2024-04-23T09:59:27.444 INFO:root:teuthology version: 0.0.1.dev269+g9481b1d 2024-04-23T09:59:27.444 DEBUG:teuthology.run:Teuthology command: teuthology --verbose --archive /home/teuthworker/archive/leonidus-2024-04-23_09:49:14-fs-wip-lusov-quiescer-distro-default-smithi/7669513 --description fs/workload/{0-centos_9.stream begin/{0-install 1-cephadm 2-logrotate 3-modules} clusters/1a11s-mds-1c-client-3node conf/{client mds mgr mon osd} mount/kclient/{base/{mount-syntax/{v1} mount overrides/{distro/stock/{centos_9.stream k-stock} ms-die-on-skipped}} ms_mode/crc wsync/no} objectstore-ec/bluestore-bitmap omap_limit/10000 overrides/{cephsqlite-timeout frag ignorelist_health ignorelist_wrongly_marked_down osd-asserts pg_health session_timeout} ranks/multi/{balancer/automatic export-check n/3 replication/always} standby-replay tasks/{0-subvolume/{with-no-extra-options} 1-check-counter 2-scrub/no 3-snaps/yes 4-flush/yes 5-quiesce/with-quiesce 6-workunit/suites/dbench}} --name leonidus-2024-04-23_09:49:14-fs-wip-lusov-quiescer-distro-default-smithi --owner scheduled_leonidus@teuthology -- /home/teuthworker/archive/leonidus-2024-04-23_09:49:14-fs-wip-lusov-quiescer-distro-default-smithi/7669513/orig.config.yaml 2024-04-23T09:59:27.505 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/ 2024-04-23T09:59:27.604 INFO:teuthology.run:Config: archive_path: /home/teuthworker/archive/leonidus-2024-04-23_09:49:14-fs-wip-lusov-quiescer-distro-default-smithi/7669513 branch: wip-lusov-quiescer description: fs/workload/{0-centos_9.stream begin/{0-install 1-cephadm 2-logrotate 3-modules} clusters/1a11s-mds-1c-client-3node conf/{client mds mgr mon osd} mount/kclient/{base/{mount-syntax/{v1} mount overrides/{distro/stock/{centos_9.stream k-stock} ms-die-on-skipped}} ms_mode/crc wsync/no} objectstore-ec/bluestore-bitmap omap_limit/10000 overrides/{cephsqlite-timeout frag ignorelist_health ignorelist_wrongly_marked_down osd-asserts pg_health session_timeout} ranks/multi/{balancer/automatic export-check n/3 replication/always} standby-replay tasks/{0-subvolume/{with-no-extra-options} 1-check-counter 2-scrub/no 3-snaps/yes 4-flush/yes 5-quiesce/with-quiesce 6-workunit/suites/dbench}} email: null first_in_suite: false job_id: '7669513' kernel: client: sha1: distro kdb: true sha1: distro ktype: distro last_in_suite: false log-rotate: ceph-mds: 10G ceph-osd: 10G machine_type: smithi mgrmodules: sequential: - print: Enabling mgr modules - exec: mon.a: - ceph mgr module enable snap_schedule - ceph config set mgr mgr/snap_schedule/allow_m_granularity true - ceph config set mgr mgr/snap_schedule/dump_on_update true name: leonidus-2024-04-23_09:49:14-fs-wip-lusov-quiescer-distro-default-smithi no_nested_subset: false nuke-on-error: true openstack: - volumes: count: 4 size: 20 - machine: disk: 200 os_type: centos os_version: 9.stream overrides: admin_socket: branch: wip-lusov-quiescer ceph: cephfs: max_mds: 3 session_timeout: 300 standby_replay: true conf: client: client mount timeout: 600 debug client: 20 debug ms: 1 rados mon op timeout: 900 rados osd op timeout: 900 global: ms die on skipped message: false mds: debug mds: 20 debug mds balancer: 20 debug ms: 1 mds bal fragment size max: 10000 mds bal merge size: 5 mds bal split bits: 3 mds bal split size: 100 mds debug frag: true mds debug scatterstat: true mds op complaint time: 180 mds verify scatter: true mds_bal_replicate_threshold: 1 osd op complaint time: 180 rados mon op timeout: 900 rados osd op timeout: 900 mgr: cephsqlite lock renewal timeout: 900000 debug cephsqlite: 20 debug client: 20 debug mgr: 20 debug ms: 1 mon: debug mon: 20 debug ms: 1 debug paxos: 20 mon down mkfs grace: 300 mon op complaint time: 120 osd: bdev async discard: true bdev enable discard: true bluestore allocator: bitmap bluestore block size: 96636764160 bluestore fsck on mount: true debug bluefs: 1/20 debug bluestore: 1/20 debug ms: 1 debug osd: 20 debug rocksdb: 4/10 mon osd backfillfull_ratio: 0.85 mon osd full ratio: 0.9 mon osd nearfull ratio: 0.8 osd failsafe full ratio: 0.95 osd objectstore: bluestore osd op complaint time: 180 osd shutdown pgref assert: true osd_max_omap_entries_per_request: 10000 flavor: default fs: xfs log-ignorelist: - \(MDS_ALL_DOWN\) - \(MDS_UP_LESS_THAN_MAX\) - FS_DEGRADED - FS_INLINE_DATA_DEPRECATED - FS_WITH_FAILED_MDS - MDS_ALL_DOWN - MDS_DAMAGE - MDS_DEGRADED - MDS_FAILED - MDS_INSUFFICIENT_STANDBY - MDS_UP_LESS_THAN_MAX - filesystem is online with fewer MDS than max_mds - POOL_APP_NOT_ENABLED - do not have an application enabled - overall HEALTH_ - Replacing daemon - deprecated feature inline_data - overall HEALTH_ - \(OSD_DOWN\) - \(OSD_ - but it is still running - is not responding - PG_AVAILABILITY - PG_DEGRADED - Reduced data availability sha1: d69f13d827952bb07bb9b7973bca14f97d764d6f subvols: create: 2 ceph-deploy: bluestore: true conf: client: log file: /var/log/ceph/ceph-$name.$pid.log mon: {} osd: bdev async discard: true bdev enable discard: true bluestore block size: 96636764160 bluestore fsck on mount: true debug bluefs: 1/20 debug bluestore: 1/20 debug rocksdb: 4/10 mon osd backfillfull_ratio: 0.85 mon osd full ratio: 0.9 mon osd nearfull ratio: 0.8 osd failsafe full ratio: 0.95 osd objectstore: bluestore fs: xfs ceph-fuse: client.0: mount_subvol_num: 0 check-counter: counters: mds: - mds.exported - mds.imported - min: 3 name: mds_cache.dir_update - min: 3 name: mds_cache.dir_update_receipt - mds.root_rsnaps - mds_server.req_mksnap_latency.avgcount - mds_server.req_rmsnap_latency.avgcount install: ceph: flavor: default sha1: d69f13d827952bb07bb9b7973bca14f97d764d6f kclient: client.0: mount_subvol_num: 1 mntopts: - ms_mode=crc - nowsync syntax: v1 selinux: allowlist: - scontext=system_u:system_r:logrotate_t:s0 - scontext=system_u:system_r:getty_t:s0 - scontext=system_u:system_r:getty_t:s0 thrashosds: bdev_inject_crash: 2 bdev_inject_crash_probability: 0.5 workunit: branch: wip-lusov-quiescer sha1: 0a6c3ed699031b80a2b419e7e795368719871394 owner: scheduled_leonidus@teuthology priority: 70 repo: https://git.ceph.com/ceph-ci.git roles: - - mon.a - mgr.x - mds.a - mds.d - mds.g - mds.j - osd.0 - osd.3 - osd.6 - osd.9 - client.0 - - mon.b - mgr.y - mds.b - mds.e - mds.h - mds.k - osd.1 - osd.4 - osd.7 - osd.10 - - mon.c - mgr.z - mds.c - mds.f - mds.i - mds.l - osd.2 - osd.5 - osd.8 - osd.11 seed: 5306 sha1: d69f13d827952bb07bb9b7973bca14f97d764d6f sleep_before_teardown: 0 subset: 200/111111 suite: fs suite_branch: wip-lusov-quiescer suite_path: /home/teuthworker/src/github.com_ceph_ceph_0a6c3ed699031b80a2b419e7e795368719871394/qa suite_relpath: qa suite_repo: https://github.com/ceph/ceph suite_sha1: 0a6c3ed699031b80a2b419e7e795368719871394 syslog: ignorelist: - WARNING*.*check_session_state - WARNING*.*__ceph_remove_cap targets: smithi114.front.sepia.ceph.com: ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBD6u3Zno1foX48Y3lv7disXfCFtZ6vZInKbwWm5POHIwlsBe/WU1nLpfVJKdmeqy65KimOB1dpAXAgWeGaqkS/w= smithi119.front.sepia.ceph.com: ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBMhNGhl/IjEU4Y4EZBE962ZOct6Ib/Whe2OtYifLHYv4C+WpTQ0fnZDi2CQLAMiR1n59QXuw1R+mG+tvEvCiy3U= smithi204.front.sepia.ceph.com: ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBAzDTSNOL1U1qUNZcRmM1lyk+mFB8Dzhokg4oU5SJYqBEDEPFordLxqpcqVZSf+Rq8cCRhZupwETwiXjgbP2VNs= tasks: - install: extra_packages: deb: - python3-cephfs - cephfs-shell - cephfs-top - cephfs-mirror rpm: - python3-cephfs - cephfs-shell - cephfs-top - cephfs-mirror extra_system_packages: deb: - bison - flex - libelf-dev - libssl-dev - network-manager - iproute2 - util-linux - dump - indent - libaio-dev - libtool-bin - uuid-dev - xfslibs-dev - postgresql - postgresql-client - postgresql-common - postgresql-contrib rpm: - bison - flex - elfutils-libelf-devel - openssl-devel - NetworkManager - iproute - util-linux - libacl-devel - libaio-devel - libattr-devel - libtool - libuuid-devel - xfsdump - xfsprogs - xfsprogs-devel - libaio-devel - libtool - libuuid-devel - xfsprogs-devel - postgresql - postgresql-server - postgresql-contrib - cephadm: roleless: false - cephadm.shell: mon.a: - ceph orch status - ceph orch ps - ceph orch ls - ceph orch host ls - ceph orch device ls - cephadm.shell: mon.a: - ceph fs dump - ceph osd dump - fs.ready: timeout: 300 - sequential: - mgrmodules - kclient: null - exec: mon.a: - ceph fs set cephfs balance_automate true - check-counter: {} - exec: mon.a: - ceph fs snap-schedule add --fs=cephfs --path=/ --snap_schedule=1m - ceph fs snap-schedule retention add --fs=cephfs --path=/ --retention-spec-or-period=6m3h - ceph fs snap-schedule status --fs=cephfs --path=/ - ceph fs snap-schedule list --fs=cephfs --path=/ --recursive=true - date +%s > START_TIME - full_sequential_finally: - exec: mon.a: - date +%s > END_TIME - START_TIME=$(cat START_TIME); END_TIME=$(cat END_TIME); DIFF_TIME=$((600-(END_TIME-START_TIME))); if [ "$DIFF_TIME" -gt 0 ]; then sleep "$DIFF_TIME"; fi - ceph fs snap-schedule status --fs=cephfs --path=/ - ceph fs snap-schedule list --fs=cephfs --path=/ --recursive=true - background_exec: mon.a: - while sleep 13; do ceph tell mds.cephfs:0 flush journal; done - quiescer: null - workunit: clients: all: - suites/dbench.sh teuthology: fragments_dropped: [] meta: {} postmerge: - local function is_kupstream() return yaml.ktype == 'upstream' end - local function is_kdistro() return yaml.ktype == 'distro' end - local function is_fuse() return false end - local function syntax_version() return yaml.overrides.kclient.syntax end teuthology_branch: main teuthology_sha1: 9481b1d62f50e7d0a4f3dd83adf6945b08d5ff17 timestamp: 2024-04-23_09:49:14 tube: smithi user: leonidus verbose: false worker_log: /home/teuthworker/archive/worker_logs/dispatcher.smithi.2226885 2024-04-23T09:59:27.605 INFO:teuthology.run:suite_path is set to /home/teuthworker/src/github.com_ceph_ceph_0a6c3ed699031b80a2b419e7e795368719871394/qa; will attempt to use it 2024-04-23T09:59:27.605 INFO:teuthology.run:Found tasks at /home/teuthworker/src/github.com_ceph_ceph_0a6c3ed699031b80a2b419e7e795368719871394/qa/tasks 2024-04-23T09:59:27.606 INFO:teuthology.run_tasks:Running task internal.check_packages... 2024-04-23T09:59:27.607 INFO:teuthology.task.internal:Checking packages... 2024-04-23T09:59:27.635 INFO:teuthology.task.internal:Checking packages for os_type 'centos', flavor 'default' and ceph hash 'd69f13d827952bb07bb9b7973bca14f97d764d6f' 2024-04-23T09:59:27.635 WARNING:teuthology.packaging:More than one of ref, tag, branch, or sha1 supplied; using branch 2024-04-23T09:59:27.635 INFO:teuthology.packaging:ref: None 2024-04-23T09:59:27.636 INFO:teuthology.packaging:tag: None 2024-04-23T09:59:27.636 INFO:teuthology.packaging:branch: wip-lusov-quiescer 2024-04-23T09:59:27.636 INFO:teuthology.packaging:sha1: d69f13d827952bb07bb9b7973bca14f97d764d6f 2024-04-23T09:59:27.636 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&ref=wip-lusov-quiescer 2024-04-23T09:59:27.888 INFO:teuthology.task.internal:Found packages for ceph version 19.0.0-3156.gd69f13d8 2024-04-23T09:59:27.890 INFO:teuthology.run_tasks:Running task internal.buildpackages_prep... 2024-04-23T09:59:27.895 INFO:teuthology.task.internal:no buildpackages task found 2024-04-23T09:59:27.895 INFO:teuthology.run_tasks:Running task internal.save_config... 2024-04-23T09:59:27.919 INFO:teuthology.task.internal:Saving configuration 2024-04-23T09:59:27.942 INFO:teuthology.run_tasks:Running task internal.check_lock... 2024-04-23T09:59:27.961 INFO:teuthology.task.internal.check_lock:Checking locks... 2024-04-23T09:59:27.982 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi114.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/leonidus-2024-04-23_09:49:14-fs-wip-lusov-quiescer-distro-default-smithi/7669513', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'centos', 'os_version': '9', 'arch': 'x86_64', 'locked': True, 'locked_since': '2024-04-23 09:51:12.360716', 'locked_by': 'scheduled_leonidus@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBD6u3Zno1foX48Y3lv7disXfCFtZ6vZInKbwWm5POHIwlsBe/WU1nLpfVJKdmeqy65KimOB1dpAXAgWeGaqkS/w='} 2024-04-23T09:59:28.001 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi119.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/leonidus-2024-04-23_09:49:14-fs-wip-lusov-quiescer-distro-default-smithi/7669513', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'centos', 'os_version': '9', 'arch': 'x86_64', 'locked': True, 'locked_since': '2024-04-23 09:51:12.357810', 'locked_by': 'scheduled_leonidus@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBMhNGhl/IjEU4Y4EZBE962ZOct6Ib/Whe2OtYifLHYv4C+WpTQ0fnZDi2CQLAMiR1n59QXuw1R+mG+tvEvCiy3U='} 2024-04-23T09:59:28.021 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi204.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/leonidus-2024-04-23_09:49:14-fs-wip-lusov-quiescer-distro-default-smithi/7669513', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'centos', 'os_version': '9', 'arch': 'x86_64', 'locked': True, 'locked_since': '2024-04-23 09:51:12.359497', 'locked_by': 'scheduled_leonidus@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBAzDTSNOL1U1qUNZcRmM1lyk+mFB8Dzhokg4oU5SJYqBEDEPFordLxqpcqVZSf+Rq8cCRhZupwETwiXjgbP2VNs='} 2024-04-23T09:59:28.021 INFO:teuthology.run_tasks:Running task internal.add_remotes... 2024-04-23T09:59:28.029 INFO:teuthology.task.internal:roles: ubuntu@smithi114.front.sepia.ceph.com - ['mon.a', 'mgr.x', 'mds.a', 'mds.d', 'mds.g', 'mds.j', 'osd.0', 'osd.3', 'osd.6', 'osd.9', 'client.0'] 2024-04-23T09:59:28.029 INFO:teuthology.task.internal:roles: ubuntu@smithi119.front.sepia.ceph.com - ['mon.b', 'mgr.y', 'mds.b', 'mds.e', 'mds.h', 'mds.k', 'osd.1', 'osd.4', 'osd.7', 'osd.10'] 2024-04-23T09:59:28.029 INFO:teuthology.task.internal:roles: ubuntu@smithi204.front.sepia.ceph.com - ['mon.c', 'mgr.z', 'mds.c', 'mds.f', 'mds.i', 'mds.l', 'osd.2', 'osd.5', 'osd.8', 'osd.11'] 2024-04-23T09:59:28.029 INFO:teuthology.run_tasks:Running task console_log... 2024-04-23T09:59:28.152 DEBUG:teuthology.exit:Installing handler: Handler(exiter=, func=.kill_console_loggers at 0x7f839ceca0d0>, signals=[15]) 2024-04-23T09:59:28.153 INFO:teuthology.run_tasks:Running task internal.connect... 2024-04-23T09:59:28.159 INFO:teuthology.task.internal:Opening connections... 2024-04-23T09:59:28.159 DEBUG:teuthology.task.internal:connecting to ubuntu@smithi114.front.sepia.ceph.com 2024-04-23T09:59:28.162 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi114.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-04-23T09:59:28.237 DEBUG:teuthology.task.internal:connecting to ubuntu@smithi119.front.sepia.ceph.com 2024-04-23T09:59:28.237 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi119.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-04-23T09:59:28.310 DEBUG:teuthology.task.internal:connecting to ubuntu@smithi204.front.sepia.ceph.com 2024-04-23T09:59:28.311 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi204.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-04-23T09:59:28.385 INFO:teuthology.run_tasks:Running task internal.push_inventory... 2024-04-23T09:59:28.391 DEBUG:teuthology.orchestra.run.smithi114:> uname -m 2024-04-23T09:59:28.409 INFO:teuthology.orchestra.run.smithi114.stdout:x86_64 2024-04-23T09:59:28.410 DEBUG:teuthology.orchestra.run.smithi114:> cat /etc/os-release 2024-04-23T09:59:28.467 INFO:teuthology.orchestra.run.smithi114.stdout:NAME="CentOS Stream" 2024-04-23T09:59:28.467 INFO:teuthology.orchestra.run.smithi114.stdout:VERSION="9" 2024-04-23T09:59:28.467 INFO:teuthology.orchestra.run.smithi114.stdout:ID="centos" 2024-04-23T09:59:28.467 INFO:teuthology.orchestra.run.smithi114.stdout:ID_LIKE="rhel fedora" 2024-04-23T09:59:28.467 INFO:teuthology.orchestra.run.smithi114.stdout:VERSION_ID="9" 2024-04-23T09:59:28.467 INFO:teuthology.orchestra.run.smithi114.stdout:PLATFORM_ID="platform:el9" 2024-04-23T09:59:28.467 INFO:teuthology.orchestra.run.smithi114.stdout:PRETTY_NAME="CentOS Stream 9" 2024-04-23T09:59:28.467 INFO:teuthology.orchestra.run.smithi114.stdout:ANSI_COLOR="0;31" 2024-04-23T09:59:28.467 INFO:teuthology.orchestra.run.smithi114.stdout:LOGO="fedora-logo-icon" 2024-04-23T09:59:28.467 INFO:teuthology.orchestra.run.smithi114.stdout:CPE_NAME="cpe:/o:centos:centos:9" 2024-04-23T09:59:28.468 INFO:teuthology.orchestra.run.smithi114.stdout:HOME_URL="https://centos.org/" 2024-04-23T09:59:28.468 INFO:teuthology.orchestra.run.smithi114.stdout:BUG_REPORT_URL="https://bugzilla.redhat.com/" 2024-04-23T09:59:28.468 INFO:teuthology.orchestra.run.smithi114.stdout:REDHAT_SUPPORT_PRODUCT="Red Hat Enterprise Linux 9" 2024-04-23T09:59:28.468 INFO:teuthology.orchestra.run.smithi114.stdout:REDHAT_SUPPORT_PRODUCT_VERSION="CentOS Stream" 2024-04-23T09:59:28.468 INFO:teuthology.lock.ops:Updating smithi114.front.sepia.ceph.com on lock server 2024-04-23T09:59:28.486 DEBUG:teuthology.orchestra.run.smithi119:> uname -m 2024-04-23T09:59:28.504 INFO:teuthology.orchestra.run.smithi119.stdout:x86_64 2024-04-23T09:59:28.504 DEBUG:teuthology.orchestra.run.smithi119:> cat /etc/os-release 2024-04-23T09:59:28.564 INFO:teuthology.orchestra.run.smithi119.stdout:NAME="CentOS Stream" 2024-04-23T09:59:28.564 INFO:teuthology.orchestra.run.smithi119.stdout:VERSION="9" 2024-04-23T09:59:28.564 INFO:teuthology.orchestra.run.smithi119.stdout:ID="centos" 2024-04-23T09:59:28.564 INFO:teuthology.orchestra.run.smithi119.stdout:ID_LIKE="rhel fedora" 2024-04-23T09:59:28.565 INFO:teuthology.orchestra.run.smithi119.stdout:VERSION_ID="9" 2024-04-23T09:59:28.565 INFO:teuthology.orchestra.run.smithi119.stdout:PLATFORM_ID="platform:el9" 2024-04-23T09:59:28.565 INFO:teuthology.orchestra.run.smithi119.stdout:PRETTY_NAME="CentOS Stream 9" 2024-04-23T09:59:28.565 INFO:teuthology.orchestra.run.smithi119.stdout:ANSI_COLOR="0;31" 2024-04-23T09:59:28.565 INFO:teuthology.orchestra.run.smithi119.stdout:LOGO="fedora-logo-icon" 2024-04-23T09:59:28.565 INFO:teuthology.orchestra.run.smithi119.stdout:CPE_NAME="cpe:/o:centos:centos:9" 2024-04-23T09:59:28.566 INFO:teuthology.orchestra.run.smithi119.stdout:HOME_URL="https://centos.org/" 2024-04-23T09:59:28.566 INFO:teuthology.orchestra.run.smithi119.stdout:BUG_REPORT_URL="https://bugzilla.redhat.com/" 2024-04-23T09:59:28.566 INFO:teuthology.orchestra.run.smithi119.stdout:REDHAT_SUPPORT_PRODUCT="Red Hat Enterprise Linux 9" 2024-04-23T09:59:28.566 INFO:teuthology.orchestra.run.smithi119.stdout:REDHAT_SUPPORT_PRODUCT_VERSION="CentOS Stream" 2024-04-23T09:59:28.567 INFO:teuthology.lock.ops:Updating smithi119.front.sepia.ceph.com on lock server 2024-04-23T09:59:28.586 DEBUG:teuthology.orchestra.run.smithi204:> uname -m 2024-04-23T09:59:28.605 INFO:teuthology.orchestra.run.smithi204.stdout:x86_64 2024-04-23T09:59:28.605 DEBUG:teuthology.orchestra.run.smithi204:> cat /etc/os-release 2024-04-23T09:59:28.664 INFO:teuthology.orchestra.run.smithi204.stdout:NAME="CentOS Stream" 2024-04-23T09:59:28.664 INFO:teuthology.orchestra.run.smithi204.stdout:VERSION="9" 2024-04-23T09:59:28.664 INFO:teuthology.orchestra.run.smithi204.stdout:ID="centos" 2024-04-23T09:59:28.664 INFO:teuthology.orchestra.run.smithi204.stdout:ID_LIKE="rhel fedora" 2024-04-23T09:59:28.664 INFO:teuthology.orchestra.run.smithi204.stdout:VERSION_ID="9" 2024-04-23T09:59:28.664 INFO:teuthology.orchestra.run.smithi204.stdout:PLATFORM_ID="platform:el9" 2024-04-23T09:59:28.665 INFO:teuthology.orchestra.run.smithi204.stdout:PRETTY_NAME="CentOS Stream 9" 2024-04-23T09:59:28.665 INFO:teuthology.orchestra.run.smithi204.stdout:ANSI_COLOR="0;31" 2024-04-23T09:59:28.665 INFO:teuthology.orchestra.run.smithi204.stdout:LOGO="fedora-logo-icon" 2024-04-23T09:59:28.665 INFO:teuthology.orchestra.run.smithi204.stdout:CPE_NAME="cpe:/o:centos:centos:9" 2024-04-23T09:59:28.665 INFO:teuthology.orchestra.run.smithi204.stdout:HOME_URL="https://centos.org/" 2024-04-23T09:59:28.665 INFO:teuthology.orchestra.run.smithi204.stdout:BUG_REPORT_URL="https://bugzilla.redhat.com/" 2024-04-23T09:59:28.665 INFO:teuthology.orchestra.run.smithi204.stdout:REDHAT_SUPPORT_PRODUCT="Red Hat Enterprise Linux 9" 2024-04-23T09:59:28.665 INFO:teuthology.orchestra.run.smithi204.stdout:REDHAT_SUPPORT_PRODUCT_VERSION="CentOS Stream" 2024-04-23T09:59:28.665 INFO:teuthology.lock.ops:Updating smithi204.front.sepia.ceph.com on lock server 2024-04-23T09:59:28.685 INFO:teuthology.run_tasks:Running task internal.serialize_remote_roles... 2024-04-23T09:59:28.695 INFO:teuthology.run_tasks:Running task internal.check_conflict... 2024-04-23T09:59:28.731 INFO:teuthology.task.internal:Checking for old test directory... 2024-04-23T09:59:28.731 DEBUG:teuthology.orchestra.run.smithi114:> test '!' -e /home/ubuntu/cephtest 2024-04-23T09:59:28.733 DEBUG:teuthology.orchestra.run.smithi119:> test '!' -e /home/ubuntu/cephtest 2024-04-23T09:59:28.736 DEBUG:teuthology.orchestra.run.smithi204:> test '!' -e /home/ubuntu/cephtest 2024-04-23T09:59:28.752 INFO:teuthology.run_tasks:Running task internal.check_ceph_data... 2024-04-23T09:59:28.792 INFO:teuthology.task.internal:Checking for non-empty /var/lib/ceph... 2024-04-23T09:59:28.792 DEBUG:teuthology.orchestra.run.smithi114:> test -z $(ls -A /var/lib/ceph) 2024-04-23T09:59:28.794 DEBUG:teuthology.orchestra.run.smithi119:> test -z $(ls -A /var/lib/ceph) 2024-04-23T09:59:28.796 DEBUG:teuthology.orchestra.run.smithi204:> test -z $(ls -A /var/lib/ceph) 2024-04-23T09:59:28.875 INFO:teuthology.run_tasks:Running task internal.vm_setup... 2024-04-23T09:59:28.954 INFO:teuthology.run_tasks:Running task kernel... 2024-04-23T09:59:28.969 INFO:teuthology.task.kernel:normalize config orig: {'client': {'sha1': 'distro'}, 'kdb': True, 'sha1': 'distro'} 2024-04-23T09:59:28.969 INFO:teuthology.task.kernel:normalize config final: {'client.0': {'sha1': 'distro'}} 2024-04-23T09:59:28.969 INFO:teuthology.task.kernel:config {'client.0': {'sha1': 'distro'}}, timeout 300 2024-04-23T09:59:28.969 DEBUG:teuthology.orchestra.run.smithi114:> test -f /run/.containerenv -o -f /.dockerenv 2024-04-23T09:59:28.986 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-23T09:59:28.986 DEBUG:teuthology.orchestra.run.smithi114:> uname -r 2024-04-23T09:59:29.044 INFO:teuthology.orchestra.run.smithi114.stdout:5.14.0-437.el9.x86_64 2024-04-23T09:59:29.045 INFO:teuthology.task.kernel:Running kernel on smithi114: 5.14.0-437.el9.x86_64 2024-04-23T09:59:29.045 DEBUG:teuthology.orchestra.run.smithi114:> sudo yum install -y kernel 2024-04-23T09:59:31.921 INFO:teuthology.orchestra.run.smithi114.stdout:CentOS Stream 9 - BaseOS 6.6 MB/s | 8.0 MB 00:01 2024-04-23T09:59:37.273 INFO:teuthology.orchestra.run.smithi114.stdout:CentOS Stream 9 - AppStream 4.4 MB/s | 19 MB 00:04 2024-04-23T09:59:49.474 INFO:teuthology.orchestra.run.smithi114.stdout:CentOS Stream 9 - CRB 737 kB/s | 6.2 MB 00:08 2024-04-23T09:59:51.030 INFO:teuthology.orchestra.run.smithi114.stdout:CentOS Stream 9 - Extras packages 40 kB/s | 16 kB 00:00 2024-04-23T09:59:52.274 INFO:teuthology.orchestra.run.smithi114.stdout:Extra Packages for Enterprise Linux 19 MB/s | 21 MB 00:01 2024-04-23T09:59:57.365 INFO:teuthology.orchestra.run.smithi114.stdout:lab-extras 33 kB/s | 1.7 kB 00:00 2024-04-23T09:59:59.245 INFO:teuthology.orchestra.run.smithi114.stdout:Package kernel-5.14.0-239.el9.x86_64 is already installed. 2024-04-23T09:59:59.246 INFO:teuthology.orchestra.run.smithi114.stdout:Package kernel-5.14.0-437.el9.x86_64 is already installed. 2024-04-23T09:59:59.335 INFO:teuthology.orchestra.run.smithi114.stdout:Dependencies resolved. 2024-04-23T09:59:59.336 INFO:teuthology.orchestra.run.smithi114.stdout:Nothing to do. 2024-04-23T09:59:59.336 INFO:teuthology.orchestra.run.smithi114.stdout:Complete! 2024-04-23T09:59:59.447 DEBUG:teuthology.orchestra.run.smithi114:> echo no | sudo yum reinstall kernel || true 2024-04-23T09:59:59.938 INFO:teuthology.orchestra.run.smithi114.stdout:Last metadata expiration check: 0:00:02 ago on Tue 23 Apr 2024 09:59:57 AM UTC. 2024-04-23T10:00:00.058 INFO:teuthology.orchestra.run.smithi114.stdout:Dependencies resolved. 2024-04-23T10:00:00.079 INFO:teuthology.orchestra.run.smithi114.stdout:================================================================================ 2024-04-23T10:00:00.079 INFO:teuthology.orchestra.run.smithi114.stdout: Package Architecture Version Repository Size 2024-04-23T10:00:00.079 INFO:teuthology.orchestra.run.smithi114.stdout:================================================================================ 2024-04-23T10:00:00.079 INFO:teuthology.orchestra.run.smithi114.stdout:Reinstalling: 2024-04-23T10:00:00.079 INFO:teuthology.orchestra.run.smithi114.stdout: kernel x86_64 5.14.0-437.el9 baseos 255 k 2024-04-23T10:00:00.079 INFO:teuthology.orchestra.run.smithi114.stdout: 2024-04-23T10:00:00.079 INFO:teuthology.orchestra.run.smithi114.stdout:Transaction Summary 2024-04-23T10:00:00.079 INFO:teuthology.orchestra.run.smithi114.stdout:================================================================================ 2024-04-23T10:00:00.079 INFO:teuthology.orchestra.run.smithi114.stdout: 2024-04-23T10:00:00.079 INFO:teuthology.orchestra.run.smithi114.stdout:Total download size: 255 k 2024-04-23T10:00:00.079 INFO:teuthology.orchestra.run.smithi114.stdout:Installed size: 0 2024-04-23T10:00:00.080 INFO:teuthology.orchestra.run.smithi114.stderr:Operation aborted. 2024-04-23T10:00:00.133 INFO:teuthology.orchestra.run.smithi114.stdout:Is this ok [y/N]: 2024-04-23T10:00:00.133 DEBUG:teuthology.orchestra.run.smithi114:> sudo yum reinstall -y kernel || true 2024-04-23T10:00:00.625 INFO:teuthology.orchestra.run.smithi114.stdout:Last metadata expiration check: 0:00:03 ago on Tue 23 Apr 2024 09:59:57 AM UTC. 2024-04-23T10:00:00.743 INFO:teuthology.orchestra.run.smithi114.stdout:Dependencies resolved. 2024-04-23T10:00:00.744 INFO:teuthology.orchestra.run.smithi114.stdout:================================================================================ 2024-04-23T10:00:00.744 INFO:teuthology.orchestra.run.smithi114.stdout: Package Architecture Version Repository Size 2024-04-23T10:00:00.744 INFO:teuthology.orchestra.run.smithi114.stdout:================================================================================ 2024-04-23T10:00:00.744 INFO:teuthology.orchestra.run.smithi114.stdout:Reinstalling: 2024-04-23T10:00:00.744 INFO:teuthology.orchestra.run.smithi114.stdout: kernel x86_64 5.14.0-437.el9 baseos 255 k 2024-04-23T10:00:00.744 INFO:teuthology.orchestra.run.smithi114.stdout: 2024-04-23T10:00:00.744 INFO:teuthology.orchestra.run.smithi114.stdout:Transaction Summary 2024-04-23T10:00:00.744 INFO:teuthology.orchestra.run.smithi114.stdout:================================================================================ 2024-04-23T10:00:00.744 INFO:teuthology.orchestra.run.smithi114.stdout: 2024-04-23T10:00:00.744 INFO:teuthology.orchestra.run.smithi114.stdout:Total download size: 255 k 2024-04-23T10:00:00.744 INFO:teuthology.orchestra.run.smithi114.stdout:Installed size: 0 2024-04-23T10:00:00.744 INFO:teuthology.orchestra.run.smithi114.stdout:Downloading Packages: 2024-04-23T10:00:01.149 INFO:teuthology.orchestra.run.smithi114.stdout:kernel-5.14.0-437.el9.x86_64.rpm 976 kB/s | 255 kB 00:00 2024-04-23T10:00:01.150 INFO:teuthology.orchestra.run.smithi114.stdout:-------------------------------------------------------------------------------- 2024-04-23T10:00:01.150 INFO:teuthology.orchestra.run.smithi114.stdout:Total 628 kB/s | 255 kB 00:00 2024-04-23T10:00:01.170 INFO:teuthology.orchestra.run.smithi114.stdout:Running transaction check 2024-04-23T10:00:01.371 INFO:teuthology.orchestra.run.smithi114.stdout:Transaction check succeeded. 2024-04-23T10:00:01.372 INFO:teuthology.orchestra.run.smithi114.stdout:Running transaction test 2024-04-23T10:00:01.377 INFO:teuthology.orchestra.run.smithi114.stdout:Transaction test succeeded. 2024-04-23T10:00:01.377 INFO:teuthology.orchestra.run.smithi114.stdout:Running transaction 2024-04-23T10:00:01.515 INFO:teuthology.orchestra.run.smithi114.stdout: Preparing : 1/1 2024-04-23T10:00:01.601 INFO:teuthology.orchestra.run.smithi114.stdout: Reinstalling : kernel-5.14.0-437.el9.x86_64 1/2 2024-04-23T10:00:01.601 INFO:teuthology.orchestra.run.smithi114.stdout: Cleanup : kernel-5.14.0-437.el9.x86_64 2/2 2024-04-23T10:00:02.241 INFO:teuthology.orchestra.run.smithi114.stdout: Running scriptlet: kernel-5.14.0-437.el9.x86_64 2/2 2024-04-23T10:00:02.241 INFO:teuthology.orchestra.run.smithi114.stdout: Verifying : kernel-5.14.0-437.el9.x86_64 1/2 2024-04-23T10:00:02.896 INFO:teuthology.orchestra.run.smithi114.stdout: Verifying : kernel-5.14.0-437.el9.x86_64 2/2 2024-04-23T10:00:02.896 INFO:teuthology.orchestra.run.smithi114.stdout: 2024-04-23T10:00:02.896 INFO:teuthology.orchestra.run.smithi114.stdout:Reinstalled: 2024-04-23T10:00:02.896 INFO:teuthology.orchestra.run.smithi114.stdout: kernel-5.14.0-437.el9.x86_64 2024-04-23T10:00:02.896 INFO:teuthology.orchestra.run.smithi114.stdout: 2024-04-23T10:00:02.897 INFO:teuthology.orchestra.run.smithi114.stdout:Complete! 2024-04-23T10:00:03.075 DEBUG:teuthology.orchestra.run.smithi114:> rpm -q kernel | sort -rV | head -n 1 2024-04-23T10:00:03.162 INFO:teuthology.orchestra.run.smithi114.stdout:kernel-5.14.0-437.el9.x86_64 2024-04-23T10:00:03.162 DEBUG:teuthology.task.kernel:get_latest_image_version_rpm: 5.14.0-437.el9.x86_64 2024-04-23T10:00:03.163 INFO:teuthology.task.kernel:Newest distro kernel installed and running 2024-04-23T10:00:03.163 DEBUG:teuthology.parallel:result is None 2024-04-23T10:00:03.163 INFO:teuthology.run_tasks:Running task internal.base... 2024-04-23T10:00:03.170 INFO:teuthology.task.internal:Creating test directory... 2024-04-23T10:00:03.171 DEBUG:teuthology.orchestra.run.smithi114:> mkdir -p -m0755 -- /home/ubuntu/cephtest 2024-04-23T10:00:03.173 DEBUG:teuthology.orchestra.run.smithi119:> mkdir -p -m0755 -- /home/ubuntu/cephtest 2024-04-23T10:00:03.175 DEBUG:teuthology.orchestra.run.smithi204:> mkdir -p -m0755 -- /home/ubuntu/cephtest 2024-04-23T10:00:03.194 INFO:teuthology.run_tasks:Running task internal.archive_upload... 2024-04-23T10:00:03.200 INFO:teuthology.run_tasks:Running task internal.archive... 2024-04-23T10:00:03.210 INFO:teuthology.task.internal:Creating archive directory... 2024-04-23T10:00:03.210 DEBUG:teuthology.orchestra.run.smithi114:> install -d -m0755 -- /home/ubuntu/cephtest/archive 2024-04-23T10:00:03.228 DEBUG:teuthology.orchestra.run.smithi119:> install -d -m0755 -- /home/ubuntu/cephtest/archive 2024-04-23T10:00:03.234 DEBUG:teuthology.orchestra.run.smithi204:> install -d -m0755 -- /home/ubuntu/cephtest/archive 2024-04-23T10:00:03.540 INFO:teuthology.run_tasks:Running task internal.coredump... 2024-04-23T10:00:03.547 INFO:teuthology.task.internal:Enabling coredump saving... 2024-04-23T10:00:03.547 DEBUG:teuthology.orchestra.run.smithi119:> test -f /run/.containerenv -o -f /.dockerenv 2024-04-23T10:00:03.563 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-23T10:00:03.564 DEBUG:teuthology.orchestra.run.smithi204:> test -f /run/.containerenv -o -f /.dockerenv 2024-04-23T10:00:03.580 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-23T10:00:03.581 DEBUG:teuthology.orchestra.run.smithi114:> install -d -m0755 -- /home/ubuntu/cephtest/archive/coredump && sudo sysctl -w kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core && echo kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core | sudo tee -a /etc/sysctl.conf 2024-04-23T10:00:03.583 DEBUG:teuthology.orchestra.run.smithi119:> install -d -m0755 -- /home/ubuntu/cephtest/archive/coredump && sudo sysctl -w kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core && echo kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core | sudo tee -a /etc/sysctl.conf 2024-04-23T10:00:03.607 DEBUG:teuthology.orchestra.run.smithi204:> install -d -m0755 -- /home/ubuntu/cephtest/archive/coredump && sudo sysctl -w kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core && echo kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core | sudo tee -a /etc/sysctl.conf 2024-04-23T10:00:03.639 INFO:teuthology.orchestra.run.smithi119.stdout:kernel.core_pattern = /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-04-23T10:00:03.639 INFO:teuthology.orchestra.run.smithi114.stdout:kernel.core_pattern = /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-04-23T10:00:03.653 INFO:teuthology.orchestra.run.smithi114.stdout:kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-04-23T10:00:03.658 INFO:teuthology.orchestra.run.smithi119.stdout:kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-04-23T10:00:03.674 INFO:teuthology.orchestra.run.smithi204.stdout:kernel.core_pattern = /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-04-23T10:00:03.689 INFO:teuthology.orchestra.run.smithi204.stdout:kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-04-23T10:00:03.691 INFO:teuthology.run_tasks:Running task internal.sudo... 2024-04-23T10:00:03.697 INFO:teuthology.task.internal:Configuring sudo... 2024-04-23T10:00:03.697 DEBUG:teuthology.orchestra.run.smithi114:> sudo sed -i.orig.teuthology -e 's/^\([^#]*\) \(requiretty\)/\1 !\2/g' -e 's/^\([^#]*\) !\(visiblepw\)/\1 \2/g' /etc/sudoers 2024-04-23T10:00:03.699 DEBUG:teuthology.orchestra.run.smithi119:> sudo sed -i.orig.teuthology -e 's/^\([^#]*\) \(requiretty\)/\1 !\2/g' -e 's/^\([^#]*\) !\(visiblepw\)/\1 \2/g' /etc/sudoers 2024-04-23T10:00:03.702 DEBUG:teuthology.orchestra.run.smithi204:> sudo sed -i.orig.teuthology -e 's/^\([^#]*\) \(requiretty\)/\1 !\2/g' -e 's/^\([^#]*\) !\(visiblepw\)/\1 \2/g' /etc/sudoers 2024-04-23T10:00:03.729 INFO:teuthology.run_tasks:Running task internal.syslog... 2024-04-23T10:00:03.749 INFO:teuthology.task.internal.syslog:Starting syslog monitoring... 2024-04-23T10:00:03.749 DEBUG:teuthology.orchestra.run.smithi114:> mkdir -p -m0755 -- /home/ubuntu/cephtest/archive/syslog 2024-04-23T10:00:03.768 DEBUG:teuthology.orchestra.run.smithi119:> mkdir -p -m0755 -- /home/ubuntu/cephtest/archive/syslog 2024-04-23T10:00:03.772 DEBUG:teuthology.orchestra.run.smithi204:> mkdir -p -m0755 -- /home/ubuntu/cephtest/archive/syslog 2024-04-23T10:00:03.791 DEBUG:teuthology.orchestra.run.smithi114:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/kern.log 2024-04-23T10:00:03.856 DEBUG:teuthology.orchestra.run.smithi114:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/kern.log 2024-04-23T10:00:03.912 DEBUG:teuthology.orchestra.run.smithi114:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/misc.log 2024-04-23T10:00:03.995 DEBUG:teuthology.orchestra.run.smithi114:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/misc.log 2024-04-23T10:00:04.033 DEBUG:teuthology.orchestra.run.smithi114:> set -ex 2024-04-23T10:00:04.033 DEBUG:teuthology.orchestra.run.smithi114:> sudo dd of=/etc/rsyslog.d/80-cephtest.conf 2024-04-23T10:00:04.096 DEBUG:teuthology.orchestra.run.smithi119:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/kern.log 2024-04-23T10:00:04.142 DEBUG:teuthology.orchestra.run.smithi119:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/kern.log 2024-04-23T10:00:04.192 DEBUG:teuthology.orchestra.run.smithi119:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/misc.log 2024-04-23T10:00:04.275 DEBUG:teuthology.orchestra.run.smithi119:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/misc.log 2024-04-23T10:00:04.318 DEBUG:teuthology.orchestra.run.smithi119:> set -ex 2024-04-23T10:00:04.318 DEBUG:teuthology.orchestra.run.smithi119:> sudo dd of=/etc/rsyslog.d/80-cephtest.conf 2024-04-23T10:00:04.383 DEBUG:teuthology.orchestra.run.smithi204:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/kern.log 2024-04-23T10:00:04.428 DEBUG:teuthology.orchestra.run.smithi204:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/kern.log 2024-04-23T10:00:04.481 DEBUG:teuthology.orchestra.run.smithi204:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/misc.log 2024-04-23T10:00:04.558 DEBUG:teuthology.orchestra.run.smithi204:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/misc.log 2024-04-23T10:00:04.606 DEBUG:teuthology.orchestra.run.smithi204:> set -ex 2024-04-23T10:00:04.606 DEBUG:teuthology.orchestra.run.smithi204:> sudo dd of=/etc/rsyslog.d/80-cephtest.conf 2024-04-23T10:00:04.671 DEBUG:teuthology.orchestra.run.smithi114:> sudo service rsyslog restart 2024-04-23T10:00:04.674 DEBUG:teuthology.orchestra.run.smithi119:> sudo service rsyslog restart 2024-04-23T10:00:04.676 DEBUG:teuthology.orchestra.run.smithi204:> sudo service rsyslog restart 2024-04-23T10:00:04.709 INFO:teuthology.orchestra.run.smithi119.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2024-04-23T10:00:04.720 INFO:teuthology.orchestra.run.smithi114.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2024-04-23T10:00:04.745 INFO:teuthology.orchestra.run.smithi204.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2024-04-23T10:00:05.113 INFO:teuthology.run_tasks:Running task internal.timer... 2024-04-23T10:00:05.120 INFO:teuthology.task.internal:Starting timer... 2024-04-23T10:00:05.120 INFO:teuthology.run_tasks:Running task pcp... 2024-04-23T10:00:05.156 INFO:teuthology.run_tasks:Running task selinux... 2024-04-23T10:00:05.165 DEBUG:teuthology.task:Applying overrides for task selinux: {'allowlist': ['scontext=system_u:system_r:logrotate_t:s0', 'scontext=system_u:system_r:getty_t:s0', 'scontext=system_u:system_r:getty_t:s0']} 2024-04-23T10:00:05.166 DEBUG:teuthology.orchestra.run.smithi114:> sudo service auditd rotate 2024-04-23T10:00:05.228 INFO:teuthology.orchestra.run.smithi114.stdout:Rotating logs: 2024-04-23T10:00:05.230 DEBUG:teuthology.orchestra.run.smithi119:> sudo service auditd rotate 2024-04-23T10:00:05.296 INFO:teuthology.orchestra.run.smithi119.stdout:Rotating logs: 2024-04-23T10:00:05.298 DEBUG:teuthology.orchestra.run.smithi204:> sudo service auditd rotate 2024-04-23T10:00:05.377 INFO:teuthology.orchestra.run.smithi204.stdout:Rotating logs: 2024-04-23T10:00:05.379 DEBUG:teuthology.task.selinux:Getting current SELinux state 2024-04-23T10:00:05.379 DEBUG:teuthology.orchestra.run.smithi114:> /usr/sbin/getenforce 2024-04-23T10:00:05.400 INFO:teuthology.orchestra.run.smithi114.stdout:Permissive 2024-04-23T10:00:05.400 DEBUG:teuthology.orchestra.run.smithi119:> /usr/sbin/getenforce 2024-04-23T10:00:05.425 INFO:teuthology.orchestra.run.smithi119.stdout:Permissive 2024-04-23T10:00:05.425 DEBUG:teuthology.orchestra.run.smithi204:> /usr/sbin/getenforce 2024-04-23T10:00:05.458 INFO:teuthology.orchestra.run.smithi204.stdout:Permissive 2024-04-23T10:00:05.458 DEBUG:teuthology.task.selinux:Existing SELinux modes: {'ubuntu@smithi114.front.sepia.ceph.com': 'permissive', 'ubuntu@smithi119.front.sepia.ceph.com': 'permissive', 'ubuntu@smithi204.front.sepia.ceph.com': 'permissive'} 2024-04-23T10:00:05.458 DEBUG:teuthology.orchestra.run.smithi114:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 -e scontext=system_u:system_r:logrotate_t:s0 -e scontext=system_u:system_r:getty_t:s0 -e scontext=system_u:system_r:getty_t:s0 2024-04-23T10:00:05.483 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-23T10:00:05.483 DEBUG:teuthology.orchestra.run.smithi119:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 -e scontext=system_u:system_r:logrotate_t:s0 -e scontext=system_u:system_r:getty_t:s0 -e scontext=system_u:system_r:getty_t:s0 2024-04-23T10:00:05.511 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-23T10:00:05.512 DEBUG:teuthology.orchestra.run.smithi204:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 -e scontext=system_u:system_r:logrotate_t:s0 -e scontext=system_u:system_r:getty_t:s0 -e scontext=system_u:system_r:getty_t:s0 2024-04-23T10:00:05.543 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-23T10:00:05.543 INFO:teuthology.task.selinux:Putting SELinux into permissive mode 2024-04-23T10:00:05.544 DEBUG:teuthology.orchestra.run.smithi114:> sudo /usr/sbin/setenforce permissive 2024-04-23T10:00:05.568 DEBUG:teuthology.orchestra.run.smithi119:> sudo /usr/sbin/setenforce permissive 2024-04-23T10:00:05.603 DEBUG:teuthology.orchestra.run.smithi204:> sudo /usr/sbin/setenforce permissive 2024-04-23T10:00:05.630 INFO:teuthology.run_tasks:Running task ansible.cephlab... 2024-04-23T10:00:05.645 DEBUG:teuthology.repo_utils:Resetting repo at /home/teuthworker/src/git.ceph.com_ceph-cm-ansible_main to origin/main 2024-04-23T10:00:05.676 INFO:teuthology.task.ansible:Playbook: [{'import_playbook': 'ansible_managed.yml'}, {'import_playbook': 'teuthology.yml'}, {'hosts': 'testnodes', 'tasks': [{'set_fact': {'ran_from_cephlab_playbook': True}}]}, {'import_playbook': 'testnodes.yml'}, {'import_playbook': 'container-host.yml'}, {'import_playbook': 'cobbler.yml'}, {'import_playbook': 'paddles.yml'}, {'import_playbook': 'pulpito.yml'}, {'hosts': 'testnodes', 'become': True, 'tasks': [{'name': 'Touch /ceph-qa-ready', 'file': {'path': '/ceph-qa-ready', 'state': 'touch'}, 'when': 'ran_from_cephlab_playbook|bool'}]}] 2024-04-23T10:00:05.677 DEBUG:teuthology.task.ansible:Running ansible-playbook -v --extra-vars '{"ansible_ssh_user": "ubuntu"}' -i /etc/ansible/hosts --limit smithi114.front.sepia.ceph.com,smithi119.front.sepia.ceph.com,smithi204.front.sepia.ceph.com /home/teuthworker/src/git.ceph.com_ceph-cm-ansible_main/cephlab.yml 2024-04-23T10:04:38.844 DEBUG:teuthology.task.ansible:Reconnecting to [Remote(name='ubuntu@smithi114.front.sepia.ceph.com'), Remote(name='ubuntu@smithi119.front.sepia.ceph.com'), Remote(name='ubuntu@smithi204.front.sepia.ceph.com')] 2024-04-23T10:04:38.845 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi114.front.sepia.ceph.com' 2024-04-23T10:04:38.846 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi114.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-04-23T10:04:38.920 DEBUG:teuthology.orchestra.run.smithi114:> true 2024-04-23T10:04:39.006 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi114.front.sepia.ceph.com' 2024-04-23T10:04:39.006 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi119.front.sepia.ceph.com' 2024-04-23T10:04:39.009 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi119.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-04-23T10:04:39.081 DEBUG:teuthology.orchestra.run.smithi119:> true 2024-04-23T10:04:39.161 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi119.front.sepia.ceph.com' 2024-04-23T10:04:39.162 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi204.front.sepia.ceph.com' 2024-04-23T10:04:39.162 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi204.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-04-23T10:04:39.236 DEBUG:teuthology.orchestra.run.smithi204:> true 2024-04-23T10:04:39.275 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi204.front.sepia.ceph.com' 2024-04-23T10:04:39.275 INFO:teuthology.run_tasks:Running task clock... 2024-04-23T10:04:39.287 INFO:teuthology.task.clock:Syncing clocks and checking initial clock skew... 2024-04-23T10:04:39.288 INFO:teuthology.orchestra.run:Running command with timeout 360 2024-04-23T10:04:39.288 DEBUG:teuthology.orchestra.run.smithi114:> sudo systemctl stop ntp.service || sudo systemctl stop ntpd.service || sudo systemctl stop chronyd.service ; sudo ntpd -gq || sudo chronyc makestep ; sudo systemctl start ntp.service || sudo systemctl start ntpd.service || sudo systemctl start chronyd.service ; PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-04-23T10:04:39.290 INFO:teuthology.orchestra.run:Running command with timeout 360 2024-04-23T10:04:39.291 DEBUG:teuthology.orchestra.run.smithi119:> sudo systemctl stop ntp.service || sudo systemctl stop ntpd.service || sudo systemctl stop chronyd.service ; sudo ntpd -gq || sudo chronyc makestep ; sudo systemctl start ntp.service || sudo systemctl start ntpd.service || sudo systemctl start chronyd.service ; PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-04-23T10:04:39.293 INFO:teuthology.orchestra.run:Running command with timeout 360 2024-04-23T10:04:39.293 DEBUG:teuthology.orchestra.run.smithi204:> sudo systemctl stop ntp.service || sudo systemctl stop ntpd.service || sudo systemctl stop chronyd.service ; sudo ntpd -gq || sudo chronyc makestep ; sudo systemctl start ntp.service || sudo systemctl start ntpd.service || sudo systemctl start chronyd.service ; PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-04-23T10:04:39.318 INFO:teuthology.orchestra.run.smithi119.stderr:Failed to stop ntp.service: Unit ntp.service not loaded. 2024-04-23T10:04:39.318 INFO:teuthology.orchestra.run.smithi114.stderr:Failed to stop ntp.service: Unit ntp.service not loaded. 2024-04-23T10:04:39.331 INFO:teuthology.orchestra.run.smithi119.stderr:Failed to stop ntpd.service: Unit ntpd.service not loaded. 2024-04-23T10:04:39.333 INFO:teuthology.orchestra.run.smithi114.stderr:Failed to stop ntpd.service: Unit ntpd.service not loaded. 2024-04-23T10:04:39.343 INFO:teuthology.orchestra.run.smithi204.stderr:Failed to stop ntp.service: Unit ntp.service not loaded. 2024-04-23T10:04:39.353 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: ntpd: command not found 2024-04-23T10:04:39.356 INFO:teuthology.orchestra.run.smithi204.stderr:Failed to stop ntpd.service: Unit ntpd.service not loaded. 2024-04-23T10:04:39.357 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: ntpd: command not found 2024-04-23T10:04:39.364 INFO:teuthology.orchestra.run.smithi119.stdout:506 Cannot talk to daemon 2024-04-23T10:04:39.367 INFO:teuthology.orchestra.run.smithi114.stdout:506 Cannot talk to daemon 2024-04-23T10:04:39.377 INFO:teuthology.orchestra.run.smithi119.stderr:Failed to start ntp.service: Unit ntp.service not found. 2024-04-23T10:04:39.379 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: ntpd: command not found 2024-04-23T10:04:39.381 INFO:teuthology.orchestra.run.smithi114.stderr:Failed to start ntp.service: Unit ntp.service not found. 2024-04-23T10:04:39.389 INFO:teuthology.orchestra.run.smithi119.stderr:Failed to start ntpd.service: Unit ntpd.service not found. 2024-04-23T10:04:39.390 INFO:teuthology.orchestra.run.smithi204.stdout:506 Cannot talk to daemon 2024-04-23T10:04:39.393 INFO:teuthology.orchestra.run.smithi114.stderr:Failed to start ntpd.service: Unit ntpd.service not found. 2024-04-23T10:04:39.403 INFO:teuthology.orchestra.run.smithi204.stderr:Failed to start ntp.service: Unit ntp.service not found. 2024-04-23T10:04:39.416 INFO:teuthology.orchestra.run.smithi204.stderr:Failed to start ntpd.service: Unit ntpd.service not found. 2024-04-23T10:04:39.450 INFO:teuthology.orchestra.run.smithi119.stderr:bash: line 1: ntpq: command not found 2024-04-23T10:04:39.452 INFO:teuthology.orchestra.run.smithi114.stderr:bash: line 1: ntpq: command not found 2024-04-23T10:04:39.454 INFO:teuthology.orchestra.run.smithi119.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2024-04-23T10:04:39.454 INFO:teuthology.orchestra.run.smithi119.stdout:=============================================================================== 2024-04-23T10:04:39.454 INFO:teuthology.orchestra.run.smithi119.stdout:^? hv01.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-04-23T10:04:39.454 INFO:teuthology.orchestra.run.smithi119.stdout:^? hv02.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-04-23T10:04:39.454 INFO:teuthology.orchestra.run.smithi119.stdout:^? hv03.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-04-23T10:04:39.455 INFO:teuthology.orchestra.run.smithi119.stdout:^? hv04.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-04-23T10:04:39.455 INFO:teuthology.orchestra.run.smithi114.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2024-04-23T10:04:39.455 INFO:teuthology.orchestra.run.smithi114.stdout:=============================================================================== 2024-04-23T10:04:39.456 INFO:teuthology.orchestra.run.smithi114.stdout:^? hv01.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-04-23T10:04:39.456 INFO:teuthology.orchestra.run.smithi114.stdout:^? hv02.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-04-23T10:04:39.456 INFO:teuthology.orchestra.run.smithi114.stdout:^? hv03.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-04-23T10:04:39.456 INFO:teuthology.orchestra.run.smithi114.stdout:^? hv04.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-04-23T10:04:39.469 INFO:teuthology.orchestra.run.smithi204.stderr:bash: line 1: ntpq: command not found 2024-04-23T10:04:39.473 INFO:teuthology.orchestra.run.smithi204.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2024-04-23T10:04:39.473 INFO:teuthology.orchestra.run.smithi204.stdout:=============================================================================== 2024-04-23T10:04:39.473 INFO:teuthology.orchestra.run.smithi204.stdout:^? hv01.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-04-23T10:04:39.473 INFO:teuthology.orchestra.run.smithi204.stdout:^? hv02.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-04-23T10:04:39.473 INFO:teuthology.orchestra.run.smithi204.stdout:^? hv03.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-04-23T10:04:39.473 INFO:teuthology.orchestra.run.smithi204.stdout:^? hv04.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-04-23T10:04:39.474 INFO:teuthology.run_tasks:Running task install... 2024-04-23T10:04:39.482 DEBUG:teuthology.task.install:project ceph 2024-04-23T10:04:39.482 DEBUG:teuthology.task.install:INSTALL overrides: {'ceph': {'flavor': 'default', 'sha1': 'd69f13d827952bb07bb9b7973bca14f97d764d6f'}} 2024-04-23T10:04:39.482 DEBUG:teuthology.task.install:config {'extra_packages': {'deb': ['python3-cephfs', 'cephfs-shell', 'cephfs-top', 'cephfs-mirror'], 'rpm': ['python3-cephfs', 'cephfs-shell', 'cephfs-top', 'cephfs-mirror']}, 'extra_system_packages': {'deb': ['bison', 'flex', 'libelf-dev', 'libssl-dev', 'network-manager', 'iproute2', 'util-linux', 'dump', 'indent', 'libaio-dev', 'libtool-bin', 'uuid-dev', 'xfslibs-dev', 'postgresql', 'postgresql-client', 'postgresql-common', 'postgresql-contrib'], 'rpm': ['bison', 'flex', 'elfutils-libelf-devel', 'openssl-devel', 'NetworkManager', 'iproute', 'util-linux', 'libacl-devel', 'libaio-devel', 'libattr-devel', 'libtool', 'libuuid-devel', 'xfsdump', 'xfsprogs', 'xfsprogs-devel', 'libaio-devel', 'libtool', 'libuuid-devel', 'xfsprogs-devel', 'postgresql', 'postgresql-server', 'postgresql-contrib']}, 'flavor': 'default', 'sha1': 'd69f13d827952bb07bb9b7973bca14f97d764d6f'} 2024-04-23T10:04:39.482 INFO:teuthology.task.install:Using flavor: default 2024-04-23T10:04:39.488 DEBUG:teuthology.task.install:Package list is: {'deb': ['ceph', 'cephadm', 'ceph-mds', 'ceph-mgr', 'ceph-common', 'ceph-fuse', 'ceph-test', 'ceph-volume', 'radosgw', 'python3-rados', 'python3-rgw', 'python3-cephfs', 'python3-rbd', 'libcephfs2', 'libcephfs-dev', 'librados2', 'librbd1', 'rbd-fuse'], 'rpm': ['ceph-radosgw', 'ceph-test', 'ceph', 'ceph-base', 'cephadm', 'ceph-immutable-object-cache', 'ceph-mgr', 'ceph-mgr-dashboard', 'ceph-mgr-diskprediction-local', 'ceph-mgr-rook', 'ceph-mgr-cephadm', 'ceph-fuse', 'ceph-volume', 'librados-devel', 'libcephfs2', 'libcephfs-devel', 'librados2', 'librbd1', 'python3-rados', 'python3-rgw', 'python3-cephfs', 'python3-rbd', 'rbd-fuse', 'rbd-mirror', 'rbd-nbd']} 2024-04-23T10:04:39.488 INFO:teuthology.task.install:extra packages: {'deb': ['python3-cephfs', 'cephfs-shell', 'cephfs-top', 'cephfs-mirror'], 'rpm': ['python3-cephfs', 'cephfs-shell', 'cephfs-top', 'cephfs-mirror']} 2024-04-23T10:04:39.489 DEBUG:teuthology.task.install.rpm:_update_package_list_and_install: config is {'branch': None, 'cleanup': None, 'debuginfo': None, 'downgrade_packages': [], 'exclude_packages': [], 'extra_packages': {'deb': ['python3-cephfs', 'cephfs-shell', 'cephfs-top', 'cephfs-mirror'], 'rpm': ['python3-cephfs', 'cephfs-shell', 'cephfs-top', 'cephfs-mirror']}, 'extra_system_packages': {'deb': ['bison', 'flex', 'libelf-dev', 'libssl-dev', 'network-manager', 'iproute2', 'util-linux', 'dump', 'indent', 'libaio-dev', 'libtool-bin', 'uuid-dev', 'xfslibs-dev', 'postgresql', 'postgresql-client', 'postgresql-common', 'postgresql-contrib'], 'rpm': ['bison', 'flex', 'elfutils-libelf-devel', 'openssl-devel', 'NetworkManager', 'iproute', 'util-linux', 'libacl-devel', 'libaio-devel', 'libattr-devel', 'libtool', 'libuuid-devel', 'xfsdump', 'xfsprogs', 'xfsprogs-devel', 'libaio-devel', 'libtool', 'libuuid-devel', 'xfsprogs-devel', 'postgresql', 'postgresql-server', 'postgresql-contrib']}, 'extras': None, 'enable_coprs': [], 'flavor': 'default', 'install_ceph_packages': True, 'packages': {}, 'project': 'ceph', 'repos_only': False, 'sha1': 'd69f13d827952bb07bb9b7973bca14f97d764d6f', 'tag': None, 'wait_for_package': False} 2024-04-23T10:04:39.489 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&sha1=d69f13d827952bb07bb9b7973bca14f97d764d6f 2024-04-23T10:04:39.491 DEBUG:teuthology.task.install.rpm:_update_package_list_and_install: config is {'branch': None, 'cleanup': None, 'debuginfo': None, 'downgrade_packages': [], 'exclude_packages': [], 'extra_packages': {'deb': ['python3-cephfs', 'cephfs-shell', 'cephfs-top', 'cephfs-mirror'], 'rpm': ['python3-cephfs', 'cephfs-shell', 'cephfs-top', 'cephfs-mirror']}, 'extra_system_packages': {'deb': ['bison', 'flex', 'libelf-dev', 'libssl-dev', 'network-manager', 'iproute2', 'util-linux', 'dump', 'indent', 'libaio-dev', 'libtool-bin', 'uuid-dev', 'xfslibs-dev', 'postgresql', 'postgresql-client', 'postgresql-common', 'postgresql-contrib'], 'rpm': ['bison', 'flex', 'elfutils-libelf-devel', 'openssl-devel', 'NetworkManager', 'iproute', 'util-linux', 'libacl-devel', 'libaio-devel', 'libattr-devel', 'libtool', 'libuuid-devel', 'xfsdump', 'xfsprogs', 'xfsprogs-devel', 'libaio-devel', 'libtool', 'libuuid-devel', 'xfsprogs-devel', 'postgresql', 'postgresql-server', 'postgresql-contrib']}, 'extras': None, 'enable_coprs': [], 'flavor': 'default', 'install_ceph_packages': True, 'packages': {}, 'project': 'ceph', 'repos_only': False, 'sha1': 'd69f13d827952bb07bb9b7973bca14f97d764d6f', 'tag': None, 'wait_for_package': False} 2024-04-23T10:04:39.491 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&sha1=d69f13d827952bb07bb9b7973bca14f97d764d6f 2024-04-23T10:04:39.492 DEBUG:teuthology.task.install.rpm:_update_package_list_and_install: config is {'branch': None, 'cleanup': None, 'debuginfo': None, 'downgrade_packages': [], 'exclude_packages': [], 'extra_packages': {'deb': ['python3-cephfs', 'cephfs-shell', 'cephfs-top', 'cephfs-mirror'], 'rpm': ['python3-cephfs', 'cephfs-shell', 'cephfs-top', 'cephfs-mirror']}, 'extra_system_packages': {'deb': ['bison', 'flex', 'libelf-dev', 'libssl-dev', 'network-manager', 'iproute2', 'util-linux', 'dump', 'indent', 'libaio-dev', 'libtool-bin', 'uuid-dev', 'xfslibs-dev', 'postgresql', 'postgresql-client', 'postgresql-common', 'postgresql-contrib'], 'rpm': ['bison', 'flex', 'elfutils-libelf-devel', 'openssl-devel', 'NetworkManager', 'iproute', 'util-linux', 'libacl-devel', 'libaio-devel', 'libattr-devel', 'libtool', 'libuuid-devel', 'xfsdump', 'xfsprogs', 'xfsprogs-devel', 'libaio-devel', 'libtool', 'libuuid-devel', 'xfsprogs-devel', 'postgresql', 'postgresql-server', 'postgresql-contrib']}, 'extras': None, 'enable_coprs': [], 'flavor': 'default', 'install_ceph_packages': True, 'packages': {}, 'project': 'ceph', 'repos_only': False, 'sha1': 'd69f13d827952bb07bb9b7973bca14f97d764d6f', 'tag': None, 'wait_for_package': False} 2024-04-23T10:04:39.493 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&sha1=d69f13d827952bb07bb9b7973bca14f97d764d6f 2024-04-23T10:04:39.645 INFO:teuthology.task.install.rpm:Pulling from https://2.chacra.ceph.com/r/ceph/wip-lusov-quiescer/d69f13d827952bb07bb9b7973bca14f97d764d6f/centos/9/flavors/default/ 2024-04-23T10:04:39.645 INFO:teuthology.task.install.rpm:Package version is 19.0.0-3156.gd69f13d8 2024-04-23T10:04:39.766 INFO:teuthology.task.install.rpm:Pulling from https://2.chacra.ceph.com/r/ceph/wip-lusov-quiescer/d69f13d827952bb07bb9b7973bca14f97d764d6f/centos/9/flavors/default/ 2024-04-23T10:04:39.766 INFO:teuthology.task.install.rpm:Package version is 19.0.0-3156.gd69f13d8 2024-04-23T10:04:39.770 INFO:teuthology.task.install.rpm:Pulling from https://2.chacra.ceph.com/r/ceph/wip-lusov-quiescer/d69f13d827952bb07bb9b7973bca14f97d764d6f/centos/9/flavors/default/ 2024-04-23T10:04:39.770 INFO:teuthology.task.install.rpm:Package version is 19.0.0-3156.gd69f13d8 2024-04-23T10:04:39.976 INFO:teuthology.packaging:Writing yum repo: [ceph] name=ceph packages for $basearch baseurl=https://2.chacra.ceph.com/r/ceph/wip-lusov-quiescer/d69f13d827952bb07bb9b7973bca14f97d764d6f/centos/9/flavors/default/$basearch enabled=1 gpgcheck=0 type=rpm-md [ceph-noarch] name=ceph noarch packages baseurl=https://2.chacra.ceph.com/r/ceph/wip-lusov-quiescer/d69f13d827952bb07bb9b7973bca14f97d764d6f/centos/9/flavors/default/noarch enabled=1 gpgcheck=0 type=rpm-md [ceph-source] name=ceph source packages baseurl=https://2.chacra.ceph.com/r/ceph/wip-lusov-quiescer/d69f13d827952bb07bb9b7973bca14f97d764d6f/centos/9/flavors/default/SRPMS enabled=1 gpgcheck=0 type=rpm-md 2024-04-23T10:04:39.976 DEBUG:teuthology.orchestra.run.smithi204:> set -ex 2024-04-23T10:04:39.976 DEBUG:teuthology.orchestra.run.smithi204:> sudo dd of=/etc/yum.repos.d/ceph.repo 2024-04-23T10:04:40.002 INFO:teuthology.task.install.rpm:Installing packages: ceph-radosgw, ceph-test, ceph, ceph-base, cephadm, ceph-immutable-object-cache, ceph-mgr, ceph-mgr-dashboard, ceph-mgr-diskprediction-local, ceph-mgr-rook, ceph-mgr-cephadm, ceph-fuse, ceph-volume, librados-devel, libcephfs2, libcephfs-devel, librados2, librbd1, python3-rados, python3-rgw, python3-cephfs, python3-rbd, rbd-fuse, rbd-mirror, rbd-nbd, python3-cephfs, cephfs-shell, cephfs-top, cephfs-mirror, bison, flex, elfutils-libelf-devel, openssl-devel, NetworkManager, iproute, util-linux, libacl-devel, libaio-devel, libattr-devel, libtool, libuuid-devel, xfsdump, xfsprogs, xfsprogs-devel, libaio-devel, libtool, libuuid-devel, xfsprogs-devel, postgresql, postgresql-server, postgresql-contrib, bison, flex, elfutils-libelf-devel, openssl-devel, NetworkManager, iproute, util-linux, libacl-devel, libaio-devel, libattr-devel, libtool, libuuid-devel, xfsdump, xfsprogs, xfsprogs-devel, libaio-devel, libtool, libuuid-devel, xfsprogs-devel, postgresql, postgresql-server, postgresql-contrib, bison, flex, elfutils-libelf-devel, openssl-devel, NetworkManager, iproute, util-linux, libacl-devel, libaio-devel, libattr-devel, libtool, libuuid-devel, xfsdump, xfsprogs, xfsprogs-devel, libaio-devel, libtool, libuuid-devel, xfsprogs-devel, postgresql, postgresql-server, postgresql-contrib, bison, flex, elfutils-libelf-devel, openssl-devel, NetworkManager, iproute, util-linux, libacl-devel, libaio-devel, libattr-devel, libtool, libuuid-devel, xfsdump, xfsprogs, xfsprogs-devel, libaio-devel, libtool, libuuid-devel, xfsprogs-devel, postgresql, postgresql-server, postgresql-contrib on remote rpm x86_64 2024-04-23T10:04:40.002 DEBUG:teuthology.orchestra.run.smithi204:> if test -f /etc/yum.repos.d/ceph.repo ; then sudo sed -i -e ':a;N;$!ba;s/enabled=1\ngpg/enabled=1\npriority=1\ngpg/g' -e 's;ref/[a-zA-Z0-9_-]*/;sha1/d69f13d827952bb07bb9b7973bca14f97d764d6f/;g' /etc/yum.repos.d/ceph.repo ; fi 2024-04-23T10:04:40.067 DEBUG:teuthology.orchestra.run.smithi204:> sudo touch -a /etc/yum/pluginconf.d/priorities.conf ; test -e /etc/yum/pluginconf.d/priorities.conf.orig || sudo cp -af /etc/yum/pluginconf.d/priorities.conf /etc/yum/pluginconf.d/priorities.conf.orig 2024-04-23T10:04:40.089 INFO:teuthology.packaging:Writing yum repo: [ceph] name=ceph packages for $basearch baseurl=https://2.chacra.ceph.com/r/ceph/wip-lusov-quiescer/d69f13d827952bb07bb9b7973bca14f97d764d6f/centos/9/flavors/default/$basearch enabled=1 gpgcheck=0 type=rpm-md [ceph-noarch] name=ceph noarch packages baseurl=https://2.chacra.ceph.com/r/ceph/wip-lusov-quiescer/d69f13d827952bb07bb9b7973bca14f97d764d6f/centos/9/flavors/default/noarch enabled=1 gpgcheck=0 type=rpm-md [ceph-source] name=ceph source packages baseurl=https://2.chacra.ceph.com/r/ceph/wip-lusov-quiescer/d69f13d827952bb07bb9b7973bca14f97d764d6f/centos/9/flavors/default/SRPMS enabled=1 gpgcheck=0 type=rpm-md 2024-04-23T10:04:40.089 DEBUG:teuthology.orchestra.run.smithi119:> set -ex 2024-04-23T10:04:40.089 DEBUG:teuthology.orchestra.run.smithi119:> sudo dd of=/etc/yum.repos.d/ceph.repo 2024-04-23T10:04:40.092 INFO:teuthology.packaging:Writing yum repo: [ceph] name=ceph packages for $basearch baseurl=https://2.chacra.ceph.com/r/ceph/wip-lusov-quiescer/d69f13d827952bb07bb9b7973bca14f97d764d6f/centos/9/flavors/default/$basearch enabled=1 gpgcheck=0 type=rpm-md [ceph-noarch] name=ceph noarch packages baseurl=https://2.chacra.ceph.com/r/ceph/wip-lusov-quiescer/d69f13d827952bb07bb9b7973bca14f97d764d6f/centos/9/flavors/default/noarch enabled=1 gpgcheck=0 type=rpm-md [ceph-source] name=ceph source packages baseurl=https://2.chacra.ceph.com/r/ceph/wip-lusov-quiescer/d69f13d827952bb07bb9b7973bca14f97d764d6f/centos/9/flavors/default/SRPMS enabled=1 gpgcheck=0 type=rpm-md 2024-04-23T10:04:40.092 DEBUG:teuthology.orchestra.run.smithi114:> set -ex 2024-04-23T10:04:40.092 DEBUG:teuthology.orchestra.run.smithi114:> sudo dd of=/etc/yum.repos.d/ceph.repo 2024-04-23T10:04:40.114 INFO:teuthology.task.install.rpm:Installing packages: ceph-radosgw, ceph-test, ceph, ceph-base, cephadm, ceph-immutable-object-cache, ceph-mgr, ceph-mgr-dashboard, ceph-mgr-diskprediction-local, ceph-mgr-rook, ceph-mgr-cephadm, ceph-fuse, ceph-volume, librados-devel, libcephfs2, libcephfs-devel, librados2, librbd1, python3-rados, python3-rgw, python3-cephfs, python3-rbd, rbd-fuse, rbd-mirror, rbd-nbd, python3-cephfs, cephfs-shell, cephfs-top, cephfs-mirror, bison, flex, elfutils-libelf-devel, openssl-devel, NetworkManager, iproute, util-linux, libacl-devel, libaio-devel, libattr-devel, libtool, libuuid-devel, xfsdump, xfsprogs, xfsprogs-devel, libaio-devel, libtool, libuuid-devel, xfsprogs-devel, postgresql, postgresql-server, postgresql-contrib, bison, flex, elfutils-libelf-devel, openssl-devel, NetworkManager, iproute, util-linux, libacl-devel, libaio-devel, libattr-devel, libtool, libuuid-devel, xfsdump, xfsprogs, xfsprogs-devel, libaio-devel, libtool, libuuid-devel, xfsprogs-devel, postgresql, postgresql-server, postgresql-contrib, bison, flex, elfutils-libelf-devel, openssl-devel, NetworkManager, iproute, util-linux, libacl-devel, libaio-devel, libattr-devel, libtool, libuuid-devel, xfsdump, xfsprogs, xfsprogs-devel, libaio-devel, libtool, libuuid-devel, xfsprogs-devel, postgresql, postgresql-server, postgresql-contrib, bison, flex, elfutils-libelf-devel, openssl-devel, NetworkManager, iproute, util-linux, libacl-devel, libaio-devel, libattr-devel, libtool, libuuid-devel, xfsdump, xfsprogs, xfsprogs-devel, libaio-devel, libtool, libuuid-devel, xfsprogs-devel, postgresql, postgresql-server, postgresql-contrib, bison, flex, elfutils-libelf-devel, openssl-devel, NetworkManager, iproute, util-linux, libacl-devel, libaio-devel, libattr-devel, libtool, libuuid-devel, xfsdump, xfsprogs, xfsprogs-devel, libaio-devel, libtool, libuuid-devel, xfsprogs-devel, postgresql, postgresql-server, postgresql-contrib on remote rpm x86_64 2024-04-23T10:04:40.114 DEBUG:teuthology.orchestra.run.smithi119:> if test -f /etc/yum.repos.d/ceph.repo ; then sudo sed -i -e ':a;N;$!ba;s/enabled=1\ngpg/enabled=1\npriority=1\ngpg/g' -e 's;ref/[a-zA-Z0-9_-]*/;sha1/d69f13d827952bb07bb9b7973bca14f97d764d6f/;g' /etc/yum.repos.d/ceph.repo ; fi 2024-04-23T10:04:40.119 INFO:teuthology.task.install.rpm:Installing packages: ceph-radosgw, ceph-test, ceph, ceph-base, cephadm, ceph-immutable-object-cache, ceph-mgr, ceph-mgr-dashboard, ceph-mgr-diskprediction-local, ceph-mgr-rook, ceph-mgr-cephadm, ceph-fuse, ceph-volume, librados-devel, libcephfs2, libcephfs-devel, librados2, librbd1, python3-rados, python3-rgw, python3-cephfs, python3-rbd, rbd-fuse, rbd-mirror, rbd-nbd, python3-cephfs, cephfs-shell, cephfs-top, cephfs-mirror, bison, flex, elfutils-libelf-devel, openssl-devel, NetworkManager, iproute, util-linux, libacl-devel, libaio-devel, libattr-devel, libtool, libuuid-devel, xfsdump, xfsprogs, xfsprogs-devel, libaio-devel, libtool, libuuid-devel, xfsprogs-devel, postgresql, postgresql-server, postgresql-contrib, bison, flex, elfutils-libelf-devel, openssl-devel, NetworkManager, iproute, util-linux, libacl-devel, libaio-devel, libattr-devel, libtool, libuuid-devel, xfsdump, xfsprogs, xfsprogs-devel, libaio-devel, libtool, libuuid-devel, xfsprogs-devel, postgresql, postgresql-server, postgresql-contrib, bison, flex, elfutils-libelf-devel, openssl-devel, NetworkManager, iproute, util-linux, libacl-devel, libaio-devel, libattr-devel, libtool, libuuid-devel, xfsdump, xfsprogs, xfsprogs-devel, libaio-devel, libtool, libuuid-devel, xfsprogs-devel, postgresql, postgresql-server, postgresql-contrib, bison, flex, elfutils-libelf-devel, openssl-devel, NetworkManager, iproute, util-linux, libacl-devel, libaio-devel, libattr-devel, libtool, libuuid-devel, xfsdump, xfsprogs, xfsprogs-devel, libaio-devel, libtool, libuuid-devel, xfsprogs-devel, postgresql, postgresql-server, postgresql-contrib, bison, flex, elfutils-libelf-devel, openssl-devel, NetworkManager, iproute, util-linux, libacl-devel, libaio-devel, libattr-devel, libtool, libuuid-devel, xfsdump, xfsprogs, xfsprogs-devel, libaio-devel, libtool, libuuid-devel, xfsprogs-devel, postgresql, postgresql-server, postgresql-contrib, bison, flex, elfutils-libelf-devel, openssl-devel, NetworkManager, iproute, util-linux, libacl-devel, libaio-devel, libattr-devel, libtool, libuuid-devel, xfsdump, xfsprogs, xfsprogs-devel, libaio-devel, libtool, libuuid-devel, xfsprogs-devel, postgresql, postgresql-server, postgresql-contrib on remote rpm x86_64 2024-04-23T10:04:40.119 DEBUG:teuthology.orchestra.run.smithi114:> if test -f /etc/yum.repos.d/ceph.repo ; then sudo sed -i -e ':a;N;$!ba;s/enabled=1\ngpg/enabled=1\npriority=1\ngpg/g' -e 's;ref/[a-zA-Z0-9_-]*/;sha1/d69f13d827952bb07bb9b7973bca14f97d764d6f/;g' /etc/yum.repos.d/ceph.repo ; fi 2024-04-23T10:04:40.141 DEBUG:teuthology.orchestra.run.smithi204:> grep check_obsoletes /etc/yum/pluginconf.d/priorities.conf && sudo sed -i 's/check_obsoletes.*0/check_obsoletes = 1/g' /etc/yum/pluginconf.d/priorities.conf || echo 'check_obsoletes = 1' | sudo tee -a /etc/yum/pluginconf.d/priorities.conf 2024-04-23T10:04:40.180 DEBUG:teuthology.orchestra.run.smithi119:> sudo touch -a /etc/yum/pluginconf.d/priorities.conf ; test -e /etc/yum/pluginconf.d/priorities.conf.orig || sudo cp -af /etc/yum/pluginconf.d/priorities.conf /etc/yum/pluginconf.d/priorities.conf.orig 2024-04-23T10:04:40.185 DEBUG:teuthology.orchestra.run.smithi114:> sudo touch -a /etc/yum/pluginconf.d/priorities.conf ; test -e /etc/yum/pluginconf.d/priorities.conf.orig || sudo cp -af /etc/yum/pluginconf.d/priorities.conf /etc/yum/pluginconf.d/priorities.conf.orig 2024-04-23T10:04:40.204 INFO:teuthology.orchestra.run.smithi204.stdout:check_obsoletes = 1 2024-04-23T10:04:40.206 DEBUG:teuthology.orchestra.run.smithi204:> sudo yum clean all 2024-04-23T10:04:40.252 DEBUG:teuthology.orchestra.run.smithi119:> grep check_obsoletes /etc/yum/pluginconf.d/priorities.conf && sudo sed -i 's/check_obsoletes.*0/check_obsoletes = 1/g' /etc/yum/pluginconf.d/priorities.conf || echo 'check_obsoletes = 1' | sudo tee -a /etc/yum/pluginconf.d/priorities.conf 2024-04-23T10:04:40.259 DEBUG:teuthology.orchestra.run.smithi114:> grep check_obsoletes /etc/yum/pluginconf.d/priorities.conf && sudo sed -i 's/check_obsoletes.*0/check_obsoletes = 1/g' /etc/yum/pluginconf.d/priorities.conf || echo 'check_obsoletes = 1' | sudo tee -a /etc/yum/pluginconf.d/priorities.conf 2024-04-23T10:04:40.315 INFO:teuthology.orchestra.run.smithi119.stdout:check_obsoletes = 1 2024-04-23T10:04:40.317 DEBUG:teuthology.orchestra.run.smithi119:> sudo yum clean all 2024-04-23T10:04:40.322 INFO:teuthology.orchestra.run.smithi114.stdout:check_obsoletes = 1 2024-04-23T10:04:40.324 DEBUG:teuthology.orchestra.run.smithi114:> sudo yum clean all 2024-04-23T10:04:40.430 INFO:teuthology.orchestra.run.smithi204.stdout:48 files removed 2024-04-23T10:04:40.456 DEBUG:teuthology.orchestra.run.smithi204:> sudo yum -y install ceph-radosgw ceph-test ceph ceph-base cephadm ceph-immutable-object-cache ceph-mgr ceph-mgr-dashboard ceph-mgr-diskprediction-local ceph-mgr-rook ceph-mgr-cephadm ceph-fuse ceph-volume librados-devel libcephfs2 libcephfs-devel librados2 librbd1 python3-rados python3-rgw python3-cephfs python3-rbd rbd-fuse rbd-mirror rbd-nbd python3-cephfs cephfs-shell cephfs-top cephfs-mirror bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib 2024-04-23T10:04:40.540 INFO:teuthology.orchestra.run.smithi119.stdout:48 files removed 2024-04-23T10:04:40.546 INFO:teuthology.orchestra.run.smithi114.stdout:48 files removed 2024-04-23T10:04:40.567 DEBUG:teuthology.orchestra.run.smithi119:> sudo yum -y install ceph-radosgw ceph-test ceph ceph-base cephadm ceph-immutable-object-cache ceph-mgr ceph-mgr-dashboard ceph-mgr-diskprediction-local ceph-mgr-rook ceph-mgr-cephadm ceph-fuse ceph-volume librados-devel libcephfs2 libcephfs-devel librados2 librbd1 python3-rados python3-rgw python3-cephfs python3-rbd rbd-fuse rbd-mirror rbd-nbd python3-cephfs cephfs-shell cephfs-top cephfs-mirror bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib 2024-04-23T10:04:40.573 DEBUG:teuthology.orchestra.run.smithi114:> sudo yum -y install ceph-radosgw ceph-test ceph ceph-base cephadm ceph-immutable-object-cache ceph-mgr ceph-mgr-dashboard ceph-mgr-diskprediction-local ceph-mgr-rook ceph-mgr-cephadm ceph-fuse ceph-volume librados-devel libcephfs2 libcephfs-devel librados2 librbd1 python3-rados python3-rgw python3-cephfs python3-rbd rbd-fuse rbd-mirror rbd-nbd python3-cephfs cephfs-shell cephfs-top cephfs-mirror bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib 2024-04-23T10:04:41.548 INFO:teuthology.orchestra.run.smithi204.stdout:ceph packages for x86_64 101 kB/s | 83 kB 00:00 2024-04-23T10:04:41.736 INFO:teuthology.orchestra.run.smithi114.stdout:ceph packages for x86_64 92 kB/s | 83 kB 00:00 2024-04-23T10:04:41.787 INFO:teuthology.orchestra.run.smithi119.stdout:ceph packages for x86_64 87 kB/s | 83 kB 00:00 2024-04-23T10:04:42.332 INFO:teuthology.orchestra.run.smithi204.stdout:ceph noarch packages 16 kB/s | 12 kB 00:00 2024-04-23T10:04:42.436 INFO:teuthology.orchestra.run.smithi114.stdout:ceph noarch packages 18 kB/s | 12 kB 00:00 2024-04-23T10:04:42.899 INFO:teuthology.orchestra.run.smithi204.stdout:ceph source packages 3.6 kB/s | 1.9 kB 00:00 2024-04-23T10:04:43.021 INFO:teuthology.orchestra.run.smithi119.stdout:ceph noarch packages 9.7 kB/s | 12 kB 00:01 2024-04-23T10:04:43.153 INFO:teuthology.orchestra.run.smithi114.stdout:ceph source packages 2.8 kB/s | 1.9 kB 00:00 2024-04-23T10:04:43.704 INFO:teuthology.orchestra.run.smithi119.stdout:ceph source packages 2.9 kB/s | 1.9 kB 00:00 2024-04-23T10:04:45.695 INFO:teuthology.orchestra.run.smithi114.stdout:CentOS Stream 9 - BaseOS 3.2 MB/s | 8.0 MB 00:02 2024-04-23T10:04:46.124 INFO:teuthology.orchestra.run.smithi204.stdout:CentOS Stream 9 - BaseOS 2.5 MB/s | 8.0 MB 00:03 2024-04-23T10:04:47.330 INFO:teuthology.orchestra.run.smithi114.stdout:CentOS Stream 9 - AppStream 32 MB/s | 19 MB 00:00 2024-04-23T10:04:48.241 INFO:teuthology.orchestra.run.smithi119.stdout:CentOS Stream 9 - BaseOS 1.8 MB/s | 8.0 MB 00:04 2024-04-23T10:04:50.059 INFO:teuthology.orchestra.run.smithi204.stdout:CentOS Stream 9 - AppStream 6.7 MB/s | 19 MB 00:02 2024-04-23T10:04:51.756 INFO:teuthology.orchestra.run.smithi114.stdout:CentOS Stream 9 - CRB 6.9 MB/s | 6.2 MB 00:00 2024-04-23T10:04:53.231 INFO:teuthology.orchestra.run.smithi114.stdout:CentOS Stream 9 - Extras packages 50 kB/s | 16 kB 00:00 2024-04-23T10:04:54.941 INFO:teuthology.orchestra.run.smithi114.stdout:Extra Packages for Enterprise Linux 13 MB/s | 21 MB 00:01 2024-04-23T10:04:55.709 INFO:teuthology.orchestra.run.smithi204.stdout:CentOS Stream 9 - CRB 3.0 MB/s | 6.2 MB 00:02 2024-04-23T10:04:57.218 INFO:teuthology.orchestra.run.smithi204.stdout:CentOS Stream 9 - Extras packages 44 kB/s | 16 kB 00:00 2024-04-23T10:04:58.937 INFO:teuthology.orchestra.run.smithi204.stdout:Extra Packages for Enterprise Linux 13 MB/s | 21 MB 00:01 2024-04-23T10:04:59.958 INFO:teuthology.orchestra.run.smithi114.stdout:lab-extras 15 kB/s | 1.7 kB 00:00 2024-04-23T10:05:02.084 INFO:teuthology.orchestra.run.smithi114.stdout:Package librados2-2:16.2.4-5.el9.x86_64 is already installed. 2024-04-23T10:05:02.085 INFO:teuthology.orchestra.run.smithi114.stdout:Package librbd1-2:16.2.4-5.el9.x86_64 is already installed. 2024-04-23T10:05:02.095 INFO:teuthology.orchestra.run.smithi114.stdout:Package NetworkManager-1:1.47.4-1.el9.x86_64 is already installed. 2024-04-23T10:05:02.096 INFO:teuthology.orchestra.run.smithi114.stdout:Package iproute-6.2.0-5.el9.x86_64 is already installed. 2024-04-23T10:05:02.097 INFO:teuthology.orchestra.run.smithi114.stdout:Package util-linux-2.37.4-18.el9.x86_64 is already installed. 2024-04-23T10:05:02.098 INFO:teuthology.orchestra.run.smithi114.stdout:Package libacl-devel-2.3.1-4.el9.x86_64 is already installed. 2024-04-23T10:05:02.099 INFO:teuthology.orchestra.run.smithi114.stdout:Package libattr-devel-2.5.1-3.el9.x86_64 is already installed. 2024-04-23T10:05:02.102 INFO:teuthology.orchestra.run.smithi114.stdout:Package xfsdump-3.1.12-4.el9.x86_64 is already installed. 2024-04-23T10:05:02.102 INFO:teuthology.orchestra.run.smithi114.stdout:Package xfsprogs-6.3.0-1.el9.x86_64 is already installed. 2024-04-23T10:05:02.163 INFO:teuthology.orchestra.run.smithi114.stdout:Dependencies resolved. 2024-04-23T10:05:02.172 INFO:teuthology.orchestra.run.smithi114.stdout:======================================================================================= 2024-04-23T10:05:02.173 INFO:teuthology.orchestra.run.smithi114.stdout: Package Arch Version Repository Size 2024-04-23T10:05:02.173 INFO:teuthology.orchestra.run.smithi114.stdout:======================================================================================= 2024-04-23T10:05:02.173 INFO:teuthology.orchestra.run.smithi114.stdout:Installing: 2024-04-23T10:05:02.173 INFO:teuthology.orchestra.run.smithi114.stdout: bison x86_64 3.7.4-5.el9 appstream 943 k 2024-04-23T10:05:02.173 INFO:teuthology.orchestra.run.smithi114.stdout: ceph x86_64 2:19.0.0-3156.gd69f13d8.el7 ceph 6.6 k 2024-04-23T10:05:02.173 INFO:teuthology.orchestra.run.smithi114.stdout: ceph-base x86_64 2:19.0.0-3156.gd69f13d8.el7 ceph 5.4 M 2024-04-23T10:05:02.174 INFO:teuthology.orchestra.run.smithi114.stdout: ceph-fuse x86_64 2:19.0.0-3156.gd69f13d8.el7 ceph 890 k 2024-04-23T10:05:02.174 INFO:teuthology.orchestra.run.smithi114.stdout: ceph-immutable-object-cache x86_64 2:19.0.0-3156.gd69f13d8.el7 ceph 145 k 2024-04-23T10:05:02.174 INFO:teuthology.orchestra.run.smithi114.stdout: ceph-mgr x86_64 2:19.0.0-3156.gd69f13d8.el7 ceph 1.5 M 2024-04-23T10:05:02.174 INFO:teuthology.orchestra.run.smithi114.stdout: ceph-mgr-cephadm noarch 2:19.0.0-3156.gd69f13d8.el7 ceph-noarch 142 k 2024-04-23T10:05:02.174 INFO:teuthology.orchestra.run.smithi114.stdout: ceph-mgr-dashboard noarch 2:19.0.0-3156.gd69f13d8.el7 ceph-noarch 3.5 M 2024-04-23T10:05:02.174 INFO:teuthology.orchestra.run.smithi114.stdout: ceph-mgr-diskprediction-local noarch 2:19.0.0-3156.gd69f13d8.el7 ceph-noarch 7.4 M 2024-04-23T10:05:02.175 INFO:teuthology.orchestra.run.smithi114.stdout: ceph-mgr-rook noarch 2:19.0.0-3156.gd69f13d8.el7 ceph-noarch 49 k 2024-04-23T10:05:02.175 INFO:teuthology.orchestra.run.smithi114.stdout: ceph-radosgw x86_64 2:19.0.0-3156.gd69f13d8.el7 ceph 11 M 2024-04-23T10:05:02.175 INFO:teuthology.orchestra.run.smithi114.stdout: ceph-test x86_64 2:19.0.0-3156.gd69f13d8.el7 ceph 49 M 2024-04-23T10:05:02.175 INFO:teuthology.orchestra.run.smithi114.stdout: ceph-volume noarch 2:19.0.0-3156.gd69f13d8.el7 ceph-noarch 273 k 2024-04-23T10:05:02.175 INFO:teuthology.orchestra.run.smithi114.stdout: cephadm noarch 2:19.0.0-3156.gd69f13d8.el7 ceph-noarch 762 k 2024-04-23T10:05:02.175 INFO:teuthology.orchestra.run.smithi114.stdout: cephfs-mirror x86_64 2:19.0.0-3156.gd69f13d8.el7 ceph 225 k 2024-04-23T10:05:02.176 INFO:teuthology.orchestra.run.smithi114.stdout: cephfs-shell x86_64 2:19.0.0-3156.gd69f13d8.el7 ceph 27 k 2024-04-23T10:05:02.176 INFO:teuthology.orchestra.run.smithi114.stdout: cephfs-top noarch 2:19.0.0-3156.gd69f13d8.el7 ceph-noarch 18 k 2024-04-23T10:05:02.176 INFO:teuthology.orchestra.run.smithi114.stdout: elfutils-libelf-devel x86_64 0.190-2.el9 appstream 21 k 2024-04-23T10:05:02.176 INFO:teuthology.orchestra.run.smithi114.stdout: flex x86_64 2.6.4-9.el9 appstream 313 k 2024-04-23T10:05:02.176 INFO:teuthology.orchestra.run.smithi114.stdout: libaio-devel x86_64 0.3.111-13.el9 appstream 11 k 2024-04-23T10:05:02.176 INFO:teuthology.orchestra.run.smithi114.stdout: libcephfs-devel x86_64 2:19.0.0-3156.gd69f13d8.el7 ceph 32 k 2024-04-23T10:05:02.177 INFO:teuthology.orchestra.run.smithi114.stdout: libcephfs2 x86_64 2:19.0.0-3156.gd69f13d8.el7 ceph 747 k 2024-04-23T10:05:02.177 INFO:teuthology.orchestra.run.smithi114.stdout: librados-devel x86_64 2:19.0.0-3156.gd69f13d8.el7 ceph 126 k 2024-04-23T10:05:02.177 INFO:teuthology.orchestra.run.smithi114.stdout: libtool x86_64 2.4.6-45.el9 appstream 582 k 2024-04-23T10:05:02.177 INFO:teuthology.orchestra.run.smithi114.stdout: libuuid-devel x86_64 2.37.4-18.el9 appstream 25 k 2024-04-23T10:05:02.177 INFO:teuthology.orchestra.run.smithi114.stdout: openssl-devel x86_64 1:3.2.1-1.el9 appstream 4.4 M 2024-04-23T10:05:02.177 INFO:teuthology.orchestra.run.smithi114.stdout: postgresql x86_64 13.14-1.el9 appstream 1.6 M 2024-04-23T10:05:02.177 INFO:teuthology.orchestra.run.smithi114.stdout: postgresql-contrib x86_64 13.14-1.el9 appstream 911 k 2024-04-23T10:05:02.178 INFO:teuthology.orchestra.run.smithi114.stdout: postgresql-server x86_64 13.14-1.el9 appstream 5.8 M 2024-04-23T10:05:02.178 INFO:teuthology.orchestra.run.smithi114.stdout: python3-cephfs x86_64 2:19.0.0-3156.gd69f13d8.el7 ceph 161 k 2024-04-23T10:05:02.178 INFO:teuthology.orchestra.run.smithi114.stdout: python3-rados x86_64 2:19.0.0-3156.gd69f13d8.el7 ceph 320 k 2024-04-23T10:05:02.178 INFO:teuthology.orchestra.run.smithi114.stdout: python3-rbd x86_64 2:19.0.0-3156.gd69f13d8.el7 ceph 298 k 2024-04-23T10:05:02.178 INFO:teuthology.orchestra.run.smithi114.stdout: python3-rgw x86_64 2:19.0.0-3156.gd69f13d8.el7 ceph 99 k 2024-04-23T10:05:02.178 INFO:teuthology.orchestra.run.smithi114.stdout: rbd-fuse x86_64 2:19.0.0-3156.gd69f13d8.el7 ceph 85 k 2024-04-23T10:05:02.179 INFO:teuthology.orchestra.run.smithi114.stdout: rbd-mirror x86_64 2:19.0.0-3156.gd69f13d8.el7 ceph 3.1 M 2024-04-23T10:05:02.179 INFO:teuthology.orchestra.run.smithi114.stdout: rbd-nbd x86_64 2:19.0.0-3156.gd69f13d8.el7 ceph 170 k 2024-04-23T10:05:02.179 INFO:teuthology.orchestra.run.smithi114.stdout: xfsprogs-devel x86_64 6.3.0-1.el9 appstream 94 k 2024-04-23T10:05:02.179 INFO:teuthology.orchestra.run.smithi114.stdout:Upgrading: 2024-04-23T10:05:02.179 INFO:teuthology.orchestra.run.smithi114.stdout: librados2 x86_64 2:19.0.0-3156.gd69f13d8.el7 ceph 3.4 M 2024-04-23T10:05:02.180 INFO:teuthology.orchestra.run.smithi114.stdout: librbd1 x86_64 2:19.0.0-3156.gd69f13d8.el7 ceph 3.1 M 2024-04-23T10:05:02.180 INFO:teuthology.orchestra.run.smithi114.stdout: openssl x86_64 1:3.2.1-1.el9 baseos 1.3 M 2024-04-23T10:05:02.180 INFO:teuthology.orchestra.run.smithi114.stdout: openssl-libs x86_64 1:3.2.1-1.el9 baseos 2.4 M 2024-04-23T10:05:02.180 INFO:teuthology.orchestra.run.smithi114.stdout:Installing dependencies: 2024-04-23T10:05:02.180 INFO:teuthology.orchestra.run.smithi114.stdout: abseil-cpp x86_64 20211102.0-3.el9 epel 548 k 2024-04-23T10:05:02.180 INFO:teuthology.orchestra.run.smithi114.stdout: automake noarch 1.16.2-8.el9 appstream 689 k 2024-04-23T10:05:02.180 INFO:teuthology.orchestra.run.smithi114.stdout: boost-program-options x86_64 1.75.0-8.el9 appstream 107 k 2024-04-23T10:05:02.181 INFO:teuthology.orchestra.run.smithi114.stdout: ceph-common x86_64 2:19.0.0-3156.gd69f13d8.el7 ceph 21 M 2024-04-23T10:05:02.181 INFO:teuthology.orchestra.run.smithi114.stdout: ceph-grafana-dashboards noarch 2:19.0.0-3156.gd69f13d8.el7 ceph-noarch 34 k 2024-04-23T10:05:02.181 INFO:teuthology.orchestra.run.smithi114.stdout: ceph-mds x86_64 2:19.0.0-3156.gd69f13d8.el7 ceph 2.4 M 2024-04-23T10:05:02.181 INFO:teuthology.orchestra.run.smithi114.stdout: ceph-mgr-modules-core noarch 2:19.0.0-3156.gd69f13d8.el7 ceph-noarch 249 k 2024-04-23T10:05:02.181 INFO:teuthology.orchestra.run.smithi114.stdout: ceph-mon x86_64 2:19.0.0-3156.gd69f13d8.el7 ceph 4.7 M 2024-04-23T10:05:02.181 INFO:teuthology.orchestra.run.smithi114.stdout: ceph-osd x86_64 2:19.0.0-3156.gd69f13d8.el7 ceph 17 M 2024-04-23T10:05:02.182 INFO:teuthology.orchestra.run.smithi114.stdout: ceph-prometheus-alerts noarch 2:19.0.0-3156.gd69f13d8.el7 ceph-noarch 16 k 2024-04-23T10:05:02.182 INFO:teuthology.orchestra.run.smithi114.stdout: ceph-selinux x86_64 2:19.0.0-3156.gd69f13d8.el7 ceph 25 k 2024-04-23T10:05:02.182 INFO:teuthology.orchestra.run.smithi114.stdout: flexiblas x86_64 3.0.4-8.el9 appstream 33 k 2024-04-23T10:05:02.182 INFO:teuthology.orchestra.run.smithi114.stdout: flexiblas-netlib x86_64 3.0.4-8.el9 appstream 3.0 M 2024-04-23T10:05:02.182 INFO:teuthology.orchestra.run.smithi114.stdout: flexiblas-openblas-openmp x86_64 3.0.4-8.el9 appstream 18 k 2024-04-23T10:05:02.183 INFO:teuthology.orchestra.run.smithi114.stdout: fuse x86_64 2.9.9-15.el9 baseos 80 k 2024-04-23T10:05:02.183 INFO:teuthology.orchestra.run.smithi114.stdout: gperftools-libs x86_64 2.9.1-2.el9 epel 309 k 2024-04-23T10:05:02.183 INFO:teuthology.orchestra.run.smithi114.stdout: grpc-data noarch 1.46.7-10.el9 epel 19 k 2024-04-23T10:05:02.183 INFO:teuthology.orchestra.run.smithi114.stdout: libarrow x86_64 9.0.0-10.el9 epel 4.4 M 2024-04-23T10:05:02.183 INFO:teuthology.orchestra.run.smithi114.stdout: libarrow-doc noarch 9.0.0-10.el9 epel 26 k 2024-04-23T10:05:02.183 INFO:teuthology.orchestra.run.smithi114.stdout: libcephsqlite x86_64 2:19.0.0-3156.gd69f13d8.el7 ceph 163 k 2024-04-23T10:05:02.183 INFO:teuthology.orchestra.run.smithi114.stdout: libgfortran x86_64 11.4.1-3.el9 baseos 803 k 2024-04-23T10:05:02.184 INFO:teuthology.orchestra.run.smithi114.stdout: liboath x86_64 2.6.7-2.el9 epel 49 k 2024-04-23T10:05:02.184 INFO:teuthology.orchestra.run.smithi114.stdout: libpmemobj x86_64 1.12.1-1.el9 appstream 160 k 2024-04-23T10:05:02.184 INFO:teuthology.orchestra.run.smithi114.stdout: libquadmath x86_64 11.4.1-3.el9 baseos 195 k 2024-04-23T10:05:02.184 INFO:teuthology.orchestra.run.smithi114.stdout: librabbitmq x86_64 0.11.0-7.el9 appstream 45 k 2024-04-23T10:05:02.184 INFO:teuthology.orchestra.run.smithi114.stdout: libradosstriper1 x86_64 2:19.0.0-3156.gd69f13d8.el7 ceph 500 k 2024-04-23T10:05:02.184 INFO:teuthology.orchestra.run.smithi114.stdout: librdkafka x86_64 1.6.1-102.el9 appstream 662 k 2024-04-23T10:05:02.185 INFO:teuthology.orchestra.run.smithi114.stdout: librgw2 x86_64 2:19.0.0-3156.gd69f13d8.el7 ceph 5.4 M 2024-04-23T10:05:02.185 INFO:teuthology.orchestra.run.smithi114.stdout: libunwind x86_64 1.6.2-1.el9 epel 67 k 2024-04-23T10:05:02.185 INFO:teuthology.orchestra.run.smithi114.stdout: libzstd-devel x86_64 1.5.1-2.el9 appstream 47 k 2024-04-23T10:05:02.185 INFO:teuthology.orchestra.run.smithi114.stdout: lttng-ust x86_64 2.12.0-6.el9 appstream 292 k 2024-04-23T10:05:02.185 INFO:teuthology.orchestra.run.smithi114.stdout: lua x86_64 5.4.4-4.el9 appstream 188 k 2024-04-23T10:05:02.185 INFO:teuthology.orchestra.run.smithi114.stdout: lua-devel x86_64 5.4.4-4.el9 crb 22 k 2024-04-23T10:05:02.186 INFO:teuthology.orchestra.run.smithi114.stdout: luarocks noarch 3.9.2-1.el9 epel 151 k 2024-04-23T10:05:02.186 INFO:teuthology.orchestra.run.smithi114.stdout: openblas x86_64 0.3.21-2.el9 appstream 35 k 2024-04-23T10:05:02.186 INFO:teuthology.orchestra.run.smithi114.stdout: openblas-openmp x86_64 0.3.21-2.el9 appstream 4.7 M 2024-04-23T10:05:02.186 INFO:teuthology.orchestra.run.smithi114.stdout: parquet-libs x86_64 9.0.0-10.el9 epel 838 k 2024-04-23T10:05:02.186 INFO:teuthology.orchestra.run.smithi114.stdout: postgresql-private-libs x86_64 13.14-1.el9 appstream 133 k 2024-04-23T10:05:02.186 INFO:teuthology.orchestra.run.smithi114.stdout: protobuf x86_64 3.14.0-13.el9 appstream 1.0 M 2024-04-23T10:05:02.187 INFO:teuthology.orchestra.run.smithi114.stdout: protobuf-compiler x86_64 3.14.0-13.el9 crb 863 k 2024-04-23T10:05:02.187 INFO:teuthology.orchestra.run.smithi114.stdout: python3-asyncssh noarch 2.13.2-5.el9 epel 548 k 2024-04-23T10:05:02.187 INFO:teuthology.orchestra.run.smithi114.stdout: python3-attrs noarch 20.3.0-7.el9 appstream 87 k 2024-04-23T10:05:02.187 INFO:teuthology.orchestra.run.smithi114.stdout: python3-babel noarch 2.9.1-2.el9 appstream 6.0 M 2024-04-23T10:05:02.187 INFO:teuthology.orchestra.run.smithi114.stdout: python3-bcrypt x86_64 3.2.2-1.el9 epel 43 k 2024-04-23T10:05:02.187 INFO:teuthology.orchestra.run.smithi114.stdout: python3-cachetools noarch 4.2.4-1.el9 epel 32 k 2024-04-23T10:05:02.188 INFO:teuthology.orchestra.run.smithi114.stdout: python3-ceph-argparse x86_64 2:19.0.0-3156.gd69f13d8.el7 ceph 45 k 2024-04-23T10:05:02.188 INFO:teuthology.orchestra.run.smithi114.stdout: python3-ceph-common x86_64 2:19.0.0-3156.gd69f13d8.el7 ceph 135 k 2024-04-23T10:05:02.188 INFO:teuthology.orchestra.run.smithi114.stdout: python3-certifi noarch 2021.10.8-2.el9 epel 15 k 2024-04-23T10:05:02.188 INFO:teuthology.orchestra.run.smithi114.stdout: python3-cffi x86_64 1.14.5-5.el9 baseos 253 k 2024-04-23T10:05:02.188 INFO:teuthology.orchestra.run.smithi114.stdout: python3-cheroot noarch 8.6.0-4.el9 epel 172 k 2024-04-23T10:05:02.188 INFO:teuthology.orchestra.run.smithi114.stdout: python3-cherrypy noarch 18.6.1-2.el9 epel 358 k 2024-04-23T10:05:02.189 INFO:teuthology.orchestra.run.smithi114.stdout: python3-cmd2 noarch 2.4.2-2.el9 epel 303 k 2024-04-23T10:05:02.189 INFO:teuthology.orchestra.run.smithi114.stdout: python3-colorama noarch 0.4.4-12.el9 epel 35 k 2024-04-23T10:05:02.189 INFO:teuthology.orchestra.run.smithi114.stdout: python3-cryptography x86_64 36.0.1-4.el9 baseos 1.2 M 2024-04-23T10:05:02.189 INFO:teuthology.orchestra.run.smithi114.stdout: python3-devel x86_64 3.9.18-3.el9 appstream 244 k 2024-04-23T10:05:02.189 INFO:teuthology.orchestra.run.smithi114.stdout: python3-google-auth noarch 1:2.29.0-1.el9 epel 206 k 2024-04-23T10:05:02.189 INFO:teuthology.orchestra.run.smithi114.stdout: python3-grpcio x86_64 1.46.7-10.el9 epel 2.0 M 2024-04-23T10:05:02.190 INFO:teuthology.orchestra.run.smithi114.stdout: python3-grpcio-tools x86_64 1.46.7-10.el9 epel 144 k 2024-04-23T10:05:02.190 INFO:teuthology.orchestra.run.smithi114.stdout: python3-jaraco noarch 8.2.1-3.el9 epel 11 k 2024-04-23T10:05:02.190 INFO:teuthology.orchestra.run.smithi114.stdout: python3-jaraco-classes noarch 3.2.1-5.el9 epel 18 k 2024-04-23T10:05:02.190 INFO:teuthology.orchestra.run.smithi114.stdout: python3-jaraco-collections noarch 3.0.0-8.el9 epel 23 k 2024-04-23T10:05:02.190 INFO:teuthology.orchestra.run.smithi114.stdout: python3-jaraco-functools noarch 3.5.0-2.el9 epel 19 k 2024-04-23T10:05:02.190 INFO:teuthology.orchestra.run.smithi114.stdout: python3-jaraco-text noarch 3.2.0-6.el9 epel 20 k 2024-04-23T10:05:02.191 INFO:teuthology.orchestra.run.smithi114.stdout: python3-jinja2 noarch 2.11.3-5.el9 appstream 248 k 2024-04-23T10:05:02.191 INFO:teuthology.orchestra.run.smithi114.stdout: python3-jsonpatch noarch 1.21-16.el9 appstream 26 k 2024-04-23T10:05:02.191 INFO:teuthology.orchestra.run.smithi114.stdout: python3-jsonpointer noarch 2.0-4.el9 appstream 19 k 2024-04-23T10:05:02.191 INFO:teuthology.orchestra.run.smithi114.stdout: python3-kubernetes noarch 1:26.1.0-2.el9 epel 1.0 M 2024-04-23T10:05:02.191 INFO:teuthology.orchestra.run.smithi114.stdout: python3-logutils noarch 0.3.5-21.el9 epel 46 k 2024-04-23T10:05:02.191 INFO:teuthology.orchestra.run.smithi114.stdout: python3-mako noarch 1.1.4-6.el9 appstream 172 k 2024-04-23T10:05:02.192 INFO:teuthology.orchestra.run.smithi114.stdout: python3-markupsafe x86_64 1.1.1-12.el9 appstream 35 k 2024-04-23T10:05:02.192 INFO:teuthology.orchestra.run.smithi114.stdout: python3-more-itertools noarch 8.12.0-2.el9 epel 79 k 2024-04-23T10:05:02.192 INFO:teuthology.orchestra.run.smithi114.stdout: python3-natsort noarch 7.1.1-5.el9 epel 58 k 2024-04-23T10:05:02.192 INFO:teuthology.orchestra.run.smithi114.stdout: python3-numpy x86_64 1:1.20.1-5.el9 appstream 5.1 M 2024-04-23T10:05:02.192 INFO:teuthology.orchestra.run.smithi114.stdout: python3-numpy-f2py x86_64 1:1.20.1-5.el9 appstream 250 k 2024-04-23T10:05:02.192 INFO:teuthology.orchestra.run.smithi114.stdout: python3-oauthlib noarch 3.1.1-5.el9 appstream 222 k 2024-04-23T10:05:02.193 INFO:teuthology.orchestra.run.smithi114.stdout: python3-packaging noarch 20.9-5.el9 appstream 77 k 2024-04-23T10:05:02.193 INFO:teuthology.orchestra.run.smithi114.stdout: python3-pecan noarch 1.4.2-3.el9 epel 272 k 2024-04-23T10:05:02.193 INFO:teuthology.orchestra.run.smithi114.stdout: python3-ply noarch 3.11-14.el9 baseos 106 k 2024-04-23T10:05:02.193 INFO:teuthology.orchestra.run.smithi114.stdout: python3-portend noarch 3.1.0-2.el9 epel 16 k 2024-04-23T10:05:02.193 INFO:teuthology.orchestra.run.smithi114.stdout: python3-prettytable noarch 0.7.2-27.el9 appstream 42 k 2024-04-23T10:05:02.193 INFO:teuthology.orchestra.run.smithi114.stdout: python3-protobuf noarch 3.14.0-13.el9 appstream 269 k 2024-04-23T10:05:02.194 INFO:teuthology.orchestra.run.smithi114.stdout: python3-pyOpenSSL noarch 21.0.0-1.el9 epel 90 k 2024-04-23T10:05:02.194 INFO:teuthology.orchestra.run.smithi114.stdout: python3-pyasn1 noarch 0.4.8-6.el9 appstream 159 k 2024-04-23T10:05:02.194 INFO:teuthology.orchestra.run.smithi114.stdout: python3-pyasn1-modules noarch 0.4.8-6.el9 appstream 279 k 2024-04-23T10:05:02.194 INFO:teuthology.orchestra.run.smithi114.stdout: python3-pycparser noarch 2.20-6.el9 baseos 135 k 2024-04-23T10:05:02.194 INFO:teuthology.orchestra.run.smithi114.stdout: python3-pyperclip noarch 1.8.2-2.el9 epel 28 k 2024-04-23T10:05:02.194 INFO:teuthology.orchestra.run.smithi114.stdout: python3-pytz noarch 2021.1-5.el9 appstream 51 k 2024-04-23T10:05:02.195 INFO:teuthology.orchestra.run.smithi114.stdout: python3-repoze-lru noarch 0.7-16.el9 epel 31 k 2024-04-23T10:05:02.195 INFO:teuthology.orchestra.run.smithi114.stdout: python3-requests noarch 2.25.1-8.el9 baseos 125 k 2024-04-23T10:05:02.195 INFO:teuthology.orchestra.run.smithi114.stdout: python3-requests-oauthlib noarch 1.3.0-12.el9 appstream 54 k 2024-04-23T10:05:02.195 INFO:teuthology.orchestra.run.smithi114.stdout: python3-routes noarch 2.5.1-5.el9 epel 188 k 2024-04-23T10:05:02.195 INFO:teuthology.orchestra.run.smithi114.stdout: python3-rsa noarch 4.9-2.el9 epel 59 k 2024-04-23T10:05:02.195 INFO:teuthology.orchestra.run.smithi114.stdout: python3-scipy x86_64 1.6.2-8.el9 appstream 16 M 2024-04-23T10:05:02.196 INFO:teuthology.orchestra.run.smithi114.stdout: python3-tempora noarch 5.0.0-2.el9 epel 36 k 2024-04-23T10:05:02.196 INFO:teuthology.orchestra.run.smithi114.stdout: python3-toml noarch 0.10.2-6.el9 appstream 42 k 2024-04-23T10:05:02.196 INFO:teuthology.orchestra.run.smithi114.stdout: python3-typing-extensions noarch 4.4.0-2.el9 epel 51 k 2024-04-23T10:05:02.196 INFO:teuthology.orchestra.run.smithi114.stdout: python3-urllib3 noarch 1.26.5-5.el9 baseos 215 k 2024-04-23T10:05:02.196 INFO:teuthology.orchestra.run.smithi114.stdout: python3-wcwidth noarch 0.2.5-8.el9 appstream 44 k 2024-04-23T10:05:02.196 INFO:teuthology.orchestra.run.smithi114.stdout: python3-webob noarch 1.8.7-6.el9 epel 230 k 2024-04-23T10:05:02.197 INFO:teuthology.orchestra.run.smithi114.stdout: python3-websocket-client noarch 1.2.3-2.el9 epel 90 k 2024-04-23T10:05:02.197 INFO:teuthology.orchestra.run.smithi114.stdout: python3-werkzeug noarch 2.0.3-3.el9.1 epel 427 k 2024-04-23T10:05:02.197 INFO:teuthology.orchestra.run.smithi114.stdout: python3-zc-lockfile noarch 2.0-10.el9 epel 20 k 2024-04-23T10:05:02.197 INFO:teuthology.orchestra.run.smithi114.stdout: qatlib x86_64 24.02.0-1.el9 appstream 221 k 2024-04-23T10:05:02.197 INFO:teuthology.orchestra.run.smithi114.stdout: qatzip-libs x86_64 1.2.0-1.el9 appstream 47 k 2024-04-23T10:05:02.197 INFO:teuthology.orchestra.run.smithi114.stdout: re2 x86_64 1:20211101-3.el9 epel 192 k 2024-04-23T10:05:02.198 INFO:teuthology.orchestra.run.smithi114.stdout: socat x86_64 1.7.4.1-5.el9 appstream 305 k 2024-04-23T10:05:02.198 INFO:teuthology.orchestra.run.smithi114.stdout: thrift x86_64 0.15.0-2.el9 epel 1.6 M 2024-04-23T10:05:02.198 INFO:teuthology.orchestra.run.smithi114.stdout: uuid x86_64 1.6.2-55.el9 appstream 58 k 2024-04-23T10:05:02.198 INFO:teuthology.orchestra.run.smithi114.stdout: xmlstarlet x86_64 1.6.1-20.el9 appstream 64 k 2024-04-23T10:05:02.198 INFO:teuthology.orchestra.run.smithi114.stdout: zlib-devel x86_64 1.2.11-41.el9 appstream 45 k 2024-04-23T10:05:02.198 INFO:teuthology.orchestra.run.smithi114.stdout:Installing weak dependencies: 2024-04-23T10:05:02.199 INFO:teuthology.orchestra.run.smithi114.stdout: qatlib-service x86_64 24.02.0-1.el9 appstream 36 k 2024-04-23T10:05:02.199 INFO:teuthology.orchestra.run.smithi114.stdout: 2024-04-23T10:05:02.199 INFO:teuthology.orchestra.run.smithi114.stdout:Transaction Summary 2024-04-23T10:05:02.199 INFO:teuthology.orchestra.run.smithi114.stdout:======================================================================================= 2024-04-23T10:05:02.199 INFO:teuthology.orchestra.run.smithi114.stdout:Install 148 Packages 2024-04-23T10:05:02.199 INFO:teuthology.orchestra.run.smithi114.stdout:Upgrade 4 Packages 2024-04-23T10:05:02.200 INFO:teuthology.orchestra.run.smithi114.stdout: 2024-04-23T10:05:02.201 INFO:teuthology.orchestra.run.smithi114.stdout:Total download size: 222 M 2024-04-23T10:05:02.201 INFO:teuthology.orchestra.run.smithi114.stdout:Downloading Packages: 2024-04-23T10:05:03.676 INFO:teuthology.orchestra.run.smithi114.stdout:(1/152): ceph-19.0.0-3156.gd69f13d8.el7.x86_64. 11 kB/s | 6.6 kB 00:00 2024-04-23T10:05:04.045 INFO:teuthology.orchestra.run.smithi204.stdout:lab-extras 11 kB/s | 1.7 kB 00:00 2024-04-23T10:05:06.180 INFO:teuthology.orchestra.run.smithi204.stdout:Package librados2-2:16.2.4-5.el9.x86_64 is already installed. 2024-04-23T10:05:06.181 INFO:teuthology.orchestra.run.smithi204.stdout:Package librbd1-2:16.2.4-5.el9.x86_64 is already installed. 2024-04-23T10:05:06.192 INFO:teuthology.orchestra.run.smithi204.stdout:Package NetworkManager-1:1.47.4-1.el9.x86_64 is already installed. 2024-04-23T10:05:06.193 INFO:teuthology.orchestra.run.smithi204.stdout:Package iproute-6.2.0-5.el9.x86_64 is already installed. 2024-04-23T10:05:06.194 INFO:teuthology.orchestra.run.smithi204.stdout:Package util-linux-2.37.4-18.el9.x86_64 is already installed. 2024-04-23T10:05:06.194 INFO:teuthology.orchestra.run.smithi204.stdout:Package libacl-devel-2.3.1-4.el9.x86_64 is already installed. 2024-04-23T10:05:06.196 INFO:teuthology.orchestra.run.smithi204.stdout:Package libattr-devel-2.5.1-3.el9.x86_64 is already installed. 2024-04-23T10:05:06.198 INFO:teuthology.orchestra.run.smithi204.stdout:Package xfsdump-3.1.12-4.el9.x86_64 is already installed. 2024-04-23T10:05:06.199 INFO:teuthology.orchestra.run.smithi204.stdout:Package xfsprogs-6.3.0-1.el9.x86_64 is already installed. 2024-04-23T10:05:06.259 INFO:teuthology.orchestra.run.smithi204.stdout:Dependencies resolved. 2024-04-23T10:05:06.273 INFO:teuthology.orchestra.run.smithi204.stdout:======================================================================================= 2024-04-23T10:05:06.273 INFO:teuthology.orchestra.run.smithi204.stdout: Package Arch Version Repository Size 2024-04-23T10:05:06.273 INFO:teuthology.orchestra.run.smithi204.stdout:======================================================================================= 2024-04-23T10:05:06.273 INFO:teuthology.orchestra.run.smithi204.stdout:Installing: 2024-04-23T10:05:06.273 INFO:teuthology.orchestra.run.smithi204.stdout: bison x86_64 3.7.4-5.el9 appstream 943 k 2024-04-23T10:05:06.274 INFO:teuthology.orchestra.run.smithi204.stdout: ceph x86_64 2:19.0.0-3156.gd69f13d8.el7 ceph 6.6 k 2024-04-23T10:05:06.274 INFO:teuthology.orchestra.run.smithi204.stdout: ceph-base x86_64 2:19.0.0-3156.gd69f13d8.el7 ceph 5.4 M 2024-04-23T10:05:06.274 INFO:teuthology.orchestra.run.smithi204.stdout: ceph-fuse x86_64 2:19.0.0-3156.gd69f13d8.el7 ceph 890 k 2024-04-23T10:05:06.274 INFO:teuthology.orchestra.run.smithi204.stdout: ceph-immutable-object-cache x86_64 2:19.0.0-3156.gd69f13d8.el7 ceph 145 k 2024-04-23T10:05:06.274 INFO:teuthology.orchestra.run.smithi204.stdout: ceph-mgr x86_64 2:19.0.0-3156.gd69f13d8.el7 ceph 1.5 M 2024-04-23T10:05:06.274 INFO:teuthology.orchestra.run.smithi204.stdout: ceph-mgr-cephadm noarch 2:19.0.0-3156.gd69f13d8.el7 ceph-noarch 142 k 2024-04-23T10:05:06.274 INFO:teuthology.orchestra.run.smithi204.stdout: ceph-mgr-dashboard noarch 2:19.0.0-3156.gd69f13d8.el7 ceph-noarch 3.5 M 2024-04-23T10:05:06.274 INFO:teuthology.orchestra.run.smithi204.stdout: ceph-mgr-diskprediction-local noarch 2:19.0.0-3156.gd69f13d8.el7 ceph-noarch 7.4 M 2024-04-23T10:05:06.274 INFO:teuthology.orchestra.run.smithi204.stdout: ceph-mgr-rook noarch 2:19.0.0-3156.gd69f13d8.el7 ceph-noarch 49 k 2024-04-23T10:05:06.274 INFO:teuthology.orchestra.run.smithi204.stdout: ceph-radosgw x86_64 2:19.0.0-3156.gd69f13d8.el7 ceph 11 M 2024-04-23T10:05:06.274 INFO:teuthology.orchestra.run.smithi204.stdout: ceph-test x86_64 2:19.0.0-3156.gd69f13d8.el7 ceph 49 M 2024-04-23T10:05:06.274 INFO:teuthology.orchestra.run.smithi204.stdout: ceph-volume noarch 2:19.0.0-3156.gd69f13d8.el7 ceph-noarch 273 k 2024-04-23T10:05:06.275 INFO:teuthology.orchestra.run.smithi204.stdout: cephadm noarch 2:19.0.0-3156.gd69f13d8.el7 ceph-noarch 762 k 2024-04-23T10:05:06.275 INFO:teuthology.orchestra.run.smithi204.stdout: cephfs-mirror x86_64 2:19.0.0-3156.gd69f13d8.el7 ceph 225 k 2024-04-23T10:05:06.275 INFO:teuthology.orchestra.run.smithi204.stdout: cephfs-shell x86_64 2:19.0.0-3156.gd69f13d8.el7 ceph 27 k 2024-04-23T10:05:06.275 INFO:teuthology.orchestra.run.smithi204.stdout: cephfs-top noarch 2:19.0.0-3156.gd69f13d8.el7 ceph-noarch 18 k 2024-04-23T10:05:06.275 INFO:teuthology.orchestra.run.smithi204.stdout: elfutils-libelf-devel x86_64 0.190-2.el9 appstream 21 k 2024-04-23T10:05:06.275 INFO:teuthology.orchestra.run.smithi204.stdout: flex x86_64 2.6.4-9.el9 appstream 313 k 2024-04-23T10:05:06.275 INFO:teuthology.orchestra.run.smithi204.stdout: libaio-devel x86_64 0.3.111-13.el9 appstream 11 k 2024-04-23T10:05:06.275 INFO:teuthology.orchestra.run.smithi204.stdout: libcephfs-devel x86_64 2:19.0.0-3156.gd69f13d8.el7 ceph 32 k 2024-04-23T10:05:06.275 INFO:teuthology.orchestra.run.smithi204.stdout: libcephfs2 x86_64 2:19.0.0-3156.gd69f13d8.el7 ceph 747 k 2024-04-23T10:05:06.275 INFO:teuthology.orchestra.run.smithi204.stdout: librados-devel x86_64 2:19.0.0-3156.gd69f13d8.el7 ceph 126 k 2024-04-23T10:05:06.275 INFO:teuthology.orchestra.run.smithi204.stdout: libtool x86_64 2.4.6-45.el9 appstream 582 k 2024-04-23T10:05:06.275 INFO:teuthology.orchestra.run.smithi204.stdout: libuuid-devel x86_64 2.37.4-18.el9 appstream 25 k 2024-04-23T10:05:06.275 INFO:teuthology.orchestra.run.smithi204.stdout: openssl-devel x86_64 1:3.2.1-1.el9 appstream 4.4 M 2024-04-23T10:05:06.276 INFO:teuthology.orchestra.run.smithi204.stdout: postgresql x86_64 13.14-1.el9 appstream 1.6 M 2024-04-23T10:05:06.276 INFO:teuthology.orchestra.run.smithi204.stdout: postgresql-contrib x86_64 13.14-1.el9 appstream 911 k 2024-04-23T10:05:06.276 INFO:teuthology.orchestra.run.smithi204.stdout: postgresql-server x86_64 13.14-1.el9 appstream 5.8 M 2024-04-23T10:05:06.276 INFO:teuthology.orchestra.run.smithi204.stdout: python3-cephfs x86_64 2:19.0.0-3156.gd69f13d8.el7 ceph 161 k 2024-04-23T10:05:06.276 INFO:teuthology.orchestra.run.smithi204.stdout: python3-rados x86_64 2:19.0.0-3156.gd69f13d8.el7 ceph 320 k 2024-04-23T10:05:06.276 INFO:teuthology.orchestra.run.smithi204.stdout: python3-rbd x86_64 2:19.0.0-3156.gd69f13d8.el7 ceph 298 k 2024-04-23T10:05:06.276 INFO:teuthology.orchestra.run.smithi204.stdout: python3-rgw x86_64 2:19.0.0-3156.gd69f13d8.el7 ceph 99 k 2024-04-23T10:05:06.276 INFO:teuthology.orchestra.run.smithi204.stdout: rbd-fuse x86_64 2:19.0.0-3156.gd69f13d8.el7 ceph 85 k 2024-04-23T10:05:06.276 INFO:teuthology.orchestra.run.smithi204.stdout: rbd-mirror x86_64 2:19.0.0-3156.gd69f13d8.el7 ceph 3.1 M 2024-04-23T10:05:06.276 INFO:teuthology.orchestra.run.smithi204.stdout: rbd-nbd x86_64 2:19.0.0-3156.gd69f13d8.el7 ceph 170 k 2024-04-23T10:05:06.276 INFO:teuthology.orchestra.run.smithi204.stdout: xfsprogs-devel x86_64 6.3.0-1.el9 appstream 94 k 2024-04-23T10:05:06.276 INFO:teuthology.orchestra.run.smithi204.stdout:Upgrading: 2024-04-23T10:05:06.276 INFO:teuthology.orchestra.run.smithi204.stdout: librados2 x86_64 2:19.0.0-3156.gd69f13d8.el7 ceph 3.4 M 2024-04-23T10:05:06.276 INFO:teuthology.orchestra.run.smithi204.stdout: librbd1 x86_64 2:19.0.0-3156.gd69f13d8.el7 ceph 3.1 M 2024-04-23T10:05:06.277 INFO:teuthology.orchestra.run.smithi204.stdout: openssl x86_64 1:3.2.1-1.el9 baseos 1.3 M 2024-04-23T10:05:06.277 INFO:teuthology.orchestra.run.smithi204.stdout: openssl-libs x86_64 1:3.2.1-1.el9 baseos 2.4 M 2024-04-23T10:05:06.277 INFO:teuthology.orchestra.run.smithi204.stdout:Installing dependencies: 2024-04-23T10:05:06.277 INFO:teuthology.orchestra.run.smithi204.stdout: abseil-cpp x86_64 20211102.0-3.el9 epel 548 k 2024-04-23T10:05:06.277 INFO:teuthology.orchestra.run.smithi204.stdout: automake noarch 1.16.2-8.el9 appstream 689 k 2024-04-23T10:05:06.277 INFO:teuthology.orchestra.run.smithi204.stdout: boost-program-options x86_64 1.75.0-8.el9 appstream 107 k 2024-04-23T10:05:06.277 INFO:teuthology.orchestra.run.smithi204.stdout: ceph-common x86_64 2:19.0.0-3156.gd69f13d8.el7 ceph 21 M 2024-04-23T10:05:06.277 INFO:teuthology.orchestra.run.smithi204.stdout: ceph-grafana-dashboards noarch 2:19.0.0-3156.gd69f13d8.el7 ceph-noarch 34 k 2024-04-23T10:05:06.277 INFO:teuthology.orchestra.run.smithi204.stdout: ceph-mds x86_64 2:19.0.0-3156.gd69f13d8.el7 ceph 2.4 M 2024-04-23T10:05:06.277 INFO:teuthology.orchestra.run.smithi204.stdout: ceph-mgr-modules-core noarch 2:19.0.0-3156.gd69f13d8.el7 ceph-noarch 249 k 2024-04-23T10:05:06.277 INFO:teuthology.orchestra.run.smithi204.stdout: ceph-mon x86_64 2:19.0.0-3156.gd69f13d8.el7 ceph 4.7 M 2024-04-23T10:05:06.277 INFO:teuthology.orchestra.run.smithi204.stdout: ceph-osd x86_64 2:19.0.0-3156.gd69f13d8.el7 ceph 17 M 2024-04-23T10:05:06.278 INFO:teuthology.orchestra.run.smithi204.stdout: ceph-prometheus-alerts noarch 2:19.0.0-3156.gd69f13d8.el7 ceph-noarch 16 k 2024-04-23T10:05:06.278 INFO:teuthology.orchestra.run.smithi204.stdout: ceph-selinux x86_64 2:19.0.0-3156.gd69f13d8.el7 ceph 25 k 2024-04-23T10:05:06.278 INFO:teuthology.orchestra.run.smithi204.stdout: flexiblas x86_64 3.0.4-8.el9 appstream 33 k 2024-04-23T10:05:06.278 INFO:teuthology.orchestra.run.smithi204.stdout: flexiblas-netlib x86_64 3.0.4-8.el9 appstream 3.0 M 2024-04-23T10:05:06.278 INFO:teuthology.orchestra.run.smithi204.stdout: flexiblas-openblas-openmp x86_64 3.0.4-8.el9 appstream 18 k 2024-04-23T10:05:06.278 INFO:teuthology.orchestra.run.smithi204.stdout: fuse x86_64 2.9.9-15.el9 baseos 80 k 2024-04-23T10:05:06.278 INFO:teuthology.orchestra.run.smithi204.stdout: gperftools-libs x86_64 2.9.1-2.el9 epel 309 k 2024-04-23T10:05:06.278 INFO:teuthology.orchestra.run.smithi204.stdout: grpc-data noarch 1.46.7-10.el9 epel 19 k 2024-04-23T10:05:06.278 INFO:teuthology.orchestra.run.smithi204.stdout: libarrow x86_64 9.0.0-10.el9 epel 4.4 M 2024-04-23T10:05:06.278 INFO:teuthology.orchestra.run.smithi204.stdout: libarrow-doc noarch 9.0.0-10.el9 epel 26 k 2024-04-23T10:05:06.278 INFO:teuthology.orchestra.run.smithi204.stdout: libcephsqlite x86_64 2:19.0.0-3156.gd69f13d8.el7 ceph 163 k 2024-04-23T10:05:06.278 INFO:teuthology.orchestra.run.smithi204.stdout: libgfortran x86_64 11.4.1-3.el9 baseos 803 k 2024-04-23T10:05:06.279 INFO:teuthology.orchestra.run.smithi204.stdout: liboath x86_64 2.6.7-2.el9 epel 49 k 2024-04-23T10:05:06.279 INFO:teuthology.orchestra.run.smithi204.stdout: libpmemobj x86_64 1.12.1-1.el9 appstream 160 k 2024-04-23T10:05:06.279 INFO:teuthology.orchestra.run.smithi204.stdout: libquadmath x86_64 11.4.1-3.el9 baseos 195 k 2024-04-23T10:05:06.279 INFO:teuthology.orchestra.run.smithi204.stdout: librabbitmq x86_64 0.11.0-7.el9 appstream 45 k 2024-04-23T10:05:06.279 INFO:teuthology.orchestra.run.smithi204.stdout: libradosstriper1 x86_64 2:19.0.0-3156.gd69f13d8.el7 ceph 500 k 2024-04-23T10:05:06.279 INFO:teuthology.orchestra.run.smithi204.stdout: librdkafka x86_64 1.6.1-102.el9 appstream 662 k 2024-04-23T10:05:06.279 INFO:teuthology.orchestra.run.smithi204.stdout: librgw2 x86_64 2:19.0.0-3156.gd69f13d8.el7 ceph 5.4 M 2024-04-23T10:05:06.279 INFO:teuthology.orchestra.run.smithi204.stdout: libunwind x86_64 1.6.2-1.el9 epel 67 k 2024-04-23T10:05:06.279 INFO:teuthology.orchestra.run.smithi204.stdout: libzstd-devel x86_64 1.5.1-2.el9 appstream 47 k 2024-04-23T10:05:06.279 INFO:teuthology.orchestra.run.smithi204.stdout: lttng-ust x86_64 2.12.0-6.el9 appstream 292 k 2024-04-23T10:05:06.279 INFO:teuthology.orchestra.run.smithi204.stdout: lua x86_64 5.4.4-4.el9 appstream 188 k 2024-04-23T10:05:06.279 INFO:teuthology.orchestra.run.smithi204.stdout: lua-devel x86_64 5.4.4-4.el9 crb 22 k 2024-04-23T10:05:06.279 INFO:teuthology.orchestra.run.smithi204.stdout: luarocks noarch 3.9.2-1.el9 epel 151 k 2024-04-23T10:05:06.280 INFO:teuthology.orchestra.run.smithi204.stdout: openblas x86_64 0.3.21-2.el9 appstream 35 k 2024-04-23T10:05:06.280 INFO:teuthology.orchestra.run.smithi204.stdout: openblas-openmp x86_64 0.3.21-2.el9 appstream 4.7 M 2024-04-23T10:05:06.280 INFO:teuthology.orchestra.run.smithi204.stdout: parquet-libs x86_64 9.0.0-10.el9 epel 838 k 2024-04-23T10:05:06.280 INFO:teuthology.orchestra.run.smithi204.stdout: postgresql-private-libs x86_64 13.14-1.el9 appstream 133 k 2024-04-23T10:05:06.280 INFO:teuthology.orchestra.run.smithi204.stdout: protobuf x86_64 3.14.0-13.el9 appstream 1.0 M 2024-04-23T10:05:06.280 INFO:teuthology.orchestra.run.smithi204.stdout: protobuf-compiler x86_64 3.14.0-13.el9 crb 863 k 2024-04-23T10:05:06.280 INFO:teuthology.orchestra.run.smithi204.stdout: python3-asyncssh noarch 2.13.2-5.el9 epel 548 k 2024-04-23T10:05:06.280 INFO:teuthology.orchestra.run.smithi204.stdout: python3-attrs noarch 20.3.0-7.el9 appstream 87 k 2024-04-23T10:05:06.280 INFO:teuthology.orchestra.run.smithi204.stdout: python3-babel noarch 2.9.1-2.el9 appstream 6.0 M 2024-04-23T10:05:06.280 INFO:teuthology.orchestra.run.smithi204.stdout: python3-bcrypt x86_64 3.2.2-1.el9 epel 43 k 2024-04-23T10:05:06.280 INFO:teuthology.orchestra.run.smithi204.stdout: python3-cachetools noarch 4.2.4-1.el9 epel 32 k 2024-04-23T10:05:06.280 INFO:teuthology.orchestra.run.smithi204.stdout: python3-ceph-argparse x86_64 2:19.0.0-3156.gd69f13d8.el7 ceph 45 k 2024-04-23T10:05:06.280 INFO:teuthology.orchestra.run.smithi204.stdout: python3-ceph-common x86_64 2:19.0.0-3156.gd69f13d8.el7 ceph 135 k 2024-04-23T10:05:06.281 INFO:teuthology.orchestra.run.smithi204.stdout: python3-certifi noarch 2021.10.8-2.el9 epel 15 k 2024-04-23T10:05:06.281 INFO:teuthology.orchestra.run.smithi204.stdout: python3-cffi x86_64 1.14.5-5.el9 baseos 253 k 2024-04-23T10:05:06.281 INFO:teuthology.orchestra.run.smithi204.stdout: python3-cheroot noarch 8.6.0-4.el9 epel 172 k 2024-04-23T10:05:06.281 INFO:teuthology.orchestra.run.smithi204.stdout: python3-cherrypy noarch 18.6.1-2.el9 epel 358 k 2024-04-23T10:05:06.281 INFO:teuthology.orchestra.run.smithi204.stdout: python3-cmd2 noarch 2.4.2-2.el9 epel 303 k 2024-04-23T10:05:06.281 INFO:teuthology.orchestra.run.smithi204.stdout: python3-colorama noarch 0.4.4-12.el9 epel 35 k 2024-04-23T10:05:06.281 INFO:teuthology.orchestra.run.smithi204.stdout: python3-cryptography x86_64 36.0.1-4.el9 baseos 1.2 M 2024-04-23T10:05:06.281 INFO:teuthology.orchestra.run.smithi204.stdout: python3-devel x86_64 3.9.18-3.el9 appstream 244 k 2024-04-23T10:05:06.281 INFO:teuthology.orchestra.run.smithi204.stdout: python3-google-auth noarch 1:2.29.0-1.el9 epel 206 k 2024-04-23T10:05:06.281 INFO:teuthology.orchestra.run.smithi204.stdout: python3-grpcio x86_64 1.46.7-10.el9 epel 2.0 M 2024-04-23T10:05:06.281 INFO:teuthology.orchestra.run.smithi204.stdout: python3-grpcio-tools x86_64 1.46.7-10.el9 epel 144 k 2024-04-23T10:05:06.281 INFO:teuthology.orchestra.run.smithi204.stdout: python3-jaraco noarch 8.2.1-3.el9 epel 11 k 2024-04-23T10:05:06.281 INFO:teuthology.orchestra.run.smithi204.stdout: python3-jaraco-classes noarch 3.2.1-5.el9 epel 18 k 2024-04-23T10:05:06.282 INFO:teuthology.orchestra.run.smithi204.stdout: python3-jaraco-collections noarch 3.0.0-8.el9 epel 23 k 2024-04-23T10:05:06.282 INFO:teuthology.orchestra.run.smithi204.stdout: python3-jaraco-functools noarch 3.5.0-2.el9 epel 19 k 2024-04-23T10:05:06.282 INFO:teuthology.orchestra.run.smithi204.stdout: python3-jaraco-text noarch 3.2.0-6.el9 epel 20 k 2024-04-23T10:05:06.282 INFO:teuthology.orchestra.run.smithi204.stdout: python3-jinja2 noarch 2.11.3-5.el9 appstream 248 k 2024-04-23T10:05:06.282 INFO:teuthology.orchestra.run.smithi204.stdout: python3-jsonpatch noarch 1.21-16.el9 appstream 26 k 2024-04-23T10:05:06.282 INFO:teuthology.orchestra.run.smithi204.stdout: python3-jsonpointer noarch 2.0-4.el9 appstream 19 k 2024-04-23T10:05:06.282 INFO:teuthology.orchestra.run.smithi204.stdout: python3-kubernetes noarch 1:26.1.0-2.el9 epel 1.0 M 2024-04-23T10:05:06.282 INFO:teuthology.orchestra.run.smithi204.stdout: python3-logutils noarch 0.3.5-21.el9 epel 46 k 2024-04-23T10:05:06.282 INFO:teuthology.orchestra.run.smithi204.stdout: python3-mako noarch 1.1.4-6.el9 appstream 172 k 2024-04-23T10:05:06.282 INFO:teuthology.orchestra.run.smithi204.stdout: python3-markupsafe x86_64 1.1.1-12.el9 appstream 35 k 2024-04-23T10:05:06.282 INFO:teuthology.orchestra.run.smithi204.stdout: python3-more-itertools noarch 8.12.0-2.el9 epel 79 k 2024-04-23T10:05:06.282 INFO:teuthology.orchestra.run.smithi204.stdout: python3-natsort noarch 7.1.1-5.el9 epel 58 k 2024-04-23T10:05:06.282 INFO:teuthology.orchestra.run.smithi204.stdout: python3-numpy x86_64 1:1.20.1-5.el9 appstream 5.1 M 2024-04-23T10:05:06.282 INFO:teuthology.orchestra.run.smithi204.stdout: python3-numpy-f2py x86_64 1:1.20.1-5.el9 appstream 250 k 2024-04-23T10:05:06.283 INFO:teuthology.orchestra.run.smithi204.stdout: python3-oauthlib noarch 3.1.1-5.el9 appstream 222 k 2024-04-23T10:05:06.283 INFO:teuthology.orchestra.run.smithi204.stdout: python3-packaging noarch 20.9-5.el9 appstream 77 k 2024-04-23T10:05:06.283 INFO:teuthology.orchestra.run.smithi204.stdout: python3-pecan noarch 1.4.2-3.el9 epel 272 k 2024-04-23T10:05:06.283 INFO:teuthology.orchestra.run.smithi204.stdout: python3-ply noarch 3.11-14.el9 baseos 106 k 2024-04-23T10:05:06.283 INFO:teuthology.orchestra.run.smithi204.stdout: python3-portend noarch 3.1.0-2.el9 epel 16 k 2024-04-23T10:05:06.283 INFO:teuthology.orchestra.run.smithi204.stdout: python3-prettytable noarch 0.7.2-27.el9 appstream 42 k 2024-04-23T10:05:06.283 INFO:teuthology.orchestra.run.smithi204.stdout: python3-protobuf noarch 3.14.0-13.el9 appstream 269 k 2024-04-23T10:05:06.283 INFO:teuthology.orchestra.run.smithi204.stdout: python3-pyOpenSSL noarch 21.0.0-1.el9 epel 90 k 2024-04-23T10:05:06.283 INFO:teuthology.orchestra.run.smithi204.stdout: python3-pyasn1 noarch 0.4.8-6.el9 appstream 159 k 2024-04-23T10:05:06.283 INFO:teuthology.orchestra.run.smithi204.stdout: python3-pyasn1-modules noarch 0.4.8-6.el9 appstream 279 k 2024-04-23T10:05:06.283 INFO:teuthology.orchestra.run.smithi204.stdout: python3-pycparser noarch 2.20-6.el9 baseos 135 k 2024-04-23T10:05:06.283 INFO:teuthology.orchestra.run.smithi204.stdout: python3-pyperclip noarch 1.8.2-2.el9 epel 28 k 2024-04-23T10:05:06.284 INFO:teuthology.orchestra.run.smithi204.stdout: python3-pytz noarch 2021.1-5.el9 appstream 51 k 2024-04-23T10:05:06.284 INFO:teuthology.orchestra.run.smithi204.stdout: python3-repoze-lru noarch 0.7-16.el9 epel 31 k 2024-04-23T10:05:06.284 INFO:teuthology.orchestra.run.smithi204.stdout: python3-requests noarch 2.25.1-8.el9 baseos 125 k 2024-04-23T10:05:06.284 INFO:teuthology.orchestra.run.smithi204.stdout: python3-requests-oauthlib noarch 1.3.0-12.el9 appstream 54 k 2024-04-23T10:05:06.284 INFO:teuthology.orchestra.run.smithi204.stdout: python3-routes noarch 2.5.1-5.el9 epel 188 k 2024-04-23T10:05:06.284 INFO:teuthology.orchestra.run.smithi204.stdout: python3-rsa noarch 4.9-2.el9 epel 59 k 2024-04-23T10:05:06.284 INFO:teuthology.orchestra.run.smithi204.stdout: python3-scipy x86_64 1.6.2-8.el9 appstream 16 M 2024-04-23T10:05:06.284 INFO:teuthology.orchestra.run.smithi204.stdout: python3-tempora noarch 5.0.0-2.el9 epel 36 k 2024-04-23T10:05:06.284 INFO:teuthology.orchestra.run.smithi204.stdout: python3-toml noarch 0.10.2-6.el9 appstream 42 k 2024-04-23T10:05:06.284 INFO:teuthology.orchestra.run.smithi204.stdout: python3-typing-extensions noarch 4.4.0-2.el9 epel 51 k 2024-04-23T10:05:06.284 INFO:teuthology.orchestra.run.smithi204.stdout: python3-urllib3 noarch 1.26.5-5.el9 baseos 215 k 2024-04-23T10:05:06.284 INFO:teuthology.orchestra.run.smithi204.stdout: python3-wcwidth noarch 0.2.5-8.el9 appstream 44 k 2024-04-23T10:05:06.284 INFO:teuthology.orchestra.run.smithi204.stdout: python3-webob noarch 1.8.7-6.el9 epel 230 k 2024-04-23T10:05:06.285 INFO:teuthology.orchestra.run.smithi204.stdout: python3-websocket-client noarch 1.2.3-2.el9 epel 90 k 2024-04-23T10:05:06.285 INFO:teuthology.orchestra.run.smithi204.stdout: python3-werkzeug noarch 2.0.3-3.el9.1 epel 427 k 2024-04-23T10:05:06.285 INFO:teuthology.orchestra.run.smithi204.stdout: python3-zc-lockfile noarch 2.0-10.el9 epel 20 k 2024-04-23T10:05:06.285 INFO:teuthology.orchestra.run.smithi204.stdout: qatlib x86_64 24.02.0-1.el9 appstream 221 k 2024-04-23T10:05:06.285 INFO:teuthology.orchestra.run.smithi204.stdout: qatzip-libs x86_64 1.2.0-1.el9 appstream 47 k 2024-04-23T10:05:06.285 INFO:teuthology.orchestra.run.smithi204.stdout: re2 x86_64 1:20211101-3.el9 epel 192 k 2024-04-23T10:05:06.285 INFO:teuthology.orchestra.run.smithi204.stdout: socat x86_64 1.7.4.1-5.el9 appstream 305 k 2024-04-23T10:05:06.285 INFO:teuthology.orchestra.run.smithi204.stdout: thrift x86_64 0.15.0-2.el9 epel 1.6 M 2024-04-23T10:05:06.285 INFO:teuthology.orchestra.run.smithi204.stdout: uuid x86_64 1.6.2-55.el9 appstream 58 k 2024-04-23T10:05:06.285 INFO:teuthology.orchestra.run.smithi204.stdout: xmlstarlet x86_64 1.6.1-20.el9 appstream 64 k 2024-04-23T10:05:06.285 INFO:teuthology.orchestra.run.smithi204.stdout: zlib-devel x86_64 1.2.11-41.el9 appstream 45 k 2024-04-23T10:05:06.285 INFO:teuthology.orchestra.run.smithi204.stdout:Installing weak dependencies: 2024-04-23T10:05:06.285 INFO:teuthology.orchestra.run.smithi204.stdout: qatlib-service x86_64 24.02.0-1.el9 appstream 36 k 2024-04-23T10:05:06.285 INFO:teuthology.orchestra.run.smithi204.stdout: 2024-04-23T10:05:06.286 INFO:teuthology.orchestra.run.smithi204.stdout:Transaction Summary 2024-04-23T10:05:06.286 INFO:teuthology.orchestra.run.smithi204.stdout:======================================================================================= 2024-04-23T10:05:06.286 INFO:teuthology.orchestra.run.smithi204.stdout:Install 148 Packages 2024-04-23T10:05:06.286 INFO:teuthology.orchestra.run.smithi204.stdout:Upgrade 4 Packages 2024-04-23T10:05:06.286 INFO:teuthology.orchestra.run.smithi204.stdout: 2024-04-23T10:05:06.286 INFO:teuthology.orchestra.run.smithi204.stdout:Total download size: 222 M 2024-04-23T10:05:06.287 INFO:teuthology.orchestra.run.smithi204.stdout:Downloading Packages: 2024-04-23T10:05:07.147 INFO:teuthology.orchestra.run.smithi204.stdout:(1/152): ceph-19.0.0-3156.gd69f13d8.el7.x86_64. 18 kB/s | 6.6 kB 00:00 2024-04-23T10:05:08.059 INFO:teuthology.orchestra.run.smithi114.stdout:(2/152): ceph-fuse-19.0.0-3156.gd69f13d8.el7.x8 203 kB/s | 890 kB 00:04 2024-04-23T10:05:08.651 INFO:teuthology.orchestra.run.smithi114.stdout:(3/152): ceph-immutable-object-cache-19.0.0-315 244 kB/s | 145 kB 00:00 2024-04-23T10:05:10.730 INFO:teuthology.orchestra.run.smithi204.stdout:(2/152): ceph-fuse-19.0.0-3156.gd69f13d8.el7.x8 248 kB/s | 890 kB 00:03 2024-04-23T10:05:11.222 INFO:teuthology.orchestra.run.smithi204.stdout:(3/152): ceph-immutable-object-cache-19.0.0-315 294 kB/s | 145 kB 00:00 2024-04-23T10:05:11.919 INFO:teuthology.orchestra.run.smithi119.stdout:CentOS Stream 9 - AppStream 861 kB/s | 19 MB 00:22 2024-04-23T10:05:16.320 INFO:teuthology.orchestra.run.smithi119.stdout:CentOS Stream 9 - CRB 6.9 MB/s | 6.2 MB 00:00 2024-04-23T10:05:18.164 INFO:teuthology.orchestra.run.smithi119.stdout:CentOS Stream 9 - Extras packages 24 kB/s | 16 kB 00:00 2024-04-23T10:05:18.896 INFO:teuthology.orchestra.run.smithi204.stdout:(4/152): ceph-mds-19.0.0-3156.gd69f13d8.el7.x86 315 kB/s | 2.4 MB 00:07 2024-04-23T10:05:19.892 INFO:teuthology.orchestra.run.smithi119.stdout:Extra Packages for Enterprise Linux 13 MB/s | 21 MB 00:01 2024-04-23T10:05:20.301 INFO:teuthology.orchestra.run.smithi114.stdout:(4/152): ceph-mds-19.0.0-3156.gd69f13d8.el7.x86 208 kB/s | 2.4 MB 00:11 2024-04-23T10:05:22.580 INFO:teuthology.orchestra.run.smithi204.stdout:(5/152): ceph-mgr-19.0.0-3156.gd69f13d8.el7.x86 416 kB/s | 1.5 MB 00:03 2024-04-23T10:05:22.959 INFO:teuthology.orchestra.run.smithi114.stdout:(5/152): ceph-base-19.0.0-3156.gd69f13d8.el7.x8 276 kB/s | 5.4 MB 00:19 2024-04-23T10:05:24.964 INFO:teuthology.orchestra.run.smithi119.stdout:lab-extras 9.0 kB/s | 1.7 kB 00:00 2024-04-23T10:05:27.081 INFO:teuthology.orchestra.run.smithi119.stdout:Package librados2-2:16.2.4-5.el9.x86_64 is already installed. 2024-04-23T10:05:27.082 INFO:teuthology.orchestra.run.smithi119.stdout:Package librbd1-2:16.2.4-5.el9.x86_64 is already installed. 2024-04-23T10:05:27.084 INFO:teuthology.orchestra.run.smithi114.stdout:(6/152): ceph-mgr-19.0.0-3156.gd69f13d8.el7.x86 226 kB/s | 1.5 MB 00:06 2024-04-23T10:05:27.094 INFO:teuthology.orchestra.run.smithi119.stdout:Package NetworkManager-1:1.47.4-1.el9.x86_64 is already installed. 2024-04-23T10:05:27.094 INFO:teuthology.orchestra.run.smithi119.stdout:Package iproute-6.2.0-5.el9.x86_64 is already installed. 2024-04-23T10:05:27.094 INFO:teuthology.orchestra.run.smithi119.stdout:Package util-linux-2.37.4-18.el9.x86_64 is already installed. 2024-04-23T10:05:27.095 INFO:teuthology.orchestra.run.smithi119.stdout:Package libacl-devel-2.3.1-4.el9.x86_64 is already installed. 2024-04-23T10:05:27.096 INFO:teuthology.orchestra.run.smithi119.stdout:Package libattr-devel-2.5.1-3.el9.x86_64 is already installed. 2024-04-23T10:05:27.098 INFO:teuthology.orchestra.run.smithi119.stdout:Package xfsdump-3.1.12-4.el9.x86_64 is already installed. 2024-04-23T10:05:27.099 INFO:teuthology.orchestra.run.smithi119.stdout:Package xfsprogs-6.3.0-1.el9.x86_64 is already installed. 2024-04-23T10:05:27.159 INFO:teuthology.orchestra.run.smithi119.stdout:Dependencies resolved. 2024-04-23T10:05:27.172 INFO:teuthology.orchestra.run.smithi119.stdout:======================================================================================= 2024-04-23T10:05:27.173 INFO:teuthology.orchestra.run.smithi119.stdout: Package Arch Version Repository Size 2024-04-23T10:05:27.173 INFO:teuthology.orchestra.run.smithi119.stdout:======================================================================================= 2024-04-23T10:05:27.173 INFO:teuthology.orchestra.run.smithi119.stdout:Installing: 2024-04-23T10:05:27.173 INFO:teuthology.orchestra.run.smithi119.stdout: bison x86_64 3.7.4-5.el9 appstream 943 k 2024-04-23T10:05:27.173 INFO:teuthology.orchestra.run.smithi119.stdout: ceph x86_64 2:19.0.0-3156.gd69f13d8.el7 ceph 6.6 k 2024-04-23T10:05:27.173 INFO:teuthology.orchestra.run.smithi119.stdout: ceph-base x86_64 2:19.0.0-3156.gd69f13d8.el7 ceph 5.4 M 2024-04-23T10:05:27.173 INFO:teuthology.orchestra.run.smithi119.stdout: ceph-fuse x86_64 2:19.0.0-3156.gd69f13d8.el7 ceph 890 k 2024-04-23T10:05:27.173 INFO:teuthology.orchestra.run.smithi119.stdout: ceph-immutable-object-cache x86_64 2:19.0.0-3156.gd69f13d8.el7 ceph 145 k 2024-04-23T10:05:27.173 INFO:teuthology.orchestra.run.smithi119.stdout: ceph-mgr x86_64 2:19.0.0-3156.gd69f13d8.el7 ceph 1.5 M 2024-04-23T10:05:27.173 INFO:teuthology.orchestra.run.smithi119.stdout: ceph-mgr-cephadm noarch 2:19.0.0-3156.gd69f13d8.el7 ceph-noarch 142 k 2024-04-23T10:05:27.173 INFO:teuthology.orchestra.run.smithi119.stdout: ceph-mgr-dashboard noarch 2:19.0.0-3156.gd69f13d8.el7 ceph-noarch 3.5 M 2024-04-23T10:05:27.173 INFO:teuthology.orchestra.run.smithi119.stdout: ceph-mgr-diskprediction-local noarch 2:19.0.0-3156.gd69f13d8.el7 ceph-noarch 7.4 M 2024-04-23T10:05:27.173 INFO:teuthology.orchestra.run.smithi119.stdout: ceph-mgr-rook noarch 2:19.0.0-3156.gd69f13d8.el7 ceph-noarch 49 k 2024-04-23T10:05:27.174 INFO:teuthology.orchestra.run.smithi119.stdout: ceph-radosgw x86_64 2:19.0.0-3156.gd69f13d8.el7 ceph 11 M 2024-04-23T10:05:27.174 INFO:teuthology.orchestra.run.smithi119.stdout: ceph-test x86_64 2:19.0.0-3156.gd69f13d8.el7 ceph 49 M 2024-04-23T10:05:27.174 INFO:teuthology.orchestra.run.smithi119.stdout: ceph-volume noarch 2:19.0.0-3156.gd69f13d8.el7 ceph-noarch 273 k 2024-04-23T10:05:27.174 INFO:teuthology.orchestra.run.smithi119.stdout: cephadm noarch 2:19.0.0-3156.gd69f13d8.el7 ceph-noarch 762 k 2024-04-23T10:05:27.174 INFO:teuthology.orchestra.run.smithi119.stdout: cephfs-mirror x86_64 2:19.0.0-3156.gd69f13d8.el7 ceph 225 k 2024-04-23T10:05:27.174 INFO:teuthology.orchestra.run.smithi119.stdout: cephfs-shell x86_64 2:19.0.0-3156.gd69f13d8.el7 ceph 27 k 2024-04-23T10:05:27.174 INFO:teuthology.orchestra.run.smithi119.stdout: cephfs-top noarch 2:19.0.0-3156.gd69f13d8.el7 ceph-noarch 18 k 2024-04-23T10:05:27.174 INFO:teuthology.orchestra.run.smithi119.stdout: elfutils-libelf-devel x86_64 0.190-2.el9 appstream 21 k 2024-04-23T10:05:27.174 INFO:teuthology.orchestra.run.smithi119.stdout: flex x86_64 2.6.4-9.el9 appstream 313 k 2024-04-23T10:05:27.174 INFO:teuthology.orchestra.run.smithi119.stdout: libaio-devel x86_64 0.3.111-13.el9 appstream 11 k 2024-04-23T10:05:27.174 INFO:teuthology.orchestra.run.smithi119.stdout: libcephfs-devel x86_64 2:19.0.0-3156.gd69f13d8.el7 ceph 32 k 2024-04-23T10:05:27.174 INFO:teuthology.orchestra.run.smithi119.stdout: libcephfs2 x86_64 2:19.0.0-3156.gd69f13d8.el7 ceph 747 k 2024-04-23T10:05:27.174 INFO:teuthology.orchestra.run.smithi119.stdout: librados-devel x86_64 2:19.0.0-3156.gd69f13d8.el7 ceph 126 k 2024-04-23T10:05:27.175 INFO:teuthology.orchestra.run.smithi119.stdout: libtool x86_64 2.4.6-45.el9 appstream 582 k 2024-04-23T10:05:27.175 INFO:teuthology.orchestra.run.smithi119.stdout: libuuid-devel x86_64 2.37.4-18.el9 appstream 25 k 2024-04-23T10:05:27.175 INFO:teuthology.orchestra.run.smithi119.stdout: openssl-devel x86_64 1:3.2.1-1.el9 appstream 4.4 M 2024-04-23T10:05:27.175 INFO:teuthology.orchestra.run.smithi119.stdout: postgresql x86_64 13.14-1.el9 appstream 1.6 M 2024-04-23T10:05:27.175 INFO:teuthology.orchestra.run.smithi119.stdout: postgresql-contrib x86_64 13.14-1.el9 appstream 911 k 2024-04-23T10:05:27.175 INFO:teuthology.orchestra.run.smithi119.stdout: postgresql-server x86_64 13.14-1.el9 appstream 5.8 M 2024-04-23T10:05:27.175 INFO:teuthology.orchestra.run.smithi119.stdout: python3-cephfs x86_64 2:19.0.0-3156.gd69f13d8.el7 ceph 161 k 2024-04-23T10:05:27.175 INFO:teuthology.orchestra.run.smithi119.stdout: python3-rados x86_64 2:19.0.0-3156.gd69f13d8.el7 ceph 320 k 2024-04-23T10:05:27.175 INFO:teuthology.orchestra.run.smithi119.stdout: python3-rbd x86_64 2:19.0.0-3156.gd69f13d8.el7 ceph 298 k 2024-04-23T10:05:27.175 INFO:teuthology.orchestra.run.smithi119.stdout: python3-rgw x86_64 2:19.0.0-3156.gd69f13d8.el7 ceph 99 k 2024-04-23T10:05:27.175 INFO:teuthology.orchestra.run.smithi119.stdout: rbd-fuse x86_64 2:19.0.0-3156.gd69f13d8.el7 ceph 85 k 2024-04-23T10:05:27.175 INFO:teuthology.orchestra.run.smithi119.stdout: rbd-mirror x86_64 2:19.0.0-3156.gd69f13d8.el7 ceph 3.1 M 2024-04-23T10:05:27.175 INFO:teuthology.orchestra.run.smithi119.stdout: rbd-nbd x86_64 2:19.0.0-3156.gd69f13d8.el7 ceph 170 k 2024-04-23T10:05:27.176 INFO:teuthology.orchestra.run.smithi119.stdout: xfsprogs-devel x86_64 6.3.0-1.el9 appstream 94 k 2024-04-23T10:05:27.176 INFO:teuthology.orchestra.run.smithi119.stdout:Upgrading: 2024-04-23T10:05:27.176 INFO:teuthology.orchestra.run.smithi119.stdout: librados2 x86_64 2:19.0.0-3156.gd69f13d8.el7 ceph 3.4 M 2024-04-23T10:05:27.176 INFO:teuthology.orchestra.run.smithi119.stdout: librbd1 x86_64 2:19.0.0-3156.gd69f13d8.el7 ceph 3.1 M 2024-04-23T10:05:27.176 INFO:teuthology.orchestra.run.smithi119.stdout: openssl x86_64 1:3.2.1-1.el9 baseos 1.3 M 2024-04-23T10:05:27.176 INFO:teuthology.orchestra.run.smithi119.stdout: openssl-libs x86_64 1:3.2.1-1.el9 baseos 2.4 M 2024-04-23T10:05:27.176 INFO:teuthology.orchestra.run.smithi119.stdout:Installing dependencies: 2024-04-23T10:05:27.176 INFO:teuthology.orchestra.run.smithi119.stdout: abseil-cpp x86_64 20211102.0-3.el9 epel 548 k 2024-04-23T10:05:27.176 INFO:teuthology.orchestra.run.smithi119.stdout: automake noarch 1.16.2-8.el9 appstream 689 k 2024-04-23T10:05:27.176 INFO:teuthology.orchestra.run.smithi119.stdout: boost-program-options x86_64 1.75.0-8.el9 appstream 107 k 2024-04-23T10:05:27.176 INFO:teuthology.orchestra.run.smithi119.stdout: ceph-common x86_64 2:19.0.0-3156.gd69f13d8.el7 ceph 21 M 2024-04-23T10:05:27.176 INFO:teuthology.orchestra.run.smithi119.stdout: ceph-grafana-dashboards noarch 2:19.0.0-3156.gd69f13d8.el7 ceph-noarch 34 k 2024-04-23T10:05:27.176 INFO:teuthology.orchestra.run.smithi119.stdout: ceph-mds x86_64 2:19.0.0-3156.gd69f13d8.el7 ceph 2.4 M 2024-04-23T10:05:27.176 INFO:teuthology.orchestra.run.smithi119.stdout: ceph-mgr-modules-core noarch 2:19.0.0-3156.gd69f13d8.el7 ceph-noarch 249 k 2024-04-23T10:05:27.177 INFO:teuthology.orchestra.run.smithi119.stdout: ceph-mon x86_64 2:19.0.0-3156.gd69f13d8.el7 ceph 4.7 M 2024-04-23T10:05:27.177 INFO:teuthology.orchestra.run.smithi119.stdout: ceph-osd x86_64 2:19.0.0-3156.gd69f13d8.el7 ceph 17 M 2024-04-23T10:05:27.177 INFO:teuthology.orchestra.run.smithi119.stdout: ceph-prometheus-alerts noarch 2:19.0.0-3156.gd69f13d8.el7 ceph-noarch 16 k 2024-04-23T10:05:27.177 INFO:teuthology.orchestra.run.smithi119.stdout: ceph-selinux x86_64 2:19.0.0-3156.gd69f13d8.el7 ceph 25 k 2024-04-23T10:05:27.177 INFO:teuthology.orchestra.run.smithi119.stdout: flexiblas x86_64 3.0.4-8.el9 appstream 33 k 2024-04-23T10:05:27.177 INFO:teuthology.orchestra.run.smithi119.stdout: flexiblas-netlib x86_64 3.0.4-8.el9 appstream 3.0 M 2024-04-23T10:05:27.177 INFO:teuthology.orchestra.run.smithi119.stdout: flexiblas-openblas-openmp x86_64 3.0.4-8.el9 appstream 18 k 2024-04-23T10:05:27.177 INFO:teuthology.orchestra.run.smithi119.stdout: fuse x86_64 2.9.9-15.el9 baseos 80 k 2024-04-23T10:05:27.177 INFO:teuthology.orchestra.run.smithi119.stdout: gperftools-libs x86_64 2.9.1-2.el9 epel 309 k 2024-04-23T10:05:27.177 INFO:teuthology.orchestra.run.smithi119.stdout: grpc-data noarch 1.46.7-10.el9 epel 19 k 2024-04-23T10:05:27.177 INFO:teuthology.orchestra.run.smithi119.stdout: libarrow x86_64 9.0.0-10.el9 epel 4.4 M 2024-04-23T10:05:27.177 INFO:teuthology.orchestra.run.smithi119.stdout: libarrow-doc noarch 9.0.0-10.el9 epel 26 k 2024-04-23T10:05:27.177 INFO:teuthology.orchestra.run.smithi119.stdout: libcephsqlite x86_64 2:19.0.0-3156.gd69f13d8.el7 ceph 163 k 2024-04-23T10:05:27.177 INFO:teuthology.orchestra.run.smithi119.stdout: libgfortran x86_64 11.4.1-3.el9 baseos 803 k 2024-04-23T10:05:27.177 INFO:teuthology.orchestra.run.smithi119.stdout: liboath x86_64 2.6.7-2.el9 epel 49 k 2024-04-23T10:05:27.178 INFO:teuthology.orchestra.run.smithi119.stdout: libpmemobj x86_64 1.12.1-1.el9 appstream 160 k 2024-04-23T10:05:27.178 INFO:teuthology.orchestra.run.smithi119.stdout: libquadmath x86_64 11.4.1-3.el9 baseos 195 k 2024-04-23T10:05:27.178 INFO:teuthology.orchestra.run.smithi119.stdout: librabbitmq x86_64 0.11.0-7.el9 appstream 45 k 2024-04-23T10:05:27.178 INFO:teuthology.orchestra.run.smithi119.stdout: libradosstriper1 x86_64 2:19.0.0-3156.gd69f13d8.el7 ceph 500 k 2024-04-23T10:05:27.178 INFO:teuthology.orchestra.run.smithi119.stdout: librdkafka x86_64 1.6.1-102.el9 appstream 662 k 2024-04-23T10:05:27.178 INFO:teuthology.orchestra.run.smithi119.stdout: librgw2 x86_64 2:19.0.0-3156.gd69f13d8.el7 ceph 5.4 M 2024-04-23T10:05:27.178 INFO:teuthology.orchestra.run.smithi119.stdout: libunwind x86_64 1.6.2-1.el9 epel 67 k 2024-04-23T10:05:27.178 INFO:teuthology.orchestra.run.smithi119.stdout: libzstd-devel x86_64 1.5.1-2.el9 appstream 47 k 2024-04-23T10:05:27.178 INFO:teuthology.orchestra.run.smithi119.stdout: lttng-ust x86_64 2.12.0-6.el9 appstream 292 k 2024-04-23T10:05:27.178 INFO:teuthology.orchestra.run.smithi119.stdout: lua x86_64 5.4.4-4.el9 appstream 188 k 2024-04-23T10:05:27.178 INFO:teuthology.orchestra.run.smithi119.stdout: lua-devel x86_64 5.4.4-4.el9 crb 22 k 2024-04-23T10:05:27.178 INFO:teuthology.orchestra.run.smithi119.stdout: luarocks noarch 3.9.2-1.el9 epel 151 k 2024-04-23T10:05:27.178 INFO:teuthology.orchestra.run.smithi119.stdout: openblas x86_64 0.3.21-2.el9 appstream 35 k 2024-04-23T10:05:27.179 INFO:teuthology.orchestra.run.smithi119.stdout: openblas-openmp x86_64 0.3.21-2.el9 appstream 4.7 M 2024-04-23T10:05:27.179 INFO:teuthology.orchestra.run.smithi119.stdout: parquet-libs x86_64 9.0.0-10.el9 epel 838 k 2024-04-23T10:05:27.179 INFO:teuthology.orchestra.run.smithi119.stdout: postgresql-private-libs x86_64 13.14-1.el9 appstream 133 k 2024-04-23T10:05:27.179 INFO:teuthology.orchestra.run.smithi119.stdout: protobuf x86_64 3.14.0-13.el9 appstream 1.0 M 2024-04-23T10:05:27.179 INFO:teuthology.orchestra.run.smithi119.stdout: protobuf-compiler x86_64 3.14.0-13.el9 crb 863 k 2024-04-23T10:05:27.179 INFO:teuthology.orchestra.run.smithi119.stdout: python3-asyncssh noarch 2.13.2-5.el9 epel 548 k 2024-04-23T10:05:27.179 INFO:teuthology.orchestra.run.smithi119.stdout: python3-attrs noarch 20.3.0-7.el9 appstream 87 k 2024-04-23T10:05:27.179 INFO:teuthology.orchestra.run.smithi119.stdout: python3-babel noarch 2.9.1-2.el9 appstream 6.0 M 2024-04-23T10:05:27.179 INFO:teuthology.orchestra.run.smithi119.stdout: python3-bcrypt x86_64 3.2.2-1.el9 epel 43 k 2024-04-23T10:05:27.179 INFO:teuthology.orchestra.run.smithi119.stdout: python3-cachetools noarch 4.2.4-1.el9 epel 32 k 2024-04-23T10:05:27.179 INFO:teuthology.orchestra.run.smithi119.stdout: python3-ceph-argparse x86_64 2:19.0.0-3156.gd69f13d8.el7 ceph 45 k 2024-04-23T10:05:27.179 INFO:teuthology.orchestra.run.smithi119.stdout: python3-ceph-common x86_64 2:19.0.0-3156.gd69f13d8.el7 ceph 135 k 2024-04-23T10:05:27.179 INFO:teuthology.orchestra.run.smithi119.stdout: python3-certifi noarch 2021.10.8-2.el9 epel 15 k 2024-04-23T10:05:27.179 INFO:teuthology.orchestra.run.smithi119.stdout: python3-cffi x86_64 1.14.5-5.el9 baseos 253 k 2024-04-23T10:05:27.180 INFO:teuthology.orchestra.run.smithi119.stdout: python3-cheroot noarch 8.6.0-4.el9 epel 172 k 2024-04-23T10:05:27.180 INFO:teuthology.orchestra.run.smithi119.stdout: python3-cherrypy noarch 18.6.1-2.el9 epel 358 k 2024-04-23T10:05:27.180 INFO:teuthology.orchestra.run.smithi119.stdout: python3-cmd2 noarch 2.4.2-2.el9 epel 303 k 2024-04-23T10:05:27.180 INFO:teuthology.orchestra.run.smithi119.stdout: python3-colorama noarch 0.4.4-12.el9 epel 35 k 2024-04-23T10:05:27.180 INFO:teuthology.orchestra.run.smithi119.stdout: python3-cryptography x86_64 36.0.1-4.el9 baseos 1.2 M 2024-04-23T10:05:27.180 INFO:teuthology.orchestra.run.smithi119.stdout: python3-devel x86_64 3.9.18-3.el9 appstream 244 k 2024-04-23T10:05:27.180 INFO:teuthology.orchestra.run.smithi119.stdout: python3-google-auth noarch 1:2.29.0-1.el9 epel 206 k 2024-04-23T10:05:27.180 INFO:teuthology.orchestra.run.smithi119.stdout: python3-grpcio x86_64 1.46.7-10.el9 epel 2.0 M 2024-04-23T10:05:27.180 INFO:teuthology.orchestra.run.smithi119.stdout: python3-grpcio-tools x86_64 1.46.7-10.el9 epel 144 k 2024-04-23T10:05:27.180 INFO:teuthology.orchestra.run.smithi119.stdout: python3-jaraco noarch 8.2.1-3.el9 epel 11 k 2024-04-23T10:05:27.180 INFO:teuthology.orchestra.run.smithi119.stdout: python3-jaraco-classes noarch 3.2.1-5.el9 epel 18 k 2024-04-23T10:05:27.180 INFO:teuthology.orchestra.run.smithi119.stdout: python3-jaraco-collections noarch 3.0.0-8.el9 epel 23 k 2024-04-23T10:05:27.180 INFO:teuthology.orchestra.run.smithi119.stdout: python3-jaraco-functools noarch 3.5.0-2.el9 epel 19 k 2024-04-23T10:05:27.181 INFO:teuthology.orchestra.run.smithi119.stdout: python3-jaraco-text noarch 3.2.0-6.el9 epel 20 k 2024-04-23T10:05:27.181 INFO:teuthology.orchestra.run.smithi119.stdout: python3-jinja2 noarch 2.11.3-5.el9 appstream 248 k 2024-04-23T10:05:27.181 INFO:teuthology.orchestra.run.smithi119.stdout: python3-jsonpatch noarch 1.21-16.el9 appstream 26 k 2024-04-23T10:05:27.181 INFO:teuthology.orchestra.run.smithi119.stdout: python3-jsonpointer noarch 2.0-4.el9 appstream 19 k 2024-04-23T10:05:27.181 INFO:teuthology.orchestra.run.smithi119.stdout: python3-kubernetes noarch 1:26.1.0-2.el9 epel 1.0 M 2024-04-23T10:05:27.181 INFO:teuthology.orchestra.run.smithi119.stdout: python3-logutils noarch 0.3.5-21.el9 epel 46 k 2024-04-23T10:05:27.181 INFO:teuthology.orchestra.run.smithi119.stdout: python3-mako noarch 1.1.4-6.el9 appstream 172 k 2024-04-23T10:05:27.181 INFO:teuthology.orchestra.run.smithi119.stdout: python3-markupsafe x86_64 1.1.1-12.el9 appstream 35 k 2024-04-23T10:05:27.181 INFO:teuthology.orchestra.run.smithi119.stdout: python3-more-itertools noarch 8.12.0-2.el9 epel 79 k 2024-04-23T10:05:27.181 INFO:teuthology.orchestra.run.smithi119.stdout: python3-natsort noarch 7.1.1-5.el9 epel 58 k 2024-04-23T10:05:27.181 INFO:teuthology.orchestra.run.smithi119.stdout: python3-numpy x86_64 1:1.20.1-5.el9 appstream 5.1 M 2024-04-23T10:05:27.181 INFO:teuthology.orchestra.run.smithi119.stdout: python3-numpy-f2py x86_64 1:1.20.1-5.el9 appstream 250 k 2024-04-23T10:05:27.181 INFO:teuthology.orchestra.run.smithi119.stdout: python3-oauthlib noarch 3.1.1-5.el9 appstream 222 k 2024-04-23T10:05:27.182 INFO:teuthology.orchestra.run.smithi119.stdout: python3-packaging noarch 20.9-5.el9 appstream 77 k 2024-04-23T10:05:27.182 INFO:teuthology.orchestra.run.smithi119.stdout: python3-pecan noarch 1.4.2-3.el9 epel 272 k 2024-04-23T10:05:27.182 INFO:teuthology.orchestra.run.smithi119.stdout: python3-ply noarch 3.11-14.el9 baseos 106 k 2024-04-23T10:05:27.182 INFO:teuthology.orchestra.run.smithi119.stdout: python3-portend noarch 3.1.0-2.el9 epel 16 k 2024-04-23T10:05:27.182 INFO:teuthology.orchestra.run.smithi119.stdout: python3-prettytable noarch 0.7.2-27.el9 appstream 42 k 2024-04-23T10:05:27.182 INFO:teuthology.orchestra.run.smithi119.stdout: python3-protobuf noarch 3.14.0-13.el9 appstream 269 k 2024-04-23T10:05:27.182 INFO:teuthology.orchestra.run.smithi119.stdout: python3-pyOpenSSL noarch 21.0.0-1.el9 epel 90 k 2024-04-23T10:05:27.182 INFO:teuthology.orchestra.run.smithi119.stdout: python3-pyasn1 noarch 0.4.8-6.el9 appstream 159 k 2024-04-23T10:05:27.182 INFO:teuthology.orchestra.run.smithi119.stdout: python3-pyasn1-modules noarch 0.4.8-6.el9 appstream 279 k 2024-04-23T10:05:27.182 INFO:teuthology.orchestra.run.smithi119.stdout: python3-pycparser noarch 2.20-6.el9 baseos 135 k 2024-04-23T10:05:27.182 INFO:teuthology.orchestra.run.smithi119.stdout: python3-pyperclip noarch 1.8.2-2.el9 epel 28 k 2024-04-23T10:05:27.182 INFO:teuthology.orchestra.run.smithi119.stdout: python3-pytz noarch 2021.1-5.el9 appstream 51 k 2024-04-23T10:05:27.182 INFO:teuthology.orchestra.run.smithi119.stdout: python3-repoze-lru noarch 0.7-16.el9 epel 31 k 2024-04-23T10:05:27.183 INFO:teuthology.orchestra.run.smithi119.stdout: python3-requests noarch 2.25.1-8.el9 baseos 125 k 2024-04-23T10:05:27.183 INFO:teuthology.orchestra.run.smithi119.stdout: python3-requests-oauthlib noarch 1.3.0-12.el9 appstream 54 k 2024-04-23T10:05:27.183 INFO:teuthology.orchestra.run.smithi119.stdout: python3-routes noarch 2.5.1-5.el9 epel 188 k 2024-04-23T10:05:27.183 INFO:teuthology.orchestra.run.smithi119.stdout: python3-rsa noarch 4.9-2.el9 epel 59 k 2024-04-23T10:05:27.183 INFO:teuthology.orchestra.run.smithi119.stdout: python3-scipy x86_64 1.6.2-8.el9 appstream 16 M 2024-04-23T10:05:27.183 INFO:teuthology.orchestra.run.smithi119.stdout: python3-tempora noarch 5.0.0-2.el9 epel 36 k 2024-04-23T10:05:27.183 INFO:teuthology.orchestra.run.smithi119.stdout: python3-toml noarch 0.10.2-6.el9 appstream 42 k 2024-04-23T10:05:27.183 INFO:teuthology.orchestra.run.smithi119.stdout: python3-typing-extensions noarch 4.4.0-2.el9 epel 51 k 2024-04-23T10:05:27.183 INFO:teuthology.orchestra.run.smithi119.stdout: python3-urllib3 noarch 1.26.5-5.el9 baseos 215 k 2024-04-23T10:05:27.183 INFO:teuthology.orchestra.run.smithi119.stdout: python3-wcwidth noarch 0.2.5-8.el9 appstream 44 k 2024-04-23T10:05:27.183 INFO:teuthology.orchestra.run.smithi119.stdout: python3-webob noarch 1.8.7-6.el9 epel 230 k 2024-04-23T10:05:27.183 INFO:teuthology.orchestra.run.smithi119.stdout: python3-websocket-client noarch 1.2.3-2.el9 epel 90 k 2024-04-23T10:05:27.183 INFO:teuthology.orchestra.run.smithi119.stdout: python3-werkzeug noarch 2.0.3-3.el9.1 epel 427 k 2024-04-23T10:05:27.183 INFO:teuthology.orchestra.run.smithi119.stdout: python3-zc-lockfile noarch 2.0-10.el9 epel 20 k 2024-04-23T10:05:27.183 INFO:teuthology.orchestra.run.smithi119.stdout: qatlib x86_64 24.02.0-1.el9 appstream 221 k 2024-04-23T10:05:27.183 INFO:teuthology.orchestra.run.smithi119.stdout: qatzip-libs x86_64 1.2.0-1.el9 appstream 47 k 2024-04-23T10:05:27.184 INFO:teuthology.orchestra.run.smithi119.stdout: re2 x86_64 1:20211101-3.el9 epel 192 k 2024-04-23T10:05:27.184 INFO:teuthology.orchestra.run.smithi119.stdout: socat x86_64 1.7.4.1-5.el9 appstream 305 k 2024-04-23T10:05:27.184 INFO:teuthology.orchestra.run.smithi119.stdout: thrift x86_64 0.15.0-2.el9 epel 1.6 M 2024-04-23T10:05:27.184 INFO:teuthology.orchestra.run.smithi119.stdout: uuid x86_64 1.6.2-55.el9 appstream 58 k 2024-04-23T10:05:27.184 INFO:teuthology.orchestra.run.smithi119.stdout: xmlstarlet x86_64 1.6.1-20.el9 appstream 64 k 2024-04-23T10:05:27.184 INFO:teuthology.orchestra.run.smithi119.stdout: zlib-devel x86_64 1.2.11-41.el9 appstream 45 k 2024-04-23T10:05:27.184 INFO:teuthology.orchestra.run.smithi119.stdout:Installing weak dependencies: 2024-04-23T10:05:27.184 INFO:teuthology.orchestra.run.smithi119.stdout: qatlib-service x86_64 24.02.0-1.el9 appstream 36 k 2024-04-23T10:05:27.184 INFO:teuthology.orchestra.run.smithi119.stdout: 2024-04-23T10:05:27.184 INFO:teuthology.orchestra.run.smithi119.stdout:Transaction Summary 2024-04-23T10:05:27.184 INFO:teuthology.orchestra.run.smithi119.stdout:======================================================================================= 2024-04-23T10:05:27.184 INFO:teuthology.orchestra.run.smithi119.stdout:Install 148 Packages 2024-04-23T10:05:27.184 INFO:teuthology.orchestra.run.smithi119.stdout:Upgrade 4 Packages 2024-04-23T10:05:27.184 INFO:teuthology.orchestra.run.smithi119.stdout: 2024-04-23T10:05:27.185 INFO:teuthology.orchestra.run.smithi119.stdout:Total download size: 222 M 2024-04-23T10:05:27.185 INFO:teuthology.orchestra.run.smithi119.stdout:Downloading Packages: 2024-04-23T10:05:28.282 INFO:teuthology.orchestra.run.smithi119.stdout:(1/152): ceph-19.0.0-3156.gd69f13d8.el7.x86_64. 18 kB/s | 6.6 kB 00:00 2024-04-23T10:05:31.513 INFO:teuthology.orchestra.run.smithi204.stdout:(6/152): ceph-base-19.0.0-3156.gd69f13d8.el7.x8 222 kB/s | 5.4 MB 00:24 2024-04-23T10:05:32.965 INFO:teuthology.orchestra.run.smithi119.stdout:(2/152): ceph-fuse-19.0.0-3156.gd69f13d8.el7.x8 190 kB/s | 890 kB 00:04 2024-04-23T10:05:33.782 INFO:teuthology.orchestra.run.smithi119.stdout:(3/152): ceph-immutable-object-cache-19.0.0-315 177 kB/s | 145 kB 00:00 2024-04-23T10:05:40.629 INFO:teuthology.orchestra.run.smithi204.stdout:(7/152): ceph-mon-19.0.0-3156.gd69f13d8.el7.x86 267 kB/s | 4.7 MB 00:18 2024-04-23T10:05:43.098 INFO:teuthology.orchestra.run.smithi119.stdout:(4/152): ceph-mds-19.0.0-3156.gd69f13d8.el7.x86 260 kB/s | 2.4 MB 00:09 2024-04-23T10:05:49.056 INFO:teuthology.orchestra.run.smithi119.stdout:(5/152): ceph-base-19.0.0-3156.gd69f13d8.el7.x8 260 kB/s | 5.4 MB 00:21 2024-04-23T10:05:50.298 INFO:teuthology.orchestra.run.smithi119.stdout:(6/152): ceph-mgr-19.0.0-3156.gd69f13d8.el7.x86 213 kB/s | 1.5 MB 00:07 2024-04-23T10:05:51.107 INFO:teuthology.orchestra.run.smithi114.stdout:(7/152): ceph-mon-19.0.0-3156.gd69f13d8.el7.x86 171 kB/s | 4.7 MB 00:28 2024-04-23T10:06:14.037 INFO:teuthology.orchestra.run.smithi119.stdout:(7/152): ceph-mon-19.0.0-3156.gd69f13d8.el7.x86 193 kB/s | 4.7 MB 00:24 2024-04-23T10:06:31.573 INFO:teuthology.orchestra.run.smithi204.stdout:(8/152): ceph-radosgw-19.0.0-3156.gd69f13d8.el7 213 kB/s | 11 MB 00:50 2024-04-23T10:06:31.757 INFO:teuthology.orchestra.run.smithi204.stdout:(9/152): ceph-selinux-19.0.0-3156.gd69f13d8.el7 137 kB/s | 25 kB 00:00 2024-04-23T10:06:55.591 INFO:teuthology.orchestra.run.smithi114.stdout:(8/152): ceph-radosgw-19.0.0-3156.gd69f13d8.el7 168 kB/s | 11 MB 01:04 2024-04-23T10:06:55.783 INFO:teuthology.orchestra.run.smithi114.stdout:(9/152): ceph-selinux-19.0.0-3156.gd69f13d8.el7 131 kB/s | 25 kB 00:00 2024-04-23T10:06:56.256 INFO:teuthology.orchestra.run.smithi119.stdout:(8/152): ceph-common-19.0.0-3156.gd69f13d8.el7. 245 kB/s | 21 MB 01:28 2024-04-23T10:06:56.640 INFO:teuthology.orchestra.run.smithi119.stdout:(9/152): ceph-selinux-19.0.0-3156.gd69f13d8.el7 66 kB/s | 25 kB 00:00 2024-04-23T10:06:57.891 INFO:teuthology.orchestra.run.smithi114.stdout:(10/152): ceph-common-19.0.0-3156.gd69f13d8.el7 188 kB/s | 21 MB 01:54 2024-04-23T10:06:59.433 INFO:teuthology.orchestra.run.smithi114.stdout:(11/152): cephfs-mirror-19.0.0-3156.gd69f13d8.e 146 kB/s | 225 kB 00:01 2024-04-23T10:06:59.850 INFO:teuthology.orchestra.run.smithi114.stdout:(12/152): cephfs-shell-19.0.0-3156.gd69f13d8.el 65 kB/s | 27 kB 00:00 2024-04-23T10:07:00.206 INFO:teuthology.orchestra.run.smithi119.stdout:(10/152): ceph-radosgw-19.0.0-3156.gd69f13d8.el 235 kB/s | 11 MB 00:46 2024-04-23T10:07:00.350 INFO:teuthology.orchestra.run.smithi114.stdout:(13/152): libcephfs-devel-19.0.0-3156.gd69f13d8 63 kB/s | 32 kB 00:00 2024-04-23T10:07:01.515 INFO:teuthology.orchestra.run.smithi119.stdout:(11/152): cephfs-mirror-19.0.0-3156.gd69f13d8.e 172 kB/s | 225 kB 00:01 2024-04-23T10:07:01.781 INFO:teuthology.orchestra.run.smithi119.stdout:(12/152): cephfs-shell-19.0.0-3156.gd69f13d8.el 101 kB/s | 27 kB 00:00 2024-04-23T10:07:02.123 INFO:teuthology.orchestra.run.smithi119.stdout:(13/152): libcephfs-devel-19.0.0-3156.gd69f13d8 93 kB/s | 32 kB 00:00 2024-04-23T10:07:05.948 INFO:teuthology.orchestra.run.smithi119.stdout:(14/152): libcephfs2-19.0.0-3156.gd69f13d8.el7. 195 kB/s | 747 kB 00:03 2024-04-23T10:07:06.457 INFO:teuthology.orchestra.run.smithi119.stdout:(15/152): ceph-osd-19.0.0-3156.gd69f13d8.el7.x8 226 kB/s | 17 MB 01:16 2024-04-23T10:07:07.125 INFO:teuthology.orchestra.run.smithi114.stdout:(14/152): libcephfs2-19.0.0-3156.gd69f13d8.el7. 110 kB/s | 747 kB 00:06 2024-04-23T10:07:07.223 INFO:teuthology.orchestra.run.smithi119.stdout:(16/152): librados-devel-19.0.0-3156.gd69f13d8. 165 kB/s | 126 kB 00:00 2024-04-23T10:07:07.390 INFO:teuthology.orchestra.run.smithi119.stdout:(17/152): libcephsqlite-19.0.0-3156.gd69f13d8.e 113 kB/s | 163 kB 00:01 2024-04-23T10:07:07.916 INFO:teuthology.orchestra.run.smithi114.stdout:(15/152): libcephsqlite-19.0.0-3156.gd69f13d8.e 206 kB/s | 163 kB 00:00 2024-04-23T10:07:08.608 INFO:teuthology.orchestra.run.smithi114.stdout:(16/152): librados-devel-19.0.0-3156.gd69f13d8. 183 kB/s | 126 kB 00:00 2024-04-23T10:07:09.274 INFO:teuthology.orchestra.run.smithi119.stdout:(18/152): libradosstriper1-19.0.0-3156.gd69f13d 244 kB/s | 500 kB 00:02 2024-04-23T10:07:09.449 INFO:teuthology.orchestra.run.smithi119.stdout:(19/152): python3-ceph-argparse-19.0.0-3156.gd6 258 kB/s | 45 kB 00:00 2024-04-23T10:07:09.916 INFO:teuthology.orchestra.run.smithi119.stdout:(20/152): python3-ceph-common-19.0.0-3156.gd69f 289 kB/s | 135 kB 00:00 2024-04-23T10:07:10.308 INFO:teuthology.orchestra.run.smithi114.stdout:(17/152): ceph-osd-19.0.0-3156.gd69f13d8.el7.x8 167 kB/s | 17 MB 01:43 2024-04-23T10:07:10.408 INFO:teuthology.orchestra.run.smithi119.stdout:(21/152): python3-cephfs-19.0.0-3156.gd69f13d8. 328 kB/s | 161 kB 00:00 2024-04-23T10:07:11.692 INFO:teuthology.orchestra.run.smithi114.stdout:(18/152): libradosstriper1-19.0.0-3156.gd69f13d 162 kB/s | 500 kB 00:03 2024-04-23T10:07:12.433 INFO:teuthology.orchestra.run.smithi119.stdout:(22/152): python3-rados-19.0.0-3156.gd69f13d8.e 158 kB/s | 320 kB 00:02 2024-04-23T10:07:12.475 INFO:teuthology.orchestra.run.smithi114.stdout:(19/152): python3-ceph-argparse-19.0.0-3156.gd6 58 kB/s | 45 kB 00:00 2024-04-23T10:07:13.859 INFO:teuthology.orchestra.run.smithi114.stdout:(20/152): python3-ceph-common-19.0.0-3156.gd69f 97 kB/s | 135 kB 00:01 2024-04-23T10:07:14.397 INFO:teuthology.orchestra.run.smithi204.stdout:(10/152): ceph-common-19.0.0-3156.gd69f13d8.el7 169 kB/s | 21 MB 02:07 2024-04-23T10:07:14.508 INFO:teuthology.orchestra.run.smithi119.stdout:(23/152): python3-rbd-19.0.0-3156.gd69f13d8.el7 143 kB/s | 298 kB 00:02 2024-04-23T10:07:15.333 INFO:teuthology.orchestra.run.smithi119.stdout:(24/152): python3-rgw-19.0.0-3156.gd69f13d8.el7 120 kB/s | 99 kB 00:00 2024-04-23T10:07:15.400 INFO:teuthology.orchestra.run.smithi114.stdout:(21/152): python3-cephfs-19.0.0-3156.gd69f13d8. 105 kB/s | 161 kB 00:01 2024-04-23T10:07:15.788 INFO:teuthology.orchestra.run.smithi204.stdout:(11/152): cephfs-mirror-19.0.0-3156.gd69f13d8.e 161 kB/s | 225 kB 00:01 2024-04-23T10:07:16.024 INFO:teuthology.orchestra.run.smithi119.stdout:(25/152): rbd-fuse-19.0.0-3156.gd69f13d8.el7.x8 123 kB/s | 85 kB 00:00 2024-04-23T10:07:16.197 INFO:teuthology.orchestra.run.smithi204.stdout:(12/152): cephfs-shell-19.0.0-3156.gd69f13d8.el 66 kB/s | 27 kB 00:00 2024-04-23T10:07:16.881 INFO:teuthology.orchestra.run.smithi204.stdout:(13/152): libcephfs-devel-19.0.0-3156.gd69f13d8 46 kB/s | 32 kB 00:00 2024-04-23T10:07:17.409 INFO:teuthology.orchestra.run.smithi114.stdout:(22/152): python3-rados-19.0.0-3156.gd69f13d8.e 160 kB/s | 320 kB 00:02 2024-04-23T10:07:19.475 INFO:teuthology.orchestra.run.smithi114.stdout:(23/152): python3-rbd-19.0.0-3156.gd69f13d8.el7 144 kB/s | 298 kB 00:02 2024-04-23T10:07:20.376 INFO:teuthology.orchestra.run.smithi114.stdout:(24/152): python3-rgw-19.0.0-3156.gd69f13d8.el7 110 kB/s | 99 kB 00:00 2024-04-23T10:07:20.976 INFO:teuthology.orchestra.run.smithi114.stdout:(25/152): rbd-fuse-19.0.0-3156.gd69f13d8.el7.x8 142 kB/s | 85 kB 00:00 2024-04-23T10:07:22.288 INFO:teuthology.orchestra.run.smithi204.stdout:(14/152): libcephfs2-19.0.0-3156.gd69f13d8.el7. 138 kB/s | 747 kB 00:05 2024-04-23T10:07:23.447 INFO:teuthology.orchestra.run.smithi204.stdout:(15/152): libcephsqlite-19.0.0-3156.gd69f13d8.e 141 kB/s | 163 kB 00:01 2024-04-23T10:07:24.131 INFO:teuthology.orchestra.run.smithi204.stdout:(16/152): librados-devel-19.0.0-3156.gd69f13d8. 185 kB/s | 126 kB 00:00 2024-04-23T10:07:26.531 INFO:teuthology.orchestra.run.smithi204.stdout:(17/152): ceph-osd-19.0.0-3156.gd69f13d8.el7.x8 150 kB/s | 17 MB 01:55 2024-04-23T10:07:26.831 INFO:teuthology.orchestra.run.smithi204.stdout:(18/152): libradosstriper1-19.0.0-3156.gd69f13d 185 kB/s | 500 kB 00:02 2024-04-23T10:07:27.090 INFO:teuthology.orchestra.run.smithi204.stdout:(19/152): python3-ceph-argparse-19.0.0-3156.gd6 174 kB/s | 45 kB 00:00 2024-04-23T10:07:27.723 INFO:teuthology.orchestra.run.smithi204.stdout:(20/152): python3-ceph-common-19.0.0-3156.gd69f 213 kB/s | 135 kB 00:00 2024-04-23T10:07:28.132 INFO:teuthology.orchestra.run.smithi119.stdout:(26/152): librgw2-19.0.0-3156.gd69f13d8.el7.x86 265 kB/s | 5.4 MB 00:20 2024-04-23T10:07:28.506 INFO:teuthology.orchestra.run.smithi204.stdout:(21/152): python3-cephfs-19.0.0-3156.gd69f13d8. 206 kB/s | 161 kB 00:00 2024-04-23T10:07:28.815 INFO:teuthology.orchestra.run.smithi119.stdout:(27/152): rbd-nbd-19.0.0-3156.gd69f13d8.el7.x86 249 kB/s | 170 kB 00:00 2024-04-23T10:07:29.007 INFO:teuthology.orchestra.run.smithi119.stdout:(28/152): ceph-grafana-dashboards-19.0.0-3156.g 178 kB/s | 34 kB 00:00 2024-04-23T10:07:29.458 INFO:teuthology.orchestra.run.smithi119.stdout:(29/152): rbd-mirror-19.0.0-3156.gd69f13d8.el7. 235 kB/s | 3.1 MB 00:13 2024-04-23T10:07:29.633 INFO:teuthology.orchestra.run.smithi119.stdout:(30/152): ceph-mgr-cephadm-19.0.0-3156.gd69f13d 227 kB/s | 142 kB 00:00 2024-04-23T10:07:30.048 INFO:teuthology.orchestra.run.smithi204.stdout:(22/152): python3-rados-19.0.0-3156.gd69f13d8.e 208 kB/s | 320 kB 00:01 2024-04-23T10:07:31.573 INFO:teuthology.orchestra.run.smithi204.stdout:(23/152): python3-rbd-19.0.0-3156.gd69f13d8.el7 195 kB/s | 298 kB 00:01 2024-04-23T10:07:33.115 INFO:teuthology.orchestra.run.smithi204.stdout:(24/152): python3-rgw-19.0.0-3156.gd69f13d8.el7 64 kB/s | 99 kB 00:01 2024-04-23T10:07:33.791 INFO:teuthology.orchestra.run.smithi204.stdout:(25/152): rbd-fuse-19.0.0-3156.gd69f13d8.el7.x8 126 kB/s | 85 kB 00:00 2024-04-23T10:07:39.566 INFO:teuthology.orchestra.run.smithi114.stdout:(26/152): rbd-mirror-19.0.0-3156.gd69f13d8.el7. 170 kB/s | 3.1 MB 00:18 2024-04-23T10:07:39.991 INFO:teuthology.orchestra.run.smithi114.stdout:(27/152): librgw2-19.0.0-3156.gd69f13d8.el7.x86 185 kB/s | 5.4 MB 00:29 2024-04-23T10:07:40.250 INFO:teuthology.orchestra.run.smithi114.stdout:(28/152): ceph-grafana-dashboards-19.0.0-3156.g 132 kB/s | 34 kB 00:00 2024-04-23T10:07:40.491 INFO:teuthology.orchestra.run.smithi114.stdout:(29/152): rbd-nbd-19.0.0-3156.gd69f13d8.el7.x86 184 kB/s | 170 kB 00:00 2024-04-23T10:07:40.858 INFO:teuthology.orchestra.run.smithi114.stdout:(30/152): ceph-mgr-cephadm-19.0.0-3156.gd69f13d 234 kB/s | 142 kB 00:00 2024-04-23T10:07:46.931 INFO:teuthology.orchestra.run.smithi119.stdout:(31/152): ceph-mgr-dashboard-19.0.0-3156.gd69f1 207 kB/s | 3.5 MB 00:17 2024-04-23T10:07:48.697 INFO:teuthology.orchestra.run.smithi119.stdout:(32/152): ceph-mgr-modules-core-19.0.0-3156.gd6 141 kB/s | 249 kB 00:01 2024-04-23T10:07:49.031 INFO:teuthology.orchestra.run.smithi119.stdout:(33/152): ceph-mgr-rook-19.0.0-3156.gd69f13d8.e 148 kB/s | 49 kB 00:00 2024-04-23T10:07:49.473 INFO:teuthology.orchestra.run.smithi119.stdout:(34/152): ceph-prometheus-alerts-19.0.0-3156.gd 37 kB/s | 16 kB 00:00 2024-04-23T10:07:51.706 INFO:teuthology.orchestra.run.smithi204.stdout:(26/152): rbd-mirror-19.0.0-3156.gd69f13d8.el7. 176 kB/s | 3.1 MB 00:17 2024-04-23T10:07:52.064 INFO:teuthology.orchestra.run.smithi119.stdout:(35/152): ceph-volume-19.0.0-3156.gd69f13d8.el7 105 kB/s | 273 kB 00:02 2024-04-23T10:07:52.431 INFO:teuthology.orchestra.run.smithi204.stdout:(27/152): rbd-nbd-19.0.0-3156.gd69f13d8.el7.x86 235 kB/s | 170 kB 00:00 2024-04-23T10:07:53.089 INFO:teuthology.orchestra.run.smithi204.stdout:(28/152): ceph-grafana-dashboards-19.0.0-3156.g 52 kB/s | 34 kB 00:00 2024-04-23T10:07:53.781 INFO:teuthology.orchestra.run.smithi204.stdout:(29/152): ceph-mgr-cephadm-19.0.0-3156.gd69f13d 206 kB/s | 142 kB 00:00 2024-04-23T10:07:55.556 INFO:teuthology.orchestra.run.smithi119.stdout:(36/152): cephadm-19.0.0-3156.gd69f13d8.el7.noa 218 kB/s | 762 kB 00:03 2024-04-23T10:07:55.664 INFO:teuthology.orchestra.run.smithi119.stdout:(37/152): cephfs-top-19.0.0-3156.gd69f13d8.el7. 168 kB/s | 18 kB 00:00 2024-04-23T10:07:55.865 INFO:teuthology.orchestra.run.smithi119.stdout:(38/152): fuse-2.9.9-15.el9.x86_64.rpm 402 kB/s | 80 kB 00:00 2024-04-23T10:07:56.015 INFO:teuthology.orchestra.run.smithi119.stdout:(39/152): libgfortran-11.4.1-3.el9.x86_64.rpm 5.2 MB/s | 803 kB 00:00 2024-04-23T10:07:56.082 INFO:teuthology.orchestra.run.smithi119.stdout:(40/152): libquadmath-11.4.1-3.el9.x86_64.rpm 2.9 MB/s | 195 kB 00:00 2024-04-23T10:07:56.149 INFO:teuthology.orchestra.run.smithi119.stdout:(41/152): python3-cffi-1.14.5-5.el9.x86_64.rpm 3.7 MB/s | 253 kB 00:00 2024-04-23T10:07:56.266 INFO:teuthology.orchestra.run.smithi119.stdout:(42/152): python3-cryptography-36.0.1-4.el9.x86 11 MB/s | 1.2 MB 00:00 2024-04-23T10:07:56.324 INFO:teuthology.orchestra.run.smithi119.stdout:(43/152): python3-ply-3.11-14.el9.noarch.rpm 1.8 MB/s | 106 kB 00:00 2024-04-23T10:07:56.383 INFO:teuthology.orchestra.run.smithi119.stdout:(44/152): python3-pycparser-2.20-6.el9.noarch.r 2.3 MB/s | 135 kB 00:00 2024-04-23T10:07:56.450 INFO:teuthology.orchestra.run.smithi119.stdout:(45/152): python3-requests-2.25.1-8.el9.noarch. 1.8 MB/s | 125 kB 00:00 2024-04-23T10:07:56.517 INFO:teuthology.orchestra.run.smithi119.stdout:(46/152): python3-urllib3-1.26.5-5.el9.noarch.r 3.2 MB/s | 215 kB 00:00 2024-04-23T10:07:56.823 INFO:teuthology.orchestra.run.smithi204.stdout:(30/152): librgw2-19.0.0-3156.gd69f13d8.el7.x86 181 kB/s | 5.4 MB 00:30 2024-04-23T10:07:58.092 INFO:teuthology.orchestra.run.smithi119.stdout:(47/152): automake-1.16.2-8.el9.noarch.rpm 437 kB/s | 689 kB 00:01 2024-04-23T10:07:58.667 INFO:teuthology.orchestra.run.smithi119.stdout:(48/152): ceph-mgr-diskprediction-local-19.0.0- 260 kB/s | 7.4 MB 00:29 2024-04-23T10:07:59.592 INFO:teuthology.orchestra.run.smithi119.stdout:(49/152): boost-program-options-1.75.0-8.el9.x8 116 kB/s | 107 kB 00:00 2024-04-23T10:08:00.168 INFO:teuthology.orchestra.run.smithi119.stdout:(50/152): bison-3.7.4-5.el9.x86_64.rpm 454 kB/s | 943 kB 00:02 2024-04-23T10:08:00.194 INFO:teuthology.orchestra.run.smithi119.stdout:(51/152): elfutils-libelf-devel-0.190-2.el9.x86 36 kB/s | 21 kB 00:00 2024-04-23T10:08:00.793 INFO:teuthology.orchestra.run.smithi119.stdout:(52/152): flexiblas-3.0.4-8.el9.x86_64.rpm 54 kB/s | 33 kB 00:00 2024-04-23T10:08:01.143 INFO:teuthology.orchestra.run.smithi119.stdout:(53/152): flex-2.6.4-9.el9.x86_64.rpm 322 kB/s | 313 kB 00:00 2024-04-23T10:08:01.173 INFO:teuthology.orchestra.run.smithi114.stdout:(31/152): ceph-mgr-dashboard-19.0.0-3156.gd69f1 175 kB/s | 3.5 MB 00:20 2024-04-23T10:08:01.685 INFO:teuthology.orchestra.run.smithi119.stdout:(54/152): flexiblas-openblas-openmp-3.0.4-8.el9 34 kB/s | 18 kB 00:00 2024-04-23T10:08:02.194 INFO:teuthology.orchestra.run.smithi119.stdout:(55/152): libaio-devel-0.3.111-13.el9.x86_64.rp 22 kB/s | 11 kB 00:00 2024-04-23T10:08:02.223 INFO:teuthology.orchestra.run.smithi114.stdout:(32/152): ceph-mgr-modules-core-19.0.0-3156.gd6 237 kB/s | 249 kB 00:01 2024-04-23T10:08:02.532 INFO:teuthology.orchestra.run.smithi114.stdout:(33/152): ceph-mgr-rook-19.0.0-3156.gd69f13d8.e 160 kB/s | 49 kB 00:00 2024-04-23T10:08:02.657 INFO:teuthology.orchestra.run.smithi114.stdout:(34/152): ceph-prometheus-alerts-19.0.0-3156.gd 131 kB/s | 16 kB 00:00 2024-04-23T10:08:03.011 INFO:teuthology.orchestra.run.smithi119.stdout:(56/152): libpmemobj-1.12.1-1.el9.x86_64.rpm 196 kB/s | 160 kB 00:00 2024-04-23T10:08:03.661 INFO:teuthology.orchestra.run.smithi119.stdout:(57/152): librabbitmq-0.11.0-7.el9.x86_64.rpm 70 kB/s | 45 kB 00:00 2024-04-23T10:08:04.098 INFO:teuthology.orchestra.run.smithi114.stdout:(35/152): ceph-volume-19.0.0-3156.gd69f13d8.el7 189 kB/s | 273 kB 00:01 2024-04-23T10:08:05.086 INFO:teuthology.orchestra.run.smithi119.stdout:(58/152): librdkafka-1.6.1-102.el9.x86_64.rpm 465 kB/s | 662 kB 00:01 2024-04-23T10:08:05.228 INFO:teuthology.orchestra.run.smithi119.stdout:(59/152): flexiblas-netlib-3.0.4-8.el9.x86_64.r 690 kB/s | 3.0 MB 00:04 2024-04-23T10:08:05.820 INFO:teuthology.orchestra.run.smithi119.stdout:(60/152): libuuid-devel-2.37.4-18.el9.x86_64.rp 43 kB/s | 25 kB 00:00 2024-04-23T10:08:06.462 INFO:teuthology.orchestra.run.smithi119.stdout:(61/152): libzstd-devel-1.5.1-2.el9.x86_64.rpm 74 kB/s | 47 kB 00:00 2024-04-23T10:08:06.487 INFO:teuthology.orchestra.run.smithi119.stdout:(62/152): libtool-2.4.6-45.el9.x86_64.rpm 416 kB/s | 582 kB 00:01 2024-04-23T10:08:07.329 INFO:teuthology.orchestra.run.smithi119.stdout:(63/152): lua-5.4.4-4.el9.x86_64.rpm 224 kB/s | 188 kB 00:00 2024-04-23T10:08:07.446 INFO:teuthology.orchestra.run.smithi119.stdout:(64/152): lttng-ust-2.12.0-6.el9.x86_64.rpm 297 kB/s | 292 kB 00:00 2024-04-23T10:08:07.955 INFO:teuthology.orchestra.run.smithi119.stdout:(65/152): openblas-0.3.21-2.el9.x86_64.rpm 56 kB/s | 35 kB 00:00 2024-04-23T10:08:11.473 INFO:teuthology.orchestra.run.smithi114.stdout:(36/152): cephadm-19.0.0-3156.gd69f13d8.el7.noa 103 kB/s | 762 kB 00:07 2024-04-23T10:08:12.073 INFO:teuthology.orchestra.run.smithi114.stdout:(37/152): cephfs-top-19.0.0-3156.gd69f13d8.el7. 30 kB/s | 18 kB 00:00 2024-04-23T10:08:12.198 INFO:teuthology.orchestra.run.smithi114.stdout:(38/152): fuse-2.9.9-15.el9.x86_64.rpm 644 kB/s | 80 kB 00:00 2024-04-23T10:08:12.299 INFO:teuthology.orchestra.run.smithi114.stdout:(39/152): libgfortran-11.4.1-3.el9.x86_64.rpm 7.8 MB/s | 803 kB 00:00 2024-04-23T10:08:12.341 INFO:teuthology.orchestra.run.smithi114.stdout:(40/152): libquadmath-11.4.1-3.el9.x86_64.rpm 4.6 MB/s | 195 kB 00:00 2024-04-23T10:08:12.383 INFO:teuthology.orchestra.run.smithi114.stdout:(41/152): python3-cffi-1.14.5-5.el9.x86_64.rpm 5.9 MB/s | 253 kB 00:00 2024-04-23T10:08:12.405 INFO:teuthology.orchestra.run.smithi204.stdout:(31/152): ceph-mgr-dashboard-19.0.0-3156.gd69f1 195 kB/s | 3.5 MB 00:18 2024-04-23T10:08:12.450 INFO:teuthology.orchestra.run.smithi114.stdout:(42/152): python3-cryptography-36.0.1-4.el9.x86 18 MB/s | 1.2 MB 00:00 2024-04-23T10:08:12.492 INFO:teuthology.orchestra.run.smithi114.stdout:(43/152): python3-ply-3.11-14.el9.noarch.rpm 2.5 MB/s | 106 kB 00:00 2024-04-23T10:08:12.533 INFO:teuthology.orchestra.run.smithi114.stdout:(44/152): python3-pycparser-2.20-6.el9.noarch.r 3.2 MB/s | 135 kB 00:00 2024-04-23T10:08:12.576 INFO:teuthology.orchestra.run.smithi114.stdout:(45/152): python3-requests-2.25.1-8.el9.noarch. 2.9 MB/s | 125 kB 00:00 2024-04-23T10:08:12.617 INFO:teuthology.orchestra.run.smithi114.stdout:(46/152): python3-urllib3-1.26.5-5.el9.noarch.r 5.0 MB/s | 215 kB 00:00 2024-04-23T10:08:12.959 INFO:teuthology.orchestra.run.smithi114.stdout:(47/152): automake-1.16.2-8.el9.noarch.rpm 2.0 MB/s | 689 kB 00:00 2024-04-23T10:08:13.218 INFO:teuthology.orchestra.run.smithi114.stdout:(48/152): bison-3.7.4-5.el9.x86_64.rpm 3.6 MB/s | 943 kB 00:00 2024-04-23T10:08:13.268 INFO:teuthology.orchestra.run.smithi114.stdout:(49/152): boost-program-options-1.75.0-8.el9.x8 2.1 MB/s | 107 kB 00:00 2024-04-23T10:08:13.319 INFO:teuthology.orchestra.run.smithi114.stdout:(50/152): elfutils-libelf-devel-0.190-2.el9.x86 430 kB/s | 21 kB 00:00 2024-04-23T10:08:13.347 INFO:teuthology.orchestra.run.smithi204.stdout:(32/152): ceph-mgr-modules-core-19.0.0-3156.gd6 264 kB/s | 249 kB 00:00 2024-04-23T10:08:13.444 INFO:teuthology.orchestra.run.smithi114.stdout:(51/152): flex-2.6.4-9.el9.x86_64.rpm 2.5 MB/s | 313 kB 00:00 2024-04-23T10:08:13.519 INFO:teuthology.orchestra.run.smithi114.stdout:(52/152): flexiblas-3.0.4-8.el9.x86_64.rpm 436 kB/s | 33 kB 00:00 2024-04-23T10:08:13.580 INFO:teuthology.orchestra.run.smithi204.stdout:(33/152): ceph-mgr-rook-19.0.0-3156.gd69f13d8.e 211 kB/s | 49 kB 00:00 2024-04-23T10:08:13.698 INFO:teuthology.orchestra.run.smithi204.stdout:(34/152): ceph-prometheus-alerts-19.0.0-3156.gd 141 kB/s | 16 kB 00:00 2024-04-23T10:08:13.703 INFO:teuthology.orchestra.run.smithi114.stdout:(53/152): ceph-mgr-diskprediction-local-19.0.0- 230 kB/s | 7.4 MB 00:32 2024-04-23T10:08:13.787 INFO:teuthology.orchestra.run.smithi114.stdout:(54/152): flexiblas-openblas-openmp-3.0.4-8.el9 219 kB/s | 18 kB 00:00 2024-04-23T10:08:13.837 INFO:teuthology.orchestra.run.smithi114.stdout:(55/152): libaio-devel-0.3.111-13.el9.x86_64.rp 224 kB/s | 11 kB 00:00 2024-04-23T10:08:13.904 INFO:teuthology.orchestra.run.smithi119.stdout:(66/152): openblas-openmp-0.3.21-2.el9.x86_64.r 751 kB/s | 4.7 MB 00:06 2024-04-23T10:08:13.953 INFO:teuthology.orchestra.run.smithi114.stdout:(56/152): libpmemobj-1.12.1-1.el9.x86_64.rpm 1.3 MB/s | 160 kB 00:00 2024-04-23T10:08:14.012 INFO:teuthology.orchestra.run.smithi114.stdout:(57/152): librabbitmq-0.11.0-7.el9.x86_64.rpm 777 kB/s | 45 kB 00:00 2024-04-23T10:08:14.054 INFO:teuthology.orchestra.run.smithi119.stdout:(67/152): openssl-devel-3.2.1-1.el9.x86_64.rpm 743 kB/s | 4.4 MB 00:06 2024-04-23T10:08:14.213 INFO:teuthology.orchestra.run.smithi114.stdout:(58/152): librdkafka-1.6.1-102.el9.x86_64.rpm 3.2 MB/s | 662 kB 00:00 2024-04-23T10:08:14.288 INFO:teuthology.orchestra.run.smithi114.stdout:(59/152): flexiblas-netlib-3.0.4-8.el9.x86_64.r 3.9 MB/s | 3.0 MB 00:00 2024-04-23T10:08:14.338 INFO:teuthology.orchestra.run.smithi114.stdout:(60/152): libuuid-devel-2.37.4-18.el9.x86_64.rp 511 kB/s | 25 kB 00:00 2024-04-23T10:08:14.389 INFO:teuthology.orchestra.run.smithi114.stdout:(61/152): libzstd-devel-1.5.1-2.el9.x86_64.rpm 950 kB/s | 47 kB 00:00 2024-04-23T10:08:14.455 INFO:teuthology.orchestra.run.smithi114.stdout:(62/152): libtool-2.4.6-45.el9.x86_64.rpm 2.3 MB/s | 582 kB 00:00 2024-04-23T10:08:14.538 INFO:teuthology.orchestra.run.smithi114.stdout:(63/152): lttng-ust-2.12.0-6.el9.x86_64.rpm 1.9 MB/s | 292 kB 00:00 2024-04-23T10:08:14.598 INFO:teuthology.orchestra.run.smithi114.stdout:(64/152): lua-5.4.4-4.el9.x86_64.rpm 1.3 MB/s | 188 kB 00:00 2024-04-23T10:08:14.623 INFO:teuthology.orchestra.run.smithi114.stdout:(65/152): openblas-0.3.21-2.el9.x86_64.rpm 420 kB/s | 35 kB 00:00 2024-04-23T10:08:14.856 INFO:teuthology.orchestra.run.smithi204.stdout:(35/152): ceph-volume-19.0.0-3156.gd69f13d8.el7 236 kB/s | 273 kB 00:01 2024-04-23T10:08:15.690 INFO:teuthology.orchestra.run.smithi114.stdout:(66/152): openssl-devel-3.2.1-1.el9.x86_64.rpm 4.1 MB/s | 4.4 MB 00:01 2024-04-23T10:08:15.996 INFO:teuthology.orchestra.run.smithi119.stdout:(68/152): postgresql-contrib-13.14-1.el9.x86_64 469 kB/s | 911 kB 00:01 2024-04-23T10:08:16.173 INFO:teuthology.orchestra.run.smithi114.stdout:(67/152): openblas-openmp-0.3.21-2.el9.x86_64.r 3.0 MB/s | 4.7 MB 00:01 2024-04-23T10:08:16.223 INFO:teuthology.orchestra.run.smithi114.stdout:(68/152): postgresql-13.14-1.el9.x86_64.rpm 3.0 MB/s | 1.6 MB 00:00 2024-04-23T10:08:16.282 INFO:teuthology.orchestra.run.smithi114.stdout:(69/152): postgresql-private-libs-13.14-1.el9.x 2.2 MB/s | 133 kB 00:00 2024-04-23T10:08:16.457 INFO:teuthology.orchestra.run.smithi114.stdout:(70/152): postgresql-contrib-13.14-1.el9.x86_64 3.1 MB/s | 911 kB 00:00 2024-04-23T10:08:16.733 INFO:teuthology.orchestra.run.smithi114.stdout:(71/152): protobuf-3.14.0-13.el9.x86_64.rpm 3.7 MB/s | 1.0 MB 00:00 2024-04-23T10:08:16.746 INFO:teuthology.orchestra.run.smithi119.stdout:(69/152): postgresql-13.14-1.el9.x86_64.rpm 572 kB/s | 1.6 MB 00:02 2024-04-23T10:08:16.771 INFO:teuthology.orchestra.run.smithi119.stdout:(70/152): postgresql-private-libs-13.14-1.el9.x 172 kB/s | 133 kB 00:00 2024-04-23T10:08:16.791 INFO:teuthology.orchestra.run.smithi114.stdout:(72/152): python3-attrs-20.3.0-7.el9.noarch.rpm 1.5 MB/s | 87 kB 00:00 2024-04-23T10:08:17.425 INFO:teuthology.orchestra.run.smithi114.stdout:(73/152): postgresql-server-13.14-1.el9.x86_64. 5.1 MB/s | 5.8 MB 00:01 2024-04-23T10:08:17.525 INFO:teuthology.orchestra.run.smithi114.stdout:(74/152): python3-devel-3.9.18-3.el9.x86_64.rpm 2.4 MB/s | 244 kB 00:00 2024-04-23T10:08:17.608 INFO:teuthology.orchestra.run.smithi114.stdout:(75/152): python3-jinja2-2.11.3-5.el9.noarch.rp 2.9 MB/s | 248 kB 00:00 2024-04-23T10:08:17.650 INFO:teuthology.orchestra.run.smithi114.stdout:(76/152): python3-jsonpatch-1.21-16.el9.noarch. 615 kB/s | 26 kB 00:00 2024-04-23T10:08:17.692 INFO:teuthology.orchestra.run.smithi114.stdout:(77/152): python3-jsonpointer-2.0-4.el9.noarch. 461 kB/s | 19 kB 00:00 2024-04-23T10:08:17.751 INFO:teuthology.orchestra.run.smithi114.stdout:(78/152): python3-mako-1.1.4-6.el9.noarch.rpm 2.9 MB/s | 172 kB 00:00 2024-04-23T10:08:17.801 INFO:teuthology.orchestra.run.smithi114.stdout:(79/152): python3-markupsafe-1.1.1-12.el9.x86_6 695 kB/s | 35 kB 00:00 2024-04-23T10:08:18.010 INFO:teuthology.orchestra.run.smithi114.stdout:(80/152): python3-babel-2.9.1-2.el9.noarch.rpm 4.9 MB/s | 6.0 MB 00:01 2024-04-23T10:08:18.102 INFO:teuthology.orchestra.run.smithi114.stdout:(81/152): python3-numpy-f2py-1.20.1-5.el9.x86_6 2.7 MB/s | 250 kB 00:00 2024-04-23T10:08:18.161 INFO:teuthology.orchestra.run.smithi114.stdout:(82/152): python3-oauthlib-3.1.1-5.el9.noarch.r 3.7 MB/s | 222 kB 00:00 2024-04-23T10:08:18.211 INFO:teuthology.orchestra.run.smithi114.stdout:(83/152): python3-packaging-20.9-5.el9.noarch.r 1.5 MB/s | 77 kB 00:00 2024-04-23T10:08:18.269 INFO:teuthology.orchestra.run.smithi114.stdout:(84/152): python3-prettytable-0.7.2-27.el9.noar 728 kB/s | 42 kB 00:00 2024-04-23T10:08:18.353 INFO:teuthology.orchestra.run.smithi114.stdout:(85/152): python3-protobuf-3.14.0-13.el9.noarch 3.2 MB/s | 269 kB 00:00 2024-04-23T10:08:18.412 INFO:teuthology.orchestra.run.smithi114.stdout:(86/152): python3-pyasn1-0.4.8-6.el9.noarch.rpm 2.7 MB/s | 159 kB 00:00 2024-04-23T10:08:18.504 INFO:teuthology.orchestra.run.smithi114.stdout:(87/152): python3-pyasn1-modules-0.4.8-6.el9.no 3.0 MB/s | 279 kB 00:00 2024-04-23T10:08:18.571 INFO:teuthology.orchestra.run.smithi114.stdout:(88/152): python3-numpy-1.20.1-5.el9.x86_64.rpm 6.6 MB/s | 5.1 MB 00:00 2024-04-23T10:08:18.596 INFO:teuthology.orchestra.run.smithi114.stdout:(89/152): python3-pytz-2021.1-5.el9.noarch.rpm 554 kB/s | 51 kB 00:00 2024-04-23T10:08:18.621 INFO:teuthology.orchestra.run.smithi114.stdout:(90/152): python3-requests-oauthlib-1.3.0-12.el 1.0 MB/s | 54 kB 00:00 2024-04-23T10:08:18.672 INFO:teuthology.orchestra.run.smithi114.stdout:(91/152): python3-toml-0.10.2-6.el9.noarch.rpm 832 kB/s | 42 kB 00:00 2024-04-23T10:08:18.722 INFO:teuthology.orchestra.run.smithi114.stdout:(92/152): python3-wcwidth-0.2.5-8.el9.noarch.rp 884 kB/s | 44 kB 00:00 2024-04-23T10:08:18.754 INFO:teuthology.orchestra.run.smithi119.stdout:(71/152): protobuf-3.14.0-13.el9.x86_64.rpm 521 kB/s | 1.0 MB 00:01 2024-04-23T10:08:18.772 INFO:teuthology.orchestra.run.smithi114.stdout:(93/152): qatlib-24.02.0-1.el9.x86_64.rpm 4.3 MB/s | 221 kB 00:00 2024-04-23T10:08:18.822 INFO:teuthology.orchestra.run.smithi114.stdout:(94/152): qatlib-service-24.02.0-1.el9.x86_64.r 718 kB/s | 36 kB 00:00 2024-04-23T10:08:18.882 INFO:teuthology.orchestra.run.smithi114.stdout:(95/152): qatzip-libs-1.2.0-1.el9.x86_64.rpm 817 kB/s | 47 kB 00:00 2024-04-23T10:08:18.940 INFO:teuthology.orchestra.run.smithi114.stdout:(96/152): socat-1.7.4.1-5.el9.x86_64.rpm 5.1 MB/s | 305 kB 00:00 2024-04-23T10:08:18.998 INFO:teuthology.orchestra.run.smithi114.stdout:(97/152): uuid-1.6.2-55.el9.x86_64.rpm 993 kB/s | 58 kB 00:00 2024-04-23T10:08:19.057 INFO:teuthology.orchestra.run.smithi114.stdout:(98/152): xfsprogs-devel-6.3.0-1.el9.x86_64.rpm 1.6 MB/s | 94 kB 00:00 2024-04-23T10:08:19.115 INFO:teuthology.orchestra.run.smithi114.stdout:(99/152): xmlstarlet-1.6.1-20.el9.x86_64.rpm 1.1 MB/s | 64 kB 00:00 2024-04-23T10:08:19.156 INFO:teuthology.orchestra.run.smithi204.stdout:(36/152): cephadm-19.0.0-3156.gd69f13d8.el7.noa 177 kB/s | 762 kB 00:04 2024-04-23T10:08:19.174 INFO:teuthology.orchestra.run.smithi114.stdout:(100/152): zlib-devel-1.2.11-41.el9.x86_64.rpm 770 kB/s | 45 kB 00:00 2024-04-23T10:08:19.232 INFO:teuthology.orchestra.run.smithi114.stdout:(101/152): lua-devel-5.4.4-4.el9.x86_64.rpm 382 kB/s | 22 kB 00:00 2024-04-23T10:08:19.358 INFO:teuthology.orchestra.run.smithi114.stdout:(102/152): protobuf-compiler-3.14.0-13.el9.x86_ 6.7 MB/s | 863 kB 00:00 2024-04-23T10:08:19.364 INFO:teuthology.orchestra.run.smithi204.stdout:(37/152): cephfs-top-19.0.0-3156.gd69f13d8.el7. 87 kB/s | 18 kB 00:00 2024-04-23T10:08:19.455 INFO:teuthology.orchestra.run.smithi119.stdout:(72/152): python3-attrs-20.3.0-7.el9.noarch.rpm 125 kB/s | 87 kB 00:00 2024-04-23T10:08:19.548 INFO:teuthology.orchestra.run.smithi204.stdout:(38/152): fuse-2.9.9-15.el9.x86_64.rpm 439 kB/s | 80 kB 00:00 2024-04-23T10:08:19.715 INFO:teuthology.orchestra.run.smithi204.stdout:(39/152): libgfortran-11.4.1-3.el9.x86_64.rpm 4.7 MB/s | 803 kB 00:00 2024-04-23T10:08:19.774 INFO:teuthology.orchestra.run.smithi204.stdout:(40/152): libquadmath-11.4.1-3.el9.x86_64.rpm 3.3 MB/s | 195 kB 00:00 2024-04-23T10:08:19.866 INFO:teuthology.orchestra.run.smithi204.stdout:(41/152): python3-cffi-1.14.5-5.el9.x86_64.rpm 2.7 MB/s | 253 kB 00:00 2024-04-23T10:08:20.016 INFO:teuthology.orchestra.run.smithi204.stdout:(42/152): python3-cryptography-36.0.1-4.el9.x86 8.2 MB/s | 1.2 MB 00:00 2024-04-23T10:08:20.083 INFO:teuthology.orchestra.run.smithi204.stdout:(43/152): python3-ply-3.11-14.el9.noarch.rpm 1.6 MB/s | 106 kB 00:00 2024-04-23T10:08:20.141 INFO:teuthology.orchestra.run.smithi204.stdout:(44/152): python3-pycparser-2.20-6.el9.noarch.r 2.3 MB/s | 135 kB 00:00 2024-04-23T10:08:20.158 INFO:teuthology.orchestra.run.smithi114.stdout:(103/152): abseil-cpp-20211102.0-3.el9.x86_64.r 685 kB/s | 548 kB 00:00 2024-04-23T10:08:20.200 INFO:teuthology.orchestra.run.smithi204.stdout:(45/152): python3-requests-2.25.1-8.el9.noarch. 2.1 MB/s | 125 kB 00:00 2024-04-23T10:08:20.259 INFO:teuthology.orchestra.run.smithi204.stdout:(46/152): python3-urllib3-1.26.5-5.el9.noarch.r 3.6 MB/s | 215 kB 00:00 2024-04-23T10:08:20.308 INFO:teuthology.orchestra.run.smithi114.stdout:(104/152): gperftools-libs-2.9.1-2.el9.x86_64.r 2.0 MB/s | 309 kB 00:00 2024-04-23T10:08:20.484 INFO:teuthology.orchestra.run.smithi114.stdout:(105/152): python3-scipy-1.6.2-8.el9.x86_64.rpm 8.6 MB/s | 16 MB 00:01 2024-04-23T10:08:20.509 INFO:teuthology.orchestra.run.smithi114.stdout:(106/152): grpc-data-1.46.7-10.el9.noarch.rpm 97 kB/s | 19 kB 00:00 2024-04-23T10:08:20.651 INFO:teuthology.orchestra.run.smithi114.stdout:(107/152): libarrow-doc-9.0.0-10.el9.noarch.rpm 183 kB/s | 26 kB 00:00 2024-04-23T10:08:20.659 INFO:teuthology.orchestra.run.smithi204.stdout:(47/152): automake-1.16.2-8.el9.noarch.rpm 1.7 MB/s | 689 kB 00:00 2024-04-23T10:08:20.768 INFO:teuthology.orchestra.run.smithi114.stdout:(108/152): liboath-2.6.7-2.el9.x86_64.rpm 415 kB/s | 49 kB 00:00 2024-04-23T10:08:20.885 INFO:teuthology.orchestra.run.smithi114.stdout:(109/152): libunwind-1.6.2-1.el9.x86_64.rpm 577 kB/s | 67 kB 00:00 2024-04-23T10:08:20.976 INFO:teuthology.orchestra.run.smithi204.stdout:(48/152): bison-3.7.4-5.el9.x86_64.rpm 2.9 MB/s | 943 kB 00:00 2024-04-23T10:08:21.010 INFO:teuthology.orchestra.run.smithi114.stdout:(110/152): luarocks-3.9.2-1.el9.noarch.rpm 1.2 MB/s | 151 kB 00:00 2024-04-23T10:08:21.026 INFO:teuthology.orchestra.run.smithi204.stdout:(49/152): boost-program-options-1.75.0-8.el9.x8 2.1 MB/s | 107 kB 00:00 2024-04-23T10:08:21.077 INFO:teuthology.orchestra.run.smithi204.stdout:(50/152): elfutils-libelf-devel-0.190-2.el9.x86 430 kB/s | 21 kB 00:00 2024-04-23T10:08:21.244 INFO:teuthology.orchestra.run.smithi204.stdout:(51/152): flex-2.6.4-9.el9.x86_64.rpm 1.8 MB/s | 313 kB 00:00 2024-04-23T10:08:21.260 INFO:teuthology.orchestra.run.smithi114.stdout:(111/152): parquet-libs-9.0.0-10.el9.x86_64.rpm 3.3 MB/s | 838 kB 00:00 2024-04-23T10:08:21.302 INFO:teuthology.orchestra.run.smithi204.stdout:(52/152): flexiblas-3.0.4-8.el9.x86_64.rpm 560 kB/s | 33 kB 00:00 2024-04-23T10:08:21.402 INFO:teuthology.orchestra.run.smithi114.stdout:(112/152): python3-asyncssh-2.13.2-5.el9.noarch 3.8 MB/s | 548 kB 00:00 2024-04-23T10:08:21.511 INFO:teuthology.orchestra.run.smithi114.stdout:(113/152): python3-bcrypt-3.2.2-1.el9.x86_64.rp 401 kB/s | 43 kB 00:00 2024-04-23T10:08:21.594 INFO:teuthology.orchestra.run.smithi114.stdout:(114/152): libarrow-9.0.0-10.el9.x86_64.rpm 4.0 MB/s | 4.4 MB 00:01 2024-04-23T10:08:21.628 INFO:teuthology.orchestra.run.smithi114.stdout:(115/152): python3-cachetools-4.2.4-1.el9.noarc 275 kB/s | 32 kB 00:00 2024-04-23T10:08:21.703 INFO:teuthology.orchestra.run.smithi114.stdout:(116/152): python3-certifi-2021.10.8-2.el9.noar 138 kB/s | 15 kB 00:00 2024-04-23T10:08:21.762 INFO:teuthology.orchestra.run.smithi114.stdout:(117/152): python3-cheroot-8.6.0-4.el9.noarch.r 1.3 MB/s | 172 kB 00:00 2024-04-23T10:08:21.854 INFO:teuthology.orchestra.run.smithi114.stdout:(118/152): python3-cherrypy-18.6.1-2.el9.noarch 2.3 MB/s | 358 kB 00:00 2024-04-23T10:08:21.912 INFO:teuthology.orchestra.run.smithi114.stdout:(119/152): python3-cmd2-2.4.2-2.el9.noarch.rpm 2.0 MB/s | 303 kB 00:00 2024-04-23T10:08:21.979 INFO:teuthology.orchestra.run.smithi114.stdout:(120/152): python3-colorama-0.4.4-12.el9.noarch 282 kB/s | 35 kB 00:00 2024-04-23T10:08:22.038 INFO:teuthology.orchestra.run.smithi114.stdout:(121/152): python3-google-auth-2.29.0-1.el9.noa 1.6 MB/s | 206 kB 00:00 2024-04-23T10:08:22.155 INFO:teuthology.orchestra.run.smithi114.stdout:(122/152): python3-grpcio-1.46.7-10.el9.x86_64. 12 MB/s | 2.0 MB 00:00 2024-04-23T10:08:22.180 INFO:teuthology.orchestra.run.smithi114.stdout:(123/152): python3-grpcio-tools-1.46.7-10.el9.x 1.0 MB/s | 144 kB 00:00 2024-04-23T10:08:22.211 INFO:teuthology.orchestra.run.smithi204.stdout:(53/152): flexiblas-netlib-3.0.4-8.el9.x86_64.r 3.3 MB/s | 3.0 MB 00:00 2024-04-23T10:08:22.264 INFO:teuthology.orchestra.run.smithi114.stdout:(124/152): python3-jaraco-8.2.1-3.el9.noarch.rp 98 kB/s | 11 kB 00:00 2024-04-23T10:08:22.278 INFO:teuthology.orchestra.run.smithi204.stdout:(54/152): flexiblas-openblas-openmp-3.0.4-8.el9 274 kB/s | 18 kB 00:00 2024-04-23T10:08:22.289 INFO:teuthology.orchestra.run.smithi114.stdout:(125/152): python3-jaraco-classes-3.2.1-5.el9.n 163 kB/s | 18 kB 00:00 2024-04-23T10:08:22.328 INFO:teuthology.orchestra.run.smithi204.stdout:(55/152): libaio-devel-0.3.111-13.el9.x86_64.rp 223 kB/s | 11 kB 00:00 2024-04-23T10:08:22.364 INFO:teuthology.orchestra.run.smithi114.stdout:(126/152): python3-jaraco-collections-3.0.0-8.e 231 kB/s | 23 kB 00:00 2024-04-23T10:08:22.387 INFO:teuthology.orchestra.run.smithi204.stdout:(56/152): libpmemobj-1.12.1-1.el9.x86_64.rpm 2.7 MB/s | 160 kB 00:00 2024-04-23T10:08:22.390 INFO:teuthology.orchestra.run.smithi114.stdout:(127/152): python3-jaraco-functools-3.5.0-2.el9 194 kB/s | 19 kB 00:00 2024-04-23T10:08:22.437 INFO:teuthology.orchestra.run.smithi204.stdout:(57/152): librabbitmq-0.11.0-7.el9.x86_64.rpm 906 kB/s | 45 kB 00:00 2024-04-23T10:08:22.473 INFO:teuthology.orchestra.run.smithi114.stdout:(128/152): python3-jaraco-text-3.2.0-6.el9.noar 181 kB/s | 20 kB 00:00 2024-04-23T10:08:22.557 INFO:teuthology.orchestra.run.smithi114.stdout:(129/152): python3-kubernetes-26.1.0-2.el9.noar 6.1 MB/s | 1.0 MB 00:00 2024-04-23T10:08:22.582 INFO:teuthology.orchestra.run.smithi114.stdout:(130/152): python3-logutils-0.3.5-21.el9.noarch 426 kB/s | 46 kB 00:00 2024-04-23T10:08:22.588 INFO:teuthology.orchestra.run.smithi204.stdout:(58/152): librdkafka-1.6.1-102.el9.x86_64.rpm 4.3 MB/s | 662 kB 00:00 2024-04-23T10:08:22.666 INFO:teuthology.orchestra.run.smithi114.stdout:(131/152): python3-more-itertools-8.12.0-2.el9. 727 kB/s | 79 kB 00:00 2024-04-23T10:08:22.691 INFO:teuthology.orchestra.run.smithi114.stdout:(132/152): python3-natsort-7.1.1-5.el9.noarch.r 532 kB/s | 58 kB 00:00 2024-04-23T10:08:22.704 INFO:teuthology.orchestra.run.smithi204.stdout:(59/152): libtool-2.4.6-45.el9.x86_64.rpm 4.9 MB/s | 582 kB 00:00 2024-04-23T10:08:22.772 INFO:teuthology.orchestra.run.smithi204.stdout:(60/152): libuuid-devel-2.37.4-18.el9.x86_64.rp 381 kB/s | 25 kB 00:00 2024-04-23T10:08:22.783 INFO:teuthology.orchestra.run.smithi114.stdout:(133/152): python3-pecan-1.4.2-3.el9.noarch.rpm 2.3 MB/s | 272 kB 00:00 2024-04-23T10:08:22.808 INFO:teuthology.orchestra.run.smithi114.stdout:(134/152): python3-portend-3.1.0-2.el9.noarch.r 140 kB/s | 16 kB 00:00 2024-04-23T10:08:22.822 INFO:teuthology.orchestra.run.smithi204.stdout:(61/152): libzstd-devel-1.5.1-2.el9.x86_64.rpm 947 kB/s | 47 kB 00:00 2024-04-23T10:08:22.906 INFO:teuthology.orchestra.run.smithi204.stdout:(62/152): lttng-ust-2.12.0-6.el9.x86_64.rpm 3.4 MB/s | 292 kB 00:00 2024-04-23T10:08:22.908 INFO:teuthology.orchestra.run.smithi114.stdout:(135/152): python3-pyOpenSSL-21.0.0-1.el9.noarc 719 kB/s | 90 kB 00:00 2024-04-23T10:08:22.934 INFO:teuthology.orchestra.run.smithi114.stdout:(136/152): python3-pyperclip-1.8.2-2.el9.noarch 225 kB/s | 28 kB 00:00 2024-04-23T10:08:22.964 INFO:teuthology.orchestra.run.smithi204.stdout:(63/152): lua-5.4.4-4.el9.x86_64.rpm 3.2 MB/s | 188 kB 00:00 2024-04-23T10:08:23.006 INFO:teuthology.orchestra.run.smithi204.stdout:(64/152): openblas-0.3.21-2.el9.x86_64.rpm 848 kB/s | 35 kB 00:00 2024-04-23T10:08:23.017 INFO:teuthology.orchestra.run.smithi114.stdout:(137/152): python3-repoze-lru-0.7-16.el9.noarch 284 kB/s | 31 kB 00:00 2024-04-23T10:08:23.043 INFO:teuthology.orchestra.run.smithi114.stdout:(138/152): python3-routes-2.5.1-5.el9.noarch.rp 1.7 MB/s | 188 kB 00:00 2024-04-23T10:08:23.126 INFO:teuthology.orchestra.run.smithi114.stdout:(139/152): python3-rsa-4.9-2.el9.noarch.rpm 543 kB/s | 59 kB 00:00 2024-04-23T10:08:23.152 INFO:teuthology.orchestra.run.smithi114.stdout:(140/152): python3-tempora-5.0.0-2.el9.noarch.r 329 kB/s | 36 kB 00:00 2024-04-23T10:08:23.235 INFO:teuthology.orchestra.run.smithi114.stdout:(141/152): python3-typing-extensions-4.4.0-2.el 470 kB/s | 51 kB 00:00 2024-04-23T10:08:23.261 INFO:teuthology.orchestra.run.smithi114.stdout:(142/152): python3-webob-1.8.7-6.el9.noarch.rpm 2.1 MB/s | 230 kB 00:00 2024-04-23T10:08:23.353 INFO:teuthology.orchestra.run.smithi114.stdout:(143/152): python3-websocket-client-1.2.3-2.el9 767 kB/s | 90 kB 00:00 2024-04-23T10:08:23.378 INFO:teuthology.orchestra.run.smithi114.stdout:(144/152): python3-werkzeug-2.0.3-3.el9.1.noarc 3.6 MB/s | 427 kB 00:00 2024-04-23T10:08:23.462 INFO:teuthology.orchestra.run.smithi114.stdout:(145/152): python3-zc-lockfile-2.0-10.el9.noarc 184 kB/s | 20 kB 00:00 2024-04-23T10:08:23.487 INFO:teuthology.orchestra.run.smithi114.stdout:(146/152): re2-20211101-3.el9.x86_64.rpm 1.7 MB/s | 192 kB 00:00 2024-04-23T10:08:23.590 INFO:teuthology.orchestra.run.smithi204.stdout:(65/152): openblas-openmp-0.3.21-2.el9.x86_64.r 8.1 MB/s | 4.7 MB 00:00 2024-04-23T10:08:23.653 INFO:teuthology.orchestra.run.smithi114.stdout:(147/152): thrift-0.15.0-2.el9.x86_64.rpm 8.2 MB/s | 1.6 MB 00:00 2024-04-23T10:08:24.007 INFO:teuthology.orchestra.run.smithi204.stdout:(66/152): openssl-devel-3.2.1-1.el9.x86_64.rpm 11 MB/s | 4.4 MB 00:00 2024-04-23T10:08:24.165 INFO:teuthology.orchestra.run.smithi204.stdout:(67/152): postgresql-13.14-1.el9.x86_64.rpm 10 MB/s | 1.6 MB 00:00 2024-04-23T10:08:24.299 INFO:teuthology.orchestra.run.smithi204.stdout:(68/152): postgresql-contrib-13.14-1.el9.x86_64 6.7 MB/s | 911 kB 00:00 2024-04-23T10:08:24.375 INFO:teuthology.orchestra.run.smithi204.stdout:(69/152): postgresql-private-libs-13.14-1.el9.x 1.7 MB/s | 133 kB 00:00 2024-04-23T10:08:25.079 INFO:teuthology.orchestra.run.smithi119.stdout:(73/152): postgresql-server-13.14-1.el9.x86_64. 710 kB/s | 5.8 MB 00:08 2024-04-23T10:08:25.483 INFO:teuthology.orchestra.run.smithi204.stdout:(70/152): postgresql-server-13.14-1.el9.x86_64. 5.2 MB/s | 5.8 MB 00:01 2024-04-23T10:08:25.717 INFO:teuthology.orchestra.run.smithi204.stdout:(71/152): protobuf-3.14.0-13.el9.x86_64.rpm 4.3 MB/s | 1.0 MB 00:00 2024-04-23T10:08:25.767 INFO:teuthology.orchestra.run.smithi204.stdout:(72/152): python3-attrs-20.3.0-7.el9.noarch.rpm 1.7 MB/s | 87 kB 00:00 2024-04-23T10:08:26.029 INFO:teuthology.orchestra.run.smithi119.stdout:(74/152): python3-devel-3.9.18-3.el9.x86_64.rpm 257 kB/s | 244 kB 00:00 2024-04-23T10:08:26.034 INFO:teuthology.orchestra.run.smithi204.stdout:(73/152): ceph-mgr-diskprediction-local-19.0.0- 259 kB/s | 7.4 MB 00:29 2024-04-23T10:08:26.234 INFO:teuthology.orchestra.run.smithi204.stdout:(74/152): python3-devel-3.9.18-3.el9.x86_64.rpm 1.2 MB/s | 244 kB 00:00 2024-04-23T10:08:26.351 INFO:teuthology.orchestra.run.smithi204.stdout:(75/152): python3-jinja2-2.11.3-5.el9.noarch.rp 2.1 MB/s | 248 kB 00:00 2024-04-23T10:08:26.402 INFO:teuthology.orchestra.run.smithi204.stdout:(76/152): python3-jsonpatch-1.21-16.el9.noarch. 511 kB/s | 26 kB 00:00 2024-04-23T10:08:26.452 INFO:teuthology.orchestra.run.smithi204.stdout:(77/152): python3-jsonpointer-2.0-4.el9.noarch. 385 kB/s | 19 kB 00:00 2024-04-23T10:08:26.535 INFO:teuthology.orchestra.run.smithi204.stdout:(78/152): python3-mako-1.1.4-6.el9.noarch.rpm 2.0 MB/s | 172 kB 00:00 2024-04-23T10:08:26.586 INFO:teuthology.orchestra.run.smithi204.stdout:(79/152): python3-markupsafe-1.1.1-12.el9.x86_6 694 kB/s | 35 kB 00:00 2024-04-23T10:08:26.904 INFO:teuthology.orchestra.run.smithi119.stdout:(75/152): python3-jinja2-2.11.3-5.el9.noarch.rp 283 kB/s | 248 kB 00:00 2024-04-23T10:08:27.345 INFO:teuthology.orchestra.run.smithi204.stdout:(80/152): python3-babel-2.9.1-2.el9.noarch.rpm 3.8 MB/s | 6.0 MB 00:01 2024-04-23T10:08:27.437 INFO:teuthology.orchestra.run.smithi204.stdout:(81/152): python3-numpy-f2py-1.20.1-5.el9.x86_6 2.7 MB/s | 250 kB 00:00 2024-04-23T10:08:27.505 INFO:teuthology.orchestra.run.smithi119.stdout:(76/152): python3-jsonpatch-1.21-16.el9.noarch. 43 kB/s | 26 kB 00:00 2024-04-23T10:08:27.528 INFO:teuthology.orchestra.run.smithi204.stdout:(82/152): python3-oauthlib-3.1.1-5.el9.noarch.r 2.4 MB/s | 222 kB 00:00 2024-04-23T10:08:27.587 INFO:teuthology.orchestra.run.smithi204.stdout:(83/152): python3-packaging-20.9-5.el9.noarch.r 1.3 MB/s | 77 kB 00:00 2024-04-23T10:08:27.646 INFO:teuthology.orchestra.run.smithi204.stdout:(84/152): python3-prettytable-0.7.2-27.el9.noar 727 kB/s | 42 kB 00:00 2024-04-23T10:08:27.730 INFO:teuthology.orchestra.run.smithi204.stdout:(85/152): python3-protobuf-3.14.0-13.el9.noarch 3.2 MB/s | 269 kB 00:00 2024-04-23T10:08:27.805 INFO:teuthology.orchestra.run.smithi204.stdout:(86/152): python3-numpy-1.20.1-5.el9.x86_64.rpm 4.2 MB/s | 5.1 MB 00:01 2024-04-23T10:08:27.830 INFO:teuthology.orchestra.run.smithi204.stdout:(87/152): python3-pyasn1-0.4.8-6.el9.noarch.rpm 1.5 MB/s | 159 kB 00:00 2024-04-23T10:08:27.881 INFO:teuthology.orchestra.run.smithi204.stdout:(88/152): python3-pytz-2021.1-5.el9.noarch.rpm 1.0 MB/s | 51 kB 00:00 2024-04-23T10:08:27.906 INFO:teuthology.orchestra.run.smithi204.stdout:(89/152): python3-pyasn1-modules-0.4.8-6.el9.no 2.7 MB/s | 279 kB 00:00 2024-04-23T10:08:27.931 INFO:teuthology.orchestra.run.smithi204.stdout:(90/152): python3-requests-oauthlib-1.3.0-12.el 1.0 MB/s | 54 kB 00:00 2024-04-23T10:08:27.981 INFO:teuthology.orchestra.run.smithi204.stdout:(91/152): python3-toml-0.10.2-6.el9.noarch.rpm 835 kB/s | 42 kB 00:00 2024-04-23T10:08:28.032 INFO:teuthology.orchestra.run.smithi204.stdout:(92/152): python3-wcwidth-0.2.5-8.el9.noarch.rp 882 kB/s | 44 kB 00:00 2024-04-23T10:08:28.055 INFO:teuthology.orchestra.run.smithi119.stdout:(77/152): python3-jsonpointer-2.0-4.el9.noarch. 35 kB/s | 19 kB 00:00 2024-04-23T10:08:28.130 INFO:teuthology.orchestra.run.smithi119.stdout:(78/152): python3-babel-2.9.1-2.el9.noarch.rpm 704 kB/s | 6.0 MB 00:08 2024-04-23T10:08:28.149 INFO:teuthology.orchestra.run.smithi204.stdout:(93/152): qatlib-24.02.0-1.el9.x86_64.rpm 1.8 MB/s | 221 kB 00:00 2024-04-23T10:08:28.199 INFO:teuthology.orchestra.run.smithi204.stdout:(94/152): qatlib-service-24.02.0-1.el9.x86_64.r 719 kB/s | 36 kB 00:00 2024-04-23T10:08:28.249 INFO:teuthology.orchestra.run.smithi204.stdout:(95/152): qatzip-libs-1.2.0-1.el9.x86_64.rpm 950 kB/s | 47 kB 00:00 2024-04-23T10:08:28.375 INFO:teuthology.orchestra.run.smithi204.stdout:(96/152): socat-1.7.4.1-5.el9.x86_64.rpm 2.4 MB/s | 305 kB 00:00 2024-04-23T10:08:28.425 INFO:teuthology.orchestra.run.smithi204.stdout:(97/152): uuid-1.6.2-55.el9.x86_64.rpm 1.1 MB/s | 58 kB 00:00 2024-04-23T10:08:28.475 INFO:teuthology.orchestra.run.smithi204.stdout:(98/152): xfsprogs-devel-6.3.0-1.el9.x86_64.rpm 1.8 MB/s | 94 kB 00:00 2024-04-23T10:08:28.526 INFO:teuthology.orchestra.run.smithi204.stdout:(99/152): xmlstarlet-1.6.1-20.el9.x86_64.rpm 1.2 MB/s | 64 kB 00:00 2024-04-23T10:08:28.568 INFO:teuthology.orchestra.run.smithi204.stdout:(100/152): zlib-devel-1.2.11-41.el9.x86_64.rpm 1.1 MB/s | 45 kB 00:00 2024-04-23T10:08:28.772 INFO:teuthology.orchestra.run.smithi119.stdout:(79/152): python3-markupsafe-1.1.1-12.el9.x86_6 54 kB/s | 35 kB 00:00 2024-04-23T10:08:28.843 INFO:teuthology.orchestra.run.smithi204.stdout:(101/152): lua-devel-5.4.4-4.el9.x86_64.rpm 81 kB/s | 22 kB 00:00 2024-04-23T10:08:28.847 INFO:teuthology.orchestra.run.smithi119.stdout:(80/152): python3-mako-1.1.4-6.el9.noarch.rpm 217 kB/s | 172 kB 00:00 2024-04-23T10:08:29.789 INFO:teuthology.orchestra.run.smithi119.stdout:(81/152): python3-numpy-f2py-1.20.1-5.el9.x86_6 266 kB/s | 250 kB 00:00 2024-04-23T10:08:30.310 INFO:teuthology.orchestra.run.smithi204.stdout:(102/152): python3-scipy-1.6.2-8.el9.x86_64.rpm 6.8 MB/s | 16 MB 00:02 2024-04-23T10:08:30.527 INFO:teuthology.orchestra.run.smithi204.stdout:(103/152): protobuf-compiler-3.14.0-13.el9.x86_ 513 kB/s | 863 kB 00:01 2024-04-23T10:08:30.635 INFO:teuthology.orchestra.run.smithi204.stdout:(104/152): abseil-cpp-20211102.0-3.el9.x86_64.r 1.6 MB/s | 548 kB 00:00 2024-04-23T10:08:30.681 INFO:teuthology.orchestra.run.smithi119.stdout:(82/152): python3-oauthlib-3.1.1-5.el9.noarch.r 249 kB/s | 222 kB 00:00 2024-04-23T10:08:30.686 INFO:teuthology.orchestra.run.smithi204.stdout:(105/152): grpc-data-1.46.7-10.el9.noarch.rpm 389 kB/s | 19 kB 00:00 2024-04-23T10:08:30.802 INFO:teuthology.orchestra.run.smithi204.stdout:(106/152): gperftools-libs-2.9.1-2.el9.x86_64.r 1.1 MB/s | 309 kB 00:00 2024-04-23T10:08:30.861 INFO:teuthology.orchestra.run.smithi204.stdout:(107/152): libarrow-doc-9.0.0-10.el9.noarch.rpm 447 kB/s | 26 kB 00:00 2024-04-23T10:08:30.936 INFO:teuthology.orchestra.run.smithi204.stdout:(108/152): libarrow-9.0.0-10.el9.x86_64.rpm 18 MB/s | 4.4 MB 00:00 2024-04-23T10:08:30.961 INFO:teuthology.orchestra.run.smithi204.stdout:(109/152): liboath-2.6.7-2.el9.x86_64.rpm 484 kB/s | 49 kB 00:00 2024-04-23T10:08:30.995 INFO:teuthology.orchestra.run.smithi204.stdout:(110/152): libunwind-1.6.2-1.el9.x86_64.rpm 1.1 MB/s | 67 kB 00:00 2024-04-23T10:08:31.029 INFO:teuthology.orchestra.run.smithi204.stdout:(111/152): luarocks-3.9.2-1.el9.noarch.rpm 2.2 MB/s | 151 kB 00:00 2024-04-23T10:08:31.063 INFO:teuthology.orchestra.run.smithi204.stdout:(112/152): parquet-libs-9.0.0-10.el9.x86_64.rpm 12 MB/s | 838 kB 00:00 2024-04-23T10:08:31.121 INFO:teuthology.orchestra.run.smithi204.stdout:(113/152): python3-bcrypt-3.2.2-1.el9.x86_64.rp 745 kB/s | 43 kB 00:00 2024-04-23T10:08:31.147 INFO:teuthology.orchestra.run.smithi204.stdout:(114/152): python3-asyncssh-2.13.2-5.el9.noarch 4.6 MB/s | 548 kB 00:00 2024-04-23T10:08:31.180 INFO:teuthology.orchestra.run.smithi204.stdout:(115/152): python3-cachetools-4.2.4-1.el9.noarc 548 kB/s | 32 kB 00:00 2024-04-23T10:08:31.206 INFO:teuthology.orchestra.run.smithi204.stdout:(116/152): python3-certifi-2021.10.8-2.el9.noar 256 kB/s | 15 kB 00:00 2024-04-23T10:08:31.239 INFO:teuthology.orchestra.run.smithi204.stdout:(117/152): python3-cheroot-8.6.0-4.el9.noarch.r 2.9 MB/s | 172 kB 00:00 2024-04-23T10:08:31.281 INFO:teuthology.orchestra.run.smithi204.stdout:(118/152): python3-cherrypy-18.6.1-2.el9.noarch 4.7 MB/s | 358 kB 00:00 2024-04-23T10:08:31.315 INFO:teuthology.orchestra.run.smithi204.stdout:(119/152): python3-cmd2-2.4.2-2.el9.noarch.rpm 3.9 MB/s | 303 kB 00:00 2024-04-23T10:08:31.340 INFO:teuthology.orchestra.run.smithi204.stdout:(120/152): python3-colorama-0.4.4-12.el9.noarch 604 kB/s | 35 kB 00:00 2024-04-23T10:08:31.373 INFO:teuthology.orchestra.run.smithi119.stdout:(83/152): python3-packaging-20.9-5.el9.noarch.r 112 kB/s | 77 kB 00:00 2024-04-23T10:08:31.373 INFO:teuthology.orchestra.run.smithi204.stdout:(121/152): python3-google-auth-2.29.0-1.el9.noa 3.4 MB/s | 206 kB 00:00 2024-04-23T10:08:31.432 INFO:teuthology.orchestra.run.smithi204.stdout:(122/152): python3-grpcio-tools-1.46.7-10.el9.x 2.4 MB/s | 144 kB 00:00 2024-04-23T10:08:31.482 INFO:teuthology.orchestra.run.smithi204.stdout:(123/152): python3-grpcio-1.46.7-10.el9.x86_64. 14 MB/s | 2.0 MB 00:00 2024-04-23T10:08:31.507 INFO:teuthology.orchestra.run.smithi204.stdout:(124/152): python3-jaraco-8.2.1-3.el9.noarch.rp 142 kB/s | 11 kB 00:00 2024-04-23T10:08:31.541 INFO:teuthology.orchestra.run.smithi204.stdout:(125/152): python3-jaraco-classes-3.2.1-5.el9.n 302 kB/s | 18 kB 00:00 2024-04-23T10:08:31.590 INFO:teuthology.orchestra.run.smithi204.stdout:(126/152): python3-jaraco-collections-3.0.0-8.e 284 kB/s | 23 kB 00:00 2024-04-23T10:08:31.617 INFO:teuthology.orchestra.run.smithi204.stdout:(127/152): python3-jaraco-functools-3.5.0-2.el9 258 kB/s | 19 kB 00:00 2024-04-23T10:08:31.642 INFO:teuthology.orchestra.run.smithi204.stdout:(128/152): python3-jaraco-text-3.2.0-6.el9.noar 376 kB/s | 20 kB 00:00 2024-04-23T10:08:31.701 INFO:teuthology.orchestra.run.smithi204.stdout:(129/152): python3-kubernetes-26.1.0-2.el9.noar 12 MB/s | 1.0 MB 00:00 2024-04-23T10:08:31.726 INFO:teuthology.orchestra.run.smithi204.stdout:(130/152): python3-logutils-0.3.5-21.el9.noarch 554 kB/s | 46 kB 00:00 2024-04-23T10:08:31.785 INFO:teuthology.orchestra.run.smithi204.stdout:(131/152): python3-natsort-7.1.1-5.el9.noarch.r 989 kB/s | 58 kB 00:00 2024-04-23T10:08:31.810 INFO:teuthology.orchestra.run.smithi204.stdout:(132/152): python3-more-itertools-8.12.0-2.el9. 724 kB/s | 79 kB 00:00 2024-04-23T10:08:31.844 INFO:teuthology.orchestra.run.smithi204.stdout:(133/152): python3-pecan-1.4.2-3.el9.noarch.rpm 4.5 MB/s | 272 kB 00:00 2024-04-23T10:08:31.869 INFO:teuthology.orchestra.run.smithi204.stdout:(134/152): python3-portend-3.1.0-2.el9.noarch.r 280 kB/s | 16 kB 00:00 2024-04-23T10:08:31.902 INFO:teuthology.orchestra.run.smithi204.stdout:(135/152): python3-pyOpenSSL-21.0.0-1.el9.noarc 1.5 MB/s | 90 kB 00:00 2024-04-23T10:08:31.936 INFO:teuthology.orchestra.run.smithi204.stdout:(136/152): python3-pyperclip-1.8.2-2.el9.noarch 422 kB/s | 28 kB 00:00 2024-04-23T10:08:31.970 INFO:teuthology.orchestra.run.smithi204.stdout:(137/152): python3-repoze-lru-0.7-16.el9.noarch 459 kB/s | 31 kB 00:00 2024-04-23T10:08:31.995 INFO:teuthology.orchestra.run.smithi204.stdout:(138/152): python3-routes-2.5.1-5.el9.noarch.rp 3.1 MB/s | 188 kB 00:00 2024-04-23T10:08:32.007 INFO:teuthology.orchestra.run.smithi119.stdout:(84/152): python3-prettytable-0.7.2-27.el9.noar 67 kB/s | 42 kB 00:00 2024-04-23T10:08:32.029 INFO:teuthology.orchestra.run.smithi204.stdout:(139/152): python3-rsa-4.9-2.el9.noarch.rpm 1.0 MB/s | 59 kB 00:00 2024-04-23T10:08:32.046 INFO:teuthology.orchestra.run.smithi204.stdout:(140/152): python3-tempora-5.0.0-2.el9.noarch.r 710 kB/s | 36 kB 00:00 2024-04-23T10:08:32.079 INFO:teuthology.orchestra.run.smithi204.stdout:(141/152): python3-typing-extensions-4.4.0-2.el 1.0 MB/s | 51 kB 00:00 2024-04-23T10:08:32.105 INFO:teuthology.orchestra.run.smithi204.stdout:(142/152): python3-webob-1.8.7-6.el9.noarch.rpm 3.8 MB/s | 230 kB 00:00 2024-04-23T10:08:32.138 INFO:teuthology.orchestra.run.smithi204.stdout:(143/152): python3-websocket-client-1.2.3-2.el9 1.5 MB/s | 90 kB 00:00 2024-04-23T10:08:32.172 INFO:teuthology.orchestra.run.smithi204.stdout:(144/152): python3-werkzeug-2.0.3-3.el9.1.noarc 6.2 MB/s | 427 kB 00:00 2024-04-23T10:08:32.197 INFO:teuthology.orchestra.run.smithi204.stdout:(145/152): python3-zc-lockfile-2.0-10.el9.noarc 341 kB/s | 20 kB 00:00 2024-04-23T10:08:32.231 INFO:teuthology.orchestra.run.smithi204.stdout:(146/152): re2-20211101-3.el9.x86_64.rpm 3.2 MB/s | 192 kB 00:00 2024-04-23T10:08:32.290 INFO:teuthology.orchestra.run.smithi204.stdout:(147/152): thrift-0.15.0-2.el9.x86_64.rpm 17 MB/s | 1.6 MB 00:00 2024-04-23T10:08:32.998 INFO:teuthology.orchestra.run.smithi119.stdout:(85/152): python3-protobuf-3.14.0-13.el9.noarch 272 kB/s | 269 kB 00:00 2024-04-23T10:08:33.840 INFO:teuthology.orchestra.run.smithi119.stdout:(86/152): python3-pyasn1-0.4.8-6.el9.noarch.rpm 189 kB/s | 159 kB 00:00 2024-04-23T10:08:34.874 INFO:teuthology.orchestra.run.smithi119.stdout:(87/152): python3-pyasn1-modules-0.4.8-6.el9.no 270 kB/s | 279 kB 00:01 2024-04-23T10:08:35.582 INFO:teuthology.orchestra.run.smithi119.stdout:(88/152): python3-pytz-2021.1-5.el9.noarch.rpm 72 kB/s | 51 kB 00:00 2024-04-23T10:08:36.041 INFO:teuthology.orchestra.run.smithi119.stdout:(89/152): python3-numpy-1.20.1-5.el9.x86_64.rpm 718 kB/s | 5.1 MB 00:07 2024-04-23T10:08:36.362 INFO:teuthology.orchestra.run.smithi114.stdout:(148/152): librados2-19.0.0-3156.gd69f13d8.el7. 270 kB/s | 3.4 MB 00:12 2024-04-23T10:08:36.366 INFO:teuthology.orchestra.run.smithi119.stdout:(90/152): python3-requests-oauthlib-1.3.0-12.el 68 kB/s | 54 kB 00:00 2024-04-23T10:08:36.411 INFO:teuthology.orchestra.run.smithi114.stdout:(149/152): librbd1-19.0.0-3156.gd69f13d8.el7.x8 252 kB/s | 3.1 MB 00:12 2024-04-23T10:08:36.537 INFO:teuthology.orchestra.run.smithi114.stdout:(150/152): openssl-3.2.1-1.el9.x86_64.rpm 7.8 MB/s | 1.3 MB 00:00 2024-04-23T10:08:36.645 INFO:teuthology.orchestra.run.smithi114.stdout:(151/152): openssl-libs-3.2.1-1.el9.x86_64.rpm 10 MB/s | 2.4 MB 00:00 2024-04-23T10:08:36.991 INFO:teuthology.orchestra.run.smithi119.stdout:(91/152): python3-toml-0.10.2-6.el9.noarch.rpm 67 kB/s | 42 kB 00:00 2024-04-23T10:08:37.608 INFO:teuthology.orchestra.run.smithi119.stdout:(92/152): python3-wcwidth-0.2.5-8.el9.noarch.rp 72 kB/s | 44 kB 00:00 2024-04-23T10:08:38.517 INFO:teuthology.orchestra.run.smithi119.stdout:(93/152): qatlib-24.02.0-1.el9.x86_64.rpm 243 kB/s | 221 kB 00:00 2024-04-23T10:08:39.125 INFO:teuthology.orchestra.run.smithi119.stdout:(94/152): qatlib-service-24.02.0-1.el9.x86_64.r 59 kB/s | 36 kB 00:00 2024-04-23T10:08:39.817 INFO:teuthology.orchestra.run.smithi119.stdout:(95/152): qatzip-libs-1.2.0-1.el9.x86_64.rpm 69 kB/s | 47 kB 00:00 2024-04-23T10:08:40.331 INFO:teuthology.orchestra.run.smithi204.stdout:(148/152): librbd1-19.0.0-3156.gd69f13d8.el7.x8 400 kB/s | 3.1 MB 00:08 2024-04-23T10:08:40.631 INFO:teuthology.orchestra.run.smithi204.stdout:(149/152): openssl-3.2.1-1.el9.x86_64.rpm 4.5 MB/s | 1.3 MB 00:00 2024-04-23T10:08:40.798 INFO:teuthology.orchestra.run.smithi204.stdout:(150/152): openssl-libs-3.2.1-1.el9.x86_64.rpm 14 MB/s | 2.4 MB 00:00 2024-04-23T10:08:41.167 INFO:teuthology.orchestra.run.smithi119.stdout:(96/152): socat-1.7.4.1-5.el9.x86_64.rpm 226 kB/s | 305 kB 00:01 2024-04-23T10:08:41.859 INFO:teuthology.orchestra.run.smithi119.stdout:(97/152): uuid-1.6.2-55.el9.x86_64.rpm 84 kB/s | 58 kB 00:00 2024-04-23T10:08:42.198 INFO:teuthology.orchestra.run.smithi204.stdout:(151/152): librados2-19.0.0-3156.gd69f13d8.el7. 349 kB/s | 3.4 MB 00:09 2024-04-23T10:08:42.567 INFO:teuthology.orchestra.run.smithi119.stdout:(98/152): xfsprogs-devel-6.3.0-1.el9.x86_64.rpm 132 kB/s | 94 kB 00:00 2024-04-23T10:08:43.226 INFO:teuthology.orchestra.run.smithi119.stdout:(99/152): xmlstarlet-1.6.1-20.el9.x86_64.rpm 97 kB/s | 64 kB 00:00 2024-04-23T10:08:43.851 INFO:teuthology.orchestra.run.smithi119.stdout:(100/152): zlib-devel-1.2.11-41.el9.x86_64.rpm 72 kB/s | 45 kB 00:00 2024-04-23T10:08:43.976 INFO:teuthology.orchestra.run.smithi119.stdout:(101/152): lua-devel-5.4.4-4.el9.x86_64.rpm 179 kB/s | 22 kB 00:00 2024-04-23T10:08:44.127 INFO:teuthology.orchestra.run.smithi119.stdout:(102/152): protobuf-compiler-3.14.0-13.el9.x86_ 5.6 MB/s | 863 kB 00:00 2024-04-23T10:08:44.602 INFO:teuthology.orchestra.run.smithi119.stdout:(103/152): abseil-cpp-20211102.0-3.el9.x86_64.r 1.1 MB/s | 548 kB 00:00 2024-04-23T10:08:44.694 INFO:teuthology.orchestra.run.smithi119.stdout:(104/152): gperftools-libs-2.9.1-2.el9.x86_64.r 3.3 MB/s | 309 kB 00:00 2024-04-23T10:08:44.769 INFO:teuthology.orchestra.run.smithi119.stdout:(105/152): grpc-data-1.46.7-10.el9.noarch.rpm 259 kB/s | 19 kB 00:00 2024-04-23T10:08:45.036 INFO:teuthology.orchestra.run.smithi119.stdout:(106/152): libarrow-9.0.0-10.el9.x86_64.rpm 17 MB/s | 4.4 MB 00:00 2024-04-23T10:08:45.111 INFO:teuthology.orchestra.run.smithi119.stdout:(107/152): libarrow-doc-9.0.0-10.el9.noarch.rpm 348 kB/s | 26 kB 00:00 2024-04-23T10:08:45.187 INFO:teuthology.orchestra.run.smithi119.stdout:(108/152): liboath-2.6.7-2.el9.x86_64.rpm 646 kB/s | 49 kB 00:00 2024-04-23T10:08:45.279 INFO:teuthology.orchestra.run.smithi119.stdout:(109/152): libunwind-1.6.2-1.el9.x86_64.rpm 736 kB/s | 67 kB 00:00 2024-04-23T10:08:45.354 INFO:teuthology.orchestra.run.smithi119.stdout:(110/152): luarocks-3.9.2-1.el9.noarch.rpm 2.0 MB/s | 151 kB 00:00 2024-04-23T10:08:45.438 INFO:teuthology.orchestra.run.smithi119.stdout:(111/152): parquet-libs-9.0.0-10.el9.x86_64.rpm 9.8 MB/s | 838 kB 00:00 2024-04-23T10:08:45.521 INFO:teuthology.orchestra.run.smithi119.stdout:(112/152): python3-asyncssh-2.13.2-5.el9.noarch 6.4 MB/s | 548 kB 00:00 2024-04-23T10:08:45.588 INFO:teuthology.orchestra.run.smithi119.stdout:(113/152): python3-bcrypt-3.2.2-1.el9.x86_64.rp 653 kB/s | 43 kB 00:00 2024-04-23T10:08:45.655 INFO:teuthology.orchestra.run.smithi119.stdout:(114/152): python3-cachetools-4.2.4-1.el9.noarc 482 kB/s | 32 kB 00:00 2024-04-23T10:08:45.722 INFO:teuthology.orchestra.run.smithi119.stdout:(115/152): python3-certifi-2021.10.8-2.el9.noar 225 kB/s | 15 kB 00:00 2024-04-23T10:08:45.798 INFO:teuthology.orchestra.run.smithi119.stdout:(116/152): python3-cheroot-8.6.0-4.el9.noarch.r 2.2 MB/s | 172 kB 00:00 2024-04-23T10:08:45.873 INFO:teuthology.orchestra.run.smithi119.stdout:(117/152): python3-cherrypy-18.6.1-2.el9.noarch 4.7 MB/s | 358 kB 00:00 2024-04-23T10:08:45.948 INFO:teuthology.orchestra.run.smithi119.stdout:(118/152): python3-cmd2-2.4.2-2.el9.noarch.rpm 4.0 MB/s | 303 kB 00:00 2024-04-23T10:08:46.023 INFO:teuthology.orchestra.run.smithi119.stdout:(119/152): python3-colorama-0.4.4-12.el9.noarch 470 kB/s | 35 kB 00:00 2024-04-23T10:08:46.098 INFO:teuthology.orchestra.run.smithi119.stdout:(120/152): python3-google-auth-2.29.0-1.el9.noa 2.7 MB/s | 206 kB 00:00 2024-04-23T10:08:46.199 INFO:teuthology.orchestra.run.smithi119.stdout:(121/152): python3-grpcio-1.46.7-10.el9.x86_64. 20 MB/s | 2.0 MB 00:00 2024-04-23T10:08:46.274 INFO:teuthology.orchestra.run.smithi119.stdout:(122/152): python3-grpcio-tools-1.46.7-10.el9.x 1.9 MB/s | 144 kB 00:00 2024-04-23T10:08:46.349 INFO:teuthology.orchestra.run.smithi119.stdout:(123/152): python3-jaraco-8.2.1-3.el9.noarch.rp 143 kB/s | 11 kB 00:00 2024-04-23T10:08:46.425 INFO:teuthology.orchestra.run.smithi119.stdout:(124/152): python3-jaraco-classes-3.2.1-5.el9.n 236 kB/s | 18 kB 00:00 2024-04-23T10:08:46.500 INFO:teuthology.orchestra.run.smithi119.stdout:(125/152): python3-jaraco-collections-3.0.0-8.e 308 kB/s | 23 kB 00:00 2024-04-23T10:08:46.575 INFO:teuthology.orchestra.run.smithi119.stdout:(126/152): python3-jaraco-functools-3.5.0-2.el9 260 kB/s | 19 kB 00:00 2024-04-23T10:08:46.650 INFO:teuthology.orchestra.run.smithi119.stdout:(127/152): python3-jaraco-text-3.2.0-6.el9.noar 262 kB/s | 20 kB 00:00 2024-04-23T10:08:46.734 INFO:teuthology.orchestra.run.smithi119.stdout:(128/152): python3-kubernetes-26.1.0-2.el9.noar 12 MB/s | 1.0 MB 00:00 2024-04-23T10:08:46.809 INFO:teuthology.orchestra.run.smithi119.stdout:(129/152): python3-logutils-0.3.5-21.el9.noarch 619 kB/s | 46 kB 00:00 2024-04-23T10:08:46.876 INFO:teuthology.orchestra.run.smithi119.stdout:(130/152): python3-more-itertools-8.12.0-2.el9. 1.2 MB/s | 79 kB 00:00 2024-04-23T10:08:46.951 INFO:teuthology.orchestra.run.smithi119.stdout:(131/152): python3-natsort-7.1.1-5.el9.noarch.r 770 kB/s | 58 kB 00:00 2024-04-23T10:08:47.027 INFO:teuthology.orchestra.run.smithi119.stdout:(132/152): python3-pecan-1.4.2-3.el9.noarch.rpm 3.5 MB/s | 272 kB 00:00 2024-04-23T10:08:47.102 INFO:teuthology.orchestra.run.smithi119.stdout:(133/152): python3-portend-3.1.0-2.el9.noarch.r 219 kB/s | 16 kB 00:00 2024-04-23T10:08:47.178 INFO:teuthology.orchestra.run.smithi119.stdout:(134/152): python3-pyOpenSSL-21.0.0-1.el9.noarc 1.2 MB/s | 90 kB 00:00 2024-04-23T10:08:47.253 INFO:teuthology.orchestra.run.smithi119.stdout:(135/152): python3-pyperclip-1.8.2-2.el9.noarch 376 kB/s | 28 kB 00:00 2024-04-23T10:08:47.328 INFO:teuthology.orchestra.run.smithi119.stdout:(136/152): python3-repoze-lru-0.7-16.el9.noarch 411 kB/s | 31 kB 00:00 2024-04-23T10:08:47.412 INFO:teuthology.orchestra.run.smithi119.stdout:(137/152): python3-routes-2.5.1-5.el9.noarch.rp 2.2 MB/s | 188 kB 00:00 2024-04-23T10:08:47.487 INFO:teuthology.orchestra.run.smithi119.stdout:(138/152): python3-rsa-4.9-2.el9.noarch.rpm 789 kB/s | 59 kB 00:00 2024-04-23T10:08:47.563 INFO:teuthology.orchestra.run.smithi119.stdout:(139/152): python3-tempora-5.0.0-2.el9.noarch.r 476 kB/s | 36 kB 00:00 2024-04-23T10:08:47.638 INFO:teuthology.orchestra.run.smithi119.stdout:(140/152): python3-typing-extensions-4.4.0-2.el 680 kB/s | 51 kB 00:00 2024-04-23T10:08:47.713 INFO:teuthology.orchestra.run.smithi119.stdout:(141/152): python3-webob-1.8.7-6.el9.noarch.rpm 3.0 MB/s | 230 kB 00:00 2024-04-23T10:08:47.822 INFO:teuthology.orchestra.run.smithi119.stdout:(142/152): python3-websocket-client-1.2.3-2.el9 829 kB/s | 90 kB 00:00 2024-04-23T10:08:47.947 INFO:teuthology.orchestra.run.smithi119.stdout:(143/152): python3-werkzeug-2.0.3-3.el9.1.noarc 3.3 MB/s | 427 kB 00:00 2024-04-23T10:08:48.023 INFO:teuthology.orchestra.run.smithi119.stdout:(144/152): python3-zc-lockfile-2.0-10.el9.noarc 267 kB/s | 20 kB 00:00 2024-04-23T10:08:48.098 INFO:teuthology.orchestra.run.smithi119.stdout:(145/152): re2-20211101-3.el9.x86_64.rpm 2.5 MB/s | 192 kB 00:00 2024-04-23T10:08:48.198 INFO:teuthology.orchestra.run.smithi119.stdout:(146/152): thrift-0.15.0-2.el9.x86_64.rpm 16 MB/s | 1.6 MB 00:00 2024-04-23T10:08:56.689 INFO:teuthology.orchestra.run.smithi119.stdout:(147/152): librados2-19.0.0-3156.gd69f13d8.el7. 409 kB/s | 3.4 MB 00:08 2024-04-23T10:08:57.147 INFO:teuthology.orchestra.run.smithi119.stdout:(148/152): python3-scipy-1.6.2-8.el9.x86_64.rpm 789 kB/s | 16 MB 00:21 2024-04-23T10:08:57.271 INFO:teuthology.orchestra.run.smithi119.stdout:[MIRROR] openssl-3.2.1-1.el9.x86_64.rpm: Status code: 404 for https://mirror.pilotfiber.com/centos-stream/9-stream/BaseOS/x86_64/os/Packages/openssl-3.2.1-1.el9.x86_64.rpm (IP: 216.158.133.254) 2024-04-23T10:08:57.290 INFO:teuthology.orchestra.run.smithi119.stdout:[MIRROR] openssl-3.2.1-1.el9.x86_64.rpm: Status code: 404 for https://mirror.pilotfiber.com/centos-stream/9-stream/BaseOS/x86_64/os/Packages/openssl-3.2.1-1.el9.x86_64.rpm (IP: 216.158.133.254) 2024-04-23T10:08:58.206 INFO:teuthology.orchestra.run.smithi119.stdout:(149/152): openssl-3.2.1-1.el9.x86_64.rpm 1.3 MB/s | 1.3 MB 00:01 2024-04-23T10:08:58.225 INFO:teuthology.orchestra.run.smithi119.stdout:[MIRROR] openssl-libs-3.2.1-1.el9.x86_64.rpm: Status code: 404 for https://mirror.pilotfiber.com/centos-stream/9-stream/BaseOS/x86_64/os/Packages/openssl-libs-3.2.1-1.el9.x86_64.rpm (IP: 216.158.133.254) 2024-04-23T10:08:58.262 INFO:teuthology.orchestra.run.smithi119.stdout:[MIRROR] openssl-libs-3.2.1-1.el9.x86_64.rpm: Status code: 404 for https://mirror.pilotfiber.com/centos-stream/9-stream/BaseOS/x86_64/os/Packages/openssl-libs-3.2.1-1.el9.x86_64.rpm (IP: 216.158.133.254) 2024-04-23T10:08:58.564 INFO:teuthology.orchestra.run.smithi119.stdout:(150/152): openssl-libs-3.2.1-1.el9.x86_64.rpm 6.7 MB/s | 2.4 MB 00:00 2024-04-23T10:08:59.598 INFO:teuthology.orchestra.run.smithi119.stdout:(151/152): ceph-test-19.0.0-3156.gd69f13d8.el7. 409 kB/s | 49 MB 02:02 2024-04-23T10:09:01.539 INFO:teuthology.orchestra.run.smithi119.stdout:(152/152): librbd1-19.0.0-3156.gd69f13d8.el7.x8 664 kB/s | 3.1 MB 00:04 2024-04-23T10:09:01.544 INFO:teuthology.orchestra.run.smithi119.stdout:-------------------------------------------------------------------------------- 2024-04-23T10:09:01.544 INFO:teuthology.orchestra.run.smithi119.stdout:Total 1.0 MB/s | 222 MB 03:34 2024-04-23T10:09:02.601 INFO:teuthology.orchestra.run.smithi119.stdout:Running transaction check 2024-04-23T10:09:02.744 INFO:teuthology.orchestra.run.smithi119.stdout:Transaction check succeeded. 2024-04-23T10:09:02.744 INFO:teuthology.orchestra.run.smithi119.stdout:Running transaction test 2024-04-23T10:09:04.509 INFO:teuthology.orchestra.run.smithi119.stdout:Transaction test succeeded. 2024-04-23T10:09:04.509 INFO:teuthology.orchestra.run.smithi119.stdout:Running transaction 2024-04-23T10:09:05.463 INFO:teuthology.orchestra.run.smithi204.stdout:(152/152): ceph-test-19.0.0-3156.gd69f13d8.el7. 327 kB/s | 49 MB 02:33 2024-04-23T10:09:05.467 INFO:teuthology.orchestra.run.smithi204.stdout:-------------------------------------------------------------------------------- 2024-04-23T10:09:05.467 INFO:teuthology.orchestra.run.smithi204.stdout:Total 949 kB/s | 222 MB 03:59 2024-04-23T10:09:06.507 INFO:teuthology.orchestra.run.smithi204.stdout:Running transaction check 2024-04-23T10:09:06.648 INFO:teuthology.orchestra.run.smithi204.stdout:Transaction check succeeded. 2024-04-23T10:09:06.648 INFO:teuthology.orchestra.run.smithi204.stdout:Running transaction test 2024-04-23T10:09:08.473 INFO:teuthology.orchestra.run.smithi204.stdout:Transaction test succeeded. 2024-04-23T10:09:08.474 INFO:teuthology.orchestra.run.smithi204.stdout:Running transaction 2024-04-23T10:09:10.465 INFO:teuthology.orchestra.run.smithi119.stdout: Preparing : 1/1 2024-04-23T10:09:10.568 INFO:teuthology.orchestra.run.smithi119.stdout: Upgrading : openssl-libs-1:3.2.1-1.el9.x86_64 1/156 2024-04-23T10:09:10.606 INFO:teuthology.orchestra.run.smithi119.stdout: Installing : librabbitmq-0.11.0-7.el9.x86_64 2/156 2024-04-23T10:09:10.625 INFO:teuthology.orchestra.run.smithi114.stdout:(152/152): ceph-test-19.0.0-3156.gd69f13d8.el7. 373 kB/s | 49 MB 02:14 2024-04-23T10:09:10.628 INFO:teuthology.orchestra.run.smithi114.stdout:-------------------------------------------------------------------------------- 2024-04-23T10:09:10.628 INFO:teuthology.orchestra.run.smithi114.stdout:Total 913 kB/s | 222 MB 04:08 2024-04-23T10:09:10.659 INFO:teuthology.orchestra.run.smithi119.stdout: Installing : librdkafka-1.6.1-102.el9.x86_64 3/156 2024-04-23T10:09:10.710 INFO:teuthology.orchestra.run.smithi119.stdout: Installing : thrift-0.15.0-2.el9.x86_64 4/156 2024-04-23T10:09:10.736 INFO:teuthology.orchestra.run.smithi119.stdout: Installing : python3-more-itertools-8.12.0-2.el9.noarch 5/156 2024-04-23T10:09:10.778 INFO:teuthology.orchestra.run.smithi119.stdout: Installing : python3-jaraco-8.2.1-3.el9.noarch 6/156 2024-04-23T10:09:11.096 INFO:teuthology.orchestra.run.smithi119.stdout: Installing : lttng-ust-2.12.0-6.el9.x86_64 7/156 2024-04-23T10:09:11.220 INFO:teuthology.orchestra.run.smithi119.stdout: Upgrading : librados2-2:19.0.0-3156.gd69f13d8.el7.x86_64 8/156 2024-04-23T10:09:11.349 INFO:teuthology.orchestra.run.smithi119.stdout: Running scriptlet: librados2-2:19.0.0-3156.gd69f13d8.el7.x86_64 8/156 2024-04-23T10:09:11.380 INFO:teuthology.orchestra.run.smithi119.stdout: Installing : libcephfs2-2:19.0.0-3156.gd69f13d8.el7.x86_64 9/156 2024-04-23T10:09:11.470 INFO:teuthology.orchestra.run.smithi119.stdout: Running scriptlet: libcephfs2-2:19.0.0-3156.gd69f13d8.el7.x86_64 9/156 2024-04-23T10:09:11.501 INFO:teuthology.orchestra.run.smithi119.stdout: Installing : python3-rados-2:19.0.0-3156.gd69f13d8.el7.x86_64 10/156 2024-04-23T10:09:11.647 INFO:teuthology.orchestra.run.smithi119.stdout: Installing : postgresql-private-libs-13.14-1.el9.x86_64 11/156 2024-04-23T10:09:11.665 INFO:teuthology.orchestra.run.smithi114.stdout:Running transaction check 2024-04-23T10:09:11.698 INFO:teuthology.orchestra.run.smithi119.stdout: Installing : postgresql-13.14-1.el9.x86_64 12/156 2024-04-23T10:09:11.726 INFO:teuthology.orchestra.run.smithi119.stdout: Installing : libcephsqlite-2:19.0.0-3156.gd69f13d8.el7.x86_64 13/156 2024-04-23T10:09:11.808 INFO:teuthology.orchestra.run.smithi114.stdout:Transaction check succeeded. 2024-04-23T10:09:11.808 INFO:teuthology.orchestra.run.smithi114.stdout:Running transaction test 2024-04-23T10:09:11.828 INFO:teuthology.orchestra.run.smithi119.stdout: Running scriptlet: libcephsqlite-2:19.0.0-3156.gd69f13d8.el7.x86_64 13/156 2024-04-23T10:09:11.852 INFO:teuthology.orchestra.run.smithi119.stdout: Installing : libradosstriper1-2:19.0.0-3156.gd69f13d8.el7.x86 14/156 2024-04-23T10:09:11.914 INFO:teuthology.orchestra.run.smithi119.stdout: Running scriptlet: libradosstriper1-2:19.0.0-3156.gd69f13d8.el7.x86 14/156 2024-04-23T10:09:11.936 INFO:teuthology.orchestra.run.smithi119.stdout: Installing : qatlib-service-24.02.0-1.el9.x86_64 15/156 2024-04-23T10:09:11.956 INFO:teuthology.orchestra.run.smithi119.stdout: Running scriptlet: qatlib-service-24.02.0-1.el9.x86_64 15/156 2024-04-23T10:09:12.103 INFO:teuthology.orchestra.run.smithi119.stdout: Running scriptlet: qatlib-24.02.0-1.el9.x86_64 16/156 2024-04-23T10:09:12.136 INFO:teuthology.orchestra.run.smithi119.stdout: Installing : qatlib-24.02.0-1.el9.x86_64 16/156 2024-04-23T10:09:12.242 INFO:teuthology.orchestra.run.smithi119.stdout: Installing : re2-1:20211101-3.el9.x86_64 17/156 2024-04-23T10:09:12.338 INFO:teuthology.orchestra.run.smithi119.stdout: Installing : libarrow-9.0.0-10.el9.x86_64 18/156 2024-04-23T10:09:12.368 INFO:teuthology.orchestra.run.smithi119.stdout: Installing : python3-werkzeug-2.0.3-3.el9.1.noarch 19/156 2024-04-23T10:09:12.403 INFO:teuthology.orchestra.run.smithi119.stdout: Installing : liboath-2.6.7-2.el9.x86_64 20/156 2024-04-23T10:09:12.468 INFO:teuthology.orchestra.run.smithi119.stdout: Installing : python3-pytz-2021.1-5.el9.noarch 21/156 2024-04-23T10:09:12.510 INFO:teuthology.orchestra.run.smithi119.stdout: Installing : python3-pyasn1-0.4.8-6.el9.noarch 22/156 2024-04-23T10:09:12.554 INFO:teuthology.orchestra.run.smithi119.stdout: Installing : python3-markupsafe-1.1.1-12.el9.x86_64 23/156 2024-04-23T10:09:12.587 INFO:teuthology.orchestra.run.smithi119.stdout: Installing : protobuf-3.14.0-13.el9.x86_64 24/156 2024-04-23T10:09:12.615 INFO:teuthology.orchestra.run.smithi119.stdout: Installing : lua-5.4.4-4.el9.x86_64 25/156 2024-04-23T10:09:12.683 INFO:teuthology.orchestra.run.smithi119.stdout: Installing : flexiblas-3.0.4-8.el9.x86_64 26/156 2024-04-23T10:09:12.737 INFO:teuthology.orchestra.run.smithi119.stdout: Installing : python3-urllib3-1.26.5-5.el9.noarch 27/156 2024-04-23T10:09:12.767 INFO:teuthology.orchestra.run.smithi119.stdout: Installing : python3-requests-2.25.1-8.el9.noarch 28/156 2024-04-23T10:09:12.802 INFO:teuthology.orchestra.run.smithi119.stdout: Installing : libquadmath-11.4.1-3.el9.x86_64 29/156 2024-04-23T10:09:12.869 INFO:teuthology.orchestra.run.smithi119.stdout: Installing : libgfortran-11.4.1-3.el9.x86_64 30/156 2024-04-23T10:09:12.905 INFO:teuthology.orchestra.run.smithi119.stdout: Installing : python3-ceph-common-2:19.0.0-3156.gd69f13d8.el7. 31/156 2024-04-23T10:09:12.948 INFO:teuthology.orchestra.run.smithi119.stdout: Installing : python3-ceph-argparse-2:19.0.0-3156.gd69f13d8.el 32/156 2024-04-23T10:09:12.977 INFO:teuthology.orchestra.run.smithi119.stdout: Installing : python3-cephfs-2:19.0.0-3156.gd69f13d8.el7.x86_6 33/156 2024-04-23T10:09:13.047 INFO:teuthology.orchestra.run.smithi119.stdout: Installing : lua-devel-5.4.4-4.el9.x86_64 34/156 2024-04-23T10:09:13.092 INFO:teuthology.orchestra.run.smithi119.stdout: Installing : luarocks-3.9.2-1.el9.noarch 35/156 2024-04-23T10:09:13.167 INFO:teuthology.orchestra.run.smithi119.stdout: Installing : protobuf-compiler-3.14.0-13.el9.x86_64 36/156 2024-04-23T10:09:13.296 INFO:teuthology.orchestra.run.smithi119.stdout: Installing : python3-mako-1.1.4-6.el9.noarch 37/156 2024-04-23T10:09:13.344 INFO:teuthology.orchestra.run.smithi119.stdout: Installing : python3-pyasn1-modules-0.4.8-6.el9.noarch 38/156 2024-04-23T10:09:13.581 INFO:teuthology.orchestra.run.smithi114.stdout:Transaction test succeeded. 2024-04-23T10:09:13.582 INFO:teuthology.orchestra.run.smithi114.stdout:Running transaction 2024-04-23T10:09:13.854 INFO:teuthology.orchestra.run.smithi119.stdout: Installing : python3-rsa-4.9-2.el9.noarch 39/156 2024-04-23T10:09:13.931 INFO:teuthology.orchestra.run.smithi119.stdout: Installing : python3-babel-2.9.1-2.el9.noarch 40/156 2024-04-23T10:09:13.960 INFO:teuthology.orchestra.run.smithi119.stdout: Installing : python3-jinja2-2.11.3-5.el9.noarch 41/156 2024-04-23T10:09:13.995 INFO:teuthology.orchestra.run.smithi119.stdout: Installing : qatzip-libs-1.2.0-1.el9.x86_64 42/156 2024-04-23T10:09:14.033 INFO:teuthology.orchestra.run.smithi119.stdout: Installing : librados-devel-2:19.0.0-3156.gd69f13d8.el7.x86_6 43/156 2024-04-23T10:09:14.074 INFO:teuthology.orchestra.run.smithi119.stdout: Installing : python3-jaraco-classes-3.2.1-5.el9.noarch 44/156 2024-04-23T10:09:14.223 INFO:teuthology.orchestra.run.smithi119.stdout: Installing : socat-1.7.4.1-5.el9.x86_64 45/156 2024-04-23T10:09:14.286 INFO:teuthology.orchestra.run.smithi119.stdout: Upgrading : openssl-1:3.2.1-1.el9.x86_64 46/156 2024-04-23T10:09:14.340 INFO:teuthology.orchestra.run.smithi119.stdout: Installing : python3-zc-lockfile-2.0-10.el9.noarch 47/156 2024-04-23T10:09:14.404 INFO:teuthology.orchestra.run.smithi204.stdout: Preparing : 1/1 2024-04-23T10:09:14.404 INFO:teuthology.orchestra.run.smithi119.stdout: Installing : python3-websocket-client-1.2.3-2.el9.noarch 48/156 2024-04-23T10:09:14.437 INFO:teuthology.orchestra.run.smithi119.stdout: Installing : python3-webob-1.8.7-6.el9.noarch 49/156 2024-04-23T10:09:14.473 INFO:teuthology.orchestra.run.smithi119.stdout: Installing : python3-typing-extensions-4.4.0-2.el9.noarch 50/156 2024-04-23T10:09:14.491 INFO:teuthology.orchestra.run.smithi204.stdout: Upgrading : openssl-libs-1:3.2.1-1.el9.x86_64 1/156 2024-04-23T10:09:14.519 INFO:teuthology.orchestra.run.smithi119.stdout: Installing : python3-repoze-lru-0.7-16.el9.noarch 51/156 2024-04-23T10:09:14.530 INFO:teuthology.orchestra.run.smithi204.stdout: Installing : librabbitmq-0.11.0-7.el9.x86_64 2/156 2024-04-23T10:09:14.556 INFO:teuthology.orchestra.run.smithi119.stdout: Installing : python3-routes-2.5.1-5.el9.noarch 52/156 2024-04-23T10:09:14.583 INFO:teuthology.orchestra.run.smithi204.stdout: Installing : librdkafka-1.6.1-102.el9.x86_64 3/156 2024-04-23T10:09:14.598 INFO:teuthology.orchestra.run.smithi119.stdout: Installing : python3-pyperclip-1.8.2-2.el9.noarch 53/156 2024-04-23T10:09:14.626 INFO:teuthology.orchestra.run.smithi204.stdout: Installing : thrift-0.15.0-2.el9.x86_64 4/156 2024-04-23T10:09:14.640 INFO:teuthology.orchestra.run.smithi119.stdout: Installing : python3-natsort-7.1.1-5.el9.noarch 54/156 2024-04-23T10:09:14.651 INFO:teuthology.orchestra.run.smithi204.stdout: Installing : python3-more-itertools-8.12.0-2.el9.noarch 5/156 2024-04-23T10:09:14.694 INFO:teuthology.orchestra.run.smithi204.stdout: Installing : python3-jaraco-8.2.1-3.el9.noarch 6/156 2024-04-23T10:09:14.775 INFO:teuthology.orchestra.run.smithi119.stdout: Installing : python3-logutils-0.3.5-21.el9.noarch 55/156 2024-04-23T10:09:14.821 INFO:teuthology.orchestra.run.smithi119.stdout: Installing : python3-pecan-1.4.2-3.el9.noarch 56/156 2024-04-23T10:09:14.852 INFO:teuthology.orchestra.run.smithi119.stdout: Installing : python3-colorama-0.4.4-12.el9.noarch 57/156 2024-04-23T10:09:14.893 INFO:teuthology.orchestra.run.smithi119.stdout: Installing : python3-certifi-2021.10.8-2.el9.noarch 58/156 2024-04-23T10:09:14.986 INFO:teuthology.orchestra.run.smithi119.stdout: Installing : python3-cachetools-4.2.4-1.el9.noarch 59/156 2024-04-23T10:09:15.014 INFO:teuthology.orchestra.run.smithi204.stdout: Installing : lttng-ust-2.12.0-6.el9.x86_64 7/156 2024-04-23T10:09:15.018 INFO:teuthology.orchestra.run.smithi119.stdout: Installing : python3-google-auth-1:2.29.0-1.el9.noarch 60/156 2024-04-23T10:09:15.067 INFO:teuthology.orchestra.run.smithi119.stdout: Installing : libunwind-1.6.2-1.el9.x86_64 61/156 2024-04-23T10:09:15.093 INFO:teuthology.orchestra.run.smithi119.stdout: Installing : gperftools-libs-2.9.1-2.el9.x86_64 62/156 2024-04-23T10:09:15.117 INFO:teuthology.orchestra.run.smithi119.stdout: Installing : libarrow-doc-9.0.0-10.el9.noarch 63/156 2024-04-23T10:09:15.136 INFO:teuthology.orchestra.run.smithi204.stdout: Upgrading : librados2-2:19.0.0-3156.gd69f13d8.el7.x86_64 8/156 2024-04-23T10:09:15.190 INFO:teuthology.orchestra.run.smithi119.stdout: Installing : grpc-data-1.46.7-10.el9.noarch 64/156 2024-04-23T10:09:15.257 INFO:teuthology.orchestra.run.smithi204.stdout: Running scriptlet: librados2-2:19.0.0-3156.gd69f13d8.el7.x86_64 8/156 2024-04-23T10:09:15.287 INFO:teuthology.orchestra.run.smithi204.stdout: Installing : libcephfs2-2:19.0.0-3156.gd69f13d8.el7.x86_64 9/156 2024-04-23T10:09:15.317 INFO:teuthology.orchestra.run.smithi119.stdout: Installing : abseil-cpp-20211102.0-3.el9.x86_64 65/156 2024-04-23T10:09:15.350 INFO:teuthology.orchestra.run.smithi119.stdout: Installing : python3-grpcio-1.46.7-10.el9.x86_64 66/156 2024-04-23T10:09:15.378 INFO:teuthology.orchestra.run.smithi119.stdout: Installing : zlib-devel-1.2.11-41.el9.x86_64 67/156 2024-04-23T10:09:15.378 INFO:teuthology.orchestra.run.smithi204.stdout: Running scriptlet: libcephfs2-2:19.0.0-3156.gd69f13d8.el7.x86_64 9/156 2024-04-23T10:09:15.412 INFO:teuthology.orchestra.run.smithi119.stdout: Installing : xmlstarlet-1.6.1-20.el9.x86_64 68/156 2024-04-23T10:09:15.416 INFO:teuthology.orchestra.run.smithi204.stdout: Installing : python3-rados-2:19.0.0-3156.gd69f13d8.el7.x86_64 10/156 2024-04-23T10:09:15.451 INFO:teuthology.orchestra.run.smithi119.stdout: Installing : uuid-1.6.2-55.el9.x86_64 69/156 2024-04-23T10:09:15.482 INFO:teuthology.orchestra.run.smithi119.stdout: Installing : python3-wcwidth-0.2.5-8.el9.noarch 70/156 2024-04-23T10:09:15.513 INFO:teuthology.orchestra.run.smithi119.stdout: Installing : python3-toml-0.10.2-6.el9.noarch 71/156 2024-04-23T10:09:15.539 INFO:teuthology.orchestra.run.smithi119.stdout: Installing : python3-jaraco-functools-3.5.0-2.el9.noarch 72/156 2024-04-23T10:09:15.564 INFO:teuthology.orchestra.run.smithi204.stdout: Installing : postgresql-private-libs-13.14-1.el9.x86_64 11/156 2024-04-23T10:09:15.572 INFO:teuthology.orchestra.run.smithi119.stdout: Installing : python3-jaraco-text-3.2.0-6.el9.noarch 73/156 2024-04-23T10:09:15.613 INFO:teuthology.orchestra.run.smithi119.stdout: Installing : python3-jaraco-collections-3.0.0-8.el9.noarch 74/156 2024-04-23T10:09:15.614 INFO:teuthology.orchestra.run.smithi204.stdout: Installing : postgresql-13.14-1.el9.x86_64 12/156 2024-04-23T10:09:15.642 INFO:teuthology.orchestra.run.smithi204.stdout: Installing : libcephsqlite-2:19.0.0-3156.gd69f13d8.el7.x86_64 13/156 2024-04-23T10:09:15.649 INFO:teuthology.orchestra.run.smithi119.stdout: Installing : python3-tempora-5.0.0-2.el9.noarch 75/156 2024-04-23T10:09:15.734 INFO:teuthology.orchestra.run.smithi119.stdout: Installing : python3-portend-3.1.0-2.el9.noarch 76/156 2024-04-23T10:09:15.744 INFO:teuthology.orchestra.run.smithi204.stdout: Running scriptlet: libcephsqlite-2:19.0.0-3156.gd69f13d8.el7.x86_64 13/156 2024-04-23T10:09:15.768 INFO:teuthology.orchestra.run.smithi204.stdout: Installing : libradosstriper1-2:19.0.0-3156.gd69f13d8.el7.x86 14/156 2024-04-23T10:09:15.778 INFO:teuthology.orchestra.run.smithi119.stdout: Installing : python3-protobuf-3.14.0-13.el9.noarch 77/156 2024-04-23T10:09:15.810 INFO:teuthology.orchestra.run.smithi119.stdout: Installing : python3-grpcio-tools-1.46.7-10.el9.x86_64 78/156 2024-04-23T10:09:15.830 INFO:teuthology.orchestra.run.smithi204.stdout: Running scriptlet: libradosstriper1-2:19.0.0-3156.gd69f13d8.el7.x86 14/156 2024-04-23T10:09:15.852 INFO:teuthology.orchestra.run.smithi204.stdout: Installing : qatlib-service-24.02.0-1.el9.x86_64 15/156 2024-04-23T10:09:15.856 INFO:teuthology.orchestra.run.smithi119.stdout: Installing : python3-prettytable-0.7.2-27.el9.noarch 79/156 2024-04-23T10:09:15.873 INFO:teuthology.orchestra.run.smithi204.stdout: Running scriptlet: qatlib-service-24.02.0-1.el9.x86_64 15/156 2024-04-23T10:09:15.954 INFO:teuthology.orchestra.run.smithi119.stdout: Installing : python3-packaging-20.9-5.el9.noarch 80/156 2024-04-23T10:09:16.011 INFO:teuthology.orchestra.run.smithi119.stdout: Installing : python3-oauthlib-3.1.1-5.el9.noarch 81/156 2024-04-23T10:09:16.069 INFO:teuthology.orchestra.run.smithi204.stdout: Running scriptlet: qatlib-24.02.0-1.el9.x86_64 16/156 2024-04-23T10:09:16.102 INFO:teuthology.orchestra.run.smithi204.stdout: Installing : qatlib-24.02.0-1.el9.x86_64 16/156 2024-04-23T10:09:16.209 INFO:teuthology.orchestra.run.smithi204.stdout: Installing : re2-1:20211101-3.el9.x86_64 17/156 2024-04-23T10:09:16.296 INFO:teuthology.orchestra.run.smithi204.stdout: Installing : libarrow-9.0.0-10.el9.x86_64 18/156 2024-04-23T10:09:16.334 INFO:teuthology.orchestra.run.smithi204.stdout: Installing : python3-werkzeug-2.0.3-3.el9.1.noarch 19/156 2024-04-23T10:09:16.378 INFO:teuthology.orchestra.run.smithi204.stdout: Installing : liboath-2.6.7-2.el9.x86_64 20/156 2024-04-23T10:09:16.467 INFO:teuthology.orchestra.run.smithi204.stdout: Installing : python3-pytz-2021.1-5.el9.noarch 21/156 2024-04-23T10:09:16.501 INFO:teuthology.orchestra.run.smithi204.stdout: Installing : python3-pyasn1-0.4.8-6.el9.noarch 22/156 2024-04-23T10:09:16.545 INFO:teuthology.orchestra.run.smithi204.stdout: Installing : python3-markupsafe-1.1.1-12.el9.x86_64 23/156 2024-04-23T10:09:16.586 INFO:teuthology.orchestra.run.smithi204.stdout: Installing : protobuf-3.14.0-13.el9.x86_64 24/156 2024-04-23T10:09:16.622 INFO:teuthology.orchestra.run.smithi204.stdout: Installing : lua-5.4.4-4.el9.x86_64 25/156 2024-04-23T10:09:16.691 INFO:teuthology.orchestra.run.smithi204.stdout: Installing : flexiblas-3.0.4-8.el9.x86_64 26/156 2024-04-23T10:09:16.719 INFO:teuthology.orchestra.run.smithi119.stdout: Installing : python3-requests-oauthlib-1.3.0-12.el9.noarch 82/156 2024-04-23T10:09:16.744 INFO:teuthology.orchestra.run.smithi204.stdout: Installing : python3-urllib3-1.26.5-5.el9.noarch 27/156 2024-04-23T10:09:16.758 INFO:teuthology.orchestra.run.smithi119.stdout: Installing : python3-kubernetes-1:26.1.0-2.el9.noarch 83/156 2024-04-23T10:09:16.775 INFO:teuthology.orchestra.run.smithi204.stdout: Installing : python3-requests-2.25.1-8.el9.noarch 28/156 2024-04-23T10:09:16.792 INFO:teuthology.orchestra.run.smithi119.stdout: Installing : python3-jsonpointer-2.0-4.el9.noarch 84/156 2024-04-23T10:09:16.811 INFO:teuthology.orchestra.run.smithi204.stdout: Installing : libquadmath-11.4.1-3.el9.x86_64 29/156 2024-04-23T10:09:16.886 INFO:teuthology.orchestra.run.smithi204.stdout: Installing : libgfortran-11.4.1-3.el9.x86_64 30/156 2024-04-23T10:09:16.889 INFO:teuthology.orchestra.run.smithi119.stdout: Installing : python3-jsonpatch-1.21-16.el9.noarch 85/156 2024-04-23T10:09:16.922 INFO:teuthology.orchestra.run.smithi204.stdout: Installing : python3-ceph-common-2:19.0.0-3156.gd69f13d8.el7. 31/156 2024-04-23T10:09:16.941 INFO:teuthology.orchestra.run.smithi119.stdout: Installing : python3-devel-3.9.18-3.el9.x86_64 86/156 2024-04-23T10:09:16.965 INFO:teuthology.orchestra.run.smithi204.stdout: Installing : python3-ceph-argparse-2:19.0.0-3156.gd69f13d8.el 32/156 2024-04-23T10:09:16.994 INFO:teuthology.orchestra.run.smithi204.stdout: Installing : python3-cephfs-2:19.0.0-3156.gd69f13d8.el7.x86_6 33/156 2024-04-23T10:09:17.021 INFO:teuthology.orchestra.run.smithi119.stdout: Installing : python3-attrs-20.3.0-7.el9.noarch 87/156 2024-04-23T10:09:17.053 INFO:teuthology.orchestra.run.smithi119.stdout: Installing : python3-cmd2-2.4.2-2.el9.noarch 88/156 2024-04-23T10:09:17.071 INFO:teuthology.orchestra.run.smithi204.stdout: Installing : lua-devel-5.4.4-4.el9.x86_64 34/156 2024-04-23T10:09:17.125 INFO:teuthology.orchestra.run.smithi204.stdout: Installing : luarocks-3.9.2-1.el9.noarch 35/156 2024-04-23T10:09:17.199 INFO:teuthology.orchestra.run.smithi204.stdout: Installing : protobuf-compiler-3.14.0-13.el9.x86_64 36/156 2024-04-23T10:09:17.229 INFO:teuthology.orchestra.run.smithi119.stdout: Installing : openblas-0.3.21-2.el9.x86_64 89/156 2024-04-23T10:09:17.259 INFO:teuthology.orchestra.run.smithi119.stdout: Installing : openblas-openmp-0.3.21-2.el9.x86_64 90/156 2024-04-23T10:09:17.331 INFO:teuthology.orchestra.run.smithi204.stdout: Installing : python3-mako-1.1.4-6.el9.noarch 37/156 2024-04-23T10:09:17.335 INFO:teuthology.orchestra.run.smithi119.stdout: Installing : flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 91/156 2024-04-23T10:09:17.385 INFO:teuthology.orchestra.run.smithi204.stdout: Installing : python3-pyasn1-modules-0.4.8-6.el9.noarch 38/156 2024-04-23T10:09:17.909 INFO:teuthology.orchestra.run.smithi204.stdout: Installing : python3-rsa-4.9-2.el9.noarch 39/156 2024-04-23T10:09:17.970 INFO:teuthology.orchestra.run.smithi119.stdout: Installing : flexiblas-netlib-3.0.4-8.el9.x86_64 92/156 2024-04-23T10:09:17.998 INFO:teuthology.orchestra.run.smithi204.stdout: Installing : python3-babel-2.9.1-2.el9.noarch 40/156 2024-04-23T10:09:18.028 INFO:teuthology.orchestra.run.smithi204.stdout: Installing : python3-jinja2-2.11.3-5.el9.noarch 41/156 2024-04-23T10:09:18.070 INFO:teuthology.orchestra.run.smithi204.stdout: Installing : qatzip-libs-1.2.0-1.el9.x86_64 42/156 2024-04-23T10:09:18.089 INFO:teuthology.orchestra.run.smithi119.stdout: Installing : python3-numpy-1:1.20.1-5.el9.x86_64 93/156 2024-04-23T10:09:18.117 INFO:teuthology.orchestra.run.smithi204.stdout: Installing : librados-devel-2:19.0.0-3156.gd69f13d8.el7.x86_6 43/156 2024-04-23T10:09:18.167 INFO:teuthology.orchestra.run.smithi204.stdout: Installing : python3-jaraco-classes-3.2.1-5.el9.noarch 44/156 2024-04-23T10:09:18.325 INFO:teuthology.orchestra.run.smithi204.stdout: Installing : socat-1.7.4.1-5.el9.x86_64 45/156 2024-04-23T10:09:18.395 INFO:teuthology.orchestra.run.smithi204.stdout: Upgrading : openssl-1:3.2.1-1.el9.x86_64 46/156 2024-04-23T10:09:18.450 INFO:teuthology.orchestra.run.smithi204.stdout: Installing : python3-zc-lockfile-2.0-10.el9.noarch 47/156 2024-04-23T10:09:18.522 INFO:teuthology.orchestra.run.smithi204.stdout: Installing : python3-websocket-client-1.2.3-2.el9.noarch 48/156 2024-04-23T10:09:18.554 INFO:teuthology.orchestra.run.smithi204.stdout: Installing : python3-webob-1.8.7-6.el9.noarch 49/156 2024-04-23T10:09:18.590 INFO:teuthology.orchestra.run.smithi204.stdout: Installing : python3-typing-extensions-4.4.0-2.el9.noarch 50/156 2024-04-23T10:09:18.637 INFO:teuthology.orchestra.run.smithi204.stdout: Installing : python3-repoze-lru-0.7-16.el9.noarch 51/156 2024-04-23T10:09:18.682 INFO:teuthology.orchestra.run.smithi204.stdout: Installing : python3-routes-2.5.1-5.el9.noarch 52/156 2024-04-23T10:09:18.733 INFO:teuthology.orchestra.run.smithi204.stdout: Installing : python3-pyperclip-1.8.2-2.el9.noarch 53/156 2024-04-23T10:09:18.775 INFO:teuthology.orchestra.run.smithi204.stdout: Installing : python3-natsort-7.1.1-5.el9.noarch 54/156 2024-04-23T10:09:18.911 INFO:teuthology.orchestra.run.smithi204.stdout: Installing : python3-logutils-0.3.5-21.el9.noarch 55/156 2024-04-23T10:09:18.956 INFO:teuthology.orchestra.run.smithi204.stdout: Installing : python3-pecan-1.4.2-3.el9.noarch 56/156 2024-04-23T10:09:18.987 INFO:teuthology.orchestra.run.smithi204.stdout: Installing : python3-colorama-0.4.4-12.el9.noarch 57/156 2024-04-23T10:09:19.028 INFO:teuthology.orchestra.run.smithi204.stdout: Installing : python3-certifi-2021.10.8-2.el9.noarch 58/156 2024-04-23T10:09:19.121 INFO:teuthology.orchestra.run.smithi204.stdout: Installing : python3-cachetools-4.2.4-1.el9.noarch 59/156 2024-04-23T10:09:19.153 INFO:teuthology.orchestra.run.smithi204.stdout: Installing : python3-google-auth-1:2.29.0-1.el9.noarch 60/156 2024-04-23T10:09:19.193 INFO:teuthology.orchestra.run.smithi204.stdout: Installing : libunwind-1.6.2-1.el9.x86_64 61/156 2024-04-23T10:09:19.227 INFO:teuthology.orchestra.run.smithi204.stdout: Installing : gperftools-libs-2.9.1-2.el9.x86_64 62/156 2024-04-23T10:09:19.252 INFO:teuthology.orchestra.run.smithi204.stdout: Installing : libarrow-doc-9.0.0-10.el9.noarch 63/156 2024-04-23T10:09:19.300 INFO:teuthology.orchestra.run.smithi119.stdout: Installing : python3-numpy-f2py-1:1.20.1-5.el9.x86_64 94/156 2024-04-23T10:09:19.326 INFO:teuthology.orchestra.run.smithi204.stdout: Installing : grpc-data-1.46.7-10.el9.noarch 64/156 2024-04-23T10:09:19.388 INFO:teuthology.orchestra.run.smithi119.stdout: Installing : python3-scipy-1.6.2-8.el9.x86_64 95/156 2024-04-23T10:09:19.416 INFO:teuthology.orchestra.run.smithi119.stdout: Installing : libzstd-devel-1.5.1-2.el9.x86_64 96/156 2024-04-23T10:09:19.450 INFO:teuthology.orchestra.run.smithi119.stdout: Installing : libuuid-devel-2.37.4-18.el9.x86_64 97/156 2024-04-23T10:09:19.454 INFO:teuthology.orchestra.run.smithi204.stdout: Installing : abseil-cpp-20211102.0-3.el9.x86_64 65/156 2024-04-23T10:09:19.484 INFO:teuthology.orchestra.run.smithi204.stdout: Installing : python3-grpcio-1.46.7-10.el9.x86_64 66/156 2024-04-23T10:09:19.512 INFO:teuthology.orchestra.run.smithi204.stdout: Installing : zlib-devel-1.2.11-41.el9.x86_64 67/156 2024-04-23T10:09:19.518 INFO:teuthology.orchestra.run.smithi114.stdout: Preparing : 1/1 2024-04-23T10:09:19.547 INFO:teuthology.orchestra.run.smithi204.stdout: Installing : xmlstarlet-1.6.1-20.el9.x86_64 68/156 2024-04-23T10:09:19.586 INFO:teuthology.orchestra.run.smithi204.stdout: Installing : uuid-1.6.2-55.el9.x86_64 69/156 2024-04-23T10:09:19.612 INFO:teuthology.orchestra.run.smithi114.stdout: Upgrading : openssl-libs-1:3.2.1-1.el9.x86_64 1/156 2024-04-23T10:09:19.618 INFO:teuthology.orchestra.run.smithi204.stdout: Installing : python3-wcwidth-0.2.5-8.el9.noarch 70/156 2024-04-23T10:09:19.647 INFO:teuthology.orchestra.run.smithi204.stdout: Installing : python3-toml-0.10.2-6.el9.noarch 71/156 2024-04-23T10:09:19.650 INFO:teuthology.orchestra.run.smithi114.stdout: Installing : librabbitmq-0.11.0-7.el9.x86_64 2/156 2024-04-23T10:09:19.682 INFO:teuthology.orchestra.run.smithi204.stdout: Installing : python3-jaraco-functools-3.5.0-2.el9.noarch 72/156 2024-04-23T10:09:19.705 INFO:teuthology.orchestra.run.smithi114.stdout: Installing : librdkafka-1.6.1-102.el9.x86_64 3/156 2024-04-23T10:09:19.724 INFO:teuthology.orchestra.run.smithi204.stdout: Installing : python3-jaraco-text-3.2.0-6.el9.noarch 73/156 2024-04-23T10:09:19.739 INFO:teuthology.orchestra.run.smithi114.stdout: Installing : thrift-0.15.0-2.el9.x86_64 4/156 2024-04-23T10:09:19.759 INFO:teuthology.orchestra.run.smithi119.stdout: Installing : libpmemobj-1.12.1-1.el9.x86_64 98/156 2024-04-23T10:09:19.765 INFO:teuthology.orchestra.run.smithi114.stdout: Installing : python3-more-itertools-8.12.0-2.el9.noarch 5/156 2024-04-23T10:09:19.774 INFO:teuthology.orchestra.run.smithi204.stdout: Installing : python3-jaraco-collections-3.0.0-8.el9.noarch 74/156 2024-04-23T10:09:19.804 INFO:teuthology.orchestra.run.smithi119.stdout: Upgrading : librbd1-2:19.0.0-3156.gd69f13d8.el7.x86_64 99/156 2024-04-23T10:09:19.807 INFO:teuthology.orchestra.run.smithi114.stdout: Installing : python3-jaraco-8.2.1-3.el9.noarch 6/156 2024-04-23T10:09:19.808 INFO:teuthology.orchestra.run.smithi204.stdout: Installing : python3-tempora-5.0.0-2.el9.noarch 75/156 2024-04-23T10:09:19.885 INFO:teuthology.orchestra.run.smithi119.stdout: Running scriptlet: librbd1-2:19.0.0-3156.gd69f13d8.el7.x86_64 99/156 2024-04-23T10:09:19.911 INFO:teuthology.orchestra.run.smithi204.stdout: Installing : python3-portend-3.1.0-2.el9.noarch 76/156 2024-04-23T10:09:19.917 INFO:teuthology.orchestra.run.smithi119.stdout: Installing : python3-rbd-2:19.0.0-3156.gd69f13d8.el7.x86_64 100/156 2024-04-23T10:09:19.953 INFO:teuthology.orchestra.run.smithi119.stdout: Installing : boost-program-options-1.75.0-8.el9.x86_64 101/156 2024-04-23T10:09:19.955 INFO:teuthology.orchestra.run.smithi204.stdout: Installing : python3-protobuf-3.14.0-13.el9.noarch 77/156 2024-04-23T10:09:19.986 INFO:teuthology.orchestra.run.smithi204.stdout: Installing : python3-grpcio-tools-1.46.7-10.el9.x86_64 78/156 2024-04-23T10:09:20.034 INFO:teuthology.orchestra.run.smithi204.stdout: Installing : python3-prettytable-0.7.2-27.el9.noarch 79/156 2024-04-23T10:09:20.131 INFO:teuthology.orchestra.run.smithi204.stdout: Installing : python3-packaging-20.9-5.el9.noarch 80/156 2024-04-23T10:09:20.134 INFO:teuthology.orchestra.run.smithi114.stdout: Installing : lttng-ust-2.12.0-6.el9.x86_64 7/156 2024-04-23T10:09:20.188 INFO:teuthology.orchestra.run.smithi204.stdout: Installing : python3-oauthlib-3.1.1-5.el9.noarch 81/156 2024-04-23T10:09:20.249 INFO:teuthology.orchestra.run.smithi114.stdout: Upgrading : librados2-2:19.0.0-3156.gd69f13d8.el7.x86_64 8/156 2024-04-23T10:09:20.369 INFO:teuthology.orchestra.run.smithi114.stdout: Running scriptlet: librados2-2:19.0.0-3156.gd69f13d8.el7.x86_64 8/156 2024-04-23T10:09:20.408 INFO:teuthology.orchestra.run.smithi114.stdout: Installing : libcephfs2-2:19.0.0-3156.gd69f13d8.el7.x86_64 9/156 2024-04-23T10:09:20.429 INFO:teuthology.orchestra.run.smithi119.stdout: Installing : parquet-libs-9.0.0-10.el9.x86_64 102/156 2024-04-23T10:09:20.457 INFO:teuthology.orchestra.run.smithi119.stdout: Installing : librgw2-2:19.0.0-3156.gd69f13d8.el7.x86_64 103/156 2024-04-23T10:09:20.499 INFO:teuthology.orchestra.run.smithi114.stdout: Running scriptlet: libcephfs2-2:19.0.0-3156.gd69f13d8.el7.x86_64 9/156 2024-04-23T10:09:20.528 INFO:teuthology.orchestra.run.smithi119.stdout: Running scriptlet: librgw2-2:19.0.0-3156.gd69f13d8.el7.x86_64 103/156 2024-04-23T10:09:20.538 INFO:teuthology.orchestra.run.smithi114.stdout: Installing : python3-rados-2:19.0.0-3156.gd69f13d8.el7.x86_64 10/156 2024-04-23T10:09:20.550 INFO:teuthology.orchestra.run.smithi119.stdout: Installing : python3-rgw-2:19.0.0-3156.gd69f13d8.el7.x86_64 104/156 2024-04-23T10:09:20.683 INFO:teuthology.orchestra.run.smithi114.stdout: Installing : postgresql-private-libs-13.14-1.el9.x86_64 11/156 2024-04-23T10:09:20.734 INFO:teuthology.orchestra.run.smithi114.stdout: Installing : postgresql-13.14-1.el9.x86_64 12/156 2024-04-23T10:09:20.763 INFO:teuthology.orchestra.run.smithi114.stdout: Installing : libcephsqlite-2:19.0.0-3156.gd69f13d8.el7.x86_64 13/156 2024-04-23T10:09:20.866 INFO:teuthology.orchestra.run.smithi114.stdout: Running scriptlet: libcephsqlite-2:19.0.0-3156.gd69f13d8.el7.x86_64 13/156 2024-04-23T10:09:20.889 INFO:teuthology.orchestra.run.smithi114.stdout: Installing : libradosstriper1-2:19.0.0-3156.gd69f13d8.el7.x86 14/156 2024-04-23T10:09:20.902 INFO:teuthology.orchestra.run.smithi204.stdout: Installing : python3-requests-oauthlib-1.3.0-12.el9.noarch 82/156 2024-04-23T10:09:20.944 INFO:teuthology.orchestra.run.smithi204.stdout: Installing : python3-kubernetes-1:26.1.0-2.el9.noarch 83/156 2024-04-23T10:09:20.951 INFO:teuthology.orchestra.run.smithi114.stdout: Running scriptlet: libradosstriper1-2:19.0.0-3156.gd69f13d8.el7.x86 14/156 2024-04-23T10:09:20.973 INFO:teuthology.orchestra.run.smithi114.stdout: Installing : qatlib-service-24.02.0-1.el9.x86_64 15/156 2024-04-23T10:09:20.977 INFO:teuthology.orchestra.run.smithi204.stdout: Installing : python3-jsonpointer-2.0-4.el9.noarch 84/156 2024-04-23T10:09:20.992 INFO:teuthology.orchestra.run.smithi114.stdout: Running scriptlet: qatlib-service-24.02.0-1.el9.x86_64 15/156 2024-04-23T10:09:21.078 INFO:teuthology.orchestra.run.smithi204.stdout: Installing : python3-jsonpatch-1.21-16.el9.noarch 85/156 2024-04-23T10:09:21.127 INFO:teuthology.orchestra.run.smithi204.stdout: Installing : python3-devel-3.9.18-3.el9.x86_64 86/156 2024-04-23T10:09:21.139 INFO:teuthology.orchestra.run.smithi114.stdout: Running scriptlet: qatlib-24.02.0-1.el9.x86_64 16/156 2024-04-23T10:09:21.173 INFO:teuthology.orchestra.run.smithi114.stdout: Installing : qatlib-24.02.0-1.el9.x86_64 16/156 2024-04-23T10:09:21.223 INFO:teuthology.orchestra.run.smithi204.stdout: Installing : python3-attrs-20.3.0-7.el9.noarch 87/156 2024-04-23T10:09:21.263 INFO:teuthology.orchestra.run.smithi204.stdout: Installing : python3-cmd2-2.4.2-2.el9.noarch 88/156 2024-04-23T10:09:21.280 INFO:teuthology.orchestra.run.smithi114.stdout: Installing : re2-1:20211101-3.el9.x86_64 17/156 2024-04-23T10:09:21.375 INFO:teuthology.orchestra.run.smithi114.stdout: Installing : libarrow-9.0.0-10.el9.x86_64 18/156 2024-04-23T10:09:21.404 INFO:teuthology.orchestra.run.smithi114.stdout: Installing : python3-werkzeug-2.0.3-3.el9.1.noarch 19/156 2024-04-23T10:09:21.439 INFO:teuthology.orchestra.run.smithi114.stdout: Installing : liboath-2.6.7-2.el9.x86_64 20/156 2024-04-23T10:09:21.440 INFO:teuthology.orchestra.run.smithi204.stdout: Installing : openblas-0.3.21-2.el9.x86_64 89/156 2024-04-23T10:09:21.469 INFO:teuthology.orchestra.run.smithi204.stdout: Installing : openblas-openmp-0.3.21-2.el9.x86_64 90/156 2024-04-23T10:09:21.507 INFO:teuthology.orchestra.run.smithi114.stdout: Installing : python3-pytz-2021.1-5.el9.noarch 21/156 2024-04-23T10:09:21.547 INFO:teuthology.orchestra.run.smithi204.stdout: Installing : flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 91/156 2024-04-23T10:09:21.547 INFO:teuthology.orchestra.run.smithi114.stdout: Installing : python3-pyasn1-0.4.8-6.el9.noarch 22/156 2024-04-23T10:09:21.591 INFO:teuthology.orchestra.run.smithi114.stdout: Installing : python3-markupsafe-1.1.1-12.el9.x86_64 23/156 2024-04-23T10:09:21.631 INFO:teuthology.orchestra.run.smithi114.stdout: Installing : protobuf-3.14.0-13.el9.x86_64 24/156 2024-04-23T10:09:21.659 INFO:teuthology.orchestra.run.smithi114.stdout: Installing : lua-5.4.4-4.el9.x86_64 25/156 2024-04-23T10:09:21.727 INFO:teuthology.orchestra.run.smithi114.stdout: Installing : flexiblas-3.0.4-8.el9.x86_64 26/156 2024-04-23T10:09:21.781 INFO:teuthology.orchestra.run.smithi114.stdout: Installing : python3-urllib3-1.26.5-5.el9.noarch 27/156 2024-04-23T10:09:21.812 INFO:teuthology.orchestra.run.smithi114.stdout: Installing : python3-requests-2.25.1-8.el9.noarch 28/156 2024-04-23T10:09:21.847 INFO:teuthology.orchestra.run.smithi114.stdout: Installing : libquadmath-11.4.1-3.el9.x86_64 29/156 2024-04-23T10:09:21.922 INFO:teuthology.orchestra.run.smithi114.stdout: Installing : libgfortran-11.4.1-3.el9.x86_64 30/156 2024-04-23T10:09:21.957 INFO:teuthology.orchestra.run.smithi114.stdout: Installing : python3-ceph-common-2:19.0.0-3156.gd69f13d8.el7. 31/156 2024-04-23T10:09:22.000 INFO:teuthology.orchestra.run.smithi114.stdout: Installing : python3-ceph-argparse-2:19.0.0-3156.gd69f13d8.el 32/156 2024-04-23T10:09:22.021 INFO:teuthology.orchestra.run.smithi114.stdout: Installing : python3-cephfs-2:19.0.0-3156.gd69f13d8.el7.x86_6 33/156 2024-04-23T10:09:22.091 INFO:teuthology.orchestra.run.smithi114.stdout: Installing : lua-devel-5.4.4-4.el9.x86_64 34/156 2024-04-23T10:09:22.136 INFO:teuthology.orchestra.run.smithi114.stdout: Installing : luarocks-3.9.2-1.el9.noarch 35/156 2024-04-23T10:09:22.186 INFO:teuthology.orchestra.run.smithi204.stdout: Installing : flexiblas-netlib-3.0.4-8.el9.x86_64 92/156 2024-04-23T10:09:22.210 INFO:teuthology.orchestra.run.smithi114.stdout: Installing : protobuf-compiler-3.14.0-13.el9.x86_64 36/156 2024-04-23T10:09:22.300 INFO:teuthology.orchestra.run.smithi204.stdout: Installing : python3-numpy-1:1.20.1-5.el9.x86_64 93/156 2024-04-23T10:09:22.351 INFO:teuthology.orchestra.run.smithi114.stdout: Installing : python3-mako-1.1.4-6.el9.noarch 37/156 2024-04-23T10:09:22.412 INFO:teuthology.orchestra.run.smithi114.stdout: Installing : python3-pyasn1-modules-0.4.8-6.el9.noarch 38/156 2024-04-23T10:09:22.837 INFO:teuthology.orchestra.run.smithi119.stdout: Running scriptlet: ceph-common-2:19.0.0-3156.gd69f13d8.el7.x86_64 105/156 2024-04-23T10:09:22.872 INFO:teuthology.orchestra.run.smithi119.stdout: Installing : ceph-common-2:19.0.0-3156.gd69f13d8.el7.x86_64 105/156 2024-04-23T10:09:22.930 INFO:teuthology.orchestra.run.smithi114.stdout: Installing : python3-rsa-4.9-2.el9.noarch 39/156 2024-04-23T10:09:23.008 INFO:teuthology.orchestra.run.smithi114.stdout: Installing : python3-babel-2.9.1-2.el9.noarch 40/156 2024-04-23T10:09:23.038 INFO:teuthology.orchestra.run.smithi114.stdout: Installing : python3-jinja2-2.11.3-5.el9.noarch 41/156 2024-04-23T10:09:23.080 INFO:teuthology.orchestra.run.smithi114.stdout: Installing : qatzip-libs-1.2.0-1.el9.x86_64 42/156 2024-04-23T10:09:23.127 INFO:teuthology.orchestra.run.smithi114.stdout: Installing : librados-devel-2:19.0.0-3156.gd69f13d8.el7.x86_6 43/156 2024-04-23T10:09:23.168 INFO:teuthology.orchestra.run.smithi114.stdout: Installing : python3-jaraco-classes-3.2.1-5.el9.noarch 44/156 2024-04-23T10:09:23.328 INFO:teuthology.orchestra.run.smithi114.stdout: Installing : socat-1.7.4.1-5.el9.x86_64 45/156 2024-04-23T10:09:23.371 INFO:teuthology.orchestra.run.smithi114.stdout: Upgrading : openssl-1:3.2.1-1.el9.x86_64 46/156 2024-04-23T10:09:23.411 INFO:teuthology.orchestra.run.smithi119.stdout: Running scriptlet: ceph-common-2:19.0.0-3156.gd69f13d8.el7.x86_64 105/156 2024-04-23T10:09:23.426 INFO:teuthology.orchestra.run.smithi114.stdout: Installing : python3-zc-lockfile-2.0-10.el9.noarch 47/156 2024-04-23T10:09:23.459 INFO:teuthology.orchestra.run.smithi119.stdout: Installing : ceph-base-2:19.0.0-3156.gd69f13d8.el7.x86_64 106/156 2024-04-23T10:09:23.498 INFO:teuthology.orchestra.run.smithi114.stdout: Installing : python3-websocket-client-1.2.3-2.el9.noarch 48/156 2024-04-23T10:09:23.517 INFO:teuthology.orchestra.run.smithi204.stdout: Installing : python3-numpy-f2py-1:1.20.1-5.el9.x86_64 94/156 2024-04-23T10:09:23.530 INFO:teuthology.orchestra.run.smithi114.stdout: Installing : python3-webob-1.8.7-6.el9.noarch 49/156 2024-04-23T10:09:23.549 INFO:teuthology.orchestra.run.smithi119.stdout: Running scriptlet: ceph-base-2:19.0.0-3156.gd69f13d8.el7.x86_64 106/156 2024-04-23T10:09:23.549 INFO:teuthology.orchestra.run.smithi119.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph.target → /usr/lib/systemd/system/ceph.target. 2024-04-23T10:09:23.549 INFO:teuthology.orchestra.run.smithi119.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-crash.service → /usr/lib/systemd/system/ceph-crash.service. 2024-04-23T10:09:23.549 INFO:teuthology.orchestra.run.smithi119.stdout: 2024-04-23T10:09:23.567 INFO:teuthology.orchestra.run.smithi114.stdout: Installing : python3-typing-extensions-4.4.0-2.el9.noarch 50/156 2024-04-23T10:09:23.585 INFO:teuthology.orchestra.run.smithi119.stdout: Installing : ceph-selinux-2:19.0.0-3156.gd69f13d8.el7.x86_64 107/156 2024-04-23T10:09:23.608 INFO:teuthology.orchestra.run.smithi204.stdout: Installing : python3-scipy-1.6.2-8.el9.x86_64 95/156 2024-04-23T10:09:23.613 INFO:teuthology.orchestra.run.smithi114.stdout: Installing : python3-repoze-lru-0.7-16.el9.noarch 51/156 2024-04-23T10:09:23.636 INFO:teuthology.orchestra.run.smithi204.stdout: Installing : libzstd-devel-1.5.1-2.el9.x86_64 96/156 2024-04-23T10:09:23.649 INFO:teuthology.orchestra.run.smithi114.stdout: Installing : python3-routes-2.5.1-5.el9.noarch 52/156 2024-04-23T10:09:23.669 INFO:teuthology.orchestra.run.smithi204.stdout: Installing : libuuid-devel-2.37.4-18.el9.x86_64 97/156 2024-04-23T10:09:23.692 INFO:teuthology.orchestra.run.smithi114.stdout: Installing : python3-pyperclip-1.8.2-2.el9.noarch 53/156 2024-04-23T10:09:23.742 INFO:teuthology.orchestra.run.smithi114.stdout: Installing : python3-natsort-7.1.1-5.el9.noarch 54/156 2024-04-23T10:09:23.878 INFO:teuthology.orchestra.run.smithi114.stdout: Installing : python3-logutils-0.3.5-21.el9.noarch 55/156 2024-04-23T10:09:23.931 INFO:teuthology.orchestra.run.smithi114.stdout: Installing : python3-pecan-1.4.2-3.el9.noarch 56/156 2024-04-23T10:09:23.962 INFO:teuthology.orchestra.run.smithi114.stdout: Installing : python3-colorama-0.4.4-12.el9.noarch 57/156 2024-04-23T10:09:23.980 INFO:teuthology.orchestra.run.smithi204.stdout: Installing : libpmemobj-1.12.1-1.el9.x86_64 98/156 2024-04-23T10:09:24.003 INFO:teuthology.orchestra.run.smithi114.stdout: Installing : python3-certifi-2021.10.8-2.el9.noarch 58/156 2024-04-23T10:09:24.024 INFO:teuthology.orchestra.run.smithi204.stdout: Upgrading : librbd1-2:19.0.0-3156.gd69f13d8.el7.x86_64 99/156 2024-04-23T10:09:24.096 INFO:teuthology.orchestra.run.smithi114.stdout: Installing : python3-cachetools-4.2.4-1.el9.noarch 59/156 2024-04-23T10:09:24.114 INFO:teuthology.orchestra.run.smithi204.stdout: Running scriptlet: librbd1-2:19.0.0-3156.gd69f13d8.el7.x86_64 99/156 2024-04-23T10:09:24.128 INFO:teuthology.orchestra.run.smithi114.stdout: Installing : python3-google-auth-1:2.29.0-1.el9.noarch 60/156 2024-04-23T10:09:24.145 INFO:teuthology.orchestra.run.smithi204.stdout: Installing : python3-rbd-2:19.0.0-3156.gd69f13d8.el7.x86_64 100/156 2024-04-23T10:09:24.168 INFO:teuthology.orchestra.run.smithi114.stdout: Installing : libunwind-1.6.2-1.el9.x86_64 61/156 2024-04-23T10:09:24.181 INFO:teuthology.orchestra.run.smithi204.stdout: Installing : boost-program-options-1.75.0-8.el9.x86_64 101/156 2024-04-23T10:09:24.194 INFO:teuthology.orchestra.run.smithi114.stdout: Installing : gperftools-libs-2.9.1-2.el9.x86_64 62/156 2024-04-23T10:09:24.218 INFO:teuthology.orchestra.run.smithi114.stdout: Installing : libarrow-doc-9.0.0-10.el9.noarch 63/156 2024-04-23T10:09:24.291 INFO:teuthology.orchestra.run.smithi114.stdout: Installing : grpc-data-1.46.7-10.el9.noarch 64/156 2024-04-23T10:09:24.419 INFO:teuthology.orchestra.run.smithi114.stdout: Installing : abseil-cpp-20211102.0-3.el9.x86_64 65/156 2024-04-23T10:09:24.459 INFO:teuthology.orchestra.run.smithi114.stdout: Installing : python3-grpcio-1.46.7-10.el9.x86_64 66/156 2024-04-23T10:09:24.487 INFO:teuthology.orchestra.run.smithi114.stdout: Installing : zlib-devel-1.2.11-41.el9.x86_64 67/156 2024-04-23T10:09:24.521 INFO:teuthology.orchestra.run.smithi114.stdout: Installing : xmlstarlet-1.6.1-20.el9.x86_64 68/156 2024-04-23T10:09:24.560 INFO:teuthology.orchestra.run.smithi114.stdout: Installing : uuid-1.6.2-55.el9.x86_64 69/156 2024-04-23T10:09:24.592 INFO:teuthology.orchestra.run.smithi114.stdout: Installing : python3-wcwidth-0.2.5-8.el9.noarch 70/156 2024-04-23T10:09:24.621 INFO:teuthology.orchestra.run.smithi114.stdout: Installing : python3-toml-0.10.2-6.el9.noarch 71/156 2024-04-23T10:09:24.656 INFO:teuthology.orchestra.run.smithi114.stdout: Installing : python3-jaraco-functools-3.5.0-2.el9.noarch 72/156 2024-04-23T10:09:24.668 INFO:teuthology.orchestra.run.smithi204.stdout: Installing : parquet-libs-9.0.0-10.el9.x86_64 102/156 2024-04-23T10:09:24.689 INFO:teuthology.orchestra.run.smithi114.stdout: Installing : python3-jaraco-text-3.2.0-6.el9.noarch 73/156 2024-04-23T10:09:24.693 INFO:teuthology.orchestra.run.smithi204.stdout: Installing : librgw2-2:19.0.0-3156.gd69f13d8.el7.x86_64 103/156 2024-04-23T10:09:24.731 INFO:teuthology.orchestra.run.smithi114.stdout: Installing : python3-jaraco-collections-3.0.0-8.el9.noarch 74/156 2024-04-23T10:09:24.764 INFO:teuthology.orchestra.run.smithi204.stdout: Running scriptlet: librgw2-2:19.0.0-3156.gd69f13d8.el7.x86_64 103/156 2024-04-23T10:09:24.765 INFO:teuthology.orchestra.run.smithi114.stdout: Installing : python3-tempora-5.0.0-2.el9.noarch 75/156 2024-04-23T10:09:24.787 INFO:teuthology.orchestra.run.smithi204.stdout: Installing : python3-rgw-2:19.0.0-3156.gd69f13d8.el7.x86_64 104/156 2024-04-23T10:09:24.851 INFO:teuthology.orchestra.run.smithi114.stdout: Installing : python3-portend-3.1.0-2.el9.noarch 76/156 2024-04-23T10:09:24.895 INFO:teuthology.orchestra.run.smithi114.stdout: Installing : python3-protobuf-3.14.0-13.el9.noarch 77/156 2024-04-23T10:09:24.926 INFO:teuthology.orchestra.run.smithi114.stdout: Installing : python3-grpcio-tools-1.46.7-10.el9.x86_64 78/156 2024-04-23T10:09:24.973 INFO:teuthology.orchestra.run.smithi114.stdout: Installing : python3-prettytable-0.7.2-27.el9.noarch 79/156 2024-04-23T10:09:25.071 INFO:teuthology.orchestra.run.smithi114.stdout: Installing : python3-packaging-20.9-5.el9.noarch 80/156 2024-04-23T10:09:25.128 INFO:teuthology.orchestra.run.smithi114.stdout: Installing : python3-oauthlib-3.1.1-5.el9.noarch 81/156 2024-04-23T10:09:25.836 INFO:teuthology.orchestra.run.smithi114.stdout: Installing : python3-requests-oauthlib-1.3.0-12.el9.noarch 82/156 2024-04-23T10:09:25.875 INFO:teuthology.orchestra.run.smithi114.stdout: Installing : python3-kubernetes-1:26.1.0-2.el9.noarch 83/156 2024-04-23T10:09:25.908 INFO:teuthology.orchestra.run.smithi114.stdout: Installing : python3-jsonpointer-2.0-4.el9.noarch 84/156 2024-04-23T10:09:26.005 INFO:teuthology.orchestra.run.smithi114.stdout: Installing : python3-jsonpatch-1.21-16.el9.noarch 85/156 2024-04-23T10:09:26.057 INFO:teuthology.orchestra.run.smithi114.stdout: Installing : python3-devel-3.9.18-3.el9.x86_64 86/156 2024-04-23T10:09:26.135 INFO:teuthology.orchestra.run.smithi114.stdout: Installing : python3-attrs-20.3.0-7.el9.noarch 87/156 2024-04-23T10:09:26.177 INFO:teuthology.orchestra.run.smithi114.stdout: Installing : python3-cmd2-2.4.2-2.el9.noarch 88/156 2024-04-23T10:09:26.355 INFO:teuthology.orchestra.run.smithi114.stdout: Installing : openblas-0.3.21-2.el9.x86_64 89/156 2024-04-23T10:09:26.383 INFO:teuthology.orchestra.run.smithi114.stdout: Installing : openblas-openmp-0.3.21-2.el9.x86_64 90/156 2024-04-23T10:09:26.460 INFO:teuthology.orchestra.run.smithi114.stdout: Installing : flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 91/156 2024-04-23T10:09:27.040 INFO:teuthology.orchestra.run.smithi204.stdout: Running scriptlet: ceph-common-2:19.0.0-3156.gd69f13d8.el7.x86_64 105/156 2024-04-23T10:09:27.075 INFO:teuthology.orchestra.run.smithi204.stdout: Installing : ceph-common-2:19.0.0-3156.gd69f13d8.el7.x86_64 105/156 2024-04-23T10:09:27.099 INFO:teuthology.orchestra.run.smithi114.stdout: Installing : flexiblas-netlib-3.0.4-8.el9.x86_64 92/156 2024-04-23T10:09:27.213 INFO:teuthology.orchestra.run.smithi114.stdout: Installing : python3-numpy-1:1.20.1-5.el9.x86_64 93/156 2024-04-23T10:09:27.612 INFO:teuthology.orchestra.run.smithi204.stdout: Running scriptlet: ceph-common-2:19.0.0-3156.gd69f13d8.el7.x86_64 105/156 2024-04-23T10:09:27.662 INFO:teuthology.orchestra.run.smithi204.stdout: Installing : ceph-base-2:19.0.0-3156.gd69f13d8.el7.x86_64 106/156 2024-04-23T10:09:27.747 INFO:teuthology.orchestra.run.smithi204.stdout: Running scriptlet: ceph-base-2:19.0.0-3156.gd69f13d8.el7.x86_64 106/156 2024-04-23T10:09:27.747 INFO:teuthology.orchestra.run.smithi204.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph.target → /usr/lib/systemd/system/ceph.target. 2024-04-23T10:09:27.747 INFO:teuthology.orchestra.run.smithi204.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-crash.service → /usr/lib/systemd/system/ceph-crash.service. 2024-04-23T10:09:27.747 INFO:teuthology.orchestra.run.smithi204.stdout: 2024-04-23T10:09:27.780 INFO:teuthology.orchestra.run.smithi204.stdout: Installing : ceph-selinux-2:19.0.0-3156.gd69f13d8.el7.x86_64 107/156 2024-04-23T10:09:28.428 INFO:teuthology.orchestra.run.smithi114.stdout: Installing : python3-numpy-f2py-1:1.20.1-5.el9.x86_64 94/156 2024-04-23T10:09:28.520 INFO:teuthology.orchestra.run.smithi114.stdout: Installing : python3-scipy-1.6.2-8.el9.x86_64 95/156 2024-04-23T10:09:28.548 INFO:teuthology.orchestra.run.smithi114.stdout: Installing : libzstd-devel-1.5.1-2.el9.x86_64 96/156 2024-04-23T10:09:28.581 INFO:teuthology.orchestra.run.smithi114.stdout: Installing : libuuid-devel-2.37.4-18.el9.x86_64 97/156 2024-04-23T10:09:28.893 INFO:teuthology.orchestra.run.smithi114.stdout: Installing : libpmemobj-1.12.1-1.el9.x86_64 98/156 2024-04-23T10:09:28.936 INFO:teuthology.orchestra.run.smithi114.stdout: Upgrading : librbd1-2:19.0.0-3156.gd69f13d8.el7.x86_64 99/156 2024-04-23T10:09:29.026 INFO:teuthology.orchestra.run.smithi114.stdout: Running scriptlet: librbd1-2:19.0.0-3156.gd69f13d8.el7.x86_64 99/156 2024-04-23T10:09:29.049 INFO:teuthology.orchestra.run.smithi114.stdout: Installing : python3-rbd-2:19.0.0-3156.gd69f13d8.el7.x86_64 100/156 2024-04-23T10:09:29.084 INFO:teuthology.orchestra.run.smithi114.stdout: Installing : boost-program-options-1.75.0-8.el9.x86_64 101/156 2024-04-23T10:09:29.563 INFO:teuthology.orchestra.run.smithi114.stdout: Installing : parquet-libs-9.0.0-10.el9.x86_64 102/156 2024-04-23T10:09:29.605 INFO:teuthology.orchestra.run.smithi114.stdout: Installing : librgw2-2:19.0.0-3156.gd69f13d8.el7.x86_64 103/156 2024-04-23T10:09:29.668 INFO:teuthology.orchestra.run.smithi114.stdout: Running scriptlet: librgw2-2:19.0.0-3156.gd69f13d8.el7.x86_64 103/156 2024-04-23T10:09:29.690 INFO:teuthology.orchestra.run.smithi114.stdout: Installing : python3-rgw-2:19.0.0-3156.gd69f13d8.el7.x86_64 104/156 2024-04-23T10:09:31.929 INFO:teuthology.orchestra.run.smithi114.stdout: Running scriptlet: ceph-common-2:19.0.0-3156.gd69f13d8.el7.x86_64 105/156 2024-04-23T10:09:31.969 INFO:teuthology.orchestra.run.smithi114.stdout: Installing : ceph-common-2:19.0.0-3156.gd69f13d8.el7.x86_64 105/156 2024-04-23T10:09:32.510 INFO:teuthology.orchestra.run.smithi114.stdout: Running scriptlet: ceph-common-2:19.0.0-3156.gd69f13d8.el7.x86_64 105/156 2024-04-23T10:09:32.548 INFO:teuthology.orchestra.run.smithi114.stdout: Installing : ceph-base-2:19.0.0-3156.gd69f13d8.el7.x86_64 106/156 2024-04-23T10:09:32.623 INFO:teuthology.orchestra.run.smithi114.stdout: Running scriptlet: ceph-base-2:19.0.0-3156.gd69f13d8.el7.x86_64 106/156 2024-04-23T10:09:32.623 INFO:teuthology.orchestra.run.smithi114.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph.target → /usr/lib/systemd/system/ceph.target. 2024-04-23T10:09:32.623 INFO:teuthology.orchestra.run.smithi114.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-crash.service → /usr/lib/systemd/system/ceph-crash.service. 2024-04-23T10:09:32.623 INFO:teuthology.orchestra.run.smithi114.stdout: 2024-04-23T10:09:32.649 INFO:teuthology.orchestra.run.smithi114.stdout: Installing : ceph-selinux-2:19.0.0-3156.gd69f13d8.el7.x86_64 107/156 2024-04-23T10:09:34.884 INFO:teuthology.orchestra.run.smithi119.stdout: Running scriptlet: ceph-selinux-2:19.0.0-3156.gd69f13d8.el7.x86_64 107/156 2024-04-23T10:09:34.884 INFO:teuthology.orchestra.run.smithi119.stdout:skipping the directory /sys 2024-04-23T10:09:34.884 INFO:teuthology.orchestra.run.smithi119.stdout:skipping the directory /proc 2024-04-23T10:09:34.884 INFO:teuthology.orchestra.run.smithi119.stdout:skipping the directory /mnt 2024-04-23T10:09:34.884 INFO:teuthology.orchestra.run.smithi119.stdout:skipping the directory /var/tmp 2024-04-23T10:09:34.884 INFO:teuthology.orchestra.run.smithi119.stdout:skipping the directory /home 2024-04-23T10:09:34.884 INFO:teuthology.orchestra.run.smithi119.stdout:skipping the directory /root 2024-04-23T10:09:34.884 INFO:teuthology.orchestra.run.smithi119.stdout:skipping the directory /tmp 2024-04-23T10:09:34.884 INFO:teuthology.orchestra.run.smithi119.stdout: 2024-04-23T10:09:35.048 INFO:teuthology.orchestra.run.smithi119.stdout: Installing : ceph-volume-2:19.0.0-3156.gd69f13d8.el7.noarch 108/156 2024-04-23T10:09:35.063 INFO:teuthology.orchestra.run.smithi119.stdout: Running scriptlet: ceph-volume-2:19.0.0-3156.gd69f13d8.el7.noarch 108/156 2024-04-23T10:09:35.063 INFO:teuthology.orchestra.run.smithi119.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-04-23T10:09:35.063 INFO:teuthology.orchestra.run.smithi119.stdout:Invalid unit name "ceph-volume@*.service" escaped as "ceph-volume@\x2a.service". 2024-04-23T10:09:35.063 INFO:teuthology.orchestra.run.smithi119.stdout: 2024-04-23T10:09:36.551 INFO:teuthology.orchestra.run.smithi119.stdout: Installing : ceph-osd-2:19.0.0-3156.gd69f13d8.el7.x86_64 109/156 2024-04-23T10:09:36.576 INFO:teuthology.orchestra.run.smithi119.stdout: Running scriptlet: ceph-osd-2:19.0.0-3156.gd69f13d8.el7.x86_64 109/156 2024-04-23T10:09:36.577 INFO:teuthology.orchestra.run.smithi119.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-04-23T10:09:36.577 INFO:teuthology.orchestra.run.smithi119.stdout:Invalid unit name "ceph-osd@*.service" escaped as "ceph-osd@\x2a.service". 2024-04-23T10:09:36.577 INFO:teuthology.orchestra.run.smithi119.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-osd.target → /usr/lib/systemd/system/ceph-osd.target. 2024-04-23T10:09:36.577 INFO:teuthology.orchestra.run.smithi119.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-osd.target → /usr/lib/systemd/system/ceph-osd.target. 2024-04-23T10:09:36.577 INFO:teuthology.orchestra.run.smithi119.stdout: 2024-04-23T10:09:36.802 INFO:teuthology.orchestra.run.smithi119.stdout: Installing : ceph-mds-2:19.0.0-3156.gd69f13d8.el7.x86_64 110/156 2024-04-23T10:09:36.857 INFO:teuthology.orchestra.run.smithi119.stdout: Running scriptlet: ceph-mds-2:19.0.0-3156.gd69f13d8.el7.x86_64 110/156 2024-04-23T10:09:36.857 INFO:teuthology.orchestra.run.smithi119.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-04-23T10:09:36.857 INFO:teuthology.orchestra.run.smithi119.stdout:Invalid unit name "ceph-mds@*.service" escaped as "ceph-mds@\x2a.service". 2024-04-23T10:09:36.857 INFO:teuthology.orchestra.run.smithi119.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mds.target → /usr/lib/systemd/system/ceph-mds.target. 2024-04-23T10:09:36.857 INFO:teuthology.orchestra.run.smithi119.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mds.target → /usr/lib/systemd/system/ceph-mds.target. 2024-04-23T10:09:36.857 INFO:teuthology.orchestra.run.smithi119.stdout: 2024-04-23T10:09:37.278 INFO:teuthology.orchestra.run.smithi119.stdout: Installing : ceph-mon-2:19.0.0-3156.gd69f13d8.el7.x86_64 111/156 2024-04-23T10:09:37.300 INFO:teuthology.orchestra.run.smithi119.stdout: Running scriptlet: ceph-mon-2:19.0.0-3156.gd69f13d8.el7.x86_64 111/156 2024-04-23T10:09:37.300 INFO:teuthology.orchestra.run.smithi119.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-04-23T10:09:37.300 INFO:teuthology.orchestra.run.smithi119.stdout:Invalid unit name "ceph-mon@*.service" escaped as "ceph-mon@\x2a.service". 2024-04-23T10:09:37.300 INFO:teuthology.orchestra.run.smithi119.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mon.target → /usr/lib/systemd/system/ceph-mon.target. 2024-04-23T10:09:37.300 INFO:teuthology.orchestra.run.smithi119.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mon.target → /usr/lib/systemd/system/ceph-mon.target. 2024-04-23T10:09:37.300 INFO:teuthology.orchestra.run.smithi119.stdout: 2024-04-23T10:09:37.399 INFO:teuthology.orchestra.run.smithi119.stdout: Installing : automake-1.16.2-8.el9.noarch 112/156 2024-04-23T10:09:37.447 INFO:teuthology.orchestra.run.smithi119.stdout: Installing : python3-ply-3.11-14.el9.noarch 113/156 2024-04-23T10:09:37.503 INFO:teuthology.orchestra.run.smithi119.stdout: Installing : python3-pycparser-2.20-6.el9.noarch 114/156 2024-04-23T10:09:37.688 INFO:teuthology.orchestra.run.smithi119.stdout: Installing : python3-cffi-1.14.5-5.el9.x86_64 115/156 2024-04-23T10:09:37.737 INFO:teuthology.orchestra.run.smithi119.stdout: Installing : python3-cryptography-36.0.1-4.el9.x86_64 116/156 2024-04-23T10:09:37.823 INFO:teuthology.orchestra.run.smithi119.stdout: Installing : python3-pyOpenSSL-21.0.0-1.el9.noarch 117/156 2024-04-23T10:09:37.919 INFO:teuthology.orchestra.run.smithi119.stdout: Installing : python3-cheroot-8.6.0-4.el9.noarch 118/156 2024-04-23T10:09:38.049 INFO:teuthology.orchestra.run.smithi119.stdout: Installing : python3-cherrypy-18.6.1-2.el9.noarch 119/156 2024-04-23T10:09:38.095 INFO:teuthology.orchestra.run.smithi119.stdout: Installing : python3-asyncssh-2.13.2-5.el9.noarch 120/156 2024-04-23T10:09:38.127 INFO:teuthology.orchestra.run.smithi119.stdout: Installing : python3-bcrypt-3.2.2-1.el9.x86_64 121/156 2024-04-23T10:09:38.155 INFO:teuthology.orchestra.run.smithi119.stdout: Installing : fuse-2.9.9-15.el9.x86_64 122/156 2024-04-23T10:09:38.539 INFO:teuthology.orchestra.run.smithi119.stdout: Running scriptlet: cephadm-2:19.0.0-3156.gd69f13d8.el7.noarch 123/156 2024-04-23T10:09:38.562 INFO:teuthology.orchestra.run.smithi119.stdout: Installing : cephadm-2:19.0.0-3156.gd69f13d8.el7.noarch 123/156 2024-04-23T10:09:38.586 INFO:teuthology.orchestra.run.smithi119.stdout: Installing : ceph-prometheus-alerts-2:19.0.0-3156.gd69f13d8.e 124/156 2024-04-23T10:09:38.647 INFO:teuthology.orchestra.run.smithi119.stdout: Installing : ceph-grafana-dashboards-2:19.0.0-3156.gd69f13d8. 125/156 2024-04-23T10:09:38.677 INFO:teuthology.orchestra.run.smithi119.stdout: Installing : ceph-mgr-cephadm-2:19.0.0-3156.gd69f13d8.el7.noa 126/156 2024-04-23T10:09:39.254 INFO:teuthology.orchestra.run.smithi204.stdout: Running scriptlet: ceph-selinux-2:19.0.0-3156.gd69f13d8.el7.x86_64 107/156 2024-04-23T10:09:39.255 INFO:teuthology.orchestra.run.smithi204.stdout:skipping the directory /sys 2024-04-23T10:09:39.255 INFO:teuthology.orchestra.run.smithi204.stdout:skipping the directory /proc 2024-04-23T10:09:39.255 INFO:teuthology.orchestra.run.smithi204.stdout:skipping the directory /mnt 2024-04-23T10:09:39.255 INFO:teuthology.orchestra.run.smithi204.stdout:skipping the directory /var/tmp 2024-04-23T10:09:39.255 INFO:teuthology.orchestra.run.smithi204.stdout:skipping the directory /home 2024-04-23T10:09:39.255 INFO:teuthology.orchestra.run.smithi204.stdout:skipping the directory /root 2024-04-23T10:09:39.255 INFO:teuthology.orchestra.run.smithi204.stdout:skipping the directory /tmp 2024-04-23T10:09:39.255 INFO:teuthology.orchestra.run.smithi204.stdout: 2024-04-23T10:09:39.418 INFO:teuthology.orchestra.run.smithi204.stdout: Installing : ceph-volume-2:19.0.0-3156.gd69f13d8.el7.noarch 108/156 2024-04-23T10:09:39.432 INFO:teuthology.orchestra.run.smithi204.stdout: Running scriptlet: ceph-volume-2:19.0.0-3156.gd69f13d8.el7.noarch 108/156 2024-04-23T10:09:39.432 INFO:teuthology.orchestra.run.smithi204.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-04-23T10:09:39.432 INFO:teuthology.orchestra.run.smithi204.stdout:Invalid unit name "ceph-volume@*.service" escaped as "ceph-volume@\x2a.service". 2024-04-23T10:09:39.432 INFO:teuthology.orchestra.run.smithi204.stdout: 2024-04-23T10:09:39.712 INFO:teuthology.orchestra.run.smithi119.stdout: Running scriptlet: ceph-mgr-cephadm-2:19.0.0-3156.gd69f13d8.el7.noa 126/156 2024-04-23T10:09:39.762 INFO:teuthology.orchestra.run.smithi119.stdout: Installing : ceph-mgr-dashboard-2:19.0.0-3156.gd69f13d8.el7.n 127/156 2024-04-23T10:09:40.720 INFO:teuthology.orchestra.run.smithi119.stdout: Running scriptlet: ceph-mgr-dashboard-2:19.0.0-3156.gd69f13d8.el7.n 127/156 2024-04-23T10:09:40.771 INFO:teuthology.orchestra.run.smithi119.stdout: Installing : ceph-mgr-diskprediction-local-2:19.0.0-3156.gd69 128/156 2024-04-23T10:09:40.873 INFO:teuthology.orchestra.run.smithi119.stdout: Running scriptlet: ceph-mgr-diskprediction-local-2:19.0.0-3156.gd69 128/156 2024-04-23T10:09:40.930 INFO:teuthology.orchestra.run.smithi204.stdout: Installing : ceph-osd-2:19.0.0-3156.gd69f13d8.el7.x86_64 109/156 2024-04-23T10:09:40.954 INFO:teuthology.orchestra.run.smithi204.stdout: Running scriptlet: ceph-osd-2:19.0.0-3156.gd69f13d8.el7.x86_64 109/156 2024-04-23T10:09:40.954 INFO:teuthology.orchestra.run.smithi204.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-04-23T10:09:40.955 INFO:teuthology.orchestra.run.smithi204.stdout:Invalid unit name "ceph-osd@*.service" escaped as "ceph-osd@\x2a.service". 2024-04-23T10:09:40.955 INFO:teuthology.orchestra.run.smithi204.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-osd.target → /usr/lib/systemd/system/ceph-osd.target. 2024-04-23T10:09:40.955 INFO:teuthology.orchestra.run.smithi204.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-osd.target → /usr/lib/systemd/system/ceph-osd.target. 2024-04-23T10:09:40.955 INFO:teuthology.orchestra.run.smithi204.stdout: 2024-04-23T10:09:41.035 INFO:teuthology.orchestra.run.smithi119.stdout: Installing : ceph-mgr-modules-core-2:19.0.0-3156.gd69f13d8.el 129/156 2024-04-23T10:09:41.064 INFO:teuthology.orchestra.run.smithi119.stdout: Installing : ceph-mgr-2:19.0.0-3156.gd69f13d8.el7.x86_64 130/156 2024-04-23T10:09:41.086 INFO:teuthology.orchestra.run.smithi119.stdout: Running scriptlet: ceph-mgr-2:19.0.0-3156.gd69f13d8.el7.x86_64 130/156 2024-04-23T10:09:41.086 INFO:teuthology.orchestra.run.smithi119.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-04-23T10:09:41.086 INFO:teuthology.orchestra.run.smithi119.stdout:Invalid unit name "ceph-mgr@*.service" escaped as "ceph-mgr@\x2a.service". 2024-04-23T10:09:41.087 INFO:teuthology.orchestra.run.smithi119.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mgr.target → /usr/lib/systemd/system/ceph-mgr.target. 2024-04-23T10:09:41.087 INFO:teuthology.orchestra.run.smithi119.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mgr.target → /usr/lib/systemd/system/ceph-mgr.target. 2024-04-23T10:09:41.087 INFO:teuthology.orchestra.run.smithi119.stdout: 2024-04-23T10:09:41.132 INFO:teuthology.orchestra.run.smithi119.stdout: Installing : ceph-mgr-rook-2:19.0.0-3156.gd69f13d8.el7.noarch 131/156 2024-04-23T10:09:41.143 INFO:teuthology.orchestra.run.smithi119.stdout: Running scriptlet: ceph-mgr-rook-2:19.0.0-3156.gd69f13d8.el7.noarch 131/156 2024-04-23T10:09:41.190 INFO:teuthology.orchestra.run.smithi204.stdout: Installing : ceph-mds-2:19.0.0-3156.gd69f13d8.el7.x86_64 110/156 2024-04-23T10:09:41.211 INFO:teuthology.orchestra.run.smithi204.stdout: Running scriptlet: ceph-mds-2:19.0.0-3156.gd69f13d8.el7.x86_64 110/156 2024-04-23T10:09:41.212 INFO:teuthology.orchestra.run.smithi204.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-04-23T10:09:41.212 INFO:teuthology.orchestra.run.smithi204.stdout:Invalid unit name "ceph-mds@*.service" escaped as "ceph-mds@\x2a.service". 2024-04-23T10:09:41.212 INFO:teuthology.orchestra.run.smithi204.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mds.target → /usr/lib/systemd/system/ceph-mds.target. 2024-04-23T10:09:41.212 INFO:teuthology.orchestra.run.smithi204.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mds.target → /usr/lib/systemd/system/ceph-mds.target. 2024-04-23T10:09:41.212 INFO:teuthology.orchestra.run.smithi204.stdout: 2024-04-23T10:09:41.240 INFO:teuthology.orchestra.run.smithi119.stdout: Installing : ceph-2:19.0.0-3156.gd69f13d8.el7.x86_64 132/156 2024-04-23T10:09:41.288 INFO:teuthology.orchestra.run.smithi119.stdout: Installing : ceph-fuse-2:19.0.0-3156.gd69f13d8.el7.x86_64 133/156 2024-04-23T10:09:41.649 INFO:teuthology.orchestra.run.smithi204.stdout: Installing : ceph-mon-2:19.0.0-3156.gd69f13d8.el7.x86_64 111/156 2024-04-23T10:09:41.670 INFO:teuthology.orchestra.run.smithi204.stdout: Running scriptlet: ceph-mon-2:19.0.0-3156.gd69f13d8.el7.x86_64 111/156 2024-04-23T10:09:41.670 INFO:teuthology.orchestra.run.smithi204.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-04-23T10:09:41.670 INFO:teuthology.orchestra.run.smithi204.stdout:Invalid unit name "ceph-mon@*.service" escaped as "ceph-mon@\x2a.service". 2024-04-23T10:09:41.670 INFO:teuthology.orchestra.run.smithi204.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mon.target → /usr/lib/systemd/system/ceph-mon.target. 2024-04-23T10:09:41.671 INFO:teuthology.orchestra.run.smithi204.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mon.target → /usr/lib/systemd/system/ceph-mon.target. 2024-04-23T10:09:41.671 INFO:teuthology.orchestra.run.smithi204.stdout: 2024-04-23T10:09:41.779 INFO:teuthology.orchestra.run.smithi204.stdout: Installing : automake-1.16.2-8.el9.noarch 112/156 2024-04-23T10:09:41.827 INFO:teuthology.orchestra.run.smithi204.stdout: Installing : python3-ply-3.11-14.el9.noarch 113/156 2024-04-23T10:09:41.882 INFO:teuthology.orchestra.run.smithi204.stdout: Installing : python3-pycparser-2.20-6.el9.noarch 114/156 2024-04-23T10:09:42.076 INFO:teuthology.orchestra.run.smithi204.stdout: Installing : python3-cffi-1.14.5-5.el9.x86_64 115/156 2024-04-23T10:09:42.124 INFO:teuthology.orchestra.run.smithi204.stdout: Installing : python3-cryptography-36.0.1-4.el9.x86_64 116/156 2024-04-23T10:09:42.202 INFO:teuthology.orchestra.run.smithi204.stdout: Installing : python3-pyOpenSSL-21.0.0-1.el9.noarch 117/156 2024-04-23T10:09:42.208 INFO:teuthology.orchestra.run.smithi119.stdout: Installing : libtool-2.4.6-45.el9.x86_64 134/156 2024-04-23T10:09:42.235 INFO:teuthology.orchestra.run.smithi119.stdout: Installing : ceph-radosgw-2:19.0.0-3156.gd69f13d8.el7.x86_64 135/156 2024-04-23T10:09:42.257 INFO:teuthology.orchestra.run.smithi119.stdout: Running scriptlet: ceph-radosgw-2:19.0.0-3156.gd69f13d8.el7.x86_64 135/156 2024-04-23T10:09:42.257 INFO:teuthology.orchestra.run.smithi119.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-04-23T10:09:42.257 INFO:teuthology.orchestra.run.smithi119.stdout:Invalid unit name "ceph-radosgw@*.service" escaped as "ceph-radosgw@\x2a.service". 2024-04-23T10:09:42.257 INFO:teuthology.orchestra.run.smithi119.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-radosgw.target → /usr/lib/systemd/system/ceph-radosgw.target. 2024-04-23T10:09:42.257 INFO:teuthology.orchestra.run.smithi119.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-radosgw.target → /usr/lib/systemd/system/ceph-radosgw.target. 2024-04-23T10:09:42.257 INFO:teuthology.orchestra.run.smithi119.stdout: 2024-04-23T10:09:42.291 INFO:teuthology.orchestra.run.smithi204.stdout: Installing : python3-cheroot-8.6.0-4.el9.noarch 118/156 2024-04-23T10:09:42.310 INFO:teuthology.orchestra.run.smithi119.stdout: Installing : ceph-immutable-object-cache-2:19.0.0-3156.gd69f1 136/156 2024-04-23T10:09:42.331 INFO:teuthology.orchestra.run.smithi119.stdout: Running scriptlet: ceph-immutable-object-cache-2:19.0.0-3156.gd69f1 136/156 2024-04-23T10:09:42.332 INFO:teuthology.orchestra.run.smithi119.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-04-23T10:09:42.332 INFO:teuthology.orchestra.run.smithi119.stdout:Invalid unit name "ceph-immutable-object-cache@*.service" escaped as "ceph-immutable-object-cache@\x2a.service". 2024-04-23T10:09:42.332 INFO:teuthology.orchestra.run.smithi119.stdout: 2024-04-23T10:09:42.377 INFO:teuthology.orchestra.run.smithi119.stdout: Installing : cephfs-mirror-2:19.0.0-3156.gd69f13d8.el7.x86_64 137/156 2024-04-23T10:09:42.399 INFO:teuthology.orchestra.run.smithi119.stdout: Running scriptlet: cephfs-mirror-2:19.0.0-3156.gd69f13d8.el7.x86_64 137/156 2024-04-23T10:09:42.399 INFO:teuthology.orchestra.run.smithi119.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-04-23T10:09:42.399 INFO:teuthology.orchestra.run.smithi119.stdout:Invalid unit name "cephfs-mirror@*.service" escaped as "cephfs-mirror@\x2a.service". 2024-04-23T10:09:42.399 INFO:teuthology.orchestra.run.smithi119.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/cephfs-mirror.target → /usr/lib/systemd/system/cephfs-mirror.target. 2024-04-23T10:09:42.400 INFO:teuthology.orchestra.run.smithi119.stdout:Created symlink /etc/systemd/system/ceph.target.wants/cephfs-mirror.target → /usr/lib/systemd/system/cephfs-mirror.target. 2024-04-23T10:09:42.400 INFO:teuthology.orchestra.run.smithi119.stdout: 2024-04-23T10:09:42.430 INFO:teuthology.orchestra.run.smithi204.stdout: Installing : python3-cherrypy-18.6.1-2.el9.noarch 119/156 2024-04-23T10:09:42.466 INFO:teuthology.orchestra.run.smithi204.stdout: Installing : python3-asyncssh-2.13.2-5.el9.noarch 120/156 2024-04-23T10:09:42.499 INFO:teuthology.orchestra.run.smithi204.stdout: Installing : python3-bcrypt-3.2.2-1.el9.x86_64 121/156 2024-04-23T10:09:42.526 INFO:teuthology.orchestra.run.smithi204.stdout: Installing : fuse-2.9.9-15.el9.x86_64 122/156 2024-04-23T10:09:42.703 INFO:teuthology.orchestra.run.smithi119.stdout: Installing : rbd-mirror-2:19.0.0-3156.gd69f13d8.el7.x86_64 138/156 2024-04-23T10:09:42.725 INFO:teuthology.orchestra.run.smithi119.stdout: Running scriptlet: rbd-mirror-2:19.0.0-3156.gd69f13d8.el7.x86_64 138/156 2024-04-23T10:09:42.725 INFO:teuthology.orchestra.run.smithi119.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-04-23T10:09:42.725 INFO:teuthology.orchestra.run.smithi119.stdout:Invalid unit name "ceph-rbd-mirror@*.service" escaped as "ceph-rbd-mirror@\x2a.service". 2024-04-23T10:09:42.725 INFO:teuthology.orchestra.run.smithi119.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-rbd-mirror.target → /usr/lib/systemd/system/ceph-rbd-mirror.target. 2024-04-23T10:09:42.726 INFO:teuthology.orchestra.run.smithi119.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-rbd-mirror.target → /usr/lib/systemd/system/ceph-rbd-mirror.target. 2024-04-23T10:09:42.726 INFO:teuthology.orchestra.run.smithi119.stdout: 2024-04-23T10:09:42.970 INFO:teuthology.orchestra.run.smithi204.stdout: Running scriptlet: cephadm-2:19.0.0-3156.gd69f13d8.el7.noarch 123/156 2024-04-23T10:09:43.011 INFO:teuthology.orchestra.run.smithi204.stdout: Installing : cephadm-2:19.0.0-3156.gd69f13d8.el7.noarch 123/156 2024-04-23T10:09:43.049 INFO:teuthology.orchestra.run.smithi204.stdout: Installing : ceph-prometheus-alerts-2:19.0.0-3156.gd69f13d8.e 124/156 2024-04-23T10:09:43.111 INFO:teuthology.orchestra.run.smithi204.stdout: Installing : ceph-grafana-dashboards-2:19.0.0-3156.gd69f13d8. 125/156 2024-04-23T10:09:43.149 INFO:teuthology.orchestra.run.smithi204.stdout: Installing : ceph-mgr-cephadm-2:19.0.0-3156.gd69f13d8.el7.noa 126/156 2024-04-23T10:09:44.187 INFO:teuthology.orchestra.run.smithi204.stdout: Running scriptlet: ceph-mgr-cephadm-2:19.0.0-3156.gd69f13d8.el7.noa 126/156 2024-04-23T10:09:44.188 INFO:teuthology.orchestra.run.smithi114.stdout: Running scriptlet: ceph-selinux-2:19.0.0-3156.gd69f13d8.el7.x86_64 107/156 2024-04-23T10:09:44.188 INFO:teuthology.orchestra.run.smithi114.stdout:skipping the directory /sys 2024-04-23T10:09:44.188 INFO:teuthology.orchestra.run.smithi114.stdout:skipping the directory /proc 2024-04-23T10:09:44.188 INFO:teuthology.orchestra.run.smithi114.stdout:skipping the directory /mnt 2024-04-23T10:09:44.188 INFO:teuthology.orchestra.run.smithi114.stdout:skipping the directory /var/tmp 2024-04-23T10:09:44.188 INFO:teuthology.orchestra.run.smithi114.stdout:skipping the directory /home 2024-04-23T10:09:44.188 INFO:teuthology.orchestra.run.smithi114.stdout:skipping the directory /root 2024-04-23T10:09:44.188 INFO:teuthology.orchestra.run.smithi114.stdout:skipping the directory /tmp 2024-04-23T10:09:44.188 INFO:teuthology.orchestra.run.smithi114.stdout: 2024-04-23T10:09:44.218 INFO:teuthology.orchestra.run.smithi204.stdout: Installing : ceph-mgr-dashboard-2:19.0.0-3156.gd69f13d8.el7.n 127/156 2024-04-23T10:09:44.370 INFO:teuthology.orchestra.run.smithi114.stdout: Installing : ceph-volume-2:19.0.0-3156.gd69f13d8.el7.noarch 108/156 2024-04-23T10:09:44.385 INFO:teuthology.orchestra.run.smithi114.stdout: Running scriptlet: ceph-volume-2:19.0.0-3156.gd69f13d8.el7.noarch 108/156 2024-04-23T10:09:44.385 INFO:teuthology.orchestra.run.smithi114.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-04-23T10:09:44.385 INFO:teuthology.orchestra.run.smithi114.stdout:Invalid unit name "ceph-volume@*.service" escaped as "ceph-volume@\x2a.service". 2024-04-23T10:09:44.385 INFO:teuthology.orchestra.run.smithi114.stdout: 2024-04-23T10:09:45.178 INFO:teuthology.orchestra.run.smithi204.stdout: Running scriptlet: ceph-mgr-dashboard-2:19.0.0-3156.gd69f13d8.el7.n 127/156 2024-04-23T10:09:45.201 INFO:teuthology.orchestra.run.smithi204.stdout: Installing : ceph-mgr-diskprediction-local-2:19.0.0-3156.gd69 128/156 2024-04-23T10:09:45.302 INFO:teuthology.orchestra.run.smithi204.stdout: Running scriptlet: ceph-mgr-diskprediction-local-2:19.0.0-3156.gd69 128/156 2024-04-23T10:09:45.465 INFO:teuthology.orchestra.run.smithi204.stdout: Installing : ceph-mgr-modules-core-2:19.0.0-3156.gd69f13d8.el 129/156 2024-04-23T10:09:45.495 INFO:teuthology.orchestra.run.smithi204.stdout: Installing : ceph-mgr-2:19.0.0-3156.gd69f13d8.el7.x86_64 130/156 2024-04-23T10:09:45.517 INFO:teuthology.orchestra.run.smithi204.stdout: Running scriptlet: ceph-mgr-2:19.0.0-3156.gd69f13d8.el7.x86_64 130/156 2024-04-23T10:09:45.517 INFO:teuthology.orchestra.run.smithi204.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-04-23T10:09:45.517 INFO:teuthology.orchestra.run.smithi204.stdout:Invalid unit name "ceph-mgr@*.service" escaped as "ceph-mgr@\x2a.service". 2024-04-23T10:09:45.517 INFO:teuthology.orchestra.run.smithi204.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mgr.target → /usr/lib/systemd/system/ceph-mgr.target. 2024-04-23T10:09:45.518 INFO:teuthology.orchestra.run.smithi204.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mgr.target → /usr/lib/systemd/system/ceph-mgr.target. 2024-04-23T10:09:45.518 INFO:teuthology.orchestra.run.smithi204.stdout: 2024-04-23T10:09:45.562 INFO:teuthology.orchestra.run.smithi204.stdout: Installing : ceph-mgr-rook-2:19.0.0-3156.gd69f13d8.el7.noarch 131/156 2024-04-23T10:09:45.573 INFO:teuthology.orchestra.run.smithi204.stdout: Running scriptlet: ceph-mgr-rook-2:19.0.0-3156.gd69f13d8.el7.noarch 131/156 2024-04-23T10:09:45.671 INFO:teuthology.orchestra.run.smithi204.stdout: Installing : ceph-2:19.0.0-3156.gd69f13d8.el7.x86_64 132/156 2024-04-23T10:09:45.719 INFO:teuthology.orchestra.run.smithi204.stdout: Installing : ceph-fuse-2:19.0.0-3156.gd69f13d8.el7.x86_64 133/156 2024-04-23T10:09:45.864 INFO:teuthology.orchestra.run.smithi114.stdout: Installing : ceph-osd-2:19.0.0-3156.gd69f13d8.el7.x86_64 109/156 2024-04-23T10:09:45.889 INFO:teuthology.orchestra.run.smithi114.stdout: Running scriptlet: ceph-osd-2:19.0.0-3156.gd69f13d8.el7.x86_64 109/156 2024-04-23T10:09:45.889 INFO:teuthology.orchestra.run.smithi114.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-04-23T10:09:45.889 INFO:teuthology.orchestra.run.smithi114.stdout:Invalid unit name "ceph-osd@*.service" escaped as "ceph-osd@\x2a.service". 2024-04-23T10:09:45.889 INFO:teuthology.orchestra.run.smithi114.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-osd.target → /usr/lib/systemd/system/ceph-osd.target. 2024-04-23T10:09:45.889 INFO:teuthology.orchestra.run.smithi114.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-osd.target → /usr/lib/systemd/system/ceph-osd.target. 2024-04-23T10:09:45.889 INFO:teuthology.orchestra.run.smithi114.stdout: 2024-04-23T10:09:46.115 INFO:teuthology.orchestra.run.smithi114.stdout: Installing : ceph-mds-2:19.0.0-3156.gd69f13d8.el7.x86_64 110/156 2024-04-23T10:09:46.137 INFO:teuthology.orchestra.run.smithi114.stdout: Running scriptlet: ceph-mds-2:19.0.0-3156.gd69f13d8.el7.x86_64 110/156 2024-04-23T10:09:46.137 INFO:teuthology.orchestra.run.smithi114.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-04-23T10:09:46.137 INFO:teuthology.orchestra.run.smithi114.stdout:Invalid unit name "ceph-mds@*.service" escaped as "ceph-mds@\x2a.service". 2024-04-23T10:09:46.137 INFO:teuthology.orchestra.run.smithi114.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mds.target → /usr/lib/systemd/system/ceph-mds.target. 2024-04-23T10:09:46.137 INFO:teuthology.orchestra.run.smithi114.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mds.target → /usr/lib/systemd/system/ceph-mds.target. 2024-04-23T10:09:46.137 INFO:teuthology.orchestra.run.smithi114.stdout: 2024-04-23T10:09:46.582 INFO:teuthology.orchestra.run.smithi114.stdout: Installing : ceph-mon-2:19.0.0-3156.gd69f13d8.el7.x86_64 111/156 2024-04-23T10:09:46.603 INFO:teuthology.orchestra.run.smithi114.stdout: Running scriptlet: ceph-mon-2:19.0.0-3156.gd69f13d8.el7.x86_64 111/156 2024-04-23T10:09:46.604 INFO:teuthology.orchestra.run.smithi114.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-04-23T10:09:46.604 INFO:teuthology.orchestra.run.smithi114.stdout:Invalid unit name "ceph-mon@*.service" escaped as "ceph-mon@\x2a.service". 2024-04-23T10:09:46.604 INFO:teuthology.orchestra.run.smithi114.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mon.target → /usr/lib/systemd/system/ceph-mon.target. 2024-04-23T10:09:46.604 INFO:teuthology.orchestra.run.smithi114.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mon.target → /usr/lib/systemd/system/ceph-mon.target. 2024-04-23T10:09:46.604 INFO:teuthology.orchestra.run.smithi114.stdout: 2024-04-23T10:09:46.640 INFO:teuthology.orchestra.run.smithi204.stdout: Installing : libtool-2.4.6-45.el9.x86_64 134/156 2024-04-23T10:09:46.673 INFO:teuthology.orchestra.run.smithi204.stdout: Installing : ceph-radosgw-2:19.0.0-3156.gd69f13d8.el7.x86_64 135/156 2024-04-23T10:09:46.695 INFO:teuthology.orchestra.run.smithi204.stdout: Running scriptlet: ceph-radosgw-2:19.0.0-3156.gd69f13d8.el7.x86_64 135/156 2024-04-23T10:09:46.695 INFO:teuthology.orchestra.run.smithi204.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-04-23T10:09:46.695 INFO:teuthology.orchestra.run.smithi204.stdout:Invalid unit name "ceph-radosgw@*.service" escaped as "ceph-radosgw@\x2a.service". 2024-04-23T10:09:46.695 INFO:teuthology.orchestra.run.smithi204.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-radosgw.target → /usr/lib/systemd/system/ceph-radosgw.target. 2024-04-23T10:09:46.695 INFO:teuthology.orchestra.run.smithi204.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-radosgw.target → /usr/lib/systemd/system/ceph-radosgw.target. 2024-04-23T10:09:46.695 INFO:teuthology.orchestra.run.smithi204.stdout: 2024-04-23T10:09:46.704 INFO:teuthology.orchestra.run.smithi114.stdout: Installing : automake-1.16.2-8.el9.noarch 112/156 2024-04-23T10:09:46.740 INFO:teuthology.orchestra.run.smithi204.stdout: Installing : ceph-immutable-object-cache-2:19.0.0-3156.gd69f1 136/156 2024-04-23T10:09:46.752 INFO:teuthology.orchestra.run.smithi114.stdout: Installing : python3-ply-3.11-14.el9.noarch 113/156 2024-04-23T10:09:46.761 INFO:teuthology.orchestra.run.smithi204.stdout: Running scriptlet: ceph-immutable-object-cache-2:19.0.0-3156.gd69f1 136/156 2024-04-23T10:09:46.761 INFO:teuthology.orchestra.run.smithi204.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-04-23T10:09:46.761 INFO:teuthology.orchestra.run.smithi204.stdout:Invalid unit name "ceph-immutable-object-cache@*.service" escaped as "ceph-immutable-object-cache@\x2a.service". 2024-04-23T10:09:46.762 INFO:teuthology.orchestra.run.smithi204.stdout: 2024-04-23T10:09:46.808 INFO:teuthology.orchestra.run.smithi204.stdout: Installing : cephfs-mirror-2:19.0.0-3156.gd69f13d8.el7.x86_64 137/156 2024-04-23T10:09:46.815 INFO:teuthology.orchestra.run.smithi114.stdout: Installing : python3-pycparser-2.20-6.el9.noarch 114/156 2024-04-23T10:09:46.830 INFO:teuthology.orchestra.run.smithi204.stdout: Running scriptlet: cephfs-mirror-2:19.0.0-3156.gd69f13d8.el7.x86_64 137/156 2024-04-23T10:09:46.830 INFO:teuthology.orchestra.run.smithi204.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-04-23T10:09:46.830 INFO:teuthology.orchestra.run.smithi204.stdout:Invalid unit name "cephfs-mirror@*.service" escaped as "cephfs-mirror@\x2a.service". 2024-04-23T10:09:46.830 INFO:teuthology.orchestra.run.smithi204.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/cephfs-mirror.target → /usr/lib/systemd/system/cephfs-mirror.target. 2024-04-23T10:09:46.830 INFO:teuthology.orchestra.run.smithi204.stdout:Created symlink /etc/systemd/system/ceph.target.wants/cephfs-mirror.target → /usr/lib/systemd/system/cephfs-mirror.target. 2024-04-23T10:09:46.830 INFO:teuthology.orchestra.run.smithi204.stdout: 2024-04-23T10:09:47.000 INFO:teuthology.orchestra.run.smithi114.stdout: Installing : python3-cffi-1.14.5-5.el9.x86_64 115/156 2024-04-23T10:09:47.041 INFO:teuthology.orchestra.run.smithi114.stdout: Installing : python3-cryptography-36.0.1-4.el9.x86_64 116/156 2024-04-23T10:09:47.119 INFO:teuthology.orchestra.run.smithi114.stdout: Installing : python3-pyOpenSSL-21.0.0-1.el9.noarch 117/156 2024-04-23T10:09:47.134 INFO:teuthology.orchestra.run.smithi204.stdout: Installing : rbd-mirror-2:19.0.0-3156.gd69f13d8.el7.x86_64 138/156 2024-04-23T10:09:47.156 INFO:teuthology.orchestra.run.smithi204.stdout: Running scriptlet: rbd-mirror-2:19.0.0-3156.gd69f13d8.el7.x86_64 138/156 2024-04-23T10:09:47.156 INFO:teuthology.orchestra.run.smithi204.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-04-23T10:09:47.156 INFO:teuthology.orchestra.run.smithi204.stdout:Invalid unit name "ceph-rbd-mirror@*.service" escaped as "ceph-rbd-mirror@\x2a.service". 2024-04-23T10:09:47.156 INFO:teuthology.orchestra.run.smithi204.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-rbd-mirror.target → /usr/lib/systemd/system/ceph-rbd-mirror.target. 2024-04-23T10:09:47.156 INFO:teuthology.orchestra.run.smithi204.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-rbd-mirror.target → /usr/lib/systemd/system/ceph-rbd-mirror.target. 2024-04-23T10:09:47.156 INFO:teuthology.orchestra.run.smithi204.stdout: 2024-04-23T10:09:47.207 INFO:teuthology.orchestra.run.smithi114.stdout: Installing : python3-cheroot-8.6.0-4.el9.noarch 118/156 2024-04-23T10:09:47.315 INFO:teuthology.orchestra.run.smithi119.stdout: Installing : ceph-test-2:19.0.0-3156.gd69f13d8.el7.x86_64 139/156 2024-04-23T10:09:47.345 INFO:teuthology.orchestra.run.smithi114.stdout: Installing : python3-cherrypy-18.6.1-2.el9.noarch 119/156 2024-04-23T10:09:47.358 INFO:teuthology.orchestra.run.smithi119.stdout: Installing : rbd-fuse-2:19.0.0-3156.gd69f13d8.el7.x86_64 140/156 2024-04-23T10:09:47.397 INFO:teuthology.orchestra.run.smithi119.stdout: Installing : rbd-nbd-2:19.0.0-3156.gd69f13d8.el7.x86_64 141/156 2024-04-23T10:09:47.407 INFO:teuthology.orchestra.run.smithi114.stdout: Installing : python3-asyncssh-2.13.2-5.el9.noarch 120/156 2024-04-23T10:09:47.430 INFO:teuthology.orchestra.run.smithi119.stdout: Installing : xfsprogs-devel-6.3.0-1.el9.x86_64 142/156 2024-04-23T10:09:47.439 INFO:teuthology.orchestra.run.smithi114.stdout: Installing : python3-bcrypt-3.2.2-1.el9.x86_64 121/156 2024-04-23T10:09:47.457 INFO:teuthology.orchestra.run.smithi119.stdout: Installing : elfutils-libelf-devel-0.190-2.el9.x86_64 143/156 2024-04-23T10:09:47.467 INFO:teuthology.orchestra.run.smithi114.stdout: Installing : fuse-2.9.9-15.el9.x86_64 122/156 2024-04-23T10:09:47.640 INFO:teuthology.orchestra.run.smithi119.stdout: Installing : cephfs-shell-2:19.0.0-3156.gd69f13d8.el7.x86_64 144/156 2024-04-23T10:09:47.678 INFO:teuthology.orchestra.run.smithi119.stdout: Installing : postgresql-contrib-13.14-1.el9.x86_64 145/156 2024-04-23T10:09:47.708 INFO:teuthology.orchestra.run.smithi119.stdout: Installing : libcephfs-devel-2:19.0.0-3156.gd69f13d8.el7.x86_ 146/156 2024-04-23T10:09:47.910 INFO:teuthology.orchestra.run.smithi114.stdout: Running scriptlet: cephadm-2:19.0.0-3156.gd69f13d8.el7.noarch 123/156 2024-04-23T10:09:47.940 INFO:teuthology.orchestra.run.smithi114.stdout: Installing : cephadm-2:19.0.0-3156.gd69f13d8.el7.noarch 123/156 2024-04-23T10:09:47.972 INFO:teuthology.orchestra.run.smithi114.stdout: Installing : ceph-prometheus-alerts-2:19.0.0-3156.gd69f13d8.e 124/156 2024-04-23T10:09:48.034 INFO:teuthology.orchestra.run.smithi114.stdout: Installing : ceph-grafana-dashboards-2:19.0.0-3156.gd69f13d8. 125/156 2024-04-23T10:09:48.072 INFO:teuthology.orchestra.run.smithi114.stdout: Installing : ceph-mgr-cephadm-2:19.0.0-3156.gd69f13d8.el7.noa 126/156 2024-04-23T10:09:48.340 INFO:teuthology.orchestra.run.smithi119.stdout: Running scriptlet: postgresql-server-13.14-1.el9.x86_64 147/156 2024-04-23T10:09:48.381 INFO:teuthology.orchestra.run.smithi119.stdout: Installing : postgresql-server-13.14-1.el9.x86_64 147/156 2024-04-23T10:09:48.400 INFO:teuthology.orchestra.run.smithi119.stdout: Running scriptlet: postgresql-server-13.14-1.el9.x86_64 147/156 2024-04-23T10:09:49.107 INFO:teuthology.orchestra.run.smithi114.stdout: Running scriptlet: ceph-mgr-cephadm-2:19.0.0-3156.gd69f13d8.el7.noa 126/156 2024-04-23T10:09:49.157 INFO:teuthology.orchestra.run.smithi114.stdout: Installing : ceph-mgr-dashboard-2:19.0.0-3156.gd69f13d8.el7.n 127/156 2024-04-23T10:09:49.234 INFO:teuthology.orchestra.run.smithi119.stdout: Installing : cephfs-top-2:19.0.0-3156.gd69f13d8.el7.noarch 148/156 2024-04-23T10:09:49.321 INFO:teuthology.orchestra.run.smithi119.stdout: Installing : openssl-devel-1:3.2.1-1.el9.x86_64 149/156 2024-04-23T10:09:49.362 INFO:teuthology.orchestra.run.smithi119.stdout: Installing : libaio-devel-0.3.111-13.el9.x86_64 150/156 2024-04-23T10:09:49.443 INFO:teuthology.orchestra.run.smithi119.stdout: Installing : flex-2.6.4-9.el9.x86_64 151/156 2024-04-23T10:09:49.475 INFO:teuthology.orchestra.run.smithi119.stdout: Installing : bison-3.7.4-5.el9.x86_64 152/156 2024-04-23T10:09:49.476 INFO:teuthology.orchestra.run.smithi119.stdout: Cleanup : librbd1-2:16.2.4-5.el9.x86_64 153/156 2024-04-23T10:09:49.559 INFO:teuthology.orchestra.run.smithi119.stdout: Running scriptlet: librbd1-2:16.2.4-5.el9.x86_64 153/156 2024-04-23T10:09:49.592 INFO:teuthology.orchestra.run.smithi119.stdout: Cleanup : openssl-1:3.0.7-27.el9.x86_64 154/156 2024-04-23T10:09:49.592 INFO:teuthology.orchestra.run.smithi119.stdout: Cleanup : librados2-2:16.2.4-5.el9.x86_64 155/156 2024-04-23T10:09:49.681 INFO:teuthology.orchestra.run.smithi119.stdout: Running scriptlet: librados2-2:16.2.4-5.el9.x86_64 155/156 2024-04-23T10:09:49.749 INFO:teuthology.orchestra.run.smithi119.stdout: Cleanup : openssl-libs-1:3.0.7-27.el9.x86_64 156/156 2024-04-23T10:09:50.114 INFO:teuthology.orchestra.run.smithi114.stdout: Running scriptlet: ceph-mgr-dashboard-2:19.0.0-3156.gd69f13d8.el7.n 127/156 2024-04-23T10:09:50.132 INFO:teuthology.orchestra.run.smithi114.stdout: Installing : ceph-mgr-diskprediction-local-2:19.0.0-3156.gd69 128/156 2024-04-23T10:09:50.232 INFO:teuthology.orchestra.run.smithi114.stdout: Running scriptlet: ceph-mgr-diskprediction-local-2:19.0.0-3156.gd69 128/156 2024-04-23T10:09:50.411 INFO:teuthology.orchestra.run.smithi114.stdout: Installing : ceph-mgr-modules-core-2:19.0.0-3156.gd69f13d8.el 129/156 2024-04-23T10:09:50.443 INFO:teuthology.orchestra.run.smithi114.stdout: Installing : ceph-mgr-2:19.0.0-3156.gd69f13d8.el7.x86_64 130/156 2024-04-23T10:09:50.464 INFO:teuthology.orchestra.run.smithi114.stdout: Running scriptlet: ceph-mgr-2:19.0.0-3156.gd69f13d8.el7.x86_64 130/156 2024-04-23T10:09:50.464 INFO:teuthology.orchestra.run.smithi114.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-04-23T10:09:50.464 INFO:teuthology.orchestra.run.smithi114.stdout:Invalid unit name "ceph-mgr@*.service" escaped as "ceph-mgr@\x2a.service". 2024-04-23T10:09:50.464 INFO:teuthology.orchestra.run.smithi114.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mgr.target → /usr/lib/systemd/system/ceph-mgr.target. 2024-04-23T10:09:50.464 INFO:teuthology.orchestra.run.smithi114.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mgr.target → /usr/lib/systemd/system/ceph-mgr.target. 2024-04-23T10:09:50.464 INFO:teuthology.orchestra.run.smithi114.stdout: 2024-04-23T10:09:50.510 INFO:teuthology.orchestra.run.smithi114.stdout: Installing : ceph-mgr-rook-2:19.0.0-3156.gd69f13d8.el7.noarch 131/156 2024-04-23T10:09:50.520 INFO:teuthology.orchestra.run.smithi114.stdout: Running scriptlet: ceph-mgr-rook-2:19.0.0-3156.gd69f13d8.el7.noarch 131/156 2024-04-23T10:09:50.619 INFO:teuthology.orchestra.run.smithi114.stdout: Installing : ceph-2:19.0.0-3156.gd69f13d8.el7.x86_64 132/156 2024-04-23T10:09:50.666 INFO:teuthology.orchestra.run.smithi114.stdout: Installing : ceph-fuse-2:19.0.0-3156.gd69f13d8.el7.x86_64 133/156 2024-04-23T10:09:51.587 INFO:teuthology.orchestra.run.smithi114.stdout: Installing : libtool-2.4.6-45.el9.x86_64 134/156 2024-04-23T10:09:51.612 INFO:teuthology.orchestra.run.smithi114.stdout: Installing : ceph-radosgw-2:19.0.0-3156.gd69f13d8.el7.x86_64 135/156 2024-04-23T10:09:51.634 INFO:teuthology.orchestra.run.smithi114.stdout: Running scriptlet: ceph-radosgw-2:19.0.0-3156.gd69f13d8.el7.x86_64 135/156 2024-04-23T10:09:51.634 INFO:teuthology.orchestra.run.smithi114.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-04-23T10:09:51.634 INFO:teuthology.orchestra.run.smithi114.stdout:Invalid unit name "ceph-radosgw@*.service" escaped as "ceph-radosgw@\x2a.service". 2024-04-23T10:09:51.634 INFO:teuthology.orchestra.run.smithi114.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-radosgw.target → /usr/lib/systemd/system/ceph-radosgw.target. 2024-04-23T10:09:51.635 INFO:teuthology.orchestra.run.smithi114.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-radosgw.target → /usr/lib/systemd/system/ceph-radosgw.target. 2024-04-23T10:09:51.635 INFO:teuthology.orchestra.run.smithi114.stdout: 2024-04-23T10:09:51.671 INFO:teuthology.orchestra.run.smithi114.stdout: Installing : ceph-immutable-object-cache-2:19.0.0-3156.gd69f1 136/156 2024-04-23T10:09:51.693 INFO:teuthology.orchestra.run.smithi114.stdout: Running scriptlet: ceph-immutable-object-cache-2:19.0.0-3156.gd69f1 136/156 2024-04-23T10:09:51.693 INFO:teuthology.orchestra.run.smithi114.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-04-23T10:09:51.693 INFO:teuthology.orchestra.run.smithi114.stdout:Invalid unit name "ceph-immutable-object-cache@*.service" escaped as "ceph-immutable-object-cache@\x2a.service". 2024-04-23T10:09:51.693 INFO:teuthology.orchestra.run.smithi114.stdout: 2024-04-23T10:09:51.739 INFO:teuthology.orchestra.run.smithi114.stdout: Installing : cephfs-mirror-2:19.0.0-3156.gd69f13d8.el7.x86_64 137/156 2024-04-23T10:09:51.754 INFO:teuthology.orchestra.run.smithi204.stdout: Installing : ceph-test-2:19.0.0-3156.gd69f13d8.el7.x86_64 139/156 2024-04-23T10:09:51.760 INFO:teuthology.orchestra.run.smithi114.stdout: Running scriptlet: cephfs-mirror-2:19.0.0-3156.gd69f13d8.el7.x86_64 137/156 2024-04-23T10:09:51.760 INFO:teuthology.orchestra.run.smithi114.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-04-23T10:09:51.761 INFO:teuthology.orchestra.run.smithi114.stdout:Invalid unit name "cephfs-mirror@*.service" escaped as "cephfs-mirror@\x2a.service". 2024-04-23T10:09:51.761 INFO:teuthology.orchestra.run.smithi114.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/cephfs-mirror.target → /usr/lib/systemd/system/cephfs-mirror.target. 2024-04-23T10:09:51.761 INFO:teuthology.orchestra.run.smithi114.stdout:Created symlink /etc/systemd/system/ceph.target.wants/cephfs-mirror.target → /usr/lib/systemd/system/cephfs-mirror.target. 2024-04-23T10:09:51.761 INFO:teuthology.orchestra.run.smithi114.stdout: 2024-04-23T10:09:51.795 INFO:teuthology.orchestra.run.smithi204.stdout: Installing : rbd-fuse-2:19.0.0-3156.gd69f13d8.el7.x86_64 140/156 2024-04-23T10:09:51.836 INFO:teuthology.orchestra.run.smithi204.stdout: Installing : rbd-nbd-2:19.0.0-3156.gd69f13d8.el7.x86_64 141/156 2024-04-23T10:09:51.860 INFO:teuthology.orchestra.run.smithi204.stdout: Installing : xfsprogs-devel-6.3.0-1.el9.x86_64 142/156 2024-04-23T10:09:51.887 INFO:teuthology.orchestra.run.smithi204.stdout: Installing : elfutils-libelf-devel-0.190-2.el9.x86_64 143/156 2024-04-23T10:09:52.064 INFO:teuthology.orchestra.run.smithi114.stdout: Installing : rbd-mirror-2:19.0.0-3156.gd69f13d8.el7.x86_64 138/156 2024-04-23T10:09:52.071 INFO:teuthology.orchestra.run.smithi204.stdout: Installing : cephfs-shell-2:19.0.0-3156.gd69f13d8.el7.x86_64 144/156 2024-04-23T10:09:52.086 INFO:teuthology.orchestra.run.smithi114.stdout: Running scriptlet: rbd-mirror-2:19.0.0-3156.gd69f13d8.el7.x86_64 138/156 2024-04-23T10:09:52.086 INFO:teuthology.orchestra.run.smithi114.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-04-23T10:09:52.086 INFO:teuthology.orchestra.run.smithi114.stdout:Invalid unit name "ceph-rbd-mirror@*.service" escaped as "ceph-rbd-mirror@\x2a.service". 2024-04-23T10:09:52.086 INFO:teuthology.orchestra.run.smithi114.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-rbd-mirror.target → /usr/lib/systemd/system/ceph-rbd-mirror.target. 2024-04-23T10:09:52.086 INFO:teuthology.orchestra.run.smithi114.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-rbd-mirror.target → /usr/lib/systemd/system/ceph-rbd-mirror.target. 2024-04-23T10:09:52.087 INFO:teuthology.orchestra.run.smithi114.stdout: 2024-04-23T10:09:52.117 INFO:teuthology.orchestra.run.smithi204.stdout: Installing : postgresql-contrib-13.14-1.el9.x86_64 145/156 2024-04-23T10:09:52.147 INFO:teuthology.orchestra.run.smithi204.stdout: Installing : libcephfs-devel-2:19.0.0-3156.gd69f13d8.el7.x86_ 146/156 2024-04-23T10:09:52.767 INFO:teuthology.orchestra.run.smithi204.stdout: Running scriptlet: postgresql-server-13.14-1.el9.x86_64 147/156 2024-04-23T10:09:52.803 INFO:teuthology.orchestra.run.smithi204.stdout: Installing : postgresql-server-13.14-1.el9.x86_64 147/156 2024-04-23T10:09:52.823 INFO:teuthology.orchestra.run.smithi204.stdout: Running scriptlet: postgresql-server-13.14-1.el9.x86_64 147/156 2024-04-23T10:09:53.654 INFO:teuthology.orchestra.run.smithi204.stdout: Installing : cephfs-top-2:19.0.0-3156.gd69f13d8.el7.noarch 148/156 2024-04-23T10:09:53.742 INFO:teuthology.orchestra.run.smithi204.stdout: Installing : openssl-devel-1:3.2.1-1.el9.x86_64 149/156 2024-04-23T10:09:53.782 INFO:teuthology.orchestra.run.smithi204.stdout: Installing : libaio-devel-0.3.111-13.el9.x86_64 150/156 2024-04-23T10:09:53.863 INFO:teuthology.orchestra.run.smithi204.stdout: Installing : flex-2.6.4-9.el9.x86_64 151/156 2024-04-23T10:09:53.897 INFO:teuthology.orchestra.run.smithi204.stdout: Installing : bison-3.7.4-5.el9.x86_64 152/156 2024-04-23T10:09:53.898 INFO:teuthology.orchestra.run.smithi204.stdout: Cleanup : librbd1-2:16.2.4-5.el9.x86_64 153/156 2024-04-23T10:09:53.980 INFO:teuthology.orchestra.run.smithi204.stdout: Running scriptlet: librbd1-2:16.2.4-5.el9.x86_64 153/156 2024-04-23T10:09:54.013 INFO:teuthology.orchestra.run.smithi204.stdout: Cleanup : openssl-1:3.0.7-27.el9.x86_64 154/156 2024-04-23T10:09:54.013 INFO:teuthology.orchestra.run.smithi204.stdout: Cleanup : librados2-2:16.2.4-5.el9.x86_64 155/156 2024-04-23T10:09:54.105 INFO:teuthology.orchestra.run.smithi204.stdout: Running scriptlet: librados2-2:16.2.4-5.el9.x86_64 155/156 2024-04-23T10:09:54.178 INFO:teuthology.orchestra.run.smithi204.stdout: Cleanup : openssl-libs-1:3.0.7-27.el9.x86_64 156/156 2024-04-23T10:09:55.946 INFO:teuthology.orchestra.run.smithi119.stdout: Running scriptlet: openssl-libs-1:3.0.7-27.el9.x86_64 156/156 2024-04-23T10:09:55.946 INFO:teuthology.orchestra.run.smithi119.stdout: Verifying : ceph-2:19.0.0-3156.gd69f13d8.el7.x86_64 1/156 2024-04-23T10:09:55.946 INFO:teuthology.orchestra.run.smithi119.stdout: Verifying : ceph-base-2:19.0.0-3156.gd69f13d8.el7.x86_64 2/156 2024-04-23T10:09:55.947 INFO:teuthology.orchestra.run.smithi119.stdout: Verifying : ceph-common-2:19.0.0-3156.gd69f13d8.el7.x86_64 3/156 2024-04-23T10:09:55.947 INFO:teuthology.orchestra.run.smithi119.stdout: Verifying : ceph-fuse-2:19.0.0-3156.gd69f13d8.el7.x86_64 4/156 2024-04-23T10:09:55.947 INFO:teuthology.orchestra.run.smithi119.stdout: Verifying : ceph-immutable-object-cache-2:19.0.0-3156.gd69f1 5/156 2024-04-23T10:09:55.947 INFO:teuthology.orchestra.run.smithi119.stdout: Verifying : ceph-mds-2:19.0.0-3156.gd69f13d8.el7.x86_64 6/156 2024-04-23T10:09:55.947 INFO:teuthology.orchestra.run.smithi119.stdout: Verifying : ceph-mgr-2:19.0.0-3156.gd69f13d8.el7.x86_64 7/156 2024-04-23T10:09:55.947 INFO:teuthology.orchestra.run.smithi119.stdout: Verifying : ceph-mon-2:19.0.0-3156.gd69f13d8.el7.x86_64 8/156 2024-04-23T10:09:55.947 INFO:teuthology.orchestra.run.smithi119.stdout: Verifying : ceph-osd-2:19.0.0-3156.gd69f13d8.el7.x86_64 9/156 2024-04-23T10:09:55.947 INFO:teuthology.orchestra.run.smithi119.stdout: Verifying : ceph-radosgw-2:19.0.0-3156.gd69f13d8.el7.x86_64 10/156 2024-04-23T10:09:55.947 INFO:teuthology.orchestra.run.smithi119.stdout: Verifying : ceph-selinux-2:19.0.0-3156.gd69f13d8.el7.x86_64 11/156 2024-04-23T10:09:55.947 INFO:teuthology.orchestra.run.smithi119.stdout: Verifying : ceph-test-2:19.0.0-3156.gd69f13d8.el7.x86_64 12/156 2024-04-23T10:09:55.947 INFO:teuthology.orchestra.run.smithi119.stdout: Verifying : cephfs-mirror-2:19.0.0-3156.gd69f13d8.el7.x86_64 13/156 2024-04-23T10:09:55.947 INFO:teuthology.orchestra.run.smithi119.stdout: Verifying : cephfs-shell-2:19.0.0-3156.gd69f13d8.el7.x86_64 14/156 2024-04-23T10:09:55.947 INFO:teuthology.orchestra.run.smithi119.stdout: Verifying : libcephfs-devel-2:19.0.0-3156.gd69f13d8.el7.x86_ 15/156 2024-04-23T10:09:55.948 INFO:teuthology.orchestra.run.smithi119.stdout: Verifying : libcephfs2-2:19.0.0-3156.gd69f13d8.el7.x86_64 16/156 2024-04-23T10:09:55.948 INFO:teuthology.orchestra.run.smithi119.stdout: Verifying : libcephsqlite-2:19.0.0-3156.gd69f13d8.el7.x86_64 17/156 2024-04-23T10:09:55.948 INFO:teuthology.orchestra.run.smithi119.stdout: Verifying : librados-devel-2:19.0.0-3156.gd69f13d8.el7.x86_6 18/156 2024-04-23T10:09:55.948 INFO:teuthology.orchestra.run.smithi119.stdout: Verifying : libradosstriper1-2:19.0.0-3156.gd69f13d8.el7.x86 19/156 2024-04-23T10:09:55.948 INFO:teuthology.orchestra.run.smithi119.stdout: Verifying : librgw2-2:19.0.0-3156.gd69f13d8.el7.x86_64 20/156 2024-04-23T10:09:55.948 INFO:teuthology.orchestra.run.smithi119.stdout: Verifying : python3-ceph-argparse-2:19.0.0-3156.gd69f13d8.el 21/156 2024-04-23T10:09:55.948 INFO:teuthology.orchestra.run.smithi119.stdout: Verifying : python3-ceph-common-2:19.0.0-3156.gd69f13d8.el7. 22/156 2024-04-23T10:09:55.948 INFO:teuthology.orchestra.run.smithi119.stdout: Verifying : python3-cephfs-2:19.0.0-3156.gd69f13d8.el7.x86_6 23/156 2024-04-23T10:09:55.948 INFO:teuthology.orchestra.run.smithi119.stdout: Verifying : python3-rados-2:19.0.0-3156.gd69f13d8.el7.x86_64 24/156 2024-04-23T10:09:55.948 INFO:teuthology.orchestra.run.smithi119.stdout: Verifying : python3-rbd-2:19.0.0-3156.gd69f13d8.el7.x86_64 25/156 2024-04-23T10:09:55.948 INFO:teuthology.orchestra.run.smithi119.stdout: Verifying : python3-rgw-2:19.0.0-3156.gd69f13d8.el7.x86_64 26/156 2024-04-23T10:09:55.948 INFO:teuthology.orchestra.run.smithi119.stdout: Verifying : rbd-fuse-2:19.0.0-3156.gd69f13d8.el7.x86_64 27/156 2024-04-23T10:09:55.948 INFO:teuthology.orchestra.run.smithi119.stdout: Verifying : rbd-mirror-2:19.0.0-3156.gd69f13d8.el7.x86_64 28/156 2024-04-23T10:09:55.948 INFO:teuthology.orchestra.run.smithi119.stdout: Verifying : rbd-nbd-2:19.0.0-3156.gd69f13d8.el7.x86_64 29/156 2024-04-23T10:09:55.949 INFO:teuthology.orchestra.run.smithi119.stdout: Verifying : ceph-grafana-dashboards-2:19.0.0-3156.gd69f13d8. 30/156 2024-04-23T10:09:55.949 INFO:teuthology.orchestra.run.smithi119.stdout: Verifying : ceph-mgr-cephadm-2:19.0.0-3156.gd69f13d8.el7.noa 31/156 2024-04-23T10:09:55.949 INFO:teuthology.orchestra.run.smithi119.stdout: Verifying : ceph-mgr-dashboard-2:19.0.0-3156.gd69f13d8.el7.n 32/156 2024-04-23T10:09:55.949 INFO:teuthology.orchestra.run.smithi119.stdout: Verifying : ceph-mgr-diskprediction-local-2:19.0.0-3156.gd69 33/156 2024-04-23T10:09:55.949 INFO:teuthology.orchestra.run.smithi119.stdout: Verifying : ceph-mgr-modules-core-2:19.0.0-3156.gd69f13d8.el 34/156 2024-04-23T10:09:55.949 INFO:teuthology.orchestra.run.smithi119.stdout: Verifying : ceph-mgr-rook-2:19.0.0-3156.gd69f13d8.el7.noarch 35/156 2024-04-23T10:09:55.949 INFO:teuthology.orchestra.run.smithi119.stdout: Verifying : ceph-prometheus-alerts-2:19.0.0-3156.gd69f13d8.e 36/156 2024-04-23T10:09:55.949 INFO:teuthology.orchestra.run.smithi119.stdout: Verifying : ceph-volume-2:19.0.0-3156.gd69f13d8.el7.noarch 37/156 2024-04-23T10:09:55.949 INFO:teuthology.orchestra.run.smithi119.stdout: Verifying : cephadm-2:19.0.0-3156.gd69f13d8.el7.noarch 38/156 2024-04-23T10:09:55.949 INFO:teuthology.orchestra.run.smithi119.stdout: Verifying : cephfs-top-2:19.0.0-3156.gd69f13d8.el7.noarch 39/156 2024-04-23T10:09:55.949 INFO:teuthology.orchestra.run.smithi119.stdout: Verifying : fuse-2.9.9-15.el9.x86_64 40/156 2024-04-23T10:09:55.949 INFO:teuthology.orchestra.run.smithi119.stdout: Verifying : libgfortran-11.4.1-3.el9.x86_64 41/156 2024-04-23T10:09:55.949 INFO:teuthology.orchestra.run.smithi119.stdout: Verifying : libquadmath-11.4.1-3.el9.x86_64 42/156 2024-04-23T10:09:55.950 INFO:teuthology.orchestra.run.smithi119.stdout: Verifying : python3-cffi-1.14.5-5.el9.x86_64 43/156 2024-04-23T10:09:55.950 INFO:teuthology.orchestra.run.smithi119.stdout: Verifying : python3-cryptography-36.0.1-4.el9.x86_64 44/156 2024-04-23T10:09:55.950 INFO:teuthology.orchestra.run.smithi119.stdout: Verifying : python3-ply-3.11-14.el9.noarch 45/156 2024-04-23T10:09:55.950 INFO:teuthology.orchestra.run.smithi119.stdout: Verifying : python3-pycparser-2.20-6.el9.noarch 46/156 2024-04-23T10:09:55.950 INFO:teuthology.orchestra.run.smithi119.stdout: Verifying : python3-requests-2.25.1-8.el9.noarch 47/156 2024-04-23T10:09:55.950 INFO:teuthology.orchestra.run.smithi119.stdout: Verifying : python3-urllib3-1.26.5-5.el9.noarch 48/156 2024-04-23T10:09:55.950 INFO:teuthology.orchestra.run.smithi119.stdout: Verifying : automake-1.16.2-8.el9.noarch 49/156 2024-04-23T10:09:55.950 INFO:teuthology.orchestra.run.smithi119.stdout: Verifying : bison-3.7.4-5.el9.x86_64 50/156 2024-04-23T10:09:55.950 INFO:teuthology.orchestra.run.smithi119.stdout: Verifying : boost-program-options-1.75.0-8.el9.x86_64 51/156 2024-04-23T10:09:55.950 INFO:teuthology.orchestra.run.smithi119.stdout: Verifying : elfutils-libelf-devel-0.190-2.el9.x86_64 52/156 2024-04-23T10:09:55.950 INFO:teuthology.orchestra.run.smithi119.stdout: Verifying : flex-2.6.4-9.el9.x86_64 53/156 2024-04-23T10:09:55.950 INFO:teuthology.orchestra.run.smithi119.stdout: Verifying : flexiblas-3.0.4-8.el9.x86_64 54/156 2024-04-23T10:09:55.951 INFO:teuthology.orchestra.run.smithi119.stdout: Verifying : flexiblas-netlib-3.0.4-8.el9.x86_64 55/156 2024-04-23T10:09:55.951 INFO:teuthology.orchestra.run.smithi119.stdout: Verifying : flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 56/156 2024-04-23T10:09:55.951 INFO:teuthology.orchestra.run.smithi119.stdout: Verifying : libaio-devel-0.3.111-13.el9.x86_64 57/156 2024-04-23T10:09:55.951 INFO:teuthology.orchestra.run.smithi119.stdout: Verifying : libpmemobj-1.12.1-1.el9.x86_64 58/156 2024-04-23T10:09:55.951 INFO:teuthology.orchestra.run.smithi119.stdout: Verifying : librabbitmq-0.11.0-7.el9.x86_64 59/156 2024-04-23T10:09:55.951 INFO:teuthology.orchestra.run.smithi119.stdout: Verifying : librdkafka-1.6.1-102.el9.x86_64 60/156 2024-04-23T10:09:55.951 INFO:teuthology.orchestra.run.smithi119.stdout: Verifying : libtool-2.4.6-45.el9.x86_64 61/156 2024-04-23T10:09:55.951 INFO:teuthology.orchestra.run.smithi119.stdout: Verifying : libuuid-devel-2.37.4-18.el9.x86_64 62/156 2024-04-23T10:09:55.951 INFO:teuthology.orchestra.run.smithi119.stdout: Verifying : libzstd-devel-1.5.1-2.el9.x86_64 63/156 2024-04-23T10:09:55.951 INFO:teuthology.orchestra.run.smithi119.stdout: Verifying : lttng-ust-2.12.0-6.el9.x86_64 64/156 2024-04-23T10:09:55.951 INFO:teuthology.orchestra.run.smithi119.stdout: Verifying : lua-5.4.4-4.el9.x86_64 65/156 2024-04-23T10:09:55.951 INFO:teuthology.orchestra.run.smithi119.stdout: Verifying : openblas-0.3.21-2.el9.x86_64 66/156 2024-04-23T10:09:55.951 INFO:teuthology.orchestra.run.smithi119.stdout: Verifying : openblas-openmp-0.3.21-2.el9.x86_64 67/156 2024-04-23T10:09:55.951 INFO:teuthology.orchestra.run.smithi119.stdout: Verifying : openssl-devel-1:3.2.1-1.el9.x86_64 68/156 2024-04-23T10:09:55.952 INFO:teuthology.orchestra.run.smithi119.stdout: Verifying : postgresql-13.14-1.el9.x86_64 69/156 2024-04-23T10:09:55.952 INFO:teuthology.orchestra.run.smithi119.stdout: Verifying : postgresql-contrib-13.14-1.el9.x86_64 70/156 2024-04-23T10:09:55.952 INFO:teuthology.orchestra.run.smithi119.stdout: Verifying : postgresql-private-libs-13.14-1.el9.x86_64 71/156 2024-04-23T10:09:55.952 INFO:teuthology.orchestra.run.smithi119.stdout: Verifying : postgresql-server-13.14-1.el9.x86_64 72/156 2024-04-23T10:09:55.952 INFO:teuthology.orchestra.run.smithi119.stdout: Verifying : protobuf-3.14.0-13.el9.x86_64 73/156 2024-04-23T10:09:55.952 INFO:teuthology.orchestra.run.smithi119.stdout: Verifying : python3-attrs-20.3.0-7.el9.noarch 74/156 2024-04-23T10:09:55.952 INFO:teuthology.orchestra.run.smithi119.stdout: Verifying : python3-babel-2.9.1-2.el9.noarch 75/156 2024-04-23T10:09:55.952 INFO:teuthology.orchestra.run.smithi119.stdout: Verifying : python3-devel-3.9.18-3.el9.x86_64 76/156 2024-04-23T10:09:55.952 INFO:teuthology.orchestra.run.smithi119.stdout: Verifying : python3-jinja2-2.11.3-5.el9.noarch 77/156 2024-04-23T10:09:55.952 INFO:teuthology.orchestra.run.smithi119.stdout: Verifying : python3-jsonpatch-1.21-16.el9.noarch 78/156 2024-04-23T10:09:55.952 INFO:teuthology.orchestra.run.smithi119.stdout: Verifying : python3-jsonpointer-2.0-4.el9.noarch 79/156 2024-04-23T10:09:55.952 INFO:teuthology.orchestra.run.smithi119.stdout: Verifying : python3-mako-1.1.4-6.el9.noarch 80/156 2024-04-23T10:09:55.952 INFO:teuthology.orchestra.run.smithi119.stdout: Verifying : python3-markupsafe-1.1.1-12.el9.x86_64 81/156 2024-04-23T10:09:55.953 INFO:teuthology.orchestra.run.smithi119.stdout: Verifying : python3-numpy-1:1.20.1-5.el9.x86_64 82/156 2024-04-23T10:09:55.953 INFO:teuthology.orchestra.run.smithi119.stdout: Verifying : python3-numpy-f2py-1:1.20.1-5.el9.x86_64 83/156 2024-04-23T10:09:55.953 INFO:teuthology.orchestra.run.smithi119.stdout: Verifying : python3-oauthlib-3.1.1-5.el9.noarch 84/156 2024-04-23T10:09:55.953 INFO:teuthology.orchestra.run.smithi119.stdout: Verifying : python3-packaging-20.9-5.el9.noarch 85/156 2024-04-23T10:09:55.953 INFO:teuthology.orchestra.run.smithi119.stdout: Verifying : python3-prettytable-0.7.2-27.el9.noarch 86/156 2024-04-23T10:09:55.953 INFO:teuthology.orchestra.run.smithi119.stdout: Verifying : python3-protobuf-3.14.0-13.el9.noarch 87/156 2024-04-23T10:09:55.953 INFO:teuthology.orchestra.run.smithi119.stdout: Verifying : python3-pyasn1-0.4.8-6.el9.noarch 88/156 2024-04-23T10:09:55.953 INFO:teuthology.orchestra.run.smithi119.stdout: Verifying : python3-pyasn1-modules-0.4.8-6.el9.noarch 89/156 2024-04-23T10:09:55.953 INFO:teuthology.orchestra.run.smithi119.stdout: Verifying : python3-pytz-2021.1-5.el9.noarch 90/156 2024-04-23T10:09:55.953 INFO:teuthology.orchestra.run.smithi119.stdout: Verifying : python3-requests-oauthlib-1.3.0-12.el9.noarch 91/156 2024-04-23T10:09:55.953 INFO:teuthology.orchestra.run.smithi119.stdout: Verifying : python3-scipy-1.6.2-8.el9.x86_64 92/156 2024-04-23T10:09:55.953 INFO:teuthology.orchestra.run.smithi119.stdout: Verifying : python3-toml-0.10.2-6.el9.noarch 93/156 2024-04-23T10:09:55.953 INFO:teuthology.orchestra.run.smithi119.stdout: Verifying : python3-wcwidth-0.2.5-8.el9.noarch 94/156 2024-04-23T10:09:55.953 INFO:teuthology.orchestra.run.smithi119.stdout: Verifying : qatlib-24.02.0-1.el9.x86_64 95/156 2024-04-23T10:09:55.954 INFO:teuthology.orchestra.run.smithi119.stdout: Verifying : qatlib-service-24.02.0-1.el9.x86_64 96/156 2024-04-23T10:09:55.954 INFO:teuthology.orchestra.run.smithi119.stdout: Verifying : qatzip-libs-1.2.0-1.el9.x86_64 97/156 2024-04-23T10:09:55.954 INFO:teuthology.orchestra.run.smithi119.stdout: Verifying : socat-1.7.4.1-5.el9.x86_64 98/156 2024-04-23T10:09:55.954 INFO:teuthology.orchestra.run.smithi119.stdout: Verifying : uuid-1.6.2-55.el9.x86_64 99/156 2024-04-23T10:09:55.954 INFO:teuthology.orchestra.run.smithi119.stdout: Verifying : xfsprogs-devel-6.3.0-1.el9.x86_64 100/156 2024-04-23T10:09:55.954 INFO:teuthology.orchestra.run.smithi119.stdout: Verifying : xmlstarlet-1.6.1-20.el9.x86_64 101/156 2024-04-23T10:09:55.954 INFO:teuthology.orchestra.run.smithi119.stdout: Verifying : zlib-devel-1.2.11-41.el9.x86_64 102/156 2024-04-23T10:09:55.954 INFO:teuthology.orchestra.run.smithi119.stdout: Verifying : lua-devel-5.4.4-4.el9.x86_64 103/156 2024-04-23T10:09:55.954 INFO:teuthology.orchestra.run.smithi119.stdout: Verifying : protobuf-compiler-3.14.0-13.el9.x86_64 104/156 2024-04-23T10:09:55.954 INFO:teuthology.orchestra.run.smithi119.stdout: Verifying : abseil-cpp-20211102.0-3.el9.x86_64 105/156 2024-04-23T10:09:55.954 INFO:teuthology.orchestra.run.smithi119.stdout: Verifying : gperftools-libs-2.9.1-2.el9.x86_64 106/156 2024-04-23T10:09:55.954 INFO:teuthology.orchestra.run.smithi119.stdout: Verifying : grpc-data-1.46.7-10.el9.noarch 107/156 2024-04-23T10:09:55.954 INFO:teuthology.orchestra.run.smithi119.stdout: Verifying : libarrow-9.0.0-10.el9.x86_64 108/156 2024-04-23T10:09:55.955 INFO:teuthology.orchestra.run.smithi119.stdout: Verifying : libarrow-doc-9.0.0-10.el9.noarch 109/156 2024-04-23T10:09:55.955 INFO:teuthology.orchestra.run.smithi119.stdout: Verifying : liboath-2.6.7-2.el9.x86_64 110/156 2024-04-23T10:09:55.955 INFO:teuthology.orchestra.run.smithi119.stdout: Verifying : libunwind-1.6.2-1.el9.x86_64 111/156 2024-04-23T10:09:55.955 INFO:teuthology.orchestra.run.smithi119.stdout: Verifying : luarocks-3.9.2-1.el9.noarch 112/156 2024-04-23T10:09:55.955 INFO:teuthology.orchestra.run.smithi119.stdout: Verifying : parquet-libs-9.0.0-10.el9.x86_64 113/156 2024-04-23T10:09:55.955 INFO:teuthology.orchestra.run.smithi119.stdout: Verifying : python3-asyncssh-2.13.2-5.el9.noarch 114/156 2024-04-23T10:09:55.955 INFO:teuthology.orchestra.run.smithi119.stdout: Verifying : python3-bcrypt-3.2.2-1.el9.x86_64 115/156 2024-04-23T10:09:55.955 INFO:teuthology.orchestra.run.smithi119.stdout: Verifying : python3-cachetools-4.2.4-1.el9.noarch 116/156 2024-04-23T10:09:55.955 INFO:teuthology.orchestra.run.smithi119.stdout: Verifying : python3-certifi-2021.10.8-2.el9.noarch 117/156 2024-04-23T10:09:55.955 INFO:teuthology.orchestra.run.smithi119.stdout: Verifying : python3-cheroot-8.6.0-4.el9.noarch 118/156 2024-04-23T10:09:55.955 INFO:teuthology.orchestra.run.smithi119.stdout: Verifying : python3-cherrypy-18.6.1-2.el9.noarch 119/156 2024-04-23T10:09:55.955 INFO:teuthology.orchestra.run.smithi119.stdout: Verifying : python3-cmd2-2.4.2-2.el9.noarch 120/156 2024-04-23T10:09:55.955 INFO:teuthology.orchestra.run.smithi119.stdout: Verifying : python3-colorama-0.4.4-12.el9.noarch 121/156 2024-04-23T10:09:55.955 INFO:teuthology.orchestra.run.smithi119.stdout: Verifying : python3-google-auth-1:2.29.0-1.el9.noarch 122/156 2024-04-23T10:09:55.955 INFO:teuthology.orchestra.run.smithi119.stdout: Verifying : python3-grpcio-1.46.7-10.el9.x86_64 123/156 2024-04-23T10:09:55.956 INFO:teuthology.orchestra.run.smithi119.stdout: Verifying : python3-grpcio-tools-1.46.7-10.el9.x86_64 124/156 2024-04-23T10:09:55.956 INFO:teuthology.orchestra.run.smithi119.stdout: Verifying : python3-jaraco-8.2.1-3.el9.noarch 125/156 2024-04-23T10:09:55.956 INFO:teuthology.orchestra.run.smithi119.stdout: Verifying : python3-jaraco-classes-3.2.1-5.el9.noarch 126/156 2024-04-23T10:09:55.956 INFO:teuthology.orchestra.run.smithi119.stdout: Verifying : python3-jaraco-collections-3.0.0-8.el9.noarch 127/156 2024-04-23T10:09:55.956 INFO:teuthology.orchestra.run.smithi119.stdout: Verifying : python3-jaraco-functools-3.5.0-2.el9.noarch 128/156 2024-04-23T10:09:55.956 INFO:teuthology.orchestra.run.smithi119.stdout: Verifying : python3-jaraco-text-3.2.0-6.el9.noarch 129/156 2024-04-23T10:09:55.956 INFO:teuthology.orchestra.run.smithi119.stdout: Verifying : python3-kubernetes-1:26.1.0-2.el9.noarch 130/156 2024-04-23T10:09:55.956 INFO:teuthology.orchestra.run.smithi119.stdout: Verifying : python3-logutils-0.3.5-21.el9.noarch 131/156 2024-04-23T10:09:55.956 INFO:teuthology.orchestra.run.smithi119.stdout: Verifying : python3-more-itertools-8.12.0-2.el9.noarch 132/156 2024-04-23T10:09:55.956 INFO:teuthology.orchestra.run.smithi119.stdout: Verifying : python3-natsort-7.1.1-5.el9.noarch 133/156 2024-04-23T10:09:55.956 INFO:teuthology.orchestra.run.smithi119.stdout: Verifying : python3-pecan-1.4.2-3.el9.noarch 134/156 2024-04-23T10:09:55.956 INFO:teuthology.orchestra.run.smithi119.stdout: Verifying : python3-portend-3.1.0-2.el9.noarch 135/156 2024-04-23T10:09:55.956 INFO:teuthology.orchestra.run.smithi119.stdout: Verifying : python3-pyOpenSSL-21.0.0-1.el9.noarch 136/156 2024-04-23T10:09:55.956 INFO:teuthology.orchestra.run.smithi119.stdout: Verifying : python3-pyperclip-1.8.2-2.el9.noarch 137/156 2024-04-23T10:09:55.956 INFO:teuthology.orchestra.run.smithi119.stdout: Verifying : python3-repoze-lru-0.7-16.el9.noarch 138/156 2024-04-23T10:09:55.957 INFO:teuthology.orchestra.run.smithi119.stdout: Verifying : python3-routes-2.5.1-5.el9.noarch 139/156 2024-04-23T10:09:55.957 INFO:teuthology.orchestra.run.smithi119.stdout: Verifying : python3-rsa-4.9-2.el9.noarch 140/156 2024-04-23T10:09:55.957 INFO:teuthology.orchestra.run.smithi119.stdout: Verifying : python3-tempora-5.0.0-2.el9.noarch 141/156 2024-04-23T10:09:55.957 INFO:teuthology.orchestra.run.smithi119.stdout: Verifying : python3-typing-extensions-4.4.0-2.el9.noarch 142/156 2024-04-23T10:09:55.957 INFO:teuthology.orchestra.run.smithi119.stdout: Verifying : python3-webob-1.8.7-6.el9.noarch 143/156 2024-04-23T10:09:55.957 INFO:teuthology.orchestra.run.smithi119.stdout: Verifying : python3-websocket-client-1.2.3-2.el9.noarch 144/156 2024-04-23T10:09:55.957 INFO:teuthology.orchestra.run.smithi119.stdout: Verifying : python3-werkzeug-2.0.3-3.el9.1.noarch 145/156 2024-04-23T10:09:55.957 INFO:teuthology.orchestra.run.smithi119.stdout: Verifying : python3-zc-lockfile-2.0-10.el9.noarch 146/156 2024-04-23T10:09:55.957 INFO:teuthology.orchestra.run.smithi119.stdout: Verifying : re2-1:20211101-3.el9.x86_64 147/156 2024-04-23T10:09:55.957 INFO:teuthology.orchestra.run.smithi119.stdout: Verifying : thrift-0.15.0-2.el9.x86_64 148/156 2024-04-23T10:09:55.957 INFO:teuthology.orchestra.run.smithi119.stdout: Verifying : librados2-2:19.0.0-3156.gd69f13d8.el7.x86_64 149/156 2024-04-23T10:09:55.957 INFO:teuthology.orchestra.run.smithi119.stdout: Verifying : librados2-2:16.2.4-5.el9.x86_64 150/156 2024-04-23T10:09:55.957 INFO:teuthology.orchestra.run.smithi119.stdout: Verifying : librbd1-2:19.0.0-3156.gd69f13d8.el7.x86_64 151/156 2024-04-23T10:09:55.957 INFO:teuthology.orchestra.run.smithi119.stdout: Verifying : librbd1-2:16.2.4-5.el9.x86_64 152/156 2024-04-23T10:09:55.958 INFO:teuthology.orchestra.run.smithi119.stdout: Verifying : openssl-1:3.2.1-1.el9.x86_64 153/156 2024-04-23T10:09:55.958 INFO:teuthology.orchestra.run.smithi119.stdout: Verifying : openssl-1:3.0.7-27.el9.x86_64 154/156 2024-04-23T10:09:55.958 INFO:teuthology.orchestra.run.smithi119.stdout: Verifying : openssl-libs-1:3.2.1-1.el9.x86_64 155/156 2024-04-23T10:09:56.735 INFO:teuthology.orchestra.run.smithi114.stdout: Installing : ceph-test-2:19.0.0-3156.gd69f13d8.el7.x86_64 139/156 2024-04-23T10:09:56.776 INFO:teuthology.orchestra.run.smithi114.stdout: Installing : rbd-fuse-2:19.0.0-3156.gd69f13d8.el7.x86_64 140/156 2024-04-23T10:09:56.817 INFO:teuthology.orchestra.run.smithi114.stdout: Installing : rbd-nbd-2:19.0.0-3156.gd69f13d8.el7.x86_64 141/156 2024-04-23T10:09:56.849 INFO:teuthology.orchestra.run.smithi114.stdout: Installing : xfsprogs-devel-6.3.0-1.el9.x86_64 142/156 2024-04-23T10:09:56.876 INFO:teuthology.orchestra.run.smithi114.stdout: Installing : elfutils-libelf-devel-0.190-2.el9.x86_64 143/156 2024-04-23T10:09:57.057 INFO:teuthology.orchestra.run.smithi114.stdout: Installing : cephfs-shell-2:19.0.0-3156.gd69f13d8.el7.x86_64 144/156 2024-04-23T10:09:57.105 INFO:teuthology.orchestra.run.smithi114.stdout: Installing : postgresql-contrib-13.14-1.el9.x86_64 145/156 2024-04-23T10:09:57.135 INFO:teuthology.orchestra.run.smithi114.stdout: Installing : libcephfs-devel-2:19.0.0-3156.gd69f13d8.el7.x86_ 146/156 2024-04-23T10:09:57.705 INFO:teuthology.orchestra.run.smithi114.stdout: Running scriptlet: postgresql-server-13.14-1.el9.x86_64 147/156 2024-04-23T10:09:57.741 INFO:teuthology.orchestra.run.smithi114.stdout: Installing : postgresql-server-13.14-1.el9.x86_64 147/156 2024-04-23T10:09:57.760 INFO:teuthology.orchestra.run.smithi114.stdout: Running scriptlet: postgresql-server-13.14-1.el9.x86_64 147/156 2024-04-23T10:09:58.057 INFO:teuthology.orchestra.run.smithi119.stdout: Verifying : openssl-libs-1:3.0.7-27.el9.x86_64 156/156 2024-04-23T10:09:58.057 INFO:teuthology.orchestra.run.smithi119.stdout: 2024-04-23T10:09:58.057 INFO:teuthology.orchestra.run.smithi119.stdout:Upgraded: 2024-04-23T10:09:58.058 INFO:teuthology.orchestra.run.smithi119.stdout: librados2-2:19.0.0-3156.gd69f13d8.el7.x86_64 2024-04-23T10:09:58.058 INFO:teuthology.orchestra.run.smithi119.stdout: librbd1-2:19.0.0-3156.gd69f13d8.el7.x86_64 2024-04-23T10:09:58.058 INFO:teuthology.orchestra.run.smithi119.stdout: openssl-1:3.2.1-1.el9.x86_64 2024-04-23T10:09:58.058 INFO:teuthology.orchestra.run.smithi119.stdout: openssl-libs-1:3.2.1-1.el9.x86_64 2024-04-23T10:09:58.058 INFO:teuthology.orchestra.run.smithi119.stdout:Installed: 2024-04-23T10:09:58.058 INFO:teuthology.orchestra.run.smithi119.stdout: abseil-cpp-20211102.0-3.el9.x86_64 2024-04-23T10:09:58.058 INFO:teuthology.orchestra.run.smithi119.stdout: automake-1.16.2-8.el9.noarch 2024-04-23T10:09:58.058 INFO:teuthology.orchestra.run.smithi119.stdout: bison-3.7.4-5.el9.x86_64 2024-04-23T10:09:58.058 INFO:teuthology.orchestra.run.smithi119.stdout: boost-program-options-1.75.0-8.el9.x86_64 2024-04-23T10:09:58.058 INFO:teuthology.orchestra.run.smithi119.stdout: ceph-2:19.0.0-3156.gd69f13d8.el7.x86_64 2024-04-23T10:09:58.058 INFO:teuthology.orchestra.run.smithi119.stdout: ceph-base-2:19.0.0-3156.gd69f13d8.el7.x86_64 2024-04-23T10:09:58.058 INFO:teuthology.orchestra.run.smithi119.stdout: ceph-common-2:19.0.0-3156.gd69f13d8.el7.x86_64 2024-04-23T10:09:58.058 INFO:teuthology.orchestra.run.smithi119.stdout: ceph-fuse-2:19.0.0-3156.gd69f13d8.el7.x86_64 2024-04-23T10:09:58.059 INFO:teuthology.orchestra.run.smithi119.stdout: ceph-grafana-dashboards-2:19.0.0-3156.gd69f13d8.el7.noarch 2024-04-23T10:09:58.059 INFO:teuthology.orchestra.run.smithi119.stdout: ceph-immutable-object-cache-2:19.0.0-3156.gd69f13d8.el7.x86_64 2024-04-23T10:09:58.059 INFO:teuthology.orchestra.run.smithi119.stdout: ceph-mds-2:19.0.0-3156.gd69f13d8.el7.x86_64 2024-04-23T10:09:58.059 INFO:teuthology.orchestra.run.smithi119.stdout: ceph-mgr-2:19.0.0-3156.gd69f13d8.el7.x86_64 2024-04-23T10:09:58.059 INFO:teuthology.orchestra.run.smithi119.stdout: ceph-mgr-cephadm-2:19.0.0-3156.gd69f13d8.el7.noarch 2024-04-23T10:09:58.059 INFO:teuthology.orchestra.run.smithi119.stdout: ceph-mgr-dashboard-2:19.0.0-3156.gd69f13d8.el7.noarch 2024-04-23T10:09:58.059 INFO:teuthology.orchestra.run.smithi119.stdout: ceph-mgr-diskprediction-local-2:19.0.0-3156.gd69f13d8.el7.noarch 2024-04-23T10:09:58.059 INFO:teuthology.orchestra.run.smithi119.stdout: ceph-mgr-modules-core-2:19.0.0-3156.gd69f13d8.el7.noarch 2024-04-23T10:09:58.059 INFO:teuthology.orchestra.run.smithi119.stdout: ceph-mgr-rook-2:19.0.0-3156.gd69f13d8.el7.noarch 2024-04-23T10:09:58.059 INFO:teuthology.orchestra.run.smithi119.stdout: ceph-mon-2:19.0.0-3156.gd69f13d8.el7.x86_64 2024-04-23T10:09:58.059 INFO:teuthology.orchestra.run.smithi119.stdout: ceph-osd-2:19.0.0-3156.gd69f13d8.el7.x86_64 2024-04-23T10:09:58.059 INFO:teuthology.orchestra.run.smithi119.stdout: ceph-prometheus-alerts-2:19.0.0-3156.gd69f13d8.el7.noarch 2024-04-23T10:09:58.059 INFO:teuthology.orchestra.run.smithi119.stdout: ceph-radosgw-2:19.0.0-3156.gd69f13d8.el7.x86_64 2024-04-23T10:09:58.059 INFO:teuthology.orchestra.run.smithi119.stdout: ceph-selinux-2:19.0.0-3156.gd69f13d8.el7.x86_64 2024-04-23T10:09:58.059 INFO:teuthology.orchestra.run.smithi119.stdout: ceph-test-2:19.0.0-3156.gd69f13d8.el7.x86_64 2024-04-23T10:09:58.060 INFO:teuthology.orchestra.run.smithi119.stdout: ceph-volume-2:19.0.0-3156.gd69f13d8.el7.noarch 2024-04-23T10:09:58.060 INFO:teuthology.orchestra.run.smithi119.stdout: cephadm-2:19.0.0-3156.gd69f13d8.el7.noarch 2024-04-23T10:09:58.060 INFO:teuthology.orchestra.run.smithi119.stdout: cephfs-mirror-2:19.0.0-3156.gd69f13d8.el7.x86_64 2024-04-23T10:09:58.060 INFO:teuthology.orchestra.run.smithi119.stdout: cephfs-shell-2:19.0.0-3156.gd69f13d8.el7.x86_64 2024-04-23T10:09:58.060 INFO:teuthology.orchestra.run.smithi119.stdout: cephfs-top-2:19.0.0-3156.gd69f13d8.el7.noarch 2024-04-23T10:09:58.060 INFO:teuthology.orchestra.run.smithi119.stdout: elfutils-libelf-devel-0.190-2.el9.x86_64 2024-04-23T10:09:58.060 INFO:teuthology.orchestra.run.smithi119.stdout: flex-2.6.4-9.el9.x86_64 2024-04-23T10:09:58.060 INFO:teuthology.orchestra.run.smithi119.stdout: flexiblas-3.0.4-8.el9.x86_64 2024-04-23T10:09:58.060 INFO:teuthology.orchestra.run.smithi119.stdout: flexiblas-netlib-3.0.4-8.el9.x86_64 2024-04-23T10:09:58.060 INFO:teuthology.orchestra.run.smithi119.stdout: flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 2024-04-23T10:09:58.060 INFO:teuthology.orchestra.run.smithi119.stdout: fuse-2.9.9-15.el9.x86_64 2024-04-23T10:09:58.060 INFO:teuthology.orchestra.run.smithi119.stdout: gperftools-libs-2.9.1-2.el9.x86_64 2024-04-23T10:09:58.060 INFO:teuthology.orchestra.run.smithi119.stdout: grpc-data-1.46.7-10.el9.noarch 2024-04-23T10:09:58.060 INFO:teuthology.orchestra.run.smithi119.stdout: libaio-devel-0.3.111-13.el9.x86_64 2024-04-23T10:09:58.061 INFO:teuthology.orchestra.run.smithi119.stdout: libarrow-9.0.0-10.el9.x86_64 2024-04-23T10:09:58.061 INFO:teuthology.orchestra.run.smithi119.stdout: libarrow-doc-9.0.0-10.el9.noarch 2024-04-23T10:09:58.061 INFO:teuthology.orchestra.run.smithi119.stdout: libcephfs-devel-2:19.0.0-3156.gd69f13d8.el7.x86_64 2024-04-23T10:09:58.061 INFO:teuthology.orchestra.run.smithi119.stdout: libcephfs2-2:19.0.0-3156.gd69f13d8.el7.x86_64 2024-04-23T10:09:58.061 INFO:teuthology.orchestra.run.smithi119.stdout: libcephsqlite-2:19.0.0-3156.gd69f13d8.el7.x86_64 2024-04-23T10:09:58.061 INFO:teuthology.orchestra.run.smithi119.stdout: libgfortran-11.4.1-3.el9.x86_64 2024-04-23T10:09:58.062 INFO:teuthology.orchestra.run.smithi119.stdout: liboath-2.6.7-2.el9.x86_64 2024-04-23T10:09:58.062 INFO:teuthology.orchestra.run.smithi119.stdout: libpmemobj-1.12.1-1.el9.x86_64 2024-04-23T10:09:58.062 INFO:teuthology.orchestra.run.smithi119.stdout: libquadmath-11.4.1-3.el9.x86_64 2024-04-23T10:09:58.062 INFO:teuthology.orchestra.run.smithi119.stdout: librabbitmq-0.11.0-7.el9.x86_64 2024-04-23T10:09:58.062 INFO:teuthology.orchestra.run.smithi119.stdout: librados-devel-2:19.0.0-3156.gd69f13d8.el7.x86_64 2024-04-23T10:09:58.062 INFO:teuthology.orchestra.run.smithi119.stdout: libradosstriper1-2:19.0.0-3156.gd69f13d8.el7.x86_64 2024-04-23T10:09:58.062 INFO:teuthology.orchestra.run.smithi119.stdout: librdkafka-1.6.1-102.el9.x86_64 2024-04-23T10:09:58.062 INFO:teuthology.orchestra.run.smithi119.stdout: librgw2-2:19.0.0-3156.gd69f13d8.el7.x86_64 2024-04-23T10:09:58.062 INFO:teuthology.orchestra.run.smithi119.stdout: libtool-2.4.6-45.el9.x86_64 2024-04-23T10:09:58.062 INFO:teuthology.orchestra.run.smithi119.stdout: libunwind-1.6.2-1.el9.x86_64 2024-04-23T10:09:58.062 INFO:teuthology.orchestra.run.smithi119.stdout: libuuid-devel-2.37.4-18.el9.x86_64 2024-04-23T10:09:58.062 INFO:teuthology.orchestra.run.smithi119.stdout: libzstd-devel-1.5.1-2.el9.x86_64 2024-04-23T10:09:58.062 INFO:teuthology.orchestra.run.smithi119.stdout: lttng-ust-2.12.0-6.el9.x86_64 2024-04-23T10:09:58.062 INFO:teuthology.orchestra.run.smithi119.stdout: lua-5.4.4-4.el9.x86_64 2024-04-23T10:09:58.063 INFO:teuthology.orchestra.run.smithi119.stdout: lua-devel-5.4.4-4.el9.x86_64 2024-04-23T10:09:58.063 INFO:teuthology.orchestra.run.smithi119.stdout: luarocks-3.9.2-1.el9.noarch 2024-04-23T10:09:58.063 INFO:teuthology.orchestra.run.smithi119.stdout: openblas-0.3.21-2.el9.x86_64 2024-04-23T10:09:58.063 INFO:teuthology.orchestra.run.smithi119.stdout: openblas-openmp-0.3.21-2.el9.x86_64 2024-04-23T10:09:58.063 INFO:teuthology.orchestra.run.smithi119.stdout: openssl-devel-1:3.2.1-1.el9.x86_64 2024-04-23T10:09:58.063 INFO:teuthology.orchestra.run.smithi119.stdout: parquet-libs-9.0.0-10.el9.x86_64 2024-04-23T10:09:58.063 INFO:teuthology.orchestra.run.smithi119.stdout: postgresql-13.14-1.el9.x86_64 2024-04-23T10:09:58.063 INFO:teuthology.orchestra.run.smithi119.stdout: postgresql-contrib-13.14-1.el9.x86_64 2024-04-23T10:09:58.063 INFO:teuthology.orchestra.run.smithi119.stdout: postgresql-private-libs-13.14-1.el9.x86_64 2024-04-23T10:09:58.063 INFO:teuthology.orchestra.run.smithi119.stdout: postgresql-server-13.14-1.el9.x86_64 2024-04-23T10:09:58.063 INFO:teuthology.orchestra.run.smithi119.stdout: protobuf-3.14.0-13.el9.x86_64 2024-04-23T10:09:58.063 INFO:teuthology.orchestra.run.smithi119.stdout: protobuf-compiler-3.14.0-13.el9.x86_64 2024-04-23T10:09:58.063 INFO:teuthology.orchestra.run.smithi119.stdout: python3-asyncssh-2.13.2-5.el9.noarch 2024-04-23T10:09:58.063 INFO:teuthology.orchestra.run.smithi119.stdout: python3-attrs-20.3.0-7.el9.noarch 2024-04-23T10:09:58.063 INFO:teuthology.orchestra.run.smithi119.stdout: python3-babel-2.9.1-2.el9.noarch 2024-04-23T10:09:58.064 INFO:teuthology.orchestra.run.smithi119.stdout: python3-bcrypt-3.2.2-1.el9.x86_64 2024-04-23T10:09:58.064 INFO:teuthology.orchestra.run.smithi119.stdout: python3-cachetools-4.2.4-1.el9.noarch 2024-04-23T10:09:58.064 INFO:teuthology.orchestra.run.smithi119.stdout: python3-ceph-argparse-2:19.0.0-3156.gd69f13d8.el7.x86_64 2024-04-23T10:09:58.064 INFO:teuthology.orchestra.run.smithi119.stdout: python3-ceph-common-2:19.0.0-3156.gd69f13d8.el7.x86_64 2024-04-23T10:09:58.064 INFO:teuthology.orchestra.run.smithi119.stdout: python3-cephfs-2:19.0.0-3156.gd69f13d8.el7.x86_64 2024-04-23T10:09:58.064 INFO:teuthology.orchestra.run.smithi119.stdout: python3-certifi-2021.10.8-2.el9.noarch 2024-04-23T10:09:58.064 INFO:teuthology.orchestra.run.smithi119.stdout: python3-cffi-1.14.5-5.el9.x86_64 2024-04-23T10:09:58.064 INFO:teuthology.orchestra.run.smithi119.stdout: python3-cheroot-8.6.0-4.el9.noarch 2024-04-23T10:09:58.064 INFO:teuthology.orchestra.run.smithi119.stdout: python3-cherrypy-18.6.1-2.el9.noarch 2024-04-23T10:09:58.064 INFO:teuthology.orchestra.run.smithi119.stdout: python3-cmd2-2.4.2-2.el9.noarch 2024-04-23T10:09:58.064 INFO:teuthology.orchestra.run.smithi119.stdout: python3-colorama-0.4.4-12.el9.noarch 2024-04-23T10:09:58.064 INFO:teuthology.orchestra.run.smithi119.stdout: python3-cryptography-36.0.1-4.el9.x86_64 2024-04-23T10:09:58.064 INFO:teuthology.orchestra.run.smithi119.stdout: python3-devel-3.9.18-3.el9.x86_64 2024-04-23T10:09:58.064 INFO:teuthology.orchestra.run.smithi119.stdout: python3-google-auth-1:2.29.0-1.el9.noarch 2024-04-23T10:09:58.065 INFO:teuthology.orchestra.run.smithi119.stdout: python3-grpcio-1.46.7-10.el9.x86_64 2024-04-23T10:09:58.065 INFO:teuthology.orchestra.run.smithi119.stdout: python3-grpcio-tools-1.46.7-10.el9.x86_64 2024-04-23T10:09:58.065 INFO:teuthology.orchestra.run.smithi119.stdout: python3-jaraco-8.2.1-3.el9.noarch 2024-04-23T10:09:58.065 INFO:teuthology.orchestra.run.smithi119.stdout: python3-jaraco-classes-3.2.1-5.el9.noarch 2024-04-23T10:09:58.065 INFO:teuthology.orchestra.run.smithi119.stdout: python3-jaraco-collections-3.0.0-8.el9.noarch 2024-04-23T10:09:58.065 INFO:teuthology.orchestra.run.smithi119.stdout: python3-jaraco-functools-3.5.0-2.el9.noarch 2024-04-23T10:09:58.065 INFO:teuthology.orchestra.run.smithi119.stdout: python3-jaraco-text-3.2.0-6.el9.noarch 2024-04-23T10:09:58.065 INFO:teuthology.orchestra.run.smithi119.stdout: python3-jinja2-2.11.3-5.el9.noarch 2024-04-23T10:09:58.065 INFO:teuthology.orchestra.run.smithi119.stdout: python3-jsonpatch-1.21-16.el9.noarch 2024-04-23T10:09:58.065 INFO:teuthology.orchestra.run.smithi119.stdout: python3-jsonpointer-2.0-4.el9.noarch 2024-04-23T10:09:58.065 INFO:teuthology.orchestra.run.smithi119.stdout: python3-kubernetes-1:26.1.0-2.el9.noarch 2024-04-23T10:09:58.065 INFO:teuthology.orchestra.run.smithi119.stdout: python3-logutils-0.3.5-21.el9.noarch 2024-04-23T10:09:58.065 INFO:teuthology.orchestra.run.smithi119.stdout: python3-mako-1.1.4-6.el9.noarch 2024-04-23T10:09:58.065 INFO:teuthology.orchestra.run.smithi119.stdout: python3-markupsafe-1.1.1-12.el9.x86_64 2024-04-23T10:09:58.065 INFO:teuthology.orchestra.run.smithi119.stdout: python3-more-itertools-8.12.0-2.el9.noarch 2024-04-23T10:09:58.066 INFO:teuthology.orchestra.run.smithi119.stdout: python3-natsort-7.1.1-5.el9.noarch 2024-04-23T10:09:58.066 INFO:teuthology.orchestra.run.smithi119.stdout: python3-numpy-1:1.20.1-5.el9.x86_64 2024-04-23T10:09:58.066 INFO:teuthology.orchestra.run.smithi119.stdout: python3-numpy-f2py-1:1.20.1-5.el9.x86_64 2024-04-23T10:09:58.066 INFO:teuthology.orchestra.run.smithi119.stdout: python3-oauthlib-3.1.1-5.el9.noarch 2024-04-23T10:09:58.066 INFO:teuthology.orchestra.run.smithi119.stdout: python3-packaging-20.9-5.el9.noarch 2024-04-23T10:09:58.066 INFO:teuthology.orchestra.run.smithi119.stdout: python3-pecan-1.4.2-3.el9.noarch 2024-04-23T10:09:58.066 INFO:teuthology.orchestra.run.smithi119.stdout: python3-ply-3.11-14.el9.noarch 2024-04-23T10:09:58.066 INFO:teuthology.orchestra.run.smithi119.stdout: python3-portend-3.1.0-2.el9.noarch 2024-04-23T10:09:58.066 INFO:teuthology.orchestra.run.smithi119.stdout: python3-prettytable-0.7.2-27.el9.noarch 2024-04-23T10:09:58.066 INFO:teuthology.orchestra.run.smithi119.stdout: python3-protobuf-3.14.0-13.el9.noarch 2024-04-23T10:09:58.066 INFO:teuthology.orchestra.run.smithi119.stdout: python3-pyOpenSSL-21.0.0-1.el9.noarch 2024-04-23T10:09:58.066 INFO:teuthology.orchestra.run.smithi119.stdout: python3-pyasn1-0.4.8-6.el9.noarch 2024-04-23T10:09:58.066 INFO:teuthology.orchestra.run.smithi119.stdout: python3-pyasn1-modules-0.4.8-6.el9.noarch 2024-04-23T10:09:58.066 INFO:teuthology.orchestra.run.smithi119.stdout: python3-pycparser-2.20-6.el9.noarch 2024-04-23T10:09:58.067 INFO:teuthology.orchestra.run.smithi119.stdout: python3-pyperclip-1.8.2-2.el9.noarch 2024-04-23T10:09:58.067 INFO:teuthology.orchestra.run.smithi119.stdout: python3-pytz-2021.1-5.el9.noarch 2024-04-23T10:09:58.067 INFO:teuthology.orchestra.run.smithi119.stdout: python3-rados-2:19.0.0-3156.gd69f13d8.el7.x86_64 2024-04-23T10:09:58.067 INFO:teuthology.orchestra.run.smithi119.stdout: python3-rbd-2:19.0.0-3156.gd69f13d8.el7.x86_64 2024-04-23T10:09:58.067 INFO:teuthology.orchestra.run.smithi119.stdout: python3-repoze-lru-0.7-16.el9.noarch 2024-04-23T10:09:58.067 INFO:teuthology.orchestra.run.smithi119.stdout: python3-requests-2.25.1-8.el9.noarch 2024-04-23T10:09:58.067 INFO:teuthology.orchestra.run.smithi119.stdout: python3-requests-oauthlib-1.3.0-12.el9.noarch 2024-04-23T10:09:58.067 INFO:teuthology.orchestra.run.smithi119.stdout: python3-rgw-2:19.0.0-3156.gd69f13d8.el7.x86_64 2024-04-23T10:09:58.067 INFO:teuthology.orchestra.run.smithi119.stdout: python3-routes-2.5.1-5.el9.noarch 2024-04-23T10:09:58.067 INFO:teuthology.orchestra.run.smithi119.stdout: python3-rsa-4.9-2.el9.noarch 2024-04-23T10:09:58.067 INFO:teuthology.orchestra.run.smithi119.stdout: python3-scipy-1.6.2-8.el9.x86_64 2024-04-23T10:09:58.067 INFO:teuthology.orchestra.run.smithi119.stdout: python3-tempora-5.0.0-2.el9.noarch 2024-04-23T10:09:58.067 INFO:teuthology.orchestra.run.smithi119.stdout: python3-toml-0.10.2-6.el9.noarch 2024-04-23T10:09:58.067 INFO:teuthology.orchestra.run.smithi119.stdout: python3-typing-extensions-4.4.0-2.el9.noarch 2024-04-23T10:09:58.068 INFO:teuthology.orchestra.run.smithi119.stdout: python3-urllib3-1.26.5-5.el9.noarch 2024-04-23T10:09:58.068 INFO:teuthology.orchestra.run.smithi119.stdout: python3-wcwidth-0.2.5-8.el9.noarch 2024-04-23T10:09:58.068 INFO:teuthology.orchestra.run.smithi119.stdout: python3-webob-1.8.7-6.el9.noarch 2024-04-23T10:09:58.068 INFO:teuthology.orchestra.run.smithi119.stdout: python3-websocket-client-1.2.3-2.el9.noarch 2024-04-23T10:09:58.068 INFO:teuthology.orchestra.run.smithi119.stdout: python3-werkzeug-2.0.3-3.el9.1.noarch 2024-04-23T10:09:58.068 INFO:teuthology.orchestra.run.smithi119.stdout: python3-zc-lockfile-2.0-10.el9.noarch 2024-04-23T10:09:58.068 INFO:teuthology.orchestra.run.smithi119.stdout: qatlib-24.02.0-1.el9.x86_64 2024-04-23T10:09:58.068 INFO:teuthology.orchestra.run.smithi119.stdout: qatlib-service-24.02.0-1.el9.x86_64 2024-04-23T10:09:58.068 INFO:teuthology.orchestra.run.smithi119.stdout: qatzip-libs-1.2.0-1.el9.x86_64 2024-04-23T10:09:58.068 INFO:teuthology.orchestra.run.smithi119.stdout: rbd-fuse-2:19.0.0-3156.gd69f13d8.el7.x86_64 2024-04-23T10:09:58.068 INFO:teuthology.orchestra.run.smithi119.stdout: rbd-mirror-2:19.0.0-3156.gd69f13d8.el7.x86_64 2024-04-23T10:09:58.068 INFO:teuthology.orchestra.run.smithi119.stdout: rbd-nbd-2:19.0.0-3156.gd69f13d8.el7.x86_64 2024-04-23T10:09:58.068 INFO:teuthology.orchestra.run.smithi119.stdout: re2-1:20211101-3.el9.x86_64 2024-04-23T10:09:58.068 INFO:teuthology.orchestra.run.smithi119.stdout: socat-1.7.4.1-5.el9.x86_64 2024-04-23T10:09:58.069 INFO:teuthology.orchestra.run.smithi119.stdout: thrift-0.15.0-2.el9.x86_64 2024-04-23T10:09:58.069 INFO:teuthology.orchestra.run.smithi119.stdout: uuid-1.6.2-55.el9.x86_64 2024-04-23T10:09:58.069 INFO:teuthology.orchestra.run.smithi119.stdout: xfsprogs-devel-6.3.0-1.el9.x86_64 2024-04-23T10:09:58.069 INFO:teuthology.orchestra.run.smithi119.stdout: xmlstarlet-1.6.1-20.el9.x86_64 2024-04-23T10:09:58.069 INFO:teuthology.orchestra.run.smithi119.stdout: zlib-devel-1.2.11-41.el9.x86_64 2024-04-23T10:09:58.069 INFO:teuthology.orchestra.run.smithi119.stdout: 2024-04-23T10:09:58.069 INFO:teuthology.orchestra.run.smithi119.stdout:Complete! 2024-04-23T10:09:58.519 DEBUG:teuthology.parallel:result is None 2024-04-23T10:09:58.581 INFO:teuthology.orchestra.run.smithi114.stdout: Installing : cephfs-top-2:19.0.0-3156.gd69f13d8.el7.noarch 148/156 2024-04-23T10:09:58.662 INFO:teuthology.orchestra.run.smithi114.stdout: Installing : openssl-devel-1:3.2.1-1.el9.x86_64 149/156 2024-04-23T10:09:58.703 INFO:teuthology.orchestra.run.smithi114.stdout: Installing : libaio-devel-0.3.111-13.el9.x86_64 150/156 2024-04-23T10:09:58.785 INFO:teuthology.orchestra.run.smithi114.stdout: Installing : flex-2.6.4-9.el9.x86_64 151/156 2024-04-23T10:09:58.817 INFO:teuthology.orchestra.run.smithi114.stdout: Installing : bison-3.7.4-5.el9.x86_64 152/156 2024-04-23T10:09:58.817 INFO:teuthology.orchestra.run.smithi114.stdout: Cleanup : librbd1-2:16.2.4-5.el9.x86_64 153/156 2024-04-23T10:09:58.901 INFO:teuthology.orchestra.run.smithi114.stdout: Running scriptlet: librbd1-2:16.2.4-5.el9.x86_64 153/156 2024-04-23T10:09:58.925 INFO:teuthology.orchestra.run.smithi114.stdout: Cleanup : openssl-1:3.0.7-27.el9.x86_64 154/156 2024-04-23T10:09:58.925 INFO:teuthology.orchestra.run.smithi114.stdout: Cleanup : librados2-2:16.2.4-5.el9.x86_64 155/156 2024-04-23T10:09:59.017 INFO:teuthology.orchestra.run.smithi114.stdout: Running scriptlet: librados2-2:16.2.4-5.el9.x86_64 155/156 2024-04-23T10:09:59.081 INFO:teuthology.orchestra.run.smithi114.stdout: Cleanup : openssl-libs-1:3.0.7-27.el9.x86_64 156/156 2024-04-23T10:10:00.594 INFO:teuthology.orchestra.run.smithi204.stdout: Running scriptlet: openssl-libs-1:3.0.7-27.el9.x86_64 156/156 2024-04-23T10:10:00.594 INFO:teuthology.orchestra.run.smithi204.stdout: Verifying : ceph-2:19.0.0-3156.gd69f13d8.el7.x86_64 1/156 2024-04-23T10:10:00.594 INFO:teuthology.orchestra.run.smithi204.stdout: Verifying : ceph-base-2:19.0.0-3156.gd69f13d8.el7.x86_64 2/156 2024-04-23T10:10:00.594 INFO:teuthology.orchestra.run.smithi204.stdout: Verifying : ceph-common-2:19.0.0-3156.gd69f13d8.el7.x86_64 3/156 2024-04-23T10:10:00.594 INFO:teuthology.orchestra.run.smithi204.stdout: Verifying : ceph-fuse-2:19.0.0-3156.gd69f13d8.el7.x86_64 4/156 2024-04-23T10:10:00.595 INFO:teuthology.orchestra.run.smithi204.stdout: Verifying : ceph-immutable-object-cache-2:19.0.0-3156.gd69f1 5/156 2024-04-23T10:10:00.595 INFO:teuthology.orchestra.run.smithi204.stdout: Verifying : ceph-mds-2:19.0.0-3156.gd69f13d8.el7.x86_64 6/156 2024-04-23T10:10:00.595 INFO:teuthology.orchestra.run.smithi204.stdout: Verifying : ceph-mgr-2:19.0.0-3156.gd69f13d8.el7.x86_64 7/156 2024-04-23T10:10:00.595 INFO:teuthology.orchestra.run.smithi204.stdout: Verifying : ceph-mon-2:19.0.0-3156.gd69f13d8.el7.x86_64 8/156 2024-04-23T10:10:00.595 INFO:teuthology.orchestra.run.smithi204.stdout: Verifying : ceph-osd-2:19.0.0-3156.gd69f13d8.el7.x86_64 9/156 2024-04-23T10:10:00.595 INFO:teuthology.orchestra.run.smithi204.stdout: Verifying : ceph-radosgw-2:19.0.0-3156.gd69f13d8.el7.x86_64 10/156 2024-04-23T10:10:00.595 INFO:teuthology.orchestra.run.smithi204.stdout: Verifying : ceph-selinux-2:19.0.0-3156.gd69f13d8.el7.x86_64 11/156 2024-04-23T10:10:00.595 INFO:teuthology.orchestra.run.smithi204.stdout: Verifying : ceph-test-2:19.0.0-3156.gd69f13d8.el7.x86_64 12/156 2024-04-23T10:10:00.595 INFO:teuthology.orchestra.run.smithi204.stdout: Verifying : cephfs-mirror-2:19.0.0-3156.gd69f13d8.el7.x86_64 13/156 2024-04-23T10:10:00.595 INFO:teuthology.orchestra.run.smithi204.stdout: Verifying : cephfs-shell-2:19.0.0-3156.gd69f13d8.el7.x86_64 14/156 2024-04-23T10:10:00.595 INFO:teuthology.orchestra.run.smithi204.stdout: Verifying : libcephfs-devel-2:19.0.0-3156.gd69f13d8.el7.x86_ 15/156 2024-04-23T10:10:00.595 INFO:teuthology.orchestra.run.smithi204.stdout: Verifying : libcephfs2-2:19.0.0-3156.gd69f13d8.el7.x86_64 16/156 2024-04-23T10:10:00.595 INFO:teuthology.orchestra.run.smithi204.stdout: Verifying : libcephsqlite-2:19.0.0-3156.gd69f13d8.el7.x86_64 17/156 2024-04-23T10:10:00.595 INFO:teuthology.orchestra.run.smithi204.stdout: Verifying : librados-devel-2:19.0.0-3156.gd69f13d8.el7.x86_6 18/156 2024-04-23T10:10:00.596 INFO:teuthology.orchestra.run.smithi204.stdout: Verifying : libradosstriper1-2:19.0.0-3156.gd69f13d8.el7.x86 19/156 2024-04-23T10:10:00.596 INFO:teuthology.orchestra.run.smithi204.stdout: Verifying : librgw2-2:19.0.0-3156.gd69f13d8.el7.x86_64 20/156 2024-04-23T10:10:00.596 INFO:teuthology.orchestra.run.smithi204.stdout: Verifying : python3-ceph-argparse-2:19.0.0-3156.gd69f13d8.el 21/156 2024-04-23T10:10:00.596 INFO:teuthology.orchestra.run.smithi204.stdout: Verifying : python3-ceph-common-2:19.0.0-3156.gd69f13d8.el7. 22/156 2024-04-23T10:10:00.596 INFO:teuthology.orchestra.run.smithi204.stdout: Verifying : python3-cephfs-2:19.0.0-3156.gd69f13d8.el7.x86_6 23/156 2024-04-23T10:10:00.596 INFO:teuthology.orchestra.run.smithi204.stdout: Verifying : python3-rados-2:19.0.0-3156.gd69f13d8.el7.x86_64 24/156 2024-04-23T10:10:00.596 INFO:teuthology.orchestra.run.smithi204.stdout: Verifying : python3-rbd-2:19.0.0-3156.gd69f13d8.el7.x86_64 25/156 2024-04-23T10:10:00.596 INFO:teuthology.orchestra.run.smithi204.stdout: Verifying : python3-rgw-2:19.0.0-3156.gd69f13d8.el7.x86_64 26/156 2024-04-23T10:10:00.596 INFO:teuthology.orchestra.run.smithi204.stdout: Verifying : rbd-fuse-2:19.0.0-3156.gd69f13d8.el7.x86_64 27/156 2024-04-23T10:10:00.596 INFO:teuthology.orchestra.run.smithi204.stdout: Verifying : rbd-mirror-2:19.0.0-3156.gd69f13d8.el7.x86_64 28/156 2024-04-23T10:10:00.596 INFO:teuthology.orchestra.run.smithi204.stdout: Verifying : rbd-nbd-2:19.0.0-3156.gd69f13d8.el7.x86_64 29/156 2024-04-23T10:10:00.596 INFO:teuthology.orchestra.run.smithi204.stdout: Verifying : ceph-grafana-dashboards-2:19.0.0-3156.gd69f13d8. 30/156 2024-04-23T10:10:00.596 INFO:teuthology.orchestra.run.smithi204.stdout: Verifying : ceph-mgr-cephadm-2:19.0.0-3156.gd69f13d8.el7.noa 31/156 2024-04-23T10:10:00.596 INFO:teuthology.orchestra.run.smithi204.stdout: Verifying : ceph-mgr-dashboard-2:19.0.0-3156.gd69f13d8.el7.n 32/156 2024-04-23T10:10:00.597 INFO:teuthology.orchestra.run.smithi204.stdout: Verifying : ceph-mgr-diskprediction-local-2:19.0.0-3156.gd69 33/156 2024-04-23T10:10:00.597 INFO:teuthology.orchestra.run.smithi204.stdout: Verifying : ceph-mgr-modules-core-2:19.0.0-3156.gd69f13d8.el 34/156 2024-04-23T10:10:00.597 INFO:teuthology.orchestra.run.smithi204.stdout: Verifying : ceph-mgr-rook-2:19.0.0-3156.gd69f13d8.el7.noarch 35/156 2024-04-23T10:10:00.597 INFO:teuthology.orchestra.run.smithi204.stdout: Verifying : ceph-prometheus-alerts-2:19.0.0-3156.gd69f13d8.e 36/156 2024-04-23T10:10:00.597 INFO:teuthology.orchestra.run.smithi204.stdout: Verifying : ceph-volume-2:19.0.0-3156.gd69f13d8.el7.noarch 37/156 2024-04-23T10:10:00.597 INFO:teuthology.orchestra.run.smithi204.stdout: Verifying : cephadm-2:19.0.0-3156.gd69f13d8.el7.noarch 38/156 2024-04-23T10:10:00.597 INFO:teuthology.orchestra.run.smithi204.stdout: Verifying : cephfs-top-2:19.0.0-3156.gd69f13d8.el7.noarch 39/156 2024-04-23T10:10:00.597 INFO:teuthology.orchestra.run.smithi204.stdout: Verifying : fuse-2.9.9-15.el9.x86_64 40/156 2024-04-23T10:10:00.597 INFO:teuthology.orchestra.run.smithi204.stdout: Verifying : libgfortran-11.4.1-3.el9.x86_64 41/156 2024-04-23T10:10:00.597 INFO:teuthology.orchestra.run.smithi204.stdout: Verifying : libquadmath-11.4.1-3.el9.x86_64 42/156 2024-04-23T10:10:00.597 INFO:teuthology.orchestra.run.smithi204.stdout: Verifying : python3-cffi-1.14.5-5.el9.x86_64 43/156 2024-04-23T10:10:00.597 INFO:teuthology.orchestra.run.smithi204.stdout: Verifying : python3-cryptography-36.0.1-4.el9.x86_64 44/156 2024-04-23T10:10:00.597 INFO:teuthology.orchestra.run.smithi204.stdout: Verifying : python3-ply-3.11-14.el9.noarch 45/156 2024-04-23T10:10:00.598 INFO:teuthology.orchestra.run.smithi204.stdout: Verifying : python3-pycparser-2.20-6.el9.noarch 46/156 2024-04-23T10:10:00.598 INFO:teuthology.orchestra.run.smithi204.stdout: Verifying : python3-requests-2.25.1-8.el9.noarch 47/156 2024-04-23T10:10:00.598 INFO:teuthology.orchestra.run.smithi204.stdout: Verifying : python3-urllib3-1.26.5-5.el9.noarch 48/156 2024-04-23T10:10:00.598 INFO:teuthology.orchestra.run.smithi204.stdout: Verifying : automake-1.16.2-8.el9.noarch 49/156 2024-04-23T10:10:00.598 INFO:teuthology.orchestra.run.smithi204.stdout: Verifying : bison-3.7.4-5.el9.x86_64 50/156 2024-04-23T10:10:00.598 INFO:teuthology.orchestra.run.smithi204.stdout: Verifying : boost-program-options-1.75.0-8.el9.x86_64 51/156 2024-04-23T10:10:00.598 INFO:teuthology.orchestra.run.smithi204.stdout: Verifying : elfutils-libelf-devel-0.190-2.el9.x86_64 52/156 2024-04-23T10:10:00.598 INFO:teuthology.orchestra.run.smithi204.stdout: Verifying : flex-2.6.4-9.el9.x86_64 53/156 2024-04-23T10:10:00.598 INFO:teuthology.orchestra.run.smithi204.stdout: Verifying : flexiblas-3.0.4-8.el9.x86_64 54/156 2024-04-23T10:10:00.598 INFO:teuthology.orchestra.run.smithi204.stdout: Verifying : flexiblas-netlib-3.0.4-8.el9.x86_64 55/156 2024-04-23T10:10:00.598 INFO:teuthology.orchestra.run.smithi204.stdout: Verifying : flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 56/156 2024-04-23T10:10:00.598 INFO:teuthology.orchestra.run.smithi204.stdout: Verifying : libaio-devel-0.3.111-13.el9.x86_64 57/156 2024-04-23T10:10:00.598 INFO:teuthology.orchestra.run.smithi204.stdout: Verifying : libpmemobj-1.12.1-1.el9.x86_64 58/156 2024-04-23T10:10:00.598 INFO:teuthology.orchestra.run.smithi204.stdout: Verifying : librabbitmq-0.11.0-7.el9.x86_64 59/156 2024-04-23T10:10:00.598 INFO:teuthology.orchestra.run.smithi204.stdout: Verifying : librdkafka-1.6.1-102.el9.x86_64 60/156 2024-04-23T10:10:00.599 INFO:teuthology.orchestra.run.smithi204.stdout: Verifying : libtool-2.4.6-45.el9.x86_64 61/156 2024-04-23T10:10:00.599 INFO:teuthology.orchestra.run.smithi204.stdout: Verifying : libuuid-devel-2.37.4-18.el9.x86_64 62/156 2024-04-23T10:10:00.599 INFO:teuthology.orchestra.run.smithi204.stdout: Verifying : libzstd-devel-1.5.1-2.el9.x86_64 63/156 2024-04-23T10:10:00.599 INFO:teuthology.orchestra.run.smithi204.stdout: Verifying : lttng-ust-2.12.0-6.el9.x86_64 64/156 2024-04-23T10:10:00.599 INFO:teuthology.orchestra.run.smithi204.stdout: Verifying : lua-5.4.4-4.el9.x86_64 65/156 2024-04-23T10:10:00.599 INFO:teuthology.orchestra.run.smithi204.stdout: Verifying : openblas-0.3.21-2.el9.x86_64 66/156 2024-04-23T10:10:00.599 INFO:teuthology.orchestra.run.smithi204.stdout: Verifying : openblas-openmp-0.3.21-2.el9.x86_64 67/156 2024-04-23T10:10:00.599 INFO:teuthology.orchestra.run.smithi204.stdout: Verifying : openssl-devel-1:3.2.1-1.el9.x86_64 68/156 2024-04-23T10:10:00.599 INFO:teuthology.orchestra.run.smithi204.stdout: Verifying : postgresql-13.14-1.el9.x86_64 69/156 2024-04-23T10:10:00.599 INFO:teuthology.orchestra.run.smithi204.stdout: Verifying : postgresql-contrib-13.14-1.el9.x86_64 70/156 2024-04-23T10:10:00.599 INFO:teuthology.orchestra.run.smithi204.stdout: Verifying : postgresql-private-libs-13.14-1.el9.x86_64 71/156 2024-04-23T10:10:00.599 INFO:teuthology.orchestra.run.smithi204.stdout: Verifying : postgresql-server-13.14-1.el9.x86_64 72/156 2024-04-23T10:10:00.599 INFO:teuthology.orchestra.run.smithi204.stdout: Verifying : protobuf-3.14.0-13.el9.x86_64 73/156 2024-04-23T10:10:00.599 INFO:teuthology.orchestra.run.smithi204.stdout: Verifying : python3-attrs-20.3.0-7.el9.noarch 74/156 2024-04-23T10:10:00.599 INFO:teuthology.orchestra.run.smithi204.stdout: Verifying : python3-babel-2.9.1-2.el9.noarch 75/156 2024-04-23T10:10:00.599 INFO:teuthology.orchestra.run.smithi204.stdout: Verifying : python3-devel-3.9.18-3.el9.x86_64 76/156 2024-04-23T10:10:00.600 INFO:teuthology.orchestra.run.smithi204.stdout: Verifying : python3-jinja2-2.11.3-5.el9.noarch 77/156 2024-04-23T10:10:00.600 INFO:teuthology.orchestra.run.smithi204.stdout: Verifying : python3-jsonpatch-1.21-16.el9.noarch 78/156 2024-04-23T10:10:00.600 INFO:teuthology.orchestra.run.smithi204.stdout: Verifying : python3-jsonpointer-2.0-4.el9.noarch 79/156 2024-04-23T10:10:00.600 INFO:teuthology.orchestra.run.smithi204.stdout: Verifying : python3-mako-1.1.4-6.el9.noarch 80/156 2024-04-23T10:10:00.600 INFO:teuthology.orchestra.run.smithi204.stdout: Verifying : python3-markupsafe-1.1.1-12.el9.x86_64 81/156 2024-04-23T10:10:00.600 INFO:teuthology.orchestra.run.smithi204.stdout: Verifying : python3-numpy-1:1.20.1-5.el9.x86_64 82/156 2024-04-23T10:10:00.600 INFO:teuthology.orchestra.run.smithi204.stdout: Verifying : python3-numpy-f2py-1:1.20.1-5.el9.x86_64 83/156 2024-04-23T10:10:00.600 INFO:teuthology.orchestra.run.smithi204.stdout: Verifying : python3-oauthlib-3.1.1-5.el9.noarch 84/156 2024-04-23T10:10:00.600 INFO:teuthology.orchestra.run.smithi204.stdout: Verifying : python3-packaging-20.9-5.el9.noarch 85/156 2024-04-23T10:10:00.600 INFO:teuthology.orchestra.run.smithi204.stdout: Verifying : python3-prettytable-0.7.2-27.el9.noarch 86/156 2024-04-23T10:10:00.600 INFO:teuthology.orchestra.run.smithi204.stdout: Verifying : python3-protobuf-3.14.0-13.el9.noarch 87/156 2024-04-23T10:10:00.600 INFO:teuthology.orchestra.run.smithi204.stdout: Verifying : python3-pyasn1-0.4.8-6.el9.noarch 88/156 2024-04-23T10:10:00.600 INFO:teuthology.orchestra.run.smithi204.stdout: Verifying : python3-pyasn1-modules-0.4.8-6.el9.noarch 89/156 2024-04-23T10:10:00.600 INFO:teuthology.orchestra.run.smithi204.stdout: Verifying : python3-pytz-2021.1-5.el9.noarch 90/156 2024-04-23T10:10:00.600 INFO:teuthology.orchestra.run.smithi204.stdout: Verifying : python3-requests-oauthlib-1.3.0-12.el9.noarch 91/156 2024-04-23T10:10:00.600 INFO:teuthology.orchestra.run.smithi204.stdout: Verifying : python3-scipy-1.6.2-8.el9.x86_64 92/156 2024-04-23T10:10:00.601 INFO:teuthology.orchestra.run.smithi204.stdout: Verifying : python3-toml-0.10.2-6.el9.noarch 93/156 2024-04-23T10:10:00.601 INFO:teuthology.orchestra.run.smithi204.stdout: Verifying : python3-wcwidth-0.2.5-8.el9.noarch 94/156 2024-04-23T10:10:00.601 INFO:teuthology.orchestra.run.smithi204.stdout: Verifying : qatlib-24.02.0-1.el9.x86_64 95/156 2024-04-23T10:10:00.601 INFO:teuthology.orchestra.run.smithi204.stdout: Verifying : qatlib-service-24.02.0-1.el9.x86_64 96/156 2024-04-23T10:10:00.601 INFO:teuthology.orchestra.run.smithi204.stdout: Verifying : qatzip-libs-1.2.0-1.el9.x86_64 97/156 2024-04-23T10:10:00.601 INFO:teuthology.orchestra.run.smithi204.stdout: Verifying : socat-1.7.4.1-5.el9.x86_64 98/156 2024-04-23T10:10:00.601 INFO:teuthology.orchestra.run.smithi204.stdout: Verifying : uuid-1.6.2-55.el9.x86_64 99/156 2024-04-23T10:10:00.601 INFO:teuthology.orchestra.run.smithi204.stdout: Verifying : xfsprogs-devel-6.3.0-1.el9.x86_64 100/156 2024-04-23T10:10:00.601 INFO:teuthology.orchestra.run.smithi204.stdout: Verifying : xmlstarlet-1.6.1-20.el9.x86_64 101/156 2024-04-23T10:10:00.601 INFO:teuthology.orchestra.run.smithi204.stdout: Verifying : zlib-devel-1.2.11-41.el9.x86_64 102/156 2024-04-23T10:10:00.601 INFO:teuthology.orchestra.run.smithi204.stdout: Verifying : lua-devel-5.4.4-4.el9.x86_64 103/156 2024-04-23T10:10:00.601 INFO:teuthology.orchestra.run.smithi204.stdout: Verifying : protobuf-compiler-3.14.0-13.el9.x86_64 104/156 2024-04-23T10:10:00.601 INFO:teuthology.orchestra.run.smithi204.stdout: Verifying : abseil-cpp-20211102.0-3.el9.x86_64 105/156 2024-04-23T10:10:00.601 INFO:teuthology.orchestra.run.smithi204.stdout: Verifying : gperftools-libs-2.9.1-2.el9.x86_64 106/156 2024-04-23T10:10:00.601 INFO:teuthology.orchestra.run.smithi204.stdout: Verifying : grpc-data-1.46.7-10.el9.noarch 107/156 2024-04-23T10:10:00.602 INFO:teuthology.orchestra.run.smithi204.stdout: Verifying : libarrow-9.0.0-10.el9.x86_64 108/156 2024-04-23T10:10:00.602 INFO:teuthology.orchestra.run.smithi204.stdout: Verifying : libarrow-doc-9.0.0-10.el9.noarch 109/156 2024-04-23T10:10:00.602 INFO:teuthology.orchestra.run.smithi204.stdout: Verifying : liboath-2.6.7-2.el9.x86_64 110/156 2024-04-23T10:10:00.602 INFO:teuthology.orchestra.run.smithi204.stdout: Verifying : libunwind-1.6.2-1.el9.x86_64 111/156 2024-04-23T10:10:00.602 INFO:teuthology.orchestra.run.smithi204.stdout: Verifying : luarocks-3.9.2-1.el9.noarch 112/156 2024-04-23T10:10:00.602 INFO:teuthology.orchestra.run.smithi204.stdout: Verifying : parquet-libs-9.0.0-10.el9.x86_64 113/156 2024-04-23T10:10:00.602 INFO:teuthology.orchestra.run.smithi204.stdout: Verifying : python3-asyncssh-2.13.2-5.el9.noarch 114/156 2024-04-23T10:10:00.602 INFO:teuthology.orchestra.run.smithi204.stdout: Verifying : python3-bcrypt-3.2.2-1.el9.x86_64 115/156 2024-04-23T10:10:00.602 INFO:teuthology.orchestra.run.smithi204.stdout: Verifying : python3-cachetools-4.2.4-1.el9.noarch 116/156 2024-04-23T10:10:00.602 INFO:teuthology.orchestra.run.smithi204.stdout: Verifying : python3-certifi-2021.10.8-2.el9.noarch 117/156 2024-04-23T10:10:00.602 INFO:teuthology.orchestra.run.smithi204.stdout: Verifying : python3-cheroot-8.6.0-4.el9.noarch 118/156 2024-04-23T10:10:00.602 INFO:teuthology.orchestra.run.smithi204.stdout: Verifying : python3-cherrypy-18.6.1-2.el9.noarch 119/156 2024-04-23T10:10:00.602 INFO:teuthology.orchestra.run.smithi204.stdout: Verifying : python3-cmd2-2.4.2-2.el9.noarch 120/156 2024-04-23T10:10:00.602 INFO:teuthology.orchestra.run.smithi204.stdout: Verifying : python3-colorama-0.4.4-12.el9.noarch 121/156 2024-04-23T10:10:00.603 INFO:teuthology.orchestra.run.smithi204.stdout: Verifying : python3-google-auth-1:2.29.0-1.el9.noarch 122/156 2024-04-23T10:10:00.603 INFO:teuthology.orchestra.run.smithi204.stdout: Verifying : python3-grpcio-1.46.7-10.el9.x86_64 123/156 2024-04-23T10:10:00.603 INFO:teuthology.orchestra.run.smithi204.stdout: Verifying : python3-grpcio-tools-1.46.7-10.el9.x86_64 124/156 2024-04-23T10:10:00.603 INFO:teuthology.orchestra.run.smithi204.stdout: Verifying : python3-jaraco-8.2.1-3.el9.noarch 125/156 2024-04-23T10:10:00.603 INFO:teuthology.orchestra.run.smithi204.stdout: Verifying : python3-jaraco-classes-3.2.1-5.el9.noarch 126/156 2024-04-23T10:10:00.603 INFO:teuthology.orchestra.run.smithi204.stdout: Verifying : python3-jaraco-collections-3.0.0-8.el9.noarch 127/156 2024-04-23T10:10:00.603 INFO:teuthology.orchestra.run.smithi204.stdout: Verifying : python3-jaraco-functools-3.5.0-2.el9.noarch 128/156 2024-04-23T10:10:00.603 INFO:teuthology.orchestra.run.smithi204.stdout: Verifying : python3-jaraco-text-3.2.0-6.el9.noarch 129/156 2024-04-23T10:10:00.603 INFO:teuthology.orchestra.run.smithi204.stdout: Verifying : python3-kubernetes-1:26.1.0-2.el9.noarch 130/156 2024-04-23T10:10:00.603 INFO:teuthology.orchestra.run.smithi204.stdout: Verifying : python3-logutils-0.3.5-21.el9.noarch 131/156 2024-04-23T10:10:00.603 INFO:teuthology.orchestra.run.smithi204.stdout: Verifying : python3-more-itertools-8.12.0-2.el9.noarch 132/156 2024-04-23T10:10:00.603 INFO:teuthology.orchestra.run.smithi204.stdout: Verifying : python3-natsort-7.1.1-5.el9.noarch 133/156 2024-04-23T10:10:00.603 INFO:teuthology.orchestra.run.smithi204.stdout: Verifying : python3-pecan-1.4.2-3.el9.noarch 134/156 2024-04-23T10:10:00.603 INFO:teuthology.orchestra.run.smithi204.stdout: Verifying : python3-portend-3.1.0-2.el9.noarch 135/156 2024-04-23T10:10:00.603 INFO:teuthology.orchestra.run.smithi204.stdout: Verifying : python3-pyOpenSSL-21.0.0-1.el9.noarch 136/156 2024-04-23T10:10:00.604 INFO:teuthology.orchestra.run.smithi204.stdout: Verifying : python3-pyperclip-1.8.2-2.el9.noarch 137/156 2024-04-23T10:10:00.604 INFO:teuthology.orchestra.run.smithi204.stdout: Verifying : python3-repoze-lru-0.7-16.el9.noarch 138/156 2024-04-23T10:10:00.604 INFO:teuthology.orchestra.run.smithi204.stdout: Verifying : python3-routes-2.5.1-5.el9.noarch 139/156 2024-04-23T10:10:00.604 INFO:teuthology.orchestra.run.smithi204.stdout: Verifying : python3-rsa-4.9-2.el9.noarch 140/156 2024-04-23T10:10:00.604 INFO:teuthology.orchestra.run.smithi204.stdout: Verifying : python3-tempora-5.0.0-2.el9.noarch 141/156 2024-04-23T10:10:00.604 INFO:teuthology.orchestra.run.smithi204.stdout: Verifying : python3-typing-extensions-4.4.0-2.el9.noarch 142/156 2024-04-23T10:10:00.604 INFO:teuthology.orchestra.run.smithi204.stdout: Verifying : python3-webob-1.8.7-6.el9.noarch 143/156 2024-04-23T10:10:00.604 INFO:teuthology.orchestra.run.smithi204.stdout: Verifying : python3-websocket-client-1.2.3-2.el9.noarch 144/156 2024-04-23T10:10:00.604 INFO:teuthology.orchestra.run.smithi204.stdout: Verifying : python3-werkzeug-2.0.3-3.el9.1.noarch 145/156 2024-04-23T10:10:00.604 INFO:teuthology.orchestra.run.smithi204.stdout: Verifying : python3-zc-lockfile-2.0-10.el9.noarch 146/156 2024-04-23T10:10:00.604 INFO:teuthology.orchestra.run.smithi204.stdout: Verifying : re2-1:20211101-3.el9.x86_64 147/156 2024-04-23T10:10:00.604 INFO:teuthology.orchestra.run.smithi204.stdout: Verifying : thrift-0.15.0-2.el9.x86_64 148/156 2024-04-23T10:10:00.604 INFO:teuthology.orchestra.run.smithi204.stdout: Verifying : librados2-2:19.0.0-3156.gd69f13d8.el7.x86_64 149/156 2024-04-23T10:10:00.604 INFO:teuthology.orchestra.run.smithi204.stdout: Verifying : librados2-2:16.2.4-5.el9.x86_64 150/156 2024-04-23T10:10:00.604 INFO:teuthology.orchestra.run.smithi204.stdout: Verifying : librbd1-2:19.0.0-3156.gd69f13d8.el7.x86_64 151/156 2024-04-23T10:10:00.604 INFO:teuthology.orchestra.run.smithi204.stdout: Verifying : librbd1-2:16.2.4-5.el9.x86_64 152/156 2024-04-23T10:10:00.605 INFO:teuthology.orchestra.run.smithi204.stdout: Verifying : openssl-1:3.2.1-1.el9.x86_64 153/156 2024-04-23T10:10:00.605 INFO:teuthology.orchestra.run.smithi204.stdout: Verifying : openssl-1:3.0.7-27.el9.x86_64 154/156 2024-04-23T10:10:00.605 INFO:teuthology.orchestra.run.smithi204.stdout: Verifying : openssl-libs-1:3.2.1-1.el9.x86_64 155/156 2024-04-23T10:10:02.726 INFO:teuthology.orchestra.run.smithi204.stdout: Verifying : openssl-libs-1:3.0.7-27.el9.x86_64 156/156 2024-04-23T10:10:02.726 INFO:teuthology.orchestra.run.smithi204.stdout: 2024-04-23T10:10:02.726 INFO:teuthology.orchestra.run.smithi204.stdout:Upgraded: 2024-04-23T10:10:02.726 INFO:teuthology.orchestra.run.smithi204.stdout: librados2-2:19.0.0-3156.gd69f13d8.el7.x86_64 2024-04-23T10:10:02.726 INFO:teuthology.orchestra.run.smithi204.stdout: librbd1-2:19.0.0-3156.gd69f13d8.el7.x86_64 2024-04-23T10:10:02.726 INFO:teuthology.orchestra.run.smithi204.stdout: openssl-1:3.2.1-1.el9.x86_64 2024-04-23T10:10:02.726 INFO:teuthology.orchestra.run.smithi204.stdout: openssl-libs-1:3.2.1-1.el9.x86_64 2024-04-23T10:10:02.726 INFO:teuthology.orchestra.run.smithi204.stdout:Installed: 2024-04-23T10:10:02.726 INFO:teuthology.orchestra.run.smithi204.stdout: abseil-cpp-20211102.0-3.el9.x86_64 2024-04-23T10:10:02.727 INFO:teuthology.orchestra.run.smithi204.stdout: automake-1.16.2-8.el9.noarch 2024-04-23T10:10:02.727 INFO:teuthology.orchestra.run.smithi204.stdout: bison-3.7.4-5.el9.x86_64 2024-04-23T10:10:02.727 INFO:teuthology.orchestra.run.smithi204.stdout: boost-program-options-1.75.0-8.el9.x86_64 2024-04-23T10:10:02.727 INFO:teuthology.orchestra.run.smithi204.stdout: ceph-2:19.0.0-3156.gd69f13d8.el7.x86_64 2024-04-23T10:10:02.727 INFO:teuthology.orchestra.run.smithi204.stdout: ceph-base-2:19.0.0-3156.gd69f13d8.el7.x86_64 2024-04-23T10:10:02.727 INFO:teuthology.orchestra.run.smithi204.stdout: ceph-common-2:19.0.0-3156.gd69f13d8.el7.x86_64 2024-04-23T10:10:02.727 INFO:teuthology.orchestra.run.smithi204.stdout: ceph-fuse-2:19.0.0-3156.gd69f13d8.el7.x86_64 2024-04-23T10:10:02.727 INFO:teuthology.orchestra.run.smithi204.stdout: ceph-grafana-dashboards-2:19.0.0-3156.gd69f13d8.el7.noarch 2024-04-23T10:10:02.727 INFO:teuthology.orchestra.run.smithi204.stdout: ceph-immutable-object-cache-2:19.0.0-3156.gd69f13d8.el7.x86_64 2024-04-23T10:10:02.727 INFO:teuthology.orchestra.run.smithi204.stdout: ceph-mds-2:19.0.0-3156.gd69f13d8.el7.x86_64 2024-04-23T10:10:02.727 INFO:teuthology.orchestra.run.smithi204.stdout: ceph-mgr-2:19.0.0-3156.gd69f13d8.el7.x86_64 2024-04-23T10:10:02.727 INFO:teuthology.orchestra.run.smithi204.stdout: ceph-mgr-cephadm-2:19.0.0-3156.gd69f13d8.el7.noarch 2024-04-23T10:10:02.727 INFO:teuthology.orchestra.run.smithi204.stdout: ceph-mgr-dashboard-2:19.0.0-3156.gd69f13d8.el7.noarch 2024-04-23T10:10:02.727 INFO:teuthology.orchestra.run.smithi204.stdout: ceph-mgr-diskprediction-local-2:19.0.0-3156.gd69f13d8.el7.noarch 2024-04-23T10:10:02.727 INFO:teuthology.orchestra.run.smithi204.stdout: ceph-mgr-modules-core-2:19.0.0-3156.gd69f13d8.el7.noarch 2024-04-23T10:10:02.728 INFO:teuthology.orchestra.run.smithi204.stdout: ceph-mgr-rook-2:19.0.0-3156.gd69f13d8.el7.noarch 2024-04-23T10:10:02.728 INFO:teuthology.orchestra.run.smithi204.stdout: ceph-mon-2:19.0.0-3156.gd69f13d8.el7.x86_64 2024-04-23T10:10:02.728 INFO:teuthology.orchestra.run.smithi204.stdout: ceph-osd-2:19.0.0-3156.gd69f13d8.el7.x86_64 2024-04-23T10:10:02.728 INFO:teuthology.orchestra.run.smithi204.stdout: ceph-prometheus-alerts-2:19.0.0-3156.gd69f13d8.el7.noarch 2024-04-23T10:10:02.728 INFO:teuthology.orchestra.run.smithi204.stdout: ceph-radosgw-2:19.0.0-3156.gd69f13d8.el7.x86_64 2024-04-23T10:10:02.728 INFO:teuthology.orchestra.run.smithi204.stdout: ceph-selinux-2:19.0.0-3156.gd69f13d8.el7.x86_64 2024-04-23T10:10:02.728 INFO:teuthology.orchestra.run.smithi204.stdout: ceph-test-2:19.0.0-3156.gd69f13d8.el7.x86_64 2024-04-23T10:10:02.728 INFO:teuthology.orchestra.run.smithi204.stdout: ceph-volume-2:19.0.0-3156.gd69f13d8.el7.noarch 2024-04-23T10:10:02.728 INFO:teuthology.orchestra.run.smithi204.stdout: cephadm-2:19.0.0-3156.gd69f13d8.el7.noarch 2024-04-23T10:10:02.728 INFO:teuthology.orchestra.run.smithi204.stdout: cephfs-mirror-2:19.0.0-3156.gd69f13d8.el7.x86_64 2024-04-23T10:10:02.728 INFO:teuthology.orchestra.run.smithi204.stdout: cephfs-shell-2:19.0.0-3156.gd69f13d8.el7.x86_64 2024-04-23T10:10:02.728 INFO:teuthology.orchestra.run.smithi204.stdout: cephfs-top-2:19.0.0-3156.gd69f13d8.el7.noarch 2024-04-23T10:10:02.728 INFO:teuthology.orchestra.run.smithi204.stdout: elfutils-libelf-devel-0.190-2.el9.x86_64 2024-04-23T10:10:02.728 INFO:teuthology.orchestra.run.smithi204.stdout: flex-2.6.4-9.el9.x86_64 2024-04-23T10:10:02.728 INFO:teuthology.orchestra.run.smithi204.stdout: flexiblas-3.0.4-8.el9.x86_64 2024-04-23T10:10:02.728 INFO:teuthology.orchestra.run.smithi204.stdout: flexiblas-netlib-3.0.4-8.el9.x86_64 2024-04-23T10:10:02.729 INFO:teuthology.orchestra.run.smithi204.stdout: flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 2024-04-23T10:10:02.729 INFO:teuthology.orchestra.run.smithi204.stdout: fuse-2.9.9-15.el9.x86_64 2024-04-23T10:10:02.729 INFO:teuthology.orchestra.run.smithi204.stdout: gperftools-libs-2.9.1-2.el9.x86_64 2024-04-23T10:10:02.729 INFO:teuthology.orchestra.run.smithi204.stdout: grpc-data-1.46.7-10.el9.noarch 2024-04-23T10:10:02.729 INFO:teuthology.orchestra.run.smithi204.stdout: libaio-devel-0.3.111-13.el9.x86_64 2024-04-23T10:10:02.729 INFO:teuthology.orchestra.run.smithi204.stdout: libarrow-9.0.0-10.el9.x86_64 2024-04-23T10:10:02.729 INFO:teuthology.orchestra.run.smithi204.stdout: libarrow-doc-9.0.0-10.el9.noarch 2024-04-23T10:10:02.729 INFO:teuthology.orchestra.run.smithi204.stdout: libcephfs-devel-2:19.0.0-3156.gd69f13d8.el7.x86_64 2024-04-23T10:10:02.729 INFO:teuthology.orchestra.run.smithi204.stdout: libcephfs2-2:19.0.0-3156.gd69f13d8.el7.x86_64 2024-04-23T10:10:02.729 INFO:teuthology.orchestra.run.smithi204.stdout: libcephsqlite-2:19.0.0-3156.gd69f13d8.el7.x86_64 2024-04-23T10:10:02.729 INFO:teuthology.orchestra.run.smithi204.stdout: libgfortran-11.4.1-3.el9.x86_64 2024-04-23T10:10:02.729 INFO:teuthology.orchestra.run.smithi204.stdout: liboath-2.6.7-2.el9.x86_64 2024-04-23T10:10:02.729 INFO:teuthology.orchestra.run.smithi204.stdout: libpmemobj-1.12.1-1.el9.x86_64 2024-04-23T10:10:02.729 INFO:teuthology.orchestra.run.smithi204.stdout: libquadmath-11.4.1-3.el9.x86_64 2024-04-23T10:10:02.729 INFO:teuthology.orchestra.run.smithi204.stdout: librabbitmq-0.11.0-7.el9.x86_64 2024-04-23T10:10:02.730 INFO:teuthology.orchestra.run.smithi204.stdout: librados-devel-2:19.0.0-3156.gd69f13d8.el7.x86_64 2024-04-23T10:10:02.730 INFO:teuthology.orchestra.run.smithi204.stdout: libradosstriper1-2:19.0.0-3156.gd69f13d8.el7.x86_64 2024-04-23T10:10:02.730 INFO:teuthology.orchestra.run.smithi204.stdout: librdkafka-1.6.1-102.el9.x86_64 2024-04-23T10:10:02.730 INFO:teuthology.orchestra.run.smithi204.stdout: librgw2-2:19.0.0-3156.gd69f13d8.el7.x86_64 2024-04-23T10:10:02.730 INFO:teuthology.orchestra.run.smithi204.stdout: libtool-2.4.6-45.el9.x86_64 2024-04-23T10:10:02.730 INFO:teuthology.orchestra.run.smithi204.stdout: libunwind-1.6.2-1.el9.x86_64 2024-04-23T10:10:02.730 INFO:teuthology.orchestra.run.smithi204.stdout: libuuid-devel-2.37.4-18.el9.x86_64 2024-04-23T10:10:02.730 INFO:teuthology.orchestra.run.smithi204.stdout: libzstd-devel-1.5.1-2.el9.x86_64 2024-04-23T10:10:02.730 INFO:teuthology.orchestra.run.smithi204.stdout: lttng-ust-2.12.0-6.el9.x86_64 2024-04-23T10:10:02.730 INFO:teuthology.orchestra.run.smithi204.stdout: lua-5.4.4-4.el9.x86_64 2024-04-23T10:10:02.730 INFO:teuthology.orchestra.run.smithi204.stdout: lua-devel-5.4.4-4.el9.x86_64 2024-04-23T10:10:02.730 INFO:teuthology.orchestra.run.smithi204.stdout: luarocks-3.9.2-1.el9.noarch 2024-04-23T10:10:02.730 INFO:teuthology.orchestra.run.smithi204.stdout: openblas-0.3.21-2.el9.x86_64 2024-04-23T10:10:02.730 INFO:teuthology.orchestra.run.smithi204.stdout: openblas-openmp-0.3.21-2.el9.x86_64 2024-04-23T10:10:02.731 INFO:teuthology.orchestra.run.smithi204.stdout: openssl-devel-1:3.2.1-1.el9.x86_64 2024-04-23T10:10:02.731 INFO:teuthology.orchestra.run.smithi204.stdout: parquet-libs-9.0.0-10.el9.x86_64 2024-04-23T10:10:02.731 INFO:teuthology.orchestra.run.smithi204.stdout: postgresql-13.14-1.el9.x86_64 2024-04-23T10:10:02.731 INFO:teuthology.orchestra.run.smithi204.stdout: postgresql-contrib-13.14-1.el9.x86_64 2024-04-23T10:10:02.731 INFO:teuthology.orchestra.run.smithi204.stdout: postgresql-private-libs-13.14-1.el9.x86_64 2024-04-23T10:10:02.731 INFO:teuthology.orchestra.run.smithi204.stdout: postgresql-server-13.14-1.el9.x86_64 2024-04-23T10:10:02.731 INFO:teuthology.orchestra.run.smithi204.stdout: protobuf-3.14.0-13.el9.x86_64 2024-04-23T10:10:02.731 INFO:teuthology.orchestra.run.smithi204.stdout: protobuf-compiler-3.14.0-13.el9.x86_64 2024-04-23T10:10:02.731 INFO:teuthology.orchestra.run.smithi204.stdout: python3-asyncssh-2.13.2-5.el9.noarch 2024-04-23T10:10:02.731 INFO:teuthology.orchestra.run.smithi204.stdout: python3-attrs-20.3.0-7.el9.noarch 2024-04-23T10:10:02.731 INFO:teuthology.orchestra.run.smithi204.stdout: python3-babel-2.9.1-2.el9.noarch 2024-04-23T10:10:02.731 INFO:teuthology.orchestra.run.smithi204.stdout: python3-bcrypt-3.2.2-1.el9.x86_64 2024-04-23T10:10:02.731 INFO:teuthology.orchestra.run.smithi204.stdout: python3-cachetools-4.2.4-1.el9.noarch 2024-04-23T10:10:02.731 INFO:teuthology.orchestra.run.smithi204.stdout: python3-ceph-argparse-2:19.0.0-3156.gd69f13d8.el7.x86_64 2024-04-23T10:10:02.731 INFO:teuthology.orchestra.run.smithi204.stdout: python3-ceph-common-2:19.0.0-3156.gd69f13d8.el7.x86_64 2024-04-23T10:10:02.731 INFO:teuthology.orchestra.run.smithi204.stdout: python3-cephfs-2:19.0.0-3156.gd69f13d8.el7.x86_64 2024-04-23T10:10:02.731 INFO:teuthology.orchestra.run.smithi204.stdout: python3-certifi-2021.10.8-2.el9.noarch 2024-04-23T10:10:02.732 INFO:teuthology.orchestra.run.smithi204.stdout: python3-cffi-1.14.5-5.el9.x86_64 2024-04-23T10:10:02.732 INFO:teuthology.orchestra.run.smithi204.stdout: python3-cheroot-8.6.0-4.el9.noarch 2024-04-23T10:10:02.732 INFO:teuthology.orchestra.run.smithi204.stdout: python3-cherrypy-18.6.1-2.el9.noarch 2024-04-23T10:10:02.732 INFO:teuthology.orchestra.run.smithi204.stdout: python3-cmd2-2.4.2-2.el9.noarch 2024-04-23T10:10:02.732 INFO:teuthology.orchestra.run.smithi204.stdout: python3-colorama-0.4.4-12.el9.noarch 2024-04-23T10:10:02.732 INFO:teuthology.orchestra.run.smithi204.stdout: python3-cryptography-36.0.1-4.el9.x86_64 2024-04-23T10:10:02.732 INFO:teuthology.orchestra.run.smithi204.stdout: python3-devel-3.9.18-3.el9.x86_64 2024-04-23T10:10:02.732 INFO:teuthology.orchestra.run.smithi204.stdout: python3-google-auth-1:2.29.0-1.el9.noarch 2024-04-23T10:10:02.732 INFO:teuthology.orchestra.run.smithi204.stdout: python3-grpcio-1.46.7-10.el9.x86_64 2024-04-23T10:10:02.732 INFO:teuthology.orchestra.run.smithi204.stdout: python3-grpcio-tools-1.46.7-10.el9.x86_64 2024-04-23T10:10:02.732 INFO:teuthology.orchestra.run.smithi204.stdout: python3-jaraco-8.2.1-3.el9.noarch 2024-04-23T10:10:02.732 INFO:teuthology.orchestra.run.smithi204.stdout: python3-jaraco-classes-3.2.1-5.el9.noarch 2024-04-23T10:10:02.732 INFO:teuthology.orchestra.run.smithi204.stdout: python3-jaraco-collections-3.0.0-8.el9.noarch 2024-04-23T10:10:02.732 INFO:teuthology.orchestra.run.smithi204.stdout: python3-jaraco-functools-3.5.0-2.el9.noarch 2024-04-23T10:10:02.732 INFO:teuthology.orchestra.run.smithi204.stdout: python3-jaraco-text-3.2.0-6.el9.noarch 2024-04-23T10:10:02.732 INFO:teuthology.orchestra.run.smithi204.stdout: python3-jinja2-2.11.3-5.el9.noarch 2024-04-23T10:10:02.733 INFO:teuthology.orchestra.run.smithi204.stdout: python3-jsonpatch-1.21-16.el9.noarch 2024-04-23T10:10:02.733 INFO:teuthology.orchestra.run.smithi204.stdout: python3-jsonpointer-2.0-4.el9.noarch 2024-04-23T10:10:02.733 INFO:teuthology.orchestra.run.smithi204.stdout: python3-kubernetes-1:26.1.0-2.el9.noarch 2024-04-23T10:10:02.733 INFO:teuthology.orchestra.run.smithi204.stdout: python3-logutils-0.3.5-21.el9.noarch 2024-04-23T10:10:02.733 INFO:teuthology.orchestra.run.smithi204.stdout: python3-mako-1.1.4-6.el9.noarch 2024-04-23T10:10:02.733 INFO:teuthology.orchestra.run.smithi204.stdout: python3-markupsafe-1.1.1-12.el9.x86_64 2024-04-23T10:10:02.733 INFO:teuthology.orchestra.run.smithi204.stdout: python3-more-itertools-8.12.0-2.el9.noarch 2024-04-23T10:10:02.733 INFO:teuthology.orchestra.run.smithi204.stdout: python3-natsort-7.1.1-5.el9.noarch 2024-04-23T10:10:02.733 INFO:teuthology.orchestra.run.smithi204.stdout: python3-numpy-1:1.20.1-5.el9.x86_64 2024-04-23T10:10:02.733 INFO:teuthology.orchestra.run.smithi204.stdout: python3-numpy-f2py-1:1.20.1-5.el9.x86_64 2024-04-23T10:10:02.733 INFO:teuthology.orchestra.run.smithi204.stdout: python3-oauthlib-3.1.1-5.el9.noarch 2024-04-23T10:10:02.733 INFO:teuthology.orchestra.run.smithi204.stdout: python3-packaging-20.9-5.el9.noarch 2024-04-23T10:10:02.733 INFO:teuthology.orchestra.run.smithi204.stdout: python3-pecan-1.4.2-3.el9.noarch 2024-04-23T10:10:02.733 INFO:teuthology.orchestra.run.smithi204.stdout: python3-ply-3.11-14.el9.noarch 2024-04-23T10:10:02.733 INFO:teuthology.orchestra.run.smithi204.stdout: python3-portend-3.1.0-2.el9.noarch 2024-04-23T10:10:02.734 INFO:teuthology.orchestra.run.smithi204.stdout: python3-prettytable-0.7.2-27.el9.noarch 2024-04-23T10:10:02.734 INFO:teuthology.orchestra.run.smithi204.stdout: python3-protobuf-3.14.0-13.el9.noarch 2024-04-23T10:10:02.734 INFO:teuthology.orchestra.run.smithi204.stdout: python3-pyOpenSSL-21.0.0-1.el9.noarch 2024-04-23T10:10:02.734 INFO:teuthology.orchestra.run.smithi204.stdout: python3-pyasn1-0.4.8-6.el9.noarch 2024-04-23T10:10:02.734 INFO:teuthology.orchestra.run.smithi204.stdout: python3-pyasn1-modules-0.4.8-6.el9.noarch 2024-04-23T10:10:02.734 INFO:teuthology.orchestra.run.smithi204.stdout: python3-pycparser-2.20-6.el9.noarch 2024-04-23T10:10:02.734 INFO:teuthology.orchestra.run.smithi204.stdout: python3-pyperclip-1.8.2-2.el9.noarch 2024-04-23T10:10:02.734 INFO:teuthology.orchestra.run.smithi204.stdout: python3-pytz-2021.1-5.el9.noarch 2024-04-23T10:10:02.734 INFO:teuthology.orchestra.run.smithi204.stdout: python3-rados-2:19.0.0-3156.gd69f13d8.el7.x86_64 2024-04-23T10:10:02.734 INFO:teuthology.orchestra.run.smithi204.stdout: python3-rbd-2:19.0.0-3156.gd69f13d8.el7.x86_64 2024-04-23T10:10:02.734 INFO:teuthology.orchestra.run.smithi204.stdout: python3-repoze-lru-0.7-16.el9.noarch 2024-04-23T10:10:02.734 INFO:teuthology.orchestra.run.smithi204.stdout: python3-requests-2.25.1-8.el9.noarch 2024-04-23T10:10:02.734 INFO:teuthology.orchestra.run.smithi204.stdout: python3-requests-oauthlib-1.3.0-12.el9.noarch 2024-04-23T10:10:02.735 INFO:teuthology.orchestra.run.smithi204.stdout: python3-rgw-2:19.0.0-3156.gd69f13d8.el7.x86_64 2024-04-23T10:10:02.735 INFO:teuthology.orchestra.run.smithi204.stdout: python3-routes-2.5.1-5.el9.noarch 2024-04-23T10:10:02.735 INFO:teuthology.orchestra.run.smithi204.stdout: python3-rsa-4.9-2.el9.noarch 2024-04-23T10:10:02.735 INFO:teuthology.orchestra.run.smithi204.stdout: python3-scipy-1.6.2-8.el9.x86_64 2024-04-23T10:10:02.735 INFO:teuthology.orchestra.run.smithi204.stdout: python3-tempora-5.0.0-2.el9.noarch 2024-04-23T10:10:02.735 INFO:teuthology.orchestra.run.smithi204.stdout: python3-toml-0.10.2-6.el9.noarch 2024-04-23T10:10:02.735 INFO:teuthology.orchestra.run.smithi204.stdout: python3-typing-extensions-4.4.0-2.el9.noarch 2024-04-23T10:10:02.735 INFO:teuthology.orchestra.run.smithi204.stdout: python3-urllib3-1.26.5-5.el9.noarch 2024-04-23T10:10:02.735 INFO:teuthology.orchestra.run.smithi204.stdout: python3-wcwidth-0.2.5-8.el9.noarch 2024-04-23T10:10:02.735 INFO:teuthology.orchestra.run.smithi204.stdout: python3-webob-1.8.7-6.el9.noarch 2024-04-23T10:10:02.735 INFO:teuthology.orchestra.run.smithi204.stdout: python3-websocket-client-1.2.3-2.el9.noarch 2024-04-23T10:10:02.735 INFO:teuthology.orchestra.run.smithi204.stdout: python3-werkzeug-2.0.3-3.el9.1.noarch 2024-04-23T10:10:02.735 INFO:teuthology.orchestra.run.smithi204.stdout: python3-zc-lockfile-2.0-10.el9.noarch 2024-04-23T10:10:02.735 INFO:teuthology.orchestra.run.smithi204.stdout: qatlib-24.02.0-1.el9.x86_64 2024-04-23T10:10:02.735 INFO:teuthology.orchestra.run.smithi204.stdout: qatlib-service-24.02.0-1.el9.x86_64 2024-04-23T10:10:02.736 INFO:teuthology.orchestra.run.smithi204.stdout: qatzip-libs-1.2.0-1.el9.x86_64 2024-04-23T10:10:02.736 INFO:teuthology.orchestra.run.smithi204.stdout: rbd-fuse-2:19.0.0-3156.gd69f13d8.el7.x86_64 2024-04-23T10:10:02.736 INFO:teuthology.orchestra.run.smithi204.stdout: rbd-mirror-2:19.0.0-3156.gd69f13d8.el7.x86_64 2024-04-23T10:10:02.736 INFO:teuthology.orchestra.run.smithi204.stdout: rbd-nbd-2:19.0.0-3156.gd69f13d8.el7.x86_64 2024-04-23T10:10:02.736 INFO:teuthology.orchestra.run.smithi204.stdout: re2-1:20211101-3.el9.x86_64 2024-04-23T10:10:02.736 INFO:teuthology.orchestra.run.smithi204.stdout: socat-1.7.4.1-5.el9.x86_64 2024-04-23T10:10:02.736 INFO:teuthology.orchestra.run.smithi204.stdout: thrift-0.15.0-2.el9.x86_64 2024-04-23T10:10:02.736 INFO:teuthology.orchestra.run.smithi204.stdout: uuid-1.6.2-55.el9.x86_64 2024-04-23T10:10:02.736 INFO:teuthology.orchestra.run.smithi204.stdout: xfsprogs-devel-6.3.0-1.el9.x86_64 2024-04-23T10:10:02.736 INFO:teuthology.orchestra.run.smithi204.stdout: xmlstarlet-1.6.1-20.el9.x86_64 2024-04-23T10:10:02.736 INFO:teuthology.orchestra.run.smithi204.stdout: zlib-devel-1.2.11-41.el9.x86_64 2024-04-23T10:10:02.736 INFO:teuthology.orchestra.run.smithi204.stdout: 2024-04-23T10:10:02.737 INFO:teuthology.orchestra.run.smithi204.stdout:Complete! 2024-04-23T10:10:03.161 DEBUG:teuthology.parallel:result is None 2024-04-23T10:10:04.822 INFO:teuthology.orchestra.run.smithi114.stdout: Running scriptlet: openssl-libs-1:3.0.7-27.el9.x86_64 156/156 2024-04-23T10:10:04.822 INFO:teuthology.orchestra.run.smithi114.stdout: Verifying : ceph-2:19.0.0-3156.gd69f13d8.el7.x86_64 1/156 2024-04-23T10:10:04.822 INFO:teuthology.orchestra.run.smithi114.stdout: Verifying : ceph-base-2:19.0.0-3156.gd69f13d8.el7.x86_64 2/156 2024-04-23T10:10:04.822 INFO:teuthology.orchestra.run.smithi114.stdout: Verifying : ceph-common-2:19.0.0-3156.gd69f13d8.el7.x86_64 3/156 2024-04-23T10:10:04.822 INFO:teuthology.orchestra.run.smithi114.stdout: Verifying : ceph-fuse-2:19.0.0-3156.gd69f13d8.el7.x86_64 4/156 2024-04-23T10:10:04.823 INFO:teuthology.orchestra.run.smithi114.stdout: Verifying : ceph-immutable-object-cache-2:19.0.0-3156.gd69f1 5/156 2024-04-23T10:10:04.823 INFO:teuthology.orchestra.run.smithi114.stdout: Verifying : ceph-mds-2:19.0.0-3156.gd69f13d8.el7.x86_64 6/156 2024-04-23T10:10:04.823 INFO:teuthology.orchestra.run.smithi114.stdout: Verifying : ceph-mgr-2:19.0.0-3156.gd69f13d8.el7.x86_64 7/156 2024-04-23T10:10:04.823 INFO:teuthology.orchestra.run.smithi114.stdout: Verifying : ceph-mon-2:19.0.0-3156.gd69f13d8.el7.x86_64 8/156 2024-04-23T10:10:04.823 INFO:teuthology.orchestra.run.smithi114.stdout: Verifying : ceph-osd-2:19.0.0-3156.gd69f13d8.el7.x86_64 9/156 2024-04-23T10:10:04.823 INFO:teuthology.orchestra.run.smithi114.stdout: Verifying : ceph-radosgw-2:19.0.0-3156.gd69f13d8.el7.x86_64 10/156 2024-04-23T10:10:04.823 INFO:teuthology.orchestra.run.smithi114.stdout: Verifying : ceph-selinux-2:19.0.0-3156.gd69f13d8.el7.x86_64 11/156 2024-04-23T10:10:04.823 INFO:teuthology.orchestra.run.smithi114.stdout: Verifying : ceph-test-2:19.0.0-3156.gd69f13d8.el7.x86_64 12/156 2024-04-23T10:10:04.823 INFO:teuthology.orchestra.run.smithi114.stdout: Verifying : cephfs-mirror-2:19.0.0-3156.gd69f13d8.el7.x86_64 13/156 2024-04-23T10:10:04.823 INFO:teuthology.orchestra.run.smithi114.stdout: Verifying : cephfs-shell-2:19.0.0-3156.gd69f13d8.el7.x86_64 14/156 2024-04-23T10:10:04.823 INFO:teuthology.orchestra.run.smithi114.stdout: Verifying : libcephfs-devel-2:19.0.0-3156.gd69f13d8.el7.x86_ 15/156 2024-04-23T10:10:04.823 INFO:teuthology.orchestra.run.smithi114.stdout: Verifying : libcephfs2-2:19.0.0-3156.gd69f13d8.el7.x86_64 16/156 2024-04-23T10:10:04.823 INFO:teuthology.orchestra.run.smithi114.stdout: Verifying : libcephsqlite-2:19.0.0-3156.gd69f13d8.el7.x86_64 17/156 2024-04-23T10:10:04.823 INFO:teuthology.orchestra.run.smithi114.stdout: Verifying : librados-devel-2:19.0.0-3156.gd69f13d8.el7.x86_6 18/156 2024-04-23T10:10:04.824 INFO:teuthology.orchestra.run.smithi114.stdout: Verifying : libradosstriper1-2:19.0.0-3156.gd69f13d8.el7.x86 19/156 2024-04-23T10:10:04.824 INFO:teuthology.orchestra.run.smithi114.stdout: Verifying : librgw2-2:19.0.0-3156.gd69f13d8.el7.x86_64 20/156 2024-04-23T10:10:04.824 INFO:teuthology.orchestra.run.smithi114.stdout: Verifying : python3-ceph-argparse-2:19.0.0-3156.gd69f13d8.el 21/156 2024-04-23T10:10:04.824 INFO:teuthology.orchestra.run.smithi114.stdout: Verifying : python3-ceph-common-2:19.0.0-3156.gd69f13d8.el7. 22/156 2024-04-23T10:10:04.824 INFO:teuthology.orchestra.run.smithi114.stdout: Verifying : python3-cephfs-2:19.0.0-3156.gd69f13d8.el7.x86_6 23/156 2024-04-23T10:10:04.824 INFO:teuthology.orchestra.run.smithi114.stdout: Verifying : python3-rados-2:19.0.0-3156.gd69f13d8.el7.x86_64 24/156 2024-04-23T10:10:04.824 INFO:teuthology.orchestra.run.smithi114.stdout: Verifying : python3-rbd-2:19.0.0-3156.gd69f13d8.el7.x86_64 25/156 2024-04-23T10:10:04.824 INFO:teuthology.orchestra.run.smithi114.stdout: Verifying : python3-rgw-2:19.0.0-3156.gd69f13d8.el7.x86_64 26/156 2024-04-23T10:10:04.824 INFO:teuthology.orchestra.run.smithi114.stdout: Verifying : rbd-fuse-2:19.0.0-3156.gd69f13d8.el7.x86_64 27/156 2024-04-23T10:10:04.824 INFO:teuthology.orchestra.run.smithi114.stdout: Verifying : rbd-mirror-2:19.0.0-3156.gd69f13d8.el7.x86_64 28/156 2024-04-23T10:10:04.824 INFO:teuthology.orchestra.run.smithi114.stdout: Verifying : rbd-nbd-2:19.0.0-3156.gd69f13d8.el7.x86_64 29/156 2024-04-23T10:10:04.836 INFO:teuthology.orchestra.run.smithi114.stdout: Verifying : ceph-grafana-dashboards-2:19.0.0-3156.gd69f13d8. 30/156 2024-04-23T10:10:04.836 INFO:teuthology.orchestra.run.smithi114.stdout: Verifying : ceph-mgr-cephadm-2:19.0.0-3156.gd69f13d8.el7.noa 31/156 2024-04-23T10:10:04.836 INFO:teuthology.orchestra.run.smithi114.stdout: Verifying : ceph-mgr-dashboard-2:19.0.0-3156.gd69f13d8.el7.n 32/156 2024-04-23T10:10:04.836 INFO:teuthology.orchestra.run.smithi114.stdout: Verifying : ceph-mgr-diskprediction-local-2:19.0.0-3156.gd69 33/156 2024-04-23T10:10:04.836 INFO:teuthology.orchestra.run.smithi114.stdout: Verifying : ceph-mgr-modules-core-2:19.0.0-3156.gd69f13d8.el 34/156 2024-04-23T10:10:04.837 INFO:teuthology.orchestra.run.smithi114.stdout: Verifying : ceph-mgr-rook-2:19.0.0-3156.gd69f13d8.el7.noarch 35/156 2024-04-23T10:10:04.837 INFO:teuthology.orchestra.run.smithi114.stdout: Verifying : ceph-prometheus-alerts-2:19.0.0-3156.gd69f13d8.e 36/156 2024-04-23T10:10:04.837 INFO:teuthology.orchestra.run.smithi114.stdout: Verifying : ceph-volume-2:19.0.0-3156.gd69f13d8.el7.noarch 37/156 2024-04-23T10:10:04.837 INFO:teuthology.orchestra.run.smithi114.stdout: Verifying : cephadm-2:19.0.0-3156.gd69f13d8.el7.noarch 38/156 2024-04-23T10:10:04.837 INFO:teuthology.orchestra.run.smithi114.stdout: Verifying : cephfs-top-2:19.0.0-3156.gd69f13d8.el7.noarch 39/156 2024-04-23T10:10:04.837 INFO:teuthology.orchestra.run.smithi114.stdout: Verifying : fuse-2.9.9-15.el9.x86_64 40/156 2024-04-23T10:10:04.837 INFO:teuthology.orchestra.run.smithi114.stdout: Verifying : libgfortran-11.4.1-3.el9.x86_64 41/156 2024-04-23T10:10:04.837 INFO:teuthology.orchestra.run.smithi114.stdout: Verifying : libquadmath-11.4.1-3.el9.x86_64 42/156 2024-04-23T10:10:04.837 INFO:teuthology.orchestra.run.smithi114.stdout: Verifying : python3-cffi-1.14.5-5.el9.x86_64 43/156 2024-04-23T10:10:04.837 INFO:teuthology.orchestra.run.smithi114.stdout: Verifying : python3-cryptography-36.0.1-4.el9.x86_64 44/156 2024-04-23T10:10:04.837 INFO:teuthology.orchestra.run.smithi114.stdout: Verifying : python3-ply-3.11-14.el9.noarch 45/156 2024-04-23T10:10:04.837 INFO:teuthology.orchestra.run.smithi114.stdout: Verifying : python3-pycparser-2.20-6.el9.noarch 46/156 2024-04-23T10:10:04.837 INFO:teuthology.orchestra.run.smithi114.stdout: Verifying : python3-requests-2.25.1-8.el9.noarch 47/156 2024-04-23T10:10:04.837 INFO:teuthology.orchestra.run.smithi114.stdout: Verifying : python3-urllib3-1.26.5-5.el9.noarch 48/156 2024-04-23T10:10:04.838 INFO:teuthology.orchestra.run.smithi114.stdout: Verifying : automake-1.16.2-8.el9.noarch 49/156 2024-04-23T10:10:04.838 INFO:teuthology.orchestra.run.smithi114.stdout: Verifying : bison-3.7.4-5.el9.x86_64 50/156 2024-04-23T10:10:04.838 INFO:teuthology.orchestra.run.smithi114.stdout: Verifying : boost-program-options-1.75.0-8.el9.x86_64 51/156 2024-04-23T10:10:04.838 INFO:teuthology.orchestra.run.smithi114.stdout: Verifying : elfutils-libelf-devel-0.190-2.el9.x86_64 52/156 2024-04-23T10:10:04.838 INFO:teuthology.orchestra.run.smithi114.stdout: Verifying : flex-2.6.4-9.el9.x86_64 53/156 2024-04-23T10:10:04.838 INFO:teuthology.orchestra.run.smithi114.stdout: Verifying : flexiblas-3.0.4-8.el9.x86_64 54/156 2024-04-23T10:10:04.838 INFO:teuthology.orchestra.run.smithi114.stdout: Verifying : flexiblas-netlib-3.0.4-8.el9.x86_64 55/156 2024-04-23T10:10:04.838 INFO:teuthology.orchestra.run.smithi114.stdout: Verifying : flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 56/156 2024-04-23T10:10:04.838 INFO:teuthology.orchestra.run.smithi114.stdout: Verifying : libaio-devel-0.3.111-13.el9.x86_64 57/156 2024-04-23T10:10:04.838 INFO:teuthology.orchestra.run.smithi114.stdout: Verifying : libpmemobj-1.12.1-1.el9.x86_64 58/156 2024-04-23T10:10:04.838 INFO:teuthology.orchestra.run.smithi114.stdout: Verifying : librabbitmq-0.11.0-7.el9.x86_64 59/156 2024-04-23T10:10:04.838 INFO:teuthology.orchestra.run.smithi114.stdout: Verifying : librdkafka-1.6.1-102.el9.x86_64 60/156 2024-04-23T10:10:04.838 INFO:teuthology.orchestra.run.smithi114.stdout: Verifying : libtool-2.4.6-45.el9.x86_64 61/156 2024-04-23T10:10:04.838 INFO:teuthology.orchestra.run.smithi114.stdout: Verifying : libuuid-devel-2.37.4-18.el9.x86_64 62/156 2024-04-23T10:10:04.839 INFO:teuthology.orchestra.run.smithi114.stdout: Verifying : libzstd-devel-1.5.1-2.el9.x86_64 63/156 2024-04-23T10:10:04.839 INFO:teuthology.orchestra.run.smithi114.stdout: Verifying : lttng-ust-2.12.0-6.el9.x86_64 64/156 2024-04-23T10:10:04.839 INFO:teuthology.orchestra.run.smithi114.stdout: Verifying : lua-5.4.4-4.el9.x86_64 65/156 2024-04-23T10:10:04.839 INFO:teuthology.orchestra.run.smithi114.stdout: Verifying : openblas-0.3.21-2.el9.x86_64 66/156 2024-04-23T10:10:04.839 INFO:teuthology.orchestra.run.smithi114.stdout: Verifying : openblas-openmp-0.3.21-2.el9.x86_64 67/156 2024-04-23T10:10:04.839 INFO:teuthology.orchestra.run.smithi114.stdout: Verifying : openssl-devel-1:3.2.1-1.el9.x86_64 68/156 2024-04-23T10:10:04.839 INFO:teuthology.orchestra.run.smithi114.stdout: Verifying : postgresql-13.14-1.el9.x86_64 69/156 2024-04-23T10:10:04.839 INFO:teuthology.orchestra.run.smithi114.stdout: Verifying : postgresql-contrib-13.14-1.el9.x86_64 70/156 2024-04-23T10:10:04.839 INFO:teuthology.orchestra.run.smithi114.stdout: Verifying : postgresql-private-libs-13.14-1.el9.x86_64 71/156 2024-04-23T10:10:04.839 INFO:teuthology.orchestra.run.smithi114.stdout: Verifying : postgresql-server-13.14-1.el9.x86_64 72/156 2024-04-23T10:10:04.839 INFO:teuthology.orchestra.run.smithi114.stdout: Verifying : protobuf-3.14.0-13.el9.x86_64 73/156 2024-04-23T10:10:04.839 INFO:teuthology.orchestra.run.smithi114.stdout: Verifying : python3-attrs-20.3.0-7.el9.noarch 74/156 2024-04-23T10:10:04.839 INFO:teuthology.orchestra.run.smithi114.stdout: Verifying : python3-babel-2.9.1-2.el9.noarch 75/156 2024-04-23T10:10:04.839 INFO:teuthology.orchestra.run.smithi114.stdout: Verifying : python3-devel-3.9.18-3.el9.x86_64 76/156 2024-04-23T10:10:04.839 INFO:teuthology.orchestra.run.smithi114.stdout: Verifying : python3-jinja2-2.11.3-5.el9.noarch 77/156 2024-04-23T10:10:04.839 INFO:teuthology.orchestra.run.smithi114.stdout: Verifying : python3-jsonpatch-1.21-16.el9.noarch 78/156 2024-04-23T10:10:04.840 INFO:teuthology.orchestra.run.smithi114.stdout: Verifying : python3-jsonpointer-2.0-4.el9.noarch 79/156 2024-04-23T10:10:04.840 INFO:teuthology.orchestra.run.smithi114.stdout: Verifying : python3-mako-1.1.4-6.el9.noarch 80/156 2024-04-23T10:10:04.840 INFO:teuthology.orchestra.run.smithi114.stdout: Verifying : python3-markupsafe-1.1.1-12.el9.x86_64 81/156 2024-04-23T10:10:04.840 INFO:teuthology.orchestra.run.smithi114.stdout: Verifying : python3-numpy-1:1.20.1-5.el9.x86_64 82/156 2024-04-23T10:10:04.840 INFO:teuthology.orchestra.run.smithi114.stdout: Verifying : python3-numpy-f2py-1:1.20.1-5.el9.x86_64 83/156 2024-04-23T10:10:04.840 INFO:teuthology.orchestra.run.smithi114.stdout: Verifying : python3-oauthlib-3.1.1-5.el9.noarch 84/156 2024-04-23T10:10:04.840 INFO:teuthology.orchestra.run.smithi114.stdout: Verifying : python3-packaging-20.9-5.el9.noarch 85/156 2024-04-23T10:10:04.840 INFO:teuthology.orchestra.run.smithi114.stdout: Verifying : python3-prettytable-0.7.2-27.el9.noarch 86/156 2024-04-23T10:10:04.840 INFO:teuthology.orchestra.run.smithi114.stdout: Verifying : python3-protobuf-3.14.0-13.el9.noarch 87/156 2024-04-23T10:10:04.840 INFO:teuthology.orchestra.run.smithi114.stdout: Verifying : python3-pyasn1-0.4.8-6.el9.noarch 88/156 2024-04-23T10:10:04.840 INFO:teuthology.orchestra.run.smithi114.stdout: Verifying : python3-pyasn1-modules-0.4.8-6.el9.noarch 89/156 2024-04-23T10:10:04.840 INFO:teuthology.orchestra.run.smithi114.stdout: Verifying : python3-pytz-2021.1-5.el9.noarch 90/156 2024-04-23T10:10:04.840 INFO:teuthology.orchestra.run.smithi114.stdout: Verifying : python3-requests-oauthlib-1.3.0-12.el9.noarch 91/156 2024-04-23T10:10:04.840 INFO:teuthology.orchestra.run.smithi114.stdout: Verifying : python3-scipy-1.6.2-8.el9.x86_64 92/156 2024-04-23T10:10:04.841 INFO:teuthology.orchestra.run.smithi114.stdout: Verifying : python3-toml-0.10.2-6.el9.noarch 93/156 2024-04-23T10:10:04.841 INFO:teuthology.orchestra.run.smithi114.stdout: Verifying : python3-wcwidth-0.2.5-8.el9.noarch 94/156 2024-04-23T10:10:04.841 INFO:teuthology.orchestra.run.smithi114.stdout: Verifying : qatlib-24.02.0-1.el9.x86_64 95/156 2024-04-23T10:10:04.841 INFO:teuthology.orchestra.run.smithi114.stdout: Verifying : qatlib-service-24.02.0-1.el9.x86_64 96/156 2024-04-23T10:10:04.841 INFO:teuthology.orchestra.run.smithi114.stdout: Verifying : qatzip-libs-1.2.0-1.el9.x86_64 97/156 2024-04-23T10:10:04.841 INFO:teuthology.orchestra.run.smithi114.stdout: Verifying : socat-1.7.4.1-5.el9.x86_64 98/156 2024-04-23T10:10:04.841 INFO:teuthology.orchestra.run.smithi114.stdout: Verifying : uuid-1.6.2-55.el9.x86_64 99/156 2024-04-23T10:10:04.841 INFO:teuthology.orchestra.run.smithi114.stdout: Verifying : xfsprogs-devel-6.3.0-1.el9.x86_64 100/156 2024-04-23T10:10:04.841 INFO:teuthology.orchestra.run.smithi114.stdout: Verifying : xmlstarlet-1.6.1-20.el9.x86_64 101/156 2024-04-23T10:10:04.841 INFO:teuthology.orchestra.run.smithi114.stdout: Verifying : zlib-devel-1.2.11-41.el9.x86_64 102/156 2024-04-23T10:10:04.841 INFO:teuthology.orchestra.run.smithi114.stdout: Verifying : lua-devel-5.4.4-4.el9.x86_64 103/156 2024-04-23T10:10:04.841 INFO:teuthology.orchestra.run.smithi114.stdout: Verifying : protobuf-compiler-3.14.0-13.el9.x86_64 104/156 2024-04-23T10:10:04.841 INFO:teuthology.orchestra.run.smithi114.stdout: Verifying : abseil-cpp-20211102.0-3.el9.x86_64 105/156 2024-04-23T10:10:04.841 INFO:teuthology.orchestra.run.smithi114.stdout: Verifying : gperftools-libs-2.9.1-2.el9.x86_64 106/156 2024-04-23T10:10:04.841 INFO:teuthology.orchestra.run.smithi114.stdout: Verifying : grpc-data-1.46.7-10.el9.noarch 107/156 2024-04-23T10:10:04.841 INFO:teuthology.orchestra.run.smithi114.stdout: Verifying : libarrow-9.0.0-10.el9.x86_64 108/156 2024-04-23T10:10:04.841 INFO:teuthology.orchestra.run.smithi114.stdout: Verifying : libarrow-doc-9.0.0-10.el9.noarch 109/156 2024-04-23T10:10:04.842 INFO:teuthology.orchestra.run.smithi114.stdout: Verifying : liboath-2.6.7-2.el9.x86_64 110/156 2024-04-23T10:10:04.842 INFO:teuthology.orchestra.run.smithi114.stdout: Verifying : libunwind-1.6.2-1.el9.x86_64 111/156 2024-04-23T10:10:04.842 INFO:teuthology.orchestra.run.smithi114.stdout: Verifying : luarocks-3.9.2-1.el9.noarch 112/156 2024-04-23T10:10:04.842 INFO:teuthology.orchestra.run.smithi114.stdout: Verifying : parquet-libs-9.0.0-10.el9.x86_64 113/156 2024-04-23T10:10:04.842 INFO:teuthology.orchestra.run.smithi114.stdout: Verifying : python3-asyncssh-2.13.2-5.el9.noarch 114/156 2024-04-23T10:10:04.842 INFO:teuthology.orchestra.run.smithi114.stdout: Verifying : python3-bcrypt-3.2.2-1.el9.x86_64 115/156 2024-04-23T10:10:04.842 INFO:teuthology.orchestra.run.smithi114.stdout: Verifying : python3-cachetools-4.2.4-1.el9.noarch 116/156 2024-04-23T10:10:04.842 INFO:teuthology.orchestra.run.smithi114.stdout: Verifying : python3-certifi-2021.10.8-2.el9.noarch 117/156 2024-04-23T10:10:04.842 INFO:teuthology.orchestra.run.smithi114.stdout: Verifying : python3-cheroot-8.6.0-4.el9.noarch 118/156 2024-04-23T10:10:04.842 INFO:teuthology.orchestra.run.smithi114.stdout: Verifying : python3-cherrypy-18.6.1-2.el9.noarch 119/156 2024-04-23T10:10:04.842 INFO:teuthology.orchestra.run.smithi114.stdout: Verifying : python3-cmd2-2.4.2-2.el9.noarch 120/156 2024-04-23T10:10:04.842 INFO:teuthology.orchestra.run.smithi114.stdout: Verifying : python3-colorama-0.4.4-12.el9.noarch 121/156 2024-04-23T10:10:04.842 INFO:teuthology.orchestra.run.smithi114.stdout: Verifying : python3-google-auth-1:2.29.0-1.el9.noarch 122/156 2024-04-23T10:10:04.843 INFO:teuthology.orchestra.run.smithi114.stdout: Verifying : python3-grpcio-1.46.7-10.el9.x86_64 123/156 2024-04-23T10:10:04.843 INFO:teuthology.orchestra.run.smithi114.stdout: Verifying : python3-grpcio-tools-1.46.7-10.el9.x86_64 124/156 2024-04-23T10:10:04.843 INFO:teuthology.orchestra.run.smithi114.stdout: Verifying : python3-jaraco-8.2.1-3.el9.noarch 125/156 2024-04-23T10:10:04.843 INFO:teuthology.orchestra.run.smithi114.stdout: Verifying : python3-jaraco-classes-3.2.1-5.el9.noarch 126/156 2024-04-23T10:10:04.843 INFO:teuthology.orchestra.run.smithi114.stdout: Verifying : python3-jaraco-collections-3.0.0-8.el9.noarch 127/156 2024-04-23T10:10:04.843 INFO:teuthology.orchestra.run.smithi114.stdout: Verifying : python3-jaraco-functools-3.5.0-2.el9.noarch 128/156 2024-04-23T10:10:04.843 INFO:teuthology.orchestra.run.smithi114.stdout: Verifying : python3-jaraco-text-3.2.0-6.el9.noarch 129/156 2024-04-23T10:10:04.843 INFO:teuthology.orchestra.run.smithi114.stdout: Verifying : python3-kubernetes-1:26.1.0-2.el9.noarch 130/156 2024-04-23T10:10:04.843 INFO:teuthology.orchestra.run.smithi114.stdout: Verifying : python3-logutils-0.3.5-21.el9.noarch 131/156 2024-04-23T10:10:04.843 INFO:teuthology.orchestra.run.smithi114.stdout: Verifying : python3-more-itertools-8.12.0-2.el9.noarch 132/156 2024-04-23T10:10:04.843 INFO:teuthology.orchestra.run.smithi114.stdout: Verifying : python3-natsort-7.1.1-5.el9.noarch 133/156 2024-04-23T10:10:04.843 INFO:teuthology.orchestra.run.smithi114.stdout: Verifying : python3-pecan-1.4.2-3.el9.noarch 134/156 2024-04-23T10:10:04.843 INFO:teuthology.orchestra.run.smithi114.stdout: Verifying : python3-portend-3.1.0-2.el9.noarch 135/156 2024-04-23T10:10:04.843 INFO:teuthology.orchestra.run.smithi114.stdout: Verifying : python3-pyOpenSSL-21.0.0-1.el9.noarch 136/156 2024-04-23T10:10:04.844 INFO:teuthology.orchestra.run.smithi114.stdout: Verifying : python3-pyperclip-1.8.2-2.el9.noarch 137/156 2024-04-23T10:10:04.844 INFO:teuthology.orchestra.run.smithi114.stdout: Verifying : python3-repoze-lru-0.7-16.el9.noarch 138/156 2024-04-23T10:10:04.844 INFO:teuthology.orchestra.run.smithi114.stdout: Verifying : python3-routes-2.5.1-5.el9.noarch 139/156 2024-04-23T10:10:04.844 INFO:teuthology.orchestra.run.smithi114.stdout: Verifying : python3-rsa-4.9-2.el9.noarch 140/156 2024-04-23T10:10:04.844 INFO:teuthology.orchestra.run.smithi114.stdout: Verifying : python3-tempora-5.0.0-2.el9.noarch 141/156 2024-04-23T10:10:04.844 INFO:teuthology.orchestra.run.smithi114.stdout: Verifying : python3-typing-extensions-4.4.0-2.el9.noarch 142/156 2024-04-23T10:10:04.844 INFO:teuthology.orchestra.run.smithi114.stdout: Verifying : python3-webob-1.8.7-6.el9.noarch 143/156 2024-04-23T10:10:04.844 INFO:teuthology.orchestra.run.smithi114.stdout: Verifying : python3-websocket-client-1.2.3-2.el9.noarch 144/156 2024-04-23T10:10:04.844 INFO:teuthology.orchestra.run.smithi114.stdout: Verifying : python3-werkzeug-2.0.3-3.el9.1.noarch 145/156 2024-04-23T10:10:04.844 INFO:teuthology.orchestra.run.smithi114.stdout: Verifying : python3-zc-lockfile-2.0-10.el9.noarch 146/156 2024-04-23T10:10:04.844 INFO:teuthology.orchestra.run.smithi114.stdout: Verifying : re2-1:20211101-3.el9.x86_64 147/156 2024-04-23T10:10:04.844 INFO:teuthology.orchestra.run.smithi114.stdout: Verifying : thrift-0.15.0-2.el9.x86_64 148/156 2024-04-23T10:10:04.844 INFO:teuthology.orchestra.run.smithi114.stdout: Verifying : librados2-2:19.0.0-3156.gd69f13d8.el7.x86_64 149/156 2024-04-23T10:10:04.844 INFO:teuthology.orchestra.run.smithi114.stdout: Verifying : librados2-2:16.2.4-5.el9.x86_64 150/156 2024-04-23T10:10:04.844 INFO:teuthology.orchestra.run.smithi114.stdout: Verifying : librbd1-2:19.0.0-3156.gd69f13d8.el7.x86_64 151/156 2024-04-23T10:10:04.845 INFO:teuthology.orchestra.run.smithi114.stdout: Verifying : librbd1-2:16.2.4-5.el9.x86_64 152/156 2024-04-23T10:10:04.845 INFO:teuthology.orchestra.run.smithi114.stdout: Verifying : openssl-1:3.2.1-1.el9.x86_64 153/156 2024-04-23T10:10:04.845 INFO:teuthology.orchestra.run.smithi114.stdout: Verifying : openssl-1:3.0.7-27.el9.x86_64 154/156 2024-04-23T10:10:04.845 INFO:teuthology.orchestra.run.smithi114.stdout: Verifying : openssl-libs-1:3.2.1-1.el9.x86_64 155/156 2024-04-23T10:10:06.960 INFO:teuthology.orchestra.run.smithi114.stdout: Verifying : openssl-libs-1:3.0.7-27.el9.x86_64 156/156 2024-04-23T10:10:06.961 INFO:teuthology.orchestra.run.smithi114.stdout: 2024-04-23T10:10:06.961 INFO:teuthology.orchestra.run.smithi114.stdout:Upgraded: 2024-04-23T10:10:06.961 INFO:teuthology.orchestra.run.smithi114.stdout: librados2-2:19.0.0-3156.gd69f13d8.el7.x86_64 2024-04-23T10:10:06.961 INFO:teuthology.orchestra.run.smithi114.stdout: librbd1-2:19.0.0-3156.gd69f13d8.el7.x86_64 2024-04-23T10:10:06.961 INFO:teuthology.orchestra.run.smithi114.stdout: openssl-1:3.2.1-1.el9.x86_64 2024-04-23T10:10:06.961 INFO:teuthology.orchestra.run.smithi114.stdout: openssl-libs-1:3.2.1-1.el9.x86_64 2024-04-23T10:10:06.961 INFO:teuthology.orchestra.run.smithi114.stdout:Installed: 2024-04-23T10:10:06.961 INFO:teuthology.orchestra.run.smithi114.stdout: abseil-cpp-20211102.0-3.el9.x86_64 2024-04-23T10:10:06.961 INFO:teuthology.orchestra.run.smithi114.stdout: automake-1.16.2-8.el9.noarch 2024-04-23T10:10:06.961 INFO:teuthology.orchestra.run.smithi114.stdout: bison-3.7.4-5.el9.x86_64 2024-04-23T10:10:06.961 INFO:teuthology.orchestra.run.smithi114.stdout: boost-program-options-1.75.0-8.el9.x86_64 2024-04-23T10:10:06.962 INFO:teuthology.orchestra.run.smithi114.stdout: ceph-2:19.0.0-3156.gd69f13d8.el7.x86_64 2024-04-23T10:10:06.962 INFO:teuthology.orchestra.run.smithi114.stdout: ceph-base-2:19.0.0-3156.gd69f13d8.el7.x86_64 2024-04-23T10:10:06.962 INFO:teuthology.orchestra.run.smithi114.stdout: ceph-common-2:19.0.0-3156.gd69f13d8.el7.x86_64 2024-04-23T10:10:06.962 INFO:teuthology.orchestra.run.smithi114.stdout: ceph-fuse-2:19.0.0-3156.gd69f13d8.el7.x86_64 2024-04-23T10:10:06.962 INFO:teuthology.orchestra.run.smithi114.stdout: ceph-grafana-dashboards-2:19.0.0-3156.gd69f13d8.el7.noarch 2024-04-23T10:10:06.962 INFO:teuthology.orchestra.run.smithi114.stdout: ceph-immutable-object-cache-2:19.0.0-3156.gd69f13d8.el7.x86_64 2024-04-23T10:10:06.962 INFO:teuthology.orchestra.run.smithi114.stdout: ceph-mds-2:19.0.0-3156.gd69f13d8.el7.x86_64 2024-04-23T10:10:06.962 INFO:teuthology.orchestra.run.smithi114.stdout: ceph-mgr-2:19.0.0-3156.gd69f13d8.el7.x86_64 2024-04-23T10:10:06.962 INFO:teuthology.orchestra.run.smithi114.stdout: ceph-mgr-cephadm-2:19.0.0-3156.gd69f13d8.el7.noarch 2024-04-23T10:10:06.962 INFO:teuthology.orchestra.run.smithi114.stdout: ceph-mgr-dashboard-2:19.0.0-3156.gd69f13d8.el7.noarch 2024-04-23T10:10:06.962 INFO:teuthology.orchestra.run.smithi114.stdout: ceph-mgr-diskprediction-local-2:19.0.0-3156.gd69f13d8.el7.noarch 2024-04-23T10:10:06.962 INFO:teuthology.orchestra.run.smithi114.stdout: ceph-mgr-modules-core-2:19.0.0-3156.gd69f13d8.el7.noarch 2024-04-23T10:10:06.962 INFO:teuthology.orchestra.run.smithi114.stdout: ceph-mgr-rook-2:19.0.0-3156.gd69f13d8.el7.noarch 2024-04-23T10:10:06.962 INFO:teuthology.orchestra.run.smithi114.stdout: ceph-mon-2:19.0.0-3156.gd69f13d8.el7.x86_64 2024-04-23T10:10:06.963 INFO:teuthology.orchestra.run.smithi114.stdout: ceph-osd-2:19.0.0-3156.gd69f13d8.el7.x86_64 2024-04-23T10:10:06.963 INFO:teuthology.orchestra.run.smithi114.stdout: ceph-prometheus-alerts-2:19.0.0-3156.gd69f13d8.el7.noarch 2024-04-23T10:10:06.963 INFO:teuthology.orchestra.run.smithi114.stdout: ceph-radosgw-2:19.0.0-3156.gd69f13d8.el7.x86_64 2024-04-23T10:10:06.963 INFO:teuthology.orchestra.run.smithi114.stdout: ceph-selinux-2:19.0.0-3156.gd69f13d8.el7.x86_64 2024-04-23T10:10:06.963 INFO:teuthology.orchestra.run.smithi114.stdout: ceph-test-2:19.0.0-3156.gd69f13d8.el7.x86_64 2024-04-23T10:10:06.963 INFO:teuthology.orchestra.run.smithi114.stdout: ceph-volume-2:19.0.0-3156.gd69f13d8.el7.noarch 2024-04-23T10:10:06.963 INFO:teuthology.orchestra.run.smithi114.stdout: cephadm-2:19.0.0-3156.gd69f13d8.el7.noarch 2024-04-23T10:10:06.963 INFO:teuthology.orchestra.run.smithi114.stdout: cephfs-mirror-2:19.0.0-3156.gd69f13d8.el7.x86_64 2024-04-23T10:10:06.963 INFO:teuthology.orchestra.run.smithi114.stdout: cephfs-shell-2:19.0.0-3156.gd69f13d8.el7.x86_64 2024-04-23T10:10:06.963 INFO:teuthology.orchestra.run.smithi114.stdout: cephfs-top-2:19.0.0-3156.gd69f13d8.el7.noarch 2024-04-23T10:10:06.963 INFO:teuthology.orchestra.run.smithi114.stdout: elfutils-libelf-devel-0.190-2.el9.x86_64 2024-04-23T10:10:06.963 INFO:teuthology.orchestra.run.smithi114.stdout: flex-2.6.4-9.el9.x86_64 2024-04-23T10:10:06.964 INFO:teuthology.orchestra.run.smithi114.stdout: flexiblas-3.0.4-8.el9.x86_64 2024-04-23T10:10:06.964 INFO:teuthology.orchestra.run.smithi114.stdout: flexiblas-netlib-3.0.4-8.el9.x86_64 2024-04-23T10:10:06.964 INFO:teuthology.orchestra.run.smithi114.stdout: flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 2024-04-23T10:10:06.964 INFO:teuthology.orchestra.run.smithi114.stdout: fuse-2.9.9-15.el9.x86_64 2024-04-23T10:10:06.964 INFO:teuthology.orchestra.run.smithi114.stdout: gperftools-libs-2.9.1-2.el9.x86_64 2024-04-23T10:10:06.964 INFO:teuthology.orchestra.run.smithi114.stdout: grpc-data-1.46.7-10.el9.noarch 2024-04-23T10:10:06.964 INFO:teuthology.orchestra.run.smithi114.stdout: libaio-devel-0.3.111-13.el9.x86_64 2024-04-23T10:10:06.964 INFO:teuthology.orchestra.run.smithi114.stdout: libarrow-9.0.0-10.el9.x86_64 2024-04-23T10:10:06.964 INFO:teuthology.orchestra.run.smithi114.stdout: libarrow-doc-9.0.0-10.el9.noarch 2024-04-23T10:10:06.964 INFO:teuthology.orchestra.run.smithi114.stdout: libcephfs-devel-2:19.0.0-3156.gd69f13d8.el7.x86_64 2024-04-23T10:10:06.964 INFO:teuthology.orchestra.run.smithi114.stdout: libcephfs2-2:19.0.0-3156.gd69f13d8.el7.x86_64 2024-04-23T10:10:06.964 INFO:teuthology.orchestra.run.smithi114.stdout: libcephsqlite-2:19.0.0-3156.gd69f13d8.el7.x86_64 2024-04-23T10:10:06.964 INFO:teuthology.orchestra.run.smithi114.stdout: libgfortran-11.4.1-3.el9.x86_64 2024-04-23T10:10:06.964 INFO:teuthology.orchestra.run.smithi114.stdout: liboath-2.6.7-2.el9.x86_64 2024-04-23T10:10:06.965 INFO:teuthology.orchestra.run.smithi114.stdout: libpmemobj-1.12.1-1.el9.x86_64 2024-04-23T10:10:06.965 INFO:teuthology.orchestra.run.smithi114.stdout: libquadmath-11.4.1-3.el9.x86_64 2024-04-23T10:10:06.965 INFO:teuthology.orchestra.run.smithi114.stdout: librabbitmq-0.11.0-7.el9.x86_64 2024-04-23T10:10:06.965 INFO:teuthology.orchestra.run.smithi114.stdout: librados-devel-2:19.0.0-3156.gd69f13d8.el7.x86_64 2024-04-23T10:10:06.965 INFO:teuthology.orchestra.run.smithi114.stdout: libradosstriper1-2:19.0.0-3156.gd69f13d8.el7.x86_64 2024-04-23T10:10:06.965 INFO:teuthology.orchestra.run.smithi114.stdout: librdkafka-1.6.1-102.el9.x86_64 2024-04-23T10:10:06.965 INFO:teuthology.orchestra.run.smithi114.stdout: librgw2-2:19.0.0-3156.gd69f13d8.el7.x86_64 2024-04-23T10:10:06.965 INFO:teuthology.orchestra.run.smithi114.stdout: libtool-2.4.6-45.el9.x86_64 2024-04-23T10:10:06.965 INFO:teuthology.orchestra.run.smithi114.stdout: libunwind-1.6.2-1.el9.x86_64 2024-04-23T10:10:06.965 INFO:teuthology.orchestra.run.smithi114.stdout: libuuid-devel-2.37.4-18.el9.x86_64 2024-04-23T10:10:06.965 INFO:teuthology.orchestra.run.smithi114.stdout: libzstd-devel-1.5.1-2.el9.x86_64 2024-04-23T10:10:06.965 INFO:teuthology.orchestra.run.smithi114.stdout: lttng-ust-2.12.0-6.el9.x86_64 2024-04-23T10:10:06.965 INFO:teuthology.orchestra.run.smithi114.stdout: lua-5.4.4-4.el9.x86_64 2024-04-23T10:10:06.966 INFO:teuthology.orchestra.run.smithi114.stdout: lua-devel-5.4.4-4.el9.x86_64 2024-04-23T10:10:06.966 INFO:teuthology.orchestra.run.smithi114.stdout: luarocks-3.9.2-1.el9.noarch 2024-04-23T10:10:06.966 INFO:teuthology.orchestra.run.smithi114.stdout: openblas-0.3.21-2.el9.x86_64 2024-04-23T10:10:06.966 INFO:teuthology.orchestra.run.smithi114.stdout: openblas-openmp-0.3.21-2.el9.x86_64 2024-04-23T10:10:06.966 INFO:teuthology.orchestra.run.smithi114.stdout: openssl-devel-1:3.2.1-1.el9.x86_64 2024-04-23T10:10:06.966 INFO:teuthology.orchestra.run.smithi114.stdout: parquet-libs-9.0.0-10.el9.x86_64 2024-04-23T10:10:06.966 INFO:teuthology.orchestra.run.smithi114.stdout: postgresql-13.14-1.el9.x86_64 2024-04-23T10:10:06.966 INFO:teuthology.orchestra.run.smithi114.stdout: postgresql-contrib-13.14-1.el9.x86_64 2024-04-23T10:10:06.966 INFO:teuthology.orchestra.run.smithi114.stdout: postgresql-private-libs-13.14-1.el9.x86_64 2024-04-23T10:10:06.966 INFO:teuthology.orchestra.run.smithi114.stdout: postgresql-server-13.14-1.el9.x86_64 2024-04-23T10:10:06.966 INFO:teuthology.orchestra.run.smithi114.stdout: protobuf-3.14.0-13.el9.x86_64 2024-04-23T10:10:06.966 INFO:teuthology.orchestra.run.smithi114.stdout: protobuf-compiler-3.14.0-13.el9.x86_64 2024-04-23T10:10:06.966 INFO:teuthology.orchestra.run.smithi114.stdout: python3-asyncssh-2.13.2-5.el9.noarch 2024-04-23T10:10:06.967 INFO:teuthology.orchestra.run.smithi114.stdout: python3-attrs-20.3.0-7.el9.noarch 2024-04-23T10:10:06.967 INFO:teuthology.orchestra.run.smithi114.stdout: python3-babel-2.9.1-2.el9.noarch 2024-04-23T10:10:06.967 INFO:teuthology.orchestra.run.smithi114.stdout: python3-bcrypt-3.2.2-1.el9.x86_64 2024-04-23T10:10:06.967 INFO:teuthology.orchestra.run.smithi114.stdout: python3-cachetools-4.2.4-1.el9.noarch 2024-04-23T10:10:06.967 INFO:teuthology.orchestra.run.smithi114.stdout: python3-ceph-argparse-2:19.0.0-3156.gd69f13d8.el7.x86_64 2024-04-23T10:10:06.967 INFO:teuthology.orchestra.run.smithi114.stdout: python3-ceph-common-2:19.0.0-3156.gd69f13d8.el7.x86_64 2024-04-23T10:10:06.967 INFO:teuthology.orchestra.run.smithi114.stdout: python3-cephfs-2:19.0.0-3156.gd69f13d8.el7.x86_64 2024-04-23T10:10:06.967 INFO:teuthology.orchestra.run.smithi114.stdout: python3-certifi-2021.10.8-2.el9.noarch 2024-04-23T10:10:06.967 INFO:teuthology.orchestra.run.smithi114.stdout: python3-cffi-1.14.5-5.el9.x86_64 2024-04-23T10:10:06.967 INFO:teuthology.orchestra.run.smithi114.stdout: python3-cheroot-8.6.0-4.el9.noarch 2024-04-23T10:10:06.967 INFO:teuthology.orchestra.run.smithi114.stdout: python3-cherrypy-18.6.1-2.el9.noarch 2024-04-23T10:10:06.967 INFO:teuthology.orchestra.run.smithi114.stdout: python3-cmd2-2.4.2-2.el9.noarch 2024-04-23T10:10:06.967 INFO:teuthology.orchestra.run.smithi114.stdout: python3-colorama-0.4.4-12.el9.noarch 2024-04-23T10:10:06.967 INFO:teuthology.orchestra.run.smithi114.stdout: python3-cryptography-36.0.1-4.el9.x86_64 2024-04-23T10:10:06.967 INFO:teuthology.orchestra.run.smithi114.stdout: python3-devel-3.9.18-3.el9.x86_64 2024-04-23T10:10:06.968 INFO:teuthology.orchestra.run.smithi114.stdout: python3-google-auth-1:2.29.0-1.el9.noarch 2024-04-23T10:10:06.968 INFO:teuthology.orchestra.run.smithi114.stdout: python3-grpcio-1.46.7-10.el9.x86_64 2024-04-23T10:10:06.968 INFO:teuthology.orchestra.run.smithi114.stdout: python3-grpcio-tools-1.46.7-10.el9.x86_64 2024-04-23T10:10:06.968 INFO:teuthology.orchestra.run.smithi114.stdout: python3-jaraco-8.2.1-3.el9.noarch 2024-04-23T10:10:06.968 INFO:teuthology.orchestra.run.smithi114.stdout: python3-jaraco-classes-3.2.1-5.el9.noarch 2024-04-23T10:10:06.968 INFO:teuthology.orchestra.run.smithi114.stdout: python3-jaraco-collections-3.0.0-8.el9.noarch 2024-04-23T10:10:06.968 INFO:teuthology.orchestra.run.smithi114.stdout: python3-jaraco-functools-3.5.0-2.el9.noarch 2024-04-23T10:10:06.968 INFO:teuthology.orchestra.run.smithi114.stdout: python3-jaraco-text-3.2.0-6.el9.noarch 2024-04-23T10:10:06.968 INFO:teuthology.orchestra.run.smithi114.stdout: python3-jinja2-2.11.3-5.el9.noarch 2024-04-23T10:10:06.968 INFO:teuthology.orchestra.run.smithi114.stdout: python3-jsonpatch-1.21-16.el9.noarch 2024-04-23T10:10:06.968 INFO:teuthology.orchestra.run.smithi114.stdout: python3-jsonpointer-2.0-4.el9.noarch 2024-04-23T10:10:06.968 INFO:teuthology.orchestra.run.smithi114.stdout: python3-kubernetes-1:26.1.0-2.el9.noarch 2024-04-23T10:10:06.968 INFO:teuthology.orchestra.run.smithi114.stdout: python3-logutils-0.3.5-21.el9.noarch 2024-04-23T10:10:06.968 INFO:teuthology.orchestra.run.smithi114.stdout: python3-mako-1.1.4-6.el9.noarch 2024-04-23T10:10:06.968 INFO:teuthology.orchestra.run.smithi114.stdout: python3-markupsafe-1.1.1-12.el9.x86_64 2024-04-23T10:10:06.968 INFO:teuthology.orchestra.run.smithi114.stdout: python3-more-itertools-8.12.0-2.el9.noarch 2024-04-23T10:10:06.969 INFO:teuthology.orchestra.run.smithi114.stdout: python3-natsort-7.1.1-5.el9.noarch 2024-04-23T10:10:06.969 INFO:teuthology.orchestra.run.smithi114.stdout: python3-numpy-1:1.20.1-5.el9.x86_64 2024-04-23T10:10:06.969 INFO:teuthology.orchestra.run.smithi114.stdout: python3-numpy-f2py-1:1.20.1-5.el9.x86_64 2024-04-23T10:10:06.969 INFO:teuthology.orchestra.run.smithi114.stdout: python3-oauthlib-3.1.1-5.el9.noarch 2024-04-23T10:10:06.969 INFO:teuthology.orchestra.run.smithi114.stdout: python3-packaging-20.9-5.el9.noarch 2024-04-23T10:10:06.969 INFO:teuthology.orchestra.run.smithi114.stdout: python3-pecan-1.4.2-3.el9.noarch 2024-04-23T10:10:06.969 INFO:teuthology.orchestra.run.smithi114.stdout: python3-ply-3.11-14.el9.noarch 2024-04-23T10:10:06.969 INFO:teuthology.orchestra.run.smithi114.stdout: python3-portend-3.1.0-2.el9.noarch 2024-04-23T10:10:06.969 INFO:teuthology.orchestra.run.smithi114.stdout: python3-prettytable-0.7.2-27.el9.noarch 2024-04-23T10:10:06.969 INFO:teuthology.orchestra.run.smithi114.stdout: python3-protobuf-3.14.0-13.el9.noarch 2024-04-23T10:10:06.969 INFO:teuthology.orchestra.run.smithi114.stdout: python3-pyOpenSSL-21.0.0-1.el9.noarch 2024-04-23T10:10:06.969 INFO:teuthology.orchestra.run.smithi114.stdout: python3-pyasn1-0.4.8-6.el9.noarch 2024-04-23T10:10:06.969 INFO:teuthology.orchestra.run.smithi114.stdout: python3-pyasn1-modules-0.4.8-6.el9.noarch 2024-04-23T10:10:06.969 INFO:teuthology.orchestra.run.smithi114.stdout: python3-pycparser-2.20-6.el9.noarch 2024-04-23T10:10:06.969 INFO:teuthology.orchestra.run.smithi114.stdout: python3-pyperclip-1.8.2-2.el9.noarch 2024-04-23T10:10:06.970 INFO:teuthology.orchestra.run.smithi114.stdout: python3-pytz-2021.1-5.el9.noarch 2024-04-23T10:10:06.970 INFO:teuthology.orchestra.run.smithi114.stdout: python3-rados-2:19.0.0-3156.gd69f13d8.el7.x86_64 2024-04-23T10:10:06.970 INFO:teuthology.orchestra.run.smithi114.stdout: python3-rbd-2:19.0.0-3156.gd69f13d8.el7.x86_64 2024-04-23T10:10:06.970 INFO:teuthology.orchestra.run.smithi114.stdout: python3-repoze-lru-0.7-16.el9.noarch 2024-04-23T10:10:06.970 INFO:teuthology.orchestra.run.smithi114.stdout: python3-requests-2.25.1-8.el9.noarch 2024-04-23T10:10:06.970 INFO:teuthology.orchestra.run.smithi114.stdout: python3-requests-oauthlib-1.3.0-12.el9.noarch 2024-04-23T10:10:06.970 INFO:teuthology.orchestra.run.smithi114.stdout: python3-rgw-2:19.0.0-3156.gd69f13d8.el7.x86_64 2024-04-23T10:10:06.970 INFO:teuthology.orchestra.run.smithi114.stdout: python3-routes-2.5.1-5.el9.noarch 2024-04-23T10:10:06.970 INFO:teuthology.orchestra.run.smithi114.stdout: python3-rsa-4.9-2.el9.noarch 2024-04-23T10:10:06.970 INFO:teuthology.orchestra.run.smithi114.stdout: python3-scipy-1.6.2-8.el9.x86_64 2024-04-23T10:10:06.970 INFO:teuthology.orchestra.run.smithi114.stdout: python3-tempora-5.0.0-2.el9.noarch 2024-04-23T10:10:06.970 INFO:teuthology.orchestra.run.smithi114.stdout: python3-toml-0.10.2-6.el9.noarch 2024-04-23T10:10:06.970 INFO:teuthology.orchestra.run.smithi114.stdout: python3-typing-extensions-4.4.0-2.el9.noarch 2024-04-23T10:10:06.970 INFO:teuthology.orchestra.run.smithi114.stdout: python3-urllib3-1.26.5-5.el9.noarch 2024-04-23T10:10:06.970 INFO:teuthology.orchestra.run.smithi114.stdout: python3-wcwidth-0.2.5-8.el9.noarch 2024-04-23T10:10:06.970 INFO:teuthology.orchestra.run.smithi114.stdout: python3-webob-1.8.7-6.el9.noarch 2024-04-23T10:10:06.971 INFO:teuthology.orchestra.run.smithi114.stdout: python3-websocket-client-1.2.3-2.el9.noarch 2024-04-23T10:10:06.971 INFO:teuthology.orchestra.run.smithi114.stdout: python3-werkzeug-2.0.3-3.el9.1.noarch 2024-04-23T10:10:06.971 INFO:teuthology.orchestra.run.smithi114.stdout: python3-zc-lockfile-2.0-10.el9.noarch 2024-04-23T10:10:06.971 INFO:teuthology.orchestra.run.smithi114.stdout: qatlib-24.02.0-1.el9.x86_64 2024-04-23T10:10:06.971 INFO:teuthology.orchestra.run.smithi114.stdout: qatlib-service-24.02.0-1.el9.x86_64 2024-04-23T10:10:06.971 INFO:teuthology.orchestra.run.smithi114.stdout: qatzip-libs-1.2.0-1.el9.x86_64 2024-04-23T10:10:06.971 INFO:teuthology.orchestra.run.smithi114.stdout: rbd-fuse-2:19.0.0-3156.gd69f13d8.el7.x86_64 2024-04-23T10:10:06.971 INFO:teuthology.orchestra.run.smithi114.stdout: rbd-mirror-2:19.0.0-3156.gd69f13d8.el7.x86_64 2024-04-23T10:10:06.971 INFO:teuthology.orchestra.run.smithi114.stdout: rbd-nbd-2:19.0.0-3156.gd69f13d8.el7.x86_64 2024-04-23T10:10:06.971 INFO:teuthology.orchestra.run.smithi114.stdout: re2-1:20211101-3.el9.x86_64 2024-04-23T10:10:06.971 INFO:teuthology.orchestra.run.smithi114.stdout: socat-1.7.4.1-5.el9.x86_64 2024-04-23T10:10:06.971 INFO:teuthology.orchestra.run.smithi114.stdout: thrift-0.15.0-2.el9.x86_64 2024-04-23T10:10:06.971 INFO:teuthology.orchestra.run.smithi114.stdout: uuid-1.6.2-55.el9.x86_64 2024-04-23T10:10:06.971 INFO:teuthology.orchestra.run.smithi114.stdout: xfsprogs-devel-6.3.0-1.el9.x86_64 2024-04-23T10:10:06.971 INFO:teuthology.orchestra.run.smithi114.stdout: xmlstarlet-1.6.1-20.el9.x86_64 2024-04-23T10:10:06.972 INFO:teuthology.orchestra.run.smithi114.stdout: zlib-devel-1.2.11-41.el9.x86_64 2024-04-23T10:10:06.972 INFO:teuthology.orchestra.run.smithi114.stdout: 2024-04-23T10:10:06.972 INFO:teuthology.orchestra.run.smithi114.stdout:Complete! 2024-04-23T10:10:07.348 DEBUG:teuthology.parallel:result is None 2024-04-23T10:10:07.348 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&sha1=d69f13d827952bb07bb9b7973bca14f97d764d6f 2024-04-23T10:10:07.559 DEBUG:teuthology.orchestra.run.smithi114:> rpm -q ceph --qf '%{VERSION}-%{RELEASE}' 2024-04-23T10:10:07.581 INFO:teuthology.orchestra.run.smithi114.stdout:19.0.0-3156.gd69f13d8.el7 2024-04-23T10:10:07.581 INFO:teuthology.packaging:The installed version of ceph is 19.0.0-3156.gd69f13d8.el7 2024-04-23T10:10:07.581 INFO:teuthology.task.install:The correct ceph version 19.0.0-3156.gd69f13d8 is installed. 2024-04-23T10:10:07.583 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&sha1=d69f13d827952bb07bb9b7973bca14f97d764d6f 2024-04-23T10:10:07.799 DEBUG:teuthology.orchestra.run.smithi119:> rpm -q ceph --qf '%{VERSION}-%{RELEASE}' 2024-04-23T10:10:07.825 INFO:teuthology.orchestra.run.smithi119.stdout:19.0.0-3156.gd69f13d8.el7 2024-04-23T10:10:07.826 INFO:teuthology.packaging:The installed version of ceph is 19.0.0-3156.gd69f13d8.el7 2024-04-23T10:10:07.826 INFO:teuthology.task.install:The correct ceph version 19.0.0-3156.gd69f13d8 is installed. 2024-04-23T10:10:07.828 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&sha1=d69f13d827952bb07bb9b7973bca14f97d764d6f 2024-04-23T10:10:07.969 DEBUG:teuthology.orchestra.run.smithi204:> rpm -q ceph --qf '%{VERSION}-%{RELEASE}' 2024-04-23T10:10:07.995 INFO:teuthology.orchestra.run.smithi204.stdout:19.0.0-3156.gd69f13d8.el7 2024-04-23T10:10:07.995 INFO:teuthology.packaging:The installed version of ceph is 19.0.0-3156.gd69f13d8.el7 2024-04-23T10:10:07.995 INFO:teuthology.task.install:The correct ceph version 19.0.0-3156.gd69f13d8 is installed. 2024-04-23T10:10:07.996 INFO:teuthology.task.install.util:Shipping valgrind.supp... 2024-04-23T10:10:07.996 DEBUG:teuthology.orchestra.run.smithi114:> set -ex 2024-04-23T10:10:07.996 DEBUG:teuthology.orchestra.run.smithi114:> sudo dd of=/home/ubuntu/cephtest/valgrind.supp 2024-04-23T10:10:08.022 DEBUG:teuthology.orchestra.run.smithi119:> set -ex 2024-04-23T10:10:08.022 DEBUG:teuthology.orchestra.run.smithi119:> sudo dd of=/home/ubuntu/cephtest/valgrind.supp 2024-04-23T10:10:08.053 DEBUG:teuthology.orchestra.run.smithi204:> set -ex 2024-04-23T10:10:08.053 DEBUG:teuthology.orchestra.run.smithi204:> sudo dd of=/home/ubuntu/cephtest/valgrind.supp 2024-04-23T10:10:08.083 INFO:teuthology.task.install.util:Shipping 'daemon-helper'... 2024-04-23T10:10:08.083 DEBUG:teuthology.orchestra.run.smithi114:> set -ex 2024-04-23T10:10:08.083 DEBUG:teuthology.orchestra.run.smithi114:> sudo dd of=/usr/bin/daemon-helper 2024-04-23T10:10:08.106 DEBUG:teuthology.orchestra.run.smithi114:> sudo chmod a=rx -- /usr/bin/daemon-helper 2024-04-23T10:10:08.169 DEBUG:teuthology.orchestra.run.smithi119:> set -ex 2024-04-23T10:10:08.170 DEBUG:teuthology.orchestra.run.smithi119:> sudo dd of=/usr/bin/daemon-helper 2024-04-23T10:10:08.198 DEBUG:teuthology.orchestra.run.smithi119:> sudo chmod a=rx -- /usr/bin/daemon-helper 2024-04-23T10:10:08.263 DEBUG:teuthology.orchestra.run.smithi204:> set -ex 2024-04-23T10:10:08.263 DEBUG:teuthology.orchestra.run.smithi204:> sudo dd of=/usr/bin/daemon-helper 2024-04-23T10:10:08.291 DEBUG:teuthology.orchestra.run.smithi204:> sudo chmod a=rx -- /usr/bin/daemon-helper 2024-04-23T10:10:08.357 INFO:teuthology.task.install.util:Shipping 'adjust-ulimits'... 2024-04-23T10:10:08.358 DEBUG:teuthology.orchestra.run.smithi114:> set -ex 2024-04-23T10:10:08.358 DEBUG:teuthology.orchestra.run.smithi114:> sudo dd of=/usr/bin/adjust-ulimits 2024-04-23T10:10:08.381 DEBUG:teuthology.orchestra.run.smithi114:> sudo chmod a=rx -- /usr/bin/adjust-ulimits 2024-04-23T10:10:08.447 DEBUG:teuthology.orchestra.run.smithi119:> set -ex 2024-04-23T10:10:08.447 DEBUG:teuthology.orchestra.run.smithi119:> sudo dd of=/usr/bin/adjust-ulimits 2024-04-23T10:10:08.475 DEBUG:teuthology.orchestra.run.smithi119:> sudo chmod a=rx -- /usr/bin/adjust-ulimits 2024-04-23T10:10:08.540 DEBUG:teuthology.orchestra.run.smithi204:> set -ex 2024-04-23T10:10:08.540 DEBUG:teuthology.orchestra.run.smithi204:> sudo dd of=/usr/bin/adjust-ulimits 2024-04-23T10:10:08.568 DEBUG:teuthology.orchestra.run.smithi204:> sudo chmod a=rx -- /usr/bin/adjust-ulimits 2024-04-23T10:10:08.636 INFO:teuthology.task.install.util:Shipping 'stdin-killer'... 2024-04-23T10:10:08.636 DEBUG:teuthology.orchestra.run.smithi114:> set -ex 2024-04-23T10:10:08.636 DEBUG:teuthology.orchestra.run.smithi114:> sudo dd of=/usr/bin/stdin-killer 2024-04-23T10:10:08.659 DEBUG:teuthology.orchestra.run.smithi114:> sudo chmod a=rx -- /usr/bin/stdin-killer 2024-04-23T10:10:08.723 DEBUG:teuthology.orchestra.run.smithi119:> set -ex 2024-04-23T10:10:08.723 DEBUG:teuthology.orchestra.run.smithi119:> sudo dd of=/usr/bin/stdin-killer 2024-04-23T10:10:08.750 DEBUG:teuthology.orchestra.run.smithi119:> sudo chmod a=rx -- /usr/bin/stdin-killer 2024-04-23T10:10:08.816 DEBUG:teuthology.orchestra.run.smithi204:> set -ex 2024-04-23T10:10:08.816 DEBUG:teuthology.orchestra.run.smithi204:> sudo dd of=/usr/bin/stdin-killer 2024-04-23T10:10:08.844 DEBUG:teuthology.orchestra.run.smithi204:> sudo chmod a=rx -- /usr/bin/stdin-killer 2024-04-23T10:10:08.911 INFO:teuthology.run_tasks:Running task cephadm... 2024-04-23T10:10:08.997 INFO:tasks.cephadm:Config: {'roleless': False, 'cephfs': {'max_mds': 3, 'session_timeout': 300, 'standby_replay': True}, 'conf': {'client': {'client mount timeout': 600, 'debug client': 20, 'debug ms': 1, 'rados mon op timeout': 900, 'rados osd op timeout': 900}, 'global': {'ms die on skipped message': False}, 'mds': {'debug mds': 20, 'debug mds balancer': 20, 'debug ms': 1, 'mds bal fragment size max': 10000, 'mds bal merge size': 5, 'mds bal split bits': 3, 'mds bal split size': 100, 'mds debug frag': True, 'mds debug scatterstat': True, 'mds op complaint time': 180, 'mds verify scatter': True, 'mds_bal_replicate_threshold': 1, 'osd op complaint time': 180, 'rados mon op timeout': 900, 'rados osd op timeout': 900}, 'mgr': {'cephsqlite lock renewal timeout': 900000, 'debug cephsqlite': 20, 'debug client': 20, 'debug mgr': 20, 'debug ms': 1}, 'mon': {'debug mon': 20, 'debug ms': 1, 'debug paxos': 20, 'mon down mkfs grace': 300, 'mon op complaint time': 120}, 'osd': {'bdev async discard': True, 'bdev enable discard': True, 'bluestore allocator': 'bitmap', 'bluestore block size': 96636764160, 'bluestore fsck on mount': True, 'debug bluefs': '1/20', 'debug bluestore': '1/20', 'debug ms': 1, 'debug osd': 20, 'debug rocksdb': '4/10', 'mon osd backfillfull_ratio': 0.85, 'mon osd full ratio': 0.9, 'mon osd nearfull ratio': 0.8, 'osd failsafe full ratio': 0.95, 'osd objectstore': 'bluestore', 'osd op complaint time': 180, 'osd shutdown pgref assert': True, 'osd_max_omap_entries_per_request': 10000}}, 'flavor': 'default', 'fs': 'xfs', 'log-ignorelist': ['\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)', 'FS_DEGRADED', 'FS_INLINE_DATA_DEPRECATED', 'FS_WITH_FAILED_MDS', 'MDS_ALL_DOWN', 'MDS_DAMAGE', 'MDS_DEGRADED', 'MDS_FAILED', 'MDS_INSUFFICIENT_STANDBY', 'MDS_UP_LESS_THAN_MAX', 'filesystem is online with fewer MDS than max_mds', 'POOL_APP_NOT_ENABLED', 'do not have an application enabled', 'overall HEALTH_', 'Replacing daemon', 'deprecated feature inline_data', 'overall HEALTH_', '\\(OSD_DOWN\\)', '\\(OSD_', 'but it is still running', 'is not responding', 'PG_AVAILABILITY', 'PG_DEGRADED', 'Reduced data availability'], 'sha1': 'd69f13d827952bb07bb9b7973bca14f97d764d6f', 'subvols': {'create': 2}} 2024-04-23T10:10:08.997 INFO:tasks.cephadm:Cluster image is quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d69f13d827952bb07bb9b7973bca14f97d764d6f 2024-04-23T10:10:08.998 INFO:tasks.cephadm:Cluster fsid is aa240572-0159-11ef-bc93-c7b262605968 2024-04-23T10:10:08.998 INFO:tasks.cephadm:Choosing monitor IPs and ports... 2024-04-23T10:10:08.998 INFO:tasks.cephadm:Monitor IPs: {'mon.a': '172.21.15.114', 'mon.b': '172.21.15.119', 'mon.c': '172.21.15.204'} 2024-04-23T10:10:08.998 INFO:tasks.cephadm:First mon is mon.a on smithi114 2024-04-23T10:10:08.998 INFO:tasks.cephadm:First mgr is x 2024-04-23T10:10:08.998 INFO:tasks.cephadm:Normalizing hostnames... 2024-04-23T10:10:08.998 DEBUG:teuthology.orchestra.run.smithi114:> sudo hostname $(hostname -s) 2024-04-23T10:10:09.023 DEBUG:teuthology.orchestra.run.smithi119:> sudo hostname $(hostname -s) 2024-04-23T10:10:09.050 DEBUG:teuthology.orchestra.run.smithi204:> sudo hostname $(hostname -s) 2024-04-23T10:10:09.079 INFO:tasks.cephadm:Downloading "compiled" cephadm from cachra 2024-04-23T10:10:09.079 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&sha1=d69f13d827952bb07bb9b7973bca14f97d764d6f 2024-04-23T10:10:09.296 INFO:tasks.cephadm:builder_project result: [{'status': 'ready', 'sha1': 'd69f13d827952bb07bb9b7973bca14f97d764d6f', 'extra': {'build_url': 'https://jenkins.ceph.com/job/ceph-dev-new-build/ARCH=x86_64,AVAILABLE_ARCH=x86_64,AVAILABLE_DIST=centos9,DIST=centos9,MACHINE_SIZE=gigantic/78795/', 'root_build_cause': 'SCMTRIGGER', 'version': '19.0.0-3156-gd69f13d8', 'node_name': '172.21.5.39+adami09', 'job_name': 'ceph-dev-new-build/ARCH=x86_64,AVAILABLE_ARCH=x86_64,AVAILABLE_DIST=centos9,DIST=centos9,MACHINE_SIZE=gigantic', 'package_manager_version': '19.0.0-3156.gd69f13d8'}, 'url': 'https://2.chacra.ceph.com/r/ceph/wip-lusov-quiescer/d69f13d827952bb07bb9b7973bca14f97d764d6f/centos/9/flavors/default/', 'distro_codename': None, 'modified': '2024-04-22 12:00:06.590653', 'distro_version': '9', 'project': 'ceph', 'flavor': 'default', 'ref': 'wip-lusov-quiescer', 'chacra_url': 'https://2.chacra.ceph.com/repos/ceph/wip-lusov-quiescer/d69f13d827952bb07bb9b7973bca14f97d764d6f/centos/9/flavors/default/', 'archs': ['source', 'x86_64'], 'distro': 'centos'}] 2024-04-23T10:10:09.431 INFO:tasks.util.chacra:got chacra host 2.chacra.ceph.com, ref wip-lusov-quiescer, sha1 d69f13d827952bb07bb9b7973bca14f97d764d6f from https://shaman.ceph.com/api/search/?project=ceph&distros=centos%2F9%2Fx86_64&flavor=default&sha1=d69f13d827952bb07bb9b7973bca14f97d764d6f 2024-04-23T10:10:09.432 INFO:tasks.cephadm:Discovered cachra url: https://2.chacra.ceph.com/binaries/ceph/wip-lusov-quiescer/d69f13d827952bb07bb9b7973bca14f97d764d6f/centos/9/x86_64/flavors/default/cephadm 2024-04-23T10:10:09.433 DEBUG:teuthology.orchestra.run.smithi114:> curl --silent -L https://2.chacra.ceph.com/binaries/ceph/wip-lusov-quiescer/d69f13d827952bb07bb9b7973bca14f97d764d6f/centos/9/x86_64/flavors/default/cephadm > /home/ubuntu/cephtest/cephadm && ls -l /home/ubuntu/cephtest/cephadm 2024-04-23T10:10:09.925 INFO:teuthology.orchestra.run.smithi114.stdout:-rw-r--r--. 1 ubuntu ubuntu 781541 Apr 23 10:10 /home/ubuntu/cephtest/cephadm 2024-04-23T10:10:09.925 DEBUG:teuthology.orchestra.run.smithi119:> curl --silent -L https://2.chacra.ceph.com/binaries/ceph/wip-lusov-quiescer/d69f13d827952bb07bb9b7973bca14f97d764d6f/centos/9/x86_64/flavors/default/cephadm > /home/ubuntu/cephtest/cephadm && ls -l /home/ubuntu/cephtest/cephadm 2024-04-23T10:10:10.390 INFO:teuthology.orchestra.run.smithi119.stdout:-rw-r--r--. 1 ubuntu ubuntu 781541 Apr 23 10:10 /home/ubuntu/cephtest/cephadm 2024-04-23T10:10:10.391 DEBUG:teuthology.orchestra.run.smithi204:> curl --silent -L https://2.chacra.ceph.com/binaries/ceph/wip-lusov-quiescer/d69f13d827952bb07bb9b7973bca14f97d764d6f/centos/9/x86_64/flavors/default/cephadm > /home/ubuntu/cephtest/cephadm && ls -l /home/ubuntu/cephtest/cephadm 2024-04-23T10:10:11.017 INFO:teuthology.orchestra.run.smithi204.stdout:-rw-r--r--. 1 ubuntu ubuntu 781541 Apr 23 10:10 /home/ubuntu/cephtest/cephadm 2024-04-23T10:10:11.018 DEBUG:teuthology.orchestra.run.smithi114:> test -s /home/ubuntu/cephtest/cephadm && test $(stat -c%s /home/ubuntu/cephtest/cephadm) -gt 1000 && chmod +x /home/ubuntu/cephtest/cephadm 2024-04-23T10:10:11.037 DEBUG:teuthology.orchestra.run.smithi119:> test -s /home/ubuntu/cephtest/cephadm && test $(stat -c%s /home/ubuntu/cephtest/cephadm) -gt 1000 && chmod +x /home/ubuntu/cephtest/cephadm 2024-04-23T10:10:11.056 DEBUG:teuthology.orchestra.run.smithi204:> test -s /home/ubuntu/cephtest/cephadm && test $(stat -c%s /home/ubuntu/cephtest/cephadm) -gt 1000 && chmod +x /home/ubuntu/cephtest/cephadm 2024-04-23T10:10:11.088 INFO:tasks.cephadm:Pulling image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d69f13d827952bb07bb9b7973bca14f97d764d6f on all hosts... 2024-04-23T10:10:11.088 DEBUG:teuthology.orchestra.run.smithi114:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d69f13d827952bb07bb9b7973bca14f97d764d6f pull 2024-04-23T10:10:11.090 DEBUG:teuthology.orchestra.run.smithi119:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d69f13d827952bb07bb9b7973bca14f97d764d6f pull 2024-04-23T10:10:11.099 DEBUG:teuthology.orchestra.run.smithi204:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d69f13d827952bb07bb9b7973bca14f97d764d6f pull 2024-04-23T10:10:12.713 INFO:teuthology.orchestra.run.smithi119.stderr:Pulling container image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d69f13d827952bb07bb9b7973bca14f97d764d6f... 2024-04-23T10:10:12.764 INFO:teuthology.orchestra.run.smithi114.stderr:Pulling container image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d69f13d827952bb07bb9b7973bca14f97d764d6f... 2024-04-23T10:10:12.833 INFO:teuthology.orchestra.run.smithi204.stderr:Pulling container image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d69f13d827952bb07bb9b7973bca14f97d764d6f... 2024-04-23T10:15:22.293 INFO:teuthology.orchestra.run.smithi119.stdout:{ 2024-04-23T10:15:22.293 INFO:teuthology.orchestra.run.smithi119.stdout: "ceph_version": "ceph version 19.0.0-3156-gd69f13d8 (d69f13d827952bb07bb9b7973bca14f97d764d6f) squid (dev)", 2024-04-23T10:15:22.293 INFO:teuthology.orchestra.run.smithi119.stdout: "image_id": "da4b004c09e820713483d039bc76fbd660be678f2ea3c03e059a6161b74dab76", 2024-04-23T10:15:22.293 INFO:teuthology.orchestra.run.smithi119.stdout: "repo_digests": [ 2024-04-23T10:15:22.294 INFO:teuthology.orchestra.run.smithi119.stdout: "quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:068a558e3af4932cf1c60dd321c0bdf500075eb52ea1bf455e0973ba114fbfd4" 2024-04-23T10:15:22.294 INFO:teuthology.orchestra.run.smithi119.stdout: ] 2024-04-23T10:15:22.294 INFO:teuthology.orchestra.run.smithi119.stdout:} 2024-04-23T10:15:26.639 INFO:teuthology.orchestra.run.smithi114.stdout:{ 2024-04-23T10:15:26.640 INFO:teuthology.orchestra.run.smithi114.stdout: "ceph_version": "ceph version 19.0.0-3156-gd69f13d8 (d69f13d827952bb07bb9b7973bca14f97d764d6f) squid (dev)", 2024-04-23T10:15:26.640 INFO:teuthology.orchestra.run.smithi114.stdout: "image_id": "da4b004c09e820713483d039bc76fbd660be678f2ea3c03e059a6161b74dab76", 2024-04-23T10:15:26.640 INFO:teuthology.orchestra.run.smithi114.stdout: "repo_digests": [ 2024-04-23T10:15:26.640 INFO:teuthology.orchestra.run.smithi114.stdout: "quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:068a558e3af4932cf1c60dd321c0bdf500075eb52ea1bf455e0973ba114fbfd4" 2024-04-23T10:15:26.640 INFO:teuthology.orchestra.run.smithi114.stdout: ] 2024-04-23T10:15:26.640 INFO:teuthology.orchestra.run.smithi114.stdout:} 2024-04-23T10:16:43.925 INFO:teuthology.orchestra.run.smithi204.stdout:{ 2024-04-23T10:16:43.925 INFO:teuthology.orchestra.run.smithi204.stdout: "ceph_version": "ceph version 19.0.0-3156-gd69f13d8 (d69f13d827952bb07bb9b7973bca14f97d764d6f) squid (dev)", 2024-04-23T10:16:43.925 INFO:teuthology.orchestra.run.smithi204.stdout: "image_id": "da4b004c09e820713483d039bc76fbd660be678f2ea3c03e059a6161b74dab76", 2024-04-23T10:16:43.926 INFO:teuthology.orchestra.run.smithi204.stdout: "repo_digests": [ 2024-04-23T10:16:43.926 INFO:teuthology.orchestra.run.smithi204.stdout: "quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:068a558e3af4932cf1c60dd321c0bdf500075eb52ea1bf455e0973ba114fbfd4" 2024-04-23T10:16:43.926 INFO:teuthology.orchestra.run.smithi204.stdout: ] 2024-04-23T10:16:43.926 INFO:teuthology.orchestra.run.smithi204.stdout:} 2024-04-23T10:16:43.950 DEBUG:teuthology.orchestra.run.smithi114:> sudo mkdir -p /etc/ceph 2024-04-23T10:16:43.983 DEBUG:teuthology.orchestra.run.smithi119:> sudo mkdir -p /etc/ceph 2024-04-23T10:16:44.017 DEBUG:teuthology.orchestra.run.smithi204:> sudo mkdir -p /etc/ceph 2024-04-23T10:16:44.045 DEBUG:teuthology.orchestra.run.smithi114:> sudo chmod 777 /etc/ceph 2024-04-23T10:16:44.071 DEBUG:teuthology.orchestra.run.smithi119:> sudo chmod 777 /etc/ceph 2024-04-23T10:16:44.100 DEBUG:teuthology.orchestra.run.smithi204:> sudo chmod 777 /etc/ceph 2024-04-23T10:16:44.123 INFO:tasks.cephadm:Writing seed config... 2024-04-23T10:16:44.125 INFO:tasks.cephadm: override: [client] client mount timeout = 600 2024-04-23T10:16:44.125 INFO:tasks.cephadm: override: [client] debug client = 20 2024-04-23T10:16:44.125 INFO:tasks.cephadm: override: [client] debug ms = 1 2024-04-23T10:16:44.125 INFO:tasks.cephadm: override: [client] rados mon op timeout = 900 2024-04-23T10:16:44.125 INFO:tasks.cephadm: override: [client] rados osd op timeout = 900 2024-04-23T10:16:44.125 INFO:tasks.cephadm: override: [global] ms die on skipped message = False 2024-04-23T10:16:44.125 INFO:tasks.cephadm: override: [mds] debug mds = 20 2024-04-23T10:16:44.125 INFO:tasks.cephadm: override: [mds] debug mds balancer = 20 2024-04-23T10:16:44.125 INFO:tasks.cephadm: override: [mds] debug ms = 1 2024-04-23T10:16:44.125 INFO:tasks.cephadm: override: [mds] mds bal fragment size max = 10000 2024-04-23T10:16:44.125 INFO:tasks.cephadm: override: [mds] mds bal merge size = 5 2024-04-23T10:16:44.126 INFO:tasks.cephadm: override: [mds] mds bal split bits = 3 2024-04-23T10:16:44.126 INFO:tasks.cephadm: override: [mds] mds bal split size = 100 2024-04-23T10:16:44.126 INFO:tasks.cephadm: override: [mds] mds debug frag = True 2024-04-23T10:16:44.126 INFO:tasks.cephadm: override: [mds] mds debug scatterstat = True 2024-04-23T10:16:44.126 INFO:tasks.cephadm: override: [mds] mds op complaint time = 180 2024-04-23T10:16:44.126 INFO:tasks.cephadm: override: [mds] mds verify scatter = True 2024-04-23T10:16:44.126 INFO:tasks.cephadm: override: [mds] mds_bal_replicate_threshold = 1 2024-04-23T10:16:44.126 INFO:tasks.cephadm: override: [mds] osd op complaint time = 180 2024-04-23T10:16:44.126 INFO:tasks.cephadm: override: [mds] rados mon op timeout = 900 2024-04-23T10:16:44.126 INFO:tasks.cephadm: override: [mds] rados osd op timeout = 900 2024-04-23T10:16:44.126 INFO:tasks.cephadm: override: [mgr] cephsqlite lock renewal timeout = 900000 2024-04-23T10:16:44.126 INFO:tasks.cephadm: override: [mgr] debug cephsqlite = 20 2024-04-23T10:16:44.127 INFO:tasks.cephadm: override: [mgr] debug client = 20 2024-04-23T10:16:44.127 INFO:tasks.cephadm: override: [mgr] debug mgr = 20 2024-04-23T10:16:44.127 INFO:tasks.cephadm: override: [mgr] debug ms = 1 2024-04-23T10:16:44.127 INFO:tasks.cephadm: override: [mon] debug mon = 20 2024-04-23T10:16:44.127 INFO:tasks.cephadm: override: [mon] debug ms = 1 2024-04-23T10:16:44.127 INFO:tasks.cephadm: override: [mon] debug paxos = 20 2024-04-23T10:16:44.127 INFO:tasks.cephadm: override: [mon] mon down mkfs grace = 300 2024-04-23T10:16:44.127 INFO:tasks.cephadm: override: [mon] mon op complaint time = 120 2024-04-23T10:16:44.127 INFO:tasks.cephadm: override: [osd] bdev async discard = True 2024-04-23T10:16:44.127 INFO:tasks.cephadm: override: [osd] bdev enable discard = True 2024-04-23T10:16:44.127 INFO:tasks.cephadm: override: [osd] bluestore allocator = bitmap 2024-04-23T10:16:44.127 INFO:tasks.cephadm: override: [osd] bluestore block size = 96636764160 2024-04-23T10:16:44.128 INFO:tasks.cephadm: override: [osd] bluestore fsck on mount = True 2024-04-23T10:16:44.128 INFO:tasks.cephadm: override: [osd] debug bluefs = 1/20 2024-04-23T10:16:44.128 INFO:tasks.cephadm: override: [osd] debug bluestore = 1/20 2024-04-23T10:16:44.128 INFO:tasks.cephadm: override: [osd] debug ms = 1 2024-04-23T10:16:44.128 INFO:tasks.cephadm: override: [osd] debug osd = 20 2024-04-23T10:16:44.128 INFO:tasks.cephadm: override: [osd] debug rocksdb = 4/10 2024-04-23T10:16:44.128 INFO:tasks.cephadm: override: [osd] mon osd backfillfull_ratio = 0.85 2024-04-23T10:16:44.128 INFO:tasks.cephadm: override: [osd] mon osd full ratio = 0.9 2024-04-23T10:16:44.128 INFO:tasks.cephadm: override: [osd] mon osd nearfull ratio = 0.8 2024-04-23T10:16:44.128 INFO:tasks.cephadm: override: [osd] osd failsafe full ratio = 0.95 2024-04-23T10:16:44.129 INFO:tasks.cephadm: override: [osd] osd objectstore = bluestore 2024-04-23T10:16:44.129 INFO:tasks.cephadm: override: [osd] osd op complaint time = 180 2024-04-23T10:16:44.129 INFO:tasks.cephadm: override: [osd] osd shutdown pgref assert = True 2024-04-23T10:16:44.129 INFO:tasks.cephadm: override: [osd] osd_max_omap_entries_per_request = 10000 2024-04-23T10:16:44.129 DEBUG:teuthology.orchestra.run.smithi114:> set -ex 2024-04-23T10:16:44.130 DEBUG:teuthology.orchestra.run.smithi114:> dd of=/home/ubuntu/cephtest/seed.ceph.conf 2024-04-23T10:16:44.146 DEBUG:tasks.cephadm:Final config: [global] # make logging friendly to teuthology log_to_file = true log_to_stderr = false log to journald = false mon cluster log to file = true mon cluster log file level = debug mon clock drift allowed = 1.000 # replicate across OSDs, not hosts osd crush chooseleaf type = 0 #osd pool default size = 2 osd pool default erasure code profile = plugin=jerasure technique=reed_sol_van k=2 m=1 crush-failure-domain=osd # enable some debugging auth debug = true ms die on old message = true ms die on bug = true debug asserts on shutdown = true # adjust warnings mon max pg per osd = 10000# >= luminous mon pg warn max object skew = 0 mon osd allow primary affinity = true mon osd allow pg remap = true mon warn on legacy crush tunables = false mon warn on crush straw calc version zero = false mon warn on no sortbitwise = false mon warn on osd down out interval zero = false mon warn on too few osds = false mon_warn_on_pool_pg_num_not_power_of_two = false # disable pg_autoscaler by default for new pools osd_pool_default_pg_autoscale_mode = off # tests delete pools mon allow pool delete = true fsid = aa240572-0159-11ef-bc93-c7b262605968 ms die on skipped message = False [osd] osd scrub load threshold = 5.0 osd scrub max interval = 600 osd mclock profile = high_recovery_ops osd recover clone overlap = true osd recovery max chunk = 1048576 osd deep scrub update digest min age = 30 osd map max advance = 10 osd memory target autotune = true # debugging osd debug shutdown = true osd debug op order = true osd debug verify stray on activate = true osd debug pg log writeout = true osd debug verify cached snaps = true osd debug verify missing on start = true osd debug misdirected ops = true osd op queue = debug_random osd op queue cut off = debug_random osd shutdown pgref assert = True bdev debug aio = true osd sloppy crc = true bdev async discard = True bdev enable discard = True bluestore allocator = bitmap bluestore block size = 96636764160 bluestore fsck on mount = True debug bluefs = 1/20 debug bluestore = 1/20 debug ms = 1 debug osd = 20 debug rocksdb = 4/10 mon osd backfillfull_ratio = 0.85 mon osd full ratio = 0.9 mon osd nearfull ratio = 0.8 osd failsafe full ratio = 0.95 osd objectstore = bluestore osd op complaint time = 180 osd_max_omap_entries_per_request = 10000 [mgr] mon reweight min pgs per osd = 4 mon reweight min bytes per osd = 10 mgr/telemetry/nag = false cephsqlite lock renewal timeout = 900000 debug cephsqlite = 20 debug client = 20 debug mgr = 20 debug ms = 1 [mon] mon data avail warn = 5 mon mgr mkfs grace = 240 mon reweight min pgs per osd = 4 mon osd reporter subtree level = osd mon osd prime pg temp = true mon reweight min bytes per osd = 10 # rotate auth tickets quickly to exercise renewal paths auth mon ticket ttl = 660# 11m auth service ticket ttl = 240# 4m # don't complain about global id reclaim mon_warn_on_insecure_global_id_reclaim = false mon_warn_on_insecure_global_id_reclaim_allowed = false debug mon = 20 debug ms = 1 debug paxos = 20 mon down mkfs grace = 300 mon op complaint time = 120 [client.rgw] rgw cache enabled = true rgw enable ops log = true rgw enable usage log = true [client] client mount timeout = 600 debug client = 20 debug ms = 1 rados mon op timeout = 900 rados osd op timeout = 900 [mds] debug mds = 20 debug mds balancer = 20 debug ms = 1 mds bal fragment size max = 10000 mds bal merge size = 5 mds bal split bits = 3 mds bal split size = 100 mds debug frag = True mds debug scatterstat = True mds op complaint time = 180 mds verify scatter = True mds_bal_replicate_threshold = 1 osd op complaint time = 180 rados mon op timeout = 900 rados osd op timeout = 900 2024-04-23T10:16:44.147 DEBUG:teuthology.orchestra.run.smithi114:mon.a> sudo journalctl -f -n 0 -u ceph-aa240572-0159-11ef-bc93-c7b262605968@mon.a.service 2024-04-23T10:16:44.190 DEBUG:teuthology.orchestra.run.smithi114:mgr.x> sudo journalctl -f -n 0 -u ceph-aa240572-0159-11ef-bc93-c7b262605968@mgr.x.service 2024-04-23T10:16:44.233 INFO:tasks.cephadm:Bootstrapping... 2024-04-23T10:16:44.233 DEBUG:teuthology.orchestra.run.smithi114:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d69f13d827952bb07bb9b7973bca14f97d764d6f -v bootstrap --fsid aa240572-0159-11ef-bc93-c7b262605968 --config /home/ubuntu/cephtest/seed.ceph.conf --output-config /etc/ceph/ceph.conf --output-keyring /etc/ceph/ceph.client.admin.keyring --output-pub-ssh-key /home/ubuntu/cephtest/ceph.pub --mon-id a --mgr-id x --orphan-initial-daemons --skip-monitoring-stack --mon-ip 172.21.15.114 --skip-admin-label && sudo chmod +r /etc/ceph/ceph.client.admin.keyring 2024-04-23T10:16:44.424 INFO:teuthology.orchestra.run.smithi114.stdout:-------------------------------------------------------------------------------- 2024-04-23T10:16:44.425 INFO:teuthology.orchestra.run.smithi114.stdout:cephadm ['--image', 'quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d69f13d827952bb07bb9b7973bca14f97d764d6f', '-v', 'bootstrap', '--fsid', 'aa240572-0159-11ef-bc93-c7b262605968', '--config', '/home/ubuntu/cephtest/seed.ceph.conf', '--output-config', '/etc/ceph/ceph.conf', '--output-keyring', '/etc/ceph/ceph.client.admin.keyring', '--output-pub-ssh-key', '/home/ubuntu/cephtest/ceph.pub', '--mon-id', 'a', '--mgr-id', 'x', '--orphan-initial-daemons', '--skip-monitoring-stack', '--mon-ip', '172.21.15.114', '--skip-admin-label'] 2024-04-23T10:16:44.425 INFO:teuthology.orchestra.run.smithi114.stderr:Specifying an fsid for your cluster offers no advantages and may increase the likelihood of fsid conflicts. 2024-04-23T10:16:44.425 INFO:teuthology.orchestra.run.smithi114.stdout:Verifying podman|docker is present... 2024-04-23T10:16:44.451 INFO:teuthology.orchestra.run.smithi114.stdout:/bin/podman: stdout 4.9.4-dev 2024-04-23T10:16:44.452 INFO:teuthology.orchestra.run.smithi114.stdout:Verifying lvm2 is present... 2024-04-23T10:16:44.452 INFO:teuthology.orchestra.run.smithi114.stdout:Verifying time synchronization is in place... 2024-04-23T10:16:44.459 INFO:teuthology.orchestra.run.smithi114.stdout:Non-zero exit code 1 from systemctl is-enabled chrony.service 2024-04-23T10:16:44.459 INFO:teuthology.orchestra.run.smithi114.stdout:systemctl: stderr Failed to get unit file state for chrony.service: No such file or directory 2024-04-23T10:16:44.465 INFO:teuthology.orchestra.run.smithi114.stdout:Non-zero exit code 3 from systemctl is-active chrony.service 2024-04-23T10:16:44.465 INFO:teuthology.orchestra.run.smithi114.stdout:systemctl: stdout inactive 2024-04-23T10:16:44.471 INFO:teuthology.orchestra.run.smithi114.stdout:systemctl: stdout enabled 2024-04-23T10:16:44.477 INFO:teuthology.orchestra.run.smithi114.stdout:systemctl: stdout active 2024-04-23T10:16:44.478 INFO:teuthology.orchestra.run.smithi114.stdout:Unit chronyd.service is enabled and running 2024-04-23T10:16:44.478 INFO:teuthology.orchestra.run.smithi114.stdout:Repeating the final host check... 2024-04-23T10:16:44.512 INFO:teuthology.orchestra.run.smithi114.stdout:/bin/podman: stdout 4.9.4-dev 2024-04-23T10:16:44.512 INFO:teuthology.orchestra.run.smithi114.stdout:podman (/bin/podman) version 4.9.4 is present 2024-04-23T10:16:44.512 INFO:teuthology.orchestra.run.smithi114.stdout:systemctl is present 2024-04-23T10:16:44.512 INFO:teuthology.orchestra.run.smithi114.stdout:lvcreate is present 2024-04-23T10:16:44.518 INFO:teuthology.orchestra.run.smithi114.stdout:Non-zero exit code 1 from systemctl is-enabled chrony.service 2024-04-23T10:16:44.519 INFO:teuthology.orchestra.run.smithi114.stdout:systemctl: stderr Failed to get unit file state for chrony.service: No such file or directory 2024-04-23T10:16:44.524 INFO:teuthology.orchestra.run.smithi114.stdout:Non-zero exit code 3 from systemctl is-active chrony.service 2024-04-23T10:16:44.524 INFO:teuthology.orchestra.run.smithi114.stdout:systemctl: stdout inactive 2024-04-23T10:16:44.530 INFO:teuthology.orchestra.run.smithi114.stdout:systemctl: stdout enabled 2024-04-23T10:16:44.537 INFO:teuthology.orchestra.run.smithi114.stdout:systemctl: stdout active 2024-04-23T10:16:44.537 INFO:teuthology.orchestra.run.smithi114.stdout:Unit chronyd.service is enabled and running 2024-04-23T10:16:44.537 INFO:teuthology.orchestra.run.smithi114.stdout:Host looks OK 2024-04-23T10:16:44.538 INFO:teuthology.orchestra.run.smithi114.stdout:Cluster fsid: aa240572-0159-11ef-bc93-c7b262605968 2024-04-23T10:16:44.538 INFO:teuthology.orchestra.run.smithi114.stdout:Acquiring lock 139969211539408 on /run/cephadm/aa240572-0159-11ef-bc93-c7b262605968.lock 2024-04-23T10:16:44.538 INFO:teuthology.orchestra.run.smithi114.stdout:Lock 139969211539408 acquired on /run/cephadm/aa240572-0159-11ef-bc93-c7b262605968.lock 2024-04-23T10:16:44.538 INFO:teuthology.orchestra.run.smithi114.stdout:Verifying IP 172.21.15.114 port 3300 ... 2024-04-23T10:16:44.538 INFO:teuthology.orchestra.run.smithi114.stdout:Verifying IP 172.21.15.114 port 6789 ... 2024-04-23T10:16:44.538 INFO:teuthology.orchestra.run.smithi114.stdout:Base mon IP(s) is [172.21.15.114:3300, 172.21.15.114:6789], mon addrv is [v2:172.21.15.114:3300,v1:172.21.15.114:6789] 2024-04-23T10:16:44.541 INFO:teuthology.orchestra.run.smithi114.stdout:/sbin/ip: stdout default via 172.21.15.254 dev enp3s0f1 proto dhcp src 172.21.15.114 metric 100 2024-04-23T10:16:44.541 INFO:teuthology.orchestra.run.smithi114.stdout:/sbin/ip: stdout 172.21.0.0/20 dev enp3s0f1 proto kernel scope link src 172.21.15.114 metric 100 2024-04-23T10:16:44.543 INFO:teuthology.orchestra.run.smithi114.stdout:/sbin/ip: stdout ::1 dev lo proto kernel metric 256 pref medium 2024-04-23T10:16:44.544 INFO:teuthology.orchestra.run.smithi114.stdout:/sbin/ip: stdout fe80::/64 dev enp3s0f1 proto kernel metric 1024 pref medium 2024-04-23T10:16:44.544 INFO:teuthology.orchestra.run.smithi114.stdout:/sbin/ip: stdout default via fe80::327c:5e00:6487:71e0 dev enp3s0f1 proto ra metric 100 pref medium 2024-04-23T10:16:44.546 INFO:teuthology.orchestra.run.smithi114.stdout:/sbin/ip: stdout 1: lo: mtu 65536 state UNKNOWN qlen 1000 2024-04-23T10:16:44.546 INFO:teuthology.orchestra.run.smithi114.stdout:/sbin/ip: stdout inet6 ::1/128 scope host 2024-04-23T10:16:44.546 INFO:teuthology.orchestra.run.smithi114.stdout:/sbin/ip: stdout valid_lft forever preferred_lft forever 2024-04-23T10:16:44.546 INFO:teuthology.orchestra.run.smithi114.stdout:/sbin/ip: stdout 5: enp3s0f1: mtu 1500 state UP qlen 1000 2024-04-23T10:16:44.547 INFO:teuthology.orchestra.run.smithi114.stdout:/sbin/ip: stdout inet6 fe80::db0f:356f:35d2:e591/64 scope link noprefixroute 2024-04-23T10:16:44.547 INFO:teuthology.orchestra.run.smithi114.stdout:/sbin/ip: stdout valid_lft forever preferred_lft forever 2024-04-23T10:16:44.547 INFO:teuthology.orchestra.run.smithi114.stdout:Mon IP `172.21.15.114` is in CIDR network `172.21.0.0/20` 2024-04-23T10:16:44.547 INFO:teuthology.orchestra.run.smithi114.stdout:Mon IP `172.21.15.114` is in CIDR network `172.21.0.0/20` 2024-04-23T10:16:44.547 INFO:teuthology.orchestra.run.smithi114.stdout:Inferred mon public CIDR from local network configuration ['172.21.0.0/20', '172.21.0.0/20'] 2024-04-23T10:16:44.548 INFO:teuthology.orchestra.run.smithi114.stdout:Internal network (--cluster-network) has not been provided, OSD replication will default to the public_network 2024-04-23T10:16:44.548 INFO:teuthology.orchestra.run.smithi114.stdout:Pulling container image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d69f13d827952bb07bb9b7973bca14f97d764d6f... 2024-04-23T10:16:45.003 INFO:teuthology.orchestra.run.smithi114.stdout:/bin/podman: stdout da4b004c09e820713483d039bc76fbd660be678f2ea3c03e059a6161b74dab76 2024-04-23T10:16:45.003 INFO:teuthology.orchestra.run.smithi114.stdout:/bin/podman: stderr Trying to pull quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d69f13d827952bb07bb9b7973bca14f97d764d6f... 2024-04-23T10:16:45.003 INFO:teuthology.orchestra.run.smithi114.stdout:/bin/podman: stderr Getting image source signatures 2024-04-23T10:16:45.003 INFO:teuthology.orchestra.run.smithi114.stdout:/bin/podman: stderr Copying blob sha256:7f03586bcf9c6a2b4d690c118c4aa5b771f9c6f252d75a20fb283dd07093a738 2024-04-23T10:16:45.003 INFO:teuthology.orchestra.run.smithi114.stdout:/bin/podman: stderr Copying blob sha256:426fede9fffa2c7dfe40a4f9322a34794b72405170134d54d2c30d9cc6c6a5be 2024-04-23T10:16:45.003 INFO:teuthology.orchestra.run.smithi114.stdout:/bin/podman: stderr Copying config sha256:da4b004c09e820713483d039bc76fbd660be678f2ea3c03e059a6161b74dab76 2024-04-23T10:16:45.003 INFO:teuthology.orchestra.run.smithi114.stdout:/bin/podman: stderr Writing manifest to image destination 2024-04-23T10:16:45.909 INFO:teuthology.orchestra.run.smithi114.stdout:ceph: stdout ceph version 19.0.0-3156-gd69f13d8 (d69f13d827952bb07bb9b7973bca14f97d764d6f) squid (dev) 2024-04-23T10:16:45.910 INFO:teuthology.orchestra.run.smithi114.stdout:Ceph version: ceph version 19.0.0-3156-gd69f13d8 (d69f13d827952bb07bb9b7973bca14f97d764d6f) squid (dev) 2024-04-23T10:16:45.910 INFO:teuthology.orchestra.run.smithi114.stdout:Extracting ceph user uid/gid from container image... 2024-04-23T10:16:46.910 INFO:teuthology.orchestra.run.smithi114.stdout:stat: stdout 167 167 2024-04-23T10:16:46.910 INFO:teuthology.orchestra.run.smithi114.stdout:Creating initial keys... 2024-04-23T10:16:47.918 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph-authtool: stdout AQCPiidmf+FEDhAAMk+WBbIZ1LmNRfLYunzVIQ== 2024-04-23T10:16:48.816 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph-authtool: stdout AQCQiidmg98oDhAAXgivHc0B9mKRsqnDfatlsA== 2024-04-23T10:16:49.954 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph-authtool: stdout AQCRiidmzweXCBAAbFqj4ggvJ5DsZGjG5NC1DA== 2024-04-23T10:16:49.955 INFO:teuthology.orchestra.run.smithi114.stdout:Creating initial monmap... 2024-04-23T10:16:50.951 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/monmaptool: stdout /usr/bin/monmaptool: monmap file /tmp/monmap 2024-04-23T10:16:50.952 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/monmaptool: stdout setting min_mon_release = quincy 2024-04-23T10:16:50.952 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/monmaptool: stdout /usr/bin/monmaptool: set fsid to aa240572-0159-11ef-bc93-c7b262605968 2024-04-23T10:16:50.952 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/monmaptool: stdout /usr/bin/monmaptool: writing epoch 0 to /tmp/monmap (1 monitors) 2024-04-23T10:16:50.952 INFO:teuthology.orchestra.run.smithi114.stdout:monmaptool for a [v2:172.21.15.114:3300,v1:172.21.15.114:6789] on /usr/bin/monmaptool: monmap file /tmp/monmap 2024-04-23T10:16:50.952 INFO:teuthology.orchestra.run.smithi114.stdout:setting min_mon_release = quincy 2024-04-23T10:16:50.952 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/monmaptool: set fsid to aa240572-0159-11ef-bc93-c7b262605968 2024-04-23T10:16:50.952 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/monmaptool: writing epoch 0 to /tmp/monmap (1 monitors) 2024-04-23T10:16:50.952 INFO:teuthology.orchestra.run.smithi114.stdout: 2024-04-23T10:16:50.952 INFO:teuthology.orchestra.run.smithi114.stdout:Creating mon... 2024-04-23T10:16:51.999 INFO:teuthology.orchestra.run.smithi114.stdout:create mon.a on 2024-04-23T10:16:52.182 INFO:teuthology.orchestra.run.smithi114.stdout:systemctl: stderr Removed "/etc/systemd/system/multi-user.target.wants/ceph.target". 2024-04-23T10:16:52.314 INFO:teuthology.orchestra.run.smithi114.stdout:systemctl: stderr Created symlink /etc/systemd/system/multi-user.target.wants/ceph.target → /etc/systemd/system/ceph.target. 2024-04-23T10:16:52.473 INFO:teuthology.orchestra.run.smithi114.stdout:systemctl: stderr Created symlink /etc/systemd/system/multi-user.target.wants/ceph-aa240572-0159-11ef-bc93-c7b262605968.target → /etc/systemd/system/ceph-aa240572-0159-11ef-bc93-c7b262605968.target. 2024-04-23T10:16:52.473 INFO:teuthology.orchestra.run.smithi114.stdout:systemctl: stderr Created symlink /etc/systemd/system/ceph.target.wants/ceph-aa240572-0159-11ef-bc93-c7b262605968.target → /etc/systemd/system/ceph-aa240572-0159-11ef-bc93-c7b262605968.target. 2024-04-23T10:16:52.684 INFO:teuthology.orchestra.run.smithi114.stdout:Non-zero exit code 1 from systemctl reset-failed ceph-aa240572-0159-11ef-bc93-c7b262605968@mon.a 2024-04-23T10:16:52.684 INFO:teuthology.orchestra.run.smithi114.stdout:systemctl: stderr Failed to reset failed state of unit ceph-aa240572-0159-11ef-bc93-c7b262605968@mon.a.service: Unit ceph-aa240572-0159-11ef-bc93-c7b262605968@mon.a.service not loaded. 2024-04-23T10:16:52.848 INFO:teuthology.orchestra.run.smithi114.stdout:systemctl: stderr Created symlink /etc/systemd/system/ceph-aa240572-0159-11ef-bc93-c7b262605968.target.wants/ceph-aa240572-0159-11ef-bc93-c7b262605968@mon.a.service → /etc/systemd/system/ceph-aa240572-0159-11ef-bc93-c7b262605968@.service. 2024-04-23T10:16:53.378 INFO:teuthology.orchestra.run.smithi114.stdout:Non-zero exit code 1 from systemctl is-enabled firewalld.service 2024-04-23T10:16:53.378 INFO:teuthology.orchestra.run.smithi114.stdout:systemctl: stdout disabled 2024-04-23T10:16:53.385 INFO:teuthology.orchestra.run.smithi114.stdout:Non-zero exit code 3 from systemctl is-active firewalld.service 2024-04-23T10:16:53.386 INFO:teuthology.orchestra.run.smithi114.stdout:systemctl: stdout inactive 2024-04-23T10:16:53.386 INFO:teuthology.orchestra.run.smithi114.stdout:firewalld.service is not enabled 2024-04-23T10:16:53.386 INFO:teuthology.orchestra.run.smithi114.stdout:Not possible to enable service . firewalld.service is not available 2024-04-23T10:16:53.386 INFO:teuthology.orchestra.run.smithi114.stdout:Waiting for mon to start... 2024-04-23T10:16:53.386 INFO:teuthology.orchestra.run.smithi114.stdout:Waiting for mon... 2024-04-23T10:16:54.487 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stdout cluster: 2024-04-23T10:16:54.487 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stdout id: aa240572-0159-11ef-bc93-c7b262605968 2024-04-23T10:16:54.487 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stdout health: HEALTH_OK 2024-04-23T10:16:54.487 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stdout 2024-04-23T10:16:54.488 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stdout services: 2024-04-23T10:16:54.488 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stdout mon: 1 daemons, quorum a (age 0.577708s) 2024-04-23T10:16:54.488 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stdout mgr: no daemons active 2024-04-23T10:16:54.488 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stdout osd: 0 osds: 0 up, 0 in 2024-04-23T10:16:54.488 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stdout 2024-04-23T10:16:54.488 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stdout data: 2024-04-23T10:16:54.488 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stdout pools: 0 pools, 0 pgs 2024-04-23T10:16:54.488 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stdout objects: 0 objects, 0 B 2024-04-23T10:16:54.488 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stdout usage: 0 B used, 0 B / 0 B avail 2024-04-23T10:16:54.488 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stdout pgs: 2024-04-23T10:16:54.488 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stdout 2024-04-23T10:16:54.488 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-04-23T10:16:53.843+0000 7fb7fcc12700 1 Processor -- start 2024-04-23T10:16:54.488 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-04-23T10:16:53.843+0000 7fb7fcc12700 1 -- start start 2024-04-23T10:16:54.488 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-04-23T10:16:53.843+0000 7fb7fcc12700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fb7f8109000 0x7fb7f810b3f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T10:16:54.488 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-04-23T10:16:53.843+0000 7fb7fcc12700 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x7fb7f807be80 con 0x7fb7f8109000 2024-04-23T10:16:54.489 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-04-23T10:16:53.844+0000 7fb7f659c700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fb7f8109000 0x7fb7f810b3f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T10:16:54.489 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-04-23T10:16:53.844+0000 7fb7f659c700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fb7f8109000 0x7fb7f810b3f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.114:40348/0 (socket says 172.21.15.114:40348) 2024-04-23T10:16:54.489 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-04-23T10:16:53.844+0000 7fb7f659c700 1 -- 172.21.15.114:0/745481626 learned_addr learned my addr 172.21.15.114:0/745481626 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-04-23T10:16:54.489 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-04-23T10:16:53.844+0000 7fb7f659c700 1 -- 172.21.15.114:0/745481626 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb7f810b930 con 0x7fb7f8109000 2024-04-23T10:16:54.489 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-04-23T10:16:53.844+0000 7fb7f659c700 1 --2- 172.21.15.114:0/745481626 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fb7f8109000 0x7fb7f810b3f0 secure :-1 s=READY pgs=1 cs=0 l=1 rev1=1 crypto rx=0x7fb7e80098a0 tx=0x7fb7e8009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=830d1062912b75f1 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T10:16:54.489 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-04-23T10:16:53.844+0000 7fb7f559a700 1 -- 172.21.15.114:0/745481626 <== mon.0 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fb7e8004160 con 0x7fb7f8109000 2024-04-23T10:16:54.489 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-04-23T10:16:53.844+0000 7fb7f559a700 1 -- 172.21.15.114:0/745481626 <== mon.0 v2:172.21.15.114:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fb7e8004300 con 0x7fb7f8109000 2024-04-23T10:16:54.489 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-04-23T10:16:53.844+0000 7fb7f559a700 1 -- 172.21.15.114:0/745481626 <== mon.0 v2:172.21.15.114:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fb7e80036a0 con 0x7fb7f8109000 2024-04-23T10:16:54.489 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-04-23T10:16:53.845+0000 7fb7fcc12700 1 -- 172.21.15.114:0/745481626 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fb7f8109000 msgr2=0x7fb7f810b3f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T10:16:54.489 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-04-23T10:16:53.845+0000 7fb7fcc12700 1 --2- 172.21.15.114:0/745481626 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fb7f8109000 0x7fb7f810b3f0 secure :-1 s=READY pgs=1 cs=0 l=1 rev1=1 crypto rx=0x7fb7e80098a0 tx=0x7fb7e8009bb0 comp rx=0 tx=0).stop 2024-04-23T10:16:54.489 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-04-23T10:16:53.845+0000 7fb7fcc12700 1 -- 172.21.15.114:0/745481626 shutdown_connections 2024-04-23T10:16:54.489 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-04-23T10:16:53.845+0000 7fb7fcc12700 1 --2- 172.21.15.114:0/745481626 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fb7f8109000 0x7fb7f810b3f0 unknown :-1 s=CLOSED pgs=1 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:16:54.489 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-04-23T10:16:53.845+0000 7fb7fcc12700 1 -- 172.21.15.114:0/745481626 >> 172.21.15.114:0/745481626 conn(0x7fb7f8102a50 msgr2=0x7fb7f8104e70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T10:16:54.489 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-04-23T10:16:53.845+0000 7fb7fcc12700 1 -- 172.21.15.114:0/745481626 shutdown_connections 2024-04-23T10:16:54.489 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-04-23T10:16:53.845+0000 7fb7fcc12700 1 -- 172.21.15.114:0/745481626 wait complete. 2024-04-23T10:16:54.490 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-04-23T10:16:53.846+0000 7fb7fcc12700 1 Processor -- start 2024-04-23T10:16:54.490 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-04-23T10:16:53.846+0000 7fb7fcc12700 1 -- start start 2024-04-23T10:16:54.490 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-04-23T10:16:53.846+0000 7fb7fcc12700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fb7f8109000 0x7fb7f8108880 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T10:16:54.490 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-04-23T10:16:53.846+0000 7fb7fcc12700 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x7fb7f810bda0 con 0x7fb7f8109000 2024-04-23T10:16:54.490 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-04-23T10:16:53.846+0000 7fb7f659c700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fb7f8109000 0x7fb7f8108880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T10:16:54.490 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-04-23T10:16:53.846+0000 7fb7f659c700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fb7f8109000 0x7fb7f8108880 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.114:40350/0 (socket says 172.21.15.114:40350) 2024-04-23T10:16:54.490 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-04-23T10:16:53.846+0000 7fb7f659c700 1 -- 172.21.15.114:0/1391900207 learned_addr learned my addr 172.21.15.114:0/1391900207 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-04-23T10:16:54.490 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-04-23T10:16:53.846+0000 7fb7f659c700 1 -- 172.21.15.114:0/1391900207 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb7f8108dc0 con 0x7fb7f8109000 2024-04-23T10:16:54.490 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-04-23T10:16:53.847+0000 7fb7f659c700 1 --2- 172.21.15.114:0/1391900207 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fb7f8109000 0x7fb7f8108880 secure :-1 s=READY pgs=2 cs=0 l=1 rev1=1 crypto rx=0x7fb7e8009fb0 tx=0x7fb7e8004750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T10:16:54.490 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-04-23T10:16:53.847+0000 7fb7e77fe700 1 -- 172.21.15.114:0/1391900207 <== mon.0 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fb7e8003de0 con 0x7fb7f8109000 2024-04-23T10:16:54.490 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-04-23T10:16:53.847+0000 7fb7e77fe700 1 -- 172.21.15.114:0/1391900207 <== mon.0 v2:172.21.15.114:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fb7e8011630 con 0x7fb7f8109000 2024-04-23T10:16:54.490 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-04-23T10:16:53.847+0000 7fb7e77fe700 1 -- 172.21.15.114:0/1391900207 <== mon.0 v2:172.21.15.114:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fb7e8011930 con 0x7fb7f8109000 2024-04-23T10:16:54.490 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-04-23T10:16:53.847+0000 7fb7fcc12700 1 -- 172.21.15.114:0/1391900207 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fb7f8106fb0 con 0x7fb7f8109000 2024-04-23T10:16:54.490 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-04-23T10:16:53.847+0000 7fb7fcc12700 1 -- 172.21.15.114:0/1391900207 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fb7f81074f0 con 0x7fb7f8109000 2024-04-23T10:16:54.491 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-04-23T10:16:53.847+0000 7fb7e77fe700 1 -- 172.21.15.114:0/1391900207 <== mon.0 v2:172.21.15.114:3300/0 4 ==== mgrmap(e 1) ==== 807+0+0 (secure 0 0 0) 0x7fb7e8024480 con 0x7fb7f8109000 2024-04-23T10:16:54.491 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-04-23T10:16:53.847+0000 7fb7e77fe700 1 -- 172.21.15.114:0/1391900207 <== mon.0 v2:172.21.15.114:3300/0 5 ==== osd_map(1..1 src has 1..1) ==== 725+0+0 (secure 0 0 0) 0x7fb7e801a7e0 con 0x7fb7f8109000 2024-04-23T10:16:54.491 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-04-23T10:16:53.848+0000 7fb7fcc12700 1 -- 172.21.15.114:0/1391900207 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fb7f80640a0 con 0x7fb7f8109000 2024-04-23T10:16:54.491 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-04-23T10:16:53.852+0000 7fb7e77fe700 1 -- 172.21.15.114:0/1391900207 <== mon.0 v2:172.21.15.114:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+73265 (secure 0 0 0) 0x7fb7e801aa80 con 0x7fb7f8109000 2024-04-23T10:16:54.491 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-04-23T10:16:53.926+0000 7fb7fcc12700 1 -- 172.21.15.114:0/1391900207 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_command({"prefix": "status"} v 0) -- 0x7fb7f8195610 con 0x7fb7f8109000 2024-04-23T10:16:54.491 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-04-23T10:16:53.927+0000 7fb7e77fe700 1 -- 172.21.15.114:0/1391900207 <== mon.0 v2:172.21.15.114:3300/0 7 ==== mon_command_ack([{"prefix": "status"}]=0 v0) ==== 54+0+317 (secure 0 0 0) 0x7fb7e801ac60 con 0x7fb7f8109000 2024-04-23T10:16:54.491 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-04-23T10:16:53.928+0000 7fb7fcc12700 1 -- 172.21.15.114:0/1391900207 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fb7f8109000 msgr2=0x7fb7f8108880 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T10:16:54.491 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-04-23T10:16:53.928+0000 7fb7fcc12700 1 --2- 172.21.15.114:0/1391900207 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fb7f8109000 0x7fb7f8108880 secure :-1 s=READY pgs=2 cs=0 l=1 rev1=1 crypto rx=0x7fb7e8009fb0 tx=0x7fb7e8004750 comp rx=0 tx=0).stop 2024-04-23T10:16:54.491 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-04-23T10:16:53.928+0000 7fb7fcc12700 1 -- 172.21.15.114:0/1391900207 shutdown_connections 2024-04-23T10:16:54.491 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-04-23T10:16:53.928+0000 7fb7fcc12700 1 --2- 172.21.15.114:0/1391900207 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fb7f8109000 0x7fb7f8108880 unknown :-1 s=CLOSED pgs=2 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:16:54.491 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-04-23T10:16:53.928+0000 7fb7fcc12700 1 -- 172.21.15.114:0/1391900207 >> 172.21.15.114:0/1391900207 conn(0x7fb7f8102a50 msgr2=0x7fb7f8103980 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T10:16:54.491 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-04-23T10:16:53.928+0000 7fb7fcc12700 1 -- 172.21.15.114:0/1391900207 shutdown_connections 2024-04-23T10:16:54.491 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-04-23T10:16:53.928+0000 7fb7fcc12700 1 -- 172.21.15.114:0/1391900207 wait complete. 2024-04-23T10:16:54.491 INFO:teuthology.orchestra.run.smithi114.stdout:mon is available 2024-04-23T10:16:54.492 INFO:teuthology.orchestra.run.smithi114.stdout:Assimilating anything we can from ceph.conf... 2024-04-23T10:16:55.531 INFO:journalctl@ceph.mon.a.smithi114.stdout:Apr 23 10:16:55 smithi114 ceph-mon[28875]: from='client.? 172.21.15.114:0/547765683' entity='client.admin' cmd='[{"prefix": "config assimilate-conf"}]': finished 2024-04-23T10:16:55.539 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stdout 2024-04-23T10:16:55.539 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stdout [global] 2024-04-23T10:16:55.539 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stdout fsid = aa240572-0159-11ef-bc93-c7b262605968 2024-04-23T10:16:55.540 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stdout mon_cluster_log_file_level = debug 2024-04-23T10:16:55.540 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stdout mon_host = [v2:172.21.15.114:3300,v1:172.21.15.114:6789] 2024-04-23T10:16:55.540 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stdout mon_osd_allow_pg_remap = true 2024-04-23T10:16:55.540 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stdout mon_osd_allow_primary_affinity = true 2024-04-23T10:16:55.540 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stdout mon_warn_on_no_sortbitwise = false 2024-04-23T10:16:55.540 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stdout osd_crush_chooseleaf_type = 0 2024-04-23T10:16:55.540 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stdout 2024-04-23T10:16:55.540 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stdout [mgr] 2024-04-23T10:16:55.540 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stdout mgr/telemetry/nag = false 2024-04-23T10:16:55.540 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stdout 2024-04-23T10:16:55.540 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stdout [osd] 2024-04-23T10:16:55.540 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stdout bdev_async_discard = True 2024-04-23T10:16:55.540 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stdout mon_osd_backfillfull_ratio = 0.85 2024-04-23T10:16:55.540 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stdout mon_osd_full_ratio = 0.9 2024-04-23T10:16:55.540 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stdout mon_osd_nearfull_ratio = 0.8 2024-04-23T10:16:55.540 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stdout osd_map_max_advance = 10 2024-04-23T10:16:55.541 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stdout osd_sloppy_crc = true 2024-04-23T10:16:55.541 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-04-23T10:16:54.899+0000 7fcf17a15700 1 Processor -- start 2024-04-23T10:16:55.541 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-04-23T10:16:54.899+0000 7fcf17a15700 1 -- start start 2024-04-23T10:16:55.541 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-04-23T10:16:54.899+0000 7fcf17a15700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fcf10108a70 0x7fcf10108e90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T10:16:55.541 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-04-23T10:16:54.899+0000 7fcf17a15700 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x7fcf10109460 con 0x7fcf10108a70 2024-04-23T10:16:55.541 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-04-23T10:16:54.899+0000 7fcf157b1700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fcf10108a70 0x7fcf10108e90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T10:16:55.541 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-04-23T10:16:54.899+0000 7fcf157b1700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fcf10108a70 0x7fcf10108e90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.114:40356/0 (socket says 172.21.15.114:40356) 2024-04-23T10:16:55.541 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-04-23T10:16:54.899+0000 7fcf157b1700 1 -- 172.21.15.114:0/3656065626 learned_addr learned my addr 172.21.15.114:0/3656065626 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-04-23T10:16:55.541 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-04-23T10:16:54.899+0000 7fcf157b1700 1 -- 172.21.15.114:0/3656065626 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fcf101095e0 con 0x7fcf10108a70 2024-04-23T10:16:55.541 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-04-23T10:16:54.900+0000 7fcf157b1700 1 --2- 172.21.15.114:0/3656065626 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fcf10108a70 0x7fcf10108e90 secure :-1 s=READY pgs=3 cs=0 l=1 rev1=1 crypto rx=0x7fcf0c0098a0 tx=0x7fcf0c009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4b012e1cc8f13d45 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T10:16:55.541 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-04-23T10:16:54.900+0000 7fcf07fff700 1 -- 172.21.15.114:0/3656065626 <== mon.0 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fcf0c004160 con 0x7fcf10108a70 2024-04-23T10:16:55.541 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-04-23T10:16:54.900+0000 7fcf07fff700 1 -- 172.21.15.114:0/3656065626 <== mon.0 v2:172.21.15.114:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fcf0c004300 con 0x7fcf10108a70 2024-04-23T10:16:55.541 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-04-23T10:16:54.900+0000 7fcf07fff700 1 -- 172.21.15.114:0/3656065626 <== mon.0 v2:172.21.15.114:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fcf0c0036a0 con 0x7fcf10108a70 2024-04-23T10:16:55.541 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-04-23T10:16:54.900+0000 7fcf17a15700 1 -- 172.21.15.114:0/3656065626 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fcf10108a70 msgr2=0x7fcf10108e90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T10:16:55.542 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-04-23T10:16:54.900+0000 7fcf17a15700 1 --2- 172.21.15.114:0/3656065626 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fcf10108a70 0x7fcf10108e90 secure :-1 s=READY pgs=3 cs=0 l=1 rev1=1 crypto rx=0x7fcf0c0098a0 tx=0x7fcf0c009bb0 comp rx=0 tx=0).stop 2024-04-23T10:16:55.542 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-04-23T10:16:54.900+0000 7fcf17a15700 1 -- 172.21.15.114:0/3656065626 shutdown_connections 2024-04-23T10:16:55.542 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-04-23T10:16:54.900+0000 7fcf17a15700 1 --2- 172.21.15.114:0/3656065626 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fcf10108a70 0x7fcf10108e90 unknown :-1 s=CLOSED pgs=3 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:16:55.542 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-04-23T10:16:54.900+0000 7fcf17a15700 1 -- 172.21.15.114:0/3656065626 >> 172.21.15.114:0/3656065626 conn(0x7fcf101021a0 msgr2=0x7fcf101045c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T10:16:55.542 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-04-23T10:16:54.900+0000 7fcf17a15700 1 -- 172.21.15.114:0/3656065626 shutdown_connections 2024-04-23T10:16:55.542 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-04-23T10:16:54.900+0000 7fcf17a15700 1 -- 172.21.15.114:0/3656065626 wait complete. 2024-04-23T10:16:55.542 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-04-23T10:16:54.901+0000 7fcf17a15700 1 Processor -- start 2024-04-23T10:16:55.542 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-04-23T10:16:54.901+0000 7fcf17a15700 1 -- start start 2024-04-23T10:16:55.542 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-04-23T10:16:54.901+0000 7fcf17a15700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fcf10108a70 0x7fcf1019ddc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T10:16:55.542 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-04-23T10:16:54.901+0000 7fcf17a15700 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x7fcf101068a0 con 0x7fcf10108a70 2024-04-23T10:16:55.542 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-04-23T10:16:54.901+0000 7fcf157b1700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fcf10108a70 0x7fcf1019ddc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T10:16:55.542 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-04-23T10:16:54.901+0000 7fcf157b1700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fcf10108a70 0x7fcf1019ddc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.114:40360/0 (socket says 172.21.15.114:40360) 2024-04-23T10:16:55.542 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-04-23T10:16:54.901+0000 7fcf157b1700 1 -- 172.21.15.114:0/547765683 learned_addr learned my addr 172.21.15.114:0/547765683 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-04-23T10:16:55.542 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-04-23T10:16:54.901+0000 7fcf157b1700 1 -- 172.21.15.114:0/547765683 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fcf1019e300 con 0x7fcf10108a70 2024-04-23T10:16:55.543 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-04-23T10:16:54.902+0000 7fcf157b1700 1 --2- 172.21.15.114:0/547765683 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fcf10108a70 0x7fcf1019ddc0 secure :-1 s=READY pgs=4 cs=0 l=1 rev1=1 crypto rx=0x7fcf0c000c00 tx=0x7fcf0c004750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T10:16:55.543 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-04-23T10:16:54.902+0000 7fcf067fc700 1 -- 172.21.15.114:0/547765683 <== mon.0 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fcf0c003db0 con 0x7fcf10108a70 2024-04-23T10:16:55.543 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-04-23T10:16:54.902+0000 7fcf067fc700 1 -- 172.21.15.114:0/547765683 <== mon.0 v2:172.21.15.114:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fcf0c011630 con 0x7fcf10108a70 2024-04-23T10:16:55.543 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-04-23T10:16:54.902+0000 7fcf17a15700 1 -- 172.21.15.114:0/547765683 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fcf1019e5f0 con 0x7fcf10108a70 2024-04-23T10:16:55.543 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-04-23T10:16:54.902+0000 7fcf067fc700 1 -- 172.21.15.114:0/547765683 <== mon.0 v2:172.21.15.114:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fcf0c011950 con 0x7fcf10108a70 2024-04-23T10:16:55.543 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-04-23T10:16:54.902+0000 7fcf17a15700 1 -- 172.21.15.114:0/547765683 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fcf101a1280 con 0x7fcf10108a70 2024-04-23T10:16:55.543 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-04-23T10:16:54.902+0000 7fcf067fc700 1 -- 172.21.15.114:0/547765683 <== mon.0 v2:172.21.15.114:3300/0 4 ==== mgrmap(e 1) ==== 807+0+0 (secure 0 0 0) 0x7fcf0c011af0 con 0x7fcf10108a70 2024-04-23T10:16:55.543 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-04-23T10:16:54.902+0000 7fcf067fc700 1 -- 172.21.15.114:0/547765683 <== mon.0 v2:172.21.15.114:3300/0 5 ==== osd_map(1..1 src has 1..1) ==== 725+0+0 (secure 0 0 0) 0x7fcf0c01c790 con 0x7fcf10108a70 2024-04-23T10:16:55.543 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-04-23T10:16:54.903+0000 7fcf17a15700 1 -- 172.21.15.114:0/547765683 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fcf100640a0 con 0x7fcf10108a70 2024-04-23T10:16:55.543 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-04-23T10:16:54.905+0000 7fcf067fc700 1 -- 172.21.15.114:0/547765683 <== mon.0 v2:172.21.15.114:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+73265 (secure 0 0 0) 0x7fcf0c004260 con 0x7fcf10108a70 2024-04-23T10:16:55.543 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-04-23T10:16:54.972+0000 7fcf17a15700 1 -- 172.21.15.114:0/547765683 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_command({"prefix": "config assimilate-conf"} v 0) -- 0x7fcf1002e200 con 0x7fcf10108a70 2024-04-23T10:16:55.543 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-04-23T10:16:54.975+0000 7fcf067fc700 1 -- 172.21.15.114:0/547765683 <== mon.0 v2:172.21.15.114:3300/0 7 ==== mon_command_ack([{"prefix": "config assimilate-conf"}]=0 v2) ==== 70+0+494 (secure 0 0 0) 0x7fcf0c01ca30 con 0x7fcf10108a70 2024-04-23T10:16:55.543 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-04-23T10:16:54.976+0000 7fcf067fc700 1 -- 172.21.15.114:0/547765683 <== mon.0 v2:172.21.15.114:3300/0 8 ==== config(26 keys) ==== 1070+0+0 (secure 0 0 0) 0x7fcf0c01b8e0 con 0x7fcf10108a70 2024-04-23T10:16:55.543 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-04-23T10:16:54.977+0000 7fcf17a15700 1 -- 172.21.15.114:0/547765683 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fcf10108a70 msgr2=0x7fcf1019ddc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T10:16:55.543 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-04-23T10:16:54.977+0000 7fcf17a15700 1 --2- 172.21.15.114:0/547765683 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fcf10108a70 0x7fcf1019ddc0 secure :-1 s=READY pgs=4 cs=0 l=1 rev1=1 crypto rx=0x7fcf0c000c00 tx=0x7fcf0c004750 comp rx=0 tx=0).stop 2024-04-23T10:16:55.544 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-04-23T10:16:54.977+0000 7fcf17a15700 1 -- 172.21.15.114:0/547765683 shutdown_connections 2024-04-23T10:16:55.544 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-04-23T10:16:54.977+0000 7fcf17a15700 1 --2- 172.21.15.114:0/547765683 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fcf10108a70 0x7fcf1019ddc0 unknown :-1 s=CLOSED pgs=4 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:16:55.544 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-04-23T10:16:54.977+0000 7fcf17a15700 1 -- 172.21.15.114:0/547765683 >> 172.21.15.114:0/547765683 conn(0x7fcf101021a0 msgr2=0x7fcf10194a80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T10:16:55.544 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-04-23T10:16:54.977+0000 7fcf17a15700 1 -- 172.21.15.114:0/547765683 shutdown_connections 2024-04-23T10:16:55.544 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-04-23T10:16:54.977+0000 7fcf17a15700 1 -- 172.21.15.114:0/547765683 wait complete. 2024-04-23T10:16:55.544 INFO:teuthology.orchestra.run.smithi114.stdout:Generating new minimal ceph.conf... 2024-04-23T10:16:56.611 INFO:journalctl@ceph.mon.a.smithi114.stdout:Apr 23 10:16:56 smithi114 ceph-mon[28875]: from='client.? 172.21.15.114:0/3007470028' entity='client.admin' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-23T10:16:56.613 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-04-23T10:16:55.947+0000 7efe14971700 1 Processor -- start 2024-04-23T10:16:56.619 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-04-23T10:16:55.947+0000 7efe14971700 1 -- start start 2024-04-23T10:16:56.619 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-04-23T10:16:55.947+0000 7efe14971700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7efe10108b20 0x7efe10108f40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T10:16:56.619 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-04-23T10:16:55.947+0000 7efe14971700 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x7efe10107060 con 0x7efe10108b20 2024-04-23T10:16:56.619 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-04-23T10:16:55.948+0000 7efe0e59c700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7efe10108b20 0x7efe10108f40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T10:16:56.619 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-04-23T10:16:55.948+0000 7efe0e59c700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7efe10108b20 0x7efe10108f40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.114:57086/0 (socket says 172.21.15.114:57086) 2024-04-23T10:16:56.619 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-04-23T10:16:55.948+0000 7efe0e59c700 1 -- 172.21.15.114:0/222249742 learned_addr learned my addr 172.21.15.114:0/222249742 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-04-23T10:16:56.619 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-04-23T10:16:55.948+0000 7efe0e59c700 1 -- 172.21.15.114:0/222249742 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7efe101071e0 con 0x7efe10108b20 2024-04-23T10:16:56.620 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-04-23T10:16:55.948+0000 7efe0e59c700 1 --2- 172.21.15.114:0/222249742 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7efe10108b20 0x7efe10108f40 secure :-1 s=READY pgs=5 cs=0 l=1 rev1=1 crypto rx=0x7efe08009e80 tx=0x7efe0800b330 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a9aef10411e97266 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T10:16:56.620 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-04-23T10:16:55.948+0000 7efe0d59a700 1 -- 172.21.15.114:0/222249742 <== mon.0 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7efe08017070 con 0x7efe10108b20 2024-04-23T10:16:56.620 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-04-23T10:16:55.948+0000 7efe0d59a700 1 -- 172.21.15.114:0/222249742 <== mon.0 v2:172.21.15.114:3300/0 2 ==== config(26 keys) ==== 1070+0+0 (secure 0 0 0) 0x7efe0800b8c0 con 0x7efe10108b20 2024-04-23T10:16:56.620 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-04-23T10:16:55.948+0000 7efe0d59a700 1 -- 172.21.15.114:0/222249742 <== mon.0 v2:172.21.15.114:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7efe08004490 con 0x7efe10108b20 2024-04-23T10:16:56.620 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-04-23T10:16:55.949+0000 7efe14971700 1 -- 172.21.15.114:0/222249742 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7efe10108b20 msgr2=0x7efe10108f40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T10:16:56.620 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-04-23T10:16:55.949+0000 7efe14971700 1 --2- 172.21.15.114:0/222249742 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7efe10108b20 0x7efe10108f40 secure :-1 s=READY pgs=5 cs=0 l=1 rev1=1 crypto rx=0x7efe08009e80 tx=0x7efe0800b330 comp rx=0 tx=0).stop 2024-04-23T10:16:56.620 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-04-23T10:16:55.949+0000 7efe14971700 1 -- 172.21.15.114:0/222249742 shutdown_connections 2024-04-23T10:16:56.620 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-04-23T10:16:55.949+0000 7efe14971700 1 --2- 172.21.15.114:0/222249742 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7efe10108b20 0x7efe10108f40 unknown :-1 s=CLOSED pgs=5 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:16:56.620 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-04-23T10:16:55.949+0000 7efe14971700 1 -- 172.21.15.114:0/222249742 >> 172.21.15.114:0/222249742 conn(0x7efe10102a60 msgr2=0x7efe10104e80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T10:16:56.620 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-04-23T10:16:55.949+0000 7efe14971700 1 -- 172.21.15.114:0/222249742 shutdown_connections 2024-04-23T10:16:56.620 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-04-23T10:16:55.949+0000 7efe14971700 1 -- 172.21.15.114:0/222249742 wait complete. 2024-04-23T10:16:56.620 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-04-23T10:16:55.950+0000 7efe14971700 1 Processor -- start 2024-04-23T10:16:56.621 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-04-23T10:16:55.950+0000 7efe14971700 1 -- start start 2024-04-23T10:16:56.621 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-04-23T10:16:55.950+0000 7efe14971700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7efe10108b20 0x7efe1019b4b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T10:16:56.621 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-04-23T10:16:55.950+0000 7efe14971700 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x7efe1010b9f0 con 0x7efe10108b20 2024-04-23T10:16:56.621 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-04-23T10:16:55.950+0000 7efe0e59c700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7efe10108b20 0x7efe1019b4b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T10:16:56.621 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-04-23T10:16:55.950+0000 7efe0e59c700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7efe10108b20 0x7efe1019b4b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.114:57094/0 (socket says 172.21.15.114:57094) 2024-04-23T10:16:56.621 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-04-23T10:16:55.950+0000 7efe0e59c700 1 -- 172.21.15.114:0/3007470028 learned_addr learned my addr 172.21.15.114:0/3007470028 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-04-23T10:16:56.621 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-04-23T10:16:55.950+0000 7efe0e59c700 1 -- 172.21.15.114:0/3007470028 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7efe1019b9f0 con 0x7efe10108b20 2024-04-23T10:16:56.621 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-04-23T10:16:55.950+0000 7efe0e59c700 1 --2- 172.21.15.114:0/3007470028 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7efe10108b20 0x7efe1019b4b0 secure :-1 s=READY pgs=6 cs=0 l=1 rev1=1 crypto rx=0x7efe08004a60 tx=0x7efe0800bd50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T10:16:56.621 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-04-23T10:16:55.950+0000 7efdff7fe700 1 -- 172.21.15.114:0/3007470028 <== mon.0 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7efe08020020 con 0x7efe10108b20 2024-04-23T10:16:56.621 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-04-23T10:16:55.951+0000 7efdff7fe700 1 -- 172.21.15.114:0/3007470028 <== mon.0 v2:172.21.15.114:3300/0 2 ==== config(26 keys) ==== 1070+0+0 (secure 0 0 0) 0x7efe08012430 con 0x7efe10108b20 2024-04-23T10:16:56.621 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-04-23T10:16:55.951+0000 7efe14971700 1 -- 172.21.15.114:0/3007470028 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7efe1019bc80 con 0x7efe10108b20 2024-04-23T10:16:56.621 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-04-23T10:16:55.951+0000 7efdff7fe700 1 -- 172.21.15.114:0/3007470028 <== mon.0 v2:172.21.15.114:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7efe08017040 con 0x7efe10108b20 2024-04-23T10:16:56.621 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-04-23T10:16:55.951+0000 7efe14971700 1 -- 172.21.15.114:0/3007470028 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7efe1019e970 con 0x7efe10108b20 2024-04-23T10:16:56.622 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-04-23T10:16:55.951+0000 7efdff7fe700 1 -- 172.21.15.114:0/3007470028 <== mon.0 v2:172.21.15.114:3300/0 4 ==== mgrmap(e 1) ==== 807+0+0 (secure 0 0 0) 0x7efe08023020 con 0x7efe10108b20 2024-04-23T10:16:56.622 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-04-23T10:16:55.951+0000 7efdff7fe700 1 -- 172.21.15.114:0/3007470028 <== mon.0 v2:172.21.15.114:3300/0 5 ==== osd_map(1..1 src has 1..1) ==== 725+0+0 (secure 0 0 0) 0x7efe0800f050 con 0x7efe10108b20 2024-04-23T10:16:56.622 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-04-23T10:16:55.952+0000 7efe14971700 1 -- 172.21.15.114:0/3007470028 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7efe1010bb70 con 0x7efe10108b20 2024-04-23T10:16:56.622 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-04-23T10:16:55.954+0000 7efdff7fe700 1 -- 172.21.15.114:0/3007470028 <== mon.0 v2:172.21.15.114:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+73265 (secure 0 0 0) 0x7efe0802a020 con 0x7efe10108b20 2024-04-23T10:16:56.622 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-04-23T10:16:56.022+0000 7efe14971700 1 -- 172.21.15.114:0/3007470028 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_command({"prefix": "config generate-minimal-conf"} v 0) -- 0x7efe100640a0 con 0x7efe10108b20 2024-04-23T10:16:56.622 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-04-23T10:16:56.022+0000 7efdff7fe700 1 -- 172.21.15.114:0/3007470028 <== mon.0 v2:172.21.15.114:3300/0 7 ==== mon_command_ack([{"prefix": "config generate-minimal-conf"}]=0 v2) ==== 76+0+177 (secure 0 0 0) 0x7efe0800b860 con 0x7efe10108b20 2024-04-23T10:16:56.622 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-04-23T10:16:56.022+0000 7efe14971700 1 -- 172.21.15.114:0/3007470028 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7efe10108b20 msgr2=0x7efe1019b4b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T10:16:56.622 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-04-23T10:16:56.023+0000 7efe14971700 1 --2- 172.21.15.114:0/3007470028 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7efe10108b20 0x7efe1019b4b0 secure :-1 s=READY pgs=6 cs=0 l=1 rev1=1 crypto rx=0x7efe08004a60 tx=0x7efe0800bd50 comp rx=0 tx=0).stop 2024-04-23T10:16:56.622 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-04-23T10:16:56.023+0000 7efe14971700 1 -- 172.21.15.114:0/3007470028 shutdown_connections 2024-04-23T10:16:56.622 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-04-23T10:16:56.023+0000 7efe14971700 1 --2- 172.21.15.114:0/3007470028 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7efe10108b20 0x7efe1019b4b0 unknown :-1 s=CLOSED pgs=6 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:16:56.622 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-04-23T10:16:56.023+0000 7efe14971700 1 -- 172.21.15.114:0/3007470028 >> 172.21.15.114:0/3007470028 conn(0x7efe10102a60 msgr2=0x7efe101035d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T10:16:56.623 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-04-23T10:16:56.023+0000 7efe14971700 1 -- 172.21.15.114:0/3007470028 shutdown_connections 2024-04-23T10:16:56.623 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-04-23T10:16:56.023+0000 7efe14971700 1 -- 172.21.15.114:0/3007470028 wait complete. 2024-04-23T10:16:56.623 INFO:teuthology.orchestra.run.smithi114.stdout:Restarting the monitor... 2024-04-23T10:16:56.899 INFO:journalctl@ceph.mon.a.smithi114.stdout:Apr 23 10:16:56 smithi114 systemd[1]: Stopping Ceph mon.a for aa240572-0159-11ef-bc93-c7b262605968... 2024-04-23T10:16:57.254 INFO:journalctl@ceph.mon.a.smithi114.stdout:Apr 23 10:16:56 smithi114 ceph-aa240572-0159-11ef-bc93-c7b262605968-mon-a[28870]: 2024-04-23T10:16:56.899+0000 7fc0768ad700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-mon -n mon.a -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false --default-mon-cluster-log-to-file=false --default-mon-cluster-log-to-journald=true --default-mon-cluster-log-to-stderr=false (PID: 1) UID: 0 2024-04-23T10:16:57.255 INFO:journalctl@ceph.mon.a.smithi114.stdout:Apr 23 10:16:56 smithi114 ceph-aa240572-0159-11ef-bc93-c7b262605968-mon-a[28870]: 2024-04-23T10:16:56.899+0000 7fc0768ad700 -1 mon.a@0(leader) e1 *** Got Signal Terminated *** 2024-04-23T10:16:57.255 INFO:journalctl@ceph.mon.a.smithi114.stdout:Apr 23 10:16:57 smithi114 podman[29128]: 2024-04-23 10:16:57.042943448 +0000 UTC m=+0.263996585 container died afc296595d8bc0d40ba5be4342ea22ed5844c07c3565d81fc249404dd9cec17f (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d69f13d827952bb07bb9b7973bca14f97d764d6f, name=ceph-aa240572-0159-11ef-bc93-c7b262605968-mon-a, CEPH_POINT_RELEASE=, org.label-schema.license=GPLv2, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, io.buildah.version=1.33.5, RELEASE=wip-lusov-quiescer-d69f13d, org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0, org.label-schema.build-date=20240416, org.label-schema.name=CentOS Stream 8 Base Image, GIT_CLEAN=True, ceph=True, maintainer=Guillaume Abrioux , GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_BRANCH=HEAD) 2024-04-23T10:16:57.565 INFO:journalctl@ceph.mon.a.smithi114.stdout:Apr 23 10:16:57 smithi114 podman[29128]: 2024-04-23 10:16:57.382796448 +0000 UTC m=+0.603849583 container cleanup afc296595d8bc0d40ba5be4342ea22ed5844c07c3565d81fc249404dd9cec17f (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d69f13d827952bb07bb9b7973bca14f97d764d6f, name=ceph-aa240572-0159-11ef-bc93-c7b262605968-mon-a, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, RELEASE=wip-lusov-quiescer-d69f13d, org.label-schema.build-date=20240416, CEPH_POINT_RELEASE=, org.label-schema.vendor=CentOS, io.buildah.version=1.33.5, ceph=True, GIT_CLEAN=True, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.schema-version=1.0) 2024-04-23T10:16:57.565 INFO:journalctl@ceph.mon.a.smithi114.stdout:Apr 23 10:16:57 smithi114 bash[29128]: ceph-aa240572-0159-11ef-bc93-c7b262605968-mon-a 2024-04-23T10:16:58.004 INFO:journalctl@ceph.mon.a.smithi114.stdout:Apr 23 10:16:57 smithi114 podman[29140]: 2024-04-23 10:16:57.566834641 +0000 UTC m=+0.519246480 container remove afc296595d8bc0d40ba5be4342ea22ed5844c07c3565d81fc249404dd9cec17f (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d69f13d827952bb07bb9b7973bca14f97d764d6f, name=ceph-aa240572-0159-11ef-bc93-c7b262605968-mon-a, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240416, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS, io.buildah.version=1.33.5, org.label-schema.name=CentOS Stream 8 Base Image, GIT_BRANCH=HEAD, GIT_CLEAN=True, RELEASE=wip-lusov-quiescer-d69f13d, ceph=True, CEPH_POINT_RELEASE=) 2024-04-23T10:16:58.004 INFO:journalctl@ceph.mon.a.smithi114.stdout:Apr 23 10:16:57 smithi114 systemd[1]: ceph-aa240572-0159-11ef-bc93-c7b262605968@mon.a.service: Deactivated successfully. 2024-04-23T10:16:58.004 INFO:journalctl@ceph.mon.a.smithi114.stdout:Apr 23 10:16:57 smithi114 systemd[1]: Stopped Ceph mon.a for aa240572-0159-11ef-bc93-c7b262605968. 2024-04-23T10:16:58.004 INFO:journalctl@ceph.mon.a.smithi114.stdout:Apr 23 10:16:57 smithi114 systemd[1]: Starting Ceph mon.a for aa240572-0159-11ef-bc93-c7b262605968... 2024-04-23T10:16:58.260 INFO:journalctl@ceph.mon.a.smithi114.stdout:Apr 23 10:16:58 smithi114 podman[29236]: 2024-04-23 10:16:58.009185675 +0000 UTC m=+0.108572644 container create 3c977ab8b867b227b23fcd666ab2002050abf6e75f25f3bb9a7d96eb2b2e973b (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d69f13d827952bb07bb9b7973bca14f97d764d6f, name=ceph-aa240572-0159-11ef-bc93-c7b262605968-mon-a, org.label-schema.build-date=20240416, io.buildah.version=1.33.5, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 8 Base Image, CEPH_POINT_RELEASE=, GIT_CLEAN=True, org.label-schema.schema-version=1.0, ceph=True, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, RELEASE=wip-lusov-quiescer-d69f13d, GIT_BRANCH=HEAD, org.label-schema.license=GPLv2, maintainer=Guillaume Abrioux ) 2024-04-23T10:16:58.260 INFO:journalctl@ceph.mon.a.smithi114.stdout:Apr 23 10:16:58 smithi114 podman[29236]: 2024-04-23 10:16:57.918228498 +0000 UTC m=+0.017615466 image pull da4b004c09e820713483d039bc76fbd660be678f2ea3c03e059a6161b74dab76 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d69f13d827952bb07bb9b7973bca14f97d764d6f 2024-04-23T10:16:58.260 INFO:journalctl@ceph.mon.a.smithi114.stdout:Apr 23 10:16:58 smithi114 podman[29236]: 2024-04-23 10:16:58.19462314 +0000 UTC m=+0.294010108 container init 3c977ab8b867b227b23fcd666ab2002050abf6e75f25f3bb9a7d96eb2b2e973b (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d69f13d827952bb07bb9b7973bca14f97d764d6f, name=ceph-aa240572-0159-11ef-bc93-c7b262605968-mon-a, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, org.label-schema.build-date=20240416, GIT_CLEAN=True, org.label-schema.schema-version=1.0, org.label-schema.name=CentOS Stream 8 Base Image, maintainer=Guillaume Abrioux , RELEASE=wip-lusov-quiescer-d69f13d, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, org.label-schema.license=GPLv2, ceph=True, io.buildah.version=1.33.5, GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-04-23T10:16:58.260 INFO:journalctl@ceph.mon.a.smithi114.stdout:Apr 23 10:16:58 smithi114 podman[29236]: 2024-04-23 10:16:58.197392667 +0000 UTC m=+0.296779638 container start 3c977ab8b867b227b23fcd666ab2002050abf6e75f25f3bb9a7d96eb2b2e973b (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d69f13d827952bb07bb9b7973bca14f97d764d6f, name=ceph-aa240572-0159-11ef-bc93-c7b262605968-mon-a, RELEASE=wip-lusov-quiescer-d69f13d, io.buildah.version=1.33.5, org.label-schema.schema-version=1.0, GIT_CLEAN=True, ceph=True, maintainer=Guillaume Abrioux , GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, org.label-schema.build-date=20240416, org.label-schema.name=CentOS Stream 8 Base Image, GIT_BRANCH=HEAD, CEPH_POINT_RELEASE=, org.label-schema.vendor=CentOS) 2024-04-23T10:16:58.260 INFO:journalctl@ceph.mon.a.smithi114.stdout:Apr 23 10:16:58 smithi114 ceph-mon[29255]: set uid:gid to 167:167 (ceph:ceph) 2024-04-23T10:16:58.260 INFO:journalctl@ceph.mon.a.smithi114.stdout:Apr 23 10:16:58 smithi114 ceph-mon[29255]: ceph version 19.0.0-3156-gd69f13d8 (d69f13d827952bb07bb9b7973bca14f97d764d6f) squid (dev), process ceph-mon, pid 2 2024-04-23T10:16:58.260 INFO:journalctl@ceph.mon.a.smithi114.stdout:Apr 23 10:16:58 smithi114 ceph-mon[29255]: pidfile_write: ignore empty --pid-file 2024-04-23T10:16:58.260 INFO:journalctl@ceph.mon.a.smithi114.stdout:Apr 23 10:16:58 smithi114 ceph-mon[29255]: load: jerasure load: lrc 2024-04-23T10:16:58.260 INFO:journalctl@ceph.mon.a.smithi114.stdout:Apr 23 10:16:58 smithi114 ceph-mon[29255]: rocksdb: RocksDB version: 7.9.2 2024-04-23T10:16:58.261 INFO:journalctl@ceph.mon.a.smithi114.stdout:Apr 23 10:16:58 smithi114 ceph-mon[29255]: rocksdb: Git sha 0 2024-04-23T10:16:58.261 INFO:journalctl@ceph.mon.a.smithi114.stdout:Apr 23 10:16:58 smithi114 ceph-mon[29255]: rocksdb: Compile date 2024-04-22 10:57:25 2024-04-23T10:16:58.261 INFO:journalctl@ceph.mon.a.smithi114.stdout:Apr 23 10:16:58 smithi114 ceph-mon[29255]: rocksdb: DB SUMMARY 2024-04-23T10:16:58.261 INFO:journalctl@ceph.mon.a.smithi114.stdout:Apr 23 10:16:58 smithi114 ceph-mon[29255]: rocksdb: DB Session ID: Q2C4SBXVK369NOD3T4L4 2024-04-23T10:16:58.261 INFO:journalctl@ceph.mon.a.smithi114.stdout:Apr 23 10:16:58 smithi114 ceph-mon[29255]: rocksdb: CURRENT file: CURRENT 2024-04-23T10:16:58.261 INFO:journalctl@ceph.mon.a.smithi114.stdout:Apr 23 10:16:58 smithi114 ceph-mon[29255]: rocksdb: IDENTITY file: IDENTITY 2024-04-23T10:16:58.261 INFO:journalctl@ceph.mon.a.smithi114.stdout:Apr 23 10:16:58 smithi114 ceph-mon[29255]: rocksdb: MANIFEST file: MANIFEST-000010 size: 179 Bytes 2024-04-23T10:16:58.261 INFO:journalctl@ceph.mon.a.smithi114.stdout:Apr 23 10:16:58 smithi114 ceph-mon[29255]: rocksdb: SST files in /var/lib/ceph/mon/ceph-a/store.db dir, Total Num: 1, files: 000008.sst 2024-04-23T10:16:58.261 INFO:journalctl@ceph.mon.a.smithi114.stdout:Apr 23 10:16:58 smithi114 ceph-mon[29255]: rocksdb: Write Ahead Log file in /var/lib/ceph/mon/ceph-a/store.db: 000009.log size: 105462 ; 2024-04-23T10:16:58.261 INFO:journalctl@ceph.mon.a.smithi114.stdout:Apr 23 10:16:58 smithi114 ceph-mon[29255]: rocksdb: Options.error_if_exists: 0 2024-04-23T10:16:58.263 INFO:journalctl@ceph.mon.a.smithi114.stdout:Apr 23 10:16:58 smithi114 ceph-mon[29255]: rocksdb: Options.create_if_missing: 0 2024-04-23T10:16:58.263 INFO:journalctl@ceph.mon.a.smithi114.stdout:Apr 23 10:16:58 smithi114 ceph-mon[29255]: rocksdb: Options.paranoid_checks: 1 2024-04-23T10:16:58.263 INFO:journalctl@ceph.mon.a.smithi114.stdout:Apr 23 10:16:58 smithi114 ceph-mon[29255]: rocksdb: Options.flush_verify_memtable_count: 1 2024-04-23T10:16:58.263 INFO:journalctl@ceph.mon.a.smithi114.stdout:Apr 23 10:16:58 smithi114 ceph-mon[29255]: rocksdb: Options.track_and_verify_wals_in_manifest: 0 2024-04-23T10:16:58.263 INFO:journalctl@ceph.mon.a.smithi114.stdout:Apr 23 10:16:58 smithi114 ceph-mon[29255]: rocksdb: Options.verify_sst_unique_id_in_manifest: 1 2024-04-23T10:16:58.263 INFO:journalctl@ceph.mon.a.smithi114.stdout:Apr 23 10:16:58 smithi114 ceph-mon[29255]: rocksdb: Options.env: 0x562d43274cc0 2024-04-23T10:16:58.263 INFO:journalctl@ceph.mon.a.smithi114.stdout:Apr 23 10:16:58 smithi114 ceph-mon[29255]: rocksdb: Options.fs: PosixFileSystem 2024-04-23T10:16:58.263 INFO:journalctl@ceph.mon.a.smithi114.stdout:Apr 23 10:16:58 smithi114 ceph-mon[29255]: rocksdb: Options.info_log: 0x562d44ea7be0 2024-04-23T10:16:58.263 INFO:journalctl@ceph.mon.a.smithi114.stdout:Apr 23 10:16:58 smithi114 ceph-mon[29255]: rocksdb: Options.max_file_opening_threads: 16 2024-04-23T10:16:58.263 INFO:journalctl@ceph.mon.a.smithi114.stdout:Apr 23 10:16:58 smithi114 ceph-mon[29255]: rocksdb: Options.statistics: (nil) 2024-04-23T10:16:58.264 INFO:journalctl@ceph.mon.a.smithi114.stdout:Apr 23 10:16:58 smithi114 ceph-mon[29255]: rocksdb: Options.use_fsync: 0 2024-04-23T10:16:58.264 INFO:journalctl@ceph.mon.a.smithi114.stdout:Apr 23 10:16:58 smithi114 ceph-mon[29255]: rocksdb: Options.max_log_file_size: 0 2024-04-23T10:16:58.264 INFO:journalctl@ceph.mon.a.smithi114.stdout:Apr 23 10:16:58 smithi114 ceph-mon[29255]: rocksdb: Options.max_manifest_file_size: 1073741824 2024-04-23T10:16:58.264 INFO:journalctl@ceph.mon.a.smithi114.stdout:Apr 23 10:16:58 smithi114 ceph-mon[29255]: rocksdb: Options.log_file_time_to_roll: 0 2024-04-23T10:16:58.264 INFO:journalctl@ceph.mon.a.smithi114.stdout:Apr 23 10:16:58 smithi114 ceph-mon[29255]: rocksdb: Options.keep_log_file_num: 1000 2024-04-23T10:16:58.264 INFO:journalctl@ceph.mon.a.smithi114.stdout:Apr 23 10:16:58 smithi114 ceph-mon[29255]: rocksdb: Options.recycle_log_file_num: 0 2024-04-23T10:16:58.264 INFO:journalctl@ceph.mon.a.smithi114.stdout:Apr 23 10:16:58 smithi114 ceph-mon[29255]: rocksdb: Options.allow_fallocate: 1 2024-04-23T10:16:58.264 INFO:journalctl@ceph.mon.a.smithi114.stdout:Apr 23 10:16:58 smithi114 ceph-mon[29255]: rocksdb: Options.allow_mmap_reads: 0 2024-04-23T10:16:58.264 INFO:journalctl@ceph.mon.a.smithi114.stdout:Apr 23 10:16:58 smithi114 ceph-mon[29255]: rocksdb: Options.allow_mmap_writes: 0 2024-04-23T10:16:58.264 INFO:journalctl@ceph.mon.a.smithi114.stdout:Apr 23 10:16:58 smithi114 ceph-mon[29255]: rocksdb: Options.use_direct_reads: 0 2024-04-23T10:16:58.264 INFO:journalctl@ceph.mon.a.smithi114.stdout:Apr 23 10:16:58 smithi114 ceph-mon[29255]: rocksdb: Options.use_direct_io_for_flush_and_compaction: 0 2024-04-23T10:16:58.264 INFO:journalctl@ceph.mon.a.smithi114.stdout:Apr 23 10:16:58 smithi114 ceph-mon[29255]: rocksdb: Options.create_missing_column_families: 0 2024-04-23T10:16:58.264 INFO:journalctl@ceph.mon.a.smithi114.stdout:Apr 23 10:16:58 smithi114 ceph-mon[29255]: rocksdb: Options.db_log_dir: 2024-04-23T10:16:58.265 INFO:journalctl@ceph.mon.a.smithi114.stdout:Apr 23 10:16:58 smithi114 ceph-mon[29255]: rocksdb: Options.wal_dir: 2024-04-23T10:16:58.265 INFO:journalctl@ceph.mon.a.smithi114.stdout:Apr 23 10:16:58 smithi114 ceph-mon[29255]: rocksdb: Options.table_cache_numshardbits: 6 2024-04-23T10:16:58.265 INFO:journalctl@ceph.mon.a.smithi114.stdout:Apr 23 10:16:58 smithi114 ceph-mon[29255]: rocksdb: Options.WAL_ttl_seconds: 0 2024-04-23T10:16:58.265 INFO:journalctl@ceph.mon.a.smithi114.stdout:Apr 23 10:16:58 smithi114 ceph-mon[29255]: rocksdb: Options.WAL_size_limit_MB: 0 2024-04-23T10:16:58.265 INFO:journalctl@ceph.mon.a.smithi114.stdout:Apr 23 10:16:58 smithi114 ceph-mon[29255]: rocksdb: Options.max_write_batch_group_size_bytes: 1048576 2024-04-23T10:16:58.265 INFO:journalctl@ceph.mon.a.smithi114.stdout:Apr 23 10:16:58 smithi114 ceph-mon[29255]: rocksdb: Options.manifest_preallocation_size: 4194304 2024-04-23T10:16:58.265 INFO:journalctl@ceph.mon.a.smithi114.stdout:Apr 23 10:16:58 smithi114 ceph-mon[29255]: rocksdb: Options.is_fd_close_on_exec: 1 2024-04-23T10:16:58.265 INFO:journalctl@ceph.mon.a.smithi114.stdout:Apr 23 10:16:58 smithi114 ceph-mon[29255]: rocksdb: Options.advise_random_on_open: 1 2024-04-23T10:16:58.265 INFO:journalctl@ceph.mon.a.smithi114.stdout:Apr 23 10:16:58 smithi114 ceph-mon[29255]: rocksdb: Options.db_write_buffer_size: 0 2024-04-23T10:16:58.265 INFO:journalctl@ceph.mon.a.smithi114.stdout:Apr 23 10:16:58 smithi114 ceph-mon[29255]: rocksdb: Options.write_buffer_manager: 0x562d44e3b0e0 2024-04-23T10:16:58.265 INFO:journalctl@ceph.mon.a.smithi114.stdout:Apr 23 10:16:58 smithi114 ceph-mon[29255]: rocksdb: Options.access_hint_on_compaction_start: 1 2024-04-23T10:16:58.265 INFO:journalctl@ceph.mon.a.smithi114.stdout:Apr 23 10:16:58 smithi114 ceph-mon[29255]: rocksdb: Options.random_access_max_buffer_size: 1048576 2024-04-23T10:16:58.265 INFO:journalctl@ceph.mon.a.smithi114.stdout:Apr 23 10:16:58 smithi114 ceph-mon[29255]: rocksdb: Options.use_adaptive_mutex: 0 2024-04-23T10:16:58.265 INFO:journalctl@ceph.mon.a.smithi114.stdout:Apr 23 10:16:58 smithi114 ceph-mon[29255]: rocksdb: Options.rate_limiter: (nil) 2024-04-23T10:16:58.266 INFO:journalctl@ceph.mon.a.smithi114.stdout:Apr 23 10:16:58 smithi114 ceph-mon[29255]: rocksdb: Options.sst_file_manager.rate_bytes_per_sec: 0 2024-04-23T10:16:58.266 INFO:journalctl@ceph.mon.a.smithi114.stdout:Apr 23 10:16:58 smithi114 ceph-mon[29255]: rocksdb: Options.wal_recovery_mode: 2 2024-04-23T10:16:58.266 INFO:journalctl@ceph.mon.a.smithi114.stdout:Apr 23 10:16:58 smithi114 ceph-mon[29255]: rocksdb: Options.enable_thread_tracking: 0 2024-04-23T10:16:58.266 INFO:journalctl@ceph.mon.a.smithi114.stdout:Apr 23 10:16:58 smithi114 ceph-mon[29255]: rocksdb: Options.enable_pipelined_write: 0 2024-04-23T10:16:58.266 INFO:journalctl@ceph.mon.a.smithi114.stdout:Apr 23 10:16:58 smithi114 ceph-mon[29255]: rocksdb: Options.unordered_write: 0 2024-04-23T10:16:58.266 INFO:journalctl@ceph.mon.a.smithi114.stdout:Apr 23 10:16:58 smithi114 ceph-mon[29255]: rocksdb: Options.allow_concurrent_memtable_write: 1 2024-04-23T10:16:58.266 INFO:journalctl@ceph.mon.a.smithi114.stdout:Apr 23 10:16:58 smithi114 ceph-mon[29255]: rocksdb: Options.enable_write_thread_adaptive_yield: 1 2024-04-23T10:16:58.266 INFO:journalctl@ceph.mon.a.smithi114.stdout:Apr 23 10:16:58 smithi114 ceph-mon[29255]: rocksdb: Options.write_thread_max_yield_usec: 100 2024-04-23T10:16:58.266 INFO:journalctl@ceph.mon.a.smithi114.stdout:Apr 23 10:16:58 smithi114 ceph-mon[29255]: rocksdb: Options.write_thread_slow_yield_usec: 3 2024-04-23T10:16:58.266 INFO:journalctl@ceph.mon.a.smithi114.stdout:Apr 23 10:16:58 smithi114 ceph-mon[29255]: rocksdb: Options.row_cache: None 2024-04-23T10:16:58.266 INFO:journalctl@ceph.mon.a.smithi114.stdout:Apr 23 10:16:58 smithi114 ceph-mon[29255]: rocksdb: Options.wal_filter: None 2024-04-23T10:16:58.266 INFO:journalctl@ceph.mon.a.smithi114.stdout:Apr 23 10:16:58 smithi114 ceph-mon[29255]: rocksdb: Options.avoid_flush_during_recovery: 0 2024-04-23T10:16:58.266 INFO:journalctl@ceph.mon.a.smithi114.stdout:Apr 23 10:16:58 smithi114 ceph-mon[29255]: rocksdb: Options.allow_ingest_behind: 0 2024-04-23T10:16:58.267 INFO:journalctl@ceph.mon.a.smithi114.stdout:Apr 23 10:16:58 smithi114 ceph-mon[29255]: rocksdb: Options.two_write_queues: 0 2024-04-23T10:16:58.267 INFO:journalctl@ceph.mon.a.smithi114.stdout:Apr 23 10:16:58 smithi114 ceph-mon[29255]: rocksdb: Options.manual_wal_flush: 0 2024-04-23T10:16:58.267 INFO:journalctl@ceph.mon.a.smithi114.stdout:Apr 23 10:16:58 smithi114 ceph-mon[29255]: rocksdb: Options.wal_compression: 0 2024-04-23T10:16:58.267 INFO:journalctl@ceph.mon.a.smithi114.stdout:Apr 23 10:16:58 smithi114 ceph-mon[29255]: rocksdb: Options.atomic_flush: 0 2024-04-23T10:16:58.267 INFO:journalctl@ceph.mon.a.smithi114.stdout:Apr 23 10:16:58 smithi114 ceph-mon[29255]: rocksdb: Options.avoid_unnecessary_blocking_io: 0 2024-04-23T10:16:58.267 INFO:journalctl@ceph.mon.a.smithi114.stdout:Apr 23 10:16:58 smithi114 ceph-mon[29255]: rocksdb: Options.persist_stats_to_disk: 0 2024-04-23T10:16:58.267 INFO:journalctl@ceph.mon.a.smithi114.stdout:Apr 23 10:16:58 smithi114 ceph-mon[29255]: rocksdb: Options.write_dbid_to_manifest: 0 2024-04-23T10:16:58.267 INFO:journalctl@ceph.mon.a.smithi114.stdout:Apr 23 10:16:58 smithi114 ceph-mon[29255]: rocksdb: Options.log_readahead_size: 0 2024-04-23T10:16:58.267 INFO:journalctl@ceph.mon.a.smithi114.stdout:Apr 23 10:16:58 smithi114 ceph-mon[29255]: rocksdb: Options.file_checksum_gen_factory: Unknown 2024-04-23T10:16:58.267 INFO:journalctl@ceph.mon.a.smithi114.stdout:Apr 23 10:16:58 smithi114 ceph-mon[29255]: rocksdb: Options.best_efforts_recovery: 0 2024-04-23T10:16:58.267 INFO:journalctl@ceph.mon.a.smithi114.stdout:Apr 23 10:16:58 smithi114 ceph-mon[29255]: rocksdb: Options.max_bgerror_resume_count: 2147483647 2024-04-23T10:16:58.267 INFO:journalctl@ceph.mon.a.smithi114.stdout:Apr 23 10:16:58 smithi114 ceph-mon[29255]: rocksdb: Options.bgerror_resume_retry_interval: 1000000 2024-04-23T10:16:58.267 INFO:journalctl@ceph.mon.a.smithi114.stdout:Apr 23 10:16:58 smithi114 ceph-mon[29255]: rocksdb: Options.allow_data_in_errors: 0 2024-04-23T10:16:58.267 INFO:journalctl@ceph.mon.a.smithi114.stdout:Apr 23 10:16:58 smithi114 ceph-mon[29255]: rocksdb: Options.db_host_id: __hostname__ 2024-04-23T10:16:58.267 INFO:journalctl@ceph.mon.a.smithi114.stdout:Apr 23 10:16:58 smithi114 ceph-mon[29255]: rocksdb: Options.enforce_single_del_contracts: true 2024-04-23T10:16:58.268 INFO:journalctl@ceph.mon.a.smithi114.stdout:Apr 23 10:16:58 smithi114 ceph-mon[29255]: rocksdb: Options.max_background_jobs: 2 2024-04-23T10:16:58.268 INFO:journalctl@ceph.mon.a.smithi114.stdout:Apr 23 10:16:58 smithi114 ceph-mon[29255]: rocksdb: Options.max_background_compactions: -1 2024-04-23T10:16:58.268 INFO:journalctl@ceph.mon.a.smithi114.stdout:Apr 23 10:16:58 smithi114 ceph-mon[29255]: rocksdb: Options.max_subcompactions: 1 2024-04-23T10:16:58.268 INFO:journalctl@ceph.mon.a.smithi114.stdout:Apr 23 10:16:58 smithi114 ceph-mon[29255]: rocksdb: Options.avoid_flush_during_shutdown: 0 2024-04-23T10:16:58.268 INFO:journalctl@ceph.mon.a.smithi114.stdout:Apr 23 10:16:58 smithi114 ceph-mon[29255]: rocksdb: Options.writable_file_max_buffer_size: 1048576 2024-04-23T10:16:58.268 INFO:journalctl@ceph.mon.a.smithi114.stdout:Apr 23 10:16:58 smithi114 ceph-mon[29255]: rocksdb: Options.delayed_write_rate : 16777216 2024-04-23T10:16:58.268 INFO:journalctl@ceph.mon.a.smithi114.stdout:Apr 23 10:16:58 smithi114 ceph-mon[29255]: rocksdb: Options.max_total_wal_size: 0 2024-04-23T10:16:58.268 INFO:journalctl@ceph.mon.a.smithi114.stdout:Apr 23 10:16:58 smithi114 ceph-mon[29255]: rocksdb: Options.delete_obsolete_files_period_micros: 21600000000 2024-04-23T10:16:58.268 INFO:journalctl@ceph.mon.a.smithi114.stdout:Apr 23 10:16:58 smithi114 ceph-mon[29255]: rocksdb: Options.stats_dump_period_sec: 600 2024-04-23T10:16:58.268 INFO:journalctl@ceph.mon.a.smithi114.stdout:Apr 23 10:16:58 smithi114 ceph-mon[29255]: rocksdb: Options.stats_persist_period_sec: 600 2024-04-23T10:16:58.268 INFO:journalctl@ceph.mon.a.smithi114.stdout:Apr 23 10:16:58 smithi114 ceph-mon[29255]: rocksdb: Options.stats_history_buffer_size: 1048576 2024-04-23T10:16:58.268 INFO:journalctl@ceph.mon.a.smithi114.stdout:Apr 23 10:16:58 smithi114 ceph-mon[29255]: rocksdb: Options.max_open_files: -1 2024-04-23T10:16:58.268 INFO:journalctl@ceph.mon.a.smithi114.stdout:Apr 23 10:16:58 smithi114 ceph-mon[29255]: rocksdb: Options.bytes_per_sync: 0 2024-04-23T10:16:58.268 INFO:journalctl@ceph.mon.a.smithi114.stdout:Apr 23 10:16:58 smithi114 ceph-mon[29255]: rocksdb: Options.wal_bytes_per_sync: 0 2024-04-23T10:16:58.269 INFO:journalctl@ceph.mon.a.smithi114.stdout:Apr 23 10:16:58 smithi114 ceph-mon[29255]: rocksdb: Options.strict_bytes_per_sync: 0 2024-04-23T10:16:58.269 INFO:journalctl@ceph.mon.a.smithi114.stdout:Apr 23 10:16:58 smithi114 ceph-mon[29255]: rocksdb: Options.compaction_readahead_size: 0 2024-04-23T10:16:58.269 INFO:journalctl@ceph.mon.a.smithi114.stdout:Apr 23 10:16:58 smithi114 ceph-mon[29255]: rocksdb: Options.max_background_flushes: -1 2024-04-23T10:16:58.269 INFO:journalctl@ceph.mon.a.smithi114.stdout:Apr 23 10:16:58 smithi114 ceph-mon[29255]: rocksdb: Compression algorithms supported: 2024-04-23T10:16:58.269 INFO:journalctl@ceph.mon.a.smithi114.stdout:Apr 23 10:16:58 smithi114 ceph-mon[29255]: rocksdb: kZSTDNotFinalCompression supported: 0 2024-04-23T10:16:58.269 INFO:journalctl@ceph.mon.a.smithi114.stdout:Apr 23 10:16:58 smithi114 ceph-mon[29255]: rocksdb: kZSTD supported: 0 2024-04-23T10:16:58.269 INFO:journalctl@ceph.mon.a.smithi114.stdout:Apr 23 10:16:58 smithi114 ceph-mon[29255]: rocksdb: kXpressCompression supported: 0 2024-04-23T10:16:58.269 INFO:journalctl@ceph.mon.a.smithi114.stdout:Apr 23 10:16:58 smithi114 ceph-mon[29255]: rocksdb: kLZ4HCCompression supported: 1 2024-04-23T10:16:58.269 INFO:journalctl@ceph.mon.a.smithi114.stdout:Apr 23 10:16:58 smithi114 ceph-mon[29255]: rocksdb: kZlibCompression supported: 1 2024-04-23T10:16:58.269 INFO:journalctl@ceph.mon.a.smithi114.stdout:Apr 23 10:16:58 smithi114 ceph-mon[29255]: rocksdb: kSnappyCompression supported: 1 2024-04-23T10:16:58.269 INFO:journalctl@ceph.mon.a.smithi114.stdout:Apr 23 10:16:58 smithi114 ceph-mon[29255]: rocksdb: kLZ4Compression supported: 1 2024-04-23T10:16:58.269 INFO:journalctl@ceph.mon.a.smithi114.stdout:Apr 23 10:16:58 smithi114 ceph-mon[29255]: rocksdb: kBZip2Compression supported: 0 2024-04-23T10:16:58.269 INFO:journalctl@ceph.mon.a.smithi114.stdout:Apr 23 10:16:58 smithi114 ceph-mon[29255]: rocksdb: Fast CRC32 supported: Supported on x86 2024-04-23T10:16:58.270 INFO:journalctl@ceph.mon.a.smithi114.stdout:Apr 23 10:16:58 smithi114 ceph-mon[29255]: rocksdb: DMutex implementation: pthread_mutex_t 2024-04-23T10:16:58.270 INFO:journalctl@ceph.mon.a.smithi114.stdout:Apr 23 10:16:58 smithi114 ceph-mon[29255]: rocksdb: [db/version_set.cc:5527] Recovering from manifest file: /var/lib/ceph/mon/ceph-a/store.db/MANIFEST-000010 2024-04-23T10:16:58.270 INFO:journalctl@ceph.mon.a.smithi114.stdout:Apr 23 10:16:58 smithi114 ceph-mon[29255]: rocksdb: [db/column_family.cc:630] --------------- Options for column family [default]: 2024-04-23T10:16:58.270 INFO:journalctl@ceph.mon.a.smithi114.stdout:Apr 23 10:16:58 smithi114 ceph-mon[29255]: rocksdb: Options.comparator: leveldb.BytewiseComparator 2024-04-23T10:16:58.270 INFO:journalctl@ceph.mon.a.smithi114.stdout:Apr 23 10:16:58 smithi114 ceph-mon[29255]: rocksdb: Options.merge_operator: 2024-04-23T10:16:58.270 INFO:journalctl@ceph.mon.a.smithi114.stdout:Apr 23 10:16:58 smithi114 ceph-mon[29255]: rocksdb: Options.compaction_filter: None 2024-04-23T10:16:58.270 INFO:journalctl@ceph.mon.a.smithi114.stdout:Apr 23 10:16:58 smithi114 ceph-mon[29255]: rocksdb: Options.compaction_filter_factory: None 2024-04-23T10:16:58.270 INFO:journalctl@ceph.mon.a.smithi114.stdout:Apr 23 10:16:58 smithi114 ceph-mon[29255]: rocksdb: Options.sst_partitioner_factory: None 2024-04-23T10:16:58.270 INFO:journalctl@ceph.mon.a.smithi114.stdout:Apr 23 10:16:58 smithi114 ceph-mon[29255]: rocksdb: Options.memtable_factory: SkipListFactory 2024-04-23T10:16:58.270 INFO:journalctl@ceph.mon.a.smithi114.stdout:Apr 23 10:16:58 smithi114 ceph-mon[29255]: rocksdb: Options.table_factory: BlockBasedTable 2024-04-23T10:16:58.270 INFO:journalctl@ceph.mon.a.smithi114.stdout:Apr 23 10:16:58 smithi114 ceph-mon[29255]: rocksdb: table_factory options: flush_block_policy_factory: FlushBlockBySizePolicyFactory (0x562d44ea7dc0) 2024-04-23T10:16:58.270 INFO:journalctl@ceph.mon.a.smithi114.stdout: cache_index_and_filter_blocks: 1 2024-04-23T10:16:58.271 INFO:journalctl@ceph.mon.a.smithi114.stdout: cache_index_and_filter_blocks_with_high_priority: 0 2024-04-23T10:16:58.271 INFO:journalctl@ceph.mon.a.smithi114.stdout: pin_l0_filter_and_index_blocks_in_cache: 0 2024-04-23T10:16:58.271 INFO:journalctl@ceph.mon.a.smithi114.stdout: pin_top_level_index_and_filter: 1 2024-04-23T10:16:58.271 INFO:journalctl@ceph.mon.a.smithi114.stdout: index_type: 0 2024-04-23T10:16:58.271 INFO:journalctl@ceph.mon.a.smithi114.stdout: data_block_index_type: 0 2024-04-23T10:16:58.271 INFO:journalctl@ceph.mon.a.smithi114.stdout: index_shortening: 1 2024-04-23T10:16:58.271 INFO:journalctl@ceph.mon.a.smithi114.stdout: data_block_hash_table_util_ratio: 0.750000 2024-04-23T10:16:58.271 INFO:journalctl@ceph.mon.a.smithi114.stdout: checksum: 4 2024-04-23T10:16:58.271 INFO:journalctl@ceph.mon.a.smithi114.stdout: no_block_cache: 0 2024-04-23T10:16:58.271 INFO:journalctl@ceph.mon.a.smithi114.stdout: block_cache: 0x562d44e6d1f0 2024-04-23T10:16:58.271 INFO:journalctl@ceph.mon.a.smithi114.stdout: block_cache_name: BinnedLRUCache 2024-04-23T10:16:58.271 INFO:journalctl@ceph.mon.a.smithi114.stdout: block_cache_options: 2024-04-23T10:16:58.271 INFO:journalctl@ceph.mon.a.smithi114.stdout: capacity : 536870912 2024-04-23T10:16:58.271 INFO:journalctl@ceph.mon.a.smithi114.stdout: num_shard_bits : 4 2024-04-23T10:16:58.272 INFO:journalctl@ceph.mon.a.smithi114.stdout: strict_capacity_limit : 0 2024-04-23T10:16:58.272 INFO:journalctl@ceph.mon.a.smithi114.stdout: high_pri_pool_ratio: 0.000 2024-04-23T10:16:58.272 INFO:journalctl@ceph.mon.a.smithi114.stdout: block_cache_compressed: (nil) 2024-04-23T10:16:58.272 INFO:journalctl@ceph.mon.a.smithi114.stdout: persistent_cache: (nil) 2024-04-23T10:16:58.272 INFO:journalctl@ceph.mon.a.smithi114.stdout: block_size: 4096 2024-04-23T10:16:58.272 INFO:journalctl@ceph.mon.a.smithi114.stdout: block_size_deviation: 10 2024-04-23T10:16:58.272 INFO:journalctl@ceph.mon.a.smithi114.stdout: block_restart_interval: 16 2024-04-23T10:16:58.272 INFO:journalctl@ceph.mon.a.smithi114.stdout: index_block_restart_interval: 1 2024-04-23T10:16:58.272 INFO:journalctl@ceph.mon.a.smithi114.stdout: metadata_block_size: 4096 2024-04-23T10:16:58.272 INFO:journalctl@ceph.mon.a.smithi114.stdout: partition_filters: 0 2024-04-23T10:16:58.272 INFO:journalctl@ceph.mon.a.smithi114.stdout: use_delta_encoding: 1 2024-04-23T10:16:58.272 INFO:journalctl@ceph.mon.a.smithi114.stdout: filter_policy: bloomfilter 2024-04-23T10:16:58.272 INFO:journalctl@ceph.mon.a.smithi114.stdout: whole_key_filtering: 1 2024-04-23T10:16:58.272 INFO:journalctl@ceph.mon.a.smithi114.stdout: verify_compression: 0 2024-04-23T10:16:58.273 INFO:journalctl@ceph.mon.a.smithi114.stdout: read_amp_bytes_per_bit: 0 2024-04-23T10:16:58.273 INFO:journalctl@ceph.mon.a.smithi114.stdout: format_version: 5 2024-04-23T10:16:58.273 INFO:journalctl@ceph.mon.a.smithi114.stdout: enable_index_compression: 1 2024-04-23T10:16:58.273 INFO:journalctl@ceph.mon.a.smithi114.stdout: block_align: 0 2024-04-23T10:16:58.273 INFO:journalctl@ceph.mon.a.smithi114.stdout: max_auto_readahead_size: 262144 2024-04-23T10:16:58.273 INFO:journalctl@ceph.mon.a.smithi114.stdout: prepopulate_block_cache: 0 2024-04-23T10:16:58.273 INFO:journalctl@ceph.mon.a.smithi114.stdout: initial_auto_readahead_size: 8192 2024-04-23T10:16:58.273 INFO:journalctl@ceph.mon.a.smithi114.stdout: num_file_reads_for_auto_readahead: 2 2024-04-23T10:16:58.273 INFO:journalctl@ceph.mon.a.smithi114.stdout:Apr 23 10:16:58 smithi114 ceph-mon[29255]: rocksdb: Options.write_buffer_size: 33554432 2024-04-23T10:16:58.273 INFO:journalctl@ceph.mon.a.smithi114.stdout:Apr 23 10:16:58 smithi114 ceph-mon[29255]: rocksdb: Options.max_write_buffer_number: 2 2024-04-23T10:16:58.273 INFO:journalctl@ceph.mon.a.smithi114.stdout:Apr 23 10:16:58 smithi114 ceph-mon[29255]: rocksdb: Options.compression: NoCompression 2024-04-23T10:16:58.273 INFO:journalctl@ceph.mon.a.smithi114.stdout:Apr 23 10:16:58 smithi114 ceph-mon[29255]: rocksdb: Options.bottommost_compression: Disabled 2024-04-23T10:16:58.273 INFO:journalctl@ceph.mon.a.smithi114.stdout:Apr 23 10:16:58 smithi114 ceph-mon[29255]: rocksdb: Options.prefix_extractor: nullptr 2024-04-23T10:16:58.273 INFO:journalctl@ceph.mon.a.smithi114.stdout:Apr 23 10:16:58 smithi114 ceph-mon[29255]: rocksdb: Options.memtable_insert_with_hint_prefix_extractor: nullptr 2024-04-23T10:16:58.273 INFO:journalctl@ceph.mon.a.smithi114.stdout:Apr 23 10:16:58 smithi114 ceph-mon[29255]: rocksdb: Options.num_levels: 7 2024-04-23T10:16:58.274 INFO:journalctl@ceph.mon.a.smithi114.stdout:Apr 23 10:16:58 smithi114 ceph-mon[29255]: rocksdb: Options.min_write_buffer_number_to_merge: 1 2024-04-23T10:16:58.274 INFO:journalctl@ceph.mon.a.smithi114.stdout:Apr 23 10:16:58 smithi114 ceph-mon[29255]: rocksdb: Options.max_write_buffer_number_to_maintain: 0 2024-04-23T10:16:58.274 INFO:journalctl@ceph.mon.a.smithi114.stdout:Apr 23 10:16:58 smithi114 ceph-mon[29255]: rocksdb: Options.max_write_buffer_size_to_maintain: 0 2024-04-23T10:16:58.274 INFO:journalctl@ceph.mon.a.smithi114.stdout:Apr 23 10:16:58 smithi114 ceph-mon[29255]: rocksdb: Options.bottommost_compression_opts.window_bits: -14 2024-04-23T10:16:58.274 INFO:journalctl@ceph.mon.a.smithi114.stdout:Apr 23 10:16:58 smithi114 ceph-mon[29255]: rocksdb: Options.bottommost_compression_opts.level: 32767 2024-04-23T10:16:58.274 INFO:journalctl@ceph.mon.a.smithi114.stdout:Apr 23 10:16:58 smithi114 ceph-mon[29255]: rocksdb: Options.bottommost_compression_opts.strategy: 0 2024-04-23T10:16:58.274 INFO:journalctl@ceph.mon.a.smithi114.stdout:Apr 23 10:16:58 smithi114 ceph-mon[29255]: rocksdb: Options.bottommost_compression_opts.max_dict_bytes: 0 2024-04-23T10:16:58.274 INFO:journalctl@ceph.mon.a.smithi114.stdout:Apr 23 10:16:58 smithi114 ceph-mon[29255]: rocksdb: Options.bottommost_compression_opts.zstd_max_train_bytes: 0 2024-04-23T10:16:58.274 INFO:journalctl@ceph.mon.a.smithi114.stdout:Apr 23 10:16:58 smithi114 ceph-mon[29255]: rocksdb: Options.bottommost_compression_opts.parallel_threads: 1 2024-04-23T10:16:58.274 INFO:journalctl@ceph.mon.a.smithi114.stdout:Apr 23 10:16:58 smithi114 ceph-mon[29255]: rocksdb: Options.bottommost_compression_opts.enabled: false 2024-04-23T10:16:58.274 INFO:journalctl@ceph.mon.a.smithi114.stdout:Apr 23 10:16:58 smithi114 ceph-mon[29255]: rocksdb: Options.bottommost_compression_opts.max_dict_buffer_bytes: 0 2024-04-23T10:16:58.274 INFO:journalctl@ceph.mon.a.smithi114.stdout:Apr 23 10:16:58 smithi114 ceph-mon[29255]: rocksdb: Options.bottommost_compression_opts.use_zstd_dict_trainer: true 2024-04-23T10:16:58.274 INFO:journalctl@ceph.mon.a.smithi114.stdout:Apr 23 10:16:58 smithi114 ceph-mon[29255]: rocksdb: Options.compression_opts.window_bits: -14 2024-04-23T10:16:58.274 INFO:journalctl@ceph.mon.a.smithi114.stdout:Apr 23 10:16:58 smithi114 ceph-mon[29255]: rocksdb: Options.compression_opts.level: 32767 2024-04-23T10:16:58.274 INFO:journalctl@ceph.mon.a.smithi114.stdout:Apr 23 10:16:58 smithi114 ceph-mon[29255]: rocksdb: Options.compression_opts.strategy: 0 2024-04-23T10:16:58.274 INFO:journalctl@ceph.mon.a.smithi114.stdout:Apr 23 10:16:58 smithi114 ceph-mon[29255]: rocksdb: Options.compression_opts.max_dict_bytes: 0 2024-04-23T10:16:58.275 INFO:journalctl@ceph.mon.a.smithi114.stdout:Apr 23 10:16:58 smithi114 ceph-mon[29255]: rocksdb: Options.compression_opts.zstd_max_train_bytes: 0 2024-04-23T10:16:58.275 INFO:journalctl@ceph.mon.a.smithi114.stdout:Apr 23 10:16:58 smithi114 ceph-mon[29255]: rocksdb: Options.compression_opts.use_zstd_dict_trainer: true 2024-04-23T10:16:58.275 INFO:journalctl@ceph.mon.a.smithi114.stdout:Apr 23 10:16:58 smithi114 ceph-mon[29255]: rocksdb: Options.compression_opts.parallel_threads: 1 2024-04-23T10:16:58.275 INFO:journalctl@ceph.mon.a.smithi114.stdout:Apr 23 10:16:58 smithi114 ceph-mon[29255]: rocksdb: Options.compression_opts.enabled: false 2024-04-23T10:16:58.275 INFO:journalctl@ceph.mon.a.smithi114.stdout:Apr 23 10:16:58 smithi114 ceph-mon[29255]: rocksdb: Options.compression_opts.max_dict_buffer_bytes: 0 2024-04-23T10:16:58.275 INFO:journalctl@ceph.mon.a.smithi114.stdout:Apr 23 10:16:58 smithi114 ceph-mon[29255]: rocksdb: Options.level0_file_num_compaction_trigger: 4 2024-04-23T10:16:58.275 INFO:journalctl@ceph.mon.a.smithi114.stdout:Apr 23 10:16:58 smithi114 ceph-mon[29255]: rocksdb: Options.level0_slowdown_writes_trigger: 20 2024-04-23T10:16:58.275 INFO:journalctl@ceph.mon.a.smithi114.stdout:Apr 23 10:16:58 smithi114 ceph-mon[29255]: rocksdb: Options.level0_stop_writes_trigger: 36 2024-04-23T10:16:58.275 INFO:journalctl@ceph.mon.a.smithi114.stdout:Apr 23 10:16:58 smithi114 ceph-mon[29255]: rocksdb: Options.target_file_size_base: 67108864 2024-04-23T10:16:58.275 INFO:journalctl@ceph.mon.a.smithi114.stdout:Apr 23 10:16:58 smithi114 ceph-mon[29255]: rocksdb: Options.target_file_size_multiplier: 1 2024-04-23T10:16:58.275 INFO:journalctl@ceph.mon.a.smithi114.stdout:Apr 23 10:16:58 smithi114 ceph-mon[29255]: rocksdb: Options.max_bytes_for_level_base: 268435456 2024-04-23T10:16:58.275 INFO:journalctl@ceph.mon.a.smithi114.stdout:Apr 23 10:16:58 smithi114 ceph-mon[29255]: rocksdb: Options.level_compaction_dynamic_level_bytes: 1 2024-04-23T10:16:58.275 INFO:journalctl@ceph.mon.a.smithi114.stdout:Apr 23 10:16:58 smithi114 ceph-mon[29255]: rocksdb: Options.max_bytes_for_level_multiplier: 10.000000 2024-04-23T10:16:58.275 INFO:journalctl@ceph.mon.a.smithi114.stdout:Apr 23 10:16:58 smithi114 ceph-mon[29255]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[0]: 1 2024-04-23T10:16:58.275 INFO:journalctl@ceph.mon.a.smithi114.stdout:Apr 23 10:16:58 smithi114 ceph-mon[29255]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[1]: 1 2024-04-23T10:16:58.276 INFO:journalctl@ceph.mon.a.smithi114.stdout:Apr 23 10:16:58 smithi114 ceph-mon[29255]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[2]: 1 2024-04-23T10:16:58.276 INFO:journalctl@ceph.mon.a.smithi114.stdout:Apr 23 10:16:58 smithi114 ceph-mon[29255]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[3]: 1 2024-04-23T10:16:58.276 INFO:journalctl@ceph.mon.a.smithi114.stdout:Apr 23 10:16:58 smithi114 ceph-mon[29255]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[4]: 1 2024-04-23T10:16:58.276 INFO:journalctl@ceph.mon.a.smithi114.stdout:Apr 23 10:16:58 smithi114 ceph-mon[29255]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[5]: 1 2024-04-23T10:16:58.276 INFO:journalctl@ceph.mon.a.smithi114.stdout:Apr 23 10:16:58 smithi114 ceph-mon[29255]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[6]: 1 2024-04-23T10:16:58.276 INFO:journalctl@ceph.mon.a.smithi114.stdout:Apr 23 10:16:58 smithi114 ceph-mon[29255]: rocksdb: Options.max_sequential_skip_in_iterations: 8 2024-04-23T10:16:58.276 INFO:journalctl@ceph.mon.a.smithi114.stdout:Apr 23 10:16:58 smithi114 ceph-mon[29255]: rocksdb: Options.max_compaction_bytes: 1677721600 2024-04-23T10:16:58.276 INFO:journalctl@ceph.mon.a.smithi114.stdout:Apr 23 10:16:58 smithi114 ceph-mon[29255]: rocksdb: Options.ignore_max_compaction_bytes_for_input: true 2024-04-23T10:16:58.276 INFO:journalctl@ceph.mon.a.smithi114.stdout:Apr 23 10:16:58 smithi114 ceph-mon[29255]: rocksdb: Options.arena_block_size: 1048576 2024-04-23T10:16:58.276 INFO:journalctl@ceph.mon.a.smithi114.stdout:Apr 23 10:16:58 smithi114 ceph-mon[29255]: rocksdb: Options.soft_pending_compaction_bytes_limit: 68719476736 2024-04-23T10:16:58.276 INFO:journalctl@ceph.mon.a.smithi114.stdout:Apr 23 10:16:58 smithi114 ceph-mon[29255]: rocksdb: Options.hard_pending_compaction_bytes_limit: 274877906944 2024-04-23T10:16:58.276 INFO:journalctl@ceph.mon.a.smithi114.stdout:Apr 23 10:16:58 smithi114 ceph-mon[29255]: rocksdb: Options.disable_auto_compactions: 0 2024-04-23T10:16:58.276 INFO:journalctl@ceph.mon.a.smithi114.stdout:Apr 23 10:16:58 smithi114 ceph-mon[29255]: rocksdb: Options.compaction_style: kCompactionStyleLevel 2024-04-23T10:16:58.276 INFO:journalctl@ceph.mon.a.smithi114.stdout:Apr 23 10:16:58 smithi114 ceph-mon[29255]: rocksdb: Options.compaction_pri: kMinOverlappingRatio 2024-04-23T10:16:58.276 INFO:journalctl@ceph.mon.a.smithi114.stdout:Apr 23 10:16:58 smithi114 ceph-mon[29255]: rocksdb: Options.compaction_options_universal.size_ratio: 1 2024-04-23T10:16:58.276 INFO:journalctl@ceph.mon.a.smithi114.stdout:Apr 23 10:16:58 smithi114 ceph-mon[29255]: rocksdb: Options.compaction_options_universal.min_merge_width: 2 2024-04-23T10:16:58.277 INFO:journalctl@ceph.mon.a.smithi114.stdout:Apr 23 10:16:58 smithi114 ceph-mon[29255]: rocksdb: Options.compaction_options_universal.max_merge_width: 4294967295 2024-04-23T10:16:58.277 INFO:journalctl@ceph.mon.a.smithi114.stdout:Apr 23 10:16:58 smithi114 ceph-mon[29255]: rocksdb: Options.compaction_options_universal.max_size_amplification_percent: 200 2024-04-23T10:16:58.277 INFO:journalctl@ceph.mon.a.smithi114.stdout:Apr 23 10:16:58 smithi114 ceph-mon[29255]: rocksdb: Options.compaction_options_universal.compression_size_percent: -1 2024-04-23T10:16:58.277 INFO:journalctl@ceph.mon.a.smithi114.stdout:Apr 23 10:16:58 smithi114 ceph-mon[29255]: rocksdb: Options.compaction_options_universal.stop_style: kCompactionStopStyleTotalSize 2024-04-23T10:16:58.277 INFO:journalctl@ceph.mon.a.smithi114.stdout:Apr 23 10:16:58 smithi114 ceph-mon[29255]: rocksdb: Options.compaction_options_fifo.max_table_files_size: 1073741824 2024-04-23T10:16:58.277 INFO:journalctl@ceph.mon.a.smithi114.stdout:Apr 23 10:16:58 smithi114 ceph-mon[29255]: rocksdb: Options.compaction_options_fifo.allow_compaction: 0 2024-04-23T10:16:58.277 INFO:journalctl@ceph.mon.a.smithi114.stdout:Apr 23 10:16:58 smithi114 ceph-mon[29255]: rocksdb: Options.table_properties_collectors: CompactOnDeletionCollector (Sliding window size = 32768 Deletion trigger = 16384 Deletion ratio = 0); 2024-04-23T10:16:58.277 INFO:journalctl@ceph.mon.a.smithi114.stdout:Apr 23 10:16:58 smithi114 ceph-mon[29255]: rocksdb: Options.inplace_update_support: 0 2024-04-23T10:16:58.277 INFO:journalctl@ceph.mon.a.smithi114.stdout:Apr 23 10:16:58 smithi114 ceph-mon[29255]: rocksdb: Options.inplace_update_num_locks: 10000 2024-04-23T10:16:58.277 INFO:journalctl@ceph.mon.a.smithi114.stdout:Apr 23 10:16:58 smithi114 ceph-mon[29255]: rocksdb: Options.memtable_prefix_bloom_size_ratio: 0.000000 2024-04-23T10:16:58.277 INFO:journalctl@ceph.mon.a.smithi114.stdout:Apr 23 10:16:58 smithi114 ceph-mon[29255]: rocksdb: Options.memtable_whole_key_filtering: 0 2024-04-23T10:16:58.277 INFO:journalctl@ceph.mon.a.smithi114.stdout:Apr 23 10:16:58 smithi114 ceph-mon[29255]: rocksdb: Options.memtable_huge_page_size: 0 2024-04-23T10:16:58.277 INFO:journalctl@ceph.mon.a.smithi114.stdout:Apr 23 10:16:58 smithi114 ceph-mon[29255]: rocksdb: Options.bloom_locality: 0 2024-04-23T10:16:58.277 INFO:journalctl@ceph.mon.a.smithi114.stdout:Apr 23 10:16:58 smithi114 ceph-mon[29255]: rocksdb: Options.max_successive_merges: 0 2024-04-23T10:16:58.277 INFO:journalctl@ceph.mon.a.smithi114.stdout:Apr 23 10:16:58 smithi114 ceph-mon[29255]: rocksdb: Options.optimize_filters_for_hits: 0 2024-04-23T10:16:58.278 INFO:journalctl@ceph.mon.a.smithi114.stdout:Apr 23 10:16:58 smithi114 ceph-mon[29255]: rocksdb: Options.paranoid_file_checks: 0 2024-04-23T10:16:58.278 INFO:journalctl@ceph.mon.a.smithi114.stdout:Apr 23 10:16:58 smithi114 ceph-mon[29255]: rocksdb: Options.force_consistency_checks: 1 2024-04-23T10:16:58.278 INFO:journalctl@ceph.mon.a.smithi114.stdout:Apr 23 10:16:58 smithi114 ceph-mon[29255]: rocksdb: Options.report_bg_io_stats: 0 2024-04-23T10:16:58.278 INFO:journalctl@ceph.mon.a.smithi114.stdout:Apr 23 10:16:58 smithi114 ceph-mon[29255]: rocksdb: Options.ttl: 2592000 2024-04-23T10:16:58.278 INFO:journalctl@ceph.mon.a.smithi114.stdout:Apr 23 10:16:58 smithi114 ceph-mon[29255]: rocksdb: Options.periodic_compaction_seconds: 0 2024-04-23T10:16:58.278 INFO:journalctl@ceph.mon.a.smithi114.stdout:Apr 23 10:16:58 smithi114 ceph-mon[29255]: rocksdb: Options.preclude_last_level_data_seconds: 0 2024-04-23T10:16:58.278 INFO:journalctl@ceph.mon.a.smithi114.stdout:Apr 23 10:16:58 smithi114 ceph-mon[29255]: rocksdb: Options.preserve_internal_time_seconds: 0 2024-04-23T10:16:58.278 INFO:journalctl@ceph.mon.a.smithi114.stdout:Apr 23 10:16:58 smithi114 ceph-mon[29255]: rocksdb: Options.enable_blob_files: false 2024-04-23T10:16:58.278 INFO:journalctl@ceph.mon.a.smithi114.stdout:Apr 23 10:16:58 smithi114 ceph-mon[29255]: rocksdb: Options.min_blob_size: 0 2024-04-23T10:16:58.278 INFO:journalctl@ceph.mon.a.smithi114.stdout:Apr 23 10:16:58 smithi114 ceph-mon[29255]: rocksdb: Options.blob_file_size: 268435456 2024-04-23T10:16:58.278 INFO:journalctl@ceph.mon.a.smithi114.stdout:Apr 23 10:16:58 smithi114 ceph-mon[29255]: rocksdb: Options.blob_compression_type: NoCompression 2024-04-23T10:16:58.278 INFO:journalctl@ceph.mon.a.smithi114.stdout:Apr 23 10:16:58 smithi114 ceph-mon[29255]: rocksdb: Options.enable_blob_garbage_collection: false 2024-04-23T10:16:58.278 INFO:journalctl@ceph.mon.a.smithi114.stdout:Apr 23 10:16:58 smithi114 ceph-mon[29255]: rocksdb: Options.blob_garbage_collection_age_cutoff: 0.250000 2024-04-23T10:16:58.278 INFO:journalctl@ceph.mon.a.smithi114.stdout:Apr 23 10:16:58 smithi114 ceph-mon[29255]: rocksdb: Options.blob_garbage_collection_force_threshold: 1.000000 2024-04-23T10:16:58.278 INFO:journalctl@ceph.mon.a.smithi114.stdout:Apr 23 10:16:58 smithi114 ceph-mon[29255]: rocksdb: Options.blob_compaction_readahead_size: 0 2024-04-23T10:16:58.279 INFO:journalctl@ceph.mon.a.smithi114.stdout:Apr 23 10:16:58 smithi114 ceph-mon[29255]: rocksdb: Options.blob_file_starting_level: 0 2024-04-23T10:16:58.279 INFO:journalctl@ceph.mon.a.smithi114.stdout:Apr 23 10:16:58 smithi114 ceph-mon[29255]: rocksdb: Options.experimental_mempurge_threshold: 0.000000 2024-04-23T10:16:58.279 INFO:journalctl@ceph.mon.a.smithi114.stdout:Apr 23 10:16:58 smithi114 ceph-mon[29255]: rocksdb: [db/version_set.cc:5566] Recovered from manifest file:/var/lib/ceph/mon/ceph-a/store.db/MANIFEST-000010 succeeded,manifest_file_number is 10, next_file_number is 12, last_sequence is 5, log_number is 5,prev_log_number is 0,max_column_family is 0,min_log_number_to_keep is 5 2024-04-23T10:16:58.279 INFO:journalctl@ceph.mon.a.smithi114.stdout:Apr 23 10:16:58 smithi114 ceph-mon[29255]: rocksdb: [db/version_set.cc:5581] Column family [default] (ID 0), log number is 5 2024-04-23T10:16:58.279 INFO:journalctl@ceph.mon.a.smithi114.stdout:Apr 23 10:16:58 smithi114 ceph-mon[29255]: rocksdb: [db/db_impl/db_impl_open.cc:539] DB ID: 37714189-1402-4dc8-8e27-61b9efcc4fb3 2024-04-23T10:16:58.279 INFO:journalctl@ceph.mon.a.smithi114.stdout:Apr 23 10:16:58 smithi114 ceph-mon[29255]: rocksdb: EVENT_LOG_v1 {"time_micros": 1713867418226554, "job": 1, "event": "recovery_started", "wal_files": [9]} 2024-04-23T10:16:58.279 INFO:journalctl@ceph.mon.a.smithi114.stdout:Apr 23 10:16:58 smithi114 ceph-mon[29255]: rocksdb: [db/db_impl/db_impl_open.cc:1043] Recovering log #9 mode 2 2024-04-23T10:16:58.279 INFO:journalctl@ceph.mon.a.smithi114.stdout:Apr 23 10:16:58 smithi114 ceph-mon[29255]: rocksdb: EVENT_LOG_v1 {"time_micros": 1713867418227578, "cf_name": "default", "job": 1, "event": "table_file_creation", "file_number": 13, "file_size": 100574, "file_checksum": "", "file_checksum_func_name": "Unknown", "smallest_seqno": 8, "largest_seqno": 328, "table_properties": {"data_size": 98459, "index_size": 307, "index_partitions": 0, "top_level_index_size": 0, "index_key_is_user_key": 1, "index_value_is_delta_encoded": 1, "filter_size": 773, "raw_key_size": 14264, "raw_average_key_size": 49, "raw_value_size": 90713, "raw_average_value_size": 316, "num_data_blocks": 12, "num_entries": 287, "num_filter_entries": 287, "num_deletions": 3, "num_merge_operands": 0, "num_range_deletions": 0, "format_version": 0, "fixed_key_len": 0, "filter_policy": "bloomfilter", "column_family_name": "default", "column_family_id": 0, "comparator": "leveldb.BytewiseComparator", "merge_operator": "", "prefix_extractor_name": "nullptr", "property_collectors": "[CompactOnDeletionCollector]", "compression": "NoCompression", "compression_options": "window_bits=-14; level=32767; strategy=0; max_dict_bytes=0; zstd_max_train_bytes=0; enabled=0; max_dict_buffer_bytes=0; use_zstd_dict_trainer=1; ", "creation_time": 1713867418, "oldest_key_time": 0, "file_creation_time": 0, "slow_compression_estimated_data_size": 0, "fast_compression_estimated_data_size": 0, "db_id": "37714189-1402-4dc8-8e27-61b9efcc4fb3", "db_session_id": "Q2C4SBXVK369NOD3T4L4", "orig_file_number": 13, "seqno_to_time_mapping": "N/A"}} 2024-04-23T10:16:58.279 INFO:journalctl@ceph.mon.a.smithi114.stdout:Apr 23 10:16:58 smithi114 ceph-mon[29255]: rocksdb: EVENT_LOG_v1 {"time_micros": 1713867418227659, "job": 1, "event": "recovery_finished"} 2024-04-23T10:16:58.279 INFO:journalctl@ceph.mon.a.smithi114.stdout:Apr 23 10:16:58 smithi114 ceph-mon[29255]: rocksdb: [db/version_set.cc:5047] Creating manifest 15 2024-04-23T10:16:58.279 INFO:journalctl@ceph.mon.a.smithi114.stdout:Apr 23 10:16:58 smithi114 ceph-mon[29255]: rocksdb: [file/delete_scheduler.cc:74] Deleted file /var/lib/ceph/mon/ceph-a/store.db/000009.log immediately, rate_bytes_per_sec 0, total_trash_size 0 max_trash_db_ratio 0.250000 2024-04-23T10:16:58.279 INFO:journalctl@ceph.mon.a.smithi114.stdout:Apr 23 10:16:58 smithi114 ceph-mon[29255]: rocksdb: [db/db_impl/db_impl_open.cc:1987] SstFileManager instance 0x562d44f82000 2024-04-23T10:16:58.279 INFO:journalctl@ceph.mon.a.smithi114.stdout:Apr 23 10:16:58 smithi114 ceph-mon[29255]: rocksdb: DB pointer 0x562d44f6c000 2024-04-23T10:16:58.279 INFO:journalctl@ceph.mon.a.smithi114.stdout:Apr 23 10:16:58 smithi114 ceph-mon[29255]: rocksdb: [db/db_impl/db_impl.cc:1109] ------- DUMPING STATS ------- 2024-04-23T10:16:58.279 INFO:journalctl@ceph.mon.a.smithi114.stdout:Apr 23 10:16:58 smithi114 ceph-mon[29255]: rocksdb: [db/db_impl/db_impl.cc:1111] 2024-04-23T10:16:58.280 INFO:journalctl@ceph.mon.a.smithi114.stdout: ** DB Stats ** 2024-04-23T10:16:58.280 INFO:journalctl@ceph.mon.a.smithi114.stdout: Uptime(secs): 0.0 total, 0.0 interval 2024-04-23T10:16:58.280 INFO:journalctl@ceph.mon.a.smithi114.stdout: Cumulative writes: 0 writes, 0 keys, 0 commit groups, 0.0 writes per commit group, ingest: 0.00 GB, 0.00 MB/s 2024-04-23T10:16:58.280 INFO:journalctl@ceph.mon.a.smithi114.stdout: Cumulative WAL: 0 writes, 0 syncs, 0.00 writes per sync, written: 0.00 GB, 0.00 MB/s 2024-04-23T10:16:58.280 INFO:journalctl@ceph.mon.a.smithi114.stdout: Cumulative stall: 00:00:0.000 H:M:S, 0.0 percent 2024-04-23T10:16:58.280 INFO:journalctl@ceph.mon.a.smithi114.stdout: Interval writes: 0 writes, 0 keys, 0 commit groups, 0.0 writes per commit group, ingest: 0.00 MB, 0.00 MB/s 2024-04-23T10:16:58.280 INFO:journalctl@ceph.mon.a.smithi114.stdout: Interval WAL: 0 writes, 0 syncs, 0.00 writes per sync, written: 0.00 GB, 0.00 MB/s 2024-04-23T10:16:58.280 INFO:journalctl@ceph.mon.a.smithi114.stdout: Interval stall: 00:00:0.000 H:M:S, 0.0 percent 2024-04-23T10:16:58.280 INFO:journalctl@ceph.mon.a.smithi114.stdout: 2024-04-23T10:16:58.280 INFO:journalctl@ceph.mon.a.smithi114.stdout: ** Compaction Stats [default] ** 2024-04-23T10:16:58.280 INFO:journalctl@ceph.mon.a.smithi114.stdout: Level Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop Rblob(GB) Wblob(GB) 2024-04-23T10:16:58.280 INFO:journalctl@ceph.mon.a.smithi114.stdout: ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ 2024-04-23T10:16:58.280 INFO:journalctl@ceph.mon.a.smithi114.stdout: L0 2/0 100.08 KB 0.5 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 147.8 0.00 0.00 1 0.001 0 0 0.0 0.0 2024-04-23T10:16:58.280 INFO:journalctl@ceph.mon.a.smithi114.stdout: Sum 2/0 100.08 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 147.8 0.00 0.00 1 0.001 0 0 0.0 0.0 2024-04-23T10:16:58.280 INFO:journalctl@ceph.mon.a.smithi114.stdout: Int 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 147.8 0.00 0.00 1 0.001 0 0 0.0 0.0 2024-04-23T10:16:58.281 INFO:journalctl@ceph.mon.a.smithi114.stdout: 2024-04-23T10:16:58.281 INFO:journalctl@ceph.mon.a.smithi114.stdout: ** Compaction Stats [default] ** 2024-04-23T10:16:58.281 INFO:journalctl@ceph.mon.a.smithi114.stdout: Priority Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop Rblob(GB) Wblob(GB) 2024-04-23T10:16:58.281 INFO:journalctl@ceph.mon.a.smithi114.stdout: --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- 2024-04-23T10:16:58.281 INFO:journalctl@ceph.mon.a.smithi114.stdout: User 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 147.8 0.00 0.00 1 0.001 0 0 0.0 0.0 2024-04-23T10:16:58.281 INFO:journalctl@ceph.mon.a.smithi114.stdout: 2024-04-23T10:16:58.281 INFO:journalctl@ceph.mon.a.smithi114.stdout: Blob file count: 0, total size: 0.0 GB, garbage size: 0.0 GB, space amp: 0.0 2024-04-23T10:16:58.281 INFO:journalctl@ceph.mon.a.smithi114.stdout: 2024-04-23T10:16:58.281 INFO:journalctl@ceph.mon.a.smithi114.stdout: Uptime(secs): 0.0 total, 0.0 interval 2024-04-23T10:16:58.281 INFO:journalctl@ceph.mon.a.smithi114.stdout: Flush(GB): cumulative 0.000, interval 0.000 2024-04-23T10:16:58.281 INFO:journalctl@ceph.mon.a.smithi114.stdout: AddFile(GB): cumulative 0.000, interval 0.000 2024-04-23T10:16:58.281 INFO:journalctl@ceph.mon.a.smithi114.stdout: AddFile(Total Files): cumulative 0, interval 0 2024-04-23T10:16:58.281 INFO:journalctl@ceph.mon.a.smithi114.stdout: AddFile(L0 Files): cumulative 0, interval 0 2024-04-23T10:16:58.281 INFO:journalctl@ceph.mon.a.smithi114.stdout: AddFile(Keys): cumulative 0, interval 0 2024-04-23T10:16:58.281 INFO:journalctl@ceph.mon.a.smithi114.stdout: Cumulative compaction: 0.00 GB write, 32.01 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds 2024-04-23T10:16:58.282 INFO:journalctl@ceph.mon.a.smithi114.stdout: Interval compaction: 0.00 GB write, 32.01 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds 2024-04-23T10:16:58.282 INFO:journalctl@ceph.mon.a.smithi114.stdout: Stalls(count): 0 level0_slowdown, 0 level0_slowdown_with_compaction, 0 level0_numfiles, 0 level0_numfiles_with_compaction, 0 stop for pending_compaction_bytes, 0 slowdown for pending_compaction_bytes, 0 memtable_compaction, 0 memtable_slowdown, interval 0 total count 2024-04-23T10:16:58.282 INFO:journalctl@ceph.mon.a.smithi114.stdout: Block cache BinnedLRUCache@0x562d44e6d1f0#2 capacity: 512.00 MB usage: 1.52 KB table_size: 0 occupancy: 18446744073709551615 collections: 1 last_copies: 0 last_secs: 9e-06 secs_since: 0 2024-04-23T10:16:58.282 INFO:journalctl@ceph.mon.a.smithi114.stdout: Block cache entry stats(count,size,portion): FilterBlock(2,1.02 KB,0.000193715%) IndexBlock(2,0.50 KB,9.53674e-05%) Misc(1,0.00 KB,0%) 2024-04-23T10:16:58.282 INFO:journalctl@ceph.mon.a.smithi114.stdout: 2024-04-23T10:16:58.282 INFO:journalctl@ceph.mon.a.smithi114.stdout: ** File Read Latency Histogram By Level [default] ** 2024-04-23T10:16:58.282 INFO:journalctl@ceph.mon.a.smithi114.stdout:Apr 23 10:16:58 smithi114 ceph-mon[29255]: starting mon.a rank 0 at public addrs [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] at bind addrs [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] mon_data /var/lib/ceph/mon/ceph-a fsid aa240572-0159-11ef-bc93-c7b262605968 2024-04-23T10:16:58.282 INFO:journalctl@ceph.mon.a.smithi114.stdout:Apr 23 10:16:58 smithi114 ceph-mon[29255]: mon.a@-1(???) e1 preinit fsid aa240572-0159-11ef-bc93-c7b262605968 2024-04-23T10:16:58.282 INFO:journalctl@ceph.mon.a.smithi114.stdout:Apr 23 10:16:58 smithi114 ceph-mon[29255]: mon.a@-1(???).mds e1 new map 2024-04-23T10:16:58.282 INFO:journalctl@ceph.mon.a.smithi114.stdout:Apr 23 10:16:58 smithi114 ceph-mon[29255]: mon.a@-1(???).mds e1 print_map 2024-04-23T10:16:58.282 INFO:journalctl@ceph.mon.a.smithi114.stdout: e1 2024-04-23T10:16:58.282 INFO:journalctl@ceph.mon.a.smithi114.stdout: enable_multiple, ever_enabled_multiple: 1,1 2024-04-23T10:16:58.282 INFO:journalctl@ceph.mon.a.smithi114.stdout: default compat: compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,8=no anchor table,9=file layout v2,10=snaprealm v2,11=minor log segments,12=quiesce subvolumes} 2024-04-23T10:16:58.283 INFO:journalctl@ceph.mon.a.smithi114.stdout: legacy client fscid: -1 2024-04-23T10:16:58.283 INFO:journalctl@ceph.mon.a.smithi114.stdout: 2024-04-23T10:16:58.283 INFO:journalctl@ceph.mon.a.smithi114.stdout: No filesystems configured 2024-04-23T10:16:58.283 INFO:journalctl@ceph.mon.a.smithi114.stdout:Apr 23 10:16:58 smithi114 ceph-mon[29255]: mon.a@-1(???).osd e1 crush map has features 3314932999778484224, adjusting msgr requires 2024-04-23T10:16:58.283 INFO:journalctl@ceph.mon.a.smithi114.stdout:Apr 23 10:16:58 smithi114 ceph-mon[29255]: mon.a@-1(???).osd e1 crush map has features 288514050185494528, adjusting msgr requires 2024-04-23T10:16:58.283 INFO:journalctl@ceph.mon.a.smithi114.stdout:Apr 23 10:16:58 smithi114 ceph-mon[29255]: mon.a@-1(???).osd e1 crush map has features 288514050185494528, adjusting msgr requires 2024-04-23T10:16:58.283 INFO:journalctl@ceph.mon.a.smithi114.stdout:Apr 23 10:16:58 smithi114 ceph-mon[29255]: mon.a@-1(???).osd e1 crush map has features 288514050185494528, adjusting msgr requires 2024-04-23T10:16:58.283 INFO:journalctl@ceph.mon.a.smithi114.stdout:Apr 23 10:16:58 smithi114 ceph-mon[29255]: mon.a@-1(???).paxosservice(auth 1..2) refresh upgraded, format 0 -> 3 2024-04-23T10:16:58.283 INFO:journalctl@ceph.mon.a.smithi114.stdout:Apr 23 10:16:58 smithi114 bash[29236]: 3c977ab8b867b227b23fcd666ab2002050abf6e75f25f3bb9a7d96eb2b2e973b 2024-04-23T10:16:58.283 INFO:journalctl@ceph.mon.a.smithi114.stdout:Apr 23 10:16:58 smithi114 systemd[1]: Started Ceph mon.a for aa240572-0159-11ef-bc93-c7b262605968. 2024-04-23T10:16:58.283 INFO:teuthology.orchestra.run.smithi114.stdout:Setting public_network to 172.21.0.0/20 in mon config section 2024-04-23T10:16:58.513 INFO:journalctl@ceph.mon.a.smithi114.stdout:Apr 23 10:16:58 smithi114 ceph-mon[29255]: mon.a is new leader, mons a in quorum (ranks 0) 2024-04-23T10:16:58.513 INFO:journalctl@ceph.mon.a.smithi114.stdout:Apr 23 10:16:58 smithi114 ceph-mon[29255]: monmap epoch 1 2024-04-23T10:16:58.513 INFO:journalctl@ceph.mon.a.smithi114.stdout:Apr 23 10:16:58 smithi114 ceph-mon[29255]: fsid aa240572-0159-11ef-bc93-c7b262605968 2024-04-23T10:16:58.513 INFO:journalctl@ceph.mon.a.smithi114.stdout:Apr 23 10:16:58 smithi114 ceph-mon[29255]: last_changed 2024-04-23T10:16:50.385874+0000 2024-04-23T10:16:58.513 INFO:journalctl@ceph.mon.a.smithi114.stdout:Apr 23 10:16:58 smithi114 ceph-mon[29255]: created 2024-04-23T10:16:50.385874+0000 2024-04-23T10:16:58.513 INFO:journalctl@ceph.mon.a.smithi114.stdout:Apr 23 10:16:58 smithi114 ceph-mon[29255]: min_mon_release 19 (squid) 2024-04-23T10:16:58.513 INFO:journalctl@ceph.mon.a.smithi114.stdout:Apr 23 10:16:58 smithi114 ceph-mon[29255]: election_strategy: 1 2024-04-23T10:16:58.514 INFO:journalctl@ceph.mon.a.smithi114.stdout:Apr 23 10:16:58 smithi114 ceph-mon[29255]: 0: [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] mon.a 2024-04-23T10:16:58.514 INFO:journalctl@ceph.mon.a.smithi114.stdout:Apr 23 10:16:58 smithi114 ceph-mon[29255]: fsmap 2024-04-23T10:16:58.514 INFO:journalctl@ceph.mon.a.smithi114.stdout:Apr 23 10:16:58 smithi114 ceph-mon[29255]: osdmap e1: 0 total, 0 up, 0 in 2024-04-23T10:16:58.514 INFO:journalctl@ceph.mon.a.smithi114.stdout:Apr 23 10:16:58 smithi114 ceph-mon[29255]: mgrmap e1: no daemons active 2024-04-23T10:16:59.383 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-04-23T10:16:58.749+0000 7f93df099700 1 Processor -- start 2024-04-23T10:16:59.384 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-04-23T10:16:58.749+0000 7f93df099700 1 -- start start 2024-04-23T10:16:59.384 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-04-23T10:16:58.749+0000 7f93df099700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f93d8108550 0x7f93d8108970 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T10:16:59.384 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-04-23T10:16:58.749+0000 7f93df099700 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x7f93d8108f40 con 0x7f93d8108550 2024-04-23T10:16:59.384 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-04-23T10:16:58.749+0000 7f93dce35700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f93d8108550 0x7f93d8108970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T10:16:59.384 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-04-23T10:16:58.750+0000 7f93dce35700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f93d8108550 0x7f93d8108970 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.114:57098/0 (socket says 172.21.15.114:57098) 2024-04-23T10:16:59.384 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-04-23T10:16:58.750+0000 7f93dce35700 1 -- 172.21.15.114:0/1043686264 learned_addr learned my addr 172.21.15.114:0/1043686264 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-04-23T10:16:59.384 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-04-23T10:16:58.750+0000 7f93dce35700 1 -- 172.21.15.114:0/1043686264 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f93d8109700 con 0x7f93d8108550 2024-04-23T10:16:59.384 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-04-23T10:16:58.750+0000 7f93dce35700 1 --2- 172.21.15.114:0/1043686264 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f93d8108550 0x7f93d8108970 secure :-1 s=READY pgs=1 cs=0 l=1 rev1=1 crypto rx=0x7f93cc0098a0 tx=0x7f93cc009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f5850c450dc0a8ec server_cookie=0 in_seq=0 out_seq=0 2024-04-23T10:16:59.384 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-04-23T10:16:58.750+0000 7f93d77fe700 1 -- 172.21.15.114:0/1043686264 <== mon.0 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f93cc004160 con 0x7f93d8108550 2024-04-23T10:16:59.385 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-04-23T10:16:58.750+0000 7f93d77fe700 1 -- 172.21.15.114:0/1043686264 <== mon.0 v2:172.21.15.114:3300/0 2 ==== config(26 keys) ==== 1070+0+0 (secure 0 0 0) 0x7f93cc004300 con 0x7f93d8108550 2024-04-23T10:16:59.385 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-04-23T10:16:58.750+0000 7f93d77fe700 1 -- 172.21.15.114:0/1043686264 <== mon.0 v2:172.21.15.114:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f93cc011af0 con 0x7f93d8108550 2024-04-23T10:16:59.385 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-04-23T10:16:58.751+0000 7f93df099700 1 -- 172.21.15.114:0/1043686264 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f93d8108550 msgr2=0x7f93d8108970 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T10:16:59.385 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-04-23T10:16:58.751+0000 7f93df099700 1 --2- 172.21.15.114:0/1043686264 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f93d8108550 0x7f93d8108970 secure :-1 s=READY pgs=1 cs=0 l=1 rev1=1 crypto rx=0x7f93cc0098a0 tx=0x7f93cc009bb0 comp rx=0 tx=0).stop 2024-04-23T10:16:59.385 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-04-23T10:16:58.751+0000 7f93df099700 1 -- 172.21.15.114:0/1043686264 shutdown_connections 2024-04-23T10:16:59.385 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-04-23T10:16:58.751+0000 7f93df099700 1 --2- 172.21.15.114:0/1043686264 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f93d8108550 0x7f93d8108970 unknown :-1 s=CLOSED pgs=1 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:16:59.385 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-04-23T10:16:58.751+0000 7f93df099700 1 -- 172.21.15.114:0/1043686264 >> 172.21.15.114:0/1043686264 conn(0x7f93d8103b10 msgr2=0x7f93d8105f30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T10:16:59.385 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-04-23T10:16:58.751+0000 7f93df099700 1 -- 172.21.15.114:0/1043686264 shutdown_connections 2024-04-23T10:16:59.385 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-04-23T10:16:58.751+0000 7f93df099700 1 -- 172.21.15.114:0/1043686264 wait complete. 2024-04-23T10:16:59.385 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-04-23T10:16:58.751+0000 7f93df099700 1 Processor -- start 2024-04-23T10:16:59.386 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-04-23T10:16:58.751+0000 7f93df099700 1 -- start start 2024-04-23T10:16:59.386 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-04-23T10:16:58.752+0000 7f93df099700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f93d8108550 0x7f93d807db80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T10:16:59.386 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-04-23T10:16:58.752+0000 7f93df099700 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x7f93d810cab0 con 0x7f93d8108550 2024-04-23T10:16:59.386 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-04-23T10:16:58.752+0000 7f93dce35700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f93d8108550 0x7f93d807db80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T10:16:59.386 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-04-23T10:16:58.752+0000 7f93dce35700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f93d8108550 0x7f93d807db80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.114:57108/0 (socket says 172.21.15.114:57108) 2024-04-23T10:16:59.386 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-04-23T10:16:58.752+0000 7f93dce35700 1 -- 172.21.15.114:0/2580794313 learned_addr learned my addr 172.21.15.114:0/2580794313 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-04-23T10:16:59.386 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-04-23T10:16:58.752+0000 7f93dce35700 1 -- 172.21.15.114:0/2580794313 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f93d807e0c0 con 0x7f93d8108550 2024-04-23T10:16:59.386 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-04-23T10:16:58.752+0000 7f93dce35700 1 --2- 172.21.15.114:0/2580794313 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f93d8108550 0x7f93d807db80 secure :-1 s=READY pgs=2 cs=0 l=1 rev1=1 crypto rx=0x7f93cc009fb0 tx=0x7f93cc011940 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T10:16:59.386 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-04-23T10:16:58.752+0000 7f93d5ffb700 1 -- 172.21.15.114:0/2580794313 <== mon.0 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f93cc0036a0 con 0x7f93d8108550 2024-04-23T10:16:59.387 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-04-23T10:16:58.752+0000 7f93df099700 1 -- 172.21.15.114:0/2580794313 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f93d807c2b0 con 0x7f93d8108550 2024-04-23T10:16:59.387 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-04-23T10:16:58.752+0000 7f93d5ffb700 1 -- 172.21.15.114:0/2580794313 <== mon.0 v2:172.21.15.114:3300/0 2 ==== config(26 keys) ==== 1070+0+0 (secure 0 0 0) 0x7f93cc01a430 con 0x7f93d8108550 2024-04-23T10:16:59.387 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-04-23T10:16:58.752+0000 7f93d5ffb700 1 -- 172.21.15.114:0/2580794313 <== mon.0 v2:172.21.15.114:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f93cc019550 con 0x7f93d8108550 2024-04-23T10:16:59.387 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-04-23T10:16:58.752+0000 7f93df099700 1 -- 172.21.15.114:0/2580794313 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f93d807c7f0 con 0x7f93d8108550 2024-04-23T10:16:59.387 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-04-23T10:16:58.753+0000 7f93d5ffb700 1 -- 172.21.15.114:0/2580794313 <== mon.0 v2:172.21.15.114:3300/0 4 ==== mgrmap(e 1) ==== 807+0+0 (secure 0 0 0) 0x7f93cc019be0 con 0x7f93d8108550 2024-04-23T10:16:59.387 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-04-23T10:16:58.753+0000 7f93d5ffb700 1 -- 172.21.15.114:0/2580794313 <== mon.0 v2:172.21.15.114:3300/0 5 ==== osd_map(1..1 src has 1..1) ==== 725+0+0 (secure 0 0 0) 0x7f93cc024730 con 0x7f93d8108550 2024-04-23T10:16:59.387 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-04-23T10:16:58.753+0000 7f93df099700 1 -- 172.21.15.114:0/2580794313 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f93d80640a0 con 0x7f93d8108550 2024-04-23T10:16:59.387 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-04-23T10:16:58.756+0000 7f93d5ffb700 1 -- 172.21.15.114:0/2580794313 <== mon.0 v2:172.21.15.114:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+73265 (secure 0 0 0) 0x7f93cc043e50 con 0x7f93d8108550 2024-04-23T10:16:59.387 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-04-23T10:16:58.824+0000 7f93df099700 1 -- 172.21.15.114:0/2580794313 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_command([{prefix=config set, name=public_network}] v 0) -- 0x7f93d810c8d0 con 0x7f93d8108550 2024-04-23T10:16:59.387 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-04-23T10:16:58.826+0000 7f93d5ffb700 1 -- 172.21.15.114:0/2580794313 <== mon.0 v2:172.21.15.114:3300/0 7 ==== mon_command_ack([{prefix=config set, name=public_network}]=0 v3)=0 v3) ==== 124+0+0 (secure 0 0 0) 0x7f93d810c8d0 con 0x7f93d8108550 2024-04-23T10:16:59.388 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-04-23T10:16:58.827+0000 7f93df099700 1 -- 172.21.15.114:0/2580794313 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f93d8108550 msgr2=0x7f93d807db80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T10:16:59.388 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-04-23T10:16:58.827+0000 7f93df099700 1 --2- 172.21.15.114:0/2580794313 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f93d8108550 0x7f93d807db80 secure :-1 s=READY pgs=2 cs=0 l=1 rev1=1 crypto rx=0x7f93cc009fb0 tx=0x7f93cc011940 comp rx=0 tx=0).stop 2024-04-23T10:16:59.388 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-04-23T10:16:58.827+0000 7f93df099700 1 -- 172.21.15.114:0/2580794313 shutdown_connections 2024-04-23T10:16:59.388 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-04-23T10:16:58.827+0000 7f93df099700 1 --2- 172.21.15.114:0/2580794313 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f93d8108550 0x7f93d807db80 unknown :-1 s=CLOSED pgs=2 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:16:59.388 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-04-23T10:16:58.827+0000 7f93df099700 1 -- 172.21.15.114:0/2580794313 >> 172.21.15.114:0/2580794313 conn(0x7f93d8103b10 msgr2=0x7f93d81046c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T10:16:59.388 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-04-23T10:16:58.827+0000 7f93df099700 1 -- 172.21.15.114:0/2580794313 shutdown_connections 2024-04-23T10:16:59.388 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-04-23T10:16:58.827+0000 7f93df099700 1 -- 172.21.15.114:0/2580794313 wait complete. 2024-04-23T10:16:59.388 INFO:teuthology.orchestra.run.smithi114.stdout:Wrote config to /etc/ceph/ceph.conf 2024-04-23T10:16:59.391 INFO:teuthology.orchestra.run.smithi114.stdout:Wrote keyring to /etc/ceph/ceph.client.admin.keyring 2024-04-23T10:16:59.391 INFO:teuthology.orchestra.run.smithi114.stdout:Creating mgr... 2024-04-23T10:16:59.391 INFO:teuthology.orchestra.run.smithi114.stdout:Verifying port 0.0.0.0:9283 ... 2024-04-23T10:16:59.392 INFO:teuthology.orchestra.run.smithi114.stdout:Verifying port 0.0.0.0:8765 ... 2024-04-23T10:16:59.659 INFO:teuthology.orchestra.run.smithi114.stdout:Non-zero exit code 1 from systemctl reset-failed ceph-aa240572-0159-11ef-bc93-c7b262605968@mgr.x 2024-04-23T10:16:59.659 INFO:teuthology.orchestra.run.smithi114.stdout:systemctl: stderr Failed to reset failed state of unit ceph-aa240572-0159-11ef-bc93-c7b262605968@mgr.x.service: Unit ceph-aa240572-0159-11ef-bc93-c7b262605968@mgr.x.service not loaded. 2024-04-23T10:16:59.806 INFO:teuthology.orchestra.run.smithi114.stdout:systemctl: stderr Created symlink /etc/systemd/system/ceph-aa240572-0159-11ef-bc93-c7b262605968.target.wants/ceph-aa240572-0159-11ef-bc93-c7b262605968@mgr.x.service → /etc/systemd/system/ceph-aa240572-0159-11ef-bc93-c7b262605968@.service. 2024-04-23T10:17:00.115 INFO:journalctl@ceph.mon.a.smithi114.stdout:Apr 23 10:16:59 smithi114 ceph-mon[29255]: from='client.? 172.21.15.114:0/2580794313' entity='client.admin' 2024-04-23T10:17:00.317 INFO:teuthology.orchestra.run.smithi114.stdout:Non-zero exit code 1 from systemctl is-enabled firewalld.service 2024-04-23T10:17:00.318 INFO:teuthology.orchestra.run.smithi114.stdout:systemctl: stdout disabled 2024-04-23T10:17:00.324 INFO:teuthology.orchestra.run.smithi114.stdout:Non-zero exit code 3 from systemctl is-active firewalld.service 2024-04-23T10:17:00.325 INFO:teuthology.orchestra.run.smithi114.stdout:systemctl: stdout inactive 2024-04-23T10:17:00.325 INFO:teuthology.orchestra.run.smithi114.stdout:firewalld.service is not enabled 2024-04-23T10:17:00.325 INFO:teuthology.orchestra.run.smithi114.stdout:Not possible to enable service . firewalld.service is not available 2024-04-23T10:17:00.333 INFO:teuthology.orchestra.run.smithi114.stdout:Non-zero exit code 1 from systemctl is-enabled firewalld.service 2024-04-23T10:17:00.333 INFO:teuthology.orchestra.run.smithi114.stdout:systemctl: stdout disabled 2024-04-23T10:17:00.340 INFO:teuthology.orchestra.run.smithi114.stdout:Non-zero exit code 3 from systemctl is-active firewalld.service 2024-04-23T10:17:00.340 INFO:teuthology.orchestra.run.smithi114.stdout:systemctl: stdout inactive 2024-04-23T10:17:00.340 INFO:teuthology.orchestra.run.smithi114.stdout:firewalld.service is not enabled 2024-04-23T10:17:00.340 INFO:teuthology.orchestra.run.smithi114.stdout:Not possible to open ports <[9283, 8765]>. firewalld.service is not available 2024-04-23T10:17:00.340 INFO:teuthology.orchestra.run.smithi114.stdout:Waiting for mgr to start... 2024-04-23T10:17:00.340 INFO:teuthology.orchestra.run.smithi114.stdout:Waiting for mgr... 2024-04-23T10:17:00.376 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Apr 23 10:17:00 smithi114 systemd[1]: Started Ceph mgr.x for aa240572-0159-11ef-bc93-c7b262605968. 2024-04-23T10:17:00.667 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Apr 23 10:17:00 smithi114 ceph-aa240572-0159-11ef-bc93-c7b262605968-mgr-x[29490]: 2024-04-23T10:17:00.376+0000 7f12a2283200 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2024-04-23T10:17:00.667 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Apr 23 10:17:00 smithi114 ceph-aa240572-0159-11ef-bc93-c7b262605968-mgr-x[29490]: 2024-04-23T10:17:00.561+0000 7f12a2283200 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2024-04-23T10:17:00.667 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Apr 23 10:17:00 smithi114 ceph-aa240572-0159-11ef-bc93-c7b262605968-mgr-x[29490]: 2024-04-23T10:17:00.667+0000 7f12a2283200 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2024-04-23T10:17:01.223 INFO:journalctl@ceph.mon.a.smithi114.stdout:Apr 23 10:17:00 smithi114 ceph-mon[29255]: from='client.? 172.21.15.114:0/621139547' entity='client.admin' cmd=[{"prefix": "status", "format": "json-pretty"}]: dispatch 2024-04-23T10:17:01.223 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Apr 23 10:17:00 smithi114 ceph-aa240572-0159-11ef-bc93-c7b262605968-mgr-x[29490]: 2024-04-23T10:17:00.991+0000 7f12a2283200 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2024-04-23T10:17:01.223 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Apr 23 10:17:01 smithi114 ceph-aa240572-0159-11ef-bc93-c7b262605968-mgr-x[29490]: 2024-04-23T10:17:01.075+0000 7f12a2283200 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2024-04-23T10:17:01.511 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stdout 2024-04-23T10:17:01.512 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stdout { 2024-04-23T10:17:01.512 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stdout "fsid": "aa240572-0159-11ef-bc93-c7b262605968", 2024-04-23T10:17:01.512 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stdout "health": { 2024-04-23T10:17:01.512 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stdout "status": "HEALTH_OK", 2024-04-23T10:17:01.512 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stdout "checks": {}, 2024-04-23T10:17:01.512 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stdout "mutes": [] 2024-04-23T10:17:01.512 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stdout }, 2024-04-23T10:17:01.512 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stdout "election_epoch": 5, 2024-04-23T10:17:01.512 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stdout "quorum": [ 2024-04-23T10:17:01.512 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stdout 0 2024-04-23T10:17:01.513 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stdout ], 2024-04-23T10:17:01.513 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stdout "quorum_names": [ 2024-04-23T10:17:01.513 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stdout "a" 2024-04-23T10:17:01.513 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stdout ], 2024-04-23T10:17:01.513 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stdout "quorum_age": 2, 2024-04-23T10:17:01.513 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stdout "monmap": { 2024-04-23T10:17:01.513 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-04-23T10:17:01.513 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stdout "min_mon_release_name": "squid", 2024-04-23T10:17:01.513 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stdout "num_mons": 1 2024-04-23T10:17:01.513 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stdout }, 2024-04-23T10:17:01.513 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stdout "osdmap": { 2024-04-23T10:17:01.514 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-04-23T10:17:01.514 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stdout "num_osds": 0, 2024-04-23T10:17:01.514 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stdout "num_up_osds": 0, 2024-04-23T10:17:01.514 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stdout "osd_up_since": 0, 2024-04-23T10:17:01.514 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stdout "num_in_osds": 0, 2024-04-23T10:17:01.514 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stdout "osd_in_since": 0, 2024-04-23T10:17:01.514 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stdout "num_remapped_pgs": 0 2024-04-23T10:17:01.514 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stdout }, 2024-04-23T10:17:01.514 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stdout "pgmap": { 2024-04-23T10:17:01.514 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stdout "pgs_by_state": [], 2024-04-23T10:17:01.515 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stdout "num_pgs": 0, 2024-04-23T10:17:01.515 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stdout "num_pools": 0, 2024-04-23T10:17:01.515 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stdout "num_objects": 0, 2024-04-23T10:17:01.515 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stdout "data_bytes": 0, 2024-04-23T10:17:01.515 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stdout "bytes_used": 0, 2024-04-23T10:17:01.515 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stdout "bytes_avail": 0, 2024-04-23T10:17:01.515 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stdout "bytes_total": 0 2024-04-23T10:17:01.515 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stdout }, 2024-04-23T10:17:01.515 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stdout "fsmap": { 2024-04-23T10:17:01.515 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-04-23T10:17:01.516 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stdout "by_rank": [], 2024-04-23T10:17:01.516 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stdout "up:standby": 0 2024-04-23T10:17:01.516 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stdout }, 2024-04-23T10:17:01.516 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stdout "mgrmap": { 2024-04-23T10:17:01.516 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stdout "available": false, 2024-04-23T10:17:01.516 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stdout "num_standbys": 0, 2024-04-23T10:17:01.516 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stdout "modules": [ 2024-04-23T10:17:01.516 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stdout "iostat", 2024-04-23T10:17:01.516 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stdout "nfs", 2024-04-23T10:17:01.516 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stdout "restful" 2024-04-23T10:17:01.516 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stdout ], 2024-04-23T10:17:01.517 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stdout "services": {} 2024-04-23T10:17:01.517 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stdout }, 2024-04-23T10:17:01.517 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stdout "servicemap": { 2024-04-23T10:17:01.517 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-04-23T10:17:01.517 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stdout "modified": "2024-04-23T10:16:53.351596+0000", 2024-04-23T10:17:01.517 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stdout "services": {} 2024-04-23T10:17:01.517 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stdout }, 2024-04-23T10:17:01.517 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stdout "progress_events": {} 2024-04-23T10:17:01.517 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stdout } 2024-04-23T10:17:01.517 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-04-23T10:17:00.791+0000 7f9331a7c700 1 Processor -- start 2024-04-23T10:17:01.517 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-04-23T10:17:00.791+0000 7f9331a7c700 1 -- start start 2024-04-23T10:17:01.518 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-04-23T10:17:00.791+0000 7f9331a7c700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f932c109b00 0x7f932c109f20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T10:17:01.518 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-04-23T10:17:00.792+0000 7f9331a7c700 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x7f932c10a4f0 con 0x7f932c109b00 2024-04-23T10:17:01.518 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-04-23T10:17:00.792+0000 7f932b7fe700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f932c109b00 0x7f932c109f20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T10:17:01.518 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-04-23T10:17:00.792+0000 7f932b7fe700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f932c109b00 0x7f932c109f20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.114:57142/0 (socket says 172.21.15.114:57142) 2024-04-23T10:17:01.518 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-04-23T10:17:00.792+0000 7f932b7fe700 1 -- 172.21.15.114:0/1938442238 learned_addr learned my addr 172.21.15.114:0/1938442238 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-04-23T10:17:01.518 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-04-23T10:17:00.792+0000 7f932b7fe700 1 -- 172.21.15.114:0/1938442238 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f932c10a670 con 0x7f932c109b00 2024-04-23T10:17:01.518 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-04-23T10:17:00.792+0000 7f932b7fe700 1 --2- 172.21.15.114:0/1938442238 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f932c109b00 0x7f932c109f20 secure :-1 s=READY pgs=5 cs=0 l=1 rev1=1 crypto rx=0x7f931c0098a0 tx=0x7f931c009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=afb71dd3ff34dcb6 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T10:17:01.518 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-04-23T10:17:00.792+0000 7f932a7fc700 1 -- 172.21.15.114:0/1938442238 <== mon.0 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f931c004160 con 0x7f932c109b00 2024-04-23T10:17:01.518 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-04-23T10:17:00.792+0000 7f932a7fc700 1 -- 172.21.15.114:0/1938442238 <== mon.0 v2:172.21.15.114:3300/0 2 ==== config(26 keys) ==== 1070+0+0 (secure 0 0 0) 0x7f931c004300 con 0x7f932c109b00 2024-04-23T10:17:01.519 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-04-23T10:17:00.792+0000 7f932a7fc700 1 -- 172.21.15.114:0/1938442238 <== mon.0 v2:172.21.15.114:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f931c011af0 con 0x7f932c109b00 2024-04-23T10:17:01.519 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-04-23T10:17:00.793+0000 7f9331a7c700 1 -- 172.21.15.114:0/1938442238 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f932c109b00 msgr2=0x7f932c109f20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T10:17:01.519 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-04-23T10:17:00.793+0000 7f9331a7c700 1 --2- 172.21.15.114:0/1938442238 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f932c109b00 0x7f932c109f20 secure :-1 s=READY pgs=5 cs=0 l=1 rev1=1 crypto rx=0x7f931c0098a0 tx=0x7f931c009bb0 comp rx=0 tx=0).stop 2024-04-23T10:17:01.519 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-04-23T10:17:00.793+0000 7f9331a7c700 1 -- 172.21.15.114:0/1938442238 shutdown_connections 2024-04-23T10:17:01.519 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-04-23T10:17:00.793+0000 7f9331a7c700 1 --2- 172.21.15.114:0/1938442238 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f932c109b00 0x7f932c109f20 unknown :-1 s=CLOSED pgs=5 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:17:01.519 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-04-23T10:17:00.793+0000 7f9331a7c700 1 -- 172.21.15.114:0/1938442238 >> 172.21.15.114:0/1938442238 conn(0x7f932c102140 msgr2=0x7f932c104560 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T10:17:01.519 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-04-23T10:17:00.793+0000 7f9331a7c700 1 -- 172.21.15.114:0/1938442238 shutdown_connections 2024-04-23T10:17:01.519 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-04-23T10:17:00.793+0000 7f9331a7c700 1 -- 172.21.15.114:0/1938442238 wait complete. 2024-04-23T10:17:01.519 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-04-23T10:17:00.794+0000 7f9331a7c700 1 Processor -- start 2024-04-23T10:17:01.519 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-04-23T10:17:00.794+0000 7f9331a7c700 1 -- start start 2024-04-23T10:17:01.520 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-04-23T10:17:00.794+0000 7f9331a7c700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f932c109b00 0x7f932c19c430 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T10:17:01.520 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-04-23T10:17:00.794+0000 7f9331a7c700 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x7f932c107300 con 0x7f932c109b00 2024-04-23T10:17:01.520 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-04-23T10:17:00.794+0000 7f932b7fe700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f932c109b00 0x7f932c19c430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T10:17:01.520 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-04-23T10:17:00.794+0000 7f932b7fe700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f932c109b00 0x7f932c19c430 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.114:57158/0 (socket says 172.21.15.114:57158) 2024-04-23T10:17:01.520 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-04-23T10:17:00.794+0000 7f932b7fe700 1 -- 172.21.15.114:0/621139547 learned_addr learned my addr 172.21.15.114:0/621139547 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-04-23T10:17:01.520 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-04-23T10:17:00.794+0000 7f932b7fe700 1 -- 172.21.15.114:0/621139547 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f932c19c970 con 0x7f932c109b00 2024-04-23T10:17:01.520 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-04-23T10:17:00.795+0000 7f932b7fe700 1 --2- 172.21.15.114:0/621139547 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f932c109b00 0x7f932c19c430 secure :-1 s=READY pgs=6 cs=0 l=1 rev1=1 crypto rx=0x7f931c003f40 tx=0x7f931c011890 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T10:17:01.520 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-04-23T10:17:00.795+0000 7f9328ff9700 1 -- 172.21.15.114:0/621139547 <== mon.0 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f931c0038b0 con 0x7f932c109b00 2024-04-23T10:17:01.520 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-04-23T10:17:00.795+0000 7f9328ff9700 1 -- 172.21.15.114:0/621139547 <== mon.0 v2:172.21.15.114:3300/0 2 ==== config(26 keys) ==== 1070+0+0 (secure 0 0 0) 0x7f931c003a50 con 0x7f932c109b00 2024-04-23T10:17:01.520 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-04-23T10:17:00.795+0000 7f9331a7c700 1 -- 172.21.15.114:0/621139547 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f932c19cc60 con 0x7f932c109b00 2024-04-23T10:17:01.521 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-04-23T10:17:00.795+0000 7f9328ff9700 1 -- 172.21.15.114:0/621139547 <== mon.0 v2:172.21.15.114:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f931c01a5c0 con 0x7f932c109b00 2024-04-23T10:17:01.521 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-04-23T10:17:00.795+0000 7f9331a7c700 1 -- 172.21.15.114:0/621139547 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f932c19f8f0 con 0x7f932c109b00 2024-04-23T10:17:01.521 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-04-23T10:17:00.795+0000 7f9328ff9700 1 -- 172.21.15.114:0/621139547 <== mon.0 v2:172.21.15.114:3300/0 4 ==== mgrmap(e 1) ==== 807+0+0 (secure 0 0 0) 0x7f931c01ac50 con 0x7f932c109b00 2024-04-23T10:17:01.521 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-04-23T10:17:00.795+0000 7f9328ff9700 1 -- 172.21.15.114:0/621139547 <== mon.0 v2:172.21.15.114:3300/0 5 ==== osd_map(1..1 src has 1..1) ==== 725+0+0 (secure 0 0 0) 0x7f931c019700 con 0x7f932c109b00 2024-04-23T10:17:01.521 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-04-23T10:17:00.796+0000 7f9331a7c700 1 -- 172.21.15.114:0/621139547 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f932c10cad0 con 0x7f932c109b00 2024-04-23T10:17:01.521 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-04-23T10:17:00.798+0000 7f9328ff9700 1 -- 172.21.15.114:0/621139547 <== mon.0 v2:172.21.15.114:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+73265 (secure 0 0 0) 0x7f931c019900 con 0x7f932c109b00 2024-04-23T10:17:01.521 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-04-23T10:17:00.870+0000 7f9331a7c700 1 -- 172.21.15.114:0/621139547 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_command({"prefix": "status", "format": "json-pretty"} v 0) -- 0x7f932c19fb60 con 0x7f932c109b00 2024-04-23T10:17:01.521 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-04-23T10:17:00.871+0000 7f9328ff9700 1 -- 172.21.15.114:0/621139547 <== mon.0 v2:172.21.15.114:3300/0 7 ==== mon_command_ack([{"prefix": "status", "format": "json-pretty"}]=0 v0) ==== 79+0+1239 (secure 0 0 0) 0x7f931c024430 con 0x7f932c109b00 2024-04-23T10:17:01.521 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-04-23T10:17:00.872+0000 7f9331a7c700 1 -- 172.21.15.114:0/621139547 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f932c109b00 msgr2=0x7f932c19c430 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T10:17:01.521 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-04-23T10:17:00.872+0000 7f9331a7c700 1 --2- 172.21.15.114:0/621139547 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f932c109b00 0x7f932c19c430 secure :-1 s=READY pgs=6 cs=0 l=1 rev1=1 crypto rx=0x7f931c003f40 tx=0x7f931c011890 comp rx=0 tx=0).stop 2024-04-23T10:17:01.522 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-04-23T10:17:00.872+0000 7f9331a7c700 1 -- 172.21.15.114:0/621139547 shutdown_connections 2024-04-23T10:17:01.522 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-04-23T10:17:00.872+0000 7f9331a7c700 1 --2- 172.21.15.114:0/621139547 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f932c109b00 0x7f932c19c430 unknown :-1 s=CLOSED pgs=6 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:17:01.522 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-04-23T10:17:00.872+0000 7f9331a7c700 1 -- 172.21.15.114:0/621139547 >> 172.21.15.114:0/621139547 conn(0x7f932c102140 msgr2=0x7f932c102cd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T10:17:01.522 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-04-23T10:17:00.872+0000 7f9331a7c700 1 -- 172.21.15.114:0/621139547 shutdown_connections 2024-04-23T10:17:01.522 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-04-23T10:17:00.872+0000 7f9331a7c700 1 -- 172.21.15.114:0/621139547 wait complete. 2024-04-23T10:17:01.522 INFO:teuthology.orchestra.run.smithi114.stdout:mgr not available, waiting (1/15)... 2024-04-23T10:17:02.601 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Apr 23 10:17:02 smithi114 ceph-aa240572-0159-11ef-bc93-c7b262605968-mgr-x[29490]: 2024-04-23T10:17:02.265+0000 7f12a2283200 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2024-04-23T10:17:02.601 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Apr 23 10:17:02 smithi114 ceph-aa240572-0159-11ef-bc93-c7b262605968-mgr-x[29490]: 2024-04-23T10:17:02.432+0000 7f12a2283200 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2024-04-23T10:17:02.601 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Apr 23 10:17:02 smithi114 ceph-aa240572-0159-11ef-bc93-c7b262605968-mgr-x[29490]: 2024-04-23T10:17:02.601+0000 7f12a2283200 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2024-04-23T10:17:03.174 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Apr 23 10:17:02 smithi114 ceph-aa240572-0159-11ef-bc93-c7b262605968-mgr-x[29490]: 2024-04-23T10:17:02.885+0000 7f12a2283200 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2024-04-23T10:17:03.174 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Apr 23 10:17:02 smithi114 ceph-aa240572-0159-11ef-bc93-c7b262605968-mgr-x[29490]: 2024-04-23T10:17:02.974+0000 7f12a2283200 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2024-04-23T10:17:03.174 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Apr 23 10:17:03 smithi114 ceph-aa240572-0159-11ef-bc93-c7b262605968-mgr-x[29490]: 2024-04-23T10:17:03.175+0000 7f12a2283200 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2024-04-23T10:17:03.504 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Apr 23 10:17:03 smithi114 ceph-aa240572-0159-11ef-bc93-c7b262605968-mgr-x[29490]: 2024-04-23T10:17:03.363+0000 7f12a2283200 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2024-04-23T10:17:04.254 INFO:journalctl@ceph.mon.a.smithi114.stdout:Apr 23 10:17:04 smithi114 ceph-mon[29255]: from='client.? 172.21.15.114:0/2553292565' entity='client.admin' cmd=[{"prefix": "status", "format": "json-pretty"}]: dispatch 2024-04-23T10:17:04.254 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Apr 23 10:17:03 smithi114 ceph-aa240572-0159-11ef-bc93-c7b262605968-mgr-x[29490]: 2024-04-23T10:17:03.976+0000 7f12a2283200 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2024-04-23T10:17:04.622 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stdout 2024-04-23T10:17:04.622 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stdout { 2024-04-23T10:17:04.622 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stdout "fsid": "aa240572-0159-11ef-bc93-c7b262605968", 2024-04-23T10:17:04.622 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stdout "health": { 2024-04-23T10:17:04.622 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stdout "status": "HEALTH_OK", 2024-04-23T10:17:04.622 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stdout "checks": {}, 2024-04-23T10:17:04.623 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stdout "mutes": [] 2024-04-23T10:17:04.623 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stdout }, 2024-04-23T10:17:04.623 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stdout "election_epoch": 5, 2024-04-23T10:17:04.623 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stdout "quorum": [ 2024-04-23T10:17:04.623 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stdout 0 2024-04-23T10:17:04.623 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stdout ], 2024-04-23T10:17:04.623 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stdout "quorum_names": [ 2024-04-23T10:17:04.623 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stdout "a" 2024-04-23T10:17:04.623 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stdout ], 2024-04-23T10:17:04.623 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stdout "quorum_age": 5, 2024-04-23T10:17:04.623 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stdout "monmap": { 2024-04-23T10:17:04.623 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-04-23T10:17:04.623 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stdout "min_mon_release_name": "squid", 2024-04-23T10:17:04.623 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stdout "num_mons": 1 2024-04-23T10:17:04.623 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stdout }, 2024-04-23T10:17:04.624 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stdout "osdmap": { 2024-04-23T10:17:04.624 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-04-23T10:17:04.624 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stdout "num_osds": 0, 2024-04-23T10:17:04.624 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stdout "num_up_osds": 0, 2024-04-23T10:17:04.624 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stdout "osd_up_since": 0, 2024-04-23T10:17:04.624 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stdout "num_in_osds": 0, 2024-04-23T10:17:04.624 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stdout "osd_in_since": 0, 2024-04-23T10:17:04.624 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stdout "num_remapped_pgs": 0 2024-04-23T10:17:04.624 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stdout }, 2024-04-23T10:17:04.624 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stdout "pgmap": { 2024-04-23T10:17:04.624 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stdout "pgs_by_state": [], 2024-04-23T10:17:04.624 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stdout "num_pgs": 0, 2024-04-23T10:17:04.624 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stdout "num_pools": 0, 2024-04-23T10:17:04.624 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stdout "num_objects": 0, 2024-04-23T10:17:04.624 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stdout "data_bytes": 0, 2024-04-23T10:17:04.625 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stdout "bytes_used": 0, 2024-04-23T10:17:04.625 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stdout "bytes_avail": 0, 2024-04-23T10:17:04.625 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stdout "bytes_total": 0 2024-04-23T10:17:04.625 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stdout }, 2024-04-23T10:17:04.625 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stdout "fsmap": { 2024-04-23T10:17:04.625 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-04-23T10:17:04.625 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stdout "by_rank": [], 2024-04-23T10:17:04.625 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stdout "up:standby": 0 2024-04-23T10:17:04.625 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stdout }, 2024-04-23T10:17:04.625 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stdout "mgrmap": { 2024-04-23T10:17:04.625 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stdout "available": false, 2024-04-23T10:17:04.625 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stdout "num_standbys": 0, 2024-04-23T10:17:04.625 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stdout "modules": [ 2024-04-23T10:17:04.625 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stdout "iostat", 2024-04-23T10:17:04.625 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stdout "nfs", 2024-04-23T10:17:04.626 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stdout "restful" 2024-04-23T10:17:04.626 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stdout ], 2024-04-23T10:17:04.626 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stdout "services": {} 2024-04-23T10:17:04.626 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stdout }, 2024-04-23T10:17:04.626 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stdout "servicemap": { 2024-04-23T10:17:04.626 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-04-23T10:17:04.626 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stdout "modified": "2024-04-23T10:16:53.351596+0000", 2024-04-23T10:17:04.626 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stdout "services": {} 2024-04-23T10:17:04.626 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stdout }, 2024-04-23T10:17:04.626 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stdout "progress_events": {} 2024-04-23T10:17:04.626 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stdout } 2024-04-23T10:17:04.626 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-04-23T10:17:03.911+0000 7fc4ad2f2700 1 Processor -- start 2024-04-23T10:17:04.626 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-04-23T10:17:03.911+0000 7fc4ad2f2700 1 -- start start 2024-04-23T10:17:04.626 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-04-23T10:17:03.912+0000 7fc4ad2f2700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fc4a8108a10 0x7fc4a8108e30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T10:17:04.627 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-04-23T10:17:03.912+0000 7fc4ad2f2700 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x7fc4a8109400 con 0x7fc4a8108a10 2024-04-23T10:17:04.627 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-04-23T10:17:03.912+0000 7fc4a6ffd700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fc4a8108a10 0x7fc4a8108e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T10:17:04.627 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-04-23T10:17:03.912+0000 7fc4a6ffd700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fc4a8108a10 0x7fc4a8108e30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.114:57172/0 (socket says 172.21.15.114:57172) 2024-04-23T10:17:04.627 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-04-23T10:17:03.912+0000 7fc4a6ffd700 1 -- 172.21.15.114:0/2970064823 learned_addr learned my addr 172.21.15.114:0/2970064823 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-04-23T10:17:04.627 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-04-23T10:17:03.912+0000 7fc4a6ffd700 1 -- 172.21.15.114:0/2970064823 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc4a8107670 con 0x7fc4a8108a10 2024-04-23T10:17:04.627 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-04-23T10:17:03.912+0000 7fc4a6ffd700 1 --2- 172.21.15.114:0/2970064823 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fc4a8108a10 0x7fc4a8108e30 secure :-1 s=READY pgs=7 cs=0 l=1 rev1=1 crypto rx=0x7fc4980098a0 tx=0x7fc498009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=19dd324a2aad0538 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T10:17:04.627 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-04-23T10:17:03.913+0000 7fc4a5ffb700 1 -- 172.21.15.114:0/2970064823 <== mon.0 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fc498004160 con 0x7fc4a8108a10 2024-04-23T10:17:04.627 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-04-23T10:17:03.913+0000 7fc4a5ffb700 1 -- 172.21.15.114:0/2970064823 <== mon.0 v2:172.21.15.114:3300/0 2 ==== config(26 keys) ==== 1070+0+0 (secure 0 0 0) 0x7fc498004300 con 0x7fc4a8108a10 2024-04-23T10:17:04.627 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-04-23T10:17:03.913+0000 7fc4a5ffb700 1 -- 172.21.15.114:0/2970064823 <== mon.0 v2:172.21.15.114:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fc498011af0 con 0x7fc4a8108a10 2024-04-23T10:17:04.627 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-04-23T10:17:03.913+0000 7fc4ad2f2700 1 -- 172.21.15.114:0/2970064823 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fc4a8108a10 msgr2=0x7fc4a8108e30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T10:17:04.627 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-04-23T10:17:03.913+0000 7fc4ad2f2700 1 --2- 172.21.15.114:0/2970064823 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fc4a8108a10 0x7fc4a8108e30 secure :-1 s=READY pgs=7 cs=0 l=1 rev1=1 crypto rx=0x7fc4980098a0 tx=0x7fc498009bb0 comp rx=0 tx=0).stop 2024-04-23T10:17:04.627 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-04-23T10:17:03.913+0000 7fc4ad2f2700 1 -- 172.21.15.114:0/2970064823 shutdown_connections 2024-04-23T10:17:04.627 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-04-23T10:17:03.913+0000 7fc4ad2f2700 1 --2- 172.21.15.114:0/2970064823 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fc4a8108a10 0x7fc4a8108e30 unknown :-1 s=CLOSED pgs=7 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:17:04.627 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-04-23T10:17:03.913+0000 7fc4ad2f2700 1 -- 172.21.15.114:0/2970064823 >> 172.21.15.114:0/2970064823 conn(0x7fc4a81021a0 msgr2=0x7fc4a81045c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T10:17:04.628 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-04-23T10:17:03.913+0000 7fc4ad2f2700 1 -- 172.21.15.114:0/2970064823 shutdown_connections 2024-04-23T10:17:04.628 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-04-23T10:17:03.913+0000 7fc4ad2f2700 1 -- 172.21.15.114:0/2970064823 wait complete. 2024-04-23T10:17:04.628 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-04-23T10:17:03.914+0000 7fc4ad2f2700 1 Processor -- start 2024-04-23T10:17:04.628 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-04-23T10:17:03.914+0000 7fc4ad2f2700 1 -- start start 2024-04-23T10:17:04.628 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-04-23T10:17:03.914+0000 7fc4ad2f2700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fc4a8108a10 0x7fc4a8106fe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T10:17:04.628 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-04-23T10:17:03.914+0000 7fc4ad2f2700 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x7fc4a810ba30 con 0x7fc4a8108a10 2024-04-23T10:17:04.628 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-04-23T10:17:03.915+0000 7fc4a6ffd700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fc4a8108a10 0x7fc4a8106fe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T10:17:04.628 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-04-23T10:17:03.915+0000 7fc4a6ffd700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fc4a8108a10 0x7fc4a8106fe0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.114:57178/0 (socket says 172.21.15.114:57178) 2024-04-23T10:17:04.628 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-04-23T10:17:03.915+0000 7fc4a6ffd700 1 -- 172.21.15.114:0/2553292565 learned_addr learned my addr 172.21.15.114:0/2553292565 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-04-23T10:17:04.628 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-04-23T10:17:03.915+0000 7fc4a6ffd700 1 -- 172.21.15.114:0/2553292565 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc4a819f1b0 con 0x7fc4a8108a10 2024-04-23T10:17:04.628 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-04-23T10:17:03.915+0000 7fc4a6ffd700 1 --2- 172.21.15.114:0/2553292565 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fc4a8108a10 0x7fc4a8106fe0 secure :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0x7fc498003f40 tx=0x7fc498011890 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T10:17:04.628 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-04-23T10:17:03.915+0000 7fc48ffff700 1 -- 172.21.15.114:0/2553292565 <== mon.0 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fc4980038b0 con 0x7fc4a8108a10 2024-04-23T10:17:04.628 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-04-23T10:17:03.915+0000 7fc48ffff700 1 -- 172.21.15.114:0/2553292565 <== mon.0 v2:172.21.15.114:3300/0 2 ==== config(26 keys) ==== 1070+0+0 (secure 0 0 0) 0x7fc498003a50 con 0x7fc4a8108a10 2024-04-23T10:17:04.628 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-04-23T10:17:03.915+0000 7fc4ad2f2700 1 -- 172.21.15.114:0/2553292565 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fc4a819f380 con 0x7fc4a8108a10 2024-04-23T10:17:04.628 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-04-23T10:17:03.915+0000 7fc48ffff700 1 -- 172.21.15.114:0/2553292565 <== mon.0 v2:172.21.15.114:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fc49801a5c0 con 0x7fc4a8108a10 2024-04-23T10:17:04.629 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-04-23T10:17:03.915+0000 7fc4ad2f2700 1 -- 172.21.15.114:0/2553292565 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fc4a819f720 con 0x7fc4a8108a10 2024-04-23T10:17:04.629 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-04-23T10:17:03.915+0000 7fc48ffff700 1 -- 172.21.15.114:0/2553292565 <== mon.0 v2:172.21.15.114:3300/0 4 ==== mgrmap(e 1) ==== 807+0+0 (secure 0 0 0) 0x7fc49801ac50 con 0x7fc4a8108a10 2024-04-23T10:17:04.629 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-04-23T10:17:03.915+0000 7fc48ffff700 1 -- 172.21.15.114:0/2553292565 <== mon.0 v2:172.21.15.114:3300/0 5 ==== osd_map(1..1 src has 1..1) ==== 725+0+0 (secure 0 0 0) 0x7fc498019700 con 0x7fc4a8108a10 2024-04-23T10:17:04.629 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-04-23T10:17:03.916+0000 7fc4ad2f2700 1 -- 172.21.15.114:0/2553292565 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fc4a80640a0 con 0x7fc4a8108a10 2024-04-23T10:17:04.629 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-04-23T10:17:03.919+0000 7fc48ffff700 1 -- 172.21.15.114:0/2553292565 <== mon.0 v2:172.21.15.114:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+73265 (secure 0 0 0) 0x7fc498019900 con 0x7fc4a8108a10 2024-04-23T10:17:04.629 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-04-23T10:17:03.992+0000 7fc4ad2f2700 1 -- 172.21.15.114:0/2553292565 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_command({"prefix": "status", "format": "json-pretty"} v 0) -- 0x7fc4a819fb60 con 0x7fc4a8108a10 2024-04-23T10:17:04.629 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-04-23T10:17:03.992+0000 7fc48ffff700 1 -- 172.21.15.114:0/2553292565 <== mon.0 v2:172.21.15.114:3300/0 7 ==== mon_command_ack([{"prefix": "status", "format": "json-pretty"}]=0 v0) ==== 79+0+1239 (secure 0 0 0) 0x7fc498019ae0 con 0x7fc4a8108a10 2024-04-23T10:17:04.629 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-04-23T10:17:03.993+0000 7fc4ad2f2700 1 -- 172.21.15.114:0/2553292565 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fc4a8108a10 msgr2=0x7fc4a8106fe0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T10:17:04.629 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-04-23T10:17:03.993+0000 7fc4ad2f2700 1 --2- 172.21.15.114:0/2553292565 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fc4a8108a10 0x7fc4a8106fe0 secure :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0x7fc498003f40 tx=0x7fc498011890 comp rx=0 tx=0).stop 2024-04-23T10:17:04.629 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-04-23T10:17:03.993+0000 7fc4ad2f2700 1 -- 172.21.15.114:0/2553292565 shutdown_connections 2024-04-23T10:17:04.629 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-04-23T10:17:03.993+0000 7fc4ad2f2700 1 --2- 172.21.15.114:0/2553292565 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fc4a8108a10 0x7fc4a8106fe0 unknown :-1 s=CLOSED pgs=8 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:17:04.629 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-04-23T10:17:03.993+0000 7fc4ad2f2700 1 -- 172.21.15.114:0/2553292565 >> 172.21.15.114:0/2553292565 conn(0x7fc4a81021a0 msgr2=0x7fc4a8102c20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T10:17:04.629 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-04-23T10:17:03.993+0000 7fc4ad2f2700 1 -- 172.21.15.114:0/2553292565 shutdown_connections 2024-04-23T10:17:04.629 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-04-23T10:17:03.993+0000 7fc4ad2f2700 1 -- 172.21.15.114:0/2553292565 wait complete. 2024-04-23T10:17:04.630 INFO:teuthology.orchestra.run.smithi114.stdout:mgr not available, waiting (2/15)... 2024-04-23T10:17:04.708 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Apr 23 10:17:04 smithi114 ceph-aa240572-0159-11ef-bc93-c7b262605968-mgr-x[29490]: 2024-04-23T10:17:04.406+0000 7f12a2283200 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2024-04-23T10:17:04.708 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Apr 23 10:17:04 smithi114 ceph-aa240572-0159-11ef-bc93-c7b262605968-mgr-x[29490]: 2024-04-23T10:17:04.506+0000 7f12a2283200 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2024-04-23T10:17:04.708 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Apr 23 10:17:04 smithi114 ceph-aa240572-0159-11ef-bc93-c7b262605968-mgr-x[29490]: 2024-04-23T10:17:04.709+0000 7f12a2283200 -1 mgr[py] Module rgw has missing NOTIFY_TYPES member 2024-04-23T10:17:05.505 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Apr 23 10:17:05 smithi114 ceph-aa240572-0159-11ef-bc93-c7b262605968-mgr-x[29490]: 2024-04-23T10:17:05.225+0000 7f12a2283200 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2024-04-23T10:17:05.505 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Apr 23 10:17:05 smithi114 ceph-aa240572-0159-11ef-bc93-c7b262605968-mgr-x[29490]: 2024-04-23T10:17:05.316+0000 7f12a2283200 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2024-04-23T10:17:05.894 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Apr 23 10:17:05 smithi114 ceph-aa240572-0159-11ef-bc93-c7b262605968-mgr-x[29490]: 2024-04-23T10:17:05.555+0000 7f12a2283200 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2024-04-23T10:17:05.894 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Apr 23 10:17:05 smithi114 ceph-aa240572-0159-11ef-bc93-c7b262605968-mgr-x[29490]: 2024-04-23T10:17:05.687+0000 7f12a2283200 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2024-04-23T10:17:05.894 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Apr 23 10:17:05 smithi114 ceph-aa240572-0159-11ef-bc93-c7b262605968-mgr-x[29490]: 2024-04-23T10:17:05.781+0000 7f12a2283200 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2024-04-23T10:17:05.894 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Apr 23 10:17:05 smithi114 ceph-aa240572-0159-11ef-bc93-c7b262605968-mgr-x[29490]: 2024-04-23T10:17:05.895+0000 7f12a2283200 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2024-04-23T10:17:06.504 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Apr 23 10:17:06 smithi114 ceph-aa240572-0159-11ef-bc93-c7b262605968-mgr-x[29490]: 2024-04-23T10:17:06.196+0000 7f12a2283200 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2024-04-23T10:17:07.004 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Apr 23 10:17:06 smithi114 ceph-aa240572-0159-11ef-bc93-c7b262605968-mgr-x[29490]: 2024-04-23T10:17:06.797+0000 7f12a2283200 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2024-04-23T10:17:07.504 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Apr 23 10:17:07 smithi114 ceph-aa240572-0159-11ef-bc93-c7b262605968-mgr-x[29490]: 2024-04-23T10:17:07.017+0000 7f12a2283200 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2024-04-23T10:17:07.505 INFO:journalctl@ceph.mon.a.smithi114.stdout:Apr 23 10:17:07 smithi114 ceph-mon[29255]: Activating manager daemon x 2024-04-23T10:17:07.505 INFO:journalctl@ceph.mon.a.smithi114.stdout:Apr 23 10:17:07 smithi114 ceph-mon[29255]: mgrmap e2: x(active, starting, since 0.00298626s) 2024-04-23T10:17:07.505 INFO:journalctl@ceph.mon.a.smithi114.stdout:Apr 23 10:17:07 smithi114 ceph-mon[29255]: from='mgr.14100 172.21.15.114:0/3906464846' entity='mgr.x' cmd=[{"prefix": "mds metadata"}]: dispatch 2024-04-23T10:17:07.505 INFO:journalctl@ceph.mon.a.smithi114.stdout:Apr 23 10:17:07 smithi114 ceph-mon[29255]: from='mgr.14100 172.21.15.114:0/3906464846' entity='mgr.x' cmd=[{"prefix": "osd metadata"}]: dispatch 2024-04-23T10:17:07.505 INFO:journalctl@ceph.mon.a.smithi114.stdout:Apr 23 10:17:07 smithi114 ceph-mon[29255]: from='mgr.14100 172.21.15.114:0/3906464846' entity='mgr.x' cmd=[{"prefix": "mon metadata"}]: dispatch 2024-04-23T10:17:07.505 INFO:journalctl@ceph.mon.a.smithi114.stdout:Apr 23 10:17:07 smithi114 ceph-mon[29255]: from='mgr.14100 172.21.15.114:0/3906464846' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-04-23T10:17:07.505 INFO:journalctl@ceph.mon.a.smithi114.stdout:Apr 23 10:17:07 smithi114 ceph-mon[29255]: from='mgr.14100 172.21.15.114:0/3906464846' entity='mgr.x' cmd=[{"prefix": "mgr metadata", "who": "x", "id": "x"}]: dispatch 2024-04-23T10:17:07.505 INFO:journalctl@ceph.mon.a.smithi114.stdout:Apr 23 10:17:07 smithi114 ceph-mon[29255]: Manager daemon x is now available 2024-04-23T10:17:07.506 INFO:journalctl@ceph.mon.a.smithi114.stdout:Apr 23 10:17:07 smithi114 ceph-mon[29255]: from='mgr.14100 172.21.15.114:0/3906464846' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2024-04-23T10:17:07.506 INFO:journalctl@ceph.mon.a.smithi114.stdout:Apr 23 10:17:07 smithi114 ceph-mon[29255]: from='mgr.14100 172.21.15.114:0/3906464846' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2024-04-23T10:17:07.506 INFO:journalctl@ceph.mon.a.smithi114.stdout:Apr 23 10:17:07 smithi114 ceph-mon[29255]: from='mgr.14100 172.21.15.114:0/3906464846' entity='mgr.x' 2024-04-23T10:17:07.506 INFO:journalctl@ceph.mon.a.smithi114.stdout:Apr 23 10:17:07 smithi114 ceph-mon[29255]: from='mgr.14100 172.21.15.114:0/3906464846' entity='mgr.x' 2024-04-23T10:17:07.506 INFO:journalctl@ceph.mon.a.smithi114.stdout:Apr 23 10:17:07 smithi114 ceph-mon[29255]: from='mgr.14100 172.21.15.114:0/3906464846' entity='mgr.x' 2024-04-23T10:17:07.678 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stdout 2024-04-23T10:17:07.679 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stdout { 2024-04-23T10:17:07.679 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stdout "fsid": "aa240572-0159-11ef-bc93-c7b262605968", 2024-04-23T10:17:07.679 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stdout "health": { 2024-04-23T10:17:07.679 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stdout "status": "HEALTH_OK", 2024-04-23T10:17:07.679 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stdout "checks": {}, 2024-04-23T10:17:07.679 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stdout "mutes": [] 2024-04-23T10:17:07.679 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stdout }, 2024-04-23T10:17:07.679 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stdout "election_epoch": 5, 2024-04-23T10:17:07.679 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stdout "quorum": [ 2024-04-23T10:17:07.679 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stdout 0 2024-04-23T10:17:07.679 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stdout ], 2024-04-23T10:17:07.679 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stdout "quorum_names": [ 2024-04-23T10:17:07.680 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stdout "a" 2024-04-23T10:17:07.680 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stdout ], 2024-04-23T10:17:07.680 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stdout "quorum_age": 8, 2024-04-23T10:17:07.680 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stdout "monmap": { 2024-04-23T10:17:07.680 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-04-23T10:17:07.680 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stdout "min_mon_release_name": "squid", 2024-04-23T10:17:07.680 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stdout "num_mons": 1 2024-04-23T10:17:07.680 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stdout }, 2024-04-23T10:17:07.680 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stdout "osdmap": { 2024-04-23T10:17:07.680 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-04-23T10:17:07.680 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stdout "num_osds": 0, 2024-04-23T10:17:07.680 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stdout "num_up_osds": 0, 2024-04-23T10:17:07.689 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stdout "osd_up_since": 0, 2024-04-23T10:17:07.689 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stdout "num_in_osds": 0, 2024-04-23T10:17:07.689 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stdout "osd_in_since": 0, 2024-04-23T10:17:07.689 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stdout "num_remapped_pgs": 0 2024-04-23T10:17:07.689 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stdout }, 2024-04-23T10:17:07.689 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stdout "pgmap": { 2024-04-23T10:17:07.689 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stdout "pgs_by_state": [], 2024-04-23T10:17:07.689 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stdout "num_pgs": 0, 2024-04-23T10:17:07.689 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stdout "num_pools": 0, 2024-04-23T10:17:07.689 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stdout "num_objects": 0, 2024-04-23T10:17:07.690 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stdout "data_bytes": 0, 2024-04-23T10:17:07.690 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stdout "bytes_used": 0, 2024-04-23T10:17:07.690 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stdout "bytes_avail": 0, 2024-04-23T10:17:07.690 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stdout "bytes_total": 0 2024-04-23T10:17:07.690 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stdout }, 2024-04-23T10:17:07.690 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stdout "fsmap": { 2024-04-23T10:17:07.690 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-04-23T10:17:07.690 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stdout "by_rank": [], 2024-04-23T10:17:07.690 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stdout "up:standby": 0 2024-04-23T10:17:07.690 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stdout }, 2024-04-23T10:17:07.690 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stdout "mgrmap": { 2024-04-23T10:17:07.690 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stdout "available": false, 2024-04-23T10:17:07.690 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stdout "num_standbys": 0, 2024-04-23T10:17:07.691 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stdout "modules": [ 2024-04-23T10:17:07.691 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stdout "iostat", 2024-04-23T10:17:07.691 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stdout "nfs", 2024-04-23T10:17:07.691 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stdout "restful" 2024-04-23T10:17:07.691 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stdout ], 2024-04-23T10:17:07.691 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stdout "services": {} 2024-04-23T10:17:07.691 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stdout }, 2024-04-23T10:17:07.691 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stdout "servicemap": { 2024-04-23T10:17:07.691 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-04-23T10:17:07.691 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stdout "modified": "2024-04-23T10:16:53.351596+0000", 2024-04-23T10:17:07.691 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stdout "services": {} 2024-04-23T10:17:07.691 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stdout }, 2024-04-23T10:17:07.691 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stdout "progress_events": {} 2024-04-23T10:17:07.691 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stdout } 2024-04-23T10:17:07.691 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-04-23T10:17:07.059+0000 7f8f7008c700 1 Processor -- start 2024-04-23T10:17:07.692 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-04-23T10:17:07.059+0000 7f8f7008c700 1 -- start start 2024-04-23T10:17:07.692 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-04-23T10:17:07.059+0000 7f8f7008c700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f8f68108530 0x7f8f68108950 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T10:17:07.692 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-04-23T10:17:07.059+0000 7f8f7008c700 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x7f8f68108f20 con 0x7f8f68108530 2024-04-23T10:17:07.692 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-04-23T10:17:07.059+0000 7f8f6de28700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f8f68108530 0x7f8f68108950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T10:17:07.692 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-04-23T10:17:07.059+0000 7f8f6de28700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f8f68108530 0x7f8f68108950 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.114:44842/0 (socket says 172.21.15.114:44842) 2024-04-23T10:17:07.692 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-04-23T10:17:07.059+0000 7f8f6de28700 1 -- 172.21.15.114:0/1738937547 learned_addr learned my addr 172.21.15.114:0/1738937547 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-04-23T10:17:07.692 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-04-23T10:17:07.059+0000 7f8f6de28700 1 -- 172.21.15.114:0/1738937547 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f8f68109730 con 0x7f8f68108530 2024-04-23T10:17:07.692 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-04-23T10:17:07.060+0000 7f8f6de28700 1 --2- 172.21.15.114:0/1738937547 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f8f68108530 0x7f8f68108950 secure :-1 s=READY pgs=16 cs=0 l=1 rev1=1 crypto rx=0x7f8f580098a0 tx=0x7f8f58009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=42740130122532a3 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T10:17:07.692 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-04-23T10:17:07.060+0000 7f8f6ce26700 1 -- 172.21.15.114:0/1738937547 <== mon.0 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f8f58004160 con 0x7f8f68108530 2024-04-23T10:17:07.692 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-04-23T10:17:07.060+0000 7f8f6ce26700 1 -- 172.21.15.114:0/1738937547 <== mon.0 v2:172.21.15.114:3300/0 2 ==== config(26 keys) ==== 1070+0+0 (secure 0 0 0) 0x7f8f58004300 con 0x7f8f68108530 2024-04-23T10:17:07.692 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-04-23T10:17:07.060+0000 7f8f6ce26700 1 -- 172.21.15.114:0/1738937547 <== mon.0 v2:172.21.15.114:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f8f58011af0 con 0x7f8f68108530 2024-04-23T10:17:07.692 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-04-23T10:17:07.060+0000 7f8f7008c700 1 -- 172.21.15.114:0/1738937547 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f8f68108530 msgr2=0x7f8f68108950 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T10:17:07.692 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-04-23T10:17:07.060+0000 7f8f7008c700 1 --2- 172.21.15.114:0/1738937547 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f8f68108530 0x7f8f68108950 secure :-1 s=READY pgs=16 cs=0 l=1 rev1=1 crypto rx=0x7f8f580098a0 tx=0x7f8f58009bb0 comp rx=0 tx=0).stop 2024-04-23T10:17:07.692 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-04-23T10:17:07.060+0000 7f8f7008c700 1 -- 172.21.15.114:0/1738937547 shutdown_connections 2024-04-23T10:17:07.693 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-04-23T10:17:07.060+0000 7f8f7008c700 1 --2- 172.21.15.114:0/1738937547 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f8f68108530 0x7f8f68108950 unknown :-1 s=CLOSED pgs=16 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:17:07.693 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-04-23T10:17:07.060+0000 7f8f7008c700 1 -- 172.21.15.114:0/1738937547 >> 172.21.15.114:0/1738937547 conn(0x7f8f68103b50 msgr2=0x7f8f68105f70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T10:17:07.693 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-04-23T10:17:07.060+0000 7f8f7008c700 1 -- 172.21.15.114:0/1738937547 shutdown_connections 2024-04-23T10:17:07.693 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-04-23T10:17:07.061+0000 7f8f7008c700 1 -- 172.21.15.114:0/1738937547 wait complete. 2024-04-23T10:17:07.693 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-04-23T10:17:07.061+0000 7f8f7008c700 1 Processor -- start 2024-04-23T10:17:07.693 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-04-23T10:17:07.061+0000 7f8f7008c700 1 -- start start 2024-04-23T10:17:07.693 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-04-23T10:17:07.061+0000 7f8f7008c700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f8f68108530 0x7f8f6819c480 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T10:17:07.693 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-04-23T10:17:07.061+0000 7f8f7008c700 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x7f8f6810cae0 con 0x7f8f68108530 2024-04-23T10:17:07.693 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-04-23T10:17:07.061+0000 7f8f6de28700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f8f68108530 0x7f8f6819c480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T10:17:07.693 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-04-23T10:17:07.061+0000 7f8f6de28700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f8f68108530 0x7f8f6819c480 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.114:44856/0 (socket says 172.21.15.114:44856) 2024-04-23T10:17:07.693 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-04-23T10:17:07.061+0000 7f8f6de28700 1 -- 172.21.15.114:0/357163463 learned_addr learned my addr 172.21.15.114:0/357163463 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-04-23T10:17:07.693 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-04-23T10:17:07.062+0000 7f8f6de28700 1 -- 172.21.15.114:0/357163463 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f8f6819c9c0 con 0x7f8f68108530 2024-04-23T10:17:07.693 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-04-23T10:17:07.062+0000 7f8f6de28700 1 --2- 172.21.15.114:0/357163463 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f8f68108530 0x7f8f6819c480 secure :-1 s=READY pgs=17 cs=0 l=1 rev1=1 crypto rx=0x7f8f58000c00 tx=0x7f8f58011b70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T10:17:07.694 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-04-23T10:17:07.062+0000 7f8f5effd700 1 -- 172.21.15.114:0/357163463 <== mon.0 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f8f58003b20 con 0x7f8f68108530 2024-04-23T10:17:07.694 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-04-23T10:17:07.062+0000 7f8f5effd700 1 -- 172.21.15.114:0/357163463 <== mon.0 v2:172.21.15.114:3300/0 2 ==== config(26 keys) ==== 1070+0+0 (secure 0 0 0) 0x7f8f5801a8c0 con 0x7f8f68108530 2024-04-23T10:17:07.694 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-04-23T10:17:07.062+0000 7f8f7008c700 1 -- 172.21.15.114:0/357163463 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f8f6819cc50 con 0x7f8f68108530 2024-04-23T10:17:07.694 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-04-23T10:17:07.062+0000 7f8f5effd700 1 -- 172.21.15.114:0/357163463 <== mon.0 v2:172.21.15.114:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f8f58024940 con 0x7f8f68108530 2024-04-23T10:17:07.694 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-04-23T10:17:07.062+0000 7f8f7008c700 1 -- 172.21.15.114:0/357163463 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f8f6819d070 con 0x7f8f68108530 2024-04-23T10:17:07.694 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-04-23T10:17:07.063+0000 7f8f5effd700 1 -- 172.21.15.114:0/357163463 <== mon.0 v2:172.21.15.114:3300/0 4 ==== mgrmap(e 2) ==== 46314+0+0 (secure 0 0 0) 0x7f8f58022020 con 0x7f8f68108530 2024-04-23T10:17:07.694 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-04-23T10:17:07.063+0000 7f8f5effd700 1 -- 172.21.15.114:0/357163463 <== mon.0 v2:172.21.15.114:3300/0 5 ==== osd_map(1..1 src has 1..1) ==== 725+0+0 (secure 0 0 0) 0x7f8f5804d070 con 0x7f8f68108530 2024-04-23T10:17:07.694 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-04-23T10:17:07.063+0000 7f8f7008c700 1 -- 172.21.15.114:0/357163463 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f8f680640a0 con 0x7f8f68108530 2024-04-23T10:17:07.694 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-04-23T10:17:07.065+0000 7f8f5effd700 1 -- 172.21.15.114:0/357163463 <== mon.0 v2:172.21.15.114:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+73265 (secure 0 0 0) 0x7f8f5802d3f0 con 0x7f8f68108530 2024-04-23T10:17:07.694 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-04-23T10:17:07.135+0000 7f8f7008c700 1 -- 172.21.15.114:0/357163463 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_command({"prefix": "status", "format": "json-pretty"} v 0) -- 0x7f8f6819ff00 con 0x7f8f68108530 2024-04-23T10:17:07.694 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-04-23T10:17:07.136+0000 7f8f5effd700 1 -- 172.21.15.114:0/357163463 <== mon.0 v2:172.21.15.114:3300/0 7 ==== mon_command_ack([{"prefix": "status", "format": "json-pretty"}]=0 v0) ==== 79+0+1239 (secure 0 0 0) 0x7f8f5800bce0 con 0x7f8f68108530 2024-04-23T10:17:07.694 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-04-23T10:17:07.137+0000 7f8f7008c700 1 -- 172.21.15.114:0/357163463 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f8f68108530 msgr2=0x7f8f6819c480 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T10:17:07.694 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-04-23T10:17:07.137+0000 7f8f7008c700 1 --2- 172.21.15.114:0/357163463 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f8f68108530 0x7f8f6819c480 secure :-1 s=READY pgs=17 cs=0 l=1 rev1=1 crypto rx=0x7f8f58000c00 tx=0x7f8f58011b70 comp rx=0 tx=0).stop 2024-04-23T10:17:07.695 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-04-23T10:17:07.137+0000 7f8f7008c700 1 -- 172.21.15.114:0/357163463 shutdown_connections 2024-04-23T10:17:07.695 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-04-23T10:17:07.137+0000 7f8f7008c700 1 --2- 172.21.15.114:0/357163463 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f8f68108530 0x7f8f6819c480 unknown :-1 s=CLOSED pgs=17 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:17:07.695 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-04-23T10:17:07.137+0000 7f8f7008c700 1 -- 172.21.15.114:0/357163463 >> 172.21.15.114:0/357163463 conn(0x7f8f68103b50 msgr2=0x7f8f681045f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T10:17:07.695 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-04-23T10:17:07.137+0000 7f8f7008c700 1 -- 172.21.15.114:0/357163463 shutdown_connections 2024-04-23T10:17:07.695 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-04-23T10:17:07.137+0000 7f8f7008c700 1 -- 172.21.15.114:0/357163463 wait complete. 2024-04-23T10:17:07.695 INFO:teuthology.orchestra.run.smithi114.stdout:mgr not available, waiting (3/15)... 2024-04-23T10:17:08.504 INFO:journalctl@ceph.mon.a.smithi114.stdout:Apr 23 10:17:08 smithi114 ceph-mon[29255]: from='client.? 172.21.15.114:0/357163463' entity='client.admin' cmd=[{"prefix": "status", "format": "json-pretty"}]: dispatch 2024-04-23T10:17:08.504 INFO:journalctl@ceph.mon.a.smithi114.stdout:Apr 23 10:17:08 smithi114 ceph-mon[29255]: mgrmap e3: x(active, since 1.00638s) 2024-04-23T10:17:10.400 INFO:journalctl@ceph.mon.a.smithi114.stdout:Apr 23 10:17:10 smithi114 ceph-mon[29255]: mgrmap e4: x(active, since 2s) 2024-04-23T10:17:10.934 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stdout 2024-04-23T10:17:10.934 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stdout { 2024-04-23T10:17:10.934 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stdout "fsid": "aa240572-0159-11ef-bc93-c7b262605968", 2024-04-23T10:17:10.934 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stdout "health": { 2024-04-23T10:17:10.935 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stdout "status": "HEALTH_OK", 2024-04-23T10:17:10.935 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stdout "checks": {}, 2024-04-23T10:17:10.935 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stdout "mutes": [] 2024-04-23T10:17:10.935 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stdout }, 2024-04-23T10:17:10.935 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stdout "election_epoch": 5, 2024-04-23T10:17:10.935 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stdout "quorum": [ 2024-04-23T10:17:10.935 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stdout 0 2024-04-23T10:17:10.935 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stdout ], 2024-04-23T10:17:10.935 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stdout "quorum_names": [ 2024-04-23T10:17:10.935 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stdout "a" 2024-04-23T10:17:10.935 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stdout ], 2024-04-23T10:17:10.935 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stdout "quorum_age": 12, 2024-04-23T10:17:10.935 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stdout "monmap": { 2024-04-23T10:17:10.935 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-04-23T10:17:10.936 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stdout "min_mon_release_name": "squid", 2024-04-23T10:17:10.936 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stdout "num_mons": 1 2024-04-23T10:17:10.936 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stdout }, 2024-04-23T10:17:10.936 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stdout "osdmap": { 2024-04-23T10:17:10.936 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-04-23T10:17:10.936 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stdout "num_osds": 0, 2024-04-23T10:17:10.936 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stdout "num_up_osds": 0, 2024-04-23T10:17:10.936 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stdout "osd_up_since": 0, 2024-04-23T10:17:10.944 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stdout "num_in_osds": 0, 2024-04-23T10:17:10.945 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stdout "osd_in_since": 0, 2024-04-23T10:17:10.945 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stdout "num_remapped_pgs": 0 2024-04-23T10:17:10.945 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stdout }, 2024-04-23T10:17:10.945 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stdout "pgmap": { 2024-04-23T10:17:10.945 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stdout "pgs_by_state": [], 2024-04-23T10:17:10.945 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stdout "num_pgs": 0, 2024-04-23T10:17:10.945 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stdout "num_pools": 0, 2024-04-23T10:17:10.945 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stdout "num_objects": 0, 2024-04-23T10:17:10.945 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stdout "data_bytes": 0, 2024-04-23T10:17:10.945 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stdout "bytes_used": 0, 2024-04-23T10:17:10.945 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stdout "bytes_avail": 0, 2024-04-23T10:17:10.945 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stdout "bytes_total": 0 2024-04-23T10:17:10.945 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stdout }, 2024-04-23T10:17:10.945 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stdout "fsmap": { 2024-04-23T10:17:10.945 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-04-23T10:17:10.946 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stdout "by_rank": [], 2024-04-23T10:17:10.946 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stdout "up:standby": 0 2024-04-23T10:17:10.946 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stdout }, 2024-04-23T10:17:10.946 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stdout "mgrmap": { 2024-04-23T10:17:10.946 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stdout "available": true, 2024-04-23T10:17:10.946 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stdout "num_standbys": 0, 2024-04-23T10:17:10.946 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stdout "modules": [ 2024-04-23T10:17:10.946 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stdout "iostat", 2024-04-23T10:17:10.946 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stdout "nfs", 2024-04-23T10:17:10.946 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stdout "restful" 2024-04-23T10:17:10.946 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stdout ], 2024-04-23T10:17:10.946 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stdout "services": {} 2024-04-23T10:17:10.946 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stdout }, 2024-04-23T10:17:10.946 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stdout "servicemap": { 2024-04-23T10:17:10.947 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-04-23T10:17:10.947 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stdout "modified": "2024-04-23T10:16:53.351596+0000", 2024-04-23T10:17:10.947 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stdout "services": {} 2024-04-23T10:17:10.947 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stdout }, 2024-04-23T10:17:10.947 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stdout "progress_events": {} 2024-04-23T10:17:10.947 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stdout } 2024-04-23T10:17:10.947 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-04-23T10:17:10.114+0000 7faff71fb700 1 Processor -- start 2024-04-23T10:17:10.947 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-04-23T10:17:10.114+0000 7faff71fb700 1 -- start start 2024-04-23T10:17:10.947 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-04-23T10:17:10.114+0000 7faff71fb700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7faff0106de0 0x7faff0107200 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T10:17:10.947 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-04-23T10:17:10.114+0000 7faff71fb700 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x7faff007be80 con 0x7faff0106de0 2024-04-23T10:17:10.947 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-04-23T10:17:10.114+0000 7faff4f97700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7faff0106de0 0x7faff0107200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T10:17:10.947 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-04-23T10:17:10.114+0000 7faff4f97700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7faff0106de0 0x7faff0107200 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.114:44858/0 (socket says 172.21.15.114:44858) 2024-04-23T10:17:10.947 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-04-23T10:17:10.114+0000 7faff4f97700 1 -- 172.21.15.114:0/2215335740 learned_addr learned my addr 172.21.15.114:0/2215335740 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-04-23T10:17:10.947 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-04-23T10:17:10.114+0000 7faff4f97700 1 -- 172.21.15.114:0/2215335740 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7faff0107740 con 0x7faff0106de0 2024-04-23T10:17:10.947 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-04-23T10:17:10.115+0000 7faff4f97700 1 --2- 172.21.15.114:0/2215335740 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7faff0106de0 0x7faff0107200 secure :-1 s=READY pgs=18 cs=0 l=1 rev1=1 crypto rx=0x7fafe40098a0 tx=0x7fafe4009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=40eb90c1cf2c259f server_cookie=0 in_seq=0 out_seq=0 2024-04-23T10:17:10.948 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-04-23T10:17:10.115+0000 7fafef7fe700 1 -- 172.21.15.114:0/2215335740 <== mon.0 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fafe4004160 con 0x7faff0106de0 2024-04-23T10:17:10.948 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-04-23T10:17:10.115+0000 7fafef7fe700 1 -- 172.21.15.114:0/2215335740 <== mon.0 v2:172.21.15.114:3300/0 2 ==== config(26 keys) ==== 1070+0+0 (secure 0 0 0) 0x7fafe4004300 con 0x7faff0106de0 2024-04-23T10:17:10.948 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-04-23T10:17:10.115+0000 7fafef7fe700 1 -- 172.21.15.114:0/2215335740 <== mon.0 v2:172.21.15.114:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fafe4011af0 con 0x7faff0106de0 2024-04-23T10:17:10.948 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-04-23T10:17:10.115+0000 7faff71fb700 1 -- 172.21.15.114:0/2215335740 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7faff0106de0 msgr2=0x7faff0107200 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T10:17:10.948 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-04-23T10:17:10.115+0000 7faff71fb700 1 --2- 172.21.15.114:0/2215335740 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7faff0106de0 0x7faff0107200 secure :-1 s=READY pgs=18 cs=0 l=1 rev1=1 crypto rx=0x7fafe40098a0 tx=0x7fafe4009bb0 comp rx=0 tx=0).stop 2024-04-23T10:17:10.948 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-04-23T10:17:10.115+0000 7faff71fb700 1 -- 172.21.15.114:0/2215335740 shutdown_connections 2024-04-23T10:17:10.948 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-04-23T10:17:10.115+0000 7faff71fb700 1 --2- 172.21.15.114:0/2215335740 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7faff0106de0 0x7faff0107200 unknown :-1 s=CLOSED pgs=18 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:17:10.948 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-04-23T10:17:10.115+0000 7faff71fb700 1 -- 172.21.15.114:0/2215335740 >> 172.21.15.114:0/2215335740 conn(0x7faff01029e0 msgr2=0x7faff0104e20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T10:17:10.948 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-04-23T10:17:10.115+0000 7faff71fb700 1 -- 172.21.15.114:0/2215335740 shutdown_connections 2024-04-23T10:17:10.948 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-04-23T10:17:10.115+0000 7faff71fb700 1 -- 172.21.15.114:0/2215335740 wait complete. 2024-04-23T10:17:10.948 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-04-23T10:17:10.116+0000 7faff71fb700 1 Processor -- start 2024-04-23T10:17:10.948 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-04-23T10:17:10.116+0000 7faff71fb700 1 -- start start 2024-04-23T10:17:10.948 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-04-23T10:17:10.116+0000 7faff71fb700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7faff0106de0 0x7faff019e6c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T10:17:10.948 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-04-23T10:17:10.116+0000 7faff71fb700 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x7faff0108770 con 0x7faff0106de0 2024-04-23T10:17:10.949 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-04-23T10:17:10.116+0000 7faff4f97700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7faff0106de0 0x7faff019e6c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T10:17:10.949 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-04-23T10:17:10.116+0000 7faff4f97700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7faff0106de0 0x7faff019e6c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.114:44860/0 (socket says 172.21.15.114:44860) 2024-04-23T10:17:10.949 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-04-23T10:17:10.116+0000 7faff4f97700 1 -- 172.21.15.114:0/127217850 learned_addr learned my addr 172.21.15.114:0/127217850 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-04-23T10:17:10.949 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-04-23T10:17:10.116+0000 7faff4f97700 1 -- 172.21.15.114:0/127217850 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7faff019ec00 con 0x7faff0106de0 2024-04-23T10:17:10.949 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-04-23T10:17:10.117+0000 7faff4f97700 1 --2- 172.21.15.114:0/127217850 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7faff0106de0 0x7faff019e6c0 secure :-1 s=READY pgs=19 cs=0 l=1 rev1=1 crypto rx=0x7fafe4011ac0 tx=0x7fafe4011ba0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T10:17:10.949 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-04-23T10:17:10.117+0000 7fafedffb700 1 -- 172.21.15.114:0/127217850 <== mon.0 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fafe4003970 con 0x7faff0106de0 2024-04-23T10:17:10.949 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-04-23T10:17:10.117+0000 7fafedffb700 1 -- 172.21.15.114:0/127217850 <== mon.0 v2:172.21.15.114:3300/0 2 ==== config(26 keys) ==== 1070+0+0 (secure 0 0 0) 0x7fafe4003b10 con 0x7faff0106de0 2024-04-23T10:17:10.949 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-04-23T10:17:10.117+0000 7faff71fb700 1 -- 172.21.15.114:0/127217850 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7faff019ee90 con 0x7faff0106de0 2024-04-23T10:17:10.949 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-04-23T10:17:10.117+0000 7fafedffb700 1 -- 172.21.15.114:0/127217850 <== mon.0 v2:172.21.15.114:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fafe4024610 con 0x7faff0106de0 2024-04-23T10:17:10.949 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-04-23T10:17:10.117+0000 7faff71fb700 1 -- 172.21.15.114:0/127217850 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7faff01a1b80 con 0x7faff0106de0 2024-04-23T10:17:10.949 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-04-23T10:17:10.118+0000 7fafedffb700 1 -- 172.21.15.114:0/127217850 <== mon.0 v2:172.21.15.114:3300/0 4 ==== mgrmap(e 4) ==== 46547+0+0 (secure 0 0 0) 0x7fafe4024840 con 0x7faff0106de0 2024-04-23T10:17:10.949 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-04-23T10:17:10.118+0000 7fafedffb700 1 --2- 172.21.15.114:0/127217850 >> [v2:172.21.15.114:6800/2831530838,v1:172.21.15.114:6801/2831530838] conn(0x7fafe00395b0 0x7fafe003ba70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T10:17:10.949 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-04-23T10:17:10.118+0000 7faff71fb700 1 -- 172.21.15.114:0/127217850 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7faff00640a0 con 0x7faff0106de0 2024-04-23T10:17:10.950 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-04-23T10:17:10.118+0000 7fafedffb700 1 -- 172.21.15.114:0/127217850 <== mon.0 v2:172.21.15.114:3300/0 5 ==== osd_map(1..1 src has 1..1) ==== 725+0+0 (secure 0 0 0) 0x7fafe404da00 con 0x7faff0106de0 2024-04-23T10:17:10.950 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-04-23T10:17:10.118+0000 7fafeffff700 1 --2- 172.21.15.114:0/127217850 >> [v2:172.21.15.114:6800/2831530838,v1:172.21.15.114:6801/2831530838] conn(0x7fafe00395b0 0x7fafe003ba70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T10:17:10.950 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-04-23T10:17:10.119+0000 7fafeffff700 1 --2- 172.21.15.114:0/127217850 >> [v2:172.21.15.114:6800/2831530838,v1:172.21.15.114:6801/2831530838] conn(0x7fafe00395b0 0x7fafe003ba70 secure :-1 s=READY pgs=6 cs=0 l=1 rev1=1 crypto rx=0x7fafdc006fd0 tx=0x7fafdc006e40 comp rx=0 tx=0).ready entity=mgr.14100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T10:17:10.950 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-04-23T10:17:10.123+0000 7fafedffb700 1 -- 172.21.15.114:0/127217850 <== mon.0 v2:172.21.15.114:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7fafe404d420 con 0x7faff0106de0 2024-04-23T10:17:10.950 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-04-23T10:17:10.401+0000 7faff71fb700 1 -- 172.21.15.114:0/127217850 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_command({"prefix": "status", "format": "json-pretty"} v 0) -- 0x7faff00008d0 con 0x7faff0106de0 2024-04-23T10:17:10.950 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-04-23T10:17:10.401+0000 7fafedffb700 1 -- 172.21.15.114:0/127217850 <== mon.0 v2:172.21.15.114:3300/0 7 ==== mon_command_ack([{"prefix": "status", "format": "json-pretty"}]=0 v0) ==== 79+0+1239 (secure 0 0 0) 0x7fafe402d770 con 0x7faff0106de0 2024-04-23T10:17:10.950 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-04-23T10:17:10.403+0000 7faff71fb700 1 -- 172.21.15.114:0/127217850 >> [v2:172.21.15.114:6800/2831530838,v1:172.21.15.114:6801/2831530838] conn(0x7fafe00395b0 msgr2=0x7fafe003ba70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T10:17:10.950 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-04-23T10:17:10.403+0000 7faff71fb700 1 --2- 172.21.15.114:0/127217850 >> [v2:172.21.15.114:6800/2831530838,v1:172.21.15.114:6801/2831530838] conn(0x7fafe00395b0 0x7fafe003ba70 secure :-1 s=READY pgs=6 cs=0 l=1 rev1=1 crypto rx=0x7fafdc006fd0 tx=0x7fafdc006e40 comp rx=0 tx=0).stop 2024-04-23T10:17:10.950 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-04-23T10:17:10.403+0000 7faff71fb700 1 -- 172.21.15.114:0/127217850 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7faff0106de0 msgr2=0x7faff019e6c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T10:17:10.950 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-04-23T10:17:10.403+0000 7faff71fb700 1 --2- 172.21.15.114:0/127217850 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7faff0106de0 0x7faff019e6c0 secure :-1 s=READY pgs=19 cs=0 l=1 rev1=1 crypto rx=0x7fafe4011ac0 tx=0x7fafe4011ba0 comp rx=0 tx=0).stop 2024-04-23T10:17:10.950 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-04-23T10:17:10.403+0000 7faff71fb700 1 -- 172.21.15.114:0/127217850 shutdown_connections 2024-04-23T10:17:10.950 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-04-23T10:17:10.403+0000 7faff71fb700 1 --2- 172.21.15.114:0/127217850 >> [v2:172.21.15.114:6800/2831530838,v1:172.21.15.114:6801/2831530838] conn(0x7fafe00395b0 0x7fafe003ba70 unknown :-1 s=CLOSED pgs=6 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:17:10.950 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-04-23T10:17:10.403+0000 7faff71fb700 1 --2- 172.21.15.114:0/127217850 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7faff0106de0 0x7faff019e6c0 unknown :-1 s=CLOSED pgs=19 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:17:10.950 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-04-23T10:17:10.403+0000 7faff71fb700 1 -- 172.21.15.114:0/127217850 >> 172.21.15.114:0/127217850 conn(0x7faff01029e0 msgr2=0x7faff0195330 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T10:17:10.950 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-04-23T10:17:10.404+0000 7faff71fb700 1 -- 172.21.15.114:0/127217850 shutdown_connections 2024-04-23T10:17:10.951 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-04-23T10:17:10.404+0000 7faff71fb700 1 -- 172.21.15.114:0/127217850 wait complete. 2024-04-23T10:17:10.951 INFO:teuthology.orchestra.run.smithi114.stdout:mgr is available 2024-04-23T10:17:11.349 INFO:journalctl@ceph.mon.a.smithi114.stdout:Apr 23 10:17:11 smithi114 ceph-mon[29255]: from='client.? 172.21.15.114:0/127217850' entity='client.admin' cmd=[{"prefix": "status", "format": "json-pretty"}]: dispatch 2024-04-23T10:17:12.136 INFO:journalctl@ceph.mon.a.smithi114.stdout:Apr 23 10:17:12 smithi114 ceph-mon[29255]: from='client.? 172.21.15.114:0/3046128780' entity='client.admin' cmd=[{"prefix": "config assimilate-conf"}]: dispatch 2024-04-23T10:17:12.148 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stdout 2024-04-23T10:17:12.148 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stdout [global] 2024-04-23T10:17:12.148 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stdout fsid = aa240572-0159-11ef-bc93-c7b262605968 2024-04-23T10:17:12.148 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stdout mon_cluster_log_file_level = debug 2024-04-23T10:17:12.148 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stdout mon_osd_allow_pg_remap = true 2024-04-23T10:17:12.148 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stdout mon_osd_allow_primary_affinity = true 2024-04-23T10:17:12.148 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stdout mon_warn_on_no_sortbitwise = false 2024-04-23T10:17:12.148 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stdout osd_crush_chooseleaf_type = 0 2024-04-23T10:17:12.148 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stdout 2024-04-23T10:17:12.149 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stdout [mgr] 2024-04-23T10:17:12.149 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stdout mgr/telemetry/nag = false 2024-04-23T10:17:12.149 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stdout 2024-04-23T10:17:12.149 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stdout [osd] 2024-04-23T10:17:12.149 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stdout bdev_async_discard = True 2024-04-23T10:17:12.149 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stdout mon_osd_backfillfull_ratio = 0.85 2024-04-23T10:17:12.149 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stdout mon_osd_full_ratio = 0.9 2024-04-23T10:17:12.149 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stdout mon_osd_nearfull_ratio = 0.8 2024-04-23T10:17:12.149 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stdout osd_map_max_advance = 10 2024-04-23T10:17:12.149 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stdout osd_sloppy_crc = true 2024-04-23T10:17:12.149 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-04-23T10:17:11.363+0000 7fde87fdb700 1 Processor -- start 2024-04-23T10:17:12.150 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-04-23T10:17:11.363+0000 7fde87fdb700 1 -- start start 2024-04-23T10:17:12.150 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-04-23T10:17:11.363+0000 7fde87fdb700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fde80106e20 0x7fde80107240 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T10:17:12.150 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-04-23T10:17:11.363+0000 7fde87fdb700 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x7fde8007be40 con 0x7fde80106e20 2024-04-23T10:17:12.150 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-04-23T10:17:11.363+0000 7fde85d77700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fde80106e20 0x7fde80107240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T10:17:12.150 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-04-23T10:17:11.363+0000 7fde85d77700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fde80106e20 0x7fde80107240 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.114:44862/0 (socket says 172.21.15.114:44862) 2024-04-23T10:17:12.150 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-04-23T10:17:11.363+0000 7fde85d77700 1 -- 172.21.15.114:0/201869208 learned_addr learned my addr 172.21.15.114:0/201869208 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-04-23T10:17:12.150 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-04-23T10:17:11.363+0000 7fde85d77700 1 -- 172.21.15.114:0/201869208 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fde80107780 con 0x7fde80106e20 2024-04-23T10:17:12.150 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-04-23T10:17:11.363+0000 7fde85d77700 1 --2- 172.21.15.114:0/201869208 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fde80106e20 0x7fde80107240 secure :-1 s=READY pgs=20 cs=0 l=1 rev1=1 crypto rx=0x7fde740098a0 tx=0x7fde74009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2ebef8b13c7a3d4 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T10:17:12.150 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-04-23T10:17:11.364+0000 7fde84d75700 1 -- 172.21.15.114:0/201869208 <== mon.0 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fde74004160 con 0x7fde80106e20 2024-04-23T10:17:12.150 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-04-23T10:17:11.364+0000 7fde84d75700 1 -- 172.21.15.114:0/201869208 <== mon.0 v2:172.21.15.114:3300/0 2 ==== config(26 keys) ==== 1070+0+0 (secure 0 0 0) 0x7fde74004300 con 0x7fde80106e20 2024-04-23T10:17:12.151 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-04-23T10:17:11.364+0000 7fde84d75700 1 -- 172.21.15.114:0/201869208 <== mon.0 v2:172.21.15.114:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fde74011af0 con 0x7fde80106e20 2024-04-23T10:17:12.151 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-04-23T10:17:11.364+0000 7fde87fdb700 1 -- 172.21.15.114:0/201869208 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fde80106e20 msgr2=0x7fde80107240 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T10:17:12.151 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-04-23T10:17:11.364+0000 7fde87fdb700 1 --2- 172.21.15.114:0/201869208 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fde80106e20 0x7fde80107240 secure :-1 s=READY pgs=20 cs=0 l=1 rev1=1 crypto rx=0x7fde740098a0 tx=0x7fde74009bb0 comp rx=0 tx=0).stop 2024-04-23T10:17:12.151 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-04-23T10:17:11.364+0000 7fde87fdb700 1 -- 172.21.15.114:0/201869208 shutdown_connections 2024-04-23T10:17:12.151 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-04-23T10:17:11.364+0000 7fde87fdb700 1 --2- 172.21.15.114:0/201869208 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fde80106e20 0x7fde80107240 unknown :-1 s=CLOSED pgs=20 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:17:12.151 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-04-23T10:17:11.364+0000 7fde87fdb700 1 -- 172.21.15.114:0/201869208 >> 172.21.15.114:0/201869208 conn(0x7fde801029e0 msgr2=0x7fde80104e20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T10:17:12.151 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-04-23T10:17:11.364+0000 7fde87fdb700 1 -- 172.21.15.114:0/201869208 shutdown_connections 2024-04-23T10:17:12.151 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-04-23T10:17:11.364+0000 7fde87fdb700 1 -- 172.21.15.114:0/201869208 wait complete. 2024-04-23T10:17:12.151 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-04-23T10:17:11.365+0000 7fde87fdb700 1 Processor -- start 2024-04-23T10:17:12.151 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-04-23T10:17:11.365+0000 7fde87fdb700 1 -- start start 2024-04-23T10:17:12.152 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-04-23T10:17:11.365+0000 7fde87fdb700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fde80106e20 0x7fde8019e6f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T10:17:12.152 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-04-23T10:17:11.365+0000 7fde87fdb700 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x7fde801087b0 con 0x7fde80106e20 2024-04-23T10:17:12.152 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-04-23T10:17:11.365+0000 7fde85d77700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fde80106e20 0x7fde8019e6f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T10:17:12.152 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-04-23T10:17:11.365+0000 7fde85d77700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fde80106e20 0x7fde8019e6f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.114:44874/0 (socket says 172.21.15.114:44874) 2024-04-23T10:17:12.152 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-04-23T10:17:11.365+0000 7fde85d77700 1 -- 172.21.15.114:0/3046128780 learned_addr learned my addr 172.21.15.114:0/3046128780 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-04-23T10:17:12.152 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-04-23T10:17:11.365+0000 7fde85d77700 1 -- 172.21.15.114:0/3046128780 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fde8019ec30 con 0x7fde80106e20 2024-04-23T10:17:12.152 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-04-23T10:17:11.365+0000 7fde85d77700 1 --2- 172.21.15.114:0/3046128780 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fde80106e20 0x7fde8019e6f0 secure :-1 s=READY pgs=21 cs=0 l=1 rev1=1 crypto rx=0x7fde74009fb0 tx=0x7fde74011940 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T10:17:12.152 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-04-23T10:17:11.366+0000 7fde72ffd700 1 -- 172.21.15.114:0/3046128780 <== mon.0 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fde740036a0 con 0x7fde80106e20 2024-04-23T10:17:12.152 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-04-23T10:17:11.366+0000 7fde72ffd700 1 -- 172.21.15.114:0/3046128780 <== mon.0 v2:172.21.15.114:3300/0 2 ==== config(26 keys) ==== 1070+0+0 (secure 0 0 0) 0x7fde7401a430 con 0x7fde80106e20 2024-04-23T10:17:12.152 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-04-23T10:17:11.366+0000 7fde72ffd700 1 -- 172.21.15.114:0/3046128780 <== mon.0 v2:172.21.15.114:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fde74019550 con 0x7fde80106e20 2024-04-23T10:17:12.152 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-04-23T10:17:11.366+0000 7fde87fdb700 1 -- 172.21.15.114:0/3046128780 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fde8019eec0 con 0x7fde80106e20 2024-04-23T10:17:12.153 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-04-23T10:17:11.366+0000 7fde87fdb700 1 -- 172.21.15.114:0/3046128780 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fde801a1bb0 con 0x7fde80106e20 2024-04-23T10:17:12.153 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-04-23T10:17:11.366+0000 7fde72ffd700 1 -- 172.21.15.114:0/3046128780 <== mon.0 v2:172.21.15.114:3300/0 4 ==== mgrmap(e 4) ==== 46547+0+0 (secure 0 0 0) 0x7fde7401a8e0 con 0x7fde80106e20 2024-04-23T10:17:12.153 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-04-23T10:17:11.367+0000 7fde72ffd700 1 --2- 172.21.15.114:0/3046128780 >> [v2:172.21.15.114:6800/2831530838,v1:172.21.15.114:6801/2831530838] conn(0x7fde6c039510 0x7fde6c03b9d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T10:17:12.153 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-04-23T10:17:11.367+0000 7fde87fdb700 1 -- 172.21.15.114:0/3046128780 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fde800640a0 con 0x7fde80106e20 2024-04-23T10:17:12.153 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-04-23T10:17:11.367+0000 7fde72ffd700 1 -- 172.21.15.114:0/3046128780 <== mon.0 v2:172.21.15.114:3300/0 5 ==== osd_map(1..1 src has 1..1) ==== 725+0+0 (secure 0 0 0) 0x7fde7404d870 con 0x7fde80106e20 2024-04-23T10:17:12.153 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-04-23T10:17:11.367+0000 7fde85576700 1 --2- 172.21.15.114:0/3046128780 >> [v2:172.21.15.114:6800/2831530838,v1:172.21.15.114:6801/2831530838] conn(0x7fde6c039510 0x7fde6c03b9d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T10:17:12.153 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-04-23T10:17:11.367+0000 7fde85576700 1 --2- 172.21.15.114:0/3046128780 >> [v2:172.21.15.114:6800/2831530838,v1:172.21.15.114:6801/2831530838] conn(0x7fde6c039510 0x7fde6c03b9d0 secure :-1 s=READY pgs=7 cs=0 l=1 rev1=1 crypto rx=0x7fde7c006fd0 tx=0x7fde7c006e40 comp rx=0 tx=0).ready entity=mgr.14100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T10:17:12.153 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-04-23T10:17:11.372+0000 7fde72ffd700 1 -- 172.21.15.114:0/3046128780 <== mon.0 v2:172.21.15.114:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7fde740172b0 con 0x7fde80106e20 2024-04-23T10:17:12.153 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-04-23T10:17:11.566+0000 7fde87fdb700 1 -- 172.21.15.114:0/3046128780 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_command({"prefix": "config assimilate-conf"} v 0) -- 0x7fde8002e210 con 0x7fde80106e20 2024-04-23T10:17:12.153 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-04-23T10:17:11.567+0000 7fde72ffd700 1 -- 172.21.15.114:0/3046128780 <== mon.0 v2:172.21.15.114:3300/0 7 ==== mon_command_ack([{"prefix": "config assimilate-conf"}]=0 v3) ==== 70+0+436 (secure 0 0 0) 0x7fde7404d290 con 0x7fde80106e20 2024-04-23T10:17:12.153 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-04-23T10:17:11.569+0000 7fde87fdb700 1 -- 172.21.15.114:0/3046128780 >> [v2:172.21.15.114:6800/2831530838,v1:172.21.15.114:6801/2831530838] conn(0x7fde6c039510 msgr2=0x7fde6c03b9d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T10:17:12.154 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-04-23T10:17:11.569+0000 7fde87fdb700 1 --2- 172.21.15.114:0/3046128780 >> [v2:172.21.15.114:6800/2831530838,v1:172.21.15.114:6801/2831530838] conn(0x7fde6c039510 0x7fde6c03b9d0 secure :-1 s=READY pgs=7 cs=0 l=1 rev1=1 crypto rx=0x7fde7c006fd0 tx=0x7fde7c006e40 comp rx=0 tx=0).stop 2024-04-23T10:17:12.154 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-04-23T10:17:11.569+0000 7fde87fdb700 1 -- 172.21.15.114:0/3046128780 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fde80106e20 msgr2=0x7fde8019e6f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T10:17:12.154 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-04-23T10:17:11.569+0000 7fde87fdb700 1 --2- 172.21.15.114:0/3046128780 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fde80106e20 0x7fde8019e6f0 secure :-1 s=READY pgs=21 cs=0 l=1 rev1=1 crypto rx=0x7fde74009fb0 tx=0x7fde74011940 comp rx=0 tx=0).stop 2024-04-23T10:17:12.154 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-04-23T10:17:11.569+0000 7fde87fdb700 1 -- 172.21.15.114:0/3046128780 shutdown_connections 2024-04-23T10:17:12.154 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-04-23T10:17:11.569+0000 7fde87fdb700 1 --2- 172.21.15.114:0/3046128780 >> [v2:172.21.15.114:6800/2831530838,v1:172.21.15.114:6801/2831530838] conn(0x7fde6c039510 0x7fde6c03b9d0 unknown :-1 s=CLOSED pgs=7 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:17:12.154 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-04-23T10:17:11.569+0000 7fde87fdb700 1 --2- 172.21.15.114:0/3046128780 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fde80106e20 0x7fde8019e6f0 unknown :-1 s=CLOSED pgs=21 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:17:12.154 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-04-23T10:17:11.569+0000 7fde87fdb700 1 -- 172.21.15.114:0/3046128780 >> 172.21.15.114:0/3046128780 conn(0x7fde801029e0 msgr2=0x7fde801952d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T10:17:12.154 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-04-23T10:17:11.570+0000 7fde87fdb700 1 -- 172.21.15.114:0/3046128780 shutdown_connections 2024-04-23T10:17:12.154 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-04-23T10:17:11.570+0000 7fde87fdb700 1 -- 172.21.15.114:0/3046128780 wait complete. 2024-04-23T10:17:12.154 INFO:teuthology.orchestra.run.smithi114.stdout:Enabling cephadm module... 2024-04-23T10:17:13.084 INFO:journalctl@ceph.mon.a.smithi114.stdout:Apr 23 10:17:13 smithi114 ceph-mon[29255]: from='client.? 172.21.15.114:0/3188535063' entity='client.admin' cmd=[{"prefix": "mgr module enable", "module": "cephadm"}]: dispatch 2024-04-23T10:17:13.342 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Apr 23 10:17:13 smithi114 ceph-aa240572-0159-11ef-bc93-c7b262605968-mgr-x[29490]: ignoring --setuser ceph since I am not root 2024-04-23T10:17:13.342 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Apr 23 10:17:13 smithi114 ceph-aa240572-0159-11ef-bc93-c7b262605968-mgr-x[29490]: ignoring --setgroup ceph since I am not root 2024-04-23T10:17:13.342 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Apr 23 10:17:13 smithi114 ceph-aa240572-0159-11ef-bc93-c7b262605968-mgr-x[29490]: 2024-04-23T10:17:13.250+0000 7f8b5dfee200 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2024-04-23T10:17:13.651 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Apr 23 10:17:13 smithi114 ceph-aa240572-0159-11ef-bc93-c7b262605968-mgr-x[29490]: 2024-04-23T10:17:13.435+0000 7f8b5dfee200 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2024-04-23T10:17:13.651 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Apr 23 10:17:13 smithi114 ceph-aa240572-0159-11ef-bc93-c7b262605968-mgr-x[29490]: 2024-04-23T10:17:13.537+0000 7f8b5dfee200 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2024-04-23T10:17:13.659 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-04-23T10:17:12.563+0000 7fa01091c700 1 Processor -- start 2024-04-23T10:17:13.659 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-04-23T10:17:12.563+0000 7fa01091c700 1 -- start start 2024-04-23T10:17:13.659 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-04-23T10:17:12.564+0000 7fa01091c700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fa00c108fb0 0x7fa00c10b3a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T10:17:13.659 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-04-23T10:17:12.564+0000 7fa01091c700 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x7fa00c063d60 con 0x7fa00c108fb0 2024-04-23T10:17:13.659 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-04-23T10:17:12.564+0000 7fa00a59c700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fa00c108fb0 0x7fa00c10b3a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T10:17:13.660 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-04-23T10:17:12.564+0000 7fa00a59c700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fa00c108fb0 0x7fa00c10b3a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.114:44876/0 (socket says 172.21.15.114:44876) 2024-04-23T10:17:13.660 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-04-23T10:17:12.564+0000 7fa00a59c700 1 -- 172.21.15.114:0/3035083968 learned_addr learned my addr 172.21.15.114:0/3035083968 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-04-23T10:17:13.660 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-04-23T10:17:12.564+0000 7fa00a59c700 1 -- 172.21.15.114:0/3035083968 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa00c10b8e0 con 0x7fa00c108fb0 2024-04-23T10:17:13.660 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-04-23T10:17:12.564+0000 7fa00a59c700 1 --2- 172.21.15.114:0/3035083968 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fa00c108fb0 0x7fa00c10b3a0 secure :-1 s=READY pgs=22 cs=0 l=1 rev1=1 crypto rx=0x7f9ffc0098a0 tx=0x7f9ffc009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9719192805232418 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T10:17:13.660 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-04-23T10:17:12.564+0000 7fa00959a700 1 -- 172.21.15.114:0/3035083968 <== mon.0 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f9ffc004160 con 0x7fa00c108fb0 2024-04-23T10:17:13.660 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-04-23T10:17:12.564+0000 7fa00959a700 1 -- 172.21.15.114:0/3035083968 <== mon.0 v2:172.21.15.114:3300/0 2 ==== config(26 keys) ==== 1070+0+0 (secure 0 0 0) 0x7f9ffc004300 con 0x7fa00c108fb0 2024-04-23T10:17:13.660 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-04-23T10:17:12.564+0000 7fa00959a700 1 -- 172.21.15.114:0/3035083968 <== mon.0 v2:172.21.15.114:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f9ffc011af0 con 0x7fa00c108fb0 2024-04-23T10:17:13.660 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-04-23T10:17:12.565+0000 7fa01091c700 1 -- 172.21.15.114:0/3035083968 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fa00c108fb0 msgr2=0x7fa00c10b3a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T10:17:13.660 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-04-23T10:17:12.565+0000 7fa01091c700 1 --2- 172.21.15.114:0/3035083968 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fa00c108fb0 0x7fa00c10b3a0 secure :-1 s=READY pgs=22 cs=0 l=1 rev1=1 crypto rx=0x7f9ffc0098a0 tx=0x7f9ffc009bb0 comp rx=0 tx=0).stop 2024-04-23T10:17:13.660 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-04-23T10:17:12.565+0000 7fa01091c700 1 -- 172.21.15.114:0/3035083968 shutdown_connections 2024-04-23T10:17:13.660 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-04-23T10:17:12.565+0000 7fa01091c700 1 --2- 172.21.15.114:0/3035083968 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fa00c108fb0 0x7fa00c10b3a0 unknown :-1 s=CLOSED pgs=22 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:17:13.660 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-04-23T10:17:12.565+0000 7fa01091c700 1 -- 172.21.15.114:0/3035083968 >> 172.21.15.114:0/3035083968 conn(0x7fa00c102a40 msgr2=0x7fa00c104e60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T10:17:13.660 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-04-23T10:17:12.565+0000 7fa01091c700 1 -- 172.21.15.114:0/3035083968 shutdown_connections 2024-04-23T10:17:13.660 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-04-23T10:17:12.565+0000 7fa01091c700 1 -- 172.21.15.114:0/3035083968 wait complete. 2024-04-23T10:17:13.661 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-04-23T10:17:12.566+0000 7fa01091c700 1 Processor -- start 2024-04-23T10:17:13.661 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-04-23T10:17:12.566+0000 7fa01091c700 1 -- start start 2024-04-23T10:17:13.661 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-04-23T10:17:12.566+0000 7fa01091c700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fa00c108fb0 0x7fa00c19c450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T10:17:13.661 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-04-23T10:17:12.566+0000 7fa01091c700 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x7fa00c10c8b0 con 0x7fa00c108fb0 2024-04-23T10:17:13.661 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-04-23T10:17:12.566+0000 7fa00a59c700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fa00c108fb0 0x7fa00c19c450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T10:17:13.661 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-04-23T10:17:12.566+0000 7fa00a59c700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fa00c108fb0 0x7fa00c19c450 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.114:44888/0 (socket says 172.21.15.114:44888) 2024-04-23T10:17:13.661 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-04-23T10:17:12.566+0000 7fa00a59c700 1 -- 172.21.15.114:0/3188535063 learned_addr learned my addr 172.21.15.114:0/3188535063 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-04-23T10:17:13.661 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-04-23T10:17:12.566+0000 7fa00a59c700 1 -- 172.21.15.114:0/3188535063 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa00c19c990 con 0x7fa00c108fb0 2024-04-23T10:17:13.661 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-04-23T10:17:12.566+0000 7fa00a59c700 1 --2- 172.21.15.114:0/3188535063 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fa00c108fb0 0x7fa00c19c450 secure :-1 s=READY pgs=23 cs=0 l=1 rev1=1 crypto rx=0x7f9ffc003f40 tx=0x7f9ffc011890 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T10:17:13.661 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-04-23T10:17:12.567+0000 7f9ffb7fe700 1 -- 172.21.15.114:0/3188535063 <== mon.0 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f9ffc0038b0 con 0x7fa00c108fb0 2024-04-23T10:17:13.661 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-04-23T10:17:12.567+0000 7f9ffb7fe700 1 -- 172.21.15.114:0/3188535063 <== mon.0 v2:172.21.15.114:3300/0 2 ==== config(26 keys) ==== 1070+0+0 (secure 0 0 0) 0x7f9ffc003a50 con 0x7fa00c108fb0 2024-04-23T10:17:13.661 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-04-23T10:17:12.567+0000 7fa01091c700 1 -- 172.21.15.114:0/3188535063 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fa00c19cc20 con 0x7fa00c108fb0 2024-04-23T10:17:13.661 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-04-23T10:17:12.567+0000 7f9ffb7fe700 1 -- 172.21.15.114:0/3188535063 <== mon.0 v2:172.21.15.114:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f9ffc01a5c0 con 0x7fa00c108fb0 2024-04-23T10:17:13.662 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-04-23T10:17:12.567+0000 7fa01091c700 1 -- 172.21.15.114:0/3188535063 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fa00c19f910 con 0x7fa00c108fb0 2024-04-23T10:17:13.662 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-04-23T10:17:12.567+0000 7f9ffb7fe700 1 -- 172.21.15.114:0/3188535063 <== mon.0 v2:172.21.15.114:3300/0 4 ==== mgrmap(e 4) ==== 46547+0+0 (secure 0 0 0) 0x7f9ffc01a870 con 0x7fa00c108fb0 2024-04-23T10:17:13.662 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-04-23T10:17:12.568+0000 7fa01091c700 1 -- 172.21.15.114:0/3188535063 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fa00c0640a0 con 0x7fa00c108fb0 2024-04-23T10:17:13.662 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-04-23T10:17:12.568+0000 7f9ffb7fe700 1 --2- 172.21.15.114:0/3188535063 >> [v2:172.21.15.114:6800/2831530838,v1:172.21.15.114:6801/2831530838] conn(0x7f9ff4039560 0x7f9ff403ba20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T10:17:13.662 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-04-23T10:17:12.568+0000 7f9ffb7fe700 1 -- 172.21.15.114:0/3188535063 <== mon.0 v2:172.21.15.114:3300/0 5 ==== osd_map(1..1 src has 1..1) ==== 725+0+0 (secure 0 0 0) 0x7f9ffc04da50 con 0x7fa00c108fb0 2024-04-23T10:17:13.662 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-04-23T10:17:12.568+0000 7fa009d9b700 1 --2- 172.21.15.114:0/3188535063 >> [v2:172.21.15.114:6800/2831530838,v1:172.21.15.114:6801/2831530838] conn(0x7f9ff4039560 0x7f9ff403ba20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T10:17:13.662 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-04-23T10:17:12.568+0000 7fa009d9b700 1 --2- 172.21.15.114:0/3188535063 >> [v2:172.21.15.114:6800/2831530838,v1:172.21.15.114:6801/2831530838] conn(0x7f9ff4039560 0x7f9ff403ba20 secure :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0x7fa000006fd0 tx=0x7fa000006e40 comp rx=0 tx=0).ready entity=mgr.14100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T10:17:13.662 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-04-23T10:17:12.573+0000 7f9ffb7fe700 1 -- 172.21.15.114:0/3188535063 <== mon.0 v2:172.21.15.114:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f9ffc019c70 con 0x7fa00c108fb0 2024-04-23T10:17:13.662 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-04-23T10:17:12.831+0000 7fa01091c700 1 -- 172.21.15.114:0/3188535063 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_command({"prefix": "mgr module enable", "module": "cephadm"} v 0) -- 0x7fa00c19fae0 con 0x7fa00c108fb0 2024-04-23T10:17:13.662 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-04-23T10:17:13.084+0000 7f9ffb7fe700 1 -- 172.21.15.114:0/3188535063 <== mon.0 v2:172.21.15.114:3300/0 7 ==== mon_command_ack([{"prefix": "mgr module enable", "module": "cephadm"}]=0 v5) ==== 86+0+0 (secure 0 0 0) 0x7fa00c19fae0 con 0x7fa00c108fb0 2024-04-23T10:17:13.662 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-04-23T10:17:13.085+0000 7f9ffb7fe700 1 -- 172.21.15.114:0/3188535063 <== mon.0 v2:172.21.15.114:3300/0 8 ==== mgrmap(e 5) ==== 46558+0+0 (secure 0 0 0) 0x7f9ffc02d910 con 0x7fa00c108fb0 2024-04-23T10:17:13.663 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-04-23T10:17:13.087+0000 7fa01091c700 1 -- 172.21.15.114:0/3188535063 >> [v2:172.21.15.114:6800/2831530838,v1:172.21.15.114:6801/2831530838] conn(0x7f9ff4039560 msgr2=0x7f9ff403ba20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T10:17:13.663 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-04-23T10:17:13.087+0000 7fa01091c700 1 --2- 172.21.15.114:0/3188535063 >> [v2:172.21.15.114:6800/2831530838,v1:172.21.15.114:6801/2831530838] conn(0x7f9ff4039560 0x7f9ff403ba20 secure :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0x7fa000006fd0 tx=0x7fa000006e40 comp rx=0 tx=0).stop 2024-04-23T10:17:13.663 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-04-23T10:17:13.087+0000 7fa01091c700 1 -- 172.21.15.114:0/3188535063 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fa00c108fb0 msgr2=0x7fa00c19c450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T10:17:13.663 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-04-23T10:17:13.087+0000 7fa01091c700 1 --2- 172.21.15.114:0/3188535063 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fa00c108fb0 0x7fa00c19c450 secure :-1 s=READY pgs=23 cs=0 l=1 rev1=1 crypto rx=0x7f9ffc003f40 tx=0x7f9ffc011890 comp rx=0 tx=0).stop 2024-04-23T10:17:13.663 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-04-23T10:17:13.087+0000 7fa01091c700 1 -- 172.21.15.114:0/3188535063 shutdown_connections 2024-04-23T10:17:13.663 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-04-23T10:17:13.087+0000 7fa01091c700 1 --2- 172.21.15.114:0/3188535063 >> [v2:172.21.15.114:6800/2831530838,v1:172.21.15.114:6801/2831530838] conn(0x7f9ff4039560 0x7f9ff403ba20 unknown :-1 s=CLOSED pgs=8 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:17:13.663 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-04-23T10:17:13.087+0000 7fa01091c700 1 --2- 172.21.15.114:0/3188535063 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fa00c108fb0 0x7fa00c19c450 unknown :-1 s=CLOSED pgs=23 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:17:13.663 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-04-23T10:17:13.087+0000 7fa01091c700 1 -- 172.21.15.114:0/3188535063 >> 172.21.15.114:0/3188535063 conn(0x7fa00c102a40 msgr2=0x7fa00c1034e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T10:17:13.663 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-04-23T10:17:13.087+0000 7fa01091c700 1 -- 172.21.15.114:0/3188535063 shutdown_connections 2024-04-23T10:17:13.663 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-04-23T10:17:13.087+0000 7fa01091c700 1 -- 172.21.15.114:0/3188535063 wait complete. 2024-04-23T10:17:13.939 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Apr 23 10:17:13 smithi114 ceph-aa240572-0159-11ef-bc93-c7b262605968-mgr-x[29490]: 2024-04-23T10:17:13.851+0000 7f8b5dfee200 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2024-04-23T10:17:13.939 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Apr 23 10:17:13 smithi114 ceph-aa240572-0159-11ef-bc93-c7b262605968-mgr-x[29490]: 2024-04-23T10:17:13.939+0000 7f8b5dfee200 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2024-04-23T10:17:14.254 INFO:journalctl@ceph.mon.a.smithi114.stdout:Apr 23 10:17:14 smithi114 ceph-mon[29255]: from='client.? 172.21.15.114:0/3188535063' entity='client.admin' cmd='[{"prefix": "mgr module enable", "module": "cephadm"}]': finished 2024-04-23T10:17:14.254 INFO:journalctl@ceph.mon.a.smithi114.stdout:Apr 23 10:17:14 smithi114 ceph-mon[29255]: mgrmap e5: x(active, since 6s) 2024-04-23T10:17:14.900 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stdout { 2024-04-23T10:17:14.900 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stdout "epoch": 5, 2024-04-23T10:17:14.900 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stdout "available": true, 2024-04-23T10:17:14.900 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stdout "active_name": "x", 2024-04-23T10:17:14.900 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stdout "num_standby": 0 2024-04-23T10:17:14.900 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stdout } 2024-04-23T10:17:14.900 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-04-23T10:17:14.069+0000 7fed90b5b700 1 Processor -- start 2024-04-23T10:17:14.900 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-04-23T10:17:14.069+0000 7fed90b5b700 1 -- start start 2024-04-23T10:17:14.901 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-04-23T10:17:14.070+0000 7fed90b5b700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fed8c108ff0 0x7fed8c10b3e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T10:17:14.901 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-04-23T10:17:14.070+0000 7fed90b5b700 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x7fed8c063d20 con 0x7fed8c108ff0 2024-04-23T10:17:14.901 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-04-23T10:17:14.070+0000 7fed8a59c700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fed8c108ff0 0x7fed8c10b3e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T10:17:14.901 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-04-23T10:17:14.070+0000 7fed8a59c700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fed8c108ff0 0x7fed8c10b3e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.114:44898/0 (socket says 172.21.15.114:44898) 2024-04-23T10:17:14.901 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-04-23T10:17:14.070+0000 7fed8a59c700 1 -- 172.21.15.114:0/1204411109 learned_addr learned my addr 172.21.15.114:0/1204411109 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-04-23T10:17:14.901 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-04-23T10:17:14.070+0000 7fed8a59c700 1 -- 172.21.15.114:0/1204411109 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fed8c10b920 con 0x7fed8c108ff0 2024-04-23T10:17:14.901 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-04-23T10:17:14.070+0000 7fed8a59c700 1 --2- 172.21.15.114:0/1204411109 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fed8c108ff0 0x7fed8c10b3e0 secure :-1 s=READY pgs=26 cs=0 l=1 rev1=1 crypto rx=0x7fed7c0098a0 tx=0x7fed7c009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e96435bd3e7b012b server_cookie=0 in_seq=0 out_seq=0 2024-04-23T10:17:14.901 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-04-23T10:17:14.071+0000 7fed8959a700 1 -- 172.21.15.114:0/1204411109 <== mon.0 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fed7c004160 con 0x7fed8c108ff0 2024-04-23T10:17:14.901 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-04-23T10:17:14.071+0000 7fed8959a700 1 -- 172.21.15.114:0/1204411109 <== mon.0 v2:172.21.15.114:3300/0 2 ==== config(26 keys) ==== 1070+0+0 (secure 0 0 0) 0x7fed7c004300 con 0x7fed8c108ff0 2024-04-23T10:17:14.901 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-04-23T10:17:14.071+0000 7fed8959a700 1 -- 172.21.15.114:0/1204411109 <== mon.0 v2:172.21.15.114:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fed7c011af0 con 0x7fed8c108ff0 2024-04-23T10:17:14.901 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-04-23T10:17:14.071+0000 7fed90b5b700 1 -- 172.21.15.114:0/1204411109 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fed8c108ff0 msgr2=0x7fed8c10b3e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T10:17:14.901 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-04-23T10:17:14.071+0000 7fed90b5b700 1 --2- 172.21.15.114:0/1204411109 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fed8c108ff0 0x7fed8c10b3e0 secure :-1 s=READY pgs=26 cs=0 l=1 rev1=1 crypto rx=0x7fed7c0098a0 tx=0x7fed7c009bb0 comp rx=0 tx=0).stop 2024-04-23T10:17:14.901 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-04-23T10:17:14.071+0000 7fed90b5b700 1 -- 172.21.15.114:0/1204411109 shutdown_connections 2024-04-23T10:17:14.901 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-04-23T10:17:14.071+0000 7fed90b5b700 1 --2- 172.21.15.114:0/1204411109 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fed8c108ff0 0x7fed8c10b3e0 unknown :-1 s=CLOSED pgs=26 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:17:14.902 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-04-23T10:17:14.071+0000 7fed90b5b700 1 -- 172.21.15.114:0/1204411109 >> 172.21.15.114:0/1204411109 conn(0x7fed8c102a40 msgr2=0x7fed8c104e60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T10:17:14.902 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-04-23T10:17:14.071+0000 7fed90b5b700 1 -- 172.21.15.114:0/1204411109 shutdown_connections 2024-04-23T10:17:14.902 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-04-23T10:17:14.071+0000 7fed90b5b700 1 -- 172.21.15.114:0/1204411109 wait complete. 2024-04-23T10:17:14.902 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-04-23T10:17:14.072+0000 7fed90b5b700 1 Processor -- start 2024-04-23T10:17:14.902 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-04-23T10:17:14.072+0000 7fed90b5b700 1 -- start start 2024-04-23T10:17:14.902 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-04-23T10:17:14.072+0000 7fed90b5b700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fed8c108ff0 0x7fed8c19c430 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T10:17:14.902 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-04-23T10:17:14.072+0000 7fed90b5b700 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x7fed8c10c8e0 con 0x7fed8c108ff0 2024-04-23T10:17:14.902 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-04-23T10:17:14.072+0000 7fed8a59c700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fed8c108ff0 0x7fed8c19c430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T10:17:14.902 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-04-23T10:17:14.072+0000 7fed8a59c700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fed8c108ff0 0x7fed8c19c430 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.114:44904/0 (socket says 172.21.15.114:44904) 2024-04-23T10:17:14.902 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-04-23T10:17:14.073+0000 7fed8a59c700 1 -- 172.21.15.114:0/1631150715 learned_addr learned my addr 172.21.15.114:0/1631150715 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-04-23T10:17:14.902 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-04-23T10:17:14.073+0000 7fed8a59c700 1 -- 172.21.15.114:0/1631150715 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fed8c19c970 con 0x7fed8c108ff0 2024-04-23T10:17:14.902 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-04-23T10:17:14.073+0000 7fed8a59c700 1 --2- 172.21.15.114:0/1631150715 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fed8c108ff0 0x7fed8c19c430 secure :-1 s=READY pgs=27 cs=0 l=1 rev1=1 crypto rx=0x7fed7c011ac0 tx=0x7fed7c011ba0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T10:17:14.902 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-04-23T10:17:14.073+0000 7fed7b7fe700 1 -- 172.21.15.114:0/1631150715 <== mon.0 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fed7c003b20 con 0x7fed8c108ff0 2024-04-23T10:17:14.903 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-04-23T10:17:14.073+0000 7fed90b5b700 1 -- 172.21.15.114:0/1631150715 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fed8c19cc00 con 0x7fed8c108ff0 2024-04-23T10:17:14.903 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-04-23T10:17:14.073+0000 7fed7b7fe700 1 -- 172.21.15.114:0/1631150715 <== mon.0 v2:172.21.15.114:3300/0 2 ==== config(26 keys) ==== 1070+0+0 (secure 0 0 0) 0x7fed7c01a8c0 con 0x7fed8c108ff0 2024-04-23T10:17:14.903 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-04-23T10:17:14.073+0000 7fed7b7fe700 1 -- 172.21.15.114:0/1631150715 <== mon.0 v2:172.21.15.114:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fed7c0248d0 con 0x7fed8c108ff0 2024-04-23T10:17:14.903 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-04-23T10:17:14.073+0000 7fed90b5b700 1 -- 172.21.15.114:0/1631150715 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fed8c19f8f0 con 0x7fed8c108ff0 2024-04-23T10:17:14.903 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-04-23T10:17:14.074+0000 7fed7b7fe700 1 -- 172.21.15.114:0/1631150715 <== mon.0 v2:172.21.15.114:3300/0 4 ==== mgrmap(e 5) ==== 46558+0+0 (secure 0 0 0) 0x7fed7c022020 con 0x7fed8c108ff0 2024-04-23T10:17:14.903 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-04-23T10:17:14.074+0000 7fed7b7fe700 1 --2- 172.21.15.114:0/1631150715 >> [v2:172.21.15.114:6800/2831530838,v1:172.21.15.114:6801/2831530838] conn(0x7fed740395b0 0x7fed7403ba70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T10:17:14.903 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-04-23T10:17:14.074+0000 7fed90b5b700 1 -- 172.21.15.114:0/1631150715 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fed8c0640a0 con 0x7fed8c108ff0 2024-04-23T10:17:14.903 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-04-23T10:17:14.074+0000 7fed7b7fe700 1 -- 172.21.15.114:0/1631150715 <== mon.0 v2:172.21.15.114:3300/0 5 ==== osd_map(1..1 src has 1..1) ==== 725+0+0 (secure 0 0 0) 0x7fed7c04d5a0 con 0x7fed8c108ff0 2024-04-23T10:17:14.903 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-04-23T10:17:14.074+0000 7fed89d9b700 1 -- 172.21.15.114:0/1631150715 >> [v2:172.21.15.114:6800/2831530838,v1:172.21.15.114:6801/2831530838] conn(0x7fed740395b0 msgr2=0x7fed7403ba70 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.114:6800/2831530838 2024-04-23T10:17:14.903 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-04-23T10:17:14.074+0000 7fed89d9b700 1 --2- 172.21.15.114:0/1631150715 >> [v2:172.21.15.114:6800/2831530838,v1:172.21.15.114:6801/2831530838] conn(0x7fed740395b0 0x7fed7403ba70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-04-23T10:17:14.903 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-04-23T10:17:14.079+0000 7fed7b7fe700 1 -- 172.21.15.114:0/1631150715 <== mon.0 v2:172.21.15.114:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7fed7c04d300 con 0x7fed8c108ff0 2024-04-23T10:17:14.903 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-04-23T10:17:14.275+0000 7fed89d9b700 1 -- 172.21.15.114:0/1631150715 >> [v2:172.21.15.114:6800/2831530838,v1:172.21.15.114:6801/2831530838] conn(0x7fed740395b0 msgr2=0x7fed7403ba70 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.114:6800/2831530838 2024-04-23T10:17:14.903 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-04-23T10:17:14.275+0000 7fed89d9b700 1 --2- 172.21.15.114:0/1631150715 >> [v2:172.21.15.114:6800/2831530838,v1:172.21.15.114:6801/2831530838] conn(0x7fed740395b0 0x7fed7403ba70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-04-23T10:17:14.904 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-04-23T10:17:14.369+0000 7fed90b5b700 1 -- 172.21.15.114:0/1631150715 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_command({"prefix": "mgr stat"} v 0) -- 0x7fed8c19fac0 con 0x7fed8c108ff0 2024-04-23T10:17:14.904 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-04-23T10:17:14.373+0000 7fed7b7fe700 1 -- 172.21.15.114:0/1631150715 <== mon.0 v2:172.21.15.114:3300/0 7 ==== mon_command_ack([{"prefix": "mgr stat"}]=0 v5) ==== 56+0+88 (secure 0 0 0) 0x7fed7c00b5a0 con 0x7fed8c108ff0 2024-04-23T10:17:14.904 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-04-23T10:17:14.376+0000 7fed797fa700 1 -- 172.21.15.114:0/1631150715 >> [v2:172.21.15.114:6800/2831530838,v1:172.21.15.114:6801/2831530838] conn(0x7fed740395b0 msgr2=0x7fed7403ba70 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-04-23T10:17:14.904 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-04-23T10:17:14.376+0000 7fed797fa700 1 --2- 172.21.15.114:0/1631150715 >> [v2:172.21.15.114:6800/2831530838,v1:172.21.15.114:6801/2831530838] conn(0x7fed740395b0 0x7fed7403ba70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:17:14.904 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-04-23T10:17:14.376+0000 7fed797fa700 1 -- 172.21.15.114:0/1631150715 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fed8c108ff0 msgr2=0x7fed8c19c430 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T10:17:14.904 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-04-23T10:17:14.376+0000 7fed797fa700 1 --2- 172.21.15.114:0/1631150715 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fed8c108ff0 0x7fed8c19c430 secure :-1 s=READY pgs=27 cs=0 l=1 rev1=1 crypto rx=0x7fed7c011ac0 tx=0x7fed7c011ba0 comp rx=0 tx=0).stop 2024-04-23T10:17:14.904 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-04-23T10:17:14.376+0000 7fed797fa700 1 -- 172.21.15.114:0/1631150715 shutdown_connections 2024-04-23T10:17:14.904 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-04-23T10:17:14.376+0000 7fed797fa700 1 --2- 172.21.15.114:0/1631150715 >> [v2:172.21.15.114:6800/2831530838,v1:172.21.15.114:6801/2831530838] conn(0x7fed740395b0 0x7fed7403ba70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:17:14.904 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-04-23T10:17:14.376+0000 7fed797fa700 1 --2- 172.21.15.114:0/1631150715 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fed8c108ff0 0x7fed8c19c430 unknown :-1 s=CLOSED pgs=27 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:17:14.904 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-04-23T10:17:14.376+0000 7fed797fa700 1 -- 172.21.15.114:0/1631150715 >> 172.21.15.114:0/1631150715 conn(0x7fed8c102a40 msgr2=0x7fed8c103530 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T10:17:14.904 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-04-23T10:17:14.376+0000 7fed797fa700 1 -- 172.21.15.114:0/1631150715 shutdown_connections 2024-04-23T10:17:14.904 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-04-23T10:17:14.376+0000 7fed797fa700 1 -- 172.21.15.114:0/1631150715 wait complete. 2024-04-23T10:17:14.904 INFO:teuthology.orchestra.run.smithi114.stdout:Waiting for the mgr to restart... 2024-04-23T10:17:14.904 INFO:teuthology.orchestra.run.smithi114.stdout:Waiting for mgr epoch 5... 2024-04-23T10:17:15.285 INFO:journalctl@ceph.mon.a.smithi114.stdout:Apr 23 10:17:15 smithi114 ceph-mon[29255]: from='client.? 172.21.15.114:0/1631150715' entity='client.admin' cmd=[{"prefix": "mgr stat"}]: dispatch 2024-04-23T10:17:15.285 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Apr 23 10:17:15 smithi114 ceph-aa240572-0159-11ef-bc93-c7b262605968-mgr-x[29490]: 2024-04-23T10:17:15.146+0000 7f8b5dfee200 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2024-04-23T10:17:15.548 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Apr 23 10:17:15 smithi114 ceph-aa240572-0159-11ef-bc93-c7b262605968-mgr-x[29490]: 2024-04-23T10:17:15.326+0000 7f8b5dfee200 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2024-04-23T10:17:15.548 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Apr 23 10:17:15 smithi114 ceph-aa240572-0159-11ef-bc93-c7b262605968-mgr-x[29490]: 2024-04-23T10:17:15.504+0000 7f8b5dfee200 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2024-04-23T10:17:16.106 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Apr 23 10:17:15 smithi114 ceph-aa240572-0159-11ef-bc93-c7b262605968-mgr-x[29490]: 2024-04-23T10:17:15.805+0000 7f8b5dfee200 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2024-04-23T10:17:16.106 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Apr 23 10:17:15 smithi114 ceph-aa240572-0159-11ef-bc93-c7b262605968-mgr-x[29490]: 2024-04-23T10:17:15.899+0000 7f8b5dfee200 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2024-04-23T10:17:16.106 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Apr 23 10:17:16 smithi114 ceph-aa240572-0159-11ef-bc93-c7b262605968-mgr-x[29490]: 2024-04-23T10:17:16.106+0000 7f8b5dfee200 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2024-04-23T10:17:16.504 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Apr 23 10:17:16 smithi114 ceph-aa240572-0159-11ef-bc93-c7b262605968-mgr-x[29490]: 2024-04-23T10:17:16.304+0000 7f8b5dfee200 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2024-04-23T10:17:17.004 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Apr 23 10:17:16 smithi114 ceph-aa240572-0159-11ef-bc93-c7b262605968-mgr-x[29490]: 2024-04-23T10:17:16.900+0000 7f8b5dfee200 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2024-04-23T10:17:17.613 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Apr 23 10:17:17 smithi114 ceph-aa240572-0159-11ef-bc93-c7b262605968-mgr-x[29490]: 2024-04-23T10:17:17.317+0000 7f8b5dfee200 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2024-04-23T10:17:17.613 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Apr 23 10:17:17 smithi114 ceph-aa240572-0159-11ef-bc93-c7b262605968-mgr-x[29490]: 2024-04-23T10:17:17.412+0000 7f8b5dfee200 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2024-04-23T10:17:17.613 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Apr 23 10:17:17 smithi114 ceph-aa240572-0159-11ef-bc93-c7b262605968-mgr-x[29490]: 2024-04-23T10:17:17.613+0000 7f8b5dfee200 -1 mgr[py] Module rgw has missing NOTIFY_TYPES member 2024-04-23T10:17:18.462 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Apr 23 10:17:18 smithi114 ceph-aa240572-0159-11ef-bc93-c7b262605968-mgr-x[29490]: 2024-04-23T10:17:18.129+0000 7f8b5dfee200 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2024-04-23T10:17:18.462 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Apr 23 10:17:18 smithi114 ceph-aa240572-0159-11ef-bc93-c7b262605968-mgr-x[29490]: 2024-04-23T10:17:18.221+0000 7f8b5dfee200 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2024-04-23T10:17:18.754 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Apr 23 10:17:18 smithi114 ceph-aa240572-0159-11ef-bc93-c7b262605968-mgr-x[29490]: 2024-04-23T10:17:18.462+0000 7f8b5dfee200 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2024-04-23T10:17:18.754 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Apr 23 10:17:18 smithi114 ceph-aa240572-0159-11ef-bc93-c7b262605968-mgr-x[29490]: 2024-04-23T10:17:18.594+0000 7f8b5dfee200 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2024-04-23T10:17:18.754 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Apr 23 10:17:18 smithi114 ceph-aa240572-0159-11ef-bc93-c7b262605968-mgr-x[29490]: 2024-04-23T10:17:18.689+0000 7f8b5dfee200 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2024-04-23T10:17:19.103 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Apr 23 10:17:18 smithi114 ceph-aa240572-0159-11ef-bc93-c7b262605968-mgr-x[29490]: 2024-04-23T10:17:18.803+0000 7f8b5dfee200 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2024-04-23T10:17:19.960 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Apr 23 10:17:19 smithi114 ceph-aa240572-0159-11ef-bc93-c7b262605968-mgr-x[29490]: 2024-04-23T10:17:19.103+0000 7f8b5dfee200 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2024-04-23T10:17:19.960 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Apr 23 10:17:19 smithi114 ceph-aa240572-0159-11ef-bc93-c7b262605968-mgr-x[29490]: 2024-04-23T10:17:19.695+0000 7f8b5dfee200 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2024-04-23T10:17:19.960 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Apr 23 10:17:19 smithi114 ceph-aa240572-0159-11ef-bc93-c7b262605968-mgr-x[29490]: 2024-04-23T10:17:19.908+0000 7f8b5dfee200 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2024-04-23T10:17:20.254 INFO:journalctl@ceph.mon.a.smithi114.stdout:Apr 23 10:17:19 smithi114 ceph-mon[29255]: Active manager daemon x restarted 2024-04-23T10:17:20.254 INFO:journalctl@ceph.mon.a.smithi114.stdout:Apr 23 10:17:19 smithi114 ceph-mon[29255]: Activating manager daemon x 2024-04-23T10:17:20.254 INFO:journalctl@ceph.mon.a.smithi114.stdout:Apr 23 10:17:19 smithi114 ceph-mon[29255]: osdmap e2: 0 total, 0 up, 0 in 2024-04-23T10:17:20.254 INFO:journalctl@ceph.mon.a.smithi114.stdout:Apr 23 10:17:19 smithi114 ceph-mon[29255]: mgrmap e6: x(active, starting, since 0.00331712s) 2024-04-23T10:17:20.255 INFO:journalctl@ceph.mon.a.smithi114.stdout:Apr 23 10:17:19 smithi114 ceph-mon[29255]: from='mgr.14120 172.21.15.114:0/1448995927' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-04-23T10:17:20.255 INFO:journalctl@ceph.mon.a.smithi114.stdout:Apr 23 10:17:19 smithi114 ceph-mon[29255]: from='mgr.14120 172.21.15.114:0/1448995927' entity='mgr.x' cmd=[{"prefix": "mgr metadata", "who": "x", "id": "x"}]: dispatch 2024-04-23T10:17:20.255 INFO:journalctl@ceph.mon.a.smithi114.stdout:Apr 23 10:17:19 smithi114 ceph-mon[29255]: from='mgr.14120 172.21.15.114:0/1448995927' entity='mgr.x' cmd=[{"prefix": "mds metadata"}]: dispatch 2024-04-23T10:17:20.255 INFO:journalctl@ceph.mon.a.smithi114.stdout:Apr 23 10:17:19 smithi114 ceph-mon[29255]: from='mgr.14120 172.21.15.114:0/1448995927' entity='mgr.x' cmd=[{"prefix": "osd metadata"}]: dispatch 2024-04-23T10:17:20.255 INFO:journalctl@ceph.mon.a.smithi114.stdout:Apr 23 10:17:19 smithi114 ceph-mon[29255]: from='mgr.14120 172.21.15.114:0/1448995927' entity='mgr.x' cmd=[{"prefix": "mon metadata"}]: dispatch 2024-04-23T10:17:20.255 INFO:journalctl@ceph.mon.a.smithi114.stdout:Apr 23 10:17:19 smithi114 ceph-mon[29255]: Manager daemon x is now available 2024-04-23T10:17:20.255 INFO:journalctl@ceph.mon.a.smithi114.stdout:Apr 23 10:17:19 smithi114 ceph-mon[29255]: from='mgr.14120 172.21.15.114:0/1448995927' entity='mgr.x' 2024-04-23T10:17:20.255 INFO:journalctl@ceph.mon.a.smithi114.stdout:Apr 23 10:17:19 smithi114 ceph-mon[29255]: from='mgr.14120 172.21.15.114:0/1448995927' entity='mgr.x' 2024-04-23T10:17:20.255 INFO:journalctl@ceph.mon.a.smithi114.stdout:Apr 23 10:17:19 smithi114 ceph-mon[29255]: from='mgr.14120 172.21.15.114:0/1448995927' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-23T10:17:20.255 INFO:journalctl@ceph.mon.a.smithi114.stdout:Apr 23 10:17:19 smithi114 ceph-mon[29255]: from='mgr.14120 172.21.15.114:0/1448995927' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-23T10:17:20.255 INFO:journalctl@ceph.mon.a.smithi114.stdout:Apr 23 10:17:19 smithi114 ceph-mon[29255]: from='mgr.14120 172.21.15.114:0/1448995927' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2024-04-23T10:17:20.255 INFO:journalctl@ceph.mon.a.smithi114.stdout:Apr 23 10:17:19 smithi114 ceph-mon[29255]: from='mgr.14120 172.21.15.114:0/1448995927' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2024-04-23T10:17:21.254 INFO:journalctl@ceph.mon.a.smithi114.stdout:Apr 23 10:17:21 smithi114 ceph-mon[29255]: Found migration_current of "None". Setting to last migration. 2024-04-23T10:17:21.254 INFO:journalctl@ceph.mon.a.smithi114.stdout:Apr 23 10:17:21 smithi114 ceph-mon[29255]: from='mgr.14120 172.21.15.114:0/1448995927' entity='mgr.x' 2024-04-23T10:17:21.254 INFO:journalctl@ceph.mon.a.smithi114.stdout:Apr 23 10:17:21 smithi114 ceph-mon[29255]: from='mgr.14120 172.21.15.114:0/1448995927' entity='mgr.x' 2024-04-23T10:17:21.254 INFO:journalctl@ceph.mon.a.smithi114.stdout:Apr 23 10:17:21 smithi114 ceph-mon[29255]: mgrmap e7: x(active, since 1.0054s) 2024-04-23T10:17:21.516 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stdout { 2024-04-23T10:17:21.524 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stdout "mgrmap_epoch": 7, 2024-04-23T10:17:21.524 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stdout "initialized": true 2024-04-23T10:17:21.524 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stdout } 2024-04-23T10:17:21.524 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-04-23T10:17:15.313+0000 7fbd6f36c700 1 Processor -- start 2024-04-23T10:17:21.524 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-04-23T10:17:15.313+0000 7fbd6f36c700 1 -- start start 2024-04-23T10:17:21.524 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-04-23T10:17:15.314+0000 7fbd6f36c700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fbd68107440 0x7fbd68107860 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T10:17:21.524 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-04-23T10:17:15.314+0000 7fbd6f36c700 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x7fbd68107e30 con 0x7fbd68107440 2024-04-23T10:17:21.524 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-04-23T10:17:15.314+0000 7fbd6d108700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fbd68107440 0x7fbd68107860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T10:17:21.524 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-04-23T10:17:15.314+0000 7fbd6d108700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fbd68107440 0x7fbd68107860 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.114:44916/0 (socket says 172.21.15.114:44916) 2024-04-23T10:17:21.525 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-04-23T10:17:15.314+0000 7fbd6d108700 1 -- 172.21.15.114:0/3966443466 learned_addr learned my addr 172.21.15.114:0/3966443466 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-04-23T10:17:21.525 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-04-23T10:17:15.314+0000 7fbd6d108700 1 -- 172.21.15.114:0/3966443466 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fbd68108640 con 0x7fbd68107440 2024-04-23T10:17:21.525 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-04-23T10:17:15.314+0000 7fbd6d108700 1 --2- 172.21.15.114:0/3966443466 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fbd68107440 0x7fbd68107860 secure :-1 s=READY pgs=28 cs=0 l=1 rev1=1 crypto rx=0x7fbd640098a0 tx=0x7fbd64009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1b2019d9448222d9 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T10:17:21.525 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-04-23T10:17:15.314+0000 7fbd5ffff700 1 -- 172.21.15.114:0/3966443466 <== mon.0 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fbd64004160 con 0x7fbd68107440 2024-04-23T10:17:21.525 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-04-23T10:17:15.314+0000 7fbd5ffff700 1 -- 172.21.15.114:0/3966443466 <== mon.0 v2:172.21.15.114:3300/0 2 ==== config(26 keys) ==== 1070+0+0 (secure 0 0 0) 0x7fbd64004300 con 0x7fbd68107440 2024-04-23T10:17:21.525 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-04-23T10:17:15.314+0000 7fbd5ffff700 1 -- 172.21.15.114:0/3966443466 <== mon.0 v2:172.21.15.114:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fbd64011af0 con 0x7fbd68107440 2024-04-23T10:17:21.525 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-04-23T10:17:15.315+0000 7fbd6f36c700 1 -- 172.21.15.114:0/3966443466 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fbd68107440 msgr2=0x7fbd68107860 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T10:17:21.525 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-04-23T10:17:15.315+0000 7fbd6f36c700 1 --2- 172.21.15.114:0/3966443466 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fbd68107440 0x7fbd68107860 secure :-1 s=READY pgs=28 cs=0 l=1 rev1=1 crypto rx=0x7fbd640098a0 tx=0x7fbd64009bb0 comp rx=0 tx=0).stop 2024-04-23T10:17:21.525 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-04-23T10:17:15.315+0000 7fbd6f36c700 1 -- 172.21.15.114:0/3966443466 shutdown_connections 2024-04-23T10:17:21.525 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-04-23T10:17:15.315+0000 7fbd6f36c700 1 --2- 172.21.15.114:0/3966443466 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fbd68107440 0x7fbd68107860 unknown :-1 s=CLOSED pgs=28 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:17:21.525 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-04-23T10:17:15.315+0000 7fbd6f36c700 1 -- 172.21.15.114:0/3966443466 >> 172.21.15.114:0/3966443466 conn(0x7fbd68102a40 msgr2=0x7fbd68104e60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T10:17:21.525 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-04-23T10:17:15.315+0000 7fbd6f36c700 1 -- 172.21.15.114:0/3966443466 shutdown_connections 2024-04-23T10:17:21.525 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-04-23T10:17:15.315+0000 7fbd6f36c700 1 -- 172.21.15.114:0/3966443466 wait complete. 2024-04-23T10:17:21.525 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-04-23T10:17:15.316+0000 7fbd6f36c700 1 Processor -- start 2024-04-23T10:17:21.526 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-04-23T10:17:15.316+0000 7fbd6f36c700 1 -- start start 2024-04-23T10:17:21.526 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-04-23T10:17:15.316+0000 7fbd6f36c700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fbd68107440 0x7fbd6819a280 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T10:17:21.526 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-04-23T10:17:15.316+0000 7fbd6f36c700 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x7fbd68062e80 con 0x7fbd68107440 2024-04-23T10:17:21.526 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-04-23T10:17:15.316+0000 7fbd6d108700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fbd68107440 0x7fbd6819a280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T10:17:21.526 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-04-23T10:17:15.316+0000 7fbd6d108700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fbd68107440 0x7fbd6819a280 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.114:44920/0 (socket says 172.21.15.114:44920) 2024-04-23T10:17:21.526 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-04-23T10:17:15.316+0000 7fbd6d108700 1 -- 172.21.15.114:0/12088556 learned_addr learned my addr 172.21.15.114:0/12088556 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-04-23T10:17:21.526 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-04-23T10:17:15.317+0000 7fbd6d108700 1 -- 172.21.15.114:0/12088556 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fbd6819a7c0 con 0x7fbd68107440 2024-04-23T10:17:21.526 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-04-23T10:17:15.317+0000 7fbd6d108700 1 --2- 172.21.15.114:0/12088556 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fbd68107440 0x7fbd6819a280 secure :-1 s=READY pgs=29 cs=0 l=1 rev1=1 crypto rx=0x7fbd64000c00 tx=0x7fbd64011b70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T10:17:21.526 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-04-23T10:17:15.317+0000 7fbd5e7fc700 1 -- 172.21.15.114:0/12088556 <== mon.0 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fbd64003b20 con 0x7fbd68107440 2024-04-23T10:17:21.526 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-04-23T10:17:15.317+0000 7fbd5e7fc700 1 -- 172.21.15.114:0/12088556 <== mon.0 v2:172.21.15.114:3300/0 2 ==== config(26 keys) ==== 1070+0+0 (secure 0 0 0) 0x7fbd6401a8c0 con 0x7fbd68107440 2024-04-23T10:17:21.526 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-04-23T10:17:15.317+0000 7fbd5e7fc700 1 -- 172.21.15.114:0/12088556 <== mon.0 v2:172.21.15.114:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fbd64024940 con 0x7fbd68107440 2024-04-23T10:17:21.526 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-04-23T10:17:15.317+0000 7fbd6f36c700 1 -- 172.21.15.114:0/12088556 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fbd6819aa50 con 0x7fbd68107440 2024-04-23T10:17:21.526 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-04-23T10:17:15.317+0000 7fbd6f36c700 1 -- 172.21.15.114:0/12088556 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fbd6819d740 con 0x7fbd68107440 2024-04-23T10:17:21.526 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-04-23T10:17:15.318+0000 7fbd5e7fc700 1 -- 172.21.15.114:0/12088556 <== mon.0 v2:172.21.15.114:3300/0 4 ==== mgrmap(e 5) ==== 46558+0+0 (secure 0 0 0) 0x7fbd64022020 con 0x7fbd68107440 2024-04-23T10:17:21.526 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-04-23T10:17:15.318+0000 7fbd5e7fc700 1 --2- 172.21.15.114:0/12088556 >> [v2:172.21.15.114:6800/2831530838,v1:172.21.15.114:6801/2831530838] conn(0x7fbd5403b810 0x7fbd5403dcd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T10:17:21.527 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-04-23T10:17:15.318+0000 7fbd5e7fc700 1 -- 172.21.15.114:0/12088556 --> [v2:172.21.15.114:6800/2831530838,v1:172.21.15.114:6801/2831530838] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x7fbd5403e3e0 con 0x7fbd5403b810 2024-04-23T10:17:21.527 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-04-23T10:17:15.318+0000 7fbd5e7fc700 1 -- 172.21.15.114:0/12088556 <== mon.0 v2:172.21.15.114:3300/0 5 ==== osd_map(1..1 src has 1..1) ==== 725+0+0 (secure 0 0 0) 0x7fbd6404d460 con 0x7fbd68107440 2024-04-23T10:17:21.527 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-04-23T10:17:15.318+0000 7fbd6c907700 1 -- 172.21.15.114:0/12088556 >> [v2:172.21.15.114:6800/2831530838,v1:172.21.15.114:6801/2831530838] conn(0x7fbd5403b810 msgr2=0x7fbd5403dcd0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.114:6800/2831530838 2024-04-23T10:17:21.527 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-04-23T10:17:15.318+0000 7fbd6c907700 1 --2- 172.21.15.114:0/12088556 >> [v2:172.21.15.114:6800/2831530838,v1:172.21.15.114:6801/2831530838] conn(0x7fbd5403b810 0x7fbd5403dcd0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-04-23T10:17:21.527 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-04-23T10:17:15.519+0000 7fbd6c907700 1 -- 172.21.15.114:0/12088556 >> [v2:172.21.15.114:6800/2831530838,v1:172.21.15.114:6801/2831530838] conn(0x7fbd5403b810 msgr2=0x7fbd5403dcd0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.114:6800/2831530838 2024-04-23T10:17:21.527 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-04-23T10:17:15.519+0000 7fbd6c907700 1 --2- 172.21.15.114:0/12088556 >> [v2:172.21.15.114:6800/2831530838,v1:172.21.15.114:6801/2831530838] conn(0x7fbd5403b810 0x7fbd5403dcd0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-04-23T10:17:21.527 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-04-23T10:17:15.920+0000 7fbd6c907700 1 -- 172.21.15.114:0/12088556 >> [v2:172.21.15.114:6800/2831530838,v1:172.21.15.114:6801/2831530838] conn(0x7fbd5403b810 msgr2=0x7fbd5403dcd0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.114:6800/2831530838 2024-04-23T10:17:21.527 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-04-23T10:17:15.920+0000 7fbd6c907700 1 --2- 172.21.15.114:0/12088556 >> [v2:172.21.15.114:6800/2831530838,v1:172.21.15.114:6801/2831530838] conn(0x7fbd5403b810 0x7fbd5403dcd0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2024-04-23T10:17:21.527 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-04-23T10:17:16.721+0000 7fbd6c907700 1 -- 172.21.15.114:0/12088556 >> [v2:172.21.15.114:6800/2831530838,v1:172.21.15.114:6801/2831530838] conn(0x7fbd5403b810 msgr2=0x7fbd5403dcd0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.114:6800/2831530838 2024-04-23T10:17:21.527 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-04-23T10:17:16.721+0000 7fbd6c907700 1 --2- 172.21.15.114:0/12088556 >> [v2:172.21.15.114:6800/2831530838,v1:172.21.15.114:6801/2831530838] conn(0x7fbd5403b810 0x7fbd5403dcd0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2024-04-23T10:17:21.527 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-04-23T10:17:18.322+0000 7fbd6c907700 1 -- 172.21.15.114:0/12088556 >> [v2:172.21.15.114:6800/2831530838,v1:172.21.15.114:6801/2831530838] conn(0x7fbd5403b810 msgr2=0x7fbd5403dcd0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.114:6800/2831530838 2024-04-23T10:17:21.527 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-04-23T10:17:18.322+0000 7fbd6c907700 1 --2- 172.21.15.114:0/12088556 >> [v2:172.21.15.114:6800/2831530838,v1:172.21.15.114:6801/2831530838] conn(0x7fbd5403b810 0x7fbd5403dcd0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 3.200000 2024-04-23T10:17:21.527 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-04-23T10:17:19.911+0000 7fbd5e7fc700 1 -- 172.21.15.114:0/12088556 <== mon.0 v2:172.21.15.114:3300/0 6 ==== mgrmap(e 6) ==== 46325+0+0 (secure 0 0 0) 0x7fbd6402d3f0 con 0x7fbd68107440 2024-04-23T10:17:21.527 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-04-23T10:17:19.911+0000 7fbd5e7fc700 1 -- 172.21.15.114:0/12088556 >> [v2:172.21.15.114:6800/2831530838,v1:172.21.15.114:6801/2831530838] conn(0x7fbd5403b810 msgr2=0x7fbd5403dcd0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-04-23T10:17:21.527 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-04-23T10:17:19.911+0000 7fbd5e7fc700 1 --2- 172.21.15.114:0/12088556 >> [v2:172.21.15.114:6800/2831530838,v1:172.21.15.114:6801/2831530838] conn(0x7fbd5403b810 0x7fbd5403dcd0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:17:21.528 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-04-23T10:17:20.912+0000 7fbd5e7fc700 1 -- 172.21.15.114:0/12088556 <== mon.0 v2:172.21.15.114:3300/0 7 ==== mgrmap(e 7) ==== 46452+0+0 (secure 0 0 0) 0x7fbd6401aa70 con 0x7fbd68107440 2024-04-23T10:17:21.528 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-04-23T10:17:20.913+0000 7fbd5e7fc700 1 --2- 172.21.15.114:0/12088556 >> [v2:172.21.15.114:6800/1604678557,v1:172.21.15.114:6801/1604678557] conn(0x7fbd5403efc0 0x7fbd540413b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T10:17:21.528 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-04-23T10:17:20.913+0000 7fbd5e7fc700 1 -- 172.21.15.114:0/12088556 --> [v2:172.21.15.114:6800/1604678557,v1:172.21.15.114:6801/1604678557] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x7fbd6402ae50 con 0x7fbd5403efc0 2024-04-23T10:17:21.528 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-04-23T10:17:20.913+0000 7fbd6c907700 1 --2- 172.21.15.114:0/12088556 >> [v2:172.21.15.114:6800/1604678557,v1:172.21.15.114:6801/1604678557] conn(0x7fbd5403efc0 0x7fbd540413b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T10:17:21.528 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-04-23T10:17:20.914+0000 7fbd6c907700 1 --2- 172.21.15.114:0/12088556 >> [v2:172.21.15.114:6800/1604678557,v1:172.21.15.114:6801/1604678557] conn(0x7fbd5403efc0 0x7fbd540413b0 secure :-1 s=READY pgs=2 cs=0 l=1 rev1=1 crypto rx=0x7fbd58003a10 tx=0x7fbd58009290 comp rx=0 tx=0).ready entity=mgr.14120 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T10:17:21.528 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-04-23T10:17:20.914+0000 7fbd5e7fc700 1 -- 172.21.15.114:0/12088556 <== mgr.14120 v2:172.21.15.114:6800/1604678557 1 ==== command_reply(tid 0: 0 ) ==== 8+0+7774 (secure 0 0 0) 0x7fbd6402ae50 con 0x7fbd5403efc0 2024-04-23T10:17:21.528 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-04-23T10:17:20.921+0000 7fbd6f36c700 1 -- 172.21.15.114:0/12088556 --> [v2:172.21.15.114:6800/1604678557,v1:172.21.15.114:6801/1604678557] -- command(tid 1: {"prefix": "mgr_status"}) -- 0x7fbd68069ba0 con 0x7fbd5403efc0 2024-04-23T10:17:21.528 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-04-23T10:17:20.921+0000 7fbd5e7fc700 1 -- 172.21.15.114:0/12088556 <== mgr.14120 v2:172.21.15.114:6800/1604678557 2 ==== command_reply(tid 1: 0 ) ==== 8+0+51 (secure 0 0 0) 0x7fbd68069ba0 con 0x7fbd5403efc0 2024-04-23T10:17:21.528 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-04-23T10:17:20.921+0000 7fbd6f36c700 1 -- 172.21.15.114:0/12088556 >> [v2:172.21.15.114:6800/1604678557,v1:172.21.15.114:6801/1604678557] conn(0x7fbd5403efc0 msgr2=0x7fbd540413b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T10:17:21.528 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-04-23T10:17:20.921+0000 7fbd6f36c700 1 --2- 172.21.15.114:0/12088556 >> [v2:172.21.15.114:6800/1604678557,v1:172.21.15.114:6801/1604678557] conn(0x7fbd5403efc0 0x7fbd540413b0 secure :-1 s=READY pgs=2 cs=0 l=1 rev1=1 crypto rx=0x7fbd58003a10 tx=0x7fbd58009290 comp rx=0 tx=0).stop 2024-04-23T10:17:21.528 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-04-23T10:17:20.921+0000 7fbd6f36c700 1 -- 172.21.15.114:0/12088556 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fbd68107440 msgr2=0x7fbd6819a280 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T10:17:21.528 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-04-23T10:17:20.921+0000 7fbd6f36c700 1 --2- 172.21.15.114:0/12088556 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fbd68107440 0x7fbd6819a280 secure :-1 s=READY pgs=29 cs=0 l=1 rev1=1 crypto rx=0x7fbd64000c00 tx=0x7fbd64011b70 comp rx=0 tx=0).stop 2024-04-23T10:17:21.528 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-04-23T10:17:20.921+0000 7fbd6f36c700 1 -- 172.21.15.114:0/12088556 shutdown_connections 2024-04-23T10:17:21.528 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-04-23T10:17:20.921+0000 7fbd6f36c700 1 --2- 172.21.15.114:0/12088556 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fbd68107440 0x7fbd6819a280 unknown :-1 s=CLOSED pgs=29 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:17:21.528 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-04-23T10:17:20.921+0000 7fbd6f36c700 1 --2- 172.21.15.114:0/12088556 >> [v2:172.21.15.114:6800/1604678557,v1:172.21.15.114:6801/1604678557] conn(0x7fbd5403efc0 0x7fbd540413b0 unknown :-1 s=CLOSED pgs=2 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:17:21.529 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-04-23T10:17:20.921+0000 7fbd6f36c700 1 --2- 172.21.15.114:0/12088556 >> [v2:172.21.15.114:6800/2831530838,v1:172.21.15.114:6801/2831530838] conn(0x7fbd5403b810 0x7fbd5403dcd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:17:21.529 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-04-23T10:17:20.921+0000 7fbd6f36c700 1 -- 172.21.15.114:0/12088556 >> 172.21.15.114:0/12088556 conn(0x7fbd68102a40 msgr2=0x7fbd681908c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T10:17:21.529 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-04-23T10:17:20.921+0000 7fbd6f36c700 1 -- 172.21.15.114:0/12088556 shutdown_connections 2024-04-23T10:17:21.529 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-04-23T10:17:20.921+0000 7fbd6f36c700 1 -- 172.21.15.114:0/12088556 wait complete. 2024-04-23T10:17:21.529 INFO:teuthology.orchestra.run.smithi114.stdout:mgr epoch 5 is available 2024-04-23T10:17:21.529 INFO:teuthology.orchestra.run.smithi114.stdout:Setting orchestrator backend to cephadm... 2024-04-23T10:17:22.178 INFO:journalctl@ceph.mon.a.smithi114.stdout:Apr 23 10:17:22 smithi114 ceph-mon[29255]: from='client.14124 -' entity='client.admin' cmd=[{"prefix": "get_command_descriptions"}]: dispatch 2024-04-23T10:17:22.179 INFO:journalctl@ceph.mon.a.smithi114.stdout:Apr 23 10:17:22 smithi114 ceph-mon[29255]: from='client.14124 -' entity='client.admin' cmd=[{"prefix": "mgr_status"}]: dispatch 2024-04-23T10:17:22.761 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-04-23T10:17:21.951+0000 7fc1ba2ea700 1 Processor -- start 2024-04-23T10:17:22.761 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-04-23T10:17:21.951+0000 7fc1ba2ea700 1 -- start start 2024-04-23T10:17:22.761 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-04-23T10:17:21.951+0000 7fc1ba2ea700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fc1b4108530 0x7fc1b4108950 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T10:17:22.761 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-04-23T10:17:21.951+0000 7fc1ba2ea700 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x7fc1b4108f20 con 0x7fc1b4108530 2024-04-23T10:17:22.761 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-04-23T10:17:21.952+0000 7fc1b3fff700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fc1b4108530 0x7fc1b4108950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T10:17:22.762 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-04-23T10:17:21.952+0000 7fc1b3fff700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fc1b4108530 0x7fc1b4108950 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.114:47264/0 (socket says 172.21.15.114:47264) 2024-04-23T10:17:22.762 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-04-23T10:17:21.952+0000 7fc1b3fff700 1 -- 172.21.15.114:0/946149968 learned_addr learned my addr 172.21.15.114:0/946149968 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-04-23T10:17:22.762 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-04-23T10:17:21.952+0000 7fc1b3fff700 1 -- 172.21.15.114:0/946149968 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc1b4109730 con 0x7fc1b4108530 2024-04-23T10:17:22.762 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-04-23T10:17:21.952+0000 7fc1b3fff700 1 --2- 172.21.15.114:0/946149968 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fc1b4108530 0x7fc1b4108950 secure :-1 s=READY pgs=37 cs=0 l=1 rev1=1 crypto rx=0x7fc19c0098a0 tx=0x7fc19c009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=476c1f0cf925f09e server_cookie=0 in_seq=0 out_seq=0 2024-04-23T10:17:22.762 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-04-23T10:17:21.952+0000 7fc1b2ffd700 1 -- 172.21.15.114:0/946149968 <== mon.0 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fc19c004160 con 0x7fc1b4108530 2024-04-23T10:17:22.762 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-04-23T10:17:21.952+0000 7fc1b2ffd700 1 -- 172.21.15.114:0/946149968 <== mon.0 v2:172.21.15.114:3300/0 2 ==== config(26 keys) ==== 1070+0+0 (secure 0 0 0) 0x7fc19c004300 con 0x7fc1b4108530 2024-04-23T10:17:22.762 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-04-23T10:17:21.952+0000 7fc1b2ffd700 1 -- 172.21.15.114:0/946149968 <== mon.0 v2:172.21.15.114:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fc19c011af0 con 0x7fc1b4108530 2024-04-23T10:17:22.762 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-04-23T10:17:21.953+0000 7fc1ba2ea700 1 -- 172.21.15.114:0/946149968 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fc1b4108530 msgr2=0x7fc1b4108950 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T10:17:22.762 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-04-23T10:17:21.953+0000 7fc1ba2ea700 1 --2- 172.21.15.114:0/946149968 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fc1b4108530 0x7fc1b4108950 secure :-1 s=READY pgs=37 cs=0 l=1 rev1=1 crypto rx=0x7fc19c0098a0 tx=0x7fc19c009bb0 comp rx=0 tx=0).stop 2024-04-23T10:17:22.762 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-04-23T10:17:21.953+0000 7fc1ba2ea700 1 -- 172.21.15.114:0/946149968 shutdown_connections 2024-04-23T10:17:22.762 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-04-23T10:17:21.953+0000 7fc1ba2ea700 1 --2- 172.21.15.114:0/946149968 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fc1b4108530 0x7fc1b4108950 unknown :-1 s=CLOSED pgs=37 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:17:22.762 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-04-23T10:17:21.953+0000 7fc1ba2ea700 1 -- 172.21.15.114:0/946149968 >> 172.21.15.114:0/946149968 conn(0x7fc1b4103b50 msgr2=0x7fc1b4105f70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T10:17:22.762 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-04-23T10:17:21.953+0000 7fc1ba2ea700 1 -- 172.21.15.114:0/946149968 shutdown_connections 2024-04-23T10:17:22.762 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-04-23T10:17:21.953+0000 7fc1ba2ea700 1 -- 172.21.15.114:0/946149968 wait complete. 2024-04-23T10:17:22.763 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-04-23T10:17:21.953+0000 7fc1ba2ea700 1 Processor -- start 2024-04-23T10:17:22.763 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-04-23T10:17:21.953+0000 7fc1ba2ea700 1 -- start start 2024-04-23T10:17:22.763 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-04-23T10:17:21.954+0000 7fc1ba2ea700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fc1b4108530 0x7fc1b407ca70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T10:17:22.763 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-04-23T10:17:21.954+0000 7fc1ba2ea700 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x7fc1b410cae0 con 0x7fc1b4108530 2024-04-23T10:17:22.763 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-04-23T10:17:21.954+0000 7fc1b3fff700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fc1b4108530 0x7fc1b407ca70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T10:17:22.763 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-04-23T10:17:21.954+0000 7fc1b3fff700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fc1b4108530 0x7fc1b407ca70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.114:47276/0 (socket says 172.21.15.114:47276) 2024-04-23T10:17:22.763 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-04-23T10:17:21.954+0000 7fc1b3fff700 1 -- 172.21.15.114:0/2466695093 learned_addr learned my addr 172.21.15.114:0/2466695093 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-04-23T10:17:22.763 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-04-23T10:17:21.954+0000 7fc1b3fff700 1 -- 172.21.15.114:0/2466695093 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc1b407cfb0 con 0x7fc1b4108530 2024-04-23T10:17:22.763 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-04-23T10:17:21.954+0000 7fc1b3fff700 1 --2- 172.21.15.114:0/2466695093 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fc1b4108530 0x7fc1b407ca70 secure :-1 s=READY pgs=38 cs=0 l=1 rev1=1 crypto rx=0x7fc19c003f40 tx=0x7fc19c011890 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T10:17:22.763 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-04-23T10:17:21.954+0000 7fc1b17fa700 1 -- 172.21.15.114:0/2466695093 <== mon.0 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fc19c0038b0 con 0x7fc1b4108530 2024-04-23T10:17:22.763 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-04-23T10:17:21.954+0000 7fc1b17fa700 1 -- 172.21.15.114:0/2466695093 <== mon.0 v2:172.21.15.114:3300/0 2 ==== config(26 keys) ==== 1070+0+0 (secure 0 0 0) 0x7fc19c003a50 con 0x7fc1b4108530 2024-04-23T10:17:22.763 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-04-23T10:17:21.954+0000 7fc1b17fa700 1 -- 172.21.15.114:0/2466695093 <== mon.0 v2:172.21.15.114:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fc19c01a5c0 con 0x7fc1b4108530 2024-04-23T10:17:22.763 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-04-23T10:17:21.954+0000 7fc1ba2ea700 1 -- 172.21.15.114:0/2466695093 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fc1b407b1a0 con 0x7fc1b4108530 2024-04-23T10:17:22.763 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-04-23T10:17:21.955+0000 7fc1ba2ea700 1 -- 172.21.15.114:0/2466695093 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fc1b407b6e0 con 0x7fc1b4108530 2024-04-23T10:17:22.763 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-04-23T10:17:21.955+0000 7fc1b17fa700 1 -- 172.21.15.114:0/2466695093 <== mon.0 v2:172.21.15.114:3300/0 4 ==== mgrmap(e 7) ==== 46452+0+0 (secure 0 0 0) 0x7fc19c01a870 con 0x7fc1b4108530 2024-04-23T10:17:22.764 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-04-23T10:17:21.955+0000 7fc1ba2ea700 1 -- 172.21.15.114:0/2466695093 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fc1b40640a0 con 0x7fc1b4108530 2024-04-23T10:17:22.764 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-04-23T10:17:21.956+0000 7fc1b17fa700 1 --2- 172.21.15.114:0/2466695093 >> [v2:172.21.15.114:6800/1604678557,v1:172.21.15.114:6801/1604678557] conn(0x7fc1a00394a0 0x7fc1a003b960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T10:17:22.764 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-04-23T10:17:21.956+0000 7fc1b17fa700 1 -- 172.21.15.114:0/2466695093 <== mon.0 v2:172.21.15.114:3300/0 5 ==== osd_map(2..2 src has 1..2) ==== 940+0+0 (secure 0 0 0) 0x7fc19c04da20 con 0x7fc1b4108530 2024-04-23T10:17:22.764 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-04-23T10:17:21.956+0000 7fc1b37fe700 1 --2- 172.21.15.114:0/2466695093 >> [v2:172.21.15.114:6800/1604678557,v1:172.21.15.114:6801/1604678557] conn(0x7fc1a00394a0 0x7fc1a003b960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T10:17:22.764 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-04-23T10:17:21.956+0000 7fc1b37fe700 1 --2- 172.21.15.114:0/2466695093 >> [v2:172.21.15.114:6800/1604678557,v1:172.21.15.114:6801/1604678557] conn(0x7fc1a00394a0 0x7fc1a003b960 secure :-1 s=READY pgs=7 cs=0 l=1 rev1=1 crypto rx=0x7fc1a4006fd0 tx=0x7fc1a4006e40 comp rx=0 tx=0).ready entity=mgr.14120 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T10:17:22.764 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-04-23T10:17:21.961+0000 7fc1b17fa700 1 -- 172.21.15.114:0/2466695093 <== mon.0 v2:172.21.15.114:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7fc19c019c70 con 0x7fc1b4108530 2024-04-23T10:17:22.764 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-04-23T10:17:22.132+0000 7fc1b17fa700 1 -- 172.21.15.114:0/2466695093 <== mon.0 v2:172.21.15.114:3300/0 7 ==== mgrmap(e 8) ==== 46558+0+0 (secure 0 0 0) 0x7fc19c02d910 con 0x7fc1b4108530 2024-04-23T10:17:22.764 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-04-23T10:17:22.179+0000 7fc1ba2ea700 1 -- 172.21.15.114:0/2466695093 --> [v2:172.21.15.114:6800/1604678557,v1:172.21.15.114:6801/1604678557] -- mgr_command(tid 0: {"prefix": "orch set backend", "module_name": "cephadm", "target": ["mon-mgr", ""]}) -- 0x7fc1b407bf40 con 0x7fc1a00394a0 2024-04-23T10:17:22.764 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-04-23T10:17:22.184+0000 7fc1b17fa700 1 -- 172.21.15.114:0/2466695093 <== mgr.14120 v2:172.21.15.114:6800/1604678557 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+0 (secure 0 0 0) 0x7fc1b407bf40 con 0x7fc1a00394a0 2024-04-23T10:17:22.764 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-04-23T10:17:22.186+0000 7fc1ba2ea700 1 -- 172.21.15.114:0/2466695093 >> [v2:172.21.15.114:6800/1604678557,v1:172.21.15.114:6801/1604678557] conn(0x7fc1a00394a0 msgr2=0x7fc1a003b960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T10:17:22.764 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-04-23T10:17:22.186+0000 7fc1ba2ea700 1 --2- 172.21.15.114:0/2466695093 >> [v2:172.21.15.114:6800/1604678557,v1:172.21.15.114:6801/1604678557] conn(0x7fc1a00394a0 0x7fc1a003b960 secure :-1 s=READY pgs=7 cs=0 l=1 rev1=1 crypto rx=0x7fc1a4006fd0 tx=0x7fc1a4006e40 comp rx=0 tx=0).stop 2024-04-23T10:17:22.764 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-04-23T10:17:22.186+0000 7fc1ba2ea700 1 -- 172.21.15.114:0/2466695093 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fc1b4108530 msgr2=0x7fc1b407ca70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T10:17:22.764 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-04-23T10:17:22.186+0000 7fc1ba2ea700 1 --2- 172.21.15.114:0/2466695093 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fc1b4108530 0x7fc1b407ca70 secure :-1 s=READY pgs=38 cs=0 l=1 rev1=1 crypto rx=0x7fc19c003f40 tx=0x7fc19c011890 comp rx=0 tx=0).stop 2024-04-23T10:17:22.764 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-04-23T10:17:22.186+0000 7fc1ba2ea700 1 -- 172.21.15.114:0/2466695093 shutdown_connections 2024-04-23T10:17:22.765 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-04-23T10:17:22.186+0000 7fc1ba2ea700 1 --2- 172.21.15.114:0/2466695093 >> [v2:172.21.15.114:6800/1604678557,v1:172.21.15.114:6801/1604678557] conn(0x7fc1a00394a0 0x7fc1a003b960 unknown :-1 s=CLOSED pgs=7 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:17:22.765 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-04-23T10:17:22.186+0000 7fc1ba2ea700 1 --2- 172.21.15.114:0/2466695093 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fc1b4108530 0x7fc1b407ca70 unknown :-1 s=CLOSED pgs=38 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:17:22.765 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-04-23T10:17:22.186+0000 7fc1ba2ea700 1 -- 172.21.15.114:0/2466695093 >> 172.21.15.114:0/2466695093 conn(0x7fc1b4103b50 msgr2=0x7fc1b41045f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T10:17:22.765 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-04-23T10:17:22.186+0000 7fc1ba2ea700 1 -- 172.21.15.114:0/2466695093 shutdown_connections 2024-04-23T10:17:22.765 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-04-23T10:17:22.186+0000 7fc1ba2ea700 1 -- 172.21.15.114:0/2466695093 wait complete. 2024-04-23T10:17:23.347 INFO:journalctl@ceph.mon.a.smithi114.stdout:Apr 23 10:17:23 smithi114 ceph-mon[29255]: [23/Apr/2024:10:17:22] ENGINE Bus STARTING 2024-04-23T10:17:23.347 INFO:journalctl@ceph.mon.a.smithi114.stdout:Apr 23 10:17:23 smithi114 ceph-mon[29255]: mgrmap e8: x(active, since 2s) 2024-04-23T10:17:23.347 INFO:journalctl@ceph.mon.a.smithi114.stdout:Apr 23 10:17:23 smithi114 ceph-mon[29255]: from='mgr.14120 172.21.15.114:0/1448995927' entity='mgr.x' 2024-04-23T10:17:23.347 INFO:journalctl@ceph.mon.a.smithi114.stdout:Apr 23 10:17:23 smithi114 ceph-mon[29255]: from='mgr.14120 172.21.15.114:0/1448995927' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-23T10:17:23.347 INFO:journalctl@ceph.mon.a.smithi114.stdout:Apr 23 10:17:23 smithi114 ceph-mon[29255]: from='mgr.14120 172.21.15.114:0/1448995927' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-23T10:17:23.909 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stdout value unchanged 2024-04-23T10:17:23.910 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-04-23T10:17:23.140+0000 7f137884c700 1 Processor -- start 2024-04-23T10:17:23.910 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-04-23T10:17:23.140+0000 7f137884c700 1 -- start start 2024-04-23T10:17:23.910 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-04-23T10:17:23.140+0000 7f137884c700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f1374107950 0x7f1374107d70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T10:17:23.910 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-04-23T10:17:23.140+0000 7f137884c700 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x7f1374108340 con 0x7f1374107950 2024-04-23T10:17:23.910 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-04-23T10:17:23.140+0000 7f137259c700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f1374107950 0x7f1374107d70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T10:17:23.910 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-04-23T10:17:23.140+0000 7f137259c700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f1374107950 0x7f1374107d70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.114:47292/0 (socket says 172.21.15.114:47292) 2024-04-23T10:17:23.910 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-04-23T10:17:23.140+0000 7f137259c700 1 -- 172.21.15.114:0/1868945238 learned_addr learned my addr 172.21.15.114:0/1868945238 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-04-23T10:17:23.910 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-04-23T10:17:23.141+0000 7f137259c700 1 -- 172.21.15.114:0/1868945238 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f1374108670 con 0x7f1374107950 2024-04-23T10:17:23.910 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-04-23T10:17:23.141+0000 7f137259c700 1 --2- 172.21.15.114:0/1868945238 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f1374107950 0x7f1374107d70 secure :-1 s=READY pgs=39 cs=0 l=1 rev1=1 crypto rx=0x7f13640098a0 tx=0x7f1364009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d759d4cbb7df545b server_cookie=0 in_seq=0 out_seq=0 2024-04-23T10:17:23.910 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-04-23T10:17:23.141+0000 7f137159a700 1 -- 172.21.15.114:0/1868945238 <== mon.0 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f1364004160 con 0x7f1374107950 2024-04-23T10:17:23.910 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-04-23T10:17:23.141+0000 7f137159a700 1 -- 172.21.15.114:0/1868945238 <== mon.0 v2:172.21.15.114:3300/0 2 ==== config(26 keys) ==== 1070+0+0 (secure 0 0 0) 0x7f1364004300 con 0x7f1374107950 2024-04-23T10:17:23.910 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-04-23T10:17:23.141+0000 7f137159a700 1 -- 172.21.15.114:0/1868945238 <== mon.0 v2:172.21.15.114:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f1364011af0 con 0x7f1374107950 2024-04-23T10:17:23.911 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-04-23T10:17:23.141+0000 7f137884c700 1 -- 172.21.15.114:0/1868945238 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f1374107950 msgr2=0x7f1374107d70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T10:17:23.911 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-04-23T10:17:23.141+0000 7f137884c700 1 --2- 172.21.15.114:0/1868945238 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f1374107950 0x7f1374107d70 secure :-1 s=READY pgs=39 cs=0 l=1 rev1=1 crypto rx=0x7f13640098a0 tx=0x7f1364009bb0 comp rx=0 tx=0).stop 2024-04-23T10:17:23.911 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-04-23T10:17:23.142+0000 7f137884c700 1 -- 172.21.15.114:0/1868945238 shutdown_connections 2024-04-23T10:17:23.911 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-04-23T10:17:23.142+0000 7f137884c700 1 --2- 172.21.15.114:0/1868945238 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f1374107950 0x7f1374107d70 unknown :-1 s=CLOSED pgs=39 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:17:23.911 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-04-23T10:17:23.142+0000 7f137884c700 1 -- 172.21.15.114:0/1868945238 >> 172.21.15.114:0/1868945238 conn(0x7f1374102150 msgr2=0x7f1374104590 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T10:17:23.911 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-04-23T10:17:23.142+0000 7f137884c700 1 -- 172.21.15.114:0/1868945238 shutdown_connections 2024-04-23T10:17:23.911 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-04-23T10:17:23.142+0000 7f137884c700 1 -- 172.21.15.114:0/1868945238 wait complete. 2024-04-23T10:17:23.911 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-04-23T10:17:23.142+0000 7f137884c700 1 Processor -- start 2024-04-23T10:17:23.911 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-04-23T10:17:23.142+0000 7f137884c700 1 -- start start 2024-04-23T10:17:23.911 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-04-23T10:17:23.142+0000 7f137884c700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f1374107950 0x7f137419c4f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T10:17:23.911 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-04-23T10:17:23.142+0000 7f137884c700 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x7f137410ba20 con 0x7f1374107950 2024-04-23T10:17:23.911 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-04-23T10:17:23.143+0000 7f137259c700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f1374107950 0x7f137419c4f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T10:17:23.911 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-04-23T10:17:23.143+0000 7f137259c700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f1374107950 0x7f137419c4f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.114:47298/0 (socket says 172.21.15.114:47298) 2024-04-23T10:17:23.911 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-04-23T10:17:23.143+0000 7f137259c700 1 -- 172.21.15.114:0/2984536749 learned_addr learned my addr 172.21.15.114:0/2984536749 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-04-23T10:17:23.911 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-04-23T10:17:23.143+0000 7f137259c700 1 -- 172.21.15.114:0/2984536749 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f137419ca30 con 0x7f1374107950 2024-04-23T10:17:23.912 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-04-23T10:17:23.143+0000 7f137259c700 1 --2- 172.21.15.114:0/2984536749 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f1374107950 0x7f137419c4f0 secure :-1 s=READY pgs=40 cs=0 l=1 rev1=1 crypto rx=0x7f1364003f40 tx=0x7f1364011890 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T10:17:23.912 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-04-23T10:17:23.143+0000 7f13637fe700 1 -- 172.21.15.114:0/2984536749 <== mon.0 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f13640038b0 con 0x7f1374107950 2024-04-23T10:17:23.912 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-04-23T10:17:23.143+0000 7f13637fe700 1 -- 172.21.15.114:0/2984536749 <== mon.0 v2:172.21.15.114:3300/0 2 ==== config(26 keys) ==== 1070+0+0 (secure 0 0 0) 0x7f1364003a50 con 0x7f1374107950 2024-04-23T10:17:23.912 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-04-23T10:17:23.143+0000 7f137884c700 1 -- 172.21.15.114:0/2984536749 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f137419ccc0 con 0x7f1374107950 2024-04-23T10:17:23.912 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-04-23T10:17:23.143+0000 7f13637fe700 1 -- 172.21.15.114:0/2984536749 <== mon.0 v2:172.21.15.114:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f136401a5c0 con 0x7f1374107950 2024-04-23T10:17:23.912 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-04-23T10:17:23.143+0000 7f137884c700 1 -- 172.21.15.114:0/2984536749 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f137419f9b0 con 0x7f1374107950 2024-04-23T10:17:23.912 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-04-23T10:17:23.144+0000 7f13637fe700 1 -- 172.21.15.114:0/2984536749 <== mon.0 v2:172.21.15.114:3300/0 4 ==== mgrmap(e 8) ==== 46558+0+0 (secure 0 0 0) 0x7f136401a870 con 0x7f1374107950 2024-04-23T10:17:23.912 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-04-23T10:17:23.144+0000 7f137884c700 1 -- 172.21.15.114:0/2984536749 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f1374069ba0 con 0x7f1374107950 2024-04-23T10:17:23.912 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-04-23T10:17:23.144+0000 7f13637fe700 1 --2- 172.21.15.114:0/2984536749 >> [v2:172.21.15.114:6800/1604678557,v1:172.21.15.114:6801/1604678557] conn(0x7f135c0395b0 0x7f135c03ba70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T10:17:23.912 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-04-23T10:17:23.144+0000 7f13637fe700 1 -- 172.21.15.114:0/2984536749 <== mon.0 v2:172.21.15.114:3300/0 5 ==== osd_map(2..2 src has 1..2) ==== 940+0+0 (secure 0 0 0) 0x7f136404da80 con 0x7f1374107950 2024-04-23T10:17:23.912 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-04-23T10:17:23.144+0000 7f1371d9b700 1 --2- 172.21.15.114:0/2984536749 >> [v2:172.21.15.114:6800/1604678557,v1:172.21.15.114:6801/1604678557] conn(0x7f135c0395b0 0x7f135c03ba70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T10:17:23.912 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-04-23T10:17:23.145+0000 7f1371d9b700 1 --2- 172.21.15.114:0/2984536749 >> [v2:172.21.15.114:6800/1604678557,v1:172.21.15.114:6801/1604678557] conn(0x7f135c0395b0 0x7f135c03ba70 secure :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0x7f1368006fd0 tx=0x7f1368006e40 comp rx=0 tx=0).ready entity=mgr.14120 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T10:17:23.912 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-04-23T10:17:23.149+0000 7f13637fe700 1 -- 172.21.15.114:0/2984536749 <== mon.0 v2:172.21.15.114:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f1364019c70 con 0x7f1374107950 2024-04-23T10:17:23.912 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-04-23T10:17:23.348+0000 7f137884c700 1 -- 172.21.15.114:0/2984536749 --> [v2:172.21.15.114:6800/1604678557,v1:172.21.15.114:6801/1604678557] -- mgr_command(tid 0: {"prefix": "cephadm set-user", "user": "root", "target": ["mon-mgr", ""]}) -- 0x7f13740008d0 con 0x7f135c0395b0 2024-04-23T10:17:23.913 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-04-23T10:17:23.348+0000 7f13637fe700 1 -- 172.21.15.114:0/2984536749 <== mgr.14120 v2:172.21.15.114:6800/1604678557 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+16 (secure 0 0 0) 0x7f13740008d0 con 0x7f135c0395b0 2024-04-23T10:17:23.913 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-04-23T10:17:23.350+0000 7f137884c700 1 -- 172.21.15.114:0/2984536749 >> [v2:172.21.15.114:6800/1604678557,v1:172.21.15.114:6801/1604678557] conn(0x7f135c0395b0 msgr2=0x7f135c03ba70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T10:17:23.913 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-04-23T10:17:23.350+0000 7f137884c700 1 --2- 172.21.15.114:0/2984536749 >> [v2:172.21.15.114:6800/1604678557,v1:172.21.15.114:6801/1604678557] conn(0x7f135c0395b0 0x7f135c03ba70 secure :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0x7f1368006fd0 tx=0x7f1368006e40 comp rx=0 tx=0).stop 2024-04-23T10:17:23.913 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-04-23T10:17:23.350+0000 7f137884c700 1 -- 172.21.15.114:0/2984536749 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f1374107950 msgr2=0x7f137419c4f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T10:17:23.913 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-04-23T10:17:23.350+0000 7f137884c700 1 --2- 172.21.15.114:0/2984536749 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f1374107950 0x7f137419c4f0 secure :-1 s=READY pgs=40 cs=0 l=1 rev1=1 crypto rx=0x7f1364003f40 tx=0x7f1364011890 comp rx=0 tx=0).stop 2024-04-23T10:17:23.913 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-04-23T10:17:23.350+0000 7f137884c700 1 -- 172.21.15.114:0/2984536749 shutdown_connections 2024-04-23T10:17:23.913 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-04-23T10:17:23.350+0000 7f137884c700 1 --2- 172.21.15.114:0/2984536749 >> [v2:172.21.15.114:6800/1604678557,v1:172.21.15.114:6801/1604678557] conn(0x7f135c0395b0 0x7f135c03ba70 unknown :-1 s=CLOSED pgs=8 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:17:23.913 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-04-23T10:17:23.350+0000 7f137884c700 1 --2- 172.21.15.114:0/2984536749 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f1374107950 0x7f137419c4f0 unknown :-1 s=CLOSED pgs=40 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:17:23.913 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-04-23T10:17:23.350+0000 7f137884c700 1 -- 172.21.15.114:0/2984536749 >> 172.21.15.114:0/2984536749 conn(0x7f1374102150 msgr2=0x7f1374103030 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T10:17:23.913 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-04-23T10:17:23.351+0000 7f137884c700 1 -- 172.21.15.114:0/2984536749 shutdown_connections 2024-04-23T10:17:23.913 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-04-23T10:17:23.351+0000 7f137884c700 1 -- 172.21.15.114:0/2984536749 wait complete. 2024-04-23T10:17:23.913 INFO:teuthology.orchestra.run.smithi114.stdout:Generating ssh key... 2024-04-23T10:17:24.154 INFO:journalctl@ceph.mon.a.smithi114.stdout:Apr 23 10:17:24 smithi114 ceph-mon[29255]: [23/Apr/2024:10:17:22] ENGINE Serving on https://172.21.15.114:7150 2024-04-23T10:17:24.154 INFO:journalctl@ceph.mon.a.smithi114.stdout:Apr 23 10:17:24 smithi114 ceph-mon[29255]: from='client.14132 -' entity='client.admin' cmd=[{"prefix": "orch set backend", "module_name": "cephadm", "target": ["mon-mgr", ""]}]: dispatch 2024-04-23T10:17:24.154 INFO:journalctl@ceph.mon.a.smithi114.stdout:Apr 23 10:17:24 smithi114 ceph-mon[29255]: [23/Apr/2024:10:17:22] ENGINE Serving on http://172.21.15.114:8765 2024-04-23T10:17:24.154 INFO:journalctl@ceph.mon.a.smithi114.stdout:Apr 23 10:17:24 smithi114 ceph-mon[29255]: [23/Apr/2024:10:17:22] ENGINE Bus STARTED 2024-04-23T10:17:25.004 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Apr 23 10:17:24 smithi114 ceph-aa240572-0159-11ef-bc93-c7b262605968-mgr-x[29490]: Generating public/private rsa key pair. 2024-04-23T10:17:25.005 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Apr 23 10:17:24 smithi114 ceph-aa240572-0159-11ef-bc93-c7b262605968-mgr-x[29490]: Your identification has been saved in /tmp/tmpa4w25ntc/key. 2024-04-23T10:17:25.005 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Apr 23 10:17:24 smithi114 ceph-aa240572-0159-11ef-bc93-c7b262605968-mgr-x[29490]: Your public key has been saved in /tmp/tmpa4w25ntc/key.pub. 2024-04-23T10:17:25.005 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Apr 23 10:17:24 smithi114 ceph-aa240572-0159-11ef-bc93-c7b262605968-mgr-x[29490]: The key fingerprint is: 2024-04-23T10:17:25.005 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Apr 23 10:17:24 smithi114 ceph-aa240572-0159-11ef-bc93-c7b262605968-mgr-x[29490]: SHA256:xx3CHP3F+zaxGBWC90EZhpv5iAXOnXxQiek1VWTmxBw ceph-aa240572-0159-11ef-bc93-c7b262605968 2024-04-23T10:17:25.005 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Apr 23 10:17:24 smithi114 ceph-aa240572-0159-11ef-bc93-c7b262605968-mgr-x[29490]: The key's randomart image is: 2024-04-23T10:17:25.005 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Apr 23 10:17:24 smithi114 ceph-aa240572-0159-11ef-bc93-c7b262605968-mgr-x[29490]: +---[RSA 3072]----+ 2024-04-23T10:17:25.005 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Apr 23 10:17:24 smithi114 ceph-aa240572-0159-11ef-bc93-c7b262605968-mgr-x[29490]: | ....*E/| 2024-04-23T10:17:25.005 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Apr 23 10:17:24 smithi114 ceph-aa240572-0159-11ef-bc93-c7b262605968-mgr-x[29490]: | o +.*o%=| 2024-04-23T10:17:25.005 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Apr 23 10:17:24 smithi114 ceph-aa240572-0159-11ef-bc93-c7b262605968-mgr-x[29490]: | * B.X.=| 2024-04-23T10:17:25.005 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Apr 23 10:17:24 smithi114 ceph-aa240572-0159-11ef-bc93-c7b262605968-mgr-x[29490]: | . = %.= | 2024-04-23T10:17:25.005 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Apr 23 10:17:24 smithi114 ceph-aa240572-0159-11ef-bc93-c7b262605968-mgr-x[29490]: | S o + * +| 2024-04-23T10:17:25.005 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Apr 23 10:17:24 smithi114 ceph-aa240572-0159-11ef-bc93-c7b262605968-mgr-x[29490]: | . . o +o| 2024-04-23T10:17:25.005 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Apr 23 10:17:24 smithi114 ceph-aa240572-0159-11ef-bc93-c7b262605968-mgr-x[29490]: | ..| 2024-04-23T10:17:25.005 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Apr 23 10:17:24 smithi114 ceph-aa240572-0159-11ef-bc93-c7b262605968-mgr-x[29490]: | | 2024-04-23T10:17:25.006 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Apr 23 10:17:24 smithi114 ceph-aa240572-0159-11ef-bc93-c7b262605968-mgr-x[29490]: | | 2024-04-23T10:17:25.006 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Apr 23 10:17:24 smithi114 ceph-aa240572-0159-11ef-bc93-c7b262605968-mgr-x[29490]: +----[SHA256]-----+ 2024-04-23T10:17:25.383 INFO:journalctl@ceph.mon.a.smithi114.stdout:Apr 23 10:17:25 smithi114 ceph-mon[29255]: from='client.14134 -' entity='client.admin' cmd=[{"prefix": "cephadm set-user", "user": "root", "target": ["mon-mgr", ""]}]: dispatch 2024-04-23T10:17:25.383 INFO:journalctl@ceph.mon.a.smithi114.stdout:Apr 23 10:17:25 smithi114 ceph-mon[29255]: from='mgr.14120 172.21.15.114:0/1448995927' entity='mgr.x' 2024-04-23T10:17:25.383 INFO:journalctl@ceph.mon.a.smithi114.stdout:Apr 23 10:17:25 smithi114 ceph-mon[29255]: from='mgr.14120 172.21.15.114:0/1448995927' entity='mgr.x' 2024-04-23T10:17:25.391 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-04-23T10:17:24.336+0000 7f02a192d700 1 Processor -- start 2024-04-23T10:17:25.391 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-04-23T10:17:24.336+0000 7f02a192d700 1 -- start start 2024-04-23T10:17:25.391 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-04-23T10:17:24.336+0000 7f02a192d700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f029c108fd0 0x7f029c10b3c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T10:17:25.391 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-04-23T10:17:24.336+0000 7f02a192d700 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x7f029c07bea0 con 0x7f029c108fd0 2024-04-23T10:17:25.391 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-04-23T10:17:24.336+0000 7f029affd700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f029c108fd0 0x7f029c10b3c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T10:17:25.391 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-04-23T10:17:24.336+0000 7f029affd700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f029c108fd0 0x7f029c10b3c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.114:47312/0 (socket says 172.21.15.114:47312) 2024-04-23T10:17:25.391 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-04-23T10:17:24.336+0000 7f029affd700 1 -- 172.21.15.114:0/193630016 learned_addr learned my addr 172.21.15.114:0/193630016 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-04-23T10:17:25.391 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-04-23T10:17:24.336+0000 7f029affd700 1 -- 172.21.15.114:0/193630016 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f029c10b900 con 0x7f029c108fd0 2024-04-23T10:17:25.391 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-04-23T10:17:24.337+0000 7f029affd700 1 --2- 172.21.15.114:0/193630016 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f029c108fd0 0x7f029c10b3c0 secure :-1 s=READY pgs=41 cs=0 l=1 rev1=1 crypto rx=0x7f028c0098a0 tx=0x7f028c009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3fe822f943008d79 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T10:17:25.391 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-04-23T10:17:24.337+0000 7f0299ffb700 1 -- 172.21.15.114:0/193630016 <== mon.0 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f028c004160 con 0x7f029c108fd0 2024-04-23T10:17:25.391 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-04-23T10:17:24.337+0000 7f0299ffb700 1 -- 172.21.15.114:0/193630016 <== mon.0 v2:172.21.15.114:3300/0 2 ==== config(26 keys) ==== 1070+0+0 (secure 0 0 0) 0x7f028c004300 con 0x7f029c108fd0 2024-04-23T10:17:25.392 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-04-23T10:17:24.337+0000 7f0299ffb700 1 -- 172.21.15.114:0/193630016 <== mon.0 v2:172.21.15.114:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f028c011af0 con 0x7f029c108fd0 2024-04-23T10:17:25.392 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-04-23T10:17:24.337+0000 7f02a192d700 1 -- 172.21.15.114:0/193630016 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f029c108fd0 msgr2=0x7f029c10b3c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T10:17:25.392 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-04-23T10:17:24.337+0000 7f02a192d700 1 --2- 172.21.15.114:0/193630016 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f029c108fd0 0x7f029c10b3c0 secure :-1 s=READY pgs=41 cs=0 l=1 rev1=1 crypto rx=0x7f028c0098a0 tx=0x7f028c009bb0 comp rx=0 tx=0).stop 2024-04-23T10:17:25.392 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-04-23T10:17:24.337+0000 7f02a192d700 1 -- 172.21.15.114:0/193630016 shutdown_connections 2024-04-23T10:17:25.392 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-04-23T10:17:24.337+0000 7f02a192d700 1 --2- 172.21.15.114:0/193630016 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f029c108fd0 0x7f029c10b3c0 unknown :-1 s=CLOSED pgs=41 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:17:25.392 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-04-23T10:17:24.337+0000 7f02a192d700 1 -- 172.21.15.114:0/193630016 >> 172.21.15.114:0/193630016 conn(0x7f029c1029e0 msgr2=0x7f029c104e20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T10:17:25.392 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-04-23T10:17:24.337+0000 7f02a192d700 1 -- 172.21.15.114:0/193630016 shutdown_connections 2024-04-23T10:17:25.392 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-04-23T10:17:24.337+0000 7f02a192d700 1 -- 172.21.15.114:0/193630016 wait complete. 2024-04-23T10:17:25.392 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-04-23T10:17:24.338+0000 7f02a192d700 1 Processor -- start 2024-04-23T10:17:25.392 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-04-23T10:17:24.338+0000 7f02a192d700 1 -- start start 2024-04-23T10:17:25.392 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-04-23T10:17:24.338+0000 7f02a192d700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f029c108fd0 0x7f029c19c480 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T10:17:25.392 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-04-23T10:17:24.338+0000 7f02a192d700 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x7f029c10c850 con 0x7f029c108fd0 2024-04-23T10:17:25.392 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-04-23T10:17:24.338+0000 7f029affd700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f029c108fd0 0x7f029c19c480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T10:17:25.392 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-04-23T10:17:24.338+0000 7f029affd700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f029c108fd0 0x7f029c19c480 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.114:47320/0 (socket says 172.21.15.114:47320) 2024-04-23T10:17:25.392 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-04-23T10:17:24.338+0000 7f029affd700 1 -- 172.21.15.114:0/3921007812 learned_addr learned my addr 172.21.15.114:0/3921007812 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-04-23T10:17:25.393 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-04-23T10:17:24.339+0000 7f029affd700 1 -- 172.21.15.114:0/3921007812 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f029c19c9c0 con 0x7f029c108fd0 2024-04-23T10:17:25.393 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-04-23T10:17:24.339+0000 7f029affd700 1 --2- 172.21.15.114:0/3921007812 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f029c108fd0 0x7f029c19c480 secure :-1 s=READY pgs=42 cs=0 l=1 rev1=1 crypto rx=0x7f028c009fb0 tx=0x7f028c011940 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T10:17:25.393 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-04-23T10:17:24.339+0000 7f02a092b700 1 -- 172.21.15.114:0/3921007812 <== mon.0 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f028c0036a0 con 0x7f029c108fd0 2024-04-23T10:17:25.393 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-04-23T10:17:24.339+0000 7f02a092b700 1 -- 172.21.15.114:0/3921007812 <== mon.0 v2:172.21.15.114:3300/0 2 ==== config(26 keys) ==== 1070+0+0 (secure 0 0 0) 0x7f028c01a430 con 0x7f029c108fd0 2024-04-23T10:17:25.393 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-04-23T10:17:24.339+0000 7f02a192d700 1 -- 172.21.15.114:0/3921007812 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f029c19ccb0 con 0x7f029c108fd0 2024-04-23T10:17:25.393 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-04-23T10:17:24.339+0000 7f02a092b700 1 -- 172.21.15.114:0/3921007812 <== mon.0 v2:172.21.15.114:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f028c019550 con 0x7f029c108fd0 2024-04-23T10:17:25.393 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-04-23T10:17:24.339+0000 7f02a192d700 1 -- 172.21.15.114:0/3921007812 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f029c19f940 con 0x7f029c108fd0 2024-04-23T10:17:25.393 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-04-23T10:17:24.340+0000 7f02a092b700 1 -- 172.21.15.114:0/3921007812 <== mon.0 v2:172.21.15.114:3300/0 4 ==== mgrmap(e 8) ==== 46558+0+0 (secure 0 0 0) 0x7f028c019850 con 0x7f029c108fd0 2024-04-23T10:17:25.393 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-04-23T10:17:24.340+0000 7f02a192d700 1 -- 172.21.15.114:0/3921007812 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f029c0640a0 con 0x7f029c108fd0 2024-04-23T10:17:25.393 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-04-23T10:17:24.340+0000 7f02a092b700 1 --2- 172.21.15.114:0/3921007812 >> [v2:172.21.15.114:6800/1604678557,v1:172.21.15.114:6801/1604678557] conn(0x7f0284039560 0x7f028403ba20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T10:17:25.393 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-04-23T10:17:24.340+0000 7f02a092b700 1 -- 172.21.15.114:0/3921007812 <== mon.0 v2:172.21.15.114:3300/0 5 ==== osd_map(2..2 src has 1..2) ==== 940+0+0 (secure 0 0 0) 0x7f028c04d890 con 0x7f029c108fd0 2024-04-23T10:17:25.393 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-04-23T10:17:24.340+0000 7f029a7fc700 1 --2- 172.21.15.114:0/3921007812 >> [v2:172.21.15.114:6800/1604678557,v1:172.21.15.114:6801/1604678557] conn(0x7f0284039560 0x7f028403ba20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T10:17:25.393 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-04-23T10:17:24.340+0000 7f029a7fc700 1 --2- 172.21.15.114:0/3921007812 >> [v2:172.21.15.114:6800/1604678557,v1:172.21.15.114:6801/1604678557] conn(0x7f0284039560 0x7f028403ba20 secure :-1 s=READY pgs=9 cs=0 l=1 rev1=1 crypto rx=0x7f0290006fd0 tx=0x7f0290006e40 comp rx=0 tx=0).ready entity=mgr.14120 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T10:17:25.393 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-04-23T10:17:24.345+0000 7f02a092b700 1 -- 172.21.15.114:0/3921007812 <== mon.0 v2:172.21.15.114:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f028c018e10 con 0x7f029c108fd0 2024-04-23T10:17:25.393 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-04-23T10:17:24.544+0000 7f02a192d700 1 -- 172.21.15.114:0/3921007812 --> [v2:172.21.15.114:6800/1604678557,v1:172.21.15.114:6801/1604678557] -- mgr_command(tid 0: {"prefix": "cephadm generate-key", "target": ["mon-mgr", ""]}) -- 0x7f029c103cf0 con 0x7f0284039560 2024-04-23T10:17:25.394 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-04-23T10:17:24.756+0000 7f02a092b700 1 -- 172.21.15.114:0/3921007812 <== mgr.14120 v2:172.21.15.114:6800/1604678557 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+0 (secure 0 0 0) 0x7f029c103cf0 con 0x7f0284039560 2024-04-23T10:17:25.394 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-04-23T10:17:24.758+0000 7f02a192d700 1 -- 172.21.15.114:0/3921007812 >> [v2:172.21.15.114:6800/1604678557,v1:172.21.15.114:6801/1604678557] conn(0x7f0284039560 msgr2=0x7f028403ba20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T10:17:25.394 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-04-23T10:17:24.758+0000 7f02a192d700 1 --2- 172.21.15.114:0/3921007812 >> [v2:172.21.15.114:6800/1604678557,v1:172.21.15.114:6801/1604678557] conn(0x7f0284039560 0x7f028403ba20 secure :-1 s=READY pgs=9 cs=0 l=1 rev1=1 crypto rx=0x7f0290006fd0 tx=0x7f0290006e40 comp rx=0 tx=0).stop 2024-04-23T10:17:25.394 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-04-23T10:17:24.758+0000 7f02a192d700 1 -- 172.21.15.114:0/3921007812 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f029c108fd0 msgr2=0x7f029c19c480 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T10:17:25.394 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-04-23T10:17:24.758+0000 7f02a192d700 1 --2- 172.21.15.114:0/3921007812 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f029c108fd0 0x7f029c19c480 secure :-1 s=READY pgs=42 cs=0 l=1 rev1=1 crypto rx=0x7f028c009fb0 tx=0x7f028c011940 comp rx=0 tx=0).stop 2024-04-23T10:17:25.394 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-04-23T10:17:24.758+0000 7f02a192d700 1 -- 172.21.15.114:0/3921007812 shutdown_connections 2024-04-23T10:17:25.394 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-04-23T10:17:24.758+0000 7f02a192d700 1 --2- 172.21.15.114:0/3921007812 >> [v2:172.21.15.114:6800/1604678557,v1:172.21.15.114:6801/1604678557] conn(0x7f0284039560 0x7f028403ba20 unknown :-1 s=CLOSED pgs=9 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:17:25.394 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-04-23T10:17:24.758+0000 7f02a192d700 1 --2- 172.21.15.114:0/3921007812 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f029c108fd0 0x7f029c19c480 unknown :-1 s=CLOSED pgs=42 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:17:25.394 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-04-23T10:17:24.758+0000 7f02a192d700 1 -- 172.21.15.114:0/3921007812 >> 172.21.15.114:0/3921007812 conn(0x7f029c1029e0 msgr2=0x7f029c1035e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T10:17:25.394 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-04-23T10:17:24.759+0000 7f02a192d700 1 -- 172.21.15.114:0/3921007812 shutdown_connections 2024-04-23T10:17:25.394 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-04-23T10:17:24.759+0000 7f02a192d700 1 -- 172.21.15.114:0/3921007812 wait complete. 2024-04-23T10:17:26.394 INFO:journalctl@ceph.mon.a.smithi114.stdout:Apr 23 10:17:26 smithi114 ceph-mon[29255]: from='client.14136 -' entity='client.admin' cmd=[{"prefix": "cephadm generate-key", "target": ["mon-mgr", ""]}]: dispatch 2024-04-23T10:17:26.394 INFO:journalctl@ceph.mon.a.smithi114.stdout:Apr 23 10:17:26 smithi114 ceph-mon[29255]: Generating ssh key... 2024-04-23T10:17:26.597 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stdout ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQC96O2tfq4F3ru+DL6BARr7cIMU2/U8e/M6SPi7vN3KakfqSi61xF0qopAsZD4nJpuRU4W+DbVrMHS+bHJ1s8ghaepC+fY+ibcriLsFcTooLk93RHYvxqSsqqJlqmoy6n9LaMx5RQrZkocXjNwj2FKHzV+xgfLZ/hEjBJPTBAl/Vg/JnfK/CA67iOeSKYWuxkARJd8zE2ePtyD9WsNmPkwccG3Q06xHl5ZzTlLNJNXDW3wgaLenxwliglHqYOmLC5jeUcjdYCJ2Xic/vnu7U8vjT0kKYrcD+BR1+ZPrem5sVV/9tVt7VS2ZnBBCo2NAWYblduMh6+M8dGxeSUxNOZ+RDjSVzsX4qHEjOBOEmaaRZOhqDge5Yqs3/wxJbalG7yvukS8uz9c2LH60vOT+2H4lBL0D/p49mvCn1OV4iEVo4EV/AgYaYE7slWP8vGhnZiX5LrzvThwslpM33hDyDZevowryFh981TVDsdIovZY4+NomFjIAzdl85EWWoWqqxcE= ceph-aa240572-0159-11ef-bc93-c7b262605968 2024-04-23T10:17:26.597 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-04-23T10:17:25.855+0000 7fe0c360d700 1 Processor -- start 2024-04-23T10:17:26.597 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-04-23T10:17:25.855+0000 7fe0c360d700 1 -- start start 2024-04-23T10:17:26.597 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-04-23T10:17:25.856+0000 7fe0c360d700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fe0bc108fd0 0x7fe0bc10b3c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T10:17:26.597 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-04-23T10:17:25.856+0000 7fe0c360d700 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x7fe0bc063d40 con 0x7fe0bc108fd0 2024-04-23T10:17:26.597 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-04-23T10:17:25.856+0000 7fe0c13a9700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fe0bc108fd0 0x7fe0bc10b3c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T10:17:26.597 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-04-23T10:17:25.856+0000 7fe0c13a9700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fe0bc108fd0 0x7fe0bc10b3c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.114:40338/0 (socket says 172.21.15.114:40338) 2024-04-23T10:17:26.597 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-04-23T10:17:25.856+0000 7fe0c13a9700 1 -- 172.21.15.114:0/177506582 learned_addr learned my addr 172.21.15.114:0/177506582 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-04-23T10:17:26.597 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-04-23T10:17:25.856+0000 7fe0c13a9700 1 -- 172.21.15.114:0/177506582 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe0bc10b900 con 0x7fe0bc108fd0 2024-04-23T10:17:26.597 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-04-23T10:17:25.856+0000 7fe0c13a9700 1 --2- 172.21.15.114:0/177506582 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fe0bc108fd0 0x7fe0bc10b3c0 secure :-1 s=READY pgs=43 cs=0 l=1 rev1=1 crypto rx=0x7fe0ac0098a0 tx=0x7fe0ac009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=aaa77d000963de3d server_cookie=0 in_seq=0 out_seq=0 2024-04-23T10:17:26.597 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-04-23T10:17:25.856+0000 7fe0b3fff700 1 -- 172.21.15.114:0/177506582 <== mon.0 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fe0ac004160 con 0x7fe0bc108fd0 2024-04-23T10:17:26.597 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-04-23T10:17:25.856+0000 7fe0b3fff700 1 -- 172.21.15.114:0/177506582 <== mon.0 v2:172.21.15.114:3300/0 2 ==== config(26 keys) ==== 1070+0+0 (secure 0 0 0) 0x7fe0ac004300 con 0x7fe0bc108fd0 2024-04-23T10:17:26.597 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-04-23T10:17:25.856+0000 7fe0b3fff700 1 -- 172.21.15.114:0/177506582 <== mon.0 v2:172.21.15.114:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fe0ac011af0 con 0x7fe0bc108fd0 2024-04-23T10:17:26.598 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-04-23T10:17:25.857+0000 7fe0c360d700 1 -- 172.21.15.114:0/177506582 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fe0bc108fd0 msgr2=0x7fe0bc10b3c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T10:17:26.598 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-04-23T10:17:25.857+0000 7fe0c360d700 1 --2- 172.21.15.114:0/177506582 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fe0bc108fd0 0x7fe0bc10b3c0 secure :-1 s=READY pgs=43 cs=0 l=1 rev1=1 crypto rx=0x7fe0ac0098a0 tx=0x7fe0ac009bb0 comp rx=0 tx=0).stop 2024-04-23T10:17:26.598 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-04-23T10:17:25.857+0000 7fe0c360d700 1 -- 172.21.15.114:0/177506582 shutdown_connections 2024-04-23T10:17:26.598 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-04-23T10:17:25.857+0000 7fe0c360d700 1 --2- 172.21.15.114:0/177506582 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fe0bc108fd0 0x7fe0bc10b3c0 unknown :-1 s=CLOSED pgs=43 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:17:26.598 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-04-23T10:17:25.857+0000 7fe0c360d700 1 -- 172.21.15.114:0/177506582 >> 172.21.15.114:0/177506582 conn(0x7fe0bc102a40 msgr2=0x7fe0bc104e60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T10:17:26.598 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-04-23T10:17:25.857+0000 7fe0c360d700 1 -- 172.21.15.114:0/177506582 shutdown_connections 2024-04-23T10:17:26.598 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-04-23T10:17:25.857+0000 7fe0c360d700 1 -- 172.21.15.114:0/177506582 wait complete. 2024-04-23T10:17:26.598 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-04-23T10:17:25.858+0000 7fe0c360d700 1 Processor -- start 2024-04-23T10:17:26.598 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-04-23T10:17:25.858+0000 7fe0c360d700 1 -- start start 2024-04-23T10:17:26.598 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-04-23T10:17:25.858+0000 7fe0c360d700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fe0bc108fd0 0x7fe0bc19c3f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T10:17:26.598 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-04-23T10:17:25.858+0000 7fe0c360d700 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x7fe0bc10c850 con 0x7fe0bc108fd0 2024-04-23T10:17:26.598 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-04-23T10:17:25.858+0000 7fe0c13a9700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fe0bc108fd0 0x7fe0bc19c3f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T10:17:26.598 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-04-23T10:17:25.858+0000 7fe0c13a9700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fe0bc108fd0 0x7fe0bc19c3f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.114:40344/0 (socket says 172.21.15.114:40344) 2024-04-23T10:17:26.598 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-04-23T10:17:25.858+0000 7fe0c13a9700 1 -- 172.21.15.114:0/2756083070 learned_addr learned my addr 172.21.15.114:0/2756083070 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-04-23T10:17:26.598 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-04-23T10:17:25.858+0000 7fe0c13a9700 1 -- 172.21.15.114:0/2756083070 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe0bc19c930 con 0x7fe0bc108fd0 2024-04-23T10:17:26.598 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-04-23T10:17:25.858+0000 7fe0c13a9700 1 --2- 172.21.15.114:0/2756083070 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fe0bc108fd0 0x7fe0bc19c3f0 secure :-1 s=READY pgs=44 cs=0 l=1 rev1=1 crypto rx=0x7fe0ac011ac0 tx=0x7fe0ac011ba0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T10:17:26.599 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-04-23T10:17:25.859+0000 7fe0b27fc700 1 -- 172.21.15.114:0/2756083070 <== mon.0 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fe0ac003b20 con 0x7fe0bc108fd0 2024-04-23T10:17:26.599 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-04-23T10:17:25.859+0000 7fe0b27fc700 1 -- 172.21.15.114:0/2756083070 <== mon.0 v2:172.21.15.114:3300/0 2 ==== config(26 keys) ==== 1070+0+0 (secure 0 0 0) 0x7fe0ac01a8c0 con 0x7fe0bc108fd0 2024-04-23T10:17:26.599 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-04-23T10:17:25.859+0000 7fe0b27fc700 1 -- 172.21.15.114:0/2756083070 <== mon.0 v2:172.21.15.114:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fe0ac0248d0 con 0x7fe0bc108fd0 2024-04-23T10:17:26.599 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-04-23T10:17:25.859+0000 7fe0c360d700 1 -- 172.21.15.114:0/2756083070 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fe0bc19cc20 con 0x7fe0bc108fd0 2024-04-23T10:17:26.599 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-04-23T10:17:25.859+0000 7fe0c360d700 1 -- 172.21.15.114:0/2756083070 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fe0bc19f8b0 con 0x7fe0bc108fd0 2024-04-23T10:17:26.599 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-04-23T10:17:25.859+0000 7fe0b27fc700 1 -- 172.21.15.114:0/2756083070 <== mon.0 v2:172.21.15.114:3300/0 4 ==== mgrmap(e 8) ==== 46558+0+0 (secure 0 0 0) 0x7fe0ac022020 con 0x7fe0bc108fd0 2024-04-23T10:17:26.599 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-04-23T10:17:25.860+0000 7fe0b27fc700 1 --2- 172.21.15.114:0/2756083070 >> [v2:172.21.15.114:6800/1604678557,v1:172.21.15.114:6801/1604678557] conn(0x7fe0a80395b0 0x7fe0a803ba70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T10:17:26.599 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-04-23T10:17:25.860+0000 7fe0c360d700 1 -- 172.21.15.114:0/2756083070 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fe0bc0640a0 con 0x7fe0bc108fd0 2024-04-23T10:17:26.599 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-04-23T10:17:25.860+0000 7fe0b27fc700 1 -- 172.21.15.114:0/2756083070 <== mon.0 v2:172.21.15.114:3300/0 5 ==== osd_map(2..2 src has 1..2) ==== 940+0+0 (secure 0 0 0) 0x7fe0ac04d870 con 0x7fe0bc108fd0 2024-04-23T10:17:26.599 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-04-23T10:17:25.860+0000 7fe0c0ba8700 1 --2- 172.21.15.114:0/2756083070 >> [v2:172.21.15.114:6800/1604678557,v1:172.21.15.114:6801/1604678557] conn(0x7fe0a80395b0 0x7fe0a803ba70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T10:17:26.599 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-04-23T10:17:25.860+0000 7fe0c0ba8700 1 --2- 172.21.15.114:0/2756083070 >> [v2:172.21.15.114:6800/1604678557,v1:172.21.15.114:6801/1604678557] conn(0x7fe0a80395b0 0x7fe0a803ba70 secure :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0x7fe0b8006fd0 tx=0x7fe0b8006e40 comp rx=0 tx=0).ready entity=mgr.14120 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T10:17:26.599 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-04-23T10:17:25.865+0000 7fe0b27fc700 1 -- 172.21.15.114:0/2756083070 <== mon.0 v2:172.21.15.114:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7fe0ac01aa70 con 0x7fe0bc108fd0 2024-04-23T10:17:26.599 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-04-23T10:17:26.069+0000 7fe0c360d700 1 -- 172.21.15.114:0/2756083070 --> [v2:172.21.15.114:6800/1604678557,v1:172.21.15.114:6801/1604678557] -- mgr_command(tid 0: {"prefix": "cephadm get-pub-key", "target": ["mon-mgr", ""]}) -- 0x7fe0bc19fa80 con 0x7fe0a80395b0 2024-04-23T10:17:26.599 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-04-23T10:17:26.070+0000 7fe0b27fc700 1 -- 172.21.15.114:0/2756083070 <== mgr.14120 v2:172.21.15.114:6800/1604678557 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+595 (secure 0 0 0) 0x7fe0bc19fa80 con 0x7fe0a80395b0 2024-04-23T10:17:26.599 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-04-23T10:17:26.072+0000 7fe0c360d700 1 -- 172.21.15.114:0/2756083070 >> [v2:172.21.15.114:6800/1604678557,v1:172.21.15.114:6801/1604678557] conn(0x7fe0a80395b0 msgr2=0x7fe0a803ba70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T10:17:26.600 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-04-23T10:17:26.072+0000 7fe0c360d700 1 --2- 172.21.15.114:0/2756083070 >> [v2:172.21.15.114:6800/1604678557,v1:172.21.15.114:6801/1604678557] conn(0x7fe0a80395b0 0x7fe0a803ba70 secure :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0x7fe0b8006fd0 tx=0x7fe0b8006e40 comp rx=0 tx=0).stop 2024-04-23T10:17:26.600 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-04-23T10:17:26.072+0000 7fe0c360d700 1 -- 172.21.15.114:0/2756083070 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fe0bc108fd0 msgr2=0x7fe0bc19c3f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T10:17:26.600 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-04-23T10:17:26.072+0000 7fe0c360d700 1 --2- 172.21.15.114:0/2756083070 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fe0bc108fd0 0x7fe0bc19c3f0 secure :-1 s=READY pgs=44 cs=0 l=1 rev1=1 crypto rx=0x7fe0ac011ac0 tx=0x7fe0ac011ba0 comp rx=0 tx=0).stop 2024-04-23T10:17:26.600 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-04-23T10:17:26.072+0000 7fe0c360d700 1 -- 172.21.15.114:0/2756083070 shutdown_connections 2024-04-23T10:17:26.600 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-04-23T10:17:26.072+0000 7fe0c360d700 1 --2- 172.21.15.114:0/2756083070 >> [v2:172.21.15.114:6800/1604678557,v1:172.21.15.114:6801/1604678557] conn(0x7fe0a80395b0 0x7fe0a803ba70 unknown :-1 s=CLOSED pgs=10 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:17:26.600 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-04-23T10:17:26.072+0000 7fe0c360d700 1 --2- 172.21.15.114:0/2756083070 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fe0bc108fd0 0x7fe0bc19c3f0 unknown :-1 s=CLOSED pgs=44 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:17:26.600 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-04-23T10:17:26.072+0000 7fe0c360d700 1 -- 172.21.15.114:0/2756083070 >> 172.21.15.114:0/2756083070 conn(0x7fe0bc102a40 msgr2=0x7fe0bc1034e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T10:17:26.600 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-04-23T10:17:26.072+0000 7fe0c360d700 1 -- 172.21.15.114:0/2756083070 shutdown_connections 2024-04-23T10:17:26.600 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-04-23T10:17:26.072+0000 7fe0c360d700 1 -- 172.21.15.114:0/2756083070 wait complete. 2024-04-23T10:17:26.600 INFO:teuthology.orchestra.run.smithi114.stdout:Wrote public SSH key to /home/ubuntu/cephtest/ceph.pub 2024-04-23T10:17:26.600 INFO:teuthology.orchestra.run.smithi114.stdout:Adding key to root@localhost authorized_keys... 2024-04-23T10:17:26.600 INFO:teuthology.orchestra.run.smithi114.stdout:Adding host smithi114... 2024-04-23T10:17:27.242 INFO:journalctl@ceph.mon.a.smithi114.stdout:Apr 23 10:17:27 smithi114 ceph-mon[29255]: from='client.14138 -' entity='client.admin' cmd=[{"prefix": "cephadm get-pub-key", "target": ["mon-mgr", ""]}]: dispatch 2024-04-23T10:17:28.504 INFO:journalctl@ceph.mon.a.smithi114.stdout:Apr 23 10:17:28 smithi114 ceph-mon[29255]: from='client.14140 -' entity='client.admin' cmd=[{"prefix": "orch host add", "hostname": "smithi114", "addr": "172.21.15.114", "target": ["mon-mgr", ""]}]: dispatch 2024-04-23T10:17:28.504 INFO:journalctl@ceph.mon.a.smithi114.stdout:Apr 23 10:17:28 smithi114 ceph-mon[29255]: Deploying cephadm binary to smithi114 2024-04-23T10:17:31.841 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stdout Added host 'smithi114' with addr '172.21.15.114' 2024-04-23T10:17:31.841 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-04-23T10:17:27.010+0000 7ff35e5f9700 1 Processor -- start 2024-04-23T10:17:31.841 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-04-23T10:17:27.010+0000 7ff35e5f9700 1 -- start start 2024-04-23T10:17:31.841 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-04-23T10:17:27.010+0000 7ff35e5f9700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7ff358107900 0x7ff358107d20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T10:17:31.842 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-04-23T10:17:27.010+0000 7ff35e5f9700 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x7ff3581082f0 con 0x7ff358107900 2024-04-23T10:17:31.842 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-04-23T10:17:27.010+0000 7ff357fff700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7ff358107900 0x7ff358107d20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T10:17:31.842 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-04-23T10:17:27.010+0000 7ff357fff700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7ff358107900 0x7ff358107d20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.114:40362/0 (socket says 172.21.15.114:40362) 2024-04-23T10:17:31.842 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-04-23T10:17:27.010+0000 7ff357fff700 1 -- 172.21.15.114:0/879267509 learned_addr learned my addr 172.21.15.114:0/879267509 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-04-23T10:17:31.842 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-04-23T10:17:27.011+0000 7ff357fff700 1 -- 172.21.15.114:0/879267509 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff358108620 con 0x7ff358107900 2024-04-23T10:17:31.842 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-04-23T10:17:27.011+0000 7ff357fff700 1 --2- 172.21.15.114:0/879267509 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7ff358107900 0x7ff358107d20 secure :-1 s=READY pgs=45 cs=0 l=1 rev1=1 crypto rx=0x7ff3480098a0 tx=0x7ff348009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6a21c91177a9c8ab server_cookie=0 in_seq=0 out_seq=0 2024-04-23T10:17:31.842 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-04-23T10:17:27.011+0000 7ff356ffd700 1 -- 172.21.15.114:0/879267509 <== mon.0 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7ff348004160 con 0x7ff358107900 2024-04-23T10:17:31.842 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-04-23T10:17:27.011+0000 7ff356ffd700 1 -- 172.21.15.114:0/879267509 <== mon.0 v2:172.21.15.114:3300/0 2 ==== config(26 keys) ==== 1070+0+0 (secure 0 0 0) 0x7ff348004300 con 0x7ff358107900 2024-04-23T10:17:31.842 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-04-23T10:17:27.011+0000 7ff356ffd700 1 -- 172.21.15.114:0/879267509 <== mon.0 v2:172.21.15.114:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7ff348011af0 con 0x7ff358107900 2024-04-23T10:17:31.843 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-04-23T10:17:27.012+0000 7ff35e5f9700 1 -- 172.21.15.114:0/879267509 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7ff358107900 msgr2=0x7ff358107d20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T10:17:31.843 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-04-23T10:17:27.012+0000 7ff35e5f9700 1 --2- 172.21.15.114:0/879267509 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7ff358107900 0x7ff358107d20 secure :-1 s=READY pgs=45 cs=0 l=1 rev1=1 crypto rx=0x7ff3480098a0 tx=0x7ff348009bb0 comp rx=0 tx=0).stop 2024-04-23T10:17:31.843 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-04-23T10:17:27.012+0000 7ff35e5f9700 1 -- 172.21.15.114:0/879267509 shutdown_connections 2024-04-23T10:17:31.843 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-04-23T10:17:27.012+0000 7ff35e5f9700 1 --2- 172.21.15.114:0/879267509 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7ff358107900 0x7ff358107d20 unknown :-1 s=CLOSED pgs=45 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:17:31.843 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-04-23T10:17:27.012+0000 7ff35e5f9700 1 -- 172.21.15.114:0/879267509 >> 172.21.15.114:0/879267509 conn(0x7ff358102140 msgr2=0x7ff358104560 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T10:17:31.843 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-04-23T10:17:27.012+0000 7ff35e5f9700 1 -- 172.21.15.114:0/879267509 shutdown_connections 2024-04-23T10:17:31.843 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-04-23T10:17:27.012+0000 7ff35e5f9700 1 -- 172.21.15.114:0/879267509 wait complete. 2024-04-23T10:17:31.843 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-04-23T10:17:27.012+0000 7ff35e5f9700 1 Processor -- start 2024-04-23T10:17:31.843 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-04-23T10:17:27.012+0000 7ff35e5f9700 1 -- start start 2024-04-23T10:17:31.844 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-04-23T10:17:27.013+0000 7ff35e5f9700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7ff358107900 0x7ff35819c480 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T10:17:31.844 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-04-23T10:17:27.013+0000 7ff35e5f9700 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x7ff35810b9d0 con 0x7ff358107900 2024-04-23T10:17:31.844 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-04-23T10:17:27.013+0000 7ff357fff700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7ff358107900 0x7ff35819c480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T10:17:31.844 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-04-23T10:17:27.013+0000 7ff357fff700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7ff358107900 0x7ff35819c480 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.114:40364/0 (socket says 172.21.15.114:40364) 2024-04-23T10:17:31.844 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-04-23T10:17:27.013+0000 7ff357fff700 1 -- 172.21.15.114:0/4128233737 learned_addr learned my addr 172.21.15.114:0/4128233737 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-04-23T10:17:31.844 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-04-23T10:17:27.013+0000 7ff357fff700 1 -- 172.21.15.114:0/4128233737 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff35819c9c0 con 0x7ff358107900 2024-04-23T10:17:31.844 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-04-23T10:17:27.013+0000 7ff357fff700 1 --2- 172.21.15.114:0/4128233737 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7ff358107900 0x7ff35819c480 secure :-1 s=READY pgs=46 cs=0 l=1 rev1=1 crypto rx=0x7ff348003f40 tx=0x7ff348011890 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T10:17:31.844 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-04-23T10:17:27.013+0000 7ff3557fa700 1 -- 172.21.15.114:0/4128233737 <== mon.0 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7ff3480038b0 con 0x7ff358107900 2024-04-23T10:17:31.844 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-04-23T10:17:27.013+0000 7ff3557fa700 1 -- 172.21.15.114:0/4128233737 <== mon.0 v2:172.21.15.114:3300/0 2 ==== config(26 keys) ==== 1070+0+0 (secure 0 0 0) 0x7ff348003a50 con 0x7ff358107900 2024-04-23T10:17:31.844 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-04-23T10:17:27.013+0000 7ff35e5f9700 1 -- 172.21.15.114:0/4128233737 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7ff35819cc50 con 0x7ff358107900 2024-04-23T10:17:31.845 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-04-23T10:17:27.013+0000 7ff35e5f9700 1 -- 172.21.15.114:0/4128233737 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7ff35819f940 con 0x7ff358107900 2024-04-23T10:17:31.845 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-04-23T10:17:27.013+0000 7ff3557fa700 1 -- 172.21.15.114:0/4128233737 <== mon.0 v2:172.21.15.114:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7ff34801a5c0 con 0x7ff358107900 2024-04-23T10:17:31.845 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-04-23T10:17:27.014+0000 7ff3557fa700 1 -- 172.21.15.114:0/4128233737 <== mon.0 v2:172.21.15.114:3300/0 4 ==== mgrmap(e 8) ==== 46558+0+0 (secure 0 0 0) 0x7ff34801a8c0 con 0x7ff358107900 2024-04-23T10:17:31.845 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-04-23T10:17:27.014+0000 7ff3557fa700 1 --2- 172.21.15.114:0/4128233737 >> [v2:172.21.15.114:6800/1604678557,v1:172.21.15.114:6801/1604678557] conn(0x7ff340039560 0x7ff34003ba20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T10:17:31.845 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-04-23T10:17:27.014+0000 7ff3557fa700 1 -- 172.21.15.114:0/4128233737 <== mon.0 v2:172.21.15.114:3300/0 5 ==== osd_map(2..2 src has 1..2) ==== 940+0+0 (secure 0 0 0) 0x7ff34804e9f0 con 0x7ff358107900 2024-04-23T10:17:31.845 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-04-23T10:17:27.014+0000 7ff3577fe700 1 --2- 172.21.15.114:0/4128233737 >> [v2:172.21.15.114:6800/1604678557,v1:172.21.15.114:6801/1604678557] conn(0x7ff340039560 0x7ff34003ba20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T10:17:31.845 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-04-23T10:17:27.014+0000 7ff35e5f9700 1 -- 172.21.15.114:0/4128233737 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7ff344005330 con 0x7ff358107900 2024-04-23T10:17:31.845 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-04-23T10:17:27.015+0000 7ff3577fe700 1 --2- 172.21.15.114:0/4128233737 >> [v2:172.21.15.114:6800/1604678557,v1:172.21.15.114:6801/1604678557] conn(0x7ff340039560 0x7ff34003ba20 secure :-1 s=READY pgs=11 cs=0 l=1 rev1=1 crypto rx=0x7ff34c006fd0 tx=0x7ff34c006e40 comp rx=0 tx=0).ready entity=mgr.14120 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T10:17:31.845 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-04-23T10:17:27.019+0000 7ff3557fa700 1 -- 172.21.15.114:0/4128233737 <== mon.0 v2:172.21.15.114:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7ff348019c70 con 0x7ff358107900 2024-04-23T10:17:31.845 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-04-23T10:17:27.224+0000 7ff35e5f9700 1 -- 172.21.15.114:0/4128233737 --> [v2:172.21.15.114:6800/1604678557,v1:172.21.15.114:6801/1604678557] -- mgr_command(tid 0: {"prefix": "orch host add", "hostname": "smithi114", "addr": "172.21.15.114", "target": ["mon-mgr", ""]}) -- 0x7ff344000bc0 con 0x7ff340039560 2024-04-23T10:17:31.845 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-04-23T10:17:31.311+0000 7ff3557fa700 1 -- 172.21.15.114:0/4128233737 <== mgr.14120 v2:172.21.15.114:6800/1604678557 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+49 (secure 0 0 0) 0x7ff344000bc0 con 0x7ff340039560 2024-04-23T10:17:31.846 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-04-23T10:17:31.314+0000 7ff35e5f9700 1 -- 172.21.15.114:0/4128233737 >> [v2:172.21.15.114:6800/1604678557,v1:172.21.15.114:6801/1604678557] conn(0x7ff340039560 msgr2=0x7ff34003ba20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T10:17:31.846 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-04-23T10:17:31.314+0000 7ff35e5f9700 1 --2- 172.21.15.114:0/4128233737 >> [v2:172.21.15.114:6800/1604678557,v1:172.21.15.114:6801/1604678557] conn(0x7ff340039560 0x7ff34003ba20 secure :-1 s=READY pgs=11 cs=0 l=1 rev1=1 crypto rx=0x7ff34c006fd0 tx=0x7ff34c006e40 comp rx=0 tx=0).stop 2024-04-23T10:17:31.846 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-04-23T10:17:31.314+0000 7ff35e5f9700 1 -- 172.21.15.114:0/4128233737 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7ff358107900 msgr2=0x7ff35819c480 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T10:17:31.846 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-04-23T10:17:31.314+0000 7ff35e5f9700 1 --2- 172.21.15.114:0/4128233737 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7ff358107900 0x7ff35819c480 secure :-1 s=READY pgs=46 cs=0 l=1 rev1=1 crypto rx=0x7ff348003f40 tx=0x7ff348011890 comp rx=0 tx=0).stop 2024-04-23T10:17:31.846 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-04-23T10:17:31.314+0000 7ff35e5f9700 1 -- 172.21.15.114:0/4128233737 shutdown_connections 2024-04-23T10:17:31.846 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-04-23T10:17:31.314+0000 7ff35e5f9700 1 --2- 172.21.15.114:0/4128233737 >> [v2:172.21.15.114:6800/1604678557,v1:172.21.15.114:6801/1604678557] conn(0x7ff340039560 0x7ff34003ba20 unknown :-1 s=CLOSED pgs=11 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:17:31.846 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-04-23T10:17:31.314+0000 7ff35e5f9700 1 --2- 172.21.15.114:0/4128233737 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7ff358107900 0x7ff35819c480 unknown :-1 s=CLOSED pgs=46 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:17:31.846 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-04-23T10:17:31.314+0000 7ff35e5f9700 1 -- 172.21.15.114:0/4128233737 >> 172.21.15.114:0/4128233737 conn(0x7ff358102140 msgr2=0x7ff358102fa0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T10:17:31.846 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-04-23T10:17:31.314+0000 7ff35e5f9700 1 -- 172.21.15.114:0/4128233737 shutdown_connections 2024-04-23T10:17:31.846 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-04-23T10:17:31.314+0000 7ff35e5f9700 1 -- 172.21.15.114:0/4128233737 wait complete. 2024-04-23T10:17:31.847 INFO:teuthology.orchestra.run.smithi114.stdout:Deploying unmanaged mon service... 2024-04-23T10:17:32.504 INFO:journalctl@ceph.mon.a.smithi114.stdout:Apr 23 10:17:32 smithi114 ceph-mon[29255]: from='mgr.14120 172.21.15.114:0/1448995927' entity='mgr.x' 2024-04-23T10:17:32.505 INFO:journalctl@ceph.mon.a.smithi114.stdout:Apr 23 10:17:32 smithi114 ceph-mon[29255]: Added host smithi114 2024-04-23T10:17:32.505 INFO:journalctl@ceph.mon.a.smithi114.stdout:Apr 23 10:17:32 smithi114 ceph-mon[29255]: from='mgr.14120 172.21.15.114:0/1448995927' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-23T10:17:33.090 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stdout Scheduled mon update... 2024-04-23T10:17:33.090 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-04-23T10:17:32.304+0000 7f632e67c700 1 Processor -- start 2024-04-23T10:17:33.090 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-04-23T10:17:32.304+0000 7f632e67c700 1 -- start start 2024-04-23T10:17:33.090 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-04-23T10:17:32.304+0000 7f632e67c700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f6328108560 0x7f6328108980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T10:17:33.091 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-04-23T10:17:32.304+0000 7f632e67c700 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x7f6328108f50 con 0x7f6328108560 2024-04-23T10:17:33.091 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-04-23T10:17:32.305+0000 7f6327fff700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f6328108560 0x7f6328108980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T10:17:33.091 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-04-23T10:17:32.305+0000 7f6327fff700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f6328108560 0x7f6328108980 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.114:40372/0 (socket says 172.21.15.114:40372) 2024-04-23T10:17:33.091 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-04-23T10:17:32.305+0000 7f6327fff700 1 -- 172.21.15.114:0/2696646524 learned_addr learned my addr 172.21.15.114:0/2696646524 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-04-23T10:17:33.091 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-04-23T10:17:32.305+0000 7f6327fff700 1 -- 172.21.15.114:0/2696646524 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f6328109710 con 0x7f6328108560 2024-04-23T10:17:33.091 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-04-23T10:17:32.305+0000 7f6327fff700 1 --2- 172.21.15.114:0/2696646524 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f6328108560 0x7f6328108980 secure :-1 s=READY pgs=47 cs=0 l=1 rev1=1 crypto rx=0x7f63180098a0 tx=0x7f6318009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7fb89fc7e5235311 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T10:17:33.091 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-04-23T10:17:32.305+0000 7f6326ffd700 1 -- 172.21.15.114:0/2696646524 <== mon.0 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f6318004160 con 0x7f6328108560 2024-04-23T10:17:33.091 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-04-23T10:17:32.305+0000 7f6326ffd700 1 -- 172.21.15.114:0/2696646524 <== mon.0 v2:172.21.15.114:3300/0 2 ==== config(26 keys) ==== 1070+0+0 (secure 0 0 0) 0x7f6318004300 con 0x7f6328108560 2024-04-23T10:17:33.091 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-04-23T10:17:32.305+0000 7f6326ffd700 1 -- 172.21.15.114:0/2696646524 <== mon.0 v2:172.21.15.114:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f6318011af0 con 0x7f6328108560 2024-04-23T10:17:33.091 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-04-23T10:17:32.306+0000 7f632e67c700 1 -- 172.21.15.114:0/2696646524 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f6328108560 msgr2=0x7f6328108980 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T10:17:33.091 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-04-23T10:17:32.306+0000 7f632e67c700 1 --2- 172.21.15.114:0/2696646524 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f6328108560 0x7f6328108980 secure :-1 s=READY pgs=47 cs=0 l=1 rev1=1 crypto rx=0x7f63180098a0 tx=0x7f6318009bb0 comp rx=0 tx=0).stop 2024-04-23T10:17:33.092 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-04-23T10:17:32.306+0000 7f632e67c700 1 -- 172.21.15.114:0/2696646524 shutdown_connections 2024-04-23T10:17:33.092 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-04-23T10:17:32.306+0000 7f632e67c700 1 --2- 172.21.15.114:0/2696646524 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f6328108560 0x7f6328108980 unknown :-1 s=CLOSED pgs=47 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:17:33.092 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-04-23T10:17:32.306+0000 7f632e67c700 1 -- 172.21.15.114:0/2696646524 >> 172.21.15.114:0/2696646524 conn(0x7f6328103b40 msgr2=0x7f6328105f60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T10:17:33.092 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-04-23T10:17:32.306+0000 7f632e67c700 1 -- 172.21.15.114:0/2696646524 shutdown_connections 2024-04-23T10:17:33.092 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-04-23T10:17:32.306+0000 7f632e67c700 1 -- 172.21.15.114:0/2696646524 wait complete. 2024-04-23T10:17:33.092 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-04-23T10:17:32.306+0000 7f632e67c700 1 Processor -- start 2024-04-23T10:17:33.092 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-04-23T10:17:32.306+0000 7f632e67c700 1 -- start start 2024-04-23T10:17:33.092 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-04-23T10:17:32.307+0000 7f632e67c700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f6328108560 0x7f632819c4f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T10:17:33.092 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-04-23T10:17:32.307+0000 7f632e67c700 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x7f632810cac0 con 0x7f6328108560 2024-04-23T10:17:33.092 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-04-23T10:17:32.307+0000 7f6327fff700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f6328108560 0x7f632819c4f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T10:17:33.092 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-04-23T10:17:32.307+0000 7f6327fff700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f6328108560 0x7f632819c4f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.114:40382/0 (socket says 172.21.15.114:40382) 2024-04-23T10:17:33.093 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-04-23T10:17:32.307+0000 7f6327fff700 1 -- 172.21.15.114:0/743717485 learned_addr learned my addr 172.21.15.114:0/743717485 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-04-23T10:17:33.093 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-04-23T10:17:32.307+0000 7f6327fff700 1 -- 172.21.15.114:0/743717485 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f632819ca30 con 0x7f6328108560 2024-04-23T10:17:33.093 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-04-23T10:17:32.307+0000 7f6327fff700 1 --2- 172.21.15.114:0/743717485 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f6328108560 0x7f632819c4f0 secure :-1 s=READY pgs=48 cs=0 l=1 rev1=1 crypto rx=0x7f6318009fb0 tx=0x7f6318011940 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T10:17:33.093 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-04-23T10:17:32.307+0000 7f63257fa700 1 -- 172.21.15.114:0/743717485 <== mon.0 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f63180036a0 con 0x7f6328108560 2024-04-23T10:17:33.093 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-04-23T10:17:32.308+0000 7f63257fa700 1 -- 172.21.15.114:0/743717485 <== mon.0 v2:172.21.15.114:3300/0 2 ==== config(26 keys) ==== 1070+0+0 (secure 0 0 0) 0x7f631801a430 con 0x7f6328108560 2024-04-23T10:17:33.093 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-04-23T10:17:32.308+0000 7f632e67c700 1 -- 172.21.15.114:0/743717485 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f632819ccc0 con 0x7f6328108560 2024-04-23T10:17:33.093 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-04-23T10:17:32.308+0000 7f632e67c700 1 -- 172.21.15.114:0/743717485 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f632819f9b0 con 0x7f6328108560 2024-04-23T10:17:33.093 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-04-23T10:17:32.308+0000 7f63257fa700 1 -- 172.21.15.114:0/743717485 <== mon.0 v2:172.21.15.114:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f6318019550 con 0x7f6328108560 2024-04-23T10:17:33.093 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-04-23T10:17:32.308+0000 7f63257fa700 1 -- 172.21.15.114:0/743717485 <== mon.0 v2:172.21.15.114:3300/0 4 ==== mgrmap(e 8) ==== 46558+0+0 (secure 0 0 0) 0x7f6318019800 con 0x7f6328108560 2024-04-23T10:17:33.093 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-04-23T10:17:32.308+0000 7f632e67c700 1 -- 172.21.15.114:0/743717485 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f6314005360 con 0x7f6328108560 2024-04-23T10:17:33.094 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-04-23T10:17:32.309+0000 7f63257fa700 1 --2- 172.21.15.114:0/743717485 >> [v2:172.21.15.114:6800/1604678557,v1:172.21.15.114:6801/1604678557] conn(0x7f63100395b0 0x7f631003ba70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T10:17:33.094 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-04-23T10:17:32.309+0000 7f63257fa700 1 -- 172.21.15.114:0/743717485 <== mon.0 v2:172.21.15.114:3300/0 5 ==== osd_map(2..2 src has 1..2) ==== 940+0+0 (secure 0 0 0) 0x7f631804d880 con 0x7f6328108560 2024-04-23T10:17:33.094 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-04-23T10:17:32.309+0000 7f63277fe700 1 --2- 172.21.15.114:0/743717485 >> [v2:172.21.15.114:6800/1604678557,v1:172.21.15.114:6801/1604678557] conn(0x7f63100395b0 0x7f631003ba70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T10:17:33.094 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-04-23T10:17:32.309+0000 7f63277fe700 1 --2- 172.21.15.114:0/743717485 >> [v2:172.21.15.114:6800/1604678557,v1:172.21.15.114:6801/1604678557] conn(0x7f63100395b0 0x7f631003ba70 secure :-1 s=READY pgs=12 cs=0 l=1 rev1=1 crypto rx=0x7f631c006fd0 tx=0x7f631c006e40 comp rx=0 tx=0).ready entity=mgr.14120 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T10:17:33.094 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-04-23T10:17:32.314+0000 7f63257fa700 1 -- 172.21.15.114:0/743717485 <== mon.0 v2:172.21.15.114:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f6318016dd0 con 0x7f6328108560 2024-04-23T10:17:33.094 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-04-23T10:17:32.527+0000 7f632e67c700 1 -- 172.21.15.114:0/743717485 --> [v2:172.21.15.114:6800/1604678557,v1:172.21.15.114:6801/1604678557] -- mgr_command(tid 0: {"prefix": "orch apply", "service_type": "mon", "unmanaged": true, "target": ["mon-mgr", ""]}) -- 0x7f6314000bf0 con 0x7f63100395b0 2024-04-23T10:17:33.094 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-04-23T10:17:32.535+0000 7f63257fa700 1 -- 172.21.15.114:0/743717485 <== mgr.14120 v2:172.21.15.114:6800/1604678557 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+24 (secure 0 0 0) 0x7f6314000bf0 con 0x7f63100395b0 2024-04-23T10:17:33.094 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-04-23T10:17:32.537+0000 7f632e67c700 1 -- 172.21.15.114:0/743717485 >> [v2:172.21.15.114:6800/1604678557,v1:172.21.15.114:6801/1604678557] conn(0x7f63100395b0 msgr2=0x7f631003ba70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T10:17:33.094 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-04-23T10:17:32.537+0000 7f632e67c700 1 --2- 172.21.15.114:0/743717485 >> [v2:172.21.15.114:6800/1604678557,v1:172.21.15.114:6801/1604678557] conn(0x7f63100395b0 0x7f631003ba70 secure :-1 s=READY pgs=12 cs=0 l=1 rev1=1 crypto rx=0x7f631c006fd0 tx=0x7f631c006e40 comp rx=0 tx=0).stop 2024-04-23T10:17:33.094 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-04-23T10:17:32.537+0000 7f632e67c700 1 -- 172.21.15.114:0/743717485 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f6328108560 msgr2=0x7f632819c4f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T10:17:33.094 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-04-23T10:17:32.537+0000 7f632e67c700 1 --2- 172.21.15.114:0/743717485 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f6328108560 0x7f632819c4f0 secure :-1 s=READY pgs=48 cs=0 l=1 rev1=1 crypto rx=0x7f6318009fb0 tx=0x7f6318011940 comp rx=0 tx=0).stop 2024-04-23T10:17:33.094 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-04-23T10:17:32.537+0000 7f632e67c700 1 -- 172.21.15.114:0/743717485 shutdown_connections 2024-04-23T10:17:33.094 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-04-23T10:17:32.537+0000 7f632e67c700 1 --2- 172.21.15.114:0/743717485 >> [v2:172.21.15.114:6800/1604678557,v1:172.21.15.114:6801/1604678557] conn(0x7f63100395b0 0x7f631003ba70 unknown :-1 s=CLOSED pgs=12 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:17:33.095 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-04-23T10:17:32.537+0000 7f632e67c700 1 --2- 172.21.15.114:0/743717485 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f6328108560 0x7f632819c4f0 unknown :-1 s=CLOSED pgs=48 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:17:33.095 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-04-23T10:17:32.537+0000 7f632e67c700 1 -- 172.21.15.114:0/743717485 >> 172.21.15.114:0/743717485 conn(0x7f6328103b40 msgr2=0x7f63281046f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T10:17:33.095 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-04-23T10:17:32.537+0000 7f632e67c700 1 -- 172.21.15.114:0/743717485 shutdown_connections 2024-04-23T10:17:33.095 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-04-23T10:17:32.537+0000 7f632e67c700 1 -- 172.21.15.114:0/743717485 wait complete. 2024-04-23T10:17:33.095 INFO:teuthology.orchestra.run.smithi114.stdout:Deploying unmanaged mgr service... 2024-04-23T10:17:33.652 INFO:journalctl@ceph.mon.a.smithi114.stdout:Apr 23 10:17:33 smithi114 ceph-mon[29255]: from='client.14142 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "mon", "unmanaged": true, "target": ["mon-mgr", ""]}]: dispatch 2024-04-23T10:17:33.652 INFO:journalctl@ceph.mon.a.smithi114.stdout:Apr 23 10:17:33 smithi114 ceph-mon[29255]: Saving service mon spec with placement count:5 2024-04-23T10:17:33.652 INFO:journalctl@ceph.mon.a.smithi114.stdout:Apr 23 10:17:33 smithi114 ceph-mon[29255]: from='mgr.14120 172.21.15.114:0/1448995927' entity='mgr.x' 2024-04-23T10:17:35.070 INFO:journalctl@ceph.mon.a.smithi114.stdout:Apr 23 10:17:35 smithi114 ceph-mon[29255]: from='client.14144 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "mgr", "unmanaged": true, "target": ["mon-mgr", ""]}]: dispatch 2024-04-23T10:17:35.070 INFO:journalctl@ceph.mon.a.smithi114.stdout:Apr 23 10:17:35 smithi114 ceph-mon[29255]: Saving service mgr spec with placement count:2 2024-04-23T10:17:35.070 INFO:journalctl@ceph.mon.a.smithi114.stdout:Apr 23 10:17:35 smithi114 ceph-mon[29255]: from='mgr.14120 172.21.15.114:0/1448995927' entity='mgr.x' 2024-04-23T10:17:35.070 INFO:journalctl@ceph.mon.a.smithi114.stdout:Apr 23 10:17:35 smithi114 ceph-mon[29255]: from='mgr.14120 172.21.15.114:0/1448995927' entity='mgr.x' 2024-04-23T10:17:35.078 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stdout Scheduled mgr update... 2024-04-23T10:17:35.078 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-04-23T10:17:33.836+0000 7fce4f87b700 1 Processor -- start 2024-04-23T10:17:35.078 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-04-23T10:17:33.836+0000 7fce4f87b700 1 -- start start 2024-04-23T10:17:35.078 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-04-23T10:17:33.836+0000 7fce4f87b700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fce48108ff0 0x7fce4810b3e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T10:17:35.078 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-04-23T10:17:33.837+0000 7fce4f87b700 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x7fce48063d20 con 0x7fce48108ff0 2024-04-23T10:17:35.078 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-04-23T10:17:33.837+0000 7fce4d617700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fce48108ff0 0x7fce4810b3e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T10:17:35.078 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-04-23T10:17:33.837+0000 7fce4d617700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fce48108ff0 0x7fce4810b3e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.114:40384/0 (socket says 172.21.15.114:40384) 2024-04-23T10:17:35.078 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-04-23T10:17:33.837+0000 7fce4d617700 1 -- 172.21.15.114:0/1558777287 learned_addr learned my addr 172.21.15.114:0/1558777287 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-04-23T10:17:35.078 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-04-23T10:17:33.837+0000 7fce4d617700 1 -- 172.21.15.114:0/1558777287 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fce4810b920 con 0x7fce48108ff0 2024-04-23T10:17:35.078 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-04-23T10:17:33.837+0000 7fce4d617700 1 --2- 172.21.15.114:0/1558777287 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fce48108ff0 0x7fce4810b3e0 secure :-1 s=READY pgs=49 cs=0 l=1 rev1=1 crypto rx=0x7fce44009e80 tx=0x7fce4400b330 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1adf0b22e72c7e55 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T10:17:35.078 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-04-23T10:17:33.837+0000 7fce3ffff700 1 -- 172.21.15.114:0/1558777287 <== mon.0 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fce44017070 con 0x7fce48108ff0 2024-04-23T10:17:35.078 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-04-23T10:17:33.837+0000 7fce3ffff700 1 -- 172.21.15.114:0/1558777287 <== mon.0 v2:172.21.15.114:3300/0 2 ==== config(26 keys) ==== 1070+0+0 (secure 0 0 0) 0x7fce4400b8c0 con 0x7fce48108ff0 2024-04-23T10:17:35.078 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-04-23T10:17:33.837+0000 7fce3ffff700 1 -- 172.21.15.114:0/1558777287 <== mon.0 v2:172.21.15.114:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fce44004490 con 0x7fce48108ff0 2024-04-23T10:17:35.078 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-04-23T10:17:33.838+0000 7fce4f87b700 1 -- 172.21.15.114:0/1558777287 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fce48108ff0 msgr2=0x7fce4810b3e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T10:17:35.079 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-04-23T10:17:33.838+0000 7fce4f87b700 1 --2- 172.21.15.114:0/1558777287 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fce48108ff0 0x7fce4810b3e0 secure :-1 s=READY pgs=49 cs=0 l=1 rev1=1 crypto rx=0x7fce44009e80 tx=0x7fce4400b330 comp rx=0 tx=0).stop 2024-04-23T10:17:35.079 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-04-23T10:17:33.838+0000 7fce4f87b700 1 -- 172.21.15.114:0/1558777287 shutdown_connections 2024-04-23T10:17:35.079 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-04-23T10:17:33.838+0000 7fce4f87b700 1 --2- 172.21.15.114:0/1558777287 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fce48108ff0 0x7fce4810b3e0 unknown :-1 s=CLOSED pgs=49 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:17:35.079 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-04-23T10:17:33.838+0000 7fce4f87b700 1 -- 172.21.15.114:0/1558777287 >> 172.21.15.114:0/1558777287 conn(0x7fce48102a40 msgr2=0x7fce48104e60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T10:17:35.079 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-04-23T10:17:33.838+0000 7fce4f87b700 1 -- 172.21.15.114:0/1558777287 shutdown_connections 2024-04-23T10:17:35.079 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-04-23T10:17:33.838+0000 7fce4f87b700 1 -- 172.21.15.114:0/1558777287 wait complete. 2024-04-23T10:17:35.079 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-04-23T10:17:33.839+0000 7fce4f87b700 1 Processor -- start 2024-04-23T10:17:35.079 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-04-23T10:17:33.839+0000 7fce4f87b700 1 -- start start 2024-04-23T10:17:35.079 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-04-23T10:17:33.839+0000 7fce4f87b700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fce48108ff0 0x7fce4819c3f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T10:17:35.079 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-04-23T10:17:33.839+0000 7fce4f87b700 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x7fce4810c8e0 con 0x7fce48108ff0 2024-04-23T10:17:35.079 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-04-23T10:17:33.839+0000 7fce4d617700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fce48108ff0 0x7fce4819c3f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T10:17:35.079 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-04-23T10:17:33.839+0000 7fce4d617700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fce48108ff0 0x7fce4819c3f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.114:40386/0 (socket says 172.21.15.114:40386) 2024-04-23T10:17:35.079 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-04-23T10:17:33.839+0000 7fce4d617700 1 -- 172.21.15.114:0/1278741194 learned_addr learned my addr 172.21.15.114:0/1278741194 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-04-23T10:17:35.079 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-04-23T10:17:33.839+0000 7fce4d617700 1 -- 172.21.15.114:0/1278741194 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fce4819c930 con 0x7fce48108ff0 2024-04-23T10:17:35.079 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-04-23T10:17:33.839+0000 7fce4d617700 1 --2- 172.21.15.114:0/1278741194 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fce48108ff0 0x7fce4819c3f0 secure :-1 s=READY pgs=50 cs=0 l=1 rev1=1 crypto rx=0x7fce4400b7b0 tx=0x7fce4400be20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T10:17:35.079 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-04-23T10:17:33.839+0000 7fce3e7fc700 1 -- 172.21.15.114:0/1278741194 <== mon.0 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fce44020020 con 0x7fce48108ff0 2024-04-23T10:17:35.080 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-04-23T10:17:33.839+0000 7fce4f87b700 1 -- 172.21.15.114:0/1278741194 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fce4819cbc0 con 0x7fce48108ff0 2024-04-23T10:17:35.080 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-04-23T10:17:33.839+0000 7fce3e7fc700 1 -- 172.21.15.114:0/1278741194 <== mon.0 v2:172.21.15.114:3300/0 2 ==== config(26 keys) ==== 1070+0+0 (secure 0 0 0) 0x7fce44012430 con 0x7fce48108ff0 2024-04-23T10:17:35.080 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-04-23T10:17:33.839+0000 7fce3e7fc700 1 -- 172.21.15.114:0/1278741194 <== mon.0 v2:172.21.15.114:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fce44017040 con 0x7fce48108ff0 2024-04-23T10:17:35.080 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-04-23T10:17:33.840+0000 7fce4f87b700 1 -- 172.21.15.114:0/1278741194 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fce4819f8b0 con 0x7fce48108ff0 2024-04-23T10:17:35.080 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-04-23T10:17:33.840+0000 7fce3e7fc700 1 -- 172.21.15.114:0/1278741194 <== mon.0 v2:172.21.15.114:3300/0 4 ==== mgrmap(e 8) ==== 46558+0+0 (secure 0 0 0) 0x7fce44023020 con 0x7fce48108ff0 2024-04-23T10:17:35.080 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-04-23T10:17:33.840+0000 7fce3e7fc700 1 --2- 172.21.15.114:0/1278741194 >> [v2:172.21.15.114:6800/1604678557,v1:172.21.15.114:6801/1604678557] conn(0x7fce34039560 0x7fce3403ba20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T10:17:35.080 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-04-23T10:17:33.840+0000 7fce3e7fc700 1 -- 172.21.15.114:0/1278741194 <== mon.0 v2:172.21.15.114:3300/0 5 ==== osd_map(2..2 src has 1..2) ==== 940+0+0 (secure 0 0 0) 0x7fce4400f050 con 0x7fce48108ff0 2024-04-23T10:17:35.080 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-04-23T10:17:33.841+0000 7fce4ce16700 1 --2- 172.21.15.114:0/1278741194 >> [v2:172.21.15.114:6800/1604678557,v1:172.21.15.114:6801/1604678557] conn(0x7fce34039560 0x7fce3403ba20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T10:17:35.080 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-04-23T10:17:33.841+0000 7fce4f87b700 1 -- 172.21.15.114:0/1278741194 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fce480640a0 con 0x7fce48108ff0 2024-04-23T10:17:35.080 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-04-23T10:17:33.841+0000 7fce4ce16700 1 --2- 172.21.15.114:0/1278741194 >> [v2:172.21.15.114:6800/1604678557,v1:172.21.15.114:6801/1604678557] conn(0x7fce34039560 0x7fce3403ba20 secure :-1 s=READY pgs=13 cs=0 l=1 rev1=1 crypto rx=0x7fce38006fd0 tx=0x7fce38006e40 comp rx=0 tx=0).ready entity=mgr.14120 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T10:17:35.080 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-04-23T10:17:33.846+0000 7fce3e7fc700 1 -- 172.21.15.114:0/1278741194 <== mon.0 v2:172.21.15.114:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7fce44023b50 con 0x7fce48108ff0 2024-04-23T10:17:35.080 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-04-23T10:17:34.060+0000 7fce4f87b700 1 -- 172.21.15.114:0/1278741194 --> [v2:172.21.15.114:6800/1604678557,v1:172.21.15.114:6801/1604678557] -- mgr_command(tid 0: {"prefix": "orch apply", "service_type": "mgr", "unmanaged": true, "target": ["mon-mgr", ""]}) -- 0x7fce48103bc0 con 0x7fce34039560 2024-04-23T10:17:35.080 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-04-23T10:17:34.065+0000 7fce3e7fc700 1 -- 172.21.15.114:0/1278741194 <== mgr.14120 v2:172.21.15.114:6800/1604678557 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+24 (secure 0 0 0) 0x7fce48103bc0 con 0x7fce34039560 2024-04-23T10:17:35.080 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-04-23T10:17:34.067+0000 7fce4f87b700 1 -- 172.21.15.114:0/1278741194 >> [v2:172.21.15.114:6800/1604678557,v1:172.21.15.114:6801/1604678557] conn(0x7fce34039560 msgr2=0x7fce3403ba20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T10:17:35.080 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-04-23T10:17:34.067+0000 7fce4f87b700 1 --2- 172.21.15.114:0/1278741194 >> [v2:172.21.15.114:6800/1604678557,v1:172.21.15.114:6801/1604678557] conn(0x7fce34039560 0x7fce3403ba20 secure :-1 s=READY pgs=13 cs=0 l=1 rev1=1 crypto rx=0x7fce38006fd0 tx=0x7fce38006e40 comp rx=0 tx=0).stop 2024-04-23T10:17:35.081 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-04-23T10:17:34.067+0000 7fce4f87b700 1 -- 172.21.15.114:0/1278741194 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fce48108ff0 msgr2=0x7fce4819c3f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T10:17:35.081 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-04-23T10:17:34.067+0000 7fce4f87b700 1 --2- 172.21.15.114:0/1278741194 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fce48108ff0 0x7fce4819c3f0 secure :-1 s=READY pgs=50 cs=0 l=1 rev1=1 crypto rx=0x7fce4400b7b0 tx=0x7fce4400be20 comp rx=0 tx=0).stop 2024-04-23T10:17:35.081 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-04-23T10:17:34.067+0000 7fce4f87b700 1 -- 172.21.15.114:0/1278741194 shutdown_connections 2024-04-23T10:17:35.081 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-04-23T10:17:34.067+0000 7fce4f87b700 1 --2- 172.21.15.114:0/1278741194 >> [v2:172.21.15.114:6800/1604678557,v1:172.21.15.114:6801/1604678557] conn(0x7fce34039560 0x7fce3403ba20 unknown :-1 s=CLOSED pgs=13 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:17:35.081 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-04-23T10:17:34.067+0000 7fce4f87b700 1 --2- 172.21.15.114:0/1278741194 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fce48108ff0 0x7fce4819c3f0 unknown :-1 s=CLOSED pgs=50 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:17:35.081 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-04-23T10:17:34.067+0000 7fce4f87b700 1 -- 172.21.15.114:0/1278741194 >> 172.21.15.114:0/1278741194 conn(0x7fce48102a40 msgr2=0x7fce481034b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T10:17:35.081 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-04-23T10:17:34.068+0000 7fce4f87b700 1 -- 172.21.15.114:0/1278741194 shutdown_connections 2024-04-23T10:17:35.081 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-04-23T10:17:34.068+0000 7fce4f87b700 1 -- 172.21.15.114:0/1278741194 wait complete. 2024-04-23T10:17:36.292 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-04-23T10:17:35.487+0000 7f3dfe94e700 1 Processor -- start 2024-04-23T10:17:36.292 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-04-23T10:17:35.487+0000 7f3dfe94e700 1 -- start start 2024-04-23T10:17:36.292 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-04-23T10:17:35.487+0000 7f3dfe94e700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f3df8106be0 0x7f3df8107000 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T10:17:36.292 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-04-23T10:17:35.487+0000 7f3dfe94e700 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x7f3df81075d0 con 0x7f3df8106be0 2024-04-23T10:17:36.292 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-04-23T10:17:35.487+0000 7f3df7fff700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f3df8106be0 0x7f3df8107000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T10:17:36.293 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-04-23T10:17:35.487+0000 7f3df7fff700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f3df8106be0 0x7f3df8107000 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.114:40402/0 (socket says 172.21.15.114:40402) 2024-04-23T10:17:36.293 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-04-23T10:17:35.487+0000 7f3df7fff700 1 -- 172.21.15.114:0/334066946 learned_addr learned my addr 172.21.15.114:0/334066946 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-04-23T10:17:36.293 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-04-23T10:17:35.488+0000 7f3df7fff700 1 -- 172.21.15.114:0/334066946 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f3df8107de0 con 0x7f3df8106be0 2024-04-23T10:17:36.293 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-04-23T10:17:35.488+0000 7f3df7fff700 1 --2- 172.21.15.114:0/334066946 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f3df8106be0 0x7f3df8107000 secure :-1 s=READY pgs=51 cs=0 l=1 rev1=1 crypto rx=0x7f3dec0098a0 tx=0x7f3dec009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3808a6f41ff01cf4 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T10:17:36.293 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-04-23T10:17:35.488+0000 7f3df6ffd700 1 -- 172.21.15.114:0/334066946 <== mon.0 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f3dec004160 con 0x7f3df8106be0 2024-04-23T10:17:36.293 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-04-23T10:17:35.488+0000 7f3df6ffd700 1 -- 172.21.15.114:0/334066946 <== mon.0 v2:172.21.15.114:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f3dec004300 con 0x7f3df8106be0 2024-04-23T10:17:36.294 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-04-23T10:17:35.488+0000 7f3df6ffd700 1 -- 172.21.15.114:0/334066946 <== mon.0 v2:172.21.15.114:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f3dec011a20 con 0x7f3df8106be0 2024-04-23T10:17:36.294 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-04-23T10:17:35.489+0000 7f3dfe94e700 1 -- 172.21.15.114:0/334066946 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f3df8106be0 msgr2=0x7f3df8107000 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T10:17:36.294 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-04-23T10:17:35.489+0000 7f3dfe94e700 1 --2- 172.21.15.114:0/334066946 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f3df8106be0 0x7f3df8107000 secure :-1 s=READY pgs=51 cs=0 l=1 rev1=1 crypto rx=0x7f3dec0098a0 tx=0x7f3dec009bb0 comp rx=0 tx=0).stop 2024-04-23T10:17:36.294 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-04-23T10:17:35.489+0000 7f3dfe94e700 1 -- 172.21.15.114:0/334066946 shutdown_connections 2024-04-23T10:17:36.294 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-04-23T10:17:35.489+0000 7f3dfe94e700 1 --2- 172.21.15.114:0/334066946 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f3df8106be0 0x7f3df8107000 unknown :-1 s=CLOSED pgs=51 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:17:36.294 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-04-23T10:17:35.489+0000 7f3dfe94e700 1 -- 172.21.15.114:0/334066946 >> 172.21.15.114:0/334066946 conn(0x7f3df81021a0 msgr2=0x7f3df81045c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T10:17:36.295 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-04-23T10:17:35.489+0000 7f3dfe94e700 1 -- 172.21.15.114:0/334066946 shutdown_connections 2024-04-23T10:17:36.295 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-04-23T10:17:35.489+0000 7f3dfe94e700 1 -- 172.21.15.114:0/334066946 wait complete. 2024-04-23T10:17:36.295 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-04-23T10:17:35.489+0000 7f3dfe94e700 1 Processor -- start 2024-04-23T10:17:36.295 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-04-23T10:17:35.489+0000 7f3dfe94e700 1 -- start start 2024-04-23T10:17:36.295 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-04-23T10:17:35.490+0000 7f3dfe94e700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f3df8106be0 0x7f3df8191d50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T10:17:36.295 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-04-23T10:17:35.490+0000 7f3dfe94e700 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x7f3df810b190 con 0x7f3df8106be0 2024-04-23T10:17:36.295 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-04-23T10:17:35.490+0000 7f3df7fff700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f3df8106be0 0x7f3df8191d50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T10:17:36.296 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-04-23T10:17:35.490+0000 7f3df7fff700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f3df8106be0 0x7f3df8191d50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.114:40416/0 (socket says 172.21.15.114:40416) 2024-04-23T10:17:36.296 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-04-23T10:17:35.490+0000 7f3df7fff700 1 -- 172.21.15.114:0/1313047620 learned_addr learned my addr 172.21.15.114:0/1313047620 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-04-23T10:17:36.296 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-04-23T10:17:35.490+0000 7f3df7fff700 1 -- 172.21.15.114:0/1313047620 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f3df8192290 con 0x7f3df8106be0 2024-04-23T10:17:36.296 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-04-23T10:17:35.490+0000 7f3df7fff700 1 --2- 172.21.15.114:0/1313047620 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f3df8106be0 0x7f3df8191d50 secure :-1 s=READY pgs=52 cs=0 l=1 rev1=1 crypto rx=0x7f3dec009520 tx=0x7f3dec011aa0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T10:17:36.296 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-04-23T10:17:35.490+0000 7f3df57fa700 1 -- 172.21.15.114:0/1313047620 <== mon.0 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f3dec0039e0 con 0x7f3df8106be0 2024-04-23T10:17:36.296 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-04-23T10:17:35.490+0000 7f3dfe94e700 1 -- 172.21.15.114:0/1313047620 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f3df8190480 con 0x7f3df8106be0 2024-04-23T10:17:36.297 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-04-23T10:17:35.490+0000 7f3df57fa700 1 -- 172.21.15.114:0/1313047620 <== mon.0 v2:172.21.15.114:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f3dec01b8f0 con 0x7f3df8106be0 2024-04-23T10:17:36.297 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-04-23T10:17:35.491+0000 7f3dfe94e700 1 -- 172.21.15.114:0/1313047620 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f3df81909c0 con 0x7f3df8106be0 2024-04-23T10:17:36.297 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-04-23T10:17:35.491+0000 7f3df57fa700 1 -- 172.21.15.114:0/1313047620 <== mon.0 v2:172.21.15.114:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f3dec019770 con 0x7f3df8106be0 2024-04-23T10:17:36.297 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-04-23T10:17:35.491+0000 7f3df57fa700 1 -- 172.21.15.114:0/1313047620 <== mon.0 v2:172.21.15.114:3300/0 4 ==== mgrmap(e 8) ==== 46558+0+0 (secure 0 0 0) 0x7f3dec019a70 con 0x7f3df8106be0 2024-04-23T10:17:36.297 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-04-23T10:17:35.491+0000 7f3df57fa700 1 --2- 172.21.15.114:0/1313047620 >> [v2:172.21.15.114:6800/1604678557,v1:172.21.15.114:6801/1604678557] conn(0x7f3de00395b0 0x7f3de003ba70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T10:17:36.297 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-04-23T10:17:35.492+0000 7f3df57fa700 1 -- 172.21.15.114:0/1313047620 <== mon.0 v2:172.21.15.114:3300/0 5 ==== osd_map(2..2 src has 1..2) ==== 940+0+0 (secure 0 0 0) 0x7f3dec04def0 con 0x7f3df8106be0 2024-04-23T10:17:36.298 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-04-23T10:17:35.492+0000 7f3df77fe700 1 --2- 172.21.15.114:0/1313047620 >> [v2:172.21.15.114:6800/1604678557,v1:172.21.15.114:6801/1604678557] conn(0x7f3de00395b0 0x7f3de003ba70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T10:17:36.298 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-04-23T10:17:35.492+0000 7f3dfe94e700 1 -- 172.21.15.114:0/1313047620 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f3df80640a0 con 0x7f3df8106be0 2024-04-23T10:17:36.298 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-04-23T10:17:35.492+0000 7f3df77fe700 1 --2- 172.21.15.114:0/1313047620 >> [v2:172.21.15.114:6800/1604678557,v1:172.21.15.114:6801/1604678557] conn(0x7f3de00395b0 0x7f3de003ba70 secure :-1 s=READY pgs=14 cs=0 l=1 rev1=1 crypto rx=0x7f3de8006fd0 tx=0x7f3de8006e40 comp rx=0 tx=0).ready entity=mgr.14120 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T10:17:36.298 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-04-23T10:17:35.497+0000 7f3df57fa700 1 -- 172.21.15.114:0/1313047620 <== mon.0 v2:172.21.15.114:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f3dec01f030 con 0x7f3df8106be0 2024-04-23T10:17:36.298 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-04-23T10:17:35.713+0000 7f3dfe94e700 1 -- 172.21.15.114:0/1313047620 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_command([{prefix=config set, name=mgr/cephadm/container_init}] v 0) -- 0x7f3df8195910 con 0x7f3df8106be0 2024-04-23T10:17:36.299 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-04-23T10:17:35.718+0000 7f3df57fa700 1 -- 172.21.15.114:0/1313047620 <== mon.0 v2:172.21.15.114:3300/0 7 ==== mon_command_ack([{prefix=config set, name=mgr/cephadm/container_init}]=0 v7)=0 v7) ==== 142+0+0 (secure 0 0 0) 0x7f3dec016df0 con 0x7f3df8106be0 2024-04-23T10:17:36.299 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-04-23T10:17:35.721+0000 7f3dfe94e700 1 -- 172.21.15.114:0/1313047620 >> [v2:172.21.15.114:6800/1604678557,v1:172.21.15.114:6801/1604678557] conn(0x7f3de00395b0 msgr2=0x7f3de003ba70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T10:17:36.299 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-04-23T10:17:35.721+0000 7f3dfe94e700 1 --2- 172.21.15.114:0/1313047620 >> [v2:172.21.15.114:6800/1604678557,v1:172.21.15.114:6801/1604678557] conn(0x7f3de00395b0 0x7f3de003ba70 secure :-1 s=READY pgs=14 cs=0 l=1 rev1=1 crypto rx=0x7f3de8006fd0 tx=0x7f3de8006e40 comp rx=0 tx=0).stop 2024-04-23T10:17:36.299 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-04-23T10:17:35.721+0000 7f3dfe94e700 1 -- 172.21.15.114:0/1313047620 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f3df8106be0 msgr2=0x7f3df8191d50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T10:17:36.299 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-04-23T10:17:35.721+0000 7f3dfe94e700 1 --2- 172.21.15.114:0/1313047620 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f3df8106be0 0x7f3df8191d50 secure :-1 s=READY pgs=52 cs=0 l=1 rev1=1 crypto rx=0x7f3dec009520 tx=0x7f3dec011aa0 comp rx=0 tx=0).stop 2024-04-23T10:17:36.299 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-04-23T10:17:35.721+0000 7f3dfe94e700 1 -- 172.21.15.114:0/1313047620 shutdown_connections 2024-04-23T10:17:36.300 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-04-23T10:17:35.721+0000 7f3dfe94e700 1 --2- 172.21.15.114:0/1313047620 >> [v2:172.21.15.114:6800/1604678557,v1:172.21.15.114:6801/1604678557] conn(0x7f3de00395b0 0x7f3de003ba70 unknown :-1 s=CLOSED pgs=14 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:17:36.300 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-04-23T10:17:35.721+0000 7f3dfe94e700 1 --2- 172.21.15.114:0/1313047620 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f3df8106be0 0x7f3df8191d50 unknown :-1 s=CLOSED pgs=52 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:17:36.300 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-04-23T10:17:35.721+0000 7f3dfe94e700 1 -- 172.21.15.114:0/1313047620 >> 172.21.15.114:0/1313047620 conn(0x7f3df81021a0 msgr2=0x7f3df8103350 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T10:17:36.300 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-04-23T10:17:35.721+0000 7f3dfe94e700 1 -- 172.21.15.114:0/1313047620 shutdown_connections 2024-04-23T10:17:36.300 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-04-23T10:17:35.721+0000 7f3dfe94e700 1 -- 172.21.15.114:0/1313047620 wait complete. 2024-04-23T10:17:36.645 INFO:journalctl@ceph.mon.a.smithi114.stdout:Apr 23 10:17:36 smithi114 ceph-mon[29255]: from='mgr.14120 172.21.15.114:0/1448995927' entity='mgr.x' 2024-04-23T10:17:36.645 INFO:journalctl@ceph.mon.a.smithi114.stdout:Apr 23 10:17:36 smithi114 ceph-mon[29255]: from='client.? 172.21.15.114:0/1313047620' entity='client.admin' 2024-04-23T10:17:37.725 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-04-23T10:17:36.746+0000 7feaf7bf8700 1 Processor -- start 2024-04-23T10:17:37.725 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-04-23T10:17:36.746+0000 7feaf7bf8700 1 -- start start 2024-04-23T10:17:37.725 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-04-23T10:17:36.746+0000 7feaf7bf8700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7feaf0107900 0x7feaf0107d20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T10:17:37.725 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-04-23T10:17:36.746+0000 7feaf7bf8700 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x7feaf01082f0 con 0x7feaf0107900 2024-04-23T10:17:37.725 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-04-23T10:17:36.746+0000 7feaf5994700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7feaf0107900 0x7feaf0107d20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T10:17:37.725 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-04-23T10:17:36.746+0000 7feaf5994700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7feaf0107900 0x7feaf0107d20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.114:47814/0 (socket says 172.21.15.114:47814) 2024-04-23T10:17:37.725 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-04-23T10:17:36.746+0000 7feaf5994700 1 -- 172.21.15.114:0/935403222 learned_addr learned my addr 172.21.15.114:0/935403222 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-04-23T10:17:37.725 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-04-23T10:17:36.747+0000 7feaf5994700 1 -- 172.21.15.114:0/935403222 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7feaf0108620 con 0x7feaf0107900 2024-04-23T10:17:37.725 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-04-23T10:17:36.747+0000 7feaf5994700 1 --2- 172.21.15.114:0/935403222 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7feaf0107900 0x7feaf0107d20 secure :-1 s=READY pgs=53 cs=0 l=1 rev1=1 crypto rx=0x7feae8009e80 tx=0x7feae800b330 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d70b5ad3f8c62a5a server_cookie=0 in_seq=0 out_seq=0 2024-04-23T10:17:37.725 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-04-23T10:17:36.747+0000 7feaf4992700 1 -- 172.21.15.114:0/935403222 <== mon.0 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7feae8017070 con 0x7feaf0107900 2024-04-23T10:17:37.725 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-04-23T10:17:36.747+0000 7feaf4992700 1 -- 172.21.15.114:0/935403222 <== mon.0 v2:172.21.15.114:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7feae800b8c0 con 0x7feaf0107900 2024-04-23T10:17:37.725 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-04-23T10:17:36.747+0000 7feaf4992700 1 -- 172.21.15.114:0/935403222 <== mon.0 v2:172.21.15.114:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7feae8004490 con 0x7feaf0107900 2024-04-23T10:17:37.725 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-04-23T10:17:36.748+0000 7feaf7bf8700 1 -- 172.21.15.114:0/935403222 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7feaf0107900 msgr2=0x7feaf0107d20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T10:17:37.726 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-04-23T10:17:36.748+0000 7feaf7bf8700 1 --2- 172.21.15.114:0/935403222 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7feaf0107900 0x7feaf0107d20 secure :-1 s=READY pgs=53 cs=0 l=1 rev1=1 crypto rx=0x7feae8009e80 tx=0x7feae800b330 comp rx=0 tx=0).stop 2024-04-23T10:17:37.726 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-04-23T10:17:36.748+0000 7feaf7bf8700 1 -- 172.21.15.114:0/935403222 shutdown_connections 2024-04-23T10:17:37.726 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-04-23T10:17:36.748+0000 7feaf7bf8700 1 --2- 172.21.15.114:0/935403222 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7feaf0107900 0x7feaf0107d20 unknown :-1 s=CLOSED pgs=53 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:17:37.726 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-04-23T10:17:36.748+0000 7feaf7bf8700 1 -- 172.21.15.114:0/935403222 >> 172.21.15.114:0/935403222 conn(0x7feaf0102140 msgr2=0x7feaf0104560 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T10:17:37.726 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-04-23T10:17:36.748+0000 7feaf7bf8700 1 -- 172.21.15.114:0/935403222 shutdown_connections 2024-04-23T10:17:37.726 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-04-23T10:17:36.748+0000 7feaf7bf8700 1 -- 172.21.15.114:0/935403222 wait complete. 2024-04-23T10:17:37.726 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-04-23T10:17:36.748+0000 7feaf7bf8700 1 Processor -- start 2024-04-23T10:17:37.726 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-04-23T10:17:36.748+0000 7feaf7bf8700 1 -- start start 2024-04-23T10:17:37.726 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-04-23T10:17:36.748+0000 7feaf7bf8700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7feaf0107900 0x7feaf019c420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T10:17:37.726 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-04-23T10:17:36.748+0000 7feaf7bf8700 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x7feaf010b9d0 con 0x7feaf0107900 2024-04-23T10:17:37.726 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-04-23T10:17:36.749+0000 7feaf5994700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7feaf0107900 0x7feaf019c420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T10:17:37.726 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-04-23T10:17:36.749+0000 7feaf5994700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7feaf0107900 0x7feaf019c420 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.114:47830/0 (socket says 172.21.15.114:47830) 2024-04-23T10:17:37.726 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-04-23T10:17:36.749+0000 7feaf5994700 1 -- 172.21.15.114:0/649648310 learned_addr learned my addr 172.21.15.114:0/649648310 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-04-23T10:17:37.727 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-04-23T10:17:36.749+0000 7feaf5994700 1 -- 172.21.15.114:0/649648310 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7feaf019c960 con 0x7feaf0107900 2024-04-23T10:17:37.727 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-04-23T10:17:36.749+0000 7feaf5994700 1 --2- 172.21.15.114:0/649648310 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7feaf0107900 0x7feaf019c420 secure :-1 s=READY pgs=54 cs=0 l=1 rev1=1 crypto rx=0x7feae8004220 tx=0x7feae80093e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T10:17:37.727 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-04-23T10:17:36.749+0000 7feae6ffd700 1 -- 172.21.15.114:0/649648310 <== mon.0 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7feae8020020 con 0x7feaf0107900 2024-04-23T10:17:37.727 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-04-23T10:17:36.749+0000 7feae6ffd700 1 -- 172.21.15.114:0/649648310 <== mon.0 v2:172.21.15.114:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7feae8003b50 con 0x7feaf0107900 2024-04-23T10:17:37.727 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-04-23T10:17:36.749+0000 7feaf7bf8700 1 -- 172.21.15.114:0/649648310 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7feaf019cbf0 con 0x7feaf0107900 2024-04-23T10:17:37.727 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-04-23T10:17:36.749+0000 7feae6ffd700 1 -- 172.21.15.114:0/649648310 <== mon.0 v2:172.21.15.114:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7feae8017040 con 0x7feaf0107900 2024-04-23T10:17:37.727 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-04-23T10:17:36.749+0000 7feaf7bf8700 1 -- 172.21.15.114:0/649648310 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7feaf019f8e0 con 0x7feaf0107900 2024-04-23T10:17:37.727 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-04-23T10:17:36.750+0000 7feae6ffd700 1 -- 172.21.15.114:0/649648310 <== mon.0 v2:172.21.15.114:3300/0 4 ==== mgrmap(e 8) ==== 46558+0+0 (secure 0 0 0) 0x7feae8023020 con 0x7feaf0107900 2024-04-23T10:17:37.727 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-04-23T10:17:36.750+0000 7feaf7bf8700 1 -- 172.21.15.114:0/649648310 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7feaf00640a0 con 0x7feaf0107900 2024-04-23T10:17:37.727 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-04-23T10:17:36.750+0000 7feae6ffd700 1 --2- 172.21.15.114:0/649648310 >> [v2:172.21.15.114:6800/1604678557,v1:172.21.15.114:6801/1604678557] conn(0x7feadc0395b0 0x7feadc03ba70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T10:17:37.727 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-04-23T10:17:36.750+0000 7feae6ffd700 1 -- 172.21.15.114:0/649648310 <== mon.0 v2:172.21.15.114:3300/0 5 ==== osd_map(2..2 src has 1..2) ==== 940+0+0 (secure 0 0 0) 0x7feae800f050 con 0x7feaf0107900 2024-04-23T10:17:37.727 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-04-23T10:17:36.750+0000 7feaf5193700 1 --2- 172.21.15.114:0/649648310 >> [v2:172.21.15.114:6800/1604678557,v1:172.21.15.114:6801/1604678557] conn(0x7feadc0395b0 0x7feadc03ba70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T10:17:37.727 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-04-23T10:17:36.751+0000 7feaf5193700 1 --2- 172.21.15.114:0/649648310 >> [v2:172.21.15.114:6800/1604678557,v1:172.21.15.114:6801/1604678557] conn(0x7feadc0395b0 0x7feadc03ba70 secure :-1 s=READY pgs=15 cs=0 l=1 rev1=1 crypto rx=0x7feae0006fd0 tx=0x7feae0006e40 comp rx=0 tx=0).ready entity=mgr.14120 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T10:17:37.727 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-04-23T10:17:36.755+0000 7feae6ffd700 1 -- 172.21.15.114:0/649648310 <== mon.0 v2:172.21.15.114:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7feae802a020 con 0x7feaf0107900 2024-04-23T10:17:37.728 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-04-23T10:17:36.965+0000 7feaf7bf8700 1 -- 172.21.15.114:0/649648310 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_command([{prefix=config set, name=mgr/dashboard/ssl_server_port}] v 0) -- 0x7feaf019ff20 con 0x7feaf0107900 2024-04-23T10:17:37.728 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-04-23T10:17:36.967+0000 7feae6ffd700 1 -- 172.21.15.114:0/649648310 <== mon.0 v2:172.21.15.114:3300/0 7 ==== mon_command_ack([{prefix=config set, name=mgr/dashboard/ssl_server_port}]=0 v8)=0 v8) ==== 130+0+0 (secure 0 0 0) 0x7feae801b370 con 0x7feaf0107900 2024-04-23T10:17:37.728 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-04-23T10:17:36.969+0000 7feaf7bf8700 1 -- 172.21.15.114:0/649648310 >> [v2:172.21.15.114:6800/1604678557,v1:172.21.15.114:6801/1604678557] conn(0x7feadc0395b0 msgr2=0x7feadc03ba70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T10:17:37.728 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-04-23T10:17:36.969+0000 7feaf7bf8700 1 --2- 172.21.15.114:0/649648310 >> [v2:172.21.15.114:6800/1604678557,v1:172.21.15.114:6801/1604678557] conn(0x7feadc0395b0 0x7feadc03ba70 secure :-1 s=READY pgs=15 cs=0 l=1 rev1=1 crypto rx=0x7feae0006fd0 tx=0x7feae0006e40 comp rx=0 tx=0).stop 2024-04-23T10:17:37.728 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-04-23T10:17:36.969+0000 7feaf7bf8700 1 -- 172.21.15.114:0/649648310 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7feaf0107900 msgr2=0x7feaf019c420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T10:17:37.728 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-04-23T10:17:36.969+0000 7feaf7bf8700 1 --2- 172.21.15.114:0/649648310 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7feaf0107900 0x7feaf019c420 secure :-1 s=READY pgs=54 cs=0 l=1 rev1=1 crypto rx=0x7feae8004220 tx=0x7feae80093e0 comp rx=0 tx=0).stop 2024-04-23T10:17:37.728 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-04-23T10:17:36.969+0000 7feaf7bf8700 1 -- 172.21.15.114:0/649648310 shutdown_connections 2024-04-23T10:17:37.728 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-04-23T10:17:36.969+0000 7feaf7bf8700 1 --2- 172.21.15.114:0/649648310 >> [v2:172.21.15.114:6800/1604678557,v1:172.21.15.114:6801/1604678557] conn(0x7feadc0395b0 0x7feadc03ba70 unknown :-1 s=CLOSED pgs=15 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:17:37.728 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-04-23T10:17:36.969+0000 7feaf7bf8700 1 --2- 172.21.15.114:0/649648310 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7feaf0107900 0x7feaf019c420 unknown :-1 s=CLOSED pgs=54 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:17:37.728 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-04-23T10:17:36.969+0000 7feaf7bf8700 1 -- 172.21.15.114:0/649648310 >> 172.21.15.114:0/649648310 conn(0x7feaf0102140 msgr2=0x7feaf0102bb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T10:17:37.728 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-04-23T10:17:36.969+0000 7feaf7bf8700 1 -- 172.21.15.114:0/649648310 shutdown_connections 2024-04-23T10:17:37.728 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-04-23T10:17:36.969+0000 7feaf7bf8700 1 -- 172.21.15.114:0/649648310 wait complete. 2024-04-23T10:17:37.728 INFO:teuthology.orchestra.run.smithi114.stdout:Enabling the dashboard module... 2024-04-23T10:17:38.137 INFO:journalctl@ceph.mon.a.smithi114.stdout:Apr 23 10:17:37 smithi114 ceph-mon[29255]: from='client.? 172.21.15.114:0/649648310' entity='client.admin' 2024-04-23T10:17:39.238 INFO:journalctl@ceph.mon.a.smithi114.stdout:Apr 23 10:17:38 smithi114 ceph-mon[29255]: from='client.? 172.21.15.114:0/3278886800' entity='client.admin' cmd=[{"prefix": "mgr module enable", "module": "dashboard"}]: dispatch 2024-04-23T10:17:39.238 INFO:journalctl@ceph.mon.a.smithi114.stdout:Apr 23 10:17:38 smithi114 ceph-mon[29255]: from='mgr.14120 172.21.15.114:0/1448995927' entity='mgr.x' 2024-04-23T10:17:39.238 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Apr 23 10:17:39 smithi114 ceph-aa240572-0159-11ef-bc93-c7b262605968-mgr-x[29490]: ignoring --setuser ceph since I am not root 2024-04-23T10:17:39.238 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Apr 23 10:17:39 smithi114 ceph-aa240572-0159-11ef-bc93-c7b262605968-mgr-x[29490]: ignoring --setgroup ceph since I am not root 2024-04-23T10:17:39.238 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Apr 23 10:17:39 smithi114 ceph-aa240572-0159-11ef-bc93-c7b262605968-mgr-x[29490]: 2024-04-23T10:17:39.152+0000 7f9be8535200 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2024-04-23T10:17:39.504 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Apr 23 10:17:39 smithi114 ceph-aa240572-0159-11ef-bc93-c7b262605968-mgr-x[29490]: 2024-04-23T10:17:39.350+0000 7f9be8535200 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2024-04-23T10:17:39.504 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Apr 23 10:17:39 smithi114 ceph-aa240572-0159-11ef-bc93-c7b262605968-mgr-x[29490]: 2024-04-23T10:17:39.454+0000 7f9be8535200 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2024-04-23T10:17:39.642 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-04-23T10:17:38.240+0000 7fbbaf39a700 1 Processor -- start 2024-04-23T10:17:39.642 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-04-23T10:17:38.240+0000 7fbbaf39a700 1 -- start start 2024-04-23T10:17:39.643 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-04-23T10:17:38.240+0000 7fbbaf39a700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fbba8109010 0x7fbba810b400 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T10:17:39.643 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-04-23T10:17:38.240+0000 7fbbaf39a700 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x7fbba807be60 con 0x7fbba8109010 2024-04-23T10:17:39.643 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-04-23T10:17:38.241+0000 7fbbad136700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fbba8109010 0x7fbba810b400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T10:17:39.643 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-04-23T10:17:38.241+0000 7fbbad136700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fbba8109010 0x7fbba810b400 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.114:47838/0 (socket says 172.21.15.114:47838) 2024-04-23T10:17:39.643 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-04-23T10:17:38.241+0000 7fbbad136700 1 -- 172.21.15.114:0/2398121311 learned_addr learned my addr 172.21.15.114:0/2398121311 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-04-23T10:17:39.643 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-04-23T10:17:38.241+0000 7fbbad136700 1 -- 172.21.15.114:0/2398121311 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fbba810b940 con 0x7fbba8109010 2024-04-23T10:17:39.643 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-04-23T10:17:38.241+0000 7fbbad136700 1 --2- 172.21.15.114:0/2398121311 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fbba8109010 0x7fbba810b400 secure :-1 s=READY pgs=55 cs=0 l=1 rev1=1 crypto rx=0x7fbb98009e80 tx=0x7fbb9800b330 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ab1b88488cb706c server_cookie=0 in_seq=0 out_seq=0 2024-04-23T10:17:39.643 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-04-23T10:17:38.241+0000 7fbba7fff700 1 -- 172.21.15.114:0/2398121311 <== mon.0 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fbb98017070 con 0x7fbba8109010 2024-04-23T10:17:39.643 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-04-23T10:17:38.241+0000 7fbba7fff700 1 -- 172.21.15.114:0/2398121311 <== mon.0 v2:172.21.15.114:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fbb9800b8c0 con 0x7fbba8109010 2024-04-23T10:17:39.643 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-04-23T10:17:38.241+0000 7fbba7fff700 1 -- 172.21.15.114:0/2398121311 <== mon.0 v2:172.21.15.114:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fbb98004490 con 0x7fbba8109010 2024-04-23T10:17:39.643 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-04-23T10:17:38.242+0000 7fbbaf39a700 1 -- 172.21.15.114:0/2398121311 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fbba8109010 msgr2=0x7fbba810b400 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T10:17:39.643 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-04-23T10:17:38.242+0000 7fbbaf39a700 1 --2- 172.21.15.114:0/2398121311 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fbba8109010 0x7fbba810b400 secure :-1 s=READY pgs=55 cs=0 l=1 rev1=1 crypto rx=0x7fbb98009e80 tx=0x7fbb9800b330 comp rx=0 tx=0).stop 2024-04-23T10:17:39.643 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-04-23T10:17:38.242+0000 7fbbaf39a700 1 -- 172.21.15.114:0/2398121311 shutdown_connections 2024-04-23T10:17:39.643 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-04-23T10:17:38.242+0000 7fbbaf39a700 1 --2- 172.21.15.114:0/2398121311 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fbba8109010 0x7fbba810b400 unknown :-1 s=CLOSED pgs=55 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:17:39.643 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-04-23T10:17:38.242+0000 7fbbaf39a700 1 -- 172.21.15.114:0/2398121311 >> 172.21.15.114:0/2398121311 conn(0x7fbba81029e0 msgr2=0x7fbba8104e20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T10:17:39.643 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-04-23T10:17:38.242+0000 7fbbaf39a700 1 -- 172.21.15.114:0/2398121311 shutdown_connections 2024-04-23T10:17:39.644 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-04-23T10:17:38.242+0000 7fbbaf39a700 1 -- 172.21.15.114:0/2398121311 wait complete. 2024-04-23T10:17:39.644 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-04-23T10:17:38.242+0000 7fbbaf39a700 1 Processor -- start 2024-04-23T10:17:39.644 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-04-23T10:17:38.243+0000 7fbbaf39a700 1 -- start start 2024-04-23T10:17:39.644 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-04-23T10:17:38.243+0000 7fbbaf39a700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fbba8109010 0x7fbba819c4d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T10:17:39.644 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-04-23T10:17:38.243+0000 7fbbaf39a700 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x7fbba810c8f0 con 0x7fbba8109010 2024-04-23T10:17:39.644 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-04-23T10:17:38.243+0000 7fbbad136700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fbba8109010 0x7fbba819c4d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T10:17:39.644 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-04-23T10:17:38.243+0000 7fbbad136700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fbba8109010 0x7fbba819c4d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.114:47846/0 (socket says 172.21.15.114:47846) 2024-04-23T10:17:39.644 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-04-23T10:17:38.243+0000 7fbbad136700 1 -- 172.21.15.114:0/3278886800 learned_addr learned my addr 172.21.15.114:0/3278886800 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-04-23T10:17:39.644 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-04-23T10:17:38.243+0000 7fbbad136700 1 -- 172.21.15.114:0/3278886800 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fbba819ca10 con 0x7fbba8109010 2024-04-23T10:17:39.644 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-04-23T10:17:38.243+0000 7fbbad136700 1 --2- 172.21.15.114:0/3278886800 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fbba8109010 0x7fbba819c4d0 secure :-1 s=READY pgs=56 cs=0 l=1 rev1=1 crypto rx=0x7fbb9800b7b0 tx=0x7fbb9800bd50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T10:17:39.644 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-04-23T10:17:38.243+0000 7fbba67fc700 1 -- 172.21.15.114:0/3278886800 <== mon.0 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fbb98020020 con 0x7fbba8109010 2024-04-23T10:17:39.644 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-04-23T10:17:38.243+0000 7fbba67fc700 1 -- 172.21.15.114:0/3278886800 <== mon.0 v2:172.21.15.114:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fbb98003c60 con 0x7fbba8109010 2024-04-23T10:17:39.644 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-04-23T10:17:38.243+0000 7fbba67fc700 1 -- 172.21.15.114:0/3278886800 <== mon.0 v2:172.21.15.114:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fbb98017040 con 0x7fbba8109010 2024-04-23T10:17:39.644 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-04-23T10:17:38.243+0000 7fbbaf39a700 1 -- 172.21.15.114:0/3278886800 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fbba819cca0 con 0x7fbba8109010 2024-04-23T10:17:39.644 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-04-23T10:17:38.243+0000 7fbbaf39a700 1 -- 172.21.15.114:0/3278886800 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fbba819f990 con 0x7fbba8109010 2024-04-23T10:17:39.645 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-04-23T10:17:38.244+0000 7fbba67fc700 1 -- 172.21.15.114:0/3278886800 <== mon.0 v2:172.21.15.114:3300/0 4 ==== mgrmap(e 8) ==== 46558+0+0 (secure 0 0 0) 0x7fbb98023020 con 0x7fbba8109010 2024-04-23T10:17:39.645 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-04-23T10:17:38.244+0000 7fbbaf39a700 1 -- 172.21.15.114:0/3278886800 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fbba80640a0 con 0x7fbba8109010 2024-04-23T10:17:39.645 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-04-23T10:17:38.244+0000 7fbba67fc700 1 --2- 172.21.15.114:0/3278886800 >> [v2:172.21.15.114:6800/1604678557,v1:172.21.15.114:6801/1604678557] conn(0x7fbb90039600 0x7fbb9003bac0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T10:17:39.645 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-04-23T10:17:38.245+0000 7fbba67fc700 1 -- 172.21.15.114:0/3278886800 <== mon.0 v2:172.21.15.114:3300/0 5 ==== osd_map(2..2 src has 1..2) ==== 940+0+0 (secure 0 0 0) 0x7fbb9800f050 con 0x7fbba8109010 2024-04-23T10:17:39.645 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-04-23T10:17:38.245+0000 7fbbac935700 1 --2- 172.21.15.114:0/3278886800 >> [v2:172.21.15.114:6800/1604678557,v1:172.21.15.114:6801/1604678557] conn(0x7fbb90039600 0x7fbb9003bac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T10:17:39.645 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-04-23T10:17:38.245+0000 7fbbac935700 1 --2- 172.21.15.114:0/3278886800 >> [v2:172.21.15.114:6800/1604678557,v1:172.21.15.114:6801/1604678557] conn(0x7fbb90039600 0x7fbb9003bac0 secure :-1 s=READY pgs=16 cs=0 l=1 rev1=1 crypto rx=0x7fbb9c006fd0 tx=0x7fbb9c006e40 comp rx=0 tx=0).ready entity=mgr.14120 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T10:17:39.645 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-04-23T10:17:38.250+0000 7fbba67fc700 1 -- 172.21.15.114:0/3278886800 <== mon.0 v2:172.21.15.114:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7fbb9802a020 con 0x7fbba8109010 2024-04-23T10:17:39.645 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-04-23T10:17:38.507+0000 7fbbaf39a700 1 -- 172.21.15.114:0/3278886800 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_command({"prefix": "mgr module enable", "module": "dashboard"} v 0) -- 0x7fbba819ff60 con 0x7fbba8109010 2024-04-23T10:17:39.645 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-04-23T10:17:38.971+0000 7fbba67fc700 1 -- 172.21.15.114:0/3278886800 <== mon.0 v2:172.21.15.114:3300/0 7 ==== mon_command_ack([{"prefix": "mgr module enable", "module": "dashboard"}]=0 v9) ==== 88+0+0 (secure 0 0 0) 0x7fbb9802c300 con 0x7fbba8109010 2024-04-23T10:17:39.645 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-04-23T10:17:38.973+0000 7fbba67fc700 1 -- 172.21.15.114:0/3278886800 <== mon.0 v2:172.21.15.114:3300/0 8 ==== mgrmap(e 9) ==== 46571+0+0 (secure 0 0 0) 0x7fbb98012430 con 0x7fbba8109010 2024-04-23T10:17:39.645 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-04-23T10:17:38.974+0000 7fbbaf39a700 1 -- 172.21.15.114:0/3278886800 >> [v2:172.21.15.114:6800/1604678557,v1:172.21.15.114:6801/1604678557] conn(0x7fbb90039600 msgr2=0x7fbb9003bac0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T10:17:39.645 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-04-23T10:17:38.974+0000 7fbbaf39a700 1 --2- 172.21.15.114:0/3278886800 >> [v2:172.21.15.114:6800/1604678557,v1:172.21.15.114:6801/1604678557] conn(0x7fbb90039600 0x7fbb9003bac0 secure :-1 s=READY pgs=16 cs=0 l=1 rev1=1 crypto rx=0x7fbb9c006fd0 tx=0x7fbb9c006e40 comp rx=0 tx=0).stop 2024-04-23T10:17:39.645 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-04-23T10:17:38.974+0000 7fbbaf39a700 1 -- 172.21.15.114:0/3278886800 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fbba8109010 msgr2=0x7fbba819c4d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T10:17:39.645 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-04-23T10:17:38.974+0000 7fbbaf39a700 1 --2- 172.21.15.114:0/3278886800 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fbba8109010 0x7fbba819c4d0 secure :-1 s=READY pgs=56 cs=0 l=1 rev1=1 crypto rx=0x7fbb9800b7b0 tx=0x7fbb9800bd50 comp rx=0 tx=0).stop 2024-04-23T10:17:39.645 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-04-23T10:17:38.974+0000 7fbbaf39a700 1 -- 172.21.15.114:0/3278886800 shutdown_connections 2024-04-23T10:17:39.645 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-04-23T10:17:38.974+0000 7fbbaf39a700 1 --2- 172.21.15.114:0/3278886800 >> [v2:172.21.15.114:6800/1604678557,v1:172.21.15.114:6801/1604678557] conn(0x7fbb90039600 0x7fbb9003bac0 unknown :-1 s=CLOSED pgs=16 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:17:39.646 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-04-23T10:17:38.974+0000 7fbbaf39a700 1 --2- 172.21.15.114:0/3278886800 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fbba8109010 0x7fbba819c4d0 unknown :-1 s=CLOSED pgs=56 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:17:39.646 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-04-23T10:17:38.974+0000 7fbbaf39a700 1 -- 172.21.15.114:0/3278886800 >> 172.21.15.114:0/3278886800 conn(0x7fbba81029e0 msgr2=0x7fbba8103520 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T10:17:39.646 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-04-23T10:17:38.975+0000 7fbbaf39a700 1 -- 172.21.15.114:0/3278886800 shutdown_connections 2024-04-23T10:17:39.646 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-04-23T10:17:38.975+0000 7fbbaf39a700 1 -- 172.21.15.114:0/3278886800 wait complete. 2024-04-23T10:17:39.890 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Apr 23 10:17:39 smithi114 ceph-aa240572-0159-11ef-bc93-c7b262605968-mgr-x[29490]: 2024-04-23T10:17:39.767+0000 7f9be8535200 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2024-04-23T10:17:39.890 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Apr 23 10:17:39 smithi114 ceph-aa240572-0159-11ef-bc93-c7b262605968-mgr-x[29490]: 2024-04-23T10:17:39.853+0000 7f9be8535200 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2024-04-23T10:17:40.254 INFO:journalctl@ceph.mon.a.smithi114.stdout:Apr 23 10:17:39 smithi114 ceph-mon[29255]: from='client.? 172.21.15.114:0/3278886800' entity='client.admin' cmd='[{"prefix": "mgr module enable", "module": "dashboard"}]': finished 2024-04-23T10:17:40.254 INFO:journalctl@ceph.mon.a.smithi114.stdout:Apr 23 10:17:39 smithi114 ceph-mon[29255]: mgrmap e9: x(active, since 19s) 2024-04-23T10:17:41.075 INFO:journalctl@ceph.mon.a.smithi114.stdout:Apr 23 10:17:40 smithi114 ceph-mon[29255]: from='client.? 172.21.15.114:0/4240247777' entity='client.admin' cmd=[{"prefix": "mgr stat"}]: dispatch 2024-04-23T10:17:41.078 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stdout { 2024-04-23T10:17:41.084 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stdout "epoch": 9, 2024-04-23T10:17:41.084 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stdout "available": true, 2024-04-23T10:17:41.084 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stdout "active_name": "x", 2024-04-23T10:17:41.084 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stdout "num_standby": 0 2024-04-23T10:17:41.084 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stdout } 2024-04-23T10:17:41.084 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-04-23T10:17:40.115+0000 7f616b750700 1 Processor -- start 2024-04-23T10:17:41.084 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-04-23T10:17:40.116+0000 7f616b750700 1 -- start start 2024-04-23T10:17:41.084 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-04-23T10:17:40.116+0000 7f616b750700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f6164109070 0x7f616410b460 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T10:17:41.084 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-04-23T10:17:40.116+0000 7f616b750700 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x7f6164063ca0 con 0x7f6164109070 2024-04-23T10:17:41.085 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-04-23T10:17:40.116+0000 7f61694ec700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f6164109070 0x7f616410b460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T10:17:41.085 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-04-23T10:17:40.116+0000 7f61694ec700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f6164109070 0x7f616410b460 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.114:47874/0 (socket says 172.21.15.114:47874) 2024-04-23T10:17:41.085 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-04-23T10:17:40.116+0000 7f61694ec700 1 -- 172.21.15.114:0/2330456234 learned_addr learned my addr 172.21.15.114:0/2330456234 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-04-23T10:17:41.085 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-04-23T10:17:40.116+0000 7f61694ec700 1 -- 172.21.15.114:0/2330456234 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f616410b9a0 con 0x7f6164109070 2024-04-23T10:17:41.085 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-04-23T10:17:40.116+0000 7f61694ec700 1 --2- 172.21.15.114:0/2330456234 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f6164109070 0x7f616410b460 secure :-1 s=READY pgs=59 cs=0 l=1 rev1=1 crypto rx=0x7f61600098a0 tx=0x7f6160009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7dcf8c1812b0d1e1 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T10:17:41.085 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-04-23T10:17:40.117+0000 7f615bfff700 1 -- 172.21.15.114:0/2330456234 <== mon.0 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f6160004160 con 0x7f6164109070 2024-04-23T10:17:41.085 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-04-23T10:17:40.117+0000 7f615bfff700 1 -- 172.21.15.114:0/2330456234 <== mon.0 v2:172.21.15.114:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f6160004300 con 0x7f6164109070 2024-04-23T10:17:41.085 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-04-23T10:17:40.117+0000 7f615bfff700 1 -- 172.21.15.114:0/2330456234 <== mon.0 v2:172.21.15.114:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f6160011a20 con 0x7f6164109070 2024-04-23T10:17:41.085 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-04-23T10:17:40.117+0000 7f616b750700 1 -- 172.21.15.114:0/2330456234 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f6164109070 msgr2=0x7f616410b460 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T10:17:41.085 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-04-23T10:17:40.117+0000 7f616b750700 1 --2- 172.21.15.114:0/2330456234 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f6164109070 0x7f616410b460 secure :-1 s=READY pgs=59 cs=0 l=1 rev1=1 crypto rx=0x7f61600098a0 tx=0x7f6160009bb0 comp rx=0 tx=0).stop 2024-04-23T10:17:41.085 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-04-23T10:17:40.117+0000 7f616b750700 1 -- 172.21.15.114:0/2330456234 shutdown_connections 2024-04-23T10:17:41.085 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-04-23T10:17:40.117+0000 7f616b750700 1 --2- 172.21.15.114:0/2330456234 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f6164109070 0x7f616410b460 unknown :-1 s=CLOSED pgs=59 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:17:41.085 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-04-23T10:17:40.117+0000 7f616b750700 1 -- 172.21.15.114:0/2330456234 >> 172.21.15.114:0/2330456234 conn(0x7f6164102a40 msgr2=0x7f6164104e60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T10:17:41.085 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-04-23T10:17:40.117+0000 7f616b750700 1 -- 172.21.15.114:0/2330456234 shutdown_connections 2024-04-23T10:17:41.086 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-04-23T10:17:40.117+0000 7f616b750700 1 -- 172.21.15.114:0/2330456234 wait complete. 2024-04-23T10:17:41.086 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-04-23T10:17:40.118+0000 7f616b750700 1 Processor -- start 2024-04-23T10:17:41.086 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-04-23T10:17:40.118+0000 7f616b750700 1 -- start start 2024-04-23T10:17:41.086 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-04-23T10:17:40.118+0000 7f616b750700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f6164109070 0x7f616419c3e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T10:17:41.086 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-04-23T10:17:40.118+0000 7f616b750700 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x7f616410c8f0 con 0x7f6164109070 2024-04-23T10:17:41.086 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-04-23T10:17:40.118+0000 7f61694ec700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f6164109070 0x7f616419c3e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T10:17:41.086 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-04-23T10:17:40.118+0000 7f61694ec700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f6164109070 0x7f616419c3e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.114:47876/0 (socket says 172.21.15.114:47876) 2024-04-23T10:17:41.086 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-04-23T10:17:40.118+0000 7f61694ec700 1 -- 172.21.15.114:0/4240247777 learned_addr learned my addr 172.21.15.114:0/4240247777 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-04-23T10:17:41.086 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-04-23T10:17:40.118+0000 7f61694ec700 1 -- 172.21.15.114:0/4240247777 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f616419c920 con 0x7f6164109070 2024-04-23T10:17:41.086 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-04-23T10:17:40.119+0000 7f61694ec700 1 --2- 172.21.15.114:0/4240247777 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f6164109070 0x7f616419c3e0 secure :-1 s=READY pgs=60 cs=0 l=1 rev1=1 crypto rx=0x7f61600116e0 tx=0x7f6160011aa0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T10:17:41.086 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-04-23T10:17:40.119+0000 7f615a7fc700 1 -- 172.21.15.114:0/4240247777 <== mon.0 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f6160003a60 con 0x7f6164109070 2024-04-23T10:17:41.086 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-04-23T10:17:40.119+0000 7f615a7fc700 1 -- 172.21.15.114:0/4240247777 <== mon.0 v2:172.21.15.114:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f616001b8f0 con 0x7f6164109070 2024-04-23T10:17:41.086 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-04-23T10:17:40.119+0000 7f616b750700 1 -- 172.21.15.114:0/4240247777 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f616419cbb0 con 0x7f6164109070 2024-04-23T10:17:41.086 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-04-23T10:17:40.119+0000 7f615a7fc700 1 -- 172.21.15.114:0/4240247777 <== mon.0 v2:172.21.15.114:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f6160019860 con 0x7f6164109070 2024-04-23T10:17:41.087 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-04-23T10:17:40.119+0000 7f616b750700 1 -- 172.21.15.114:0/4240247777 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f616419f8a0 con 0x7f6164109070 2024-04-23T10:17:41.087 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-04-23T10:17:40.120+0000 7f615a7fc700 1 -- 172.21.15.114:0/4240247777 <== mon.0 v2:172.21.15.114:3300/0 4 ==== mgrmap(e 9) ==== 46571+0+0 (secure 0 0 0) 0x7f6160019b60 con 0x7f6164109070 2024-04-23T10:17:41.087 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-04-23T10:17:40.120+0000 7f615a7fc700 1 --2- 172.21.15.114:0/4240247777 >> [v2:172.21.15.114:6800/1604678557,v1:172.21.15.114:6801/1604678557] conn(0x7f61500395b0 0x7f615003ba70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T10:17:41.087 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-04-23T10:17:40.120+0000 7f615a7fc700 1 -- 172.21.15.114:0/4240247777 <== mon.0 v2:172.21.15.114:3300/0 5 ==== osd_map(2..2 src has 1..2) ==== 940+0+0 (secure 0 0 0) 0x7f616004dd30 con 0x7f6164109070 2024-04-23T10:17:41.087 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-04-23T10:17:40.120+0000 7f6168ceb700 1 -- 172.21.15.114:0/4240247777 >> [v2:172.21.15.114:6800/1604678557,v1:172.21.15.114:6801/1604678557] conn(0x7f61500395b0 msgr2=0x7f615003ba70 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.114:6800/1604678557 2024-04-23T10:17:41.087 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-04-23T10:17:40.120+0000 7f6168ceb700 1 --2- 172.21.15.114:0/4240247777 >> [v2:172.21.15.114:6800/1604678557,v1:172.21.15.114:6801/1604678557] conn(0x7f61500395b0 0x7f615003ba70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-04-23T10:17:41.087 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-04-23T10:17:40.120+0000 7f616b750700 1 -- 172.21.15.114:0/4240247777 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f61640640a0 con 0x7f6164109070 2024-04-23T10:17:41.087 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-04-23T10:17:40.125+0000 7f615a7fc700 1 -- 172.21.15.114:0/4240247777 <== mon.0 v2:172.21.15.114:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f616001f030 con 0x7f6164109070 2024-04-23T10:17:41.087 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-04-23T10:17:40.321+0000 7f6168ceb700 1 -- 172.21.15.114:0/4240247777 >> [v2:172.21.15.114:6800/1604678557,v1:172.21.15.114:6801/1604678557] conn(0x7f61500395b0 msgr2=0x7f615003ba70 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.114:6800/1604678557 2024-04-23T10:17:41.087 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-04-23T10:17:40.321+0000 7f6168ceb700 1 --2- 172.21.15.114:0/4240247777 >> [v2:172.21.15.114:6800/1604678557,v1:172.21.15.114:6801/1604678557] conn(0x7f61500395b0 0x7f615003ba70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-04-23T10:17:41.087 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-04-23T10:17:40.452+0000 7f616b750700 1 -- 172.21.15.114:0/4240247777 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_command({"prefix": "mgr stat"} v 0) -- 0x7f616419fa70 con 0x7f6164109070 2024-04-23T10:17:41.087 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-04-23T10:17:40.453+0000 7f615a7fc700 1 -- 172.21.15.114:0/4240247777 <== mon.0 v2:172.21.15.114:3300/0 7 ==== mon_command_ack([{"prefix": "mgr stat"}]=0 v9) ==== 56+0+88 (secure 0 0 0) 0x7f6160029260 con 0x7f6164109070 2024-04-23T10:17:41.088 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-04-23T10:17:40.455+0000 7f616b750700 1 -- 172.21.15.114:0/4240247777 >> [v2:172.21.15.114:6800/1604678557,v1:172.21.15.114:6801/1604678557] conn(0x7f61500395b0 msgr2=0x7f615003ba70 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-04-23T10:17:41.088 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-04-23T10:17:40.455+0000 7f616b750700 1 --2- 172.21.15.114:0/4240247777 >> [v2:172.21.15.114:6800/1604678557,v1:172.21.15.114:6801/1604678557] conn(0x7f61500395b0 0x7f615003ba70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:17:41.088 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-04-23T10:17:40.455+0000 7f616b750700 1 -- 172.21.15.114:0/4240247777 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f6164109070 msgr2=0x7f616419c3e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T10:17:41.088 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-04-23T10:17:40.455+0000 7f616b750700 1 --2- 172.21.15.114:0/4240247777 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f6164109070 0x7f616419c3e0 secure :-1 s=READY pgs=60 cs=0 l=1 rev1=1 crypto rx=0x7f61600116e0 tx=0x7f6160011aa0 comp rx=0 tx=0).stop 2024-04-23T10:17:41.088 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-04-23T10:17:40.455+0000 7f616b750700 1 -- 172.21.15.114:0/4240247777 shutdown_connections 2024-04-23T10:17:41.088 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-04-23T10:17:40.455+0000 7f616b750700 1 --2- 172.21.15.114:0/4240247777 >> [v2:172.21.15.114:6800/1604678557,v1:172.21.15.114:6801/1604678557] conn(0x7f61500395b0 0x7f615003ba70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:17:41.088 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-04-23T10:17:40.455+0000 7f616b750700 1 --2- 172.21.15.114:0/4240247777 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f6164109070 0x7f616419c3e0 unknown :-1 s=CLOSED pgs=60 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:17:41.088 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-04-23T10:17:40.455+0000 7f616b750700 1 -- 172.21.15.114:0/4240247777 >> 172.21.15.114:0/4240247777 conn(0x7f6164102a40 msgr2=0x7f61641034e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T10:17:41.088 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-04-23T10:17:40.455+0000 7f616b750700 1 -- 172.21.15.114:0/4240247777 shutdown_connections 2024-04-23T10:17:41.088 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-04-23T10:17:40.455+0000 7f616b750700 1 -- 172.21.15.114:0/4240247777 wait complete. 2024-04-23T10:17:41.088 INFO:teuthology.orchestra.run.smithi114.stdout:Waiting for the mgr to restart... 2024-04-23T10:17:41.088 INFO:teuthology.orchestra.run.smithi114.stdout:Waiting for mgr epoch 9... 2024-04-23T10:17:41.368 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Apr 23 10:17:41 smithi114 ceph-aa240572-0159-11ef-bc93-c7b262605968-mgr-x[29490]: 2024-04-23T10:17:41.087+0000 7f9be8535200 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2024-04-23T10:17:41.369 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Apr 23 10:17:41 smithi114 ceph-aa240572-0159-11ef-bc93-c7b262605968-mgr-x[29490]: 2024-04-23T10:17:41.262+0000 7f9be8535200 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2024-04-23T10:17:41.674 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Apr 23 10:17:41 smithi114 ceph-aa240572-0159-11ef-bc93-c7b262605968-mgr-x[29490]: 2024-04-23T10:17:41.437+0000 7f9be8535200 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2024-04-23T10:17:42.004 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Apr 23 10:17:41 smithi114 ceph-aa240572-0159-11ef-bc93-c7b262605968-mgr-x[29490]: 2024-04-23T10:17:41.728+0000 7f9be8535200 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2024-04-23T10:17:42.004 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Apr 23 10:17:41 smithi114 ceph-aa240572-0159-11ef-bc93-c7b262605968-mgr-x[29490]: 2024-04-23T10:17:41.819+0000 7f9be8535200 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2024-04-23T10:17:42.504 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Apr 23 10:17:42 smithi114 ceph-aa240572-0159-11ef-bc93-c7b262605968-mgr-x[29490]: 2024-04-23T10:17:42.018+0000 7f9be8535200 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2024-04-23T10:17:42.504 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Apr 23 10:17:42 smithi114 ceph-aa240572-0159-11ef-bc93-c7b262605968-mgr-x[29490]: 2024-04-23T10:17:42.210+0000 7f9be8535200 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2024-04-23T10:17:43.215 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Apr 23 10:17:42 smithi114 ceph-aa240572-0159-11ef-bc93-c7b262605968-mgr-x[29490]: 2024-04-23T10:17:42.797+0000 7f9be8535200 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2024-04-23T10:17:43.504 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Apr 23 10:17:43 smithi114 ceph-aa240572-0159-11ef-bc93-c7b262605968-mgr-x[29490]: 2024-04-23T10:17:43.216+0000 7f9be8535200 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2024-04-23T10:17:43.504 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Apr 23 10:17:43 smithi114 ceph-aa240572-0159-11ef-bc93-c7b262605968-mgr-x[29490]: 2024-04-23T10:17:43.312+0000 7f9be8535200 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2024-04-23T10:17:44.004 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Apr 23 10:17:43 smithi114 ceph-aa240572-0159-11ef-bc93-c7b262605968-mgr-x[29490]: 2024-04-23T10:17:43.514+0000 7f9be8535200 -1 mgr[py] Module rgw has missing NOTIFY_TYPES member 2024-04-23T10:17:44.282 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Apr 23 10:17:44 smithi114 ceph-aa240572-0159-11ef-bc93-c7b262605968-mgr-x[29490]: 2024-04-23T10:17:44.027+0000 7f9be8535200 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2024-04-23T10:17:44.282 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Apr 23 10:17:44 smithi114 ceph-aa240572-0159-11ef-bc93-c7b262605968-mgr-x[29490]: 2024-04-23T10:17:44.124+0000 7f9be8535200 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2024-04-23T10:17:44.615 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Apr 23 10:17:44 smithi114 ceph-aa240572-0159-11ef-bc93-c7b262605968-mgr-x[29490]: 2024-04-23T10:17:44.379+0000 7f9be8535200 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2024-04-23T10:17:44.615 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Apr 23 10:17:44 smithi114 ceph-aa240572-0159-11ef-bc93-c7b262605968-mgr-x[29490]: 2024-04-23T10:17:44.518+0000 7f9be8535200 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2024-04-23T10:17:44.615 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Apr 23 10:17:44 smithi114 ceph-aa240572-0159-11ef-bc93-c7b262605968-mgr-x[29490]: 2024-04-23T10:17:44.615+0000 7f9be8535200 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2024-04-23T10:17:45.004 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Apr 23 10:17:44 smithi114 ceph-aa240572-0159-11ef-bc93-c7b262605968-mgr-x[29490]: 2024-04-23T10:17:44.743+0000 7f9be8535200 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2024-04-23T10:17:45.504 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Apr 23 10:17:45 smithi114 ceph-aa240572-0159-11ef-bc93-c7b262605968-mgr-x[29490]: 2024-04-23T10:17:45.051+0000 7f9be8535200 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2024-04-23T10:17:45.925 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Apr 23 10:17:45 smithi114 ceph-aa240572-0159-11ef-bc93-c7b262605968-mgr-x[29490]: 2024-04-23T10:17:45.657+0000 7f9be8535200 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2024-04-23T10:17:45.925 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Apr 23 10:17:45 smithi114 ceph-aa240572-0159-11ef-bc93-c7b262605968-mgr-x[29490]: 2024-04-23T10:17:45.872+0000 7f9be8535200 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2024-04-23T10:17:46.254 INFO:journalctl@ceph.mon.a.smithi114.stdout:Apr 23 10:17:45 smithi114 ceph-mon[29255]: Active manager daemon x restarted 2024-04-23T10:17:46.255 INFO:journalctl@ceph.mon.a.smithi114.stdout:Apr 23 10:17:45 smithi114 ceph-mon[29255]: Activating manager daemon x 2024-04-23T10:17:46.255 INFO:journalctl@ceph.mon.a.smithi114.stdout:Apr 23 10:17:45 smithi114 ceph-mon[29255]: osdmap e3: 0 total, 0 up, 0 in 2024-04-23T10:17:46.255 INFO:journalctl@ceph.mon.a.smithi114.stdout:Apr 23 10:17:45 smithi114 ceph-mon[29255]: mgrmap e10: x(active, starting, since 0.00333362s) 2024-04-23T10:17:46.255 INFO:journalctl@ceph.mon.a.smithi114.stdout:Apr 23 10:17:45 smithi114 ceph-mon[29255]: from='mgr.14152 172.21.15.114:0/1529706138' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-04-23T10:17:46.255 INFO:journalctl@ceph.mon.a.smithi114.stdout:Apr 23 10:17:45 smithi114 ceph-mon[29255]: from='mgr.14152 172.21.15.114:0/1529706138' entity='mgr.x' cmd=[{"prefix": "mgr metadata", "who": "x", "id": "x"}]: dispatch 2024-04-23T10:17:46.255 INFO:journalctl@ceph.mon.a.smithi114.stdout:Apr 23 10:17:45 smithi114 ceph-mon[29255]: from='mgr.14152 172.21.15.114:0/1529706138' entity='mgr.x' cmd=[{"prefix": "mds metadata"}]: dispatch 2024-04-23T10:17:46.255 INFO:journalctl@ceph.mon.a.smithi114.stdout:Apr 23 10:17:45 smithi114 ceph-mon[29255]: from='mgr.14152 172.21.15.114:0/1529706138' entity='mgr.x' cmd=[{"prefix": "osd metadata"}]: dispatch 2024-04-23T10:17:46.255 INFO:journalctl@ceph.mon.a.smithi114.stdout:Apr 23 10:17:45 smithi114 ceph-mon[29255]: from='mgr.14152 172.21.15.114:0/1529706138' entity='mgr.x' cmd=[{"prefix": "mon metadata"}]: dispatch 2024-04-23T10:17:46.255 INFO:journalctl@ceph.mon.a.smithi114.stdout:Apr 23 10:17:45 smithi114 ceph-mon[29255]: Manager daemon x is now available 2024-04-23T10:17:46.255 INFO:journalctl@ceph.mon.a.smithi114.stdout:Apr 23 10:17:45 smithi114 ceph-mon[29255]: from='mgr.14152 172.21.15.114:0/1529706138' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-23T10:17:46.255 INFO:journalctl@ceph.mon.a.smithi114.stdout:Apr 23 10:17:45 smithi114 ceph-mon[29255]: from='mgr.14152 172.21.15.114:0/1529706138' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2024-04-23T10:17:47.235 INFO:journalctl@ceph.mon.a.smithi114.stdout:Apr 23 10:17:46 smithi114 ceph-mon[29255]: from='mgr.14152 172.21.15.114:0/1529706138' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2024-04-23T10:17:47.235 INFO:journalctl@ceph.mon.a.smithi114.stdout:Apr 23 10:17:46 smithi114 ceph-mon[29255]: mgrmap e11: x(active, since 1.00651s) 2024-04-23T10:17:47.744 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stdout { 2024-04-23T10:17:47.744 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stdout "mgrmap_epoch": 11, 2024-04-23T10:17:47.744 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stdout "initialized": true 2024-04-23T10:17:47.744 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stdout } 2024-04-23T10:17:47.744 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-04-23T10:17:41.469+0000 7f5b68e33700 1 Processor -- start 2024-04-23T10:17:47.744 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-04-23T10:17:41.470+0000 7f5b68e33700 1 -- start start 2024-04-23T10:17:47.744 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-04-23T10:17:41.470+0000 7f5b68e33700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f5b64108ff0 0x7f5b6410b3e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T10:17:47.744 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-04-23T10:17:41.470+0000 7f5b68e33700 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x7f5b6410b920 con 0x7f5b64108ff0 2024-04-23T10:17:47.744 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-04-23T10:17:41.470+0000 7f5b6259c700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f5b64108ff0 0x7f5b6410b3e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T10:17:47.745 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-04-23T10:17:41.470+0000 7f5b6259c700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f5b64108ff0 0x7f5b6410b3e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.114:47882/0 (socket says 172.21.15.114:47882) 2024-04-23T10:17:47.745 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-04-23T10:17:41.470+0000 7f5b6259c700 1 -- 172.21.15.114:0/3025014391 learned_addr learned my addr 172.21.15.114:0/3025014391 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-04-23T10:17:47.745 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-04-23T10:17:41.470+0000 7f5b6259c700 1 -- 172.21.15.114:0/3025014391 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f5b6410baa0 con 0x7f5b64108ff0 2024-04-23T10:17:47.745 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-04-23T10:17:41.471+0000 7f5b6259c700 1 --2- 172.21.15.114:0/3025014391 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f5b64108ff0 0x7f5b6410b3e0 secure :-1 s=READY pgs=61 cs=0 l=1 rev1=1 crypto rx=0x7f5b540098a0 tx=0x7f5b54009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ff58797d95c2707b server_cookie=0 in_seq=0 out_seq=0 2024-04-23T10:17:47.745 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-04-23T10:17:41.471+0000 7f5b6159a700 1 -- 172.21.15.114:0/3025014391 <== mon.0 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f5b54004160 con 0x7f5b64108ff0 2024-04-23T10:17:47.745 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-04-23T10:17:41.471+0000 7f5b6159a700 1 -- 172.21.15.114:0/3025014391 <== mon.0 v2:172.21.15.114:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f5b54004300 con 0x7f5b64108ff0 2024-04-23T10:17:47.745 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-04-23T10:17:41.471+0000 7f5b6159a700 1 -- 172.21.15.114:0/3025014391 <== mon.0 v2:172.21.15.114:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f5b54011a20 con 0x7f5b64108ff0 2024-04-23T10:17:47.745 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-04-23T10:17:41.471+0000 7f5b68e33700 1 -- 172.21.15.114:0/3025014391 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f5b64108ff0 msgr2=0x7f5b6410b3e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T10:17:47.745 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-04-23T10:17:41.471+0000 7f5b68e33700 1 --2- 172.21.15.114:0/3025014391 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f5b64108ff0 0x7f5b6410b3e0 secure :-1 s=READY pgs=61 cs=0 l=1 rev1=1 crypto rx=0x7f5b540098a0 tx=0x7f5b54009bb0 comp rx=0 tx=0).stop 2024-04-23T10:17:47.745 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-04-23T10:17:41.471+0000 7f5b68e33700 1 -- 172.21.15.114:0/3025014391 shutdown_connections 2024-04-23T10:17:47.745 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-04-23T10:17:41.471+0000 7f5b68e33700 1 --2- 172.21.15.114:0/3025014391 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f5b64108ff0 0x7f5b6410b3e0 unknown :-1 s=CLOSED pgs=61 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:17:47.745 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-04-23T10:17:41.471+0000 7f5b68e33700 1 -- 172.21.15.114:0/3025014391 >> 172.21.15.114:0/3025014391 conn(0x7f5b64102a60 msgr2=0x7f5b64104e80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T10:17:47.745 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-04-23T10:17:41.472+0000 7f5b68e33700 1 -- 172.21.15.114:0/3025014391 shutdown_connections 2024-04-23T10:17:47.745 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-04-23T10:17:41.472+0000 7f5b68e33700 1 -- 172.21.15.114:0/3025014391 wait complete. 2024-04-23T10:17:47.745 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-04-23T10:17:41.472+0000 7f5b68e33700 1 Processor -- start 2024-04-23T10:17:47.745 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-04-23T10:17:41.472+0000 7f5b68e33700 1 -- start start 2024-04-23T10:17:47.746 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-04-23T10:17:41.472+0000 7f5b68e33700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f5b64108ff0 0x7f5b6419c570 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T10:17:47.746 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-04-23T10:17:41.472+0000 7f5b68e33700 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x7f5b6410c9f0 con 0x7f5b64108ff0 2024-04-23T10:17:47.746 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-04-23T10:17:41.473+0000 7f5b6259c700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f5b64108ff0 0x7f5b6419c570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T10:17:47.746 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-04-23T10:17:41.473+0000 7f5b6259c700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f5b64108ff0 0x7f5b6419c570 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.114:47890/0 (socket says 172.21.15.114:47890) 2024-04-23T10:17:47.746 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-04-23T10:17:41.473+0000 7f5b6259c700 1 -- 172.21.15.114:0/96543514 learned_addr learned my addr 172.21.15.114:0/96543514 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-04-23T10:17:47.746 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-04-23T10:17:41.473+0000 7f5b6259c700 1 -- 172.21.15.114:0/96543514 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f5b6419cab0 con 0x7f5b64108ff0 2024-04-23T10:17:47.746 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-04-23T10:17:41.473+0000 7f5b6259c700 1 --2- 172.21.15.114:0/96543514 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f5b64108ff0 0x7f5b6419c570 secure :-1 s=READY pgs=62 cs=0 l=1 rev1=1 crypto rx=0x7f5b54004660 tx=0x7f5b54011aa0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T10:17:47.746 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-04-23T10:17:41.473+0000 7f5b537fe700 1 -- 172.21.15.114:0/96543514 <== mon.0 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f5b54003aa0 con 0x7f5b64108ff0 2024-04-23T10:17:47.746 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-04-23T10:17:41.473+0000 7f5b68e33700 1 -- 172.21.15.114:0/96543514 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f5b6419cd40 con 0x7f5b64108ff0 2024-04-23T10:17:47.746 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-04-23T10:17:41.473+0000 7f5b537fe700 1 -- 172.21.15.114:0/96543514 <== mon.0 v2:172.21.15.114:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f5b5401b8f0 con 0x7f5b64108ff0 2024-04-23T10:17:47.746 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-04-23T10:17:41.473+0000 7f5b537fe700 1 -- 172.21.15.114:0/96543514 <== mon.0 v2:172.21.15.114:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f5b54023900 con 0x7f5b64108ff0 2024-04-23T10:17:47.746 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-04-23T10:17:41.473+0000 7f5b68e33700 1 -- 172.21.15.114:0/96543514 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f5b6419fa30 con 0x7f5b64108ff0 2024-04-23T10:17:47.746 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-04-23T10:17:41.474+0000 7f5b537fe700 1 -- 172.21.15.114:0/96543514 <== mon.0 v2:172.21.15.114:3300/0 4 ==== mgrmap(e 9) ==== 46571+0+0 (secure 0 0 0) 0x7f5b54023bc0 con 0x7f5b64108ff0 2024-04-23T10:17:47.746 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-04-23T10:17:41.474+0000 7f5b537fe700 1 --2- 172.21.15.114:0/96543514 >> [v2:172.21.15.114:6800/1604678557,v1:172.21.15.114:6801/1604678557] conn(0x7f5b4c039600 0x7f5b4c03bac0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T10:17:47.746 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-04-23T10:17:41.474+0000 7f5b537fe700 1 -- 172.21.15.114:0/96543514 --> [v2:172.21.15.114:6800/1604678557,v1:172.21.15.114:6801/1604678557] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x7f5b4c03c1d0 con 0x7f5b4c039600 2024-04-23T10:17:47.747 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-04-23T10:17:41.474+0000 7f5b537fe700 1 -- 172.21.15.114:0/96543514 <== mon.0 v2:172.21.15.114:3300/0 5 ==== osd_map(2..2 src has 1..2) ==== 940+0+0 (secure 0 0 0) 0x7f5b5404e120 con 0x7f5b64108ff0 2024-04-23T10:17:47.747 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-04-23T10:17:41.474+0000 7f5b61d9b700 1 -- 172.21.15.114:0/96543514 >> [v2:172.21.15.114:6800/1604678557,v1:172.21.15.114:6801/1604678557] conn(0x7f5b4c039600 msgr2=0x7f5b4c03bac0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.114:6800/1604678557 2024-04-23T10:17:47.747 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-04-23T10:17:41.474+0000 7f5b61d9b700 1 --2- 172.21.15.114:0/96543514 >> [v2:172.21.15.114:6800/1604678557,v1:172.21.15.114:6801/1604678557] conn(0x7f5b4c039600 0x7f5b4c03bac0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-04-23T10:17:47.747 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-04-23T10:17:41.675+0000 7f5b61d9b700 1 -- 172.21.15.114:0/96543514 >> [v2:172.21.15.114:6800/1604678557,v1:172.21.15.114:6801/1604678557] conn(0x7f5b4c039600 msgr2=0x7f5b4c03bac0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.114:6800/1604678557 2024-04-23T10:17:47.747 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-04-23T10:17:41.675+0000 7f5b61d9b700 1 --2- 172.21.15.114:0/96543514 >> [v2:172.21.15.114:6800/1604678557,v1:172.21.15.114:6801/1604678557] conn(0x7f5b4c039600 0x7f5b4c03bac0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-04-23T10:17:47.747 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-04-23T10:17:42.076+0000 7f5b61d9b700 1 -- 172.21.15.114:0/96543514 >> [v2:172.21.15.114:6800/1604678557,v1:172.21.15.114:6801/1604678557] conn(0x7f5b4c039600 msgr2=0x7f5b4c03bac0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.114:6800/1604678557 2024-04-23T10:17:47.747 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-04-23T10:17:42.076+0000 7f5b61d9b700 1 --2- 172.21.15.114:0/96543514 >> [v2:172.21.15.114:6800/1604678557,v1:172.21.15.114:6801/1604678557] conn(0x7f5b4c039600 0x7f5b4c03bac0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2024-04-23T10:17:47.747 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-04-23T10:17:42.877+0000 7f5b61d9b700 1 -- 172.21.15.114:0/96543514 >> [v2:172.21.15.114:6800/1604678557,v1:172.21.15.114:6801/1604678557] conn(0x7f5b4c039600 msgr2=0x7f5b4c03bac0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.114:6800/1604678557 2024-04-23T10:17:47.747 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-04-23T10:17:42.877+0000 7f5b61d9b700 1 --2- 172.21.15.114:0/96543514 >> [v2:172.21.15.114:6800/1604678557,v1:172.21.15.114:6801/1604678557] conn(0x7f5b4c039600 0x7f5b4c03bac0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2024-04-23T10:17:47.747 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-04-23T10:17:44.478+0000 7f5b61d9b700 1 -- 172.21.15.114:0/96543514 >> [v2:172.21.15.114:6800/1604678557,v1:172.21.15.114:6801/1604678557] conn(0x7f5b4c039600 msgr2=0x7f5b4c03bac0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.114:6800/1604678557 2024-04-23T10:17:47.747 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-04-23T10:17:44.478+0000 7f5b61d9b700 1 --2- 172.21.15.114:0/96543514 >> [v2:172.21.15.114:6800/1604678557,v1:172.21.15.114:6801/1604678557] conn(0x7f5b4c039600 0x7f5b4c03bac0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 3.200000 2024-04-23T10:17:47.747 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-04-23T10:17:45.875+0000 7f5b537fe700 1 -- 172.21.15.114:0/96543514 <== mon.0 v2:172.21.15.114:3300/0 6 ==== mgrmap(e 10) ==== 46338+0+0 (secure 0 0 0) 0x7f5b5402c3f0 con 0x7f5b64108ff0 2024-04-23T10:17:47.747 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-04-23T10:17:45.875+0000 7f5b537fe700 1 -- 172.21.15.114:0/96543514 >> [v2:172.21.15.114:6800/1604678557,v1:172.21.15.114:6801/1604678557] conn(0x7f5b4c039600 msgr2=0x7f5b4c03bac0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-04-23T10:17:47.747 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-04-23T10:17:45.875+0000 7f5b537fe700 1 --2- 172.21.15.114:0/96543514 >> [v2:172.21.15.114:6800/1604678557,v1:172.21.15.114:6801/1604678557] conn(0x7f5b4c039600 0x7f5b4c03bac0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:17:47.747 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-04-23T10:17:46.877+0000 7f5b537fe700 1 -- 172.21.15.114:0/96543514 <== mon.0 v2:172.21.15.114:3300/0 7 ==== mgrmap(e 11) ==== 46465+0+0 (secure 0 0 0) 0x7f5b5404fd20 con 0x7f5b64108ff0 2024-04-23T10:17:47.748 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-04-23T10:17:46.877+0000 7f5b537fe700 1 --2- 172.21.15.114:0/96543514 >> [v2:172.21.15.114:6800/465333808,v1:172.21.15.114:6801/465333808] conn(0x7f5b4c03cfe0 0x7f5b4c03f3d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T10:17:47.748 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-04-23T10:17:46.877+0000 7f5b537fe700 1 -- 172.21.15.114:0/96543514 --> [v2:172.21.15.114:6800/465333808,v1:172.21.15.114:6801/465333808] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x7f5b54001d30 con 0x7f5b4c03cfe0 2024-04-23T10:17:47.748 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-04-23T10:17:46.878+0000 7f5b61d9b700 1 --2- 172.21.15.114:0/96543514 >> [v2:172.21.15.114:6800/465333808,v1:172.21.15.114:6801/465333808] conn(0x7f5b4c03cfe0 0x7f5b4c03f3d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T10:17:47.748 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-04-23T10:17:46.878+0000 7f5b61d9b700 1 --2- 172.21.15.114:0/96543514 >> [v2:172.21.15.114:6800/465333808,v1:172.21.15.114:6801/465333808] conn(0x7f5b4c03cfe0 0x7f5b4c03f3d0 secure :-1 s=READY pgs=2 cs=0 l=1 rev1=1 crypto rx=0x7f5b58003a10 tx=0x7f5b580092b0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T10:17:47.748 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-04-23T10:17:46.879+0000 7f5b537fe700 1 -- 172.21.15.114:0/96543514 <== mgr.14152 v2:172.21.15.114:6800/465333808 1 ==== command_reply(tid 0: 0 ) ==== 8+0+7774 (secure 0 0 0) 0x7f5b54001d30 con 0x7f5b4c03cfe0 2024-04-23T10:17:47.748 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-04-23T10:17:46.890+0000 7f5b68e33700 1 -- 172.21.15.114:0/96543514 --> [v2:172.21.15.114:6800/465333808,v1:172.21.15.114:6801/465333808] -- command(tid 1: {"prefix": "mgr_status"}) -- 0x7f5b6419d220 con 0x7f5b4c03cfe0 2024-04-23T10:17:47.748 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-04-23T10:17:46.890+0000 7f5b537fe700 1 -- 172.21.15.114:0/96543514 <== mgr.14152 v2:172.21.15.114:6800/465333808 2 ==== command_reply(tid 1: 0 ) ==== 8+0+52 (secure 0 0 0) 0x7f5b6419d220 con 0x7f5b4c03cfe0 2024-04-23T10:17:47.748 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-04-23T10:17:46.891+0000 7f5b68e33700 1 -- 172.21.15.114:0/96543514 >> [v2:172.21.15.114:6800/465333808,v1:172.21.15.114:6801/465333808] conn(0x7f5b4c03cfe0 msgr2=0x7f5b4c03f3d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T10:17:47.748 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-04-23T10:17:46.891+0000 7f5b68e33700 1 --2- 172.21.15.114:0/96543514 >> [v2:172.21.15.114:6800/465333808,v1:172.21.15.114:6801/465333808] conn(0x7f5b4c03cfe0 0x7f5b4c03f3d0 secure :-1 s=READY pgs=2 cs=0 l=1 rev1=1 crypto rx=0x7f5b58003a10 tx=0x7f5b580092b0 comp rx=0 tx=0).stop 2024-04-23T10:17:47.748 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-04-23T10:17:46.891+0000 7f5b68e33700 1 -- 172.21.15.114:0/96543514 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f5b64108ff0 msgr2=0x7f5b6419c570 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T10:17:47.748 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-04-23T10:17:46.891+0000 7f5b68e33700 1 --2- 172.21.15.114:0/96543514 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f5b64108ff0 0x7f5b6419c570 secure :-1 s=READY pgs=62 cs=0 l=1 rev1=1 crypto rx=0x7f5b54004660 tx=0x7f5b54011aa0 comp rx=0 tx=0).stop 2024-04-23T10:17:47.748 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-04-23T10:17:46.891+0000 7f5b68e33700 1 -- 172.21.15.114:0/96543514 shutdown_connections 2024-04-23T10:17:47.748 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-04-23T10:17:46.891+0000 7f5b68e33700 1 --2- 172.21.15.114:0/96543514 >> [v2:172.21.15.114:6800/465333808,v1:172.21.15.114:6801/465333808] conn(0x7f5b4c03cfe0 0x7f5b4c03f3d0 unknown :-1 s=CLOSED pgs=2 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:17:47.748 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-04-23T10:17:46.891+0000 7f5b68e33700 1 --2- 172.21.15.114:0/96543514 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f5b64108ff0 0x7f5b6419c570 unknown :-1 s=CLOSED pgs=62 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:17:47.748 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-04-23T10:17:46.891+0000 7f5b68e33700 1 --2- 172.21.15.114:0/96543514 >> [v2:172.21.15.114:6800/1604678557,v1:172.21.15.114:6801/1604678557] conn(0x7f5b4c039600 0x7f5b4c03bac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:17:47.748 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-04-23T10:17:46.891+0000 7f5b68e33700 1 -- 172.21.15.114:0/96543514 >> 172.21.15.114:0/96543514 conn(0x7f5b64102a60 msgr2=0x7f5b641034f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T10:17:47.749 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-04-23T10:17:46.891+0000 7f5b68e33700 1 -- 172.21.15.114:0/96543514 shutdown_connections 2024-04-23T10:17:47.749 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-04-23T10:17:46.891+0000 7f5b68e33700 1 -- 172.21.15.114:0/96543514 wait complete. 2024-04-23T10:17:47.749 INFO:teuthology.orchestra.run.smithi114.stdout:mgr epoch 9 is available 2024-04-23T10:17:47.749 INFO:teuthology.orchestra.run.smithi114.stdout:Generating a dashboard self-signed certificate... 2024-04-23T10:17:48.162 INFO:journalctl@ceph.mon.a.smithi114.stdout:Apr 23 10:17:47 smithi114 ceph-mon[29255]: [23/Apr/2024:10:17:46] ENGINE Bus STARTING 2024-04-23T10:17:48.162 INFO:journalctl@ceph.mon.a.smithi114.stdout:Apr 23 10:17:47 smithi114 ceph-mon[29255]: from='mgr.14152 172.21.15.114:0/1529706138' entity='mgr.x' 2024-04-23T10:17:49.032 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stdout Self-signed certificate created 2024-04-23T10:17:49.032 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-04-23T10:17:48.171+0000 7f5b30579700 1 Processor -- start 2024-04-23T10:17:49.032 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-04-23T10:17:48.171+0000 7f5b30579700 1 -- start start 2024-04-23T10:17:49.032 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-04-23T10:17:48.172+0000 7f5b30579700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f5b28109010 0x7f5b2810b400 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T10:17:49.032 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-04-23T10:17:48.172+0000 7f5b30579700 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x7f5b28063d00 con 0x7f5b28109010 2024-04-23T10:17:49.032 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-04-23T10:17:48.172+0000 7f5b2e315700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f5b28109010 0x7f5b2810b400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T10:17:49.032 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-04-23T10:17:48.172+0000 7f5b2e315700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f5b28109010 0x7f5b2810b400 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.114:44786/0 (socket says 172.21.15.114:44786) 2024-04-23T10:17:49.032 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-04-23T10:17:48.172+0000 7f5b2e315700 1 -- 172.21.15.114:0/1631772189 learned_addr learned my addr 172.21.15.114:0/1631772189 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-04-23T10:17:49.033 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-04-23T10:17:48.172+0000 7f5b2e315700 1 -- 172.21.15.114:0/1631772189 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f5b2810b940 con 0x7f5b28109010 2024-04-23T10:17:49.033 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-04-23T10:17:48.172+0000 7f5b2e315700 1 --2- 172.21.15.114:0/1631772189 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f5b28109010 0x7f5b2810b400 secure :-1 s=READY pgs=70 cs=0 l=1 rev1=1 crypto rx=0x7f5b240098a0 tx=0x7f5b24009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3844deb281de741d server_cookie=0 in_seq=0 out_seq=0 2024-04-23T10:17:49.033 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-04-23T10:17:48.173+0000 7f5b2d313700 1 -- 172.21.15.114:0/1631772189 <== mon.0 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f5b24004160 con 0x7f5b28109010 2024-04-23T10:17:49.033 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-04-23T10:17:48.173+0000 7f5b2d313700 1 -- 172.21.15.114:0/1631772189 <== mon.0 v2:172.21.15.114:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f5b24004300 con 0x7f5b28109010 2024-04-23T10:17:49.033 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-04-23T10:17:48.173+0000 7f5b2d313700 1 -- 172.21.15.114:0/1631772189 <== mon.0 v2:172.21.15.114:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f5b24011a20 con 0x7f5b28109010 2024-04-23T10:17:49.033 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-04-23T10:17:48.173+0000 7f5b30579700 1 -- 172.21.15.114:0/1631772189 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f5b28109010 msgr2=0x7f5b2810b400 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T10:17:49.033 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-04-23T10:17:48.173+0000 7f5b30579700 1 --2- 172.21.15.114:0/1631772189 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f5b28109010 0x7f5b2810b400 secure :-1 s=READY pgs=70 cs=0 l=1 rev1=1 crypto rx=0x7f5b240098a0 tx=0x7f5b24009bb0 comp rx=0 tx=0).stop 2024-04-23T10:17:49.033 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-04-23T10:17:48.173+0000 7f5b30579700 1 -- 172.21.15.114:0/1631772189 shutdown_connections 2024-04-23T10:17:49.033 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-04-23T10:17:48.173+0000 7f5b30579700 1 --2- 172.21.15.114:0/1631772189 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f5b28109010 0x7f5b2810b400 unknown :-1 s=CLOSED pgs=70 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:17:49.033 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-04-23T10:17:48.173+0000 7f5b30579700 1 -- 172.21.15.114:0/1631772189 >> 172.21.15.114:0/1631772189 conn(0x7f5b28102a40 msgr2=0x7f5b28104e60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T10:17:49.033 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-04-23T10:17:48.173+0000 7f5b30579700 1 -- 172.21.15.114:0/1631772189 shutdown_connections 2024-04-23T10:17:49.033 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-04-23T10:17:48.173+0000 7f5b30579700 1 -- 172.21.15.114:0/1631772189 wait complete. 2024-04-23T10:17:49.033 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-04-23T10:17:48.174+0000 7f5b30579700 1 Processor -- start 2024-04-23T10:17:49.033 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-04-23T10:17:48.174+0000 7f5b30579700 1 -- start start 2024-04-23T10:17:49.034 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-04-23T10:17:48.174+0000 7f5b30579700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f5b28109010 0x7f5b2819c3f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T10:17:49.034 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-04-23T10:17:48.174+0000 7f5b30579700 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x7f5b2810c8f0 con 0x7f5b28109010 2024-04-23T10:17:49.034 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-04-23T10:17:48.174+0000 7f5b2e315700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f5b28109010 0x7f5b2819c3f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T10:17:49.034 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-04-23T10:17:48.174+0000 7f5b2e315700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f5b28109010 0x7f5b2819c3f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.114:44794/0 (socket says 172.21.15.114:44794) 2024-04-23T10:17:49.034 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-04-23T10:17:48.174+0000 7f5b2e315700 1 -- 172.21.15.114:0/489689643 learned_addr learned my addr 172.21.15.114:0/489689643 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-04-23T10:17:49.034 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-04-23T10:17:48.175+0000 7f5b2e315700 1 -- 172.21.15.114:0/489689643 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f5b2819c930 con 0x7f5b28109010 2024-04-23T10:17:49.034 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-04-23T10:17:48.175+0000 7f5b2e315700 1 --2- 172.21.15.114:0/489689643 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f5b28109010 0x7f5b2819c3f0 secure :-1 s=READY pgs=71 cs=0 l=1 rev1=1 crypto rx=0x7f5b24000c00 tx=0x7f5b24011aa0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T10:17:49.034 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-04-23T10:17:48.175+0000 7f5b1f7fe700 1 -- 172.21.15.114:0/489689643 <== mon.0 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f5b24003aa0 con 0x7f5b28109010 2024-04-23T10:17:49.034 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-04-23T10:17:48.175+0000 7f5b30579700 1 -- 172.21.15.114:0/489689643 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f5b2819cbc0 con 0x7f5b28109010 2024-04-23T10:17:49.034 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-04-23T10:17:48.175+0000 7f5b1f7fe700 1 -- 172.21.15.114:0/489689643 <== mon.0 v2:172.21.15.114:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f5b2401b8f0 con 0x7f5b28109010 2024-04-23T10:17:49.034 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-04-23T10:17:48.175+0000 7f5b1f7fe700 1 -- 172.21.15.114:0/489689643 <== mon.0 v2:172.21.15.114:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f5b24023870 con 0x7f5b28109010 2024-04-23T10:17:49.034 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-04-23T10:17:48.175+0000 7f5b30579700 1 -- 172.21.15.114:0/489689643 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f5b2819f8b0 con 0x7f5b28109010 2024-04-23T10:17:49.034 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-04-23T10:17:48.176+0000 7f5b1f7fe700 1 -- 172.21.15.114:0/489689643 <== mon.0 v2:172.21.15.114:3300/0 4 ==== mgrmap(e 11) ==== 46465+0+0 (secure 0 0 0) 0x7f5b24023b70 con 0x7f5b28109010 2024-04-23T10:17:49.034 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-04-23T10:17:48.176+0000 7f5b30579700 1 -- 172.21.15.114:0/489689643 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f5b280640a0 con 0x7f5b28109010 2024-04-23T10:17:49.035 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-04-23T10:17:48.176+0000 7f5b1f7fe700 1 --2- 172.21.15.114:0/489689643 >> [v2:172.21.15.114:6800/465333808,v1:172.21.15.114:6801/465333808] conn(0x7f5b140394e0 0x7f5b1403b9a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T10:17:49.035 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-04-23T10:17:48.176+0000 7f5b1f7fe700 1 -- 172.21.15.114:0/489689643 <== mon.0 v2:172.21.15.114:3300/0 5 ==== osd_map(3..3 src has 1..3) ==== 1155+0+0 (secure 0 0 0) 0x7f5b2404de80 con 0x7f5b28109010 2024-04-23T10:17:49.035 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-04-23T10:17:48.176+0000 7f5b2db14700 1 --2- 172.21.15.114:0/489689643 >> [v2:172.21.15.114:6800/465333808,v1:172.21.15.114:6801/465333808] conn(0x7f5b140394e0 0x7f5b1403b9a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T10:17:49.035 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-04-23T10:17:48.177+0000 7f5b2db14700 1 --2- 172.21.15.114:0/489689643 >> [v2:172.21.15.114:6800/465333808,v1:172.21.15.114:6801/465333808] conn(0x7f5b140394e0 0x7f5b1403b9a0 secure :-1 s=READY pgs=7 cs=0 l=1 rev1=1 crypto rx=0x7f5b18006fd0 tx=0x7f5b18006e40 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T10:17:49.035 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-04-23T10:17:48.181+0000 7f5b1f7fe700 1 -- 172.21.15.114:0/489689643 <== mon.0 v2:172.21.15.114:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f5b2401f030 con 0x7f5b28109010 2024-04-23T10:17:49.035 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-04-23T10:17:48.237+0000 7f5b1f7fe700 1 -- 172.21.15.114:0/489689643 <== mon.0 v2:172.21.15.114:3300/0 7 ==== mgrmap(e 12) ==== 46571+0+0 (secure 0 0 0) 0x7f5b2402c910 con 0x7f5b28109010 2024-04-23T10:17:49.035 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-04-23T10:17:48.394+0000 7f5b30579700 1 -- 172.21.15.114:0/489689643 --> [v2:172.21.15.114:6800/465333808,v1:172.21.15.114:6801/465333808] -- mgr_command(tid 0: {"prefix": "dashboard create-self-signed-cert", "target": ["mon-mgr", ""]}) -- 0x7f5b28103ba0 con 0x7f5b140394e0 2024-04-23T10:17:49.035 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-04-23T10:17:48.452+0000 7f5b1f7fe700 1 -- 172.21.15.114:0/489689643 <== mgr.14152 v2:172.21.15.114:6800/465333808 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+32 (secure 0 0 0) 0x7f5b28103ba0 con 0x7f5b140394e0 2024-04-23T10:17:49.035 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-04-23T10:17:48.454+0000 7f5b30579700 1 -- 172.21.15.114:0/489689643 >> [v2:172.21.15.114:6800/465333808,v1:172.21.15.114:6801/465333808] conn(0x7f5b140394e0 msgr2=0x7f5b1403b9a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T10:17:49.035 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-04-23T10:17:48.454+0000 7f5b30579700 1 --2- 172.21.15.114:0/489689643 >> [v2:172.21.15.114:6800/465333808,v1:172.21.15.114:6801/465333808] conn(0x7f5b140394e0 0x7f5b1403b9a0 secure :-1 s=READY pgs=7 cs=0 l=1 rev1=1 crypto rx=0x7f5b18006fd0 tx=0x7f5b18006e40 comp rx=0 tx=0).stop 2024-04-23T10:17:49.035 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-04-23T10:17:48.454+0000 7f5b30579700 1 -- 172.21.15.114:0/489689643 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f5b28109010 msgr2=0x7f5b2819c3f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T10:17:49.035 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-04-23T10:17:48.454+0000 7f5b30579700 1 --2- 172.21.15.114:0/489689643 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f5b28109010 0x7f5b2819c3f0 secure :-1 s=READY pgs=71 cs=0 l=1 rev1=1 crypto rx=0x7f5b24000c00 tx=0x7f5b24011aa0 comp rx=0 tx=0).stop 2024-04-23T10:17:49.035 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-04-23T10:17:48.454+0000 7f5b30579700 1 -- 172.21.15.114:0/489689643 shutdown_connections 2024-04-23T10:17:49.035 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-04-23T10:17:48.454+0000 7f5b30579700 1 --2- 172.21.15.114:0/489689643 >> [v2:172.21.15.114:6800/465333808,v1:172.21.15.114:6801/465333808] conn(0x7f5b140394e0 0x7f5b1403b9a0 unknown :-1 s=CLOSED pgs=7 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:17:49.036 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-04-23T10:17:48.454+0000 7f5b30579700 1 --2- 172.21.15.114:0/489689643 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f5b28109010 0x7f5b2819c3f0 unknown :-1 s=CLOSED pgs=71 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:17:49.036 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-04-23T10:17:48.454+0000 7f5b30579700 1 -- 172.21.15.114:0/489689643 >> 172.21.15.114:0/489689643 conn(0x7f5b28102a40 msgr2=0x7f5b28103490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T10:17:49.036 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-04-23T10:17:48.454+0000 7f5b30579700 1 -- 172.21.15.114:0/489689643 shutdown_connections 2024-04-23T10:17:49.036 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-04-23T10:17:48.454+0000 7f5b30579700 1 -- 172.21.15.114:0/489689643 wait complete. 2024-04-23T10:17:49.036 INFO:teuthology.orchestra.run.smithi114.stdout:Creating initial admin user... 2024-04-23T10:17:49.156 INFO:journalctl@ceph.mon.a.smithi114.stdout:Apr 23 10:17:48 smithi114 ceph-mon[29255]: [23/Apr/2024:10:17:47] ENGINE Serving on http://172.21.15.114:8765 2024-04-23T10:17:49.156 INFO:journalctl@ceph.mon.a.smithi114.stdout:Apr 23 10:17:48 smithi114 ceph-mon[29255]: [23/Apr/2024:10:17:47] ENGINE Serving on https://172.21.15.114:7150 2024-04-23T10:17:49.156 INFO:journalctl@ceph.mon.a.smithi114.stdout:Apr 23 10:17:48 smithi114 ceph-mon[29255]: [23/Apr/2024:10:17:47] ENGINE Bus STARTED 2024-04-23T10:17:49.156 INFO:journalctl@ceph.mon.a.smithi114.stdout:Apr 23 10:17:48 smithi114 ceph-mon[29255]: mgrmap e12: x(active, since 2s) 2024-04-23T10:17:49.156 INFO:journalctl@ceph.mon.a.smithi114.stdout:Apr 23 10:17:48 smithi114 ceph-mon[29255]: from='mgr.14152 172.21.15.114:0/1529706138' entity='mgr.x' 2024-04-23T10:17:49.156 INFO:journalctl@ceph.mon.a.smithi114.stdout:Apr 23 10:17:48 smithi114 ceph-mon[29255]: from='mgr.14152 172.21.15.114:0/1529706138' entity='mgr.x' 2024-04-23T10:17:49.157 INFO:journalctl@ceph.mon.a.smithi114.stdout:Apr 23 10:17:48 smithi114 ceph-mon[29255]: from='mgr.14152 172.21.15.114:0/1529706138' entity='mgr.x' 2024-04-23T10:17:50.004 INFO:journalctl@ceph.mon.a.smithi114.stdout:Apr 23 10:17:49 smithi114 ceph-mon[29255]: from='client.14164 -' entity='client.admin' cmd=[{"prefix": "dashboard create-self-signed-cert", "target": ["mon-mgr", ""]}]: dispatch 2024-04-23T10:17:50.675 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stdout {"username": "admin", "password": "$2b$12$gxa7jVj6Jxnd7XCVynA7euLRxS10KaMUbcT5jgy7VyCNTMD65/UM6", "roles": ["administrator"], "name": null, "email": null, "lastUpdate": 1713867469, "enabled": true, "pwdExpirationDate": null, "pwdUpdateRequired": true} 2024-04-23T10:17:50.675 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-04-23T10:17:49.492+0000 7f35e238c700 1 Processor -- start 2024-04-23T10:17:50.675 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-04-23T10:17:49.492+0000 7f35e238c700 1 -- start start 2024-04-23T10:17:50.675 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-04-23T10:17:49.492+0000 7f35e238c700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f35dc106810 0x7f35dc106c30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T10:17:50.675 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-04-23T10:17:49.492+0000 7f35e238c700 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x7f35dc107200 con 0x7f35dc106810 2024-04-23T10:17:50.675 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-04-23T10:17:49.492+0000 7f35dbfff700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f35dc106810 0x7f35dc106c30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T10:17:50.675 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-04-23T10:17:49.492+0000 7f35dbfff700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f35dc106810 0x7f35dc106c30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.114:44804/0 (socket says 172.21.15.114:44804) 2024-04-23T10:17:50.675 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-04-23T10:17:49.492+0000 7f35dbfff700 1 -- 172.21.15.114:0/3641760470 learned_addr learned my addr 172.21.15.114:0/3641760470 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-04-23T10:17:50.675 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-04-23T10:17:49.493+0000 7f35dbfff700 1 -- 172.21.15.114:0/3641760470 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f35dc107380 con 0x7f35dc106810 2024-04-23T10:17:50.676 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-04-23T10:17:49.493+0000 7f35dbfff700 1 --2- 172.21.15.114:0/3641760470 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f35dc106810 0x7f35dc106c30 secure :-1 s=READY pgs=72 cs=0 l=1 rev1=1 crypto rx=0x7f35cc0098a0 tx=0x7f35cc009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=76d6d8574ad111f0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T10:17:50.676 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-04-23T10:17:49.493+0000 7f35daffd700 1 -- 172.21.15.114:0/3641760470 <== mon.0 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f35cc004160 con 0x7f35dc106810 2024-04-23T10:17:50.676 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-04-23T10:17:49.493+0000 7f35daffd700 1 -- 172.21.15.114:0/3641760470 <== mon.0 v2:172.21.15.114:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f35cc004300 con 0x7f35dc106810 2024-04-23T10:17:50.676 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-04-23T10:17:49.493+0000 7f35daffd700 1 -- 172.21.15.114:0/3641760470 <== mon.0 v2:172.21.15.114:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f35cc011a20 con 0x7f35dc106810 2024-04-23T10:17:50.676 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-04-23T10:17:49.494+0000 7f35e238c700 1 -- 172.21.15.114:0/3641760470 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f35dc106810 msgr2=0x7f35dc106c30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T10:17:50.676 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-04-23T10:17:49.494+0000 7f35e238c700 1 --2- 172.21.15.114:0/3641760470 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f35dc106810 0x7f35dc106c30 secure :-1 s=READY pgs=72 cs=0 l=1 rev1=1 crypto rx=0x7f35cc0098a0 tx=0x7f35cc009bb0 comp rx=0 tx=0).stop 2024-04-23T10:17:50.676 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-04-23T10:17:49.494+0000 7f35e238c700 1 -- 172.21.15.114:0/3641760470 shutdown_connections 2024-04-23T10:17:50.676 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-04-23T10:17:49.494+0000 7f35e238c700 1 --2- 172.21.15.114:0/3641760470 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f35dc106810 0x7f35dc106c30 unknown :-1 s=CLOSED pgs=72 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:17:50.676 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-04-23T10:17:49.494+0000 7f35e238c700 1 -- 172.21.15.114:0/3641760470 >> 172.21.15.114:0/3641760470 conn(0x7f35dc102150 msgr2=0x7f35dc104590 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T10:17:50.676 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-04-23T10:17:49.494+0000 7f35e238c700 1 -- 172.21.15.114:0/3641760470 shutdown_connections 2024-04-23T10:17:50.676 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-04-23T10:17:49.494+0000 7f35e238c700 1 -- 172.21.15.114:0/3641760470 wait complete. 2024-04-23T10:17:50.676 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-04-23T10:17:49.494+0000 7f35e238c700 1 Processor -- start 2024-04-23T10:17:50.676 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-04-23T10:17:49.494+0000 7f35e238c700 1 -- start start 2024-04-23T10:17:50.676 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-04-23T10:17:49.495+0000 7f35e238c700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f35dc106810 0x7f35dc19c400 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T10:17:50.677 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-04-23T10:17:49.495+0000 7f35e238c700 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x7f35dc10a720 con 0x7f35dc106810 2024-04-23T10:17:50.677 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-04-23T10:17:49.495+0000 7f35dbfff700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f35dc106810 0x7f35dc19c400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T10:17:50.677 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-04-23T10:17:49.495+0000 7f35dbfff700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f35dc106810 0x7f35dc19c400 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.114:44812/0 (socket says 172.21.15.114:44812) 2024-04-23T10:17:50.677 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-04-23T10:17:49.495+0000 7f35dbfff700 1 -- 172.21.15.114:0/1531179744 learned_addr learned my addr 172.21.15.114:0/1531179744 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-04-23T10:17:50.677 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-04-23T10:17:49.495+0000 7f35dbfff700 1 -- 172.21.15.114:0/1531179744 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f35dc19c940 con 0x7f35dc106810 2024-04-23T10:17:50.677 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-04-23T10:17:49.495+0000 7f35dbfff700 1 --2- 172.21.15.114:0/1531179744 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f35dc106810 0x7f35dc19c400 secure :-1 s=READY pgs=73 cs=0 l=1 rev1=1 crypto rx=0x7f35cc0116e0 tx=0x7f35cc011aa0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T10:17:50.677 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-04-23T10:17:49.496+0000 7f35d97fa700 1 -- 172.21.15.114:0/1531179744 <== mon.0 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f35cc0039e0 con 0x7f35dc106810 2024-04-23T10:17:50.677 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-04-23T10:17:49.496+0000 7f35d97fa700 1 -- 172.21.15.114:0/1531179744 <== mon.0 v2:172.21.15.114:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f35cc01b8f0 con 0x7f35dc106810 2024-04-23T10:17:50.677 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-04-23T10:17:49.496+0000 7f35e238c700 1 -- 172.21.15.114:0/1531179744 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f35dc19cc30 con 0x7f35dc106810 2024-04-23T10:17:50.677 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-04-23T10:17:49.496+0000 7f35d97fa700 1 -- 172.21.15.114:0/1531179744 <== mon.0 v2:172.21.15.114:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f35cc023700 con 0x7f35dc106810 2024-04-23T10:17:50.677 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-04-23T10:17:49.496+0000 7f35e238c700 1 -- 172.21.15.114:0/1531179744 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f35dc19f8c0 con 0x7f35dc106810 2024-04-23T10:17:50.677 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-04-23T10:17:49.497+0000 7f35d97fa700 1 -- 172.21.15.114:0/1531179744 <== mon.0 v2:172.21.15.114:3300/0 4 ==== mgrmap(e 12) ==== 46571+0+0 (secure 0 0 0) 0x7f35cc023970 con 0x7f35dc106810 2024-04-23T10:17:50.677 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-04-23T10:17:49.497+0000 7f35d97fa700 1 --2- 172.21.15.114:0/1531179744 >> [v2:172.21.15.114:6800/465333808,v1:172.21.15.114:6801/465333808] conn(0x7f35c40395b0 0x7f35c403ba70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T10:17:50.677 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-04-23T10:17:49.497+0000 7f35d97fa700 1 -- 172.21.15.114:0/1531179744 <== mon.0 v2:172.21.15.114:3300/0 5 ==== osd_map(3..3 src has 1..3) ==== 1155+0+0 (secure 0 0 0) 0x7f35cc04dfd0 con 0x7f35dc106810 2024-04-23T10:17:50.677 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-04-23T10:17:49.497+0000 7f35db7fe700 1 --2- 172.21.15.114:0/1531179744 >> [v2:172.21.15.114:6800/465333808,v1:172.21.15.114:6801/465333808] conn(0x7f35c40395b0 0x7f35c403ba70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T10:17:50.678 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-04-23T10:17:49.497+0000 7f35e238c700 1 -- 172.21.15.114:0/1531179744 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f35dc0640a0 con 0x7f35dc106810 2024-04-23T10:17:50.678 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-04-23T10:17:49.497+0000 7f35db7fe700 1 --2- 172.21.15.114:0/1531179744 >> [v2:172.21.15.114:6800/465333808,v1:172.21.15.114:6801/465333808] conn(0x7f35c40395b0 0x7f35c403ba70 secure :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0x7f35d0006fd0 tx=0x7f35d0006e40 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T10:17:50.678 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-04-23T10:17:49.505+0000 7f35d97fa700 1 -- 172.21.15.114:0/1531179744 <== mon.0 v2:172.21.15.114:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f35cc01f030 con 0x7f35dc106810 2024-04-23T10:17:50.678 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-04-23T10:17:49.721+0000 7f35e238c700 1 -- 172.21.15.114:0/1531179744 --> [v2:172.21.15.114:6800/465333808,v1:172.21.15.114:6801/465333808] -- mgr_command(tid 0: {"prefix": "dashboard ac-user-create", "username": "admin", "rolename": "administrator", "force_password": true, "pwd_update_required": true, "target": ["mon-mgr", ""]}) -- 0x7f35dc103350 con 0x7f35c40395b0 2024-04-23T10:17:50.678 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-04-23T10:17:49.948+0000 7f35d97fa700 1 -- 172.21.15.114:0/1531179744 <== mgr.14152 v2:172.21.15.114:6800/465333808 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+252 (secure 0 0 0) 0x7f35dc103350 con 0x7f35c40395b0 2024-04-23T10:17:50.678 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-04-23T10:17:49.950+0000 7f35e238c700 1 -- 172.21.15.114:0/1531179744 >> [v2:172.21.15.114:6800/465333808,v1:172.21.15.114:6801/465333808] conn(0x7f35c40395b0 msgr2=0x7f35c403ba70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T10:17:50.678 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-04-23T10:17:49.950+0000 7f35e238c700 1 --2- 172.21.15.114:0/1531179744 >> [v2:172.21.15.114:6800/465333808,v1:172.21.15.114:6801/465333808] conn(0x7f35c40395b0 0x7f35c403ba70 secure :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0x7f35d0006fd0 tx=0x7f35d0006e40 comp rx=0 tx=0).stop 2024-04-23T10:17:50.678 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-04-23T10:17:49.950+0000 7f35e238c700 1 -- 172.21.15.114:0/1531179744 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f35dc106810 msgr2=0x7f35dc19c400 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T10:17:50.678 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-04-23T10:17:49.950+0000 7f35e238c700 1 --2- 172.21.15.114:0/1531179744 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f35dc106810 0x7f35dc19c400 secure :-1 s=READY pgs=73 cs=0 l=1 rev1=1 crypto rx=0x7f35cc0116e0 tx=0x7f35cc011aa0 comp rx=0 tx=0).stop 2024-04-23T10:17:50.678 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-04-23T10:17:49.950+0000 7f35e238c700 1 -- 172.21.15.114:0/1531179744 shutdown_connections 2024-04-23T10:17:50.678 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-04-23T10:17:49.950+0000 7f35e238c700 1 --2- 172.21.15.114:0/1531179744 >> [v2:172.21.15.114:6800/465333808,v1:172.21.15.114:6801/465333808] conn(0x7f35c40395b0 0x7f35c403ba70 unknown :-1 s=CLOSED pgs=8 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:17:50.678 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-04-23T10:17:49.950+0000 7f35e238c700 1 --2- 172.21.15.114:0/1531179744 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f35dc106810 0x7f35dc19c400 unknown :-1 s=CLOSED pgs=73 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:17:50.679 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-04-23T10:17:49.950+0000 7f35e238c700 1 -- 172.21.15.114:0/1531179744 >> 172.21.15.114:0/1531179744 conn(0x7f35dc102150 msgr2=0x7f35dc102c40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T10:17:50.679 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-04-23T10:17:49.951+0000 7f35e238c700 1 -- 172.21.15.114:0/1531179744 shutdown_connections 2024-04-23T10:17:50.679 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-04-23T10:17:49.951+0000 7f35e238c700 1 -- 172.21.15.114:0/1531179744 wait complete. 2024-04-23T10:17:50.679 INFO:teuthology.orchestra.run.smithi114.stdout:Fetching dashboard port number... 2024-04-23T10:17:51.254 INFO:journalctl@ceph.mon.a.smithi114.stdout:Apr 23 10:17:50 smithi114 ceph-mon[29255]: from='client.14166 -' entity='client.admin' cmd=[{"prefix": "dashboard ac-user-create", "username": "admin", "rolename": "administrator", "force_password": true, "pwd_update_required": true, "target": ["mon-mgr", ""]}]: dispatch 2024-04-23T10:17:51.254 INFO:journalctl@ceph.mon.a.smithi114.stdout:Apr 23 10:17:50 smithi114 ceph-mon[29255]: from='mgr.14152 172.21.15.114:0/1529706138' entity='mgr.x' 2024-04-23T10:17:52.051 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stdout 8443 2024-04-23T10:17:52.051 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-04-23T10:17:51.104+0000 7fed0fc69700 1 Processor -- start 2024-04-23T10:17:52.051 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-04-23T10:17:51.104+0000 7fed0fc69700 1 -- start start 2024-04-23T10:17:52.051 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-04-23T10:17:51.104+0000 7fed0fc69700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fed08108fb0 0x7fed0810b3a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T10:17:52.051 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-04-23T10:17:51.104+0000 7fed0fc69700 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x7fed08063d60 con 0x7fed08108fb0 2024-04-23T10:17:52.051 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-04-23T10:17:51.105+0000 7fed0da05700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fed08108fb0 0x7fed0810b3a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T10:17:52.051 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-04-23T10:17:51.105+0000 7fed0da05700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fed08108fb0 0x7fed0810b3a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.114:44824/0 (socket says 172.21.15.114:44824) 2024-04-23T10:17:52.051 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-04-23T10:17:51.105+0000 7fed0da05700 1 -- 172.21.15.114:0/2578102467 learned_addr learned my addr 172.21.15.114:0/2578102467 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-04-23T10:17:52.051 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-04-23T10:17:51.105+0000 7fed0da05700 1 -- 172.21.15.114:0/2578102467 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fed0810b8e0 con 0x7fed08108fb0 2024-04-23T10:17:52.051 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-04-23T10:17:51.105+0000 7fed0da05700 1 --2- 172.21.15.114:0/2578102467 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fed08108fb0 0x7fed0810b3a0 secure :-1 s=READY pgs=74 cs=0 l=1 rev1=1 crypto rx=0x7fed040098a0 tx=0x7fed04009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=46066139819c247 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T10:17:52.052 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-04-23T10:17:51.105+0000 7fed0ca03700 1 -- 172.21.15.114:0/2578102467 <== mon.0 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fed04004160 con 0x7fed08108fb0 2024-04-23T10:17:52.052 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-04-23T10:17:51.105+0000 7fed0ca03700 1 -- 172.21.15.114:0/2578102467 <== mon.0 v2:172.21.15.114:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fed04004300 con 0x7fed08108fb0 2024-04-23T10:17:52.052 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-04-23T10:17:51.105+0000 7fed0ca03700 1 -- 172.21.15.114:0/2578102467 <== mon.0 v2:172.21.15.114:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fed04011a20 con 0x7fed08108fb0 2024-04-23T10:17:52.052 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-04-23T10:17:51.106+0000 7fed0fc69700 1 -- 172.21.15.114:0/2578102467 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fed08108fb0 msgr2=0x7fed0810b3a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T10:17:52.052 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-04-23T10:17:51.106+0000 7fed0fc69700 1 --2- 172.21.15.114:0/2578102467 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fed08108fb0 0x7fed0810b3a0 secure :-1 s=READY pgs=74 cs=0 l=1 rev1=1 crypto rx=0x7fed040098a0 tx=0x7fed04009bb0 comp rx=0 tx=0).stop 2024-04-23T10:17:52.052 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-04-23T10:17:51.106+0000 7fed0fc69700 1 -- 172.21.15.114:0/2578102467 shutdown_connections 2024-04-23T10:17:52.052 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-04-23T10:17:51.106+0000 7fed0fc69700 1 --2- 172.21.15.114:0/2578102467 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fed08108fb0 0x7fed0810b3a0 unknown :-1 s=CLOSED pgs=74 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:17:52.052 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-04-23T10:17:51.106+0000 7fed0fc69700 1 -- 172.21.15.114:0/2578102467 >> 172.21.15.114:0/2578102467 conn(0x7fed08102a40 msgr2=0x7fed08104e60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T10:17:52.052 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-04-23T10:17:51.106+0000 7fed0fc69700 1 -- 172.21.15.114:0/2578102467 shutdown_connections 2024-04-23T10:17:52.052 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-04-23T10:17:51.106+0000 7fed0fc69700 1 -- 172.21.15.114:0/2578102467 wait complete. 2024-04-23T10:17:52.052 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-04-23T10:17:51.107+0000 7fed0fc69700 1 Processor -- start 2024-04-23T10:17:52.052 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-04-23T10:17:51.107+0000 7fed0fc69700 1 -- start start 2024-04-23T10:17:52.052 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-04-23T10:17:51.107+0000 7fed0fc69700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fed08108fb0 0x7fed0819c3b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T10:17:52.052 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-04-23T10:17:51.107+0000 7fed0fc69700 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x7fed0810c8f0 con 0x7fed08108fb0 2024-04-23T10:17:52.053 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-04-23T10:17:51.107+0000 7fed0da05700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fed08108fb0 0x7fed0819c3b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T10:17:52.053 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-04-23T10:17:51.107+0000 7fed0da05700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fed08108fb0 0x7fed0819c3b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.114:44836/0 (socket says 172.21.15.114:44836) 2024-04-23T10:17:52.053 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-04-23T10:17:51.107+0000 7fed0da05700 1 -- 172.21.15.114:0/4214499060 learned_addr learned my addr 172.21.15.114:0/4214499060 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-04-23T10:17:52.053 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-04-23T10:17:51.107+0000 7fed0da05700 1 -- 172.21.15.114:0/4214499060 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fed0819c8f0 con 0x7fed08108fb0 2024-04-23T10:17:52.053 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-04-23T10:17:51.107+0000 7fed0da05700 1 --2- 172.21.15.114:0/4214499060 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fed08108fb0 0x7fed0819c3b0 secure :-1 s=READY pgs=75 cs=0 l=1 rev1=1 crypto rx=0x7fed040116e0 tx=0x7fed04011aa0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T10:17:52.053 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-04-23T10:17:51.107+0000 7fecfeffd700 1 -- 172.21.15.114:0/4214499060 <== mon.0 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fed04003a60 con 0x7fed08108fb0 2024-04-23T10:17:52.053 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-04-23T10:17:51.107+0000 7fecfeffd700 1 -- 172.21.15.114:0/4214499060 <== mon.0 v2:172.21.15.114:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fed0401b8f0 con 0x7fed08108fb0 2024-04-23T10:17:52.053 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-04-23T10:17:51.107+0000 7fed0fc69700 1 -- 172.21.15.114:0/4214499060 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fed0819cb80 con 0x7fed08108fb0 2024-04-23T10:17:52.053 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-04-23T10:17:51.107+0000 7fecfeffd700 1 -- 172.21.15.114:0/4214499060 <== mon.0 v2:172.21.15.114:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fed04019860 con 0x7fed08108fb0 2024-04-23T10:17:52.053 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-04-23T10:17:51.108+0000 7fed0fc69700 1 -- 172.21.15.114:0/4214499060 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fed0819f870 con 0x7fed08108fb0 2024-04-23T10:17:52.053 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-04-23T10:17:51.108+0000 7fecfeffd700 1 -- 172.21.15.114:0/4214499060 <== mon.0 v2:172.21.15.114:3300/0 4 ==== mgrmap(e 12) ==== 46571+0+0 (secure 0 0 0) 0x7fed0401b440 con 0x7fed08108fb0 2024-04-23T10:17:52.053 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-04-23T10:17:51.108+0000 7fed0fc69700 1 -- 172.21.15.114:0/4214499060 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fed080640a0 con 0x7fed08108fb0 2024-04-23T10:17:52.053 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-04-23T10:17:51.109+0000 7fecfeffd700 1 --2- 172.21.15.114:0/4214499060 >> [v2:172.21.15.114:6800/465333808,v1:172.21.15.114:6801/465333808] conn(0x7fecf4039610 0x7fecf403bad0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T10:17:52.053 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-04-23T10:17:51.109+0000 7fecfeffd700 1 -- 172.21.15.114:0/4214499060 <== mon.0 v2:172.21.15.114:3300/0 5 ==== osd_map(3..3 src has 1..3) ==== 1155+0+0 (secure 0 0 0) 0x7fed0404ddd0 con 0x7fed08108fb0 2024-04-23T10:17:52.053 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-04-23T10:17:51.109+0000 7fed0d204700 1 --2- 172.21.15.114:0/4214499060 >> [v2:172.21.15.114:6800/465333808,v1:172.21.15.114:6801/465333808] conn(0x7fecf4039610 0x7fecf403bad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T10:17:52.054 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-04-23T10:17:51.109+0000 7fed0d204700 1 --2- 172.21.15.114:0/4214499060 >> [v2:172.21.15.114:6800/465333808,v1:172.21.15.114:6801/465333808] conn(0x7fecf4039610 0x7fecf403bad0 secure :-1 s=READY pgs=9 cs=0 l=1 rev1=1 crypto rx=0x7fecf8006fd0 tx=0x7fecf8006e40 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T10:17:52.054 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-04-23T10:17:51.114+0000 7fecfeffd700 1 -- 172.21.15.114:0/4214499060 <== mon.0 v2:172.21.15.114:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7fed04051030 con 0x7fed08108fb0 2024-04-23T10:17:52.054 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-04-23T10:17:51.318+0000 7fed0fc69700 1 -- 172.21.15.114:0/4214499060 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_command({"prefix": "config get", "who": "mgr", "key": "mgr/dashboard/ssl_server_port"} v 0) -- 0x7fed08107180 con 0x7fed08108fb0 2024-04-23T10:17:52.054 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-04-23T10:17:51.318+0000 7fecfeffd700 1 -- 172.21.15.114:0/4214499060 <== mon.0 v2:172.21.15.114:3300/0 7 ==== mon_command_ack([{"prefix": "config get", "who": "mgr", "key": "mgr/dashboard/ssl_server_port"}]=0 v8) ==== 112+0+5 (secure 0 0 0) 0x7fed0401f030 con 0x7fed08108fb0 2024-04-23T10:17:52.054 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-04-23T10:17:51.320+0000 7fed0fc69700 1 -- 172.21.15.114:0/4214499060 >> [v2:172.21.15.114:6800/465333808,v1:172.21.15.114:6801/465333808] conn(0x7fecf4039610 msgr2=0x7fecf403bad0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T10:17:52.054 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-04-23T10:17:51.320+0000 7fed0fc69700 1 --2- 172.21.15.114:0/4214499060 >> [v2:172.21.15.114:6800/465333808,v1:172.21.15.114:6801/465333808] conn(0x7fecf4039610 0x7fecf403bad0 secure :-1 s=READY pgs=9 cs=0 l=1 rev1=1 crypto rx=0x7fecf8006fd0 tx=0x7fecf8006e40 comp rx=0 tx=0).stop 2024-04-23T10:17:52.054 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-04-23T10:17:51.320+0000 7fed0fc69700 1 -- 172.21.15.114:0/4214499060 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fed08108fb0 msgr2=0x7fed0819c3b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T10:17:52.054 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-04-23T10:17:51.321+0000 7fed0fc69700 1 --2- 172.21.15.114:0/4214499060 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fed08108fb0 0x7fed0819c3b0 secure :-1 s=READY pgs=75 cs=0 l=1 rev1=1 crypto rx=0x7fed040116e0 tx=0x7fed04011aa0 comp rx=0 tx=0).stop 2024-04-23T10:17:52.054 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-04-23T10:17:51.321+0000 7fed0fc69700 1 -- 172.21.15.114:0/4214499060 shutdown_connections 2024-04-23T10:17:52.054 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-04-23T10:17:51.321+0000 7fed0fc69700 1 --2- 172.21.15.114:0/4214499060 >> [v2:172.21.15.114:6800/465333808,v1:172.21.15.114:6801/465333808] conn(0x7fecf4039610 0x7fecf403bad0 unknown :-1 s=CLOSED pgs=9 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:17:52.054 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-04-23T10:17:51.321+0000 7fed0fc69700 1 --2- 172.21.15.114:0/4214499060 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fed08108fb0 0x7fed0819c3b0 unknown :-1 s=CLOSED pgs=75 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:17:52.054 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-04-23T10:17:51.321+0000 7fed0fc69700 1 -- 172.21.15.114:0/4214499060 >> 172.21.15.114:0/4214499060 conn(0x7fed08102a40 msgr2=0x7fed08103490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T10:17:52.054 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-04-23T10:17:51.321+0000 7fed0fc69700 1 -- 172.21.15.114:0/4214499060 shutdown_connections 2024-04-23T10:17:52.054 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-04-23T10:17:51.321+0000 7fed0fc69700 1 -- 172.21.15.114:0/4214499060 wait complete. 2024-04-23T10:17:52.056 INFO:teuthology.orchestra.run.smithi114.stdout:Non-zero exit code 1 from systemctl is-enabled firewalld.service 2024-04-23T10:17:52.056 INFO:teuthology.orchestra.run.smithi114.stdout:systemctl: stdout disabled 2024-04-23T10:17:52.063 INFO:teuthology.orchestra.run.smithi114.stdout:Non-zero exit code 3 from systemctl is-active firewalld.service 2024-04-23T10:17:52.063 INFO:teuthology.orchestra.run.smithi114.stdout:systemctl: stdout inactive 2024-04-23T10:17:52.063 INFO:teuthology.orchestra.run.smithi114.stdout:firewalld.service is not enabled 2024-04-23T10:17:52.064 INFO:teuthology.orchestra.run.smithi114.stdout:Not possible to open ports <[8443]>. firewalld.service is not available 2024-04-23T10:17:52.065 INFO:teuthology.orchestra.run.smithi114.stdout:Ceph Dashboard is now available at: 2024-04-23T10:17:52.065 INFO:teuthology.orchestra.run.smithi114.stdout: 2024-04-23T10:17:52.065 INFO:teuthology.orchestra.run.smithi114.stdout: URL: https://smithi114:8443/ 2024-04-23T10:17:52.065 INFO:teuthology.orchestra.run.smithi114.stdout: User: admin 2024-04-23T10:17:52.065 INFO:teuthology.orchestra.run.smithi114.stdout: Password: pl0hdvli49 2024-04-23T10:17:52.065 INFO:teuthology.orchestra.run.smithi114.stdout: 2024-04-23T10:17:52.066 INFO:teuthology.orchestra.run.smithi114.stdout:Saving cluster configuration to /var/lib/ceph/aa240572-0159-11ef-bc93-c7b262605968/config directory 2024-04-23T10:17:52.066 INFO:teuthology.orchestra.run.smithi114.stdout:Enabling autotune for osd_memory_target 2024-04-23T10:17:52.187 INFO:journalctl@ceph.mon.a.smithi114.stdout:Apr 23 10:17:51 smithi114 ceph-mon[29255]: from='client.? 172.21.15.114:0/4214499060' entity='client.admin' cmd=[{"prefix": "config get", "who": "mgr", "key": "mgr/dashboard/ssl_server_port"}]: dispatch 2024-04-23T10:17:53.254 INFO:journalctl@ceph.mon.a.smithi114.stdout:Apr 23 10:17:52 smithi114 ceph-mon[29255]: mgrmap e13: x(active, since 6s) 2024-04-23T10:17:53.299 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-04-23T10:17:52.489+0000 7fe00c831700 1 Processor -- start 2024-04-23T10:17:53.299 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-04-23T10:17:52.489+0000 7fe00c831700 1 -- start start 2024-04-23T10:17:53.299 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-04-23T10:17:52.489+0000 7fe00c831700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fe008108530 0x7fe008108950 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T10:17:53.299 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-04-23T10:17:52.489+0000 7fe00c831700 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x7fe008108f20 con 0x7fe008108530 2024-04-23T10:17:53.299 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-04-23T10:17:52.489+0000 7fe00659c700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fe008108530 0x7fe008108950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T10:17:53.299 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-04-23T10:17:52.489+0000 7fe00659c700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fe008108530 0x7fe008108950 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.114:44848/0 (socket says 172.21.15.114:44848) 2024-04-23T10:17:53.299 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-04-23T10:17:52.489+0000 7fe00659c700 1 -- 172.21.15.114:0/606510515 learned_addr learned my addr 172.21.15.114:0/606510515 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-04-23T10:17:53.299 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-04-23T10:17:52.490+0000 7fe00659c700 1 -- 172.21.15.114:0/606510515 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe0081096e0 con 0x7fe008108530 2024-04-23T10:17:53.299 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-04-23T10:17:52.490+0000 7fe00659c700 1 --2- 172.21.15.114:0/606510515 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fe008108530 0x7fe008108950 secure :-1 s=READY pgs=76 cs=0 l=1 rev1=1 crypto rx=0x7fdff80098a0 tx=0x7fdff8009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=cc006feb3168fcac server_cookie=0 in_seq=0 out_seq=0 2024-04-23T10:17:53.299 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-04-23T10:17:52.490+0000 7fe00559a700 1 -- 172.21.15.114:0/606510515 <== mon.0 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fdff8004160 con 0x7fe008108530 2024-04-23T10:17:53.300 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-04-23T10:17:52.490+0000 7fe00559a700 1 -- 172.21.15.114:0/606510515 <== mon.0 v2:172.21.15.114:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fdff8004300 con 0x7fe008108530 2024-04-23T10:17:53.300 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-04-23T10:17:52.490+0000 7fe00559a700 1 -- 172.21.15.114:0/606510515 <== mon.0 v2:172.21.15.114:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fdff8011a20 con 0x7fe008108530 2024-04-23T10:17:53.300 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-04-23T10:17:52.491+0000 7fe00c831700 1 -- 172.21.15.114:0/606510515 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fe008108530 msgr2=0x7fe008108950 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T10:17:53.300 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-04-23T10:17:52.491+0000 7fe00c831700 1 --2- 172.21.15.114:0/606510515 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fe008108530 0x7fe008108950 secure :-1 s=READY pgs=76 cs=0 l=1 rev1=1 crypto rx=0x7fdff80098a0 tx=0x7fdff8009bb0 comp rx=0 tx=0).stop 2024-04-23T10:17:53.300 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-04-23T10:17:52.491+0000 7fe00c831700 1 -- 172.21.15.114:0/606510515 shutdown_connections 2024-04-23T10:17:53.300 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-04-23T10:17:52.491+0000 7fe00c831700 1 --2- 172.21.15.114:0/606510515 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fe008108530 0x7fe008108950 unknown :-1 s=CLOSED pgs=76 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:17:53.300 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-04-23T10:17:52.491+0000 7fe00c831700 1 -- 172.21.15.114:0/606510515 >> 172.21.15.114:0/606510515 conn(0x7fe008103ab0 msgr2=0x7fe008105f10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T10:17:53.300 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-04-23T10:17:52.491+0000 7fe00c831700 1 -- 172.21.15.114:0/606510515 shutdown_connections 2024-04-23T10:17:53.300 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-04-23T10:17:52.491+0000 7fe00c831700 1 -- 172.21.15.114:0/606510515 wait complete. 2024-04-23T10:17:53.300 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-04-23T10:17:52.491+0000 7fe00c831700 1 Processor -- start 2024-04-23T10:17:53.300 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-04-23T10:17:52.491+0000 7fe00c831700 1 -- start start 2024-04-23T10:17:53.300 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-04-23T10:17:52.492+0000 7fe00c831700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fe008108530 0x7fe00819c470 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T10:17:53.300 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-04-23T10:17:52.492+0000 7fe00c831700 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x7fe00810ca90 con 0x7fe008108530 2024-04-23T10:17:53.300 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-04-23T10:17:52.492+0000 7fe00659c700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fe008108530 0x7fe00819c470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T10:17:53.300 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-04-23T10:17:52.492+0000 7fe00659c700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fe008108530 0x7fe00819c470 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.114:44864/0 (socket says 172.21.15.114:44864) 2024-04-23T10:17:53.301 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-04-23T10:17:52.492+0000 7fe00659c700 1 -- 172.21.15.114:0/2889751091 learned_addr learned my addr 172.21.15.114:0/2889751091 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-04-23T10:17:53.301 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-04-23T10:17:52.492+0000 7fe00659c700 1 -- 172.21.15.114:0/2889751091 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe00819c9b0 con 0x7fe008108530 2024-04-23T10:17:53.301 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-04-23T10:17:52.492+0000 7fe00659c700 1 --2- 172.21.15.114:0/2889751091 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fe008108530 0x7fe00819c470 secure :-1 s=READY pgs=77 cs=0 l=1 rev1=1 crypto rx=0x7fdff8000c00 tx=0x7fdff8011aa0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T10:17:53.301 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-04-23T10:17:52.492+0000 7fdff77fe700 1 -- 172.21.15.114:0/2889751091 <== mon.0 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fdff8003aa0 con 0x7fe008108530 2024-04-23T10:17:53.301 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-04-23T10:17:52.492+0000 7fdff77fe700 1 -- 172.21.15.114:0/2889751091 <== mon.0 v2:172.21.15.114:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fdff801b8f0 con 0x7fe008108530 2024-04-23T10:17:53.301 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-04-23T10:17:52.492+0000 7fe00c831700 1 -- 172.21.15.114:0/2889751091 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fe00819cc40 con 0x7fe008108530 2024-04-23T10:17:53.301 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-04-23T10:17:52.492+0000 7fdff77fe700 1 -- 172.21.15.114:0/2889751091 <== mon.0 v2:172.21.15.114:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fdff8023890 con 0x7fe008108530 2024-04-23T10:17:53.301 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-04-23T10:17:52.492+0000 7fe00c831700 1 -- 172.21.15.114:0/2889751091 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fe00819f930 con 0x7fe008108530 2024-04-23T10:17:53.301 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-04-23T10:17:52.493+0000 7fdff77fe700 1 -- 172.21.15.114:0/2889751091 <== mon.0 v2:172.21.15.114:3300/0 4 ==== mgrmap(e 13) ==== 46615+0+0 (secure 0 0 0) 0x7fdff802c3f0 con 0x7fe008108530 2024-04-23T10:17:53.301 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-04-23T10:17:52.493+0000 7fdff77fe700 1 --2- 172.21.15.114:0/2889751091 >> [v2:172.21.15.114:6800/465333808,v1:172.21.15.114:6801/465333808] conn(0x7fdff00396a0 0x7fdff003bb60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T10:17:53.301 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-04-23T10:17:52.493+0000 7fdff77fe700 1 -- 172.21.15.114:0/2889751091 <== mon.0 v2:172.21.15.114:3300/0 5 ==== osd_map(3..3 src has 1..3) ==== 1155+0+0 (secure 0 0 0) 0x7fdff804e370 con 0x7fe008108530 2024-04-23T10:17:53.301 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-04-23T10:17:52.493+0000 7fe005d9b700 1 --2- 172.21.15.114:0/2889751091 >> [v2:172.21.15.114:6800/465333808,v1:172.21.15.114:6801/465333808] conn(0x7fdff00396a0 0x7fdff003bb60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T10:17:53.301 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-04-23T10:17:52.493+0000 7fe00c831700 1 -- 172.21.15.114:0/2889751091 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fe0080640a0 con 0x7fe008108530 2024-04-23T10:17:53.301 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-04-23T10:17:52.494+0000 7fe005d9b700 1 --2- 172.21.15.114:0/2889751091 >> [v2:172.21.15.114:6800/465333808,v1:172.21.15.114:6801/465333808] conn(0x7fdff00396a0 0x7fdff003bb60 secure :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0x7fdffc006fd0 tx=0x7fdffc006e40 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T10:17:53.302 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-04-23T10:17:52.498+0000 7fdff77fe700 1 -- 172.21.15.114:0/2889751091 <== mon.0 v2:172.21.15.114:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7fdff801f030 con 0x7fe008108530 2024-04-23T10:17:53.302 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-04-23T10:17:52.696+0000 7fe00c831700 1 -- 172.21.15.114:0/2889751091 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_command([{prefix=config set, name=osd_memory_target_autotune}] v 0) -- 0x7fe00819fdc0 con 0x7fe008108530 2024-04-23T10:17:53.302 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-04-23T10:17:52.697+0000 7fdff77fe700 1 -- 172.21.15.114:0/2889751091 <== mon.0 v2:172.21.15.114:3300/0 7 ==== mon_command_ack([{prefix=config set, name=osd_memory_target_autotune}]=0 v8)=0 v8) ==== 127+0+0 (secure 0 0 0) 0x7fdff80105e0 con 0x7fe008108530 2024-04-23T10:17:53.302 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-04-23T10:17:52.698+0000 7fe00c831700 1 -- 172.21.15.114:0/2889751091 >> [v2:172.21.15.114:6800/465333808,v1:172.21.15.114:6801/465333808] conn(0x7fdff00396a0 msgr2=0x7fdff003bb60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T10:17:53.302 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-04-23T10:17:52.698+0000 7fe00c831700 1 --2- 172.21.15.114:0/2889751091 >> [v2:172.21.15.114:6800/465333808,v1:172.21.15.114:6801/465333808] conn(0x7fdff00396a0 0x7fdff003bb60 secure :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0x7fdffc006fd0 tx=0x7fdffc006e40 comp rx=0 tx=0).stop 2024-04-23T10:17:53.302 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-04-23T10:17:52.699+0000 7fe00c831700 1 -- 172.21.15.114:0/2889751091 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fe008108530 msgr2=0x7fe00819c470 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T10:17:53.302 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-04-23T10:17:52.699+0000 7fe00c831700 1 --2- 172.21.15.114:0/2889751091 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fe008108530 0x7fe00819c470 secure :-1 s=READY pgs=77 cs=0 l=1 rev1=1 crypto rx=0x7fdff8000c00 tx=0x7fdff8011aa0 comp rx=0 tx=0).stop 2024-04-23T10:17:53.302 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-04-23T10:17:52.699+0000 7fe00c831700 1 -- 172.21.15.114:0/2889751091 shutdown_connections 2024-04-23T10:17:53.302 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-04-23T10:17:52.699+0000 7fe00c831700 1 --2- 172.21.15.114:0/2889751091 >> [v2:172.21.15.114:6800/465333808,v1:172.21.15.114:6801/465333808] conn(0x7fdff00396a0 0x7fdff003bb60 unknown :-1 s=CLOSED pgs=10 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:17:53.302 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-04-23T10:17:52.699+0000 7fe00c831700 1 --2- 172.21.15.114:0/2889751091 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fe008108530 0x7fe00819c470 unknown :-1 s=CLOSED pgs=77 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:17:53.302 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-04-23T10:17:52.699+0000 7fe00c831700 1 -- 172.21.15.114:0/2889751091 >> 172.21.15.114:0/2889751091 conn(0x7fe008103ab0 msgr2=0x7fe008104650 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T10:17:53.302 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-04-23T10:17:52.699+0000 7fe00c831700 1 -- 172.21.15.114:0/2889751091 shutdown_connections 2024-04-23T10:17:53.302 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-04-23T10:17:52.699+0000 7fe00c831700 1 -- 172.21.15.114:0/2889751091 wait complete. 2024-04-23T10:17:54.589 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-04-23T10:17:53.706+0000 7f60f07a2700 1 Processor -- start 2024-04-23T10:17:54.589 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-04-23T10:17:53.706+0000 7f60f07a2700 1 -- start start 2024-04-23T10:17:54.589 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-04-23T10:17:53.706+0000 7f60f07a2700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f60e8107440 0x7f60e8107860 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T10:17:54.589 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-04-23T10:17:53.706+0000 7f60f07a2700 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x7f60e8107e30 con 0x7f60e8107440 2024-04-23T10:17:54.590 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-04-23T10:17:53.706+0000 7f60ee53e700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f60e8107440 0x7f60e8107860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T10:17:54.590 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-04-23T10:17:53.706+0000 7f60ee53e700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f60e8107440 0x7f60e8107860 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.114:44866/0 (socket says 172.21.15.114:44866) 2024-04-23T10:17:54.590 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-04-23T10:17:53.706+0000 7f60ee53e700 1 -- 172.21.15.114:0/2269396785 learned_addr learned my addr 172.21.15.114:0/2269396785 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-04-23T10:17:54.590 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-04-23T10:17:53.706+0000 7f60ee53e700 1 -- 172.21.15.114:0/2269396785 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f60e8108640 con 0x7f60e8107440 2024-04-23T10:17:54.590 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-04-23T10:17:53.706+0000 7f60ee53e700 1 --2- 172.21.15.114:0/2269396785 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f60e8107440 0x7f60e8107860 secure :-1 s=READY pgs=78 cs=0 l=1 rev1=1 crypto rx=0x7f60d80098a0 tx=0x7f60d8009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c7595fd1dbe14407 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T10:17:54.590 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-04-23T10:17:53.707+0000 7f60ed53c700 1 -- 172.21.15.114:0/2269396785 <== mon.0 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f60d8004160 con 0x7f60e8107440 2024-04-23T10:17:54.590 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-04-23T10:17:53.707+0000 7f60ed53c700 1 -- 172.21.15.114:0/2269396785 <== mon.0 v2:172.21.15.114:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f60d8004300 con 0x7f60e8107440 2024-04-23T10:17:54.590 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-04-23T10:17:53.707+0000 7f60ed53c700 1 -- 172.21.15.114:0/2269396785 <== mon.0 v2:172.21.15.114:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f60d8011a20 con 0x7f60e8107440 2024-04-23T10:17:54.590 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-04-23T10:17:53.707+0000 7f60f07a2700 1 -- 172.21.15.114:0/2269396785 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f60e8107440 msgr2=0x7f60e8107860 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T10:17:54.590 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-04-23T10:17:53.707+0000 7f60f07a2700 1 --2- 172.21.15.114:0/2269396785 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f60e8107440 0x7f60e8107860 secure :-1 s=READY pgs=78 cs=0 l=1 rev1=1 crypto rx=0x7f60d80098a0 tx=0x7f60d8009bb0 comp rx=0 tx=0).stop 2024-04-23T10:17:54.590 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-04-23T10:17:53.707+0000 7f60f07a2700 1 -- 172.21.15.114:0/2269396785 shutdown_connections 2024-04-23T10:17:54.590 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-04-23T10:17:53.707+0000 7f60f07a2700 1 --2- 172.21.15.114:0/2269396785 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f60e8107440 0x7f60e8107860 unknown :-1 s=CLOSED pgs=78 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:17:54.590 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-04-23T10:17:53.707+0000 7f60f07a2700 1 -- 172.21.15.114:0/2269396785 >> 172.21.15.114:0/2269396785 conn(0x7f60e81029e0 msgr2=0x7f60e8104e20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T10:17:54.591 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-04-23T10:17:53.707+0000 7f60f07a2700 1 -- 172.21.15.114:0/2269396785 shutdown_connections 2024-04-23T10:17:54.591 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-04-23T10:17:53.707+0000 7f60f07a2700 1 -- 172.21.15.114:0/2269396785 wait complete. 2024-04-23T10:17:54.591 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-04-23T10:17:53.708+0000 7f60f07a2700 1 Processor -- start 2024-04-23T10:17:54.591 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-04-23T10:17:53.708+0000 7f60f07a2700 1 -- start start 2024-04-23T10:17:54.591 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-04-23T10:17:53.708+0000 7f60f07a2700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f60e8107440 0x7f60e8192560 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T10:17:54.591 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-04-23T10:17:53.708+0000 7f60f07a2700 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x7f60e810b9f0 con 0x7f60e8107440 2024-04-23T10:17:54.591 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-04-23T10:17:53.708+0000 7f60ee53e700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f60e8107440 0x7f60e8192560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T10:17:54.591 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-04-23T10:17:53.708+0000 7f60ee53e700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f60e8107440 0x7f60e8192560 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.114:44882/0 (socket says 172.21.15.114:44882) 2024-04-23T10:17:54.591 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-04-23T10:17:53.708+0000 7f60ee53e700 1 -- 172.21.15.114:0/3546529694 learned_addr learned my addr 172.21.15.114:0/3546529694 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-04-23T10:17:54.591 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-04-23T10:17:53.708+0000 7f60ee53e700 1 -- 172.21.15.114:0/3546529694 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f60e8192aa0 con 0x7f60e8107440 2024-04-23T10:17:54.591 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-04-23T10:17:53.709+0000 7f60ee53e700 1 --2- 172.21.15.114:0/3546529694 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f60e8107440 0x7f60e8192560 secure :-1 s=READY pgs=79 cs=0 l=1 rev1=1 crypto rx=0x7f60d80119f0 tx=0x7f60d8011ad0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T10:17:54.591 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-04-23T10:17:53.709+0000 7f60df7fe700 1 -- 172.21.15.114:0/3546529694 <== mon.0 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f60d80039e0 con 0x7f60e8107440 2024-04-23T10:17:54.591 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-04-23T10:17:53.709+0000 7f60df7fe700 1 -- 172.21.15.114:0/3546529694 <== mon.0 v2:172.21.15.114:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f60d801b8f0 con 0x7f60e8107440 2024-04-23T10:17:54.591 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-04-23T10:17:53.709+0000 7f60df7fe700 1 -- 172.21.15.114:0/3546529694 <== mon.0 v2:172.21.15.114:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f60d80237a0 con 0x7f60e8107440 2024-04-23T10:17:54.591 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-04-23T10:17:53.709+0000 7f60f07a2700 1 -- 172.21.15.114:0/3546529694 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f60e8190ce0 con 0x7f60e8107440 2024-04-23T10:17:54.592 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-04-23T10:17:53.709+0000 7f60f07a2700 1 -- 172.21.15.114:0/3546529694 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f60e81911c0 con 0x7f60e8107440 2024-04-23T10:17:54.592 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-04-23T10:17:53.710+0000 7f60df7fe700 1 -- 172.21.15.114:0/3546529694 <== mon.0 v2:172.21.15.114:3300/0 4 ==== mgrmap(e 13) ==== 46615+0+0 (secure 0 0 0) 0x7f60d801b440 con 0x7f60e8107440 2024-04-23T10:17:54.592 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-04-23T10:17:53.710+0000 7f60f07a2700 1 -- 172.21.15.114:0/3546529694 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f60e80640a0 con 0x7f60e8107440 2024-04-23T10:17:54.592 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-04-23T10:17:53.710+0000 7f60df7fe700 1 --2- 172.21.15.114:0/3546529694 >> [v2:172.21.15.114:6800/465333808,v1:172.21.15.114:6801/465333808] conn(0x7f60d40396b0 0x7f60d403bb70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T10:17:54.592 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-04-23T10:17:53.710+0000 7f60df7fe700 1 -- 172.21.15.114:0/3546529694 <== mon.0 v2:172.21.15.114:3300/0 5 ==== osd_map(3..3 src has 1..3) ==== 1155+0+0 (secure 0 0 0) 0x7f60d804dfd0 con 0x7f60e8107440 2024-04-23T10:17:54.592 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-04-23T10:17:53.710+0000 7f60edd3d700 1 --2- 172.21.15.114:0/3546529694 >> [v2:172.21.15.114:6800/465333808,v1:172.21.15.114:6801/465333808] conn(0x7f60d40396b0 0x7f60d403bb70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T10:17:54.592 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-04-23T10:17:53.710+0000 7f60edd3d700 1 --2- 172.21.15.114:0/3546529694 >> [v2:172.21.15.114:6800/465333808,v1:172.21.15.114:6801/465333808] conn(0x7f60d40396b0 0x7f60d403bb70 secure :-1 s=READY pgs=11 cs=0 l=1 rev1=1 crypto rx=0x7f60e4006fd0 tx=0x7f60e4006e40 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T10:17:54.592 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-04-23T10:17:53.715+0000 7f60df7fe700 1 -- 172.21.15.114:0/3546529694 <== mon.0 v2:172.21.15.114:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f60d801f030 con 0x7f60e8107440 2024-04-23T10:17:54.592 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-04-23T10:17:54.030+0000 7f60f07a2700 1 -- 172.21.15.114:0/3546529694 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_command([{prefix=config-key set, key=mgr/dashboard/cluster/status}] v 0) -- 0x7f60e8196290 con 0x7f60e8107440 2024-04-23T10:17:54.592 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-04-23T10:17:54.032+0000 7f60df7fe700 1 -- 172.21.15.114:0/3546529694 <== mon.0 v2:172.21.15.114:3300/0 7 ==== mon_command_ack([{prefix=config-key set, key=mgr/dashboard/cluster/status}]=0 set mgr/dashboard/cluster/status v27)=0 set mgr/dashboard/cluster/status v27) ==== 153+0+0 (secure 0 0 0) 0x7f60d8016260 con 0x7f60e8107440 2024-04-23T10:17:54.592 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-04-23T10:17:54.034+0000 7f60f07a2700 1 -- 172.21.15.114:0/3546529694 >> [v2:172.21.15.114:6800/465333808,v1:172.21.15.114:6801/465333808] conn(0x7f60d40396b0 msgr2=0x7f60d403bb70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T10:17:54.592 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-04-23T10:17:54.034+0000 7f60f07a2700 1 --2- 172.21.15.114:0/3546529694 >> [v2:172.21.15.114:6800/465333808,v1:172.21.15.114:6801/465333808] conn(0x7f60d40396b0 0x7f60d403bb70 secure :-1 s=READY pgs=11 cs=0 l=1 rev1=1 crypto rx=0x7f60e4006fd0 tx=0x7f60e4006e40 comp rx=0 tx=0).stop 2024-04-23T10:17:54.592 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-04-23T10:17:54.034+0000 7f60f07a2700 1 -- 172.21.15.114:0/3546529694 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f60e8107440 msgr2=0x7f60e8192560 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T10:17:54.592 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-04-23T10:17:54.034+0000 7f60f07a2700 1 --2- 172.21.15.114:0/3546529694 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f60e8107440 0x7f60e8192560 secure :-1 s=READY pgs=79 cs=0 l=1 rev1=1 crypto rx=0x7f60d80119f0 tx=0x7f60d8011ad0 comp rx=0 tx=0).stop 2024-04-23T10:17:54.593 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-04-23T10:17:54.034+0000 7f60f07a2700 1 -- 172.21.15.114:0/3546529694 shutdown_connections 2024-04-23T10:17:54.593 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-04-23T10:17:54.034+0000 7f60f07a2700 1 --2- 172.21.15.114:0/3546529694 >> [v2:172.21.15.114:6800/465333808,v1:172.21.15.114:6801/465333808] conn(0x7f60d40396b0 0x7f60d403bb70 unknown :-1 s=CLOSED pgs=11 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:17:54.593 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-04-23T10:17:54.034+0000 7f60f07a2700 1 --2- 172.21.15.114:0/3546529694 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f60e8107440 0x7f60e8192560 unknown :-1 s=CLOSED pgs=79 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:17:54.593 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-04-23T10:17:54.034+0000 7f60f07a2700 1 -- 172.21.15.114:0/3546529694 >> 172.21.15.114:0/3546529694 conn(0x7f60e81029e0 msgr2=0x7f60e81038a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T10:17:54.593 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-04-23T10:17:54.034+0000 7f60f07a2700 1 -- 172.21.15.114:0/3546529694 shutdown_connections 2024-04-23T10:17:54.593 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr 2024-04-23T10:17:54.034+0000 7f60f07a2700 1 -- 172.21.15.114:0/3546529694 wait complete. 2024-04-23T10:17:54.593 INFO:teuthology.orchestra.run.smithi114.stdout:/usr/bin/ceph: stderr set mgr/dashboard/cluster/status 2024-04-23T10:17:54.593 INFO:teuthology.orchestra.run.smithi114.stdout:You can access the Ceph CLI as following in case of multi-cluster or non-default config: 2024-04-23T10:17:54.593 INFO:teuthology.orchestra.run.smithi114.stdout: 2024-04-23T10:17:54.593 INFO:teuthology.orchestra.run.smithi114.stdout: sudo /home/ubuntu/cephtest/cephadm shell --fsid aa240572-0159-11ef-bc93-c7b262605968 -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring 2024-04-23T10:17:54.593 INFO:teuthology.orchestra.run.smithi114.stdout: 2024-04-23T10:17:54.593 INFO:teuthology.orchestra.run.smithi114.stdout:Or, if you are only running a single cluster on this host: 2024-04-23T10:17:54.593 INFO:teuthology.orchestra.run.smithi114.stdout: 2024-04-23T10:17:54.593 INFO:teuthology.orchestra.run.smithi114.stdout: sudo /home/ubuntu/cephtest/cephadm shell 2024-04-23T10:17:54.593 INFO:teuthology.orchestra.run.smithi114.stdout: 2024-04-23T10:17:54.593 INFO:teuthology.orchestra.run.smithi114.stdout:Please consider enabling telemetry to help improve Ceph: 2024-04-23T10:17:54.593 INFO:teuthology.orchestra.run.smithi114.stdout: 2024-04-23T10:17:54.594 INFO:teuthology.orchestra.run.smithi114.stdout: ceph telemetry on 2024-04-23T10:17:54.594 INFO:teuthology.orchestra.run.smithi114.stdout: 2024-04-23T10:17:54.594 INFO:teuthology.orchestra.run.smithi114.stdout:For more information see: 2024-04-23T10:17:54.594 INFO:teuthology.orchestra.run.smithi114.stdout: 2024-04-23T10:17:54.594 INFO:teuthology.orchestra.run.smithi114.stdout: https://docs.ceph.com/en/latest/mgr/telemetry/ 2024-04-23T10:17:54.594 INFO:teuthology.orchestra.run.smithi114.stdout: 2024-04-23T10:17:54.594 INFO:teuthology.orchestra.run.smithi114.stdout:Bootstrap complete. 2024-04-23T10:17:54.622 INFO:tasks.cephadm:Fetching config... 2024-04-23T10:17:54.623 DEBUG:teuthology.orchestra.run.smithi114:> set -ex 2024-04-23T10:17:54.623 DEBUG:teuthology.orchestra.run.smithi114:> dd if=/etc/ceph/ceph.conf of=/dev/stdout 2024-04-23T10:17:54.638 INFO:tasks.cephadm:Fetching client.admin keyring... 2024-04-23T10:17:54.638 DEBUG:teuthology.orchestra.run.smithi114:> set -ex 2024-04-23T10:17:54.638 DEBUG:teuthology.orchestra.run.smithi114:> dd if=/etc/ceph/ceph.client.admin.keyring of=/dev/stdout 2024-04-23T10:17:54.695 INFO:tasks.cephadm:Fetching mon keyring... 2024-04-23T10:17:54.695 DEBUG:teuthology.orchestra.run.smithi114:> set -ex 2024-04-23T10:17:54.695 DEBUG:teuthology.orchestra.run.smithi114:> sudo dd if=/var/lib/ceph/aa240572-0159-11ef-bc93-c7b262605968/mon.a/keyring of=/dev/stdout 2024-04-23T10:17:54.759 INFO:tasks.cephadm:Fetching pub ssh key... 2024-04-23T10:17:54.759 DEBUG:teuthology.orchestra.run.smithi114:> set -ex 2024-04-23T10:17:54.759 DEBUG:teuthology.orchestra.run.smithi114:> dd if=/home/ubuntu/cephtest/ceph.pub of=/dev/stdout 2024-04-23T10:17:54.813 INFO:tasks.cephadm:Installing pub ssh key for root users... 2024-04-23T10:17:54.814 DEBUG:teuthology.orchestra.run.smithi114:> sudo install -d -m 0700 /root/.ssh && echo 'ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQC96O2tfq4F3ru+DL6BARr7cIMU2/U8e/M6SPi7vN3KakfqSi61xF0qopAsZD4nJpuRU4W+DbVrMHS+bHJ1s8ghaepC+fY+ibcriLsFcTooLk93RHYvxqSsqqJlqmoy6n9LaMx5RQrZkocXjNwj2FKHzV+xgfLZ/hEjBJPTBAl/Vg/JnfK/CA67iOeSKYWuxkARJd8zE2ePtyD9WsNmPkwccG3Q06xHl5ZzTlLNJNXDW3wgaLenxwliglHqYOmLC5jeUcjdYCJ2Xic/vnu7U8vjT0kKYrcD+BR1+ZPrem5sVV/9tVt7VS2ZnBBCo2NAWYblduMh6+M8dGxeSUxNOZ+RDjSVzsX4qHEjOBOEmaaRZOhqDge5Yqs3/wxJbalG7yvukS8uz9c2LH60vOT+2H4lBL0D/p49mvCn1OV4iEVo4EV/AgYaYE7slWP8vGhnZiX5LrzvThwslpM33hDyDZevowryFh981TVDsdIovZY4+NomFjIAzdl85EWWoWqqxcE= ceph-aa240572-0159-11ef-bc93-c7b262605968' | sudo tee -a /root/.ssh/authorized_keys && sudo chmod 0600 /root/.ssh/authorized_keys 2024-04-23T10:17:54.885 INFO:teuthology.orchestra.run.smithi114.stdout:ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQC96O2tfq4F3ru+DL6BARr7cIMU2/U8e/M6SPi7vN3KakfqSi61xF0qopAsZD4nJpuRU4W+DbVrMHS+bHJ1s8ghaepC+fY+ibcriLsFcTooLk93RHYvxqSsqqJlqmoy6n9LaMx5RQrZkocXjNwj2FKHzV+xgfLZ/hEjBJPTBAl/Vg/JnfK/CA67iOeSKYWuxkARJd8zE2ePtyD9WsNmPkwccG3Q06xHl5ZzTlLNJNXDW3wgaLenxwliglHqYOmLC5jeUcjdYCJ2Xic/vnu7U8vjT0kKYrcD+BR1+ZPrem5sVV/9tVt7VS2ZnBBCo2NAWYblduMh6+M8dGxeSUxNOZ+RDjSVzsX4qHEjOBOEmaaRZOhqDge5Yqs3/wxJbalG7yvukS8uz9c2LH60vOT+2H4lBL0D/p49mvCn1OV4iEVo4EV/AgYaYE7slWP8vGhnZiX5LrzvThwslpM33hDyDZevowryFh981TVDsdIovZY4+NomFjIAzdl85EWWoWqqxcE= ceph-aa240572-0159-11ef-bc93-c7b262605968 2024-04-23T10:17:54.895 DEBUG:teuthology.orchestra.run.smithi119:> sudo install -d -m 0700 /root/.ssh && echo 'ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQC96O2tfq4F3ru+DL6BARr7cIMU2/U8e/M6SPi7vN3KakfqSi61xF0qopAsZD4nJpuRU4W+DbVrMHS+bHJ1s8ghaepC+fY+ibcriLsFcTooLk93RHYvxqSsqqJlqmoy6n9LaMx5RQrZkocXjNwj2FKHzV+xgfLZ/hEjBJPTBAl/Vg/JnfK/CA67iOeSKYWuxkARJd8zE2ePtyD9WsNmPkwccG3Q06xHl5ZzTlLNJNXDW3wgaLenxwliglHqYOmLC5jeUcjdYCJ2Xic/vnu7U8vjT0kKYrcD+BR1+ZPrem5sVV/9tVt7VS2ZnBBCo2NAWYblduMh6+M8dGxeSUxNOZ+RDjSVzsX4qHEjOBOEmaaRZOhqDge5Yqs3/wxJbalG7yvukS8uz9c2LH60vOT+2H4lBL0D/p49mvCn1OV4iEVo4EV/AgYaYE7slWP8vGhnZiX5LrzvThwslpM33hDyDZevowryFh981TVDsdIovZY4+NomFjIAzdl85EWWoWqqxcE= ceph-aa240572-0159-11ef-bc93-c7b262605968' | sudo tee -a /root/.ssh/authorized_keys && sudo chmod 0600 /root/.ssh/authorized_keys 2024-04-23T10:17:54.935 INFO:teuthology.orchestra.run.smithi119.stdout:ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQC96O2tfq4F3ru+DL6BARr7cIMU2/U8e/M6SPi7vN3KakfqSi61xF0qopAsZD4nJpuRU4W+DbVrMHS+bHJ1s8ghaepC+fY+ibcriLsFcTooLk93RHYvxqSsqqJlqmoy6n9LaMx5RQrZkocXjNwj2FKHzV+xgfLZ/hEjBJPTBAl/Vg/JnfK/CA67iOeSKYWuxkARJd8zE2ePtyD9WsNmPkwccG3Q06xHl5ZzTlLNJNXDW3wgaLenxwliglHqYOmLC5jeUcjdYCJ2Xic/vnu7U8vjT0kKYrcD+BR1+ZPrem5sVV/9tVt7VS2ZnBBCo2NAWYblduMh6+M8dGxeSUxNOZ+RDjSVzsX4qHEjOBOEmaaRZOhqDge5Yqs3/wxJbalG7yvukS8uz9c2LH60vOT+2H4lBL0D/p49mvCn1OV4iEVo4EV/AgYaYE7slWP8vGhnZiX5LrzvThwslpM33hDyDZevowryFh981TVDsdIovZY4+NomFjIAzdl85EWWoWqqxcE= ceph-aa240572-0159-11ef-bc93-c7b262605968 2024-04-23T10:17:54.947 DEBUG:teuthology.orchestra.run.smithi204:> sudo install -d -m 0700 /root/.ssh && echo 'ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQC96O2tfq4F3ru+DL6BARr7cIMU2/U8e/M6SPi7vN3KakfqSi61xF0qopAsZD4nJpuRU4W+DbVrMHS+bHJ1s8ghaepC+fY+ibcriLsFcTooLk93RHYvxqSsqqJlqmoy6n9LaMx5RQrZkocXjNwj2FKHzV+xgfLZ/hEjBJPTBAl/Vg/JnfK/CA67iOeSKYWuxkARJd8zE2ePtyD9WsNmPkwccG3Q06xHl5ZzTlLNJNXDW3wgaLenxwliglHqYOmLC5jeUcjdYCJ2Xic/vnu7U8vjT0kKYrcD+BR1+ZPrem5sVV/9tVt7VS2ZnBBCo2NAWYblduMh6+M8dGxeSUxNOZ+RDjSVzsX4qHEjOBOEmaaRZOhqDge5Yqs3/wxJbalG7yvukS8uz9c2LH60vOT+2H4lBL0D/p49mvCn1OV4iEVo4EV/AgYaYE7slWP8vGhnZiX5LrzvThwslpM33hDyDZevowryFh981TVDsdIovZY4+NomFjIAzdl85EWWoWqqxcE= ceph-aa240572-0159-11ef-bc93-c7b262605968' | sudo tee -a /root/.ssh/authorized_keys && sudo chmod 0600 /root/.ssh/authorized_keys 2024-04-23T10:17:54.985 INFO:teuthology.orchestra.run.smithi204.stdout:ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQC96O2tfq4F3ru+DL6BARr7cIMU2/U8e/M6SPi7vN3KakfqSi61xF0qopAsZD4nJpuRU4W+DbVrMHS+bHJ1s8ghaepC+fY+ibcriLsFcTooLk93RHYvxqSsqqJlqmoy6n9LaMx5RQrZkocXjNwj2FKHzV+xgfLZ/hEjBJPTBAl/Vg/JnfK/CA67iOeSKYWuxkARJd8zE2ePtyD9WsNmPkwccG3Q06xHl5ZzTlLNJNXDW3wgaLenxwliglHqYOmLC5jeUcjdYCJ2Xic/vnu7U8vjT0kKYrcD+BR1+ZPrem5sVV/9tVt7VS2ZnBBCo2NAWYblduMh6+M8dGxeSUxNOZ+RDjSVzsX4qHEjOBOEmaaRZOhqDge5Yqs3/wxJbalG7yvukS8uz9c2LH60vOT+2H4lBL0D/p49mvCn1OV4iEVo4EV/AgYaYE7slWP8vGhnZiX5LrzvThwslpM33hDyDZevowryFh981TVDsdIovZY4+NomFjIAzdl85EWWoWqqxcE= ceph-aa240572-0159-11ef-bc93-c7b262605968 2024-04-23T10:17:54.997 DEBUG:teuthology.orchestra.run.smithi114:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d69f13d827952bb07bb9b7973bca14f97d764d6f shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid aa240572-0159-11ef-bc93-c7b262605968 -- ceph config set mgr mgr/cephadm/allow_ptrace true 2024-04-23T10:17:55.126 INFO:journalctl@ceph.mon.a.smithi114.stdout:Apr 23 10:17:55 smithi114 ceph-mon[29255]: from='client.? 172.21.15.114:0/3546529694' entity='client.admin' 2024-04-23T10:17:55.234 INFO:teuthology.orchestra.run.smithi114.stderr:Inferring config /var/lib/ceph/aa240572-0159-11ef-bc93-c7b262605968/mon.a/config 2024-04-23T10:17:57.423 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:17:57.422+0000 7f4932f20700 1 -- 172.21.15.114:0/1842246437 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f492c102490 msgr2=0x7f492c1028b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T10:17:57.423 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:17:57.422+0000 7f4932f20700 1 --2- 172.21.15.114:0/1842246437 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f492c102490 0x7f492c1028b0 secure :-1 s=READY pgs=80 cs=0 l=1 rev1=1 crypto rx=0x7f4920009910 tx=0x7f4920009c20 comp rx=0 tx=0).stop 2024-04-23T10:17:57.423 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:17:57.422+0000 7f4932f20700 1 -- 172.21.15.114:0/1842246437 shutdown_connections 2024-04-23T10:17:57.423 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:17:57.422+0000 7f4932f20700 1 --2- 172.21.15.114:0/1842246437 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f492c102490 0x7f492c1028b0 unknown :-1 s=CLOSED pgs=80 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:17:57.423 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:17:57.422+0000 7f4932f20700 1 -- 172.21.15.114:0/1842246437 >> 172.21.15.114:0/1842246437 conn(0x7f492c0fda30 msgr2=0x7f492c0ffe70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T10:17:57.423 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:17:57.422+0000 7f4932f20700 1 -- 172.21.15.114:0/1842246437 shutdown_connections 2024-04-23T10:17:57.423 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:17:57.422+0000 7f4932f20700 1 -- 172.21.15.114:0/1842246437 wait complete. 2024-04-23T10:17:57.424 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:17:57.422+0000 7f4932f20700 1 Processor -- start 2024-04-23T10:17:57.424 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:17:57.422+0000 7f4932f20700 1 -- start start 2024-04-23T10:17:57.424 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:17:57.423+0000 7f4932f20700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f492c102490 0x7f492c197ad0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T10:17:57.424 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:17:57.423+0000 7f4932f20700 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x7f492c1083d0 con 0x7f492c102490 2024-04-23T10:17:57.424 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:17:57.423+0000 7f4930cbc700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f492c102490 0x7f492c197ad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T10:17:57.424 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:17:57.423+0000 7f4930cbc700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f492c102490 0x7f492c197ad0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.114:60596/0 (socket says 172.21.15.114:60596) 2024-04-23T10:17:57.424 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:17:57.423+0000 7f4930cbc700 1 -- 172.21.15.114:0/2314980317 learned_addr learned my addr 172.21.15.114:0/2314980317 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-04-23T10:17:57.424 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:17:57.423+0000 7f4930cbc700 1 -- 172.21.15.114:0/2314980317 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f492c198010 con 0x7f492c102490 2024-04-23T10:17:57.424 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:17:57.423+0000 7f4930cbc700 1 --2- 172.21.15.114:0/2314980317 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f492c102490 0x7f492c197ad0 secure :-1 s=READY pgs=81 cs=0 l=1 rev1=1 crypto rx=0x7f4920005800 tx=0x7f4920009f60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T10:17:57.424 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:17:57.423+0000 7f4929ffb700 1 -- 172.21.15.114:0/2314980317 <== mon.0 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f4920027070 con 0x7f492c102490 2024-04-23T10:17:57.424 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:17:57.423+0000 7f4932f20700 1 -- 172.21.15.114:0/2314980317 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f492c1982a0 con 0x7f492c102490 2024-04-23T10:17:57.424 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:17:57.423+0000 7f4929ffb700 1 -- 172.21.15.114:0/2314980317 <== mon.0 v2:172.21.15.114:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f492000be10 con 0x7f492c102490 2024-04-23T10:17:57.425 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:17:57.423+0000 7f4929ffb700 1 -- 172.21.15.114:0/2314980317 <== mon.0 v2:172.21.15.114:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f492000f460 con 0x7f492c102490 2024-04-23T10:17:57.425 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:17:57.424+0000 7f4932f20700 1 -- 172.21.15.114:0/2314980317 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f492c19af90 con 0x7f492c102490 2024-04-23T10:17:57.425 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:17:57.424+0000 7f4932f20700 1 -- 172.21.15.114:0/2314980317 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f492c108b50 con 0x7f492c102490 2024-04-23T10:17:57.426 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:17:57.425+0000 7f4929ffb700 1 -- 172.21.15.114:0/2314980317 <== mon.0 v2:172.21.15.114:3300/0 4 ==== mgrmap(e 13) ==== 46615+0+0 (secure 0 0 0) 0x7f492002a070 con 0x7f492c102490 2024-04-23T10:17:57.426 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:17:57.425+0000 7f4929ffb700 1 --2- 172.21.15.114:0/2314980317 >> [v2:172.21.15.114:6800/465333808,v1:172.21.15.114:6801/465333808] conn(0x7f491c039650 0x7f491c03bb10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T10:17:57.426 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:17:57.425+0000 7f4929ffb700 1 -- 172.21.15.114:0/2314980317 <== mon.0 v2:172.21.15.114:3300/0 5 ==== osd_map(3..3 src has 1..3) ==== 1155+0+0 (secure 0 0 0) 0x7f492001e030 con 0x7f492c102490 2024-04-23T10:17:57.426 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:17:57.425+0000 7f492bfff700 1 --2- 172.21.15.114:0/2314980317 >> [v2:172.21.15.114:6800/465333808,v1:172.21.15.114:6801/465333808] conn(0x7f491c039650 0x7f491c03bb10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T10:17:57.426 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:17:57.425+0000 7f492bfff700 1 --2- 172.21.15.114:0/2314980317 >> [v2:172.21.15.114:6800/465333808,v1:172.21.15.114:6801/465333808] conn(0x7f491c039650 0x7f491c03bb10 secure :-1 s=READY pgs=12 cs=0 l=1 rev1=1 crypto rx=0x7f4918006fd0 tx=0x7f4918006e40 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T10:17:57.429 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:17:57.430+0000 7f4929ffb700 1 -- 172.21.15.114:0/2314980317 <== mon.0 v2:172.21.15.114:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f4920013070 con 0x7f492c102490 2024-04-23T10:17:57.633 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:17:57.634+0000 7f4932f20700 1 -- 172.21.15.114:0/2314980317 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_command([{prefix=config set, name=mgr/cephadm/allow_ptrace}] v 0) -- 0x7f492c02e090 con 0x7f492c102490 2024-04-23T10:17:57.635 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:17:57.636+0000 7f4929ffb700 1 -- 172.21.15.114:0/2314980317 <== mon.0 v2:172.21.15.114:3300/0 7 ==== mon_command_ack([{prefix=config set, name=mgr/cephadm/allow_ptrace}]=0 v9)=0 v9) ==== 125+0+0 (secure 0 0 0) 0x7f4920031030 con 0x7f492c102490 2024-04-23T10:17:57.638 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:17:57.638+0000 7f4932f20700 1 -- 172.21.15.114:0/2314980317 >> [v2:172.21.15.114:6800/465333808,v1:172.21.15.114:6801/465333808] conn(0x7f491c039650 msgr2=0x7f491c03bb10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T10:17:57.638 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:17:57.638+0000 7f4932f20700 1 --2- 172.21.15.114:0/2314980317 >> [v2:172.21.15.114:6800/465333808,v1:172.21.15.114:6801/465333808] conn(0x7f491c039650 0x7f491c03bb10 secure :-1 s=READY pgs=12 cs=0 l=1 rev1=1 crypto rx=0x7f4918006fd0 tx=0x7f4918006e40 comp rx=0 tx=0).stop 2024-04-23T10:17:57.638 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:17:57.638+0000 7f4932f20700 1 -- 172.21.15.114:0/2314980317 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f492c102490 msgr2=0x7f492c197ad0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T10:17:57.638 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:17:57.638+0000 7f4932f20700 1 --2- 172.21.15.114:0/2314980317 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f492c102490 0x7f492c197ad0 secure :-1 s=READY pgs=81 cs=0 l=1 rev1=1 crypto rx=0x7f4920005800 tx=0x7f4920009f60 comp rx=0 tx=0).stop 2024-04-23T10:17:57.638 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:17:57.638+0000 7f4932f20700 1 -- 172.21.15.114:0/2314980317 shutdown_connections 2024-04-23T10:17:57.638 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:17:57.638+0000 7f4932f20700 1 --2- 172.21.15.114:0/2314980317 >> [v2:172.21.15.114:6800/465333808,v1:172.21.15.114:6801/465333808] conn(0x7f491c039650 0x7f491c03bb10 unknown :-1 s=CLOSED pgs=12 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:17:57.638 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:17:57.638+0000 7f4932f20700 1 --2- 172.21.15.114:0/2314980317 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f492c102490 0x7f492c197ad0 unknown :-1 s=CLOSED pgs=81 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:17:57.638 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:17:57.638+0000 7f4932f20700 1 -- 172.21.15.114:0/2314980317 >> 172.21.15.114:0/2314980317 conn(0x7f492c0fda30 msgr2=0x7f492c0fe580 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T10:17:57.638 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:17:57.638+0000 7f4932f20700 1 -- 172.21.15.114:0/2314980317 shutdown_connections 2024-04-23T10:17:57.638 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:17:57.639+0000 7f4932f20700 1 -- 172.21.15.114:0/2314980317 wait complete. 2024-04-23T10:17:57.650 INFO:journalctl@ceph.mon.a.smithi114.stdout:Apr 23 10:17:57 smithi114 ceph-mon[29255]: from='mgr.14152 172.21.15.114:0/1529706138' entity='mgr.x' 2024-04-23T10:17:57.650 INFO:journalctl@ceph.mon.a.smithi114.stdout:Apr 23 10:17:57 smithi114 ceph-mon[29255]: from='mgr.14152 172.21.15.114:0/1529706138' entity='mgr.x' 2024-04-23T10:17:57.650 INFO:journalctl@ceph.mon.a.smithi114.stdout:Apr 23 10:17:57 smithi114 ceph-mon[29255]: from='mgr.14152 172.21.15.114:0/1529706138' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd/host:smithi114", "name": "osd_memory_target"}]: dispatch 2024-04-23T10:17:57.650 INFO:journalctl@ceph.mon.a.smithi114.stdout:Apr 23 10:17:57 smithi114 ceph-mon[29255]: from='mgr.14152 172.21.15.114:0/1529706138' entity='mgr.x' 2024-04-23T10:17:57.650 INFO:journalctl@ceph.mon.a.smithi114.stdout:Apr 23 10:17:57 smithi114 ceph-mon[29255]: from='mgr.14152 172.21.15.114:0/1529706138' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-23T10:17:57.650 INFO:journalctl@ceph.mon.a.smithi114.stdout:Apr 23 10:17:57 smithi114 ceph-mon[29255]: from='mgr.14152 172.21.15.114:0/1529706138' entity='mgr.x' 2024-04-23T10:17:58.289 INFO:tasks.cephadm:Distributing conf and client.admin keyring to all hosts + 0755 2024-04-23T10:17:58.289 DEBUG:teuthology.orchestra.run.smithi114:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d69f13d827952bb07bb9b7973bca14f97d764d6f shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid aa240572-0159-11ef-bc93-c7b262605968 -- ceph orch client-keyring set client.admin '*' --mode 0755 2024-04-23T10:17:58.524 INFO:teuthology.orchestra.run.smithi114.stderr:Inferring config /var/lib/ceph/aa240572-0159-11ef-bc93-c7b262605968/mon.a/config 2024-04-23T10:17:58.972 INFO:journalctl@ceph.mon.a.smithi114.stdout:Apr 23 10:17:58 smithi114 ceph-mon[29255]: from='client.? 172.21.15.114:0/2314980317' entity='client.admin' 2024-04-23T10:17:58.972 INFO:journalctl@ceph.mon.a.smithi114.stdout:Apr 23 10:17:58 smithi114 ceph-mon[29255]: from='mgr.14152 172.21.15.114:0/1529706138' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-23T10:17:58.972 INFO:journalctl@ceph.mon.a.smithi114.stdout:Apr 23 10:17:58 smithi114 ceph-mon[29255]: from='mgr.14152 172.21.15.114:0/1529706138' entity='mgr.x' 2024-04-23T10:18:00.078 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:18:00.075+0000 7f36e44e0700 1 -- 172.21.15.114:0/569676857 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f36dc101410 msgr2=0x7f36dc101830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T10:18:00.078 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:18:00.075+0000 7f36e44e0700 1 --2- 172.21.15.114:0/569676857 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f36dc101410 0x7f36dc101830 secure :-1 s=READY pgs=82 cs=0 l=1 rev1=1 crypto rx=0x7f36d8009910 tx=0x7f36d8009c20 comp rx=0 tx=0).stop 2024-04-23T10:18:00.078 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:18:00.076+0000 7f36e44e0700 1 -- 172.21.15.114:0/569676857 shutdown_connections 2024-04-23T10:18:00.078 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:18:00.076+0000 7f36e44e0700 1 --2- 172.21.15.114:0/569676857 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f36dc101410 0x7f36dc101830 unknown :-1 s=CLOSED pgs=82 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:18:00.078 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:18:00.076+0000 7f36e44e0700 1 -- 172.21.15.114:0/569676857 >> 172.21.15.114:0/569676857 conn(0x7f36dc0fc9b0 msgr2=0x7f36dc0fedf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T10:18:00.078 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:18:00.076+0000 7f36e44e0700 1 -- 172.21.15.114:0/569676857 shutdown_connections 2024-04-23T10:18:00.078 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:18:00.076+0000 7f36e44e0700 1 -- 172.21.15.114:0/569676857 wait complete. 2024-04-23T10:18:00.078 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:18:00.076+0000 7f36e44e0700 1 Processor -- start 2024-04-23T10:18:00.078 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:18:00.076+0000 7f36e44e0700 1 -- start start 2024-04-23T10:18:00.078 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:18:00.076+0000 7f36e44e0700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f36dc101410 0x7f36dc06a910 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T10:18:00.079 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:18:00.076+0000 7f36e44e0700 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x7f36dc1072e0 con 0x7f36dc101410 2024-04-23T10:18:00.079 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:18:00.076+0000 7f36e227c700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f36dc101410 0x7f36dc06a910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T10:18:00.079 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:18:00.076+0000 7f36e227c700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f36dc101410 0x7f36dc06a910 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.114:60616/0 (socket says 172.21.15.114:60616) 2024-04-23T10:18:00.079 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:18:00.076+0000 7f36e227c700 1 -- 172.21.15.114:0/697789493 learned_addr learned my addr 172.21.15.114:0/697789493 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-04-23T10:18:00.079 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:18:00.077+0000 7f36e227c700 1 -- 172.21.15.114:0/697789493 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f36dc19a820 con 0x7f36dc101410 2024-04-23T10:18:00.079 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:18:00.077+0000 7f36e227c700 1 --2- 172.21.15.114:0/697789493 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f36dc101410 0x7f36dc06a910 secure :-1 s=READY pgs=83 cs=0 l=1 rev1=1 crypto rx=0x7f36d8005800 tx=0x7f36d8009f60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T10:18:00.079 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:18:00.077+0000 7f36d37fe700 1 -- 172.21.15.114:0/697789493 <== mon.0 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f36d8027070 con 0x7f36dc101410 2024-04-23T10:18:00.079 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:18:00.077+0000 7f36d37fe700 1 -- 172.21.15.114:0/697789493 <== mon.0 v2:172.21.15.114:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f36d800be10 con 0x7f36dc101410 2024-04-23T10:18:00.079 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:18:00.077+0000 7f36d37fe700 1 -- 172.21.15.114:0/697789493 <== mon.0 v2:172.21.15.114:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f36d800f460 con 0x7f36dc101410 2024-04-23T10:18:00.079 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:18:00.077+0000 7f36e44e0700 1 -- 172.21.15.114:0/697789493 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f36dc19a9f0 con 0x7f36dc101410 2024-04-23T10:18:00.079 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:18:00.077+0000 7f36e44e0700 1 -- 172.21.15.114:0/697789493 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f36dc19ae10 con 0x7f36dc101410 2024-04-23T10:18:00.079 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:18:00.078+0000 7f36d37fe700 1 -- 172.21.15.114:0/697789493 <== mon.0 v2:172.21.15.114:3300/0 4 ==== mgrmap(e 13) ==== 46615+0+0 (secure 0 0 0) 0x7f36d802a070 con 0x7f36dc101410 2024-04-23T10:18:00.079 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:18:00.078+0000 7f36e44e0700 1 -- 172.21.15.114:0/697789493 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f36dc107a60 con 0x7f36dc101410 2024-04-23T10:18:00.079 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:18:00.078+0000 7f36d37fe700 1 --2- 172.21.15.114:0/697789493 >> [v2:172.21.15.114:6800/465333808,v1:172.21.15.114:6801/465333808] conn(0x7f36c8039650 0x7f36c803bb10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T10:18:00.080 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:18:00.078+0000 7f36d37fe700 1 -- 172.21.15.114:0/697789493 <== mon.0 v2:172.21.15.114:3300/0 5 ==== osd_map(3..3 src has 1..3) ==== 1155+0+0 (secure 0 0 0) 0x7f36d801e030 con 0x7f36dc101410 2024-04-23T10:18:00.080 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:18:00.078+0000 7f36e1a7b700 1 --2- 172.21.15.114:0/697789493 >> [v2:172.21.15.114:6800/465333808,v1:172.21.15.114:6801/465333808] conn(0x7f36c8039650 0x7f36c803bb10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T10:18:00.080 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:18:00.078+0000 7f36e1a7b700 1 --2- 172.21.15.114:0/697789493 >> [v2:172.21.15.114:6800/465333808,v1:172.21.15.114:6801/465333808] conn(0x7f36c8039650 0x7f36c803bb10 secure :-1 s=READY pgs=13 cs=0 l=1 rev1=1 crypto rx=0x7f36cc006fd0 tx=0x7f36cc006e40 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T10:18:00.082 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:18:00.083+0000 7f36d37fe700 1 -- 172.21.15.114:0/697789493 <== mon.0 v2:172.21.15.114:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f36d8013070 con 0x7f36dc101410 2024-04-23T10:18:00.285 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:18:00.286+0000 7f36e44e0700 1 -- 172.21.15.114:0/697789493 --> [v2:172.21.15.114:6800/465333808,v1:172.21.15.114:6801/465333808] -- mgr_command(tid 0: {"prefix": "orch client-keyring set", "entity": "client.admin", "placement": "*", "mode": "0755", "target": ["mon-mgr", ""]}) -- 0x7f36dc02e240 con 0x7f36c8039650 2024-04-23T10:18:00.288 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:18:00.289+0000 7f36d37fe700 1 -- 172.21.15.114:0/697789493 <== mgr.14152 v2:172.21.15.114:6800/465333808 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+0 (secure 0 0 0) 0x7f36dc02e240 con 0x7f36c8039650 2024-04-23T10:18:00.291 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:18:00.291+0000 7f36e44e0700 1 -- 172.21.15.114:0/697789493 >> [v2:172.21.15.114:6800/465333808,v1:172.21.15.114:6801/465333808] conn(0x7f36c8039650 msgr2=0x7f36c803bb10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T10:18:00.291 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:18:00.291+0000 7f36e44e0700 1 --2- 172.21.15.114:0/697789493 >> [v2:172.21.15.114:6800/465333808,v1:172.21.15.114:6801/465333808] conn(0x7f36c8039650 0x7f36c803bb10 secure :-1 s=READY pgs=13 cs=0 l=1 rev1=1 crypto rx=0x7f36cc006fd0 tx=0x7f36cc006e40 comp rx=0 tx=0).stop 2024-04-23T10:18:00.291 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:18:00.291+0000 7f36e44e0700 1 -- 172.21.15.114:0/697789493 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f36dc101410 msgr2=0x7f36dc06a910 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T10:18:00.291 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:18:00.291+0000 7f36e44e0700 1 --2- 172.21.15.114:0/697789493 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f36dc101410 0x7f36dc06a910 secure :-1 s=READY pgs=83 cs=0 l=1 rev1=1 crypto rx=0x7f36d8005800 tx=0x7f36d8009f60 comp rx=0 tx=0).stop 2024-04-23T10:18:00.291 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:18:00.291+0000 7f36e44e0700 1 -- 172.21.15.114:0/697789493 shutdown_connections 2024-04-23T10:18:00.291 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:18:00.291+0000 7f36e44e0700 1 --2- 172.21.15.114:0/697789493 >> [v2:172.21.15.114:6800/465333808,v1:172.21.15.114:6801/465333808] conn(0x7f36c8039650 0x7f36c803bb10 unknown :-1 s=CLOSED pgs=13 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:18:00.291 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:18:00.291+0000 7f36e44e0700 1 --2- 172.21.15.114:0/697789493 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f36dc101410 0x7f36dc06a910 unknown :-1 s=CLOSED pgs=83 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:18:00.291 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:18:00.291+0000 7f36e44e0700 1 -- 172.21.15.114:0/697789493 >> 172.21.15.114:0/697789493 conn(0x7f36dc0fc9b0 msgr2=0x7f36dc0fd4a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T10:18:00.291 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:18:00.292+0000 7f36e44e0700 1 -- 172.21.15.114:0/697789493 shutdown_connections 2024-04-23T10:18:00.292 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:18:00.292+0000 7f36e44e0700 1 -- 172.21.15.114:0/697789493 wait complete. 2024-04-23T10:18:01.063 INFO:tasks.cephadm:Writing (initial) conf and keyring to smithi119 2024-04-23T10:18:01.063 DEBUG:teuthology.orchestra.run.smithi119:> set -ex 2024-04-23T10:18:01.063 DEBUG:teuthology.orchestra.run.smithi119:> dd of=/etc/ceph/ceph.conf 2024-04-23T10:18:01.080 DEBUG:teuthology.orchestra.run.smithi119:> set -ex 2024-04-23T10:18:01.081 DEBUG:teuthology.orchestra.run.smithi119:> dd of=/etc/ceph/ceph.client.admin.keyring 2024-04-23T10:18:01.138 INFO:tasks.cephadm:Adding host smithi119 to orchestrator... 2024-04-23T10:18:01.138 DEBUG:teuthology.orchestra.run.smithi114:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d69f13d827952bb07bb9b7973bca14f97d764d6f shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid aa240572-0159-11ef-bc93-c7b262605968 -- ceph orch host add smithi119 2024-04-23T10:18:01.371 INFO:teuthology.orchestra.run.smithi114.stderr:Inferring config /var/lib/ceph/aa240572-0159-11ef-bc93-c7b262605968/mon.a/config 2024-04-23T10:18:01.592 INFO:journalctl@ceph.mon.a.smithi114.stdout:Apr 23 10:18:01 smithi114 ceph-mon[29255]: from='client.14176 -' entity='client.admin' cmd=[{"prefix": "orch client-keyring set", "entity": "client.admin", "placement": "*", "mode": "0755", "target": ["mon-mgr", ""]}]: dispatch 2024-04-23T10:18:01.592 INFO:journalctl@ceph.mon.a.smithi114.stdout:Apr 23 10:18:01 smithi114 ceph-mon[29255]: from='mgr.14152 172.21.15.114:0/1529706138' entity='mgr.x' 2024-04-23T10:18:01.592 INFO:journalctl@ceph.mon.a.smithi114.stdout:Apr 23 10:18:01 smithi114 ceph-mon[29255]: from='mgr.14152 172.21.15.114:0/1529706138' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-23T10:18:01.592 INFO:journalctl@ceph.mon.a.smithi114.stdout:Apr 23 10:18:01 smithi114 ceph-mon[29255]: from='mgr.14152 172.21.15.114:0/1529706138' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-23T10:18:01.593 INFO:journalctl@ceph.mon.a.smithi114.stdout:Apr 23 10:18:01 smithi114 ceph-mon[29255]: from='mgr.14152 172.21.15.114:0/1529706138' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-23T10:18:01.593 INFO:journalctl@ceph.mon.a.smithi114.stdout:Apr 23 10:18:01 smithi114 ceph-mon[29255]: Updating smithi114:/etc/ceph/ceph.conf 2024-04-23T10:18:01.593 INFO:journalctl@ceph.mon.a.smithi114.stdout:Apr 23 10:18:01 smithi114 ceph-mon[29255]: Updating smithi114:/var/lib/ceph/aa240572-0159-11ef-bc93-c7b262605968/config/ceph.conf 2024-04-23T10:18:01.593 INFO:journalctl@ceph.mon.a.smithi114.stdout:Apr 23 10:18:01 smithi114 ceph-mon[29255]: Updating smithi114:/etc/ceph/ceph.client.admin.keyring 2024-04-23T10:18:01.593 INFO:journalctl@ceph.mon.a.smithi114.stdout:Apr 23 10:18:01 smithi114 ceph-mon[29255]: Updating smithi114:/var/lib/ceph/aa240572-0159-11ef-bc93-c7b262605968/config/ceph.client.admin.keyring 2024-04-23T10:18:01.593 INFO:journalctl@ceph.mon.a.smithi114.stdout:Apr 23 10:18:01 smithi114 ceph-mon[29255]: from='mgr.14152 172.21.15.114:0/1529706138' entity='mgr.x' 2024-04-23T10:18:01.593 INFO:journalctl@ceph.mon.a.smithi114.stdout:Apr 23 10:18:01 smithi114 ceph-mon[29255]: from='mgr.14152 172.21.15.114:0/1529706138' entity='mgr.x' 2024-04-23T10:18:01.593 INFO:journalctl@ceph.mon.a.smithi114.stdout:Apr 23 10:18:01 smithi114 ceph-mon[29255]: from='mgr.14152 172.21.15.114:0/1529706138' entity='mgr.x' 2024-04-23T10:18:02.883 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:18:02.882+0000 7f28f8809700 1 -- 172.21.15.114:0/3310938865 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f28f41084c0 msgr2=0x7f28f41088a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T10:18:02.883 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:18:02.882+0000 7f28f8809700 1 --2- 172.21.15.114:0/3310938865 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f28f41084c0 0x7f28f41088a0 secure :-1 s=READY pgs=84 cs=0 l=1 rev1=1 crypto rx=0x7f28e4009910 tx=0x7f28e4009c20 comp rx=0 tx=0).stop 2024-04-23T10:18:02.883 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:18:02.882+0000 7f28f8809700 1 -- 172.21.15.114:0/3310938865 shutdown_connections 2024-04-23T10:18:02.883 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:18:02.882+0000 7f28f8809700 1 --2- 172.21.15.114:0/3310938865 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f28f41084c0 0x7f28f41088a0 unknown :-1 s=CLOSED pgs=84 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:18:02.883 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:18:02.882+0000 7f28f8809700 1 -- 172.21.15.114:0/3310938865 >> 172.21.15.114:0/3310938865 conn(0x7f28f40fdf60 msgr2=0x7f28f4100380 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T10:18:02.884 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:18:02.882+0000 7f28f8809700 1 -- 172.21.15.114:0/3310938865 shutdown_connections 2024-04-23T10:18:02.884 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:18:02.882+0000 7f28f8809700 1 -- 172.21.15.114:0/3310938865 wait complete. 2024-04-23T10:18:02.884 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:18:02.883+0000 7f28f8809700 1 Processor -- start 2024-04-23T10:18:02.884 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:18:02.883+0000 7f28f8809700 1 -- start start 2024-04-23T10:18:02.884 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:18:02.883+0000 7f28f8809700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f28f41084c0 0x7f28f4197e70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T10:18:02.884 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:18:02.883+0000 7f28f8809700 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x7f28f410c920 con 0x7f28f41084c0 2024-04-23T10:18:02.884 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:18:02.883+0000 7f28f259c700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f28f41084c0 0x7f28f4197e70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T10:18:02.884 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:18:02.883+0000 7f28f259c700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f28f41084c0 0x7f28f4197e70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.114:60628/0 (socket says 172.21.15.114:60628) 2024-04-23T10:18:02.884 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:18:02.883+0000 7f28f259c700 1 -- 172.21.15.114:0/1217935402 learned_addr learned my addr 172.21.15.114:0/1217935402 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-04-23T10:18:02.884 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:18:02.883+0000 7f28f259c700 1 -- 172.21.15.114:0/1217935402 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f28f41983b0 con 0x7f28f41084c0 2024-04-23T10:18:02.884 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:18:02.883+0000 7f28f259c700 1 --2- 172.21.15.114:0/1217935402 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f28f41084c0 0x7f28f4197e70 secure :-1 s=READY pgs=85 cs=0 l=1 rev1=1 crypto rx=0x7f28e40056d0 tx=0x7f28e40057b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T10:18:02.884 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:18:02.884+0000 7f28ebfff700 1 -- 172.21.15.114:0/1217935402 <== mon.0 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f28e4026070 con 0x7f28f41084c0 2024-04-23T10:18:02.884 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:18:02.884+0000 7f28ebfff700 1 -- 172.21.15.114:0/1217935402 <== mon.0 v2:172.21.15.114:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f28e400bba0 con 0x7f28f41084c0 2024-04-23T10:18:02.884 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:18:02.884+0000 7f28f8809700 1 -- 172.21.15.114:0/1217935402 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f28f41986a0 con 0x7f28f41084c0 2024-04-23T10:18:02.884 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:18:02.884+0000 7f28ebfff700 1 -- 172.21.15.114:0/1217935402 <== mon.0 v2:172.21.15.114:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f28e4003920 con 0x7f28f41084c0 2024-04-23T10:18:02.885 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:18:02.884+0000 7f28f8809700 1 -- 172.21.15.114:0/1217935402 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f28f419c740 con 0x7f28f41084c0 2024-04-23T10:18:02.885 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:18:02.885+0000 7f28ebfff700 1 -- 172.21.15.114:0/1217935402 <== mon.0 v2:172.21.15.114:3300/0 4 ==== mgrmap(e 13) ==== 46615+0+0 (secure 0 0 0) 0x7f28e4029070 con 0x7f28f41084c0 2024-04-23T10:18:02.885 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:18:02.885+0000 7f28f8809700 1 -- 172.21.15.114:0/1217935402 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f28f4103b70 con 0x7f28f41084c0 2024-04-23T10:18:02.886 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:18:02.885+0000 7f28ebfff700 1 --2- 172.21.15.114:0/1217935402 >> [v2:172.21.15.114:6800/465333808,v1:172.21.15.114:6801/465333808] conn(0x7f28e0039650 0x7f28e003bb10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T10:18:02.886 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:18:02.885+0000 7f28ebfff700 1 -- 172.21.15.114:0/1217935402 <== mon.0 v2:172.21.15.114:3300/0 5 ==== osd_map(3..3 src has 1..3) ==== 1155+0+0 (secure 0 0 0) 0x7f28e401e070 con 0x7f28f41084c0 2024-04-23T10:18:02.886 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:18:02.885+0000 7f28e9d9b700 1 --2- 172.21.15.114:0/1217935402 >> [v2:172.21.15.114:6800/465333808,v1:172.21.15.114:6801/465333808] conn(0x7f28e0039650 0x7f28e003bb10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T10:18:02.886 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:18:02.885+0000 7f28e9d9b700 1 --2- 172.21.15.114:0/1217935402 >> [v2:172.21.15.114:6800/465333808,v1:172.21.15.114:6801/465333808] conn(0x7f28e0039650 0x7f28e003bb10 secure :-1 s=READY pgs=14 cs=0 l=1 rev1=1 crypto rx=0x7f28dc006fd0 tx=0x7f28dc006e40 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T10:18:02.889 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:18:02.890+0000 7f28ebfff700 1 -- 172.21.15.114:0/1217935402 <== mon.0 v2:172.21.15.114:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f28e4013070 con 0x7f28f41084c0 2024-04-23T10:18:03.090 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:18:03.091+0000 7f28f8809700 1 -- 172.21.15.114:0/1217935402 --> [v2:172.21.15.114:6800/465333808,v1:172.21.15.114:6801/465333808] -- mgr_command(tid 0: {"prefix": "orch host add", "hostname": "smithi119", "target": ["mon-mgr", ""]}) -- 0x7f28f41991d0 con 0x7f28e0039650 2024-04-23T10:18:04.504 INFO:journalctl@ceph.mon.a.smithi114.stdout:Apr 23 10:18:04 smithi114 ceph-mon[29255]: from='client.14178 -' entity='client.admin' cmd=[{"prefix": "orch host add", "hostname": "smithi119", "target": ["mon-mgr", ""]}]: dispatch 2024-04-23T10:18:04.504 INFO:journalctl@ceph.mon.a.smithi114.stdout:Apr 23 10:18:04 smithi114 ceph-mon[29255]: Deploying cephadm binary to smithi119 2024-04-23T10:18:07.200 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:18:07.201+0000 7f28ebfff700 1 -- 172.21.15.114:0/1217935402 <== mgr.14152 v2:172.21.15.114:6800/465333808 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+49 (secure 0 0 0) 0x7f28f41991d0 con 0x7f28e0039650 2024-04-23T10:18:07.201 INFO:journalctl@ceph.mon.a.smithi114.stdout:Apr 23 10:18:06 smithi114 ceph-mon[29255]: pgmap v3: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-23T10:18:07.201 INFO:teuthology.orchestra.run.smithi114.stdout:Added host 'smithi119' with addr '172.21.15.119' 2024-04-23T10:18:07.203 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:18:07.203+0000 7f28f8809700 1 -- 172.21.15.114:0/1217935402 >> [v2:172.21.15.114:6800/465333808,v1:172.21.15.114:6801/465333808] conn(0x7f28e0039650 msgr2=0x7f28e003bb10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T10:18:07.203 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:18:07.203+0000 7f28f8809700 1 --2- 172.21.15.114:0/1217935402 >> [v2:172.21.15.114:6800/465333808,v1:172.21.15.114:6801/465333808] conn(0x7f28e0039650 0x7f28e003bb10 secure :-1 s=READY pgs=14 cs=0 l=1 rev1=1 crypto rx=0x7f28dc006fd0 tx=0x7f28dc006e40 comp rx=0 tx=0).stop 2024-04-23T10:18:07.203 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:18:07.203+0000 7f28f8809700 1 -- 172.21.15.114:0/1217935402 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f28f41084c0 msgr2=0x7f28f4197e70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T10:18:07.203 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:18:07.203+0000 7f28f8809700 1 --2- 172.21.15.114:0/1217935402 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f28f41084c0 0x7f28f4197e70 secure :-1 s=READY pgs=85 cs=0 l=1 rev1=1 crypto rx=0x7f28e40056d0 tx=0x7f28e40057b0 comp rx=0 tx=0).stop 2024-04-23T10:18:07.203 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:18:07.203+0000 7f28f8809700 1 -- 172.21.15.114:0/1217935402 shutdown_connections 2024-04-23T10:18:07.203 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:18:07.203+0000 7f28f8809700 1 --2- 172.21.15.114:0/1217935402 >> [v2:172.21.15.114:6800/465333808,v1:172.21.15.114:6801/465333808] conn(0x7f28e0039650 0x7f28e003bb10 unknown :-1 s=CLOSED pgs=14 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:18:07.203 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:18:07.203+0000 7f28f8809700 1 --2- 172.21.15.114:0/1217935402 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f28f41084c0 0x7f28f4197e70 unknown :-1 s=CLOSED pgs=85 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:18:07.203 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:18:07.203+0000 7f28f8809700 1 -- 172.21.15.114:0/1217935402 >> 172.21.15.114:0/1217935402 conn(0x7f28f40fdf60 msgr2=0x7f28f40fed10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T10:18:07.203 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:18:07.204+0000 7f28f8809700 1 -- 172.21.15.114:0/1217935402 shutdown_connections 2024-04-23T10:18:07.203 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:18:07.204+0000 7f28f8809700 1 -- 172.21.15.114:0/1217935402 wait complete. 2024-04-23T10:18:07.808 DEBUG:teuthology.orchestra.run.smithi114:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d69f13d827952bb07bb9b7973bca14f97d764d6f shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid aa240572-0159-11ef-bc93-c7b262605968 -- ceph orch host ls --format=json 2024-04-23T10:18:08.045 INFO:teuthology.orchestra.run.smithi114.stderr:Inferring config /var/lib/ceph/aa240572-0159-11ef-bc93-c7b262605968/mon.a/config 2024-04-23T10:18:08.426 INFO:journalctl@ceph.mon.a.smithi114.stdout:Apr 23 10:18:08 smithi114 ceph-mon[29255]: from='mgr.14152 172.21.15.114:0/1529706138' entity='mgr.x' 2024-04-23T10:18:08.426 INFO:journalctl@ceph.mon.a.smithi114.stdout:Apr 23 10:18:08 smithi114 ceph-mon[29255]: Added host smithi119 2024-04-23T10:18:08.426 INFO:journalctl@ceph.mon.a.smithi114.stdout:Apr 23 10:18:08 smithi114 ceph-mon[29255]: from='mgr.14152 172.21.15.114:0/1529706138' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-23T10:18:08.426 INFO:journalctl@ceph.mon.a.smithi114.stdout:Apr 23 10:18:08 smithi114 ceph-mon[29255]: from='mgr.14152 172.21.15.114:0/1529706138' entity='mgr.x' 2024-04-23T10:18:08.426 INFO:journalctl@ceph.mon.a.smithi114.stdout:Apr 23 10:18:08 smithi114 ceph-mon[29255]: pgmap v4: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-23T10:18:09.441 INFO:journalctl@ceph.mon.a.smithi114.stdout:Apr 23 10:18:09 smithi114 ceph-mon[29255]: from='mgr.14152 172.21.15.114:0/1529706138' entity='mgr.x' 2024-04-23T10:18:09.669 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:18:09.667+0000 7f6d031e8700 1 -- 172.21.15.114:0/3065384394 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f6cfc101580 msgr2=0x7f6cfc101960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T10:18:09.669 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:18:09.667+0000 7f6d031e8700 1 --2- 172.21.15.114:0/3065384394 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f6cfc101580 0x7f6cfc101960 secure :-1 s=READY pgs=86 cs=0 l=1 rev1=1 crypto rx=0x7f6cf8009910 tx=0x7f6cf8009c20 comp rx=0 tx=0).stop 2024-04-23T10:18:09.670 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:18:09.667+0000 7f6d031e8700 1 -- 172.21.15.114:0/3065384394 shutdown_connections 2024-04-23T10:18:09.670 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:18:09.667+0000 7f6d031e8700 1 --2- 172.21.15.114:0/3065384394 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f6cfc101580 0x7f6cfc101960 unknown :-1 s=CLOSED pgs=86 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:18:09.670 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:18:09.667+0000 7f6d031e8700 1 -- 172.21.15.114:0/3065384394 >> 172.21.15.114:0/3065384394 conn(0x7f6cfc078010 msgr2=0x7f6cfc0ff2c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T10:18:09.670 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:18:09.667+0000 7f6d031e8700 1 -- 172.21.15.114:0/3065384394 shutdown_connections 2024-04-23T10:18:09.670 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:18:09.667+0000 7f6d031e8700 1 -- 172.21.15.114:0/3065384394 wait complete. 2024-04-23T10:18:09.670 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:18:09.668+0000 7f6d031e8700 1 Processor -- start 2024-04-23T10:18:09.670 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:18:09.668+0000 7f6d031e8700 1 -- start start 2024-04-23T10:18:09.670 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:18:09.668+0000 7f6d031e8700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f6cfc101580 0x7f6cfc196dd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T10:18:09.670 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:18:09.668+0000 7f6d031e8700 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x7f6cfc10b800 con 0x7f6cfc101580 2024-04-23T10:18:09.671 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:18:09.668+0000 7f6d00f84700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f6cfc101580 0x7f6cfc196dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T10:18:09.671 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:18:09.668+0000 7f6d00f84700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f6cfc101580 0x7f6cfc196dd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.114:36144/0 (socket says 172.21.15.114:36144) 2024-04-23T10:18:09.671 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:18:09.668+0000 7f6d00f84700 1 -- 172.21.15.114:0/1727757171 learned_addr learned my addr 172.21.15.114:0/1727757171 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-04-23T10:18:09.671 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:18:09.668+0000 7f6d00f84700 1 -- 172.21.15.114:0/1727757171 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f6cfc197310 con 0x7f6cfc101580 2024-04-23T10:18:09.671 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:18:09.668+0000 7f6d00f84700 1 --2- 172.21.15.114:0/1727757171 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f6cfc101580 0x7f6cfc196dd0 secure :-1 s=READY pgs=87 cs=0 l=1 rev1=1 crypto rx=0x7f6cf80056d0 tx=0x7f6cf80057b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T10:18:09.671 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:18:09.668+0000 7f6cf1ffb700 1 -- 172.21.15.114:0/1727757171 <== mon.0 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f6cf8026070 con 0x7f6cfc101580 2024-04-23T10:18:09.671 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:18:09.668+0000 7f6cf1ffb700 1 -- 172.21.15.114:0/1727757171 <== mon.0 v2:172.21.15.114:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f6cf800bba0 con 0x7f6cfc101580 2024-04-23T10:18:09.671 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:18:09.668+0000 7f6d031e8700 1 -- 172.21.15.114:0/1727757171 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f6cfc197600 con 0x7f6cfc101580 2024-04-23T10:18:09.671 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:18:09.668+0000 7f6cf1ffb700 1 -- 172.21.15.114:0/1727757171 <== mon.0 v2:172.21.15.114:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f6cf8003920 con 0x7f6cfc101580 2024-04-23T10:18:09.671 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:18:09.669+0000 7f6d031e8700 1 -- 172.21.15.114:0/1727757171 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f6cfc19b6a0 con 0x7f6cfc101580 2024-04-23T10:18:09.671 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:18:09.670+0000 7f6cf1ffb700 1 -- 172.21.15.114:0/1727757171 <== mon.0 v2:172.21.15.114:3300/0 4 ==== mgrmap(e 13) ==== 46615+0+0 (secure 0 0 0) 0x7f6cf8029070 con 0x7f6cfc101580 2024-04-23T10:18:09.672 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:18:09.670+0000 7f6d031e8700 1 -- 172.21.15.114:0/1727757171 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f6cfc102a60 con 0x7f6cfc101580 2024-04-23T10:18:09.672 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:18:09.670+0000 7f6cf1ffb700 1 --2- 172.21.15.114:0/1727757171 >> [v2:172.21.15.114:6800/465333808,v1:172.21.15.114:6801/465333808] conn(0x7f6ce4039650 0x7f6ce403bb10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T10:18:09.672 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:18:09.670+0000 7f6cf1ffb700 1 -- 172.21.15.114:0/1727757171 <== mon.0 v2:172.21.15.114:3300/0 5 ==== osd_map(3..3 src has 1..3) ==== 1155+0+0 (secure 0 0 0) 0x7f6cf801e070 con 0x7f6cfc101580 2024-04-23T10:18:09.672 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:18:09.670+0000 7f6cf3fff700 1 --2- 172.21.15.114:0/1727757171 >> [v2:172.21.15.114:6800/465333808,v1:172.21.15.114:6801/465333808] conn(0x7f6ce4039650 0x7f6ce403bb10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T10:18:09.672 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:18:09.670+0000 7f6cf3fff700 1 --2- 172.21.15.114:0/1727757171 >> [v2:172.21.15.114:6800/465333808,v1:172.21.15.114:6801/465333808] conn(0x7f6ce4039650 0x7f6ce403bb10 secure :-1 s=READY pgs=15 cs=0 l=1 rev1=1 crypto rx=0x7f6cec006fd0 tx=0x7f6cec006e40 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T10:18:09.674 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:18:09.675+0000 7f6cf1ffb700 1 -- 172.21.15.114:0/1727757171 <== mon.0 v2:172.21.15.114:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f6cf8013070 con 0x7f6cfc101580 2024-04-23T10:18:09.881 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:18:09.882+0000 7f6d031e8700 1 -- 172.21.15.114:0/1727757171 --> [v2:172.21.15.114:6800/465333808,v1:172.21.15.114:6801/465333808] -- mgr_command(tid 0: {"prefix": "orch host ls", "target": ["mon-mgr", ""], "format": "json"}) -- 0x7f6cfc198130 con 0x7f6ce4039650 2024-04-23T10:18:09.882 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:18:09.883+0000 7f6cf1ffb700 1 -- 172.21.15.114:0/1727757171 <== mgr.14152 v2:172.21.15.114:6800/465333808 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+161 (secure 0 0 0) 0x7f6cfc198130 con 0x7f6ce4039650 2024-04-23T10:18:09.882 INFO:teuthology.orchestra.run.smithi114.stdout: 2024-04-23T10:18:09.882 INFO:teuthology.orchestra.run.smithi114.stdout:[{"addr": "172.21.15.114", "hostname": "smithi114", "labels": [], "status": ""}, {"addr": "172.21.15.119", "hostname": "smithi119", "labels": [], "status": ""}] 2024-04-23T10:18:09.884 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:18:09.885+0000 7f6d031e8700 1 -- 172.21.15.114:0/1727757171 >> [v2:172.21.15.114:6800/465333808,v1:172.21.15.114:6801/465333808] conn(0x7f6ce4039650 msgr2=0x7f6ce403bb10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T10:18:09.884 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:18:09.885+0000 7f6d031e8700 1 --2- 172.21.15.114:0/1727757171 >> [v2:172.21.15.114:6800/465333808,v1:172.21.15.114:6801/465333808] conn(0x7f6ce4039650 0x7f6ce403bb10 secure :-1 s=READY pgs=15 cs=0 l=1 rev1=1 crypto rx=0x7f6cec006fd0 tx=0x7f6cec006e40 comp rx=0 tx=0).stop 2024-04-23T10:18:09.885 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:18:09.885+0000 7f6d031e8700 1 -- 172.21.15.114:0/1727757171 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f6cfc101580 msgr2=0x7f6cfc196dd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T10:18:09.885 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:18:09.885+0000 7f6d031e8700 1 --2- 172.21.15.114:0/1727757171 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f6cfc101580 0x7f6cfc196dd0 secure :-1 s=READY pgs=87 cs=0 l=1 rev1=1 crypto rx=0x7f6cf80056d0 tx=0x7f6cf80057b0 comp rx=0 tx=0).stop 2024-04-23T10:18:09.885 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:18:09.885+0000 7f6d031e8700 1 -- 172.21.15.114:0/1727757171 shutdown_connections 2024-04-23T10:18:09.885 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:18:09.885+0000 7f6d031e8700 1 --2- 172.21.15.114:0/1727757171 >> [v2:172.21.15.114:6800/465333808,v1:172.21.15.114:6801/465333808] conn(0x7f6ce4039650 0x7f6ce403bb10 unknown :-1 s=CLOSED pgs=15 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:18:09.885 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:18:09.885+0000 7f6d031e8700 1 --2- 172.21.15.114:0/1727757171 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f6cfc101580 0x7f6cfc196dd0 unknown :-1 s=CLOSED pgs=87 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:18:09.885 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:18:09.885+0000 7f6d031e8700 1 -- 172.21.15.114:0/1727757171 >> 172.21.15.114:0/1727757171 conn(0x7f6cfc078010 msgr2=0x7f6cfc102f50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T10:18:09.885 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:18:09.885+0000 7f6d031e8700 1 -- 172.21.15.114:0/1727757171 shutdown_connections 2024-04-23T10:18:09.885 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:18:09.885+0000 7f6d031e8700 1 -- 172.21.15.114:0/1727757171 wait complete. 2024-04-23T10:18:10.491 INFO:tasks.cephadm:Writing (initial) conf and keyring to smithi204 2024-04-23T10:18:10.491 DEBUG:teuthology.orchestra.run.smithi204:> set -ex 2024-04-23T10:18:10.491 DEBUG:teuthology.orchestra.run.smithi204:> dd of=/etc/ceph/ceph.conf 2024-04-23T10:18:10.510 DEBUG:teuthology.orchestra.run.smithi204:> set -ex 2024-04-23T10:18:10.510 DEBUG:teuthology.orchestra.run.smithi204:> dd of=/etc/ceph/ceph.client.admin.keyring 2024-04-23T10:18:10.567 INFO:tasks.cephadm:Adding host smithi204 to orchestrator... 2024-04-23T10:18:10.568 DEBUG:teuthology.orchestra.run.smithi114:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d69f13d827952bb07bb9b7973bca14f97d764d6f shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid aa240572-0159-11ef-bc93-c7b262605968 -- ceph orch host add smithi204 2024-04-23T10:18:10.804 INFO:teuthology.orchestra.run.smithi114.stderr:Inferring config /var/lib/ceph/aa240572-0159-11ef-bc93-c7b262605968/mon.a/config 2024-04-23T10:18:11.026 INFO:journalctl@ceph.mon.a.smithi114.stdout:Apr 23 10:18:10 smithi114 ceph-mon[29255]: from='mgr.14152 172.21.15.114:0/1529706138' entity='mgr.x' 2024-04-23T10:18:11.026 INFO:journalctl@ceph.mon.a.smithi114.stdout:Apr 23 10:18:10 smithi114 ceph-mon[29255]: pgmap v5: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-23T10:18:11.026 INFO:journalctl@ceph.mon.a.smithi114.stdout:Apr 23 10:18:10 smithi114 ceph-mon[29255]: from='client.14180 -' entity='client.admin' cmd=[{"prefix": "orch host ls", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-04-23T10:18:12.267 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:18:12.267+0000 7fc4a49d2700 1 -- 172.21.15.114:0/420548721 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fc4a01015b0 msgr2=0x7fc4a0101990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T10:18:12.267 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:18:12.267+0000 7fc4a49d2700 1 --2- 172.21.15.114:0/420548721 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fc4a01015b0 0x7fc4a0101990 secure :-1 s=READY pgs=88 cs=0 l=1 rev1=1 crypto rx=0x7fc490009910 tx=0x7fc490009c20 comp rx=0 tx=0).stop 2024-04-23T10:18:12.267 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:18:12.267+0000 7fc4a49d2700 1 -- 172.21.15.114:0/420548721 shutdown_connections 2024-04-23T10:18:12.267 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:18:12.267+0000 7fc4a49d2700 1 --2- 172.21.15.114:0/420548721 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fc4a01015b0 0x7fc4a0101990 unknown :-1 s=CLOSED pgs=88 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:18:12.267 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:18:12.267+0000 7fc4a49d2700 1 -- 172.21.15.114:0/420548721 >> 172.21.15.114:0/420548721 conn(0x7fc4a00fce80 msgr2=0x7fc4a00ff2a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T10:18:12.267 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:18:12.267+0000 7fc4a49d2700 1 -- 172.21.15.114:0/420548721 shutdown_connections 2024-04-23T10:18:12.267 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:18:12.267+0000 7fc4a49d2700 1 -- 172.21.15.114:0/420548721 wait complete. 2024-04-23T10:18:12.267 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:18:12.267+0000 7fc4a49d2700 1 Processor -- start 2024-04-23T10:18:12.267 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:18:12.268+0000 7fc4a49d2700 1 -- start start 2024-04-23T10:18:12.269 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:18:12.268+0000 7fc4a49d2700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fc4a01015b0 0x7fc4a0196d90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T10:18:12.269 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:18:12.268+0000 7fc4a49d2700 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x7fc4a010b830 con 0x7fc4a01015b0 2024-04-23T10:18:12.269 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:18:12.268+0000 7fc49e59c700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fc4a01015b0 0x7fc4a0196d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T10:18:12.270 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:18:12.268+0000 7fc49e59c700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fc4a01015b0 0x7fc4a0196d90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.114:36166/0 (socket says 172.21.15.114:36166) 2024-04-23T10:18:12.270 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:18:12.268+0000 7fc49e59c700 1 -- 172.21.15.114:0/636261191 learned_addr learned my addr 172.21.15.114:0/636261191 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-04-23T10:18:12.270 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:18:12.268+0000 7fc49e59c700 1 -- 172.21.15.114:0/636261191 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc4a01972d0 con 0x7fc4a01015b0 2024-04-23T10:18:12.270 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:18:12.268+0000 7fc49e59c700 1 --2- 172.21.15.114:0/636261191 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fc4a01015b0 0x7fc4a0196d90 secure :-1 s=READY pgs=89 cs=0 l=1 rev1=1 crypto rx=0x7fc4900056d0 tx=0x7fc4900057b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T10:18:12.270 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:18:12.269+0000 7fc497fff700 1 -- 172.21.15.114:0/636261191 <== mon.0 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fc490026070 con 0x7fc4a01015b0 2024-04-23T10:18:12.270 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:18:12.269+0000 7fc497fff700 1 -- 172.21.15.114:0/636261191 <== mon.0 v2:172.21.15.114:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fc49000bba0 con 0x7fc4a01015b0 2024-04-23T10:18:12.270 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:18:12.269+0000 7fc4a49d2700 1 -- 172.21.15.114:0/636261191 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fc4a01975c0 con 0x7fc4a01015b0 2024-04-23T10:18:12.270 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:18:12.269+0000 7fc497fff700 1 -- 172.21.15.114:0/636261191 <== mon.0 v2:172.21.15.114:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fc490003920 con 0x7fc4a01015b0 2024-04-23T10:18:12.270 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:18:12.269+0000 7fc4a49d2700 1 -- 172.21.15.114:0/636261191 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fc4a019b660 con 0x7fc4a01015b0 2024-04-23T10:18:12.270 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:18:12.269+0000 7fc497fff700 1 -- 172.21.15.114:0/636261191 <== mon.0 v2:172.21.15.114:3300/0 4 ==== mgrmap(e 13) ==== 46615+0+0 (secure 0 0 0) 0x7fc490029070 con 0x7fc4a01015b0 2024-04-23T10:18:12.270 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:18:12.270+0000 7fc4a49d2700 1 -- 172.21.15.114:0/636261191 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fc4a0102a90 con 0x7fc4a01015b0 2024-04-23T10:18:12.270 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:18:12.270+0000 7fc497fff700 1 --2- 172.21.15.114:0/636261191 >> [v2:172.21.15.114:6800/465333808,v1:172.21.15.114:6801/465333808] conn(0x7fc48c039650 0x7fc48c03bb10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T10:18:12.270 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:18:12.270+0000 7fc497fff700 1 -- 172.21.15.114:0/636261191 <== mon.0 v2:172.21.15.114:3300/0 5 ==== osd_map(3..3 src has 1..3) ==== 1155+0+0 (secure 0 0 0) 0x7fc49001e070 con 0x7fc4a01015b0 2024-04-23T10:18:12.270 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:18:12.270+0000 7fc495d9b700 1 --2- 172.21.15.114:0/636261191 >> [v2:172.21.15.114:6800/465333808,v1:172.21.15.114:6801/465333808] conn(0x7fc48c039650 0x7fc48c03bb10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T10:18:12.270 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:18:12.270+0000 7fc495d9b700 1 --2- 172.21.15.114:0/636261191 >> [v2:172.21.15.114:6800/465333808,v1:172.21.15.114:6801/465333808] conn(0x7fc48c039650 0x7fc48c03bb10 secure :-1 s=READY pgs=16 cs=0 l=1 rev1=1 crypto rx=0x7fc488006fd0 tx=0x7fc488006e40 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T10:18:12.274 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:18:12.275+0000 7fc497fff700 1 -- 172.21.15.114:0/636261191 <== mon.0 v2:172.21.15.114:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7fc490013070 con 0x7fc4a01015b0 2024-04-23T10:18:12.478 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:18:12.478+0000 7fc4a49d2700 1 -- 172.21.15.114:0/636261191 --> [v2:172.21.15.114:6800/465333808,v1:172.21.15.114:6801/465333808] -- mgr_command(tid 0: {"prefix": "orch host add", "hostname": "smithi204", "target": ["mon-mgr", ""]}) -- 0x7fc4a01980f0 con 0x7fc48c039650 2024-04-23T10:18:13.254 INFO:journalctl@ceph.mon.a.smithi114.stdout:Apr 23 10:18:12 smithi114 ceph-mon[29255]: pgmap v6: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-23T10:18:13.254 INFO:journalctl@ceph.mon.a.smithi114.stdout:Apr 23 10:18:12 smithi114 ceph-mon[29255]: from='mgr.14152 172.21.15.114:0/1529706138' entity='mgr.x' 2024-04-23T10:18:13.254 INFO:journalctl@ceph.mon.a.smithi114.stdout:Apr 23 10:18:12 smithi114 ceph-mon[29255]: from='mgr.14152 172.21.15.114:0/1529706138' entity='mgr.x' 2024-04-23T10:18:13.254 INFO:journalctl@ceph.mon.a.smithi114.stdout:Apr 23 10:18:12 smithi114 ceph-mon[29255]: from='mgr.14152 172.21.15.114:0/1529706138' entity='mgr.x' 2024-04-23T10:18:13.254 INFO:journalctl@ceph.mon.a.smithi114.stdout:Apr 23 10:18:12 smithi114 ceph-mon[29255]: from='mgr.14152 172.21.15.114:0/1529706138' entity='mgr.x' 2024-04-23T10:18:13.255 INFO:journalctl@ceph.mon.a.smithi114.stdout:Apr 23 10:18:12 smithi114 ceph-mon[29255]: from='mgr.14152 172.21.15.114:0/1529706138' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd/host:smithi119", "name": "osd_memory_target"}]: dispatch 2024-04-23T10:18:13.255 INFO:journalctl@ceph.mon.a.smithi114.stdout:Apr 23 10:18:12 smithi114 ceph-mon[29255]: from='mgr.14152 172.21.15.114:0/1529706138' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-23T10:18:13.255 INFO:journalctl@ceph.mon.a.smithi114.stdout:Apr 23 10:18:12 smithi114 ceph-mon[29255]: from='mgr.14152 172.21.15.114:0/1529706138' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-23T10:18:14.254 INFO:journalctl@ceph.mon.a.smithi114.stdout:Apr 23 10:18:13 smithi114 ceph-mon[29255]: from='client.14182 -' entity='client.admin' cmd=[{"prefix": "orch host add", "hostname": "smithi204", "target": ["mon-mgr", ""]}]: dispatch 2024-04-23T10:18:14.255 INFO:journalctl@ceph.mon.a.smithi114.stdout:Apr 23 10:18:13 smithi114 ceph-mon[29255]: Updating smithi119:/etc/ceph/ceph.conf 2024-04-23T10:18:14.255 INFO:journalctl@ceph.mon.a.smithi114.stdout:Apr 23 10:18:13 smithi114 ceph-mon[29255]: Updating smithi119:/var/lib/ceph/aa240572-0159-11ef-bc93-c7b262605968/config/ceph.conf 2024-04-23T10:18:14.255 INFO:journalctl@ceph.mon.a.smithi114.stdout:Apr 23 10:18:13 smithi114 ceph-mon[29255]: Updating smithi119:/etc/ceph/ceph.client.admin.keyring 2024-04-23T10:18:14.255 INFO:journalctl@ceph.mon.a.smithi114.stdout:Apr 23 10:18:13 smithi114 ceph-mon[29255]: from='mgr.14152 172.21.15.114:0/1529706138' entity='mgr.x' 2024-04-23T10:18:14.255 INFO:journalctl@ceph.mon.a.smithi114.stdout:Apr 23 10:18:13 smithi114 ceph-mon[29255]: from='mgr.14152 172.21.15.114:0/1529706138' entity='mgr.x' 2024-04-23T10:18:14.255 INFO:journalctl@ceph.mon.a.smithi114.stdout:Apr 23 10:18:13 smithi114 ceph-mon[29255]: from='mgr.14152 172.21.15.114:0/1529706138' entity='mgr.x' 2024-04-23T10:18:15.254 INFO:journalctl@ceph.mon.a.smithi114.stdout:Apr 23 10:18:14 smithi114 ceph-mon[29255]: Deploying cephadm binary to smithi204 2024-04-23T10:18:15.255 INFO:journalctl@ceph.mon.a.smithi114.stdout:Apr 23 10:18:14 smithi114 ceph-mon[29255]: Updating smithi119:/var/lib/ceph/aa240572-0159-11ef-bc93-c7b262605968/config/ceph.client.admin.keyring 2024-04-23T10:18:15.255 INFO:journalctl@ceph.mon.a.smithi114.stdout:Apr 23 10:18:14 smithi114 ceph-mon[29255]: pgmap v7: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-23T10:18:16.687 INFO:teuthology.orchestra.run.smithi114.stdout:Added host 'smithi204' with addr '172.21.15.204' 2024-04-23T10:18:16.687 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:18:16.687+0000 7fc497fff700 1 -- 172.21.15.114:0/636261191 <== mgr.14152 v2:172.21.15.114:6800/465333808 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+49 (secure 0 0 0) 0x7fc4a01980f0 con 0x7fc48c039650 2024-04-23T10:18:16.690 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:18:16.690+0000 7fc4a49d2700 1 -- 172.21.15.114:0/636261191 >> [v2:172.21.15.114:6800/465333808,v1:172.21.15.114:6801/465333808] conn(0x7fc48c039650 msgr2=0x7fc48c03bb10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T10:18:16.690 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:18:16.690+0000 7fc4a49d2700 1 --2- 172.21.15.114:0/636261191 >> [v2:172.21.15.114:6800/465333808,v1:172.21.15.114:6801/465333808] conn(0x7fc48c039650 0x7fc48c03bb10 secure :-1 s=READY pgs=16 cs=0 l=1 rev1=1 crypto rx=0x7fc488006fd0 tx=0x7fc488006e40 comp rx=0 tx=0).stop 2024-04-23T10:18:16.690 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:18:16.690+0000 7fc4a49d2700 1 -- 172.21.15.114:0/636261191 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fc4a01015b0 msgr2=0x7fc4a0196d90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T10:18:16.690 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:18:16.690+0000 7fc4a49d2700 1 --2- 172.21.15.114:0/636261191 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fc4a01015b0 0x7fc4a0196d90 secure :-1 s=READY pgs=89 cs=0 l=1 rev1=1 crypto rx=0x7fc4900056d0 tx=0x7fc4900057b0 comp rx=0 tx=0).stop 2024-04-23T10:18:16.690 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:18:16.690+0000 7fc4a49d2700 1 -- 172.21.15.114:0/636261191 shutdown_connections 2024-04-23T10:18:16.690 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:18:16.690+0000 7fc4a49d2700 1 --2- 172.21.15.114:0/636261191 >> [v2:172.21.15.114:6800/465333808,v1:172.21.15.114:6801/465333808] conn(0x7fc48c039650 0x7fc48c03bb10 unknown :-1 s=CLOSED pgs=16 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:18:16.691 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:18:16.690+0000 7fc4a49d2700 1 --2- 172.21.15.114:0/636261191 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fc4a01015b0 0x7fc4a0196d90 unknown :-1 s=CLOSED pgs=89 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:18:16.691 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:18:16.690+0000 7fc4a49d2700 1 -- 172.21.15.114:0/636261191 >> 172.21.15.114:0/636261191 conn(0x7fc4a00fce80 msgr2=0x7fc4a0102f80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T10:18:16.691 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:18:16.691+0000 7fc4a49d2700 1 -- 172.21.15.114:0/636261191 shutdown_connections 2024-04-23T10:18:16.691 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:18:16.691+0000 7fc4a49d2700 1 -- 172.21.15.114:0/636261191 wait complete. 2024-04-23T10:18:16.937 INFO:journalctl@ceph.mon.a.smithi114.stdout:Apr 23 10:18:16 smithi114 ceph-mon[29255]: pgmap v8: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-23T10:18:16.937 INFO:journalctl@ceph.mon.a.smithi114.stdout:Apr 23 10:18:16 smithi114 ceph-mon[29255]: from='mgr.14152 172.21.15.114:0/1529706138' entity='mgr.x' 2024-04-23T10:18:17.254 INFO:journalctl@ceph.mon.a.smithi114.stdout:Apr 23 10:18:16 smithi114 ceph-mon[29255]: from='mgr.14152 172.21.15.114:0/1529706138' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-23T10:18:17.312 DEBUG:teuthology.orchestra.run.smithi114:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d69f13d827952bb07bb9b7973bca14f97d764d6f shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid aa240572-0159-11ef-bc93-c7b262605968 -- ceph orch host ls --format=json 2024-04-23T10:18:17.548 INFO:teuthology.orchestra.run.smithi114.stderr:Inferring config /var/lib/ceph/aa240572-0159-11ef-bc93-c7b262605968/mon.a/config 2024-04-23T10:18:18.254 INFO:journalctl@ceph.mon.a.smithi114.stdout:Apr 23 10:18:17 smithi114 ceph-mon[29255]: Added host smithi204 2024-04-23T10:18:18.254 INFO:journalctl@ceph.mon.a.smithi114.stdout:Apr 23 10:18:17 smithi114 ceph-mon[29255]: from='mgr.14152 172.21.15.114:0/1529706138' entity='mgr.x' 2024-04-23T10:18:18.254 INFO:journalctl@ceph.mon.a.smithi114.stdout:Apr 23 10:18:17 smithi114 ceph-mon[29255]: from='mgr.14152 172.21.15.114:0/1529706138' entity='mgr.x' 2024-04-23T10:18:19.111 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:18:19.111+0000 7f5621aa9700 1 -- 172.21.15.114:0/3090888035 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f561c101580 msgr2=0x7f561c101960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T10:18:19.111 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:18:19.111+0000 7f5621aa9700 1 --2- 172.21.15.114:0/3090888035 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f561c101580 0x7f561c101960 secure :-1 s=READY pgs=90 cs=0 l=1 rev1=1 crypto rx=0x7f5614009910 tx=0x7f5614009c20 comp rx=0 tx=0).stop 2024-04-23T10:18:19.112 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:18:19.111+0000 7f5621aa9700 1 -- 172.21.15.114:0/3090888035 shutdown_connections 2024-04-23T10:18:19.112 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:18:19.111+0000 7f5621aa9700 1 --2- 172.21.15.114:0/3090888035 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f561c101580 0x7f561c101960 unknown :-1 s=CLOSED pgs=90 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:18:19.112 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:18:19.111+0000 7f5621aa9700 1 -- 172.21.15.114:0/3090888035 >> 172.21.15.114:0/3090888035 conn(0x7f561c078010 msgr2=0x7f561c0ff2c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T10:18:19.112 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:18:19.111+0000 7f5621aa9700 1 -- 172.21.15.114:0/3090888035 shutdown_connections 2024-04-23T10:18:19.112 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:18:19.111+0000 7f5621aa9700 1 -- 172.21.15.114:0/3090888035 wait complete. 2024-04-23T10:18:19.112 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:18:19.112+0000 7f5621aa9700 1 Processor -- start 2024-04-23T10:18:19.112 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:18:19.112+0000 7f5621aa9700 1 -- start start 2024-04-23T10:18:19.112 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:18:19.112+0000 7f5621aa9700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f561c101580 0x7f561c198d40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T10:18:19.112 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:18:19.112+0000 7f5621aa9700 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x7f561c10b800 con 0x7f561c101580 2024-04-23T10:18:19.112 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:18:19.112+0000 7f561b7fe700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f561c101580 0x7f561c198d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T10:18:19.112 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:18:19.112+0000 7f561b7fe700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f561c101580 0x7f561c198d40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.114:36532/0 (socket says 172.21.15.114:36532) 2024-04-23T10:18:19.112 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:18:19.112+0000 7f561b7fe700 1 -- 172.21.15.114:0/474311088 learned_addr learned my addr 172.21.15.114:0/474311088 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-04-23T10:18:19.113 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:18:19.112+0000 7f561b7fe700 1 -- 172.21.15.114:0/474311088 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f561c199280 con 0x7f561c101580 2024-04-23T10:18:19.113 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:18:19.112+0000 7f561b7fe700 1 --2- 172.21.15.114:0/474311088 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f561c101580 0x7f561c198d40 secure :-1 s=READY pgs=91 cs=0 l=1 rev1=1 crypto rx=0x7f56140056d0 tx=0x7f56140057b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T10:18:19.113 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:18:19.113+0000 7f5618ff9700 1 -- 172.21.15.114:0/474311088 <== mon.0 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f5614026070 con 0x7f561c101580 2024-04-23T10:18:19.114 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:18:19.113+0000 7f5621aa9700 1 -- 172.21.15.114:0/474311088 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f561c199570 con 0x7f561c101580 2024-04-23T10:18:19.114 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:18:19.113+0000 7f5618ff9700 1 -- 172.21.15.114:0/474311088 <== mon.0 v2:172.21.15.114:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f561400bba0 con 0x7f561c101580 2024-04-23T10:18:19.114 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:18:19.113+0000 7f5618ff9700 1 -- 172.21.15.114:0/474311088 <== mon.0 v2:172.21.15.114:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f5614003920 con 0x7f561c101580 2024-04-23T10:18:19.114 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:18:19.113+0000 7f5621aa9700 1 -- 172.21.15.114:0/474311088 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f561c199a30 con 0x7f561c101580 2024-04-23T10:18:19.114 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:18:19.113+0000 7f5618ff9700 1 -- 172.21.15.114:0/474311088 <== mon.0 v2:172.21.15.114:3300/0 4 ==== mgrmap(e 13) ==== 46615+0+0 (secure 0 0 0) 0x7f5614029070 con 0x7f561c101580 2024-04-23T10:18:19.114 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:18:19.113+0000 7f5621aa9700 1 -- 172.21.15.114:0/474311088 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f561c102a60 con 0x7f561c101580 2024-04-23T10:18:19.114 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:18:19.114+0000 7f5618ff9700 1 --2- 172.21.15.114:0/474311088 >> [v2:172.21.15.114:6800/465333808,v1:172.21.15.114:6801/465333808] conn(0x7f5604039650 0x7f560403bb10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T10:18:19.114 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:18:19.114+0000 7f5618ff9700 1 -- 172.21.15.114:0/474311088 <== mon.0 v2:172.21.15.114:3300/0 5 ==== osd_map(3..3 src has 1..3) ==== 1155+0+0 (secure 0 0 0) 0x7f561401e070 con 0x7f561c101580 2024-04-23T10:18:19.114 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:18:19.114+0000 7f561affd700 1 --2- 172.21.15.114:0/474311088 >> [v2:172.21.15.114:6800/465333808,v1:172.21.15.114:6801/465333808] conn(0x7f5604039650 0x7f560403bb10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T10:18:19.114 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:18:19.114+0000 7f561affd700 1 --2- 172.21.15.114:0/474311088 >> [v2:172.21.15.114:6800/465333808,v1:172.21.15.114:6801/465333808] conn(0x7f5604039650 0x7f560403bb10 secure :-1 s=READY pgs=17 cs=0 l=1 rev1=1 crypto rx=0x7f561000ad30 tx=0x7f56100093f0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T10:18:19.118 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:18:19.119+0000 7f5618ff9700 1 -- 172.21.15.114:0/474311088 <== mon.0 v2:172.21.15.114:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f5614013070 con 0x7f561c101580 2024-04-23T10:18:19.118 INFO:journalctl@ceph.mon.a.smithi114.stdout:Apr 23 10:18:18 smithi114 ceph-mon[29255]: pgmap v9: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-23T10:18:19.323 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:18:19.324+0000 7f5621aa9700 1 -- 172.21.15.114:0/474311088 --> [v2:172.21.15.114:6800/465333808,v1:172.21.15.114:6801/465333808] -- mgr_command(tid 0: {"prefix": "orch host ls", "target": ["mon-mgr", ""], "format": "json"}) -- 0x7f561c06a460 con 0x7f5604039650 2024-04-23T10:18:19.324 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:18:19.325+0000 7f5618ff9700 1 -- 172.21.15.114:0/474311088 <== mgr.14152 v2:172.21.15.114:6800/465333808 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+241 (secure 0 0 0) 0x7f561c06a460 con 0x7f5604039650 2024-04-23T10:18:19.324 INFO:teuthology.orchestra.run.smithi114.stdout: 2024-04-23T10:18:19.324 INFO:teuthology.orchestra.run.smithi114.stdout:[{"addr": "172.21.15.114", "hostname": "smithi114", "labels": [], "status": ""}, {"addr": "172.21.15.119", "hostname": "smithi119", "labels": [], "status": ""}, {"addr": "172.21.15.204", "hostname": "smithi204", "labels": [], "status": ""}] 2024-04-23T10:18:19.326 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:18:19.327+0000 7f5621aa9700 1 -- 172.21.15.114:0/474311088 >> [v2:172.21.15.114:6800/465333808,v1:172.21.15.114:6801/465333808] conn(0x7f5604039650 msgr2=0x7f560403bb10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T10:18:19.326 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:18:19.327+0000 7f5621aa9700 1 --2- 172.21.15.114:0/474311088 >> [v2:172.21.15.114:6800/465333808,v1:172.21.15.114:6801/465333808] conn(0x7f5604039650 0x7f560403bb10 secure :-1 s=READY pgs=17 cs=0 l=1 rev1=1 crypto rx=0x7f561000ad30 tx=0x7f56100093f0 comp rx=0 tx=0).stop 2024-04-23T10:18:19.326 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:18:19.327+0000 7f5621aa9700 1 -- 172.21.15.114:0/474311088 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f561c101580 msgr2=0x7f561c198d40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T10:18:19.326 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:18:19.327+0000 7f5621aa9700 1 --2- 172.21.15.114:0/474311088 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f561c101580 0x7f561c198d40 secure :-1 s=READY pgs=91 cs=0 l=1 rev1=1 crypto rx=0x7f56140056d0 tx=0x7f56140057b0 comp rx=0 tx=0).stop 2024-04-23T10:18:19.326 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:18:19.327+0000 7f5621aa9700 1 -- 172.21.15.114:0/474311088 shutdown_connections 2024-04-23T10:18:19.327 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:18:19.327+0000 7f5621aa9700 1 --2- 172.21.15.114:0/474311088 >> [v2:172.21.15.114:6800/465333808,v1:172.21.15.114:6801/465333808] conn(0x7f5604039650 0x7f560403bb10 unknown :-1 s=CLOSED pgs=17 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:18:19.327 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:18:19.327+0000 7f5621aa9700 1 --2- 172.21.15.114:0/474311088 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f561c101580 0x7f561c198d40 unknown :-1 s=CLOSED pgs=91 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:18:19.327 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:18:19.327+0000 7f5621aa9700 1 -- 172.21.15.114:0/474311088 >> 172.21.15.114:0/474311088 conn(0x7f561c078010 msgr2=0x7f561c102f50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T10:18:19.327 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:18:19.327+0000 7f5621aa9700 1 -- 172.21.15.114:0/474311088 shutdown_connections 2024-04-23T10:18:19.327 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:18:19.327+0000 7f5621aa9700 1 -- 172.21.15.114:0/474311088 wait complete. 2024-04-23T10:18:19.898 INFO:tasks.cephadm:Setting crush tunables to default 2024-04-23T10:18:19.899 DEBUG:teuthology.orchestra.run.smithi114:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d69f13d827952bb07bb9b7973bca14f97d764d6f shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid aa240572-0159-11ef-bc93-c7b262605968 -- ceph osd crush tunables default 2024-04-23T10:18:20.131 INFO:teuthology.orchestra.run.smithi114.stderr:Inferring config /var/lib/ceph/aa240572-0159-11ef-bc93-c7b262605968/mon.a/config 2024-04-23T10:18:20.504 INFO:journalctl@ceph.mon.a.smithi114.stdout:Apr 23 10:18:20 smithi114 ceph-mon[29255]: from='mgr.14152 172.21.15.114:0/1529706138' entity='mgr.x' 2024-04-23T10:18:20.504 INFO:journalctl@ceph.mon.a.smithi114.stdout:Apr 23 10:18:20 smithi114 ceph-mon[29255]: from='client.14184 -' entity='client.admin' cmd=[{"prefix": "orch host ls", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-04-23T10:18:20.505 INFO:journalctl@ceph.mon.a.smithi114.stdout:Apr 23 10:18:20 smithi114 ceph-mon[29255]: pgmap v10: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-23T10:18:21.590 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:18:21.589+0000 7fdd33dcb700 1 -- 172.21.15.114:0/984585715 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fdd2c100c90 msgr2=0x7fdd2c101070 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T10:18:21.590 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:18:21.589+0000 7fdd33dcb700 1 --2- 172.21.15.114:0/984585715 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fdd2c100c90 0x7fdd2c101070 secure :-1 s=READY pgs=92 cs=0 l=1 rev1=1 crypto rx=0x7fdd28009920 tx=0x7fdd28009c30 comp rx=0 tx=0).stop 2024-04-23T10:18:21.590 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:18:21.589+0000 7fdd33dcb700 1 -- 172.21.15.114:0/984585715 shutdown_connections 2024-04-23T10:18:21.590 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:18:21.589+0000 7fdd33dcb700 1 --2- 172.21.15.114:0/984585715 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fdd2c100c90 0x7fdd2c101070 unknown :-1 s=CLOSED pgs=92 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:18:21.590 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:18:21.589+0000 7fdd33dcb700 1 -- 172.21.15.114:0/984585715 >> 172.21.15.114:0/984585715 conn(0x7fdd2c0779f0 msgr2=0x7fdd2c077e00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T10:18:21.590 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:18:21.589+0000 7fdd33dcb700 1 -- 172.21.15.114:0/984585715 shutdown_connections 2024-04-23T10:18:21.590 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:18:21.589+0000 7fdd33dcb700 1 -- 172.21.15.114:0/984585715 wait complete. 2024-04-23T10:18:21.590 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:18:21.589+0000 7fdd33dcb700 1 Processor -- start 2024-04-23T10:18:21.590 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:18:21.589+0000 7fdd33dcb700 1 -- start start 2024-04-23T10:18:21.591 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:18:21.590+0000 7fdd33dcb700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fdd2c100c90 0x7fdd2c101c10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T10:18:21.591 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:18:21.590+0000 7fdd33dcb700 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x7fdd2c10c930 con 0x7fdd2c100c90 2024-04-23T10:18:21.591 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:18:21.590+0000 7fdd31b67700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fdd2c100c90 0x7fdd2c101c10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T10:18:21.591 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:18:21.590+0000 7fdd31b67700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fdd2c100c90 0x7fdd2c101c10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.114:36546/0 (socket says 172.21.15.114:36546) 2024-04-23T10:18:21.591 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:18:21.590+0000 7fdd31b67700 1 -- 172.21.15.114:0/261917642 learned_addr learned my addr 172.21.15.114:0/261917642 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-04-23T10:18:21.591 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:18:21.590+0000 7fdd31b67700 1 -- 172.21.15.114:0/261917642 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fdd2c103b00 con 0x7fdd2c100c90 2024-04-23T10:18:21.591 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:18:21.590+0000 7fdd31b67700 1 --2- 172.21.15.114:0/261917642 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fdd2c100c90 0x7fdd2c101c10 secure :-1 s=READY pgs=93 cs=0 l=1 rev1=1 crypto rx=0x7fdd2800b5c0 tx=0x7fdd28005530 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T10:18:21.591 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:18:21.590+0000 7fdd22ffd700 1 -- 172.21.15.114:0/261917642 <== mon.0 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fdd28026070 con 0x7fdd2c100c90 2024-04-23T10:18:21.591 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:18:21.590+0000 7fdd22ffd700 1 -- 172.21.15.114:0/261917642 <== mon.0 v2:172.21.15.114:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fdd2800b750 con 0x7fdd2c100c90 2024-04-23T10:18:21.591 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:18:21.590+0000 7fdd33dcb700 1 -- 172.21.15.114:0/261917642 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fdd2c1021b0 con 0x7fdd2c100c90 2024-04-23T10:18:21.591 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:18:21.590+0000 7fdd22ffd700 1 -- 172.21.15.114:0/261917642 <== mon.0 v2:172.21.15.114:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fdd28003930 con 0x7fdd2c100c90 2024-04-23T10:18:21.591 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:18:21.590+0000 7fdd33dcb700 1 -- 172.21.15.114:0/261917642 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fdd2c102670 con 0x7fdd2c100c90 2024-04-23T10:18:21.592 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:18:21.591+0000 7fdd22ffd700 1 -- 172.21.15.114:0/261917642 <== mon.0 v2:172.21.15.114:3300/0 4 ==== mgrmap(e 13) ==== 46615+0+0 (secure 0 0 0) 0x7fdd28029070 con 0x7fdd2c100c90 2024-04-23T10:18:21.593 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:18:21.591+0000 7fdd33dcb700 1 -- 172.21.15.114:0/261917642 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fdd2c0640a0 con 0x7fdd2c100c90 2024-04-23T10:18:21.593 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:18:21.591+0000 7fdd22ffd700 1 --2- 172.21.15.114:0/261917642 >> [v2:172.21.15.114:6800/465333808,v1:172.21.15.114:6801/465333808] conn(0x7fdd180396b0 0x7fdd1803bb70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T10:18:21.593 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:18:21.591+0000 7fdd22ffd700 1 -- 172.21.15.114:0/261917642 <== mon.0 v2:172.21.15.114:3300/0 5 ==== osd_map(3..3 src has 1..3) ==== 1155+0+0 (secure 0 0 0) 0x7fdd2801e070 con 0x7fdd2c100c90 2024-04-23T10:18:21.593 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:18:21.591+0000 7fdd31366700 1 --2- 172.21.15.114:0/261917642 >> [v2:172.21.15.114:6800/465333808,v1:172.21.15.114:6801/465333808] conn(0x7fdd180396b0 0x7fdd1803bb70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T10:18:21.593 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:18:21.592+0000 7fdd31366700 1 --2- 172.21.15.114:0/261917642 >> [v2:172.21.15.114:6800/465333808,v1:172.21.15.114:6801/465333808] conn(0x7fdd180396b0 0x7fdd1803bb70 secure :-1 s=READY pgs=18 cs=0 l=1 rev1=1 crypto rx=0x7fdd1c006fd0 tx=0x7fdd1c006e40 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T10:18:21.595 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:18:21.596+0000 7fdd22ffd700 1 -- 172.21.15.114:0/261917642 <== mon.0 v2:172.21.15.114:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7fdd28013020 con 0x7fdd2c100c90 2024-04-23T10:18:21.800 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:18:21.800+0000 7fdd33dcb700 1 -- 172.21.15.114:0/261917642 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_command({"prefix": "osd crush tunables", "profile": "default"} v 0) -- 0x7fdd2c103cd0 con 0x7fdd2c100c90 2024-04-23T10:18:21.851 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:18:21.852+0000 7fdd22ffd700 1 -- 172.21.15.114:0/261917642 <== mon.0 v2:172.21.15.114:3300/0 7 ==== mon_command_ack([{"prefix": "osd crush tunables", "profile": "default"}]=0 adjusted tunables profile to default v4) ==== 124+0+0 (secure 0 0 0) 0x7fdd2800e3e0 con 0x7fdd2c100c90 2024-04-23T10:18:21.854 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:18:21.854+0000 7fdd33dcb700 1 -- 172.21.15.114:0/261917642 >> [v2:172.21.15.114:6800/465333808,v1:172.21.15.114:6801/465333808] conn(0x7fdd180396b0 msgr2=0x7fdd1803bb70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T10:18:21.854 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:18:21.854+0000 7fdd33dcb700 1 --2- 172.21.15.114:0/261917642 >> [v2:172.21.15.114:6800/465333808,v1:172.21.15.114:6801/465333808] conn(0x7fdd180396b0 0x7fdd1803bb70 secure :-1 s=READY pgs=18 cs=0 l=1 rev1=1 crypto rx=0x7fdd1c006fd0 tx=0x7fdd1c006e40 comp rx=0 tx=0).stop 2024-04-23T10:18:21.854 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:18:21.854+0000 7fdd33dcb700 1 -- 172.21.15.114:0/261917642 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fdd2c100c90 msgr2=0x7fdd2c101c10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T10:18:21.854 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:18:21.854+0000 7fdd33dcb700 1 --2- 172.21.15.114:0/261917642 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fdd2c100c90 0x7fdd2c101c10 secure :-1 s=READY pgs=93 cs=0 l=1 rev1=1 crypto rx=0x7fdd2800b5c0 tx=0x7fdd28005530 comp rx=0 tx=0).stop 2024-04-23T10:18:21.854 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:18:21.854+0000 7fdd33dcb700 1 -- 172.21.15.114:0/261917642 shutdown_connections 2024-04-23T10:18:21.854 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:18:21.854+0000 7fdd33dcb700 1 --2- 172.21.15.114:0/261917642 >> [v2:172.21.15.114:6800/465333808,v1:172.21.15.114:6801/465333808] conn(0x7fdd180396b0 0x7fdd1803bb70 unknown :-1 s=CLOSED pgs=18 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:18:21.854 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:18:21.854+0000 7fdd33dcb700 1 --2- 172.21.15.114:0/261917642 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fdd2c100c90 0x7fdd2c101c10 unknown :-1 s=CLOSED pgs=93 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:18:21.854 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:18:21.854+0000 7fdd33dcb700 1 -- 172.21.15.114:0/261917642 >> 172.21.15.114:0/261917642 conn(0x7fdd2c0779f0 msgr2=0x7fdd2c104070 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T10:18:21.854 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:18:21.854+0000 7fdd33dcb700 1 -- 172.21.15.114:0/261917642 shutdown_connections 2024-04-23T10:18:21.854 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:18:21.854+0000 7fdd33dcb700 1 -- 172.21.15.114:0/261917642 wait complete. 2024-04-23T10:18:21.854 INFO:teuthology.orchestra.run.smithi114.stderr:adjusted tunables profile to default 2024-04-23T10:18:22.254 INFO:journalctl@ceph.mon.a.smithi114.stdout:Apr 23 10:18:21 smithi114 ceph-mon[29255]: from='client.? 172.21.15.114:0/261917642' entity='client.admin' cmd=[{"prefix": "osd crush tunables", "profile": "default"}]: dispatch 2024-04-23T10:18:22.441 INFO:tasks.cephadm:Adding mon.a on smithi114 2024-04-23T10:18:22.441 INFO:tasks.cephadm:Adding mon.b on smithi119 2024-04-23T10:18:22.441 INFO:tasks.cephadm:Adding mon.c on smithi204 2024-04-23T10:18:22.442 DEBUG:teuthology.orchestra.run.smithi204:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d69f13d827952bb07bb9b7973bca14f97d764d6f shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid aa240572-0159-11ef-bc93-c7b262605968 -- ceph orch apply mon '3;smithi114:172.21.15.114=a;smithi119:172.21.15.119=b;smithi204:172.21.15.204=c' 2024-04-23T10:18:22.671 INFO:teuthology.orchestra.run.smithi204.stderr:Inferring config /etc/ceph/ceph.conf 2024-04-23T10:18:22.719 INFO:teuthology.orchestra.run.smithi204.stderr:Inferring config /etc/ceph/ceph.conf 2024-04-23T10:18:23.254 INFO:journalctl@ceph.mon.a.smithi114.stdout:Apr 23 10:18:22 smithi114 ceph-mon[29255]: from='client.? 172.21.15.114:0/261917642' entity='client.admin' cmd='[{"prefix": "osd crush tunables", "profile": "default"}]': finished 2024-04-23T10:18:23.254 INFO:journalctl@ceph.mon.a.smithi114.stdout:Apr 23 10:18:22 smithi114 ceph-mon[29255]: osdmap e4: 0 total, 0 up, 0 in 2024-04-23T10:18:23.255 INFO:journalctl@ceph.mon.a.smithi114.stdout:Apr 23 10:18:22 smithi114 ceph-mon[29255]: pgmap v12: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-23T10:18:23.255 INFO:journalctl@ceph.mon.a.smithi114.stdout:Apr 23 10:18:22 smithi114 ceph-mon[29255]: from='mgr.14152 172.21.15.114:0/1529706138' entity='mgr.x' 2024-04-23T10:18:23.255 INFO:journalctl@ceph.mon.a.smithi114.stdout:Apr 23 10:18:22 smithi114 ceph-mon[29255]: from='mgr.14152 172.21.15.114:0/1529706138' entity='mgr.x' 2024-04-23T10:18:23.255 INFO:journalctl@ceph.mon.a.smithi114.stdout:Apr 23 10:18:22 smithi114 ceph-mon[29255]: from='mgr.14152 172.21.15.114:0/1529706138' entity='mgr.x' 2024-04-23T10:18:23.255 INFO:journalctl@ceph.mon.a.smithi114.stdout:Apr 23 10:18:22 smithi114 ceph-mon[29255]: from='mgr.14152 172.21.15.114:0/1529706138' entity='mgr.x' 2024-04-23T10:18:23.255 INFO:journalctl@ceph.mon.a.smithi114.stdout:Apr 23 10:18:22 smithi114 ceph-mon[29255]: from='mgr.14152 172.21.15.114:0/1529706138' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd/host:smithi204", "name": "osd_memory_target"}]: dispatch 2024-04-23T10:18:23.255 INFO:journalctl@ceph.mon.a.smithi114.stdout:Apr 23 10:18:22 smithi114 ceph-mon[29255]: from='mgr.14152 172.21.15.114:0/1529706138' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-23T10:18:23.255 INFO:journalctl@ceph.mon.a.smithi114.stdout:Apr 23 10:18:22 smithi114 ceph-mon[29255]: from='mgr.14152 172.21.15.114:0/1529706138' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-23T10:18:24.254 INFO:journalctl@ceph.mon.a.smithi114.stdout:Apr 23 10:18:23 smithi114 ceph-mon[29255]: Updating smithi204:/etc/ceph/ceph.conf 2024-04-23T10:18:24.254 INFO:journalctl@ceph.mon.a.smithi114.stdout:Apr 23 10:18:23 smithi114 ceph-mon[29255]: Updating smithi204:/var/lib/ceph/aa240572-0159-11ef-bc93-c7b262605968/config/ceph.conf 2024-04-23T10:18:24.255 INFO:journalctl@ceph.mon.a.smithi114.stdout:Apr 23 10:18:23 smithi114 ceph-mon[29255]: Updating smithi204:/etc/ceph/ceph.client.admin.keyring 2024-04-23T10:18:24.255 INFO:journalctl@ceph.mon.a.smithi114.stdout:Apr 23 10:18:23 smithi114 ceph-mon[29255]: Updating smithi204:/var/lib/ceph/aa240572-0159-11ef-bc93-c7b262605968/config/ceph.client.admin.keyring 2024-04-23T10:18:24.255 INFO:journalctl@ceph.mon.a.smithi114.stdout:Apr 23 10:18:23 smithi114 ceph-mon[29255]: from='mgr.14152 172.21.15.114:0/1529706138' entity='mgr.x' 2024-04-23T10:18:24.255 INFO:journalctl@ceph.mon.a.smithi114.stdout:Apr 23 10:18:23 smithi114 ceph-mon[29255]: from='mgr.14152 172.21.15.114:0/1529706138' entity='mgr.x' 2024-04-23T10:18:24.255 INFO:journalctl@ceph.mon.a.smithi114.stdout:Apr 23 10:18:23 smithi114 ceph-mon[29255]: from='mgr.14152 172.21.15.114:0/1529706138' entity='mgr.x' 2024-04-23T10:18:24.300 INFO:teuthology.orchestra.run.smithi204.stderr:2024-04-23T10:18:24.301+0000 7f4b4c25f700 1 -- 172.21.15.204:0/2451686441 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f4b441084b0 msgr2=0x7f4b44108890 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T10:18:24.301 INFO:teuthology.orchestra.run.smithi204.stderr:2024-04-23T10:18:24.301+0000 7f4b4c25f700 1 --2- 172.21.15.204:0/2451686441 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f4b441084b0 0x7f4b44108890 secure :-1 s=READY pgs=94 cs=0 l=1 rev1=1 crypto rx=0x7f4b40009910 tx=0x7f4b40009c20 comp rx=0 tx=0).stop 2024-04-23T10:18:24.301 INFO:teuthology.orchestra.run.smithi204.stderr:2024-04-23T10:18:24.301+0000 7f4b4c25f700 1 -- 172.21.15.204:0/2451686441 shutdown_connections 2024-04-23T10:18:24.301 INFO:teuthology.orchestra.run.smithi204.stderr:2024-04-23T10:18:24.301+0000 7f4b4c25f700 1 --2- 172.21.15.204:0/2451686441 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f4b441084b0 0x7f4b44108890 unknown :-1 s=CLOSED pgs=94 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:18:24.301 INFO:teuthology.orchestra.run.smithi204.stderr:2024-04-23T10:18:24.301+0000 7f4b4c25f700 1 -- 172.21.15.204:0/2451686441 >> 172.21.15.204:0/2451686441 conn(0x7f4b440fdf30 msgr2=0x7f4b44100350 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T10:18:24.301 INFO:teuthology.orchestra.run.smithi204.stderr:2024-04-23T10:18:24.302+0000 7f4b4c25f700 1 -- 172.21.15.204:0/2451686441 shutdown_connections 2024-04-23T10:18:24.301 INFO:teuthology.orchestra.run.smithi204.stderr:2024-04-23T10:18:24.302+0000 7f4b4c25f700 1 -- 172.21.15.204:0/2451686441 wait complete. 2024-04-23T10:18:24.303 INFO:teuthology.orchestra.run.smithi204.stderr:2024-04-23T10:18:24.302+0000 7f4b4c25f700 1 Processor -- start 2024-04-23T10:18:24.303 INFO:teuthology.orchestra.run.smithi204.stderr:2024-04-23T10:18:24.302+0000 7f4b4c25f700 1 -- start start 2024-04-23T10:18:24.303 INFO:teuthology.orchestra.run.smithi204.stderr:2024-04-23T10:18:24.302+0000 7f4b4c25f700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f4b441084b0 0x7f4b44074d40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T10:18:24.303 INFO:teuthology.orchestra.run.smithi204.stderr:2024-04-23T10:18:24.302+0000 7f4b4c25f700 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x7f4b4410c910 con 0x7f4b441084b0 2024-04-23T10:18:24.303 INFO:teuthology.orchestra.run.smithi204.stderr:2024-04-23T10:18:24.303+0000 7f4b49ffb700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f4b441084b0 0x7f4b44074d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T10:18:24.303 INFO:teuthology.orchestra.run.smithi204.stderr:2024-04-23T10:18:24.303+0000 7f4b49ffb700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f4b441084b0 0x7f4b44074d40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.204:51374/0 (socket says 172.21.15.204:51374) 2024-04-23T10:18:24.303 INFO:teuthology.orchestra.run.smithi204.stderr:2024-04-23T10:18:24.303+0000 7f4b49ffb700 1 -- 172.21.15.204:0/3472365321 learned_addr learned my addr 172.21.15.204:0/3472365321 (peer_addr_for_me v2:172.21.15.204:0/0) 2024-04-23T10:18:24.304 INFO:teuthology.orchestra.run.smithi204.stderr:2024-04-23T10:18:24.303+0000 7f4b49ffb700 1 -- 172.21.15.204:0/3472365321 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f4b44076be0 con 0x7f4b441084b0 2024-04-23T10:18:24.304 INFO:teuthology.orchestra.run.smithi204.stderr:2024-04-23T10:18:24.303+0000 7f4b49ffb700 1 --2- 172.21.15.204:0/3472365321 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f4b441084b0 0x7f4b44074d40 secure :-1 s=READY pgs=95 cs=0 l=1 rev1=1 crypto rx=0x7f4b400056d0 tx=0x7f4b400057b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T10:18:24.304 INFO:teuthology.orchestra.run.smithi204.stderr:2024-04-23T10:18:24.303+0000 7f4b3affd700 1 -- 172.21.15.204:0/3472365321 <== mon.0 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f4b40026070 con 0x7f4b441084b0 2024-04-23T10:18:24.304 INFO:teuthology.orchestra.run.smithi204.stderr:2024-04-23T10:18:24.303+0000 7f4b4c25f700 1 -- 172.21.15.204:0/3472365321 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f4b44075280 con 0x7f4b441084b0 2024-04-23T10:18:24.304 INFO:teuthology.orchestra.run.smithi204.stderr:2024-04-23T10:18:24.303+0000 7f4b3affd700 1 -- 172.21.15.204:0/3472365321 <== mon.0 v2:172.21.15.114:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f4b4000bc00 con 0x7f4b441084b0 2024-04-23T10:18:24.304 INFO:teuthology.orchestra.run.smithi204.stderr:2024-04-23T10:18:24.303+0000 7f4b3affd700 1 -- 172.21.15.204:0/3472365321 <== mon.0 v2:172.21.15.114:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f4b40003920 con 0x7f4b441084b0 2024-04-23T10:18:24.304 INFO:teuthology.orchestra.run.smithi204.stderr:2024-04-23T10:18:24.304+0000 7f4b4c25f700 1 -- 172.21.15.204:0/3472365321 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f4b44075740 con 0x7f4b441084b0 2024-04-23T10:18:24.305 INFO:teuthology.orchestra.run.smithi204.stderr:2024-04-23T10:18:24.305+0000 7f4b3affd700 1 -- 172.21.15.204:0/3472365321 <== mon.0 v2:172.21.15.114:3300/0 4 ==== mgrmap(e 13) ==== 46615+0+0 (secure 0 0 0) 0x7f4b40029070 con 0x7f4b441084b0 2024-04-23T10:18:24.305 INFO:teuthology.orchestra.run.smithi204.stderr:2024-04-23T10:18:24.304+0000 7f4b4c25f700 1 -- 172.21.15.204:0/3472365321 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f4b44103b60 con 0x7f4b441084b0 2024-04-23T10:18:24.305 INFO:teuthology.orchestra.run.smithi204.stderr:2024-04-23T10:18:24.305+0000 7f4b3affd700 1 --2- 172.21.15.204:0/3472365321 >> [v2:172.21.15.114:6800/465333808,v1:172.21.15.114:6801/465333808] conn(0x7f4b30039650 0x7f4b3003bb10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T10:18:24.305 INFO:teuthology.orchestra.run.smithi204.stderr:2024-04-23T10:18:24.305+0000 7f4b3affd700 1 -- 172.21.15.204:0/3472365321 <== mon.0 v2:172.21.15.114:3300/0 5 ==== osd_map(4..4 src has 1..4) ==== 1155+0+0 (secure 0 0 0) 0x7f4b4001e070 con 0x7f4b441084b0 2024-04-23T10:18:24.305 INFO:teuthology.orchestra.run.smithi204.stderr:2024-04-23T10:18:24.305+0000 7f4b497fa700 1 --2- 172.21.15.204:0/3472365321 >> [v2:172.21.15.114:6800/465333808,v1:172.21.15.114:6801/465333808] conn(0x7f4b30039650 0x7f4b3003bb10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T10:18:24.305 INFO:teuthology.orchestra.run.smithi204.stderr:2024-04-23T10:18:24.305+0000 7f4b497fa700 1 --2- 172.21.15.204:0/3472365321 >> [v2:172.21.15.114:6800/465333808,v1:172.21.15.114:6801/465333808] conn(0x7f4b30039650 0x7f4b3003bb10 secure :-1 s=READY pgs=19 cs=0 l=1 rev1=1 crypto rx=0x7f4b34006fd0 tx=0x7f4b34006e40 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T10:18:24.309 INFO:teuthology.orchestra.run.smithi204.stderr:2024-04-23T10:18:24.310+0000 7f4b3affd700 1 -- 172.21.15.204:0/3472365321 <== mon.0 v2:172.21.15.114:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f4b40013070 con 0x7f4b441084b0 2024-04-23T10:18:24.519 INFO:teuthology.orchestra.run.smithi204.stderr:2024-04-23T10:18:24.521+0000 7f4b4c25f700 1 -- 172.21.15.204:0/3472365321 --> [v2:172.21.15.114:6800/465333808,v1:172.21.15.114:6801/465333808] -- mgr_command(tid 0: {"prefix": "orch apply", "service_type": "mon", "placement": "3;smithi114:172.21.15.114=a;smithi119:172.21.15.119=b;smithi204:172.21.15.204=c", "target": ["mon-mgr", ""]}) -- 0x7f4b44076270 con 0x7f4b30039650 2024-04-23T10:18:24.526 INFO:teuthology.orchestra.run.smithi204.stderr:2024-04-23T10:18:24.527+0000 7f4b3affd700 1 -- 172.21.15.204:0/3472365321 <== mgr.14152 v2:172.21.15.114:6800/465333808 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+24 (secure 0 0 0) 0x7f4b44076270 con 0x7f4b30039650 2024-04-23T10:18:24.526 INFO:teuthology.orchestra.run.smithi204.stdout:Scheduled mon update... 2024-04-23T10:18:24.528 INFO:teuthology.orchestra.run.smithi204.stderr:2024-04-23T10:18:24.529+0000 7f4b4c25f700 1 -- 172.21.15.204:0/3472365321 >> [v2:172.21.15.114:6800/465333808,v1:172.21.15.114:6801/465333808] conn(0x7f4b30039650 msgr2=0x7f4b3003bb10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T10:18:24.529 INFO:teuthology.orchestra.run.smithi204.stderr:2024-04-23T10:18:24.530+0000 7f4b4c25f700 1 --2- 172.21.15.204:0/3472365321 >> [v2:172.21.15.114:6800/465333808,v1:172.21.15.114:6801/465333808] conn(0x7f4b30039650 0x7f4b3003bb10 secure :-1 s=READY pgs=19 cs=0 l=1 rev1=1 crypto rx=0x7f4b34006fd0 tx=0x7f4b34006e40 comp rx=0 tx=0).stop 2024-04-23T10:18:24.529 INFO:teuthology.orchestra.run.smithi204.stderr:2024-04-23T10:18:24.530+0000 7f4b4c25f700 1 -- 172.21.15.204:0/3472365321 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f4b441084b0 msgr2=0x7f4b44074d40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T10:18:24.529 INFO:teuthology.orchestra.run.smithi204.stderr:2024-04-23T10:18:24.530+0000 7f4b4c25f700 1 --2- 172.21.15.204:0/3472365321 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f4b441084b0 0x7f4b44074d40 secure :-1 s=READY pgs=95 cs=0 l=1 rev1=1 crypto rx=0x7f4b400056d0 tx=0x7f4b400057b0 comp rx=0 tx=0).stop 2024-04-23T10:18:24.529 INFO:teuthology.orchestra.run.smithi204.stderr:2024-04-23T10:18:24.530+0000 7f4b4c25f700 1 -- 172.21.15.204:0/3472365321 shutdown_connections 2024-04-23T10:18:24.529 INFO:teuthology.orchestra.run.smithi204.stderr:2024-04-23T10:18:24.530+0000 7f4b4c25f700 1 --2- 172.21.15.204:0/3472365321 >> [v2:172.21.15.114:6800/465333808,v1:172.21.15.114:6801/465333808] conn(0x7f4b30039650 0x7f4b3003bb10 unknown :-1 s=CLOSED pgs=19 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:18:24.529 INFO:teuthology.orchestra.run.smithi204.stderr:2024-04-23T10:18:24.530+0000 7f4b4c25f700 1 --2- 172.21.15.204:0/3472365321 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f4b441084b0 0x7f4b44074d40 unknown :-1 s=CLOSED pgs=95 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:18:24.529 INFO:teuthology.orchestra.run.smithi204.stderr:2024-04-23T10:18:24.530+0000 7f4b4c25f700 1 -- 172.21.15.204:0/3472365321 >> 172.21.15.204:0/3472365321 conn(0x7f4b440fdf30 msgr2=0x7f4b4410a030 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T10:18:24.529 INFO:teuthology.orchestra.run.smithi204.stderr:2024-04-23T10:18:24.530+0000 7f4b4c25f700 1 -- 172.21.15.204:0/3472365321 shutdown_connections 2024-04-23T10:18:24.529 INFO:teuthology.orchestra.run.smithi204.stderr:2024-04-23T10:18:24.530+0000 7f4b4c25f700 1 -- 172.21.15.204:0/3472365321 wait complete. 2024-04-23T10:18:25.172 DEBUG:teuthology.orchestra.run.smithi119:mon.b> sudo journalctl -f -n 0 -u ceph-aa240572-0159-11ef-bc93-c7b262605968@mon.b.service 2024-04-23T10:18:25.175 DEBUG:teuthology.orchestra.run.smithi204:mon.c> sudo journalctl -f -n 0 -u ceph-aa240572-0159-11ef-bc93-c7b262605968@mon.c.service 2024-04-23T10:18:25.177 INFO:tasks.cephadm:Waiting for 3 mons in monmap... 2024-04-23T10:18:25.178 DEBUG:teuthology.orchestra.run.smithi204:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d69f13d827952bb07bb9b7973bca14f97d764d6f shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid aa240572-0159-11ef-bc93-c7b262605968 -- ceph mon dump -f json 2024-04-23T10:18:25.254 INFO:journalctl@ceph.mon.a.smithi114.stdout:Apr 23 10:18:24 smithi114 ceph-mon[29255]: pgmap v13: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-23T10:18:25.254 INFO:journalctl@ceph.mon.a.smithi114.stdout:Apr 23 10:18:24 smithi114 ceph-mon[29255]: from='mgr.14152 172.21.15.114:0/1529706138' entity='mgr.x' 2024-04-23T10:18:25.254 INFO:journalctl@ceph.mon.a.smithi114.stdout:Apr 23 10:18:24 smithi114 ceph-mon[29255]: from='mgr.14152 172.21.15.114:0/1529706138' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-23T10:18:25.255 INFO:journalctl@ceph.mon.a.smithi114.stdout:Apr 23 10:18:24 smithi114 ceph-mon[29255]: from='mgr.14152 172.21.15.114:0/1529706138' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-23T10:18:25.255 INFO:journalctl@ceph.mon.a.smithi114.stdout:Apr 23 10:18:24 smithi114 ceph-mon[29255]: from='mgr.14152 172.21.15.114:0/1529706138' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-23T10:18:25.255 INFO:journalctl@ceph.mon.a.smithi114.stdout:Apr 23 10:18:24 smithi114 ceph-mon[29255]: from='mgr.14152 172.21.15.114:0/1529706138' entity='mgr.x' 2024-04-23T10:18:25.255 INFO:journalctl@ceph.mon.a.smithi114.stdout:Apr 23 10:18:24 smithi114 ceph-mon[29255]: from='mgr.14152 172.21.15.114:0/1529706138' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-04-23T10:18:25.255 INFO:journalctl@ceph.mon.a.smithi114.stdout:Apr 23 10:18:24 smithi114 ceph-mon[29255]: from='mgr.14152 172.21.15.114:0/1529706138' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-23T10:18:25.905 INFO:teuthology.orchestra.run.smithi204.stderr:Inferring config /var/lib/ceph/aa240572-0159-11ef-bc93-c7b262605968/config/ceph.conf 2024-04-23T10:18:26.254 INFO:journalctl@ceph.mon.a.smithi114.stdout:Apr 23 10:18:25 smithi114 ceph-mon[29255]: from='client.14188 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "mon", "placement": "3;smithi114:172.21.15.114=a;smithi119:172.21.15.119=b;smithi204:172.21.15.204=c", "target": ["mon-mgr", ""]}]: dispatch 2024-04-23T10:18:26.254 INFO:journalctl@ceph.mon.a.smithi114.stdout:Apr 23 10:18:25 smithi114 ceph-mon[29255]: Saving service mon spec with placement smithi114:172.21.15.114=a;smithi119:172.21.15.119=b;smithi204:172.21.15.204=c;count:3 2024-04-23T10:18:26.254 INFO:journalctl@ceph.mon.a.smithi114.stdout:Apr 23 10:18:25 smithi114 ceph-mon[29255]: Deploying daemon mon.c on smithi204 2024-04-23T10:18:27.254 INFO:journalctl@ceph.mon.a.smithi114.stdout:Apr 23 10:18:26 smithi114 ceph-mon[29255]: pgmap v14: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-23T10:18:28.821 INFO:teuthology.orchestra.run.smithi204.stderr:2024-04-23T10:18:28.820+0000 7f27aff34700 1 -- 172.21.15.204:0/2973336719 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f27a8108490 msgr2=0x7f27a8108870 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T10:18:28.821 INFO:teuthology.orchestra.run.smithi204.stderr:2024-04-23T10:18:28.820+0000 7f27aff34700 1 --2- 172.21.15.204:0/2973336719 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f27a8108490 0x7f27a8108870 secure :-1 s=READY pgs=96 cs=0 l=1 rev1=1 crypto rx=0x7f27a4009910 tx=0x7f27a4009c20 comp rx=0 tx=0).stop 2024-04-23T10:18:28.821 INFO:teuthology.orchestra.run.smithi204.stderr:2024-04-23T10:18:28.820+0000 7f27aff34700 1 -- 172.21.15.204:0/2973336719 shutdown_connections 2024-04-23T10:18:28.821 INFO:teuthology.orchestra.run.smithi204.stderr:2024-04-23T10:18:28.820+0000 7f27aff34700 1 --2- 172.21.15.204:0/2973336719 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f27a8108490 0x7f27a8108870 unknown :-1 s=CLOSED pgs=96 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:18:28.821 INFO:teuthology.orchestra.run.smithi204.stderr:2024-04-23T10:18:28.820+0000 7f27aff34700 1 -- 172.21.15.204:0/2973336719 >> 172.21.15.204:0/2973336719 conn(0x7f27a8077a30 msgr2=0x7f27a8077e40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T10:18:28.821 INFO:teuthology.orchestra.run.smithi204.stderr:2024-04-23T10:18:28.820+0000 7f27aff34700 1 -- 172.21.15.204:0/2973336719 shutdown_connections 2024-04-23T10:18:28.821 INFO:teuthology.orchestra.run.smithi204.stderr:2024-04-23T10:18:28.820+0000 7f27aff34700 1 -- 172.21.15.204:0/2973336719 wait complete. 2024-04-23T10:18:28.821 INFO:teuthology.orchestra.run.smithi204.stderr:2024-04-23T10:18:28.821+0000 7f27aff34700 1 Processor -- start 2024-04-23T10:18:28.821 INFO:teuthology.orchestra.run.smithi204.stderr:2024-04-23T10:18:28.821+0000 7f27aff34700 1 -- start start 2024-04-23T10:18:28.822 INFO:teuthology.orchestra.run.smithi204.stderr:2024-04-23T10:18:28.821+0000 7f27aff34700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f27a8108490 0x7f27a8101c10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T10:18:28.822 INFO:teuthology.orchestra.run.smithi204.stderr:2024-04-23T10:18:28.821+0000 7f27aff34700 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x7f27a810c880 con 0x7f27a8108490 2024-04-23T10:18:28.822 INFO:teuthology.orchestra.run.smithi204.stderr:2024-04-23T10:18:28.821+0000 7f27adcd0700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f27a8108490 0x7f27a8101c10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T10:18:28.822 INFO:teuthology.orchestra.run.smithi204.stderr:2024-04-23T10:18:28.821+0000 7f27adcd0700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f27a8108490 0x7f27a8101c10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.204:51390/0 (socket says 172.21.15.204:51390) 2024-04-23T10:18:28.822 INFO:teuthology.orchestra.run.smithi204.stderr:2024-04-23T10:18:28.821+0000 7f27adcd0700 1 -- 172.21.15.204:0/2143757240 learned_addr learned my addr 172.21.15.204:0/2143757240 (peer_addr_for_me v2:172.21.15.204:0/0) 2024-04-23T10:18:28.822 INFO:teuthology.orchestra.run.smithi204.stderr:2024-04-23T10:18:28.822+0000 7f27adcd0700 1 -- 172.21.15.204:0/2143757240 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f27a8103b00 con 0x7f27a8108490 2024-04-23T10:18:28.822 INFO:teuthology.orchestra.run.smithi204.stderr:2024-04-23T10:18:28.822+0000 7f27adcd0700 1 --2- 172.21.15.204:0/2143757240 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f27a8108490 0x7f27a8101c10 secure :-1 s=READY pgs=97 cs=0 l=1 rev1=1 crypto rx=0x7f27a40056d0 tx=0x7f27a40057b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T10:18:28.822 INFO:teuthology.orchestra.run.smithi204.stderr:2024-04-23T10:18:28.822+0000 7f279effd700 1 -- 172.21.15.204:0/2143757240 <== mon.0 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f27a4026070 con 0x7f27a8108490 2024-04-23T10:18:28.822 INFO:teuthology.orchestra.run.smithi204.stderr:2024-04-23T10:18:28.822+0000 7f27aff34700 1 -- 172.21.15.204:0/2143757240 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f27a81021b0 con 0x7f27a8108490 2024-04-23T10:18:28.822 INFO:teuthology.orchestra.run.smithi204.stderr:2024-04-23T10:18:28.822+0000 7f279effd700 1 -- 172.21.15.204:0/2143757240 <== mon.0 v2:172.21.15.114:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f27a400bc00 con 0x7f27a8108490 2024-04-23T10:18:28.822 INFO:teuthology.orchestra.run.smithi204.stderr:2024-04-23T10:18:28.822+0000 7f27aff34700 1 -- 172.21.15.204:0/2143757240 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f27a8102670 con 0x7f27a8108490 2024-04-23T10:18:28.822 INFO:teuthology.orchestra.run.smithi204.stderr:2024-04-23T10:18:28.822+0000 7f279effd700 1 -- 172.21.15.204:0/2143757240 <== mon.0 v2:172.21.15.114:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f27a4003910 con 0x7f27a8108490 2024-04-23T10:18:28.823 INFO:teuthology.orchestra.run.smithi204.stderr:2024-04-23T10:18:28.823+0000 7f279effd700 1 -- 172.21.15.204:0/2143757240 <== mon.0 v2:172.21.15.114:3300/0 4 ==== mgrmap(e 13) ==== 46615+0+0 (secure 0 0 0) 0x7f27a4029070 con 0x7f27a8108490 2024-04-23T10:18:28.823 INFO:teuthology.orchestra.run.smithi204.stderr:2024-04-23T10:18:28.823+0000 7f27aff34700 1 -- 172.21.15.204:0/2143757240 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f27a80640a0 con 0x7f27a8108490 2024-04-23T10:18:28.823 INFO:teuthology.orchestra.run.smithi204.stderr:2024-04-23T10:18:28.823+0000 7f279effd700 1 --2- 172.21.15.204:0/2143757240 >> [v2:172.21.15.114:6800/465333808,v1:172.21.15.114:6801/465333808] conn(0x7f2794039650 0x7f279403bb10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T10:18:28.823 INFO:teuthology.orchestra.run.smithi204.stderr:2024-04-23T10:18:28.823+0000 7f279effd700 1 -- 172.21.15.204:0/2143757240 <== mon.0 v2:172.21.15.114:3300/0 5 ==== osd_map(4..4 src has 1..4) ==== 1155+0+0 (secure 0 0 0) 0x7f27a401e070 con 0x7f27a8108490 2024-04-23T10:18:28.824 INFO:teuthology.orchestra.run.smithi204.stderr:2024-04-23T10:18:28.823+0000 7f27ad4cf700 1 --2- 172.21.15.204:0/2143757240 >> [v2:172.21.15.114:6800/465333808,v1:172.21.15.114:6801/465333808] conn(0x7f2794039650 0x7f279403bb10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T10:18:28.824 INFO:teuthology.orchestra.run.smithi204.stderr:2024-04-23T10:18:28.824+0000 7f27ad4cf700 1 --2- 172.21.15.204:0/2143757240 >> [v2:172.21.15.114:6800/465333808,v1:172.21.15.114:6801/465333808] conn(0x7f2794039650 0x7f279403bb10 secure :-1 s=READY pgs=20 cs=0 l=1 rev1=1 crypto rx=0x7f2798006fd0 tx=0x7f2798006e40 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T10:18:28.827 INFO:teuthology.orchestra.run.smithi204.stderr:2024-04-23T10:18:28.829+0000 7f279effd700 1 -- 172.21.15.204:0/2143757240 <== mon.0 v2:172.21.15.114:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f27a4013070 con 0x7f27a8108490 2024-04-23T10:18:29.144 INFO:teuthology.orchestra.run.smithi204.stderr:2024-04-23T10:18:29.144+0000 7f27aff34700 1 -- 172.21.15.204:0/2143757240 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) -- 0x7f27a8103cd0 con 0x7f27a8108490 2024-04-23T10:18:29.144 INFO:teuthology.orchestra.run.smithi204.stderr:2024-04-23T10:18:29.144+0000 7f279effd700 1 -- 172.21.15.204:0/2143757240 <== mon.0 v2:172.21.15.114:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) ==== 95+0+749 (secure 0 0 0) 0x7f27a40282c0 con 0x7f27a8108490 2024-04-23T10:18:29.144 INFO:teuthology.orchestra.run.smithi204.stdout: 2024-04-23T10:18:29.144 INFO:teuthology.orchestra.run.smithi204.stdout:{"epoch":1,"fsid":"aa240572-0159-11ef-bc93-c7b262605968","modified":"2024-04-23T10:16:50.385874Z","created":"2024-04-23T10:16:50.385874Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:3300","nonce":0},{"type":"v1","addr":"172.21.15.114:6789","nonce":0}]},"addr":"172.21.15.114:6789/0","public_addr":"172.21.15.114:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-04-23T10:18:29.146 INFO:teuthology.orchestra.run.smithi204.stderr:2024-04-23T10:18:29.147+0000 7f27aff34700 1 -- 172.21.15.204:0/2143757240 >> [v2:172.21.15.114:6800/465333808,v1:172.21.15.114:6801/465333808] conn(0x7f2794039650 msgr2=0x7f279403bb10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T10:18:29.146 INFO:teuthology.orchestra.run.smithi204.stderr:2024-04-23T10:18:29.147+0000 7f27aff34700 1 --2- 172.21.15.204:0/2143757240 >> [v2:172.21.15.114:6800/465333808,v1:172.21.15.114:6801/465333808] conn(0x7f2794039650 0x7f279403bb10 secure :-1 s=READY pgs=20 cs=0 l=1 rev1=1 crypto rx=0x7f2798006fd0 tx=0x7f2798006e40 comp rx=0 tx=0).stop 2024-04-23T10:18:29.146 INFO:teuthology.orchestra.run.smithi204.stderr:2024-04-23T10:18:29.147+0000 7f27aff34700 1 -- 172.21.15.204:0/2143757240 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f27a8108490 msgr2=0x7f27a8101c10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T10:18:29.146 INFO:teuthology.orchestra.run.smithi204.stderr:2024-04-23T10:18:29.147+0000 7f27aff34700 1 --2- 172.21.15.204:0/2143757240 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f27a8108490 0x7f27a8101c10 secure :-1 s=READY pgs=97 cs=0 l=1 rev1=1 crypto rx=0x7f27a40056d0 tx=0x7f27a40057b0 comp rx=0 tx=0).stop 2024-04-23T10:18:29.146 INFO:teuthology.orchestra.run.smithi204.stderr:2024-04-23T10:18:29.147+0000 7f27aff34700 1 -- 172.21.15.204:0/2143757240 shutdown_connections 2024-04-23T10:18:29.146 INFO:teuthology.orchestra.run.smithi204.stderr:2024-04-23T10:18:29.147+0000 7f27aff34700 1 --2- 172.21.15.204:0/2143757240 >> [v2:172.21.15.114:6800/465333808,v1:172.21.15.114:6801/465333808] conn(0x7f2794039650 0x7f279403bb10 unknown :-1 s=CLOSED pgs=20 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:18:29.146 INFO:teuthology.orchestra.run.smithi204.stderr:2024-04-23T10:18:29.147+0000 7f27aff34700 1 --2- 172.21.15.204:0/2143757240 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f27a8108490 0x7f27a8101c10 unknown :-1 s=CLOSED pgs=97 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:18:29.146 INFO:teuthology.orchestra.run.smithi204.stderr:2024-04-23T10:18:29.147+0000 7f27aff34700 1 -- 172.21.15.204:0/2143757240 >> 172.21.15.204:0/2143757240 conn(0x7f27a8077a30 msgr2=0x7f27a80fdfa0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T10:18:29.146 INFO:teuthology.orchestra.run.smithi204.stderr:2024-04-23T10:18:29.147+0000 7f27aff34700 1 -- 172.21.15.204:0/2143757240 shutdown_connections 2024-04-23T10:18:29.146 INFO:teuthology.orchestra.run.smithi204.stderr:2024-04-23T10:18:29.147+0000 7f27aff34700 1 -- 172.21.15.204:0/2143757240 wait complete. 2024-04-23T10:18:29.147 INFO:teuthology.orchestra.run.smithi204.stderr:dumped monmap epoch 1 2024-04-23T10:18:29.254 INFO:journalctl@ceph.mon.a.smithi114.stdout:Apr 23 10:18:28 smithi114 ceph-mon[29255]: pgmap v15: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-23T10:18:29.451 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 23 10:18:29 smithi204 systemd[1]: Starting Ceph mon.c for aa240572-0159-11ef-bc93-c7b262605968... 2024-04-23T10:18:30.034 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 23 10:18:29 smithi204 podman[32630]: 2024-04-23 10:18:29.834750107 +0000 UTC m=+0.018791376 image pull da4b004c09e820713483d039bc76fbd660be678f2ea3c03e059a6161b74dab76 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:068a558e3af4932cf1c60dd321c0bdf500075eb52ea1bf455e0973ba114fbfd4 2024-04-23T10:18:30.034 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 23 10:18:29 smithi204 podman[32630]: 2024-04-23 10:18:29.937673744 +0000 UTC m=+0.121715042 container create adbe74a807a5b0af3405c83cbe2d8a0564a1166fa90cea7fd099dcf9651880f8 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:068a558e3af4932cf1c60dd321c0bdf500075eb52ea1bf455e0973ba114fbfd4, name=ceph-aa240572-0159-11ef-bc93-c7b262605968-mon-c, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240416, GIT_BRANCH=HEAD, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 8 Base Image, GIT_CLEAN=True, ceph=True, maintainer=Guillaume Abrioux , RELEASE=wip-lusov-quiescer-d69f13d, io.buildah.version=1.33.5, CEPH_POINT_RELEASE=) 2024-04-23T10:18:30.254 INFO:journalctl@ceph.mon.a.smithi114.stdout:Apr 23 10:18:29 smithi114 ceph-mon[29255]: from='client.? 172.21.15.204:0/2143757240' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-04-23T10:18:30.328 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 23 10:18:30 smithi204 podman[32630]: 2024-04-23 10:18:30.131617353 +0000 UTC m=+0.315658609 container init adbe74a807a5b0af3405c83cbe2d8a0564a1166fa90cea7fd099dcf9651880f8 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:068a558e3af4932cf1c60dd321c0bdf500075eb52ea1bf455e0973ba114fbfd4, name=ceph-aa240572-0159-11ef-bc93-c7b262605968-mon-c, ceph=True, maintainer=Guillaume Abrioux , RELEASE=wip-lusov-quiescer-d69f13d, CEPH_POINT_RELEASE=, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, org.label-schema.schema-version=1.0, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.build-date=20240416, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2, GIT_CLEAN=True, io.buildah.version=1.33.5, GIT_BRANCH=HEAD) 2024-04-23T10:18:30.328 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 23 10:18:30 smithi204 podman[32630]: 2024-04-23 10:18:30.134443773 +0000 UTC m=+0.318485029 container start adbe74a807a5b0af3405c83cbe2d8a0564a1166fa90cea7fd099dcf9651880f8 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:068a558e3af4932cf1c60dd321c0bdf500075eb52ea1bf455e0973ba114fbfd4, name=ceph-aa240572-0159-11ef-bc93-c7b262605968-mon-c, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240416, io.buildah.version=1.33.5, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, ceph=True, org.label-schema.schema-version=1.0, GIT_CLEAN=True, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=wip-lusov-quiescer-d69f13d) 2024-04-23T10:18:30.328 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 23 10:18:30 smithi204 ceph-mon[32649]: set uid:gid to 167:167 (ceph:ceph) 2024-04-23T10:18:30.328 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 23 10:18:30 smithi204 ceph-mon[32649]: ceph version 19.0.0-3156-gd69f13d8 (d69f13d827952bb07bb9b7973bca14f97d764d6f) squid (dev), process ceph-mon, pid 2 2024-04-23T10:18:30.328 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 23 10:18:30 smithi204 ceph-mon[32649]: pidfile_write: ignore empty --pid-file 2024-04-23T10:18:30.329 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 23 10:18:30 smithi204 ceph-mon[32649]: load: jerasure load: lrc 2024-04-23T10:18:30.329 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 23 10:18:30 smithi204 ceph-mon[32649]: rocksdb: RocksDB version: 7.9.2 2024-04-23T10:18:30.329 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 23 10:18:30 smithi204 ceph-mon[32649]: rocksdb: Git sha 0 2024-04-23T10:18:30.329 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 23 10:18:30 smithi204 ceph-mon[32649]: rocksdb: Compile date 2024-04-22 10:57:25 2024-04-23T10:18:30.329 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 23 10:18:30 smithi204 ceph-mon[32649]: rocksdb: DB SUMMARY 2024-04-23T10:18:30.329 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 23 10:18:30 smithi204 ceph-mon[32649]: rocksdb: DB Session ID: Y6EGWCOW67BHFAJ42T8V 2024-04-23T10:18:30.329 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 23 10:18:30 smithi204 ceph-mon[32649]: rocksdb: CURRENT file: CURRENT 2024-04-23T10:18:30.329 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 23 10:18:30 smithi204 ceph-mon[32649]: rocksdb: IDENTITY file: IDENTITY 2024-04-23T10:18:30.329 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 23 10:18:30 smithi204 ceph-mon[32649]: rocksdb: MANIFEST file: MANIFEST-000005 size: 59 Bytes 2024-04-23T10:18:30.329 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 23 10:18:30 smithi204 ceph-mon[32649]: rocksdb: SST files in /var/lib/ceph/mon/ceph-c/store.db dir, Total Num: 0, files: 2024-04-23T10:18:30.329 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 23 10:18:30 smithi204 ceph-mon[32649]: rocksdb: Write Ahead Log file in /var/lib/ceph/mon/ceph-c/store.db: 000004.log size: 511 ; 2024-04-23T10:18:30.329 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 23 10:18:30 smithi204 ceph-mon[32649]: rocksdb: Options.error_if_exists: 0 2024-04-23T10:18:30.329 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 23 10:18:30 smithi204 ceph-mon[32649]: rocksdb: Options.create_if_missing: 0 2024-04-23T10:18:30.329 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 23 10:18:30 smithi204 ceph-mon[32649]: rocksdb: Options.paranoid_checks: 1 2024-04-23T10:18:30.330 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 23 10:18:30 smithi204 ceph-mon[32649]: rocksdb: Options.flush_verify_memtable_count: 1 2024-04-23T10:18:30.330 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 23 10:18:30 smithi204 ceph-mon[32649]: rocksdb: Options.track_and_verify_wals_in_manifest: 0 2024-04-23T10:18:30.330 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 23 10:18:30 smithi204 ceph-mon[32649]: rocksdb: Options.verify_sst_unique_id_in_manifest: 1 2024-04-23T10:18:30.330 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 23 10:18:30 smithi204 ceph-mon[32649]: rocksdb: Options.env: 0x555bca9e3cc0 2024-04-23T10:18:30.330 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 23 10:18:30 smithi204 ceph-mon[32649]: rocksdb: Options.fs: PosixFileSystem 2024-04-23T10:18:30.330 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 23 10:18:30 smithi204 ceph-mon[32649]: rocksdb: Options.info_log: 0x555bcc217be0 2024-04-23T10:18:30.330 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 23 10:18:30 smithi204 ceph-mon[32649]: rocksdb: Options.max_file_opening_threads: 16 2024-04-23T10:18:30.332 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 23 10:18:30 smithi204 ceph-mon[32649]: rocksdb: Options.statistics: (nil) 2024-04-23T10:18:30.332 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 23 10:18:30 smithi204 ceph-mon[32649]: rocksdb: Options.use_fsync: 0 2024-04-23T10:18:30.332 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 23 10:18:30 smithi204 ceph-mon[32649]: rocksdb: Options.max_log_file_size: 0 2024-04-23T10:18:30.332 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 23 10:18:30 smithi204 ceph-mon[32649]: rocksdb: Options.max_manifest_file_size: 1073741824 2024-04-23T10:18:30.332 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 23 10:18:30 smithi204 ceph-mon[32649]: rocksdb: Options.log_file_time_to_roll: 0 2024-04-23T10:18:30.332 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 23 10:18:30 smithi204 ceph-mon[32649]: rocksdb: Options.keep_log_file_num: 1000 2024-04-23T10:18:30.332 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 23 10:18:30 smithi204 ceph-mon[32649]: rocksdb: Options.recycle_log_file_num: 0 2024-04-23T10:18:30.332 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 23 10:18:30 smithi204 ceph-mon[32649]: rocksdb: Options.allow_fallocate: 1 2024-04-23T10:18:30.333 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 23 10:18:30 smithi204 ceph-mon[32649]: rocksdb: Options.allow_mmap_reads: 0 2024-04-23T10:18:30.333 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 23 10:18:30 smithi204 ceph-mon[32649]: rocksdb: Options.allow_mmap_writes: 0 2024-04-23T10:18:30.333 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 23 10:18:30 smithi204 ceph-mon[32649]: rocksdb: Options.use_direct_reads: 0 2024-04-23T10:18:30.333 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 23 10:18:30 smithi204 ceph-mon[32649]: rocksdb: Options.use_direct_io_for_flush_and_compaction: 0 2024-04-23T10:18:30.333 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 23 10:18:30 smithi204 ceph-mon[32649]: rocksdb: Options.create_missing_column_families: 0 2024-04-23T10:18:30.333 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 23 10:18:30 smithi204 ceph-mon[32649]: rocksdb: Options.db_log_dir: 2024-04-23T10:18:30.333 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 23 10:18:30 smithi204 ceph-mon[32649]: rocksdb: Options.wal_dir: 2024-04-23T10:18:30.333 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 23 10:18:30 smithi204 ceph-mon[32649]: rocksdb: Options.table_cache_numshardbits: 6 2024-04-23T10:18:30.333 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 23 10:18:30 smithi204 ceph-mon[32649]: rocksdb: Options.WAL_ttl_seconds: 0 2024-04-23T10:18:30.333 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 23 10:18:30 smithi204 ceph-mon[32649]: rocksdb: Options.WAL_size_limit_MB: 0 2024-04-23T10:18:30.333 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 23 10:18:30 smithi204 ceph-mon[32649]: rocksdb: Options.max_write_batch_group_size_bytes: 1048576 2024-04-23T10:18:30.333 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 23 10:18:30 smithi204 ceph-mon[32649]: rocksdb: Options.manifest_preallocation_size: 4194304 2024-04-23T10:18:30.333 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 23 10:18:30 smithi204 ceph-mon[32649]: rocksdb: Options.is_fd_close_on_exec: 1 2024-04-23T10:18:30.333 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 23 10:18:30 smithi204 ceph-mon[32649]: rocksdb: Options.advise_random_on_open: 1 2024-04-23T10:18:30.333 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 23 10:18:30 smithi204 ceph-mon[32649]: rocksdb: Options.db_write_buffer_size: 0 2024-04-23T10:18:30.334 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 23 10:18:30 smithi204 ceph-mon[32649]: rocksdb: Options.write_buffer_manager: 0x555bcc1ab0e0 2024-04-23T10:18:30.334 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 23 10:18:30 smithi204 ceph-mon[32649]: rocksdb: Options.access_hint_on_compaction_start: 1 2024-04-23T10:18:30.334 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 23 10:18:30 smithi204 ceph-mon[32649]: rocksdb: Options.random_access_max_buffer_size: 1048576 2024-04-23T10:18:30.334 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 23 10:18:30 smithi204 ceph-mon[32649]: rocksdb: Options.use_adaptive_mutex: 0 2024-04-23T10:18:30.334 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 23 10:18:30 smithi204 ceph-mon[32649]: rocksdb: Options.rate_limiter: (nil) 2024-04-23T10:18:30.334 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 23 10:18:30 smithi204 ceph-mon[32649]: rocksdb: Options.sst_file_manager.rate_bytes_per_sec: 0 2024-04-23T10:18:30.334 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 23 10:18:30 smithi204 ceph-mon[32649]: rocksdb: Options.wal_recovery_mode: 2 2024-04-23T10:18:30.334 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 23 10:18:30 smithi204 ceph-mon[32649]: rocksdb: Options.enable_thread_tracking: 0 2024-04-23T10:18:30.334 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 23 10:18:30 smithi204 ceph-mon[32649]: rocksdb: Options.enable_pipelined_write: 0 2024-04-23T10:18:30.334 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 23 10:18:30 smithi204 ceph-mon[32649]: rocksdb: Options.unordered_write: 0 2024-04-23T10:18:30.334 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 23 10:18:30 smithi204 ceph-mon[32649]: rocksdb: Options.allow_concurrent_memtable_write: 1 2024-04-23T10:18:30.334 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 23 10:18:30 smithi204 ceph-mon[32649]: rocksdb: Options.enable_write_thread_adaptive_yield: 1 2024-04-23T10:18:30.334 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 23 10:18:30 smithi204 ceph-mon[32649]: rocksdb: Options.write_thread_max_yield_usec: 100 2024-04-23T10:18:30.334 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 23 10:18:30 smithi204 ceph-mon[32649]: rocksdb: Options.write_thread_slow_yield_usec: 3 2024-04-23T10:18:30.334 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 23 10:18:30 smithi204 ceph-mon[32649]: rocksdb: Options.row_cache: None 2024-04-23T10:18:30.334 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 23 10:18:30 smithi204 ceph-mon[32649]: rocksdb: Options.wal_filter: None 2024-04-23T10:18:30.335 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 23 10:18:30 smithi204 ceph-mon[32649]: rocksdb: Options.avoid_flush_during_recovery: 0 2024-04-23T10:18:30.335 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 23 10:18:30 smithi204 ceph-mon[32649]: rocksdb: Options.allow_ingest_behind: 0 2024-04-23T10:18:30.335 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 23 10:18:30 smithi204 ceph-mon[32649]: rocksdb: Options.two_write_queues: 0 2024-04-23T10:18:30.335 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 23 10:18:30 smithi204 ceph-mon[32649]: rocksdb: Options.manual_wal_flush: 0 2024-04-23T10:18:30.335 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 23 10:18:30 smithi204 ceph-mon[32649]: rocksdb: Options.wal_compression: 0 2024-04-23T10:18:30.335 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 23 10:18:30 smithi204 ceph-mon[32649]: rocksdb: Options.atomic_flush: 0 2024-04-23T10:18:30.335 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 23 10:18:30 smithi204 ceph-mon[32649]: rocksdb: Options.avoid_unnecessary_blocking_io: 0 2024-04-23T10:18:30.335 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 23 10:18:30 smithi204 ceph-mon[32649]: rocksdb: Options.persist_stats_to_disk: 0 2024-04-23T10:18:30.335 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 23 10:18:30 smithi204 ceph-mon[32649]: rocksdb: Options.write_dbid_to_manifest: 0 2024-04-23T10:18:30.335 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 23 10:18:30 smithi204 ceph-mon[32649]: rocksdb: Options.log_readahead_size: 0 2024-04-23T10:18:30.335 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 23 10:18:30 smithi204 ceph-mon[32649]: rocksdb: Options.file_checksum_gen_factory: Unknown 2024-04-23T10:18:30.335 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 23 10:18:30 smithi204 ceph-mon[32649]: rocksdb: Options.best_efforts_recovery: 0 2024-04-23T10:18:30.335 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 23 10:18:30 smithi204 ceph-mon[32649]: rocksdb: Options.max_bgerror_resume_count: 2147483647 2024-04-23T10:18:30.335 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 23 10:18:30 smithi204 ceph-mon[32649]: rocksdb: Options.bgerror_resume_retry_interval: 1000000 2024-04-23T10:18:30.335 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 23 10:18:30 smithi204 ceph-mon[32649]: rocksdb: Options.allow_data_in_errors: 0 2024-04-23T10:18:30.336 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 23 10:18:30 smithi204 ceph-mon[32649]: rocksdb: Options.db_host_id: __hostname__ 2024-04-23T10:18:30.336 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 23 10:18:30 smithi204 ceph-mon[32649]: rocksdb: Options.enforce_single_del_contracts: true 2024-04-23T10:18:30.336 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 23 10:18:30 smithi204 ceph-mon[32649]: rocksdb: Options.max_background_jobs: 2 2024-04-23T10:18:30.336 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 23 10:18:30 smithi204 ceph-mon[32649]: rocksdb: Options.max_background_compactions: -1 2024-04-23T10:18:30.336 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 23 10:18:30 smithi204 ceph-mon[32649]: rocksdb: Options.max_subcompactions: 1 2024-04-23T10:18:30.336 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 23 10:18:30 smithi204 ceph-mon[32649]: rocksdb: Options.avoid_flush_during_shutdown: 0 2024-04-23T10:18:30.336 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 23 10:18:30 smithi204 ceph-mon[32649]: rocksdb: Options.writable_file_max_buffer_size: 1048576 2024-04-23T10:18:30.336 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 23 10:18:30 smithi204 ceph-mon[32649]: rocksdb: Options.delayed_write_rate : 16777216 2024-04-23T10:18:30.336 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 23 10:18:30 smithi204 ceph-mon[32649]: rocksdb: Options.max_total_wal_size: 0 2024-04-23T10:18:30.336 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 23 10:18:30 smithi204 ceph-mon[32649]: rocksdb: Options.delete_obsolete_files_period_micros: 21600000000 2024-04-23T10:18:30.336 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 23 10:18:30 smithi204 ceph-mon[32649]: rocksdb: Options.stats_dump_period_sec: 600 2024-04-23T10:18:30.336 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 23 10:18:30 smithi204 ceph-mon[32649]: rocksdb: Options.stats_persist_period_sec: 600 2024-04-23T10:18:30.336 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 23 10:18:30 smithi204 ceph-mon[32649]: rocksdb: Options.stats_history_buffer_size: 1048576 2024-04-23T10:18:30.336 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 23 10:18:30 smithi204 ceph-mon[32649]: rocksdb: Options.max_open_files: -1 2024-04-23T10:18:30.336 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 23 10:18:30 smithi204 ceph-mon[32649]: rocksdb: Options.bytes_per_sync: 0 2024-04-23T10:18:30.336 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 23 10:18:30 smithi204 ceph-mon[32649]: rocksdb: Options.wal_bytes_per_sync: 0 2024-04-23T10:18:30.337 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 23 10:18:30 smithi204 ceph-mon[32649]: rocksdb: Options.strict_bytes_per_sync: 0 2024-04-23T10:18:30.337 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 23 10:18:30 smithi204 ceph-mon[32649]: rocksdb: Options.compaction_readahead_size: 0 2024-04-23T10:18:30.337 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 23 10:18:30 smithi204 ceph-mon[32649]: rocksdb: Options.max_background_flushes: -1 2024-04-23T10:18:30.337 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 23 10:18:30 smithi204 ceph-mon[32649]: rocksdb: Compression algorithms supported: 2024-04-23T10:18:30.337 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 23 10:18:30 smithi204 ceph-mon[32649]: rocksdb: kZSTDNotFinalCompression supported: 0 2024-04-23T10:18:30.337 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 23 10:18:30 smithi204 ceph-mon[32649]: rocksdb: kZSTD supported: 0 2024-04-23T10:18:30.337 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 23 10:18:30 smithi204 ceph-mon[32649]: rocksdb: kXpressCompression supported: 0 2024-04-23T10:18:30.337 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 23 10:18:30 smithi204 ceph-mon[32649]: rocksdb: kLZ4HCCompression supported: 1 2024-04-23T10:18:30.337 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 23 10:18:30 smithi204 ceph-mon[32649]: rocksdb: kZlibCompression supported: 1 2024-04-23T10:18:30.337 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 23 10:18:30 smithi204 ceph-mon[32649]: rocksdb: kSnappyCompression supported: 1 2024-04-23T10:18:30.337 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 23 10:18:30 smithi204 ceph-mon[32649]: rocksdb: kLZ4Compression supported: 1 2024-04-23T10:18:30.337 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 23 10:18:30 smithi204 ceph-mon[32649]: rocksdb: kBZip2Compression supported: 0 2024-04-23T10:18:30.337 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 23 10:18:30 smithi204 ceph-mon[32649]: rocksdb: Fast CRC32 supported: Supported on x86 2024-04-23T10:18:30.337 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 23 10:18:30 smithi204 ceph-mon[32649]: rocksdb: DMutex implementation: pthread_mutex_t 2024-04-23T10:18:30.337 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 23 10:18:30 smithi204 ceph-mon[32649]: rocksdb: [db/version_set.cc:5527] Recovering from manifest file: /var/lib/ceph/mon/ceph-c/store.db/MANIFEST-000005 2024-04-23T10:18:30.337 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 23 10:18:30 smithi204 ceph-mon[32649]: rocksdb: [db/column_family.cc:630] --------------- Options for column family [default]: 2024-04-23T10:18:30.338 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 23 10:18:30 smithi204 ceph-mon[32649]: rocksdb: Options.comparator: leveldb.BytewiseComparator 2024-04-23T10:18:30.338 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 23 10:18:30 smithi204 ceph-mon[32649]: rocksdb: Options.merge_operator: 2024-04-23T10:18:30.338 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 23 10:18:30 smithi204 ceph-mon[32649]: rocksdb: Options.compaction_filter: None 2024-04-23T10:18:30.338 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 23 10:18:30 smithi204 ceph-mon[32649]: rocksdb: Options.compaction_filter_factory: None 2024-04-23T10:18:30.338 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 23 10:18:30 smithi204 ceph-mon[32649]: rocksdb: Options.sst_partitioner_factory: None 2024-04-23T10:18:30.338 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 23 10:18:30 smithi204 ceph-mon[32649]: rocksdb: Options.memtable_factory: SkipListFactory 2024-04-23T10:18:30.338 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 23 10:18:30 smithi204 ceph-mon[32649]: rocksdb: Options.table_factory: BlockBasedTable 2024-04-23T10:18:30.338 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 23 10:18:30 smithi204 ceph-mon[32649]: rocksdb: table_factory options: flush_block_policy_factory: FlushBlockBySizePolicyFactory (0x555bcc217dc0) 2024-04-23T10:18:30.338 INFO:journalctl@ceph.mon.c.smithi204.stdout: cache_index_and_filter_blocks: 1 2024-04-23T10:18:30.338 INFO:journalctl@ceph.mon.c.smithi204.stdout: cache_index_and_filter_blocks_with_high_priority: 0 2024-04-23T10:18:30.338 INFO:journalctl@ceph.mon.c.smithi204.stdout: pin_l0_filter_and_index_blocks_in_cache: 0 2024-04-23T10:18:30.338 INFO:journalctl@ceph.mon.c.smithi204.stdout: pin_top_level_index_and_filter: 1 2024-04-23T10:18:30.338 INFO:journalctl@ceph.mon.c.smithi204.stdout: index_type: 0 2024-04-23T10:18:30.338 INFO:journalctl@ceph.mon.c.smithi204.stdout: data_block_index_type: 0 2024-04-23T10:18:30.338 INFO:journalctl@ceph.mon.c.smithi204.stdout: index_shortening: 1 2024-04-23T10:18:30.338 INFO:journalctl@ceph.mon.c.smithi204.stdout: data_block_hash_table_util_ratio: 0.750000 2024-04-23T10:18:30.339 INFO:journalctl@ceph.mon.c.smithi204.stdout: checksum: 4 2024-04-23T10:18:30.339 INFO:journalctl@ceph.mon.c.smithi204.stdout: no_block_cache: 0 2024-04-23T10:18:30.339 INFO:journalctl@ceph.mon.c.smithi204.stdout: block_cache: 0x555bcc1dd1f0 2024-04-23T10:18:30.339 INFO:journalctl@ceph.mon.c.smithi204.stdout: block_cache_name: BinnedLRUCache 2024-04-23T10:18:30.339 INFO:journalctl@ceph.mon.c.smithi204.stdout: block_cache_options: 2024-04-23T10:18:30.339 INFO:journalctl@ceph.mon.c.smithi204.stdout: capacity : 536870912 2024-04-23T10:18:30.339 INFO:journalctl@ceph.mon.c.smithi204.stdout: num_shard_bits : 4 2024-04-23T10:18:30.339 INFO:journalctl@ceph.mon.c.smithi204.stdout: strict_capacity_limit : 0 2024-04-23T10:18:30.339 INFO:journalctl@ceph.mon.c.smithi204.stdout: high_pri_pool_ratio: 0.000 2024-04-23T10:18:30.339 INFO:journalctl@ceph.mon.c.smithi204.stdout: block_cache_compressed: (nil) 2024-04-23T10:18:30.339 INFO:journalctl@ceph.mon.c.smithi204.stdout: persistent_cache: (nil) 2024-04-23T10:18:30.339 INFO:journalctl@ceph.mon.c.smithi204.stdout: block_size: 4096 2024-04-23T10:18:30.339 INFO:journalctl@ceph.mon.c.smithi204.stdout: block_size_deviation: 10 2024-04-23T10:18:30.339 INFO:journalctl@ceph.mon.c.smithi204.stdout: block_restart_interval: 16 2024-04-23T10:18:30.339 INFO:journalctl@ceph.mon.c.smithi204.stdout: index_block_restart_interval: 1 2024-04-23T10:18:30.340 INFO:journalctl@ceph.mon.c.smithi204.stdout: metadata_block_size: 4096 2024-04-23T10:18:30.340 INFO:journalctl@ceph.mon.c.smithi204.stdout: partition_filters: 0 2024-04-23T10:18:30.340 INFO:journalctl@ceph.mon.c.smithi204.stdout: use_delta_encoding: 1 2024-04-23T10:18:30.340 INFO:journalctl@ceph.mon.c.smithi204.stdout: filter_policy: bloomfilter 2024-04-23T10:18:30.340 INFO:journalctl@ceph.mon.c.smithi204.stdout: whole_key_filtering: 1 2024-04-23T10:18:30.340 INFO:journalctl@ceph.mon.c.smithi204.stdout: verify_compression: 0 2024-04-23T10:18:30.340 INFO:journalctl@ceph.mon.c.smithi204.stdout: read_amp_bytes_per_bit: 0 2024-04-23T10:18:30.340 INFO:journalctl@ceph.mon.c.smithi204.stdout: format_version: 5 2024-04-23T10:18:30.340 INFO:journalctl@ceph.mon.c.smithi204.stdout: enable_index_compression: 1 2024-04-23T10:18:30.340 INFO:journalctl@ceph.mon.c.smithi204.stdout: block_align: 0 2024-04-23T10:18:30.340 INFO:journalctl@ceph.mon.c.smithi204.stdout: max_auto_readahead_size: 262144 2024-04-23T10:18:30.340 INFO:journalctl@ceph.mon.c.smithi204.stdout: prepopulate_block_cache: 0 2024-04-23T10:18:30.340 INFO:journalctl@ceph.mon.c.smithi204.stdout: initial_auto_readahead_size: 8192 2024-04-23T10:18:30.340 INFO:journalctl@ceph.mon.c.smithi204.stdout: num_file_reads_for_auto_readahead: 2 2024-04-23T10:18:30.340 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 23 10:18:30 smithi204 ceph-mon[32649]: rocksdb: Options.write_buffer_size: 33554432 2024-04-23T10:18:30.340 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 23 10:18:30 smithi204 ceph-mon[32649]: rocksdb: Options.max_write_buffer_number: 2 2024-04-23T10:18:30.341 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 23 10:18:30 smithi204 ceph-mon[32649]: rocksdb: Options.compression: NoCompression 2024-04-23T10:18:30.341 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 23 10:18:30 smithi204 ceph-mon[32649]: rocksdb: Options.bottommost_compression: Disabled 2024-04-23T10:18:30.341 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 23 10:18:30 smithi204 ceph-mon[32649]: rocksdb: Options.prefix_extractor: nullptr 2024-04-23T10:18:30.341 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 23 10:18:30 smithi204 ceph-mon[32649]: rocksdb: Options.memtable_insert_with_hint_prefix_extractor: nullptr 2024-04-23T10:18:30.341 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 23 10:18:30 smithi204 ceph-mon[32649]: rocksdb: Options.num_levels: 7 2024-04-23T10:18:30.341 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 23 10:18:30 smithi204 ceph-mon[32649]: rocksdb: Options.min_write_buffer_number_to_merge: 1 2024-04-23T10:18:30.341 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 23 10:18:30 smithi204 ceph-mon[32649]: rocksdb: Options.max_write_buffer_number_to_maintain: 0 2024-04-23T10:18:30.341 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 23 10:18:30 smithi204 ceph-mon[32649]: rocksdb: Options.max_write_buffer_size_to_maintain: 0 2024-04-23T10:18:30.341 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 23 10:18:30 smithi204 ceph-mon[32649]: rocksdb: Options.bottommost_compression_opts.window_bits: -14 2024-04-23T10:18:30.341 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 23 10:18:30 smithi204 ceph-mon[32649]: rocksdb: Options.bottommost_compression_opts.level: 32767 2024-04-23T10:18:30.341 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 23 10:18:30 smithi204 ceph-mon[32649]: rocksdb: Options.bottommost_compression_opts.strategy: 0 2024-04-23T10:18:30.341 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 23 10:18:30 smithi204 ceph-mon[32649]: rocksdb: Options.bottommost_compression_opts.max_dict_bytes: 0 2024-04-23T10:18:30.341 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 23 10:18:30 smithi204 ceph-mon[32649]: rocksdb: Options.bottommost_compression_opts.zstd_max_train_bytes: 0 2024-04-23T10:18:30.341 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 23 10:18:30 smithi204 ceph-mon[32649]: rocksdb: Options.bottommost_compression_opts.parallel_threads: 1 2024-04-23T10:18:30.341 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 23 10:18:30 smithi204 ceph-mon[32649]: rocksdb: Options.bottommost_compression_opts.enabled: false 2024-04-23T10:18:30.341 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 23 10:18:30 smithi204 ceph-mon[32649]: rocksdb: Options.bottommost_compression_opts.max_dict_buffer_bytes: 0 2024-04-23T10:18:30.341 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 23 10:18:30 smithi204 ceph-mon[32649]: rocksdb: Options.bottommost_compression_opts.use_zstd_dict_trainer: true 2024-04-23T10:18:30.342 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 23 10:18:30 smithi204 ceph-mon[32649]: rocksdb: Options.compression_opts.window_bits: -14 2024-04-23T10:18:30.342 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 23 10:18:30 smithi204 ceph-mon[32649]: rocksdb: Options.compression_opts.level: 32767 2024-04-23T10:18:30.342 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 23 10:18:30 smithi204 ceph-mon[32649]: rocksdb: Options.compression_opts.strategy: 0 2024-04-23T10:18:30.342 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 23 10:18:30 smithi204 ceph-mon[32649]: rocksdb: Options.compression_opts.max_dict_bytes: 0 2024-04-23T10:18:30.342 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 23 10:18:30 smithi204 ceph-mon[32649]: rocksdb: Options.compression_opts.zstd_max_train_bytes: 0 2024-04-23T10:18:30.342 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 23 10:18:30 smithi204 ceph-mon[32649]: rocksdb: Options.compression_opts.use_zstd_dict_trainer: true 2024-04-23T10:18:30.342 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 23 10:18:30 smithi204 ceph-mon[32649]: rocksdb: Options.compression_opts.parallel_threads: 1 2024-04-23T10:18:30.342 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 23 10:18:30 smithi204 ceph-mon[32649]: rocksdb: Options.compression_opts.enabled: false 2024-04-23T10:18:30.342 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 23 10:18:30 smithi204 ceph-mon[32649]: rocksdb: Options.compression_opts.max_dict_buffer_bytes: 0 2024-04-23T10:18:30.342 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 23 10:18:30 smithi204 ceph-mon[32649]: rocksdb: Options.level0_file_num_compaction_trigger: 4 2024-04-23T10:18:30.342 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 23 10:18:30 smithi204 ceph-mon[32649]: rocksdb: Options.level0_slowdown_writes_trigger: 20 2024-04-23T10:18:30.342 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 23 10:18:30 smithi204 ceph-mon[32649]: rocksdb: Options.level0_stop_writes_trigger: 36 2024-04-23T10:18:30.342 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 23 10:18:30 smithi204 ceph-mon[32649]: rocksdb: Options.target_file_size_base: 67108864 2024-04-23T10:18:30.343 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 23 10:18:30 smithi204 ceph-mon[32649]: rocksdb: Options.target_file_size_multiplier: 1 2024-04-23T10:18:30.343 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 23 10:18:30 smithi204 ceph-mon[32649]: rocksdb: Options.max_bytes_for_level_base: 268435456 2024-04-23T10:18:30.343 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 23 10:18:30 smithi204 ceph-mon[32649]: rocksdb: Options.level_compaction_dynamic_level_bytes: 1 2024-04-23T10:18:30.343 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 23 10:18:30 smithi204 ceph-mon[32649]: rocksdb: Options.max_bytes_for_level_multiplier: 10.000000 2024-04-23T10:18:30.343 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 23 10:18:30 smithi204 ceph-mon[32649]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[0]: 1 2024-04-23T10:18:30.343 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 23 10:18:30 smithi204 ceph-mon[32649]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[1]: 1 2024-04-23T10:18:30.343 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 23 10:18:30 smithi204 ceph-mon[32649]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[2]: 1 2024-04-23T10:18:30.343 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 23 10:18:30 smithi204 ceph-mon[32649]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[3]: 1 2024-04-23T10:18:30.343 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 23 10:18:30 smithi204 ceph-mon[32649]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[4]: 1 2024-04-23T10:18:30.343 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 23 10:18:30 smithi204 ceph-mon[32649]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[5]: 1 2024-04-23T10:18:30.343 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 23 10:18:30 smithi204 ceph-mon[32649]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[6]: 1 2024-04-23T10:18:30.343 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 23 10:18:30 smithi204 ceph-mon[32649]: rocksdb: Options.max_sequential_skip_in_iterations: 8 2024-04-23T10:18:30.343 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 23 10:18:30 smithi204 ceph-mon[32649]: rocksdb: Options.max_compaction_bytes: 1677721600 2024-04-23T10:18:30.343 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 23 10:18:30 smithi204 ceph-mon[32649]: rocksdb: Options.ignore_max_compaction_bytes_for_input: true 2024-04-23T10:18:30.343 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 23 10:18:30 smithi204 ceph-mon[32649]: rocksdb: Options.arena_block_size: 1048576 2024-04-23T10:18:30.344 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 23 10:18:30 smithi204 ceph-mon[32649]: rocksdb: Options.soft_pending_compaction_bytes_limit: 68719476736 2024-04-23T10:18:30.344 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 23 10:18:30 smithi204 ceph-mon[32649]: rocksdb: Options.hard_pending_compaction_bytes_limit: 274877906944 2024-04-23T10:18:30.344 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 23 10:18:30 smithi204 ceph-mon[32649]: rocksdb: Options.disable_auto_compactions: 0 2024-04-23T10:18:30.344 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 23 10:18:30 smithi204 ceph-mon[32649]: rocksdb: Options.compaction_style: kCompactionStyleLevel 2024-04-23T10:18:30.344 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 23 10:18:30 smithi204 ceph-mon[32649]: rocksdb: Options.compaction_pri: kMinOverlappingRatio 2024-04-23T10:18:30.344 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 23 10:18:30 smithi204 ceph-mon[32649]: rocksdb: Options.compaction_options_universal.size_ratio: 1 2024-04-23T10:18:30.344 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 23 10:18:30 smithi204 ceph-mon[32649]: rocksdb: Options.compaction_options_universal.min_merge_width: 2 2024-04-23T10:18:30.344 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 23 10:18:30 smithi204 ceph-mon[32649]: rocksdb: Options.compaction_options_universal.max_merge_width: 4294967295 2024-04-23T10:18:30.344 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 23 10:18:30 smithi204 ceph-mon[32649]: rocksdb: Options.compaction_options_universal.max_size_amplification_percent: 200 2024-04-23T10:18:30.344 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 23 10:18:30 smithi204 ceph-mon[32649]: rocksdb: Options.compaction_options_universal.compression_size_percent: -1 2024-04-23T10:18:30.344 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 23 10:18:30 smithi204 ceph-mon[32649]: rocksdb: Options.compaction_options_universal.stop_style: kCompactionStopStyleTotalSize 2024-04-23T10:18:30.344 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 23 10:18:30 smithi204 ceph-mon[32649]: rocksdb: Options.compaction_options_fifo.max_table_files_size: 1073741824 2024-04-23T10:18:30.344 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 23 10:18:30 smithi204 ceph-mon[32649]: rocksdb: Options.compaction_options_fifo.allow_compaction: 0 2024-04-23T10:18:30.344 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 23 10:18:30 smithi204 ceph-mon[32649]: rocksdb: Options.table_properties_collectors: CompactOnDeletionCollector (Sliding window size = 32768 Deletion trigger = 16384 Deletion ratio = 0); 2024-04-23T10:18:30.344 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 23 10:18:30 smithi204 ceph-mon[32649]: rocksdb: Options.inplace_update_support: 0 2024-04-23T10:18:30.344 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 23 10:18:30 smithi204 ceph-mon[32649]: rocksdb: Options.inplace_update_num_locks: 10000 2024-04-23T10:18:30.345 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 23 10:18:30 smithi204 ceph-mon[32649]: rocksdb: Options.memtable_prefix_bloom_size_ratio: 0.000000 2024-04-23T10:18:30.345 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 23 10:18:30 smithi204 ceph-mon[32649]: rocksdb: Options.memtable_whole_key_filtering: 0 2024-04-23T10:18:30.345 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 23 10:18:30 smithi204 ceph-mon[32649]: rocksdb: Options.memtable_huge_page_size: 0 2024-04-23T10:18:30.345 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 23 10:18:30 smithi204 ceph-mon[32649]: rocksdb: Options.bloom_locality: 0 2024-04-23T10:18:30.345 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 23 10:18:30 smithi204 ceph-mon[32649]: rocksdb: Options.max_successive_merges: 0 2024-04-23T10:18:30.345 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 23 10:18:30 smithi204 ceph-mon[32649]: rocksdb: Options.optimize_filters_for_hits: 0 2024-04-23T10:18:30.345 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 23 10:18:30 smithi204 ceph-mon[32649]: rocksdb: Options.paranoid_file_checks: 0 2024-04-23T10:18:30.345 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 23 10:18:30 smithi204 ceph-mon[32649]: rocksdb: Options.force_consistency_checks: 1 2024-04-23T10:18:30.345 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 23 10:18:30 smithi204 ceph-mon[32649]: rocksdb: Options.report_bg_io_stats: 0 2024-04-23T10:18:30.345 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 23 10:18:30 smithi204 ceph-mon[32649]: rocksdb: Options.ttl: 2592000 2024-04-23T10:18:30.345 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 23 10:18:30 smithi204 ceph-mon[32649]: rocksdb: Options.periodic_compaction_seconds: 0 2024-04-23T10:18:30.345 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 23 10:18:30 smithi204 ceph-mon[32649]: rocksdb: Options.preclude_last_level_data_seconds: 0 2024-04-23T10:18:30.345 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 23 10:18:30 smithi204 ceph-mon[32649]: rocksdb: Options.preserve_internal_time_seconds: 0 2024-04-23T10:18:30.345 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 23 10:18:30 smithi204 ceph-mon[32649]: rocksdb: Options.enable_blob_files: false 2024-04-23T10:18:30.345 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 23 10:18:30 smithi204 ceph-mon[32649]: rocksdb: Options.min_blob_size: 0 2024-04-23T10:18:30.345 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 23 10:18:30 smithi204 ceph-mon[32649]: rocksdb: Options.blob_file_size: 268435456 2024-04-23T10:18:30.346 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 23 10:18:30 smithi204 ceph-mon[32649]: rocksdb: Options.blob_compression_type: NoCompression 2024-04-23T10:18:30.346 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 23 10:18:30 smithi204 ceph-mon[32649]: rocksdb: Options.enable_blob_garbage_collection: false 2024-04-23T10:18:30.346 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 23 10:18:30 smithi204 ceph-mon[32649]: rocksdb: Options.blob_garbage_collection_age_cutoff: 0.250000 2024-04-23T10:18:30.346 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 23 10:18:30 smithi204 ceph-mon[32649]: rocksdb: Options.blob_garbage_collection_force_threshold: 1.000000 2024-04-23T10:18:30.346 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 23 10:18:30 smithi204 ceph-mon[32649]: rocksdb: Options.blob_compaction_readahead_size: 0 2024-04-23T10:18:30.346 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 23 10:18:30 smithi204 ceph-mon[32649]: rocksdb: Options.blob_file_starting_level: 0 2024-04-23T10:18:30.346 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 23 10:18:30 smithi204 ceph-mon[32649]: rocksdb: Options.experimental_mempurge_threshold: 0.000000 2024-04-23T10:18:30.346 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 23 10:18:30 smithi204 ceph-mon[32649]: rocksdb: [db/version_set.cc:5566] Recovered from manifest file:/var/lib/ceph/mon/ceph-c/store.db/MANIFEST-000005 succeeded,manifest_file_number is 5, next_file_number is 7, last_sequence is 0, log_number is 0,prev_log_number is 0,max_column_family is 0,min_log_number_to_keep is 0 2024-04-23T10:18:30.346 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 23 10:18:30 smithi204 ceph-mon[32649]: rocksdb: [db/version_set.cc:5581] Column family [default] (ID 0), log number is 0 2024-04-23T10:18:30.346 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 23 10:18:30 smithi204 ceph-mon[32649]: rocksdb: [db/db_impl/db_impl_open.cc:539] DB ID: c81bc822-86fe-43ca-b3d4-d6b214ad425e 2024-04-23T10:18:30.346 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 23 10:18:30 smithi204 ceph-mon[32649]: rocksdb: EVENT_LOG_v1 {"time_micros": 1713867510163982, "job": 1, "event": "recovery_started", "wal_files": [4]} 2024-04-23T10:18:30.346 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 23 10:18:30 smithi204 ceph-mon[32649]: rocksdb: [db/db_impl/db_impl_open.cc:1043] Recovering log #4 mode 2 2024-04-23T10:18:30.346 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 23 10:18:30 smithi204 ceph-mon[32649]: rocksdb: EVENT_LOG_v1 {"time_micros": 1713867510164399, "cf_name": "default", "job": 1, "event": "table_file_creation", "file_number": 8, "file_size": 1648, "file_checksum": "", "file_checksum_func_name": "Unknown", "smallest_seqno": 1, "largest_seqno": 5, "table_properties": {"data_size": 523, "index_size": 31, "index_partitions": 0, "top_level_index_size": 0, "index_key_is_user_key": 1, "index_value_is_delta_encoded": 1, "filter_size": 69, "raw_key_size": 115, "raw_average_key_size": 23, "raw_value_size": 401, "raw_average_value_size": 80, "num_data_blocks": 1, "num_entries": 5, "num_filter_entries": 5, "num_deletions": 0, "num_merge_operands": 0, "num_range_deletions": 0, "format_version": 0, "fixed_key_len": 0, "filter_policy": "bloomfilter", "column_family_name": "default", "column_family_id": 0, "comparator": "leveldb.BytewiseComparator", "merge_operator": "", "prefix_extractor_name": "nullptr", "property_collectors": "[CompactOnDeletionCollector]", "compression": "NoCompression", "compression_options": "window_bits=-14; level=32767; strategy=0; max_dict_bytes=0; zstd_max_train_bytes=0; enabled=0; max_dict_buffer_bytes=0; use_zstd_dict_trainer=1; ", "creation_time": 1713867510, "oldest_key_time": 0, "file_creation_time": 0, "slow_compression_estimated_data_size": 0, "fast_compression_estimated_data_size": 0, "db_id": "c81bc822-86fe-43ca-b3d4-d6b214ad425e", "db_session_id": "Y6EGWCOW67BHFAJ42T8V", "orig_file_number": 8, "seqno_to_time_mapping": "N/A"}} 2024-04-23T10:18:30.346 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 23 10:18:30 smithi204 ceph-mon[32649]: rocksdb: EVENT_LOG_v1 {"time_micros": 1713867510164478, "job": 1, "event": "recovery_finished"} 2024-04-23T10:18:30.346 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 23 10:18:30 smithi204 ceph-mon[32649]: rocksdb: [db/version_set.cc:5047] Creating manifest 10 2024-04-23T10:18:30.346 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 23 10:18:30 smithi204 ceph-mon[32649]: rocksdb: [file/delete_scheduler.cc:74] Deleted file /var/lib/ceph/mon/ceph-c/store.db/000004.log immediately, rate_bytes_per_sec 0, total_trash_size 0 max_trash_db_ratio 0.250000 2024-04-23T10:18:30.347 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 23 10:18:30 smithi204 ceph-mon[32649]: rocksdb: [db/db_impl/db_impl_open.cc:1987] SstFileManager instance 0x555bcc2f2000 2024-04-23T10:18:30.347 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 23 10:18:30 smithi204 ceph-mon[32649]: rocksdb: DB pointer 0x555bcc2dc000 2024-04-23T10:18:30.347 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 23 10:18:30 smithi204 ceph-mon[32649]: rocksdb: [db/db_impl/db_impl.cc:1109] ------- DUMPING STATS ------- 2024-04-23T10:18:30.347 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 23 10:18:30 smithi204 ceph-mon[32649]: rocksdb: [db/db_impl/db_impl.cc:1111] 2024-04-23T10:18:30.347 INFO:journalctl@ceph.mon.c.smithi204.stdout: ** DB Stats ** 2024-04-23T10:18:30.347 INFO:journalctl@ceph.mon.c.smithi204.stdout: Uptime(secs): 0.0 total, 0.0 interval 2024-04-23T10:18:30.347 INFO:journalctl@ceph.mon.c.smithi204.stdout: Cumulative writes: 0 writes, 0 keys, 0 commit groups, 0.0 writes per commit group, ingest: 0.00 GB, 0.00 MB/s 2024-04-23T10:18:30.347 INFO:journalctl@ceph.mon.c.smithi204.stdout: Cumulative WAL: 0 writes, 0 syncs, 0.00 writes per sync, written: 0.00 GB, 0.00 MB/s 2024-04-23T10:18:30.347 INFO:journalctl@ceph.mon.c.smithi204.stdout: Cumulative stall: 00:00:0.000 H:M:S, 0.0 percent 2024-04-23T10:18:30.347 INFO:journalctl@ceph.mon.c.smithi204.stdout: Interval writes: 0 writes, 0 keys, 0 commit groups, 0.0 writes per commit group, ingest: 0.00 MB, 0.00 MB/s 2024-04-23T10:18:30.347 INFO:journalctl@ceph.mon.c.smithi204.stdout: Interval WAL: 0 writes, 0 syncs, 0.00 writes per sync, written: 0.00 GB, 0.00 MB/s 2024-04-23T10:18:30.347 INFO:journalctl@ceph.mon.c.smithi204.stdout: Interval stall: 00:00:0.000 H:M:S, 0.0 percent 2024-04-23T10:18:30.347 INFO:journalctl@ceph.mon.c.smithi204.stdout: 2024-04-23T10:18:30.347 INFO:journalctl@ceph.mon.c.smithi204.stdout: ** Compaction Stats [default] ** 2024-04-23T10:18:30.348 INFO:journalctl@ceph.mon.c.smithi204.stdout: Level Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop Rblob(GB) Wblob(GB) 2024-04-23T10:18:30.348 INFO:journalctl@ceph.mon.c.smithi204.stdout: ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ 2024-04-23T10:18:30.348 INFO:journalctl@ceph.mon.c.smithi204.stdout: L0 1/0 1.61 KB 0.2 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 3.9 0.00 0.00 1 0.000 0 0 0.0 0.0 2024-04-23T10:18:30.348 INFO:journalctl@ceph.mon.c.smithi204.stdout: Sum 1/0 1.61 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 3.9 0.00 0.00 1 0.000 0 0 0.0 0.0 2024-04-23T10:18:30.348 INFO:journalctl@ceph.mon.c.smithi204.stdout: Int 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 3.9 0.00 0.00 1 0.000 0 0 0.0 0.0 2024-04-23T10:18:30.348 INFO:journalctl@ceph.mon.c.smithi204.stdout: 2024-04-23T10:18:30.348 INFO:journalctl@ceph.mon.c.smithi204.stdout: ** Compaction Stats [default] ** 2024-04-23T10:18:30.348 INFO:journalctl@ceph.mon.c.smithi204.stdout: Priority Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop Rblob(GB) Wblob(GB) 2024-04-23T10:18:30.348 INFO:journalctl@ceph.mon.c.smithi204.stdout: --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- 2024-04-23T10:18:30.348 INFO:journalctl@ceph.mon.c.smithi204.stdout: User 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 3.9 0.00 0.00 1 0.000 0 0 0.0 0.0 2024-04-23T10:18:30.348 INFO:journalctl@ceph.mon.c.smithi204.stdout: 2024-04-23T10:18:30.348 INFO:journalctl@ceph.mon.c.smithi204.stdout: Blob file count: 0, total size: 0.0 GB, garbage size: 0.0 GB, space amp: 0.0 2024-04-23T10:18:30.348 INFO:journalctl@ceph.mon.c.smithi204.stdout: 2024-04-23T10:18:30.348 INFO:journalctl@ceph.mon.c.smithi204.stdout: Uptime(secs): 0.0 total, 0.0 interval 2024-04-23T10:18:30.348 INFO:journalctl@ceph.mon.c.smithi204.stdout: Flush(GB): cumulative 0.000, interval 0.000 2024-04-23T10:18:30.348 INFO:journalctl@ceph.mon.c.smithi204.stdout: AddFile(GB): cumulative 0.000, interval 0.000 2024-04-23T10:18:30.349 INFO:journalctl@ceph.mon.c.smithi204.stdout: AddFile(Total Files): cumulative 0, interval 0 2024-04-23T10:18:30.349 INFO:journalctl@ceph.mon.c.smithi204.stdout: AddFile(L0 Files): cumulative 0, interval 0 2024-04-23T10:18:30.349 INFO:journalctl@ceph.mon.c.smithi204.stdout: AddFile(Keys): cumulative 0, interval 0 2024-04-23T10:18:30.349 INFO:journalctl@ceph.mon.c.smithi204.stdout: Cumulative compaction: 0.00 GB write, 0.65 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds 2024-04-23T10:18:30.349 INFO:journalctl@ceph.mon.c.smithi204.stdout: Interval compaction: 0.00 GB write, 0.65 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds 2024-04-23T10:18:30.349 INFO:journalctl@ceph.mon.c.smithi204.stdout: Stalls(count): 0 level0_slowdown, 0 level0_slowdown_with_compaction, 0 level0_numfiles, 0 level0_numfiles_with_compaction, 0 stop for pending_compaction_bytes, 0 slowdown for pending_compaction_bytes, 0 memtable_compaction, 0 memtable_slowdown, interval 0 total count 2024-04-23T10:18:30.349 INFO:journalctl@ceph.mon.c.smithi204.stdout: Block cache BinnedLRUCache@0x555bcc1dd1f0#2 capacity: 512.00 MB usage: 0.22 KB table_size: 0 occupancy: 18446744073709551615 collections: 1 last_copies: 0 last_secs: 8e-06 secs_since: 0 2024-04-23T10:18:30.349 INFO:journalctl@ceph.mon.c.smithi204.stdout: Block cache entry stats(count,size,portion): FilterBlock(1,0.11 KB,2.08616e-05%) IndexBlock(1,0.11 KB,2.08616e-05%) Misc(1,0.00 KB,0%) 2024-04-23T10:18:30.349 INFO:journalctl@ceph.mon.c.smithi204.stdout: 2024-04-23T10:18:30.349 INFO:journalctl@ceph.mon.c.smithi204.stdout: ** File Read Latency Histogram By Level [default] ** 2024-04-23T10:18:30.349 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 23 10:18:30 smithi204 ceph-mon[32649]: mon.c does not exist in monmap, will attempt to join an existing cluster 2024-04-23T10:18:30.349 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 23 10:18:30 smithi204 ceph-mon[32649]: using public_addr v2:172.21.15.204:0/0 -> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] 2024-04-23T10:18:30.349 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 23 10:18:30 smithi204 ceph-mon[32649]: starting mon.c rank -1 at public addrs [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] at bind addrs [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] mon_data /var/lib/ceph/mon/ceph-c fsid aa240572-0159-11ef-bc93-c7b262605968 2024-04-23T10:18:30.350 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 23 10:18:30 smithi204 ceph-mon[32649]: mon.c@-1(???) e0 preinit fsid aa240572-0159-11ef-bc93-c7b262605968 2024-04-23T10:18:30.350 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 23 10:18:30 smithi204 ceph-mon[32649]: mon.c@-1(synchronizing).mds e1 new map 2024-04-23T10:18:30.350 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 23 10:18:30 smithi204 ceph-mon[32649]: mon.c@-1(synchronizing).mds e1 print_map 2024-04-23T10:18:30.350 INFO:journalctl@ceph.mon.c.smithi204.stdout: e1 2024-04-23T10:18:30.350 INFO:journalctl@ceph.mon.c.smithi204.stdout: enable_multiple, ever_enabled_multiple: 1,1 2024-04-23T10:18:30.350 INFO:journalctl@ceph.mon.c.smithi204.stdout: default compat: compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,8=no anchor table,9=file layout v2,10=snaprealm v2,11=minor log segments,12=quiesce subvolumes} 2024-04-23T10:18:30.350 INFO:journalctl@ceph.mon.c.smithi204.stdout: legacy client fscid: -1 2024-04-23T10:18:30.350 INFO:journalctl@ceph.mon.c.smithi204.stdout: 2024-04-23T10:18:30.350 INFO:journalctl@ceph.mon.c.smithi204.stdout: No filesystems configured 2024-04-23T10:18:30.350 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 23 10:18:30 smithi204 ceph-mon[32649]: mon.c@-1(synchronizing).osd e0 _set_cache_ratios kv ratio 0.25 inc ratio 0.375 full ratio 0.375 2024-04-23T10:18:30.350 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 23 10:18:30 smithi204 ceph-mon[32649]: mon.c@-1(synchronizing).osd e0 register_cache_with_pcm pcm target: 2147483648 pcm max: 1020054732 pcm min: 134217728 inc_osd_cache size: 1 2024-04-23T10:18:30.350 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 23 10:18:30 smithi204 ceph-mon[32649]: mon.c@-1(synchronizing).osd e1 e1: 0 total, 0 up, 0 in 2024-04-23T10:18:30.350 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 23 10:18:30 smithi204 ceph-mon[32649]: mon.c@-1(synchronizing).osd e2 e2: 0 total, 0 up, 0 in 2024-04-23T10:18:30.351 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 23 10:18:30 smithi204 ceph-mon[32649]: mon.c@-1(synchronizing).osd e3 e3: 0 total, 0 up, 0 in 2024-04-23T10:18:30.351 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 23 10:18:30 smithi204 ceph-mon[32649]: mon.c@-1(synchronizing).osd e4 e4: 0 total, 0 up, 0 in 2024-04-23T10:18:30.351 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 23 10:18:30 smithi204 ceph-mon[32649]: mon.c@-1(synchronizing).osd e4 crush map has features 3314932999778484224, adjusting msgr requires 2024-04-23T10:18:30.351 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 23 10:18:30 smithi204 ceph-mon[32649]: mon.c@-1(synchronizing).osd e4 crush map has features 288514050185494528, adjusting msgr requires 2024-04-23T10:18:30.351 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 23 10:18:30 smithi204 ceph-mon[32649]: mon.c@-1(synchronizing).osd e4 crush map has features 288514050185494528, adjusting msgr requires 2024-04-23T10:18:30.351 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 23 10:18:30 smithi204 ceph-mon[32649]: mon.c@-1(synchronizing).osd e4 crush map has features 288514050185494528, adjusting msgr requires 2024-04-23T10:18:30.351 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 23 10:18:30 smithi204 ceph-mon[32649]: pgmap v6: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-23T10:18:30.351 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 23 10:18:30 smithi204 ceph-mon[32649]: from='mgr.14152 172.21.15.114:0/1529706138' entity='mgr.x' 2024-04-23T10:18:30.351 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 23 10:18:30 smithi204 ceph-mon[32649]: from='mgr.14152 172.21.15.114:0/1529706138' entity='mgr.x' 2024-04-23T10:18:30.351 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 23 10:18:30 smithi204 ceph-mon[32649]: from='mgr.14152 172.21.15.114:0/1529706138' entity='mgr.x' 2024-04-23T10:18:30.351 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 23 10:18:30 smithi204 ceph-mon[32649]: from='mgr.14152 172.21.15.114:0/1529706138' entity='mgr.x' 2024-04-23T10:18:30.351 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 23 10:18:30 smithi204 ceph-mon[32649]: from='mgr.14152 172.21.15.114:0/1529706138' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd/host:smithi119", "name": "osd_memory_target"}]: dispatch 2024-04-23T10:18:30.351 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 23 10:18:30 smithi204 ceph-mon[32649]: from='mgr.14152 172.21.15.114:0/1529706138' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-23T10:18:30.351 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 23 10:18:30 smithi204 ceph-mon[32649]: from='mgr.14152 172.21.15.114:0/1529706138' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-23T10:18:30.351 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 23 10:18:30 smithi204 ceph-mon[32649]: from='client.14182 -' entity='client.admin' cmd=[{"prefix": "orch host add", "hostname": "smithi204", "target": ["mon-mgr", ""]}]: dispatch 2024-04-23T10:18:30.352 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 23 10:18:30 smithi204 ceph-mon[32649]: Updating smithi119:/etc/ceph/ceph.conf 2024-04-23T10:18:30.352 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 23 10:18:30 smithi204 ceph-mon[32649]: Updating smithi119:/var/lib/ceph/aa240572-0159-11ef-bc93-c7b262605968/config/ceph.conf 2024-04-23T10:18:30.352 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 23 10:18:30 smithi204 ceph-mon[32649]: Updating smithi119:/etc/ceph/ceph.client.admin.keyring 2024-04-23T10:18:30.352 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 23 10:18:30 smithi204 ceph-mon[32649]: from='mgr.14152 172.21.15.114:0/1529706138' entity='mgr.x' 2024-04-23T10:18:30.352 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 23 10:18:30 smithi204 ceph-mon[32649]: from='mgr.14152 172.21.15.114:0/1529706138' entity='mgr.x' 2024-04-23T10:18:30.352 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 23 10:18:30 smithi204 ceph-mon[32649]: from='mgr.14152 172.21.15.114:0/1529706138' entity='mgr.x' 2024-04-23T10:18:30.352 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 23 10:18:30 smithi204 ceph-mon[32649]: Deploying cephadm binary to smithi204 2024-04-23T10:18:30.352 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 23 10:18:30 smithi204 ceph-mon[32649]: Updating smithi119:/var/lib/ceph/aa240572-0159-11ef-bc93-c7b262605968/config/ceph.client.admin.keyring 2024-04-23T10:18:30.352 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 23 10:18:30 smithi204 ceph-mon[32649]: pgmap v7: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-23T10:18:30.352 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 23 10:18:30 smithi204 ceph-mon[32649]: pgmap v8: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-23T10:18:30.352 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 23 10:18:30 smithi204 ceph-mon[32649]: from='mgr.14152 172.21.15.114:0/1529706138' entity='mgr.x' 2024-04-23T10:18:30.352 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 23 10:18:30 smithi204 ceph-mon[32649]: from='mgr.14152 172.21.15.114:0/1529706138' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-23T10:18:30.352 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 23 10:18:30 smithi204 ceph-mon[32649]: Added host smithi204 2024-04-23T10:18:30.353 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 23 10:18:30 smithi204 ceph-mon[32649]: from='mgr.14152 172.21.15.114:0/1529706138' entity='mgr.x' 2024-04-23T10:18:30.353 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 23 10:18:30 smithi204 ceph-mon[32649]: from='mgr.14152 172.21.15.114:0/1529706138' entity='mgr.x' 2024-04-23T10:18:30.353 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 23 10:18:30 smithi204 ceph-mon[32649]: pgmap v9: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-23T10:18:30.353 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 23 10:18:30 smithi204 ceph-mon[32649]: from='mgr.14152 172.21.15.114:0/1529706138' entity='mgr.x' 2024-04-23T10:18:30.353 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 23 10:18:30 smithi204 ceph-mon[32649]: from='client.14184 -' entity='client.admin' cmd=[{"prefix": "orch host ls", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-04-23T10:18:30.353 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 23 10:18:30 smithi204 ceph-mon[32649]: pgmap v10: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-23T10:18:30.353 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 23 10:18:30 smithi204 ceph-mon[32649]: from='client.? 172.21.15.114:0/261917642' entity='client.admin' cmd=[{"prefix": "osd crush tunables", "profile": "default"}]: dispatch 2024-04-23T10:18:30.353 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 23 10:18:30 smithi204 ceph-mon[32649]: from='client.? 172.21.15.114:0/261917642' entity='client.admin' cmd='[{"prefix": "osd crush tunables", "profile": "default"}]': finished 2024-04-23T10:18:30.353 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 23 10:18:30 smithi204 ceph-mon[32649]: osdmap e4: 0 total, 0 up, 0 in 2024-04-23T10:18:30.353 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 23 10:18:30 smithi204 ceph-mon[32649]: pgmap v12: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-23T10:18:30.353 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 23 10:18:30 smithi204 ceph-mon[32649]: from='mgr.14152 172.21.15.114:0/1529706138' entity='mgr.x' 2024-04-23T10:18:30.353 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 23 10:18:30 smithi204 ceph-mon[32649]: from='mgr.14152 172.21.15.114:0/1529706138' entity='mgr.x' 2024-04-23T10:18:30.353 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 23 10:18:30 smithi204 ceph-mon[32649]: from='mgr.14152 172.21.15.114:0/1529706138' entity='mgr.x' 2024-04-23T10:18:30.353 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 23 10:18:30 smithi204 ceph-mon[32649]: from='mgr.14152 172.21.15.114:0/1529706138' entity='mgr.x' 2024-04-23T10:18:30.353 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 23 10:18:30 smithi204 ceph-mon[32649]: from='mgr.14152 172.21.15.114:0/1529706138' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd/host:smithi204", "name": "osd_memory_target"}]: dispatch 2024-04-23T10:18:30.353 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 23 10:18:30 smithi204 ceph-mon[32649]: from='mgr.14152 172.21.15.114:0/1529706138' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-23T10:18:30.354 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 23 10:18:30 smithi204 ceph-mon[32649]: from='mgr.14152 172.21.15.114:0/1529706138' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-23T10:18:30.354 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 23 10:18:30 smithi204 ceph-mon[32649]: Updating smithi204:/etc/ceph/ceph.conf 2024-04-23T10:18:30.354 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 23 10:18:30 smithi204 ceph-mon[32649]: Updating smithi204:/var/lib/ceph/aa240572-0159-11ef-bc93-c7b262605968/config/ceph.conf 2024-04-23T10:18:30.354 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 23 10:18:30 smithi204 ceph-mon[32649]: Updating smithi204:/etc/ceph/ceph.client.admin.keyring 2024-04-23T10:18:30.354 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 23 10:18:30 smithi204 ceph-mon[32649]: Updating smithi204:/var/lib/ceph/aa240572-0159-11ef-bc93-c7b262605968/config/ceph.client.admin.keyring 2024-04-23T10:18:30.354 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 23 10:18:30 smithi204 ceph-mon[32649]: from='mgr.14152 172.21.15.114:0/1529706138' entity='mgr.x' 2024-04-23T10:18:30.354 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 23 10:18:30 smithi204 ceph-mon[32649]: from='mgr.14152 172.21.15.114:0/1529706138' entity='mgr.x' 2024-04-23T10:18:30.354 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 23 10:18:30 smithi204 ceph-mon[32649]: from='mgr.14152 172.21.15.114:0/1529706138' entity='mgr.x' 2024-04-23T10:18:30.354 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 23 10:18:30 smithi204 ceph-mon[32649]: pgmap v13: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-23T10:18:30.354 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 23 10:18:30 smithi204 ceph-mon[32649]: from='mgr.14152 172.21.15.114:0/1529706138' entity='mgr.x' 2024-04-23T10:18:30.354 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 23 10:18:30 smithi204 ceph-mon[32649]: from='mgr.14152 172.21.15.114:0/1529706138' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-23T10:18:30.354 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 23 10:18:30 smithi204 ceph-mon[32649]: from='mgr.14152 172.21.15.114:0/1529706138' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-23T10:18:30.354 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 23 10:18:30 smithi204 ceph-mon[32649]: from='mgr.14152 172.21.15.114:0/1529706138' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-23T10:18:30.354 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 23 10:18:30 smithi204 ceph-mon[32649]: from='mgr.14152 172.21.15.114:0/1529706138' entity='mgr.x' 2024-04-23T10:18:30.354 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 23 10:18:30 smithi204 ceph-mon[32649]: from='mgr.14152 172.21.15.114:0/1529706138' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-04-23T10:18:30.355 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 23 10:18:30 smithi204 ceph-mon[32649]: from='mgr.14152 172.21.15.114:0/1529706138' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-23T10:18:30.355 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 23 10:18:30 smithi204 ceph-mon[32649]: from='client.14188 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "mon", "placement": "3;smithi114:172.21.15.114=a;smithi119:172.21.15.119=b;smithi204:172.21.15.204=c", "target": ["mon-mgr", ""]}]: dispatch 2024-04-23T10:18:30.355 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 23 10:18:30 smithi204 ceph-mon[32649]: Saving service mon spec with placement smithi114:172.21.15.114=a;smithi119:172.21.15.119=b;smithi204:172.21.15.204=c;count:3 2024-04-23T10:18:30.355 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 23 10:18:30 smithi204 ceph-mon[32649]: Deploying daemon mon.c on smithi204 2024-04-23T10:18:30.355 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 23 10:18:30 smithi204 ceph-mon[32649]: pgmap v14: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-23T10:18:30.355 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 23 10:18:30 smithi204 ceph-mon[32649]: pgmap v15: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-23T10:18:30.355 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 23 10:18:30 smithi204 ceph-mon[32649]: from='client.? 172.21.15.204:0/2143757240' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-04-23T10:18:30.355 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 23 10:18:30 smithi204 ceph-mon[32649]: mon.c@-1(synchronizing).paxosservice(auth 1..3) refresh upgraded, format 0 -> 3 2024-04-23T10:18:30.355 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 23 10:18:30 smithi204 bash[32630]: adbe74a807a5b0af3405c83cbe2d8a0564a1166fa90cea7fd099dcf9651880f8 2024-04-23T10:18:30.355 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 23 10:18:30 smithi204 systemd[1]: Started Ceph mon.c for aa240572-0159-11ef-bc93-c7b262605968. 2024-04-23T10:18:30.781 INFO:tasks.cephadm:Waiting for 3 mons in monmap... 2024-04-23T10:18:30.781 DEBUG:teuthology.orchestra.run.smithi204:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d69f13d827952bb07bb9b7973bca14f97d764d6f shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid aa240572-0159-11ef-bc93-c7b262605968 -- ceph mon dump -f json 2024-04-23T10:18:31.014 INFO:teuthology.orchestra.run.smithi204.stderr:Inferring config /var/lib/ceph/aa240572-0159-11ef-bc93-c7b262605968/mon.c/config 2024-04-23T10:18:35.505 INFO:journalctl@ceph.mon.a.smithi114.stdout:Apr 23 10:18:35 smithi114 ceph-mon[29255]: from='mgr.14152 172.21.15.114:0/1529706138' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-04-23T10:18:35.505 INFO:journalctl@ceph.mon.a.smithi114.stdout:Apr 23 10:18:35 smithi114 ceph-mon[29255]: from='mgr.14152 172.21.15.114:0/1529706138' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-04-23T10:18:35.505 INFO:journalctl@ceph.mon.a.smithi114.stdout:Apr 23 10:18:35 smithi114 ceph-mon[29255]: mon.a calling monitor election 2024-04-23T10:18:35.505 INFO:journalctl@ceph.mon.a.smithi114.stdout:Apr 23 10:18:35 smithi114 ceph-mon[29255]: from='mgr.14152 172.21.15.114:0/1529706138' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-04-23T10:18:35.505 INFO:journalctl@ceph.mon.a.smithi114.stdout:Apr 23 10:18:35 smithi114 ceph-mon[29255]: pgmap v17: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-23T10:18:35.505 INFO:journalctl@ceph.mon.a.smithi114.stdout:Apr 23 10:18:35 smithi114 ceph-mon[29255]: from='mgr.14152 172.21.15.114:0/1529706138' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-04-23T10:18:35.505 INFO:journalctl@ceph.mon.a.smithi114.stdout:Apr 23 10:18:35 smithi114 ceph-mon[29255]: mon.c calling monitor election 2024-04-23T10:18:35.505 INFO:journalctl@ceph.mon.a.smithi114.stdout:Apr 23 10:18:35 smithi114 ceph-mon[29255]: from='mgr.14152 172.21.15.114:0/1529706138' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-04-23T10:18:35.505 INFO:journalctl@ceph.mon.a.smithi114.stdout:Apr 23 10:18:35 smithi114 ceph-mon[29255]: pgmap v18: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-23T10:18:35.505 INFO:journalctl@ceph.mon.a.smithi114.stdout:Apr 23 10:18:35 smithi114 ceph-mon[29255]: from='mgr.14152 172.21.15.114:0/1529706138' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-04-23T10:18:35.505 INFO:journalctl@ceph.mon.a.smithi114.stdout:Apr 23 10:18:35 smithi114 ceph-mon[29255]: from='mgr.14152 172.21.15.114:0/1529706138' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-04-23T10:18:35.506 INFO:journalctl@ceph.mon.a.smithi114.stdout:Apr 23 10:18:35 smithi114 ceph-mon[29255]: mon.a is new leader, mons a,c in quorum (ranks 0,1) 2024-04-23T10:18:35.506 INFO:journalctl@ceph.mon.a.smithi114.stdout:Apr 23 10:18:35 smithi114 ceph-mon[29255]: monmap epoch 2 2024-04-23T10:18:35.506 INFO:journalctl@ceph.mon.a.smithi114.stdout:Apr 23 10:18:35 smithi114 ceph-mon[29255]: fsid aa240572-0159-11ef-bc93-c7b262605968 2024-04-23T10:18:35.506 INFO:journalctl@ceph.mon.a.smithi114.stdout:Apr 23 10:18:35 smithi114 ceph-mon[29255]: last_changed 2024-04-23T10:18:30.187213+0000 2024-04-23T10:18:35.506 INFO:journalctl@ceph.mon.a.smithi114.stdout:Apr 23 10:18:35 smithi114 ceph-mon[29255]: created 2024-04-23T10:16:50.385874+0000 2024-04-23T10:18:35.506 INFO:journalctl@ceph.mon.a.smithi114.stdout:Apr 23 10:18:35 smithi114 ceph-mon[29255]: min_mon_release 19 (squid) 2024-04-23T10:18:35.506 INFO:journalctl@ceph.mon.a.smithi114.stdout:Apr 23 10:18:35 smithi114 ceph-mon[29255]: election_strategy: 1 2024-04-23T10:18:35.506 INFO:journalctl@ceph.mon.a.smithi114.stdout:Apr 23 10:18:35 smithi114 ceph-mon[29255]: 0: [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] mon.a 2024-04-23T10:18:35.506 INFO:journalctl@ceph.mon.a.smithi114.stdout:Apr 23 10:18:35 smithi114 ceph-mon[29255]: 1: [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] mon.c 2024-04-23T10:18:35.506 INFO:journalctl@ceph.mon.a.smithi114.stdout:Apr 23 10:18:35 smithi114 ceph-mon[29255]: fsmap 2024-04-23T10:18:35.506 INFO:journalctl@ceph.mon.a.smithi114.stdout:Apr 23 10:18:35 smithi114 ceph-mon[29255]: osdmap e4: 0 total, 0 up, 0 in 2024-04-23T10:18:35.506 INFO:journalctl@ceph.mon.a.smithi114.stdout:Apr 23 10:18:35 smithi114 ceph-mon[29255]: mgrmap e13: x(active, since 49s) 2024-04-23T10:18:35.506 INFO:journalctl@ceph.mon.a.smithi114.stdout:Apr 23 10:18:35 smithi114 ceph-mon[29255]: overall HEALTH_OK 2024-04-23T10:18:35.506 INFO:journalctl@ceph.mon.a.smithi114.stdout:Apr 23 10:18:35 smithi114 ceph-mon[29255]: from='mgr.14152 172.21.15.114:0/1529706138' entity='mgr.x' 2024-04-23T10:18:35.507 INFO:journalctl@ceph.mon.a.smithi114.stdout:Apr 23 10:18:35 smithi114 ceph-mon[29255]: from='mgr.14152 172.21.15.114:0/1529706138' entity='mgr.x' 2024-04-23T10:18:35.507 INFO:journalctl@ceph.mon.a.smithi114.stdout:Apr 23 10:18:35 smithi114 ceph-mon[29255]: from='mgr.14152 172.21.15.114:0/1529706138' entity='mgr.x' 2024-04-23T10:18:35.507 INFO:journalctl@ceph.mon.a.smithi114.stdout:Apr 23 10:18:35 smithi114 ceph-mon[29255]: from='mgr.14152 172.21.15.114:0/1529706138' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-04-23T10:18:35.507 INFO:journalctl@ceph.mon.a.smithi114.stdout:Apr 23 10:18:35 smithi114 ceph-mon[29255]: from='mgr.14152 172.21.15.114:0/1529706138' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-23T10:18:35.573 INFO:teuthology.orchestra.run.smithi204.stderr:2024-04-23T10:18:35.574+0000 7f90abca1700 1 -- 172.21.15.204:0/3848265364 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f90900035c0 msgr2=0x7f9090005a50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T10:18:35.573 INFO:teuthology.orchestra.run.smithi204.stderr:2024-04-23T10:18:35.574+0000 7f90abca1700 1 --2- 172.21.15.204:0/3848265364 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f90900035c0 0x7f9090005a50 secure :-1 s=READY pgs=100 cs=0 l=1 rev1=1 crypto rx=0x7f90a406ad30 tx=0x7f90a0009e80 comp rx=0 tx=0).stop 2024-04-23T10:18:35.574 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 23 10:18:35 smithi204 ceph-mon[32649]: from='mgr.14152 172.21.15.114:0/1529706138' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-04-23T10:18:35.574 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 23 10:18:35 smithi204 ceph-mon[32649]: from='mgr.14152 172.21.15.114:0/1529706138' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-04-23T10:18:35.574 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 23 10:18:35 smithi204 ceph-mon[32649]: mon.a calling monitor election 2024-04-23T10:18:35.574 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 23 10:18:35 smithi204 ceph-mon[32649]: from='mgr.14152 172.21.15.114:0/1529706138' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-04-23T10:18:35.574 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 23 10:18:35 smithi204 ceph-mon[32649]: pgmap v17: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-23T10:18:35.574 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 23 10:18:35 smithi204 ceph-mon[32649]: from='mgr.14152 172.21.15.114:0/1529706138' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-04-23T10:18:35.574 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 23 10:18:35 smithi204 ceph-mon[32649]: mon.c calling monitor election 2024-04-23T10:18:35.574 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 23 10:18:35 smithi204 ceph-mon[32649]: from='mgr.14152 172.21.15.114:0/1529706138' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-04-23T10:18:35.575 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 23 10:18:35 smithi204 ceph-mon[32649]: pgmap v18: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-23T10:18:35.575 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 23 10:18:35 smithi204 ceph-mon[32649]: from='mgr.14152 172.21.15.114:0/1529706138' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-04-23T10:18:35.575 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 23 10:18:35 smithi204 ceph-mon[32649]: from='mgr.14152 172.21.15.114:0/1529706138' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-04-23T10:18:35.575 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 23 10:18:35 smithi204 ceph-mon[32649]: mon.a is new leader, mons a,c in quorum (ranks 0,1) 2024-04-23T10:18:35.575 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 23 10:18:35 smithi204 ceph-mon[32649]: monmap epoch 2 2024-04-23T10:18:35.575 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 23 10:18:35 smithi204 ceph-mon[32649]: fsid aa240572-0159-11ef-bc93-c7b262605968 2024-04-23T10:18:35.575 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 23 10:18:35 smithi204 ceph-mon[32649]: last_changed 2024-04-23T10:18:30.187213+0000 2024-04-23T10:18:35.575 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 23 10:18:35 smithi204 ceph-mon[32649]: created 2024-04-23T10:16:50.385874+0000 2024-04-23T10:18:35.575 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 23 10:18:35 smithi204 ceph-mon[32649]: min_mon_release 19 (squid) 2024-04-23T10:18:35.575 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 23 10:18:35 smithi204 ceph-mon[32649]: election_strategy: 1 2024-04-23T10:18:35.575 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 23 10:18:35 smithi204 ceph-mon[32649]: 0: [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] mon.a 2024-04-23T10:18:35.575 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 23 10:18:35 smithi204 ceph-mon[32649]: 1: [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] mon.c 2024-04-23T10:18:35.576 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 23 10:18:35 smithi204 ceph-mon[32649]: fsmap 2024-04-23T10:18:35.576 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 23 10:18:35 smithi204 ceph-mon[32649]: osdmap e4: 0 total, 0 up, 0 in 2024-04-23T10:18:35.576 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 23 10:18:35 smithi204 ceph-mon[32649]: mgrmap e13: x(active, since 49s) 2024-04-23T10:18:35.576 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 23 10:18:35 smithi204 ceph-mon[32649]: overall HEALTH_OK 2024-04-23T10:18:35.576 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 23 10:18:35 smithi204 ceph-mon[32649]: from='mgr.14152 172.21.15.114:0/1529706138' entity='mgr.x' 2024-04-23T10:18:35.576 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 23 10:18:35 smithi204 ceph-mon[32649]: from='mgr.14152 172.21.15.114:0/1529706138' entity='mgr.x' 2024-04-23T10:18:35.576 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 23 10:18:35 smithi204 ceph-mon[32649]: from='mgr.14152 172.21.15.114:0/1529706138' entity='mgr.x' 2024-04-23T10:18:35.576 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 23 10:18:35 smithi204 ceph-mon[32649]: from='mgr.14152 172.21.15.114:0/1529706138' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-04-23T10:18:35.576 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 23 10:18:35 smithi204 ceph-mon[32649]: from='mgr.14152 172.21.15.114:0/1529706138' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-23T10:18:35.579 INFO:teuthology.orchestra.run.smithi204.stderr:2024-04-23T10:18:35.574+0000 7f90abca1700 1 -- 172.21.15.204:0/3848265364 shutdown_connections 2024-04-23T10:18:35.579 INFO:teuthology.orchestra.run.smithi204.stderr:2024-04-23T10:18:35.574+0000 7f90abca1700 1 --2- 172.21.15.204:0/3848265364 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f90900035c0 0x7f9090005a50 unknown :-1 s=CLOSED pgs=100 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:18:35.579 INFO:teuthology.orchestra.run.smithi204.stderr:2024-04-23T10:18:35.574+0000 7f90abca1700 1 -- 172.21.15.204:0/3848265364 >> 172.21.15.204:0/3848265364 conn(0x7f90a40fcf80 msgr2=0x7f90a40ff3a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T10:18:35.580 INFO:teuthology.orchestra.run.smithi204.stderr:2024-04-23T10:18:35.574+0000 7f90abca1700 1 -- 172.21.15.204:0/3848265364 shutdown_connections 2024-04-23T10:18:35.580 INFO:teuthology.orchestra.run.smithi204.stderr:2024-04-23T10:18:35.574+0000 7f90abca1700 1 -- 172.21.15.204:0/3848265364 wait complete. 2024-04-23T10:18:35.580 INFO:teuthology.orchestra.run.smithi204.stderr:2024-04-23T10:18:35.575+0000 7f90abca1700 1 Processor -- start 2024-04-23T10:18:35.580 INFO:teuthology.orchestra.run.smithi204.stderr:2024-04-23T10:18:35.575+0000 7f90abca1700 1 -- start start 2024-04-23T10:18:35.580 INFO:teuthology.orchestra.run.smithi204.stderr:2024-04-23T10:18:35.576+0000 7f90abca1700 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f90900035c0 0x7f90a419e6a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T10:18:35.580 INFO:teuthology.orchestra.run.smithi204.stderr:2024-04-23T10:18:35.576+0000 7f90abca1700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f90a419ebe0 0x7f90a41a2f70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T10:18:35.580 INFO:teuthology.orchestra.run.smithi204.stderr:2024-04-23T10:18:35.576+0000 7f90abca1700 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x7f90a4110ab0 con 0x7f90a419ebe0 2024-04-23T10:18:35.580 INFO:teuthology.orchestra.run.smithi204.stderr:2024-04-23T10:18:35.576+0000 7f90abca1700 1 -- --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_getmap magic: 0 -- 0x7f90a4110930 con 0x7f90900035c0 2024-04-23T10:18:35.580 INFO:teuthology.orchestra.run.smithi204.stderr:2024-04-23T10:18:35.576+0000 7f90a9a3d700 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f90900035c0 0x7f90a419e6a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T10:18:35.580 INFO:teuthology.orchestra.run.smithi204.stderr:2024-04-23T10:18:35.576+0000 7f90a9a3d700 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f90900035c0 0x7f90a419e6a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.204:3300/0 says I am v2:172.21.15.204:39172/0 (socket says 172.21.15.204:39172) 2024-04-23T10:18:35.580 INFO:teuthology.orchestra.run.smithi204.stderr:2024-04-23T10:18:35.576+0000 7f90a9a3d700 1 -- 172.21.15.204:0/2078698815 learned_addr learned my addr 172.21.15.204:0/2078698815 (peer_addr_for_me v2:172.21.15.204:0/0) 2024-04-23T10:18:35.580 INFO:teuthology.orchestra.run.smithi204.stderr:2024-04-23T10:18:35.576+0000 7f90a923c700 1 --2- 172.21.15.204:0/2078698815 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f90a419ebe0 0x7f90a41a2f70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T10:18:35.580 INFO:teuthology.orchestra.run.smithi204.stderr:2024-04-23T10:18:35.576+0000 7f90a9a3d700 1 -- 172.21.15.204:0/2078698815 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f90900035c0 msgr2=0x7f90a419e6a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 12 2024-04-23T10:18:35.581 INFO:teuthology.orchestra.run.smithi204.stderr:2024-04-23T10:18:35.576+0000 7f90a9a3d700 1 -- 172.21.15.204:0/2078698815 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f90900035c0 msgr2=0x7f90a419e6a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-04-23T10:18:35.581 INFO:teuthology.orchestra.run.smithi204.stderr:2024-04-23T10:18:35.576+0000 7f90a9a3d700 1 --2- 172.21.15.204:0/2078698815 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f90900035c0 0x7f90a419e6a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-04-23T10:18:35.581 INFO:teuthology.orchestra.run.smithi204.stderr:2024-04-23T10:18:35.576+0000 7f90a9a3d700 1 --2- 172.21.15.204:0/2078698815 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f90900035c0 0x7f90a419e6a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-04-23T10:18:35.581 INFO:teuthology.orchestra.run.smithi204.stderr:2024-04-23T10:18:35.576+0000 7f90a923c700 1 -- 172.21.15.204:0/2078698815 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f90900035c0 msgr2=0x7f90a419e6a0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-04-23T10:18:35.581 INFO:teuthology.orchestra.run.smithi204.stderr:2024-04-23T10:18:35.576+0000 7f90a923c700 1 --2- 172.21.15.204:0/2078698815 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f90900035c0 0x7f90a419e6a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:18:35.581 INFO:teuthology.orchestra.run.smithi204.stderr:2024-04-23T10:18:35.576+0000 7f90a923c700 1 -- 172.21.15.204:0/2078698815 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f90a419f220 con 0x7f90a419ebe0 2024-04-23T10:18:35.581 INFO:teuthology.orchestra.run.smithi204.stderr:2024-04-23T10:18:35.577+0000 7f90a923c700 1 --2- 172.21.15.204:0/2078698815 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f90a419ebe0 0x7f90a41a2f70 secure :-1 s=READY pgs=101 cs=0 l=1 rev1=1 crypto rx=0x7f909400b4d0 tx=0x7f909400b890 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T10:18:35.581 INFO:teuthology.orchestra.run.smithi204.stderr:2024-04-23T10:18:35.577+0000 7f909affd700 1 -- 172.21.15.204:0/2078698815 <== mon.0 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 ==== 309+0+0 (secure 0 0 0) 0x7f9094011630 con 0x7f90a419ebe0 2024-04-23T10:18:35.581 INFO:teuthology.orchestra.run.smithi204.stderr:2024-04-23T10:18:35.577+0000 7f909affd700 1 -- 172.21.15.204:0/2078698815 <== mon.0 v2:172.21.15.114:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f9094011c80 con 0x7f90a419ebe0 2024-04-23T10:18:35.581 INFO:teuthology.orchestra.run.smithi204.stderr:2024-04-23T10:18:35.577+0000 7f90abca1700 1 -- 172.21.15.204:0/2078698815 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f90a41a34b0 con 0x7f90a419ebe0 2024-04-23T10:18:35.581 INFO:teuthology.orchestra.run.smithi204.stderr:2024-04-23T10:18:35.577+0000 7f909affd700 1 -- 172.21.15.204:0/2078698815 <== mon.0 v2:172.21.15.114:3300/0 3 ==== mon_map magic: 0 ==== 309+0+0 (secure 0 0 0) 0x7f9094010da0 con 0x7f90a419ebe0 2024-04-23T10:18:35.581 INFO:teuthology.orchestra.run.smithi204.stderr:2024-04-23T10:18:35.577+0000 7f90abca1700 1 -- 172.21.15.204:0/2078698815 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f90a41a3a10 con 0x7f90a419ebe0 2024-04-23T10:18:35.581 INFO:teuthology.orchestra.run.smithi204.stderr:2024-04-23T10:18:35.578+0000 7f909affd700 1 -- 172.21.15.204:0/2078698815 <== mon.0 v2:172.21.15.114:3300/0 4 ==== mgrmap(e 13) ==== 46615+0+0 (secure 0 0 0) 0x7f9094018020 con 0x7f90a419ebe0 2024-04-23T10:18:35.582 INFO:teuthology.orchestra.run.smithi204.stderr:2024-04-23T10:18:35.578+0000 7f909affd700 1 --2- 172.21.15.204:0/2078698815 >> [v2:172.21.15.114:6800/465333808,v1:172.21.15.114:6801/465333808] conn(0x7f90880394b0 0x7f908803b970 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T10:18:35.582 INFO:teuthology.orchestra.run.smithi204.stderr:2024-04-23T10:18:35.579+0000 7f909affd700 1 -- 172.21.15.204:0/2078698815 <== mon.0 v2:172.21.15.114:3300/0 5 ==== osd_map(4..4 src has 1..4) ==== 1155+0+0 (secure 0 0 0) 0x7f909404c220 con 0x7f90a419ebe0 2024-04-23T10:18:35.582 INFO:teuthology.orchestra.run.smithi204.stderr:2024-04-23T10:18:35.579+0000 7f90abca1700 1 -- 172.21.15.204:0/2078698815 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f90a406b530 con 0x7f90a419ebe0 2024-04-23T10:18:35.582 INFO:teuthology.orchestra.run.smithi204.stderr:2024-04-23T10:18:35.579+0000 7f90a9a3d700 1 --2- 172.21.15.204:0/2078698815 >> [v2:172.21.15.114:6800/465333808,v1:172.21.15.114:6801/465333808] conn(0x7f90880394b0 0x7f908803b970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T10:18:35.582 INFO:teuthology.orchestra.run.smithi204.stderr:2024-04-23T10:18:35.579+0000 7f90a9a3d700 1 --2- 172.21.15.204:0/2078698815 >> [v2:172.21.15.114:6800/465333808,v1:172.21.15.114:6801/465333808] conn(0x7f90880394b0 0x7f908803b970 secure :-1 s=READY pgs=27 cs=0 l=1 rev1=1 crypto rx=0x7f90a0009a30 tx=0x7f90a00090b0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T10:18:35.583 INFO:teuthology.orchestra.run.smithi204.stderr:2024-04-23T10:18:35.584+0000 7f909affd700 1 -- 172.21.15.204:0/2078698815 <== mon.0 v2:172.21.15.114:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f909401f020 con 0x7f90a419ebe0 2024-04-23T10:18:35.866 INFO:teuthology.orchestra.run.smithi204.stderr:2024-04-23T10:18:35.867+0000 7f90abca1700 1 -- 172.21.15.204:0/2078698815 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) -- 0x7f90a41a3d20 con 0x7f90a419ebe0 2024-04-23T10:18:35.867 INFO:teuthology.orchestra.run.smithi204.stderr:2024-04-23T10:18:35.868+0000 7f909affd700 1 -- 172.21.15.204:0/2078698815 <== mon.0 v2:172.21.15.114:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 2 v2) ==== 95+0+1019 (secure 0 0 0) 0x7f9094018570 con 0x7f90a419ebe0 2024-04-23T10:18:35.867 INFO:teuthology.orchestra.run.smithi204.stdout: 2024-04-23T10:18:35.867 INFO:teuthology.orchestra.run.smithi204.stdout:{"epoch":2,"fsid":"aa240572-0159-11ef-bc93-c7b262605968","modified":"2024-04-23T10:18:30.187213Z","created":"2024-04-23T10:16:50.385874Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:3300","nonce":0},{"type":"v1","addr":"172.21.15.114:6789","nonce":0}]},"addr":"172.21.15.114:6789/0","public_addr":"172.21.15.114:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:3300","nonce":0},{"type":"v1","addr":"172.21.15.204:6789","nonce":0}]},"addr":"172.21.15.204:6789/0","public_addr":"172.21.15.204:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0,1]} 2024-04-23T10:18:35.870 INFO:teuthology.orchestra.run.smithi204.stderr:2024-04-23T10:18:35.870+0000 7f90abca1700 1 -- 172.21.15.204:0/2078698815 >> [v2:172.21.15.114:6800/465333808,v1:172.21.15.114:6801/465333808] conn(0x7f90880394b0 msgr2=0x7f908803b970 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T10:18:35.870 INFO:teuthology.orchestra.run.smithi204.stderr:2024-04-23T10:18:35.870+0000 7f90abca1700 1 --2- 172.21.15.204:0/2078698815 >> [v2:172.21.15.114:6800/465333808,v1:172.21.15.114:6801/465333808] conn(0x7f90880394b0 0x7f908803b970 secure :-1 s=READY pgs=27 cs=0 l=1 rev1=1 crypto rx=0x7f90a0009a30 tx=0x7f90a00090b0 comp rx=0 tx=0).stop 2024-04-23T10:18:35.870 INFO:teuthology.orchestra.run.smithi204.stderr:2024-04-23T10:18:35.870+0000 7f90abca1700 1 -- 172.21.15.204:0/2078698815 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f90a419ebe0 msgr2=0x7f90a41a2f70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T10:18:35.870 INFO:teuthology.orchestra.run.smithi204.stderr:2024-04-23T10:18:35.870+0000 7f90abca1700 1 --2- 172.21.15.204:0/2078698815 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f90a419ebe0 0x7f90a41a2f70 secure :-1 s=READY pgs=101 cs=0 l=1 rev1=1 crypto rx=0x7f909400b4d0 tx=0x7f909400b890 comp rx=0 tx=0).stop 2024-04-23T10:18:35.870 INFO:teuthology.orchestra.run.smithi204.stderr:2024-04-23T10:18:35.870+0000 7f90abca1700 1 -- 172.21.15.204:0/2078698815 shutdown_connections 2024-04-23T10:18:35.870 INFO:teuthology.orchestra.run.smithi204.stderr:2024-04-23T10:18:35.870+0000 7f90abca1700 1 --2- 172.21.15.204:0/2078698815 >> [v2:172.21.15.114:6800/465333808,v1:172.21.15.114:6801/465333808] conn(0x7f90880394b0 0x7f908803b970 unknown :-1 s=CLOSED pgs=27 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:18:35.870 INFO:teuthology.orchestra.run.smithi204.stderr:2024-04-23T10:18:35.870+0000 7f90abca1700 1 --2- 172.21.15.204:0/2078698815 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f90900035c0 0x7f90a419e6a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:18:35.870 INFO:teuthology.orchestra.run.smithi204.stderr:2024-04-23T10:18:35.870+0000 7f90abca1700 1 --2- 172.21.15.204:0/2078698815 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f90a419ebe0 0x7f90a41a2f70 unknown :-1 s=CLOSED pgs=101 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:18:35.870 INFO:teuthology.orchestra.run.smithi204.stderr:2024-04-23T10:18:35.870+0000 7f90abca1700 1 -- 172.21.15.204:0/2078698815 >> 172.21.15.204:0/2078698815 conn(0x7f90a40fcf80 msgr2=0x7f90a40fef50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T10:18:35.870 INFO:teuthology.orchestra.run.smithi204.stderr:2024-04-23T10:18:35.870+0000 7f90abca1700 1 -- 172.21.15.204:0/2078698815 shutdown_connections 2024-04-23T10:18:35.871 INFO:teuthology.orchestra.run.smithi204.stderr:2024-04-23T10:18:35.870+0000 7f90abca1700 1 -- 172.21.15.204:0/2078698815 wait complete. 2024-04-23T10:18:35.871 INFO:teuthology.orchestra.run.smithi204.stderr:dumped monmap epoch 2 2024-04-23T10:18:36.578 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 23 10:18:36 smithi204 ceph-mon[32649]: Deploying daemon mon.b on smithi119 2024-04-23T10:18:36.578 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 23 10:18:36 smithi204 ceph-mon[32649]: from='client.? 172.21.15.204:0/2078698815' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-04-23T10:18:36.578 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 23 10:18:36 smithi204 ceph-mon[32649]: pgmap v19: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-23T10:18:36.578 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 23 10:18:36 smithi204 ceph-mon[32649]: from='mgr.14152 172.21.15.114:0/1529706138' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-04-23T10:18:36.754 INFO:journalctl@ceph.mon.a.smithi114.stdout:Apr 23 10:18:36 smithi114 ceph-mon[29255]: Deploying daemon mon.b on smithi119 2024-04-23T10:18:36.754 INFO:journalctl@ceph.mon.a.smithi114.stdout:Apr 23 10:18:36 smithi114 ceph-mon[29255]: from='client.? 172.21.15.204:0/2078698815' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-04-23T10:18:36.754 INFO:journalctl@ceph.mon.a.smithi114.stdout:Apr 23 10:18:36 smithi114 ceph-mon[29255]: pgmap v19: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-23T10:18:36.755 INFO:journalctl@ceph.mon.a.smithi114.stdout:Apr 23 10:18:36 smithi114 ceph-mon[29255]: from='mgr.14152 172.21.15.114:0/1529706138' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-04-23T10:18:37.410 INFO:tasks.cephadm:Waiting for 3 mons in monmap... 2024-04-23T10:18:37.411 DEBUG:teuthology.orchestra.run.smithi204:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d69f13d827952bb07bb9b7973bca14f97d764d6f shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid aa240572-0159-11ef-bc93-c7b262605968 -- ceph mon dump -f json 2024-04-23T10:18:37.653 INFO:teuthology.orchestra.run.smithi204.stderr:Inferring config /var/lib/ceph/aa240572-0159-11ef-bc93-c7b262605968/mon.c/config 2024-04-23T10:18:39.036 INFO:teuthology.orchestra.run.smithi204.stderr:2024-04-23T10:18:39.037+0000 7fd76e5fa700 1 -- 172.21.15.204:0/3447758512 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fd768108630 msgr2=0x7fd768108a10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T10:18:39.037 INFO:teuthology.orchestra.run.smithi204.stderr:2024-04-23T10:18:39.037+0000 7fd76e5fa700 1 --2- 172.21.15.204:0/3447758512 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fd768108630 0x7fd768108a10 secure :-1 s=READY pgs=102 cs=0 l=1 rev1=1 crypto rx=0x7fd758009910 tx=0x7fd758009c20 comp rx=0 tx=0).stop 2024-04-23T10:18:39.037 INFO:teuthology.orchestra.run.smithi204.stderr:2024-04-23T10:18:39.037+0000 7fd76e5fa700 1 -- 172.21.15.204:0/3447758512 shutdown_connections 2024-04-23T10:18:39.037 INFO:teuthology.orchestra.run.smithi204.stderr:2024-04-23T10:18:39.037+0000 7fd76e5fa700 1 --2- 172.21.15.204:0/3447758512 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fd768108630 0x7fd768108a10 unknown :-1 s=CLOSED pgs=102 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:18:39.037 INFO:teuthology.orchestra.run.smithi204.stderr:2024-04-23T10:18:39.037+0000 7fd76e5fa700 1 -- 172.21.15.204:0/3447758512 >> 172.21.15.204:0/3447758512 conn(0x7fd7680fe050 msgr2=0x7fd768100470 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T10:18:39.037 INFO:teuthology.orchestra.run.smithi204.stderr:2024-04-23T10:18:39.037+0000 7fd76e5fa700 1 -- 172.21.15.204:0/3447758512 shutdown_connections 2024-04-23T10:18:39.037 INFO:teuthology.orchestra.run.smithi204.stderr:2024-04-23T10:18:39.037+0000 7fd76e5fa700 1 -- 172.21.15.204:0/3447758512 wait complete. 2024-04-23T10:18:39.039 INFO:teuthology.orchestra.run.smithi204.stderr:2024-04-23T10:18:39.038+0000 7fd76e5fa700 1 Processor -- start 2024-04-23T10:18:39.039 INFO:teuthology.orchestra.run.smithi204.stderr:2024-04-23T10:18:39.038+0000 7fd76e5fa700 1 -- start start 2024-04-23T10:18:39.039 INFO:teuthology.orchestra.run.smithi204.stderr:2024-04-23T10:18:39.038+0000 7fd76e5fa700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fd768108630 0x7fd7681981c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T10:18:39.039 INFO:teuthology.orchestra.run.smithi204.stderr:2024-04-23T10:18:39.038+0000 7fd76e5fa700 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fd768198700 0x7fd76819ca90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T10:18:39.039 INFO:teuthology.orchestra.run.smithi204.stderr:2024-04-23T10:18:39.038+0000 7fd76e5fa700 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x7fd76810caa0 con 0x7fd768108630 2024-04-23T10:18:39.040 INFO:teuthology.orchestra.run.smithi204.stderr:2024-04-23T10:18:39.038+0000 7fd76e5fa700 1 -- --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_getmap magic: 0 -- 0x7fd76810c920 con 0x7fd768198700 2024-04-23T10:18:39.040 INFO:teuthology.orchestra.run.smithi204.stderr:2024-04-23T10:18:39.039+0000 7fd7677fe700 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fd768198700 0x7fd76819ca90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T10:18:39.040 INFO:teuthology.orchestra.run.smithi204.stderr:2024-04-23T10:18:39.039+0000 7fd7677fe700 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fd768198700 0x7fd76819ca90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.204:3300/0 says I am v2:172.21.15.204:39176/0 (socket says 172.21.15.204:39176) 2024-04-23T10:18:39.040 INFO:teuthology.orchestra.run.smithi204.stderr:2024-04-23T10:18:39.039+0000 7fd7677fe700 1 -- 172.21.15.204:0/4292325444 learned_addr learned my addr 172.21.15.204:0/4292325444 (peer_addr_for_me v2:172.21.15.204:0/0) 2024-04-23T10:18:39.040 INFO:teuthology.orchestra.run.smithi204.stderr:2024-04-23T10:18:39.039+0000 7fd767fff700 1 --2- 172.21.15.204:0/4292325444 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fd768108630 0x7fd7681981c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T10:18:39.040 INFO:teuthology.orchestra.run.smithi204.stderr:2024-04-23T10:18:39.039+0000 7fd7677fe700 1 -- 172.21.15.204:0/4292325444 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fd768198700 msgr2=0x7fd76819ca90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 13 2024-04-23T10:18:39.040 INFO:teuthology.orchestra.run.smithi204.stderr:2024-04-23T10:18:39.039+0000 7fd7677fe700 1 -- 172.21.15.204:0/4292325444 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fd768198700 msgr2=0x7fd76819ca90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-04-23T10:18:39.040 INFO:teuthology.orchestra.run.smithi204.stderr:2024-04-23T10:18:39.039+0000 7fd7677fe700 1 --2- 172.21.15.204:0/4292325444 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fd768198700 0x7fd76819ca90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-04-23T10:18:39.040 INFO:teuthology.orchestra.run.smithi204.stderr:2024-04-23T10:18:39.039+0000 7fd7677fe700 1 --2- 172.21.15.204:0/4292325444 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fd768198700 0x7fd76819ca90 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-04-23T10:18:39.040 INFO:teuthology.orchestra.run.smithi204.stderr:2024-04-23T10:18:39.039+0000 7fd767fff700 1 -- 172.21.15.204:0/4292325444 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fd768198700 msgr2=0x7fd76819ca90 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-04-23T10:18:39.040 INFO:teuthology.orchestra.run.smithi204.stderr:2024-04-23T10:18:39.039+0000 7fd767fff700 1 --2- 172.21.15.204:0/4292325444 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fd768198700 0x7fd76819ca90 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:18:39.041 INFO:teuthology.orchestra.run.smithi204.stderr:2024-04-23T10:18:39.039+0000 7fd767fff700 1 -- 172.21.15.204:0/4292325444 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd768198d40 con 0x7fd768108630 2024-04-23T10:18:39.041 INFO:teuthology.orchestra.run.smithi204.stderr:2024-04-23T10:18:39.039+0000 7fd767fff700 1 --2- 172.21.15.204:0/4292325444 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fd768108630 0x7fd7681981c0 secure :-1 s=READY pgs=103 cs=0 l=1 rev1=1 crypto rx=0x7fd758014000 tx=0x7fd758005030 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T10:18:39.041 INFO:teuthology.orchestra.run.smithi204.stderr:2024-04-23T10:18:39.040+0000 7fd7657fa700 1 -- 172.21.15.204:0/4292325444 <== mon.0 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 ==== 309+0+0 (secure 0 0 0) 0x7fd758005dc0 con 0x7fd768108630 2024-04-23T10:18:39.041 INFO:teuthology.orchestra.run.smithi204.stderr:2024-04-23T10:18:39.040+0000 7fd7657fa700 1 -- 172.21.15.204:0/4292325444 <== mon.0 v2:172.21.15.114:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fd75800b750 con 0x7fd768108630 2024-04-23T10:18:39.041 INFO:teuthology.orchestra.run.smithi204.stderr:2024-04-23T10:18:39.040+0000 7fd76e5fa700 1 -- 172.21.15.204:0/4292325444 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fd76819cfd0 con 0x7fd768108630 2024-04-23T10:18:39.041 INFO:teuthology.orchestra.run.smithi204.stderr:2024-04-23T10:18:39.040+0000 7fd7657fa700 1 -- 172.21.15.204:0/4292325444 <== mon.0 v2:172.21.15.114:3300/0 3 ==== mon_map magic: 0 ==== 309+0+0 (secure 0 0 0) 0x7fd7580039a0 con 0x7fd768108630 2024-04-23T10:18:39.041 INFO:teuthology.orchestra.run.smithi204.stderr:2024-04-23T10:18:39.040+0000 7fd76e5fa700 1 -- 172.21.15.204:0/4292325444 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fd76819d4d0 con 0x7fd768108630 2024-04-23T10:18:39.042 INFO:teuthology.orchestra.run.smithi204.stderr:2024-04-23T10:18:39.041+0000 7fd76e5fa700 1 -- 172.21.15.204:0/4292325444 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fd768103ca0 con 0x7fd768108630 2024-04-23T10:18:39.042 INFO:teuthology.orchestra.run.smithi204.stderr:2024-04-23T10:18:39.041+0000 7fd7657fa700 1 -- 172.21.15.204:0/4292325444 <== mon.0 v2:172.21.15.114:3300/0 4 ==== mgrmap(e 13) ==== 46615+0+0 (secure 0 0 0) 0x7fd758029070 con 0x7fd768108630 2024-04-23T10:18:39.042 INFO:teuthology.orchestra.run.smithi204.stderr:2024-04-23T10:18:39.041+0000 7fd7657fa700 1 --2- 172.21.15.204:0/4292325444 >> [v2:172.21.15.114:6800/465333808,v1:172.21.15.114:6801/465333808] conn(0x7fd750039440 0x7fd75003b900 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T10:18:39.042 INFO:teuthology.orchestra.run.smithi204.stderr:2024-04-23T10:18:39.041+0000 7fd7657fa700 1 -- 172.21.15.204:0/4292325444 <== mon.0 v2:172.21.15.114:3300/0 5 ==== osd_map(4..4 src has 1..4) ==== 1155+0+0 (secure 0 0 0) 0x7fd75801e070 con 0x7fd768108630 2024-04-23T10:18:39.042 INFO:teuthology.orchestra.run.smithi204.stderr:2024-04-23T10:18:39.041+0000 7fd7677fe700 1 --2- 172.21.15.204:0/4292325444 >> [v2:172.21.15.114:6800/465333808,v1:172.21.15.114:6801/465333808] conn(0x7fd750039440 0x7fd75003b900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T10:18:39.042 INFO:teuthology.orchestra.run.smithi204.stderr:2024-04-23T10:18:39.042+0000 7fd7677fe700 1 --2- 172.21.15.204:0/4292325444 >> [v2:172.21.15.114:6800/465333808,v1:172.21.15.114:6801/465333808] conn(0x7fd750039440 0x7fd75003b900 secure :-1 s=READY pgs=30 cs=0 l=1 rev1=1 crypto rx=0x7fd75c009180 tx=0x7fd75c008040 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T10:18:39.046 INFO:teuthology.orchestra.run.smithi204.stderr:2024-04-23T10:18:39.047+0000 7fd7657fa700 1 -- 172.21.15.204:0/4292325444 <== mon.0 v2:172.21.15.114:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7fd758013070 con 0x7fd768108630 2024-04-23T10:18:39.078 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 23 10:18:38 smithi204 ceph-mon[32649]: pgmap v20: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-23T10:18:39.186 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 23 10:18:39 smithi119 podman[32588]: 2024-04-23 10:18:39.162451619 +0000 UTC m=+0.355660308 container start 903c3f3bd1b4516ae98306d6c9d5ed5e42d049cd068de3764e2881d530a5c7ad (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:068a558e3af4932cf1c60dd321c0bdf500075eb52ea1bf455e0973ba114fbfd4, name=ceph-aa240572-0159-11ef-bc93-c7b262605968-mon-b, org.label-schema.vendor=CentOS, org.label-schema.name=CentOS Stream 8 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0, maintainer=Guillaume Abrioux , io.buildah.version=1.33.5, RELEASE=wip-lusov-quiescer-d69f13d, GIT_BRANCH=HEAD, ceph=True, GIT_CLEAN=True, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, org.label-schema.build-date=20240416, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67) 2024-04-23T10:18:39.186 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 23 10:18:39 smithi119 ceph-mon[32608]: set uid:gid to 167:167 (ceph:ceph) 2024-04-23T10:18:39.187 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 23 10:18:39 smithi119 ceph-mon[32608]: ceph version 19.0.0-3156-gd69f13d8 (d69f13d827952bb07bb9b7973bca14f97d764d6f) squid (dev), process ceph-mon, pid 2 2024-04-23T10:18:39.187 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 23 10:18:39 smithi119 ceph-mon[32608]: pidfile_write: ignore empty --pid-file 2024-04-23T10:18:39.217 INFO:teuthology.orchestra.run.smithi204.stderr:2024-04-23T10:18:39.218+0000 7fd7657fa700 1 -- 172.21.15.204:0/4292325444 <== mon.0 v2:172.21.15.114:3300/0 7 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd75802ae20 con 0x7fd768108630 2024-04-23T10:18:39.254 INFO:journalctl@ceph.mon.a.smithi114.stdout:Apr 23 10:18:38 smithi114 ceph-mon[29255]: pgmap v20: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-23T10:18:39.331 INFO:teuthology.orchestra.run.smithi204.stderr:2024-04-23T10:18:39.332+0000 7fd76e5fa700 1 -- 172.21.15.204:0/4292325444 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) -- 0x7fd76819d780 con 0x7fd768108630 2024-04-23T10:18:39.447 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 23 10:18:39 smithi119 ceph-mon[32608]: load: jerasure load: lrc 2024-04-23T10:18:39.447 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 23 10:18:39 smithi119 ceph-mon[32608]: rocksdb: RocksDB version: 7.9.2 2024-04-23T10:18:39.447 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 23 10:18:39 smithi119 ceph-mon[32608]: rocksdb: Git sha 0 2024-04-23T10:18:39.448 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 23 10:18:39 smithi119 ceph-mon[32608]: rocksdb: Compile date 2024-04-22 10:57:25 2024-04-23T10:18:39.448 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 23 10:18:39 smithi119 ceph-mon[32608]: rocksdb: DB SUMMARY 2024-04-23T10:18:39.448 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 23 10:18:39 smithi119 ceph-mon[32608]: rocksdb: DB Session ID: JU99V4O8SURC5HIG6FAI 2024-04-23T10:18:39.448 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 23 10:18:39 smithi119 ceph-mon[32608]: rocksdb: CURRENT file: CURRENT 2024-04-23T10:18:39.448 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 23 10:18:39 smithi119 ceph-mon[32608]: rocksdb: IDENTITY file: IDENTITY 2024-04-23T10:18:39.448 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 23 10:18:39 smithi119 ceph-mon[32608]: rocksdb: MANIFEST file: MANIFEST-000005 size: 59 Bytes 2024-04-23T10:18:39.448 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 23 10:18:39 smithi119 ceph-mon[32608]: rocksdb: SST files in /var/lib/ceph/mon/ceph-b/store.db dir, Total Num: 0, files: 2024-04-23T10:18:39.448 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 23 10:18:39 smithi119 ceph-mon[32608]: rocksdb: Write Ahead Log file in /var/lib/ceph/mon/ceph-b/store.db: 000004.log size: 636 ; 2024-04-23T10:18:39.448 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 23 10:18:39 smithi119 ceph-mon[32608]: rocksdb: Options.error_if_exists: 0 2024-04-23T10:18:39.448 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 23 10:18:39 smithi119 ceph-mon[32608]: rocksdb: Options.create_if_missing: 0 2024-04-23T10:18:39.449 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 23 10:18:39 smithi119 ceph-mon[32608]: rocksdb: Options.paranoid_checks: 1 2024-04-23T10:18:39.449 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 23 10:18:39 smithi119 ceph-mon[32608]: rocksdb: Options.flush_verify_memtable_count: 1 2024-04-23T10:18:39.449 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 23 10:18:39 smithi119 ceph-mon[32608]: rocksdb: Options.track_and_verify_wals_in_manifest: 0 2024-04-23T10:18:39.449 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 23 10:18:39 smithi119 ceph-mon[32608]: rocksdb: Options.verify_sst_unique_id_in_manifest: 1 2024-04-23T10:18:39.449 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 23 10:18:39 smithi119 ceph-mon[32608]: rocksdb: Options.env: 0x5572472c3cc0 2024-04-23T10:18:39.449 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 23 10:18:39 smithi119 ceph-mon[32608]: rocksdb: Options.fs: PosixFileSystem 2024-04-23T10:18:39.449 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 23 10:18:39 smithi119 ceph-mon[32608]: rocksdb: Options.info_log: 0x55724965bbe0 2024-04-23T10:18:39.449 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 23 10:18:39 smithi119 ceph-mon[32608]: rocksdb: Options.max_file_opening_threads: 16 2024-04-23T10:18:39.449 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 23 10:18:39 smithi119 ceph-mon[32608]: rocksdb: Options.statistics: (nil) 2024-04-23T10:18:39.449 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 23 10:18:39 smithi119 ceph-mon[32608]: rocksdb: Options.use_fsync: 0 2024-04-23T10:18:39.450 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 23 10:18:39 smithi119 ceph-mon[32608]: rocksdb: Options.max_log_file_size: 0 2024-04-23T10:18:39.450 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 23 10:18:39 smithi119 ceph-mon[32608]: rocksdb: Options.max_manifest_file_size: 1073741824 2024-04-23T10:18:39.450 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 23 10:18:39 smithi119 ceph-mon[32608]: rocksdb: Options.log_file_time_to_roll: 0 2024-04-23T10:18:39.450 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 23 10:18:39 smithi119 ceph-mon[32608]: rocksdb: Options.keep_log_file_num: 1000 2024-04-23T10:18:39.450 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 23 10:18:39 smithi119 ceph-mon[32608]: rocksdb: Options.recycle_log_file_num: 0 2024-04-23T10:18:39.450 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 23 10:18:39 smithi119 ceph-mon[32608]: rocksdb: Options.allow_fallocate: 1 2024-04-23T10:18:39.450 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 23 10:18:39 smithi119 ceph-mon[32608]: rocksdb: Options.allow_mmap_reads: 0 2024-04-23T10:18:39.450 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 23 10:18:39 smithi119 ceph-mon[32608]: rocksdb: Options.allow_mmap_writes: 0 2024-04-23T10:18:39.450 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 23 10:18:39 smithi119 ceph-mon[32608]: rocksdb: Options.use_direct_reads: 0 2024-04-23T10:18:39.450 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 23 10:18:39 smithi119 ceph-mon[32608]: rocksdb: Options.use_direct_io_for_flush_and_compaction: 0 2024-04-23T10:18:39.451 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 23 10:18:39 smithi119 ceph-mon[32608]: rocksdb: Options.create_missing_column_families: 0 2024-04-23T10:18:39.451 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 23 10:18:39 smithi119 ceph-mon[32608]: rocksdb: Options.db_log_dir: 2024-04-23T10:18:39.451 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 23 10:18:39 smithi119 ceph-mon[32608]: rocksdb: Options.wal_dir: 2024-04-23T10:18:39.451 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 23 10:18:39 smithi119 ceph-mon[32608]: rocksdb: Options.table_cache_numshardbits: 6 2024-04-23T10:18:39.451 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 23 10:18:39 smithi119 ceph-mon[32608]: rocksdb: Options.WAL_ttl_seconds: 0 2024-04-23T10:18:39.451 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 23 10:18:39 smithi119 ceph-mon[32608]: rocksdb: Options.WAL_size_limit_MB: 0 2024-04-23T10:18:39.451 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 23 10:18:39 smithi119 ceph-mon[32608]: rocksdb: Options.max_write_batch_group_size_bytes: 1048576 2024-04-23T10:18:39.453 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 23 10:18:39 smithi119 ceph-mon[32608]: rocksdb: Options.manifest_preallocation_size: 4194304 2024-04-23T10:18:39.453 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 23 10:18:39 smithi119 ceph-mon[32608]: rocksdb: Options.is_fd_close_on_exec: 1 2024-04-23T10:18:39.453 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 23 10:18:39 smithi119 ceph-mon[32608]: rocksdb: Options.advise_random_on_open: 1 2024-04-23T10:18:39.453 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 23 10:18:39 smithi119 ceph-mon[32608]: rocksdb: Options.db_write_buffer_size: 0 2024-04-23T10:18:39.453 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 23 10:18:39 smithi119 ceph-mon[32608]: rocksdb: Options.write_buffer_manager: 0x5572495ef0e0 2024-04-23T10:18:39.453 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 23 10:18:39 smithi119 ceph-mon[32608]: rocksdb: Options.access_hint_on_compaction_start: 1 2024-04-23T10:18:39.453 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 23 10:18:39 smithi119 ceph-mon[32608]: rocksdb: Options.random_access_max_buffer_size: 1048576 2024-04-23T10:18:39.453 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 23 10:18:39 smithi119 ceph-mon[32608]: rocksdb: Options.use_adaptive_mutex: 0 2024-04-23T10:18:39.453 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 23 10:18:39 smithi119 ceph-mon[32608]: rocksdb: Options.rate_limiter: (nil) 2024-04-23T10:18:39.454 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 23 10:18:39 smithi119 ceph-mon[32608]: rocksdb: Options.sst_file_manager.rate_bytes_per_sec: 0 2024-04-23T10:18:39.454 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 23 10:18:39 smithi119 ceph-mon[32608]: rocksdb: Options.wal_recovery_mode: 2 2024-04-23T10:18:39.454 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 23 10:18:39 smithi119 ceph-mon[32608]: rocksdb: Options.enable_thread_tracking: 0 2024-04-23T10:18:39.454 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 23 10:18:39 smithi119 ceph-mon[32608]: rocksdb: Options.enable_pipelined_write: 0 2024-04-23T10:18:39.454 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 23 10:18:39 smithi119 ceph-mon[32608]: rocksdb: Options.unordered_write: 0 2024-04-23T10:18:39.454 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 23 10:18:39 smithi119 ceph-mon[32608]: rocksdb: Options.allow_concurrent_memtable_write: 1 2024-04-23T10:18:39.454 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 23 10:18:39 smithi119 ceph-mon[32608]: rocksdb: Options.enable_write_thread_adaptive_yield: 1 2024-04-23T10:18:39.454 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 23 10:18:39 smithi119 ceph-mon[32608]: rocksdb: Options.write_thread_max_yield_usec: 100 2024-04-23T10:18:39.454 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 23 10:18:39 smithi119 ceph-mon[32608]: rocksdb: Options.write_thread_slow_yield_usec: 3 2024-04-23T10:18:39.454 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 23 10:18:39 smithi119 ceph-mon[32608]: rocksdb: Options.row_cache: None 2024-04-23T10:18:39.454 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 23 10:18:39 smithi119 ceph-mon[32608]: rocksdb: Options.wal_filter: None 2024-04-23T10:18:39.454 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 23 10:18:39 smithi119 ceph-mon[32608]: rocksdb: Options.avoid_flush_during_recovery: 0 2024-04-23T10:18:39.454 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 23 10:18:39 smithi119 ceph-mon[32608]: rocksdb: Options.allow_ingest_behind: 0 2024-04-23T10:18:39.455 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 23 10:18:39 smithi119 ceph-mon[32608]: rocksdb: Options.two_write_queues: 0 2024-04-23T10:18:39.455 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 23 10:18:39 smithi119 ceph-mon[32608]: rocksdb: Options.manual_wal_flush: 0 2024-04-23T10:18:39.455 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 23 10:18:39 smithi119 ceph-mon[32608]: rocksdb: Options.wal_compression: 0 2024-04-23T10:18:39.455 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 23 10:18:39 smithi119 ceph-mon[32608]: rocksdb: Options.atomic_flush: 0 2024-04-23T10:18:39.455 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 23 10:18:39 smithi119 ceph-mon[32608]: rocksdb: Options.avoid_unnecessary_blocking_io: 0 2024-04-23T10:18:39.455 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 23 10:18:39 smithi119 ceph-mon[32608]: rocksdb: Options.persist_stats_to_disk: 0 2024-04-23T10:18:39.455 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 23 10:18:39 smithi119 ceph-mon[32608]: rocksdb: Options.write_dbid_to_manifest: 0 2024-04-23T10:18:39.455 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 23 10:18:39 smithi119 ceph-mon[32608]: rocksdb: Options.log_readahead_size: 0 2024-04-23T10:18:39.455 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 23 10:18:39 smithi119 ceph-mon[32608]: rocksdb: Options.file_checksum_gen_factory: Unknown 2024-04-23T10:18:39.455 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 23 10:18:39 smithi119 ceph-mon[32608]: rocksdb: Options.best_efforts_recovery: 0 2024-04-23T10:18:39.455 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 23 10:18:39 smithi119 ceph-mon[32608]: rocksdb: Options.max_bgerror_resume_count: 2147483647 2024-04-23T10:18:39.455 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 23 10:18:39 smithi119 ceph-mon[32608]: rocksdb: Options.bgerror_resume_retry_interval: 1000000 2024-04-23T10:18:39.455 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 23 10:18:39 smithi119 ceph-mon[32608]: rocksdb: Options.allow_data_in_errors: 0 2024-04-23T10:18:39.455 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 23 10:18:39 smithi119 ceph-mon[32608]: rocksdb: Options.db_host_id: __hostname__ 2024-04-23T10:18:39.455 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 23 10:18:39 smithi119 ceph-mon[32608]: rocksdb: Options.enforce_single_del_contracts: true 2024-04-23T10:18:39.455 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 23 10:18:39 smithi119 ceph-mon[32608]: rocksdb: Options.max_background_jobs: 2 2024-04-23T10:18:39.456 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 23 10:18:39 smithi119 ceph-mon[32608]: rocksdb: Options.max_background_compactions: -1 2024-04-23T10:18:39.456 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 23 10:18:39 smithi119 ceph-mon[32608]: rocksdb: Options.max_subcompactions: 1 2024-04-23T10:18:39.456 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 23 10:18:39 smithi119 ceph-mon[32608]: rocksdb: Options.avoid_flush_during_shutdown: 0 2024-04-23T10:18:39.456 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 23 10:18:39 smithi119 ceph-mon[32608]: rocksdb: Options.writable_file_max_buffer_size: 1048576 2024-04-23T10:18:39.456 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 23 10:18:39 smithi119 ceph-mon[32608]: rocksdb: Options.delayed_write_rate : 16777216 2024-04-23T10:18:39.456 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 23 10:18:39 smithi119 ceph-mon[32608]: rocksdb: Options.max_total_wal_size: 0 2024-04-23T10:18:39.456 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 23 10:18:39 smithi119 ceph-mon[32608]: rocksdb: Options.delete_obsolete_files_period_micros: 21600000000 2024-04-23T10:18:39.456 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 23 10:18:39 smithi119 ceph-mon[32608]: rocksdb: Options.stats_dump_period_sec: 600 2024-04-23T10:18:39.456 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 23 10:18:39 smithi119 ceph-mon[32608]: rocksdb: Options.stats_persist_period_sec: 600 2024-04-23T10:18:39.456 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 23 10:18:39 smithi119 ceph-mon[32608]: rocksdb: Options.stats_history_buffer_size: 1048576 2024-04-23T10:18:39.456 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 23 10:18:39 smithi119 ceph-mon[32608]: rocksdb: Options.max_open_files: -1 2024-04-23T10:18:39.456 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 23 10:18:39 smithi119 ceph-mon[32608]: rocksdb: Options.bytes_per_sync: 0 2024-04-23T10:18:39.456 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 23 10:18:39 smithi119 ceph-mon[32608]: rocksdb: Options.wal_bytes_per_sync: 0 2024-04-23T10:18:39.457 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 23 10:18:39 smithi119 ceph-mon[32608]: rocksdb: Options.strict_bytes_per_sync: 0 2024-04-23T10:18:39.457 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 23 10:18:39 smithi119 ceph-mon[32608]: rocksdb: Options.compaction_readahead_size: 0 2024-04-23T10:18:39.457 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 23 10:18:39 smithi119 ceph-mon[32608]: rocksdb: Options.max_background_flushes: -1 2024-04-23T10:18:39.457 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 23 10:18:39 smithi119 ceph-mon[32608]: rocksdb: Compression algorithms supported: 2024-04-23T10:18:39.457 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 23 10:18:39 smithi119 ceph-mon[32608]: rocksdb: kZSTDNotFinalCompression supported: 0 2024-04-23T10:18:39.457 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 23 10:18:39 smithi119 ceph-mon[32608]: rocksdb: kZSTD supported: 0 2024-04-23T10:18:39.457 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 23 10:18:39 smithi119 ceph-mon[32608]: rocksdb: kXpressCompression supported: 0 2024-04-23T10:18:39.457 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 23 10:18:39 smithi119 ceph-mon[32608]: rocksdb: kLZ4HCCompression supported: 1 2024-04-23T10:18:39.457 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 23 10:18:39 smithi119 ceph-mon[32608]: rocksdb: kZlibCompression supported: 1 2024-04-23T10:18:39.457 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 23 10:18:39 smithi119 ceph-mon[32608]: rocksdb: kSnappyCompression supported: 1 2024-04-23T10:18:39.457 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 23 10:18:39 smithi119 ceph-mon[32608]: rocksdb: kLZ4Compression supported: 1 2024-04-23T10:18:39.457 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 23 10:18:39 smithi119 ceph-mon[32608]: rocksdb: kBZip2Compression supported: 0 2024-04-23T10:18:39.457 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 23 10:18:39 smithi119 ceph-mon[32608]: rocksdb: Fast CRC32 supported: Supported on x86 2024-04-23T10:18:39.457 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 23 10:18:39 smithi119 ceph-mon[32608]: rocksdb: DMutex implementation: pthread_mutex_t 2024-04-23T10:18:39.457 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 23 10:18:39 smithi119 ceph-mon[32608]: rocksdb: [db/version_set.cc:5527] Recovering from manifest file: /var/lib/ceph/mon/ceph-b/store.db/MANIFEST-000005 2024-04-23T10:18:39.457 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 23 10:18:39 smithi119 ceph-mon[32608]: rocksdb: [db/column_family.cc:630] --------------- Options for column family [default]: 2024-04-23T10:18:39.458 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 23 10:18:39 smithi119 ceph-mon[32608]: rocksdb: Options.comparator: leveldb.BytewiseComparator 2024-04-23T10:18:39.458 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 23 10:18:39 smithi119 ceph-mon[32608]: rocksdb: Options.merge_operator: 2024-04-23T10:18:39.458 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 23 10:18:39 smithi119 ceph-mon[32608]: rocksdb: Options.compaction_filter: None 2024-04-23T10:18:39.458 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 23 10:18:39 smithi119 ceph-mon[32608]: rocksdb: Options.compaction_filter_factory: None 2024-04-23T10:18:39.458 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 23 10:18:39 smithi119 ceph-mon[32608]: rocksdb: Options.sst_partitioner_factory: None 2024-04-23T10:18:39.458 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 23 10:18:39 smithi119 ceph-mon[32608]: rocksdb: Options.memtable_factory: SkipListFactory 2024-04-23T10:18:39.458 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 23 10:18:39 smithi119 ceph-mon[32608]: rocksdb: Options.table_factory: BlockBasedTable 2024-04-23T10:18:39.458 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 23 10:18:39 smithi119 ceph-mon[32608]: rocksdb: table_factory options: flush_block_policy_factory: FlushBlockBySizePolicyFactory (0x55724965bdc0) 2024-04-23T10:18:39.458 INFO:journalctl@ceph.mon.b.smithi119.stdout: cache_index_and_filter_blocks: 1 2024-04-23T10:18:39.458 INFO:journalctl@ceph.mon.b.smithi119.stdout: cache_index_and_filter_blocks_with_high_priority: 0 2024-04-23T10:18:39.458 INFO:journalctl@ceph.mon.b.smithi119.stdout: pin_l0_filter_and_index_blocks_in_cache: 0 2024-04-23T10:18:39.458 INFO:journalctl@ceph.mon.b.smithi119.stdout: pin_top_level_index_and_filter: 1 2024-04-23T10:18:39.458 INFO:journalctl@ceph.mon.b.smithi119.stdout: index_type: 0 2024-04-23T10:18:39.458 INFO:journalctl@ceph.mon.b.smithi119.stdout: data_block_index_type: 0 2024-04-23T10:18:39.459 INFO:journalctl@ceph.mon.b.smithi119.stdout: index_shortening: 1 2024-04-23T10:18:39.459 INFO:journalctl@ceph.mon.b.smithi119.stdout: data_block_hash_table_util_ratio: 0.750000 2024-04-23T10:18:39.459 INFO:journalctl@ceph.mon.b.smithi119.stdout: checksum: 4 2024-04-23T10:18:39.459 INFO:journalctl@ceph.mon.b.smithi119.stdout: no_block_cache: 0 2024-04-23T10:18:39.459 INFO:journalctl@ceph.mon.b.smithi119.stdout: block_cache: 0x5572496211f0 2024-04-23T10:18:39.459 INFO:journalctl@ceph.mon.b.smithi119.stdout: block_cache_name: BinnedLRUCache 2024-04-23T10:18:39.459 INFO:journalctl@ceph.mon.b.smithi119.stdout: block_cache_options: 2024-04-23T10:18:39.459 INFO:journalctl@ceph.mon.b.smithi119.stdout: capacity : 536870912 2024-04-23T10:18:39.459 INFO:journalctl@ceph.mon.b.smithi119.stdout: num_shard_bits : 4 2024-04-23T10:18:39.459 INFO:journalctl@ceph.mon.b.smithi119.stdout: strict_capacity_limit : 0 2024-04-23T10:18:39.459 INFO:journalctl@ceph.mon.b.smithi119.stdout: high_pri_pool_ratio: 0.000 2024-04-23T10:18:39.459 INFO:journalctl@ceph.mon.b.smithi119.stdout: block_cache_compressed: (nil) 2024-04-23T10:18:39.459 INFO:journalctl@ceph.mon.b.smithi119.stdout: persistent_cache: (nil) 2024-04-23T10:18:39.459 INFO:journalctl@ceph.mon.b.smithi119.stdout: block_size: 4096 2024-04-23T10:18:39.459 INFO:journalctl@ceph.mon.b.smithi119.stdout: block_size_deviation: 10 2024-04-23T10:18:39.459 INFO:journalctl@ceph.mon.b.smithi119.stdout: block_restart_interval: 16 2024-04-23T10:18:39.460 INFO:journalctl@ceph.mon.b.smithi119.stdout: index_block_restart_interval: 1 2024-04-23T10:18:39.460 INFO:journalctl@ceph.mon.b.smithi119.stdout: metadata_block_size: 4096 2024-04-23T10:18:39.460 INFO:journalctl@ceph.mon.b.smithi119.stdout: partition_filters: 0 2024-04-23T10:18:39.460 INFO:journalctl@ceph.mon.b.smithi119.stdout: use_delta_encoding: 1 2024-04-23T10:18:39.460 INFO:journalctl@ceph.mon.b.smithi119.stdout: filter_policy: bloomfilter 2024-04-23T10:18:39.460 INFO:journalctl@ceph.mon.b.smithi119.stdout: whole_key_filtering: 1 2024-04-23T10:18:39.460 INFO:journalctl@ceph.mon.b.smithi119.stdout: verify_compression: 0 2024-04-23T10:18:39.460 INFO:journalctl@ceph.mon.b.smithi119.stdout: read_amp_bytes_per_bit: 0 2024-04-23T10:18:39.460 INFO:journalctl@ceph.mon.b.smithi119.stdout: format_version: 5 2024-04-23T10:18:39.460 INFO:journalctl@ceph.mon.b.smithi119.stdout: enable_index_compression: 1 2024-04-23T10:18:39.460 INFO:journalctl@ceph.mon.b.smithi119.stdout: block_align: 0 2024-04-23T10:18:39.460 INFO:journalctl@ceph.mon.b.smithi119.stdout: max_auto_readahead_size: 262144 2024-04-23T10:18:39.460 INFO:journalctl@ceph.mon.b.smithi119.stdout: prepopulate_block_cache: 0 2024-04-23T10:18:39.460 INFO:journalctl@ceph.mon.b.smithi119.stdout: initial_auto_readahead_size: 8192 2024-04-23T10:18:39.460 INFO:journalctl@ceph.mon.b.smithi119.stdout: num_file_reads_for_auto_readahead: 2 2024-04-23T10:18:39.460 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 23 10:18:39 smithi119 ceph-mon[32608]: rocksdb: Options.write_buffer_size: 33554432 2024-04-23T10:18:39.460 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 23 10:18:39 smithi119 ceph-mon[32608]: rocksdb: Options.max_write_buffer_number: 2 2024-04-23T10:18:39.461 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 23 10:18:39 smithi119 ceph-mon[32608]: rocksdb: Options.compression: NoCompression 2024-04-23T10:18:39.461 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 23 10:18:39 smithi119 ceph-mon[32608]: rocksdb: Options.bottommost_compression: Disabled 2024-04-23T10:18:39.461 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 23 10:18:39 smithi119 ceph-mon[32608]: rocksdb: Options.prefix_extractor: nullptr 2024-04-23T10:18:39.461 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 23 10:18:39 smithi119 ceph-mon[32608]: rocksdb: Options.memtable_insert_with_hint_prefix_extractor: nullptr 2024-04-23T10:18:39.461 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 23 10:18:39 smithi119 ceph-mon[32608]: rocksdb: Options.num_levels: 7 2024-04-23T10:18:39.461 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 23 10:18:39 smithi119 ceph-mon[32608]: rocksdb: Options.min_write_buffer_number_to_merge: 1 2024-04-23T10:18:39.461 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 23 10:18:39 smithi119 ceph-mon[32608]: rocksdb: Options.max_write_buffer_number_to_maintain: 0 2024-04-23T10:18:39.461 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 23 10:18:39 smithi119 ceph-mon[32608]: rocksdb: Options.max_write_buffer_size_to_maintain: 0 2024-04-23T10:18:39.461 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 23 10:18:39 smithi119 ceph-mon[32608]: rocksdb: Options.bottommost_compression_opts.window_bits: -14 2024-04-23T10:18:39.461 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 23 10:18:39 smithi119 ceph-mon[32608]: rocksdb: Options.bottommost_compression_opts.level: 32767 2024-04-23T10:18:39.461 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 23 10:18:39 smithi119 ceph-mon[32608]: rocksdb: Options.bottommost_compression_opts.strategy: 0 2024-04-23T10:18:39.461 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 23 10:18:39 smithi119 ceph-mon[32608]: rocksdb: Options.bottommost_compression_opts.max_dict_bytes: 0 2024-04-23T10:18:39.461 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 23 10:18:39 smithi119 ceph-mon[32608]: rocksdb: Options.bottommost_compression_opts.zstd_max_train_bytes: 0 2024-04-23T10:18:39.461 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 23 10:18:39 smithi119 ceph-mon[32608]: rocksdb: Options.bottommost_compression_opts.parallel_threads: 1 2024-04-23T10:18:39.461 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 23 10:18:39 smithi119 ceph-mon[32608]: rocksdb: Options.bottommost_compression_opts.enabled: false 2024-04-23T10:18:39.461 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 23 10:18:39 smithi119 ceph-mon[32608]: rocksdb: Options.bottommost_compression_opts.max_dict_buffer_bytes: 0 2024-04-23T10:18:39.462 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 23 10:18:39 smithi119 ceph-mon[32608]: rocksdb: Options.bottommost_compression_opts.use_zstd_dict_trainer: true 2024-04-23T10:18:39.462 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 23 10:18:39 smithi119 ceph-mon[32608]: rocksdb: Options.compression_opts.window_bits: -14 2024-04-23T10:18:39.462 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 23 10:18:39 smithi119 ceph-mon[32608]: rocksdb: Options.compression_opts.level: 32767 2024-04-23T10:18:39.462 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 23 10:18:39 smithi119 ceph-mon[32608]: rocksdb: Options.compression_opts.strategy: 0 2024-04-23T10:18:39.462 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 23 10:18:39 smithi119 ceph-mon[32608]: rocksdb: Options.compression_opts.max_dict_bytes: 0 2024-04-23T10:18:39.462 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 23 10:18:39 smithi119 ceph-mon[32608]: rocksdb: Options.compression_opts.zstd_max_train_bytes: 0 2024-04-23T10:18:39.462 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 23 10:18:39 smithi119 ceph-mon[32608]: rocksdb: Options.compression_opts.use_zstd_dict_trainer: true 2024-04-23T10:18:39.462 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 23 10:18:39 smithi119 ceph-mon[32608]: rocksdb: Options.compression_opts.parallel_threads: 1 2024-04-23T10:18:39.462 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 23 10:18:39 smithi119 ceph-mon[32608]: rocksdb: Options.compression_opts.enabled: false 2024-04-23T10:18:39.462 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 23 10:18:39 smithi119 ceph-mon[32608]: rocksdb: Options.compression_opts.max_dict_buffer_bytes: 0 2024-04-23T10:18:39.462 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 23 10:18:39 smithi119 ceph-mon[32608]: rocksdb: Options.level0_file_num_compaction_trigger: 4 2024-04-23T10:18:39.462 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 23 10:18:39 smithi119 ceph-mon[32608]: rocksdb: Options.level0_slowdown_writes_trigger: 20 2024-04-23T10:18:39.462 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 23 10:18:39 smithi119 ceph-mon[32608]: rocksdb: Options.level0_stop_writes_trigger: 36 2024-04-23T10:18:39.463 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 23 10:18:39 smithi119 ceph-mon[32608]: rocksdb: Options.target_file_size_base: 67108864 2024-04-23T10:18:39.463 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 23 10:18:39 smithi119 ceph-mon[32608]: rocksdb: Options.target_file_size_multiplier: 1 2024-04-23T10:18:39.463 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 23 10:18:39 smithi119 ceph-mon[32608]: rocksdb: Options.max_bytes_for_level_base: 268435456 2024-04-23T10:18:39.463 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 23 10:18:39 smithi119 ceph-mon[32608]: rocksdb: Options.level_compaction_dynamic_level_bytes: 1 2024-04-23T10:18:39.463 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 23 10:18:39 smithi119 ceph-mon[32608]: rocksdb: Options.max_bytes_for_level_multiplier: 10.000000 2024-04-23T10:18:39.463 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 23 10:18:39 smithi119 ceph-mon[32608]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[0]: 1 2024-04-23T10:18:39.463 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 23 10:18:39 smithi119 ceph-mon[32608]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[1]: 1 2024-04-23T10:18:39.463 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 23 10:18:39 smithi119 ceph-mon[32608]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[2]: 1 2024-04-23T10:18:39.463 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 23 10:18:39 smithi119 ceph-mon[32608]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[3]: 1 2024-04-23T10:18:39.463 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 23 10:18:39 smithi119 ceph-mon[32608]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[4]: 1 2024-04-23T10:18:39.463 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 23 10:18:39 smithi119 ceph-mon[32608]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[5]: 1 2024-04-23T10:18:39.463 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 23 10:18:39 smithi119 ceph-mon[32608]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[6]: 1 2024-04-23T10:18:39.463 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 23 10:18:39 smithi119 ceph-mon[32608]: rocksdb: Options.max_sequential_skip_in_iterations: 8 2024-04-23T10:18:39.463 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 23 10:18:39 smithi119 ceph-mon[32608]: rocksdb: Options.max_compaction_bytes: 1677721600 2024-04-23T10:18:39.463 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 23 10:18:39 smithi119 ceph-mon[32608]: rocksdb: Options.ignore_max_compaction_bytes_for_input: true 2024-04-23T10:18:39.463 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 23 10:18:39 smithi119 ceph-mon[32608]: rocksdb: Options.arena_block_size: 1048576 2024-04-23T10:18:39.464 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 23 10:18:39 smithi119 ceph-mon[32608]: rocksdb: Options.soft_pending_compaction_bytes_limit: 68719476736 2024-04-23T10:18:39.464 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 23 10:18:39 smithi119 ceph-mon[32608]: rocksdb: Options.hard_pending_compaction_bytes_limit: 274877906944 2024-04-23T10:18:39.464 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 23 10:18:39 smithi119 ceph-mon[32608]: rocksdb: Options.disable_auto_compactions: 0 2024-04-23T10:18:39.464 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 23 10:18:39 smithi119 ceph-mon[32608]: rocksdb: Options.compaction_style: kCompactionStyleLevel 2024-04-23T10:18:39.464 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 23 10:18:39 smithi119 ceph-mon[32608]: rocksdb: Options.compaction_pri: kMinOverlappingRatio 2024-04-23T10:18:39.464 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 23 10:18:39 smithi119 ceph-mon[32608]: rocksdb: Options.compaction_options_universal.size_ratio: 1 2024-04-23T10:18:39.464 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 23 10:18:39 smithi119 ceph-mon[32608]: rocksdb: Options.compaction_options_universal.min_merge_width: 2 2024-04-23T10:18:39.464 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 23 10:18:39 smithi119 ceph-mon[32608]: rocksdb: Options.compaction_options_universal.max_merge_width: 4294967295 2024-04-23T10:18:39.464 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 23 10:18:39 smithi119 ceph-mon[32608]: rocksdb: Options.compaction_options_universal.max_size_amplification_percent: 200 2024-04-23T10:18:39.464 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 23 10:18:39 smithi119 ceph-mon[32608]: rocksdb: Options.compaction_options_universal.compression_size_percent: -1 2024-04-23T10:18:39.464 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 23 10:18:39 smithi119 ceph-mon[32608]: rocksdb: Options.compaction_options_universal.stop_style: kCompactionStopStyleTotalSize 2024-04-23T10:18:39.464 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 23 10:18:39 smithi119 ceph-mon[32608]: rocksdb: Options.compaction_options_fifo.max_table_files_size: 1073741824 2024-04-23T10:18:39.464 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 23 10:18:39 smithi119 ceph-mon[32608]: rocksdb: Options.compaction_options_fifo.allow_compaction: 0 2024-04-23T10:18:39.464 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 23 10:18:39 smithi119 ceph-mon[32608]: rocksdb: Options.table_properties_collectors: CompactOnDeletionCollector (Sliding window size = 32768 Deletion trigger = 16384 Deletion ratio = 0); 2024-04-23T10:18:39.464 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 23 10:18:39 smithi119 ceph-mon[32608]: rocksdb: Options.inplace_update_support: 0 2024-04-23T10:18:39.464 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 23 10:18:39 smithi119 ceph-mon[32608]: rocksdb: Options.inplace_update_num_locks: 10000 2024-04-23T10:18:39.465 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 23 10:18:39 smithi119 ceph-mon[32608]: rocksdb: Options.memtable_prefix_bloom_size_ratio: 0.000000 2024-04-23T10:18:39.465 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 23 10:18:39 smithi119 ceph-mon[32608]: rocksdb: Options.memtable_whole_key_filtering: 0 2024-04-23T10:18:39.465 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 23 10:18:39 smithi119 ceph-mon[32608]: rocksdb: Options.memtable_huge_page_size: 0 2024-04-23T10:18:39.465 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 23 10:18:39 smithi119 ceph-mon[32608]: rocksdb: Options.bloom_locality: 0 2024-04-23T10:18:39.465 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 23 10:18:39 smithi119 ceph-mon[32608]: rocksdb: Options.max_successive_merges: 0 2024-04-23T10:18:39.465 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 23 10:18:39 smithi119 ceph-mon[32608]: rocksdb: Options.optimize_filters_for_hits: 0 2024-04-23T10:18:39.465 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 23 10:18:39 smithi119 ceph-mon[32608]: rocksdb: Options.paranoid_file_checks: 0 2024-04-23T10:18:39.465 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 23 10:18:39 smithi119 ceph-mon[32608]: rocksdb: Options.force_consistency_checks: 1 2024-04-23T10:18:39.465 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 23 10:18:39 smithi119 ceph-mon[32608]: rocksdb: Options.report_bg_io_stats: 0 2024-04-23T10:18:39.465 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 23 10:18:39 smithi119 ceph-mon[32608]: rocksdb: Options.ttl: 2592000 2024-04-23T10:18:39.465 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 23 10:18:39 smithi119 ceph-mon[32608]: rocksdb: Options.periodic_compaction_seconds: 0 2024-04-23T10:18:39.465 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 23 10:18:39 smithi119 ceph-mon[32608]: rocksdb: Options.preclude_last_level_data_seconds: 0 2024-04-23T10:18:39.465 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 23 10:18:39 smithi119 ceph-mon[32608]: rocksdb: Options.preserve_internal_time_seconds: 0 2024-04-23T10:18:39.465 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 23 10:18:39 smithi119 ceph-mon[32608]: rocksdb: Options.enable_blob_files: false 2024-04-23T10:18:39.466 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 23 10:18:39 smithi119 ceph-mon[32608]: rocksdb: Options.min_blob_size: 0 2024-04-23T10:18:39.466 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 23 10:18:39 smithi119 ceph-mon[32608]: rocksdb: Options.blob_file_size: 268435456 2024-04-23T10:18:39.466 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 23 10:18:39 smithi119 ceph-mon[32608]: rocksdb: Options.blob_compression_type: NoCompression 2024-04-23T10:18:39.466 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 23 10:18:39 smithi119 ceph-mon[32608]: rocksdb: Options.enable_blob_garbage_collection: false 2024-04-23T10:18:39.466 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 23 10:18:39 smithi119 ceph-mon[32608]: rocksdb: Options.blob_garbage_collection_age_cutoff: 0.250000 2024-04-23T10:18:39.466 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 23 10:18:39 smithi119 ceph-mon[32608]: rocksdb: Options.blob_garbage_collection_force_threshold: 1.000000 2024-04-23T10:18:39.466 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 23 10:18:39 smithi119 ceph-mon[32608]: rocksdb: Options.blob_compaction_readahead_size: 0 2024-04-23T10:18:39.466 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 23 10:18:39 smithi119 ceph-mon[32608]: rocksdb: Options.blob_file_starting_level: 0 2024-04-23T10:18:39.466 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 23 10:18:39 smithi119 ceph-mon[32608]: rocksdb: Options.experimental_mempurge_threshold: 0.000000 2024-04-23T10:18:39.466 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 23 10:18:39 smithi119 ceph-mon[32608]: rocksdb: [db/version_set.cc:5566] Recovered from manifest file:/var/lib/ceph/mon/ceph-b/store.db/MANIFEST-000005 succeeded,manifest_file_number is 5, next_file_number is 7, last_sequence is 0, log_number is 0,prev_log_number is 0,max_column_family is 0,min_log_number_to_keep is 0 2024-04-23T10:18:39.466 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 23 10:18:39 smithi119 ceph-mon[32608]: rocksdb: [db/version_set.cc:5581] Column family [default] (ID 0), log number is 0 2024-04-23T10:18:39.466 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 23 10:18:39 smithi119 ceph-mon[32608]: rocksdb: [db/db_impl/db_impl_open.cc:539] DB ID: 4c7216c2-0847-4eeb-95ed-be0fa1e06c48 2024-04-23T10:18:39.467 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 23 10:18:39 smithi119 ceph-mon[32608]: rocksdb: EVENT_LOG_v1 {"time_micros": 1713867519194084, "job": 1, "event": "recovery_started", "wal_files": [4]} 2024-04-23T10:18:39.467 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 23 10:18:39 smithi119 ceph-mon[32608]: rocksdb: [db/db_impl/db_impl_open.cc:1043] Recovering log #4 mode 2 2024-04-23T10:18:39.467 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 23 10:18:39 smithi119 ceph-mon[32608]: rocksdb: EVENT_LOG_v1 {"time_micros": 1713867519194508, "cf_name": "default", "job": 1, "event": "table_file_creation", "file_number": 8, "file_size": 1773, "file_checksum": "", "file_checksum_func_name": "Unknown", "smallest_seqno": 1, "largest_seqno": 5, "table_properties": {"data_size": 648, "index_size": 31, "index_partitions": 0, "top_level_index_size": 0, "index_key_is_user_key": 1, "index_value_is_delta_encoded": 1, "filter_size": 69, "raw_key_size": 115, "raw_average_key_size": 23, "raw_value_size": 526, "raw_average_value_size": 105, "num_data_blocks": 1, "num_entries": 5, "num_filter_entries": 5, "num_deletions": 0, "num_merge_operands": 0, "num_range_deletions": 0, "format_version": 0, "fixed_key_len": 0, "filter_policy": "bloomfilter", "column_family_name": "default", "column_family_id": 0, "comparator": "leveldb.BytewiseComparator", "merge_operator": "", "prefix_extractor_name": "nullptr", "property_collectors": "[CompactOnDeletionCollector]", "compression": "NoCompression", "compression_options": "window_bits=-14; level=32767; strategy=0; max_dict_bytes=0; zstd_max_train_bytes=0; enabled=0; max_dict_buffer_bytes=0; use_zstd_dict_trainer=1; ", "creation_time": 1713867519, "oldest_key_time": 0, "file_creation_time": 0, "slow_compression_estimated_data_size": 0, "fast_compression_estimated_data_size": 0, "db_id": "4c7216c2-0847-4eeb-95ed-be0fa1e06c48", "db_session_id": "JU99V4O8SURC5HIG6FAI", "orig_file_number": 8, "seqno_to_time_mapping": "N/A"}} 2024-04-23T10:18:39.467 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 23 10:18:39 smithi119 ceph-mon[32608]: rocksdb: EVENT_LOG_v1 {"time_micros": 1713867519194591, "job": 1, "event": "recovery_finished"} 2024-04-23T10:18:39.467 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 23 10:18:39 smithi119 ceph-mon[32608]: rocksdb: [db/version_set.cc:5047] Creating manifest 10 2024-04-23T10:18:39.467 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 23 10:18:39 smithi119 ceph-mon[32608]: rocksdb: [file/delete_scheduler.cc:74] Deleted file /var/lib/ceph/mon/ceph-b/store.db/000004.log immediately, rate_bytes_per_sec 0, total_trash_size 0 max_trash_db_ratio 0.250000 2024-04-23T10:18:39.467 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 23 10:18:39 smithi119 ceph-mon[32608]: rocksdb: [db/db_impl/db_impl_open.cc:1987] SstFileManager instance 0x557249736000 2024-04-23T10:18:39.467 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 23 10:18:39 smithi119 ceph-mon[32608]: rocksdb: DB pointer 0x557249720000 2024-04-23T10:18:39.467 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 23 10:18:39 smithi119 ceph-mon[32608]: rocksdb: [db/db_impl/db_impl.cc:1109] ------- DUMPING STATS ------- 2024-04-23T10:18:39.467 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 23 10:18:39 smithi119 ceph-mon[32608]: rocksdb: [db/db_impl/db_impl.cc:1111] 2024-04-23T10:18:39.467 INFO:journalctl@ceph.mon.b.smithi119.stdout: ** DB Stats ** 2024-04-23T10:18:39.467 INFO:journalctl@ceph.mon.b.smithi119.stdout: Uptime(secs): 0.0 total, 0.0 interval 2024-04-23T10:18:39.467 INFO:journalctl@ceph.mon.b.smithi119.stdout: Cumulative writes: 0 writes, 0 keys, 0 commit groups, 0.0 writes per commit group, ingest: 0.00 GB, 0.00 MB/s 2024-04-23T10:18:39.467 INFO:journalctl@ceph.mon.b.smithi119.stdout: Cumulative WAL: 0 writes, 0 syncs, 0.00 writes per sync, written: 0.00 GB, 0.00 MB/s 2024-04-23T10:18:39.468 INFO:journalctl@ceph.mon.b.smithi119.stdout: Cumulative stall: 00:00:0.000 H:M:S, 0.0 percent 2024-04-23T10:18:39.468 INFO:journalctl@ceph.mon.b.smithi119.stdout: Interval writes: 0 writes, 0 keys, 0 commit groups, 0.0 writes per commit group, ingest: 0.00 MB, 0.00 MB/s 2024-04-23T10:18:39.468 INFO:journalctl@ceph.mon.b.smithi119.stdout: Interval WAL: 0 writes, 0 syncs, 0.00 writes per sync, written: 0.00 GB, 0.00 MB/s 2024-04-23T10:18:39.468 INFO:journalctl@ceph.mon.b.smithi119.stdout: Interval stall: 00:00:0.000 H:M:S, 0.0 percent 2024-04-23T10:18:39.468 INFO:journalctl@ceph.mon.b.smithi119.stdout: 2024-04-23T10:18:39.468 INFO:journalctl@ceph.mon.b.smithi119.stdout: ** Compaction Stats [default] ** 2024-04-23T10:18:39.468 INFO:journalctl@ceph.mon.b.smithi119.stdout: Level Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop Rblob(GB) Wblob(GB) 2024-04-23T10:18:39.468 INFO:journalctl@ceph.mon.b.smithi119.stdout: ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ 2024-04-23T10:18:39.468 INFO:journalctl@ceph.mon.b.smithi119.stdout: L0 1/0 1.73 KB 0.2 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 4.2 0.00 0.00 1 0.000 0 0 0.0 0.0 2024-04-23T10:18:39.468 INFO:journalctl@ceph.mon.b.smithi119.stdout: Sum 1/0 1.73 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 4.2 0.00 0.00 1 0.000 0 0 0.0 0.0 2024-04-23T10:18:39.468 INFO:journalctl@ceph.mon.b.smithi119.stdout: Int 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 4.2 0.00 0.00 1 0.000 0 0 0.0 0.0 2024-04-23T10:18:39.468 INFO:journalctl@ceph.mon.b.smithi119.stdout: 2024-04-23T10:18:39.468 INFO:journalctl@ceph.mon.b.smithi119.stdout: ** Compaction Stats [default] ** 2024-04-23T10:18:39.469 INFO:journalctl@ceph.mon.b.smithi119.stdout: Priority Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop Rblob(GB) Wblob(GB) 2024-04-23T10:18:39.469 INFO:journalctl@ceph.mon.b.smithi119.stdout: --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- 2024-04-23T10:18:39.469 INFO:journalctl@ceph.mon.b.smithi119.stdout: User 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 4.2 0.00 0.00 1 0.000 0 0 0.0 0.0 2024-04-23T10:18:39.469 INFO:journalctl@ceph.mon.b.smithi119.stdout: 2024-04-23T10:18:39.469 INFO:journalctl@ceph.mon.b.smithi119.stdout: Blob file count: 0, total size: 0.0 GB, garbage size: 0.0 GB, space amp: 0.0 2024-04-23T10:18:39.469 INFO:journalctl@ceph.mon.b.smithi119.stdout: 2024-04-23T10:18:39.469 INFO:journalctl@ceph.mon.b.smithi119.stdout: Uptime(secs): 0.0 total, 0.0 interval 2024-04-23T10:18:39.469 INFO:journalctl@ceph.mon.b.smithi119.stdout: Flush(GB): cumulative 0.000, interval 0.000 2024-04-23T10:18:39.469 INFO:journalctl@ceph.mon.b.smithi119.stdout: AddFile(GB): cumulative 0.000, interval 0.000 2024-04-23T10:18:39.469 INFO:journalctl@ceph.mon.b.smithi119.stdout: AddFile(Total Files): cumulative 0, interval 0 2024-04-23T10:18:39.469 INFO:journalctl@ceph.mon.b.smithi119.stdout: AddFile(L0 Files): cumulative 0, interval 0 2024-04-23T10:18:39.469 INFO:journalctl@ceph.mon.b.smithi119.stdout: AddFile(Keys): cumulative 0, interval 0 2024-04-23T10:18:39.469 INFO:journalctl@ceph.mon.b.smithi119.stdout: Cumulative compaction: 0.00 GB write, 0.65 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds 2024-04-23T10:18:39.469 INFO:journalctl@ceph.mon.b.smithi119.stdout: Interval compaction: 0.00 GB write, 0.65 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds 2024-04-23T10:18:39.469 INFO:journalctl@ceph.mon.b.smithi119.stdout: Stalls(count): 0 level0_slowdown, 0 level0_slowdown_with_compaction, 0 level0_numfiles, 0 level0_numfiles_with_compaction, 0 stop for pending_compaction_bytes, 0 slowdown for pending_compaction_bytes, 0 memtable_compaction, 0 memtable_slowdown, interval 0 total count 2024-04-23T10:18:39.469 INFO:journalctl@ceph.mon.b.smithi119.stdout: Block cache BinnedLRUCache@0x5572496211f0#2 capacity: 512.00 MB usage: 0.22 KB table_size: 0 occupancy: 18446744073709551615 collections: 1 last_copies: 0 last_secs: 9e-06 secs_since: 0 2024-04-23T10:18:39.470 INFO:journalctl@ceph.mon.b.smithi119.stdout: Block cache entry stats(count,size,portion): FilterBlock(1,0.11 KB,2.08616e-05%) IndexBlock(1,0.11 KB,2.08616e-05%) Misc(1,0.00 KB,0%) 2024-04-23T10:18:39.470 INFO:journalctl@ceph.mon.b.smithi119.stdout: 2024-04-23T10:18:39.470 INFO:journalctl@ceph.mon.b.smithi119.stdout: ** File Read Latency Histogram By Level [default] ** 2024-04-23T10:18:39.470 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 23 10:18:39 smithi119 ceph-mon[32608]: mon.b does not exist in monmap, will attempt to join an existing cluster 2024-04-23T10:18:39.470 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 23 10:18:39 smithi119 ceph-mon[32608]: using public_addr v2:172.21.15.119:0/0 -> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] 2024-04-23T10:18:39.470 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 23 10:18:39 smithi119 ceph-mon[32608]: starting mon.b rank -1 at public addrs [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] at bind addrs [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] mon_data /var/lib/ceph/mon/ceph-b fsid aa240572-0159-11ef-bc93-c7b262605968 2024-04-23T10:18:39.470 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 23 10:18:39 smithi119 ceph-mon[32608]: mon.b@-1(???) e0 preinit fsid aa240572-0159-11ef-bc93-c7b262605968 2024-04-23T10:18:39.470 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 23 10:18:39 smithi119 ceph-mon[32608]: mon.b@-1(synchronizing).mds e1 new map 2024-04-23T10:18:39.470 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 23 10:18:39 smithi119 ceph-mon[32608]: mon.b@-1(synchronizing).mds e1 print_map 2024-04-23T10:18:39.470 INFO:journalctl@ceph.mon.b.smithi119.stdout: e1 2024-04-23T10:18:39.470 INFO:journalctl@ceph.mon.b.smithi119.stdout: enable_multiple, ever_enabled_multiple: 1,1 2024-04-23T10:18:39.470 INFO:journalctl@ceph.mon.b.smithi119.stdout: default compat: compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,8=no anchor table,9=file layout v2,10=snaprealm v2,11=minor log segments,12=quiesce subvolumes} 2024-04-23T10:18:39.470 INFO:journalctl@ceph.mon.b.smithi119.stdout: legacy client fscid: -1 2024-04-23T10:18:39.470 INFO:journalctl@ceph.mon.b.smithi119.stdout: 2024-04-23T10:18:39.470 INFO:journalctl@ceph.mon.b.smithi119.stdout: No filesystems configured 2024-04-23T10:18:39.470 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 23 10:18:39 smithi119 ceph-mon[32608]: mon.b@-1(synchronizing).osd e0 _set_cache_ratios kv ratio 0.25 inc ratio 0.375 full ratio 0.375 2024-04-23T10:18:39.471 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 23 10:18:39 smithi119 ceph-mon[32608]: mon.b@-1(synchronizing).osd e0 register_cache_with_pcm pcm target: 2147483648 pcm max: 1020054732 pcm min: 134217728 inc_osd_cache size: 1 2024-04-23T10:18:39.471 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 23 10:18:39 smithi119 ceph-mon[32608]: mon.b@-1(synchronizing).osd e1 e1: 0 total, 0 up, 0 in 2024-04-23T10:18:39.471 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 23 10:18:39 smithi119 ceph-mon[32608]: mon.b@-1(synchronizing).osd e2 e2: 0 total, 0 up, 0 in 2024-04-23T10:18:39.471 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 23 10:18:39 smithi119 ceph-mon[32608]: mon.b@-1(synchronizing).osd e3 e3: 0 total, 0 up, 0 in 2024-04-23T10:18:39.471 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 23 10:18:39 smithi119 ceph-mon[32608]: mon.b@-1(synchronizing).osd e4 e4: 0 total, 0 up, 0 in 2024-04-23T10:18:39.471 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 23 10:18:39 smithi119 ceph-mon[32608]: mon.b@-1(synchronizing).osd e4 crush map has features 3314932999778484224, adjusting msgr requires 2024-04-23T10:18:39.471 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 23 10:18:39 smithi119 ceph-mon[32608]: mon.b@-1(synchronizing).osd e4 crush map has features 288514050185494528, adjusting msgr requires 2024-04-23T10:18:39.471 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 23 10:18:39 smithi119 ceph-mon[32608]: mon.b@-1(synchronizing).osd e4 crush map has features 288514050185494528, adjusting msgr requires 2024-04-23T10:18:39.471 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 23 10:18:39 smithi119 ceph-mon[32608]: mon.b@-1(synchronizing).osd e4 crush map has features 288514050185494528, adjusting msgr requires 2024-04-23T10:18:39.471 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 23 10:18:39 smithi119 ceph-mon[32608]: pgmap v6: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-23T10:18:39.471 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 23 10:18:39 smithi119 ceph-mon[32608]: from='mgr.14152 172.21.15.114:0/1529706138' entity='mgr.x' 2024-04-23T10:18:39.471 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 23 10:18:39 smithi119 ceph-mon[32608]: from='mgr.14152 172.21.15.114:0/1529706138' entity='mgr.x' 2024-04-23T10:18:39.471 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 23 10:18:39 smithi119 ceph-mon[32608]: from='mgr.14152 172.21.15.114:0/1529706138' entity='mgr.x' 2024-04-23T10:18:39.471 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 23 10:18:39 smithi119 ceph-mon[32608]: from='mgr.14152 172.21.15.114:0/1529706138' entity='mgr.x' 2024-04-23T10:18:39.471 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 23 10:18:39 smithi119 ceph-mon[32608]: from='mgr.14152 172.21.15.114:0/1529706138' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd/host:smithi119", "name": "osd_memory_target"}]: dispatch 2024-04-23T10:18:39.471 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 23 10:18:39 smithi119 ceph-mon[32608]: from='mgr.14152 172.21.15.114:0/1529706138' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-23T10:18:39.471 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 23 10:18:39 smithi119 ceph-mon[32608]: from='mgr.14152 172.21.15.114:0/1529706138' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-23T10:18:39.472 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 23 10:18:39 smithi119 ceph-mon[32608]: from='client.14182 -' entity='client.admin' cmd=[{"prefix": "orch host add", "hostname": "smithi204", "target": ["mon-mgr", ""]}]: dispatch 2024-04-23T10:18:39.472 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 23 10:18:39 smithi119 ceph-mon[32608]: Updating smithi119:/etc/ceph/ceph.conf 2024-04-23T10:18:39.472 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 23 10:18:39 smithi119 ceph-mon[32608]: Updating smithi119:/var/lib/ceph/aa240572-0159-11ef-bc93-c7b262605968/config/ceph.conf 2024-04-23T10:18:39.472 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 23 10:18:39 smithi119 ceph-mon[32608]: Updating smithi119:/etc/ceph/ceph.client.admin.keyring 2024-04-23T10:18:39.472 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 23 10:18:39 smithi119 ceph-mon[32608]: from='mgr.14152 172.21.15.114:0/1529706138' entity='mgr.x' 2024-04-23T10:18:39.472 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 23 10:18:39 smithi119 ceph-mon[32608]: from='mgr.14152 172.21.15.114:0/1529706138' entity='mgr.x' 2024-04-23T10:18:39.472 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 23 10:18:39 smithi119 ceph-mon[32608]: from='mgr.14152 172.21.15.114:0/1529706138' entity='mgr.x' 2024-04-23T10:18:39.472 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 23 10:18:39 smithi119 ceph-mon[32608]: Deploying cephadm binary to smithi204 2024-04-23T10:18:39.472 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 23 10:18:39 smithi119 ceph-mon[32608]: Updating smithi119:/var/lib/ceph/aa240572-0159-11ef-bc93-c7b262605968/config/ceph.client.admin.keyring 2024-04-23T10:18:39.472 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 23 10:18:39 smithi119 ceph-mon[32608]: pgmap v7: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-23T10:18:39.472 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 23 10:18:39 smithi119 ceph-mon[32608]: pgmap v8: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-23T10:18:39.472 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 23 10:18:39 smithi119 ceph-mon[32608]: from='mgr.14152 172.21.15.114:0/1529706138' entity='mgr.x' 2024-04-23T10:18:39.472 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 23 10:18:39 smithi119 ceph-mon[32608]: from='mgr.14152 172.21.15.114:0/1529706138' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-23T10:18:39.472 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 23 10:18:39 smithi119 ceph-mon[32608]: Added host smithi204 2024-04-23T10:18:39.472 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 23 10:18:39 smithi119 ceph-mon[32608]: from='mgr.14152 172.21.15.114:0/1529706138' entity='mgr.x' 2024-04-23T10:18:39.473 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 23 10:18:39 smithi119 ceph-mon[32608]: from='mgr.14152 172.21.15.114:0/1529706138' entity='mgr.x' 2024-04-23T10:18:39.473 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 23 10:18:39 smithi119 ceph-mon[32608]: pgmap v9: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-23T10:18:39.473 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 23 10:18:39 smithi119 ceph-mon[32608]: from='mgr.14152 172.21.15.114:0/1529706138' entity='mgr.x' 2024-04-23T10:18:39.473 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 23 10:18:39 smithi119 ceph-mon[32608]: from='client.14184 -' entity='client.admin' cmd=[{"prefix": "orch host ls", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-04-23T10:18:39.473 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 23 10:18:39 smithi119 ceph-mon[32608]: pgmap v10: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-23T10:18:39.473 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 23 10:18:39 smithi119 ceph-mon[32608]: from='client.? 172.21.15.114:0/261917642' entity='client.admin' cmd=[{"prefix": "osd crush tunables", "profile": "default"}]: dispatch 2024-04-23T10:18:39.473 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 23 10:18:39 smithi119 ceph-mon[32608]: from='client.? 172.21.15.114:0/261917642' entity='client.admin' cmd='[{"prefix": "osd crush tunables", "profile": "default"}]': finished 2024-04-23T10:18:39.473 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 23 10:18:39 smithi119 ceph-mon[32608]: osdmap e4: 0 total, 0 up, 0 in 2024-04-23T10:18:39.473 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 23 10:18:39 smithi119 ceph-mon[32608]: pgmap v12: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-23T10:18:39.473 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 23 10:18:39 smithi119 ceph-mon[32608]: from='mgr.14152 172.21.15.114:0/1529706138' entity='mgr.x' 2024-04-23T10:18:39.473 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 23 10:18:39 smithi119 ceph-mon[32608]: from='mgr.14152 172.21.15.114:0/1529706138' entity='mgr.x' 2024-04-23T10:18:39.473 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 23 10:18:39 smithi119 ceph-mon[32608]: from='mgr.14152 172.21.15.114:0/1529706138' entity='mgr.x' 2024-04-23T10:18:39.473 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 23 10:18:39 smithi119 ceph-mon[32608]: from='mgr.14152 172.21.15.114:0/1529706138' entity='mgr.x' 2024-04-23T10:18:39.473 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 23 10:18:39 smithi119 ceph-mon[32608]: from='mgr.14152 172.21.15.114:0/1529706138' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd/host:smithi204", "name": "osd_memory_target"}]: dispatch 2024-04-23T10:18:39.473 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 23 10:18:39 smithi119 ceph-mon[32608]: from='mgr.14152 172.21.15.114:0/1529706138' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-23T10:18:39.473 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 23 10:18:39 smithi119 ceph-mon[32608]: from='mgr.14152 172.21.15.114:0/1529706138' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-23T10:18:39.474 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 23 10:18:39 smithi119 ceph-mon[32608]: Updating smithi204:/etc/ceph/ceph.conf 2024-04-23T10:18:39.474 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 23 10:18:39 smithi119 ceph-mon[32608]: Updating smithi204:/var/lib/ceph/aa240572-0159-11ef-bc93-c7b262605968/config/ceph.conf 2024-04-23T10:18:39.474 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 23 10:18:39 smithi119 ceph-mon[32608]: Updating smithi204:/etc/ceph/ceph.client.admin.keyring 2024-04-23T10:18:39.474 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 23 10:18:39 smithi119 ceph-mon[32608]: Updating smithi204:/var/lib/ceph/aa240572-0159-11ef-bc93-c7b262605968/config/ceph.client.admin.keyring 2024-04-23T10:18:39.474 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 23 10:18:39 smithi119 ceph-mon[32608]: from='mgr.14152 172.21.15.114:0/1529706138' entity='mgr.x' 2024-04-23T10:18:39.474 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 23 10:18:39 smithi119 ceph-mon[32608]: from='mgr.14152 172.21.15.114:0/1529706138' entity='mgr.x' 2024-04-23T10:18:39.474 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 23 10:18:39 smithi119 ceph-mon[32608]: from='mgr.14152 172.21.15.114:0/1529706138' entity='mgr.x' 2024-04-23T10:18:39.474 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 23 10:18:39 smithi119 ceph-mon[32608]: pgmap v13: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-23T10:18:39.474 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 23 10:18:39 smithi119 ceph-mon[32608]: from='mgr.14152 172.21.15.114:0/1529706138' entity='mgr.x' 2024-04-23T10:18:39.474 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 23 10:18:39 smithi119 ceph-mon[32608]: from='mgr.14152 172.21.15.114:0/1529706138' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-23T10:18:39.474 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 23 10:18:39 smithi119 ceph-mon[32608]: from='mgr.14152 172.21.15.114:0/1529706138' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-23T10:18:39.474 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 23 10:18:39 smithi119 ceph-mon[32608]: from='mgr.14152 172.21.15.114:0/1529706138' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-23T10:18:39.474 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 23 10:18:39 smithi119 ceph-mon[32608]: from='mgr.14152 172.21.15.114:0/1529706138' entity='mgr.x' 2024-04-23T10:18:39.475 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 23 10:18:39 smithi119 ceph-mon[32608]: from='mgr.14152 172.21.15.114:0/1529706138' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-04-23T10:18:39.475 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 23 10:18:39 smithi119 ceph-mon[32608]: from='mgr.14152 172.21.15.114:0/1529706138' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-23T10:18:39.475 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 23 10:18:39 smithi119 ceph-mon[32608]: from='client.14188 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "mon", "placement": "3;smithi114:172.21.15.114=a;smithi119:172.21.15.119=b;smithi204:172.21.15.204=c", "target": ["mon-mgr", ""]}]: dispatch 2024-04-23T10:18:39.475 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 23 10:18:39 smithi119 ceph-mon[32608]: Saving service mon spec with placement smithi114:172.21.15.114=a;smithi119:172.21.15.119=b;smithi204:172.21.15.204=c;count:3 2024-04-23T10:18:39.475 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 23 10:18:39 smithi119 ceph-mon[32608]: Deploying daemon mon.c on smithi204 2024-04-23T10:18:39.475 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 23 10:18:39 smithi119 ceph-mon[32608]: pgmap v14: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-23T10:18:39.475 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 23 10:18:39 smithi119 ceph-mon[32608]: pgmap v15: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-23T10:18:39.475 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 23 10:18:39 smithi119 ceph-mon[32608]: from='client.? 172.21.15.204:0/2143757240' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-04-23T10:18:39.475 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 23 10:18:39 smithi119 ceph-mon[32608]: from='mgr.14152 172.21.15.114:0/1529706138' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-04-23T10:18:39.475 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 23 10:18:39 smithi119 ceph-mon[32608]: from='mgr.14152 172.21.15.114:0/1529706138' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-04-23T10:18:39.475 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 23 10:18:39 smithi119 ceph-mon[32608]: mon.a calling monitor election 2024-04-23T10:18:39.476 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 23 10:18:39 smithi119 ceph-mon[32608]: from='mgr.14152 172.21.15.114:0/1529706138' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-04-23T10:18:39.476 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 23 10:18:39 smithi119 ceph-mon[32608]: pgmap v17: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-23T10:18:39.476 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 23 10:18:39 smithi119 ceph-mon[32608]: from='mgr.14152 172.21.15.114:0/1529706138' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-04-23T10:18:39.476 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 23 10:18:39 smithi119 ceph-mon[32608]: mon.c calling monitor election 2024-04-23T10:18:39.476 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 23 10:18:39 smithi119 ceph-mon[32608]: from='mgr.14152 172.21.15.114:0/1529706138' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-04-23T10:18:39.476 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 23 10:18:39 smithi119 ceph-mon[32608]: pgmap v18: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-23T10:18:39.476 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 23 10:18:39 smithi119 ceph-mon[32608]: from='mgr.14152 172.21.15.114:0/1529706138' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-04-23T10:18:39.476 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 23 10:18:39 smithi119 ceph-mon[32608]: from='mgr.14152 172.21.15.114:0/1529706138' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-04-23T10:18:39.476 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 23 10:18:39 smithi119 ceph-mon[32608]: mon.a is new leader, mons a,c in quorum (ranks 0,1) 2024-04-23T10:18:39.476 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 23 10:18:39 smithi119 ceph-mon[32608]: monmap epoch 2 2024-04-23T10:18:39.476 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 23 10:18:39 smithi119 ceph-mon[32608]: fsid aa240572-0159-11ef-bc93-c7b262605968 2024-04-23T10:18:39.476 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 23 10:18:39 smithi119 ceph-mon[32608]: last_changed 2024-04-23T10:18:30.187213+0000 2024-04-23T10:18:39.476 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 23 10:18:39 smithi119 ceph-mon[32608]: created 2024-04-23T10:16:50.385874+0000 2024-04-23T10:18:39.476 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 23 10:18:39 smithi119 ceph-mon[32608]: min_mon_release 19 (squid) 2024-04-23T10:18:39.476 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 23 10:18:39 smithi119 ceph-mon[32608]: election_strategy: 1 2024-04-23T10:18:39.476 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 23 10:18:39 smithi119 ceph-mon[32608]: 0: [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] mon.a 2024-04-23T10:18:39.477 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 23 10:18:39 smithi119 ceph-mon[32608]: 1: [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] mon.c 2024-04-23T10:18:39.477 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 23 10:18:39 smithi119 ceph-mon[32608]: fsmap 2024-04-23T10:18:39.477 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 23 10:18:39 smithi119 ceph-mon[32608]: osdmap e4: 0 total, 0 up, 0 in 2024-04-23T10:18:39.477 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 23 10:18:39 smithi119 ceph-mon[32608]: mgrmap e13: x(active, since 49s) 2024-04-23T10:18:39.477 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 23 10:18:39 smithi119 ceph-mon[32608]: overall HEALTH_OK 2024-04-23T10:18:39.477 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 23 10:18:39 smithi119 ceph-mon[32608]: from='mgr.14152 172.21.15.114:0/1529706138' entity='mgr.x' 2024-04-23T10:18:39.477 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 23 10:18:39 smithi119 ceph-mon[32608]: from='mgr.14152 172.21.15.114:0/1529706138' entity='mgr.x' 2024-04-23T10:18:39.477 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 23 10:18:39 smithi119 ceph-mon[32608]: from='mgr.14152 172.21.15.114:0/1529706138' entity='mgr.x' 2024-04-23T10:18:39.477 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 23 10:18:39 smithi119 ceph-mon[32608]: from='mgr.14152 172.21.15.114:0/1529706138' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-04-23T10:18:39.477 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 23 10:18:39 smithi119 ceph-mon[32608]: from='mgr.14152 172.21.15.114:0/1529706138' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-23T10:18:39.477 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 23 10:18:39 smithi119 ceph-mon[32608]: Deploying daemon mon.b on smithi119 2024-04-23T10:18:39.477 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 23 10:18:39 smithi119 ceph-mon[32608]: from='client.? 172.21.15.204:0/2078698815' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-04-23T10:18:39.477 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 23 10:18:39 smithi119 ceph-mon[32608]: pgmap v19: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-23T10:18:39.477 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 23 10:18:39 smithi119 ceph-mon[32608]: from='mgr.14152 172.21.15.114:0/1529706138' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-04-23T10:18:39.477 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 23 10:18:39 smithi119 ceph-mon[32608]: pgmap v20: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-23T10:18:39.477 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 23 10:18:39 smithi119 ceph-mon[32608]: mon.b@-1(synchronizing).paxosservice(auth 1..3) refresh upgraded, format 0 -> 3 2024-04-23T10:18:39.478 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 23 10:18:39 smithi119 bash[32588]: 903c3f3bd1b4516ae98306d6c9d5ed5e42d049cd068de3764e2881d530a5c7ad 2024-04-23T10:18:39.478 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 23 10:18:39 smithi119 systemd[1]: Started Ceph mon.b for aa240572-0159-11ef-bc93-c7b262605968. 2024-04-23T10:18:44.226 INFO:teuthology.orchestra.run.smithi204.stderr:2024-04-23T10:18:44.227+0000 7fd7657fa700 1 -- 172.21.15.204:0/4292325444 <== mon.0 v2:172.21.15.114:3300/0 8 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 3 v3) ==== 95+0+1287 (secure 0 0 0) 0x7fd758030090 con 0x7fd768108630 2024-04-23T10:18:44.226 INFO:teuthology.orchestra.run.smithi204.stdout: 2024-04-23T10:18:44.227 INFO:teuthology.orchestra.run.smithi204.stdout:{"epoch":3,"fsid":"aa240572-0159-11ef-bc93-c7b262605968","modified":"2024-04-23T10:18:39.217399Z","created":"2024-04-23T10:16:50.385874Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:3300","nonce":0},{"type":"v1","addr":"172.21.15.114:6789","nonce":0}]},"addr":"172.21.15.114:6789/0","public_addr":"172.21.15.114:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:3300","nonce":0},{"type":"v1","addr":"172.21.15.204:6789","nonce":0}]},"addr":"172.21.15.204:6789/0","public_addr":"172.21.15.204:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:3300","nonce":0},{"type":"v1","addr":"172.21.15.119:6789","nonce":0}]},"addr":"172.21.15.119:6789/0","public_addr":"172.21.15.119:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0,1]} 2024-04-23T10:18:44.229 INFO:teuthology.orchestra.run.smithi204.stderr:2024-04-23T10:18:44.230+0000 7fd76e5fa700 1 -- 172.21.15.204:0/4292325444 >> [v2:172.21.15.114:6800/465333808,v1:172.21.15.114:6801/465333808] conn(0x7fd750039440 msgr2=0x7fd75003b900 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T10:18:44.230 INFO:teuthology.orchestra.run.smithi204.stderr:2024-04-23T10:18:44.230+0000 7fd76e5fa700 1 --2- 172.21.15.204:0/4292325444 >> [v2:172.21.15.114:6800/465333808,v1:172.21.15.114:6801/465333808] conn(0x7fd750039440 0x7fd75003b900 secure :-1 s=READY pgs=30 cs=0 l=1 rev1=1 crypto rx=0x7fd75c009180 tx=0x7fd75c008040 comp rx=0 tx=0).stop 2024-04-23T10:18:44.230 INFO:teuthology.orchestra.run.smithi204.stderr:2024-04-23T10:18:44.230+0000 7fd76e5fa700 1 -- 172.21.15.204:0/4292325444 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fd768108630 msgr2=0x7fd7681981c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T10:18:44.230 INFO:teuthology.orchestra.run.smithi204.stderr:2024-04-23T10:18:44.230+0000 7fd76e5fa700 1 --2- 172.21.15.204:0/4292325444 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fd768108630 0x7fd7681981c0 secure :-1 s=READY pgs=103 cs=0 l=1 rev1=1 crypto rx=0x7fd758014000 tx=0x7fd758005030 comp rx=0 tx=0).stop 2024-04-23T10:18:44.230 INFO:teuthology.orchestra.run.smithi204.stderr:2024-04-23T10:18:44.230+0000 7fd76e5fa700 1 -- 172.21.15.204:0/4292325444 shutdown_connections 2024-04-23T10:18:44.230 INFO:teuthology.orchestra.run.smithi204.stderr:2024-04-23T10:18:44.230+0000 7fd76e5fa700 1 --2- 172.21.15.204:0/4292325444 >> [v2:172.21.15.114:6800/465333808,v1:172.21.15.114:6801/465333808] conn(0x7fd750039440 0x7fd75003b900 unknown :-1 s=CLOSED pgs=30 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:18:44.230 INFO:teuthology.orchestra.run.smithi204.stderr:2024-04-23T10:18:44.230+0000 7fd76e5fa700 1 --2- 172.21.15.204:0/4292325444 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fd768108630 0x7fd7681981c0 unknown :-1 s=CLOSED pgs=103 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:18:44.230 INFO:teuthology.orchestra.run.smithi204.stderr:2024-04-23T10:18:44.230+0000 7fd76e5fa700 1 --2- 172.21.15.204:0/4292325444 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fd768198700 0x7fd76819ca90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:18:44.230 INFO:teuthology.orchestra.run.smithi204.stderr:2024-04-23T10:18:44.230+0000 7fd76e5fa700 1 -- 172.21.15.204:0/4292325444 >> 172.21.15.204:0/4292325444 conn(0x7fd7680fe050 msgr2=0x7fd7680fe430 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T10:18:44.231 INFO:teuthology.orchestra.run.smithi204.stderr:2024-04-23T10:18:44.231+0000 7fd76e5fa700 1 -- 172.21.15.204:0/4292325444 shutdown_connections 2024-04-23T10:18:44.231 INFO:teuthology.orchestra.run.smithi204.stderr:2024-04-23T10:18:44.231+0000 7fd76e5fa700 1 -- 172.21.15.204:0/4292325444 wait complete. 2024-04-23T10:18:44.231 INFO:teuthology.orchestra.run.smithi204.stderr:dumped monmap epoch 3 2024-04-23T10:18:44.546 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 23 10:18:44 smithi204 ceph-mon[32649]: from='mgr.14152 172.21.15.114:0/1529706138' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-04-23T10:18:44.546 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 23 10:18:44 smithi204 ceph-mon[32649]: mon.a calling monitor election 2024-04-23T10:18:44.546 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 23 10:18:44 smithi204 ceph-mon[32649]: mon.c calling monitor election 2024-04-23T10:18:44.546 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 23 10:18:44 smithi204 ceph-mon[32649]: from='mgr.14152 172.21.15.114:0/1529706138' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-04-23T10:18:44.546 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 23 10:18:44 smithi204 ceph-mon[32649]: from='mgr.14152 172.21.15.114:0/1529706138' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-04-23T10:18:44.546 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 23 10:18:44 smithi204 ceph-mon[32649]: from='client.? 172.21.15.204:0/4292325444' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-04-23T10:18:44.546 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 23 10:18:44 smithi204 ceph-mon[32649]: pgmap v21: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-23T10:18:44.547 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 23 10:18:44 smithi204 ceph-mon[32649]: from='mgr.14152 172.21.15.114:0/1529706138' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-04-23T10:18:44.547 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 23 10:18:44 smithi204 ceph-mon[32649]: from='mgr.14152 172.21.15.114:0/1529706138' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-04-23T10:18:44.547 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 23 10:18:44 smithi204 ceph-mon[32649]: pgmap v22: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-23T10:18:44.547 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 23 10:18:44 smithi204 ceph-mon[32649]: from='mgr.14152 172.21.15.114:0/1529706138' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-04-23T10:18:44.547 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 23 10:18:44 smithi204 ceph-mon[32649]: from='mgr.14152 172.21.15.114:0/1529706138' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-04-23T10:18:44.547 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 23 10:18:44 smithi204 ceph-mon[32649]: pgmap v23: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-23T10:18:44.547 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 23 10:18:44 smithi204 ceph-mon[32649]: from='mgr.14152 172.21.15.114:0/1529706138' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-04-23T10:18:44.547 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 23 10:18:44 smithi204 ceph-mon[32649]: mon.a is new leader, mons a,c in quorum (ranks 0,1) 2024-04-23T10:18:44.547 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 23 10:18:44 smithi204 ceph-mon[32649]: monmap epoch 3 2024-04-23T10:18:44.547 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 23 10:18:44 smithi204 ceph-mon[32649]: fsid aa240572-0159-11ef-bc93-c7b262605968 2024-04-23T10:18:44.547 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 23 10:18:44 smithi204 ceph-mon[32649]: last_changed 2024-04-23T10:18:39.217399+0000 2024-04-23T10:18:44.547 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 23 10:18:44 smithi204 ceph-mon[32649]: created 2024-04-23T10:16:50.385874+0000 2024-04-23T10:18:44.548 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 23 10:18:44 smithi204 ceph-mon[32649]: min_mon_release 19 (squid) 2024-04-23T10:18:44.548 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 23 10:18:44 smithi204 ceph-mon[32649]: election_strategy: 1 2024-04-23T10:18:44.548 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 23 10:18:44 smithi204 ceph-mon[32649]: 0: [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] mon.a 2024-04-23T10:18:44.548 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 23 10:18:44 smithi204 ceph-mon[32649]: 1: [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] mon.c 2024-04-23T10:18:44.548 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 23 10:18:44 smithi204 ceph-mon[32649]: 2: [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] mon.b 2024-04-23T10:18:44.548 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 23 10:18:44 smithi204 ceph-mon[32649]: fsmap 2024-04-23T10:18:44.548 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 23 10:18:44 smithi204 ceph-mon[32649]: osdmap e4: 0 total, 0 up, 0 in 2024-04-23T10:18:44.548 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 23 10:18:44 smithi204 ceph-mon[32649]: mgrmap e13: x(active, since 58s) 2024-04-23T10:18:44.548 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 23 10:18:44 smithi204 ceph-mon[32649]: overall HEALTH_OK 2024-04-23T10:18:44.548 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 23 10:18:44 smithi204 ceph-mon[32649]: from='mgr.14152 172.21.15.114:0/1529706138' entity='mgr.x' 2024-04-23T10:18:44.548 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 23 10:18:44 smithi204 ceph-mon[32649]: from='mgr.14152 172.21.15.114:0/1529706138' entity='mgr.x' 2024-04-23T10:18:44.661 INFO:journalctl@ceph.mon.a.smithi114.stdout:Apr 23 10:18:44 smithi114 ceph-mon[29255]: from='mgr.14152 172.21.15.114:0/1529706138' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-04-23T10:18:44.773 INFO:journalctl@ceph.mon.a.smithi114.stdout:Apr 23 10:18:44 smithi114 ceph-mon[29255]: mon.a calling monitor election 2024-04-23T10:18:44.773 INFO:journalctl@ceph.mon.a.smithi114.stdout:Apr 23 10:18:44 smithi114 ceph-mon[29255]: mon.c calling monitor election 2024-04-23T10:18:44.773 INFO:journalctl@ceph.mon.a.smithi114.stdout:Apr 23 10:18:44 smithi114 ceph-mon[29255]: from='mgr.14152 172.21.15.114:0/1529706138' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-04-23T10:18:44.773 INFO:journalctl@ceph.mon.a.smithi114.stdout:Apr 23 10:18:44 smithi114 ceph-mon[29255]: from='mgr.14152 172.21.15.114:0/1529706138' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-04-23T10:18:44.773 INFO:journalctl@ceph.mon.a.smithi114.stdout:Apr 23 10:18:44 smithi114 ceph-mon[29255]: from='client.? 172.21.15.204:0/4292325444' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-04-23T10:18:44.774 INFO:journalctl@ceph.mon.a.smithi114.stdout:Apr 23 10:18:44 smithi114 ceph-mon[29255]: pgmap v21: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-23T10:18:44.774 INFO:journalctl@ceph.mon.a.smithi114.stdout:Apr 23 10:18:44 smithi114 ceph-mon[29255]: from='mgr.14152 172.21.15.114:0/1529706138' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-04-23T10:18:44.774 INFO:journalctl@ceph.mon.a.smithi114.stdout:Apr 23 10:18:44 smithi114 ceph-mon[29255]: from='mgr.14152 172.21.15.114:0/1529706138' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-04-23T10:18:44.774 INFO:journalctl@ceph.mon.a.smithi114.stdout:Apr 23 10:18:44 smithi114 ceph-mon[29255]: pgmap v22: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-23T10:18:44.774 INFO:journalctl@ceph.mon.a.smithi114.stdout:Apr 23 10:18:44 smithi114 ceph-mon[29255]: from='mgr.14152 172.21.15.114:0/1529706138' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-04-23T10:18:44.774 INFO:journalctl@ceph.mon.a.smithi114.stdout:Apr 23 10:18:44 smithi114 ceph-mon[29255]: from='mgr.14152 172.21.15.114:0/1529706138' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-04-23T10:18:44.774 INFO:journalctl@ceph.mon.a.smithi114.stdout:Apr 23 10:18:44 smithi114 ceph-mon[29255]: pgmap v23: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-23T10:18:44.774 INFO:journalctl@ceph.mon.a.smithi114.stdout:Apr 23 10:18:44 smithi114 ceph-mon[29255]: from='mgr.14152 172.21.15.114:0/1529706138' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-04-23T10:18:44.774 INFO:journalctl@ceph.mon.a.smithi114.stdout:Apr 23 10:18:44 smithi114 ceph-mon[29255]: mon.a is new leader, mons a,c in quorum (ranks 0,1) 2024-04-23T10:18:44.774 INFO:journalctl@ceph.mon.a.smithi114.stdout:Apr 23 10:18:44 smithi114 ceph-mon[29255]: monmap epoch 3 2024-04-23T10:18:44.774 INFO:journalctl@ceph.mon.a.smithi114.stdout:Apr 23 10:18:44 smithi114 ceph-mon[29255]: fsid aa240572-0159-11ef-bc93-c7b262605968 2024-04-23T10:18:44.775 INFO:journalctl@ceph.mon.a.smithi114.stdout:Apr 23 10:18:44 smithi114 ceph-mon[29255]: last_changed 2024-04-23T10:18:39.217399+0000 2024-04-23T10:18:44.775 INFO:journalctl@ceph.mon.a.smithi114.stdout:Apr 23 10:18:44 smithi114 ceph-mon[29255]: created 2024-04-23T10:16:50.385874+0000 2024-04-23T10:18:44.775 INFO:journalctl@ceph.mon.a.smithi114.stdout:Apr 23 10:18:44 smithi114 ceph-mon[29255]: min_mon_release 19 (squid) 2024-04-23T10:18:44.775 INFO:journalctl@ceph.mon.a.smithi114.stdout:Apr 23 10:18:44 smithi114 ceph-mon[29255]: election_strategy: 1 2024-04-23T10:18:44.775 INFO:journalctl@ceph.mon.a.smithi114.stdout:Apr 23 10:18:44 smithi114 ceph-mon[29255]: 0: [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] mon.a 2024-04-23T10:18:44.775 INFO:journalctl@ceph.mon.a.smithi114.stdout:Apr 23 10:18:44 smithi114 ceph-mon[29255]: 1: [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] mon.c 2024-04-23T10:18:44.775 INFO:journalctl@ceph.mon.a.smithi114.stdout:Apr 23 10:18:44 smithi114 ceph-mon[29255]: 2: [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] mon.b 2024-04-23T10:18:44.775 INFO:journalctl@ceph.mon.a.smithi114.stdout:Apr 23 10:18:44 smithi114 ceph-mon[29255]: fsmap 2024-04-23T10:18:44.775 INFO:journalctl@ceph.mon.a.smithi114.stdout:Apr 23 10:18:44 smithi114 ceph-mon[29255]: osdmap e4: 0 total, 0 up, 0 in 2024-04-23T10:18:44.775 INFO:journalctl@ceph.mon.a.smithi114.stdout:Apr 23 10:18:44 smithi114 ceph-mon[29255]: mgrmap e13: x(active, since 58s) 2024-04-23T10:18:44.775 INFO:journalctl@ceph.mon.a.smithi114.stdout:Apr 23 10:18:44 smithi114 ceph-mon[29255]: overall HEALTH_OK 2024-04-23T10:18:44.775 INFO:journalctl@ceph.mon.a.smithi114.stdout:Apr 23 10:18:44 smithi114 ceph-mon[29255]: from='mgr.14152 172.21.15.114:0/1529706138' entity='mgr.x' 2024-04-23T10:18:44.776 INFO:journalctl@ceph.mon.a.smithi114.stdout:Apr 23 10:18:44 smithi114 ceph-mon[29255]: from='mgr.14152 172.21.15.114:0/1529706138' entity='mgr.x' 2024-04-23T10:18:44.816 INFO:tasks.cephadm:Generating final ceph.conf file... 2024-04-23T10:18:44.816 DEBUG:teuthology.orchestra.run.smithi114:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d69f13d827952bb07bb9b7973bca14f97d764d6f shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid aa240572-0159-11ef-bc93-c7b262605968 -- ceph config generate-minimal-conf 2024-04-23T10:18:45.055 INFO:teuthology.orchestra.run.smithi114.stderr:Inferring config /var/lib/ceph/aa240572-0159-11ef-bc93-c7b262605968/mon.a/config 2024-04-23T10:18:45.504 INFO:journalctl@ceph.mon.a.smithi114.stdout:Apr 23 10:18:45 smithi114 ceph-mon[29255]: from='mgr.14152 172.21.15.114:0/1529706138' entity='mgr.x' 2024-04-23T10:18:45.504 INFO:journalctl@ceph.mon.a.smithi114.stdout:Apr 23 10:18:45 smithi114 ceph-mon[29255]: from='mgr.14152 172.21.15.114:0/1529706138' entity='mgr.x' 2024-04-23T10:18:45.505 INFO:journalctl@ceph.mon.a.smithi114.stdout:Apr 23 10:18:45 smithi114 ceph-mon[29255]: from='mgr.14152 172.21.15.114:0/1529706138' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-23T10:18:45.505 INFO:journalctl@ceph.mon.a.smithi114.stdout:Apr 23 10:18:45 smithi114 ceph-mon[29255]: from='mgr.14152 172.21.15.114:0/1529706138' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-04-23T10:18:45.578 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 23 10:18:45 smithi204 ceph-mon[32649]: from='mgr.14152 172.21.15.114:0/1529706138' entity='mgr.x' 2024-04-23T10:18:45.578 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 23 10:18:45 smithi204 ceph-mon[32649]: from='mgr.14152 172.21.15.114:0/1529706138' entity='mgr.x' 2024-04-23T10:18:45.578 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 23 10:18:45 smithi204 ceph-mon[32649]: from='mgr.14152 172.21.15.114:0/1529706138' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-23T10:18:45.578 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 23 10:18:45 smithi204 ceph-mon[32649]: from='mgr.14152 172.21.15.114:0/1529706138' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-04-23T10:18:46.493 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 23 10:18:46 smithi119 ceph-mon[32608]: from='mgr.14152 172.21.15.114:0/1529706138' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-04-23T10:18:46.493 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 23 10:18:46 smithi119 ceph-mon[32608]: mon.a calling monitor election 2024-04-23T10:18:46.493 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 23 10:18:46 smithi119 ceph-mon[32608]: mon.c calling monitor election 2024-04-23T10:18:46.493 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 23 10:18:46 smithi119 ceph-mon[32608]: from='mgr.14152 172.21.15.114:0/1529706138' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-04-23T10:18:46.493 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 23 10:18:46 smithi119 ceph-mon[32608]: from='mgr.14152 172.21.15.114:0/1529706138' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-04-23T10:18:46.493 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 23 10:18:46 smithi119 ceph-mon[32608]: from='client.? 172.21.15.204:0/4292325444' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-04-23T10:18:46.493 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 23 10:18:46 smithi119 ceph-mon[32608]: pgmap v21: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-23T10:18:46.493 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 23 10:18:46 smithi119 ceph-mon[32608]: from='mgr.14152 172.21.15.114:0/1529706138' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-04-23T10:18:46.494 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 23 10:18:46 smithi119 ceph-mon[32608]: from='mgr.14152 172.21.15.114:0/1529706138' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-04-23T10:18:46.494 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 23 10:18:46 smithi119 ceph-mon[32608]: pgmap v22: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-23T10:18:46.494 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 23 10:18:46 smithi119 ceph-mon[32608]: from='mgr.14152 172.21.15.114:0/1529706138' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-04-23T10:18:46.494 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 23 10:18:46 smithi119 ceph-mon[32608]: from='mgr.14152 172.21.15.114:0/1529706138' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-04-23T10:18:46.494 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 23 10:18:46 smithi119 ceph-mon[32608]: pgmap v23: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-23T10:18:46.494 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 23 10:18:46 smithi119 ceph-mon[32608]: from='mgr.14152 172.21.15.114:0/1529706138' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-04-23T10:18:46.494 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 23 10:18:46 smithi119 ceph-mon[32608]: mon.a is new leader, mons a,c in quorum (ranks 0,1) 2024-04-23T10:18:46.494 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 23 10:18:46 smithi119 ceph-mon[32608]: monmap epoch 3 2024-04-23T10:18:46.494 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 23 10:18:46 smithi119 ceph-mon[32608]: fsid aa240572-0159-11ef-bc93-c7b262605968 2024-04-23T10:18:46.495 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 23 10:18:46 smithi119 ceph-mon[32608]: last_changed 2024-04-23T10:18:39.217399+0000 2024-04-23T10:18:46.495 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 23 10:18:46 smithi119 ceph-mon[32608]: created 2024-04-23T10:16:50.385874+0000 2024-04-23T10:18:46.495 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 23 10:18:46 smithi119 ceph-mon[32608]: min_mon_release 19 (squid) 2024-04-23T10:18:46.495 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 23 10:18:46 smithi119 ceph-mon[32608]: election_strategy: 1 2024-04-23T10:18:46.495 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 23 10:18:46 smithi119 ceph-mon[32608]: 0: [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] mon.a 2024-04-23T10:18:46.495 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 23 10:18:46 smithi119 ceph-mon[32608]: 1: [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] mon.c 2024-04-23T10:18:46.495 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 23 10:18:46 smithi119 ceph-mon[32608]: 2: [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] mon.b 2024-04-23T10:18:46.495 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 23 10:18:46 smithi119 ceph-mon[32608]: fsmap 2024-04-23T10:18:46.495 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 23 10:18:46 smithi119 ceph-mon[32608]: osdmap e4: 0 total, 0 up, 0 in 2024-04-23T10:18:46.495 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 23 10:18:46 smithi119 ceph-mon[32608]: mgrmap e13: x(active, since 58s) 2024-04-23T10:18:46.496 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 23 10:18:46 smithi119 ceph-mon[32608]: overall HEALTH_OK 2024-04-23T10:18:46.496 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 23 10:18:46 smithi119 ceph-mon[32608]: from='mgr.14152 172.21.15.114:0/1529706138' entity='mgr.x' 2024-04-23T10:18:46.496 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 23 10:18:46 smithi119 ceph-mon[32608]: from='mgr.14152 172.21.15.114:0/1529706138' entity='mgr.x' 2024-04-23T10:18:46.496 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 23 10:18:46 smithi119 ceph-mon[32608]: from='mgr.14152 172.21.15.114:0/1529706138' entity='mgr.x' 2024-04-23T10:18:46.496 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 23 10:18:46 smithi119 ceph-mon[32608]: from='mgr.14152 172.21.15.114:0/1529706138' entity='mgr.x' 2024-04-23T10:18:46.496 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 23 10:18:46 smithi119 ceph-mon[32608]: from='mgr.14152 172.21.15.114:0/1529706138' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-23T10:18:46.496 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 23 10:18:46 smithi119 ceph-mon[32608]: from='mgr.14152 172.21.15.114:0/1529706138' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-04-23T10:18:46.780 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:18:46.780+0000 7fdf44e24700 1 -- 172.21.15.114:0/2974895372 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fdf401084d0 msgr2=0x7fdf401088b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T10:18:46.780 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:18:46.780+0000 7fdf44e24700 1 --2- 172.21.15.114:0/2974895372 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fdf401084d0 0x7fdf401088b0 secure :-1 s=READY pgs=106 cs=0 l=1 rev1=1 crypto rx=0x7fdf30009910 tx=0x7fdf30009c20 comp rx=0 tx=0).stop 2024-04-23T10:18:46.780 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:18:46.780+0000 7fdf44e24700 1 -- 172.21.15.114:0/2974895372 shutdown_connections 2024-04-23T10:18:46.780 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:18:46.780+0000 7fdf44e24700 1 --2- 172.21.15.114:0/2974895372 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fdf401084d0 0x7fdf401088b0 unknown :-1 s=CLOSED pgs=106 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:18:46.780 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:18:46.780+0000 7fdf44e24700 1 -- 172.21.15.114:0/2974895372 >> 172.21.15.114:0/2974895372 conn(0x7fdf400fdf50 msgr2=0x7fdf40100370 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T10:18:46.780 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:18:46.780+0000 7fdf44e24700 1 -- 172.21.15.114:0/2974895372 shutdown_connections 2024-04-23T10:18:46.780 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:18:46.780+0000 7fdf44e24700 1 -- 172.21.15.114:0/2974895372 wait complete. 2024-04-23T10:18:46.784 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:18:46.781+0000 7fdf44e24700 1 Processor -- start 2024-04-23T10:18:46.784 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:18:46.781+0000 7fdf44e24700 1 -- start start 2024-04-23T10:18:46.784 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:18:46.781+0000 7fdf44e24700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fdf401084d0 0x7fdf40198170 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T10:18:46.784 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:18:46.781+0000 7fdf44e24700 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fdf401986b0 0x7fdf4019ca40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T10:18:46.784 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:18:46.781+0000 7fdf44e24700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fdf4019cf80 0x7fdf4019d430 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T10:18:46.784 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:18:46.781+0000 7fdf44e24700 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x7fdf4010c930 con 0x7fdf401084d0 2024-04-23T10:18:46.784 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:18:46.781+0000 7fdf44e24700 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 -- 0x7fdf4010c7b0 con 0x7fdf4019cf80 2024-04-23T10:18:46.784 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:18:46.781+0000 7fdf44e24700 1 -- --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_getmap magic: 0 -- 0x7fdf4010cab0 con 0x7fdf401986b0 2024-04-23T10:18:46.784 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:18:46.781+0000 7fdf3e59c700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fdf401084d0 0x7fdf40198170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T10:18:46.785 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:18:46.781+0000 7fdf3e59c700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fdf401084d0 0x7fdf40198170 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.114:35700/0 (socket says 172.21.15.114:35700) 2024-04-23T10:18:46.785 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:18:46.781+0000 7fdf3e59c700 1 -- 172.21.15.114:0/1551542762 learned_addr learned my addr 172.21.15.114:0/1551542762 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-04-23T10:18:46.785 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:18:46.781+0000 7fdf3ed9d700 1 --2- 172.21.15.114:0/1551542762 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fdf4019cf80 0x7fdf4019d430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T10:18:46.785 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:18:46.781+0000 7fdf3dd9b700 1 --2- 172.21.15.114:0/1551542762 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fdf401986b0 0x7fdf4019ca40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T10:18:46.785 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:18:46.782+0000 7fdf3e59c700 1 -- 172.21.15.114:0/1551542762 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fdf401986b0 msgr2=0x7fdf4019ca40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T10:18:46.785 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:18:46.782+0000 7fdf3e59c700 1 --2- 172.21.15.114:0/1551542762 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fdf401986b0 0x7fdf4019ca40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:18:46.785 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:18:46.782+0000 7fdf3e59c700 1 -- 172.21.15.114:0/1551542762 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fdf4019cf80 msgr2=0x7fdf4019d430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T10:18:46.785 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:18:46.782+0000 7fdf3e59c700 1 --2- 172.21.15.114:0/1551542762 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fdf4019cf80 0x7fdf4019d430 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:18:46.785 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:18:46.782+0000 7fdf3e59c700 1 -- 172.21.15.114:0/1551542762 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fdf401a4cf0 con 0x7fdf401084d0 2024-04-23T10:18:46.785 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:18:46.782+0000 7fdf3e59c700 1 --2- 172.21.15.114:0/1551542762 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fdf401084d0 0x7fdf40198170 secure :-1 s=READY pgs=107 cs=0 l=1 rev1=1 crypto rx=0x7fdf30005b40 tx=0x7fdf30004f80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T10:18:46.786 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:18:46.782+0000 7fdf2f7fe700 1 -- 172.21.15.114:0/1551542762 <== mon.0 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdf30026070 con 0x7fdf401084d0 2024-04-23T10:18:46.786 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:18:46.782+0000 7fdf2f7fe700 1 -- 172.21.15.114:0/1551542762 <== mon.0 v2:172.21.15.114:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fdf3000b6f0 con 0x7fdf401084d0 2024-04-23T10:18:46.786 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:18:46.782+0000 7fdf44e24700 1 -- 172.21.15.114:0/1551542762 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fdf40074cd0 con 0x7fdf401084d0 2024-04-23T10:18:46.786 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:18:46.782+0000 7fdf2f7fe700 1 -- 172.21.15.114:0/1551542762 <== mon.0 v2:172.21.15.114:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdf30009e60 con 0x7fdf401084d0 2024-04-23T10:18:46.786 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:18:46.782+0000 7fdf44e24700 1 -- 172.21.15.114:0/1551542762 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fdf40075260 con 0x7fdf401084d0 2024-04-23T10:18:46.786 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:18:46.783+0000 7fdf2f7fe700 1 -- 172.21.15.114:0/1551542762 <== mon.0 v2:172.21.15.114:3300/0 4 ==== mgrmap(e 13) ==== 46615+0+0 (secure 0 0 0) 0x7fdf30029070 con 0x7fdf401084d0 2024-04-23T10:18:46.786 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:18:46.783+0000 7fdf44e24700 1 -- 172.21.15.114:0/1551542762 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fdf400640a0 con 0x7fdf401084d0 2024-04-23T10:18:46.786 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:18:46.783+0000 7fdf2f7fe700 1 --2- 172.21.15.114:0/1551542762 >> [v2:172.21.15.114:6800/465333808,v1:172.21.15.114:6801/465333808] conn(0x7fdf28039ae0 0x7fdf2803bfa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T10:18:46.786 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:18:46.783+0000 7fdf2f7fe700 1 -- 172.21.15.114:0/1551542762 <== mon.0 v2:172.21.15.114:3300/0 5 ==== osd_map(4..4 src has 1..4) ==== 1155+0+0 (secure 0 0 0) 0x7fdf3004d470 con 0x7fdf401084d0 2024-04-23T10:18:46.787 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:18:46.784+0000 7fdf3dd9b700 1 --2- 172.21.15.114:0/1551542762 >> [v2:172.21.15.114:6800/465333808,v1:172.21.15.114:6801/465333808] conn(0x7fdf28039ae0 0x7fdf2803bfa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T10:18:46.787 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:18:46.784+0000 7fdf3dd9b700 1 --2- 172.21.15.114:0/1551542762 >> [v2:172.21.15.114:6800/465333808,v1:172.21.15.114:6801/465333808] conn(0x7fdf28039ae0 0x7fdf2803bfa0 secure :-1 s=READY pgs=39 cs=0 l=1 rev1=1 crypto rx=0x7fdf34009ad0 tx=0x7fdf34009210 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T10:18:46.788 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:18:46.788+0000 7fdf2f7fe700 1 -- 172.21.15.114:0/1551542762 <== mon.0 v2:172.21.15.114:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7fdf30013070 con 0x7fdf401084d0 2024-04-23T10:18:46.999 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:18:46.999+0000 7fdf44e24700 1 -- 172.21.15.114:0/1551542762 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_command({"prefix": "config generate-minimal-conf"} v 0) -- 0x7fdf40075580 con 0x7fdf401084d0 2024-04-23T10:18:46.999 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:18:47.000+0000 7fdf2f7fe700 1 -- 172.21.15.114:0/1551542762 <== mon.0 v2:172.21.15.114:3300/0 7 ==== mon_command_ack([{"prefix": "config generate-minimal-conf"}]=0 v9) ==== 76+0+277 (secure 0 0 0) 0x7fdf30030090 con 0x7fdf401084d0 2024-04-23T10:18:46.999 INFO:teuthology.orchestra.run.smithi114.stdout:# minimal ceph.conf for aa240572-0159-11ef-bc93-c7b262605968 2024-04-23T10:18:47.000 INFO:teuthology.orchestra.run.smithi114.stdout:[global] 2024-04-23T10:18:47.000 INFO:teuthology.orchestra.run.smithi114.stdout: fsid = aa240572-0159-11ef-bc93-c7b262605968 2024-04-23T10:18:47.000 INFO:teuthology.orchestra.run.smithi114.stdout: mon_host = [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] 2024-04-23T10:18:47.001 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:18:47.002+0000 7fdf44e24700 1 -- 172.21.15.114:0/1551542762 >> [v2:172.21.15.114:6800/465333808,v1:172.21.15.114:6801/465333808] conn(0x7fdf28039ae0 msgr2=0x7fdf2803bfa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T10:18:47.002 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:18:47.002+0000 7fdf44e24700 1 --2- 172.21.15.114:0/1551542762 >> [v2:172.21.15.114:6800/465333808,v1:172.21.15.114:6801/465333808] conn(0x7fdf28039ae0 0x7fdf2803bfa0 secure :-1 s=READY pgs=39 cs=0 l=1 rev1=1 crypto rx=0x7fdf34009ad0 tx=0x7fdf34009210 comp rx=0 tx=0).stop 2024-04-23T10:18:47.002 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:18:47.002+0000 7fdf44e24700 1 -- 172.21.15.114:0/1551542762 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fdf401084d0 msgr2=0x7fdf40198170 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T10:18:47.002 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:18:47.002+0000 7fdf44e24700 1 --2- 172.21.15.114:0/1551542762 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fdf401084d0 0x7fdf40198170 secure :-1 s=READY pgs=107 cs=0 l=1 rev1=1 crypto rx=0x7fdf30005b40 tx=0x7fdf30004f80 comp rx=0 tx=0).stop 2024-04-23T10:18:47.002 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:18:47.002+0000 7fdf44e24700 1 -- 172.21.15.114:0/1551542762 shutdown_connections 2024-04-23T10:18:47.002 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:18:47.002+0000 7fdf44e24700 1 --2- 172.21.15.114:0/1551542762 >> [v2:172.21.15.114:6800/465333808,v1:172.21.15.114:6801/465333808] conn(0x7fdf28039ae0 0x7fdf2803bfa0 unknown :-1 s=CLOSED pgs=39 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:18:47.002 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:18:47.002+0000 7fdf44e24700 1 --2- 172.21.15.114:0/1551542762 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fdf4019cf80 0x7fdf4019d430 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:18:47.002 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:18:47.002+0000 7fdf44e24700 1 --2- 172.21.15.114:0/1551542762 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fdf401084d0 0x7fdf40198170 unknown :-1 s=CLOSED pgs=107 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:18:47.002 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:18:47.002+0000 7fdf44e24700 1 --2- 172.21.15.114:0/1551542762 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fdf401986b0 0x7fdf4019ca40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:18:47.003 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:18:47.002+0000 7fdf44e24700 1 -- 172.21.15.114:0/1551542762 >> 172.21.15.114:0/1551542762 conn(0x7fdf400fdf50 msgr2=0x7fdf4010a110 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T10:18:47.003 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:18:47.002+0000 7fdf44e24700 1 -- 172.21.15.114:0/1551542762 shutdown_connections 2024-04-23T10:18:47.003 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:18:47.002+0000 7fdf44e24700 1 -- 172.21.15.114:0/1551542762 wait complete. 2024-04-23T10:18:47.242 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 23 10:18:46 smithi119 ceph-mon[32608]: mon.b calling monitor election 2024-04-23T10:18:47.242 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 23 10:18:46 smithi119 ceph-mon[32608]: mon.b calling monitor election 2024-04-23T10:18:47.243 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 23 10:18:46 smithi119 ceph-mon[32608]: mon.a calling monitor election 2024-04-23T10:18:47.243 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 23 10:18:46 smithi119 ceph-mon[32608]: mon.c calling monitor election 2024-04-23T10:18:47.243 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 23 10:18:46 smithi119 ceph-mon[32608]: mon.a is new leader, mons a,c,b in quorum (ranks 0,1,2) 2024-04-23T10:18:47.243 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 23 10:18:46 smithi119 ceph-mon[32608]: monmap epoch 3 2024-04-23T10:18:47.243 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 23 10:18:46 smithi119 ceph-mon[32608]: fsid aa240572-0159-11ef-bc93-c7b262605968 2024-04-23T10:18:47.243 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 23 10:18:46 smithi119 ceph-mon[32608]: last_changed 2024-04-23T10:18:39.217399+0000 2024-04-23T10:18:47.243 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 23 10:18:46 smithi119 ceph-mon[32608]: created 2024-04-23T10:16:50.385874+0000 2024-04-23T10:18:47.243 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 23 10:18:46 smithi119 ceph-mon[32608]: min_mon_release 19 (squid) 2024-04-23T10:18:47.243 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 23 10:18:46 smithi119 ceph-mon[32608]: election_strategy: 1 2024-04-23T10:18:47.243 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 23 10:18:46 smithi119 ceph-mon[32608]: 0: [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] mon.a 2024-04-23T10:18:47.244 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 23 10:18:46 smithi119 ceph-mon[32608]: 1: [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] mon.c 2024-04-23T10:18:47.244 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 23 10:18:46 smithi119 ceph-mon[32608]: 2: [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] mon.b 2024-04-23T10:18:47.244 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 23 10:18:46 smithi119 ceph-mon[32608]: fsmap 2024-04-23T10:18:47.244 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 23 10:18:46 smithi119 ceph-mon[32608]: osdmap e4: 0 total, 0 up, 0 in 2024-04-23T10:18:47.244 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 23 10:18:46 smithi119 ceph-mon[32608]: mgrmap e13: x(active, since 60s) 2024-04-23T10:18:47.244 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 23 10:18:46 smithi119 ceph-mon[32608]: overall HEALTH_OK 2024-04-23T10:18:47.244 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 23 10:18:46 smithi119 ceph-mon[32608]: from='mgr.14152 172.21.15.114:0/1529706138' entity='mgr.x' 2024-04-23T10:18:47.244 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 23 10:18:46 smithi119 ceph-mon[32608]: from='mgr.14152 172.21.15.114:0/1529706138' entity='mgr.x' 2024-04-23T10:18:47.244 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 23 10:18:46 smithi119 ceph-mon[32608]: from='mgr.14152 172.21.15.114:0/1529706138' entity='mgr.x' 2024-04-23T10:18:47.244 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 23 10:18:46 smithi119 ceph-mon[32608]: from='mgr.14152 172.21.15.114:0/1529706138' entity='mgr.x' 2024-04-23T10:18:47.244 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 23 10:18:46 smithi119 ceph-mon[32608]: from='mgr.14152 172.21.15.114:0/1529706138' entity='mgr.x' 2024-04-23T10:18:47.245 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 23 10:18:46 smithi119 ceph-mon[32608]: from='mgr.14152 172.21.15.114:0/1529706138' entity='mgr.x' 2024-04-23T10:18:47.245 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 23 10:18:46 smithi119 ceph-mon[32608]: from='mgr.14152 172.21.15.114:0/1529706138' entity='mgr.x' 2024-04-23T10:18:47.245 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 23 10:18:46 smithi119 ceph-mon[32608]: from='mgr.14152 172.21.15.114:0/1529706138' entity='mgr.x' 2024-04-23T10:18:47.245 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 23 10:18:46 smithi119 ceph-mon[32608]: from='mgr.14152 172.21.15.114:0/1529706138' entity='mgr.x' 2024-04-23T10:18:47.245 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 23 10:18:46 smithi119 ceph-mon[32608]: from='mgr.14152 172.21.15.114:0/1529706138' entity='mgr.x' 2024-04-23T10:18:47.245 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 23 10:18:46 smithi119 ceph-mon[32608]: from='mgr.14152 172.21.15.114:0/1529706138' entity='mgr.x' 2024-04-23T10:18:47.245 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 23 10:18:46 smithi119 ceph-mon[32608]: from='mgr.14152 172.21.15.114:0/1529706138' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-04-23T10:18:47.245 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 23 10:18:46 smithi119 ceph-mon[32608]: from='mgr.14152 172.21.15.114:0/1529706138' entity='mgr.x' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-04-23T10:18:47.245 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 23 10:18:46 smithi119 ceph-mon[32608]: from='mgr.14152 172.21.15.114:0/1529706138' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-23T10:18:47.255 INFO:journalctl@ceph.mon.a.smithi114.stdout:Apr 23 10:18:46 smithi114 ceph-mon[29255]: mon.b calling monitor election 2024-04-23T10:18:47.255 INFO:journalctl@ceph.mon.a.smithi114.stdout:Apr 23 10:18:46 smithi114 ceph-mon[29255]: mon.b calling monitor election 2024-04-23T10:18:47.255 INFO:journalctl@ceph.mon.a.smithi114.stdout:Apr 23 10:18:46 smithi114 ceph-mon[29255]: mon.a calling monitor election 2024-04-23T10:18:47.255 INFO:journalctl@ceph.mon.a.smithi114.stdout:Apr 23 10:18:46 smithi114 ceph-mon[29255]: mon.c calling monitor election 2024-04-23T10:18:47.255 INFO:journalctl@ceph.mon.a.smithi114.stdout:Apr 23 10:18:46 smithi114 ceph-mon[29255]: mon.a is new leader, mons a,c,b in quorum (ranks 0,1,2) 2024-04-23T10:18:47.255 INFO:journalctl@ceph.mon.a.smithi114.stdout:Apr 23 10:18:46 smithi114 ceph-mon[29255]: monmap epoch 3 2024-04-23T10:18:47.255 INFO:journalctl@ceph.mon.a.smithi114.stdout:Apr 23 10:18:46 smithi114 ceph-mon[29255]: fsid aa240572-0159-11ef-bc93-c7b262605968 2024-04-23T10:18:47.256 INFO:journalctl@ceph.mon.a.smithi114.stdout:Apr 23 10:18:46 smithi114 ceph-mon[29255]: last_changed 2024-04-23T10:18:39.217399+0000 2024-04-23T10:18:47.256 INFO:journalctl@ceph.mon.a.smithi114.stdout:Apr 23 10:18:46 smithi114 ceph-mon[29255]: created 2024-04-23T10:16:50.385874+0000 2024-04-23T10:18:47.256 INFO:journalctl@ceph.mon.a.smithi114.stdout:Apr 23 10:18:46 smithi114 ceph-mon[29255]: min_mon_release 19 (squid) 2024-04-23T10:18:47.256 INFO:journalctl@ceph.mon.a.smithi114.stdout:Apr 23 10:18:46 smithi114 ceph-mon[29255]: election_strategy: 1 2024-04-23T10:18:47.256 INFO:journalctl@ceph.mon.a.smithi114.stdout:Apr 23 10:18:46 smithi114 ceph-mon[29255]: 0: [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] mon.a 2024-04-23T10:18:47.256 INFO:journalctl@ceph.mon.a.smithi114.stdout:Apr 23 10:18:46 smithi114 ceph-mon[29255]: 1: [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] mon.c 2024-04-23T10:18:47.256 INFO:journalctl@ceph.mon.a.smithi114.stdout:Apr 23 10:18:46 smithi114 ceph-mon[29255]: 2: [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] mon.b 2024-04-23T10:18:47.256 INFO:journalctl@ceph.mon.a.smithi114.stdout:Apr 23 10:18:46 smithi114 ceph-mon[29255]: fsmap 2024-04-23T10:18:47.257 INFO:journalctl@ceph.mon.a.smithi114.stdout:Apr 23 10:18:46 smithi114 ceph-mon[29255]: osdmap e4: 0 total, 0 up, 0 in 2024-04-23T10:18:47.257 INFO:journalctl@ceph.mon.a.smithi114.stdout:Apr 23 10:18:46 smithi114 ceph-mon[29255]: mgrmap e13: x(active, since 60s) 2024-04-23T10:18:47.257 INFO:journalctl@ceph.mon.a.smithi114.stdout:Apr 23 10:18:46 smithi114 ceph-mon[29255]: overall HEALTH_OK 2024-04-23T10:18:47.257 INFO:journalctl@ceph.mon.a.smithi114.stdout:Apr 23 10:18:46 smithi114 ceph-mon[29255]: from='mgr.14152 172.21.15.114:0/1529706138' entity='mgr.x' 2024-04-23T10:18:47.257 INFO:journalctl@ceph.mon.a.smithi114.stdout:Apr 23 10:18:46 smithi114 ceph-mon[29255]: from='mgr.14152 172.21.15.114:0/1529706138' entity='mgr.x' 2024-04-23T10:18:47.257 INFO:journalctl@ceph.mon.a.smithi114.stdout:Apr 23 10:18:46 smithi114 ceph-mon[29255]: from='mgr.14152 172.21.15.114:0/1529706138' entity='mgr.x' 2024-04-23T10:18:47.257 INFO:journalctl@ceph.mon.a.smithi114.stdout:Apr 23 10:18:46 smithi114 ceph-mon[29255]: from='mgr.14152 172.21.15.114:0/1529706138' entity='mgr.x' 2024-04-23T10:18:47.257 INFO:journalctl@ceph.mon.a.smithi114.stdout:Apr 23 10:18:46 smithi114 ceph-mon[29255]: from='mgr.14152 172.21.15.114:0/1529706138' entity='mgr.x' 2024-04-23T10:18:47.257 INFO:journalctl@ceph.mon.a.smithi114.stdout:Apr 23 10:18:46 smithi114 ceph-mon[29255]: from='mgr.14152 172.21.15.114:0/1529706138' entity='mgr.x' 2024-04-23T10:18:47.257 INFO:journalctl@ceph.mon.a.smithi114.stdout:Apr 23 10:18:46 smithi114 ceph-mon[29255]: from='mgr.14152 172.21.15.114:0/1529706138' entity='mgr.x' 2024-04-23T10:18:47.258 INFO:journalctl@ceph.mon.a.smithi114.stdout:Apr 23 10:18:46 smithi114 ceph-mon[29255]: from='mgr.14152 172.21.15.114:0/1529706138' entity='mgr.x' 2024-04-23T10:18:47.258 INFO:journalctl@ceph.mon.a.smithi114.stdout:Apr 23 10:18:46 smithi114 ceph-mon[29255]: from='mgr.14152 172.21.15.114:0/1529706138' entity='mgr.x' 2024-04-23T10:18:47.258 INFO:journalctl@ceph.mon.a.smithi114.stdout:Apr 23 10:18:46 smithi114 ceph-mon[29255]: from='mgr.14152 172.21.15.114:0/1529706138' entity='mgr.x' 2024-04-23T10:18:47.258 INFO:journalctl@ceph.mon.a.smithi114.stdout:Apr 23 10:18:46 smithi114 ceph-mon[29255]: from='mgr.14152 172.21.15.114:0/1529706138' entity='mgr.x' 2024-04-23T10:18:47.258 INFO:journalctl@ceph.mon.a.smithi114.stdout:Apr 23 10:18:46 smithi114 ceph-mon[29255]: from='mgr.14152 172.21.15.114:0/1529706138' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-04-23T10:18:47.258 INFO:journalctl@ceph.mon.a.smithi114.stdout:Apr 23 10:18:46 smithi114 ceph-mon[29255]: from='mgr.14152 172.21.15.114:0/1529706138' entity='mgr.x' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-04-23T10:18:47.258 INFO:journalctl@ceph.mon.a.smithi114.stdout:Apr 23 10:18:46 smithi114 ceph-mon[29255]: from='mgr.14152 172.21.15.114:0/1529706138' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-23T10:18:47.328 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 23 10:18:46 smithi204 ceph-mon[32649]: mon.b calling monitor election 2024-04-23T10:18:47.329 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 23 10:18:46 smithi204 ceph-mon[32649]: mon.b calling monitor election 2024-04-23T10:18:47.329 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 23 10:18:46 smithi204 ceph-mon[32649]: mon.a calling monitor election 2024-04-23T10:18:47.329 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 23 10:18:46 smithi204 ceph-mon[32649]: mon.c calling monitor election 2024-04-23T10:18:47.329 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 23 10:18:46 smithi204 ceph-mon[32649]: mon.a is new leader, mons a,c,b in quorum (ranks 0,1,2) 2024-04-23T10:18:47.329 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 23 10:18:46 smithi204 ceph-mon[32649]: monmap epoch 3 2024-04-23T10:18:47.329 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 23 10:18:46 smithi204 ceph-mon[32649]: fsid aa240572-0159-11ef-bc93-c7b262605968 2024-04-23T10:18:47.329 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 23 10:18:46 smithi204 ceph-mon[32649]: last_changed 2024-04-23T10:18:39.217399+0000 2024-04-23T10:18:47.330 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 23 10:18:46 smithi204 ceph-mon[32649]: created 2024-04-23T10:16:50.385874+0000 2024-04-23T10:18:47.330 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 23 10:18:46 smithi204 ceph-mon[32649]: min_mon_release 19 (squid) 2024-04-23T10:18:47.330 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 23 10:18:46 smithi204 ceph-mon[32649]: election_strategy: 1 2024-04-23T10:18:47.330 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 23 10:18:46 smithi204 ceph-mon[32649]: 0: [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] mon.a 2024-04-23T10:18:47.330 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 23 10:18:46 smithi204 ceph-mon[32649]: 1: [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] mon.c 2024-04-23T10:18:47.330 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 23 10:18:46 smithi204 ceph-mon[32649]: 2: [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] mon.b 2024-04-23T10:18:47.330 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 23 10:18:46 smithi204 ceph-mon[32649]: fsmap 2024-04-23T10:18:47.330 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 23 10:18:46 smithi204 ceph-mon[32649]: osdmap e4: 0 total, 0 up, 0 in 2024-04-23T10:18:47.331 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 23 10:18:46 smithi204 ceph-mon[32649]: mgrmap e13: x(active, since 60s) 2024-04-23T10:18:47.331 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 23 10:18:46 smithi204 ceph-mon[32649]: overall HEALTH_OK 2024-04-23T10:18:47.331 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 23 10:18:46 smithi204 ceph-mon[32649]: from='mgr.14152 172.21.15.114:0/1529706138' entity='mgr.x' 2024-04-23T10:18:47.331 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 23 10:18:46 smithi204 ceph-mon[32649]: from='mgr.14152 172.21.15.114:0/1529706138' entity='mgr.x' 2024-04-23T10:18:47.331 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 23 10:18:46 smithi204 ceph-mon[32649]: from='mgr.14152 172.21.15.114:0/1529706138' entity='mgr.x' 2024-04-23T10:18:47.331 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 23 10:18:46 smithi204 ceph-mon[32649]: from='mgr.14152 172.21.15.114:0/1529706138' entity='mgr.x' 2024-04-23T10:18:47.331 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 23 10:18:46 smithi204 ceph-mon[32649]: from='mgr.14152 172.21.15.114:0/1529706138' entity='mgr.x' 2024-04-23T10:18:47.332 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 23 10:18:46 smithi204 ceph-mon[32649]: from='mgr.14152 172.21.15.114:0/1529706138' entity='mgr.x' 2024-04-23T10:18:47.332 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 23 10:18:46 smithi204 ceph-mon[32649]: from='mgr.14152 172.21.15.114:0/1529706138' entity='mgr.x' 2024-04-23T10:18:47.332 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 23 10:18:46 smithi204 ceph-mon[32649]: from='mgr.14152 172.21.15.114:0/1529706138' entity='mgr.x' 2024-04-23T10:18:47.332 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 23 10:18:46 smithi204 ceph-mon[32649]: from='mgr.14152 172.21.15.114:0/1529706138' entity='mgr.x' 2024-04-23T10:18:47.332 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 23 10:18:46 smithi204 ceph-mon[32649]: from='mgr.14152 172.21.15.114:0/1529706138' entity='mgr.x' 2024-04-23T10:18:47.332 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 23 10:18:46 smithi204 ceph-mon[32649]: from='mgr.14152 172.21.15.114:0/1529706138' entity='mgr.x' 2024-04-23T10:18:47.332 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 23 10:18:46 smithi204 ceph-mon[32649]: from='mgr.14152 172.21.15.114:0/1529706138' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-04-23T10:18:47.332 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 23 10:18:46 smithi204 ceph-mon[32649]: from='mgr.14152 172.21.15.114:0/1529706138' entity='mgr.x' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-04-23T10:18:47.333 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 23 10:18:46 smithi204 ceph-mon[32649]: from='mgr.14152 172.21.15.114:0/1529706138' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-23T10:18:47.717 INFO:tasks.cephadm:Distributing (final) config and client.admin keyring... 2024-04-23T10:18:47.718 DEBUG:teuthology.orchestra.run.smithi114:> set -ex 2024-04-23T10:18:47.718 DEBUG:teuthology.orchestra.run.smithi114:> sudo dd of=/etc/ceph/ceph.conf 2024-04-23T10:18:47.745 DEBUG:teuthology.orchestra.run.smithi114:> set -ex 2024-04-23T10:18:47.745 DEBUG:teuthology.orchestra.run.smithi114:> sudo dd of=/etc/ceph/ceph.client.admin.keyring 2024-04-23T10:18:47.809 DEBUG:teuthology.orchestra.run.smithi119:> set -ex 2024-04-23T10:18:47.809 DEBUG:teuthology.orchestra.run.smithi119:> sudo dd of=/etc/ceph/ceph.conf 2024-04-23T10:18:47.841 DEBUG:teuthology.orchestra.run.smithi119:> set -ex 2024-04-23T10:18:47.841 DEBUG:teuthology.orchestra.run.smithi119:> sudo dd of=/etc/ceph/ceph.client.admin.keyring 2024-04-23T10:18:47.909 DEBUG:teuthology.orchestra.run.smithi204:> set -ex 2024-04-23T10:18:47.910 DEBUG:teuthology.orchestra.run.smithi204:> sudo dd of=/etc/ceph/ceph.conf 2024-04-23T10:18:47.942 DEBUG:teuthology.orchestra.run.smithi204:> set -ex 2024-04-23T10:18:47.942 DEBUG:teuthology.orchestra.run.smithi204:> sudo dd of=/etc/ceph/ceph.client.admin.keyring 2024-04-23T10:18:47.998 INFO:journalctl@ceph.mon.a.smithi114.stdout:Apr 23 10:18:47 smithi114 ceph-mon[29255]: Reconfiguring mon.a (unknown last config time)... 2024-04-23T10:18:47.998 INFO:journalctl@ceph.mon.a.smithi114.stdout:Apr 23 10:18:47 smithi114 ceph-mon[29255]: Reconfiguring daemon mon.a on smithi114 2024-04-23T10:18:47.999 INFO:journalctl@ceph.mon.a.smithi114.stdout:Apr 23 10:18:47 smithi114 ceph-mon[29255]: from='client.? 172.21.15.114:0/1551542762' entity='client.admin' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-23T10:18:47.999 INFO:journalctl@ceph.mon.a.smithi114.stdout:Apr 23 10:18:47 smithi114 ceph-mon[29255]: from='mgr.14152 172.21.15.114:0/1529706138' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-04-23T10:18:48.011 INFO:tasks.cephadm:Adding mgr.x on smithi114 2024-04-23T10:18:48.011 INFO:tasks.cephadm:Adding mgr.y on smithi119 2024-04-23T10:18:48.011 INFO:tasks.cephadm:Adding mgr.z on smithi204 2024-04-23T10:18:48.011 DEBUG:teuthology.orchestra.run.smithi204:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d69f13d827952bb07bb9b7973bca14f97d764d6f shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid aa240572-0159-11ef-bc93-c7b262605968 -- ceph orch apply mgr '3;smithi114=x;smithi119=y;smithi204=z' 2024-04-23T10:18:48.242 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 23 10:18:47 smithi119 ceph-mon[32608]: Reconfiguring mon.a (unknown last config time)... 2024-04-23T10:18:48.242 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 23 10:18:47 smithi119 ceph-mon[32608]: Reconfiguring daemon mon.a on smithi114 2024-04-23T10:18:48.242 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 23 10:18:47 smithi119 ceph-mon[32608]: from='client.? 172.21.15.114:0/1551542762' entity='client.admin' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-23T10:18:48.242 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 23 10:18:47 smithi119 ceph-mon[32608]: from='mgr.14152 172.21.15.114:0/1529706138' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-04-23T10:18:48.282 INFO:teuthology.orchestra.run.smithi204.stderr:Inferring config /var/lib/ceph/aa240572-0159-11ef-bc93-c7b262605968/mon.c/config 2024-04-23T10:18:48.328 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 23 10:18:47 smithi204 ceph-mon[32649]: Reconfiguring mon.a (unknown last config time)... 2024-04-23T10:18:48.328 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 23 10:18:47 smithi204 ceph-mon[32649]: Reconfiguring daemon mon.a on smithi114 2024-04-23T10:18:48.328 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 23 10:18:47 smithi204 ceph-mon[32649]: from='client.? 172.21.15.114:0/1551542762' entity='client.admin' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-23T10:18:48.328 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 23 10:18:47 smithi204 ceph-mon[32649]: from='mgr.14152 172.21.15.114:0/1529706138' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-04-23T10:18:49.200 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 23 10:18:48 smithi119 ceph-mon[32608]: pgmap v25: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-23T10:18:49.201 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 23 10:18:48 smithi119 ceph-mon[32608]: from='mgr.14152 172.21.15.114:0/1529706138' entity='mgr.x' 2024-04-23T10:18:49.201 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 23 10:18:48 smithi119 ceph-mon[32608]: from='mgr.14152 172.21.15.114:0/1529706138' entity='mgr.x' 2024-04-23T10:18:49.201 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 23 10:18:48 smithi119 ceph-mon[32608]: from='mgr.14152 172.21.15.114:0/1529706138' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-04-23T10:18:49.201 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 23 10:18:48 smithi119 ceph-mon[32608]: from='mgr.14152 172.21.15.114:0/1529706138' entity='mgr.x' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-04-23T10:18:49.201 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 23 10:18:48 smithi119 ceph-mon[32608]: from='mgr.14152 172.21.15.114:0/1529706138' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-23T10:18:49.254 INFO:journalctl@ceph.mon.a.smithi114.stdout:Apr 23 10:18:48 smithi114 ceph-mon[29255]: pgmap v25: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-23T10:18:49.254 INFO:journalctl@ceph.mon.a.smithi114.stdout:Apr 23 10:18:48 smithi114 ceph-mon[29255]: from='mgr.14152 172.21.15.114:0/1529706138' entity='mgr.x' 2024-04-23T10:18:49.254 INFO:journalctl@ceph.mon.a.smithi114.stdout:Apr 23 10:18:48 smithi114 ceph-mon[29255]: from='mgr.14152 172.21.15.114:0/1529706138' entity='mgr.x' 2024-04-23T10:18:49.254 INFO:journalctl@ceph.mon.a.smithi114.stdout:Apr 23 10:18:48 smithi114 ceph-mon[29255]: from='mgr.14152 172.21.15.114:0/1529706138' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-04-23T10:18:49.254 INFO:journalctl@ceph.mon.a.smithi114.stdout:Apr 23 10:18:48 smithi114 ceph-mon[29255]: from='mgr.14152 172.21.15.114:0/1529706138' entity='mgr.x' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-04-23T10:18:49.255 INFO:journalctl@ceph.mon.a.smithi114.stdout:Apr 23 10:18:48 smithi114 ceph-mon[29255]: from='mgr.14152 172.21.15.114:0/1529706138' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-23T10:18:49.328 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 23 10:18:48 smithi204 ceph-mon[32649]: pgmap v25: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-23T10:18:49.328 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 23 10:18:48 smithi204 ceph-mon[32649]: from='mgr.14152 172.21.15.114:0/1529706138' entity='mgr.x' 2024-04-23T10:18:49.328 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 23 10:18:48 smithi204 ceph-mon[32649]: from='mgr.14152 172.21.15.114:0/1529706138' entity='mgr.x' 2024-04-23T10:18:49.328 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 23 10:18:48 smithi204 ceph-mon[32649]: from='mgr.14152 172.21.15.114:0/1529706138' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-04-23T10:18:49.328 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 23 10:18:48 smithi204 ceph-mon[32649]: from='mgr.14152 172.21.15.114:0/1529706138' entity='mgr.x' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-04-23T10:18:49.328 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 23 10:18:48 smithi204 ceph-mon[32649]: from='mgr.14152 172.21.15.114:0/1529706138' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-23T10:18:49.637 INFO:teuthology.orchestra.run.smithi204.stderr:2024-04-23T10:18:49.636+0000 7f3c9abba700 1 -- 172.21.15.204:0/1473093998 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f3c941014e0 msgr2=0x7f3c94101900 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T10:18:49.638 INFO:teuthology.orchestra.run.smithi204.stderr:2024-04-23T10:18:49.636+0000 7f3c9abba700 1 --2- 172.21.15.204:0/1473093998 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f3c941014e0 0x7f3c94101900 secure :-1 s=READY pgs=108 cs=0 l=1 rev1=1 crypto rx=0x7f3c88009910 tx=0x7f3c88009c20 comp rx=0 tx=0).stop 2024-04-23T10:18:49.638 INFO:teuthology.orchestra.run.smithi204.stderr:2024-04-23T10:18:49.636+0000 7f3c9abba700 1 -- 172.21.15.204:0/1473093998 shutdown_connections 2024-04-23T10:18:49.638 INFO:teuthology.orchestra.run.smithi204.stderr:2024-04-23T10:18:49.636+0000 7f3c9abba700 1 --2- 172.21.15.204:0/1473093998 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f3c941014e0 0x7f3c94101900 unknown :-1 s=CLOSED pgs=108 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:18:49.638 INFO:teuthology.orchestra.run.smithi204.stderr:2024-04-23T10:18:49.636+0000 7f3c9abba700 1 -- 172.21.15.204:0/1473093998 >> 172.21.15.204:0/1473093998 conn(0x7f3c940fca80 msgr2=0x7f3c940feec0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T10:18:49.638 INFO:teuthology.orchestra.run.smithi204.stderr:2024-04-23T10:18:49.636+0000 7f3c9abba700 1 -- 172.21.15.204:0/1473093998 shutdown_connections 2024-04-23T10:18:49.638 INFO:teuthology.orchestra.run.smithi204.stderr:2024-04-23T10:18:49.636+0000 7f3c9abba700 1 -- 172.21.15.204:0/1473093998 wait complete. 2024-04-23T10:18:49.638 INFO:teuthology.orchestra.run.smithi204.stderr:2024-04-23T10:18:49.636+0000 7f3c9abba700 1 Processor -- start 2024-04-23T10:18:49.638 INFO:teuthology.orchestra.run.smithi204.stderr:2024-04-23T10:18:49.636+0000 7f3c9abba700 1 -- start start 2024-04-23T10:18:49.638 INFO:teuthology.orchestra.run.smithi204.stderr:2024-04-23T10:18:49.637+0000 7f3c9abba700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f3c941014e0 0x7f3c9406b7b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T10:18:49.638 INFO:teuthology.orchestra.run.smithi204.stderr:2024-04-23T10:18:49.637+0000 7f3c9abba700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f3c9419aad0 0x7f3c9419cec0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T10:18:49.638 INFO:teuthology.orchestra.run.smithi204.stderr:2024-04-23T10:18:49.637+0000 7f3c9abba700 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f3c9419d450 0x7f3c9419f8e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T10:18:49.638 INFO:teuthology.orchestra.run.smithi204.stderr:2024-04-23T10:18:49.637+0000 7f3c9abba700 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x7f3c94107420 con 0x7f3c9419aad0 2024-04-23T10:18:49.639 INFO:teuthology.orchestra.run.smithi204.stderr:2024-04-23T10:18:49.637+0000 7f3c9abba700 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 -- 0x7f3c941072a0 con 0x7f3c941014e0 2024-04-23T10:18:49.639 INFO:teuthology.orchestra.run.smithi204.stderr:2024-04-23T10:18:49.637+0000 7f3c9abba700 1 -- --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_getmap magic: 0 -- 0x7f3c941075a0 con 0x7f3c9419d450 2024-04-23T10:18:49.639 INFO:teuthology.orchestra.run.smithi204.stderr:2024-04-23T10:18:49.637+0000 7f3c93fff700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f3c9419aad0 0x7f3c9419cec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T10:18:49.639 INFO:teuthology.orchestra.run.smithi204.stderr:2024-04-23T10:18:49.637+0000 7f3c93fff700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f3c9419aad0 0x7f3c9419cec0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.204:37952/0 (socket says 172.21.15.204:37952) 2024-04-23T10:18:49.639 INFO:teuthology.orchestra.run.smithi204.stderr:2024-04-23T10:18:49.637+0000 7f3c93fff700 1 -- 172.21.15.204:0/3645080897 learned_addr learned my addr 172.21.15.204:0/3645080897 (peer_addr_for_me v2:172.21.15.204:0/0) 2024-04-23T10:18:49.639 INFO:teuthology.orchestra.run.smithi204.stderr:2024-04-23T10:18:49.637+0000 7f3c99157700 1 --2- 172.21.15.204:0/3645080897 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f3c9419d450 0x7f3c9419f8e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T10:18:49.639 INFO:teuthology.orchestra.run.smithi204.stderr:2024-04-23T10:18:49.637+0000 7f3c93fff700 1 -- 172.21.15.204:0/3645080897 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f3c9419d450 msgr2=0x7f3c9419f8e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T10:18:49.639 INFO:teuthology.orchestra.run.smithi204.stderr:2024-04-23T10:18:49.637+0000 7f3c93fff700 1 --2- 172.21.15.204:0/3645080897 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f3c9419d450 0x7f3c9419f8e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:18:49.639 INFO:teuthology.orchestra.run.smithi204.stderr:2024-04-23T10:18:49.637+0000 7f3c98956700 1 --2- 172.21.15.204:0/3645080897 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f3c941014e0 0x7f3c9406b7b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T10:18:49.639 INFO:teuthology.orchestra.run.smithi204.stderr:2024-04-23T10:18:49.637+0000 7f3c93fff700 1 -- 172.21.15.204:0/3645080897 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f3c941014e0 msgr2=0x7f3c9406b7b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T10:18:49.639 INFO:teuthology.orchestra.run.smithi204.stderr:2024-04-23T10:18:49.637+0000 7f3c93fff700 1 --2- 172.21.15.204:0/3645080897 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f3c941014e0 0x7f3c9406b7b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:18:49.639 INFO:teuthology.orchestra.run.smithi204.stderr:2024-04-23T10:18:49.637+0000 7f3c93fff700 1 -- 172.21.15.204:0/3645080897 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f3c941a01b0 con 0x7f3c9419aad0 2024-04-23T10:18:49.639 INFO:teuthology.orchestra.run.smithi204.stderr:2024-04-23T10:18:49.637+0000 7f3c93fff700 1 --2- 172.21.15.204:0/3645080897 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f3c9419aad0 0x7f3c9419cec0 secure :-1 s=READY pgs=109 cs=0 l=1 rev1=1 crypto rx=0x7f3c8000b780 tx=0x7f3c8000bb40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T10:18:49.640 INFO:teuthology.orchestra.run.smithi204.stderr:2024-04-23T10:18:49.637+0000 7f3c91ffb700 1 -- 172.21.15.204:0/3645080897 <== mon.0 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3c80004230 con 0x7f3c9419aad0 2024-04-23T10:18:49.640 INFO:teuthology.orchestra.run.smithi204.stderr:2024-04-23T10:18:49.638+0000 7f3c91ffb700 1 -- 172.21.15.204:0/3645080897 <== mon.0 v2:172.21.15.114:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f3c800043d0 con 0x7f3c9419aad0 2024-04-23T10:18:49.640 INFO:teuthology.orchestra.run.smithi204.stderr:2024-04-23T10:18:49.638+0000 7f3c91ffb700 1 -- 172.21.15.204:0/3645080897 <== mon.0 v2:172.21.15.114:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3c80011bd0 con 0x7f3c9419aad0 2024-04-23T10:18:49.640 INFO:teuthology.orchestra.run.smithi204.stderr:2024-04-23T10:18:49.638+0000 7f3c9abba700 1 -- 172.21.15.204:0/3645080897 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f3c941a04a0 con 0x7f3c9419aad0 2024-04-23T10:18:49.640 INFO:teuthology.orchestra.run.smithi204.stderr:2024-04-23T10:18:49.638+0000 7f3c9abba700 1 -- 172.21.15.204:0/3645080897 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f3c941a0ab0 con 0x7f3c9419aad0 2024-04-23T10:18:49.641 INFO:teuthology.orchestra.run.smithi204.stderr:2024-04-23T10:18:49.638+0000 7f3c91ffb700 1 -- 172.21.15.204:0/3645080897 <== mon.0 v2:172.21.15.114:3300/0 4 ==== mgrmap(e 13) ==== 46615+0+0 (secure 0 0 0) 0x7f3c8001a400 con 0x7f3c9419aad0 2024-04-23T10:18:49.641 INFO:teuthology.orchestra.run.smithi204.stderr:2024-04-23T10:18:49.639+0000 7f3c91ffb700 1 --2- 172.21.15.204:0/3645080897 >> [v2:172.21.15.114:6800/465333808,v1:172.21.15.114:6801/465333808] conn(0x7f3c84039ae0 0x7f3c8403bfa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T10:18:49.641 INFO:teuthology.orchestra.run.smithi204.stderr:2024-04-23T10:18:49.639+0000 7f3c91ffb700 1 -- 172.21.15.204:0/3645080897 <== mon.0 v2:172.21.15.114:3300/0 5 ==== osd_map(4..4 src has 1..4) ==== 1155+0+0 (secure 0 0 0) 0x7f3c8004c5a0 con 0x7f3c9419aad0 2024-04-23T10:18:49.641 INFO:teuthology.orchestra.run.smithi204.stderr:2024-04-23T10:18:49.639+0000 7f3c9abba700 1 -- 172.21.15.204:0/3645080897 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f3c940640a0 con 0x7f3c9419aad0 2024-04-23T10:18:49.641 INFO:teuthology.orchestra.run.smithi204.stderr:2024-04-23T10:18:49.639+0000 7f3c98956700 1 --2- 172.21.15.204:0/3645080897 >> [v2:172.21.15.114:6800/465333808,v1:172.21.15.114:6801/465333808] conn(0x7f3c84039ae0 0x7f3c8403bfa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T10:18:49.641 INFO:teuthology.orchestra.run.smithi204.stderr:2024-04-23T10:18:49.639+0000 7f3c98956700 1 --2- 172.21.15.204:0/3645080897 >> [v2:172.21.15.114:6800/465333808,v1:172.21.15.114:6801/465333808] conn(0x7f3c84039ae0 0x7f3c8403bfa0 secure :-1 s=READY pgs=42 cs=0 l=1 rev1=1 crypto rx=0x7f3c88000c00 tx=0x7f3c88005fb0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T10:18:49.644 INFO:teuthology.orchestra.run.smithi204.stderr:2024-04-23T10:18:49.644+0000 7f3c91ffb700 1 -- 172.21.15.204:0/3645080897 <== mon.0 v2:172.21.15.114:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f3c80003be0 con 0x7f3c9419aad0 2024-04-23T10:18:49.852 INFO:teuthology.orchestra.run.smithi204.stderr:2024-04-23T10:18:49.852+0000 7f3c9abba700 1 -- 172.21.15.204:0/3645080897 --> [v2:172.21.15.114:6800/465333808,v1:172.21.15.114:6801/465333808] -- mgr_command(tid 0: {"prefix": "orch apply", "service_type": "mgr", "placement": "3;smithi114=x;smithi119=y;smithi204=z", "target": ["mon-mgr", ""]}) -- 0x7f3c94062e80 con 0x7f3c84039ae0 2024-04-23T10:18:49.858 INFO:teuthology.orchestra.run.smithi204.stderr:2024-04-23T10:18:49.859+0000 7f3c91ffb700 1 -- 172.21.15.204:0/3645080897 <== mgr.14152 v2:172.21.15.114:6800/465333808 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+24 (secure 0 0 0) 0x7f3c94062e80 con 0x7f3c84039ae0 2024-04-23T10:18:49.859 INFO:teuthology.orchestra.run.smithi204.stdout:Scheduled mgr update... 2024-04-23T10:18:49.861 INFO:teuthology.orchestra.run.smithi204.stderr:2024-04-23T10:18:49.861+0000 7f3c9abba700 1 -- 172.21.15.204:0/3645080897 >> [v2:172.21.15.114:6800/465333808,v1:172.21.15.114:6801/465333808] conn(0x7f3c84039ae0 msgr2=0x7f3c8403bfa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T10:18:49.861 INFO:teuthology.orchestra.run.smithi204.stderr:2024-04-23T10:18:49.861+0000 7f3c9abba700 1 --2- 172.21.15.204:0/3645080897 >> [v2:172.21.15.114:6800/465333808,v1:172.21.15.114:6801/465333808] conn(0x7f3c84039ae0 0x7f3c8403bfa0 secure :-1 s=READY pgs=42 cs=0 l=1 rev1=1 crypto rx=0x7f3c88000c00 tx=0x7f3c88005fb0 comp rx=0 tx=0).stop 2024-04-23T10:18:49.861 INFO:teuthology.orchestra.run.smithi204.stderr:2024-04-23T10:18:49.861+0000 7f3c9abba700 1 -- 172.21.15.204:0/3645080897 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f3c9419aad0 msgr2=0x7f3c9419cec0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T10:18:49.861 INFO:teuthology.orchestra.run.smithi204.stderr:2024-04-23T10:18:49.861+0000 7f3c9abba700 1 --2- 172.21.15.204:0/3645080897 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f3c9419aad0 0x7f3c9419cec0 secure :-1 s=READY pgs=109 cs=0 l=1 rev1=1 crypto rx=0x7f3c8000b780 tx=0x7f3c8000bb40 comp rx=0 tx=0).stop 2024-04-23T10:18:49.861 INFO:teuthology.orchestra.run.smithi204.stderr:2024-04-23T10:18:49.861+0000 7f3c9abba700 1 -- 172.21.15.204:0/3645080897 shutdown_connections 2024-04-23T10:18:49.861 INFO:teuthology.orchestra.run.smithi204.stderr:2024-04-23T10:18:49.861+0000 7f3c9abba700 1 --2- 172.21.15.204:0/3645080897 >> [v2:172.21.15.114:6800/465333808,v1:172.21.15.114:6801/465333808] conn(0x7f3c84039ae0 0x7f3c8403bfa0 unknown :-1 s=CLOSED pgs=42 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:18:49.861 INFO:teuthology.orchestra.run.smithi204.stderr:2024-04-23T10:18:49.861+0000 7f3c9abba700 1 --2- 172.21.15.204:0/3645080897 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f3c941014e0 0x7f3c9406b7b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:18:49.862 INFO:teuthology.orchestra.run.smithi204.stderr:2024-04-23T10:18:49.861+0000 7f3c9abba700 1 --2- 172.21.15.204:0/3645080897 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f3c9419d450 0x7f3c9419f8e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:18:49.862 INFO:teuthology.orchestra.run.smithi204.stderr:2024-04-23T10:18:49.861+0000 7f3c9abba700 1 --2- 172.21.15.204:0/3645080897 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f3c9419aad0 0x7f3c9419cec0 unknown :-1 s=CLOSED pgs=109 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:18:49.862 INFO:teuthology.orchestra.run.smithi204.stderr:2024-04-23T10:18:49.861+0000 7f3c9abba700 1 -- 172.21.15.204:0/3645080897 >> 172.21.15.204:0/3645080897 conn(0x7f3c940fca80 msgr2=0x7f3c940fd750 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T10:18:49.862 INFO:teuthology.orchestra.run.smithi204.stderr:2024-04-23T10:18:49.861+0000 7f3c9abba700 1 -- 172.21.15.204:0/3645080897 shutdown_connections 2024-04-23T10:18:49.862 INFO:teuthology.orchestra.run.smithi204.stderr:2024-04-23T10:18:49.861+0000 7f3c9abba700 1 -- 172.21.15.204:0/3645080897 wait complete. 2024-04-23T10:18:50.242 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 23 10:18:49 smithi119 ceph-mon[32608]: Reconfiguring mon.b (monmap changed)... 2024-04-23T10:18:50.242 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 23 10:18:49 smithi119 ceph-mon[32608]: Reconfiguring daemon mon.b on smithi119 2024-04-23T10:18:50.242 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 23 10:18:49 smithi119 ceph-mon[32608]: from='mgr.14152 172.21.15.114:0/1529706138' entity='mgr.x' 2024-04-23T10:18:50.254 INFO:journalctl@ceph.mon.a.smithi114.stdout:Apr 23 10:18:49 smithi114 ceph-mon[29255]: Reconfiguring mon.b (monmap changed)... 2024-04-23T10:18:50.254 INFO:journalctl@ceph.mon.a.smithi114.stdout:Apr 23 10:18:49 smithi114 ceph-mon[29255]: Reconfiguring daemon mon.b on smithi119 2024-04-23T10:18:50.254 INFO:journalctl@ceph.mon.a.smithi114.stdout:Apr 23 10:18:49 smithi114 ceph-mon[29255]: from='mgr.14152 172.21.15.114:0/1529706138' entity='mgr.x' 2024-04-23T10:18:50.328 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 23 10:18:49 smithi204 ceph-mon[32649]: Reconfiguring mon.b (monmap changed)... 2024-04-23T10:18:50.328 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 23 10:18:49 smithi204 ceph-mon[32649]: Reconfiguring daemon mon.b on smithi119 2024-04-23T10:18:50.328 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 23 10:18:49 smithi204 ceph-mon[32649]: from='mgr.14152 172.21.15.114:0/1529706138' entity='mgr.x' 2024-04-23T10:18:50.432 DEBUG:teuthology.orchestra.run.smithi119:mgr.y> sudo journalctl -f -n 0 -u ceph-aa240572-0159-11ef-bc93-c7b262605968@mgr.y.service 2024-04-23T10:18:50.435 DEBUG:teuthology.orchestra.run.smithi204:mgr.z> sudo journalctl -f -n 0 -u ceph-aa240572-0159-11ef-bc93-c7b262605968@mgr.z.service 2024-04-23T10:18:50.437 INFO:tasks.cephadm:Deploying OSDs... 2024-04-23T10:18:50.438 DEBUG:teuthology.orchestra.run.smithi114:> set -ex 2024-04-23T10:18:50.438 DEBUG:teuthology.orchestra.run.smithi114:> dd if=/scratch_devs of=/dev/stdout 2024-04-23T10:18:50.457 DEBUG:teuthology.misc:devs=['/dev/vg_nvme/lv_1', '/dev/vg_nvme/lv_2', '/dev/vg_nvme/lv_3', '/dev/vg_nvme/lv_4'] 2024-04-23T10:18:50.457 DEBUG:teuthology.orchestra.run.smithi114:> stat /dev/vg_nvme/lv_1 2024-04-23T10:18:50.514 INFO:teuthology.orchestra.run.smithi114.stdout: File: /dev/vg_nvme/lv_1 -> ../dm-0 2024-04-23T10:18:50.515 INFO:teuthology.orchestra.run.smithi114.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-04-23T10:18:50.515 INFO:teuthology.orchestra.run.smithi114.stdout:Device: 5h/5d Inode: 666 Links: 1 2024-04-23T10:18:50.515 INFO:teuthology.orchestra.run.smithi114.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-04-23T10:18:50.515 INFO:teuthology.orchestra.run.smithi114.stdout:Context: system_u:object_r:device_t:s0 2024-04-23T10:18:50.515 INFO:teuthology.orchestra.run.smithi114.stdout:Access: 2024-04-23 10:17:55.639013430 +0000 2024-04-23T10:18:50.515 INFO:teuthology.orchestra.run.smithi114.stdout:Modify: 2024-04-23 10:03:42.525971369 +0000 2024-04-23T10:18:50.515 INFO:teuthology.orchestra.run.smithi114.stdout:Change: 2024-04-23 10:03:42.525971369 +0000 2024-04-23T10:18:50.515 INFO:teuthology.orchestra.run.smithi114.stdout: Birth: 2024-04-23 10:03:42.525971369 +0000 2024-04-23T10:18:50.515 DEBUG:teuthology.orchestra.run.smithi114:> sudo dd if=/dev/vg_nvme/lv_1 of=/dev/null count=1 2024-04-23T10:18:50.580 INFO:teuthology.orchestra.run.smithi114.stderr:1+0 records in 2024-04-23T10:18:50.580 INFO:teuthology.orchestra.run.smithi114.stderr:1+0 records out 2024-04-23T10:18:50.581 INFO:teuthology.orchestra.run.smithi114.stderr:512 bytes copied, 0.000233872 s, 2.2 MB/s 2024-04-23T10:18:50.582 DEBUG:teuthology.orchestra.run.smithi114:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_1 2024-04-23T10:18:50.640 DEBUG:teuthology.orchestra.run.smithi114:> stat /dev/vg_nvme/lv_2 2024-04-23T10:18:50.696 INFO:teuthology.orchestra.run.smithi114.stdout: File: /dev/vg_nvme/lv_2 -> ../dm-1 2024-04-23T10:18:50.696 INFO:teuthology.orchestra.run.smithi114.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-04-23T10:18:50.696 INFO:teuthology.orchestra.run.smithi114.stdout:Device: 5h/5d Inode: 685 Links: 1 2024-04-23T10:18:50.696 INFO:teuthology.orchestra.run.smithi114.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-04-23T10:18:50.696 INFO:teuthology.orchestra.run.smithi114.stdout:Context: system_u:object_r:device_t:s0 2024-04-23T10:18:50.697 INFO:teuthology.orchestra.run.smithi114.stdout:Access: 2024-04-23 10:17:55.665012835 +0000 2024-04-23T10:18:50.697 INFO:teuthology.orchestra.run.smithi114.stdout:Modify: 2024-04-23 10:03:42.882963240 +0000 2024-04-23T10:18:50.697 INFO:teuthology.orchestra.run.smithi114.stdout:Change: 2024-04-23 10:03:42.882963240 +0000 2024-04-23T10:18:50.697 INFO:teuthology.orchestra.run.smithi114.stdout: Birth: 2024-04-23 10:03:42.882963240 +0000 2024-04-23T10:18:50.697 DEBUG:teuthology.orchestra.run.smithi114:> sudo dd if=/dev/vg_nvme/lv_2 of=/dev/null count=1 2024-04-23T10:18:50.761 INFO:teuthology.orchestra.run.smithi114.stderr:1+0 records in 2024-04-23T10:18:50.761 INFO:teuthology.orchestra.run.smithi114.stderr:1+0 records out 2024-04-23T10:18:50.761 INFO:teuthology.orchestra.run.smithi114.stderr:512 bytes copied, 0.000230542 s, 2.2 MB/s 2024-04-23T10:18:50.762 DEBUG:teuthology.orchestra.run.smithi114:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_2 2024-04-23T10:18:50.820 DEBUG:teuthology.orchestra.run.smithi114:> stat /dev/vg_nvme/lv_3 2024-04-23T10:18:50.876 INFO:teuthology.orchestra.run.smithi114.stdout: File: /dev/vg_nvme/lv_3 -> ../dm-2 2024-04-23T10:18:50.877 INFO:teuthology.orchestra.run.smithi114.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-04-23T10:18:50.877 INFO:teuthology.orchestra.run.smithi114.stdout:Device: 5h/5d Inode: 703 Links: 1 2024-04-23T10:18:50.877 INFO:teuthology.orchestra.run.smithi114.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-04-23T10:18:50.877 INFO:teuthology.orchestra.run.smithi114.stdout:Context: system_u:object_r:device_t:s0 2024-04-23T10:18:50.877 INFO:teuthology.orchestra.run.smithi114.stdout:Access: 2024-04-23 10:17:55.689012285 +0000 2024-04-23T10:18:50.877 INFO:teuthology.orchestra.run.smithi114.stdout:Modify: 2024-04-23 10:03:43.245954975 +0000 2024-04-23T10:18:50.877 INFO:teuthology.orchestra.run.smithi114.stdout:Change: 2024-04-23 10:03:43.245954975 +0000 2024-04-23T10:18:50.877 INFO:teuthology.orchestra.run.smithi114.stdout: Birth: 2024-04-23 10:03:43.245954975 +0000 2024-04-23T10:18:50.877 DEBUG:teuthology.orchestra.run.smithi114:> sudo dd if=/dev/vg_nvme/lv_3 of=/dev/null count=1 2024-04-23T10:18:50.941 INFO:teuthology.orchestra.run.smithi114.stderr:1+0 records in 2024-04-23T10:18:50.941 INFO:teuthology.orchestra.run.smithi114.stderr:1+0 records out 2024-04-23T10:18:50.941 INFO:teuthology.orchestra.run.smithi114.stderr:512 bytes copied, 0.000202603 s, 2.5 MB/s 2024-04-23T10:18:50.942 DEBUG:teuthology.orchestra.run.smithi114:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_3 2024-04-23T10:18:50.998 DEBUG:teuthology.orchestra.run.smithi114:> stat /dev/vg_nvme/lv_4 2024-04-23T10:18:51.054 INFO:teuthology.orchestra.run.smithi114.stdout: File: /dev/vg_nvme/lv_4 -> ../dm-3 2024-04-23T10:18:51.054 INFO:teuthology.orchestra.run.smithi114.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-04-23T10:18:51.054 INFO:teuthology.orchestra.run.smithi114.stdout:Device: 5h/5d Inode: 722 Links: 1 2024-04-23T10:18:51.054 INFO:teuthology.orchestra.run.smithi114.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-04-23T10:18:51.054 INFO:teuthology.orchestra.run.smithi114.stdout:Context: system_u:object_r:device_t:s0 2024-04-23T10:18:51.054 INFO:teuthology.orchestra.run.smithi114.stdout:Access: 2024-04-23 10:17:55.715011690 +0000 2024-04-23T10:18:51.054 INFO:teuthology.orchestra.run.smithi114.stdout:Modify: 2024-04-23 10:03:43.608946709 +0000 2024-04-23T10:18:51.055 INFO:teuthology.orchestra.run.smithi114.stdout:Change: 2024-04-23 10:03:43.608946709 +0000 2024-04-23T10:18:51.055 INFO:teuthology.orchestra.run.smithi114.stdout: Birth: 2024-04-23 10:03:43.608946709 +0000 2024-04-23T10:18:51.055 DEBUG:teuthology.orchestra.run.smithi114:> sudo dd if=/dev/vg_nvme/lv_4 of=/dev/null count=1 2024-04-23T10:18:51.118 INFO:teuthology.orchestra.run.smithi114.stderr:1+0 records in 2024-04-23T10:18:51.118 INFO:teuthology.orchestra.run.smithi114.stderr:1+0 records out 2024-04-23T10:18:51.119 INFO:teuthology.orchestra.run.smithi114.stderr:512 bytes copied, 0.000199345 s, 2.6 MB/s 2024-04-23T10:18:51.120 DEBUG:teuthology.orchestra.run.smithi114:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_4 2024-04-23T10:18:51.176 DEBUG:teuthology.orchestra.run.smithi119:> set -ex 2024-04-23T10:18:51.176 DEBUG:teuthology.orchestra.run.smithi119:> dd if=/scratch_devs of=/dev/stdout 2024-04-23T10:18:51.194 DEBUG:teuthology.misc:devs=['/dev/vg_nvme/lv_1', '/dev/vg_nvme/lv_2', '/dev/vg_nvme/lv_3', '/dev/vg_nvme/lv_4'] 2024-04-23T10:18:51.195 DEBUG:teuthology.orchestra.run.smithi119:> stat /dev/vg_nvme/lv_1 2024-04-23T10:18:51.224 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 23 10:18:50 smithi204 ceph-mon[32649]: from='client.14208 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "mgr", "placement": "3;smithi114=x;smithi119=y;smithi204=z", "target": ["mon-mgr", ""]}]: dispatch 2024-04-23T10:18:51.224 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 23 10:18:50 smithi204 ceph-mon[32649]: Saving service mgr spec with placement smithi114=x;smithi119=y;smithi204=z;count:3 2024-04-23T10:18:51.224 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 23 10:18:50 smithi204 ceph-mon[32649]: pgmap v26: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-23T10:18:51.224 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 23 10:18:50 smithi204 ceph-mon[32649]: from='mgr.14152 172.21.15.114:0/1529706138' entity='mgr.x' 2024-04-23T10:18:51.224 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 23 10:18:50 smithi204 ceph-mon[32649]: from='mgr.14152 172.21.15.114:0/1529706138' entity='mgr.x' 2024-04-23T10:18:51.225 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 23 10:18:50 smithi204 ceph-mon[32649]: from='mgr.14152 172.21.15.114:0/1529706138' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-04-23T10:18:51.225 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 23 10:18:50 smithi204 ceph-mon[32649]: from='mgr.14152 172.21.15.114:0/1529706138' entity='mgr.x' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-04-23T10:18:51.225 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 23 10:18:50 smithi204 ceph-mon[32649]: from='mgr.14152 172.21.15.114:0/1529706138' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-23T10:18:51.242 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 23 10:18:50 smithi119 ceph-mon[32608]: from='client.14208 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "mgr", "placement": "3;smithi114=x;smithi119=y;smithi204=z", "target": ["mon-mgr", ""]}]: dispatch 2024-04-23T10:18:51.242 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 23 10:18:50 smithi119 ceph-mon[32608]: Saving service mgr spec with placement smithi114=x;smithi119=y;smithi204=z;count:3 2024-04-23T10:18:51.242 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 23 10:18:50 smithi119 ceph-mon[32608]: pgmap v26: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-23T10:18:51.242 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 23 10:18:50 smithi119 ceph-mon[32608]: from='mgr.14152 172.21.15.114:0/1529706138' entity='mgr.x' 2024-04-23T10:18:51.242 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 23 10:18:50 smithi119 ceph-mon[32608]: from='mgr.14152 172.21.15.114:0/1529706138' entity='mgr.x' 2024-04-23T10:18:51.242 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 23 10:18:50 smithi119 ceph-mon[32608]: from='mgr.14152 172.21.15.114:0/1529706138' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-04-23T10:18:51.243 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 23 10:18:50 smithi119 ceph-mon[32608]: from='mgr.14152 172.21.15.114:0/1529706138' entity='mgr.x' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-04-23T10:18:51.243 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 23 10:18:50 smithi119 ceph-mon[32608]: from='mgr.14152 172.21.15.114:0/1529706138' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-23T10:18:51.251 INFO:teuthology.orchestra.run.smithi119.stdout: File: /dev/vg_nvme/lv_1 -> ../dm-0 2024-04-23T10:18:51.251 INFO:teuthology.orchestra.run.smithi119.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-04-23T10:18:51.252 INFO:teuthology.orchestra.run.smithi119.stdout:Device: 5h/5d Inode: 666 Links: 1 2024-04-23T10:18:51.252 INFO:teuthology.orchestra.run.smithi119.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-04-23T10:18:51.252 INFO:teuthology.orchestra.run.smithi119.stdout:Context: system_u:object_r:device_t:s0 2024-04-23T10:18:51.252 INFO:teuthology.orchestra.run.smithi119.stdout:Access: 2024-04-23 10:18:38.623612408 +0000 2024-04-23T10:18:51.252 INFO:teuthology.orchestra.run.smithi119.stdout:Modify: 2024-04-23 10:04:19.968218869 +0000 2024-04-23T10:18:51.252 INFO:teuthology.orchestra.run.smithi119.stdout:Change: 2024-04-23 10:04:19.968218869 +0000 2024-04-23T10:18:51.252 INFO:teuthology.orchestra.run.smithi119.stdout: Birth: 2024-04-23 10:04:19.968218869 +0000 2024-04-23T10:18:51.252 DEBUG:teuthology.orchestra.run.smithi119:> sudo dd if=/dev/vg_nvme/lv_1 of=/dev/null count=1 2024-04-23T10:18:51.254 INFO:journalctl@ceph.mon.a.smithi114.stdout:Apr 23 10:18:50 smithi114 ceph-mon[29255]: from='client.14208 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "mgr", "placement": "3;smithi114=x;smithi119=y;smithi204=z", "target": ["mon-mgr", ""]}]: dispatch 2024-04-23T10:18:51.254 INFO:journalctl@ceph.mon.a.smithi114.stdout:Apr 23 10:18:50 smithi114 ceph-mon[29255]: Saving service mgr spec with placement smithi114=x;smithi119=y;smithi204=z;count:3 2024-04-23T10:18:51.254 INFO:journalctl@ceph.mon.a.smithi114.stdout:Apr 23 10:18:50 smithi114 ceph-mon[29255]: pgmap v26: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-23T10:18:51.254 INFO:journalctl@ceph.mon.a.smithi114.stdout:Apr 23 10:18:50 smithi114 ceph-mon[29255]: from='mgr.14152 172.21.15.114:0/1529706138' entity='mgr.x' 2024-04-23T10:18:51.254 INFO:journalctl@ceph.mon.a.smithi114.stdout:Apr 23 10:18:50 smithi114 ceph-mon[29255]: from='mgr.14152 172.21.15.114:0/1529706138' entity='mgr.x' 2024-04-23T10:18:51.255 INFO:journalctl@ceph.mon.a.smithi114.stdout:Apr 23 10:18:50 smithi114 ceph-mon[29255]: from='mgr.14152 172.21.15.114:0/1529706138' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-04-23T10:18:51.255 INFO:journalctl@ceph.mon.a.smithi114.stdout:Apr 23 10:18:50 smithi114 ceph-mon[29255]: from='mgr.14152 172.21.15.114:0/1529706138' entity='mgr.x' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-04-23T10:18:51.255 INFO:journalctl@ceph.mon.a.smithi114.stdout:Apr 23 10:18:50 smithi114 ceph-mon[29255]: from='mgr.14152 172.21.15.114:0/1529706138' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-23T10:18:51.317 INFO:teuthology.orchestra.run.smithi119.stderr:1+0 records in 2024-04-23T10:18:51.317 INFO:teuthology.orchestra.run.smithi119.stderr:1+0 records out 2024-04-23T10:18:51.317 INFO:teuthology.orchestra.run.smithi119.stderr:512 bytes copied, 0.000265783 s, 1.9 MB/s 2024-04-23T10:18:51.319 DEBUG:teuthology.orchestra.run.smithi119:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_1 2024-04-23T10:18:51.377 DEBUG:teuthology.orchestra.run.smithi119:> stat /dev/vg_nvme/lv_2 2024-04-23T10:18:51.435 INFO:teuthology.orchestra.run.smithi119.stdout: File: /dev/vg_nvme/lv_2 -> ../dm-1 2024-04-23T10:18:51.435 INFO:teuthology.orchestra.run.smithi119.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-04-23T10:18:51.435 INFO:teuthology.orchestra.run.smithi119.stdout:Device: 5h/5d Inode: 685 Links: 1 2024-04-23T10:18:51.435 INFO:teuthology.orchestra.run.smithi119.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-04-23T10:18:51.435 INFO:teuthology.orchestra.run.smithi119.stdout:Context: system_u:object_r:device_t:s0 2024-04-23T10:18:51.435 INFO:teuthology.orchestra.run.smithi119.stdout:Access: 2024-04-23 10:18:38.623612408 +0000 2024-04-23T10:18:51.435 INFO:teuthology.orchestra.run.smithi119.stdout:Modify: 2024-04-23 10:04:20.345209541 +0000 2024-04-23T10:18:51.435 INFO:teuthology.orchestra.run.smithi119.stdout:Change: 2024-04-23 10:04:20.345209541 +0000 2024-04-23T10:18:51.435 INFO:teuthology.orchestra.run.smithi119.stdout: Birth: 2024-04-23 10:04:20.345209541 +0000 2024-04-23T10:18:51.436 DEBUG:teuthology.orchestra.run.smithi119:> sudo dd if=/dev/vg_nvme/lv_2 of=/dev/null count=1 2024-04-23T10:18:51.500 INFO:teuthology.orchestra.run.smithi119.stderr:1+0 records in 2024-04-23T10:18:51.501 INFO:teuthology.orchestra.run.smithi119.stderr:1+0 records out 2024-04-23T10:18:51.501 INFO:teuthology.orchestra.run.smithi119.stderr:512 bytes copied, 0.00021334 s, 2.4 MB/s 2024-04-23T10:18:51.503 DEBUG:teuthology.orchestra.run.smithi119:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_2 2024-04-23T10:18:51.560 DEBUG:teuthology.orchestra.run.smithi119:> stat /dev/vg_nvme/lv_3 2024-04-23T10:18:51.617 INFO:teuthology.orchestra.run.smithi119.stdout: File: /dev/vg_nvme/lv_3 -> ../dm-2 2024-04-23T10:18:51.617 INFO:teuthology.orchestra.run.smithi119.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-04-23T10:18:51.617 INFO:teuthology.orchestra.run.smithi119.stdout:Device: 5h/5d Inode: 705 Links: 1 2024-04-23T10:18:51.617 INFO:teuthology.orchestra.run.smithi119.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-04-23T10:18:51.617 INFO:teuthology.orchestra.run.smithi119.stdout:Context: system_u:object_r:device_t:s0 2024-04-23T10:18:51.617 INFO:teuthology.orchestra.run.smithi119.stdout:Access: 2024-04-23 10:18:38.623612408 +0000 2024-04-23T10:18:51.617 INFO:teuthology.orchestra.run.smithi119.stdout:Modify: 2024-04-23 10:04:20.696200855 +0000 2024-04-23T10:18:51.617 INFO:teuthology.orchestra.run.smithi119.stdout:Change: 2024-04-23 10:04:20.696200855 +0000 2024-04-23T10:18:51.617 INFO:teuthology.orchestra.run.smithi119.stdout: Birth: 2024-04-23 10:04:20.696200855 +0000 2024-04-23T10:18:51.618 DEBUG:teuthology.orchestra.run.smithi119:> sudo dd if=/dev/vg_nvme/lv_3 of=/dev/null count=1 2024-04-23T10:18:51.681 INFO:teuthology.orchestra.run.smithi119.stderr:1+0 records in 2024-04-23T10:18:51.682 INFO:teuthology.orchestra.run.smithi119.stderr:1+0 records out 2024-04-23T10:18:51.682 INFO:teuthology.orchestra.run.smithi119.stderr:512 bytes copied, 0.000246043 s, 2.1 MB/s 2024-04-23T10:18:51.683 DEBUG:teuthology.orchestra.run.smithi119:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_3 2024-04-23T10:18:51.740 DEBUG:teuthology.orchestra.run.smithi119:> stat /dev/vg_nvme/lv_4 2024-04-23T10:18:51.796 INFO:teuthology.orchestra.run.smithi119.stdout: File: /dev/vg_nvme/lv_4 -> ../dm-3 2024-04-23T10:18:51.796 INFO:teuthology.orchestra.run.smithi119.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-04-23T10:18:51.796 INFO:teuthology.orchestra.run.smithi119.stdout:Device: 5h/5d Inode: 723 Links: 1 2024-04-23T10:18:51.796 INFO:teuthology.orchestra.run.smithi119.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-04-23T10:18:51.797 INFO:teuthology.orchestra.run.smithi119.stdout:Context: system_u:object_r:device_t:s0 2024-04-23T10:18:51.797 INFO:teuthology.orchestra.run.smithi119.stdout:Access: 2024-04-23 10:18:38.624612383 +0000 2024-04-23T10:18:51.797 INFO:teuthology.orchestra.run.smithi119.stdout:Modify: 2024-04-23 10:04:21.040192342 +0000 2024-04-23T10:18:51.797 INFO:teuthology.orchestra.run.smithi119.stdout:Change: 2024-04-23 10:04:21.040192342 +0000 2024-04-23T10:18:51.797 INFO:teuthology.orchestra.run.smithi119.stdout: Birth: 2024-04-23 10:04:21.040192342 +0000 2024-04-23T10:18:51.797 DEBUG:teuthology.orchestra.run.smithi119:> sudo dd if=/dev/vg_nvme/lv_4 of=/dev/null count=1 2024-04-23T10:18:51.860 INFO:teuthology.orchestra.run.smithi119.stderr:1+0 records in 2024-04-23T10:18:51.860 INFO:teuthology.orchestra.run.smithi119.stderr:1+0 records out 2024-04-23T10:18:51.860 INFO:teuthology.orchestra.run.smithi119.stderr:512 bytes copied, 0.000201849 s, 2.5 MB/s 2024-04-23T10:18:51.862 DEBUG:teuthology.orchestra.run.smithi119:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_4 2024-04-23T10:18:51.919 DEBUG:teuthology.orchestra.run.smithi204:> set -ex 2024-04-23T10:18:51.919 DEBUG:teuthology.orchestra.run.smithi204:> dd if=/scratch_devs of=/dev/stdout 2024-04-23T10:18:51.935 DEBUG:teuthology.misc:devs=['/dev/vg_nvme/lv_1', '/dev/vg_nvme/lv_2', '/dev/vg_nvme/lv_3', '/dev/vg_nvme/lv_4'] 2024-04-23T10:18:51.935 DEBUG:teuthology.orchestra.run.smithi204:> stat /dev/vg_nvme/lv_1 2024-04-23T10:18:51.990 INFO:teuthology.orchestra.run.smithi204.stdout: File: /dev/vg_nvme/lv_1 -> ../dm-0 2024-04-23T10:18:51.991 INFO:teuthology.orchestra.run.smithi204.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-04-23T10:18:51.991 INFO:teuthology.orchestra.run.smithi204.stdout:Device: 5h/5d Inode: 667 Links: 1 2024-04-23T10:18:51.991 INFO:teuthology.orchestra.run.smithi204.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-04-23T10:18:51.991 INFO:teuthology.orchestra.run.smithi204.stdout:Context: system_u:object_r:device_t:s0 2024-04-23T10:18:51.991 INFO:teuthology.orchestra.run.smithi204.stdout:Access: 2024-04-23 10:18:29.151279160 +0000 2024-04-23T10:18:51.991 INFO:teuthology.orchestra.run.smithi204.stdout:Modify: 2024-04-23 10:04:08.070527537 +0000 2024-04-23T10:18:51.991 INFO:teuthology.orchestra.run.smithi204.stdout:Change: 2024-04-23 10:04:08.070527537 +0000 2024-04-23T10:18:51.991 INFO:teuthology.orchestra.run.smithi204.stdout: Birth: 2024-04-23 10:04:08.070527537 +0000 2024-04-23T10:18:51.991 DEBUG:teuthology.orchestra.run.smithi204:> sudo dd if=/dev/vg_nvme/lv_1 of=/dev/null count=1 2024-04-23T10:18:52.054 INFO:teuthology.orchestra.run.smithi204.stderr:1+0 records in 2024-04-23T10:18:52.054 INFO:teuthology.orchestra.run.smithi204.stderr:1+0 records out 2024-04-23T10:18:52.054 INFO:teuthology.orchestra.run.smithi204.stderr:512 bytes copied, 0.000271225 s, 1.9 MB/s 2024-04-23T10:18:52.055 DEBUG:teuthology.orchestra.run.smithi204:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_1 2024-04-23T10:18:52.111 DEBUG:teuthology.orchestra.run.smithi204:> stat /dev/vg_nvme/lv_2 2024-04-23T10:18:52.165 INFO:teuthology.orchestra.run.smithi204.stdout: File: /dev/vg_nvme/lv_2 -> ../dm-1 2024-04-23T10:18:52.165 INFO:teuthology.orchestra.run.smithi204.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-04-23T10:18:52.165 INFO:teuthology.orchestra.run.smithi204.stdout:Device: 5h/5d Inode: 684 Links: 1 2024-04-23T10:18:52.165 INFO:teuthology.orchestra.run.smithi204.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-04-23T10:18:52.165 INFO:teuthology.orchestra.run.smithi204.stdout:Context: system_u:object_r:device_t:s0 2024-04-23T10:18:52.165 INFO:teuthology.orchestra.run.smithi204.stdout:Access: 2024-04-23 10:18:29.152279146 +0000 2024-04-23T10:18:52.165 INFO:teuthology.orchestra.run.smithi204.stdout:Modify: 2024-04-23 10:04:08.412522791 +0000 2024-04-23T10:18:52.166 INFO:teuthology.orchestra.run.smithi204.stdout:Change: 2024-04-23 10:04:08.412522791 +0000 2024-04-23T10:18:52.166 INFO:teuthology.orchestra.run.smithi204.stdout: Birth: 2024-04-23 10:04:08.412522791 +0000 2024-04-23T10:18:52.166 DEBUG:teuthology.orchestra.run.smithi204:> sudo dd if=/dev/vg_nvme/lv_2 of=/dev/null count=1 2024-04-23T10:18:52.225 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 23 10:18:51 smithi204 ceph-mon[32649]: Reconfiguring mon.c (monmap changed)... 2024-04-23T10:18:52.225 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 23 10:18:51 smithi204 ceph-mon[32649]: Reconfiguring daemon mon.c on smithi204 2024-04-23T10:18:52.228 INFO:teuthology.orchestra.run.smithi204.stderr:1+0 records in 2024-04-23T10:18:52.228 INFO:teuthology.orchestra.run.smithi204.stderr:1+0 records out 2024-04-23T10:18:52.228 INFO:teuthology.orchestra.run.smithi204.stderr:512 bytes copied, 0.000274249 s, 1.9 MB/s 2024-04-23T10:18:52.229 DEBUG:teuthology.orchestra.run.smithi204:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_2 2024-04-23T10:18:52.242 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 23 10:18:51 smithi119 ceph-mon[32608]: Reconfiguring mon.c (monmap changed)... 2024-04-23T10:18:52.242 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 23 10:18:51 smithi119 ceph-mon[32608]: Reconfiguring daemon mon.c on smithi204 2024-04-23T10:18:52.254 INFO:journalctl@ceph.mon.a.smithi114.stdout:Apr 23 10:18:51 smithi114 ceph-mon[29255]: Reconfiguring mon.c (monmap changed)... 2024-04-23T10:18:52.254 INFO:journalctl@ceph.mon.a.smithi114.stdout:Apr 23 10:18:51 smithi114 ceph-mon[29255]: Reconfiguring daemon mon.c on smithi204 2024-04-23T10:18:52.285 DEBUG:teuthology.orchestra.run.smithi204:> stat /dev/vg_nvme/lv_3 2024-04-23T10:18:52.341 INFO:teuthology.orchestra.run.smithi204.stdout: File: /dev/vg_nvme/lv_3 -> ../dm-2 2024-04-23T10:18:52.341 INFO:teuthology.orchestra.run.smithi204.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-04-23T10:18:52.341 INFO:teuthology.orchestra.run.smithi204.stdout:Device: 5h/5d Inode: 702 Links: 1 2024-04-23T10:18:52.341 INFO:teuthology.orchestra.run.smithi204.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-04-23T10:18:52.341 INFO:teuthology.orchestra.run.smithi204.stdout:Context: system_u:object_r:device_t:s0 2024-04-23T10:18:52.341 INFO:teuthology.orchestra.run.smithi204.stdout:Access: 2024-04-23 10:18:29.152279146 +0000 2024-04-23T10:18:52.342 INFO:teuthology.orchestra.run.smithi204.stdout:Modify: 2024-04-23 10:04:08.769517837 +0000 2024-04-23T10:18:52.342 INFO:teuthology.orchestra.run.smithi204.stdout:Change: 2024-04-23 10:04:08.769517837 +0000 2024-04-23T10:18:52.342 INFO:teuthology.orchestra.run.smithi204.stdout: Birth: 2024-04-23 10:04:08.769517837 +0000 2024-04-23T10:18:52.342 DEBUG:teuthology.orchestra.run.smithi204:> sudo dd if=/dev/vg_nvme/lv_3 of=/dev/null count=1 2024-04-23T10:18:52.402 INFO:teuthology.orchestra.run.smithi204.stderr:1+0 records in 2024-04-23T10:18:52.402 INFO:teuthology.orchestra.run.smithi204.stderr:1+0 records out 2024-04-23T10:18:52.402 INFO:teuthology.orchestra.run.smithi204.stderr:512 bytes copied, 0.000224178 s, 2.3 MB/s 2024-04-23T10:18:52.403 DEBUG:teuthology.orchestra.run.smithi204:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_3 2024-04-23T10:18:52.460 DEBUG:teuthology.orchestra.run.smithi204:> stat /dev/vg_nvme/lv_4 2024-04-23T10:18:52.514 INFO:teuthology.orchestra.run.smithi204.stdout: File: /dev/vg_nvme/lv_4 -> ../dm-3 2024-04-23T10:18:52.514 INFO:teuthology.orchestra.run.smithi204.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-04-23T10:18:52.514 INFO:teuthology.orchestra.run.smithi204.stdout:Device: 5h/5d Inode: 724 Links: 1 2024-04-23T10:18:52.514 INFO:teuthology.orchestra.run.smithi204.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-04-23T10:18:52.514 INFO:teuthology.orchestra.run.smithi204.stdout:Context: system_u:object_r:device_t:s0 2024-04-23T10:18:52.514 INFO:teuthology.orchestra.run.smithi204.stdout:Access: 2024-04-23 10:18:29.152279146 +0000 2024-04-23T10:18:52.514 INFO:teuthology.orchestra.run.smithi204.stdout:Modify: 2024-04-23 10:04:09.180512134 +0000 2024-04-23T10:18:52.514 INFO:teuthology.orchestra.run.smithi204.stdout:Change: 2024-04-23 10:04:09.180512134 +0000 2024-04-23T10:18:52.514 INFO:teuthology.orchestra.run.smithi204.stdout: Birth: 2024-04-23 10:04:09.180512134 +0000 2024-04-23T10:18:52.515 DEBUG:teuthology.orchestra.run.smithi204:> sudo dd if=/dev/vg_nvme/lv_4 of=/dev/null count=1 2024-04-23T10:18:52.576 INFO:teuthology.orchestra.run.smithi204.stderr:1+0 records in 2024-04-23T10:18:52.576 INFO:teuthology.orchestra.run.smithi204.stderr:1+0 records out 2024-04-23T10:18:52.576 INFO:teuthology.orchestra.run.smithi204.stderr:512 bytes copied, 0.000214739 s, 2.4 MB/s 2024-04-23T10:18:52.577 DEBUG:teuthology.orchestra.run.smithi204:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_4 2024-04-23T10:18:52.632 INFO:tasks.cephadm:Deploying osd.0 on smithi114 with /dev/vg_nvme/lv_4... 2024-04-23T10:18:52.632 DEBUG:teuthology.orchestra.run.smithi114:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d69f13d827952bb07bb9b7973bca14f97d764d6f shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid aa240572-0159-11ef-bc93-c7b262605968 -- ceph-volume lvm zap /dev/vg_nvme/lv_4 2024-04-23T10:18:52.861 INFO:teuthology.orchestra.run.smithi114.stderr:Inferring config /var/lib/ceph/aa240572-0159-11ef-bc93-c7b262605968/mon.a/config 2024-04-23T10:18:53.242 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 23 10:18:52 smithi119 ceph-mon[32608]: pgmap v27: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-23T10:18:53.242 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 23 10:18:52 smithi119 ceph-mon[32608]: from='mgr.14152 172.21.15.114:0/1529706138' entity='mgr.x' 2024-04-23T10:18:53.242 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 23 10:18:52 smithi119 ceph-mon[32608]: from='mgr.14152 172.21.15.114:0/1529706138' entity='mgr.x' 2024-04-23T10:18:53.242 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 23 10:18:52 smithi119 ceph-mon[32608]: from='mgr.14152 172.21.15.114:0/1529706138' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-23T10:18:53.242 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 23 10:18:52 smithi119 ceph-mon[32608]: from='mgr.14152 172.21.15.114:0/1529706138' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-23T10:18:53.243 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 23 10:18:52 smithi119 ceph-mon[32608]: from='mgr.14152 172.21.15.114:0/1529706138' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-23T10:18:53.243 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 23 10:18:52 smithi119 ceph-mon[32608]: from='mgr.14152 172.21.15.114:0/1529706138' entity='mgr.x' 2024-04-23T10:18:53.243 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 23 10:18:52 smithi119 ceph-mon[32608]: from='mgr.14152 172.21.15.114:0/1529706138' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.z", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-04-23T10:18:53.243 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 23 10:18:52 smithi119 ceph-mon[32608]: from='mgr.14152 172.21.15.114:0/1529706138' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mgr.z", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]': finished 2024-04-23T10:18:53.243 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 23 10:18:52 smithi119 ceph-mon[32608]: from='mgr.14152 172.21.15.114:0/1529706138' entity='mgr.x' cmd=[{"prefix": "mgr services"}]: dispatch 2024-04-23T10:18:53.243 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 23 10:18:52 smithi119 ceph-mon[32608]: from='mgr.14152 172.21.15.114:0/1529706138' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-23T10:18:53.254 INFO:journalctl@ceph.mon.a.smithi114.stdout:Apr 23 10:18:52 smithi114 ceph-mon[29255]: pgmap v27: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-23T10:18:53.254 INFO:journalctl@ceph.mon.a.smithi114.stdout:Apr 23 10:18:52 smithi114 ceph-mon[29255]: from='mgr.14152 172.21.15.114:0/1529706138' entity='mgr.x' 2024-04-23T10:18:53.254 INFO:journalctl@ceph.mon.a.smithi114.stdout:Apr 23 10:18:52 smithi114 ceph-mon[29255]: from='mgr.14152 172.21.15.114:0/1529706138' entity='mgr.x' 2024-04-23T10:18:53.254 INFO:journalctl@ceph.mon.a.smithi114.stdout:Apr 23 10:18:52 smithi114 ceph-mon[29255]: from='mgr.14152 172.21.15.114:0/1529706138' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-23T10:18:53.254 INFO:journalctl@ceph.mon.a.smithi114.stdout:Apr 23 10:18:52 smithi114 ceph-mon[29255]: from='mgr.14152 172.21.15.114:0/1529706138' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-23T10:18:53.254 INFO:journalctl@ceph.mon.a.smithi114.stdout:Apr 23 10:18:52 smithi114 ceph-mon[29255]: from='mgr.14152 172.21.15.114:0/1529706138' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-23T10:18:53.254 INFO:journalctl@ceph.mon.a.smithi114.stdout:Apr 23 10:18:52 smithi114 ceph-mon[29255]: from='mgr.14152 172.21.15.114:0/1529706138' entity='mgr.x' 2024-04-23T10:18:53.255 INFO:journalctl@ceph.mon.a.smithi114.stdout:Apr 23 10:18:52 smithi114 ceph-mon[29255]: from='mgr.14152 172.21.15.114:0/1529706138' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.z", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-04-23T10:18:53.255 INFO:journalctl@ceph.mon.a.smithi114.stdout:Apr 23 10:18:52 smithi114 ceph-mon[29255]: from='mgr.14152 172.21.15.114:0/1529706138' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mgr.z", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]': finished 2024-04-23T10:18:53.255 INFO:journalctl@ceph.mon.a.smithi114.stdout:Apr 23 10:18:52 smithi114 ceph-mon[29255]: from='mgr.14152 172.21.15.114:0/1529706138' entity='mgr.x' cmd=[{"prefix": "mgr services"}]: dispatch 2024-04-23T10:18:53.255 INFO:journalctl@ceph.mon.a.smithi114.stdout:Apr 23 10:18:52 smithi114 ceph-mon[29255]: from='mgr.14152 172.21.15.114:0/1529706138' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-23T10:18:53.293 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 23 10:18:52 smithi204 ceph-mon[32649]: pgmap v27: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-23T10:18:53.293 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 23 10:18:52 smithi204 ceph-mon[32649]: from='mgr.14152 172.21.15.114:0/1529706138' entity='mgr.x' 2024-04-23T10:18:53.293 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 23 10:18:52 smithi204 ceph-mon[32649]: from='mgr.14152 172.21.15.114:0/1529706138' entity='mgr.x' 2024-04-23T10:18:53.293 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 23 10:18:52 smithi204 ceph-mon[32649]: from='mgr.14152 172.21.15.114:0/1529706138' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-23T10:18:53.293 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 23 10:18:52 smithi204 ceph-mon[32649]: from='mgr.14152 172.21.15.114:0/1529706138' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-23T10:18:53.293 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 23 10:18:52 smithi204 ceph-mon[32649]: from='mgr.14152 172.21.15.114:0/1529706138' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-23T10:18:53.293 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 23 10:18:52 smithi204 ceph-mon[32649]: from='mgr.14152 172.21.15.114:0/1529706138' entity='mgr.x' 2024-04-23T10:18:53.293 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 23 10:18:52 smithi204 ceph-mon[32649]: from='mgr.14152 172.21.15.114:0/1529706138' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.z", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-04-23T10:18:53.293 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 23 10:18:52 smithi204 ceph-mon[32649]: from='mgr.14152 172.21.15.114:0/1529706138' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mgr.z", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]': finished 2024-04-23T10:18:53.293 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 23 10:18:52 smithi204 ceph-mon[32649]: from='mgr.14152 172.21.15.114:0/1529706138' entity='mgr.x' cmd=[{"prefix": "mgr services"}]: dispatch 2024-04-23T10:18:53.293 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 23 10:18:52 smithi204 ceph-mon[32649]: from='mgr.14152 172.21.15.114:0/1529706138' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-23T10:18:54.203 INFO:journalctl@ceph.mon.a.smithi114.stdout:Apr 23 10:18:53 smithi114 ceph-mon[29255]: Deploying daemon mgr.z on smithi204 2024-04-23T10:18:54.242 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 23 10:18:53 smithi119 ceph-mon[32608]: Deploying daemon mgr.z on smithi204 2024-04-23T10:18:54.313 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 23 10:18:53 smithi204 ceph-mon[32649]: Deploying daemon mgr.z on smithi204 2024-04-23T10:18:54.578 INFO:journalctl@ceph.mgr.z.smithi204.stdout:Apr 23 10:18:54 smithi204 systemd[1]: Starting Ceph mgr.z for aa240572-0159-11ef-bc93-c7b262605968... 2024-04-23T10:18:54.764 INFO:teuthology.orchestra.run.smithi114.stderr:--> Zapping: /dev/vg_nvme/lv_4 2024-04-23T10:18:54.765 INFO:teuthology.orchestra.run.smithi114.stderr:Running command: /usr/bin/dd if=/dev/zero of=/dev/vg_nvme/lv_4 bs=1M count=10 conv=fsync 2024-04-23T10:18:54.765 INFO:teuthology.orchestra.run.smithi114.stderr: stderr: 10+0 records in 2024-04-23T10:18:54.765 INFO:teuthology.orchestra.run.smithi114.stderr:10+0 records out 2024-04-23T10:18:54.765 INFO:teuthology.orchestra.run.smithi114.stderr: stderr: 10485760 bytes (10 MB, 10 MiB) copied, 0.0123041 s, 852 MB/s 2024-04-23T10:18:54.765 INFO:teuthology.orchestra.run.smithi114.stderr:--> Zapping successful for: 2024-04-23T10:18:55.014 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 23 10:18:54 smithi204 ceph-mon[32649]: pgmap v28: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-23T10:18:55.014 INFO:journalctl@ceph.mgr.z.smithi204.stdout:Apr 23 10:18:54 smithi204 podman[34479]: 2024-04-23 10:18:54.729544528 +0000 UTC m=+0.104104235 container create 8a04d6bd3ee9040467b0c927f8cfdfa384601c95813ea4b16200a29e028e95c2 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:068a558e3af4932cf1c60dd321c0bdf500075eb52ea1bf455e0973ba114fbfd4, name=ceph-aa240572-0159-11ef-bc93-c7b262605968-mgr-z, org.label-schema.build-date=20240416, org.label-schema.schema-version=1.0, org.label-schema.name=CentOS Stream 8 Base Image, io.buildah.version=1.33.5, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, GIT_CLEAN=True, org.label-schema.license=GPLv2, ceph=True, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_BRANCH=HEAD, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux , RELEASE=wip-lusov-quiescer-d69f13d, org.label-schema.vendor=CentOS) 2024-04-23T10:18:55.014 INFO:journalctl@ceph.mgr.z.smithi204.stdout:Apr 23 10:18:54 smithi204 podman[34479]: 2024-04-23 10:18:54.643335456 +0000 UTC m=+0.017895142 image pull da4b004c09e820713483d039bc76fbd660be678f2ea3c03e059a6161b74dab76 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:068a558e3af4932cf1c60dd321c0bdf500075eb52ea1bf455e0973ba114fbfd4 2024-04-23T10:18:55.014 INFO:journalctl@ceph.mgr.z.smithi204.stdout:Apr 23 10:18:54 smithi204 podman[34479]: 2024-04-23 10:18:54.932073356 +0000 UTC m=+0.306633018 container init 8a04d6bd3ee9040467b0c927f8cfdfa384601c95813ea4b16200a29e028e95c2 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:068a558e3af4932cf1c60dd321c0bdf500075eb52ea1bf455e0973ba114fbfd4, name=ceph-aa240572-0159-11ef-bc93-c7b262605968-mgr-z, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_CLEAN=True, io.buildah.version=1.33.5, org.label-schema.build-date=20240416, ceph=True, CEPH_POINT_RELEASE=, org.label-schema.vendor=CentOS, org.label-schema.name=CentOS Stream 8 Base Image, RELEASE=wip-lusov-quiescer-d69f13d) 2024-04-23T10:18:55.014 INFO:journalctl@ceph.mgr.z.smithi204.stdout:Apr 23 10:18:54 smithi204 podman[34479]: 2024-04-23 10:18:54.934479704 +0000 UTC m=+0.309039378 container start 8a04d6bd3ee9040467b0c927f8cfdfa384601c95813ea4b16200a29e028e95c2 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:068a558e3af4932cf1c60dd321c0bdf500075eb52ea1bf455e0973ba114fbfd4, name=ceph-aa240572-0159-11ef-bc93-c7b262605968-mgr-z, maintainer=Guillaume Abrioux , ceph=True, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.build-date=20240416, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, GIT_CLEAN=True, io.buildah.version=1.33.5, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=wip-lusov-quiescer-d69f13d, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, CEPH_POINT_RELEASE=) 2024-04-23T10:18:55.014 INFO:journalctl@ceph.mgr.z.smithi204.stdout:Apr 23 10:18:54 smithi204 ceph-mgr[34497]: -- 172.21.15.204:0/1326515344 <== mon.1 v2:172.21.15.204:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 194+0+0 (secure 0 0 0) 0x563a8b68c5a0 con 0x563a8b660800 2024-04-23T10:18:55.014 INFO:journalctl@ceph.mgr.z.smithi204.stdout:Apr 23 10:18:55 smithi204 bash[34479]: 8a04d6bd3ee9040467b0c927f8cfdfa384601c95813ea4b16200a29e028e95c2 2024-04-23T10:18:55.079 INFO:journalctl@ceph.mon.a.smithi114.stdout:Apr 23 10:18:54 smithi114 ceph-mon[29255]: pgmap v28: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-23T10:18:55.242 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 23 10:18:54 smithi119 ceph-mon[32608]: pgmap v28: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-23T10:18:55.272 INFO:journalctl@ceph.mgr.z.smithi204.stdout:Apr 23 10:18:55 smithi204 systemd[1]: Started Ceph mgr.z for aa240572-0159-11ef-bc93-c7b262605968. 2024-04-23T10:18:55.272 INFO:journalctl@ceph.mgr.z.smithi204.stdout:Apr 23 10:18:55 smithi204 ceph-aa240572-0159-11ef-bc93-c7b262605968-mgr-z[34493]: 2024-04-23T10:18:55.087+0000 7f5199df1200 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2024-04-23T10:18:55.382 DEBUG:teuthology.orchestra.run.smithi114:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d69f13d827952bb07bb9b7973bca14f97d764d6f shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid aa240572-0159-11ef-bc93-c7b262605968 -- ceph orch daemon add osd smithi114:vg_nvme/lv_4 2024-04-23T10:18:55.578 INFO:journalctl@ceph.mgr.z.smithi204.stdout:Apr 23 10:18:55 smithi204 ceph-aa240572-0159-11ef-bc93-c7b262605968-mgr-z[34493]: 2024-04-23T10:18:55.272+0000 7f5199df1200 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2024-04-23T10:18:55.578 INFO:journalctl@ceph.mgr.z.smithi204.stdout:Apr 23 10:18:55 smithi204 ceph-aa240572-0159-11ef-bc93-c7b262605968-mgr-z[34493]: 2024-04-23T10:18:55.373+0000 7f5199df1200 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2024-04-23T10:18:55.614 INFO:teuthology.orchestra.run.smithi114.stderr:Inferring config /var/lib/ceph/aa240572-0159-11ef-bc93-c7b262605968/mon.a/config 2024-04-23T10:18:56.078 INFO:journalctl@ceph.mgr.z.smithi204.stdout:Apr 23 10:18:55 smithi204 ceph-aa240572-0159-11ef-bc93-c7b262605968-mgr-z[34493]: 2024-04-23T10:18:55.679+0000 7f5199df1200 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2024-04-23T10:18:56.078 INFO:journalctl@ceph.mgr.z.smithi204.stdout:Apr 23 10:18:55 smithi204 ceph-aa240572-0159-11ef-bc93-c7b262605968-mgr-z[34493]: 2024-04-23T10:18:55.764+0000 7f5199df1200 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2024-04-23T10:18:56.374 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 23 10:18:56 smithi204 ceph-mon[32649]: from='mgr.14152 172.21.15.114:0/1529706138' entity='mgr.x' 2024-04-23T10:18:56.374 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 23 10:18:56 smithi204 ceph-mon[32649]: from='mgr.14152 172.21.15.114:0/1529706138' entity='mgr.x' 2024-04-23T10:18:56.374 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 23 10:18:56 smithi204 ceph-mon[32649]: from='mgr.14152 172.21.15.114:0/1529706138' entity='mgr.x' 2024-04-23T10:18:56.374 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 23 10:18:56 smithi204 ceph-mon[32649]: from='mgr.14152 172.21.15.114:0/1529706138' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.y", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-04-23T10:18:56.374 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 23 10:18:56 smithi204 ceph-mon[32649]: from='mgr.14152 172.21.15.114:0/1529706138' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mgr.y", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]': finished 2024-04-23T10:18:56.374 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 23 10:18:56 smithi204 ceph-mon[32649]: from='mgr.14152 172.21.15.114:0/1529706138' entity='mgr.x' cmd=[{"prefix": "mgr services"}]: dispatch 2024-04-23T10:18:56.374 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 23 10:18:56 smithi204 ceph-mon[32649]: from='mgr.14152 172.21.15.114:0/1529706138' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-23T10:18:56.374 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 23 10:18:56 smithi204 ceph-mon[32649]: Deploying daemon mgr.y on smithi119 2024-04-23T10:18:56.374 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 23 10:18:56 smithi204 ceph-mon[32649]: pgmap v29: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-23T10:18:56.378 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 23 10:18:56 smithi119 ceph-mon[32608]: from='mgr.14152 172.21.15.114:0/1529706138' entity='mgr.x' 2024-04-23T10:18:56.378 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 23 10:18:56 smithi119 ceph-mon[32608]: from='mgr.14152 172.21.15.114:0/1529706138' entity='mgr.x' 2024-04-23T10:18:56.378 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 23 10:18:56 smithi119 ceph-mon[32608]: from='mgr.14152 172.21.15.114:0/1529706138' entity='mgr.x' 2024-04-23T10:18:56.378 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 23 10:18:56 smithi119 ceph-mon[32608]: from='mgr.14152 172.21.15.114:0/1529706138' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.y", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-04-23T10:18:56.379 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 23 10:18:56 smithi119 ceph-mon[32608]: from='mgr.14152 172.21.15.114:0/1529706138' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mgr.y", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]': finished 2024-04-23T10:18:56.379 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 23 10:18:56 smithi119 ceph-mon[32608]: from='mgr.14152 172.21.15.114:0/1529706138' entity='mgr.x' cmd=[{"prefix": "mgr services"}]: dispatch 2024-04-23T10:18:56.379 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 23 10:18:56 smithi119 ceph-mon[32608]: from='mgr.14152 172.21.15.114:0/1529706138' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-23T10:18:56.379 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 23 10:18:56 smithi119 ceph-mon[32608]: Deploying daemon mgr.y on smithi119 2024-04-23T10:18:56.379 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 23 10:18:56 smithi119 ceph-mon[32608]: pgmap v29: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-23T10:18:56.395 INFO:journalctl@ceph.mon.a.smithi114.stdout:Apr 23 10:18:56 smithi114 ceph-mon[29255]: from='mgr.14152 172.21.15.114:0/1529706138' entity='mgr.x' 2024-04-23T10:18:56.395 INFO:journalctl@ceph.mon.a.smithi114.stdout:Apr 23 10:18:56 smithi114 ceph-mon[29255]: from='mgr.14152 172.21.15.114:0/1529706138' entity='mgr.x' 2024-04-23T10:18:56.395 INFO:journalctl@ceph.mon.a.smithi114.stdout:Apr 23 10:18:56 smithi114 ceph-mon[29255]: from='mgr.14152 172.21.15.114:0/1529706138' entity='mgr.x' 2024-04-23T10:18:56.395 INFO:journalctl@ceph.mon.a.smithi114.stdout:Apr 23 10:18:56 smithi114 ceph-mon[29255]: from='mgr.14152 172.21.15.114:0/1529706138' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.y", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-04-23T10:18:56.395 INFO:journalctl@ceph.mon.a.smithi114.stdout:Apr 23 10:18:56 smithi114 ceph-mon[29255]: from='mgr.14152 172.21.15.114:0/1529706138' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mgr.y", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]': finished 2024-04-23T10:18:56.395 INFO:journalctl@ceph.mon.a.smithi114.stdout:Apr 23 10:18:56 smithi114 ceph-mon[29255]: from='mgr.14152 172.21.15.114:0/1529706138' entity='mgr.x' cmd=[{"prefix": "mgr services"}]: dispatch 2024-04-23T10:18:56.395 INFO:journalctl@ceph.mon.a.smithi114.stdout:Apr 23 10:18:56 smithi114 ceph-mon[29255]: from='mgr.14152 172.21.15.114:0/1529706138' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-23T10:18:56.395 INFO:journalctl@ceph.mon.a.smithi114.stdout:Apr 23 10:18:56 smithi114 ceph-mon[29255]: Deploying daemon mgr.y on smithi119 2024-04-23T10:18:56.395 INFO:journalctl@ceph.mon.a.smithi114.stdout:Apr 23 10:18:56 smithi114 ceph-mon[29255]: pgmap v29: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-23T10:18:57.156 INFO:journalctl@ceph.mgr.y.smithi119.stdout:Apr 23 10:18:56 smithi119 systemd[1]: Starting Ceph mgr.y for aa240572-0159-11ef-bc93-c7b262605968... 2024-04-23T10:18:57.205 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:18:57.204+0000 7f0fc93b4700 1 -- 172.21.15.114:0/2437013431 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f0fc41014d0 msgr2=0x7f0fc410b080 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T10:18:57.205 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:18:57.204+0000 7f0fc93b4700 1 --2- 172.21.15.114:0/2437013431 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f0fc41014d0 0x7f0fc410b080 secure :-1 s=READY pgs=110 cs=0 l=1 rev1=1 crypto rx=0x7f0fbc0099c0 tx=0x7f0fbc009cd0 comp rx=0 tx=0).stop 2024-04-23T10:18:57.205 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:18:57.204+0000 7f0fc93b4700 1 -- 172.21.15.114:0/2437013431 shutdown_connections 2024-04-23T10:18:57.205 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:18:57.204+0000 7f0fc93b4700 1 --2- 172.21.15.114:0/2437013431 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f0fc41014d0 0x7f0fc410b080 unknown :-1 s=CLOSED pgs=110 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:18:57.205 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:18:57.204+0000 7f0fc93b4700 1 --2- 172.21.15.114:0/2437013431 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f0fc4104090 0x7f0fc41044b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:18:57.205 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:18:57.204+0000 7f0fc93b4700 1 --2- 172.21.15.114:0/2437013431 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f0fc4100b10 0x7f0fc4100f90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:18:57.205 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:18:57.204+0000 7f0fc93b4700 1 -- 172.21.15.114:0/2437013431 >> 172.21.15.114:0/2437013431 conn(0x7f0fc40fc6d0 msgr2=0x7f0fc40feaf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T10:18:57.205 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:18:57.204+0000 7f0fc93b4700 1 -- 172.21.15.114:0/2437013431 shutdown_connections 2024-04-23T10:18:57.205 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:18:57.204+0000 7f0fc93b4700 1 -- 172.21.15.114:0/2437013431 wait complete. 2024-04-23T10:18:57.206 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:18:57.204+0000 7f0fc93b4700 1 Processor -- start 2024-04-23T10:18:57.206 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:18:57.205+0000 7f0fc93b4700 1 -- start start 2024-04-23T10:18:57.206 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:18:57.205+0000 7f0fc93b4700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f0fc4100b10 0x7f0fc4103790 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T10:18:57.206 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:18:57.205+0000 7f0fc93b4700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f0fc41014d0 0x7f0fc4101de0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T10:18:57.206 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:18:57.205+0000 7f0fc93b4700 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f0fc4104090 0x7f0fc4102320 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T10:18:57.206 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:18:57.205+0000 7f0fc93b4700 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x7f0fc410e5c0 con 0x7f0fc4100b10 2024-04-23T10:18:57.206 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:18:57.205+0000 7f0fc93b4700 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 -- 0x7f0fc410e440 con 0x7f0fc41014d0 2024-04-23T10:18:57.206 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:18:57.205+0000 7f0fc93b4700 1 -- --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_getmap magic: 0 -- 0x7f0fc410e740 con 0x7f0fc4104090 2024-04-23T10:18:57.206 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:18:57.205+0000 7f0fc2ffd700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f0fc4100b10 0x7f0fc4103790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T10:18:57.206 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:18:57.205+0000 7f0fc2ffd700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f0fc4100b10 0x7f0fc4103790 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.114:47754/0 (socket says 172.21.15.114:47754) 2024-04-23T10:18:57.206 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:18:57.205+0000 7f0fc2ffd700 1 -- 172.21.15.114:0/934458444 learned_addr learned my addr 172.21.15.114:0/934458444 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-04-23T10:18:57.206 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:18:57.205+0000 7f0fc37fe700 1 --2- 172.21.15.114:0/934458444 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f0fc4104090 0x7f0fc4102320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T10:18:57.206 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:18:57.205+0000 7f0fc27fc700 1 --2- 172.21.15.114:0/934458444 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f0fc41014d0 0x7f0fc4101de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T10:18:57.206 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:18:57.205+0000 7f0fc2ffd700 1 -- 172.21.15.114:0/934458444 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f0fc4104090 msgr2=0x7f0fc4102320 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T10:18:57.206 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:18:57.205+0000 7f0fc2ffd700 1 --2- 172.21.15.114:0/934458444 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f0fc4104090 0x7f0fc4102320 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:18:57.207 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:18:57.205+0000 7f0fc2ffd700 1 -- 172.21.15.114:0/934458444 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f0fc41014d0 msgr2=0x7f0fc4101de0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T10:18:57.207 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:18:57.205+0000 7f0fc2ffd700 1 --2- 172.21.15.114:0/934458444 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f0fc41014d0 0x7f0fc4101de0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:18:57.207 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:18:57.205+0000 7f0fc2ffd700 1 -- 172.21.15.114:0/934458444 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f0fc4102bd0 con 0x7f0fc4100b10 2024-04-23T10:18:57.207 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:18:57.205+0000 7f0fc2ffd700 1 --2- 172.21.15.114:0/934458444 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f0fc4100b10 0x7f0fc4103790 secure :-1 s=READY pgs=111 cs=0 l=1 rev1=1 crypto rx=0x7f0fb4007f10 tx=0x7f0fb400d3b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T10:18:57.207 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:18:57.206+0000 7f0fbbfff700 1 -- 172.21.15.114:0/934458444 <== mon.0 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0fb400db20 con 0x7f0fc4100b10 2024-04-23T10:18:57.207 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:18:57.206+0000 7f0fbbfff700 1 -- 172.21.15.114:0/934458444 <== mon.0 v2:172.21.15.114:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f0fb400dcc0 con 0x7f0fc4100b10 2024-04-23T10:18:57.207 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:18:57.206+0000 7f0fc93b4700 1 -- 172.21.15.114:0/934458444 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f0fc41a2d40 con 0x7f0fc4100b10 2024-04-23T10:18:57.207 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:18:57.206+0000 7f0fbbfff700 1 -- 172.21.15.114:0/934458444 <== mon.0 v2:172.21.15.114:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0fb4004850 con 0x7f0fc4100b10 2024-04-23T10:18:57.207 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:18:57.206+0000 7f0fc93b4700 1 -- 172.21.15.114:0/934458444 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f0fc41a31a0 con 0x7f0fc4100b10 2024-04-23T10:18:57.208 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:18:57.207+0000 7f0fbbfff700 1 -- 172.21.15.114:0/934458444 <== mon.0 v2:172.21.15.114:3300/0 4 ==== mgrmap(e 13) ==== 46615+0+0 (secure 0 0 0) 0x7f0fb401a020 con 0x7f0fc4100b10 2024-04-23T10:18:57.208 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:18:57.207+0000 7f0fc93b4700 1 -- 172.21.15.114:0/934458444 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f0fc4069730 con 0x7f0fc4100b10 2024-04-23T10:18:57.208 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:18:57.207+0000 7f0fbbfff700 1 --2- 172.21.15.114:0/934458444 >> [v2:172.21.15.114:6800/465333808,v1:172.21.15.114:6801/465333808] conn(0x7f0fb0039890 0x7f0fb003bd50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T10:18:57.208 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:18:57.207+0000 7f0fbbfff700 1 -- 172.21.15.114:0/934458444 <== mon.0 v2:172.21.15.114:3300/0 5 ==== osd_map(4..4 src has 1..4) ==== 1155+0+0 (secure 0 0 0) 0x7f0fb404c0e0 con 0x7f0fc4100b10 2024-04-23T10:18:57.208 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:18:57.207+0000 7f0fc27fc700 1 --2- 172.21.15.114:0/934458444 >> [v2:172.21.15.114:6800/465333808,v1:172.21.15.114:6801/465333808] conn(0x7f0fb0039890 0x7f0fb003bd50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T10:18:57.208 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:18:57.207+0000 7f0fc27fc700 1 --2- 172.21.15.114:0/934458444 >> [v2:172.21.15.114:6800/465333808,v1:172.21.15.114:6801/465333808] conn(0x7f0fb0039890 0x7f0fb003bd50 secure :-1 s=READY pgs=43 cs=0 l=1 rev1=1 crypto rx=0x7f0fc406a690 tx=0x7f0fac009190 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T10:18:57.211 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:18:57.212+0000 7f0fbbfff700 1 -- 172.21.15.114:0/934458444 <== mon.0 v2:172.21.15.114:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f0fb4010070 con 0x7f0fc4100b10 2024-04-23T10:18:57.313 INFO:journalctl@ceph.mgr.z.smithi204.stdout:Apr 23 10:18:56 smithi204 ceph-aa240572-0159-11ef-bc93-c7b262605968-mgr-z[34493]: 2024-04-23T10:18:56.987+0000 7f5199df1200 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2024-04-23T10:18:57.313 INFO:journalctl@ceph.mgr.z.smithi204.stdout:Apr 23 10:18:57 smithi204 ceph-aa240572-0159-11ef-bc93-c7b262605968-mgr-z[34493]: 2024-04-23T10:18:57.158+0000 7f5199df1200 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2024-04-23T10:18:57.422 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:18:57.423+0000 7f0fc93b4700 1 -- 172.21.15.114:0/934458444 --> [v2:172.21.15.114:6800/465333808,v1:172.21.15.114:6801/465333808] -- mgr_command(tid 0: {"prefix": "orch daemon add osd", "svc_arg": "smithi114:vg_nvme/lv_4", "target": ["mon-mgr", ""]}) -- 0x7f0fc4062e80 con 0x7f0fb0039890 2024-04-23T10:18:57.424 INFO:journalctl@ceph.mgr.y.smithi119.stdout:Apr 23 10:18:57 smithi119 podman[34004]: 2024-04-23 10:18:57.157221355 +0000 UTC m=+0.096519333 container create c8a099050ba05aa9e6840787cdb34c260b1a6299c3be20f3499f5cf3f0bc522d (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:068a558e3af4932cf1c60dd321c0bdf500075eb52ea1bf455e0973ba114fbfd4, name=ceph-aa240572-0159-11ef-bc93-c7b262605968-mgr-y, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, ceph=True, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, io.buildah.version=1.33.5, GIT_CLEAN=True, RELEASE=wip-lusov-quiescer-d69f13d, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240416, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 8 Base Image) 2024-04-23T10:18:57.424 INFO:journalctl@ceph.mgr.y.smithi119.stdout:Apr 23 10:18:57 smithi119 podman[34004]: 2024-04-23 10:18:57.078859999 +0000 UTC m=+0.018157938 image pull da4b004c09e820713483d039bc76fbd660be678f2ea3c03e059a6161b74dab76 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:068a558e3af4932cf1c60dd321c0bdf500075eb52ea1bf455e0973ba114fbfd4 2024-04-23T10:18:57.424 INFO:journalctl@ceph.mgr.y.smithi119.stdout:Apr 23 10:18:57 smithi119 podman[34004]: 2024-04-23 10:18:57.34257907 +0000 UTC m=+0.281876986 container init c8a099050ba05aa9e6840787cdb34c260b1a6299c3be20f3499f5cf3f0bc522d (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:068a558e3af4932cf1c60dd321c0bdf500075eb52ea1bf455e0973ba114fbfd4, name=ceph-aa240572-0159-11ef-bc93-c7b262605968-mgr-y, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.vendor=CentOS, RELEASE=wip-lusov-quiescer-d69f13d, org.label-schema.license=GPLv2, org.label-schema.build-date=20240416, org.label-schema.schema-version=1.0, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, io.buildah.version=1.33.5, GIT_BRANCH=HEAD) 2024-04-23T10:18:57.424 INFO:journalctl@ceph.mgr.y.smithi119.stdout:Apr 23 10:18:57 smithi119 podman[34004]: 2024-04-23 10:18:57.345064546 +0000 UTC m=+0.284362471 container start c8a099050ba05aa9e6840787cdb34c260b1a6299c3be20f3499f5cf3f0bc522d (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:068a558e3af4932cf1c60dd321c0bdf500075eb52ea1bf455e0973ba114fbfd4, name=ceph-aa240572-0159-11ef-bc93-c7b262605968-mgr-y, org.label-schema.build-date=20240416, maintainer=Guillaume Abrioux , GIT_CLEAN=True, GIT_BRANCH=HEAD, RELEASE=wip-lusov-quiescer-d69f13d, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, io.buildah.version=1.33.5, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=) 2024-04-23T10:18:57.424 INFO:journalctl@ceph.mgr.y.smithi119.stdout:Apr 23 10:18:57 smithi119 ceph-mgr[34023]: -- 172.21.15.119:0/1170496124 <== mon.2 v2:172.21.15.119:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 194+0+0 (secure 0 0 0) 0x5618b4ae05a0 con 0x5618b4ab5000 2024-04-23T10:18:57.578 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 23 10:18:57 smithi204 ceph-mon[32649]: from='mgr.14152 172.21.15.114:0/1529706138' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-04-23T10:18:57.578 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 23 10:18:57 smithi204 ceph-mon[32649]: from='mgr.14152 172.21.15.114:0/1529706138' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-04-23T10:18:57.578 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 23 10:18:57 smithi204 ceph-mon[32649]: from='mgr.14152 172.21.15.114:0/1529706138' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-23T10:18:57.578 INFO:journalctl@ceph.mgr.z.smithi204.stdout:Apr 23 10:18:57 smithi204 ceph-aa240572-0159-11ef-bc93-c7b262605968-mgr-z[34493]: 2024-04-23T10:18:57.333+0000 7f5199df1200 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2024-04-23T10:18:57.691 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 23 10:18:57 smithi119 ceph-mon[32608]: from='mgr.14152 172.21.15.114:0/1529706138' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-04-23T10:18:57.691 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 23 10:18:57 smithi119 ceph-mon[32608]: from='mgr.14152 172.21.15.114:0/1529706138' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-04-23T10:18:57.691 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 23 10:18:57 smithi119 ceph-mon[32608]: from='mgr.14152 172.21.15.114:0/1529706138' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-23T10:18:57.692 INFO:journalctl@ceph.mgr.y.smithi119.stdout:Apr 23 10:18:57 smithi119 bash[34004]: c8a099050ba05aa9e6840787cdb34c260b1a6299c3be20f3499f5cf3f0bc522d 2024-04-23T10:18:57.692 INFO:journalctl@ceph.mgr.y.smithi119.stdout:Apr 23 10:18:57 smithi119 systemd[1]: Started Ceph mgr.y for aa240572-0159-11ef-bc93-c7b262605968. 2024-04-23T10:18:57.692 INFO:journalctl@ceph.mgr.y.smithi119.stdout:Apr 23 10:18:57 smithi119 ceph-aa240572-0159-11ef-bc93-c7b262605968-mgr-y[34019]: 2024-04-23T10:18:57.504+0000 7f14e65c9200 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2024-04-23T10:18:57.692 INFO:journalctl@ceph.mgr.y.smithi119.stdout:Apr 23 10:18:57 smithi119 ceph-aa240572-0159-11ef-bc93-c7b262605968-mgr-y[34019]: 2024-04-23T10:18:57.693+0000 7f14e65c9200 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2024-04-23T10:18:57.754 INFO:journalctl@ceph.mon.a.smithi114.stdout:Apr 23 10:18:57 smithi114 ceph-mon[29255]: from='mgr.14152 172.21.15.114:0/1529706138' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-04-23T10:18:57.754 INFO:journalctl@ceph.mon.a.smithi114.stdout:Apr 23 10:18:57 smithi114 ceph-mon[29255]: from='mgr.14152 172.21.15.114:0/1529706138' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-04-23T10:18:57.754 INFO:journalctl@ceph.mon.a.smithi114.stdout:Apr 23 10:18:57 smithi114 ceph-mon[29255]: from='mgr.14152 172.21.15.114:0/1529706138' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-23T10:18:57.940 INFO:journalctl@ceph.mgr.z.smithi204.stdout:Apr 23 10:18:57 smithi204 ceph-aa240572-0159-11ef-bc93-c7b262605968-mgr-z[34493]: 2024-04-23T10:18:57.628+0000 7f5199df1200 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2024-04-23T10:18:57.940 INFO:journalctl@ceph.mgr.z.smithi204.stdout:Apr 23 10:18:57 smithi204 ceph-aa240572-0159-11ef-bc93-c7b262605968-mgr-z[34493]: 2024-04-23T10:18:57.725+0000 7f5199df1200 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2024-04-23T10:18:57.940 INFO:journalctl@ceph.mgr.z.smithi204.stdout:Apr 23 10:18:57 smithi204 ceph-aa240572-0159-11ef-bc93-c7b262605968-mgr-z[34493]: 2024-04-23T10:18:57.939+0000 7f5199df1200 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2024-04-23T10:18:57.992 INFO:journalctl@ceph.mgr.y.smithi119.stdout:Apr 23 10:18:57 smithi119 ceph-aa240572-0159-11ef-bc93-c7b262605968-mgr-y[34019]: 2024-04-23T10:18:57.800+0000 7f14e65c9200 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2024-04-23T10:18:58.328 INFO:journalctl@ceph.mgr.z.smithi204.stdout:Apr 23 10:18:58 smithi204 ceph-aa240572-0159-11ef-bc93-c7b262605968-mgr-z[34493]: 2024-04-23T10:18:58.141+0000 7f5199df1200 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2024-04-23T10:18:58.469 INFO:journalctl@ceph.mgr.y.smithi119.stdout:Apr 23 10:18:58 smithi119 ceph-aa240572-0159-11ef-bc93-c7b262605968-mgr-y[34019]: 2024-04-23T10:18:58.123+0000 7f14e65c9200 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2024-04-23T10:18:58.469 INFO:journalctl@ceph.mgr.y.smithi119.stdout:Apr 23 10:18:58 smithi119 ceph-aa240572-0159-11ef-bc93-c7b262605968-mgr-y[34019]: 2024-04-23T10:18:58.216+0000 7f14e65c9200 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2024-04-23T10:18:58.742 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 23 10:18:58 smithi119 ceph-mon[32608]: from='client.14217 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi114:vg_nvme/lv_4", "target": ["mon-mgr", ""]}]: dispatch 2024-04-23T10:18:58.742 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 23 10:18:58 smithi119 ceph-mon[32608]: from='mgr.14152 172.21.15.114:0/1529706138' entity='mgr.x' 2024-04-23T10:18:58.742 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 23 10:18:58 smithi119 ceph-mon[32608]: from='mgr.14152 172.21.15.114:0/1529706138' entity='mgr.x' 2024-04-23T10:18:58.742 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 23 10:18:58 smithi119 ceph-mon[32608]: from='mgr.14152 172.21.15.114:0/1529706138' entity='mgr.x' 2024-04-23T10:18:58.742 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 23 10:18:58 smithi119 ceph-mon[32608]: from='mgr.14152 172.21.15.114:0/1529706138' entity='mgr.x' 2024-04-23T10:18:58.742 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 23 10:18:58 smithi119 ceph-mon[32608]: from='mgr.14152 172.21.15.114:0/1529706138' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-23T10:18:58.742 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 23 10:18:58 smithi119 ceph-mon[32608]: pgmap v30: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-23T10:18:58.754 INFO:journalctl@ceph.mon.a.smithi114.stdout:Apr 23 10:18:58 smithi114 ceph-mon[29255]: from='client.14217 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi114:vg_nvme/lv_4", "target": ["mon-mgr", ""]}]: dispatch 2024-04-23T10:18:58.754 INFO:journalctl@ceph.mon.a.smithi114.stdout:Apr 23 10:18:58 smithi114 ceph-mon[29255]: from='mgr.14152 172.21.15.114:0/1529706138' entity='mgr.x' 2024-04-23T10:18:58.754 INFO:journalctl@ceph.mon.a.smithi114.stdout:Apr 23 10:18:58 smithi114 ceph-mon[29255]: from='mgr.14152 172.21.15.114:0/1529706138' entity='mgr.x' 2024-04-23T10:18:58.754 INFO:journalctl@ceph.mon.a.smithi114.stdout:Apr 23 10:18:58 smithi114 ceph-mon[29255]: from='mgr.14152 172.21.15.114:0/1529706138' entity='mgr.x' 2024-04-23T10:18:58.754 INFO:journalctl@ceph.mon.a.smithi114.stdout:Apr 23 10:18:58 smithi114 ceph-mon[29255]: from='mgr.14152 172.21.15.114:0/1529706138' entity='mgr.x' 2024-04-23T10:18:58.754 INFO:journalctl@ceph.mon.a.smithi114.stdout:Apr 23 10:18:58 smithi114 ceph-mon[29255]: from='mgr.14152 172.21.15.114:0/1529706138' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-23T10:18:58.755 INFO:journalctl@ceph.mon.a.smithi114.stdout:Apr 23 10:18:58 smithi114 ceph-mon[29255]: pgmap v30: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-23T10:18:58.778 INFO:journalctl@ceph.mgr.z.smithi204.stdout:Apr 23 10:18:58 smithi204 ceph-aa240572-0159-11ef-bc93-c7b262605968-mgr-z[34493]: 2024-04-23T10:18:58.778+0000 7f5199df1200 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2024-04-23T10:18:58.779 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 23 10:18:58 smithi204 ceph-mon[32649]: from='client.14217 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi114:vg_nvme/lv_4", "target": ["mon-mgr", ""]}]: dispatch 2024-04-23T10:18:58.779 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 23 10:18:58 smithi204 ceph-mon[32649]: from='mgr.14152 172.21.15.114:0/1529706138' entity='mgr.x' 2024-04-23T10:18:58.779 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 23 10:18:58 smithi204 ceph-mon[32649]: from='mgr.14152 172.21.15.114:0/1529706138' entity='mgr.x' 2024-04-23T10:18:58.779 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 23 10:18:58 smithi204 ceph-mon[32649]: from='mgr.14152 172.21.15.114:0/1529706138' entity='mgr.x' 2024-04-23T10:18:58.779 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 23 10:18:58 smithi204 ceph-mon[32649]: from='mgr.14152 172.21.15.114:0/1529706138' entity='mgr.x' 2024-04-23T10:18:58.779 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 23 10:18:58 smithi204 ceph-mon[32649]: from='mgr.14152 172.21.15.114:0/1529706138' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-23T10:18:58.779 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 23 10:18:58 smithi204 ceph-mon[32649]: pgmap v30: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-23T10:18:59.513 INFO:journalctl@ceph.mgr.z.smithi204.stdout:Apr 23 10:18:59 smithi204 ceph-aa240572-0159-11ef-bc93-c7b262605968-mgr-z[34493]: 2024-04-23T10:18:59.214+0000 7f5199df1200 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2024-04-23T10:18:59.513 INFO:journalctl@ceph.mgr.z.smithi204.stdout:Apr 23 10:18:59 smithi204 ceph-aa240572-0159-11ef-bc93-c7b262605968-mgr-z[34493]: 2024-04-23T10:18:59.310+0000 7f5199df1200 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2024-04-23T10:18:59.513 INFO:journalctl@ceph.mgr.z.smithi204.stdout:Apr 23 10:18:59 smithi204 ceph-aa240572-0159-11ef-bc93-c7b262605968-mgr-z[34493]: 2024-04-23T10:18:59.513+0000 7f5199df1200 -1 mgr[py] Module rgw has missing NOTIFY_TYPES member 2024-04-23T10:18:59.742 INFO:journalctl@ceph.mgr.y.smithi119.stdout:Apr 23 10:18:59 smithi119 ceph-aa240572-0159-11ef-bc93-c7b262605968-mgr-y[34019]: 2024-04-23T10:18:59.435+0000 7f14e65c9200 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2024-04-23T10:18:59.742 INFO:journalctl@ceph.mgr.y.smithi119.stdout:Apr 23 10:18:59 smithi119 ceph-aa240572-0159-11ef-bc93-c7b262605968-mgr-y[34019]: 2024-04-23T10:18:59.602+0000 7f14e65c9200 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2024-04-23T10:19:00.053 INFO:journalctl@ceph.mgr.y.smithi119.stdout:Apr 23 10:18:59 smithi119 ceph-aa240572-0159-11ef-bc93-c7b262605968-mgr-y[34019]: 2024-04-23T10:18:59.770+0000 7f14e65c9200 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2024-04-23T10:19:00.053 INFO:journalctl@ceph.mgr.y.smithi119.stdout:Apr 23 10:19:00 smithi119 ceph-aa240572-0159-11ef-bc93-c7b262605968-mgr-y[34019]: 2024-04-23T10:19:00.054+0000 7f14e65c9200 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2024-04-23T10:19:00.054 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 23 10:19:00 smithi119 ceph-mon[32608]: from='mgr.14152 172.21.15.114:0/1529706138' entity='mgr.x' 2024-04-23T10:19:00.054 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 23 10:19:00 smithi119 ceph-mon[32608]: from='mgr.14152 172.21.15.114:0/1529706138' entity='mgr.x' 2024-04-23T10:19:00.054 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 23 10:19:00 smithi119 ceph-mon[32608]: from='mgr.14152 172.21.15.114:0/1529706138' entity='mgr.x' 2024-04-23T10:19:00.054 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 23 10:19:00 smithi119 ceph-mon[32608]: from='mgr.14152 172.21.15.114:0/1529706138' entity='mgr.x' 2024-04-23T10:19:00.054 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 23 10:19:00 smithi119 ceph-mon[32608]: from='mgr.14152 172.21.15.114:0/1529706138' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-23T10:19:00.054 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 23 10:19:00 smithi119 ceph-mon[32608]: from='mgr.14152 172.21.15.114:0/1529706138' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-23T10:19:00.054 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 23 10:19:00 smithi119 ceph-mon[32608]: from='mgr.14152 172.21.15.114:0/1529706138' entity='mgr.x' 2024-04-23T10:19:00.054 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 23 10:19:00 smithi119 ceph-mon[32608]: from='mgr.14152 172.21.15.114:0/1529706138' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.x", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-04-23T10:19:00.054 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 23 10:19:00 smithi119 ceph-mon[32608]: from='mgr.14152 172.21.15.114:0/1529706138' entity='mgr.x' cmd=[{"prefix": "mgr services"}]: dispatch 2024-04-23T10:19:00.054 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 23 10:19:00 smithi119 ceph-mon[32608]: from='mgr.14152 172.21.15.114:0/1529706138' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-23T10:19:00.254 INFO:journalctl@ceph.mon.a.smithi114.stdout:Apr 23 10:19:00 smithi114 ceph-mon[29255]: from='mgr.14152 172.21.15.114:0/1529706138' entity='mgr.x' 2024-04-23T10:19:00.254 INFO:journalctl@ceph.mon.a.smithi114.stdout:Apr 23 10:19:00 smithi114 ceph-mon[29255]: from='mgr.14152 172.21.15.114:0/1529706138' entity='mgr.x' 2024-04-23T10:19:00.254 INFO:journalctl@ceph.mon.a.smithi114.stdout:Apr 23 10:19:00 smithi114 ceph-mon[29255]: from='mgr.14152 172.21.15.114:0/1529706138' entity='mgr.x' 2024-04-23T10:19:00.255 INFO:journalctl@ceph.mon.a.smithi114.stdout:Apr 23 10:19:00 smithi114 ceph-mon[29255]: from='mgr.14152 172.21.15.114:0/1529706138' entity='mgr.x' 2024-04-23T10:19:00.255 INFO:journalctl@ceph.mon.a.smithi114.stdout:Apr 23 10:19:00 smithi114 ceph-mon[29255]: from='mgr.14152 172.21.15.114:0/1529706138' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-23T10:19:00.255 INFO:journalctl@ceph.mon.a.smithi114.stdout:Apr 23 10:19:00 smithi114 ceph-mon[29255]: from='mgr.14152 172.21.15.114:0/1529706138' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-23T10:19:00.255 INFO:journalctl@ceph.mon.a.smithi114.stdout:Apr 23 10:19:00 smithi114 ceph-mon[29255]: from='mgr.14152 172.21.15.114:0/1529706138' entity='mgr.x' 2024-04-23T10:19:00.255 INFO:journalctl@ceph.mon.a.smithi114.stdout:Apr 23 10:19:00 smithi114 ceph-mon[29255]: from='mgr.14152 172.21.15.114:0/1529706138' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.x", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-04-23T10:19:00.255 INFO:journalctl@ceph.mon.a.smithi114.stdout:Apr 23 10:19:00 smithi114 ceph-mon[29255]: from='mgr.14152 172.21.15.114:0/1529706138' entity='mgr.x' cmd=[{"prefix": "mgr services"}]: dispatch 2024-04-23T10:19:00.255 INFO:journalctl@ceph.mon.a.smithi114.stdout:Apr 23 10:19:00 smithi114 ceph-mon[29255]: from='mgr.14152 172.21.15.114:0/1529706138' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-23T10:19:00.328 INFO:journalctl@ceph.mgr.z.smithi204.stdout:Apr 23 10:19:00 smithi204 ceph-aa240572-0159-11ef-bc93-c7b262605968-mgr-z[34493]: 2024-04-23T10:19:00.062+0000 7f5199df1200 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2024-04-23T10:19:00.328 INFO:journalctl@ceph.mgr.z.smithi204.stdout:Apr 23 10:19:00 smithi204 ceph-aa240572-0159-11ef-bc93-c7b262605968-mgr-z[34493]: 2024-04-23T10:19:00.155+0000 7f5199df1200 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2024-04-23T10:19:00.328 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 23 10:19:00 smithi204 ceph-mon[32649]: from='mgr.14152 172.21.15.114:0/1529706138' entity='mgr.x' 2024-04-23T10:19:00.328 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 23 10:19:00 smithi204 ceph-mon[32649]: from='mgr.14152 172.21.15.114:0/1529706138' entity='mgr.x' 2024-04-23T10:19:00.328 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 23 10:19:00 smithi204 ceph-mon[32649]: from='mgr.14152 172.21.15.114:0/1529706138' entity='mgr.x' 2024-04-23T10:19:00.328 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 23 10:19:00 smithi204 ceph-mon[32649]: from='mgr.14152 172.21.15.114:0/1529706138' entity='mgr.x' 2024-04-23T10:19:00.328 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 23 10:19:00 smithi204 ceph-mon[32649]: from='mgr.14152 172.21.15.114:0/1529706138' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-23T10:19:00.328 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 23 10:19:00 smithi204 ceph-mon[32649]: from='mgr.14152 172.21.15.114:0/1529706138' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-23T10:19:00.328 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 23 10:19:00 smithi204 ceph-mon[32649]: from='mgr.14152 172.21.15.114:0/1529706138' entity='mgr.x' 2024-04-23T10:19:00.329 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 23 10:19:00 smithi204 ceph-mon[32649]: from='mgr.14152 172.21.15.114:0/1529706138' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.x", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-04-23T10:19:00.329 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 23 10:19:00 smithi204 ceph-mon[32649]: from='mgr.14152 172.21.15.114:0/1529706138' entity='mgr.x' cmd=[{"prefix": "mgr services"}]: dispatch 2024-04-23T10:19:00.329 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 23 10:19:00 smithi204 ceph-mon[32649]: from='mgr.14152 172.21.15.114:0/1529706138' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-23T10:19:00.345 INFO:journalctl@ceph.mgr.y.smithi119.stdout:Apr 23 10:19:00 smithi119 ceph-aa240572-0159-11ef-bc93-c7b262605968-mgr-y[34019]: 2024-04-23T10:19:00.146+0000 7f14e65c9200 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2024-04-23T10:19:00.345 INFO:journalctl@ceph.mgr.y.smithi119.stdout:Apr 23 10:19:00 smithi119 ceph-aa240572-0159-11ef-bc93-c7b262605968-mgr-y[34019]: 2024-04-23T10:19:00.346+0000 7f14e65c9200 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2024-04-23T10:19:00.738 INFO:journalctl@ceph.mgr.z.smithi204.stdout:Apr 23 10:19:00 smithi204 ceph-aa240572-0159-11ef-bc93-c7b262605968-mgr-z[34493]: 2024-04-23T10:19:00.397+0000 7f5199df1200 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2024-04-23T10:19:00.738 INFO:journalctl@ceph.mgr.z.smithi204.stdout:Apr 23 10:19:00 smithi204 ceph-aa240572-0159-11ef-bc93-c7b262605968-mgr-z[34493]: 2024-04-23T10:19:00.528+0000 7f5199df1200 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2024-04-23T10:19:00.738 INFO:journalctl@ceph.mgr.z.smithi204.stdout:Apr 23 10:19:00 smithi204 ceph-aa240572-0159-11ef-bc93-c7b262605968-mgr-z[34493]: 2024-04-23T10:19:00.622+0000 7f5199df1200 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2024-04-23T10:19:00.738 INFO:journalctl@ceph.mgr.z.smithi204.stdout:Apr 23 10:19:00 smithi204 ceph-aa240572-0159-11ef-bc93-c7b262605968-mgr-z[34493]: 2024-04-23T10:19:00.738+0000 7f5199df1200 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2024-04-23T10:19:00.742 INFO:journalctl@ceph.mgr.y.smithi119.stdout:Apr 23 10:19:00 smithi119 ceph-aa240572-0159-11ef-bc93-c7b262605968-mgr-y[34019]: 2024-04-23T10:19:00.536+0000 7f14e65c9200 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2024-04-23T10:19:01.328 INFO:journalctl@ceph.mgr.z.smithi204.stdout:Apr 23 10:19:01 smithi204 ceph-aa240572-0159-11ef-bc93-c7b262605968-mgr-z[34493]: 2024-04-23T10:19:01.042+0000 7f5199df1200 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2024-04-23T10:19:01.328 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 23 10:19:01 smithi204 ceph-mon[32649]: Reconfiguring mgr.x (unknown last config time)... 2024-04-23T10:19:01.328 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 23 10:19:01 smithi204 ceph-mon[32649]: Reconfiguring daemon mgr.x on smithi114 2024-04-23T10:19:01.328 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 23 10:19:01 smithi204 ceph-mon[32649]: pgmap v31: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-23T10:19:01.328 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 23 10:19:01 smithi204 ceph-mon[32649]: from='client.? 172.21.15.114:0/1209566228' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "fd64adde-9445-45a4-a171-ef656e2c5709"}]: dispatch 2024-04-23T10:19:01.328 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 23 10:19:01 smithi204 ceph-mon[32649]: from='client.? 172.21.15.114:0/1209566228' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "fd64adde-9445-45a4-a171-ef656e2c5709"}]': finished 2024-04-23T10:19:01.329 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 23 10:19:01 smithi204 ceph-mon[32649]: osdmap e5: 1 total, 0 up, 1 in 2024-04-23T10:19:01.329 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 23 10:19:01 smithi204 ceph-mon[32649]: from='mgr.14152 172.21.15.114:0/1529706138' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-04-23T10:19:01.329 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 23 10:19:01 smithi204 ceph-mon[32649]: from='client.? 172.21.15.114:0/1556672770' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-04-23T10:19:01.329 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 23 10:19:01 smithi204 ceph-mon[32649]: from='mgr.14152 172.21.15.114:0/1529706138' entity='mgr.x' 2024-04-23T10:19:01.492 INFO:journalctl@ceph.mgr.y.smithi119.stdout:Apr 23 10:19:01 smithi119 ceph-aa240572-0159-11ef-bc93-c7b262605968-mgr-y[34019]: 2024-04-23T10:19:01.130+0000 7f14e65c9200 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2024-04-23T10:19:01.492 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 23 10:19:01 smithi119 ceph-mon[32608]: Reconfiguring mgr.x (unknown last config time)... 2024-04-23T10:19:01.492 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 23 10:19:01 smithi119 ceph-mon[32608]: Reconfiguring daemon mgr.x on smithi114 2024-04-23T10:19:01.492 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 23 10:19:01 smithi119 ceph-mon[32608]: pgmap v31: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-23T10:19:01.493 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 23 10:19:01 smithi119 ceph-mon[32608]: from='client.? 172.21.15.114:0/1209566228' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "fd64adde-9445-45a4-a171-ef656e2c5709"}]: dispatch 2024-04-23T10:19:01.493 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 23 10:19:01 smithi119 ceph-mon[32608]: from='client.? 172.21.15.114:0/1209566228' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "fd64adde-9445-45a4-a171-ef656e2c5709"}]': finished 2024-04-23T10:19:01.493 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 23 10:19:01 smithi119 ceph-mon[32608]: osdmap e5: 1 total, 0 up, 1 in 2024-04-23T10:19:01.493 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 23 10:19:01 smithi119 ceph-mon[32608]: from='mgr.14152 172.21.15.114:0/1529706138' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-04-23T10:19:01.493 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 23 10:19:01 smithi119 ceph-mon[32608]: from='client.? 172.21.15.114:0/1556672770' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-04-23T10:19:01.493 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 23 10:19:01 smithi119 ceph-mon[32608]: from='mgr.14152 172.21.15.114:0/1529706138' entity='mgr.x' 2024-04-23T10:19:01.504 INFO:journalctl@ceph.mon.a.smithi114.stdout:Apr 23 10:19:01 smithi114 ceph-mon[29255]: Reconfiguring mgr.x (unknown last config time)... 2024-04-23T10:19:01.504 INFO:journalctl@ceph.mon.a.smithi114.stdout:Apr 23 10:19:01 smithi114 ceph-mon[29255]: Reconfiguring daemon mgr.x on smithi114 2024-04-23T10:19:01.505 INFO:journalctl@ceph.mon.a.smithi114.stdout:Apr 23 10:19:01 smithi114 ceph-mon[29255]: pgmap v31: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-23T10:19:01.505 INFO:journalctl@ceph.mon.a.smithi114.stdout:Apr 23 10:19:01 smithi114 ceph-mon[29255]: from='client.? 172.21.15.114:0/1209566228' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "fd64adde-9445-45a4-a171-ef656e2c5709"}]: dispatch 2024-04-23T10:19:01.505 INFO:journalctl@ceph.mon.a.smithi114.stdout:Apr 23 10:19:01 smithi114 ceph-mon[29255]: from='client.? 172.21.15.114:0/1209566228' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "fd64adde-9445-45a4-a171-ef656e2c5709"}]': finished 2024-04-23T10:19:01.505 INFO:journalctl@ceph.mon.a.smithi114.stdout:Apr 23 10:19:01 smithi114 ceph-mon[29255]: osdmap e5: 1 total, 0 up, 1 in 2024-04-23T10:19:01.505 INFO:journalctl@ceph.mon.a.smithi114.stdout:Apr 23 10:19:01 smithi114 ceph-mon[29255]: from='mgr.14152 172.21.15.114:0/1529706138' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-04-23T10:19:01.505 INFO:journalctl@ceph.mon.a.smithi114.stdout:Apr 23 10:19:01 smithi114 ceph-mon[29255]: from='client.? 172.21.15.114:0/1556672770' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-04-23T10:19:01.505 INFO:journalctl@ceph.mon.a.smithi114.stdout:Apr 23 10:19:01 smithi114 ceph-mon[29255]: from='mgr.14152 172.21.15.114:0/1529706138' entity='mgr.x' 2024-04-23T10:19:01.838 INFO:journalctl@ceph.mgr.y.smithi119.stdout:Apr 23 10:19:01 smithi119 ceph-aa240572-0159-11ef-bc93-c7b262605968-mgr-y[34019]: 2024-04-23T10:19:01.546+0000 7f14e65c9200 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2024-04-23T10:19:01.838 INFO:journalctl@ceph.mgr.y.smithi119.stdout:Apr 23 10:19:01 smithi119 ceph-aa240572-0159-11ef-bc93-c7b262605968-mgr-y[34019]: 2024-04-23T10:19:01.641+0000 7f14e65c9200 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2024-04-23T10:19:01.838 INFO:journalctl@ceph.mgr.y.smithi119.stdout:Apr 23 10:19:01 smithi119 ceph-aa240572-0159-11ef-bc93-c7b262605968-mgr-y[34019]: 2024-04-23T10:19:01.840+0000 7f14e65c9200 -1 mgr[py] Module rgw has missing NOTIFY_TYPES member 2024-04-23T10:19:02.070 INFO:journalctl@ceph.mgr.z.smithi204.stdout:Apr 23 10:19:01 smithi204 ceph-aa240572-0159-11ef-bc93-c7b262605968-mgr-z[34493]: 2024-04-23T10:19:01.642+0000 7f5199df1200 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2024-04-23T10:19:02.070 INFO:journalctl@ceph.mgr.z.smithi204.stdout:Apr 23 10:19:01 smithi204 ceph-aa240572-0159-11ef-bc93-c7b262605968-mgr-z[34493]: 2024-04-23T10:19:01.859+0000 7f5199df1200 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2024-04-23T10:19:02.072 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:19:02.072+0000 7f0fbbfff700 1 -- 172.21.15.114:0/934458444 <== mon.0 v2:172.21.15.114:3300/0 7 ==== mgrmap(e 14) ==== 92603+0+0 (secure 0 0 0) 0x7f0fb4012530 con 0x7f0fc4100b10 2024-04-23T10:19:02.242 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 23 10:19:02 smithi119 ceph-mon[32608]: Standby manager daemon z started 2024-04-23T10:19:02.242 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 23 10:19:02 smithi119 ceph-mon[32608]: from='mgr.? 172.21.15.204:0/23901281' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/z/crt"}]: dispatch 2024-04-23T10:19:02.243 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 23 10:19:02 smithi119 ceph-mon[32608]: from='mgr.? 172.21.15.204:0/23901281' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-04-23T10:19:02.243 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 23 10:19:02 smithi119 ceph-mon[32608]: from='mgr.? 172.21.15.204:0/23901281' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/z/key"}]: dispatch 2024-04-23T10:19:02.243 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 23 10:19:02 smithi119 ceph-mon[32608]: from='mgr.? 172.21.15.204:0/23901281' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-04-23T10:19:02.243 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 23 10:19:02 smithi119 ceph-mon[32608]: pgmap v33: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-23T10:19:02.328 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 23 10:19:02 smithi204 ceph-mon[32649]: Standby manager daemon z started 2024-04-23T10:19:02.328 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 23 10:19:02 smithi204 ceph-mon[32649]: from='mgr.? 172.21.15.204:0/23901281' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/z/crt"}]: dispatch 2024-04-23T10:19:02.328 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 23 10:19:02 smithi204 ceph-mon[32649]: from='mgr.? 172.21.15.204:0/23901281' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-04-23T10:19:02.328 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 23 10:19:02 smithi204 ceph-mon[32649]: from='mgr.? 172.21.15.204:0/23901281' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/z/key"}]: dispatch 2024-04-23T10:19:02.328 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 23 10:19:02 smithi204 ceph-mon[32649]: from='mgr.? 172.21.15.204:0/23901281' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-04-23T10:19:02.328 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 23 10:19:02 smithi204 ceph-mon[32649]: pgmap v33: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-23T10:19:02.504 INFO:journalctl@ceph.mon.a.smithi114.stdout:Apr 23 10:19:02 smithi114 ceph-mon[29255]: Standby manager daemon z started 2024-04-23T10:19:02.504 INFO:journalctl@ceph.mon.a.smithi114.stdout:Apr 23 10:19:02 smithi114 ceph-mon[29255]: from='mgr.? 172.21.15.204:0/23901281' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/z/crt"}]: dispatch 2024-04-23T10:19:02.504 INFO:journalctl@ceph.mon.a.smithi114.stdout:Apr 23 10:19:02 smithi114 ceph-mon[29255]: from='mgr.? 172.21.15.204:0/23901281' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-04-23T10:19:02.504 INFO:journalctl@ceph.mon.a.smithi114.stdout:Apr 23 10:19:02 smithi114 ceph-mon[29255]: from='mgr.? 172.21.15.204:0/23901281' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/z/key"}]: dispatch 2024-04-23T10:19:02.504 INFO:journalctl@ceph.mon.a.smithi114.stdout:Apr 23 10:19:02 smithi114 ceph-mon[29255]: from='mgr.? 172.21.15.204:0/23901281' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-04-23T10:19:02.504 INFO:journalctl@ceph.mon.a.smithi114.stdout:Apr 23 10:19:02 smithi114 ceph-mon[29255]: pgmap v33: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-23T10:19:02.685 INFO:journalctl@ceph.mgr.y.smithi119.stdout:Apr 23 10:19:02 smithi119 ceph-aa240572-0159-11ef-bc93-c7b262605968-mgr-y[34019]: 2024-04-23T10:19:02.361+0000 7f14e65c9200 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2024-04-23T10:19:02.685 INFO:journalctl@ceph.mgr.y.smithi119.stdout:Apr 23 10:19:02 smithi119 ceph-aa240572-0159-11ef-bc93-c7b262605968-mgr-y[34019]: 2024-04-23T10:19:02.450+0000 7f14e65c9200 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2024-04-23T10:19:02.992 INFO:journalctl@ceph.mgr.y.smithi119.stdout:Apr 23 10:19:02 smithi119 ceph-aa240572-0159-11ef-bc93-c7b262605968-mgr-y[34019]: 2024-04-23T10:19:02.686+0000 7f14e65c9200 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2024-04-23T10:19:02.992 INFO:journalctl@ceph.mgr.y.smithi119.stdout:Apr 23 10:19:02 smithi119 ceph-aa240572-0159-11ef-bc93-c7b262605968-mgr-y[34019]: 2024-04-23T10:19:02.817+0000 7f14e65c9200 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2024-04-23T10:19:02.992 INFO:journalctl@ceph.mgr.y.smithi119.stdout:Apr 23 10:19:02 smithi119 ceph-aa240572-0159-11ef-bc93-c7b262605968-mgr-y[34019]: 2024-04-23T10:19:02.910+0000 7f14e65c9200 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2024-04-23T10:19:03.319 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 23 10:19:03 smithi119 ceph-mon[32608]: mgrmap e14: x(active, since 76s), standbys: z 2024-04-23T10:19:03.319 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 23 10:19:03 smithi119 ceph-mon[32608]: from='mgr.14152 172.21.15.114:0/1529706138' entity='mgr.x' cmd=[{"prefix": "mgr metadata", "who": "z", "id": "z"}]: dispatch 2024-04-23T10:19:03.319 INFO:journalctl@ceph.mgr.y.smithi119.stdout:Apr 23 10:19:03 smithi119 ceph-aa240572-0159-11ef-bc93-c7b262605968-mgr-y[34019]: 2024-04-23T10:19:03.024+0000 7f14e65c9200 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2024-04-23T10:19:03.320 INFO:journalctl@ceph.mgr.y.smithi119.stdout:Apr 23 10:19:03 smithi119 ceph-aa240572-0159-11ef-bc93-c7b262605968-mgr-y[34019]: 2024-04-23T10:19:03.320+0000 7f14e65c9200 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2024-04-23T10:19:03.328 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 23 10:19:03 smithi204 ceph-mon[32649]: mgrmap e14: x(active, since 76s), standbys: z 2024-04-23T10:19:03.328 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 23 10:19:03 smithi204 ceph-mon[32649]: from='mgr.14152 172.21.15.114:0/1529706138' entity='mgr.x' cmd=[{"prefix": "mgr metadata", "who": "z", "id": "z"}]: dispatch 2024-04-23T10:19:03.504 INFO:journalctl@ceph.mon.a.smithi114.stdout:Apr 23 10:19:03 smithi114 ceph-mon[29255]: mgrmap e14: x(active, since 76s), standbys: z 2024-04-23T10:19:03.504 INFO:journalctl@ceph.mon.a.smithi114.stdout:Apr 23 10:19:03 smithi114 ceph-mon[29255]: from='mgr.14152 172.21.15.114:0/1529706138' entity='mgr.x' cmd=[{"prefix": "mgr metadata", "who": "z", "id": "z"}]: dispatch 2024-04-23T10:19:04.242 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 23 10:19:04 smithi119 ceph-mon[32608]: pgmap v34: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-23T10:19:04.242 INFO:journalctl@ceph.mgr.y.smithi119.stdout:Apr 23 10:19:03 smithi119 ceph-aa240572-0159-11ef-bc93-c7b262605968-mgr-y[34019]: 2024-04-23T10:19:03.906+0000 7f14e65c9200 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2024-04-23T10:19:04.242 INFO:journalctl@ceph.mgr.y.smithi119.stdout:Apr 23 10:19:04 smithi119 ceph-aa240572-0159-11ef-bc93-c7b262605968-mgr-y[34019]: 2024-04-23T10:19:04.118+0000 7f14e65c9200 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2024-04-23T10:19:04.254 INFO:journalctl@ceph.mon.a.smithi114.stdout:Apr 23 10:19:04 smithi114 ceph-mon[29255]: pgmap v34: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-23T10:19:04.328 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 23 10:19:04 smithi204 ceph-mon[32649]: pgmap v34: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-23T10:19:05.090 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:19:05.091+0000 7f0fbbfff700 1 -- 172.21.15.114:0/934458444 <== mon.0 v2:172.21.15.114:3300/0 8 ==== mgrmap(e 15) ==== 138591+0+0 (secure 0 0 0) 0x7f0fb405ed50 con 0x7f0fc4100b10 2024-04-23T10:19:05.254 INFO:journalctl@ceph.mon.a.smithi114.stdout:Apr 23 10:19:05 smithi114 ceph-mon[29255]: Standby manager daemon y started 2024-04-23T10:19:05.254 INFO:journalctl@ceph.mon.a.smithi114.stdout:Apr 23 10:19:05 smithi114 ceph-mon[29255]: from='mgr.? 172.21.15.119:0/1221870383' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/y/crt"}]: dispatch 2024-04-23T10:19:05.254 INFO:journalctl@ceph.mon.a.smithi114.stdout:Apr 23 10:19:05 smithi114 ceph-mon[29255]: from='mgr.? 172.21.15.119:0/1221870383' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-04-23T10:19:05.254 INFO:journalctl@ceph.mon.a.smithi114.stdout:Apr 23 10:19:05 smithi114 ceph-mon[29255]: from='mgr.? 172.21.15.119:0/1221870383' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/y/key"}]: dispatch 2024-04-23T10:19:05.254 INFO:journalctl@ceph.mon.a.smithi114.stdout:Apr 23 10:19:05 smithi114 ceph-mon[29255]: from='mgr.? 172.21.15.119:0/1221870383' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-04-23T10:19:05.492 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 23 10:19:05 smithi119 ceph-mon[32608]: Standby manager daemon y started 2024-04-23T10:19:05.492 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 23 10:19:05 smithi119 ceph-mon[32608]: from='mgr.? 172.21.15.119:0/1221870383' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/y/crt"}]: dispatch 2024-04-23T10:19:05.492 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 23 10:19:05 smithi119 ceph-mon[32608]: from='mgr.? 172.21.15.119:0/1221870383' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-04-23T10:19:05.492 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 23 10:19:05 smithi119 ceph-mon[32608]: from='mgr.? 172.21.15.119:0/1221870383' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/y/key"}]: dispatch 2024-04-23T10:19:05.492 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 23 10:19:05 smithi119 ceph-mon[32608]: from='mgr.? 172.21.15.119:0/1221870383' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-04-23T10:19:05.578 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 23 10:19:05 smithi204 ceph-mon[32649]: Standby manager daemon y started 2024-04-23T10:19:05.578 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 23 10:19:05 smithi204 ceph-mon[32649]: from='mgr.? 172.21.15.119:0/1221870383' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/y/crt"}]: dispatch 2024-04-23T10:19:05.578 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 23 10:19:05 smithi204 ceph-mon[32649]: from='mgr.? 172.21.15.119:0/1221870383' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-04-23T10:19:05.578 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 23 10:19:05 smithi204 ceph-mon[32649]: from='mgr.? 172.21.15.119:0/1221870383' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/y/key"}]: dispatch 2024-04-23T10:19:05.578 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 23 10:19:05 smithi204 ceph-mon[32649]: from='mgr.? 172.21.15.119:0/1221870383' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-04-23T10:19:06.242 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 23 10:19:06 smithi119 ceph-mon[32608]: mgrmap e15: x(active, since 79s), standbys: z, y 2024-04-23T10:19:06.242 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 23 10:19:06 smithi119 ceph-mon[32608]: from='mgr.14152 172.21.15.114:0/1529706138' entity='mgr.x' cmd=[{"prefix": "mgr metadata", "who": "y", "id": "y"}]: dispatch 2024-04-23T10:19:06.242 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 23 10:19:06 smithi119 ceph-mon[32608]: from='mgr.14152 172.21.15.114:0/1529706138' entity='mgr.x' 2024-04-23T10:19:06.242 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 23 10:19:06 smithi119 ceph-mon[32608]: from='mgr.14152 172.21.15.114:0/1529706138' entity='mgr.x' 2024-04-23T10:19:06.242 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 23 10:19:06 smithi119 ceph-mon[32608]: from='mgr.14152 172.21.15.114:0/1529706138' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-23T10:19:06.242 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 23 10:19:06 smithi119 ceph-mon[32608]: from='mgr.14152 172.21.15.114:0/1529706138' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-23T10:19:06.242 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 23 10:19:06 smithi119 ceph-mon[32608]: from='mgr.14152 172.21.15.114:0/1529706138' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-23T10:19:06.243 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 23 10:19:06 smithi119 ceph-mon[32608]: from='mgr.14152 172.21.15.114:0/1529706138' entity='mgr.x' 2024-04-23T10:19:06.243 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 23 10:19:06 smithi119 ceph-mon[32608]: pgmap v35: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-23T10:19:06.289 INFO:journalctl@ceph.mon.a.smithi114.stdout:Apr 23 10:19:06 smithi114 ceph-mon[29255]: mgrmap e15: x(active, since 79s), standbys: z, y 2024-04-23T10:19:06.289 INFO:journalctl@ceph.mon.a.smithi114.stdout:Apr 23 10:19:06 smithi114 ceph-mon[29255]: from='mgr.14152 172.21.15.114:0/1529706138' entity='mgr.x' cmd=[{"prefix": "mgr metadata", "who": "y", "id": "y"}]: dispatch 2024-04-23T10:19:06.289 INFO:journalctl@ceph.mon.a.smithi114.stdout:Apr 23 10:19:06 smithi114 ceph-mon[29255]: from='mgr.14152 172.21.15.114:0/1529706138' entity='mgr.x' 2024-04-23T10:19:06.289 INFO:journalctl@ceph.mon.a.smithi114.stdout:Apr 23 10:19:06 smithi114 ceph-mon[29255]: from='mgr.14152 172.21.15.114:0/1529706138' entity='mgr.x' 2024-04-23T10:19:06.289 INFO:journalctl@ceph.mon.a.smithi114.stdout:Apr 23 10:19:06 smithi114 ceph-mon[29255]: from='mgr.14152 172.21.15.114:0/1529706138' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-23T10:19:06.289 INFO:journalctl@ceph.mon.a.smithi114.stdout:Apr 23 10:19:06 smithi114 ceph-mon[29255]: from='mgr.14152 172.21.15.114:0/1529706138' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-23T10:19:06.289 INFO:journalctl@ceph.mon.a.smithi114.stdout:Apr 23 10:19:06 smithi114 ceph-mon[29255]: from='mgr.14152 172.21.15.114:0/1529706138' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-23T10:19:06.289 INFO:journalctl@ceph.mon.a.smithi114.stdout:Apr 23 10:19:06 smithi114 ceph-mon[29255]: from='mgr.14152 172.21.15.114:0/1529706138' entity='mgr.x' 2024-04-23T10:19:06.289 INFO:journalctl@ceph.mon.a.smithi114.stdout:Apr 23 10:19:06 smithi114 ceph-mon[29255]: pgmap v35: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-23T10:19:06.578 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 23 10:19:06 smithi204 ceph-mon[32649]: mgrmap e15: x(active, since 79s), standbys: z, y 2024-04-23T10:19:06.578 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 23 10:19:06 smithi204 ceph-mon[32649]: from='mgr.14152 172.21.15.114:0/1529706138' entity='mgr.x' cmd=[{"prefix": "mgr metadata", "who": "y", "id": "y"}]: dispatch 2024-04-23T10:19:06.578 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 23 10:19:06 smithi204 ceph-mon[32649]: from='mgr.14152 172.21.15.114:0/1529706138' entity='mgr.x' 2024-04-23T10:19:06.578 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 23 10:19:06 smithi204 ceph-mon[32649]: from='mgr.14152 172.21.15.114:0/1529706138' entity='mgr.x' 2024-04-23T10:19:06.578 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 23 10:19:06 smithi204 ceph-mon[32649]: from='mgr.14152 172.21.15.114:0/1529706138' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-23T10:19:06.578 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 23 10:19:06 smithi204 ceph-mon[32649]: from='mgr.14152 172.21.15.114:0/1529706138' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-23T10:19:06.578 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 23 10:19:06 smithi204 ceph-mon[32649]: from='mgr.14152 172.21.15.114:0/1529706138' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-23T10:19:06.578 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 23 10:19:06 smithi204 ceph-mon[32649]: from='mgr.14152 172.21.15.114:0/1529706138' entity='mgr.x' 2024-04-23T10:19:06.579 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 23 10:19:06 smithi204 ceph-mon[32649]: pgmap v35: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-23T10:19:09.051 INFO:journalctl@ceph.mon.a.smithi114.stdout:Apr 23 10:19:08 smithi114 ceph-mon[29255]: pgmap v36: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-23T10:19:09.052 INFO:journalctl@ceph.mon.a.smithi114.stdout:Apr 23 10:19:08 smithi114 ceph-mon[29255]: from='mgr.14152 172.21.15.114:0/1529706138' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.0"}]: dispatch 2024-04-23T10:19:09.052 INFO:journalctl@ceph.mon.a.smithi114.stdout:Apr 23 10:19:08 smithi114 ceph-mon[29255]: from='mgr.14152 172.21.15.114:0/1529706138' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-23T10:19:09.242 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 23 10:19:08 smithi119 ceph-mon[32608]: pgmap v36: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-23T10:19:09.242 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 23 10:19:08 smithi119 ceph-mon[32608]: from='mgr.14152 172.21.15.114:0/1529706138' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.0"}]: dispatch 2024-04-23T10:19:09.242 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 23 10:19:08 smithi119 ceph-mon[32608]: from='mgr.14152 172.21.15.114:0/1529706138' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-23T10:19:09.328 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 23 10:19:08 smithi204 ceph-mon[32649]: pgmap v36: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-23T10:19:09.328 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 23 10:19:08 smithi204 ceph-mon[32649]: from='mgr.14152 172.21.15.114:0/1529706138' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.0"}]: dispatch 2024-04-23T10:19:09.328 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 23 10:19:08 smithi204 ceph-mon[32649]: from='mgr.14152 172.21.15.114:0/1529706138' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-23T10:19:10.243 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 23 10:19:09 smithi119 ceph-mon[32608]: Deploying daemon osd.0 on smithi114 2024-04-23T10:19:10.254 INFO:journalctl@ceph.mon.a.smithi114.stdout:Apr 23 10:19:09 smithi114 ceph-mon[29255]: Deploying daemon osd.0 on smithi114 2024-04-23T10:19:10.328 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 23 10:19:09 smithi204 ceph-mon[32649]: Deploying daemon osd.0 on smithi114 2024-04-23T10:19:11.036 INFO:journalctl@ceph.mon.a.smithi114.stdout:Apr 23 10:19:10 smithi114 ceph-mon[29255]: pgmap v37: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-23T10:19:11.242 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 23 10:19:10 smithi119 ceph-mon[32608]: pgmap v37: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-23T10:19:11.328 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 23 10:19:10 smithi204 ceph-mon[32649]: pgmap v37: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-23T10:19:13.225 INFO:journalctl@ceph.mon.a.smithi114.stdout:Apr 23 10:19:12 smithi114 ceph-mon[29255]: pgmap v38: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-23T10:19:13.242 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 23 10:19:12 smithi119 ceph-mon[32608]: pgmap v38: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-23T10:19:13.328 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 23 10:19:12 smithi204 ceph-mon[32649]: pgmap v38: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-23T10:19:14.742 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 23 10:19:14 smithi119 ceph-mon[32608]: from='mgr.14152 172.21.15.114:0/1529706138' entity='mgr.x' 2024-04-23T10:19:14.742 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 23 10:19:14 smithi119 ceph-mon[32608]: from='mgr.14152 172.21.15.114:0/1529706138' entity='mgr.x' 2024-04-23T10:19:14.742 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 23 10:19:14 smithi119 ceph-mon[32608]: from='mgr.14152 172.21.15.114:0/1529706138' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-23T10:19:14.742 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 23 10:19:14 smithi119 ceph-mon[32608]: pgmap v39: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-23T10:19:14.754 INFO:journalctl@ceph.mon.a.smithi114.stdout:Apr 23 10:19:14 smithi114 ceph-mon[29255]: from='mgr.14152 172.21.15.114:0/1529706138' entity='mgr.x' 2024-04-23T10:19:14.754 INFO:journalctl@ceph.mon.a.smithi114.stdout:Apr 23 10:19:14 smithi114 ceph-mon[29255]: from='mgr.14152 172.21.15.114:0/1529706138' entity='mgr.x' 2024-04-23T10:19:14.754 INFO:journalctl@ceph.mon.a.smithi114.stdout:Apr 23 10:19:14 smithi114 ceph-mon[29255]: from='mgr.14152 172.21.15.114:0/1529706138' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-23T10:19:14.754 INFO:journalctl@ceph.mon.a.smithi114.stdout:Apr 23 10:19:14 smithi114 ceph-mon[29255]: pgmap v39: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-23T10:19:14.828 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 23 10:19:14 smithi204 ceph-mon[32649]: from='mgr.14152 172.21.15.114:0/1529706138' entity='mgr.x' 2024-04-23T10:19:14.828 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 23 10:19:14 smithi204 ceph-mon[32649]: from='mgr.14152 172.21.15.114:0/1529706138' entity='mgr.x' 2024-04-23T10:19:14.828 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 23 10:19:14 smithi204 ceph-mon[32649]: from='mgr.14152 172.21.15.114:0/1529706138' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-23T10:19:14.828 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 23 10:19:14 smithi204 ceph-mon[32649]: pgmap v39: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-23T10:19:17.206 INFO:journalctl@ceph.mon.a.smithi114.stdout:Apr 23 10:19:16 smithi114 ceph-mon[29255]: from='mgr.14152 172.21.15.114:0/1529706138' entity='mgr.x' 2024-04-23T10:19:17.206 INFO:journalctl@ceph.mon.a.smithi114.stdout:Apr 23 10:19:16 smithi114 ceph-mon[29255]: from='mgr.14152 172.21.15.114:0/1529706138' entity='mgr.x' 2024-04-23T10:19:17.206 INFO:journalctl@ceph.mon.a.smithi114.stdout:Apr 23 10:19:16 smithi114 ceph-mon[29255]: from='mgr.14152 172.21.15.114:0/1529706138' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-23T10:19:17.206 INFO:journalctl@ceph.mon.a.smithi114.stdout:Apr 23 10:19:16 smithi114 ceph-mon[29255]: from='mgr.14152 172.21.15.114:0/1529706138' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-23T10:19:17.206 INFO:journalctl@ceph.mon.a.smithi114.stdout:Apr 23 10:19:16 smithi114 ceph-mon[29255]: from='mgr.14152 172.21.15.114:0/1529706138' entity='mgr.x' 2024-04-23T10:19:17.206 INFO:journalctl@ceph.mon.a.smithi114.stdout:Apr 23 10:19:16 smithi114 ceph-mon[29255]: pgmap v40: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-23T10:19:17.242 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 23 10:19:16 smithi119 ceph-mon[32608]: from='mgr.14152 172.21.15.114:0/1529706138' entity='mgr.x' 2024-04-23T10:19:17.242 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 23 10:19:16 smithi119 ceph-mon[32608]: from='mgr.14152 172.21.15.114:0/1529706138' entity='mgr.x' 2024-04-23T10:19:17.242 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 23 10:19:16 smithi119 ceph-mon[32608]: from='mgr.14152 172.21.15.114:0/1529706138' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-23T10:19:17.242 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 23 10:19:16 smithi119 ceph-mon[32608]: from='mgr.14152 172.21.15.114:0/1529706138' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-23T10:19:17.242 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 23 10:19:16 smithi119 ceph-mon[32608]: from='mgr.14152 172.21.15.114:0/1529706138' entity='mgr.x' 2024-04-23T10:19:17.242 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 23 10:19:16 smithi119 ceph-mon[32608]: pgmap v40: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-23T10:19:17.283 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:19:17.283+0000 7f0fbbfff700 1 -- 172.21.15.114:0/934458444 <== mgr.14152 v2:172.21.15.114:6800/465333808 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+37 (secure 0 0 0) 0x7f0fc4062e80 con 0x7f0fb0039890 2024-04-23T10:19:17.283 INFO:teuthology.orchestra.run.smithi114.stdout:Created osd(s) 0 on host 'smithi114' 2024-04-23T10:19:17.285 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:19:17.286+0000 7f0fc93b4700 1 -- 172.21.15.114:0/934458444 >> [v2:172.21.15.114:6800/465333808,v1:172.21.15.114:6801/465333808] conn(0x7f0fb0039890 msgr2=0x7f0fb003bd50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T10:19:17.285 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:19:17.286+0000 7f0fc93b4700 1 --2- 172.21.15.114:0/934458444 >> [v2:172.21.15.114:6800/465333808,v1:172.21.15.114:6801/465333808] conn(0x7f0fb0039890 0x7f0fb003bd50 secure :-1 s=READY pgs=43 cs=0 l=1 rev1=1 crypto rx=0x7f0fc406a690 tx=0x7f0fac009190 comp rx=0 tx=0).stop 2024-04-23T10:19:17.285 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:19:17.286+0000 7f0fc93b4700 1 -- 172.21.15.114:0/934458444 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f0fc4100b10 msgr2=0x7f0fc4103790 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T10:19:17.285 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:19:17.286+0000 7f0fc93b4700 1 --2- 172.21.15.114:0/934458444 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f0fc4100b10 0x7f0fc4103790 secure :-1 s=READY pgs=111 cs=0 l=1 rev1=1 crypto rx=0x7f0fb4007f10 tx=0x7f0fb400d3b0 comp rx=0 tx=0).stop 2024-04-23T10:19:17.285 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:19:17.286+0000 7f0fc93b4700 1 -- 172.21.15.114:0/934458444 shutdown_connections 2024-04-23T10:19:17.286 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:19:17.286+0000 7f0fc93b4700 1 --2- 172.21.15.114:0/934458444 >> [v2:172.21.15.114:6800/465333808,v1:172.21.15.114:6801/465333808] conn(0x7f0fb0039890 0x7f0fb003bd50 unknown :-1 s=CLOSED pgs=43 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:19:17.286 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:19:17.286+0000 7f0fc93b4700 1 --2- 172.21.15.114:0/934458444 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f0fc4104090 0x7f0fc4102320 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:19:17.286 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:19:17.286+0000 7f0fc93b4700 1 --2- 172.21.15.114:0/934458444 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f0fc4100b10 0x7f0fc4103790 unknown :-1 s=CLOSED pgs=111 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:19:17.286 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:19:17.286+0000 7f0fc93b4700 1 --2- 172.21.15.114:0/934458444 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f0fc41014d0 0x7f0fc4101de0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:19:17.286 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:19:17.286+0000 7f0fc93b4700 1 -- 172.21.15.114:0/934458444 >> 172.21.15.114:0/934458444 conn(0x7f0fc40fc6d0 msgr2=0x7f0fc4107130 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T10:19:17.286 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:19:17.286+0000 7f0fc93b4700 1 -- 172.21.15.114:0/934458444 shutdown_connections 2024-04-23T10:19:17.286 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:19:17.286+0000 7f0fc93b4700 1 -- 172.21.15.114:0/934458444 wait complete. 2024-04-23T10:19:17.328 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 23 10:19:16 smithi204 ceph-mon[32649]: from='mgr.14152 172.21.15.114:0/1529706138' entity='mgr.x' 2024-04-23T10:19:17.328 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 23 10:19:16 smithi204 ceph-mon[32649]: from='mgr.14152 172.21.15.114:0/1529706138' entity='mgr.x' 2024-04-23T10:19:17.328 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 23 10:19:16 smithi204 ceph-mon[32649]: from='mgr.14152 172.21.15.114:0/1529706138' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-23T10:19:17.328 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 23 10:19:16 smithi204 ceph-mon[32649]: from='mgr.14152 172.21.15.114:0/1529706138' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-23T10:19:17.328 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 23 10:19:16 smithi204 ceph-mon[32649]: from='mgr.14152 172.21.15.114:0/1529706138' entity='mgr.x' 2024-04-23T10:19:17.328 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 23 10:19:16 smithi204 ceph-mon[32649]: pgmap v40: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-23T10:19:17.870 DEBUG:teuthology.orchestra.run.smithi114:osd.0> sudo journalctl -f -n 0 -u ceph-aa240572-0159-11ef-bc93-c7b262605968@osd.0.service 2024-04-23T10:19:17.871 INFO:tasks.cephadm:Deploying osd.1 on smithi119 with /dev/vg_nvme/lv_4... 2024-04-23T10:19:17.872 DEBUG:teuthology.orchestra.run.smithi119:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d69f13d827952bb07bb9b7973bca14f97d764d6f shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid aa240572-0159-11ef-bc93-c7b262605968 -- ceph-volume lvm zap /dev/vg_nvme/lv_4 2024-04-23T10:19:18.113 INFO:teuthology.orchestra.run.smithi119.stderr:Inferring config /var/lib/ceph/aa240572-0159-11ef-bc93-c7b262605968/mon.b/config 2024-04-23T10:19:18.492 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 23 10:19:18 smithi119 ceph-mon[32608]: from='mgr.14152 172.21.15.114:0/1529706138' entity='mgr.x' 2024-04-23T10:19:18.492 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 23 10:19:18 smithi119 ceph-mon[32608]: from='mgr.14152 172.21.15.114:0/1529706138' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-23T10:19:18.492 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 23 10:19:18 smithi119 ceph-mon[32608]: from='mgr.14152 172.21.15.114:0/1529706138' entity='mgr.x' 2024-04-23T10:19:18.493 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 23 10:19:18 smithi119 ceph-mon[32608]: from='osd.0 [v2:172.21.15.114:6802/736366878,v1:172.21.15.114:6803/736366878]' entity='osd.0' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["0"]}]: dispatch 2024-04-23T10:19:18.493 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 23 10:19:18 smithi119 ceph-mon[32608]: pgmap v41: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-23T10:19:18.504 INFO:journalctl@ceph.mon.a.smithi114.stdout:Apr 23 10:19:18 smithi114 ceph-mon[29255]: from='mgr.14152 172.21.15.114:0/1529706138' entity='mgr.x' 2024-04-23T10:19:18.504 INFO:journalctl@ceph.mon.a.smithi114.stdout:Apr 23 10:19:18 smithi114 ceph-mon[29255]: from='mgr.14152 172.21.15.114:0/1529706138' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-23T10:19:18.504 INFO:journalctl@ceph.mon.a.smithi114.stdout:Apr 23 10:19:18 smithi114 ceph-mon[29255]: from='mgr.14152 172.21.15.114:0/1529706138' entity='mgr.x' 2024-04-23T10:19:18.504 INFO:journalctl@ceph.mon.a.smithi114.stdout:Apr 23 10:19:18 smithi114 ceph-mon[29255]: from='osd.0 [v2:172.21.15.114:6802/736366878,v1:172.21.15.114:6803/736366878]' entity='osd.0' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["0"]}]: dispatch 2024-04-23T10:19:18.504 INFO:journalctl@ceph.mon.a.smithi114.stdout:Apr 23 10:19:18 smithi114 ceph-mon[29255]: pgmap v41: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-23T10:19:18.578 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 23 10:19:18 smithi204 ceph-mon[32649]: from='mgr.14152 172.21.15.114:0/1529706138' entity='mgr.x' 2024-04-23T10:19:18.578 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 23 10:19:18 smithi204 ceph-mon[32649]: from='mgr.14152 172.21.15.114:0/1529706138' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-23T10:19:18.578 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 23 10:19:18 smithi204 ceph-mon[32649]: from='mgr.14152 172.21.15.114:0/1529706138' entity='mgr.x' 2024-04-23T10:19:18.578 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 23 10:19:18 smithi204 ceph-mon[32649]: from='osd.0 [v2:172.21.15.114:6802/736366878,v1:172.21.15.114:6803/736366878]' entity='osd.0' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["0"]}]: dispatch 2024-04-23T10:19:18.578 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 23 10:19:18 smithi204 ceph-mon[32649]: pgmap v41: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-23T10:19:19.467 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 23 10:19:19 smithi119 ceph-mon[32608]: from='osd.0 [v2:172.21.15.114:6802/736366878,v1:172.21.15.114:6803/736366878]' entity='osd.0' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["0"]}]': finished 2024-04-23T10:19:19.467 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 23 10:19:19 smithi119 ceph-mon[32608]: osdmap e6: 1 total, 0 up, 1 in 2024-04-23T10:19:19.467 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 23 10:19:19 smithi119 ceph-mon[32608]: from='osd.0 [v2:172.21.15.114:6802/736366878,v1:172.21.15.114:6803/736366878]' entity='osd.0' cmd=[{"prefix": "osd crush create-or-move", "id": 0, "weight":0.0873, "args": ["host=smithi114", "root=default"]}]: dispatch 2024-04-23T10:19:19.467 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 23 10:19:19 smithi119 ceph-mon[32608]: from='mgr.14152 172.21.15.114:0/1529706138' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-04-23T10:19:19.504 INFO:journalctl@ceph.mon.a.smithi114.stdout:Apr 23 10:19:19 smithi114 ceph-mon[29255]: from='osd.0 [v2:172.21.15.114:6802/736366878,v1:172.21.15.114:6803/736366878]' entity='osd.0' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["0"]}]': finished 2024-04-23T10:19:19.504 INFO:journalctl@ceph.mon.a.smithi114.stdout:Apr 23 10:19:19 smithi114 ceph-mon[29255]: osdmap e6: 1 total, 0 up, 1 in 2024-04-23T10:19:19.504 INFO:journalctl@ceph.mon.a.smithi114.stdout:Apr 23 10:19:19 smithi114 ceph-mon[29255]: from='osd.0 [v2:172.21.15.114:6802/736366878,v1:172.21.15.114:6803/736366878]' entity='osd.0' cmd=[{"prefix": "osd crush create-or-move", "id": 0, "weight":0.0873, "args": ["host=smithi114", "root=default"]}]: dispatch 2024-04-23T10:19:19.504 INFO:journalctl@ceph.mon.a.smithi114.stdout:Apr 23 10:19:19 smithi114 ceph-mon[29255]: from='mgr.14152 172.21.15.114:0/1529706138' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-04-23T10:19:19.578 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 23 10:19:19 smithi204 ceph-mon[32649]: from='osd.0 [v2:172.21.15.114:6802/736366878,v1:172.21.15.114:6803/736366878]' entity='osd.0' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["0"]}]': finished 2024-04-23T10:19:19.578 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 23 10:19:19 smithi204 ceph-mon[32649]: osdmap e6: 1 total, 0 up, 1 in 2024-04-23T10:19:19.578 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 23 10:19:19 smithi204 ceph-mon[32649]: from='osd.0 [v2:172.21.15.114:6802/736366878,v1:172.21.15.114:6803/736366878]' entity='osd.0' cmd=[{"prefix": "osd crush create-or-move", "id": 0, "weight":0.0873, "args": ["host=smithi114", "root=default"]}]: dispatch 2024-04-23T10:19:19.578 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 23 10:19:19 smithi204 ceph-mon[32649]: from='mgr.14152 172.21.15.114:0/1529706138' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-04-23T10:19:20.028 INFO:teuthology.orchestra.run.smithi119.stderr:--> Zapping: /dev/vg_nvme/lv_4 2024-04-23T10:19:20.029 INFO:teuthology.orchestra.run.smithi119.stderr:Running command: /usr/bin/dd if=/dev/zero of=/dev/vg_nvme/lv_4 bs=1M count=10 conv=fsync 2024-04-23T10:19:20.029 INFO:teuthology.orchestra.run.smithi119.stderr: stderr: 10+0 records in 2024-04-23T10:19:20.029 INFO:teuthology.orchestra.run.smithi119.stderr:10+0 records out 2024-04-23T10:19:20.029 INFO:teuthology.orchestra.run.smithi119.stderr: stderr: 10485760 bytes (10 MB, 10 MiB) copied, 0.0124839 s, 840 MB/s 2024-04-23T10:19:20.029 INFO:teuthology.orchestra.run.smithi119.stderr:--> Zapping successful for: 2024-04-23T10:19:20.331 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 23 10:19:20 smithi119 ceph-mon[32608]: from='osd.0 [v2:172.21.15.114:6802/736366878,v1:172.21.15.114:6803/736366878]' entity='osd.0' cmd='[{"prefix": "osd crush create-or-move", "id": 0, "weight":0.0873, "args": ["host=smithi114", "root=default"]}]': finished 2024-04-23T10:19:20.331 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 23 10:19:20 smithi119 ceph-mon[32608]: osdmap e7: 1 total, 0 up, 1 in 2024-04-23T10:19:20.331 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 23 10:19:20 smithi119 ceph-mon[32608]: from='mgr.14152 172.21.15.114:0/1529706138' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-04-23T10:19:20.331 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 23 10:19:20 smithi119 ceph-mon[32608]: from='mgr.14152 172.21.15.114:0/1529706138' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-04-23T10:19:20.332 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 23 10:19:20 smithi119 ceph-mon[32608]: pgmap v44: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-23T10:19:20.332 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 23 10:19:20 smithi119 ceph-mon[32608]: from='osd.0 [v2:172.21.15.114:6802/736366878,v1:172.21.15.114:6803/736366878]' entity='osd.0' 2024-04-23T10:19:20.504 INFO:journalctl@ceph.mon.a.smithi114.stdout:Apr 23 10:19:20 smithi114 ceph-mon[29255]: from='osd.0 [v2:172.21.15.114:6802/736366878,v1:172.21.15.114:6803/736366878]' entity='osd.0' cmd='[{"prefix": "osd crush create-or-move", "id": 0, "weight":0.0873, "args": ["host=smithi114", "root=default"]}]': finished 2024-04-23T10:19:20.504 INFO:journalctl@ceph.mon.a.smithi114.stdout:Apr 23 10:19:20 smithi114 ceph-mon[29255]: osdmap e7: 1 total, 0 up, 1 in 2024-04-23T10:19:20.504 INFO:journalctl@ceph.mon.a.smithi114.stdout:Apr 23 10:19:20 smithi114 ceph-mon[29255]: from='mgr.14152 172.21.15.114:0/1529706138' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-04-23T10:19:20.504 INFO:journalctl@ceph.mon.a.smithi114.stdout:Apr 23 10:19:20 smithi114 ceph-mon[29255]: from='mgr.14152 172.21.15.114:0/1529706138' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-04-23T10:19:20.505 INFO:journalctl@ceph.mon.a.smithi114.stdout:Apr 23 10:19:20 smithi114 ceph-mon[29255]: pgmap v44: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-23T10:19:20.505 INFO:journalctl@ceph.mon.a.smithi114.stdout:Apr 23 10:19:20 smithi114 ceph-mon[29255]: from='osd.0 [v2:172.21.15.114:6802/736366878,v1:172.21.15.114:6803/736366878]' entity='osd.0' 2024-04-23T10:19:20.505 INFO:journalctl@ceph.osd.0.smithi114.stdout:Apr 23 10:19:20 smithi114 ceph-aa240572-0159-11ef-bc93-c7b262605968-osd-0[41269]: 2024-04-23T10:19:20.224+0000 7f5e80984700 -1 osd.0 0 waiting for initial osdmap 2024-04-23T10:19:20.505 INFO:journalctl@ceph.osd.0.smithi114.stdout:Apr 23 10:19:20 smithi114 ceph-aa240572-0159-11ef-bc93-c7b262605968-osd-0[41269]: 2024-04-23T10:19:20.405+0000 7f5e7876f700 -1 osd.0 7 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-04-23T10:19:20.578 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 23 10:19:20 smithi204 ceph-mon[32649]: from='osd.0 [v2:172.21.15.114:6802/736366878,v1:172.21.15.114:6803/736366878]' entity='osd.0' cmd='[{"prefix": "osd crush create-or-move", "id": 0, "weight":0.0873, "args": ["host=smithi114", "root=default"]}]': finished 2024-04-23T10:19:20.578 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 23 10:19:20 smithi204 ceph-mon[32649]: osdmap e7: 1 total, 0 up, 1 in 2024-04-23T10:19:20.578 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 23 10:19:20 smithi204 ceph-mon[32649]: from='mgr.14152 172.21.15.114:0/1529706138' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-04-23T10:19:20.578 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 23 10:19:20 smithi204 ceph-mon[32649]: from='mgr.14152 172.21.15.114:0/1529706138' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-04-23T10:19:20.578 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 23 10:19:20 smithi204 ceph-mon[32649]: pgmap v44: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-23T10:19:20.578 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 23 10:19:20 smithi204 ceph-mon[32649]: from='osd.0 [v2:172.21.15.114:6802/736366878,v1:172.21.15.114:6803/736366878]' entity='osd.0' 2024-04-23T10:19:20.691 DEBUG:teuthology.orchestra.run.smithi119:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d69f13d827952bb07bb9b7973bca14f97d764d6f shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid aa240572-0159-11ef-bc93-c7b262605968 -- ceph orch daemon add osd smithi119:vg_nvme/lv_4 2024-04-23T10:19:20.933 INFO:teuthology.orchestra.run.smithi119.stderr:Inferring config /var/lib/ceph/aa240572-0159-11ef-bc93-c7b262605968/mon.b/config 2024-04-23T10:19:21.325 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 23 10:19:21 smithi119 ceph-mon[32608]: purged_snaps scrub starts 2024-04-23T10:19:21.325 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 23 10:19:21 smithi119 ceph-mon[32608]: purged_snaps scrub ok 2024-04-23T10:19:21.325 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 23 10:19:21 smithi119 ceph-mon[32608]: from='mgr.14152 172.21.15.114:0/1529706138' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-04-23T10:19:21.325 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 23 10:19:21 smithi119 ceph-mon[32608]: Detected new or changed devices on smithi114 2024-04-23T10:19:21.325 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 23 10:19:21 smithi119 ceph-mon[32608]: from='mgr.14152 172.21.15.114:0/1529706138' entity='mgr.x' 2024-04-23T10:19:21.325 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 23 10:19:21 smithi119 ceph-mon[32608]: from='mgr.14152 172.21.15.114:0/1529706138' entity='mgr.x' 2024-04-23T10:19:21.326 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 23 10:19:21 smithi119 ceph-mon[32608]: from='mgr.14152 172.21.15.114:0/1529706138' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-04-23T10:19:21.326 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 23 10:19:21 smithi119 ceph-mon[32608]: Adjusting osd_memory_target on smithi114 to 17088M 2024-04-23T10:19:21.326 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 23 10:19:21 smithi119 ceph-mon[32608]: from='mgr.14152 172.21.15.114:0/1529706138' entity='mgr.x' 2024-04-23T10:19:21.326 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 23 10:19:21 smithi119 ceph-mon[32608]: from='mgr.14152 172.21.15.114:0/1529706138' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-23T10:19:21.326 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 23 10:19:21 smithi119 ceph-mon[32608]: from='mgr.14152 172.21.15.114:0/1529706138' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-23T10:19:21.326 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 23 10:19:21 smithi119 ceph-mon[32608]: from='mgr.14152 172.21.15.114:0/1529706138' entity='mgr.x' 2024-04-23T10:19:21.326 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 23 10:19:21 smithi119 ceph-mon[32608]: from='mgr.14152 172.21.15.114:0/1529706138' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-04-23T10:19:21.578 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 23 10:19:21 smithi204 ceph-mon[32649]: purged_snaps scrub starts 2024-04-23T10:19:21.578 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 23 10:19:21 smithi204 ceph-mon[32649]: purged_snaps scrub ok 2024-04-23T10:19:21.578 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 23 10:19:21 smithi204 ceph-mon[32649]: from='mgr.14152 172.21.15.114:0/1529706138' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-04-23T10:19:21.578 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 23 10:19:21 smithi204 ceph-mon[32649]: Detected new or changed devices on smithi114 2024-04-23T10:19:21.579 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 23 10:19:21 smithi204 ceph-mon[32649]: from='mgr.14152 172.21.15.114:0/1529706138' entity='mgr.x' 2024-04-23T10:19:21.579 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 23 10:19:21 smithi204 ceph-mon[32649]: from='mgr.14152 172.21.15.114:0/1529706138' entity='mgr.x' 2024-04-23T10:19:21.579 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 23 10:19:21 smithi204 ceph-mon[32649]: from='mgr.14152 172.21.15.114:0/1529706138' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-04-23T10:19:21.579 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 23 10:19:21 smithi204 ceph-mon[32649]: Adjusting osd_memory_target on smithi114 to 17088M 2024-04-23T10:19:21.579 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 23 10:19:21 smithi204 ceph-mon[32649]: from='mgr.14152 172.21.15.114:0/1529706138' entity='mgr.x' 2024-04-23T10:19:21.579 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 23 10:19:21 smithi204 ceph-mon[32649]: from='mgr.14152 172.21.15.114:0/1529706138' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-23T10:19:21.579 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 23 10:19:21 smithi204 ceph-mon[32649]: from='mgr.14152 172.21.15.114:0/1529706138' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-23T10:19:21.579 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 23 10:19:21 smithi204 ceph-mon[32649]: from='mgr.14152 172.21.15.114:0/1529706138' entity='mgr.x' 2024-04-23T10:19:21.579 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 23 10:19:21 smithi204 ceph-mon[32649]: from='mgr.14152 172.21.15.114:0/1529706138' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-04-23T10:19:21.754 INFO:journalctl@ceph.mon.a.smithi114.stdout:Apr 23 10:19:21 smithi114 ceph-mon[29255]: purged_snaps scrub starts 2024-04-23T10:19:21.754 INFO:journalctl@ceph.mon.a.smithi114.stdout:Apr 23 10:19:21 smithi114 ceph-mon[29255]: purged_snaps scrub ok 2024-04-23T10:19:21.755 INFO:journalctl@ceph.mon.a.smithi114.stdout:Apr 23 10:19:21 smithi114 ceph-mon[29255]: from='mgr.14152 172.21.15.114:0/1529706138' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-04-23T10:19:21.755 INFO:journalctl@ceph.mon.a.smithi114.stdout:Apr 23 10:19:21 smithi114 ceph-mon[29255]: Detected new or changed devices on smithi114 2024-04-23T10:19:21.755 INFO:journalctl@ceph.mon.a.smithi114.stdout:Apr 23 10:19:21 smithi114 ceph-mon[29255]: from='mgr.14152 172.21.15.114:0/1529706138' entity='mgr.x' 2024-04-23T10:19:21.755 INFO:journalctl@ceph.mon.a.smithi114.stdout:Apr 23 10:19:21 smithi114 ceph-mon[29255]: from='mgr.14152 172.21.15.114:0/1529706138' entity='mgr.x' 2024-04-23T10:19:21.755 INFO:journalctl@ceph.mon.a.smithi114.stdout:Apr 23 10:19:21 smithi114 ceph-mon[29255]: from='mgr.14152 172.21.15.114:0/1529706138' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-04-23T10:19:21.755 INFO:journalctl@ceph.mon.a.smithi114.stdout:Apr 23 10:19:21 smithi114 ceph-mon[29255]: Adjusting osd_memory_target on smithi114 to 17088M 2024-04-23T10:19:21.755 INFO:journalctl@ceph.mon.a.smithi114.stdout:Apr 23 10:19:21 smithi114 ceph-mon[29255]: from='mgr.14152 172.21.15.114:0/1529706138' entity='mgr.x' 2024-04-23T10:19:21.755 INFO:journalctl@ceph.mon.a.smithi114.stdout:Apr 23 10:19:21 smithi114 ceph-mon[29255]: from='mgr.14152 172.21.15.114:0/1529706138' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-23T10:19:21.755 INFO:journalctl@ceph.mon.a.smithi114.stdout:Apr 23 10:19:21 smithi114 ceph-mon[29255]: from='mgr.14152 172.21.15.114:0/1529706138' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-23T10:19:21.755 INFO:journalctl@ceph.mon.a.smithi114.stdout:Apr 23 10:19:21 smithi114 ceph-mon[29255]: from='mgr.14152 172.21.15.114:0/1529706138' entity='mgr.x' 2024-04-23T10:19:21.755 INFO:journalctl@ceph.mon.a.smithi114.stdout:Apr 23 10:19:21 smithi114 ceph-mon[29255]: from='mgr.14152 172.21.15.114:0/1529706138' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-04-23T10:19:22.414 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 23 10:19:22 smithi119 ceph-mon[32608]: osd.0 [v2:172.21.15.114:6802/736366878,v1:172.21.15.114:6803/736366878] boot 2024-04-23T10:19:22.414 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 23 10:19:22 smithi119 ceph-mon[32608]: osdmap e8: 1 total, 1 up, 1 in 2024-04-23T10:19:22.414 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 23 10:19:22 smithi119 ceph-mon[32608]: from='mgr.14152 172.21.15.114:0/1529706138' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-04-23T10:19:22.415 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 23 10:19:22 smithi119 ceph-mon[32608]: pgmap v46: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-23T10:19:22.517 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-23T10:19:22.518+0000 7f1eda588700 1 -- 172.21.15.119:0/3750017323 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f1ed4102bb0 msgr2=0x7f1ed4103010 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T10:19:22.517 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-23T10:19:22.518+0000 7f1eda588700 1 --2- 172.21.15.119:0/3750017323 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f1ed4102bb0 0x7f1ed4103010 secure :-1 s=READY pgs=7 cs=0 l=1 rev1=1 crypto rx=0x7f1ec80099c0 tx=0x7f1ec8009cd0 comp rx=0 tx=0).stop 2024-04-23T10:19:22.518 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-23T10:19:22.518+0000 7f1eda588700 1 -- 172.21.15.119:0/3750017323 shutdown_connections 2024-04-23T10:19:22.518 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-23T10:19:22.518+0000 7f1eda588700 1 --2- 172.21.15.119:0/3750017323 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f1ed4103550 0x7f1ed4109e00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:19:22.518 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-23T10:19:22.518+0000 7f1eda588700 1 --2- 172.21.15.119:0/3750017323 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f1ed41019b0 0x7f1ed4101dd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:19:22.518 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-23T10:19:22.518+0000 7f1eda588700 1 --2- 172.21.15.119:0/3750017323 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f1ed4102bb0 0x7f1ed4103010 unknown :-1 s=CLOSED pgs=7 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:19:22.518 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-23T10:19:22.518+0000 7f1eda588700 1 -- 172.21.15.119:0/3750017323 >> 172.21.15.119:0/3750017323 conn(0x7f1ed40fcf70 msgr2=0x7f1ed40ff390 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T10:19:22.518 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-23T10:19:22.518+0000 7f1eda588700 1 -- 172.21.15.119:0/3750017323 shutdown_connections 2024-04-23T10:19:22.518 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-23T10:19:22.518+0000 7f1eda588700 1 -- 172.21.15.119:0/3750017323 wait complete. 2024-04-23T10:19:22.520 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-23T10:19:22.518+0000 7f1eda588700 1 Processor -- start 2024-04-23T10:19:22.521 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-23T10:19:22.518+0000 7f1eda588700 1 -- start start 2024-04-23T10:19:22.521 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-23T10:19:22.518+0000 7f1eda588700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f1ed41019b0 0x7f1ed406bd10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T10:19:22.521 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-23T10:19:22.518+0000 7f1eda588700 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f1ed4102bb0 0x7f1ed419d8a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T10:19:22.521 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-23T10:19:22.518+0000 7f1eda588700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f1ed4103550 0x7f1ed419fef0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T10:19:22.521 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-23T10:19:22.518+0000 7f1eda588700 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x7f1ed410d470 con 0x7f1ed4103550 2024-04-23T10:19:22.521 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-23T10:19:22.519+0000 7f1eda588700 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 -- 0x7f1ed410d2f0 con 0x7f1ed41019b0 2024-04-23T10:19:22.521 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-23T10:19:22.519+0000 7f1eda588700 1 -- --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_getmap magic: 0 -- 0x7f1ed410d5f0 con 0x7f1ed4102bb0 2024-04-23T10:19:22.521 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-23T10:19:22.519+0000 7f1ed3fff700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f1ed41019b0 0x7f1ed406bd10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T10:19:22.521 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-23T10:19:22.519+0000 7f1ed3fff700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f1ed41019b0 0x7f1ed406bd10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.119:3300/0 says I am v2:172.21.15.119:50824/0 (socket says 172.21.15.119:50824) 2024-04-23T10:19:22.522 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-23T10:19:22.519+0000 7f1ed3fff700 1 -- 172.21.15.119:0/2776983548 learned_addr learned my addr 172.21.15.119:0/2776983548 (peer_addr_for_me v2:172.21.15.119:0/0) 2024-04-23T10:19:22.522 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-23T10:19:22.519+0000 7f1ed37fe700 1 --2- 172.21.15.119:0/2776983548 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f1ed4102bb0 0x7f1ed419d8a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T10:19:22.522 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-23T10:19:22.519+0000 7f1ed8b25700 1 --2- 172.21.15.119:0/2776983548 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f1ed4103550 0x7f1ed419fef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T10:19:22.522 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-23T10:19:22.519+0000 7f1ed3fff700 1 -- 172.21.15.119:0/2776983548 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f1ed4102bb0 msgr2=0x7f1ed419d8a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T10:19:22.522 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-23T10:19:22.519+0000 7f1ed3fff700 1 --2- 172.21.15.119:0/2776983548 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f1ed4102bb0 0x7f1ed419d8a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:19:22.522 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-23T10:19:22.519+0000 7f1ed3fff700 1 -- 172.21.15.119:0/2776983548 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f1ed4103550 msgr2=0x7f1ed419fef0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T10:19:22.522 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-23T10:19:22.519+0000 7f1ed3fff700 1 --2- 172.21.15.119:0/2776983548 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f1ed4103550 0x7f1ed419fef0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:19:22.522 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-23T10:19:22.519+0000 7f1ed3fff700 1 -- 172.21.15.119:0/2776983548 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f1ed41a07c0 con 0x7f1ed41019b0 2024-04-23T10:19:22.522 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-23T10:19:22.519+0000 7f1ed3fff700 1 --2- 172.21.15.119:0/2776983548 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f1ed41019b0 0x7f1ed406bd10 secure :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0x7f1ec400c990 tx=0x7f1ec400cca0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T10:19:22.522 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-23T10:19:22.519+0000 7f1ed17fa700 1 -- 172.21.15.119:0/2776983548 <== mon.2 v2:172.21.15.119:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1ec4007940 con 0x7f1ed41019b0 2024-04-23T10:19:22.522 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-23T10:19:22.519+0000 7f1ed17fa700 1 -- 172.21.15.119:0/2776983548 <== mon.2 v2:172.21.15.119:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f1ec400f450 con 0x7f1ed41019b0 2024-04-23T10:19:22.522 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-23T10:19:22.519+0000 7f1eda588700 1 -- 172.21.15.119:0/2776983548 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f1ed41a0ab0 con 0x7f1ed41019b0 2024-04-23T10:19:22.523 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-23T10:19:22.519+0000 7f1ed17fa700 1 -- 172.21.15.119:0/2776983548 <== mon.2 v2:172.21.15.119:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1ec4018660 con 0x7f1ed41019b0 2024-04-23T10:19:22.523 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-23T10:19:22.519+0000 7f1eda588700 1 -- 172.21.15.119:0/2776983548 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f1ed41a0fc0 con 0x7f1ed41019b0 2024-04-23T10:19:22.523 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-23T10:19:22.520+0000 7f1eda588700 1 -- 172.21.15.119:0/2776983548 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f1ed4069730 con 0x7f1ed41019b0 2024-04-23T10:19:22.523 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-23T10:19:22.522+0000 7f1ed17fa700 1 -- 172.21.15.119:0/2776983548 <== mon.2 v2:172.21.15.119:3300/0 4 ==== mgrmap(e 15) ==== 138591+0+0 (secure 0 0 0) 0x7f1ec4007ae0 con 0x7f1ed41019b0 2024-04-23T10:19:22.524 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-23T10:19:22.522+0000 7f1ed17fa700 1 --2- 172.21.15.119:0/2776983548 >> [v2:172.21.15.114:6800/465333808,v1:172.21.15.114:6801/465333808] conn(0x7f1ebc0a3bb0 0x7f1ebc0a6070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T10:19:22.524 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-23T10:19:22.522+0000 7f1ed17fa700 1 -- 172.21.15.119:0/2776983548 <== mon.2 v2:172.21.15.119:3300/0 5 ==== osd_map(9..9 src has 1..9) ==== 1767+0+0 (secure 0 0 0) 0x7f1ec40ce950 con 0x7f1ed41019b0 2024-04-23T10:19:22.524 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-23T10:19:22.522+0000 7f1ed37fe700 1 --2- 172.21.15.119:0/2776983548 >> [v2:172.21.15.114:6800/465333808,v1:172.21.15.114:6801/465333808] conn(0x7f1ebc0a3bb0 0x7f1ebc0a6070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T10:19:22.524 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-23T10:19:22.523+0000 7f1ed37fe700 1 --2- 172.21.15.119:0/2776983548 >> [v2:172.21.15.114:6800/465333808,v1:172.21.15.114:6801/465333808] conn(0x7f1ebc0a3bb0 0x7f1ebc0a6070 secure :-1 s=READY pgs=54 cs=0 l=1 rev1=1 crypto rx=0x7f1ec8009990 tx=0x7f1ec8003880 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T10:19:22.525 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-23T10:19:22.526+0000 7f1ed17fa700 1 -- 172.21.15.119:0/2776983548 <== mon.2 v2:172.21.15.119:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f1ec4014030 con 0x7f1ed41019b0 2024-04-23T10:19:22.578 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 23 10:19:22 smithi204 ceph-mon[32649]: osd.0 [v2:172.21.15.114:6802/736366878,v1:172.21.15.114:6803/736366878] boot 2024-04-23T10:19:22.578 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 23 10:19:22 smithi204 ceph-mon[32649]: osdmap e8: 1 total, 1 up, 1 in 2024-04-23T10:19:22.578 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 23 10:19:22 smithi204 ceph-mon[32649]: from='mgr.14152 172.21.15.114:0/1529706138' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-04-23T10:19:22.578 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 23 10:19:22 smithi204 ceph-mon[32649]: pgmap v46: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-23T10:19:22.729 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-23T10:19:22.729+0000 7f1eda588700 1 -- 172.21.15.119:0/2776983548 --> [v2:172.21.15.114:6800/465333808,v1:172.21.15.114:6801/465333808] -- mgr_command(tid 0: {"prefix": "orch daemon add osd", "svc_arg": "smithi119:vg_nvme/lv_4", "target": ["mon-mgr", ""]}) -- 0x7f1ed4062e80 con 0x7f1ebc0a3bb0 2024-04-23T10:19:22.754 INFO:journalctl@ceph.mon.a.smithi114.stdout:Apr 23 10:19:22 smithi114 ceph-mon[29255]: osd.0 [v2:172.21.15.114:6802/736366878,v1:172.21.15.114:6803/736366878] boot 2024-04-23T10:19:22.754 INFO:journalctl@ceph.mon.a.smithi114.stdout:Apr 23 10:19:22 smithi114 ceph-mon[29255]: osdmap e8: 1 total, 1 up, 1 in 2024-04-23T10:19:22.754 INFO:journalctl@ceph.mon.a.smithi114.stdout:Apr 23 10:19:22 smithi114 ceph-mon[29255]: from='mgr.14152 172.21.15.114:0/1529706138' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-04-23T10:19:22.754 INFO:journalctl@ceph.mon.a.smithi114.stdout:Apr 23 10:19:22 smithi114 ceph-mon[29255]: pgmap v46: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-23T10:19:23.558 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 23 10:19:23 smithi119 ceph-mon[32608]: osdmap e9: 1 total, 1 up, 1 in 2024-04-23T10:19:23.558 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 23 10:19:23 smithi119 ceph-mon[32608]: from='client.24143 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi119:vg_nvme/lv_4", "target": ["mon-mgr", ""]}]: dispatch 2024-04-23T10:19:23.559 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 23 10:19:23 smithi119 ceph-mon[32608]: from='mgr.14152 172.21.15.114:0/1529706138' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-04-23T10:19:23.559 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 23 10:19:23 smithi119 ceph-mon[32608]: from='mgr.14152 172.21.15.114:0/1529706138' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-04-23T10:19:23.559 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 23 10:19:23 smithi119 ceph-mon[32608]: from='mgr.14152 172.21.15.114:0/1529706138' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-23T10:19:23.578 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 23 10:19:23 smithi204 ceph-mon[32649]: osdmap e9: 1 total, 1 up, 1 in 2024-04-23T10:19:23.578 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 23 10:19:23 smithi204 ceph-mon[32649]: from='client.24143 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi119:vg_nvme/lv_4", "target": ["mon-mgr", ""]}]: dispatch 2024-04-23T10:19:23.578 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 23 10:19:23 smithi204 ceph-mon[32649]: from='mgr.14152 172.21.15.114:0/1529706138' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-04-23T10:19:23.578 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 23 10:19:23 smithi204 ceph-mon[32649]: from='mgr.14152 172.21.15.114:0/1529706138' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-04-23T10:19:23.578 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 23 10:19:23 smithi204 ceph-mon[32649]: from='mgr.14152 172.21.15.114:0/1529706138' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-23T10:19:23.754 INFO:journalctl@ceph.mon.a.smithi114.stdout:Apr 23 10:19:23 smithi114 ceph-mon[29255]: osdmap e9: 1 total, 1 up, 1 in 2024-04-23T10:19:23.754 INFO:journalctl@ceph.mon.a.smithi114.stdout:Apr 23 10:19:23 smithi114 ceph-mon[29255]: from='client.24143 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi119:vg_nvme/lv_4", "target": ["mon-mgr", ""]}]: dispatch 2024-04-23T10:19:23.754 INFO:journalctl@ceph.mon.a.smithi114.stdout:Apr 23 10:19:23 smithi114 ceph-mon[29255]: from='mgr.14152 172.21.15.114:0/1529706138' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-04-23T10:19:23.754 INFO:journalctl@ceph.mon.a.smithi114.stdout:Apr 23 10:19:23 smithi114 ceph-mon[29255]: from='mgr.14152 172.21.15.114:0/1529706138' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-04-23T10:19:23.754 INFO:journalctl@ceph.mon.a.smithi114.stdout:Apr 23 10:19:23 smithi114 ceph-mon[29255]: from='mgr.14152 172.21.15.114:0/1529706138' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-23T10:19:24.428 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 23 10:19:24 smithi119 ceph-mon[32608]: pgmap v48: 0 pgs: ; 0 B data, 31 MiB used, 89 GiB / 89 GiB avail 2024-04-23T10:19:24.578 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 23 10:19:24 smithi204 ceph-mon[32649]: pgmap v48: 0 pgs: ; 0 B data, 31 MiB used, 89 GiB / 89 GiB avail 2024-04-23T10:19:24.754 INFO:journalctl@ceph.mon.a.smithi114.stdout:Apr 23 10:19:24 smithi114 ceph-mon[29255]: pgmap v48: 0 pgs: ; 0 B data, 31 MiB used, 89 GiB / 89 GiB avail 2024-04-23T10:19:25.578 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 23 10:19:25 smithi204 ceph-mon[32649]: from='client.? 172.21.15.119:0/53717918' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "e279a1e6-2702-472a-930a-e08ece76a861"}]: dispatch 2024-04-23T10:19:25.578 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 23 10:19:25 smithi204 ceph-mon[32649]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "e279a1e6-2702-472a-930a-e08ece76a861"}]: dispatch 2024-04-23T10:19:25.579 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 23 10:19:25 smithi204 ceph-mon[32649]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "e279a1e6-2702-472a-930a-e08ece76a861"}]': finished 2024-04-23T10:19:25.579 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 23 10:19:25 smithi204 ceph-mon[32649]: osdmap e10: 2 total, 1 up, 2 in 2024-04-23T10:19:25.579 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 23 10:19:25 smithi204 ceph-mon[32649]: from='mgr.14152 172.21.15.114:0/1529706138' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-04-23T10:19:25.589 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 23 10:19:25 smithi119 ceph-mon[32608]: from='client.? 172.21.15.119:0/53717918' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "e279a1e6-2702-472a-930a-e08ece76a861"}]: dispatch 2024-04-23T10:19:25.589 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 23 10:19:25 smithi119 ceph-mon[32608]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "e279a1e6-2702-472a-930a-e08ece76a861"}]: dispatch 2024-04-23T10:19:25.589 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 23 10:19:25 smithi119 ceph-mon[32608]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "e279a1e6-2702-472a-930a-e08ece76a861"}]': finished 2024-04-23T10:19:25.589 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 23 10:19:25 smithi119 ceph-mon[32608]: osdmap e10: 2 total, 1 up, 2 in 2024-04-23T10:19:25.590 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 23 10:19:25 smithi119 ceph-mon[32608]: from='mgr.14152 172.21.15.114:0/1529706138' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-04-23T10:19:25.754 INFO:journalctl@ceph.mon.a.smithi114.stdout:Apr 23 10:19:25 smithi114 ceph-mon[29255]: from='client.? 172.21.15.119:0/53717918' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "e279a1e6-2702-472a-930a-e08ece76a861"}]: dispatch 2024-04-23T10:19:25.754 INFO:journalctl@ceph.mon.a.smithi114.stdout:Apr 23 10:19:25 smithi114 ceph-mon[29255]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "e279a1e6-2702-472a-930a-e08ece76a861"}]: dispatch 2024-04-23T10:19:25.754 INFO:journalctl@ceph.mon.a.smithi114.stdout:Apr 23 10:19:25 smithi114 ceph-mon[29255]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "e279a1e6-2702-472a-930a-e08ece76a861"}]': finished 2024-04-23T10:19:25.755 INFO:journalctl@ceph.mon.a.smithi114.stdout:Apr 23 10:19:25 smithi114 ceph-mon[29255]: osdmap e10: 2 total, 1 up, 2 in 2024-04-23T10:19:25.755 INFO:journalctl@ceph.mon.a.smithi114.stdout:Apr 23 10:19:25 smithi114 ceph-mon[29255]: from='mgr.14152 172.21.15.114:0/1529706138' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-04-23T10:19:26.563 INFO:journalctl@ceph.mon.a.smithi114.stdout:Apr 23 10:19:26 smithi114 ceph-mon[29255]: from='client.? 172.21.15.119:0/97324799' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-04-23T10:19:26.563 INFO:journalctl@ceph.mon.a.smithi114.stdout:Apr 23 10:19:26 smithi114 ceph-mon[29255]: pgmap v50: 0 pgs: ; 0 B data, 31 MiB used, 89 GiB / 89 GiB avail 2024-04-23T10:19:26.578 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 23 10:19:26 smithi204 ceph-mon[32649]: from='client.? 172.21.15.119:0/97324799' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-04-23T10:19:26.578 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 23 10:19:26 smithi204 ceph-mon[32649]: pgmap v50: 0 pgs: ; 0 B data, 31 MiB used, 89 GiB / 89 GiB avail 2024-04-23T10:19:26.599 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 23 10:19:26 smithi119 ceph-mon[32608]: from='client.? 172.21.15.119:0/97324799' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-04-23T10:19:26.600 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 23 10:19:26 smithi119 ceph-mon[32608]: pgmap v50: 0 pgs: ; 0 B data, 31 MiB used, 89 GiB / 89 GiB avail 2024-04-23T10:19:29.242 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 23 10:19:28 smithi119 ceph-mon[32608]: pgmap v51: 0 pgs: ; 0 B data, 31 MiB used, 89 GiB / 89 GiB avail 2024-04-23T10:19:29.254 INFO:journalctl@ceph.mon.a.smithi114.stdout:Apr 23 10:19:28 smithi114 ceph-mon[29255]: pgmap v51: 0 pgs: ; 0 B data, 31 MiB used, 89 GiB / 89 GiB avail 2024-04-23T10:19:29.328 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 23 10:19:28 smithi204 ceph-mon[32649]: pgmap v51: 0 pgs: ; 0 B data, 31 MiB used, 89 GiB / 89 GiB avail 2024-04-23T10:19:31.241 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 23 10:19:30 smithi119 ceph-mon[32608]: pgmap v52: 0 pgs: ; 0 B data, 31 MiB used, 89 GiB / 89 GiB avail 2024-04-23T10:19:31.254 INFO:journalctl@ceph.mon.a.smithi114.stdout:Apr 23 10:19:30 smithi114 ceph-mon[29255]: pgmap v52: 0 pgs: ; 0 B data, 31 MiB used, 89 GiB / 89 GiB avail 2024-04-23T10:19:31.328 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 23 10:19:30 smithi204 ceph-mon[32649]: pgmap v52: 0 pgs: ; 0 B data, 31 MiB used, 89 GiB / 89 GiB avail 2024-04-23T10:19:33.237 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 23 10:19:32 smithi119 ceph-mon[32608]: pgmap v53: 0 pgs: ; 0 B data, 31 MiB used, 89 GiB / 89 GiB avail 2024-04-23T10:19:33.238 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 23 10:19:32 smithi119 ceph-mon[32608]: from='mgr.14152 172.21.15.114:0/1529706138' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.1"}]: dispatch 2024-04-23T10:19:33.238 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 23 10:19:32 smithi119 ceph-mon[32608]: from='mgr.14152 172.21.15.114:0/1529706138' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-23T10:19:33.254 INFO:journalctl@ceph.mon.a.smithi114.stdout:Apr 23 10:19:32 smithi114 ceph-mon[29255]: pgmap v53: 0 pgs: ; 0 B data, 31 MiB used, 89 GiB / 89 GiB avail 2024-04-23T10:19:33.254 INFO:journalctl@ceph.mon.a.smithi114.stdout:Apr 23 10:19:32 smithi114 ceph-mon[29255]: from='mgr.14152 172.21.15.114:0/1529706138' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.1"}]: dispatch 2024-04-23T10:19:33.254 INFO:journalctl@ceph.mon.a.smithi114.stdout:Apr 23 10:19:32 smithi114 ceph-mon[29255]: from='mgr.14152 172.21.15.114:0/1529706138' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-23T10:19:33.328 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 23 10:19:32 smithi204 ceph-mon[32649]: pgmap v53: 0 pgs: ; 0 B data, 31 MiB used, 89 GiB / 89 GiB avail 2024-04-23T10:19:33.328 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 23 10:19:32 smithi204 ceph-mon[32649]: from='mgr.14152 172.21.15.114:0/1529706138' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.1"}]: dispatch 2024-04-23T10:19:33.328 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 23 10:19:32 smithi204 ceph-mon[32649]: from='mgr.14152 172.21.15.114:0/1529706138' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-23T10:19:34.167 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 23 10:19:33 smithi119 ceph-mon[32608]: Deploying daemon osd.1 on smithi119 2024-04-23T10:19:34.254 INFO:journalctl@ceph.mon.a.smithi114.stdout:Apr 23 10:19:33 smithi114 ceph-mon[29255]: Deploying daemon osd.1 on smithi119 2024-04-23T10:19:34.328 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 23 10:19:33 smithi204 ceph-mon[32649]: Deploying daemon osd.1 on smithi119 2024-04-23T10:19:35.210 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 23 10:19:34 smithi119 ceph-mon[32608]: pgmap v54: 0 pgs: ; 0 B data, 31 MiB used, 89 GiB / 89 GiB avail 2024-04-23T10:19:35.254 INFO:journalctl@ceph.mon.a.smithi114.stdout:Apr 23 10:19:34 smithi114 ceph-mon[29255]: pgmap v54: 0 pgs: ; 0 B data, 31 MiB used, 89 GiB / 89 GiB avail 2024-04-23T10:19:35.328 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 23 10:19:34 smithi204 ceph-mon[32649]: pgmap v54: 0 pgs: ; 0 B data, 31 MiB used, 89 GiB / 89 GiB avail 2024-04-23T10:19:37.242 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 23 10:19:36 smithi119 ceph-mon[32608]: pgmap v55: 0 pgs: ; 0 B data, 31 MiB used, 89 GiB / 89 GiB avail 2024-04-23T10:19:37.254 INFO:journalctl@ceph.mon.a.smithi114.stdout:Apr 23 10:19:36 smithi114 ceph-mon[29255]: pgmap v55: 0 pgs: ; 0 B data, 31 MiB used, 89 GiB / 89 GiB avail 2024-04-23T10:19:37.328 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 23 10:19:36 smithi204 ceph-mon[32649]: pgmap v55: 0 pgs: ; 0 B data, 31 MiB used, 89 GiB / 89 GiB avail 2024-04-23T10:19:38.992 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 23 10:19:38 smithi119 ceph-mon[32608]: from='mgr.14152 172.21.15.114:0/1529706138' entity='mgr.x' 2024-04-23T10:19:38.992 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 23 10:19:38 smithi119 ceph-mon[32608]: from='mgr.14152 172.21.15.114:0/1529706138' entity='mgr.x' 2024-04-23T10:19:38.992 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 23 10:19:38 smithi119 ceph-mon[32608]: from='mgr.14152 172.21.15.114:0/1529706138' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-23T10:19:38.992 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 23 10:19:38 smithi119 ceph-mon[32608]: pgmap v56: 0 pgs: ; 0 B data, 31 MiB used, 89 GiB / 89 GiB avail 2024-04-23T10:19:39.004 INFO:journalctl@ceph.mon.a.smithi114.stdout:Apr 23 10:19:38 smithi114 ceph-mon[29255]: from='mgr.14152 172.21.15.114:0/1529706138' entity='mgr.x' 2024-04-23T10:19:39.004 INFO:journalctl@ceph.mon.a.smithi114.stdout:Apr 23 10:19:38 smithi114 ceph-mon[29255]: from='mgr.14152 172.21.15.114:0/1529706138' entity='mgr.x' 2024-04-23T10:19:39.004 INFO:journalctl@ceph.mon.a.smithi114.stdout:Apr 23 10:19:38 smithi114 ceph-mon[29255]: from='mgr.14152 172.21.15.114:0/1529706138' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-23T10:19:39.004 INFO:journalctl@ceph.mon.a.smithi114.stdout:Apr 23 10:19:38 smithi114 ceph-mon[29255]: pgmap v56: 0 pgs: ; 0 B data, 31 MiB used, 89 GiB / 89 GiB avail 2024-04-23T10:19:39.078 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 23 10:19:38 smithi204 ceph-mon[32649]: from='mgr.14152 172.21.15.114:0/1529706138' entity='mgr.x' 2024-04-23T10:19:39.078 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 23 10:19:38 smithi204 ceph-mon[32649]: from='mgr.14152 172.21.15.114:0/1529706138' entity='mgr.x' 2024-04-23T10:19:39.078 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 23 10:19:38 smithi204 ceph-mon[32649]: from='mgr.14152 172.21.15.114:0/1529706138' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-23T10:19:39.078 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 23 10:19:38 smithi204 ceph-mon[32649]: pgmap v56: 0 pgs: ; 0 B data, 31 MiB used, 89 GiB / 89 GiB avail 2024-04-23T10:19:41.236 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 23 10:19:40 smithi119 ceph-mon[32608]: pgmap v57: 0 pgs: ; 0 B data, 31 MiB used, 89 GiB / 89 GiB avail 2024-04-23T10:19:41.236 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 23 10:19:40 smithi119 ceph-mon[32608]: from='mgr.14152 172.21.15.114:0/1529706138' entity='mgr.x' 2024-04-23T10:19:41.236 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 23 10:19:40 smithi119 ceph-mon[32608]: from='mgr.14152 172.21.15.114:0/1529706138' entity='mgr.x' 2024-04-23T10:19:41.236 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 23 10:19:40 smithi119 ceph-mon[32608]: from='mgr.14152 172.21.15.114:0/1529706138' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-23T10:19:41.236 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 23 10:19:40 smithi119 ceph-mon[32608]: from='mgr.14152 172.21.15.114:0/1529706138' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-23T10:19:41.236 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 23 10:19:40 smithi119 ceph-mon[32608]: from='mgr.14152 172.21.15.114:0/1529706138' entity='mgr.x' 2024-04-23T10:19:41.254 INFO:journalctl@ceph.mon.a.smithi114.stdout:Apr 23 10:19:40 smithi114 ceph-mon[29255]: pgmap v57: 0 pgs: ; 0 B data, 31 MiB used, 89 GiB / 89 GiB avail 2024-04-23T10:19:41.254 INFO:journalctl@ceph.mon.a.smithi114.stdout:Apr 23 10:19:40 smithi114 ceph-mon[29255]: from='mgr.14152 172.21.15.114:0/1529706138' entity='mgr.x' 2024-04-23T10:19:41.254 INFO:journalctl@ceph.mon.a.smithi114.stdout:Apr 23 10:19:40 smithi114 ceph-mon[29255]: from='mgr.14152 172.21.15.114:0/1529706138' entity='mgr.x' 2024-04-23T10:19:41.254 INFO:journalctl@ceph.mon.a.smithi114.stdout:Apr 23 10:19:40 smithi114 ceph-mon[29255]: from='mgr.14152 172.21.15.114:0/1529706138' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-23T10:19:41.254 INFO:journalctl@ceph.mon.a.smithi114.stdout:Apr 23 10:19:40 smithi114 ceph-mon[29255]: from='mgr.14152 172.21.15.114:0/1529706138' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-23T10:19:41.254 INFO:journalctl@ceph.mon.a.smithi114.stdout:Apr 23 10:19:40 smithi114 ceph-mon[29255]: from='mgr.14152 172.21.15.114:0/1529706138' entity='mgr.x' 2024-04-23T10:19:41.297 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-23T10:19:41.298+0000 7f1ed17fa700 1 -- 172.21.15.119:0/2776983548 <== mgr.14152 v2:172.21.15.114:6800/465333808 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+37 (secure 0 0 0) 0x7f1ed4062e80 con 0x7f1ebc0a3bb0 2024-04-23T10:19:41.297 INFO:teuthology.orchestra.run.smithi119.stdout:Created osd(s) 1 on host 'smithi119' 2024-04-23T10:19:41.299 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-23T10:19:41.300+0000 7f1eda588700 1 -- 172.21.15.119:0/2776983548 >> [v2:172.21.15.114:6800/465333808,v1:172.21.15.114:6801/465333808] conn(0x7f1ebc0a3bb0 msgr2=0x7f1ebc0a6070 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T10:19:41.299 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-23T10:19:41.300+0000 7f1eda588700 1 --2- 172.21.15.119:0/2776983548 >> [v2:172.21.15.114:6800/465333808,v1:172.21.15.114:6801/465333808] conn(0x7f1ebc0a3bb0 0x7f1ebc0a6070 secure :-1 s=READY pgs=54 cs=0 l=1 rev1=1 crypto rx=0x7f1ec8009990 tx=0x7f1ec8003880 comp rx=0 tx=0).stop 2024-04-23T10:19:41.299 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-23T10:19:41.300+0000 7f1eda588700 1 -- 172.21.15.119:0/2776983548 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f1ed41019b0 msgr2=0x7f1ed406bd10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T10:19:41.299 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-23T10:19:41.300+0000 7f1eda588700 1 --2- 172.21.15.119:0/2776983548 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f1ed41019b0 0x7f1ed406bd10 secure :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0x7f1ec400c990 tx=0x7f1ec400cca0 comp rx=0 tx=0).stop 2024-04-23T10:19:41.299 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-23T10:19:41.300+0000 7f1eda588700 1 -- 172.21.15.119:0/2776983548 shutdown_connections 2024-04-23T10:19:41.299 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-23T10:19:41.300+0000 7f1eda588700 1 --2- 172.21.15.119:0/2776983548 >> [v2:172.21.15.114:6800/465333808,v1:172.21.15.114:6801/465333808] conn(0x7f1ebc0a3bb0 0x7f1ebc0a6070 unknown :-1 s=CLOSED pgs=54 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:19:41.299 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-23T10:19:41.300+0000 7f1eda588700 1 --2- 172.21.15.119:0/2776983548 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f1ed41019b0 0x7f1ed406bd10 unknown :-1 s=CLOSED pgs=8 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:19:41.299 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-23T10:19:41.300+0000 7f1eda588700 1 --2- 172.21.15.119:0/2776983548 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f1ed4103550 0x7f1ed419fef0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:19:41.300 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-23T10:19:41.300+0000 7f1eda588700 1 --2- 172.21.15.119:0/2776983548 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f1ed4102bb0 0x7f1ed419d8a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:19:41.300 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-23T10:19:41.300+0000 7f1eda588700 1 -- 172.21.15.119:0/2776983548 >> 172.21.15.119:0/2776983548 conn(0x7f1ed40fcf70 msgr2=0x7f1ed4105f30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T10:19:41.300 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-23T10:19:41.300+0000 7f1eda588700 1 -- 172.21.15.119:0/2776983548 shutdown_connections 2024-04-23T10:19:41.300 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-23T10:19:41.300+0000 7f1eda588700 1 -- 172.21.15.119:0/2776983548 wait complete. 2024-04-23T10:19:41.328 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 23 10:19:40 smithi204 ceph-mon[32649]: pgmap v57: 0 pgs: ; 0 B data, 31 MiB used, 89 GiB / 89 GiB avail 2024-04-23T10:19:41.328 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 23 10:19:40 smithi204 ceph-mon[32649]: from='mgr.14152 172.21.15.114:0/1529706138' entity='mgr.x' 2024-04-23T10:19:41.328 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 23 10:19:40 smithi204 ceph-mon[32649]: from='mgr.14152 172.21.15.114:0/1529706138' entity='mgr.x' 2024-04-23T10:19:41.328 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 23 10:19:40 smithi204 ceph-mon[32649]: from='mgr.14152 172.21.15.114:0/1529706138' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-23T10:19:41.328 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 23 10:19:40 smithi204 ceph-mon[32649]: from='mgr.14152 172.21.15.114:0/1529706138' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-23T10:19:41.328 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 23 10:19:40 smithi204 ceph-mon[32649]: from='mgr.14152 172.21.15.114:0/1529706138' entity='mgr.x' 2024-04-23T10:19:41.975 DEBUG:teuthology.orchestra.run.smithi119:osd.1> sudo journalctl -f -n 0 -u ceph-aa240572-0159-11ef-bc93-c7b262605968@osd.1.service 2024-04-23T10:19:41.978 INFO:tasks.cephadm:Deploying osd.2 on smithi204 with /dev/vg_nvme/lv_4... 2024-04-23T10:19:41.978 DEBUG:teuthology.orchestra.run.smithi204:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d69f13d827952bb07bb9b7973bca14f97d764d6f shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid aa240572-0159-11ef-bc93-c7b262605968 -- ceph-volume lvm zap /dev/vg_nvme/lv_4 2024-04-23T10:19:42.221 INFO:teuthology.orchestra.run.smithi204.stderr:Inferring config /var/lib/ceph/aa240572-0159-11ef-bc93-c7b262605968/mon.c/config 2024-04-23T10:19:42.492 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 23 10:19:42 smithi119 ceph-mon[32608]: from='mgr.14152 172.21.15.114:0/1529706138' entity='mgr.x' 2024-04-23T10:19:42.492 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 23 10:19:42 smithi119 ceph-mon[32608]: from='mgr.14152 172.21.15.114:0/1529706138' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-23T10:19:42.492 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 23 10:19:42 smithi119 ceph-mon[32608]: from='mgr.14152 172.21.15.114:0/1529706138' entity='mgr.x' 2024-04-23T10:19:42.492 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 23 10:19:42 smithi119 ceph-mon[32608]: from='osd.1 [v2:172.21.15.119:6800/1133540779,v1:172.21.15.119:6801/1133540779]' entity='osd.1' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]: dispatch 2024-04-23T10:19:42.492 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 23 10:19:42 smithi119 ceph-mon[32608]: from='osd.1 ' entity='osd.1' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]: dispatch 2024-04-23T10:19:42.492 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 23 10:19:42 smithi119 ceph-mon[32608]: pgmap v58: 0 pgs: ; 0 B data, 31 MiB used, 89 GiB / 89 GiB avail 2024-04-23T10:19:42.559 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 23 10:19:42 smithi204 ceph-mon[32649]: from='mgr.14152 172.21.15.114:0/1529706138' entity='mgr.x' 2024-04-23T10:19:42.559 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 23 10:19:42 smithi204 ceph-mon[32649]: from='mgr.14152 172.21.15.114:0/1529706138' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-23T10:19:42.559 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 23 10:19:42 smithi204 ceph-mon[32649]: from='mgr.14152 172.21.15.114:0/1529706138' entity='mgr.x' 2024-04-23T10:19:42.559 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 23 10:19:42 smithi204 ceph-mon[32649]: from='osd.1 [v2:172.21.15.119:6800/1133540779,v1:172.21.15.119:6801/1133540779]' entity='osd.1' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]: dispatch 2024-04-23T10:19:42.559 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 23 10:19:42 smithi204 ceph-mon[32649]: from='osd.1 ' entity='osd.1' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]: dispatch 2024-04-23T10:19:42.559 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 23 10:19:42 smithi204 ceph-mon[32649]: pgmap v58: 0 pgs: ; 0 B data, 31 MiB used, 89 GiB / 89 GiB avail 2024-04-23T10:19:42.754 INFO:journalctl@ceph.mon.a.smithi114.stdout:Apr 23 10:19:42 smithi114 ceph-mon[29255]: from='mgr.14152 172.21.15.114:0/1529706138' entity='mgr.x' 2024-04-23T10:19:42.754 INFO:journalctl@ceph.mon.a.smithi114.stdout:Apr 23 10:19:42 smithi114 ceph-mon[29255]: from='mgr.14152 172.21.15.114:0/1529706138' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-23T10:19:42.755 INFO:journalctl@ceph.mon.a.smithi114.stdout:Apr 23 10:19:42 smithi114 ceph-mon[29255]: from='mgr.14152 172.21.15.114:0/1529706138' entity='mgr.x' 2024-04-23T10:19:42.755 INFO:journalctl@ceph.mon.a.smithi114.stdout:Apr 23 10:19:42 smithi114 ceph-mon[29255]: from='osd.1 [v2:172.21.15.119:6800/1133540779,v1:172.21.15.119:6801/1133540779]' entity='osd.1' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]: dispatch 2024-04-23T10:19:42.755 INFO:journalctl@ceph.mon.a.smithi114.stdout:Apr 23 10:19:42 smithi114 ceph-mon[29255]: from='osd.1 ' entity='osd.1' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]: dispatch 2024-04-23T10:19:42.755 INFO:journalctl@ceph.mon.a.smithi114.stdout:Apr 23 10:19:42 smithi114 ceph-mon[29255]: pgmap v58: 0 pgs: ; 0 B data, 31 MiB used, 89 GiB / 89 GiB avail 2024-04-23T10:19:43.492 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 23 10:19:43 smithi119 ceph-mon[32608]: from='osd.1 ' entity='osd.1' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]': finished 2024-04-23T10:19:43.492 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 23 10:19:43 smithi119 ceph-mon[32608]: from='osd.1 [v2:172.21.15.119:6800/1133540779,v1:172.21.15.119:6801/1133540779]' entity='osd.1' cmd=[{"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi119", "root=default"]}]: dispatch 2024-04-23T10:19:43.493 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 23 10:19:43 smithi119 ceph-mon[32608]: osdmap e11: 2 total, 1 up, 2 in 2024-04-23T10:19:43.493 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 23 10:19:43 smithi119 ceph-mon[32608]: from='mgr.14152 172.21.15.114:0/1529706138' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-04-23T10:19:43.493 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 23 10:19:43 smithi119 ceph-mon[32608]: from='osd.1 ' entity='osd.1' cmd=[{"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi119", "root=default"]}]: dispatch 2024-04-23T10:19:43.493 INFO:journalctl@ceph.osd.1.smithi119.stdout:Apr 23 10:19:43 smithi119 ceph-aa240572-0159-11ef-bc93-c7b262605968-osd-1[37220]: 2024-04-23T10:19:43.292+0000 7fefbf916700 -1 osd.1 0 waiting for initial osdmap 2024-04-23T10:19:43.493 INFO:journalctl@ceph.osd.1.smithi119.stdout:Apr 23 10:19:43 smithi119 ceph-aa240572-0159-11ef-bc93-c7b262605968-osd-1[37220]: 2024-04-23T10:19:43.299+0000 7fefbb709700 -1 osd.1 12 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-04-23T10:19:43.506 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 23 10:19:43 smithi204 ceph-mon[32649]: from='osd.1 ' entity='osd.1' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]': finished 2024-04-23T10:19:43.506 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 23 10:19:43 smithi204 ceph-mon[32649]: from='osd.1 [v2:172.21.15.119:6800/1133540779,v1:172.21.15.119:6801/1133540779]' entity='osd.1' cmd=[{"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi119", "root=default"]}]: dispatch 2024-04-23T10:19:43.506 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 23 10:19:43 smithi204 ceph-mon[32649]: osdmap e11: 2 total, 1 up, 2 in 2024-04-23T10:19:43.506 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 23 10:19:43 smithi204 ceph-mon[32649]: from='mgr.14152 172.21.15.114:0/1529706138' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-04-23T10:19:43.506 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 23 10:19:43 smithi204 ceph-mon[32649]: from='osd.1 ' entity='osd.1' cmd=[{"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi119", "root=default"]}]: dispatch 2024-04-23T10:19:43.754 INFO:journalctl@ceph.mon.a.smithi114.stdout:Apr 23 10:19:43 smithi114 ceph-mon[29255]: from='osd.1 ' entity='osd.1' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]': finished 2024-04-23T10:19:43.754 INFO:journalctl@ceph.mon.a.smithi114.stdout:Apr 23 10:19:43 smithi114 ceph-mon[29255]: from='osd.1 [v2:172.21.15.119:6800/1133540779,v1:172.21.15.119:6801/1133540779]' entity='osd.1' cmd=[{"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi119", "root=default"]}]: dispatch 2024-04-23T10:19:43.754 INFO:journalctl@ceph.mon.a.smithi114.stdout:Apr 23 10:19:43 smithi114 ceph-mon[29255]: osdmap e11: 2 total, 1 up, 2 in 2024-04-23T10:19:43.754 INFO:journalctl@ceph.mon.a.smithi114.stdout:Apr 23 10:19:43 smithi114 ceph-mon[29255]: from='mgr.14152 172.21.15.114:0/1529706138' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-04-23T10:19:43.754 INFO:journalctl@ceph.mon.a.smithi114.stdout:Apr 23 10:19:43 smithi114 ceph-mon[29255]: from='osd.1 ' entity='osd.1' cmd=[{"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi119", "root=default"]}]: dispatch 2024-04-23T10:19:44.090 INFO:teuthology.orchestra.run.smithi204.stderr:--> Zapping: /dev/vg_nvme/lv_4 2024-04-23T10:19:44.090 INFO:teuthology.orchestra.run.smithi204.stderr:Running command: /usr/bin/dd if=/dev/zero of=/dev/vg_nvme/lv_4 bs=1M count=10 conv=fsync 2024-04-23T10:19:44.090 INFO:teuthology.orchestra.run.smithi204.stderr: stderr: 10+0 records in 2024-04-23T10:19:44.090 INFO:teuthology.orchestra.run.smithi204.stderr:10+0 records out 2024-04-23T10:19:44.091 INFO:teuthology.orchestra.run.smithi204.stderr: stderr: 10485760 bytes (10 MB, 10 MiB) copied, 0.0185469 s, 565 MB/s 2024-04-23T10:19:44.091 INFO:teuthology.orchestra.run.smithi204.stderr:--> Zapping successful for: 2024-04-23T10:19:44.385 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 23 10:19:44 smithi204 ceph-mon[32649]: from='osd.1 ' entity='osd.1' cmd='[{"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi119", "root=default"]}]': finished 2024-04-23T10:19:44.385 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 23 10:19:44 smithi204 ceph-mon[32649]: osdmap e12: 2 total, 1 up, 2 in 2024-04-23T10:19:44.385 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 23 10:19:44 smithi204 ceph-mon[32649]: from='mgr.14152 172.21.15.114:0/1529706138' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-04-23T10:19:44.385 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 23 10:19:44 smithi204 ceph-mon[32649]: from='mgr.14152 172.21.15.114:0/1529706138' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-04-23T10:19:44.386 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 23 10:19:44 smithi204 ceph-mon[32649]: pgmap v61: 0 pgs: ; 0 B data, 31 MiB used, 89 GiB / 89 GiB avail 2024-04-23T10:19:44.610 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 23 10:19:44 smithi119 ceph-mon[32608]: from='osd.1 ' entity='osd.1' cmd='[{"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi119", "root=default"]}]': finished 2024-04-23T10:19:44.610 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 23 10:19:44 smithi119 ceph-mon[32608]: osdmap e12: 2 total, 1 up, 2 in 2024-04-23T10:19:44.610 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 23 10:19:44 smithi119 ceph-mon[32608]: from='mgr.14152 172.21.15.114:0/1529706138' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-04-23T10:19:44.610 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 23 10:19:44 smithi119 ceph-mon[32608]: from='mgr.14152 172.21.15.114:0/1529706138' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-04-23T10:19:44.610 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 23 10:19:44 smithi119 ceph-mon[32608]: pgmap v61: 0 pgs: ; 0 B data, 31 MiB used, 89 GiB / 89 GiB avail 2024-04-23T10:19:44.655 INFO:journalctl@ceph.mon.a.smithi114.stdout:Apr 23 10:19:44 smithi114 ceph-mon[29255]: from='osd.1 ' entity='osd.1' cmd='[{"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi119", "root=default"]}]': finished 2024-04-23T10:19:44.656 INFO:journalctl@ceph.mon.a.smithi114.stdout:Apr 23 10:19:44 smithi114 ceph-mon[29255]: osdmap e12: 2 total, 1 up, 2 in 2024-04-23T10:19:44.656 INFO:journalctl@ceph.mon.a.smithi114.stdout:Apr 23 10:19:44 smithi114 ceph-mon[29255]: from='mgr.14152 172.21.15.114:0/1529706138' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-04-23T10:19:44.656 INFO:journalctl@ceph.mon.a.smithi114.stdout:Apr 23 10:19:44 smithi114 ceph-mon[29255]: from='mgr.14152 172.21.15.114:0/1529706138' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-04-23T10:19:44.656 INFO:journalctl@ceph.mon.a.smithi114.stdout:Apr 23 10:19:44 smithi114 ceph-mon[29255]: pgmap v61: 0 pgs: ; 0 B data, 31 MiB used, 89 GiB / 89 GiB avail 2024-04-23T10:19:44.747 DEBUG:teuthology.orchestra.run.smithi204:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d69f13d827952bb07bb9b7973bca14f97d764d6f shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid aa240572-0159-11ef-bc93-c7b262605968 -- ceph orch daemon add osd smithi204:vg_nvme/lv_4 2024-04-23T10:19:44.983 INFO:teuthology.orchestra.run.smithi204.stderr:Inferring config /var/lib/ceph/aa240572-0159-11ef-bc93-c7b262605968/mon.c/config 2024-04-23T10:19:45.414 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 23 10:19:45 smithi204 ceph-mon[32649]: purged_snaps scrub starts 2024-04-23T10:19:45.414 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 23 10:19:45 smithi204 ceph-mon[32649]: purged_snaps scrub ok 2024-04-23T10:19:45.414 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 23 10:19:45 smithi204 ceph-mon[32649]: from='mgr.14152 172.21.15.114:0/1529706138' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-04-23T10:19:45.414 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 23 10:19:45 smithi204 ceph-mon[32649]: osd.1 [v2:172.21.15.119:6800/1133540779,v1:172.21.15.119:6801/1133540779] boot 2024-04-23T10:19:45.414 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 23 10:19:45 smithi204 ceph-mon[32649]: osdmap e13: 2 total, 2 up, 2 in 2024-04-23T10:19:45.414 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 23 10:19:45 smithi204 ceph-mon[32649]: from='mgr.14152 172.21.15.114:0/1529706138' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-04-23T10:19:45.414 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 23 10:19:45 smithi204 ceph-mon[32649]: Detected new or changed devices on smithi119 2024-04-23T10:19:45.414 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 23 10:19:45 smithi204 ceph-mon[32649]: from='mgr.14152 172.21.15.114:0/1529706138' entity='mgr.x' 2024-04-23T10:19:45.414 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 23 10:19:45 smithi204 ceph-mon[32649]: from='mgr.14152 172.21.15.114:0/1529706138' entity='mgr.x' 2024-04-23T10:19:45.414 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 23 10:19:45 smithi204 ceph-mon[32649]: from='mgr.14152 172.21.15.114:0/1529706138' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-04-23T10:19:45.415 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 23 10:19:45 smithi204 ceph-mon[32649]: Adjusting osd_memory_target on smithi119 to 17088M 2024-04-23T10:19:45.415 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 23 10:19:45 smithi204 ceph-mon[32649]: from='mgr.14152 172.21.15.114:0/1529706138' entity='mgr.x' 2024-04-23T10:19:45.415 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 23 10:19:45 smithi204 ceph-mon[32649]: from='mgr.14152 172.21.15.114:0/1529706138' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-23T10:19:45.415 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 23 10:19:45 smithi204 ceph-mon[32649]: from='mgr.14152 172.21.15.114:0/1529706138' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-23T10:19:45.415 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 23 10:19:45 smithi204 ceph-mon[32649]: from='mgr.14152 172.21.15.114:0/1529706138' entity='mgr.x' 2024-04-23T10:19:45.742 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 23 10:19:45 smithi119 ceph-mon[32608]: purged_snaps scrub starts 2024-04-23T10:19:45.742 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 23 10:19:45 smithi119 ceph-mon[32608]: purged_snaps scrub ok 2024-04-23T10:19:45.743 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 23 10:19:45 smithi119 ceph-mon[32608]: from='mgr.14152 172.21.15.114:0/1529706138' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-04-23T10:19:45.743 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 23 10:19:45 smithi119 ceph-mon[32608]: osd.1 [v2:172.21.15.119:6800/1133540779,v1:172.21.15.119:6801/1133540779] boot 2024-04-23T10:19:45.743 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 23 10:19:45 smithi119 ceph-mon[32608]: osdmap e13: 2 total, 2 up, 2 in 2024-04-23T10:19:45.743 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 23 10:19:45 smithi119 ceph-mon[32608]: from='mgr.14152 172.21.15.114:0/1529706138' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-04-23T10:19:45.743 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 23 10:19:45 smithi119 ceph-mon[32608]: Detected new or changed devices on smithi119 2024-04-23T10:19:45.743 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 23 10:19:45 smithi119 ceph-mon[32608]: from='mgr.14152 172.21.15.114:0/1529706138' entity='mgr.x' 2024-04-23T10:19:45.743 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 23 10:19:45 smithi119 ceph-mon[32608]: from='mgr.14152 172.21.15.114:0/1529706138' entity='mgr.x' 2024-04-23T10:19:45.743 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 23 10:19:45 smithi119 ceph-mon[32608]: from='mgr.14152 172.21.15.114:0/1529706138' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-04-23T10:19:45.743 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 23 10:19:45 smithi119 ceph-mon[32608]: Adjusting osd_memory_target on smithi119 to 17088M 2024-04-23T10:19:45.743 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 23 10:19:45 smithi119 ceph-mon[32608]: from='mgr.14152 172.21.15.114:0/1529706138' entity='mgr.x' 2024-04-23T10:19:45.743 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 23 10:19:45 smithi119 ceph-mon[32608]: from='mgr.14152 172.21.15.114:0/1529706138' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-23T10:19:45.743 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 23 10:19:45 smithi119 ceph-mon[32608]: from='mgr.14152 172.21.15.114:0/1529706138' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-23T10:19:45.744 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 23 10:19:45 smithi119 ceph-mon[32608]: from='mgr.14152 172.21.15.114:0/1529706138' entity='mgr.x' 2024-04-23T10:19:45.754 INFO:journalctl@ceph.mon.a.smithi114.stdout:Apr 23 10:19:45 smithi114 ceph-mon[29255]: purged_snaps scrub starts 2024-04-23T10:19:45.755 INFO:journalctl@ceph.mon.a.smithi114.stdout:Apr 23 10:19:45 smithi114 ceph-mon[29255]: purged_snaps scrub ok 2024-04-23T10:19:45.755 INFO:journalctl@ceph.mon.a.smithi114.stdout:Apr 23 10:19:45 smithi114 ceph-mon[29255]: from='mgr.14152 172.21.15.114:0/1529706138' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-04-23T10:19:45.755 INFO:journalctl@ceph.mon.a.smithi114.stdout:Apr 23 10:19:45 smithi114 ceph-mon[29255]: osd.1 [v2:172.21.15.119:6800/1133540779,v1:172.21.15.119:6801/1133540779] boot 2024-04-23T10:19:45.755 INFO:journalctl@ceph.mon.a.smithi114.stdout:Apr 23 10:19:45 smithi114 ceph-mon[29255]: osdmap e13: 2 total, 2 up, 2 in 2024-04-23T10:19:45.755 INFO:journalctl@ceph.mon.a.smithi114.stdout:Apr 23 10:19:45 smithi114 ceph-mon[29255]: from='mgr.14152 172.21.15.114:0/1529706138' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-04-23T10:19:45.755 INFO:journalctl@ceph.mon.a.smithi114.stdout:Apr 23 10:19:45 smithi114 ceph-mon[29255]: Detected new or changed devices on smithi119 2024-04-23T10:19:45.755 INFO:journalctl@ceph.mon.a.smithi114.stdout:Apr 23 10:19:45 smithi114 ceph-mon[29255]: from='mgr.14152 172.21.15.114:0/1529706138' entity='mgr.x' 2024-04-23T10:19:45.755 INFO:journalctl@ceph.mon.a.smithi114.stdout:Apr 23 10:19:45 smithi114 ceph-mon[29255]: from='mgr.14152 172.21.15.114:0/1529706138' entity='mgr.x' 2024-04-23T10:19:45.755 INFO:journalctl@ceph.mon.a.smithi114.stdout:Apr 23 10:19:45 smithi114 ceph-mon[29255]: from='mgr.14152 172.21.15.114:0/1529706138' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-04-23T10:19:45.755 INFO:journalctl@ceph.mon.a.smithi114.stdout:Apr 23 10:19:45 smithi114 ceph-mon[29255]: Adjusting osd_memory_target on smithi119 to 17088M 2024-04-23T10:19:45.755 INFO:journalctl@ceph.mon.a.smithi114.stdout:Apr 23 10:19:45 smithi114 ceph-mon[29255]: from='mgr.14152 172.21.15.114:0/1529706138' entity='mgr.x' 2024-04-23T10:19:45.755 INFO:journalctl@ceph.mon.a.smithi114.stdout:Apr 23 10:19:45 smithi114 ceph-mon[29255]: from='mgr.14152 172.21.15.114:0/1529706138' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-23T10:19:45.756 INFO:journalctl@ceph.mon.a.smithi114.stdout:Apr 23 10:19:45 smithi114 ceph-mon[29255]: from='mgr.14152 172.21.15.114:0/1529706138' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-23T10:19:45.756 INFO:journalctl@ceph.mon.a.smithi114.stdout:Apr 23 10:19:45 smithi114 ceph-mon[29255]: from='mgr.14152 172.21.15.114:0/1529706138' entity='mgr.x' 2024-04-23T10:19:46.504 INFO:journalctl@ceph.mon.a.smithi114.stdout:Apr 23 10:19:46 smithi114 ceph-mon[29255]: osdmap e14: 2 total, 2 up, 2 in 2024-04-23T10:19:46.504 INFO:journalctl@ceph.mon.a.smithi114.stdout:Apr 23 10:19:46 smithi114 ceph-mon[29255]: pgmap v64: 0 pgs: ; 0 B data, 57 MiB used, 179 GiB / 179 GiB avail 2024-04-23T10:19:46.542 INFO:teuthology.orchestra.run.smithi204.stderr:2024-04-23T10:19:46.541+0000 7f2f3f838700 1 -- 172.21.15.204:0/1785651139 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f2f38108830 msgr2=0x7f2f3810ac20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T10:19:46.542 INFO:teuthology.orchestra.run.smithi204.stderr:2024-04-23T10:19:46.541+0000 7f2f3f838700 1 --2- 172.21.15.204:0/1785651139 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f2f38108830 0x7f2f3810ac20 secure :-1 s=READY pgs=9 cs=0 l=1 rev1=1 crypto rx=0x7f2f3000b0a0 tx=0x7f2f3000b3b0 comp rx=0 tx=0).stop 2024-04-23T10:19:46.542 INFO:teuthology.orchestra.run.smithi204.stderr:2024-04-23T10:19:46.541+0000 7f2f3f838700 1 -- 172.21.15.204:0/1785651139 shutdown_connections 2024-04-23T10:19:46.542 INFO:teuthology.orchestra.run.smithi204.stderr:2024-04-23T10:19:46.541+0000 7f2f3f838700 1 --2- 172.21.15.204:0/1785651139 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f2f38108830 0x7f2f3810ac20 unknown :-1 s=CLOSED pgs=9 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:19:46.542 INFO:teuthology.orchestra.run.smithi204.stderr:2024-04-23T10:19:46.541+0000 7f2f3f838700 1 --2- 172.21.15.204:0/1785651139 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f2f381035d0 0x7f2f381059c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:19:46.542 INFO:teuthology.orchestra.run.smithi204.stderr:2024-04-23T10:19:46.541+0000 7f2f3f838700 1 --2- 172.21.15.204:0/1785651139 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f2f38105f00 0x7f2f381082f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:19:46.542 INFO:teuthology.orchestra.run.smithi204.stderr:2024-04-23T10:19:46.541+0000 7f2f3f838700 1 -- 172.21.15.204:0/1785651139 >> 172.21.15.204:0/1785651139 conn(0x7f2f380fcf40 msgr2=0x7f2f380ff380 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T10:19:46.542 INFO:teuthology.orchestra.run.smithi204.stderr:2024-04-23T10:19:46.541+0000 7f2f3f838700 1 -- 172.21.15.204:0/1785651139 shutdown_connections 2024-04-23T10:19:46.543 INFO:teuthology.orchestra.run.smithi204.stderr:2024-04-23T10:19:46.541+0000 7f2f3f838700 1 -- 172.21.15.204:0/1785651139 wait complete. 2024-04-23T10:19:46.543 INFO:teuthology.orchestra.run.smithi204.stderr:2024-04-23T10:19:46.542+0000 7f2f3f838700 1 Processor -- start 2024-04-23T10:19:46.543 INFO:teuthology.orchestra.run.smithi204.stderr:2024-04-23T10:19:46.542+0000 7f2f3f838700 1 -- start start 2024-04-23T10:19:46.543 INFO:teuthology.orchestra.run.smithi204.stderr:2024-04-23T10:19:46.542+0000 7f2f3f838700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f2f381035d0 0x7f2f38102f30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T10:19:46.543 INFO:teuthology.orchestra.run.smithi204.stderr:2024-04-23T10:19:46.542+0000 7f2f3f838700 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f2f38105f00 0x7f2f38101580 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T10:19:46.543 INFO:teuthology.orchestra.run.smithi204.stderr:2024-04-23T10:19:46.542+0000 7f2f3f838700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f2f38108830 0x7f2f38101ac0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T10:19:46.543 INFO:teuthology.orchestra.run.smithi204.stderr:2024-04-23T10:19:46.542+0000 7f2f3f838700 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x7f2f3810e500 con 0x7f2f38108830 2024-04-23T10:19:46.543 INFO:teuthology.orchestra.run.smithi204.stderr:2024-04-23T10:19:46.542+0000 7f2f3f838700 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 -- 0x7f2f3810e380 con 0x7f2f381035d0 2024-04-23T10:19:46.543 INFO:teuthology.orchestra.run.smithi204.stderr:2024-04-23T10:19:46.542+0000 7f2f3f838700 1 -- --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_getmap magic: 0 -- 0x7f2f3810e680 con 0x7f2f38105f00 2024-04-23T10:19:46.543 INFO:teuthology.orchestra.run.smithi204.stderr:2024-04-23T10:19:46.542+0000 7f2f3cdd3700 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f2f38105f00 0x7f2f38101580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T10:19:46.543 INFO:teuthology.orchestra.run.smithi204.stderr:2024-04-23T10:19:46.542+0000 7f2f3cdd3700 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f2f38105f00 0x7f2f38101580 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.204:3300/0 says I am v2:172.21.15.204:34484/0 (socket says 172.21.15.204:34484) 2024-04-23T10:19:46.543 INFO:teuthology.orchestra.run.smithi204.stderr:2024-04-23T10:19:46.542+0000 7f2f3cdd3700 1 -- 172.21.15.204:0/2029816795 learned_addr learned my addr 172.21.15.204:0/2029816795 (peer_addr_for_me v2:172.21.15.204:0/0) 2024-04-23T10:19:46.544 INFO:teuthology.orchestra.run.smithi204.stderr:2024-04-23T10:19:46.542+0000 7f2f3ddd5700 1 --2- 172.21.15.204:0/2029816795 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f2f38108830 0x7f2f38101ac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T10:19:46.544 INFO:teuthology.orchestra.run.smithi204.stderr:2024-04-23T10:19:46.542+0000 7f2f3d5d4700 1 --2- 172.21.15.204:0/2029816795 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f2f381035d0 0x7f2f38102f30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T10:19:46.544 INFO:teuthology.orchestra.run.smithi204.stderr:2024-04-23T10:19:46.542+0000 7f2f3cdd3700 1 -- 172.21.15.204:0/2029816795 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f2f381035d0 msgr2=0x7f2f38102f30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T10:19:46.544 INFO:teuthology.orchestra.run.smithi204.stderr:2024-04-23T10:19:46.542+0000 7f2f3cdd3700 1 --2- 172.21.15.204:0/2029816795 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f2f381035d0 0x7f2f38102f30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:19:46.544 INFO:teuthology.orchestra.run.smithi204.stderr:2024-04-23T10:19:46.542+0000 7f2f3cdd3700 1 -- 172.21.15.204:0/2029816795 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f2f38108830 msgr2=0x7f2f38101ac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T10:19:46.544 INFO:teuthology.orchestra.run.smithi204.stderr:2024-04-23T10:19:46.542+0000 7f2f3cdd3700 1 --2- 172.21.15.204:0/2029816795 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f2f38108830 0x7f2f38101ac0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:19:46.544 INFO:teuthology.orchestra.run.smithi204.stderr:2024-04-23T10:19:46.542+0000 7f2f3cdd3700 1 -- 172.21.15.204:0/2029816795 --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f2f38102370 con 0x7f2f38105f00 2024-04-23T10:19:46.544 INFO:teuthology.orchestra.run.smithi204.stderr:2024-04-23T10:19:46.542+0000 7f2f3d5d4700 1 --2- 172.21.15.204:0/2029816795 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f2f381035d0 0x7f2f38102f30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-23T10:19:46.544 INFO:teuthology.orchestra.run.smithi204.stderr:2024-04-23T10:19:46.542+0000 7f2f3ddd5700 1 --2- 172.21.15.204:0/2029816795 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f2f38108830 0x7f2f38101ac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-23T10:19:46.544 INFO:teuthology.orchestra.run.smithi204.stderr:2024-04-23T10:19:46.543+0000 7f2f3cdd3700 1 --2- 172.21.15.204:0/2029816795 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f2f38105f00 0x7f2f38101580 secure :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0x7f2f2c00d950 tx=0x7f2f2c00dd10 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T10:19:46.544 INFO:teuthology.orchestra.run.smithi204.stderr:2024-04-23T10:19:46.543+0000 7f2f367fc700 1 -- 172.21.15.204:0/2029816795 <== mon.1 v2:172.21.15.204:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2f2c016070 con 0x7f2f38105f00 2024-04-23T10:19:46.545 INFO:teuthology.orchestra.run.smithi204.stderr:2024-04-23T10:19:46.543+0000 7f2f3f838700 1 -- 172.21.15.204:0/2029816795 --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f2f381a2d30 con 0x7f2f38105f00 2024-04-23T10:19:46.545 INFO:teuthology.orchestra.run.smithi204.stderr:2024-04-23T10:19:46.543+0000 7f2f367fc700 1 -- 172.21.15.204:0/2029816795 <== mon.1 v2:172.21.15.204:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f2f2c010460 con 0x7f2f38105f00 2024-04-23T10:19:46.545 INFO:teuthology.orchestra.run.smithi204.stderr:2024-04-23T10:19:46.543+0000 7f2f367fc700 1 -- 172.21.15.204:0/2029816795 <== mon.1 v2:172.21.15.204:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2f2c00f620 con 0x7f2f38105f00 2024-04-23T10:19:46.545 INFO:teuthology.orchestra.run.smithi204.stderr:2024-04-23T10:19:46.543+0000 7f2f3f838700 1 -- 172.21.15.204:0/2029816795 --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f2f381a31c0 con 0x7f2f38105f00 2024-04-23T10:19:46.546 INFO:teuthology.orchestra.run.smithi204.stderr:2024-04-23T10:19:46.544+0000 7f2f3f838700 1 -- 172.21.15.204:0/2029816795 --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f2f38069730 con 0x7f2f38105f00 2024-04-23T10:19:46.546 INFO:teuthology.orchestra.run.smithi204.stderr:2024-04-23T10:19:46.545+0000 7f2f367fc700 1 -- 172.21.15.204:0/2029816795 <== mon.1 v2:172.21.15.204:3300/0 4 ==== mgrmap(e 15) ==== 138591+0+0 (secure 0 0 0) 0x7f2f2c019070 con 0x7f2f38105f00 2024-04-23T10:19:46.546 INFO:teuthology.orchestra.run.smithi204.stderr:2024-04-23T10:19:46.545+0000 7f2f367fc700 1 --2- 172.21.15.204:0/2029816795 >> [v2:172.21.15.114:6800/465333808,v1:172.21.15.114:6801/465333808] conn(0x7f2f200a3bb0 0x7f2f200a6070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T10:19:46.546 INFO:teuthology.orchestra.run.smithi204.stderr:2024-04-23T10:19:46.546+0000 7f2f367fc700 1 -- 172.21.15.204:0/2029816795 <== mon.1 v2:172.21.15.204:3300/0 5 ==== osd_map(14..14 src has 1..14) ==== 2309+0+0 (secure 0 0 0) 0x7f2f2c0ce800 con 0x7f2f38105f00 2024-04-23T10:19:46.546 INFO:teuthology.orchestra.run.smithi204.stderr:2024-04-23T10:19:46.546+0000 7f2f3d5d4700 1 --2- 172.21.15.204:0/2029816795 >> [v2:172.21.15.114:6800/465333808,v1:172.21.15.114:6801/465333808] conn(0x7f2f200a3bb0 0x7f2f200a6070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T10:19:46.546 INFO:teuthology.orchestra.run.smithi204.stderr:2024-04-23T10:19:46.546+0000 7f2f3d5d4700 1 --2- 172.21.15.204:0/2029816795 >> [v2:172.21.15.114:6800/465333808,v1:172.21.15.114:6801/465333808] conn(0x7f2f200a3bb0 0x7f2f200a6070 secure :-1 s=READY pgs=60 cs=0 l=1 rev1=1 crypto rx=0x7f2f28005fd0 tx=0x7f2f28005ee0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T10:19:46.548 INFO:teuthology.orchestra.run.smithi204.stderr:2024-04-23T10:19:46.549+0000 7f2f367fc700 1 -- 172.21.15.204:0/2029816795 <== mon.1 v2:172.21.15.204:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f2f2c09ca30 con 0x7f2f38105f00 2024-04-23T10:19:46.578 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 23 10:19:46 smithi204 ceph-mon[32649]: osdmap e14: 2 total, 2 up, 2 in 2024-04-23T10:19:46.578 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 23 10:19:46 smithi204 ceph-mon[32649]: pgmap v64: 0 pgs: ; 0 B data, 57 MiB used, 179 GiB / 179 GiB avail 2024-04-23T10:19:46.742 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 23 10:19:46 smithi119 ceph-mon[32608]: osdmap e14: 2 total, 2 up, 2 in 2024-04-23T10:19:46.742 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 23 10:19:46 smithi119 ceph-mon[32608]: pgmap v64: 0 pgs: ; 0 B data, 57 MiB used, 179 GiB / 179 GiB avail 2024-04-23T10:19:46.752 INFO:teuthology.orchestra.run.smithi204.stderr:2024-04-23T10:19:46.753+0000 7f2f3f838700 1 -- 172.21.15.204:0/2029816795 --> [v2:172.21.15.114:6800/465333808,v1:172.21.15.114:6801/465333808] -- mgr_command(tid 0: {"prefix": "orch daemon add osd", "svc_arg": "smithi204:vg_nvme/lv_4", "target": ["mon-mgr", ""]}) -- 0x7f2f38062e80 con 0x7f2f200a3bb0 2024-04-23T10:19:47.578 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 23 10:19:47 smithi204 ceph-mon[32649]: from='client.24160 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi204:vg_nvme/lv_4", "target": ["mon-mgr", ""]}]: dispatch 2024-04-23T10:19:47.578 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 23 10:19:47 smithi204 ceph-mon[32649]: from='mgr.14152 172.21.15.114:0/1529706138' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-04-23T10:19:47.579 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 23 10:19:47 smithi204 ceph-mon[32649]: from='mgr.14152 172.21.15.114:0/1529706138' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-04-23T10:19:47.579 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 23 10:19:47 smithi204 ceph-mon[32649]: from='mgr.14152 172.21.15.114:0/1529706138' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-23T10:19:47.742 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 23 10:19:47 smithi119 ceph-mon[32608]: from='client.24160 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi204:vg_nvme/lv_4", "target": ["mon-mgr", ""]}]: dispatch 2024-04-23T10:19:47.743 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 23 10:19:47 smithi119 ceph-mon[32608]: from='mgr.14152 172.21.15.114:0/1529706138' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-04-23T10:19:47.743 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 23 10:19:47 smithi119 ceph-mon[32608]: from='mgr.14152 172.21.15.114:0/1529706138' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-04-23T10:19:47.743 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 23 10:19:47 smithi119 ceph-mon[32608]: from='mgr.14152 172.21.15.114:0/1529706138' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-23T10:19:47.754 INFO:journalctl@ceph.mon.a.smithi114.stdout:Apr 23 10:19:47 smithi114 ceph-mon[29255]: from='client.24160 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi204:vg_nvme/lv_4", "target": ["mon-mgr", ""]}]: dispatch 2024-04-23T10:19:47.754 INFO:journalctl@ceph.mon.a.smithi114.stdout:Apr 23 10:19:47 smithi114 ceph-mon[29255]: from='mgr.14152 172.21.15.114:0/1529706138' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-04-23T10:19:47.755 INFO:journalctl@ceph.mon.a.smithi114.stdout:Apr 23 10:19:47 smithi114 ceph-mon[29255]: from='mgr.14152 172.21.15.114:0/1529706138' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-04-23T10:19:47.755 INFO:journalctl@ceph.mon.a.smithi114.stdout:Apr 23 10:19:47 smithi114 ceph-mon[29255]: from='mgr.14152 172.21.15.114:0/1529706138' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-23T10:19:48.393 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 23 10:19:48 smithi204 ceph-mon[32649]: pgmap v65: 0 pgs: ; 0 B data, 57 MiB used, 179 GiB / 179 GiB avail 2024-04-23T10:19:48.742 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 23 10:19:48 smithi119 ceph-mon[32608]: pgmap v65: 0 pgs: ; 0 B data, 57 MiB used, 179 GiB / 179 GiB avail 2024-04-23T10:19:48.754 INFO:journalctl@ceph.mon.a.smithi114.stdout:Apr 23 10:19:48 smithi114 ceph-mon[29255]: pgmap v65: 0 pgs: ; 0 B data, 57 MiB used, 179 GiB / 179 GiB avail 2024-04-23T10:19:49.828 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 23 10:19:49 smithi204 ceph-mon[32649]: from='client.? 172.21.15.204:0/4200297040' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "f4c9f2b4-ff49-4e14-89cd-b477feeb9382"}]: dispatch 2024-04-23T10:19:49.829 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 23 10:19:49 smithi204 ceph-mon[32649]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "f4c9f2b4-ff49-4e14-89cd-b477feeb9382"}]: dispatch 2024-04-23T10:19:49.829 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 23 10:19:49 smithi204 ceph-mon[32649]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "f4c9f2b4-ff49-4e14-89cd-b477feeb9382"}]': finished 2024-04-23T10:19:49.829 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 23 10:19:49 smithi204 ceph-mon[32649]: osdmap e15: 3 total, 2 up, 3 in 2024-04-23T10:19:49.829 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 23 10:19:49 smithi204 ceph-mon[32649]: from='mgr.14152 172.21.15.114:0/1529706138' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-04-23T10:19:49.992 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 23 10:19:49 smithi119 ceph-mon[32608]: from='client.? 172.21.15.204:0/4200297040' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "f4c9f2b4-ff49-4e14-89cd-b477feeb9382"}]: dispatch 2024-04-23T10:19:49.992 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 23 10:19:49 smithi119 ceph-mon[32608]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "f4c9f2b4-ff49-4e14-89cd-b477feeb9382"}]: dispatch 2024-04-23T10:19:49.992 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 23 10:19:49 smithi119 ceph-mon[32608]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "f4c9f2b4-ff49-4e14-89cd-b477feeb9382"}]': finished 2024-04-23T10:19:49.992 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 23 10:19:49 smithi119 ceph-mon[32608]: osdmap e15: 3 total, 2 up, 3 in 2024-04-23T10:19:49.992 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 23 10:19:49 smithi119 ceph-mon[32608]: from='mgr.14152 172.21.15.114:0/1529706138' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-04-23T10:19:50.004 INFO:journalctl@ceph.mon.a.smithi114.stdout:Apr 23 10:19:49 smithi114 ceph-mon[29255]: from='client.? 172.21.15.204:0/4200297040' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "f4c9f2b4-ff49-4e14-89cd-b477feeb9382"}]: dispatch 2024-04-23T10:19:50.004 INFO:journalctl@ceph.mon.a.smithi114.stdout:Apr 23 10:19:49 smithi114 ceph-mon[29255]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "f4c9f2b4-ff49-4e14-89cd-b477feeb9382"}]: dispatch 2024-04-23T10:19:50.004 INFO:journalctl@ceph.mon.a.smithi114.stdout:Apr 23 10:19:49 smithi114 ceph-mon[29255]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "f4c9f2b4-ff49-4e14-89cd-b477feeb9382"}]': finished 2024-04-23T10:19:50.004 INFO:journalctl@ceph.mon.a.smithi114.stdout:Apr 23 10:19:49 smithi114 ceph-mon[29255]: osdmap e15: 3 total, 2 up, 3 in 2024-04-23T10:19:50.004 INFO:journalctl@ceph.mon.a.smithi114.stdout:Apr 23 10:19:49 smithi114 ceph-mon[29255]: from='mgr.14152 172.21.15.114:0/1529706138' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-04-23T10:19:50.828 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 23 10:19:50 smithi204 ceph-mon[32649]: pgmap v67: 0 pgs: ; 0 B data, 57 MiB used, 179 GiB / 179 GiB avail 2024-04-23T10:19:50.828 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 23 10:19:50 smithi204 ceph-mon[32649]: from='client.? 172.21.15.204:0/2218870773' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-04-23T10:19:50.992 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 23 10:19:50 smithi119 ceph-mon[32608]: pgmap v67: 0 pgs: ; 0 B data, 57 MiB used, 179 GiB / 179 GiB avail 2024-04-23T10:19:50.992 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 23 10:19:50 smithi119 ceph-mon[32608]: from='client.? 172.21.15.204:0/2218870773' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-04-23T10:19:51.004 INFO:journalctl@ceph.mon.a.smithi114.stdout:Apr 23 10:19:50 smithi114 ceph-mon[29255]: pgmap v67: 0 pgs: ; 0 B data, 57 MiB used, 179 GiB / 179 GiB avail 2024-04-23T10:19:51.004 INFO:journalctl@ceph.mon.a.smithi114.stdout:Apr 23 10:19:50 smithi114 ceph-mon[29255]: from='client.? 172.21.15.204:0/2218870773' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-04-23T10:19:53.242 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 23 10:19:52 smithi119 ceph-mon[32608]: pgmap v68: 0 pgs: ; 0 B data, 57 MiB used, 179 GiB / 179 GiB avail 2024-04-23T10:19:53.254 INFO:journalctl@ceph.mon.a.smithi114.stdout:Apr 23 10:19:52 smithi114 ceph-mon[29255]: pgmap v68: 0 pgs: ; 0 B data, 57 MiB used, 179 GiB / 179 GiB avail 2024-04-23T10:19:53.328 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 23 10:19:52 smithi204 ceph-mon[32649]: pgmap v68: 0 pgs: ; 0 B data, 57 MiB used, 179 GiB / 179 GiB avail 2024-04-23T10:19:55.242 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 23 10:19:54 smithi119 ceph-mon[32608]: pgmap v69: 0 pgs: ; 0 B data, 57 MiB used, 179 GiB / 179 GiB avail 2024-04-23T10:19:55.254 INFO:journalctl@ceph.mon.a.smithi114.stdout:Apr 23 10:19:54 smithi114 ceph-mon[29255]: pgmap v69: 0 pgs: ; 0 B data, 57 MiB used, 179 GiB / 179 GiB avail 2024-04-23T10:19:55.328 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 23 10:19:54 smithi204 ceph-mon[32649]: pgmap v69: 0 pgs: ; 0 B data, 57 MiB used, 179 GiB / 179 GiB avail 2024-04-23T10:19:57.242 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 23 10:19:56 smithi119 ceph-mon[32608]: pgmap v70: 0 pgs: ; 0 B data, 57 MiB used, 179 GiB / 179 GiB avail 2024-04-23T10:19:57.242 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 23 10:19:56 smithi119 ceph-mon[32608]: from='mgr.14152 172.21.15.114:0/1529706138' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.2"}]: dispatch 2024-04-23T10:19:57.242 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 23 10:19:56 smithi119 ceph-mon[32608]: from='mgr.14152 172.21.15.114:0/1529706138' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-23T10:19:57.254 INFO:journalctl@ceph.mon.a.smithi114.stdout:Apr 23 10:19:56 smithi114 ceph-mon[29255]: pgmap v70: 0 pgs: ; 0 B data, 57 MiB used, 179 GiB / 179 GiB avail 2024-04-23T10:19:57.254 INFO:journalctl@ceph.mon.a.smithi114.stdout:Apr 23 10:19:56 smithi114 ceph-mon[29255]: from='mgr.14152 172.21.15.114:0/1529706138' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.2"}]: dispatch 2024-04-23T10:19:57.254 INFO:journalctl@ceph.mon.a.smithi114.stdout:Apr 23 10:19:56 smithi114 ceph-mon[29255]: from='mgr.14152 172.21.15.114:0/1529706138' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-23T10:19:57.328 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 23 10:19:56 smithi204 ceph-mon[32649]: pgmap v70: 0 pgs: ; 0 B data, 57 MiB used, 179 GiB / 179 GiB avail 2024-04-23T10:19:57.328 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 23 10:19:56 smithi204 ceph-mon[32649]: from='mgr.14152 172.21.15.114:0/1529706138' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.2"}]: dispatch 2024-04-23T10:19:57.328 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 23 10:19:56 smithi204 ceph-mon[32649]: from='mgr.14152 172.21.15.114:0/1529706138' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-23T10:19:58.212 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 23 10:19:57 smithi204 ceph-mon[32649]: Deploying daemon osd.2 on smithi204 2024-04-23T10:19:58.242 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 23 10:19:57 smithi119 ceph-mon[32608]: Deploying daemon osd.2 on smithi204 2024-04-23T10:19:58.254 INFO:journalctl@ceph.mon.a.smithi114.stdout:Apr 23 10:19:57 smithi114 ceph-mon[29255]: Deploying daemon osd.2 on smithi204 2024-04-23T10:19:59.078 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 23 10:19:58 smithi204 ceph-mon[32649]: pgmap v71: 0 pgs: ; 0 B data, 57 MiB used, 179 GiB / 179 GiB avail 2024-04-23T10:19:59.242 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 23 10:19:58 smithi119 ceph-mon[32608]: pgmap v71: 0 pgs: ; 0 B data, 57 MiB used, 179 GiB / 179 GiB avail 2024-04-23T10:19:59.254 INFO:journalctl@ceph.mon.a.smithi114.stdout:Apr 23 10:19:58 smithi114 ceph-mon[29255]: pgmap v71: 0 pgs: ; 0 B data, 57 MiB used, 179 GiB / 179 GiB avail 2024-04-23T10:20:01.078 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 23 10:20:00 smithi204 ceph-mon[32649]: pgmap v72: 0 pgs: ; 0 B data, 57 MiB used, 179 GiB / 179 GiB avail 2024-04-23T10:20:01.079 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 23 10:20:00 smithi204 ceph-mon[32649]: overall HEALTH_OK 2024-04-23T10:20:01.242 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 23 10:20:00 smithi119 ceph-mon[32608]: pgmap v72: 0 pgs: ; 0 B data, 57 MiB used, 179 GiB / 179 GiB avail 2024-04-23T10:20:01.242 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 23 10:20:00 smithi119 ceph-mon[32608]: overall HEALTH_OK 2024-04-23T10:20:01.254 INFO:journalctl@ceph.mon.a.smithi114.stdout:Apr 23 10:20:00 smithi114 ceph-mon[29255]: pgmap v72: 0 pgs: ; 0 B data, 57 MiB used, 179 GiB / 179 GiB avail 2024-04-23T10:20:01.254 INFO:journalctl@ceph.mon.a.smithi114.stdout:Apr 23 10:20:00 smithi114 ceph-mon[29255]: overall HEALTH_OK 2024-04-23T10:20:03.036 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 23 10:20:02 smithi204 ceph-mon[32649]: pgmap v73: 0 pgs: ; 0 B data, 57 MiB used, 179 GiB / 179 GiB avail 2024-04-23T10:20:03.036 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 23 10:20:02 smithi204 ceph-mon[32649]: from='mgr.14152 172.21.15.114:0/1529706138' entity='mgr.x' 2024-04-23T10:20:03.036 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 23 10:20:02 smithi204 ceph-mon[32649]: from='mgr.14152 172.21.15.114:0/1529706138' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-23T10:20:03.036 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 23 10:20:02 smithi204 ceph-mon[32649]: from='mgr.14152 172.21.15.114:0/1529706138' entity='mgr.x' 2024-04-23T10:20:03.242 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 23 10:20:02 smithi119 ceph-mon[32608]: pgmap v73: 0 pgs: ; 0 B data, 57 MiB used, 179 GiB / 179 GiB avail 2024-04-23T10:20:03.242 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 23 10:20:02 smithi119 ceph-mon[32608]: from='mgr.14152 172.21.15.114:0/1529706138' entity='mgr.x' 2024-04-23T10:20:03.242 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 23 10:20:02 smithi119 ceph-mon[32608]: from='mgr.14152 172.21.15.114:0/1529706138' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-23T10:20:03.242 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 23 10:20:02 smithi119 ceph-mon[32608]: from='mgr.14152 172.21.15.114:0/1529706138' entity='mgr.x' 2024-04-23T10:20:03.254 INFO:journalctl@ceph.mon.a.smithi114.stdout:Apr 23 10:20:02 smithi114 ceph-mon[29255]: pgmap v73: 0 pgs: ; 0 B data, 57 MiB used, 179 GiB / 179 GiB avail 2024-04-23T10:20:03.254 INFO:journalctl@ceph.mon.a.smithi114.stdout:Apr 23 10:20:02 smithi114 ceph-mon[29255]: from='mgr.14152 172.21.15.114:0/1529706138' entity='mgr.x' 2024-04-23T10:20:03.254 INFO:journalctl@ceph.mon.a.smithi114.stdout:Apr 23 10:20:02 smithi114 ceph-mon[29255]: from='mgr.14152 172.21.15.114:0/1529706138' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-23T10:20:03.254 INFO:journalctl@ceph.mon.a.smithi114.stdout:Apr 23 10:20:02 smithi114 ceph-mon[29255]: from='mgr.14152 172.21.15.114:0/1529706138' entity='mgr.x' 2024-04-23T10:20:05.242 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 23 10:20:04 smithi119 ceph-mon[32608]: pgmap v74: 0 pgs: ; 0 B data, 57 MiB used, 179 GiB / 179 GiB avail 2024-04-23T10:20:05.254 INFO:journalctl@ceph.mon.a.smithi114.stdout:Apr 23 10:20:04 smithi114 ceph-mon[29255]: pgmap v74: 0 pgs: ; 0 B data, 57 MiB used, 179 GiB / 179 GiB avail 2024-04-23T10:20:05.255 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 23 10:20:04 smithi204 ceph-mon[32649]: pgmap v74: 0 pgs: ; 0 B data, 57 MiB used, 179 GiB / 179 GiB avail 2024-04-23T10:20:05.616 INFO:teuthology.orchestra.run.smithi204.stderr:2024-04-23T10:20:05.616+0000 7f2f367fc700 1 -- 172.21.15.204:0/2029816795 <== mgr.14152 v2:172.21.15.114:6800/465333808 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+37 (secure 0 0 0) 0x7f2f38062e80 con 0x7f2f200a3bb0 2024-04-23T10:20:05.617 INFO:teuthology.orchestra.run.smithi204.stdout:Created osd(s) 2 on host 'smithi204' 2024-04-23T10:20:05.618 INFO:teuthology.orchestra.run.smithi204.stderr:2024-04-23T10:20:05.618+0000 7f2f3f838700 1 -- 172.21.15.204:0/2029816795 >> [v2:172.21.15.114:6800/465333808,v1:172.21.15.114:6801/465333808] conn(0x7f2f200a3bb0 msgr2=0x7f2f200a6070 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T10:20:05.618 INFO:teuthology.orchestra.run.smithi204.stderr:2024-04-23T10:20:05.618+0000 7f2f3f838700 1 --2- 172.21.15.204:0/2029816795 >> [v2:172.21.15.114:6800/465333808,v1:172.21.15.114:6801/465333808] conn(0x7f2f200a3bb0 0x7f2f200a6070 secure :-1 s=READY pgs=60 cs=0 l=1 rev1=1 crypto rx=0x7f2f28005fd0 tx=0x7f2f28005ee0 comp rx=0 tx=0).stop 2024-04-23T10:20:05.618 INFO:teuthology.orchestra.run.smithi204.stderr:2024-04-23T10:20:05.618+0000 7f2f3f838700 1 -- 172.21.15.204:0/2029816795 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f2f38105f00 msgr2=0x7f2f38101580 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T10:20:05.619 INFO:teuthology.orchestra.run.smithi204.stderr:2024-04-23T10:20:05.618+0000 7f2f3f838700 1 --2- 172.21.15.204:0/2029816795 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f2f38105f00 0x7f2f38101580 secure :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0x7f2f2c00d950 tx=0x7f2f2c00dd10 comp rx=0 tx=0).stop 2024-04-23T10:20:05.619 INFO:teuthology.orchestra.run.smithi204.stderr:2024-04-23T10:20:05.618+0000 7f2f3f838700 1 -- 172.21.15.204:0/2029816795 shutdown_connections 2024-04-23T10:20:05.619 INFO:teuthology.orchestra.run.smithi204.stderr:2024-04-23T10:20:05.618+0000 7f2f3f838700 1 --2- 172.21.15.204:0/2029816795 >> [v2:172.21.15.114:6800/465333808,v1:172.21.15.114:6801/465333808] conn(0x7f2f200a3bb0 0x7f2f200a6070 unknown :-1 s=CLOSED pgs=60 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:20:05.619 INFO:teuthology.orchestra.run.smithi204.stderr:2024-04-23T10:20:05.618+0000 7f2f3f838700 1 --2- 172.21.15.204:0/2029816795 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f2f381035d0 0x7f2f38102f30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:20:05.619 INFO:teuthology.orchestra.run.smithi204.stderr:2024-04-23T10:20:05.618+0000 7f2f3f838700 1 --2- 172.21.15.204:0/2029816795 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f2f38108830 0x7f2f38101ac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:20:05.619 INFO:teuthology.orchestra.run.smithi204.stderr:2024-04-23T10:20:05.618+0000 7f2f3f838700 1 --2- 172.21.15.204:0/2029816795 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f2f38105f00 0x7f2f38101580 unknown :-1 s=CLOSED pgs=10 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:20:05.619 INFO:teuthology.orchestra.run.smithi204.stderr:2024-04-23T10:20:05.618+0000 7f2f3f838700 1 -- 172.21.15.204:0/2029816795 >> 172.21.15.204:0/2029816795 conn(0x7f2f380fcf40 msgr2=0x7f2f38104010 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T10:20:05.619 INFO:teuthology.orchestra.run.smithi204.stderr:2024-04-23T10:20:05.618+0000 7f2f3f838700 1 -- 172.21.15.204:0/2029816795 shutdown_connections 2024-04-23T10:20:05.619 INFO:teuthology.orchestra.run.smithi204.stderr:2024-04-23T10:20:05.618+0000 7f2f3f838700 1 -- 172.21.15.204:0/2029816795 wait complete. 2024-04-23T10:20:06.209 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 23 10:20:06 smithi204 ceph-mon[32649]: from='mgr.14152 172.21.15.114:0/1529706138' entity='mgr.x' 2024-04-23T10:20:06.209 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 23 10:20:06 smithi204 ceph-mon[32649]: from='mgr.14152 172.21.15.114:0/1529706138' entity='mgr.x' 2024-04-23T10:20:06.209 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 23 10:20:06 smithi204 ceph-mon[32649]: from='mgr.14152 172.21.15.114:0/1529706138' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-23T10:20:06.209 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 23 10:20:06 smithi204 ceph-mon[32649]: from='mgr.14152 172.21.15.114:0/1529706138' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-23T10:20:06.209 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 23 10:20:06 smithi204 ceph-mon[32649]: from='mgr.14152 172.21.15.114:0/1529706138' entity='mgr.x' 2024-04-23T10:20:06.209 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 23 10:20:06 smithi204 ceph-mon[32649]: from='mgr.14152 172.21.15.114:0/1529706138' entity='mgr.x' 2024-04-23T10:20:06.210 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 23 10:20:06 smithi204 ceph-mon[32649]: from='mgr.14152 172.21.15.114:0/1529706138' entity='mgr.x' 2024-04-23T10:20:06.210 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 23 10:20:06 smithi204 ceph-mon[32649]: from='mgr.14152 172.21.15.114:0/1529706138' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-23T10:20:06.210 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 23 10:20:06 smithi204 ceph-mon[32649]: pgmap v75: 0 pgs: ; 0 B data, 57 MiB used, 179 GiB / 179 GiB avail 2024-04-23T10:20:06.234 DEBUG:teuthology.orchestra.run.smithi204:osd.2> sudo journalctl -f -n 0 -u ceph-aa240572-0159-11ef-bc93-c7b262605968@osd.2.service 2024-04-23T10:20:06.237 INFO:tasks.cephadm:Deploying osd.3 on smithi114 with /dev/vg_nvme/lv_3... 2024-04-23T10:20:06.237 DEBUG:teuthology.orchestra.run.smithi114:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d69f13d827952bb07bb9b7973bca14f97d764d6f shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid aa240572-0159-11ef-bc93-c7b262605968 -- ceph-volume lvm zap /dev/vg_nvme/lv_3 2024-04-23T10:20:06.482 INFO:teuthology.orchestra.run.smithi114.stderr:Inferring config /var/lib/ceph/aa240572-0159-11ef-bc93-c7b262605968/mon.a/config 2024-04-23T10:20:06.492 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 23 10:20:06 smithi119 ceph-mon[32608]: from='mgr.14152 172.21.15.114:0/1529706138' entity='mgr.x' 2024-04-23T10:20:06.492 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 23 10:20:06 smithi119 ceph-mon[32608]: from='mgr.14152 172.21.15.114:0/1529706138' entity='mgr.x' 2024-04-23T10:20:06.492 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 23 10:20:06 smithi119 ceph-mon[32608]: from='mgr.14152 172.21.15.114:0/1529706138' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-23T10:20:06.492 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 23 10:20:06 smithi119 ceph-mon[32608]: from='mgr.14152 172.21.15.114:0/1529706138' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-23T10:20:06.492 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 23 10:20:06 smithi119 ceph-mon[32608]: from='mgr.14152 172.21.15.114:0/1529706138' entity='mgr.x' 2024-04-23T10:20:06.492 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 23 10:20:06 smithi119 ceph-mon[32608]: from='mgr.14152 172.21.15.114:0/1529706138' entity='mgr.x' 2024-04-23T10:20:06.492 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 23 10:20:06 smithi119 ceph-mon[32608]: from='mgr.14152 172.21.15.114:0/1529706138' entity='mgr.x' 2024-04-23T10:20:06.493 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 23 10:20:06 smithi119 ceph-mon[32608]: from='mgr.14152 172.21.15.114:0/1529706138' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-23T10:20:06.493 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 23 10:20:06 smithi119 ceph-mon[32608]: pgmap v75: 0 pgs: ; 0 B data, 57 MiB used, 179 GiB / 179 GiB avail 2024-04-23T10:20:06.504 INFO:journalctl@ceph.mon.a.smithi114.stdout:Apr 23 10:20:06 smithi114 ceph-mon[29255]: from='mgr.14152 172.21.15.114:0/1529706138' entity='mgr.x' 2024-04-23T10:20:06.504 INFO:journalctl@ceph.mon.a.smithi114.stdout:Apr 23 10:20:06 smithi114 ceph-mon[29255]: from='mgr.14152 172.21.15.114:0/1529706138' entity='mgr.x' 2024-04-23T10:20:06.504 INFO:journalctl@ceph.mon.a.smithi114.stdout:Apr 23 10:20:06 smithi114 ceph-mon[29255]: from='mgr.14152 172.21.15.114:0/1529706138' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-23T10:20:06.505 INFO:journalctl@ceph.mon.a.smithi114.stdout:Apr 23 10:20:06 smithi114 ceph-mon[29255]: from='mgr.14152 172.21.15.114:0/1529706138' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-23T10:20:06.505 INFO:journalctl@ceph.mon.a.smithi114.stdout:Apr 23 10:20:06 smithi114 ceph-mon[29255]: from='mgr.14152 172.21.15.114:0/1529706138' entity='mgr.x' 2024-04-23T10:20:06.505 INFO:journalctl@ceph.mon.a.smithi114.stdout:Apr 23 10:20:06 smithi114 ceph-mon[29255]: from='mgr.14152 172.21.15.114:0/1529706138' entity='mgr.x' 2024-04-23T10:20:06.505 INFO:journalctl@ceph.mon.a.smithi114.stdout:Apr 23 10:20:06 smithi114 ceph-mon[29255]: from='mgr.14152 172.21.15.114:0/1529706138' entity='mgr.x' 2024-04-23T10:20:06.505 INFO:journalctl@ceph.mon.a.smithi114.stdout:Apr 23 10:20:06 smithi114 ceph-mon[29255]: from='mgr.14152 172.21.15.114:0/1529706138' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-23T10:20:06.505 INFO:journalctl@ceph.mon.a.smithi114.stdout:Apr 23 10:20:06 smithi114 ceph-mon[29255]: pgmap v75: 0 pgs: ; 0 B data, 57 MiB used, 179 GiB / 179 GiB avail 2024-04-23T10:20:06.554 INFO:journalctl@ceph.osd.2.smithi204.stdout:Apr 23 10:20:06 smithi204 ceph-aa240572-0159-11ef-bc93-c7b262605968-osd-2[37811]: 2024-04-23T10:20:06.278+0000 7fd9ce16c7c0 -1 osd.2 0 log_to_monitors true 2024-04-23T10:20:07.254 INFO:journalctl@ceph.mon.a.smithi114.stdout:Apr 23 10:20:07 smithi114 ceph-mon[29255]: from='osd.2 [v2:172.21.15.204:6800/799652511,v1:172.21.15.204:6801/799652511]' entity='osd.2' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]: dispatch 2024-04-23T10:20:07.254 INFO:journalctl@ceph.mon.a.smithi114.stdout:Apr 23 10:20:07 smithi114 ceph-mon[29255]: from='osd.2 ' entity='osd.2' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]: dispatch 2024-04-23T10:20:07.328 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 23 10:20:07 smithi204 ceph-mon[32649]: from='osd.2 [v2:172.21.15.204:6800/799652511,v1:172.21.15.204:6801/799652511]' entity='osd.2' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]: dispatch 2024-04-23T10:20:07.328 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 23 10:20:07 smithi204 ceph-mon[32649]: from='osd.2 ' entity='osd.2' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]: dispatch 2024-04-23T10:20:07.492 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 23 10:20:07 smithi119 ceph-mon[32608]: from='osd.2 [v2:172.21.15.204:6800/799652511,v1:172.21.15.204:6801/799652511]' entity='osd.2' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]: dispatch 2024-04-23T10:20:07.492 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 23 10:20:07 smithi119 ceph-mon[32608]: from='osd.2 ' entity='osd.2' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]: dispatch 2024-04-23T10:20:08.295 INFO:teuthology.orchestra.run.smithi114.stderr:--> Zapping: /dev/vg_nvme/lv_3 2024-04-23T10:20:08.295 INFO:teuthology.orchestra.run.smithi114.stderr:Running command: /usr/bin/dd if=/dev/zero of=/dev/vg_nvme/lv_3 bs=1M count=10 conv=fsync 2024-04-23T10:20:08.295 INFO:teuthology.orchestra.run.smithi114.stderr: stderr: 10+0 records in 2024-04-23T10:20:08.296 INFO:teuthology.orchestra.run.smithi114.stderr:10+0 records out 2024-04-23T10:20:08.296 INFO:teuthology.orchestra.run.smithi114.stderr:10485760 bytes (10 MB, 10 MiB) copied, 0.0117785 s, 890 MB/s 2024-04-23T10:20:08.296 INFO:teuthology.orchestra.run.smithi114.stderr:--> Zapping successful for: 2024-04-23T10:20:08.492 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 23 10:20:08 smithi119 ceph-mon[32608]: from='osd.2 ' entity='osd.2' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]': finished 2024-04-23T10:20:08.492 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 23 10:20:08 smithi119 ceph-mon[32608]: from='osd.2 [v2:172.21.15.204:6800/799652511,v1:172.21.15.204:6801/799652511]' entity='osd.2' cmd=[{"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi204", "root=default"]}]: dispatch 2024-04-23T10:20:08.492 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 23 10:20:08 smithi119 ceph-mon[32608]: osdmap e16: 3 total, 2 up, 3 in 2024-04-23T10:20:08.492 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 23 10:20:08 smithi119 ceph-mon[32608]: from='mgr.14152 172.21.15.114:0/1529706138' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-04-23T10:20:08.492 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 23 10:20:08 smithi119 ceph-mon[32608]: from='osd.2 ' entity='osd.2' cmd=[{"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi204", "root=default"]}]: dispatch 2024-04-23T10:20:08.492 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 23 10:20:08 smithi119 ceph-mon[32608]: pgmap v77: 0 pgs: ; 0 B data, 57 MiB used, 179 GiB / 179 GiB avail 2024-04-23T10:20:08.504 INFO:journalctl@ceph.mon.a.smithi114.stdout:Apr 23 10:20:08 smithi114 ceph-mon[29255]: from='osd.2 ' entity='osd.2' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]': finished 2024-04-23T10:20:08.504 INFO:journalctl@ceph.mon.a.smithi114.stdout:Apr 23 10:20:08 smithi114 ceph-mon[29255]: from='osd.2 [v2:172.21.15.204:6800/799652511,v1:172.21.15.204:6801/799652511]' entity='osd.2' cmd=[{"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi204", "root=default"]}]: dispatch 2024-04-23T10:20:08.504 INFO:journalctl@ceph.mon.a.smithi114.stdout:Apr 23 10:20:08 smithi114 ceph-mon[29255]: osdmap e16: 3 total, 2 up, 3 in 2024-04-23T10:20:08.504 INFO:journalctl@ceph.mon.a.smithi114.stdout:Apr 23 10:20:08 smithi114 ceph-mon[29255]: from='mgr.14152 172.21.15.114:0/1529706138' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-04-23T10:20:08.504 INFO:journalctl@ceph.mon.a.smithi114.stdout:Apr 23 10:20:08 smithi114 ceph-mon[29255]: from='osd.2 ' entity='osd.2' cmd=[{"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi204", "root=default"]}]: dispatch 2024-04-23T10:20:08.504 INFO:journalctl@ceph.mon.a.smithi114.stdout:Apr 23 10:20:08 smithi114 ceph-mon[29255]: pgmap v77: 0 pgs: ; 0 B data, 57 MiB used, 179 GiB / 179 GiB avail 2024-04-23T10:20:08.578 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 23 10:20:08 smithi204 ceph-mon[32649]: from='osd.2 ' entity='osd.2' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]': finished 2024-04-23T10:20:08.578 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 23 10:20:08 smithi204 ceph-mon[32649]: from='osd.2 [v2:172.21.15.204:6800/799652511,v1:172.21.15.204:6801/799652511]' entity='osd.2' cmd=[{"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi204", "root=default"]}]: dispatch 2024-04-23T10:20:08.578 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 23 10:20:08 smithi204 ceph-mon[32649]: osdmap e16: 3 total, 2 up, 3 in 2024-04-23T10:20:08.578 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 23 10:20:08 smithi204 ceph-mon[32649]: from='mgr.14152 172.21.15.114:0/1529706138' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-04-23T10:20:08.578 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 23 10:20:08 smithi204 ceph-mon[32649]: from='osd.2 ' entity='osd.2' cmd=[{"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi204", "root=default"]}]: dispatch 2024-04-23T10:20:08.578 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 23 10:20:08 smithi204 ceph-mon[32649]: pgmap v77: 0 pgs: ; 0 B data, 57 MiB used, 179 GiB / 179 GiB avail 2024-04-23T10:20:08.900 DEBUG:teuthology.orchestra.run.smithi114:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d69f13d827952bb07bb9b7973bca14f97d764d6f shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid aa240572-0159-11ef-bc93-c7b262605968 -- ceph orch daemon add osd smithi114:vg_nvme/lv_3 2024-04-23T10:20:09.157 INFO:teuthology.orchestra.run.smithi114.stderr:Inferring config /var/lib/ceph/aa240572-0159-11ef-bc93-c7b262605968/mon.a/config 2024-04-23T10:20:09.492 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 23 10:20:09 smithi119 ceph-mon[32608]: from='osd.2 ' entity='osd.2' cmd='[{"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi204", "root=default"]}]': finished 2024-04-23T10:20:09.492 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 23 10:20:09 smithi119 ceph-mon[32608]: osdmap e17: 3 total, 2 up, 3 in 2024-04-23T10:20:09.492 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 23 10:20:09 smithi119 ceph-mon[32608]: from='mgr.14152 172.21.15.114:0/1529706138' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-04-23T10:20:09.492 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 23 10:20:09 smithi119 ceph-mon[32608]: from='mgr.14152 172.21.15.114:0/1529706138' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-04-23T10:20:09.492 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 23 10:20:09 smithi119 ceph-mon[32608]: Detected new or changed devices on smithi204 2024-04-23T10:20:09.492 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 23 10:20:09 smithi119 ceph-mon[32608]: from='mgr.14152 172.21.15.114:0/1529706138' entity='mgr.x' 2024-04-23T10:20:09.493 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 23 10:20:09 smithi119 ceph-mon[32608]: from='mgr.14152 172.21.15.114:0/1529706138' entity='mgr.x' 2024-04-23T10:20:09.493 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 23 10:20:09 smithi119 ceph-mon[32608]: from='mgr.14152 172.21.15.114:0/1529706138' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-04-23T10:20:09.493 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 23 10:20:09 smithi119 ceph-mon[32608]: Adjusting osd_memory_target on smithi204 to 17088M 2024-04-23T10:20:09.493 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 23 10:20:09 smithi119 ceph-mon[32608]: from='mgr.14152 172.21.15.114:0/1529706138' entity='mgr.x' 2024-04-23T10:20:09.493 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 23 10:20:09 smithi119 ceph-mon[32608]: from='mgr.14152 172.21.15.114:0/1529706138' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-23T10:20:09.493 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 23 10:20:09 smithi119 ceph-mon[32608]: from='mgr.14152 172.21.15.114:0/1529706138' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-23T10:20:09.493 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 23 10:20:09 smithi119 ceph-mon[32608]: from='mgr.14152 172.21.15.114:0/1529706138' entity='mgr.x' 2024-04-23T10:20:09.493 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 23 10:20:09 smithi119 ceph-mon[32608]: from='osd.2 ' entity='osd.2' 2024-04-23T10:20:09.500 INFO:journalctl@ceph.mon.a.smithi114.stdout:Apr 23 10:20:09 smithi114 ceph-mon[29255]: from='osd.2 ' entity='osd.2' cmd='[{"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi204", "root=default"]}]': finished 2024-04-23T10:20:09.500 INFO:journalctl@ceph.mon.a.smithi114.stdout:Apr 23 10:20:09 smithi114 ceph-mon[29255]: osdmap e17: 3 total, 2 up, 3 in 2024-04-23T10:20:09.500 INFO:journalctl@ceph.mon.a.smithi114.stdout:Apr 23 10:20:09 smithi114 ceph-mon[29255]: from='mgr.14152 172.21.15.114:0/1529706138' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-04-23T10:20:09.500 INFO:journalctl@ceph.mon.a.smithi114.stdout:Apr 23 10:20:09 smithi114 ceph-mon[29255]: from='mgr.14152 172.21.15.114:0/1529706138' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-04-23T10:20:09.500 INFO:journalctl@ceph.mon.a.smithi114.stdout:Apr 23 10:20:09 smithi114 ceph-mon[29255]: Detected new or changed devices on smithi204 2024-04-23T10:20:09.500 INFO:journalctl@ceph.mon.a.smithi114.stdout:Apr 23 10:20:09 smithi114 ceph-mon[29255]: from='mgr.14152 172.21.15.114:0/1529706138' entity='mgr.x' 2024-04-23T10:20:09.500 INFO:journalctl@ceph.mon.a.smithi114.stdout:Apr 23 10:20:09 smithi114 ceph-mon[29255]: from='mgr.14152 172.21.15.114:0/1529706138' entity='mgr.x' 2024-04-23T10:20:09.500 INFO:journalctl@ceph.mon.a.smithi114.stdout:Apr 23 10:20:09 smithi114 ceph-mon[29255]: from='mgr.14152 172.21.15.114:0/1529706138' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-04-23T10:20:09.500 INFO:journalctl@ceph.mon.a.smithi114.stdout:Apr 23 10:20:09 smithi114 ceph-mon[29255]: Adjusting osd_memory_target on smithi204 to 17088M 2024-04-23T10:20:09.500 INFO:journalctl@ceph.mon.a.smithi114.stdout:Apr 23 10:20:09 smithi114 ceph-mon[29255]: from='mgr.14152 172.21.15.114:0/1529706138' entity='mgr.x' 2024-04-23T10:20:09.500 INFO:journalctl@ceph.mon.a.smithi114.stdout:Apr 23 10:20:09 smithi114 ceph-mon[29255]: from='mgr.14152 172.21.15.114:0/1529706138' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-23T10:20:09.500 INFO:journalctl@ceph.mon.a.smithi114.stdout:Apr 23 10:20:09 smithi114 ceph-mon[29255]: from='mgr.14152 172.21.15.114:0/1529706138' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-23T10:20:09.500 INFO:journalctl@ceph.mon.a.smithi114.stdout:Apr 23 10:20:09 smithi114 ceph-mon[29255]: from='mgr.14152 172.21.15.114:0/1529706138' entity='mgr.x' 2024-04-23T10:20:09.500 INFO:journalctl@ceph.mon.a.smithi114.stdout:Apr 23 10:20:09 smithi114 ceph-mon[29255]: from='osd.2 ' entity='osd.2' 2024-04-23T10:20:09.578 INFO:journalctl@ceph.osd.2.smithi204.stdout:Apr 23 10:20:09 smithi204 ceph-aa240572-0159-11ef-bc93-c7b262605968-osd-2[37811]: 2024-04-23T10:20:09.079+0000 7fd9c37d1700 -1 osd.2 0 waiting for initial osdmap 2024-04-23T10:20:09.578 INFO:journalctl@ceph.osd.2.smithi204.stdout:Apr 23 10:20:09 smithi204 ceph-aa240572-0159-11ef-bc93-c7b262605968-osd-2[37811]: 2024-04-23T10:20:09.238+0000 7fd9bb5bc700 -1 osd.2 17 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-04-23T10:20:09.578 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 23 10:20:09 smithi204 ceph-mon[32649]: from='osd.2 ' entity='osd.2' cmd='[{"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi204", "root=default"]}]': finished 2024-04-23T10:20:09.578 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 23 10:20:09 smithi204 ceph-mon[32649]: osdmap e17: 3 total, 2 up, 3 in 2024-04-23T10:20:09.578 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 23 10:20:09 smithi204 ceph-mon[32649]: from='mgr.14152 172.21.15.114:0/1529706138' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-04-23T10:20:09.578 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 23 10:20:09 smithi204 ceph-mon[32649]: from='mgr.14152 172.21.15.114:0/1529706138' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-04-23T10:20:09.578 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 23 10:20:09 smithi204 ceph-mon[32649]: Detected new or changed devices on smithi204 2024-04-23T10:20:09.579 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 23 10:20:09 smithi204 ceph-mon[32649]: from='mgr.14152 172.21.15.114:0/1529706138' entity='mgr.x' 2024-04-23T10:20:09.579 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 23 10:20:09 smithi204 ceph-mon[32649]: from='mgr.14152 172.21.15.114:0/1529706138' entity='mgr.x' 2024-04-23T10:20:09.579 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 23 10:20:09 smithi204 ceph-mon[32649]: from='mgr.14152 172.21.15.114:0/1529706138' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-04-23T10:20:09.579 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 23 10:20:09 smithi204 ceph-mon[32649]: Adjusting osd_memory_target on smithi204 to 17088M 2024-04-23T10:20:09.579 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 23 10:20:09 smithi204 ceph-mon[32649]: from='mgr.14152 172.21.15.114:0/1529706138' entity='mgr.x' 2024-04-23T10:20:09.579 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 23 10:20:09 smithi204 ceph-mon[32649]: from='mgr.14152 172.21.15.114:0/1529706138' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-23T10:20:09.579 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 23 10:20:09 smithi204 ceph-mon[32649]: from='mgr.14152 172.21.15.114:0/1529706138' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-23T10:20:09.579 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 23 10:20:09 smithi204 ceph-mon[32649]: from='mgr.14152 172.21.15.114:0/1529706138' entity='mgr.x' 2024-04-23T10:20:09.579 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 23 10:20:09 smithi204 ceph-mon[32649]: from='osd.2 ' entity='osd.2' 2024-04-23T10:20:10.397 INFO:journalctl@ceph.mon.a.smithi114.stdout:Apr 23 10:20:10 smithi114 ceph-mon[29255]: purged_snaps scrub starts 2024-04-23T10:20:10.397 INFO:journalctl@ceph.mon.a.smithi114.stdout:Apr 23 10:20:10 smithi114 ceph-mon[29255]: purged_snaps scrub ok 2024-04-23T10:20:10.397 INFO:journalctl@ceph.mon.a.smithi114.stdout:Apr 23 10:20:10 smithi114 ceph-mon[29255]: from='mgr.14152 172.21.15.114:0/1529706138' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-04-23T10:20:10.398 INFO:journalctl@ceph.mon.a.smithi114.stdout:Apr 23 10:20:10 smithi114 ceph-mon[29255]: pgmap v79: 0 pgs: ; 0 B data, 57 MiB used, 179 GiB / 179 GiB avail 2024-04-23T10:20:10.492 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 23 10:20:10 smithi119 ceph-mon[32608]: purged_snaps scrub starts 2024-04-23T10:20:10.492 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 23 10:20:10 smithi119 ceph-mon[32608]: purged_snaps scrub ok 2024-04-23T10:20:10.492 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 23 10:20:10 smithi119 ceph-mon[32608]: from='mgr.14152 172.21.15.114:0/1529706138' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-04-23T10:20:10.492 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 23 10:20:10 smithi119 ceph-mon[32608]: pgmap v79: 0 pgs: ; 0 B data, 57 MiB used, 179 GiB / 179 GiB avail 2024-04-23T10:20:10.578 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 23 10:20:10 smithi204 ceph-mon[32649]: purged_snaps scrub starts 2024-04-23T10:20:10.578 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 23 10:20:10 smithi204 ceph-mon[32649]: purged_snaps scrub ok 2024-04-23T10:20:10.578 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 23 10:20:10 smithi204 ceph-mon[32649]: from='mgr.14152 172.21.15.114:0/1529706138' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-04-23T10:20:10.578 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 23 10:20:10 smithi204 ceph-mon[32649]: pgmap v79: 0 pgs: ; 0 B data, 57 MiB used, 179 GiB / 179 GiB avail 2024-04-23T10:20:10.610 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:20:10.609+0000 7f208db19700 1 -- 172.21.15.114:0/2695176488 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f2088102bb0 msgr2=0x7f2088103010 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T10:20:10.610 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:20:10.609+0000 7f208db19700 1 --2- 172.21.15.114:0/2695176488 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f2088102bb0 0x7f2088103010 secure :-1 s=READY pgs=119 cs=0 l=1 rev1=1 crypto rx=0x7f207c0099c0 tx=0x7f207c009cd0 comp rx=0 tx=0).stop 2024-04-23T10:20:10.610 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:20:10.609+0000 7f208db19700 1 -- 172.21.15.114:0/2695176488 shutdown_connections 2024-04-23T10:20:10.610 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:20:10.609+0000 7f208db19700 1 --2- 172.21.15.114:0/2695176488 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f2088103550 0x7f2088109e00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:20:10.611 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:20:10.609+0000 7f208db19700 1 --2- 172.21.15.114:0/2695176488 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f20881019b0 0x7f2088101dd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:20:10.611 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:20:10.609+0000 7f208db19700 1 --2- 172.21.15.114:0/2695176488 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f2088102bb0 0x7f2088103010 unknown :-1 s=CLOSED pgs=119 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:20:10.611 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:20:10.609+0000 7f208db19700 1 -- 172.21.15.114:0/2695176488 >> 172.21.15.114:0/2695176488 conn(0x7f20880fcf70 msgr2=0x7f20880ff390 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T10:20:10.611 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:20:10.609+0000 7f208db19700 1 -- 172.21.15.114:0/2695176488 shutdown_connections 2024-04-23T10:20:10.611 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:20:10.609+0000 7f208db19700 1 -- 172.21.15.114:0/2695176488 wait complete. 2024-04-23T10:20:10.611 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:20:10.609+0000 7f208db19700 1 Processor -- start 2024-04-23T10:20:10.611 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:20:10.610+0000 7f208db19700 1 -- start start 2024-04-23T10:20:10.611 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:20:10.610+0000 7f208db19700 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f20881019b0 0x7f2088198bc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T10:20:10.611 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:20:10.610+0000 7f208db19700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f2088102bb0 0x7f2088199100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T10:20:10.611 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:20:10.610+0000 7f208db19700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f2088103550 0x7f20881a0180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T10:20:10.611 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:20:10.610+0000 7f208db19700 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x7f208810d420 con 0x7f2088103550 2024-04-23T10:20:10.611 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:20:10.610+0000 7f208db19700 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 -- 0x7f208810d2a0 con 0x7f2088102bb0 2024-04-23T10:20:10.611 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:20:10.610+0000 7f208db19700 1 -- --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_getmap magic: 0 -- 0x7f208810d5a0 con 0x7f20881019b0 2024-04-23T10:20:10.611 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:20:10.610+0000 7f2087fff700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f2088103550 0x7f20881a0180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T10:20:10.612 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:20:10.610+0000 7f2087fff700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f2088103550 0x7f20881a0180 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.114:59114/0 (socket says 172.21.15.114:59114) 2024-04-23T10:20:10.612 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:20:10.610+0000 7f2087fff700 1 -- 172.21.15.114:0/19850472 learned_addr learned my addr 172.21.15.114:0/19850472 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-04-23T10:20:10.612 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:20:10.610+0000 7f2086ffd700 1 --2- 172.21.15.114:0/19850472 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f2088102bb0 0x7f2088199100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T10:20:10.612 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:20:10.610+0000 7f20877fe700 1 --2- 172.21.15.114:0/19850472 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f20881019b0 0x7f2088198bc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T10:20:10.612 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:20:10.610+0000 7f2087fff700 1 -- 172.21.15.114:0/19850472 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f20881019b0 msgr2=0x7f2088198bc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T10:20:10.612 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:20:10.610+0000 7f2087fff700 1 --2- 172.21.15.114:0/19850472 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f20881019b0 0x7f2088198bc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:20:10.612 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:20:10.610+0000 7f2087fff700 1 -- 172.21.15.114:0/19850472 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f2088102bb0 msgr2=0x7f2088199100 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T10:20:10.612 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:20:10.610+0000 7f2087fff700 1 --2- 172.21.15.114:0/19850472 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f2088102bb0 0x7f2088199100 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:20:10.612 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:20:10.610+0000 7f2087fff700 1 -- 172.21.15.114:0/19850472 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f20881a0820 con 0x7f2088103550 2024-04-23T10:20:10.612 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:20:10.610+0000 7f20877fe700 1 --2- 172.21.15.114:0/19850472 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f20881019b0 0x7f2088198bc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-23T10:20:10.612 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:20:10.610+0000 7f2087fff700 1 --2- 172.21.15.114:0/19850472 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f2088103550 0x7f20881a0180 secure :-1 s=READY pgs=120 cs=0 l=1 rev1=1 crypto rx=0x7f208000add0 tx=0x7f208000c330 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T10:20:10.612 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:20:10.610+0000 7f2084ff9700 1 -- 172.21.15.114:0/19850472 <== mon.0 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2080017020 con 0x7f2088103550 2024-04-23T10:20:10.612 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:20:10.611+0000 7f2084ff9700 1 -- 172.21.15.114:0/19850472 <== mon.0 v2:172.21.15.114:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f208000cdf0 con 0x7f2088103550 2024-04-23T10:20:10.612 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:20:10.611+0000 7f208db19700 1 -- 172.21.15.114:0/19850472 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f20881a0b10 con 0x7f2088103550 2024-04-23T10:20:10.612 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:20:10.611+0000 7f2084ff9700 1 -- 172.21.15.114:0/19850472 <== mon.0 v2:172.21.15.114:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2080005460 con 0x7f2088103550 2024-04-23T10:20:10.613 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:20:10.611+0000 7f208db19700 1 -- 172.21.15.114:0/19850472 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f20881a1000 con 0x7f2088103550 2024-04-23T10:20:10.613 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:20:10.612+0000 7f208db19700 1 -- 172.21.15.114:0/19850472 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f2088069730 con 0x7f2088103550 2024-04-23T10:20:10.614 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:20:10.613+0000 7f2084ff9700 1 -- 172.21.15.114:0/19850472 <== mon.0 v2:172.21.15.114:3300/0 4 ==== mgrmap(e 15) ==== 138591+0+0 (secure 0 0 0) 0x7f2080004ad0 con 0x7f2088103550 2024-04-23T10:20:10.614 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:20:10.614+0000 7f2084ff9700 1 --2- 172.21.15.114:0/19850472 >> [v2:172.21.15.114:6800/465333808,v1:172.21.15.114:6801/465333808] conn(0x7f20700a3bb0 0x7f20700a6070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T10:20:10.614 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:20:10.614+0000 7f20877fe700 1 --2- 172.21.15.114:0/19850472 >> [v2:172.21.15.114:6800/465333808,v1:172.21.15.114:6801/465333808] conn(0x7f20700a3bb0 0x7f20700a6070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T10:20:10.614 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:20:10.614+0000 7f2084ff9700 1 -- 172.21.15.114:0/19850472 <== mon.0 v2:172.21.15.114:3300/0 5 ==== osd_map(18..18 src has 1..18) ==== 2851+0+0 (secure 0 0 0) 0x7f20800cebf0 con 0x7f2088103550 2024-04-23T10:20:10.614 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:20:10.614+0000 7f20877fe700 1 --2- 172.21.15.114:0/19850472 >> [v2:172.21.15.114:6800/465333808,v1:172.21.15.114:6801/465333808] conn(0x7f20700a3bb0 0x7f20700a6070 secure :-1 s=READY pgs=66 cs=0 l=1 rev1=1 crypto rx=0x7f2078005fd0 tx=0x7f2078005e40 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T10:20:10.616 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:20:10.618+0000 7f2084ff9700 1 -- 172.21.15.114:0/19850472 <== mon.0 v2:172.21.15.114:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f2080021020 con 0x7f2088103550 2024-04-23T10:20:10.825 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:20:10.826+0000 7f208db19700 1 -- 172.21.15.114:0/19850472 --> [v2:172.21.15.114:6800/465333808,v1:172.21.15.114:6801/465333808] -- mgr_command(tid 0: {"prefix": "orch daemon add osd", "svc_arg": "smithi114:vg_nvme/lv_3", "target": ["mon-mgr", ""]}) -- 0x7f208806b8a0 con 0x7f20700a3bb0 2024-04-23T10:20:11.448 INFO:journalctl@ceph.mon.a.smithi114.stdout:Apr 23 10:20:11 smithi114 ceph-mon[29255]: from='mgr.14152 172.21.15.114:0/1529706138' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-04-23T10:20:11.449 INFO:journalctl@ceph.mon.a.smithi114.stdout:Apr 23 10:20:11 smithi114 ceph-mon[29255]: osd.2 [v2:172.21.15.204:6800/799652511,v1:172.21.15.204:6801/799652511] boot 2024-04-23T10:20:11.449 INFO:journalctl@ceph.mon.a.smithi114.stdout:Apr 23 10:20:11 smithi114 ceph-mon[29255]: osdmap e18: 3 total, 3 up, 3 in 2024-04-23T10:20:11.449 INFO:journalctl@ceph.mon.a.smithi114.stdout:Apr 23 10:20:11 smithi114 ceph-mon[29255]: from='mgr.14152 172.21.15.114:0/1529706138' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-04-23T10:20:11.449 INFO:journalctl@ceph.mon.a.smithi114.stdout:Apr 23 10:20:11 smithi114 ceph-mon[29255]: from='client.14286 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi114:vg_nvme/lv_3", "target": ["mon-mgr", ""]}]: dispatch 2024-04-23T10:20:11.449 INFO:journalctl@ceph.mon.a.smithi114.stdout:Apr 23 10:20:11 smithi114 ceph-mon[29255]: from='mgr.14152 172.21.15.114:0/1529706138' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-04-23T10:20:11.449 INFO:journalctl@ceph.mon.a.smithi114.stdout:Apr 23 10:20:11 smithi114 ceph-mon[29255]: from='mgr.14152 172.21.15.114:0/1529706138' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-04-23T10:20:11.449 INFO:journalctl@ceph.mon.a.smithi114.stdout:Apr 23 10:20:11 smithi114 ceph-mon[29255]: from='mgr.14152 172.21.15.114:0/1529706138' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-23T10:20:11.492 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 23 10:20:11 smithi119 ceph-mon[32608]: from='mgr.14152 172.21.15.114:0/1529706138' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-04-23T10:20:11.492 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 23 10:20:11 smithi119 ceph-mon[32608]: osd.2 [v2:172.21.15.204:6800/799652511,v1:172.21.15.204:6801/799652511] boot 2024-04-23T10:20:11.492 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 23 10:20:11 smithi119 ceph-mon[32608]: osdmap e18: 3 total, 3 up, 3 in 2024-04-23T10:20:11.492 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 23 10:20:11 smithi119 ceph-mon[32608]: from='mgr.14152 172.21.15.114:0/1529706138' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-04-23T10:20:11.492 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 23 10:20:11 smithi119 ceph-mon[32608]: from='client.14286 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi114:vg_nvme/lv_3", "target": ["mon-mgr", ""]}]: dispatch 2024-04-23T10:20:11.492 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 23 10:20:11 smithi119 ceph-mon[32608]: from='mgr.14152 172.21.15.114:0/1529706138' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-04-23T10:20:11.492 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 23 10:20:11 smithi119 ceph-mon[32608]: from='mgr.14152 172.21.15.114:0/1529706138' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-04-23T10:20:11.493 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 23 10:20:11 smithi119 ceph-mon[32608]: from='mgr.14152 172.21.15.114:0/1529706138' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-23T10:20:11.578 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 23 10:20:11 smithi204 ceph-mon[32649]: from='mgr.14152 172.21.15.114:0/1529706138' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-04-23T10:20:11.578 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 23 10:20:11 smithi204 ceph-mon[32649]: osd.2 [v2:172.21.15.204:6800/799652511,v1:172.21.15.204:6801/799652511] boot 2024-04-23T10:20:11.578 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 23 10:20:11 smithi204 ceph-mon[32649]: osdmap e18: 3 total, 3 up, 3 in 2024-04-23T10:20:11.578 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 23 10:20:11 smithi204 ceph-mon[32649]: from='mgr.14152 172.21.15.114:0/1529706138' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-04-23T10:20:11.578 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 23 10:20:11 smithi204 ceph-mon[32649]: from='client.14286 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi114:vg_nvme/lv_3", "target": ["mon-mgr", ""]}]: dispatch 2024-04-23T10:20:11.578 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 23 10:20:11 smithi204 ceph-mon[32649]: from='mgr.14152 172.21.15.114:0/1529706138' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-04-23T10:20:11.578 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 23 10:20:11 smithi204 ceph-mon[32649]: from='mgr.14152 172.21.15.114:0/1529706138' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-04-23T10:20:11.578 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 23 10:20:11 smithi204 ceph-mon[32649]: from='mgr.14152 172.21.15.114:0/1529706138' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-23T10:20:12.437 INFO:journalctl@ceph.mon.a.smithi114.stdout:Apr 23 10:20:12 smithi114 ceph-mon[29255]: osdmap e19: 3 total, 3 up, 3 in 2024-04-23T10:20:12.437 INFO:journalctl@ceph.mon.a.smithi114.stdout:Apr 23 10:20:12 smithi114 ceph-mon[29255]: pgmap v82: 0 pgs: ; 0 B data, 88 MiB used, 268 GiB / 268 GiB avail 2024-04-23T10:20:12.437 INFO:journalctl@ceph.mon.a.smithi114.stdout:Apr 23 10:20:12 smithi114 ceph-mon[29255]: from='mgr.14152 172.21.15.114:0/1529706138' entity='mgr.x' cmd=[{"prefix": "osd pool create", "format": "json", "pool": ".mgr", "pg_num": 1, "pg_num_min": 1, "pg_num_max": 32, "yes_i_really_mean_it": true}]: dispatch 2024-04-23T10:20:12.492 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 23 10:20:12 smithi119 ceph-mon[32608]: osdmap e19: 3 total, 3 up, 3 in 2024-04-23T10:20:12.492 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 23 10:20:12 smithi119 ceph-mon[32608]: pgmap v82: 0 pgs: ; 0 B data, 88 MiB used, 268 GiB / 268 GiB avail 2024-04-23T10:20:12.492 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 23 10:20:12 smithi119 ceph-mon[32608]: from='mgr.14152 172.21.15.114:0/1529706138' entity='mgr.x' cmd=[{"prefix": "osd pool create", "format": "json", "pool": ".mgr", "pg_num": 1, "pg_num_min": 1, "pg_num_max": 32, "yes_i_really_mean_it": true}]: dispatch 2024-04-23T10:20:12.578 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 23 10:20:12 smithi204 ceph-mon[32649]: osdmap e19: 3 total, 3 up, 3 in 2024-04-23T10:20:12.578 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 23 10:20:12 smithi204 ceph-mon[32649]: pgmap v82: 0 pgs: ; 0 B data, 88 MiB used, 268 GiB / 268 GiB avail 2024-04-23T10:20:12.578 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 23 10:20:12 smithi204 ceph-mon[32649]: from='mgr.14152 172.21.15.114:0/1529706138' entity='mgr.x' cmd=[{"prefix": "osd pool create", "format": "json", "pool": ".mgr", "pg_num": 1, "pg_num_min": 1, "pg_num_max": 32, "yes_i_really_mean_it": true}]: dispatch 2024-04-23T10:20:13.468 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 23 10:20:13 smithi204 ceph-mon[32649]: from='mgr.14152 172.21.15.114:0/1529706138' entity='mgr.x' cmd='[{"prefix": "osd pool create", "format": "json", "pool": ".mgr", "pg_num": 1, "pg_num_min": 1, "pg_num_max": 32, "yes_i_really_mean_it": true}]': finished 2024-04-23T10:20:13.469 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 23 10:20:13 smithi204 ceph-mon[32649]: osdmap e20: 3 total, 3 up, 3 in 2024-04-23T10:20:13.469 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 23 10:20:13 smithi204 ceph-mon[32649]: from='mgr.14152 172.21.15.114:0/1529706138' entity='mgr.x' cmd=[{"prefix": "osd pool application enable", "format": "json", "pool": ".mgr", "app": "mgr", "yes_i_really_mean_it": true}]: dispatch 2024-04-23T10:20:13.469 INFO:journalctl@ceph.osd.2.smithi204.stdout:Apr 23 10:20:13 smithi204 sudo[41426]: ceph : PWD=/ ; USER=root ; COMMAND=/usr/sbin/smartctl -x --json=o /dev/nvme0n1 2024-04-23T10:20:13.469 INFO:journalctl@ceph.osd.2.smithi204.stdout:Apr 23 10:20:13 smithi204 sudo[41426]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-04-23T10:20:13.469 INFO:journalctl@ceph.osd.2.smithi204.stdout:Apr 23 10:20:13 smithi204 sudo[41426]: pam_unix(sudo:session): session opened for user root by (uid=167) 2024-04-23T10:20:13.469 INFO:journalctl@ceph.osd.2.smithi204.stdout:Apr 23 10:20:13 smithi204 sudo[41426]: pam_unix(sudo:session): session closed for user root 2024-04-23T10:20:13.492 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 23 10:20:13 smithi119 ceph-mon[32608]: from='mgr.14152 172.21.15.114:0/1529706138' entity='mgr.x' cmd='[{"prefix": "osd pool create", "format": "json", "pool": ".mgr", "pg_num": 1, "pg_num_min": 1, "pg_num_max": 32, "yes_i_really_mean_it": true}]': finished 2024-04-23T10:20:13.492 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 23 10:20:13 smithi119 ceph-mon[32608]: osdmap e20: 3 total, 3 up, 3 in 2024-04-23T10:20:13.492 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 23 10:20:13 smithi119 ceph-mon[32608]: from='mgr.14152 172.21.15.114:0/1529706138' entity='mgr.x' cmd=[{"prefix": "osd pool application enable", "format": "json", "pool": ".mgr", "app": "mgr", "yes_i_really_mean_it": true}]: dispatch 2024-04-23T10:20:13.492 INFO:journalctl@ceph.osd.1.smithi119.stdout:Apr 23 10:20:13 smithi119 sudo[41080]: ceph : PWD=/ ; USER=root ; COMMAND=/usr/sbin/smartctl -x --json=o /dev/nvme0n1 2024-04-23T10:20:13.492 INFO:journalctl@ceph.osd.1.smithi119.stdout:Apr 23 10:20:13 smithi119 sudo[41080]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-04-23T10:20:13.492 INFO:journalctl@ceph.osd.1.smithi119.stdout:Apr 23 10:20:13 smithi119 sudo[41080]: pam_unix(sudo:session): session opened for user root by (uid=167) 2024-04-23T10:20:13.492 INFO:journalctl@ceph.osd.1.smithi119.stdout:Apr 23 10:20:13 smithi119 sudo[41080]: pam_unix(sudo:session): session closed for user root 2024-04-23T10:20:13.492 INFO:journalctl@ceph.osd.1.smithi119.stdout:Apr 23 10:20:13 smithi119 sudo[41083]: ceph : PWD=/ ; USER=root ; COMMAND=/usr/sbin/nvme intel smart-log-add --json /dev/nvme0n1 2024-04-23T10:20:13.492 INFO:journalctl@ceph.osd.1.smithi119.stdout:Apr 23 10:20:13 smithi119 sudo[41083]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-04-23T10:20:13.493 INFO:journalctl@ceph.osd.1.smithi119.stdout:Apr 23 10:20:13 smithi119 sudo[41083]: pam_unix(sudo:session): session opened for user root by (uid=167) 2024-04-23T10:20:13.493 INFO:journalctl@ceph.osd.1.smithi119.stdout:Apr 23 10:20:13 smithi119 sudo[41083]: pam_unix(sudo:session): session closed for user root 2024-04-23T10:20:13.501 INFO:journalctl@ceph.mon.a.smithi114.stdout:Apr 23 10:20:13 smithi114 ceph-mon[29255]: from='mgr.14152 172.21.15.114:0/1529706138' entity='mgr.x' cmd='[{"prefix": "osd pool create", "format": "json", "pool": ".mgr", "pg_num": 1, "pg_num_min": 1, "pg_num_max": 32, "yes_i_really_mean_it": true}]': finished 2024-04-23T10:20:13.501 INFO:journalctl@ceph.mon.a.smithi114.stdout:Apr 23 10:20:13 smithi114 ceph-mon[29255]: osdmap e20: 3 total, 3 up, 3 in 2024-04-23T10:20:13.501 INFO:journalctl@ceph.mon.a.smithi114.stdout:Apr 23 10:20:13 smithi114 ceph-mon[29255]: from='mgr.14152 172.21.15.114:0/1529706138' entity='mgr.x' cmd=[{"prefix": "osd pool application enable", "format": "json", "pool": ".mgr", "app": "mgr", "yes_i_really_mean_it": true}]: dispatch 2024-04-23T10:20:13.501 INFO:journalctl@ceph.osd.0.smithi114.stdout:Apr 23 10:20:13 smithi114 sudo[46023]: ceph : PWD=/ ; USER=root ; COMMAND=/usr/sbin/smartctl -x --json=o /dev/nvme0n1 2024-04-23T10:20:13.502 INFO:journalctl@ceph.osd.0.smithi114.stdout:Apr 23 10:20:13 smithi114 sudo[46023]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-04-23T10:20:13.502 INFO:journalctl@ceph.osd.0.smithi114.stdout:Apr 23 10:20:13 smithi114 sudo[46023]: pam_unix(sudo:session): session opened for user root by (uid=167) 2024-04-23T10:20:13.502 INFO:journalctl@ceph.osd.0.smithi114.stdout:Apr 23 10:20:13 smithi114 sudo[46023]: pam_unix(sudo:session): session closed for user root 2024-04-23T10:20:13.502 INFO:journalctl@ceph.osd.0.smithi114.stdout:Apr 23 10:20:13 smithi114 sudo[46026]: ceph : PWD=/ ; USER=root ; COMMAND=/usr/sbin/nvme intel smart-log-add --json /dev/nvme0n1 2024-04-23T10:20:13.502 INFO:journalctl@ceph.osd.0.smithi114.stdout:Apr 23 10:20:13 smithi114 sudo[46026]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-04-23T10:20:13.502 INFO:journalctl@ceph.osd.0.smithi114.stdout:Apr 23 10:20:13 smithi114 sudo[46026]: pam_unix(sudo:session): session opened for user root by (uid=167) 2024-04-23T10:20:13.502 INFO:journalctl@ceph.osd.0.smithi114.stdout:Apr 23 10:20:13 smithi114 sudo[46026]: pam_unix(sudo:session): session closed for user root 2024-04-23T10:20:13.754 INFO:journalctl@ceph.mon.a.smithi114.stdout:Apr 23 10:20:13 smithi114 sudo[46049]: ceph : PWD=/ ; USER=root ; COMMAND=/usr/sbin/smartctl -x --json=o /dev/nvme0n1 2024-04-23T10:20:13.754 INFO:journalctl@ceph.mon.a.smithi114.stdout:Apr 23 10:20:13 smithi114 sudo[46049]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-04-23T10:20:13.754 INFO:journalctl@ceph.mon.a.smithi114.stdout:Apr 23 10:20:13 smithi114 sudo[46049]: pam_unix(sudo:session): session opened for user root by (uid=167) 2024-04-23T10:20:13.754 INFO:journalctl@ceph.mon.a.smithi114.stdout:Apr 23 10:20:13 smithi114 sudo[46049]: pam_unix(sudo:session): session closed for user root 2024-04-23T10:20:13.754 INFO:journalctl@ceph.mon.a.smithi114.stdout:Apr 23 10:20:13 smithi114 sudo[46052]: ceph : PWD=/ ; USER=root ; COMMAND=/usr/sbin/nvme intel smart-log-add --json /dev/nvme0n1 2024-04-23T10:20:13.755 INFO:journalctl@ceph.mon.a.smithi114.stdout:Apr 23 10:20:13 smithi114 sudo[46052]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-04-23T10:20:13.755 INFO:journalctl@ceph.mon.a.smithi114.stdout:Apr 23 10:20:13 smithi114 sudo[46052]: pam_unix(sudo:session): session opened for user root by (uid=167) 2024-04-23T10:20:13.755 INFO:journalctl@ceph.mon.a.smithi114.stdout:Apr 23 10:20:13 smithi114 sudo[46052]: pam_unix(sudo:session): session closed for user root 2024-04-23T10:20:13.828 INFO:journalctl@ceph.osd.2.smithi204.stdout:Apr 23 10:20:13 smithi204 sudo[41429]: ceph : PWD=/ ; USER=root ; COMMAND=/usr/sbin/nvme intel smart-log-add --json /dev/nvme0n1 2024-04-23T10:20:13.828 INFO:journalctl@ceph.osd.2.smithi204.stdout:Apr 23 10:20:13 smithi204 sudo[41429]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-04-23T10:20:13.828 INFO:journalctl@ceph.osd.2.smithi204.stdout:Apr 23 10:20:13 smithi204 sudo[41429]: pam_unix(sudo:session): session opened for user root by (uid=167) 2024-04-23T10:20:13.828 INFO:journalctl@ceph.osd.2.smithi204.stdout:Apr 23 10:20:13 smithi204 sudo[41429]: pam_unix(sudo:session): session closed for user root 2024-04-23T10:20:13.829 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 23 10:20:13 smithi204 sudo[41432]: ceph : PWD=/ ; USER=root ; COMMAND=/usr/sbin/smartctl -x --json=o /dev/nvme0n1 2024-04-23T10:20:13.829 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 23 10:20:13 smithi204 sudo[41432]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-04-23T10:20:13.829 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 23 10:20:13 smithi204 sudo[41432]: pam_unix(sudo:session): session opened for user root by (uid=167) 2024-04-23T10:20:13.829 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 23 10:20:13 smithi204 sudo[41432]: pam_unix(sudo:session): session closed for user root 2024-04-23T10:20:13.829 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 23 10:20:13 smithi204 sudo[41435]: ceph : PWD=/ ; USER=root ; COMMAND=/usr/sbin/nvme intel smart-log-add --json /dev/nvme0n1 2024-04-23T10:20:13.829 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 23 10:20:13 smithi204 sudo[41435]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-04-23T10:20:13.829 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 23 10:20:13 smithi204 sudo[41435]: pam_unix(sudo:session): session opened for user root by (uid=167) 2024-04-23T10:20:13.829 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 23 10:20:13 smithi204 sudo[41435]: pam_unix(sudo:session): session closed for user root 2024-04-23T10:20:13.992 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 23 10:20:13 smithi119 sudo[41086]: ceph : PWD=/ ; USER=root ; COMMAND=/usr/sbin/smartctl -x --json=o /dev/nvme0n1 2024-04-23T10:20:13.992 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 23 10:20:13 smithi119 sudo[41086]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-04-23T10:20:13.992 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 23 10:20:13 smithi119 sudo[41086]: pam_unix(sudo:session): session opened for user root by (uid=167) 2024-04-23T10:20:13.992 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 23 10:20:13 smithi119 sudo[41086]: pam_unix(sudo:session): session closed for user root 2024-04-23T10:20:13.992 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 23 10:20:13 smithi119 sudo[41089]: ceph : PWD=/ ; USER=root ; COMMAND=/usr/sbin/nvme intel smart-log-add --json /dev/nvme0n1 2024-04-23T10:20:13.992 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 23 10:20:13 smithi119 sudo[41089]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-04-23T10:20:13.993 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 23 10:20:13 smithi119 sudo[41089]: pam_unix(sudo:session): session opened for user root by (uid=167) 2024-04-23T10:20:13.993 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 23 10:20:13 smithi119 sudo[41089]: pam_unix(sudo:session): session closed for user root 2024-04-23T10:20:14.255 INFO:journalctl@ceph.mon.a.smithi114.stdout:Apr 23 10:20:14 smithi114 ceph-mon[29255]: from='mgr.14152 172.21.15.114:0/1529706138' entity='mgr.x' cmd='[{"prefix": "osd pool application enable", "format": "json", "pool": ".mgr", "app": "mgr", "yes_i_really_mean_it": true}]': finished 2024-04-23T10:20:14.255 INFO:journalctl@ceph.mon.a.smithi114.stdout:Apr 23 10:20:14 smithi114 ceph-mon[29255]: osdmap e21: 3 total, 3 up, 3 in 2024-04-23T10:20:14.255 INFO:journalctl@ceph.mon.a.smithi114.stdout:Apr 23 10:20:14 smithi114 ceph-mon[29255]: from='mgr.14152 172.21.15.114:0/1529706138' entity='mgr.x' 2024-04-23T10:20:14.255 INFO:journalctl@ceph.mon.a.smithi114.stdout:Apr 23 10:20:14 smithi114 ceph-mon[29255]: from='mgr.14152 172.21.15.114:0/1529706138' entity='mgr.x' 2024-04-23T10:20:14.255 INFO:journalctl@ceph.mon.a.smithi114.stdout:Apr 23 10:20:14 smithi114 ceph-mon[29255]: from='mgr.14152 172.21.15.114:0/1529706138' entity='mgr.x' 2024-04-23T10:20:14.255 INFO:journalctl@ceph.mon.a.smithi114.stdout:Apr 23 10:20:14 smithi114 ceph-mon[29255]: from='admin socket' entity='admin socket' cmd='smart' args=[json]: dispatch 2024-04-23T10:20:14.255 INFO:journalctl@ceph.mon.a.smithi114.stdout:Apr 23 10:20:14 smithi114 ceph-mon[29255]: from='admin socket' entity='admin socket' cmd=smart args=[json]: finished 2024-04-23T10:20:14.255 INFO:journalctl@ceph.mon.a.smithi114.stdout:Apr 23 10:20:14 smithi114 ceph-mon[29255]: from='mgr.14152 172.21.15.114:0/1529706138' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-04-23T10:20:14.255 INFO:journalctl@ceph.mon.a.smithi114.stdout:Apr 23 10:20:14 smithi114 ceph-mon[29255]: from='mgr.14152 172.21.15.114:0/1529706138' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-04-23T10:20:14.255 INFO:journalctl@ceph.mon.a.smithi114.stdout:Apr 23 10:20:14 smithi114 ceph-mon[29255]: from='mgr.14152 172.21.15.114:0/1529706138' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-04-23T10:20:14.256 INFO:journalctl@ceph.mon.a.smithi114.stdout:Apr 23 10:20:14 smithi114 ceph-mon[29255]: from='admin socket' entity='admin socket' cmd='smart' args=[json]: dispatch 2024-04-23T10:20:14.256 INFO:journalctl@ceph.mon.a.smithi114.stdout:Apr 23 10:20:14 smithi114 ceph-mon[29255]: from='mgr.14152 172.21.15.114:0/1529706138' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-04-23T10:20:14.256 INFO:journalctl@ceph.mon.a.smithi114.stdout:Apr 23 10:20:14 smithi114 ceph-mon[29255]: from='mgr.14152 172.21.15.114:0/1529706138' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-04-23T10:20:14.256 INFO:journalctl@ceph.mon.a.smithi114.stdout:Apr 23 10:20:14 smithi114 ceph-mon[29255]: from='mgr.14152 172.21.15.114:0/1529706138' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-04-23T10:20:14.256 INFO:journalctl@ceph.mon.a.smithi114.stdout:Apr 23 10:20:14 smithi114 ceph-mon[29255]: from='admin socket' entity='admin socket' cmd=smart args=[json]: finished 2024-04-23T10:20:14.256 INFO:journalctl@ceph.mon.a.smithi114.stdout:Apr 23 10:20:14 smithi114 ceph-mon[29255]: from='admin socket' entity='admin socket' cmd='smart' args=[json]: dispatch 2024-04-23T10:20:14.256 INFO:journalctl@ceph.mon.a.smithi114.stdout:Apr 23 10:20:14 smithi114 ceph-mon[29255]: from='mgr.14152 172.21.15.114:0/1529706138' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-04-23T10:20:14.256 INFO:journalctl@ceph.mon.a.smithi114.stdout:Apr 23 10:20:14 smithi114 ceph-mon[29255]: from='mgr.14152 172.21.15.114:0/1529706138' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-04-23T10:20:14.256 INFO:journalctl@ceph.mon.a.smithi114.stdout:Apr 23 10:20:14 smithi114 ceph-mon[29255]: from='mgr.14152 172.21.15.114:0/1529706138' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-04-23T10:20:14.256 INFO:journalctl@ceph.mon.a.smithi114.stdout:Apr 23 10:20:14 smithi114 ceph-mon[29255]: from='admin socket' entity='admin socket' cmd=smart args=[json]: finished 2024-04-23T10:20:14.256 INFO:journalctl@ceph.mon.a.smithi114.stdout:Apr 23 10:20:14 smithi114 ceph-mon[29255]: from='client.? 172.21.15.114:0/814378052' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "52d2a566-e71c-4bb3-8d6c-5abdd43177c2"}]: dispatch 2024-04-23T10:20:14.256 INFO:journalctl@ceph.mon.a.smithi114.stdout:Apr 23 10:20:14 smithi114 ceph-mon[29255]: from='client.? 172.21.15.114:0/814378052' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "52d2a566-e71c-4bb3-8d6c-5abdd43177c2"}]': finished 2024-04-23T10:20:14.256 INFO:journalctl@ceph.mon.a.smithi114.stdout:Apr 23 10:20:14 smithi114 ceph-mon[29255]: osdmap e22: 4 total, 3 up, 4 in 2024-04-23T10:20:14.257 INFO:journalctl@ceph.mon.a.smithi114.stdout:Apr 23 10:20:14 smithi114 ceph-mon[29255]: from='mgr.14152 172.21.15.114:0/1529706138' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-04-23T10:20:14.257 INFO:journalctl@ceph.mon.a.smithi114.stdout:Apr 23 10:20:14 smithi114 ceph-mon[29255]: pgmap v86: 1 pgs: 1 unknown; 0 B data, 88 MiB used, 268 GiB / 268 GiB avail 2024-04-23T10:20:14.492 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 23 10:20:14 smithi119 ceph-mon[32608]: from='mgr.14152 172.21.15.114:0/1529706138' entity='mgr.x' cmd='[{"prefix": "osd pool application enable", "format": "json", "pool": ".mgr", "app": "mgr", "yes_i_really_mean_it": true}]': finished 2024-04-23T10:20:14.492 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 23 10:20:14 smithi119 ceph-mon[32608]: osdmap e21: 3 total, 3 up, 3 in 2024-04-23T10:20:14.492 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 23 10:20:14 smithi119 ceph-mon[32608]: from='mgr.14152 172.21.15.114:0/1529706138' entity='mgr.x' 2024-04-23T10:20:14.493 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 23 10:20:14 smithi119 ceph-mon[32608]: from='mgr.14152 172.21.15.114:0/1529706138' entity='mgr.x' 2024-04-23T10:20:14.493 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 23 10:20:14 smithi119 ceph-mon[32608]: from='mgr.14152 172.21.15.114:0/1529706138' entity='mgr.x' 2024-04-23T10:20:14.493 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 23 10:20:14 smithi119 ceph-mon[32608]: from='admin socket' entity='admin socket' cmd='smart' args=[json]: dispatch 2024-04-23T10:20:14.493 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 23 10:20:14 smithi119 ceph-mon[32608]: from='admin socket' entity='admin socket' cmd=smart args=[json]: finished 2024-04-23T10:20:14.493 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 23 10:20:14 smithi119 ceph-mon[32608]: from='mgr.14152 172.21.15.114:0/1529706138' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-04-23T10:20:14.493 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 23 10:20:14 smithi119 ceph-mon[32608]: from='mgr.14152 172.21.15.114:0/1529706138' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-04-23T10:20:14.493 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 23 10:20:14 smithi119 ceph-mon[32608]: from='mgr.14152 172.21.15.114:0/1529706138' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-04-23T10:20:14.493 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 23 10:20:14 smithi119 ceph-mon[32608]: from='admin socket' entity='admin socket' cmd='smart' args=[json]: dispatch 2024-04-23T10:20:14.493 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 23 10:20:14 smithi119 ceph-mon[32608]: from='mgr.14152 172.21.15.114:0/1529706138' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-04-23T10:20:14.493 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 23 10:20:14 smithi119 ceph-mon[32608]: from='mgr.14152 172.21.15.114:0/1529706138' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-04-23T10:20:14.493 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 23 10:20:14 smithi119 ceph-mon[32608]: from='mgr.14152 172.21.15.114:0/1529706138' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-04-23T10:20:14.493 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 23 10:20:14 smithi119 ceph-mon[32608]: from='admin socket' entity='admin socket' cmd=smart args=[json]: finished 2024-04-23T10:20:14.494 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 23 10:20:14 smithi119 ceph-mon[32608]: from='admin socket' entity='admin socket' cmd='smart' args=[json]: dispatch 2024-04-23T10:20:14.494 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 23 10:20:14 smithi119 ceph-mon[32608]: from='mgr.14152 172.21.15.114:0/1529706138' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-04-23T10:20:14.494 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 23 10:20:14 smithi119 ceph-mon[32608]: from='mgr.14152 172.21.15.114:0/1529706138' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-04-23T10:20:14.494 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 23 10:20:14 smithi119 ceph-mon[32608]: from='mgr.14152 172.21.15.114:0/1529706138' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-04-23T10:20:14.494 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 23 10:20:14 smithi119 ceph-mon[32608]: from='admin socket' entity='admin socket' cmd=smart args=[json]: finished 2024-04-23T10:20:14.494 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 23 10:20:14 smithi119 ceph-mon[32608]: from='client.? 172.21.15.114:0/814378052' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "52d2a566-e71c-4bb3-8d6c-5abdd43177c2"}]: dispatch 2024-04-23T10:20:14.494 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 23 10:20:14 smithi119 ceph-mon[32608]: from='client.? 172.21.15.114:0/814378052' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "52d2a566-e71c-4bb3-8d6c-5abdd43177c2"}]': finished 2024-04-23T10:20:14.494 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 23 10:20:14 smithi119 ceph-mon[32608]: osdmap e22: 4 total, 3 up, 4 in 2024-04-23T10:20:14.494 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 23 10:20:14 smithi119 ceph-mon[32608]: from='mgr.14152 172.21.15.114:0/1529706138' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-04-23T10:20:14.494 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 23 10:20:14 smithi119 ceph-mon[32608]: pgmap v86: 1 pgs: 1 unknown; 0 B data, 88 MiB used, 268 GiB / 268 GiB avail 2024-04-23T10:20:14.578 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 23 10:20:14 smithi204 ceph-mon[32649]: from='mgr.14152 172.21.15.114:0/1529706138' entity='mgr.x' cmd='[{"prefix": "osd pool application enable", "format": "json", "pool": ".mgr", "app": "mgr", "yes_i_really_mean_it": true}]': finished 2024-04-23T10:20:14.578 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 23 10:20:14 smithi204 ceph-mon[32649]: osdmap e21: 3 total, 3 up, 3 in 2024-04-23T10:20:14.578 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 23 10:20:14 smithi204 ceph-mon[32649]: from='mgr.14152 172.21.15.114:0/1529706138' entity='mgr.x' 2024-04-23T10:20:14.578 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 23 10:20:14 smithi204 ceph-mon[32649]: from='mgr.14152 172.21.15.114:0/1529706138' entity='mgr.x' 2024-04-23T10:20:14.579 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 23 10:20:14 smithi204 ceph-mon[32649]: from='mgr.14152 172.21.15.114:0/1529706138' entity='mgr.x' 2024-04-23T10:20:14.579 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 23 10:20:14 smithi204 ceph-mon[32649]: from='admin socket' entity='admin socket' cmd='smart' args=[json]: dispatch 2024-04-23T10:20:14.579 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 23 10:20:14 smithi204 ceph-mon[32649]: from='admin socket' entity='admin socket' cmd=smart args=[json]: finished 2024-04-23T10:20:14.579 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 23 10:20:14 smithi204 ceph-mon[32649]: from='mgr.14152 172.21.15.114:0/1529706138' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-04-23T10:20:14.579 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 23 10:20:14 smithi204 ceph-mon[32649]: from='mgr.14152 172.21.15.114:0/1529706138' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-04-23T10:20:14.579 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 23 10:20:14 smithi204 ceph-mon[32649]: from='mgr.14152 172.21.15.114:0/1529706138' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-04-23T10:20:14.579 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 23 10:20:14 smithi204 ceph-mon[32649]: from='admin socket' entity='admin socket' cmd='smart' args=[json]: dispatch 2024-04-23T10:20:14.579 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 23 10:20:14 smithi204 ceph-mon[32649]: from='mgr.14152 172.21.15.114:0/1529706138' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-04-23T10:20:14.579 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 23 10:20:14 smithi204 ceph-mon[32649]: from='mgr.14152 172.21.15.114:0/1529706138' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-04-23T10:20:14.579 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 23 10:20:14 smithi204 ceph-mon[32649]: from='mgr.14152 172.21.15.114:0/1529706138' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-04-23T10:20:14.579 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 23 10:20:14 smithi204 ceph-mon[32649]: from='admin socket' entity='admin socket' cmd=smart args=[json]: finished 2024-04-23T10:20:14.579 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 23 10:20:14 smithi204 ceph-mon[32649]: from='admin socket' entity='admin socket' cmd='smart' args=[json]: dispatch 2024-04-23T10:20:14.579 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 23 10:20:14 smithi204 ceph-mon[32649]: from='mgr.14152 172.21.15.114:0/1529706138' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-04-23T10:20:14.579 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 23 10:20:14 smithi204 ceph-mon[32649]: from='mgr.14152 172.21.15.114:0/1529706138' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-04-23T10:20:14.580 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 23 10:20:14 smithi204 ceph-mon[32649]: from='mgr.14152 172.21.15.114:0/1529706138' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-04-23T10:20:14.580 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 23 10:20:14 smithi204 ceph-mon[32649]: from='admin socket' entity='admin socket' cmd=smart args=[json]: finished 2024-04-23T10:20:14.580 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 23 10:20:14 smithi204 ceph-mon[32649]: from='client.? 172.21.15.114:0/814378052' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "52d2a566-e71c-4bb3-8d6c-5abdd43177c2"}]: dispatch 2024-04-23T10:20:14.580 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 23 10:20:14 smithi204 ceph-mon[32649]: from='client.? 172.21.15.114:0/814378052' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "52d2a566-e71c-4bb3-8d6c-5abdd43177c2"}]': finished 2024-04-23T10:20:14.580 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 23 10:20:14 smithi204 ceph-mon[32649]: osdmap e22: 4 total, 3 up, 4 in 2024-04-23T10:20:14.580 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 23 10:20:14 smithi204 ceph-mon[32649]: from='mgr.14152 172.21.15.114:0/1529706138' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-04-23T10:20:14.580 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 23 10:20:14 smithi204 ceph-mon[32649]: pgmap v86: 1 pgs: 1 unknown; 0 B data, 88 MiB used, 268 GiB / 268 GiB avail 2024-04-23T10:20:14.681 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:20:14.681+0000 7f2084ff9700 1 -- 172.21.15.114:0/19850472 <== mon.0 v2:172.21.15.114:3300/0 7 ==== mgrmap(e 16) ==== 138647+0+0 (secure 0 0 0) 0x7f2080098be0 con 0x7f2088103550 2024-04-23T10:20:15.492 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 23 10:20:15 smithi119 ceph-mon[32608]: from='client.? 172.21.15.114:0/1875139678' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-04-23T10:20:15.492 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 23 10:20:15 smithi119 ceph-mon[32608]: mgrmap e16: x(active, since 2m), standbys: z, y 2024-04-23T10:20:15.504 INFO:journalctl@ceph.mon.a.smithi114.stdout:Apr 23 10:20:15 smithi114 ceph-mon[29255]: from='client.? 172.21.15.114:0/1875139678' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-04-23T10:20:15.504 INFO:journalctl@ceph.mon.a.smithi114.stdout:Apr 23 10:20:15 smithi114 ceph-mon[29255]: mgrmap e16: x(active, since 2m), standbys: z, y 2024-04-23T10:20:15.578 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 23 10:20:15 smithi204 ceph-mon[32649]: from='client.? 172.21.15.114:0/1875139678' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-04-23T10:20:15.578 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 23 10:20:15 smithi204 ceph-mon[32649]: mgrmap e16: x(active, since 2m), standbys: z, y 2024-04-23T10:20:16.492 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 23 10:20:16 smithi119 ceph-mon[32608]: pgmap v87: 1 pgs: 1 active+clean; 577 KiB data, 89 MiB used, 268 GiB / 268 GiB avail 2024-04-23T10:20:16.504 INFO:journalctl@ceph.mon.a.smithi114.stdout:Apr 23 10:20:16 smithi114 ceph-mon[29255]: pgmap v87: 1 pgs: 1 active+clean; 577 KiB data, 89 MiB used, 268 GiB / 268 GiB avail 2024-04-23T10:20:16.578 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 23 10:20:16 smithi204 ceph-mon[32649]: pgmap v87: 1 pgs: 1 active+clean; 577 KiB data, 89 MiB used, 268 GiB / 268 GiB avail 2024-04-23T10:20:19.041 INFO:journalctl@ceph.mon.a.smithi114.stdout:Apr 23 10:20:18 smithi114 ceph-mon[29255]: pgmap v88: 1 pgs: 1 active+clean; 577 KiB data, 91 MiB used, 268 GiB / 268 GiB avail 2024-04-23T10:20:19.242 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 23 10:20:18 smithi119 ceph-mon[32608]: pgmap v88: 1 pgs: 1 active+clean; 577 KiB data, 91 MiB used, 268 GiB / 268 GiB avail 2024-04-23T10:20:19.328 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 23 10:20:18 smithi204 ceph-mon[32649]: pgmap v88: 1 pgs: 1 active+clean; 577 KiB data, 91 MiB used, 268 GiB / 268 GiB avail 2024-04-23T10:20:21.242 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 23 10:20:20 smithi119 ceph-mon[32608]: pgmap v89: 1 pgs: 1 active+clean; 577 KiB data, 91 MiB used, 268 GiB / 268 GiB avail 2024-04-23T10:20:21.254 INFO:journalctl@ceph.mon.a.smithi114.stdout:Apr 23 10:20:20 smithi114 ceph-mon[29255]: pgmap v89: 1 pgs: 1 active+clean; 577 KiB data, 91 MiB used, 268 GiB / 268 GiB avail 2024-04-23T10:20:21.328 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 23 10:20:20 smithi204 ceph-mon[32649]: pgmap v89: 1 pgs: 1 active+clean; 577 KiB data, 91 MiB used, 268 GiB / 268 GiB avail 2024-04-23T10:20:22.238 INFO:journalctl@ceph.mon.a.smithi114.stdout:Apr 23 10:20:21 smithi114 ceph-mon[29255]: from='mgr.14152 172.21.15.114:0/1529706138' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.3"}]: dispatch 2024-04-23T10:20:22.238 INFO:journalctl@ceph.mon.a.smithi114.stdout:Apr 23 10:20:21 smithi114 ceph-mon[29255]: from='mgr.14152 172.21.15.114:0/1529706138' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-23T10:20:22.238 INFO:journalctl@ceph.mon.a.smithi114.stdout:Apr 23 10:20:21 smithi114 ceph-mon[29255]: Deploying daemon osd.3 on smithi114 2024-04-23T10:20:22.242 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 23 10:20:21 smithi119 ceph-mon[32608]: from='mgr.14152 172.21.15.114:0/1529706138' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.3"}]: dispatch 2024-04-23T10:20:22.242 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 23 10:20:21 smithi119 ceph-mon[32608]: from='mgr.14152 172.21.15.114:0/1529706138' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-23T10:20:22.242 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 23 10:20:21 smithi119 ceph-mon[32608]: Deploying daemon osd.3 on smithi114 2024-04-23T10:20:22.328 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 23 10:20:21 smithi204 ceph-mon[32649]: from='mgr.14152 172.21.15.114:0/1529706138' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.3"}]: dispatch 2024-04-23T10:20:22.328 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 23 10:20:21 smithi204 ceph-mon[32649]: from='mgr.14152 172.21.15.114:0/1529706138' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-23T10:20:22.328 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 23 10:20:21 smithi204 ceph-mon[32649]: Deploying daemon osd.3 on smithi114 2024-04-23T10:20:23.187 INFO:journalctl@ceph.mon.a.smithi114.stdout:Apr 23 10:20:22 smithi114 ceph-mon[29255]: pgmap v90: 1 pgs: 1 active+clean; 577 KiB data, 91 MiB used, 268 GiB / 268 GiB avail 2024-04-23T10:20:23.242 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 23 10:20:22 smithi119 ceph-mon[32608]: pgmap v90: 1 pgs: 1 active+clean; 577 KiB data, 91 MiB used, 268 GiB / 268 GiB avail 2024-04-23T10:20:23.328 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 23 10:20:22 smithi204 ceph-mon[32649]: pgmap v90: 1 pgs: 1 active+clean; 577 KiB data, 91 MiB used, 268 GiB / 268 GiB avail 2024-04-23T10:20:25.188 INFO:journalctl@ceph.mon.a.smithi114.stdout:Apr 23 10:20:24 smithi114 ceph-mon[29255]: pgmap v91: 1 pgs: 1 active+clean; 577 KiB data, 91 MiB used, 268 GiB / 268 GiB avail 2024-04-23T10:20:25.242 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 23 10:20:24 smithi119 ceph-mon[32608]: pgmap v91: 1 pgs: 1 active+clean; 577 KiB data, 91 MiB used, 268 GiB / 268 GiB avail 2024-04-23T10:20:25.328 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 23 10:20:24 smithi204 ceph-mon[32649]: pgmap v91: 1 pgs: 1 active+clean; 577 KiB data, 91 MiB used, 268 GiB / 268 GiB avail 2024-04-23T10:20:27.242 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 23 10:20:26 smithi119 ceph-mon[32608]: pgmap v92: 1 pgs: 1 active+clean; 577 KiB data, 91 MiB used, 268 GiB / 268 GiB avail 2024-04-23T10:20:27.242 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 23 10:20:26 smithi119 ceph-mon[32608]: from='mgr.14152 172.21.15.114:0/1529706138' entity='mgr.x' 2024-04-23T10:20:27.242 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 23 10:20:26 smithi119 ceph-mon[32608]: from='mgr.14152 172.21.15.114:0/1529706138' entity='mgr.x' 2024-04-23T10:20:27.242 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 23 10:20:26 smithi119 ceph-mon[32608]: from='mgr.14152 172.21.15.114:0/1529706138' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-23T10:20:27.254 INFO:journalctl@ceph.mon.a.smithi114.stdout:Apr 23 10:20:26 smithi114 ceph-mon[29255]: pgmap v92: 1 pgs: 1 active+clean; 577 KiB data, 91 MiB used, 268 GiB / 268 GiB avail 2024-04-23T10:20:27.255 INFO:journalctl@ceph.mon.a.smithi114.stdout:Apr 23 10:20:26 smithi114 ceph-mon[29255]: from='mgr.14152 172.21.15.114:0/1529706138' entity='mgr.x' 2024-04-23T10:20:27.255 INFO:journalctl@ceph.mon.a.smithi114.stdout:Apr 23 10:20:26 smithi114 ceph-mon[29255]: from='mgr.14152 172.21.15.114:0/1529706138' entity='mgr.x' 2024-04-23T10:20:27.255 INFO:journalctl@ceph.mon.a.smithi114.stdout:Apr 23 10:20:26 smithi114 ceph-mon[29255]: from='mgr.14152 172.21.15.114:0/1529706138' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-23T10:20:27.328 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 23 10:20:26 smithi204 ceph-mon[32649]: pgmap v92: 1 pgs: 1 active+clean; 577 KiB data, 91 MiB used, 268 GiB / 268 GiB avail 2024-04-23T10:20:27.328 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 23 10:20:26 smithi204 ceph-mon[32649]: from='mgr.14152 172.21.15.114:0/1529706138' entity='mgr.x' 2024-04-23T10:20:27.328 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 23 10:20:26 smithi204 ceph-mon[32649]: from='mgr.14152 172.21.15.114:0/1529706138' entity='mgr.x' 2024-04-23T10:20:27.328 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 23 10:20:26 smithi204 ceph-mon[32649]: from='mgr.14152 172.21.15.114:0/1529706138' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-23T10:20:29.242 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 23 10:20:28 smithi119 ceph-mon[32608]: pgmap v93: 1 pgs: 1 active+clean; 577 KiB data, 91 MiB used, 268 GiB / 268 GiB avail 2024-04-23T10:20:29.254 INFO:journalctl@ceph.mon.a.smithi114.stdout:Apr 23 10:20:28 smithi114 ceph-mon[29255]: pgmap v93: 1 pgs: 1 active+clean; 577 KiB data, 91 MiB used, 268 GiB / 268 GiB avail 2024-04-23T10:20:29.328 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 23 10:20:28 smithi204 ceph-mon[32649]: pgmap v93: 1 pgs: 1 active+clean; 577 KiB data, 91 MiB used, 268 GiB / 268 GiB avail 2024-04-23T10:20:29.946 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:20:29.947+0000 7f2084ff9700 1 -- 172.21.15.114:0/19850472 <== mgr.14152 v2:172.21.15.114:6800/465333808 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+37 (secure 0 0 0) 0x7f208806b8a0 con 0x7f20700a3bb0 2024-04-23T10:20:29.947 INFO:teuthology.orchestra.run.smithi114.stdout:Created osd(s) 3 on host 'smithi114' 2024-04-23T10:20:29.949 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:20:29.949+0000 7f208db19700 1 -- 172.21.15.114:0/19850472 >> [v2:172.21.15.114:6800/465333808,v1:172.21.15.114:6801/465333808] conn(0x7f20700a3bb0 msgr2=0x7f20700a6070 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T10:20:29.949 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:20:29.949+0000 7f208db19700 1 --2- 172.21.15.114:0/19850472 >> [v2:172.21.15.114:6800/465333808,v1:172.21.15.114:6801/465333808] conn(0x7f20700a3bb0 0x7f20700a6070 secure :-1 s=READY pgs=66 cs=0 l=1 rev1=1 crypto rx=0x7f2078005fd0 tx=0x7f2078005e40 comp rx=0 tx=0).stop 2024-04-23T10:20:29.949 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:20:29.950+0000 7f208db19700 1 -- 172.21.15.114:0/19850472 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f2088103550 msgr2=0x7f20881a0180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T10:20:29.949 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:20:29.950+0000 7f208db19700 1 --2- 172.21.15.114:0/19850472 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f2088103550 0x7f20881a0180 secure :-1 s=READY pgs=120 cs=0 l=1 rev1=1 crypto rx=0x7f208000add0 tx=0x7f208000c330 comp rx=0 tx=0).stop 2024-04-23T10:20:29.949 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:20:29.950+0000 7f208db19700 1 -- 172.21.15.114:0/19850472 shutdown_connections 2024-04-23T10:20:29.949 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:20:29.950+0000 7f208db19700 1 --2- 172.21.15.114:0/19850472 >> [v2:172.21.15.114:6800/465333808,v1:172.21.15.114:6801/465333808] conn(0x7f20700a3bb0 0x7f20700a6070 unknown :-1 s=CLOSED pgs=66 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:20:29.949 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:20:29.950+0000 7f208db19700 1 --2- 172.21.15.114:0/19850472 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f2088103550 0x7f20881a0180 unknown :-1 s=CLOSED pgs=120 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:20:29.949 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:20:29.950+0000 7f208db19700 1 --2- 172.21.15.114:0/19850472 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f20881019b0 0x7f2088198bc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:20:29.949 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:20:29.950+0000 7f208db19700 1 --2- 172.21.15.114:0/19850472 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f2088102bb0 0x7f2088199100 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:20:29.949 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:20:29.950+0000 7f208db19700 1 -- 172.21.15.114:0/19850472 >> 172.21.15.114:0/19850472 conn(0x7f20880fcf70 msgr2=0x7f208806b190 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T10:20:29.950 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:20:29.950+0000 7f208db19700 1 -- 172.21.15.114:0/19850472 shutdown_connections 2024-04-23T10:20:29.950 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:20:29.950+0000 7f208db19700 1 -- 172.21.15.114:0/19850472 wait complete. 2024-04-23T10:20:30.205 INFO:journalctl@ceph.mon.a.smithi114.stdout:Apr 23 10:20:30 smithi114 ceph-mon[29255]: from='mgr.14152 172.21.15.114:0/1529706138' entity='mgr.x' 2024-04-23T10:20:30.205 INFO:journalctl@ceph.mon.a.smithi114.stdout:Apr 23 10:20:30 smithi114 ceph-mon[29255]: from='mgr.14152 172.21.15.114:0/1529706138' entity='mgr.x' 2024-04-23T10:20:30.205 INFO:journalctl@ceph.mon.a.smithi114.stdout:Apr 23 10:20:30 smithi114 ceph-mon[29255]: from='mgr.14152 172.21.15.114:0/1529706138' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-23T10:20:30.206 INFO:journalctl@ceph.mon.a.smithi114.stdout:Apr 23 10:20:30 smithi114 ceph-mon[29255]: from='mgr.14152 172.21.15.114:0/1529706138' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-23T10:20:30.206 INFO:journalctl@ceph.mon.a.smithi114.stdout:Apr 23 10:20:30 smithi114 ceph-mon[29255]: from='mgr.14152 172.21.15.114:0/1529706138' entity='mgr.x' 2024-04-23T10:20:30.206 INFO:journalctl@ceph.mon.a.smithi114.stdout:Apr 23 10:20:30 smithi114 ceph-mon[29255]: from='mgr.14152 172.21.15.114:0/1529706138' entity='mgr.x' 2024-04-23T10:20:30.206 INFO:journalctl@ceph.mon.a.smithi114.stdout:Apr 23 10:20:30 smithi114 ceph-mon[29255]: from='mgr.14152 172.21.15.114:0/1529706138' entity='mgr.x' 2024-04-23T10:20:30.206 INFO:journalctl@ceph.mon.a.smithi114.stdout:Apr 23 10:20:30 smithi114 ceph-mon[29255]: from='mgr.14152 172.21.15.114:0/1529706138' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-23T10:20:30.328 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 23 10:20:30 smithi204 ceph-mon[32649]: from='mgr.14152 172.21.15.114:0/1529706138' entity='mgr.x' 2024-04-23T10:20:30.328 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 23 10:20:30 smithi204 ceph-mon[32649]: from='mgr.14152 172.21.15.114:0/1529706138' entity='mgr.x' 2024-04-23T10:20:30.328 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 23 10:20:30 smithi204 ceph-mon[32649]: from='mgr.14152 172.21.15.114:0/1529706138' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-23T10:20:30.328 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 23 10:20:30 smithi204 ceph-mon[32649]: from='mgr.14152 172.21.15.114:0/1529706138' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-23T10:20:30.328 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 23 10:20:30 smithi204 ceph-mon[32649]: from='mgr.14152 172.21.15.114:0/1529706138' entity='mgr.x' 2024-04-23T10:20:30.328 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 23 10:20:30 smithi204 ceph-mon[32649]: from='mgr.14152 172.21.15.114:0/1529706138' entity='mgr.x' 2024-04-23T10:20:30.328 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 23 10:20:30 smithi204 ceph-mon[32649]: from='mgr.14152 172.21.15.114:0/1529706138' entity='mgr.x' 2024-04-23T10:20:30.329 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 23 10:20:30 smithi204 ceph-mon[32649]: from='mgr.14152 172.21.15.114:0/1529706138' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-23T10:20:30.492 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 23 10:20:30 smithi119 ceph-mon[32608]: from='mgr.14152 172.21.15.114:0/1529706138' entity='mgr.x' 2024-04-23T10:20:30.492 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 23 10:20:30 smithi119 ceph-mon[32608]: from='mgr.14152 172.21.15.114:0/1529706138' entity='mgr.x' 2024-04-23T10:20:30.492 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 23 10:20:30 smithi119 ceph-mon[32608]: from='mgr.14152 172.21.15.114:0/1529706138' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-23T10:20:30.492 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 23 10:20:30 smithi119 ceph-mon[32608]: from='mgr.14152 172.21.15.114:0/1529706138' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-23T10:20:30.492 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 23 10:20:30 smithi119 ceph-mon[32608]: from='mgr.14152 172.21.15.114:0/1529706138' entity='mgr.x' 2024-04-23T10:20:30.492 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 23 10:20:30 smithi119 ceph-mon[32608]: from='mgr.14152 172.21.15.114:0/1529706138' entity='mgr.x' 2024-04-23T10:20:30.493 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 23 10:20:30 smithi119 ceph-mon[32608]: from='mgr.14152 172.21.15.114:0/1529706138' entity='mgr.x' 2024-04-23T10:20:30.493 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 23 10:20:30 smithi119 ceph-mon[32608]: from='mgr.14152 172.21.15.114:0/1529706138' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-23T10:20:30.711 DEBUG:teuthology.orchestra.run.smithi114:osd.3> sudo journalctl -f -n 0 -u ceph-aa240572-0159-11ef-bc93-c7b262605968@osd.3.service 2024-04-23T10:20:30.714 INFO:tasks.cephadm:Deploying osd.4 on smithi119 with /dev/vg_nvme/lv_3... 2024-04-23T10:20:30.714 DEBUG:teuthology.orchestra.run.smithi119:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d69f13d827952bb07bb9b7973bca14f97d764d6f shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid aa240572-0159-11ef-bc93-c7b262605968 -- ceph-volume lvm zap /dev/vg_nvme/lv_3 2024-04-23T10:20:30.958 INFO:teuthology.orchestra.run.smithi119.stderr:Inferring config /var/lib/ceph/aa240572-0159-11ef-bc93-c7b262605968/mon.b/config 2024-04-23T10:20:31.229 INFO:journalctl@ceph.mon.a.smithi114.stdout:Apr 23 10:20:31 smithi114 ceph-mon[29255]: pgmap v94: 1 pgs: 1 active+clean; 577 KiB data, 91 MiB used, 268 GiB / 268 GiB avail 2024-04-23T10:20:31.229 INFO:journalctl@ceph.mon.a.smithi114.stdout:Apr 23 10:20:31 smithi114 ceph-mon[29255]: from='osd.3 [v2:172.21.15.114:6810/4275595091,v1:172.21.15.114:6811/4275595091]' entity='osd.3' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["3"]}]: dispatch 2024-04-23T10:20:31.309 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 23 10:20:31 smithi119 ceph-mon[32608]: pgmap v94: 1 pgs: 1 active+clean; 577 KiB data, 91 MiB used, 268 GiB / 268 GiB avail 2024-04-23T10:20:31.309 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 23 10:20:31 smithi119 ceph-mon[32608]: from='osd.3 [v2:172.21.15.114:6810/4275595091,v1:172.21.15.114:6811/4275595091]' entity='osd.3' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["3"]}]: dispatch 2024-04-23T10:20:31.328 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 23 10:20:31 smithi204 ceph-mon[32649]: pgmap v94: 1 pgs: 1 active+clean; 577 KiB data, 91 MiB used, 268 GiB / 268 GiB avail 2024-04-23T10:20:31.328 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 23 10:20:31 smithi204 ceph-mon[32649]: from='osd.3 [v2:172.21.15.114:6810/4275595091,v1:172.21.15.114:6811/4275595091]' entity='osd.3' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["3"]}]: dispatch 2024-04-23T10:20:32.134 INFO:journalctl@ceph.mon.a.smithi114.stdout:Apr 23 10:20:32 smithi114 ceph-mon[29255]: from='osd.3 [v2:172.21.15.114:6810/4275595091,v1:172.21.15.114:6811/4275595091]' entity='osd.3' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["3"]}]': finished 2024-04-23T10:20:32.135 INFO:journalctl@ceph.mon.a.smithi114.stdout:Apr 23 10:20:32 smithi114 ceph-mon[29255]: osdmap e23: 4 total, 3 up, 4 in 2024-04-23T10:20:32.135 INFO:journalctl@ceph.mon.a.smithi114.stdout:Apr 23 10:20:32 smithi114 ceph-mon[29255]: from='osd.3 [v2:172.21.15.114:6810/4275595091,v1:172.21.15.114:6811/4275595091]' entity='osd.3' cmd=[{"prefix": "osd crush create-or-move", "id": 3, "weight":0.0873, "args": ["host=smithi114", "root=default"]}]: dispatch 2024-04-23T10:20:32.135 INFO:journalctl@ceph.mon.a.smithi114.stdout:Apr 23 10:20:32 smithi114 ceph-mon[29255]: from='mgr.14152 172.21.15.114:0/1529706138' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-04-23T10:20:32.135 INFO:journalctl@ceph.mon.a.smithi114.stdout:Apr 23 10:20:32 smithi114 ceph-mon[29255]: from='osd.3 [v2:172.21.15.114:6810/4275595091,v1:172.21.15.114:6811/4275595091]' entity='osd.3' cmd='[{"prefix": "osd crush create-or-move", "id": 3, "weight":0.0873, "args": ["host=smithi114", "root=default"]}]': finished 2024-04-23T10:20:32.135 INFO:journalctl@ceph.osd.3.smithi114.stdout:Apr 23 10:20:32 smithi114 ceph-aa240572-0159-11ef-bc93-c7b262605968-osd-3[48091]: 2024-04-23T10:20:32.025+0000 7f1aa3ef8700 -1 osd.3 0 waiting for initial osdmap 2024-04-23T10:20:32.136 INFO:journalctl@ceph.osd.3.smithi114.stdout:Apr 23 10:20:32 smithi114 ceph-aa240572-0159-11ef-bc93-c7b262605968-osd-3[48091]: 2024-04-23T10:20:32.032+0000 7f1a9fceb700 -1 osd.3 24 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-04-23T10:20:32.290 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 23 10:20:32 smithi119 ceph-mon[32608]: from='osd.3 [v2:172.21.15.114:6810/4275595091,v1:172.21.15.114:6811/4275595091]' entity='osd.3' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["3"]}]': finished 2024-04-23T10:20:32.290 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 23 10:20:32 smithi119 ceph-mon[32608]: osdmap e23: 4 total, 3 up, 4 in 2024-04-23T10:20:32.290 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 23 10:20:32 smithi119 ceph-mon[32608]: from='osd.3 [v2:172.21.15.114:6810/4275595091,v1:172.21.15.114:6811/4275595091]' entity='osd.3' cmd=[{"prefix": "osd crush create-or-move", "id": 3, "weight":0.0873, "args": ["host=smithi114", "root=default"]}]: dispatch 2024-04-23T10:20:32.290 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 23 10:20:32 smithi119 ceph-mon[32608]: from='mgr.14152 172.21.15.114:0/1529706138' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-04-23T10:20:32.290 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 23 10:20:32 smithi119 ceph-mon[32608]: from='osd.3 [v2:172.21.15.114:6810/4275595091,v1:172.21.15.114:6811/4275595091]' entity='osd.3' cmd='[{"prefix": "osd crush create-or-move", "id": 3, "weight":0.0873, "args": ["host=smithi114", "root=default"]}]': finished 2024-04-23T10:20:32.328 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 23 10:20:32 smithi204 ceph-mon[32649]: from='osd.3 [v2:172.21.15.114:6810/4275595091,v1:172.21.15.114:6811/4275595091]' entity='osd.3' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["3"]}]': finished 2024-04-23T10:20:32.328 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 23 10:20:32 smithi204 ceph-mon[32649]: osdmap e23: 4 total, 3 up, 4 in 2024-04-23T10:20:32.328 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 23 10:20:32 smithi204 ceph-mon[32649]: from='osd.3 [v2:172.21.15.114:6810/4275595091,v1:172.21.15.114:6811/4275595091]' entity='osd.3' cmd=[{"prefix": "osd crush create-or-move", "id": 3, "weight":0.0873, "args": ["host=smithi114", "root=default"]}]: dispatch 2024-04-23T10:20:32.328 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 23 10:20:32 smithi204 ceph-mon[32649]: from='mgr.14152 172.21.15.114:0/1529706138' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-04-23T10:20:32.328 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 23 10:20:32 smithi204 ceph-mon[32649]: from='osd.3 [v2:172.21.15.114:6810/4275595091,v1:172.21.15.114:6811/4275595091]' entity='osd.3' cmd='[{"prefix": "osd crush create-or-move", "id": 3, "weight":0.0873, "args": ["host=smithi114", "root=default"]}]': finished 2024-04-23T10:20:32.975 INFO:teuthology.orchestra.run.smithi119.stderr:--> Zapping: /dev/vg_nvme/lv_3 2024-04-23T10:20:32.975 INFO:teuthology.orchestra.run.smithi119.stderr:Running command: /usr/bin/dd if=/dev/zero of=/dev/vg_nvme/lv_3 bs=1M count=10 conv=fsync 2024-04-23T10:20:32.975 INFO:teuthology.orchestra.run.smithi119.stderr: stderr: 10+0 records in 2024-04-23T10:20:32.975 INFO:teuthology.orchestra.run.smithi119.stderr:10+0 records out 2024-04-23T10:20:32.975 INFO:teuthology.orchestra.run.smithi119.stderr: stderr: 10485760 bytes (10 MB, 10 MiB) copied, 0.0112228 s, 934 MB/s 2024-04-23T10:20:32.975 INFO:teuthology.orchestra.run.smithi119.stderr:--> Zapping successful for: 2024-04-23T10:20:33.242 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 23 10:20:33 smithi119 ceph-mon[32608]: pgmap v96: 1 pgs: 1 active+clean; 577 KiB data, 91 MiB used, 268 GiB / 268 GiB avail 2024-04-23T10:20:33.242 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 23 10:20:33 smithi119 ceph-mon[32608]: osdmap e24: 4 total, 3 up, 4 in 2024-04-23T10:20:33.242 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 23 10:20:33 smithi119 ceph-mon[32608]: from='mgr.14152 172.21.15.114:0/1529706138' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-04-23T10:20:33.243 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 23 10:20:33 smithi119 ceph-mon[32608]: from='mgr.14152 172.21.15.114:0/1529706138' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-04-23T10:20:33.254 INFO:journalctl@ceph.mon.a.smithi114.stdout:Apr 23 10:20:33 smithi114 ceph-mon[29255]: pgmap v96: 1 pgs: 1 active+clean; 577 KiB data, 91 MiB used, 268 GiB / 268 GiB avail 2024-04-23T10:20:33.254 INFO:journalctl@ceph.mon.a.smithi114.stdout:Apr 23 10:20:33 smithi114 ceph-mon[29255]: osdmap e24: 4 total, 3 up, 4 in 2024-04-23T10:20:33.254 INFO:journalctl@ceph.mon.a.smithi114.stdout:Apr 23 10:20:33 smithi114 ceph-mon[29255]: from='mgr.14152 172.21.15.114:0/1529706138' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-04-23T10:20:33.254 INFO:journalctl@ceph.mon.a.smithi114.stdout:Apr 23 10:20:33 smithi114 ceph-mon[29255]: from='mgr.14152 172.21.15.114:0/1529706138' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-04-23T10:20:33.328 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 23 10:20:33 smithi204 ceph-mon[32649]: pgmap v96: 1 pgs: 1 active+clean; 577 KiB data, 91 MiB used, 268 GiB / 268 GiB avail 2024-04-23T10:20:33.328 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 23 10:20:33 smithi204 ceph-mon[32649]: osdmap e24: 4 total, 3 up, 4 in 2024-04-23T10:20:33.328 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 23 10:20:33 smithi204 ceph-mon[32649]: from='mgr.14152 172.21.15.114:0/1529706138' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-04-23T10:20:33.328 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 23 10:20:33 smithi204 ceph-mon[32649]: from='mgr.14152 172.21.15.114:0/1529706138' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-04-23T10:20:33.590 DEBUG:teuthology.orchestra.run.smithi119:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d69f13d827952bb07bb9b7973bca14f97d764d6f shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid aa240572-0159-11ef-bc93-c7b262605968 -- ceph orch daemon add osd smithi119:vg_nvme/lv_3 2024-04-23T10:20:33.828 INFO:teuthology.orchestra.run.smithi119.stderr:Inferring config /var/lib/ceph/aa240572-0159-11ef-bc93-c7b262605968/mon.b/config 2024-04-23T10:20:34.242 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 23 10:20:34 smithi119 ceph-mon[32608]: purged_snaps scrub starts 2024-04-23T10:20:34.242 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 23 10:20:34 smithi119 ceph-mon[32608]: purged_snaps scrub ok 2024-04-23T10:20:34.242 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 23 10:20:34 smithi119 ceph-mon[32608]: osd.3 [v2:172.21.15.114:6810/4275595091,v1:172.21.15.114:6811/4275595091] boot 2024-04-23T10:20:34.243 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 23 10:20:34 smithi119 ceph-mon[32608]: osdmap e25: 4 total, 4 up, 4 in 2024-04-23T10:20:34.243 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 23 10:20:34 smithi119 ceph-mon[32608]: from='mgr.14152 172.21.15.114:0/1529706138' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-04-23T10:20:34.243 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 23 10:20:34 smithi119 ceph-mon[32608]: osdmap e26: 4 total, 4 up, 4 in 2024-04-23T10:20:34.243 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 23 10:20:34 smithi119 ceph-mon[32608]: from='mgr.14152 172.21.15.114:0/1529706138' entity='mgr.x' 2024-04-23T10:20:34.243 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 23 10:20:34 smithi119 ceph-mon[32608]: from='mgr.14152 172.21.15.114:0/1529706138' entity='mgr.x' 2024-04-23T10:20:34.243 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 23 10:20:34 smithi119 ceph-mon[32608]: from='mgr.14152 172.21.15.114:0/1529706138' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-04-23T10:20:34.243 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 23 10:20:34 smithi119 ceph-mon[32608]: from='mgr.14152 172.21.15.114:0/1529706138' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-04-23T10:20:34.243 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 23 10:20:34 smithi119 ceph-mon[32608]: from='mgr.14152 172.21.15.114:0/1529706138' entity='mgr.x' 2024-04-23T10:20:34.243 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 23 10:20:34 smithi119 ceph-mon[32608]: from='mgr.14152 172.21.15.114:0/1529706138' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-23T10:20:34.243 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 23 10:20:34 smithi119 ceph-mon[32608]: from='mgr.14152 172.21.15.114:0/1529706138' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-23T10:20:34.243 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 23 10:20:34 smithi119 ceph-mon[32608]: from='mgr.14152 172.21.15.114:0/1529706138' entity='mgr.x' 2024-04-23T10:20:34.328 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 23 10:20:34 smithi204 ceph-mon[32649]: purged_snaps scrub starts 2024-04-23T10:20:34.328 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 23 10:20:34 smithi204 ceph-mon[32649]: purged_snaps scrub ok 2024-04-23T10:20:34.328 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 23 10:20:34 smithi204 ceph-mon[32649]: osd.3 [v2:172.21.15.114:6810/4275595091,v1:172.21.15.114:6811/4275595091] boot 2024-04-23T10:20:34.328 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 23 10:20:34 smithi204 ceph-mon[32649]: osdmap e25: 4 total, 4 up, 4 in 2024-04-23T10:20:34.328 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 23 10:20:34 smithi204 ceph-mon[32649]: from='mgr.14152 172.21.15.114:0/1529706138' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-04-23T10:20:34.328 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 23 10:20:34 smithi204 ceph-mon[32649]: osdmap e26: 4 total, 4 up, 4 in 2024-04-23T10:20:34.328 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 23 10:20:34 smithi204 ceph-mon[32649]: from='mgr.14152 172.21.15.114:0/1529706138' entity='mgr.x' 2024-04-23T10:20:34.328 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 23 10:20:34 smithi204 ceph-mon[32649]: from='mgr.14152 172.21.15.114:0/1529706138' entity='mgr.x' 2024-04-23T10:20:34.328 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 23 10:20:34 smithi204 ceph-mon[32649]: from='mgr.14152 172.21.15.114:0/1529706138' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-04-23T10:20:34.328 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 23 10:20:34 smithi204 ceph-mon[32649]: from='mgr.14152 172.21.15.114:0/1529706138' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-04-23T10:20:34.329 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 23 10:20:34 smithi204 ceph-mon[32649]: from='mgr.14152 172.21.15.114:0/1529706138' entity='mgr.x' 2024-04-23T10:20:34.329 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 23 10:20:34 smithi204 ceph-mon[32649]: from='mgr.14152 172.21.15.114:0/1529706138' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-23T10:20:34.329 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 23 10:20:34 smithi204 ceph-mon[32649]: from='mgr.14152 172.21.15.114:0/1529706138' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-23T10:20:34.329 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 23 10:20:34 smithi204 ceph-mon[32649]: from='mgr.14152 172.21.15.114:0/1529706138' entity='mgr.x' 2024-04-23T10:20:34.504 INFO:journalctl@ceph.mon.a.smithi114.stdout:Apr 23 10:20:34 smithi114 ceph-mon[29255]: purged_snaps scrub starts 2024-04-23T10:20:34.504 INFO:journalctl@ceph.mon.a.smithi114.stdout:Apr 23 10:20:34 smithi114 ceph-mon[29255]: purged_snaps scrub ok 2024-04-23T10:20:34.504 INFO:journalctl@ceph.mon.a.smithi114.stdout:Apr 23 10:20:34 smithi114 ceph-mon[29255]: osd.3 [v2:172.21.15.114:6810/4275595091,v1:172.21.15.114:6811/4275595091] boot 2024-04-23T10:20:34.505 INFO:journalctl@ceph.mon.a.smithi114.stdout:Apr 23 10:20:34 smithi114 ceph-mon[29255]: osdmap e25: 4 total, 4 up, 4 in 2024-04-23T10:20:34.505 INFO:journalctl@ceph.mon.a.smithi114.stdout:Apr 23 10:20:34 smithi114 ceph-mon[29255]: from='mgr.14152 172.21.15.114:0/1529706138' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-04-23T10:20:34.505 INFO:journalctl@ceph.mon.a.smithi114.stdout:Apr 23 10:20:34 smithi114 ceph-mon[29255]: osdmap e26: 4 total, 4 up, 4 in 2024-04-23T10:20:34.505 INFO:journalctl@ceph.mon.a.smithi114.stdout:Apr 23 10:20:34 smithi114 ceph-mon[29255]: from='mgr.14152 172.21.15.114:0/1529706138' entity='mgr.x' 2024-04-23T10:20:34.505 INFO:journalctl@ceph.mon.a.smithi114.stdout:Apr 23 10:20:34 smithi114 ceph-mon[29255]: from='mgr.14152 172.21.15.114:0/1529706138' entity='mgr.x' 2024-04-23T10:20:34.505 INFO:journalctl@ceph.mon.a.smithi114.stdout:Apr 23 10:20:34 smithi114 ceph-mon[29255]: from='mgr.14152 172.21.15.114:0/1529706138' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-04-23T10:20:34.505 INFO:journalctl@ceph.mon.a.smithi114.stdout:Apr 23 10:20:34 smithi114 ceph-mon[29255]: from='mgr.14152 172.21.15.114:0/1529706138' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-04-23T10:20:34.505 INFO:journalctl@ceph.mon.a.smithi114.stdout:Apr 23 10:20:34 smithi114 ceph-mon[29255]: from='mgr.14152 172.21.15.114:0/1529706138' entity='mgr.x' 2024-04-23T10:20:34.505 INFO:journalctl@ceph.mon.a.smithi114.stdout:Apr 23 10:20:34 smithi114 ceph-mon[29255]: from='mgr.14152 172.21.15.114:0/1529706138' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-23T10:20:34.505 INFO:journalctl@ceph.mon.a.smithi114.stdout:Apr 23 10:20:34 smithi114 ceph-mon[29255]: from='mgr.14152 172.21.15.114:0/1529706138' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-23T10:20:34.505 INFO:journalctl@ceph.mon.a.smithi114.stdout:Apr 23 10:20:34 smithi114 ceph-mon[29255]: from='mgr.14152 172.21.15.114:0/1529706138' entity='mgr.x' 2024-04-23T10:20:35.160 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 23 10:20:35 smithi119 ceph-mon[32608]: Detected new or changed devices on smithi114 2024-04-23T10:20:35.161 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 23 10:20:35 smithi119 ceph-mon[32608]: Adjusting osd_memory_target on smithi114 to 8544M 2024-04-23T10:20:35.161 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 23 10:20:35 smithi119 ceph-mon[32608]: pgmap v100: 1 pgs: 1 active+clean; 577 KiB data, 91 MiB used, 268 GiB / 268 GiB avail 2024-04-23T10:20:35.161 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 23 10:20:35 smithi119 ceph-mon[32608]: osdmap e27: 4 total, 4 up, 4 in 2024-04-23T10:20:35.328 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 23 10:20:35 smithi204 ceph-mon[32649]: Detected new or changed devices on smithi114 2024-04-23T10:20:35.328 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 23 10:20:35 smithi204 ceph-mon[32649]: Adjusting osd_memory_target on smithi114 to 8544M 2024-04-23T10:20:35.328 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 23 10:20:35 smithi204 ceph-mon[32649]: pgmap v100: 1 pgs: 1 active+clean; 577 KiB data, 91 MiB used, 268 GiB / 268 GiB avail 2024-04-23T10:20:35.328 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 23 10:20:35 smithi204 ceph-mon[32649]: osdmap e27: 4 total, 4 up, 4 in 2024-04-23T10:20:35.383 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-23T10:20:35.381+0000 7fe9c7f72700 1 -- 172.21.15.119:0/1740534875 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fe9c0103540 msgr2=0x7fe9c0109df0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T10:20:35.383 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-23T10:20:35.381+0000 7fe9c7f72700 1 --2- 172.21.15.119:0/1740534875 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fe9c0103540 0x7fe9c0109df0 secure :-1 s=READY pgs=17 cs=0 l=1 rev1=1 crypto rx=0x7fe9b800b0d0 tx=0x7fe9b800b3e0 comp rx=0 tx=0).stop 2024-04-23T10:20:35.383 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-23T10:20:35.381+0000 7fe9c7f72700 1 -- 172.21.15.119:0/1740534875 shutdown_connections 2024-04-23T10:20:35.383 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-23T10:20:35.381+0000 7fe9c7f72700 1 --2- 172.21.15.119:0/1740534875 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fe9c0103540 0x7fe9c0109df0 unknown :-1 s=CLOSED pgs=17 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:20:35.383 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-23T10:20:35.381+0000 7fe9c7f72700 1 --2- 172.21.15.119:0/1740534875 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fe9c01019a0 0x7fe9c0101dc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:20:35.383 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-23T10:20:35.381+0000 7fe9c7f72700 1 --2- 172.21.15.119:0/1740534875 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fe9c0102ba0 0x7fe9c0103000 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:20:35.383 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-23T10:20:35.381+0000 7fe9c7f72700 1 -- 172.21.15.119:0/1740534875 >> 172.21.15.119:0/1740534875 conn(0x7fe9c00fcf40 msgr2=0x7fe9c00ff380 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T10:20:35.383 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-23T10:20:35.381+0000 7fe9c7f72700 1 -- 172.21.15.119:0/1740534875 shutdown_connections 2024-04-23T10:20:35.383 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-23T10:20:35.381+0000 7fe9c7f72700 1 -- 172.21.15.119:0/1740534875 wait complete. 2024-04-23T10:20:35.384 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-23T10:20:35.382+0000 7fe9c7f72700 1 Processor -- start 2024-04-23T10:20:35.384 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-23T10:20:35.382+0000 7fe9c7f72700 1 -- start start 2024-04-23T10:20:35.384 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-23T10:20:35.382+0000 7fe9c7f72700 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fe9c01019a0 0x7fe9c0197ab0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T10:20:35.384 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-23T10:20:35.382+0000 7fe9c7f72700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fe9c0102ba0 0x7fe9c0197ff0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T10:20:35.384 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-23T10:20:35.382+0000 7fe9c7f72700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fe9c0103540 0x7fe9c019f070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T10:20:35.384 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-23T10:20:35.382+0000 7fe9c7f72700 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x7fe9c010d440 con 0x7fe9c0103540 2024-04-23T10:20:35.384 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-23T10:20:35.382+0000 7fe9c7f72700 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 -- 0x7fe9c010d2c0 con 0x7fe9c0102ba0 2024-04-23T10:20:35.384 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-23T10:20:35.382+0000 7fe9c7f72700 1 -- --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_getmap magic: 0 -- 0x7fe9c010d5c0 con 0x7fe9c01019a0 2024-04-23T10:20:35.384 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-23T10:20:35.382+0000 7fe9c550d700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fe9c0102ba0 0x7fe9c0197ff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T10:20:35.384 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-23T10:20:35.382+0000 7fe9c550d700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fe9c0102ba0 0x7fe9c0197ff0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.119:3300/0 says I am v2:172.21.15.119:52358/0 (socket says 172.21.15.119:52358) 2024-04-23T10:20:35.384 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-23T10:20:35.382+0000 7fe9c550d700 1 -- 172.21.15.119:0/1560854742 learned_addr learned my addr 172.21.15.119:0/1560854742 (peer_addr_for_me v2:172.21.15.119:0/0) 2024-04-23T10:20:35.384 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-23T10:20:35.382+0000 7fe9c5d0e700 1 --2- 172.21.15.119:0/1560854742 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fe9c01019a0 0x7fe9c0197ab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T10:20:35.384 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-23T10:20:35.382+0000 7fe9c650f700 1 --2- 172.21.15.119:0/1560854742 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fe9c0103540 0x7fe9c019f070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T10:20:35.384 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-23T10:20:35.382+0000 7fe9c550d700 1 -- 172.21.15.119:0/1560854742 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fe9c01019a0 msgr2=0x7fe9c0197ab0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T10:20:35.384 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-23T10:20:35.382+0000 7fe9c550d700 1 --2- 172.21.15.119:0/1560854742 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fe9c01019a0 0x7fe9c0197ab0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:20:35.385 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-23T10:20:35.382+0000 7fe9c550d700 1 -- 172.21.15.119:0/1560854742 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fe9c0103540 msgr2=0x7fe9c019f070 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T10:20:35.385 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-23T10:20:35.382+0000 7fe9c550d700 1 --2- 172.21.15.119:0/1560854742 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fe9c0103540 0x7fe9c019f070 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:20:35.385 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-23T10:20:35.382+0000 7fe9c550d700 1 -- 172.21.15.119:0/1560854742 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe9c019f710 con 0x7fe9c0102ba0 2024-04-23T10:20:35.385 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-23T10:20:35.383+0000 7fe9c550d700 1 --2- 172.21.15.119:0/1560854742 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fe9c0102ba0 0x7fe9c0197ff0 secure :-1 s=READY pgs=18 cs=0 l=1 rev1=1 crypto rx=0x7fe9b0009970 tx=0x7fe9b0009c80 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T10:20:35.385 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-23T10:20:35.383+0000 7fe9b6ffd700 1 -- 172.21.15.119:0/1560854742 <== mon.2 v2:172.21.15.119:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe9b001c070 con 0x7fe9c0102ba0 2024-04-23T10:20:35.385 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-23T10:20:35.383+0000 7fe9b6ffd700 1 -- 172.21.15.119:0/1560854742 <== mon.2 v2:172.21.15.119:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fe9b000d260 con 0x7fe9c0102ba0 2024-04-23T10:20:35.385 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-23T10:20:35.383+0000 7fe9c7f72700 1 -- 172.21.15.119:0/1560854742 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fe9c006a780 con 0x7fe9c0102ba0 2024-04-23T10:20:35.385 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-23T10:20:35.383+0000 7fe9b6ffd700 1 -- 172.21.15.119:0/1560854742 <== mon.2 v2:172.21.15.119:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe9b00179d0 con 0x7fe9c0102ba0 2024-04-23T10:20:35.385 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-23T10:20:35.383+0000 7fe9c7f72700 1 -- 172.21.15.119:0/1560854742 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fe9c006acc0 con 0x7fe9c0102ba0 2024-04-23T10:20:35.385 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-23T10:20:35.384+0000 7fe9c7f72700 1 -- 172.21.15.119:0/1560854742 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fe9c0069730 con 0x7fe9c0102ba0 2024-04-23T10:20:35.386 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-23T10:20:35.385+0000 7fe9b6ffd700 1 -- 172.21.15.119:0/1560854742 <== mon.2 v2:172.21.15.119:3300/0 4 ==== mgrmap(e 16) ==== 138647+0+0 (secure 0 0 0) 0x7fe9b001f070 con 0x7fe9c0102ba0 2024-04-23T10:20:35.386 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-23T10:20:35.386+0000 7fe9b6ffd700 1 --2- 172.21.15.119:0/1560854742 >> [v2:172.21.15.114:6800/465333808,v1:172.21.15.114:6801/465333808] conn(0x7fe9ac0a3c40 0x7fe9ac0a6100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T10:20:35.386 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-23T10:20:35.386+0000 7fe9b6ffd700 1 -- 172.21.15.119:0/1560854742 <== mon.2 v2:172.21.15.119:3300/0 5 ==== osd_map(28..28 src has 1..28) ==== 3669+0+0 (secure 0 0 0) 0x7fe9b00d0190 con 0x7fe9c0102ba0 2024-04-23T10:20:35.386 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-23T10:20:35.386+0000 7fe9c5d0e700 1 --2- 172.21.15.119:0/1560854742 >> [v2:172.21.15.114:6800/465333808,v1:172.21.15.114:6801/465333808] conn(0x7fe9ac0a3c40 0x7fe9ac0a6100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T10:20:35.386 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-23T10:20:35.387+0000 7fe9c5d0e700 1 --2- 172.21.15.119:0/1560854742 >> [v2:172.21.15.114:6800/465333808,v1:172.21.15.114:6801/465333808] conn(0x7fe9ac0a3c40 0x7fe9ac0a6100 secure :-1 s=READY pgs=74 cs=0 l=1 rev1=1 crypto rx=0x7fe9bc005950 tx=0x7fe9bc00a300 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T10:20:35.389 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-23T10:20:35.390+0000 7fe9b6ffd700 1 -- 172.21.15.119:0/1560854742 <== mon.2 v2:172.21.15.119:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7fe9b009dd90 con 0x7fe9c0102ba0 2024-04-23T10:20:35.504 INFO:journalctl@ceph.mon.a.smithi114.stdout:Apr 23 10:20:35 smithi114 ceph-mon[29255]: Detected new or changed devices on smithi114 2024-04-23T10:20:35.504 INFO:journalctl@ceph.mon.a.smithi114.stdout:Apr 23 10:20:35 smithi114 ceph-mon[29255]: Adjusting osd_memory_target on smithi114 to 8544M 2024-04-23T10:20:35.504 INFO:journalctl@ceph.mon.a.smithi114.stdout:Apr 23 10:20:35 smithi114 ceph-mon[29255]: pgmap v100: 1 pgs: 1 active+clean; 577 KiB data, 91 MiB used, 268 GiB / 268 GiB avail 2024-04-23T10:20:35.504 INFO:journalctl@ceph.mon.a.smithi114.stdout:Apr 23 10:20:35 smithi114 ceph-mon[29255]: osdmap e27: 4 total, 4 up, 4 in 2024-04-23T10:20:35.596 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-23T10:20:35.596+0000 7fe9c7f72700 1 -- 172.21.15.119:0/1560854742 --> [v2:172.21.15.114:6800/465333808,v1:172.21.15.114:6801/465333808] -- mgr_command(tid 0: {"prefix": "orch daemon add osd", "svc_arg": "smithi119:vg_nvme/lv_3", "target": ["mon-mgr", ""]}) -- 0x7fe9c006afe0 con 0x7fe9ac0a3c40 2024-04-23T10:20:36.492 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 23 10:20:36 smithi119 ceph-mon[32608]: osdmap e28: 4 total, 4 up, 4 in 2024-04-23T10:20:36.492 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 23 10:20:36 smithi119 ceph-mon[32608]: from='client.24224 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi119:vg_nvme/lv_3", "target": ["mon-mgr", ""]}]: dispatch 2024-04-23T10:20:36.492 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 23 10:20:36 smithi119 ceph-mon[32608]: from='mgr.14152 172.21.15.114:0/1529706138' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-04-23T10:20:36.492 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 23 10:20:36 smithi119 ceph-mon[32608]: from='mgr.14152 172.21.15.114:0/1529706138' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-04-23T10:20:36.492 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 23 10:20:36 smithi119 ceph-mon[32608]: from='mgr.14152 172.21.15.114:0/1529706138' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-23T10:20:36.492 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 23 10:20:36 smithi119 ceph-mon[32608]: pgmap v103: 1 pgs: 1 peering; 577 KiB data, 117 MiB used, 357 GiB / 358 GiB avail; 148 KiB/s, 0 objects/s recovering 2024-04-23T10:20:36.578 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 23 10:20:36 smithi204 ceph-mon[32649]: osdmap e28: 4 total, 4 up, 4 in 2024-04-23T10:20:36.578 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 23 10:20:36 smithi204 ceph-mon[32649]: from='client.24224 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi119:vg_nvme/lv_3", "target": ["mon-mgr", ""]}]: dispatch 2024-04-23T10:20:36.578 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 23 10:20:36 smithi204 ceph-mon[32649]: from='mgr.14152 172.21.15.114:0/1529706138' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-04-23T10:20:36.578 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 23 10:20:36 smithi204 ceph-mon[32649]: from='mgr.14152 172.21.15.114:0/1529706138' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-04-23T10:20:36.578 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 23 10:20:36 smithi204 ceph-mon[32649]: from='mgr.14152 172.21.15.114:0/1529706138' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-23T10:20:36.578 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 23 10:20:36 smithi204 ceph-mon[32649]: pgmap v103: 1 pgs: 1 peering; 577 KiB data, 117 MiB used, 357 GiB / 358 GiB avail; 148 KiB/s, 0 objects/s recovering 2024-04-23T10:20:36.754 INFO:journalctl@ceph.mon.a.smithi114.stdout:Apr 23 10:20:36 smithi114 ceph-mon[29255]: osdmap e28: 4 total, 4 up, 4 in 2024-04-23T10:20:36.754 INFO:journalctl@ceph.mon.a.smithi114.stdout:Apr 23 10:20:36 smithi114 ceph-mon[29255]: from='client.24224 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi119:vg_nvme/lv_3", "target": ["mon-mgr", ""]}]: dispatch 2024-04-23T10:20:36.754 INFO:journalctl@ceph.mon.a.smithi114.stdout:Apr 23 10:20:36 smithi114 ceph-mon[29255]: from='mgr.14152 172.21.15.114:0/1529706138' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-04-23T10:20:36.754 INFO:journalctl@ceph.mon.a.smithi114.stdout:Apr 23 10:20:36 smithi114 ceph-mon[29255]: from='mgr.14152 172.21.15.114:0/1529706138' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-04-23T10:20:36.754 INFO:journalctl@ceph.mon.a.smithi114.stdout:Apr 23 10:20:36 smithi114 ceph-mon[29255]: from='mgr.14152 172.21.15.114:0/1529706138' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-23T10:20:36.754 INFO:journalctl@ceph.mon.a.smithi114.stdout:Apr 23 10:20:36 smithi114 ceph-mon[29255]: pgmap v103: 1 pgs: 1 peering; 577 KiB data, 117 MiB used, 357 GiB / 358 GiB avail; 148 KiB/s, 0 objects/s recovering 2024-04-23T10:20:37.380 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 23 10:20:37 smithi119 ceph-mon[32608]: osdmap e29: 4 total, 4 up, 4 in 2024-04-23T10:20:37.578 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 23 10:20:37 smithi204 ceph-mon[32649]: osdmap e29: 4 total, 4 up, 4 in 2024-04-23T10:20:37.754 INFO:journalctl@ceph.mon.a.smithi114.stdout:Apr 23 10:20:37 smithi114 ceph-mon[29255]: osdmap e29: 4 total, 4 up, 4 in 2024-04-23T10:20:38.578 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 23 10:20:38 smithi204 ceph-mon[32649]: pgmap v105: 1 pgs: 1 peering; 577 KiB data, 116 MiB used, 357 GiB / 358 GiB avail; 124 KiB/s, 0 objects/s recovering 2024-04-23T10:20:38.578 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 23 10:20:38 smithi204 ceph-mon[32649]: from='client.? 172.21.15.119:0/3614878736' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "d911a7c1-4bdc-4d28-b8a6-73896a6e7de6"}]: dispatch 2024-04-23T10:20:38.578 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 23 10:20:38 smithi204 ceph-mon[32649]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "d911a7c1-4bdc-4d28-b8a6-73896a6e7de6"}]: dispatch 2024-04-23T10:20:38.578 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 23 10:20:38 smithi204 ceph-mon[32649]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "d911a7c1-4bdc-4d28-b8a6-73896a6e7de6"}]': finished 2024-04-23T10:20:38.578 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 23 10:20:38 smithi204 ceph-mon[32649]: osdmap e30: 5 total, 4 up, 5 in 2024-04-23T10:20:38.578 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 23 10:20:38 smithi204 ceph-mon[32649]: from='mgr.14152 172.21.15.114:0/1529706138' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-04-23T10:20:38.579 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 23 10:20:38 smithi119 ceph-mon[32608]: pgmap v105: 1 pgs: 1 peering; 577 KiB data, 116 MiB used, 357 GiB / 358 GiB avail; 124 KiB/s, 0 objects/s recovering 2024-04-23T10:20:38.579 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 23 10:20:38 smithi119 ceph-mon[32608]: from='client.? 172.21.15.119:0/3614878736' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "d911a7c1-4bdc-4d28-b8a6-73896a6e7de6"}]: dispatch 2024-04-23T10:20:38.579 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 23 10:20:38 smithi119 ceph-mon[32608]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "d911a7c1-4bdc-4d28-b8a6-73896a6e7de6"}]: dispatch 2024-04-23T10:20:38.579 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 23 10:20:38 smithi119 ceph-mon[32608]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "d911a7c1-4bdc-4d28-b8a6-73896a6e7de6"}]': finished 2024-04-23T10:20:38.579 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 23 10:20:38 smithi119 ceph-mon[32608]: osdmap e30: 5 total, 4 up, 5 in 2024-04-23T10:20:38.579 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 23 10:20:38 smithi119 ceph-mon[32608]: from='mgr.14152 172.21.15.114:0/1529706138' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-04-23T10:20:38.754 INFO:journalctl@ceph.mon.a.smithi114.stdout:Apr 23 10:20:38 smithi114 ceph-mon[29255]: pgmap v105: 1 pgs: 1 peering; 577 KiB data, 116 MiB used, 357 GiB / 358 GiB avail; 124 KiB/s, 0 objects/s recovering 2024-04-23T10:20:38.754 INFO:journalctl@ceph.mon.a.smithi114.stdout:Apr 23 10:20:38 smithi114 ceph-mon[29255]: from='client.? 172.21.15.119:0/3614878736' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "d911a7c1-4bdc-4d28-b8a6-73896a6e7de6"}]: dispatch 2024-04-23T10:20:38.754 INFO:journalctl@ceph.mon.a.smithi114.stdout:Apr 23 10:20:38 smithi114 ceph-mon[29255]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "d911a7c1-4bdc-4d28-b8a6-73896a6e7de6"}]: dispatch 2024-04-23T10:20:38.754 INFO:journalctl@ceph.mon.a.smithi114.stdout:Apr 23 10:20:38 smithi114 ceph-mon[29255]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "d911a7c1-4bdc-4d28-b8a6-73896a6e7de6"}]': finished 2024-04-23T10:20:38.754 INFO:journalctl@ceph.mon.a.smithi114.stdout:Apr 23 10:20:38 smithi114 ceph-mon[29255]: osdmap e30: 5 total, 4 up, 5 in 2024-04-23T10:20:38.754 INFO:journalctl@ceph.mon.a.smithi114.stdout:Apr 23 10:20:38 smithi114 ceph-mon[29255]: from='mgr.14152 172.21.15.114:0/1529706138' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-04-23T10:20:39.578 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 23 10:20:39 smithi204 ceph-mon[32649]: from='client.? 172.21.15.119:0/3548374416' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-04-23T10:20:39.742 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 23 10:20:39 smithi119 ceph-mon[32608]: from='client.? 172.21.15.119:0/3548374416' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-04-23T10:20:39.754 INFO:journalctl@ceph.mon.a.smithi114.stdout:Apr 23 10:20:39 smithi114 ceph-mon[29255]: from='client.? 172.21.15.119:0/3548374416' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-04-23T10:20:40.578 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 23 10:20:40 smithi204 ceph-mon[32649]: pgmap v107: 1 pgs: 1 peering; 577 KiB data, 116 MiB used, 357 GiB / 358 GiB avail; 102 KiB/s, 0 objects/s recovering 2024-04-23T10:20:40.742 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 23 10:20:40 smithi119 ceph-mon[32608]: pgmap v107: 1 pgs: 1 peering; 577 KiB data, 116 MiB used, 357 GiB / 358 GiB avail; 102 KiB/s, 0 objects/s recovering 2024-04-23T10:20:40.754 INFO:journalctl@ceph.mon.a.smithi114.stdout:Apr 23 10:20:40 smithi114 ceph-mon[29255]: pgmap v107: 1 pgs: 1 peering; 577 KiB data, 116 MiB used, 357 GiB / 358 GiB avail; 102 KiB/s, 0 objects/s recovering 2024-04-23T10:20:43.242 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 23 10:20:42 smithi119 ceph-mon[32608]: pgmap v108: 1 pgs: 1 active+clean; 577 KiB data, 116 MiB used, 357 GiB / 358 GiB avail; 87 KiB/s, 0 objects/s recovering 2024-04-23T10:20:43.254 INFO:journalctl@ceph.mon.a.smithi114.stdout:Apr 23 10:20:42 smithi114 ceph-mon[29255]: pgmap v108: 1 pgs: 1 active+clean; 577 KiB data, 116 MiB used, 357 GiB / 358 GiB avail; 87 KiB/s, 0 objects/s recovering 2024-04-23T10:20:43.328 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 23 10:20:42 smithi204 ceph-mon[32649]: pgmap v108: 1 pgs: 1 active+clean; 577 KiB data, 116 MiB used, 357 GiB / 358 GiB avail; 87 KiB/s, 0 objects/s recovering 2024-04-23T10:20:45.243 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 23 10:20:44 smithi119 ceph-mon[32608]: pgmap v109: 1 pgs: 1 active+clean; 577 KiB data, 116 MiB used, 357 GiB / 358 GiB avail 2024-04-23T10:20:45.254 INFO:journalctl@ceph.mon.a.smithi114.stdout:Apr 23 10:20:44 smithi114 ceph-mon[29255]: pgmap v109: 1 pgs: 1 active+clean; 577 KiB data, 116 MiB used, 357 GiB / 358 GiB avail 2024-04-23T10:20:45.328 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 23 10:20:44 smithi204 ceph-mon[32649]: pgmap v109: 1 pgs: 1 active+clean; 577 KiB data, 116 MiB used, 357 GiB / 358 GiB avail 2024-04-23T10:20:46.231 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 23 10:20:45 smithi119 ceph-mon[32608]: from='mgr.14152 172.21.15.114:0/1529706138' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.4"}]: dispatch 2024-04-23T10:20:46.232 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 23 10:20:45 smithi119 ceph-mon[32608]: from='mgr.14152 172.21.15.114:0/1529706138' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-23T10:20:46.254 INFO:journalctl@ceph.mon.a.smithi114.stdout:Apr 23 10:20:45 smithi114 ceph-mon[29255]: from='mgr.14152 172.21.15.114:0/1529706138' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.4"}]: dispatch 2024-04-23T10:20:46.254 INFO:journalctl@ceph.mon.a.smithi114.stdout:Apr 23 10:20:45 smithi114 ceph-mon[29255]: from='mgr.14152 172.21.15.114:0/1529706138' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-23T10:20:46.328 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 23 10:20:45 smithi204 ceph-mon[32649]: from='mgr.14152 172.21.15.114:0/1529706138' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.4"}]: dispatch 2024-04-23T10:20:46.328 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 23 10:20:45 smithi204 ceph-mon[32649]: from='mgr.14152 172.21.15.114:0/1529706138' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-23T10:20:47.146 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 23 10:20:46 smithi119 ceph-mon[32608]: Deploying daemon osd.4 on smithi119 2024-04-23T10:20:47.146 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 23 10:20:46 smithi119 ceph-mon[32608]: pgmap v110: 1 pgs: 1 active+clean; 577 KiB data, 116 MiB used, 357 GiB / 358 GiB avail 2024-04-23T10:20:47.254 INFO:journalctl@ceph.mon.a.smithi114.stdout:Apr 23 10:20:46 smithi114 ceph-mon[29255]: Deploying daemon osd.4 on smithi119 2024-04-23T10:20:47.254 INFO:journalctl@ceph.mon.a.smithi114.stdout:Apr 23 10:20:46 smithi114 ceph-mon[29255]: pgmap v110: 1 pgs: 1 active+clean; 577 KiB data, 116 MiB used, 357 GiB / 358 GiB avail 2024-04-23T10:20:47.328 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 23 10:20:46 smithi204 ceph-mon[32649]: Deploying daemon osd.4 on smithi119 2024-04-23T10:20:47.328 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 23 10:20:46 smithi204 ceph-mon[32649]: pgmap v110: 1 pgs: 1 active+clean; 577 KiB data, 116 MiB used, 357 GiB / 358 GiB avail 2024-04-23T10:20:49.144 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 23 10:20:48 smithi119 ceph-mon[32608]: pgmap v111: 1 pgs: 1 active+clean; 577 KiB data, 116 MiB used, 357 GiB / 358 GiB avail 2024-04-23T10:20:49.254 INFO:journalctl@ceph.mon.a.smithi114.stdout:Apr 23 10:20:48 smithi114 ceph-mon[29255]: pgmap v111: 1 pgs: 1 active+clean; 577 KiB data, 116 MiB used, 357 GiB / 358 GiB avail 2024-04-23T10:20:49.328 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 23 10:20:48 smithi204 ceph-mon[32649]: pgmap v111: 1 pgs: 1 active+clean; 577 KiB data, 116 MiB used, 357 GiB / 358 GiB avail 2024-04-23T10:20:51.242 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 23 10:20:50 smithi119 ceph-mon[32608]: pgmap v112: 1 pgs: 1 active+clean; 577 KiB data, 116 MiB used, 357 GiB / 358 GiB avail 2024-04-23T10:20:51.254 INFO:journalctl@ceph.mon.a.smithi114.stdout:Apr 23 10:20:50 smithi114 ceph-mon[29255]: pgmap v112: 1 pgs: 1 active+clean; 577 KiB data, 116 MiB used, 357 GiB / 358 GiB avail 2024-04-23T10:20:51.328 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 23 10:20:50 smithi204 ceph-mon[32649]: pgmap v112: 1 pgs: 1 active+clean; 577 KiB data, 116 MiB used, 357 GiB / 358 GiB avail 2024-04-23T10:20:52.242 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 23 10:20:52 smithi119 ceph-mon[32608]: from='mgr.14152 172.21.15.114:0/1529706138' entity='mgr.x' 2024-04-23T10:20:52.242 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 23 10:20:52 smithi119 ceph-mon[32608]: from='mgr.14152 172.21.15.114:0/1529706138' entity='mgr.x' 2024-04-23T10:20:52.242 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 23 10:20:52 smithi119 ceph-mon[32608]: from='mgr.14152 172.21.15.114:0/1529706138' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-23T10:20:52.254 INFO:journalctl@ceph.mon.a.smithi114.stdout:Apr 23 10:20:51 smithi114 ceph-mon[29255]: from='mgr.14152 172.21.15.114:0/1529706138' entity='mgr.x' 2024-04-23T10:20:52.254 INFO:journalctl@ceph.mon.a.smithi114.stdout:Apr 23 10:20:52 smithi114 ceph-mon[29255]: from='mgr.14152 172.21.15.114:0/1529706138' entity='mgr.x' 2024-04-23T10:20:52.254 INFO:journalctl@ceph.mon.a.smithi114.stdout:Apr 23 10:20:52 smithi114 ceph-mon[29255]: from='mgr.14152 172.21.15.114:0/1529706138' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-23T10:20:52.328 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 23 10:20:52 smithi204 ceph-mon[32649]: from='mgr.14152 172.21.15.114:0/1529706138' entity='mgr.x' 2024-04-23T10:20:52.328 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 23 10:20:52 smithi204 ceph-mon[32649]: from='mgr.14152 172.21.15.114:0/1529706138' entity='mgr.x' 2024-04-23T10:20:52.328 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 23 10:20:52 smithi204 ceph-mon[32649]: from='mgr.14152 172.21.15.114:0/1529706138' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-23T10:20:53.092 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 23 10:20:53 smithi119 ceph-mon[32608]: pgmap v113: 1 pgs: 1 active+clean; 577 KiB data, 116 MiB used, 357 GiB / 358 GiB avail 2024-04-23T10:20:53.328 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 23 10:20:53 smithi204 ceph-mon[32649]: pgmap v113: 1 pgs: 1 active+clean; 577 KiB data, 116 MiB used, 357 GiB / 358 GiB avail 2024-04-23T10:20:53.504 INFO:journalctl@ceph.mon.a.smithi114.stdout:Apr 23 10:20:53 smithi114 ceph-mon[29255]: pgmap v113: 1 pgs: 1 active+clean; 577 KiB data, 116 MiB used, 357 GiB / 358 GiB avail 2024-04-23T10:20:54.263 INFO:teuthology.orchestra.run.smithi119.stdout:Created osd(s) 4 on host 'smithi119' 2024-04-23T10:20:54.263 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-23T10:20:54.263+0000 7fe9b6ffd700 1 -- 172.21.15.119:0/1560854742 <== mgr.14152 v2:172.21.15.114:6800/465333808 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+37 (secure 0 0 0) 0x7fe9c006afe0 con 0x7fe9ac0a3c40 2024-04-23T10:20:54.264 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-23T10:20:54.265+0000 7fe9c7f72700 1 -- 172.21.15.119:0/1560854742 >> [v2:172.21.15.114:6800/465333808,v1:172.21.15.114:6801/465333808] conn(0x7fe9ac0a3c40 msgr2=0x7fe9ac0a6100 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T10:20:54.264 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-23T10:20:54.265+0000 7fe9c7f72700 1 --2- 172.21.15.119:0/1560854742 >> [v2:172.21.15.114:6800/465333808,v1:172.21.15.114:6801/465333808] conn(0x7fe9ac0a3c40 0x7fe9ac0a6100 secure :-1 s=READY pgs=74 cs=0 l=1 rev1=1 crypto rx=0x7fe9bc005950 tx=0x7fe9bc00a300 comp rx=0 tx=0).stop 2024-04-23T10:20:54.264 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-23T10:20:54.265+0000 7fe9c7f72700 1 -- 172.21.15.119:0/1560854742 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fe9c0102ba0 msgr2=0x7fe9c0197ff0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T10:20:54.264 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-23T10:20:54.265+0000 7fe9c7f72700 1 --2- 172.21.15.119:0/1560854742 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fe9c0102ba0 0x7fe9c0197ff0 secure :-1 s=READY pgs=18 cs=0 l=1 rev1=1 crypto rx=0x7fe9b0009970 tx=0x7fe9b0009c80 comp rx=0 tx=0).stop 2024-04-23T10:20:54.264 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-23T10:20:54.265+0000 7fe9c7f72700 1 -- 172.21.15.119:0/1560854742 shutdown_connections 2024-04-23T10:20:54.265 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-23T10:20:54.265+0000 7fe9c7f72700 1 --2- 172.21.15.119:0/1560854742 >> [v2:172.21.15.114:6800/465333808,v1:172.21.15.114:6801/465333808] conn(0x7fe9ac0a3c40 0x7fe9ac0a6100 unknown :-1 s=CLOSED pgs=74 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:20:54.265 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-23T10:20:54.265+0000 7fe9c7f72700 1 --2- 172.21.15.119:0/1560854742 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fe9c0103540 0x7fe9c019f070 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:20:54.265 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-23T10:20:54.265+0000 7fe9c7f72700 1 --2- 172.21.15.119:0/1560854742 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fe9c01019a0 0x7fe9c0197ab0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:20:54.265 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-23T10:20:54.265+0000 7fe9c7f72700 1 --2- 172.21.15.119:0/1560854742 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fe9c0102ba0 0x7fe9c0197ff0 unknown :-1 s=CLOSED pgs=18 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:20:54.265 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-23T10:20:54.265+0000 7fe9c7f72700 1 -- 172.21.15.119:0/1560854742 >> 172.21.15.119:0/1560854742 conn(0x7fe9c00fcf40 msgr2=0x7fe9c0105f20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T10:20:54.265 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-23T10:20:54.265+0000 7fe9c7f72700 1 -- 172.21.15.119:0/1560854742 shutdown_connections 2024-04-23T10:20:54.265 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-23T10:20:54.265+0000 7fe9c7f72700 1 -- 172.21.15.119:0/1560854742 wait complete. 2024-04-23T10:20:54.988 DEBUG:teuthology.orchestra.run.smithi119:osd.4> sudo journalctl -f -n 0 -u ceph-aa240572-0159-11ef-bc93-c7b262605968@osd.4.service 2024-04-23T10:20:54.990 INFO:tasks.cephadm:Deploying osd.5 on smithi204 with /dev/vg_nvme/lv_3... 2024-04-23T10:20:54.990 DEBUG:teuthology.orchestra.run.smithi204:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d69f13d827952bb07bb9b7973bca14f97d764d6f shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid aa240572-0159-11ef-bc93-c7b262605968 -- ceph-volume lvm zap /dev/vg_nvme/lv_3 2024-04-23T10:20:55.226 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 23 10:20:55 smithi119 ceph-mon[32608]: pgmap v114: 1 pgs: 1 active+clean; 577 KiB data, 116 MiB used, 357 GiB / 358 GiB avail 2024-04-23T10:20:55.226 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 23 10:20:55 smithi119 ceph-mon[32608]: from='mgr.14152 172.21.15.114:0/1529706138' entity='mgr.x' 2024-04-23T10:20:55.226 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 23 10:20:55 smithi119 ceph-mon[32608]: from='mgr.14152 172.21.15.114:0/1529706138' entity='mgr.x' 2024-04-23T10:20:55.226 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 23 10:20:55 smithi119 ceph-mon[32608]: from='mgr.14152 172.21.15.114:0/1529706138' entity='mgr.x' 2024-04-23T10:20:55.226 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 23 10:20:55 smithi119 ceph-mon[32608]: from='mgr.14152 172.21.15.114:0/1529706138' entity='mgr.x' 2024-04-23T10:20:55.226 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 23 10:20:55 smithi119 ceph-mon[32608]: from='osd.4 [v2:172.21.15.119:6808/1611007172,v1:172.21.15.119:6809/1611007172]' entity='osd.4' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]: dispatch 2024-04-23T10:20:55.226 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 23 10:20:55 smithi119 ceph-mon[32608]: from='osd.4 ' entity='osd.4' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]: dispatch 2024-04-23T10:20:55.234 INFO:teuthology.orchestra.run.smithi204.stderr:Inferring config /var/lib/ceph/aa240572-0159-11ef-bc93-c7b262605968/mon.c/config 2024-04-23T10:20:55.328 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 23 10:20:55 smithi204 ceph-mon[32649]: pgmap v114: 1 pgs: 1 active+clean; 577 KiB data, 116 MiB used, 357 GiB / 358 GiB avail 2024-04-23T10:20:55.328 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 23 10:20:55 smithi204 ceph-mon[32649]: from='mgr.14152 172.21.15.114:0/1529706138' entity='mgr.x' 2024-04-23T10:20:55.328 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 23 10:20:55 smithi204 ceph-mon[32649]: from='mgr.14152 172.21.15.114:0/1529706138' entity='mgr.x' 2024-04-23T10:20:55.328 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 23 10:20:55 smithi204 ceph-mon[32649]: from='mgr.14152 172.21.15.114:0/1529706138' entity='mgr.x' 2024-04-23T10:20:55.328 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 23 10:20:55 smithi204 ceph-mon[32649]: from='mgr.14152 172.21.15.114:0/1529706138' entity='mgr.x' 2024-04-23T10:20:55.328 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 23 10:20:55 smithi204 ceph-mon[32649]: from='osd.4 [v2:172.21.15.119:6808/1611007172,v1:172.21.15.119:6809/1611007172]' entity='osd.4' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]: dispatch 2024-04-23T10:20:55.328 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 23 10:20:55 smithi204 ceph-mon[32649]: from='osd.4 ' entity='osd.4' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]: dispatch 2024-04-23T10:20:55.504 INFO:journalctl@ceph.mon.a.smithi114.stdout:Apr 23 10:20:55 smithi114 ceph-mon[29255]: pgmap v114: 1 pgs: 1 active+clean; 577 KiB data, 116 MiB used, 357 GiB / 358 GiB avail 2024-04-23T10:20:55.504 INFO:journalctl@ceph.mon.a.smithi114.stdout:Apr 23 10:20:55 smithi114 ceph-mon[29255]: from='mgr.14152 172.21.15.114:0/1529706138' entity='mgr.x' 2024-04-23T10:20:55.505 INFO:journalctl@ceph.mon.a.smithi114.stdout:Apr 23 10:20:55 smithi114 ceph-mon[29255]: from='mgr.14152 172.21.15.114:0/1529706138' entity='mgr.x' 2024-04-23T10:20:55.505 INFO:journalctl@ceph.mon.a.smithi114.stdout:Apr 23 10:20:55 smithi114 ceph-mon[29255]: from='mgr.14152 172.21.15.114:0/1529706138' entity='mgr.x' 2024-04-23T10:20:55.505 INFO:journalctl@ceph.mon.a.smithi114.stdout:Apr 23 10:20:55 smithi114 ceph-mon[29255]: from='mgr.14152 172.21.15.114:0/1529706138' entity='mgr.x' 2024-04-23T10:20:55.505 INFO:journalctl@ceph.mon.a.smithi114.stdout:Apr 23 10:20:55 smithi114 ceph-mon[29255]: from='osd.4 [v2:172.21.15.119:6808/1611007172,v1:172.21.15.119:6809/1611007172]' entity='osd.4' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]: dispatch 2024-04-23T10:20:55.505 INFO:journalctl@ceph.mon.a.smithi114.stdout:Apr 23 10:20:55 smithi114 ceph-mon[29255]: from='osd.4 ' entity='osd.4' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]: dispatch 2024-04-23T10:20:56.710 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 23 10:20:56 smithi119 ceph-mon[32608]: from='osd.4 ' entity='osd.4' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]': finished 2024-04-23T10:20:56.710 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 23 10:20:56 smithi119 ceph-mon[32608]: osdmap e31: 5 total, 4 up, 5 in 2024-04-23T10:20:56.710 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 23 10:20:56 smithi119 ceph-mon[32608]: from='mgr.14152 172.21.15.114:0/1529706138' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-04-23T10:20:56.710 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 23 10:20:56 smithi119 ceph-mon[32608]: from='osd.4 [v2:172.21.15.119:6808/1611007172,v1:172.21.15.119:6809/1611007172]' entity='osd.4' cmd=[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi119", "root=default"]}]: dispatch 2024-04-23T10:20:56.710 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 23 10:20:56 smithi119 ceph-mon[32608]: from='osd.4 ' entity='osd.4' cmd=[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi119", "root=default"]}]: dispatch 2024-04-23T10:20:56.710 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 23 10:20:56 smithi119 ceph-mon[32608]: pgmap v116: 1 pgs: 1 active+clean; 577 KiB data, 116 MiB used, 357 GiB / 358 GiB avail 2024-04-23T10:20:56.711 INFO:journalctl@ceph.osd.4.smithi119.stdout:Apr 23 10:20:56 smithi119 ceph-aa240572-0159-11ef-bc93-c7b262605968-osd-4[43889]: 2024-04-23T10:20:56.624+0000 7f4e9d9a2700 -1 osd.4 0 waiting for initial osdmap 2024-04-23T10:20:56.711 INFO:journalctl@ceph.osd.4.smithi119.stdout:Apr 23 10:20:56 smithi119 ceph-aa240572-0159-11ef-bc93-c7b262605968-osd-4[43889]: 2024-04-23T10:20:56.634+0000 7f4e99795700 -1 osd.4 32 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-04-23T10:20:57.004 INFO:journalctl@ceph.mon.a.smithi114.stdout:Apr 23 10:20:56 smithi114 ceph-mon[29255]: from='osd.4 ' entity='osd.4' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]': finished 2024-04-23T10:20:57.004 INFO:journalctl@ceph.mon.a.smithi114.stdout:Apr 23 10:20:56 smithi114 ceph-mon[29255]: osdmap e31: 5 total, 4 up, 5 in 2024-04-23T10:20:57.004 INFO:journalctl@ceph.mon.a.smithi114.stdout:Apr 23 10:20:56 smithi114 ceph-mon[29255]: from='mgr.14152 172.21.15.114:0/1529706138' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-04-23T10:20:57.004 INFO:journalctl@ceph.mon.a.smithi114.stdout:Apr 23 10:20:56 smithi114 ceph-mon[29255]: from='osd.4 [v2:172.21.15.119:6808/1611007172,v1:172.21.15.119:6809/1611007172]' entity='osd.4' cmd=[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi119", "root=default"]}]: dispatch 2024-04-23T10:20:57.004 INFO:journalctl@ceph.mon.a.smithi114.stdout:Apr 23 10:20:56 smithi114 ceph-mon[29255]: from='osd.4 ' entity='osd.4' cmd=[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi119", "root=default"]}]: dispatch 2024-04-23T10:20:57.004 INFO:journalctl@ceph.mon.a.smithi114.stdout:Apr 23 10:20:56 smithi114 ceph-mon[29255]: pgmap v116: 1 pgs: 1 active+clean; 577 KiB data, 116 MiB used, 357 GiB / 358 GiB avail 2024-04-23T10:20:57.078 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 23 10:20:56 smithi204 ceph-mon[32649]: from='osd.4 ' entity='osd.4' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]': finished 2024-04-23T10:20:57.078 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 23 10:20:56 smithi204 ceph-mon[32649]: osdmap e31: 5 total, 4 up, 5 in 2024-04-23T10:20:57.078 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 23 10:20:56 smithi204 ceph-mon[32649]: from='mgr.14152 172.21.15.114:0/1529706138' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-04-23T10:20:57.078 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 23 10:20:56 smithi204 ceph-mon[32649]: from='osd.4 [v2:172.21.15.119:6808/1611007172,v1:172.21.15.119:6809/1611007172]' entity='osd.4' cmd=[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi119", "root=default"]}]: dispatch 2024-04-23T10:20:57.078 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 23 10:20:56 smithi204 ceph-mon[32649]: from='osd.4 ' entity='osd.4' cmd=[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi119", "root=default"]}]: dispatch 2024-04-23T10:20:57.078 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 23 10:20:56 smithi204 ceph-mon[32649]: pgmap v116: 1 pgs: 1 active+clean; 577 KiB data, 116 MiB used, 357 GiB / 358 GiB avail 2024-04-23T10:20:57.220 INFO:teuthology.orchestra.run.smithi204.stderr:--> Zapping: /dev/vg_nvme/lv_3 2024-04-23T10:20:57.221 INFO:teuthology.orchestra.run.smithi204.stderr:Running command: /usr/bin/dd if=/dev/zero of=/dev/vg_nvme/lv_3 bs=1M count=10 conv=fsync 2024-04-23T10:20:57.221 INFO:teuthology.orchestra.run.smithi204.stderr: stderr: 10+0 records in 2024-04-23T10:20:57.221 INFO:teuthology.orchestra.run.smithi204.stderr:10+0 records out 2024-04-23T10:20:57.221 INFO:teuthology.orchestra.run.smithi204.stderr: stderr: 10485760 bytes (10 MB, 10 MiB) copied, 0.0112863 s, 929 MB/s 2024-04-23T10:20:57.221 INFO:teuthology.orchestra.run.smithi204.stderr:--> Zapping successful for: 2024-04-23T10:20:57.870 DEBUG:teuthology.orchestra.run.smithi204:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d69f13d827952bb07bb9b7973bca14f97d764d6f shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid aa240572-0159-11ef-bc93-c7b262605968 -- ceph orch daemon add osd smithi204:vg_nvme/lv_3 2024-04-23T10:20:57.890 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 23 10:20:57 smithi119 ceph-mon[32608]: from='osd.4 ' entity='osd.4' cmd='[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi119", "root=default"]}]': finished 2024-04-23T10:20:57.890 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 23 10:20:57 smithi119 ceph-mon[32608]: osdmap e32: 5 total, 4 up, 5 in 2024-04-23T10:20:57.891 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 23 10:20:57 smithi119 ceph-mon[32608]: from='mgr.14152 172.21.15.114:0/1529706138' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-04-23T10:20:57.891 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 23 10:20:57 smithi119 ceph-mon[32608]: from='mgr.14152 172.21.15.114:0/1529706138' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-04-23T10:20:57.891 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 23 10:20:57 smithi204 ceph-mon[32649]: from='osd.4 ' entity='osd.4' cmd='[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi119", "root=default"]}]': finished 2024-04-23T10:20:57.891 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 23 10:20:57 smithi204 ceph-mon[32649]: osdmap e32: 5 total, 4 up, 5 in 2024-04-23T10:20:57.891 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 23 10:20:57 smithi204 ceph-mon[32649]: from='mgr.14152 172.21.15.114:0/1529706138' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-04-23T10:20:57.892 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 23 10:20:57 smithi204 ceph-mon[32649]: from='mgr.14152 172.21.15.114:0/1529706138' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-04-23T10:20:58.004 INFO:journalctl@ceph.mon.a.smithi114.stdout:Apr 23 10:20:57 smithi114 ceph-mon[29255]: from='osd.4 ' entity='osd.4' cmd='[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi119", "root=default"]}]': finished 2024-04-23T10:20:58.004 INFO:journalctl@ceph.mon.a.smithi114.stdout:Apr 23 10:20:57 smithi114 ceph-mon[29255]: osdmap e32: 5 total, 4 up, 5 in 2024-04-23T10:20:58.004 INFO:journalctl@ceph.mon.a.smithi114.stdout:Apr 23 10:20:57 smithi114 ceph-mon[29255]: from='mgr.14152 172.21.15.114:0/1529706138' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-04-23T10:20:58.004 INFO:journalctl@ceph.mon.a.smithi114.stdout:Apr 23 10:20:57 smithi114 ceph-mon[29255]: from='mgr.14152 172.21.15.114:0/1529706138' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-04-23T10:20:58.115 INFO:teuthology.orchestra.run.smithi204.stderr:Inferring config /var/lib/ceph/aa240572-0159-11ef-bc93-c7b262605968/mon.c/config 2024-04-23T10:20:58.828 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 23 10:20:58 smithi204 ceph-mon[32649]: purged_snaps scrub starts 2024-04-23T10:20:58.829 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 23 10:20:58 smithi204 ceph-mon[32649]: purged_snaps scrub ok 2024-04-23T10:20:58.829 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 23 10:20:58 smithi204 ceph-mon[32649]: osd.4 [v2:172.21.15.119:6808/1611007172,v1:172.21.15.119:6809/1611007172] boot 2024-04-23T10:20:58.829 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 23 10:20:58 smithi204 ceph-mon[32649]: osdmap e33: 5 total, 5 up, 5 in 2024-04-23T10:20:58.829 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 23 10:20:58 smithi204 ceph-mon[32649]: from='mgr.14152 172.21.15.114:0/1529706138' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-04-23T10:20:58.829 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 23 10:20:58 smithi204 ceph-mon[32649]: pgmap v119: 1 pgs: 1 active+clean; 577 KiB data, 143 MiB used, 447 GiB / 447 GiB avail 2024-04-23T10:20:58.829 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 23 10:20:58 smithi204 ceph-mon[32649]: Detected new or changed devices on smithi119 2024-04-23T10:20:58.829 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 23 10:20:58 smithi204 ceph-mon[32649]: from='mgr.14152 172.21.15.114:0/1529706138' entity='mgr.x' 2024-04-23T10:20:58.829 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 23 10:20:58 smithi204 ceph-mon[32649]: from='mgr.14152 172.21.15.114:0/1529706138' entity='mgr.x' 2024-04-23T10:20:58.829 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 23 10:20:58 smithi204 ceph-mon[32649]: from='mgr.14152 172.21.15.114:0/1529706138' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-04-23T10:20:58.829 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 23 10:20:58 smithi204 ceph-mon[32649]: from='mgr.14152 172.21.15.114:0/1529706138' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-04-23T10:20:58.829 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 23 10:20:58 smithi204 ceph-mon[32649]: Adjusting osd_memory_target on smithi119 to 8544M 2024-04-23T10:20:58.829 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 23 10:20:58 smithi204 ceph-mon[32649]: from='mgr.14152 172.21.15.114:0/1529706138' entity='mgr.x' 2024-04-23T10:20:58.829 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 23 10:20:58 smithi204 ceph-mon[32649]: from='mgr.14152 172.21.15.114:0/1529706138' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-23T10:20:58.829 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 23 10:20:58 smithi204 ceph-mon[32649]: from='mgr.14152 172.21.15.114:0/1529706138' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-23T10:20:58.830 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 23 10:20:58 smithi204 ceph-mon[32649]: from='mgr.14152 172.21.15.114:0/1529706138' entity='mgr.x' 2024-04-23T10:20:58.830 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 23 10:20:58 smithi204 ceph-mon[32649]: from='mgr.14152 172.21.15.114:0/1529706138' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-23T10:20:58.830 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 23 10:20:58 smithi204 ceph-mon[32649]: from='mgr.14152 172.21.15.114:0/1529706138' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-23T10:20:58.830 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 23 10:20:58 smithi204 ceph-mon[32649]: from='mgr.14152 172.21.15.114:0/1529706138' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-23T10:20:58.830 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 23 10:20:58 smithi204 ceph-mon[32649]: from='mgr.14152 172.21.15.114:0/1529706138' entity='mgr.x' 2024-04-23T10:20:58.992 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 23 10:20:58 smithi119 ceph-mon[32608]: purged_snaps scrub starts 2024-04-23T10:20:58.992 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 23 10:20:58 smithi119 ceph-mon[32608]: purged_snaps scrub ok 2024-04-23T10:20:58.992 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 23 10:20:58 smithi119 ceph-mon[32608]: osd.4 [v2:172.21.15.119:6808/1611007172,v1:172.21.15.119:6809/1611007172] boot 2024-04-23T10:20:58.993 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 23 10:20:58 smithi119 ceph-mon[32608]: osdmap e33: 5 total, 5 up, 5 in 2024-04-23T10:20:58.993 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 23 10:20:58 smithi119 ceph-mon[32608]: from='mgr.14152 172.21.15.114:0/1529706138' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-04-23T10:20:58.993 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 23 10:20:58 smithi119 ceph-mon[32608]: pgmap v119: 1 pgs: 1 active+clean; 577 KiB data, 143 MiB used, 447 GiB / 447 GiB avail 2024-04-23T10:20:58.993 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 23 10:20:58 smithi119 ceph-mon[32608]: Detected new or changed devices on smithi119 2024-04-23T10:20:58.993 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 23 10:20:58 smithi119 ceph-mon[32608]: from='mgr.14152 172.21.15.114:0/1529706138' entity='mgr.x' 2024-04-23T10:20:58.993 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 23 10:20:58 smithi119 ceph-mon[32608]: from='mgr.14152 172.21.15.114:0/1529706138' entity='mgr.x' 2024-04-23T10:20:58.993 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 23 10:20:58 smithi119 ceph-mon[32608]: from='mgr.14152 172.21.15.114:0/1529706138' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-04-23T10:20:58.993 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 23 10:20:58 smithi119 ceph-mon[32608]: from='mgr.14152 172.21.15.114:0/1529706138' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-04-23T10:20:58.993 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 23 10:20:58 smithi119 ceph-mon[32608]: Adjusting osd_memory_target on smithi119 to 8544M 2024-04-23T10:20:58.993 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 23 10:20:58 smithi119 ceph-mon[32608]: from='mgr.14152 172.21.15.114:0/1529706138' entity='mgr.x' 2024-04-23T10:20:58.993 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 23 10:20:58 smithi119 ceph-mon[32608]: from='mgr.14152 172.21.15.114:0/1529706138' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-23T10:20:58.993 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 23 10:20:58 smithi119 ceph-mon[32608]: from='mgr.14152 172.21.15.114:0/1529706138' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-23T10:20:58.993 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 23 10:20:58 smithi119 ceph-mon[32608]: from='mgr.14152 172.21.15.114:0/1529706138' entity='mgr.x' 2024-04-23T10:20:58.993 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 23 10:20:58 smithi119 ceph-mon[32608]: from='mgr.14152 172.21.15.114:0/1529706138' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-23T10:20:58.994 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 23 10:20:58 smithi119 ceph-mon[32608]: from='mgr.14152 172.21.15.114:0/1529706138' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-23T10:20:58.994 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 23 10:20:58 smithi119 ceph-mon[32608]: from='mgr.14152 172.21.15.114:0/1529706138' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-23T10:20:58.994 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 23 10:20:58 smithi119 ceph-mon[32608]: from='mgr.14152 172.21.15.114:0/1529706138' entity='mgr.x' 2024-04-23T10:20:59.004 INFO:journalctl@ceph.mon.a.smithi114.stdout:Apr 23 10:20:58 smithi114 ceph-mon[29255]: purged_snaps scrub starts 2024-04-23T10:20:59.004 INFO:journalctl@ceph.mon.a.smithi114.stdout:Apr 23 10:20:58 smithi114 ceph-mon[29255]: purged_snaps scrub ok 2024-04-23T10:20:59.004 INFO:journalctl@ceph.mon.a.smithi114.stdout:Apr 23 10:20:58 smithi114 ceph-mon[29255]: osd.4 [v2:172.21.15.119:6808/1611007172,v1:172.21.15.119:6809/1611007172] boot 2024-04-23T10:20:59.005 INFO:journalctl@ceph.mon.a.smithi114.stdout:Apr 23 10:20:58 smithi114 ceph-mon[29255]: osdmap e33: 5 total, 5 up, 5 in 2024-04-23T10:20:59.005 INFO:journalctl@ceph.mon.a.smithi114.stdout:Apr 23 10:20:58 smithi114 ceph-mon[29255]: from='mgr.14152 172.21.15.114:0/1529706138' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-04-23T10:20:59.005 INFO:journalctl@ceph.mon.a.smithi114.stdout:Apr 23 10:20:58 smithi114 ceph-mon[29255]: pgmap v119: 1 pgs: 1 active+clean; 577 KiB data, 143 MiB used, 447 GiB / 447 GiB avail 2024-04-23T10:20:59.005 INFO:journalctl@ceph.mon.a.smithi114.stdout:Apr 23 10:20:58 smithi114 ceph-mon[29255]: Detected new or changed devices on smithi119 2024-04-23T10:20:59.005 INFO:journalctl@ceph.mon.a.smithi114.stdout:Apr 23 10:20:58 smithi114 ceph-mon[29255]: from='mgr.14152 172.21.15.114:0/1529706138' entity='mgr.x' 2024-04-23T10:20:59.005 INFO:journalctl@ceph.mon.a.smithi114.stdout:Apr 23 10:20:58 smithi114 ceph-mon[29255]: from='mgr.14152 172.21.15.114:0/1529706138' entity='mgr.x' 2024-04-23T10:20:59.005 INFO:journalctl@ceph.mon.a.smithi114.stdout:Apr 23 10:20:58 smithi114 ceph-mon[29255]: from='mgr.14152 172.21.15.114:0/1529706138' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-04-23T10:20:59.005 INFO:journalctl@ceph.mon.a.smithi114.stdout:Apr 23 10:20:58 smithi114 ceph-mon[29255]: from='mgr.14152 172.21.15.114:0/1529706138' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-04-23T10:20:59.005 INFO:journalctl@ceph.mon.a.smithi114.stdout:Apr 23 10:20:58 smithi114 ceph-mon[29255]: Adjusting osd_memory_target on smithi119 to 8544M 2024-04-23T10:20:59.005 INFO:journalctl@ceph.mon.a.smithi114.stdout:Apr 23 10:20:58 smithi114 ceph-mon[29255]: from='mgr.14152 172.21.15.114:0/1529706138' entity='mgr.x' 2024-04-23T10:20:59.005 INFO:journalctl@ceph.mon.a.smithi114.stdout:Apr 23 10:20:58 smithi114 ceph-mon[29255]: from='mgr.14152 172.21.15.114:0/1529706138' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-23T10:20:59.005 INFO:journalctl@ceph.mon.a.smithi114.stdout:Apr 23 10:20:58 smithi114 ceph-mon[29255]: from='mgr.14152 172.21.15.114:0/1529706138' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-23T10:20:59.005 INFO:journalctl@ceph.mon.a.smithi114.stdout:Apr 23 10:20:58 smithi114 ceph-mon[29255]: from='mgr.14152 172.21.15.114:0/1529706138' entity='mgr.x' 2024-04-23T10:20:59.005 INFO:journalctl@ceph.mon.a.smithi114.stdout:Apr 23 10:20:58 smithi114 ceph-mon[29255]: from='mgr.14152 172.21.15.114:0/1529706138' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-23T10:20:59.006 INFO:journalctl@ceph.mon.a.smithi114.stdout:Apr 23 10:20:58 smithi114 ceph-mon[29255]: from='mgr.14152 172.21.15.114:0/1529706138' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-23T10:20:59.006 INFO:journalctl@ceph.mon.a.smithi114.stdout:Apr 23 10:20:58 smithi114 ceph-mon[29255]: from='mgr.14152 172.21.15.114:0/1529706138' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-23T10:20:59.006 INFO:journalctl@ceph.mon.a.smithi114.stdout:Apr 23 10:20:58 smithi114 ceph-mon[29255]: from='mgr.14152 172.21.15.114:0/1529706138' entity='mgr.x' 2024-04-23T10:20:59.624 INFO:teuthology.orchestra.run.smithi204.stderr:2024-04-23T10:20:59.625+0000 7f8678ee8700 1 -- 172.21.15.204:0/2333337213 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f86741019a0 msgr2=0x7f8674101dc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T10:20:59.624 INFO:teuthology.orchestra.run.smithi204.stderr:2024-04-23T10:20:59.625+0000 7f8678ee8700 1 --2- 172.21.15.204:0/2333337213 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f86741019a0 0x7f8674101dc0 secure :-1 s=READY pgs=19 cs=0 l=1 rev1=1 crypto rx=0x7f8664009990 tx=0x7f8664009ca0 comp rx=0 tx=0).stop 2024-04-23T10:20:59.624 INFO:teuthology.orchestra.run.smithi204.stderr:2024-04-23T10:20:59.625+0000 7f8678ee8700 1 -- 172.21.15.204:0/2333337213 shutdown_connections 2024-04-23T10:20:59.624 INFO:teuthology.orchestra.run.smithi204.stderr:2024-04-23T10:20:59.625+0000 7f8678ee8700 1 --2- 172.21.15.204:0/2333337213 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f8674103540 0x7f8674109df0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:20:59.624 INFO:teuthology.orchestra.run.smithi204.stderr:2024-04-23T10:20:59.625+0000 7f8678ee8700 1 --2- 172.21.15.204:0/2333337213 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f86741019a0 0x7f8674101dc0 unknown :-1 s=CLOSED pgs=19 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:20:59.624 INFO:teuthology.orchestra.run.smithi204.stderr:2024-04-23T10:20:59.625+0000 7f8678ee8700 1 --2- 172.21.15.204:0/2333337213 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f8674102ba0 0x7f8674103000 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:20:59.624 INFO:teuthology.orchestra.run.smithi204.stderr:2024-04-23T10:20:59.625+0000 7f8678ee8700 1 -- 172.21.15.204:0/2333337213 >> 172.21.15.204:0/2333337213 conn(0x7f86740fcf40 msgr2=0x7f86740ff380 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T10:20:59.624 INFO:teuthology.orchestra.run.smithi204.stderr:2024-04-23T10:20:59.625+0000 7f8678ee8700 1 -- 172.21.15.204:0/2333337213 shutdown_connections 2024-04-23T10:20:59.624 INFO:teuthology.orchestra.run.smithi204.stderr:2024-04-23T10:20:59.625+0000 7f8678ee8700 1 -- 172.21.15.204:0/2333337213 wait complete. 2024-04-23T10:20:59.627 INFO:teuthology.orchestra.run.smithi204.stderr:2024-04-23T10:20:59.626+0000 7f8678ee8700 1 Processor -- start 2024-04-23T10:20:59.627 INFO:teuthology.orchestra.run.smithi204.stderr:2024-04-23T10:20:59.626+0000 7f8678ee8700 1 -- start start 2024-04-23T10:20:59.627 INFO:teuthology.orchestra.run.smithi204.stderr:2024-04-23T10:20:59.626+0000 7f8678ee8700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f86741019a0 0x7f8674198d30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T10:20:59.627 INFO:teuthology.orchestra.run.smithi204.stderr:2024-04-23T10:20:59.626+0000 7f8678ee8700 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f8674102ba0 0x7f8674199270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T10:20:59.627 INFO:teuthology.orchestra.run.smithi204.stderr:2024-04-23T10:20:59.626+0000 7f8678ee8700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f8674103540 0x7f86741a0260 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T10:20:59.627 INFO:teuthology.orchestra.run.smithi204.stderr:2024-04-23T10:20:59.626+0000 7f8678ee8700 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x7f867410d430 con 0x7f86741019a0 2024-04-23T10:20:59.627 INFO:teuthology.orchestra.run.smithi204.stderr:2024-04-23T10:20:59.626+0000 7f8678ee8700 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 -- 0x7f867410d2b0 con 0x7f8674103540 2024-04-23T10:20:59.627 INFO:teuthology.orchestra.run.smithi204.stderr:2024-04-23T10:20:59.626+0000 7f8678ee8700 1 -- --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_getmap magic: 0 -- 0x7f867410d5b0 con 0x7f8674102ba0 2024-04-23T10:20:59.627 INFO:teuthology.orchestra.run.smithi204.stderr:2024-04-23T10:20:59.626+0000 7f8669d9b700 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f8674102ba0 0x7f8674199270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T10:20:59.627 INFO:teuthology.orchestra.run.smithi204.stderr:2024-04-23T10:20:59.626+0000 7f8669d9b700 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f8674102ba0 0x7f8674199270 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.204:3300/0 says I am v2:172.21.15.204:51106/0 (socket says 172.21.15.204:51106) 2024-04-23T10:20:59.627 INFO:teuthology.orchestra.run.smithi204.stderr:2024-04-23T10:20:59.626+0000 7f8669d9b700 1 -- 172.21.15.204:0/2377941356 learned_addr learned my addr 172.21.15.204:0/2377941356 (peer_addr_for_me v2:172.21.15.204:0/0) 2024-04-23T10:20:59.627 INFO:teuthology.orchestra.run.smithi204.stderr:2024-04-23T10:20:59.626+0000 7f867259c700 1 --2- 172.21.15.204:0/2377941356 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f86741019a0 0x7f8674198d30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T10:20:59.627 INFO:teuthology.orchestra.run.smithi204.stderr:2024-04-23T10:20:59.626+0000 7f8672d9d700 1 --2- 172.21.15.204:0/2377941356 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f8674103540 0x7f86741a0260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T10:20:59.627 INFO:teuthology.orchestra.run.smithi204.stderr:2024-04-23T10:20:59.626+0000 7f8669d9b700 1 -- 172.21.15.204:0/2377941356 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f8674103540 msgr2=0x7f86741a0260 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T10:20:59.628 INFO:teuthology.orchestra.run.smithi204.stderr:2024-04-23T10:20:59.626+0000 7f8669d9b700 1 --2- 172.21.15.204:0/2377941356 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f8674103540 0x7f86741a0260 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:20:59.628 INFO:teuthology.orchestra.run.smithi204.stderr:2024-04-23T10:20:59.626+0000 7f8669d9b700 1 -- 172.21.15.204:0/2377941356 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f86741019a0 msgr2=0x7f8674198d30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T10:20:59.628 INFO:teuthology.orchestra.run.smithi204.stderr:2024-04-23T10:20:59.626+0000 7f8669d9b700 1 --2- 172.21.15.204:0/2377941356 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f86741019a0 0x7f8674198d30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:20:59.628 INFO:teuthology.orchestra.run.smithi204.stderr:2024-04-23T10:20:59.626+0000 7f8669d9b700 1 -- 172.21.15.204:0/2377941356 --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f86741a0930 con 0x7f8674102ba0 2024-04-23T10:20:59.628 INFO:teuthology.orchestra.run.smithi204.stderr:2024-04-23T10:20:59.627+0000 7f8669d9b700 1 --2- 172.21.15.204:0/2377941356 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f8674102ba0 0x7f8674199270 secure :-1 s=READY pgs=20 cs=0 l=1 rev1=1 crypto rx=0x7f865c00e9c0 tx=0x7f865c00ecd0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T10:20:59.628 INFO:teuthology.orchestra.run.smithi204.stderr:2024-04-23T10:20:59.627+0000 7f866bfff700 1 -- 172.21.15.204:0/2377941356 <== mon.1 v2:172.21.15.204:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f865c00cbd0 con 0x7f8674102ba0 2024-04-23T10:20:59.628 INFO:teuthology.orchestra.run.smithi204.stderr:2024-04-23T10:20:59.627+0000 7f8678ee8700 1 -- 172.21.15.204:0/2377941356 --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f86741a0c20 con 0x7f8674102ba0 2024-04-23T10:20:59.628 INFO:teuthology.orchestra.run.smithi204.stderr:2024-04-23T10:20:59.627+0000 7f866bfff700 1 -- 172.21.15.204:0/2377941356 <== mon.1 v2:172.21.15.204:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f865c00ee60 con 0x7f8674102ba0 2024-04-23T10:20:59.628 INFO:teuthology.orchestra.run.smithi204.stderr:2024-04-23T10:20:59.627+0000 7f866bfff700 1 -- 172.21.15.204:0/2377941356 <== mon.1 v2:172.21.15.204:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f865c0055c0 con 0x7f8674102ba0 2024-04-23T10:20:59.628 INFO:teuthology.orchestra.run.smithi204.stderr:2024-04-23T10:20:59.627+0000 7f8678ee8700 1 -- 172.21.15.204:0/2377941356 --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f86741a1160 con 0x7f8674102ba0 2024-04-23T10:20:59.628 INFO:teuthology.orchestra.run.smithi204.stderr:2024-04-23T10:20:59.628+0000 7f8678ee8700 1 -- 172.21.15.204:0/2377941356 --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f8674069730 con 0x7f8674102ba0 2024-04-23T10:20:59.629 INFO:teuthology.orchestra.run.smithi204.stderr:2024-04-23T10:20:59.629+0000 7f866bfff700 1 -- 172.21.15.204:0/2377941356 <== mon.1 v2:172.21.15.204:3300/0 4 ==== mgrmap(e 16) ==== 138647+0+0 (secure 0 0 0) 0x7f865c004750 con 0x7f8674102ba0 2024-04-23T10:20:59.629 INFO:teuthology.orchestra.run.smithi204.stderr:2024-04-23T10:20:59.630+0000 7f866bfff700 1 --2- 172.21.15.204:0/2377941356 >> [v2:172.21.15.114:6800/465333808,v1:172.21.15.114:6801/465333808] conn(0x7f86600a3c40 0x7f86600a6100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T10:20:59.629 INFO:teuthology.orchestra.run.smithi204.stderr:2024-04-23T10:20:59.630+0000 7f866bfff700 1 -- 172.21.15.204:0/2377941356 <== mon.1 v2:172.21.15.204:3300/0 5 ==== osd_map(34..34 src has 1..34) ==== 4101+0+0 (secure 0 0 0) 0x7f865c0cf640 con 0x7f8674102ba0 2024-04-23T10:20:59.629 INFO:teuthology.orchestra.run.smithi204.stderr:2024-04-23T10:20:59.630+0000 7f867259c700 1 --2- 172.21.15.204:0/2377941356 >> [v2:172.21.15.114:6800/465333808,v1:172.21.15.114:6801/465333808] conn(0x7f86600a3c40 0x7f86600a6100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T10:20:59.629 INFO:teuthology.orchestra.run.smithi204.stderr:2024-04-23T10:20:59.630+0000 7f867259c700 1 --2- 172.21.15.204:0/2377941356 >> [v2:172.21.15.114:6800/465333808,v1:172.21.15.114:6801/465333808] conn(0x7f86600a3c40 0x7f86600a6100 secure :-1 s=READY pgs=79 cs=0 l=1 rev1=1 crypto rx=0x7f867419a1c0 tx=0x7f866401a040 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T10:20:59.633 INFO:teuthology.orchestra.run.smithi204.stderr:2024-04-23T10:20:59.635+0000 7f866bfff700 1 -- 172.21.15.204:0/2377941356 <== mon.1 v2:172.21.15.204:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f865c09d090 con 0x7f8674102ba0 2024-04-23T10:20:59.839 INFO:teuthology.orchestra.run.smithi204.stderr:2024-04-23T10:20:59.840+0000 7f8678ee8700 1 -- 172.21.15.204:0/2377941356 --> [v2:172.21.15.114:6800/465333808,v1:172.21.15.114:6801/465333808] -- mgr_command(tid 0: {"prefix": "orch daemon add osd", "svc_arg": "smithi204:vg_nvme/lv_3", "target": ["mon-mgr", ""]}) -- 0x7f867406aea0 con 0x7f86600a3c40 2024-04-23T10:20:59.992 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 23 10:20:59 smithi119 ceph-mon[32608]: osdmap e34: 5 total, 5 up, 5 in 2024-04-23T10:21:00.004 INFO:journalctl@ceph.mon.a.smithi114.stdout:Apr 23 10:20:59 smithi114 ceph-mon[29255]: osdmap e34: 5 total, 5 up, 5 in 2024-04-23T10:21:00.078 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 23 10:20:59 smithi204 ceph-mon[32649]: osdmap e34: 5 total, 5 up, 5 in 2024-04-23T10:21:00.730 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 23 10:21:00 smithi204 ceph-mon[32649]: from='client.24217 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi204:vg_nvme/lv_3", "target": ["mon-mgr", ""]}]: dispatch 2024-04-23T10:21:00.731 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 23 10:21:00 smithi204 ceph-mon[32649]: from='mgr.14152 172.21.15.114:0/1529706138' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-04-23T10:21:00.731 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 23 10:21:00 smithi204 ceph-mon[32649]: from='mgr.14152 172.21.15.114:0/1529706138' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-04-23T10:21:00.731 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 23 10:21:00 smithi204 ceph-mon[32649]: from='mgr.14152 172.21.15.114:0/1529706138' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-23T10:21:00.731 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 23 10:21:00 smithi204 ceph-mon[32649]: pgmap v121: 1 pgs: 1 active+clean; 577 KiB data, 143 MiB used, 447 GiB / 447 GiB avail 2024-04-23T10:21:00.992 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 23 10:21:00 smithi119 ceph-mon[32608]: from='client.24217 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi204:vg_nvme/lv_3", "target": ["mon-mgr", ""]}]: dispatch 2024-04-23T10:21:00.992 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 23 10:21:00 smithi119 ceph-mon[32608]: from='mgr.14152 172.21.15.114:0/1529706138' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-04-23T10:21:00.992 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 23 10:21:00 smithi119 ceph-mon[32608]: from='mgr.14152 172.21.15.114:0/1529706138' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-04-23T10:21:00.992 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 23 10:21:00 smithi119 ceph-mon[32608]: from='mgr.14152 172.21.15.114:0/1529706138' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-23T10:21:00.992 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 23 10:21:00 smithi119 ceph-mon[32608]: pgmap v121: 1 pgs: 1 active+clean; 577 KiB data, 143 MiB used, 447 GiB / 447 GiB avail 2024-04-23T10:21:01.004 INFO:journalctl@ceph.mon.a.smithi114.stdout:Apr 23 10:21:00 smithi114 ceph-mon[29255]: from='client.24217 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi204:vg_nvme/lv_3", "target": ["mon-mgr", ""]}]: dispatch 2024-04-23T10:21:01.004 INFO:journalctl@ceph.mon.a.smithi114.stdout:Apr 23 10:21:00 smithi114 ceph-mon[29255]: from='mgr.14152 172.21.15.114:0/1529706138' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-04-23T10:21:01.004 INFO:journalctl@ceph.mon.a.smithi114.stdout:Apr 23 10:21:00 smithi114 ceph-mon[29255]: from='mgr.14152 172.21.15.114:0/1529706138' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-04-23T10:21:01.004 INFO:journalctl@ceph.mon.a.smithi114.stdout:Apr 23 10:21:00 smithi114 ceph-mon[29255]: from='mgr.14152 172.21.15.114:0/1529706138' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-23T10:21:01.004 INFO:journalctl@ceph.mon.a.smithi114.stdout:Apr 23 10:21:00 smithi114 ceph-mon[29255]: pgmap v121: 1 pgs: 1 active+clean; 577 KiB data, 143 MiB used, 447 GiB / 447 GiB avail 2024-04-23T10:21:03.078 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 23 10:21:02 smithi204 ceph-mon[32649]: pgmap v122: 1 pgs: 1 active+clean; 577 KiB data, 143 MiB used, 447 GiB / 447 GiB avail 2024-04-23T10:21:03.078 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 23 10:21:02 smithi204 ceph-mon[32649]: from='client.? 172.21.15.204:0/886770309' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "f7590e1a-e93c-4c29-b15f-38ee52d1f921"}]: dispatch 2024-04-23T10:21:03.078 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 23 10:21:02 smithi204 ceph-mon[32649]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "f7590e1a-e93c-4c29-b15f-38ee52d1f921"}]: dispatch 2024-04-23T10:21:03.078 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 23 10:21:02 smithi204 ceph-mon[32649]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "f7590e1a-e93c-4c29-b15f-38ee52d1f921"}]': finished 2024-04-23T10:21:03.079 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 23 10:21:02 smithi204 ceph-mon[32649]: osdmap e35: 6 total, 5 up, 6 in 2024-04-23T10:21:03.079 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 23 10:21:02 smithi204 ceph-mon[32649]: from='mgr.14152 172.21.15.114:0/1529706138' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-04-23T10:21:03.242 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 23 10:21:02 smithi119 ceph-mon[32608]: pgmap v122: 1 pgs: 1 active+clean; 577 KiB data, 143 MiB used, 447 GiB / 447 GiB avail 2024-04-23T10:21:03.242 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 23 10:21:02 smithi119 ceph-mon[32608]: from='client.? 172.21.15.204:0/886770309' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "f7590e1a-e93c-4c29-b15f-38ee52d1f921"}]: dispatch 2024-04-23T10:21:03.242 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 23 10:21:02 smithi119 ceph-mon[32608]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "f7590e1a-e93c-4c29-b15f-38ee52d1f921"}]: dispatch 2024-04-23T10:21:03.242 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 23 10:21:02 smithi119 ceph-mon[32608]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "f7590e1a-e93c-4c29-b15f-38ee52d1f921"}]': finished 2024-04-23T10:21:03.242 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 23 10:21:02 smithi119 ceph-mon[32608]: osdmap e35: 6 total, 5 up, 6 in 2024-04-23T10:21:03.242 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 23 10:21:02 smithi119 ceph-mon[32608]: from='mgr.14152 172.21.15.114:0/1529706138' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-04-23T10:21:03.254 INFO:journalctl@ceph.mon.a.smithi114.stdout:Apr 23 10:21:02 smithi114 ceph-mon[29255]: pgmap v122: 1 pgs: 1 active+clean; 577 KiB data, 143 MiB used, 447 GiB / 447 GiB avail 2024-04-23T10:21:03.254 INFO:journalctl@ceph.mon.a.smithi114.stdout:Apr 23 10:21:02 smithi114 ceph-mon[29255]: from='client.? 172.21.15.204:0/886770309' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "f7590e1a-e93c-4c29-b15f-38ee52d1f921"}]: dispatch 2024-04-23T10:21:03.254 INFO:journalctl@ceph.mon.a.smithi114.stdout:Apr 23 10:21:02 smithi114 ceph-mon[29255]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "f7590e1a-e93c-4c29-b15f-38ee52d1f921"}]: dispatch 2024-04-23T10:21:03.254 INFO:journalctl@ceph.mon.a.smithi114.stdout:Apr 23 10:21:02 smithi114 ceph-mon[29255]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "f7590e1a-e93c-4c29-b15f-38ee52d1f921"}]': finished 2024-04-23T10:21:03.254 INFO:journalctl@ceph.mon.a.smithi114.stdout:Apr 23 10:21:02 smithi114 ceph-mon[29255]: osdmap e35: 6 total, 5 up, 6 in 2024-04-23T10:21:03.254 INFO:journalctl@ceph.mon.a.smithi114.stdout:Apr 23 10:21:02 smithi114 ceph-mon[29255]: from='mgr.14152 172.21.15.114:0/1529706138' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-04-23T10:21:04.242 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 23 10:21:03 smithi119 ceph-mon[32608]: from='client.? 172.21.15.204:0/3457402239' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-04-23T10:21:04.254 INFO:journalctl@ceph.mon.a.smithi114.stdout:Apr 23 10:21:03 smithi114 ceph-mon[29255]: from='client.? 172.21.15.204:0/3457402239' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-04-23T10:21:04.327 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 23 10:21:03 smithi204 ceph-mon[32649]: from='client.? 172.21.15.204:0/3457402239' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-04-23T10:21:05.242 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 23 10:21:04 smithi119 ceph-mon[32608]: pgmap v124: 1 pgs: 1 active+clean; 577 KiB data, 143 MiB used, 447 GiB / 447 GiB avail 2024-04-23T10:21:05.254 INFO:journalctl@ceph.mon.a.smithi114.stdout:Apr 23 10:21:04 smithi114 ceph-mon[29255]: pgmap v124: 1 pgs: 1 active+clean; 577 KiB data, 143 MiB used, 447 GiB / 447 GiB avail 2024-04-23T10:21:05.327 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 23 10:21:04 smithi204 ceph-mon[32649]: pgmap v124: 1 pgs: 1 active+clean; 577 KiB data, 143 MiB used, 447 GiB / 447 GiB avail 2024-04-23T10:21:07.242 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 23 10:21:06 smithi119 ceph-mon[32608]: pgmap v125: 1 pgs: 1 active+clean; 577 KiB data, 143 MiB used, 447 GiB / 447 GiB avail 2024-04-23T10:21:07.254 INFO:journalctl@ceph.mon.a.smithi114.stdout:Apr 23 10:21:06 smithi114 ceph-mon[29255]: pgmap v125: 1 pgs: 1 active+clean; 577 KiB data, 143 MiB used, 447 GiB / 447 GiB avail 2024-04-23T10:21:07.327 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 23 10:21:06 smithi204 ceph-mon[32649]: pgmap v125: 1 pgs: 1 active+clean; 577 KiB data, 143 MiB used, 447 GiB / 447 GiB avail 2024-04-23T10:21:09.242 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 23 10:21:08 smithi119 ceph-mon[32608]: pgmap v126: 1 pgs: 1 active+clean; 577 KiB data, 143 MiB used, 447 GiB / 447 GiB avail 2024-04-23T10:21:09.254 INFO:journalctl@ceph.mon.a.smithi114.stdout:Apr 23 10:21:08 smithi114 ceph-mon[29255]: pgmap v126: 1 pgs: 1 active+clean; 577 KiB data, 143 MiB used, 447 GiB / 447 GiB avail 2024-04-23T10:21:09.328 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 23 10:21:08 smithi204 ceph-mon[32649]: pgmap v126: 1 pgs: 1 active+clean; 577 KiB data, 143 MiB used, 447 GiB / 447 GiB avail 2024-04-23T10:21:10.242 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 23 10:21:09 smithi119 ceph-mon[32608]: from='mgr.14152 172.21.15.114:0/1529706138' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.5"}]: dispatch 2024-04-23T10:21:10.242 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 23 10:21:09 smithi119 ceph-mon[32608]: from='mgr.14152 172.21.15.114:0/1529706138' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-23T10:21:10.254 INFO:journalctl@ceph.mon.a.smithi114.stdout:Apr 23 10:21:09 smithi114 ceph-mon[29255]: from='mgr.14152 172.21.15.114:0/1529706138' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.5"}]: dispatch 2024-04-23T10:21:10.254 INFO:journalctl@ceph.mon.a.smithi114.stdout:Apr 23 10:21:09 smithi114 ceph-mon[29255]: from='mgr.14152 172.21.15.114:0/1529706138' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-23T10:21:10.291 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 23 10:21:09 smithi204 ceph-mon[32649]: from='mgr.14152 172.21.15.114:0/1529706138' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.5"}]: dispatch 2024-04-23T10:21:10.291 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 23 10:21:09 smithi204 ceph-mon[32649]: from='mgr.14152 172.21.15.114:0/1529706138' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-23T10:21:11.242 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 23 10:21:10 smithi119 ceph-mon[32608]: Deploying daemon osd.5 on smithi204 2024-04-23T10:21:11.242 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 23 10:21:10 smithi119 ceph-mon[32608]: pgmap v127: 1 pgs: 1 active+clean; 577 KiB data, 143 MiB used, 447 GiB / 447 GiB avail 2024-04-23T10:21:11.254 INFO:journalctl@ceph.mon.a.smithi114.stdout:Apr 23 10:21:10 smithi114 ceph-mon[29255]: Deploying daemon osd.5 on smithi204 2024-04-23T10:21:11.254 INFO:journalctl@ceph.mon.a.smithi114.stdout:Apr 23 10:21:10 smithi114 ceph-mon[29255]: pgmap v127: 1 pgs: 1 active+clean; 577 KiB data, 143 MiB used, 447 GiB / 447 GiB avail 2024-04-23T10:21:11.317 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 23 10:21:10 smithi204 ceph-mon[32649]: Deploying daemon osd.5 on smithi204 2024-04-23T10:21:11.318 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 23 10:21:10 smithi204 ceph-mon[32649]: pgmap v127: 1 pgs: 1 active+clean; 577 KiB data, 143 MiB used, 447 GiB / 447 GiB avail 2024-04-23T10:21:13.242 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 23 10:21:12 smithi119 ceph-mon[32608]: pgmap v128: 1 pgs: 1 active+clean; 577 KiB data, 143 MiB used, 447 GiB / 447 GiB avail 2024-04-23T10:21:13.246 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 23 10:21:12 smithi204 ceph-mon[32649]: pgmap v128: 1 pgs: 1 active+clean; 577 KiB data, 143 MiB used, 447 GiB / 447 GiB avail 2024-04-23T10:21:13.254 INFO:journalctl@ceph.mon.a.smithi114.stdout:Apr 23 10:21:12 smithi114 ceph-mon[29255]: pgmap v128: 1 pgs: 1 active+clean; 577 KiB data, 143 MiB used, 447 GiB / 447 GiB avail 2024-04-23T10:21:15.242 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 23 10:21:14 smithi119 ceph-mon[32608]: pgmap v129: 1 pgs: 1 active+clean; 577 KiB data, 143 MiB used, 447 GiB / 447 GiB avail 2024-04-23T10:21:15.254 INFO:journalctl@ceph.mon.a.smithi114.stdout:Apr 23 10:21:14 smithi114 ceph-mon[29255]: pgmap v129: 1 pgs: 1 active+clean; 577 KiB data, 143 MiB used, 447 GiB / 447 GiB avail 2024-04-23T10:21:15.328 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 23 10:21:14 smithi204 ceph-mon[32649]: pgmap v129: 1 pgs: 1 active+clean; 577 KiB data, 143 MiB used, 447 GiB / 447 GiB avail 2024-04-23T10:21:16.078 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 23 10:21:16 smithi204 ceph-mon[32649]: from='mgr.14152 172.21.15.114:0/1529706138' entity='mgr.x' 2024-04-23T10:21:16.078 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 23 10:21:16 smithi204 ceph-mon[32649]: from='mgr.14152 172.21.15.114:0/1529706138' entity='mgr.x' 2024-04-23T10:21:16.078 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 23 10:21:16 smithi204 ceph-mon[32649]: from='mgr.14152 172.21.15.114:0/1529706138' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-23T10:21:16.492 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 23 10:21:16 smithi119 ceph-mon[32608]: from='mgr.14152 172.21.15.114:0/1529706138' entity='mgr.x' 2024-04-23T10:21:16.492 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 23 10:21:16 smithi119 ceph-mon[32608]: from='mgr.14152 172.21.15.114:0/1529706138' entity='mgr.x' 2024-04-23T10:21:16.492 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 23 10:21:16 smithi119 ceph-mon[32608]: from='mgr.14152 172.21.15.114:0/1529706138' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-23T10:21:16.504 INFO:journalctl@ceph.mon.a.smithi114.stdout:Apr 23 10:21:16 smithi114 ceph-mon[29255]: from='mgr.14152 172.21.15.114:0/1529706138' entity='mgr.x' 2024-04-23T10:21:16.504 INFO:journalctl@ceph.mon.a.smithi114.stdout:Apr 23 10:21:16 smithi114 ceph-mon[29255]: from='mgr.14152 172.21.15.114:0/1529706138' entity='mgr.x' 2024-04-23T10:21:16.504 INFO:journalctl@ceph.mon.a.smithi114.stdout:Apr 23 10:21:16 smithi114 ceph-mon[29255]: from='mgr.14152 172.21.15.114:0/1529706138' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-23T10:21:17.324 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 23 10:21:17 smithi204 ceph-mon[32649]: pgmap v130: 1 pgs: 1 active+clean; 577 KiB data, 143 MiB used, 447 GiB / 447 GiB avail 2024-04-23T10:21:17.492 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 23 10:21:17 smithi119 ceph-mon[32608]: pgmap v130: 1 pgs: 1 active+clean; 577 KiB data, 143 MiB used, 447 GiB / 447 GiB avail 2024-04-23T10:21:17.504 INFO:journalctl@ceph.mon.a.smithi114.stdout:Apr 23 10:21:17 smithi114 ceph-mon[29255]: pgmap v130: 1 pgs: 1 active+clean; 577 KiB data, 143 MiB used, 447 GiB / 447 GiB avail 2024-04-23T10:21:18.607 INFO:teuthology.orchestra.run.smithi204.stderr:2024-04-23T10:21:18.607+0000 7f866bfff700 1 -- 172.21.15.204:0/2377941356 <== mgr.14152 v2:172.21.15.114:6800/465333808 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+37 (secure 0 0 0) 0x7f867406aea0 con 0x7f86600a3c40 2024-04-23T10:21:18.608 INFO:teuthology.orchestra.run.smithi204.stdout:Created osd(s) 5 on host 'smithi204' 2024-04-23T10:21:18.609 INFO:teuthology.orchestra.run.smithi204.stderr:2024-04-23T10:21:18.609+0000 7f8678ee8700 1 -- 172.21.15.204:0/2377941356 >> [v2:172.21.15.114:6800/465333808,v1:172.21.15.114:6801/465333808] conn(0x7f86600a3c40 msgr2=0x7f86600a6100 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T10:21:18.609 INFO:teuthology.orchestra.run.smithi204.stderr:2024-04-23T10:21:18.609+0000 7f8678ee8700 1 --2- 172.21.15.204:0/2377941356 >> [v2:172.21.15.114:6800/465333808,v1:172.21.15.114:6801/465333808] conn(0x7f86600a3c40 0x7f86600a6100 secure :-1 s=READY pgs=79 cs=0 l=1 rev1=1 crypto rx=0x7f867419a1c0 tx=0x7f866401a040 comp rx=0 tx=0).stop 2024-04-23T10:21:18.610 INFO:teuthology.orchestra.run.smithi204.stderr:2024-04-23T10:21:18.609+0000 7f8678ee8700 1 -- 172.21.15.204:0/2377941356 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f8674102ba0 msgr2=0x7f8674199270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T10:21:18.610 INFO:teuthology.orchestra.run.smithi204.stderr:2024-04-23T10:21:18.609+0000 7f8678ee8700 1 --2- 172.21.15.204:0/2377941356 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f8674102ba0 0x7f8674199270 secure :-1 s=READY pgs=20 cs=0 l=1 rev1=1 crypto rx=0x7f865c00e9c0 tx=0x7f865c00ecd0 comp rx=0 tx=0).stop 2024-04-23T10:21:18.610 INFO:teuthology.orchestra.run.smithi204.stderr:2024-04-23T10:21:18.610+0000 7f8678ee8700 1 -- 172.21.15.204:0/2377941356 shutdown_connections 2024-04-23T10:21:18.610 INFO:teuthology.orchestra.run.smithi204.stderr:2024-04-23T10:21:18.610+0000 7f8678ee8700 1 --2- 172.21.15.204:0/2377941356 >> [v2:172.21.15.114:6800/465333808,v1:172.21.15.114:6801/465333808] conn(0x7f86600a3c40 0x7f86600a6100 unknown :-1 s=CLOSED pgs=79 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:21:18.610 INFO:teuthology.orchestra.run.smithi204.stderr:2024-04-23T10:21:18.610+0000 7f8678ee8700 1 --2- 172.21.15.204:0/2377941356 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f8674103540 0x7f86741a0260 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:21:18.610 INFO:teuthology.orchestra.run.smithi204.stderr:2024-04-23T10:21:18.610+0000 7f8678ee8700 1 --2- 172.21.15.204:0/2377941356 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f86741019a0 0x7f8674198d30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:21:18.610 INFO:teuthology.orchestra.run.smithi204.stderr:2024-04-23T10:21:18.610+0000 7f8678ee8700 1 --2- 172.21.15.204:0/2377941356 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f8674102ba0 0x7f8674199270 unknown :-1 s=CLOSED pgs=20 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:21:18.610 INFO:teuthology.orchestra.run.smithi204.stderr:2024-04-23T10:21:18.610+0000 7f8678ee8700 1 -- 172.21.15.204:0/2377941356 >> 172.21.15.204:0/2377941356 conn(0x7f86740fcf40 msgr2=0x7f867406a790 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T10:21:18.610 INFO:teuthology.orchestra.run.smithi204.stderr:2024-04-23T10:21:18.610+0000 7f8678ee8700 1 -- 172.21.15.204:0/2377941356 shutdown_connections 2024-04-23T10:21:18.610 INFO:teuthology.orchestra.run.smithi204.stderr:2024-04-23T10:21:18.610+0000 7f8678ee8700 1 -- 172.21.15.204:0/2377941356 wait complete. 2024-04-23T10:21:19.078 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 23 10:21:18 smithi204 ceph-mon[32649]: from='mgr.14152 172.21.15.114:0/1529706138' entity='mgr.x' 2024-04-23T10:21:19.078 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 23 10:21:18 smithi204 ceph-mon[32649]: from='mgr.14152 172.21.15.114:0/1529706138' entity='mgr.x' 2024-04-23T10:21:19.078 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 23 10:21:18 smithi204 ceph-mon[32649]: from='mgr.14152 172.21.15.114:0/1529706138' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-23T10:21:19.078 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 23 10:21:18 smithi204 ceph-mon[32649]: from='mgr.14152 172.21.15.114:0/1529706138' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-23T10:21:19.078 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 23 10:21:18 smithi204 ceph-mon[32649]: from='mgr.14152 172.21.15.114:0/1529706138' entity='mgr.x' 2024-04-23T10:21:19.078 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 23 10:21:18 smithi204 ceph-mon[32649]: pgmap v131: 1 pgs: 1 active+clean; 577 KiB data, 143 MiB used, 447 GiB / 447 GiB avail 2024-04-23T10:21:19.078 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 23 10:21:18 smithi204 ceph-mon[32649]: from='mgr.14152 172.21.15.114:0/1529706138' entity='mgr.x' 2024-04-23T10:21:19.078 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 23 10:21:18 smithi204 ceph-mon[32649]: from='mgr.14152 172.21.15.114:0/1529706138' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-23T10:21:19.078 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 23 10:21:18 smithi204 ceph-mon[32649]: from='mgr.14152 172.21.15.114:0/1529706138' entity='mgr.x' 2024-04-23T10:21:19.242 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 23 10:21:18 smithi119 ceph-mon[32608]: from='mgr.14152 172.21.15.114:0/1529706138' entity='mgr.x' 2024-04-23T10:21:19.242 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 23 10:21:18 smithi119 ceph-mon[32608]: from='mgr.14152 172.21.15.114:0/1529706138' entity='mgr.x' 2024-04-23T10:21:19.242 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 23 10:21:18 smithi119 ceph-mon[32608]: from='mgr.14152 172.21.15.114:0/1529706138' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-23T10:21:19.242 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 23 10:21:18 smithi119 ceph-mon[32608]: from='mgr.14152 172.21.15.114:0/1529706138' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-23T10:21:19.242 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 23 10:21:18 smithi119 ceph-mon[32608]: from='mgr.14152 172.21.15.114:0/1529706138' entity='mgr.x' 2024-04-23T10:21:19.242 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 23 10:21:18 smithi119 ceph-mon[32608]: pgmap v131: 1 pgs: 1 active+clean; 577 KiB data, 143 MiB used, 447 GiB / 447 GiB avail 2024-04-23T10:21:19.242 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 23 10:21:18 smithi119 ceph-mon[32608]: from='mgr.14152 172.21.15.114:0/1529706138' entity='mgr.x' 2024-04-23T10:21:19.243 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 23 10:21:18 smithi119 ceph-mon[32608]: from='mgr.14152 172.21.15.114:0/1529706138' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-23T10:21:19.243 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 23 10:21:18 smithi119 ceph-mon[32608]: from='mgr.14152 172.21.15.114:0/1529706138' entity='mgr.x' 2024-04-23T10:21:19.246 DEBUG:teuthology.orchestra.run.smithi204:osd.5> sudo journalctl -f -n 0 -u ceph-aa240572-0159-11ef-bc93-c7b262605968@osd.5.service 2024-04-23T10:21:19.254 INFO:journalctl@ceph.mon.a.smithi114.stdout:Apr 23 10:21:18 smithi114 ceph-mon[29255]: from='mgr.14152 172.21.15.114:0/1529706138' entity='mgr.x' 2024-04-23T10:21:19.254 INFO:journalctl@ceph.mon.a.smithi114.stdout:Apr 23 10:21:18 smithi114 ceph-mon[29255]: from='mgr.14152 172.21.15.114:0/1529706138' entity='mgr.x' 2024-04-23T10:21:19.254 INFO:journalctl@ceph.mon.a.smithi114.stdout:Apr 23 10:21:18 smithi114 ceph-mon[29255]: from='mgr.14152 172.21.15.114:0/1529706138' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-23T10:21:19.255 INFO:journalctl@ceph.mon.a.smithi114.stdout:Apr 23 10:21:18 smithi114 ceph-mon[29255]: from='mgr.14152 172.21.15.114:0/1529706138' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-23T10:21:19.255 INFO:journalctl@ceph.mon.a.smithi114.stdout:Apr 23 10:21:18 smithi114 ceph-mon[29255]: from='mgr.14152 172.21.15.114:0/1529706138' entity='mgr.x' 2024-04-23T10:21:19.255 INFO:journalctl@ceph.mon.a.smithi114.stdout:Apr 23 10:21:18 smithi114 ceph-mon[29255]: pgmap v131: 1 pgs: 1 active+clean; 577 KiB data, 143 MiB used, 447 GiB / 447 GiB avail 2024-04-23T10:21:19.255 INFO:journalctl@ceph.mon.a.smithi114.stdout:Apr 23 10:21:18 smithi114 ceph-mon[29255]: from='mgr.14152 172.21.15.114:0/1529706138' entity='mgr.x' 2024-04-23T10:21:19.255 INFO:journalctl@ceph.mon.a.smithi114.stdout:Apr 23 10:21:18 smithi114 ceph-mon[29255]: from='mgr.14152 172.21.15.114:0/1529706138' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-23T10:21:19.255 INFO:journalctl@ceph.mon.a.smithi114.stdout:Apr 23 10:21:18 smithi114 ceph-mon[29255]: from='mgr.14152 172.21.15.114:0/1529706138' entity='mgr.x' 2024-04-23T10:21:19.290 INFO:tasks.cephadm:Deploying osd.6 on smithi114 with /dev/vg_nvme/lv_2... 2024-04-23T10:21:19.290 DEBUG:teuthology.orchestra.run.smithi114:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d69f13d827952bb07bb9b7973bca14f97d764d6f shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid aa240572-0159-11ef-bc93-c7b262605968 -- ceph-volume lvm zap /dev/vg_nvme/lv_2 2024-04-23T10:21:19.535 INFO:teuthology.orchestra.run.smithi114.stderr:Inferring config /var/lib/ceph/aa240572-0159-11ef-bc93-c7b262605968/mon.a/config 2024-04-23T10:21:19.924 INFO:journalctl@ceph.mon.a.smithi114.stdout:Apr 23 10:21:19 smithi114 ceph-mon[29255]: from='osd.5 [v2:172.21.15.204:6808/1427563312,v1:172.21.15.204:6809/1427563312]' entity='osd.5' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]: dispatch 2024-04-23T10:21:19.924 INFO:journalctl@ceph.mon.a.smithi114.stdout:Apr 23 10:21:19 smithi114 ceph-mon[29255]: from='osd.5 ' entity='osd.5' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]: dispatch 2024-04-23T10:21:20.198 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 23 10:21:19 smithi204 ceph-mon[32649]: from='osd.5 [v2:172.21.15.204:6808/1427563312,v1:172.21.15.204:6809/1427563312]' entity='osd.5' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]: dispatch 2024-04-23T10:21:20.199 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 23 10:21:19 smithi204 ceph-mon[32649]: from='osd.5 ' entity='osd.5' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]: dispatch 2024-04-23T10:21:20.242 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 23 10:21:19 smithi119 ceph-mon[32608]: from='osd.5 [v2:172.21.15.204:6808/1427563312,v1:172.21.15.204:6809/1427563312]' entity='osd.5' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]: dispatch 2024-04-23T10:21:20.242 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 23 10:21:19 smithi119 ceph-mon[32608]: from='osd.5 ' entity='osd.5' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]: dispatch 2024-04-23T10:21:21.242 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 23 10:21:20 smithi119 ceph-mon[32608]: from='osd.5 ' entity='osd.5' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]': finished 2024-04-23T10:21:21.242 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 23 10:21:20 smithi119 ceph-mon[32608]: osdmap e36: 6 total, 5 up, 6 in 2024-04-23T10:21:21.242 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 23 10:21:20 smithi119 ceph-mon[32608]: from='mgr.14152 172.21.15.114:0/1529706138' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-04-23T10:21:21.242 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 23 10:21:20 smithi119 ceph-mon[32608]: from='osd.5 [v2:172.21.15.204:6808/1427563312,v1:172.21.15.204:6809/1427563312]' entity='osd.5' cmd=[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi204", "root=default"]}]: dispatch 2024-04-23T10:21:21.243 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 23 10:21:20 smithi119 ceph-mon[32608]: from='osd.5 ' entity='osd.5' cmd=[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi204", "root=default"]}]: dispatch 2024-04-23T10:21:21.243 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 23 10:21:20 smithi119 ceph-mon[32608]: pgmap v133: 1 pgs: 1 active+clean; 577 KiB data, 143 MiB used, 447 GiB / 447 GiB avail 2024-04-23T10:21:21.254 INFO:journalctl@ceph.mon.a.smithi114.stdout:Apr 23 10:21:20 smithi114 ceph-mon[29255]: from='osd.5 ' entity='osd.5' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]': finished 2024-04-23T10:21:21.254 INFO:journalctl@ceph.mon.a.smithi114.stdout:Apr 23 10:21:20 smithi114 ceph-mon[29255]: osdmap e36: 6 total, 5 up, 6 in 2024-04-23T10:21:21.254 INFO:journalctl@ceph.mon.a.smithi114.stdout:Apr 23 10:21:20 smithi114 ceph-mon[29255]: from='mgr.14152 172.21.15.114:0/1529706138' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-04-23T10:21:21.254 INFO:journalctl@ceph.mon.a.smithi114.stdout:Apr 23 10:21:20 smithi114 ceph-mon[29255]: from='osd.5 [v2:172.21.15.204:6808/1427563312,v1:172.21.15.204:6809/1427563312]' entity='osd.5' cmd=[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi204", "root=default"]}]: dispatch 2024-04-23T10:21:21.255 INFO:journalctl@ceph.mon.a.smithi114.stdout:Apr 23 10:21:20 smithi114 ceph-mon[29255]: from='osd.5 ' entity='osd.5' cmd=[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi204", "root=default"]}]: dispatch 2024-04-23T10:21:21.255 INFO:journalctl@ceph.mon.a.smithi114.stdout:Apr 23 10:21:20 smithi114 ceph-mon[29255]: pgmap v133: 1 pgs: 1 active+clean; 577 KiB data, 143 MiB used, 447 GiB / 447 GiB avail 2024-04-23T10:21:21.328 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 23 10:21:20 smithi204 ceph-mon[32649]: from='osd.5 ' entity='osd.5' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]': finished 2024-04-23T10:21:21.328 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 23 10:21:20 smithi204 ceph-mon[32649]: osdmap e36: 6 total, 5 up, 6 in 2024-04-23T10:21:21.328 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 23 10:21:20 smithi204 ceph-mon[32649]: from='mgr.14152 172.21.15.114:0/1529706138' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-04-23T10:21:21.328 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 23 10:21:20 smithi204 ceph-mon[32649]: from='osd.5 [v2:172.21.15.204:6808/1427563312,v1:172.21.15.204:6809/1427563312]' entity='osd.5' cmd=[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi204", "root=default"]}]: dispatch 2024-04-23T10:21:21.328 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 23 10:21:20 smithi204 ceph-mon[32649]: from='osd.5 ' entity='osd.5' cmd=[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi204", "root=default"]}]: dispatch 2024-04-23T10:21:21.328 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 23 10:21:20 smithi204 ceph-mon[32649]: pgmap v133: 1 pgs: 1 active+clean; 577 KiB data, 143 MiB used, 447 GiB / 447 GiB avail 2024-04-23T10:21:21.384 INFO:teuthology.orchestra.run.smithi114.stderr:--> Zapping: /dev/vg_nvme/lv_2 2024-04-23T10:21:21.384 INFO:teuthology.orchestra.run.smithi114.stderr:Running command: /usr/bin/dd if=/dev/zero of=/dev/vg_nvme/lv_2 bs=1M count=10 conv=fsync 2024-04-23T10:21:21.384 INFO:teuthology.orchestra.run.smithi114.stderr: stderr: 10+0 records in 2024-04-23T10:21:21.384 INFO:teuthology.orchestra.run.smithi114.stderr:10+0 records out 2024-04-23T10:21:21.385 INFO:teuthology.orchestra.run.smithi114.stderr: stderr: 10485760 bytes (10 MB, 10 MiB) copied, 0.0114394 s, 917 MB/s 2024-04-23T10:21:21.385 INFO:teuthology.orchestra.run.smithi114.stderr:--> Zapping successful for: 2024-04-23T10:21:21.828 INFO:journalctl@ceph.osd.5.smithi204.stdout:Apr 23 10:21:21 smithi204 ceph-aa240572-0159-11ef-bc93-c7b262605968-osd-5[44334]: 2024-04-23T10:21:21.762+0000 7fbf0936f700 -1 osd.5 0 waiting for initial osdmap 2024-04-23T10:21:22.012 DEBUG:teuthology.orchestra.run.smithi114:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d69f13d827952bb07bb9b7973bca14f97d764d6f shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid aa240572-0159-11ef-bc93-c7b262605968 -- ceph orch daemon add osd smithi114:vg_nvme/lv_2 2024-04-23T10:21:22.150 INFO:journalctl@ceph.osd.5.smithi204.stdout:Apr 23 10:21:21 smithi204 ceph-aa240572-0159-11ef-bc93-c7b262605968-osd-5[44334]: 2024-04-23T10:21:21.857+0000 7fbf00959700 -1 osd.5 37 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-04-23T10:21:22.150 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 23 10:21:21 smithi204 ceph-mon[32649]: from='osd.5 ' entity='osd.5' cmd='[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi204", "root=default"]}]': finished 2024-04-23T10:21:22.150 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 23 10:21:21 smithi204 ceph-mon[32649]: osdmap e37: 6 total, 5 up, 6 in 2024-04-23T10:21:22.150 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 23 10:21:21 smithi204 ceph-mon[32649]: from='mgr.14152 172.21.15.114:0/1529706138' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-04-23T10:21:22.150 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 23 10:21:21 smithi204 ceph-mon[32649]: from='mgr.14152 172.21.15.114:0/1529706138' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-04-23T10:21:22.150 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 23 10:21:21 smithi204 ceph-mon[32649]: from='osd.5 ' entity='osd.5' 2024-04-23T10:21:22.242 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 23 10:21:21 smithi119 ceph-mon[32608]: from='osd.5 ' entity='osd.5' cmd='[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi204", "root=default"]}]': finished 2024-04-23T10:21:22.242 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 23 10:21:21 smithi119 ceph-mon[32608]: osdmap e37: 6 total, 5 up, 6 in 2024-04-23T10:21:22.242 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 23 10:21:21 smithi119 ceph-mon[32608]: from='mgr.14152 172.21.15.114:0/1529706138' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-04-23T10:21:22.242 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 23 10:21:21 smithi119 ceph-mon[32608]: from='mgr.14152 172.21.15.114:0/1529706138' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-04-23T10:21:22.242 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 23 10:21:21 smithi119 ceph-mon[32608]: from='osd.5 ' entity='osd.5' 2024-04-23T10:21:22.254 INFO:journalctl@ceph.mon.a.smithi114.stdout:Apr 23 10:21:21 smithi114 ceph-mon[29255]: from='osd.5 ' entity='osd.5' cmd='[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi204", "root=default"]}]': finished 2024-04-23T10:21:22.254 INFO:journalctl@ceph.mon.a.smithi114.stdout:Apr 23 10:21:21 smithi114 ceph-mon[29255]: osdmap e37: 6 total, 5 up, 6 in 2024-04-23T10:21:22.254 INFO:journalctl@ceph.mon.a.smithi114.stdout:Apr 23 10:21:21 smithi114 ceph-mon[29255]: from='mgr.14152 172.21.15.114:0/1529706138' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-04-23T10:21:22.254 INFO:journalctl@ceph.mon.a.smithi114.stdout:Apr 23 10:21:21 smithi114 ceph-mon[29255]: from='mgr.14152 172.21.15.114:0/1529706138' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-04-23T10:21:22.254 INFO:journalctl@ceph.mon.a.smithi114.stdout:Apr 23 10:21:21 smithi114 ceph-mon[29255]: from='osd.5 ' entity='osd.5' 2024-04-23T10:21:22.262 INFO:teuthology.orchestra.run.smithi114.stderr:Inferring config /var/lib/ceph/aa240572-0159-11ef-bc93-c7b262605968/mon.a/config 2024-04-23T10:21:23.004 INFO:journalctl@ceph.mon.a.smithi114.stdout:Apr 23 10:21:22 smithi114 ceph-mon[29255]: purged_snaps scrub starts 2024-04-23T10:21:23.005 INFO:journalctl@ceph.mon.a.smithi114.stdout:Apr 23 10:21:22 smithi114 ceph-mon[29255]: purged_snaps scrub ok 2024-04-23T10:21:23.005 INFO:journalctl@ceph.mon.a.smithi114.stdout:Apr 23 10:21:22 smithi114 ceph-mon[29255]: osdmap e38: 6 total, 5 up, 6 in 2024-04-23T10:21:23.005 INFO:journalctl@ceph.mon.a.smithi114.stdout:Apr 23 10:21:22 smithi114 ceph-mon[29255]: from='mgr.14152 172.21.15.114:0/1529706138' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-04-23T10:21:23.005 INFO:journalctl@ceph.mon.a.smithi114.stdout:Apr 23 10:21:22 smithi114 ceph-mon[29255]: pgmap v136: 1 pgs: 1 active+clean; 577 KiB data, 143 MiB used, 447 GiB / 447 GiB avail 2024-04-23T10:21:23.005 INFO:journalctl@ceph.mon.a.smithi114.stdout:Apr 23 10:21:22 smithi114 ceph-mon[29255]: from='mgr.14152 172.21.15.114:0/1529706138' entity='mgr.x' 2024-04-23T10:21:23.005 INFO:journalctl@ceph.mon.a.smithi114.stdout:Apr 23 10:21:22 smithi114 ceph-mon[29255]: from='mgr.14152 172.21.15.114:0/1529706138' entity='mgr.x' 2024-04-23T10:21:23.005 INFO:journalctl@ceph.mon.a.smithi114.stdout:Apr 23 10:21:22 smithi114 ceph-mon[29255]: from='mgr.14152 172.21.15.114:0/1529706138' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-04-23T10:21:23.005 INFO:journalctl@ceph.mon.a.smithi114.stdout:Apr 23 10:21:22 smithi114 ceph-mon[29255]: from='mgr.14152 172.21.15.114:0/1529706138' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-04-23T10:21:23.005 INFO:journalctl@ceph.mon.a.smithi114.stdout:Apr 23 10:21:22 smithi114 ceph-mon[29255]: from='mgr.14152 172.21.15.114:0/1529706138' entity='mgr.x' 2024-04-23T10:21:23.005 INFO:journalctl@ceph.mon.a.smithi114.stdout:Apr 23 10:21:22 smithi114 ceph-mon[29255]: from='mgr.14152 172.21.15.114:0/1529706138' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-23T10:21:23.006 INFO:journalctl@ceph.mon.a.smithi114.stdout:Apr 23 10:21:22 smithi114 ceph-mon[29255]: from='mgr.14152 172.21.15.114:0/1529706138' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-23T10:21:23.006 INFO:journalctl@ceph.mon.a.smithi114.stdout:Apr 23 10:21:22 smithi114 ceph-mon[29255]: from='mgr.14152 172.21.15.114:0/1529706138' entity='mgr.x' 2024-04-23T10:21:23.006 INFO:journalctl@ceph.mon.a.smithi114.stdout:Apr 23 10:21:22 smithi114 ceph-mon[29255]: from='mgr.14152 172.21.15.114:0/1529706138' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-04-23T10:21:23.242 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 23 10:21:22 smithi119 ceph-mon[32608]: purged_snaps scrub starts 2024-04-23T10:21:23.242 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 23 10:21:22 smithi119 ceph-mon[32608]: purged_snaps scrub ok 2024-04-23T10:21:23.242 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 23 10:21:22 smithi119 ceph-mon[32608]: osdmap e38: 6 total, 5 up, 6 in 2024-04-23T10:21:23.242 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 23 10:21:22 smithi119 ceph-mon[32608]: from='mgr.14152 172.21.15.114:0/1529706138' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-04-23T10:21:23.243 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 23 10:21:22 smithi119 ceph-mon[32608]: pgmap v136: 1 pgs: 1 active+clean; 577 KiB data, 143 MiB used, 447 GiB / 447 GiB avail 2024-04-23T10:21:23.243 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 23 10:21:22 smithi119 ceph-mon[32608]: from='mgr.14152 172.21.15.114:0/1529706138' entity='mgr.x' 2024-04-23T10:21:23.243 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 23 10:21:22 smithi119 ceph-mon[32608]: from='mgr.14152 172.21.15.114:0/1529706138' entity='mgr.x' 2024-04-23T10:21:23.243 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 23 10:21:22 smithi119 ceph-mon[32608]: from='mgr.14152 172.21.15.114:0/1529706138' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-04-23T10:21:23.243 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 23 10:21:22 smithi119 ceph-mon[32608]: from='mgr.14152 172.21.15.114:0/1529706138' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-04-23T10:21:23.243 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 23 10:21:22 smithi119 ceph-mon[32608]: from='mgr.14152 172.21.15.114:0/1529706138' entity='mgr.x' 2024-04-23T10:21:23.243 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 23 10:21:22 smithi119 ceph-mon[32608]: from='mgr.14152 172.21.15.114:0/1529706138' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-23T10:21:23.243 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 23 10:21:22 smithi119 ceph-mon[32608]: from='mgr.14152 172.21.15.114:0/1529706138' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-23T10:21:23.243 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 23 10:21:22 smithi119 ceph-mon[32608]: from='mgr.14152 172.21.15.114:0/1529706138' entity='mgr.x' 2024-04-23T10:21:23.243 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 23 10:21:22 smithi119 ceph-mon[32608]: from='mgr.14152 172.21.15.114:0/1529706138' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-04-23T10:21:23.328 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 23 10:21:22 smithi204 ceph-mon[32649]: purged_snaps scrub starts 2024-04-23T10:21:23.328 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 23 10:21:22 smithi204 ceph-mon[32649]: purged_snaps scrub ok 2024-04-23T10:21:23.328 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 23 10:21:22 smithi204 ceph-mon[32649]: osdmap e38: 6 total, 5 up, 6 in 2024-04-23T10:21:23.328 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 23 10:21:22 smithi204 ceph-mon[32649]: from='mgr.14152 172.21.15.114:0/1529706138' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-04-23T10:21:23.328 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 23 10:21:22 smithi204 ceph-mon[32649]: pgmap v136: 1 pgs: 1 active+clean; 577 KiB data, 143 MiB used, 447 GiB / 447 GiB avail 2024-04-23T10:21:23.328 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 23 10:21:22 smithi204 ceph-mon[32649]: from='mgr.14152 172.21.15.114:0/1529706138' entity='mgr.x' 2024-04-23T10:21:23.329 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 23 10:21:22 smithi204 ceph-mon[32649]: from='mgr.14152 172.21.15.114:0/1529706138' entity='mgr.x' 2024-04-23T10:21:23.329 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 23 10:21:22 smithi204 ceph-mon[32649]: from='mgr.14152 172.21.15.114:0/1529706138' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-04-23T10:21:23.329 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 23 10:21:22 smithi204 ceph-mon[32649]: from='mgr.14152 172.21.15.114:0/1529706138' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-04-23T10:21:23.329 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 23 10:21:22 smithi204 ceph-mon[32649]: from='mgr.14152 172.21.15.114:0/1529706138' entity='mgr.x' 2024-04-23T10:21:23.329 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 23 10:21:22 smithi204 ceph-mon[32649]: from='mgr.14152 172.21.15.114:0/1529706138' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-23T10:21:23.329 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 23 10:21:22 smithi204 ceph-mon[32649]: from='mgr.14152 172.21.15.114:0/1529706138' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-23T10:21:23.329 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 23 10:21:22 smithi204 ceph-mon[32649]: from='mgr.14152 172.21.15.114:0/1529706138' entity='mgr.x' 2024-04-23T10:21:23.329 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 23 10:21:22 smithi204 ceph-mon[32649]: from='mgr.14152 172.21.15.114:0/1529706138' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-04-23T10:21:23.801 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:21:23.798+0000 7f784628f700 1 -- 172.21.15.114:0/4053763300 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f7840101530 msgr2=0x7f784010b100 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T10:21:23.801 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:21:23.798+0000 7f784628f700 1 --2- 172.21.15.114:0/4053763300 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f7840101530 0x7f784010b100 secure :-1 s=READY pgs=131 cs=0 l=1 rev1=1 crypto rx=0x7f783800b0a0 tx=0x7f783800b3b0 comp rx=0 tx=0).stop 2024-04-23T10:21:23.801 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:21:23.799+0000 7f784628f700 1 -- 172.21.15.114:0/4053763300 shutdown_connections 2024-04-23T10:21:23.801 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:21:23.799+0000 7f784628f700 1 --2- 172.21.15.114:0/4053763300 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f7840101530 0x7f784010b100 unknown :-1 s=CLOSED pgs=131 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:21:23.801 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:21:23.799+0000 7f784628f700 1 --2- 172.21.15.114:0/4053763300 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f7840104110 0x7f7840104530 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:21:23.802 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:21:23.799+0000 7f784628f700 1 --2- 172.21.15.114:0/4053763300 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f7840100b90 0x7f7840100ff0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:21:23.802 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:21:23.799+0000 7f784628f700 1 -- 172.21.15.114:0/4053763300 >> 172.21.15.114:0/4053763300 conn(0x7f78400fc710 msgr2=0x7f78400feb50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T10:21:23.802 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:21:23.799+0000 7f784628f700 1 -- 172.21.15.114:0/4053763300 shutdown_connections 2024-04-23T10:21:23.802 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:21:23.799+0000 7f784628f700 1 -- 172.21.15.114:0/4053763300 wait complete. 2024-04-23T10:21:23.802 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:21:23.799+0000 7f784628f700 1 Processor -- start 2024-04-23T10:21:23.802 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:21:23.799+0000 7f784628f700 1 -- start start 2024-04-23T10:21:23.802 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:21:23.800+0000 7f784628f700 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f7840100b90 0x7f7840198ba0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T10:21:23.802 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:21:23.800+0000 7f784628f700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f7840101530 0x7f78401990e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T10:21:23.802 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:21:23.800+0000 7f784628f700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f7840104110 0x7f78401a0160 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T10:21:23.803 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:21:23.800+0000 7f784628f700 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x7f784010e5f0 con 0x7f7840104110 2024-04-23T10:21:23.803 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:21:23.800+0000 7f784628f700 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 -- 0x7f784010e470 con 0x7f7840101530 2024-04-23T10:21:23.803 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:21:23.800+0000 7f784628f700 1 -- --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_getmap magic: 0 -- 0x7f784010e770 con 0x7f7840100b90 2024-04-23T10:21:23.803 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:21:23.800+0000 7f784482c700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f7840104110 0x7f78401a0160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T10:21:23.803 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:21:23.800+0000 7f784482c700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f7840104110 0x7f78401a0160 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.114:55498/0 (socket says 172.21.15.114:55498) 2024-04-23T10:21:23.803 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:21:23.800+0000 7f784482c700 1 -- 172.21.15.114:0/235501402 learned_addr learned my addr 172.21.15.114:0/235501402 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-04-23T10:21:23.803 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:21:23.800+0000 7f783f7fe700 1 --2- 172.21.15.114:0/235501402 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f7840101530 0x7f78401990e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T10:21:23.803 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:21:23.800+0000 7f783ffff700 1 --2- 172.21.15.114:0/235501402 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f7840100b90 0x7f7840198ba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T10:21:23.803 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:21:23.800+0000 7f784482c700 1 -- 172.21.15.114:0/235501402 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f7840100b90 msgr2=0x7f7840198ba0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T10:21:23.804 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:21:23.800+0000 7f784482c700 1 --2- 172.21.15.114:0/235501402 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f7840100b90 0x7f7840198ba0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:21:23.804 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:21:23.800+0000 7f784482c700 1 -- 172.21.15.114:0/235501402 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f7840101530 msgr2=0x7f78401990e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T10:21:23.804 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:21:23.800+0000 7f784482c700 1 --2- 172.21.15.114:0/235501402 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f7840101530 0x7f78401990e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:21:23.804 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:21:23.800+0000 7f784482c700 1 -- 172.21.15.114:0/235501402 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f78401a0800 con 0x7f7840104110 2024-04-23T10:21:23.804 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:21:23.800+0000 7f784482c700 1 --2- 172.21.15.114:0/235501402 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f7840104110 0x7f78401a0160 secure :-1 s=READY pgs=132 cs=0 l=1 rev1=1 crypto rx=0x7f783800cf00 tx=0x7f7838007770 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T10:21:23.804 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:21:23.800+0000 7f783d7fa700 1 -- 172.21.15.114:0/235501402 <== mon.0 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7838021070 con 0x7f7840104110 2024-04-23T10:21:23.804 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:21:23.801+0000 7f784628f700 1 -- 172.21.15.114:0/235501402 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f7840101ca0 con 0x7f7840104110 2024-04-23T10:21:23.804 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:21:23.801+0000 7f783d7fa700 1 -- 172.21.15.114:0/235501402 <== mon.0 v2:172.21.15.114:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f783801d900 con 0x7f7840104110 2024-04-23T10:21:23.804 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:21:23.801+0000 7f783d7fa700 1 -- 172.21.15.114:0/235501402 <== mon.0 v2:172.21.15.114:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7838005460 con 0x7f7840104110 2024-04-23T10:21:23.804 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:21:23.801+0000 7f784628f700 1 -- 172.21.15.114:0/235501402 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f78401022b0 con 0x7f7840104110 2024-04-23T10:21:23.805 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:21:23.802+0000 7f784628f700 1 -- 172.21.15.114:0/235501402 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f7840069730 con 0x7f7840104110 2024-04-23T10:21:23.806 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:21:23.803+0000 7f783d7fa700 1 -- 172.21.15.114:0/235501402 <== mon.0 v2:172.21.15.114:3300/0 4 ==== mgrmap(e 16) ==== 138647+0+0 (secure 0 0 0) 0x7f7838024070 con 0x7f7840104110 2024-04-23T10:21:23.806 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:21:23.803+0000 7f783d7fa700 1 --2- 172.21.15.114:0/235501402 >> [v2:172.21.15.114:6800/465333808,v1:172.21.15.114:6801/465333808] conn(0x7f78280a3c40 0x7f78280a6100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T10:21:23.806 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:21:23.803+0000 7f783ffff700 1 --2- 172.21.15.114:0/235501402 >> [v2:172.21.15.114:6800/465333808,v1:172.21.15.114:6801/465333808] conn(0x7f78280a3c40 0x7f78280a6100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T10:21:23.806 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:21:23.803+0000 7f783d7fa700 1 -- 172.21.15.114:0/235501402 <== mon.0 v2:172.21.15.114:3300/0 5 ==== osd_map(40..40 src has 1..40) ==== 4562+0+0 (secure 0 0 0) 0x7f783800f020 con 0x7f7840104110 2024-04-23T10:21:23.806 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:21:23.804+0000 7f783ffff700 1 --2- 172.21.15.114:0/235501402 >> [v2:172.21.15.114:6800/465333808,v1:172.21.15.114:6801/465333808] conn(0x7f78280a3c40 0x7f78280a6100 secure :-1 s=READY pgs=85 cs=0 l=1 rev1=1 crypto rx=0x7f7830006fd0 tx=0x7f7830008040 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T10:21:23.807 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:21:23.808+0000 7f783d7fa700 1 -- 172.21.15.114:0/235501402 <== mon.0 v2:172.21.15.114:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f783802b020 con 0x7f7840104110 2024-04-23T10:21:24.016 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:21:24.017+0000 7f784628f700 1 -- 172.21.15.114:0/235501402 --> [v2:172.21.15.114:6800/465333808,v1:172.21.15.114:6801/465333808] -- mgr_command(tid 0: {"prefix": "orch daemon add osd", "svc_arg": "smithi114:vg_nvme/lv_2", "target": ["mon-mgr", ""]}) -- 0x7f7840062e80 con 0x7f78280a3c40 2024-04-23T10:21:24.242 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 23 10:21:23 smithi119 ceph-mon[32608]: Detected new or changed devices on smithi204 2024-04-23T10:21:24.242 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 23 10:21:23 smithi119 ceph-mon[32608]: Adjusting osd_memory_target on smithi204 to 8544M 2024-04-23T10:21:24.242 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 23 10:21:23 smithi119 ceph-mon[32608]: osd.5 [v2:172.21.15.204:6808/1427563312,v1:172.21.15.204:6809/1427563312] boot 2024-04-23T10:21:24.242 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 23 10:21:23 smithi119 ceph-mon[32608]: osdmap e39: 6 total, 6 up, 6 in 2024-04-23T10:21:24.242 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 23 10:21:23 smithi119 ceph-mon[32608]: from='mgr.14152 172.21.15.114:0/1529706138' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-04-23T10:21:24.242 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 23 10:21:23 smithi119 ceph-mon[32608]: osdmap e40: 6 total, 6 up, 6 in 2024-04-23T10:21:24.254 INFO:journalctl@ceph.mon.a.smithi114.stdout:Apr 23 10:21:23 smithi114 ceph-mon[29255]: Detected new or changed devices on smithi204 2024-04-23T10:21:24.254 INFO:journalctl@ceph.mon.a.smithi114.stdout:Apr 23 10:21:23 smithi114 ceph-mon[29255]: Adjusting osd_memory_target on smithi204 to 8544M 2024-04-23T10:21:24.254 INFO:journalctl@ceph.mon.a.smithi114.stdout:Apr 23 10:21:23 smithi114 ceph-mon[29255]: osd.5 [v2:172.21.15.204:6808/1427563312,v1:172.21.15.204:6809/1427563312] boot 2024-04-23T10:21:24.254 INFO:journalctl@ceph.mon.a.smithi114.stdout:Apr 23 10:21:23 smithi114 ceph-mon[29255]: osdmap e39: 6 total, 6 up, 6 in 2024-04-23T10:21:24.254 INFO:journalctl@ceph.mon.a.smithi114.stdout:Apr 23 10:21:23 smithi114 ceph-mon[29255]: from='mgr.14152 172.21.15.114:0/1529706138' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-04-23T10:21:24.254 INFO:journalctl@ceph.mon.a.smithi114.stdout:Apr 23 10:21:23 smithi114 ceph-mon[29255]: osdmap e40: 6 total, 6 up, 6 in 2024-04-23T10:21:24.328 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 23 10:21:23 smithi204 ceph-mon[32649]: Detected new or changed devices on smithi204 2024-04-23T10:21:24.328 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 23 10:21:23 smithi204 ceph-mon[32649]: Adjusting osd_memory_target on smithi204 to 8544M 2024-04-23T10:21:24.328 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 23 10:21:23 smithi204 ceph-mon[32649]: osd.5 [v2:172.21.15.204:6808/1427563312,v1:172.21.15.204:6809/1427563312] boot 2024-04-23T10:21:24.328 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 23 10:21:23 smithi204 ceph-mon[32649]: osdmap e39: 6 total, 6 up, 6 in 2024-04-23T10:21:24.328 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 23 10:21:23 smithi204 ceph-mon[32649]: from='mgr.14152 172.21.15.114:0/1529706138' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-04-23T10:21:24.328 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 23 10:21:23 smithi204 ceph-mon[32649]: osdmap e40: 6 total, 6 up, 6 in 2024-04-23T10:21:25.143 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 23 10:21:24 smithi119 ceph-mon[32608]: pgmap v139: 1 pgs: 1 active+clean; 577 KiB data, 174 MiB used, 536 GiB / 536 GiB avail 2024-04-23T10:21:25.143 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 23 10:21:24 smithi119 ceph-mon[32608]: from='client.14367 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi114:vg_nvme/lv_2", "target": ["mon-mgr", ""]}]: dispatch 2024-04-23T10:21:25.143 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 23 10:21:24 smithi119 ceph-mon[32608]: from='mgr.14152 172.21.15.114:0/1529706138' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-04-23T10:21:25.144 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 23 10:21:24 smithi119 ceph-mon[32608]: from='mgr.14152 172.21.15.114:0/1529706138' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-04-23T10:21:25.144 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 23 10:21:24 smithi119 ceph-mon[32608]: from='mgr.14152 172.21.15.114:0/1529706138' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-23T10:21:25.144 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 23 10:21:24 smithi119 ceph-mon[32608]: osdmap e41: 6 total, 6 up, 6 in 2024-04-23T10:21:25.158 INFO:journalctl@ceph.mon.a.smithi114.stdout:Apr 23 10:21:24 smithi114 ceph-mon[29255]: pgmap v139: 1 pgs: 1 active+clean; 577 KiB data, 174 MiB used, 536 GiB / 536 GiB avail 2024-04-23T10:21:25.158 INFO:journalctl@ceph.mon.a.smithi114.stdout:Apr 23 10:21:24 smithi114 ceph-mon[29255]: from='client.14367 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi114:vg_nvme/lv_2", "target": ["mon-mgr", ""]}]: dispatch 2024-04-23T10:21:25.158 INFO:journalctl@ceph.mon.a.smithi114.stdout:Apr 23 10:21:24 smithi114 ceph-mon[29255]: from='mgr.14152 172.21.15.114:0/1529706138' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-04-23T10:21:25.158 INFO:journalctl@ceph.mon.a.smithi114.stdout:Apr 23 10:21:24 smithi114 ceph-mon[29255]: from='mgr.14152 172.21.15.114:0/1529706138' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-04-23T10:21:25.158 INFO:journalctl@ceph.mon.a.smithi114.stdout:Apr 23 10:21:24 smithi114 ceph-mon[29255]: from='mgr.14152 172.21.15.114:0/1529706138' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-23T10:21:25.159 INFO:journalctl@ceph.mon.a.smithi114.stdout:Apr 23 10:21:24 smithi114 ceph-mon[29255]: osdmap e41: 6 total, 6 up, 6 in 2024-04-23T10:21:25.229 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 23 10:21:24 smithi204 ceph-mon[32649]: pgmap v139: 1 pgs: 1 active+clean; 577 KiB data, 174 MiB used, 536 GiB / 536 GiB avail 2024-04-23T10:21:25.229 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 23 10:21:24 smithi204 ceph-mon[32649]: from='client.14367 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi114:vg_nvme/lv_2", "target": ["mon-mgr", ""]}]: dispatch 2024-04-23T10:21:25.229 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 23 10:21:24 smithi204 ceph-mon[32649]: from='mgr.14152 172.21.15.114:0/1529706138' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-04-23T10:21:25.229 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 23 10:21:24 smithi204 ceph-mon[32649]: from='mgr.14152 172.21.15.114:0/1529706138' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-04-23T10:21:25.229 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 23 10:21:24 smithi204 ceph-mon[32649]: from='mgr.14152 172.21.15.114:0/1529706138' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-23T10:21:25.230 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 23 10:21:24 smithi204 ceph-mon[32649]: osdmap e41: 6 total, 6 up, 6 in 2024-04-23T10:21:26.578 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 23 10:21:26 smithi204 ceph-mon[32649]: osdmap e42: 6 total, 6 up, 6 in 2024-04-23T10:21:26.578 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 23 10:21:26 smithi204 ceph-mon[32649]: pgmap v142: 1 pgs: 1 peering; 577 KiB data, 174 MiB used, 536 GiB / 536 GiB avail; 144 KiB/s, 0 objects/s recovering 2024-04-23T10:21:26.742 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 23 10:21:26 smithi119 ceph-mon[32608]: osdmap e42: 6 total, 6 up, 6 in 2024-04-23T10:21:26.742 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 23 10:21:26 smithi119 ceph-mon[32608]: pgmap v142: 1 pgs: 1 peering; 577 KiB data, 174 MiB used, 536 GiB / 536 GiB avail; 144 KiB/s, 0 objects/s recovering 2024-04-23T10:21:26.754 INFO:journalctl@ceph.mon.a.smithi114.stdout:Apr 23 10:21:26 smithi114 ceph-mon[29255]: osdmap e42: 6 total, 6 up, 6 in 2024-04-23T10:21:26.754 INFO:journalctl@ceph.mon.a.smithi114.stdout:Apr 23 10:21:26 smithi114 ceph-mon[29255]: pgmap v142: 1 pgs: 1 peering; 577 KiB data, 174 MiB used, 536 GiB / 536 GiB avail; 144 KiB/s, 0 objects/s recovering 2024-04-23T10:21:27.504 INFO:journalctl@ceph.mon.a.smithi114.stdout:Apr 23 10:21:27 smithi114 ceph-mon[29255]: osdmap e43: 6 total, 6 up, 6 in 2024-04-23T10:21:27.504 INFO:journalctl@ceph.mon.a.smithi114.stdout:Apr 23 10:21:27 smithi114 ceph-mon[29255]: from='client.? 172.21.15.114:0/3931874388' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "534d20e8-3e54-4db8-a845-6119f49c7a78"}]: dispatch 2024-04-23T10:21:27.504 INFO:journalctl@ceph.mon.a.smithi114.stdout:Apr 23 10:21:27 smithi114 ceph-mon[29255]: from='client.? 172.21.15.114:0/3931874388' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "534d20e8-3e54-4db8-a845-6119f49c7a78"}]': finished 2024-04-23T10:21:27.504 INFO:journalctl@ceph.mon.a.smithi114.stdout:Apr 23 10:21:27 smithi114 ceph-mon[29255]: osdmap e44: 7 total, 6 up, 7 in 2024-04-23T10:21:27.504 INFO:journalctl@ceph.mon.a.smithi114.stdout:Apr 23 10:21:27 smithi114 ceph-mon[29255]: from='mgr.14152 172.21.15.114:0/1529706138' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-04-23T10:21:27.578 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 23 10:21:27 smithi204 ceph-mon[32649]: osdmap e43: 6 total, 6 up, 6 in 2024-04-23T10:21:27.578 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 23 10:21:27 smithi204 ceph-mon[32649]: from='client.? 172.21.15.114:0/3931874388' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "534d20e8-3e54-4db8-a845-6119f49c7a78"}]: dispatch 2024-04-23T10:21:27.578 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 23 10:21:27 smithi204 ceph-mon[32649]: from='client.? 172.21.15.114:0/3931874388' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "534d20e8-3e54-4db8-a845-6119f49c7a78"}]': finished 2024-04-23T10:21:27.578 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 23 10:21:27 smithi204 ceph-mon[32649]: osdmap e44: 7 total, 6 up, 7 in 2024-04-23T10:21:27.578 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 23 10:21:27 smithi204 ceph-mon[32649]: from='mgr.14152 172.21.15.114:0/1529706138' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-04-23T10:21:27.742 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 23 10:21:27 smithi119 ceph-mon[32608]: osdmap e43: 6 total, 6 up, 6 in 2024-04-23T10:21:27.742 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 23 10:21:27 smithi119 ceph-mon[32608]: from='client.? 172.21.15.114:0/3931874388' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "534d20e8-3e54-4db8-a845-6119f49c7a78"}]: dispatch 2024-04-23T10:21:27.742 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 23 10:21:27 smithi119 ceph-mon[32608]: from='client.? 172.21.15.114:0/3931874388' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "534d20e8-3e54-4db8-a845-6119f49c7a78"}]': finished 2024-04-23T10:21:27.742 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 23 10:21:27 smithi119 ceph-mon[32608]: osdmap e44: 7 total, 6 up, 7 in 2024-04-23T10:21:27.742 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 23 10:21:27 smithi119 ceph-mon[32608]: from='mgr.14152 172.21.15.114:0/1529706138' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-04-23T10:21:28.578 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 23 10:21:28 smithi204 ceph-mon[32649]: from='client.? 172.21.15.114:0/357599187' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-04-23T10:21:28.578 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 23 10:21:28 smithi204 ceph-mon[32649]: pgmap v145: 1 pgs: 1 peering; 577 KiB data, 174 MiB used, 536 GiB / 536 GiB avail; 144 KiB/s, 0 objects/s recovering 2024-04-23T10:21:28.742 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 23 10:21:28 smithi119 ceph-mon[32608]: from='client.? 172.21.15.114:0/357599187' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-04-23T10:21:28.742 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 23 10:21:28 smithi119 ceph-mon[32608]: pgmap v145: 1 pgs: 1 peering; 577 KiB data, 174 MiB used, 536 GiB / 536 GiB avail; 144 KiB/s, 0 objects/s recovering 2024-04-23T10:21:28.754 INFO:journalctl@ceph.mon.a.smithi114.stdout:Apr 23 10:21:28 smithi114 ceph-mon[29255]: from='client.? 172.21.15.114:0/357599187' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-04-23T10:21:28.754 INFO:journalctl@ceph.mon.a.smithi114.stdout:Apr 23 10:21:28 smithi114 ceph-mon[29255]: pgmap v145: 1 pgs: 1 peering; 577 KiB data, 174 MiB used, 536 GiB / 536 GiB avail; 144 KiB/s, 0 objects/s recovering 2024-04-23T10:21:31.235 INFO:journalctl@ceph.mon.a.smithi114.stdout:Apr 23 10:21:30 smithi114 ceph-mon[29255]: pgmap v146: 1 pgs: 1 peering; 577 KiB data, 174 MiB used, 536 GiB / 536 GiB avail; 102 KiB/s, 0 objects/s recovering 2024-04-23T10:21:31.242 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 23 10:21:30 smithi119 ceph-mon[32608]: pgmap v146: 1 pgs: 1 peering; 577 KiB data, 174 MiB used, 536 GiB / 536 GiB avail; 102 KiB/s, 0 objects/s recovering 2024-04-23T10:21:31.328 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 23 10:21:30 smithi204 ceph-mon[32649]: pgmap v146: 1 pgs: 1 peering; 577 KiB data, 174 MiB used, 536 GiB / 536 GiB avail; 102 KiB/s, 0 objects/s recovering 2024-04-23T10:21:33.178 INFO:journalctl@ceph.mon.a.smithi114.stdout:Apr 23 10:21:32 smithi114 ceph-mon[29255]: pgmap v147: 1 pgs: 1 active+clean; 577 KiB data, 175 MiB used, 536 GiB / 536 GiB avail; 87 KiB/s, 0 objects/s recovering 2024-04-23T10:21:33.242 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 23 10:21:32 smithi119 ceph-mon[32608]: pgmap v147: 1 pgs: 1 active+clean; 577 KiB data, 175 MiB used, 536 GiB / 536 GiB avail; 87 KiB/s, 0 objects/s recovering 2024-04-23T10:21:33.328 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 23 10:21:32 smithi204 ceph-mon[32649]: pgmap v147: 1 pgs: 1 active+clean; 577 KiB data, 175 MiB used, 536 GiB / 536 GiB avail; 87 KiB/s, 0 objects/s recovering 2024-04-23T10:21:35.004 INFO:journalctl@ceph.mon.a.smithi114.stdout:Apr 23 10:21:34 smithi114 ceph-mon[29255]: pgmap v148: 1 pgs: 1 active+clean; 577 KiB data, 175 MiB used, 536 GiB / 536 GiB avail 2024-04-23T10:21:35.004 INFO:journalctl@ceph.mon.a.smithi114.stdout:Apr 23 10:21:34 smithi114 ceph-mon[29255]: from='mgr.14152 172.21.15.114:0/1529706138' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.6"}]: dispatch 2024-04-23T10:21:35.004 INFO:journalctl@ceph.mon.a.smithi114.stdout:Apr 23 10:21:34 smithi114 ceph-mon[29255]: from='mgr.14152 172.21.15.114:0/1529706138' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-23T10:21:35.242 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 23 10:21:34 smithi119 ceph-mon[32608]: pgmap v148: 1 pgs: 1 active+clean; 577 KiB data, 175 MiB used, 536 GiB / 536 GiB avail 2024-04-23T10:21:35.242 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 23 10:21:34 smithi119 ceph-mon[32608]: from='mgr.14152 172.21.15.114:0/1529706138' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.6"}]: dispatch 2024-04-23T10:21:35.242 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 23 10:21:34 smithi119 ceph-mon[32608]: from='mgr.14152 172.21.15.114:0/1529706138' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-23T10:21:35.328 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 23 10:21:34 smithi204 ceph-mon[32649]: pgmap v148: 1 pgs: 1 active+clean; 577 KiB data, 175 MiB used, 536 GiB / 536 GiB avail 2024-04-23T10:21:35.328 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 23 10:21:34 smithi204 ceph-mon[32649]: from='mgr.14152 172.21.15.114:0/1529706138' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.6"}]: dispatch 2024-04-23T10:21:35.328 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 23 10:21:34 smithi204 ceph-mon[32649]: from='mgr.14152 172.21.15.114:0/1529706138' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-23T10:21:36.242 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 23 10:21:35 smithi119 ceph-mon[32608]: Deploying daemon osd.6 on smithi114 2024-04-23T10:21:36.254 INFO:journalctl@ceph.mon.a.smithi114.stdout:Apr 23 10:21:35 smithi114 ceph-mon[29255]: Deploying daemon osd.6 on smithi114 2024-04-23T10:21:36.328 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 23 10:21:35 smithi204 ceph-mon[32649]: Deploying daemon osd.6 on smithi114 2024-04-23T10:21:37.242 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 23 10:21:36 smithi119 ceph-mon[32608]: pgmap v149: 1 pgs: 1 active+clean; 577 KiB data, 175 MiB used, 536 GiB / 536 GiB avail 2024-04-23T10:21:37.254 INFO:journalctl@ceph.mon.a.smithi114.stdout:Apr 23 10:21:36 smithi114 ceph-mon[29255]: pgmap v149: 1 pgs: 1 active+clean; 577 KiB data, 175 MiB used, 536 GiB / 536 GiB avail 2024-04-23T10:21:37.328 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 23 10:21:36 smithi204 ceph-mon[32649]: pgmap v149: 1 pgs: 1 active+clean; 577 KiB data, 175 MiB used, 536 GiB / 536 GiB avail 2024-04-23T10:21:39.238 INFO:journalctl@ceph.mon.a.smithi114.stdout:Apr 23 10:21:38 smithi114 ceph-mon[29255]: pgmap v150: 1 pgs: 1 active+clean; 577 KiB data, 175 MiB used, 536 GiB / 536 GiB avail 2024-04-23T10:21:39.242 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 23 10:21:38 smithi119 ceph-mon[32608]: pgmap v150: 1 pgs: 1 active+clean; 577 KiB data, 175 MiB used, 536 GiB / 536 GiB avail 2024-04-23T10:21:39.328 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 23 10:21:38 smithi204 ceph-mon[32649]: pgmap v150: 1 pgs: 1 active+clean; 577 KiB data, 175 MiB used, 536 GiB / 536 GiB avail 2024-04-23T10:21:40.754 INFO:journalctl@ceph.mon.a.smithi114.stdout:Apr 23 10:21:40 smithi114 ceph-mon[29255]: from='mgr.14152 172.21.15.114:0/1529706138' entity='mgr.x' 2024-04-23T10:21:40.754 INFO:journalctl@ceph.mon.a.smithi114.stdout:Apr 23 10:21:40 smithi114 ceph-mon[29255]: from='mgr.14152 172.21.15.114:0/1529706138' entity='mgr.x' 2024-04-23T10:21:40.754 INFO:journalctl@ceph.mon.a.smithi114.stdout:Apr 23 10:21:40 smithi114 ceph-mon[29255]: from='mgr.14152 172.21.15.114:0/1529706138' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-23T10:21:40.754 INFO:journalctl@ceph.mon.a.smithi114.stdout:Apr 23 10:21:40 smithi114 ceph-mon[29255]: pgmap v151: 1 pgs: 1 active+clean; 577 KiB data, 175 MiB used, 536 GiB / 536 GiB avail 2024-04-23T10:21:40.828 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 23 10:21:40 smithi204 ceph-mon[32649]: from='mgr.14152 172.21.15.114:0/1529706138' entity='mgr.x' 2024-04-23T10:21:40.828 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 23 10:21:40 smithi204 ceph-mon[32649]: from='mgr.14152 172.21.15.114:0/1529706138' entity='mgr.x' 2024-04-23T10:21:40.828 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 23 10:21:40 smithi204 ceph-mon[32649]: from='mgr.14152 172.21.15.114:0/1529706138' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-23T10:21:40.828 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 23 10:21:40 smithi204 ceph-mon[32649]: pgmap v151: 1 pgs: 1 active+clean; 577 KiB data, 175 MiB used, 536 GiB / 536 GiB avail 2024-04-23T10:21:40.992 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 23 10:21:40 smithi119 ceph-mon[32608]: from='mgr.14152 172.21.15.114:0/1529706138' entity='mgr.x' 2024-04-23T10:21:40.992 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 23 10:21:40 smithi119 ceph-mon[32608]: from='mgr.14152 172.21.15.114:0/1529706138' entity='mgr.x' 2024-04-23T10:21:40.992 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 23 10:21:40 smithi119 ceph-mon[32608]: from='mgr.14152 172.21.15.114:0/1529706138' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-23T10:21:40.992 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 23 10:21:40 smithi119 ceph-mon[32608]: pgmap v151: 1 pgs: 1 active+clean; 577 KiB data, 175 MiB used, 536 GiB / 536 GiB avail 2024-04-23T10:21:43.024 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:21:43.024+0000 7f783d7fa700 1 -- 172.21.15.114:0/235501402 <== mgr.14152 v2:172.21.15.114:6800/465333808 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+37 (secure 0 0 0) 0x7f7840062e80 con 0x7f78280a3c40 2024-04-23T10:21:43.024 INFO:teuthology.orchestra.run.smithi114.stdout:Created osd(s) 6 on host 'smithi114' 2024-04-23T10:21:43.026 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:21:43.027+0000 7f784628f700 1 -- 172.21.15.114:0/235501402 >> [v2:172.21.15.114:6800/465333808,v1:172.21.15.114:6801/465333808] conn(0x7f78280a3c40 msgr2=0x7f78280a6100 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T10:21:43.026 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:21:43.027+0000 7f784628f700 1 --2- 172.21.15.114:0/235501402 >> [v2:172.21.15.114:6800/465333808,v1:172.21.15.114:6801/465333808] conn(0x7f78280a3c40 0x7f78280a6100 secure :-1 s=READY pgs=85 cs=0 l=1 rev1=1 crypto rx=0x7f7830006fd0 tx=0x7f7830008040 comp rx=0 tx=0).stop 2024-04-23T10:21:43.026 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:21:43.027+0000 7f784628f700 1 -- 172.21.15.114:0/235501402 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f7840104110 msgr2=0x7f78401a0160 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T10:21:43.026 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:21:43.027+0000 7f784628f700 1 --2- 172.21.15.114:0/235501402 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f7840104110 0x7f78401a0160 secure :-1 s=READY pgs=132 cs=0 l=1 rev1=1 crypto rx=0x7f783800cf00 tx=0x7f7838007770 comp rx=0 tx=0).stop 2024-04-23T10:21:43.027 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:21:43.027+0000 7f784628f700 1 -- 172.21.15.114:0/235501402 shutdown_connections 2024-04-23T10:21:43.027 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:21:43.027+0000 7f784628f700 1 --2- 172.21.15.114:0/235501402 >> [v2:172.21.15.114:6800/465333808,v1:172.21.15.114:6801/465333808] conn(0x7f78280a3c40 0x7f78280a6100 unknown :-1 s=CLOSED pgs=85 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:21:43.027 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:21:43.027+0000 7f784628f700 1 --2- 172.21.15.114:0/235501402 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f7840104110 0x7f78401a0160 unknown :-1 s=CLOSED pgs=132 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:21:43.027 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:21:43.027+0000 7f784628f700 1 --2- 172.21.15.114:0/235501402 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f7840100b90 0x7f7840198ba0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:21:43.027 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:21:43.027+0000 7f784628f700 1 --2- 172.21.15.114:0/235501402 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f7840101530 0x7f78401990e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:21:43.027 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:21:43.027+0000 7f784628f700 1 -- 172.21.15.114:0/235501402 >> 172.21.15.114:0/235501402 conn(0x7f78400fc710 msgr2=0x7f7840107150 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T10:21:43.027 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:21:43.027+0000 7f784628f700 1 -- 172.21.15.114:0/235501402 shutdown_connections 2024-04-23T10:21:43.027 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:21:43.027+0000 7f784628f700 1 -- 172.21.15.114:0/235501402 wait complete. 2024-04-23T10:21:43.242 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 23 10:21:42 smithi119 ceph-mon[32608]: pgmap v152: 1 pgs: 1 active+clean; 577 KiB data, 175 MiB used, 536 GiB / 536 GiB avail 2024-04-23T10:21:43.254 INFO:journalctl@ceph.mon.a.smithi114.stdout:Apr 23 10:21:42 smithi114 ceph-mon[29255]: pgmap v152: 1 pgs: 1 active+clean; 577 KiB data, 175 MiB used, 536 GiB / 536 GiB avail 2024-04-23T10:21:43.328 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 23 10:21:42 smithi204 ceph-mon[32649]: pgmap v152: 1 pgs: 1 active+clean; 577 KiB data, 175 MiB used, 536 GiB / 536 GiB avail 2024-04-23T10:21:43.685 DEBUG:teuthology.orchestra.run.smithi114:osd.6> sudo journalctl -f -n 0 -u ceph-aa240572-0159-11ef-bc93-c7b262605968@osd.6.service 2024-04-23T10:21:43.687 INFO:tasks.cephadm:Deploying osd.7 on smithi119 with /dev/vg_nvme/lv_2... 2024-04-23T10:21:43.687 DEBUG:teuthology.orchestra.run.smithi119:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d69f13d827952bb07bb9b7973bca14f97d764d6f shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid aa240572-0159-11ef-bc93-c7b262605968 -- ceph-volume lvm zap /dev/vg_nvme/lv_2 2024-04-23T10:21:43.933 INFO:teuthology.orchestra.run.smithi119.stderr:Inferring config /var/lib/ceph/aa240572-0159-11ef-bc93-c7b262605968/mon.b/config 2024-04-23T10:21:44.242 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 23 10:21:43 smithi119 ceph-mon[32608]: from='mgr.14152 172.21.15.114:0/1529706138' entity='mgr.x' 2024-04-23T10:21:44.242 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 23 10:21:43 smithi119 ceph-mon[32608]: from='mgr.14152 172.21.15.114:0/1529706138' entity='mgr.x' 2024-04-23T10:21:44.242 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 23 10:21:43 smithi119 ceph-mon[32608]: from='osd.6 [v2:172.21.15.114:6818/4072912544,v1:172.21.15.114:6819/4072912544]' entity='osd.6' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["6"]}]: dispatch 2024-04-23T10:21:44.242 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 23 10:21:43 smithi119 ceph-mon[32608]: from='mgr.14152 172.21.15.114:0/1529706138' entity='mgr.x' 2024-04-23T10:21:44.243 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 23 10:21:43 smithi119 ceph-mon[32608]: from='mgr.14152 172.21.15.114:0/1529706138' entity='mgr.x' 2024-04-23T10:21:44.254 INFO:journalctl@ceph.mon.a.smithi114.stdout:Apr 23 10:21:43 smithi114 ceph-mon[29255]: from='mgr.14152 172.21.15.114:0/1529706138' entity='mgr.x' 2024-04-23T10:21:44.254 INFO:journalctl@ceph.mon.a.smithi114.stdout:Apr 23 10:21:43 smithi114 ceph-mon[29255]: from='mgr.14152 172.21.15.114:0/1529706138' entity='mgr.x' 2024-04-23T10:21:44.254 INFO:journalctl@ceph.mon.a.smithi114.stdout:Apr 23 10:21:43 smithi114 ceph-mon[29255]: from='osd.6 [v2:172.21.15.114:6818/4072912544,v1:172.21.15.114:6819/4072912544]' entity='osd.6' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["6"]}]: dispatch 2024-04-23T10:21:44.254 INFO:journalctl@ceph.mon.a.smithi114.stdout:Apr 23 10:21:43 smithi114 ceph-mon[29255]: from='mgr.14152 172.21.15.114:0/1529706138' entity='mgr.x' 2024-04-23T10:21:44.254 INFO:journalctl@ceph.mon.a.smithi114.stdout:Apr 23 10:21:43 smithi114 ceph-mon[29255]: from='mgr.14152 172.21.15.114:0/1529706138' entity='mgr.x' 2024-04-23T10:21:44.328 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 23 10:21:43 smithi204 ceph-mon[32649]: from='mgr.14152 172.21.15.114:0/1529706138' entity='mgr.x' 2024-04-23T10:21:44.328 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 23 10:21:43 smithi204 ceph-mon[32649]: from='mgr.14152 172.21.15.114:0/1529706138' entity='mgr.x' 2024-04-23T10:21:44.328 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 23 10:21:43 smithi204 ceph-mon[32649]: from='osd.6 [v2:172.21.15.114:6818/4072912544,v1:172.21.15.114:6819/4072912544]' entity='osd.6' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["6"]}]: dispatch 2024-04-23T10:21:44.328 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 23 10:21:43 smithi204 ceph-mon[32649]: from='mgr.14152 172.21.15.114:0/1529706138' entity='mgr.x' 2024-04-23T10:21:44.328 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 23 10:21:43 smithi204 ceph-mon[32649]: from='mgr.14152 172.21.15.114:0/1529706138' entity='mgr.x' 2024-04-23T10:21:45.270 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 23 10:21:45 smithi119 ceph-mon[32608]: pgmap v153: 1 pgs: 1 active+clean; 577 KiB data, 175 MiB used, 536 GiB / 536 GiB avail 2024-04-23T10:21:45.270 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 23 10:21:45 smithi119 ceph-mon[32608]: from='osd.6 [v2:172.21.15.114:6818/4072912544,v1:172.21.15.114:6819/4072912544]' entity='osd.6' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["6"]}]': finished 2024-04-23T10:21:45.270 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 23 10:21:45 smithi119 ceph-mon[32608]: osdmap e45: 7 total, 6 up, 7 in 2024-04-23T10:21:45.270 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 23 10:21:45 smithi119 ceph-mon[32608]: from='osd.6 [v2:172.21.15.114:6818/4072912544,v1:172.21.15.114:6819/4072912544]' entity='osd.6' cmd=[{"prefix": "osd crush create-or-move", "id": 6, "weight":0.0873, "args": ["host=smithi114", "root=default"]}]: dispatch 2024-04-23T10:21:45.270 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 23 10:21:45 smithi119 ceph-mon[32608]: from='mgr.14152 172.21.15.114:0/1529706138' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-04-23T10:21:45.328 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 23 10:21:45 smithi204 ceph-mon[32649]: pgmap v153: 1 pgs: 1 active+clean; 577 KiB data, 175 MiB used, 536 GiB / 536 GiB avail 2024-04-23T10:21:45.328 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 23 10:21:45 smithi204 ceph-mon[32649]: from='osd.6 [v2:172.21.15.114:6818/4072912544,v1:172.21.15.114:6819/4072912544]' entity='osd.6' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["6"]}]': finished 2024-04-23T10:21:45.328 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 23 10:21:45 smithi204 ceph-mon[32649]: osdmap e45: 7 total, 6 up, 7 in 2024-04-23T10:21:45.328 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 23 10:21:45 smithi204 ceph-mon[32649]: from='osd.6 [v2:172.21.15.114:6818/4072912544,v1:172.21.15.114:6819/4072912544]' entity='osd.6' cmd=[{"prefix": "osd crush create-or-move", "id": 6, "weight":0.0873, "args": ["host=smithi114", "root=default"]}]: dispatch 2024-04-23T10:21:45.328 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 23 10:21:45 smithi204 ceph-mon[32649]: from='mgr.14152 172.21.15.114:0/1529706138' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-04-23T10:21:45.419 INFO:journalctl@ceph.mon.a.smithi114.stdout:Apr 23 10:21:45 smithi114 ceph-mon[29255]: pgmap v153: 1 pgs: 1 active+clean; 577 KiB data, 175 MiB used, 536 GiB / 536 GiB avail 2024-04-23T10:21:45.419 INFO:journalctl@ceph.mon.a.smithi114.stdout:Apr 23 10:21:45 smithi114 ceph-mon[29255]: from='osd.6 [v2:172.21.15.114:6818/4072912544,v1:172.21.15.114:6819/4072912544]' entity='osd.6' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["6"]}]': finished 2024-04-23T10:21:45.419 INFO:journalctl@ceph.mon.a.smithi114.stdout:Apr 23 10:21:45 smithi114 ceph-mon[29255]: osdmap e45: 7 total, 6 up, 7 in 2024-04-23T10:21:45.419 INFO:journalctl@ceph.mon.a.smithi114.stdout:Apr 23 10:21:45 smithi114 ceph-mon[29255]: from='osd.6 [v2:172.21.15.114:6818/4072912544,v1:172.21.15.114:6819/4072912544]' entity='osd.6' cmd=[{"prefix": "osd crush create-or-move", "id": 6, "weight":0.0873, "args": ["host=smithi114", "root=default"]}]: dispatch 2024-04-23T10:21:45.419 INFO:journalctl@ceph.mon.a.smithi114.stdout:Apr 23 10:21:45 smithi114 ceph-mon[29255]: from='mgr.14152 172.21.15.114:0/1529706138' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-04-23T10:21:45.914 INFO:teuthology.orchestra.run.smithi119.stderr:--> Zapping: /dev/vg_nvme/lv_2 2024-04-23T10:21:45.914 INFO:teuthology.orchestra.run.smithi119.stderr:Running command: /usr/bin/dd if=/dev/zero of=/dev/vg_nvme/lv_2 bs=1M count=10 conv=fsync 2024-04-23T10:21:45.914 INFO:teuthology.orchestra.run.smithi119.stderr: stderr: 10+0 records in 2024-04-23T10:21:45.914 INFO:teuthology.orchestra.run.smithi119.stderr:10+0 records out 2024-04-23T10:21:45.914 INFO:teuthology.orchestra.run.smithi119.stderr:10485760 bytes (10 MB, 10 MiB) copied, 0.0112848 s, 929 MB/s 2024-04-23T10:21:45.914 INFO:teuthology.orchestra.run.smithi119.stderr:--> Zapping successful for: 2024-04-23T10:21:46.004 INFO:journalctl@ceph.osd.6.smithi114.stdout:Apr 23 10:21:45 smithi114 ceph-aa240572-0159-11ef-bc93-c7b262605968-osd-6[55044]: 2024-04-23T10:21:45.900+0000 7fdbab9b6700 -1 osd.6 0 waiting for initial osdmap 2024-04-23T10:21:46.242 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 23 10:21:46 smithi119 ceph-mon[32608]: from='osd.6 [v2:172.21.15.114:6818/4072912544,v1:172.21.15.114:6819/4072912544]' entity='osd.6' cmd='[{"prefix": "osd crush create-or-move", "id": 6, "weight":0.0873, "args": ["host=smithi114", "root=default"]}]': finished 2024-04-23T10:21:46.242 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 23 10:21:46 smithi119 ceph-mon[32608]: osdmap e46: 7 total, 6 up, 7 in 2024-04-23T10:21:46.242 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 23 10:21:46 smithi119 ceph-mon[32608]: from='mgr.14152 172.21.15.114:0/1529706138' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-04-23T10:21:46.242 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 23 10:21:46 smithi119 ceph-mon[32608]: from='mgr.14152 172.21.15.114:0/1529706138' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-04-23T10:21:46.242 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 23 10:21:46 smithi119 ceph-mon[32608]: from='osd.6 [v2:172.21.15.114:6818/4072912544,v1:172.21.15.114:6819/4072912544]' entity='osd.6' 2024-04-23T10:21:46.242 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 23 10:21:46 smithi119 ceph-mon[32608]: from='mgr.14152 172.21.15.114:0/1529706138' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-04-23T10:21:46.328 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 23 10:21:46 smithi204 ceph-mon[32649]: from='osd.6 [v2:172.21.15.114:6818/4072912544,v1:172.21.15.114:6819/4072912544]' entity='osd.6' cmd='[{"prefix": "osd crush create-or-move", "id": 6, "weight":0.0873, "args": ["host=smithi114", "root=default"]}]': finished 2024-04-23T10:21:46.328 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 23 10:21:46 smithi204 ceph-mon[32649]: osdmap e46: 7 total, 6 up, 7 in 2024-04-23T10:21:46.328 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 23 10:21:46 smithi204 ceph-mon[32649]: from='mgr.14152 172.21.15.114:0/1529706138' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-04-23T10:21:46.328 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 23 10:21:46 smithi204 ceph-mon[32649]: from='mgr.14152 172.21.15.114:0/1529706138' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-04-23T10:21:46.328 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 23 10:21:46 smithi204 ceph-mon[32649]: from='osd.6 [v2:172.21.15.114:6818/4072912544,v1:172.21.15.114:6819/4072912544]' entity='osd.6' 2024-04-23T10:21:46.328 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 23 10:21:46 smithi204 ceph-mon[32649]: from='mgr.14152 172.21.15.114:0/1529706138' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-04-23T10:21:46.504 INFO:journalctl@ceph.osd.6.smithi114.stdout:Apr 23 10:21:46 smithi114 ceph-aa240572-0159-11ef-bc93-c7b262605968-osd-6[55044]: 2024-04-23T10:21:46.034+0000 7fdba37a1700 -1 osd.6 46 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-04-23T10:21:46.504 INFO:journalctl@ceph.mon.a.smithi114.stdout:Apr 23 10:21:46 smithi114 ceph-mon[29255]: from='osd.6 [v2:172.21.15.114:6818/4072912544,v1:172.21.15.114:6819/4072912544]' entity='osd.6' cmd='[{"prefix": "osd crush create-or-move", "id": 6, "weight":0.0873, "args": ["host=smithi114", "root=default"]}]': finished 2024-04-23T10:21:46.504 INFO:journalctl@ceph.mon.a.smithi114.stdout:Apr 23 10:21:46 smithi114 ceph-mon[29255]: osdmap e46: 7 total, 6 up, 7 in 2024-04-23T10:21:46.504 INFO:journalctl@ceph.mon.a.smithi114.stdout:Apr 23 10:21:46 smithi114 ceph-mon[29255]: from='mgr.14152 172.21.15.114:0/1529706138' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-04-23T10:21:46.505 INFO:journalctl@ceph.mon.a.smithi114.stdout:Apr 23 10:21:46 smithi114 ceph-mon[29255]: from='mgr.14152 172.21.15.114:0/1529706138' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-04-23T10:21:46.505 INFO:journalctl@ceph.mon.a.smithi114.stdout:Apr 23 10:21:46 smithi114 ceph-mon[29255]: from='osd.6 [v2:172.21.15.114:6818/4072912544,v1:172.21.15.114:6819/4072912544]' entity='osd.6' 2024-04-23T10:21:46.505 INFO:journalctl@ceph.mon.a.smithi114.stdout:Apr 23 10:21:46 smithi114 ceph-mon[29255]: from='mgr.14152 172.21.15.114:0/1529706138' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-04-23T10:21:46.549 DEBUG:teuthology.orchestra.run.smithi119:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d69f13d827952bb07bb9b7973bca14f97d764d6f shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid aa240572-0159-11ef-bc93-c7b262605968 -- ceph orch daemon add osd smithi119:vg_nvme/lv_2 2024-04-23T10:21:46.793 INFO:teuthology.orchestra.run.smithi119.stderr:Inferring config /var/lib/ceph/aa240572-0159-11ef-bc93-c7b262605968/mon.b/config 2024-04-23T10:21:47.178 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 23 10:21:47 smithi119 ceph-mon[32608]: purged_snaps scrub starts 2024-04-23T10:21:47.178 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 23 10:21:47 smithi119 ceph-mon[32608]: purged_snaps scrub ok 2024-04-23T10:21:47.178 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 23 10:21:47 smithi119 ceph-mon[32608]: pgmap v156: 1 pgs: 1 active+clean; 577 KiB data, 175 MiB used, 536 GiB / 536 GiB avail 2024-04-23T10:21:47.178 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 23 10:21:47 smithi119 ceph-mon[32608]: from='mgr.14152 172.21.15.114:0/1529706138' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-04-23T10:21:47.219 INFO:journalctl@ceph.mon.a.smithi114.stdout:Apr 23 10:21:47 smithi114 ceph-mon[29255]: purged_snaps scrub starts 2024-04-23T10:21:47.219 INFO:journalctl@ceph.mon.a.smithi114.stdout:Apr 23 10:21:47 smithi114 ceph-mon[29255]: purged_snaps scrub ok 2024-04-23T10:21:47.219 INFO:journalctl@ceph.mon.a.smithi114.stdout:Apr 23 10:21:47 smithi114 ceph-mon[29255]: pgmap v156: 1 pgs: 1 active+clean; 577 KiB data, 175 MiB used, 536 GiB / 536 GiB avail 2024-04-23T10:21:47.219 INFO:journalctl@ceph.mon.a.smithi114.stdout:Apr 23 10:21:47 smithi114 ceph-mon[29255]: from='mgr.14152 172.21.15.114:0/1529706138' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-04-23T10:21:47.328 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 23 10:21:47 smithi204 ceph-mon[32649]: purged_snaps scrub starts 2024-04-23T10:21:47.328 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 23 10:21:47 smithi204 ceph-mon[32649]: purged_snaps scrub ok 2024-04-23T10:21:47.328 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 23 10:21:47 smithi204 ceph-mon[32649]: pgmap v156: 1 pgs: 1 active+clean; 577 KiB data, 175 MiB used, 536 GiB / 536 GiB avail 2024-04-23T10:21:47.328 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 23 10:21:47 smithi204 ceph-mon[32649]: from='mgr.14152 172.21.15.114:0/1529706138' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-04-23T10:21:48.238 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 23 10:21:48 smithi119 ceph-mon[32608]: osd.6 [v2:172.21.15.114:6818/4072912544,v1:172.21.15.114:6819/4072912544] boot 2024-04-23T10:21:48.238 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 23 10:21:48 smithi119 ceph-mon[32608]: osdmap e47: 7 total, 7 up, 7 in 2024-04-23T10:21:48.238 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 23 10:21:48 smithi119 ceph-mon[32608]: from='mgr.14152 172.21.15.114:0/1529706138' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-04-23T10:21:48.238 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 23 10:21:48 smithi119 ceph-mon[32608]: from='mgr.14152 172.21.15.114:0/1529706138' entity='mgr.x' 2024-04-23T10:21:48.238 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 23 10:21:48 smithi119 ceph-mon[32608]: from='mgr.14152 172.21.15.114:0/1529706138' entity='mgr.x' 2024-04-23T10:21:48.238 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 23 10:21:48 smithi119 ceph-mon[32608]: from='mgr.14152 172.21.15.114:0/1529706138' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-04-23T10:21:48.238 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 23 10:21:48 smithi119 ceph-mon[32608]: from='mgr.14152 172.21.15.114:0/1529706138' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-04-23T10:21:48.238 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 23 10:21:48 smithi119 ceph-mon[32608]: from='mgr.14152 172.21.15.114:0/1529706138' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-04-23T10:21:48.238 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 23 10:21:48 smithi119 ceph-mon[32608]: from='mgr.14152 172.21.15.114:0/1529706138' entity='mgr.x' 2024-04-23T10:21:48.239 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 23 10:21:48 smithi119 ceph-mon[32608]: from='mgr.14152 172.21.15.114:0/1529706138' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-23T10:21:48.239 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 23 10:21:48 smithi119 ceph-mon[32608]: from='mgr.14152 172.21.15.114:0/1529706138' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-23T10:21:48.239 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 23 10:21:48 smithi119 ceph-mon[32608]: from='mgr.14152 172.21.15.114:0/1529706138' entity='mgr.x' 2024-04-23T10:21:48.239 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 23 10:21:48 smithi119 ceph-mon[32608]: from='mgr.14152 172.21.15.114:0/1529706138' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-23T10:21:48.239 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 23 10:21:48 smithi119 ceph-mon[32608]: from='mgr.14152 172.21.15.114:0/1529706138' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-23T10:21:48.239 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 23 10:21:48 smithi119 ceph-mon[32608]: from='mgr.14152 172.21.15.114:0/1529706138' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-23T10:21:48.239 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 23 10:21:48 smithi119 ceph-mon[32608]: from='mgr.14152 172.21.15.114:0/1529706138' entity='mgr.x' 2024-04-23T10:21:48.254 INFO:journalctl@ceph.mon.a.smithi114.stdout:Apr 23 10:21:48 smithi114 ceph-mon[29255]: osd.6 [v2:172.21.15.114:6818/4072912544,v1:172.21.15.114:6819/4072912544] boot 2024-04-23T10:21:48.254 INFO:journalctl@ceph.mon.a.smithi114.stdout:Apr 23 10:21:48 smithi114 ceph-mon[29255]: osdmap e47: 7 total, 7 up, 7 in 2024-04-23T10:21:48.254 INFO:journalctl@ceph.mon.a.smithi114.stdout:Apr 23 10:21:48 smithi114 ceph-mon[29255]: from='mgr.14152 172.21.15.114:0/1529706138' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-04-23T10:21:48.254 INFO:journalctl@ceph.mon.a.smithi114.stdout:Apr 23 10:21:48 smithi114 ceph-mon[29255]: from='mgr.14152 172.21.15.114:0/1529706138' entity='mgr.x' 2024-04-23T10:21:48.255 INFO:journalctl@ceph.mon.a.smithi114.stdout:Apr 23 10:21:48 smithi114 ceph-mon[29255]: from='mgr.14152 172.21.15.114:0/1529706138' entity='mgr.x' 2024-04-23T10:21:48.255 INFO:journalctl@ceph.mon.a.smithi114.stdout:Apr 23 10:21:48 smithi114 ceph-mon[29255]: from='mgr.14152 172.21.15.114:0/1529706138' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-04-23T10:21:48.255 INFO:journalctl@ceph.mon.a.smithi114.stdout:Apr 23 10:21:48 smithi114 ceph-mon[29255]: from='mgr.14152 172.21.15.114:0/1529706138' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-04-23T10:21:48.255 INFO:journalctl@ceph.mon.a.smithi114.stdout:Apr 23 10:21:48 smithi114 ceph-mon[29255]: from='mgr.14152 172.21.15.114:0/1529706138' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-04-23T10:21:48.255 INFO:journalctl@ceph.mon.a.smithi114.stdout:Apr 23 10:21:48 smithi114 ceph-mon[29255]: from='mgr.14152 172.21.15.114:0/1529706138' entity='mgr.x' 2024-04-23T10:21:48.255 INFO:journalctl@ceph.mon.a.smithi114.stdout:Apr 23 10:21:48 smithi114 ceph-mon[29255]: from='mgr.14152 172.21.15.114:0/1529706138' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-23T10:21:48.255 INFO:journalctl@ceph.mon.a.smithi114.stdout:Apr 23 10:21:48 smithi114 ceph-mon[29255]: from='mgr.14152 172.21.15.114:0/1529706138' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-23T10:21:48.255 INFO:journalctl@ceph.mon.a.smithi114.stdout:Apr 23 10:21:48 smithi114 ceph-mon[29255]: from='mgr.14152 172.21.15.114:0/1529706138' entity='mgr.x' 2024-04-23T10:21:48.255 INFO:journalctl@ceph.mon.a.smithi114.stdout:Apr 23 10:21:48 smithi114 ceph-mon[29255]: from='mgr.14152 172.21.15.114:0/1529706138' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-23T10:21:48.255 INFO:journalctl@ceph.mon.a.smithi114.stdout:Apr 23 10:21:48 smithi114 ceph-mon[29255]: from='mgr.14152 172.21.15.114:0/1529706138' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-23T10:21:48.255 INFO:journalctl@ceph.mon.a.smithi114.stdout:Apr 23 10:21:48 smithi114 ceph-mon[29255]: from='mgr.14152 172.21.15.114:0/1529706138' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-23T10:21:48.255 INFO:journalctl@ceph.mon.a.smithi114.stdout:Apr 23 10:21:48 smithi114 ceph-mon[29255]: from='mgr.14152 172.21.15.114:0/1529706138' entity='mgr.x' 2024-04-23T10:21:48.328 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 23 10:21:48 smithi204 ceph-mon[32649]: osd.6 [v2:172.21.15.114:6818/4072912544,v1:172.21.15.114:6819/4072912544] boot 2024-04-23T10:21:48.328 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 23 10:21:48 smithi204 ceph-mon[32649]: osdmap e47: 7 total, 7 up, 7 in 2024-04-23T10:21:48.328 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 23 10:21:48 smithi204 ceph-mon[32649]: from='mgr.14152 172.21.15.114:0/1529706138' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-04-23T10:21:48.328 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 23 10:21:48 smithi204 ceph-mon[32649]: from='mgr.14152 172.21.15.114:0/1529706138' entity='mgr.x' 2024-04-23T10:21:48.328 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 23 10:21:48 smithi204 ceph-mon[32649]: from='mgr.14152 172.21.15.114:0/1529706138' entity='mgr.x' 2024-04-23T10:21:48.328 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 23 10:21:48 smithi204 ceph-mon[32649]: from='mgr.14152 172.21.15.114:0/1529706138' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-04-23T10:21:48.328 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 23 10:21:48 smithi204 ceph-mon[32649]: from='mgr.14152 172.21.15.114:0/1529706138' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-04-23T10:21:48.328 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 23 10:21:48 smithi204 ceph-mon[32649]: from='mgr.14152 172.21.15.114:0/1529706138' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-04-23T10:21:48.329 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 23 10:21:48 smithi204 ceph-mon[32649]: from='mgr.14152 172.21.15.114:0/1529706138' entity='mgr.x' 2024-04-23T10:21:48.329 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 23 10:21:48 smithi204 ceph-mon[32649]: from='mgr.14152 172.21.15.114:0/1529706138' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-23T10:21:48.329 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 23 10:21:48 smithi204 ceph-mon[32649]: from='mgr.14152 172.21.15.114:0/1529706138' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-23T10:21:48.329 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 23 10:21:48 smithi204 ceph-mon[32649]: from='mgr.14152 172.21.15.114:0/1529706138' entity='mgr.x' 2024-04-23T10:21:48.329 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 23 10:21:48 smithi204 ceph-mon[32649]: from='mgr.14152 172.21.15.114:0/1529706138' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-23T10:21:48.329 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 23 10:21:48 smithi204 ceph-mon[32649]: from='mgr.14152 172.21.15.114:0/1529706138' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-23T10:21:48.329 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 23 10:21:48 smithi204 ceph-mon[32649]: from='mgr.14152 172.21.15.114:0/1529706138' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-23T10:21:48.329 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 23 10:21:48 smithi204 ceph-mon[32649]: from='mgr.14152 172.21.15.114:0/1529706138' entity='mgr.x' 2024-04-23T10:21:48.459 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-23T10:21:48.460+0000 7f376e229700 1 -- 172.21.15.119:0/2270024352 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f3768102ab0 msgr2=0x7f3768102ed0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T10:21:48.460 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-23T10:21:48.460+0000 7f376e229700 1 --2- 172.21.15.119:0/2270024352 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f3768102ab0 0x7f3768102ed0 secure :-1 s=READY pgs=26 cs=0 l=1 rev1=1 crypto rx=0x7f37580099c0 tx=0x7f3758009cd0 comp rx=0 tx=0).stop 2024-04-23T10:21:48.460 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-23T10:21:48.460+0000 7f376e229700 1 -- 172.21.15.119:0/2270024352 shutdown_connections 2024-04-23T10:21:48.460 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-23T10:21:48.460+0000 7f376e229700 1 --2- 172.21.15.119:0/2270024352 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f3768102ab0 0x7f3768102ed0 unknown :-1 s=CLOSED pgs=26 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:21:48.460 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-23T10:21:48.460+0000 7f376e229700 1 --2- 172.21.15.119:0/2270024352 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f3768104650 0x7f376810af00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:21:48.460 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-23T10:21:48.460+0000 7f376e229700 1 --2- 172.21.15.119:0/2270024352 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f3768103cb0 0x7f3768104110 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:21:48.460 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-23T10:21:48.460+0000 7f376e229700 1 -- 172.21.15.119:0/2270024352 >> 172.21.15.119:0/2270024352 conn(0x7f37680fe050 msgr2=0x7f3768100490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T10:21:48.460 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-23T10:21:48.460+0000 7f376e229700 1 -- 172.21.15.119:0/2270024352 shutdown_connections 2024-04-23T10:21:48.460 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-23T10:21:48.460+0000 7f376e229700 1 -- 172.21.15.119:0/2270024352 wait complete. 2024-04-23T10:21:48.462 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-23T10:21:48.461+0000 7f376e229700 1 Processor -- start 2024-04-23T10:21:48.462 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-23T10:21:48.461+0000 7f376e229700 1 -- start start 2024-04-23T10:21:48.462 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-23T10:21:48.461+0000 7f376e229700 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f3768102ab0 0x7f37680767e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T10:21:48.462 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-23T10:21:48.461+0000 7f376e229700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f3768103cb0 0x7f3768074e30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T10:21:48.462 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-23T10:21:48.461+0000 7f376e229700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f3768104650 0x7f3768075370 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T10:21:48.462 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-23T10:21:48.461+0000 7f376e229700 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x7f376810e540 con 0x7f3768103cb0 2024-04-23T10:21:48.462 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-23T10:21:48.461+0000 7f376e229700 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 -- 0x7f376810e3c0 con 0x7f3768104650 2024-04-23T10:21:48.462 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-23T10:21:48.461+0000 7f376e229700 1 -- --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_getmap magic: 0 -- 0x7f376810e6c0 con 0x7f3768102ab0 2024-04-23T10:21:48.462 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-23T10:21:48.461+0000 7f3767fff700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f3768104650 0x7f3768075370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T10:21:48.462 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-23T10:21:48.461+0000 7f3767fff700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f3768104650 0x7f3768075370 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.119:3300/0 says I am v2:172.21.15.119:48778/0 (socket says 172.21.15.119:48778) 2024-04-23T10:21:48.462 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-23T10:21:48.461+0000 7f3767fff700 1 -- 172.21.15.119:0/655104942 learned_addr learned my addr 172.21.15.119:0/655104942 (peer_addr_for_me v2:172.21.15.119:0/0) 2024-04-23T10:21:48.463 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-23T10:21:48.461+0000 7f375effd700 1 --2- 172.21.15.119:0/655104942 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f3768103cb0 0x7f3768074e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T10:21:48.463 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-23T10:21:48.461+0000 7f37677fe700 1 --2- 172.21.15.119:0/655104942 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f3768102ab0 0x7f37680767e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T10:21:48.463 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-23T10:21:48.461+0000 7f3767fff700 1 -- 172.21.15.119:0/655104942 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f3768102ab0 msgr2=0x7f37680767e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T10:21:48.463 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-23T10:21:48.461+0000 7f3767fff700 1 --2- 172.21.15.119:0/655104942 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f3768102ab0 0x7f37680767e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:21:48.463 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-23T10:21:48.461+0000 7f3767fff700 1 -- 172.21.15.119:0/655104942 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f3768103cb0 msgr2=0x7f3768074e30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T10:21:48.463 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-23T10:21:48.461+0000 7f3767fff700 1 --2- 172.21.15.119:0/655104942 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f3768103cb0 0x7f3768074e30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:21:48.463 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-23T10:21:48.461+0000 7f3767fff700 1 -- 172.21.15.119:0/655104942 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f3768075c20 con 0x7f3768104650 2024-04-23T10:21:48.463 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-23T10:21:48.462+0000 7f3767fff700 1 --2- 172.21.15.119:0/655104942 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f3768104650 0x7f3768075370 secure :-1 s=READY pgs=27 cs=0 l=1 rev1=1 crypto rx=0x7f3760007ee0 tx=0x7f376000d3b0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T10:21:48.463 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-23T10:21:48.462+0000 7f37657fa700 1 -- 172.21.15.119:0/655104942 <== mon.2 v2:172.21.15.119:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3760004bb0 con 0x7f3768104650 2024-04-23T10:21:48.463 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-23T10:21:48.462+0000 7f376e229700 1 -- 172.21.15.119:0/655104942 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f37681a2d50 con 0x7f3768104650 2024-04-23T10:21:48.463 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-23T10:21:48.462+0000 7f37657fa700 1 -- 172.21.15.119:0/655104942 <== mon.2 v2:172.21.15.119:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f3760004d50 con 0x7f3768104650 2024-04-23T10:21:48.463 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-23T10:21:48.462+0000 7f37657fa700 1 -- 172.21.15.119:0/655104942 <== mon.2 v2:172.21.15.119:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3760005920 con 0x7f3768104650 2024-04-23T10:21:48.463 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-23T10:21:48.462+0000 7f376e229700 1 -- 172.21.15.119:0/655104942 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f37681a3200 con 0x7f3768104650 2024-04-23T10:21:48.464 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-23T10:21:48.463+0000 7f376e229700 1 -- 172.21.15.119:0/655104942 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f3768069730 con 0x7f3768104650 2024-04-23T10:21:48.464 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-23T10:21:48.464+0000 7f37657fa700 1 -- 172.21.15.119:0/655104942 <== mon.2 v2:172.21.15.119:3300/0 4 ==== mgrmap(e 16) ==== 138647+0+0 (secure 0 0 0) 0x7f376001a020 con 0x7f3768104650 2024-04-23T10:21:48.464 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-23T10:21:48.464+0000 7f37657fa700 1 --2- 172.21.15.119:0/655104942 >> [v2:172.21.15.114:6800/465333808,v1:172.21.15.114:6801/465333808] conn(0x7f37540a3c40 0x7f37540a6100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T10:21:48.465 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-23T10:21:48.465+0000 7f37657fa700 1 -- 172.21.15.119:0/655104942 <== mon.2 v2:172.21.15.119:3300/0 5 ==== osd_map(48..48 src has 1..48) ==== 4965+0+0 (secure 0 0 0) 0x7f37600cf320 con 0x7f3768104650 2024-04-23T10:21:48.465 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-23T10:21:48.465+0000 7f37677fe700 1 --2- 172.21.15.119:0/655104942 >> [v2:172.21.15.114:6800/465333808,v1:172.21.15.114:6801/465333808] conn(0x7f37540a3c40 0x7f37540a6100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T10:21:48.465 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-23T10:21:48.465+0000 7f37677fe700 1 --2- 172.21.15.119:0/655104942 >> [v2:172.21.15.114:6800/465333808,v1:172.21.15.114:6801/465333808] conn(0x7f37540a3c40 0x7f37540a6100 secure :-1 s=READY pgs=93 cs=0 l=1 rev1=1 crypto rx=0x7f37580038c0 tx=0x7f375800b540 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T10:21:48.468 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-23T10:21:48.468+0000 7f37657fa700 1 -- 172.21.15.119:0/655104942 <== mon.2 v2:172.21.15.119:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f3760010070 con 0x7f3768104650 2024-04-23T10:21:48.674 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-23T10:21:48.674+0000 7f376e229700 1 -- 172.21.15.119:0/655104942 --> [v2:172.21.15.114:6800/465333808,v1:172.21.15.114:6801/465333808] -- mgr_command(tid 0: {"prefix": "orch daemon add osd", "svc_arg": "smithi119:vg_nvme/lv_2", "target": ["mon-mgr", ""]}) -- 0x7f3768062e80 con 0x7f37540a3c40 2024-04-23T10:21:49.284 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 23 10:21:49 smithi119 ceph-mon[32608]: Detected new or changed devices on smithi114 2024-04-23T10:21:49.284 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 23 10:21:49 smithi119 ceph-mon[32608]: Adjusting osd_memory_target on smithi114 to 5696M 2024-04-23T10:21:49.284 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 23 10:21:49 smithi119 ceph-mon[32608]: pgmap v158: 1 pgs: 1 active+clean; 577 KiB data, 175 MiB used, 536 GiB / 536 GiB avail 2024-04-23T10:21:49.284 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 23 10:21:49 smithi119 ceph-mon[32608]: osdmap e48: 7 total, 7 up, 7 in 2024-04-23T10:21:49.284 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 23 10:21:49 smithi119 ceph-mon[32608]: from='mgr.14152 172.21.15.114:0/1529706138' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-04-23T10:21:49.284 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 23 10:21:49 smithi119 ceph-mon[32608]: from='mgr.14152 172.21.15.114:0/1529706138' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-04-23T10:21:49.284 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 23 10:21:49 smithi119 ceph-mon[32608]: from='mgr.14152 172.21.15.114:0/1529706138' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-23T10:21:49.328 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 23 10:21:49 smithi204 ceph-mon[32649]: Detected new or changed devices on smithi114 2024-04-23T10:21:49.328 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 23 10:21:49 smithi204 ceph-mon[32649]: Adjusting osd_memory_target on smithi114 to 5696M 2024-04-23T10:21:49.328 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 23 10:21:49 smithi204 ceph-mon[32649]: pgmap v158: 1 pgs: 1 active+clean; 577 KiB data, 175 MiB used, 536 GiB / 536 GiB avail 2024-04-23T10:21:49.328 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 23 10:21:49 smithi204 ceph-mon[32649]: osdmap e48: 7 total, 7 up, 7 in 2024-04-23T10:21:49.328 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 23 10:21:49 smithi204 ceph-mon[32649]: from='mgr.14152 172.21.15.114:0/1529706138' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-04-23T10:21:49.328 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 23 10:21:49 smithi204 ceph-mon[32649]: from='mgr.14152 172.21.15.114:0/1529706138' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-04-23T10:21:49.328 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 23 10:21:49 smithi204 ceph-mon[32649]: from='mgr.14152 172.21.15.114:0/1529706138' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-23T10:21:49.504 INFO:journalctl@ceph.mon.a.smithi114.stdout:Apr 23 10:21:49 smithi114 ceph-mon[29255]: Detected new or changed devices on smithi114 2024-04-23T10:21:49.504 INFO:journalctl@ceph.mon.a.smithi114.stdout:Apr 23 10:21:49 smithi114 ceph-mon[29255]: Adjusting osd_memory_target on smithi114 to 5696M 2024-04-23T10:21:49.504 INFO:journalctl@ceph.mon.a.smithi114.stdout:Apr 23 10:21:49 smithi114 ceph-mon[29255]: pgmap v158: 1 pgs: 1 active+clean; 577 KiB data, 175 MiB used, 536 GiB / 536 GiB avail 2024-04-23T10:21:49.504 INFO:journalctl@ceph.mon.a.smithi114.stdout:Apr 23 10:21:49 smithi114 ceph-mon[29255]: osdmap e48: 7 total, 7 up, 7 in 2024-04-23T10:21:49.505 INFO:journalctl@ceph.mon.a.smithi114.stdout:Apr 23 10:21:49 smithi114 ceph-mon[29255]: from='mgr.14152 172.21.15.114:0/1529706138' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-04-23T10:21:49.505 INFO:journalctl@ceph.mon.a.smithi114.stdout:Apr 23 10:21:49 smithi114 ceph-mon[29255]: from='mgr.14152 172.21.15.114:0/1529706138' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-04-23T10:21:49.505 INFO:journalctl@ceph.mon.a.smithi114.stdout:Apr 23 10:21:49 smithi114 ceph-mon[29255]: from='mgr.14152 172.21.15.114:0/1529706138' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-23T10:21:50.315 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 23 10:21:50 smithi119 ceph-mon[32608]: from='client.24296 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi119:vg_nvme/lv_2", "target": ["mon-mgr", ""]}]: dispatch 2024-04-23T10:21:50.328 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 23 10:21:50 smithi204 ceph-mon[32649]: from='client.24296 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi119:vg_nvme/lv_2", "target": ["mon-mgr", ""]}]: dispatch 2024-04-23T10:21:50.505 INFO:journalctl@ceph.mon.a.smithi114.stdout:Apr 23 10:21:50 smithi114 ceph-mon[29255]: from='client.24296 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi119:vg_nvme/lv_2", "target": ["mon-mgr", ""]}]: dispatch 2024-04-23T10:21:51.328 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 23 10:21:51 smithi204 ceph-mon[32649]: pgmap v160: 1 pgs: 1 active+clean; 577 KiB data, 206 MiB used, 626 GiB / 626 GiB avail 2024-04-23T10:21:51.492 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 23 10:21:51 smithi119 ceph-mon[32608]: pgmap v160: 1 pgs: 1 active+clean; 577 KiB data, 206 MiB used, 626 GiB / 626 GiB avail 2024-04-23T10:21:51.504 INFO:journalctl@ceph.mon.a.smithi114.stdout:Apr 23 10:21:51 smithi114 ceph-mon[29255]: pgmap v160: 1 pgs: 1 active+clean; 577 KiB data, 206 MiB used, 626 GiB / 626 GiB avail 2024-04-23T10:21:52.328 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 23 10:21:52 smithi204 ceph-mon[32649]: from='client.? 172.21.15.119:0/1515059230' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "f0ec22b8-1b48-4859-9d65-b00403aff3ad"}]: dispatch 2024-04-23T10:21:52.328 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 23 10:21:52 smithi204 ceph-mon[32649]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "f0ec22b8-1b48-4859-9d65-b00403aff3ad"}]: dispatch 2024-04-23T10:21:52.328 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 23 10:21:52 smithi204 ceph-mon[32649]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "f0ec22b8-1b48-4859-9d65-b00403aff3ad"}]': finished 2024-04-23T10:21:52.328 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 23 10:21:52 smithi204 ceph-mon[32649]: osdmap e49: 8 total, 7 up, 8 in 2024-04-23T10:21:52.328 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 23 10:21:52 smithi204 ceph-mon[32649]: from='mgr.14152 172.21.15.114:0/1529706138' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-04-23T10:21:52.329 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 23 10:21:52 smithi204 ceph-mon[32649]: from='client.? 172.21.15.119:0/1252765963' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-04-23T10:21:52.492 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 23 10:21:52 smithi119 ceph-mon[32608]: from='client.? 172.21.15.119:0/1515059230' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "f0ec22b8-1b48-4859-9d65-b00403aff3ad"}]: dispatch 2024-04-23T10:21:52.492 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 23 10:21:52 smithi119 ceph-mon[32608]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "f0ec22b8-1b48-4859-9d65-b00403aff3ad"}]: dispatch 2024-04-23T10:21:52.492 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 23 10:21:52 smithi119 ceph-mon[32608]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "f0ec22b8-1b48-4859-9d65-b00403aff3ad"}]': finished 2024-04-23T10:21:52.492 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 23 10:21:52 smithi119 ceph-mon[32608]: osdmap e49: 8 total, 7 up, 8 in 2024-04-23T10:21:52.492 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 23 10:21:52 smithi119 ceph-mon[32608]: from='mgr.14152 172.21.15.114:0/1529706138' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-04-23T10:21:52.492 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 23 10:21:52 smithi119 ceph-mon[32608]: from='client.? 172.21.15.119:0/1252765963' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-04-23T10:21:52.504 INFO:journalctl@ceph.mon.a.smithi114.stdout:Apr 23 10:21:52 smithi114 ceph-mon[29255]: from='client.? 172.21.15.119:0/1515059230' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "f0ec22b8-1b48-4859-9d65-b00403aff3ad"}]: dispatch 2024-04-23T10:21:52.504 INFO:journalctl@ceph.mon.a.smithi114.stdout:Apr 23 10:21:52 smithi114 ceph-mon[29255]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "f0ec22b8-1b48-4859-9d65-b00403aff3ad"}]: dispatch 2024-04-23T10:21:52.504 INFO:journalctl@ceph.mon.a.smithi114.stdout:Apr 23 10:21:52 smithi114 ceph-mon[29255]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "f0ec22b8-1b48-4859-9d65-b00403aff3ad"}]': finished 2024-04-23T10:21:52.504 INFO:journalctl@ceph.mon.a.smithi114.stdout:Apr 23 10:21:52 smithi114 ceph-mon[29255]: osdmap e49: 8 total, 7 up, 8 in 2024-04-23T10:21:52.504 INFO:journalctl@ceph.mon.a.smithi114.stdout:Apr 23 10:21:52 smithi114 ceph-mon[29255]: from='mgr.14152 172.21.15.114:0/1529706138' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-04-23T10:21:52.505 INFO:journalctl@ceph.mon.a.smithi114.stdout:Apr 23 10:21:52 smithi114 ceph-mon[29255]: from='client.? 172.21.15.119:0/1252765963' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-04-23T10:21:53.328 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 23 10:21:53 smithi204 ceph-mon[32649]: pgmap v162: 1 pgs: 1 active+clean; 577 KiB data, 206 MiB used, 626 GiB / 626 GiB avail 2024-04-23T10:21:53.492 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 23 10:21:53 smithi119 ceph-mon[32608]: pgmap v162: 1 pgs: 1 active+clean; 577 KiB data, 206 MiB used, 626 GiB / 626 GiB avail 2024-04-23T10:21:53.504 INFO:journalctl@ceph.mon.a.smithi114.stdout:Apr 23 10:21:53 smithi114 ceph-mon[29255]: pgmap v162: 1 pgs: 1 active+clean; 577 KiB data, 206 MiB used, 626 GiB / 626 GiB avail 2024-04-23T10:21:55.294 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 23 10:21:55 smithi119 ceph-mon[32608]: pgmap v163: 1 pgs: 1 active+clean; 577 KiB data, 206 MiB used, 626 GiB / 626 GiB avail 2024-04-23T10:21:55.328 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 23 10:21:55 smithi204 ceph-mon[32649]: pgmap v163: 1 pgs: 1 active+clean; 577 KiB data, 206 MiB used, 626 GiB / 626 GiB avail 2024-04-23T10:21:55.504 INFO:journalctl@ceph.mon.a.smithi114.stdout:Apr 23 10:21:55 smithi114 ceph-mon[29255]: pgmap v163: 1 pgs: 1 active+clean; 577 KiB data, 206 MiB used, 626 GiB / 626 GiB avail 2024-04-23T10:21:57.242 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 23 10:21:57 smithi119 ceph-mon[32608]: pgmap v164: 1 pgs: 1 active+clean; 577 KiB data, 206 MiB used, 626 GiB / 626 GiB avail 2024-04-23T10:21:57.328 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 23 10:21:57 smithi204 ceph-mon[32649]: pgmap v164: 1 pgs: 1 active+clean; 577 KiB data, 206 MiB used, 626 GiB / 626 GiB avail 2024-04-23T10:21:57.504 INFO:journalctl@ceph.mon.a.smithi114.stdout:Apr 23 10:21:57 smithi114 ceph-mon[29255]: pgmap v164: 1 pgs: 1 active+clean; 577 KiB data, 206 MiB used, 626 GiB / 626 GiB avail 2024-04-23T10:21:59.242 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 23 10:21:59 smithi119 ceph-mon[32608]: pgmap v165: 1 pgs: 1 active+clean; 577 KiB data, 206 MiB used, 626 GiB / 626 GiB avail 2024-04-23T10:21:59.242 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 23 10:21:59 smithi119 ceph-mon[32608]: from='mgr.14152 172.21.15.114:0/1529706138' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.7"}]: dispatch 2024-04-23T10:21:59.242 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 23 10:21:59 smithi119 ceph-mon[32608]: from='mgr.14152 172.21.15.114:0/1529706138' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-23T10:21:59.328 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 23 10:21:59 smithi204 ceph-mon[32649]: pgmap v165: 1 pgs: 1 active+clean; 577 KiB data, 206 MiB used, 626 GiB / 626 GiB avail 2024-04-23T10:21:59.328 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 23 10:21:59 smithi204 ceph-mon[32649]: from='mgr.14152 172.21.15.114:0/1529706138' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.7"}]: dispatch 2024-04-23T10:21:59.328 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 23 10:21:59 smithi204 ceph-mon[32649]: from='mgr.14152 172.21.15.114:0/1529706138' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-23T10:21:59.504 INFO:journalctl@ceph.mon.a.smithi114.stdout:Apr 23 10:21:59 smithi114 ceph-mon[29255]: pgmap v165: 1 pgs: 1 active+clean; 577 KiB data, 206 MiB used, 626 GiB / 626 GiB avail 2024-04-23T10:21:59.504 INFO:journalctl@ceph.mon.a.smithi114.stdout:Apr 23 10:21:59 smithi114 ceph-mon[29255]: from='mgr.14152 172.21.15.114:0/1529706138' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.7"}]: dispatch 2024-04-23T10:21:59.504 INFO:journalctl@ceph.mon.a.smithi114.stdout:Apr 23 10:21:59 smithi114 ceph-mon[29255]: from='mgr.14152 172.21.15.114:0/1529706138' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-23T10:22:00.328 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 23 10:22:00 smithi204 ceph-mon[32649]: Deploying daemon osd.7 on smithi119 2024-04-23T10:22:00.369 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 23 10:22:00 smithi119 ceph-mon[32608]: Deploying daemon osd.7 on smithi119 2024-04-23T10:22:00.504 INFO:journalctl@ceph.mon.a.smithi114.stdout:Apr 23 10:22:00 smithi114 ceph-mon[29255]: Deploying daemon osd.7 on smithi119 2024-04-23T10:22:01.328 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 23 10:22:01 smithi204 ceph-mon[32649]: pgmap v166: 1 pgs: 1 active+clean; 577 KiB data, 206 MiB used, 626 GiB / 626 GiB avail 2024-04-23T10:22:01.344 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 23 10:22:01 smithi119 ceph-mon[32608]: pgmap v166: 1 pgs: 1 active+clean; 577 KiB data, 206 MiB used, 626 GiB / 626 GiB avail 2024-04-23T10:22:01.504 INFO:journalctl@ceph.mon.a.smithi114.stdout:Apr 23 10:22:01 smithi114 ceph-mon[29255]: pgmap v166: 1 pgs: 1 active+clean; 577 KiB data, 206 MiB used, 626 GiB / 626 GiB avail 2024-04-23T10:22:03.328 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 23 10:22:03 smithi204 ceph-mon[32649]: pgmap v167: 1 pgs: 1 active+clean; 577 KiB data, 206 MiB used, 626 GiB / 626 GiB avail 2024-04-23T10:22:03.395 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 23 10:22:03 smithi119 ceph-mon[32608]: pgmap v167: 1 pgs: 1 active+clean; 577 KiB data, 206 MiB used, 626 GiB / 626 GiB avail 2024-04-23T10:22:03.504 INFO:journalctl@ceph.mon.a.smithi114.stdout:Apr 23 10:22:03 smithi114 ceph-mon[29255]: pgmap v167: 1 pgs: 1 active+clean; 577 KiB data, 206 MiB used, 626 GiB / 626 GiB avail 2024-04-23T10:22:05.235 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 23 10:22:05 smithi119 ceph-mon[32608]: pgmap v168: 1 pgs: 1 active+clean; 577 KiB data, 206 MiB used, 626 GiB / 626 GiB avail 2024-04-23T10:22:05.236 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 23 10:22:05 smithi119 ceph-mon[32608]: from='mgr.14152 172.21.15.114:0/1529706138' entity='mgr.x' 2024-04-23T10:22:05.236 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 23 10:22:05 smithi119 ceph-mon[32608]: from='mgr.14152 172.21.15.114:0/1529706138' entity='mgr.x' 2024-04-23T10:22:05.236 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 23 10:22:05 smithi119 ceph-mon[32608]: from='mgr.14152 172.21.15.114:0/1529706138' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-23T10:22:05.328 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 23 10:22:05 smithi204 ceph-mon[32649]: pgmap v168: 1 pgs: 1 active+clean; 577 KiB data, 206 MiB used, 626 GiB / 626 GiB avail 2024-04-23T10:22:05.328 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 23 10:22:05 smithi204 ceph-mon[32649]: from='mgr.14152 172.21.15.114:0/1529706138' entity='mgr.x' 2024-04-23T10:22:05.328 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 23 10:22:05 smithi204 ceph-mon[32649]: from='mgr.14152 172.21.15.114:0/1529706138' entity='mgr.x' 2024-04-23T10:22:05.328 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 23 10:22:05 smithi204 ceph-mon[32649]: from='mgr.14152 172.21.15.114:0/1529706138' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-23T10:22:05.504 INFO:journalctl@ceph.mon.a.smithi114.stdout:Apr 23 10:22:05 smithi114 ceph-mon[29255]: pgmap v168: 1 pgs: 1 active+clean; 577 KiB data, 206 MiB used, 626 GiB / 626 GiB avail 2024-04-23T10:22:05.504 INFO:journalctl@ceph.mon.a.smithi114.stdout:Apr 23 10:22:05 smithi114 ceph-mon[29255]: from='mgr.14152 172.21.15.114:0/1529706138' entity='mgr.x' 2024-04-23T10:22:05.504 INFO:journalctl@ceph.mon.a.smithi114.stdout:Apr 23 10:22:05 smithi114 ceph-mon[29255]: from='mgr.14152 172.21.15.114:0/1529706138' entity='mgr.x' 2024-04-23T10:22:05.504 INFO:journalctl@ceph.mon.a.smithi114.stdout:Apr 23 10:22:05 smithi114 ceph-mon[29255]: from='mgr.14152 172.21.15.114:0/1529706138' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-23T10:22:07.243 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 23 10:22:07 smithi119 ceph-mon[32608]: pgmap v169: 1 pgs: 1 active+clean; 577 KiB data, 206 MiB used, 626 GiB / 626 GiB avail 2024-04-23T10:22:07.328 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 23 10:22:07 smithi204 ceph-mon[32649]: pgmap v169: 1 pgs: 1 active+clean; 577 KiB data, 206 MiB used, 626 GiB / 626 GiB avail 2024-04-23T10:22:07.504 INFO:journalctl@ceph.mon.a.smithi114.stdout:Apr 23 10:22:07 smithi114 ceph-mon[29255]: pgmap v169: 1 pgs: 1 active+clean; 577 KiB data, 206 MiB used, 626 GiB / 626 GiB avail 2024-04-23T10:22:08.240 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-23T10:22:08.241+0000 7f37657fa700 1 -- 172.21.15.119:0/655104942 <== mgr.14152 v2:172.21.15.114:6800/465333808 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+37 (secure 0 0 0) 0x7f3768062e80 con 0x7f37540a3c40 2024-04-23T10:22:08.240 INFO:teuthology.orchestra.run.smithi119.stdout:Created osd(s) 7 on host 'smithi119' 2024-04-23T10:22:08.242 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-23T10:22:08.243+0000 7f376e229700 1 -- 172.21.15.119:0/655104942 >> [v2:172.21.15.114:6800/465333808,v1:172.21.15.114:6801/465333808] conn(0x7f37540a3c40 msgr2=0x7f37540a6100 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T10:22:08.242 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-23T10:22:08.243+0000 7f376e229700 1 --2- 172.21.15.119:0/655104942 >> [v2:172.21.15.114:6800/465333808,v1:172.21.15.114:6801/465333808] conn(0x7f37540a3c40 0x7f37540a6100 secure :-1 s=READY pgs=93 cs=0 l=1 rev1=1 crypto rx=0x7f37580038c0 tx=0x7f375800b540 comp rx=0 tx=0).stop 2024-04-23T10:22:08.242 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-23T10:22:08.243+0000 7f376e229700 1 -- 172.21.15.119:0/655104942 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f3768104650 msgr2=0x7f3768075370 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T10:22:08.242 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-23T10:22:08.243+0000 7f376e229700 1 --2- 172.21.15.119:0/655104942 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f3768104650 0x7f3768075370 secure :-1 s=READY pgs=27 cs=0 l=1 rev1=1 crypto rx=0x7f3760007ee0 tx=0x7f376000d3b0 comp rx=0 tx=0).stop 2024-04-23T10:22:08.242 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-23T10:22:08.243+0000 7f376e229700 1 -- 172.21.15.119:0/655104942 shutdown_connections 2024-04-23T10:22:08.243 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-23T10:22:08.243+0000 7f376e229700 1 --2- 172.21.15.119:0/655104942 >> [v2:172.21.15.114:6800/465333808,v1:172.21.15.114:6801/465333808] conn(0x7f37540a3c40 0x7f37540a6100 unknown :-1 s=CLOSED pgs=93 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:22:08.243 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-23T10:22:08.243+0000 7f376e229700 1 --2- 172.21.15.119:0/655104942 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f3768104650 0x7f3768075370 unknown :-1 s=CLOSED pgs=27 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:22:08.243 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-23T10:22:08.243+0000 7f376e229700 1 --2- 172.21.15.119:0/655104942 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f3768102ab0 0x7f37680767e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:22:08.243 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-23T10:22:08.243+0000 7f376e229700 1 --2- 172.21.15.119:0/655104942 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f3768103cb0 0x7f3768074e30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:22:08.243 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-23T10:22:08.243+0000 7f376e229700 1 -- 172.21.15.119:0/655104942 >> 172.21.15.119:0/655104942 conn(0x7f37680fe050 msgr2=0x7f3768106fd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T10:22:08.243 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-23T10:22:08.243+0000 7f376e229700 1 -- 172.21.15.119:0/655104942 shutdown_connections 2024-04-23T10:22:08.243 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-23T10:22:08.243+0000 7f376e229700 1 -- 172.21.15.119:0/655104942 wait complete. 2024-04-23T10:22:08.492 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 23 10:22:08 smithi119 ceph-mon[32608]: from='mgr.14152 172.21.15.114:0/1529706138' entity='mgr.x' 2024-04-23T10:22:08.492 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 23 10:22:08 smithi119 ceph-mon[32608]: from='mgr.14152 172.21.15.114:0/1529706138' entity='mgr.x' 2024-04-23T10:22:08.492 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 23 10:22:08 smithi119 ceph-mon[32608]: from='mgr.14152 172.21.15.114:0/1529706138' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-23T10:22:08.492 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 23 10:22:08 smithi119 ceph-mon[32608]: from='mgr.14152 172.21.15.114:0/1529706138' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-23T10:22:08.492 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 23 10:22:08 smithi119 ceph-mon[32608]: from='mgr.14152 172.21.15.114:0/1529706138' entity='mgr.x' 2024-04-23T10:22:08.493 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 23 10:22:08 smithi119 ceph-mon[32608]: pgmap v170: 1 pgs: 1 active+clean; 577 KiB data, 206 MiB used, 626 GiB / 626 GiB avail 2024-04-23T10:22:08.493 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 23 10:22:08 smithi119 ceph-mon[32608]: from='osd.7 [v2:172.21.15.119:6816/2237458793,v1:172.21.15.119:6817/2237458793]' entity='osd.7' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["7"]}]: dispatch 2024-04-23T10:22:08.493 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 23 10:22:08 smithi119 ceph-mon[32608]: from='osd.7 ' entity='osd.7' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["7"]}]: dispatch 2024-04-23T10:22:08.493 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 23 10:22:08 smithi119 ceph-mon[32608]: from='mgr.14152 172.21.15.114:0/1529706138' entity='mgr.x' 2024-04-23T10:22:08.493 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 23 10:22:08 smithi119 ceph-mon[32608]: from='mgr.14152 172.21.15.114:0/1529706138' entity='mgr.x' 2024-04-23T10:22:08.493 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 23 10:22:08 smithi119 ceph-mon[32608]: from='mgr.14152 172.21.15.114:0/1529706138' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-23T10:22:08.754 INFO:journalctl@ceph.mon.a.smithi114.stdout:Apr 23 10:22:08 smithi114 ceph-mon[29255]: from='mgr.14152 172.21.15.114:0/1529706138' entity='mgr.x' 2024-04-23T10:22:08.754 INFO:journalctl@ceph.mon.a.smithi114.stdout:Apr 23 10:22:08 smithi114 ceph-mon[29255]: from='mgr.14152 172.21.15.114:0/1529706138' entity='mgr.x' 2024-04-23T10:22:08.755 INFO:journalctl@ceph.mon.a.smithi114.stdout:Apr 23 10:22:08 smithi114 ceph-mon[29255]: from='mgr.14152 172.21.15.114:0/1529706138' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-23T10:22:08.755 INFO:journalctl@ceph.mon.a.smithi114.stdout:Apr 23 10:22:08 smithi114 ceph-mon[29255]: from='mgr.14152 172.21.15.114:0/1529706138' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-23T10:22:08.755 INFO:journalctl@ceph.mon.a.smithi114.stdout:Apr 23 10:22:08 smithi114 ceph-mon[29255]: from='mgr.14152 172.21.15.114:0/1529706138' entity='mgr.x' 2024-04-23T10:22:08.755 INFO:journalctl@ceph.mon.a.smithi114.stdout:Apr 23 10:22:08 smithi114 ceph-mon[29255]: pgmap v170: 1 pgs: 1 active+clean; 577 KiB data, 206 MiB used, 626 GiB / 626 GiB avail 2024-04-23T10:22:08.755 INFO:journalctl@ceph.mon.a.smithi114.stdout:Apr 23 10:22:08 smithi114 ceph-mon[29255]: from='osd.7 [v2:172.21.15.119:6816/2237458793,v1:172.21.15.119:6817/2237458793]' entity='osd.7' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["7"]}]: dispatch 2024-04-23T10:22:08.755 INFO:journalctl@ceph.mon.a.smithi114.stdout:Apr 23 10:22:08 smithi114 ceph-mon[29255]: from='osd.7 ' entity='osd.7' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["7"]}]: dispatch 2024-04-23T10:22:08.755 INFO:journalctl@ceph.mon.a.smithi114.stdout:Apr 23 10:22:08 smithi114 ceph-mon[29255]: from='mgr.14152 172.21.15.114:0/1529706138' entity='mgr.x' 2024-04-23T10:22:08.755 INFO:journalctl@ceph.mon.a.smithi114.stdout:Apr 23 10:22:08 smithi114 ceph-mon[29255]: from='mgr.14152 172.21.15.114:0/1529706138' entity='mgr.x' 2024-04-23T10:22:08.755 INFO:journalctl@ceph.mon.a.smithi114.stdout:Apr 23 10:22:08 smithi114 ceph-mon[29255]: from='mgr.14152 172.21.15.114:0/1529706138' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-23T10:22:08.828 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 23 10:22:08 smithi204 ceph-mon[32649]: from='mgr.14152 172.21.15.114:0/1529706138' entity='mgr.x' 2024-04-23T10:22:08.828 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 23 10:22:08 smithi204 ceph-mon[32649]: from='mgr.14152 172.21.15.114:0/1529706138' entity='mgr.x' 2024-04-23T10:22:08.828 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 23 10:22:08 smithi204 ceph-mon[32649]: from='mgr.14152 172.21.15.114:0/1529706138' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-23T10:22:08.828 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 23 10:22:08 smithi204 ceph-mon[32649]: from='mgr.14152 172.21.15.114:0/1529706138' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-23T10:22:08.828 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 23 10:22:08 smithi204 ceph-mon[32649]: from='mgr.14152 172.21.15.114:0/1529706138' entity='mgr.x' 2024-04-23T10:22:08.828 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 23 10:22:08 smithi204 ceph-mon[32649]: pgmap v170: 1 pgs: 1 active+clean; 577 KiB data, 206 MiB used, 626 GiB / 626 GiB avail 2024-04-23T10:22:08.828 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 23 10:22:08 smithi204 ceph-mon[32649]: from='osd.7 [v2:172.21.15.119:6816/2237458793,v1:172.21.15.119:6817/2237458793]' entity='osd.7' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["7"]}]: dispatch 2024-04-23T10:22:08.829 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 23 10:22:08 smithi204 ceph-mon[32649]: from='osd.7 ' entity='osd.7' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["7"]}]: dispatch 2024-04-23T10:22:08.829 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 23 10:22:08 smithi204 ceph-mon[32649]: from='mgr.14152 172.21.15.114:0/1529706138' entity='mgr.x' 2024-04-23T10:22:08.829 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 23 10:22:08 smithi204 ceph-mon[32649]: from='mgr.14152 172.21.15.114:0/1529706138' entity='mgr.x' 2024-04-23T10:22:08.829 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 23 10:22:08 smithi204 ceph-mon[32649]: from='mgr.14152 172.21.15.114:0/1529706138' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-23T10:22:08.904 DEBUG:teuthology.orchestra.run.smithi119:osd.7> sudo journalctl -f -n 0 -u ceph-aa240572-0159-11ef-bc93-c7b262605968@osd.7.service 2024-04-23T10:22:08.906 INFO:tasks.cephadm:Deploying osd.8 on smithi204 with /dev/vg_nvme/lv_2... 2024-04-23T10:22:08.907 DEBUG:teuthology.orchestra.run.smithi204:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d69f13d827952bb07bb9b7973bca14f97d764d6f shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid aa240572-0159-11ef-bc93-c7b262605968 -- ceph-volume lvm zap /dev/vg_nvme/lv_2 2024-04-23T10:22:09.153 INFO:teuthology.orchestra.run.smithi204.stderr:Inferring config /var/lib/ceph/aa240572-0159-11ef-bc93-c7b262605968/mon.c/config 2024-04-23T10:22:09.492 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 23 10:22:09 smithi119 ceph-mon[32608]: from='osd.7 ' entity='osd.7' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["7"]}]': finished 2024-04-23T10:22:09.492 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 23 10:22:09 smithi119 ceph-mon[32608]: from='osd.7 [v2:172.21.15.119:6816/2237458793,v1:172.21.15.119:6817/2237458793]' entity='osd.7' cmd=[{"prefix": "osd crush create-or-move", "id": 7, "weight":0.0873, "args": ["host=smithi119", "root=default"]}]: dispatch 2024-04-23T10:22:09.492 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 23 10:22:09 smithi119 ceph-mon[32608]: osdmap e50: 8 total, 7 up, 8 in 2024-04-23T10:22:09.492 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 23 10:22:09 smithi119 ceph-mon[32608]: from='mgr.14152 172.21.15.114:0/1529706138' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-04-23T10:22:09.492 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 23 10:22:09 smithi119 ceph-mon[32608]: from='osd.7 ' entity='osd.7' cmd=[{"prefix": "osd crush create-or-move", "id": 7, "weight":0.0873, "args": ["host=smithi119", "root=default"]}]: dispatch 2024-04-23T10:22:09.532 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 23 10:22:09 smithi204 ceph-mon[32649]: from='osd.7 ' entity='osd.7' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["7"]}]': finished 2024-04-23T10:22:09.532 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 23 10:22:09 smithi204 ceph-mon[32649]: from='osd.7 [v2:172.21.15.119:6816/2237458793,v1:172.21.15.119:6817/2237458793]' entity='osd.7' cmd=[{"prefix": "osd crush create-or-move", "id": 7, "weight":0.0873, "args": ["host=smithi119", "root=default"]}]: dispatch 2024-04-23T10:22:09.533 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 23 10:22:09 smithi204 ceph-mon[32649]: osdmap e50: 8 total, 7 up, 8 in 2024-04-23T10:22:09.533 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 23 10:22:09 smithi204 ceph-mon[32649]: from='mgr.14152 172.21.15.114:0/1529706138' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-04-23T10:22:09.533 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 23 10:22:09 smithi204 ceph-mon[32649]: from='osd.7 ' entity='osd.7' cmd=[{"prefix": "osd crush create-or-move", "id": 7, "weight":0.0873, "args": ["host=smithi119", "root=default"]}]: dispatch 2024-04-23T10:22:09.754 INFO:journalctl@ceph.mon.a.smithi114.stdout:Apr 23 10:22:09 smithi114 ceph-mon[29255]: from='osd.7 ' entity='osd.7' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["7"]}]': finished 2024-04-23T10:22:09.754 INFO:journalctl@ceph.mon.a.smithi114.stdout:Apr 23 10:22:09 smithi114 ceph-mon[29255]: from='osd.7 [v2:172.21.15.119:6816/2237458793,v1:172.21.15.119:6817/2237458793]' entity='osd.7' cmd=[{"prefix": "osd crush create-or-move", "id": 7, "weight":0.0873, "args": ["host=smithi119", "root=default"]}]: dispatch 2024-04-23T10:22:09.754 INFO:journalctl@ceph.mon.a.smithi114.stdout:Apr 23 10:22:09 smithi114 ceph-mon[29255]: osdmap e50: 8 total, 7 up, 8 in 2024-04-23T10:22:09.754 INFO:journalctl@ceph.mon.a.smithi114.stdout:Apr 23 10:22:09 smithi114 ceph-mon[29255]: from='mgr.14152 172.21.15.114:0/1529706138' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-04-23T10:22:09.754 INFO:journalctl@ceph.mon.a.smithi114.stdout:Apr 23 10:22:09 smithi114 ceph-mon[29255]: from='osd.7 ' entity='osd.7' cmd=[{"prefix": "osd crush create-or-move", "id": 7, "weight":0.0873, "args": ["host=smithi119", "root=default"]}]: dispatch 2024-04-23T10:22:10.353 INFO:journalctl@ceph.osd.7.smithi119.stdout:Apr 23 10:22:10 smithi119 ceph-aa240572-0159-11ef-bc93-c7b262605968-osd-7[50859]: 2024-04-23T10:22:10.256+0000 7f0165ba6700 -1 osd.7 0 waiting for initial osdmap 2024-04-23T10:22:10.480 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 23 10:22:10 smithi204 ceph-mon[32649]: from='osd.7 ' entity='osd.7' cmd='[{"prefix": "osd crush create-or-move", "id": 7, "weight":0.0873, "args": ["host=smithi119", "root=default"]}]': finished 2024-04-23T10:22:10.480 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 23 10:22:10 smithi204 ceph-mon[32649]: osdmap e51: 8 total, 7 up, 8 in 2024-04-23T10:22:10.480 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 23 10:22:10 smithi204 ceph-mon[32649]: from='mgr.14152 172.21.15.114:0/1529706138' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-04-23T10:22:10.481 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 23 10:22:10 smithi204 ceph-mon[32649]: from='mgr.14152 172.21.15.114:0/1529706138' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-04-23T10:22:10.482 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 23 10:22:10 smithi204 ceph-mon[32649]: pgmap v173: 1 pgs: 1 active+clean; 577 KiB data, 206 MiB used, 626 GiB / 626 GiB avail 2024-04-23T10:22:10.482 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 23 10:22:10 smithi204 ceph-mon[32649]: from='osd.7 ' entity='osd.7' 2024-04-23T10:22:10.742 INFO:journalctl@ceph.osd.7.smithi119.stdout:Apr 23 10:22:10 smithi119 ceph-aa240572-0159-11ef-bc93-c7b262605968-osd-7[50859]: 2024-04-23T10:22:10.418+0000 7f015d991700 -1 osd.7 52 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-04-23T10:22:10.742 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 23 10:22:10 smithi119 ceph-mon[32608]: from='osd.7 ' entity='osd.7' cmd='[{"prefix": "osd crush create-or-move", "id": 7, "weight":0.0873, "args": ["host=smithi119", "root=default"]}]': finished 2024-04-23T10:22:10.742 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 23 10:22:10 smithi119 ceph-mon[32608]: osdmap e51: 8 total, 7 up, 8 in 2024-04-23T10:22:10.742 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 23 10:22:10 smithi119 ceph-mon[32608]: from='mgr.14152 172.21.15.114:0/1529706138' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-04-23T10:22:10.742 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 23 10:22:10 smithi119 ceph-mon[32608]: from='mgr.14152 172.21.15.114:0/1529706138' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-04-23T10:22:10.742 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 23 10:22:10 smithi119 ceph-mon[32608]: pgmap v173: 1 pgs: 1 active+clean; 577 KiB data, 206 MiB used, 626 GiB / 626 GiB avail 2024-04-23T10:22:10.743 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 23 10:22:10 smithi119 ceph-mon[32608]: from='osd.7 ' entity='osd.7' 2024-04-23T10:22:10.754 INFO:journalctl@ceph.mon.a.smithi114.stdout:Apr 23 10:22:10 smithi114 ceph-mon[29255]: from='osd.7 ' entity='osd.7' cmd='[{"prefix": "osd crush create-or-move", "id": 7, "weight":0.0873, "args": ["host=smithi119", "root=default"]}]': finished 2024-04-23T10:22:10.754 INFO:journalctl@ceph.mon.a.smithi114.stdout:Apr 23 10:22:10 smithi114 ceph-mon[29255]: osdmap e51: 8 total, 7 up, 8 in 2024-04-23T10:22:10.754 INFO:journalctl@ceph.mon.a.smithi114.stdout:Apr 23 10:22:10 smithi114 ceph-mon[29255]: from='mgr.14152 172.21.15.114:0/1529706138' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-04-23T10:22:10.754 INFO:journalctl@ceph.mon.a.smithi114.stdout:Apr 23 10:22:10 smithi114 ceph-mon[29255]: from='mgr.14152 172.21.15.114:0/1529706138' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-04-23T10:22:10.754 INFO:journalctl@ceph.mon.a.smithi114.stdout:Apr 23 10:22:10 smithi114 ceph-mon[29255]: pgmap v173: 1 pgs: 1 active+clean; 577 KiB data, 206 MiB used, 626 GiB / 626 GiB avail 2024-04-23T10:22:10.754 INFO:journalctl@ceph.mon.a.smithi114.stdout:Apr 23 10:22:10 smithi114 ceph-mon[29255]: from='osd.7 ' entity='osd.7' 2024-04-23T10:22:11.036 INFO:teuthology.orchestra.run.smithi204.stderr:--> Zapping: /dev/vg_nvme/lv_2 2024-04-23T10:22:11.036 INFO:teuthology.orchestra.run.smithi204.stderr:Running command: /usr/bin/dd if=/dev/zero of=/dev/vg_nvme/lv_2 bs=1M count=10 conv=fsync 2024-04-23T10:22:11.036 INFO:teuthology.orchestra.run.smithi204.stderr: stderr: 10+0 records in 2024-04-23T10:22:11.036 INFO:teuthology.orchestra.run.smithi204.stderr:10+0 records out 2024-04-23T10:22:11.036 INFO:teuthology.orchestra.run.smithi204.stderr: stderr: 10485760 bytes (10 MB, 10 MiB) copied, 0.0114963 s, 912 MB/s 2024-04-23T10:22:11.036 INFO:teuthology.orchestra.run.smithi204.stderr:--> Zapping successful for: 2024-04-23T10:22:11.466 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 23 10:22:11 smithi119 ceph-mon[32608]: purged_snaps scrub starts 2024-04-23T10:22:11.467 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 23 10:22:11 smithi119 ceph-mon[32608]: purged_snaps scrub ok 2024-04-23T10:22:11.467 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 23 10:22:11 smithi119 ceph-mon[32608]: from='mgr.14152 172.21.15.114:0/1529706138' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-04-23T10:22:11.467 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 23 10:22:11 smithi119 ceph-mon[32608]: osdmap e52: 8 total, 7 up, 8 in 2024-04-23T10:22:11.467 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 23 10:22:11 smithi119 ceph-mon[32608]: from='mgr.14152 172.21.15.114:0/1529706138' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-04-23T10:22:11.467 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 23 10:22:11 smithi119 ceph-mon[32608]: from='mgr.14152 172.21.15.114:0/1529706138' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-04-23T10:22:11.613 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 23 10:22:11 smithi204 ceph-mon[32649]: purged_snaps scrub starts 2024-04-23T10:22:11.613 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 23 10:22:11 smithi204 ceph-mon[32649]: purged_snaps scrub ok 2024-04-23T10:22:11.613 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 23 10:22:11 smithi204 ceph-mon[32649]: from='mgr.14152 172.21.15.114:0/1529706138' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-04-23T10:22:11.613 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 23 10:22:11 smithi204 ceph-mon[32649]: osdmap e52: 8 total, 7 up, 8 in 2024-04-23T10:22:11.613 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 23 10:22:11 smithi204 ceph-mon[32649]: from='mgr.14152 172.21.15.114:0/1529706138' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-04-23T10:22:11.613 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 23 10:22:11 smithi204 ceph-mon[32649]: from='mgr.14152 172.21.15.114:0/1529706138' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-04-23T10:22:11.615 DEBUG:teuthology.orchestra.run.smithi204:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d69f13d827952bb07bb9b7973bca14f97d764d6f shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid aa240572-0159-11ef-bc93-c7b262605968 -- ceph orch daemon add osd smithi204:vg_nvme/lv_2 2024-04-23T10:22:11.754 INFO:journalctl@ceph.mon.a.smithi114.stdout:Apr 23 10:22:11 smithi114 ceph-mon[29255]: purged_snaps scrub starts 2024-04-23T10:22:11.754 INFO:journalctl@ceph.mon.a.smithi114.stdout:Apr 23 10:22:11 smithi114 ceph-mon[29255]: purged_snaps scrub ok 2024-04-23T10:22:11.754 INFO:journalctl@ceph.mon.a.smithi114.stdout:Apr 23 10:22:11 smithi114 ceph-mon[29255]: from='mgr.14152 172.21.15.114:0/1529706138' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-04-23T10:22:11.754 INFO:journalctl@ceph.mon.a.smithi114.stdout:Apr 23 10:22:11 smithi114 ceph-mon[29255]: osdmap e52: 8 total, 7 up, 8 in 2024-04-23T10:22:11.754 INFO:journalctl@ceph.mon.a.smithi114.stdout:Apr 23 10:22:11 smithi114 ceph-mon[29255]: from='mgr.14152 172.21.15.114:0/1529706138' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-04-23T10:22:11.754 INFO:journalctl@ceph.mon.a.smithi114.stdout:Apr 23 10:22:11 smithi114 ceph-mon[29255]: from='mgr.14152 172.21.15.114:0/1529706138' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-04-23T10:22:11.859 INFO:teuthology.orchestra.run.smithi204.stderr:Inferring config /var/lib/ceph/aa240572-0159-11ef-bc93-c7b262605968/mon.c/config 2024-04-23T10:22:12.578 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 23 10:22:12 smithi204 ceph-mon[32649]: osd.7 [v2:172.21.15.119:6816/2237458793,v1:172.21.15.119:6817/2237458793] boot 2024-04-23T10:22:12.578 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 23 10:22:12 smithi204 ceph-mon[32649]: osdmap e53: 8 total, 8 up, 8 in 2024-04-23T10:22:12.578 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 23 10:22:12 smithi204 ceph-mon[32649]: from='mgr.14152 172.21.15.114:0/1529706138' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-04-23T10:22:12.578 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 23 10:22:12 smithi204 ceph-mon[32649]: Detected new or changed devices on smithi119 2024-04-23T10:22:12.579 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 23 10:22:12 smithi204 ceph-mon[32649]: from='mgr.14152 172.21.15.114:0/1529706138' entity='mgr.x' 2024-04-23T10:22:12.579 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 23 10:22:12 smithi204 ceph-mon[32649]: from='mgr.14152 172.21.15.114:0/1529706138' entity='mgr.x' 2024-04-23T10:22:12.579 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 23 10:22:12 smithi204 ceph-mon[32649]: from='mgr.14152 172.21.15.114:0/1529706138' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-04-23T10:22:12.579 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 23 10:22:12 smithi204 ceph-mon[32649]: from='mgr.14152 172.21.15.114:0/1529706138' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-04-23T10:22:12.579 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 23 10:22:12 smithi204 ceph-mon[32649]: from='mgr.14152 172.21.15.114:0/1529706138' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-04-23T10:22:12.579 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 23 10:22:12 smithi204 ceph-mon[32649]: Adjusting osd_memory_target on smithi119 to 5696M 2024-04-23T10:22:12.579 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 23 10:22:12 smithi204 ceph-mon[32649]: from='mgr.14152 172.21.15.114:0/1529706138' entity='mgr.x' 2024-04-23T10:22:12.579 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 23 10:22:12 smithi204 ceph-mon[32649]: from='mgr.14152 172.21.15.114:0/1529706138' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-23T10:22:12.579 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 23 10:22:12 smithi204 ceph-mon[32649]: from='mgr.14152 172.21.15.114:0/1529706138' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-23T10:22:12.579 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 23 10:22:12 smithi204 ceph-mon[32649]: from='mgr.14152 172.21.15.114:0/1529706138' entity='mgr.x' 2024-04-23T10:22:12.579 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 23 10:22:12 smithi204 ceph-mon[32649]: pgmap v176: 1 pgs: 1 active+clean; 577 KiB data, 206 MiB used, 626 GiB / 626 GiB avail 2024-04-23T10:22:12.742 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 23 10:22:12 smithi119 ceph-mon[32608]: osd.7 [v2:172.21.15.119:6816/2237458793,v1:172.21.15.119:6817/2237458793] boot 2024-04-23T10:22:12.742 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 23 10:22:12 smithi119 ceph-mon[32608]: osdmap e53: 8 total, 8 up, 8 in 2024-04-23T10:22:12.742 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 23 10:22:12 smithi119 ceph-mon[32608]: from='mgr.14152 172.21.15.114:0/1529706138' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-04-23T10:22:12.743 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 23 10:22:12 smithi119 ceph-mon[32608]: Detected new or changed devices on smithi119 2024-04-23T10:22:12.743 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 23 10:22:12 smithi119 ceph-mon[32608]: from='mgr.14152 172.21.15.114:0/1529706138' entity='mgr.x' 2024-04-23T10:22:12.743 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 23 10:22:12 smithi119 ceph-mon[32608]: from='mgr.14152 172.21.15.114:0/1529706138' entity='mgr.x' 2024-04-23T10:22:12.743 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 23 10:22:12 smithi119 ceph-mon[32608]: from='mgr.14152 172.21.15.114:0/1529706138' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-04-23T10:22:12.743 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 23 10:22:12 smithi119 ceph-mon[32608]: from='mgr.14152 172.21.15.114:0/1529706138' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-04-23T10:22:12.743 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 23 10:22:12 smithi119 ceph-mon[32608]: from='mgr.14152 172.21.15.114:0/1529706138' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-04-23T10:22:12.743 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 23 10:22:12 smithi119 ceph-mon[32608]: Adjusting osd_memory_target on smithi119 to 5696M 2024-04-23T10:22:12.743 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 23 10:22:12 smithi119 ceph-mon[32608]: from='mgr.14152 172.21.15.114:0/1529706138' entity='mgr.x' 2024-04-23T10:22:12.743 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 23 10:22:12 smithi119 ceph-mon[32608]: from='mgr.14152 172.21.15.114:0/1529706138' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-23T10:22:12.743 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 23 10:22:12 smithi119 ceph-mon[32608]: from='mgr.14152 172.21.15.114:0/1529706138' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-23T10:22:12.744 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 23 10:22:12 smithi119 ceph-mon[32608]: from='mgr.14152 172.21.15.114:0/1529706138' entity='mgr.x' 2024-04-23T10:22:12.744 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 23 10:22:12 smithi119 ceph-mon[32608]: pgmap v176: 1 pgs: 1 active+clean; 577 KiB data, 206 MiB used, 626 GiB / 626 GiB avail 2024-04-23T10:22:12.754 INFO:journalctl@ceph.mon.a.smithi114.stdout:Apr 23 10:22:12 smithi114 ceph-mon[29255]: osd.7 [v2:172.21.15.119:6816/2237458793,v1:172.21.15.119:6817/2237458793] boot 2024-04-23T10:22:12.754 INFO:journalctl@ceph.mon.a.smithi114.stdout:Apr 23 10:22:12 smithi114 ceph-mon[29255]: osdmap e53: 8 total, 8 up, 8 in 2024-04-23T10:22:12.754 INFO:journalctl@ceph.mon.a.smithi114.stdout:Apr 23 10:22:12 smithi114 ceph-mon[29255]: from='mgr.14152 172.21.15.114:0/1529706138' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-04-23T10:22:12.754 INFO:journalctl@ceph.mon.a.smithi114.stdout:Apr 23 10:22:12 smithi114 ceph-mon[29255]: Detected new or changed devices on smithi119 2024-04-23T10:22:12.754 INFO:journalctl@ceph.mon.a.smithi114.stdout:Apr 23 10:22:12 smithi114 ceph-mon[29255]: from='mgr.14152 172.21.15.114:0/1529706138' entity='mgr.x' 2024-04-23T10:22:12.755 INFO:journalctl@ceph.mon.a.smithi114.stdout:Apr 23 10:22:12 smithi114 ceph-mon[29255]: from='mgr.14152 172.21.15.114:0/1529706138' entity='mgr.x' 2024-04-23T10:22:12.755 INFO:journalctl@ceph.mon.a.smithi114.stdout:Apr 23 10:22:12 smithi114 ceph-mon[29255]: from='mgr.14152 172.21.15.114:0/1529706138' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-04-23T10:22:12.755 INFO:journalctl@ceph.mon.a.smithi114.stdout:Apr 23 10:22:12 smithi114 ceph-mon[29255]: from='mgr.14152 172.21.15.114:0/1529706138' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-04-23T10:22:12.755 INFO:journalctl@ceph.mon.a.smithi114.stdout:Apr 23 10:22:12 smithi114 ceph-mon[29255]: from='mgr.14152 172.21.15.114:0/1529706138' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-04-23T10:22:12.755 INFO:journalctl@ceph.mon.a.smithi114.stdout:Apr 23 10:22:12 smithi114 ceph-mon[29255]: Adjusting osd_memory_target on smithi119 to 5696M 2024-04-23T10:22:12.755 INFO:journalctl@ceph.mon.a.smithi114.stdout:Apr 23 10:22:12 smithi114 ceph-mon[29255]: from='mgr.14152 172.21.15.114:0/1529706138' entity='mgr.x' 2024-04-23T10:22:12.755 INFO:journalctl@ceph.mon.a.smithi114.stdout:Apr 23 10:22:12 smithi114 ceph-mon[29255]: from='mgr.14152 172.21.15.114:0/1529706138' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-23T10:22:12.755 INFO:journalctl@ceph.mon.a.smithi114.stdout:Apr 23 10:22:12 smithi114 ceph-mon[29255]: from='mgr.14152 172.21.15.114:0/1529706138' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-23T10:22:12.755 INFO:journalctl@ceph.mon.a.smithi114.stdout:Apr 23 10:22:12 smithi114 ceph-mon[29255]: from='mgr.14152 172.21.15.114:0/1529706138' entity='mgr.x' 2024-04-23T10:22:12.755 INFO:journalctl@ceph.mon.a.smithi114.stdout:Apr 23 10:22:12 smithi114 ceph-mon[29255]: pgmap v176: 1 pgs: 1 active+clean; 577 KiB data, 206 MiB used, 626 GiB / 626 GiB avail 2024-04-23T10:22:13.472 INFO:teuthology.orchestra.run.smithi204.stderr:2024-04-23T10:22:13.470+0000 7fe7eb5ae700 1 -- 172.21.15.204:0/988896737 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fe7e4102ba0 msgr2=0x7fe7e4103000 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T10:22:13.472 INFO:teuthology.orchestra.run.smithi204.stderr:2024-04-23T10:22:13.470+0000 7fe7eb5ae700 1 --2- 172.21.15.204:0/988896737 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fe7e4102ba0 0x7fe7e4103000 secure :-1 s=READY pgs=28 cs=0 l=1 rev1=1 crypto rx=0x7fe7e0009990 tx=0x7fe7e0009ca0 comp rx=0 tx=0).stop 2024-04-23T10:22:13.472 INFO:teuthology.orchestra.run.smithi204.stderr:2024-04-23T10:22:13.470+0000 7fe7eb5ae700 1 -- 172.21.15.204:0/988896737 shutdown_connections 2024-04-23T10:22:13.473 INFO:teuthology.orchestra.run.smithi204.stderr:2024-04-23T10:22:13.470+0000 7fe7eb5ae700 1 --2- 172.21.15.204:0/988896737 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fe7e41019a0 0x7fe7e4101dc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:22:13.473 INFO:teuthology.orchestra.run.smithi204.stderr:2024-04-23T10:22:13.470+0000 7fe7eb5ae700 1 --2- 172.21.15.204:0/988896737 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fe7e4103540 0x7fe7e4109df0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:22:13.473 INFO:teuthology.orchestra.run.smithi204.stderr:2024-04-23T10:22:13.470+0000 7fe7eb5ae700 1 --2- 172.21.15.204:0/988896737 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fe7e4102ba0 0x7fe7e4103000 unknown :-1 s=CLOSED pgs=28 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:22:13.473 INFO:teuthology.orchestra.run.smithi204.stderr:2024-04-23T10:22:13.470+0000 7fe7eb5ae700 1 -- 172.21.15.204:0/988896737 >> 172.21.15.204:0/988896737 conn(0x7fe7e40fcf40 msgr2=0x7fe7e40ff380 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T10:22:13.473 INFO:teuthology.orchestra.run.smithi204.stderr:2024-04-23T10:22:13.470+0000 7fe7eb5ae700 1 -- 172.21.15.204:0/988896737 shutdown_connections 2024-04-23T10:22:13.473 INFO:teuthology.orchestra.run.smithi204.stderr:2024-04-23T10:22:13.470+0000 7fe7eb5ae700 1 -- 172.21.15.204:0/988896737 wait complete. 2024-04-23T10:22:13.473 INFO:teuthology.orchestra.run.smithi204.stderr:2024-04-23T10:22:13.471+0000 7fe7eb5ae700 1 Processor -- start 2024-04-23T10:22:13.473 INFO:teuthology.orchestra.run.smithi204.stderr:2024-04-23T10:22:13.471+0000 7fe7eb5ae700 1 -- start start 2024-04-23T10:22:13.473 INFO:teuthology.orchestra.run.smithi204.stderr:2024-04-23T10:22:13.471+0000 7fe7eb5ae700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fe7e41019a0 0x7fe7e406b250 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T10:22:13.473 INFO:teuthology.orchestra.run.smithi204.stderr:2024-04-23T10:22:13.471+0000 7fe7eb5ae700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fe7e4102ba0 0x7fe7e419da60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T10:22:13.473 INFO:teuthology.orchestra.run.smithi204.stderr:2024-04-23T10:22:13.471+0000 7fe7eb5ae700 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fe7e4103540 0x7fe7e41a0050 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T10:22:13.474 INFO:teuthology.orchestra.run.smithi204.stderr:2024-04-23T10:22:13.471+0000 7fe7eb5ae700 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x7fe7e410d4b0 con 0x7fe7e41019a0 2024-04-23T10:22:13.474 INFO:teuthology.orchestra.run.smithi204.stderr:2024-04-23T10:22:13.471+0000 7fe7eb5ae700 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 -- 0x7fe7e410d330 con 0x7fe7e4102ba0 2024-04-23T10:22:13.474 INFO:teuthology.orchestra.run.smithi204.stderr:2024-04-23T10:22:13.471+0000 7fe7eb5ae700 1 -- --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_getmap magic: 0 -- 0x7fe7e410d630 con 0x7fe7e4103540 2024-04-23T10:22:13.474 INFO:teuthology.orchestra.run.smithi204.stderr:2024-04-23T10:22:13.471+0000 7fe7e9b4b700 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fe7e4103540 0x7fe7e41a0050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T10:22:13.474 INFO:teuthology.orchestra.run.smithi204.stderr:2024-04-23T10:22:13.471+0000 7fe7e9b4b700 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fe7e4103540 0x7fe7e41a0050 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.204:3300/0 says I am v2:172.21.15.204:52720/0 (socket says 172.21.15.204:52720) 2024-04-23T10:22:13.474 INFO:teuthology.orchestra.run.smithi204.stderr:2024-04-23T10:22:13.471+0000 7fe7e9b4b700 1 -- 172.21.15.204:0/708081393 learned_addr learned my addr 172.21.15.204:0/708081393 (peer_addr_for_me v2:172.21.15.204:0/0) 2024-04-23T10:22:13.474 INFO:teuthology.orchestra.run.smithi204.stderr:2024-04-23T10:22:13.471+0000 7fe7e8b49700 1 --2- 172.21.15.204:0/708081393 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fe7e4102ba0 0x7fe7e419da60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T10:22:13.474 INFO:teuthology.orchestra.run.smithi204.stderr:2024-04-23T10:22:13.471+0000 7fe7e934a700 1 --2- 172.21.15.204:0/708081393 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fe7e41019a0 0x7fe7e406b250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T10:22:13.474 INFO:teuthology.orchestra.run.smithi204.stderr:2024-04-23T10:22:13.471+0000 7fe7e9b4b700 1 -- 172.21.15.204:0/708081393 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fe7e4102ba0 msgr2=0x7fe7e419da60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T10:22:13.474 INFO:teuthology.orchestra.run.smithi204.stderr:2024-04-23T10:22:13.471+0000 7fe7e9b4b700 1 --2- 172.21.15.204:0/708081393 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fe7e4102ba0 0x7fe7e419da60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:22:13.475 INFO:teuthology.orchestra.run.smithi204.stderr:2024-04-23T10:22:13.471+0000 7fe7e9b4b700 1 -- 172.21.15.204:0/708081393 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fe7e41019a0 msgr2=0x7fe7e406b250 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T10:22:13.475 INFO:teuthology.orchestra.run.smithi204.stderr:2024-04-23T10:22:13.471+0000 7fe7e9b4b700 1 --2- 172.21.15.204:0/708081393 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fe7e41019a0 0x7fe7e406b250 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:22:13.475 INFO:teuthology.orchestra.run.smithi204.stderr:2024-04-23T10:22:13.471+0000 7fe7e9b4b700 1 -- 172.21.15.204:0/708081393 --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe7e4197830 con 0x7fe7e4103540 2024-04-23T10:22:13.475 INFO:teuthology.orchestra.run.smithi204.stderr:2024-04-23T10:22:13.472+0000 7fe7e9b4b700 1 --2- 172.21.15.204:0/708081393 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fe7e4103540 0x7fe7e41a0050 secure :-1 s=READY pgs=29 cs=0 l=1 rev1=1 crypto rx=0x7fe7dc00ada0 tx=0x7fe7dc00c330 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T10:22:13.475 INFO:teuthology.orchestra.run.smithi204.stderr:2024-04-23T10:22:13.472+0000 7fe7da7fc700 1 -- 172.21.15.204:0/708081393 <== mon.1 v2:172.21.15.204:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe7dc017020 con 0x7fe7e4103540 2024-04-23T10:22:13.475 INFO:teuthology.orchestra.run.smithi204.stderr:2024-04-23T10:22:13.472+0000 7fe7eb5ae700 1 -- 172.21.15.204:0/708081393 --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fe7e4197b20 con 0x7fe7e4103540 2024-04-23T10:22:13.475 INFO:teuthology.orchestra.run.smithi204.stderr:2024-04-23T10:22:13.472+0000 7fe7da7fc700 1 -- 172.21.15.204:0/708081393 <== mon.1 v2:172.21.15.204:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fe7dc00cd80 con 0x7fe7e4103540 2024-04-23T10:22:13.475 INFO:teuthology.orchestra.run.smithi204.stderr:2024-04-23T10:22:13.472+0000 7fe7da7fc700 1 -- 172.21.15.204:0/708081393 <== mon.1 v2:172.21.15.204:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe7dc005280 con 0x7fe7e4103540 2024-04-23T10:22:13.475 INFO:teuthology.orchestra.run.smithi204.stderr:2024-04-23T10:22:13.472+0000 7fe7eb5ae700 1 -- 172.21.15.204:0/708081393 --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fe7e41980b0 con 0x7fe7e4103540 2024-04-23T10:22:13.475 INFO:teuthology.orchestra.run.smithi204.stderr:2024-04-23T10:22:13.473+0000 7fe7eb5ae700 1 -- 172.21.15.204:0/708081393 --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fe7e4069730 con 0x7fe7e4103540 2024-04-23T10:22:13.476 INFO:teuthology.orchestra.run.smithi204.stderr:2024-04-23T10:22:13.474+0000 7fe7da7fc700 1 -- 172.21.15.204:0/708081393 <== mon.1 v2:172.21.15.204:3300/0 4 ==== mgrmap(e 16) ==== 138647+0+0 (secure 0 0 0) 0x7fe7dc00a090 con 0x7fe7e4103540 2024-04-23T10:22:13.476 INFO:teuthology.orchestra.run.smithi204.stderr:2024-04-23T10:22:13.474+0000 7fe7da7fc700 1 --2- 172.21.15.204:0/708081393 >> [v2:172.21.15.114:6800/465333808,v1:172.21.15.114:6801/465333808] conn(0x7fe7d00a3c40 0x7fe7d00a6100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T10:22:13.477 INFO:teuthology.orchestra.run.smithi204.stderr:2024-04-23T10:22:13.474+0000 7fe7da7fc700 1 -- 172.21.15.204:0/708081393 <== mon.1 v2:172.21.15.204:3300/0 5 ==== osd_map(55..55 src has 1..55) ==== 5426+0+0 (secure 0 0 0) 0x7fe7dc0cf550 con 0x7fe7e4103540 2024-04-23T10:22:13.477 INFO:teuthology.orchestra.run.smithi204.stderr:2024-04-23T10:22:13.475+0000 7fe7e934a700 1 --2- 172.21.15.204:0/708081393 >> [v2:172.21.15.114:6800/465333808,v1:172.21.15.114:6801/465333808] conn(0x7fe7d00a3c40 0x7fe7d00a6100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T10:22:13.477 INFO:teuthology.orchestra.run.smithi204.stderr:2024-04-23T10:22:13.475+0000 7fe7e934a700 1 --2- 172.21.15.204:0/708081393 >> [v2:172.21.15.114:6800/465333808,v1:172.21.15.114:6801/465333808] conn(0x7fe7d00a3c40 0x7fe7d00a6100 secure :-1 s=READY pgs=100 cs=0 l=1 rev1=1 crypto rx=0x7fe7e4102a00 tx=0x7fe7d4009190 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T10:22:13.477 INFO:teuthology.orchestra.run.smithi204.stderr:2024-04-23T10:22:13.478+0000 7fe7da7fc700 1 -- 172.21.15.204:0/708081393 <== mon.1 v2:172.21.15.204:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7fe7dc021020 con 0x7fe7e4103540 2024-04-23T10:22:13.578 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 23 10:22:13 smithi204 ceph-mon[32649]: osdmap e54: 8 total, 8 up, 8 in 2024-04-23T10:22:13.686 INFO:teuthology.orchestra.run.smithi204.stderr:2024-04-23T10:22:13.687+0000 7fe7eb5ae700 1 -- 172.21.15.204:0/708081393 --> [v2:172.21.15.114:6800/465333808,v1:172.21.15.114:6801/465333808] -- mgr_command(tid 0: {"prefix": "orch daemon add osd", "svc_arg": "smithi204:vg_nvme/lv_2", "target": ["mon-mgr", ""]}) -- 0x7fe7e41089c0 con 0x7fe7d00a3c40 2024-04-23T10:22:13.742 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 23 10:22:13 smithi119 ceph-mon[32608]: osdmap e54: 8 total, 8 up, 8 in 2024-04-23T10:22:13.754 INFO:journalctl@ceph.mon.a.smithi114.stdout:Apr 23 10:22:13 smithi114 ceph-mon[29255]: osdmap e54: 8 total, 8 up, 8 in 2024-04-23T10:22:14.578 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 23 10:22:14 smithi204 ceph-mon[32649]: osdmap e55: 8 total, 8 up, 8 in 2024-04-23T10:22:14.578 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 23 10:22:14 smithi204 ceph-mon[32649]: from='client.24262 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi204:vg_nvme/lv_2", "target": ["mon-mgr", ""]}]: dispatch 2024-04-23T10:22:14.578 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 23 10:22:14 smithi204 ceph-mon[32649]: from='mgr.14152 172.21.15.114:0/1529706138' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-04-23T10:22:14.578 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 23 10:22:14 smithi204 ceph-mon[32649]: from='mgr.14152 172.21.15.114:0/1529706138' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-04-23T10:22:14.578 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 23 10:22:14 smithi204 ceph-mon[32649]: from='mgr.14152 172.21.15.114:0/1529706138' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-23T10:22:14.578 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 23 10:22:14 smithi204 ceph-mon[32649]: pgmap v179: 1 pgs: 1 active+undersized+remapped; 577 KiB data, 241 MiB used, 715 GiB / 715 GiB avail 2024-04-23T10:22:14.742 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 23 10:22:14 smithi119 ceph-mon[32608]: osdmap e55: 8 total, 8 up, 8 in 2024-04-23T10:22:14.742 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 23 10:22:14 smithi119 ceph-mon[32608]: from='client.24262 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi204:vg_nvme/lv_2", "target": ["mon-mgr", ""]}]: dispatch 2024-04-23T10:22:14.742 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 23 10:22:14 smithi119 ceph-mon[32608]: from='mgr.14152 172.21.15.114:0/1529706138' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-04-23T10:22:14.742 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 23 10:22:14 smithi119 ceph-mon[32608]: from='mgr.14152 172.21.15.114:0/1529706138' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-04-23T10:22:14.742 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 23 10:22:14 smithi119 ceph-mon[32608]: from='mgr.14152 172.21.15.114:0/1529706138' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-23T10:22:14.742 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 23 10:22:14 smithi119 ceph-mon[32608]: pgmap v179: 1 pgs: 1 active+undersized+remapped; 577 KiB data, 241 MiB used, 715 GiB / 715 GiB avail 2024-04-23T10:22:14.754 INFO:journalctl@ceph.mon.a.smithi114.stdout:Apr 23 10:22:14 smithi114 ceph-mon[29255]: osdmap e55: 8 total, 8 up, 8 in 2024-04-23T10:22:14.754 INFO:journalctl@ceph.mon.a.smithi114.stdout:Apr 23 10:22:14 smithi114 ceph-mon[29255]: from='client.24262 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi204:vg_nvme/lv_2", "target": ["mon-mgr", ""]}]: dispatch 2024-04-23T10:22:14.754 INFO:journalctl@ceph.mon.a.smithi114.stdout:Apr 23 10:22:14 smithi114 ceph-mon[29255]: from='mgr.14152 172.21.15.114:0/1529706138' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-04-23T10:22:14.754 INFO:journalctl@ceph.mon.a.smithi114.stdout:Apr 23 10:22:14 smithi114 ceph-mon[29255]: from='mgr.14152 172.21.15.114:0/1529706138' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-04-23T10:22:14.754 INFO:journalctl@ceph.mon.a.smithi114.stdout:Apr 23 10:22:14 smithi114 ceph-mon[29255]: from='mgr.14152 172.21.15.114:0/1529706138' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-23T10:22:14.754 INFO:journalctl@ceph.mon.a.smithi114.stdout:Apr 23 10:22:14 smithi114 ceph-mon[29255]: pgmap v179: 1 pgs: 1 active+undersized+remapped; 577 KiB data, 241 MiB used, 715 GiB / 715 GiB avail 2024-04-23T10:22:15.529 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 23 10:22:15 smithi204 ceph-mon[32649]: osdmap e56: 8 total, 8 up, 8 in 2024-04-23T10:22:15.655 INFO:journalctl@ceph.mon.a.smithi114.stdout:Apr 23 10:22:15 smithi114 ceph-mon[29255]: osdmap e56: 8 total, 8 up, 8 in 2024-04-23T10:22:15.742 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 23 10:22:15 smithi119 ceph-mon[32608]: osdmap e56: 8 total, 8 up, 8 in 2024-04-23T10:22:16.671 INFO:journalctl@ceph.mon.a.smithi114.stdout:Apr 23 10:22:16 smithi114 ceph-mon[29255]: osdmap e57: 8 total, 8 up, 8 in 2024-04-23T10:22:16.671 INFO:journalctl@ceph.mon.a.smithi114.stdout:Apr 23 10:22:16 smithi114 ceph-mon[29255]: pgmap v182: 1 pgs: 1 active+undersized+remapped; 577 KiB data, 241 MiB used, 715 GiB / 715 GiB avail 2024-04-23T10:22:16.671 INFO:journalctl@ceph.mon.a.smithi114.stdout:Apr 23 10:22:16 smithi114 ceph-mon[29255]: from='client.? 172.21.15.204:0/1281308728' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "3ac89686-6d76-4cef-9089-ac29e4b3bcac"}]: dispatch 2024-04-23T10:22:16.671 INFO:journalctl@ceph.mon.a.smithi114.stdout:Apr 23 10:22:16 smithi114 ceph-mon[29255]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "3ac89686-6d76-4cef-9089-ac29e4b3bcac"}]: dispatch 2024-04-23T10:22:16.671 INFO:journalctl@ceph.mon.a.smithi114.stdout:Apr 23 10:22:16 smithi114 ceph-mon[29255]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "3ac89686-6d76-4cef-9089-ac29e4b3bcac"}]': finished 2024-04-23T10:22:16.672 INFO:journalctl@ceph.mon.a.smithi114.stdout:Apr 23 10:22:16 smithi114 ceph-mon[29255]: osdmap e58: 9 total, 8 up, 9 in 2024-04-23T10:22:16.672 INFO:journalctl@ceph.mon.a.smithi114.stdout:Apr 23 10:22:16 smithi114 ceph-mon[29255]: from='mgr.14152 172.21.15.114:0/1529706138' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 8}]: dispatch 2024-04-23T10:22:16.698 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 23 10:22:16 smithi204 ceph-mon[32649]: osdmap e57: 8 total, 8 up, 8 in 2024-04-23T10:22:16.698 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 23 10:22:16 smithi204 ceph-mon[32649]: pgmap v182: 1 pgs: 1 active+undersized+remapped; 577 KiB data, 241 MiB used, 715 GiB / 715 GiB avail 2024-04-23T10:22:16.698 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 23 10:22:16 smithi204 ceph-mon[32649]: from='client.? 172.21.15.204:0/1281308728' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "3ac89686-6d76-4cef-9089-ac29e4b3bcac"}]: dispatch 2024-04-23T10:22:16.698 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 23 10:22:16 smithi204 ceph-mon[32649]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "3ac89686-6d76-4cef-9089-ac29e4b3bcac"}]: dispatch 2024-04-23T10:22:16.698 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 23 10:22:16 smithi204 ceph-mon[32649]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "3ac89686-6d76-4cef-9089-ac29e4b3bcac"}]': finished 2024-04-23T10:22:16.698 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 23 10:22:16 smithi204 ceph-mon[32649]: osdmap e58: 9 total, 8 up, 9 in 2024-04-23T10:22:16.698 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 23 10:22:16 smithi204 ceph-mon[32649]: from='mgr.14152 172.21.15.114:0/1529706138' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 8}]: dispatch 2024-04-23T10:22:16.742 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 23 10:22:16 smithi119 ceph-mon[32608]: osdmap e57: 8 total, 8 up, 8 in 2024-04-23T10:22:16.742 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 23 10:22:16 smithi119 ceph-mon[32608]: pgmap v182: 1 pgs: 1 active+undersized+remapped; 577 KiB data, 241 MiB used, 715 GiB / 715 GiB avail 2024-04-23T10:22:16.742 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 23 10:22:16 smithi119 ceph-mon[32608]: from='client.? 172.21.15.204:0/1281308728' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "3ac89686-6d76-4cef-9089-ac29e4b3bcac"}]: dispatch 2024-04-23T10:22:16.742 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 23 10:22:16 smithi119 ceph-mon[32608]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "3ac89686-6d76-4cef-9089-ac29e4b3bcac"}]: dispatch 2024-04-23T10:22:16.742 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 23 10:22:16 smithi119 ceph-mon[32608]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "3ac89686-6d76-4cef-9089-ac29e4b3bcac"}]': finished 2024-04-23T10:22:16.742 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 23 10:22:16 smithi119 ceph-mon[32608]: osdmap e58: 9 total, 8 up, 9 in 2024-04-23T10:22:16.742 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 23 10:22:16 smithi119 ceph-mon[32608]: from='mgr.14152 172.21.15.114:0/1529706138' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 8}]: dispatch 2024-04-23T10:22:17.742 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 23 10:22:17 smithi119 ceph-mon[32608]: from='client.? 172.21.15.204:0/79120735' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-04-23T10:22:17.754 INFO:journalctl@ceph.mon.a.smithi114.stdout:Apr 23 10:22:17 smithi114 ceph-mon[29255]: from='client.? 172.21.15.204:0/79120735' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-04-23T10:22:17.829 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 23 10:22:17 smithi204 ceph-mon[32649]: from='client.? 172.21.15.204:0/79120735' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-04-23T10:22:18.742 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 23 10:22:18 smithi119 ceph-mon[32608]: pgmap v184: 1 pgs: 1 active+clean; 577 KiB data, 242 MiB used, 715 GiB / 715 GiB avail 2024-04-23T10:22:18.754 INFO:journalctl@ceph.mon.a.smithi114.stdout:Apr 23 10:22:18 smithi114 ceph-mon[29255]: pgmap v184: 1 pgs: 1 active+clean; 577 KiB data, 242 MiB used, 715 GiB / 715 GiB avail 2024-04-23T10:22:18.828 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 23 10:22:18 smithi204 ceph-mon[32649]: pgmap v184: 1 pgs: 1 active+clean; 577 KiB data, 242 MiB used, 715 GiB / 715 GiB avail 2024-04-23T10:22:21.242 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 23 10:22:20 smithi119 ceph-mon[32608]: pgmap v185: 1 pgs: 1 active+clean; 577 KiB data, 242 MiB used, 715 GiB / 715 GiB avail 2024-04-23T10:22:21.254 INFO:journalctl@ceph.mon.a.smithi114.stdout:Apr 23 10:22:20 smithi114 ceph-mon[29255]: pgmap v185: 1 pgs: 1 active+clean; 577 KiB data, 242 MiB used, 715 GiB / 715 GiB avail 2024-04-23T10:22:21.327 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 23 10:22:20 smithi204 ceph-mon[32649]: pgmap v185: 1 pgs: 1 active+clean; 577 KiB data, 242 MiB used, 715 GiB / 715 GiB avail 2024-04-23T10:22:23.242 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 23 10:22:22 smithi119 ceph-mon[32608]: pgmap v186: 1 pgs: 1 active+clean; 577 KiB data, 242 MiB used, 715 GiB / 715 GiB avail 2024-04-23T10:22:23.254 INFO:journalctl@ceph.mon.a.smithi114.stdout:Apr 23 10:22:22 smithi114 ceph-mon[29255]: pgmap v186: 1 pgs: 1 active+clean; 577 KiB data, 242 MiB used, 715 GiB / 715 GiB avail 2024-04-23T10:22:23.328 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 23 10:22:22 smithi204 ceph-mon[32649]: pgmap v186: 1 pgs: 1 active+clean; 577 KiB data, 242 MiB used, 715 GiB / 715 GiB avail 2024-04-23T10:22:24.242 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 23 10:22:23 smithi119 ceph-mon[32608]: from='mgr.14152 172.21.15.114:0/1529706138' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.8"}]: dispatch 2024-04-23T10:22:24.242 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 23 10:22:23 smithi119 ceph-mon[32608]: from='mgr.14152 172.21.15.114:0/1529706138' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-23T10:22:24.254 INFO:journalctl@ceph.mon.a.smithi114.stdout:Apr 23 10:22:23 smithi114 ceph-mon[29255]: from='mgr.14152 172.21.15.114:0/1529706138' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.8"}]: dispatch 2024-04-23T10:22:24.254 INFO:journalctl@ceph.mon.a.smithi114.stdout:Apr 23 10:22:23 smithi114 ceph-mon[29255]: from='mgr.14152 172.21.15.114:0/1529706138' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-23T10:22:24.283 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 23 10:22:23 smithi204 ceph-mon[32649]: from='mgr.14152 172.21.15.114:0/1529706138' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.8"}]: dispatch 2024-04-23T10:22:24.283 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 23 10:22:23 smithi204 ceph-mon[32649]: from='mgr.14152 172.21.15.114:0/1529706138' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-23T10:22:25.164 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 23 10:22:24 smithi204 ceph-mon[32649]: Deploying daemon osd.8 on smithi204 2024-04-23T10:22:25.164 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 23 10:22:24 smithi204 ceph-mon[32649]: pgmap v187: 1 pgs: 1 active+clean; 577 KiB data, 242 MiB used, 715 GiB / 715 GiB avail 2024-04-23T10:22:25.242 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 23 10:22:24 smithi119 ceph-mon[32608]: Deploying daemon osd.8 on smithi204 2024-04-23T10:22:25.242 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 23 10:22:24 smithi119 ceph-mon[32608]: pgmap v187: 1 pgs: 1 active+clean; 577 KiB data, 242 MiB used, 715 GiB / 715 GiB avail 2024-04-23T10:22:25.254 INFO:journalctl@ceph.mon.a.smithi114.stdout:Apr 23 10:22:24 smithi114 ceph-mon[29255]: Deploying daemon osd.8 on smithi204 2024-04-23T10:22:25.254 INFO:journalctl@ceph.mon.a.smithi114.stdout:Apr 23 10:22:24 smithi114 ceph-mon[29255]: pgmap v187: 1 pgs: 1 active+clean; 577 KiB data, 242 MiB used, 715 GiB / 715 GiB avail 2024-04-23T10:22:27.075 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 23 10:22:26 smithi204 ceph-mon[32649]: pgmap v188: 1 pgs: 1 active+clean; 577 KiB data, 242 MiB used, 715 GiB / 715 GiB avail 2024-04-23T10:22:27.254 INFO:journalctl@ceph.mon.a.smithi114.stdout:Apr 23 10:22:26 smithi114 ceph-mon[29255]: pgmap v188: 1 pgs: 1 active+clean; 577 KiB data, 242 MiB used, 715 GiB / 715 GiB avail 2024-04-23T10:22:27.492 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 23 10:22:26 smithi119 ceph-mon[32608]: pgmap v188: 1 pgs: 1 active+clean; 577 KiB data, 242 MiB used, 715 GiB / 715 GiB avail 2024-04-23T10:22:29.254 INFO:journalctl@ceph.mon.a.smithi114.stdout:Apr 23 10:22:28 smithi114 ceph-mon[29255]: pgmap v189: 1 pgs: 1 active+clean; 577 KiB data, 242 MiB used, 715 GiB / 715 GiB avail 2024-04-23T10:22:29.327 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 23 10:22:28 smithi204 ceph-mon[32649]: pgmap v189: 1 pgs: 1 active+clean; 577 KiB data, 242 MiB used, 715 GiB / 715 GiB avail 2024-04-23T10:22:29.492 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 23 10:22:28 smithi119 ceph-mon[32608]: pgmap v189: 1 pgs: 1 active+clean; 577 KiB data, 242 MiB used, 715 GiB / 715 GiB avail 2024-04-23T10:22:30.492 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 23 10:22:30 smithi119 ceph-mon[32608]: from='mgr.14152 172.21.15.114:0/1529706138' entity='mgr.x' 2024-04-23T10:22:30.492 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 23 10:22:30 smithi119 ceph-mon[32608]: from='mgr.14152 172.21.15.114:0/1529706138' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-23T10:22:30.492 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 23 10:22:30 smithi119 ceph-mon[32608]: from='mgr.14152 172.21.15.114:0/1529706138' entity='mgr.x' 2024-04-23T10:22:30.492 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 23 10:22:30 smithi119 ceph-mon[32608]: pgmap v190: 1 pgs: 1 active+clean; 577 KiB data, 242 MiB used, 715 GiB / 715 GiB avail 2024-04-23T10:22:30.504 INFO:journalctl@ceph.mon.a.smithi114.stdout:Apr 23 10:22:30 smithi114 ceph-mon[29255]: from='mgr.14152 172.21.15.114:0/1529706138' entity='mgr.x' 2024-04-23T10:22:30.504 INFO:journalctl@ceph.mon.a.smithi114.stdout:Apr 23 10:22:30 smithi114 ceph-mon[29255]: from='mgr.14152 172.21.15.114:0/1529706138' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-23T10:22:30.504 INFO:journalctl@ceph.mon.a.smithi114.stdout:Apr 23 10:22:30 smithi114 ceph-mon[29255]: from='mgr.14152 172.21.15.114:0/1529706138' entity='mgr.x' 2024-04-23T10:22:30.504 INFO:journalctl@ceph.mon.a.smithi114.stdout:Apr 23 10:22:30 smithi114 ceph-mon[29255]: pgmap v190: 1 pgs: 1 active+clean; 577 KiB data, 242 MiB used, 715 GiB / 715 GiB avail 2024-04-23T10:22:30.578 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 23 10:22:30 smithi204 ceph-mon[32649]: from='mgr.14152 172.21.15.114:0/1529706138' entity='mgr.x' 2024-04-23T10:22:30.578 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 23 10:22:30 smithi204 ceph-mon[32649]: from='mgr.14152 172.21.15.114:0/1529706138' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-23T10:22:30.578 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 23 10:22:30 smithi204 ceph-mon[32649]: from='mgr.14152 172.21.15.114:0/1529706138' entity='mgr.x' 2024-04-23T10:22:30.578 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 23 10:22:30 smithi204 ceph-mon[32649]: pgmap v190: 1 pgs: 1 active+clean; 577 KiB data, 242 MiB used, 715 GiB / 715 GiB avail 2024-04-23T10:22:32.300 INFO:teuthology.orchestra.run.smithi204.stdout:Created osd(s) 8 on host 'smithi204' 2024-04-23T10:22:32.300 INFO:teuthology.orchestra.run.smithi204.stderr:2024-04-23T10:22:32.299+0000 7fe7da7fc700 1 -- 172.21.15.204:0/708081393 <== mgr.14152 v2:172.21.15.114:6800/465333808 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+37 (secure 0 0 0) 0x7fe7e41089c0 con 0x7fe7d00a3c40 2024-04-23T10:22:32.302 INFO:teuthology.orchestra.run.smithi204.stderr:2024-04-23T10:22:32.302+0000 7fe7eb5ae700 1 -- 172.21.15.204:0/708081393 >> [v2:172.21.15.114:6800/465333808,v1:172.21.15.114:6801/465333808] conn(0x7fe7d00a3c40 msgr2=0x7fe7d00a6100 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T10:22:32.303 INFO:teuthology.orchestra.run.smithi204.stderr:2024-04-23T10:22:32.302+0000 7fe7eb5ae700 1 --2- 172.21.15.204:0/708081393 >> [v2:172.21.15.114:6800/465333808,v1:172.21.15.114:6801/465333808] conn(0x7fe7d00a3c40 0x7fe7d00a6100 secure :-1 s=READY pgs=100 cs=0 l=1 rev1=1 crypto rx=0x7fe7e4102a00 tx=0x7fe7d4009190 comp rx=0 tx=0).stop 2024-04-23T10:22:32.303 INFO:teuthology.orchestra.run.smithi204.stderr:2024-04-23T10:22:32.302+0000 7fe7eb5ae700 1 -- 172.21.15.204:0/708081393 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fe7e4103540 msgr2=0x7fe7e41a0050 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T10:22:32.303 INFO:teuthology.orchestra.run.smithi204.stderr:2024-04-23T10:22:32.302+0000 7fe7eb5ae700 1 --2- 172.21.15.204:0/708081393 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fe7e4103540 0x7fe7e41a0050 secure :-1 s=READY pgs=29 cs=0 l=1 rev1=1 crypto rx=0x7fe7dc00ada0 tx=0x7fe7dc00c330 comp rx=0 tx=0).stop 2024-04-23T10:22:32.303 INFO:teuthology.orchestra.run.smithi204.stderr:2024-04-23T10:22:32.302+0000 7fe7eb5ae700 1 -- 172.21.15.204:0/708081393 shutdown_connections 2024-04-23T10:22:32.303 INFO:teuthology.orchestra.run.smithi204.stderr:2024-04-23T10:22:32.302+0000 7fe7eb5ae700 1 --2- 172.21.15.204:0/708081393 >> [v2:172.21.15.114:6800/465333808,v1:172.21.15.114:6801/465333808] conn(0x7fe7d00a3c40 0x7fe7d00a6100 unknown :-1 s=CLOSED pgs=100 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:22:32.303 INFO:teuthology.orchestra.run.smithi204.stderr:2024-04-23T10:22:32.302+0000 7fe7eb5ae700 1 --2- 172.21.15.204:0/708081393 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fe7e4103540 0x7fe7e41a0050 unknown :-1 s=CLOSED pgs=29 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:22:32.303 INFO:teuthology.orchestra.run.smithi204.stderr:2024-04-23T10:22:32.302+0000 7fe7eb5ae700 1 --2- 172.21.15.204:0/708081393 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fe7e41019a0 0x7fe7e406b250 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:22:32.303 INFO:teuthology.orchestra.run.smithi204.stderr:2024-04-23T10:22:32.302+0000 7fe7eb5ae700 1 --2- 172.21.15.204:0/708081393 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fe7e4102ba0 0x7fe7e419da60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:22:32.303 INFO:teuthology.orchestra.run.smithi204.stderr:2024-04-23T10:22:32.302+0000 7fe7eb5ae700 1 -- 172.21.15.204:0/708081393 >> 172.21.15.204:0/708081393 conn(0x7fe7e40fcf40 msgr2=0x7fe7e4106010 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T10:22:32.303 INFO:teuthology.orchestra.run.smithi204.stderr:2024-04-23T10:22:32.302+0000 7fe7eb5ae700 1 -- 172.21.15.204:0/708081393 shutdown_connections 2024-04-23T10:22:32.303 INFO:teuthology.orchestra.run.smithi204.stderr:2024-04-23T10:22:32.302+0000 7fe7eb5ae700 1 -- 172.21.15.204:0/708081393 wait complete. 2024-04-23T10:22:32.887 DEBUG:teuthology.orchestra.run.smithi204:osd.8> sudo journalctl -f -n 0 -u ceph-aa240572-0159-11ef-bc93-c7b262605968@osd.8.service 2024-04-23T10:22:32.890 INFO:tasks.cephadm:Deploying osd.9 on smithi114 with /dev/vg_nvme/lv_1... 2024-04-23T10:22:32.890 DEBUG:teuthology.orchestra.run.smithi114:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d69f13d827952bb07bb9b7973bca14f97d764d6f shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid aa240572-0159-11ef-bc93-c7b262605968 -- ceph-volume lvm zap /dev/vg_nvme/lv_1 2024-04-23T10:22:33.140 INFO:teuthology.orchestra.run.smithi114.stderr:Inferring config /var/lib/ceph/aa240572-0159-11ef-bc93-c7b262605968/mon.a/config 2024-04-23T10:22:33.242 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 23 10:22:32 smithi119 ceph-mon[32608]: pgmap v191: 1 pgs: 1 active+clean; 577 KiB data, 242 MiB used, 715 GiB / 715 GiB avail 2024-04-23T10:22:33.242 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 23 10:22:32 smithi119 ceph-mon[32608]: from='mgr.14152 172.21.15.114:0/1529706138' entity='mgr.x' 2024-04-23T10:22:33.242 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 23 10:22:32 smithi119 ceph-mon[32608]: from='mgr.14152 172.21.15.114:0/1529706138' entity='mgr.x' 2024-04-23T10:22:33.254 INFO:journalctl@ceph.mon.a.smithi114.stdout:Apr 23 10:22:32 smithi114 ceph-mon[29255]: pgmap v191: 1 pgs: 1 active+clean; 577 KiB data, 242 MiB used, 715 GiB / 715 GiB avail 2024-04-23T10:22:33.254 INFO:journalctl@ceph.mon.a.smithi114.stdout:Apr 23 10:22:32 smithi114 ceph-mon[29255]: from='mgr.14152 172.21.15.114:0/1529706138' entity='mgr.x' 2024-04-23T10:22:33.254 INFO:journalctl@ceph.mon.a.smithi114.stdout:Apr 23 10:22:32 smithi114 ceph-mon[29255]: from='mgr.14152 172.21.15.114:0/1529706138' entity='mgr.x' 2024-04-23T10:22:33.328 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 23 10:22:32 smithi204 ceph-mon[32649]: pgmap v191: 1 pgs: 1 active+clean; 577 KiB data, 242 MiB used, 715 GiB / 715 GiB avail 2024-04-23T10:22:33.328 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 23 10:22:32 smithi204 ceph-mon[32649]: from='mgr.14152 172.21.15.114:0/1529706138' entity='mgr.x' 2024-04-23T10:22:33.328 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 23 10:22:32 smithi204 ceph-mon[32649]: from='mgr.14152 172.21.15.114:0/1529706138' entity='mgr.x' 2024-04-23T10:22:33.329 INFO:journalctl@ceph.osd.8.smithi204.stdout:Apr 23 10:22:33 smithi204 ceph-aa240572-0159-11ef-bc93-c7b262605968-osd-8[51050]: 2024-04-23T10:22:33.059+0000 7f8d7767a7c0 -1 osd.8 0 log_to_monitors true 2024-04-23T10:22:34.242 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 23 10:22:33 smithi119 ceph-mon[32608]: from='osd.8 [v2:172.21.15.204:6816/1426318231,v1:172.21.15.204:6817/1426318231]' entity='osd.8' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["8"]}]: dispatch 2024-04-23T10:22:34.243 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 23 10:22:33 smithi119 ceph-mon[32608]: from='osd.8 ' entity='osd.8' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["8"]}]: dispatch 2024-04-23T10:22:34.243 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 23 10:22:33 smithi119 ceph-mon[32608]: from='mgr.14152 172.21.15.114:0/1529706138' entity='mgr.x' 2024-04-23T10:22:34.243 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 23 10:22:33 smithi119 ceph-mon[32608]: from='mgr.14152 172.21.15.114:0/1529706138' entity='mgr.x' 2024-04-23T10:22:34.254 INFO:journalctl@ceph.mon.a.smithi114.stdout:Apr 23 10:22:33 smithi114 ceph-mon[29255]: from='osd.8 [v2:172.21.15.204:6816/1426318231,v1:172.21.15.204:6817/1426318231]' entity='osd.8' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["8"]}]: dispatch 2024-04-23T10:22:34.254 INFO:journalctl@ceph.mon.a.smithi114.stdout:Apr 23 10:22:33 smithi114 ceph-mon[29255]: from='osd.8 ' entity='osd.8' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["8"]}]: dispatch 2024-04-23T10:22:34.254 INFO:journalctl@ceph.mon.a.smithi114.stdout:Apr 23 10:22:33 smithi114 ceph-mon[29255]: from='mgr.14152 172.21.15.114:0/1529706138' entity='mgr.x' 2024-04-23T10:22:34.254 INFO:journalctl@ceph.mon.a.smithi114.stdout:Apr 23 10:22:33 smithi114 ceph-mon[29255]: from='mgr.14152 172.21.15.114:0/1529706138' entity='mgr.x' 2024-04-23T10:22:34.273 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 23 10:22:33 smithi204 ceph-mon[32649]: from='osd.8 [v2:172.21.15.204:6816/1426318231,v1:172.21.15.204:6817/1426318231]' entity='osd.8' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["8"]}]: dispatch 2024-04-23T10:22:34.274 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 23 10:22:33 smithi204 ceph-mon[32649]: from='osd.8 ' entity='osd.8' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["8"]}]: dispatch 2024-04-23T10:22:34.274 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 23 10:22:33 smithi204 ceph-mon[32649]: from='mgr.14152 172.21.15.114:0/1529706138' entity='mgr.x' 2024-04-23T10:22:34.274 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 23 10:22:33 smithi204 ceph-mon[32649]: from='mgr.14152 172.21.15.114:0/1529706138' entity='mgr.x' 2024-04-23T10:22:35.108 INFO:teuthology.orchestra.run.smithi114.stderr:--> Zapping: /dev/vg_nvme/lv_1 2024-04-23T10:22:35.108 INFO:teuthology.orchestra.run.smithi114.stderr:Running command: /usr/bin/dd if=/dev/zero of=/dev/vg_nvme/lv_1 bs=1M count=10 conv=fsync 2024-04-23T10:22:35.108 INFO:teuthology.orchestra.run.smithi114.stderr: stderr: 10+0 records in 2024-04-23T10:22:35.108 INFO:teuthology.orchestra.run.smithi114.stderr:10+0 records out 2024-04-23T10:22:35.108 INFO:teuthology.orchestra.run.smithi114.stderr: stderr: 10485760 bytes (10 MB, 10 MiB) copied, 0.0114415 s, 916 MB/s 2024-04-23T10:22:35.108 INFO:teuthology.orchestra.run.smithi114.stderr:--> Zapping successful for: 2024-04-23T10:22:35.254 INFO:journalctl@ceph.mon.a.smithi114.stdout:Apr 23 10:22:34 smithi114 ceph-mon[29255]: pgmap v192: 1 pgs: 1 active+clean; 577 KiB data, 242 MiB used, 715 GiB / 715 GiB avail 2024-04-23T10:22:35.255 INFO:journalctl@ceph.mon.a.smithi114.stdout:Apr 23 10:22:34 smithi114 ceph-mon[29255]: from='osd.8 ' entity='osd.8' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["8"]}]': finished 2024-04-23T10:22:35.255 INFO:journalctl@ceph.mon.a.smithi114.stdout:Apr 23 10:22:34 smithi114 ceph-mon[29255]: osdmap e59: 9 total, 8 up, 9 in 2024-04-23T10:22:35.255 INFO:journalctl@ceph.mon.a.smithi114.stdout:Apr 23 10:22:34 smithi114 ceph-mon[29255]: from='mgr.14152 172.21.15.114:0/1529706138' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 8}]: dispatch 2024-04-23T10:22:35.255 INFO:journalctl@ceph.mon.a.smithi114.stdout:Apr 23 10:22:34 smithi114 ceph-mon[29255]: from='osd.8 [v2:172.21.15.204:6816/1426318231,v1:172.21.15.204:6817/1426318231]' entity='osd.8' cmd=[{"prefix": "osd crush create-or-move", "id": 8, "weight":0.0873, "args": ["host=smithi204", "root=default"]}]: dispatch 2024-04-23T10:22:35.255 INFO:journalctl@ceph.mon.a.smithi114.stdout:Apr 23 10:22:34 smithi114 ceph-mon[29255]: from='osd.8 ' entity='osd.8' cmd=[{"prefix": "osd crush create-or-move", "id": 8, "weight":0.0873, "args": ["host=smithi204", "root=default"]}]: dispatch 2024-04-23T10:22:35.291 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 23 10:22:34 smithi204 ceph-mon[32649]: pgmap v192: 1 pgs: 1 active+clean; 577 KiB data, 242 MiB used, 715 GiB / 715 GiB avail 2024-04-23T10:22:35.291 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 23 10:22:34 smithi204 ceph-mon[32649]: from='osd.8 ' entity='osd.8' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["8"]}]': finished 2024-04-23T10:22:35.291 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 23 10:22:34 smithi204 ceph-mon[32649]: osdmap e59: 9 total, 8 up, 9 in 2024-04-23T10:22:35.291 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 23 10:22:34 smithi204 ceph-mon[32649]: from='mgr.14152 172.21.15.114:0/1529706138' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 8}]: dispatch 2024-04-23T10:22:35.291 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 23 10:22:34 smithi204 ceph-mon[32649]: from='osd.8 [v2:172.21.15.204:6816/1426318231,v1:172.21.15.204:6817/1426318231]' entity='osd.8' cmd=[{"prefix": "osd crush create-or-move", "id": 8, "weight":0.0873, "args": ["host=smithi204", "root=default"]}]: dispatch 2024-04-23T10:22:35.291 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 23 10:22:34 smithi204 ceph-mon[32649]: from='osd.8 ' entity='osd.8' cmd=[{"prefix": "osd crush create-or-move", "id": 8, "weight":0.0873, "args": ["host=smithi204", "root=default"]}]: dispatch 2024-04-23T10:22:35.492 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 23 10:22:34 smithi119 ceph-mon[32608]: pgmap v192: 1 pgs: 1 active+clean; 577 KiB data, 242 MiB used, 715 GiB / 715 GiB avail 2024-04-23T10:22:35.492 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 23 10:22:34 smithi119 ceph-mon[32608]: from='osd.8 ' entity='osd.8' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["8"]}]': finished 2024-04-23T10:22:35.492 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 23 10:22:34 smithi119 ceph-mon[32608]: osdmap e59: 9 total, 8 up, 9 in 2024-04-23T10:22:35.492 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 23 10:22:34 smithi119 ceph-mon[32608]: from='mgr.14152 172.21.15.114:0/1529706138' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 8}]: dispatch 2024-04-23T10:22:35.492 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 23 10:22:34 smithi119 ceph-mon[32608]: from='osd.8 [v2:172.21.15.204:6816/1426318231,v1:172.21.15.204:6817/1426318231]' entity='osd.8' cmd=[{"prefix": "osd crush create-or-move", "id": 8, "weight":0.0873, "args": ["host=smithi204", "root=default"]}]: dispatch 2024-04-23T10:22:35.492 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 23 10:22:34 smithi119 ceph-mon[32608]: from='osd.8 ' entity='osd.8' cmd=[{"prefix": "osd crush create-or-move", "id": 8, "weight":0.0873, "args": ["host=smithi204", "root=default"]}]: dispatch 2024-04-23T10:22:35.843 DEBUG:teuthology.orchestra.run.smithi114:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d69f13d827952bb07bb9b7973bca14f97d764d6f shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid aa240572-0159-11ef-bc93-c7b262605968 -- ceph orch daemon add osd smithi114:vg_nvme/lv_1 2024-04-23T10:22:36.087 INFO:teuthology.orchestra.run.smithi114.stderr:Inferring config /var/lib/ceph/aa240572-0159-11ef-bc93-c7b262605968/mon.a/config 2024-04-23T10:22:36.154 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 23 10:22:36 smithi204 ceph-mon[32649]: from='osd.8 ' entity='osd.8' cmd='[{"prefix": "osd crush create-or-move", "id": 8, "weight":0.0873, "args": ["host=smithi204", "root=default"]}]': finished 2024-04-23T10:22:36.154 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 23 10:22:36 smithi204 ceph-mon[32649]: osdmap e60: 9 total, 8 up, 9 in 2024-04-23T10:22:36.154 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 23 10:22:36 smithi204 ceph-mon[32649]: from='mgr.14152 172.21.15.114:0/1529706138' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 8}]: dispatch 2024-04-23T10:22:36.154 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 23 10:22:36 smithi204 ceph-mon[32649]: from='mgr.14152 172.21.15.114:0/1529706138' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 8}]: dispatch 2024-04-23T10:22:36.154 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 23 10:22:36 smithi204 ceph-mon[32649]: from='osd.8 ' entity='osd.8' 2024-04-23T10:22:36.154 INFO:journalctl@ceph.osd.8.smithi204.stdout:Apr 23 10:22:35 smithi204 ceph-aa240572-0159-11ef-bc93-c7b262605968-osd-8[51050]: 2024-04-23T10:22:35.896+0000 7f8d6ccdf700 -1 osd.8 0 waiting for initial osdmap 2024-04-23T10:22:36.155 INFO:journalctl@ceph.osd.8.smithi204.stdout:Apr 23 10:22:36 smithi204 ceph-aa240572-0159-11ef-bc93-c7b262605968-osd-8[51050]: 2024-04-23T10:22:36.075+0000 7f8d64aca700 -1 osd.8 61 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-04-23T10:22:36.208 INFO:journalctl@ceph.mon.a.smithi114.stdout:Apr 23 10:22:36 smithi114 ceph-mon[29255]: from='osd.8 ' entity='osd.8' cmd='[{"prefix": "osd crush create-or-move", "id": 8, "weight":0.0873, "args": ["host=smithi204", "root=default"]}]': finished 2024-04-23T10:22:36.208 INFO:journalctl@ceph.mon.a.smithi114.stdout:Apr 23 10:22:36 smithi114 ceph-mon[29255]: osdmap e60: 9 total, 8 up, 9 in 2024-04-23T10:22:36.208 INFO:journalctl@ceph.mon.a.smithi114.stdout:Apr 23 10:22:36 smithi114 ceph-mon[29255]: from='mgr.14152 172.21.15.114:0/1529706138' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 8}]: dispatch 2024-04-23T10:22:36.208 INFO:journalctl@ceph.mon.a.smithi114.stdout:Apr 23 10:22:36 smithi114 ceph-mon[29255]: from='mgr.14152 172.21.15.114:0/1529706138' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 8}]: dispatch 2024-04-23T10:22:36.208 INFO:journalctl@ceph.mon.a.smithi114.stdout:Apr 23 10:22:36 smithi114 ceph-mon[29255]: from='osd.8 ' entity='osd.8' 2024-04-23T10:22:36.242 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 23 10:22:36 smithi119 ceph-mon[32608]: from='osd.8 ' entity='osd.8' cmd='[{"prefix": "osd crush create-or-move", "id": 8, "weight":0.0873, "args": ["host=smithi204", "root=default"]}]': finished 2024-04-23T10:22:36.242 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 23 10:22:36 smithi119 ceph-mon[32608]: osdmap e60: 9 total, 8 up, 9 in 2024-04-23T10:22:36.242 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 23 10:22:36 smithi119 ceph-mon[32608]: from='mgr.14152 172.21.15.114:0/1529706138' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 8}]: dispatch 2024-04-23T10:22:36.242 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 23 10:22:36 smithi119 ceph-mon[32608]: from='mgr.14152 172.21.15.114:0/1529706138' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 8}]: dispatch 2024-04-23T10:22:36.242 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 23 10:22:36 smithi119 ceph-mon[32608]: from='osd.8 ' entity='osd.8' 2024-04-23T10:22:37.078 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 23 10:22:37 smithi204 ceph-mon[32649]: purged_snaps scrub starts 2024-04-23T10:22:37.078 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 23 10:22:37 smithi204 ceph-mon[32649]: purged_snaps scrub ok 2024-04-23T10:22:37.078 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 23 10:22:37 smithi204 ceph-mon[32649]: pgmap v195: 1 pgs: 1 active+clean; 577 KiB data, 242 MiB used, 715 GiB / 715 GiB avail 2024-04-23T10:22:37.078 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 23 10:22:37 smithi204 ceph-mon[32649]: osdmap e61: 9 total, 8 up, 9 in 2024-04-23T10:22:37.078 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 23 10:22:37 smithi204 ceph-mon[32649]: from='mgr.14152 172.21.15.114:0/1529706138' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 8}]: dispatch 2024-04-23T10:22:37.078 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 23 10:22:37 smithi204 ceph-mon[32649]: from='mgr.14152 172.21.15.114:0/1529706138' entity='mgr.x' 2024-04-23T10:22:37.078 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 23 10:22:37 smithi204 ceph-mon[32649]: from='mgr.14152 172.21.15.114:0/1529706138' entity='mgr.x' 2024-04-23T10:22:37.079 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 23 10:22:37 smithi204 ceph-mon[32649]: from='mgr.14152 172.21.15.114:0/1529706138' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-04-23T10:22:37.079 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 23 10:22:37 smithi204 ceph-mon[32649]: from='mgr.14152 172.21.15.114:0/1529706138' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-04-23T10:22:37.079 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 23 10:22:37 smithi204 ceph-mon[32649]: from='mgr.14152 172.21.15.114:0/1529706138' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2024-04-23T10:22:37.079 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 23 10:22:37 smithi204 ceph-mon[32649]: from='mgr.14152 172.21.15.114:0/1529706138' entity='mgr.x' 2024-04-23T10:22:37.079 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 23 10:22:37 smithi204 ceph-mon[32649]: from='mgr.14152 172.21.15.114:0/1529706138' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-23T10:22:37.079 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 23 10:22:37 smithi204 ceph-mon[32649]: from='mgr.14152 172.21.15.114:0/1529706138' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-23T10:22:37.079 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 23 10:22:37 smithi204 ceph-mon[32649]: from='mgr.14152 172.21.15.114:0/1529706138' entity='mgr.x' 2024-04-23T10:22:37.079 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 23 10:22:37 smithi204 ceph-mon[32649]: from='mgr.14152 172.21.15.114:0/1529706138' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 8}]: dispatch 2024-04-23T10:22:37.254 INFO:journalctl@ceph.mon.a.smithi114.stdout:Apr 23 10:22:37 smithi114 ceph-mon[29255]: purged_snaps scrub starts 2024-04-23T10:22:37.254 INFO:journalctl@ceph.mon.a.smithi114.stdout:Apr 23 10:22:37 smithi114 ceph-mon[29255]: purged_snaps scrub ok 2024-04-23T10:22:37.254 INFO:journalctl@ceph.mon.a.smithi114.stdout:Apr 23 10:22:37 smithi114 ceph-mon[29255]: pgmap v195: 1 pgs: 1 active+clean; 577 KiB data, 242 MiB used, 715 GiB / 715 GiB avail 2024-04-23T10:22:37.254 INFO:journalctl@ceph.mon.a.smithi114.stdout:Apr 23 10:22:37 smithi114 ceph-mon[29255]: osdmap e61: 9 total, 8 up, 9 in 2024-04-23T10:22:37.255 INFO:journalctl@ceph.mon.a.smithi114.stdout:Apr 23 10:22:37 smithi114 ceph-mon[29255]: from='mgr.14152 172.21.15.114:0/1529706138' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 8}]: dispatch 2024-04-23T10:22:37.255 INFO:journalctl@ceph.mon.a.smithi114.stdout:Apr 23 10:22:37 smithi114 ceph-mon[29255]: from='mgr.14152 172.21.15.114:0/1529706138' entity='mgr.x' 2024-04-23T10:22:37.255 INFO:journalctl@ceph.mon.a.smithi114.stdout:Apr 23 10:22:37 smithi114 ceph-mon[29255]: from='mgr.14152 172.21.15.114:0/1529706138' entity='mgr.x' 2024-04-23T10:22:37.255 INFO:journalctl@ceph.mon.a.smithi114.stdout:Apr 23 10:22:37 smithi114 ceph-mon[29255]: from='mgr.14152 172.21.15.114:0/1529706138' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-04-23T10:22:37.255 INFO:journalctl@ceph.mon.a.smithi114.stdout:Apr 23 10:22:37 smithi114 ceph-mon[29255]: from='mgr.14152 172.21.15.114:0/1529706138' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-04-23T10:22:37.255 INFO:journalctl@ceph.mon.a.smithi114.stdout:Apr 23 10:22:37 smithi114 ceph-mon[29255]: from='mgr.14152 172.21.15.114:0/1529706138' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2024-04-23T10:22:37.255 INFO:journalctl@ceph.mon.a.smithi114.stdout:Apr 23 10:22:37 smithi114 ceph-mon[29255]: from='mgr.14152 172.21.15.114:0/1529706138' entity='mgr.x' 2024-04-23T10:22:37.255 INFO:journalctl@ceph.mon.a.smithi114.stdout:Apr 23 10:22:37 smithi114 ceph-mon[29255]: from='mgr.14152 172.21.15.114:0/1529706138' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-23T10:22:37.255 INFO:journalctl@ceph.mon.a.smithi114.stdout:Apr 23 10:22:37 smithi114 ceph-mon[29255]: from='mgr.14152 172.21.15.114:0/1529706138' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-23T10:22:37.255 INFO:journalctl@ceph.mon.a.smithi114.stdout:Apr 23 10:22:37 smithi114 ceph-mon[29255]: from='mgr.14152 172.21.15.114:0/1529706138' entity='mgr.x' 2024-04-23T10:22:37.255 INFO:journalctl@ceph.mon.a.smithi114.stdout:Apr 23 10:22:37 smithi114 ceph-mon[29255]: from='mgr.14152 172.21.15.114:0/1529706138' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 8}]: dispatch 2024-04-23T10:22:37.492 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 23 10:22:37 smithi119 ceph-mon[32608]: purged_snaps scrub starts 2024-04-23T10:22:37.492 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 23 10:22:37 smithi119 ceph-mon[32608]: purged_snaps scrub ok 2024-04-23T10:22:37.492 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 23 10:22:37 smithi119 ceph-mon[32608]: pgmap v195: 1 pgs: 1 active+clean; 577 KiB data, 242 MiB used, 715 GiB / 715 GiB avail 2024-04-23T10:22:37.492 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 23 10:22:37 smithi119 ceph-mon[32608]: osdmap e61: 9 total, 8 up, 9 in 2024-04-23T10:22:37.492 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 23 10:22:37 smithi119 ceph-mon[32608]: from='mgr.14152 172.21.15.114:0/1529706138' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 8}]: dispatch 2024-04-23T10:22:37.492 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 23 10:22:37 smithi119 ceph-mon[32608]: from='mgr.14152 172.21.15.114:0/1529706138' entity='mgr.x' 2024-04-23T10:22:37.492 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 23 10:22:37 smithi119 ceph-mon[32608]: from='mgr.14152 172.21.15.114:0/1529706138' entity='mgr.x' 2024-04-23T10:22:37.492 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 23 10:22:37 smithi119 ceph-mon[32608]: from='mgr.14152 172.21.15.114:0/1529706138' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-04-23T10:22:37.492 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 23 10:22:37 smithi119 ceph-mon[32608]: from='mgr.14152 172.21.15.114:0/1529706138' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-04-23T10:22:37.493 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 23 10:22:37 smithi119 ceph-mon[32608]: from='mgr.14152 172.21.15.114:0/1529706138' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2024-04-23T10:22:37.493 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 23 10:22:37 smithi119 ceph-mon[32608]: from='mgr.14152 172.21.15.114:0/1529706138' entity='mgr.x' 2024-04-23T10:22:37.493 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 23 10:22:37 smithi119 ceph-mon[32608]: from='mgr.14152 172.21.15.114:0/1529706138' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-23T10:22:37.493 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 23 10:22:37 smithi119 ceph-mon[32608]: from='mgr.14152 172.21.15.114:0/1529706138' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-23T10:22:37.493 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 23 10:22:37 smithi119 ceph-mon[32608]: from='mgr.14152 172.21.15.114:0/1529706138' entity='mgr.x' 2024-04-23T10:22:37.493 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 23 10:22:37 smithi119 ceph-mon[32608]: from='mgr.14152 172.21.15.114:0/1529706138' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 8}]: dispatch 2024-04-23T10:22:37.823 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:22:37.823+0000 7fc6799c8700 1 -- 172.21.15.114:0/1276498488 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fc6741019c0 msgr2=0x7fc674101de0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T10:22:37.823 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:22:37.823+0000 7fc6799c8700 1 --2- 172.21.15.114:0/1276498488 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fc6741019c0 0x7fc674101de0 secure :-1 s=READY pgs=140 cs=0 l=1 rev1=1 crypto rx=0x7fc6640099c0 tx=0x7fc664009cd0 comp rx=0 tx=0).stop 2024-04-23T10:22:37.823 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:22:37.823+0000 7fc6799c8700 1 -- 172.21.15.114:0/1276498488 shutdown_connections 2024-04-23T10:22:37.823 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:22:37.823+0000 7fc6799c8700 1 --2- 172.21.15.114:0/1276498488 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fc674103560 0x7fc674109e10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:22:37.823 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:22:37.823+0000 7fc6799c8700 1 --2- 172.21.15.114:0/1276498488 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fc6741019c0 0x7fc674101de0 unknown :-1 s=CLOSED pgs=140 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:22:37.823 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:22:37.823+0000 7fc6799c8700 1 --2- 172.21.15.114:0/1276498488 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fc674102bc0 0x7fc674103020 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:22:37.823 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:22:37.823+0000 7fc6799c8700 1 -- 172.21.15.114:0/1276498488 >> 172.21.15.114:0/1276498488 conn(0x7fc6740fcfa0 msgr2=0x7fc6740ff3c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T10:22:37.823 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:22:37.823+0000 7fc6799c8700 1 -- 172.21.15.114:0/1276498488 shutdown_connections 2024-04-23T10:22:37.824 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:22:37.823+0000 7fc6799c8700 1 -- 172.21.15.114:0/1276498488 wait complete. 2024-04-23T10:22:37.824 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:22:37.824+0000 7fc6799c8700 1 Processor -- start 2024-04-23T10:22:37.824 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:22:37.824+0000 7fc6799c8700 1 -- start start 2024-04-23T10:22:37.825 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:22:37.824+0000 7fc6799c8700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fc6741019c0 0x7fc674198c70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T10:22:37.825 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:22:37.824+0000 7fc6799c8700 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fc674102bc0 0x7fc6741991b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T10:22:37.826 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:22:37.824+0000 7fc6799c8700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fc674103560 0x7fc6741a01f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T10:22:37.826 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:22:37.824+0000 7fc6799c8700 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x7fc67410e5d0 con 0x7fc6741019c0 2024-04-23T10:22:37.826 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:22:37.824+0000 7fc6799c8700 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 -- 0x7fc67410e450 con 0x7fc674103560 2024-04-23T10:22:37.826 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:22:37.824+0000 7fc6799c8700 1 -- --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_getmap magic: 0 -- 0x7fc67410e750 con 0x7fc674102bc0 2024-04-23T10:22:37.826 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:22:37.825+0000 7fc672ffd700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fc6741019c0 0x7fc674198c70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T10:22:37.826 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:22:37.825+0000 7fc672ffd700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fc6741019c0 0x7fc674198c70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.114:51950/0 (socket says 172.21.15.114:51950) 2024-04-23T10:22:37.826 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:22:37.825+0000 7fc672ffd700 1 -- 172.21.15.114:0/3759512728 learned_addr learned my addr 172.21.15.114:0/3759512728 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-04-23T10:22:37.826 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:22:37.825+0000 7fc6727fc700 1 --2- 172.21.15.114:0/3759512728 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fc674102bc0 0x7fc6741991b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T10:22:37.826 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:22:37.825+0000 7fc6737fe700 1 --2- 172.21.15.114:0/3759512728 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fc674103560 0x7fc6741a01f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T10:22:37.826 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:22:37.825+0000 7fc672ffd700 1 -- 172.21.15.114:0/3759512728 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fc674102bc0 msgr2=0x7fc6741991b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T10:22:37.826 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:22:37.825+0000 7fc672ffd700 1 --2- 172.21.15.114:0/3759512728 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fc674102bc0 0x7fc6741991b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:22:37.826 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:22:37.825+0000 7fc672ffd700 1 -- 172.21.15.114:0/3759512728 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fc674103560 msgr2=0x7fc6741a01f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T10:22:37.826 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:22:37.825+0000 7fc672ffd700 1 --2- 172.21.15.114:0/3759512728 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fc674103560 0x7fc6741a01f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:22:37.826 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:22:37.825+0000 7fc672ffd700 1 -- 172.21.15.114:0/3759512728 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc6741a08c0 con 0x7fc6741019c0 2024-04-23T10:22:37.827 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:22:37.825+0000 7fc6737fe700 1 --2- 172.21.15.114:0/3759512728 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fc674103560 0x7fc6741a01f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-23T10:22:37.827 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:22:37.825+0000 7fc672ffd700 1 --2- 172.21.15.114:0/3759512728 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fc6741019c0 0x7fc674198c70 secure :-1 s=READY pgs=141 cs=0 l=1 rev1=1 crypto rx=0x7fc6640038c0 tx=0x7fc6640047b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T10:22:37.827 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:22:37.825+0000 7fc6789c6700 1 -- 172.21.15.114:0/3759512728 <== mon.0 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc664027070 con 0x7fc6741019c0 2024-04-23T10:22:37.827 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:22:37.825+0000 7fc6789c6700 1 -- 172.21.15.114:0/3759512728 <== mon.0 v2:172.21.15.114:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fc664004160 con 0x7fc6741019c0 2024-04-23T10:22:37.827 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:22:37.825+0000 7fc6799c8700 1 -- 172.21.15.114:0/3759512728 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fc6741a0b50 con 0x7fc6741019c0 2024-04-23T10:22:37.827 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:22:37.825+0000 7fc6789c6700 1 -- 172.21.15.114:0/3759512728 <== mon.0 v2:172.21.15.114:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc664022e20 con 0x7fc6741019c0 2024-04-23T10:22:37.827 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:22:37.825+0000 7fc6799c8700 1 -- 172.21.15.114:0/3759512728 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fc6741a1030 con 0x7fc6741019c0 2024-04-23T10:22:37.827 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:22:37.826+0000 7fc6799c8700 1 -- 172.21.15.114:0/3759512728 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fc674069730 con 0x7fc6741019c0 2024-04-23T10:22:37.828 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:22:37.828+0000 7fc6789c6700 1 -- 172.21.15.114:0/3759512728 <== mon.0 v2:172.21.15.114:3300/0 4 ==== mgrmap(e 16) ==== 138647+0+0 (secure 0 0 0) 0x7fc66402a070 con 0x7fc6741019c0 2024-04-23T10:22:37.828 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:22:37.829+0000 7fc6789c6700 1 --2- 172.21.15.114:0/3759512728 >> [v2:172.21.15.114:6800/465333808,v1:172.21.15.114:6801/465333808] conn(0x7fc65c0a3c40 0x7fc65c0a6100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T10:22:37.828 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:22:37.829+0000 7fc6727fc700 1 --2- 172.21.15.114:0/3759512728 >> [v2:172.21.15.114:6800/465333808,v1:172.21.15.114:6801/465333808] conn(0x7fc65c0a3c40 0x7fc65c0a6100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T10:22:37.828 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:22:37.829+0000 7fc6789c6700 1 -- 172.21.15.114:0/3759512728 <== mon.0 v2:172.21.15.114:3300/0 5 ==== osd_map(62..62 src has 1..62) ==== 5862+0+0 (secure 0 0 0) 0x7fc6640d06f0 con 0x7fc6741019c0 2024-04-23T10:22:37.828 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:22:37.829+0000 7fc6727fc700 1 --2- 172.21.15.114:0/3759512728 >> [v2:172.21.15.114:6800/465333808,v1:172.21.15.114:6801/465333808] conn(0x7fc65c0a3c40 0x7fc65c0a6100 secure :-1 s=READY pgs=106 cs=0 l=1 rev1=1 crypto rx=0x7fc668005950 tx=0x7fc66800b410 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T10:22:37.831 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:22:37.832+0000 7fc6789c6700 1 -- 172.21.15.114:0/3759512728 <== mon.0 v2:172.21.15.114:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7fc664013070 con 0x7fc6741019c0 2024-04-23T10:22:38.047 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:22:38.048+0000 7fc6799c8700 1 -- 172.21.15.114:0/3759512728 --> [v2:172.21.15.114:6800/465333808,v1:172.21.15.114:6801/465333808] -- mgr_command(tid 0: {"prefix": "orch daemon add osd", "svc_arg": "smithi114:vg_nvme/lv_1", "target": ["mon-mgr", ""]}) -- 0x7fc674062e80 con 0x7fc65c0a3c40 2024-04-23T10:22:38.328 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 23 10:22:38 smithi204 ceph-mon[32649]: Detected new or changed devices on smithi204 2024-04-23T10:22:38.328 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 23 10:22:38 smithi204 ceph-mon[32649]: Adjusting osd_memory_target on smithi204 to 5696M 2024-04-23T10:22:38.328 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 23 10:22:38 smithi204 ceph-mon[32649]: osd.8 [v2:172.21.15.204:6816/1426318231,v1:172.21.15.204:6817/1426318231] boot 2024-04-23T10:22:38.328 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 23 10:22:38 smithi204 ceph-mon[32649]: osdmap e62: 9 total, 9 up, 9 in 2024-04-23T10:22:38.328 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 23 10:22:38 smithi204 ceph-mon[32649]: from='mgr.14152 172.21.15.114:0/1529706138' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 8}]: dispatch 2024-04-23T10:22:38.328 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 23 10:22:38 smithi204 ceph-mon[32649]: from='mgr.14152 172.21.15.114:0/1529706138' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-23T10:22:38.328 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 23 10:22:38 smithi204 ceph-mon[32649]: from='mgr.14152 172.21.15.114:0/1529706138' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-23T10:22:38.328 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 23 10:22:38 smithi204 ceph-mon[32649]: from='mgr.14152 172.21.15.114:0/1529706138' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-23T10:22:38.329 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 23 10:22:38 smithi204 ceph-mon[32649]: from='mgr.14152 172.21.15.114:0/1529706138' entity='mgr.x' 2024-04-23T10:22:38.492 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 23 10:22:38 smithi119 ceph-mon[32608]: Detected new or changed devices on smithi204 2024-04-23T10:22:38.492 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 23 10:22:38 smithi119 ceph-mon[32608]: Adjusting osd_memory_target on smithi204 to 5696M 2024-04-23T10:22:38.492 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 23 10:22:38 smithi119 ceph-mon[32608]: osd.8 [v2:172.21.15.204:6816/1426318231,v1:172.21.15.204:6817/1426318231] boot 2024-04-23T10:22:38.492 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 23 10:22:38 smithi119 ceph-mon[32608]: osdmap e62: 9 total, 9 up, 9 in 2024-04-23T10:22:38.492 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 23 10:22:38 smithi119 ceph-mon[32608]: from='mgr.14152 172.21.15.114:0/1529706138' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 8}]: dispatch 2024-04-23T10:22:38.492 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 23 10:22:38 smithi119 ceph-mon[32608]: from='mgr.14152 172.21.15.114:0/1529706138' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-23T10:22:38.492 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 23 10:22:38 smithi119 ceph-mon[32608]: from='mgr.14152 172.21.15.114:0/1529706138' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-23T10:22:38.492 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 23 10:22:38 smithi119 ceph-mon[32608]: from='mgr.14152 172.21.15.114:0/1529706138' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-23T10:22:38.492 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 23 10:22:38 smithi119 ceph-mon[32608]: from='mgr.14152 172.21.15.114:0/1529706138' entity='mgr.x' 2024-04-23T10:22:38.504 INFO:journalctl@ceph.mon.a.smithi114.stdout:Apr 23 10:22:38 smithi114 ceph-mon[29255]: Detected new or changed devices on smithi204 2024-04-23T10:22:38.504 INFO:journalctl@ceph.mon.a.smithi114.stdout:Apr 23 10:22:38 smithi114 ceph-mon[29255]: Adjusting osd_memory_target on smithi204 to 5696M 2024-04-23T10:22:38.504 INFO:journalctl@ceph.mon.a.smithi114.stdout:Apr 23 10:22:38 smithi114 ceph-mon[29255]: osd.8 [v2:172.21.15.204:6816/1426318231,v1:172.21.15.204:6817/1426318231] boot 2024-04-23T10:22:38.504 INFO:journalctl@ceph.mon.a.smithi114.stdout:Apr 23 10:22:38 smithi114 ceph-mon[29255]: osdmap e62: 9 total, 9 up, 9 in 2024-04-23T10:22:38.504 INFO:journalctl@ceph.mon.a.smithi114.stdout:Apr 23 10:22:38 smithi114 ceph-mon[29255]: from='mgr.14152 172.21.15.114:0/1529706138' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 8}]: dispatch 2024-04-23T10:22:38.504 INFO:journalctl@ceph.mon.a.smithi114.stdout:Apr 23 10:22:38 smithi114 ceph-mon[29255]: from='mgr.14152 172.21.15.114:0/1529706138' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-23T10:22:38.504 INFO:journalctl@ceph.mon.a.smithi114.stdout:Apr 23 10:22:38 smithi114 ceph-mon[29255]: from='mgr.14152 172.21.15.114:0/1529706138' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-23T10:22:38.504 INFO:journalctl@ceph.mon.a.smithi114.stdout:Apr 23 10:22:38 smithi114 ceph-mon[29255]: from='mgr.14152 172.21.15.114:0/1529706138' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-23T10:22:38.504 INFO:journalctl@ceph.mon.a.smithi114.stdout:Apr 23 10:22:38 smithi114 ceph-mon[29255]: from='mgr.14152 172.21.15.114:0/1529706138' entity='mgr.x' 2024-04-23T10:22:39.254 INFO:journalctl@ceph.mon.a.smithi114.stdout:Apr 23 10:22:39 smithi114 ceph-mon[29255]: pgmap v198: 1 pgs: 1 remapped+peering; 577 KiB data, 242 MiB used, 715 GiB / 715 GiB avail 2024-04-23T10:22:39.254 INFO:journalctl@ceph.mon.a.smithi114.stdout:Apr 23 10:22:39 smithi114 ceph-mon[29255]: osdmap e63: 9 total, 9 up, 9 in 2024-04-23T10:22:39.254 INFO:journalctl@ceph.mon.a.smithi114.stdout:Apr 23 10:22:39 smithi114 ceph-mon[29255]: from='client.14433 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi114:vg_nvme/lv_1", "target": ["mon-mgr", ""]}]: dispatch 2024-04-23T10:22:39.254 INFO:journalctl@ceph.mon.a.smithi114.stdout:Apr 23 10:22:39 smithi114 ceph-mon[29255]: from='mgr.14152 172.21.15.114:0/1529706138' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-04-23T10:22:39.254 INFO:journalctl@ceph.mon.a.smithi114.stdout:Apr 23 10:22:39 smithi114 ceph-mon[29255]: from='mgr.14152 172.21.15.114:0/1529706138' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-04-23T10:22:39.254 INFO:journalctl@ceph.mon.a.smithi114.stdout:Apr 23 10:22:39 smithi114 ceph-mon[29255]: from='mgr.14152 172.21.15.114:0/1529706138' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-23T10:22:39.328 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 23 10:22:39 smithi204 ceph-mon[32649]: pgmap v198: 1 pgs: 1 remapped+peering; 577 KiB data, 242 MiB used, 715 GiB / 715 GiB avail 2024-04-23T10:22:39.328 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 23 10:22:39 smithi204 ceph-mon[32649]: osdmap e63: 9 total, 9 up, 9 in 2024-04-23T10:22:39.328 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 23 10:22:39 smithi204 ceph-mon[32649]: from='client.14433 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi114:vg_nvme/lv_1", "target": ["mon-mgr", ""]}]: dispatch 2024-04-23T10:22:39.328 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 23 10:22:39 smithi204 ceph-mon[32649]: from='mgr.14152 172.21.15.114:0/1529706138' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-04-23T10:22:39.328 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 23 10:22:39 smithi204 ceph-mon[32649]: from='mgr.14152 172.21.15.114:0/1529706138' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-04-23T10:22:39.328 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 23 10:22:39 smithi204 ceph-mon[32649]: from='mgr.14152 172.21.15.114:0/1529706138' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-23T10:22:39.492 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 23 10:22:39 smithi119 ceph-mon[32608]: pgmap v198: 1 pgs: 1 remapped+peering; 577 KiB data, 242 MiB used, 715 GiB / 715 GiB avail 2024-04-23T10:22:39.492 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 23 10:22:39 smithi119 ceph-mon[32608]: osdmap e63: 9 total, 9 up, 9 in 2024-04-23T10:22:39.492 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 23 10:22:39 smithi119 ceph-mon[32608]: from='client.14433 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi114:vg_nvme/lv_1", "target": ["mon-mgr", ""]}]: dispatch 2024-04-23T10:22:39.492 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 23 10:22:39 smithi119 ceph-mon[32608]: from='mgr.14152 172.21.15.114:0/1529706138' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-04-23T10:22:39.492 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 23 10:22:39 smithi119 ceph-mon[32608]: from='mgr.14152 172.21.15.114:0/1529706138' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-04-23T10:22:39.492 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 23 10:22:39 smithi119 ceph-mon[32608]: from='mgr.14152 172.21.15.114:0/1529706138' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-23T10:22:40.143 INFO:journalctl@ceph.mon.a.smithi114.stdout:Apr 23 10:22:40 smithi114 ceph-mon[29255]: osdmap e64: 9 total, 9 up, 9 in 2024-04-23T10:22:40.328 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 23 10:22:40 smithi204 ceph-mon[32649]: osdmap e64: 9 total, 9 up, 9 in 2024-04-23T10:22:40.492 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 23 10:22:40 smithi119 ceph-mon[32608]: osdmap e64: 9 total, 9 up, 9 in 2024-04-23T10:22:41.328 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 23 10:22:41 smithi204 ceph-mon[32649]: pgmap v201: 1 pgs: 1 remapped+peering; 577 KiB data, 273 MiB used, 804 GiB / 805 GiB avail 2024-04-23T10:22:41.328 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 23 10:22:41 smithi204 ceph-mon[32649]: osdmap e65: 9 total, 9 up, 9 in 2024-04-23T10:22:41.328 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 23 10:22:41 smithi204 ceph-mon[32649]: from='client.? 172.21.15.114:0/4091144089' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "a80ffe4d-1e98-460f-b046-c4754285cc5a"}]: dispatch 2024-04-23T10:22:41.328 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 23 10:22:41 smithi204 ceph-mon[32649]: from='client.? 172.21.15.114:0/4091144089' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "a80ffe4d-1e98-460f-b046-c4754285cc5a"}]': finished 2024-04-23T10:22:41.328 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 23 10:22:41 smithi204 ceph-mon[32649]: osdmap e66: 10 total, 9 up, 10 in 2024-04-23T10:22:41.328 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 23 10:22:41 smithi204 ceph-mon[32649]: from='mgr.14152 172.21.15.114:0/1529706138' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 9}]: dispatch 2024-04-23T10:22:41.492 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 23 10:22:41 smithi119 ceph-mon[32608]: pgmap v201: 1 pgs: 1 remapped+peering; 577 KiB data, 273 MiB used, 804 GiB / 805 GiB avail 2024-04-23T10:22:41.492 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 23 10:22:41 smithi119 ceph-mon[32608]: osdmap e65: 9 total, 9 up, 9 in 2024-04-23T10:22:41.492 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 23 10:22:41 smithi119 ceph-mon[32608]: from='client.? 172.21.15.114:0/4091144089' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "a80ffe4d-1e98-460f-b046-c4754285cc5a"}]: dispatch 2024-04-23T10:22:41.492 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 23 10:22:41 smithi119 ceph-mon[32608]: from='client.? 172.21.15.114:0/4091144089' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "a80ffe4d-1e98-460f-b046-c4754285cc5a"}]': finished 2024-04-23T10:22:41.492 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 23 10:22:41 smithi119 ceph-mon[32608]: osdmap e66: 10 total, 9 up, 10 in 2024-04-23T10:22:41.492 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 23 10:22:41 smithi119 ceph-mon[32608]: from='mgr.14152 172.21.15.114:0/1529706138' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 9}]: dispatch 2024-04-23T10:22:41.504 INFO:journalctl@ceph.mon.a.smithi114.stdout:Apr 23 10:22:41 smithi114 ceph-mon[29255]: pgmap v201: 1 pgs: 1 remapped+peering; 577 KiB data, 273 MiB used, 804 GiB / 805 GiB avail 2024-04-23T10:22:41.504 INFO:journalctl@ceph.mon.a.smithi114.stdout:Apr 23 10:22:41 smithi114 ceph-mon[29255]: osdmap e65: 9 total, 9 up, 9 in 2024-04-23T10:22:41.504 INFO:journalctl@ceph.mon.a.smithi114.stdout:Apr 23 10:22:41 smithi114 ceph-mon[29255]: from='client.? 172.21.15.114:0/4091144089' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "a80ffe4d-1e98-460f-b046-c4754285cc5a"}]: dispatch 2024-04-23T10:22:41.504 INFO:journalctl@ceph.mon.a.smithi114.stdout:Apr 23 10:22:41 smithi114 ceph-mon[29255]: from='client.? 172.21.15.114:0/4091144089' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "a80ffe4d-1e98-460f-b046-c4754285cc5a"}]': finished 2024-04-23T10:22:41.504 INFO:journalctl@ceph.mon.a.smithi114.stdout:Apr 23 10:22:41 smithi114 ceph-mon[29255]: osdmap e66: 10 total, 9 up, 10 in 2024-04-23T10:22:41.504 INFO:journalctl@ceph.mon.a.smithi114.stdout:Apr 23 10:22:41 smithi114 ceph-mon[29255]: from='mgr.14152 172.21.15.114:0/1529706138' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 9}]: dispatch 2024-04-23T10:22:42.328 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 23 10:22:42 smithi204 ceph-mon[32649]: from='client.? 172.21.15.114:0/1631569958' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-04-23T10:22:42.492 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 23 10:22:42 smithi119 ceph-mon[32608]: from='client.? 172.21.15.114:0/1631569958' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-04-23T10:22:42.504 INFO:journalctl@ceph.mon.a.smithi114.stdout:Apr 23 10:22:42 smithi114 ceph-mon[29255]: from='client.? 172.21.15.114:0/1631569958' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-04-23T10:22:43.328 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 23 10:22:43 smithi204 ceph-mon[32649]: pgmap v204: 1 pgs: 1 remapped+peering; 577 KiB data, 273 MiB used, 804 GiB / 805 GiB avail 2024-04-23T10:22:43.492 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 23 10:22:43 smithi119 ceph-mon[32608]: pgmap v204: 1 pgs: 1 remapped+peering; 577 KiB data, 273 MiB used, 804 GiB / 805 GiB avail 2024-04-23T10:22:43.504 INFO:journalctl@ceph.mon.a.smithi114.stdout:Apr 23 10:22:43 smithi114 ceph-mon[29255]: pgmap v204: 1 pgs: 1 remapped+peering; 577 KiB data, 273 MiB used, 804 GiB / 805 GiB avail 2024-04-23T10:22:45.328 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 23 10:22:45 smithi204 ceph-mon[32649]: pgmap v205: 1 pgs: 1 active+clean; 577 KiB data, 273 MiB used, 804 GiB / 805 GiB avail; 98 KiB/s, 0 objects/s recovering 2024-04-23T10:22:45.492 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 23 10:22:45 smithi119 ceph-mon[32608]: pgmap v205: 1 pgs: 1 active+clean; 577 KiB data, 273 MiB used, 804 GiB / 805 GiB avail; 98 KiB/s, 0 objects/s recovering 2024-04-23T10:22:45.504 INFO:journalctl@ceph.mon.a.smithi114.stdout:Apr 23 10:22:45 smithi114 ceph-mon[29255]: pgmap v205: 1 pgs: 1 active+clean; 577 KiB data, 273 MiB used, 804 GiB / 805 GiB avail; 98 KiB/s, 0 objects/s recovering 2024-04-23T10:22:47.324 INFO:journalctl@ceph.mon.a.smithi114.stdout:Apr 23 10:22:47 smithi114 ceph-mon[29255]: pgmap v206: 1 pgs: 1 active+clean; 577 KiB data, 274 MiB used, 804 GiB / 805 GiB avail; 84 KiB/s, 0 objects/s recovering 2024-04-23T10:22:47.328 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 23 10:22:47 smithi204 ceph-mon[32649]: pgmap v206: 1 pgs: 1 active+clean; 577 KiB data, 274 MiB used, 804 GiB / 805 GiB avail; 84 KiB/s, 0 objects/s recovering 2024-04-23T10:22:47.492 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 23 10:22:47 smithi119 ceph-mon[32608]: pgmap v206: 1 pgs: 1 active+clean; 577 KiB data, 274 MiB used, 804 GiB / 805 GiB avail; 84 KiB/s, 0 objects/s recovering 2024-04-23T10:22:49.328 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 23 10:22:49 smithi204 ceph-mon[32649]: pgmap v207: 1 pgs: 1 active+clean; 577 KiB data, 274 MiB used, 804 GiB / 805 GiB avail; 72 KiB/s, 0 objects/s recovering 2024-04-23T10:22:49.328 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 23 10:22:49 smithi204 ceph-mon[32649]: from='mgr.14152 172.21.15.114:0/1529706138' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.9"}]: dispatch 2024-04-23T10:22:49.328 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 23 10:22:49 smithi204 ceph-mon[32649]: from='mgr.14152 172.21.15.114:0/1529706138' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-23T10:22:49.433 INFO:journalctl@ceph.mon.a.smithi114.stdout:Apr 23 10:22:49 smithi114 ceph-mon[29255]: pgmap v207: 1 pgs: 1 active+clean; 577 KiB data, 274 MiB used, 804 GiB / 805 GiB avail; 72 KiB/s, 0 objects/s recovering 2024-04-23T10:22:49.433 INFO:journalctl@ceph.mon.a.smithi114.stdout:Apr 23 10:22:49 smithi114 ceph-mon[29255]: from='mgr.14152 172.21.15.114:0/1529706138' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.9"}]: dispatch 2024-04-23T10:22:49.433 INFO:journalctl@ceph.mon.a.smithi114.stdout:Apr 23 10:22:49 smithi114 ceph-mon[29255]: from='mgr.14152 172.21.15.114:0/1529706138' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-23T10:22:49.492 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 23 10:22:49 smithi119 ceph-mon[32608]: pgmap v207: 1 pgs: 1 active+clean; 577 KiB data, 274 MiB used, 804 GiB / 805 GiB avail; 72 KiB/s, 0 objects/s recovering 2024-04-23T10:22:49.492 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 23 10:22:49 smithi119 ceph-mon[32608]: from='mgr.14152 172.21.15.114:0/1529706138' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.9"}]: dispatch 2024-04-23T10:22:49.492 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 23 10:22:49 smithi119 ceph-mon[32608]: from='mgr.14152 172.21.15.114:0/1529706138' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-23T10:22:50.328 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 23 10:22:50 smithi204 ceph-mon[32649]: Deploying daemon osd.9 on smithi114 2024-04-23T10:22:50.492 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 23 10:22:50 smithi119 ceph-mon[32608]: Deploying daemon osd.9 on smithi114 2024-04-23T10:22:50.504 INFO:journalctl@ceph.mon.a.smithi114.stdout:Apr 23 10:22:50 smithi114 ceph-mon[29255]: Deploying daemon osd.9 on smithi114 2024-04-23T10:22:51.234 INFO:journalctl@ceph.mon.a.smithi114.stdout:Apr 23 10:22:51 smithi114 ceph-mon[29255]: pgmap v208: 1 pgs: 1 active+clean; 577 KiB data, 274 MiB used, 804 GiB / 805 GiB avail; 58 KiB/s, 0 objects/s recovering 2024-04-23T10:22:51.328 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 23 10:22:51 smithi204 ceph-mon[32649]: pgmap v208: 1 pgs: 1 active+clean; 577 KiB data, 274 MiB used, 804 GiB / 805 GiB avail; 58 KiB/s, 0 objects/s recovering 2024-04-23T10:22:51.492 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 23 10:22:51 smithi119 ceph-mon[32608]: pgmap v208: 1 pgs: 1 active+clean; 577 KiB data, 274 MiB used, 804 GiB / 805 GiB avail; 58 KiB/s, 0 objects/s recovering 2024-04-23T10:22:53.254 INFO:journalctl@ceph.mon.a.smithi114.stdout:Apr 23 10:22:53 smithi114 ceph-mon[29255]: pgmap v209: 1 pgs: 1 active+clean; 577 KiB data, 274 MiB used, 804 GiB / 805 GiB avail; 52 KiB/s, 0 objects/s recovering 2024-04-23T10:22:53.328 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 23 10:22:53 smithi204 ceph-mon[32649]: pgmap v209: 1 pgs: 1 active+clean; 577 KiB data, 274 MiB used, 804 GiB / 805 GiB avail; 52 KiB/s, 0 objects/s recovering 2024-04-23T10:22:53.492 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 23 10:22:53 smithi119 ceph-mon[32608]: pgmap v209: 1 pgs: 1 active+clean; 577 KiB data, 274 MiB used, 804 GiB / 805 GiB avail; 52 KiB/s, 0 objects/s recovering 2024-04-23T10:22:55.328 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 23 10:22:55 smithi204 ceph-mon[32649]: pgmap v210: 1 pgs: 1 active+clean; 577 KiB data, 274 MiB used, 804 GiB / 805 GiB avail; 48 KiB/s, 0 objects/s recovering 2024-04-23T10:22:55.328 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 23 10:22:55 smithi204 ceph-mon[32649]: from='mgr.14152 172.21.15.114:0/1529706138' entity='mgr.x' 2024-04-23T10:22:55.328 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 23 10:22:55 smithi204 ceph-mon[32649]: from='mgr.14152 172.21.15.114:0/1529706138' entity='mgr.x' 2024-04-23T10:22:55.328 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 23 10:22:55 smithi204 ceph-mon[32649]: from='mgr.14152 172.21.15.114:0/1529706138' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-23T10:22:55.468 INFO:journalctl@ceph.mon.a.smithi114.stdout:Apr 23 10:22:55 smithi114 ceph-mon[29255]: pgmap v210: 1 pgs: 1 active+clean; 577 KiB data, 274 MiB used, 804 GiB / 805 GiB avail; 48 KiB/s, 0 objects/s recovering 2024-04-23T10:22:55.468 INFO:journalctl@ceph.mon.a.smithi114.stdout:Apr 23 10:22:55 smithi114 ceph-mon[29255]: from='mgr.14152 172.21.15.114:0/1529706138' entity='mgr.x' 2024-04-23T10:22:55.468 INFO:journalctl@ceph.mon.a.smithi114.stdout:Apr 23 10:22:55 smithi114 ceph-mon[29255]: from='mgr.14152 172.21.15.114:0/1529706138' entity='mgr.x' 2024-04-23T10:22:55.468 INFO:journalctl@ceph.mon.a.smithi114.stdout:Apr 23 10:22:55 smithi114 ceph-mon[29255]: from='mgr.14152 172.21.15.114:0/1529706138' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-23T10:22:55.492 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 23 10:22:55 smithi119 ceph-mon[32608]: pgmap v210: 1 pgs: 1 active+clean; 577 KiB data, 274 MiB used, 804 GiB / 805 GiB avail; 48 KiB/s, 0 objects/s recovering 2024-04-23T10:22:55.492 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 23 10:22:55 smithi119 ceph-mon[32608]: from='mgr.14152 172.21.15.114:0/1529706138' entity='mgr.x' 2024-04-23T10:22:55.492 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 23 10:22:55 smithi119 ceph-mon[32608]: from='mgr.14152 172.21.15.114:0/1529706138' entity='mgr.x' 2024-04-23T10:22:55.492 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 23 10:22:55 smithi119 ceph-mon[32608]: from='mgr.14152 172.21.15.114:0/1529706138' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-23T10:22:57.187 INFO:journalctl@ceph.mon.a.smithi114.stdout:Apr 23 10:22:57 smithi114 ceph-mon[29255]: pgmap v211: 1 pgs: 1 active+clean; 577 KiB data, 274 MiB used, 804 GiB / 805 GiB avail 2024-04-23T10:22:57.327 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 23 10:22:57 smithi204 ceph-mon[32649]: pgmap v211: 1 pgs: 1 active+clean; 577 KiB data, 274 MiB used, 804 GiB / 805 GiB avail 2024-04-23T10:22:57.491 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 23 10:22:57 smithi119 ceph-mon[32608]: pgmap v211: 1 pgs: 1 active+clean; 577 KiB data, 274 MiB used, 804 GiB / 805 GiB avail 2024-04-23T10:22:58.797 INFO:teuthology.orchestra.run.smithi114.stdout:Created osd(s) 9 on host 'smithi114' 2024-04-23T10:22:58.798 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:22:58.798+0000 7fc6789c6700 1 -- 172.21.15.114:0/3759512728 <== mgr.14152 v2:172.21.15.114:6800/465333808 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+37 (secure 0 0 0) 0x7fc674062e80 con 0x7fc65c0a3c40 2024-04-23T10:22:58.799 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:22:58.800+0000 7fc6799c8700 1 -- 172.21.15.114:0/3759512728 >> [v2:172.21.15.114:6800/465333808,v1:172.21.15.114:6801/465333808] conn(0x7fc65c0a3c40 msgr2=0x7fc65c0a6100 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T10:22:58.800 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:22:58.800+0000 7fc6799c8700 1 --2- 172.21.15.114:0/3759512728 >> [v2:172.21.15.114:6800/465333808,v1:172.21.15.114:6801/465333808] conn(0x7fc65c0a3c40 0x7fc65c0a6100 secure :-1 s=READY pgs=106 cs=0 l=1 rev1=1 crypto rx=0x7fc668005950 tx=0x7fc66800b410 comp rx=0 tx=0).stop 2024-04-23T10:22:58.800 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:22:58.800+0000 7fc6799c8700 1 -- 172.21.15.114:0/3759512728 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fc6741019c0 msgr2=0x7fc674198c70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T10:22:58.800 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:22:58.800+0000 7fc6799c8700 1 --2- 172.21.15.114:0/3759512728 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fc6741019c0 0x7fc674198c70 secure :-1 s=READY pgs=141 cs=0 l=1 rev1=1 crypto rx=0x7fc6640038c0 tx=0x7fc6640047b0 comp rx=0 tx=0).stop 2024-04-23T10:22:58.800 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:22:58.800+0000 7fc6799c8700 1 -- 172.21.15.114:0/3759512728 shutdown_connections 2024-04-23T10:22:58.800 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:22:58.800+0000 7fc6799c8700 1 --2- 172.21.15.114:0/3759512728 >> [v2:172.21.15.114:6800/465333808,v1:172.21.15.114:6801/465333808] conn(0x7fc65c0a3c40 0x7fc65c0a6100 unknown :-1 s=CLOSED pgs=106 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:22:58.800 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:22:58.800+0000 7fc6799c8700 1 --2- 172.21.15.114:0/3759512728 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fc674103560 0x7fc6741a01f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:22:58.800 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:22:58.800+0000 7fc6799c8700 1 --2- 172.21.15.114:0/3759512728 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fc6741019c0 0x7fc674198c70 unknown :-1 s=CLOSED pgs=141 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:22:58.800 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:22:58.800+0000 7fc6799c8700 1 --2- 172.21.15.114:0/3759512728 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fc674102bc0 0x7fc6741991b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:22:58.800 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:22:58.800+0000 7fc6799c8700 1 -- 172.21.15.114:0/3759512728 >> 172.21.15.114:0/3759512728 conn(0x7fc6740fcfa0 msgr2=0x7fc674105f90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T10:22:58.800 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:22:58.800+0000 7fc6799c8700 1 -- 172.21.15.114:0/3759512728 shutdown_connections 2024-04-23T10:22:58.800 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:22:58.800+0000 7fc6799c8700 1 -- 172.21.15.114:0/3759512728 wait complete. 2024-04-23T10:22:59.004 INFO:journalctl@ceph.mon.a.smithi114.stdout:Apr 23 10:22:58 smithi114 ceph-mon[29255]: from='mgr.14152 172.21.15.114:0/1529706138' entity='mgr.x' 2024-04-23T10:22:59.004 INFO:journalctl@ceph.mon.a.smithi114.stdout:Apr 23 10:22:58 smithi114 ceph-mon[29255]: from='mgr.14152 172.21.15.114:0/1529706138' entity='mgr.x' 2024-04-23T10:22:59.004 INFO:journalctl@ceph.mon.a.smithi114.stdout:Apr 23 10:22:58 smithi114 ceph-mon[29255]: from='mgr.14152 172.21.15.114:0/1529706138' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-23T10:22:59.004 INFO:journalctl@ceph.mon.a.smithi114.stdout:Apr 23 10:22:58 smithi114 ceph-mon[29255]: from='mgr.14152 172.21.15.114:0/1529706138' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-23T10:22:59.004 INFO:journalctl@ceph.mon.a.smithi114.stdout:Apr 23 10:22:58 smithi114 ceph-mon[29255]: from='mgr.14152 172.21.15.114:0/1529706138' entity='mgr.x' 2024-04-23T10:22:59.004 INFO:journalctl@ceph.mon.a.smithi114.stdout:Apr 23 10:22:58 smithi114 ceph-mon[29255]: pgmap v212: 1 pgs: 1 active+clean; 577 KiB data, 274 MiB used, 804 GiB / 805 GiB avail 2024-04-23T10:22:59.004 INFO:journalctl@ceph.mon.a.smithi114.stdout:Apr 23 10:22:58 smithi114 ceph-mon[29255]: from='osd.9 [v2:172.21.15.114:6826/2750635721,v1:172.21.15.114:6827/2750635721]' entity='osd.9' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["9"]}]: dispatch 2024-04-23T10:22:59.004 INFO:journalctl@ceph.mon.a.smithi114.stdout:Apr 23 10:22:58 smithi114 ceph-mon[29255]: from='mgr.14152 172.21.15.114:0/1529706138' entity='mgr.x' 2024-04-23T10:22:59.005 INFO:journalctl@ceph.mon.a.smithi114.stdout:Apr 23 10:22:58 smithi114 ceph-mon[29255]: from='mgr.14152 172.21.15.114:0/1529706138' entity='mgr.x' 2024-04-23T10:22:59.005 INFO:journalctl@ceph.mon.a.smithi114.stdout:Apr 23 10:22:58 smithi114 ceph-mon[29255]: from='mgr.14152 172.21.15.114:0/1529706138' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-23T10:22:59.242 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 23 10:22:58 smithi119 ceph-mon[32608]: from='mgr.14152 172.21.15.114:0/1529706138' entity='mgr.x' 2024-04-23T10:22:59.242 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 23 10:22:58 smithi119 ceph-mon[32608]: from='mgr.14152 172.21.15.114:0/1529706138' entity='mgr.x' 2024-04-23T10:22:59.242 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 23 10:22:58 smithi119 ceph-mon[32608]: from='mgr.14152 172.21.15.114:0/1529706138' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-23T10:22:59.242 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 23 10:22:58 smithi119 ceph-mon[32608]: from='mgr.14152 172.21.15.114:0/1529706138' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-23T10:22:59.242 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 23 10:22:58 smithi119 ceph-mon[32608]: from='mgr.14152 172.21.15.114:0/1529706138' entity='mgr.x' 2024-04-23T10:22:59.242 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 23 10:22:58 smithi119 ceph-mon[32608]: pgmap v212: 1 pgs: 1 active+clean; 577 KiB data, 274 MiB used, 804 GiB / 805 GiB avail 2024-04-23T10:22:59.242 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 23 10:22:58 smithi119 ceph-mon[32608]: from='osd.9 [v2:172.21.15.114:6826/2750635721,v1:172.21.15.114:6827/2750635721]' entity='osd.9' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["9"]}]: dispatch 2024-04-23T10:22:59.242 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 23 10:22:58 smithi119 ceph-mon[32608]: from='mgr.14152 172.21.15.114:0/1529706138' entity='mgr.x' 2024-04-23T10:22:59.242 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 23 10:22:58 smithi119 ceph-mon[32608]: from='mgr.14152 172.21.15.114:0/1529706138' entity='mgr.x' 2024-04-23T10:22:59.243 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 23 10:22:58 smithi119 ceph-mon[32608]: from='mgr.14152 172.21.15.114:0/1529706138' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-23T10:22:59.328 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 23 10:22:58 smithi204 ceph-mon[32649]: from='mgr.14152 172.21.15.114:0/1529706138' entity='mgr.x' 2024-04-23T10:22:59.328 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 23 10:22:58 smithi204 ceph-mon[32649]: from='mgr.14152 172.21.15.114:0/1529706138' entity='mgr.x' 2024-04-23T10:22:59.328 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 23 10:22:58 smithi204 ceph-mon[32649]: from='mgr.14152 172.21.15.114:0/1529706138' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-23T10:22:59.328 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 23 10:22:58 smithi204 ceph-mon[32649]: from='mgr.14152 172.21.15.114:0/1529706138' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-23T10:22:59.328 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 23 10:22:58 smithi204 ceph-mon[32649]: from='mgr.14152 172.21.15.114:0/1529706138' entity='mgr.x' 2024-04-23T10:22:59.328 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 23 10:22:58 smithi204 ceph-mon[32649]: pgmap v212: 1 pgs: 1 active+clean; 577 KiB data, 274 MiB used, 804 GiB / 805 GiB avail 2024-04-23T10:22:59.328 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 23 10:22:58 smithi204 ceph-mon[32649]: from='osd.9 [v2:172.21.15.114:6826/2750635721,v1:172.21.15.114:6827/2750635721]' entity='osd.9' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["9"]}]: dispatch 2024-04-23T10:22:59.328 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 23 10:22:58 smithi204 ceph-mon[32649]: from='mgr.14152 172.21.15.114:0/1529706138' entity='mgr.x' 2024-04-23T10:22:59.328 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 23 10:22:58 smithi204 ceph-mon[32649]: from='mgr.14152 172.21.15.114:0/1529706138' entity='mgr.x' 2024-04-23T10:22:59.328 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 23 10:22:58 smithi204 ceph-mon[32649]: from='mgr.14152 172.21.15.114:0/1529706138' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-23T10:22:59.409 DEBUG:teuthology.orchestra.run.smithi114:osd.9> sudo journalctl -f -n 0 -u ceph-aa240572-0159-11ef-bc93-c7b262605968@osd.9.service 2024-04-23T10:22:59.411 INFO:tasks.cephadm:Deploying osd.10 on smithi119 with /dev/vg_nvme/lv_1... 2024-04-23T10:22:59.412 DEBUG:teuthology.orchestra.run.smithi119:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d69f13d827952bb07bb9b7973bca14f97d764d6f shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid aa240572-0159-11ef-bc93-c7b262605968 -- ceph-volume lvm zap /dev/vg_nvme/lv_1 2024-04-23T10:22:59.657 INFO:teuthology.orchestra.run.smithi119.stderr:Inferring config /var/lib/ceph/aa240572-0159-11ef-bc93-c7b262605968/mon.b/config 2024-04-23T10:23:00.092 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 23 10:22:59 smithi119 ceph-mon[32608]: from='osd.9 [v2:172.21.15.114:6826/2750635721,v1:172.21.15.114:6827/2750635721]' entity='osd.9' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["9"]}]': finished 2024-04-23T10:23:00.093 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 23 10:22:59 smithi119 ceph-mon[32608]: osdmap e67: 10 total, 9 up, 10 in 2024-04-23T10:23:00.093 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 23 10:22:59 smithi119 ceph-mon[32608]: from='osd.9 [v2:172.21.15.114:6826/2750635721,v1:172.21.15.114:6827/2750635721]' entity='osd.9' cmd=[{"prefix": "osd crush create-or-move", "id": 9, "weight":0.0873, "args": ["host=smithi114", "root=default"]}]: dispatch 2024-04-23T10:23:00.093 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 23 10:22:59 smithi119 ceph-mon[32608]: from='mgr.14152 172.21.15.114:0/1529706138' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 9}]: dispatch 2024-04-23T10:23:00.254 INFO:journalctl@ceph.mon.a.smithi114.stdout:Apr 23 10:22:59 smithi114 ceph-mon[29255]: from='osd.9 [v2:172.21.15.114:6826/2750635721,v1:172.21.15.114:6827/2750635721]' entity='osd.9' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["9"]}]': finished 2024-04-23T10:23:00.254 INFO:journalctl@ceph.mon.a.smithi114.stdout:Apr 23 10:22:59 smithi114 ceph-mon[29255]: osdmap e67: 10 total, 9 up, 10 in 2024-04-23T10:23:00.254 INFO:journalctl@ceph.mon.a.smithi114.stdout:Apr 23 10:22:59 smithi114 ceph-mon[29255]: from='osd.9 [v2:172.21.15.114:6826/2750635721,v1:172.21.15.114:6827/2750635721]' entity='osd.9' cmd=[{"prefix": "osd crush create-or-move", "id": 9, "weight":0.0873, "args": ["host=smithi114", "root=default"]}]: dispatch 2024-04-23T10:23:00.255 INFO:journalctl@ceph.mon.a.smithi114.stdout:Apr 23 10:22:59 smithi114 ceph-mon[29255]: from='mgr.14152 172.21.15.114:0/1529706138' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 9}]: dispatch 2024-04-23T10:23:00.255 INFO:journalctl@ceph.osd.9.smithi114.stdout:Apr 23 10:22:59 smithi114 ceph-aa240572-0159-11ef-bc93-c7b262605968-osd-9[62058]: 2024-04-23T10:22:59.914+0000 7feeb75d1700 -1 osd.9 0 waiting for initial osdmap 2024-04-23T10:23:00.255 INFO:journalctl@ceph.osd.9.smithi114.stdout:Apr 23 10:22:59 smithi114 ceph-aa240572-0159-11ef-bc93-c7b262605968-osd-9[62058]: 2024-04-23T10:22:59.930+0000 7feeb33c4700 -1 osd.9 68 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-04-23T10:23:00.328 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 23 10:22:59 smithi204 ceph-mon[32649]: from='osd.9 [v2:172.21.15.114:6826/2750635721,v1:172.21.15.114:6827/2750635721]' entity='osd.9' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["9"]}]': finished 2024-04-23T10:23:00.328 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 23 10:22:59 smithi204 ceph-mon[32649]: osdmap e67: 10 total, 9 up, 10 in 2024-04-23T10:23:00.328 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 23 10:22:59 smithi204 ceph-mon[32649]: from='osd.9 [v2:172.21.15.114:6826/2750635721,v1:172.21.15.114:6827/2750635721]' entity='osd.9' cmd=[{"prefix": "osd crush create-or-move", "id": 9, "weight":0.0873, "args": ["host=smithi114", "root=default"]}]: dispatch 2024-04-23T10:23:00.328 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 23 10:22:59 smithi204 ceph-mon[32649]: from='mgr.14152 172.21.15.114:0/1529706138' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 9}]: dispatch 2024-04-23T10:23:00.992 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 23 10:23:00 smithi119 ceph-mon[32608]: from='osd.9 [v2:172.21.15.114:6826/2750635721,v1:172.21.15.114:6827/2750635721]' entity='osd.9' cmd='[{"prefix": "osd crush create-or-move", "id": 9, "weight":0.0873, "args": ["host=smithi114", "root=default"]}]': finished 2024-04-23T10:23:00.992 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 23 10:23:00 smithi119 ceph-mon[32608]: osdmap e68: 10 total, 9 up, 10 in 2024-04-23T10:23:00.992 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 23 10:23:00 smithi119 ceph-mon[32608]: from='mgr.14152 172.21.15.114:0/1529706138' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 9}]: dispatch 2024-04-23T10:23:00.992 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 23 10:23:00 smithi119 ceph-mon[32608]: from='mgr.14152 172.21.15.114:0/1529706138' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 9}]: dispatch 2024-04-23T10:23:00.992 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 23 10:23:00 smithi119 ceph-mon[32608]: pgmap v215: 1 pgs: 1 active+clean; 577 KiB data, 274 MiB used, 804 GiB / 805 GiB avail 2024-04-23T10:23:01.004 INFO:journalctl@ceph.mon.a.smithi114.stdout:Apr 23 10:23:00 smithi114 ceph-mon[29255]: from='osd.9 [v2:172.21.15.114:6826/2750635721,v1:172.21.15.114:6827/2750635721]' entity='osd.9' cmd='[{"prefix": "osd crush create-or-move", "id": 9, "weight":0.0873, "args": ["host=smithi114", "root=default"]}]': finished 2024-04-23T10:23:01.004 INFO:journalctl@ceph.mon.a.smithi114.stdout:Apr 23 10:23:00 smithi114 ceph-mon[29255]: osdmap e68: 10 total, 9 up, 10 in 2024-04-23T10:23:01.004 INFO:journalctl@ceph.mon.a.smithi114.stdout:Apr 23 10:23:00 smithi114 ceph-mon[29255]: from='mgr.14152 172.21.15.114:0/1529706138' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 9}]: dispatch 2024-04-23T10:23:01.004 INFO:journalctl@ceph.mon.a.smithi114.stdout:Apr 23 10:23:00 smithi114 ceph-mon[29255]: from='mgr.14152 172.21.15.114:0/1529706138' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 9}]: dispatch 2024-04-23T10:23:01.004 INFO:journalctl@ceph.mon.a.smithi114.stdout:Apr 23 10:23:00 smithi114 ceph-mon[29255]: pgmap v215: 1 pgs: 1 active+clean; 577 KiB data, 274 MiB used, 804 GiB / 805 GiB avail 2024-04-23T10:23:01.328 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 23 10:23:00 smithi204 ceph-mon[32649]: from='osd.9 [v2:172.21.15.114:6826/2750635721,v1:172.21.15.114:6827/2750635721]' entity='osd.9' cmd='[{"prefix": "osd crush create-or-move", "id": 9, "weight":0.0873, "args": ["host=smithi114", "root=default"]}]': finished 2024-04-23T10:23:01.328 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 23 10:23:00 smithi204 ceph-mon[32649]: osdmap e68: 10 total, 9 up, 10 in 2024-04-23T10:23:01.328 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 23 10:23:00 smithi204 ceph-mon[32649]: from='mgr.14152 172.21.15.114:0/1529706138' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 9}]: dispatch 2024-04-23T10:23:01.328 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 23 10:23:00 smithi204 ceph-mon[32649]: from='mgr.14152 172.21.15.114:0/1529706138' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 9}]: dispatch 2024-04-23T10:23:01.328 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 23 10:23:00 smithi204 ceph-mon[32649]: pgmap v215: 1 pgs: 1 active+clean; 577 KiB data, 274 MiB used, 804 GiB / 805 GiB avail 2024-04-23T10:23:01.616 INFO:teuthology.orchestra.run.smithi119.stderr:--> Zapping: /dev/vg_nvme/lv_1 2024-04-23T10:23:01.616 INFO:teuthology.orchestra.run.smithi119.stderr:Running command: /usr/bin/dd if=/dev/zero of=/dev/vg_nvme/lv_1 bs=1M count=10 conv=fsync 2024-04-23T10:23:01.616 INFO:teuthology.orchestra.run.smithi119.stderr: stderr: 10+0 records in 2024-04-23T10:23:01.616 INFO:teuthology.orchestra.run.smithi119.stderr:10+0 records out 2024-04-23T10:23:01.616 INFO:teuthology.orchestra.run.smithi119.stderr: stderr: 10485760 bytes (10 MB, 10 MiB) copied, 0.0115937 s, 904 MB/s 2024-04-23T10:23:01.617 INFO:teuthology.orchestra.run.smithi119.stderr:--> Zapping successful for: 2024-04-23T10:23:02.091 INFO:journalctl@ceph.mon.a.smithi114.stdout:Apr 23 10:23:01 smithi114 ceph-mon[29255]: purged_snaps scrub starts 2024-04-23T10:23:02.091 INFO:journalctl@ceph.mon.a.smithi114.stdout:Apr 23 10:23:01 smithi114 ceph-mon[29255]: purged_snaps scrub ok 2024-04-23T10:23:02.091 INFO:journalctl@ceph.mon.a.smithi114.stdout:Apr 23 10:23:01 smithi114 ceph-mon[29255]: from='mgr.14152 172.21.15.114:0/1529706138' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 9}]: dispatch 2024-04-23T10:23:02.091 INFO:journalctl@ceph.mon.a.smithi114.stdout:Apr 23 10:23:01 smithi114 ceph-mon[29255]: osd.9 [v2:172.21.15.114:6826/2750635721,v1:172.21.15.114:6827/2750635721] boot 2024-04-23T10:23:02.091 INFO:journalctl@ceph.mon.a.smithi114.stdout:Apr 23 10:23:01 smithi114 ceph-mon[29255]: osdmap e69: 10 total, 10 up, 10 in 2024-04-23T10:23:02.091 INFO:journalctl@ceph.mon.a.smithi114.stdout:Apr 23 10:23:01 smithi114 ceph-mon[29255]: from='mgr.14152 172.21.15.114:0/1529706138' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 9}]: dispatch 2024-04-23T10:23:02.242 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 23 10:23:01 smithi119 ceph-mon[32608]: purged_snaps scrub starts 2024-04-23T10:23:02.242 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 23 10:23:01 smithi119 ceph-mon[32608]: purged_snaps scrub ok 2024-04-23T10:23:02.242 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 23 10:23:01 smithi119 ceph-mon[32608]: from='mgr.14152 172.21.15.114:0/1529706138' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 9}]: dispatch 2024-04-23T10:23:02.242 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 23 10:23:01 smithi119 ceph-mon[32608]: osd.9 [v2:172.21.15.114:6826/2750635721,v1:172.21.15.114:6827/2750635721] boot 2024-04-23T10:23:02.243 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 23 10:23:01 smithi119 ceph-mon[32608]: osdmap e69: 10 total, 10 up, 10 in 2024-04-23T10:23:02.243 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 23 10:23:01 smithi119 ceph-mon[32608]: from='mgr.14152 172.21.15.114:0/1529706138' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 9}]: dispatch 2024-04-23T10:23:02.326 DEBUG:teuthology.orchestra.run.smithi119:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d69f13d827952bb07bb9b7973bca14f97d764d6f shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid aa240572-0159-11ef-bc93-c7b262605968 -- ceph orch daemon add osd smithi119:vg_nvme/lv_1 2024-04-23T10:23:02.328 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 23 10:23:01 smithi204 ceph-mon[32649]: purged_snaps scrub starts 2024-04-23T10:23:02.328 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 23 10:23:01 smithi204 ceph-mon[32649]: purged_snaps scrub ok 2024-04-23T10:23:02.328 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 23 10:23:01 smithi204 ceph-mon[32649]: from='mgr.14152 172.21.15.114:0/1529706138' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 9}]: dispatch 2024-04-23T10:23:02.328 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 23 10:23:01 smithi204 ceph-mon[32649]: osd.9 [v2:172.21.15.114:6826/2750635721,v1:172.21.15.114:6827/2750635721] boot 2024-04-23T10:23:02.328 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 23 10:23:01 smithi204 ceph-mon[32649]: osdmap e69: 10 total, 10 up, 10 in 2024-04-23T10:23:02.328 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 23 10:23:01 smithi204 ceph-mon[32649]: from='mgr.14152 172.21.15.114:0/1529706138' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 9}]: dispatch 2024-04-23T10:23:02.571 INFO:teuthology.orchestra.run.smithi119.stderr:Inferring config /var/lib/ceph/aa240572-0159-11ef-bc93-c7b262605968/mon.b/config 2024-04-23T10:23:03.242 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 23 10:23:02 smithi119 ceph-mon[32608]: osdmap e70: 10 total, 10 up, 10 in 2024-04-23T10:23:03.243 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 23 10:23:02 smithi119 ceph-mon[32608]: pgmap v218: 1 pgs: 1 active+clean; 577 KiB data, 301 MiB used, 894 GiB / 894 GiB avail 2024-04-23T10:23:03.243 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 23 10:23:02 smithi119 ceph-mon[32608]: from='mgr.14152 172.21.15.114:0/1529706138' entity='mgr.x' 2024-04-23T10:23:03.243 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 23 10:23:02 smithi119 ceph-mon[32608]: from='mgr.14152 172.21.15.114:0/1529706138' entity='mgr.x' 2024-04-23T10:23:03.243 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 23 10:23:02 smithi119 ceph-mon[32608]: from='mgr.14152 172.21.15.114:0/1529706138' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-04-23T10:23:03.243 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 23 10:23:02 smithi119 ceph-mon[32608]: from='mgr.14152 172.21.15.114:0/1529706138' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-04-23T10:23:03.243 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 23 10:23:02 smithi119 ceph-mon[32608]: from='mgr.14152 172.21.15.114:0/1529706138' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-04-23T10:23:03.243 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 23 10:23:02 smithi119 ceph-mon[32608]: from='mgr.14152 172.21.15.114:0/1529706138' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"}]: dispatch 2024-04-23T10:23:03.243 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 23 10:23:02 smithi119 ceph-mon[32608]: from='mgr.14152 172.21.15.114:0/1529706138' entity='mgr.x' 2024-04-23T10:23:03.243 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 23 10:23:02 smithi119 ceph-mon[32608]: from='mgr.14152 172.21.15.114:0/1529706138' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-23T10:23:03.243 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 23 10:23:02 smithi119 ceph-mon[32608]: from='mgr.14152 172.21.15.114:0/1529706138' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-23T10:23:03.243 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 23 10:23:02 smithi119 ceph-mon[32608]: from='mgr.14152 172.21.15.114:0/1529706138' entity='mgr.x' 2024-04-23T10:23:03.254 INFO:journalctl@ceph.mon.a.smithi114.stdout:Apr 23 10:23:02 smithi114 ceph-mon[29255]: osdmap e70: 10 total, 10 up, 10 in 2024-04-23T10:23:03.254 INFO:journalctl@ceph.mon.a.smithi114.stdout:Apr 23 10:23:02 smithi114 ceph-mon[29255]: pgmap v218: 1 pgs: 1 active+clean; 577 KiB data, 301 MiB used, 894 GiB / 894 GiB avail 2024-04-23T10:23:03.254 INFO:journalctl@ceph.mon.a.smithi114.stdout:Apr 23 10:23:02 smithi114 ceph-mon[29255]: from='mgr.14152 172.21.15.114:0/1529706138' entity='mgr.x' 2024-04-23T10:23:03.254 INFO:journalctl@ceph.mon.a.smithi114.stdout:Apr 23 10:23:02 smithi114 ceph-mon[29255]: from='mgr.14152 172.21.15.114:0/1529706138' entity='mgr.x' 2024-04-23T10:23:03.254 INFO:journalctl@ceph.mon.a.smithi114.stdout:Apr 23 10:23:02 smithi114 ceph-mon[29255]: from='mgr.14152 172.21.15.114:0/1529706138' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-04-23T10:23:03.255 INFO:journalctl@ceph.mon.a.smithi114.stdout:Apr 23 10:23:02 smithi114 ceph-mon[29255]: from='mgr.14152 172.21.15.114:0/1529706138' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-04-23T10:23:03.255 INFO:journalctl@ceph.mon.a.smithi114.stdout:Apr 23 10:23:02 smithi114 ceph-mon[29255]: from='mgr.14152 172.21.15.114:0/1529706138' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-04-23T10:23:03.255 INFO:journalctl@ceph.mon.a.smithi114.stdout:Apr 23 10:23:02 smithi114 ceph-mon[29255]: from='mgr.14152 172.21.15.114:0/1529706138' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"}]: dispatch 2024-04-23T10:23:03.255 INFO:journalctl@ceph.mon.a.smithi114.stdout:Apr 23 10:23:02 smithi114 ceph-mon[29255]: from='mgr.14152 172.21.15.114:0/1529706138' entity='mgr.x' 2024-04-23T10:23:03.255 INFO:journalctl@ceph.mon.a.smithi114.stdout:Apr 23 10:23:02 smithi114 ceph-mon[29255]: from='mgr.14152 172.21.15.114:0/1529706138' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-23T10:23:03.255 INFO:journalctl@ceph.mon.a.smithi114.stdout:Apr 23 10:23:02 smithi114 ceph-mon[29255]: from='mgr.14152 172.21.15.114:0/1529706138' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-23T10:23:03.255 INFO:journalctl@ceph.mon.a.smithi114.stdout:Apr 23 10:23:02 smithi114 ceph-mon[29255]: from='mgr.14152 172.21.15.114:0/1529706138' entity='mgr.x' 2024-04-23T10:23:03.328 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 23 10:23:02 smithi204 ceph-mon[32649]: osdmap e70: 10 total, 10 up, 10 in 2024-04-23T10:23:03.328 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 23 10:23:02 smithi204 ceph-mon[32649]: pgmap v218: 1 pgs: 1 active+clean; 577 KiB data, 301 MiB used, 894 GiB / 894 GiB avail 2024-04-23T10:23:03.328 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 23 10:23:02 smithi204 ceph-mon[32649]: from='mgr.14152 172.21.15.114:0/1529706138' entity='mgr.x' 2024-04-23T10:23:03.328 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 23 10:23:02 smithi204 ceph-mon[32649]: from='mgr.14152 172.21.15.114:0/1529706138' entity='mgr.x' 2024-04-23T10:23:03.328 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 23 10:23:02 smithi204 ceph-mon[32649]: from='mgr.14152 172.21.15.114:0/1529706138' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-04-23T10:23:03.329 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 23 10:23:02 smithi204 ceph-mon[32649]: from='mgr.14152 172.21.15.114:0/1529706138' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-04-23T10:23:03.329 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 23 10:23:02 smithi204 ceph-mon[32649]: from='mgr.14152 172.21.15.114:0/1529706138' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-04-23T10:23:03.329 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 23 10:23:02 smithi204 ceph-mon[32649]: from='mgr.14152 172.21.15.114:0/1529706138' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"}]: dispatch 2024-04-23T10:23:03.329 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 23 10:23:02 smithi204 ceph-mon[32649]: from='mgr.14152 172.21.15.114:0/1529706138' entity='mgr.x' 2024-04-23T10:23:03.329 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 23 10:23:02 smithi204 ceph-mon[32649]: from='mgr.14152 172.21.15.114:0/1529706138' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-23T10:23:03.329 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 23 10:23:02 smithi204 ceph-mon[32649]: from='mgr.14152 172.21.15.114:0/1529706138' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-23T10:23:03.329 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 23 10:23:02 smithi204 ceph-mon[32649]: from='mgr.14152 172.21.15.114:0/1529706138' entity='mgr.x' 2024-04-23T10:23:04.106 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 23 10:23:03 smithi119 ceph-mon[32608]: Detected new or changed devices on smithi114 2024-04-23T10:23:04.106 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 23 10:23:03 smithi119 ceph-mon[32608]: Adjusting osd_memory_target on smithi114 to 4272M 2024-04-23T10:23:04.254 INFO:journalctl@ceph.mon.a.smithi114.stdout:Apr 23 10:23:03 smithi114 ceph-mon[29255]: Detected new or changed devices on smithi114 2024-04-23T10:23:04.254 INFO:journalctl@ceph.mon.a.smithi114.stdout:Apr 23 10:23:03 smithi114 ceph-mon[29255]: Adjusting osd_memory_target on smithi114 to 4272M 2024-04-23T10:23:04.308 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-23T10:23:04.309+0000 7f60ce68d700 1 -- 172.21.15.119:0/1317335967 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f60c8103ca0 msgr2=0x7f60c8104100 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T10:23:04.308 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-23T10:23:04.309+0000 7f60ce68d700 1 --2- 172.21.15.119:0/1317335967 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f60c8103ca0 0x7f60c8104100 secure :-1 s=READY pgs=35 cs=0 l=1 rev1=1 crypto rx=0x7f60bc0099c0 tx=0x7f60bc009cd0 comp rx=0 tx=0).stop 2024-04-23T10:23:04.308 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-23T10:23:04.309+0000 7f60ce68d700 1 -- 172.21.15.119:0/1317335967 shutdown_connections 2024-04-23T10:23:04.308 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-23T10:23:04.309+0000 7f60ce68d700 1 --2- 172.21.15.119:0/1317335967 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f60c8104640 0x7f60c810aef0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:23:04.308 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-23T10:23:04.309+0000 7f60ce68d700 1 --2- 172.21.15.119:0/1317335967 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f60c8102aa0 0x7f60c8102ec0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:23:04.308 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-23T10:23:04.309+0000 7f60ce68d700 1 --2- 172.21.15.119:0/1317335967 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f60c8103ca0 0x7f60c8104100 unknown :-1 s=CLOSED pgs=35 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:23:04.309 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-23T10:23:04.309+0000 7f60ce68d700 1 -- 172.21.15.119:0/1317335967 >> 172.21.15.119:0/1317335967 conn(0x7f60c80fe040 msgr2=0x7f60c8100480 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T10:23:04.309 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-23T10:23:04.309+0000 7f60ce68d700 1 -- 172.21.15.119:0/1317335967 shutdown_connections 2024-04-23T10:23:04.309 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-23T10:23:04.309+0000 7f60ce68d700 1 -- 172.21.15.119:0/1317335967 wait complete. 2024-04-23T10:23:04.309 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-23T10:23:04.310+0000 7f60ce68d700 1 Processor -- start 2024-04-23T10:23:04.309 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-23T10:23:04.310+0000 7f60ce68d700 1 -- start start 2024-04-23T10:23:04.311 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-23T10:23:04.310+0000 7f60ce68d700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f60c8102aa0 0x7f60c8198cf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T10:23:04.311 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-23T10:23:04.310+0000 7f60ce68d700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f60c8103ca0 0x7f60c8199230 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T10:23:04.311 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-23T10:23:04.310+0000 7f60ce68d700 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f60c8104640 0x7f60c81a0270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T10:23:04.312 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-23T10:23:04.310+0000 7f60ce68d700 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x7f60c810e5b0 con 0x7f60c8102aa0 2024-04-23T10:23:04.312 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-23T10:23:04.310+0000 7f60ce68d700 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 -- 0x7f60c810e430 con 0x7f60c8103ca0 2024-04-23T10:23:04.312 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-23T10:23:04.310+0000 7f60ce68d700 1 -- --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_getmap magic: 0 -- 0x7f60c810e730 con 0x7f60c8104640 2024-04-23T10:23:04.312 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-23T10:23:04.310+0000 7f60c77fe700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f60c8103ca0 0x7f60c8199230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T10:23:04.312 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-23T10:23:04.310+0000 7f60c77fe700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f60c8103ca0 0x7f60c8199230 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.119:3300/0 says I am v2:172.21.15.119:42618/0 (socket says 172.21.15.119:42618) 2024-04-23T10:23:04.312 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-23T10:23:04.310+0000 7f60c77fe700 1 -- 172.21.15.119:0/1062796981 learned_addr learned my addr 172.21.15.119:0/1062796981 (peer_addr_for_me v2:172.21.15.119:0/0) 2024-04-23T10:23:04.312 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-23T10:23:04.310+0000 7f60c7fff700 1 --2- 172.21.15.119:0/1062796981 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f60c8102aa0 0x7f60c8198cf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T10:23:04.312 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-23T10:23:04.311+0000 7f60ccc2a700 1 --2- 172.21.15.119:0/1062796981 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f60c8104640 0x7f60c81a0270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T10:23:04.312 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-23T10:23:04.311+0000 7f60c77fe700 1 -- 172.21.15.119:0/1062796981 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f60c8104640 msgr2=0x7f60c81a0270 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T10:23:04.312 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-23T10:23:04.311+0000 7f60c77fe700 1 --2- 172.21.15.119:0/1062796981 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f60c8104640 0x7f60c81a0270 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:23:04.312 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-23T10:23:04.311+0000 7f60c77fe700 1 -- 172.21.15.119:0/1062796981 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f60c8102aa0 msgr2=0x7f60c8198cf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T10:23:04.312 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-23T10:23:04.311+0000 7f60c77fe700 1 --2- 172.21.15.119:0/1062796981 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f60c8102aa0 0x7f60c8198cf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:23:04.312 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-23T10:23:04.311+0000 7f60c77fe700 1 -- 172.21.15.119:0/1062796981 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f60c81a0940 con 0x7f60c8103ca0 2024-04-23T10:23:04.313 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-23T10:23:04.311+0000 7f60ccc2a700 1 --2- 172.21.15.119:0/1062796981 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f60c8104640 0x7f60c81a0270 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-23T10:23:04.313 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-23T10:23:04.311+0000 7f60c77fe700 1 --2- 172.21.15.119:0/1062796981 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f60c8103ca0 0x7f60c8199230 secure :-1 s=READY pgs=36 cs=0 l=1 rev1=1 crypto rx=0x7f60bc003790 tx=0x7f60bc005ef0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T10:23:04.313 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-23T10:23:04.311+0000 7f60c57fa700 1 -- 172.21.15.119:0/1062796981 <== mon.2 v2:172.21.15.119:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f60bc027070 con 0x7f60c8103ca0 2024-04-23T10:23:04.313 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-23T10:23:04.311+0000 7f60ce68d700 1 -- 172.21.15.119:0/1062796981 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f60c81a0bd0 con 0x7f60c8103ca0 2024-04-23T10:23:04.313 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-23T10:23:04.311+0000 7f60c57fa700 1 -- 172.21.15.119:0/1062796981 <== mon.2 v2:172.21.15.119:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f60bc022470 con 0x7f60c8103ca0 2024-04-23T10:23:04.313 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-23T10:23:04.311+0000 7f60c57fa700 1 -- 172.21.15.119:0/1062796981 <== mon.2 v2:172.21.15.119:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f60bc00f6e0 con 0x7f60c8103ca0 2024-04-23T10:23:04.313 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-23T10:23:04.311+0000 7f60ce68d700 1 -- 172.21.15.119:0/1062796981 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f60c81a1100 con 0x7f60c8103ca0 2024-04-23T10:23:04.313 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-23T10:23:04.312+0000 7f60ce68d700 1 -- 172.21.15.119:0/1062796981 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f60c8069730 con 0x7f60c8103ca0 2024-04-23T10:23:04.313 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-23T10:23:04.313+0000 7f60c57fa700 1 -- 172.21.15.119:0/1062796981 <== mon.2 v2:172.21.15.119:3300/0 4 ==== mgrmap(e 16) ==== 138647+0+0 (secure 0 0 0) 0x7f60bc02a070 con 0x7f60c8103ca0 2024-04-23T10:23:04.314 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-23T10:23:04.313+0000 7f60c57fa700 1 --2- 172.21.15.119:0/1062796981 >> [v2:172.21.15.114:6800/465333808,v1:172.21.15.114:6801/465333808] conn(0x7f60b00a3c40 0x7f60b00a6100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T10:23:04.314 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-23T10:23:04.314+0000 7f60c57fa700 1 -- 172.21.15.119:0/1062796981 <== mon.2 v2:172.21.15.119:3300/0 5 ==== osd_map(70..70 src has 1..70) ==== 6261+0+0 (secure 0 0 0) 0x7f60bc0d0a70 con 0x7f60c8103ca0 2024-04-23T10:23:04.314 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-23T10:23:04.314+0000 7f60c7fff700 1 --2- 172.21.15.119:0/1062796981 >> [v2:172.21.15.114:6800/465333808,v1:172.21.15.114:6801/465333808] conn(0x7f60b00a3c40 0x7f60b00a6100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T10:23:04.314 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-23T10:23:04.314+0000 7f60c7fff700 1 --2- 172.21.15.119:0/1062796981 >> [v2:172.21.15.114:6800/465333808,v1:172.21.15.114:6801/465333808] conn(0x7f60b00a3c40 0x7f60b00a6100 secure :-1 s=READY pgs=113 cs=0 l=1 rev1=1 crypto rx=0x7f60c819a1d0 tx=0x7f60b8009190 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T10:23:04.316 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-23T10:23:04.318+0000 7f60c57fa700 1 -- 172.21.15.119:0/1062796981 <== mon.2 v2:172.21.15.119:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f60bc013070 con 0x7f60c8103ca0 2024-04-23T10:23:04.328 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 23 10:23:03 smithi204 ceph-mon[32649]: Detected new or changed devices on smithi114 2024-04-23T10:23:04.328 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 23 10:23:03 smithi204 ceph-mon[32649]: Adjusting osd_memory_target on smithi114 to 4272M 2024-04-23T10:23:04.524 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-23T10:23:04.525+0000 7f60ce68d700 1 -- 172.21.15.119:0/1062796981 --> [v2:172.21.15.114:6800/465333808,v1:172.21.15.114:6801/465333808] -- mgr_command(tid 0: {"prefix": "orch daemon add osd", "svc_arg": "smithi119:vg_nvme/lv_1", "target": ["mon-mgr", ""]}) -- 0x7f60c8062e80 con 0x7f60b00a3c40 2024-04-23T10:23:05.242 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 23 10:23:04 smithi119 ceph-mon[32608]: pgmap v219: 1 pgs: 1 active+clean; 577 KiB data, 301 MiB used, 894 GiB / 894 GiB avail 2024-04-23T10:23:05.242 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 23 10:23:04 smithi119 ceph-mon[32608]: from='mgr.14152 172.21.15.114:0/1529706138' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-04-23T10:23:05.242 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 23 10:23:04 smithi119 ceph-mon[32608]: from='mgr.14152 172.21.15.114:0/1529706138' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-04-23T10:23:05.242 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 23 10:23:04 smithi119 ceph-mon[32608]: from='mgr.14152 172.21.15.114:0/1529706138' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-23T10:23:05.254 INFO:journalctl@ceph.mon.a.smithi114.stdout:Apr 23 10:23:04 smithi114 ceph-mon[29255]: pgmap v219: 1 pgs: 1 active+clean; 577 KiB data, 301 MiB used, 894 GiB / 894 GiB avail 2024-04-23T10:23:05.254 INFO:journalctl@ceph.mon.a.smithi114.stdout:Apr 23 10:23:04 smithi114 ceph-mon[29255]: from='mgr.14152 172.21.15.114:0/1529706138' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-04-23T10:23:05.254 INFO:journalctl@ceph.mon.a.smithi114.stdout:Apr 23 10:23:04 smithi114 ceph-mon[29255]: from='mgr.14152 172.21.15.114:0/1529706138' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-04-23T10:23:05.254 INFO:journalctl@ceph.mon.a.smithi114.stdout:Apr 23 10:23:04 smithi114 ceph-mon[29255]: from='mgr.14152 172.21.15.114:0/1529706138' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-23T10:23:05.328 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 23 10:23:04 smithi204 ceph-mon[32649]: pgmap v219: 1 pgs: 1 active+clean; 577 KiB data, 301 MiB used, 894 GiB / 894 GiB avail 2024-04-23T10:23:05.328 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 23 10:23:04 smithi204 ceph-mon[32649]: from='mgr.14152 172.21.15.114:0/1529706138' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-04-23T10:23:05.328 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 23 10:23:04 smithi204 ceph-mon[32649]: from='mgr.14152 172.21.15.114:0/1529706138' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-04-23T10:23:05.328 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 23 10:23:04 smithi204 ceph-mon[32649]: from='mgr.14152 172.21.15.114:0/1529706138' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-23T10:23:06.254 INFO:journalctl@ceph.mon.a.smithi114.stdout:Apr 23 10:23:05 smithi114 ceph-mon[29255]: from='client.24359 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi119:vg_nvme/lv_1", "target": ["mon-mgr", ""]}]: dispatch 2024-04-23T10:23:06.323 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 23 10:23:05 smithi119 ceph-mon[32608]: from='client.24359 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi119:vg_nvme/lv_1", "target": ["mon-mgr", ""]}]: dispatch 2024-04-23T10:23:06.327 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 23 10:23:05 smithi204 ceph-mon[32649]: from='client.24359 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi119:vg_nvme/lv_1", "target": ["mon-mgr", ""]}]: dispatch 2024-04-23T10:23:07.254 INFO:journalctl@ceph.mon.a.smithi114.stdout:Apr 23 10:23:06 smithi114 ceph-mon[29255]: pgmap v220: 1 pgs: 1 active+clean; 577 KiB data, 301 MiB used, 894 GiB / 894 GiB avail 2024-04-23T10:23:07.328 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 23 10:23:06 smithi204 ceph-mon[32649]: pgmap v220: 1 pgs: 1 active+clean; 577 KiB data, 301 MiB used, 894 GiB / 894 GiB avail 2024-04-23T10:23:07.492 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 23 10:23:06 smithi119 ceph-mon[32608]: pgmap v220: 1 pgs: 1 active+clean; 577 KiB data, 301 MiB used, 894 GiB / 894 GiB avail 2024-04-23T10:23:08.254 INFO:journalctl@ceph.mon.a.smithi114.stdout:Apr 23 10:23:07 smithi114 ceph-mon[29255]: from='client.? 172.21.15.119:0/3176563496' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "cd75a579-bf12-4600-ac09-a8bfce772ced"}]: dispatch 2024-04-23T10:23:08.254 INFO:journalctl@ceph.mon.a.smithi114.stdout:Apr 23 10:23:07 smithi114 ceph-mon[29255]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "cd75a579-bf12-4600-ac09-a8bfce772ced"}]: dispatch 2024-04-23T10:23:08.254 INFO:journalctl@ceph.mon.a.smithi114.stdout:Apr 23 10:23:07 smithi114 ceph-mon[29255]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "cd75a579-bf12-4600-ac09-a8bfce772ced"}]': finished 2024-04-23T10:23:08.255 INFO:journalctl@ceph.mon.a.smithi114.stdout:Apr 23 10:23:07 smithi114 ceph-mon[29255]: osdmap e71: 11 total, 10 up, 11 in 2024-04-23T10:23:08.255 INFO:journalctl@ceph.mon.a.smithi114.stdout:Apr 23 10:23:07 smithi114 ceph-mon[29255]: from='mgr.14152 172.21.15.114:0/1529706138' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 10}]: dispatch 2024-04-23T10:23:08.255 INFO:journalctl@ceph.mon.a.smithi114.stdout:Apr 23 10:23:07 smithi114 ceph-mon[29255]: from='client.? 172.21.15.119:0/1464397119' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-04-23T10:23:08.328 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 23 10:23:07 smithi204 ceph-mon[32649]: from='client.? 172.21.15.119:0/3176563496' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "cd75a579-bf12-4600-ac09-a8bfce772ced"}]: dispatch 2024-04-23T10:23:08.328 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 23 10:23:07 smithi204 ceph-mon[32649]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "cd75a579-bf12-4600-ac09-a8bfce772ced"}]: dispatch 2024-04-23T10:23:08.328 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 23 10:23:07 smithi204 ceph-mon[32649]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "cd75a579-bf12-4600-ac09-a8bfce772ced"}]': finished 2024-04-23T10:23:08.328 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 23 10:23:07 smithi204 ceph-mon[32649]: osdmap e71: 11 total, 10 up, 11 in 2024-04-23T10:23:08.328 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 23 10:23:07 smithi204 ceph-mon[32649]: from='mgr.14152 172.21.15.114:0/1529706138' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 10}]: dispatch 2024-04-23T10:23:08.328 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 23 10:23:07 smithi204 ceph-mon[32649]: from='client.? 172.21.15.119:0/1464397119' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-04-23T10:23:08.492 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 23 10:23:07 smithi119 ceph-mon[32608]: from='client.? 172.21.15.119:0/3176563496' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "cd75a579-bf12-4600-ac09-a8bfce772ced"}]: dispatch 2024-04-23T10:23:08.492 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 23 10:23:07 smithi119 ceph-mon[32608]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "cd75a579-bf12-4600-ac09-a8bfce772ced"}]: dispatch 2024-04-23T10:23:08.492 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 23 10:23:07 smithi119 ceph-mon[32608]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "cd75a579-bf12-4600-ac09-a8bfce772ced"}]': finished 2024-04-23T10:23:08.492 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 23 10:23:07 smithi119 ceph-mon[32608]: osdmap e71: 11 total, 10 up, 11 in 2024-04-23T10:23:08.492 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 23 10:23:07 smithi119 ceph-mon[32608]: from='mgr.14152 172.21.15.114:0/1529706138' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 10}]: dispatch 2024-04-23T10:23:08.492 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 23 10:23:07 smithi119 ceph-mon[32608]: from='client.? 172.21.15.119:0/1464397119' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-04-23T10:23:09.254 INFO:journalctl@ceph.mon.a.smithi114.stdout:Apr 23 10:23:08 smithi114 ceph-mon[29255]: pgmap v222: 1 pgs: 1 active+clean; 577 KiB data, 301 MiB used, 894 GiB / 894 GiB avail 2024-04-23T10:23:09.328 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 23 10:23:08 smithi204 ceph-mon[32649]: pgmap v222: 1 pgs: 1 active+clean; 577 KiB data, 301 MiB used, 894 GiB / 894 GiB avail 2024-04-23T10:23:09.492 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 23 10:23:09 smithi119 ceph-mon[32608]: pgmap v222: 1 pgs: 1 active+clean; 577 KiB data, 301 MiB used, 894 GiB / 894 GiB avail 2024-04-23T10:23:11.254 INFO:journalctl@ceph.mon.a.smithi114.stdout:Apr 23 10:23:11 smithi114 ceph-mon[29255]: pgmap v223: 1 pgs: 1 active+clean; 577 KiB data, 301 MiB used, 894 GiB / 894 GiB avail 2024-04-23T10:23:11.255 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 23 10:23:11 smithi119 ceph-mon[32608]: pgmap v223: 1 pgs: 1 active+clean; 577 KiB data, 301 MiB used, 894 GiB / 894 GiB avail 2024-04-23T10:23:11.328 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 23 10:23:11 smithi204 ceph-mon[32649]: pgmap v223: 1 pgs: 1 active+clean; 577 KiB data, 301 MiB used, 894 GiB / 894 GiB avail 2024-04-23T10:23:13.242 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 23 10:23:13 smithi119 ceph-mon[32608]: pgmap v224: 1 pgs: 1 active+clean; 577 KiB data, 301 MiB used, 894 GiB / 894 GiB avail 2024-04-23T10:23:13.254 INFO:journalctl@ceph.mon.a.smithi114.stdout:Apr 23 10:23:13 smithi114 ceph-mon[29255]: pgmap v224: 1 pgs: 1 active+clean; 577 KiB data, 301 MiB used, 894 GiB / 894 GiB avail 2024-04-23T10:23:13.328 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 23 10:23:13 smithi204 ceph-mon[32649]: pgmap v224: 1 pgs: 1 active+clean; 577 KiB data, 301 MiB used, 894 GiB / 894 GiB avail 2024-04-23T10:23:15.103 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 23 10:23:15 smithi119 ceph-mon[32608]: pgmap v225: 1 pgs: 1 active+clean; 577 KiB data, 301 MiB used, 894 GiB / 894 GiB avail 2024-04-23T10:23:15.328 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 23 10:23:15 smithi204 ceph-mon[32649]: pgmap v225: 1 pgs: 1 active+clean; 577 KiB data, 301 MiB used, 894 GiB / 894 GiB avail 2024-04-23T10:23:15.504 INFO:journalctl@ceph.mon.a.smithi114.stdout:Apr 23 10:23:15 smithi114 ceph-mon[29255]: pgmap v225: 1 pgs: 1 active+clean; 577 KiB data, 301 MiB used, 894 GiB / 894 GiB avail 2024-04-23T10:23:16.328 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 23 10:23:16 smithi204 ceph-mon[32649]: from='mgr.14152 172.21.15.114:0/1529706138' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.10"}]: dispatch 2024-04-23T10:23:16.328 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 23 10:23:16 smithi204 ceph-mon[32649]: from='mgr.14152 172.21.15.114:0/1529706138' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-23T10:23:16.329 INFO:journalctl@ceph.mon.a.smithi114.stdout:Apr 23 10:23:16 smithi114 ceph-mon[29255]: from='mgr.14152 172.21.15.114:0/1529706138' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.10"}]: dispatch 2024-04-23T10:23:16.330 INFO:journalctl@ceph.mon.a.smithi114.stdout:Apr 23 10:23:16 smithi114 ceph-mon[29255]: from='mgr.14152 172.21.15.114:0/1529706138' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-23T10:23:16.492 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 23 10:23:16 smithi119 ceph-mon[32608]: from='mgr.14152 172.21.15.114:0/1529706138' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.10"}]: dispatch 2024-04-23T10:23:16.492 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 23 10:23:16 smithi119 ceph-mon[32608]: from='mgr.14152 172.21.15.114:0/1529706138' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-23T10:23:17.265 INFO:journalctl@ceph.mon.a.smithi114.stdout:Apr 23 10:23:17 smithi114 ceph-mon[29255]: Deploying daemon osd.10 on smithi119 2024-04-23T10:23:17.265 INFO:journalctl@ceph.mon.a.smithi114.stdout:Apr 23 10:23:17 smithi114 ceph-mon[29255]: pgmap v226: 1 pgs: 1 active+clean; 577 KiB data, 301 MiB used, 894 GiB / 894 GiB avail 2024-04-23T10:23:17.328 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 23 10:23:17 smithi204 ceph-mon[32649]: Deploying daemon osd.10 on smithi119 2024-04-23T10:23:17.328 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 23 10:23:17 smithi204 ceph-mon[32649]: pgmap v226: 1 pgs: 1 active+clean; 577 KiB data, 301 MiB used, 894 GiB / 894 GiB avail 2024-04-23T10:23:17.348 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 23 10:23:17 smithi119 ceph-mon[32608]: Deploying daemon osd.10 on smithi119 2024-04-23T10:23:17.348 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 23 10:23:17 smithi119 ceph-mon[32608]: pgmap v226: 1 pgs: 1 active+clean; 577 KiB data, 301 MiB used, 894 GiB / 894 GiB avail 2024-04-23T10:23:19.328 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 23 10:23:19 smithi204 ceph-mon[32649]: pgmap v227: 1 pgs: 1 active+clean; 577 KiB data, 301 MiB used, 894 GiB / 894 GiB avail 2024-04-23T10:23:19.465 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 23 10:23:19 smithi119 ceph-mon[32608]: pgmap v227: 1 pgs: 1 active+clean; 577 KiB data, 301 MiB used, 894 GiB / 894 GiB avail 2024-04-23T10:23:19.504 INFO:journalctl@ceph.mon.a.smithi114.stdout:Apr 23 10:23:19 smithi114 ceph-mon[29255]: pgmap v227: 1 pgs: 1 active+clean; 577 KiB data, 301 MiB used, 894 GiB / 894 GiB avail 2024-04-23T10:23:21.328 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 23 10:23:21 smithi204 ceph-mon[32649]: pgmap v228: 1 pgs: 1 active+clean; 577 KiB data, 301 MiB used, 894 GiB / 894 GiB avail 2024-04-23T10:23:21.328 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 23 10:23:21 smithi204 ceph-mon[32649]: from='mgr.14152 172.21.15.114:0/1529706138' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-23T10:23:21.328 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 23 10:23:21 smithi204 ceph-mon[32649]: from='mgr.14152 172.21.15.114:0/1529706138' entity='mgr.x' 2024-04-23T10:23:21.328 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 23 10:23:21 smithi204 ceph-mon[32649]: from='mgr.14152 172.21.15.114:0/1529706138' entity='mgr.x' 2024-04-23T10:23:21.403 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 23 10:23:21 smithi119 ceph-mon[32608]: pgmap v228: 1 pgs: 1 active+clean; 577 KiB data, 301 MiB used, 894 GiB / 894 GiB avail 2024-04-23T10:23:21.403 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 23 10:23:21 smithi119 ceph-mon[32608]: from='mgr.14152 172.21.15.114:0/1529706138' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-23T10:23:21.403 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 23 10:23:21 smithi119 ceph-mon[32608]: from='mgr.14152 172.21.15.114:0/1529706138' entity='mgr.x' 2024-04-23T10:23:21.403 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 23 10:23:21 smithi119 ceph-mon[32608]: from='mgr.14152 172.21.15.114:0/1529706138' entity='mgr.x' 2024-04-23T10:23:21.504 INFO:journalctl@ceph.mon.a.smithi114.stdout:Apr 23 10:23:21 smithi114 ceph-mon[29255]: pgmap v228: 1 pgs: 1 active+clean; 577 KiB data, 301 MiB used, 894 GiB / 894 GiB avail 2024-04-23T10:23:21.504 INFO:journalctl@ceph.mon.a.smithi114.stdout:Apr 23 10:23:21 smithi114 ceph-mon[29255]: from='mgr.14152 172.21.15.114:0/1529706138' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-23T10:23:21.504 INFO:journalctl@ceph.mon.a.smithi114.stdout:Apr 23 10:23:21 smithi114 ceph-mon[29255]: from='mgr.14152 172.21.15.114:0/1529706138' entity='mgr.x' 2024-04-23T10:23:21.504 INFO:journalctl@ceph.mon.a.smithi114.stdout:Apr 23 10:23:21 smithi114 ceph-mon[29255]: from='mgr.14152 172.21.15.114:0/1529706138' entity='mgr.x' 2024-04-23T10:23:23.282 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 23 10:23:23 smithi119 ceph-mon[32608]: pgmap v229: 1 pgs: 1 active+clean; 577 KiB data, 301 MiB used, 894 GiB / 894 GiB avail 2024-04-23T10:23:23.327 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 23 10:23:23 smithi204 ceph-mon[32649]: pgmap v229: 1 pgs: 1 active+clean; 577 KiB data, 301 MiB used, 894 GiB / 894 GiB avail 2024-04-23T10:23:23.504 INFO:journalctl@ceph.mon.a.smithi114.stdout:Apr 23 10:23:23 smithi114 ceph-mon[29255]: pgmap v229: 1 pgs: 1 active+clean; 577 KiB data, 301 MiB used, 894 GiB / 894 GiB avail 2024-04-23T10:23:23.988 INFO:teuthology.orchestra.run.smithi119.stdout:Created osd(s) 10 on host 'smithi119' 2024-04-23T10:23:23.988 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-23T10:23:23.988+0000 7f60c57fa700 1 -- 172.21.15.119:0/1062796981 <== mgr.14152 v2:172.21.15.114:6800/465333808 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+38 (secure 0 0 0) 0x7f60c8062e80 con 0x7f60b00a3c40 2024-04-23T10:23:23.990 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-23T10:23:23.990+0000 7f60ce68d700 1 -- 172.21.15.119:0/1062796981 >> [v2:172.21.15.114:6800/465333808,v1:172.21.15.114:6801/465333808] conn(0x7f60b00a3c40 msgr2=0x7f60b00a6100 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T10:23:23.990 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-23T10:23:23.990+0000 7f60ce68d700 1 --2- 172.21.15.119:0/1062796981 >> [v2:172.21.15.114:6800/465333808,v1:172.21.15.114:6801/465333808] conn(0x7f60b00a3c40 0x7f60b00a6100 secure :-1 s=READY pgs=113 cs=0 l=1 rev1=1 crypto rx=0x7f60c819a1d0 tx=0x7f60b8009190 comp rx=0 tx=0).stop 2024-04-23T10:23:23.990 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-23T10:23:23.991+0000 7f60ce68d700 1 -- 172.21.15.119:0/1062796981 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f60c8103ca0 msgr2=0x7f60c8199230 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T10:23:23.990 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-23T10:23:23.991+0000 7f60ce68d700 1 --2- 172.21.15.119:0/1062796981 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f60c8103ca0 0x7f60c8199230 secure :-1 s=READY pgs=36 cs=0 l=1 rev1=1 crypto rx=0x7f60bc003790 tx=0x7f60bc005ef0 comp rx=0 tx=0).stop 2024-04-23T10:23:23.990 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-23T10:23:23.991+0000 7f60ce68d700 1 -- 172.21.15.119:0/1062796981 shutdown_connections 2024-04-23T10:23:23.990 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-23T10:23:23.991+0000 7f60ce68d700 1 --2- 172.21.15.119:0/1062796981 >> [v2:172.21.15.114:6800/465333808,v1:172.21.15.114:6801/465333808] conn(0x7f60b00a3c40 0x7f60b00a6100 unknown :-1 s=CLOSED pgs=113 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:23:23.991 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-23T10:23:23.991+0000 7f60ce68d700 1 --2- 172.21.15.119:0/1062796981 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f60c8104640 0x7f60c81a0270 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:23:23.991 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-23T10:23:23.991+0000 7f60ce68d700 1 --2- 172.21.15.119:0/1062796981 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f60c8102aa0 0x7f60c8198cf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:23:23.991 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-23T10:23:23.991+0000 7f60ce68d700 1 --2- 172.21.15.119:0/1062796981 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f60c8103ca0 0x7f60c8199230 unknown :-1 s=CLOSED pgs=36 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:23:23.991 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-23T10:23:23.991+0000 7f60ce68d700 1 -- 172.21.15.119:0/1062796981 >> 172.21.15.119:0/1062796981 conn(0x7f60c80fe040 msgr2=0x7f60c8107060 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T10:23:23.991 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-23T10:23:23.991+0000 7f60ce68d700 1 -- 172.21.15.119:0/1062796981 shutdown_connections 2024-04-23T10:23:23.991 INFO:teuthology.orchestra.run.smithi119.stderr:2024-04-23T10:23:23.991+0000 7f60ce68d700 1 -- 172.21.15.119:0/1062796981 wait complete. 2024-04-23T10:23:24.845 DEBUG:teuthology.orchestra.run.smithi119:osd.10> sudo journalctl -f -n 0 -u ceph-aa240572-0159-11ef-bc93-c7b262605968@osd.10.service 2024-04-23T10:23:24.848 INFO:tasks.cephadm:Deploying osd.11 on smithi204 with /dev/vg_nvme/lv_1... 2024-04-23T10:23:24.848 DEBUG:teuthology.orchestra.run.smithi204:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d69f13d827952bb07bb9b7973bca14f97d764d6f shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid aa240572-0159-11ef-bc93-c7b262605968 -- ceph-volume lvm zap /dev/vg_nvme/lv_1 2024-04-23T10:23:25.095 INFO:teuthology.orchestra.run.smithi204.stderr:Inferring config /var/lib/ceph/aa240572-0159-11ef-bc93-c7b262605968/mon.c/config 2024-04-23T10:23:25.130 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 23 10:23:24 smithi204 ceph-mon[32649]: pgmap v230: 1 pgs: 1 active+clean; 577 KiB data, 301 MiB used, 894 GiB / 894 GiB avail 2024-04-23T10:23:25.130 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 23 10:23:24 smithi204 ceph-mon[32649]: from='mgr.14152 172.21.15.114:0/1529706138' entity='mgr.x' 2024-04-23T10:23:25.130 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 23 10:23:24 smithi204 ceph-mon[32649]: from='mgr.14152 172.21.15.114:0/1529706138' entity='mgr.x' 2024-04-23T10:23:25.130 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 23 10:23:24 smithi204 ceph-mon[32649]: from='osd.10 [v2:172.21.15.119:6824/1400187701,v1:172.21.15.119:6825/1400187701]' entity='osd.10' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["10"]}]: dispatch 2024-04-23T10:23:25.130 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 23 10:23:24 smithi204 ceph-mon[32649]: from='osd.10 ' entity='osd.10' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["10"]}]: dispatch 2024-04-23T10:23:25.242 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 23 10:23:24 smithi119 ceph-mon[32608]: pgmap v230: 1 pgs: 1 active+clean; 577 KiB data, 301 MiB used, 894 GiB / 894 GiB avail 2024-04-23T10:23:25.242 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 23 10:23:24 smithi119 ceph-mon[32608]: from='mgr.14152 172.21.15.114:0/1529706138' entity='mgr.x' 2024-04-23T10:23:25.242 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 23 10:23:24 smithi119 ceph-mon[32608]: from='mgr.14152 172.21.15.114:0/1529706138' entity='mgr.x' 2024-04-23T10:23:25.242 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 23 10:23:24 smithi119 ceph-mon[32608]: from='osd.10 [v2:172.21.15.119:6824/1400187701,v1:172.21.15.119:6825/1400187701]' entity='osd.10' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["10"]}]: dispatch 2024-04-23T10:23:25.242 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 23 10:23:24 smithi119 ceph-mon[32608]: from='osd.10 ' entity='osd.10' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["10"]}]: dispatch 2024-04-23T10:23:25.254 INFO:journalctl@ceph.mon.a.smithi114.stdout:Apr 23 10:23:24 smithi114 ceph-mon[29255]: pgmap v230: 1 pgs: 1 active+clean; 577 KiB data, 301 MiB used, 894 GiB / 894 GiB avail 2024-04-23T10:23:25.254 INFO:journalctl@ceph.mon.a.smithi114.stdout:Apr 23 10:23:24 smithi114 ceph-mon[29255]: from='mgr.14152 172.21.15.114:0/1529706138' entity='mgr.x' 2024-04-23T10:23:25.254 INFO:journalctl@ceph.mon.a.smithi114.stdout:Apr 23 10:23:24 smithi114 ceph-mon[29255]: from='mgr.14152 172.21.15.114:0/1529706138' entity='mgr.x' 2024-04-23T10:23:25.254 INFO:journalctl@ceph.mon.a.smithi114.stdout:Apr 23 10:23:24 smithi114 ceph-mon[29255]: from='osd.10 [v2:172.21.15.119:6824/1400187701,v1:172.21.15.119:6825/1400187701]' entity='osd.10' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["10"]}]: dispatch 2024-04-23T10:23:25.254 INFO:journalctl@ceph.mon.a.smithi114.stdout:Apr 23 10:23:24 smithi114 ceph-mon[29255]: from='osd.10 ' entity='osd.10' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["10"]}]: dispatch 2024-04-23T10:23:26.239 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 23 10:23:26 smithi204 ceph-mon[32649]: from='osd.10 ' entity='osd.10' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["10"]}]': finished 2024-04-23T10:23:26.239 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 23 10:23:26 smithi204 ceph-mon[32649]: from='osd.10 [v2:172.21.15.119:6824/1400187701,v1:172.21.15.119:6825/1400187701]' entity='osd.10' cmd=[{"prefix": "osd crush create-or-move", "id": 10, "weight":0.0873, "args": ["host=smithi119", "root=default"]}]: dispatch 2024-04-23T10:23:26.239 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 23 10:23:26 smithi204 ceph-mon[32649]: osdmap e72: 11 total, 10 up, 11 in 2024-04-23T10:23:26.240 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 23 10:23:26 smithi204 ceph-mon[32649]: from='mgr.14152 172.21.15.114:0/1529706138' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 10}]: dispatch 2024-04-23T10:23:26.240 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 23 10:23:26 smithi204 ceph-mon[32649]: from='osd.10 ' entity='osd.10' cmd=[{"prefix": "osd crush create-or-move", "id": 10, "weight":0.0873, "args": ["host=smithi119", "root=default"]}]: dispatch 2024-04-23T10:23:26.240 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 23 10:23:26 smithi204 ceph-mon[32649]: from='mgr.14152 172.21.15.114:0/1529706138' entity='mgr.x' 2024-04-23T10:23:26.240 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 23 10:23:26 smithi204 ceph-mon[32649]: from='mgr.14152 172.21.15.114:0/1529706138' entity='mgr.x' 2024-04-23T10:23:26.492 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 23 10:23:26 smithi119 ceph-mon[32608]: from='osd.10 ' entity='osd.10' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["10"]}]': finished 2024-04-23T10:23:26.492 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 23 10:23:26 smithi119 ceph-mon[32608]: from='osd.10 [v2:172.21.15.119:6824/1400187701,v1:172.21.15.119:6825/1400187701]' entity='osd.10' cmd=[{"prefix": "osd crush create-or-move", "id": 10, "weight":0.0873, "args": ["host=smithi119", "root=default"]}]: dispatch 2024-04-23T10:23:26.492 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 23 10:23:26 smithi119 ceph-mon[32608]: osdmap e72: 11 total, 10 up, 11 in 2024-04-23T10:23:26.492 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 23 10:23:26 smithi119 ceph-mon[32608]: from='mgr.14152 172.21.15.114:0/1529706138' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 10}]: dispatch 2024-04-23T10:23:26.492 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 23 10:23:26 smithi119 ceph-mon[32608]: from='osd.10 ' entity='osd.10' cmd=[{"prefix": "osd crush create-or-move", "id": 10, "weight":0.0873, "args": ["host=smithi119", "root=default"]}]: dispatch 2024-04-23T10:23:26.492 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 23 10:23:26 smithi119 ceph-mon[32608]: from='mgr.14152 172.21.15.114:0/1529706138' entity='mgr.x' 2024-04-23T10:23:26.492 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 23 10:23:26 smithi119 ceph-mon[32608]: from='mgr.14152 172.21.15.114:0/1529706138' entity='mgr.x' 2024-04-23T10:23:26.492 INFO:journalctl@ceph.osd.10.smithi119.stdout:Apr 23 10:23:26 smithi119 ceph-aa240572-0159-11ef-bc93-c7b262605968-osd-10[57454]: 2024-04-23T10:23:26.034+0000 7fb1873c9700 -1 osd.10 0 waiting for initial osdmap 2024-04-23T10:23:26.492 INFO:journalctl@ceph.osd.10.smithi119.stdout:Apr 23 10:23:26 smithi119 ceph-aa240572-0159-11ef-bc93-c7b262605968-osd-10[57454]: 2024-04-23T10:23:26.047+0000 7fb1831bc700 -1 osd.10 73 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-04-23T10:23:26.504 INFO:journalctl@ceph.mon.a.smithi114.stdout:Apr 23 10:23:26 smithi114 ceph-mon[29255]: from='osd.10 ' entity='osd.10' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["10"]}]': finished 2024-04-23T10:23:26.504 INFO:journalctl@ceph.mon.a.smithi114.stdout:Apr 23 10:23:26 smithi114 ceph-mon[29255]: from='osd.10 [v2:172.21.15.119:6824/1400187701,v1:172.21.15.119:6825/1400187701]' entity='osd.10' cmd=[{"prefix": "osd crush create-or-move", "id": 10, "weight":0.0873, "args": ["host=smithi119", "root=default"]}]: dispatch 2024-04-23T10:23:26.504 INFO:journalctl@ceph.mon.a.smithi114.stdout:Apr 23 10:23:26 smithi114 ceph-mon[29255]: osdmap e72: 11 total, 10 up, 11 in 2024-04-23T10:23:26.504 INFO:journalctl@ceph.mon.a.smithi114.stdout:Apr 23 10:23:26 smithi114 ceph-mon[29255]: from='mgr.14152 172.21.15.114:0/1529706138' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 10}]: dispatch 2024-04-23T10:23:26.504 INFO:journalctl@ceph.mon.a.smithi114.stdout:Apr 23 10:23:26 smithi114 ceph-mon[29255]: from='osd.10 ' entity='osd.10' cmd=[{"prefix": "osd crush create-or-move", "id": 10, "weight":0.0873, "args": ["host=smithi119", "root=default"]}]: dispatch 2024-04-23T10:23:26.504 INFO:journalctl@ceph.mon.a.smithi114.stdout:Apr 23 10:23:26 smithi114 ceph-mon[29255]: from='mgr.14152 172.21.15.114:0/1529706138' entity='mgr.x' 2024-04-23T10:23:26.504 INFO:journalctl@ceph.mon.a.smithi114.stdout:Apr 23 10:23:26 smithi114 ceph-mon[29255]: from='mgr.14152 172.21.15.114:0/1529706138' entity='mgr.x' 2024-04-23T10:23:27.082 INFO:teuthology.orchestra.run.smithi204.stderr:--> Zapping: /dev/vg_nvme/lv_1 2024-04-23T10:23:27.082 INFO:teuthology.orchestra.run.smithi204.stderr:Running command: /usr/bin/dd if=/dev/zero of=/dev/vg_nvme/lv_1 bs=1M count=10 conv=fsync 2024-04-23T10:23:27.082 INFO:teuthology.orchestra.run.smithi204.stderr: stderr: 10+0 records in 2024-04-23T10:23:27.082 INFO:teuthology.orchestra.run.smithi204.stderr:10+0 records out 2024-04-23T10:23:27.083 INFO:teuthology.orchestra.run.smithi204.stderr: stderr: 10485760 bytes (10 MB, 10 MiB) copied, 0.0113454 s, 924 MB/s 2024-04-23T10:23:27.083 INFO:teuthology.orchestra.run.smithi204.stderr:--> Zapping successful for: 2024-04-23T10:23:27.083 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 23 10:23:27 smithi204 ceph-mon[32649]: pgmap v232: 1 pgs: 1 active+clean; 577 KiB data, 301 MiB used, 894 GiB / 894 GiB avail 2024-04-23T10:23:27.083 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 23 10:23:27 smithi204 ceph-mon[32649]: from='osd.10 ' entity='osd.10' cmd='[{"prefix": "osd crush create-or-move", "id": 10, "weight":0.0873, "args": ["host=smithi119", "root=default"]}]': finished 2024-04-23T10:23:27.083 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 23 10:23:27 smithi204 ceph-mon[32649]: osdmap e73: 11 total, 10 up, 11 in 2024-04-23T10:23:27.083 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 23 10:23:27 smithi204 ceph-mon[32649]: from='mgr.14152 172.21.15.114:0/1529706138' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 10}]: dispatch 2024-04-23T10:23:27.083 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 23 10:23:27 smithi204 ceph-mon[32649]: from='mgr.14152 172.21.15.114:0/1529706138' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 10}]: dispatch 2024-04-23T10:23:27.083 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 23 10:23:27 smithi204 ceph-mon[32649]: osd.10 [v2:172.21.15.119:6824/1400187701,v1:172.21.15.119:6825/1400187701] boot 2024-04-23T10:23:27.083 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 23 10:23:27 smithi204 ceph-mon[32649]: osdmap e74: 11 total, 11 up, 11 in 2024-04-23T10:23:27.083 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 23 10:23:27 smithi204 ceph-mon[32649]: from='mgr.14152 172.21.15.114:0/1529706138' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 10}]: dispatch 2024-04-23T10:23:27.239 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 23 10:23:27 smithi119 ceph-mon[32608]: pgmap v232: 1 pgs: 1 active+clean; 577 KiB data, 301 MiB used, 894 GiB / 894 GiB avail 2024-04-23T10:23:27.239 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 23 10:23:27 smithi119 ceph-mon[32608]: from='osd.10 ' entity='osd.10' cmd='[{"prefix": "osd crush create-or-move", "id": 10, "weight":0.0873, "args": ["host=smithi119", "root=default"]}]': finished 2024-04-23T10:23:27.239 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 23 10:23:27 smithi119 ceph-mon[32608]: osdmap e73: 11 total, 10 up, 11 in 2024-04-23T10:23:27.240 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 23 10:23:27 smithi119 ceph-mon[32608]: from='mgr.14152 172.21.15.114:0/1529706138' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 10}]: dispatch 2024-04-23T10:23:27.240 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 23 10:23:27 smithi119 ceph-mon[32608]: from='mgr.14152 172.21.15.114:0/1529706138' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 10}]: dispatch 2024-04-23T10:23:27.240 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 23 10:23:27 smithi119 ceph-mon[32608]: osd.10 [v2:172.21.15.119:6824/1400187701,v1:172.21.15.119:6825/1400187701] boot 2024-04-23T10:23:27.240 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 23 10:23:27 smithi119 ceph-mon[32608]: osdmap e74: 11 total, 11 up, 11 in 2024-04-23T10:23:27.240 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 23 10:23:27 smithi119 ceph-mon[32608]: from='mgr.14152 172.21.15.114:0/1529706138' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 10}]: dispatch 2024-04-23T10:23:27.337 INFO:journalctl@ceph.mon.a.smithi114.stdout:Apr 23 10:23:27 smithi114 ceph-mon[29255]: pgmap v232: 1 pgs: 1 active+clean; 577 KiB data, 301 MiB used, 894 GiB / 894 GiB avail 2024-04-23T10:23:27.337 INFO:journalctl@ceph.mon.a.smithi114.stdout:Apr 23 10:23:27 smithi114 ceph-mon[29255]: from='osd.10 ' entity='osd.10' cmd='[{"prefix": "osd crush create-or-move", "id": 10, "weight":0.0873, "args": ["host=smithi119", "root=default"]}]': finished 2024-04-23T10:23:27.338 INFO:journalctl@ceph.mon.a.smithi114.stdout:Apr 23 10:23:27 smithi114 ceph-mon[29255]: osdmap e73: 11 total, 10 up, 11 in 2024-04-23T10:23:27.338 INFO:journalctl@ceph.mon.a.smithi114.stdout:Apr 23 10:23:27 smithi114 ceph-mon[29255]: from='mgr.14152 172.21.15.114:0/1529706138' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 10}]: dispatch 2024-04-23T10:23:27.338 INFO:journalctl@ceph.mon.a.smithi114.stdout:Apr 23 10:23:27 smithi114 ceph-mon[29255]: from='mgr.14152 172.21.15.114:0/1529706138' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 10}]: dispatch 2024-04-23T10:23:27.338 INFO:journalctl@ceph.mon.a.smithi114.stdout:Apr 23 10:23:27 smithi114 ceph-mon[29255]: osd.10 [v2:172.21.15.119:6824/1400187701,v1:172.21.15.119:6825/1400187701] boot 2024-04-23T10:23:27.338 INFO:journalctl@ceph.mon.a.smithi114.stdout:Apr 23 10:23:27 smithi114 ceph-mon[29255]: osdmap e74: 11 total, 11 up, 11 in 2024-04-23T10:23:27.338 INFO:journalctl@ceph.mon.a.smithi114.stdout:Apr 23 10:23:27 smithi114 ceph-mon[29255]: from='mgr.14152 172.21.15.114:0/1529706138' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 10}]: dispatch 2024-04-23T10:23:27.835 DEBUG:teuthology.orchestra.run.smithi204:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d69f13d827952bb07bb9b7973bca14f97d764d6f shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid aa240572-0159-11ef-bc93-c7b262605968 -- ceph orch daemon add osd smithi204:vg_nvme/lv_1 2024-04-23T10:23:28.074 INFO:teuthology.orchestra.run.smithi204.stderr:Inferring config /var/lib/ceph/aa240572-0159-11ef-bc93-c7b262605968/mon.c/config 2024-04-23T10:23:28.181 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 23 10:23:28 smithi119 ceph-mon[32608]: purged_snaps scrub starts 2024-04-23T10:23:28.181 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 23 10:23:28 smithi119 ceph-mon[32608]: purged_snaps scrub ok 2024-04-23T10:23:28.327 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 23 10:23:28 smithi204 ceph-mon[32649]: purged_snaps scrub starts 2024-04-23T10:23:28.328 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 23 10:23:28 smithi204 ceph-mon[32649]: purged_snaps scrub ok 2024-04-23T10:23:28.504 INFO:journalctl@ceph.mon.a.smithi114.stdout:Apr 23 10:23:28 smithi114 ceph-mon[29255]: purged_snaps scrub starts 2024-04-23T10:23:28.504 INFO:journalctl@ceph.mon.a.smithi114.stdout:Apr 23 10:23:28 smithi114 ceph-mon[29255]: purged_snaps scrub ok 2024-04-23T10:23:29.296 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 23 10:23:29 smithi204 ceph-mon[32649]: pgmap v235: 1 pgs: 1 remapped+peering; 577 KiB data, 328 MiB used, 983 GiB / 983 GiB avail 2024-04-23T10:23:29.296 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 23 10:23:29 smithi204 ceph-mon[32649]: osdmap e75: 11 total, 11 up, 11 in 2024-04-23T10:23:29.296 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 23 10:23:29 smithi204 ceph-mon[32649]: from='mgr.14152 172.21.15.114:0/1529706138' entity='mgr.x' 2024-04-23T10:23:29.297 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 23 10:23:29 smithi204 ceph-mon[32649]: from='mgr.14152 172.21.15.114:0/1529706138' entity='mgr.x' 2024-04-23T10:23:29.297 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 23 10:23:29 smithi204 ceph-mon[32649]: from='mgr.14152 172.21.15.114:0/1529706138' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-04-23T10:23:29.297 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 23 10:23:29 smithi204 ceph-mon[32649]: from='mgr.14152 172.21.15.114:0/1529706138' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-04-23T10:23:29.297 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 23 10:23:29 smithi204 ceph-mon[32649]: from='mgr.14152 172.21.15.114:0/1529706138' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-04-23T10:23:29.297 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 23 10:23:29 smithi204 ceph-mon[32649]: from='mgr.14152 172.21.15.114:0/1529706138' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2024-04-23T10:23:29.297 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 23 10:23:29 smithi204 ceph-mon[32649]: from='mgr.14152 172.21.15.114:0/1529706138' entity='mgr.x' 2024-04-23T10:23:29.297 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 23 10:23:29 smithi204 ceph-mon[32649]: from='mgr.14152 172.21.15.114:0/1529706138' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-23T10:23:29.297 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 23 10:23:29 smithi204 ceph-mon[32649]: from='mgr.14152 172.21.15.114:0/1529706138' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-23T10:23:29.297 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 23 10:23:29 smithi204 ceph-mon[32649]: from='mgr.14152 172.21.15.114:0/1529706138' entity='mgr.x' 2024-04-23T10:23:29.297 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 23 10:23:29 smithi204 ceph-mon[32649]: from='mgr.14152 172.21.15.114:0/1529706138' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-23T10:23:29.297 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 23 10:23:29 smithi204 ceph-mon[32649]: from='mgr.14152 172.21.15.114:0/1529706138' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-23T10:23:29.297 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 23 10:23:29 smithi204 ceph-mon[32649]: from='mgr.14152 172.21.15.114:0/1529706138' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-23T10:23:29.297 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 23 10:23:29 smithi204 ceph-mon[32649]: from='mgr.14152 172.21.15.114:0/1529706138' entity='mgr.x' 2024-04-23T10:23:29.492 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 23 10:23:29 smithi119 ceph-mon[32608]: pgmap v235: 1 pgs: 1 remapped+peering; 577 KiB data, 328 MiB used, 983 GiB / 983 GiB avail 2024-04-23T10:23:29.492 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 23 10:23:29 smithi119 ceph-mon[32608]: osdmap e75: 11 total, 11 up, 11 in 2024-04-23T10:23:29.492 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 23 10:23:29 smithi119 ceph-mon[32608]: from='mgr.14152 172.21.15.114:0/1529706138' entity='mgr.x' 2024-04-23T10:23:29.492 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 23 10:23:29 smithi119 ceph-mon[32608]: from='mgr.14152 172.21.15.114:0/1529706138' entity='mgr.x' 2024-04-23T10:23:29.492 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 23 10:23:29 smithi119 ceph-mon[32608]: from='mgr.14152 172.21.15.114:0/1529706138' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-04-23T10:23:29.492 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 23 10:23:29 smithi119 ceph-mon[32608]: from='mgr.14152 172.21.15.114:0/1529706138' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-04-23T10:23:29.492 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 23 10:23:29 smithi119 ceph-mon[32608]: from='mgr.14152 172.21.15.114:0/1529706138' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-04-23T10:23:29.492 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 23 10:23:29 smithi119 ceph-mon[32608]: from='mgr.14152 172.21.15.114:0/1529706138' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2024-04-23T10:23:29.492 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 23 10:23:29 smithi119 ceph-mon[32608]: from='mgr.14152 172.21.15.114:0/1529706138' entity='mgr.x' 2024-04-23T10:23:29.492 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 23 10:23:29 smithi119 ceph-mon[32608]: from='mgr.14152 172.21.15.114:0/1529706138' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-23T10:23:29.492 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 23 10:23:29 smithi119 ceph-mon[32608]: from='mgr.14152 172.21.15.114:0/1529706138' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-23T10:23:29.492 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 23 10:23:29 smithi119 ceph-mon[32608]: from='mgr.14152 172.21.15.114:0/1529706138' entity='mgr.x' 2024-04-23T10:23:29.493 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 23 10:23:29 smithi119 ceph-mon[32608]: from='mgr.14152 172.21.15.114:0/1529706138' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-23T10:23:29.493 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 23 10:23:29 smithi119 ceph-mon[32608]: from='mgr.14152 172.21.15.114:0/1529706138' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-23T10:23:29.493 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 23 10:23:29 smithi119 ceph-mon[32608]: from='mgr.14152 172.21.15.114:0/1529706138' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-23T10:23:29.493 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 23 10:23:29 smithi119 ceph-mon[32608]: from='mgr.14152 172.21.15.114:0/1529706138' entity='mgr.x' 2024-04-23T10:23:29.504 INFO:journalctl@ceph.mon.a.smithi114.stdout:Apr 23 10:23:29 smithi114 ceph-mon[29255]: pgmap v235: 1 pgs: 1 remapped+peering; 577 KiB data, 328 MiB used, 983 GiB / 983 GiB avail 2024-04-23T10:23:29.504 INFO:journalctl@ceph.mon.a.smithi114.stdout:Apr 23 10:23:29 smithi114 ceph-mon[29255]: osdmap e75: 11 total, 11 up, 11 in 2024-04-23T10:23:29.504 INFO:journalctl@ceph.mon.a.smithi114.stdout:Apr 23 10:23:29 smithi114 ceph-mon[29255]: from='mgr.14152 172.21.15.114:0/1529706138' entity='mgr.x' 2024-04-23T10:23:29.504 INFO:journalctl@ceph.mon.a.smithi114.stdout:Apr 23 10:23:29 smithi114 ceph-mon[29255]: from='mgr.14152 172.21.15.114:0/1529706138' entity='mgr.x' 2024-04-23T10:23:29.504 INFO:journalctl@ceph.mon.a.smithi114.stdout:Apr 23 10:23:29 smithi114 ceph-mon[29255]: from='mgr.14152 172.21.15.114:0/1529706138' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-04-23T10:23:29.504 INFO:journalctl@ceph.mon.a.smithi114.stdout:Apr 23 10:23:29 smithi114 ceph-mon[29255]: from='mgr.14152 172.21.15.114:0/1529706138' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-04-23T10:23:29.505 INFO:journalctl@ceph.mon.a.smithi114.stdout:Apr 23 10:23:29 smithi114 ceph-mon[29255]: from='mgr.14152 172.21.15.114:0/1529706138' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-04-23T10:23:29.505 INFO:journalctl@ceph.mon.a.smithi114.stdout:Apr 23 10:23:29 smithi114 ceph-mon[29255]: from='mgr.14152 172.21.15.114:0/1529706138' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2024-04-23T10:23:29.505 INFO:journalctl@ceph.mon.a.smithi114.stdout:Apr 23 10:23:29 smithi114 ceph-mon[29255]: from='mgr.14152 172.21.15.114:0/1529706138' entity='mgr.x' 2024-04-23T10:23:29.505 INFO:journalctl@ceph.mon.a.smithi114.stdout:Apr 23 10:23:29 smithi114 ceph-mon[29255]: from='mgr.14152 172.21.15.114:0/1529706138' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-23T10:23:29.505 INFO:journalctl@ceph.mon.a.smithi114.stdout:Apr 23 10:23:29 smithi114 ceph-mon[29255]: from='mgr.14152 172.21.15.114:0/1529706138' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-23T10:23:29.505 INFO:journalctl@ceph.mon.a.smithi114.stdout:Apr 23 10:23:29 smithi114 ceph-mon[29255]: from='mgr.14152 172.21.15.114:0/1529706138' entity='mgr.x' 2024-04-23T10:23:29.505 INFO:journalctl@ceph.mon.a.smithi114.stdout:Apr 23 10:23:29 smithi114 ceph-mon[29255]: from='mgr.14152 172.21.15.114:0/1529706138' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-23T10:23:29.505 INFO:journalctl@ceph.mon.a.smithi114.stdout:Apr 23 10:23:29 smithi114 ceph-mon[29255]: from='mgr.14152 172.21.15.114:0/1529706138' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-23T10:23:29.505 INFO:journalctl@ceph.mon.a.smithi114.stdout:Apr 23 10:23:29 smithi114 ceph-mon[29255]: from='mgr.14152 172.21.15.114:0/1529706138' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-23T10:23:29.505 INFO:journalctl@ceph.mon.a.smithi114.stdout:Apr 23 10:23:29 smithi114 ceph-mon[29255]: from='mgr.14152 172.21.15.114:0/1529706138' entity='mgr.x' 2024-04-23T10:23:29.788 INFO:teuthology.orchestra.run.smithi204.stderr:2024-04-23T10:23:29.787+0000 7f5cd545f700 1 -- 172.21.15.204:0/2445466445 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f5cd0103ce0 msgr2=0x7f5cd0104140 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T10:23:29.788 INFO:teuthology.orchestra.run.smithi204.stderr:2024-04-23T10:23:29.787+0000 7f5cd545f700 1 --2- 172.21.15.204:0/2445466445 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f5cd0103ce0 0x7f5cd0104140 secure :-1 s=READY pgs=37 cs=0 l=1 rev1=1 crypto rx=0x7f5cb8009990 tx=0x7f5cb8009ca0 comp rx=0 tx=0).stop 2024-04-23T10:23:29.788 INFO:teuthology.orchestra.run.smithi204.stderr:2024-04-23T10:23:29.787+0000 7f5cd545f700 1 -- 172.21.15.204:0/2445466445 shutdown_connections 2024-04-23T10:23:29.788 INFO:teuthology.orchestra.run.smithi204.stderr:2024-04-23T10:23:29.787+0000 7f5cd545f700 1 --2- 172.21.15.204:0/2445466445 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f5cd0104680 0x7f5cd010af30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:23:29.788 INFO:teuthology.orchestra.run.smithi204.stderr:2024-04-23T10:23:29.787+0000 7f5cd545f700 1 --2- 172.21.15.204:0/2445466445 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f5cd0102ae0 0x7f5cd0102f00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:23:29.788 INFO:teuthology.orchestra.run.smithi204.stderr:2024-04-23T10:23:29.787+0000 7f5cd545f700 1 --2- 172.21.15.204:0/2445466445 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f5cd0103ce0 0x7f5cd0104140 unknown :-1 s=CLOSED pgs=37 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:23:29.788 INFO:teuthology.orchestra.run.smithi204.stderr:2024-04-23T10:23:29.787+0000 7f5cd545f700 1 -- 172.21.15.204:0/2445466445 >> 172.21.15.204:0/2445466445 conn(0x7f5cd00fe080 msgr2=0x7f5cd01004c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T10:23:29.788 INFO:teuthology.orchestra.run.smithi204.stderr:2024-04-23T10:23:29.787+0000 7f5cd545f700 1 -- 172.21.15.204:0/2445466445 shutdown_connections 2024-04-23T10:23:29.788 INFO:teuthology.orchestra.run.smithi204.stderr:2024-04-23T10:23:29.787+0000 7f5cd545f700 1 -- 172.21.15.204:0/2445466445 wait complete. 2024-04-23T10:23:29.788 INFO:teuthology.orchestra.run.smithi204.stderr:2024-04-23T10:23:29.788+0000 7f5cd545f700 1 Processor -- start 2024-04-23T10:23:29.788 INFO:teuthology.orchestra.run.smithi204.stderr:2024-04-23T10:23:29.788+0000 7f5cd545f700 1 -- start start 2024-04-23T10:23:29.788 INFO:teuthology.orchestra.run.smithi204.stderr:2024-04-23T10:23:29.788+0000 7f5cd545f700 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f5cd0102ae0 0x7f5cd0198b40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T10:23:29.788 INFO:teuthology.orchestra.run.smithi204.stderr:2024-04-23T10:23:29.788+0000 7f5cd545f700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f5cd0103ce0 0x7f5cd0199080 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T10:23:29.789 INFO:teuthology.orchestra.run.smithi204.stderr:2024-04-23T10:23:29.788+0000 7f5cd545f700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f5cd0104680 0x7f5cd01a0100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T10:23:29.789 INFO:teuthology.orchestra.run.smithi204.stderr:2024-04-23T10:23:29.788+0000 7f5cd545f700 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x7f5cd010e550 con 0x7f5cd0104680 2024-04-23T10:23:29.789 INFO:teuthology.orchestra.run.smithi204.stderr:2024-04-23T10:23:29.788+0000 7f5cd545f700 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 -- 0x7f5cd010e3d0 con 0x7f5cd0103ce0 2024-04-23T10:23:29.789 INFO:teuthology.orchestra.run.smithi204.stderr:2024-04-23T10:23:29.788+0000 7f5cd545f700 1 -- --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_getmap magic: 0 -- 0x7f5cd010e6d0 con 0x7f5cd0102ae0 2024-04-23T10:23:29.789 INFO:teuthology.orchestra.run.smithi204.stderr:2024-04-23T10:23:29.788+0000 7f5cceffd700 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f5cd0102ae0 0x7f5cd0198b40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T10:23:29.789 INFO:teuthology.orchestra.run.smithi204.stderr:2024-04-23T10:23:29.788+0000 7f5cceffd700 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f5cd0102ae0 0x7f5cd0198b40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.204:3300/0 says I am v2:172.21.15.204:32836/0 (socket says 172.21.15.204:32836) 2024-04-23T10:23:29.789 INFO:teuthology.orchestra.run.smithi204.stderr:2024-04-23T10:23:29.788+0000 7f5cceffd700 1 -- 172.21.15.204:0/1626262897 learned_addr learned my addr 172.21.15.204:0/1626262897 (peer_addr_for_me v2:172.21.15.204:0/0) 2024-04-23T10:23:29.789 INFO:teuthology.orchestra.run.smithi204.stderr:2024-04-23T10:23:29.788+0000 7f5ccf7fe700 1 --2- 172.21.15.204:0/1626262897 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f5cd0104680 0x7f5cd01a0100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T10:23:29.789 INFO:teuthology.orchestra.run.smithi204.stderr:2024-04-23T10:23:29.788+0000 7f5cce7fc700 1 --2- 172.21.15.204:0/1626262897 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f5cd0103ce0 0x7f5cd0199080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T10:23:29.789 INFO:teuthology.orchestra.run.smithi204.stderr:2024-04-23T10:23:29.788+0000 7f5cceffd700 1 -- 172.21.15.204:0/1626262897 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f5cd0103ce0 msgr2=0x7f5cd0199080 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T10:23:29.789 INFO:teuthology.orchestra.run.smithi204.stderr:2024-04-23T10:23:29.788+0000 7f5cceffd700 1 --2- 172.21.15.204:0/1626262897 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f5cd0103ce0 0x7f5cd0199080 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:23:29.789 INFO:teuthology.orchestra.run.smithi204.stderr:2024-04-23T10:23:29.788+0000 7f5cceffd700 1 -- 172.21.15.204:0/1626262897 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f5cd0104680 msgr2=0x7f5cd01a0100 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T10:23:29.789 INFO:teuthology.orchestra.run.smithi204.stderr:2024-04-23T10:23:29.788+0000 7f5cceffd700 1 --2- 172.21.15.204:0/1626262897 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f5cd0104680 0x7f5cd01a0100 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:23:29.789 INFO:teuthology.orchestra.run.smithi204.stderr:2024-04-23T10:23:29.788+0000 7f5cceffd700 1 -- 172.21.15.204:0/1626262897 --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f5cd01a07a0 con 0x7f5cd0102ae0 2024-04-23T10:23:29.790 INFO:teuthology.orchestra.run.smithi204.stderr:2024-04-23T10:23:29.789+0000 7f5cceffd700 1 --2- 172.21.15.204:0/1626262897 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f5cd0102ae0 0x7f5cd0198b40 secure :-1 s=READY pgs=38 cs=0 l=1 rev1=1 crypto rx=0x7f5cc000add0 tx=0x7f5cc000c330 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T10:23:29.790 INFO:teuthology.orchestra.run.smithi204.stderr:2024-04-23T10:23:29.789+0000 7f5cc7fff700 1 -- 172.21.15.204:0/1626262897 <== mon.1 v2:172.21.15.204:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5cc0017020 con 0x7f5cd0102ae0 2024-04-23T10:23:29.790 INFO:teuthology.orchestra.run.smithi204.stderr:2024-04-23T10:23:29.789+0000 7f5cd545f700 1 -- 172.21.15.204:0/1626262897 --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f5cd01a0a90 con 0x7f5cd0102ae0 2024-04-23T10:23:29.790 INFO:teuthology.orchestra.run.smithi204.stderr:2024-04-23T10:23:29.789+0000 7f5cc7fff700 1 -- 172.21.15.204:0/1626262897 <== mon.1 v2:172.21.15.204:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f5cc000c940 con 0x7f5cd0102ae0 2024-04-23T10:23:29.790 INFO:teuthology.orchestra.run.smithi204.stderr:2024-04-23T10:23:29.789+0000 7f5cc7fff700 1 -- 172.21.15.204:0/1626262897 <== mon.1 v2:172.21.15.204:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5cc0004280 con 0x7f5cd0102ae0 2024-04-23T10:23:29.790 INFO:teuthology.orchestra.run.smithi204.stderr:2024-04-23T10:23:29.789+0000 7f5cd545f700 1 -- 172.21.15.204:0/1626262897 --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f5cd01a0fd0 con 0x7f5cd0102ae0 2024-04-23T10:23:29.791 INFO:teuthology.orchestra.run.smithi204.stderr:2024-04-23T10:23:29.790+0000 7f5cd545f700 1 -- 172.21.15.204:0/1626262897 --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f5cd0069730 con 0x7f5cd0102ae0 2024-04-23T10:23:29.791 INFO:teuthology.orchestra.run.smithi204.stderr:2024-04-23T10:23:29.792+0000 7f5cc7fff700 1 -- 172.21.15.204:0/1626262897 <== mon.1 v2:172.21.15.204:3300/0 4 ==== mgrmap(e 16) ==== 138647+0+0 (secure 0 0 0) 0x7f5cc000caf0 con 0x7f5cd0102ae0 2024-04-23T10:23:29.791 INFO:teuthology.orchestra.run.smithi204.stderr:2024-04-23T10:23:29.792+0000 7f5cc7fff700 1 --2- 172.21.15.204:0/1626262897 >> [v2:172.21.15.114:6800/465333808,v1:172.21.15.114:6801/465333808] conn(0x7f5cbc0a3c40 0x7f5cbc0a6100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T10:23:29.791 INFO:teuthology.orchestra.run.smithi204.stderr:2024-04-23T10:23:29.792+0000 7f5cc7fff700 1 -- 172.21.15.204:0/1626262897 <== mon.1 v2:172.21.15.204:3300/0 5 ==== osd_map(76..76 src has 1..76) ==== 6723+0+0 (secure 0 0 0) 0x7f5cc00cf7b0 con 0x7f5cd0102ae0 2024-04-23T10:23:29.791 INFO:teuthology.orchestra.run.smithi204.stderr:2024-04-23T10:23:29.792+0000 7f5cce7fc700 1 --2- 172.21.15.204:0/1626262897 >> [v2:172.21.15.114:6800/465333808,v1:172.21.15.114:6801/465333808] conn(0x7f5cbc0a3c40 0x7f5cbc0a6100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T10:23:29.791 INFO:teuthology.orchestra.run.smithi204.stderr:2024-04-23T10:23:29.793+0000 7f5cce7fc700 1 --2- 172.21.15.204:0/1626262897 >> [v2:172.21.15.114:6800/465333808,v1:172.21.15.114:6801/465333808] conn(0x7f5cbc0a3c40 0x7f5cbc0a6100 secure :-1 s=READY pgs=118 cs=0 l=1 rev1=1 crypto rx=0x7f5cb800b5c0 tx=0x7f5cb8005fd0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T10:23:29.794 INFO:teuthology.orchestra.run.smithi204.stderr:2024-04-23T10:23:29.796+0000 7f5cc7fff700 1 -- 172.21.15.204:0/1626262897 <== mon.1 v2:172.21.15.204:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f5cc0021020 con 0x7f5cd0102ae0 2024-04-23T10:23:30.001 INFO:teuthology.orchestra.run.smithi204.stderr:2024-04-23T10:23:30.002+0000 7f5cd545f700 1 -- 172.21.15.204:0/1626262897 --> [v2:172.21.15.114:6800/465333808,v1:172.21.15.114:6801/465333808] -- mgr_command(tid 0: {"prefix": "orch daemon add osd", "svc_arg": "smithi204:vg_nvme/lv_1", "target": ["mon-mgr", ""]}) -- 0x7f5cd0062e80 con 0x7f5cbc0a3c40 2024-04-23T10:23:30.328 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 23 10:23:30 smithi204 ceph-mon[32649]: Detected new or changed devices on smithi119 2024-04-23T10:23:30.328 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 23 10:23:30 smithi204 ceph-mon[32649]: Adjusting osd_memory_target on smithi119 to 4272M 2024-04-23T10:23:30.328 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 23 10:23:30 smithi204 ceph-mon[32649]: osdmap e76: 11 total, 11 up, 11 in 2024-04-23T10:23:30.328 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 23 10:23:30 smithi204 ceph-mon[32649]: from='mgr.14152 172.21.15.114:0/1529706138' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-04-23T10:23:30.328 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 23 10:23:30 smithi204 ceph-mon[32649]: from='mgr.14152 172.21.15.114:0/1529706138' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-04-23T10:23:30.328 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 23 10:23:30 smithi204 ceph-mon[32649]: from='mgr.14152 172.21.15.114:0/1529706138' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-23T10:23:30.492 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 23 10:23:30 smithi119 ceph-mon[32608]: Detected new or changed devices on smithi119 2024-04-23T10:23:30.492 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 23 10:23:30 smithi119 ceph-mon[32608]: Adjusting osd_memory_target on smithi119 to 4272M 2024-04-23T10:23:30.492 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 23 10:23:30 smithi119 ceph-mon[32608]: osdmap e76: 11 total, 11 up, 11 in 2024-04-23T10:23:30.492 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 23 10:23:30 smithi119 ceph-mon[32608]: from='mgr.14152 172.21.15.114:0/1529706138' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-04-23T10:23:30.492 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 23 10:23:30 smithi119 ceph-mon[32608]: from='mgr.14152 172.21.15.114:0/1529706138' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-04-23T10:23:30.492 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 23 10:23:30 smithi119 ceph-mon[32608]: from='mgr.14152 172.21.15.114:0/1529706138' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-23T10:23:30.504 INFO:journalctl@ceph.mon.a.smithi114.stdout:Apr 23 10:23:30 smithi114 ceph-mon[29255]: Detected new or changed devices on smithi119 2024-04-23T10:23:30.504 INFO:journalctl@ceph.mon.a.smithi114.stdout:Apr 23 10:23:30 smithi114 ceph-mon[29255]: Adjusting osd_memory_target on smithi119 to 4272M 2024-04-23T10:23:30.504 INFO:journalctl@ceph.mon.a.smithi114.stdout:Apr 23 10:23:30 smithi114 ceph-mon[29255]: osdmap e76: 11 total, 11 up, 11 in 2024-04-23T10:23:30.504 INFO:journalctl@ceph.mon.a.smithi114.stdout:Apr 23 10:23:30 smithi114 ceph-mon[29255]: from='mgr.14152 172.21.15.114:0/1529706138' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-04-23T10:23:30.504 INFO:journalctl@ceph.mon.a.smithi114.stdout:Apr 23 10:23:30 smithi114 ceph-mon[29255]: from='mgr.14152 172.21.15.114:0/1529706138' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-04-23T10:23:30.504 INFO:journalctl@ceph.mon.a.smithi114.stdout:Apr 23 10:23:30 smithi114 ceph-mon[29255]: from='mgr.14152 172.21.15.114:0/1529706138' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-23T10:23:31.328 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 23 10:23:31 smithi204 ceph-mon[32649]: pgmap v238: 1 pgs: 1 remapped+peering; 577 KiB data, 328 MiB used, 983 GiB / 983 GiB avail 2024-04-23T10:23:31.328 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 23 10:23:31 smithi204 ceph-mon[32649]: from='client.24319 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi204:vg_nvme/lv_1", "target": ["mon-mgr", ""]}]: dispatch 2024-04-23T10:23:31.328 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 23 10:23:31 smithi204 ceph-mon[32649]: osdmap e77: 11 total, 11 up, 11 in 2024-04-23T10:23:31.492 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 23 10:23:31 smithi119 ceph-mon[32608]: pgmap v238: 1 pgs: 1 remapped+peering; 577 KiB data, 328 MiB used, 983 GiB / 983 GiB avail 2024-04-23T10:23:31.492 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 23 10:23:31 smithi119 ceph-mon[32608]: from='client.24319 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi204:vg_nvme/lv_1", "target": ["mon-mgr", ""]}]: dispatch 2024-04-23T10:23:31.492 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 23 10:23:31 smithi119 ceph-mon[32608]: osdmap e77: 11 total, 11 up, 11 in 2024-04-23T10:23:31.504 INFO:journalctl@ceph.mon.a.smithi114.stdout:Apr 23 10:23:31 smithi114 ceph-mon[29255]: pgmap v238: 1 pgs: 1 remapped+peering; 577 KiB data, 328 MiB used, 983 GiB / 983 GiB avail 2024-04-23T10:23:31.504 INFO:journalctl@ceph.mon.a.smithi114.stdout:Apr 23 10:23:31 smithi114 ceph-mon[29255]: from='client.24319 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi204:vg_nvme/lv_1", "target": ["mon-mgr", ""]}]: dispatch 2024-04-23T10:23:31.504 INFO:journalctl@ceph.mon.a.smithi114.stdout:Apr 23 10:23:31 smithi114 ceph-mon[29255]: osdmap e77: 11 total, 11 up, 11 in 2024-04-23T10:23:32.492 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 23 10:23:32 smithi119 ceph-mon[32608]: osdmap e78: 11 total, 11 up, 11 in 2024-04-23T10:23:32.492 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 23 10:23:32 smithi119 ceph-mon[32608]: pgmap v241: 1 pgs: 1 remapped+peering; 577 KiB data, 329 MiB used, 983 GiB / 983 GiB avail 2024-04-23T10:23:32.504 INFO:journalctl@ceph.mon.a.smithi114.stdout:Apr 23 10:23:32 smithi114 ceph-mon[29255]: osdmap e78: 11 total, 11 up, 11 in 2024-04-23T10:23:32.504 INFO:journalctl@ceph.mon.a.smithi114.stdout:Apr 23 10:23:32 smithi114 ceph-mon[29255]: pgmap v241: 1 pgs: 1 remapped+peering; 577 KiB data, 329 MiB used, 983 GiB / 983 GiB avail 2024-04-23T10:23:32.578 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 23 10:23:32 smithi204 ceph-mon[32649]: osdmap e78: 11 total, 11 up, 11 in 2024-04-23T10:23:32.578 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 23 10:23:32 smithi204 ceph-mon[32649]: pgmap v241: 1 pgs: 1 remapped+peering; 577 KiB data, 329 MiB used, 983 GiB / 983 GiB avail 2024-04-23T10:23:33.492 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 23 10:23:33 smithi119 ceph-mon[32608]: from='client.? 172.21.15.204:0/3281511985' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "53f037a5-3ec0-483a-80a1-dfee923a68e5"}]: dispatch 2024-04-23T10:23:33.492 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 23 10:23:33 smithi119 ceph-mon[32608]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "53f037a5-3ec0-483a-80a1-dfee923a68e5"}]: dispatch 2024-04-23T10:23:33.492 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 23 10:23:33 smithi119 ceph-mon[32608]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "53f037a5-3ec0-483a-80a1-dfee923a68e5"}]': finished 2024-04-23T10:23:33.492 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 23 10:23:33 smithi119 ceph-mon[32608]: osdmap e79: 12 total, 11 up, 12 in 2024-04-23T10:23:33.492 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 23 10:23:33 smithi119 ceph-mon[32608]: from='mgr.14152 172.21.15.114:0/1529706138' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 11}]: dispatch 2024-04-23T10:23:33.504 INFO:journalctl@ceph.mon.a.smithi114.stdout:Apr 23 10:23:33 smithi114 ceph-mon[29255]: from='client.? 172.21.15.204:0/3281511985' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "53f037a5-3ec0-483a-80a1-dfee923a68e5"}]: dispatch 2024-04-23T10:23:33.504 INFO:journalctl@ceph.mon.a.smithi114.stdout:Apr 23 10:23:33 smithi114 ceph-mon[29255]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "53f037a5-3ec0-483a-80a1-dfee923a68e5"}]: dispatch 2024-04-23T10:23:33.504 INFO:journalctl@ceph.mon.a.smithi114.stdout:Apr 23 10:23:33 smithi114 ceph-mon[29255]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "53f037a5-3ec0-483a-80a1-dfee923a68e5"}]': finished 2024-04-23T10:23:33.504 INFO:journalctl@ceph.mon.a.smithi114.stdout:Apr 23 10:23:33 smithi114 ceph-mon[29255]: osdmap e79: 12 total, 11 up, 12 in 2024-04-23T10:23:33.504 INFO:journalctl@ceph.mon.a.smithi114.stdout:Apr 23 10:23:33 smithi114 ceph-mon[29255]: from='mgr.14152 172.21.15.114:0/1529706138' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 11}]: dispatch 2024-04-23T10:23:33.578 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 23 10:23:33 smithi204 ceph-mon[32649]: from='client.? 172.21.15.204:0/3281511985' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "53f037a5-3ec0-483a-80a1-dfee923a68e5"}]: dispatch 2024-04-23T10:23:33.578 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 23 10:23:33 smithi204 ceph-mon[32649]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "53f037a5-3ec0-483a-80a1-dfee923a68e5"}]: dispatch 2024-04-23T10:23:33.578 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 23 10:23:33 smithi204 ceph-mon[32649]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "53f037a5-3ec0-483a-80a1-dfee923a68e5"}]': finished 2024-04-23T10:23:33.578 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 23 10:23:33 smithi204 ceph-mon[32649]: osdmap e79: 12 total, 11 up, 12 in 2024-04-23T10:23:33.578 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 23 10:23:33 smithi204 ceph-mon[32649]: from='mgr.14152 172.21.15.114:0/1529706138' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 11}]: dispatch 2024-04-23T10:23:34.492 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 23 10:23:34 smithi119 ceph-mon[32608]: from='client.? 172.21.15.204:0/1155592237' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-04-23T10:23:34.492 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 23 10:23:34 smithi119 ceph-mon[32608]: pgmap v243: 1 pgs: 1 active+clean; 577 KiB data, 329 MiB used, 983 GiB / 983 GiB avail; 118 KiB/s, 0 objects/s recovering 2024-04-23T10:23:34.504 INFO:journalctl@ceph.mon.a.smithi114.stdout:Apr 23 10:23:34 smithi114 ceph-mon[29255]: from='client.? 172.21.15.204:0/1155592237' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-04-23T10:23:34.504 INFO:journalctl@ceph.mon.a.smithi114.stdout:Apr 23 10:23:34 smithi114 ceph-mon[29255]: pgmap v243: 1 pgs: 1 active+clean; 577 KiB data, 329 MiB used, 983 GiB / 983 GiB avail; 118 KiB/s, 0 objects/s recovering 2024-04-23T10:23:34.578 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 23 10:23:34 smithi204 ceph-mon[32649]: from='client.? 172.21.15.204:0/1155592237' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-04-23T10:23:34.578 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 23 10:23:34 smithi204 ceph-mon[32649]: pgmap v243: 1 pgs: 1 active+clean; 577 KiB data, 329 MiB used, 983 GiB / 983 GiB avail; 118 KiB/s, 0 objects/s recovering 2024-04-23T10:23:37.162 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 23 10:23:36 smithi204 ceph-mon[32649]: pgmap v244: 1 pgs: 1 active+clean; 577 KiB data, 329 MiB used, 983 GiB / 983 GiB avail; 96 KiB/s, 0 objects/s recovering 2024-04-23T10:23:37.254 INFO:journalctl@ceph.mon.a.smithi114.stdout:Apr 23 10:23:36 smithi114 ceph-mon[29255]: pgmap v244: 1 pgs: 1 active+clean; 577 KiB data, 329 MiB used, 983 GiB / 983 GiB avail; 96 KiB/s, 0 objects/s recovering 2024-04-23T10:23:37.492 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 23 10:23:36 smithi119 ceph-mon[32608]: pgmap v244: 1 pgs: 1 active+clean; 577 KiB data, 329 MiB used, 983 GiB / 983 GiB avail; 96 KiB/s, 0 objects/s recovering 2024-04-23T10:23:39.247 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 23 10:23:38 smithi204 ceph-mon[32649]: pgmap v245: 1 pgs: 1 active+clean; 577 KiB data, 333 MiB used, 983 GiB / 983 GiB avail; 73 KiB/s, 0 objects/s recovering 2024-04-23T10:23:39.254 INFO:journalctl@ceph.mon.a.smithi114.stdout:Apr 23 10:23:38 smithi114 ceph-mon[29255]: pgmap v245: 1 pgs: 1 active+clean; 577 KiB data, 333 MiB used, 983 GiB / 983 GiB avail; 73 KiB/s, 0 objects/s recovering 2024-04-23T10:23:39.492 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 23 10:23:38 smithi119 ceph-mon[32608]: pgmap v245: 1 pgs: 1 active+clean; 577 KiB data, 333 MiB used, 983 GiB / 983 GiB avail; 73 KiB/s, 0 objects/s recovering 2024-04-23T10:23:41.254 INFO:journalctl@ceph.mon.a.smithi114.stdout:Apr 23 10:23:41 smithi114 ceph-mon[29255]: pgmap v246: 1 pgs: 1 active+clean; 577 KiB data, 333 MiB used, 983 GiB / 983 GiB avail; 65 KiB/s, 0 objects/s recovering 2024-04-23T10:23:41.254 INFO:journalctl@ceph.mon.a.smithi114.stdout:Apr 23 10:23:41 smithi114 ceph-mon[29255]: from='mgr.14152 172.21.15.114:0/1529706138' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.11"}]: dispatch 2024-04-23T10:23:41.254 INFO:journalctl@ceph.mon.a.smithi114.stdout:Apr 23 10:23:41 smithi114 ceph-mon[29255]: from='mgr.14152 172.21.15.114:0/1529706138' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-23T10:23:41.264 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 23 10:23:41 smithi204 ceph-mon[32649]: pgmap v246: 1 pgs: 1 active+clean; 577 KiB data, 333 MiB used, 983 GiB / 983 GiB avail; 65 KiB/s, 0 objects/s recovering 2024-04-23T10:23:41.264 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 23 10:23:41 smithi204 ceph-mon[32649]: from='mgr.14152 172.21.15.114:0/1529706138' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.11"}]: dispatch 2024-04-23T10:23:41.264 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 23 10:23:41 smithi204 ceph-mon[32649]: from='mgr.14152 172.21.15.114:0/1529706138' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-23T10:23:41.492 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 23 10:23:41 smithi119 ceph-mon[32608]: pgmap v246: 1 pgs: 1 active+clean; 577 KiB data, 333 MiB used, 983 GiB / 983 GiB avail; 65 KiB/s, 0 objects/s recovering 2024-04-23T10:23:41.492 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 23 10:23:41 smithi119 ceph-mon[32608]: from='mgr.14152 172.21.15.114:0/1529706138' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.11"}]: dispatch 2024-04-23T10:23:41.492 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 23 10:23:41 smithi119 ceph-mon[32608]: from='mgr.14152 172.21.15.114:0/1529706138' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-23T10:23:42.328 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 23 10:23:42 smithi204 ceph-mon[32649]: Deploying daemon osd.11 on smithi204 2024-04-23T10:23:42.492 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 23 10:23:42 smithi119 ceph-mon[32608]: Deploying daemon osd.11 on smithi204 2024-04-23T10:23:42.504 INFO:journalctl@ceph.mon.a.smithi114.stdout:Apr 23 10:23:42 smithi114 ceph-mon[29255]: Deploying daemon osd.11 on smithi204 2024-04-23T10:23:43.328 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 23 10:23:43 smithi204 ceph-mon[32649]: pgmap v247: 1 pgs: 1 active+clean; 577 KiB data, 333 MiB used, 983 GiB / 983 GiB avail; 58 KiB/s, 0 objects/s recovering 2024-04-23T10:23:43.491 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 23 10:23:43 smithi119 ceph-mon[32608]: pgmap v247: 1 pgs: 1 active+clean; 577 KiB data, 333 MiB used, 983 GiB / 983 GiB avail; 58 KiB/s, 0 objects/s recovering 2024-04-23T10:23:43.504 INFO:journalctl@ceph.mon.a.smithi114.stdout:Apr 23 10:23:43 smithi114 ceph-mon[29255]: pgmap v247: 1 pgs: 1 active+clean; 577 KiB data, 333 MiB used, 983 GiB / 983 GiB avail; 58 KiB/s, 0 objects/s recovering 2024-04-23T10:23:45.306 INFO:journalctl@ceph.mon.a.smithi114.stdout:Apr 23 10:23:45 smithi114 ceph-mon[29255]: pgmap v248: 1 pgs: 1 active+clean; 577 KiB data, 333 MiB used, 983 GiB / 983 GiB avail 2024-04-23T10:23:45.328 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 23 10:23:45 smithi204 ceph-mon[32649]: pgmap v248: 1 pgs: 1 active+clean; 577 KiB data, 333 MiB used, 983 GiB / 983 GiB avail 2024-04-23T10:23:45.492 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 23 10:23:45 smithi119 ceph-mon[32608]: pgmap v248: 1 pgs: 1 active+clean; 577 KiB data, 333 MiB used, 983 GiB / 983 GiB avail 2024-04-23T10:23:47.279 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 23 10:23:47 smithi204 ceph-mon[32649]: pgmap v249: 1 pgs: 1 active+clean; 577 KiB data, 333 MiB used, 983 GiB / 983 GiB avail 2024-04-23T10:23:47.279 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 23 10:23:47 smithi204 ceph-mon[32649]: from='mgr.14152 172.21.15.114:0/1529706138' entity='mgr.x' 2024-04-23T10:23:47.279 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 23 10:23:47 smithi204 ceph-mon[32649]: from='mgr.14152 172.21.15.114:0/1529706138' entity='mgr.x' 2024-04-23T10:23:47.280 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 23 10:23:47 smithi204 ceph-mon[32649]: from='mgr.14152 172.21.15.114:0/1529706138' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-23T10:23:47.492 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 23 10:23:47 smithi119 ceph-mon[32608]: pgmap v249: 1 pgs: 1 active+clean; 577 KiB data, 333 MiB used, 983 GiB / 983 GiB avail 2024-04-23T10:23:47.492 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 23 10:23:47 smithi119 ceph-mon[32608]: from='mgr.14152 172.21.15.114:0/1529706138' entity='mgr.x' 2024-04-23T10:23:47.492 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 23 10:23:47 smithi119 ceph-mon[32608]: from='mgr.14152 172.21.15.114:0/1529706138' entity='mgr.x' 2024-04-23T10:23:47.492 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 23 10:23:47 smithi119 ceph-mon[32608]: from='mgr.14152 172.21.15.114:0/1529706138' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-23T10:23:47.504 INFO:journalctl@ceph.mon.a.smithi114.stdout:Apr 23 10:23:47 smithi114 ceph-mon[29255]: pgmap v249: 1 pgs: 1 active+clean; 577 KiB data, 333 MiB used, 983 GiB / 983 GiB avail 2024-04-23T10:23:47.504 INFO:journalctl@ceph.mon.a.smithi114.stdout:Apr 23 10:23:47 smithi114 ceph-mon[29255]: from='mgr.14152 172.21.15.114:0/1529706138' entity='mgr.x' 2024-04-23T10:23:47.504 INFO:journalctl@ceph.mon.a.smithi114.stdout:Apr 23 10:23:47 smithi114 ceph-mon[29255]: from='mgr.14152 172.21.15.114:0/1529706138' entity='mgr.x' 2024-04-23T10:23:47.504 INFO:journalctl@ceph.mon.a.smithi114.stdout:Apr 23 10:23:47 smithi114 ceph-mon[29255]: from='mgr.14152 172.21.15.114:0/1529706138' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-23T10:23:49.078 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 23 10:23:49 smithi204 ceph-mon[32649]: pgmap v250: 1 pgs: 1 active+clean; 577 KiB data, 333 MiB used, 983 GiB / 983 GiB avail 2024-04-23T10:23:49.492 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 23 10:23:49 smithi119 ceph-mon[32608]: pgmap v250: 1 pgs: 1 active+clean; 577 KiB data, 333 MiB used, 983 GiB / 983 GiB avail 2024-04-23T10:23:49.504 INFO:journalctl@ceph.mon.a.smithi114.stdout:Apr 23 10:23:49 smithi114 ceph-mon[29255]: pgmap v250: 1 pgs: 1 active+clean; 577 KiB data, 333 MiB used, 983 GiB / 983 GiB avail 2024-04-23T10:23:50.176 INFO:teuthology.orchestra.run.smithi204.stdout:Created osd(s) 11 on host 'smithi204' 2024-04-23T10:23:50.176 INFO:teuthology.orchestra.run.smithi204.stderr:2024-04-23T10:23:50.176+0000 7f5cc7fff700 1 -- 172.21.15.204:0/1626262897 <== mgr.14152 v2:172.21.15.114:6800/465333808 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+38 (secure 0 0 0) 0x7f5cd0062e80 con 0x7f5cbc0a3c40 2024-04-23T10:23:50.179 INFO:teuthology.orchestra.run.smithi204.stderr:2024-04-23T10:23:50.179+0000 7f5cd545f700 1 -- 172.21.15.204:0/1626262897 >> [v2:172.21.15.114:6800/465333808,v1:172.21.15.114:6801/465333808] conn(0x7f5cbc0a3c40 msgr2=0x7f5cbc0a6100 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T10:23:50.179 INFO:teuthology.orchestra.run.smithi204.stderr:2024-04-23T10:23:50.179+0000 7f5cd545f700 1 --2- 172.21.15.204:0/1626262897 >> [v2:172.21.15.114:6800/465333808,v1:172.21.15.114:6801/465333808] conn(0x7f5cbc0a3c40 0x7f5cbc0a6100 secure :-1 s=READY pgs=118 cs=0 l=1 rev1=1 crypto rx=0x7f5cb800b5c0 tx=0x7f5cb8005fd0 comp rx=0 tx=0).stop 2024-04-23T10:23:50.179 INFO:teuthology.orchestra.run.smithi204.stderr:2024-04-23T10:23:50.179+0000 7f5cd545f700 1 -- 172.21.15.204:0/1626262897 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f5cd0102ae0 msgr2=0x7f5cd0198b40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T10:23:50.179 INFO:teuthology.orchestra.run.smithi204.stderr:2024-04-23T10:23:50.179+0000 7f5cd545f700 1 --2- 172.21.15.204:0/1626262897 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f5cd0102ae0 0x7f5cd0198b40 secure :-1 s=READY pgs=38 cs=0 l=1 rev1=1 crypto rx=0x7f5cc000add0 tx=0x7f5cc000c330 comp rx=0 tx=0).stop 2024-04-23T10:23:50.179 INFO:teuthology.orchestra.run.smithi204.stderr:2024-04-23T10:23:50.179+0000 7f5cd545f700 1 -- 172.21.15.204:0/1626262897 shutdown_connections 2024-04-23T10:23:50.179 INFO:teuthology.orchestra.run.smithi204.stderr:2024-04-23T10:23:50.179+0000 7f5cd545f700 1 --2- 172.21.15.204:0/1626262897 >> [v2:172.21.15.114:6800/465333808,v1:172.21.15.114:6801/465333808] conn(0x7f5cbc0a3c40 0x7f5cbc0a6100 unknown :-1 s=CLOSED pgs=118 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:23:50.179 INFO:teuthology.orchestra.run.smithi204.stderr:2024-04-23T10:23:50.179+0000 7f5cd545f700 1 --2- 172.21.15.204:0/1626262897 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f5cd0104680 0x7f5cd01a0100 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:23:50.179 INFO:teuthology.orchestra.run.smithi204.stderr:2024-04-23T10:23:50.179+0000 7f5cd545f700 1 --2- 172.21.15.204:0/1626262897 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f5cd0102ae0 0x7f5cd0198b40 unknown :-1 s=CLOSED pgs=38 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:23:50.179 INFO:teuthology.orchestra.run.smithi204.stderr:2024-04-23T10:23:50.179+0000 7f5cd545f700 1 --2- 172.21.15.204:0/1626262897 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f5cd0103ce0 0x7f5cd0199080 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:23:50.179 INFO:teuthology.orchestra.run.smithi204.stderr:2024-04-23T10:23:50.179+0000 7f5cd545f700 1 -- 172.21.15.204:0/1626262897 >> 172.21.15.204:0/1626262897 conn(0x7f5cd00fe080 msgr2=0x7f5cd0106fe0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T10:23:50.179 INFO:teuthology.orchestra.run.smithi204.stderr:2024-04-23T10:23:50.179+0000 7f5cd545f700 1 -- 172.21.15.204:0/1626262897 shutdown_connections 2024-04-23T10:23:50.179 INFO:teuthology.orchestra.run.smithi204.stderr:2024-04-23T10:23:50.179+0000 7f5cd545f700 1 -- 172.21.15.204:0/1626262897 wait complete. 2024-04-23T10:23:50.481 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 23 10:23:50 smithi204 ceph-mon[32649]: from='mgr.14152 172.21.15.114:0/1529706138' entity='mgr.x' 2024-04-23T10:23:50.481 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 23 10:23:50 smithi204 ceph-mon[32649]: from='mgr.14152 172.21.15.114:0/1529706138' entity='mgr.x' 2024-04-23T10:23:50.481 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 23 10:23:50 smithi204 ceph-mon[32649]: from='mgr.14152 172.21.15.114:0/1529706138' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-23T10:23:50.482 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 23 10:23:50 smithi204 ceph-mon[32649]: from='mgr.14152 172.21.15.114:0/1529706138' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-23T10:23:50.482 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 23 10:23:50 smithi204 ceph-mon[32649]: from='mgr.14152 172.21.15.114:0/1529706138' entity='mgr.x' 2024-04-23T10:23:50.482 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 23 10:23:50 smithi204 ceph-mon[32649]: pgmap v251: 1 pgs: 1 active+clean; 577 KiB data, 333 MiB used, 983 GiB / 983 GiB avail 2024-04-23T10:23:50.482 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 23 10:23:50 smithi204 ceph-mon[32649]: from='mgr.14152 172.21.15.114:0/1529706138' entity='mgr.x' 2024-04-23T10:23:50.482 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 23 10:23:50 smithi204 ceph-mon[32649]: from='mgr.14152 172.21.15.114:0/1529706138' entity='mgr.x' 2024-04-23T10:23:50.482 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 23 10:23:50 smithi204 ceph-mon[32649]: from='osd.11 [v2:172.21.15.204:6824/2407298490,v1:172.21.15.204:6825/2407298490]' entity='osd.11' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["11"]}]: dispatch 2024-04-23T10:23:50.482 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 23 10:23:50 smithi204 ceph-mon[32649]: from='osd.11 ' entity='osd.11' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["11"]}]: dispatch 2024-04-23T10:23:50.482 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 23 10:23:50 smithi204 ceph-mon[32649]: from='mgr.14152 172.21.15.114:0/1529706138' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-23T10:23:50.742 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 23 10:23:50 smithi119 ceph-mon[32608]: from='mgr.14152 172.21.15.114:0/1529706138' entity='mgr.x' 2024-04-23T10:23:50.742 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 23 10:23:50 smithi119 ceph-mon[32608]: from='mgr.14152 172.21.15.114:0/1529706138' entity='mgr.x' 2024-04-23T10:23:50.742 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 23 10:23:50 smithi119 ceph-mon[32608]: from='mgr.14152 172.21.15.114:0/1529706138' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-23T10:23:50.742 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 23 10:23:50 smithi119 ceph-mon[32608]: from='mgr.14152 172.21.15.114:0/1529706138' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-23T10:23:50.742 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 23 10:23:50 smithi119 ceph-mon[32608]: from='mgr.14152 172.21.15.114:0/1529706138' entity='mgr.x' 2024-04-23T10:23:50.742 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 23 10:23:50 smithi119 ceph-mon[32608]: pgmap v251: 1 pgs: 1 active+clean; 577 KiB data, 333 MiB used, 983 GiB / 983 GiB avail 2024-04-23T10:23:50.742 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 23 10:23:50 smithi119 ceph-mon[32608]: from='mgr.14152 172.21.15.114:0/1529706138' entity='mgr.x' 2024-04-23T10:23:50.742 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 23 10:23:50 smithi119 ceph-mon[32608]: from='mgr.14152 172.21.15.114:0/1529706138' entity='mgr.x' 2024-04-23T10:23:50.743 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 23 10:23:50 smithi119 ceph-mon[32608]: from='osd.11 [v2:172.21.15.204:6824/2407298490,v1:172.21.15.204:6825/2407298490]' entity='osd.11' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["11"]}]: dispatch 2024-04-23T10:23:50.743 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 23 10:23:50 smithi119 ceph-mon[32608]: from='osd.11 ' entity='osd.11' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["11"]}]: dispatch 2024-04-23T10:23:50.743 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 23 10:23:50 smithi119 ceph-mon[32608]: from='mgr.14152 172.21.15.114:0/1529706138' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-23T10:23:50.754 INFO:journalctl@ceph.mon.a.smithi114.stdout:Apr 23 10:23:50 smithi114 ceph-mon[29255]: from='mgr.14152 172.21.15.114:0/1529706138' entity='mgr.x' 2024-04-23T10:23:50.754 INFO:journalctl@ceph.mon.a.smithi114.stdout:Apr 23 10:23:50 smithi114 ceph-mon[29255]: from='mgr.14152 172.21.15.114:0/1529706138' entity='mgr.x' 2024-04-23T10:23:50.754 INFO:journalctl@ceph.mon.a.smithi114.stdout:Apr 23 10:23:50 smithi114 ceph-mon[29255]: from='mgr.14152 172.21.15.114:0/1529706138' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-23T10:23:50.754 INFO:journalctl@ceph.mon.a.smithi114.stdout:Apr 23 10:23:50 smithi114 ceph-mon[29255]: from='mgr.14152 172.21.15.114:0/1529706138' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-23T10:23:50.754 INFO:journalctl@ceph.mon.a.smithi114.stdout:Apr 23 10:23:50 smithi114 ceph-mon[29255]: from='mgr.14152 172.21.15.114:0/1529706138' entity='mgr.x' 2024-04-23T10:23:50.754 INFO:journalctl@ceph.mon.a.smithi114.stdout:Apr 23 10:23:50 smithi114 ceph-mon[29255]: pgmap v251: 1 pgs: 1 active+clean; 577 KiB data, 333 MiB used, 983 GiB / 983 GiB avail 2024-04-23T10:23:50.755 INFO:journalctl@ceph.mon.a.smithi114.stdout:Apr 23 10:23:50 smithi114 ceph-mon[29255]: from='mgr.14152 172.21.15.114:0/1529706138' entity='mgr.x' 2024-04-23T10:23:50.755 INFO:journalctl@ceph.mon.a.smithi114.stdout:Apr 23 10:23:50 smithi114 ceph-mon[29255]: from='mgr.14152 172.21.15.114:0/1529706138' entity='mgr.x' 2024-04-23T10:23:50.755 INFO:journalctl@ceph.mon.a.smithi114.stdout:Apr 23 10:23:50 smithi114 ceph-mon[29255]: from='osd.11 [v2:172.21.15.204:6824/2407298490,v1:172.21.15.204:6825/2407298490]' entity='osd.11' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["11"]}]: dispatch 2024-04-23T10:23:50.755 INFO:journalctl@ceph.mon.a.smithi114.stdout:Apr 23 10:23:50 smithi114 ceph-mon[29255]: from='osd.11 ' entity='osd.11' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["11"]}]: dispatch 2024-04-23T10:23:50.755 INFO:journalctl@ceph.mon.a.smithi114.stdout:Apr 23 10:23:50 smithi114 ceph-mon[29255]: from='mgr.14152 172.21.15.114:0/1529706138' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-23T10:23:50.799 DEBUG:teuthology.orchestra.run.smithi204:osd.11> sudo journalctl -f -n 0 -u ceph-aa240572-0159-11ef-bc93-c7b262605968@osd.11.service 2024-04-23T10:23:50.801 INFO:tasks.cephadm:Waiting for 12 OSDs to come up... 2024-04-23T10:23:50.801 DEBUG:teuthology.orchestra.run.smithi114:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d69f13d827952bb07bb9b7973bca14f97d764d6f shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid aa240572-0159-11ef-bc93-c7b262605968 -- ceph osd stat -f json 2024-04-23T10:23:51.050 INFO:teuthology.orchestra.run.smithi114.stderr:Inferring config /var/lib/ceph/aa240572-0159-11ef-bc93-c7b262605968/mon.a/config 2024-04-23T10:23:52.279 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 23 10:23:52 smithi204 ceph-mon[32649]: from='osd.11 ' entity='osd.11' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["11"]}]': finished 2024-04-23T10:23:52.279 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 23 10:23:52 smithi204 ceph-mon[32649]: osdmap e80: 12 total, 11 up, 12 in 2024-04-23T10:23:52.279 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 23 10:23:52 smithi204 ceph-mon[32649]: from='mgr.14152 172.21.15.114:0/1529706138' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 11}]: dispatch 2024-04-23T10:23:52.279 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 23 10:23:52 smithi204 ceph-mon[32649]: from='osd.11 [v2:172.21.15.204:6824/2407298490,v1:172.21.15.204:6825/2407298490]' entity='osd.11' cmd=[{"prefix": "osd crush create-or-move", "id": 11, "weight":0.0873, "args": ["host=smithi204", "root=default"]}]: dispatch 2024-04-23T10:23:52.279 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 23 10:23:52 smithi204 ceph-mon[32649]: from='osd.11 ' entity='osd.11' cmd=[{"prefix": "osd crush create-or-move", "id": 11, "weight":0.0873, "args": ["host=smithi204", "root=default"]}]: dispatch 2024-04-23T10:23:52.279 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 23 10:23:52 smithi204 ceph-mon[32649]: pgmap v253: 1 pgs: 1 active+clean; 577 KiB data, 333 MiB used, 983 GiB / 983 GiB avail 2024-04-23T10:23:52.492 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 23 10:23:52 smithi119 ceph-mon[32608]: from='osd.11 ' entity='osd.11' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["11"]}]': finished 2024-04-23T10:23:52.492 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 23 10:23:52 smithi119 ceph-mon[32608]: osdmap e80: 12 total, 11 up, 12 in 2024-04-23T10:23:52.492 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 23 10:23:52 smithi119 ceph-mon[32608]: from='mgr.14152 172.21.15.114:0/1529706138' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 11}]: dispatch 2024-04-23T10:23:52.492 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 23 10:23:52 smithi119 ceph-mon[32608]: from='osd.11 [v2:172.21.15.204:6824/2407298490,v1:172.21.15.204:6825/2407298490]' entity='osd.11' cmd=[{"prefix": "osd crush create-or-move", "id": 11, "weight":0.0873, "args": ["host=smithi204", "root=default"]}]: dispatch 2024-04-23T10:23:52.492 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 23 10:23:52 smithi119 ceph-mon[32608]: from='osd.11 ' entity='osd.11' cmd=[{"prefix": "osd crush create-or-move", "id": 11, "weight":0.0873, "args": ["host=smithi204", "root=default"]}]: dispatch 2024-04-23T10:23:52.492 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 23 10:23:52 smithi119 ceph-mon[32608]: pgmap v253: 1 pgs: 1 active+clean; 577 KiB data, 333 MiB used, 983 GiB / 983 GiB avail 2024-04-23T10:23:52.504 INFO:journalctl@ceph.mon.a.smithi114.stdout:Apr 23 10:23:52 smithi114 ceph-mon[29255]: from='osd.11 ' entity='osd.11' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["11"]}]': finished 2024-04-23T10:23:52.504 INFO:journalctl@ceph.mon.a.smithi114.stdout:Apr 23 10:23:52 smithi114 ceph-mon[29255]: osdmap e80: 12 total, 11 up, 12 in 2024-04-23T10:23:52.504 INFO:journalctl@ceph.mon.a.smithi114.stdout:Apr 23 10:23:52 smithi114 ceph-mon[29255]: from='mgr.14152 172.21.15.114:0/1529706138' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 11}]: dispatch 2024-04-23T10:23:52.504 INFO:journalctl@ceph.mon.a.smithi114.stdout:Apr 23 10:23:52 smithi114 ceph-mon[29255]: from='osd.11 [v2:172.21.15.204:6824/2407298490,v1:172.21.15.204:6825/2407298490]' entity='osd.11' cmd=[{"prefix": "osd crush create-or-move", "id": 11, "weight":0.0873, "args": ["host=smithi204", "root=default"]}]: dispatch 2024-04-23T10:23:52.504 INFO:journalctl@ceph.mon.a.smithi114.stdout:Apr 23 10:23:52 smithi114 ceph-mon[29255]: from='osd.11 ' entity='osd.11' cmd=[{"prefix": "osd crush create-or-move", "id": 11, "weight":0.0873, "args": ["host=smithi204", "root=default"]}]: dispatch 2024-04-23T10:23:52.505 INFO:journalctl@ceph.mon.a.smithi114.stdout:Apr 23 10:23:52 smithi114 ceph-mon[29255]: pgmap v253: 1 pgs: 1 active+clean; 577 KiB data, 333 MiB used, 983 GiB / 983 GiB avail 2024-04-23T10:23:52.840 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:23:52.839+0000 7f9ca7653700 1 -- 172.21.15.114:0/4267202178 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f9ca0102ba0 msgr2=0x7f9ca0103000 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T10:23:52.840 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:23:52.839+0000 7f9ca7653700 1 --2- 172.21.15.114:0/4267202178 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f9ca0102ba0 0x7f9ca0103000 secure :-1 s=READY pgs=149 cs=0 l=1 rev1=1 crypto rx=0x7f9c900099c0 tx=0x7f9c90009cd0 comp rx=0 tx=0).stop 2024-04-23T10:23:52.840 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:23:52.839+0000 7f9ca7653700 1 -- 172.21.15.114:0/4267202178 shutdown_connections 2024-04-23T10:23:52.840 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:23:52.839+0000 7f9ca7653700 1 --2- 172.21.15.114:0/4267202178 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f9ca0103540 0x7f9ca0109df0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:23:52.840 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:23:52.839+0000 7f9ca7653700 1 --2- 172.21.15.114:0/4267202178 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f9ca01019a0 0x7f9ca0101dc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:23:52.840 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:23:52.839+0000 7f9ca7653700 1 --2- 172.21.15.114:0/4267202178 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f9ca0102ba0 0x7f9ca0103000 unknown :-1 s=CLOSED pgs=149 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:23:52.840 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:23:52.839+0000 7f9ca7653700 1 -- 172.21.15.114:0/4267202178 >> 172.21.15.114:0/4267202178 conn(0x7f9ca00fcf40 msgr2=0x7f9ca00ff380 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T10:23:52.840 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:23:52.839+0000 7f9ca7653700 1 -- 172.21.15.114:0/4267202178 shutdown_connections 2024-04-23T10:23:52.840 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:23:52.839+0000 7f9ca7653700 1 -- 172.21.15.114:0/4267202178 wait complete. 2024-04-23T10:23:52.840 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:23:52.839+0000 7f9ca7653700 1 Processor -- start 2024-04-23T10:23:52.840 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:23:52.839+0000 7f9ca7653700 1 -- start start 2024-04-23T10:23:52.841 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:23:52.839+0000 7f9ca7653700 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f9ca01019a0 0x7f9ca006b2a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T10:23:52.841 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:23:52.839+0000 7f9ca7653700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f9ca0102ba0 0x7f9ca019d920 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T10:23:52.841 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:23:52.839+0000 7f9ca7653700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f9ca0103540 0x7f9ca019ff70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T10:23:52.841 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:23:52.839+0000 7f9ca7653700 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x7f9ca010d410 con 0x7f9ca0102ba0 2024-04-23T10:23:52.841 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:23:52.839+0000 7f9ca7653700 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 -- 0x7f9ca010d290 con 0x7f9ca0103540 2024-04-23T10:23:52.841 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:23:52.839+0000 7f9ca7653700 1 -- --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_getmap magic: 0 -- 0x7f9ca010d590 con 0x7f9ca01019a0 2024-04-23T10:23:52.841 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:23:52.840+0000 7f9ca4bee700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f9ca0102ba0 0x7f9ca019d920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T10:23:52.841 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:23:52.840+0000 7f9ca4bee700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f9ca0102ba0 0x7f9ca019d920 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.114:53460/0 (socket says 172.21.15.114:53460) 2024-04-23T10:23:52.841 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:23:52.840+0000 7f9ca4bee700 1 -- 172.21.15.114:0/526042645 learned_addr learned my addr 172.21.15.114:0/526042645 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-04-23T10:23:52.841 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:23:52.840+0000 7f9ca53ef700 1 --2- 172.21.15.114:0/526042645 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f9ca01019a0 0x7f9ca006b2a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T10:23:52.841 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:23:52.840+0000 7f9ca4bee700 1 -- 172.21.15.114:0/526042645 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f9ca01019a0 msgr2=0x7f9ca006b2a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T10:23:52.842 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:23:52.840+0000 7f9ca5bf0700 1 --2- 172.21.15.114:0/526042645 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f9ca0103540 0x7f9ca019ff70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T10:23:52.842 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:23:52.840+0000 7f9ca4bee700 1 --2- 172.21.15.114:0/526042645 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f9ca01019a0 0x7f9ca006b2a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:23:52.842 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:23:52.840+0000 7f9ca4bee700 1 -- 172.21.15.114:0/526042645 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f9ca0103540 msgr2=0x7f9ca019ff70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T10:23:52.842 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:23:52.840+0000 7f9ca4bee700 1 --2- 172.21.15.114:0/526042645 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f9ca0103540 0x7f9ca019ff70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:23:52.842 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:23:52.840+0000 7f9ca4bee700 1 -- 172.21.15.114:0/526042645 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f9ca01a0840 con 0x7f9ca0102ba0 2024-04-23T10:23:52.842 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:23:52.840+0000 7f9ca4bee700 1 --2- 172.21.15.114:0/526042645 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f9ca0102ba0 0x7f9ca019d920 secure :-1 s=READY pgs=150 cs=0 l=1 rev1=1 crypto rx=0x7f9c90000c00 tx=0x7f9c90009580 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T10:23:52.842 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:23:52.840+0000 7f9c967fc700 1 -- 172.21.15.114:0/526042645 <== mon.0 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9c90027070 con 0x7f9ca0102ba0 2024-04-23T10:23:52.842 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:23:52.840+0000 7f9c967fc700 1 -- 172.21.15.114:0/526042645 <== mon.0 v2:172.21.15.114:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f9c900040a0 con 0x7f9ca0102ba0 2024-04-23T10:23:52.842 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:23:52.840+0000 7f9ca7653700 1 -- 172.21.15.114:0/526042645 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f9ca01a0ad0 con 0x7f9ca0102ba0 2024-04-23T10:23:52.842 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:23:52.840+0000 7f9c967fc700 1 -- 172.21.15.114:0/526042645 <== mon.0 v2:172.21.15.114:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9c90022da0 con 0x7f9ca0102ba0 2024-04-23T10:23:52.842 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:23:52.840+0000 7f9ca7653700 1 -- 172.21.15.114:0/526042645 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f9ca01a0fb0 con 0x7f9ca0102ba0 2024-04-23T10:23:52.843 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:23:52.841+0000 7f9ca7653700 1 -- 172.21.15.114:0/526042645 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f9ca0069730 con 0x7f9ca0102ba0 2024-04-23T10:23:52.843 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:23:52.842+0000 7f9c967fc700 1 -- 172.21.15.114:0/526042645 <== mon.0 v2:172.21.15.114:3300/0 4 ==== mgrmap(e 16) ==== 138647+0+0 (secure 0 0 0) 0x7f9c9002a070 con 0x7f9ca0102ba0 2024-04-23T10:23:52.843 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:23:52.843+0000 7f9c967fc700 1 --2- 172.21.15.114:0/526042645 >> [v2:172.21.15.114:6800/465333808,v1:172.21.15.114:6801/465333808] conn(0x7f9c8c0a3c40 0x7f9c8c0a6100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T10:23:52.843 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:23:52.843+0000 7f9ca53ef700 1 --2- 172.21.15.114:0/526042645 >> [v2:172.21.15.114:6800/465333808,v1:172.21.15.114:6801/465333808] conn(0x7f9c8c0a3c40 0x7f9c8c0a6100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T10:23:52.843 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:23:52.843+0000 7f9c967fc700 1 -- 172.21.15.114:0/526042645 <== mon.0 v2:172.21.15.114:3300/0 5 ==== osd_map(81..81 src has 1..81) ==== 6847+0+0 (secure 0 0 0) 0x7f9c900d0b80 con 0x7f9ca0102ba0 2024-04-23T10:23:52.844 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:23:52.843+0000 7f9ca53ef700 1 --2- 172.21.15.114:0/526042645 >> [v2:172.21.15.114:6800/465333808,v1:172.21.15.114:6801/465333808] conn(0x7f9c8c0a3c40 0x7f9c8c0a6100 secure :-1 s=READY pgs=122 cs=0 l=1 rev1=1 crypto rx=0x7f9ca0102a00 tx=0x7f9c980091c0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T10:23:52.846 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:23:52.848+0000 7f9c967fc700 1 -- 172.21.15.114:0/526042645 <== mon.0 v2:172.21.15.114:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f9c9005eff0 con 0x7f9ca0102ba0 2024-04-23T10:23:53.047 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:23:53.047+0000 7f9ca7653700 1 -- 172.21.15.114:0/526042645 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_command({"prefix": "osd stat", "format": "json"} v 0) -- 0x7f9ca0063650 con 0x7f9ca0102ba0 2024-04-23T10:23:53.047 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:23:53.048+0000 7f9c967fc700 1 -- 172.21.15.114:0/526042645 <== mon.0 v2:172.21.15.114:3300/0 7 ==== mon_command_ack([{"prefix": "osd stat", "format": "json"}]=0 v81) ==== 74+0+133 (secure 0 0 0) 0x7f9c90013070 con 0x7f9ca0102ba0 2024-04-23T10:23:53.047 INFO:teuthology.orchestra.run.smithi114.stdout: 2024-04-23T10:23:53.049 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:23:53.050+0000 7f9ca7653700 1 -- 172.21.15.114:0/526042645 >> [v2:172.21.15.114:6800/465333808,v1:172.21.15.114:6801/465333808] conn(0x7f9c8c0a3c40 msgr2=0x7f9c8c0a6100 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T10:23:53.049 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:23:53.050+0000 7f9ca7653700 1 --2- 172.21.15.114:0/526042645 >> [v2:172.21.15.114:6800/465333808,v1:172.21.15.114:6801/465333808] conn(0x7f9c8c0a3c40 0x7f9c8c0a6100 secure :-1 s=READY pgs=122 cs=0 l=1 rev1=1 crypto rx=0x7f9ca0102a00 tx=0x7f9c980091c0 comp rx=0 tx=0).stop 2024-04-23T10:23:53.049 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:23:53.050+0000 7f9ca7653700 1 -- 172.21.15.114:0/526042645 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f9ca0102ba0 msgr2=0x7f9ca019d920 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T10:23:53.049 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:23:53.050+0000 7f9ca7653700 1 --2- 172.21.15.114:0/526042645 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f9ca0102ba0 0x7f9ca019d920 secure :-1 s=READY pgs=150 cs=0 l=1 rev1=1 crypto rx=0x7f9c90000c00 tx=0x7f9c90009580 comp rx=0 tx=0).stop 2024-04-23T10:23:53.049 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:23:53.050+0000 7f9ca7653700 1 -- 172.21.15.114:0/526042645 shutdown_connections 2024-04-23T10:23:53.049 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:23:53.050+0000 7f9ca7653700 1 --2- 172.21.15.114:0/526042645 >> [v2:172.21.15.114:6800/465333808,v1:172.21.15.114:6801/465333808] conn(0x7f9c8c0a3c40 0x7f9c8c0a6100 unknown :-1 s=CLOSED pgs=122 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:23:53.049 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:23:53.050+0000 7f9ca7653700 1 --2- 172.21.15.114:0/526042645 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f9ca0103540 0x7f9ca019ff70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:23:53.049 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:23:53.050+0000 7f9ca7653700 1 --2- 172.21.15.114:0/526042645 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f9ca01019a0 0x7f9ca006b2a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:23:53.050 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:23:53.050+0000 7f9ca7653700 1 --2- 172.21.15.114:0/526042645 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f9ca0102ba0 0x7f9ca019d920 unknown :-1 s=CLOSED pgs=150 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:23:53.050 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:23:53.050+0000 7f9ca7653700 1 -- 172.21.15.114:0/526042645 >> 172.21.15.114:0/526042645 conn(0x7f9ca00fcf40 msgr2=0x7f9ca0105fe0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T10:23:53.050 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:23:53.050+0000 7f9ca7653700 1 -- 172.21.15.114:0/526042645 shutdown_connections 2024-04-23T10:23:53.050 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:23:53.050+0000 7f9ca7653700 1 -- 172.21.15.114:0/526042645 wait complete. 2024-04-23T10:23:53.328 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 23 10:23:53 smithi204 ceph-mon[32649]: from='osd.11 ' entity='osd.11' cmd='[{"prefix": "osd crush create-or-move", "id": 11, "weight":0.0873, "args": ["host=smithi204", "root=default"]}]': finished 2024-04-23T10:23:53.328 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 23 10:23:53 smithi204 ceph-mon[32649]: osdmap e81: 12 total, 11 up, 12 in 2024-04-23T10:23:53.328 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 23 10:23:53 smithi204 ceph-mon[32649]: from='mgr.14152 172.21.15.114:0/1529706138' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 11}]: dispatch 2024-04-23T10:23:53.328 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 23 10:23:53 smithi204 ceph-mon[32649]: from='mgr.14152 172.21.15.114:0/1529706138' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 11}]: dispatch 2024-04-23T10:23:53.328 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 23 10:23:53 smithi204 ceph-mon[32649]: from='client.? 172.21.15.114:0/526042645' entity='client.admin' cmd=[{"prefix": "osd stat", "format": "json"}]: dispatch 2024-04-23T10:23:53.328 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 23 10:23:53 smithi204 ceph-mon[32649]: from='osd.11 ' entity='osd.11' 2024-04-23T10:23:53.328 INFO:journalctl@ceph.osd.11.smithi204.stdout:Apr 23 10:23:53 smithi204 ceph-aa240572-0159-11ef-bc93-c7b262605968-osd-11[57644]: 2024-04-23T10:23:53.068+0000 7fcbcb575700 -1 osd.11 0 waiting for initial osdmap 2024-04-23T10:23:53.329 INFO:journalctl@ceph.osd.11.smithi204.stdout:Apr 23 10:23:53 smithi204 ceph-aa240572-0159-11ef-bc93-c7b262605968-osd-11[57644]: 2024-04-23T10:23:53.229+0000 7fcbc3360700 -1 osd.11 81 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-04-23T10:23:53.342 INFO:journalctl@ceph.mon.a.smithi114.stdout:Apr 23 10:23:53 smithi114 ceph-mon[29255]: from='osd.11 ' entity='osd.11' cmd='[{"prefix": "osd crush create-or-move", "id": 11, "weight":0.0873, "args": ["host=smithi204", "root=default"]}]': finished 2024-04-23T10:23:53.342 INFO:journalctl@ceph.mon.a.smithi114.stdout:Apr 23 10:23:53 smithi114 ceph-mon[29255]: osdmap e81: 12 total, 11 up, 12 in 2024-04-23T10:23:53.342 INFO:journalctl@ceph.mon.a.smithi114.stdout:Apr 23 10:23:53 smithi114 ceph-mon[29255]: from='mgr.14152 172.21.15.114:0/1529706138' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 11}]: dispatch 2024-04-23T10:23:53.342 INFO:journalctl@ceph.mon.a.smithi114.stdout:Apr 23 10:23:53 smithi114 ceph-mon[29255]: from='mgr.14152 172.21.15.114:0/1529706138' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 11}]: dispatch 2024-04-23T10:23:53.342 INFO:journalctl@ceph.mon.a.smithi114.stdout:Apr 23 10:23:53 smithi114 ceph-mon[29255]: from='client.? 172.21.15.114:0/526042645' entity='client.admin' cmd=[{"prefix": "osd stat", "format": "json"}]: dispatch 2024-04-23T10:23:53.342 INFO:journalctl@ceph.mon.a.smithi114.stdout:Apr 23 10:23:53 smithi114 ceph-mon[29255]: from='osd.11 ' entity='osd.11' 2024-04-23T10:23:53.492 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 23 10:23:53 smithi119 ceph-mon[32608]: from='osd.11 ' entity='osd.11' cmd='[{"prefix": "osd crush create-or-move", "id": 11, "weight":0.0873, "args": ["host=smithi204", "root=default"]}]': finished 2024-04-23T10:23:53.492 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 23 10:23:53 smithi119 ceph-mon[32608]: osdmap e81: 12 total, 11 up, 12 in 2024-04-23T10:23:53.492 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 23 10:23:53 smithi119 ceph-mon[32608]: from='mgr.14152 172.21.15.114:0/1529706138' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 11}]: dispatch 2024-04-23T10:23:53.492 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 23 10:23:53 smithi119 ceph-mon[32608]: from='mgr.14152 172.21.15.114:0/1529706138' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 11}]: dispatch 2024-04-23T10:23:53.492 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 23 10:23:53 smithi119 ceph-mon[32608]: from='client.? 172.21.15.114:0/526042645' entity='client.admin' cmd=[{"prefix": "osd stat", "format": "json"}]: dispatch 2024-04-23T10:23:53.492 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 23 10:23:53 smithi119 ceph-mon[32608]: from='osd.11 ' entity='osd.11' 2024-04-23T10:23:53.823 INFO:teuthology.orchestra.run.smithi114.stdout:{"epoch":81,"num_osds":12,"num_up_osds":11,"osd_up_since":1713867807,"num_in_osds":12,"osd_in_since":1713867812,"num_remapped_pgs":0} 2024-04-23T10:23:54.328 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 23 10:23:54 smithi204 ceph-mon[32649]: purged_snaps scrub starts 2024-04-23T10:23:54.329 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 23 10:23:54 smithi204 ceph-mon[32649]: purged_snaps scrub ok 2024-04-23T10:23:54.329 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 23 10:23:54 smithi204 ceph-mon[32649]: from='mgr.14152 172.21.15.114:0/1529706138' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 11}]: dispatch 2024-04-23T10:23:54.329 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 23 10:23:54 smithi204 ceph-mon[32649]: pgmap v255: 1 pgs: 1 active+clean; 577 KiB data, 333 MiB used, 983 GiB / 983 GiB avail 2024-04-23T10:23:54.329 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 23 10:23:54 smithi204 ceph-mon[32649]: Detected new or changed devices on smithi204 2024-04-23T10:23:54.329 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 23 10:23:54 smithi204 ceph-mon[32649]: from='mgr.14152 172.21.15.114:0/1529706138' entity='mgr.x' 2024-04-23T10:23:54.329 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 23 10:23:54 smithi204 ceph-mon[32649]: from='mgr.14152 172.21.15.114:0/1529706138' entity='mgr.x' 2024-04-23T10:23:54.329 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 23 10:23:54 smithi204 ceph-mon[32649]: from='mgr.14152 172.21.15.114:0/1529706138' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-04-23T10:23:54.329 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 23 10:23:54 smithi204 ceph-mon[32649]: from='mgr.14152 172.21.15.114:0/1529706138' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-04-23T10:23:54.329 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 23 10:23:54 smithi204 ceph-mon[32649]: from='mgr.14152 172.21.15.114:0/1529706138' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2024-04-23T10:23:54.329 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 23 10:23:54 smithi204 ceph-mon[32649]: from='mgr.14152 172.21.15.114:0/1529706138' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2024-04-23T10:23:54.329 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 23 10:23:54 smithi204 ceph-mon[32649]: Adjusting osd_memory_target on smithi204 to 4272M 2024-04-23T10:23:54.329 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 23 10:23:54 smithi204 ceph-mon[32649]: from='mgr.14152 172.21.15.114:0/1529706138' entity='mgr.x' 2024-04-23T10:23:54.329 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 23 10:23:54 smithi204 ceph-mon[32649]: from='mgr.14152 172.21.15.114:0/1529706138' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-23T10:23:54.329 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 23 10:23:54 smithi204 ceph-mon[32649]: from='mgr.14152 172.21.15.114:0/1529706138' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-23T10:23:54.330 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 23 10:23:54 smithi204 ceph-mon[32649]: from='mgr.14152 172.21.15.114:0/1529706138' entity='mgr.x' 2024-04-23T10:23:54.492 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 23 10:23:54 smithi119 ceph-mon[32608]: purged_snaps scrub starts 2024-04-23T10:23:54.492 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 23 10:23:54 smithi119 ceph-mon[32608]: purged_snaps scrub ok 2024-04-23T10:23:54.492 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 23 10:23:54 smithi119 ceph-mon[32608]: from='mgr.14152 172.21.15.114:0/1529706138' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 11}]: dispatch 2024-04-23T10:23:54.492 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 23 10:23:54 smithi119 ceph-mon[32608]: pgmap v255: 1 pgs: 1 active+clean; 577 KiB data, 333 MiB used, 983 GiB / 983 GiB avail 2024-04-23T10:23:54.492 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 23 10:23:54 smithi119 ceph-mon[32608]: Detected new or changed devices on smithi204 2024-04-23T10:23:54.492 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 23 10:23:54 smithi119 ceph-mon[32608]: from='mgr.14152 172.21.15.114:0/1529706138' entity='mgr.x' 2024-04-23T10:23:54.492 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 23 10:23:54 smithi119 ceph-mon[32608]: from='mgr.14152 172.21.15.114:0/1529706138' entity='mgr.x' 2024-04-23T10:23:54.492 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 23 10:23:54 smithi119 ceph-mon[32608]: from='mgr.14152 172.21.15.114:0/1529706138' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-04-23T10:23:54.492 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 23 10:23:54 smithi119 ceph-mon[32608]: from='mgr.14152 172.21.15.114:0/1529706138' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-04-23T10:23:54.493 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 23 10:23:54 smithi119 ceph-mon[32608]: from='mgr.14152 172.21.15.114:0/1529706138' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2024-04-23T10:23:54.493 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 23 10:23:54 smithi119 ceph-mon[32608]: from='mgr.14152 172.21.15.114:0/1529706138' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2024-04-23T10:23:54.493 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 23 10:23:54 smithi119 ceph-mon[32608]: Adjusting osd_memory_target on smithi204 to 4272M 2024-04-23T10:23:54.493 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 23 10:23:54 smithi119 ceph-mon[32608]: from='mgr.14152 172.21.15.114:0/1529706138' entity='mgr.x' 2024-04-23T10:23:54.493 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 23 10:23:54 smithi119 ceph-mon[32608]: from='mgr.14152 172.21.15.114:0/1529706138' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-23T10:23:54.493 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 23 10:23:54 smithi119 ceph-mon[32608]: from='mgr.14152 172.21.15.114:0/1529706138' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-23T10:23:54.493 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 23 10:23:54 smithi119 ceph-mon[32608]: from='mgr.14152 172.21.15.114:0/1529706138' entity='mgr.x' 2024-04-23T10:23:54.504 INFO:journalctl@ceph.mon.a.smithi114.stdout:Apr 23 10:23:54 smithi114 ceph-mon[29255]: purged_snaps scrub starts 2024-04-23T10:23:54.504 INFO:journalctl@ceph.mon.a.smithi114.stdout:Apr 23 10:23:54 smithi114 ceph-mon[29255]: purged_snaps scrub ok 2024-04-23T10:23:54.504 INFO:journalctl@ceph.mon.a.smithi114.stdout:Apr 23 10:23:54 smithi114 ceph-mon[29255]: from='mgr.14152 172.21.15.114:0/1529706138' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 11}]: dispatch 2024-04-23T10:23:54.504 INFO:journalctl@ceph.mon.a.smithi114.stdout:Apr 23 10:23:54 smithi114 ceph-mon[29255]: pgmap v255: 1 pgs: 1 active+clean; 577 KiB data, 333 MiB used, 983 GiB / 983 GiB avail 2024-04-23T10:23:54.504 INFO:journalctl@ceph.mon.a.smithi114.stdout:Apr 23 10:23:54 smithi114 ceph-mon[29255]: Detected new or changed devices on smithi204 2024-04-23T10:23:54.504 INFO:journalctl@ceph.mon.a.smithi114.stdout:Apr 23 10:23:54 smithi114 ceph-mon[29255]: from='mgr.14152 172.21.15.114:0/1529706138' entity='mgr.x' 2024-04-23T10:23:54.504 INFO:journalctl@ceph.mon.a.smithi114.stdout:Apr 23 10:23:54 smithi114 ceph-mon[29255]: from='mgr.14152 172.21.15.114:0/1529706138' entity='mgr.x' 2024-04-23T10:23:54.504 INFO:journalctl@ceph.mon.a.smithi114.stdout:Apr 23 10:23:54 smithi114 ceph-mon[29255]: from='mgr.14152 172.21.15.114:0/1529706138' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-04-23T10:23:54.504 INFO:journalctl@ceph.mon.a.smithi114.stdout:Apr 23 10:23:54 smithi114 ceph-mon[29255]: from='mgr.14152 172.21.15.114:0/1529706138' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-04-23T10:23:54.505 INFO:journalctl@ceph.mon.a.smithi114.stdout:Apr 23 10:23:54 smithi114 ceph-mon[29255]: from='mgr.14152 172.21.15.114:0/1529706138' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2024-04-23T10:23:54.505 INFO:journalctl@ceph.mon.a.smithi114.stdout:Apr 23 10:23:54 smithi114 ceph-mon[29255]: from='mgr.14152 172.21.15.114:0/1529706138' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2024-04-23T10:23:54.505 INFO:journalctl@ceph.mon.a.smithi114.stdout:Apr 23 10:23:54 smithi114 ceph-mon[29255]: Adjusting osd_memory_target on smithi204 to 4272M 2024-04-23T10:23:54.505 INFO:journalctl@ceph.mon.a.smithi114.stdout:Apr 23 10:23:54 smithi114 ceph-mon[29255]: from='mgr.14152 172.21.15.114:0/1529706138' entity='mgr.x' 2024-04-23T10:23:54.505 INFO:journalctl@ceph.mon.a.smithi114.stdout:Apr 23 10:23:54 smithi114 ceph-mon[29255]: from='mgr.14152 172.21.15.114:0/1529706138' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-23T10:23:54.505 INFO:journalctl@ceph.mon.a.smithi114.stdout:Apr 23 10:23:54 smithi114 ceph-mon[29255]: from='mgr.14152 172.21.15.114:0/1529706138' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-23T10:23:54.505 INFO:journalctl@ceph.mon.a.smithi114.stdout:Apr 23 10:23:54 smithi114 ceph-mon[29255]: from='mgr.14152 172.21.15.114:0/1529706138' entity='mgr.x' 2024-04-23T10:23:54.824 DEBUG:teuthology.orchestra.run.smithi114:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d69f13d827952bb07bb9b7973bca14f97d764d6f shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid aa240572-0159-11ef-bc93-c7b262605968 -- ceph osd stat -f json 2024-04-23T10:23:55.063 INFO:teuthology.orchestra.run.smithi114.stderr:Inferring config /var/lib/ceph/aa240572-0159-11ef-bc93-c7b262605968/mon.a/config 2024-04-23T10:23:55.479 INFO:journalctl@ceph.mon.a.smithi114.stdout:Apr 23 10:23:55 smithi114 ceph-mon[29255]: from='mgr.14152 172.21.15.114:0/1529706138' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 11}]: dispatch 2024-04-23T10:23:55.479 INFO:journalctl@ceph.mon.a.smithi114.stdout:Apr 23 10:23:55 smithi114 ceph-mon[29255]: osd.11 [v2:172.21.15.204:6824/2407298490,v1:172.21.15.204:6825/2407298490] boot 2024-04-23T10:23:55.479 INFO:journalctl@ceph.mon.a.smithi114.stdout:Apr 23 10:23:55 smithi114 ceph-mon[29255]: osdmap e82: 12 total, 12 up, 12 in 2024-04-23T10:23:55.479 INFO:journalctl@ceph.mon.a.smithi114.stdout:Apr 23 10:23:55 smithi114 ceph-mon[29255]: from='mgr.14152 172.21.15.114:0/1529706138' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 11}]: dispatch 2024-04-23T10:23:55.492 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 23 10:23:55 smithi119 ceph-mon[32608]: from='mgr.14152 172.21.15.114:0/1529706138' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 11}]: dispatch 2024-04-23T10:23:55.492 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 23 10:23:55 smithi119 ceph-mon[32608]: osd.11 [v2:172.21.15.204:6824/2407298490,v1:172.21.15.204:6825/2407298490] boot 2024-04-23T10:23:55.492 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 23 10:23:55 smithi119 ceph-mon[32608]: osdmap e82: 12 total, 12 up, 12 in 2024-04-23T10:23:55.492 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 23 10:23:55 smithi119 ceph-mon[32608]: from='mgr.14152 172.21.15.114:0/1529706138' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 11}]: dispatch 2024-04-23T10:23:55.511 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 23 10:23:55 smithi204 ceph-mon[32649]: from='mgr.14152 172.21.15.114:0/1529706138' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 11}]: dispatch 2024-04-23T10:23:55.511 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 23 10:23:55 smithi204 ceph-mon[32649]: osd.11 [v2:172.21.15.204:6824/2407298490,v1:172.21.15.204:6825/2407298490] boot 2024-04-23T10:23:55.512 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 23 10:23:55 smithi204 ceph-mon[32649]: osdmap e82: 12 total, 12 up, 12 in 2024-04-23T10:23:55.512 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 23 10:23:55 smithi204 ceph-mon[32649]: from='mgr.14152 172.21.15.114:0/1529706138' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 11}]: dispatch 2024-04-23T10:23:56.435 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 23 10:23:56 smithi204 ceph-mon[32649]: osdmap e83: 12 total, 12 up, 12 in 2024-04-23T10:23:56.435 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 23 10:23:56 smithi204 ceph-mon[32649]: pgmap v258: 1 pgs: 1 active+clean; 577 KiB data, 365 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-23T10:23:56.478 INFO:journalctl@ceph.mon.a.smithi114.stdout:Apr 23 10:23:56 smithi114 ceph-mon[29255]: osdmap e83: 12 total, 12 up, 12 in 2024-04-23T10:23:56.478 INFO:journalctl@ceph.mon.a.smithi114.stdout:Apr 23 10:23:56 smithi114 ceph-mon[29255]: pgmap v258: 1 pgs: 1 active+clean; 577 KiB data, 365 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-23T10:23:56.491 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 23 10:23:56 smithi119 ceph-mon[32608]: osdmap e83: 12 total, 12 up, 12 in 2024-04-23T10:23:56.492 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 23 10:23:56 smithi119 ceph-mon[32608]: pgmap v258: 1 pgs: 1 active+clean; 577 KiB data, 365 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-23T10:23:56.679 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:23:56.678+0000 7feb2f40b700 1 -- 172.21.15.114:0/1617262119 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7feb28103c90 msgr2=0x7feb281040f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T10:23:56.679 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:23:56.678+0000 7feb2f40b700 1 --2- 172.21.15.114:0/1617262119 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7feb28103c90 0x7feb281040f0 secure :-1 s=READY pgs=151 cs=0 l=1 rev1=1 crypto rx=0x7feb240099c0 tx=0x7feb24009cd0 comp rx=0 tx=0).stop 2024-04-23T10:23:56.680 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:23:56.678+0000 7feb2f40b700 1 -- 172.21.15.114:0/1617262119 shutdown_connections 2024-04-23T10:23:56.680 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:23:56.678+0000 7feb2f40b700 1 --2- 172.21.15.114:0/1617262119 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7feb28102a90 0x7feb28102eb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:23:56.680 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:23:56.678+0000 7feb2f40b700 1 --2- 172.21.15.114:0/1617262119 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7feb28104630 0x7feb2810aee0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:23:56.680 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:23:56.678+0000 7feb2f40b700 1 --2- 172.21.15.114:0/1617262119 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7feb28103c90 0x7feb281040f0 unknown :-1 s=CLOSED pgs=151 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:23:56.680 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:23:56.678+0000 7feb2f40b700 1 -- 172.21.15.114:0/1617262119 >> 172.21.15.114:0/1617262119 conn(0x7feb280fe030 msgr2=0x7feb28100470 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T10:23:56.680 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:23:56.678+0000 7feb2f40b700 1 -- 172.21.15.114:0/1617262119 shutdown_connections 2024-04-23T10:23:56.680 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:23:56.678+0000 7feb2f40b700 1 -- 172.21.15.114:0/1617262119 wait complete. 2024-04-23T10:23:56.680 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:23:56.679+0000 7feb2f40b700 1 Processor -- start 2024-04-23T10:23:56.680 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:23:56.679+0000 7feb2f40b700 1 -- start start 2024-04-23T10:23:56.680 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:23:56.679+0000 7feb2f40b700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7feb28102a90 0x7feb28198c40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T10:23:56.680 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:23:56.679+0000 7feb2f40b700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7feb28103c90 0x7feb28199180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T10:23:56.680 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:23:56.679+0000 7feb2f40b700 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7feb28104630 0x7feb281a01c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T10:23:56.680 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:23:56.679+0000 7feb2f40b700 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x7feb2810e580 con 0x7feb28102a90 2024-04-23T10:23:56.680 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:23:56.679+0000 7feb2d1a7700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7feb28102a90 0x7feb28198c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T10:23:56.681 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:23:56.679+0000 7feb2f40b700 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 -- 0x7feb2810e400 con 0x7feb28103c90 2024-04-23T10:23:56.681 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:23:56.679+0000 7feb2f40b700 1 -- --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_getmap magic: 0 -- 0x7feb2810e700 con 0x7feb28104630 2024-04-23T10:23:56.681 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:23:56.679+0000 7feb2d1a7700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7feb28102a90 0x7feb28198c40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.114:36092/0 (socket says 172.21.15.114:36092) 2024-04-23T10:23:56.681 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:23:56.679+0000 7feb2d1a7700 1 -- 172.21.15.114:0/2649997417 learned_addr learned my addr 172.21.15.114:0/2649997417 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-04-23T10:23:56.681 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:23:56.679+0000 7feb2d9a8700 1 --2- 172.21.15.114:0/2649997417 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7feb28104630 0x7feb281a01c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T10:23:56.681 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:23:56.679+0000 7feb2c9a6700 1 --2- 172.21.15.114:0/2649997417 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7feb28103c90 0x7feb28199180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T10:23:56.681 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:23:56.680+0000 7feb2d1a7700 1 -- 172.21.15.114:0/2649997417 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7feb28104630 msgr2=0x7feb281a01c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T10:23:56.681 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:23:56.680+0000 7feb2d1a7700 1 --2- 172.21.15.114:0/2649997417 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7feb28104630 0x7feb281a01c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:23:56.681 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:23:56.680+0000 7feb2d1a7700 1 -- 172.21.15.114:0/2649997417 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7feb28103c90 msgr2=0x7feb28199180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T10:23:56.681 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:23:56.680+0000 7feb2d1a7700 1 --2- 172.21.15.114:0/2649997417 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7feb28103c90 0x7feb28199180 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:23:56.681 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:23:56.680+0000 7feb2d1a7700 1 -- 172.21.15.114:0/2649997417 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7feb281a0890 con 0x7feb28102a90 2024-04-23T10:23:56.681 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:23:56.680+0000 7feb2d1a7700 1 --2- 172.21.15.114:0/2649997417 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7feb28102a90 0x7feb28198c40 secure :-1 s=READY pgs=152 cs=0 l=1 rev1=1 crypto rx=0x7feb1800d980 tx=0x7feb1800dc90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T10:23:56.681 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:23:56.680+0000 7feb1e7fc700 1 -- 172.21.15.114:0/2649997417 <== mon.0 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7feb18017030 con 0x7feb28102a90 2024-04-23T10:23:56.682 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:23:56.680+0000 7feb1e7fc700 1 -- 172.21.15.114:0/2649997417 <== mon.0 v2:172.21.15.114:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7feb180044d0 con 0x7feb28102a90 2024-04-23T10:23:56.682 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:23:56.680+0000 7feb1e7fc700 1 -- 172.21.15.114:0/2649997417 <== mon.0 v2:172.21.15.114:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7feb18003970 con 0x7feb28102a90 2024-04-23T10:23:56.682 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:23:56.680+0000 7feb2f40b700 1 -- 172.21.15.114:0/2649997417 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7feb281a0b80 con 0x7feb28102a90 2024-04-23T10:23:56.682 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:23:56.680+0000 7feb2f40b700 1 -- 172.21.15.114:0/2649997417 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7feb281a1110 con 0x7feb28102a90 2024-04-23T10:23:56.683 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:23:56.681+0000 7feb2f40b700 1 -- 172.21.15.114:0/2649997417 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7feb28069730 con 0x7feb28102a90 2024-04-23T10:23:56.683 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:23:56.682+0000 7feb1e7fc700 1 -- 172.21.15.114:0/2649997417 <== mon.0 v2:172.21.15.114:3300/0 4 ==== mgrmap(e 16) ==== 138647+0+0 (secure 0 0 0) 0x7feb1801a070 con 0x7feb28102a90 2024-04-23T10:23:56.683 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:23:56.683+0000 7feb1e7fc700 1 --2- 172.21.15.114:0/2649997417 >> [v2:172.21.15.114:6800/465333808,v1:172.21.15.114:6801/465333808] conn(0x7feb140a3cc0 0x7feb140a6180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T10:23:56.683 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:23:56.683+0000 7feb2c9a6700 1 --2- 172.21.15.114:0/2649997417 >> [v2:172.21.15.114:6800/465333808,v1:172.21.15.114:6801/465333808] conn(0x7feb140a3cc0 0x7feb140a6180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T10:23:56.683 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:23:56.683+0000 7feb1e7fc700 1 -- 172.21.15.114:0/2649997417 <== mon.0 v2:172.21.15.114:3300/0 5 ==== osd_map(83..83 src has 1..83) ==== 7127+0+0 (secure 0 0 0) 0x7feb180cf790 con 0x7feb28102a90 2024-04-23T10:23:56.683 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:23:56.683+0000 7feb2c9a6700 1 --2- 172.21.15.114:0/2649997417 >> [v2:172.21.15.114:6800/465333808,v1:172.21.15.114:6801/465333808] conn(0x7feb140a3cc0 0x7feb140a6180 secure :-1 s=READY pgs=126 cs=0 l=1 rev1=1 crypto rx=0x7feb24009990 tx=0x7feb2400b870 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T10:23:56.686 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:23:56.687+0000 7feb1e7fc700 1 -- 172.21.15.114:0/2649997417 <== mon.0 v2:172.21.15.114:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7feb1809c610 con 0x7feb28102a90 2024-04-23T10:23:56.883 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:23:56.884+0000 7feb2f40b700 1 -- 172.21.15.114:0/2649997417 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_command({"prefix": "osd stat", "format": "json"} v 0) -- 0x7feb2802e050 con 0x7feb28102a90 2024-04-23T10:23:56.883 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:23:56.885+0000 7feb1e7fc700 1 -- 172.21.15.114:0/2649997417 <== mon.0 v2:172.21.15.114:3300/0 7 ==== mon_command_ack([{"prefix": "osd stat", "format": "json"}]=0 v83) ==== 74+0+133 (secure 0 0 0) 0x7feb1809bf10 con 0x7feb28102a90 2024-04-23T10:23:56.884 INFO:teuthology.orchestra.run.smithi114.stdout: 2024-04-23T10:23:56.886 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:23:56.887+0000 7feb2f40b700 1 -- 172.21.15.114:0/2649997417 >> [v2:172.21.15.114:6800/465333808,v1:172.21.15.114:6801/465333808] conn(0x7feb140a3cc0 msgr2=0x7feb140a6180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T10:23:56.886 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:23:56.887+0000 7feb2f40b700 1 --2- 172.21.15.114:0/2649997417 >> [v2:172.21.15.114:6800/465333808,v1:172.21.15.114:6801/465333808] conn(0x7feb140a3cc0 0x7feb140a6180 secure :-1 s=READY pgs=126 cs=0 l=1 rev1=1 crypto rx=0x7feb24009990 tx=0x7feb2400b870 comp rx=0 tx=0).stop 2024-04-23T10:23:56.886 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:23:56.887+0000 7feb2f40b700 1 -- 172.21.15.114:0/2649997417 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7feb28102a90 msgr2=0x7feb28198c40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T10:23:56.886 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:23:56.887+0000 7feb2f40b700 1 --2- 172.21.15.114:0/2649997417 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7feb28102a90 0x7feb28198c40 secure :-1 s=READY pgs=152 cs=0 l=1 rev1=1 crypto rx=0x7feb1800d980 tx=0x7feb1800dc90 comp rx=0 tx=0).stop 2024-04-23T10:23:56.886 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:23:56.887+0000 7feb2f40b700 1 -- 172.21.15.114:0/2649997417 shutdown_connections 2024-04-23T10:23:56.886 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:23:56.887+0000 7feb2f40b700 1 --2- 172.21.15.114:0/2649997417 >> [v2:172.21.15.114:6800/465333808,v1:172.21.15.114:6801/465333808] conn(0x7feb140a3cc0 0x7feb140a6180 unknown :-1 s=CLOSED pgs=126 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:23:56.886 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:23:56.887+0000 7feb2f40b700 1 --2- 172.21.15.114:0/2649997417 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7feb28104630 0x7feb281a01c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:23:56.887 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:23:56.887+0000 7feb2f40b700 1 --2- 172.21.15.114:0/2649997417 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7feb28102a90 0x7feb28198c40 unknown :-1 s=CLOSED pgs=152 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:23:56.887 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:23:56.887+0000 7feb2f40b700 1 --2- 172.21.15.114:0/2649997417 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7feb28103c90 0x7feb28199180 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:23:56.887 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:23:56.887+0000 7feb2f40b700 1 -- 172.21.15.114:0/2649997417 >> 172.21.15.114:0/2649997417 conn(0x7feb280fe030 msgr2=0x7feb28107010 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T10:23:56.887 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:23:56.887+0000 7feb2f40b700 1 -- 172.21.15.114:0/2649997417 shutdown_connections 2024-04-23T10:23:56.887 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:23:56.887+0000 7feb2f40b700 1 -- 172.21.15.114:0/2649997417 wait complete. 2024-04-23T10:23:57.492 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 23 10:23:57 smithi119 ceph-mon[32608]: from='client.? 172.21.15.114:0/2649997417' entity='client.admin' cmd=[{"prefix": "osd stat", "format": "json"}]: dispatch 2024-04-23T10:23:57.504 INFO:journalctl@ceph.mon.a.smithi114.stdout:Apr 23 10:23:57 smithi114 ceph-mon[29255]: from='client.? 172.21.15.114:0/2649997417' entity='client.admin' cmd=[{"prefix": "osd stat", "format": "json"}]: dispatch 2024-04-23T10:23:57.578 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 23 10:23:57 smithi204 ceph-mon[32649]: from='client.? 172.21.15.114:0/2649997417' entity='client.admin' cmd=[{"prefix": "osd stat", "format": "json"}]: dispatch 2024-04-23T10:23:57.640 INFO:teuthology.orchestra.run.smithi114.stdout:{"epoch":83,"num_osds":12,"num_up_osds":12,"osd_up_since":1713867834,"num_in_osds":12,"osd_in_since":1713867812,"num_remapped_pgs":0} 2024-04-23T10:23:57.640 DEBUG:teuthology.orchestra.run.smithi114:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d69f13d827952bb07bb9b7973bca14f97d764d6f shell --fsid aa240572-0159-11ef-bc93-c7b262605968 -- ceph osd dump --format=json 2024-04-23T10:23:57.878 INFO:teuthology.orchestra.run.smithi114.stderr:Inferring config /var/lib/ceph/aa240572-0159-11ef-bc93-c7b262605968/mon.a/config 2024-04-23T10:23:58.254 INFO:journalctl@ceph.mon.a.smithi114.stdout:Apr 23 10:23:58 smithi114 ceph-mon[29255]: pgmap v259: 1 pgs: 1 active+clean; 577 KiB data, 365 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-23T10:23:58.492 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 23 10:23:58 smithi119 ceph-mon[32608]: pgmap v259: 1 pgs: 1 active+clean; 577 KiB data, 365 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-23T10:23:58.578 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 23 10:23:58 smithi204 ceph-mon[32649]: pgmap v259: 1 pgs: 1 active+clean; 577 KiB data, 365 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-23T10:23:59.465 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:23:59.463+0000 7f2e240dd700 1 -- 172.21.15.114:0/989917041 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f2e1c101dc0 msgr2=0x7f2e1c10f830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T10:23:59.465 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:23:59.463+0000 7f2e240dd700 1 --2- 172.21.15.114:0/989917041 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f2e1c101dc0 0x7f2e1c10f830 secure :-1 s=READY pgs=153 cs=0 l=1 rev1=1 crypto rx=0x7f2e1400b0a0 tx=0x7f2e1400b3b0 comp rx=0 tx=0).stop 2024-04-23T10:23:59.465 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:23:59.463+0000 7f2e240dd700 1 -- 172.21.15.114:0/989917041 shutdown_connections 2024-04-23T10:23:59.465 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:23:59.463+0000 7f2e240dd700 1 --2- 172.21.15.114:0/989917041 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f2e1c107b20 0x7f2e1c107f00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:23:59.466 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:23:59.463+0000 7f2e240dd700 1 --2- 172.21.15.114:0/989917041 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f2e1c101dc0 0x7f2e1c10f830 unknown :-1 s=CLOSED pgs=153 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:23:59.466 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:23:59.463+0000 7f2e240dd700 1 --2- 172.21.15.114:0/989917041 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f2e1c101400 0x7f2e1c101880 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:23:59.466 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:23:59.463+0000 7f2e240dd700 1 -- 172.21.15.114:0/989917041 >> 172.21.15.114:0/989917041 conn(0x7f2e1c076c60 msgr2=0x7f2e1c077070 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T10:23:59.466 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:23:59.464+0000 7f2e240dd700 1 -- 172.21.15.114:0/989917041 shutdown_connections 2024-04-23T10:23:59.466 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:23:59.464+0000 7f2e240dd700 1 -- 172.21.15.114:0/989917041 wait complete. 2024-04-23T10:23:59.466 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:23:59.464+0000 7f2e240dd700 1 Processor -- start 2024-04-23T10:23:59.466 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:23:59.464+0000 7f2e240dd700 1 -- start start 2024-04-23T10:23:59.466 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:23:59.464+0000 7f2e240dd700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f2e1c101400 0x7f2e1c1025e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T10:23:59.466 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:23:59.464+0000 7f2e240dd700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f2e1c101dc0 0x7f2e1c102b20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T10:23:59.466 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:23:59.464+0000 7f2e240dd700 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f2e1c107b20 0x7f2e1c103060 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T10:23:59.466 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:23:59.464+0000 7f2e240dd700 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x7f2e1c112cb0 con 0x7f2e1c101400 2024-04-23T10:23:59.466 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:23:59.464+0000 7f2e240dd700 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 -- 0x7f2e1c112b30 con 0x7f2e1c101dc0 2024-04-23T10:23:59.467 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:23:59.464+0000 7f2e240dd700 1 -- --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_getmap magic: 0 -- 0x7f2e1c112e30 con 0x7f2e1c107b20 2024-04-23T10:23:59.467 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:23:59.464+0000 7f2e21e79700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f2e1c101400 0x7f2e1c1025e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T10:23:59.467 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:23:59.464+0000 7f2e21e79700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f2e1c101400 0x7f2e1c1025e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.114:36110/0 (socket says 172.21.15.114:36110) 2024-04-23T10:23:59.467 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:23:59.464+0000 7f2e21e79700 1 -- 172.21.15.114:0/381236297 learned_addr learned my addr 172.21.15.114:0/381236297 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-04-23T10:23:59.467 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:23:59.465+0000 7f2e2267a700 1 --2- 172.21.15.114:0/381236297 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f2e1c107b20 0x7f2e1c103060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T10:23:59.467 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:23:59.465+0000 7f2e21678700 1 --2- 172.21.15.114:0/381236297 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f2e1c101dc0 0x7f2e1c102b20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T10:23:59.467 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:23:59.465+0000 7f2e21e79700 1 -- 172.21.15.114:0/381236297 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f2e1c107b20 msgr2=0x7f2e1c103060 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T10:23:59.467 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:23:59.465+0000 7f2e21e79700 1 --2- 172.21.15.114:0/381236297 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f2e1c107b20 0x7f2e1c103060 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:23:59.467 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:23:59.465+0000 7f2e21e79700 1 -- 172.21.15.114:0/381236297 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f2e1c101dc0 msgr2=0x7f2e1c102b20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T10:23:59.467 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:23:59.465+0000 7f2e21e79700 1 --2- 172.21.15.114:0/381236297 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f2e1c101dc0 0x7f2e1c102b20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:23:59.467 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:23:59.465+0000 7f2e21e79700 1 -- 172.21.15.114:0/381236297 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f2e1c1a7520 con 0x7f2e1c101400 2024-04-23T10:23:59.467 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:23:59.465+0000 7f2e21678700 1 --2- 172.21.15.114:0/381236297 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f2e1c101dc0 0x7f2e1c102b20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-23T10:23:59.468 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:23:59.465+0000 7f2e2267a700 1 --2- 172.21.15.114:0/381236297 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f2e1c107b20 0x7f2e1c103060 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-23T10:23:59.468 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:23:59.465+0000 7f2e21e79700 1 --2- 172.21.15.114:0/381236297 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f2e1c101400 0x7f2e1c1025e0 secure :-1 s=READY pgs=154 cs=0 l=1 rev1=1 crypto rx=0x7f2e1800d980 tx=0x7f2e1800dd40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T10:23:59.468 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:23:59.465+0000 7f2e12ffd700 1 -- 172.21.15.114:0/381236297 <== mon.0 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2e18017030 con 0x7f2e1c101400 2024-04-23T10:23:59.468 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:23:59.465+0000 7f2e240dd700 1 -- 172.21.15.114:0/381236297 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f2e1c1a7810 con 0x7f2e1c101400 2024-04-23T10:23:59.468 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:23:59.465+0000 7f2e12ffd700 1 -- 172.21.15.114:0/381236297 <== mon.0 v2:172.21.15.114:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f2e180044d0 con 0x7f2e1c101400 2024-04-23T10:23:59.468 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:23:59.465+0000 7f2e12ffd700 1 -- 172.21.15.114:0/381236297 <== mon.0 v2:172.21.15.114:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2e18003aa0 con 0x7f2e1c101400 2024-04-23T10:23:59.468 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:23:59.465+0000 7f2e240dd700 1 -- 172.21.15.114:0/381236297 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f2e1c1a7d50 con 0x7f2e1c101400 2024-04-23T10:23:59.468 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:23:59.466+0000 7f2e240dd700 1 -- 172.21.15.114:0/381236297 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f2e1c069730 con 0x7f2e1c101400 2024-04-23T10:23:59.469 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:23:59.467+0000 7f2e12ffd700 1 -- 172.21.15.114:0/381236297 <== mon.0 v2:172.21.15.114:3300/0 4 ==== mgrmap(e 16) ==== 138647+0+0 (secure 0 0 0) 0x7f2e1801a070 con 0x7f2e1c101400 2024-04-23T10:23:59.469 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:23:59.468+0000 7f2e12ffd700 1 --2- 172.21.15.114:0/381236297 >> [v2:172.21.15.114:6800/465333808,v1:172.21.15.114:6801/465333808] conn(0x7f2e080a3c40 0x7f2e080a6100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T10:23:59.469 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:23:59.468+0000 7f2e21678700 1 --2- 172.21.15.114:0/381236297 >> [v2:172.21.15.114:6800/465333808,v1:172.21.15.114:6801/465333808] conn(0x7f2e080a3c40 0x7f2e080a6100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T10:23:59.469 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:23:59.468+0000 7f2e12ffd700 1 -- 172.21.15.114:0/381236297 <== mon.0 v2:172.21.15.114:3300/0 5 ==== osd_map(83..83 src has 1..83) ==== 7127+0+0 (secure 0 0 0) 0x7f2e180cfc90 con 0x7f2e1c101400 2024-04-23T10:23:59.469 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:23:59.468+0000 7f2e21678700 1 --2- 172.21.15.114:0/381236297 >> [v2:172.21.15.114:6800/465333808,v1:172.21.15.114:6801/465333808] conn(0x7f2e080a3c40 0x7f2e080a6100 secure :-1 s=READY pgs=127 cs=0 l=1 rev1=1 crypto rx=0x7f2e0c005fd0 tx=0x7f2e0c005e20 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T10:23:59.470 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:23:59.471+0000 7f2e12ffd700 1 -- 172.21.15.114:0/381236297 <== mon.0 v2:172.21.15.114:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f2e1809cb10 con 0x7f2e1c101400 2024-04-23T10:23:59.672 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:23:59.673+0000 7f2e240dd700 1 -- 172.21.15.114:0/381236297 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f2e1c02df80 con 0x7f2e1c101400 2024-04-23T10:23:59.673 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:23:59.674+0000 7f2e12ffd700 1 -- 172.21.15.114:0/381236297 <== mon.0 v2:172.21.15.114:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v83) ==== 74+0+18528 (secure 0 0 0) 0x7f2e1809c410 con 0x7f2e1c101400 2024-04-23T10:23:59.674 INFO:teuthology.orchestra.run.smithi114.stdout: 2024-04-23T10:23:59.674 INFO:teuthology.orchestra.run.smithi114.stdout:{"epoch":83,"fsid":"aa240572-0159-11ef-bc93-c7b262605968","created":"2024-04-23T10:16:53.351136+0000","modified":"2024-04-23T10:23:55.163231+0000","last_up_change":"2024-04-23T10:23:54.153269+0000","last_in_change":"2024-04-23T10:23:32.963085+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":26,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":1,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-04-23T10:20:11.970297+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":12,"score_stable":12,"optimal_score":0.25,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"fd64adde-9445-45a4-a171-ef656e2c5709","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6802","nonce":736366878},{"type":"v1","addr":"172.21.15.114:6803","nonce":736366878}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6804","nonce":736366878},{"type":"v1","addr":"172.21.15.114:6805","nonce":736366878}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6808","nonce":736366878},{"type":"v1","addr":"172.21.15.114:6809","nonce":736366878}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6806","nonce":736366878},{"type":"v1","addr":"172.21.15.114:6807","nonce":736366878}]},"public_addr":"172.21.15.114:6803/736366878","cluster_addr":"172.21.15.114:6805/736366878","heartbeat_back_addr":"172.21.15.114:6809/736366878","heartbeat_front_addr":"172.21.15.114:6807/736366878","state":["exists","up"]},{"osd":1,"uuid":"e279a1e6-2702-472a-930a-e08ece76a861","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":42,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6800","nonce":1133540779},{"type":"v1","addr":"172.21.15.119:6801","nonce":1133540779}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6802","nonce":1133540779},{"type":"v1","addr":"172.21.15.119:6803","nonce":1133540779}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6806","nonce":1133540779},{"type":"v1","addr":"172.21.15.119:6807","nonce":1133540779}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6804","nonce":1133540779},{"type":"v1","addr":"172.21.15.119:6805","nonce":1133540779}]},"public_addr":"172.21.15.119:6801/1133540779","cluster_addr":"172.21.15.119:6803/1133540779","heartbeat_back_addr":"172.21.15.119:6807/1133540779","heartbeat_front_addr":"172.21.15.119:6805/1133540779","state":["exists","up"]},{"osd":2,"uuid":"f4c9f2b4-ff49-4e14-89cd-b477feeb9382","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6800","nonce":799652511},{"type":"v1","addr":"172.21.15.204:6801","nonce":799652511}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6802","nonce":799652511},{"type":"v1","addr":"172.21.15.204:6803","nonce":799652511}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6806","nonce":799652511},{"type":"v1","addr":"172.21.15.204:6807","nonce":799652511}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6804","nonce":799652511},{"type":"v1","addr":"172.21.15.204:6805","nonce":799652511}]},"public_addr":"172.21.15.204:6801/799652511","cluster_addr":"172.21.15.204:6803/799652511","heartbeat_back_addr":"172.21.15.204:6807/799652511","heartbeat_front_addr":"172.21.15.204:6805/799652511","state":["exists","up"]},{"osd":3,"uuid":"52d2a566-e71c-4bb3-8d6c-5abdd43177c2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":25,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6810","nonce":4275595091},{"type":"v1","addr":"172.21.15.114:6811","nonce":4275595091}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6812","nonce":4275595091},{"type":"v1","addr":"172.21.15.114:6813","nonce":4275595091}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6816","nonce":4275595091},{"type":"v1","addr":"172.21.15.114:6817","nonce":4275595091}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6814","nonce":4275595091},{"type":"v1","addr":"172.21.15.114:6815","nonce":4275595091}]},"public_addr":"172.21.15.114:6811/4275595091","cluster_addr":"172.21.15.114:6813/4275595091","heartbeat_back_addr":"172.21.15.114:6817/4275595091","heartbeat_front_addr":"172.21.15.114:6815/4275595091","state":["exists","up"]},{"osd":4,"uuid":"d911a7c1-4bdc-4d28-b8a6-73896a6e7de6","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":33,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6808","nonce":1611007172},{"type":"v1","addr":"172.21.15.119:6809","nonce":1611007172}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6810","nonce":1611007172},{"type":"v1","addr":"172.21.15.119:6811","nonce":1611007172}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6814","nonce":1611007172},{"type":"v1","addr":"172.21.15.119:6815","nonce":1611007172}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6812","nonce":1611007172},{"type":"v1","addr":"172.21.15.119:6813","nonce":1611007172}]},"public_addr":"172.21.15.119:6809/1611007172","cluster_addr":"172.21.15.119:6811/1611007172","heartbeat_back_addr":"172.21.15.119:6815/1611007172","heartbeat_front_addr":"172.21.15.119:6813/1611007172","state":["exists","up"]},{"osd":5,"uuid":"f7590e1a-e93c-4c29-b15f-38ee52d1f921","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":39,"up_thru":55,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6808","nonce":1427563312},{"type":"v1","addr":"172.21.15.204:6809","nonce":1427563312}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6810","nonce":1427563312},{"type":"v1","addr":"172.21.15.204:6811","nonce":1427563312}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6814","nonce":1427563312},{"type":"v1","addr":"172.21.15.204:6815","nonce":1427563312}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6812","nonce":1427563312},{"type":"v1","addr":"172.21.15.204:6813","nonce":1427563312}]},"public_addr":"172.21.15.204:6809/1427563312","cluster_addr":"172.21.15.204:6811/1427563312","heartbeat_back_addr":"172.21.15.204:6815/1427563312","heartbeat_front_addr":"172.21.15.204:6813/1427563312","state":["exists","up"]},{"osd":6,"uuid":"534d20e8-3e54-4db8-a845-6119f49c7a78","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":47,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6818","nonce":4072912544},{"type":"v1","addr":"172.21.15.114:6819","nonce":4072912544}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6820","nonce":4072912544},{"type":"v1","addr":"172.21.15.114:6821","nonce":4072912544}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6824","nonce":4072912544},{"type":"v1","addr":"172.21.15.114:6825","nonce":4072912544}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6822","nonce":4072912544},{"type":"v1","addr":"172.21.15.114:6823","nonce":4072912544}]},"public_addr":"172.21.15.114:6819/4072912544","cluster_addr":"172.21.15.114:6821/4072912544","heartbeat_back_addr":"172.21.15.114:6825/4072912544","heartbeat_front_addr":"172.21.15.114:6823/4072912544","state":["exists","up"]},{"osd":7,"uuid":"f0ec22b8-1b48-4859-9d65-b00403aff3ad","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":53,"up_thru":77,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6816","nonce":2237458793},{"type":"v1","addr":"172.21.15.119:6817","nonce":2237458793}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6818","nonce":2237458793},{"type":"v1","addr":"172.21.15.119:6819","nonce":2237458793}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6822","nonce":2237458793},{"type":"v1","addr":"172.21.15.119:6823","nonce":2237458793}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6820","nonce":2237458793},{"type":"v1","addr":"172.21.15.119:6821","nonce":2237458793}]},"public_addr":"172.21.15.119:6817/2237458793","cluster_addr":"172.21.15.119:6819/2237458793","heartbeat_back_addr":"172.21.15.119:6823/2237458793","heartbeat_front_addr":"172.21.15.119:6821/2237458793","state":["exists","up"]},{"osd":8,"uuid":"3ac89686-6d76-4cef-9089-ac29e4b3bcac","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":62,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6816","nonce":1426318231},{"type":"v1","addr":"172.21.15.204:6817","nonce":1426318231}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6818","nonce":1426318231},{"type":"v1","addr":"172.21.15.204:6819","nonce":1426318231}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6822","nonce":1426318231},{"type":"v1","addr":"172.21.15.204:6823","nonce":1426318231}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6820","nonce":1426318231},{"type":"v1","addr":"172.21.15.204:6821","nonce":1426318231}]},"public_addr":"172.21.15.204:6817/1426318231","cluster_addr":"172.21.15.204:6819/1426318231","heartbeat_back_addr":"172.21.15.204:6823/1426318231","heartbeat_front_addr":"172.21.15.204:6821/1426318231","state":["exists","up"]},{"osd":9,"uuid":"a80ffe4d-1e98-460f-b046-c4754285cc5a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":69,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6826","nonce":2750635721},{"type":"v1","addr":"172.21.15.114:6827","nonce":2750635721}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6828","nonce":2750635721},{"type":"v1","addr":"172.21.15.114:6829","nonce":2750635721}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6832","nonce":2750635721},{"type":"v1","addr":"172.21.15.114:6833","nonce":2750635721}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6830","nonce":2750635721},{"type":"v1","addr":"172.21.15.114:6831","nonce":2750635721}]},"public_addr":"172.21.15.114:6827/2750635721","cluster_addr":"172.21.15.114:6829/2750635721","heartbeat_back_addr":"172.21.15.114:6833/2750635721","heartbeat_front_addr":"172.21.15.114:6831/2750635721","state":["exists","up"]},{"osd":10,"uuid":"cd75a579-bf12-4600-ac09-a8bfce772ced","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":74,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6824","nonce":1400187701},{"type":"v1","addr":"172.21.15.119:6825","nonce":1400187701}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6826","nonce":1400187701},{"type":"v1","addr":"172.21.15.119:6827","nonce":1400187701}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6830","nonce":1400187701},{"type":"v1","addr":"172.21.15.119:6831","nonce":1400187701}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6828","nonce":1400187701},{"type":"v1","addr":"172.21.15.119:6829","nonce":1400187701}]},"public_addr":"172.21.15.119:6825/1400187701","cluster_addr":"172.21.15.119:6827/1400187701","heartbeat_back_addr":"172.21.15.119:6831/1400187701","heartbeat_front_addr":"172.21.15.119:6829/1400187701","state":["exists","up"]},{"osd":11,"uuid":"53f037a5-3ec0-483a-80a1-dfee923a68e5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":82,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6824","nonce":2407298490},{"type":"v1","addr":"172.21.15.204:6825","nonce":2407298490}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6826","nonce":2407298490},{"type":"v1","addr":"172.21.15.204:6827","nonce":2407298490}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6830","nonce":2407298490},{"type":"v1","addr":"172.21.15.204:6831","nonce":2407298490}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6828","nonce":2407298490},{"type":"v1","addr":"172.21.15.204:6829","nonce":2407298490}]},"public_addr":"172.21.15.204:6825/2407298490","cluster_addr":"172.21.15.204:6827/2407298490","heartbeat_back_addr":"172.21.15.204:6831/2407298490","heartbeat_front_addr":"172.21.15.204:6829/2407298490","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-23T10:19:18.401147+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-23T10:19:42.611299+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-23T10:20:07.277367+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-23T10:20:31.208368+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-23T10:20:55.866225+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-23T10:21:19.961237+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-23T10:21:44.480783+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-23T10:22:09.173641+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-23T10:22:34.030328+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-23T10:22:59.652293+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-23T10:23:25.659673+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-23T10:23:51.168171+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.114:6801/1604678557":"2024-04-24T10:17:45.874609+0000","172.21.15.114:0/4025385854":"2024-04-24T10:17:45.874609+0000","172.21.15.114:0/2228273469":"2024-04-24T10:17:19.910348+0000","172.21.15.114:6800/2831530838":"2024-04-24T10:17:19.910348+0000","172.21.15.114:0/1333447450":"2024-04-24T10:17:45.874609+0000","172.21.15.114:0/2233026840":"2024-04-24T10:17:19.910348+0000","172.21.15.114:0/1902281545":"2024-04-24T10:17:19.910348+0000","172.21.15.114:6801/2831530838":"2024-04-24T10:17:19.910348+0000","172.21.15.114:6800/1604678557":"2024-04-24T10:17:45.874609+0000","172.21.15.114:0/1623553811":"2024-04-24T10:17:45.874609+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-23T10:23:59.676 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:23:59.677+0000 7f2e240dd700 1 -- 172.21.15.114:0/381236297 >> [v2:172.21.15.114:6800/465333808,v1:172.21.15.114:6801/465333808] conn(0x7f2e080a3c40 msgr2=0x7f2e080a6100 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T10:23:59.676 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:23:59.677+0000 7f2e240dd700 1 --2- 172.21.15.114:0/381236297 >> [v2:172.21.15.114:6800/465333808,v1:172.21.15.114:6801/465333808] conn(0x7f2e080a3c40 0x7f2e080a6100 secure :-1 s=READY pgs=127 cs=0 l=1 rev1=1 crypto rx=0x7f2e0c005fd0 tx=0x7f2e0c005e20 comp rx=0 tx=0).stop 2024-04-23T10:23:59.676 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:23:59.677+0000 7f2e240dd700 1 -- 172.21.15.114:0/381236297 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f2e1c101400 msgr2=0x7f2e1c1025e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T10:23:59.676 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:23:59.677+0000 7f2e240dd700 1 --2- 172.21.15.114:0/381236297 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f2e1c101400 0x7f2e1c1025e0 secure :-1 s=READY pgs=154 cs=0 l=1 rev1=1 crypto rx=0x7f2e1800d980 tx=0x7f2e1800dd40 comp rx=0 tx=0).stop 2024-04-23T10:23:59.677 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:23:59.677+0000 7f2e240dd700 1 -- 172.21.15.114:0/381236297 shutdown_connections 2024-04-23T10:23:59.677 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:23:59.677+0000 7f2e240dd700 1 --2- 172.21.15.114:0/381236297 >> [v2:172.21.15.114:6800/465333808,v1:172.21.15.114:6801/465333808] conn(0x7f2e080a3c40 0x7f2e080a6100 unknown :-1 s=CLOSED pgs=127 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:23:59.677 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:23:59.677+0000 7f2e240dd700 1 --2- 172.21.15.114:0/381236297 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f2e1c107b20 0x7f2e1c103060 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:23:59.677 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:23:59.677+0000 7f2e240dd700 1 --2- 172.21.15.114:0/381236297 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f2e1c101400 0x7f2e1c1025e0 unknown :-1 s=CLOSED pgs=154 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:23:59.677 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:23:59.677+0000 7f2e240dd700 1 --2- 172.21.15.114:0/381236297 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f2e1c101dc0 0x7f2e1c102b20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:23:59.677 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:23:59.677+0000 7f2e240dd700 1 -- 172.21.15.114:0/381236297 >> 172.21.15.114:0/381236297 conn(0x7f2e1c076c60 msgr2=0x7f2e1c0ffe80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T10:23:59.677 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:23:59.677+0000 7f2e240dd700 1 -- 172.21.15.114:0/381236297 shutdown_connections 2024-04-23T10:23:59.677 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:23:59.677+0000 7f2e240dd700 1 -- 172.21.15.114:0/381236297 wait complete. 2024-04-23T10:23:59.992 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 23 10:23:59 smithi119 ceph-mon[32608]: from='client.? 172.21.15.114:0/381236297' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-23T10:24:00.004 INFO:journalctl@ceph.mon.a.smithi114.stdout:Apr 23 10:23:59 smithi114 ceph-mon[29255]: from='client.? 172.21.15.114:0/381236297' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-23T10:24:00.078 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 23 10:23:59 smithi204 ceph-mon[32649]: from='client.? 172.21.15.114:0/381236297' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-23T10:24:00.307 INFO:tasks.cephadm.ceph_manager.ceph:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-04-23T10:20:11.970297+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '22', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 12, 'score_stable': 12, 'optimal_score': 0.25, 'raw_score_acting': 3, 'raw_score_stable': 3, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-04-23T10:24:00.308 DEBUG:teuthology.orchestra.run.smithi114:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d69f13d827952bb07bb9b7973bca14f97d764d6f shell --fsid aa240572-0159-11ef-bc93-c7b262605968 -- ceph osd pool get .mgr pg_num 2024-04-23T10:24:00.543 INFO:teuthology.orchestra.run.smithi114.stderr:Inferring config /var/lib/ceph/aa240572-0159-11ef-bc93-c7b262605968/mon.a/config 2024-04-23T10:24:00.925 INFO:journalctl@ceph.mon.a.smithi114.stdout:Apr 23 10:24:00 smithi114 ceph-mon[29255]: pgmap v260: 1 pgs: 1 active+clean; 577 KiB data, 365 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-23T10:24:00.992 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 23 10:24:00 smithi119 ceph-mon[32608]: pgmap v260: 1 pgs: 1 active+clean; 577 KiB data, 365 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-23T10:24:01.078 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 23 10:24:00 smithi204 ceph-mon[32649]: pgmap v260: 1 pgs: 1 active+clean; 577 KiB data, 365 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-23T10:24:02.087 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:24:02.086+0000 7ff11b66d700 1 -- 172.21.15.114:0/1148696884 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7ff114102300 msgr2=0x7ff11410e7f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T10:24:02.087 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:24:02.086+0000 7ff11b66d700 1 --2- 172.21.15.114:0/1148696884 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7ff114102300 0x7ff11410e7f0 secure :-1 s=READY pgs=155 cs=0 l=1 rev1=1 crypto rx=0x7ff10c00b0a0 tx=0x7ff10c00b3b0 comp rx=0 tx=0).stop 2024-04-23T10:24:02.087 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:24:02.086+0000 7ff11b66d700 1 -- 172.21.15.114:0/1148696884 shutdown_connections 2024-04-23T10:24:02.087 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:24:02.086+0000 7ff11b66d700 1 --2- 172.21.15.114:0/1148696884 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7ff114102300 0x7ff11410e7f0 unknown :-1 s=CLOSED pgs=155 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:24:02.088 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:24:02.086+0000 7ff11b66d700 1 --2- 172.21.15.114:0/1148696884 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7ff114107990 0x7ff114107d70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:24:02.088 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:24:02.086+0000 7ff11b66d700 1 --2- 172.21.15.114:0/1148696884 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7ff114101940 0x7ff114101dc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:24:02.088 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:24:02.086+0000 7ff11b66d700 1 -- 172.21.15.114:0/1148696884 >> 172.21.15.114:0/1148696884 conn(0x7ff1140788f0 msgr2=0x7ff114078d00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T10:24:02.088 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:24:02.086+0000 7ff11b66d700 1 -- 172.21.15.114:0/1148696884 shutdown_connections 2024-04-23T10:24:02.088 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:24:02.086+0000 7ff11b66d700 1 -- 172.21.15.114:0/1148696884 wait complete. 2024-04-23T10:24:02.088 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:24:02.087+0000 7ff11b66d700 1 Processor -- start 2024-04-23T10:24:02.088 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:24:02.087+0000 7ff11b66d700 1 -- start start 2024-04-23T10:24:02.088 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:24:02.087+0000 7ff11b66d700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7ff114101940 0x7ff114199e70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T10:24:02.088 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:24:02.087+0000 7ff11b66d700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7ff114102300 0x7ff11419a3b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T10:24:02.088 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:24:02.087+0000 7ff11b66d700 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7ff114107990 0x7ff1141a4ef0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T10:24:02.088 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:24:02.087+0000 7ff11b66d700 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x7ff114111b50 con 0x7ff114102300 2024-04-23T10:24:02.088 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:24:02.087+0000 7ff11b66d700 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 -- 0x7ff1141119d0 con 0x7ff114101940 2024-04-23T10:24:02.088 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:24:02.087+0000 7ff11b66d700 1 -- --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_getmap magic: 0 -- 0x7ff114111cd0 con 0x7ff114107990 2024-04-23T10:24:02.088 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:24:02.087+0000 7ff118c08700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7ff114102300 0x7ff11419a3b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T10:24:02.088 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:24:02.087+0000 7ff118c08700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7ff114102300 0x7ff11419a3b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.114:36132/0 (socket says 172.21.15.114:36132) 2024-04-23T10:24:02.089 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:24:02.087+0000 7ff118c08700 1 -- 172.21.15.114:0/4021022996 learned_addr learned my addr 172.21.15.114:0/4021022996 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-04-23T10:24:02.089 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:24:02.087+0000 7ff119409700 1 --2- 172.21.15.114:0/4021022996 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7ff114101940 0x7ff114199e70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T10:24:02.089 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:24:02.087+0000 7ff119c0a700 1 --2- 172.21.15.114:0/4021022996 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7ff114107990 0x7ff1141a4ef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T10:24:02.089 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:24:02.087+0000 7ff118c08700 1 -- 172.21.15.114:0/4021022996 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7ff114107990 msgr2=0x7ff1141a4ef0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T10:24:02.089 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:24:02.087+0000 7ff118c08700 1 --2- 172.21.15.114:0/4021022996 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7ff114107990 0x7ff1141a4ef0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:24:02.089 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:24:02.087+0000 7ff118c08700 1 -- 172.21.15.114:0/4021022996 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7ff114101940 msgr2=0x7ff114199e70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T10:24:02.089 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:24:02.087+0000 7ff118c08700 1 --2- 172.21.15.114:0/4021022996 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7ff114101940 0x7ff114199e70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:24:02.089 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:24:02.087+0000 7ff118c08700 1 -- 172.21.15.114:0/4021022996 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff1141a5430 con 0x7ff114102300 2024-04-23T10:24:02.089 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:24:02.087+0000 7ff119c0a700 1 --2- 172.21.15.114:0/4021022996 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7ff114107990 0x7ff1141a4ef0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-23T10:24:02.089 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:24:02.088+0000 7ff118c08700 1 --2- 172.21.15.114:0/4021022996 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7ff114102300 0x7ff11419a3b0 secure :-1 s=READY pgs=156 cs=0 l=1 rev1=1 crypto rx=0x7ff10400b530 tx=0x7ff10400b840 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T10:24:02.089 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:24:02.088+0000 7ff10a7fc700 1 -- 172.21.15.114:0/4021022996 <== mon.0 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff104015020 con 0x7ff114102300 2024-04-23T10:24:02.089 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:24:02.088+0000 7ff10a7fc700 1 -- 172.21.15.114:0/4021022996 <== mon.0 v2:172.21.15.114:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7ff104011ce0 con 0x7ff114102300 2024-04-23T10:24:02.089 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:24:02.088+0000 7ff11b66d700 1 -- 172.21.15.114:0/4021022996 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7ff1141a56c0 con 0x7ff114102300 2024-04-23T10:24:02.089 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:24:02.088+0000 7ff10a7fc700 1 -- 172.21.15.114:0/4021022996 <== mon.0 v2:172.21.15.114:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff104010e40 con 0x7ff114102300 2024-04-23T10:24:02.089 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:24:02.088+0000 7ff11b66d700 1 -- 172.21.15.114:0/4021022996 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7ff1141a5bd0 con 0x7ff114102300 2024-04-23T10:24:02.090 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:24:02.089+0000 7ff11b66d700 1 -- 172.21.15.114:0/4021022996 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7ff114069730 con 0x7ff114102300 2024-04-23T10:24:02.090 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:24:02.090+0000 7ff10a7fc700 1 -- 172.21.15.114:0/4021022996 <== mon.0 v2:172.21.15.114:3300/0 4 ==== mgrmap(e 16) ==== 138647+0+0 (secure 0 0 0) 0x7ff104018020 con 0x7ff114102300 2024-04-23T10:24:02.091 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:24:02.091+0000 7ff10a7fc700 1 --2- 172.21.15.114:0/4021022996 >> [v2:172.21.15.114:6800/465333808,v1:172.21.15.114:6801/465333808] conn(0x7ff1000a3c40 0x7ff1000a6100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T10:24:02.091 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:24:02.091+0000 7ff119409700 1 --2- 172.21.15.114:0/4021022996 >> [v2:172.21.15.114:6800/465333808,v1:172.21.15.114:6801/465333808] conn(0x7ff1000a3c40 0x7ff1000a6100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T10:24:02.091 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:24:02.091+0000 7ff10a7fc700 1 -- 172.21.15.114:0/4021022996 <== mon.0 v2:172.21.15.114:3300/0 5 ==== osd_map(83..83 src has 1..83) ==== 7127+0+0 (secure 0 0 0) 0x7ff1040cfd50 con 0x7ff114102300 2024-04-23T10:24:02.091 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:24:02.091+0000 7ff119409700 1 --2- 172.21.15.114:0/4021022996 >> [v2:172.21.15.114:6800/465333808,v1:172.21.15.114:6801/465333808] conn(0x7ff1000a3c40 0x7ff1000a6100 secure :-1 s=READY pgs=128 cs=0 l=1 rev1=1 crypto rx=0x7ff11000ac50 tx=0x7ff11000a380 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T10:24:02.094 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:24:02.095+0000 7ff10a7fc700 1 -- 172.21.15.114:0/4021022996 <== mon.0 v2:172.21.15.114:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7ff10401f020 con 0x7ff114102300 2024-04-23T10:24:02.291 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:24:02.292+0000 7ff11b66d700 1 -- 172.21.15.114:0/4021022996 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) -- 0x7ff11402df80 con 0x7ff114102300 2024-04-23T10:24:02.291 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:24:02.292+0000 7ff10a7fc700 1 -- 172.21.15.114:0/4021022996 <== mon.0 v2:172.21.15.114:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v83) ==== 93+0+10 (secure 0 0 0) 0x7ff10409cb30 con 0x7ff114102300 2024-04-23T10:24:02.291 INFO:teuthology.orchestra.run.smithi114.stdout:pg_num: 1 2024-04-23T10:24:02.294 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:24:02.294+0000 7ff11b66d700 1 -- 172.21.15.114:0/4021022996 >> [v2:172.21.15.114:6800/465333808,v1:172.21.15.114:6801/465333808] conn(0x7ff1000a3c40 msgr2=0x7ff1000a6100 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T10:24:02.294 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:24:02.294+0000 7ff11b66d700 1 --2- 172.21.15.114:0/4021022996 >> [v2:172.21.15.114:6800/465333808,v1:172.21.15.114:6801/465333808] conn(0x7ff1000a3c40 0x7ff1000a6100 secure :-1 s=READY pgs=128 cs=0 l=1 rev1=1 crypto rx=0x7ff11000ac50 tx=0x7ff11000a380 comp rx=0 tx=0).stop 2024-04-23T10:24:02.294 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:24:02.294+0000 7ff11b66d700 1 -- 172.21.15.114:0/4021022996 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7ff114102300 msgr2=0x7ff11419a3b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T10:24:02.294 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:24:02.294+0000 7ff11b66d700 1 --2- 172.21.15.114:0/4021022996 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7ff114102300 0x7ff11419a3b0 secure :-1 s=READY pgs=156 cs=0 l=1 rev1=1 crypto rx=0x7ff10400b530 tx=0x7ff10400b840 comp rx=0 tx=0).stop 2024-04-23T10:24:02.294 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:24:02.294+0000 7ff11b66d700 1 -- 172.21.15.114:0/4021022996 shutdown_connections 2024-04-23T10:24:02.294 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:24:02.294+0000 7ff11b66d700 1 --2- 172.21.15.114:0/4021022996 >> [v2:172.21.15.114:6800/465333808,v1:172.21.15.114:6801/465333808] conn(0x7ff1000a3c40 0x7ff1000a6100 unknown :-1 s=CLOSED pgs=128 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:24:02.294 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:24:02.294+0000 7ff11b66d700 1 --2- 172.21.15.114:0/4021022996 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7ff114101940 0x7ff114199e70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:24:02.294 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:24:02.294+0000 7ff11b66d700 1 --2- 172.21.15.114:0/4021022996 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7ff114107990 0x7ff1141a4ef0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:24:02.294 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:24:02.294+0000 7ff11b66d700 1 --2- 172.21.15.114:0/4021022996 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7ff114102300 0x7ff11419a3b0 unknown :-1 s=CLOSED pgs=156 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:24:02.294 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:24:02.295+0000 7ff11b66d700 1 -- 172.21.15.114:0/4021022996 >> 172.21.15.114:0/4021022996 conn(0x7ff1140788f0 msgr2=0x7ff1140ff950 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T10:24:02.294 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:24:02.295+0000 7ff11b66d700 1 -- 172.21.15.114:0/4021022996 shutdown_connections 2024-04-23T10:24:02.294 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:24:02.295+0000 7ff11b66d700 1 -- 172.21.15.114:0/4021022996 wait complete. 2024-04-23T10:24:02.848 INFO:tasks.cephadm:Adding mds.a on smithi114 2024-04-23T10:24:02.848 INFO:tasks.cephadm:Adding mds.d on smithi114 2024-04-23T10:24:02.848 INFO:tasks.cephadm:Adding mds.g on smithi114 2024-04-23T10:24:02.848 INFO:tasks.cephadm:Adding mds.j on smithi114 2024-04-23T10:24:02.848 INFO:tasks.cephadm:Adding mds.b on smithi119 2024-04-23T10:24:02.848 INFO:tasks.cephadm:Adding mds.e on smithi119 2024-04-23T10:24:02.848 INFO:tasks.cephadm:Adding mds.h on smithi119 2024-04-23T10:24:02.848 INFO:tasks.cephadm:Adding mds.k on smithi119 2024-04-23T10:24:02.848 INFO:tasks.cephadm:Adding mds.c on smithi204 2024-04-23T10:24:02.848 INFO:tasks.cephadm:Adding mds.f on smithi204 2024-04-23T10:24:02.848 INFO:tasks.cephadm:Adding mds.i on smithi204 2024-04-23T10:24:02.848 INFO:tasks.cephadm:Adding mds.l on smithi204 2024-04-23T10:24:02.849 DEBUG:teuthology.orchestra.run.smithi204:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d69f13d827952bb07bb9b7973bca14f97d764d6f shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid aa240572-0159-11ef-bc93-c7b262605968 -- ceph orch apply mds all '12;smithi114=a;smithi114=d;smithi114=g;smithi114=j;smithi119=b;smithi119=e;smithi119=h;smithi119=k;smithi204=c;smithi204=f;smithi204=i;smithi204=l' 2024-04-23T10:24:03.100 INFO:teuthology.orchestra.run.smithi204.stderr:Inferring config /var/lib/ceph/aa240572-0159-11ef-bc93-c7b262605968/mon.c/config 2024-04-23T10:24:03.221 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 23 10:24:03 smithi204 ceph-mon[32649]: pgmap v261: 1 pgs: 1 active+clean; 577 KiB data, 365 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-23T10:24:03.221 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 23 10:24:03 smithi204 ceph-mon[32649]: from='client.? 172.21.15.114:0/4021022996' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-04-23T10:24:03.254 INFO:journalctl@ceph.mon.a.smithi114.stdout:Apr 23 10:24:03 smithi114 ceph-mon[29255]: pgmap v261: 1 pgs: 1 active+clean; 577 KiB data, 365 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-23T10:24:03.254 INFO:journalctl@ceph.mon.a.smithi114.stdout:Apr 23 10:24:03 smithi114 ceph-mon[29255]: from='client.? 172.21.15.114:0/4021022996' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-04-23T10:24:03.492 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 23 10:24:03 smithi119 ceph-mon[32608]: pgmap v261: 1 pgs: 1 active+clean; 577 KiB data, 365 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-23T10:24:03.492 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 23 10:24:03 smithi119 ceph-mon[32608]: from='client.? 172.21.15.114:0/4021022996' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-04-23T10:24:04.670 INFO:teuthology.orchestra.run.smithi204.stderr:2024-04-23T10:24:04.669+0000 7f9540482700 1 -- 172.21.15.204:0/2417278740 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f9538103ca0 msgr2=0x7f9538104100 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T10:24:04.670 INFO:teuthology.orchestra.run.smithi204.stderr:2024-04-23T10:24:04.669+0000 7f9540482700 1 --2- 172.21.15.204:0/2417278740 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f9538103ca0 0x7f9538104100 secure :-1 s=READY pgs=46 cs=0 l=1 rev1=1 crypto rx=0x7f9528009990 tx=0x7f9528009ca0 comp rx=0 tx=0).stop 2024-04-23T10:24:04.670 INFO:teuthology.orchestra.run.smithi204.stderr:2024-04-23T10:24:04.669+0000 7f9540482700 1 -- 172.21.15.204:0/2417278740 shutdown_connections 2024-04-23T10:24:04.670 INFO:teuthology.orchestra.run.smithi204.stderr:2024-04-23T10:24:04.669+0000 7f9540482700 1 --2- 172.21.15.204:0/2417278740 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f9538104640 0x7f953810aef0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:24:04.670 INFO:teuthology.orchestra.run.smithi204.stderr:2024-04-23T10:24:04.669+0000 7f9540482700 1 --2- 172.21.15.204:0/2417278740 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f9538102aa0 0x7f9538102ec0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:24:04.670 INFO:teuthology.orchestra.run.smithi204.stderr:2024-04-23T10:24:04.669+0000 7f9540482700 1 --2- 172.21.15.204:0/2417278740 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f9538103ca0 0x7f9538104100 unknown :-1 s=CLOSED pgs=46 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:24:04.670 INFO:teuthology.orchestra.run.smithi204.stderr:2024-04-23T10:24:04.669+0000 7f9540482700 1 -- 172.21.15.204:0/2417278740 >> 172.21.15.204:0/2417278740 conn(0x7f95380fe040 msgr2=0x7f9538100480 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T10:24:04.670 INFO:teuthology.orchestra.run.smithi204.stderr:2024-04-23T10:24:04.669+0000 7f9540482700 1 -- 172.21.15.204:0/2417278740 shutdown_connections 2024-04-23T10:24:04.671 INFO:teuthology.orchestra.run.smithi204.stderr:2024-04-23T10:24:04.669+0000 7f9540482700 1 -- 172.21.15.204:0/2417278740 wait complete. 2024-04-23T10:24:04.671 INFO:teuthology.orchestra.run.smithi204.stderr:2024-04-23T10:24:04.669+0000 7f9540482700 1 Processor -- start 2024-04-23T10:24:04.671 INFO:teuthology.orchestra.run.smithi204.stderr:2024-04-23T10:24:04.670+0000 7f9540482700 1 -- start start 2024-04-23T10:24:04.671 INFO:teuthology.orchestra.run.smithi204.stderr:2024-04-23T10:24:04.670+0000 7f9540482700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f9538102aa0 0x7f9538198c70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T10:24:04.671 INFO:teuthology.orchestra.run.smithi204.stderr:2024-04-23T10:24:04.670+0000 7f9540482700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f9538103ca0 0x7f95381991b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T10:24:04.671 INFO:teuthology.orchestra.run.smithi204.stderr:2024-04-23T10:24:04.670+0000 7f9540482700 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f9538104640 0x7f95381a0220 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T10:24:04.671 INFO:teuthology.orchestra.run.smithi204.stderr:2024-04-23T10:24:04.670+0000 7f9540482700 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x7f953810e5d0 con 0x7f9538103ca0 2024-04-23T10:24:04.671 INFO:teuthology.orchestra.run.smithi204.stderr:2024-04-23T10:24:04.670+0000 7f9540482700 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 -- 0x7f953810e450 con 0x7f9538102aa0 2024-04-23T10:24:04.671 INFO:teuthology.orchestra.run.smithi204.stderr:2024-04-23T10:24:04.670+0000 7f9540482700 1 -- --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_getmap magic: 0 -- 0x7f953810e750 con 0x7f9538104640 2024-04-23T10:24:04.671 INFO:teuthology.orchestra.run.smithi204.stderr:2024-04-23T10:24:04.670+0000 7f953ea1f700 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f9538104640 0x7f95381a0220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T10:24:04.671 INFO:teuthology.orchestra.run.smithi204.stderr:2024-04-23T10:24:04.670+0000 7f953ea1f700 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f9538104640 0x7f95381a0220 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.204:3300/0 says I am v2:172.21.15.204:37122/0 (socket says 172.21.15.204:37122) 2024-04-23T10:24:04.671 INFO:teuthology.orchestra.run.smithi204.stderr:2024-04-23T10:24:04.670+0000 7f953ea1f700 1 -- 172.21.15.204:0/3424011162 learned_addr learned my addr 172.21.15.204:0/3424011162 (peer_addr_for_me v2:172.21.15.204:0/0) 2024-04-23T10:24:04.671 INFO:teuthology.orchestra.run.smithi204.stderr:2024-04-23T10:24:04.670+0000 7f953e21e700 1 --2- 172.21.15.204:0/3424011162 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f9538102aa0 0x7f9538198c70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T10:24:04.671 INFO:teuthology.orchestra.run.smithi204.stderr:2024-04-23T10:24:04.670+0000 7f953da1d700 1 --2- 172.21.15.204:0/3424011162 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f9538103ca0 0x7f95381991b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T10:24:04.672 INFO:teuthology.orchestra.run.smithi204.stderr:2024-04-23T10:24:04.670+0000 7f953ea1f700 1 -- 172.21.15.204:0/3424011162 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f9538102aa0 msgr2=0x7f9538198c70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T10:24:04.672 INFO:teuthology.orchestra.run.smithi204.stderr:2024-04-23T10:24:04.670+0000 7f953ea1f700 1 --2- 172.21.15.204:0/3424011162 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f9538102aa0 0x7f9538198c70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:24:04.672 INFO:teuthology.orchestra.run.smithi204.stderr:2024-04-23T10:24:04.670+0000 7f953ea1f700 1 -- 172.21.15.204:0/3424011162 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f9538103ca0 msgr2=0x7f95381991b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T10:24:04.672 INFO:teuthology.orchestra.run.smithi204.stderr:2024-04-23T10:24:04.670+0000 7f953ea1f700 1 --2- 172.21.15.204:0/3424011162 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f9538103ca0 0x7f95381991b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:24:04.672 INFO:teuthology.orchestra.run.smithi204.stderr:2024-04-23T10:24:04.670+0000 7f953ea1f700 1 -- 172.21.15.204:0/3424011162 --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f95381a08f0 con 0x7f9538104640 2024-04-23T10:24:04.672 INFO:teuthology.orchestra.run.smithi204.stderr:2024-04-23T10:24:04.670+0000 7f953da1d700 1 --2- 172.21.15.204:0/3424011162 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f9538103ca0 0x7f95381991b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-23T10:24:04.672 INFO:teuthology.orchestra.run.smithi204.stderr:2024-04-23T10:24:04.670+0000 7f953ea1f700 1 --2- 172.21.15.204:0/3424011162 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f9538104640 0x7f95381a0220 secure :-1 s=READY pgs=47 cs=0 l=1 rev1=1 crypto rx=0x7f9530007ee0 tx=0x7f953000d3b0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T10:24:04.672 INFO:teuthology.orchestra.run.smithi204.stderr:2024-04-23T10:24:04.671+0000 7f952f7fe700 1 -- 172.21.15.204:0/3424011162 <== mon.1 v2:172.21.15.204:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f953000dbd0 con 0x7f9538104640 2024-04-23T10:24:04.672 INFO:teuthology.orchestra.run.smithi204.stderr:2024-04-23T10:24:04.671+0000 7f952f7fe700 1 -- 172.21.15.204:0/3424011162 <== mon.1 v2:172.21.15.204:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f953000dd70 con 0x7f9538104640 2024-04-23T10:24:04.672 INFO:teuthology.orchestra.run.smithi204.stderr:2024-04-23T10:24:04.671+0000 7f9540482700 1 -- 172.21.15.204:0/3424011162 --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f95381a0be0 con 0x7f9538104640 2024-04-23T10:24:04.672 INFO:teuthology.orchestra.run.smithi204.stderr:2024-04-23T10:24:04.671+0000 7f952f7fe700 1 -- 172.21.15.204:0/3424011162 <== mon.1 v2:172.21.15.204:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9530005990 con 0x7f9538104640 2024-04-23T10:24:04.672 INFO:teuthology.orchestra.run.smithi204.stderr:2024-04-23T10:24:04.671+0000 7f9540482700 1 -- 172.21.15.204:0/3424011162 --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f95381a1170 con 0x7f9538104640 2024-04-23T10:24:04.672 INFO:teuthology.orchestra.run.smithi204.stderr:2024-04-23T10:24:04.671+0000 7f9540482700 1 -- 172.21.15.204:0/3424011162 --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f9538069730 con 0x7f9538104640 2024-04-23T10:24:04.673 INFO:teuthology.orchestra.run.smithi204.stderr:2024-04-23T10:24:04.673+0000 7f952f7fe700 1 -- 172.21.15.204:0/3424011162 <== mon.1 v2:172.21.15.204:3300/0 4 ==== mgrmap(e 16) ==== 138647+0+0 (secure 0 0 0) 0x7f953001a020 con 0x7f9538104640 2024-04-23T10:24:04.673 INFO:teuthology.orchestra.run.smithi204.stderr:2024-04-23T10:24:04.674+0000 7f952f7fe700 1 --2- 172.21.15.204:0/3424011162 >> [v2:172.21.15.114:6800/465333808,v1:172.21.15.114:6801/465333808] conn(0x7f95240a3cc0 0x7f95240a6180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T10:24:04.673 INFO:teuthology.orchestra.run.smithi204.stderr:2024-04-23T10:24:04.674+0000 7f952f7fe700 1 -- 172.21.15.204:0/3424011162 <== mon.1 v2:172.21.15.204:3300/0 5 ==== osd_map(83..83 src has 1..83) ==== 7127+0+0 (secure 0 0 0) 0x7f95300cfc80 con 0x7f9538104640 2024-04-23T10:24:04.673 INFO:teuthology.orchestra.run.smithi204.stderr:2024-04-23T10:24:04.674+0000 7f953e21e700 1 --2- 172.21.15.204:0/3424011162 >> [v2:172.21.15.114:6800/465333808,v1:172.21.15.114:6801/465333808] conn(0x7f95240a3cc0 0x7f95240a6180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T10:24:04.673 INFO:teuthology.orchestra.run.smithi204.stderr:2024-04-23T10:24:04.675+0000 7f953e21e700 1 --2- 172.21.15.204:0/3424011162 >> [v2:172.21.15.114:6800/465333808,v1:172.21.15.114:6801/465333808] conn(0x7f95240a3cc0 0x7f95240a6180 secure :-1 s=READY pgs=129 cs=0 l=1 rev1=1 crypto rx=0x7f9538199b50 tx=0x7f9534009190 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T10:24:04.676 INFO:teuthology.orchestra.run.smithi204.stderr:2024-04-23T10:24:04.678+0000 7f952f7fe700 1 -- 172.21.15.204:0/3424011162 <== mon.1 v2:172.21.15.204:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f9530010070 con 0x7f9538104640 2024-04-23T10:24:04.889 INFO:teuthology.orchestra.run.smithi204.stderr:2024-04-23T10:24:04.891+0000 7f9540482700 1 -- 172.21.15.204:0/3424011162 --> [v2:172.21.15.114:6800/465333808,v1:172.21.15.114:6801/465333808] -- mgr_command(tid 0: {"prefix": "orch apply mds", "fs_name": "all", "placement": "12;smithi114=a;smithi114=d;smithi114=g;smithi114=j;smithi119=b;smithi119=e;smithi119=h;smithi119=k;smithi204=c;smithi204=f;smithi204=i;smithi204=l", "target": ["mon-mgr", ""]}) -- 0x7f9538063140 con 0x7f95240a3cc0 2024-04-23T10:24:04.897 INFO:teuthology.orchestra.run.smithi204.stderr:2024-04-23T10:24:04.899+0000 7f952f7fe700 1 -- 172.21.15.204:0/3424011162 <== mgr.14152 v2:172.21.15.114:6800/465333808 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+28 (secure 0 0 0) 0x7f9538063140 con 0x7f95240a3cc0 2024-04-23T10:24:04.897 INFO:teuthology.orchestra.run.smithi204.stdout:Scheduled mds.all update... 2024-04-23T10:24:04.900 INFO:teuthology.orchestra.run.smithi204.stderr:2024-04-23T10:24:04.901+0000 7f9540482700 1 -- 172.21.15.204:0/3424011162 >> [v2:172.21.15.114:6800/465333808,v1:172.21.15.114:6801/465333808] conn(0x7f95240a3cc0 msgr2=0x7f95240a6180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T10:24:04.900 INFO:teuthology.orchestra.run.smithi204.stderr:2024-04-23T10:24:04.901+0000 7f9540482700 1 --2- 172.21.15.204:0/3424011162 >> [v2:172.21.15.114:6800/465333808,v1:172.21.15.114:6801/465333808] conn(0x7f95240a3cc0 0x7f95240a6180 secure :-1 s=READY pgs=129 cs=0 l=1 rev1=1 crypto rx=0x7f9538199b50 tx=0x7f9534009190 comp rx=0 tx=0).stop 2024-04-23T10:24:04.900 INFO:teuthology.orchestra.run.smithi204.stderr:2024-04-23T10:24:04.901+0000 7f9540482700 1 -- 172.21.15.204:0/3424011162 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f9538104640 msgr2=0x7f95381a0220 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T10:24:04.900 INFO:teuthology.orchestra.run.smithi204.stderr:2024-04-23T10:24:04.901+0000 7f9540482700 1 --2- 172.21.15.204:0/3424011162 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f9538104640 0x7f95381a0220 secure :-1 s=READY pgs=47 cs=0 l=1 rev1=1 crypto rx=0x7f9530007ee0 tx=0x7f953000d3b0 comp rx=0 tx=0).stop 2024-04-23T10:24:04.901 INFO:teuthology.orchestra.run.smithi204.stderr:2024-04-23T10:24:04.901+0000 7f9540482700 1 -- 172.21.15.204:0/3424011162 shutdown_connections 2024-04-23T10:24:04.901 INFO:teuthology.orchestra.run.smithi204.stderr:2024-04-23T10:24:04.901+0000 7f9540482700 1 --2- 172.21.15.204:0/3424011162 >> [v2:172.21.15.114:6800/465333808,v1:172.21.15.114:6801/465333808] conn(0x7f95240a3cc0 0x7f95240a6180 unknown :-1 s=CLOSED pgs=129 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:24:04.901 INFO:teuthology.orchestra.run.smithi204.stderr:2024-04-23T10:24:04.901+0000 7f9540482700 1 --2- 172.21.15.204:0/3424011162 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f9538102aa0 0x7f9538198c70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:24:04.901 INFO:teuthology.orchestra.run.smithi204.stderr:2024-04-23T10:24:04.901+0000 7f9540482700 1 --2- 172.21.15.204:0/3424011162 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f9538104640 0x7f95381a0220 unknown :-1 s=CLOSED pgs=47 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:24:04.901 INFO:teuthology.orchestra.run.smithi204.stderr:2024-04-23T10:24:04.901+0000 7f9540482700 1 --2- 172.21.15.204:0/3424011162 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f9538103ca0 0x7f95381991b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:24:04.901 INFO:teuthology.orchestra.run.smithi204.stderr:2024-04-23T10:24:04.901+0000 7f9540482700 1 -- 172.21.15.204:0/3424011162 >> 172.21.15.204:0/3424011162 conn(0x7f95380fe040 msgr2=0x7f95381070c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T10:24:04.901 INFO:teuthology.orchestra.run.smithi204.stderr:2024-04-23T10:24:04.901+0000 7f9540482700 1 -- 172.21.15.204:0/3424011162 shutdown_connections 2024-04-23T10:24:04.901 INFO:teuthology.orchestra.run.smithi204.stderr:2024-04-23T10:24:04.901+0000 7f9540482700 1 -- 172.21.15.204:0/3424011162 wait complete. 2024-04-23T10:24:05.233 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 23 10:24:05 smithi204 ceph-mon[32649]: pgmap v262: 1 pgs: 1 active+clean; 577 KiB data, 365 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-23T10:24:05.233 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 23 10:24:05 smithi204 ceph-mon[32649]: from='mgr.14152 172.21.15.114:0/1529706138' entity='mgr.x' 2024-04-23T10:24:05.233 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 23 10:24:05 smithi204 ceph-mon[32649]: from='mgr.14152 172.21.15.114:0/1529706138' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-23T10:24:05.254 INFO:journalctl@ceph.mon.a.smithi114.stdout:Apr 23 10:24:05 smithi114 ceph-mon[29255]: pgmap v262: 1 pgs: 1 active+clean; 577 KiB data, 365 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-23T10:24:05.254 INFO:journalctl@ceph.mon.a.smithi114.stdout:Apr 23 10:24:05 smithi114 ceph-mon[29255]: from='mgr.14152 172.21.15.114:0/1529706138' entity='mgr.x' 2024-04-23T10:24:05.254 INFO:journalctl@ceph.mon.a.smithi114.stdout:Apr 23 10:24:05 smithi114 ceph-mon[29255]: from='mgr.14152 172.21.15.114:0/1529706138' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-23T10:24:05.492 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 23 10:24:05 smithi119 ceph-mon[32608]: pgmap v262: 1 pgs: 1 active+clean; 577 KiB data, 365 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-23T10:24:05.492 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 23 10:24:05 smithi119 ceph-mon[32608]: from='mgr.14152 172.21.15.114:0/1529706138' entity='mgr.x' 2024-04-23T10:24:05.492 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 23 10:24:05 smithi119 ceph-mon[32608]: from='mgr.14152 172.21.15.114:0/1529706138' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-23T10:24:05.519 DEBUG:teuthology.orchestra.run.smithi114:mds.a> sudo journalctl -f -n 0 -u ceph-aa240572-0159-11ef-bc93-c7b262605968@mds.a.service 2024-04-23T10:24:05.521 DEBUG:teuthology.orchestra.run.smithi114:mds.d> sudo journalctl -f -n 0 -u ceph-aa240572-0159-11ef-bc93-c7b262605968@mds.d.service 2024-04-23T10:24:05.563 DEBUG:teuthology.orchestra.run.smithi114:mds.g> sudo journalctl -f -n 0 -u ceph-aa240572-0159-11ef-bc93-c7b262605968@mds.g.service 2024-04-23T10:24:05.607 DEBUG:teuthology.orchestra.run.smithi114:mds.j> sudo journalctl -f -n 0 -u ceph-aa240572-0159-11ef-bc93-c7b262605968@mds.j.service 2024-04-23T10:24:05.650 DEBUG:teuthology.orchestra.run.smithi119:mds.b> sudo journalctl -f -n 0 -u ceph-aa240572-0159-11ef-bc93-c7b262605968@mds.b.service 2024-04-23T10:24:05.653 DEBUG:teuthology.orchestra.run.smithi119:mds.e> sudo journalctl -f -n 0 -u ceph-aa240572-0159-11ef-bc93-c7b262605968@mds.e.service 2024-04-23T10:24:05.697 DEBUG:teuthology.orchestra.run.smithi119:mds.h> sudo journalctl -f -n 0 -u ceph-aa240572-0159-11ef-bc93-c7b262605968@mds.h.service 2024-04-23T10:24:05.740 DEBUG:teuthology.orchestra.run.smithi119:mds.k> sudo journalctl -f -n 0 -u ceph-aa240572-0159-11ef-bc93-c7b262605968@mds.k.service 2024-04-23T10:24:05.783 DEBUG:teuthology.orchestra.run.smithi204:mds.c> sudo journalctl -f -n 0 -u ceph-aa240572-0159-11ef-bc93-c7b262605968@mds.c.service 2024-04-23T10:24:05.786 DEBUG:teuthology.orchestra.run.smithi204:mds.f> sudo journalctl -f -n 0 -u ceph-aa240572-0159-11ef-bc93-c7b262605968@mds.f.service 2024-04-23T10:24:05.828 DEBUG:teuthology.orchestra.run.smithi204:mds.i> sudo journalctl -f -n 0 -u ceph-aa240572-0159-11ef-bc93-c7b262605968@mds.i.service 2024-04-23T10:24:05.871 DEBUG:teuthology.orchestra.run.smithi204:mds.l> sudo journalctl -f -n 0 -u ceph-aa240572-0159-11ef-bc93-c7b262605968@mds.l.service 2024-04-23T10:24:05.914 INFO:tasks.cephadm:Setting up CephFS filesystem(s)... 2024-04-23T10:24:05.914 DEBUG:teuthology.orchestra.run.smithi114:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-04-23T10:24:06.028 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:24:06.027+0000 7f25bd4ae640 1 -- 172.21.15.114:0/316236323 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f25b8101fc0 msgr2=0x7f25b81023e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T10:24:06.028 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:24:06.027+0000 7f25bd4ae640 1 --2- 172.21.15.114:0/316236323 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f25b8101fc0 0x7f25b81023e0 secure :-1 s=READY pgs=157 cs=0 l=1 rev1=1 crypto rx=0x7f25a8009900 tx=0x7f25a802a410 comp rx=0 tx=0).stop 2024-04-23T10:24:06.029 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:24:06.027+0000 7f25bd4ae640 1 -- 172.21.15.114:0/316236323 shutdown_connections 2024-04-23T10:24:06.029 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:24:06.027+0000 7f25bd4ae640 1 --2- 172.21.15.114:0/316236323 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f25b8103b70 0x7f25b810a400 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:24:06.029 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:24:06.027+0000 7f25bd4ae640 1 --2- 172.21.15.114:0/316236323 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f25b81031b0 0x7f25b8103630 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:24:06.029 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:24:06.027+0000 7f25bd4ae640 1 --2- 172.21.15.114:0/316236323 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f25b8101fc0 0x7f25b81023e0 unknown :-1 s=CLOSED pgs=157 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:24:06.029 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:24:06.027+0000 7f25bd4ae640 1 -- 172.21.15.114:0/316236323 >> 172.21.15.114:0/316236323 conn(0x7f25b80fd550 msgr2=0x7f25b80ff970 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T10:24:06.029 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:24:06.027+0000 7f25bd4ae640 1 -- 172.21.15.114:0/316236323 shutdown_connections 2024-04-23T10:24:06.029 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:24:06.027+0000 7f25bd4ae640 1 -- 172.21.15.114:0/316236323 wait complete. 2024-04-23T10:24:06.029 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:24:06.028+0000 7f25bd4ae640 1 Processor -- start 2024-04-23T10:24:06.029 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:24:06.028+0000 7f25bd4ae640 1 -- start start 2024-04-23T10:24:06.029 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:24:06.028+0000 7f25bd4ae640 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f25b8101fc0 0x7f25b8198280 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T10:24:06.029 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:24:06.028+0000 7f25bd4ae640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f25b81031b0 0x7f25b81987c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T10:24:06.029 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:24:06.028+0000 7f25bd4ae640 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f25b8103b70 0x7f25b819f7e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T10:24:06.029 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:24:06.028+0000 7f25bd4ae640 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x7f25b810db00 con 0x7f25b81031b0 2024-04-23T10:24:06.029 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:24:06.028+0000 7f25bd4ae640 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 -- 0x7f25b810d980 con 0x7f25b8101fc0 2024-04-23T10:24:06.029 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:24:06.028+0000 7f25bd4ae640 1 -- --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_getmap magic: 0 -- 0x7f25b810dc80 con 0x7f25b8103b70 2024-04-23T10:24:06.030 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:24:06.028+0000 7f25affff640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f25b81031b0 0x7f25b81987c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T10:24:06.030 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:24:06.028+0000 7f25affff640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f25b81031b0 0x7f25b81987c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.114:38902/0 (socket says 172.21.15.114:38902) 2024-04-23T10:24:06.030 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:24:06.028+0000 7f25affff640 1 -- 172.21.15.114:0/2473285021 learned_addr learned my addr 172.21.15.114:0/2473285021 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-04-23T10:24:06.030 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:24:06.028+0000 7f25b77fe640 1 --2- 172.21.15.114:0/2473285021 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f25b8103b70 0x7f25b819f7e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T10:24:06.030 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:24:06.028+0000 7f25b6ffd640 1 --2- 172.21.15.114:0/2473285021 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f25b8101fc0 0x7f25b8198280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T10:24:06.030 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:24:06.028+0000 7f25affff640 1 -- 172.21.15.114:0/2473285021 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f25b8103b70 msgr2=0x7f25b819f7e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T10:24:06.030 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:24:06.028+0000 7f25affff640 1 --2- 172.21.15.114:0/2473285021 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f25b8103b70 0x7f25b819f7e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:24:06.030 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:24:06.028+0000 7f25affff640 1 -- 172.21.15.114:0/2473285021 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f25b8101fc0 msgr2=0x7f25b8198280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T10:24:06.030 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:24:06.028+0000 7f25affff640 1 --2- 172.21.15.114:0/2473285021 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f25b8101fc0 0x7f25b8198280 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:24:06.030 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:24:06.028+0000 7f25affff640 1 -- 172.21.15.114:0/2473285021 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f25b819fe50 con 0x7f25b81031b0 2024-04-23T10:24:06.030 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:24:06.029+0000 7f25b6ffd640 1 --2- 172.21.15.114:0/2473285021 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f25b8101fc0 0x7f25b8198280 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-23T10:24:06.030 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:24:06.029+0000 7f25affff640 1 --2- 172.21.15.114:0/2473285021 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f25b81031b0 0x7f25b81987c0 secure :-1 s=READY pgs=158 cs=0 l=1 rev1=1 crypto rx=0x7f25a000e9f0 tx=0x7f25a000edf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T10:24:06.030 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:24:06.029+0000 7f25b4ff9640 1 -- 172.21.15.114:0/2473285021 <== mon.0 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f25a000cd80 con 0x7f25b81031b0 2024-04-23T10:24:06.030 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:24:06.029+0000 7f25b4ff9640 1 -- 172.21.15.114:0/2473285021 <== mon.0 v2:172.21.15.114:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f25a00054d0 con 0x7f25b81031b0 2024-04-23T10:24:06.031 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:24:06.029+0000 7f25bd4ae640 1 -- 172.21.15.114:0/2473285021 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f25b81a0140 con 0x7f25b81031b0 2024-04-23T10:24:06.031 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:24:06.029+0000 7f25b4ff9640 1 -- 172.21.15.114:0/2473285021 <== mon.0 v2:172.21.15.114:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f25a0010690 con 0x7f25b81031b0 2024-04-23T10:24:06.031 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:24:06.029+0000 7f25bd4ae640 1 -- 172.21.15.114:0/2473285021 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f25b81a0640 con 0x7f25b81031b0 2024-04-23T10:24:06.031 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:24:06.030+0000 7f25bd4ae640 1 -- 172.21.15.114:0/2473285021 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f25b8109450 con 0x7f25b81031b0 2024-04-23T10:24:06.031 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:24:06.031+0000 7f25b4ff9640 1 -- 172.21.15.114:0/2473285021 <== mon.0 v2:172.21.15.114:3300/0 4 ==== mgrmap(e 16) ==== 138647+0+0 (secure 0 0 0) 0x7f25a0004e30 con 0x7f25b81031b0 2024-04-23T10:24:06.032 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:24:06.031+0000 7f25b4ff9640 1 --2- 172.21.15.114:0/2473285021 >> [v2:172.21.15.114:6800/465333808,v1:172.21.15.114:6801/465333808] conn(0x7f25a40a3bb0 0x7f25a40a6070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T10:24:06.032 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:24:06.031+0000 7f25b6ffd640 1 --2- 172.21.15.114:0/2473285021 >> [v2:172.21.15.114:6800/465333808,v1:172.21.15.114:6801/465333808] conn(0x7f25a40a3bb0 0x7f25a40a6070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T10:24:06.032 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:24:06.032+0000 7f25b4ff9640 1 -- 172.21.15.114:0/2473285021 <== mon.0 v2:172.21.15.114:3300/0 5 ==== osd_map(83..83 src has 1..83) ==== 7127+0+0 (secure 0 0 0) 0x7f25a00d06c0 con 0x7f25b81031b0 2024-04-23T10:24:06.032 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:24:06.032+0000 7f25b6ffd640 1 --2- 172.21.15.114:0/2473285021 >> [v2:172.21.15.114:6800/465333808,v1:172.21.15.114:6801/465333808] conn(0x7f25a40a3bb0 0x7f25a40a6070 secure :-1 s=READY pgs=130 cs=0 l=1 rev1=1 crypto rx=0x7f25a8000c00 tx=0x7f25a80314a0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T10:24:06.034 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:24:06.035+0000 7f25b4ff9640 1 -- 172.21.15.114:0/2473285021 <== mon.0 v2:172.21.15.114:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f25a0053ff0 con 0x7f25b81031b0 2024-04-23T10:24:06.078 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 23 10:24:06 smithi204 ceph-mon[32649]: from='client.24361 -' entity='client.admin' cmd=[{"prefix": "orch apply mds", "fs_name": "all", "placement": "12;smithi114=a;smithi114=d;smithi114=g;smithi114=j;smithi119=b;smithi119=e;smithi119=h;smithi119=k;smithi204=c;smithi204=f;smithi204=i;smithi204=l", "target": ["mon-mgr", ""]}]: dispatch 2024-04-23T10:24:06.078 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 23 10:24:06 smithi204 ceph-mon[32649]: Saving service mds.all spec with placement smithi114=a;smithi114=d;smithi114=g;smithi114=j;smithi119=b;smithi119=e;smithi119=h;smithi119=k;smithi204=c;smithi204=f;smithi204=i;smithi204=l;count:12 2024-04-23T10:24:06.078 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 23 10:24:06 smithi204 ceph-mon[32649]: from='mgr.14152 172.21.15.114:0/1529706138' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-23T10:24:06.078 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 23 10:24:06 smithi204 ceph-mon[32649]: from='mgr.14152 172.21.15.114:0/1529706138' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-23T10:24:06.078 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 23 10:24:06 smithi204 ceph-mon[32649]: from='mgr.14152 172.21.15.114:0/1529706138' entity='mgr.x' 2024-04-23T10:24:06.078 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 23 10:24:06 smithi204 ceph-mon[32649]: from='mgr.14152 172.21.15.114:0/1529706138' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.l", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-04-23T10:24:06.078 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 23 10:24:06 smithi204 ceph-mon[32649]: from='mgr.14152 172.21.15.114:0/1529706138' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.l", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-04-23T10:24:06.078 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 23 10:24:06 smithi204 ceph-mon[32649]: from='mgr.14152 172.21.15.114:0/1529706138' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-23T10:24:06.212 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:24:06.213+0000 7f25bd4ae640 1 -- 172.21.15.114:0/2473285021 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f25b802d810 con 0x7f25b81031b0 2024-04-23T10:24:06.213 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:24:06.214+0000 7f25b4ff9640 1 -- 172.21.15.114:0/2473285021 <== mon.0 v2:172.21.15.114:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v83) ==== 74+0+18528 (secure 0 0 0) 0x7f25a009d4a0 con 0x7f25b81031b0 2024-04-23T10:24:06.213 INFO:teuthology.orchestra.run.smithi114.stdout: 2024-04-23T10:24:06.214 INFO:teuthology.orchestra.run.smithi114.stdout:{"epoch":83,"fsid":"aa240572-0159-11ef-bc93-c7b262605968","created":"2024-04-23T10:16:53.351136+0000","modified":"2024-04-23T10:23:55.163231+0000","last_up_change":"2024-04-23T10:23:54.153269+0000","last_in_change":"2024-04-23T10:23:32.963085+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":26,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":1,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-04-23T10:20:11.970297+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":12,"score_stable":12,"optimal_score":0.25,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"fd64adde-9445-45a4-a171-ef656e2c5709","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6802","nonce":736366878},{"type":"v1","addr":"172.21.15.114:6803","nonce":736366878}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6804","nonce":736366878},{"type":"v1","addr":"172.21.15.114:6805","nonce":736366878}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6808","nonce":736366878},{"type":"v1","addr":"172.21.15.114:6809","nonce":736366878}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6806","nonce":736366878},{"type":"v1","addr":"172.21.15.114:6807","nonce":736366878}]},"public_addr":"172.21.15.114:6803/736366878","cluster_addr":"172.21.15.114:6805/736366878","heartbeat_back_addr":"172.21.15.114:6809/736366878","heartbeat_front_addr":"172.21.15.114:6807/736366878","state":["exists","up"]},{"osd":1,"uuid":"e279a1e6-2702-472a-930a-e08ece76a861","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":42,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6800","nonce":1133540779},{"type":"v1","addr":"172.21.15.119:6801","nonce":1133540779}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6802","nonce":1133540779},{"type":"v1","addr":"172.21.15.119:6803","nonce":1133540779}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6806","nonce":1133540779},{"type":"v1","addr":"172.21.15.119:6807","nonce":1133540779}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6804","nonce":1133540779},{"type":"v1","addr":"172.21.15.119:6805","nonce":1133540779}]},"public_addr":"172.21.15.119:6801/1133540779","cluster_addr":"172.21.15.119:6803/1133540779","heartbeat_back_addr":"172.21.15.119:6807/1133540779","heartbeat_front_addr":"172.21.15.119:6805/1133540779","state":["exists","up"]},{"osd":2,"uuid":"f4c9f2b4-ff49-4e14-89cd-b477feeb9382","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6800","nonce":799652511},{"type":"v1","addr":"172.21.15.204:6801","nonce":799652511}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6802","nonce":799652511},{"type":"v1","addr":"172.21.15.204:6803","nonce":799652511}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6806","nonce":799652511},{"type":"v1","addr":"172.21.15.204:6807","nonce":799652511}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6804","nonce":799652511},{"type":"v1","addr":"172.21.15.204:6805","nonce":799652511}]},"public_addr":"172.21.15.204:6801/799652511","cluster_addr":"172.21.15.204:6803/799652511","heartbeat_back_addr":"172.21.15.204:6807/799652511","heartbeat_front_addr":"172.21.15.204:6805/799652511","state":["exists","up"]},{"osd":3,"uuid":"52d2a566-e71c-4bb3-8d6c-5abdd43177c2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":25,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6810","nonce":4275595091},{"type":"v1","addr":"172.21.15.114:6811","nonce":4275595091}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6812","nonce":4275595091},{"type":"v1","addr":"172.21.15.114:6813","nonce":4275595091}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6816","nonce":4275595091},{"type":"v1","addr":"172.21.15.114:6817","nonce":4275595091}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6814","nonce":4275595091},{"type":"v1","addr":"172.21.15.114:6815","nonce":4275595091}]},"public_addr":"172.21.15.114:6811/4275595091","cluster_addr":"172.21.15.114:6813/4275595091","heartbeat_back_addr":"172.21.15.114:6817/4275595091","heartbeat_front_addr":"172.21.15.114:6815/4275595091","state":["exists","up"]},{"osd":4,"uuid":"d911a7c1-4bdc-4d28-b8a6-73896a6e7de6","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":33,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6808","nonce":1611007172},{"type":"v1","addr":"172.21.15.119:6809","nonce":1611007172}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6810","nonce":1611007172},{"type":"v1","addr":"172.21.15.119:6811","nonce":1611007172}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6814","nonce":1611007172},{"type":"v1","addr":"172.21.15.119:6815","nonce":1611007172}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6812","nonce":1611007172},{"type":"v1","addr":"172.21.15.119:6813","nonce":1611007172}]},"public_addr":"172.21.15.119:6809/1611007172","cluster_addr":"172.21.15.119:6811/1611007172","heartbeat_back_addr":"172.21.15.119:6815/1611007172","heartbeat_front_addr":"172.21.15.119:6813/1611007172","state":["exists","up"]},{"osd":5,"uuid":"f7590e1a-e93c-4c29-b15f-38ee52d1f921","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":39,"up_thru":55,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6808","nonce":1427563312},{"type":"v1","addr":"172.21.15.204:6809","nonce":1427563312}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6810","nonce":1427563312},{"type":"v1","addr":"172.21.15.204:6811","nonce":1427563312}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6814","nonce":1427563312},{"type":"v1","addr":"172.21.15.204:6815","nonce":1427563312}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6812","nonce":1427563312},{"type":"v1","addr":"172.21.15.204:6813","nonce":1427563312}]},"public_addr":"172.21.15.204:6809/1427563312","cluster_addr":"172.21.15.204:6811/1427563312","heartbeat_back_addr":"172.21.15.204:6815/1427563312","heartbeat_front_addr":"172.21.15.204:6813/1427563312","state":["exists","up"]},{"osd":6,"uuid":"534d20e8-3e54-4db8-a845-6119f49c7a78","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":47,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6818","nonce":4072912544},{"type":"v1","addr":"172.21.15.114:6819","nonce":4072912544}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6820","nonce":4072912544},{"type":"v1","addr":"172.21.15.114:6821","nonce":4072912544}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6824","nonce":4072912544},{"type":"v1","addr":"172.21.15.114:6825","nonce":4072912544}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6822","nonce":4072912544},{"type":"v1","addr":"172.21.15.114:6823","nonce":4072912544}]},"public_addr":"172.21.15.114:6819/4072912544","cluster_addr":"172.21.15.114:6821/4072912544","heartbeat_back_addr":"172.21.15.114:6825/4072912544","heartbeat_front_addr":"172.21.15.114:6823/4072912544","state":["exists","up"]},{"osd":7,"uuid":"f0ec22b8-1b48-4859-9d65-b00403aff3ad","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":53,"up_thru":77,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6816","nonce":2237458793},{"type":"v1","addr":"172.21.15.119:6817","nonce":2237458793}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6818","nonce":2237458793},{"type":"v1","addr":"172.21.15.119:6819","nonce":2237458793}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6822","nonce":2237458793},{"type":"v1","addr":"172.21.15.119:6823","nonce":2237458793}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6820","nonce":2237458793},{"type":"v1","addr":"172.21.15.119:6821","nonce":2237458793}]},"public_addr":"172.21.15.119:6817/2237458793","cluster_addr":"172.21.15.119:6819/2237458793","heartbeat_back_addr":"172.21.15.119:6823/2237458793","heartbeat_front_addr":"172.21.15.119:6821/2237458793","state":["exists","up"]},{"osd":8,"uuid":"3ac89686-6d76-4cef-9089-ac29e4b3bcac","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":62,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6816","nonce":1426318231},{"type":"v1","addr":"172.21.15.204:6817","nonce":1426318231}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6818","nonce":1426318231},{"type":"v1","addr":"172.21.15.204:6819","nonce":1426318231}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6822","nonce":1426318231},{"type":"v1","addr":"172.21.15.204:6823","nonce":1426318231}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6820","nonce":1426318231},{"type":"v1","addr":"172.21.15.204:6821","nonce":1426318231}]},"public_addr":"172.21.15.204:6817/1426318231","cluster_addr":"172.21.15.204:6819/1426318231","heartbeat_back_addr":"172.21.15.204:6823/1426318231","heartbeat_front_addr":"172.21.15.204:6821/1426318231","state":["exists","up"]},{"osd":9,"uuid":"a80ffe4d-1e98-460f-b046-c4754285cc5a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":69,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6826","nonce":2750635721},{"type":"v1","addr":"172.21.15.114:6827","nonce":2750635721}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6828","nonce":2750635721},{"type":"v1","addr":"172.21.15.114:6829","nonce":2750635721}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6832","nonce":2750635721},{"type":"v1","addr":"172.21.15.114:6833","nonce":2750635721}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6830","nonce":2750635721},{"type":"v1","addr":"172.21.15.114:6831","nonce":2750635721}]},"public_addr":"172.21.15.114:6827/2750635721","cluster_addr":"172.21.15.114:6829/2750635721","heartbeat_back_addr":"172.21.15.114:6833/2750635721","heartbeat_front_addr":"172.21.15.114:6831/2750635721","state":["exists","up"]},{"osd":10,"uuid":"cd75a579-bf12-4600-ac09-a8bfce772ced","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":74,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6824","nonce":1400187701},{"type":"v1","addr":"172.21.15.119:6825","nonce":1400187701}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6826","nonce":1400187701},{"type":"v1","addr":"172.21.15.119:6827","nonce":1400187701}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6830","nonce":1400187701},{"type":"v1","addr":"172.21.15.119:6831","nonce":1400187701}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6828","nonce":1400187701},{"type":"v1","addr":"172.21.15.119:6829","nonce":1400187701}]},"public_addr":"172.21.15.119:6825/1400187701","cluster_addr":"172.21.15.119:6827/1400187701","heartbeat_back_addr":"172.21.15.119:6831/1400187701","heartbeat_front_addr":"172.21.15.119:6829/1400187701","state":["exists","up"]},{"osd":11,"uuid":"53f037a5-3ec0-483a-80a1-dfee923a68e5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":82,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6824","nonce":2407298490},{"type":"v1","addr":"172.21.15.204:6825","nonce":2407298490}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6826","nonce":2407298490},{"type":"v1","addr":"172.21.15.204:6827","nonce":2407298490}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6830","nonce":2407298490},{"type":"v1","addr":"172.21.15.204:6831","nonce":2407298490}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6828","nonce":2407298490},{"type":"v1","addr":"172.21.15.204:6829","nonce":2407298490}]},"public_addr":"172.21.15.204:6825/2407298490","cluster_addr":"172.21.15.204:6827/2407298490","heartbeat_back_addr":"172.21.15.204:6831/2407298490","heartbeat_front_addr":"172.21.15.204:6829/2407298490","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-23T10:19:18.401147+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-23T10:19:42.611299+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-23T10:20:07.277367+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-23T10:20:31.208368+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-23T10:20:55.866225+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-23T10:21:19.961237+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-23T10:21:44.480783+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-23T10:22:09.173641+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-23T10:22:34.030328+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-23T10:22:59.652293+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-23T10:23:25.659673+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-23T10:23:51.168171+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.114:6801/1604678557":"2024-04-24T10:17:45.874609+0000","172.21.15.114:0/4025385854":"2024-04-24T10:17:45.874609+0000","172.21.15.114:0/2228273469":"2024-04-24T10:17:19.910348+0000","172.21.15.114:6800/2831530838":"2024-04-24T10:17:19.910348+0000","172.21.15.114:0/1333447450":"2024-04-24T10:17:45.874609+0000","172.21.15.114:0/2233026840":"2024-04-24T10:17:19.910348+0000","172.21.15.114:0/1902281545":"2024-04-24T10:17:19.910348+0000","172.21.15.114:6801/2831530838":"2024-04-24T10:17:19.910348+0000","172.21.15.114:6800/1604678557":"2024-04-24T10:17:45.874609+0000","172.21.15.114:0/1623553811":"2024-04-24T10:17:45.874609+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-23T10:24:06.215 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:24:06.216+0000 7f25bd4ae640 1 -- 172.21.15.114:0/2473285021 >> [v2:172.21.15.114:6800/465333808,v1:172.21.15.114:6801/465333808] conn(0x7f25a40a3bb0 msgr2=0x7f25a40a6070 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T10:24:06.215 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:24:06.216+0000 7f25bd4ae640 1 --2- 172.21.15.114:0/2473285021 >> [v2:172.21.15.114:6800/465333808,v1:172.21.15.114:6801/465333808] conn(0x7f25a40a3bb0 0x7f25a40a6070 secure :-1 s=READY pgs=130 cs=0 l=1 rev1=1 crypto rx=0x7f25a8000c00 tx=0x7f25a80314a0 comp rx=0 tx=0).stop 2024-04-23T10:24:06.215 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:24:06.216+0000 7f25bd4ae640 1 -- 172.21.15.114:0/2473285021 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f25b81031b0 msgr2=0x7f25b81987c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T10:24:06.216 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:24:06.216+0000 7f25bd4ae640 1 --2- 172.21.15.114:0/2473285021 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f25b81031b0 0x7f25b81987c0 secure :-1 s=READY pgs=158 cs=0 l=1 rev1=1 crypto rx=0x7f25a000e9f0 tx=0x7f25a000edf0 comp rx=0 tx=0).stop 2024-04-23T10:24:06.216 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:24:06.216+0000 7f25bd4ae640 1 -- 172.21.15.114:0/2473285021 shutdown_connections 2024-04-23T10:24:06.216 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:24:06.216+0000 7f25bd4ae640 1 --2- 172.21.15.114:0/2473285021 >> [v2:172.21.15.114:6800/465333808,v1:172.21.15.114:6801/465333808] conn(0x7f25a40a3bb0 0x7f25a40a6070 unknown :-1 s=CLOSED pgs=130 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:24:06.216 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:24:06.216+0000 7f25bd4ae640 1 --2- 172.21.15.114:0/2473285021 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f25b8103b70 0x7f25b819f7e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:24:06.216 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:24:06.216+0000 7f25bd4ae640 1 --2- 172.21.15.114:0/2473285021 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f25b81031b0 0x7f25b81987c0 unknown :-1 s=CLOSED pgs=158 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:24:06.216 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:24:06.216+0000 7f25bd4ae640 1 --2- 172.21.15.114:0/2473285021 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f25b8101fc0 0x7f25b8198280 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:24:06.216 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:24:06.216+0000 7f25bd4ae640 1 -- 172.21.15.114:0/2473285021 >> 172.21.15.114:0/2473285021 conn(0x7f25b80fd550 msgr2=0x7f25b8106620 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T10:24:06.216 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:24:06.216+0000 7f25bd4ae640 1 -- 172.21.15.114:0/2473285021 shutdown_connections 2024-04-23T10:24:06.216 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:24:06.216+0000 7f25bd4ae640 1 -- 172.21.15.114:0/2473285021 wait complete. 2024-04-23T10:24:06.225 INFO:journalctl@ceph.mon.a.smithi114.stdout:Apr 23 10:24:06 smithi114 ceph-mon[29255]: from='client.24361 -' entity='client.admin' cmd=[{"prefix": "orch apply mds", "fs_name": "all", "placement": "12;smithi114=a;smithi114=d;smithi114=g;smithi114=j;smithi119=b;smithi119=e;smithi119=h;smithi119=k;smithi204=c;smithi204=f;smithi204=i;smithi204=l", "target": ["mon-mgr", ""]}]: dispatch 2024-04-23T10:24:06.225 INFO:journalctl@ceph.mon.a.smithi114.stdout:Apr 23 10:24:06 smithi114 ceph-mon[29255]: Saving service mds.all spec with placement smithi114=a;smithi114=d;smithi114=g;smithi114=j;smithi119=b;smithi119=e;smithi119=h;smithi119=k;smithi204=c;smithi204=f;smithi204=i;smithi204=l;count:12 2024-04-23T10:24:06.225 INFO:journalctl@ceph.mon.a.smithi114.stdout:Apr 23 10:24:06 smithi114 ceph-mon[29255]: from='mgr.14152 172.21.15.114:0/1529706138' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-23T10:24:06.225 INFO:journalctl@ceph.mon.a.smithi114.stdout:Apr 23 10:24:06 smithi114 ceph-mon[29255]: from='mgr.14152 172.21.15.114:0/1529706138' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-23T10:24:06.225 INFO:journalctl@ceph.mon.a.smithi114.stdout:Apr 23 10:24:06 smithi114 ceph-mon[29255]: from='mgr.14152 172.21.15.114:0/1529706138' entity='mgr.x' 2024-04-23T10:24:06.225 INFO:journalctl@ceph.mon.a.smithi114.stdout:Apr 23 10:24:06 smithi114 ceph-mon[29255]: from='mgr.14152 172.21.15.114:0/1529706138' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.l", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-04-23T10:24:06.225 INFO:journalctl@ceph.mon.a.smithi114.stdout:Apr 23 10:24:06 smithi114 ceph-mon[29255]: from='mgr.14152 172.21.15.114:0/1529706138' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.l", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-04-23T10:24:06.225 INFO:journalctl@ceph.mon.a.smithi114.stdout:Apr 23 10:24:06 smithi114 ceph-mon[29255]: from='mgr.14152 172.21.15.114:0/1529706138' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-23T10:24:06.227 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-04-23T10:20:11.970297+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '22', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 12, 'score_stable': 12, 'optimal_score': 0.25, 'raw_score_acting': 3, 'raw_score_stable': 3, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-04-23T10:24:06.227 DEBUG:teuthology.orchestra.run.smithi114:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-04-23T10:24:06.330 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:24:06.329+0000 7f3657f08640 1 -- 172.21.15.114:0/2306169959 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f36501029a0 msgr2=0x7f36501092c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T10:24:06.330 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:24:06.329+0000 7f3657f08640 1 --2- 172.21.15.114:0/2306169959 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f36501029a0 0x7f36501092c0 secure :-1 s=READY pgs=159 cs=0 l=1 rev1=1 crypto rx=0x7f3648009ca0 tx=0x7f364802a8c0 comp rx=0 tx=0).stop 2024-04-23T10:24:06.330 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:24:06.329+0000 7f3657f08640 1 -- 172.21.15.114:0/2306169959 shutdown_connections 2024-04-23T10:24:06.330 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:24:06.329+0000 7f3657f08640 1 --2- 172.21.15.114:0/2306169959 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f36501029a0 0x7f36501092c0 unknown :-1 s=CLOSED pgs=159 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:24:06.330 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:24:06.329+0000 7f3657f08640 1 --2- 172.21.15.114:0/2306169959 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f3650101fe0 0x7f3650102460 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:24:06.331 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:24:06.329+0000 7f3657f08640 1 --2- 172.21.15.114:0/2306169959 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f3650100e90 0x7f36501012b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:24:06.331 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:24:06.329+0000 7f3657f08640 1 -- 172.21.15.114:0/2306169959 >> 172.21.15.114:0/2306169959 conn(0x7f36500fc440 msgr2=0x7f36500fe860 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T10:24:06.331 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:24:06.329+0000 7f3657f08640 1 -- 172.21.15.114:0/2306169959 shutdown_connections 2024-04-23T10:24:06.331 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:24:06.329+0000 7f3657f08640 1 -- 172.21.15.114:0/2306169959 wait complete. 2024-04-23T10:24:06.331 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:24:06.329+0000 7f3657f08640 1 Processor -- start 2024-04-23T10:24:06.331 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:24:06.329+0000 7f3657f08640 1 -- start start 2024-04-23T10:24:06.331 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:24:06.330+0000 7f3657f08640 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f3650100e90 0x7f365006b100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T10:24:06.331 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:24:06.330+0000 7f3657f08640 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f3650101fe0 0x7f365019f340 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T10:24:06.331 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:24:06.330+0000 7f3657f08640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f36501029a0 0x7f36501a1890 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T10:24:06.331 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:24:06.330+0000 7f3657f08640 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x7f365010ca40 con 0x7f36501029a0 2024-04-23T10:24:06.331 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:24:06.330+0000 7f3657f08640 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 -- 0x7f365010c8c0 con 0x7f3650100e90 2024-04-23T10:24:06.331 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:24:06.330+0000 7f3657f08640 1 -- --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_getmap magic: 0 -- 0x7f365010cbc0 con 0x7f3650101fe0 2024-04-23T10:24:06.331 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:24:06.330+0000 7f36564a5640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f36501029a0 0x7f36501a1890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T10:24:06.332 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:24:06.330+0000 7f36564a5640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f36501029a0 0x7f36501a1890 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.114:38926/0 (socket says 172.21.15.114:38926) 2024-04-23T10:24:06.332 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:24:06.330+0000 7f36564a5640 1 -- 172.21.15.114:0/704862242 learned_addr learned my addr 172.21.15.114:0/704862242 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-04-23T10:24:06.332 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:24:06.330+0000 7f36554a3640 1 --2- 172.21.15.114:0/704862242 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f3650101fe0 0x7f365019f340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T10:24:06.332 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:24:06.330+0000 7f3655ca4640 1 --2- 172.21.15.114:0/704862242 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f3650100e90 0x7f365006b100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T10:24:06.332 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:24:06.330+0000 7f36564a5640 1 -- 172.21.15.114:0/704862242 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f3650101fe0 msgr2=0x7f365019f340 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T10:24:06.332 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:24:06.330+0000 7f36564a5640 1 --2- 172.21.15.114:0/704862242 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f3650101fe0 0x7f365019f340 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:24:06.332 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:24:06.330+0000 7f36564a5640 1 -- 172.21.15.114:0/704862242 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f3650100e90 msgr2=0x7f365006b100 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T10:24:06.332 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:24:06.330+0000 7f36564a5640 1 --2- 172.21.15.114:0/704862242 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f3650100e90 0x7f365006b100 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:24:06.332 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:24:06.330+0000 7f36564a5640 1 -- 172.21.15.114:0/704862242 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f36501a2080 con 0x7f36501029a0 2024-04-23T10:24:06.332 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:24:06.330+0000 7f36564a5640 1 --2- 172.21.15.114:0/704862242 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f36501029a0 0x7f36501a1890 secure :-1 s=READY pgs=160 cs=0 l=1 rev1=1 crypto rx=0x7f3648009430 tx=0x7f36480316f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T10:24:06.332 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:24:06.330+0000 7f3646ffd640 1 -- 172.21.15.114:0/704862242 <== mon.0 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3648003b70 con 0x7f36501029a0 2024-04-23T10:24:06.332 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:24:06.330+0000 7f3646ffd640 1 -- 172.21.15.114:0/704862242 <== mon.0 v2:172.21.15.114:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f3648003d10 con 0x7f36501029a0 2024-04-23T10:24:06.332 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:24:06.330+0000 7f3657f08640 1 -- 172.21.15.114:0/704862242 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f36501a22b0 con 0x7f36501029a0 2024-04-23T10:24:06.333 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:24:06.330+0000 7f3646ffd640 1 -- 172.21.15.114:0/704862242 <== mon.0 v2:172.21.15.114:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f364802fb40 con 0x7f36501029a0 2024-04-23T10:24:06.333 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:24:06.330+0000 7f3657f08640 1 -- 172.21.15.114:0/704862242 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f36501a2850 con 0x7f36501029a0 2024-04-23T10:24:06.333 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:24:06.331+0000 7f3657f08640 1 -- 172.21.15.114:0/704862242 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f3650108140 con 0x7f36501029a0 2024-04-23T10:24:06.333 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:24:06.332+0000 7f3646ffd640 1 -- 172.21.15.114:0/704862242 <== mon.0 v2:172.21.15.114:3300/0 4 ==== mgrmap(e 16) ==== 138647+0+0 (secure 0 0 0) 0x7f3648004610 con 0x7f36501029a0 2024-04-23T10:24:06.334 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:24:06.333+0000 7f3646ffd640 1 --2- 172.21.15.114:0/704862242 >> [v2:172.21.15.114:6800/465333808,v1:172.21.15.114:6801/465333808] conn(0x7f363c0a3c60 0x7f363c0a6120 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T10:24:06.334 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:24:06.333+0000 7f3655ca4640 1 --2- 172.21.15.114:0/704862242 >> [v2:172.21.15.114:6800/465333808,v1:172.21.15.114:6801/465333808] conn(0x7f363c0a3c60 0x7f363c0a6120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T10:24:06.334 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:24:06.333+0000 7f3646ffd640 1 -- 172.21.15.114:0/704862242 <== mon.0 v2:172.21.15.114:3300/0 5 ==== osd_map(83..83 src has 1..83) ==== 7127+0+0 (secure 0 0 0) 0x7f36480f1260 con 0x7f36501029a0 2024-04-23T10:24:06.334 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:24:06.333+0000 7f3655ca4640 1 --2- 172.21.15.114:0/704862242 >> [v2:172.21.15.114:6800/465333808,v1:172.21.15.114:6801/465333808] conn(0x7f363c0a3c60 0x7f363c0a6120 secure :-1 s=READY pgs=131 cs=0 l=1 rev1=1 crypto rx=0x7f364c009bd0 tx=0x7f364c005f90 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T10:24:06.336 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:24:06.338+0000 7f3646ffd640 1 -- 172.21.15.114:0/704862242 <== mon.0 v2:172.21.15.114:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f36480be0e0 con 0x7f36501029a0 2024-04-23T10:24:06.492 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 23 10:24:06 smithi119 ceph-mon[32608]: from='client.24361 -' entity='client.admin' cmd=[{"prefix": "orch apply mds", "fs_name": "all", "placement": "12;smithi114=a;smithi114=d;smithi114=g;smithi114=j;smithi119=b;smithi119=e;smithi119=h;smithi119=k;smithi204=c;smithi204=f;smithi204=i;smithi204=l", "target": ["mon-mgr", ""]}]: dispatch 2024-04-23T10:24:06.492 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 23 10:24:06 smithi119 ceph-mon[32608]: Saving service mds.all spec with placement smithi114=a;smithi114=d;smithi114=g;smithi114=j;smithi119=b;smithi119=e;smithi119=h;smithi119=k;smithi204=c;smithi204=f;smithi204=i;smithi204=l;count:12 2024-04-23T10:24:06.492 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 23 10:24:06 smithi119 ceph-mon[32608]: from='mgr.14152 172.21.15.114:0/1529706138' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-23T10:24:06.492 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 23 10:24:06 smithi119 ceph-mon[32608]: from='mgr.14152 172.21.15.114:0/1529706138' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-23T10:24:06.492 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 23 10:24:06 smithi119 ceph-mon[32608]: from='mgr.14152 172.21.15.114:0/1529706138' entity='mgr.x' 2024-04-23T10:24:06.492 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 23 10:24:06 smithi119 ceph-mon[32608]: from='mgr.14152 172.21.15.114:0/1529706138' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.l", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-04-23T10:24:06.492 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 23 10:24:06 smithi119 ceph-mon[32608]: from='mgr.14152 172.21.15.114:0/1529706138' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.l", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-04-23T10:24:06.492 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 23 10:24:06 smithi119 ceph-mon[32608]: from='mgr.14152 172.21.15.114:0/1529706138' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-23T10:24:06.515 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:24:06.516+0000 7f3657f08640 1 -- 172.21.15.114:0/704862242 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) -- 0x7f365002d770 con 0x7f36501029a0 2024-04-23T10:24:06.516 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:24:06.517+0000 7f3646ffd640 1 -- 172.21.15.114:0/704862242 <== mon.0 v2:172.21.15.114:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v83) ==== 93+0+10 (secure 0 0 0) 0x7f36480bd9e0 con 0x7f36501029a0 2024-04-23T10:24:06.516 INFO:teuthology.orchestra.run.smithi114.stdout:pg_num: 1 2024-04-23T10:24:06.518 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:24:06.519+0000 7f3657f08640 1 -- 172.21.15.114:0/704862242 >> [v2:172.21.15.114:6800/465333808,v1:172.21.15.114:6801/465333808] conn(0x7f363c0a3c60 msgr2=0x7f363c0a6120 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T10:24:06.518 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:24:06.519+0000 7f3657f08640 1 --2- 172.21.15.114:0/704862242 >> [v2:172.21.15.114:6800/465333808,v1:172.21.15.114:6801/465333808] conn(0x7f363c0a3c60 0x7f363c0a6120 secure :-1 s=READY pgs=131 cs=0 l=1 rev1=1 crypto rx=0x7f364c009bd0 tx=0x7f364c005f90 comp rx=0 tx=0).stop 2024-04-23T10:24:06.518 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:24:06.519+0000 7f3657f08640 1 -- 172.21.15.114:0/704862242 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f36501029a0 msgr2=0x7f36501a1890 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T10:24:06.518 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:24:06.519+0000 7f3657f08640 1 --2- 172.21.15.114:0/704862242 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f36501029a0 0x7f36501a1890 secure :-1 s=READY pgs=160 cs=0 l=1 rev1=1 crypto rx=0x7f3648009430 tx=0x7f36480316f0 comp rx=0 tx=0).stop 2024-04-23T10:24:06.519 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:24:06.519+0000 7f3657f08640 1 -- 172.21.15.114:0/704862242 shutdown_connections 2024-04-23T10:24:06.519 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:24:06.519+0000 7f3657f08640 1 --2- 172.21.15.114:0/704862242 >> [v2:172.21.15.114:6800/465333808,v1:172.21.15.114:6801/465333808] conn(0x7f363c0a3c60 0x7f363c0a6120 unknown :-1 s=CLOSED pgs=131 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:24:06.519 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:24:06.519+0000 7f3657f08640 1 --2- 172.21.15.114:0/704862242 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f36501029a0 0x7f36501a1890 unknown :-1 s=CLOSED pgs=160 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:24:06.519 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:24:06.519+0000 7f3657f08640 1 --2- 172.21.15.114:0/704862242 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f3650101fe0 0x7f365019f340 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:24:06.519 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:24:06.519+0000 7f3657f08640 1 --2- 172.21.15.114:0/704862242 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f3650100e90 0x7f365006b100 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:24:06.519 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:24:06.519+0000 7f3657f08640 1 -- 172.21.15.114:0/704862242 >> 172.21.15.114:0/704862242 conn(0x7f36500fc440 msgr2=0x7f3650105310 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T10:24:06.519 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:24:06.519+0000 7f3657f08640 1 -- 172.21.15.114:0/704862242 shutdown_connections 2024-04-23T10:24:06.519 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:24:06.519+0000 7f3657f08640 1 -- 172.21.15.114:0/704862242 wait complete. 2024-04-23T10:24:06.529 DEBUG:teuthology.orchestra.run.smithi114:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-23T10:24:06.626 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:24:06.627+0000 7f7c167fc640 1 -- 172.21.15.114:0/2372625938 <== mon.0 v2:172.21.15.114:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7c0c0318a0 con 0x7f7c18102040 2024-04-23T10:24:06.626 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:24:06.627+0000 7f7c1e108640 1 -- 172.21.15.114:0/2372625938 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f7c18102040 msgr2=0x7f7c181024c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T10:24:06.627 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:24:06.627+0000 7f7c1e108640 1 --2- 172.21.15.114:0/2372625938 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f7c18102040 0x7f7c181024c0 secure :-1 s=READY pgs=161 cs=0 l=1 rev1=1 crypto rx=0x7f7c0c009900 tx=0x7f7c0c02a410 comp rx=0 tx=0).stop 2024-04-23T10:24:06.627 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:24:06.627+0000 7f7c1e108640 1 -- 172.21.15.114:0/2372625938 shutdown_connections 2024-04-23T10:24:06.627 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:24:06.627+0000 7f7c1e108640 1 --2- 172.21.15.114:0/2372625938 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f7c18102a00 0x7f7c18109320 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:24:06.627 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:24:06.627+0000 7f7c1e108640 1 --2- 172.21.15.114:0/2372625938 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f7c18102040 0x7f7c181024c0 unknown :-1 s=CLOSED pgs=161 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:24:06.627 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:24:06.627+0000 7f7c1e108640 1 --2- 172.21.15.114:0/2372625938 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f7c18100e60 0x7f7c18101280 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:24:06.627 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:24:06.627+0000 7f7c1e108640 1 -- 172.21.15.114:0/2372625938 >> 172.21.15.114:0/2372625938 conn(0x7f7c180fc440 msgr2=0x7f7c180fe860 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T10:24:06.627 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:24:06.627+0000 7f7c1e108640 1 -- 172.21.15.114:0/2372625938 shutdown_connections 2024-04-23T10:24:06.627 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:24:06.627+0000 7f7c1e108640 1 -- 172.21.15.114:0/2372625938 wait complete. 2024-04-23T10:24:06.627 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:24:06.628+0000 7f7c1e108640 1 Processor -- start 2024-04-23T10:24:06.629 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:24:06.628+0000 7f7c1e108640 1 -- start start 2024-04-23T10:24:06.629 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:24:06.628+0000 7f7c1e108640 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f7c18100e60 0x7f7c1806b100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T10:24:06.629 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:24:06.628+0000 7f7c1e108640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f7c18102040 0x7f7c1819cf20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T10:24:06.629 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:24:06.628+0000 7f7c1e108640 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f7c18102a00 0x7f7c1819f540 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T10:24:06.629 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:24:06.628+0000 7f7c1e108640 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x7f7c1810ca50 con 0x7f7c18102040 2024-04-23T10:24:06.630 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:24:06.628+0000 7f7c1e108640 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 -- 0x7f7c1810c8d0 con 0x7f7c18102a00 2024-04-23T10:24:06.630 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:24:06.628+0000 7f7c1e108640 1 -- --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_getmap magic: 0 -- 0x7f7c1810cbd0 con 0x7f7c18100e60 2024-04-23T10:24:06.630 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:24:06.628+0000 7f7c16ffd640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f7c18102040 0x7f7c1819cf20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T10:24:06.630 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:24:06.628+0000 7f7c16ffd640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f7c18102040 0x7f7c1819cf20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.114:38946/0 (socket says 172.21.15.114:38946) 2024-04-23T10:24:06.630 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:24:06.628+0000 7f7c16ffd640 1 -- 172.21.15.114:0/1441704618 learned_addr learned my addr 172.21.15.114:0/1441704618 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-04-23T10:24:06.630 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:24:06.628+0000 7f7c17fff640 1 --2- 172.21.15.114:0/1441704618 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f7c18102a00 0x7f7c1819f540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T10:24:06.630 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:24:06.628+0000 7f7c177fe640 1 --2- 172.21.15.114:0/1441704618 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f7c18100e60 0x7f7c1806b100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T10:24:06.630 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:24:06.628+0000 7f7c16ffd640 1 -- 172.21.15.114:0/1441704618 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f7c18100e60 msgr2=0x7f7c1806b100 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T10:24:06.630 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:24:06.629+0000 7f7c16ffd640 1 --2- 172.21.15.114:0/1441704618 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f7c18100e60 0x7f7c1806b100 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:24:06.630 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:24:06.629+0000 7f7c16ffd640 1 -- 172.21.15.114:0/1441704618 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f7c18102a00 msgr2=0x7f7c1819f540 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T10:24:06.630 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:24:06.629+0000 7f7c16ffd640 1 --2- 172.21.15.114:0/1441704618 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f7c18102a00 0x7f7c1819f540 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:24:06.630 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:24:06.629+0000 7f7c16ffd640 1 -- 172.21.15.114:0/1441704618 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f7c1819fe50 con 0x7f7c18102040 2024-04-23T10:24:06.631 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:24:06.629+0000 7f7c177fe640 1 --2- 172.21.15.114:0/1441704618 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f7c18100e60 0x7f7c1806b100 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-23T10:24:06.631 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:24:06.629+0000 7f7c16ffd640 1 --2- 172.21.15.114:0/1441704618 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f7c18102040 0x7f7c1819cf20 secure :-1 s=READY pgs=162 cs=0 l=1 rev1=1 crypto rx=0x7f7c0c005940 tx=0x7f7c0c031a50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T10:24:06.631 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:24:06.629+0000 7f7c14ff9640 1 -- 172.21.15.114:0/1441704618 <== mon.0 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7c0c03f030 con 0x7f7c18102040 2024-04-23T10:24:06.631 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:24:06.629+0000 7f7c14ff9640 1 -- 172.21.15.114:0/1441704618 <== mon.0 v2:172.21.15.114:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f7c0c003aa0 con 0x7f7c18102040 2024-04-23T10:24:06.631 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:24:06.629+0000 7f7c1e108640 1 -- 172.21.15.114:0/1441704618 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f7c181a0080 con 0x7f7c18102040 2024-04-23T10:24:06.631 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:24:06.629+0000 7f7c14ff9640 1 -- 172.21.15.114:0/1441704618 <== mon.0 v2:172.21.15.114:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7c0c035070 con 0x7f7c18102040 2024-04-23T10:24:06.631 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:24:06.629+0000 7f7c1e108640 1 -- 172.21.15.114:0/1441704618 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f7c181a05c0 con 0x7f7c18102040 2024-04-23T10:24:06.631 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:24:06.630+0000 7f7c1e108640 1 -- 172.21.15.114:0/1441704618 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f7c18108200 con 0x7f7c18102040 2024-04-23T10:24:06.632 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:24:06.631+0000 7f7c14ff9640 1 -- 172.21.15.114:0/1441704618 <== mon.0 v2:172.21.15.114:3300/0 4 ==== mgrmap(e 16) ==== 138647+0+0 (secure 0 0 0) 0x7f7c0c042030 con 0x7f7c18102040 2024-04-23T10:24:06.632 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:24:06.631+0000 7f7c14ff9640 1 --2- 172.21.15.114:0/1441704618 >> [v2:172.21.15.114:6800/465333808,v1:172.21.15.114:6801/465333808] conn(0x7f7c000a3c30 0x7f7c000a60f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T10:24:06.632 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:24:06.631+0000 7f7c177fe640 1 --2- 172.21.15.114:0/1441704618 >> [v2:172.21.15.114:6800/465333808,v1:172.21.15.114:6801/465333808] conn(0x7f7c000a3c30 0x7f7c000a60f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T10:24:06.632 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:24:06.632+0000 7f7c14ff9640 1 -- 172.21.15.114:0/1441704618 <== mon.0 v2:172.21.15.114:3300/0 5 ==== osd_map(83..83 src has 1..83) ==== 7127+0+0 (secure 0 0 0) 0x7f7c0c0f0520 con 0x7f7c18102040 2024-04-23T10:24:06.632 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:24:06.632+0000 7f7c177fe640 1 --2- 172.21.15.114:0/1441704618 >> [v2:172.21.15.114:6800/465333808,v1:172.21.15.114:6801/465333808] conn(0x7f7c000a3c30 0x7f7c000a60f0 secure :-1 s=READY pgs=132 cs=0 l=1 rev1=1 crypto rx=0x7f7c080058c0 tx=0x7f7c08005850 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T10:24:06.635 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:24:06.636+0000 7f7c14ff9640 1 -- 172.21.15.114:0/1441704618 <== mon.0 v2:172.21.15.114:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f7c0c0bd3a0 con 0x7f7c18102040 2024-04-23T10:24:06.850 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:24:06.850+0000 7f7c1e108640 1 -- 172.21.15.114:0/1441704618 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f7c1802d810 con 0x7f7c18102040 2024-04-23T10:24:06.850 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:24:06.851+0000 7f7c14ff9640 1 -- 172.21.15.114:0/1441704618 <== mon.0 v2:172.21.15.114:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 1 v1) ==== 93+0+569 (secure 0 0 0) 0x7f7c0c0bcca0 con 0x7f7c18102040 2024-04-23T10:24:06.850 INFO:teuthology.orchestra.run.smithi114.stdout: 2024-04-23T10:24:06.850 INFO:teuthology.orchestra.run.smithi114.stdout:{"epoch":1,"default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[]} 2024-04-23T10:24:06.850 INFO:teuthology.orchestra.run.smithi114.stderr:dumped fsmap epoch 1 2024-04-23T10:24:06.852 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:24:06.853+0000 7f7c1e108640 1 -- 172.21.15.114:0/1441704618 >> [v2:172.21.15.114:6800/465333808,v1:172.21.15.114:6801/465333808] conn(0x7f7c000a3c30 msgr2=0x7f7c000a60f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T10:24:06.852 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:24:06.853+0000 7f7c1e108640 1 --2- 172.21.15.114:0/1441704618 >> [v2:172.21.15.114:6800/465333808,v1:172.21.15.114:6801/465333808] conn(0x7f7c000a3c30 0x7f7c000a60f0 secure :-1 s=READY pgs=132 cs=0 l=1 rev1=1 crypto rx=0x7f7c080058c0 tx=0x7f7c08005850 comp rx=0 tx=0).stop 2024-04-23T10:24:06.852 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:24:06.853+0000 7f7c1e108640 1 -- 172.21.15.114:0/1441704618 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f7c18102040 msgr2=0x7f7c1819cf20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T10:24:06.852 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:24:06.853+0000 7f7c1e108640 1 --2- 172.21.15.114:0/1441704618 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f7c18102040 0x7f7c1819cf20 secure :-1 s=READY pgs=162 cs=0 l=1 rev1=1 crypto rx=0x7f7c0c005940 tx=0x7f7c0c031a50 comp rx=0 tx=0).stop 2024-04-23T10:24:06.852 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:24:06.853+0000 7f7c1e108640 1 -- 172.21.15.114:0/1441704618 shutdown_connections 2024-04-23T10:24:06.852 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:24:06.853+0000 7f7c1e108640 1 --2- 172.21.15.114:0/1441704618 >> [v2:172.21.15.114:6800/465333808,v1:172.21.15.114:6801/465333808] conn(0x7f7c000a3c30 0x7f7c000a60f0 unknown :-1 s=CLOSED pgs=132 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:24:06.853 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:24:06.853+0000 7f7c1e108640 1 --2- 172.21.15.114:0/1441704618 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f7c18102a00 0x7f7c1819f540 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:24:06.853 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:24:06.853+0000 7f7c1e108640 1 --2- 172.21.15.114:0/1441704618 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f7c18102040 0x7f7c1819cf20 unknown :-1 s=CLOSED pgs=162 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:24:06.853 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:24:06.853+0000 7f7c1e108640 1 --2- 172.21.15.114:0/1441704618 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f7c18100e60 0x7f7c1806b100 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:24:06.853 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:24:06.853+0000 7f7c1e108640 1 -- 172.21.15.114:0/1441704618 >> 172.21.15.114:0/1441704618 conn(0x7f7c180fc440 msgr2=0x7f7c181053d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T10:24:06.853 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:24:06.853+0000 7f7c1e108640 1 -- 172.21.15.114:0/1441704618 shutdown_connections 2024-04-23T10:24:06.853 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:24:06.853+0000 7f7c1e108640 1 -- 172.21.15.114:0/1441704618 wait complete. 2024-04-23T10:24:07.078 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 23 10:24:07 smithi204 ceph-mon[32649]: Deploying daemon mds.l on smithi204 2024-04-23T10:24:07.078 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 23 10:24:07 smithi204 ceph-mon[32649]: pgmap v263: 1 pgs: 1 active+clean; 577 KiB data, 365 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-23T10:24:07.078 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 23 10:24:07 smithi204 ceph-mon[32649]: from='client.? 172.21.15.114:0/2473285021' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-23T10:24:07.078 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 23 10:24:07 smithi204 ceph-mon[32649]: from='client.? 172.21.15.114:0/704862242' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-04-23T10:24:07.078 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 23 10:24:07 smithi204 ceph-mon[32649]: from='client.? 172.21.15.114:0/1441704618' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-23T10:24:07.254 INFO:journalctl@ceph.mon.a.smithi114.stdout:Apr 23 10:24:07 smithi114 ceph-mon[29255]: Deploying daemon mds.l on smithi204 2024-04-23T10:24:07.254 INFO:journalctl@ceph.mon.a.smithi114.stdout:Apr 23 10:24:07 smithi114 ceph-mon[29255]: pgmap v263: 1 pgs: 1 active+clean; 577 KiB data, 365 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-23T10:24:07.254 INFO:journalctl@ceph.mon.a.smithi114.stdout:Apr 23 10:24:07 smithi114 ceph-mon[29255]: from='client.? 172.21.15.114:0/2473285021' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-23T10:24:07.254 INFO:journalctl@ceph.mon.a.smithi114.stdout:Apr 23 10:24:07 smithi114 ceph-mon[29255]: from='client.? 172.21.15.114:0/704862242' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-04-23T10:24:07.254 INFO:journalctl@ceph.mon.a.smithi114.stdout:Apr 23 10:24:07 smithi114 ceph-mon[29255]: from='client.? 172.21.15.114:0/1441704618' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-23T10:24:07.492 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 23 10:24:07 smithi119 ceph-mon[32608]: Deploying daemon mds.l on smithi204 2024-04-23T10:24:07.492 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 23 10:24:07 smithi119 ceph-mon[32608]: pgmap v263: 1 pgs: 1 active+clean; 577 KiB data, 365 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-23T10:24:07.492 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 23 10:24:07 smithi119 ceph-mon[32608]: from='client.? 172.21.15.114:0/2473285021' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-23T10:24:07.492 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 23 10:24:07 smithi119 ceph-mon[32608]: from='client.? 172.21.15.114:0/704862242' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-04-23T10:24:07.492 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 23 10:24:07 smithi119 ceph-mon[32608]: from='client.? 172.21.15.114:0/1441704618' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-23T10:24:07.792 INFO:journalctl@ceph.mds.l.smithi204.stdout:Apr 23 10:24:07 smithi204 systemd[1]: Starting Ceph mds.l for aa240572-0159-11ef-bc93-c7b262605968... 2024-04-23T10:24:08.048 INFO:journalctl@ceph.mds.l.smithi204.stdout:Apr 23 10:24:07 smithi204 podman[62356]: 2024-04-23 10:24:07.696633141 +0000 UTC m=+0.018892073 image pull da4b004c09e820713483d039bc76fbd660be678f2ea3c03e059a6161b74dab76 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:068a558e3af4932cf1c60dd321c0bdf500075eb52ea1bf455e0973ba114fbfd4 2024-04-23T10:24:08.048 INFO:journalctl@ceph.mds.l.smithi204.stdout:Apr 23 10:24:07 smithi204 podman[62356]: 2024-04-23 10:24:07.804800083 +0000 UTC m=+0.127058988 container create ffce6fd328936cd910c50e0e68d0bb3ac574b03fff9108e0718d8090fdb12017 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:068a558e3af4932cf1c60dd321c0bdf500075eb52ea1bf455e0973ba114fbfd4, name=ceph-aa240572-0159-11ef-bc93-c7b262605968-mds-l, org.label-schema.build-date=20240416, org.label-schema.vendor=CentOS, GIT_CLEAN=True, org.label-schema.license=GPLv2, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD, io.buildah.version=1.33.5, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=wip-lusov-quiescer-d69f13d, org.label-schema.name=CentOS Stream 8 Base Image, CEPH_POINT_RELEASE=, org.label-schema.schema-version=1.0, ceph=True) 2024-04-23T10:24:08.048 INFO:journalctl@ceph.mds.l.smithi204.stdout:Apr 23 10:24:08 smithi204 podman[62356]: 2024-04-23 10:24:08.049432702 +0000 UTC m=+0.371691608 container init ffce6fd328936cd910c50e0e68d0bb3ac574b03fff9108e0718d8090fdb12017 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:068a558e3af4932cf1c60dd321c0bdf500075eb52ea1bf455e0973ba114fbfd4, name=ceph-aa240572-0159-11ef-bc93-c7b262605968-mds-l, RELEASE=wip-lusov-quiescer-d69f13d, io.buildah.version=1.33.5, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2, org.label-schema.build-date=20240416, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.schema-version=1.0, GIT_CLEAN=True, ceph=True) 2024-04-23T10:24:08.328 INFO:journalctl@ceph.mds.l.smithi204.stdout:Apr 23 10:24:08 smithi204 podman[62356]: 2024-04-23 10:24:08.052114536 +0000 UTC m=+0.374373450 container start ffce6fd328936cd910c50e0e68d0bb3ac574b03fff9108e0718d8090fdb12017 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:068a558e3af4932cf1c60dd321c0bdf500075eb52ea1bf455e0973ba114fbfd4, name=ceph-aa240572-0159-11ef-bc93-c7b262605968-mds-l, io.buildah.version=1.33.5, ceph=True, org.label-schema.name=CentOS Stream 8 Base Image, RELEASE=wip-lusov-quiescer-d69f13d, org.label-schema.schema-version=1.0, org.label-schema.build-date=20240416, maintainer=Guillaume Abrioux , GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, GIT_CLEAN=True) 2024-04-23T10:24:08.328 INFO:journalctl@ceph.mds.l.smithi204.stdout:Apr 23 10:24:08 smithi204 ceph-mds[62376]: -- 172.21.15.204:0/4287628177 <== mon.1 v2:172.21.15.204:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 194+0+0 (secure 0 0 0) 0x5623dbfd5a40 con 0x5623dbfa8800 2024-04-23T10:24:08.328 INFO:journalctl@ceph.mds.l.smithi204.stdout:Apr 23 10:24:08 smithi204 ceph-aa240572-0159-11ef-bc93-c7b262605968-mds-l[62372]: starting mds.l at 2024-04-23T10:24:08.328 INFO:journalctl@ceph.mds.l.smithi204.stdout:Apr 23 10:24:08 smithi204 bash[62356]: ffce6fd328936cd910c50e0e68d0bb3ac574b03fff9108e0718d8090fdb12017 2024-04-23T10:24:08.328 INFO:journalctl@ceph.mds.l.smithi204.stdout:Apr 23 10:24:08 smithi204 systemd[1]: Started Ceph mds.l for aa240572-0159-11ef-bc93-c7b262605968. 2024-04-23T10:24:08.864 DEBUG:teuthology.orchestra.run.smithi114:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-23T10:24:08.976 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:24:08.975+0000 7f6954320640 1 -- 172.21.15.114:0/1044936615 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f694c1029b0 msgr2=0x7f694c1092d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T10:24:08.977 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:24:08.975+0000 7f6954320640 1 --2- 172.21.15.114:0/1044936615 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f694c1029b0 0x7f694c1092d0 secure :-1 s=READY pgs=163 cs=0 l=1 rev1=1 crypto rx=0x7f6944009ca0 tx=0x7f694402a8c0 comp rx=0 tx=0).stop 2024-04-23T10:24:08.977 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:24:08.975+0000 7f6954320640 1 -- 172.21.15.114:0/1044936615 shutdown_connections 2024-04-23T10:24:08.977 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:24:08.975+0000 7f6954320640 1 --2- 172.21.15.114:0/1044936615 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f694c1029b0 0x7f694c1092d0 unknown :-1 s=CLOSED pgs=163 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:24:08.977 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:24:08.975+0000 7f6954320640 1 --2- 172.21.15.114:0/1044936615 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f694c101ff0 0x7f694c102470 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:24:08.977 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:24:08.975+0000 7f6954320640 1 --2- 172.21.15.114:0/1044936615 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f694c100ea0 0x7f694c1012c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:24:08.977 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:24:08.975+0000 7f6954320640 1 -- 172.21.15.114:0/1044936615 >> 172.21.15.114:0/1044936615 conn(0x7f694c0fc430 msgr2=0x7f694c0fe850 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T10:24:08.977 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:24:08.975+0000 7f6954320640 1 -- 172.21.15.114:0/1044936615 shutdown_connections 2024-04-23T10:24:08.977 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:24:08.975+0000 7f6954320640 1 -- 172.21.15.114:0/1044936615 wait complete. 2024-04-23T10:24:08.977 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:24:08.976+0000 7f6954320640 1 Processor -- start 2024-04-23T10:24:08.977 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:24:08.976+0000 7f6954320640 1 -- start start 2024-04-23T10:24:08.977 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:24:08.976+0000 7f6954320640 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f694c100ea0 0x7f694c06b120 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T10:24:08.977 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:24:08.976+0000 7f6954320640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f694c101ff0 0x7f694c19cf00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T10:24:08.977 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:24:08.976+0000 7f6954320640 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f694c1029b0 0x7f694c19f520 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T10:24:08.977 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:24:08.976+0000 7f6954320640 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x7f694c10ca00 con 0x7f694c101ff0 2024-04-23T10:24:08.978 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:24:08.976+0000 7f6954320640 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 -- 0x7f694c10c880 con 0x7f694c100ea0 2024-04-23T10:24:08.978 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:24:08.976+0000 7f6954320640 1 -- --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_getmap magic: 0 -- 0x7f694c10cb80 con 0x7f694c1029b0 2024-04-23T10:24:08.978 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:24:08.976+0000 7f69518bb640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f694c101ff0 0x7f694c19cf00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T10:24:08.978 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:24:08.976+0000 7f69518bb640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f694c101ff0 0x7f694c19cf00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.114:38970/0 (socket says 172.21.15.114:38970) 2024-04-23T10:24:08.978 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:24:08.976+0000 7f69518bb640 1 -- 172.21.15.114:0/250891510 learned_addr learned my addr 172.21.15.114:0/250891510 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-04-23T10:24:08.978 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:24:08.976+0000 7f69520bc640 1 --2- 172.21.15.114:0/250891510 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f694c100ea0 0x7f694c06b120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T10:24:08.978 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:24:08.976+0000 7f69528bd640 1 --2- 172.21.15.114:0/250891510 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f694c1029b0 0x7f694c19f520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T10:24:08.978 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:24:08.976+0000 7f69518bb640 1 -- 172.21.15.114:0/250891510 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f694c1029b0 msgr2=0x7f694c19f520 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T10:24:08.978 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:24:08.976+0000 7f69518bb640 1 --2- 172.21.15.114:0/250891510 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f694c1029b0 0x7f694c19f520 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:24:08.978 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:24:08.976+0000 7f69518bb640 1 -- 172.21.15.114:0/250891510 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f694c100ea0 msgr2=0x7f694c06b120 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T10:24:08.978 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:24:08.976+0000 7f69518bb640 1 --2- 172.21.15.114:0/250891510 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f694c100ea0 0x7f694c06b120 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:24:08.978 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:24:08.977+0000 7f69518bb640 1 -- 172.21.15.114:0/250891510 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f694c19fe30 con 0x7f694c101ff0 2024-04-23T10:24:08.978 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:24:08.977+0000 7f69528bd640 1 --2- 172.21.15.114:0/250891510 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f694c1029b0 0x7f694c19f520 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-23T10:24:08.978 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:24:08.977+0000 7f69518bb640 1 --2- 172.21.15.114:0/250891510 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f694c101ff0 0x7f694c19cf00 secure :-1 s=READY pgs=164 cs=0 l=1 rev1=1 crypto rx=0x7f693c00e9c0 tx=0x7f693c00ee80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T10:24:08.979 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:24:08.977+0000 7f69437fe640 1 -- 172.21.15.114:0/250891510 <== mon.0 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f693c00cde0 con 0x7f694c101ff0 2024-04-23T10:24:08.979 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:24:08.977+0000 7f6954320640 1 -- 172.21.15.114:0/250891510 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f694c1a00c0 con 0x7f694c101ff0 2024-04-23T10:24:08.979 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:24:08.977+0000 7f69437fe640 1 -- 172.21.15.114:0/250891510 <== mon.0 v2:172.21.15.114:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f693c0108e0 con 0x7f694c101ff0 2024-04-23T10:24:08.979 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:24:08.977+0000 7f69437fe640 1 -- 172.21.15.114:0/250891510 <== mon.0 v2:172.21.15.114:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f693c0188d0 con 0x7f694c101ff0 2024-04-23T10:24:08.979 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:24:08.977+0000 7f6954320640 1 -- 172.21.15.114:0/250891510 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f694c1a0610 con 0x7f694c101ff0 2024-04-23T10:24:08.980 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:24:08.978+0000 7f6954320640 1 -- 172.21.15.114:0/250891510 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f694c108280 con 0x7f694c101ff0 2024-04-23T10:24:08.980 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:24:08.979+0000 7f69437fe640 1 -- 172.21.15.114:0/250891510 <== mon.0 v2:172.21.15.114:3300/0 4 ==== mgrmap(e 16) ==== 138647+0+0 (secure 0 0 0) 0x7f693c01f020 con 0x7f694c101ff0 2024-04-23T10:24:08.980 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:24:08.979+0000 7f69437fe640 1 --2- 172.21.15.114:0/250891510 >> [v2:172.21.15.114:6800/465333808,v1:172.21.15.114:6801/465333808] conn(0x7f69380a3bb0 0x7f69380a6070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T10:24:08.980 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:24:08.980+0000 7f69437fe640 1 -- 172.21.15.114:0/250891510 <== mon.0 v2:172.21.15.114:3300/0 5 ==== osd_map(83..83 src has 1..83) ==== 7127+0+0 (secure 0 0 0) 0x7f693c0d0160 con 0x7f694c101ff0 2024-04-23T10:24:08.980 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:24:08.980+0000 7f69520bc640 1 --2- 172.21.15.114:0/250891510 >> [v2:172.21.15.114:6800/465333808,v1:172.21.15.114:6801/465333808] conn(0x7f69380a3bb0 0x7f69380a6070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T10:24:08.980 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:24:08.980+0000 7f69520bc640 1 --2- 172.21.15.114:0/250891510 >> [v2:172.21.15.114:6800/465333808,v1:172.21.15.114:6801/465333808] conn(0x7f69380a3bb0 0x7f69380a6070 secure :-1 s=READY pgs=134 cs=0 l=1 rev1=1 crypto rx=0x7f6948005fd0 tx=0x7f694800b390 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T10:24:08.983 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:24:08.984+0000 7f69437fe640 1 -- 172.21.15.114:0/250891510 <== mon.0 v2:172.21.15.114:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f693c09cfe0 con 0x7f694c101ff0 2024-04-23T10:24:09.174 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 23 10:24:09 smithi119 ceph-mon[32608]: pgmap v264: 1 pgs: 1 active+clean; 577 KiB data, 365 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-23T10:24:09.175 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 23 10:24:09 smithi119 ceph-mon[32608]: mds.? [v2:172.21.15.204:6832/3866561754,v1:172.21.15.204:6833/3866561754] up:boot 2024-04-23T10:24:09.175 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 23 10:24:09 smithi119 ceph-mon[32608]: fsmap 1 up:standby 2024-04-23T10:24:09.175 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 23 10:24:09 smithi119 ceph-mon[32608]: from='mgr.14152 172.21.15.114:0/1529706138' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "l"}]: dispatch 2024-04-23T10:24:09.175 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 23 10:24:09 smithi119 ceph-mon[32608]: from='mgr.14152 172.21.15.114:0/1529706138' entity='mgr.x' 2024-04-23T10:24:09.175 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 23 10:24:09 smithi119 ceph-mon[32608]: from='mgr.14152 172.21.15.114:0/1529706138' entity='mgr.x' 2024-04-23T10:24:09.175 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 23 10:24:09 smithi119 ceph-mon[32608]: from='mgr.14152 172.21.15.114:0/1529706138' entity='mgr.x' 2024-04-23T10:24:09.175 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 23 10:24:09 smithi119 ceph-mon[32608]: from='mgr.14152 172.21.15.114:0/1529706138' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.k", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-04-23T10:24:09.175 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 23 10:24:09 smithi119 ceph-mon[32608]: from='mgr.14152 172.21.15.114:0/1529706138' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.k", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-04-23T10:24:09.175 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 23 10:24:09 smithi119 ceph-mon[32608]: from='mgr.14152 172.21.15.114:0/1529706138' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-23T10:24:09.202 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:24:09.203+0000 7f6954320640 1 -- 172.21.15.114:0/250891510 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f694c02d790 con 0x7f694c101ff0 2024-04-23T10:24:09.202 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:24:09.203+0000 7f69437fe640 1 -- 172.21.15.114:0/250891510 <== mon.0 v2:172.21.15.114:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 2 v2) ==== 93+0+1396 (secure 0 0 0) 0x7f693c09c8e0 con 0x7f694c101ff0 2024-04-23T10:24:09.202 INFO:teuthology.orchestra.run.smithi114.stdout: 2024-04-23T10:24:09.202 INFO:teuthology.orchestra.run.smithi114.stdout:{"epoch":2,"default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":24385,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.204:6833/3866561754","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6832","nonce":3866561754},{"type":"v1","addr":"172.21.15.204:6833","nonce":3866561754}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2}],"filesystems":[]} 2024-04-23T10:24:09.203 INFO:teuthology.orchestra.run.smithi114.stderr:dumped fsmap epoch 2 2024-04-23T10:24:09.204 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:24:09.205+0000 7f6954320640 1 -- 172.21.15.114:0/250891510 >> [v2:172.21.15.114:6800/465333808,v1:172.21.15.114:6801/465333808] conn(0x7f69380a3bb0 msgr2=0x7f69380a6070 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T10:24:09.204 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:24:09.205+0000 7f6954320640 1 --2- 172.21.15.114:0/250891510 >> [v2:172.21.15.114:6800/465333808,v1:172.21.15.114:6801/465333808] conn(0x7f69380a3bb0 0x7f69380a6070 secure :-1 s=READY pgs=134 cs=0 l=1 rev1=1 crypto rx=0x7f6948005fd0 tx=0x7f694800b390 comp rx=0 tx=0).stop 2024-04-23T10:24:09.205 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:24:09.205+0000 7f6954320640 1 -- 172.21.15.114:0/250891510 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f694c101ff0 msgr2=0x7f694c19cf00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T10:24:09.205 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:24:09.205+0000 7f6954320640 1 --2- 172.21.15.114:0/250891510 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f694c101ff0 0x7f694c19cf00 secure :-1 s=READY pgs=164 cs=0 l=1 rev1=1 crypto rx=0x7f693c00e9c0 tx=0x7f693c00ee80 comp rx=0 tx=0).stop 2024-04-23T10:24:09.205 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:24:09.205+0000 7f6954320640 1 -- 172.21.15.114:0/250891510 shutdown_connections 2024-04-23T10:24:09.205 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:24:09.205+0000 7f6954320640 1 --2- 172.21.15.114:0/250891510 >> [v2:172.21.15.114:6800/465333808,v1:172.21.15.114:6801/465333808] conn(0x7f69380a3bb0 0x7f69380a6070 unknown :-1 s=CLOSED pgs=134 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:24:09.205 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:24:09.205+0000 7f6954320640 1 --2- 172.21.15.114:0/250891510 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f694c1029b0 0x7f694c19f520 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:24:09.205 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:24:09.205+0000 7f6954320640 1 --2- 172.21.15.114:0/250891510 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f694c101ff0 0x7f694c19cf00 unknown :-1 s=CLOSED pgs=164 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:24:09.205 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:24:09.205+0000 7f6954320640 1 --2- 172.21.15.114:0/250891510 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f694c100ea0 0x7f694c06b120 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:24:09.205 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:24:09.205+0000 7f6954320640 1 -- 172.21.15.114:0/250891510 >> 172.21.15.114:0/250891510 conn(0x7f694c0fc430 msgr2=0x7f694c105450 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T10:24:09.205 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:24:09.205+0000 7f6954320640 1 -- 172.21.15.114:0/250891510 shutdown_connections 2024-04-23T10:24:09.205 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:24:09.205+0000 7f6954320640 1 -- 172.21.15.114:0/250891510 wait complete. 2024-04-23T10:24:09.214 INFO:journalctl@ceph.mon.a.smithi114.stdout:Apr 23 10:24:09 smithi114 ceph-mon[29255]: pgmap v264: 1 pgs: 1 active+clean; 577 KiB data, 365 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-23T10:24:09.214 INFO:journalctl@ceph.mon.a.smithi114.stdout:Apr 23 10:24:09 smithi114 ceph-mon[29255]: mds.? [v2:172.21.15.204:6832/3866561754,v1:172.21.15.204:6833/3866561754] up:boot 2024-04-23T10:24:09.214 INFO:journalctl@ceph.mon.a.smithi114.stdout:Apr 23 10:24:09 smithi114 ceph-mon[29255]: fsmap 1 up:standby 2024-04-23T10:24:09.215 INFO:journalctl@ceph.mon.a.smithi114.stdout:Apr 23 10:24:09 smithi114 ceph-mon[29255]: from='mgr.14152 172.21.15.114:0/1529706138' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "l"}]: dispatch 2024-04-23T10:24:09.215 INFO:journalctl@ceph.mon.a.smithi114.stdout:Apr 23 10:24:09 smithi114 ceph-mon[29255]: from='mgr.14152 172.21.15.114:0/1529706138' entity='mgr.x' 2024-04-23T10:24:09.215 INFO:journalctl@ceph.mon.a.smithi114.stdout:Apr 23 10:24:09 smithi114 ceph-mon[29255]: from='mgr.14152 172.21.15.114:0/1529706138' entity='mgr.x' 2024-04-23T10:24:09.215 INFO:journalctl@ceph.mon.a.smithi114.stdout:Apr 23 10:24:09 smithi114 ceph-mon[29255]: from='mgr.14152 172.21.15.114:0/1529706138' entity='mgr.x' 2024-04-23T10:24:09.215 INFO:journalctl@ceph.mon.a.smithi114.stdout:Apr 23 10:24:09 smithi114 ceph-mon[29255]: from='mgr.14152 172.21.15.114:0/1529706138' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.k", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-04-23T10:24:09.215 INFO:journalctl@ceph.mon.a.smithi114.stdout:Apr 23 10:24:09 smithi114 ceph-mon[29255]: from='mgr.14152 172.21.15.114:0/1529706138' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.k", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-04-23T10:24:09.215 INFO:journalctl@ceph.mon.a.smithi114.stdout:Apr 23 10:24:09 smithi114 ceph-mon[29255]: from='mgr.14152 172.21.15.114:0/1529706138' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-23T10:24:09.578 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 23 10:24:09 smithi204 ceph-mon[32649]: pgmap v264: 1 pgs: 1 active+clean; 577 KiB data, 365 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-23T10:24:09.578 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 23 10:24:09 smithi204 ceph-mon[32649]: mds.? [v2:172.21.15.204:6832/3866561754,v1:172.21.15.204:6833/3866561754] up:boot 2024-04-23T10:24:09.578 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 23 10:24:09 smithi204 ceph-mon[32649]: fsmap 1 up:standby 2024-04-23T10:24:09.578 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 23 10:24:09 smithi204 ceph-mon[32649]: from='mgr.14152 172.21.15.114:0/1529706138' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "l"}]: dispatch 2024-04-23T10:24:09.578 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 23 10:24:09 smithi204 ceph-mon[32649]: from='mgr.14152 172.21.15.114:0/1529706138' entity='mgr.x' 2024-04-23T10:24:09.578 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 23 10:24:09 smithi204 ceph-mon[32649]: from='mgr.14152 172.21.15.114:0/1529706138' entity='mgr.x' 2024-04-23T10:24:09.578 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 23 10:24:09 smithi204 ceph-mon[32649]: from='mgr.14152 172.21.15.114:0/1529706138' entity='mgr.x' 2024-04-23T10:24:09.579 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 23 10:24:09 smithi204 ceph-mon[32649]: from='mgr.14152 172.21.15.114:0/1529706138' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.k", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-04-23T10:24:09.579 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 23 10:24:09 smithi204 ceph-mon[32649]: from='mgr.14152 172.21.15.114:0/1529706138' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.k", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-04-23T10:24:09.579 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 23 10:24:09 smithi204 ceph-mon[32649]: from='mgr.14152 172.21.15.114:0/1529706138' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-23T10:24:10.191 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 23 10:24:10 smithi119 ceph-mon[32608]: Deploying daemon mds.k on smithi119 2024-04-23T10:24:10.191 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 23 10:24:10 smithi119 ceph-mon[32608]: from='client.? 172.21.15.114:0/250891510' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-23T10:24:10.492 INFO:journalctl@ceph.mds.k.smithi119.stdout:Apr 23 10:24:10 smithi119 systemd[1]: Starting Ceph mds.k for aa240572-0159-11ef-bc93-c7b262605968... 2024-04-23T10:24:10.504 INFO:journalctl@ceph.mon.a.smithi114.stdout:Apr 23 10:24:10 smithi114 ceph-mon[29255]: Deploying daemon mds.k on smithi119 2024-04-23T10:24:10.504 INFO:journalctl@ceph.mon.a.smithi114.stdout:Apr 23 10:24:10 smithi114 ceph-mon[29255]: from='client.? 172.21.15.114:0/250891510' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-23T10:24:10.578 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 23 10:24:10 smithi204 ceph-mon[32649]: Deploying daemon mds.k on smithi119 2024-04-23T10:24:10.578 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 23 10:24:10 smithi204 ceph-mon[32649]: from='client.? 172.21.15.114:0/250891510' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-23T10:24:10.884 INFO:journalctl@ceph.mds.k.smithi119.stdout:Apr 23 10:24:10 smithi119 podman[61730]: 2024-04-23 10:24:10.490316676 +0000 UTC m=+0.018049018 image pull da4b004c09e820713483d039bc76fbd660be678f2ea3c03e059a6161b74dab76 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:068a558e3af4932cf1c60dd321c0bdf500075eb52ea1bf455e0973ba114fbfd4 2024-04-23T10:24:10.884 INFO:journalctl@ceph.mds.k.smithi119.stdout:Apr 23 10:24:10 smithi119 podman[61730]: 2024-04-23 10:24:10.607878243 +0000 UTC m=+0.135610570 container create ef9a53e050a9ec51cd1080c4cb18e56466bd97e8d997e444ca4d15b372981bcd (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:068a558e3af4932cf1c60dd321c0bdf500075eb52ea1bf455e0973ba114fbfd4, name=ceph-aa240572-0159-11ef-bc93-c7b262605968-mds-k, GIT_BRANCH=HEAD, org.label-schema.license=GPLv2, GIT_CLEAN=True, io.buildah.version=1.33.5, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240416, org.label-schema.vendor=CentOS, org.label-schema.name=CentOS Stream 8 Base Image, CEPH_POINT_RELEASE=, RELEASE=wip-lusov-quiescer-d69f13d, maintainer=Guillaume Abrioux , ceph=True, org.label-schema.schema-version=1.0) 2024-04-23T10:24:10.884 INFO:journalctl@ceph.mds.k.smithi119.stdout:Apr 23 10:24:10 smithi119 podman[61730]: 2024-04-23 10:24:10.885551197 +0000 UTC m=+0.413283525 container init ef9a53e050a9ec51cd1080c4cb18e56466bd97e8d997e444ca4d15b372981bcd (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:068a558e3af4932cf1c60dd321c0bdf500075eb52ea1bf455e0973ba114fbfd4, name=ceph-aa240572-0159-11ef-bc93-c7b262605968-mds-k, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, RELEASE=wip-lusov-quiescer-d69f13d, maintainer=Guillaume Abrioux , ceph=True, GIT_CLEAN=True, org.label-schema.build-date=20240416, org.label-schema.schema-version=1.0, io.buildah.version=1.33.5, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-04-23T10:24:11.217 DEBUG:teuthology.orchestra.run.smithi114:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-23T10:24:11.242 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 23 10:24:11 smithi119 ceph-mon[32608]: pgmap v265: 1 pgs: 1 active+clean; 577 KiB data, 365 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-23T10:24:11.242 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 23 10:24:11 smithi119 ceph-mon[32608]: from='mgr.14152 172.21.15.114:0/1529706138' entity='mgr.x' 2024-04-23T10:24:11.242 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 23 10:24:11 smithi119 ceph-mon[32608]: from='mgr.14152 172.21.15.114:0/1529706138' entity='mgr.x' 2024-04-23T10:24:11.242 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 23 10:24:11 smithi119 ceph-mon[32608]: from='mgr.14152 172.21.15.114:0/1529706138' entity='mgr.x' 2024-04-23T10:24:11.242 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 23 10:24:11 smithi119 ceph-mon[32608]: from='mgr.14152 172.21.15.114:0/1529706138' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.j", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-04-23T10:24:11.242 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 23 10:24:11 smithi119 ceph-mon[32608]: from='mgr.14152 172.21.15.114:0/1529706138' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.j", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-04-23T10:24:11.242 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 23 10:24:11 smithi119 ceph-mon[32608]: from='mgr.14152 172.21.15.114:0/1529706138' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-23T10:24:11.242 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 23 10:24:11 smithi119 ceph-mon[32608]: Deploying daemon mds.j on smithi114 2024-04-23T10:24:11.243 INFO:journalctl@ceph.mds.k.smithi119.stdout:Apr 23 10:24:10 smithi119 podman[61730]: 2024-04-23 10:24:10.888160614 +0000 UTC m=+0.415892941 container start ef9a53e050a9ec51cd1080c4cb18e56466bd97e8d997e444ca4d15b372981bcd (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:068a558e3af4932cf1c60dd321c0bdf500075eb52ea1bf455e0973ba114fbfd4, name=ceph-aa240572-0159-11ef-bc93-c7b262605968-mds-k, org.label-schema.build-date=20240416, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, ceph=True, GIT_CLEAN=True, CEPH_POINT_RELEASE=, io.buildah.version=1.33.5, RELEASE=wip-lusov-quiescer-d69f13d, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.license=GPLv2, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, maintainer=Guillaume Abrioux ) 2024-04-23T10:24:11.243 INFO:journalctl@ceph.mds.k.smithi119.stdout:Apr 23 10:24:10 smithi119 ceph-mds[61750]: -- 172.21.15.119:0/335132777 <== mon.2 v2:172.21.15.119:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 194+0+0 (secure 0 0 0) 0x5611764cba40 con 0x56117649ec00 2024-04-23T10:24:11.243 INFO:journalctl@ceph.mds.k.smithi119.stdout:Apr 23 10:24:10 smithi119 ceph-aa240572-0159-11ef-bc93-c7b262605968-mds-k[61746]: starting mds.k at 2024-04-23T10:24:11.243 INFO:journalctl@ceph.mds.k.smithi119.stdout:Apr 23 10:24:10 smithi119 bash[61730]: ef9a53e050a9ec51cd1080c4cb18e56466bd97e8d997e444ca4d15b372981bcd 2024-04-23T10:24:11.243 INFO:journalctl@ceph.mds.k.smithi119.stdout:Apr 23 10:24:10 smithi119 systemd[1]: Started Ceph mds.k for aa240572-0159-11ef-bc93-c7b262605968. 2024-04-23T10:24:11.317 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:24:11.318+0000 7f5b1cb68640 1 -- 172.21.15.114:0/2906891065 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f5b181053e0 msgr2=0x7f5b181077d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T10:24:11.317 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:24:11.318+0000 7f5b1cb68640 1 --2- 172.21.15.114:0/2906891065 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f5b181053e0 0x7f5b181077d0 secure :-1 s=READY pgs=165 cs=0 l=1 rev1=1 crypto rx=0x7f5b0c009900 tx=0x7f5b0c02a410 comp rx=0 tx=0).stop 2024-04-23T10:24:11.317 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:24:11.318+0000 7f5b1cb68640 1 -- 172.21.15.114:0/2906891065 shutdown_connections 2024-04-23T10:24:11.317 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:24:11.318+0000 7f5b1cb68640 1 --2- 172.21.15.114:0/2906891065 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f5b18107d10 0x7f5b1810a100 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:24:11.317 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:24:11.318+0000 7f5b1cb68640 1 --2- 172.21.15.114:0/2906891065 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f5b181053e0 0x7f5b181077d0 unknown :-1 s=CLOSED pgs=165 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:24:11.318 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:24:11.318+0000 7f5b1cb68640 1 --2- 172.21.15.114:0/2906891065 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f5b18102ab0 0x7f5b18104ea0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:24:11.318 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:24:11.318+0000 7f5b1cb68640 1 -- 172.21.15.114:0/2906891065 >> 172.21.15.114:0/2906891065 conn(0x7f5b180fc400 msgr2=0x7f5b180fe840 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T10:24:11.318 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:24:11.318+0000 7f5b1cb68640 1 -- 172.21.15.114:0/2906891065 shutdown_connections 2024-04-23T10:24:11.318 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:24:11.318+0000 7f5b1cb68640 1 -- 172.21.15.114:0/2906891065 wait complete. 2024-04-23T10:24:11.320 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:24:11.319+0000 7f5b1cb68640 1 Processor -- start 2024-04-23T10:24:11.320 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:24:11.319+0000 7f5b1cb68640 1 -- start start 2024-04-23T10:24:11.320 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:24:11.319+0000 7f5b1cb68640 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f5b18102ab0 0x7f5b181981b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T10:24:11.320 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:24:11.319+0000 7f5b1cb68640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f5b181053e0 0x7f5b181986f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T10:24:11.320 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:24:11.319+0000 7f5b1cb68640 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f5b18107d10 0x7f5b1819f760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T10:24:11.320 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:24:11.319+0000 7f5b1cb68640 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x7f5b1810daf0 con 0x7f5b181053e0 2024-04-23T10:24:11.321 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:24:11.319+0000 7f5b1cb68640 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 -- 0x7f5b1810d970 con 0x7f5b18107d10 2024-04-23T10:24:11.321 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:24:11.319+0000 7f5b1cb68640 1 -- --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_getmap magic: 0 -- 0x7f5b1810dc70 con 0x7f5b18102ab0 2024-04-23T10:24:11.321 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:24:11.319+0000 7f5b15d9b640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f5b181053e0 0x7f5b181986f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T10:24:11.321 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:24:11.319+0000 7f5b15d9b640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f5b181053e0 0x7f5b181986f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.114:38978/0 (socket says 172.21.15.114:38978) 2024-04-23T10:24:11.321 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:24:11.319+0000 7f5b15d9b640 1 -- 172.21.15.114:0/952472848 learned_addr learned my addr 172.21.15.114:0/952472848 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-04-23T10:24:11.321 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:24:11.319+0000 7f5b16d9d640 1 --2- 172.21.15.114:0/952472848 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f5b18107d10 0x7f5b1819f760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T10:24:11.321 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:24:11.319+0000 7f5b1659c640 1 --2- 172.21.15.114:0/952472848 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f5b18102ab0 0x7f5b181981b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T10:24:11.321 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:24:11.319+0000 7f5b15d9b640 1 -- 172.21.15.114:0/952472848 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f5b18102ab0 msgr2=0x7f5b181981b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T10:24:11.321 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:24:11.319+0000 7f5b15d9b640 1 --2- 172.21.15.114:0/952472848 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f5b18102ab0 0x7f5b181981b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:24:11.321 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:24:11.319+0000 7f5b15d9b640 1 -- 172.21.15.114:0/952472848 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f5b18107d10 msgr2=0x7f5b1819f760 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T10:24:11.321 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:24:11.319+0000 7f5b15d9b640 1 --2- 172.21.15.114:0/952472848 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f5b18107d10 0x7f5b1819f760 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:24:11.321 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:24:11.319+0000 7f5b15d9b640 1 -- 172.21.15.114:0/952472848 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f5b1819fdd0 con 0x7f5b181053e0 2024-04-23T10:24:11.321 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:24:11.319+0000 7f5b1659c640 1 --2- 172.21.15.114:0/952472848 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f5b18102ab0 0x7f5b181981b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-23T10:24:11.321 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:24:11.319+0000 7f5b16d9d640 1 --2- 172.21.15.114:0/952472848 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f5b18107d10 0x7f5b1819f760 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-23T10:24:11.321 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:24:11.320+0000 7f5b15d9b640 1 --2- 172.21.15.114:0/952472848 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f5b181053e0 0x7f5b181986f0 secure :-1 s=READY pgs=166 cs=0 l=1 rev1=1 crypto rx=0x7f5b0c006010 tx=0x7f5b0c004430 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T10:24:11.322 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:24:11.320+0000 7f5b077fe640 1 -- 172.21.15.114:0/952472848 <== mon.0 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5b0c03f030 con 0x7f5b181053e0 2024-04-23T10:24:11.322 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:24:11.320+0000 7f5b1cb68640 1 -- 172.21.15.114:0/952472848 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f5b181a0060 con 0x7f5b181053e0 2024-04-23T10:24:11.322 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:24:11.320+0000 7f5b077fe640 1 -- 172.21.15.114:0/952472848 <== mon.0 v2:172.21.15.114:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f5b0c003cb0 con 0x7f5b181053e0 2024-04-23T10:24:11.322 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:24:11.320+0000 7f5b077fe640 1 -- 172.21.15.114:0/952472848 <== mon.0 v2:172.21.15.114:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5b0c035070 con 0x7f5b181053e0 2024-04-23T10:24:11.322 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:24:11.320+0000 7f5b1cb68640 1 -- 172.21.15.114:0/952472848 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f5b181a0550 con 0x7f5b181053e0 2024-04-23T10:24:11.322 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:24:11.321+0000 7f5b1cb68640 1 -- 172.21.15.114:0/952472848 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f5b180fdef0 con 0x7f5b181053e0 2024-04-23T10:24:11.323 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:24:11.323+0000 7f5b077fe640 1 -- 172.21.15.114:0/952472848 <== mon.0 v2:172.21.15.114:3300/0 4 ==== mgrmap(e 16) ==== 138647+0+0 (secure 0 0 0) 0x7f5b0c042030 con 0x7f5b181053e0 2024-04-23T10:24:11.323 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:24:11.323+0000 7f5b077fe640 1 --2- 172.21.15.114:0/952472848 >> [v2:172.21.15.114:6800/465333808,v1:172.21.15.114:6801/465333808] conn(0x7f5b000a3bb0 0x7f5b000a6070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T10:24:11.323 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:24:11.323+0000 7f5b1659c640 1 --2- 172.21.15.114:0/952472848 >> [v2:172.21.15.114:6800/465333808,v1:172.21.15.114:6801/465333808] conn(0x7f5b000a3bb0 0x7f5b000a6070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T10:24:11.323 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:24:11.324+0000 7f5b077fe640 1 -- 172.21.15.114:0/952472848 <== mon.0 v2:172.21.15.114:3300/0 5 ==== osd_map(83..83 src has 1..83) ==== 7127+0+0 (secure 0 0 0) 0x7f5b0c0f0640 con 0x7f5b181053e0 2024-04-23T10:24:11.323 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:24:11.324+0000 7f5b1659c640 1 --2- 172.21.15.114:0/952472848 >> [v2:172.21.15.114:6800/465333808,v1:172.21.15.114:6801/465333808] conn(0x7f5b000a3bb0 0x7f5b000a6070 secure :-1 s=READY pgs=136 cs=0 l=1 rev1=1 crypto rx=0x7f5b08005fd0 tx=0x7f5b08005ea0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T10:24:11.327 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:24:11.328+0000 7f5b077fe640 1 -- 172.21.15.114:0/952472848 <== mon.0 v2:172.21.15.114:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f5b0c003e60 con 0x7f5b181053e0 2024-04-23T10:24:11.505 INFO:journalctl@ceph.mon.a.smithi114.stdout:Apr 23 10:24:11 smithi114 ceph-mon[29255]: pgmap v265: 1 pgs: 1 active+clean; 577 KiB data, 365 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-23T10:24:11.506 INFO:journalctl@ceph.mon.a.smithi114.stdout:Apr 23 10:24:11 smithi114 ceph-mon[29255]: from='mgr.14152 172.21.15.114:0/1529706138' entity='mgr.x' 2024-04-23T10:24:11.506 INFO:journalctl@ceph.mon.a.smithi114.stdout:Apr 23 10:24:11 smithi114 ceph-mon[29255]: from='mgr.14152 172.21.15.114:0/1529706138' entity='mgr.x' 2024-04-23T10:24:11.506 INFO:journalctl@ceph.mon.a.smithi114.stdout:Apr 23 10:24:11 smithi114 ceph-mon[29255]: from='mgr.14152 172.21.15.114:0/1529706138' entity='mgr.x' 2024-04-23T10:24:11.506 INFO:journalctl@ceph.mon.a.smithi114.stdout:Apr 23 10:24:11 smithi114 ceph-mon[29255]: from='mgr.14152 172.21.15.114:0/1529706138' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.j", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-04-23T10:24:11.506 INFO:journalctl@ceph.mon.a.smithi114.stdout:Apr 23 10:24:11 smithi114 ceph-mon[29255]: from='mgr.14152 172.21.15.114:0/1529706138' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.j", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-04-23T10:24:11.506 INFO:journalctl@ceph.mon.a.smithi114.stdout:Apr 23 10:24:11 smithi114 ceph-mon[29255]: from='mgr.14152 172.21.15.114:0/1529706138' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-23T10:24:11.506 INFO:journalctl@ceph.mon.a.smithi114.stdout:Apr 23 10:24:11 smithi114 ceph-mon[29255]: Deploying daemon mds.j on smithi114 2024-04-23T10:24:11.551 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:24:11.552+0000 7f5b1cb68640 1 -- 172.21.15.114:0/952472848 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f5b1802d5c0 con 0x7f5b181053e0 2024-04-23T10:24:11.552 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:24:11.553+0000 7f5b077fe640 1 -- 172.21.15.114:0/952472848 <== mon.0 v2:172.21.15.114:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 3 v3) ==== 93+0+2221 (secure 0 0 0) 0x7f5b0c0bd4c0 con 0x7f5b181053e0 2024-04-23T10:24:11.552 INFO:teuthology.orchestra.run.smithi114.stdout: 2024-04-23T10:24:11.552 INFO:teuthology.orchestra.run.smithi114.stdout:{"epoch":3,"default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":24385,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.204:6833/3866561754","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6832","nonce":3866561754},{"type":"v1","addr":"172.21.15.204:6833","nonce":3866561754}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2},{"gid":24443,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.119:6833/783244660","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6832","nonce":783244660},{"type":"v1","addr":"172.21.15.119:6833","nonce":783244660}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3}],"filesystems":[]} 2024-04-23T10:24:11.552 INFO:teuthology.orchestra.run.smithi114.stderr:dumped fsmap epoch 3 2024-04-23T10:24:11.554 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:24:11.555+0000 7f5b1cb68640 1 -- 172.21.15.114:0/952472848 >> [v2:172.21.15.114:6800/465333808,v1:172.21.15.114:6801/465333808] conn(0x7f5b000a3bb0 msgr2=0x7f5b000a6070 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T10:24:11.554 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:24:11.555+0000 7f5b1cb68640 1 --2- 172.21.15.114:0/952472848 >> [v2:172.21.15.114:6800/465333808,v1:172.21.15.114:6801/465333808] conn(0x7f5b000a3bb0 0x7f5b000a6070 secure :-1 s=READY pgs=136 cs=0 l=1 rev1=1 crypto rx=0x7f5b08005fd0 tx=0x7f5b08005ea0 comp rx=0 tx=0).stop 2024-04-23T10:24:11.554 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:24:11.555+0000 7f5b1cb68640 1 -- 172.21.15.114:0/952472848 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f5b181053e0 msgr2=0x7f5b181986f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T10:24:11.554 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:24:11.555+0000 7f5b1cb68640 1 --2- 172.21.15.114:0/952472848 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f5b181053e0 0x7f5b181986f0 secure :-1 s=READY pgs=166 cs=0 l=1 rev1=1 crypto rx=0x7f5b0c006010 tx=0x7f5b0c004430 comp rx=0 tx=0).stop 2024-04-23T10:24:11.554 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:24:11.555+0000 7f5b1cb68640 1 -- 172.21.15.114:0/952472848 shutdown_connections 2024-04-23T10:24:11.554 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:24:11.555+0000 7f5b1cb68640 1 --2- 172.21.15.114:0/952472848 >> [v2:172.21.15.114:6800/465333808,v1:172.21.15.114:6801/465333808] conn(0x7f5b000a3bb0 0x7f5b000a6070 unknown :-1 s=CLOSED pgs=136 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:24:11.554 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:24:11.555+0000 7f5b1cb68640 1 --2- 172.21.15.114:0/952472848 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f5b18107d10 0x7f5b1819f760 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:24:11.554 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:24:11.555+0000 7f5b1cb68640 1 --2- 172.21.15.114:0/952472848 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f5b181053e0 0x7f5b181986f0 unknown :-1 s=CLOSED pgs=166 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:24:11.555 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:24:11.555+0000 7f5b1cb68640 1 --2- 172.21.15.114:0/952472848 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f5b18102ab0 0x7f5b181981b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:24:11.555 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:24:11.555+0000 7f5b1cb68640 1 -- 172.21.15.114:0/952472848 >> 172.21.15.114:0/952472848 conn(0x7f5b180fc400 msgr2=0x7f5b181035b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T10:24:11.555 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:24:11.555+0000 7f5b1cb68640 1 -- 172.21.15.114:0/952472848 shutdown_connections 2024-04-23T10:24:11.555 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:24:11.555+0000 7f5b1cb68640 1 -- 172.21.15.114:0/952472848 wait complete. 2024-04-23T10:24:11.578 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 23 10:24:11 smithi204 ceph-mon[32649]: pgmap v265: 1 pgs: 1 active+clean; 577 KiB data, 365 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-23T10:24:11.578 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 23 10:24:11 smithi204 ceph-mon[32649]: from='mgr.14152 172.21.15.114:0/1529706138' entity='mgr.x' 2024-04-23T10:24:11.578 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 23 10:24:11 smithi204 ceph-mon[32649]: from='mgr.14152 172.21.15.114:0/1529706138' entity='mgr.x' 2024-04-23T10:24:11.578 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 23 10:24:11 smithi204 ceph-mon[32649]: from='mgr.14152 172.21.15.114:0/1529706138' entity='mgr.x' 2024-04-23T10:24:11.578 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 23 10:24:11 smithi204 ceph-mon[32649]: from='mgr.14152 172.21.15.114:0/1529706138' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.j", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-04-23T10:24:11.578 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 23 10:24:11 smithi204 ceph-mon[32649]: from='mgr.14152 172.21.15.114:0/1529706138' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.j", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-04-23T10:24:11.578 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 23 10:24:11 smithi204 ceph-mon[32649]: from='mgr.14152 172.21.15.114:0/1529706138' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-23T10:24:11.578 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 23 10:24:11 smithi204 ceph-mon[32649]: Deploying daemon mds.j on smithi114 2024-04-23T10:24:12.254 INFO:journalctl@ceph.mon.a.smithi114.stdout:Apr 23 10:24:12 smithi114 ceph-mon[29255]: mds.? [v2:172.21.15.119:6832/783244660,v1:172.21.15.119:6833/783244660] up:boot 2024-04-23T10:24:12.254 INFO:journalctl@ceph.mon.a.smithi114.stdout:Apr 23 10:24:12 smithi114 ceph-mon[29255]: fsmap 2 up:standby 2024-04-23T10:24:12.254 INFO:journalctl@ceph.mon.a.smithi114.stdout:Apr 23 10:24:12 smithi114 ceph-mon[29255]: from='mgr.14152 172.21.15.114:0/1529706138' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "k"}]: dispatch 2024-04-23T10:24:12.254 INFO:journalctl@ceph.mon.a.smithi114.stdout:Apr 23 10:24:12 smithi114 ceph-mon[29255]: from='client.? 172.21.15.114:0/952472848' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-23T10:24:12.254 INFO:journalctl@ceph.mon.a.smithi114.stdout:Apr 23 10:24:12 smithi114 ceph-mon[29255]: pgmap v266: 1 pgs: 1 active+clean; 577 KiB data, 365 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-23T10:24:12.302 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 23 10:24:12 smithi119 ceph-mon[32608]: mds.? [v2:172.21.15.119:6832/783244660,v1:172.21.15.119:6833/783244660] up:boot 2024-04-23T10:24:12.302 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 23 10:24:12 smithi119 ceph-mon[32608]: fsmap 2 up:standby 2024-04-23T10:24:12.302 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 23 10:24:12 smithi119 ceph-mon[32608]: from='mgr.14152 172.21.15.114:0/1529706138' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "k"}]: dispatch 2024-04-23T10:24:12.302 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 23 10:24:12 smithi119 ceph-mon[32608]: from='client.? 172.21.15.114:0/952472848' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-23T10:24:12.302 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 23 10:24:12 smithi119 ceph-mon[32608]: pgmap v266: 1 pgs: 1 active+clean; 577 KiB data, 365 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-23T10:24:12.578 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 23 10:24:12 smithi204 ceph-mon[32649]: mds.? [v2:172.21.15.119:6832/783244660,v1:172.21.15.119:6833/783244660] up:boot 2024-04-23T10:24:12.578 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 23 10:24:12 smithi204 ceph-mon[32649]: fsmap 2 up:standby 2024-04-23T10:24:12.578 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 23 10:24:12 smithi204 ceph-mon[32649]: from='mgr.14152 172.21.15.114:0/1529706138' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "k"}]: dispatch 2024-04-23T10:24:12.578 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 23 10:24:12 smithi204 ceph-mon[32649]: from='client.? 172.21.15.114:0/952472848' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-23T10:24:12.578 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 23 10:24:12 smithi204 ceph-mon[32649]: pgmap v266: 1 pgs: 1 active+clean; 577 KiB data, 365 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-23T10:24:13.254 INFO:journalctl@ceph.mds.j.smithi114.stdout:Apr 23 10:24:13 smithi114 systemd[1]: Starting Ceph mds.j for aa240572-0159-11ef-bc93-c7b262605968... 2024-04-23T10:24:13.567 DEBUG:teuthology.orchestra.run.smithi114:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-23T10:24:13.575 INFO:journalctl@ceph.mds.j.smithi114.stdout:Apr 23 10:24:13 smithi114 podman[67242]: 2024-04-23 10:24:13.222134165 +0000 UTC m=+0.018349397 image pull da4b004c09e820713483d039bc76fbd660be678f2ea3c03e059a6161b74dab76 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:068a558e3af4932cf1c60dd321c0bdf500075eb52ea1bf455e0973ba114fbfd4 2024-04-23T10:24:13.575 INFO:journalctl@ceph.mds.j.smithi114.stdout:Apr 23 10:24:13 smithi114 podman[67242]: 2024-04-23 10:24:13.356734421 +0000 UTC m=+0.152949640 container create 383afeb57c86c4986b4fbb48e40044e048c5fe4426ad7fb6e6913f5ab2228fb1 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:068a558e3af4932cf1c60dd321c0bdf500075eb52ea1bf455e0973ba114fbfd4, name=ceph-aa240572-0159-11ef-bc93-c7b262605968-mds-j, org.label-schema.build-date=20240416, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, org.label-schema.name=CentOS Stream 8 Base Image, maintainer=Guillaume Abrioux , ceph=True, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.33.5, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, GIT_CLEAN=True, org.label-schema.schema-version=1.0, RELEASE=wip-lusov-quiescer-d69f13d, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=) 2024-04-23T10:24:13.575 INFO:journalctl@ceph.mds.j.smithi114.stdout:Apr 23 10:24:13 smithi114 podman[67242]: 2024-04-23 10:24:13.550862944 +0000 UTC m=+0.347078163 container init 383afeb57c86c4986b4fbb48e40044e048c5fe4426ad7fb6e6913f5ab2228fb1 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:068a558e3af4932cf1c60dd321c0bdf500075eb52ea1bf455e0973ba114fbfd4, name=ceph-aa240572-0159-11ef-bc93-c7b262605968-mds-j, GIT_CLEAN=True, org.label-schema.name=CentOS Stream 8 Base Image, RELEASE=wip-lusov-quiescer-d69f13d, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux , ceph=True, org.label-schema.build-date=20240416, org.label-schema.license=GPLv2, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, io.buildah.version=1.33.5, CEPH_POINT_RELEASE=) 2024-04-23T10:24:13.575 INFO:journalctl@ceph.mds.j.smithi114.stdout:Apr 23 10:24:13 smithi114 podman[67242]: 2024-04-23 10:24:13.553385586 +0000 UTC m=+0.349600807 container start 383afeb57c86c4986b4fbb48e40044e048c5fe4426ad7fb6e6913f5ab2228fb1 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:068a558e3af4932cf1c60dd321c0bdf500075eb52ea1bf455e0973ba114fbfd4, name=ceph-aa240572-0159-11ef-bc93-c7b262605968-mds-j, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.schema-version=1.0, ceph=True, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, io.buildah.version=1.33.5, org.label-schema.license=GPLv2, RELEASE=wip-lusov-quiescer-d69f13d, maintainer=Guillaume Abrioux , GIT_CLEAN=True, org.label-schema.build-date=20240416) 2024-04-23T10:24:13.676 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:24:13.677+0000 7f0c0ca4e640 1 -- 172.21.15.114:0/2722500233 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f0c08107c90 msgr2=0x7f0c0810a0e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T10:24:13.676 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:24:13.677+0000 7f0c0ca4e640 1 --2- 172.21.15.114:0/2722500233 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f0c08107c90 0x7f0c0810a0e0 secure :-1 s=READY pgs=169 cs=0 l=1 rev1=1 crypto rx=0x7f0c00009ca0 tx=0x7f0c0002a8c0 comp rx=0 tx=0).stop 2024-04-23T10:24:13.676 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:24:13.677+0000 7f0c0ca4e640 1 -- 172.21.15.114:0/2722500233 shutdown_connections 2024-04-23T10:24:13.676 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:24:13.677+0000 7f0c0ca4e640 1 --2- 172.21.15.114:0/2722500233 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f0c08107c90 0x7f0c0810a0e0 unknown :-1 s=CLOSED pgs=169 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:24:13.676 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:24:13.677+0000 7f0c0ca4e640 1 --2- 172.21.15.114:0/2722500233 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f0c08105360 0x7f0c08107750 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:24:13.676 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:24:13.677+0000 7f0c0ca4e640 1 --2- 172.21.15.114:0/2722500233 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f0c08102a30 0x7f0c08104e20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:24:13.676 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:24:13.677+0000 7f0c0ca4e640 1 -- 172.21.15.114:0/2722500233 >> 172.21.15.114:0/2722500233 conn(0x7f0c080fc440 msgr2=0x7f0c080fe860 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T10:24:13.676 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:24:13.677+0000 7f0c0ca4e640 1 -- 172.21.15.114:0/2722500233 shutdown_connections 2024-04-23T10:24:13.677 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:24:13.677+0000 7f0c0ca4e640 1 -- 172.21.15.114:0/2722500233 wait complete. 2024-04-23T10:24:13.679 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:24:13.678+0000 7f0c0ca4e640 1 Processor -- start 2024-04-23T10:24:13.679 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:24:13.678+0000 7f0c0ca4e640 1 -- start start 2024-04-23T10:24:13.679 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:24:13.678+0000 7f0c0ca4e640 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f0c08102a30 0x7f0c08198110 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T10:24:13.679 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:24:13.678+0000 7f0c0ca4e640 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f0c08105360 0x7f0c08198650 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T10:24:13.679 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:24:13.678+0000 7f0c0ca4e640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f0c08107c90 0x7f0c0819f6c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T10:24:13.679 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:24:13.678+0000 7f0c0ca4e640 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x7f0c0810daf0 con 0x7f0c08107c90 2024-04-23T10:24:13.679 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:24:13.678+0000 7f0c0ca4e640 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 -- 0x7f0c0810d970 con 0x7f0c08102a30 2024-04-23T10:24:13.679 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:24:13.678+0000 7f0c0ca4e640 1 -- --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_getmap magic: 0 -- 0x7f0c0810dc70 con 0x7f0c08105360 2024-04-23T10:24:13.679 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:24:13.678+0000 7f0c06d9d640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f0c08107c90 0x7f0c0819f6c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T10:24:13.680 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:24:13.678+0000 7f0c06d9d640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f0c08107c90 0x7f0c0819f6c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.114:39008/0 (socket says 172.21.15.114:39008) 2024-04-23T10:24:13.680 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:24:13.678+0000 7f0c06d9d640 1 -- 172.21.15.114:0/2987354431 learned_addr learned my addr 172.21.15.114:0/2987354431 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-04-23T10:24:13.680 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:24:13.678+0000 7f0c0659c640 1 --2- 172.21.15.114:0/2987354431 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f0c08102a30 0x7f0c08198110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T10:24:13.680 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:24:13.678+0000 7f0c05d9b640 1 --2- 172.21.15.114:0/2987354431 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f0c08105360 0x7f0c08198650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T10:24:13.680 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:24:13.678+0000 7f0c06d9d640 1 -- 172.21.15.114:0/2987354431 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f0c08105360 msgr2=0x7f0c08198650 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T10:24:13.680 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:24:13.678+0000 7f0c06d9d640 1 --2- 172.21.15.114:0/2987354431 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f0c08105360 0x7f0c08198650 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:24:13.680 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:24:13.678+0000 7f0c06d9d640 1 -- 172.21.15.114:0/2987354431 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f0c08102a30 msgr2=0x7f0c08198110 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T10:24:13.680 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:24:13.678+0000 7f0c06d9d640 1 --2- 172.21.15.114:0/2987354431 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f0c08102a30 0x7f0c08198110 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:24:13.680 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:24:13.678+0000 7f0c06d9d640 1 -- 172.21.15.114:0/2987354431 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f0c0819fd30 con 0x7f0c08107c90 2024-04-23T10:24:13.680 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:24:13.679+0000 7f0c06d9d640 1 --2- 172.21.15.114:0/2987354431 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f0c08107c90 0x7f0c0819f6c0 secure :-1 s=READY pgs=170 cs=0 l=1 rev1=1 crypto rx=0x7f0c000097b0 tx=0x7f0c00031840 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T10:24:13.680 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:24:13.679+0000 7f0bf77fe640 1 -- 172.21.15.114:0/2987354431 <== mon.0 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0c00003bf0 con 0x7f0c08107c90 2024-04-23T10:24:13.680 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:24:13.679+0000 7f0c0ca4e640 1 -- 172.21.15.114:0/2987354431 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f0c0819ffc0 con 0x7f0c08107c90 2024-04-23T10:24:13.680 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:24:13.679+0000 7f0bf77fe640 1 -- 172.21.15.114:0/2987354431 <== mon.0 v2:172.21.15.114:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f0c00003d90 con 0x7f0c08107c90 2024-04-23T10:24:13.680 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:24:13.679+0000 7f0bf77fe640 1 -- 172.21.15.114:0/2987354431 <== mon.0 v2:172.21.15.114:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0c0002fc10 con 0x7f0c08107c90 2024-04-23T10:24:13.680 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:24:13.679+0000 7f0c0ca4e640 1 -- 172.21.15.114:0/2987354431 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f0c081a0480 con 0x7f0c08107c90 2024-04-23T10:24:13.680 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:24:13.680+0000 7f0c0ca4e640 1 -- 172.21.15.114:0/2987354431 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f0c080691c0 con 0x7f0c08107c90 2024-04-23T10:24:13.681 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:24:13.681+0000 7f0bf77fe640 1 -- 172.21.15.114:0/2987354431 <== mon.0 v2:172.21.15.114:3300/0 4 ==== mgrmap(e 16) ==== 138647+0+0 (secure 0 0 0) 0x7f0c00004040 con 0x7f0c08107c90 2024-04-23T10:24:13.681 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:24:13.682+0000 7f0bf77fe640 1 --2- 172.21.15.114:0/2987354431 >> [v2:172.21.15.114:6800/465333808,v1:172.21.15.114:6801/465333808] conn(0x7f0bf00a3bb0 0x7f0bf00a6070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T10:24:13.681 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:24:13.682+0000 7f0c0659c640 1 --2- 172.21.15.114:0/2987354431 >> [v2:172.21.15.114:6800/465333808,v1:172.21.15.114:6801/465333808] conn(0x7f0bf00a3bb0 0x7f0bf00a6070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T10:24:13.681 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:24:13.682+0000 7f0bf77fe640 1 -- 172.21.15.114:0/2987354431 <== mon.0 v2:172.21.15.114:3300/0 5 ==== osd_map(83..83 src has 1..83) ==== 7127+0+0 (secure 0 0 0) 0x7f0c000f18b0 con 0x7f0c08107c90 2024-04-23T10:24:13.682 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:24:13.682+0000 7f0c0659c640 1 --2- 172.21.15.114:0/2987354431 >> [v2:172.21.15.114:6800/465333808,v1:172.21.15.114:6801/465333808] conn(0x7f0bf00a3bb0 0x7f0bf00a6070 secure :-1 s=READY pgs=138 cs=0 l=1 rev1=1 crypto rx=0x7f0bf800abe0 tx=0x7f0bf800a2d0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T10:24:13.685 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:24:13.686+0000 7f0bf77fe640 1 -- 172.21.15.114:0/2987354431 <== mon.0 v2:172.21.15.114:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f0c000be7b0 con 0x7f0c08107c90 2024-04-23T10:24:13.927 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:24:13.928+0000 7f0c0ca4e640 1 -- 172.21.15.114:0/2987354431 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f0c08062b50 con 0x7f0c08107c90 2024-04-23T10:24:13.927 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:24:13.929+0000 7f0bf77fe640 1 -- 172.21.15.114:0/2987354431 <== mon.0 v2:172.21.15.114:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 4 v4) ==== 93+0+3046 (secure 0 0 0) 0x7f0c000be0b0 con 0x7f0c08107c90 2024-04-23T10:24:13.928 INFO:teuthology.orchestra.run.smithi114.stdout: 2024-04-23T10:24:13.928 INFO:teuthology.orchestra.run.smithi114.stdout:{"epoch":4,"default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14547,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.114:6835/663493580","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6834","nonce":663493580},{"type":"v1","addr":"172.21.15.114:6835","nonce":663493580}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":24385,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.204:6833/3866561754","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6832","nonce":3866561754},{"type":"v1","addr":"172.21.15.204:6833","nonce":3866561754}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2},{"gid":24443,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.119:6833/783244660","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6832","nonce":783244660},{"type":"v1","addr":"172.21.15.119:6833","nonce":783244660}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3}],"filesystems":[]} 2024-04-23T10:24:13.928 INFO:teuthology.orchestra.run.smithi114.stderr:dumped fsmap epoch 4 2024-04-23T10:24:13.930 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:24:13.931+0000 7f0c0ca4e640 1 -- 172.21.15.114:0/2987354431 >> [v2:172.21.15.114:6800/465333808,v1:172.21.15.114:6801/465333808] conn(0x7f0bf00a3bb0 msgr2=0x7f0bf00a6070 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T10:24:13.930 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:24:13.931+0000 7f0c0ca4e640 1 --2- 172.21.15.114:0/2987354431 >> [v2:172.21.15.114:6800/465333808,v1:172.21.15.114:6801/465333808] conn(0x7f0bf00a3bb0 0x7f0bf00a6070 secure :-1 s=READY pgs=138 cs=0 l=1 rev1=1 crypto rx=0x7f0bf800abe0 tx=0x7f0bf800a2d0 comp rx=0 tx=0).stop 2024-04-23T10:24:13.930 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:24:13.931+0000 7f0c0ca4e640 1 -- 172.21.15.114:0/2987354431 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f0c08107c90 msgr2=0x7f0c0819f6c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T10:24:13.930 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:24:13.931+0000 7f0c0ca4e640 1 --2- 172.21.15.114:0/2987354431 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f0c08107c90 0x7f0c0819f6c0 secure :-1 s=READY pgs=170 cs=0 l=1 rev1=1 crypto rx=0x7f0c000097b0 tx=0x7f0c00031840 comp rx=0 tx=0).stop 2024-04-23T10:24:13.930 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:24:13.931+0000 7f0c0ca4e640 1 -- 172.21.15.114:0/2987354431 shutdown_connections 2024-04-23T10:24:13.930 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:24:13.931+0000 7f0c0ca4e640 1 --2- 172.21.15.114:0/2987354431 >> [v2:172.21.15.114:6800/465333808,v1:172.21.15.114:6801/465333808] conn(0x7f0bf00a3bb0 0x7f0bf00a6070 unknown :-1 s=CLOSED pgs=138 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:24:13.930 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:24:13.931+0000 7f0c0ca4e640 1 --2- 172.21.15.114:0/2987354431 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f0c08107c90 0x7f0c0819f6c0 unknown :-1 s=CLOSED pgs=170 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:24:13.931 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:24:13.931+0000 7f0c0ca4e640 1 --2- 172.21.15.114:0/2987354431 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f0c08105360 0x7f0c08198650 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:24:13.931 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:24:13.931+0000 7f0c0ca4e640 1 --2- 172.21.15.114:0/2987354431 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f0c08102a30 0x7f0c08198110 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:24:13.931 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:24:13.931+0000 7f0c0ca4e640 1 -- 172.21.15.114:0/2987354431 >> 172.21.15.114:0/2987354431 conn(0x7f0c080fc440 msgr2=0x7f0c08103370 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T10:24:13.931 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:24:13.931+0000 7f0c0ca4e640 1 -- 172.21.15.114:0/2987354431 shutdown_connections 2024-04-23T10:24:13.931 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:24:13.931+0000 7f0c0ca4e640 1 -- 172.21.15.114:0/2987354431 wait complete. 2024-04-23T10:24:13.941 INFO:journalctl@ceph.mds.j.smithi114.stdout:Apr 23 10:24:13 smithi114 ceph-mds[67261]: -- 172.21.15.114:0/2934516277 <== mon.0 v2:172.21.15.114:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 194+0+0 (secure 0 0 0) 0x56257ede1a40 con 0x56257edb4800 2024-04-23T10:24:13.941 INFO:journalctl@ceph.mds.j.smithi114.stdout:Apr 23 10:24:13 smithi114 ceph-aa240572-0159-11ef-bc93-c7b262605968-mds-j[67257]: starting mds.j at 2024-04-23T10:24:13.941 INFO:journalctl@ceph.mds.j.smithi114.stdout:Apr 23 10:24:13 smithi114 bash[67242]: 383afeb57c86c4986b4fbb48e40044e048c5fe4426ad7fb6e6913f5ab2228fb1 2024-04-23T10:24:13.941 INFO:journalctl@ceph.mds.j.smithi114.stdout:Apr 23 10:24:13 smithi114 systemd[1]: Started Ceph mds.j for aa240572-0159-11ef-bc93-c7b262605968. 2024-04-23T10:24:14.744 INFO:journalctl@ceph.mon.a.smithi114.stdout:Apr 23 10:24:14 smithi114 ceph-mon[29255]: mds.? [v2:172.21.15.114:6834/663493580,v1:172.21.15.114:6835/663493580] up:boot 2024-04-23T10:24:14.745 INFO:journalctl@ceph.mon.a.smithi114.stdout:Apr 23 10:24:14 smithi114 ceph-mon[29255]: fsmap 3 up:standby 2024-04-23T10:24:14.745 INFO:journalctl@ceph.mon.a.smithi114.stdout:Apr 23 10:24:14 smithi114 ceph-mon[29255]: from='mgr.14152 172.21.15.114:0/1529706138' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "j"}]: dispatch 2024-04-23T10:24:14.745 INFO:journalctl@ceph.mon.a.smithi114.stdout:Apr 23 10:24:14 smithi114 ceph-mon[29255]: from='mgr.14152 172.21.15.114:0/1529706138' entity='mgr.x' 2024-04-23T10:24:14.745 INFO:journalctl@ceph.mon.a.smithi114.stdout:Apr 23 10:24:14 smithi114 ceph-mon[29255]: from='mgr.14152 172.21.15.114:0/1529706138' entity='mgr.x' 2024-04-23T10:24:14.745 INFO:journalctl@ceph.mon.a.smithi114.stdout:Apr 23 10:24:14 smithi114 ceph-mon[29255]: from='mgr.14152 172.21.15.114:0/1529706138' entity='mgr.x' 2024-04-23T10:24:14.745 INFO:journalctl@ceph.mon.a.smithi114.stdout:Apr 23 10:24:14 smithi114 ceph-mon[29255]: from='mgr.14152 172.21.15.114:0/1529706138' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.d", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-04-23T10:24:14.745 INFO:journalctl@ceph.mon.a.smithi114.stdout:Apr 23 10:24:14 smithi114 ceph-mon[29255]: from='mgr.14152 172.21.15.114:0/1529706138' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.d", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-04-23T10:24:14.745 INFO:journalctl@ceph.mon.a.smithi114.stdout:Apr 23 10:24:14 smithi114 ceph-mon[29255]: from='mgr.14152 172.21.15.114:0/1529706138' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-23T10:24:14.745 INFO:journalctl@ceph.mon.a.smithi114.stdout:Apr 23 10:24:14 smithi114 ceph-mon[29255]: Deploying daemon mds.d on smithi114 2024-04-23T10:24:14.745 INFO:journalctl@ceph.mon.a.smithi114.stdout:Apr 23 10:24:14 smithi114 ceph-mon[29255]: from='client.? 172.21.15.114:0/2987354431' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-23T10:24:14.745 INFO:journalctl@ceph.mon.a.smithi114.stdout:Apr 23 10:24:14 smithi114 ceph-mon[29255]: pgmap v267: 1 pgs: 1 active+clean; 577 KiB data, 365 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-23T10:24:14.992 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 23 10:24:14 smithi119 ceph-mon[32608]: mds.? [v2:172.21.15.114:6834/663493580,v1:172.21.15.114:6835/663493580] up:boot 2024-04-23T10:24:14.992 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 23 10:24:14 smithi119 ceph-mon[32608]: fsmap 3 up:standby 2024-04-23T10:24:14.992 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 23 10:24:14 smithi119 ceph-mon[32608]: from='mgr.14152 172.21.15.114:0/1529706138' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "j"}]: dispatch 2024-04-23T10:24:14.992 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 23 10:24:14 smithi119 ceph-mon[32608]: from='mgr.14152 172.21.15.114:0/1529706138' entity='mgr.x' 2024-04-23T10:24:14.992 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 23 10:24:14 smithi119 ceph-mon[32608]: from='mgr.14152 172.21.15.114:0/1529706138' entity='mgr.x' 2024-04-23T10:24:14.992 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 23 10:24:14 smithi119 ceph-mon[32608]: from='mgr.14152 172.21.15.114:0/1529706138' entity='mgr.x' 2024-04-23T10:24:14.992 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 23 10:24:14 smithi119 ceph-mon[32608]: from='mgr.14152 172.21.15.114:0/1529706138' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.d", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-04-23T10:24:14.992 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 23 10:24:14 smithi119 ceph-mon[32608]: from='mgr.14152 172.21.15.114:0/1529706138' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.d", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-04-23T10:24:14.992 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 23 10:24:14 smithi119 ceph-mon[32608]: from='mgr.14152 172.21.15.114:0/1529706138' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-23T10:24:14.992 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 23 10:24:14 smithi119 ceph-mon[32608]: Deploying daemon mds.d on smithi114 2024-04-23T10:24:14.992 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 23 10:24:14 smithi119 ceph-mon[32608]: from='client.? 172.21.15.114:0/2987354431' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-23T10:24:14.992 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 23 10:24:14 smithi119 ceph-mon[32608]: pgmap v267: 1 pgs: 1 active+clean; 577 KiB data, 365 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-23T10:24:15.078 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 23 10:24:14 smithi204 ceph-mon[32649]: mds.? [v2:172.21.15.114:6834/663493580,v1:172.21.15.114:6835/663493580] up:boot 2024-04-23T10:24:15.078 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 23 10:24:14 smithi204 ceph-mon[32649]: fsmap 3 up:standby 2024-04-23T10:24:15.078 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 23 10:24:14 smithi204 ceph-mon[32649]: from='mgr.14152 172.21.15.114:0/1529706138' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "j"}]: dispatch 2024-04-23T10:24:15.078 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 23 10:24:14 smithi204 ceph-mon[32649]: from='mgr.14152 172.21.15.114:0/1529706138' entity='mgr.x' 2024-04-23T10:24:15.078 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 23 10:24:14 smithi204 ceph-mon[32649]: from='mgr.14152 172.21.15.114:0/1529706138' entity='mgr.x' 2024-04-23T10:24:15.078 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 23 10:24:14 smithi204 ceph-mon[32649]: from='mgr.14152 172.21.15.114:0/1529706138' entity='mgr.x' 2024-04-23T10:24:15.078 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 23 10:24:14 smithi204 ceph-mon[32649]: from='mgr.14152 172.21.15.114:0/1529706138' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.d", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-04-23T10:24:15.078 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 23 10:24:14 smithi204 ceph-mon[32649]: from='mgr.14152 172.21.15.114:0/1529706138' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.d", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-04-23T10:24:15.079 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 23 10:24:14 smithi204 ceph-mon[32649]: from='mgr.14152 172.21.15.114:0/1529706138' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-23T10:24:15.079 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 23 10:24:14 smithi204 ceph-mon[32649]: Deploying daemon mds.d on smithi114 2024-04-23T10:24:15.079 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 23 10:24:14 smithi204 ceph-mon[32649]: from='client.? 172.21.15.114:0/2987354431' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-23T10:24:15.079 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 23 10:24:14 smithi204 ceph-mon[32649]: pgmap v267: 1 pgs: 1 active+clean; 577 KiB data, 365 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-23T10:24:15.943 DEBUG:teuthology.orchestra.run.smithi114:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-23T10:24:15.967 INFO:journalctl@ceph.mds.d.smithi114.stdout:Apr 23 10:24:15 smithi114 systemd[1]: Starting Ceph mds.d for aa240572-0159-11ef-bc93-c7b262605968... 2024-04-23T10:24:16.059 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:24:16.058+0000 7fc1fc568640 1 -- 172.21.15.114:0/2571029743 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fc1f41029a0 msgr2=0x7fc1f41092c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T10:24:16.059 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:24:16.058+0000 7fc1fc568640 1 --2- 172.21.15.114:0/2571029743 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fc1f41029a0 0x7fc1f41092c0 secure :-1 s=READY pgs=171 cs=0 l=1 rev1=1 crypto rx=0x7fc1e4009900 tx=0x7fc1e402a410 comp rx=0 tx=0).stop 2024-04-23T10:24:16.060 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:24:16.058+0000 7fc1fc568640 1 -- 172.21.15.114:0/2571029743 shutdown_connections 2024-04-23T10:24:16.060 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:24:16.058+0000 7fc1fc568640 1 --2- 172.21.15.114:0/2571029743 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fc1f41029a0 0x7fc1f41092c0 unknown :-1 s=CLOSED pgs=171 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:24:16.060 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:24:16.058+0000 7fc1fc568640 1 --2- 172.21.15.114:0/2571029743 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fc1f4101fe0 0x7fc1f4102460 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:24:16.060 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:24:16.058+0000 7fc1fc568640 1 --2- 172.21.15.114:0/2571029743 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fc1f4100e90 0x7fc1f41012b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:24:16.060 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:24:16.059+0000 7fc1fc568640 1 -- 172.21.15.114:0/2571029743 >> 172.21.15.114:0/2571029743 conn(0x7fc1f40fc400 msgr2=0x7fc1f40fe840 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T10:24:16.060 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:24:16.059+0000 7fc1fc568640 1 -- 172.21.15.114:0/2571029743 shutdown_connections 2024-04-23T10:24:16.060 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:24:16.059+0000 7fc1fc568640 1 -- 172.21.15.114:0/2571029743 wait complete. 2024-04-23T10:24:16.060 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:24:16.059+0000 7fc1fc568640 1 Processor -- start 2024-04-23T10:24:16.060 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:24:16.059+0000 7fc1fc568640 1 -- start start 2024-04-23T10:24:16.060 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:24:16.059+0000 7fc1fc568640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fc1f4100e90 0x7fc1f4069c70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T10:24:16.060 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:24:16.059+0000 7fc1fc568640 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fc1f4101fe0 0x7fc1f419cec0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T10:24:16.060 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:24:16.059+0000 7fc1fc568640 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fc1f41029a0 0x7fc1f419f500 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T10:24:16.060 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:24:16.059+0000 7fc1fc568640 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x7fc1f410c9f0 con 0x7fc1f4100e90 2024-04-23T10:24:16.060 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:24:16.059+0000 7fc1fc568640 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 -- 0x7fc1f410c870 con 0x7fc1f41029a0 2024-04-23T10:24:16.060 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:24:16.059+0000 7fc1fc568640 1 -- --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_getmap magic: 0 -- 0x7fc1f410cb70 con 0x7fc1f4101fe0 2024-04-23T10:24:16.060 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:24:16.059+0000 7fc1fa304640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fc1f4100e90 0x7fc1f4069c70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T10:24:16.061 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:24:16.059+0000 7fc1fa304640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fc1f4100e90 0x7fc1f4069c70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.114:42566/0 (socket says 172.21.15.114:42566) 2024-04-23T10:24:16.061 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:24:16.059+0000 7fc1fa304640 1 -- 172.21.15.114:0/92967189 learned_addr learned my addr 172.21.15.114:0/92967189 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-04-23T10:24:16.061 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:24:16.059+0000 7fc1f9b03640 1 --2- 172.21.15.114:0/92967189 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fc1f4101fe0 0x7fc1f419cec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T10:24:16.061 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:24:16.060+0000 7fc1fab05640 1 --2- 172.21.15.114:0/92967189 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fc1f41029a0 0x7fc1f419f500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T10:24:16.061 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:24:16.060+0000 7fc1fa304640 1 -- 172.21.15.114:0/92967189 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fc1f4101fe0 msgr2=0x7fc1f419cec0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T10:24:16.061 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:24:16.060+0000 7fc1fa304640 1 --2- 172.21.15.114:0/92967189 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fc1f4101fe0 0x7fc1f419cec0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:24:16.061 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:24:16.060+0000 7fc1fa304640 1 -- 172.21.15.114:0/92967189 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fc1f41029a0 msgr2=0x7fc1f419f500 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T10:24:16.061 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:24:16.060+0000 7fc1fa304640 1 --2- 172.21.15.114:0/92967189 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fc1f41029a0 0x7fc1f419f500 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:24:16.061 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:24:16.060+0000 7fc1fa304640 1 -- 172.21.15.114:0/92967189 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc1f419fdc0 con 0x7fc1f4100e90 2024-04-23T10:24:16.061 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:24:16.060+0000 7fc1f9b03640 1 --2- 172.21.15.114:0/92967189 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fc1f4101fe0 0x7fc1f419cec0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-23T10:24:16.061 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:24:16.060+0000 7fc1fab05640 1 --2- 172.21.15.114:0/92967189 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fc1f41029a0 0x7fc1f419f500 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-23T10:24:16.061 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:24:16.060+0000 7fc1fa304640 1 --2- 172.21.15.114:0/92967189 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fc1f4100e90 0x7fc1f4069c70 secure :-1 s=READY pgs=172 cs=0 l=1 rev1=1 crypto rx=0x7fc1ec00d770 tx=0x7fc1ec00dc30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T10:24:16.061 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:24:16.060+0000 7fc1f37fe640 1 -- 172.21.15.114:0/92967189 <== mon.0 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc1ec017070 con 0x7fc1f4100e90 2024-04-23T10:24:16.061 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:24:16.060+0000 7fc1f37fe640 1 -- 172.21.15.114:0/92967189 <== mon.0 v2:172.21.15.114:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fc1ec0054d0 con 0x7fc1f4100e90 2024-04-23T10:24:16.061 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:24:16.060+0000 7fc1f37fe640 1 -- 172.21.15.114:0/92967189 <== mon.0 v2:172.21.15.114:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc1ec004c30 con 0x7fc1f4100e90 2024-04-23T10:24:16.062 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:24:16.060+0000 7fc1fc568640 1 -- 172.21.15.114:0/92967189 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fc1f41a00b0 con 0x7fc1f4100e90 2024-04-23T10:24:16.062 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:24:16.060+0000 7fc1fc568640 1 -- 172.21.15.114:0/92967189 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fc1f41a05b0 con 0x7fc1f4100e90 2024-04-23T10:24:16.063 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:24:16.061+0000 7fc1fc568640 1 -- 172.21.15.114:0/92967189 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fc1f41081a0 con 0x7fc1f4100e90 2024-04-23T10:24:16.063 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:24:16.062+0000 7fc1f37fe640 1 -- 172.21.15.114:0/92967189 <== mon.0 v2:172.21.15.114:3300/0 4 ==== mgrmap(e 16) ==== 138647+0+0 (secure 0 0 0) 0x7fc1ec004dd0 con 0x7fc1f4100e90 2024-04-23T10:24:16.063 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:24:16.063+0000 7fc1f37fe640 1 --2- 172.21.15.114:0/92967189 >> [v2:172.21.15.114:6800/465333808,v1:172.21.15.114:6801/465333808] conn(0x7fc1dc0a3c30 0x7fc1dc0a60f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T10:24:16.063 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:24:16.063+0000 7fc1f9b03640 1 --2- 172.21.15.114:0/92967189 >> [v2:172.21.15.114:6800/465333808,v1:172.21.15.114:6801/465333808] conn(0x7fc1dc0a3c30 0x7fc1dc0a60f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T10:24:16.063 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:24:16.063+0000 7fc1f37fe640 1 -- 172.21.15.114:0/92967189 <== mon.0 v2:172.21.15.114:3300/0 5 ==== osd_map(83..83 src has 1..83) ==== 7127+0+0 (secure 0 0 0) 0x7fc1ec0cfe80 con 0x7fc1f4100e90 2024-04-23T10:24:16.063 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:24:16.063+0000 7fc1f9b03640 1 --2- 172.21.15.114:0/92967189 >> [v2:172.21.15.114:6800/465333808,v1:172.21.15.114:6801/465333808] conn(0x7fc1dc0a3c30 0x7fc1dc0a60f0 secure :-1 s=READY pgs=139 cs=0 l=1 rev1=1 crypto rx=0x7fc1f40699e0 tx=0x7fc1e8009540 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T10:24:16.065 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:24:16.066+0000 7fc1f37fe640 1 -- 172.21.15.114:0/92967189 <== mon.0 v2:172.21.15.114:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7fc1ec09cd80 con 0x7fc1f4100e90 2024-04-23T10:24:16.220 INFO:journalctl@ceph.mds.d.smithi114.stdout:Apr 23 10:24:16 smithi114 podman[67606]: 2024-04-23 10:24:16.002900699 +0000 UTC m=+0.026144643 image pull da4b004c09e820713483d039bc76fbd660be678f2ea3c03e059a6161b74dab76 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:068a558e3af4932cf1c60dd321c0bdf500075eb52ea1bf455e0973ba114fbfd4 2024-04-23T10:24:16.220 INFO:journalctl@ceph.mds.d.smithi114.stdout:Apr 23 10:24:16 smithi114 podman[67606]: 2024-04-23 10:24:16.111808884 +0000 UTC m=+0.135052813 container create 6fd229fdfcca3f474f5659a8f9c7602d73c060537c379b7946717b0039b87e17 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:068a558e3af4932cf1c60dd321c0bdf500075eb52ea1bf455e0973ba114fbfd4, name=ceph-aa240572-0159-11ef-bc93-c7b262605968-mds-d, io.buildah.version=1.33.5, GIT_BRANCH=HEAD, GIT_CLEAN=True, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, org.label-schema.name=CentOS Stream 8 Base Image, RELEASE=wip-lusov-quiescer-d69f13d, ceph=True, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=, org.label-schema.build-date=20240416) 2024-04-23T10:24:16.291 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:24:16.291+0000 7fc1fc568640 1 -- 172.21.15.114:0/92967189 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fc1f402d5c0 con 0x7fc1f4100e90 2024-04-23T10:24:16.291 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:24:16.292+0000 7fc1f37fe640 1 -- 172.21.15.114:0/92967189 <== mon.0 v2:172.21.15.114:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 4 v4) ==== 93+0+3046 (secure 0 0 0) 0x7fc1ec09c680 con 0x7fc1f4100e90 2024-04-23T10:24:16.291 INFO:teuthology.orchestra.run.smithi114.stdout: 2024-04-23T10:24:16.291 INFO:teuthology.orchestra.run.smithi114.stdout:{"epoch":4,"default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14547,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.114:6835/663493580","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6834","nonce":663493580},{"type":"v1","addr":"172.21.15.114:6835","nonce":663493580}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":24385,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.204:6833/3866561754","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6832","nonce":3866561754},{"type":"v1","addr":"172.21.15.204:6833","nonce":3866561754}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2},{"gid":24443,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.119:6833/783244660","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6832","nonce":783244660},{"type":"v1","addr":"172.21.15.119:6833","nonce":783244660}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3}],"filesystems":[]} 2024-04-23T10:24:16.291 INFO:teuthology.orchestra.run.smithi114.stderr:dumped fsmap epoch 4 2024-04-23T10:24:16.293 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:24:16.294+0000 7fc1fc568640 1 -- 172.21.15.114:0/92967189 >> [v2:172.21.15.114:6800/465333808,v1:172.21.15.114:6801/465333808] conn(0x7fc1dc0a3c30 msgr2=0x7fc1dc0a60f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T10:24:16.293 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:24:16.294+0000 7fc1fc568640 1 --2- 172.21.15.114:0/92967189 >> [v2:172.21.15.114:6800/465333808,v1:172.21.15.114:6801/465333808] conn(0x7fc1dc0a3c30 0x7fc1dc0a60f0 secure :-1 s=READY pgs=139 cs=0 l=1 rev1=1 crypto rx=0x7fc1f40699e0 tx=0x7fc1e8009540 comp rx=0 tx=0).stop 2024-04-23T10:24:16.293 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:24:16.294+0000 7fc1fc568640 1 -- 172.21.15.114:0/92967189 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fc1f4100e90 msgr2=0x7fc1f4069c70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T10:24:16.293 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:24:16.294+0000 7fc1fc568640 1 --2- 172.21.15.114:0/92967189 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fc1f4100e90 0x7fc1f4069c70 secure :-1 s=READY pgs=172 cs=0 l=1 rev1=1 crypto rx=0x7fc1ec00d770 tx=0x7fc1ec00dc30 comp rx=0 tx=0).stop 2024-04-23T10:24:16.294 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:24:16.294+0000 7fc1fc568640 1 -- 172.21.15.114:0/92967189 shutdown_connections 2024-04-23T10:24:16.294 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:24:16.294+0000 7fc1fc568640 1 --2- 172.21.15.114:0/92967189 >> [v2:172.21.15.114:6800/465333808,v1:172.21.15.114:6801/465333808] conn(0x7fc1dc0a3c30 0x7fc1dc0a60f0 unknown :-1 s=CLOSED pgs=139 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:24:16.294 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:24:16.294+0000 7fc1fc568640 1 --2- 172.21.15.114:0/92967189 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fc1f41029a0 0x7fc1f419f500 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:24:16.294 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:24:16.294+0000 7fc1fc568640 1 --2- 172.21.15.114:0/92967189 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fc1f4101fe0 0x7fc1f419cec0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:24:16.294 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:24:16.294+0000 7fc1fc568640 1 --2- 172.21.15.114:0/92967189 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fc1f4100e90 0x7fc1f4069c70 unknown :-1 s=CLOSED pgs=172 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:24:16.294 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:24:16.294+0000 7fc1fc568640 1 -- 172.21.15.114:0/92967189 >> 172.21.15.114:0/92967189 conn(0x7fc1f40fc400 msgr2=0x7fc1f4105370 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T10:24:16.294 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:24:16.294+0000 7fc1fc568640 1 -- 172.21.15.114:0/92967189 shutdown_connections 2024-04-23T10:24:16.294 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:24:16.294+0000 7fc1fc568640 1 -- 172.21.15.114:0/92967189 wait complete. 2024-04-23T10:24:16.754 INFO:journalctl@ceph.mds.d.smithi114.stdout:Apr 23 10:24:16 smithi114 podman[67606]: 2024-04-23 10:24:16.347736752 +0000 UTC m=+0.370980687 container init 6fd229fdfcca3f474f5659a8f9c7602d73c060537c379b7946717b0039b87e17 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:068a558e3af4932cf1c60dd321c0bdf500075eb52ea1bf455e0973ba114fbfd4, name=ceph-aa240572-0159-11ef-bc93-c7b262605968-mds-d, RELEASE=wip-lusov-quiescer-d69f13d, io.buildah.version=1.33.5, org.label-schema.schema-version=1.0, GIT_CLEAN=True, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, GIT_BRANCH=HEAD, org.label-schema.build-date=20240416, org.label-schema.vendor=CentOS, ceph=True, org.label-schema.license=GPLv2, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 8 Base Image) 2024-04-23T10:24:16.754 INFO:journalctl@ceph.mds.d.smithi114.stdout:Apr 23 10:24:16 smithi114 podman[67606]: 2024-04-23 10:24:16.350438929 +0000 UTC m=+0.373682858 container start 6fd229fdfcca3f474f5659a8f9c7602d73c060537c379b7946717b0039b87e17 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:068a558e3af4932cf1c60dd321c0bdf500075eb52ea1bf455e0973ba114fbfd4, name=ceph-aa240572-0159-11ef-bc93-c7b262605968-mds-d, io.buildah.version=1.33.5, org.label-schema.build-date=20240416, maintainer=Guillaume Abrioux , GIT_CLEAN=True, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, RELEASE=wip-lusov-quiescer-d69f13d, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True) 2024-04-23T10:24:16.754 INFO:journalctl@ceph.mds.d.smithi114.stdout:Apr 23 10:24:16 smithi114 ceph-mds[67649]: -- 172.21.15.114:0/1855749257 <== mon.0 v2:172.21.15.114:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 194+0+0 (secure 0 0 0) 0x556b77a5ba40 con 0x556b77a2ec00 2024-04-23T10:24:16.754 INFO:journalctl@ceph.mds.d.smithi114.stdout:Apr 23 10:24:16 smithi114 ceph-aa240572-0159-11ef-bc93-c7b262605968-mds-d[67643]: starting mds.d at 2024-04-23T10:24:16.754 INFO:journalctl@ceph.mds.d.smithi114.stdout:Apr 23 10:24:16 smithi114 bash[67606]: 6fd229fdfcca3f474f5659a8f9c7602d73c060537c379b7946717b0039b87e17 2024-04-23T10:24:16.754 INFO:journalctl@ceph.mds.d.smithi114.stdout:Apr 23 10:24:16 smithi114 systemd[1]: Started Ceph mds.d for aa240572-0159-11ef-bc93-c7b262605968. 2024-04-23T10:24:17.297 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 23 10:24:17 smithi119 ceph-mon[32608]: pgmap v268: 1 pgs: 1 active+clean; 577 KiB data, 365 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-23T10:24:17.297 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 23 10:24:17 smithi119 ceph-mon[32608]: from='client.? 172.21.15.114:0/92967189' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-23T10:24:17.297 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 23 10:24:17 smithi119 ceph-mon[32608]: from='mgr.14152 172.21.15.114:0/1529706138' entity='mgr.x' 2024-04-23T10:24:17.297 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 23 10:24:17 smithi119 ceph-mon[32608]: from='mgr.14152 172.21.15.114:0/1529706138' entity='mgr.x' 2024-04-23T10:24:17.297 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 23 10:24:17 smithi119 ceph-mon[32608]: from='mgr.14152 172.21.15.114:0/1529706138' entity='mgr.x' 2024-04-23T10:24:17.297 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 23 10:24:17 smithi119 ceph-mon[32608]: from='mgr.14152 172.21.15.114:0/1529706138' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.e", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-04-23T10:24:17.297 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 23 10:24:17 smithi119 ceph-mon[32608]: from='mgr.14152 172.21.15.114:0/1529706138' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.e", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-04-23T10:24:17.297 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 23 10:24:17 smithi119 ceph-mon[32608]: from='mgr.14152 172.21.15.114:0/1529706138' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-23T10:24:17.328 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 23 10:24:17 smithi204 ceph-mon[32649]: pgmap v268: 1 pgs: 1 active+clean; 577 KiB data, 365 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-23T10:24:17.328 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 23 10:24:17 smithi204 ceph-mon[32649]: from='client.? 172.21.15.114:0/92967189' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-23T10:24:17.328 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 23 10:24:17 smithi204 ceph-mon[32649]: from='mgr.14152 172.21.15.114:0/1529706138' entity='mgr.x' 2024-04-23T10:24:17.328 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 23 10:24:17 smithi204 ceph-mon[32649]: from='mgr.14152 172.21.15.114:0/1529706138' entity='mgr.x' 2024-04-23T10:24:17.328 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 23 10:24:17 smithi204 ceph-mon[32649]: from='mgr.14152 172.21.15.114:0/1529706138' entity='mgr.x' 2024-04-23T10:24:17.328 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 23 10:24:17 smithi204 ceph-mon[32649]: from='mgr.14152 172.21.15.114:0/1529706138' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.e", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-04-23T10:24:17.328 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 23 10:24:17 smithi204 ceph-mon[32649]: from='mgr.14152 172.21.15.114:0/1529706138' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.e", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-04-23T10:24:17.328 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 23 10:24:17 smithi204 ceph-mon[32649]: from='mgr.14152 172.21.15.114:0/1529706138' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-23T10:24:17.504 INFO:journalctl@ceph.mon.a.smithi114.stdout:Apr 23 10:24:17 smithi114 ceph-mon[29255]: pgmap v268: 1 pgs: 1 active+clean; 577 KiB data, 365 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-23T10:24:17.504 INFO:journalctl@ceph.mon.a.smithi114.stdout:Apr 23 10:24:17 smithi114 ceph-mon[29255]: from='client.? 172.21.15.114:0/92967189' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-23T10:24:17.504 INFO:journalctl@ceph.mon.a.smithi114.stdout:Apr 23 10:24:17 smithi114 ceph-mon[29255]: from='mgr.14152 172.21.15.114:0/1529706138' entity='mgr.x' 2024-04-23T10:24:17.504 INFO:journalctl@ceph.mon.a.smithi114.stdout:Apr 23 10:24:17 smithi114 ceph-mon[29255]: from='mgr.14152 172.21.15.114:0/1529706138' entity='mgr.x' 2024-04-23T10:24:17.504 INFO:journalctl@ceph.mon.a.smithi114.stdout:Apr 23 10:24:17 smithi114 ceph-mon[29255]: from='mgr.14152 172.21.15.114:0/1529706138' entity='mgr.x' 2024-04-23T10:24:17.504 INFO:journalctl@ceph.mon.a.smithi114.stdout:Apr 23 10:24:17 smithi114 ceph-mon[29255]: from='mgr.14152 172.21.15.114:0/1529706138' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.e", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-04-23T10:24:17.505 INFO:journalctl@ceph.mon.a.smithi114.stdout:Apr 23 10:24:17 smithi114 ceph-mon[29255]: from='mgr.14152 172.21.15.114:0/1529706138' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.e", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-04-23T10:24:17.505 INFO:journalctl@ceph.mon.a.smithi114.stdout:Apr 23 10:24:17 smithi114 ceph-mon[29255]: from='mgr.14152 172.21.15.114:0/1529706138' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-23T10:24:18.254 INFO:journalctl@ceph.mon.a.smithi114.stdout:Apr 23 10:24:18 smithi114 ceph-mon[29255]: Deploying daemon mds.e on smithi119 2024-04-23T10:24:18.254 INFO:journalctl@ceph.mon.a.smithi114.stdout:Apr 23 10:24:18 smithi114 ceph-mon[29255]: mds.? [v2:172.21.15.114:6836/1439252154,v1:172.21.15.114:6837/1439252154] up:boot 2024-04-23T10:24:18.254 INFO:journalctl@ceph.mon.a.smithi114.stdout:Apr 23 10:24:18 smithi114 ceph-mon[29255]: fsmap 4 up:standby 2024-04-23T10:24:18.254 INFO:journalctl@ceph.mon.a.smithi114.stdout:Apr 23 10:24:18 smithi114 ceph-mon[29255]: from='mgr.14152 172.21.15.114:0/1529706138' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "d"}]: dispatch 2024-04-23T10:24:18.305 DEBUG:teuthology.orchestra.run.smithi114:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-23T10:24:18.328 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 23 10:24:18 smithi204 ceph-mon[32649]: Deploying daemon mds.e on smithi119 2024-04-23T10:24:18.328 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 23 10:24:18 smithi204 ceph-mon[32649]: mds.? [v2:172.21.15.114:6836/1439252154,v1:172.21.15.114:6837/1439252154] up:boot 2024-04-23T10:24:18.328 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 23 10:24:18 smithi204 ceph-mon[32649]: fsmap 4 up:standby 2024-04-23T10:24:18.328 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 23 10:24:18 smithi204 ceph-mon[32649]: from='mgr.14152 172.21.15.114:0/1529706138' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "d"}]: dispatch 2024-04-23T10:24:18.409 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:24:18.407+0000 7f2f31507640 1 -- 172.21.15.114:0/4025914622 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f2f2c102a10 msgr2=0x7f2f2c104e00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T10:24:18.409 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:24:18.407+0000 7f2f31507640 1 --2- 172.21.15.114:0/4025914622 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f2f2c102a10 0x7f2f2c104e00 secure :-1 s=READY pgs=175 cs=0 l=1 rev1=1 crypto rx=0x7f2f20009900 tx=0x7f2f2002a410 comp rx=0 tx=0).stop 2024-04-23T10:24:18.409 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:24:18.407+0000 7f2f31507640 1 -- 172.21.15.114:0/4025914622 shutdown_connections 2024-04-23T10:24:18.409 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:24:18.407+0000 7f2f31507640 1 --2- 172.21.15.114:0/4025914622 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f2f2c107c70 0x7f2f2c10a0c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:24:18.409 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:24:18.407+0000 7f2f31507640 1 --2- 172.21.15.114:0/4025914622 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f2f2c105340 0x7f2f2c107730 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:24:18.409 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:24:18.407+0000 7f2f31507640 1 --2- 172.21.15.114:0/4025914622 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f2f2c102a10 0x7f2f2c104e00 unknown :-1 s=CLOSED pgs=175 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:24:18.409 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:24:18.407+0000 7f2f31507640 1 -- 172.21.15.114:0/4025914622 >> 172.21.15.114:0/4025914622 conn(0x7f2f2c0fc400 msgr2=0x7f2f2c0fe840 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T10:24:18.409 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:24:18.407+0000 7f2f31507640 1 -- 172.21.15.114:0/4025914622 shutdown_connections 2024-04-23T10:24:18.409 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:24:18.407+0000 7f2f31507640 1 -- 172.21.15.114:0/4025914622 wait complete. 2024-04-23T10:24:18.409 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:24:18.408+0000 7f2f31507640 1 Processor -- start 2024-04-23T10:24:18.409 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:24:18.408+0000 7f2f31507640 1 -- start start 2024-04-23T10:24:18.409 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:24:18.408+0000 7f2f31507640 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f2f2c102a10 0x7f2f2c198200 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T10:24:18.410 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:24:18.408+0000 7f2f31507640 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f2f2c105340 0x7f2f2c198740 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T10:24:18.410 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:24:18.408+0000 7f2f31507640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f2f2c107c70 0x7f2f2c19f7b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T10:24:18.410 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:24:18.408+0000 7f2f31507640 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x7f2f2c10dad0 con 0x7f2f2c107c70 2024-04-23T10:24:18.410 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:24:18.408+0000 7f2f31507640 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 -- 0x7f2f2c10d950 con 0x7f2f2c105340 2024-04-23T10:24:18.410 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:24:18.408+0000 7f2f31507640 1 -- --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_getmap magic: 0 -- 0x7f2f2c10dc50 con 0x7f2f2c102a10 2024-04-23T10:24:18.410 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:24:18.408+0000 7f2f2b7fe640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f2f2c107c70 0x7f2f2c19f7b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T10:24:18.410 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:24:18.408+0000 7f2f2a7fc640 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f2f2c105340 0x7f2f2c198740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T10:24:18.410 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:24:18.408+0000 7f2f2b7fe640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f2f2c107c70 0x7f2f2c19f7b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.114:42596/0 (socket says 172.21.15.114:42596) 2024-04-23T10:24:18.410 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:24:18.408+0000 7f2f2a7fc640 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f2f2c105340 0x7f2f2c198740 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.119:3300/0 says I am v2:172.21.15.114:51094/0 (socket says 172.21.15.114:51094) 2024-04-23T10:24:18.410 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:24:18.408+0000 7f2f2b7fe640 1 -- 172.21.15.114:0/4204900988 learned_addr learned my addr 172.21.15.114:0/4204900988 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-04-23T10:24:18.410 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:24:18.408+0000 7f2f2affd640 1 --2- 172.21.15.114:0/4204900988 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f2f2c102a10 0x7f2f2c198200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T10:24:18.410 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:24:18.408+0000 7f2f2b7fe640 1 -- 172.21.15.114:0/4204900988 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f2f2c102a10 msgr2=0x7f2f2c198200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T10:24:18.410 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:24:18.408+0000 7f2f2b7fe640 1 --2- 172.21.15.114:0/4204900988 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f2f2c102a10 0x7f2f2c198200 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:24:18.410 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:24:18.409+0000 7f2f2b7fe640 1 -- 172.21.15.114:0/4204900988 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f2f2c105340 msgr2=0x7f2f2c198740 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T10:24:18.411 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:24:18.409+0000 7f2f2b7fe640 1 --2- 172.21.15.114:0/4204900988 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f2f2c105340 0x7f2f2c198740 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:24:18.411 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:24:18.409+0000 7f2f2b7fe640 1 -- 172.21.15.114:0/4204900988 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f2f2c19fe20 con 0x7f2f2c107c70 2024-04-23T10:24:18.411 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:24:18.409+0000 7f2f2a7fc640 1 --2- 172.21.15.114:0/4204900988 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f2f2c105340 0x7f2f2c198740 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-23T10:24:18.411 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:24:18.409+0000 7f2f2b7fe640 1 --2- 172.21.15.114:0/4204900988 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f2f2c107c70 0x7f2f2c19f7b0 secure :-1 s=READY pgs=176 cs=0 l=1 rev1=1 crypto rx=0x7f2f2400eaa0 tx=0x7f2f2400ef60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T10:24:18.411 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:24:18.409+0000 7f2f13fff640 1 -- 172.21.15.114:0/4204900988 <== mon.0 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2f2400cdb0 con 0x7f2f2c107c70 2024-04-23T10:24:18.411 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:24:18.409+0000 7f2f31507640 1 -- 172.21.15.114:0/4204900988 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f2f2c1a0110 con 0x7f2f2c107c70 2024-04-23T10:24:18.411 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:24:18.409+0000 7f2f13fff640 1 -- 172.21.15.114:0/4204900988 <== mon.0 v2:172.21.15.114:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f2f24005020 con 0x7f2f2c107c70 2024-04-23T10:24:18.411 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:24:18.409+0000 7f2f13fff640 1 -- 172.21.15.114:0/4204900988 <== mon.0 v2:172.21.15.114:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2f240116d0 con 0x7f2f2c107c70 2024-04-23T10:24:18.411 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:24:18.409+0000 7f2f31507640 1 -- 172.21.15.114:0/4204900988 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f2f2c1a0610 con 0x7f2f2c107c70 2024-04-23T10:24:18.411 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:24:18.410+0000 7f2f31507640 1 -- 172.21.15.114:0/4204900988 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f2f2c0fdef0 con 0x7f2f2c107c70 2024-04-23T10:24:18.412 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:24:18.411+0000 7f2f13fff640 1 -- 172.21.15.114:0/4204900988 <== mon.0 v2:172.21.15.114:3300/0 4 ==== mgrmap(e 16) ==== 138647+0+0 (secure 0 0 0) 0x7f2f240051d0 con 0x7f2f2c107c70 2024-04-23T10:24:18.412 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:24:18.412+0000 7f2f13fff640 1 --2- 172.21.15.114:0/4204900988 >> [v2:172.21.15.114:6800/465333808,v1:172.21.15.114:6801/465333808] conn(0x7f2f140a3bb0 0x7f2f140a6070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T10:24:18.412 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:24:18.412+0000 7f2f13fff640 1 -- 172.21.15.114:0/4204900988 <== mon.0 v2:172.21.15.114:3300/0 5 ==== osd_map(83..83 src has 1..83) ==== 7127+0+0 (secure 0 0 0) 0x7f2f240d04a0 con 0x7f2f2c107c70 2024-04-23T10:24:18.412 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:24:18.412+0000 7f2f2affd640 1 --2- 172.21.15.114:0/4204900988 >> [v2:172.21.15.114:6800/465333808,v1:172.21.15.114:6801/465333808] conn(0x7f2f140a3bb0 0x7f2f140a6070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T10:24:18.412 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:24:18.412+0000 7f2f2affd640 1 --2- 172.21.15.114:0/4204900988 >> [v2:172.21.15.114:6800/465333808,v1:172.21.15.114:6801/465333808] conn(0x7f2f140a3bb0 0x7f2f140a6070 secure :-1 s=READY pgs=141 cs=0 l=1 rev1=1 crypto rx=0x7f2f20000c00 tx=0x7f2f200314a0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T10:24:18.414 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:24:18.415+0000 7f2f13fff640 1 -- 172.21.15.114:0/4204900988 <== mon.0 v2:172.21.15.114:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f2f24016030 con 0x7f2f2c107c70 2024-04-23T10:24:18.431 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 23 10:24:18 smithi119 ceph-mon[32608]: Deploying daemon mds.e on smithi119 2024-04-23T10:24:18.431 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 23 10:24:18 smithi119 ceph-mon[32608]: mds.? [v2:172.21.15.114:6836/1439252154,v1:172.21.15.114:6837/1439252154] up:boot 2024-04-23T10:24:18.431 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 23 10:24:18 smithi119 ceph-mon[32608]: fsmap 4 up:standby 2024-04-23T10:24:18.431 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 23 10:24:18 smithi119 ceph-mon[32608]: from='mgr.14152 172.21.15.114:0/1529706138' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "d"}]: dispatch 2024-04-23T10:24:18.641 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:24:18.641+0000 7f2f31507640 1 -- 172.21.15.114:0/4204900988 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f2f2c02d5c0 con 0x7f2f2c107c70 2024-04-23T10:24:18.641 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:24:18.642+0000 7f2f13fff640 1 -- 172.21.15.114:0/4204900988 <== mon.0 v2:172.21.15.114:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 5 v5) ==== 93+0+3874 (secure 0 0 0) 0x7f2f2409ccd0 con 0x7f2f2c107c70 2024-04-23T10:24:18.641 INFO:teuthology.orchestra.run.smithi114.stdout: 2024-04-23T10:24:18.641 INFO:teuthology.orchestra.run.smithi114.stdout:{"epoch":5,"default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14547,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.114:6835/663493580","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6834","nonce":663493580},{"type":"v1","addr":"172.21.15.114:6835","nonce":663493580}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14565,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.114:6837/1439252154","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6836","nonce":1439252154},{"type":"v1","addr":"172.21.15.114:6837","nonce":1439252154}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5},{"gid":24385,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.204:6833/3866561754","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6832","nonce":3866561754},{"type":"v1","addr":"172.21.15.204:6833","nonce":3866561754}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2},{"gid":24443,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.119:6833/783244660","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6832","nonce":783244660},{"type":"v1","addr":"172.21.15.119:6833","nonce":783244660}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3}],"filesystems":[]} 2024-04-23T10:24:18.641 INFO:teuthology.orchestra.run.smithi114.stderr:dumped fsmap epoch 5 2024-04-23T10:24:18.643 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:24:18.644+0000 7f2f31507640 1 -- 172.21.15.114:0/4204900988 >> [v2:172.21.15.114:6800/465333808,v1:172.21.15.114:6801/465333808] conn(0x7f2f140a3bb0 msgr2=0x7f2f140a6070 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T10:24:18.643 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:24:18.644+0000 7f2f31507640 1 --2- 172.21.15.114:0/4204900988 >> [v2:172.21.15.114:6800/465333808,v1:172.21.15.114:6801/465333808] conn(0x7f2f140a3bb0 0x7f2f140a6070 secure :-1 s=READY pgs=141 cs=0 l=1 rev1=1 crypto rx=0x7f2f20000c00 tx=0x7f2f200314a0 comp rx=0 tx=0).stop 2024-04-23T10:24:18.643 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:24:18.644+0000 7f2f31507640 1 -- 172.21.15.114:0/4204900988 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f2f2c107c70 msgr2=0x7f2f2c19f7b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T10:24:18.643 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:24:18.644+0000 7f2f31507640 1 --2- 172.21.15.114:0/4204900988 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f2f2c107c70 0x7f2f2c19f7b0 secure :-1 s=READY pgs=176 cs=0 l=1 rev1=1 crypto rx=0x7f2f2400eaa0 tx=0x7f2f2400ef60 comp rx=0 tx=0).stop 2024-04-23T10:24:18.644 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:24:18.644+0000 7f2f31507640 1 -- 172.21.15.114:0/4204900988 shutdown_connections 2024-04-23T10:24:18.644 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:24:18.644+0000 7f2f31507640 1 --2- 172.21.15.114:0/4204900988 >> [v2:172.21.15.114:6800/465333808,v1:172.21.15.114:6801/465333808] conn(0x7f2f140a3bb0 0x7f2f140a6070 unknown :-1 s=CLOSED pgs=141 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:24:18.644 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:24:18.644+0000 7f2f31507640 1 --2- 172.21.15.114:0/4204900988 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f2f2c107c70 0x7f2f2c19f7b0 unknown :-1 s=CLOSED pgs=176 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:24:18.644 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:24:18.644+0000 7f2f31507640 1 --2- 172.21.15.114:0/4204900988 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f2f2c105340 0x7f2f2c198740 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:24:18.644 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:24:18.644+0000 7f2f31507640 1 --2- 172.21.15.114:0/4204900988 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f2f2c102a10 0x7f2f2c198200 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:24:18.644 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:24:18.644+0000 7f2f31507640 1 -- 172.21.15.114:0/4204900988 >> 172.21.15.114:0/4204900988 conn(0x7f2f2c0fc400 msgr2=0x7f2f2c103490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T10:24:18.644 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:24:18.644+0000 7f2f31507640 1 -- 172.21.15.114:0/4204900988 shutdown_connections 2024-04-23T10:24:18.644 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:24:18.644+0000 7f2f31507640 1 -- 172.21.15.114:0/4204900988 wait complete. 2024-04-23T10:24:18.742 INFO:journalctl@ceph.mds.e.smithi119.stdout:Apr 23 10:24:18 smithi119 systemd[1]: Starting Ceph mds.e for aa240572-0159-11ef-bc93-c7b262605968... 2024-04-23T10:24:19.111 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 23 10:24:19 smithi119 ceph-mon[32608]: pgmap v269: 1 pgs: 1 active+clean; 577 KiB data, 365 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-23T10:24:19.112 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 23 10:24:19 smithi119 ceph-mon[32608]: from='client.? 172.21.15.114:0/4204900988' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-23T10:24:19.112 INFO:journalctl@ceph.mds.e.smithi119.stdout:Apr 23 10:24:18 smithi119 podman[62048]: 2024-04-23 10:24:18.735419944 +0000 UTC m=+0.018054485 image pull da4b004c09e820713483d039bc76fbd660be678f2ea3c03e059a6161b74dab76 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:068a558e3af4932cf1c60dd321c0bdf500075eb52ea1bf455e0973ba114fbfd4 2024-04-23T10:24:19.112 INFO:journalctl@ceph.mds.e.smithi119.stdout:Apr 23 10:24:18 smithi119 podman[62048]: 2024-04-23 10:24:18.861255351 +0000 UTC m=+0.143889925 container create d5e5a1bdb2a73aa81c32c18b1ed7110745620f6bcb082deb1c33a64c184a08b7 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:068a558e3af4932cf1c60dd321c0bdf500075eb52ea1bf455e0973ba114fbfd4, name=ceph-aa240572-0159-11ef-bc93-c7b262605968-mds-e, GIT_CLEAN=True, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, io.buildah.version=1.33.5, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0, ceph=True, org.label-schema.build-date=20240416, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 8 Base Image, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, RELEASE=wip-lusov-quiescer-d69f13d, GIT_BRANCH=HEAD) 2024-04-23T10:24:19.328 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 23 10:24:19 smithi204 ceph-mon[32649]: pgmap v269: 1 pgs: 1 active+clean; 577 KiB data, 365 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-23T10:24:19.328 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 23 10:24:19 smithi204 ceph-mon[32649]: from='client.? 172.21.15.114:0/4204900988' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-23T10:24:19.492 INFO:journalctl@ceph.mds.e.smithi119.stdout:Apr 23 10:24:19 smithi119 podman[62048]: 2024-04-23 10:24:19.113616502 +0000 UTC m=+0.396251033 container init d5e5a1bdb2a73aa81c32c18b1ed7110745620f6bcb082deb1c33a64c184a08b7 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:068a558e3af4932cf1c60dd321c0bdf500075eb52ea1bf455e0973ba114fbfd4, name=ceph-aa240572-0159-11ef-bc93-c7b262605968-mds-e, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_CLEAN=True, RELEASE=wip-lusov-quiescer-d69f13d, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, org.label-schema.name=CentOS Stream 8 Base Image, io.buildah.version=1.33.5, ceph=True, GIT_BRANCH=HEAD, org.label-schema.build-date=20240416, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67) 2024-04-23T10:24:19.493 INFO:journalctl@ceph.mds.e.smithi119.stdout:Apr 23 10:24:19 smithi119 podman[62048]: 2024-04-23 10:24:19.116435951 +0000 UTC m=+0.399070492 container start d5e5a1bdb2a73aa81c32c18b1ed7110745620f6bcb082deb1c33a64c184a08b7 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:068a558e3af4932cf1c60dd321c0bdf500075eb52ea1bf455e0973ba114fbfd4, name=ceph-aa240572-0159-11ef-bc93-c7b262605968-mds-e, ceph=True, GIT_BRANCH=HEAD, GIT_CLEAN=True, RELEASE=wip-lusov-quiescer-d69f13d, org.label-schema.name=CentOS Stream 8 Base Image, io.buildah.version=1.33.5, org.label-schema.build-date=20240416, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67) 2024-04-23T10:24:19.493 INFO:journalctl@ceph.mds.e.smithi119.stdout:Apr 23 10:24:19 smithi119 ceph-mds[62067]: -- 172.21.15.119:0/3221685009 <== mon.2 v2:172.21.15.119:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 194+0+0 (secure 0 0 0) 0x55c68aa19a40 con 0x55c68a9ecc00 2024-04-23T10:24:19.493 INFO:journalctl@ceph.mds.e.smithi119.stdout:Apr 23 10:24:19 smithi119 ceph-aa240572-0159-11ef-bc93-c7b262605968-mds-e[62063]: starting mds.e at 2024-04-23T10:24:19.493 INFO:journalctl@ceph.mds.e.smithi119.stdout:Apr 23 10:24:19 smithi119 bash[62048]: d5e5a1bdb2a73aa81c32c18b1ed7110745620f6bcb082deb1c33a64c184a08b7 2024-04-23T10:24:19.493 INFO:journalctl@ceph.mds.e.smithi119.stdout:Apr 23 10:24:19 smithi119 systemd[1]: Started Ceph mds.e for aa240572-0159-11ef-bc93-c7b262605968. 2024-04-23T10:24:19.504 INFO:journalctl@ceph.mon.a.smithi114.stdout:Apr 23 10:24:19 smithi114 ceph-mon[29255]: pgmap v269: 1 pgs: 1 active+clean; 577 KiB data, 365 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-23T10:24:19.504 INFO:journalctl@ceph.mon.a.smithi114.stdout:Apr 23 10:24:19 smithi114 ceph-mon[29255]: from='client.? 172.21.15.114:0/4204900988' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-23T10:24:20.492 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 23 10:24:20 smithi119 ceph-mon[32608]: from='mgr.14152 172.21.15.114:0/1529706138' entity='mgr.x' 2024-04-23T10:24:20.493 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 23 10:24:20 smithi119 ceph-mon[32608]: from='mgr.14152 172.21.15.114:0/1529706138' entity='mgr.x' 2024-04-23T10:24:20.493 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 23 10:24:20 smithi119 ceph-mon[32608]: from='mgr.14152 172.21.15.114:0/1529706138' entity='mgr.x' 2024-04-23T10:24:20.493 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 23 10:24:20 smithi119 ceph-mon[32608]: from='mgr.14152 172.21.15.114:0/1529706138' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.f", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-04-23T10:24:20.493 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 23 10:24:20 smithi119 ceph-mon[32608]: from='mgr.14152 172.21.15.114:0/1529706138' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.f", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-04-23T10:24:20.493 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 23 10:24:20 smithi119 ceph-mon[32608]: from='mgr.14152 172.21.15.114:0/1529706138' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-23T10:24:20.493 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 23 10:24:20 smithi119 ceph-mon[32608]: Deploying daemon mds.f on smithi204 2024-04-23T10:24:20.493 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 23 10:24:20 smithi119 ceph-mon[32608]: pgmap v270: 1 pgs: 1 active+clean; 577 KiB data, 365 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-23T10:24:20.493 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 23 10:24:20 smithi119 ceph-mon[32608]: mds.? [v2:172.21.15.119:6834/2033916709,v1:172.21.15.119:6835/2033916709] up:boot 2024-04-23T10:24:20.493 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 23 10:24:20 smithi119 ceph-mon[32608]: fsmap 5 up:standby 2024-04-23T10:24:20.493 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 23 10:24:20 smithi119 ceph-mon[32608]: from='mgr.14152 172.21.15.114:0/1529706138' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "e"}]: dispatch 2024-04-23T10:24:20.504 INFO:journalctl@ceph.mon.a.smithi114.stdout:Apr 23 10:24:20 smithi114 ceph-mon[29255]: from='mgr.14152 172.21.15.114:0/1529706138' entity='mgr.x' 2024-04-23T10:24:20.504 INFO:journalctl@ceph.mon.a.smithi114.stdout:Apr 23 10:24:20 smithi114 ceph-mon[29255]: from='mgr.14152 172.21.15.114:0/1529706138' entity='mgr.x' 2024-04-23T10:24:20.504 INFO:journalctl@ceph.mon.a.smithi114.stdout:Apr 23 10:24:20 smithi114 ceph-mon[29255]: from='mgr.14152 172.21.15.114:0/1529706138' entity='mgr.x' 2024-04-23T10:24:20.504 INFO:journalctl@ceph.mon.a.smithi114.stdout:Apr 23 10:24:20 smithi114 ceph-mon[29255]: from='mgr.14152 172.21.15.114:0/1529706138' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.f", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-04-23T10:24:20.504 INFO:journalctl@ceph.mon.a.smithi114.stdout:Apr 23 10:24:20 smithi114 ceph-mon[29255]: from='mgr.14152 172.21.15.114:0/1529706138' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.f", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-04-23T10:24:20.504 INFO:journalctl@ceph.mon.a.smithi114.stdout:Apr 23 10:24:20 smithi114 ceph-mon[29255]: from='mgr.14152 172.21.15.114:0/1529706138' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-23T10:24:20.504 INFO:journalctl@ceph.mon.a.smithi114.stdout:Apr 23 10:24:20 smithi114 ceph-mon[29255]: Deploying daemon mds.f on smithi204 2024-04-23T10:24:20.504 INFO:journalctl@ceph.mon.a.smithi114.stdout:Apr 23 10:24:20 smithi114 ceph-mon[29255]: pgmap v270: 1 pgs: 1 active+clean; 577 KiB data, 365 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-23T10:24:20.505 INFO:journalctl@ceph.mon.a.smithi114.stdout:Apr 23 10:24:20 smithi114 ceph-mon[29255]: mds.? [v2:172.21.15.119:6834/2033916709,v1:172.21.15.119:6835/2033916709] up:boot 2024-04-23T10:24:20.505 INFO:journalctl@ceph.mon.a.smithi114.stdout:Apr 23 10:24:20 smithi114 ceph-mon[29255]: fsmap 5 up:standby 2024-04-23T10:24:20.505 INFO:journalctl@ceph.mon.a.smithi114.stdout:Apr 23 10:24:20 smithi114 ceph-mon[29255]: from='mgr.14152 172.21.15.114:0/1529706138' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "e"}]: dispatch 2024-04-23T10:24:20.578 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 23 10:24:20 smithi204 ceph-mon[32649]: from='mgr.14152 172.21.15.114:0/1529706138' entity='mgr.x' 2024-04-23T10:24:20.578 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 23 10:24:20 smithi204 ceph-mon[32649]: from='mgr.14152 172.21.15.114:0/1529706138' entity='mgr.x' 2024-04-23T10:24:20.578 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 23 10:24:20 smithi204 ceph-mon[32649]: from='mgr.14152 172.21.15.114:0/1529706138' entity='mgr.x' 2024-04-23T10:24:20.578 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 23 10:24:20 smithi204 ceph-mon[32649]: from='mgr.14152 172.21.15.114:0/1529706138' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.f", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-04-23T10:24:20.578 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 23 10:24:20 smithi204 ceph-mon[32649]: from='mgr.14152 172.21.15.114:0/1529706138' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.f", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-04-23T10:24:20.578 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 23 10:24:20 smithi204 ceph-mon[32649]: from='mgr.14152 172.21.15.114:0/1529706138' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-23T10:24:20.578 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 23 10:24:20 smithi204 ceph-mon[32649]: Deploying daemon mds.f on smithi204 2024-04-23T10:24:20.578 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 23 10:24:20 smithi204 ceph-mon[32649]: pgmap v270: 1 pgs: 1 active+clean; 577 KiB data, 365 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-23T10:24:20.578 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 23 10:24:20 smithi204 ceph-mon[32649]: mds.? [v2:172.21.15.119:6834/2033916709,v1:172.21.15.119:6835/2033916709] up:boot 2024-04-23T10:24:20.579 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 23 10:24:20 smithi204 ceph-mon[32649]: fsmap 5 up:standby 2024-04-23T10:24:20.579 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 23 10:24:20 smithi204 ceph-mon[32649]: from='mgr.14152 172.21.15.114:0/1529706138' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "e"}]: dispatch 2024-04-23T10:24:20.654 DEBUG:teuthology.orchestra.run.smithi114:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-23T10:24:20.766 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:24:20.765+0000 7f4a47944640 1 -- 172.21.15.114:0/4116560786 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f4a401029b0 msgr2=0x7f4a401092d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T10:24:20.766 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:24:20.765+0000 7f4a47944640 1 --2- 172.21.15.114:0/4116560786 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f4a401029b0 0x7f4a401092d0 secure :-1 s=READY pgs=177 cs=0 l=1 rev1=1 crypto rx=0x7f4a38009ca0 tx=0x7f4a3802a8c0 comp rx=0 tx=0).stop 2024-04-23T10:24:20.766 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:24:20.765+0000 7f4a47944640 1 -- 172.21.15.114:0/4116560786 shutdown_connections 2024-04-23T10:24:20.766 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:24:20.765+0000 7f4a47944640 1 --2- 172.21.15.114:0/4116560786 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f4a401029b0 0x7f4a401092d0 unknown :-1 s=CLOSED pgs=177 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:24:20.766 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:24:20.765+0000 7f4a47944640 1 --2- 172.21.15.114:0/4116560786 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f4a40101ff0 0x7f4a40102470 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:24:20.767 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:24:20.765+0000 7f4a47944640 1 --2- 172.21.15.114:0/4116560786 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f4a40100ea0 0x7f4a401012c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:24:20.767 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:24:20.765+0000 7f4a47944640 1 -- 172.21.15.114:0/4116560786 >> 172.21.15.114:0/4116560786 conn(0x7f4a400fc430 msgr2=0x7f4a400fe850 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T10:24:20.767 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:24:20.765+0000 7f4a47944640 1 -- 172.21.15.114:0/4116560786 shutdown_connections 2024-04-23T10:24:20.767 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:24:20.765+0000 7f4a47944640 1 -- 172.21.15.114:0/4116560786 wait complete. 2024-04-23T10:24:20.767 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:24:20.766+0000 7f4a47944640 1 Processor -- start 2024-04-23T10:24:20.767 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:24:20.766+0000 7f4a47944640 1 -- start start 2024-04-23T10:24:20.767 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:24:20.766+0000 7f4a47944640 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f4a40100ea0 0x7f4a401981a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T10:24:20.767 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:24:20.766+0000 7f4a47944640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f4a40101ff0 0x7f4a401986e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T10:24:20.767 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:24:20.766+0000 7f4a47944640 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f4a401029b0 0x7f4a4019f750 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T10:24:20.767 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:24:20.766+0000 7f4a47944640 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x7f4a4010ca00 con 0x7f4a40101ff0 2024-04-23T10:24:20.767 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:24:20.766+0000 7f4a47944640 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 -- 0x7f4a4010c880 con 0x7f4a401029b0 2024-04-23T10:24:20.767 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:24:20.766+0000 7f4a47944640 1 -- --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_getmap magic: 0 -- 0x7f4a4010cb80 con 0x7f4a40100ea0 2024-04-23T10:24:20.767 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:24:20.766+0000 7f4a44edf640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f4a40101ff0 0x7f4a401986e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T10:24:20.767 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:24:20.766+0000 7f4a44edf640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f4a40101ff0 0x7f4a401986e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.114:42614/0 (socket says 172.21.15.114:42614) 2024-04-23T10:24:20.767 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:24:20.766+0000 7f4a44edf640 1 -- 172.21.15.114:0/4190350639 learned_addr learned my addr 172.21.15.114:0/4190350639 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-04-23T10:24:20.768 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:24:20.766+0000 7f4a456e0640 1 --2- 172.21.15.114:0/4190350639 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f4a40100ea0 0x7f4a401981a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T10:24:20.768 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:24:20.766+0000 7f4a45ee1640 1 --2- 172.21.15.114:0/4190350639 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f4a401029b0 0x7f4a4019f750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T10:24:20.768 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:24:20.767+0000 7f4a44edf640 1 -- 172.21.15.114:0/4190350639 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f4a40100ea0 msgr2=0x7f4a401981a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T10:24:20.768 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:24:20.767+0000 7f4a44edf640 1 --2- 172.21.15.114:0/4190350639 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f4a40100ea0 0x7f4a401981a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:24:20.768 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:24:20.767+0000 7f4a44edf640 1 -- 172.21.15.114:0/4190350639 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f4a401029b0 msgr2=0x7f4a4019f750 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T10:24:20.768 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:24:20.767+0000 7f4a44edf640 1 --2- 172.21.15.114:0/4190350639 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f4a401029b0 0x7f4a4019f750 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:24:20.768 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:24:20.767+0000 7f4a44edf640 1 -- 172.21.15.114:0/4190350639 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f4a4019fdc0 con 0x7f4a40101ff0 2024-04-23T10:24:20.768 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:24:20.767+0000 7f4a45ee1640 1 --2- 172.21.15.114:0/4190350639 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f4a401029b0 0x7f4a4019f750 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-23T10:24:20.768 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:24:20.767+0000 7f4a44edf640 1 --2- 172.21.15.114:0/4190350639 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f4a40101ff0 0x7f4a401986e0 secure :-1 s=READY pgs=178 cs=0 l=1 rev1=1 crypto rx=0x7f4a3000d950 tx=0x7f4a3000de10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T10:24:20.768 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:24:20.767+0000 7f4a367fc640 1 -- 172.21.15.114:0/4190350639 <== mon.0 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4a30017020 con 0x7f4a40101ff0 2024-04-23T10:24:20.768 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:24:20.767+0000 7f4a367fc640 1 -- 172.21.15.114:0/4190350639 <== mon.0 v2:172.21.15.114:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f4a30009c30 con 0x7f4a40101ff0 2024-04-23T10:24:20.768 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:24:20.767+0000 7f4a47944640 1 -- 172.21.15.114:0/4190350639 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f4a401a00b0 con 0x7f4a40101ff0 2024-04-23T10:24:20.768 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:24:20.767+0000 7f4a367fc640 1 -- 172.21.15.114:0/4190350639 <== mon.0 v2:172.21.15.114:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4a3000f690 con 0x7f4a40101ff0 2024-04-23T10:24:20.768 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:24:20.767+0000 7f4a47944640 1 -- 172.21.15.114:0/4190350639 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f4a401a0600 con 0x7f4a40101ff0 2024-04-23T10:24:20.769 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:24:20.768+0000 7f4a47944640 1 -- 172.21.15.114:0/4190350639 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f4a401a08c0 con 0x7f4a40101ff0 2024-04-23T10:24:20.769 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:24:20.769+0000 7f4a367fc640 1 -- 172.21.15.114:0/4190350639 <== mon.0 v2:172.21.15.114:3300/0 4 ==== mgrmap(e 16) ==== 138647+0+0 (secure 0 0 0) 0x7f4a3001a020 con 0x7f4a40101ff0 2024-04-23T10:24:20.769 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:24:20.769+0000 7f4a367fc640 1 --2- 172.21.15.114:0/4190350639 >> [v2:172.21.15.114:6800/465333808,v1:172.21.15.114:6801/465333808] conn(0x7f4a2c0a3bb0 0x7f4a2c0a6070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T10:24:20.769 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:24:20.770+0000 7f4a456e0640 1 --2- 172.21.15.114:0/4190350639 >> [v2:172.21.15.114:6800/465333808,v1:172.21.15.114:6801/465333808] conn(0x7f4a2c0a3bb0 0x7f4a2c0a6070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T10:24:20.769 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:24:20.770+0000 7f4a367fc640 1 -- 172.21.15.114:0/4190350639 <== mon.0 v2:172.21.15.114:3300/0 5 ==== osd_map(83..83 src has 1..83) ==== 7127+0+0 (secure 0 0 0) 0x7f4a300cfe70 con 0x7f4a40101ff0 2024-04-23T10:24:20.770 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:24:20.770+0000 7f4a456e0640 1 --2- 172.21.15.114:0/4190350639 >> [v2:172.21.15.114:6800/465333808,v1:172.21.15.114:6801/465333808] conn(0x7f4a2c0a3bb0 0x7f4a2c0a6070 secure :-1 s=READY pgs=143 cs=0 l=1 rev1=1 crypto rx=0x7f4a3c009bd0 tx=0x7f4a3c005f90 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T10:24:20.773 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:24:20.774+0000 7f4a367fc640 1 -- 172.21.15.114:0/4190350639 <== mon.0 v2:172.21.15.114:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f4a30021020 con 0x7f4a40101ff0 2024-04-23T10:24:20.986 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:24:20.987+0000 7f4a47944640 1 -- 172.21.15.114:0/4190350639 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f4a401a0a90 con 0x7f4a40101ff0 2024-04-23T10:24:20.986 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:24:20.987+0000 7f4a367fc640 1 -- 172.21.15.114:0/4190350639 <== mon.0 v2:172.21.15.114:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 6 v6) ==== 93+0+4702 (secure 0 0 0) 0x7f4a30009de0 con 0x7f4a40101ff0 2024-04-23T10:24:20.986 INFO:teuthology.orchestra.run.smithi114.stdout: 2024-04-23T10:24:20.986 INFO:teuthology.orchestra.run.smithi114.stdout:{"epoch":6,"default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14547,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.114:6835/663493580","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6834","nonce":663493580},{"type":"v1","addr":"172.21.15.114:6835","nonce":663493580}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14565,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.114:6837/1439252154","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6836","nonce":1439252154},{"type":"v1","addr":"172.21.15.114:6837","nonce":1439252154}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5},{"gid":24385,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.204:6833/3866561754","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6832","nonce":3866561754},{"type":"v1","addr":"172.21.15.204:6833","nonce":3866561754}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2},{"gid":24443,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.119:6833/783244660","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6832","nonce":783244660},{"type":"v1","addr":"172.21.15.119:6833","nonce":783244660}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":24479,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.119:6835/2033916709","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6834","nonce":2033916709},{"type":"v1","addr":"172.21.15.119:6835","nonce":2033916709}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":6}],"filesystems":[]} 2024-04-23T10:24:20.987 INFO:teuthology.orchestra.run.smithi114.stderr:dumped fsmap epoch 6 2024-04-23T10:24:20.988 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:24:20.989+0000 7f4a47944640 1 -- 172.21.15.114:0/4190350639 >> [v2:172.21.15.114:6800/465333808,v1:172.21.15.114:6801/465333808] conn(0x7f4a2c0a3bb0 msgr2=0x7f4a2c0a6070 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T10:24:20.989 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:24:20.989+0000 7f4a47944640 1 --2- 172.21.15.114:0/4190350639 >> [v2:172.21.15.114:6800/465333808,v1:172.21.15.114:6801/465333808] conn(0x7f4a2c0a3bb0 0x7f4a2c0a6070 secure :-1 s=READY pgs=143 cs=0 l=1 rev1=1 crypto rx=0x7f4a3c009bd0 tx=0x7f4a3c005f90 comp rx=0 tx=0).stop 2024-04-23T10:24:20.989 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:24:20.989+0000 7f4a47944640 1 -- 172.21.15.114:0/4190350639 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f4a40101ff0 msgr2=0x7f4a401986e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T10:24:20.989 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:24:20.989+0000 7f4a47944640 1 --2- 172.21.15.114:0/4190350639 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f4a40101ff0 0x7f4a401986e0 secure :-1 s=READY pgs=178 cs=0 l=1 rev1=1 crypto rx=0x7f4a3000d950 tx=0x7f4a3000de10 comp rx=0 tx=0).stop 2024-04-23T10:24:20.989 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:24:20.989+0000 7f4a47944640 1 -- 172.21.15.114:0/4190350639 shutdown_connections 2024-04-23T10:24:20.989 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:24:20.989+0000 7f4a47944640 1 --2- 172.21.15.114:0/4190350639 >> [v2:172.21.15.114:6800/465333808,v1:172.21.15.114:6801/465333808] conn(0x7f4a2c0a3bb0 0x7f4a2c0a6070 unknown :-1 s=CLOSED pgs=143 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:24:20.989 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:24:20.989+0000 7f4a47944640 1 --2- 172.21.15.114:0/4190350639 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f4a401029b0 0x7f4a4019f750 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:24:20.989 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:24:20.989+0000 7f4a47944640 1 --2- 172.21.15.114:0/4190350639 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f4a40101ff0 0x7f4a401986e0 unknown :-1 s=CLOSED pgs=178 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:24:20.989 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:24:20.989+0000 7f4a47944640 1 --2- 172.21.15.114:0/4190350639 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f4a40100ea0 0x7f4a401981a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:24:20.989 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:24:20.989+0000 7f4a47944640 1 -- 172.21.15.114:0/4190350639 >> 172.21.15.114:0/4190350639 conn(0x7f4a400fc430 msgr2=0x7f4a400fdef0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T10:24:20.989 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:24:20.989+0000 7f4a47944640 1 -- 172.21.15.114:0/4190350639 shutdown_connections 2024-04-23T10:24:20.989 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:24:20.989+0000 7f4a47944640 1 -- 172.21.15.114:0/4190350639 wait complete. 2024-04-23T10:24:21.578 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 23 10:24:21 smithi204 ceph-mon[32649]: from='client.? 172.21.15.114:0/4190350639' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-23T10:24:21.578 INFO:journalctl@ceph.mds.f.smithi204.stdout:Apr 23 10:24:21 smithi204 systemd[1]: Starting Ceph mds.f for aa240572-0159-11ef-bc93-c7b262605968... 2024-04-23T10:24:21.742 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 23 10:24:21 smithi119 ceph-mon[32608]: from='client.? 172.21.15.114:0/4190350639' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-23T10:24:21.754 INFO:journalctl@ceph.mon.a.smithi114.stdout:Apr 23 10:24:21 smithi114 ceph-mon[29255]: from='client.? 172.21.15.114:0/4190350639' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-23T10:24:21.835 INFO:journalctl@ceph.mds.f.smithi204.stdout:Apr 23 10:24:21 smithi204 podman[62663]: 2024-04-23 10:24:21.487111434 +0000 UTC m=+0.018474307 image pull da4b004c09e820713483d039bc76fbd660be678f2ea3c03e059a6161b74dab76 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:068a558e3af4932cf1c60dd321c0bdf500075eb52ea1bf455e0973ba114fbfd4 2024-04-23T10:24:21.835 INFO:journalctl@ceph.mds.f.smithi204.stdout:Apr 23 10:24:21 smithi204 podman[62663]: 2024-04-23 10:24:21.591477954 +0000 UTC m=+0.122840813 container create 4fd9d2764c4fb78010077cffd6d0ec8d8ec5fcfe8503b3bd2e85eac2f95b4010 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:068a558e3af4932cf1c60dd321c0bdf500075eb52ea1bf455e0973ba114fbfd4, name=ceph-aa240572-0159-11ef-bc93-c7b262605968-mds-f, RELEASE=wip-lusov-quiescer-d69f13d, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 8 Base Image, ceph=True, GIT_CLEAN=True, io.buildah.version=1.33.5, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240416, CEPH_POINT_RELEASE=, org.label-schema.license=GPLv2) 2024-04-23T10:24:21.835 INFO:journalctl@ceph.mds.f.smithi204.stdout:Apr 23 10:24:21 smithi204 podman[62663]: 2024-04-23 10:24:21.810846599 +0000 UTC m=+0.342209461 container init 4fd9d2764c4fb78010077cffd6d0ec8d8ec5fcfe8503b3bd2e85eac2f95b4010 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:068a558e3af4932cf1c60dd321c0bdf500075eb52ea1bf455e0973ba114fbfd4, name=ceph-aa240572-0159-11ef-bc93-c7b262605968-mds-f, ceph=True, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240416, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS, io.buildah.version=1.33.5, RELEASE=wip-lusov-quiescer-d69f13d, org.label-schema.schema-version=1.0, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, GIT_CLEAN=True, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 8 Base Image) 2024-04-23T10:24:21.835 INFO:journalctl@ceph.mds.f.smithi204.stdout:Apr 23 10:24:21 smithi204 podman[62663]: 2024-04-23 10:24:21.813318833 +0000 UTC m=+0.344681697 container start 4fd9d2764c4fb78010077cffd6d0ec8d8ec5fcfe8503b3bd2e85eac2f95b4010 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:068a558e3af4932cf1c60dd321c0bdf500075eb52ea1bf455e0973ba114fbfd4, name=ceph-aa240572-0159-11ef-bc93-c7b262605968-mds-f, RELEASE=wip-lusov-quiescer-d69f13d, org.label-schema.build-date=20240416, ceph=True, CEPH_POINT_RELEASE=, org.label-schema.license=GPLv2, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS, org.label-schema.name=CentOS Stream 8 Base Image, GIT_CLEAN=True, GIT_BRANCH=HEAD, io.buildah.version=1.33.5) 2024-04-23T10:24:21.836 INFO:journalctl@ceph.mds.f.smithi204.stdout:Apr 23 10:24:21 smithi204 ceph-mds[62682]: -- 172.21.15.204:0/4019874088 <== mon.1 v2:172.21.15.204:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 194+0+0 (secure 0 0 0) 0x56088d339a40 con 0x56088d30d000 2024-04-23T10:24:22.328 INFO:journalctl@ceph.mds.f.smithi204.stdout:Apr 23 10:24:21 smithi204 ceph-aa240572-0159-11ef-bc93-c7b262605968-mds-f[62678]: starting mds.f at 2024-04-23T10:24:22.328 INFO:journalctl@ceph.mds.f.smithi204.stdout:Apr 23 10:24:21 smithi204 bash[62663]: 4fd9d2764c4fb78010077cffd6d0ec8d8ec5fcfe8503b3bd2e85eac2f95b4010 2024-04-23T10:24:22.328 INFO:journalctl@ceph.mds.f.smithi204.stdout:Apr 23 10:24:21 smithi204 systemd[1]: Started Ceph mds.f for aa240572-0159-11ef-bc93-c7b262605968. 2024-04-23T10:24:22.992 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 23 10:24:22 smithi119 ceph-mon[32608]: from='mgr.14152 172.21.15.114:0/1529706138' entity='mgr.x' 2024-04-23T10:24:22.992 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 23 10:24:22 smithi119 ceph-mon[32608]: from='mgr.14152 172.21.15.114:0/1529706138' entity='mgr.x' 2024-04-23T10:24:22.992 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 23 10:24:22 smithi119 ceph-mon[32608]: pgmap v271: 1 pgs: 1 active+clean; 577 KiB data, 365 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-23T10:24:22.992 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 23 10:24:22 smithi119 ceph-mon[32608]: from='mgr.14152 172.21.15.114:0/1529706138' entity='mgr.x' 2024-04-23T10:24:22.993 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 23 10:24:22 smithi119 ceph-mon[32608]: from='mgr.14152 172.21.15.114:0/1529706138' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.c", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-04-23T10:24:22.993 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 23 10:24:22 smithi119 ceph-mon[32608]: from='mgr.14152 172.21.15.114:0/1529706138' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.c", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-04-23T10:24:22.993 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 23 10:24:22 smithi119 ceph-mon[32608]: from='mgr.14152 172.21.15.114:0/1529706138' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-23T10:24:22.993 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 23 10:24:22 smithi119 ceph-mon[32608]: Deploying daemon mds.c on smithi204 2024-04-23T10:24:22.993 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 23 10:24:22 smithi119 ceph-mon[32608]: mds.? [v2:172.21.15.204:6834/2533793172,v1:172.21.15.204:6835/2533793172] up:boot 2024-04-23T10:24:22.993 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 23 10:24:22 smithi119 ceph-mon[32608]: fsmap 6 up:standby 2024-04-23T10:24:22.993 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 23 10:24:22 smithi119 ceph-mon[32608]: from='mgr.14152 172.21.15.114:0/1529706138' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "f"}]: dispatch 2024-04-23T10:24:22.999 DEBUG:teuthology.orchestra.run.smithi114:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-23T10:24:23.111 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:24:23.110+0000 7f27178b9640 1 -- 172.21.15.114:0/884398278 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f2710105b40 msgr2=0x7f2710105fc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T10:24:23.112 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:24:23.110+0000 7f27178b9640 1 --2- 172.21.15.114:0/884398278 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f2710105b40 0x7f2710105fc0 secure :-1 s=READY pgs=179 cs=0 l=1 rev1=1 crypto rx=0x7f2708009ca0 tx=0x7f270802a8c0 comp rx=0 tx=0).stop 2024-04-23T10:24:23.112 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:24:23.110+0000 7f27178b9640 1 -- 172.21.15.114:0/884398278 shutdown_connections 2024-04-23T10:24:23.112 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:24:23.110+0000 7f27178b9640 1 --2- 172.21.15.114:0/884398278 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f2710105b40 0x7f2710105fc0 unknown :-1 s=CLOSED pgs=179 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:24:23.112 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:24:23.110+0000 7f27178b9640 1 --2- 172.21.15.114:0/884398278 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f2710103190 0x7f2710105600 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:24:23.112 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:24:23.110+0000 7f27178b9640 1 --2- 172.21.15.114:0/884398278 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f2710100820 0x7f2710102c50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:24:23.112 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:24:23.110+0000 7f27178b9640 1 -- 172.21.15.114:0/884398278 >> 172.21.15.114:0/884398278 conn(0x7f27100fc440 msgr2=0x7f27100fe860 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T10:24:23.112 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:24:23.110+0000 7f27178b9640 1 -- 172.21.15.114:0/884398278 shutdown_connections 2024-04-23T10:24:23.112 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:24:23.110+0000 7f27178b9640 1 -- 172.21.15.114:0/884398278 wait complete. 2024-04-23T10:24:23.112 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:24:23.111+0000 7f27178b9640 1 Processor -- start 2024-04-23T10:24:23.112 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:24:23.111+0000 7f27178b9640 1 -- start start 2024-04-23T10:24:23.112 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:24:23.111+0000 7f27178b9640 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f2710100820 0x7f271019e730 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T10:24:23.112 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:24:23.111+0000 7f27178b9640 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f2710103190 0x7f271019ec70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T10:24:23.112 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:24:23.111+0000 7f27178b9640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f2710105b40 0x7f27101a5ce0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T10:24:23.113 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:24:23.111+0000 7f27178b9640 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x7f27101140b0 con 0x7f2710105b40 2024-04-23T10:24:23.113 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:24:23.111+0000 7f27178b9640 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 -- 0x7f2710113f30 con 0x7f2710100820 2024-04-23T10:24:23.113 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:24:23.111+0000 7f27178b9640 1 -- --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_getmap magic: 0 -- 0x7f2710114230 con 0x7f2710103190 2024-04-23T10:24:23.113 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:24:23.111+0000 7f2715e56640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f2710105b40 0x7f27101a5ce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T10:24:23.113 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:24:23.111+0000 7f2715e56640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f2710105b40 0x7f27101a5ce0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.114:42634/0 (socket says 172.21.15.114:42634) 2024-04-23T10:24:23.113 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:24:23.111+0000 7f2715e56640 1 -- 172.21.15.114:0/2091763711 learned_addr learned my addr 172.21.15.114:0/2091763711 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-04-23T10:24:23.113 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:24:23.111+0000 7f2714e54640 1 --2- 172.21.15.114:0/2091763711 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f2710103190 0x7f271019ec70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T10:24:23.113 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:24:23.111+0000 7f2715655640 1 --2- 172.21.15.114:0/2091763711 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f2710100820 0x7f271019e730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T10:24:23.113 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:24:23.111+0000 7f2715e56640 1 -- 172.21.15.114:0/2091763711 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f2710103190 msgr2=0x7f271019ec70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T10:24:23.113 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:24:23.111+0000 7f2715e56640 1 --2- 172.21.15.114:0/2091763711 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f2710103190 0x7f271019ec70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:24:23.113 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:24:23.111+0000 7f2715e56640 1 -- 172.21.15.114:0/2091763711 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f2710100820 msgr2=0x7f271019e730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T10:24:23.113 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:24:23.111+0000 7f2715e56640 1 --2- 172.21.15.114:0/2091763711 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f2710100820 0x7f271019e730 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:24:23.113 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:24:23.111+0000 7f2715e56640 1 -- 172.21.15.114:0/2091763711 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f27101a6350 con 0x7f2710105b40 2024-04-23T10:24:23.113 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:24:23.111+0000 7f2714e54640 1 --2- 172.21.15.114:0/2091763711 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f2710103190 0x7f271019ec70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-23T10:24:23.114 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:24:23.112+0000 7f2715e56640 1 --2- 172.21.15.114:0/2091763711 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f2710105b40 0x7f27101a5ce0 secure :-1 s=READY pgs=180 cs=0 l=1 rev1=1 crypto rx=0x7f27080097b0 tx=0x7f2708003c20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T10:24:23.114 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:24:23.112+0000 7f27067fc640 1 -- 172.21.15.114:0/2091763711 <== mon.0 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2708004240 con 0x7f2710105b40 2024-04-23T10:24:23.114 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:24:23.112+0000 7f27067fc640 1 -- 172.21.15.114:0/2091763711 <== mon.0 v2:172.21.15.114:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f27080317c0 con 0x7f2710105b40 2024-04-23T10:24:23.114 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:24:23.112+0000 7f27067fc640 1 -- 172.21.15.114:0/2091763711 <== mon.0 v2:172.21.15.114:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f270802fd80 con 0x7f2710105b40 2024-04-23T10:24:23.114 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:24:23.112+0000 7f27178b9640 1 -- 172.21.15.114:0/2091763711 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f27101a65e0 con 0x7f2710105b40 2024-04-23T10:24:23.114 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:24:23.112+0000 7f27178b9640 1 -- 172.21.15.114:0/2091763711 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f27101a6b30 con 0x7f2710105b40 2024-04-23T10:24:23.114 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:24:23.113+0000 7f27178b9640 1 -- 172.21.15.114:0/2091763711 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f27100fdf30 con 0x7f2710105b40 2024-04-23T10:24:23.115 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:24:23.114+0000 7f27067fc640 1 -- 172.21.15.114:0/2091763711 <== mon.0 v2:172.21.15.114:3300/0 4 ==== mgrmap(e 16) ==== 138647+0+0 (secure 0 0 0) 0x7f27080043e0 con 0x7f2710105b40 2024-04-23T10:24:23.115 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:24:23.114+0000 7f27067fc640 1 --2- 172.21.15.114:0/2091763711 >> [v2:172.21.15.114:6800/465333808,v1:172.21.15.114:6801/465333808] conn(0x7f26fc0a3bb0 0x7f26fc0a6070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T10:24:23.115 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:24:23.114+0000 7f2715655640 1 --2- 172.21.15.114:0/2091763711 >> [v2:172.21.15.114:6800/465333808,v1:172.21.15.114:6801/465333808] conn(0x7f26fc0a3bb0 0x7f26fc0a6070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T10:24:23.115 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:24:23.114+0000 7f27067fc640 1 -- 172.21.15.114:0/2091763711 <== mon.0 v2:172.21.15.114:3300/0 5 ==== osd_map(83..83 src has 1..83) ==== 7127+0+0 (secure 0 0 0) 0x7f27080f17f0 con 0x7f2710105b40 2024-04-23T10:24:23.115 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:24:23.115+0000 7f2715655640 1 --2- 172.21.15.114:0/2091763711 >> [v2:172.21.15.114:6800/465333808,v1:172.21.15.114:6801/465333808] conn(0x7f26fc0a3bb0 0x7f26fc0a6070 secure :-1 s=READY pgs=145 cs=0 l=1 rev1=1 crypto rx=0x7f270000bbe0 tx=0x7f270000b390 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T10:24:23.118 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:24:23.119+0000 7f27067fc640 1 -- 172.21.15.114:0/2091763711 <== mon.0 v2:172.21.15.114:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f27080be670 con 0x7f2710105b40 2024-04-23T10:24:23.254 INFO:journalctl@ceph.mon.a.smithi114.stdout:Apr 23 10:24:22 smithi114 ceph-mon[29255]: from='mgr.14152 172.21.15.114:0/1529706138' entity='mgr.x' 2024-04-23T10:24:23.254 INFO:journalctl@ceph.mon.a.smithi114.stdout:Apr 23 10:24:22 smithi114 ceph-mon[29255]: from='mgr.14152 172.21.15.114:0/1529706138' entity='mgr.x' 2024-04-23T10:24:23.254 INFO:journalctl@ceph.mon.a.smithi114.stdout:Apr 23 10:24:22 smithi114 ceph-mon[29255]: pgmap v271: 1 pgs: 1 active+clean; 577 KiB data, 365 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-23T10:24:23.254 INFO:journalctl@ceph.mon.a.smithi114.stdout:Apr 23 10:24:22 smithi114 ceph-mon[29255]: from='mgr.14152 172.21.15.114:0/1529706138' entity='mgr.x' 2024-04-23T10:24:23.254 INFO:journalctl@ceph.mon.a.smithi114.stdout:Apr 23 10:24:22 smithi114 ceph-mon[29255]: from='mgr.14152 172.21.15.114:0/1529706138' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.c", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-04-23T10:24:23.254 INFO:journalctl@ceph.mon.a.smithi114.stdout:Apr 23 10:24:22 smithi114 ceph-mon[29255]: from='mgr.14152 172.21.15.114:0/1529706138' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.c", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-04-23T10:24:23.254 INFO:journalctl@ceph.mon.a.smithi114.stdout:Apr 23 10:24:22 smithi114 ceph-mon[29255]: from='mgr.14152 172.21.15.114:0/1529706138' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-23T10:24:23.254 INFO:journalctl@ceph.mon.a.smithi114.stdout:Apr 23 10:24:22 smithi114 ceph-mon[29255]: Deploying daemon mds.c on smithi204 2024-04-23T10:24:23.255 INFO:journalctl@ceph.mon.a.smithi114.stdout:Apr 23 10:24:22 smithi114 ceph-mon[29255]: mds.? [v2:172.21.15.204:6834/2533793172,v1:172.21.15.204:6835/2533793172] up:boot 2024-04-23T10:24:23.255 INFO:journalctl@ceph.mon.a.smithi114.stdout:Apr 23 10:24:22 smithi114 ceph-mon[29255]: fsmap 6 up:standby 2024-04-23T10:24:23.255 INFO:journalctl@ceph.mon.a.smithi114.stdout:Apr 23 10:24:22 smithi114 ceph-mon[29255]: from='mgr.14152 172.21.15.114:0/1529706138' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "f"}]: dispatch 2024-04-23T10:24:23.328 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 23 10:24:22 smithi204 ceph-mon[32649]: from='mgr.14152 172.21.15.114:0/1529706138' entity='mgr.x' 2024-04-23T10:24:23.328 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 23 10:24:22 smithi204 ceph-mon[32649]: from='mgr.14152 172.21.15.114:0/1529706138' entity='mgr.x' 2024-04-23T10:24:23.328 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 23 10:24:22 smithi204 ceph-mon[32649]: pgmap v271: 1 pgs: 1 active+clean; 577 KiB data, 365 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-23T10:24:23.328 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 23 10:24:22 smithi204 ceph-mon[32649]: from='mgr.14152 172.21.15.114:0/1529706138' entity='mgr.x' 2024-04-23T10:24:23.328 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 23 10:24:22 smithi204 ceph-mon[32649]: from='mgr.14152 172.21.15.114:0/1529706138' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.c", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-04-23T10:24:23.328 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 23 10:24:22 smithi204 ceph-mon[32649]: from='mgr.14152 172.21.15.114:0/1529706138' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.c", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-04-23T10:24:23.328 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 23 10:24:22 smithi204 ceph-mon[32649]: from='mgr.14152 172.21.15.114:0/1529706138' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-23T10:24:23.328 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 23 10:24:22 smithi204 ceph-mon[32649]: Deploying daemon mds.c on smithi204 2024-04-23T10:24:23.328 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 23 10:24:22 smithi204 ceph-mon[32649]: mds.? [v2:172.21.15.204:6834/2533793172,v1:172.21.15.204:6835/2533793172] up:boot 2024-04-23T10:24:23.328 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 23 10:24:22 smithi204 ceph-mon[32649]: fsmap 6 up:standby 2024-04-23T10:24:23.328 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 23 10:24:22 smithi204 ceph-mon[32649]: from='mgr.14152 172.21.15.114:0/1529706138' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "f"}]: dispatch 2024-04-23T10:24:23.333 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:24:23.334+0000 7f27178b9640 1 -- 172.21.15.114:0/2091763711 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f2710062b50 con 0x7f2710105b40 2024-04-23T10:24:23.333 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:24:23.335+0000 7f27067fc640 1 -- 172.21.15.114:0/2091763711 <== mon.0 v2:172.21.15.114:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 7 v7) ==== 93+0+5530 (secure 0 0 0) 0x7f27080098b0 con 0x7f2710105b40 2024-04-23T10:24:23.334 INFO:teuthology.orchestra.run.smithi114.stdout: 2024-04-23T10:24:23.334 INFO:teuthology.orchestra.run.smithi114.stdout:{"epoch":7,"default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14547,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.114:6835/663493580","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6834","nonce":663493580},{"type":"v1","addr":"172.21.15.114:6835","nonce":663493580}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14565,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.114:6837/1439252154","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6836","nonce":1439252154},{"type":"v1","addr":"172.21.15.114:6837","nonce":1439252154}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5},{"gid":24385,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.204:6833/3866561754","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6832","nonce":3866561754},{"type":"v1","addr":"172.21.15.204:6833","nonce":3866561754}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2},{"gid":24427,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.204:6835/2533793172","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6834","nonce":2533793172},{"type":"v1","addr":"172.21.15.204:6835","nonce":2533793172}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":7},{"gid":24443,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.119:6833/783244660","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6832","nonce":783244660},{"type":"v1","addr":"172.21.15.119:6833","nonce":783244660}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":24479,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.119:6835/2033916709","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6834","nonce":2033916709},{"type":"v1","addr":"172.21.15.119:6835","nonce":2033916709}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":6}],"filesystems":[]} 2024-04-23T10:24:23.334 INFO:teuthology.orchestra.run.smithi114.stderr:dumped fsmap epoch 7 2024-04-23T10:24:23.336 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:24:23.337+0000 7f27178b9640 1 -- 172.21.15.114:0/2091763711 >> [v2:172.21.15.114:6800/465333808,v1:172.21.15.114:6801/465333808] conn(0x7f26fc0a3bb0 msgr2=0x7f26fc0a6070 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T10:24:23.336 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:24:23.337+0000 7f27178b9640 1 --2- 172.21.15.114:0/2091763711 >> [v2:172.21.15.114:6800/465333808,v1:172.21.15.114:6801/465333808] conn(0x7f26fc0a3bb0 0x7f26fc0a6070 secure :-1 s=READY pgs=145 cs=0 l=1 rev1=1 crypto rx=0x7f270000bbe0 tx=0x7f270000b390 comp rx=0 tx=0).stop 2024-04-23T10:24:23.336 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:24:23.337+0000 7f27178b9640 1 -- 172.21.15.114:0/2091763711 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f2710105b40 msgr2=0x7f27101a5ce0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T10:24:23.336 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:24:23.337+0000 7f27178b9640 1 --2- 172.21.15.114:0/2091763711 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f2710105b40 0x7f27101a5ce0 secure :-1 s=READY pgs=180 cs=0 l=1 rev1=1 crypto rx=0x7f27080097b0 tx=0x7f2708003c20 comp rx=0 tx=0).stop 2024-04-23T10:24:23.336 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:24:23.337+0000 7f27178b9640 1 -- 172.21.15.114:0/2091763711 shutdown_connections 2024-04-23T10:24:23.336 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:24:23.337+0000 7f27178b9640 1 --2- 172.21.15.114:0/2091763711 >> [v2:172.21.15.114:6800/465333808,v1:172.21.15.114:6801/465333808] conn(0x7f26fc0a3bb0 0x7f26fc0a6070 unknown :-1 s=CLOSED pgs=145 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:24:23.336 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:24:23.337+0000 7f27178b9640 1 --2- 172.21.15.114:0/2091763711 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f2710105b40 0x7f27101a5ce0 unknown :-1 s=CLOSED pgs=180 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:24:23.336 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:24:23.337+0000 7f27178b9640 1 --2- 172.21.15.114:0/2091763711 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f2710103190 0x7f271019ec70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:24:23.337 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:24:23.337+0000 7f27178b9640 1 --2- 172.21.15.114:0/2091763711 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f2710100820 0x7f271019e730 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:24:23.337 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:24:23.337+0000 7f27178b9640 1 -- 172.21.15.114:0/2091763711 >> 172.21.15.114:0/2091763711 conn(0x7f27100fc440 msgr2=0x7f27101011a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T10:24:23.337 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:24:23.337+0000 7f27178b9640 1 -- 172.21.15.114:0/2091763711 shutdown_connections 2024-04-23T10:24:23.337 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:24:23.337+0000 7f27178b9640 1 -- 172.21.15.114:0/2091763711 wait complete. 2024-04-23T10:24:24.242 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 23 10:24:23 smithi119 ceph-mon[32608]: from='client.? 172.21.15.114:0/2091763711' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-23T10:24:24.254 INFO:journalctl@ceph.mon.a.smithi114.stdout:Apr 23 10:24:23 smithi114 ceph-mon[29255]: from='client.? 172.21.15.114:0/2091763711' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-23T10:24:24.278 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 23 10:24:23 smithi204 ceph-mon[32649]: from='client.? 172.21.15.114:0/2091763711' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-23T10:24:24.279 INFO:journalctl@ceph.mds.c.smithi204.stdout:Apr 23 10:24:24 smithi204 systemd[1]: Starting Ceph mds.c for aa240572-0159-11ef-bc93-c7b262605968... 2024-04-23T10:24:24.530 INFO:journalctl@ceph.mds.c.smithi204.stdout:Apr 23 10:24:24 smithi204 podman[62975]: 2024-04-23 10:24:24.18307136 +0000 UTC m=+0.017955864 image pull da4b004c09e820713483d039bc76fbd660be678f2ea3c03e059a6161b74dab76 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:068a558e3af4932cf1c60dd321c0bdf500075eb52ea1bf455e0973ba114fbfd4 2024-04-23T10:24:24.530 INFO:journalctl@ceph.mds.c.smithi204.stdout:Apr 23 10:24:24 smithi204 podman[62975]: 2024-04-23 10:24:24.303738309 +0000 UTC m=+0.138622824 container create 084acf2b16c700cde87a30383114cd49b4d1e8dfe3c3df1fa7a6952468a190b9 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:068a558e3af4932cf1c60dd321c0bdf500075eb52ea1bf455e0973ba114fbfd4, name=ceph-aa240572-0159-11ef-bc93-c7b262605968-mds-c, GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2, io.buildah.version=1.33.5, RELEASE=wip-lusov-quiescer-d69f13d, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0, GIT_CLEAN=True, ceph=True, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, org.label-schema.build-date=20240416, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 8 Base Image) 2024-04-23T10:24:24.530 INFO:journalctl@ceph.mds.c.smithi204.stdout:Apr 23 10:24:24 smithi204 podman[62975]: 2024-04-23 10:24:24.531415757 +0000 UTC m=+0.366300234 container init 084acf2b16c700cde87a30383114cd49b4d1e8dfe3c3df1fa7a6952468a190b9 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:068a558e3af4932cf1c60dd321c0bdf500075eb52ea1bf455e0973ba114fbfd4, name=ceph-aa240572-0159-11ef-bc93-c7b262605968-mds-c, org.label-schema.vendor=CentOS, org.label-schema.build-date=20240416, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, RELEASE=wip-lusov-quiescer-d69f13d, io.buildah.version=1.33.5, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, ceph=True, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.schema-version=1.0, GIT_CLEAN=True, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-04-23T10:24:24.828 INFO:journalctl@ceph.mds.c.smithi204.stdout:Apr 23 10:24:24 smithi204 podman[62975]: 2024-04-23 10:24:24.534261971 +0000 UTC m=+0.369146443 container start 084acf2b16c700cde87a30383114cd49b4d1e8dfe3c3df1fa7a6952468a190b9 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:068a558e3af4932cf1c60dd321c0bdf500075eb52ea1bf455e0973ba114fbfd4, name=ceph-aa240572-0159-11ef-bc93-c7b262605968-mds-c, org.label-schema.vendor=CentOS, ceph=True, org.label-schema.schema-version=1.0, GIT_CLEAN=True, io.buildah.version=1.33.5, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux , GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, org.label-schema.build-date=20240416, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 8 Base Image, GIT_BRANCH=HEAD, org.label-schema.license=GPLv2, RELEASE=wip-lusov-quiescer-d69f13d) 2024-04-23T10:24:24.828 INFO:journalctl@ceph.mds.c.smithi204.stdout:Apr 23 10:24:24 smithi204 ceph-mds[62994]: -- 172.21.15.204:0/792201730 <== mon.1 v2:172.21.15.204:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 194+0+0 (secure 0 0 0) 0x559318763a40 con 0x559318737000 2024-04-23T10:24:24.828 INFO:journalctl@ceph.mds.c.smithi204.stdout:Apr 23 10:24:24 smithi204 ceph-aa240572-0159-11ef-bc93-c7b262605968-mds-c[62990]: starting mds.c at 2024-04-23T10:24:24.828 INFO:journalctl@ceph.mds.c.smithi204.stdout:Apr 23 10:24:24 smithi204 bash[62975]: 084acf2b16c700cde87a30383114cd49b4d1e8dfe3c3df1fa7a6952468a190b9 2024-04-23T10:24:24.828 INFO:journalctl@ceph.mds.c.smithi204.stdout:Apr 23 10:24:24 smithi204 systemd[1]: Started Ceph mds.c for aa240572-0159-11ef-bc93-c7b262605968. 2024-04-23T10:24:25.242 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 23 10:24:24 smithi119 ceph-mon[32608]: pgmap v272: 1 pgs: 1 active+clean; 577 KiB data, 365 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-23T10:24:25.242 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 23 10:24:24 smithi119 ceph-mon[32608]: from='mgr.14152 172.21.15.114:0/1529706138' entity='mgr.x' 2024-04-23T10:24:25.242 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 23 10:24:24 smithi119 ceph-mon[32608]: from='mgr.14152 172.21.15.114:0/1529706138' entity='mgr.x' 2024-04-23T10:24:25.242 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 23 10:24:24 smithi119 ceph-mon[32608]: from='mgr.14152 172.21.15.114:0/1529706138' entity='mgr.x' 2024-04-23T10:24:25.243 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 23 10:24:24 smithi119 ceph-mon[32608]: from='mgr.14152 172.21.15.114:0/1529706138' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.h", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-04-23T10:24:25.243 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 23 10:24:24 smithi119 ceph-mon[32608]: from='mgr.14152 172.21.15.114:0/1529706138' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.h", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-04-23T10:24:25.243 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 23 10:24:24 smithi119 ceph-mon[32608]: from='mgr.14152 172.21.15.114:0/1529706138' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-23T10:24:25.254 INFO:journalctl@ceph.mon.a.smithi114.stdout:Apr 23 10:24:24 smithi114 ceph-mon[29255]: pgmap v272: 1 pgs: 1 active+clean; 577 KiB data, 365 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-23T10:24:25.254 INFO:journalctl@ceph.mon.a.smithi114.stdout:Apr 23 10:24:24 smithi114 ceph-mon[29255]: from='mgr.14152 172.21.15.114:0/1529706138' entity='mgr.x' 2024-04-23T10:24:25.254 INFO:journalctl@ceph.mon.a.smithi114.stdout:Apr 23 10:24:24 smithi114 ceph-mon[29255]: from='mgr.14152 172.21.15.114:0/1529706138' entity='mgr.x' 2024-04-23T10:24:25.254 INFO:journalctl@ceph.mon.a.smithi114.stdout:Apr 23 10:24:24 smithi114 ceph-mon[29255]: from='mgr.14152 172.21.15.114:0/1529706138' entity='mgr.x' 2024-04-23T10:24:25.254 INFO:journalctl@ceph.mon.a.smithi114.stdout:Apr 23 10:24:24 smithi114 ceph-mon[29255]: from='mgr.14152 172.21.15.114:0/1529706138' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.h", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-04-23T10:24:25.254 INFO:journalctl@ceph.mon.a.smithi114.stdout:Apr 23 10:24:24 smithi114 ceph-mon[29255]: from='mgr.14152 172.21.15.114:0/1529706138' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.h", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-04-23T10:24:25.254 INFO:journalctl@ceph.mon.a.smithi114.stdout:Apr 23 10:24:24 smithi114 ceph-mon[29255]: from='mgr.14152 172.21.15.114:0/1529706138' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-23T10:24:25.328 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 23 10:24:24 smithi204 ceph-mon[32649]: pgmap v272: 1 pgs: 1 active+clean; 577 KiB data, 365 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-23T10:24:25.328 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 23 10:24:24 smithi204 ceph-mon[32649]: from='mgr.14152 172.21.15.114:0/1529706138' entity='mgr.x' 2024-04-23T10:24:25.328 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 23 10:24:24 smithi204 ceph-mon[32649]: from='mgr.14152 172.21.15.114:0/1529706138' entity='mgr.x' 2024-04-23T10:24:25.328 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 23 10:24:24 smithi204 ceph-mon[32649]: from='mgr.14152 172.21.15.114:0/1529706138' entity='mgr.x' 2024-04-23T10:24:25.328 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 23 10:24:24 smithi204 ceph-mon[32649]: from='mgr.14152 172.21.15.114:0/1529706138' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.h", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-04-23T10:24:25.329 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 23 10:24:24 smithi204 ceph-mon[32649]: from='mgr.14152 172.21.15.114:0/1529706138' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.h", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-04-23T10:24:25.329 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 23 10:24:24 smithi204 ceph-mon[32649]: from='mgr.14152 172.21.15.114:0/1529706138' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-23T10:24:25.347 DEBUG:teuthology.orchestra.run.smithi114:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-23T10:24:25.457 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:24:25.458+0000 7fec358cf640 1 -- 172.21.15.114:0/2571317060 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fec30100e90 msgr2=0x7fec301012b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T10:24:25.457 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:24:25.458+0000 7fec358cf640 1 --2- 172.21.15.114:0/2571317060 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fec30100e90 0x7fec301012b0 secure :-1 s=READY pgs=181 cs=0 l=1 rev1=1 crypto rx=0x7fec20009900 tx=0x7fec2002a410 comp rx=0 tx=0).stop 2024-04-23T10:24:25.458 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:24:25.458+0000 7fec358cf640 1 -- 172.21.15.114:0/2571317060 shutdown_connections 2024-04-23T10:24:25.458 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:24:25.458+0000 7fec358cf640 1 --2- 172.21.15.114:0/2571317060 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fec301029a0 0x7fec301092c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:24:25.458 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:24:25.458+0000 7fec358cf640 1 --2- 172.21.15.114:0/2571317060 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fec30101fe0 0x7fec30102460 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:24:25.458 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:24:25.458+0000 7fec358cf640 1 --2- 172.21.15.114:0/2571317060 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fec30100e90 0x7fec301012b0 unknown :-1 s=CLOSED pgs=181 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:24:25.458 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:24:25.458+0000 7fec358cf640 1 -- 172.21.15.114:0/2571317060 >> 172.21.15.114:0/2571317060 conn(0x7fec300fc400 msgr2=0x7fec300fe840 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T10:24:25.458 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:24:25.458+0000 7fec358cf640 1 -- 172.21.15.114:0/2571317060 shutdown_connections 2024-04-23T10:24:25.458 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:24:25.458+0000 7fec358cf640 1 -- 172.21.15.114:0/2571317060 wait complete. 2024-04-23T10:24:25.458 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:24:25.459+0000 7fec358cf640 1 Processor -- start 2024-04-23T10:24:25.460 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:24:25.459+0000 7fec358cf640 1 -- start start 2024-04-23T10:24:25.460 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:24:25.459+0000 7fec358cf640 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fec30100e90 0x7fec30069c50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T10:24:25.460 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:24:25.459+0000 7fec358cf640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fec30101fe0 0x7fec3019cf40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T10:24:25.461 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:24:25.459+0000 7fec358cf640 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fec301029a0 0x7fec3019f560 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T10:24:25.461 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:24:25.459+0000 7fec358cf640 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x7fec3010c9e0 con 0x7fec30101fe0 2024-04-23T10:24:25.461 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:24:25.459+0000 7fec358cf640 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 -- 0x7fec3010c860 con 0x7fec301029a0 2024-04-23T10:24:25.461 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:24:25.459+0000 7fec358cf640 1 -- --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_getmap magic: 0 -- 0x7fec3010cb60 con 0x7fec30100e90 2024-04-23T10:24:25.461 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:24:25.459+0000 7fec2e7fc640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fec30101fe0 0x7fec3019cf40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T10:24:25.461 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:24:25.459+0000 7fec2e7fc640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fec30101fe0 0x7fec3019cf40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.114:42650/0 (socket says 172.21.15.114:42650) 2024-04-23T10:24:25.461 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:24:25.459+0000 7fec2e7fc640 1 -- 172.21.15.114:0/1277400952 learned_addr learned my addr 172.21.15.114:0/1277400952 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-04-23T10:24:25.461 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:24:25.459+0000 7fec2f7fe640 1 --2- 172.21.15.114:0/1277400952 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fec301029a0 0x7fec3019f560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T10:24:25.461 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:24:25.459+0000 7fec2effd640 1 --2- 172.21.15.114:0/1277400952 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fec30100e90 0x7fec30069c50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T10:24:25.461 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:24:25.459+0000 7fec2e7fc640 1 -- 172.21.15.114:0/1277400952 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fec30100e90 msgr2=0x7fec30069c50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T10:24:25.461 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:24:25.459+0000 7fec2e7fc640 1 --2- 172.21.15.114:0/1277400952 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fec30100e90 0x7fec30069c50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:24:25.461 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:24:25.460+0000 7fec2e7fc640 1 -- 172.21.15.114:0/1277400952 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fec301029a0 msgr2=0x7fec3019f560 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T10:24:25.462 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:24:25.460+0000 7fec2e7fc640 1 --2- 172.21.15.114:0/1277400952 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fec301029a0 0x7fec3019f560 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:24:25.462 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:24:25.460+0000 7fec2e7fc640 1 -- 172.21.15.114:0/1277400952 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fec3019fe70 con 0x7fec30101fe0 2024-04-23T10:24:25.462 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:24:25.460+0000 7fec2e7fc640 1 --2- 172.21.15.114:0/1277400952 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fec30101fe0 0x7fec3019cf40 secure :-1 s=READY pgs=182 cs=0 l=1 rev1=1 crypto rx=0x7fec2400e9c0 tx=0x7fec2400ee80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T10:24:25.462 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:24:25.460+0000 7fec348cd640 1 -- 172.21.15.114:0/1277400952 <== mon.0 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fec2400cd80 con 0x7fec30101fe0 2024-04-23T10:24:25.462 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:24:25.460+0000 7fec348cd640 1 -- 172.21.15.114:0/1277400952 <== mon.0 v2:172.21.15.114:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fec240035f0 con 0x7fec30101fe0 2024-04-23T10:24:25.462 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:24:25.460+0000 7fec358cf640 1 -- 172.21.15.114:0/1277400952 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fec301a0100 con 0x7fec30101fe0 2024-04-23T10:24:25.462 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:24:25.460+0000 7fec348cd640 1 -- 172.21.15.114:0/1277400952 <== mon.0 v2:172.21.15.114:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fec24010690 con 0x7fec30101fe0 2024-04-23T10:24:25.462 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:24:25.460+0000 7fec358cf640 1 -- 172.21.15.114:0/1277400952 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fec301a0650 con 0x7fec30101fe0 2024-04-23T10:24:25.462 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:24:25.461+0000 7fec358cf640 1 -- 172.21.15.114:0/1277400952 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fec301081f0 con 0x7fec30101fe0 2024-04-23T10:24:25.463 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:24:25.462+0000 7fec348cd640 1 -- 172.21.15.114:0/1277400952 <== mon.0 v2:172.21.15.114:3300/0 4 ==== mgrmap(e 16) ==== 138647+0+0 (secure 0 0 0) 0x7fec240037a0 con 0x7fec30101fe0 2024-04-23T10:24:25.463 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:24:25.462+0000 7fec348cd640 1 --2- 172.21.15.114:0/1277400952 >> [v2:172.21.15.114:6800/465333808,v1:172.21.15.114:6801/465333808] conn(0x7fec180a3bb0 0x7fec180a6070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T10:24:25.463 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:24:25.463+0000 7fec2effd640 1 --2- 172.21.15.114:0/1277400952 >> [v2:172.21.15.114:6800/465333808,v1:172.21.15.114:6801/465333808] conn(0x7fec180a3bb0 0x7fec180a6070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T10:24:25.463 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:24:25.463+0000 7fec348cd640 1 -- 172.21.15.114:0/1277400952 <== mon.0 v2:172.21.15.114:3300/0 5 ==== osd_map(83..83 src has 1..83) ==== 7127+0+0 (secure 0 0 0) 0x7fec240d0370 con 0x7fec30101fe0 2024-04-23T10:24:25.463 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:24:25.463+0000 7fec2effd640 1 --2- 172.21.15.114:0/1277400952 >> [v2:172.21.15.114:6800/465333808,v1:172.21.15.114:6801/465333808] conn(0x7fec180a3bb0 0x7fec180a6070 secure :-1 s=READY pgs=147 cs=0 l=1 rev1=1 crypto rx=0x7fec20000c00 tx=0x7fec200314a0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T10:24:25.466 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:24:25.467+0000 7fec348cd640 1 -- 172.21.15.114:0/1277400952 <== mon.0 v2:172.21.15.114:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7fec2409d1f0 con 0x7fec30101fe0 2024-04-23T10:24:25.683 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:24:25.684+0000 7fec358cf640 1 -- 172.21.15.114:0/1277400952 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fec3002d810 con 0x7fec30101fe0 2024-04-23T10:24:25.683 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:24:25.684+0000 7fec348cd640 1 -- 172.21.15.114:0/1277400952 <== mon.0 v2:172.21.15.114:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 8 v8) ==== 93+0+6355 (secure 0 0 0) 0x7fec2409de20 con 0x7fec30101fe0 2024-04-23T10:24:25.684 INFO:teuthology.orchestra.run.smithi114.stdout: 2024-04-23T10:24:25.684 INFO:teuthology.orchestra.run.smithi114.stdout:{"epoch":8,"default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14547,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.114:6835/663493580","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6834","nonce":663493580},{"type":"v1","addr":"172.21.15.114:6835","nonce":663493580}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14565,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.114:6837/1439252154","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6836","nonce":1439252154},{"type":"v1","addr":"172.21.15.114:6837","nonce":1439252154}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5},{"gid":24385,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.204:6833/3866561754","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6832","nonce":3866561754},{"type":"v1","addr":"172.21.15.204:6833","nonce":3866561754}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2},{"gid":24427,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.204:6835/2533793172","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6834","nonce":2533793172},{"type":"v1","addr":"172.21.15.204:6835","nonce":2533793172}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":7},{"gid":24439,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.204:6837/513822179","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6836","nonce":513822179},{"type":"v1","addr":"172.21.15.204:6837","nonce":513822179}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":8},{"gid":24443,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.119:6833/783244660","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6832","nonce":783244660},{"type":"v1","addr":"172.21.15.119:6833","nonce":783244660}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":24479,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.119:6835/2033916709","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6834","nonce":2033916709},{"type":"v1","addr":"172.21.15.119:6835","nonce":2033916709}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":6}],"filesystems":[]} 2024-04-23T10:24:25.684 INFO:teuthology.orchestra.run.smithi114.stderr:dumped fsmap epoch 8 2024-04-23T10:24:25.686 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:24:25.686+0000 7fec358cf640 1 -- 172.21.15.114:0/1277400952 >> [v2:172.21.15.114:6800/465333808,v1:172.21.15.114:6801/465333808] conn(0x7fec180a3bb0 msgr2=0x7fec180a6070 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T10:24:25.686 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:24:25.686+0000 7fec358cf640 1 --2- 172.21.15.114:0/1277400952 >> [v2:172.21.15.114:6800/465333808,v1:172.21.15.114:6801/465333808] conn(0x7fec180a3bb0 0x7fec180a6070 secure :-1 s=READY pgs=147 cs=0 l=1 rev1=1 crypto rx=0x7fec20000c00 tx=0x7fec200314a0 comp rx=0 tx=0).stop 2024-04-23T10:24:25.686 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:24:25.686+0000 7fec358cf640 1 -- 172.21.15.114:0/1277400952 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fec30101fe0 msgr2=0x7fec3019cf40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T10:24:25.686 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:24:25.686+0000 7fec358cf640 1 --2- 172.21.15.114:0/1277400952 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fec30101fe0 0x7fec3019cf40 secure :-1 s=READY pgs=182 cs=0 l=1 rev1=1 crypto rx=0x7fec2400e9c0 tx=0x7fec2400ee80 comp rx=0 tx=0).stop 2024-04-23T10:24:25.686 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:24:25.686+0000 7fec358cf640 1 -- 172.21.15.114:0/1277400952 shutdown_connections 2024-04-23T10:24:25.686 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:24:25.686+0000 7fec358cf640 1 --2- 172.21.15.114:0/1277400952 >> [v2:172.21.15.114:6800/465333808,v1:172.21.15.114:6801/465333808] conn(0x7fec180a3bb0 0x7fec180a6070 unknown :-1 s=CLOSED pgs=147 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:24:25.686 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:24:25.686+0000 7fec358cf640 1 --2- 172.21.15.114:0/1277400952 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fec301029a0 0x7fec3019f560 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:24:25.686 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:24:25.686+0000 7fec358cf640 1 --2- 172.21.15.114:0/1277400952 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fec30101fe0 0x7fec3019cf40 unknown :-1 s=CLOSED pgs=182 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:24:25.687 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:24:25.686+0000 7fec358cf640 1 --2- 172.21.15.114:0/1277400952 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fec30100e90 0x7fec30069c50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:24:25.687 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:24:25.686+0000 7fec358cf640 1 -- 172.21.15.114:0/1277400952 >> 172.21.15.114:0/1277400952 conn(0x7fec300fc400 msgr2=0x7fec301053c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T10:24:25.687 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:24:25.687+0000 7fec358cf640 1 -- 172.21.15.114:0/1277400952 shutdown_connections 2024-04-23T10:24:25.687 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:24:25.687+0000 7fec358cf640 1 -- 172.21.15.114:0/1277400952 wait complete. 2024-04-23T10:24:26.242 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 23 10:24:25 smithi119 ceph-mon[32608]: Deploying daemon mds.h on smithi119 2024-04-23T10:24:26.242 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 23 10:24:25 smithi119 ceph-mon[32608]: mds.? [v2:172.21.15.204:6836/513822179,v1:172.21.15.204:6837/513822179] up:boot 2024-04-23T10:24:26.242 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 23 10:24:25 smithi119 ceph-mon[32608]: fsmap 7 up:standby 2024-04-23T10:24:26.242 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 23 10:24:25 smithi119 ceph-mon[32608]: from='mgr.14152 172.21.15.114:0/1529706138' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "c"}]: dispatch 2024-04-23T10:24:26.242 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 23 10:24:25 smithi119 ceph-mon[32608]: from='client.? 172.21.15.114:0/1277400952' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-23T10:24:26.254 INFO:journalctl@ceph.mon.a.smithi114.stdout:Apr 23 10:24:25 smithi114 ceph-mon[29255]: Deploying daemon mds.h on smithi119 2024-04-23T10:24:26.254 INFO:journalctl@ceph.mon.a.smithi114.stdout:Apr 23 10:24:25 smithi114 ceph-mon[29255]: mds.? [v2:172.21.15.204:6836/513822179,v1:172.21.15.204:6837/513822179] up:boot 2024-04-23T10:24:26.254 INFO:journalctl@ceph.mon.a.smithi114.stdout:Apr 23 10:24:25 smithi114 ceph-mon[29255]: fsmap 7 up:standby 2024-04-23T10:24:26.254 INFO:journalctl@ceph.mon.a.smithi114.stdout:Apr 23 10:24:25 smithi114 ceph-mon[29255]: from='mgr.14152 172.21.15.114:0/1529706138' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "c"}]: dispatch 2024-04-23T10:24:26.254 INFO:journalctl@ceph.mon.a.smithi114.stdout:Apr 23 10:24:25 smithi114 ceph-mon[29255]: from='client.? 172.21.15.114:0/1277400952' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-23T10:24:26.328 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 23 10:24:25 smithi204 ceph-mon[32649]: Deploying daemon mds.h on smithi119 2024-04-23T10:24:26.328 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 23 10:24:25 smithi204 ceph-mon[32649]: mds.? [v2:172.21.15.204:6836/513822179,v1:172.21.15.204:6837/513822179] up:boot 2024-04-23T10:24:26.328 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 23 10:24:25 smithi204 ceph-mon[32649]: fsmap 7 up:standby 2024-04-23T10:24:26.328 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 23 10:24:25 smithi204 ceph-mon[32649]: from='mgr.14152 172.21.15.114:0/1529706138' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "c"}]: dispatch 2024-04-23T10:24:26.328 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 23 10:24:25 smithi204 ceph-mon[32649]: from='client.? 172.21.15.114:0/1277400952' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-23T10:24:26.992 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 23 10:24:26 smithi119 ceph-mon[32608]: pgmap v273: 1 pgs: 1 active+clean; 577 KiB data, 365 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-23T10:24:27.254 INFO:journalctl@ceph.mon.a.smithi114.stdout:Apr 23 10:24:26 smithi114 ceph-mon[29255]: pgmap v273: 1 pgs: 1 active+clean; 577 KiB data, 365 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-23T10:24:27.328 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 23 10:24:26 smithi204 ceph-mon[32649]: pgmap v273: 1 pgs: 1 active+clean; 577 KiB data, 365 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-23T10:24:27.697 DEBUG:teuthology.orchestra.run.smithi114:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-23T10:24:27.742 INFO:journalctl@ceph.mds.h.smithi119.stdout:Apr 23 10:24:27 smithi119 systemd[1]: Started Ceph mds.h for aa240572-0159-11ef-bc93-c7b262605968. 2024-04-23T10:24:27.845 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:24:27.846+0000 7f6659260640 1 -- 172.21.15.114:0/1455151970 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f6654102aa0 msgr2=0x7f6654104e90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T10:24:27.845 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:24:27.846+0000 7f6659260640 1 --2- 172.21.15.114:0/1455151970 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f6654102aa0 0x7f6654104e90 secure :-1 s=READY pgs=183 cs=0 l=1 rev1=1 crypto rx=0x7f664c009ca0 tx=0x7f664c02a8c0 comp rx=0 tx=0).stop 2024-04-23T10:24:27.845 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:24:27.846+0000 7f6659260640 1 -- 172.21.15.114:0/1455151970 shutdown_connections 2024-04-23T10:24:27.845 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:24:27.846+0000 7f6659260640 1 --2- 172.21.15.114:0/1455151970 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f6654107d00 0x7f665410a0f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:24:27.845 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:24:27.846+0000 7f6659260640 1 --2- 172.21.15.114:0/1455151970 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f66541053d0 0x7f66541077c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:24:27.845 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:24:27.846+0000 7f6659260640 1 --2- 172.21.15.114:0/1455151970 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f6654102aa0 0x7f6654104e90 unknown :-1 s=CLOSED pgs=183 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:24:27.845 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:24:27.846+0000 7f6659260640 1 -- 172.21.15.114:0/1455151970 >> 172.21.15.114:0/1455151970 conn(0x7f66540fc430 msgr2=0x7f66540fe850 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T10:24:27.845 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:24:27.846+0000 7f6659260640 1 -- 172.21.15.114:0/1455151970 shutdown_connections 2024-04-23T10:24:27.846 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:24:27.846+0000 7f6659260640 1 -- 172.21.15.114:0/1455151970 wait complete. 2024-04-23T10:24:27.848 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:24:27.846+0000 7f6659260640 1 Processor -- start 2024-04-23T10:24:27.848 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:24:27.847+0000 7f6659260640 1 -- start start 2024-04-23T10:24:27.848 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:24:27.847+0000 7f6659260640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f6654102aa0 0x7f6654198280 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T10:24:27.848 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:24:27.847+0000 7f6659260640 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f66541053d0 0x7f66541987c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T10:24:27.848 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:24:27.847+0000 7f6659260640 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f6654107d00 0x7f665419f7e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T10:24:27.848 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:24:27.847+0000 7f6659260640 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x7f665410dae0 con 0x7f6654102aa0 2024-04-23T10:24:27.848 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:24:27.847+0000 7f6659260640 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 -- 0x7f665410d960 con 0x7f66541053d0 2024-04-23T10:24:27.848 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:24:27.847+0000 7f6659260640 1 -- --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_getmap magic: 0 -- 0x7f665410dc60 con 0x7f6654107d00 2024-04-23T10:24:27.848 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:24:27.847+0000 7f6652d9d640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f6654102aa0 0x7f6654198280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T10:24:27.848 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:24:27.847+0000 7f6652d9d640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f6654102aa0 0x7f6654198280 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.114:46124/0 (socket says 172.21.15.114:46124) 2024-04-23T10:24:27.849 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:24:27.847+0000 7f6652d9d640 1 -- 172.21.15.114:0/3914709749 learned_addr learned my addr 172.21.15.114:0/3914709749 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-04-23T10:24:27.849 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:24:27.847+0000 7f665359e640 1 --2- 172.21.15.114:0/3914709749 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f6654107d00 0x7f665419f7e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T10:24:27.849 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:24:27.847+0000 7f665259c640 1 --2- 172.21.15.114:0/3914709749 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f66541053d0 0x7f66541987c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T10:24:27.849 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:24:27.847+0000 7f6652d9d640 1 -- 172.21.15.114:0/3914709749 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f6654107d00 msgr2=0x7f665419f7e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T10:24:27.849 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:24:27.847+0000 7f6652d9d640 1 --2- 172.21.15.114:0/3914709749 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f6654107d00 0x7f665419f7e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:24:27.849 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:24:27.847+0000 7f6652d9d640 1 -- 172.21.15.114:0/3914709749 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f66541053d0 msgr2=0x7f66541987c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T10:24:27.849 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:24:27.847+0000 7f6652d9d640 1 --2- 172.21.15.114:0/3914709749 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f66541053d0 0x7f66541987c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:24:27.849 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:24:27.847+0000 7f6652d9d640 1 -- 172.21.15.114:0/3914709749 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f665419fe50 con 0x7f6654102aa0 2024-04-23T10:24:27.849 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:24:27.848+0000 7f6652d9d640 1 --2- 172.21.15.114:0/3914709749 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f6654102aa0 0x7f6654198280 secure :-1 s=READY pgs=184 cs=0 l=1 rev1=1 crypto rx=0x7f664c006010 tx=0x7f664c0317a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T10:24:27.849 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:24:27.848+0000 7f664bfff640 1 -- 172.21.15.114:0/3914709749 <== mon.0 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f664c003c30 con 0x7f6654102aa0 2024-04-23T10:24:27.849 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:24:27.848+0000 7f6659260640 1 -- 172.21.15.114:0/3914709749 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f66541a00e0 con 0x7f6654102aa0 2024-04-23T10:24:27.849 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:24:27.848+0000 7f664bfff640 1 -- 172.21.15.114:0/3914709749 <== mon.0 v2:172.21.15.114:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f664c003dd0 con 0x7f6654102aa0 2024-04-23T10:24:27.849 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:24:27.848+0000 7f664bfff640 1 -- 172.21.15.114:0/3914709749 <== mon.0 v2:172.21.15.114:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f664c02fc20 con 0x7f6654102aa0 2024-04-23T10:24:27.849 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:24:27.848+0000 7f6659260640 1 -- 172.21.15.114:0/3914709749 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f66541a0630 con 0x7f6654102aa0 2024-04-23T10:24:27.849 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:24:27.849+0000 7f6659260640 1 -- 172.21.15.114:0/3914709749 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f66540fdf20 con 0x7f6654102aa0 2024-04-23T10:24:27.850 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:24:27.850+0000 7f664bfff640 1 -- 172.21.15.114:0/3914709749 <== mon.0 v2:172.21.15.114:3300/0 4 ==== mgrmap(e 16) ==== 138647+0+0 (secure 0 0 0) 0x7f664c0046a0 con 0x7f6654102aa0 2024-04-23T10:24:27.850 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:24:27.850+0000 7f664bfff640 1 --2- 172.21.15.114:0/3914709749 >> [v2:172.21.15.114:6800/465333808,v1:172.21.15.114:6801/465333808] conn(0x7f66400a3bb0 0x7f66400a6070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T10:24:27.850 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:24:27.851+0000 7f665259c640 1 --2- 172.21.15.114:0/3914709749 >> [v2:172.21.15.114:6800/465333808,v1:172.21.15.114:6801/465333808] conn(0x7f66400a3bb0 0x7f66400a6070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T10:24:27.851 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:24:27.851+0000 7f664bfff640 1 -- 172.21.15.114:0/3914709749 <== mon.0 v2:172.21.15.114:3300/0 5 ==== osd_map(83..83 src has 1..83) ==== 7127+0+0 (secure 0 0 0) 0x7f664c0f1600 con 0x7f6654102aa0 2024-04-23T10:24:27.851 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:24:27.851+0000 7f665259c640 1 --2- 172.21.15.114:0/3914709749 >> [v2:172.21.15.114:6800/465333808,v1:172.21.15.114:6801/465333808] conn(0x7f66400a3bb0 0x7f66400a6070 secure :-1 s=READY pgs=148 cs=0 l=1 rev1=1 crypto rx=0x7f6654199740 tx=0x7f663c005f90 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T10:24:27.853 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:24:27.854+0000 7f664bfff640 1 -- 172.21.15.114:0/3914709749 <== mon.0 v2:172.21.15.114:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f664c0be510 con 0x7f6654102aa0 2024-04-23T10:24:28.070 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:24:28.071+0000 7f6659260640 1 -- 172.21.15.114:0/3914709749 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f665402d5c0 con 0x7f6654102aa0 2024-04-23T10:24:28.070 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:24:28.071+0000 7f664bfff640 1 -- 172.21.15.114:0/3914709749 <== mon.0 v2:172.21.15.114:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 9 v9) ==== 93+0+7183 (secure 0 0 0) 0x7f664c0bde10 con 0x7f6654102aa0 2024-04-23T10:24:28.070 INFO:teuthology.orchestra.run.smithi114.stdout: 2024-04-23T10:24:28.071 INFO:teuthology.orchestra.run.smithi114.stdout:{"epoch":9,"default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14547,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.114:6835/663493580","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6834","nonce":663493580},{"type":"v1","addr":"172.21.15.114:6835","nonce":663493580}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14565,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.114:6837/1439252154","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6836","nonce":1439252154},{"type":"v1","addr":"172.21.15.114:6837","nonce":1439252154}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5},{"gid":24385,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.204:6833/3866561754","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6832","nonce":3866561754},{"type":"v1","addr":"172.21.15.204:6833","nonce":3866561754}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2},{"gid":24427,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.204:6835/2533793172","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6834","nonce":2533793172},{"type":"v1","addr":"172.21.15.204:6835","nonce":2533793172}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":7},{"gid":24439,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.204:6837/513822179","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6836","nonce":513822179},{"type":"v1","addr":"172.21.15.204:6837","nonce":513822179}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":8},{"gid":24443,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.119:6833/783244660","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6832","nonce":783244660},{"type":"v1","addr":"172.21.15.119:6833","nonce":783244660}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":24479,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.119:6835/2033916709","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6834","nonce":2033916709},{"type":"v1","addr":"172.21.15.119:6835","nonce":2033916709}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":6},{"gid":24509,"name":"h","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.119:6837/1872363128","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6836","nonce":1872363128},{"type":"v1","addr":"172.21.15.119:6837","nonce":1872363128}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":9}],"filesystems":[]} 2024-04-23T10:24:28.071 INFO:teuthology.orchestra.run.smithi114.stderr:dumped fsmap epoch 9 2024-04-23T10:24:28.072 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:24:28.073+0000 7f6659260640 1 -- 172.21.15.114:0/3914709749 >> [v2:172.21.15.114:6800/465333808,v1:172.21.15.114:6801/465333808] conn(0x7f66400a3bb0 msgr2=0x7f66400a6070 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T10:24:28.073 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:24:28.073+0000 7f6659260640 1 --2- 172.21.15.114:0/3914709749 >> [v2:172.21.15.114:6800/465333808,v1:172.21.15.114:6801/465333808] conn(0x7f66400a3bb0 0x7f66400a6070 secure :-1 s=READY pgs=148 cs=0 l=1 rev1=1 crypto rx=0x7f6654199740 tx=0x7f663c005f90 comp rx=0 tx=0).stop 2024-04-23T10:24:28.073 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:24:28.073+0000 7f6659260640 1 -- 172.21.15.114:0/3914709749 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f6654102aa0 msgr2=0x7f6654198280 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T10:24:28.073 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:24:28.073+0000 7f6659260640 1 --2- 172.21.15.114:0/3914709749 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f6654102aa0 0x7f6654198280 secure :-1 s=READY pgs=184 cs=0 l=1 rev1=1 crypto rx=0x7f664c006010 tx=0x7f664c0317a0 comp rx=0 tx=0).stop 2024-04-23T10:24:28.073 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:24:28.073+0000 7f6659260640 1 -- 172.21.15.114:0/3914709749 shutdown_connections 2024-04-23T10:24:28.073 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:24:28.073+0000 7f6659260640 1 --2- 172.21.15.114:0/3914709749 >> [v2:172.21.15.114:6800/465333808,v1:172.21.15.114:6801/465333808] conn(0x7f66400a3bb0 0x7f66400a6070 unknown :-1 s=CLOSED pgs=148 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:24:28.073 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:24:28.073+0000 7f6659260640 1 --2- 172.21.15.114:0/3914709749 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f6654107d00 0x7f665419f7e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:24:28.073 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:24:28.073+0000 7f6659260640 1 --2- 172.21.15.114:0/3914709749 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f66541053d0 0x7f66541987c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:24:28.073 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:24:28.073+0000 7f6659260640 1 --2- 172.21.15.114:0/3914709749 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f6654102aa0 0x7f6654198280 unknown :-1 s=CLOSED pgs=184 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:24:28.073 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:24:28.073+0000 7f6659260640 1 -- 172.21.15.114:0/3914709749 >> 172.21.15.114:0/3914709749 conn(0x7f66540fc430 msgr2=0x7f66541038f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T10:24:28.073 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:24:28.073+0000 7f6659260640 1 -- 172.21.15.114:0/3914709749 shutdown_connections 2024-04-23T10:24:28.073 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:24:28.073+0000 7f6659260640 1 -- 172.21.15.114:0/3914709749 wait complete. 2024-04-23T10:24:28.800 INFO:journalctl@ceph.mon.a.smithi114.stdout:Apr 23 10:24:28 smithi114 ceph-mon[29255]: from='mgr.14152 172.21.15.114:0/1529706138' entity='mgr.x' 2024-04-23T10:24:28.800 INFO:journalctl@ceph.mon.a.smithi114.stdout:Apr 23 10:24:28 smithi114 ceph-mon[29255]: from='mgr.14152 172.21.15.114:0/1529706138' entity='mgr.x' 2024-04-23T10:24:28.800 INFO:journalctl@ceph.mon.a.smithi114.stdout:Apr 23 10:24:28 smithi114 ceph-mon[29255]: from='mgr.14152 172.21.15.114:0/1529706138' entity='mgr.x' 2024-04-23T10:24:28.800 INFO:journalctl@ceph.mon.a.smithi114.stdout:Apr 23 10:24:28 smithi114 ceph-mon[29255]: from='mgr.14152 172.21.15.114:0/1529706138' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.i", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-04-23T10:24:28.800 INFO:journalctl@ceph.mon.a.smithi114.stdout:Apr 23 10:24:28 smithi114 ceph-mon[29255]: from='mgr.14152 172.21.15.114:0/1529706138' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.i", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-04-23T10:24:28.800 INFO:journalctl@ceph.mon.a.smithi114.stdout:Apr 23 10:24:28 smithi114 ceph-mon[29255]: from='mgr.14152 172.21.15.114:0/1529706138' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-23T10:24:28.800 INFO:journalctl@ceph.mon.a.smithi114.stdout:Apr 23 10:24:28 smithi114 ceph-mon[29255]: Deploying daemon mds.i on smithi204 2024-04-23T10:24:28.800 INFO:journalctl@ceph.mon.a.smithi114.stdout:Apr 23 10:24:28 smithi114 ceph-mon[29255]: pgmap v274: 1 pgs: 1 active+clean; 577 KiB data, 365 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-23T10:24:28.801 INFO:journalctl@ceph.mon.a.smithi114.stdout:Apr 23 10:24:28 smithi114 ceph-mon[29255]: mds.? [v2:172.21.15.119:6836/1872363128,v1:172.21.15.119:6837/1872363128] up:boot 2024-04-23T10:24:28.801 INFO:journalctl@ceph.mon.a.smithi114.stdout:Apr 23 10:24:28 smithi114 ceph-mon[29255]: fsmap 8 up:standby 2024-04-23T10:24:28.801 INFO:journalctl@ceph.mon.a.smithi114.stdout:Apr 23 10:24:28 smithi114 ceph-mon[29255]: from='mgr.14152 172.21.15.114:0/1529706138' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "h"}]: dispatch 2024-04-23T10:24:28.801 INFO:journalctl@ceph.mon.a.smithi114.stdout:Apr 23 10:24:28 smithi114 ceph-mon[29255]: from='client.? 172.21.15.114:0/3914709749' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-23T10:24:28.828 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 23 10:24:28 smithi204 ceph-mon[32649]: from='mgr.14152 172.21.15.114:0/1529706138' entity='mgr.x' 2024-04-23T10:24:28.828 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 23 10:24:28 smithi204 ceph-mon[32649]: from='mgr.14152 172.21.15.114:0/1529706138' entity='mgr.x' 2024-04-23T10:24:28.828 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 23 10:24:28 smithi204 ceph-mon[32649]: from='mgr.14152 172.21.15.114:0/1529706138' entity='mgr.x' 2024-04-23T10:24:28.828 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 23 10:24:28 smithi204 ceph-mon[32649]: from='mgr.14152 172.21.15.114:0/1529706138' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.i", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-04-23T10:24:28.828 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 23 10:24:28 smithi204 ceph-mon[32649]: from='mgr.14152 172.21.15.114:0/1529706138' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.i", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-04-23T10:24:28.828 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 23 10:24:28 smithi204 ceph-mon[32649]: from='mgr.14152 172.21.15.114:0/1529706138' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-23T10:24:28.828 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 23 10:24:28 smithi204 ceph-mon[32649]: Deploying daemon mds.i on smithi204 2024-04-23T10:24:28.828 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 23 10:24:28 smithi204 ceph-mon[32649]: pgmap v274: 1 pgs: 1 active+clean; 577 KiB data, 365 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-23T10:24:28.828 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 23 10:24:28 smithi204 ceph-mon[32649]: mds.? [v2:172.21.15.119:6836/1872363128,v1:172.21.15.119:6837/1872363128] up:boot 2024-04-23T10:24:28.828 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 23 10:24:28 smithi204 ceph-mon[32649]: fsmap 8 up:standby 2024-04-23T10:24:28.828 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 23 10:24:28 smithi204 ceph-mon[32649]: from='mgr.14152 172.21.15.114:0/1529706138' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "h"}]: dispatch 2024-04-23T10:24:28.829 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 23 10:24:28 smithi204 ceph-mon[32649]: from='client.? 172.21.15.114:0/3914709749' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-23T10:24:28.992 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 23 10:24:28 smithi119 ceph-mon[32608]: from='mgr.14152 172.21.15.114:0/1529706138' entity='mgr.x' 2024-04-23T10:24:28.992 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 23 10:24:28 smithi119 ceph-mon[32608]: from='mgr.14152 172.21.15.114:0/1529706138' entity='mgr.x' 2024-04-23T10:24:28.992 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 23 10:24:28 smithi119 ceph-mon[32608]: from='mgr.14152 172.21.15.114:0/1529706138' entity='mgr.x' 2024-04-23T10:24:28.992 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 23 10:24:28 smithi119 ceph-mon[32608]: from='mgr.14152 172.21.15.114:0/1529706138' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.i", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-04-23T10:24:28.992 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 23 10:24:28 smithi119 ceph-mon[32608]: from='mgr.14152 172.21.15.114:0/1529706138' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.i", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-04-23T10:24:28.992 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 23 10:24:28 smithi119 ceph-mon[32608]: from='mgr.14152 172.21.15.114:0/1529706138' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-23T10:24:28.992 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 23 10:24:28 smithi119 ceph-mon[32608]: Deploying daemon mds.i on smithi204 2024-04-23T10:24:28.992 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 23 10:24:28 smithi119 ceph-mon[32608]: pgmap v274: 1 pgs: 1 active+clean; 577 KiB data, 365 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-23T10:24:28.992 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 23 10:24:28 smithi119 ceph-mon[32608]: mds.? [v2:172.21.15.119:6836/1872363128,v1:172.21.15.119:6837/1872363128] up:boot 2024-04-23T10:24:28.992 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 23 10:24:28 smithi119 ceph-mon[32608]: fsmap 8 up:standby 2024-04-23T10:24:28.993 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 23 10:24:28 smithi119 ceph-mon[32608]: from='mgr.14152 172.21.15.114:0/1529706138' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "h"}]: dispatch 2024-04-23T10:24:28.993 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 23 10:24:28 smithi119 ceph-mon[32608]: from='client.? 172.21.15.114:0/3914709749' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-23T10:24:29.819 INFO:journalctl@ceph.mds.i.smithi204.stdout:Apr 23 10:24:29 smithi204 systemd[1]: Starting Ceph mds.i for aa240572-0159-11ef-bc93-c7b262605968... 2024-04-23T10:24:30.078 INFO:journalctl@ceph.mds.i.smithi204.stdout:Apr 23 10:24:29 smithi204 podman[63289]: 2024-04-23 10:24:29.724212709 +0000 UTC m=+0.018019112 image pull da4b004c09e820713483d039bc76fbd660be678f2ea3c03e059a6161b74dab76 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:068a558e3af4932cf1c60dd321c0bdf500075eb52ea1bf455e0973ba114fbfd4 2024-04-23T10:24:30.078 INFO:journalctl@ceph.mds.i.smithi204.stdout:Apr 23 10:24:29 smithi204 podman[63289]: 2024-04-23 10:24:29.823487348 +0000 UTC m=+0.117293743 container create 6e31504a857cb22736200eb8bd364a6ae52fae3d213bde84ebb24ffddd2652a3 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:068a558e3af4932cf1c60dd321c0bdf500075eb52ea1bf455e0973ba114fbfd4, name=ceph-aa240572-0159-11ef-bc93-c7b262605968-mds-i, ceph=True, org.label-schema.license=GPLv2, io.buildah.version=1.33.5, RELEASE=wip-lusov-quiescer-d69f13d, org.label-schema.schema-version=1.0, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux , GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_CLEAN=True, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, org.label-schema.build-date=20240416, org.label-schema.name=CentOS Stream 8 Base Image) 2024-04-23T10:24:30.084 DEBUG:teuthology.orchestra.run.smithi114:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-23T10:24:30.195 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:24:30.194+0000 7fca84860640 1 -- 172.21.15.114:0/1234356496 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fca80107cd0 msgr2=0x7fca8010a0c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T10:24:30.195 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:24:30.194+0000 7fca84860640 1 --2- 172.21.15.114:0/1234356496 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fca80107cd0 0x7fca8010a0c0 secure :-1 s=READY pgs=185 cs=0 l=1 rev1=1 crypto rx=0x7fca70009900 tx=0x7fca7002a410 comp rx=0 tx=0).stop 2024-04-23T10:24:30.195 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:24:30.195+0000 7fca84860640 1 -- 172.21.15.114:0/1234356496 shutdown_connections 2024-04-23T10:24:30.195 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:24:30.195+0000 7fca84860640 1 --2- 172.21.15.114:0/1234356496 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fca80107cd0 0x7fca8010a0c0 unknown :-1 s=CLOSED pgs=185 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:24:30.195 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:24:30.195+0000 7fca84860640 1 --2- 172.21.15.114:0/1234356496 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fca801053a0 0x7fca80107790 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:24:30.195 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:24:30.195+0000 7fca84860640 1 --2- 172.21.15.114:0/1234356496 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fca80102a70 0x7fca80104e60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:24:30.195 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:24:30.195+0000 7fca84860640 1 -- 172.21.15.114:0/1234356496 >> 172.21.15.114:0/1234356496 conn(0x7fca800fc400 msgr2=0x7fca800fe840 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T10:24:30.195 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:24:30.195+0000 7fca84860640 1 -- 172.21.15.114:0/1234356496 shutdown_connections 2024-04-23T10:24:30.195 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:24:30.195+0000 7fca84860640 1 -- 172.21.15.114:0/1234356496 wait complete. 2024-04-23T10:24:30.197 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:24:30.195+0000 7fca84860640 1 Processor -- start 2024-04-23T10:24:30.197 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:24:30.195+0000 7fca84860640 1 -- start start 2024-04-23T10:24:30.197 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:24:30.196+0000 7fca84860640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fca80102a70 0x7fca801023e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T10:24:30.197 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:24:30.196+0000 7fca84860640 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fca801053a0 0x7fca80100a20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T10:24:30.198 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:24:30.196+0000 7fca84860640 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fca80107cd0 0x7fca80100f60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T10:24:30.198 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:24:30.196+0000 7fca84860640 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x7fca8010dae0 con 0x7fca80102a70 2024-04-23T10:24:30.198 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:24:30.196+0000 7fca84860640 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 -- 0x7fca8010d960 con 0x7fca80107cd0 2024-04-23T10:24:30.198 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:24:30.196+0000 7fca84860640 1 -- --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_getmap magic: 0 -- 0x7fca8010dc60 con 0x7fca801053a0 2024-04-23T10:24:30.198 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:24:30.196+0000 7fca7e59c640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fca80102a70 0x7fca801023e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T10:24:30.198 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:24:30.196+0000 7fca7e59c640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fca80102a70 0x7fca801023e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.114:46144/0 (socket says 172.21.15.114:46144) 2024-04-23T10:24:30.198 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:24:30.196+0000 7fca7e59c640 1 -- 172.21.15.114:0/708627321 learned_addr learned my addr 172.21.15.114:0/708627321 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-04-23T10:24:30.198 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:24:30.196+0000 7fca7dd9b640 1 --2- 172.21.15.114:0/708627321 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fca801053a0 0x7fca80100a20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T10:24:30.198 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:24:30.196+0000 7fca7e59c640 1 -- 172.21.15.114:0/708627321 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fca801053a0 msgr2=0x7fca80100a20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T10:24:30.198 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:24:30.196+0000 7fca7e59c640 1 --2- 172.21.15.114:0/708627321 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fca801053a0 0x7fca80100a20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:24:30.198 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:24:30.196+0000 7fca7e59c640 1 -- 172.21.15.114:0/708627321 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fca80107cd0 msgr2=0x7fca80100f60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T10:24:30.198 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:24:30.196+0000 7fca7ed9d640 1 --2- 172.21.15.114:0/708627321 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fca80107cd0 0x7fca80100f60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T10:24:30.198 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:24:30.196+0000 7fca7e59c640 1 --2- 172.21.15.114:0/708627321 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fca80107cd0 0x7fca80100f60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:24:30.198 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:24:30.196+0000 7fca7e59c640 1 -- 172.21.15.114:0/708627321 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fca801017f0 con 0x7fca80102a70 2024-04-23T10:24:30.199 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:24:30.196+0000 7fca7e59c640 1 --2- 172.21.15.114:0/708627321 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fca80102a70 0x7fca801023e0 secure :-1 s=READY pgs=186 cs=0 l=1 rev1=1 crypto rx=0x7fca7800e9f0 tx=0x7fca7800edf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T10:24:30.199 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:24:30.196+0000 7fca6f7fe640 1 -- 172.21.15.114:0/708627321 <== mon.0 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fca7800cd80 con 0x7fca80102a70 2024-04-23T10:24:30.199 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:24:30.196+0000 7fca84860640 1 -- 172.21.15.114:0/708627321 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fca801a2270 con 0x7fca80102a70 2024-04-23T10:24:30.199 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:24:30.196+0000 7fca6f7fe640 1 -- 172.21.15.114:0/708627321 <== mon.0 v2:172.21.15.114:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fca780054d0 con 0x7fca80102a70 2024-04-23T10:24:30.199 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:24:30.196+0000 7fca84860640 1 -- 172.21.15.114:0/708627321 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fca801a27c0 con 0x7fca80102a70 2024-04-23T10:24:30.199 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:24:30.196+0000 7fca6f7fe640 1 -- 172.21.15.114:0/708627321 <== mon.0 v2:172.21.15.114:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fca78010690 con 0x7fca80102a70 2024-04-23T10:24:30.199 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:24:30.197+0000 7fca84860640 1 -- 172.21.15.114:0/708627321 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fca800fdef0 con 0x7fca80102a70 2024-04-23T10:24:30.200 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:24:30.199+0000 7fca6f7fe640 1 -- 172.21.15.114:0/708627321 <== mon.0 v2:172.21.15.114:3300/0 4 ==== mgrmap(e 16) ==== 138647+0+0 (secure 0 0 0) 0x7fca78004e50 con 0x7fca80102a70 2024-04-23T10:24:30.200 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:24:30.199+0000 7fca6f7fe640 1 --2- 172.21.15.114:0/708627321 >> [v2:172.21.15.114:6800/465333808,v1:172.21.15.114:6801/465333808] conn(0x7fca680a3bb0 0x7fca680a6070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T10:24:30.200 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:24:30.199+0000 7fca7dd9b640 1 --2- 172.21.15.114:0/708627321 >> [v2:172.21.15.114:6800/465333808,v1:172.21.15.114:6801/465333808] conn(0x7fca680a3bb0 0x7fca680a6070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T10:24:30.200 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:24:30.199+0000 7fca6f7fe640 1 -- 172.21.15.114:0/708627321 <== mon.0 v2:172.21.15.114:3300/0 5 ==== osd_map(83..83 src has 1..83) ==== 7127+0+0 (secure 0 0 0) 0x7fca780d0690 con 0x7fca80102a70 2024-04-23T10:24:30.200 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:24:30.200+0000 7fca7dd9b640 1 --2- 172.21.15.114:0/708627321 >> [v2:172.21.15.114:6800/465333808,v1:172.21.15.114:6801/465333808] conn(0x7fca680a3bb0 0x7fca680a6070 secure :-1 s=READY pgs=150 cs=0 l=1 rev1=1 crypto rx=0x7fca80102150 tx=0x7fca740075b0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T10:24:30.204 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:24:30.205+0000 7fca6f7fe640 1 -- 172.21.15.114:0/708627321 <== mon.0 v2:172.21.15.114:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7fca78053ff0 con 0x7fca80102a70 2024-04-23T10:24:30.431 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:24:30.431+0000 7fca84860640 1 -- 172.21.15.114:0/708627321 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fca8002d850 con 0x7fca80102a70 2024-04-23T10:24:30.432 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:24:30.432+0000 7fca6f7fe640 1 -- 172.21.15.114:0/708627321 <== mon.0 v2:172.21.15.114:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 9 v9) ==== 93+0+7183 (secure 0 0 0) 0x7fca7809d510 con 0x7fca80102a70 2024-04-23T10:24:30.432 INFO:teuthology.orchestra.run.smithi114.stdout: 2024-04-23T10:24:30.432 INFO:teuthology.orchestra.run.smithi114.stdout:{"epoch":9,"default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14547,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.114:6835/663493580","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6834","nonce":663493580},{"type":"v1","addr":"172.21.15.114:6835","nonce":663493580}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14565,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.114:6837/1439252154","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6836","nonce":1439252154},{"type":"v1","addr":"172.21.15.114:6837","nonce":1439252154}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5},{"gid":24385,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.204:6833/3866561754","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6832","nonce":3866561754},{"type":"v1","addr":"172.21.15.204:6833","nonce":3866561754}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2},{"gid":24427,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.204:6835/2533793172","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6834","nonce":2533793172},{"type":"v1","addr":"172.21.15.204:6835","nonce":2533793172}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":7},{"gid":24439,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.204:6837/513822179","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6836","nonce":513822179},{"type":"v1","addr":"172.21.15.204:6837","nonce":513822179}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":8},{"gid":24443,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.119:6833/783244660","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6832","nonce":783244660},{"type":"v1","addr":"172.21.15.119:6833","nonce":783244660}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":24479,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.119:6835/2033916709","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6834","nonce":2033916709},{"type":"v1","addr":"172.21.15.119:6835","nonce":2033916709}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":6},{"gid":24509,"name":"h","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.119:6837/1872363128","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6836","nonce":1872363128},{"type":"v1","addr":"172.21.15.119:6837","nonce":1872363128}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":9}],"filesystems":[]} 2024-04-23T10:24:30.432 INFO:teuthology.orchestra.run.smithi114.stderr:dumped fsmap epoch 9 2024-04-23T10:24:30.435 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:24:30.434+0000 7fca84860640 1 -- 172.21.15.114:0/708627321 >> [v2:172.21.15.114:6800/465333808,v1:172.21.15.114:6801/465333808] conn(0x7fca680a3bb0 msgr2=0x7fca680a6070 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T10:24:30.435 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:24:30.434+0000 7fca84860640 1 --2- 172.21.15.114:0/708627321 >> [v2:172.21.15.114:6800/465333808,v1:172.21.15.114:6801/465333808] conn(0x7fca680a3bb0 0x7fca680a6070 secure :-1 s=READY pgs=150 cs=0 l=1 rev1=1 crypto rx=0x7fca80102150 tx=0x7fca740075b0 comp rx=0 tx=0).stop 2024-04-23T10:24:30.435 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:24:30.434+0000 7fca84860640 1 -- 172.21.15.114:0/708627321 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fca80102a70 msgr2=0x7fca801023e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T10:24:30.435 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:24:30.434+0000 7fca84860640 1 --2- 172.21.15.114:0/708627321 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fca80102a70 0x7fca801023e0 secure :-1 s=READY pgs=186 cs=0 l=1 rev1=1 crypto rx=0x7fca7800e9f0 tx=0x7fca7800edf0 comp rx=0 tx=0).stop 2024-04-23T10:24:30.435 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:24:30.434+0000 7fca84860640 1 -- 172.21.15.114:0/708627321 shutdown_connections 2024-04-23T10:24:30.435 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:24:30.434+0000 7fca84860640 1 --2- 172.21.15.114:0/708627321 >> [v2:172.21.15.114:6800/465333808,v1:172.21.15.114:6801/465333808] conn(0x7fca680a3bb0 0x7fca680a6070 unknown :-1 s=CLOSED pgs=150 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:24:30.435 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:24:30.434+0000 7fca84860640 1 --2- 172.21.15.114:0/708627321 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fca80107cd0 0x7fca80100f60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:24:30.435 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:24:30.434+0000 7fca84860640 1 --2- 172.21.15.114:0/708627321 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fca801053a0 0x7fca80100a20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:24:30.435 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:24:30.434+0000 7fca84860640 1 --2- 172.21.15.114:0/708627321 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fca80102a70 0x7fca801023e0 unknown :-1 s=CLOSED pgs=186 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:24:30.435 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:24:30.434+0000 7fca84860640 1 -- 172.21.15.114:0/708627321 >> 172.21.15.114:0/708627321 conn(0x7fca800fc400 msgr2=0x7fca801033b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T10:24:30.435 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:24:30.435+0000 7fca84860640 1 -- 172.21.15.114:0/708627321 shutdown_connections 2024-04-23T10:24:30.435 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:24:30.435+0000 7fca84860640 1 -- 172.21.15.114:0/708627321 wait complete. 2024-04-23T10:24:30.578 INFO:journalctl@ceph.mds.i.smithi204.stdout:Apr 23 10:24:30 smithi204 podman[63289]: 2024-04-23 10:24:30.084718828 +0000 UTC m=+0.378525225 container init 6e31504a857cb22736200eb8bd364a6ae52fae3d213bde84ebb24ffddd2652a3 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:068a558e3af4932cf1c60dd321c0bdf500075eb52ea1bf455e0973ba114fbfd4, name=ceph-aa240572-0159-11ef-bc93-c7b262605968-mds-i, RELEASE=wip-lusov-quiescer-d69f13d, GIT_BRANCH=HEAD, org.label-schema.build-date=20240416, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, ceph=True, org.label-schema.name=CentOS Stream 8 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, maintainer=Guillaume Abrioux , GIT_CLEAN=True, io.buildah.version=1.33.5) 2024-04-23T10:24:30.578 INFO:journalctl@ceph.mds.i.smithi204.stdout:Apr 23 10:24:30 smithi204 podman[63289]: 2024-04-23 10:24:30.087330215 +0000 UTC m=+0.381136617 container start 6e31504a857cb22736200eb8bd364a6ae52fae3d213bde84ebb24ffddd2652a3 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:068a558e3af4932cf1c60dd321c0bdf500075eb52ea1bf455e0973ba114fbfd4, name=ceph-aa240572-0159-11ef-bc93-c7b262605968-mds-i, org.label-schema.schema-version=1.0, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, ceph=True, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_CLEAN=True, org.label-schema.vendor=CentOS, RELEASE=wip-lusov-quiescer-d69f13d, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 8 Base Image, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240416, io.buildah.version=1.33.5) 2024-04-23T10:24:30.578 INFO:journalctl@ceph.mds.i.smithi204.stdout:Apr 23 10:24:30 smithi204 ceph-mds[63308]: -- 172.21.15.204:0/89414897 <== mon.1 v2:172.21.15.204:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 194+0+0 (secure 0 0 0) 0x5610ed9d7a40 con 0x5610ed9ab000 2024-04-23T10:24:30.578 INFO:journalctl@ceph.mds.i.smithi204.stdout:Apr 23 10:24:30 smithi204 ceph-aa240572-0159-11ef-bc93-c7b262605968-mds-i[63304]: starting mds.i at 2024-04-23T10:24:30.579 INFO:journalctl@ceph.mds.i.smithi204.stdout:Apr 23 10:24:30 smithi204 bash[63289]: 6e31504a857cb22736200eb8bd364a6ae52fae3d213bde84ebb24ffddd2652a3 2024-04-23T10:24:30.579 INFO:journalctl@ceph.mds.i.smithi204.stdout:Apr 23 10:24:30 smithi204 systemd[1]: Started Ceph mds.i for aa240572-0159-11ef-bc93-c7b262605968. 2024-04-23T10:24:31.217 INFO:journalctl@ceph.mon.a.smithi114.stdout:Apr 23 10:24:31 smithi114 ceph-mon[29255]: pgmap v275: 1 pgs: 1 active+clean; 577 KiB data, 365 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-23T10:24:31.217 INFO:journalctl@ceph.mon.a.smithi114.stdout:Apr 23 10:24:31 smithi114 ceph-mon[29255]: from='mgr.14152 172.21.15.114:0/1529706138' entity='mgr.x' 2024-04-23T10:24:31.217 INFO:journalctl@ceph.mon.a.smithi114.stdout:Apr 23 10:24:31 smithi114 ceph-mon[29255]: from='mgr.14152 172.21.15.114:0/1529706138' entity='mgr.x' 2024-04-23T10:24:31.218 INFO:journalctl@ceph.mon.a.smithi114.stdout:Apr 23 10:24:31 smithi114 ceph-mon[29255]: from='mgr.14152 172.21.15.114:0/1529706138' entity='mgr.x' 2024-04-23T10:24:31.218 INFO:journalctl@ceph.mon.a.smithi114.stdout:Apr 23 10:24:31 smithi114 ceph-mon[29255]: from='mgr.14152 172.21.15.114:0/1529706138' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.g", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-04-23T10:24:31.218 INFO:journalctl@ceph.mon.a.smithi114.stdout:Apr 23 10:24:31 smithi114 ceph-mon[29255]: from='mgr.14152 172.21.15.114:0/1529706138' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.g", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-04-23T10:24:31.218 INFO:journalctl@ceph.mon.a.smithi114.stdout:Apr 23 10:24:31 smithi114 ceph-mon[29255]: from='mgr.14152 172.21.15.114:0/1529706138' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-23T10:24:31.218 INFO:journalctl@ceph.mon.a.smithi114.stdout:Apr 23 10:24:31 smithi114 ceph-mon[29255]: from='client.? 172.21.15.114:0/708627321' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-23T10:24:31.328 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 23 10:24:31 smithi204 ceph-mon[32649]: pgmap v275: 1 pgs: 1 active+clean; 577 KiB data, 365 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-23T10:24:31.328 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 23 10:24:31 smithi204 ceph-mon[32649]: from='mgr.14152 172.21.15.114:0/1529706138' entity='mgr.x' 2024-04-23T10:24:31.328 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 23 10:24:31 smithi204 ceph-mon[32649]: from='mgr.14152 172.21.15.114:0/1529706138' entity='mgr.x' 2024-04-23T10:24:31.328 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 23 10:24:31 smithi204 ceph-mon[32649]: from='mgr.14152 172.21.15.114:0/1529706138' entity='mgr.x' 2024-04-23T10:24:31.328 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 23 10:24:31 smithi204 ceph-mon[32649]: from='mgr.14152 172.21.15.114:0/1529706138' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.g", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-04-23T10:24:31.328 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 23 10:24:31 smithi204 ceph-mon[32649]: from='mgr.14152 172.21.15.114:0/1529706138' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.g", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-04-23T10:24:31.328 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 23 10:24:31 smithi204 ceph-mon[32649]: from='mgr.14152 172.21.15.114:0/1529706138' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-23T10:24:31.328 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 23 10:24:31 smithi204 ceph-mon[32649]: from='client.? 172.21.15.114:0/708627321' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-23T10:24:31.492 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 23 10:24:31 smithi119 ceph-mon[32608]: pgmap v275: 1 pgs: 1 active+clean; 577 KiB data, 365 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-23T10:24:31.492 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 23 10:24:31 smithi119 ceph-mon[32608]: from='mgr.14152 172.21.15.114:0/1529706138' entity='mgr.x' 2024-04-23T10:24:31.492 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 23 10:24:31 smithi119 ceph-mon[32608]: from='mgr.14152 172.21.15.114:0/1529706138' entity='mgr.x' 2024-04-23T10:24:31.492 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 23 10:24:31 smithi119 ceph-mon[32608]: from='mgr.14152 172.21.15.114:0/1529706138' entity='mgr.x' 2024-04-23T10:24:31.492 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 23 10:24:31 smithi119 ceph-mon[32608]: from='mgr.14152 172.21.15.114:0/1529706138' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.g", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-04-23T10:24:31.492 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 23 10:24:31 smithi119 ceph-mon[32608]: from='mgr.14152 172.21.15.114:0/1529706138' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.g", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-04-23T10:24:31.492 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 23 10:24:31 smithi119 ceph-mon[32608]: from='mgr.14152 172.21.15.114:0/1529706138' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-23T10:24:31.492 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 23 10:24:31 smithi119 ceph-mon[32608]: from='client.? 172.21.15.114:0/708627321' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-23T10:24:32.254 INFO:journalctl@ceph.mon.a.smithi114.stdout:Apr 23 10:24:32 smithi114 ceph-mon[29255]: Deploying daemon mds.g on smithi114 2024-04-23T10:24:32.254 INFO:journalctl@ceph.mon.a.smithi114.stdout:Apr 23 10:24:32 smithi114 ceph-mon[29255]: mds.? [v2:172.21.15.204:6838/3222658893,v1:172.21.15.204:6839/3222658893] up:boot 2024-04-23T10:24:32.254 INFO:journalctl@ceph.mon.a.smithi114.stdout:Apr 23 10:24:32 smithi114 ceph-mon[29255]: fsmap 9 up:standby 2024-04-23T10:24:32.254 INFO:journalctl@ceph.mon.a.smithi114.stdout:Apr 23 10:24:32 smithi114 ceph-mon[29255]: from='mgr.14152 172.21.15.114:0/1529706138' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "i"}]: dispatch 2024-04-23T10:24:32.328 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 23 10:24:32 smithi204 ceph-mon[32649]: Deploying daemon mds.g on smithi114 2024-04-23T10:24:32.328 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 23 10:24:32 smithi204 ceph-mon[32649]: mds.? [v2:172.21.15.204:6838/3222658893,v1:172.21.15.204:6839/3222658893] up:boot 2024-04-23T10:24:32.328 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 23 10:24:32 smithi204 ceph-mon[32649]: fsmap 9 up:standby 2024-04-23T10:24:32.328 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 23 10:24:32 smithi204 ceph-mon[32649]: from='mgr.14152 172.21.15.114:0/1529706138' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "i"}]: dispatch 2024-04-23T10:24:32.447 DEBUG:teuthology.orchestra.run.smithi114:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-23T10:24:32.492 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 23 10:24:32 smithi119 ceph-mon[32608]: Deploying daemon mds.g on smithi114 2024-04-23T10:24:32.492 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 23 10:24:32 smithi119 ceph-mon[32608]: mds.? [v2:172.21.15.204:6838/3222658893,v1:172.21.15.204:6839/3222658893] up:boot 2024-04-23T10:24:32.492 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 23 10:24:32 smithi119 ceph-mon[32608]: fsmap 9 up:standby 2024-04-23T10:24:32.492 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 23 10:24:32 smithi119 ceph-mon[32608]: from='mgr.14152 172.21.15.114:0/1529706138' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "i"}]: dispatch 2024-04-23T10:24:32.568 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:24:32.566+0000 7ff8e9d5d640 1 -- 172.21.15.114:0/1729109576 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7ff8e4101fc0 msgr2=0x7ff8e41023e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T10:24:32.568 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:24:32.566+0000 7ff8e9d5d640 1 --2- 172.21.15.114:0/1729109576 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7ff8e4101fc0 0x7ff8e41023e0 secure :-1 s=READY pgs=187 cs=0 l=1 rev1=1 crypto rx=0x7ff8d4009900 tx=0x7ff8d402a410 comp rx=0 tx=0).stop 2024-04-23T10:24:32.568 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:24:32.566+0000 7ff8e9d5d640 1 -- 172.21.15.114:0/1729109576 shutdown_connections 2024-04-23T10:24:32.568 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:24:32.566+0000 7ff8e9d5d640 1 --2- 172.21.15.114:0/1729109576 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7ff8e4103b70 0x7ff8e410a400 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:24:32.568 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:24:32.566+0000 7ff8e9d5d640 1 --2- 172.21.15.114:0/1729109576 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7ff8e41031b0 0x7ff8e4103630 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:24:32.568 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:24:32.566+0000 7ff8e9d5d640 1 --2- 172.21.15.114:0/1729109576 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7ff8e4101fc0 0x7ff8e41023e0 unknown :-1 s=CLOSED pgs=187 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:24:32.569 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:24:32.566+0000 7ff8e9d5d640 1 -- 172.21.15.114:0/1729109576 >> 172.21.15.114:0/1729109576 conn(0x7ff8e40fd530 msgr2=0x7ff8e40ff970 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T10:24:32.569 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:24:32.566+0000 7ff8e9d5d640 1 -- 172.21.15.114:0/1729109576 shutdown_connections 2024-04-23T10:24:32.569 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:24:32.566+0000 7ff8e9d5d640 1 -- 172.21.15.114:0/1729109576 wait complete. 2024-04-23T10:24:32.569 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:24:32.567+0000 7ff8e9d5d640 1 Processor -- start 2024-04-23T10:24:32.569 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:24:32.567+0000 7ff8e9d5d640 1 -- start start 2024-04-23T10:24:32.569 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:24:32.567+0000 7ff8e9d5d640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7ff8e4101fc0 0x7ff8e41982b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T10:24:32.569 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:24:32.567+0000 7ff8e9d5d640 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7ff8e41031b0 0x7ff8e41987f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T10:24:32.569 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:24:32.567+0000 7ff8e9d5d640 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7ff8e4103b70 0x7ff8e419f7c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T10:24:32.569 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:24:32.567+0000 7ff8e9d5d640 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x7ff8e410db50 con 0x7ff8e4101fc0 2024-04-23T10:24:32.569 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:24:32.567+0000 7ff8e9d5d640 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 -- 0x7ff8e410d9d0 con 0x7ff8e4103b70 2024-04-23T10:24:32.569 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:24:32.567+0000 7ff8e9d5d640 1 -- --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_getmap magic: 0 -- 0x7ff8e410dcd0 con 0x7ff8e41031b0 2024-04-23T10:24:32.569 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:24:32.567+0000 7ff8e37fe640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7ff8e4101fc0 0x7ff8e41982b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T10:24:32.569 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:24:32.568+0000 7ff8e37fe640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7ff8e4101fc0 0x7ff8e41982b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.114:46156/0 (socket says 172.21.15.114:46156) 2024-04-23T10:24:32.569 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:24:32.568+0000 7ff8e37fe640 1 -- 172.21.15.114:0/3662582290 learned_addr learned my addr 172.21.15.114:0/3662582290 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-04-23T10:24:32.569 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:24:32.568+0000 7ff8e2ffd640 1 --2- 172.21.15.114:0/3662582290 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7ff8e41031b0 0x7ff8e41987f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T10:24:32.570 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:24:32.568+0000 7ff8e3fff640 1 --2- 172.21.15.114:0/3662582290 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7ff8e4103b70 0x7ff8e419f7c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T10:24:32.570 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:24:32.568+0000 7ff8e37fe640 1 -- 172.21.15.114:0/3662582290 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7ff8e41031b0 msgr2=0x7ff8e41987f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T10:24:32.570 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:24:32.568+0000 7ff8e37fe640 1 --2- 172.21.15.114:0/3662582290 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7ff8e41031b0 0x7ff8e41987f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:24:32.570 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:24:32.568+0000 7ff8e37fe640 1 -- 172.21.15.114:0/3662582290 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7ff8e4103b70 msgr2=0x7ff8e419f7c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T10:24:32.570 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:24:32.568+0000 7ff8e37fe640 1 --2- 172.21.15.114:0/3662582290 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7ff8e4103b70 0x7ff8e419f7c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:24:32.570 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:24:32.568+0000 7ff8e37fe640 1 -- 172.21.15.114:0/3662582290 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff8e419fe30 con 0x7ff8e4101fc0 2024-04-23T10:24:32.570 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:24:32.568+0000 7ff8e37fe640 1 --2- 172.21.15.114:0/3662582290 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7ff8e4101fc0 0x7ff8e41982b0 secure :-1 s=READY pgs=188 cs=0 l=1 rev1=1 crypto rx=0x7ff8d4031500 tx=0x7ff8d4003c80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T10:24:32.570 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:24:32.568+0000 7ff8e0ff9640 1 -- 172.21.15.114:0/3662582290 <== mon.0 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff8d403f070 con 0x7ff8e4101fc0 2024-04-23T10:24:32.570 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:24:32.568+0000 7ff8e9d5d640 1 -- 172.21.15.114:0/3662582290 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7ff8e41a00c0 con 0x7ff8e4101fc0 2024-04-23T10:24:32.570 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:24:32.568+0000 7ff8e0ff9640 1 -- 172.21.15.114:0/3662582290 <== mon.0 v2:172.21.15.114:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7ff8d4003e70 con 0x7ff8e4101fc0 2024-04-23T10:24:32.570 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:24:32.568+0000 7ff8e9d5d640 1 -- 172.21.15.114:0/3662582290 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7ff8e41a0600 con 0x7ff8e4101fc0 2024-04-23T10:24:32.570 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:24:32.568+0000 7ff8e0ff9640 1 -- 172.21.15.114:0/3662582290 <== mon.0 v2:172.21.15.114:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff8d4035030 con 0x7ff8e4101fc0 2024-04-23T10:24:32.570 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:24:32.569+0000 7ff8e9d5d640 1 -- 172.21.15.114:0/3662582290 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7ff8e4109300 con 0x7ff8e4101fc0 2024-04-23T10:24:32.571 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:24:32.571+0000 7ff8e0ff9640 1 -- 172.21.15.114:0/3662582290 <== mon.0 v2:172.21.15.114:3300/0 4 ==== mgrmap(e 16) ==== 138647+0+0 (secure 0 0 0) 0x7ff8d4042070 con 0x7ff8e4101fc0 2024-04-23T10:24:32.571 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:24:32.572+0000 7ff8e0ff9640 1 --2- 172.21.15.114:0/3662582290 >> [v2:172.21.15.114:6800/465333808,v1:172.21.15.114:6801/465333808] conn(0x7ff8cc0a3bb0 0x7ff8cc0a6070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T10:24:32.571 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:24:32.572+0000 7ff8e2ffd640 1 --2- 172.21.15.114:0/3662582290 >> [v2:172.21.15.114:6800/465333808,v1:172.21.15.114:6801/465333808] conn(0x7ff8cc0a3bb0 0x7ff8cc0a6070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T10:24:32.571 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:24:32.572+0000 7ff8e0ff9640 1 -- 172.21.15.114:0/3662582290 <== mon.0 v2:172.21.15.114:3300/0 5 ==== osd_map(83..83 src has 1..83) ==== 7127+0+0 (secure 0 0 0) 0x7ff8d40f0d80 con 0x7ff8e4101fc0 2024-04-23T10:24:32.572 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:24:32.572+0000 7ff8e2ffd640 1 --2- 172.21.15.114:0/3662582290 >> [v2:172.21.15.114:6800/465333808,v1:172.21.15.114:6801/465333808] conn(0x7ff8cc0a3bb0 0x7ff8cc0a6070 secure :-1 s=READY pgs=152 cs=0 l=1 rev1=1 crypto rx=0x7ff8e4199720 tx=0x7ff8d800b410 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T10:24:32.574 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:24:32.575+0000 7ff8e0ff9640 1 -- 172.21.15.114:0/3662582290 <== mon.0 v2:172.21.15.114:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7ff8d40bdc00 con 0x7ff8e4101fc0 2024-04-23T10:24:32.795 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:24:32.796+0000 7ff8e9d5d640 1 -- 172.21.15.114:0/3662582290 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7ff8e402d5c0 con 0x7ff8e4101fc0 2024-04-23T10:24:32.796 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:24:32.797+0000 7ff8e0ff9640 1 -- 172.21.15.114:0/3662582290 <== mon.0 v2:172.21.15.114:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 10 v10) ==== 94+0+8013 (secure 0 0 0) 0x7ff8d40bd500 con 0x7ff8e4101fc0 2024-04-23T10:24:32.796 INFO:teuthology.orchestra.run.smithi114.stdout: 2024-04-23T10:24:32.796 INFO:teuthology.orchestra.run.smithi114.stdout:{"epoch":10,"default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14547,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.114:6835/663493580","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6834","nonce":663493580},{"type":"v1","addr":"172.21.15.114:6835","nonce":663493580}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14565,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.114:6837/1439252154","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6836","nonce":1439252154},{"type":"v1","addr":"172.21.15.114:6837","nonce":1439252154}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5},{"gid":24385,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.204:6833/3866561754","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6832","nonce":3866561754},{"type":"v1","addr":"172.21.15.204:6833","nonce":3866561754}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2},{"gid":24427,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.204:6835/2533793172","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6834","nonce":2533793172},{"type":"v1","addr":"172.21.15.204:6835","nonce":2533793172}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":7},{"gid":24439,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.204:6837/513822179","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6836","nonce":513822179},{"type":"v1","addr":"172.21.15.204:6837","nonce":513822179}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":8},{"gid":24443,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.119:6833/783244660","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6832","nonce":783244660},{"type":"v1","addr":"172.21.15.119:6833","nonce":783244660}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":24463,"name":"i","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.204:6839/3222658893","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6838","nonce":3222658893},{"type":"v1","addr":"172.21.15.204:6839","nonce":3222658893}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":10},{"gid":24479,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.119:6835/2033916709","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6834","nonce":2033916709},{"type":"v1","addr":"172.21.15.119:6835","nonce":2033916709}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":6},{"gid":24509,"name":"h","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.119:6837/1872363128","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6836","nonce":1872363128},{"type":"v1","addr":"172.21.15.119:6837","nonce":1872363128}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":9}],"filesystems":[]} 2024-04-23T10:24:32.796 INFO:teuthology.orchestra.run.smithi114.stderr:dumped fsmap epoch 10 2024-04-23T10:24:32.798 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:24:32.799+0000 7ff8e9d5d640 1 -- 172.21.15.114:0/3662582290 >> [v2:172.21.15.114:6800/465333808,v1:172.21.15.114:6801/465333808] conn(0x7ff8cc0a3bb0 msgr2=0x7ff8cc0a6070 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T10:24:32.798 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:24:32.799+0000 7ff8e9d5d640 1 --2- 172.21.15.114:0/3662582290 >> [v2:172.21.15.114:6800/465333808,v1:172.21.15.114:6801/465333808] conn(0x7ff8cc0a3bb0 0x7ff8cc0a6070 secure :-1 s=READY pgs=152 cs=0 l=1 rev1=1 crypto rx=0x7ff8e4199720 tx=0x7ff8d800b410 comp rx=0 tx=0).stop 2024-04-23T10:24:32.798 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:24:32.799+0000 7ff8e9d5d640 1 -- 172.21.15.114:0/3662582290 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7ff8e4101fc0 msgr2=0x7ff8e41982b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T10:24:32.799 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:24:32.799+0000 7ff8e9d5d640 1 --2- 172.21.15.114:0/3662582290 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7ff8e4101fc0 0x7ff8e41982b0 secure :-1 s=READY pgs=188 cs=0 l=1 rev1=1 crypto rx=0x7ff8d4031500 tx=0x7ff8d4003c80 comp rx=0 tx=0).stop 2024-04-23T10:24:32.799 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:24:32.799+0000 7ff8e9d5d640 1 -- 172.21.15.114:0/3662582290 shutdown_connections 2024-04-23T10:24:32.799 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:24:32.799+0000 7ff8e9d5d640 1 --2- 172.21.15.114:0/3662582290 >> [v2:172.21.15.114:6800/465333808,v1:172.21.15.114:6801/465333808] conn(0x7ff8cc0a3bb0 0x7ff8cc0a6070 unknown :-1 s=CLOSED pgs=152 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:24:32.799 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:24:32.799+0000 7ff8e9d5d640 1 --2- 172.21.15.114:0/3662582290 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7ff8e4103b70 0x7ff8e419f7c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:24:32.799 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:24:32.799+0000 7ff8e9d5d640 1 --2- 172.21.15.114:0/3662582290 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7ff8e41031b0 0x7ff8e41987f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:24:32.799 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:24:32.799+0000 7ff8e9d5d640 1 --2- 172.21.15.114:0/3662582290 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7ff8e4101fc0 0x7ff8e41982b0 unknown :-1 s=CLOSED pgs=188 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:24:32.799 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:24:32.799+0000 7ff8e9d5d640 1 -- 172.21.15.114:0/3662582290 >> 172.21.15.114:0/3662582290 conn(0x7ff8e40fd530 msgr2=0x7ff8e41064d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T10:24:32.799 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:24:32.799+0000 7ff8e9d5d640 1 -- 172.21.15.114:0/3662582290 shutdown_connections 2024-04-23T10:24:32.799 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:24:32.799+0000 7ff8e9d5d640 1 -- 172.21.15.114:0/3662582290 wait complete. 2024-04-23T10:24:32.986 INFO:journalctl@ceph.mds.g.smithi114.stdout:Apr 23 10:24:32 smithi114 ceph-aa240572-0159-11ef-bc93-c7b262605968-mds-g[68253]: starting mds.g at 2024-04-23T10:24:32.986 INFO:journalctl@ceph.mds.g.smithi114.stdout:Apr 23 10:24:32 smithi114 bash[68218]: 9c754624738df274e46e58ec1fd175dcb4b4ca97ac7da78b9741766662e88227 2024-04-23T10:24:33.254 INFO:journalctl@ceph.mon.a.smithi114.stdout:Apr 23 10:24:33 smithi114 ceph-mon[29255]: pgmap v276: 1 pgs: 1 active+clean; 577 KiB data, 365 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-23T10:24:33.254 INFO:journalctl@ceph.mon.a.smithi114.stdout:Apr 23 10:24:33 smithi114 ceph-mon[29255]: from='client.? 172.21.15.114:0/3662582290' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-23T10:24:33.254 INFO:journalctl@ceph.mds.g.smithi114.stdout:Apr 23 10:24:32 smithi114 systemd[1]: Started Ceph mds.g for aa240572-0159-11ef-bc93-c7b262605968. 2024-04-23T10:24:33.328 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 23 10:24:33 smithi204 ceph-mon[32649]: pgmap v276: 1 pgs: 1 active+clean; 577 KiB data, 365 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-23T10:24:33.328 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 23 10:24:33 smithi204 ceph-mon[32649]: from='client.? 172.21.15.114:0/3662582290' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-23T10:24:33.492 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 23 10:24:33 smithi119 ceph-mon[32608]: pgmap v276: 1 pgs: 1 active+clean; 577 KiB data, 365 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-23T10:24:33.492 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 23 10:24:33 smithi119 ceph-mon[32608]: from='client.? 172.21.15.114:0/3662582290' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-23T10:24:34.254 INFO:journalctl@ceph.mon.a.smithi114.stdout:Apr 23 10:24:34 smithi114 ceph-mon[29255]: mds.? [v2:172.21.15.114:6838/3849796939,v1:172.21.15.114:6839/3849796939] up:boot 2024-04-23T10:24:34.254 INFO:journalctl@ceph.mon.a.smithi114.stdout:Apr 23 10:24:34 smithi114 ceph-mon[29255]: fsmap 10 up:standby 2024-04-23T10:24:34.255 INFO:journalctl@ceph.mon.a.smithi114.stdout:Apr 23 10:24:34 smithi114 ceph-mon[29255]: from='mgr.14152 172.21.15.114:0/1529706138' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "g"}]: dispatch 2024-04-23T10:24:34.255 INFO:journalctl@ceph.mon.a.smithi114.stdout:Apr 23 10:24:34 smithi114 ceph-mon[29255]: from='mgr.14152 172.21.15.114:0/1529706138' entity='mgr.x' 2024-04-23T10:24:34.255 INFO:journalctl@ceph.mon.a.smithi114.stdout:Apr 23 10:24:34 smithi114 ceph-mon[29255]: from='mgr.14152 172.21.15.114:0/1529706138' entity='mgr.x' 2024-04-23T10:24:34.255 INFO:journalctl@ceph.mon.a.smithi114.stdout:Apr 23 10:24:34 smithi114 ceph-mon[29255]: from='mgr.14152 172.21.15.114:0/1529706138' entity='mgr.x' 2024-04-23T10:24:34.255 INFO:journalctl@ceph.mon.a.smithi114.stdout:Apr 23 10:24:34 smithi114 ceph-mon[29255]: from='mgr.14152 172.21.15.114:0/1529706138' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.a", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-04-23T10:24:34.255 INFO:journalctl@ceph.mon.a.smithi114.stdout:Apr 23 10:24:34 smithi114 ceph-mon[29255]: from='mgr.14152 172.21.15.114:0/1529706138' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.a", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-04-23T10:24:34.255 INFO:journalctl@ceph.mon.a.smithi114.stdout:Apr 23 10:24:34 smithi114 ceph-mon[29255]: from='mgr.14152 172.21.15.114:0/1529706138' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-23T10:24:34.328 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 23 10:24:34 smithi204 ceph-mon[32649]: mds.? [v2:172.21.15.114:6838/3849796939,v1:172.21.15.114:6839/3849796939] up:boot 2024-04-23T10:24:34.328 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 23 10:24:34 smithi204 ceph-mon[32649]: fsmap 10 up:standby 2024-04-23T10:24:34.328 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 23 10:24:34 smithi204 ceph-mon[32649]: from='mgr.14152 172.21.15.114:0/1529706138' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "g"}]: dispatch 2024-04-23T10:24:34.328 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 23 10:24:34 smithi204 ceph-mon[32649]: from='mgr.14152 172.21.15.114:0/1529706138' entity='mgr.x' 2024-04-23T10:24:34.328 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 23 10:24:34 smithi204 ceph-mon[32649]: from='mgr.14152 172.21.15.114:0/1529706138' entity='mgr.x' 2024-04-23T10:24:34.328 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 23 10:24:34 smithi204 ceph-mon[32649]: from='mgr.14152 172.21.15.114:0/1529706138' entity='mgr.x' 2024-04-23T10:24:34.328 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 23 10:24:34 smithi204 ceph-mon[32649]: from='mgr.14152 172.21.15.114:0/1529706138' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.a", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-04-23T10:24:34.328 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 23 10:24:34 smithi204 ceph-mon[32649]: from='mgr.14152 172.21.15.114:0/1529706138' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.a", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-04-23T10:24:34.328 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 23 10:24:34 smithi204 ceph-mon[32649]: from='mgr.14152 172.21.15.114:0/1529706138' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-23T10:24:34.492 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 23 10:24:34 smithi119 ceph-mon[32608]: mds.? [v2:172.21.15.114:6838/3849796939,v1:172.21.15.114:6839/3849796939] up:boot 2024-04-23T10:24:34.492 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 23 10:24:34 smithi119 ceph-mon[32608]: fsmap 10 up:standby 2024-04-23T10:24:34.492 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 23 10:24:34 smithi119 ceph-mon[32608]: from='mgr.14152 172.21.15.114:0/1529706138' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "g"}]: dispatch 2024-04-23T10:24:34.492 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 23 10:24:34 smithi119 ceph-mon[32608]: from='mgr.14152 172.21.15.114:0/1529706138' entity='mgr.x' 2024-04-23T10:24:34.492 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 23 10:24:34 smithi119 ceph-mon[32608]: from='mgr.14152 172.21.15.114:0/1529706138' entity='mgr.x' 2024-04-23T10:24:34.492 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 23 10:24:34 smithi119 ceph-mon[32608]: from='mgr.14152 172.21.15.114:0/1529706138' entity='mgr.x' 2024-04-23T10:24:34.492 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 23 10:24:34 smithi119 ceph-mon[32608]: from='mgr.14152 172.21.15.114:0/1529706138' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.a", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-04-23T10:24:34.492 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 23 10:24:34 smithi119 ceph-mon[32608]: from='mgr.14152 172.21.15.114:0/1529706138' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.a", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-04-23T10:24:34.493 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 23 10:24:34 smithi119 ceph-mon[32608]: from='mgr.14152 172.21.15.114:0/1529706138' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-23T10:24:34.810 DEBUG:teuthology.orchestra.run.smithi114:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-23T10:24:35.001 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:24:35.001+0000 7f24bc5cc640 1 -- 172.21.15.114:0/3306724969 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f24b41029a0 msgr2=0x7f24b41092c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T10:24:35.001 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:24:35.001+0000 7f24bc5cc640 1 --2- 172.21.15.114:0/3306724969 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f24b41029a0 0x7f24b41092c0 secure :-1 s=READY pgs=191 cs=0 l=1 rev1=1 crypto rx=0x7f24ac009be0 tx=0x7f24ac02a6c0 comp rx=0 tx=0).stop 2024-04-23T10:24:35.001 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:24:35.002+0000 7f24bc5cc640 1 -- 172.21.15.114:0/3306724969 shutdown_connections 2024-04-23T10:24:35.001 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:24:35.002+0000 7f24bc5cc640 1 --2- 172.21.15.114:0/3306724969 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f24b41029a0 0x7f24b41092c0 unknown :-1 s=CLOSED pgs=191 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:24:35.001 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:24:35.002+0000 7f24bc5cc640 1 --2- 172.21.15.114:0/3306724969 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f24b4101fe0 0x7f24b4102460 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:24:35.001 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:24:35.002+0000 7f24bc5cc640 1 --2- 172.21.15.114:0/3306724969 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f24b4100e90 0x7f24b41012b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:24:35.002 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:24:35.002+0000 7f24bc5cc640 1 -- 172.21.15.114:0/3306724969 >> 172.21.15.114:0/3306724969 conn(0x7f24b40fc400 msgr2=0x7f24b40fe840 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T10:24:35.002 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:24:35.002+0000 7f24bc5cc640 1 -- 172.21.15.114:0/3306724969 shutdown_connections 2024-04-23T10:24:35.002 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:24:35.002+0000 7f24bc5cc640 1 -- 172.21.15.114:0/3306724969 wait complete. 2024-04-23T10:24:35.004 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:24:35.003+0000 7f24bc5cc640 1 Processor -- start 2024-04-23T10:24:35.004 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:24:35.003+0000 7f24bc5cc640 1 -- start start 2024-04-23T10:24:35.004 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:24:35.003+0000 7f24bc5cc640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f24b4100e90 0x7f24b4198150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T10:24:35.004 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:24:35.003+0000 7f24bc5cc640 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f24b4101fe0 0x7f24b4198690 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T10:24:35.004 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:24:35.003+0000 7f24bc5cc640 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f24b41029a0 0x7f24b419f700 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T10:24:35.004 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:24:35.003+0000 7f24bc5cc640 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x7f24b410daf0 con 0x7f24b4100e90 2024-04-23T10:24:35.004 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:24:35.003+0000 7f24bc5cc640 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 -- 0x7f24b410d970 con 0x7f24b41029a0 2024-04-23T10:24:35.005 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:24:35.003+0000 7f24bc5cc640 1 -- --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_getmap magic: 0 -- 0x7f24b410dc70 con 0x7f24b4101fe0 2024-04-23T10:24:35.005 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:24:35.003+0000 7f24ba368640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f24b4100e90 0x7f24b4198150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T10:24:35.005 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:24:35.003+0000 7f24ba368640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f24b4100e90 0x7f24b4198150 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.114:46196/0 (socket says 172.21.15.114:46196) 2024-04-23T10:24:35.005 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:24:35.003+0000 7f24ba368640 1 -- 172.21.15.114:0/3316450038 learned_addr learned my addr 172.21.15.114:0/3316450038 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-04-23T10:24:35.005 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:24:35.003+0000 7f24bab69640 1 --2- 172.21.15.114:0/3316450038 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f24b41029a0 0x7f24b419f700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T10:24:35.005 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:24:35.003+0000 7f24b9b67640 1 --2- 172.21.15.114:0/3316450038 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f24b4101fe0 0x7f24b4198690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T10:24:35.005 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:24:35.003+0000 7f24ba368640 1 -- 172.21.15.114:0/3316450038 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f24b4101fe0 msgr2=0x7f24b4198690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T10:24:35.005 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:24:35.003+0000 7f24ba368640 1 --2- 172.21.15.114:0/3316450038 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f24b4101fe0 0x7f24b4198690 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:24:35.005 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:24:35.003+0000 7f24ba368640 1 -- 172.21.15.114:0/3316450038 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f24b41029a0 msgr2=0x7f24b419f700 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T10:24:35.005 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:24:35.003+0000 7f24ba368640 1 --2- 172.21.15.114:0/3316450038 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f24b41029a0 0x7f24b419f700 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:24:35.005 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:24:35.003+0000 7f24ba368640 1 -- 172.21.15.114:0/3316450038 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f24b419fd70 con 0x7f24b4100e90 2024-04-23T10:24:35.005 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:24:35.003+0000 7f24ba368640 1 --2- 172.21.15.114:0/3316450038 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f24b4100e90 0x7f24b4198150 secure :-1 s=READY pgs=192 cs=0 l=1 rev1=1 crypto rx=0x7f24b000e9c0 tx=0x7f24b000ee80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T10:24:35.005 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:24:35.003+0000 7f24ab7fe640 1 -- 172.21.15.114:0/3316450038 <== mon.0 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f24b000cde0 con 0x7f24b4100e90 2024-04-23T10:24:35.005 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:24:35.004+0000 7f24ab7fe640 1 -- 172.21.15.114:0/3316450038 <== mon.0 v2:172.21.15.114:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f24b00035f0 con 0x7f24b4100e90 2024-04-23T10:24:35.006 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:24:35.004+0000 7f24bc5cc640 1 -- 172.21.15.114:0/3316450038 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f24b41a0060 con 0x7f24b4100e90 2024-04-23T10:24:35.006 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:24:35.004+0000 7f24ab7fe640 1 -- 172.21.15.114:0/3316450038 <== mon.0 v2:172.21.15.114:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f24b0010690 con 0x7f24b4100e90 2024-04-23T10:24:35.006 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:24:35.004+0000 7f24bc5cc640 1 -- 172.21.15.114:0/3316450038 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f24b41a05b0 con 0x7f24b4100e90 2024-04-23T10:24:35.006 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:24:35.005+0000 7f24bc5cc640 1 -- 172.21.15.114:0/3316450038 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f24b41081a0 con 0x7f24b4100e90 2024-04-23T10:24:35.007 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:24:35.006+0000 7f24ab7fe640 1 -- 172.21.15.114:0/3316450038 <== mon.0 v2:172.21.15.114:3300/0 4 ==== mgrmap(e 16) ==== 138647+0+0 (secure 0 0 0) 0x7f24b0004000 con 0x7f24b4100e90 2024-04-23T10:24:35.007 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:24:35.006+0000 7f24ab7fe640 1 --2- 172.21.15.114:0/3316450038 >> [v2:172.21.15.114:6800/465333808,v1:172.21.15.114:6801/465333808] conn(0x7f24a00a3bb0 0x7f24a00a6070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T10:24:35.007 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:24:35.006+0000 7f24b9b67640 1 --2- 172.21.15.114:0/3316450038 >> [v2:172.21.15.114:6800/465333808,v1:172.21.15.114:6801/465333808] conn(0x7f24a00a3bb0 0x7f24a00a6070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T10:24:35.007 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:24:35.006+0000 7f24ab7fe640 1 -- 172.21.15.114:0/3316450038 <== mon.0 v2:172.21.15.114:3300/0 5 ==== osd_map(83..83 src has 1..83) ==== 7127+0+0 (secure 0 0 0) 0x7f24b00d0430 con 0x7f24b4100e90 2024-04-23T10:24:35.007 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:24:35.007+0000 7f24b9b67640 1 --2- 172.21.15.114:0/3316450038 >> [v2:172.21.15.114:6800/465333808,v1:172.21.15.114:6801/465333808] conn(0x7f24a00a3bb0 0x7f24a00a6070 secure :-1 s=READY pgs=154 cs=0 l=1 rev1=1 crypto rx=0x7f24b4199660 tx=0x7f24a4005f90 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T10:24:35.010 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:24:35.011+0000 7f24ab7fe640 1 -- 172.21.15.114:0/3316450038 <== mon.0 v2:172.21.15.114:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f24b009d210 con 0x7f24b4100e90 2024-04-23T10:24:35.151 INFO:journalctl@ceph.mon.a.smithi114.stdout:Apr 23 10:24:35 smithi114 ceph-mon[29255]: Deploying daemon mds.a on smithi114 2024-04-23T10:24:35.151 INFO:journalctl@ceph.mon.a.smithi114.stdout:Apr 23 10:24:35 smithi114 ceph-mon[29255]: pgmap v277: 1 pgs: 1 active+clean; 577 KiB data, 365 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-23T10:24:35.240 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:24:35.241+0000 7f24bc5cc640 1 -- 172.21.15.114:0/3316450038 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f24b402d5c0 con 0x7f24b4100e90 2024-04-23T10:24:35.240 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:24:35.242+0000 7f24ab7fe640 1 -- 172.21.15.114:0/3316450038 <== mon.0 v2:172.21.15.114:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 11 v11) ==== 94+0+8842 (secure 0 0 0) 0x7f24b009cb10 con 0x7f24b4100e90 2024-04-23T10:24:35.241 INFO:teuthology.orchestra.run.smithi114.stdout: 2024-04-23T10:24:35.241 INFO:teuthology.orchestra.run.smithi114.stdout:{"epoch":11,"default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14547,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.114:6835/663493580","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6834","nonce":663493580},{"type":"v1","addr":"172.21.15.114:6835","nonce":663493580}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14565,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.114:6837/1439252154","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6836","nonce":1439252154},{"type":"v1","addr":"172.21.15.114:6837","nonce":1439252154}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5},{"gid":14634,"name":"g","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.114:6839/3849796939","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6838","nonce":3849796939},{"type":"v1","addr":"172.21.15.114:6839","nonce":3849796939}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":11},{"gid":24385,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.204:6833/3866561754","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6832","nonce":3866561754},{"type":"v1","addr":"172.21.15.204:6833","nonce":3866561754}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2},{"gid":24427,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.204:6835/2533793172","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6834","nonce":2533793172},{"type":"v1","addr":"172.21.15.204:6835","nonce":2533793172}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":7},{"gid":24439,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.204:6837/513822179","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6836","nonce":513822179},{"type":"v1","addr":"172.21.15.204:6837","nonce":513822179}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":8},{"gid":24443,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.119:6833/783244660","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6832","nonce":783244660},{"type":"v1","addr":"172.21.15.119:6833","nonce":783244660}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":24463,"name":"i","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.204:6839/3222658893","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6838","nonce":3222658893},{"type":"v1","addr":"172.21.15.204:6839","nonce":3222658893}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":10},{"gid":24479,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.119:6835/2033916709","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6834","nonce":2033916709},{"type":"v1","addr":"172.21.15.119:6835","nonce":2033916709}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":6},{"gid":24509,"name":"h","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.119:6837/1872363128","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6836","nonce":1872363128},{"type":"v1","addr":"172.21.15.119:6837","nonce":1872363128}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":9}],"filesystems":[]} 2024-04-23T10:24:35.241 INFO:teuthology.orchestra.run.smithi114.stderr:dumped fsmap epoch 11 2024-04-23T10:24:35.243 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:24:35.245+0000 7f24bc5cc640 1 -- 172.21.15.114:0/3316450038 >> [v2:172.21.15.114:6800/465333808,v1:172.21.15.114:6801/465333808] conn(0x7f24a00a3bb0 msgr2=0x7f24a00a6070 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T10:24:35.244 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:24:35.245+0000 7f24bc5cc640 1 --2- 172.21.15.114:0/3316450038 >> [v2:172.21.15.114:6800/465333808,v1:172.21.15.114:6801/465333808] conn(0x7f24a00a3bb0 0x7f24a00a6070 secure :-1 s=READY pgs=154 cs=0 l=1 rev1=1 crypto rx=0x7f24b4199660 tx=0x7f24a4005f90 comp rx=0 tx=0).stop 2024-04-23T10:24:35.244 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:24:35.245+0000 7f24bc5cc640 1 -- 172.21.15.114:0/3316450038 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f24b4100e90 msgr2=0x7f24b4198150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T10:24:35.244 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:24:35.245+0000 7f24bc5cc640 1 --2- 172.21.15.114:0/3316450038 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f24b4100e90 0x7f24b4198150 secure :-1 s=READY pgs=192 cs=0 l=1 rev1=1 crypto rx=0x7f24b000e9c0 tx=0x7f24b000ee80 comp rx=0 tx=0).stop 2024-04-23T10:24:35.244 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:24:35.245+0000 7f24bc5cc640 1 -- 172.21.15.114:0/3316450038 shutdown_connections 2024-04-23T10:24:35.244 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:24:35.245+0000 7f24bc5cc640 1 --2- 172.21.15.114:0/3316450038 >> [v2:172.21.15.114:6800/465333808,v1:172.21.15.114:6801/465333808] conn(0x7f24a00a3bb0 0x7f24a00a6070 unknown :-1 s=CLOSED pgs=154 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:24:35.244 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:24:35.245+0000 7f24bc5cc640 1 --2- 172.21.15.114:0/3316450038 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f24b41029a0 0x7f24b419f700 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:24:35.244 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:24:35.245+0000 7f24bc5cc640 1 --2- 172.21.15.114:0/3316450038 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f24b4101fe0 0x7f24b4198690 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:24:35.244 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:24:35.245+0000 7f24bc5cc640 1 --2- 172.21.15.114:0/3316450038 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f24b4100e90 0x7f24b4198150 unknown :-1 s=CLOSED pgs=192 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:24:35.244 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:24:35.245+0000 7f24bc5cc640 1 -- 172.21.15.114:0/3316450038 >> 172.21.15.114:0/3316450038 conn(0x7f24b40fc400 msgr2=0x7f24b4105370 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T10:24:35.244 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:24:35.245+0000 7f24bc5cc640 1 -- 172.21.15.114:0/3316450038 shutdown_connections 2024-04-23T10:24:35.245 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:24:35.245+0000 7f24bc5cc640 1 -- 172.21.15.114:0/3316450038 wait complete. 2024-04-23T10:24:35.405 INFO:journalctl@ceph.mds.a.smithi114.stdout:Apr 23 10:24:35 smithi114 podman[68581]: 2024-04-23 10:24:35.286355052 +0000 UTC m=+0.018459370 image pull da4b004c09e820713483d039bc76fbd660be678f2ea3c03e059a6161b74dab76 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:068a558e3af4932cf1c60dd321c0bdf500075eb52ea1bf455e0973ba114fbfd4 2024-04-23T10:24:35.492 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 23 10:24:35 smithi119 ceph-mon[32608]: Deploying daemon mds.a on smithi114 2024-04-23T10:24:35.492 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 23 10:24:35 smithi119 ceph-mon[32608]: pgmap v277: 1 pgs: 1 active+clean; 577 KiB data, 365 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-23T10:24:35.578 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 23 10:24:35 smithi204 ceph-mon[32649]: Deploying daemon mds.a on smithi114 2024-04-23T10:24:35.578 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 23 10:24:35 smithi204 ceph-mon[32649]: pgmap v277: 1 pgs: 1 active+clean; 577 KiB data, 365 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-23T10:24:35.666 INFO:journalctl@ceph.mds.a.smithi114.stdout:Apr 23 10:24:35 smithi114 podman[68581]: 2024-04-23 10:24:35.406111112 +0000 UTC m=+0.138215431 container create b4bc3a39121733c1ecb40711b4f4880211c6ddc3656a0c4f1887d99d32bb941a (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:068a558e3af4932cf1c60dd321c0bdf500075eb52ea1bf455e0973ba114fbfd4, name=ceph-aa240572-0159-11ef-bc93-c7b262605968-mds-a, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 8 Base Image, RELEASE=wip-lusov-quiescer-d69f13d, org.label-schema.schema-version=1.0, org.label-schema.license=GPLv2, GIT_CLEAN=True, ceph=True, org.label-schema.build-date=20240416, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, maintainer=Guillaume Abrioux , io.buildah.version=1.33.5, GIT_BRANCH=HEAD) 2024-04-23T10:24:35.666 INFO:journalctl@ceph.mds.a.smithi114.stdout:Apr 23 10:24:35 smithi114 podman[68581]: 2024-04-23 10:24:35.64216281 +0000 UTC m=+0.374267129 container init b4bc3a39121733c1ecb40711b4f4880211c6ddc3656a0c4f1887d99d32bb941a (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:068a558e3af4932cf1c60dd321c0bdf500075eb52ea1bf455e0973ba114fbfd4, name=ceph-aa240572-0159-11ef-bc93-c7b262605968-mds-a, ceph=True, org.label-schema.vendor=CentOS, GIT_CLEAN=True, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.build-date=20240416, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, io.buildah.version=1.33.5, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=wip-lusov-quiescer-d69f13d) 2024-04-23T10:24:35.667 INFO:journalctl@ceph.mds.a.smithi114.stdout:Apr 23 10:24:35 smithi114 podman[68581]: 2024-04-23 10:24:35.645127276 +0000 UTC m=+0.377231604 container start b4bc3a39121733c1ecb40711b4f4880211c6ddc3656a0c4f1887d99d32bb941a (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:068a558e3af4932cf1c60dd321c0bdf500075eb52ea1bf455e0973ba114fbfd4, name=ceph-aa240572-0159-11ef-bc93-c7b262605968-mds-a, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240416, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, RELEASE=wip-lusov-quiescer-d69f13d, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, GIT_CLEAN=True, GIT_BRANCH=HEAD, io.buildah.version=1.33.5, ceph=True, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.vendor=CentOS) 2024-04-23T10:24:36.004 INFO:journalctl@ceph.mds.a.smithi114.stdout:Apr 23 10:24:35 smithi114 ceph-mds[68610]: -- 172.21.15.114:0/51959741 <== mon.0 v2:172.21.15.114:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 194+0+0 (secure 0 0 0) 0x55b9be1d5a40 con 0x55b9be1a8c00 2024-04-23T10:24:36.004 INFO:journalctl@ceph.mds.a.smithi114.stdout:Apr 23 10:24:35 smithi114 ceph-aa240572-0159-11ef-bc93-c7b262605968-mds-a[68604]: starting mds.a at 2024-04-23T10:24:36.004 INFO:journalctl@ceph.mds.a.smithi114.stdout:Apr 23 10:24:35 smithi114 bash[68581]: b4bc3a39121733c1ecb40711b4f4880211c6ddc3656a0c4f1887d99d32bb941a 2024-04-23T10:24:36.004 INFO:journalctl@ceph.mds.a.smithi114.stdout:Apr 23 10:24:35 smithi114 systemd[1]: Started Ceph mds.a for aa240572-0159-11ef-bc93-c7b262605968. 2024-04-23T10:24:36.242 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 23 10:24:36 smithi119 ceph-mon[32608]: from='client.? 172.21.15.114:0/3316450038' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-23T10:24:36.242 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 23 10:24:36 smithi119 ceph-mon[32608]: from='mgr.14152 172.21.15.114:0/1529706138' entity='mgr.x' 2024-04-23T10:24:36.242 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 23 10:24:36 smithi119 ceph-mon[32608]: from='mgr.14152 172.21.15.114:0/1529706138' entity='mgr.x' 2024-04-23T10:24:36.242 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 23 10:24:36 smithi119 ceph-mon[32608]: from='mgr.14152 172.21.15.114:0/1529706138' entity='mgr.x' 2024-04-23T10:24:36.242 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 23 10:24:36 smithi119 ceph-mon[32608]: from='mgr.14152 172.21.15.114:0/1529706138' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.b", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-04-23T10:24:36.242 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 23 10:24:36 smithi119 ceph-mon[32608]: from='mgr.14152 172.21.15.114:0/1529706138' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.b", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-04-23T10:24:36.242 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 23 10:24:36 smithi119 ceph-mon[32608]: from='mgr.14152 172.21.15.114:0/1529706138' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-23T10:24:36.504 INFO:journalctl@ceph.mon.a.smithi114.stdout:Apr 23 10:24:36 smithi114 ceph-mon[29255]: from='client.? 172.21.15.114:0/3316450038' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-23T10:24:36.504 INFO:journalctl@ceph.mon.a.smithi114.stdout:Apr 23 10:24:36 smithi114 ceph-mon[29255]: from='mgr.14152 172.21.15.114:0/1529706138' entity='mgr.x' 2024-04-23T10:24:36.504 INFO:journalctl@ceph.mon.a.smithi114.stdout:Apr 23 10:24:36 smithi114 ceph-mon[29255]: from='mgr.14152 172.21.15.114:0/1529706138' entity='mgr.x' 2024-04-23T10:24:36.504 INFO:journalctl@ceph.mon.a.smithi114.stdout:Apr 23 10:24:36 smithi114 ceph-mon[29255]: from='mgr.14152 172.21.15.114:0/1529706138' entity='mgr.x' 2024-04-23T10:24:36.504 INFO:journalctl@ceph.mon.a.smithi114.stdout:Apr 23 10:24:36 smithi114 ceph-mon[29255]: from='mgr.14152 172.21.15.114:0/1529706138' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.b", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-04-23T10:24:36.504 INFO:journalctl@ceph.mon.a.smithi114.stdout:Apr 23 10:24:36 smithi114 ceph-mon[29255]: from='mgr.14152 172.21.15.114:0/1529706138' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.b", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-04-23T10:24:36.504 INFO:journalctl@ceph.mon.a.smithi114.stdout:Apr 23 10:24:36 smithi114 ceph-mon[29255]: from='mgr.14152 172.21.15.114:0/1529706138' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-23T10:24:36.578 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 23 10:24:36 smithi204 ceph-mon[32649]: from='client.? 172.21.15.114:0/3316450038' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-23T10:24:36.578 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 23 10:24:36 smithi204 ceph-mon[32649]: from='mgr.14152 172.21.15.114:0/1529706138' entity='mgr.x' 2024-04-23T10:24:36.578 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 23 10:24:36 smithi204 ceph-mon[32649]: from='mgr.14152 172.21.15.114:0/1529706138' entity='mgr.x' 2024-04-23T10:24:36.578 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 23 10:24:36 smithi204 ceph-mon[32649]: from='mgr.14152 172.21.15.114:0/1529706138' entity='mgr.x' 2024-04-23T10:24:36.578 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 23 10:24:36 smithi204 ceph-mon[32649]: from='mgr.14152 172.21.15.114:0/1529706138' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.b", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-04-23T10:24:36.578 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 23 10:24:36 smithi204 ceph-mon[32649]: from='mgr.14152 172.21.15.114:0/1529706138' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.b", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-04-23T10:24:36.578 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 23 10:24:36 smithi204 ceph-mon[32649]: from='mgr.14152 172.21.15.114:0/1529706138' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-23T10:24:37.257 DEBUG:teuthology.orchestra.run.smithi114:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-23T10:24:37.328 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 23 10:24:37 smithi204 ceph-mon[32649]: Deploying daemon mds.b on smithi119 2024-04-23T10:24:37.328 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 23 10:24:37 smithi204 ceph-mon[32649]: pgmap v278: 1 pgs: 1 active+clean; 577 KiB data, 365 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-23T10:24:37.328 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 23 10:24:37 smithi204 ceph-mon[32649]: mds.? [v2:172.21.15.114:6840/3544523891,v1:172.21.15.114:6841/3544523891] up:boot 2024-04-23T10:24:37.328 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 23 10:24:37 smithi204 ceph-mon[32649]: fsmap 11 up:standby 2024-04-23T10:24:37.328 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 23 10:24:37 smithi204 ceph-mon[32649]: from='mgr.14152 172.21.15.114:0/1529706138' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "a"}]: dispatch 2024-04-23T10:24:37.373 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:24:37.372+0000 7f28f6dcd640 1 -- 172.21.15.114:0/2754595983 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f28f0103190 msgr2=0x7f28f0103610 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T10:24:37.374 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:24:37.372+0000 7f28f6dcd640 1 --2- 172.21.15.114:0/2754595983 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f28f0103190 0x7f28f0103610 secure :-1 s=READY pgs=195 cs=0 l=1 rev1=1 crypto rx=0x7f28dc009900 tx=0x7f28dc02a410 comp rx=0 tx=0).stop 2024-04-23T10:24:37.374 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:24:37.372+0000 7f28f6dcd640 1 -- 172.21.15.114:0/2754595983 shutdown_connections 2024-04-23T10:24:37.374 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:24:37.372+0000 7f28f6dcd640 1 --2- 172.21.15.114:0/2754595983 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f28f0103b50 0x7f28f010a3e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:24:37.374 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:24:37.372+0000 7f28f6dcd640 1 --2- 172.21.15.114:0/2754595983 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f28f0103190 0x7f28f0103610 unknown :-1 s=CLOSED pgs=195 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:24:37.374 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:24:37.372+0000 7f28f6dcd640 1 --2- 172.21.15.114:0/2754595983 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f28f0101fa0 0x7f28f01023c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:24:37.374 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:24:37.372+0000 7f28f6dcd640 1 -- 172.21.15.114:0/2754595983 >> 172.21.15.114:0/2754595983 conn(0x7f28f00fd530 msgr2=0x7f28f00ff950 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T10:24:37.374 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:24:37.372+0000 7f28f6dcd640 1 -- 172.21.15.114:0/2754595983 shutdown_connections 2024-04-23T10:24:37.374 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:24:37.372+0000 7f28f6dcd640 1 -- 172.21.15.114:0/2754595983 wait complete. 2024-04-23T10:24:37.374 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:24:37.372+0000 7f28f6dcd640 1 Processor -- start 2024-04-23T10:24:37.374 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:24:37.373+0000 7f28f6dcd640 1 -- start start 2024-04-23T10:24:37.374 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:24:37.373+0000 7f28f6dcd640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f28f0101fa0 0x7f28f0075d00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T10:24:37.374 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:24:37.373+0000 7f28f6dcd640 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f28f0103190 0x7f28f0074340 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T10:24:37.375 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:24:37.373+0000 7f28f6dcd640 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f28f0103b50 0x7f28f0074880 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T10:24:37.375 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:24:37.373+0000 7f28f6dcd640 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x7f28f010dae0 con 0x7f28f0101fa0 2024-04-23T10:24:37.375 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:24:37.373+0000 7f28f6dcd640 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 -- 0x7f28f010d960 con 0x7f28f0103b50 2024-04-23T10:24:37.375 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:24:37.373+0000 7f28f6dcd640 1 -- --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_getmap magic: 0 -- 0x7f28f010dc60 con 0x7f28f0103190 2024-04-23T10:24:37.375 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:24:37.373+0000 7f28f4b69640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f28f0101fa0 0x7f28f0075d00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T10:24:37.375 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:24:37.373+0000 7f28f4b69640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f28f0101fa0 0x7f28f0075d00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.114:44382/0 (socket says 172.21.15.114:44382) 2024-04-23T10:24:37.375 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:24:37.373+0000 7f28f4b69640 1 -- 172.21.15.114:0/3641715768 learned_addr learned my addr 172.21.15.114:0/3641715768 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-04-23T10:24:37.375 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:24:37.373+0000 7f28effff640 1 --2- 172.21.15.114:0/3641715768 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f28f0103190 0x7f28f0074340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T10:24:37.375 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:24:37.373+0000 7f28f536a640 1 --2- 172.21.15.114:0/3641715768 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f28f0103b50 0x7f28f0074880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T10:24:37.375 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:24:37.373+0000 7f28f4b69640 1 -- 172.21.15.114:0/3641715768 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f28f0103190 msgr2=0x7f28f0074340 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T10:24:37.375 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:24:37.373+0000 7f28f4b69640 1 --2- 172.21.15.114:0/3641715768 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f28f0103190 0x7f28f0074340 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:24:37.375 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:24:37.373+0000 7f28f4b69640 1 -- 172.21.15.114:0/3641715768 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f28f0103b50 msgr2=0x7f28f0074880 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T10:24:37.375 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:24:37.373+0000 7f28f4b69640 1 --2- 172.21.15.114:0/3641715768 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f28f0103b50 0x7f28f0074880 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:24:37.376 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:24:37.373+0000 7f28f4b69640 1 -- 172.21.15.114:0/3641715768 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f28f0075110 con 0x7f28f0101fa0 2024-04-23T10:24:37.376 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:24:37.373+0000 7f28f4b69640 1 --2- 172.21.15.114:0/3641715768 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f28f0101fa0 0x7f28f0075d00 secure :-1 s=READY pgs=196 cs=0 l=1 rev1=1 crypto rx=0x7f28e400d950 tx=0x7f28e400de10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T10:24:37.376 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:24:37.373+0000 7f28edffb640 1 -- 172.21.15.114:0/3641715768 <== mon.0 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f28e4017070 con 0x7f28f0101fa0 2024-04-23T10:24:37.376 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:24:37.373+0000 7f28f6dcd640 1 -- 172.21.15.114:0/3641715768 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f28f01a22e0 con 0x7f28f0101fa0 2024-04-23T10:24:37.376 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:24:37.373+0000 7f28edffb640 1 -- 172.21.15.114:0/3641715768 <== mon.0 v2:172.21.15.114:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f28e4009c30 con 0x7f28f0101fa0 2024-04-23T10:24:37.376 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:24:37.373+0000 7f28edffb640 1 -- 172.21.15.114:0/3641715768 <== mon.0 v2:172.21.15.114:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f28e400f620 con 0x7f28f0101fa0 2024-04-23T10:24:37.376 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:24:37.374+0000 7f28f6dcd640 1 -- 172.21.15.114:0/3641715768 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f28f01a27e0 con 0x7f28f0101fa0 2024-04-23T10:24:37.376 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:24:37.375+0000 7f28f6dcd640 1 -- 172.21.15.114:0/3641715768 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f28f0109390 con 0x7f28f0101fa0 2024-04-23T10:24:37.377 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:24:37.376+0000 7f28edffb640 1 -- 172.21.15.114:0/3641715768 <== mon.0 v2:172.21.15.114:3300/0 4 ==== mgrmap(e 16) ==== 138647+0+0 (secure 0 0 0) 0x7f28e401a070 con 0x7f28f0101fa0 2024-04-23T10:24:37.377 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:24:37.376+0000 7f28edffb640 1 --2- 172.21.15.114:0/3641715768 >> [v2:172.21.15.114:6800/465333808,v1:172.21.15.114:6801/465333808] conn(0x7f28e00a3bb0 0x7f28e00a6070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T10:24:37.377 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:24:37.377+0000 7f28effff640 1 --2- 172.21.15.114:0/3641715768 >> [v2:172.21.15.114:6800/465333808,v1:172.21.15.114:6801/465333808] conn(0x7f28e00a3bb0 0x7f28e00a6070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T10:24:37.377 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:24:37.377+0000 7f28edffb640 1 -- 172.21.15.114:0/3641715768 <== mon.0 v2:172.21.15.114:3300/0 5 ==== osd_map(83..83 src has 1..83) ==== 7127+0+0 (secure 0 0 0) 0x7f28e40cfa50 con 0x7f28f0101fa0 2024-04-23T10:24:37.377 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:24:37.377+0000 7f28effff640 1 --2- 172.21.15.114:0/3641715768 >> [v2:172.21.15.114:6800/465333808,v1:172.21.15.114:6801/465333808] conn(0x7f28e00a3bb0 0x7f28e00a6070 secure :-1 s=READY pgs=156 cs=0 l=1 rev1=1 crypto rx=0x7f28f0075a70 tx=0x7f28dc0314a0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T10:24:37.380 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:24:37.381+0000 7f28edffb640 1 -- 172.21.15.114:0/3641715768 <== mon.0 v2:172.21.15.114:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f28e4009de0 con 0x7f28f0101fa0 2024-04-23T10:24:37.470 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 23 10:24:37 smithi119 ceph-mon[32608]: Deploying daemon mds.b on smithi119 2024-04-23T10:24:37.471 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 23 10:24:37 smithi119 ceph-mon[32608]: pgmap v278: 1 pgs: 1 active+clean; 577 KiB data, 365 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-23T10:24:37.471 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 23 10:24:37 smithi119 ceph-mon[32608]: mds.? [v2:172.21.15.114:6840/3544523891,v1:172.21.15.114:6841/3544523891] up:boot 2024-04-23T10:24:37.471 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 23 10:24:37 smithi119 ceph-mon[32608]: fsmap 11 up:standby 2024-04-23T10:24:37.471 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 23 10:24:37 smithi119 ceph-mon[32608]: from='mgr.14152 172.21.15.114:0/1529706138' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "a"}]: dispatch 2024-04-23T10:24:37.504 INFO:journalctl@ceph.mon.a.smithi114.stdout:Apr 23 10:24:37 smithi114 ceph-mon[29255]: Deploying daemon mds.b on smithi119 2024-04-23T10:24:37.504 INFO:journalctl@ceph.mon.a.smithi114.stdout:Apr 23 10:24:37 smithi114 ceph-mon[29255]: pgmap v278: 1 pgs: 1 active+clean; 577 KiB data, 365 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-23T10:24:37.504 INFO:journalctl@ceph.mon.a.smithi114.stdout:Apr 23 10:24:37 smithi114 ceph-mon[29255]: mds.? [v2:172.21.15.114:6840/3544523891,v1:172.21.15.114:6841/3544523891] up:boot 2024-04-23T10:24:37.504 INFO:journalctl@ceph.mon.a.smithi114.stdout:Apr 23 10:24:37 smithi114 ceph-mon[29255]: fsmap 11 up:standby 2024-04-23T10:24:37.504 INFO:journalctl@ceph.mon.a.smithi114.stdout:Apr 23 10:24:37 smithi114 ceph-mon[29255]: from='mgr.14152 172.21.15.114:0/1529706138' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "a"}]: dispatch 2024-04-23T10:24:37.596 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:24:37.597+0000 7f28f6dcd640 1 -- 172.21.15.114:0/3641715768 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f28f002d7e0 con 0x7f28f0101fa0 2024-04-23T10:24:37.597 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:24:37.598+0000 7f28edffb640 1 -- 172.21.15.114:0/3641715768 <== mon.0 v2:172.21.15.114:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 12 v12) ==== 94+0+9671 (secure 0 0 0) 0x7f28e409c8d0 con 0x7f28f0101fa0 2024-04-23T10:24:37.597 INFO:teuthology.orchestra.run.smithi114.stdout: 2024-04-23T10:24:37.597 INFO:teuthology.orchestra.run.smithi114.stdout:{"epoch":12,"default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14547,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.114:6835/663493580","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6834","nonce":663493580},{"type":"v1","addr":"172.21.15.114:6835","nonce":663493580}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14565,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.114:6837/1439252154","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6836","nonce":1439252154},{"type":"v1","addr":"172.21.15.114:6837","nonce":1439252154}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5},{"gid":14634,"name":"g","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.114:6839/3849796939","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6838","nonce":3849796939},{"type":"v1","addr":"172.21.15.114:6839","nonce":3849796939}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":11},{"gid":14646,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.114:6841/3544523891","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6840","nonce":3544523891},{"type":"v1","addr":"172.21.15.114:6841","nonce":3544523891}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":12},{"gid":24385,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.204:6833/3866561754","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6832","nonce":3866561754},{"type":"v1","addr":"172.21.15.204:6833","nonce":3866561754}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2},{"gid":24427,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.204:6835/2533793172","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6834","nonce":2533793172},{"type":"v1","addr":"172.21.15.204:6835","nonce":2533793172}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":7},{"gid":24439,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.204:6837/513822179","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6836","nonce":513822179},{"type":"v1","addr":"172.21.15.204:6837","nonce":513822179}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":8},{"gid":24443,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.119:6833/783244660","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6832","nonce":783244660},{"type":"v1","addr":"172.21.15.119:6833","nonce":783244660}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":24463,"name":"i","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.204:6839/3222658893","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6838","nonce":3222658893},{"type":"v1","addr":"172.21.15.204:6839","nonce":3222658893}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":10},{"gid":24479,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.119:6835/2033916709","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6834","nonce":2033916709},{"type":"v1","addr":"172.21.15.119:6835","nonce":2033916709}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":6},{"gid":24509,"name":"h","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.119:6837/1872363128","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6836","nonce":1872363128},{"type":"v1","addr":"172.21.15.119:6837","nonce":1872363128}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":9}],"filesystems":[]} 2024-04-23T10:24:37.597 INFO:teuthology.orchestra.run.smithi114.stderr:dumped fsmap epoch 12 2024-04-23T10:24:37.599 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:24:37.600+0000 7f28f6dcd640 1 -- 172.21.15.114:0/3641715768 >> [v2:172.21.15.114:6800/465333808,v1:172.21.15.114:6801/465333808] conn(0x7f28e00a3bb0 msgr2=0x7f28e00a6070 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T10:24:37.599 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:24:37.600+0000 7f28f6dcd640 1 --2- 172.21.15.114:0/3641715768 >> [v2:172.21.15.114:6800/465333808,v1:172.21.15.114:6801/465333808] conn(0x7f28e00a3bb0 0x7f28e00a6070 secure :-1 s=READY pgs=156 cs=0 l=1 rev1=1 crypto rx=0x7f28f0075a70 tx=0x7f28dc0314a0 comp rx=0 tx=0).stop 2024-04-23T10:24:37.599 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:24:37.600+0000 7f28f6dcd640 1 -- 172.21.15.114:0/3641715768 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f28f0101fa0 msgr2=0x7f28f0075d00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T10:24:37.599 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:24:37.600+0000 7f28f6dcd640 1 --2- 172.21.15.114:0/3641715768 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f28f0101fa0 0x7f28f0075d00 secure :-1 s=READY pgs=196 cs=0 l=1 rev1=1 crypto rx=0x7f28e400d950 tx=0x7f28e400de10 comp rx=0 tx=0).stop 2024-04-23T10:24:37.599 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:24:37.600+0000 7f28f6dcd640 1 -- 172.21.15.114:0/3641715768 shutdown_connections 2024-04-23T10:24:37.599 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:24:37.600+0000 7f28f6dcd640 1 --2- 172.21.15.114:0/3641715768 >> [v2:172.21.15.114:6800/465333808,v1:172.21.15.114:6801/465333808] conn(0x7f28e00a3bb0 0x7f28e00a6070 unknown :-1 s=CLOSED pgs=156 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:24:37.599 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:24:37.600+0000 7f28f6dcd640 1 --2- 172.21.15.114:0/3641715768 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f28f0103b50 0x7f28f0074880 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:24:37.600 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:24:37.600+0000 7f28f6dcd640 1 --2- 172.21.15.114:0/3641715768 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f28f0103190 0x7f28f0074340 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:24:37.600 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:24:37.600+0000 7f28f6dcd640 1 --2- 172.21.15.114:0/3641715768 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f28f0101fa0 0x7f28f0075d00 unknown :-1 s=CLOSED pgs=196 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:24:37.600 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:24:37.600+0000 7f28f6dcd640 1 -- 172.21.15.114:0/3641715768 >> 172.21.15.114:0/3641715768 conn(0x7f28f00fd530 msgr2=0x7f28f0106560 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T10:24:37.600 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:24:37.600+0000 7f28f6dcd640 1 -- 172.21.15.114:0/3641715768 shutdown_connections 2024-04-23T10:24:37.600 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:24:37.600+0000 7f28f6dcd640 1 -- 172.21.15.114:0/3641715768 wait complete. 2024-04-23T10:24:38.106 INFO:journalctl@ceph.mds.b.smithi119.stdout:Apr 23 10:24:37 smithi119 systemd[1]: Starting Ceph mds.b for aa240572-0159-11ef-bc93-c7b262605968... 2024-04-23T10:24:38.383 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 23 10:24:38 smithi119 ceph-mon[32608]: from='client.? 172.21.15.114:0/3641715768' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-23T10:24:38.383 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 23 10:24:38 smithi119 ceph-mon[32608]: pgmap v279: 1 pgs: 1 active+clean; 577 KiB data, 365 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-23T10:24:38.384 INFO:journalctl@ceph.mds.b.smithi119.stdout:Apr 23 10:24:38 smithi119 podman[62674]: 2024-04-23 10:24:38.070345367 +0000 UTC m=+0.017883196 image pull da4b004c09e820713483d039bc76fbd660be678f2ea3c03e059a6161b74dab76 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:068a558e3af4932cf1c60dd321c0bdf500075eb52ea1bf455e0973ba114fbfd4 2024-04-23T10:24:38.384 INFO:journalctl@ceph.mds.b.smithi119.stdout:Apr 23 10:24:38 smithi119 podman[62674]: 2024-04-23 10:24:38.182769356 +0000 UTC m=+0.130307212 container create c3c18069909792cef80b3deb85df8e647afc0cad3b2249403961570a05f53d8e (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:068a558e3af4932cf1c60dd321c0bdf500075eb52ea1bf455e0973ba114fbfd4, name=ceph-aa240572-0159-11ef-bc93-c7b262605968-mds-b, RELEASE=wip-lusov-quiescer-d69f13d, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_BRANCH=HEAD, org.label-schema.build-date=20240416, org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, GIT_CLEAN=True, org.label-schema.license=GPLv2, ceph=True, org.label-schema.name=CentOS Stream 8 Base Image, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, io.buildah.version=1.33.5, maintainer=Guillaume Abrioux ) 2024-04-23T10:24:38.504 INFO:journalctl@ceph.mon.a.smithi114.stdout:Apr 23 10:24:38 smithi114 ceph-mon[29255]: from='client.? 172.21.15.114:0/3641715768' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-23T10:24:38.504 INFO:journalctl@ceph.mon.a.smithi114.stdout:Apr 23 10:24:38 smithi114 ceph-mon[29255]: pgmap v279: 1 pgs: 1 active+clean; 577 KiB data, 365 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-23T10:24:38.578 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 23 10:24:38 smithi204 ceph-mon[32649]: from='client.? 172.21.15.114:0/3641715768' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-23T10:24:38.578 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 23 10:24:38 smithi204 ceph-mon[32649]: pgmap v279: 1 pgs: 1 active+clean; 577 KiB data, 365 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-23T10:24:38.742 INFO:journalctl@ceph.mds.b.smithi119.stdout:Apr 23 10:24:38 smithi119 podman[62674]: 2024-04-23 10:24:38.385195821 +0000 UTC m=+0.332733638 container init c3c18069909792cef80b3deb85df8e647afc0cad3b2249403961570a05f53d8e (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:068a558e3af4932cf1c60dd321c0bdf500075eb52ea1bf455e0973ba114fbfd4, name=ceph-aa240572-0159-11ef-bc93-c7b262605968-mds-b, maintainer=Guillaume Abrioux , RELEASE=wip-lusov-quiescer-d69f13d, CEPH_POINT_RELEASE=, org.label-schema.license=GPLv2, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, GIT_CLEAN=True, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, org.label-schema.build-date=20240416, org.label-schema.name=CentOS Stream 8 Base Image, GIT_BRANCH=HEAD, io.buildah.version=1.33.5, org.label-schema.schema-version=1.0) 2024-04-23T10:24:38.742 INFO:journalctl@ceph.mds.b.smithi119.stdout:Apr 23 10:24:38 smithi119 podman[62674]: 2024-04-23 10:24:38.387852482 +0000 UTC m=+0.335390297 container start c3c18069909792cef80b3deb85df8e647afc0cad3b2249403961570a05f53d8e (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:068a558e3af4932cf1c60dd321c0bdf500075eb52ea1bf455e0973ba114fbfd4, name=ceph-aa240572-0159-11ef-bc93-c7b262605968-mds-b, ceph=True, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_CLEAN=True, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0, org.label-schema.build-date=20240416, RELEASE=wip-lusov-quiescer-d69f13d, org.label-schema.name=CentOS Stream 8 Base Image, CEPH_POINT_RELEASE=, io.buildah.version=1.33.5, org.label-schema.license=GPLv2, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67) 2024-04-23T10:24:38.742 INFO:journalctl@ceph.mds.b.smithi119.stdout:Apr 23 10:24:38 smithi119 ceph-mds[62693]: -- 172.21.15.119:0/2330987501 <== mon.2 v2:172.21.15.119:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 194+0+0 (secure 0 0 0) 0x5571ce1a5a40 con 0x5571ce178c00 2024-04-23T10:24:38.742 INFO:journalctl@ceph.mds.b.smithi119.stdout:Apr 23 10:24:38 smithi119 ceph-aa240572-0159-11ef-bc93-c7b262605968-mds-b[62689]: starting mds.b at 2024-04-23T10:24:38.742 INFO:journalctl@ceph.mds.b.smithi119.stdout:Apr 23 10:24:38 smithi119 bash[62674]: c3c18069909792cef80b3deb85df8e647afc0cad3b2249403961570a05f53d8e 2024-04-23T10:24:38.742 INFO:journalctl@ceph.mds.b.smithi119.stdout:Apr 23 10:24:38 smithi119 systemd[1]: Started Ceph mds.b for aa240572-0159-11ef-bc93-c7b262605968. 2024-04-23T10:24:39.611 DEBUG:teuthology.orchestra.run.smithi114:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-23T10:24:39.717 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:24:39.716+0000 7f2153308640 1 -- 172.21.15.114:0/2660824578 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f214c105360 msgr2=0x7f214c107750 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T10:24:39.717 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:24:39.716+0000 7f2153308640 1 --2- 172.21.15.114:0/2660824578 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f214c105360 0x7f214c107750 secure :-1 s=READY pgs=197 cs=0 l=1 rev1=1 crypto rx=0x7f213c009900 tx=0x7f213c02a410 comp rx=0 tx=0).stop 2024-04-23T10:24:39.717 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:24:39.716+0000 7f2153308640 1 -- 172.21.15.114:0/2660824578 shutdown_connections 2024-04-23T10:24:39.717 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:24:39.716+0000 7f2153308640 1 --2- 172.21.15.114:0/2660824578 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f214c107c90 0x7f214c10a0e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:24:39.717 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:24:39.716+0000 7f2153308640 1 --2- 172.21.15.114:0/2660824578 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f214c105360 0x7f214c107750 unknown :-1 s=CLOSED pgs=197 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:24:39.717 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:24:39.716+0000 7f2153308640 1 --2- 172.21.15.114:0/2660824578 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f214c102a30 0x7f214c104e20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:24:39.718 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:24:39.716+0000 7f2153308640 1 -- 172.21.15.114:0/2660824578 >> 172.21.15.114:0/2660824578 conn(0x7f214c0fc440 msgr2=0x7f214c0fe860 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T10:24:39.718 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:24:39.716+0000 7f2153308640 1 -- 172.21.15.114:0/2660824578 shutdown_connections 2024-04-23T10:24:39.718 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:24:39.716+0000 7f2153308640 1 -- 172.21.15.114:0/2660824578 wait complete. 2024-04-23T10:24:39.718 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:24:39.716+0000 7f2153308640 1 Processor -- start 2024-04-23T10:24:39.718 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:24:39.716+0000 7f2153308640 1 -- start start 2024-04-23T10:24:39.718 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:24:39.717+0000 7f2153308640 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f214c102a30 0x7f214c1023a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T10:24:39.718 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:24:39.717+0000 7f2153308640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f214c105360 0x7f214c1009e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T10:24:39.718 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:24:39.717+0000 7f2153308640 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f214c107c90 0x7f214c100f20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T10:24:39.718 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:24:39.717+0000 7f2153308640 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x7f214c10daf0 con 0x7f214c105360 2024-04-23T10:24:39.718 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:24:39.717+0000 7f2153308640 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 -- 0x7f214c10d970 con 0x7f214c107c90 2024-04-23T10:24:39.718 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:24:39.717+0000 7f2153308640 1 -- --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_getmap magic: 0 -- 0x7f214c10dc70 con 0x7f214c102a30 2024-04-23T10:24:39.718 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:24:39.717+0000 7f21508a3640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f214c105360 0x7f214c1009e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T10:24:39.718 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:24:39.717+0000 7f21508a3640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f214c105360 0x7f214c1009e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.114:44402/0 (socket says 172.21.15.114:44402) 2024-04-23T10:24:39.719 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:24:39.717+0000 7f21508a3640 1 -- 172.21.15.114:0/2261177185 learned_addr learned my addr 172.21.15.114:0/2261177185 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-04-23T10:24:39.719 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:24:39.717+0000 7f21510a4640 1 --2- 172.21.15.114:0/2261177185 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f214c102a30 0x7f214c1023a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T10:24:39.719 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:24:39.717+0000 7f21508a3640 1 -- 172.21.15.114:0/2261177185 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f214c102a30 msgr2=0x7f214c1023a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T10:24:39.719 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:24:39.717+0000 7f21508a3640 1 --2- 172.21.15.114:0/2261177185 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f214c102a30 0x7f214c1023a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:24:39.719 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:24:39.717+0000 7f21508a3640 1 -- 172.21.15.114:0/2261177185 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f214c107c90 msgr2=0x7f214c100f20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T10:24:39.719 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:24:39.717+0000 7f21508a3640 1 --2- 172.21.15.114:0/2261177185 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f214c107c90 0x7f214c100f20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:24:39.719 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:24:39.717+0000 7f21508a3640 1 -- 172.21.15.114:0/2261177185 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f214c1017b0 con 0x7f214c105360 2024-04-23T10:24:39.719 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:24:39.717+0000 7f21508a3640 1 --2- 172.21.15.114:0/2261177185 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f214c105360 0x7f214c1009e0 secure :-1 s=READY pgs=198 cs=0 l=1 rev1=1 crypto rx=0x7f213c006010 tx=0x7f213c0042f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T10:24:39.719 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:24:39.717+0000 7f21427fc640 1 -- 172.21.15.114:0/2261177185 <== mon.0 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f213c03f030 con 0x7f214c105360 2024-04-23T10:24:39.719 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:24:39.717+0000 7f21427fc640 1 -- 172.21.15.114:0/2261177185 <== mon.0 v2:172.21.15.114:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f213c003cb0 con 0x7f214c105360 2024-04-23T10:24:39.719 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:24:39.717+0000 7f2153308640 1 -- 172.21.15.114:0/2261177185 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f214c1a2300 con 0x7f214c105360 2024-04-23T10:24:39.719 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:24:39.717+0000 7f21427fc640 1 -- 172.21.15.114:0/2261177185 <== mon.0 v2:172.21.15.114:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f213c035070 con 0x7f214c105360 2024-04-23T10:24:39.719 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:24:39.717+0000 7f2153308640 1 -- 172.21.15.114:0/2261177185 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f214c1a2850 con 0x7f214c105360 2024-04-23T10:24:39.719 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:24:39.718+0000 7f2153308640 1 -- 172.21.15.114:0/2261177185 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f214c0fdf30 con 0x7f214c105360 2024-04-23T10:24:39.720 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:24:39.719+0000 7f21427fc640 1 -- 172.21.15.114:0/2261177185 <== mon.0 v2:172.21.15.114:3300/0 4 ==== mgrmap(e 16) ==== 138647+0+0 (secure 0 0 0) 0x7f213c042030 con 0x7f214c105360 2024-04-23T10:24:39.720 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:24:39.720+0000 7f21427fc640 1 --2- 172.21.15.114:0/2261177185 >> [v2:172.21.15.114:6800/465333808,v1:172.21.15.114:6801/465333808] conn(0x7f21380a3bb0 0x7f21380a6070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T10:24:39.720 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:24:39.720+0000 7f21510a4640 1 --2- 172.21.15.114:0/2261177185 >> [v2:172.21.15.114:6800/465333808,v1:172.21.15.114:6801/465333808] conn(0x7f21380a3bb0 0x7f21380a6070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T10:24:39.721 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:24:39.720+0000 7f21427fc640 1 -- 172.21.15.114:0/2261177185 <== mon.0 v2:172.21.15.114:3300/0 5 ==== osd_map(83..83 src has 1..83) ==== 7127+0+0 (secure 0 0 0) 0x7f213c0f0a60 con 0x7f214c105360 2024-04-23T10:24:39.721 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:24:39.721+0000 7f21510a4640 1 --2- 172.21.15.114:0/2261177185 >> [v2:172.21.15.114:6800/465333808,v1:172.21.15.114:6801/465333808] conn(0x7f21380a3bb0 0x7f21380a6070 secure :-1 s=READY pgs=158 cs=0 l=1 rev1=1 crypto rx=0x7f2148005fd0 tx=0x7f2148005850 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T10:24:39.723 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:24:39.724+0000 7f21427fc640 1 -- 172.21.15.114:0/2261177185 <== mon.0 v2:172.21.15.114:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f213c0bd8c0 con 0x7f214c105360 2024-04-23T10:24:39.801 INFO:journalctl@ceph.mon.a.smithi114.stdout:Apr 23 10:24:39 smithi114 ceph-mon[29255]: from='mgr.14152 172.21.15.114:0/1529706138' entity='mgr.x' 2024-04-23T10:24:39.801 INFO:journalctl@ceph.mon.a.smithi114.stdout:Apr 23 10:24:39 smithi114 ceph-mon[29255]: from='mgr.14152 172.21.15.114:0/1529706138' entity='mgr.x' 2024-04-23T10:24:39.801 INFO:journalctl@ceph.mon.a.smithi114.stdout:Apr 23 10:24:39 smithi114 ceph-mon[29255]: from='mgr.14152 172.21.15.114:0/1529706138' entity='mgr.x' 2024-04-23T10:24:39.801 INFO:journalctl@ceph.mon.a.smithi114.stdout:Apr 23 10:24:39 smithi114 ceph-mon[29255]: from='mgr.14152 172.21.15.114:0/1529706138' entity='mgr.x' 2024-04-23T10:24:39.801 INFO:journalctl@ceph.mon.a.smithi114.stdout:Apr 23 10:24:39 smithi114 ceph-mon[29255]: from='mgr.14152 172.21.15.114:0/1529706138' entity='mgr.x' 2024-04-23T10:24:39.801 INFO:journalctl@ceph.mon.a.smithi114.stdout:Apr 23 10:24:39 smithi114 ceph-mon[29255]: from='mgr.14152 172.21.15.114:0/1529706138' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-23T10:24:39.801 INFO:journalctl@ceph.mon.a.smithi114.stdout:Apr 23 10:24:39 smithi114 ceph-mon[29255]: mds.? [v2:172.21.15.119:6838/937297340,v1:172.21.15.119:6839/937297340] up:boot 2024-04-23T10:24:39.801 INFO:journalctl@ceph.mon.a.smithi114.stdout:Apr 23 10:24:39 smithi114 ceph-mon[29255]: fsmap 12 up:standby 2024-04-23T10:24:39.801 INFO:journalctl@ceph.mon.a.smithi114.stdout:Apr 23 10:24:39 smithi114 ceph-mon[29255]: from='mgr.14152 172.21.15.114:0/1529706138' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "b"}]: dispatch 2024-04-23T10:24:39.828 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 23 10:24:39 smithi204 ceph-mon[32649]: from='mgr.14152 172.21.15.114:0/1529706138' entity='mgr.x' 2024-04-23T10:24:39.828 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 23 10:24:39 smithi204 ceph-mon[32649]: from='mgr.14152 172.21.15.114:0/1529706138' entity='mgr.x' 2024-04-23T10:24:39.828 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 23 10:24:39 smithi204 ceph-mon[32649]: from='mgr.14152 172.21.15.114:0/1529706138' entity='mgr.x' 2024-04-23T10:24:39.828 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 23 10:24:39 smithi204 ceph-mon[32649]: from='mgr.14152 172.21.15.114:0/1529706138' entity='mgr.x' 2024-04-23T10:24:39.828 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 23 10:24:39 smithi204 ceph-mon[32649]: from='mgr.14152 172.21.15.114:0/1529706138' entity='mgr.x' 2024-04-23T10:24:39.828 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 23 10:24:39 smithi204 ceph-mon[32649]: from='mgr.14152 172.21.15.114:0/1529706138' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-23T10:24:39.828 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 23 10:24:39 smithi204 ceph-mon[32649]: mds.? [v2:172.21.15.119:6838/937297340,v1:172.21.15.119:6839/937297340] up:boot 2024-04-23T10:24:39.828 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 23 10:24:39 smithi204 ceph-mon[32649]: fsmap 12 up:standby 2024-04-23T10:24:39.828 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 23 10:24:39 smithi204 ceph-mon[32649]: from='mgr.14152 172.21.15.114:0/1529706138' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "b"}]: dispatch 2024-04-23T10:24:39.941 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:24:39.942+0000 7f2153308640 1 -- 172.21.15.114:0/2261177185 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f214c02d5c0 con 0x7f214c105360 2024-04-23T10:24:39.942 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:24:39.943+0000 7f21427fc640 1 -- 172.21.15.114:0/2261177185 <== mon.0 v2:172.21.15.114:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 13 v13) ==== 94+0+10497 (secure 0 0 0) 0x7f213c0bd1c0 con 0x7f214c105360 2024-04-23T10:24:39.942 INFO:teuthology.orchestra.run.smithi114.stdout: 2024-04-23T10:24:39.942 INFO:teuthology.orchestra.run.smithi114.stdout:{"epoch":13,"default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14547,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.114:6835/663493580","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6834","nonce":663493580},{"type":"v1","addr":"172.21.15.114:6835","nonce":663493580}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14565,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.114:6837/1439252154","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6836","nonce":1439252154},{"type":"v1","addr":"172.21.15.114:6837","nonce":1439252154}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5},{"gid":14634,"name":"g","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.114:6839/3849796939","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6838","nonce":3849796939},{"type":"v1","addr":"172.21.15.114:6839","nonce":3849796939}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":11},{"gid":14646,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.114:6841/3544523891","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6840","nonce":3544523891},{"type":"v1","addr":"172.21.15.114:6841","nonce":3544523891}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":12},{"gid":24385,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.204:6833/3866561754","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6832","nonce":3866561754},{"type":"v1","addr":"172.21.15.204:6833","nonce":3866561754}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2},{"gid":24427,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.204:6835/2533793172","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6834","nonce":2533793172},{"type":"v1","addr":"172.21.15.204:6835","nonce":2533793172}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":7},{"gid":24439,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.204:6837/513822179","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6836","nonce":513822179},{"type":"v1","addr":"172.21.15.204:6837","nonce":513822179}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":8},{"gid":24443,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.119:6833/783244660","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6832","nonce":783244660},{"type":"v1","addr":"172.21.15.119:6833","nonce":783244660}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":24463,"name":"i","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.204:6839/3222658893","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6838","nonce":3222658893},{"type":"v1","addr":"172.21.15.204:6839","nonce":3222658893}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":10},{"gid":24479,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.119:6835/2033916709","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6834","nonce":2033916709},{"type":"v1","addr":"172.21.15.119:6835","nonce":2033916709}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":6},{"gid":24509,"name":"h","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.119:6837/1872363128","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6836","nonce":1872363128},{"type":"v1","addr":"172.21.15.119:6837","nonce":1872363128}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":9},{"gid":24545,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.119:6839/937297340","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6838","nonce":937297340},{"type":"v1","addr":"172.21.15.119:6839","nonce":937297340}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":13}],"filesystems":[]} 2024-04-23T10:24:39.942 INFO:teuthology.orchestra.run.smithi114.stderr:dumped fsmap epoch 13 2024-04-23T10:24:39.944 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:24:39.945+0000 7f2153308640 1 -- 172.21.15.114:0/2261177185 >> [v2:172.21.15.114:6800/465333808,v1:172.21.15.114:6801/465333808] conn(0x7f21380a3bb0 msgr2=0x7f21380a6070 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T10:24:39.944 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:24:39.945+0000 7f2153308640 1 --2- 172.21.15.114:0/2261177185 >> [v2:172.21.15.114:6800/465333808,v1:172.21.15.114:6801/465333808] conn(0x7f21380a3bb0 0x7f21380a6070 secure :-1 s=READY pgs=158 cs=0 l=1 rev1=1 crypto rx=0x7f2148005fd0 tx=0x7f2148005850 comp rx=0 tx=0).stop 2024-04-23T10:24:39.944 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:24:39.945+0000 7f2153308640 1 -- 172.21.15.114:0/2261177185 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f214c105360 msgr2=0x7f214c1009e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T10:24:39.944 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:24:39.945+0000 7f2153308640 1 --2- 172.21.15.114:0/2261177185 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f214c105360 0x7f214c1009e0 secure :-1 s=READY pgs=198 cs=0 l=1 rev1=1 crypto rx=0x7f213c006010 tx=0x7f213c0042f0 comp rx=0 tx=0).stop 2024-04-23T10:24:39.944 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:24:39.945+0000 7f2153308640 1 -- 172.21.15.114:0/2261177185 shutdown_connections 2024-04-23T10:24:39.945 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:24:39.945+0000 7f2153308640 1 --2- 172.21.15.114:0/2261177185 >> [v2:172.21.15.114:6800/465333808,v1:172.21.15.114:6801/465333808] conn(0x7f21380a3bb0 0x7f21380a6070 unknown :-1 s=CLOSED pgs=158 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:24:39.945 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:24:39.945+0000 7f2153308640 1 --2- 172.21.15.114:0/2261177185 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f214c107c90 0x7f214c100f20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:24:39.945 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:24:39.945+0000 7f2153308640 1 --2- 172.21.15.114:0/2261177185 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f214c105360 0x7f214c1009e0 unknown :-1 s=CLOSED pgs=198 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:24:39.945 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:24:39.945+0000 7f2153308640 1 --2- 172.21.15.114:0/2261177185 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f214c102a30 0x7f214c1023a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:24:39.945 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:24:39.945+0000 7f2153308640 1 -- 172.21.15.114:0/2261177185 >> 172.21.15.114:0/2261177185 conn(0x7f214c0fc440 msgr2=0x7f214c103470 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T10:24:39.945 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:24:39.945+0000 7f2153308640 1 -- 172.21.15.114:0/2261177185 shutdown_connections 2024-04-23T10:24:39.945 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:24:39.945+0000 7f2153308640 1 -- 172.21.15.114:0/2261177185 wait complete. 2024-04-23T10:24:39.955 DEBUG:teuthology.orchestra.run.smithi114:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-04-23T10:24:39.992 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 23 10:24:39 smithi119 ceph-mon[32608]: from='mgr.14152 172.21.15.114:0/1529706138' entity='mgr.x' 2024-04-23T10:24:39.992 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 23 10:24:39 smithi119 ceph-mon[32608]: from='mgr.14152 172.21.15.114:0/1529706138' entity='mgr.x' 2024-04-23T10:24:39.992 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 23 10:24:39 smithi119 ceph-mon[32608]: from='mgr.14152 172.21.15.114:0/1529706138' entity='mgr.x' 2024-04-23T10:24:39.992 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 23 10:24:39 smithi119 ceph-mon[32608]: from='mgr.14152 172.21.15.114:0/1529706138' entity='mgr.x' 2024-04-23T10:24:39.992 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 23 10:24:39 smithi119 ceph-mon[32608]: from='mgr.14152 172.21.15.114:0/1529706138' entity='mgr.x' 2024-04-23T10:24:39.992 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 23 10:24:39 smithi119 ceph-mon[32608]: from='mgr.14152 172.21.15.114:0/1529706138' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-23T10:24:39.992 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 23 10:24:39 smithi119 ceph-mon[32608]: mds.? [v2:172.21.15.119:6838/937297340,v1:172.21.15.119:6839/937297340] up:boot 2024-04-23T10:24:39.992 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 23 10:24:39 smithi119 ceph-mon[32608]: fsmap 12 up:standby 2024-04-23T10:24:39.992 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 23 10:24:39 smithi119 ceph-mon[32608]: from='mgr.14152 172.21.15.114:0/1529706138' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "b"}]: dispatch 2024-04-23T10:24:40.055 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:24:40.056+0000 7fa68a3f7640 1 -- 172.21.15.114:0/746295215 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fa6841031b0 msgr2=0x7fa684103630 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T10:24:40.056 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:24:40.056+0000 7fa68a3f7640 1 --2- 172.21.15.114:0/746295215 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fa6841031b0 0x7fa684103630 secure :-1 s=READY pgs=199 cs=0 l=1 rev1=1 crypto rx=0x7fa674009900 tx=0x7fa67402a410 comp rx=0 tx=0).stop 2024-04-23T10:24:40.056 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:24:40.056+0000 7fa68a3f7640 1 -- 172.21.15.114:0/746295215 shutdown_connections 2024-04-23T10:24:40.056 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:24:40.056+0000 7fa68a3f7640 1 --2- 172.21.15.114:0/746295215 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fa684103b70 0x7fa68410a400 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:24:40.056 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:24:40.056+0000 7fa68a3f7640 1 --2- 172.21.15.114:0/746295215 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fa6841031b0 0x7fa684103630 unknown :-1 s=CLOSED pgs=199 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:24:40.056 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:24:40.056+0000 7fa68a3f7640 1 --2- 172.21.15.114:0/746295215 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fa684101fc0 0x7fa6841023e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:24:40.056 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:24:40.056+0000 7fa68a3f7640 1 -- 172.21.15.114:0/746295215 >> 172.21.15.114:0/746295215 conn(0x7fa6840fd550 msgr2=0x7fa6840ff970 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T10:24:40.056 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:24:40.057+0000 7fa68a3f7640 1 -- 172.21.15.114:0/746295215 shutdown_connections 2024-04-23T10:24:40.056 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:24:40.057+0000 7fa68a3f7640 1 -- 172.21.15.114:0/746295215 wait complete. 2024-04-23T10:24:40.058 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:24:40.057+0000 7fa68a3f7640 1 Processor -- start 2024-04-23T10:24:40.058 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:24:40.057+0000 7fa68a3f7640 1 -- start start 2024-04-23T10:24:40.058 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:24:40.058+0000 7fa68a3f7640 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fa684101fc0 0x7fa6841981d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T10:24:40.058 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:24:40.058+0000 7fa68a3f7640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fa6841031b0 0x7fa684198710 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T10:24:40.058 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:24:40.058+0000 7fa68a3f7640 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fa684103b70 0x7fa68419f780 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T10:24:40.059 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:24:40.058+0000 7fa68a3f7640 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x7fa68410db00 con 0x7fa6841031b0 2024-04-23T10:24:40.059 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:24:40.058+0000 7fa68a3f7640 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 -- 0x7fa68410d980 con 0x7fa684101fc0 2024-04-23T10:24:40.059 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:24:40.058+0000 7fa68a3f7640 1 -- --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_getmap magic: 0 -- 0x7fa68410dc80 con 0x7fa684103b70 2024-04-23T10:24:40.059 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:24:40.058+0000 7fa6837fe640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fa6841031b0 0x7fa684198710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T10:24:40.059 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:24:40.058+0000 7fa6837fe640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fa6841031b0 0x7fa684198710 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.114:44408/0 (socket says 172.21.15.114:44408) 2024-04-23T10:24:40.059 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:24:40.058+0000 7fa6837fe640 1 -- 172.21.15.114:0/3397831736 learned_addr learned my addr 172.21.15.114:0/3397831736 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-04-23T10:24:40.059 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:24:40.058+0000 7fa683fff640 1 --2- 172.21.15.114:0/3397831736 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fa684101fc0 0x7fa6841981d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T10:24:40.059 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:24:40.058+0000 7fa688994640 1 --2- 172.21.15.114:0/3397831736 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fa684103b70 0x7fa68419f780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T10:24:40.059 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:24:40.058+0000 7fa683fff640 1 -- 172.21.15.114:0/3397831736 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fa684103b70 msgr2=0x7fa68419f780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T10:24:40.059 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:24:40.058+0000 7fa683fff640 1 --2- 172.21.15.114:0/3397831736 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fa684103b70 0x7fa68419f780 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:24:40.059 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:24:40.058+0000 7fa683fff640 1 -- 172.21.15.114:0/3397831736 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fa6841031b0 msgr2=0x7fa684198710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T10:24:40.059 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:24:40.058+0000 7fa683fff640 1 --2- 172.21.15.114:0/3397831736 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fa6841031b0 0x7fa684198710 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:24:40.059 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:24:40.058+0000 7fa683fff640 1 -- 172.21.15.114:0/3397831736 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa68419fdf0 con 0x7fa684101fc0 2024-04-23T10:24:40.059 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:24:40.058+0000 7fa6837fe640 1 --2- 172.21.15.114:0/3397831736 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fa6841031b0 0x7fa684198710 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-23T10:24:40.059 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:24:40.058+0000 7fa683fff640 1 --2- 172.21.15.114:0/3397831736 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fa684101fc0 0x7fa6841981d0 secure :-1 s=READY pgs=52 cs=0 l=1 rev1=1 crypto rx=0x7fa67800e9c0 tx=0x7fa67800ee80 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T10:24:40.060 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:24:40.058+0000 7fa6817fa640 1 -- 172.21.15.114:0/3397831736 <== mon.2 v2:172.21.15.119:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa67800ccc0 con 0x7fa684101fc0 2024-04-23T10:24:40.060 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:24:40.059+0000 7fa6817fa640 1 -- 172.21.15.114:0/3397831736 <== mon.2 v2:172.21.15.119:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fa6780035f0 con 0x7fa684101fc0 2024-04-23T10:24:40.060 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:24:40.059+0000 7fa68a3f7640 1 -- 172.21.15.114:0/3397831736 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fa6841a00e0 con 0x7fa684101fc0 2024-04-23T10:24:40.060 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:24:40.059+0000 7fa6817fa640 1 -- 172.21.15.114:0/3397831736 <== mon.2 v2:172.21.15.119:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa67800ce60 con 0x7fa684101fc0 2024-04-23T10:24:40.060 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:24:40.059+0000 7fa68a3f7640 1 -- 172.21.15.114:0/3397831736 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fa6841a0600 con 0x7fa684101fc0 2024-04-23T10:24:40.061 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:24:40.060+0000 7fa68a3f7640 1 -- 172.21.15.114:0/3397831736 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fa6840684e0 con 0x7fa684101fc0 2024-04-23T10:24:40.061 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:24:40.061+0000 7fa6817fa640 1 -- 172.21.15.114:0/3397831736 <== mon.2 v2:172.21.15.119:3300/0 4 ==== mgrmap(e 16) ==== 138647+0+0 (secure 0 0 0) 0x7fa678004000 con 0x7fa684101fc0 2024-04-23T10:24:40.061 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:24:40.061+0000 7fa6817fa640 1 --2- 172.21.15.114:0/3397831736 >> [v2:172.21.15.114:6800/465333808,v1:172.21.15.114:6801/465333808] conn(0x7fa66c0a3bb0 0x7fa66c0a6070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T10:24:40.061 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:24:40.061+0000 7fa6837fe640 1 --2- 172.21.15.114:0/3397831736 >> [v2:172.21.15.114:6800/465333808,v1:172.21.15.114:6801/465333808] conn(0x7fa66c0a3bb0 0x7fa66c0a6070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T10:24:40.061 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:24:40.061+0000 7fa6817fa640 1 -- 172.21.15.114:0/3397831736 <== mon.2 v2:172.21.15.119:3300/0 5 ==== osd_map(83..83 src has 1..83) ==== 7127+0+0 (secure 0 0 0) 0x7fa6780d0400 con 0x7fa684101fc0 2024-04-23T10:24:40.061 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:24:40.062+0000 7fa6837fe640 1 --2- 172.21.15.114:0/3397831736 >> [v2:172.21.15.114:6800/465333808,v1:172.21.15.114:6801/465333808] conn(0x7fa66c0a3bb0 0x7fa66c0a6070 secure :-1 s=READY pgs=159 cs=0 l=1 rev1=1 crypto rx=0x7fa674009230 tx=0x7fa6740058d0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T10:24:40.065 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:24:40.066+0000 7fa6817fa640 1 -- 172.21.15.114:0/3397831736 <== mon.2 v2:172.21.15.119:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7fa67809d1e0 con 0x7fa684101fc0 2024-04-23T10:24:40.265 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:24:40.266+0000 7fa68a3f7640 1 -- 172.21.15.114:0/3397831736 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7fa684062b50 con 0x7fa684101fc0 2024-04-23T10:24:40.266 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:24:40.267+0000 7fa6817fa640 1 -- 172.21.15.114:0/3397831736 <== mon.2 v2:172.21.15.119:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v83) ==== 74+0+18528 (secure 0 0 0) 0x7fa67809cae0 con 0x7fa684101fc0 2024-04-23T10:24:40.266 INFO:teuthology.orchestra.run.smithi114.stdout: 2024-04-23T10:24:40.266 INFO:teuthology.orchestra.run.smithi114.stdout:{"epoch":83,"fsid":"aa240572-0159-11ef-bc93-c7b262605968","created":"2024-04-23T10:16:53.351136+0000","modified":"2024-04-23T10:23:55.163231+0000","last_up_change":"2024-04-23T10:23:54.153269+0000","last_in_change":"2024-04-23T10:23:32.963085+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":26,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":1,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-04-23T10:20:11.970297+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":12,"score_stable":12,"optimal_score":0.25,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"fd64adde-9445-45a4-a171-ef656e2c5709","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6802","nonce":736366878},{"type":"v1","addr":"172.21.15.114:6803","nonce":736366878}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6804","nonce":736366878},{"type":"v1","addr":"172.21.15.114:6805","nonce":736366878}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6808","nonce":736366878},{"type":"v1","addr":"172.21.15.114:6809","nonce":736366878}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6806","nonce":736366878},{"type":"v1","addr":"172.21.15.114:6807","nonce":736366878}]},"public_addr":"172.21.15.114:6803/736366878","cluster_addr":"172.21.15.114:6805/736366878","heartbeat_back_addr":"172.21.15.114:6809/736366878","heartbeat_front_addr":"172.21.15.114:6807/736366878","state":["exists","up"]},{"osd":1,"uuid":"e279a1e6-2702-472a-930a-e08ece76a861","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":42,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6800","nonce":1133540779},{"type":"v1","addr":"172.21.15.119:6801","nonce":1133540779}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6802","nonce":1133540779},{"type":"v1","addr":"172.21.15.119:6803","nonce":1133540779}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6806","nonce":1133540779},{"type":"v1","addr":"172.21.15.119:6807","nonce":1133540779}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6804","nonce":1133540779},{"type":"v1","addr":"172.21.15.119:6805","nonce":1133540779}]},"public_addr":"172.21.15.119:6801/1133540779","cluster_addr":"172.21.15.119:6803/1133540779","heartbeat_back_addr":"172.21.15.119:6807/1133540779","heartbeat_front_addr":"172.21.15.119:6805/1133540779","state":["exists","up"]},{"osd":2,"uuid":"f4c9f2b4-ff49-4e14-89cd-b477feeb9382","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6800","nonce":799652511},{"type":"v1","addr":"172.21.15.204:6801","nonce":799652511}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6802","nonce":799652511},{"type":"v1","addr":"172.21.15.204:6803","nonce":799652511}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6806","nonce":799652511},{"type":"v1","addr":"172.21.15.204:6807","nonce":799652511}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6804","nonce":799652511},{"type":"v1","addr":"172.21.15.204:6805","nonce":799652511}]},"public_addr":"172.21.15.204:6801/799652511","cluster_addr":"172.21.15.204:6803/799652511","heartbeat_back_addr":"172.21.15.204:6807/799652511","heartbeat_front_addr":"172.21.15.204:6805/799652511","state":["exists","up"]},{"osd":3,"uuid":"52d2a566-e71c-4bb3-8d6c-5abdd43177c2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":25,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6810","nonce":4275595091},{"type":"v1","addr":"172.21.15.114:6811","nonce":4275595091}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6812","nonce":4275595091},{"type":"v1","addr":"172.21.15.114:6813","nonce":4275595091}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6816","nonce":4275595091},{"type":"v1","addr":"172.21.15.114:6817","nonce":4275595091}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6814","nonce":4275595091},{"type":"v1","addr":"172.21.15.114:6815","nonce":4275595091}]},"public_addr":"172.21.15.114:6811/4275595091","cluster_addr":"172.21.15.114:6813/4275595091","heartbeat_back_addr":"172.21.15.114:6817/4275595091","heartbeat_front_addr":"172.21.15.114:6815/4275595091","state":["exists","up"]},{"osd":4,"uuid":"d911a7c1-4bdc-4d28-b8a6-73896a6e7de6","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":33,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6808","nonce":1611007172},{"type":"v1","addr":"172.21.15.119:6809","nonce":1611007172}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6810","nonce":1611007172},{"type":"v1","addr":"172.21.15.119:6811","nonce":1611007172}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6814","nonce":1611007172},{"type":"v1","addr":"172.21.15.119:6815","nonce":1611007172}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6812","nonce":1611007172},{"type":"v1","addr":"172.21.15.119:6813","nonce":1611007172}]},"public_addr":"172.21.15.119:6809/1611007172","cluster_addr":"172.21.15.119:6811/1611007172","heartbeat_back_addr":"172.21.15.119:6815/1611007172","heartbeat_front_addr":"172.21.15.119:6813/1611007172","state":["exists","up"]},{"osd":5,"uuid":"f7590e1a-e93c-4c29-b15f-38ee52d1f921","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":39,"up_thru":55,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6808","nonce":1427563312},{"type":"v1","addr":"172.21.15.204:6809","nonce":1427563312}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6810","nonce":1427563312},{"type":"v1","addr":"172.21.15.204:6811","nonce":1427563312}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6814","nonce":1427563312},{"type":"v1","addr":"172.21.15.204:6815","nonce":1427563312}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6812","nonce":1427563312},{"type":"v1","addr":"172.21.15.204:6813","nonce":1427563312}]},"public_addr":"172.21.15.204:6809/1427563312","cluster_addr":"172.21.15.204:6811/1427563312","heartbeat_back_addr":"172.21.15.204:6815/1427563312","heartbeat_front_addr":"172.21.15.204:6813/1427563312","state":["exists","up"]},{"osd":6,"uuid":"534d20e8-3e54-4db8-a845-6119f49c7a78","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":47,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6818","nonce":4072912544},{"type":"v1","addr":"172.21.15.114:6819","nonce":4072912544}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6820","nonce":4072912544},{"type":"v1","addr":"172.21.15.114:6821","nonce":4072912544}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6824","nonce":4072912544},{"type":"v1","addr":"172.21.15.114:6825","nonce":4072912544}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6822","nonce":4072912544},{"type":"v1","addr":"172.21.15.114:6823","nonce":4072912544}]},"public_addr":"172.21.15.114:6819/4072912544","cluster_addr":"172.21.15.114:6821/4072912544","heartbeat_back_addr":"172.21.15.114:6825/4072912544","heartbeat_front_addr":"172.21.15.114:6823/4072912544","state":["exists","up"]},{"osd":7,"uuid":"f0ec22b8-1b48-4859-9d65-b00403aff3ad","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":53,"up_thru":77,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6816","nonce":2237458793},{"type":"v1","addr":"172.21.15.119:6817","nonce":2237458793}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6818","nonce":2237458793},{"type":"v1","addr":"172.21.15.119:6819","nonce":2237458793}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6822","nonce":2237458793},{"type":"v1","addr":"172.21.15.119:6823","nonce":2237458793}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6820","nonce":2237458793},{"type":"v1","addr":"172.21.15.119:6821","nonce":2237458793}]},"public_addr":"172.21.15.119:6817/2237458793","cluster_addr":"172.21.15.119:6819/2237458793","heartbeat_back_addr":"172.21.15.119:6823/2237458793","heartbeat_front_addr":"172.21.15.119:6821/2237458793","state":["exists","up"]},{"osd":8,"uuid":"3ac89686-6d76-4cef-9089-ac29e4b3bcac","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":62,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6816","nonce":1426318231},{"type":"v1","addr":"172.21.15.204:6817","nonce":1426318231}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6818","nonce":1426318231},{"type":"v1","addr":"172.21.15.204:6819","nonce":1426318231}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6822","nonce":1426318231},{"type":"v1","addr":"172.21.15.204:6823","nonce":1426318231}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6820","nonce":1426318231},{"type":"v1","addr":"172.21.15.204:6821","nonce":1426318231}]},"public_addr":"172.21.15.204:6817/1426318231","cluster_addr":"172.21.15.204:6819/1426318231","heartbeat_back_addr":"172.21.15.204:6823/1426318231","heartbeat_front_addr":"172.21.15.204:6821/1426318231","state":["exists","up"]},{"osd":9,"uuid":"a80ffe4d-1e98-460f-b046-c4754285cc5a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":69,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6826","nonce":2750635721},{"type":"v1","addr":"172.21.15.114:6827","nonce":2750635721}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6828","nonce":2750635721},{"type":"v1","addr":"172.21.15.114:6829","nonce":2750635721}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6832","nonce":2750635721},{"type":"v1","addr":"172.21.15.114:6833","nonce":2750635721}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6830","nonce":2750635721},{"type":"v1","addr":"172.21.15.114:6831","nonce":2750635721}]},"public_addr":"172.21.15.114:6827/2750635721","cluster_addr":"172.21.15.114:6829/2750635721","heartbeat_back_addr":"172.21.15.114:6833/2750635721","heartbeat_front_addr":"172.21.15.114:6831/2750635721","state":["exists","up"]},{"osd":10,"uuid":"cd75a579-bf12-4600-ac09-a8bfce772ced","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":74,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6824","nonce":1400187701},{"type":"v1","addr":"172.21.15.119:6825","nonce":1400187701}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6826","nonce":1400187701},{"type":"v1","addr":"172.21.15.119:6827","nonce":1400187701}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6830","nonce":1400187701},{"type":"v1","addr":"172.21.15.119:6831","nonce":1400187701}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6828","nonce":1400187701},{"type":"v1","addr":"172.21.15.119:6829","nonce":1400187701}]},"public_addr":"172.21.15.119:6825/1400187701","cluster_addr":"172.21.15.119:6827/1400187701","heartbeat_back_addr":"172.21.15.119:6831/1400187701","heartbeat_front_addr":"172.21.15.119:6829/1400187701","state":["exists","up"]},{"osd":11,"uuid":"53f037a5-3ec0-483a-80a1-dfee923a68e5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":82,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6824","nonce":2407298490},{"type":"v1","addr":"172.21.15.204:6825","nonce":2407298490}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6826","nonce":2407298490},{"type":"v1","addr":"172.21.15.204:6827","nonce":2407298490}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6830","nonce":2407298490},{"type":"v1","addr":"172.21.15.204:6831","nonce":2407298490}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6828","nonce":2407298490},{"type":"v1","addr":"172.21.15.204:6829","nonce":2407298490}]},"public_addr":"172.21.15.204:6825/2407298490","cluster_addr":"172.21.15.204:6827/2407298490","heartbeat_back_addr":"172.21.15.204:6831/2407298490","heartbeat_front_addr":"172.21.15.204:6829/2407298490","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-23T10:19:18.401147+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-23T10:19:42.611299+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-23T10:20:07.277367+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-23T10:20:31.208368+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-23T10:20:55.866225+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-23T10:21:19.961237+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-23T10:21:44.480783+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-23T10:22:09.173641+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-23T10:22:34.030328+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-23T10:22:59.652293+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-23T10:23:25.659673+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-23T10:23:51.168171+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.114:6801/1604678557":"2024-04-24T10:17:45.874609+0000","172.21.15.114:0/4025385854":"2024-04-24T10:17:45.874609+0000","172.21.15.114:0/2228273469":"2024-04-24T10:17:19.910348+0000","172.21.15.114:6800/2831530838":"2024-04-24T10:17:19.910348+0000","172.21.15.114:0/1333447450":"2024-04-24T10:17:45.874609+0000","172.21.15.114:0/2233026840":"2024-04-24T10:17:19.910348+0000","172.21.15.114:0/1902281545":"2024-04-24T10:17:19.910348+0000","172.21.15.114:6801/2831530838":"2024-04-24T10:17:19.910348+0000","172.21.15.114:6800/1604678557":"2024-04-24T10:17:45.874609+0000","172.21.15.114:0/1623553811":"2024-04-24T10:17:45.874609+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-23T10:24:40.268 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:24:40.269+0000 7fa68a3f7640 1 -- 172.21.15.114:0/3397831736 >> [v2:172.21.15.114:6800/465333808,v1:172.21.15.114:6801/465333808] conn(0x7fa66c0a3bb0 msgr2=0x7fa66c0a6070 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T10:24:40.268 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:24:40.269+0000 7fa68a3f7640 1 --2- 172.21.15.114:0/3397831736 >> [v2:172.21.15.114:6800/465333808,v1:172.21.15.114:6801/465333808] conn(0x7fa66c0a3bb0 0x7fa66c0a6070 secure :-1 s=READY pgs=159 cs=0 l=1 rev1=1 crypto rx=0x7fa674009230 tx=0x7fa6740058d0 comp rx=0 tx=0).stop 2024-04-23T10:24:40.268 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:24:40.269+0000 7fa68a3f7640 1 -- 172.21.15.114:0/3397831736 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fa684101fc0 msgr2=0x7fa6841981d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T10:24:40.268 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:24:40.269+0000 7fa68a3f7640 1 --2- 172.21.15.114:0/3397831736 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fa684101fc0 0x7fa6841981d0 secure :-1 s=READY pgs=52 cs=0 l=1 rev1=1 crypto rx=0x7fa67800e9c0 tx=0x7fa67800ee80 comp rx=0 tx=0).stop 2024-04-23T10:24:40.268 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:24:40.269+0000 7fa68a3f7640 1 -- 172.21.15.114:0/3397831736 shutdown_connections 2024-04-23T10:24:40.268 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:24:40.269+0000 7fa68a3f7640 1 --2- 172.21.15.114:0/3397831736 >> [v2:172.21.15.114:6800/465333808,v1:172.21.15.114:6801/465333808] conn(0x7fa66c0a3bb0 0x7fa66c0a6070 unknown :-1 s=CLOSED pgs=159 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:24:40.268 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:24:40.269+0000 7fa68a3f7640 1 --2- 172.21.15.114:0/3397831736 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fa684103b70 0x7fa68419f780 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:24:40.269 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:24:40.269+0000 7fa68a3f7640 1 --2- 172.21.15.114:0/3397831736 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fa6841031b0 0x7fa684198710 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:24:40.269 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:24:40.269+0000 7fa68a3f7640 1 --2- 172.21.15.114:0/3397831736 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fa684101fc0 0x7fa6841981d0 unknown :-1 s=CLOSED pgs=52 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:24:40.269 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:24:40.269+0000 7fa68a3f7640 1 -- 172.21.15.114:0/3397831736 >> 172.21.15.114:0/3397831736 conn(0x7fa6840fd550 msgr2=0x7fa6841064b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T10:24:40.269 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:24:40.269+0000 7fa68a3f7640 1 -- 172.21.15.114:0/3397831736 shutdown_connections 2024-04-23T10:24:40.269 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:24:40.269+0000 7fa68a3f7640 1 -- 172.21.15.114:0/3397831736 wait complete. 2024-04-23T10:24:40.280 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-04-23T10:20:11.970297+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '22', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 12, 'score_stable': 12, 'optimal_score': 0.25, 'raw_score_acting': 3, 'raw_score_stable': 3, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-04-23T10:24:40.280 DEBUG:teuthology.orchestra.run.smithi114:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-04-23T10:24:40.388 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:24:40.389+0000 7f03a1968640 1 -- 172.21.15.114:0/2541427437 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f039c0749b0 msgr2=0x7f039c074e30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T10:24:40.388 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:24:40.389+0000 7f03a1968640 1 --2- 172.21.15.114:0/2541427437 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f039c0749b0 0x7f039c074e30 secure :-1 s=READY pgs=200 cs=0 l=1 rev1=1 crypto rx=0x7f0394009cd0 tx=0x7f039402a8e0 comp rx=0 tx=0).stop 2024-04-23T10:24:40.388 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:24:40.389+0000 7f03a1968640 1 -- 172.21.15.114:0/2541427437 shutdown_connections 2024-04-23T10:24:40.389 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:24:40.389+0000 7f03a1968640 1 --2- 172.21.15.114:0/2541427437 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f039c10e570 0x7f039c110960 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:24:40.389 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:24:40.389+0000 7f03a1968640 1 --2- 172.21.15.114:0/2541427437 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f039c0749b0 0x7f039c074e30 unknown :-1 s=CLOSED pgs=200 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:24:40.389 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:24:40.389+0000 7f03a1968640 1 --2- 172.21.15.114:0/2541427437 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f039c075f90 0x7f039c0743e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:24:40.389 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:24:40.389+0000 7f03a1968640 1 -- 172.21.15.114:0/2541427437 >> 172.21.15.114:0/2541427437 conn(0x7f039c0fd530 msgr2=0x7f039c0ff950 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T10:24:40.389 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:24:40.389+0000 7f03a1968640 1 -- 172.21.15.114:0/2541427437 shutdown_connections 2024-04-23T10:24:40.389 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:24:40.389+0000 7f03a1968640 1 -- 172.21.15.114:0/2541427437 wait complete. 2024-04-23T10:24:40.389 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:24:40.390+0000 7f03a1968640 1 Processor -- start 2024-04-23T10:24:40.391 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:24:40.390+0000 7f03a1968640 1 -- start start 2024-04-23T10:24:40.391 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:24:40.390+0000 7f03a1968640 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f039c0749b0 0x7f039c19e890 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T10:24:40.391 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:24:40.390+0000 7f03a1968640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f039c075f90 0x7f039c19edd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T10:24:40.391 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:24:40.390+0000 7f03a1968640 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f039c10e570 0x7f039c1a5df0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T10:24:40.391 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:24:40.390+0000 7f03a1968640 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x7f039c1140e0 con 0x7f039c075f90 2024-04-23T10:24:40.391 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:24:40.390+0000 7f03a1968640 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 -- 0x7f039c113f60 con 0x7f039c0749b0 2024-04-23T10:24:40.391 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:24:40.390+0000 7f03a1968640 1 -- --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_getmap magic: 0 -- 0x7f039c114260 con 0x7f039c10e570 2024-04-23T10:24:40.391 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:24:40.390+0000 7f039a7fc640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f039c075f90 0x7f039c19edd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T10:24:40.391 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:24:40.390+0000 7f039affd640 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f039c0749b0 0x7f039c19e890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T10:24:40.392 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:24:40.390+0000 7f039a7fc640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f039c075f90 0x7f039c19edd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.114:44420/0 (socket says 172.21.15.114:44420) 2024-04-23T10:24:40.392 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:24:40.390+0000 7f039a7fc640 1 -- 172.21.15.114:0/3942888588 learned_addr learned my addr 172.21.15.114:0/3942888588 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-04-23T10:24:40.392 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:24:40.390+0000 7f039b7fe640 1 --2- 172.21.15.114:0/3942888588 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f039c10e570 0x7f039c1a5df0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T10:24:40.392 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:24:40.391+0000 7f039a7fc640 1 -- 172.21.15.114:0/3942888588 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f039c10e570 msgr2=0x7f039c1a5df0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T10:24:40.392 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:24:40.391+0000 7f039a7fc640 1 --2- 172.21.15.114:0/3942888588 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f039c10e570 0x7f039c1a5df0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:24:40.392 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:24:40.391+0000 7f039a7fc640 1 -- 172.21.15.114:0/3942888588 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f039c0749b0 msgr2=0x7f039c19e890 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T10:24:40.392 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:24:40.391+0000 7f039a7fc640 1 --2- 172.21.15.114:0/3942888588 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f039c0749b0 0x7f039c19e890 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:24:40.392 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:24:40.391+0000 7f039a7fc640 1 -- 172.21.15.114:0/3942888588 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f039c1a6460 con 0x7f039c075f90 2024-04-23T10:24:40.392 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:24:40.391+0000 7f039affd640 1 --2- 172.21.15.114:0/3942888588 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f039c0749b0 0x7f039c19e890 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-23T10:24:40.392 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:24:40.391+0000 7f039a7fc640 1 --2- 172.21.15.114:0/3942888588 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f039c075f90 0x7f039c19edd0 secure :-1 s=READY pgs=201 cs=0 l=1 rev1=1 crypto rx=0x7f0394005e30 tx=0x7f0394003c20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T10:24:40.392 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:24:40.391+0000 7f03a0966640 1 -- 172.21.15.114:0/3942888588 <== mon.0 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0394004240 con 0x7f039c075f90 2024-04-23T10:24:40.392 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:24:40.391+0000 7f03a1968640 1 -- 172.21.15.114:0/3942888588 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f039c1a66f0 con 0x7f039c075f90 2024-04-23T10:24:40.392 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:24:40.391+0000 7f03a0966640 1 -- 172.21.15.114:0/3942888588 <== mon.0 v2:172.21.15.114:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f0394003e10 con 0x7f039c075f90 2024-04-23T10:24:40.392 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:24:40.391+0000 7f03a0966640 1 -- 172.21.15.114:0/3942888588 <== mon.0 v2:172.21.15.114:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f039403b400 con 0x7f039c075f90 2024-04-23T10:24:40.392 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:24:40.391+0000 7f03a1968640 1 -- 172.21.15.114:0/3942888588 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f039c1a6c30 con 0x7f039c075f90 2024-04-23T10:24:40.393 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:24:40.392+0000 7f03a1968640 1 -- 172.21.15.114:0/3942888588 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f039c104630 con 0x7f039c075f90 2024-04-23T10:24:40.393 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:24:40.393+0000 7f03a0966640 1 -- 172.21.15.114:0/3942888588 <== mon.0 v2:172.21.15.114:3300/0 4 ==== mgrmap(e 16) ==== 138647+0+0 (secure 0 0 0) 0x7f03940306c0 con 0x7f039c075f90 2024-04-23T10:24:40.393 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:24:40.394+0000 7f03a0966640 1 --2- 172.21.15.114:0/3942888588 >> [v2:172.21.15.114:6800/465333808,v1:172.21.15.114:6801/465333808] conn(0x7f03840a3bb0 0x7f03840a6070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T10:24:40.393 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:24:40.394+0000 7f03a0966640 1 -- 172.21.15.114:0/3942888588 <== mon.0 v2:172.21.15.114:3300/0 5 ==== osd_map(83..83 src has 1..83) ==== 7127+0+0 (secure 0 0 0) 0x7f039402e030 con 0x7f039c075f90 2024-04-23T10:24:40.394 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:24:40.394+0000 7f039affd640 1 --2- 172.21.15.114:0/3942888588 >> [v2:172.21.15.114:6800/465333808,v1:172.21.15.114:6801/465333808] conn(0x7f03840a3bb0 0x7f03840a6070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T10:24:40.394 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:24:40.394+0000 7f039affd640 1 --2- 172.21.15.114:0/3942888588 >> [v2:172.21.15.114:6800/465333808,v1:172.21.15.114:6801/465333808] conn(0x7f03840a3bb0 0x7f03840a6070 secure :-1 s=READY pgs=160 cs=0 l=1 rev1=1 crypto rx=0x7f038c00bfd0 tx=0x7f038c00b390 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T10:24:40.396 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:24:40.398+0000 7f03a0966640 1 -- 172.21.15.114:0/3942888588 <== mon.0 v2:172.21.15.114:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f03940be150 con 0x7f039c075f90 2024-04-23T10:24:40.602 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:24:40.603+0000 7f03a1968640 1 -- 172.21.15.114:0/3942888588 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) -- 0x7f039c02d5c0 con 0x7f039c075f90 2024-04-23T10:24:40.603 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:24:40.604+0000 7f03a0966640 1 -- 172.21.15.114:0/3942888588 <== mon.0 v2:172.21.15.114:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v83) ==== 93+0+10 (secure 0 0 0) 0x7f03940bda50 con 0x7f039c075f90 2024-04-23T10:24:40.603 INFO:teuthology.orchestra.run.smithi114.stdout:pg_num: 1 2024-04-23T10:24:40.605 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:24:40.606+0000 7f03a1968640 1 -- 172.21.15.114:0/3942888588 >> [v2:172.21.15.114:6800/465333808,v1:172.21.15.114:6801/465333808] conn(0x7f03840a3bb0 msgr2=0x7f03840a6070 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T10:24:40.605 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:24:40.606+0000 7f03a1968640 1 --2- 172.21.15.114:0/3942888588 >> [v2:172.21.15.114:6800/465333808,v1:172.21.15.114:6801/465333808] conn(0x7f03840a3bb0 0x7f03840a6070 secure :-1 s=READY pgs=160 cs=0 l=1 rev1=1 crypto rx=0x7f038c00bfd0 tx=0x7f038c00b390 comp rx=0 tx=0).stop 2024-04-23T10:24:40.605 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:24:40.606+0000 7f03a1968640 1 -- 172.21.15.114:0/3942888588 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f039c075f90 msgr2=0x7f039c19edd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T10:24:40.605 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:24:40.606+0000 7f03a1968640 1 --2- 172.21.15.114:0/3942888588 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f039c075f90 0x7f039c19edd0 secure :-1 s=READY pgs=201 cs=0 l=1 rev1=1 crypto rx=0x7f0394005e30 tx=0x7f0394003c20 comp rx=0 tx=0).stop 2024-04-23T10:24:40.605 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:24:40.606+0000 7f03a1968640 1 -- 172.21.15.114:0/3942888588 shutdown_connections 2024-04-23T10:24:40.605 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:24:40.606+0000 7f03a1968640 1 --2- 172.21.15.114:0/3942888588 >> [v2:172.21.15.114:6800/465333808,v1:172.21.15.114:6801/465333808] conn(0x7f03840a3bb0 0x7f03840a6070 unknown :-1 s=CLOSED pgs=160 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:24:40.605 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:24:40.606+0000 7f03a1968640 1 --2- 172.21.15.114:0/3942888588 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f039c10e570 0x7f039c1a5df0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:24:40.606 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:24:40.606+0000 7f03a1968640 1 --2- 172.21.15.114:0/3942888588 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f039c075f90 0x7f039c19edd0 unknown :-1 s=CLOSED pgs=201 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:24:40.606 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:24:40.606+0000 7f03a1968640 1 --2- 172.21.15.114:0/3942888588 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f039c0749b0 0x7f039c19e890 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:24:40.606 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:24:40.606+0000 7f03a1968640 1 -- 172.21.15.114:0/3942888588 >> 172.21.15.114:0/3942888588 conn(0x7f039c0fd530 msgr2=0x7f039c101800 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T10:24:40.606 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:24:40.606+0000 7f03a1968640 1 -- 172.21.15.114:0/3942888588 shutdown_connections 2024-04-23T10:24:40.606 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:24:40.606+0000 7f03a1968640 1 -- 172.21.15.114:0/3942888588 wait complete. 2024-04-23T10:24:40.616 DEBUG:teuthology.orchestra.run.smithi114:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph --format=json-pretty osd lspools 2024-04-23T10:24:40.716 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:24:40.715+0000 7fbf8c249640 1 -- 172.21.15.114:0/2545458166 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fbf841029b0 msgr2=0x7fbf841092d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T10:24:40.716 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:24:40.715+0000 7fbf8c249640 1 --2- 172.21.15.114:0/2545458166 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fbf841029b0 0x7fbf841092d0 secure :-1 s=READY pgs=202 cs=0 l=1 rev1=1 crypto rx=0x7fbf7c009ca0 tx=0x7fbf7c02a8c0 comp rx=0 tx=0).stop 2024-04-23T10:24:40.716 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:24:40.715+0000 7fbf8c249640 1 -- 172.21.15.114:0/2545458166 shutdown_connections 2024-04-23T10:24:40.716 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:24:40.715+0000 7fbf8c249640 1 --2- 172.21.15.114:0/2545458166 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fbf841029b0 0x7fbf841092d0 unknown :-1 s=CLOSED pgs=202 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:24:40.716 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:24:40.715+0000 7fbf8c249640 1 --2- 172.21.15.114:0/2545458166 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fbf84101ff0 0x7fbf84102470 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:24:40.716 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:24:40.715+0000 7fbf8c249640 1 --2- 172.21.15.114:0/2545458166 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fbf84100ea0 0x7fbf841012c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:24:40.717 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:24:40.715+0000 7fbf8c249640 1 -- 172.21.15.114:0/2545458166 >> 172.21.15.114:0/2545458166 conn(0x7fbf840fc430 msgr2=0x7fbf840fe850 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T10:24:40.717 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:24:40.715+0000 7fbf8c249640 1 -- 172.21.15.114:0/2545458166 shutdown_connections 2024-04-23T10:24:40.717 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:24:40.715+0000 7fbf8c249640 1 -- 172.21.15.114:0/2545458166 wait complete. 2024-04-23T10:24:40.717 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:24:40.716+0000 7fbf8c249640 1 Processor -- start 2024-04-23T10:24:40.717 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:24:40.716+0000 7fbf8c249640 1 -- start start 2024-04-23T10:24:40.717 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:24:40.716+0000 7fbf8c249640 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fbf84100ea0 0x7fbf841970b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T10:24:40.717 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:24:40.716+0000 7fbf8c249640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fbf84101ff0 0x7fbf841975f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T10:24:40.717 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:24:40.716+0000 7fbf8c249640 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fbf841029b0 0x7fbf8419e660 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T10:24:40.717 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:24:40.716+0000 7fbf8c249640 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x7fbf8410ca00 con 0x7fbf84101ff0 2024-04-23T10:24:40.717 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:24:40.716+0000 7fbf8c249640 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 -- 0x7fbf8410c880 con 0x7fbf841029b0 2024-04-23T10:24:40.717 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:24:40.716+0000 7fbf8c249640 1 -- --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_getmap magic: 0 -- 0x7fbf8410cb80 con 0x7fbf84100ea0 2024-04-23T10:24:40.717 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:24:40.716+0000 7fbf897e4640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fbf84101ff0 0x7fbf841975f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T10:24:40.717 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:24:40.716+0000 7fbf897e4640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fbf84101ff0 0x7fbf841975f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.114:44438/0 (socket says 172.21.15.114:44438) 2024-04-23T10:24:40.717 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:24:40.716+0000 7fbf897e4640 1 -- 172.21.15.114:0/784199017 learned_addr learned my addr 172.21.15.114:0/784199017 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-04-23T10:24:40.717 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:24:40.716+0000 7fbf89fe5640 1 --2- 172.21.15.114:0/784199017 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fbf84100ea0 0x7fbf841970b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T10:24:40.717 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:24:40.716+0000 7fbf8a7e6640 1 --2- 172.21.15.114:0/784199017 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fbf841029b0 0x7fbf8419e660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T10:24:40.718 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:24:40.716+0000 7fbf897e4640 1 -- 172.21.15.114:0/784199017 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fbf84100ea0 msgr2=0x7fbf841970b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T10:24:40.718 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:24:40.716+0000 7fbf897e4640 1 --2- 172.21.15.114:0/784199017 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fbf84100ea0 0x7fbf841970b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:24:40.718 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:24:40.716+0000 7fbf897e4640 1 -- 172.21.15.114:0/784199017 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fbf841029b0 msgr2=0x7fbf8419e660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T10:24:40.718 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:24:40.716+0000 7fbf897e4640 1 --2- 172.21.15.114:0/784199017 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fbf841029b0 0x7fbf8419e660 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:24:40.718 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:24:40.716+0000 7fbf897e4640 1 -- 172.21.15.114:0/784199017 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fbf8419ecd0 con 0x7fbf84101ff0 2024-04-23T10:24:40.718 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:24:40.716+0000 7fbf897e4640 1 --2- 172.21.15.114:0/784199017 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fbf84101ff0 0x7fbf841975f0 secure :-1 s=READY pgs=203 cs=0 l=1 rev1=1 crypto rx=0x7fbf7400d950 tx=0x7fbf7400de10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T10:24:40.718 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:24:40.716+0000 7fbf7affd640 1 -- 172.21.15.114:0/784199017 <== mon.0 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbf74017020 con 0x7fbf84101ff0 2024-04-23T10:24:40.718 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:24:40.716+0000 7fbf7affd640 1 -- 172.21.15.114:0/784199017 <== mon.0 v2:172.21.15.114:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fbf74009c30 con 0x7fbf84101ff0 2024-04-23T10:24:40.718 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:24:40.717+0000 7fbf8c249640 1 -- 172.21.15.114:0/784199017 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fbf8419efc0 con 0x7fbf84101ff0 2024-04-23T10:24:40.718 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:24:40.717+0000 7fbf7affd640 1 -- 172.21.15.114:0/784199017 <== mon.0 v2:172.21.15.114:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbf7400f6b0 con 0x7fbf84101ff0 2024-04-23T10:24:40.718 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:24:40.717+0000 7fbf8c249640 1 -- 172.21.15.114:0/784199017 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fbf8419f480 con 0x7fbf84101ff0 2024-04-23T10:24:40.719 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:24:40.718+0000 7fbf8c249640 1 -- 172.21.15.114:0/784199017 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fbf84108280 con 0x7fbf84101ff0 2024-04-23T10:24:40.719 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:24:40.718+0000 7fbf7affd640 1 -- 172.21.15.114:0/784199017 <== mon.0 v2:172.21.15.114:3300/0 4 ==== mgrmap(e 16) ==== 138647+0+0 (secure 0 0 0) 0x7fbf7401a020 con 0x7fbf84101ff0 2024-04-23T10:24:40.719 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:24:40.719+0000 7fbf7affd640 1 --2- 172.21.15.114:0/784199017 >> [v2:172.21.15.114:6800/465333808,v1:172.21.15.114:6801/465333808] conn(0x7fbf700a3bb0 0x7fbf700a6070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T10:24:40.719 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:24:40.719+0000 7fbf89fe5640 1 --2- 172.21.15.114:0/784199017 >> [v2:172.21.15.114:6800/465333808,v1:172.21.15.114:6801/465333808] conn(0x7fbf700a3bb0 0x7fbf700a6070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T10:24:40.720 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:24:40.719+0000 7fbf7affd640 1 -- 172.21.15.114:0/784199017 <== mon.0 v2:172.21.15.114:3300/0 5 ==== osd_map(83..83 src has 1..83) ==== 7127+0+0 (secure 0 0 0) 0x7fbf740cfac0 con 0x7fbf84101ff0 2024-04-23T10:24:40.720 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:24:40.719+0000 7fbf89fe5640 1 --2- 172.21.15.114:0/784199017 >> [v2:172.21.15.114:6800/465333808,v1:172.21.15.114:6801/465333808] conn(0x7fbf700a3bb0 0x7fbf700a6070 secure :-1 s=READY pgs=161 cs=0 l=1 rev1=1 crypto rx=0x7fbf80009bd0 tx=0x7fbf80005f90 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T10:24:40.722 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:24:40.724+0000 7fbf7affd640 1 -- 172.21.15.114:0/784199017 <== mon.0 v2:172.21.15.114:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7fbf74009de0 con 0x7fbf84101ff0 2024-04-23T10:24:40.828 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 23 10:24:40 smithi204 ceph-mon[32649]: from='client.? 172.21.15.114:0/2261177185' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-23T10:24:40.828 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 23 10:24:40 smithi204 ceph-mon[32649]: pgmap v280: 1 pgs: 1 active+clean; 577 KiB data, 365 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-23T10:24:40.828 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 23 10:24:40 smithi204 ceph-mon[32649]: from='client.? 172.21.15.114:0/3397831736' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-23T10:24:40.828 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 23 10:24:40 smithi204 ceph-mon[32649]: from='mgr.14152 172.21.15.114:0/1529706138' entity='mgr.x' 2024-04-23T10:24:40.828 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 23 10:24:40 smithi204 ceph-mon[32649]: from='mgr.14152 172.21.15.114:0/1529706138' entity='mgr.x' 2024-04-23T10:24:40.828 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 23 10:24:40 smithi204 ceph-mon[32649]: from='mgr.14152 172.21.15.114:0/1529706138' entity='mgr.x' 2024-04-23T10:24:40.828 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 23 10:24:40 smithi204 ceph-mon[32649]: from='mgr.14152 172.21.15.114:0/1529706138' entity='mgr.x' 2024-04-23T10:24:40.828 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 23 10:24:40 smithi204 ceph-mon[32649]: from='mgr.14152 172.21.15.114:0/1529706138' entity='mgr.x' 2024-04-23T10:24:40.828 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 23 10:24:40 smithi204 ceph-mon[32649]: from='mgr.14152 172.21.15.114:0/1529706138' entity='mgr.x' 2024-04-23T10:24:40.900 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:24:40.901+0000 7fbf8c249640 1 -- 172.21.15.114:0/784199017 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_command({"prefix": "osd lspools", "format": "json-pretty"} v 0) -- 0x7fbf8402d770 con 0x7fbf84101ff0 2024-04-23T10:24:40.901 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:24:40.902+0000 7fbf7affd640 1 -- 172.21.15.114:0/784199017 <== mon.0 v2:172.21.15.114:3300/0 7 ==== mon_command_ack([{"prefix": "osd lspools", "format": "json-pretty"}]=0 v83) ==== 84+0+65 (secure 0 0 0) 0x7fbf7409c940 con 0x7fbf84101ff0 2024-04-23T10:24:40.901 INFO:teuthology.orchestra.run.smithi114.stdout: 2024-04-23T10:24:40.901 INFO:teuthology.orchestra.run.smithi114.stdout:[ 2024-04-23T10:24:40.901 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-04-23T10:24:40.901 INFO:teuthology.orchestra.run.smithi114.stdout: "poolnum": 1, 2024-04-23T10:24:40.901 INFO:teuthology.orchestra.run.smithi114.stdout: "poolname": ".mgr" 2024-04-23T10:24:40.901 INFO:teuthology.orchestra.run.smithi114.stdout: } 2024-04-23T10:24:40.901 INFO:teuthology.orchestra.run.smithi114.stdout:] 2024-04-23T10:24:40.903 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:24:40.904+0000 7fbf8c249640 1 -- 172.21.15.114:0/784199017 >> [v2:172.21.15.114:6800/465333808,v1:172.21.15.114:6801/465333808] conn(0x7fbf700a3bb0 msgr2=0x7fbf700a6070 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T10:24:40.903 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:24:40.904+0000 7fbf8c249640 1 --2- 172.21.15.114:0/784199017 >> [v2:172.21.15.114:6800/465333808,v1:172.21.15.114:6801/465333808] conn(0x7fbf700a3bb0 0x7fbf700a6070 secure :-1 s=READY pgs=161 cs=0 l=1 rev1=1 crypto rx=0x7fbf80009bd0 tx=0x7fbf80005f90 comp rx=0 tx=0).stop 2024-04-23T10:24:40.903 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:24:40.904+0000 7fbf8c249640 1 -- 172.21.15.114:0/784199017 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fbf84101ff0 msgr2=0x7fbf841975f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T10:24:40.903 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:24:40.904+0000 7fbf8c249640 1 --2- 172.21.15.114:0/784199017 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fbf84101ff0 0x7fbf841975f0 secure :-1 s=READY pgs=203 cs=0 l=1 rev1=1 crypto rx=0x7fbf7400d950 tx=0x7fbf7400de10 comp rx=0 tx=0).stop 2024-04-23T10:24:40.904 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:24:40.904+0000 7fbf8c249640 1 -- 172.21.15.114:0/784199017 shutdown_connections 2024-04-23T10:24:40.904 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:24:40.904+0000 7fbf8c249640 1 --2- 172.21.15.114:0/784199017 >> [v2:172.21.15.114:6800/465333808,v1:172.21.15.114:6801/465333808] conn(0x7fbf700a3bb0 0x7fbf700a6070 unknown :-1 s=CLOSED pgs=161 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:24:40.904 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:24:40.904+0000 7fbf8c249640 1 --2- 172.21.15.114:0/784199017 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fbf841029b0 0x7fbf8419e660 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:24:40.904 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:24:40.904+0000 7fbf8c249640 1 --2- 172.21.15.114:0/784199017 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fbf84101ff0 0x7fbf841975f0 unknown :-1 s=CLOSED pgs=203 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:24:40.904 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:24:40.904+0000 7fbf8c249640 1 --2- 172.21.15.114:0/784199017 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fbf84100ea0 0x7fbf841970b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:24:40.904 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:24:40.904+0000 7fbf8c249640 1 -- 172.21.15.114:0/784199017 >> 172.21.15.114:0/784199017 conn(0x7fbf840fc430 msgr2=0x7fbf84105450 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T10:24:40.904 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:24:40.904+0000 7fbf8c249640 1 -- 172.21.15.114:0/784199017 shutdown_connections 2024-04-23T10:24:40.904 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:24:40.904+0000 7fbf8c249640 1 -- 172.21.15.114:0/784199017 wait complete. 2024-04-23T10:24:40.913 INFO:journalctl@ceph.mon.a.smithi114.stdout:Apr 23 10:24:40 smithi114 ceph-mon[29255]: from='client.? 172.21.15.114:0/2261177185' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-23T10:24:40.913 INFO:journalctl@ceph.mon.a.smithi114.stdout:Apr 23 10:24:40 smithi114 ceph-mon[29255]: pgmap v280: 1 pgs: 1 active+clean; 577 KiB data, 365 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-23T10:24:40.913 INFO:journalctl@ceph.mon.a.smithi114.stdout:Apr 23 10:24:40 smithi114 ceph-mon[29255]: from='client.? 172.21.15.114:0/3397831736' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-23T10:24:40.913 INFO:journalctl@ceph.mon.a.smithi114.stdout:Apr 23 10:24:40 smithi114 ceph-mon[29255]: from='mgr.14152 172.21.15.114:0/1529706138' entity='mgr.x' 2024-04-23T10:24:40.913 INFO:journalctl@ceph.mon.a.smithi114.stdout:Apr 23 10:24:40 smithi114 ceph-mon[29255]: from='mgr.14152 172.21.15.114:0/1529706138' entity='mgr.x' 2024-04-23T10:24:40.913 INFO:journalctl@ceph.mon.a.smithi114.stdout:Apr 23 10:24:40 smithi114 ceph-mon[29255]: from='mgr.14152 172.21.15.114:0/1529706138' entity='mgr.x' 2024-04-23T10:24:40.913 INFO:journalctl@ceph.mon.a.smithi114.stdout:Apr 23 10:24:40 smithi114 ceph-mon[29255]: from='mgr.14152 172.21.15.114:0/1529706138' entity='mgr.x' 2024-04-23T10:24:40.913 INFO:journalctl@ceph.mon.a.smithi114.stdout:Apr 23 10:24:40 smithi114 ceph-mon[29255]: from='mgr.14152 172.21.15.114:0/1529706138' entity='mgr.x' 2024-04-23T10:24:40.913 INFO:journalctl@ceph.mon.a.smithi114.stdout:Apr 23 10:24:40 smithi114 ceph-mon[29255]: from='mgr.14152 172.21.15.114:0/1529706138' entity='mgr.x' 2024-04-23T10:24:40.914 DEBUG:tasks.cephfs.filesystem:Creating filesystem 'cephfs' 2024-04-23T10:24:40.914 DEBUG:teuthology.orchestra.run.smithi114:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool create cephfs_metadata --pg_num_min 64 2024-04-23T10:24:40.992 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 23 10:24:40 smithi119 ceph-mon[32608]: from='client.? 172.21.15.114:0/2261177185' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-23T10:24:40.992 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 23 10:24:40 smithi119 ceph-mon[32608]: pgmap v280: 1 pgs: 1 active+clean; 577 KiB data, 365 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-23T10:24:40.992 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 23 10:24:40 smithi119 ceph-mon[32608]: from='client.? 172.21.15.114:0/3397831736' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-23T10:24:40.992 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 23 10:24:40 smithi119 ceph-mon[32608]: from='mgr.14152 172.21.15.114:0/1529706138' entity='mgr.x' 2024-04-23T10:24:40.992 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 23 10:24:40 smithi119 ceph-mon[32608]: from='mgr.14152 172.21.15.114:0/1529706138' entity='mgr.x' 2024-04-23T10:24:40.992 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 23 10:24:40 smithi119 ceph-mon[32608]: from='mgr.14152 172.21.15.114:0/1529706138' entity='mgr.x' 2024-04-23T10:24:40.992 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 23 10:24:40 smithi119 ceph-mon[32608]: from='mgr.14152 172.21.15.114:0/1529706138' entity='mgr.x' 2024-04-23T10:24:40.992 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 23 10:24:40 smithi119 ceph-mon[32608]: from='mgr.14152 172.21.15.114:0/1529706138' entity='mgr.x' 2024-04-23T10:24:40.992 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 23 10:24:40 smithi119 ceph-mon[32608]: from='mgr.14152 172.21.15.114:0/1529706138' entity='mgr.x' 2024-04-23T10:24:41.018 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:24:41.018+0000 7f2cdd213640 1 -- 172.21.15.114:0/1732097895 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f2cd8100ea0 msgr2=0x7f2cd81012c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T10:24:41.018 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:24:41.018+0000 7f2cdd213640 1 --2- 172.21.15.114:0/1732097895 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f2cd8100ea0 0x7f2cd81012c0 secure :-1 s=READY pgs=204 cs=0 l=1 rev1=1 crypto rx=0x7f2cc8009900 tx=0x7f2cc802a410 comp rx=0 tx=0).stop 2024-04-23T10:24:41.018 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:24:41.018+0000 7f2cdd213640 1 -- 172.21.15.114:0/1732097895 shutdown_connections 2024-04-23T10:24:41.018 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:24:41.018+0000 7f2cdd213640 1 --2- 172.21.15.114:0/1732097895 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f2cd81029b0 0x7f2cd81092d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:24:41.018 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:24:41.018+0000 7f2cdd213640 1 --2- 172.21.15.114:0/1732097895 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f2cd8101ff0 0x7f2cd8102470 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:24:41.018 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:24:41.018+0000 7f2cdd213640 1 --2- 172.21.15.114:0/1732097895 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f2cd8100ea0 0x7f2cd81012c0 unknown :-1 s=CLOSED pgs=204 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:24:41.018 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:24:41.018+0000 7f2cdd213640 1 -- 172.21.15.114:0/1732097895 >> 172.21.15.114:0/1732097895 conn(0x7f2cd80fc410 msgr2=0x7f2cd80fe850 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T10:24:41.019 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:24:41.018+0000 7f2cdd213640 1 -- 172.21.15.114:0/1732097895 shutdown_connections 2024-04-23T10:24:41.019 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:24:41.018+0000 7f2cdd213640 1 -- 172.21.15.114:0/1732097895 wait complete. 2024-04-23T10:24:41.019 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:24:41.019+0000 7f2cdd213640 1 Processor -- start 2024-04-23T10:24:41.019 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:24:41.019+0000 7f2cdd213640 1 -- start start 2024-04-23T10:24:41.021 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:24:41.020+0000 7f2cdd213640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f2cd8100ea0 0x7f2cd81982b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T10:24:41.021 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:24:41.020+0000 7f2cdd213640 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f2cd8101ff0 0x7f2cd81987f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T10:24:41.021 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:24:41.020+0000 7f2cdd213640 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f2cd81029b0 0x7f2cd819f860 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T10:24:41.021 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:24:41.020+0000 7f2cdd213640 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x7f2cd810c9f0 con 0x7f2cd8100ea0 2024-04-23T10:24:41.021 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:24:41.020+0000 7f2cdd213640 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 -- 0x7f2cd810c870 con 0x7f2cd8101ff0 2024-04-23T10:24:41.021 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:24:41.020+0000 7f2cdd213640 1 -- --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_getmap magic: 0 -- 0x7f2cd810cb70 con 0x7f2cd81029b0 2024-04-23T10:24:41.021 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:24:41.020+0000 7f2cd6d9d640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f2cd8100ea0 0x7f2cd81982b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T10:24:41.021 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:24:41.020+0000 7f2cd6d9d640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f2cd8100ea0 0x7f2cd81982b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.114:44468/0 (socket says 172.21.15.114:44468) 2024-04-23T10:24:41.021 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:24:41.020+0000 7f2cd6d9d640 1 -- 172.21.15.114:0/1102954452 learned_addr learned my addr 172.21.15.114:0/1102954452 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-04-23T10:24:41.021 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:24:41.020+0000 7f2cd659c640 1 --2- 172.21.15.114:0/1102954452 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f2cd8101ff0 0x7f2cd81987f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T10:24:41.021 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:24:41.020+0000 7f2cd759e640 1 --2- 172.21.15.114:0/1102954452 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f2cd81029b0 0x7f2cd819f860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T10:24:41.021 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:24:41.020+0000 7f2cd6d9d640 1 -- 172.21.15.114:0/1102954452 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f2cd81029b0 msgr2=0x7f2cd819f860 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T10:24:41.022 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:24:41.020+0000 7f2cd6d9d640 1 --2- 172.21.15.114:0/1102954452 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f2cd81029b0 0x7f2cd819f860 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:24:41.022 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:24:41.020+0000 7f2cd6d9d640 1 -- 172.21.15.114:0/1102954452 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f2cd8101ff0 msgr2=0x7f2cd81987f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T10:24:41.022 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:24:41.020+0000 7f2cd6d9d640 1 --2- 172.21.15.114:0/1102954452 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f2cd8101ff0 0x7f2cd81987f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:24:41.022 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:24:41.020+0000 7f2cd6d9d640 1 -- 172.21.15.114:0/1102954452 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f2cd819fed0 con 0x7f2cd8100ea0 2024-04-23T10:24:41.022 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:24:41.020+0000 7f2cd759e640 1 --2- 172.21.15.114:0/1102954452 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f2cd81029b0 0x7f2cd819f860 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-23T10:24:41.022 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:24:41.020+0000 7f2cd6d9d640 1 --2- 172.21.15.114:0/1102954452 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f2cd8100ea0 0x7f2cd81982b0 secure :-1 s=READY pgs=205 cs=0 l=1 rev1=1 crypto rx=0x7f2cc8005940 tx=0x7f2cc8031a50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T10:24:41.022 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:24:41.020+0000 7f2cbffff640 1 -- 172.21.15.114:0/1102954452 <== mon.0 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2cc803f030 con 0x7f2cd8100ea0 2024-04-23T10:24:41.022 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:24:41.020+0000 7f2cbffff640 1 -- 172.21.15.114:0/1102954452 <== mon.0 v2:172.21.15.114:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f2cc8003aa0 con 0x7f2cd8100ea0 2024-04-23T10:24:41.022 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:24:41.020+0000 7f2cdd213640 1 -- 172.21.15.114:0/1102954452 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f2cd81a0160 con 0x7f2cd8100ea0 2024-04-23T10:24:41.022 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:24:41.020+0000 7f2cbffff640 1 -- 172.21.15.114:0/1102954452 <== mon.0 v2:172.21.15.114:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2cc8035070 con 0x7f2cd8100ea0 2024-04-23T10:24:41.022 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:24:41.020+0000 7f2cdd213640 1 -- 172.21.15.114:0/1102954452 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f2cd81a0650 con 0x7f2cd8100ea0 2024-04-23T10:24:41.022 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:24:41.021+0000 7f2cdd213640 1 -- 172.21.15.114:0/1102954452 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f2cd81a08b0 con 0x7f2cd8100ea0 2024-04-23T10:24:41.023 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:24:41.023+0000 7f2cbffff640 1 -- 172.21.15.114:0/1102954452 <== mon.0 v2:172.21.15.114:3300/0 4 ==== mgrmap(e 16) ==== 138647+0+0 (secure 0 0 0) 0x7f2cc8042030 con 0x7f2cd8100ea0 2024-04-23T10:24:41.023 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:24:41.023+0000 7f2cbffff640 1 --2- 172.21.15.114:0/1102954452 >> [v2:172.21.15.114:6800/465333808,v1:172.21.15.114:6801/465333808] conn(0x7f2cc00a3bb0 0x7f2cc00a6070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T10:24:41.023 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:24:41.023+0000 7f2cd659c640 1 --2- 172.21.15.114:0/1102954452 >> [v2:172.21.15.114:6800/465333808,v1:172.21.15.114:6801/465333808] conn(0x7f2cc00a3bb0 0x7f2cc00a6070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T10:24:41.023 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:24:41.023+0000 7f2cbffff640 1 -- 172.21.15.114:0/1102954452 <== mon.0 v2:172.21.15.114:3300/0 5 ==== osd_map(83..83 src has 1..83) ==== 7127+0+0 (secure 0 0 0) 0x7f2cc80f0770 con 0x7f2cd8100ea0 2024-04-23T10:24:41.023 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:24:41.024+0000 7f2cd659c640 1 --2- 172.21.15.114:0/1102954452 >> [v2:172.21.15.114:6800/465333808,v1:172.21.15.114:6801/465333808] conn(0x7f2cc00a3bb0 0x7f2cc00a6070 secure :-1 s=READY pgs=162 cs=0 l=1 rev1=1 crypto rx=0x7f2ccc005fd0 tx=0x7f2ccc005850 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T10:24:41.028 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:24:41.029+0000 7f2cbffff640 1 -- 172.21.15.114:0/1102954452 <== mon.0 v2:172.21.15.114:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f2cc80bd5f0 con 0x7f2cd8100ea0 2024-04-23T10:24:41.212 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:24:41.213+0000 7f2cdd213640 1 -- 172.21.15.114:0/1102954452 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_command({"prefix": "osd pool create", "pool": "cephfs_metadata", "pg_num_min": 64} v 0) -- 0x7f2cd81a0a80 con 0x7f2cd8100ea0 2024-04-23T10:24:41.828 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 23 10:24:41 smithi204 ceph-mon[32649]: from='client.? 172.21.15.114:0/3942888588' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-04-23T10:24:41.828 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 23 10:24:41 smithi204 ceph-mon[32649]: from='client.? 172.21.15.114:0/784199017' entity='client.admin' cmd=[{"prefix": "osd lspools", "format": "json-pretty"}]: dispatch 2024-04-23T10:24:41.828 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 23 10:24:41 smithi204 ceph-mon[32649]: from='mgr.14152 172.21.15.114:0/1529706138' entity='mgr.x' 2024-04-23T10:24:41.828 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 23 10:24:41 smithi204 ceph-mon[32649]: from='mgr.14152 172.21.15.114:0/1529706138' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-23T10:24:41.828 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 23 10:24:41 smithi204 ceph-mon[32649]: from='mgr.14152 172.21.15.114:0/1529706138' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-23T10:24:41.828 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 23 10:24:41 smithi204 ceph-mon[32649]: from='mgr.14152 172.21.15.114:0/1529706138' entity='mgr.x' 2024-04-23T10:24:41.828 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 23 10:24:41 smithi204 ceph-mon[32649]: from='mgr.14152 172.21.15.114:0/1529706138' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-23T10:24:41.828 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 23 10:24:41 smithi204 ceph-mon[32649]: from='client.? 172.21.15.114:0/1102954452' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "cephfs_metadata", "pg_num_min": 64}]: dispatch 2024-04-23T10:24:41.904 INFO:journalctl@ceph.mon.a.smithi114.stdout:Apr 23 10:24:41 smithi114 ceph-mon[29255]: from='client.? 172.21.15.114:0/3942888588' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-04-23T10:24:41.904 INFO:journalctl@ceph.mon.a.smithi114.stdout:Apr 23 10:24:41 smithi114 ceph-mon[29255]: from='client.? 172.21.15.114:0/784199017' entity='client.admin' cmd=[{"prefix": "osd lspools", "format": "json-pretty"}]: dispatch 2024-04-23T10:24:41.904 INFO:journalctl@ceph.mon.a.smithi114.stdout:Apr 23 10:24:41 smithi114 ceph-mon[29255]: from='mgr.14152 172.21.15.114:0/1529706138' entity='mgr.x' 2024-04-23T10:24:41.904 INFO:journalctl@ceph.mon.a.smithi114.stdout:Apr 23 10:24:41 smithi114 ceph-mon[29255]: from='mgr.14152 172.21.15.114:0/1529706138' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-23T10:24:41.904 INFO:journalctl@ceph.mon.a.smithi114.stdout:Apr 23 10:24:41 smithi114 ceph-mon[29255]: from='mgr.14152 172.21.15.114:0/1529706138' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-23T10:24:41.904 INFO:journalctl@ceph.mon.a.smithi114.stdout:Apr 23 10:24:41 smithi114 ceph-mon[29255]: from='mgr.14152 172.21.15.114:0/1529706138' entity='mgr.x' 2024-04-23T10:24:41.904 INFO:journalctl@ceph.mon.a.smithi114.stdout:Apr 23 10:24:41 smithi114 ceph-mon[29255]: from='mgr.14152 172.21.15.114:0/1529706138' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-23T10:24:41.904 INFO:journalctl@ceph.mon.a.smithi114.stdout:Apr 23 10:24:41 smithi114 ceph-mon[29255]: from='client.? 172.21.15.114:0/1102954452' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "cephfs_metadata", "pg_num_min": 64}]: dispatch 2024-04-23T10:24:41.955 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 23 10:24:41 smithi119 ceph-mon[32608]: from='client.? 172.21.15.114:0/3942888588' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-04-23T10:24:41.955 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 23 10:24:41 smithi119 ceph-mon[32608]: from='client.? 172.21.15.114:0/784199017' entity='client.admin' cmd=[{"prefix": "osd lspools", "format": "json-pretty"}]: dispatch 2024-04-23T10:24:41.956 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 23 10:24:41 smithi119 ceph-mon[32608]: from='mgr.14152 172.21.15.114:0/1529706138' entity='mgr.x' 2024-04-23T10:24:41.956 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 23 10:24:41 smithi119 ceph-mon[32608]: from='mgr.14152 172.21.15.114:0/1529706138' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-23T10:24:41.956 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 23 10:24:41 smithi119 ceph-mon[32608]: from='mgr.14152 172.21.15.114:0/1529706138' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-23T10:24:41.956 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 23 10:24:41 smithi119 ceph-mon[32608]: from='mgr.14152 172.21.15.114:0/1529706138' entity='mgr.x' 2024-04-23T10:24:41.956 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 23 10:24:41 smithi119 ceph-mon[32608]: from='mgr.14152 172.21.15.114:0/1529706138' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-23T10:24:41.956 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 23 10:24:41 smithi119 ceph-mon[32608]: from='client.? 172.21.15.114:0/1102954452' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "cephfs_metadata", "pg_num_min": 64}]: dispatch 2024-04-23T10:24:42.022 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:24:42.023+0000 7f2cbffff640 1 -- 172.21.15.114:0/1102954452 <== mon.0 v2:172.21.15.114:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool create", "pool": "cephfs_metadata", "pg_num_min": 64}]=0 pool 'cephfs_metadata' created v84) ==== 138+0+0 (secure 0 0 0) 0x7f2cc80bcef0 con 0x7f2cd8100ea0 2024-04-23T10:24:42.022 INFO:teuthology.orchestra.run.smithi114.stderr:pool 'cephfs_metadata' created 2024-04-23T10:24:42.026 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:24:42.026+0000 7f2cdd213640 1 -- 172.21.15.114:0/1102954452 >> [v2:172.21.15.114:6800/465333808,v1:172.21.15.114:6801/465333808] conn(0x7f2cc00a3bb0 msgr2=0x7f2cc00a6070 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T10:24:42.026 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:24:42.026+0000 7f2cdd213640 1 --2- 172.21.15.114:0/1102954452 >> [v2:172.21.15.114:6800/465333808,v1:172.21.15.114:6801/465333808] conn(0x7f2cc00a3bb0 0x7f2cc00a6070 secure :-1 s=READY pgs=162 cs=0 l=1 rev1=1 crypto rx=0x7f2ccc005fd0 tx=0x7f2ccc005850 comp rx=0 tx=0).stop 2024-04-23T10:24:42.026 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:24:42.026+0000 7f2cdd213640 1 -- 172.21.15.114:0/1102954452 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f2cd8100ea0 msgr2=0x7f2cd81982b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T10:24:42.026 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:24:42.026+0000 7f2cdd213640 1 --2- 172.21.15.114:0/1102954452 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f2cd8100ea0 0x7f2cd81982b0 secure :-1 s=READY pgs=205 cs=0 l=1 rev1=1 crypto rx=0x7f2cc8005940 tx=0x7f2cc8031a50 comp rx=0 tx=0).stop 2024-04-23T10:24:42.026 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:24:42.027+0000 7f2cdd213640 1 -- 172.21.15.114:0/1102954452 shutdown_connections 2024-04-23T10:24:42.026 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:24:42.027+0000 7f2cdd213640 1 --2- 172.21.15.114:0/1102954452 >> [v2:172.21.15.114:6800/465333808,v1:172.21.15.114:6801/465333808] conn(0x7f2cc00a3bb0 0x7f2cc00a6070 unknown :-1 s=CLOSED pgs=162 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:24:42.026 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:24:42.027+0000 7f2cdd213640 1 --2- 172.21.15.114:0/1102954452 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f2cd81029b0 0x7f2cd819f860 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:24:42.026 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:24:42.027+0000 7f2cdd213640 1 --2- 172.21.15.114:0/1102954452 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f2cd8101ff0 0x7f2cd81987f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:24:42.026 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:24:42.027+0000 7f2cdd213640 1 --2- 172.21.15.114:0/1102954452 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f2cd8100ea0 0x7f2cd81982b0 unknown :-1 s=CLOSED pgs=205 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:24:42.026 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:24:42.027+0000 7f2cdd213640 1 -- 172.21.15.114:0/1102954452 >> 172.21.15.114:0/1102954452 conn(0x7f2cd80fc410 msgr2=0x7f2cd806a2d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T10:24:42.026 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:24:42.027+0000 7f2cdd213640 1 -- 172.21.15.114:0/1102954452 shutdown_connections 2024-04-23T10:24:42.026 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:24:42.027+0000 7f2cdd213640 1 -- 172.21.15.114:0/1102954452 wait complete. 2024-04-23T10:24:42.039 DEBUG:teuthology.orchestra.run.smithi114:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool create cephfs_data 64 --pg_num_min 64 --target_size_ratio 0.9 2024-04-23T10:24:42.148 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:24:42.147+0000 7f4c846a3640 1 -- 172.21.15.114:0/1219310993 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f4c7c1053e0 msgr2=0x7f4c7c1077d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T10:24:42.148 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:24:42.147+0000 7f4c846a3640 1 --2- 172.21.15.114:0/1219310993 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f4c7c1053e0 0x7f4c7c1077d0 secure :-1 s=READY pgs=206 cs=0 l=1 rev1=1 crypto rx=0x7f4c6c009900 tx=0x7f4c6c02a410 comp rx=0 tx=0).stop 2024-04-23T10:24:42.148 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:24:42.147+0000 7f4c846a3640 1 -- 172.21.15.114:0/1219310993 shutdown_connections 2024-04-23T10:24:42.148 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:24:42.147+0000 7f4c846a3640 1 --2- 172.21.15.114:0/1219310993 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f4c7c107d10 0x7f4c7c10a100 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:24:42.148 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:24:42.147+0000 7f4c846a3640 1 --2- 172.21.15.114:0/1219310993 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f4c7c1053e0 0x7f4c7c1077d0 unknown :-1 s=CLOSED pgs=206 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:24:42.149 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:24:42.147+0000 7f4c846a3640 1 --2- 172.21.15.114:0/1219310993 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f4c7c102ab0 0x7f4c7c104ea0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:24:42.149 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:24:42.147+0000 7f4c846a3640 1 -- 172.21.15.114:0/1219310993 >> 172.21.15.114:0/1219310993 conn(0x7f4c7c0fc400 msgr2=0x7f4c7c0fe840 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T10:24:42.149 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:24:42.147+0000 7f4c846a3640 1 -- 172.21.15.114:0/1219310993 shutdown_connections 2024-04-23T10:24:42.149 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:24:42.147+0000 7f4c846a3640 1 -- 172.21.15.114:0/1219310993 wait complete. 2024-04-23T10:24:42.149 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:24:42.148+0000 7f4c846a3640 1 Processor -- start 2024-04-23T10:24:42.149 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:24:42.148+0000 7f4c846a3640 1 -- start start 2024-04-23T10:24:42.149 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:24:42.148+0000 7f4c846a3640 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f4c7c102ab0 0x7f4c7c1981b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T10:24:42.149 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:24:42.148+0000 7f4c846a3640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f4c7c1053e0 0x7f4c7c1986f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T10:24:42.149 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:24:42.148+0000 7f4c846a3640 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f4c7c107d10 0x7f4c7c19f760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T10:24:42.149 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:24:42.148+0000 7f4c846a3640 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x7f4c7c10daf0 con 0x7f4c7c1053e0 2024-04-23T10:24:42.149 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:24:42.148+0000 7f4c846a3640 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 -- 0x7f4c7c10d970 con 0x7f4c7c107d10 2024-04-23T10:24:42.149 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:24:42.148+0000 7f4c846a3640 1 -- --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_getmap magic: 0 -- 0x7f4c7c10dc70 con 0x7f4c7c102ab0 2024-04-23T10:24:42.149 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:24:42.148+0000 7f4c81c3e640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f4c7c1053e0 0x7f4c7c1986f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T10:24:42.149 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:24:42.148+0000 7f4c81c3e640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f4c7c1053e0 0x7f4c7c1986f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.114:44496/0 (socket says 172.21.15.114:44496) 2024-04-23T10:24:42.149 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:24:42.148+0000 7f4c81c3e640 1 -- 172.21.15.114:0/1930247895 learned_addr learned my addr 172.21.15.114:0/1930247895 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-04-23T10:24:42.150 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:24:42.148+0000 7f4c82c40640 1 --2- 172.21.15.114:0/1930247895 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f4c7c107d10 0x7f4c7c19f760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T10:24:42.150 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:24:42.148+0000 7f4c8243f640 1 --2- 172.21.15.114:0/1930247895 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f4c7c102ab0 0x7f4c7c1981b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T10:24:42.150 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:24:42.149+0000 7f4c81c3e640 1 -- 172.21.15.114:0/1930247895 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f4c7c102ab0 msgr2=0x7f4c7c1981b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T10:24:42.150 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:24:42.149+0000 7f4c81c3e640 1 --2- 172.21.15.114:0/1930247895 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f4c7c102ab0 0x7f4c7c1981b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:24:42.150 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:24:42.149+0000 7f4c81c3e640 1 -- 172.21.15.114:0/1930247895 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f4c7c107d10 msgr2=0x7f4c7c19f760 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T10:24:42.150 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:24:42.149+0000 7f4c81c3e640 1 --2- 172.21.15.114:0/1930247895 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f4c7c107d10 0x7f4c7c19f760 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:24:42.150 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:24:42.149+0000 7f4c81c3e640 1 -- 172.21.15.114:0/1930247895 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f4c7c19fdd0 con 0x7f4c7c1053e0 2024-04-23T10:24:42.150 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:24:42.149+0000 7f4c81c3e640 1 --2- 172.21.15.114:0/1930247895 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f4c7c1053e0 0x7f4c7c1986f0 secure :-1 s=READY pgs=207 cs=0 l=1 rev1=1 crypto rx=0x7f4c6c006010 tx=0x7f4c6c0042f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T10:24:42.150 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:24:42.149+0000 7f4c737fe640 1 -- 172.21.15.114:0/1930247895 <== mon.0 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4c6c03f030 con 0x7f4c7c1053e0 2024-04-23T10:24:42.150 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:24:42.149+0000 7f4c846a3640 1 -- 172.21.15.114:0/1930247895 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f4c7c1a0060 con 0x7f4c7c1053e0 2024-04-23T10:24:42.150 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:24:42.149+0000 7f4c737fe640 1 -- 172.21.15.114:0/1930247895 <== mon.0 v2:172.21.15.114:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f4c6c003cb0 con 0x7f4c7c1053e0 2024-04-23T10:24:42.150 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:24:42.149+0000 7f4c737fe640 1 -- 172.21.15.114:0/1930247895 <== mon.0 v2:172.21.15.114:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4c6c035070 con 0x7f4c7c1053e0 2024-04-23T10:24:42.150 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:24:42.149+0000 7f4c846a3640 1 -- 172.21.15.114:0/1930247895 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f4c7c1a0550 con 0x7f4c7c1053e0 2024-04-23T10:24:42.151 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:24:42.150+0000 7f4c846a3640 1 -- 172.21.15.114:0/1930247895 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f4c7c0fdef0 con 0x7f4c7c1053e0 2024-04-23T10:24:42.151 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:24:42.151+0000 7f4c737fe640 1 -- 172.21.15.114:0/1930247895 <== mon.0 v2:172.21.15.114:3300/0 4 ==== mgrmap(e 16) ==== 138647+0+0 (secure 0 0 0) 0x7f4c6c042030 con 0x7f4c7c1053e0 2024-04-23T10:24:42.151 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:24:42.152+0000 7f4c737fe640 1 --2- 172.21.15.114:0/1930247895 >> [v2:172.21.15.114:6800/465333808,v1:172.21.15.114:6801/465333808] conn(0x7f4c680a3bb0 0x7f4c680a6070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T10:24:42.151 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:24:42.152+0000 7f4c8243f640 1 --2- 172.21.15.114:0/1930247895 >> [v2:172.21.15.114:6800/465333808,v1:172.21.15.114:6801/465333808] conn(0x7f4c680a3bb0 0x7f4c680a6070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T10:24:42.152 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:24:42.152+0000 7f4c737fe640 1 -- 172.21.15.114:0/1930247895 <== mon.0 v2:172.21.15.114:3300/0 5 ==== osd_map(84..84 src has 1..84) ==== 7497+0+0 (secure 0 0 0) 0x7f4c6c0f0a30 con 0x7f4c7c1053e0 2024-04-23T10:24:42.152 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:24:42.152+0000 7f4c8243f640 1 --2- 172.21.15.114:0/1930247895 >> [v2:172.21.15.114:6800/465333808,v1:172.21.15.114:6801/465333808] conn(0x7f4c680a3bb0 0x7f4c680a6070 secure :-1 s=READY pgs=163 cs=0 l=1 rev1=1 crypto rx=0x7f4c7800abe0 tx=0x7f4c7800a2d0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T10:24:42.154 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:24:42.155+0000 7f4c737fe640 1 -- 172.21.15.114:0/1930247895 <== mon.0 v2:172.21.15.114:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f4c6c0bd740 con 0x7f4c7c1053e0 2024-04-23T10:24:42.336 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:24:42.337+0000 7f4c846a3640 1 -- 172.21.15.114:0/1930247895 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_command({"prefix": "osd pool create", "pool": "cephfs_data", "pg_num": 64, "pg_num_min": 64, "target_size_ratio": 0.9} v 0) -- 0x7f4c7c02d5c0 con 0x7f4c7c1053e0 2024-04-23T10:24:43.028 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:24:43.029+0000 7f4c737fe640 1 -- 172.21.15.114:0/1930247895 <== mon.0 v2:172.21.15.114:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool create", "pool": "cephfs_data", "pg_num": 64, "pg_num_min": 64, "target_size_ratio": 0.9}]=0 pool 'cephfs_data' created v85) ==== 170+0+0 (secure 0 0 0) 0x7f4c6c0bd040 con 0x7f4c7c1053e0 2024-04-23T10:24:43.028 INFO:teuthology.orchestra.run.smithi114.stderr:pool 'cephfs_data' created 2024-04-23T10:24:43.031 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:24:43.032+0000 7f4c846a3640 1 -- 172.21.15.114:0/1930247895 >> [v2:172.21.15.114:6800/465333808,v1:172.21.15.114:6801/465333808] conn(0x7f4c680a3bb0 msgr2=0x7f4c680a6070 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T10:24:43.031 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:24:43.032+0000 7f4c846a3640 1 --2- 172.21.15.114:0/1930247895 >> [v2:172.21.15.114:6800/465333808,v1:172.21.15.114:6801/465333808] conn(0x7f4c680a3bb0 0x7f4c680a6070 secure :-1 s=READY pgs=163 cs=0 l=1 rev1=1 crypto rx=0x7f4c7800abe0 tx=0x7f4c7800a2d0 comp rx=0 tx=0).stop 2024-04-23T10:24:43.031 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:24:43.032+0000 7f4c846a3640 1 -- 172.21.15.114:0/1930247895 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f4c7c1053e0 msgr2=0x7f4c7c1986f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T10:24:43.031 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:24:43.032+0000 7f4c846a3640 1 --2- 172.21.15.114:0/1930247895 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f4c7c1053e0 0x7f4c7c1986f0 secure :-1 s=READY pgs=207 cs=0 l=1 rev1=1 crypto rx=0x7f4c6c006010 tx=0x7f4c6c0042f0 comp rx=0 tx=0).stop 2024-04-23T10:24:43.031 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:24:43.032+0000 7f4c846a3640 1 -- 172.21.15.114:0/1930247895 shutdown_connections 2024-04-23T10:24:43.031 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:24:43.032+0000 7f4c846a3640 1 --2- 172.21.15.114:0/1930247895 >> [v2:172.21.15.114:6800/465333808,v1:172.21.15.114:6801/465333808] conn(0x7f4c680a3bb0 0x7f4c680a6070 unknown :-1 s=CLOSED pgs=163 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:24:43.031 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:24:43.032+0000 7f4c846a3640 1 --2- 172.21.15.114:0/1930247895 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f4c7c107d10 0x7f4c7c19f760 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:24:43.031 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:24:43.032+0000 7f4c846a3640 1 --2- 172.21.15.114:0/1930247895 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f4c7c1053e0 0x7f4c7c1986f0 secure :-1 s=CLOSED pgs=207 cs=0 l=1 rev1=1 crypto rx=0x7f4c6c006010 tx=0x7f4c6c0042f0 comp rx=0 tx=0).stop 2024-04-23T10:24:43.032 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:24:43.032+0000 7f4c846a3640 1 --2- 172.21.15.114:0/1930247895 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f4c7c102ab0 0x7f4c7c1981b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:24:43.032 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:24:43.032+0000 7f4c846a3640 1 -- 172.21.15.114:0/1930247895 >> 172.21.15.114:0/1930247895 conn(0x7f4c7c0fc400 msgr2=0x7f4c7c1034d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T10:24:43.032 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:24:43.032+0000 7f4c846a3640 1 -- 172.21.15.114:0/1930247895 shutdown_connections 2024-04-23T10:24:43.032 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:24:43.032+0000 7f4c846a3640 1 -- 172.21.15.114:0/1930247895 wait complete. 2024-04-23T10:24:43.052 DEBUG:teuthology.orchestra.run.smithi114:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs new cephfs cephfs_metadata cephfs_data 2024-04-23T10:24:43.156 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:24:43.155+0000 7fef13ea7640 1 -- 172.21.15.114:0/2474003791 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fef0c101fe0 msgr2=0x7fef0c102460 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T10:24:43.156 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:24:43.155+0000 7fef13ea7640 1 --2- 172.21.15.114:0/2474003791 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fef0c101fe0 0x7fef0c102460 secure :-1 s=READY pgs=208 cs=0 l=1 rev1=1 crypto rx=0x7feefc009900 tx=0x7feefc02a410 comp rx=0 tx=0).stop 2024-04-23T10:24:43.156 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:24:43.155+0000 7fef13ea7640 1 -- 172.21.15.114:0/2474003791 shutdown_connections 2024-04-23T10:24:43.156 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:24:43.155+0000 7fef13ea7640 1 --2- 172.21.15.114:0/2474003791 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fef0c1029a0 0x7fef0c1092c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:24:43.156 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:24:43.155+0000 7fef13ea7640 1 --2- 172.21.15.114:0/2474003791 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fef0c101fe0 0x7fef0c102460 unknown :-1 s=CLOSED pgs=208 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:24:43.157 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:24:43.155+0000 7fef13ea7640 1 --2- 172.21.15.114:0/2474003791 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fef0c100e90 0x7fef0c1012b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:24:43.157 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:24:43.155+0000 7fef13ea7640 1 -- 172.21.15.114:0/2474003791 >> 172.21.15.114:0/2474003791 conn(0x7fef0c0fc400 msgr2=0x7fef0c0fe840 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T10:24:43.157 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:24:43.155+0000 7fef13ea7640 1 -- 172.21.15.114:0/2474003791 shutdown_connections 2024-04-23T10:24:43.157 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:24:43.155+0000 7fef13ea7640 1 -- 172.21.15.114:0/2474003791 wait complete. 2024-04-23T10:24:43.157 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:24:43.156+0000 7fef13ea7640 1 Processor -- start 2024-04-23T10:24:43.157 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:24:43.156+0000 7fef13ea7640 1 -- start start 2024-04-23T10:24:43.157 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:24:43.156+0000 7fef13ea7640 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fef0c100e90 0x7fef0c06b0e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T10:24:43.157 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:24:43.156+0000 7fef13ea7640 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fef0c101fe0 0x7fef0c19cf80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T10:24:43.157 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:24:43.156+0000 7fef13ea7640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fef0c1029a0 0x7fef0c19f5a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T10:24:43.157 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:24:43.156+0000 7fef13ea7640 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x7fef0c10c9e0 con 0x7fef0c1029a0 2024-04-23T10:24:43.157 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:24:43.156+0000 7fef13ea7640 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 -- 0x7fef0c10c860 con 0x7fef0c101fe0 2024-04-23T10:24:43.157 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:24:43.156+0000 7fef13ea7640 1 -- --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_getmap magic: 0 -- 0x7fef0c10cb60 con 0x7fef0c100e90 2024-04-23T10:24:43.157 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:24:43.156+0000 7fef12444640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fef0c1029a0 0x7fef0c19f5a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T10:24:43.158 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:24:43.156+0000 7fef12444640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fef0c1029a0 0x7fef0c19f5a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.114:44516/0 (socket says 172.21.15.114:44516) 2024-04-23T10:24:43.158 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:24:43.156+0000 7fef12444640 1 -- 172.21.15.114:0/2263953334 learned_addr learned my addr 172.21.15.114:0/2263953334 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-04-23T10:24:43.158 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:24:43.156+0000 7fef11442640 1 --2- 172.21.15.114:0/2263953334 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fef0c101fe0 0x7fef0c19cf80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T10:24:43.158 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:24:43.156+0000 7fef11c43640 1 --2- 172.21.15.114:0/2263953334 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fef0c100e90 0x7fef0c06b0e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T10:24:43.158 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:24:43.156+0000 7fef12444640 1 -- 172.21.15.114:0/2263953334 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fef0c100e90 msgr2=0x7fef0c06b0e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T10:24:43.158 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:24:43.156+0000 7fef12444640 1 --2- 172.21.15.114:0/2263953334 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fef0c100e90 0x7fef0c06b0e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:24:43.158 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:24:43.156+0000 7fef12444640 1 -- 172.21.15.114:0/2263953334 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fef0c101fe0 msgr2=0x7fef0c19cf80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T10:24:43.158 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:24:43.156+0000 7fef12444640 1 --2- 172.21.15.114:0/2263953334 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fef0c101fe0 0x7fef0c19cf80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:24:43.158 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:24:43.156+0000 7fef12444640 1 -- 172.21.15.114:0/2263953334 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fef0c19feb0 con 0x7fef0c1029a0 2024-04-23T10:24:43.158 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:24:43.156+0000 7fef12444640 1 --2- 172.21.15.114:0/2263953334 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fef0c1029a0 0x7fef0c19f5a0 secure :-1 s=READY pgs=209 cs=0 l=1 rev1=1 crypto rx=0x7fef0400cad0 tx=0x7fef0400cf90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T10:24:43.158 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:24:43.156+0000 7fef02ffd640 1 -- 172.21.15.114:0/2263953334 <== mon.0 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fef0400ae20 con 0x7fef0c1029a0 2024-04-23T10:24:43.158 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:24:43.157+0000 7fef02ffd640 1 -- 172.21.15.114:0/2263953334 <== mon.0 v2:172.21.15.114:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fef04005020 con 0x7fef0c1029a0 2024-04-23T10:24:43.158 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:24:43.157+0000 7fef13ea7640 1 -- 172.21.15.114:0/2263953334 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fef0c1a0140 con 0x7fef0c1029a0 2024-04-23T10:24:43.158 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:24:43.157+0000 7fef02ffd640 1 -- 172.21.15.114:0/2263953334 <== mon.0 v2:172.21.15.114:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fef0400f6b0 con 0x7fef0c1029a0 2024-04-23T10:24:43.158 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:24:43.157+0000 7fef13ea7640 1 -- 172.21.15.114:0/2263953334 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fef0c1a06b0 con 0x7fef0c1029a0 2024-04-23T10:24:43.159 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:24:43.158+0000 7fef13ea7640 1 -- 172.21.15.114:0/2263953334 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fef0c108240 con 0x7fef0c1029a0 2024-04-23T10:24:43.159 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:24:43.159+0000 7fef02ffd640 1 -- 172.21.15.114:0/2263953334 <== mon.0 v2:172.21.15.114:3300/0 4 ==== mgrmap(e 16) ==== 138647+0+0 (secure 0 0 0) 0x7fef040051d0 con 0x7fef0c1029a0 2024-04-23T10:24:43.159 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:24:43.159+0000 7fef02ffd640 1 --2- 172.21.15.114:0/2263953334 >> [v2:172.21.15.114:6800/465333808,v1:172.21.15.114:6801/465333808] conn(0x7feef80a3c30 0x7feef80a60f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T10:24:43.159 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:24:43.159+0000 7fef11c43640 1 --2- 172.21.15.114:0/2263953334 >> [v2:172.21.15.114:6800/465333808,v1:172.21.15.114:6801/465333808] conn(0x7feef80a3c30 0x7feef80a60f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T10:24:43.160 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:24:43.159+0000 7fef02ffd640 1 -- 172.21.15.114:0/2263953334 <== mon.0 v2:172.21.15.114:3300/0 5 ==== osd_map(85..85 src has 1..85) ==== 7879+0+0 (secure 0 0 0) 0x7fef040d05d0 con 0x7fef0c1029a0 2024-04-23T10:24:43.160 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:24:43.160+0000 7fef11c43640 1 --2- 172.21.15.114:0/2263953334 >> [v2:172.21.15.114:6800/465333808,v1:172.21.15.114:6801/465333808] conn(0x7feef80a3c30 0x7feef80a60f0 secure :-1 s=READY pgs=164 cs=0 l=1 rev1=1 crypto rx=0x7fef08009b60 tx=0x7fef08005f90 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T10:24:43.162 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:24:43.163+0000 7fef02ffd640 1 -- 172.21.15.114:0/2263953334 <== mon.0 v2:172.21.15.114:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7fef04014030 con 0x7fef0c1029a0 2024-04-23T10:24:43.328 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 23 10:24:43 smithi204 ceph-mon[32649]: pgmap v281: 1 pgs: 1 active+clean; 577 KiB data, 365 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-23T10:24:43.328 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 23 10:24:43 smithi204 ceph-mon[32649]: from='client.? 172.21.15.114:0/1102954452' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "cephfs_metadata", "pg_num_min": 64}]': finished 2024-04-23T10:24:43.328 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 23 10:24:43 smithi204 ceph-mon[32649]: osdmap e84: 12 total, 12 up, 12 in 2024-04-23T10:24:43.328 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 23 10:24:43 smithi204 ceph-mon[32649]: from='client.? 172.21.15.114:0/1930247895' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "cephfs_data", "pg_num": 64, "pg_num_min": 64, "target_size_ratio": 0.9}]: dispatch 2024-04-23T10:24:43.328 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 23 10:24:43 smithi204 ceph-mon[32649]: from='mgr.14152 172.21.15.114:0/1529706138' entity='mgr.x' 2024-04-23T10:24:43.328 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 23 10:24:43 smithi204 ceph-mon[32649]: from='mgr.14152 172.21.15.114:0/1529706138' entity='mgr.x' 2024-04-23T10:24:43.328 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 23 10:24:43 smithi204 ceph-mon[32649]: from='mgr.14152 172.21.15.114:0/1529706138' entity='mgr.x' 2024-04-23T10:24:43.328 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 23 10:24:43 smithi204 ceph-mon[32649]: from='mgr.14152 172.21.15.114:0/1529706138' entity='mgr.x' 2024-04-23T10:24:43.328 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 23 10:24:43 smithi204 ceph-mon[32649]: from='mgr.14152 172.21.15.114:0/1529706138' entity='mgr.x' 2024-04-23T10:24:43.328 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 23 10:24:43 smithi204 ceph-mon[32649]: from='mgr.14152 172.21.15.114:0/1529706138' entity='mgr.x' 2024-04-23T10:24:43.328 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 23 10:24:43 smithi204 ceph-mon[32649]: from='mgr.14152 172.21.15.114:0/1529706138' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-23T10:24:43.329 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 23 10:24:43 smithi204 ceph-mon[32649]: from='mgr.14152 172.21.15.114:0/1529706138' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-23T10:24:43.329 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 23 10:24:43 smithi204 ceph-mon[32649]: from='mgr.14152 172.21.15.114:0/1529706138' entity='mgr.x' 2024-04-23T10:24:43.380 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:24:43.381+0000 7fef13ea7640 1 -- 172.21.15.114:0/2263953334 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_command({"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs_metadata", "data": "cephfs_data"} v 0) -- 0x7fef0c062bc0 con 0x7fef0c1029a0 2024-04-23T10:24:43.492 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 23 10:24:43 smithi119 ceph-mon[32608]: pgmap v281: 1 pgs: 1 active+clean; 577 KiB data, 365 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-23T10:24:43.492 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 23 10:24:43 smithi119 ceph-mon[32608]: from='client.? 172.21.15.114:0/1102954452' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "cephfs_metadata", "pg_num_min": 64}]': finished 2024-04-23T10:24:43.492 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 23 10:24:43 smithi119 ceph-mon[32608]: osdmap e84: 12 total, 12 up, 12 in 2024-04-23T10:24:43.492 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 23 10:24:43 smithi119 ceph-mon[32608]: from='client.? 172.21.15.114:0/1930247895' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "cephfs_data", "pg_num": 64, "pg_num_min": 64, "target_size_ratio": 0.9}]: dispatch 2024-04-23T10:24:43.492 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 23 10:24:43 smithi119 ceph-mon[32608]: from='mgr.14152 172.21.15.114:0/1529706138' entity='mgr.x' 2024-04-23T10:24:43.492 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 23 10:24:43 smithi119 ceph-mon[32608]: from='mgr.14152 172.21.15.114:0/1529706138' entity='mgr.x' 2024-04-23T10:24:43.492 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 23 10:24:43 smithi119 ceph-mon[32608]: from='mgr.14152 172.21.15.114:0/1529706138' entity='mgr.x' 2024-04-23T10:24:43.492 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 23 10:24:43 smithi119 ceph-mon[32608]: from='mgr.14152 172.21.15.114:0/1529706138' entity='mgr.x' 2024-04-23T10:24:43.492 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 23 10:24:43 smithi119 ceph-mon[32608]: from='mgr.14152 172.21.15.114:0/1529706138' entity='mgr.x' 2024-04-23T10:24:43.492 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 23 10:24:43 smithi119 ceph-mon[32608]: from='mgr.14152 172.21.15.114:0/1529706138' entity='mgr.x' 2024-04-23T10:24:43.492 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 23 10:24:43 smithi119 ceph-mon[32608]: from='mgr.14152 172.21.15.114:0/1529706138' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-23T10:24:43.492 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 23 10:24:43 smithi119 ceph-mon[32608]: from='mgr.14152 172.21.15.114:0/1529706138' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-23T10:24:43.492 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 23 10:24:43 smithi119 ceph-mon[32608]: from='mgr.14152 172.21.15.114:0/1529706138' entity='mgr.x' 2024-04-23T10:24:43.504 INFO:journalctl@ceph.mon.a.smithi114.stdout:Apr 23 10:24:43 smithi114 ceph-mon[29255]: pgmap v281: 1 pgs: 1 active+clean; 577 KiB data, 365 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-23T10:24:43.504 INFO:journalctl@ceph.mon.a.smithi114.stdout:Apr 23 10:24:43 smithi114 ceph-mon[29255]: from='client.? 172.21.15.114:0/1102954452' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "cephfs_metadata", "pg_num_min": 64}]': finished 2024-04-23T10:24:43.504 INFO:journalctl@ceph.mon.a.smithi114.stdout:Apr 23 10:24:43 smithi114 ceph-mon[29255]: osdmap e84: 12 total, 12 up, 12 in 2024-04-23T10:24:43.504 INFO:journalctl@ceph.mon.a.smithi114.stdout:Apr 23 10:24:43 smithi114 ceph-mon[29255]: from='client.? 172.21.15.114:0/1930247895' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "cephfs_data", "pg_num": 64, "pg_num_min": 64, "target_size_ratio": 0.9}]: dispatch 2024-04-23T10:24:43.504 INFO:journalctl@ceph.mon.a.smithi114.stdout:Apr 23 10:24:43 smithi114 ceph-mon[29255]: from='mgr.14152 172.21.15.114:0/1529706138' entity='mgr.x' 2024-04-23T10:24:43.504 INFO:journalctl@ceph.mon.a.smithi114.stdout:Apr 23 10:24:43 smithi114 ceph-mon[29255]: from='mgr.14152 172.21.15.114:0/1529706138' entity='mgr.x' 2024-04-23T10:24:43.504 INFO:journalctl@ceph.mon.a.smithi114.stdout:Apr 23 10:24:43 smithi114 ceph-mon[29255]: from='mgr.14152 172.21.15.114:0/1529706138' entity='mgr.x' 2024-04-23T10:24:43.504 INFO:journalctl@ceph.mon.a.smithi114.stdout:Apr 23 10:24:43 smithi114 ceph-mon[29255]: from='mgr.14152 172.21.15.114:0/1529706138' entity='mgr.x' 2024-04-23T10:24:43.504 INFO:journalctl@ceph.mon.a.smithi114.stdout:Apr 23 10:24:43 smithi114 ceph-mon[29255]: from='mgr.14152 172.21.15.114:0/1529706138' entity='mgr.x' 2024-04-23T10:24:43.504 INFO:journalctl@ceph.mon.a.smithi114.stdout:Apr 23 10:24:43 smithi114 ceph-mon[29255]: from='mgr.14152 172.21.15.114:0/1529706138' entity='mgr.x' 2024-04-23T10:24:43.504 INFO:journalctl@ceph.mon.a.smithi114.stdout:Apr 23 10:24:43 smithi114 ceph-mon[29255]: from='mgr.14152 172.21.15.114:0/1529706138' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-23T10:24:43.504 INFO:journalctl@ceph.mon.a.smithi114.stdout:Apr 23 10:24:43 smithi114 ceph-mon[29255]: from='mgr.14152 172.21.15.114:0/1529706138' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-23T10:24:43.505 INFO:journalctl@ceph.mon.a.smithi114.stdout:Apr 23 10:24:43 smithi114 ceph-mon[29255]: from='mgr.14152 172.21.15.114:0/1529706138' entity='mgr.x' 2024-04-23T10:24:44.315 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:24:44.316+0000 7fef02ffd640 1 -- 172.21.15.114:0/2263953334 <== mon.0 v2:172.21.15.114:3300/0 7 ==== mon_command_ack([{"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs_metadata", "data": "cephfs_data"}]=0 new fs with metadata pool 2 and data pool 3 v14) ==== 172+0+0 (secure 0 0 0) 0x7fef0409d1c0 con 0x7fef0c1029a0 2024-04-23T10:24:44.315 INFO:teuthology.orchestra.run.smithi114.stderr:new fs with metadata pool 2 and data pool 3 2024-04-23T10:24:44.318 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:24:44.319+0000 7fef13ea7640 1 -- 172.21.15.114:0/2263953334 >> [v2:172.21.15.114:6800/465333808,v1:172.21.15.114:6801/465333808] conn(0x7feef80a3c30 msgr2=0x7feef80a60f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T10:24:44.318 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:24:44.319+0000 7fef13ea7640 1 --2- 172.21.15.114:0/2263953334 >> [v2:172.21.15.114:6800/465333808,v1:172.21.15.114:6801/465333808] conn(0x7feef80a3c30 0x7feef80a60f0 secure :-1 s=READY pgs=164 cs=0 l=1 rev1=1 crypto rx=0x7fef08009b60 tx=0x7fef08005f90 comp rx=0 tx=0).stop 2024-04-23T10:24:44.318 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:24:44.319+0000 7fef13ea7640 1 -- 172.21.15.114:0/2263953334 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fef0c1029a0 msgr2=0x7fef0c19f5a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T10:24:44.318 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:24:44.319+0000 7fef13ea7640 1 --2- 172.21.15.114:0/2263953334 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fef0c1029a0 0x7fef0c19f5a0 secure :-1 s=READY pgs=209 cs=0 l=1 rev1=1 crypto rx=0x7fef0400cad0 tx=0x7fef0400cf90 comp rx=0 tx=0).stop 2024-04-23T10:24:44.318 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:24:44.319+0000 7fef13ea7640 1 -- 172.21.15.114:0/2263953334 shutdown_connections 2024-04-23T10:24:44.318 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:24:44.319+0000 7fef13ea7640 1 --2- 172.21.15.114:0/2263953334 >> [v2:172.21.15.114:6800/465333808,v1:172.21.15.114:6801/465333808] conn(0x7feef80a3c30 0x7feef80a60f0 unknown :-1 s=CLOSED pgs=164 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:24:44.318 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:24:44.319+0000 7fef13ea7640 1 --2- 172.21.15.114:0/2263953334 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fef0c1029a0 0x7fef0c19f5a0 unknown :-1 s=CLOSED pgs=209 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:24:44.318 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:24:44.319+0000 7fef13ea7640 1 --2- 172.21.15.114:0/2263953334 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fef0c101fe0 0x7fef0c19cf80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:24:44.319 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:24:44.319+0000 7fef13ea7640 1 --2- 172.21.15.114:0/2263953334 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fef0c100e90 0x7fef0c06b0e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:24:44.319 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:24:44.319+0000 7fef13ea7640 1 -- 172.21.15.114:0/2263953334 >> 172.21.15.114:0/2263953334 conn(0x7fef0c0fc400 msgr2=0x7fef0c105410 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T10:24:44.319 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:24:44.319+0000 7fef13ea7640 1 -- 172.21.15.114:0/2263953334 shutdown_connections 2024-04-23T10:24:44.319 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:24:44.319+0000 7fef13ea7640 1 -- 172.21.15.114:0/2263953334 wait complete. 2024-04-23T10:24:44.328 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 23 10:24:44 smithi204 ceph-mon[32649]: from='client.? 172.21.15.114:0/1930247895' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "cephfs_data", "pg_num": 64, "pg_num_min": 64, "target_size_ratio": 0.9}]': finished 2024-04-23T10:24:44.328 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 23 10:24:44 smithi204 ceph-mon[32649]: osdmap e85: 12 total, 12 up, 12 in 2024-04-23T10:24:44.328 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 23 10:24:44 smithi204 ceph-mon[32649]: from='client.? 172.21.15.114:0/2263953334' entity='client.admin' cmd=[{"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs_metadata", "data": "cephfs_data"}]: dispatch 2024-04-23T10:24:44.328 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 23 10:24:44 smithi204 ceph-mon[32649]: daemon mds.b assigned to filesystem cephfs as rank 0 2024-04-23T10:24:44.333 INFO:journalctl@ceph.mon.a.smithi114.stdout:Apr 23 10:24:44 smithi114 ceph-mon[29255]: from='client.? 172.21.15.114:0/1930247895' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "cephfs_data", "pg_num": 64, "pg_num_min": 64, "target_size_ratio": 0.9}]': finished 2024-04-23T10:24:44.333 INFO:journalctl@ceph.mon.a.smithi114.stdout:Apr 23 10:24:44 smithi114 ceph-mon[29255]: osdmap e85: 12 total, 12 up, 12 in 2024-04-23T10:24:44.333 INFO:journalctl@ceph.mon.a.smithi114.stdout:Apr 23 10:24:44 smithi114 ceph-mon[29255]: from='client.? 172.21.15.114:0/2263953334' entity='client.admin' cmd=[{"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs_metadata", "data": "cephfs_data"}]: dispatch 2024-04-23T10:24:44.333 INFO:journalctl@ceph.mon.a.smithi114.stdout:Apr 23 10:24:44 smithi114 ceph-mon[29255]: daemon mds.b assigned to filesystem cephfs as rank 0 2024-04-23T10:24:44.334 DEBUG:teuthology.orchestra.run.smithi114:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-04-23T10:24:44.439 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:24:44.438+0000 7f1b3fa20640 1 -- 172.21.15.114:0/1059191291 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f1b38107d30 msgr2=0x7f1b3810a120 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T10:24:44.440 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:24:44.438+0000 7f1b3fa20640 1 --2- 172.21.15.114:0/1059191291 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f1b38107d30 0x7f1b3810a120 secure :-1 s=READY pgs=210 cs=0 l=1 rev1=1 crypto rx=0x7f1b30009ca0 tx=0x7f1b3002a8c0 comp rx=0 tx=0).stop 2024-04-23T10:24:44.440 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:24:44.438+0000 7f1b3fa20640 1 -- 172.21.15.114:0/1059191291 shutdown_connections 2024-04-23T10:24:44.440 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:24:44.438+0000 7f1b3fa20640 1 --2- 172.21.15.114:0/1059191291 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f1b38107d30 0x7f1b3810a120 unknown :-1 s=CLOSED pgs=210 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:24:44.440 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:24:44.438+0000 7f1b3fa20640 1 --2- 172.21.15.114:0/1059191291 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f1b38105400 0x7f1b381077f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:24:44.440 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:24:44.438+0000 7f1b3fa20640 1 --2- 172.21.15.114:0/1059191291 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f1b38102ad0 0x7f1b38104ec0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:24:44.440 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:24:44.438+0000 7f1b3fa20640 1 -- 172.21.15.114:0/1059191291 >> 172.21.15.114:0/1059191291 conn(0x7f1b380fc400 msgr2=0x7f1b380fe840 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T10:24:44.440 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:24:44.438+0000 7f1b3fa20640 1 -- 172.21.15.114:0/1059191291 shutdown_connections 2024-04-23T10:24:44.440 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:24:44.438+0000 7f1b3fa20640 1 -- 172.21.15.114:0/1059191291 wait complete. 2024-04-23T10:24:44.440 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:24:44.439+0000 7f1b3fa20640 1 Processor -- start 2024-04-23T10:24:44.440 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:24:44.439+0000 7f1b3fa20640 1 -- start start 2024-04-23T10:24:44.440 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:24:44.439+0000 7f1b3fa20640 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f1b38102ad0 0x7f1b381981e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T10:24:44.440 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:24:44.439+0000 7f1b3fa20640 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f1b38105400 0x7f1b38198720 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T10:24:44.440 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:24:44.439+0000 7f1b3fa20640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f1b38107d30 0x7f1b3819f790 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T10:24:44.440 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:24:44.439+0000 7f1b3fa20640 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x7f1b3810dae0 con 0x7f1b38107d30 2024-04-23T10:24:44.441 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:24:44.439+0000 7f1b3fa20640 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 -- 0x7f1b3810d960 con 0x7f1b38102ad0 2024-04-23T10:24:44.441 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:24:44.439+0000 7f1b3fa20640 1 -- --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_getmap magic: 0 -- 0x7f1b3810dc60 con 0x7f1b38105400 2024-04-23T10:24:44.441 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:24:44.439+0000 7f1b3dfbd640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f1b38107d30 0x7f1b3819f790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T10:24:44.441 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:24:44.439+0000 7f1b3dfbd640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f1b38107d30 0x7f1b3819f790 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.114:44520/0 (socket says 172.21.15.114:44520) 2024-04-23T10:24:44.441 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:24:44.439+0000 7f1b3dfbd640 1 -- 172.21.15.114:0/2509541380 learned_addr learned my addr 172.21.15.114:0/2509541380 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-04-23T10:24:44.441 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:24:44.439+0000 7f1b3d7bc640 1 --2- 172.21.15.114:0/2509541380 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f1b38102ad0 0x7f1b381981e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T10:24:44.441 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:24:44.439+0000 7f1b3cfbb640 1 --2- 172.21.15.114:0/2509541380 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f1b38105400 0x7f1b38198720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T10:24:44.441 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:24:44.439+0000 7f1b3dfbd640 1 -- 172.21.15.114:0/2509541380 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f1b38105400 msgr2=0x7f1b38198720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T10:24:44.441 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:24:44.439+0000 7f1b3dfbd640 1 --2- 172.21.15.114:0/2509541380 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f1b38105400 0x7f1b38198720 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:24:44.441 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:24:44.439+0000 7f1b3dfbd640 1 -- 172.21.15.114:0/2509541380 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f1b38102ad0 msgr2=0x7f1b381981e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T10:24:44.441 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:24:44.439+0000 7f1b3dfbd640 1 --2- 172.21.15.114:0/2509541380 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f1b38102ad0 0x7f1b381981e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:24:44.441 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:24:44.439+0000 7f1b3dfbd640 1 -- 172.21.15.114:0/2509541380 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f1b3819fe00 con 0x7f1b38107d30 2024-04-23T10:24:44.441 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:24:44.439+0000 7f1b3dfbd640 1 --2- 172.21.15.114:0/2509541380 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f1b38107d30 0x7f1b3819f790 secure :-1 s=READY pgs=211 cs=0 l=1 rev1=1 crypto rx=0x7f1b300097b0 tx=0x7f1b30003c20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T10:24:44.441 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:24:44.440+0000 7f1b2e7fc640 1 -- 172.21.15.114:0/2509541380 <== mon.0 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1b30004240 con 0x7f1b38107d30 2024-04-23T10:24:44.442 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:24:44.440+0000 7f1b2e7fc640 1 -- 172.21.15.114:0/2509541380 <== mon.0 v2:172.21.15.114:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f1b300317c0 con 0x7f1b38107d30 2024-04-23T10:24:44.442 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:24:44.440+0000 7f1b2e7fc640 1 -- 172.21.15.114:0/2509541380 <== mon.0 v2:172.21.15.114:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1b3002fd80 con 0x7f1b38107d30 2024-04-23T10:24:44.442 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:24:44.440+0000 7f1b3fa20640 1 -- 172.21.15.114:0/2509541380 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f1b381a0090 con 0x7f1b38107d30 2024-04-23T10:24:44.442 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:24:44.440+0000 7f1b3fa20640 1 -- 172.21.15.114:0/2509541380 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f1b381a05e0 con 0x7f1b38107d30 2024-04-23T10:24:44.442 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:24:44.441+0000 7f1b3fa20640 1 -- 172.21.15.114:0/2509541380 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f1b380fdef0 con 0x7f1b38107d30 2024-04-23T10:24:44.443 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:24:44.442+0000 7f1b2e7fc640 1 -- 172.21.15.114:0/2509541380 <== mon.0 v2:172.21.15.114:3300/0 4 ==== mgrmap(e 16) ==== 138647+0+0 (secure 0 0 0) 0x7f1b300043e0 con 0x7f1b38107d30 2024-04-23T10:24:44.443 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:24:44.442+0000 7f1b2e7fc640 1 --2- 172.21.15.114:0/2509541380 >> [v2:172.21.15.114:6800/465333808,v1:172.21.15.114:6801/465333808] conn(0x7f1b240a3bb0 0x7f1b240a6070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T10:24:44.443 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:24:44.442+0000 7f1b3d7bc640 1 --2- 172.21.15.114:0/2509541380 >> [v2:172.21.15.114:6800/465333808,v1:172.21.15.114:6801/465333808] conn(0x7f1b240a3bb0 0x7f1b240a6070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T10:24:44.443 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:24:44.442+0000 7f1b2e7fc640 1 -- 172.21.15.114:0/2509541380 <== mon.0 v2:172.21.15.114:3300/0 5 ==== osd_map(86..86 src has 1..86) ==== 7979+0+0 (secure 0 0 0) 0x7f1b300f1c00 con 0x7f1b38107d30 2024-04-23T10:24:44.443 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:24:44.443+0000 7f1b3d7bc640 1 --2- 172.21.15.114:0/2509541380 >> [v2:172.21.15.114:6800/465333808,v1:172.21.15.114:6801/465333808] conn(0x7f1b240a3bb0 0x7f1b240a6070 secure :-1 s=READY pgs=165 cs=0 l=1 rev1=1 crypto rx=0x7f1b34005fd0 tx=0x7f1b3400b390 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T10:24:44.444 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:24:44.446+0000 7f1b2e7fc640 1 -- 172.21.15.114:0/2509541380 <== mon.0 v2:172.21.15.114:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f1b300be850 con 0x7f1b38107d30 2024-04-23T10:24:44.492 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 23 10:24:44 smithi119 ceph-mon[32608]: from='client.? 172.21.15.114:0/1930247895' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "cephfs_data", "pg_num": 64, "pg_num_min": 64, "target_size_ratio": 0.9}]': finished 2024-04-23T10:24:44.492 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 23 10:24:44 smithi119 ceph-mon[32608]: osdmap e85: 12 total, 12 up, 12 in 2024-04-23T10:24:44.492 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 23 10:24:44 smithi119 ceph-mon[32608]: from='client.? 172.21.15.114:0/2263953334' entity='client.admin' cmd=[{"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs_metadata", "data": "cephfs_data"}]: dispatch 2024-04-23T10:24:44.492 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 23 10:24:44 smithi119 ceph-mon[32608]: daemon mds.b assigned to filesystem cephfs as rank 0 2024-04-23T10:24:44.625 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:24:44.626+0000 7f1b3fa20640 1 -- 172.21.15.114:0/2509541380 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f1b38062b50 con 0x7f1b38107d30 2024-04-23T10:24:44.626 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:24:44.628+0000 7f1b2e7fc640 1 -- 172.21.15.114:0/2509541380 <== mon.0 v2:172.21.15.114:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v86) ==== 74+0+22304 (secure 0 0 0) 0x7f1b300be150 con 0x7f1b38107d30 2024-04-23T10:24:44.627 INFO:teuthology.orchestra.run.smithi114.stdout: 2024-04-23T10:24:44.627 INFO:teuthology.orchestra.run.smithi114.stdout:{"epoch":86,"fsid":"aa240572-0159-11ef-bc93-c7b262605968","created":"2024-04-23T10:16:53.351136+0000","modified":"2024-04-23T10:24:43.383206+0000","last_up_change":"2024-04-23T10:23:54.153269+0000","last_in_change":"2024-04-23T10:23:32.963085+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":26,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-04-23T10:20:11.970297+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":12,"score_stable":12,"optimal_score":0.25,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-04-23T10:24:41.218540+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"86","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":2.25,"score_stable":2.25,"optimal_score":1,"raw_score_acting":2.25,"raw_score_stable":2.25,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-04-23T10:24:42.341799+0000","flags":32769,"flags_names":"hashpspool,creating","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"86","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"fd64adde-9445-45a4-a171-ef656e2c5709","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":85,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6802","nonce":736366878},{"type":"v1","addr":"172.21.15.114:6803","nonce":736366878}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6804","nonce":736366878},{"type":"v1","addr":"172.21.15.114:6805","nonce":736366878}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6808","nonce":736366878},{"type":"v1","addr":"172.21.15.114:6809","nonce":736366878}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6806","nonce":736366878},{"type":"v1","addr":"172.21.15.114:6807","nonce":736366878}]},"public_addr":"172.21.15.114:6803/736366878","cluster_addr":"172.21.15.114:6805/736366878","heartbeat_back_addr":"172.21.15.114:6809/736366878","heartbeat_front_addr":"172.21.15.114:6807/736366878","state":["exists","up"]},{"osd":1,"uuid":"e279a1e6-2702-472a-930a-e08ece76a861","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":85,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6800","nonce":1133540779},{"type":"v1","addr":"172.21.15.119:6801","nonce":1133540779}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6802","nonce":1133540779},{"type":"v1","addr":"172.21.15.119:6803","nonce":1133540779}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6806","nonce":1133540779},{"type":"v1","addr":"172.21.15.119:6807","nonce":1133540779}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6804","nonce":1133540779},{"type":"v1","addr":"172.21.15.119:6805","nonce":1133540779}]},"public_addr":"172.21.15.119:6801/1133540779","cluster_addr":"172.21.15.119:6803/1133540779","heartbeat_back_addr":"172.21.15.119:6807/1133540779","heartbeat_front_addr":"172.21.15.119:6805/1133540779","state":["exists","up"]},{"osd":2,"uuid":"f4c9f2b4-ff49-4e14-89cd-b477feeb9382","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":85,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6800","nonce":799652511},{"type":"v1","addr":"172.21.15.204:6801","nonce":799652511}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6802","nonce":799652511},{"type":"v1","addr":"172.21.15.204:6803","nonce":799652511}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6806","nonce":799652511},{"type":"v1","addr":"172.21.15.204:6807","nonce":799652511}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6804","nonce":799652511},{"type":"v1","addr":"172.21.15.204:6805","nonce":799652511}]},"public_addr":"172.21.15.204:6801/799652511","cluster_addr":"172.21.15.204:6803/799652511","heartbeat_back_addr":"172.21.15.204:6807/799652511","heartbeat_front_addr":"172.21.15.204:6805/799652511","state":["exists","up"]},{"osd":3,"uuid":"52d2a566-e71c-4bb3-8d6c-5abdd43177c2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":25,"up_thru":85,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6810","nonce":4275595091},{"type":"v1","addr":"172.21.15.114:6811","nonce":4275595091}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6812","nonce":4275595091},{"type":"v1","addr":"172.21.15.114:6813","nonce":4275595091}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6816","nonce":4275595091},{"type":"v1","addr":"172.21.15.114:6817","nonce":4275595091}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6814","nonce":4275595091},{"type":"v1","addr":"172.21.15.114:6815","nonce":4275595091}]},"public_addr":"172.21.15.114:6811/4275595091","cluster_addr":"172.21.15.114:6813/4275595091","heartbeat_back_addr":"172.21.15.114:6817/4275595091","heartbeat_front_addr":"172.21.15.114:6815/4275595091","state":["exists","up"]},{"osd":4,"uuid":"d911a7c1-4bdc-4d28-b8a6-73896a6e7de6","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":33,"up_thru":85,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6808","nonce":1611007172},{"type":"v1","addr":"172.21.15.119:6809","nonce":1611007172}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6810","nonce":1611007172},{"type":"v1","addr":"172.21.15.119:6811","nonce":1611007172}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6814","nonce":1611007172},{"type":"v1","addr":"172.21.15.119:6815","nonce":1611007172}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6812","nonce":1611007172},{"type":"v1","addr":"172.21.15.119:6813","nonce":1611007172}]},"public_addr":"172.21.15.119:6809/1611007172","cluster_addr":"172.21.15.119:6811/1611007172","heartbeat_back_addr":"172.21.15.119:6815/1611007172","heartbeat_front_addr":"172.21.15.119:6813/1611007172","state":["exists","up"]},{"osd":5,"uuid":"f7590e1a-e93c-4c29-b15f-38ee52d1f921","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":39,"up_thru":85,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6808","nonce":1427563312},{"type":"v1","addr":"172.21.15.204:6809","nonce":1427563312}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6810","nonce":1427563312},{"type":"v1","addr":"172.21.15.204:6811","nonce":1427563312}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6814","nonce":1427563312},{"type":"v1","addr":"172.21.15.204:6815","nonce":1427563312}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6812","nonce":1427563312},{"type":"v1","addr":"172.21.15.204:6813","nonce":1427563312}]},"public_addr":"172.21.15.204:6809/1427563312","cluster_addr":"172.21.15.204:6811/1427563312","heartbeat_back_addr":"172.21.15.204:6815/1427563312","heartbeat_front_addr":"172.21.15.204:6813/1427563312","state":["exists","up"]},{"osd":6,"uuid":"534d20e8-3e54-4db8-a845-6119f49c7a78","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":47,"up_thru":85,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6818","nonce":4072912544},{"type":"v1","addr":"172.21.15.114:6819","nonce":4072912544}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6820","nonce":4072912544},{"type":"v1","addr":"172.21.15.114:6821","nonce":4072912544}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6824","nonce":4072912544},{"type":"v1","addr":"172.21.15.114:6825","nonce":4072912544}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6822","nonce":4072912544},{"type":"v1","addr":"172.21.15.114:6823","nonce":4072912544}]},"public_addr":"172.21.15.114:6819/4072912544","cluster_addr":"172.21.15.114:6821/4072912544","heartbeat_back_addr":"172.21.15.114:6825/4072912544","heartbeat_front_addr":"172.21.15.114:6823/4072912544","state":["exists","up"]},{"osd":7,"uuid":"f0ec22b8-1b48-4859-9d65-b00403aff3ad","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":53,"up_thru":85,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6816","nonce":2237458793},{"type":"v1","addr":"172.21.15.119:6817","nonce":2237458793}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6818","nonce":2237458793},{"type":"v1","addr":"172.21.15.119:6819","nonce":2237458793}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6822","nonce":2237458793},{"type":"v1","addr":"172.21.15.119:6823","nonce":2237458793}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6820","nonce":2237458793},{"type":"v1","addr":"172.21.15.119:6821","nonce":2237458793}]},"public_addr":"172.21.15.119:6817/2237458793","cluster_addr":"172.21.15.119:6819/2237458793","heartbeat_back_addr":"172.21.15.119:6823/2237458793","heartbeat_front_addr":"172.21.15.119:6821/2237458793","state":["exists","up"]},{"osd":8,"uuid":"3ac89686-6d76-4cef-9089-ac29e4b3bcac","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":62,"up_thru":85,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6816","nonce":1426318231},{"type":"v1","addr":"172.21.15.204:6817","nonce":1426318231}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6818","nonce":1426318231},{"type":"v1","addr":"172.21.15.204:6819","nonce":1426318231}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6822","nonce":1426318231},{"type":"v1","addr":"172.21.15.204:6823","nonce":1426318231}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6820","nonce":1426318231},{"type":"v1","addr":"172.21.15.204:6821","nonce":1426318231}]},"public_addr":"172.21.15.204:6817/1426318231","cluster_addr":"172.21.15.204:6819/1426318231","heartbeat_back_addr":"172.21.15.204:6823/1426318231","heartbeat_front_addr":"172.21.15.204:6821/1426318231","state":["exists","up"]},{"osd":9,"uuid":"a80ffe4d-1e98-460f-b046-c4754285cc5a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":69,"up_thru":85,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6826","nonce":2750635721},{"type":"v1","addr":"172.21.15.114:6827","nonce":2750635721}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6828","nonce":2750635721},{"type":"v1","addr":"172.21.15.114:6829","nonce":2750635721}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6832","nonce":2750635721},{"type":"v1","addr":"172.21.15.114:6833","nonce":2750635721}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6830","nonce":2750635721},{"type":"v1","addr":"172.21.15.114:6831","nonce":2750635721}]},"public_addr":"172.21.15.114:6827/2750635721","cluster_addr":"172.21.15.114:6829/2750635721","heartbeat_back_addr":"172.21.15.114:6833/2750635721","heartbeat_front_addr":"172.21.15.114:6831/2750635721","state":["exists","up"]},{"osd":10,"uuid":"cd75a579-bf12-4600-ac09-a8bfce772ced","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":74,"up_thru":85,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6824","nonce":1400187701},{"type":"v1","addr":"172.21.15.119:6825","nonce":1400187701}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6826","nonce":1400187701},{"type":"v1","addr":"172.21.15.119:6827","nonce":1400187701}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6830","nonce":1400187701},{"type":"v1","addr":"172.21.15.119:6831","nonce":1400187701}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6828","nonce":1400187701},{"type":"v1","addr":"172.21.15.119:6829","nonce":1400187701}]},"public_addr":"172.21.15.119:6825/1400187701","cluster_addr":"172.21.15.119:6827/1400187701","heartbeat_back_addr":"172.21.15.119:6831/1400187701","heartbeat_front_addr":"172.21.15.119:6829/1400187701","state":["exists","up"]},{"osd":11,"uuid":"53f037a5-3ec0-483a-80a1-dfee923a68e5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":82,"up_thru":85,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6824","nonce":2407298490},{"type":"v1","addr":"172.21.15.204:6825","nonce":2407298490}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6826","nonce":2407298490},{"type":"v1","addr":"172.21.15.204:6827","nonce":2407298490}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6830","nonce":2407298490},{"type":"v1","addr":"172.21.15.204:6831","nonce":2407298490}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6828","nonce":2407298490},{"type":"v1","addr":"172.21.15.204:6829","nonce":2407298490}]},"public_addr":"172.21.15.204:6825/2407298490","cluster_addr":"172.21.15.204:6827/2407298490","heartbeat_back_addr":"172.21.15.204:6831/2407298490","heartbeat_front_addr":"172.21.15.204:6829/2407298490","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-23T10:19:18.401147+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-23T10:19:42.611299+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-23T10:20:07.277367+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-23T10:20:31.208368+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-23T10:20:55.866225+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-23T10:21:19.961237+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-23T10:21:44.480783+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-23T10:22:09.173641+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-23T10:22:34.030328+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-23T10:22:59.652293+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-23T10:23:25.659673+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-23T10:23:51.168171+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.114:6801/1604678557":"2024-04-24T10:17:45.874609+0000","172.21.15.114:0/4025385854":"2024-04-24T10:17:45.874609+0000","172.21.15.114:0/2228273469":"2024-04-24T10:17:19.910348+0000","172.21.15.114:6800/2831530838":"2024-04-24T10:17:19.910348+0000","172.21.15.114:0/1333447450":"2024-04-24T10:17:45.874609+0000","172.21.15.114:0/2233026840":"2024-04-24T10:17:19.910348+0000","172.21.15.114:0/1902281545":"2024-04-24T10:17:19.910348+0000","172.21.15.114:6801/2831530838":"2024-04-24T10:17:19.910348+0000","172.21.15.114:6800/1604678557":"2024-04-24T10:17:45.874609+0000","172.21.15.114:0/1623553811":"2024-04-24T10:17:45.874609+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-23T10:24:44.629 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:24:44.630+0000 7f1b3fa20640 1 -- 172.21.15.114:0/2509541380 >> [v2:172.21.15.114:6800/465333808,v1:172.21.15.114:6801/465333808] conn(0x7f1b240a3bb0 msgr2=0x7f1b240a6070 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T10:24:44.629 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:24:44.630+0000 7f1b3fa20640 1 --2- 172.21.15.114:0/2509541380 >> [v2:172.21.15.114:6800/465333808,v1:172.21.15.114:6801/465333808] conn(0x7f1b240a3bb0 0x7f1b240a6070 secure :-1 s=READY pgs=165 cs=0 l=1 rev1=1 crypto rx=0x7f1b34005fd0 tx=0x7f1b3400b390 comp rx=0 tx=0).stop 2024-04-23T10:24:44.629 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:24:44.630+0000 7f1b3fa20640 1 -- 172.21.15.114:0/2509541380 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f1b38107d30 msgr2=0x7f1b3819f790 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T10:24:44.629 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:24:44.630+0000 7f1b3fa20640 1 --2- 172.21.15.114:0/2509541380 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f1b38107d30 0x7f1b3819f790 secure :-1 s=READY pgs=211 cs=0 l=1 rev1=1 crypto rx=0x7f1b300097b0 tx=0x7f1b30003c20 comp rx=0 tx=0).stop 2024-04-23T10:24:44.629 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:24:44.630+0000 7f1b3fa20640 1 -- 172.21.15.114:0/2509541380 shutdown_connections 2024-04-23T10:24:44.629 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:24:44.630+0000 7f1b3fa20640 1 --2- 172.21.15.114:0/2509541380 >> [v2:172.21.15.114:6800/465333808,v1:172.21.15.114:6801/465333808] conn(0x7f1b240a3bb0 0x7f1b240a6070 unknown :-1 s=CLOSED pgs=165 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:24:44.629 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:24:44.630+0000 7f1b3fa20640 1 --2- 172.21.15.114:0/2509541380 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f1b38107d30 0x7f1b3819f790 unknown :-1 s=CLOSED pgs=211 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:24:44.629 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:24:44.630+0000 7f1b3fa20640 1 --2- 172.21.15.114:0/2509541380 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f1b38105400 0x7f1b38198720 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:24:44.629 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:24:44.630+0000 7f1b3fa20640 1 --2- 172.21.15.114:0/2509541380 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f1b38102ad0 0x7f1b381981e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:24:44.629 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:24:44.630+0000 7f1b3fa20640 1 -- 172.21.15.114:0/2509541380 >> 172.21.15.114:0/2509541380 conn(0x7f1b380fc400 msgr2=0x7f1b38103550 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T10:24:44.630 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:24:44.630+0000 7f1b3fa20640 1 -- 172.21.15.114:0/2509541380 shutdown_connections 2024-04-23T10:24:44.630 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:24:44.630+0000 7f1b3fa20640 1 -- 172.21.15.114:0/2509541380 wait complete. 2024-04-23T10:24:44.640 DEBUG:teuthology.orchestra.run.smithi114:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-04-23T10:24:44.737 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:24:44.738+0000 7f6a6cb20640 1 -- 172.21.15.114:0/1785267543 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f6a68101b80 msgr2=0x7f6a68102000 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T10:24:44.738 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:24:44.738+0000 7f6a6cb20640 1 --2- 172.21.15.114:0/1785267543 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f6a68101b80 0x7f6a68102000 secure :-1 s=READY pgs=212 cs=0 l=1 rev1=1 crypto rx=0x7f6a60009ca0 tx=0x7f6a6002a950 comp rx=0 tx=0).stop 2024-04-23T10:24:44.738 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:24:44.738+0000 7f6a6cb20640 1 -- 172.21.15.114:0/1785267543 shutdown_connections 2024-04-23T10:24:44.738 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:24:44.738+0000 7f6a6cb20640 1 --2- 172.21.15.114:0/1785267543 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f6a68101b80 0x7f6a68102000 unknown :-1 s=CLOSED pgs=212 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:24:44.738 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:24:44.738+0000 7f6a6cb20640 1 --2- 172.21.15.114:0/1785267543 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f6a681011e0 0x7f6a68101640 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:24:44.738 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:24:44.738+0000 7f6a6cb20640 1 --2- 172.21.15.114:0/1785267543 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f6a68100880 0x7f6a68100ca0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:24:44.738 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:24:44.738+0000 7f6a6cb20640 1 -- 172.21.15.114:0/1785267543 >> 172.21.15.114:0/1785267543 conn(0x7f6a680fc400 msgr2=0x7f6a680fe840 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T10:24:44.738 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:24:44.739+0000 7f6a6cb20640 1 -- 172.21.15.114:0/1785267543 shutdown_connections 2024-04-23T10:24:44.738 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:24:44.739+0000 7f6a6cb20640 1 -- 172.21.15.114:0/1785267543 wait complete. 2024-04-23T10:24:44.740 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:24:44.739+0000 7f6a6cb20640 1 Processor -- start 2024-04-23T10:24:44.740 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:24:44.739+0000 7f6a6cb20640 1 -- start start 2024-04-23T10:24:44.741 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:24:44.739+0000 7f6a6cb20640 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f6a68100880 0x7f6a6819a4f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T10:24:44.741 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:24:44.739+0000 7f6a6cb20640 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f6a681011e0 0x7f6a6819aa30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T10:24:44.741 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:24:44.739+0000 7f6a6cb20640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f6a68101b80 0x7f6a681a1aa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T10:24:44.741 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:24:44.739+0000 7f6a6cb20640 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x7f6a6810fd90 con 0x7f6a68101b80 2024-04-23T10:24:44.741 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:24:44.739+0000 7f6a6cb20640 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 -- 0x7f6a6810fc10 con 0x7f6a681011e0 2024-04-23T10:24:44.741 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:24:44.739+0000 7f6a6cb20640 1 -- --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_getmap magic: 0 -- 0x7f6a6810ff10 con 0x7f6a68100880 2024-04-23T10:24:44.741 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:24:44.740+0000 7f6a66d9d640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f6a68101b80 0x7f6a681a1aa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T10:24:44.741 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:24:44.740+0000 7f6a66d9d640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f6a68101b80 0x7f6a681a1aa0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.114:44536/0 (socket says 172.21.15.114:44536) 2024-04-23T10:24:44.741 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:24:44.740+0000 7f6a66d9d640 1 -- 172.21.15.114:0/3696231242 learned_addr learned my addr 172.21.15.114:0/3696231242 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-04-23T10:24:44.741 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:24:44.740+0000 7f6a65d9b640 1 --2- 172.21.15.114:0/3696231242 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f6a681011e0 0x7f6a6819aa30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T10:24:44.741 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:24:44.740+0000 7f6a6659c640 1 --2- 172.21.15.114:0/3696231242 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f6a68100880 0x7f6a6819a4f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T10:24:44.741 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:24:44.740+0000 7f6a66d9d640 1 -- 172.21.15.114:0/3696231242 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f6a68100880 msgr2=0x7f6a6819a4f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T10:24:44.741 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:24:44.740+0000 7f6a66d9d640 1 --2- 172.21.15.114:0/3696231242 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f6a68100880 0x7f6a6819a4f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:24:44.741 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:24:44.740+0000 7f6a66d9d640 1 -- 172.21.15.114:0/3696231242 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f6a681011e0 msgr2=0x7f6a6819aa30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T10:24:44.742 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:24:44.740+0000 7f6a66d9d640 1 --2- 172.21.15.114:0/3696231242 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f6a681011e0 0x7f6a6819aa30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:24:44.742 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:24:44.740+0000 7f6a66d9d640 1 -- 172.21.15.114:0/3696231242 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f6a681a2110 con 0x7f6a68101b80 2024-04-23T10:24:44.742 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:24:44.740+0000 7f6a6659c640 1 --2- 172.21.15.114:0/3696231242 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f6a68100880 0x7f6a6819a4f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-23T10:24:44.742 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:24:44.740+0000 7f6a66d9d640 1 --2- 172.21.15.114:0/3696231242 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f6a68101b80 0x7f6a681a1aa0 secure :-1 s=READY pgs=213 cs=0 l=1 rev1=1 crypto rx=0x7f6a60009dc0 tx=0x7f6a60031cb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T10:24:44.742 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:24:44.740+0000 7f6a577fe640 1 -- 172.21.15.114:0/3696231242 <== mon.0 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6a60034040 con 0x7f6a68101b80 2024-04-23T10:24:44.742 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:24:44.740+0000 7f6a6cb20640 1 -- 172.21.15.114:0/3696231242 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f6a681a23a0 con 0x7f6a68101b80 2024-04-23T10:24:44.742 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:24:44.740+0000 7f6a577fe640 1 -- 172.21.15.114:0/3696231242 <== mon.0 v2:172.21.15.114:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f6a60004040 con 0x7f6a68101b80 2024-04-23T10:24:44.742 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:24:44.740+0000 7f6a577fe640 1 -- 172.21.15.114:0/3696231242 <== mon.0 v2:172.21.15.114:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6a600456e0 con 0x7f6a68101b80 2024-04-23T10:24:44.742 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:24:44.740+0000 7f6a6cb20640 1 -- 172.21.15.114:0/3696231242 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f6a681a2960 con 0x7f6a68101b80 2024-04-23T10:24:44.742 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:24:44.741+0000 7f6a6cb20640 1 -- 172.21.15.114:0/3696231242 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f6a6810b860 con 0x7f6a68101b80 2024-04-23T10:24:44.743 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:24:44.742+0000 7f6a577fe640 1 -- 172.21.15.114:0/3696231242 <== mon.0 v2:172.21.15.114:3300/0 4 ==== mgrmap(e 16) ==== 138647+0+0 (secure 0 0 0) 0x7f6a60037020 con 0x7f6a68101b80 2024-04-23T10:24:44.743 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:24:44.743+0000 7f6a577fe640 1 --2- 172.21.15.114:0/3696231242 >> [v2:172.21.15.114:6800/465333808,v1:172.21.15.114:6801/465333808] conn(0x7f6a500a3bb0 0x7f6a500a6070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T10:24:44.743 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:24:44.743+0000 7f6a6659c640 1 --2- 172.21.15.114:0/3696231242 >> [v2:172.21.15.114:6800/465333808,v1:172.21.15.114:6801/465333808] conn(0x7f6a500a3bb0 0x7f6a500a6070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T10:24:44.743 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:24:44.743+0000 7f6a577fe640 1 -- 172.21.15.114:0/3696231242 <== mon.0 v2:172.21.15.114:3300/0 5 ==== osd_map(86..86 src has 1..86) ==== 7979+0+0 (secure 0 0 0) 0x7f6a600f1520 con 0x7f6a68101b80 2024-04-23T10:24:44.743 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:24:44.743+0000 7f6a6659c640 1 --2- 172.21.15.114:0/3696231242 >> [v2:172.21.15.114:6800/465333808,v1:172.21.15.114:6801/465333808] conn(0x7f6a500a3bb0 0x7f6a500a6070 secure :-1 s=READY pgs=166 cs=0 l=1 rev1=1 crypto rx=0x7f6a58009930 tx=0x7f6a58008040 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T10:24:44.745 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:24:44.746+0000 7f6a577fe640 1 -- 172.21.15.114:0/3696231242 <== mon.0 v2:172.21.15.114:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f6a6004d350 con 0x7f6a68101b80 2024-04-23T10:24:44.924 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:24:44.925+0000 7f6a6cb20640 1 -- 172.21.15.114:0/3696231242 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f6a68062b50 con 0x7f6a68101b80 2024-04-23T10:24:44.926 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:24:44.927+0000 7f6a577fe640 1 -- 172.21.15.114:0/3696231242 <== mon.0 v2:172.21.15.114:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v86) ==== 74+0+22304 (secure 0 0 0) 0x7f6a600be050 con 0x7f6a68101b80 2024-04-23T10:24:44.926 INFO:teuthology.orchestra.run.smithi114.stdout: 2024-04-23T10:24:44.926 INFO:teuthology.orchestra.run.smithi114.stdout:{"epoch":86,"fsid":"aa240572-0159-11ef-bc93-c7b262605968","created":"2024-04-23T10:16:53.351136+0000","modified":"2024-04-23T10:24:43.383206+0000","last_up_change":"2024-04-23T10:23:54.153269+0000","last_in_change":"2024-04-23T10:23:32.963085+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":26,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-04-23T10:20:11.970297+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":12,"score_stable":12,"optimal_score":0.25,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-04-23T10:24:41.218540+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"86","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":2.25,"score_stable":2.25,"optimal_score":1,"raw_score_acting":2.25,"raw_score_stable":2.25,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-04-23T10:24:42.341799+0000","flags":32769,"flags_names":"hashpspool,creating","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"86","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"fd64adde-9445-45a4-a171-ef656e2c5709","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":85,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6802","nonce":736366878},{"type":"v1","addr":"172.21.15.114:6803","nonce":736366878}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6804","nonce":736366878},{"type":"v1","addr":"172.21.15.114:6805","nonce":736366878}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6808","nonce":736366878},{"type":"v1","addr":"172.21.15.114:6809","nonce":736366878}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6806","nonce":736366878},{"type":"v1","addr":"172.21.15.114:6807","nonce":736366878}]},"public_addr":"172.21.15.114:6803/736366878","cluster_addr":"172.21.15.114:6805/736366878","heartbeat_back_addr":"172.21.15.114:6809/736366878","heartbeat_front_addr":"172.21.15.114:6807/736366878","state":["exists","up"]},{"osd":1,"uuid":"e279a1e6-2702-472a-930a-e08ece76a861","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":85,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6800","nonce":1133540779},{"type":"v1","addr":"172.21.15.119:6801","nonce":1133540779}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6802","nonce":1133540779},{"type":"v1","addr":"172.21.15.119:6803","nonce":1133540779}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6806","nonce":1133540779},{"type":"v1","addr":"172.21.15.119:6807","nonce":1133540779}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6804","nonce":1133540779},{"type":"v1","addr":"172.21.15.119:6805","nonce":1133540779}]},"public_addr":"172.21.15.119:6801/1133540779","cluster_addr":"172.21.15.119:6803/1133540779","heartbeat_back_addr":"172.21.15.119:6807/1133540779","heartbeat_front_addr":"172.21.15.119:6805/1133540779","state":["exists","up"]},{"osd":2,"uuid":"f4c9f2b4-ff49-4e14-89cd-b477feeb9382","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":85,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6800","nonce":799652511},{"type":"v1","addr":"172.21.15.204:6801","nonce":799652511}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6802","nonce":799652511},{"type":"v1","addr":"172.21.15.204:6803","nonce":799652511}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6806","nonce":799652511},{"type":"v1","addr":"172.21.15.204:6807","nonce":799652511}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6804","nonce":799652511},{"type":"v1","addr":"172.21.15.204:6805","nonce":799652511}]},"public_addr":"172.21.15.204:6801/799652511","cluster_addr":"172.21.15.204:6803/799652511","heartbeat_back_addr":"172.21.15.204:6807/799652511","heartbeat_front_addr":"172.21.15.204:6805/799652511","state":["exists","up"]},{"osd":3,"uuid":"52d2a566-e71c-4bb3-8d6c-5abdd43177c2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":25,"up_thru":85,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6810","nonce":4275595091},{"type":"v1","addr":"172.21.15.114:6811","nonce":4275595091}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6812","nonce":4275595091},{"type":"v1","addr":"172.21.15.114:6813","nonce":4275595091}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6816","nonce":4275595091},{"type":"v1","addr":"172.21.15.114:6817","nonce":4275595091}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6814","nonce":4275595091},{"type":"v1","addr":"172.21.15.114:6815","nonce":4275595091}]},"public_addr":"172.21.15.114:6811/4275595091","cluster_addr":"172.21.15.114:6813/4275595091","heartbeat_back_addr":"172.21.15.114:6817/4275595091","heartbeat_front_addr":"172.21.15.114:6815/4275595091","state":["exists","up"]},{"osd":4,"uuid":"d911a7c1-4bdc-4d28-b8a6-73896a6e7de6","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":33,"up_thru":85,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6808","nonce":1611007172},{"type":"v1","addr":"172.21.15.119:6809","nonce":1611007172}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6810","nonce":1611007172},{"type":"v1","addr":"172.21.15.119:6811","nonce":1611007172}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6814","nonce":1611007172},{"type":"v1","addr":"172.21.15.119:6815","nonce":1611007172}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6812","nonce":1611007172},{"type":"v1","addr":"172.21.15.119:6813","nonce":1611007172}]},"public_addr":"172.21.15.119:6809/1611007172","cluster_addr":"172.21.15.119:6811/1611007172","heartbeat_back_addr":"172.21.15.119:6815/1611007172","heartbeat_front_addr":"172.21.15.119:6813/1611007172","state":["exists","up"]},{"osd":5,"uuid":"f7590e1a-e93c-4c29-b15f-38ee52d1f921","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":39,"up_thru":85,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6808","nonce":1427563312},{"type":"v1","addr":"172.21.15.204:6809","nonce":1427563312}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6810","nonce":1427563312},{"type":"v1","addr":"172.21.15.204:6811","nonce":1427563312}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6814","nonce":1427563312},{"type":"v1","addr":"172.21.15.204:6815","nonce":1427563312}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6812","nonce":1427563312},{"type":"v1","addr":"172.21.15.204:6813","nonce":1427563312}]},"public_addr":"172.21.15.204:6809/1427563312","cluster_addr":"172.21.15.204:6811/1427563312","heartbeat_back_addr":"172.21.15.204:6815/1427563312","heartbeat_front_addr":"172.21.15.204:6813/1427563312","state":["exists","up"]},{"osd":6,"uuid":"534d20e8-3e54-4db8-a845-6119f49c7a78","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":47,"up_thru":85,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6818","nonce":4072912544},{"type":"v1","addr":"172.21.15.114:6819","nonce":4072912544}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6820","nonce":4072912544},{"type":"v1","addr":"172.21.15.114:6821","nonce":4072912544}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6824","nonce":4072912544},{"type":"v1","addr":"172.21.15.114:6825","nonce":4072912544}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6822","nonce":4072912544},{"type":"v1","addr":"172.21.15.114:6823","nonce":4072912544}]},"public_addr":"172.21.15.114:6819/4072912544","cluster_addr":"172.21.15.114:6821/4072912544","heartbeat_back_addr":"172.21.15.114:6825/4072912544","heartbeat_front_addr":"172.21.15.114:6823/4072912544","state":["exists","up"]},{"osd":7,"uuid":"f0ec22b8-1b48-4859-9d65-b00403aff3ad","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":53,"up_thru":85,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6816","nonce":2237458793},{"type":"v1","addr":"172.21.15.119:6817","nonce":2237458793}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6818","nonce":2237458793},{"type":"v1","addr":"172.21.15.119:6819","nonce":2237458793}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6822","nonce":2237458793},{"type":"v1","addr":"172.21.15.119:6823","nonce":2237458793}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6820","nonce":2237458793},{"type":"v1","addr":"172.21.15.119:6821","nonce":2237458793}]},"public_addr":"172.21.15.119:6817/2237458793","cluster_addr":"172.21.15.119:6819/2237458793","heartbeat_back_addr":"172.21.15.119:6823/2237458793","heartbeat_front_addr":"172.21.15.119:6821/2237458793","state":["exists","up"]},{"osd":8,"uuid":"3ac89686-6d76-4cef-9089-ac29e4b3bcac","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":62,"up_thru":85,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6816","nonce":1426318231},{"type":"v1","addr":"172.21.15.204:6817","nonce":1426318231}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6818","nonce":1426318231},{"type":"v1","addr":"172.21.15.204:6819","nonce":1426318231}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6822","nonce":1426318231},{"type":"v1","addr":"172.21.15.204:6823","nonce":1426318231}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6820","nonce":1426318231},{"type":"v1","addr":"172.21.15.204:6821","nonce":1426318231}]},"public_addr":"172.21.15.204:6817/1426318231","cluster_addr":"172.21.15.204:6819/1426318231","heartbeat_back_addr":"172.21.15.204:6823/1426318231","heartbeat_front_addr":"172.21.15.204:6821/1426318231","state":["exists","up"]},{"osd":9,"uuid":"a80ffe4d-1e98-460f-b046-c4754285cc5a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":69,"up_thru":85,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6826","nonce":2750635721},{"type":"v1","addr":"172.21.15.114:6827","nonce":2750635721}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6828","nonce":2750635721},{"type":"v1","addr":"172.21.15.114:6829","nonce":2750635721}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6832","nonce":2750635721},{"type":"v1","addr":"172.21.15.114:6833","nonce":2750635721}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6830","nonce":2750635721},{"type":"v1","addr":"172.21.15.114:6831","nonce":2750635721}]},"public_addr":"172.21.15.114:6827/2750635721","cluster_addr":"172.21.15.114:6829/2750635721","heartbeat_back_addr":"172.21.15.114:6833/2750635721","heartbeat_front_addr":"172.21.15.114:6831/2750635721","state":["exists","up"]},{"osd":10,"uuid":"cd75a579-bf12-4600-ac09-a8bfce772ced","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":74,"up_thru":85,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6824","nonce":1400187701},{"type":"v1","addr":"172.21.15.119:6825","nonce":1400187701}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6826","nonce":1400187701},{"type":"v1","addr":"172.21.15.119:6827","nonce":1400187701}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6830","nonce":1400187701},{"type":"v1","addr":"172.21.15.119:6831","nonce":1400187701}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6828","nonce":1400187701},{"type":"v1","addr":"172.21.15.119:6829","nonce":1400187701}]},"public_addr":"172.21.15.119:6825/1400187701","cluster_addr":"172.21.15.119:6827/1400187701","heartbeat_back_addr":"172.21.15.119:6831/1400187701","heartbeat_front_addr":"172.21.15.119:6829/1400187701","state":["exists","up"]},{"osd":11,"uuid":"53f037a5-3ec0-483a-80a1-dfee923a68e5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":82,"up_thru":85,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6824","nonce":2407298490},{"type":"v1","addr":"172.21.15.204:6825","nonce":2407298490}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6826","nonce":2407298490},{"type":"v1","addr":"172.21.15.204:6827","nonce":2407298490}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6830","nonce":2407298490},{"type":"v1","addr":"172.21.15.204:6831","nonce":2407298490}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6828","nonce":2407298490},{"type":"v1","addr":"172.21.15.204:6829","nonce":2407298490}]},"public_addr":"172.21.15.204:6825/2407298490","cluster_addr":"172.21.15.204:6827/2407298490","heartbeat_back_addr":"172.21.15.204:6831/2407298490","heartbeat_front_addr":"172.21.15.204:6829/2407298490","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-23T10:19:18.401147+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-23T10:19:42.611299+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-23T10:20:07.277367+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-23T10:20:31.208368+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-23T10:20:55.866225+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-23T10:21:19.961237+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-23T10:21:44.480783+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-23T10:22:09.173641+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-23T10:22:34.030328+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-23T10:22:59.652293+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-23T10:23:25.659673+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-23T10:23:51.168171+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.114:6801/1604678557":"2024-04-24T10:17:45.874609+0000","172.21.15.114:0/4025385854":"2024-04-24T10:17:45.874609+0000","172.21.15.114:0/2228273469":"2024-04-24T10:17:19.910348+0000","172.21.15.114:6800/2831530838":"2024-04-24T10:17:19.910348+0000","172.21.15.114:0/1333447450":"2024-04-24T10:17:45.874609+0000","172.21.15.114:0/2233026840":"2024-04-24T10:17:19.910348+0000","172.21.15.114:0/1902281545":"2024-04-24T10:17:19.910348+0000","172.21.15.114:6801/2831530838":"2024-04-24T10:17:19.910348+0000","172.21.15.114:6800/1604678557":"2024-04-24T10:17:45.874609+0000","172.21.15.114:0/1623553811":"2024-04-24T10:17:45.874609+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-23T10:24:44.928 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:24:44.929+0000 7f6a6cb20640 1 -- 172.21.15.114:0/3696231242 >> [v2:172.21.15.114:6800/465333808,v1:172.21.15.114:6801/465333808] conn(0x7f6a500a3bb0 msgr2=0x7f6a500a6070 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T10:24:44.928 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:24:44.929+0000 7f6a6cb20640 1 --2- 172.21.15.114:0/3696231242 >> [v2:172.21.15.114:6800/465333808,v1:172.21.15.114:6801/465333808] conn(0x7f6a500a3bb0 0x7f6a500a6070 secure :-1 s=READY pgs=166 cs=0 l=1 rev1=1 crypto rx=0x7f6a58009930 tx=0x7f6a58008040 comp rx=0 tx=0).stop 2024-04-23T10:24:44.928 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:24:44.929+0000 7f6a6cb20640 1 -- 172.21.15.114:0/3696231242 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f6a68101b80 msgr2=0x7f6a681a1aa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T10:24:44.929 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:24:44.929+0000 7f6a6cb20640 1 --2- 172.21.15.114:0/3696231242 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f6a68101b80 0x7f6a681a1aa0 secure :-1 s=READY pgs=213 cs=0 l=1 rev1=1 crypto rx=0x7f6a60009dc0 tx=0x7f6a60031cb0 comp rx=0 tx=0).stop 2024-04-23T10:24:44.929 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:24:44.929+0000 7f6a6cb20640 1 -- 172.21.15.114:0/3696231242 shutdown_connections 2024-04-23T10:24:44.929 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:24:44.929+0000 7f6a6cb20640 1 --2- 172.21.15.114:0/3696231242 >> [v2:172.21.15.114:6800/465333808,v1:172.21.15.114:6801/465333808] conn(0x7f6a500a3bb0 0x7f6a500a6070 unknown :-1 s=CLOSED pgs=166 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:24:44.929 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:24:44.929+0000 7f6a6cb20640 1 --2- 172.21.15.114:0/3696231242 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f6a68101b80 0x7f6a681a1aa0 unknown :-1 s=CLOSED pgs=213 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:24:44.929 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:24:44.929+0000 7f6a6cb20640 1 --2- 172.21.15.114:0/3696231242 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f6a681011e0 0x7f6a6819aa30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:24:44.929 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:24:44.929+0000 7f6a6cb20640 1 --2- 172.21.15.114:0/3696231242 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f6a68100880 0x7f6a6819a4f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:24:44.929 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:24:44.929+0000 7f6a6cb20640 1 -- 172.21.15.114:0/3696231242 >> 172.21.15.114:0/3696231242 conn(0x7f6a680fc400 msgr2=0x7f6a680fdbe0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T10:24:44.929 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:24:44.929+0000 7f6a6cb20640 1 -- 172.21.15.114:0/3696231242 shutdown_connections 2024-04-23T10:24:44.929 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:24:44.929+0000 7f6a6cb20640 1 -- 172.21.15.114:0/3696231242 wait complete. 2024-04-23T10:24:44.939 DEBUG:teuthology.orchestra.run.smithi114:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs set cephfs standby_count_wanted 0 2024-04-23T10:24:45.044 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:24:45.044+0000 7f1b39787640 1 -- 172.21.15.114:0/2455671584 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f1b34100610 msgr2=0x7f1b34100a30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T10:24:45.045 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:24:45.044+0000 7f1b39787640 1 --2- 172.21.15.114:0/2455671584 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f1b34100610 0x7f1b34100a30 secure :-1 s=READY pgs=214 cs=0 l=1 rev1=1 crypto rx=0x7f1b24009900 tx=0x7f1b2402a410 comp rx=0 tx=0).stop 2024-04-23T10:24:45.045 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:24:45.044+0000 7f1b39787640 1 -- 172.21.15.114:0/2455671584 shutdown_connections 2024-04-23T10:24:45.045 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:24:45.044+0000 7f1b39787640 1 --2- 172.21.15.114:0/2455671584 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f1b341021c0 0x7f1b34108a50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:24:45.045 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:24:45.044+0000 7f1b39787640 1 --2- 172.21.15.114:0/2455671584 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f1b34101800 0x7f1b34101c80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:24:45.045 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:24:45.044+0000 7f1b39787640 1 --2- 172.21.15.114:0/2455671584 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f1b34100610 0x7f1b34100a30 unknown :-1 s=CLOSED pgs=214 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:24:45.045 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:24:45.044+0000 7f1b39787640 1 -- 172.21.15.114:0/2455671584 >> 172.21.15.114:0/2455671584 conn(0x7f1b340fbb80 msgr2=0x7f1b340fdfc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T10:24:45.045 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:24:45.044+0000 7f1b39787640 1 -- 172.21.15.114:0/2455671584 shutdown_connections 2024-04-23T10:24:45.045 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:24:45.044+0000 7f1b39787640 1 -- 172.21.15.114:0/2455671584 wait complete. 2024-04-23T10:24:45.045 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:24:45.044+0000 7f1b39787640 1 Processor -- start 2024-04-23T10:24:45.045 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:24:45.044+0000 7f1b39787640 1 -- start start 2024-04-23T10:24:45.045 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:24:45.045+0000 7f1b39787640 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f1b34100610 0x7f1b34198250 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T10:24:45.045 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:24:45.045+0000 7f1b39787640 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f1b34101800 0x7f1b34198790 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T10:24:45.045 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:24:45.045+0000 7f1b39787640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f1b341021c0 0x7f1b3419f800 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T10:24:45.045 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:24:45.045+0000 7f1b39787640 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x7f1b3410db10 con 0x7f1b341021c0 2024-04-23T10:24:45.045 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:24:45.045+0000 7f1b39787640 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 -- 0x7f1b3410d990 con 0x7f1b34101800 2024-04-23T10:24:45.046 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:24:45.045+0000 7f1b39787640 1 -- --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_getmap magic: 0 -- 0x7f1b3410dc90 con 0x7f1b34100610 2024-04-23T10:24:45.046 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:24:45.045+0000 7f1b337fe640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f1b341021c0 0x7f1b3419f800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T10:24:45.046 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:24:45.045+0000 7f1b337fe640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f1b341021c0 0x7f1b3419f800 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.114:44540/0 (socket says 172.21.15.114:44540) 2024-04-23T10:24:45.046 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:24:45.045+0000 7f1b337fe640 1 -- 172.21.15.114:0/3723116402 learned_addr learned my addr 172.21.15.114:0/3723116402 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-04-23T10:24:45.046 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:24:45.045+0000 7f1b32ffd640 1 --2- 172.21.15.114:0/3723116402 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f1b34100610 0x7f1b34198250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T10:24:45.046 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:24:45.045+0000 7f1b337fe640 1 -- 172.21.15.114:0/3723116402 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f1b34100610 msgr2=0x7f1b34198250 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T10:24:45.046 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:24:45.045+0000 7f1b337fe640 1 --2- 172.21.15.114:0/3723116402 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f1b34100610 0x7f1b34198250 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:24:45.046 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:24:45.045+0000 7f1b337fe640 1 -- 172.21.15.114:0/3723116402 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f1b34101800 msgr2=0x7f1b34198790 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T10:24:45.046 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:24:45.045+0000 7f1b337fe640 1 --2- 172.21.15.114:0/3723116402 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f1b34101800 0x7f1b34198790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:24:45.046 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:24:45.045+0000 7f1b337fe640 1 -- 172.21.15.114:0/3723116402 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f1b3419fe70 con 0x7f1b341021c0 2024-04-23T10:24:45.046 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:24:45.045+0000 7f1b337fe640 1 --2- 172.21.15.114:0/3723116402 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f1b341021c0 0x7f1b3419f800 secure :-1 s=READY pgs=215 cs=0 l=1 rev1=1 crypto rx=0x7f1b2c00eb10 tx=0x7f1b2c00c2d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T10:24:45.046 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:24:45.045+0000 7f1b30ff9640 1 -- 172.21.15.114:0/3723116402 <== mon.0 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1b2c004b20 con 0x7f1b341021c0 2024-04-23T10:24:45.046 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:24:45.045+0000 7f1b30ff9640 1 -- 172.21.15.114:0/3723116402 <== mon.0 v2:172.21.15.114:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f1b2c004cc0 con 0x7f1b341021c0 2024-04-23T10:24:45.046 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:24:45.045+0000 7f1b39787640 1 -- 172.21.15.114:0/3723116402 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f1b341a0160 con 0x7f1b341021c0 2024-04-23T10:24:45.046 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:24:45.045+0000 7f1b30ff9640 1 -- 172.21.15.114:0/3723116402 <== mon.0 v2:172.21.15.114:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1b2c010670 con 0x7f1b341021c0 2024-04-23T10:24:45.046 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:24:45.045+0000 7f1b39787640 1 -- 172.21.15.114:0/3723116402 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f1b341a0660 con 0x7f1b341021c0 2024-04-23T10:24:45.047 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:24:45.046+0000 7f1b39787640 1 -- 172.21.15.114:0/3723116402 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f1b34107a50 con 0x7f1b341021c0 2024-04-23T10:24:45.047 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:24:45.047+0000 7f1b30ff9640 1 -- 172.21.15.114:0/3723116402 <== mon.0 v2:172.21.15.114:3300/0 4 ==== mgrmap(e 16) ==== 138647+0+0 (secure 0 0 0) 0x7f1b2c005020 con 0x7f1b341021c0 2024-04-23T10:24:45.047 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:24:45.048+0000 7f1b30ff9640 1 --2- 172.21.15.114:0/3723116402 >> [v2:172.21.15.114:6800/465333808,v1:172.21.15.114:6801/465333808] conn(0x7f1b200a3bb0 0x7f1b200a6070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T10:24:45.048 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:24:45.048+0000 7f1b32ffd640 1 --2- 172.21.15.114:0/3723116402 >> [v2:172.21.15.114:6800/465333808,v1:172.21.15.114:6801/465333808] conn(0x7f1b200a3bb0 0x7f1b200a6070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T10:24:45.048 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:24:45.048+0000 7f1b30ff9640 1 -- 172.21.15.114:0/3723116402 <== mon.0 v2:172.21.15.114:3300/0 5 ==== osd_map(87..87 src has 1..87) ==== 7979+0+0 (secure 0 0 0) 0x7f1b2c0d0910 con 0x7f1b341021c0 2024-04-23T10:24:45.048 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:24:45.048+0000 7f1b32ffd640 1 --2- 172.21.15.114:0/3723116402 >> [v2:172.21.15.114:6800/465333808,v1:172.21.15.114:6801/465333808] conn(0x7f1b200a3bb0 0x7f1b200a6070 secure :-1 s=READY pgs=167 cs=0 l=1 rev1=1 crypto rx=0x7f1b24009230 tx=0x7f1b240314a0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T10:24:45.051 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:24:45.053+0000 7f1b30ff9640 1 -- 172.21.15.114:0/3723116402 <== mon.0 v2:172.21.15.114:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f1b2c09d440 con 0x7f1b341021c0 2024-04-23T10:24:45.254 INFO:journalctl@ceph.mon.a.smithi114.stdout:Apr 23 10:24:45 smithi114 ceph-mon[29255]: pgmap v284: 97 pgs: 23 creating+peering, 68 unknown, 6 active+clean; 577 KiB data, 365 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-23T10:24:45.254 INFO:journalctl@ceph.mon.a.smithi114.stdout:Apr 23 10:24:45 smithi114 ceph-mon[29255]: osdmap e86: 12 total, 12 up, 12 in 2024-04-23T10:24:45.254 INFO:journalctl@ceph.mon.a.smithi114.stdout:Apr 23 10:24:45 smithi114 ceph-mon[29255]: Health check failed: 2 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-04-23T10:24:45.254 INFO:journalctl@ceph.mon.a.smithi114.stdout:Apr 23 10:24:45 smithi114 ceph-mon[29255]: from='client.? 172.21.15.114:0/2263953334' entity='client.admin' cmd='[{"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs_metadata", "data": "cephfs_data"}]': finished 2024-04-23T10:24:45.254 INFO:journalctl@ceph.mon.a.smithi114.stdout:Apr 23 10:24:45 smithi114 ceph-mon[29255]: fsmap cephfs:1 {0=b=up:creating} 11 up:standby 2024-04-23T10:24:45.254 INFO:journalctl@ceph.mon.a.smithi114.stdout:Apr 23 10:24:45 smithi114 ceph-mon[29255]: daemon mds.b is now active in filesystem cephfs as rank 0 2024-04-23T10:24:45.254 INFO:journalctl@ceph.mon.a.smithi114.stdout:Apr 23 10:24:45 smithi114 ceph-mon[29255]: from='client.? 172.21.15.114:0/2509541380' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-23T10:24:45.254 INFO:journalctl@ceph.mon.a.smithi114.stdout:Apr 23 10:24:45 smithi114 ceph-mon[29255]: from='client.? 172.21.15.114:0/3696231242' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-23T10:24:45.266 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:24:45.268+0000 7f1b39787640 1 -- 172.21.15.114:0/3723116402 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_command({"prefix": "fs set", "fs_name": "cephfs", "var": "standby_count_wanted", "val": "0"} v 0) -- 0x7f1b3402d810 con 0x7f1b341021c0 2024-04-23T10:24:45.321 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:24:45.322+0000 7f1b30ff9640 1 -- 172.21.15.114:0/3723116402 <== mon.0 v2:172.21.15.114:3300/0 7 ==== mon_command_ack([{"prefix": "fs set", "fs_name": "cephfs", "var": "standby_count_wanted", "val": "0"}]=0 v15) ==== 118+0+0 (secure 0 0 0) 0x7f1b2c09cd40 con 0x7f1b341021c0 2024-04-23T10:24:45.325 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:24:45.325+0000 7f1b39787640 1 -- 172.21.15.114:0/3723116402 >> [v2:172.21.15.114:6800/465333808,v1:172.21.15.114:6801/465333808] conn(0x7f1b200a3bb0 msgr2=0x7f1b200a6070 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T10:24:45.325 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:24:45.325+0000 7f1b39787640 1 --2- 172.21.15.114:0/3723116402 >> [v2:172.21.15.114:6800/465333808,v1:172.21.15.114:6801/465333808] conn(0x7f1b200a3bb0 0x7f1b200a6070 secure :-1 s=READY pgs=167 cs=0 l=1 rev1=1 crypto rx=0x7f1b24009230 tx=0x7f1b240314a0 comp rx=0 tx=0).stop 2024-04-23T10:24:45.325 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:24:45.325+0000 7f1b39787640 1 -- 172.21.15.114:0/3723116402 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f1b341021c0 msgr2=0x7f1b3419f800 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T10:24:45.325 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:24:45.325+0000 7f1b39787640 1 --2- 172.21.15.114:0/3723116402 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f1b341021c0 0x7f1b3419f800 secure :-1 s=READY pgs=215 cs=0 l=1 rev1=1 crypto rx=0x7f1b2c00eb10 tx=0x7f1b2c00c2d0 comp rx=0 tx=0).stop 2024-04-23T10:24:45.325 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:24:45.325+0000 7f1b39787640 1 -- 172.21.15.114:0/3723116402 shutdown_connections 2024-04-23T10:24:45.325 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:24:45.325+0000 7f1b39787640 1 --2- 172.21.15.114:0/3723116402 >> [v2:172.21.15.114:6800/465333808,v1:172.21.15.114:6801/465333808] conn(0x7f1b200a3bb0 0x7f1b200a6070 unknown :-1 s=CLOSED pgs=167 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:24:45.325 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:24:45.325+0000 7f1b39787640 1 --2- 172.21.15.114:0/3723116402 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f1b341021c0 0x7f1b3419f800 unknown :-1 s=CLOSED pgs=215 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:24:45.325 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:24:45.325+0000 7f1b39787640 1 --2- 172.21.15.114:0/3723116402 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f1b34101800 0x7f1b34198790 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:24:45.325 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:24:45.325+0000 7f1b39787640 1 --2- 172.21.15.114:0/3723116402 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f1b34100610 0x7f1b34198250 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:24:45.325 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:24:45.325+0000 7f1b39787640 1 -- 172.21.15.114:0/3723116402 >> 172.21.15.114:0/3723116402 conn(0x7f1b340fbb80 msgr2=0x7f1b34104c20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T10:24:45.326 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:24:45.325+0000 7f1b39787640 1 -- 172.21.15.114:0/3723116402 shutdown_connections 2024-04-23T10:24:45.326 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:24:45.325+0000 7f1b39787640 1 -- 172.21.15.114:0/3723116402 wait complete. 2024-04-23T10:24:45.328 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 23 10:24:45 smithi204 ceph-mon[32649]: pgmap v284: 97 pgs: 23 creating+peering, 68 unknown, 6 active+clean; 577 KiB data, 365 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-23T10:24:45.328 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 23 10:24:45 smithi204 ceph-mon[32649]: osdmap e86: 12 total, 12 up, 12 in 2024-04-23T10:24:45.328 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 23 10:24:45 smithi204 ceph-mon[32649]: Health check failed: 2 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-04-23T10:24:45.328 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 23 10:24:45 smithi204 ceph-mon[32649]: from='client.? 172.21.15.114:0/2263953334' entity='client.admin' cmd='[{"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs_metadata", "data": "cephfs_data"}]': finished 2024-04-23T10:24:45.328 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 23 10:24:45 smithi204 ceph-mon[32649]: fsmap cephfs:1 {0=b=up:creating} 11 up:standby 2024-04-23T10:24:45.329 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 23 10:24:45 smithi204 ceph-mon[32649]: daemon mds.b is now active in filesystem cephfs as rank 0 2024-04-23T10:24:45.329 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 23 10:24:45 smithi204 ceph-mon[32649]: from='client.? 172.21.15.114:0/2509541380' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-23T10:24:45.329 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 23 10:24:45 smithi204 ceph-mon[32649]: from='client.? 172.21.15.114:0/3696231242' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-23T10:24:45.336 DEBUG:tasks.cephfs.filesystem:fs_config: {'max_mds': 3, 'session_timeout': 300, 'standby_replay': True, 'subvols': {'create': 2}} 2024-04-23T10:24:45.336 DEBUG:teuthology.orchestra.run.smithi114:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs set cephfs max_mds 3 2024-04-23T10:24:45.440 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:24:45.439+0000 7f860335d640 1 -- 172.21.15.114:0/1087481940 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f85fc103190 msgr2=0x7f85fc103610 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T10:24:45.440 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:24:45.439+0000 7f860335d640 1 --2- 172.21.15.114:0/1087481940 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f85fc103190 0x7f85fc103610 secure :-1 s=READY pgs=216 cs=0 l=1 rev1=1 crypto rx=0x7f85ec009900 tx=0x7f85ec02a410 comp rx=0 tx=0).stop 2024-04-23T10:24:45.440 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:24:45.439+0000 7f860335d640 1 -- 172.21.15.114:0/1087481940 shutdown_connections 2024-04-23T10:24:45.440 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:24:45.439+0000 7f860335d640 1 --2- 172.21.15.114:0/1087481940 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f85fc103b50 0x7f85fc10a3e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:24:45.440 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:24:45.439+0000 7f860335d640 1 --2- 172.21.15.114:0/1087481940 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f85fc103190 0x7f85fc103610 unknown :-1 s=CLOSED pgs=216 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:24:45.440 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:24:45.439+0000 7f860335d640 1 --2- 172.21.15.114:0/1087481940 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f85fc101fa0 0x7f85fc1023c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:24:45.441 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:24:45.439+0000 7f860335d640 1 -- 172.21.15.114:0/1087481940 >> 172.21.15.114:0/1087481940 conn(0x7f85fc0fd530 msgr2=0x7f85fc0ff950 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T10:24:45.441 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:24:45.439+0000 7f860335d640 1 -- 172.21.15.114:0/1087481940 shutdown_connections 2024-04-23T10:24:45.441 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:24:45.439+0000 7f860335d640 1 -- 172.21.15.114:0/1087481940 wait complete. 2024-04-23T10:24:45.441 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:24:45.440+0000 7f860335d640 1 Processor -- start 2024-04-23T10:24:45.441 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:24:45.440+0000 7f860335d640 1 -- start start 2024-04-23T10:24:45.441 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:24:45.440+0000 7f860335d640 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f85fc101fa0 0x7f85fc198290 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T10:24:45.441 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:24:45.440+0000 7f860335d640 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f85fc103190 0x7f85fc1987d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T10:24:45.441 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:24:45.440+0000 7f860335d640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f85fc103b50 0x7f85fc19f840 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T10:24:45.441 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:24:45.440+0000 7f860335d640 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x7f85fc10db40 con 0x7f85fc103b50 2024-04-23T10:24:45.441 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:24:45.440+0000 7f860335d640 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 -- 0x7f85fc10d9c0 con 0x7f85fc101fa0 2024-04-23T10:24:45.441 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:24:45.440+0000 7f860335d640 1 -- --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_getmap magic: 0 -- 0x7f85fc10dcc0 con 0x7f85fc103190 2024-04-23T10:24:45.441 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:24:45.440+0000 7f86018fa640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f85fc103b50 0x7f85fc19f840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T10:24:45.441 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:24:45.440+0000 7f86018fa640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f85fc103b50 0x7f85fc19f840 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.114:44550/0 (socket says 172.21.15.114:44550) 2024-04-23T10:24:45.442 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:24:45.440+0000 7f86018fa640 1 -- 172.21.15.114:0/2932909767 learned_addr learned my addr 172.21.15.114:0/2932909767 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-04-23T10:24:45.442 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:24:45.440+0000 7f86018fa640 1 -- 172.21.15.114:0/2932909767 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f85fc103190 msgr2=0x7f85fc1987d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T10:24:45.442 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:24:45.440+0000 7f86008f8640 1 --2- 172.21.15.114:0/2932909767 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f85fc103190 0x7f85fc1987d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T10:24:45.442 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:24:45.440+0000 7f86018fa640 1 --2- 172.21.15.114:0/2932909767 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f85fc103190 0x7f85fc1987d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:24:45.442 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:24:45.440+0000 7f86018fa640 1 -- 172.21.15.114:0/2932909767 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f85fc101fa0 msgr2=0x7f85fc198290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T10:24:45.442 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:24:45.440+0000 7f86010f9640 1 --2- 172.21.15.114:0/2932909767 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f85fc101fa0 0x7f85fc198290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T10:24:45.442 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:24:45.440+0000 7f86018fa640 1 --2- 172.21.15.114:0/2932909767 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f85fc101fa0 0x7f85fc198290 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:24:45.442 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:24:45.440+0000 7f86018fa640 1 -- 172.21.15.114:0/2932909767 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f85fc19feb0 con 0x7f85fc103b50 2024-04-23T10:24:45.442 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:24:45.440+0000 7f86018fa640 1 --2- 172.21.15.114:0/2932909767 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f85fc103b50 0x7f85fc19f840 secure :-1 s=READY pgs=217 cs=0 l=1 rev1=1 crypto rx=0x7f85f400eaa0 tx=0x7f85f400ef60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T10:24:45.442 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:24:45.440+0000 7f85f27fc640 1 -- 172.21.15.114:0/2932909767 <== mon.0 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f85f400cdb0 con 0x7f85fc103b50 2024-04-23T10:24:45.442 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:24:45.440+0000 7f85f27fc640 1 -- 172.21.15.114:0/2932909767 <== mon.0 v2:172.21.15.114:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f85f4005020 con 0x7f85fc103b50 2024-04-23T10:24:45.442 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:24:45.440+0000 7f860335d640 1 -- 172.21.15.114:0/2932909767 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f85fc1a01a0 con 0x7f85fc103b50 2024-04-23T10:24:45.442 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:24:45.440+0000 7f85f27fc640 1 -- 172.21.15.114:0/2932909767 <== mon.0 v2:172.21.15.114:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f85f40116d0 con 0x7f85fc103b50 2024-04-23T10:24:45.442 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:24:45.440+0000 7f860335d640 1 -- 172.21.15.114:0/2932909767 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f85fc1a0720 con 0x7f85fc103b50 2024-04-23T10:24:45.443 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:24:45.441+0000 7f860335d640 1 -- 172.21.15.114:0/2932909767 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f85fc109390 con 0x7f85fc103b50 2024-04-23T10:24:45.443 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:24:45.442+0000 7f85f27fc640 1 -- 172.21.15.114:0/2932909767 <== mon.0 v2:172.21.15.114:3300/0 4 ==== mgrmap(e 16) ==== 138647+0+0 (secure 0 0 0) 0x7f85f40051d0 con 0x7f85fc103b50 2024-04-23T10:24:45.443 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:24:45.443+0000 7f85f27fc640 1 --2- 172.21.15.114:0/2932909767 >> [v2:172.21.15.114:6800/465333808,v1:172.21.15.114:6801/465333808] conn(0x7f85e80a3bb0 0x7f85e80a6070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T10:24:45.444 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:24:45.443+0000 7f86010f9640 1 --2- 172.21.15.114:0/2932909767 >> [v2:172.21.15.114:6800/465333808,v1:172.21.15.114:6801/465333808] conn(0x7f85e80a3bb0 0x7f85e80a6070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T10:24:45.444 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:24:45.443+0000 7f85f27fc640 1 -- 172.21.15.114:0/2932909767 <== mon.0 v2:172.21.15.114:3300/0 5 ==== osd_map(87..87 src has 1..87) ==== 7979+0+0 (secure 0 0 0) 0x7f85f40d0560 con 0x7f85fc103b50 2024-04-23T10:24:45.444 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:24:45.443+0000 7f86010f9640 1 --2- 172.21.15.114:0/2932909767 >> [v2:172.21.15.114:6800/465333808,v1:172.21.15.114:6801/465333808] conn(0x7f85e80a3bb0 0x7f85e80a6070 secure :-1 s=READY pgs=168 cs=0 l=1 rev1=1 crypto rx=0x7f85fc102ff0 tx=0x7f85f8008040 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T10:24:45.446 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:24:45.447+0000 7f85f27fc640 1 -- 172.21.15.114:0/2932909767 <== mon.0 v2:172.21.15.114:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f85f4016030 con 0x7f85fc103b50 2024-04-23T10:24:45.492 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 23 10:24:45 smithi119 ceph-mon[32608]: pgmap v284: 97 pgs: 23 creating+peering, 68 unknown, 6 active+clean; 577 KiB data, 365 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-23T10:24:45.492 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 23 10:24:45 smithi119 ceph-mon[32608]: osdmap e86: 12 total, 12 up, 12 in 2024-04-23T10:24:45.492 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 23 10:24:45 smithi119 ceph-mon[32608]: Health check failed: 2 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-04-23T10:24:45.492 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 23 10:24:45 smithi119 ceph-mon[32608]: from='client.? 172.21.15.114:0/2263953334' entity='client.admin' cmd='[{"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs_metadata", "data": "cephfs_data"}]': finished 2024-04-23T10:24:45.492 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 23 10:24:45 smithi119 ceph-mon[32608]: fsmap cephfs:1 {0=b=up:creating} 11 up:standby 2024-04-23T10:24:45.492 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 23 10:24:45 smithi119 ceph-mon[32608]: daemon mds.b is now active in filesystem cephfs as rank 0 2024-04-23T10:24:45.492 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 23 10:24:45 smithi119 ceph-mon[32608]: from='client.? 172.21.15.114:0/2509541380' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-23T10:24:45.492 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 23 10:24:45 smithi119 ceph-mon[32608]: from='client.? 172.21.15.114:0/3696231242' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-23T10:24:45.665 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:24:45.667+0000 7f860335d640 1 -- 172.21.15.114:0/2932909767 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_command({"prefix": "fs set", "fs_name": "cephfs", "var": "max_mds", "val": "3"} v 0) -- 0x7f85fc062b50 con 0x7f85fc103b50 2024-04-23T10:24:46.327 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:24:46.328+0000 7f85f27fc640 1 -- 172.21.15.114:0/2932909767 <== mon.0 v2:172.21.15.114:3300/0 7 ==== mon_command_ack([{"prefix": "fs set", "fs_name": "cephfs", "var": "max_mds", "val": "3"}]=0 v16) ==== 105+0+0 (secure 0 0 0) 0x7f85f409d090 con 0x7f85fc103b50 2024-04-23T10:24:46.328 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 23 10:24:46 smithi204 ceph-mon[32649]: osdmap e87: 12 total, 12 up, 12 in 2024-04-23T10:24:46.328 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 23 10:24:46 smithi204 ceph-mon[32649]: from='client.? 172.21.15.114:0/3723116402' entity='client.admin' cmd=[{"prefix": "fs set", "fs_name": "cephfs", "var": "standby_count_wanted", "val": "0"}]: dispatch 2024-04-23T10:24:46.328 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 23 10:24:46 smithi204 ceph-mon[32649]: from='client.? 172.21.15.114:0/3723116402' entity='client.admin' cmd='[{"prefix": "fs set", "fs_name": "cephfs", "var": "standby_count_wanted", "val": "0"}]': finished 2024-04-23T10:24:46.328 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 23 10:24:46 smithi204 ceph-mon[32649]: mds.? [v2:172.21.15.119:6838/937297340,v1:172.21.15.119:6839/937297340] up:active 2024-04-23T10:24:46.328 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 23 10:24:46 smithi204 ceph-mon[32649]: fsmap cephfs:1 {0=b=up:active} 11 up:standby 2024-04-23T10:24:46.328 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 23 10:24:46 smithi204 ceph-mon[32649]: from='client.? 172.21.15.114:0/2932909767' entity='client.admin' cmd=[{"prefix": "fs set", "fs_name": "cephfs", "var": "max_mds", "val": "3"}]: dispatch 2024-04-23T10:24:46.330 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:24:46.331+0000 7f860335d640 1 -- 172.21.15.114:0/2932909767 >> [v2:172.21.15.114:6800/465333808,v1:172.21.15.114:6801/465333808] conn(0x7f85e80a3bb0 msgr2=0x7f85e80a6070 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T10:24:46.330 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:24:46.331+0000 7f860335d640 1 --2- 172.21.15.114:0/2932909767 >> [v2:172.21.15.114:6800/465333808,v1:172.21.15.114:6801/465333808] conn(0x7f85e80a3bb0 0x7f85e80a6070 secure :-1 s=READY pgs=168 cs=0 l=1 rev1=1 crypto rx=0x7f85fc102ff0 tx=0x7f85f8008040 comp rx=0 tx=0).stop 2024-04-23T10:24:46.331 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:24:46.331+0000 7f860335d640 1 -- 172.21.15.114:0/2932909767 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f85fc103b50 msgr2=0x7f85fc19f840 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T10:24:46.331 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:24:46.331+0000 7f860335d640 1 --2- 172.21.15.114:0/2932909767 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f85fc103b50 0x7f85fc19f840 secure :-1 s=READY pgs=217 cs=0 l=1 rev1=1 crypto rx=0x7f85f400eaa0 tx=0x7f85f400ef60 comp rx=0 tx=0).stop 2024-04-23T10:24:46.331 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:24:46.331+0000 7f860335d640 1 -- 172.21.15.114:0/2932909767 shutdown_connections 2024-04-23T10:24:46.331 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:24:46.331+0000 7f860335d640 1 --2- 172.21.15.114:0/2932909767 >> [v2:172.21.15.114:6800/465333808,v1:172.21.15.114:6801/465333808] conn(0x7f85e80a3bb0 0x7f85e80a6070 unknown :-1 s=CLOSED pgs=168 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:24:46.331 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:24:46.331+0000 7f860335d640 1 --2- 172.21.15.114:0/2932909767 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f85fc103b50 0x7f85fc19f840 unknown :-1 s=CLOSED pgs=217 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:24:46.331 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:24:46.331+0000 7f860335d640 1 --2- 172.21.15.114:0/2932909767 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f85fc103190 0x7f85fc1987d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:24:46.331 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:24:46.331+0000 7f860335d640 1 --2- 172.21.15.114:0/2932909767 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f85fc101fa0 0x7f85fc198290 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:24:46.331 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:24:46.331+0000 7f860335d640 1 -- 172.21.15.114:0/2932909767 >> 172.21.15.114:0/2932909767 conn(0x7f85fc0fd530 msgr2=0x7f85fc106560 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T10:24:46.331 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:24:46.331+0000 7f860335d640 1 -- 172.21.15.114:0/2932909767 shutdown_connections 2024-04-23T10:24:46.331 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:24:46.331+0000 7f860335d640 1 -- 172.21.15.114:0/2932909767 wait complete. 2024-04-23T10:24:46.344 INFO:journalctl@ceph.mon.a.smithi114.stdout:Apr 23 10:24:46 smithi114 ceph-mon[29255]: osdmap e87: 12 total, 12 up, 12 in 2024-04-23T10:24:46.344 INFO:journalctl@ceph.mon.a.smithi114.stdout:Apr 23 10:24:46 smithi114 ceph-mon[29255]: from='client.? 172.21.15.114:0/3723116402' entity='client.admin' cmd=[{"prefix": "fs set", "fs_name": "cephfs", "var": "standby_count_wanted", "val": "0"}]: dispatch 2024-04-23T10:24:46.344 INFO:journalctl@ceph.mon.a.smithi114.stdout:Apr 23 10:24:46 smithi114 ceph-mon[29255]: from='client.? 172.21.15.114:0/3723116402' entity='client.admin' cmd='[{"prefix": "fs set", "fs_name": "cephfs", "var": "standby_count_wanted", "val": "0"}]': finished 2024-04-23T10:24:46.344 INFO:journalctl@ceph.mon.a.smithi114.stdout:Apr 23 10:24:46 smithi114 ceph-mon[29255]: mds.? [v2:172.21.15.119:6838/937297340,v1:172.21.15.119:6839/937297340] up:active 2024-04-23T10:24:46.344 INFO:journalctl@ceph.mon.a.smithi114.stdout:Apr 23 10:24:46 smithi114 ceph-mon[29255]: fsmap cephfs:1 {0=b=up:active} 11 up:standby 2024-04-23T10:24:46.344 INFO:journalctl@ceph.mon.a.smithi114.stdout:Apr 23 10:24:46 smithi114 ceph-mon[29255]: from='client.? 172.21.15.114:0/2932909767' entity='client.admin' cmd=[{"prefix": "fs set", "fs_name": "cephfs", "var": "max_mds", "val": "3"}]: dispatch 2024-04-23T10:24:46.345 DEBUG:teuthology.orchestra.run.smithi114:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs set cephfs allow_standby_replay true 2024-04-23T10:24:46.453 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:24:46.451+0000 7f8c7f1ad640 1 -- 172.21.15.114:0/1763281982 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f8c78102a00 msgr2=0x7f8c78109320 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T10:24:46.453 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:24:46.451+0000 7f8c7f1ad640 1 --2- 172.21.15.114:0/1763281982 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f8c78102a00 0x7f8c78109320 secure :-1 s=READY pgs=218 cs=0 l=1 rev1=1 crypto rx=0x7f8c70009ca0 tx=0x7f8c7002a8c0 comp rx=0 tx=0).stop 2024-04-23T10:24:46.453 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:24:46.451+0000 7f8c7f1ad640 1 -- 172.21.15.114:0/1763281982 shutdown_connections 2024-04-23T10:24:46.453 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:24:46.451+0000 7f8c7f1ad640 1 --2- 172.21.15.114:0/1763281982 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f8c78102a00 0x7f8c78109320 unknown :-1 s=CLOSED pgs=218 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:24:46.453 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:24:46.451+0000 7f8c7f1ad640 1 --2- 172.21.15.114:0/1763281982 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f8c78102040 0x7f8c781024c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:24:46.453 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:24:46.451+0000 7f8c7f1ad640 1 --2- 172.21.15.114:0/1763281982 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f8c78100e60 0x7f8c78101280 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:24:46.453 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:24:46.451+0000 7f8c7f1ad640 1 -- 172.21.15.114:0/1763281982 >> 172.21.15.114:0/1763281982 conn(0x7f8c780fc420 msgr2=0x7f8c780fe860 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T10:24:46.453 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:24:46.451+0000 7f8c7f1ad640 1 -- 172.21.15.114:0/1763281982 shutdown_connections 2024-04-23T10:24:46.453 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:24:46.451+0000 7f8c7f1ad640 1 -- 172.21.15.114:0/1763281982 wait complete. 2024-04-23T10:24:46.453 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:24:46.452+0000 7f8c7f1ad640 1 Processor -- start 2024-04-23T10:24:46.453 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:24:46.452+0000 7f8c7f1ad640 1 -- start start 2024-04-23T10:24:46.453 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:24:46.452+0000 7f8c7f1ad640 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f8c78100e60 0x7f8c781970e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T10:24:46.453 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:24:46.452+0000 7f8c7f1ad640 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f8c78102040 0x7f8c78197620 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T10:24:46.454 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:24:46.452+0000 7f8c7f1ad640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f8c78102a00 0x7f8c7819e690 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T10:24:46.454 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:24:46.452+0000 7f8c7f1ad640 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x7f8c7810ca70 con 0x7f8c78102a00 2024-04-23T10:24:46.454 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:24:46.452+0000 7f8c7f1ad640 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 -- 0x7f8c7810c8f0 con 0x7f8c78100e60 2024-04-23T10:24:46.454 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:24:46.452+0000 7f8c7f1ad640 1 -- --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_getmap magic: 0 -- 0x7f8c7810cbf0 con 0x7f8c78102040 2024-04-23T10:24:46.454 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:24:46.452+0000 7f8c7d74a640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f8c78102a00 0x7f8c7819e690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T10:24:46.454 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:24:46.452+0000 7f8c7d74a640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f8c78102a00 0x7f8c7819e690 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.114:43262/0 (socket says 172.21.15.114:43262) 2024-04-23T10:24:46.454 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:24:46.452+0000 7f8c7d74a640 1 -- 172.21.15.114:0/1853258673 learned_addr learned my addr 172.21.15.114:0/1853258673 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-04-23T10:24:46.454 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:24:46.452+0000 7f8c77fff640 1 --2- 172.21.15.114:0/1853258673 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f8c78102040 0x7f8c78197620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T10:24:46.454 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:24:46.452+0000 7f8c7cf49640 1 --2- 172.21.15.114:0/1853258673 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f8c78100e60 0x7f8c781970e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T10:24:46.454 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:24:46.453+0000 7f8c7d74a640 1 -- 172.21.15.114:0/1853258673 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f8c78102040 msgr2=0x7f8c78197620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T10:24:46.454 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:24:46.453+0000 7f8c7d74a640 1 --2- 172.21.15.114:0/1853258673 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f8c78102040 0x7f8c78197620 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:24:46.454 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:24:46.453+0000 7f8c7d74a640 1 -- 172.21.15.114:0/1853258673 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f8c78100e60 msgr2=0x7f8c781970e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T10:24:46.454 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:24:46.453+0000 7f8c7d74a640 1 --2- 172.21.15.114:0/1853258673 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f8c78100e60 0x7f8c781970e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:24:46.455 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:24:46.453+0000 7f8c7d74a640 1 -- 172.21.15.114:0/1853258673 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f8c7819ed00 con 0x7f8c78102a00 2024-04-23T10:24:46.455 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:24:46.453+0000 7f8c7d74a640 1 --2- 172.21.15.114:0/1853258673 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f8c78102a00 0x7f8c7819e690 secure :-1 s=READY pgs=219 cs=0 l=1 rev1=1 crypto rx=0x7f8c700097b0 tx=0x7f8c70003c20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T10:24:46.455 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:24:46.453+0000 7f8c75ffb640 1 -- 172.21.15.114:0/1853258673 <== mon.0 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8c70004240 con 0x7f8c78102a00 2024-04-23T10:24:46.455 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:24:46.453+0000 7f8c75ffb640 1 -- 172.21.15.114:0/1853258673 <== mon.0 v2:172.21.15.114:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f8c700317c0 con 0x7f8c78102a00 2024-04-23T10:24:46.455 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:24:46.453+0000 7f8c7f1ad640 1 -- 172.21.15.114:0/1853258673 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f8c7819ef90 con 0x7f8c78102a00 2024-04-23T10:24:46.455 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:24:46.453+0000 7f8c75ffb640 1 -- 172.21.15.114:0/1853258673 <== mon.0 v2:172.21.15.114:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8c7002fc70 con 0x7f8c78102a00 2024-04-23T10:24:46.455 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:24:46.453+0000 7f8c7f1ad640 1 -- 172.21.15.114:0/1853258673 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f8c7819f4d0 con 0x7f8c78102a00 2024-04-23T10:24:46.455 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:24:46.454+0000 7f8c7f1ad640 1 -- 172.21.15.114:0/1853258673 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f8c78108250 con 0x7f8c78102a00 2024-04-23T10:24:46.456 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:24:46.455+0000 7f8c75ffb640 1 -- 172.21.15.114:0/1853258673 <== mon.0 v2:172.21.15.114:3300/0 4 ==== mgrmap(e 16) ==== 138647+0+0 (secure 0 0 0) 0x7f8c700043e0 con 0x7f8c78102a00 2024-04-23T10:24:46.456 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:24:46.456+0000 7f8c75ffb640 1 --2- 172.21.15.114:0/1853258673 >> [v2:172.21.15.114:6800/465333808,v1:172.21.15.114:6801/465333808] conn(0x7f8c680a3bb0 0x7f8c680a6070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T10:24:46.456 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:24:46.456+0000 7f8c7cf49640 1 --2- 172.21.15.114:0/1853258673 >> [v2:172.21.15.114:6800/465333808,v1:172.21.15.114:6801/465333808] conn(0x7f8c680a3bb0 0x7f8c680a6070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T10:24:46.456 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:24:46.456+0000 7f8c75ffb640 1 -- 172.21.15.114:0/1853258673 <== mon.0 v2:172.21.15.114:3300/0 5 ==== osd_map(87..87 src has 1..87) ==== 7979+0+0 (secure 0 0 0) 0x7f8c700f1af0 con 0x7f8c78102a00 2024-04-23T10:24:46.456 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:24:46.456+0000 7f8c7cf49640 1 --2- 172.21.15.114:0/1853258673 >> [v2:172.21.15.114:6800/465333808,v1:172.21.15.114:6801/465333808] conn(0x7f8c680a3bb0 0x7f8c680a6070 secure :-1 s=READY pgs=169 cs=0 l=1 rev1=1 crypto rx=0x7f8c78101bb0 tx=0x7f8c6c009110 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T10:24:46.459 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:24:46.460+0000 7f8c75ffb640 1 -- 172.21.15.114:0/1853258673 <== mon.0 v2:172.21.15.114:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f8c700be620 con 0x7f8c78102a00 2024-04-23T10:24:46.492 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 23 10:24:46 smithi119 ceph-mon[32608]: osdmap e87: 12 total, 12 up, 12 in 2024-04-23T10:24:46.492 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 23 10:24:46 smithi119 ceph-mon[32608]: from='client.? 172.21.15.114:0/3723116402' entity='client.admin' cmd=[{"prefix": "fs set", "fs_name": "cephfs", "var": "standby_count_wanted", "val": "0"}]: dispatch 2024-04-23T10:24:46.492 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 23 10:24:46 smithi119 ceph-mon[32608]: from='client.? 172.21.15.114:0/3723116402' entity='client.admin' cmd='[{"prefix": "fs set", "fs_name": "cephfs", "var": "standby_count_wanted", "val": "0"}]': finished 2024-04-23T10:24:46.492 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 23 10:24:46 smithi119 ceph-mon[32608]: mds.? [v2:172.21.15.119:6838/937297340,v1:172.21.15.119:6839/937297340] up:active 2024-04-23T10:24:46.492 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 23 10:24:46 smithi119 ceph-mon[32608]: fsmap cephfs:1 {0=b=up:active} 11 up:standby 2024-04-23T10:24:46.492 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 23 10:24:46 smithi119 ceph-mon[32608]: from='client.? 172.21.15.114:0/2932909767' entity='client.admin' cmd=[{"prefix": "fs set", "fs_name": "cephfs", "var": "max_mds", "val": "3"}]: dispatch 2024-04-23T10:24:46.677 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:24:46.678+0000 7f8c7f1ad640 1 -- 172.21.15.114:0/1853258673 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_command({"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "true"} v 0) -- 0x7f8c7802d810 con 0x7f8c78102a00 2024-04-23T10:24:47.328 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 23 10:24:47 smithi204 ceph-mon[32649]: pgmap v287: 97 pgs: 5 creating+activating, 23 creating+peering, 44 unknown, 25 active+clean; 577 KiB data, 365 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-23T10:24:47.328 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 23 10:24:47 smithi204 ceph-mon[32649]: Health check cleared: POOL_APP_NOT_ENABLED (was: 2 pool(s) do not have an application enabled) 2024-04-23T10:24:47.328 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 23 10:24:47 smithi204 ceph-mon[32649]: Cluster is now healthy 2024-04-23T10:24:47.328 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 23 10:24:47 smithi204 ceph-mon[32649]: Health check failed: 1 filesystem is online with fewer MDS than max_mds (MDS_UP_LESS_THAN_MAX) 2024-04-23T10:24:47.328 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 23 10:24:47 smithi204 ceph-mon[32649]: from='client.? 172.21.15.114:0/2932909767' entity='client.admin' cmd='[{"prefix": "fs set", "fs_name": "cephfs", "var": "max_mds", "val": "3"}]': finished 2024-04-23T10:24:47.328 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 23 10:24:47 smithi204 ceph-mon[32649]: daemon mds.h assigned to filesystem cephfs as rank 1 (now has 2 ranks) 2024-04-23T10:24:47.328 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 23 10:24:47 smithi204 ceph-mon[32649]: fsmap cephfs:1 {0=b=up:active} 11 up:standby 2024-04-23T10:24:47.328 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 23 10:24:47 smithi204 ceph-mon[32649]: fsmap cephfs:2 {0=b=up:active,1=h=up:creating} 10 up:standby 2024-04-23T10:24:47.329 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 23 10:24:47 smithi204 ceph-mon[32649]: daemon mds.h is now active in filesystem cephfs as rank 1 2024-04-23T10:24:47.329 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 23 10:24:47 smithi204 ceph-mon[32649]: from='client.? 172.21.15.114:0/1853258673' entity='client.admin' cmd=[{"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "true"}]: dispatch 2024-04-23T10:24:47.334 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:24:47.335+0000 7f8c75ffb640 1 -- 172.21.15.114:0/1853258673 <== mon.0 v2:172.21.15.114:3300/0 7 ==== mon_command_ack([{"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "true"}]=0 v18) ==== 121+0+0 (secure 0 0 0) 0x7f8c700098b0 con 0x7f8c78102a00 2024-04-23T10:24:47.336 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:24:47.337+0000 7f8c7f1ad640 1 -- 172.21.15.114:0/1853258673 >> [v2:172.21.15.114:6800/465333808,v1:172.21.15.114:6801/465333808] conn(0x7f8c680a3bb0 msgr2=0x7f8c680a6070 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T10:24:47.336 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:24:47.337+0000 7f8c7f1ad640 1 --2- 172.21.15.114:0/1853258673 >> [v2:172.21.15.114:6800/465333808,v1:172.21.15.114:6801/465333808] conn(0x7f8c680a3bb0 0x7f8c680a6070 secure :-1 s=READY pgs=169 cs=0 l=1 rev1=1 crypto rx=0x7f8c78101bb0 tx=0x7f8c6c009110 comp rx=0 tx=0).stop 2024-04-23T10:24:47.337 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:24:47.337+0000 7f8c7f1ad640 1 -- 172.21.15.114:0/1853258673 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f8c78102a00 msgr2=0x7f8c7819e690 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T10:24:47.337 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:24:47.337+0000 7f8c7f1ad640 1 --2- 172.21.15.114:0/1853258673 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f8c78102a00 0x7f8c7819e690 secure :-1 s=READY pgs=219 cs=0 l=1 rev1=1 crypto rx=0x7f8c700097b0 tx=0x7f8c70003c20 comp rx=0 tx=0).stop 2024-04-23T10:24:47.337 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:24:47.337+0000 7f8c7f1ad640 1 -- 172.21.15.114:0/1853258673 shutdown_connections 2024-04-23T10:24:47.337 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:24:47.337+0000 7f8c7f1ad640 1 --2- 172.21.15.114:0/1853258673 >> [v2:172.21.15.114:6800/465333808,v1:172.21.15.114:6801/465333808] conn(0x7f8c680a3bb0 0x7f8c680a6070 unknown :-1 s=CLOSED pgs=169 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:24:47.337 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:24:47.337+0000 7f8c7f1ad640 1 --2- 172.21.15.114:0/1853258673 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f8c78102a00 0x7f8c7819e690 unknown :-1 s=CLOSED pgs=219 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:24:47.337 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:24:47.337+0000 7f8c7f1ad640 1 --2- 172.21.15.114:0/1853258673 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f8c78102040 0x7f8c78197620 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:24:47.337 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:24:47.337+0000 7f8c7f1ad640 1 --2- 172.21.15.114:0/1853258673 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f8c78100e60 0x7f8c781970e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:24:47.337 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:24:47.337+0000 7f8c7f1ad640 1 -- 172.21.15.114:0/1853258673 >> 172.21.15.114:0/1853258673 conn(0x7f8c780fc420 msgr2=0x7f8c78105420 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T10:24:47.337 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:24:47.337+0000 7f8c7f1ad640 1 -- 172.21.15.114:0/1853258673 shutdown_connections 2024-04-23T10:24:47.337 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:24:47.337+0000 7f8c7f1ad640 1 -- 172.21.15.114:0/1853258673 wait complete. 2024-04-23T10:24:47.351 INFO:journalctl@ceph.mon.a.smithi114.stdout:Apr 23 10:24:47 smithi114 ceph-mon[29255]: pgmap v287: 97 pgs: 5 creating+activating, 23 creating+peering, 44 unknown, 25 active+clean; 577 KiB data, 365 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-23T10:24:47.351 INFO:journalctl@ceph.mon.a.smithi114.stdout:Apr 23 10:24:47 smithi114 ceph-mon[29255]: Health check cleared: POOL_APP_NOT_ENABLED (was: 2 pool(s) do not have an application enabled) 2024-04-23T10:24:47.351 INFO:journalctl@ceph.mon.a.smithi114.stdout:Apr 23 10:24:47 smithi114 ceph-mon[29255]: Cluster is now healthy 2024-04-23T10:24:47.351 INFO:journalctl@ceph.mon.a.smithi114.stdout:Apr 23 10:24:47 smithi114 ceph-mon[29255]: Health check failed: 1 filesystem is online with fewer MDS than max_mds (MDS_UP_LESS_THAN_MAX) 2024-04-23T10:24:47.351 INFO:journalctl@ceph.mon.a.smithi114.stdout:Apr 23 10:24:47 smithi114 ceph-mon[29255]: from='client.? 172.21.15.114:0/2932909767' entity='client.admin' cmd='[{"prefix": "fs set", "fs_name": "cephfs", "var": "max_mds", "val": "3"}]': finished 2024-04-23T10:24:47.351 INFO:journalctl@ceph.mon.a.smithi114.stdout:Apr 23 10:24:47 smithi114 ceph-mon[29255]: daemon mds.h assigned to filesystem cephfs as rank 1 (now has 2 ranks) 2024-04-23T10:24:47.351 INFO:journalctl@ceph.mon.a.smithi114.stdout:Apr 23 10:24:47 smithi114 ceph-mon[29255]: fsmap cephfs:1 {0=b=up:active} 11 up:standby 2024-04-23T10:24:47.351 INFO:journalctl@ceph.mon.a.smithi114.stdout:Apr 23 10:24:47 smithi114 ceph-mon[29255]: fsmap cephfs:2 {0=b=up:active,1=h=up:creating} 10 up:standby 2024-04-23T10:24:47.351 INFO:journalctl@ceph.mon.a.smithi114.stdout:Apr 23 10:24:47 smithi114 ceph-mon[29255]: daemon mds.h is now active in filesystem cephfs as rank 1 2024-04-23T10:24:47.351 INFO:journalctl@ceph.mon.a.smithi114.stdout:Apr 23 10:24:47 smithi114 ceph-mon[29255]: from='client.? 172.21.15.114:0/1853258673' entity='client.admin' cmd=[{"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "true"}]: dispatch 2024-04-23T10:24:47.353 DEBUG:teuthology.orchestra.run.smithi114:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs set cephfs session_timeout 300 2024-04-23T10:24:47.460 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:24:47.459+0000 7f1deb1f7640 1 -- 172.21.15.114:0/3053197017 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f1de4102a10 msgr2=0x7f1de4104e00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T10:24:47.461 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:24:47.459+0000 7f1deb1f7640 1 --2- 172.21.15.114:0/3053197017 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f1de4102a10 0x7f1de4104e00 secure :-1 s=READY pgs=220 cs=0 l=1 rev1=1 crypto rx=0x7f1dd8009900 tx=0x7f1dd802a410 comp rx=0 tx=0).stop 2024-04-23T10:24:47.461 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:24:47.459+0000 7f1deb1f7640 1 -- 172.21.15.114:0/3053197017 shutdown_connections 2024-04-23T10:24:47.461 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:24:47.459+0000 7f1deb1f7640 1 --2- 172.21.15.114:0/3053197017 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f1de4107c70 0x7f1de410a0c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:24:47.461 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:24:47.459+0000 7f1deb1f7640 1 --2- 172.21.15.114:0/3053197017 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f1de4105340 0x7f1de4107730 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:24:47.461 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:24:47.459+0000 7f1deb1f7640 1 --2- 172.21.15.114:0/3053197017 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f1de4102a10 0x7f1de4104e00 unknown :-1 s=CLOSED pgs=220 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:24:47.461 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:24:47.459+0000 7f1deb1f7640 1 -- 172.21.15.114:0/3053197017 >> 172.21.15.114:0/3053197017 conn(0x7f1de40fc400 msgr2=0x7f1de40fe840 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T10:24:47.461 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:24:47.459+0000 7f1deb1f7640 1 -- 172.21.15.114:0/3053197017 shutdown_connections 2024-04-23T10:24:47.461 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:24:47.459+0000 7f1deb1f7640 1 -- 172.21.15.114:0/3053197017 wait complete. 2024-04-23T10:24:47.461 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:24:47.459+0000 7f1deb1f7640 1 Processor -- start 2024-04-23T10:24:47.461 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:24:47.460+0000 7f1deb1f7640 1 -- start start 2024-04-23T10:24:47.461 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:24:47.460+0000 7f1deb1f7640 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f1de4102a10 0x7f1de4102330 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T10:24:47.461 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:24:47.460+0000 7f1deb1f7640 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f1de4105340 0x7f1de41009c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T10:24:47.461 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:24:47.460+0000 7f1deb1f7640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f1de4107c70 0x7f1de4100f00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T10:24:47.462 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:24:47.460+0000 7f1deb1f7640 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x7f1de410dac0 con 0x7f1de4107c70 2024-04-23T10:24:47.462 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:24:47.460+0000 7f1deb1f7640 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 -- 0x7f1de410d940 con 0x7f1de4102a10 2024-04-23T10:24:47.462 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:24:47.460+0000 7f1deb1f7640 1 -- --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_getmap magic: 0 -- 0x7f1de410dc40 con 0x7f1de4105340 2024-04-23T10:24:47.462 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:24:47.460+0000 7f1de9794640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f1de4107c70 0x7f1de4100f00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T10:24:47.462 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:24:47.460+0000 7f1de9794640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f1de4107c70 0x7f1de4100f00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.114:43282/0 (socket says 172.21.15.114:43282) 2024-04-23T10:24:47.462 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:24:47.460+0000 7f1de9794640 1 -- 172.21.15.114:0/347304495 learned_addr learned my addr 172.21.15.114:0/347304495 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-04-23T10:24:47.462 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:24:47.460+0000 7f1de8f93640 1 --2- 172.21.15.114:0/347304495 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f1de4102a10 0x7f1de4102330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T10:24:47.462 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:24:47.460+0000 7f1de9794640 1 -- 172.21.15.114:0/347304495 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f1de4105340 msgr2=0x7f1de41009c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T10:24:47.462 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:24:47.460+0000 7f1de9794640 1 --2- 172.21.15.114:0/347304495 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f1de4105340 0x7f1de41009c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:24:47.462 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:24:47.460+0000 7f1de9794640 1 -- 172.21.15.114:0/347304495 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f1de4102a10 msgr2=0x7f1de4102330 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T10:24:47.462 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:24:47.460+0000 7f1de9794640 1 --2- 172.21.15.114:0/347304495 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f1de4102a10 0x7f1de4102330 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:24:47.462 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:24:47.460+0000 7f1de9794640 1 -- 172.21.15.114:0/347304495 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f1de41017c0 con 0x7f1de4107c70 2024-04-23T10:24:47.462 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:24:47.460+0000 7f1de9794640 1 --2- 172.21.15.114:0/347304495 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f1de4107c70 0x7f1de4100f00 secure :-1 s=READY pgs=221 cs=0 l=1 rev1=1 crypto rx=0x7f1ddc00eb10 tx=0x7f1ddc00c2d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T10:24:47.463 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:24:47.460+0000 7f1de1ffb640 1 -- 172.21.15.114:0/347304495 <== mon.0 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1ddc004b20 con 0x7f1de4107c70 2024-04-23T10:24:47.463 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:24:47.460+0000 7f1de1ffb640 1 -- 172.21.15.114:0/347304495 <== mon.0 v2:172.21.15.114:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f1ddc004cc0 con 0x7f1de4107c70 2024-04-23T10:24:47.463 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:24:47.460+0000 7f1deb1f7640 1 -- 172.21.15.114:0/347304495 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f1de41a2380 con 0x7f1de4107c70 2024-04-23T10:24:47.463 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:24:47.460+0000 7f1de1ffb640 1 -- 172.21.15.114:0/347304495 <== mon.0 v2:172.21.15.114:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1ddc010670 con 0x7f1de4107c70 2024-04-23T10:24:47.463 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:24:47.461+0000 7f1deb1f7640 1 -- 172.21.15.114:0/347304495 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f1de41a2850 con 0x7f1de4107c70 2024-04-23T10:24:47.463 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:24:47.462+0000 7f1deb1f7640 1 -- 172.21.15.114:0/347304495 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f1de40fdef0 con 0x7f1de4107c70 2024-04-23T10:24:47.464 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:24:47.463+0000 7f1de1ffb640 1 -- 172.21.15.114:0/347304495 <== mon.0 v2:172.21.15.114:3300/0 4 ==== mgrmap(e 16) ==== 138647+0+0 (secure 0 0 0) 0x7f1ddc005020 con 0x7f1de4107c70 2024-04-23T10:24:47.464 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:24:47.463+0000 7f1de1ffb640 1 --2- 172.21.15.114:0/347304495 >> [v2:172.21.15.114:6800/465333808,v1:172.21.15.114:6801/465333808] conn(0x7f1dd40a3bb0 0x7f1dd40a6070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T10:24:47.464 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:24:47.463+0000 7f1de8f93640 1 --2- 172.21.15.114:0/347304495 >> [v2:172.21.15.114:6800/465333808,v1:172.21.15.114:6801/465333808] conn(0x7f1dd40a3bb0 0x7f1dd40a6070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T10:24:47.464 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:24:47.463+0000 7f1de1ffb640 1 -- 172.21.15.114:0/347304495 <== mon.0 v2:172.21.15.114:3300/0 5 ==== osd_map(87..87 src has 1..87) ==== 7979+0+0 (secure 0 0 0) 0x7f1ddc0d08c0 con 0x7f1de4107c70 2024-04-23T10:24:47.464 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:24:47.464+0000 7f1de8f93640 1 --2- 172.21.15.114:0/347304495 >> [v2:172.21.15.114:6800/465333808,v1:172.21.15.114:6801/465333808] conn(0x7f1dd40a3bb0 0x7f1dd40a6070 secure :-1 s=READY pgs=170 cs=0 l=1 rev1=1 crypto rx=0x7f1dd8000c00 tx=0x7f1dd80314a0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T10:24:47.467 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:24:47.468+0000 7f1de1ffb640 1 -- 172.21.15.114:0/347304495 <== mon.0 v2:172.21.15.114:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f1ddc09d3f0 con 0x7f1de4107c70 2024-04-23T10:24:47.492 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 23 10:24:47 smithi119 ceph-mon[32608]: pgmap v287: 97 pgs: 5 creating+activating, 23 creating+peering, 44 unknown, 25 active+clean; 577 KiB data, 365 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-23T10:24:47.492 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 23 10:24:47 smithi119 ceph-mon[32608]: Health check cleared: POOL_APP_NOT_ENABLED (was: 2 pool(s) do not have an application enabled) 2024-04-23T10:24:47.492 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 23 10:24:47 smithi119 ceph-mon[32608]: Cluster is now healthy 2024-04-23T10:24:47.492 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 23 10:24:47 smithi119 ceph-mon[32608]: Health check failed: 1 filesystem is online with fewer MDS than max_mds (MDS_UP_LESS_THAN_MAX) 2024-04-23T10:24:47.492 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 23 10:24:47 smithi119 ceph-mon[32608]: from='client.? 172.21.15.114:0/2932909767' entity='client.admin' cmd='[{"prefix": "fs set", "fs_name": "cephfs", "var": "max_mds", "val": "3"}]': finished 2024-04-23T10:24:47.492 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 23 10:24:47 smithi119 ceph-mon[32608]: daemon mds.h assigned to filesystem cephfs as rank 1 (now has 2 ranks) 2024-04-23T10:24:47.492 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 23 10:24:47 smithi119 ceph-mon[32608]: fsmap cephfs:1 {0=b=up:active} 11 up:standby 2024-04-23T10:24:47.492 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 23 10:24:47 smithi119 ceph-mon[32608]: fsmap cephfs:2 {0=b=up:active,1=h=up:creating} 10 up:standby 2024-04-23T10:24:47.492 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 23 10:24:47 smithi119 ceph-mon[32608]: daemon mds.h is now active in filesystem cephfs as rank 1 2024-04-23T10:24:47.493 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 23 10:24:47 smithi119 ceph-mon[32608]: from='client.? 172.21.15.114:0/1853258673' entity='client.admin' cmd=[{"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "true"}]: dispatch 2024-04-23T10:24:47.680 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:24:47.681+0000 7f1deb1f7640 1 -- 172.21.15.114:0/347304495 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_command({"prefix": "fs set", "fs_name": "cephfs", "var": "session_timeout", "val": "300"} v 0) -- 0x7f1de402d810 con 0x7f1de4107c70 2024-04-23T10:24:48.314 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:24:48.315+0000 7f1de1ffb640 1 -- 172.21.15.114:0/347304495 <== mon.0 v2:172.21.15.114:3300/0 7 ==== mon_command_ack([{"prefix": "fs set", "fs_name": "cephfs", "var": "session_timeout", "val": "300"}]=0 v20) ==== 115+0+0 (secure 0 0 0) 0x7f1ddc09ccf0 con 0x7f1de4107c70 2024-04-23T10:24:48.317 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:24:48.318+0000 7f1deb1f7640 1 -- 172.21.15.114:0/347304495 >> [v2:172.21.15.114:6800/465333808,v1:172.21.15.114:6801/465333808] conn(0x7f1dd40a3bb0 msgr2=0x7f1dd40a6070 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T10:24:48.317 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:24:48.318+0000 7f1deb1f7640 1 --2- 172.21.15.114:0/347304495 >> [v2:172.21.15.114:6800/465333808,v1:172.21.15.114:6801/465333808] conn(0x7f1dd40a3bb0 0x7f1dd40a6070 secure :-1 s=READY pgs=170 cs=0 l=1 rev1=1 crypto rx=0x7f1dd8000c00 tx=0x7f1dd80314a0 comp rx=0 tx=0).stop 2024-04-23T10:24:48.317 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:24:48.318+0000 7f1deb1f7640 1 -- 172.21.15.114:0/347304495 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f1de4107c70 msgr2=0x7f1de4100f00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T10:24:48.318 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:24:48.318+0000 7f1deb1f7640 1 --2- 172.21.15.114:0/347304495 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f1de4107c70 0x7f1de4100f00 secure :-1 s=READY pgs=221 cs=0 l=1 rev1=1 crypto rx=0x7f1ddc00eb10 tx=0x7f1ddc00c2d0 comp rx=0 tx=0).stop 2024-04-23T10:24:48.318 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:24:48.318+0000 7f1deb1f7640 1 -- 172.21.15.114:0/347304495 shutdown_connections 2024-04-23T10:24:48.318 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:24:48.318+0000 7f1deb1f7640 1 --2- 172.21.15.114:0/347304495 >> [v2:172.21.15.114:6800/465333808,v1:172.21.15.114:6801/465333808] conn(0x7f1dd40a3bb0 0x7f1dd40a6070 unknown :-1 s=CLOSED pgs=170 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:24:48.318 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:24:48.318+0000 7f1deb1f7640 1 --2- 172.21.15.114:0/347304495 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f1de4107c70 0x7f1de4100f00 unknown :-1 s=CLOSED pgs=221 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:24:48.318 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:24:48.318+0000 7f1deb1f7640 1 --2- 172.21.15.114:0/347304495 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f1de4105340 0x7f1de41009c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:24:48.318 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:24:48.318+0000 7f1deb1f7640 1 --2- 172.21.15.114:0/347304495 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f1de4102a10 0x7f1de4102330 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:24:48.318 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:24:48.318+0000 7f1deb1f7640 1 -- 172.21.15.114:0/347304495 >> 172.21.15.114:0/347304495 conn(0x7f1de40fc400 msgr2=0x7f1de4103510 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T10:24:48.318 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:24:48.318+0000 7f1deb1f7640 1 -- 172.21.15.114:0/347304495 shutdown_connections 2024-04-23T10:24:48.318 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:24:48.318+0000 7f1deb1f7640 1 -- 172.21.15.114:0/347304495 wait complete. 2024-04-23T10:24:48.331 DEBUG:tasks.cephfs.filesystem:Creating {'create': 2} subvols for filesystem 'cephfs' 2024-04-23T10:24:48.331 DEBUG:teuthology.orchestra.run.smithi114:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs subvolumegroup create cephfs qa 2024-04-23T10:24:48.440 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:24:48.438+0000 7ff6954cf640 1 -- 172.21.15.114:0/3851751585 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7ff690101fb0 msgr2=0x7ff6901023d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T10:24:48.440 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:24:48.438+0000 7ff6954cf640 1 --2- 172.21.15.114:0/3851751585 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7ff690101fb0 0x7ff6901023d0 secure :-1 s=READY pgs=222 cs=0 l=1 rev1=1 crypto rx=0x7ff680009900 tx=0x7ff68002a410 comp rx=0 tx=0).stop 2024-04-23T10:24:48.440 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:24:48.438+0000 7ff6954cf640 1 -- 172.21.15.114:0/3851751585 shutdown_connections 2024-04-23T10:24:48.440 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:24:48.438+0000 7ff6954cf640 1 --2- 172.21.15.114:0/3851751585 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7ff690103b60 0x7ff69010a3f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:24:48.440 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:24:48.438+0000 7ff6954cf640 1 --2- 172.21.15.114:0/3851751585 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7ff6901031a0 0x7ff690103620 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:24:48.440 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:24:48.438+0000 7ff6954cf640 1 --2- 172.21.15.114:0/3851751585 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7ff690101fb0 0x7ff6901023d0 unknown :-1 s=CLOSED pgs=222 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:24:48.440 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:24:48.438+0000 7ff6954cf640 1 -- 172.21.15.114:0/3851751585 >> 172.21.15.114:0/3851751585 conn(0x7ff6900fd520 msgr2=0x7ff6900ff960 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T10:24:48.440 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:24:48.438+0000 7ff6954cf640 1 -- 172.21.15.114:0/3851751585 shutdown_connections 2024-04-23T10:24:48.440 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:24:48.438+0000 7ff6954cf640 1 -- 172.21.15.114:0/3851751585 wait complete. 2024-04-23T10:24:48.440 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:24:48.439+0000 7ff6954cf640 1 Processor -- start 2024-04-23T10:24:48.440 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:24:48.439+0000 7ff6954cf640 1 -- start start 2024-04-23T10:24:48.441 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:24:48.439+0000 7ff6954cf640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7ff690101fb0 0x7ff690075d00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T10:24:48.441 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:24:48.439+0000 7ff6954cf640 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7ff6901031a0 0x7ff690074340 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T10:24:48.441 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:24:48.439+0000 7ff6954cf640 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7ff690103b60 0x7ff690074880 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T10:24:48.441 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:24:48.439+0000 7ff6954cf640 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x7ff69010db30 con 0x7ff690101fb0 2024-04-23T10:24:48.441 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:24:48.439+0000 7ff6954cf640 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 -- 0x7ff69010d9b0 con 0x7ff6901031a0 2024-04-23T10:24:48.441 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:24:48.439+0000 7ff6954cf640 1 -- --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_getmap magic: 0 -- 0x7ff69010dcb0 con 0x7ff690103b60 2024-04-23T10:24:48.441 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:24:48.439+0000 7ff68effd640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7ff690101fb0 0x7ff690075d00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T10:24:48.441 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:24:48.439+0000 7ff68effd640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7ff690101fb0 0x7ff690075d00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.114:43302/0 (socket says 172.21.15.114:43302) 2024-04-23T10:24:48.441 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:24:48.439+0000 7ff68effd640 1 -- 172.21.15.114:0/187897024 learned_addr learned my addr 172.21.15.114:0/187897024 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-04-23T10:24:48.441 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:24:48.439+0000 7ff687fff640 1 --2- 172.21.15.114:0/187897024 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7ff6901031a0 0x7ff690074340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T10:24:48.441 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:24:48.439+0000 7ff68effd640 1 -- 172.21.15.114:0/187897024 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7ff690103b60 msgr2=0x7ff690074880 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T10:24:48.441 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:24:48.439+0000 7ff68effd640 1 --2- 172.21.15.114:0/187897024 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7ff690103b60 0x7ff690074880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:24:48.441 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:24:48.439+0000 7ff68effd640 1 -- 172.21.15.114:0/187897024 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7ff6901031a0 msgr2=0x7ff690074340 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T10:24:48.441 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:24:48.439+0000 7ff68effd640 1 --2- 172.21.15.114:0/187897024 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7ff6901031a0 0x7ff690074340 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:24:48.442 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:24:48.439+0000 7ff68effd640 1 -- 172.21.15.114:0/187897024 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff690075110 con 0x7ff690101fb0 2024-04-23T10:24:48.442 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:24:48.439+0000 7ff68effd640 1 --2- 172.21.15.114:0/187897024 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7ff690101fb0 0x7ff690075d00 secure :-1 s=READY pgs=223 cs=0 l=1 rev1=1 crypto rx=0x7ff680006010 tx=0x7ff6800042f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T10:24:48.442 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:24:48.439+0000 7ff68cff9640 1 -- 172.21.15.114:0/187897024 <== mon.0 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff68003f030 con 0x7ff690101fb0 2024-04-23T10:24:48.442 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:24:48.439+0000 7ff68cff9640 1 -- 172.21.15.114:0/187897024 <== mon.0 v2:172.21.15.114:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7ff6800318f0 con 0x7ff690101fb0 2024-04-23T10:24:48.442 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:24:48.439+0000 7ff6954cf640 1 -- 172.21.15.114:0/187897024 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7ff6901a2330 con 0x7ff690101fb0 2024-04-23T10:24:48.442 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:24:48.439+0000 7ff68cff9640 1 -- 172.21.15.114:0/187897024 <== mon.0 v2:172.21.15.114:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff680035070 con 0x7ff690101fb0 2024-04-23T10:24:48.442 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:24:48.440+0000 7ff6954cf640 1 -- 172.21.15.114:0/187897024 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7ff6901a2950 con 0x7ff690101fb0 2024-04-23T10:24:48.442 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:24:48.441+0000 7ff6954cf640 1 -- 172.21.15.114:0/187897024 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7ff690109340 con 0x7ff690101fb0 2024-04-23T10:24:48.443 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:24:48.441+0000 7ff68cff9640 1 -- 172.21.15.114:0/187897024 <== mon.0 v2:172.21.15.114:3300/0 4 ==== mgrmap(e 16) ==== 138647+0+0 (secure 0 0 0) 0x7ff680042030 con 0x7ff690101fb0 2024-04-23T10:24:48.443 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:24:48.442+0000 7ff68cff9640 1 --2- 172.21.15.114:0/187897024 >> [v2:172.21.15.114:6800/465333808,v1:172.21.15.114:6801/465333808] conn(0x7ff67c0a3c30 0x7ff67c0a60f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T10:24:48.443 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:24:48.442+0000 7ff687fff640 1 --2- 172.21.15.114:0/187897024 >> [v2:172.21.15.114:6800/465333808,v1:172.21.15.114:6801/465333808] conn(0x7ff67c0a3c30 0x7ff67c0a60f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T10:24:48.443 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:24:48.442+0000 7ff68cff9640 1 -- 172.21.15.114:0/187897024 <== mon.0 v2:172.21.15.114:3300/0 5 ==== osd_map(87..87 src has 1..87) ==== 7979+0+0 (secure 0 0 0) 0x7ff6800f07d0 con 0x7ff690101fb0 2024-04-23T10:24:48.443 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:24:48.443+0000 7ff687fff640 1 --2- 172.21.15.114:0/187897024 >> [v2:172.21.15.114:6800/465333808,v1:172.21.15.114:6801/465333808] conn(0x7ff67c0a3c30 0x7ff67c0a60f0 secure :-1 s=READY pgs=171 cs=0 l=1 rev1=1 crypto rx=0x7ff690075a70 tx=0x7ff678005f90 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T10:24:48.446 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:24:48.447+0000 7ff68cff9640 1 -- 172.21.15.114:0/187897024 <== mon.0 v2:172.21.15.114:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7ff6800bd300 con 0x7ff690101fb0 2024-04-23T10:24:48.661 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:24:48.662+0000 7ff6954cf640 1 -- 172.21.15.114:0/187897024 --> [v2:172.21.15.114:6800/465333808,v1:172.21.15.114:6801/465333808] -- mgr_command(tid 0: {"prefix": "fs subvolumegroup create", "vol_name": "cephfs", "group_name": "qa", "target": ["mon-mgr", ""]}) -- 0x7ff690062390 con 0x7ff67c0a3c30 2024-04-23T10:24:48.667 INFO:journalctl@ceph.mon.a.smithi114.stdout:Apr 23 10:24:48 smithi114 ceph-mon[29255]: from='client.? 172.21.15.114:0/1853258673' entity='client.admin' cmd='[{"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "true"}]': finished 2024-04-23T10:24:48.667 INFO:journalctl@ceph.mon.a.smithi114.stdout:Apr 23 10:24:48 smithi114 ceph-mon[29255]: mds.? [v2:172.21.15.119:6836/1872363128,v1:172.21.15.119:6837/1872363128] up:active 2024-04-23T10:24:48.667 INFO:journalctl@ceph.mon.a.smithi114.stdout:Apr 23 10:24:48 smithi114 ceph-mon[29255]: daemon mds.e assigned to filesystem cephfs as rank 2 (now has 3 ranks) 2024-04-23T10:24:48.668 INFO:journalctl@ceph.mon.a.smithi114.stdout:Apr 23 10:24:48 smithi114 ceph-mon[29255]: Health check cleared: MDS_UP_LESS_THAN_MAX (was: 1 filesystem is online with fewer MDS than max_mds) 2024-04-23T10:24:48.668 INFO:journalctl@ceph.mon.a.smithi114.stdout:Apr 23 10:24:48 smithi114 ceph-mon[29255]: Cluster is now healthy 2024-04-23T10:24:48.668 INFO:journalctl@ceph.mon.a.smithi114.stdout:Apr 23 10:24:48 smithi114 ceph-mon[29255]: fsmap cephfs:2 {0=b=up:active,1=h=up:active} 10 up:standby 2024-04-23T10:24:48.668 INFO:journalctl@ceph.mon.a.smithi114.stdout:Apr 23 10:24:48 smithi114 ceph-mon[29255]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=e=up:creating} 9 up:standby 2024-04-23T10:24:48.668 INFO:journalctl@ceph.mon.a.smithi114.stdout:Apr 23 10:24:48 smithi114 ceph-mon[29255]: daemon mds.e is now active in filesystem cephfs as rank 2 2024-04-23T10:24:48.668 INFO:journalctl@ceph.mon.a.smithi114.stdout:Apr 23 10:24:48 smithi114 ceph-mon[29255]: from='client.? 172.21.15.114:0/347304495' entity='client.admin' cmd=[{"prefix": "fs set", "fs_name": "cephfs", "var": "session_timeout", "val": "300"}]: dispatch 2024-04-23T10:24:48.668 INFO:journalctl@ceph.mon.a.smithi114.stdout:Apr 23 10:24:48 smithi114 ceph-mon[29255]: pgmap v288: 97 pgs: 5 creating+activating, 14 creating+peering, 10 unknown, 68 active+clean; 579 KiB data, 370 MiB used, 1.0 TiB / 1.0 TiB avail; 3.4 KiB/s wr, 6 op/s 2024-04-23T10:24:48.668 INFO:journalctl@ceph.mon.a.smithi114.stdout:Apr 23 10:24:48 smithi114 ceph-mon[29255]: from='client.? 172.21.15.114:0/347304495' entity='client.admin' cmd='[{"prefix": "fs set", "fs_name": "cephfs", "var": "session_timeout", "val": "300"}]': finished 2024-04-23T10:24:48.668 INFO:journalctl@ceph.mon.a.smithi114.stdout:Apr 23 10:24:48 smithi114 ceph-mon[29255]: mds.? [v2:172.21.15.119:6834/2033916709,v1:172.21.15.119:6835/2033916709] up:active 2024-04-23T10:24:48.668 INFO:journalctl@ceph.mon.a.smithi114.stdout:Apr 23 10:24:48 smithi114 ceph-mon[29255]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=e=up:active} 3 up:standby-replay 6 up:standby 2024-04-23T10:24:48.687 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:24:48.689+0000 7ff68cff9640 1 -- 172.21.15.114:0/187897024 <== mgr.14152 v2:172.21.15.114:6800/465333808 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+0 (secure 0 0 0) 0x7ff690062390 con 0x7ff67c0a3c30 2024-04-23T10:24:48.690 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:24:48.691+0000 7ff6954cf640 1 -- 172.21.15.114:0/187897024 >> [v2:172.21.15.114:6800/465333808,v1:172.21.15.114:6801/465333808] conn(0x7ff67c0a3c30 msgr2=0x7ff67c0a60f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T10:24:48.690 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:24:48.691+0000 7ff6954cf640 1 --2- 172.21.15.114:0/187897024 >> [v2:172.21.15.114:6800/465333808,v1:172.21.15.114:6801/465333808] conn(0x7ff67c0a3c30 0x7ff67c0a60f0 secure :-1 s=READY pgs=171 cs=0 l=1 rev1=1 crypto rx=0x7ff690075a70 tx=0x7ff678005f90 comp rx=0 tx=0).stop 2024-04-23T10:24:48.690 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:24:48.691+0000 7ff6954cf640 1 -- 172.21.15.114:0/187897024 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7ff690101fb0 msgr2=0x7ff690075d00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T10:24:48.690 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:24:48.691+0000 7ff6954cf640 1 --2- 172.21.15.114:0/187897024 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7ff690101fb0 0x7ff690075d00 secure :-1 s=READY pgs=223 cs=0 l=1 rev1=1 crypto rx=0x7ff680006010 tx=0x7ff6800042f0 comp rx=0 tx=0).stop 2024-04-23T10:24:48.690 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:24:48.691+0000 7ff6954cf640 1 -- 172.21.15.114:0/187897024 shutdown_connections 2024-04-23T10:24:48.691 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:24:48.691+0000 7ff6954cf640 1 --2- 172.21.15.114:0/187897024 >> [v2:172.21.15.114:6800/465333808,v1:172.21.15.114:6801/465333808] conn(0x7ff67c0a3c30 0x7ff67c0a60f0 unknown :-1 s=CLOSED pgs=171 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:24:48.691 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:24:48.691+0000 7ff6954cf640 1 --2- 172.21.15.114:0/187897024 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7ff690103b60 0x7ff690074880 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:24:48.691 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:24:48.691+0000 7ff6954cf640 1 --2- 172.21.15.114:0/187897024 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7ff6901031a0 0x7ff690074340 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:24:48.691 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:24:48.691+0000 7ff6954cf640 1 --2- 172.21.15.114:0/187897024 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7ff690101fb0 0x7ff690075d00 unknown :-1 s=CLOSED pgs=223 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:24:48.691 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:24:48.691+0000 7ff6954cf640 1 -- 172.21.15.114:0/187897024 >> 172.21.15.114:0/187897024 conn(0x7ff6900fd520 msgr2=0x7ff690106510 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T10:24:48.691 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:24:48.691+0000 7ff6954cf640 1 -- 172.21.15.114:0/187897024 shutdown_connections 2024-04-23T10:24:48.691 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:24:48.691+0000 7ff6954cf640 1 -- 172.21.15.114:0/187897024 wait complete. 2024-04-23T10:24:48.701 DEBUG:teuthology.orchestra.run.smithi114:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs subvolume create cephfs sv_0 --group_name qa 2024-04-23T10:24:48.742 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 23 10:24:48 smithi119 ceph-mon[32608]: from='client.? 172.21.15.114:0/1853258673' entity='client.admin' cmd='[{"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "true"}]': finished 2024-04-23T10:24:48.742 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 23 10:24:48 smithi119 ceph-mon[32608]: mds.? [v2:172.21.15.119:6836/1872363128,v1:172.21.15.119:6837/1872363128] up:active 2024-04-23T10:24:48.742 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 23 10:24:48 smithi119 ceph-mon[32608]: daemon mds.e assigned to filesystem cephfs as rank 2 (now has 3 ranks) 2024-04-23T10:24:48.742 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 23 10:24:48 smithi119 ceph-mon[32608]: Health check cleared: MDS_UP_LESS_THAN_MAX (was: 1 filesystem is online with fewer MDS than max_mds) 2024-04-23T10:24:48.742 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 23 10:24:48 smithi119 ceph-mon[32608]: Cluster is now healthy 2024-04-23T10:24:48.742 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 23 10:24:48 smithi119 ceph-mon[32608]: fsmap cephfs:2 {0=b=up:active,1=h=up:active} 10 up:standby 2024-04-23T10:24:48.742 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 23 10:24:48 smithi119 ceph-mon[32608]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=e=up:creating} 9 up:standby 2024-04-23T10:24:48.742 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 23 10:24:48 smithi119 ceph-mon[32608]: daemon mds.e is now active in filesystem cephfs as rank 2 2024-04-23T10:24:48.743 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 23 10:24:48 smithi119 ceph-mon[32608]: from='client.? 172.21.15.114:0/347304495' entity='client.admin' cmd=[{"prefix": "fs set", "fs_name": "cephfs", "var": "session_timeout", "val": "300"}]: dispatch 2024-04-23T10:24:48.743 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 23 10:24:48 smithi119 ceph-mon[32608]: pgmap v288: 97 pgs: 5 creating+activating, 14 creating+peering, 10 unknown, 68 active+clean; 579 KiB data, 370 MiB used, 1.0 TiB / 1.0 TiB avail; 3.4 KiB/s wr, 6 op/s 2024-04-23T10:24:48.743 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 23 10:24:48 smithi119 ceph-mon[32608]: from='client.? 172.21.15.114:0/347304495' entity='client.admin' cmd='[{"prefix": "fs set", "fs_name": "cephfs", "var": "session_timeout", "val": "300"}]': finished 2024-04-23T10:24:48.743 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 23 10:24:48 smithi119 ceph-mon[32608]: mds.? [v2:172.21.15.119:6834/2033916709,v1:172.21.15.119:6835/2033916709] up:active 2024-04-23T10:24:48.743 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 23 10:24:48 smithi119 ceph-mon[32608]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=e=up:active} 3 up:standby-replay 6 up:standby 2024-04-23T10:24:48.806 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:24:48.805+0000 7f9be0bf9640 1 -- 172.21.15.114:0/3632497507 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f9bdc102040 msgr2=0x7f9bdc1024c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T10:24:48.806 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:24:48.805+0000 7f9be0bf9640 1 --2- 172.21.15.114:0/3632497507 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f9bdc102040 0x7f9bdc1024c0 secure :-1 s=READY pgs=226 cs=0 l=1 rev1=1 crypto rx=0x7f9bc4009900 tx=0x7f9bc402a410 comp rx=0 tx=0).stop 2024-04-23T10:24:48.806 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:24:48.805+0000 7f9be0bf9640 1 -- 172.21.15.114:0/3632497507 shutdown_connections 2024-04-23T10:24:48.807 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:24:48.805+0000 7f9be0bf9640 1 --2- 172.21.15.114:0/3632497507 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f9bdc102a00 0x7f9bdc109320 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:24:48.807 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:24:48.805+0000 7f9be0bf9640 1 --2- 172.21.15.114:0/3632497507 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f9bdc102040 0x7f9bdc1024c0 unknown :-1 s=CLOSED pgs=226 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:24:48.807 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:24:48.805+0000 7f9be0bf9640 1 --2- 172.21.15.114:0/3632497507 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f9bdc100e60 0x7f9bdc101280 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:24:48.807 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:24:48.805+0000 7f9be0bf9640 1 -- 172.21.15.114:0/3632497507 >> 172.21.15.114:0/3632497507 conn(0x7f9bdc0fc420 msgr2=0x7f9bdc0fe860 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T10:24:48.807 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:24:48.805+0000 7f9be0bf9640 1 -- 172.21.15.114:0/3632497507 shutdown_connections 2024-04-23T10:24:48.807 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:24:48.805+0000 7f9be0bf9640 1 -- 172.21.15.114:0/3632497507 wait complete. 2024-04-23T10:24:48.807 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:24:48.806+0000 7f9be0bf9640 1 Processor -- start 2024-04-23T10:24:48.807 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:24:48.806+0000 7f9be0bf9640 1 -- start start 2024-04-23T10:24:48.807 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:24:48.806+0000 7f9be0bf9640 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f9bdc100e60 0x7f9bdc198220 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T10:24:48.807 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:24:48.806+0000 7f9be0bf9640 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f9bdc102040 0x7f9bdc198760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T10:24:48.807 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:24:48.806+0000 7f9be0bf9640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f9bdc102a00 0x7f9bdc19f7d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T10:24:48.807 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:24:48.806+0000 7f9be0bf9640 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x7f9bdc10c9f0 con 0x7f9bdc102a00 2024-04-23T10:24:48.807 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:24:48.806+0000 7f9be0bf9640 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 -- 0x7f9bdc10c870 con 0x7f9bdc100e60 2024-04-23T10:24:48.808 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:24:48.806+0000 7f9be0bf9640 1 -- --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_getmap magic: 0 -- 0x7f9bdc10cb70 con 0x7f9bdc102040 2024-04-23T10:24:48.808 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:24:48.806+0000 7f9bdad9d640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f9bdc102a00 0x7f9bdc19f7d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T10:24:48.808 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:24:48.806+0000 7f9bdad9d640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f9bdc102a00 0x7f9bdc19f7d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.114:43334/0 (socket says 172.21.15.114:43334) 2024-04-23T10:24:48.808 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:24:48.806+0000 7f9bdad9d640 1 -- 172.21.15.114:0/1895313244 learned_addr learned my addr 172.21.15.114:0/1895313244 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-04-23T10:24:48.808 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:24:48.806+0000 7f9bda59c640 1 --2- 172.21.15.114:0/1895313244 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f9bdc100e60 0x7f9bdc198220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T10:24:48.808 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:24:48.806+0000 7f9bd3fff640 1 --2- 172.21.15.114:0/1895313244 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f9bdc102040 0x7f9bdc198760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T10:24:48.808 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:24:48.806+0000 7f9bdad9d640 1 -- 172.21.15.114:0/1895313244 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f9bdc102040 msgr2=0x7f9bdc198760 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T10:24:48.808 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:24:48.806+0000 7f9bdad9d640 1 --2- 172.21.15.114:0/1895313244 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f9bdc102040 0x7f9bdc198760 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:24:48.808 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:24:48.806+0000 7f9bdad9d640 1 -- 172.21.15.114:0/1895313244 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f9bdc100e60 msgr2=0x7f9bdc198220 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T10:24:48.808 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:24:48.806+0000 7f9bdad9d640 1 --2- 172.21.15.114:0/1895313244 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f9bdc100e60 0x7f9bdc198220 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:24:48.808 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:24:48.806+0000 7f9bdad9d640 1 -- 172.21.15.114:0/1895313244 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f9bdc19fe40 con 0x7f9bdc102a00 2024-04-23T10:24:48.808 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:24:48.806+0000 7f9bd3fff640 1 --2- 172.21.15.114:0/1895313244 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f9bdc102040 0x7f9bdc198760 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-23T10:24:48.809 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:24:48.806+0000 7f9bda59c640 1 --2- 172.21.15.114:0/1895313244 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f9bdc100e60 0x7f9bdc198220 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-23T10:24:48.809 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:24:48.806+0000 7f9bdad9d640 1 --2- 172.21.15.114:0/1895313244 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f9bdc102a00 0x7f9bdc19f7d0 secure :-1 s=READY pgs=227 cs=0 l=1 rev1=1 crypto rx=0x7f9bd400eb10 tx=0x7f9bd400c2d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T10:24:48.809 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:24:48.806+0000 7f9bd37fe640 1 -- 172.21.15.114:0/1895313244 <== mon.0 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9bd4004b20 con 0x7f9bdc102a00 2024-04-23T10:24:48.809 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:24:48.807+0000 7f9be0bf9640 1 -- 172.21.15.114:0/1895313244 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f9bdc1a0130 con 0x7f9bdc102a00 2024-04-23T10:24:48.809 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:24:48.807+0000 7f9bd37fe640 1 -- 172.21.15.114:0/1895313244 <== mon.0 v2:172.21.15.114:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f9bd4004cc0 con 0x7f9bdc102a00 2024-04-23T10:24:48.809 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:24:48.807+0000 7f9bd37fe640 1 -- 172.21.15.114:0/1895313244 <== mon.0 v2:172.21.15.114:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9bd4010670 con 0x7f9bdc102a00 2024-04-23T10:24:48.809 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:24:48.807+0000 7f9be0bf9640 1 -- 172.21.15.114:0/1895313244 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f9bdc1a0630 con 0x7f9bdc102a00 2024-04-23T10:24:48.809 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:24:48.807+0000 7f9be0bf9640 1 -- 172.21.15.114:0/1895313244 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f9bdc1a08f0 con 0x7f9bdc102a00 2024-04-23T10:24:48.810 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:24:48.809+0000 7f9bd37fe640 1 -- 172.21.15.114:0/1895313244 <== mon.0 v2:172.21.15.114:3300/0 4 ==== mgrmap(e 16) ==== 138647+0+0 (secure 0 0 0) 0x7f9bd4005000 con 0x7f9bdc102a00 2024-04-23T10:24:48.810 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:24:48.809+0000 7f9bd37fe640 1 --2- 172.21.15.114:0/1895313244 >> [v2:172.21.15.114:6800/465333808,v1:172.21.15.114:6801/465333808] conn(0x7f9bc80a3bb0 0x7f9bc80a6070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T10:24:48.810 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:24:48.809+0000 7f9bda59c640 1 --2- 172.21.15.114:0/1895313244 >> [v2:172.21.15.114:6800/465333808,v1:172.21.15.114:6801/465333808] conn(0x7f9bc80a3bb0 0x7f9bc80a6070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T10:24:48.810 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:24:48.809+0000 7f9bd37fe640 1 -- 172.21.15.114:0/1895313244 <== mon.0 v2:172.21.15.114:3300/0 5 ==== osd_map(87..87 src has 1..87) ==== 7979+0+0 (secure 0 0 0) 0x7f9bd40d0af0 con 0x7f9bdc102a00 2024-04-23T10:24:48.810 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:24:48.810+0000 7f9bda59c640 1 --2- 172.21.15.114:0/1895313244 >> [v2:172.21.15.114:6800/465333808,v1:172.21.15.114:6801/465333808] conn(0x7f9bc80a3bb0 0x7f9bc80a6070 secure :-1 s=READY pgs=172 cs=0 l=1 rev1=1 crypto rx=0x7f9bdc101bb0 tx=0x7f9bcc00a2d0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T10:24:48.812 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:24:48.813+0000 7f9bd37fe640 1 -- 172.21.15.114:0/1895313244 <== mon.0 v2:172.21.15.114:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f9bd4053ff0 con 0x7f9bdc102a00 2024-04-23T10:24:48.828 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 23 10:24:48 smithi204 ceph-mon[32649]: from='client.? 172.21.15.114:0/1853258673' entity='client.admin' cmd='[{"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "true"}]': finished 2024-04-23T10:24:48.828 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 23 10:24:48 smithi204 ceph-mon[32649]: mds.? [v2:172.21.15.119:6836/1872363128,v1:172.21.15.119:6837/1872363128] up:active 2024-04-23T10:24:48.828 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 23 10:24:48 smithi204 ceph-mon[32649]: daemon mds.e assigned to filesystem cephfs as rank 2 (now has 3 ranks) 2024-04-23T10:24:48.828 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 23 10:24:48 smithi204 ceph-mon[32649]: Health check cleared: MDS_UP_LESS_THAN_MAX (was: 1 filesystem is online with fewer MDS than max_mds) 2024-04-23T10:24:48.828 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 23 10:24:48 smithi204 ceph-mon[32649]: Cluster is now healthy 2024-04-23T10:24:48.828 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 23 10:24:48 smithi204 ceph-mon[32649]: fsmap cephfs:2 {0=b=up:active,1=h=up:active} 10 up:standby 2024-04-23T10:24:48.828 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 23 10:24:48 smithi204 ceph-mon[32649]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=e=up:creating} 9 up:standby 2024-04-23T10:24:48.828 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 23 10:24:48 smithi204 ceph-mon[32649]: daemon mds.e is now active in filesystem cephfs as rank 2 2024-04-23T10:24:48.828 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 23 10:24:48 smithi204 ceph-mon[32649]: from='client.? 172.21.15.114:0/347304495' entity='client.admin' cmd=[{"prefix": "fs set", "fs_name": "cephfs", "var": "session_timeout", "val": "300"}]: dispatch 2024-04-23T10:24:48.828 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 23 10:24:48 smithi204 ceph-mon[32649]: pgmap v288: 97 pgs: 5 creating+activating, 14 creating+peering, 10 unknown, 68 active+clean; 579 KiB data, 370 MiB used, 1.0 TiB / 1.0 TiB avail; 3.4 KiB/s wr, 6 op/s 2024-04-23T10:24:48.829 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 23 10:24:48 smithi204 ceph-mon[32649]: from='client.? 172.21.15.114:0/347304495' entity='client.admin' cmd='[{"prefix": "fs set", "fs_name": "cephfs", "var": "session_timeout", "val": "300"}]': finished 2024-04-23T10:24:48.829 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 23 10:24:48 smithi204 ceph-mon[32649]: mds.? [v2:172.21.15.119:6834/2033916709,v1:172.21.15.119:6835/2033916709] up:active 2024-04-23T10:24:48.829 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 23 10:24:48 smithi204 ceph-mon[32649]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=e=up:active} 3 up:standby-replay 6 up:standby 2024-04-23T10:24:49.004 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Apr 23 10:24:48 smithi114 ceph-aa240572-0159-11ef-bc93-c7b262605968-mgr-x[29490]: 2024-04-23T10:24:48.667+0000 7f9b4a92e700 -1 client.0 error registering admin socket command: (17) File exists 2024-04-23T10:24:49.004 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Apr 23 10:24:48 smithi114 ceph-aa240572-0159-11ef-bc93-c7b262605968-mgr-x[29490]: 2024-04-23T10:24:48.667+0000 7f9b4a92e700 -1 client.0 error registering admin socket command: (17) File exists 2024-04-23T10:24:49.004 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Apr 23 10:24:48 smithi114 ceph-aa240572-0159-11ef-bc93-c7b262605968-mgr-x[29490]: 2024-04-23T10:24:48.667+0000 7f9b4a92e700 -1 client.0 error registering admin socket command: (17) File exists 2024-04-23T10:24:49.004 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Apr 23 10:24:48 smithi114 ceph-aa240572-0159-11ef-bc93-c7b262605968-mgr-x[29490]: 2024-04-23T10:24:48.667+0000 7f9b4a92e700 -1 client.0 error registering admin socket command: (17) File exists 2024-04-23T10:24:49.005 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Apr 23 10:24:48 smithi114 ceph-aa240572-0159-11ef-bc93-c7b262605968-mgr-x[29490]: 2024-04-23T10:24:48.667+0000 7f9b4a92e700 -1 client.0 error registering admin socket command: (17) File exists 2024-04-23T10:24:49.016 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:24:49.017+0000 7f9be0bf9640 1 -- 172.21.15.114:0/1895313244 --> [v2:172.21.15.114:6800/465333808,v1:172.21.15.114:6801/465333808] -- mgr_command(tid 0: {"prefix": "fs subvolume create", "vol_name": "cephfs", "sub_name": "sv_0", "group_name": "qa", "target": ["mon-mgr", ""]}) -- 0x7f9bdc0fe650 con 0x7f9bc80a3bb0 2024-04-23T10:24:49.041 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:24:49.042+0000 7f9bd37fe640 1 -- 172.21.15.114:0/1895313244 <== mgr.14152 v2:172.21.15.114:6800/465333808 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+0 (secure 0 0 0) 0x7f9bdc0fe650 con 0x7f9bc80a3bb0 2024-04-23T10:24:49.043 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:24:49.044+0000 7f9be0bf9640 1 -- 172.21.15.114:0/1895313244 >> [v2:172.21.15.114:6800/465333808,v1:172.21.15.114:6801/465333808] conn(0x7f9bc80a3bb0 msgr2=0x7f9bc80a6070 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T10:24:49.043 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:24:49.044+0000 7f9be0bf9640 1 --2- 172.21.15.114:0/1895313244 >> [v2:172.21.15.114:6800/465333808,v1:172.21.15.114:6801/465333808] conn(0x7f9bc80a3bb0 0x7f9bc80a6070 secure :-1 s=READY pgs=172 cs=0 l=1 rev1=1 crypto rx=0x7f9bdc101bb0 tx=0x7f9bcc00a2d0 comp rx=0 tx=0).stop 2024-04-23T10:24:49.043 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:24:49.044+0000 7f9be0bf9640 1 -- 172.21.15.114:0/1895313244 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f9bdc102a00 msgr2=0x7f9bdc19f7d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T10:24:49.043 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:24:49.044+0000 7f9be0bf9640 1 --2- 172.21.15.114:0/1895313244 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f9bdc102a00 0x7f9bdc19f7d0 secure :-1 s=READY pgs=227 cs=0 l=1 rev1=1 crypto rx=0x7f9bd400eb10 tx=0x7f9bd400c2d0 comp rx=0 tx=0).stop 2024-04-23T10:24:49.043 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:24:49.044+0000 7f9be0bf9640 1 -- 172.21.15.114:0/1895313244 shutdown_connections 2024-04-23T10:24:49.044 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:24:49.044+0000 7f9be0bf9640 1 --2- 172.21.15.114:0/1895313244 >> [v2:172.21.15.114:6800/465333808,v1:172.21.15.114:6801/465333808] conn(0x7f9bc80a3bb0 0x7f9bc80a6070 unknown :-1 s=CLOSED pgs=172 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:24:49.044 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:24:49.044+0000 7f9be0bf9640 1 --2- 172.21.15.114:0/1895313244 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f9bdc102a00 0x7f9bdc19f7d0 unknown :-1 s=CLOSED pgs=227 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:24:49.044 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:24:49.044+0000 7f9be0bf9640 1 --2- 172.21.15.114:0/1895313244 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f9bdc102040 0x7f9bdc198760 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:24:49.044 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:24:49.044+0000 7f9be0bf9640 1 --2- 172.21.15.114:0/1895313244 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f9bdc100e60 0x7f9bdc198220 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:24:49.044 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:24:49.044+0000 7f9be0bf9640 1 -- 172.21.15.114:0/1895313244 >> 172.21.15.114:0/1895313244 conn(0x7f9bdc0fc420 msgr2=0x7f9bdc0fdf40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T10:24:49.044 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:24:49.044+0000 7f9be0bf9640 1 -- 172.21.15.114:0/1895313244 shutdown_connections 2024-04-23T10:24:49.044 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:24:49.044+0000 7f9be0bf9640 1 -- 172.21.15.114:0/1895313244 wait complete. 2024-04-23T10:24:49.054 DEBUG:teuthology.orchestra.run.smithi114:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs subvolume getpath cephfs --group_name qa sv_0 2024-04-23T10:24:49.156 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:24:49.157+0000 7f9dd8e20640 1 -- 172.21.15.114:0/1738762062 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f9dd4101ac0 msgr2=0x7f9dd4109360 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T10:24:49.156 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:24:49.157+0000 7f9dd8e20640 1 --2- 172.21.15.114:0/1738762062 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f9dd4101ac0 0x7f9dd4109360 secure :-1 s=READY pgs=228 cs=0 l=1 rev1=1 crypto rx=0x7f9dcc009ca0 tx=0x7f9dcc02a8c0 comp rx=0 tx=0).stop 2024-04-23T10:24:49.156 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:24:49.157+0000 7f9dd8e20640 1 -- 172.21.15.114:0/1738762062 shutdown_connections 2024-04-23T10:24:49.156 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:24:49.157+0000 7f9dd8e20640 1 --2- 172.21.15.114:0/1738762062 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f9dd4101ac0 0x7f9dd4109360 unknown :-1 s=CLOSED pgs=228 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:24:49.157 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:24:49.157+0000 7f9dd8e20640 1 --2- 172.21.15.114:0/1738762062 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f9dd4101100 0x7f9dd4101580 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:24:49.157 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:24:49.157+0000 7f9dd8e20640 1 --2- 172.21.15.114:0/1738762062 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f9dd41024b0 0x7f9dd41028d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:24:49.157 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:24:49.157+0000 7f9dd8e20640 1 -- 172.21.15.114:0/1738762062 >> 172.21.15.114:0/1738762062 conn(0x7f9dd40fbb90 msgr2=0x7f9dd40fdfd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T10:24:49.157 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:24:49.157+0000 7f9dd8e20640 1 -- 172.21.15.114:0/1738762062 shutdown_connections 2024-04-23T10:24:49.157 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:24:49.157+0000 7f9dd8e20640 1 -- 172.21.15.114:0/1738762062 wait complete. 2024-04-23T10:24:49.159 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:24:49.158+0000 7f9dd8e20640 1 Processor -- start 2024-04-23T10:24:49.159 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:24:49.158+0000 7f9dd8e20640 1 -- start start 2024-04-23T10:24:49.159 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:24:49.158+0000 7f9dd8e20640 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f9dd4101100 0x7f9dd4100b60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T10:24:49.159 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:24:49.158+0000 7f9dd8e20640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f9dd4101ac0 0x7f9dd419cf80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T10:24:49.159 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:24:49.158+0000 7f9dd8e20640 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f9dd41024b0 0x7f9dd419f5a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T10:24:49.159 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:24:49.158+0000 7f9dd8e20640 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x7f9dd410ca60 con 0x7f9dd4101ac0 2024-04-23T10:24:49.159 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:24:49.158+0000 7f9dd8e20640 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 -- 0x7f9dd410c8e0 con 0x7f9dd41024b0 2024-04-23T10:24:49.159 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:24:49.158+0000 7f9dd8e20640 1 -- --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_getmap magic: 0 -- 0x7f9dd410cbe0 con 0x7f9dd4101100 2024-04-23T10:24:49.160 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:24:49.158+0000 7f9dcbfff640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f9dd4101ac0 0x7f9dd419cf80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T10:24:49.160 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:24:49.158+0000 7f9dcbfff640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f9dd4101ac0 0x7f9dd419cf80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.114:43362/0 (socket says 172.21.15.114:43362) 2024-04-23T10:24:49.160 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:24:49.158+0000 7f9dcbfff640 1 -- 172.21.15.114:0/3889510303 learned_addr learned my addr 172.21.15.114:0/3889510303 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-04-23T10:24:49.160 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:24:49.158+0000 7f9dd2d9d640 1 --2- 172.21.15.114:0/3889510303 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f9dd41024b0 0x7f9dd419f5a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T10:24:49.160 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:24:49.158+0000 7f9dd259c640 1 --2- 172.21.15.114:0/3889510303 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f9dd4101100 0x7f9dd4100b60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T10:24:49.160 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:24:49.158+0000 7f9dcbfff640 1 -- 172.21.15.114:0/3889510303 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f9dd4101100 msgr2=0x7f9dd4100b60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T10:24:49.160 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:24:49.158+0000 7f9dcbfff640 1 --2- 172.21.15.114:0/3889510303 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f9dd4101100 0x7f9dd4100b60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:24:49.160 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:24:49.158+0000 7f9dcbfff640 1 -- 172.21.15.114:0/3889510303 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f9dd41024b0 msgr2=0x7f9dd419f5a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T10:24:49.160 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:24:49.158+0000 7f9dcbfff640 1 --2- 172.21.15.114:0/3889510303 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f9dd41024b0 0x7f9dd419f5a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:24:49.160 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:24:49.158+0000 7f9dcbfff640 1 -- 172.21.15.114:0/3889510303 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f9dd419feb0 con 0x7f9dd4101ac0 2024-04-23T10:24:49.160 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:24:49.159+0000 7f9dcbfff640 1 --2- 172.21.15.114:0/3889510303 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f9dd4101ac0 0x7f9dd419cf80 secure :-1 s=READY pgs=229 cs=0 l=1 rev1=1 crypto rx=0x7f9dbc00b450 tx=0x7f9dbc00b910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T10:24:49.160 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:24:49.159+0000 7f9dcb7fe640 1 -- 172.21.15.114:0/3889510303 <== mon.0 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9dbc015020 con 0x7f9dd4101ac0 2024-04-23T10:24:49.160 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:24:49.159+0000 7f9dcb7fe640 1 -- 172.21.15.114:0/3889510303 <== mon.0 v2:172.21.15.114:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f9dbc011d40 con 0x7f9dd4101ac0 2024-04-23T10:24:49.160 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:24:49.159+0000 7f9dcb7fe640 1 -- 172.21.15.114:0/3889510303 <== mon.0 v2:172.21.15.114:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9dbc00f340 con 0x7f9dd4101ac0 2024-04-23T10:24:49.161 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:24:49.159+0000 7f9dd8e20640 1 -- 172.21.15.114:0/3889510303 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f9dd41a0140 con 0x7f9dd4101ac0 2024-04-23T10:24:49.161 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:24:49.159+0000 7f9dd8e20640 1 -- 172.21.15.114:0/3889510303 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f9dd41a0710 con 0x7f9dd4101ac0 2024-04-23T10:24:49.161 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:24:49.160+0000 7f9dd8e20640 1 -- 172.21.15.114:0/3889510303 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f9dd4108310 con 0x7f9dd4101ac0 2024-04-23T10:24:49.162 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:24:49.161+0000 7f9dcb7fe640 1 -- 172.21.15.114:0/3889510303 <== mon.0 v2:172.21.15.114:3300/0 4 ==== mgrmap(e 16) ==== 138647+0+0 (secure 0 0 0) 0x7f9dbc018020 con 0x7f9dd4101ac0 2024-04-23T10:24:49.162 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:24:49.162+0000 7f9dcb7fe640 1 --2- 172.21.15.114:0/3889510303 >> [v2:172.21.15.114:6800/465333808,v1:172.21.15.114:6801/465333808] conn(0x7f9dc00a3bb0 0x7f9dc00a6070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T10:24:49.162 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:24:49.162+0000 7f9dd259c640 1 --2- 172.21.15.114:0/3889510303 >> [v2:172.21.15.114:6800/465333808,v1:172.21.15.114:6801/465333808] conn(0x7f9dc00a3bb0 0x7f9dc00a6070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T10:24:49.162 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:24:49.162+0000 7f9dcb7fe640 1 -- 172.21.15.114:0/3889510303 <== mon.0 v2:172.21.15.114:3300/0 5 ==== osd_map(87..87 src has 1..87) ==== 7979+0+0 (secure 0 0 0) 0x7f9dbc0cfdf0 con 0x7f9dd4101ac0 2024-04-23T10:24:49.162 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:24:49.162+0000 7f9dd259c640 1 --2- 172.21.15.114:0/3889510303 >> [v2:172.21.15.114:6800/465333808,v1:172.21.15.114:6801/465333808] conn(0x7f9dc00a3bb0 0x7f9dc00a6070 secure :-1 s=READY pgs=173 cs=0 l=1 rev1=1 crypto rx=0x7f9dc4005db0 tx=0x7f9dc4005d40 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T10:24:49.165 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:24:49.166+0000 7f9dcb7fe640 1 -- 172.21.15.114:0/3889510303 <== mon.0 v2:172.21.15.114:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f9dbc01f020 con 0x7f9dd4101ac0 2024-04-23T10:24:49.375 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:24:49.377+0000 7f9dd8e20640 1 -- 172.21.15.114:0/3889510303 --> [v2:172.21.15.114:6800/465333808,v1:172.21.15.114:6801/465333808] -- mgr_command(tid 0: {"prefix": "fs subvolume getpath", "vol_name": "cephfs", "group_name": "qa", "sub_name": "sv_0", "target": ["mon-mgr", ""]}) -- 0x7f9dd4062420 con 0x7f9dc00a3bb0 2024-04-23T10:24:49.381 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:24:49.382+0000 7f9dcb7fe640 1 -- 172.21.15.114:0/3889510303 <== mgr.14152 v2:172.21.15.114:6800/465333808 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+54 (secure 0 0 0) 0x7f9dd4062420 con 0x7f9dc00a3bb0 2024-04-23T10:24:49.381 INFO:teuthology.orchestra.run.smithi114.stdout:/volumes/qa/sv_0/30055ca5-bafc-4745-832b-fb90b980738b 2024-04-23T10:24:49.383 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:24:49.384+0000 7f9dd8e20640 1 -- 172.21.15.114:0/3889510303 >> [v2:172.21.15.114:6800/465333808,v1:172.21.15.114:6801/465333808] conn(0x7f9dc00a3bb0 msgr2=0x7f9dc00a6070 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T10:24:49.383 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:24:49.384+0000 7f9dd8e20640 1 --2- 172.21.15.114:0/3889510303 >> [v2:172.21.15.114:6800/465333808,v1:172.21.15.114:6801/465333808] conn(0x7f9dc00a3bb0 0x7f9dc00a6070 secure :-1 s=READY pgs=173 cs=0 l=1 rev1=1 crypto rx=0x7f9dc4005db0 tx=0x7f9dc4005d40 comp rx=0 tx=0).stop 2024-04-23T10:24:49.383 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:24:49.384+0000 7f9dd8e20640 1 -- 172.21.15.114:0/3889510303 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f9dd4101ac0 msgr2=0x7f9dd419cf80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T10:24:49.383 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:24:49.384+0000 7f9dd8e20640 1 --2- 172.21.15.114:0/3889510303 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f9dd4101ac0 0x7f9dd419cf80 secure :-1 s=READY pgs=229 cs=0 l=1 rev1=1 crypto rx=0x7f9dbc00b450 tx=0x7f9dbc00b910 comp rx=0 tx=0).stop 2024-04-23T10:24:49.384 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:24:49.384+0000 7f9dd8e20640 1 -- 172.21.15.114:0/3889510303 shutdown_connections 2024-04-23T10:24:49.384 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:24:49.384+0000 7f9dd8e20640 1 --2- 172.21.15.114:0/3889510303 >> [v2:172.21.15.114:6800/465333808,v1:172.21.15.114:6801/465333808] conn(0x7f9dc00a3bb0 0x7f9dc00a6070 unknown :-1 s=CLOSED pgs=173 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:24:49.384 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:24:49.384+0000 7f9dd8e20640 1 --2- 172.21.15.114:0/3889510303 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f9dd41024b0 0x7f9dd419f5a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:24:49.384 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:24:49.384+0000 7f9dd8e20640 1 --2- 172.21.15.114:0/3889510303 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f9dd4101ac0 0x7f9dd419cf80 unknown :-1 s=CLOSED pgs=229 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:24:49.384 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:24:49.384+0000 7f9dd8e20640 1 --2- 172.21.15.114:0/3889510303 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f9dd4101100 0x7f9dd4100b60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:24:49.384 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:24:49.384+0000 7f9dd8e20640 1 -- 172.21.15.114:0/3889510303 >> 172.21.15.114:0/3889510303 conn(0x7f9dd40fbb90 msgr2=0x7f9dd41054e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T10:24:49.384 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:24:49.384+0000 7f9dd8e20640 1 -- 172.21.15.114:0/3889510303 shutdown_connections 2024-04-23T10:24:49.384 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:24:49.384+0000 7f9dd8e20640 1 -- 172.21.15.114:0/3889510303 wait complete. 2024-04-23T10:24:49.394 DEBUG:teuthology.orchestra.run.smithi114:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs subvolume create cephfs sv_1 --group_name qa 2024-04-23T10:24:49.492 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 23 10:24:49 smithi119 ceph-mon[32608]: from='client.14742 -' entity='client.admin' cmd=[{"prefix": "fs subvolumegroup create", "vol_name": "cephfs", "group_name": "qa", "target": ["mon-mgr", ""]}]: dispatch 2024-04-23T10:24:49.492 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 23 10:24:49 smithi119 ceph-mon[32608]: from='client.14754 -' entity='client.admin' cmd=[{"prefix": "fs subvolume create", "vol_name": "cephfs", "sub_name": "sv_0", "group_name": "qa", "target": ["mon-mgr", ""]}]: dispatch 2024-04-23T10:24:49.492 INFO:journalctl@ceph.mds.b.smithi119.stdout:Apr 23 10:24:49 smithi119 ceph-aa240572-0159-11ef-bc93-c7b262605968-mds-b[62689]: 2024-04-23T10:24:49.113+0000 7f2f46d47700 -1 mds.pinger is_rank_lagging: rank=0 was never sent ping request. 2024-04-23T10:24:49.492 INFO:journalctl@ceph.mds.b.smithi119.stdout:Apr 23 10:24:49 smithi119 ceph-aa240572-0159-11ef-bc93-c7b262605968-mds-b[62689]: 2024-04-23T10:24:49.113+0000 7f2f46d47700 -1 mds.pinger is_rank_lagging: rank=1 was never sent ping request. 2024-04-23T10:24:49.492 INFO:journalctl@ceph.mds.b.smithi119.stdout:Apr 23 10:24:49 smithi119 ceph-aa240572-0159-11ef-bc93-c7b262605968-mds-b[62689]: 2024-04-23T10:24:49.113+0000 7f2f46d47700 -1 mds.pinger is_rank_lagging: rank=2 was never sent ping request. 2024-04-23T10:24:49.500 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:24:49.498+0000 7ff4c0ffe640 1 -- 172.21.15.114:0/1600186015 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7ff4bc102a10 msgr2=0x7ff4bc104e00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T10:24:49.500 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:24:49.498+0000 7ff4c0ffe640 1 --2- 172.21.15.114:0/1600186015 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7ff4bc102a10 0x7ff4bc104e00 secure :-1 s=READY pgs=230 cs=0 l=1 rev1=1 crypto rx=0x7ff4ac009ca0 tx=0x7ff4ac02a8c0 comp rx=0 tx=0).stop 2024-04-23T10:24:49.500 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:24:49.498+0000 7ff4c0ffe640 1 -- 172.21.15.114:0/1600186015 shutdown_connections 2024-04-23T10:24:49.500 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:24:49.498+0000 7ff4c0ffe640 1 --2- 172.21.15.114:0/1600186015 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7ff4bc107c70 0x7ff4bc10a0c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:24:49.500 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:24:49.498+0000 7ff4c0ffe640 1 --2- 172.21.15.114:0/1600186015 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7ff4bc105340 0x7ff4bc107730 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:24:49.500 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:24:49.498+0000 7ff4c0ffe640 1 --2- 172.21.15.114:0/1600186015 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7ff4bc102a10 0x7ff4bc104e00 unknown :-1 s=CLOSED pgs=230 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:24:49.500 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:24:49.498+0000 7ff4c0ffe640 1 -- 172.21.15.114:0/1600186015 >> 172.21.15.114:0/1600186015 conn(0x7ff4bc0fc400 msgr2=0x7ff4bc0fe840 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T10:24:49.500 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:24:49.498+0000 7ff4c0ffe640 1 -- 172.21.15.114:0/1600186015 shutdown_connections 2024-04-23T10:24:49.500 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:24:49.498+0000 7ff4c0ffe640 1 -- 172.21.15.114:0/1600186015 wait complete. 2024-04-23T10:24:49.500 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:24:49.499+0000 7ff4c0ffe640 1 Processor -- start 2024-04-23T10:24:49.501 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:24:49.499+0000 7ff4c0ffe640 1 -- start start 2024-04-23T10:24:49.501 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:24:49.499+0000 7ff4c0ffe640 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7ff4bc102a10 0x7ff4bc198200 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T10:24:49.501 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:24:49.499+0000 7ff4c0ffe640 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7ff4bc105340 0x7ff4bc198740 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T10:24:49.501 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:24:49.499+0000 7ff4c0ffe640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7ff4bc107c70 0x7ff4bc19f7b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T10:24:49.501 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:24:49.499+0000 7ff4c0ffe640 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x7ff4bc10dac0 con 0x7ff4bc107c70 2024-04-23T10:24:49.501 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:24:49.499+0000 7ff4c0ffe640 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 -- 0x7ff4bc10d940 con 0x7ff4bc102a10 2024-04-23T10:24:49.501 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:24:49.499+0000 7ff4c0ffe640 1 -- --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_getmap magic: 0 -- 0x7ff4bc10dc40 con 0x7ff4bc105340 2024-04-23T10:24:49.501 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:24:49.499+0000 7ff4bad9d640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7ff4bc107c70 0x7ff4bc19f7b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T10:24:49.501 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:24:49.499+0000 7ff4bad9d640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7ff4bc107c70 0x7ff4bc19f7b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.114:43378/0 (socket says 172.21.15.114:43378) 2024-04-23T10:24:49.501 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:24:49.499+0000 7ff4bad9d640 1 -- 172.21.15.114:0/3560602326 learned_addr learned my addr 172.21.15.114:0/3560602326 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-04-23T10:24:49.501 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:24:49.499+0000 7ff4b9d9b640 1 --2- 172.21.15.114:0/3560602326 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7ff4bc105340 0x7ff4bc198740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T10:24:49.501 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:24:49.499+0000 7ff4ba59c640 1 --2- 172.21.15.114:0/3560602326 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7ff4bc102a10 0x7ff4bc198200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T10:24:49.501 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:24:49.500+0000 7ff4bad9d640 1 -- 172.21.15.114:0/3560602326 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7ff4bc105340 msgr2=0x7ff4bc198740 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T10:24:49.502 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:24:49.500+0000 7ff4bad9d640 1 --2- 172.21.15.114:0/3560602326 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7ff4bc105340 0x7ff4bc198740 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:24:49.502 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:24:49.500+0000 7ff4bad9d640 1 -- 172.21.15.114:0/3560602326 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7ff4bc102a10 msgr2=0x7ff4bc198200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T10:24:49.502 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:24:49.500+0000 7ff4bad9d640 1 --2- 172.21.15.114:0/3560602326 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7ff4bc102a10 0x7ff4bc198200 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:24:49.502 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:24:49.500+0000 7ff4bad9d640 1 -- 172.21.15.114:0/3560602326 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff4bc19fe20 con 0x7ff4bc107c70 2024-04-23T10:24:49.502 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:24:49.500+0000 7ff4bad9d640 1 --2- 172.21.15.114:0/3560602326 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7ff4bc107c70 0x7ff4bc19f7b0 secure :-1 s=READY pgs=231 cs=0 l=1 rev1=1 crypto rx=0x7ff4b400c970 tx=0x7ff4b400cd70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T10:24:49.502 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:24:49.500+0000 7ff4ab7fe640 1 -- 172.21.15.114:0/3560602326 <== mon.0 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff4b400ae60 con 0x7ff4bc107c70 2024-04-23T10:24:49.502 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:24:49.500+0000 7ff4ab7fe640 1 -- 172.21.15.114:0/3560602326 <== mon.0 v2:172.21.15.114:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7ff4b40054d0 con 0x7ff4bc107c70 2024-04-23T10:24:49.502 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:24:49.500+0000 7ff4c0ffe640 1 -- 172.21.15.114:0/3560602326 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7ff4bc1a0110 con 0x7ff4bc107c70 2024-04-23T10:24:49.502 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:24:49.500+0000 7ff4ab7fe640 1 -- 172.21.15.114:0/3560602326 <== mon.0 v2:172.21.15.114:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff4b4004d60 con 0x7ff4bc107c70 2024-04-23T10:24:49.502 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:24:49.500+0000 7ff4c0ffe640 1 -- 172.21.15.114:0/3560602326 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7ff4bc1a0610 con 0x7ff4bc107c70 2024-04-23T10:24:49.502 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:24:49.501+0000 7ff4c0ffe640 1 -- 172.21.15.114:0/3560602326 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7ff4bc0fdef0 con 0x7ff4bc107c70 2024-04-23T10:24:49.503 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:24:49.502+0000 7ff4ab7fe640 1 -- 172.21.15.114:0/3560602326 <== mon.0 v2:172.21.15.114:3300/0 4 ==== mgrmap(e 16) ==== 138647+0+0 (secure 0 0 0) 0x7ff4b4005020 con 0x7ff4bc107c70 2024-04-23T10:24:49.503 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:24:49.502+0000 7ff4ab7fe640 1 --2- 172.21.15.114:0/3560602326 >> [v2:172.21.15.114:6800/465333808,v1:172.21.15.114:6801/465333808] conn(0x7ff4a40a3bb0 0x7ff4a40a6070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T10:24:49.503 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:24:49.503+0000 7ff4ba59c640 1 --2- 172.21.15.114:0/3560602326 >> [v2:172.21.15.114:6800/465333808,v1:172.21.15.114:6801/465333808] conn(0x7ff4a40a3bb0 0x7ff4a40a6070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T10:24:49.503 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:24:49.503+0000 7ff4ab7fe640 1 -- 172.21.15.114:0/3560602326 <== mon.0 v2:172.21.15.114:3300/0 5 ==== osd_map(87..87 src has 1..87) ==== 7979+0+0 (secure 0 0 0) 0x7ff4b40cffc0 con 0x7ff4bc107c70 2024-04-23T10:24:49.503 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:24:49.503+0000 7ff4ba59c640 1 --2- 172.21.15.114:0/3560602326 >> [v2:172.21.15.114:6800/465333808,v1:172.21.15.114:6801/465333808] conn(0x7ff4a40a3bb0 0x7ff4a40a6070 secure :-1 s=READY pgs=174 cs=0 l=1 rev1=1 crypto rx=0x7ff4ac005cc0 tx=0x7ff4ac005c30 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T10:24:49.506 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:24:49.507+0000 7ff4ab7fe640 1 -- 172.21.15.114:0/3560602326 <== mon.0 v2:172.21.15.114:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7ff4b4014030 con 0x7ff4bc107c70 2024-04-23T10:24:49.709 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:24:49.710+0000 7ff4c0ffe640 1 -- 172.21.15.114:0/3560602326 --> [v2:172.21.15.114:6800/465333808,v1:172.21.15.114:6801/465333808] -- mgr_command(tid 0: {"prefix": "fs subvolume create", "vol_name": "cephfs", "sub_name": "sv_1", "group_name": "qa", "target": ["mon-mgr", ""]}) -- 0x7ff4bc062390 con 0x7ff4a40a3bb0 2024-04-23T10:24:49.735 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:24:49.737+0000 7ff4ab7fe640 1 -- 172.21.15.114:0/3560602326 <== mgr.14152 v2:172.21.15.114:6800/465333808 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+0 (secure 0 0 0) 0x7ff4bc062390 con 0x7ff4a40a3bb0 2024-04-23T10:24:49.738 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:24:49.739+0000 7ff4c0ffe640 1 -- 172.21.15.114:0/3560602326 >> [v2:172.21.15.114:6800/465333808,v1:172.21.15.114:6801/465333808] conn(0x7ff4a40a3bb0 msgr2=0x7ff4a40a6070 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T10:24:49.738 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:24:49.739+0000 7ff4c0ffe640 1 --2- 172.21.15.114:0/3560602326 >> [v2:172.21.15.114:6800/465333808,v1:172.21.15.114:6801/465333808] conn(0x7ff4a40a3bb0 0x7ff4a40a6070 secure :-1 s=READY pgs=174 cs=0 l=1 rev1=1 crypto rx=0x7ff4ac005cc0 tx=0x7ff4ac005c30 comp rx=0 tx=0).stop 2024-04-23T10:24:49.738 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:24:49.739+0000 7ff4c0ffe640 1 -- 172.21.15.114:0/3560602326 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7ff4bc107c70 msgr2=0x7ff4bc19f7b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T10:24:49.738 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:24:49.739+0000 7ff4c0ffe640 1 --2- 172.21.15.114:0/3560602326 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7ff4bc107c70 0x7ff4bc19f7b0 secure :-1 s=READY pgs=231 cs=0 l=1 rev1=1 crypto rx=0x7ff4b400c970 tx=0x7ff4b400cd70 comp rx=0 tx=0).stop 2024-04-23T10:24:49.739 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:24:49.739+0000 7ff4c0ffe640 1 -- 172.21.15.114:0/3560602326 shutdown_connections 2024-04-23T10:24:49.739 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:24:49.739+0000 7ff4c0ffe640 1 --2- 172.21.15.114:0/3560602326 >> [v2:172.21.15.114:6800/465333808,v1:172.21.15.114:6801/465333808] conn(0x7ff4a40a3bb0 0x7ff4a40a6070 unknown :-1 s=CLOSED pgs=174 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:24:49.739 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:24:49.739+0000 7ff4c0ffe640 1 --2- 172.21.15.114:0/3560602326 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7ff4bc107c70 0x7ff4bc19f7b0 unknown :-1 s=CLOSED pgs=231 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:24:49.739 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:24:49.739+0000 7ff4c0ffe640 1 --2- 172.21.15.114:0/3560602326 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7ff4bc105340 0x7ff4bc198740 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:24:49.739 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:24:49.739+0000 7ff4c0ffe640 1 --2- 172.21.15.114:0/3560602326 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7ff4bc102a10 0x7ff4bc198200 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:24:49.739 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:24:49.739+0000 7ff4c0ffe640 1 -- 172.21.15.114:0/3560602326 >> 172.21.15.114:0/3560602326 conn(0x7ff4bc0fc400 msgr2=0x7ff4bc103370 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T10:24:49.739 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:24:49.739+0000 7ff4c0ffe640 1 -- 172.21.15.114:0/3560602326 shutdown_connections 2024-04-23T10:24:49.739 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:24:49.739+0000 7ff4c0ffe640 1 -- 172.21.15.114:0/3560602326 wait complete. 2024-04-23T10:24:49.748 INFO:journalctl@ceph.mon.a.smithi114.stdout:Apr 23 10:24:49 smithi114 ceph-mon[29255]: from='client.14742 -' entity='client.admin' cmd=[{"prefix": "fs subvolumegroup create", "vol_name": "cephfs", "group_name": "qa", "target": ["mon-mgr", ""]}]: dispatch 2024-04-23T10:24:49.748 INFO:journalctl@ceph.mon.a.smithi114.stdout:Apr 23 10:24:49 smithi114 ceph-mon[29255]: from='client.14754 -' entity='client.admin' cmd=[{"prefix": "fs subvolume create", "vol_name": "cephfs", "sub_name": "sv_0", "group_name": "qa", "target": ["mon-mgr", ""]}]: dispatch 2024-04-23T10:24:49.749 DEBUG:teuthology.orchestra.run.smithi114:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs subvolume getpath cephfs --group_name qa sv_1 2024-04-23T10:24:49.828 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 23 10:24:49 smithi204 ceph-mon[32649]: from='client.14742 -' entity='client.admin' cmd=[{"prefix": "fs subvolumegroup create", "vol_name": "cephfs", "group_name": "qa", "target": ["mon-mgr", ""]}]: dispatch 2024-04-23T10:24:49.828 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 23 10:24:49 smithi204 ceph-mon[32649]: from='client.14754 -' entity='client.admin' cmd=[{"prefix": "fs subvolume create", "vol_name": "cephfs", "sub_name": "sv_0", "group_name": "qa", "target": ["mon-mgr", ""]}]: dispatch 2024-04-23T10:24:49.853 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:24:49.852+0000 7f42c5cea640 1 -- 172.21.15.114:0/4194934848 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f42c0107c70 msgr2=0x7f42c010a0c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T10:24:49.854 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:24:49.852+0000 7f42c5cea640 1 --2- 172.21.15.114:0/4194934848 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f42c0107c70 0x7f42c010a0c0 secure :-1 s=READY pgs=232 cs=0 l=1 rev1=1 crypto rx=0x7f42b8009ca0 tx=0x7f42b802a8c0 comp rx=0 tx=0).stop 2024-04-23T10:24:49.854 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:24:49.853+0000 7f42c5cea640 1 -- 172.21.15.114:0/4194934848 shutdown_connections 2024-04-23T10:24:49.854 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:24:49.853+0000 7f42c5cea640 1 --2- 172.21.15.114:0/4194934848 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f42c0107c70 0x7f42c010a0c0 unknown :-1 s=CLOSED pgs=232 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:24:49.854 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:24:49.853+0000 7f42c5cea640 1 --2- 172.21.15.114:0/4194934848 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f42c0105340 0x7f42c0107730 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:24:49.854 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:24:49.853+0000 7f42c5cea640 1 --2- 172.21.15.114:0/4194934848 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f42c0102a10 0x7f42c0104e00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:24:49.854 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:24:49.853+0000 7f42c5cea640 1 -- 172.21.15.114:0/4194934848 >> 172.21.15.114:0/4194934848 conn(0x7f42c00fc400 msgr2=0x7f42c00fe840 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T10:24:49.854 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:24:49.853+0000 7f42c5cea640 1 -- 172.21.15.114:0/4194934848 shutdown_connections 2024-04-23T10:24:49.854 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:24:49.853+0000 7f42c5cea640 1 -- 172.21.15.114:0/4194934848 wait complete. 2024-04-23T10:24:49.854 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:24:49.853+0000 7f42c5cea640 1 Processor -- start 2024-04-23T10:24:49.854 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:24:49.853+0000 7f42c5cea640 1 -- start start 2024-04-23T10:24:49.854 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:24:49.853+0000 7f42c5cea640 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f42c0102a10 0x7f42c01981d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T10:24:49.854 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:24:49.853+0000 7f42c5cea640 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f42c0105340 0x7f42c0198710 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T10:24:49.854 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:24:49.853+0000 7f42c5cea640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f42c0107c70 0x7f42c019f780 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T10:24:49.854 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:24:49.853+0000 7f42c5cea640 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x7f42c010dad0 con 0x7f42c0107c70 2024-04-23T10:24:49.855 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:24:49.853+0000 7f42c5cea640 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 -- 0x7f42c010d950 con 0x7f42c0105340 2024-04-23T10:24:49.855 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:24:49.853+0000 7f42c5cea640 1 -- --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_getmap magic: 0 -- 0x7f42c010dc50 con 0x7f42c0102a10 2024-04-23T10:24:49.855 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:24:49.853+0000 7f42bffff640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f42c0107c70 0x7f42c019f780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T10:24:49.855 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:24:49.853+0000 7f42bffff640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f42c0107c70 0x7f42c019f780 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.114:43398/0 (socket says 172.21.15.114:43398) 2024-04-23T10:24:49.855 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:24:49.853+0000 7f42bffff640 1 -- 172.21.15.114:0/2980259448 learned_addr learned my addr 172.21.15.114:0/2980259448 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-04-23T10:24:49.855 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:24:49.853+0000 7f42bf7fe640 1 --2- 172.21.15.114:0/2980259448 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f42c0102a10 0x7f42c01981d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T10:24:49.855 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:24:49.854+0000 7f42bffff640 1 -- 172.21.15.114:0/2980259448 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f42c0102a10 msgr2=0x7f42c01981d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T10:24:49.855 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:24:49.854+0000 7f42bffff640 1 --2- 172.21.15.114:0/2980259448 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f42c0102a10 0x7f42c01981d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:24:49.855 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:24:49.854+0000 7f42b7fff640 1 --2- 172.21.15.114:0/2980259448 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f42c0105340 0x7f42c0198710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T10:24:49.855 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:24:49.854+0000 7f42bffff640 1 -- 172.21.15.114:0/2980259448 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f42c0105340 msgr2=0x7f42c0198710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T10:24:49.855 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:24:49.854+0000 7f42bffff640 1 --2- 172.21.15.114:0/2980259448 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f42c0105340 0x7f42c0198710 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:24:49.855 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:24:49.854+0000 7f42bffff640 1 -- 172.21.15.114:0/2980259448 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f42c019fdf0 con 0x7f42c0107c70 2024-04-23T10:24:49.855 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:24:49.854+0000 7f42bffff640 1 --2- 172.21.15.114:0/2980259448 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f42c0107c70 0x7f42c019f780 secure :-1 s=READY pgs=233 cs=0 l=1 rev1=1 crypto rx=0x7f42b80097b0 tx=0x7f42b8003c20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T10:24:49.855 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:24:49.854+0000 7f42bd7fa640 1 -- 172.21.15.114:0/2980259448 <== mon.0 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f42b8004240 con 0x7f42c0107c70 2024-04-23T10:24:49.855 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:24:49.854+0000 7f42bd7fa640 1 -- 172.21.15.114:0/2980259448 <== mon.0 v2:172.21.15.114:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f42b80317c0 con 0x7f42c0107c70 2024-04-23T10:24:49.855 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:24:49.854+0000 7f42c5cea640 1 -- 172.21.15.114:0/2980259448 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f42c01a0080 con 0x7f42c0107c70 2024-04-23T10:24:49.856 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:24:49.854+0000 7f42bd7fa640 1 -- 172.21.15.114:0/2980259448 <== mon.0 v2:172.21.15.114:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f42b802fd80 con 0x7f42c0107c70 2024-04-23T10:24:49.856 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:24:49.854+0000 7f42c5cea640 1 -- 172.21.15.114:0/2980259448 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f42c01a05d0 con 0x7f42c0107c70 2024-04-23T10:24:49.856 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:24:49.855+0000 7f42c5cea640 1 -- 172.21.15.114:0/2980259448 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f42c00fdef0 con 0x7f42c0107c70 2024-04-23T10:24:49.857 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:24:49.856+0000 7f42bd7fa640 1 -- 172.21.15.114:0/2980259448 <== mon.0 v2:172.21.15.114:3300/0 4 ==== mgrmap(e 16) ==== 138647+0+0 (secure 0 0 0) 0x7f42b80043e0 con 0x7f42c0107c70 2024-04-23T10:24:49.857 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:24:49.857+0000 7f42bd7fa640 1 --2- 172.21.15.114:0/2980259448 >> [v2:172.21.15.114:6800/465333808,v1:172.21.15.114:6801/465333808] conn(0x7f42ac0a3bb0 0x7f42ac0a6070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T10:24:49.857 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:24:49.857+0000 7f42bf7fe640 1 --2- 172.21.15.114:0/2980259448 >> [v2:172.21.15.114:6800/465333808,v1:172.21.15.114:6801/465333808] conn(0x7f42ac0a3bb0 0x7f42ac0a6070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T10:24:49.857 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:24:49.857+0000 7f42bd7fa640 1 -- 172.21.15.114:0/2980259448 <== mon.0 v2:172.21.15.114:3300/0 5 ==== osd_map(87..87 src has 1..87) ==== 7979+0+0 (secure 0 0 0) 0x7f42b80f1cd0 con 0x7f42c0107c70 2024-04-23T10:24:49.857 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:24:49.857+0000 7f42bf7fe640 1 --2- 172.21.15.114:0/2980259448 >> [v2:172.21.15.114:6800/465333808,v1:172.21.15.114:6801/465333808] conn(0x7f42ac0a3bb0 0x7f42ac0a6070 secure :-1 s=READY pgs=175 cs=0 l=1 rev1=1 crypto rx=0x7f42b000bbe0 tx=0x7f42b000b390 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T10:24:49.859 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:24:49.861+0000 7f42bd7fa640 1 -- 172.21.15.114:0/2980259448 <== mon.0 v2:172.21.15.114:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f42b807fff0 con 0x7f42c0107c70 2024-04-23T10:24:50.070 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:24:50.071+0000 7f42c5cea640 1 -- 172.21.15.114:0/2980259448 --> [v2:172.21.15.114:6800/465333808,v1:172.21.15.114:6801/465333808] -- mgr_command(tid 0: {"prefix": "fs subvolume getpath", "vol_name": "cephfs", "group_name": "qa", "sub_name": "sv_1", "target": ["mon-mgr", ""]}) -- 0x7f42c0062390 con 0x7f42ac0a3bb0 2024-04-23T10:24:50.075 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:24:50.076+0000 7f42bd7fa640 1 -- 172.21.15.114:0/2980259448 <== mgr.14152 v2:172.21.15.114:6800/465333808 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+54 (secure 0 0 0) 0x7f42c0062390 con 0x7f42ac0a3bb0 2024-04-23T10:24:50.075 INFO:teuthology.orchestra.run.smithi114.stdout:/volumes/qa/sv_1/b206794e-7622-4ab8-8943-f3ae4e65b7a3 2024-04-23T10:24:50.078 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:24:50.079+0000 7f42c5cea640 1 -- 172.21.15.114:0/2980259448 >> [v2:172.21.15.114:6800/465333808,v1:172.21.15.114:6801/465333808] conn(0x7f42ac0a3bb0 msgr2=0x7f42ac0a6070 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T10:24:50.078 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:24:50.079+0000 7f42c5cea640 1 --2- 172.21.15.114:0/2980259448 >> [v2:172.21.15.114:6800/465333808,v1:172.21.15.114:6801/465333808] conn(0x7f42ac0a3bb0 0x7f42ac0a6070 secure :-1 s=READY pgs=175 cs=0 l=1 rev1=1 crypto rx=0x7f42b000bbe0 tx=0x7f42b000b390 comp rx=0 tx=0).stop 2024-04-23T10:24:50.078 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:24:50.079+0000 7f42c5cea640 1 -- 172.21.15.114:0/2980259448 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f42c0107c70 msgr2=0x7f42c019f780 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T10:24:50.078 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:24:50.079+0000 7f42c5cea640 1 --2- 172.21.15.114:0/2980259448 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f42c0107c70 0x7f42c019f780 secure :-1 s=READY pgs=233 cs=0 l=1 rev1=1 crypto rx=0x7f42b80097b0 tx=0x7f42b8003c20 comp rx=0 tx=0).stop 2024-04-23T10:24:50.078 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:24:50.079+0000 7f42c5cea640 1 -- 172.21.15.114:0/2980259448 shutdown_connections 2024-04-23T10:24:50.078 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:24:50.079+0000 7f42c5cea640 1 --2- 172.21.15.114:0/2980259448 >> [v2:172.21.15.114:6800/465333808,v1:172.21.15.114:6801/465333808] conn(0x7f42ac0a3bb0 0x7f42ac0a6070 unknown :-1 s=CLOSED pgs=175 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:24:50.078 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:24:50.079+0000 7f42c5cea640 1 --2- 172.21.15.114:0/2980259448 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f42c0107c70 0x7f42c019f780 unknown :-1 s=CLOSED pgs=233 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:24:50.078 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:24:50.079+0000 7f42c5cea640 1 --2- 172.21.15.114:0/2980259448 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f42c0105340 0x7f42c0198710 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:24:50.078 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:24:50.079+0000 7f42c5cea640 1 --2- 172.21.15.114:0/2980259448 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f42c0102a10 0x7f42c01981d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:24:50.078 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:24:50.079+0000 7f42c5cea640 1 -- 172.21.15.114:0/2980259448 >> 172.21.15.114:0/2980259448 conn(0x7f42c00fc400 msgr2=0x7f42c01033f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T10:24:50.078 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:24:50.079+0000 7f42c5cea640 1 -- 172.21.15.114:0/2980259448 shutdown_connections 2024-04-23T10:24:50.078 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:24:50.079+0000 7f42c5cea640 1 -- 172.21.15.114:0/2980259448 wait complete. 2024-04-23T10:24:50.088 DEBUG:teuthology.orchestra.run.smithi114:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-23T10:24:50.188 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:24:50.188+0000 7fef4b47c640 1 -- 172.21.15.114:0/1990512412 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fef44100e60 msgr2=0x7fef44101280 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T10:24:50.189 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:24:50.188+0000 7fef4b47c640 1 --2- 172.21.15.114:0/1990512412 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fef44100e60 0x7fef44101280 secure :-1 s=READY pgs=234 cs=0 l=1 rev1=1 crypto rx=0x7fef40009900 tx=0x7fef4002a410 comp rx=0 tx=0).stop 2024-04-23T10:24:50.189 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:24:50.188+0000 7fef4b47c640 1 -- 172.21.15.114:0/1990512412 shutdown_connections 2024-04-23T10:24:50.189 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:24:50.188+0000 7fef4b47c640 1 --2- 172.21.15.114:0/1990512412 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fef44102a00 0x7fef44109320 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:24:50.189 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:24:50.188+0000 7fef4b47c640 1 --2- 172.21.15.114:0/1990512412 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fef44102040 0x7fef441024c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:24:50.189 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:24:50.188+0000 7fef4b47c640 1 --2- 172.21.15.114:0/1990512412 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fef44100e60 0x7fef44101280 unknown :-1 s=CLOSED pgs=234 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:24:50.189 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:24:50.188+0000 7fef4b47c640 1 -- 172.21.15.114:0/1990512412 >> 172.21.15.114:0/1990512412 conn(0x7fef440fc440 msgr2=0x7fef440fe860 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T10:24:50.189 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:24:50.188+0000 7fef4b47c640 1 -- 172.21.15.114:0/1990512412 shutdown_connections 2024-04-23T10:24:50.189 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:24:50.188+0000 7fef4b47c640 1 -- 172.21.15.114:0/1990512412 wait complete. 2024-04-23T10:24:50.189 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:24:50.188+0000 7fef4b47c640 1 Processor -- start 2024-04-23T10:24:50.189 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:24:50.188+0000 7fef4b47c640 1 -- start start 2024-04-23T10:24:50.189 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:24:50.188+0000 7fef4b47c640 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fef44100e60 0x7fef44198240 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T10:24:50.189 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:24:50.189+0000 7fef4b47c640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fef44102040 0x7fef44198780 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T10:24:50.189 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:24:50.189+0000 7fef4b47c640 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fef44102a00 0x7fef4419f7f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T10:24:50.189 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:24:50.189+0000 7fef4b47c640 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x7fef4410db70 con 0x7fef44102040 2024-04-23T10:24:50.189 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:24:50.189+0000 7fef4b47c640 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 -- 0x7fef4410d9f0 con 0x7fef44100e60 2024-04-23T10:24:50.189 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:24:50.189+0000 7fef4b47c640 1 -- --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_getmap magic: 0 -- 0x7fef4410dcf0 con 0x7fef44102a00 2024-04-23T10:24:50.190 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:24:50.189+0000 7fef48a17640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fef44102040 0x7fef44198780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T10:24:50.190 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:24:50.189+0000 7fef48a17640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fef44102040 0x7fef44198780 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.114:43422/0 (socket says 172.21.15.114:43422) 2024-04-23T10:24:50.190 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:24:50.189+0000 7fef48a17640 1 -- 172.21.15.114:0/3392172770 learned_addr learned my addr 172.21.15.114:0/3392172770 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-04-23T10:24:50.190 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:24:50.189+0000 7fef49a19640 1 --2- 172.21.15.114:0/3392172770 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fef44102a00 0x7fef4419f7f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T10:24:50.190 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:24:50.189+0000 7fef48a17640 1 -- 172.21.15.114:0/3392172770 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fef44102a00 msgr2=0x7fef4419f7f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T10:24:50.190 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:24:50.189+0000 7fef48a17640 1 --2- 172.21.15.114:0/3392172770 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fef44102a00 0x7fef4419f7f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:24:50.190 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:24:50.189+0000 7fef48a17640 1 -- 172.21.15.114:0/3392172770 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fef44100e60 msgr2=0x7fef44198240 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T10:24:50.190 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:24:50.189+0000 7fef48a17640 1 --2- 172.21.15.114:0/3392172770 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fef44100e60 0x7fef44198240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:24:50.190 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:24:50.189+0000 7fef48a17640 1 -- 172.21.15.114:0/3392172770 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fef4419fe60 con 0x7fef44102040 2024-04-23T10:24:50.190 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:24:50.189+0000 7fef48a17640 1 --2- 172.21.15.114:0/3392172770 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fef44102040 0x7fef44198780 secure :-1 s=READY pgs=235 cs=0 l=1 rev1=1 crypto rx=0x7fef3400b4a0 tx=0x7fef3400b960 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T10:24:50.190 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:24:50.189+0000 7fef3a7fc640 1 -- 172.21.15.114:0/3392172770 <== mon.0 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fef34014070 con 0x7fef44102040 2024-04-23T10:24:50.190 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:24:50.189+0000 7fef4b47c640 1 -- 172.21.15.114:0/3392172770 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fef441a0150 con 0x7fef44102040 2024-04-23T10:24:50.190 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:24:50.189+0000 7fef3a7fc640 1 -- 172.21.15.114:0/3392172770 <== mon.0 v2:172.21.15.114:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fef34010ac0 con 0x7fef44102040 2024-04-23T10:24:50.190 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:24:50.189+0000 7fef3a7fc640 1 -- 172.21.15.114:0/3392172770 <== mon.0 v2:172.21.15.114:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fef34017c30 con 0x7fef44102040 2024-04-23T10:24:50.191 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:24:50.189+0000 7fef4b47c640 1 -- 172.21.15.114:0/3392172770 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fef441a0720 con 0x7fef44102040 2024-04-23T10:24:50.191 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:24:50.190+0000 7fef4b47c640 1 -- 172.21.15.114:0/3392172770 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fef44108220 con 0x7fef44102040 2024-04-23T10:24:50.192 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:24:50.191+0000 7fef3a7fc640 1 -- 172.21.15.114:0/3392172770 <== mon.0 v2:172.21.15.114:3300/0 4 ==== mgrmap(e 16) ==== 138647+0+0 (secure 0 0 0) 0x7fef34017070 con 0x7fef44102040 2024-04-23T10:24:50.192 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:24:50.192+0000 7fef3a7fc640 1 --2- 172.21.15.114:0/3392172770 >> [v2:172.21.15.114:6800/465333808,v1:172.21.15.114:6801/465333808] conn(0x7fef300a3bb0 0x7fef300a6070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T10:24:50.192 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:24:50.192+0000 7fef49218640 1 --2- 172.21.15.114:0/3392172770 >> [v2:172.21.15.114:6800/465333808,v1:172.21.15.114:6801/465333808] conn(0x7fef300a3bb0 0x7fef300a6070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T10:24:50.192 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:24:50.192+0000 7fef3a7fc640 1 -- 172.21.15.114:0/3392172770 <== mon.0 v2:172.21.15.114:3300/0 5 ==== osd_map(87..87 src has 1..87) ==== 7979+0+0 (secure 0 0 0) 0x7fef340cfa80 con 0x7fef44102040 2024-04-23T10:24:50.192 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:24:50.192+0000 7fef49218640 1 --2- 172.21.15.114:0/3392172770 >> [v2:172.21.15.114:6800/465333808,v1:172.21.15.114:6801/465333808] conn(0x7fef300a3bb0 0x7fef300a6070 secure :-1 s=READY pgs=176 cs=0 l=1 rev1=1 crypto rx=0x7fef400096c0 tx=0x7fef40000bc0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T10:24:50.194 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:24:50.195+0000 7fef3a7fc640 1 -- 172.21.15.114:0/3392172770 <== mon.0 v2:172.21.15.114:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7fef3409c5b0 con 0x7fef44102040 2024-04-23T10:24:50.362 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:24:50.363+0000 7fef3a7fc640 1 -- 172.21.15.114:0/3392172770 <== mon.0 v2:172.21.15.114:3300/0 7 ==== mgrmap(e 17) ==== 138698+0+0 (secure 0 0 0) 0x7fef34010c70 con 0x7fef44102040 2024-04-23T10:24:50.421 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:24:50.422+0000 7fef4b47c640 1 -- 172.21.15.114:0/3392172770 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fef4402d5c0 con 0x7fef44102040 2024-04-23T10:24:50.422 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:24:50.423+0000 7fef3a7fc640 1 -- 172.21.15.114:0/3392172770 <== mon.0 v2:172.21.15.114:3300/0 8 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 20 v20) ==== 94+0+11884 (secure 0 0 0) 0x7fef340c7b50 con 0x7fef44102040 2024-04-23T10:24:50.422 INFO:teuthology.orchestra.run.smithi114.stdout: 2024-04-23T10:24:50.422 INFO:teuthology.orchestra.run.smithi114.stdout:{"epoch":20,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14547,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.114:6835/663493580","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6834","nonce":663493580},{"type":"v1","addr":"172.21.15.114:6835","nonce":663493580}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14565,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.114:6837/1439252154","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6836","nonce":1439252154},{"type":"v1","addr":"172.21.15.114:6837","nonce":1439252154}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5},{"gid":14634,"name":"g","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.114:6839/3849796939","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6838","nonce":3849796939},{"type":"v1","addr":"172.21.15.114:6839","nonce":3849796939}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":11},{"gid":14646,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.114:6841/3544523891","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6840","nonce":3544523891},{"type":"v1","addr":"172.21.15.114:6841","nonce":3544523891}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":12},{"gid":24385,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.204:6833/3866561754","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6832","nonce":3866561754},{"type":"v1","addr":"172.21.15.204:6833","nonce":3866561754}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2},{"gid":24427,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.204:6835/2533793172","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6834","nonce":2533793172},{"type":"v1","addr":"172.21.15.204:6835","nonce":2533793172}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":7}],"filesystems":[{"mdsmap":{"epoch":20,"flags":50,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-04-23T10:24:43.383158+0000","modified":"2024-04-23T10:24:48.315128+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":3,"in":[0,1,2],"up":{"mds_0":24545,"mds_1":24509,"mds_2":24479},"failed":[],"damaged":[],"stopped":[],"info":{"gid_24439":{"gid":24439,"name":"c","rank":2,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.204:6837/513822179","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6836","nonce":513822179},{"type":"v1","addr":"172.21.15.204:6837","nonce":513822179}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24443":{"gid":24443,"name":"k","rank":1,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.119:6833/783244660","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6832","nonce":783244660},{"type":"v1","addr":"172.21.15.119:6833","nonce":783244660}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24463":{"gid":24463,"name":"i","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.204:6839/3222658893","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6838","nonce":3222658893},{"type":"v1","addr":"172.21.15.204:6839","nonce":3222658893}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24479":{"gid":24479,"name":"e","rank":2,"incarnation":19,"state":"up:active","state_seq":9,"addr":"172.21.15.119:6835/2033916709","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6834","nonce":2033916709},{"type":"v1","addr":"172.21.15.119:6835","nonce":2033916709}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24509":{"gid":24509,"name":"h","rank":1,"incarnation":17,"state":"up:active","state_seq":6,"addr":"172.21.15.119:6837/1872363128","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6836","nonce":1872363128},{"type":"v1","addr":"172.21.15.119:6837","nonce":1872363128}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24545":{"gid":24545,"name":"b","rank":0,"incarnation":14,"state":"up:active","state_seq":3,"addr":"172.21.15.119:6839/937297340","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6838","nonce":937297340},{"type":"v1","addr":"172.21.15.119:6839","nonce":937297340}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":24545,"qdb_cluster":[24509,24545,24479]},"id":1}]} 2024-04-23T10:24:50.423 INFO:teuthology.orchestra.run.smithi114.stderr:dumped fsmap epoch 20 2024-04-23T10:24:50.424 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:24:50.425+0000 7fef4b47c640 1 -- 172.21.15.114:0/3392172770 >> [v2:172.21.15.114:6800/465333808,v1:172.21.15.114:6801/465333808] conn(0x7fef300a3bb0 msgr2=0x7fef300a6070 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T10:24:50.424 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:24:50.425+0000 7fef4b47c640 1 --2- 172.21.15.114:0/3392172770 >> [v2:172.21.15.114:6800/465333808,v1:172.21.15.114:6801/465333808] conn(0x7fef300a3bb0 0x7fef300a6070 secure :-1 s=READY pgs=176 cs=0 l=1 rev1=1 crypto rx=0x7fef400096c0 tx=0x7fef40000bc0 comp rx=0 tx=0).stop 2024-04-23T10:24:50.424 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:24:50.425+0000 7fef4b47c640 1 -- 172.21.15.114:0/3392172770 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fef44102040 msgr2=0x7fef44198780 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T10:24:50.424 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:24:50.425+0000 7fef4b47c640 1 --2- 172.21.15.114:0/3392172770 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fef44102040 0x7fef44198780 secure :-1 s=READY pgs=235 cs=0 l=1 rev1=1 crypto rx=0x7fef3400b4a0 tx=0x7fef3400b960 comp rx=0 tx=0).stop 2024-04-23T10:24:50.424 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:24:50.425+0000 7fef4b47c640 1 -- 172.21.15.114:0/3392172770 shutdown_connections 2024-04-23T10:24:50.425 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:24:50.425+0000 7fef4b47c640 1 --2- 172.21.15.114:0/3392172770 >> [v2:172.21.15.114:6800/465333808,v1:172.21.15.114:6801/465333808] conn(0x7fef300a3bb0 0x7fef300a6070 unknown :-1 s=CLOSED pgs=176 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:24:50.425 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:24:50.425+0000 7fef4b47c640 1 --2- 172.21.15.114:0/3392172770 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fef44102a00 0x7fef4419f7f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:24:50.425 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:24:50.425+0000 7fef4b47c640 1 --2- 172.21.15.114:0/3392172770 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fef44102040 0x7fef44198780 unknown :-1 s=CLOSED pgs=235 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:24:50.425 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:24:50.425+0000 7fef4b47c640 1 --2- 172.21.15.114:0/3392172770 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fef44100e60 0x7fef44198240 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:24:50.425 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:24:50.425+0000 7fef4b47c640 1 -- 172.21.15.114:0/3392172770 >> 172.21.15.114:0/3392172770 conn(0x7fef440fc440 msgr2=0x7fef441053f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T10:24:50.425 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:24:50.425+0000 7fef4b47c640 1 -- 172.21.15.114:0/3392172770 shutdown_connections 2024-04-23T10:24:50.425 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:24:50.425+0000 7fef4b47c640 1 -- 172.21.15.114:0/3392172770 wait complete. 2024-04-23T10:24:50.435 DEBUG:teuthology.orchestra.run.smithi114:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-04-23T10:24:50.535 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:24:50.534+0000 7f9a379fe640 1 -- 172.21.15.114:0/236245431 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f9a301029a0 msgr2=0x7f9a301092c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T10:24:50.535 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:24:50.534+0000 7f9a379fe640 1 --2- 172.21.15.114:0/236245431 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f9a301029a0 0x7f9a301092c0 secure :-1 s=READY pgs=236 cs=0 l=1 rev1=1 crypto rx=0x7f9a2c009ca0 tx=0x7f9a2c02a8c0 comp rx=0 tx=0).stop 2024-04-23T10:24:50.535 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:24:50.534+0000 7f9a379fe640 1 -- 172.21.15.114:0/236245431 shutdown_connections 2024-04-23T10:24:50.535 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:24:50.534+0000 7f9a379fe640 1 --2- 172.21.15.114:0/236245431 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f9a301029a0 0x7f9a301092c0 unknown :-1 s=CLOSED pgs=236 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:24:50.535 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:24:50.534+0000 7f9a379fe640 1 --2- 172.21.15.114:0/236245431 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f9a30101fe0 0x7f9a30102460 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:24:50.535 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:24:50.534+0000 7f9a379fe640 1 --2- 172.21.15.114:0/236245431 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f9a30100e90 0x7f9a301012b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:24:50.535 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:24:50.534+0000 7f9a379fe640 1 -- 172.21.15.114:0/236245431 >> 172.21.15.114:0/236245431 conn(0x7f9a300fc400 msgr2=0x7f9a300fe840 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T10:24:50.536 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:24:50.534+0000 7f9a379fe640 1 -- 172.21.15.114:0/236245431 shutdown_connections 2024-04-23T10:24:50.536 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:24:50.534+0000 7f9a379fe640 1 -- 172.21.15.114:0/236245431 wait complete. 2024-04-23T10:24:50.536 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:24:50.534+0000 7f9a379fe640 1 Processor -- start 2024-04-23T10:24:50.536 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:24:50.534+0000 7f9a379fe640 1 -- start start 2024-04-23T10:24:50.536 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:24:50.534+0000 7f9a379fe640 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f9a30100e90 0x7f9a30069c50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T10:24:50.536 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:24:50.534+0000 7f9a379fe640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f9a30101fe0 0x7f9a3019f2e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T10:24:50.536 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:24:50.534+0000 7f9a379fe640 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f9a301029a0 0x7f9a301a1830 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T10:24:50.536 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:24:50.534+0000 7f9a379fe640 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x7f9a3010c9f0 con 0x7f9a30101fe0 2024-04-23T10:24:50.536 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:24:50.534+0000 7f9a379fe640 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 -- 0x7f9a3010c870 con 0x7f9a301029a0 2024-04-23T10:24:50.536 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:24:50.534+0000 7f9a379fe640 1 -- --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_getmap magic: 0 -- 0x7f9a3010cb70 con 0x7f9a30100e90 2024-04-23T10:24:50.536 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:24:50.534+0000 7f9a34f99640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f9a30101fe0 0x7f9a3019f2e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T10:24:50.536 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:24:50.535+0000 7f9a34f99640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f9a30101fe0 0x7f9a3019f2e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.114:43452/0 (socket says 172.21.15.114:43452) 2024-04-23T10:24:50.536 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:24:50.535+0000 7f9a34f99640 1 -- 172.21.15.114:0/541711716 learned_addr learned my addr 172.21.15.114:0/541711716 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-04-23T10:24:50.536 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:24:50.535+0000 7f9a3579a640 1 --2- 172.21.15.114:0/541711716 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f9a30100e90 0x7f9a30069c50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T10:24:50.537 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:24:50.535+0000 7f9a35f9b640 1 --2- 172.21.15.114:0/541711716 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f9a301029a0 0x7f9a301a1830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T10:24:50.537 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:24:50.535+0000 7f9a34f99640 1 -- 172.21.15.114:0/541711716 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f9a30100e90 msgr2=0x7f9a30069c50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T10:24:50.537 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:24:50.535+0000 7f9a34f99640 1 --2- 172.21.15.114:0/541711716 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f9a30100e90 0x7f9a30069c50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:24:50.537 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:24:50.535+0000 7f9a34f99640 1 -- 172.21.15.114:0/541711716 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f9a301029a0 msgr2=0x7f9a301a1830 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T10:24:50.537 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:24:50.535+0000 7f9a34f99640 1 --2- 172.21.15.114:0/541711716 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f9a301029a0 0x7f9a301a1830 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:24:50.537 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:24:50.535+0000 7f9a34f99640 1 -- 172.21.15.114:0/541711716 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f9a301a2020 con 0x7f9a30101fe0 2024-04-23T10:24:50.537 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:24:50.535+0000 7f9a34f99640 1 --2- 172.21.15.114:0/541711716 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f9a30101fe0 0x7f9a3019f2e0 secure :-1 s=READY pgs=237 cs=0 l=1 rev1=1 crypto rx=0x7f9a2000d770 tx=0x7f9a2000dc30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T10:24:50.537 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:24:50.535+0000 7f9a267fc640 1 -- 172.21.15.114:0/541711716 <== mon.0 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9a20017070 con 0x7f9a30101fe0 2024-04-23T10:24:50.537 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:24:50.535+0000 7f9a267fc640 1 -- 172.21.15.114:0/541711716 <== mon.0 v2:172.21.15.114:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f9a20009c30 con 0x7f9a30101fe0 2024-04-23T10:24:50.537 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:24:50.535+0000 7f9a379fe640 1 -- 172.21.15.114:0/541711716 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f9a301a22b0 con 0x7f9a30101fe0 2024-04-23T10:24:50.537 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:24:50.535+0000 7f9a267fc640 1 -- 172.21.15.114:0/541711716 <== mon.0 v2:172.21.15.114:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9a20003cc0 con 0x7f9a30101fe0 2024-04-23T10:24:50.537 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:24:50.535+0000 7f9a379fe640 1 -- 172.21.15.114:0/541711716 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f9a301a2800 con 0x7f9a30101fe0 2024-04-23T10:24:50.537 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:24:50.536+0000 7f9a379fe640 1 -- 172.21.15.114:0/541711716 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f9a301081a0 con 0x7f9a30101fe0 2024-04-23T10:24:50.538 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:24:50.537+0000 7f9a267fc640 1 -- 172.21.15.114:0/541711716 <== mon.0 v2:172.21.15.114:3300/0 4 ==== mgrmap(e 17) ==== 138698+0+0 (secure 0 0 0) 0x7f9a20003e60 con 0x7f9a30101fe0 2024-04-23T10:24:50.538 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:24:50.538+0000 7f9a267fc640 1 --2- 172.21.15.114:0/541711716 >> [v2:172.21.15.114:6800/465333808,v1:172.21.15.114:6801/465333808] conn(0x7f9a1c0a3c70 0x7f9a1c0a6130 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T10:24:50.538 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:24:50.538+0000 7f9a3579a640 1 --2- 172.21.15.114:0/541711716 >> [v2:172.21.15.114:6800/465333808,v1:172.21.15.114:6801/465333808] conn(0x7f9a1c0a3c70 0x7f9a1c0a6130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T10:24:50.538 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:24:50.538+0000 7f9a267fc640 1 -- 172.21.15.114:0/541711716 <== mon.0 v2:172.21.15.114:3300/0 5 ==== osd_map(87..87 src has 1..87) ==== 7979+0+0 (secure 0 0 0) 0x7f9a200d00b0 con 0x7f9a30101fe0 2024-04-23T10:24:50.538 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:24:50.538+0000 7f9a3579a640 1 --2- 172.21.15.114:0/541711716 >> [v2:172.21.15.114:6800/465333808,v1:172.21.15.114:6801/465333808] conn(0x7f9a1c0a3c70 0x7f9a1c0a6130 secure :-1 s=READY pgs=177 cs=0 l=1 rev1=1 crypto rx=0x7f9a28009bd0 tx=0x7f9a28005f90 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T10:24:50.540 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:24:50.542+0000 7f9a267fc640 1 -- 172.21.15.114:0/541711716 <== mon.0 v2:172.21.15.114:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f9a2009c490 con 0x7f9a30101fe0 2024-04-23T10:24:50.718 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:24:50.720+0000 7f9a379fe640 1 -- 172.21.15.114:0/541711716 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f9a30062b50 con 0x7f9a30101fe0 2024-04-23T10:24:50.720 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:24:50.721+0000 7f9a267fc640 1 -- 172.21.15.114:0/541711716 <== mon.0 v2:172.21.15.114:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v87) ==== 74+0+22291 (secure 0 0 0) 0x7f9a200a0790 con 0x7f9a30101fe0 2024-04-23T10:24:50.720 INFO:teuthology.orchestra.run.smithi114.stdout: 2024-04-23T10:24:50.720 INFO:teuthology.orchestra.run.smithi114.stdout:{"epoch":87,"fsid":"aa240572-0159-11ef-bc93-c7b262605968","created":"2024-04-23T10:16:53.351136+0000","modified":"2024-04-23T10:24:45.032245+0000","last_up_change":"2024-04-23T10:23:54.153269+0000","last_in_change":"2024-04-23T10:23:32.963085+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":26,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-04-23T10:20:11.970297+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":12,"score_stable":12,"optimal_score":0.25,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-04-23T10:24:41.218540+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"86","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":2.25,"score_stable":2.25,"optimal_score":1,"raw_score_acting":2.25,"raw_score_stable":2.25,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-04-23T10:24:42.341799+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"87","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"fd64adde-9445-45a4-a171-ef656e2c5709","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":85,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6802","nonce":736366878},{"type":"v1","addr":"172.21.15.114:6803","nonce":736366878}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6804","nonce":736366878},{"type":"v1","addr":"172.21.15.114:6805","nonce":736366878}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6808","nonce":736366878},{"type":"v1","addr":"172.21.15.114:6809","nonce":736366878}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6806","nonce":736366878},{"type":"v1","addr":"172.21.15.114:6807","nonce":736366878}]},"public_addr":"172.21.15.114:6803/736366878","cluster_addr":"172.21.15.114:6805/736366878","heartbeat_back_addr":"172.21.15.114:6809/736366878","heartbeat_front_addr":"172.21.15.114:6807/736366878","state":["exists","up"]},{"osd":1,"uuid":"e279a1e6-2702-472a-930a-e08ece76a861","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":85,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6800","nonce":1133540779},{"type":"v1","addr":"172.21.15.119:6801","nonce":1133540779}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6802","nonce":1133540779},{"type":"v1","addr":"172.21.15.119:6803","nonce":1133540779}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6806","nonce":1133540779},{"type":"v1","addr":"172.21.15.119:6807","nonce":1133540779}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6804","nonce":1133540779},{"type":"v1","addr":"172.21.15.119:6805","nonce":1133540779}]},"public_addr":"172.21.15.119:6801/1133540779","cluster_addr":"172.21.15.119:6803/1133540779","heartbeat_back_addr":"172.21.15.119:6807/1133540779","heartbeat_front_addr":"172.21.15.119:6805/1133540779","state":["exists","up"]},{"osd":2,"uuid":"f4c9f2b4-ff49-4e14-89cd-b477feeb9382","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":85,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6800","nonce":799652511},{"type":"v1","addr":"172.21.15.204:6801","nonce":799652511}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6802","nonce":799652511},{"type":"v1","addr":"172.21.15.204:6803","nonce":799652511}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6806","nonce":799652511},{"type":"v1","addr":"172.21.15.204:6807","nonce":799652511}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6804","nonce":799652511},{"type":"v1","addr":"172.21.15.204:6805","nonce":799652511}]},"public_addr":"172.21.15.204:6801/799652511","cluster_addr":"172.21.15.204:6803/799652511","heartbeat_back_addr":"172.21.15.204:6807/799652511","heartbeat_front_addr":"172.21.15.204:6805/799652511","state":["exists","up"]},{"osd":3,"uuid":"52d2a566-e71c-4bb3-8d6c-5abdd43177c2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":25,"up_thru":85,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6810","nonce":4275595091},{"type":"v1","addr":"172.21.15.114:6811","nonce":4275595091}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6812","nonce":4275595091},{"type":"v1","addr":"172.21.15.114:6813","nonce":4275595091}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6816","nonce":4275595091},{"type":"v1","addr":"172.21.15.114:6817","nonce":4275595091}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6814","nonce":4275595091},{"type":"v1","addr":"172.21.15.114:6815","nonce":4275595091}]},"public_addr":"172.21.15.114:6811/4275595091","cluster_addr":"172.21.15.114:6813/4275595091","heartbeat_back_addr":"172.21.15.114:6817/4275595091","heartbeat_front_addr":"172.21.15.114:6815/4275595091","state":["exists","up"]},{"osd":4,"uuid":"d911a7c1-4bdc-4d28-b8a6-73896a6e7de6","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":33,"up_thru":85,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6808","nonce":1611007172},{"type":"v1","addr":"172.21.15.119:6809","nonce":1611007172}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6810","nonce":1611007172},{"type":"v1","addr":"172.21.15.119:6811","nonce":1611007172}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6814","nonce":1611007172},{"type":"v1","addr":"172.21.15.119:6815","nonce":1611007172}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6812","nonce":1611007172},{"type":"v1","addr":"172.21.15.119:6813","nonce":1611007172}]},"public_addr":"172.21.15.119:6809/1611007172","cluster_addr":"172.21.15.119:6811/1611007172","heartbeat_back_addr":"172.21.15.119:6815/1611007172","heartbeat_front_addr":"172.21.15.119:6813/1611007172","state":["exists","up"]},{"osd":5,"uuid":"f7590e1a-e93c-4c29-b15f-38ee52d1f921","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":39,"up_thru":85,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6808","nonce":1427563312},{"type":"v1","addr":"172.21.15.204:6809","nonce":1427563312}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6810","nonce":1427563312},{"type":"v1","addr":"172.21.15.204:6811","nonce":1427563312}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6814","nonce":1427563312},{"type":"v1","addr":"172.21.15.204:6815","nonce":1427563312}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6812","nonce":1427563312},{"type":"v1","addr":"172.21.15.204:6813","nonce":1427563312}]},"public_addr":"172.21.15.204:6809/1427563312","cluster_addr":"172.21.15.204:6811/1427563312","heartbeat_back_addr":"172.21.15.204:6815/1427563312","heartbeat_front_addr":"172.21.15.204:6813/1427563312","state":["exists","up"]},{"osd":6,"uuid":"534d20e8-3e54-4db8-a845-6119f49c7a78","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":47,"up_thru":85,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6818","nonce":4072912544},{"type":"v1","addr":"172.21.15.114:6819","nonce":4072912544}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6820","nonce":4072912544},{"type":"v1","addr":"172.21.15.114:6821","nonce":4072912544}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6824","nonce":4072912544},{"type":"v1","addr":"172.21.15.114:6825","nonce":4072912544}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6822","nonce":4072912544},{"type":"v1","addr":"172.21.15.114:6823","nonce":4072912544}]},"public_addr":"172.21.15.114:6819/4072912544","cluster_addr":"172.21.15.114:6821/4072912544","heartbeat_back_addr":"172.21.15.114:6825/4072912544","heartbeat_front_addr":"172.21.15.114:6823/4072912544","state":["exists","up"]},{"osd":7,"uuid":"f0ec22b8-1b48-4859-9d65-b00403aff3ad","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":53,"up_thru":85,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6816","nonce":2237458793},{"type":"v1","addr":"172.21.15.119:6817","nonce":2237458793}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6818","nonce":2237458793},{"type":"v1","addr":"172.21.15.119:6819","nonce":2237458793}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6822","nonce":2237458793},{"type":"v1","addr":"172.21.15.119:6823","nonce":2237458793}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6820","nonce":2237458793},{"type":"v1","addr":"172.21.15.119:6821","nonce":2237458793}]},"public_addr":"172.21.15.119:6817/2237458793","cluster_addr":"172.21.15.119:6819/2237458793","heartbeat_back_addr":"172.21.15.119:6823/2237458793","heartbeat_front_addr":"172.21.15.119:6821/2237458793","state":["exists","up"]},{"osd":8,"uuid":"3ac89686-6d76-4cef-9089-ac29e4b3bcac","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":62,"up_thru":85,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6816","nonce":1426318231},{"type":"v1","addr":"172.21.15.204:6817","nonce":1426318231}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6818","nonce":1426318231},{"type":"v1","addr":"172.21.15.204:6819","nonce":1426318231}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6822","nonce":1426318231},{"type":"v1","addr":"172.21.15.204:6823","nonce":1426318231}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6820","nonce":1426318231},{"type":"v1","addr":"172.21.15.204:6821","nonce":1426318231}]},"public_addr":"172.21.15.204:6817/1426318231","cluster_addr":"172.21.15.204:6819/1426318231","heartbeat_back_addr":"172.21.15.204:6823/1426318231","heartbeat_front_addr":"172.21.15.204:6821/1426318231","state":["exists","up"]},{"osd":9,"uuid":"a80ffe4d-1e98-460f-b046-c4754285cc5a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":69,"up_thru":85,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6826","nonce":2750635721},{"type":"v1","addr":"172.21.15.114:6827","nonce":2750635721}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6828","nonce":2750635721},{"type":"v1","addr":"172.21.15.114:6829","nonce":2750635721}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6832","nonce":2750635721},{"type":"v1","addr":"172.21.15.114:6833","nonce":2750635721}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6830","nonce":2750635721},{"type":"v1","addr":"172.21.15.114:6831","nonce":2750635721}]},"public_addr":"172.21.15.114:6827/2750635721","cluster_addr":"172.21.15.114:6829/2750635721","heartbeat_back_addr":"172.21.15.114:6833/2750635721","heartbeat_front_addr":"172.21.15.114:6831/2750635721","state":["exists","up"]},{"osd":10,"uuid":"cd75a579-bf12-4600-ac09-a8bfce772ced","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":74,"up_thru":85,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6824","nonce":1400187701},{"type":"v1","addr":"172.21.15.119:6825","nonce":1400187701}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6826","nonce":1400187701},{"type":"v1","addr":"172.21.15.119:6827","nonce":1400187701}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6830","nonce":1400187701},{"type":"v1","addr":"172.21.15.119:6831","nonce":1400187701}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6828","nonce":1400187701},{"type":"v1","addr":"172.21.15.119:6829","nonce":1400187701}]},"public_addr":"172.21.15.119:6825/1400187701","cluster_addr":"172.21.15.119:6827/1400187701","heartbeat_back_addr":"172.21.15.119:6831/1400187701","heartbeat_front_addr":"172.21.15.119:6829/1400187701","state":["exists","up"]},{"osd":11,"uuid":"53f037a5-3ec0-483a-80a1-dfee923a68e5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":82,"up_thru":85,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6824","nonce":2407298490},{"type":"v1","addr":"172.21.15.204:6825","nonce":2407298490}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6826","nonce":2407298490},{"type":"v1","addr":"172.21.15.204:6827","nonce":2407298490}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6830","nonce":2407298490},{"type":"v1","addr":"172.21.15.204:6831","nonce":2407298490}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6828","nonce":2407298490},{"type":"v1","addr":"172.21.15.204:6829","nonce":2407298490}]},"public_addr":"172.21.15.204:6825/2407298490","cluster_addr":"172.21.15.204:6827/2407298490","heartbeat_back_addr":"172.21.15.204:6831/2407298490","heartbeat_front_addr":"172.21.15.204:6829/2407298490","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-23T10:19:18.401147+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-23T10:19:42.611299+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-23T10:20:07.277367+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-23T10:20:31.208368+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-23T10:20:55.866225+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-23T10:21:19.961237+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-23T10:21:44.480783+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-23T10:22:09.173641+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-23T10:22:34.030328+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-23T10:22:59.652293+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-23T10:23:25.659673+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-23T10:23:51.168171+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.114:6801/1604678557":"2024-04-24T10:17:45.874609+0000","172.21.15.114:0/4025385854":"2024-04-24T10:17:45.874609+0000","172.21.15.114:0/2228273469":"2024-04-24T10:17:19.910348+0000","172.21.15.114:6800/2831530838":"2024-04-24T10:17:19.910348+0000","172.21.15.114:0/1333447450":"2024-04-24T10:17:45.874609+0000","172.21.15.114:0/2233026840":"2024-04-24T10:17:19.910348+0000","172.21.15.114:0/1902281545":"2024-04-24T10:17:19.910348+0000","172.21.15.114:6801/2831530838":"2024-04-24T10:17:19.910348+0000","172.21.15.114:6800/1604678557":"2024-04-24T10:17:45.874609+0000","172.21.15.114:0/1623553811":"2024-04-24T10:17:45.874609+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-23T10:24:50.722 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:24:50.723+0000 7f9a379fe640 1 -- 172.21.15.114:0/541711716 >> [v2:172.21.15.114:6800/465333808,v1:172.21.15.114:6801/465333808] conn(0x7f9a1c0a3c70 msgr2=0x7f9a1c0a6130 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T10:24:50.722 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:24:50.723+0000 7f9a379fe640 1 --2- 172.21.15.114:0/541711716 >> [v2:172.21.15.114:6800/465333808,v1:172.21.15.114:6801/465333808] conn(0x7f9a1c0a3c70 0x7f9a1c0a6130 secure :-1 s=READY pgs=177 cs=0 l=1 rev1=1 crypto rx=0x7f9a28009bd0 tx=0x7f9a28005f90 comp rx=0 tx=0).stop 2024-04-23T10:24:50.722 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:24:50.723+0000 7f9a379fe640 1 -- 172.21.15.114:0/541711716 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f9a30101fe0 msgr2=0x7f9a3019f2e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T10:24:50.722 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:24:50.723+0000 7f9a379fe640 1 --2- 172.21.15.114:0/541711716 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f9a30101fe0 0x7f9a3019f2e0 secure :-1 s=READY pgs=237 cs=0 l=1 rev1=1 crypto rx=0x7f9a2000d770 tx=0x7f9a2000dc30 comp rx=0 tx=0).stop 2024-04-23T10:24:50.722 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:24:50.723+0000 7f9a379fe640 1 -- 172.21.15.114:0/541711716 shutdown_connections 2024-04-23T10:24:50.722 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:24:50.723+0000 7f9a379fe640 1 --2- 172.21.15.114:0/541711716 >> [v2:172.21.15.114:6800/465333808,v1:172.21.15.114:6801/465333808] conn(0x7f9a1c0a3c70 0x7f9a1c0a6130 unknown :-1 s=CLOSED pgs=177 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:24:50.722 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:24:50.723+0000 7f9a379fe640 1 --2- 172.21.15.114:0/541711716 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f9a301029a0 0x7f9a301a1830 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:24:50.723 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:24:50.723+0000 7f9a379fe640 1 --2- 172.21.15.114:0/541711716 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f9a30101fe0 0x7f9a3019f2e0 unknown :-1 s=CLOSED pgs=237 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:24:50.723 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:24:50.723+0000 7f9a379fe640 1 --2- 172.21.15.114:0/541711716 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f9a30100e90 0x7f9a30069c50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:24:50.723 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:24:50.723+0000 7f9a379fe640 1 -- 172.21.15.114:0/541711716 >> 172.21.15.114:0/541711716 conn(0x7f9a300fc400 msgr2=0x7f9a30105370 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T10:24:50.723 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:24:50.723+0000 7f9a379fe640 1 -- 172.21.15.114:0/541711716 shutdown_connections 2024-04-23T10:24:50.723 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:24:50.723+0000 7f9a379fe640 1 -- 172.21.15.114:0/541711716 wait complete. 2024-04-23T10:24:50.732 INFO:journalctl@ceph.mon.a.smithi114.stdout:Apr 23 10:24:50 smithi114 ceph-mon[29255]: from='client.14760 -' entity='client.admin' cmd=[{"prefix": "fs subvolume getpath", "vol_name": "cephfs", "group_name": "qa", "sub_name": "sv_0", "target": ["mon-mgr", ""]}]: dispatch 2024-04-23T10:24:50.732 INFO:journalctl@ceph.mon.a.smithi114.stdout:Apr 23 10:24:50 smithi114 ceph-mon[29255]: from='client.14766 -' entity='client.admin' cmd=[{"prefix": "fs subvolume create", "vol_name": "cephfs", "sub_name": "sv_1", "group_name": "qa", "target": ["mon-mgr", ""]}]: dispatch 2024-04-23T10:24:50.732 INFO:journalctl@ceph.mon.a.smithi114.stdout:Apr 23 10:24:50 smithi114 ceph-mon[29255]: pgmap v289: 97 pgs: 97 active+clean; 581 KiB data, 370 MiB used, 1.0 TiB / 1.0 TiB avail; 591 B/s rd, 5.5 KiB/s wr, 15 op/s 2024-04-23T10:24:50.732 INFO:journalctl@ceph.mon.a.smithi114.stdout:Apr 23 10:24:50 smithi114 ceph-mon[29255]: from='client.14772 -' entity='client.admin' cmd=[{"prefix": "fs subvolume getpath", "vol_name": "cephfs", "group_name": "qa", "sub_name": "sv_1", "target": ["mon-mgr", ""]}]: dispatch 2024-04-23T10:24:50.733 INFO:tasks.cephfs.filesystem.ceph_manager:waiting for clean 2024-04-23T10:24:50.734 DEBUG:teuthology.orchestra.run.smithi114:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-04-23T10:24:50.742 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 23 10:24:50 smithi119 ceph-mon[32608]: from='client.14760 -' entity='client.admin' cmd=[{"prefix": "fs subvolume getpath", "vol_name": "cephfs", "group_name": "qa", "sub_name": "sv_0", "target": ["mon-mgr", ""]}]: dispatch 2024-04-23T10:24:50.742 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 23 10:24:50 smithi119 ceph-mon[32608]: from='client.14766 -' entity='client.admin' cmd=[{"prefix": "fs subvolume create", "vol_name": "cephfs", "sub_name": "sv_1", "group_name": "qa", "target": ["mon-mgr", ""]}]: dispatch 2024-04-23T10:24:50.742 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 23 10:24:50 smithi119 ceph-mon[32608]: pgmap v289: 97 pgs: 97 active+clean; 581 KiB data, 370 MiB used, 1.0 TiB / 1.0 TiB avail; 591 B/s rd, 5.5 KiB/s wr, 15 op/s 2024-04-23T10:24:50.742 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 23 10:24:50 smithi119 ceph-mon[32608]: from='client.14772 -' entity='client.admin' cmd=[{"prefix": "fs subvolume getpath", "vol_name": "cephfs", "group_name": "qa", "sub_name": "sv_1", "target": ["mon-mgr", ""]}]: dispatch 2024-04-23T10:24:50.828 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 23 10:24:50 smithi204 ceph-mon[32649]: from='client.14760 -' entity='client.admin' cmd=[{"prefix": "fs subvolume getpath", "vol_name": "cephfs", "group_name": "qa", "sub_name": "sv_0", "target": ["mon-mgr", ""]}]: dispatch 2024-04-23T10:24:50.828 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 23 10:24:50 smithi204 ceph-mon[32649]: from='client.14766 -' entity='client.admin' cmd=[{"prefix": "fs subvolume create", "vol_name": "cephfs", "sub_name": "sv_1", "group_name": "qa", "target": ["mon-mgr", ""]}]: dispatch 2024-04-23T10:24:50.828 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 23 10:24:50 smithi204 ceph-mon[32649]: pgmap v289: 97 pgs: 97 active+clean; 581 KiB data, 370 MiB used, 1.0 TiB / 1.0 TiB avail; 591 B/s rd, 5.5 KiB/s wr, 15 op/s 2024-04-23T10:24:50.828 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 23 10:24:50 smithi204 ceph-mon[32649]: from='client.14772 -' entity='client.admin' cmd=[{"prefix": "fs subvolume getpath", "vol_name": "cephfs", "group_name": "qa", "sub_name": "sv_1", "target": ["mon-mgr", ""]}]: dispatch 2024-04-23T10:24:50.835 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:24:50.833+0000 7faa1e9e2640 1 -- 172.21.15.114:0/3923891308 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7faa18105400 msgr2=0x7faa181077f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T10:24:50.835 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:24:50.833+0000 7faa1e9e2640 1 --2- 172.21.15.114:0/3923891308 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7faa18105400 0x7faa181077f0 secure :-1 s=READY pgs=238 cs=0 l=1 rev1=1 crypto rx=0x7faa08009900 tx=0x7faa0802a410 comp rx=0 tx=0).stop 2024-04-23T10:24:50.835 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:24:50.833+0000 7faa1e9e2640 1 -- 172.21.15.114:0/3923891308 shutdown_connections 2024-04-23T10:24:50.835 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:24:50.833+0000 7faa1e9e2640 1 --2- 172.21.15.114:0/3923891308 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7faa18107d30 0x7faa1810a120 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:24:50.835 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:24:50.833+0000 7faa1e9e2640 1 --2- 172.21.15.114:0/3923891308 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7faa18105400 0x7faa181077f0 unknown :-1 s=CLOSED pgs=238 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:24:50.835 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:24:50.833+0000 7faa1e9e2640 1 --2- 172.21.15.114:0/3923891308 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7faa18102ad0 0x7faa18104ec0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:24:50.835 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:24:50.833+0000 7faa1e9e2640 1 -- 172.21.15.114:0/3923891308 >> 172.21.15.114:0/3923891308 conn(0x7faa180fc400 msgr2=0x7faa180fe840 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T10:24:50.835 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:24:50.833+0000 7faa1e9e2640 1 -- 172.21.15.114:0/3923891308 shutdown_connections 2024-04-23T10:24:50.835 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:24:50.833+0000 7faa1e9e2640 1 -- 172.21.15.114:0/3923891308 wait complete. 2024-04-23T10:24:50.835 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:24:50.834+0000 7faa1e9e2640 1 Processor -- start 2024-04-23T10:24:50.835 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:24:50.834+0000 7faa1e9e2640 1 -- start start 2024-04-23T10:24:50.835 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:24:50.834+0000 7faa1e9e2640 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7faa18102ad0 0x7faa181982a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T10:24:50.835 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:24:50.834+0000 7faa1e9e2640 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7faa18105400 0x7faa181987e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T10:24:50.835 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:24:50.834+0000 7faa1e9e2640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7faa18107d30 0x7faa1819f850 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T10:24:50.836 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:24:50.834+0000 7faa1e9e2640 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x7faa1810db40 con 0x7faa18107d30 2024-04-23T10:24:50.836 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:24:50.834+0000 7faa1e9e2640 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 -- 0x7faa1810d9c0 con 0x7faa18102ad0 2024-04-23T10:24:50.836 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:24:50.834+0000 7faa1e9e2640 1 -- --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_getmap magic: 0 -- 0x7faa1810dcc0 con 0x7faa18105400 2024-04-23T10:24:50.836 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:24:50.834+0000 7faa1cf7f640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7faa18107d30 0x7faa1819f850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T10:24:50.836 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:24:50.834+0000 7faa1cf7f640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7faa18107d30 0x7faa1819f850 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.114:43476/0 (socket says 172.21.15.114:43476) 2024-04-23T10:24:50.836 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:24:50.834+0000 7faa1cf7f640 1 -- 172.21.15.114:0/1010351351 learned_addr learned my addr 172.21.15.114:0/1010351351 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-04-23T10:24:50.836 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:24:50.834+0000 7faa177fe640 1 --2- 172.21.15.114:0/1010351351 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7faa18105400 0x7faa181987e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T10:24:50.836 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:24:50.835+0000 7faa17fff640 1 --2- 172.21.15.114:0/1010351351 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7faa18102ad0 0x7faa181982a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T10:24:50.836 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:24:50.835+0000 7faa1cf7f640 1 -- 172.21.15.114:0/1010351351 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7faa18105400 msgr2=0x7faa181987e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T10:24:50.836 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:24:50.835+0000 7faa1cf7f640 1 --2- 172.21.15.114:0/1010351351 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7faa18105400 0x7faa181987e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:24:50.836 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:24:50.835+0000 7faa1cf7f640 1 -- 172.21.15.114:0/1010351351 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7faa18102ad0 msgr2=0x7faa181982a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T10:24:50.836 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:24:50.835+0000 7faa1cf7f640 1 --2- 172.21.15.114:0/1010351351 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7faa18102ad0 0x7faa181982a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:24:50.836 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:24:50.835+0000 7faa1cf7f640 1 -- 172.21.15.114:0/1010351351 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7faa1819fec0 con 0x7faa18107d30 2024-04-23T10:24:50.836 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:24:50.835+0000 7faa1cf7f640 1 --2- 172.21.15.114:0/1010351351 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7faa18107d30 0x7faa1819f850 secure :-1 s=READY pgs=239 cs=0 l=1 rev1=1 crypto rx=0x7faa1000b970 tx=0x7faa1000be30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T10:24:50.836 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:24:50.835+0000 7faa157fa640 1 -- 172.21.15.114:0/1010351351 <== mon.0 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7faa10019070 con 0x7faa18107d30 2024-04-23T10:24:50.837 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:24:50.835+0000 7faa157fa640 1 -- 172.21.15.114:0/1010351351 <== mon.0 v2:172.21.15.114:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7faa10014440 con 0x7faa18107d30 2024-04-23T10:24:50.837 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:24:50.835+0000 7faa1e9e2640 1 -- 172.21.15.114:0/1010351351 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7faa181a01b0 con 0x7faa18107d30 2024-04-23T10:24:50.837 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:24:50.835+0000 7faa157fa640 1 -- 172.21.15.114:0/1010351351 <== mon.0 v2:172.21.15.114:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7faa100075d0 con 0x7faa18107d30 2024-04-23T10:24:50.837 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:24:50.835+0000 7faa1e9e2640 1 -- 172.21.15.114:0/1010351351 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7faa181a0700 con 0x7faa18107d30 2024-04-23T10:24:50.837 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:24:50.836+0000 7faa1e9e2640 1 -- 172.21.15.114:0/1010351351 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7faa180fdef0 con 0x7faa18107d30 2024-04-23T10:24:50.838 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:24:50.837+0000 7faa157fa640 1 -- 172.21.15.114:0/1010351351 <== mon.0 v2:172.21.15.114:3300/0 4 ==== mgrmap(e 17) ==== 138698+0+0 (secure 0 0 0) 0x7faa1001c070 con 0x7faa18107d30 2024-04-23T10:24:50.838 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:24:50.838+0000 7faa157fa640 1 --2- 172.21.15.114:0/1010351351 >> [v2:172.21.15.114:6800/465333808,v1:172.21.15.114:6801/465333808] conn(0x7faa000a3c40 0x7faa000a6100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T10:24:50.838 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:24:50.838+0000 7faa17fff640 1 --2- 172.21.15.114:0/1010351351 >> [v2:172.21.15.114:6800/465333808,v1:172.21.15.114:6801/465333808] conn(0x7faa000a3c40 0x7faa000a6100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T10:24:50.838 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:24:50.838+0000 7faa157fa640 1 -- 172.21.15.114:0/1010351351 <== mon.0 v2:172.21.15.114:3300/0 5 ==== osd_map(87..87 src has 1..87) ==== 7979+0+0 (secure 0 0 0) 0x7faa100cfd90 con 0x7faa18107d30 2024-04-23T10:24:50.838 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:24:50.838+0000 7faa17fff640 1 --2- 172.21.15.114:0/1010351351 >> [v2:172.21.15.114:6800/465333808,v1:172.21.15.114:6801/465333808] conn(0x7faa000a3c40 0x7faa000a6100 secure :-1 s=READY pgs=178 cs=0 l=1 rev1=1 crypto rx=0x7faa0c005fd0 tx=0x7faa0c00b390 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T10:24:50.840 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:24:50.841+0000 7faa157fa640 1 -- 172.21.15.114:0/1010351351 <== mon.0 v2:172.21.15.114:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7faa1009c280 con 0x7faa18107d30 2024-04-23T10:24:51.015 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:24:51.016+0000 7faa1e9e2640 1 -- 172.21.15.114:0/1010351351 --> [v2:172.21.15.114:6800/465333808,v1:172.21.15.114:6801/465333808] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) -- 0x7faa18062390 con 0x7faa000a3c40 2024-04-23T10:24:51.023 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:24:51.024+0000 7faa157fa640 1 -- 172.21.15.114:0/1010351351 <== mgr.14152 v2:172.21.15.114:6800/465333808 1 ==== mgr_command_reply(tid 0: 0 dumped all) ==== 18+0+258753 (secure 0 0 0) 0x7faa18062390 con 0x7faa000a3c40 2024-04-23T10:24:51.027 INFO:teuthology.orchestra.run.smithi114.stdout: 2024-04-23T10:24:51.028 INFO:teuthology.orchestra.run.smithi114.stderr:dumped all 2024-04-23T10:24:51.030 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:24:51.027+0000 7faa1e9e2640 1 -- 172.21.15.114:0/1010351351 >> [v2:172.21.15.114:6800/465333808,v1:172.21.15.114:6801/465333808] conn(0x7faa000a3c40 msgr2=0x7faa000a6100 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T10:24:51.030 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:24:51.027+0000 7faa1e9e2640 1 --2- 172.21.15.114:0/1010351351 >> [v2:172.21.15.114:6800/465333808,v1:172.21.15.114:6801/465333808] conn(0x7faa000a3c40 0x7faa000a6100 secure :-1 s=READY pgs=178 cs=0 l=1 rev1=1 crypto rx=0x7faa0c005fd0 tx=0x7faa0c00b390 comp rx=0 tx=0).stop 2024-04-23T10:24:51.030 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:24:51.027+0000 7faa1e9e2640 1 -- 172.21.15.114:0/1010351351 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7faa18107d30 msgr2=0x7faa1819f850 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T10:24:51.030 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:24:51.027+0000 7faa1e9e2640 1 --2- 172.21.15.114:0/1010351351 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7faa18107d30 0x7faa1819f850 secure :-1 s=READY pgs=239 cs=0 l=1 rev1=1 crypto rx=0x7faa1000b970 tx=0x7faa1000be30 comp rx=0 tx=0).stop 2024-04-23T10:24:51.030 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:24:51.027+0000 7faa1e9e2640 1 -- 172.21.15.114:0/1010351351 shutdown_connections 2024-04-23T10:24:51.031 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:24:51.027+0000 7faa1e9e2640 1 --2- 172.21.15.114:0/1010351351 >> [v2:172.21.15.114:6800/465333808,v1:172.21.15.114:6801/465333808] conn(0x7faa000a3c40 0x7faa000a6100 unknown :-1 s=CLOSED pgs=178 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:24:51.031 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:24:51.027+0000 7faa1e9e2640 1 --2- 172.21.15.114:0/1010351351 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7faa18107d30 0x7faa1819f850 unknown :-1 s=CLOSED pgs=239 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:24:51.031 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:24:51.027+0000 7faa1e9e2640 1 --2- 172.21.15.114:0/1010351351 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7faa18105400 0x7faa181987e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:24:51.031 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:24:51.027+0000 7faa1e9e2640 1 --2- 172.21.15.114:0/1010351351 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7faa18102ad0 0x7faa181982a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:24:51.031 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:24:51.027+0000 7faa1e9e2640 1 -- 172.21.15.114:0/1010351351 >> 172.21.15.114:0/1010351351 conn(0x7faa180fc400 msgr2=0x7faa181034f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T10:24:51.031 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:24:51.027+0000 7faa1e9e2640 1 -- 172.21.15.114:0/1010351351 shutdown_connections 2024-04-23T10:24:51.031 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:24:51.027+0000 7faa1e9e2640 1 -- 172.21.15.114:0/1010351351 wait complete. 2024-04-23T10:24:51.038 INFO:teuthology.orchestra.run.smithi114.stdout:{"pg_ready":true,"pg_map":{"version":289,"stamp":"2024-04-23T10:24:49.957169+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":595399,"num_objects":51,"num_object_clones":0,"num_object_copies":153,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":51,"num_whiteouts":0,"num_read":141,"num_read_kb":123,"num_write":280,"num_write_kb":2976,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":1180736,"num_keys_recovered":0,"num_objects_omap":30,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":227,"ondisk_log_size":227,"up":291,"acting":291,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":262,"num_osds":12,"num_per_pool_osds":12,"num_per_pool_omap_osds":12,"kb":1124941824,"kb_used":378716,"kb_used_data":12860,"kb_used_omap":18,"kb_used_meta":365677,"kb_avail":1124563108,"statfs":{"total":1151940427776,"available":1151552622592,"internally_reserved":0,"allocated":13168640,"data_stored":7137094,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":19142,"internal_metadata":374453562},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":10,"apply_latency_ms":10,"commit_latency_ns":10000000,"apply_latency_ns":10000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":5031,"num_objects":49,"num_object_clones":0,"num_object_copies":147,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":49,"num_whiteouts":0,"num_read":5,"num_read_kb":4,"num_write":99,"num_write_kb":38,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":30,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"6.926225"},"pg_stats":[{"pgid":"3.3f","version":"0'0","reported_seq":10,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-04-23T10:24:44.052035+0000","last_change":"2024-04-23T10:24:44.051465+0000","last_active":"2024-04-23T10:24:44.052035+0000","last_peered":"2024-04-23T10:24:44.052035+0000","last_clean":"2024-04-23T10:24:44.052035+0000","last_became_active":"2024-04-23T10:24:44.050863+0000","last_became_peered":"2024-04-23T10:24:44.050863+0000","last_unstale":"2024-04-23T10:24:44.052035+0000","last_undegraded":"2024-04-23T10:24:44.052035+0000","last_fullsized":"2024-04-23T10:24:44.052035+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T10:24:43.025324+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T10:24:43.025324+0000","last_clean_scrub_stamp":"2024-04-23T10:24:43.025324+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T15:52:57.416236+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,1,0],"acting":[11,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.3e","version":"0'0","reported_seq":13,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-04-23T10:24:45.041886+0000","last_change":"2024-04-23T10:24:44.050739+0000","last_active":"2024-04-23T10:24:45.041886+0000","last_peered":"2024-04-23T10:24:45.041886+0000","last_clean":"2024-04-23T10:24:45.041886+0000","last_became_active":"2024-04-23T10:24:44.050430+0000","last_became_peered":"2024-04-23T10:24:44.050430+0000","last_unstale":"2024-04-23T10:24:45.041886+0000","last_undegraded":"2024-04-23T10:24:45.041886+0000","last_fullsized":"2024-04-23T10:24:45.041886+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T10:24:43.025324+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T10:24:43.025324+0000","last_clean_scrub_stamp":"2024-04-23T10:24:43.025324+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T17:36:25.983147+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,11,10],"acting":[0,11,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3d","version":"0'0","reported_seq":13,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-04-23T10:24:45.041944+0000","last_change":"2024-04-23T10:24:44.054805+0000","last_active":"2024-04-23T10:24:45.041944+0000","last_peered":"2024-04-23T10:24:45.041944+0000","last_clean":"2024-04-23T10:24:45.041944+0000","last_became_active":"2024-04-23T10:24:44.054423+0000","last_became_peered":"2024-04-23T10:24:44.054423+0000","last_unstale":"2024-04-23T10:24:45.041944+0000","last_undegraded":"2024-04-23T10:24:45.041944+0000","last_fullsized":"2024-04-23T10:24:45.041944+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T10:24:43.025324+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T10:24:43.025324+0000","last_clean_scrub_stamp":"2024-04-23T10:24:43.025324+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T21:34:26.793624+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,1],"acting":[0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3c","version":"0'0","reported_seq":13,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-04-23T10:24:45.809205+0000","last_change":"2024-04-23T10:24:44.055966+0000","last_active":"2024-04-23T10:24:45.809205+0000","last_peered":"2024-04-23T10:24:45.809205+0000","last_clean":"2024-04-23T10:24:45.809205+0000","last_became_active":"2024-04-23T10:24:44.055028+0000","last_became_peered":"2024-04-23T10:24:44.055028+0000","last_unstale":"2024-04-23T10:24:45.809205+0000","last_undegraded":"2024-04-23T10:24:45.809205+0000","last_fullsized":"2024-04-23T10:24:45.809205+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T10:24:43.025324+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T10:24:43.025324+0000","last_clean_scrub_stamp":"2024-04-23T10:24:43.025324+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T19:41:04.594025+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,8,2],"acting":[4,8,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.3b","version":"0'0","reported_seq":13,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-04-23T10:24:45.044272+0000","last_change":"2024-04-23T10:24:44.046954+0000","last_active":"2024-04-23T10:24:45.044272+0000","last_peered":"2024-04-23T10:24:45.044272+0000","last_clean":"2024-04-23T10:24:45.044272+0000","last_became_active":"2024-04-23T10:24:44.046631+0000","last_became_peered":"2024-04-23T10:24:44.046631+0000","last_unstale":"2024-04-23T10:24:45.044272+0000","last_undegraded":"2024-04-23T10:24:45.044272+0000","last_fullsized":"2024-04-23T10:24:45.044272+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T10:24:43.025324+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T10:24:43.025324+0000","last_clean_scrub_stamp":"2024-04-23T10:24:43.025324+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T15:04:26.878894+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,3,6],"acting":[8,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.3a","version":"0'0","reported_seq":13,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-04-23T10:24:45.235291+0000","last_change":"2024-04-23T10:24:44.051550+0000","last_active":"2024-04-23T10:24:45.235291+0000","last_peered":"2024-04-23T10:24:45.235291+0000","last_clean":"2024-04-23T10:24:45.235291+0000","last_became_active":"2024-04-23T10:24:44.050469+0000","last_became_peered":"2024-04-23T10:24:44.050469+0000","last_unstale":"2024-04-23T10:24:45.235291+0000","last_undegraded":"2024-04-23T10:24:45.235291+0000","last_fullsized":"2024-04-23T10:24:45.235291+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T10:24:43.025324+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T10:24:43.025324+0000","last_clean_scrub_stamp":"2024-04-23T10:24:43.025324+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T21:13:42.469252+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,9,4],"acting":[10,9,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.39","version":"0'0","reported_seq":13,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-04-23T10:24:45.042070+0000","last_change":"2024-04-23T10:24:44.054916+0000","last_active":"2024-04-23T10:24:45.042070+0000","last_peered":"2024-04-23T10:24:45.042070+0000","last_clean":"2024-04-23T10:24:45.042070+0000","last_became_active":"2024-04-23T10:24:44.054536+0000","last_became_peered":"2024-04-23T10:24:44.054536+0000","last_unstale":"2024-04-23T10:24:45.042070+0000","last_undegraded":"2024-04-23T10:24:45.042070+0000","last_fullsized":"2024-04-23T10:24:45.042070+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T10:24:43.025324+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T10:24:43.025324+0000","last_clean_scrub_stamp":"2024-04-23T10:24:43.025324+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T19:04:13.402939+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,9,5],"acting":[7,9,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.38","version":"0'0","reported_seq":13,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-04-23T10:24:46.310399+0000","last_change":"2024-04-23T10:24:44.049026+0000","last_active":"2024-04-23T10:24:46.310399+0000","last_peered":"2024-04-23T10:24:46.310399+0000","last_clean":"2024-04-23T10:24:46.310399+0000","last_became_active":"2024-04-23T10:24:44.048683+0000","last_became_peered":"2024-04-23T10:24:44.048683+0000","last_unstale":"2024-04-23T10:24:46.310399+0000","last_undegraded":"2024-04-23T10:24:46.310399+0000","last_fullsized":"2024-04-23T10:24:46.310399+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T10:24:43.025324+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T10:24:43.025324+0000","last_clean_scrub_stamp":"2024-04-23T10:24:43.025324+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T19:26:53.489327+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,10],"acting":[3,5,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.37","version":"0'0","reported_seq":10,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-04-23T10:24:44.057965+0000","last_change":"2024-04-23T10:24:44.055261+0000","last_active":"2024-04-23T10:24:44.057965+0000","last_peered":"2024-04-23T10:24:44.057965+0000","last_clean":"2024-04-23T10:24:44.057965+0000","last_became_active":"2024-04-23T10:24:44.054405+0000","last_became_peered":"2024-04-23T10:24:44.054405+0000","last_unstale":"2024-04-23T10:24:44.057965+0000","last_undegraded":"2024-04-23T10:24:44.057965+0000","last_fullsized":"2024-04-23T10:24:44.057965+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T10:24:43.025324+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T10:24:43.025324+0000","last_clean_scrub_stamp":"2024-04-23T10:24:43.025324+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T21:11:43.804818+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,11],"acting":[1,5,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.36","version":"0'0","reported_seq":13,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-04-23T10:24:45.235708+0000","last_change":"2024-04-23T10:24:44.055854+0000","last_active":"2024-04-23T10:24:45.235708+0000","last_peered":"2024-04-23T10:24:45.235708+0000","last_clean":"2024-04-23T10:24:45.235708+0000","last_became_active":"2024-04-23T10:24:44.055275+0000","last_became_peered":"2024-04-23T10:24:44.055275+0000","last_unstale":"2024-04-23T10:24:45.235708+0000","last_undegraded":"2024-04-23T10:24:45.235708+0000","last_fullsized":"2024-04-23T10:24:45.235708+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T10:24:43.025324+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T10:24:43.025324+0000","last_clean_scrub_stamp":"2024-04-23T10:24:43.025324+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T21:48:25.934878+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,2,4],"acting":[10,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.35","version":"0'0","reported_seq":13,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-04-23T10:24:45.042491+0000","last_change":"2024-04-23T10:24:44.061314+0000","last_active":"2024-04-23T10:24:45.042491+0000","last_peered":"2024-04-23T10:24:45.042491+0000","last_clean":"2024-04-23T10:24:45.042491+0000","last_became_active":"2024-04-23T10:24:44.060981+0000","last_became_peered":"2024-04-23T10:24:44.060981+0000","last_unstale":"2024-04-23T10:24:45.042491+0000","last_undegraded":"2024-04-23T10:24:45.042491+0000","last_fullsized":"2024-04-23T10:24:45.042491+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T10:24:43.025324+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T10:24:43.025324+0000","last_clean_scrub_stamp":"2024-04-23T10:24:43.025324+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T11:48:08.387575+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,8,1],"acting":[7,8,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.34","version":"0'0","reported_seq":13,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-04-23T10:24:45.809201+0000","last_change":"2024-04-23T10:24:44.055335+0000","last_active":"2024-04-23T10:24:45.809201+0000","last_peered":"2024-04-23T10:24:45.809201+0000","last_clean":"2024-04-23T10:24:45.809201+0000","last_became_active":"2024-04-23T10:24:44.051257+0000","last_became_peered":"2024-04-23T10:24:44.051257+0000","last_unstale":"2024-04-23T10:24:45.809201+0000","last_undegraded":"2024-04-23T10:24:45.809201+0000","last_fullsized":"2024-04-23T10:24:45.809201+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T10:24:43.025324+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T10:24:43.025324+0000","last_clean_scrub_stamp":"2024-04-23T10:24:43.025324+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T11:57:45.999751+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,10,8],"acting":[4,10,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.33","version":"0'0","reported_seq":13,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-04-23T10:24:46.310013+0000","last_change":"2024-04-23T10:24:44.052078+0000","last_active":"2024-04-23T10:24:46.310013+0000","last_peered":"2024-04-23T10:24:46.310013+0000","last_clean":"2024-04-23T10:24:46.310013+0000","last_became_active":"2024-04-23T10:24:44.051747+0000","last_became_peered":"2024-04-23T10:24:44.051747+0000","last_unstale":"2024-04-23T10:24:46.310013+0000","last_undegraded":"2024-04-23T10:24:46.310013+0000","last_fullsized":"2024-04-23T10:24:46.310013+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T10:24:43.025324+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T10:24:43.025324+0000","last_clean_scrub_stamp":"2024-04-23T10:24:43.025324+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T19:14:16.495967+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,10,5],"acting":[6,10,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.32","version":"0'0","reported_seq":13,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-04-23T10:24:45.041784+0000","last_change":"2024-04-23T10:24:44.051994+0000","last_active":"2024-04-23T10:24:45.041784+0000","last_peered":"2024-04-23T10:24:45.041784+0000","last_clean":"2024-04-23T10:24:45.041784+0000","last_became_active":"2024-04-23T10:24:44.051638+0000","last_became_peered":"2024-04-23T10:24:44.051638+0000","last_unstale":"2024-04-23T10:24:45.041784+0000","last_undegraded":"2024-04-23T10:24:45.041784+0000","last_fullsized":"2024-04-23T10:24:45.041784+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T10:24:43.025324+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T10:24:43.025324+0000","last_clean_scrub_stamp":"2024-04-23T10:24:43.025324+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T14:28:30.447198+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,8,10],"acting":[0,8,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.31","version":"0'0","reported_seq":13,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-04-23T10:24:45.041962+0000","last_change":"2024-04-23T10:24:44.055779+0000","last_active":"2024-04-23T10:24:45.041962+0000","last_peered":"2024-04-23T10:24:45.041962+0000","last_clean":"2024-04-23T10:24:45.041962+0000","last_became_active":"2024-04-23T10:24:44.054872+0000","last_became_peered":"2024-04-23T10:24:44.054872+0000","last_unstale":"2024-04-23T10:24:45.041962+0000","last_undegraded":"2024-04-23T10:24:45.041962+0000","last_fullsized":"2024-04-23T10:24:45.041962+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T10:24:43.025324+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T10:24:43.025324+0000","last_clean_scrub_stamp":"2024-04-23T10:24:43.025324+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T16:45:32.273256+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,2],"acting":[0,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.30","version":"0'0","reported_seq":10,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-04-23T10:24:44.058403+0000","last_change":"2024-04-23T10:24:44.057154+0000","last_active":"2024-04-23T10:24:44.058403+0000","last_peered":"2024-04-23T10:24:44.058403+0000","last_clean":"2024-04-23T10:24:44.058403+0000","last_became_active":"2024-04-23T10:24:44.056836+0000","last_became_peered":"2024-04-23T10:24:44.056836+0000","last_unstale":"2024-04-23T10:24:44.058403+0000","last_undegraded":"2024-04-23T10:24:44.058403+0000","last_fullsized":"2024-04-23T10:24:44.058403+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T10:24:43.025324+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T10:24:43.025324+0000","last_clean_scrub_stamp":"2024-04-23T10:24:43.025324+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T21:21:50.715307+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,0],"acting":[1,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.2f","version":"0'0","reported_seq":10,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-04-23T10:24:44.052014+0000","last_change":"2024-04-23T10:24:44.051175+0000","last_active":"2024-04-23T10:24:44.052014+0000","last_peered":"2024-04-23T10:24:44.052014+0000","last_clean":"2024-04-23T10:24:44.052014+0000","last_became_active":"2024-04-23T10:24:44.050402+0000","last_became_peered":"2024-04-23T10:24:44.050402+0000","last_unstale":"2024-04-23T10:24:44.052014+0000","last_undegraded":"2024-04-23T10:24:44.052014+0000","last_fullsized":"2024-04-23T10:24:44.052014+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T10:24:43.025324+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T10:24:43.025324+0000","last_clean_scrub_stamp":"2024-04-23T10:24:43.025324+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T16:29:04.971067+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,1,0],"acting":[11,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.2e","version":"0'0","reported_seq":13,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-04-23T10:24:45.609038+0000","last_change":"2024-04-23T10:24:44.053390+0000","last_active":"2024-04-23T10:24:45.609038+0000","last_peered":"2024-04-23T10:24:45.609038+0000","last_clean":"2024-04-23T10:24:45.609038+0000","last_became_active":"2024-04-23T10:24:44.052970+0000","last_became_peered":"2024-04-23T10:24:44.052970+0000","last_unstale":"2024-04-23T10:24:45.609038+0000","last_undegraded":"2024-04-23T10:24:45.609038+0000","last_fullsized":"2024-04-23T10:24:45.609038+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T10:24:43.025324+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T10:24:43.025324+0000","last_clean_scrub_stamp":"2024-04-23T10:24:43.025324+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T13:27:55.704328+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,1,10],"acting":[9,1,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.2d","version":"0'0","reported_seq":13,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-04-23T10:24:45.124137+0000","last_change":"2024-04-23T10:24:44.049307+0000","last_active":"2024-04-23T10:24:45.124137+0000","last_peered":"2024-04-23T10:24:45.124137+0000","last_clean":"2024-04-23T10:24:45.124137+0000","last_became_active":"2024-04-23T10:24:44.048981+0000","last_became_peered":"2024-04-23T10:24:44.048981+0000","last_unstale":"2024-04-23T10:24:45.124137+0000","last_undegraded":"2024-04-23T10:24:45.124137+0000","last_fullsized":"2024-04-23T10:24:45.124137+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T10:24:43.025324+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T10:24:43.025324+0000","last_clean_scrub_stamp":"2024-04-23T10:24:43.025324+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T12:42:21.804591+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,6],"acting":[2,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.1f","version":"86'4","reported_seq":19,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-04-23T10:24:45.608885+0000","last_change":"2024-04-23T10:24:43.045804+0000","last_active":"2024-04-23T10:24:45.608885+0000","last_peered":"2024-04-23T10:24:45.608885+0000","last_clean":"2024-04-23T10:24:45.608885+0000","last_became_active":"2024-04-23T10:24:43.045135+0000","last_became_peered":"2024-04-23T10:24:43.045135+0000","last_unstale":"2024-04-23T10:24:45.608885+0000","last_undegraded":"2024-04-23T10:24:45.608885+0000","last_fullsized":"2024-04-23T10:24:45.608885+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T10:24:42.023127+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T10:24:42.023127+0000","last_clean_scrub_stamp":"2024-04-23T10:24:42.023127+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T20:27:28.421644+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,7,8],"acting":[9,7,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.1e","version":"0'0","reported_seq":13,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-04-23T10:24:46.309996+0000","last_change":"2024-04-23T10:24:44.055008+0000","last_active":"2024-04-23T10:24:46.309996+0000","last_peered":"2024-04-23T10:24:46.309996+0000","last_clean":"2024-04-23T10:24:46.309996+0000","last_became_active":"2024-04-23T10:24:44.054682+0000","last_became_peered":"2024-04-23T10:24:44.054682+0000","last_unstale":"2024-04-23T10:24:46.309996+0000","last_undegraded":"2024-04-23T10:24:46.309996+0000","last_fullsized":"2024-04-23T10:24:46.309996+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T10:24:43.025324+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T10:24:43.025324+0000","last_clean_scrub_stamp":"2024-04-23T10:24:43.025324+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T15:49:58.394958+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,5],"acting":[3,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.1e","version":"87'2","reported_seq":17,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-04-23T10:24:46.310088+0000","last_change":"2024-04-23T10:24:43.052303+0000","last_active":"2024-04-23T10:24:46.310088+0000","last_peered":"2024-04-23T10:24:46.310088+0000","last_clean":"2024-04-23T10:24:46.310088+0000","last_became_active":"2024-04-23T10:24:43.052010+0000","last_became_peered":"2024-04-23T10:24:43.052010+0000","last_unstale":"2024-04-23T10:24:46.310088+0000","last_undegraded":"2024-04-23T10:24:46.310088+0000","last_fullsized":"2024-04-23T10:24:46.310088+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T10:24:42.023127+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T10:24:42.023127+0000","last_clean_scrub_stamp":"2024-04-23T10:24:42.023127+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T13:03:15.440610+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,5],"acting":[3,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1f","version":"0'0","reported_seq":13,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-04-23T10:24:45.042148+0000","last_change":"2024-04-23T10:24:44.050749+0000","last_active":"2024-04-23T10:24:45.042148+0000","last_peered":"2024-04-23T10:24:45.042148+0000","last_clean":"2024-04-23T10:24:45.042148+0000","last_became_active":"2024-04-23T10:24:44.049780+0000","last_became_peered":"2024-04-23T10:24:44.049780+0000","last_unstale":"2024-04-23T10:24:45.042148+0000","last_undegraded":"2024-04-23T10:24:45.042148+0000","last_fullsized":"2024-04-23T10:24:45.042148+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T10:24:43.025324+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T10:24:43.025324+0000","last_clean_scrub_stamp":"2024-04-23T10:24:43.025324+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T16:32:37.963776+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,9],"acting":[0,1,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1d","version":"87'4","reported_seq":19,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-04-23T10:24:47.353492+0000","last_change":"2024-04-23T10:24:43.045847+0000","last_active":"2024-04-23T10:24:47.353492+0000","last_peered":"2024-04-23T10:24:47.353492+0000","last_clean":"2024-04-23T10:24:47.353492+0000","last_became_active":"2024-04-23T10:24:43.045554+0000","last_became_peered":"2024-04-23T10:24:43.045554+0000","last_unstale":"2024-04-23T10:24:47.353492+0000","last_undegraded":"2024-04-23T10:24:47.353492+0000","last_fullsized":"2024-04-23T10:24:47.353492+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T10:24:42.023127+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T10:24:42.023127+0000","last_clean_scrub_stamp":"2024-04-23T10:24:42.023127+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T18:27:50.240680+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,2,7],"acting":[8,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.1c","version":"0'0","reported_seq":10,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-04-23T10:24:44.052233+0000","last_change":"2024-04-23T10:24:44.051606+0000","last_active":"2024-04-23T10:24:44.052233+0000","last_peered":"2024-04-23T10:24:44.052233+0000","last_clean":"2024-04-23T10:24:44.052233+0000","last_became_active":"2024-04-23T10:24:44.051191+0000","last_became_peered":"2024-04-23T10:24:44.051191+0000","last_unstale":"2024-04-23T10:24:44.052233+0000","last_undegraded":"2024-04-23T10:24:44.052233+0000","last_fullsized":"2024-04-23T10:24:44.052233+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T10:24:43.025324+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T10:24:43.025324+0000","last_clean_scrub_stamp":"2024-04-23T10:24:43.025324+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T16:46:57.770438+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,10,6],"acting":[1,10,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1c","version":"86'2","reported_seq":17,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-04-23T10:24:45.044439+0000","last_change":"2024-04-23T10:24:43.048594+0000","last_active":"2024-04-23T10:24:45.044439+0000","last_peered":"2024-04-23T10:24:45.044439+0000","last_clean":"2024-04-23T10:24:45.044439+0000","last_became_active":"2024-04-23T10:24:43.048276+0000","last_became_peered":"2024-04-23T10:24:43.048276+0000","last_unstale":"2024-04-23T10:24:45.044439+0000","last_undegraded":"2024-04-23T10:24:45.044439+0000","last_fullsized":"2024-04-23T10:24:45.044439+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T10:24:42.023127+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T10:24:42.023127+0000","last_clean_scrub_stamp":"2024-04-23T10:24:42.023127+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T12:20:40.196674+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,1,5],"acting":[8,1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.1d","version":"0'0","reported_seq":13,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-04-23T10:24:46.310431+0000","last_change":"2024-04-23T10:24:44.050970+0000","last_active":"2024-04-23T10:24:46.310431+0000","last_peered":"2024-04-23T10:24:46.310431+0000","last_clean":"2024-04-23T10:24:46.310431+0000","last_became_active":"2024-04-23T10:24:44.050640+0000","last_became_peered":"2024-04-23T10:24:44.050640+0000","last_unstale":"2024-04-23T10:24:46.310431+0000","last_undegraded":"2024-04-23T10:24:46.310431+0000","last_fullsized":"2024-04-23T10:24:46.310431+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T10:24:43.025324+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T10:24:43.025324+0000","last_clean_scrub_stamp":"2024-04-23T10:24:43.025324+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T21:03:24.570290+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,1],"acting":[5,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.1b","version":"87'6","reported_seq":24,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-04-23T10:24:48.324814+0000","last_change":"2024-04-23T10:24:43.051742+0000","last_active":"2024-04-23T10:24:48.324814+0000","last_peered":"2024-04-23T10:24:48.324814+0000","last_clean":"2024-04-23T10:24:48.324814+0000","last_became_active":"2024-04-23T10:24:43.051420+0000","last_became_peered":"2024-04-23T10:24:43.051420+0000","last_unstale":"2024-04-23T10:24:48.324814+0000","last_undegraded":"2024-04-23T10:24:48.324814+0000","last_fullsized":"2024-04-23T10:24:48.324814+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T10:24:42.023127+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T10:24:42.023127+0000","last_clean_scrub_stamp":"2024-04-23T10:24:42.023127+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T17:54:37.182731+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1013,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":3,"num_read_kb":2,"num_write":3,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,7],"acting":[5,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.1a","version":"0'0","reported_seq":13,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-04-23T10:24:45.044567+0000","last_change":"2024-04-23T10:24:44.049314+0000","last_active":"2024-04-23T10:24:45.044567+0000","last_peered":"2024-04-23T10:24:45.044567+0000","last_clean":"2024-04-23T10:24:45.044567+0000","last_became_active":"2024-04-23T10:24:44.048352+0000","last_became_peered":"2024-04-23T10:24:44.048352+0000","last_unstale":"2024-04-23T10:24:45.044567+0000","last_undegraded":"2024-04-23T10:24:45.044567+0000","last_fullsized":"2024-04-23T10:24:45.044567+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T10:24:43.025324+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T10:24:43.025324+0000","last_clean_scrub_stamp":"2024-04-23T10:24:43.025324+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T16:57:29.957716+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,9,5],"acting":[8,9,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"2.1a","version":"86'1","reported_seq":16,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-04-23T10:24:45.235084+0000","last_change":"2024-04-23T10:24:43.051362+0000","last_active":"2024-04-23T10:24:45.235084+0000","last_peered":"2024-04-23T10:24:45.235084+0000","last_clean":"2024-04-23T10:24:45.235084+0000","last_became_active":"2024-04-23T10:24:43.051045+0000","last_became_peered":"2024-04-23T10:24:43.051045+0000","last_unstale":"2024-04-23T10:24:45.235084+0000","last_undegraded":"2024-04-23T10:24:45.235084+0000","last_fullsized":"2024-04-23T10:24:45.235084+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T10:24:42.023127+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T10:24:42.023127+0000","last_clean_scrub_stamp":"2024-04-23T10:24:42.023127+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T15:40:53.618281+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,8,6],"acting":[10,8,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.1b","version":"0'0","reported_seq":13,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-04-23T10:24:46.310386+0000","last_change":"2024-04-23T10:24:44.040323+0000","last_active":"2024-04-23T10:24:46.310386+0000","last_peered":"2024-04-23T10:24:46.310386+0000","last_clean":"2024-04-23T10:24:46.310386+0000","last_became_active":"2024-04-23T10:24:44.039995+0000","last_became_peered":"2024-04-23T10:24:44.039995+0000","last_unstale":"2024-04-23T10:24:46.310386+0000","last_undegraded":"2024-04-23T10:24:46.310386+0000","last_fullsized":"2024-04-23T10:24:46.310386+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T10:24:43.025324+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T10:24:43.025324+0000","last_clean_scrub_stamp":"2024-04-23T10:24:43.025324+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T16:38:48.756776+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,3],"acting":[5,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.19","version":"87'2","reported_seq":17,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-04-23T10:24:47.355005+0000","last_change":"2024-04-23T10:24:43.048754+0000","last_active":"2024-04-23T10:24:47.355005+0000","last_peered":"2024-04-23T10:24:47.355005+0000","last_clean":"2024-04-23T10:24:47.355005+0000","last_became_active":"2024-04-23T10:24:43.047962+0000","last_became_peered":"2024-04-23T10:24:43.047962+0000","last_unstale":"2024-04-23T10:24:47.355005+0000","last_undegraded":"2024-04-23T10:24:47.355005+0000","last_fullsized":"2024-04-23T10:24:47.355005+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T10:24:42.023127+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T10:24:42.023127+0000","last_clean_scrub_stamp":"2024-04-23T10:24:42.023127+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T21:08:38.487185+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,9],"acting":[6,4,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.18","version":"0'0","reported_seq":13,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-04-23T10:24:46.310429+0000","last_change":"2024-04-23T10:24:44.045022+0000","last_active":"2024-04-23T10:24:46.310429+0000","last_peered":"2024-04-23T10:24:46.310429+0000","last_clean":"2024-04-23T10:24:46.310429+0000","last_became_active":"2024-04-23T10:24:44.044590+0000","last_became_peered":"2024-04-23T10:24:44.044590+0000","last_unstale":"2024-04-23T10:24:46.310429+0000","last_undegraded":"2024-04-23T10:24:46.310429+0000","last_fullsized":"2024-04-23T10:24:46.310429+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T10:24:43.025324+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T10:24:43.025324+0000","last_clean_scrub_stamp":"2024-04-23T10:24:43.025324+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T13:20:49.892013+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,11,6],"acting":[3,11,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.18","version":"87'1","reported_seq":16,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-04-23T10:24:46.309777+0000","last_change":"2024-04-23T10:24:43.050170+0000","last_active":"2024-04-23T10:24:46.309777+0000","last_peered":"2024-04-23T10:24:46.309777+0000","last_clean":"2024-04-23T10:24:46.309777+0000","last_became_active":"2024-04-23T10:24:43.049070+0000","last_became_peered":"2024-04-23T10:24:43.049070+0000","last_unstale":"2024-04-23T10:24:46.309777+0000","last_undegraded":"2024-04-23T10:24:46.309777+0000","last_fullsized":"2024-04-23T10:24:46.309777+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T10:24:42.023127+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T10:24:42.023127+0000","last_clean_scrub_stamp":"2024-04-23T10:24:42.023127+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T19:34:13.983628+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,7],"acting":[5,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.19","version":"0'0","reported_seq":13,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-04-23T10:24:45.608801+0000","last_change":"2024-04-23T10:24:44.047165+0000","last_active":"2024-04-23T10:24:45.608801+0000","last_peered":"2024-04-23T10:24:45.608801+0000","last_clean":"2024-04-23T10:24:45.608801+0000","last_became_active":"2024-04-23T10:24:44.046776+0000","last_became_peered":"2024-04-23T10:24:44.046776+0000","last_unstale":"2024-04-23T10:24:45.608801+0000","last_undegraded":"2024-04-23T10:24:45.608801+0000","last_fullsized":"2024-04-23T10:24:45.608801+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T10:24:43.025324+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T10:24:43.025324+0000","last_clean_scrub_stamp":"2024-04-23T10:24:43.025324+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T18:11:57.181747+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,3,10],"acting":[9,3,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"2.17","version":"87'3","reported_seq":18,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-04-23T10:24:46.346825+0000","last_change":"2024-04-23T10:24:43.049785+0000","last_active":"2024-04-23T10:24:46.346825+0000","last_peered":"2024-04-23T10:24:46.346825+0000","last_clean":"2024-04-23T10:24:46.346825+0000","last_became_active":"2024-04-23T10:24:43.049402+0000","last_became_peered":"2024-04-23T10:24:43.049402+0000","last_unstale":"2024-04-23T10:24:46.346825+0000","last_undegraded":"2024-04-23T10:24:46.346825+0000","last_fullsized":"2024-04-23T10:24:46.346825+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T10:24:42.023127+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T10:24:42.023127+0000","last_clean_scrub_stamp":"2024-04-23T10:24:42.023127+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T14:20:13.774082+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,5,9],"acting":[10,5,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.16","version":"0'0","reported_seq":10,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-04-23T10:24:44.053989+0000","last_change":"2024-04-23T10:24:44.051768+0000","last_active":"2024-04-23T10:24:44.053989+0000","last_peered":"2024-04-23T10:24:44.053989+0000","last_clean":"2024-04-23T10:24:44.053989+0000","last_became_active":"2024-04-23T10:24:44.051228+0000","last_became_peered":"2024-04-23T10:24:44.051228+0000","last_unstale":"2024-04-23T10:24:44.053989+0000","last_undegraded":"2024-04-23T10:24:44.053989+0000","last_fullsized":"2024-04-23T10:24:44.053989+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T10:24:43.025324+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T10:24:43.025324+0000","last_clean_scrub_stamp":"2024-04-23T10:24:43.025324+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T11:23:59.992132+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,10,9],"acting":[1,10,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.16","version":"87'2","reported_seq":17,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-04-23T10:24:46.346750+0000","last_change":"2024-04-23T10:24:43.043944+0000","last_active":"2024-04-23T10:24:46.346750+0000","last_peered":"2024-04-23T10:24:46.346750+0000","last_clean":"2024-04-23T10:24:46.346750+0000","last_became_active":"2024-04-23T10:24:43.043648+0000","last_became_peered":"2024-04-23T10:24:43.043648+0000","last_unstale":"2024-04-23T10:24:46.346750+0000","last_undegraded":"2024-04-23T10:24:46.346750+0000","last_fullsized":"2024-04-23T10:24:46.346750+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T10:24:42.023127+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T10:24:42.023127+0000","last_clean_scrub_stamp":"2024-04-23T10:24:42.023127+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T13:41:15.724662+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,2],"acting":[7,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.17","version":"0'0","reported_seq":13,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-04-23T10:24:45.042106+0000","last_change":"2024-04-23T10:24:44.052369+0000","last_active":"2024-04-23T10:24:45.042106+0000","last_peered":"2024-04-23T10:24:45.042106+0000","last_clean":"2024-04-23T10:24:45.042106+0000","last_became_active":"2024-04-23T10:24:44.052009+0000","last_became_peered":"2024-04-23T10:24:44.052009+0000","last_unstale":"2024-04-23T10:24:45.042106+0000","last_undegraded":"2024-04-23T10:24:45.042106+0000","last_fullsized":"2024-04-23T10:24:45.042106+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T10:24:43.025324+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T10:24:43.025324+0000","last_clean_scrub_stamp":"2024-04-23T10:24:43.025324+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T22:23:01.913219+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,5],"acting":[0,4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.15","version":"87'4","reported_seq":20,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-04-23T10:24:46.346970+0000","last_change":"2024-04-23T10:24:43.048517+0000","last_active":"2024-04-23T10:24:46.346970+0000","last_peered":"2024-04-23T10:24:46.346970+0000","last_clean":"2024-04-23T10:24:46.346970+0000","last_became_active":"2024-04-23T10:24:43.048183+0000","last_became_peered":"2024-04-23T10:24:43.048183+0000","last_unstale":"2024-04-23T10:24:46.346970+0000","last_undegraded":"2024-04-23T10:24:46.346970+0000","last_fullsized":"2024-04-23T10:24:46.346970+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T10:24:42.023127+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T10:24:42.023127+0000","last_clean_scrub_stamp":"2024-04-23T10:24:42.023127+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T20:50:15.017833+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":672,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":5,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,10],"acting":[5,0,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.14","version":"0'0","reported_seq":10,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-04-23T10:24:44.058007+0000","last_change":"2024-04-23T10:24:44.055598+0000","last_active":"2024-04-23T10:24:44.058007+0000","last_peered":"2024-04-23T10:24:44.058007+0000","last_clean":"2024-04-23T10:24:44.058007+0000","last_became_active":"2024-04-23T10:24:44.053787+0000","last_became_peered":"2024-04-23T10:24:44.053787+0000","last_unstale":"2024-04-23T10:24:44.058007+0000","last_undegraded":"2024-04-23T10:24:44.058007+0000","last_fullsized":"2024-04-23T10:24:44.058007+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T10:24:43.025324+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T10:24:43.025324+0000","last_clean_scrub_stamp":"2024-04-23T10:24:43.025324+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T17:55:30.488748+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,8,4],"acting":[1,8,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.14","version":"87'8","reported_seq":23,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-04-23T10:24:46.345804+0000","last_change":"2024-04-23T10:24:43.044601+0000","last_active":"2024-04-23T10:24:46.345804+0000","last_peered":"2024-04-23T10:24:46.345804+0000","last_clean":"2024-04-23T10:24:46.345804+0000","last_became_active":"2024-04-23T10:24:43.044238+0000","last_became_peered":"2024-04-23T10:24:43.044238+0000","last_unstale":"2024-04-23T10:24:46.345804+0000","last_undegraded":"2024-04-23T10:24:46.345804+0000","last_fullsized":"2024-04-23T10:24:46.345804+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T10:24:42.023127+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T10:24:42.023127+0000","last_clean_scrub_stamp":"2024-04-23T10:24:42.023127+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T12:53:26.275771+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1612,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,1,2],"acting":[10,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.15","version":"0'0","reported_seq":13,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-04-23T10:24:45.044495+0000","last_change":"2024-04-23T10:24:44.051375+0000","last_active":"2024-04-23T10:24:45.044495+0000","last_peered":"2024-04-23T10:24:45.044495+0000","last_clean":"2024-04-23T10:24:45.044495+0000","last_became_active":"2024-04-23T10:24:44.050924+0000","last_became_peered":"2024-04-23T10:24:44.050924+0000","last_unstale":"2024-04-23T10:24:45.044495+0000","last_undegraded":"2024-04-23T10:24:45.044495+0000","last_fullsized":"2024-04-23T10:24:45.044495+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T10:24:43.025324+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T10:24:43.025324+0000","last_clean_scrub_stamp":"2024-04-23T10:24:43.025324+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T21:50:57.534760+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,11,7],"acting":[8,11,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"2.13","version":"86'2","reported_seq":17,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-04-23T10:24:45.123954+0000","last_change":"2024-04-23T10:24:43.044457+0000","last_active":"2024-04-23T10:24:45.123954+0000","last_peered":"2024-04-23T10:24:45.123954+0000","last_clean":"2024-04-23T10:24:45.123954+0000","last_became_active":"2024-04-23T10:24:43.043956+0000","last_became_peered":"2024-04-23T10:24:43.043956+0000","last_unstale":"2024-04-23T10:24:45.123954+0000","last_undegraded":"2024-04-23T10:24:45.123954+0000","last_fullsized":"2024-04-23T10:24:45.123954+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T10:24:42.023127+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T10:24:42.023127+0000","last_clean_scrub_stamp":"2024-04-23T10:24:42.023127+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T20:16:08.186729+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,10,3],"acting":[2,10,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.12","version":"0'0","reported_seq":13,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-04-23T10:24:46.309871+0000","last_change":"2024-04-23T10:24:44.059571+0000","last_active":"2024-04-23T10:24:46.309871+0000","last_peered":"2024-04-23T10:24:46.309871+0000","last_clean":"2024-04-23T10:24:46.309871+0000","last_became_active":"2024-04-23T10:24:44.059196+0000","last_became_peered":"2024-04-23T10:24:44.059196+0000","last_unstale":"2024-04-23T10:24:46.309871+0000","last_undegraded":"2024-04-23T10:24:46.309871+0000","last_fullsized":"2024-04-23T10:24:46.309871+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T10:24:43.025324+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T10:24:43.025324+0000","last_clean_scrub_stamp":"2024-04-23T10:24:43.025324+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T22:09:15.659402+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,3],"acting":[6,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.12","version":"87'3","reported_seq":18,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-04-23T10:24:46.347868+0000","last_change":"2024-04-23T10:24:43.052181+0000","last_active":"2024-04-23T10:24:46.347868+0000","last_peered":"2024-04-23T10:24:46.347868+0000","last_clean":"2024-04-23T10:24:46.347868+0000","last_became_active":"2024-04-23T10:24:43.051793+0000","last_became_peered":"2024-04-23T10:24:43.051793+0000","last_unstale":"2024-04-23T10:24:46.347868+0000","last_undegraded":"2024-04-23T10:24:46.347868+0000","last_fullsized":"2024-04-23T10:24:46.347868+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T10:24:42.023127+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T10:24:42.023127+0000","last_clean_scrub_stamp":"2024-04-23T10:24:42.023127+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T20:57:59.015115+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,7],"acting":[5,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.13","version":"0'0","reported_seq":13,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-04-23T10:24:45.042533+0000","last_change":"2024-04-23T10:24:44.062689+0000","last_active":"2024-04-23T10:24:45.042533+0000","last_peered":"2024-04-23T10:24:45.042533+0000","last_clean":"2024-04-23T10:24:45.042533+0000","last_became_active":"2024-04-23T10:24:44.062331+0000","last_became_peered":"2024-04-23T10:24:44.062331+0000","last_unstale":"2024-04-23T10:24:45.042533+0000","last_undegraded":"2024-04-23T10:24:45.042533+0000","last_fullsized":"2024-04-23T10:24:45.042533+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T10:24:43.025324+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T10:24:43.025324+0000","last_clean_scrub_stamp":"2024-04-23T10:24:43.025324+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T14:06:04.989156+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,6],"acting":[7,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.11","version":"87'3","reported_seq":18,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-04-23T10:24:47.354217+0000","last_change":"2024-04-23T10:24:43.043901+0000","last_active":"2024-04-23T10:24:47.354217+0000","last_peered":"2024-04-23T10:24:47.354217+0000","last_clean":"2024-04-23T10:24:47.354217+0000","last_became_active":"2024-04-23T10:24:43.043598+0000","last_became_peered":"2024-04-23T10:24:43.043598+0000","last_unstale":"2024-04-23T10:24:47.354217+0000","last_undegraded":"2024-04-23T10:24:47.354217+0000","last_fullsized":"2024-04-23T10:24:47.354217+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T10:24:42.023127+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T10:24:42.023127+0000","last_clean_scrub_stamp":"2024-04-23T10:24:42.023127+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T14:00:06.338084+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":588,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,10],"acting":[4,7,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.10","version":"0'0","reported_seq":13,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-04-23T10:24:45.042074+0000","last_change":"2024-04-23T10:24:44.058773+0000","last_active":"2024-04-23T10:24:45.042074+0000","last_peered":"2024-04-23T10:24:45.042074+0000","last_clean":"2024-04-23T10:24:45.042074+0000","last_became_active":"2024-04-23T10:24:44.058420+0000","last_became_peered":"2024-04-23T10:24:44.058420+0000","last_unstale":"2024-04-23T10:24:45.042074+0000","last_undegraded":"2024-04-23T10:24:45.042074+0000","last_fullsized":"2024-04-23T10:24:45.042074+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T10:24:43.025324+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T10:24:43.025324+0000","last_clean_scrub_stamp":"2024-04-23T10:24:43.025324+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T12:44:30.023047+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,10],"acting":[7,5,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.10","version":"87'15","reported_seq":30,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-04-23T10:24:47.355199+0000","last_change":"2024-04-23T10:24:43.044700+0000","last_active":"2024-04-23T10:24:47.355199+0000","last_peered":"2024-04-23T10:24:47.355199+0000","last_clean":"2024-04-23T10:24:47.355199+0000","last_became_active":"2024-04-23T10:24:43.044394+0000","last_became_peered":"2024-04-23T10:24:43.044394+0000","last_unstale":"2024-04-23T10:24:47.355199+0000","last_undegraded":"2024-04-23T10:24:47.355199+0000","last_fullsized":"2024-04-23T10:24:47.355199+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T10:24:42.023127+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T10:24:42.023127+0000","last_clean_scrub_stamp":"2024-04-23T10:24:42.023127+0000","objects_scrubbed":0,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T15:16:41.367517+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":112,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":20,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":6,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,0,2],"acting":[8,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.11","version":"0'0","reported_seq":13,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-04-23T10:24:46.310005+0000","last_change":"2024-04-23T10:24:44.049739+0000","last_active":"2024-04-23T10:24:46.310005+0000","last_peered":"2024-04-23T10:24:46.310005+0000","last_clean":"2024-04-23T10:24:46.310005+0000","last_became_active":"2024-04-23T10:24:44.049414+0000","last_became_peered":"2024-04-23T10:24:44.049414+0000","last_unstale":"2024-04-23T10:24:46.310005+0000","last_undegraded":"2024-04-23T10:24:46.310005+0000","last_fullsized":"2024-04-23T10:24:46.310005+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T10:24:43.025324+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T10:24:43.025324+0000","last_clean_scrub_stamp":"2024-04-23T10:24:43.025324+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T12:51:02.645259+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,1],"acting":[5,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.f","version":"86'1","reported_seq":16,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-04-23T10:24:45.044492+0000","last_change":"2024-04-23T10:24:43.048884+0000","last_active":"2024-04-23T10:24:45.044492+0000","last_peered":"2024-04-23T10:24:45.044492+0000","last_clean":"2024-04-23T10:24:45.044492+0000","last_became_active":"2024-04-23T10:24:43.048583+0000","last_became_peered":"2024-04-23T10:24:43.048583+0000","last_unstale":"2024-04-23T10:24:45.044492+0000","last_undegraded":"2024-04-23T10:24:45.044492+0000","last_fullsized":"2024-04-23T10:24:45.044492+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T10:24:42.023127+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T10:24:42.023127+0000","last_clean_scrub_stamp":"2024-04-23T10:24:42.023127+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T17:55:57.869700+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,5,4],"acting":[8,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.e","version":"0'0","reported_seq":10,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-04-23T10:24:44.053368+0000","last_change":"2024-04-23T10:24:44.052817+0000","last_active":"2024-04-23T10:24:44.053368+0000","last_peered":"2024-04-23T10:24:44.053368+0000","last_clean":"2024-04-23T10:24:44.053368+0000","last_became_active":"2024-04-23T10:24:44.052323+0000","last_became_peered":"2024-04-23T10:24:44.052323+0000","last_unstale":"2024-04-23T10:24:44.053368+0000","last_undegraded":"2024-04-23T10:24:44.053368+0000","last_fullsized":"2024-04-23T10:24:44.053368+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T10:24:43.025324+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T10:24:43.025324+0000","last_clean_scrub_stamp":"2024-04-23T10:24:43.025324+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T16:13:09.933338+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,5,4],"acting":[11,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":15,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-04-23T10:24:45.608893+0000","last_change":"2024-04-23T10:24:43.044375+0000","last_active":"2024-04-23T10:24:45.608893+0000","last_peered":"2024-04-23T10:24:45.608893+0000","last_clean":"2024-04-23T10:24:45.608893+0000","last_became_active":"2024-04-23T10:24:43.043914+0000","last_became_peered":"2024-04-23T10:24:43.043914+0000","last_unstale":"2024-04-23T10:24:45.608893+0000","last_undegraded":"2024-04-23T10:24:45.608893+0000","last_fullsized":"2024-04-23T10:24:45.608893+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T10:24:42.023127+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T10:24:42.023127+0000","last_clean_scrub_stamp":"2024-04-23T10:24:42.023127+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T16:22:44.165156+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,2,0],"acting":[9,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.5","version":"0'0","reported_seq":13,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-04-23T10:24:45.235612+0000","last_change":"2024-04-23T10:24:44.057272+0000","last_active":"2024-04-23T10:24:45.235612+0000","last_peered":"2024-04-23T10:24:45.235612+0000","last_clean":"2024-04-23T10:24:45.235612+0000","last_became_active":"2024-04-23T10:24:44.056968+0000","last_became_peered":"2024-04-23T10:24:44.056968+0000","last_unstale":"2024-04-23T10:24:45.235612+0000","last_undegraded":"2024-04-23T10:24:45.235612+0000","last_fullsized":"2024-04-23T10:24:45.235612+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T10:24:43.025324+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T10:24:43.025324+0000","last_clean_scrub_stamp":"2024-04-23T10:24:43.025324+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T11:10:09.021845+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,3,6],"acting":[10,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"2.2","version":"87'1","reported_seq":16,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-04-23T10:24:45.042157+0000","last_change":"2024-04-23T10:24:43.051822+0000","last_active":"2024-04-23T10:24:45.042157+0000","last_peered":"2024-04-23T10:24:45.042157+0000","last_clean":"2024-04-23T10:24:45.042157+0000","last_became_active":"2024-04-23T10:24:43.051491+0000","last_became_peered":"2024-04-23T10:24:43.051491+0000","last_unstale":"2024-04-23T10:24:45.042157+0000","last_undegraded":"2024-04-23T10:24:45.042157+0000","last_fullsized":"2024-04-23T10:24:45.042157+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T10:24:42.023127+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T10:24:42.023127+0000","last_clean_scrub_stamp":"2024-04-23T10:24:42.023127+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T13:49:23.109064+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,1],"acting":[7,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.3","version":"0'0","reported_seq":13,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-04-23T10:24:45.235454+0000","last_change":"2024-04-23T10:24:44.053604+0000","last_active":"2024-04-23T10:24:45.235454+0000","last_peered":"2024-04-23T10:24:45.235454+0000","last_clean":"2024-04-23T10:24:45.235454+0000","last_became_active":"2024-04-23T10:24:44.053147+0000","last_became_peered":"2024-04-23T10:24:44.053147+0000","last_unstale":"2024-04-23T10:24:45.235454+0000","last_undegraded":"2024-04-23T10:24:45.235454+0000","last_fullsized":"2024-04-23T10:24:45.235454+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T10:24:43.025324+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T10:24:43.025324+0000","last_clean_scrub_stamp":"2024-04-23T10:24:43.025324+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T21:58:38.981282+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,4,0],"acting":[10,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.0","version":"0'0","reported_seq":13,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-04-23T10:24:46.310228+0000","last_change":"2024-04-23T10:24:44.052564+0000","last_active":"2024-04-23T10:24:46.310228+0000","last_peered":"2024-04-23T10:24:46.310228+0000","last_clean":"2024-04-23T10:24:46.310228+0000","last_became_active":"2024-04-23T10:24:44.052201+0000","last_became_peered":"2024-04-23T10:24:44.052201+0000","last_unstale":"2024-04-23T10:24:46.310228+0000","last_undegraded":"2024-04-23T10:24:46.310228+0000","last_fullsized":"2024-04-23T10:24:46.310228+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T10:24:43.025324+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T10:24:43.025324+0000","last_clean_scrub_stamp":"2024-04-23T10:24:43.025324+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T16:25:54.725347+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,4],"acting":[5,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.1","version":"87'4","reported_seq":19,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-04-23T10:24:47.356133+0000","last_change":"2024-04-23T10:24:43.049067+0000","last_active":"2024-04-23T10:24:47.356133+0000","last_peered":"2024-04-23T10:24:47.356133+0000","last_clean":"2024-04-23T10:24:47.356133+0000","last_became_active":"2024-04-23T10:24:43.048400+0000","last_became_peered":"2024-04-23T10:24:43.048400+0000","last_unstale":"2024-04-23T10:24:47.356133+0000","last_undegraded":"2024-04-23T10:24:47.356133+0000","last_fullsized":"2024-04-23T10:24:47.356133+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T10:24:42.023127+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T10:24:42.023127+0000","last_clean_scrub_stamp":"2024-04-23T10:24:42.023127+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T12:17:02.439054+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,0],"acting":[6,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.2","version":"0'0","reported_seq":13,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-04-23T10:24:46.310403+0000","last_change":"2024-04-23T10:24:44.059114+0000","last_active":"2024-04-23T10:24:46.310403+0000","last_peered":"2024-04-23T10:24:46.310403+0000","last_clean":"2024-04-23T10:24:46.310403+0000","last_became_active":"2024-04-23T10:24:44.058618+0000","last_became_peered":"2024-04-23T10:24:44.058618+0000","last_unstale":"2024-04-23T10:24:46.310403+0000","last_undegraded":"2024-04-23T10:24:46.310403+0000","last_fullsized":"2024-04-23T10:24:46.310403+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T10:24:43.025324+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T10:24:43.025324+0000","last_clean_scrub_stamp":"2024-04-23T10:24:43.025324+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T20:18:09.707062+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,1],"acting":[3,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.3","version":"87'3","reported_seq":18,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-04-23T10:24:46.347609+0000","last_change":"2024-04-23T10:24:43.039725+0000","last_active":"2024-04-23T10:24:46.347609+0000","last_peered":"2024-04-23T10:24:46.347609+0000","last_clean":"2024-04-23T10:24:46.347609+0000","last_became_active":"2024-04-23T10:24:43.039187+0000","last_became_peered":"2024-04-23T10:24:43.039187+0000","last_unstale":"2024-04-23T10:24:46.347609+0000","last_undegraded":"2024-04-23T10:24:46.347609+0000","last_fullsized":"2024-04-23T10:24:46.347609+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T10:24:42.023127+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T10:24:42.023127+0000","last_clean_scrub_stamp":"2024-04-23T10:24:42.023127+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T17:20:50.844878+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,10,9],"acting":[4,10,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.0","version":"22'137","reported_seq":346,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-04-23T10:24:45.041954+0000","last_change":"2024-04-23T10:23:31.092860+0000","last_active":"2024-04-23T10:24:45.041954+0000","last_peered":"2024-04-23T10:24:45.041954+0000","last_clean":"2024-04-23T10:24:45.041954+0000","last_became_active":"2024-04-23T10:23:31.092357+0000","last_became_peered":"2024-04-23T10:23:31.092357+0000","last_unstale":"2024-04-23T10:24:45.041954+0000","last_undegraded":"2024-04-23T10:24:45.041954+0000","last_fullsized":"2024-04-23T10:24:45.041954+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T10:20:12.201233+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T10:20:12.201233+0000","last_clean_scrub_stamp":"2024-04-23T10:20:12.201233+0000","objects_scrubbed":0,"log_size":137,"log_dups_size":0,"ondisk_log_size":137,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T21:28:56.804053+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":136,"num_read_kb":119,"num_write":181,"num_write_kb":2938,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":1180736,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,10],"acting":[7,5,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1","version":"0'0","reported_seq":13,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-04-23T10:24:46.309600+0000","last_change":"2024-04-23T10:24:44.058393+0000","last_active":"2024-04-23T10:24:46.309600+0000","last_peered":"2024-04-23T10:24:46.309600+0000","last_clean":"2024-04-23T10:24:46.309600+0000","last_became_active":"2024-04-23T10:24:44.058075+0000","last_became_peered":"2024-04-23T10:24:44.058075+0000","last_unstale":"2024-04-23T10:24:46.309600+0000","last_undegraded":"2024-04-23T10:24:46.309600+0000","last_fullsized":"2024-04-23T10:24:46.309600+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T10:24:43.025324+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T10:24:43.025324+0000","last_clean_scrub_stamp":"2024-04-23T10:24:43.025324+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T10:32:05.693311+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,2],"acting":[6,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.0","version":"87'3","reported_seq":18,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-04-23T10:24:47.354743+0000","last_change":"2024-04-23T10:24:43.048678+0000","last_active":"2024-04-23T10:24:47.354743+0000","last_peered":"2024-04-23T10:24:47.354743+0000","last_clean":"2024-04-23T10:24:47.354743+0000","last_became_active":"2024-04-23T10:24:43.048341+0000","last_became_peered":"2024-04-23T10:24:43.048341+0000","last_unstale":"2024-04-23T10:24:47.354743+0000","last_undegraded":"2024-04-23T10:24:47.354743+0000","last_fullsized":"2024-04-23T10:24:47.354743+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T10:24:42.023127+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T10:24:42.023127+0000","last_clean_scrub_stamp":"2024-04-23T10:24:42.023127+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T12:24:33.705325+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,8],"acting":[7,5,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.5","version":"87'1","reported_seq":16,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-04-23T10:24:45.044415+0000","last_change":"2024-04-23T10:24:43.046105+0000","last_active":"2024-04-23T10:24:45.044415+0000","last_peered":"2024-04-23T10:24:45.044415+0000","last_clean":"2024-04-23T10:24:45.044415+0000","last_became_active":"2024-04-23T10:24:43.045804+0000","last_became_peered":"2024-04-23T10:24:43.045804+0000","last_unstale":"2024-04-23T10:24:45.044415+0000","last_undegraded":"2024-04-23T10:24:45.044415+0000","last_fullsized":"2024-04-23T10:24:45.044415+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T10:24:42.023127+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T10:24:42.023127+0000","last_clean_scrub_stamp":"2024-04-23T10:24:42.023127+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T22:23:35.927468+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,0,9],"acting":[8,0,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.4","version":"0'0","reported_seq":13,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-04-23T10:24:46.310103+0000","last_change":"2024-04-23T10:24:44.059731+0000","last_active":"2024-04-23T10:24:46.310103+0000","last_peered":"2024-04-23T10:24:46.310103+0000","last_clean":"2024-04-23T10:24:46.310103+0000","last_became_active":"2024-04-23T10:24:44.059352+0000","last_became_peered":"2024-04-23T10:24:44.059352+0000","last_unstale":"2024-04-23T10:24:46.310103+0000","last_undegraded":"2024-04-23T10:24:46.310103+0000","last_fullsized":"2024-04-23T10:24:46.310103+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T10:24:43.025324+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T10:24:43.025324+0000","last_clean_scrub_stamp":"2024-04-23T10:24:43.025324+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T16:36:08.985444+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2,9],"acting":[6,2,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.6","version":"87'1","reported_seq":16,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-04-23T10:24:46.323682+0000","last_change":"2024-04-23T10:24:43.044527+0000","last_active":"2024-04-23T10:24:46.323682+0000","last_peered":"2024-04-23T10:24:46.323682+0000","last_clean":"2024-04-23T10:24:46.323682+0000","last_became_active":"2024-04-23T10:24:43.044212+0000","last_became_peered":"2024-04-23T10:24:43.044212+0000","last_unstale":"2024-04-23T10:24:46.323682+0000","last_undegraded":"2024-04-23T10:24:46.323682+0000","last_fullsized":"2024-04-23T10:24:46.323682+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T10:24:42.023127+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T10:24:42.023127+0000","last_clean_scrub_stamp":"2024-04-23T10:24:42.023127+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T19:39:46.369132+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,8,3],"acting":[9,8,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.7","version":"0'0","reported_seq":13,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-04-23T10:24:46.309988+0000","last_change":"2024-04-23T10:24:44.055920+0000","last_active":"2024-04-23T10:24:46.309988+0000","last_peered":"2024-04-23T10:24:46.309988+0000","last_clean":"2024-04-23T10:24:46.309988+0000","last_became_active":"2024-04-23T10:24:44.055411+0000","last_became_peered":"2024-04-23T10:24:44.055411+0000","last_unstale":"2024-04-23T10:24:46.309988+0000","last_undegraded":"2024-04-23T10:24:46.309988+0000","last_fullsized":"2024-04-23T10:24:46.309988+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T10:24:43.025324+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T10:24:43.025324+0000","last_clean_scrub_stamp":"2024-04-23T10:24:43.025324+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T14:43:10.446790+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,11,7],"acting":[3,11,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.7","version":"87'4","reported_seq":19,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-04-23T10:24:46.345036+0000","last_change":"2024-04-23T10:24:43.047989+0000","last_active":"2024-04-23T10:24:46.345036+0000","last_peered":"2024-04-23T10:24:46.345036+0000","last_clean":"2024-04-23T10:24:46.345036+0000","last_became_active":"2024-04-23T10:24:43.047625+0000","last_became_peered":"2024-04-23T10:24:43.047625+0000","last_unstale":"2024-04-23T10:24:46.345036+0000","last_undegraded":"2024-04-23T10:24:46.345036+0000","last_fullsized":"2024-04-23T10:24:46.345036+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T10:24:42.023127+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T10:24:42.023127+0000","last_clean_scrub_stamp":"2024-04-23T10:24:42.023127+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T13:50:16.647176+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,11,6],"acting":[4,11,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.6","version":"0'0","reported_seq":13,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-04-23T10:24:45.042053+0000","last_change":"2024-04-23T10:24:44.049214+0000","last_active":"2024-04-23T10:24:45.042053+0000","last_peered":"2024-04-23T10:24:45.042053+0000","last_clean":"2024-04-23T10:24:45.042053+0000","last_became_active":"2024-04-23T10:24:44.048869+0000","last_became_peered":"2024-04-23T10:24:44.048869+0000","last_unstale":"2024-04-23T10:24:45.042053+0000","last_undegraded":"2024-04-23T10:24:45.042053+0000","last_fullsized":"2024-04-23T10:24:45.042053+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T10:24:43.025324+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T10:24:43.025324+0000","last_clean_scrub_stamp":"2024-04-23T10:24:43.025324+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T18:17:09.772428+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,4],"acting":[0,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.8","version":"0'0","reported_seq":13,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-04-23T10:24:44.047149+0000","last_change":"2024-04-23T10:24:43.046189+0000","last_active":"2024-04-23T10:24:44.047149+0000","last_peered":"2024-04-23T10:24:44.047149+0000","last_clean":"2024-04-23T10:24:44.047149+0000","last_became_active":"2024-04-23T10:24:43.045848+0000","last_became_peered":"2024-04-23T10:24:43.045848+0000","last_unstale":"2024-04-23T10:24:44.047149+0000","last_undegraded":"2024-04-23T10:24:44.047149+0000","last_fullsized":"2024-04-23T10:24:44.047149+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T10:24:42.023127+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T10:24:42.023127+0000","last_clean_scrub_stamp":"2024-04-23T10:24:42.023127+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T22:23:39.180377+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,0],"acting":[1,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.9","version":"0'0","reported_seq":13,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-04-23T10:24:45.044247+0000","last_change":"2024-04-23T10:24:44.048415+0000","last_active":"2024-04-23T10:24:45.044247+0000","last_peered":"2024-04-23T10:24:45.044247+0000","last_clean":"2024-04-23T10:24:45.044247+0000","last_became_active":"2024-04-23T10:24:44.048030+0000","last_became_peered":"2024-04-23T10:24:44.048030+0000","last_unstale":"2024-04-23T10:24:45.044247+0000","last_undegraded":"2024-04-23T10:24:45.044247+0000","last_fullsized":"2024-04-23T10:24:45.044247+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T10:24:43.025324+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T10:24:43.025324+0000","last_clean_scrub_stamp":"2024-04-23T10:24:43.025324+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T14:42:09.920927+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,11,10],"acting":[8,11,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"2.9","version":"87'3","reported_seq":18,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-04-23T10:24:46.346630+0000","last_change":"2024-04-23T10:24:43.041525+0000","last_active":"2024-04-23T10:24:46.346630+0000","last_peered":"2024-04-23T10:24:46.346630+0000","last_clean":"2024-04-23T10:24:46.346630+0000","last_became_active":"2024-04-23T10:24:43.041200+0000","last_became_peered":"2024-04-23T10:24:43.041200+0000","last_unstale":"2024-04-23T10:24:46.346630+0000","last_undegraded":"2024-04-23T10:24:46.346630+0000","last_fullsized":"2024-04-23T10:24:46.346630+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T10:24:42.023127+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T10:24:42.023127+0000","last_clean_scrub_stamp":"2024-04-23T10:24:42.023127+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T16:45:06.846991+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,9,10],"acting":[3,9,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.8","version":"0'0","reported_seq":10,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-04-23T10:24:44.055845+0000","last_change":"2024-04-23T10:24:44.055341+0000","last_active":"2024-04-23T10:24:44.055845+0000","last_peered":"2024-04-23T10:24:44.055845+0000","last_clean":"2024-04-23T10:24:44.055845+0000","last_became_active":"2024-04-23T10:24:44.054216+0000","last_became_peered":"2024-04-23T10:24:44.054216+0000","last_unstale":"2024-04-23T10:24:44.055845+0000","last_undegraded":"2024-04-23T10:24:44.055845+0000","last_fullsized":"2024-04-23T10:24:44.055845+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T10:24:43.025324+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T10:24:43.025324+0000","last_clean_scrub_stamp":"2024-04-23T10:24:43.025324+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T10:32:02.420748+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,0,2],"acting":[11,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"2.a","version":"0'0","reported_seq":15,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-04-23T10:24:45.123774+0000","last_change":"2024-04-23T10:24:43.045400+0000","last_active":"2024-04-23T10:24:45.123774+0000","last_peered":"2024-04-23T10:24:45.123774+0000","last_clean":"2024-04-23T10:24:45.123774+0000","last_became_active":"2024-04-23T10:24:43.044677+0000","last_became_peered":"2024-04-23T10:24:43.044677+0000","last_unstale":"2024-04-23T10:24:45.123774+0000","last_undegraded":"2024-04-23T10:24:45.123774+0000","last_fullsized":"2024-04-23T10:24:45.123774+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T10:24:42.023127+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T10:24:42.023127+0000","last_clean_scrub_stamp":"2024-04-23T10:24:42.023127+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T19:06:19.700132+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,8],"acting":[2,3,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.b","version":"0'0","reported_seq":13,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-04-23T10:24:46.310072+0000","last_change":"2024-04-23T10:24:44.049968+0000","last_active":"2024-04-23T10:24:46.310072+0000","last_peered":"2024-04-23T10:24:46.310072+0000","last_clean":"2024-04-23T10:24:46.310072+0000","last_became_active":"2024-04-23T10:24:44.049621+0000","last_became_peered":"2024-04-23T10:24:44.049621+0000","last_unstale":"2024-04-23T10:24:46.310072+0000","last_undegraded":"2024-04-23T10:24:46.310072+0000","last_fullsized":"2024-04-23T10:24:46.310072+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T10:24:43.025324+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T10:24:43.025324+0000","last_clean_scrub_stamp":"2024-04-23T10:24:43.025324+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T16:23:43.082831+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,10],"acting":[3,0,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.b","version":"87'4","reported_seq":19,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-04-23T10:24:46.344828+0000","last_change":"2024-04-23T10:24:43.041828+0000","last_active":"2024-04-23T10:24:46.344828+0000","last_peered":"2024-04-23T10:24:46.344828+0000","last_clean":"2024-04-23T10:24:46.344828+0000","last_became_active":"2024-04-23T10:24:43.041480+0000","last_became_peered":"2024-04-23T10:24:43.041480+0000","last_unstale":"2024-04-23T10:24:46.344828+0000","last_undegraded":"2024-04-23T10:24:46.344828+0000","last_fullsized":"2024-04-23T10:24:46.344828+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T10:24:42.023127+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T10:24:42.023127+0000","last_clean_scrub_stamp":"2024-04-23T10:24:42.023127+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T14:43:03.203971+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,2,7],"acting":[8,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.a","version":"0'0","reported_seq":13,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-04-23T10:24:45.042560+0000","last_change":"2024-04-23T10:24:44.062103+0000","last_active":"2024-04-23T10:24:45.042560+0000","last_peered":"2024-04-23T10:24:45.042560+0000","last_clean":"2024-04-23T10:24:45.042560+0000","last_became_active":"2024-04-23T10:24:44.061794+0000","last_became_peered":"2024-04-23T10:24:44.061794+0000","last_unstale":"2024-04-23T10:24:45.042560+0000","last_undegraded":"2024-04-23T10:24:45.042560+0000","last_fullsized":"2024-04-23T10:24:45.042560+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T10:24:43.025324+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T10:24:43.025324+0000","last_clean_scrub_stamp":"2024-04-23T10:24:43.025324+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T10:45:30.130047+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,6],"acting":[7,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.c","version":"0'0","reported_seq":15,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-04-23T10:24:45.124150+0000","last_change":"2024-04-23T10:24:43.045702+0000","last_active":"2024-04-23T10:24:45.124150+0000","last_peered":"2024-04-23T10:24:45.124150+0000","last_clean":"2024-04-23T10:24:45.124150+0000","last_became_active":"2024-04-23T10:24:43.045258+0000","last_became_peered":"2024-04-23T10:24:43.045258+0000","last_unstale":"2024-04-23T10:24:45.124150+0000","last_undegraded":"2024-04-23T10:24:45.124150+0000","last_fullsized":"2024-04-23T10:24:45.124150+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T10:24:42.023127+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T10:24:42.023127+0000","last_clean_scrub_stamp":"2024-04-23T10:24:42.023127+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T12:54:30.965080+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,11,0],"acting":[2,11,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.d","version":"0'0","reported_seq":13,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-04-23T10:24:45.042632+0000","last_change":"2024-04-23T10:24:44.061118+0000","last_active":"2024-04-23T10:24:45.042632+0000","last_peered":"2024-04-23T10:24:45.042632+0000","last_clean":"2024-04-23T10:24:45.042632+0000","last_became_active":"2024-04-23T10:24:44.060743+0000","last_became_peered":"2024-04-23T10:24:44.060743+0000","last_unstale":"2024-04-23T10:24:45.042632+0000","last_undegraded":"2024-04-23T10:24:45.042632+0000","last_fullsized":"2024-04-23T10:24:45.042632+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T10:24:43.025324+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T10:24:43.025324+0000","last_clean_scrub_stamp":"2024-04-23T10:24:43.025324+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T17:17:46.731121+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,11],"acting":[7,1,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.d","version":"86'1","reported_seq":17,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-04-23T10:24:46.310477+0000","last_change":"2024-04-23T10:24:43.052519+0000","last_active":"2024-04-23T10:24:46.310477+0000","last_peered":"2024-04-23T10:24:46.310477+0000","last_clean":"2024-04-23T10:24:46.310477+0000","last_became_active":"2024-04-23T10:24:43.052187+0000","last_became_peered":"2024-04-23T10:24:43.052187+0000","last_unstale":"2024-04-23T10:24:46.310477+0000","last_undegraded":"2024-04-23T10:24:46.310477+0000","last_fullsized":"2024-04-23T10:24:46.310477+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T10:24:42.023127+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T10:24:42.023127+0000","last_clean_scrub_stamp":"2024-04-23T10:24:42.023127+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T15:51:52.839409+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,10,3],"acting":[5,10,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.c","version":"0'0","reported_seq":13,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-04-23T10:24:45.042342+0000","last_change":"2024-04-23T10:24:44.053086+0000","last_active":"2024-04-23T10:24:45.042342+0000","last_peered":"2024-04-23T10:24:45.042342+0000","last_clean":"2024-04-23T10:24:45.042342+0000","last_became_active":"2024-04-23T10:24:44.052687+0000","last_became_peered":"2024-04-23T10:24:44.052687+0000","last_unstale":"2024-04-23T10:24:45.042342+0000","last_undegraded":"2024-04-23T10:24:45.042342+0000","last_fullsized":"2024-04-23T10:24:45.042342+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T10:24:43.025324+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T10:24:43.025324+0000","last_clean_scrub_stamp":"2024-04-23T10:24:43.025324+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T14:35:44.958742+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,5],"acting":[7,3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.e","version":"86'2","reported_seq":17,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-04-23T10:24:46.309746+0000","last_change":"2024-04-23T10:24:43.047716+0000","last_active":"2024-04-23T10:24:46.309746+0000","last_peered":"2024-04-23T10:24:46.309746+0000","last_clean":"2024-04-23T10:24:46.309746+0000","last_became_active":"2024-04-23T10:24:43.047357+0000","last_became_peered":"2024-04-23T10:24:43.047357+0000","last_unstale":"2024-04-23T10:24:46.309746+0000","last_undegraded":"2024-04-23T10:24:46.309746+0000","last_fullsized":"2024-04-23T10:24:46.309746+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T10:24:42.023127+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T10:24:42.023127+0000","last_clean_scrub_stamp":"2024-04-23T10:24:42.023127+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T14:25:11.593207+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,2],"acting":[6,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.f","version":"0'0","reported_seq":13,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-04-23T10:24:45.042610+0000","last_change":"2024-04-23T10:24:44.049543+0000","last_active":"2024-04-23T10:24:45.042610+0000","last_peered":"2024-04-23T10:24:45.042610+0000","last_clean":"2024-04-23T10:24:45.042610+0000","last_became_active":"2024-04-23T10:24:44.049198+0000","last_became_peered":"2024-04-23T10:24:44.049198+0000","last_unstale":"2024-04-23T10:24:45.042610+0000","last_undegraded":"2024-04-23T10:24:45.042610+0000","last_fullsized":"2024-04-23T10:24:45.042610+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T10:24:43.025324+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T10:24:43.025324+0000","last_clean_scrub_stamp":"2024-04-23T10:24:43.025324+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T12:25:15.974267+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,11,8],"acting":[7,11,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.20","version":"0'0","reported_seq":10,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-04-23T10:24:44.055115+0000","last_change":"2024-04-23T10:24:44.054440+0000","last_active":"2024-04-23T10:24:44.055115+0000","last_peered":"2024-04-23T10:24:44.055115+0000","last_clean":"2024-04-23T10:24:44.055115+0000","last_became_active":"2024-04-23T10:24:44.053951+0000","last_became_peered":"2024-04-23T10:24:44.053951+0000","last_unstale":"2024-04-23T10:24:44.055115+0000","last_undegraded":"2024-04-23T10:24:44.055115+0000","last_fullsized":"2024-04-23T10:24:44.055115+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T10:24:43.025324+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T10:24:43.025324+0000","last_clean_scrub_stamp":"2024-04-23T10:24:43.025324+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T20:26:51.035255+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,2,5],"acting":[11,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.21","version":"0'0","reported_seq":13,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-04-23T10:24:45.123991+0000","last_change":"2024-04-23T10:24:44.059015+0000","last_active":"2024-04-23T10:24:45.123991+0000","last_peered":"2024-04-23T10:24:45.123991+0000","last_clean":"2024-04-23T10:24:45.123991+0000","last_became_active":"2024-04-23T10:24:44.058691+0000","last_became_peered":"2024-04-23T10:24:44.058691+0000","last_unstale":"2024-04-23T10:24:45.123991+0000","last_undegraded":"2024-04-23T10:24:45.123991+0000","last_fullsized":"2024-04-23T10:24:45.123991+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T10:24:43.025324+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T10:24:43.025324+0000","last_clean_scrub_stamp":"2024-04-23T10:24:43.025324+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T14:57:48.153094+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,4],"acting":[2,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.22","version":"0'0","reported_seq":13,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-04-23T10:24:45.123994+0000","last_change":"2024-04-23T10:24:44.055534+0000","last_active":"2024-04-23T10:24:45.123994+0000","last_peered":"2024-04-23T10:24:45.123994+0000","last_clean":"2024-04-23T10:24:45.123994+0000","last_became_active":"2024-04-23T10:24:44.055181+0000","last_became_peered":"2024-04-23T10:24:44.055181+0000","last_unstale":"2024-04-23T10:24:45.123994+0000","last_undegraded":"2024-04-23T10:24:45.123994+0000","last_fullsized":"2024-04-23T10:24:45.123994+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T10:24:43.025324+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T10:24:43.025324+0000","last_clean_scrub_stamp":"2024-04-23T10:24:43.025324+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T13:56:33.878232+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.23","version":"0'0","reported_seq":13,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-04-23T10:24:45.041988+0000","last_change":"2024-04-23T10:24:44.053349+0000","last_active":"2024-04-23T10:24:45.041988+0000","last_peered":"2024-04-23T10:24:45.041988+0000","last_clean":"2024-04-23T10:24:45.041988+0000","last_became_active":"2024-04-23T10:24:44.053016+0000","last_became_peered":"2024-04-23T10:24:44.053016+0000","last_unstale":"2024-04-23T10:24:45.041988+0000","last_undegraded":"2024-04-23T10:24:45.041988+0000","last_fullsized":"2024-04-23T10:24:45.041988+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T10:24:43.025324+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T10:24:43.025324+0000","last_clean_scrub_stamp":"2024-04-23T10:24:43.025324+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T14:24:36.945889+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,4],"acting":[0,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.24","version":"0'0","reported_seq":13,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-04-23T10:24:46.309748+0000","last_change":"2024-04-23T10:24:44.050741+0000","last_active":"2024-04-23T10:24:46.309748+0000","last_peered":"2024-04-23T10:24:46.309748+0000","last_clean":"2024-04-23T10:24:46.309748+0000","last_became_active":"2024-04-23T10:24:44.050398+0000","last_became_peered":"2024-04-23T10:24:44.050398+0000","last_unstale":"2024-04-23T10:24:46.309748+0000","last_undegraded":"2024-04-23T10:24:46.309748+0000","last_fullsized":"2024-04-23T10:24:46.309748+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T10:24:43.025324+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T10:24:43.025324+0000","last_clean_scrub_stamp":"2024-04-23T10:24:43.025324+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T10:49:07.146673+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,0],"acting":[6,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.25","version":"0'0","reported_seq":10,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-04-23T10:24:44.049538+0000","last_change":"2024-04-23T10:24:44.048933+0000","last_active":"2024-04-23T10:24:44.049538+0000","last_peered":"2024-04-23T10:24:44.049538+0000","last_clean":"2024-04-23T10:24:44.049538+0000","last_became_active":"2024-04-23T10:24:44.048553+0000","last_became_peered":"2024-04-23T10:24:44.048553+0000","last_unstale":"2024-04-23T10:24:44.049538+0000","last_undegraded":"2024-04-23T10:24:44.049538+0000","last_fullsized":"2024-04-23T10:24:44.049538+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T10:24:43.025324+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T10:24:43.025324+0000","last_clean_scrub_stamp":"2024-04-23T10:24:43.025324+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T22:19:54.013031+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,8,5],"acting":[11,8,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.26","version":"0'0","reported_seq":13,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-04-23T10:24:45.124258+0000","last_change":"2024-04-23T10:24:44.051895+0000","last_active":"2024-04-23T10:24:45.124258+0000","last_peered":"2024-04-23T10:24:45.124258+0000","last_clean":"2024-04-23T10:24:45.124258+0000","last_became_active":"2024-04-23T10:24:44.051005+0000","last_became_peered":"2024-04-23T10:24:44.051005+0000","last_unstale":"2024-04-23T10:24:45.124258+0000","last_undegraded":"2024-04-23T10:24:45.124258+0000","last_fullsized":"2024-04-23T10:24:45.124258+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T10:24:43.025324+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T10:24:43.025324+0000","last_clean_scrub_stamp":"2024-04-23T10:24:43.025324+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T18:13:05.227365+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,8,1],"acting":[2,8,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.27","version":"0'0","reported_seq":13,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-04-23T10:24:46.309796+0000","last_change":"2024-04-23T10:24:44.051863+0000","last_active":"2024-04-23T10:24:46.309796+0000","last_peered":"2024-04-23T10:24:46.309796+0000","last_clean":"2024-04-23T10:24:46.309796+0000","last_became_active":"2024-04-23T10:24:44.051208+0000","last_became_peered":"2024-04-23T10:24:44.051208+0000","last_unstale":"2024-04-23T10:24:46.309796+0000","last_undegraded":"2024-04-23T10:24:46.309796+0000","last_fullsized":"2024-04-23T10:24:46.309796+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T10:24:43.025324+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T10:24:43.025324+0000","last_clean_scrub_stamp":"2024-04-23T10:24:43.025324+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T13:35:07.072598+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,2],"acting":[3,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.28","version":"0'0","reported_seq":13,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-04-23T10:24:46.309909+0000","last_change":"2024-04-23T10:24:44.044158+0000","last_active":"2024-04-23T10:24:46.309909+0000","last_peered":"2024-04-23T10:24:46.309909+0000","last_clean":"2024-04-23T10:24:46.309909+0000","last_became_active":"2024-04-23T10:24:44.043829+0000","last_became_peered":"2024-04-23T10:24:44.043829+0000","last_unstale":"2024-04-23T10:24:46.309909+0000","last_undegraded":"2024-04-23T10:24:46.309909+0000","last_fullsized":"2024-04-23T10:24:46.309909+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T10:24:43.025324+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T10:24:43.025324+0000","last_clean_scrub_stamp":"2024-04-23T10:24:43.025324+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T15:10:08.081831+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,9,3],"acting":[5,9,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.29","version":"0'0","reported_seq":13,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-04-23T10:24:46.310279+0000","last_change":"2024-04-23T10:24:44.051267+0000","last_active":"2024-04-23T10:24:46.310279+0000","last_peered":"2024-04-23T10:24:46.310279+0000","last_clean":"2024-04-23T10:24:46.310279+0000","last_became_active":"2024-04-23T10:24:44.050911+0000","last_became_peered":"2024-04-23T10:24:44.050911+0000","last_unstale":"2024-04-23T10:24:46.310279+0000","last_undegraded":"2024-04-23T10:24:46.310279+0000","last_fullsized":"2024-04-23T10:24:46.310279+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T10:24:43.025324+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T10:24:43.025324+0000","last_clean_scrub_stamp":"2024-04-23T10:24:43.025324+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T18:09:28.871427+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,11],"acting":[3,2,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.2a","version":"0'0","reported_seq":10,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-04-23T10:24:44.058465+0000","last_change":"2024-04-23T10:24:44.057653+0000","last_active":"2024-04-23T10:24:44.058465+0000","last_peered":"2024-04-23T10:24:44.058465+0000","last_clean":"2024-04-23T10:24:44.058465+0000","last_became_active":"2024-04-23T10:24:44.056627+0000","last_became_peered":"2024-04-23T10:24:44.056627+0000","last_unstale":"2024-04-23T10:24:44.058465+0000","last_undegraded":"2024-04-23T10:24:44.058465+0000","last_fullsized":"2024-04-23T10:24:44.058465+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T10:24:43.025324+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T10:24:43.025324+0000","last_clean_scrub_stamp":"2024-04-23T10:24:43.025324+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T16:06:51.295540+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,3],"acting":[1,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.2b","version":"0'0","reported_seq":13,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-04-23T10:24:46.310216+0000","last_change":"2024-04-23T10:24:44.059735+0000","last_active":"2024-04-23T10:24:46.310216+0000","last_peered":"2024-04-23T10:24:46.310216+0000","last_clean":"2024-04-23T10:24:46.310216+0000","last_became_active":"2024-04-23T10:24:44.059052+0000","last_became_peered":"2024-04-23T10:24:44.059052+0000","last_unstale":"2024-04-23T10:24:46.310216+0000","last_undegraded":"2024-04-23T10:24:46.310216+0000","last_fullsized":"2024-04-23T10:24:46.310216+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T10:24:43.025324+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T10:24:43.025324+0000","last_clean_scrub_stamp":"2024-04-23T10:24:43.025324+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T14:29:25.794655+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,0],"acting":[6,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.2c","version":"0'0","reported_seq":10,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-04-23T10:24:44.053091+0000","last_change":"2024-04-23T10:24:44.051831+0000","last_active":"2024-04-23T10:24:44.053091+0000","last_peered":"2024-04-23T10:24:44.053091+0000","last_clean":"2024-04-23T10:24:44.053091+0000","last_became_active":"2024-04-23T10:24:44.051480+0000","last_became_peered":"2024-04-23T10:24:44.051480+0000","last_unstale":"2024-04-23T10:24:44.053091+0000","last_undegraded":"2024-04-23T10:24:44.053091+0000","last_fullsized":"2024-04-23T10:24:44.053091+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T10:24:43.025324+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T10:24:43.025324+0000","last_clean_scrub_stamp":"2024-04-23T10:24:43.025324+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T18:53:58.599937+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,4,8],"acting":[11,4,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]}],"pool_stats":[{"poolid":3,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":117,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":192,"acting":192,"num_store_stats":12},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":136,"num_read_kb":119,"num_write":181,"num_write_kb":2938,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":1180736,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":4751360,"data_stored":4722944,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":137,"ondisk_log_size":137,"up":3,"acting":3,"num_store_stats":8},{"poolid":2,"num_pg":32,"stat_sum":{"num_bytes":5031,"num_objects":49,"num_object_clones":0,"num_object_copies":147,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":49,"num_whiteouts":0,"num_read":5,"num_read_kb":4,"num_write":99,"num_write_kb":38,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":30,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":192512,"data_stored":12719,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":90,"ondisk_log_size":90,"up":96,"acting":96,"num_store_stats":12}],"osd_stats":[{"osd":11,"up_from":82,"seq":352187318286,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32080,"kb_used_data":904,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93713072,"statfs":{"total":95995035648,"available":95962185728,"internally_reserved":0,"allocated":925696,"data_stored":439178,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1596,"internal_metadata":31914436},"hb_peers":[0,1,2,3,4,5,6,7,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":10,"up_from":74,"seq":317827579922,"num_pgs":26,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28548,"kb_used_data":1520,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93716604,"statfs":{"total":95995035648,"available":95965802496,"internally_reserved":0,"allocated":1556480,"data_stored":1039922,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1600,"internal_metadata":27654592},"hb_peers":[0,1,2,3,4,5,6,8,9,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":9,"up_from":69,"seq":296352743448,"num_pgs":17,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":27944,"kb_used_data":920,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717208,"statfs":{"total":95995035648,"available":95966420992,"internally_reserved":0,"allocated":942080,"data_stored":448330,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1600,"internal_metadata":27654592},"hb_peers":[0,1,2,3,4,5,6,7,8,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":8,"up_from":62,"seq":266287972382,"num_pgs":24,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32112,"kb_used_data":936,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93713040,"statfs":{"total":95995035648,"available":95962152960,"internally_reserved":0,"allocated":958464,"data_stored":448486,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,2,3,4,5,6,7,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":7,"up_from":53,"seq":227633266722,"num_pgs":25,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":36852,"kb_used_data":1508,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":93708300,"statfs":{"total":95995035648,"available":95957299200,"internally_reserved":0,"allocated":1544192,"data_stored":1039320,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":36174282},"hb_peers":[1,2,3,4,5,6,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":6,"up_from":47,"seq":201863462952,"num_pgs":20,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32100,"kb_used_data":920,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93713052,"statfs":{"total":95995035648,"available":95962165248,"internally_reserved":0,"allocated":942080,"data_stored":447838,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":31914438},"hb_peers":[0,1,2,3,4,5,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":1,"up_from":13,"seq":55834574911,"num_pgs":24,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":27944,"kb_used_data":916,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717208,"statfs":{"total":95995035648,"available":95966420992,"internally_reserved":0,"allocated":937984,"data_stored":440790,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,2,3,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":0,"up_from":8,"seq":34359738437,"num_pgs":25,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32104,"kb_used_data":928,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93713048,"statfs":{"total":95995035648,"available":95962161152,"internally_reserved":0,"allocated":950272,"data_stored":448442,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":31914438},"hb_peers":[1,2,3,4,5,6,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":2,"up_from":18,"seq":77309411387,"num_pgs":28,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":36272,"kb_used_data":932,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":93708880,"statfs":{"total":95995035648,"available":95957893120,"internally_reserved":0,"allocated":954368,"data_stored":449394,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":36174277},"hb_peers":[0,1,3,4,5,6,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":3,"up_from":25,"seq":107374182452,"num_pgs":17,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":27952,"kb_used_data":932,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717200,"statfs":{"total":95995035648,"available":95966412800,"internally_reserved":0,"allocated":954368,"data_stored":448032,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[0,1,2,4,5,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":4,"up_from":33,"seq":141733920816,"num_pgs":26,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":27944,"kb_used_data":920,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717208,"statfs":{"total":95995035648,"available":95966420992,"internally_reserved":0,"allocated":942080,"data_stored":447770,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[1,2,3,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":5,"up_from":39,"seq":167503724589,"num_pgs":30,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":36864,"kb_used_data":1524,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":93708288,"statfs":{"total":95995035648,"available":95957286912,"internally_reserved":0,"allocated":1560576,"data_stored":1039592,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":36174277},"hb_peers":[0,1,2,3,4,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":694,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":1612,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":1646,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":284,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":644,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":28672,"data_stored":1867,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":90,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":1792,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":738,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":784,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":36864,"data_stored":2568,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":117,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-04-23T10:24:51.042 DEBUG:teuthology.orchestra.run.smithi114:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-04-23T10:24:51.178 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:24:51.177+0000 7ff8b4260640 1 -- 172.21.15.114:0/929183351 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7ff8ac101fc0 msgr2=0x7ff8ac1023e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T10:24:51.178 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:24:51.177+0000 7ff8b4260640 1 --2- 172.21.15.114:0/929183351 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7ff8ac101fc0 0x7ff8ac1023e0 secure :-1 s=READY pgs=240 cs=0 l=1 rev1=1 crypto rx=0x7ff8a8009ca0 tx=0x7ff8a802a8c0 comp rx=0 tx=0).stop 2024-04-23T10:24:51.178 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:24:51.177+0000 7ff8b4260640 1 -- 172.21.15.114:0/929183351 shutdown_connections 2024-04-23T10:24:51.178 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:24:51.177+0000 7ff8b4260640 1 --2- 172.21.15.114:0/929183351 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7ff8ac103b70 0x7ff8ac10a400 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:24:51.178 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:24:51.177+0000 7ff8b4260640 1 --2- 172.21.15.114:0/929183351 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7ff8ac1031b0 0x7ff8ac103630 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:24:51.178 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:24:51.177+0000 7ff8b4260640 1 --2- 172.21.15.114:0/929183351 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7ff8ac101fc0 0x7ff8ac1023e0 unknown :-1 s=CLOSED pgs=240 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:24:51.178 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:24:51.177+0000 7ff8b4260640 1 -- 172.21.15.114:0/929183351 >> 172.21.15.114:0/929183351 conn(0x7ff8ac0fd530 msgr2=0x7ff8ac0ff970 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T10:24:51.178 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:24:51.177+0000 7ff8b4260640 1 -- 172.21.15.114:0/929183351 shutdown_connections 2024-04-23T10:24:51.179 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:24:51.177+0000 7ff8b4260640 1 -- 172.21.15.114:0/929183351 wait complete. 2024-04-23T10:24:51.179 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:24:51.177+0000 7ff8b4260640 1 Processor -- start 2024-04-23T10:24:51.179 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:24:51.177+0000 7ff8b4260640 1 -- start start 2024-04-23T10:24:51.179 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:24:51.177+0000 7ff8b4260640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7ff8ac101fc0 0x7ff8ac198270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T10:24:51.179 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:24:51.177+0000 7ff8b4260640 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7ff8ac1031b0 0x7ff8ac1987b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T10:24:51.179 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:24:51.177+0000 7ff8b4260640 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7ff8ac103b70 0x7ff8ac19f7d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T10:24:51.179 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:24:51.177+0000 7ff8b4260640 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x7ff8ac10db00 con 0x7ff8ac101fc0 2024-04-23T10:24:51.179 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:24:51.177+0000 7ff8b4260640 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 -- 0x7ff8ac10d980 con 0x7ff8ac103b70 2024-04-23T10:24:51.179 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:24:51.177+0000 7ff8b4260640 1 -- --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_getmap magic: 0 -- 0x7ff8ac10dc80 con 0x7ff8ac1031b0 2024-04-23T10:24:51.179 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:24:51.177+0000 7ff8b1ffc640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7ff8ac101fc0 0x7ff8ac198270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T10:24:51.179 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:24:51.177+0000 7ff8b1ffc640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7ff8ac101fc0 0x7ff8ac198270 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.114:43494/0 (socket says 172.21.15.114:43494) 2024-04-23T10:24:51.179 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:24:51.177+0000 7ff8b1ffc640 1 -- 172.21.15.114:0/4244578347 learned_addr learned my addr 172.21.15.114:0/4244578347 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-04-23T10:24:51.179 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:24:51.178+0000 7ff8b17fb640 1 --2- 172.21.15.114:0/4244578347 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7ff8ac1031b0 0x7ff8ac1987b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T10:24:51.179 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:24:51.178+0000 7ff8b27fd640 1 --2- 172.21.15.114:0/4244578347 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7ff8ac103b70 0x7ff8ac19f7d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T10:24:51.180 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:24:51.178+0000 7ff8b1ffc640 1 -- 172.21.15.114:0/4244578347 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7ff8ac1031b0 msgr2=0x7ff8ac1987b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T10:24:51.180 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:24:51.178+0000 7ff8b1ffc640 1 --2- 172.21.15.114:0/4244578347 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7ff8ac1031b0 0x7ff8ac1987b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:24:51.180 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:24:51.178+0000 7ff8b1ffc640 1 -- 172.21.15.114:0/4244578347 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7ff8ac103b70 msgr2=0x7ff8ac19f7d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T10:24:51.180 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:24:51.178+0000 7ff8b1ffc640 1 --2- 172.21.15.114:0/4244578347 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7ff8ac103b70 0x7ff8ac19f7d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:24:51.180 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:24:51.178+0000 7ff8b1ffc640 1 -- 172.21.15.114:0/4244578347 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff8ac19fe40 con 0x7ff8ac101fc0 2024-04-23T10:24:51.180 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:24:51.178+0000 7ff8b17fb640 1 --2- 172.21.15.114:0/4244578347 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7ff8ac1031b0 0x7ff8ac1987b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-23T10:24:51.180 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:24:51.178+0000 7ff8b27fd640 1 --2- 172.21.15.114:0/4244578347 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7ff8ac103b70 0x7ff8ac19f7d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-23T10:24:51.180 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:24:51.178+0000 7ff8b1ffc640 1 --2- 172.21.15.114:0/4244578347 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7ff8ac101fc0 0x7ff8ac198270 secure :-1 s=READY pgs=241 cs=0 l=1 rev1=1 crypto rx=0x7ff8a8009430 tx=0x7ff8a8031840 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T10:24:51.180 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:24:51.178+0000 7ff8a2ffd640 1 -- 172.21.15.114:0/4244578347 <== mon.0 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff8a8003cb0 con 0x7ff8ac101fc0 2024-04-23T10:24:51.180 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:24:51.178+0000 7ff8a2ffd640 1 -- 172.21.15.114:0/4244578347 <== mon.0 v2:172.21.15.114:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7ff8a8003e50 con 0x7ff8ac101fc0 2024-04-23T10:24:51.180 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:24:51.178+0000 7ff8b4260640 1 -- 172.21.15.114:0/4244578347 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7ff8ac1a00d0 con 0x7ff8ac101fc0 2024-04-23T10:24:51.180 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:24:51.178+0000 7ff8a2ffd640 1 -- 172.21.15.114:0/4244578347 <== mon.0 v2:172.21.15.114:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff8a802fc60 con 0x7ff8ac101fc0 2024-04-23T10:24:51.180 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:24:51.178+0000 7ff8b4260640 1 -- 172.21.15.114:0/4244578347 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7ff8ac1a0620 con 0x7ff8ac101fc0 2024-04-23T10:24:51.180 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:24:51.179+0000 7ff8b4260640 1 -- 172.21.15.114:0/4244578347 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7ff8ac1094a0 con 0x7ff8ac101fc0 2024-04-23T10:24:51.181 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:24:51.180+0000 7ff8a2ffd640 1 -- 172.21.15.114:0/4244578347 <== mon.0 v2:172.21.15.114:3300/0 4 ==== mgrmap(e 17) ==== 138698+0+0 (secure 0 0 0) 0x7ff8a8004040 con 0x7ff8ac101fc0 2024-04-23T10:24:51.181 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:24:51.181+0000 7ff8a2ffd640 1 --2- 172.21.15.114:0/4244578347 >> [v2:172.21.15.114:6800/465333808,v1:172.21.15.114:6801/465333808] conn(0x7ff8980a3c40 0x7ff8980a6100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T10:24:51.181 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:24:51.181+0000 7ff8b17fb640 1 --2- 172.21.15.114:0/4244578347 >> [v2:172.21.15.114:6800/465333808,v1:172.21.15.114:6801/465333808] conn(0x7ff8980a3c40 0x7ff8980a6100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T10:24:51.181 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:24:51.181+0000 7ff8a2ffd640 1 -- 172.21.15.114:0/4244578347 <== mon.0 v2:172.21.15.114:3300/0 5 ==== osd_map(87..87 src has 1..87) ==== 7979+0+0 (secure 0 0 0) 0x7ff8a80f19d0 con 0x7ff8ac101fc0 2024-04-23T10:24:51.181 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:24:51.181+0000 7ff8b17fb640 1 --2- 172.21.15.114:0/4244578347 >> [v2:172.21.15.114:6800/465333808,v1:172.21.15.114:6801/465333808] conn(0x7ff8980a3c40 0x7ff8980a6100 secure :-1 s=READY pgs=179 cs=0 l=1 rev1=1 crypto rx=0x7ff8ac199730 tx=0x7ff89c005e50 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T10:24:51.184 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:24:51.185+0000 7ff8a2ffd640 1 -- 172.21.15.114:0/4244578347 <== mon.0 v2:172.21.15.114:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7ff8a80be490 con 0x7ff8ac101fc0 2024-04-23T10:24:51.360 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:24:51.361+0000 7ff8b4260640 1 -- 172.21.15.114:0/4244578347 --> [v2:172.21.15.114:6800/465333808,v1:172.21.15.114:6801/465333808] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) -- 0x7ff8ac062390 con 0x7ff8980a3c40 2024-04-23T10:24:51.367 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:24:51.369+0000 7ff8a2ffd640 1 -- 172.21.15.114:0/4244578347 <== mgr.14152 v2:172.21.15.114:6800/465333808 1 ==== mgr_command_reply(tid 0: 0 dumped all) ==== 18+0+258753 (secure 0 0 0) 0x7ff8ac062390 con 0x7ff8980a3c40 2024-04-23T10:24:51.370 INFO:teuthology.orchestra.run.smithi114.stdout: 2024-04-23T10:24:51.371 INFO:teuthology.orchestra.run.smithi114.stderr:dumped all 2024-04-23T10:24:51.373 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:24:51.371+0000 7ff8b4260640 1 -- 172.21.15.114:0/4244578347 >> [v2:172.21.15.114:6800/465333808,v1:172.21.15.114:6801/465333808] conn(0x7ff8980a3c40 msgr2=0x7ff8980a6100 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T10:24:51.373 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:24:51.371+0000 7ff8b4260640 1 --2- 172.21.15.114:0/4244578347 >> [v2:172.21.15.114:6800/465333808,v1:172.21.15.114:6801/465333808] conn(0x7ff8980a3c40 0x7ff8980a6100 secure :-1 s=READY pgs=179 cs=0 l=1 rev1=1 crypto rx=0x7ff8ac199730 tx=0x7ff89c005e50 comp rx=0 tx=0).stop 2024-04-23T10:24:51.373 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:24:51.371+0000 7ff8b4260640 1 -- 172.21.15.114:0/4244578347 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7ff8ac101fc0 msgr2=0x7ff8ac198270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T10:24:51.373 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:24:51.371+0000 7ff8b4260640 1 --2- 172.21.15.114:0/4244578347 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7ff8ac101fc0 0x7ff8ac198270 secure :-1 s=READY pgs=241 cs=0 l=1 rev1=1 crypto rx=0x7ff8a8009430 tx=0x7ff8a8031840 comp rx=0 tx=0).stop 2024-04-23T10:24:51.373 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:24:51.371+0000 7ff8b4260640 1 -- 172.21.15.114:0/4244578347 shutdown_connections 2024-04-23T10:24:51.373 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:24:51.371+0000 7ff8b4260640 1 --2- 172.21.15.114:0/4244578347 >> [v2:172.21.15.114:6800/465333808,v1:172.21.15.114:6801/465333808] conn(0x7ff8980a3c40 0x7ff8980a6100 unknown :-1 s=CLOSED pgs=179 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:24:51.373 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:24:51.371+0000 7ff8b4260640 1 --2- 172.21.15.114:0/4244578347 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7ff8ac103b70 0x7ff8ac19f7d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:24:51.373 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:24:51.371+0000 7ff8b4260640 1 --2- 172.21.15.114:0/4244578347 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7ff8ac1031b0 0x7ff8ac1987b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:24:51.373 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:24:51.371+0000 7ff8b4260640 1 --2- 172.21.15.114:0/4244578347 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7ff8ac101fc0 0x7ff8ac198270 unknown :-1 s=CLOSED pgs=241 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:24:51.373 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:24:51.371+0000 7ff8b4260640 1 -- 172.21.15.114:0/4244578347 >> 172.21.15.114:0/4244578347 conn(0x7ff8ac0fd530 msgr2=0x7ff8ac106670 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T10:24:51.373 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:24:51.371+0000 7ff8b4260640 1 -- 172.21.15.114:0/4244578347 shutdown_connections 2024-04-23T10:24:51.374 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:24:51.371+0000 7ff8b4260640 1 -- 172.21.15.114:0/4244578347 wait complete. 2024-04-23T10:24:51.381 INFO:teuthology.orchestra.run.smithi114.stdout:{"pg_ready":true,"pg_map":{"version":289,"stamp":"2024-04-23T10:24:49.957169+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":595399,"num_objects":51,"num_object_clones":0,"num_object_copies":153,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":51,"num_whiteouts":0,"num_read":141,"num_read_kb":123,"num_write":280,"num_write_kb":2976,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":1180736,"num_keys_recovered":0,"num_objects_omap":30,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":227,"ondisk_log_size":227,"up":291,"acting":291,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":262,"num_osds":12,"num_per_pool_osds":12,"num_per_pool_omap_osds":12,"kb":1124941824,"kb_used":378716,"kb_used_data":12860,"kb_used_omap":18,"kb_used_meta":365677,"kb_avail":1124563108,"statfs":{"total":1151940427776,"available":1151552622592,"internally_reserved":0,"allocated":13168640,"data_stored":7137094,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":19142,"internal_metadata":374453562},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":10,"apply_latency_ms":10,"commit_latency_ns":10000000,"apply_latency_ns":10000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":5031,"num_objects":49,"num_object_clones":0,"num_object_copies":147,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":49,"num_whiteouts":0,"num_read":5,"num_read_kb":4,"num_write":99,"num_write_kb":38,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":30,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"6.926225"},"pg_stats":[{"pgid":"3.3f","version":"0'0","reported_seq":10,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-04-23T10:24:44.052035+0000","last_change":"2024-04-23T10:24:44.051465+0000","last_active":"2024-04-23T10:24:44.052035+0000","last_peered":"2024-04-23T10:24:44.052035+0000","last_clean":"2024-04-23T10:24:44.052035+0000","last_became_active":"2024-04-23T10:24:44.050863+0000","last_became_peered":"2024-04-23T10:24:44.050863+0000","last_unstale":"2024-04-23T10:24:44.052035+0000","last_undegraded":"2024-04-23T10:24:44.052035+0000","last_fullsized":"2024-04-23T10:24:44.052035+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T10:24:43.025324+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T10:24:43.025324+0000","last_clean_scrub_stamp":"2024-04-23T10:24:43.025324+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T15:52:57.416236+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,1,0],"acting":[11,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.3e","version":"0'0","reported_seq":13,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-04-23T10:24:45.041886+0000","last_change":"2024-04-23T10:24:44.050739+0000","last_active":"2024-04-23T10:24:45.041886+0000","last_peered":"2024-04-23T10:24:45.041886+0000","last_clean":"2024-04-23T10:24:45.041886+0000","last_became_active":"2024-04-23T10:24:44.050430+0000","last_became_peered":"2024-04-23T10:24:44.050430+0000","last_unstale":"2024-04-23T10:24:45.041886+0000","last_undegraded":"2024-04-23T10:24:45.041886+0000","last_fullsized":"2024-04-23T10:24:45.041886+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T10:24:43.025324+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T10:24:43.025324+0000","last_clean_scrub_stamp":"2024-04-23T10:24:43.025324+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T17:36:25.983147+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,11,10],"acting":[0,11,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3d","version":"0'0","reported_seq":13,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-04-23T10:24:45.041944+0000","last_change":"2024-04-23T10:24:44.054805+0000","last_active":"2024-04-23T10:24:45.041944+0000","last_peered":"2024-04-23T10:24:45.041944+0000","last_clean":"2024-04-23T10:24:45.041944+0000","last_became_active":"2024-04-23T10:24:44.054423+0000","last_became_peered":"2024-04-23T10:24:44.054423+0000","last_unstale":"2024-04-23T10:24:45.041944+0000","last_undegraded":"2024-04-23T10:24:45.041944+0000","last_fullsized":"2024-04-23T10:24:45.041944+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T10:24:43.025324+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T10:24:43.025324+0000","last_clean_scrub_stamp":"2024-04-23T10:24:43.025324+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T21:34:26.793624+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,1],"acting":[0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3c","version":"0'0","reported_seq":13,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-04-23T10:24:45.809205+0000","last_change":"2024-04-23T10:24:44.055966+0000","last_active":"2024-04-23T10:24:45.809205+0000","last_peered":"2024-04-23T10:24:45.809205+0000","last_clean":"2024-04-23T10:24:45.809205+0000","last_became_active":"2024-04-23T10:24:44.055028+0000","last_became_peered":"2024-04-23T10:24:44.055028+0000","last_unstale":"2024-04-23T10:24:45.809205+0000","last_undegraded":"2024-04-23T10:24:45.809205+0000","last_fullsized":"2024-04-23T10:24:45.809205+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T10:24:43.025324+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T10:24:43.025324+0000","last_clean_scrub_stamp":"2024-04-23T10:24:43.025324+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T19:41:04.594025+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,8,2],"acting":[4,8,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.3b","version":"0'0","reported_seq":13,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-04-23T10:24:45.044272+0000","last_change":"2024-04-23T10:24:44.046954+0000","last_active":"2024-04-23T10:24:45.044272+0000","last_peered":"2024-04-23T10:24:45.044272+0000","last_clean":"2024-04-23T10:24:45.044272+0000","last_became_active":"2024-04-23T10:24:44.046631+0000","last_became_peered":"2024-04-23T10:24:44.046631+0000","last_unstale":"2024-04-23T10:24:45.044272+0000","last_undegraded":"2024-04-23T10:24:45.044272+0000","last_fullsized":"2024-04-23T10:24:45.044272+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T10:24:43.025324+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T10:24:43.025324+0000","last_clean_scrub_stamp":"2024-04-23T10:24:43.025324+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T15:04:26.878894+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,3,6],"acting":[8,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.3a","version":"0'0","reported_seq":13,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-04-23T10:24:45.235291+0000","last_change":"2024-04-23T10:24:44.051550+0000","last_active":"2024-04-23T10:24:45.235291+0000","last_peered":"2024-04-23T10:24:45.235291+0000","last_clean":"2024-04-23T10:24:45.235291+0000","last_became_active":"2024-04-23T10:24:44.050469+0000","last_became_peered":"2024-04-23T10:24:44.050469+0000","last_unstale":"2024-04-23T10:24:45.235291+0000","last_undegraded":"2024-04-23T10:24:45.235291+0000","last_fullsized":"2024-04-23T10:24:45.235291+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T10:24:43.025324+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T10:24:43.025324+0000","last_clean_scrub_stamp":"2024-04-23T10:24:43.025324+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T21:13:42.469252+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,9,4],"acting":[10,9,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.39","version":"0'0","reported_seq":13,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-04-23T10:24:45.042070+0000","last_change":"2024-04-23T10:24:44.054916+0000","last_active":"2024-04-23T10:24:45.042070+0000","last_peered":"2024-04-23T10:24:45.042070+0000","last_clean":"2024-04-23T10:24:45.042070+0000","last_became_active":"2024-04-23T10:24:44.054536+0000","last_became_peered":"2024-04-23T10:24:44.054536+0000","last_unstale":"2024-04-23T10:24:45.042070+0000","last_undegraded":"2024-04-23T10:24:45.042070+0000","last_fullsized":"2024-04-23T10:24:45.042070+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T10:24:43.025324+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T10:24:43.025324+0000","last_clean_scrub_stamp":"2024-04-23T10:24:43.025324+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T19:04:13.402939+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,9,5],"acting":[7,9,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.38","version":"0'0","reported_seq":13,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-04-23T10:24:46.310399+0000","last_change":"2024-04-23T10:24:44.049026+0000","last_active":"2024-04-23T10:24:46.310399+0000","last_peered":"2024-04-23T10:24:46.310399+0000","last_clean":"2024-04-23T10:24:46.310399+0000","last_became_active":"2024-04-23T10:24:44.048683+0000","last_became_peered":"2024-04-23T10:24:44.048683+0000","last_unstale":"2024-04-23T10:24:46.310399+0000","last_undegraded":"2024-04-23T10:24:46.310399+0000","last_fullsized":"2024-04-23T10:24:46.310399+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T10:24:43.025324+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T10:24:43.025324+0000","last_clean_scrub_stamp":"2024-04-23T10:24:43.025324+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T19:26:53.489327+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,10],"acting":[3,5,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.37","version":"0'0","reported_seq":10,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-04-23T10:24:44.057965+0000","last_change":"2024-04-23T10:24:44.055261+0000","last_active":"2024-04-23T10:24:44.057965+0000","last_peered":"2024-04-23T10:24:44.057965+0000","last_clean":"2024-04-23T10:24:44.057965+0000","last_became_active":"2024-04-23T10:24:44.054405+0000","last_became_peered":"2024-04-23T10:24:44.054405+0000","last_unstale":"2024-04-23T10:24:44.057965+0000","last_undegraded":"2024-04-23T10:24:44.057965+0000","last_fullsized":"2024-04-23T10:24:44.057965+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T10:24:43.025324+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T10:24:43.025324+0000","last_clean_scrub_stamp":"2024-04-23T10:24:43.025324+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T21:11:43.804818+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,11],"acting":[1,5,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.36","version":"0'0","reported_seq":13,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-04-23T10:24:45.235708+0000","last_change":"2024-04-23T10:24:44.055854+0000","last_active":"2024-04-23T10:24:45.235708+0000","last_peered":"2024-04-23T10:24:45.235708+0000","last_clean":"2024-04-23T10:24:45.235708+0000","last_became_active":"2024-04-23T10:24:44.055275+0000","last_became_peered":"2024-04-23T10:24:44.055275+0000","last_unstale":"2024-04-23T10:24:45.235708+0000","last_undegraded":"2024-04-23T10:24:45.235708+0000","last_fullsized":"2024-04-23T10:24:45.235708+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T10:24:43.025324+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T10:24:43.025324+0000","last_clean_scrub_stamp":"2024-04-23T10:24:43.025324+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T21:48:25.934878+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,2,4],"acting":[10,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.35","version":"0'0","reported_seq":13,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-04-23T10:24:45.042491+0000","last_change":"2024-04-23T10:24:44.061314+0000","last_active":"2024-04-23T10:24:45.042491+0000","last_peered":"2024-04-23T10:24:45.042491+0000","last_clean":"2024-04-23T10:24:45.042491+0000","last_became_active":"2024-04-23T10:24:44.060981+0000","last_became_peered":"2024-04-23T10:24:44.060981+0000","last_unstale":"2024-04-23T10:24:45.042491+0000","last_undegraded":"2024-04-23T10:24:45.042491+0000","last_fullsized":"2024-04-23T10:24:45.042491+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T10:24:43.025324+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T10:24:43.025324+0000","last_clean_scrub_stamp":"2024-04-23T10:24:43.025324+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T11:48:08.387575+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,8,1],"acting":[7,8,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.34","version":"0'0","reported_seq":13,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-04-23T10:24:45.809201+0000","last_change":"2024-04-23T10:24:44.055335+0000","last_active":"2024-04-23T10:24:45.809201+0000","last_peered":"2024-04-23T10:24:45.809201+0000","last_clean":"2024-04-23T10:24:45.809201+0000","last_became_active":"2024-04-23T10:24:44.051257+0000","last_became_peered":"2024-04-23T10:24:44.051257+0000","last_unstale":"2024-04-23T10:24:45.809201+0000","last_undegraded":"2024-04-23T10:24:45.809201+0000","last_fullsized":"2024-04-23T10:24:45.809201+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T10:24:43.025324+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T10:24:43.025324+0000","last_clean_scrub_stamp":"2024-04-23T10:24:43.025324+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T11:57:45.999751+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,10,8],"acting":[4,10,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.33","version":"0'0","reported_seq":13,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-04-23T10:24:46.310013+0000","last_change":"2024-04-23T10:24:44.052078+0000","last_active":"2024-04-23T10:24:46.310013+0000","last_peered":"2024-04-23T10:24:46.310013+0000","last_clean":"2024-04-23T10:24:46.310013+0000","last_became_active":"2024-04-23T10:24:44.051747+0000","last_became_peered":"2024-04-23T10:24:44.051747+0000","last_unstale":"2024-04-23T10:24:46.310013+0000","last_undegraded":"2024-04-23T10:24:46.310013+0000","last_fullsized":"2024-04-23T10:24:46.310013+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T10:24:43.025324+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T10:24:43.025324+0000","last_clean_scrub_stamp":"2024-04-23T10:24:43.025324+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T19:14:16.495967+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,10,5],"acting":[6,10,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.32","version":"0'0","reported_seq":13,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-04-23T10:24:45.041784+0000","last_change":"2024-04-23T10:24:44.051994+0000","last_active":"2024-04-23T10:24:45.041784+0000","last_peered":"2024-04-23T10:24:45.041784+0000","last_clean":"2024-04-23T10:24:45.041784+0000","last_became_active":"2024-04-23T10:24:44.051638+0000","last_became_peered":"2024-04-23T10:24:44.051638+0000","last_unstale":"2024-04-23T10:24:45.041784+0000","last_undegraded":"2024-04-23T10:24:45.041784+0000","last_fullsized":"2024-04-23T10:24:45.041784+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T10:24:43.025324+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T10:24:43.025324+0000","last_clean_scrub_stamp":"2024-04-23T10:24:43.025324+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T14:28:30.447198+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,8,10],"acting":[0,8,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.31","version":"0'0","reported_seq":13,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-04-23T10:24:45.041962+0000","last_change":"2024-04-23T10:24:44.055779+0000","last_active":"2024-04-23T10:24:45.041962+0000","last_peered":"2024-04-23T10:24:45.041962+0000","last_clean":"2024-04-23T10:24:45.041962+0000","last_became_active":"2024-04-23T10:24:44.054872+0000","last_became_peered":"2024-04-23T10:24:44.054872+0000","last_unstale":"2024-04-23T10:24:45.041962+0000","last_undegraded":"2024-04-23T10:24:45.041962+0000","last_fullsized":"2024-04-23T10:24:45.041962+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T10:24:43.025324+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T10:24:43.025324+0000","last_clean_scrub_stamp":"2024-04-23T10:24:43.025324+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T16:45:32.273256+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,2],"acting":[0,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.30","version":"0'0","reported_seq":10,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-04-23T10:24:44.058403+0000","last_change":"2024-04-23T10:24:44.057154+0000","last_active":"2024-04-23T10:24:44.058403+0000","last_peered":"2024-04-23T10:24:44.058403+0000","last_clean":"2024-04-23T10:24:44.058403+0000","last_became_active":"2024-04-23T10:24:44.056836+0000","last_became_peered":"2024-04-23T10:24:44.056836+0000","last_unstale":"2024-04-23T10:24:44.058403+0000","last_undegraded":"2024-04-23T10:24:44.058403+0000","last_fullsized":"2024-04-23T10:24:44.058403+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T10:24:43.025324+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T10:24:43.025324+0000","last_clean_scrub_stamp":"2024-04-23T10:24:43.025324+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T21:21:50.715307+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,0],"acting":[1,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.2f","version":"0'0","reported_seq":10,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-04-23T10:24:44.052014+0000","last_change":"2024-04-23T10:24:44.051175+0000","last_active":"2024-04-23T10:24:44.052014+0000","last_peered":"2024-04-23T10:24:44.052014+0000","last_clean":"2024-04-23T10:24:44.052014+0000","last_became_active":"2024-04-23T10:24:44.050402+0000","last_became_peered":"2024-04-23T10:24:44.050402+0000","last_unstale":"2024-04-23T10:24:44.052014+0000","last_undegraded":"2024-04-23T10:24:44.052014+0000","last_fullsized":"2024-04-23T10:24:44.052014+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T10:24:43.025324+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T10:24:43.025324+0000","last_clean_scrub_stamp":"2024-04-23T10:24:43.025324+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T16:29:04.971067+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,1,0],"acting":[11,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.2e","version":"0'0","reported_seq":13,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-04-23T10:24:45.609038+0000","last_change":"2024-04-23T10:24:44.053390+0000","last_active":"2024-04-23T10:24:45.609038+0000","last_peered":"2024-04-23T10:24:45.609038+0000","last_clean":"2024-04-23T10:24:45.609038+0000","last_became_active":"2024-04-23T10:24:44.052970+0000","last_became_peered":"2024-04-23T10:24:44.052970+0000","last_unstale":"2024-04-23T10:24:45.609038+0000","last_undegraded":"2024-04-23T10:24:45.609038+0000","last_fullsized":"2024-04-23T10:24:45.609038+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T10:24:43.025324+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T10:24:43.025324+0000","last_clean_scrub_stamp":"2024-04-23T10:24:43.025324+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T13:27:55.704328+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,1,10],"acting":[9,1,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.2d","version":"0'0","reported_seq":13,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-04-23T10:24:45.124137+0000","last_change":"2024-04-23T10:24:44.049307+0000","last_active":"2024-04-23T10:24:45.124137+0000","last_peered":"2024-04-23T10:24:45.124137+0000","last_clean":"2024-04-23T10:24:45.124137+0000","last_became_active":"2024-04-23T10:24:44.048981+0000","last_became_peered":"2024-04-23T10:24:44.048981+0000","last_unstale":"2024-04-23T10:24:45.124137+0000","last_undegraded":"2024-04-23T10:24:45.124137+0000","last_fullsized":"2024-04-23T10:24:45.124137+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T10:24:43.025324+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T10:24:43.025324+0000","last_clean_scrub_stamp":"2024-04-23T10:24:43.025324+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T12:42:21.804591+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,6],"acting":[2,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.1f","version":"86'4","reported_seq":19,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-04-23T10:24:45.608885+0000","last_change":"2024-04-23T10:24:43.045804+0000","last_active":"2024-04-23T10:24:45.608885+0000","last_peered":"2024-04-23T10:24:45.608885+0000","last_clean":"2024-04-23T10:24:45.608885+0000","last_became_active":"2024-04-23T10:24:43.045135+0000","last_became_peered":"2024-04-23T10:24:43.045135+0000","last_unstale":"2024-04-23T10:24:45.608885+0000","last_undegraded":"2024-04-23T10:24:45.608885+0000","last_fullsized":"2024-04-23T10:24:45.608885+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T10:24:42.023127+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T10:24:42.023127+0000","last_clean_scrub_stamp":"2024-04-23T10:24:42.023127+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T20:27:28.421644+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,7,8],"acting":[9,7,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.1e","version":"0'0","reported_seq":13,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-04-23T10:24:46.309996+0000","last_change":"2024-04-23T10:24:44.055008+0000","last_active":"2024-04-23T10:24:46.309996+0000","last_peered":"2024-04-23T10:24:46.309996+0000","last_clean":"2024-04-23T10:24:46.309996+0000","last_became_active":"2024-04-23T10:24:44.054682+0000","last_became_peered":"2024-04-23T10:24:44.054682+0000","last_unstale":"2024-04-23T10:24:46.309996+0000","last_undegraded":"2024-04-23T10:24:46.309996+0000","last_fullsized":"2024-04-23T10:24:46.309996+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T10:24:43.025324+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T10:24:43.025324+0000","last_clean_scrub_stamp":"2024-04-23T10:24:43.025324+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T15:49:58.394958+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,5],"acting":[3,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.1e","version":"87'2","reported_seq":17,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-04-23T10:24:46.310088+0000","last_change":"2024-04-23T10:24:43.052303+0000","last_active":"2024-04-23T10:24:46.310088+0000","last_peered":"2024-04-23T10:24:46.310088+0000","last_clean":"2024-04-23T10:24:46.310088+0000","last_became_active":"2024-04-23T10:24:43.052010+0000","last_became_peered":"2024-04-23T10:24:43.052010+0000","last_unstale":"2024-04-23T10:24:46.310088+0000","last_undegraded":"2024-04-23T10:24:46.310088+0000","last_fullsized":"2024-04-23T10:24:46.310088+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T10:24:42.023127+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T10:24:42.023127+0000","last_clean_scrub_stamp":"2024-04-23T10:24:42.023127+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T13:03:15.440610+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,5],"acting":[3,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1f","version":"0'0","reported_seq":13,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-04-23T10:24:45.042148+0000","last_change":"2024-04-23T10:24:44.050749+0000","last_active":"2024-04-23T10:24:45.042148+0000","last_peered":"2024-04-23T10:24:45.042148+0000","last_clean":"2024-04-23T10:24:45.042148+0000","last_became_active":"2024-04-23T10:24:44.049780+0000","last_became_peered":"2024-04-23T10:24:44.049780+0000","last_unstale":"2024-04-23T10:24:45.042148+0000","last_undegraded":"2024-04-23T10:24:45.042148+0000","last_fullsized":"2024-04-23T10:24:45.042148+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T10:24:43.025324+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T10:24:43.025324+0000","last_clean_scrub_stamp":"2024-04-23T10:24:43.025324+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T16:32:37.963776+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,9],"acting":[0,1,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1d","version":"87'4","reported_seq":19,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-04-23T10:24:47.353492+0000","last_change":"2024-04-23T10:24:43.045847+0000","last_active":"2024-04-23T10:24:47.353492+0000","last_peered":"2024-04-23T10:24:47.353492+0000","last_clean":"2024-04-23T10:24:47.353492+0000","last_became_active":"2024-04-23T10:24:43.045554+0000","last_became_peered":"2024-04-23T10:24:43.045554+0000","last_unstale":"2024-04-23T10:24:47.353492+0000","last_undegraded":"2024-04-23T10:24:47.353492+0000","last_fullsized":"2024-04-23T10:24:47.353492+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T10:24:42.023127+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T10:24:42.023127+0000","last_clean_scrub_stamp":"2024-04-23T10:24:42.023127+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T18:27:50.240680+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,2,7],"acting":[8,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.1c","version":"0'0","reported_seq":10,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-04-23T10:24:44.052233+0000","last_change":"2024-04-23T10:24:44.051606+0000","last_active":"2024-04-23T10:24:44.052233+0000","last_peered":"2024-04-23T10:24:44.052233+0000","last_clean":"2024-04-23T10:24:44.052233+0000","last_became_active":"2024-04-23T10:24:44.051191+0000","last_became_peered":"2024-04-23T10:24:44.051191+0000","last_unstale":"2024-04-23T10:24:44.052233+0000","last_undegraded":"2024-04-23T10:24:44.052233+0000","last_fullsized":"2024-04-23T10:24:44.052233+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T10:24:43.025324+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T10:24:43.025324+0000","last_clean_scrub_stamp":"2024-04-23T10:24:43.025324+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T16:46:57.770438+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,10,6],"acting":[1,10,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1c","version":"86'2","reported_seq":17,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-04-23T10:24:45.044439+0000","last_change":"2024-04-23T10:24:43.048594+0000","last_active":"2024-04-23T10:24:45.044439+0000","last_peered":"2024-04-23T10:24:45.044439+0000","last_clean":"2024-04-23T10:24:45.044439+0000","last_became_active":"2024-04-23T10:24:43.048276+0000","last_became_peered":"2024-04-23T10:24:43.048276+0000","last_unstale":"2024-04-23T10:24:45.044439+0000","last_undegraded":"2024-04-23T10:24:45.044439+0000","last_fullsized":"2024-04-23T10:24:45.044439+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T10:24:42.023127+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T10:24:42.023127+0000","last_clean_scrub_stamp":"2024-04-23T10:24:42.023127+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T12:20:40.196674+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,1,5],"acting":[8,1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.1d","version":"0'0","reported_seq":13,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-04-23T10:24:46.310431+0000","last_change":"2024-04-23T10:24:44.050970+0000","last_active":"2024-04-23T10:24:46.310431+0000","last_peered":"2024-04-23T10:24:46.310431+0000","last_clean":"2024-04-23T10:24:46.310431+0000","last_became_active":"2024-04-23T10:24:44.050640+0000","last_became_peered":"2024-04-23T10:24:44.050640+0000","last_unstale":"2024-04-23T10:24:46.310431+0000","last_undegraded":"2024-04-23T10:24:46.310431+0000","last_fullsized":"2024-04-23T10:24:46.310431+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T10:24:43.025324+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T10:24:43.025324+0000","last_clean_scrub_stamp":"2024-04-23T10:24:43.025324+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T21:03:24.570290+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,1],"acting":[5,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.1b","version":"87'6","reported_seq":24,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-04-23T10:24:48.324814+0000","last_change":"2024-04-23T10:24:43.051742+0000","last_active":"2024-04-23T10:24:48.324814+0000","last_peered":"2024-04-23T10:24:48.324814+0000","last_clean":"2024-04-23T10:24:48.324814+0000","last_became_active":"2024-04-23T10:24:43.051420+0000","last_became_peered":"2024-04-23T10:24:43.051420+0000","last_unstale":"2024-04-23T10:24:48.324814+0000","last_undegraded":"2024-04-23T10:24:48.324814+0000","last_fullsized":"2024-04-23T10:24:48.324814+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T10:24:42.023127+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T10:24:42.023127+0000","last_clean_scrub_stamp":"2024-04-23T10:24:42.023127+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T17:54:37.182731+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1013,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":3,"num_read_kb":2,"num_write":3,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,7],"acting":[5,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.1a","version":"0'0","reported_seq":13,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-04-23T10:24:45.044567+0000","last_change":"2024-04-23T10:24:44.049314+0000","last_active":"2024-04-23T10:24:45.044567+0000","last_peered":"2024-04-23T10:24:45.044567+0000","last_clean":"2024-04-23T10:24:45.044567+0000","last_became_active":"2024-04-23T10:24:44.048352+0000","last_became_peered":"2024-04-23T10:24:44.048352+0000","last_unstale":"2024-04-23T10:24:45.044567+0000","last_undegraded":"2024-04-23T10:24:45.044567+0000","last_fullsized":"2024-04-23T10:24:45.044567+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T10:24:43.025324+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T10:24:43.025324+0000","last_clean_scrub_stamp":"2024-04-23T10:24:43.025324+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T16:57:29.957716+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,9,5],"acting":[8,9,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"2.1a","version":"86'1","reported_seq":16,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-04-23T10:24:45.235084+0000","last_change":"2024-04-23T10:24:43.051362+0000","last_active":"2024-04-23T10:24:45.235084+0000","last_peered":"2024-04-23T10:24:45.235084+0000","last_clean":"2024-04-23T10:24:45.235084+0000","last_became_active":"2024-04-23T10:24:43.051045+0000","last_became_peered":"2024-04-23T10:24:43.051045+0000","last_unstale":"2024-04-23T10:24:45.235084+0000","last_undegraded":"2024-04-23T10:24:45.235084+0000","last_fullsized":"2024-04-23T10:24:45.235084+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T10:24:42.023127+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T10:24:42.023127+0000","last_clean_scrub_stamp":"2024-04-23T10:24:42.023127+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T15:40:53.618281+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,8,6],"acting":[10,8,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.1b","version":"0'0","reported_seq":13,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-04-23T10:24:46.310386+0000","last_change":"2024-04-23T10:24:44.040323+0000","last_active":"2024-04-23T10:24:46.310386+0000","last_peered":"2024-04-23T10:24:46.310386+0000","last_clean":"2024-04-23T10:24:46.310386+0000","last_became_active":"2024-04-23T10:24:44.039995+0000","last_became_peered":"2024-04-23T10:24:44.039995+0000","last_unstale":"2024-04-23T10:24:46.310386+0000","last_undegraded":"2024-04-23T10:24:46.310386+0000","last_fullsized":"2024-04-23T10:24:46.310386+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T10:24:43.025324+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T10:24:43.025324+0000","last_clean_scrub_stamp":"2024-04-23T10:24:43.025324+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T16:38:48.756776+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,3],"acting":[5,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.19","version":"87'2","reported_seq":17,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-04-23T10:24:47.355005+0000","last_change":"2024-04-23T10:24:43.048754+0000","last_active":"2024-04-23T10:24:47.355005+0000","last_peered":"2024-04-23T10:24:47.355005+0000","last_clean":"2024-04-23T10:24:47.355005+0000","last_became_active":"2024-04-23T10:24:43.047962+0000","last_became_peered":"2024-04-23T10:24:43.047962+0000","last_unstale":"2024-04-23T10:24:47.355005+0000","last_undegraded":"2024-04-23T10:24:47.355005+0000","last_fullsized":"2024-04-23T10:24:47.355005+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T10:24:42.023127+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T10:24:42.023127+0000","last_clean_scrub_stamp":"2024-04-23T10:24:42.023127+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T21:08:38.487185+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,9],"acting":[6,4,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.18","version":"0'0","reported_seq":13,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-04-23T10:24:46.310429+0000","last_change":"2024-04-23T10:24:44.045022+0000","last_active":"2024-04-23T10:24:46.310429+0000","last_peered":"2024-04-23T10:24:46.310429+0000","last_clean":"2024-04-23T10:24:46.310429+0000","last_became_active":"2024-04-23T10:24:44.044590+0000","last_became_peered":"2024-04-23T10:24:44.044590+0000","last_unstale":"2024-04-23T10:24:46.310429+0000","last_undegraded":"2024-04-23T10:24:46.310429+0000","last_fullsized":"2024-04-23T10:24:46.310429+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T10:24:43.025324+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T10:24:43.025324+0000","last_clean_scrub_stamp":"2024-04-23T10:24:43.025324+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T13:20:49.892013+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,11,6],"acting":[3,11,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.18","version":"87'1","reported_seq":16,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-04-23T10:24:46.309777+0000","last_change":"2024-04-23T10:24:43.050170+0000","last_active":"2024-04-23T10:24:46.309777+0000","last_peered":"2024-04-23T10:24:46.309777+0000","last_clean":"2024-04-23T10:24:46.309777+0000","last_became_active":"2024-04-23T10:24:43.049070+0000","last_became_peered":"2024-04-23T10:24:43.049070+0000","last_unstale":"2024-04-23T10:24:46.309777+0000","last_undegraded":"2024-04-23T10:24:46.309777+0000","last_fullsized":"2024-04-23T10:24:46.309777+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T10:24:42.023127+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T10:24:42.023127+0000","last_clean_scrub_stamp":"2024-04-23T10:24:42.023127+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T19:34:13.983628+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,7],"acting":[5,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.19","version":"0'0","reported_seq":13,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-04-23T10:24:45.608801+0000","last_change":"2024-04-23T10:24:44.047165+0000","last_active":"2024-04-23T10:24:45.608801+0000","last_peered":"2024-04-23T10:24:45.608801+0000","last_clean":"2024-04-23T10:24:45.608801+0000","last_became_active":"2024-04-23T10:24:44.046776+0000","last_became_peered":"2024-04-23T10:24:44.046776+0000","last_unstale":"2024-04-23T10:24:45.608801+0000","last_undegraded":"2024-04-23T10:24:45.608801+0000","last_fullsized":"2024-04-23T10:24:45.608801+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T10:24:43.025324+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T10:24:43.025324+0000","last_clean_scrub_stamp":"2024-04-23T10:24:43.025324+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T18:11:57.181747+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,3,10],"acting":[9,3,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"2.17","version":"87'3","reported_seq":18,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-04-23T10:24:46.346825+0000","last_change":"2024-04-23T10:24:43.049785+0000","last_active":"2024-04-23T10:24:46.346825+0000","last_peered":"2024-04-23T10:24:46.346825+0000","last_clean":"2024-04-23T10:24:46.346825+0000","last_became_active":"2024-04-23T10:24:43.049402+0000","last_became_peered":"2024-04-23T10:24:43.049402+0000","last_unstale":"2024-04-23T10:24:46.346825+0000","last_undegraded":"2024-04-23T10:24:46.346825+0000","last_fullsized":"2024-04-23T10:24:46.346825+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T10:24:42.023127+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T10:24:42.023127+0000","last_clean_scrub_stamp":"2024-04-23T10:24:42.023127+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T14:20:13.774082+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,5,9],"acting":[10,5,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.16","version":"0'0","reported_seq":10,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-04-23T10:24:44.053989+0000","last_change":"2024-04-23T10:24:44.051768+0000","last_active":"2024-04-23T10:24:44.053989+0000","last_peered":"2024-04-23T10:24:44.053989+0000","last_clean":"2024-04-23T10:24:44.053989+0000","last_became_active":"2024-04-23T10:24:44.051228+0000","last_became_peered":"2024-04-23T10:24:44.051228+0000","last_unstale":"2024-04-23T10:24:44.053989+0000","last_undegraded":"2024-04-23T10:24:44.053989+0000","last_fullsized":"2024-04-23T10:24:44.053989+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T10:24:43.025324+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T10:24:43.025324+0000","last_clean_scrub_stamp":"2024-04-23T10:24:43.025324+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T11:23:59.992132+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,10,9],"acting":[1,10,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.16","version":"87'2","reported_seq":17,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-04-23T10:24:46.346750+0000","last_change":"2024-04-23T10:24:43.043944+0000","last_active":"2024-04-23T10:24:46.346750+0000","last_peered":"2024-04-23T10:24:46.346750+0000","last_clean":"2024-04-23T10:24:46.346750+0000","last_became_active":"2024-04-23T10:24:43.043648+0000","last_became_peered":"2024-04-23T10:24:43.043648+0000","last_unstale":"2024-04-23T10:24:46.346750+0000","last_undegraded":"2024-04-23T10:24:46.346750+0000","last_fullsized":"2024-04-23T10:24:46.346750+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T10:24:42.023127+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T10:24:42.023127+0000","last_clean_scrub_stamp":"2024-04-23T10:24:42.023127+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T13:41:15.724662+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,2],"acting":[7,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.17","version":"0'0","reported_seq":13,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-04-23T10:24:45.042106+0000","last_change":"2024-04-23T10:24:44.052369+0000","last_active":"2024-04-23T10:24:45.042106+0000","last_peered":"2024-04-23T10:24:45.042106+0000","last_clean":"2024-04-23T10:24:45.042106+0000","last_became_active":"2024-04-23T10:24:44.052009+0000","last_became_peered":"2024-04-23T10:24:44.052009+0000","last_unstale":"2024-04-23T10:24:45.042106+0000","last_undegraded":"2024-04-23T10:24:45.042106+0000","last_fullsized":"2024-04-23T10:24:45.042106+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T10:24:43.025324+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T10:24:43.025324+0000","last_clean_scrub_stamp":"2024-04-23T10:24:43.025324+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T22:23:01.913219+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,5],"acting":[0,4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.15","version":"87'4","reported_seq":20,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-04-23T10:24:46.346970+0000","last_change":"2024-04-23T10:24:43.048517+0000","last_active":"2024-04-23T10:24:46.346970+0000","last_peered":"2024-04-23T10:24:46.346970+0000","last_clean":"2024-04-23T10:24:46.346970+0000","last_became_active":"2024-04-23T10:24:43.048183+0000","last_became_peered":"2024-04-23T10:24:43.048183+0000","last_unstale":"2024-04-23T10:24:46.346970+0000","last_undegraded":"2024-04-23T10:24:46.346970+0000","last_fullsized":"2024-04-23T10:24:46.346970+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T10:24:42.023127+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T10:24:42.023127+0000","last_clean_scrub_stamp":"2024-04-23T10:24:42.023127+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T20:50:15.017833+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":672,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":5,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,10],"acting":[5,0,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.14","version":"0'0","reported_seq":10,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-04-23T10:24:44.058007+0000","last_change":"2024-04-23T10:24:44.055598+0000","last_active":"2024-04-23T10:24:44.058007+0000","last_peered":"2024-04-23T10:24:44.058007+0000","last_clean":"2024-04-23T10:24:44.058007+0000","last_became_active":"2024-04-23T10:24:44.053787+0000","last_became_peered":"2024-04-23T10:24:44.053787+0000","last_unstale":"2024-04-23T10:24:44.058007+0000","last_undegraded":"2024-04-23T10:24:44.058007+0000","last_fullsized":"2024-04-23T10:24:44.058007+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T10:24:43.025324+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T10:24:43.025324+0000","last_clean_scrub_stamp":"2024-04-23T10:24:43.025324+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T17:55:30.488748+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,8,4],"acting":[1,8,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.14","version":"87'8","reported_seq":23,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-04-23T10:24:46.345804+0000","last_change":"2024-04-23T10:24:43.044601+0000","last_active":"2024-04-23T10:24:46.345804+0000","last_peered":"2024-04-23T10:24:46.345804+0000","last_clean":"2024-04-23T10:24:46.345804+0000","last_became_active":"2024-04-23T10:24:43.044238+0000","last_became_peered":"2024-04-23T10:24:43.044238+0000","last_unstale":"2024-04-23T10:24:46.345804+0000","last_undegraded":"2024-04-23T10:24:46.345804+0000","last_fullsized":"2024-04-23T10:24:46.345804+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T10:24:42.023127+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T10:24:42.023127+0000","last_clean_scrub_stamp":"2024-04-23T10:24:42.023127+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T12:53:26.275771+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1612,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,1,2],"acting":[10,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.15","version":"0'0","reported_seq":13,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-04-23T10:24:45.044495+0000","last_change":"2024-04-23T10:24:44.051375+0000","last_active":"2024-04-23T10:24:45.044495+0000","last_peered":"2024-04-23T10:24:45.044495+0000","last_clean":"2024-04-23T10:24:45.044495+0000","last_became_active":"2024-04-23T10:24:44.050924+0000","last_became_peered":"2024-04-23T10:24:44.050924+0000","last_unstale":"2024-04-23T10:24:45.044495+0000","last_undegraded":"2024-04-23T10:24:45.044495+0000","last_fullsized":"2024-04-23T10:24:45.044495+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T10:24:43.025324+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T10:24:43.025324+0000","last_clean_scrub_stamp":"2024-04-23T10:24:43.025324+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T21:50:57.534760+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,11,7],"acting":[8,11,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"2.13","version":"86'2","reported_seq":17,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-04-23T10:24:45.123954+0000","last_change":"2024-04-23T10:24:43.044457+0000","last_active":"2024-04-23T10:24:45.123954+0000","last_peered":"2024-04-23T10:24:45.123954+0000","last_clean":"2024-04-23T10:24:45.123954+0000","last_became_active":"2024-04-23T10:24:43.043956+0000","last_became_peered":"2024-04-23T10:24:43.043956+0000","last_unstale":"2024-04-23T10:24:45.123954+0000","last_undegraded":"2024-04-23T10:24:45.123954+0000","last_fullsized":"2024-04-23T10:24:45.123954+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T10:24:42.023127+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T10:24:42.023127+0000","last_clean_scrub_stamp":"2024-04-23T10:24:42.023127+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T20:16:08.186729+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,10,3],"acting":[2,10,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.12","version":"0'0","reported_seq":13,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-04-23T10:24:46.309871+0000","last_change":"2024-04-23T10:24:44.059571+0000","last_active":"2024-04-23T10:24:46.309871+0000","last_peered":"2024-04-23T10:24:46.309871+0000","last_clean":"2024-04-23T10:24:46.309871+0000","last_became_active":"2024-04-23T10:24:44.059196+0000","last_became_peered":"2024-04-23T10:24:44.059196+0000","last_unstale":"2024-04-23T10:24:46.309871+0000","last_undegraded":"2024-04-23T10:24:46.309871+0000","last_fullsized":"2024-04-23T10:24:46.309871+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T10:24:43.025324+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T10:24:43.025324+0000","last_clean_scrub_stamp":"2024-04-23T10:24:43.025324+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T22:09:15.659402+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,3],"acting":[6,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.12","version":"87'3","reported_seq":18,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-04-23T10:24:46.347868+0000","last_change":"2024-04-23T10:24:43.052181+0000","last_active":"2024-04-23T10:24:46.347868+0000","last_peered":"2024-04-23T10:24:46.347868+0000","last_clean":"2024-04-23T10:24:46.347868+0000","last_became_active":"2024-04-23T10:24:43.051793+0000","last_became_peered":"2024-04-23T10:24:43.051793+0000","last_unstale":"2024-04-23T10:24:46.347868+0000","last_undegraded":"2024-04-23T10:24:46.347868+0000","last_fullsized":"2024-04-23T10:24:46.347868+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T10:24:42.023127+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T10:24:42.023127+0000","last_clean_scrub_stamp":"2024-04-23T10:24:42.023127+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T20:57:59.015115+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,7],"acting":[5,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.13","version":"0'0","reported_seq":13,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-04-23T10:24:45.042533+0000","last_change":"2024-04-23T10:24:44.062689+0000","last_active":"2024-04-23T10:24:45.042533+0000","last_peered":"2024-04-23T10:24:45.042533+0000","last_clean":"2024-04-23T10:24:45.042533+0000","last_became_active":"2024-04-23T10:24:44.062331+0000","last_became_peered":"2024-04-23T10:24:44.062331+0000","last_unstale":"2024-04-23T10:24:45.042533+0000","last_undegraded":"2024-04-23T10:24:45.042533+0000","last_fullsized":"2024-04-23T10:24:45.042533+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T10:24:43.025324+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T10:24:43.025324+0000","last_clean_scrub_stamp":"2024-04-23T10:24:43.025324+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T14:06:04.989156+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,6],"acting":[7,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.11","version":"87'3","reported_seq":18,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-04-23T10:24:47.354217+0000","last_change":"2024-04-23T10:24:43.043901+0000","last_active":"2024-04-23T10:24:47.354217+0000","last_peered":"2024-04-23T10:24:47.354217+0000","last_clean":"2024-04-23T10:24:47.354217+0000","last_became_active":"2024-04-23T10:24:43.043598+0000","last_became_peered":"2024-04-23T10:24:43.043598+0000","last_unstale":"2024-04-23T10:24:47.354217+0000","last_undegraded":"2024-04-23T10:24:47.354217+0000","last_fullsized":"2024-04-23T10:24:47.354217+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T10:24:42.023127+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T10:24:42.023127+0000","last_clean_scrub_stamp":"2024-04-23T10:24:42.023127+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T14:00:06.338084+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":588,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,10],"acting":[4,7,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.10","version":"0'0","reported_seq":13,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-04-23T10:24:45.042074+0000","last_change":"2024-04-23T10:24:44.058773+0000","last_active":"2024-04-23T10:24:45.042074+0000","last_peered":"2024-04-23T10:24:45.042074+0000","last_clean":"2024-04-23T10:24:45.042074+0000","last_became_active":"2024-04-23T10:24:44.058420+0000","last_became_peered":"2024-04-23T10:24:44.058420+0000","last_unstale":"2024-04-23T10:24:45.042074+0000","last_undegraded":"2024-04-23T10:24:45.042074+0000","last_fullsized":"2024-04-23T10:24:45.042074+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T10:24:43.025324+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T10:24:43.025324+0000","last_clean_scrub_stamp":"2024-04-23T10:24:43.025324+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T12:44:30.023047+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,10],"acting":[7,5,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.10","version":"87'15","reported_seq":30,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-04-23T10:24:47.355199+0000","last_change":"2024-04-23T10:24:43.044700+0000","last_active":"2024-04-23T10:24:47.355199+0000","last_peered":"2024-04-23T10:24:47.355199+0000","last_clean":"2024-04-23T10:24:47.355199+0000","last_became_active":"2024-04-23T10:24:43.044394+0000","last_became_peered":"2024-04-23T10:24:43.044394+0000","last_unstale":"2024-04-23T10:24:47.355199+0000","last_undegraded":"2024-04-23T10:24:47.355199+0000","last_fullsized":"2024-04-23T10:24:47.355199+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T10:24:42.023127+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T10:24:42.023127+0000","last_clean_scrub_stamp":"2024-04-23T10:24:42.023127+0000","objects_scrubbed":0,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T15:16:41.367517+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":112,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":20,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":6,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,0,2],"acting":[8,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.11","version":"0'0","reported_seq":13,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-04-23T10:24:46.310005+0000","last_change":"2024-04-23T10:24:44.049739+0000","last_active":"2024-04-23T10:24:46.310005+0000","last_peered":"2024-04-23T10:24:46.310005+0000","last_clean":"2024-04-23T10:24:46.310005+0000","last_became_active":"2024-04-23T10:24:44.049414+0000","last_became_peered":"2024-04-23T10:24:44.049414+0000","last_unstale":"2024-04-23T10:24:46.310005+0000","last_undegraded":"2024-04-23T10:24:46.310005+0000","last_fullsized":"2024-04-23T10:24:46.310005+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T10:24:43.025324+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T10:24:43.025324+0000","last_clean_scrub_stamp":"2024-04-23T10:24:43.025324+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T12:51:02.645259+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,1],"acting":[5,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.f","version":"86'1","reported_seq":16,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-04-23T10:24:45.044492+0000","last_change":"2024-04-23T10:24:43.048884+0000","last_active":"2024-04-23T10:24:45.044492+0000","last_peered":"2024-04-23T10:24:45.044492+0000","last_clean":"2024-04-23T10:24:45.044492+0000","last_became_active":"2024-04-23T10:24:43.048583+0000","last_became_peered":"2024-04-23T10:24:43.048583+0000","last_unstale":"2024-04-23T10:24:45.044492+0000","last_undegraded":"2024-04-23T10:24:45.044492+0000","last_fullsized":"2024-04-23T10:24:45.044492+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T10:24:42.023127+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T10:24:42.023127+0000","last_clean_scrub_stamp":"2024-04-23T10:24:42.023127+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T17:55:57.869700+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,5,4],"acting":[8,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.e","version":"0'0","reported_seq":10,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-04-23T10:24:44.053368+0000","last_change":"2024-04-23T10:24:44.052817+0000","last_active":"2024-04-23T10:24:44.053368+0000","last_peered":"2024-04-23T10:24:44.053368+0000","last_clean":"2024-04-23T10:24:44.053368+0000","last_became_active":"2024-04-23T10:24:44.052323+0000","last_became_peered":"2024-04-23T10:24:44.052323+0000","last_unstale":"2024-04-23T10:24:44.053368+0000","last_undegraded":"2024-04-23T10:24:44.053368+0000","last_fullsized":"2024-04-23T10:24:44.053368+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T10:24:43.025324+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T10:24:43.025324+0000","last_clean_scrub_stamp":"2024-04-23T10:24:43.025324+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T16:13:09.933338+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,5,4],"acting":[11,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":15,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-04-23T10:24:45.608893+0000","last_change":"2024-04-23T10:24:43.044375+0000","last_active":"2024-04-23T10:24:45.608893+0000","last_peered":"2024-04-23T10:24:45.608893+0000","last_clean":"2024-04-23T10:24:45.608893+0000","last_became_active":"2024-04-23T10:24:43.043914+0000","last_became_peered":"2024-04-23T10:24:43.043914+0000","last_unstale":"2024-04-23T10:24:45.608893+0000","last_undegraded":"2024-04-23T10:24:45.608893+0000","last_fullsized":"2024-04-23T10:24:45.608893+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T10:24:42.023127+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T10:24:42.023127+0000","last_clean_scrub_stamp":"2024-04-23T10:24:42.023127+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T16:22:44.165156+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,2,0],"acting":[9,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.5","version":"0'0","reported_seq":13,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-04-23T10:24:45.235612+0000","last_change":"2024-04-23T10:24:44.057272+0000","last_active":"2024-04-23T10:24:45.235612+0000","last_peered":"2024-04-23T10:24:45.235612+0000","last_clean":"2024-04-23T10:24:45.235612+0000","last_became_active":"2024-04-23T10:24:44.056968+0000","last_became_peered":"2024-04-23T10:24:44.056968+0000","last_unstale":"2024-04-23T10:24:45.235612+0000","last_undegraded":"2024-04-23T10:24:45.235612+0000","last_fullsized":"2024-04-23T10:24:45.235612+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T10:24:43.025324+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T10:24:43.025324+0000","last_clean_scrub_stamp":"2024-04-23T10:24:43.025324+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T11:10:09.021845+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,3,6],"acting":[10,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"2.2","version":"87'1","reported_seq":16,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-04-23T10:24:45.042157+0000","last_change":"2024-04-23T10:24:43.051822+0000","last_active":"2024-04-23T10:24:45.042157+0000","last_peered":"2024-04-23T10:24:45.042157+0000","last_clean":"2024-04-23T10:24:45.042157+0000","last_became_active":"2024-04-23T10:24:43.051491+0000","last_became_peered":"2024-04-23T10:24:43.051491+0000","last_unstale":"2024-04-23T10:24:45.042157+0000","last_undegraded":"2024-04-23T10:24:45.042157+0000","last_fullsized":"2024-04-23T10:24:45.042157+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T10:24:42.023127+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T10:24:42.023127+0000","last_clean_scrub_stamp":"2024-04-23T10:24:42.023127+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T13:49:23.109064+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,1],"acting":[7,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.3","version":"0'0","reported_seq":13,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-04-23T10:24:45.235454+0000","last_change":"2024-04-23T10:24:44.053604+0000","last_active":"2024-04-23T10:24:45.235454+0000","last_peered":"2024-04-23T10:24:45.235454+0000","last_clean":"2024-04-23T10:24:45.235454+0000","last_became_active":"2024-04-23T10:24:44.053147+0000","last_became_peered":"2024-04-23T10:24:44.053147+0000","last_unstale":"2024-04-23T10:24:45.235454+0000","last_undegraded":"2024-04-23T10:24:45.235454+0000","last_fullsized":"2024-04-23T10:24:45.235454+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T10:24:43.025324+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T10:24:43.025324+0000","last_clean_scrub_stamp":"2024-04-23T10:24:43.025324+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T21:58:38.981282+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,4,0],"acting":[10,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.0","version":"0'0","reported_seq":13,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-04-23T10:24:46.310228+0000","last_change":"2024-04-23T10:24:44.052564+0000","last_active":"2024-04-23T10:24:46.310228+0000","last_peered":"2024-04-23T10:24:46.310228+0000","last_clean":"2024-04-23T10:24:46.310228+0000","last_became_active":"2024-04-23T10:24:44.052201+0000","last_became_peered":"2024-04-23T10:24:44.052201+0000","last_unstale":"2024-04-23T10:24:46.310228+0000","last_undegraded":"2024-04-23T10:24:46.310228+0000","last_fullsized":"2024-04-23T10:24:46.310228+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T10:24:43.025324+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T10:24:43.025324+0000","last_clean_scrub_stamp":"2024-04-23T10:24:43.025324+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T16:25:54.725347+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,4],"acting":[5,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.1","version":"87'4","reported_seq":19,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-04-23T10:24:47.356133+0000","last_change":"2024-04-23T10:24:43.049067+0000","last_active":"2024-04-23T10:24:47.356133+0000","last_peered":"2024-04-23T10:24:47.356133+0000","last_clean":"2024-04-23T10:24:47.356133+0000","last_became_active":"2024-04-23T10:24:43.048400+0000","last_became_peered":"2024-04-23T10:24:43.048400+0000","last_unstale":"2024-04-23T10:24:47.356133+0000","last_undegraded":"2024-04-23T10:24:47.356133+0000","last_fullsized":"2024-04-23T10:24:47.356133+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T10:24:42.023127+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T10:24:42.023127+0000","last_clean_scrub_stamp":"2024-04-23T10:24:42.023127+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T12:17:02.439054+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,0],"acting":[6,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.2","version":"0'0","reported_seq":13,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-04-23T10:24:46.310403+0000","last_change":"2024-04-23T10:24:44.059114+0000","last_active":"2024-04-23T10:24:46.310403+0000","last_peered":"2024-04-23T10:24:46.310403+0000","last_clean":"2024-04-23T10:24:46.310403+0000","last_became_active":"2024-04-23T10:24:44.058618+0000","last_became_peered":"2024-04-23T10:24:44.058618+0000","last_unstale":"2024-04-23T10:24:46.310403+0000","last_undegraded":"2024-04-23T10:24:46.310403+0000","last_fullsized":"2024-04-23T10:24:46.310403+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T10:24:43.025324+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T10:24:43.025324+0000","last_clean_scrub_stamp":"2024-04-23T10:24:43.025324+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T20:18:09.707062+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,1],"acting":[3,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.3","version":"87'3","reported_seq":18,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-04-23T10:24:46.347609+0000","last_change":"2024-04-23T10:24:43.039725+0000","last_active":"2024-04-23T10:24:46.347609+0000","last_peered":"2024-04-23T10:24:46.347609+0000","last_clean":"2024-04-23T10:24:46.347609+0000","last_became_active":"2024-04-23T10:24:43.039187+0000","last_became_peered":"2024-04-23T10:24:43.039187+0000","last_unstale":"2024-04-23T10:24:46.347609+0000","last_undegraded":"2024-04-23T10:24:46.347609+0000","last_fullsized":"2024-04-23T10:24:46.347609+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T10:24:42.023127+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T10:24:42.023127+0000","last_clean_scrub_stamp":"2024-04-23T10:24:42.023127+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T17:20:50.844878+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,10,9],"acting":[4,10,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.0","version":"22'137","reported_seq":346,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-04-23T10:24:45.041954+0000","last_change":"2024-04-23T10:23:31.092860+0000","last_active":"2024-04-23T10:24:45.041954+0000","last_peered":"2024-04-23T10:24:45.041954+0000","last_clean":"2024-04-23T10:24:45.041954+0000","last_became_active":"2024-04-23T10:23:31.092357+0000","last_became_peered":"2024-04-23T10:23:31.092357+0000","last_unstale":"2024-04-23T10:24:45.041954+0000","last_undegraded":"2024-04-23T10:24:45.041954+0000","last_fullsized":"2024-04-23T10:24:45.041954+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T10:20:12.201233+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T10:20:12.201233+0000","last_clean_scrub_stamp":"2024-04-23T10:20:12.201233+0000","objects_scrubbed":0,"log_size":137,"log_dups_size":0,"ondisk_log_size":137,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T21:28:56.804053+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":136,"num_read_kb":119,"num_write":181,"num_write_kb":2938,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":1180736,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,10],"acting":[7,5,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1","version":"0'0","reported_seq":13,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-04-23T10:24:46.309600+0000","last_change":"2024-04-23T10:24:44.058393+0000","last_active":"2024-04-23T10:24:46.309600+0000","last_peered":"2024-04-23T10:24:46.309600+0000","last_clean":"2024-04-23T10:24:46.309600+0000","last_became_active":"2024-04-23T10:24:44.058075+0000","last_became_peered":"2024-04-23T10:24:44.058075+0000","last_unstale":"2024-04-23T10:24:46.309600+0000","last_undegraded":"2024-04-23T10:24:46.309600+0000","last_fullsized":"2024-04-23T10:24:46.309600+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T10:24:43.025324+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T10:24:43.025324+0000","last_clean_scrub_stamp":"2024-04-23T10:24:43.025324+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T10:32:05.693311+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,2],"acting":[6,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.0","version":"87'3","reported_seq":18,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-04-23T10:24:47.354743+0000","last_change":"2024-04-23T10:24:43.048678+0000","last_active":"2024-04-23T10:24:47.354743+0000","last_peered":"2024-04-23T10:24:47.354743+0000","last_clean":"2024-04-23T10:24:47.354743+0000","last_became_active":"2024-04-23T10:24:43.048341+0000","last_became_peered":"2024-04-23T10:24:43.048341+0000","last_unstale":"2024-04-23T10:24:47.354743+0000","last_undegraded":"2024-04-23T10:24:47.354743+0000","last_fullsized":"2024-04-23T10:24:47.354743+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T10:24:42.023127+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T10:24:42.023127+0000","last_clean_scrub_stamp":"2024-04-23T10:24:42.023127+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T12:24:33.705325+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,8],"acting":[7,5,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.5","version":"87'1","reported_seq":16,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-04-23T10:24:45.044415+0000","last_change":"2024-04-23T10:24:43.046105+0000","last_active":"2024-04-23T10:24:45.044415+0000","last_peered":"2024-04-23T10:24:45.044415+0000","last_clean":"2024-04-23T10:24:45.044415+0000","last_became_active":"2024-04-23T10:24:43.045804+0000","last_became_peered":"2024-04-23T10:24:43.045804+0000","last_unstale":"2024-04-23T10:24:45.044415+0000","last_undegraded":"2024-04-23T10:24:45.044415+0000","last_fullsized":"2024-04-23T10:24:45.044415+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T10:24:42.023127+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T10:24:42.023127+0000","last_clean_scrub_stamp":"2024-04-23T10:24:42.023127+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T22:23:35.927468+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,0,9],"acting":[8,0,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.4","version":"0'0","reported_seq":13,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-04-23T10:24:46.310103+0000","last_change":"2024-04-23T10:24:44.059731+0000","last_active":"2024-04-23T10:24:46.310103+0000","last_peered":"2024-04-23T10:24:46.310103+0000","last_clean":"2024-04-23T10:24:46.310103+0000","last_became_active":"2024-04-23T10:24:44.059352+0000","last_became_peered":"2024-04-23T10:24:44.059352+0000","last_unstale":"2024-04-23T10:24:46.310103+0000","last_undegraded":"2024-04-23T10:24:46.310103+0000","last_fullsized":"2024-04-23T10:24:46.310103+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T10:24:43.025324+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T10:24:43.025324+0000","last_clean_scrub_stamp":"2024-04-23T10:24:43.025324+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T16:36:08.985444+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2,9],"acting":[6,2,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.6","version":"87'1","reported_seq":16,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-04-23T10:24:46.323682+0000","last_change":"2024-04-23T10:24:43.044527+0000","last_active":"2024-04-23T10:24:46.323682+0000","last_peered":"2024-04-23T10:24:46.323682+0000","last_clean":"2024-04-23T10:24:46.323682+0000","last_became_active":"2024-04-23T10:24:43.044212+0000","last_became_peered":"2024-04-23T10:24:43.044212+0000","last_unstale":"2024-04-23T10:24:46.323682+0000","last_undegraded":"2024-04-23T10:24:46.323682+0000","last_fullsized":"2024-04-23T10:24:46.323682+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T10:24:42.023127+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T10:24:42.023127+0000","last_clean_scrub_stamp":"2024-04-23T10:24:42.023127+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T19:39:46.369132+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,8,3],"acting":[9,8,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.7","version":"0'0","reported_seq":13,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-04-23T10:24:46.309988+0000","last_change":"2024-04-23T10:24:44.055920+0000","last_active":"2024-04-23T10:24:46.309988+0000","last_peered":"2024-04-23T10:24:46.309988+0000","last_clean":"2024-04-23T10:24:46.309988+0000","last_became_active":"2024-04-23T10:24:44.055411+0000","last_became_peered":"2024-04-23T10:24:44.055411+0000","last_unstale":"2024-04-23T10:24:46.309988+0000","last_undegraded":"2024-04-23T10:24:46.309988+0000","last_fullsized":"2024-04-23T10:24:46.309988+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T10:24:43.025324+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T10:24:43.025324+0000","last_clean_scrub_stamp":"2024-04-23T10:24:43.025324+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T14:43:10.446790+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,11,7],"acting":[3,11,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.7","version":"87'4","reported_seq":19,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-04-23T10:24:46.345036+0000","last_change":"2024-04-23T10:24:43.047989+0000","last_active":"2024-04-23T10:24:46.345036+0000","last_peered":"2024-04-23T10:24:46.345036+0000","last_clean":"2024-04-23T10:24:46.345036+0000","last_became_active":"2024-04-23T10:24:43.047625+0000","last_became_peered":"2024-04-23T10:24:43.047625+0000","last_unstale":"2024-04-23T10:24:46.345036+0000","last_undegraded":"2024-04-23T10:24:46.345036+0000","last_fullsized":"2024-04-23T10:24:46.345036+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T10:24:42.023127+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T10:24:42.023127+0000","last_clean_scrub_stamp":"2024-04-23T10:24:42.023127+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T13:50:16.647176+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,11,6],"acting":[4,11,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.6","version":"0'0","reported_seq":13,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-04-23T10:24:45.042053+0000","last_change":"2024-04-23T10:24:44.049214+0000","last_active":"2024-04-23T10:24:45.042053+0000","last_peered":"2024-04-23T10:24:45.042053+0000","last_clean":"2024-04-23T10:24:45.042053+0000","last_became_active":"2024-04-23T10:24:44.048869+0000","last_became_peered":"2024-04-23T10:24:44.048869+0000","last_unstale":"2024-04-23T10:24:45.042053+0000","last_undegraded":"2024-04-23T10:24:45.042053+0000","last_fullsized":"2024-04-23T10:24:45.042053+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T10:24:43.025324+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T10:24:43.025324+0000","last_clean_scrub_stamp":"2024-04-23T10:24:43.025324+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T18:17:09.772428+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,4],"acting":[0,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.8","version":"0'0","reported_seq":13,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-04-23T10:24:44.047149+0000","last_change":"2024-04-23T10:24:43.046189+0000","last_active":"2024-04-23T10:24:44.047149+0000","last_peered":"2024-04-23T10:24:44.047149+0000","last_clean":"2024-04-23T10:24:44.047149+0000","last_became_active":"2024-04-23T10:24:43.045848+0000","last_became_peered":"2024-04-23T10:24:43.045848+0000","last_unstale":"2024-04-23T10:24:44.047149+0000","last_undegraded":"2024-04-23T10:24:44.047149+0000","last_fullsized":"2024-04-23T10:24:44.047149+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T10:24:42.023127+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T10:24:42.023127+0000","last_clean_scrub_stamp":"2024-04-23T10:24:42.023127+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T22:23:39.180377+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,0],"acting":[1,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.9","version":"0'0","reported_seq":13,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-04-23T10:24:45.044247+0000","last_change":"2024-04-23T10:24:44.048415+0000","last_active":"2024-04-23T10:24:45.044247+0000","last_peered":"2024-04-23T10:24:45.044247+0000","last_clean":"2024-04-23T10:24:45.044247+0000","last_became_active":"2024-04-23T10:24:44.048030+0000","last_became_peered":"2024-04-23T10:24:44.048030+0000","last_unstale":"2024-04-23T10:24:45.044247+0000","last_undegraded":"2024-04-23T10:24:45.044247+0000","last_fullsized":"2024-04-23T10:24:45.044247+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T10:24:43.025324+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T10:24:43.025324+0000","last_clean_scrub_stamp":"2024-04-23T10:24:43.025324+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T14:42:09.920927+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,11,10],"acting":[8,11,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"2.9","version":"87'3","reported_seq":18,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-04-23T10:24:46.346630+0000","last_change":"2024-04-23T10:24:43.041525+0000","last_active":"2024-04-23T10:24:46.346630+0000","last_peered":"2024-04-23T10:24:46.346630+0000","last_clean":"2024-04-23T10:24:46.346630+0000","last_became_active":"2024-04-23T10:24:43.041200+0000","last_became_peered":"2024-04-23T10:24:43.041200+0000","last_unstale":"2024-04-23T10:24:46.346630+0000","last_undegraded":"2024-04-23T10:24:46.346630+0000","last_fullsized":"2024-04-23T10:24:46.346630+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T10:24:42.023127+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T10:24:42.023127+0000","last_clean_scrub_stamp":"2024-04-23T10:24:42.023127+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T16:45:06.846991+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,9,10],"acting":[3,9,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.8","version":"0'0","reported_seq":10,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-04-23T10:24:44.055845+0000","last_change":"2024-04-23T10:24:44.055341+0000","last_active":"2024-04-23T10:24:44.055845+0000","last_peered":"2024-04-23T10:24:44.055845+0000","last_clean":"2024-04-23T10:24:44.055845+0000","last_became_active":"2024-04-23T10:24:44.054216+0000","last_became_peered":"2024-04-23T10:24:44.054216+0000","last_unstale":"2024-04-23T10:24:44.055845+0000","last_undegraded":"2024-04-23T10:24:44.055845+0000","last_fullsized":"2024-04-23T10:24:44.055845+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T10:24:43.025324+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T10:24:43.025324+0000","last_clean_scrub_stamp":"2024-04-23T10:24:43.025324+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T10:32:02.420748+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,0,2],"acting":[11,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"2.a","version":"0'0","reported_seq":15,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-04-23T10:24:45.123774+0000","last_change":"2024-04-23T10:24:43.045400+0000","last_active":"2024-04-23T10:24:45.123774+0000","last_peered":"2024-04-23T10:24:45.123774+0000","last_clean":"2024-04-23T10:24:45.123774+0000","last_became_active":"2024-04-23T10:24:43.044677+0000","last_became_peered":"2024-04-23T10:24:43.044677+0000","last_unstale":"2024-04-23T10:24:45.123774+0000","last_undegraded":"2024-04-23T10:24:45.123774+0000","last_fullsized":"2024-04-23T10:24:45.123774+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T10:24:42.023127+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T10:24:42.023127+0000","last_clean_scrub_stamp":"2024-04-23T10:24:42.023127+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T19:06:19.700132+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,8],"acting":[2,3,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.b","version":"0'0","reported_seq":13,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-04-23T10:24:46.310072+0000","last_change":"2024-04-23T10:24:44.049968+0000","last_active":"2024-04-23T10:24:46.310072+0000","last_peered":"2024-04-23T10:24:46.310072+0000","last_clean":"2024-04-23T10:24:46.310072+0000","last_became_active":"2024-04-23T10:24:44.049621+0000","last_became_peered":"2024-04-23T10:24:44.049621+0000","last_unstale":"2024-04-23T10:24:46.310072+0000","last_undegraded":"2024-04-23T10:24:46.310072+0000","last_fullsized":"2024-04-23T10:24:46.310072+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T10:24:43.025324+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T10:24:43.025324+0000","last_clean_scrub_stamp":"2024-04-23T10:24:43.025324+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T16:23:43.082831+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,10],"acting":[3,0,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.b","version":"87'4","reported_seq":19,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-04-23T10:24:46.344828+0000","last_change":"2024-04-23T10:24:43.041828+0000","last_active":"2024-04-23T10:24:46.344828+0000","last_peered":"2024-04-23T10:24:46.344828+0000","last_clean":"2024-04-23T10:24:46.344828+0000","last_became_active":"2024-04-23T10:24:43.041480+0000","last_became_peered":"2024-04-23T10:24:43.041480+0000","last_unstale":"2024-04-23T10:24:46.344828+0000","last_undegraded":"2024-04-23T10:24:46.344828+0000","last_fullsized":"2024-04-23T10:24:46.344828+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T10:24:42.023127+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T10:24:42.023127+0000","last_clean_scrub_stamp":"2024-04-23T10:24:42.023127+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T14:43:03.203971+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,2,7],"acting":[8,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.a","version":"0'0","reported_seq":13,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-04-23T10:24:45.042560+0000","last_change":"2024-04-23T10:24:44.062103+0000","last_active":"2024-04-23T10:24:45.042560+0000","last_peered":"2024-04-23T10:24:45.042560+0000","last_clean":"2024-04-23T10:24:45.042560+0000","last_became_active":"2024-04-23T10:24:44.061794+0000","last_became_peered":"2024-04-23T10:24:44.061794+0000","last_unstale":"2024-04-23T10:24:45.042560+0000","last_undegraded":"2024-04-23T10:24:45.042560+0000","last_fullsized":"2024-04-23T10:24:45.042560+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T10:24:43.025324+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T10:24:43.025324+0000","last_clean_scrub_stamp":"2024-04-23T10:24:43.025324+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T10:45:30.130047+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,6],"acting":[7,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.c","version":"0'0","reported_seq":15,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-04-23T10:24:45.124150+0000","last_change":"2024-04-23T10:24:43.045702+0000","last_active":"2024-04-23T10:24:45.124150+0000","last_peered":"2024-04-23T10:24:45.124150+0000","last_clean":"2024-04-23T10:24:45.124150+0000","last_became_active":"2024-04-23T10:24:43.045258+0000","last_became_peered":"2024-04-23T10:24:43.045258+0000","last_unstale":"2024-04-23T10:24:45.124150+0000","last_undegraded":"2024-04-23T10:24:45.124150+0000","last_fullsized":"2024-04-23T10:24:45.124150+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T10:24:42.023127+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T10:24:42.023127+0000","last_clean_scrub_stamp":"2024-04-23T10:24:42.023127+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T12:54:30.965080+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,11,0],"acting":[2,11,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.d","version":"0'0","reported_seq":13,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-04-23T10:24:45.042632+0000","last_change":"2024-04-23T10:24:44.061118+0000","last_active":"2024-04-23T10:24:45.042632+0000","last_peered":"2024-04-23T10:24:45.042632+0000","last_clean":"2024-04-23T10:24:45.042632+0000","last_became_active":"2024-04-23T10:24:44.060743+0000","last_became_peered":"2024-04-23T10:24:44.060743+0000","last_unstale":"2024-04-23T10:24:45.042632+0000","last_undegraded":"2024-04-23T10:24:45.042632+0000","last_fullsized":"2024-04-23T10:24:45.042632+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T10:24:43.025324+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T10:24:43.025324+0000","last_clean_scrub_stamp":"2024-04-23T10:24:43.025324+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T17:17:46.731121+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,11],"acting":[7,1,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.d","version":"86'1","reported_seq":17,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-04-23T10:24:46.310477+0000","last_change":"2024-04-23T10:24:43.052519+0000","last_active":"2024-04-23T10:24:46.310477+0000","last_peered":"2024-04-23T10:24:46.310477+0000","last_clean":"2024-04-23T10:24:46.310477+0000","last_became_active":"2024-04-23T10:24:43.052187+0000","last_became_peered":"2024-04-23T10:24:43.052187+0000","last_unstale":"2024-04-23T10:24:46.310477+0000","last_undegraded":"2024-04-23T10:24:46.310477+0000","last_fullsized":"2024-04-23T10:24:46.310477+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T10:24:42.023127+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T10:24:42.023127+0000","last_clean_scrub_stamp":"2024-04-23T10:24:42.023127+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T15:51:52.839409+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,10,3],"acting":[5,10,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.c","version":"0'0","reported_seq":13,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-04-23T10:24:45.042342+0000","last_change":"2024-04-23T10:24:44.053086+0000","last_active":"2024-04-23T10:24:45.042342+0000","last_peered":"2024-04-23T10:24:45.042342+0000","last_clean":"2024-04-23T10:24:45.042342+0000","last_became_active":"2024-04-23T10:24:44.052687+0000","last_became_peered":"2024-04-23T10:24:44.052687+0000","last_unstale":"2024-04-23T10:24:45.042342+0000","last_undegraded":"2024-04-23T10:24:45.042342+0000","last_fullsized":"2024-04-23T10:24:45.042342+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T10:24:43.025324+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T10:24:43.025324+0000","last_clean_scrub_stamp":"2024-04-23T10:24:43.025324+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T14:35:44.958742+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,5],"acting":[7,3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.e","version":"86'2","reported_seq":17,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-04-23T10:24:46.309746+0000","last_change":"2024-04-23T10:24:43.047716+0000","last_active":"2024-04-23T10:24:46.309746+0000","last_peered":"2024-04-23T10:24:46.309746+0000","last_clean":"2024-04-23T10:24:46.309746+0000","last_became_active":"2024-04-23T10:24:43.047357+0000","last_became_peered":"2024-04-23T10:24:43.047357+0000","last_unstale":"2024-04-23T10:24:46.309746+0000","last_undegraded":"2024-04-23T10:24:46.309746+0000","last_fullsized":"2024-04-23T10:24:46.309746+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T10:24:42.023127+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T10:24:42.023127+0000","last_clean_scrub_stamp":"2024-04-23T10:24:42.023127+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T14:25:11.593207+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,2],"acting":[6,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.f","version":"0'0","reported_seq":13,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-04-23T10:24:45.042610+0000","last_change":"2024-04-23T10:24:44.049543+0000","last_active":"2024-04-23T10:24:45.042610+0000","last_peered":"2024-04-23T10:24:45.042610+0000","last_clean":"2024-04-23T10:24:45.042610+0000","last_became_active":"2024-04-23T10:24:44.049198+0000","last_became_peered":"2024-04-23T10:24:44.049198+0000","last_unstale":"2024-04-23T10:24:45.042610+0000","last_undegraded":"2024-04-23T10:24:45.042610+0000","last_fullsized":"2024-04-23T10:24:45.042610+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T10:24:43.025324+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T10:24:43.025324+0000","last_clean_scrub_stamp":"2024-04-23T10:24:43.025324+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T12:25:15.974267+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,11,8],"acting":[7,11,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.20","version":"0'0","reported_seq":10,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-04-23T10:24:44.055115+0000","last_change":"2024-04-23T10:24:44.054440+0000","last_active":"2024-04-23T10:24:44.055115+0000","last_peered":"2024-04-23T10:24:44.055115+0000","last_clean":"2024-04-23T10:24:44.055115+0000","last_became_active":"2024-04-23T10:24:44.053951+0000","last_became_peered":"2024-04-23T10:24:44.053951+0000","last_unstale":"2024-04-23T10:24:44.055115+0000","last_undegraded":"2024-04-23T10:24:44.055115+0000","last_fullsized":"2024-04-23T10:24:44.055115+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T10:24:43.025324+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T10:24:43.025324+0000","last_clean_scrub_stamp":"2024-04-23T10:24:43.025324+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T20:26:51.035255+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,2,5],"acting":[11,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.21","version":"0'0","reported_seq":13,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-04-23T10:24:45.123991+0000","last_change":"2024-04-23T10:24:44.059015+0000","last_active":"2024-04-23T10:24:45.123991+0000","last_peered":"2024-04-23T10:24:45.123991+0000","last_clean":"2024-04-23T10:24:45.123991+0000","last_became_active":"2024-04-23T10:24:44.058691+0000","last_became_peered":"2024-04-23T10:24:44.058691+0000","last_unstale":"2024-04-23T10:24:45.123991+0000","last_undegraded":"2024-04-23T10:24:45.123991+0000","last_fullsized":"2024-04-23T10:24:45.123991+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T10:24:43.025324+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T10:24:43.025324+0000","last_clean_scrub_stamp":"2024-04-23T10:24:43.025324+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T14:57:48.153094+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,4],"acting":[2,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.22","version":"0'0","reported_seq":13,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-04-23T10:24:45.123994+0000","last_change":"2024-04-23T10:24:44.055534+0000","last_active":"2024-04-23T10:24:45.123994+0000","last_peered":"2024-04-23T10:24:45.123994+0000","last_clean":"2024-04-23T10:24:45.123994+0000","last_became_active":"2024-04-23T10:24:44.055181+0000","last_became_peered":"2024-04-23T10:24:44.055181+0000","last_unstale":"2024-04-23T10:24:45.123994+0000","last_undegraded":"2024-04-23T10:24:45.123994+0000","last_fullsized":"2024-04-23T10:24:45.123994+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T10:24:43.025324+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T10:24:43.025324+0000","last_clean_scrub_stamp":"2024-04-23T10:24:43.025324+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T13:56:33.878232+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.23","version":"0'0","reported_seq":13,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-04-23T10:24:45.041988+0000","last_change":"2024-04-23T10:24:44.053349+0000","last_active":"2024-04-23T10:24:45.041988+0000","last_peered":"2024-04-23T10:24:45.041988+0000","last_clean":"2024-04-23T10:24:45.041988+0000","last_became_active":"2024-04-23T10:24:44.053016+0000","last_became_peered":"2024-04-23T10:24:44.053016+0000","last_unstale":"2024-04-23T10:24:45.041988+0000","last_undegraded":"2024-04-23T10:24:45.041988+0000","last_fullsized":"2024-04-23T10:24:45.041988+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T10:24:43.025324+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T10:24:43.025324+0000","last_clean_scrub_stamp":"2024-04-23T10:24:43.025324+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T14:24:36.945889+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,4],"acting":[0,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.24","version":"0'0","reported_seq":13,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-04-23T10:24:46.309748+0000","last_change":"2024-04-23T10:24:44.050741+0000","last_active":"2024-04-23T10:24:46.309748+0000","last_peered":"2024-04-23T10:24:46.309748+0000","last_clean":"2024-04-23T10:24:46.309748+0000","last_became_active":"2024-04-23T10:24:44.050398+0000","last_became_peered":"2024-04-23T10:24:44.050398+0000","last_unstale":"2024-04-23T10:24:46.309748+0000","last_undegraded":"2024-04-23T10:24:46.309748+0000","last_fullsized":"2024-04-23T10:24:46.309748+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T10:24:43.025324+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T10:24:43.025324+0000","last_clean_scrub_stamp":"2024-04-23T10:24:43.025324+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T10:49:07.146673+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,0],"acting":[6,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.25","version":"0'0","reported_seq":10,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-04-23T10:24:44.049538+0000","last_change":"2024-04-23T10:24:44.048933+0000","last_active":"2024-04-23T10:24:44.049538+0000","last_peered":"2024-04-23T10:24:44.049538+0000","last_clean":"2024-04-23T10:24:44.049538+0000","last_became_active":"2024-04-23T10:24:44.048553+0000","last_became_peered":"2024-04-23T10:24:44.048553+0000","last_unstale":"2024-04-23T10:24:44.049538+0000","last_undegraded":"2024-04-23T10:24:44.049538+0000","last_fullsized":"2024-04-23T10:24:44.049538+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T10:24:43.025324+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T10:24:43.025324+0000","last_clean_scrub_stamp":"2024-04-23T10:24:43.025324+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T22:19:54.013031+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,8,5],"acting":[11,8,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.26","version":"0'0","reported_seq":13,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-04-23T10:24:45.124258+0000","last_change":"2024-04-23T10:24:44.051895+0000","last_active":"2024-04-23T10:24:45.124258+0000","last_peered":"2024-04-23T10:24:45.124258+0000","last_clean":"2024-04-23T10:24:45.124258+0000","last_became_active":"2024-04-23T10:24:44.051005+0000","last_became_peered":"2024-04-23T10:24:44.051005+0000","last_unstale":"2024-04-23T10:24:45.124258+0000","last_undegraded":"2024-04-23T10:24:45.124258+0000","last_fullsized":"2024-04-23T10:24:45.124258+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T10:24:43.025324+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T10:24:43.025324+0000","last_clean_scrub_stamp":"2024-04-23T10:24:43.025324+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T18:13:05.227365+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,8,1],"acting":[2,8,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.27","version":"0'0","reported_seq":13,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-04-23T10:24:46.309796+0000","last_change":"2024-04-23T10:24:44.051863+0000","last_active":"2024-04-23T10:24:46.309796+0000","last_peered":"2024-04-23T10:24:46.309796+0000","last_clean":"2024-04-23T10:24:46.309796+0000","last_became_active":"2024-04-23T10:24:44.051208+0000","last_became_peered":"2024-04-23T10:24:44.051208+0000","last_unstale":"2024-04-23T10:24:46.309796+0000","last_undegraded":"2024-04-23T10:24:46.309796+0000","last_fullsized":"2024-04-23T10:24:46.309796+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T10:24:43.025324+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T10:24:43.025324+0000","last_clean_scrub_stamp":"2024-04-23T10:24:43.025324+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T13:35:07.072598+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,2],"acting":[3,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.28","version":"0'0","reported_seq":13,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-04-23T10:24:46.309909+0000","last_change":"2024-04-23T10:24:44.044158+0000","last_active":"2024-04-23T10:24:46.309909+0000","last_peered":"2024-04-23T10:24:46.309909+0000","last_clean":"2024-04-23T10:24:46.309909+0000","last_became_active":"2024-04-23T10:24:44.043829+0000","last_became_peered":"2024-04-23T10:24:44.043829+0000","last_unstale":"2024-04-23T10:24:46.309909+0000","last_undegraded":"2024-04-23T10:24:46.309909+0000","last_fullsized":"2024-04-23T10:24:46.309909+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T10:24:43.025324+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T10:24:43.025324+0000","last_clean_scrub_stamp":"2024-04-23T10:24:43.025324+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T15:10:08.081831+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,9,3],"acting":[5,9,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.29","version":"0'0","reported_seq":13,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-04-23T10:24:46.310279+0000","last_change":"2024-04-23T10:24:44.051267+0000","last_active":"2024-04-23T10:24:46.310279+0000","last_peered":"2024-04-23T10:24:46.310279+0000","last_clean":"2024-04-23T10:24:46.310279+0000","last_became_active":"2024-04-23T10:24:44.050911+0000","last_became_peered":"2024-04-23T10:24:44.050911+0000","last_unstale":"2024-04-23T10:24:46.310279+0000","last_undegraded":"2024-04-23T10:24:46.310279+0000","last_fullsized":"2024-04-23T10:24:46.310279+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T10:24:43.025324+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T10:24:43.025324+0000","last_clean_scrub_stamp":"2024-04-23T10:24:43.025324+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T18:09:28.871427+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,11],"acting":[3,2,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.2a","version":"0'0","reported_seq":10,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-04-23T10:24:44.058465+0000","last_change":"2024-04-23T10:24:44.057653+0000","last_active":"2024-04-23T10:24:44.058465+0000","last_peered":"2024-04-23T10:24:44.058465+0000","last_clean":"2024-04-23T10:24:44.058465+0000","last_became_active":"2024-04-23T10:24:44.056627+0000","last_became_peered":"2024-04-23T10:24:44.056627+0000","last_unstale":"2024-04-23T10:24:44.058465+0000","last_undegraded":"2024-04-23T10:24:44.058465+0000","last_fullsized":"2024-04-23T10:24:44.058465+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T10:24:43.025324+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T10:24:43.025324+0000","last_clean_scrub_stamp":"2024-04-23T10:24:43.025324+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T16:06:51.295540+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,3],"acting":[1,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.2b","version":"0'0","reported_seq":13,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-04-23T10:24:46.310216+0000","last_change":"2024-04-23T10:24:44.059735+0000","last_active":"2024-04-23T10:24:46.310216+0000","last_peered":"2024-04-23T10:24:46.310216+0000","last_clean":"2024-04-23T10:24:46.310216+0000","last_became_active":"2024-04-23T10:24:44.059052+0000","last_became_peered":"2024-04-23T10:24:44.059052+0000","last_unstale":"2024-04-23T10:24:46.310216+0000","last_undegraded":"2024-04-23T10:24:46.310216+0000","last_fullsized":"2024-04-23T10:24:46.310216+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T10:24:43.025324+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T10:24:43.025324+0000","last_clean_scrub_stamp":"2024-04-23T10:24:43.025324+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T14:29:25.794655+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,0],"acting":[6,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.2c","version":"0'0","reported_seq":10,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-04-23T10:24:44.053091+0000","last_change":"2024-04-23T10:24:44.051831+0000","last_active":"2024-04-23T10:24:44.053091+0000","last_peered":"2024-04-23T10:24:44.053091+0000","last_clean":"2024-04-23T10:24:44.053091+0000","last_became_active":"2024-04-23T10:24:44.051480+0000","last_became_peered":"2024-04-23T10:24:44.051480+0000","last_unstale":"2024-04-23T10:24:44.053091+0000","last_undegraded":"2024-04-23T10:24:44.053091+0000","last_fullsized":"2024-04-23T10:24:44.053091+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T10:24:43.025324+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T10:24:43.025324+0000","last_clean_scrub_stamp":"2024-04-23T10:24:43.025324+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T18:53:58.599937+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,4,8],"acting":[11,4,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]}],"pool_stats":[{"poolid":3,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":117,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":192,"acting":192,"num_store_stats":12},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":136,"num_read_kb":119,"num_write":181,"num_write_kb":2938,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":1180736,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":4751360,"data_stored":4722944,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":137,"ondisk_log_size":137,"up":3,"acting":3,"num_store_stats":8},{"poolid":2,"num_pg":32,"stat_sum":{"num_bytes":5031,"num_objects":49,"num_object_clones":0,"num_object_copies":147,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":49,"num_whiteouts":0,"num_read":5,"num_read_kb":4,"num_write":99,"num_write_kb":38,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":30,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":192512,"data_stored":12719,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":90,"ondisk_log_size":90,"up":96,"acting":96,"num_store_stats":12}],"osd_stats":[{"osd":11,"up_from":82,"seq":352187318286,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32080,"kb_used_data":904,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93713072,"statfs":{"total":95995035648,"available":95962185728,"internally_reserved":0,"allocated":925696,"data_stored":439178,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1596,"internal_metadata":31914436},"hb_peers":[0,1,2,3,4,5,6,7,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":10,"up_from":74,"seq":317827579922,"num_pgs":26,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28548,"kb_used_data":1520,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93716604,"statfs":{"total":95995035648,"available":95965802496,"internally_reserved":0,"allocated":1556480,"data_stored":1039922,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1600,"internal_metadata":27654592},"hb_peers":[0,1,2,3,4,5,6,8,9,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":9,"up_from":69,"seq":296352743448,"num_pgs":17,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":27944,"kb_used_data":920,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717208,"statfs":{"total":95995035648,"available":95966420992,"internally_reserved":0,"allocated":942080,"data_stored":448330,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1600,"internal_metadata":27654592},"hb_peers":[0,1,2,3,4,5,6,7,8,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":8,"up_from":62,"seq":266287972382,"num_pgs":24,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32112,"kb_used_data":936,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93713040,"statfs":{"total":95995035648,"available":95962152960,"internally_reserved":0,"allocated":958464,"data_stored":448486,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,2,3,4,5,6,7,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":7,"up_from":53,"seq":227633266722,"num_pgs":25,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":36852,"kb_used_data":1508,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":93708300,"statfs":{"total":95995035648,"available":95957299200,"internally_reserved":0,"allocated":1544192,"data_stored":1039320,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":36174282},"hb_peers":[1,2,3,4,5,6,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":6,"up_from":47,"seq":201863462952,"num_pgs":20,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32100,"kb_used_data":920,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93713052,"statfs":{"total":95995035648,"available":95962165248,"internally_reserved":0,"allocated":942080,"data_stored":447838,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":31914438},"hb_peers":[0,1,2,3,4,5,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":1,"up_from":13,"seq":55834574911,"num_pgs":24,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":27944,"kb_used_data":916,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717208,"statfs":{"total":95995035648,"available":95966420992,"internally_reserved":0,"allocated":937984,"data_stored":440790,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,2,3,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":0,"up_from":8,"seq":34359738437,"num_pgs":25,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32104,"kb_used_data":928,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93713048,"statfs":{"total":95995035648,"available":95962161152,"internally_reserved":0,"allocated":950272,"data_stored":448442,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":31914438},"hb_peers":[1,2,3,4,5,6,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":2,"up_from":18,"seq":77309411387,"num_pgs":28,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":36272,"kb_used_data":932,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":93708880,"statfs":{"total":95995035648,"available":95957893120,"internally_reserved":0,"allocated":954368,"data_stored":449394,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":36174277},"hb_peers":[0,1,3,4,5,6,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":3,"up_from":25,"seq":107374182452,"num_pgs":17,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":27952,"kb_used_data":932,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717200,"statfs":{"total":95995035648,"available":95966412800,"internally_reserved":0,"allocated":954368,"data_stored":448032,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[0,1,2,4,5,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":4,"up_from":33,"seq":141733920816,"num_pgs":26,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":27944,"kb_used_data":920,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717208,"statfs":{"total":95995035648,"available":95966420992,"internally_reserved":0,"allocated":942080,"data_stored":447770,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[1,2,3,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":5,"up_from":39,"seq":167503724589,"num_pgs":30,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":36864,"kb_used_data":1524,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":93708288,"statfs":{"total":95995035648,"available":95957286912,"internally_reserved":0,"allocated":1560576,"data_stored":1039592,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":36174277},"hb_peers":[0,1,2,3,4,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":694,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":1612,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":1646,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":284,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":644,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":28672,"data_stored":1867,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":90,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":1792,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":738,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":784,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":36864,"data_stored":2568,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":117,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-04-23T10:24:51.385 INFO:tasks.cephfs.filesystem.ceph_manager:clean! 2024-04-23T10:24:51.385 INFO:tasks.cephadm:Setting up client nodes... 2024-04-23T10:24:51.385 DEBUG:teuthology.orchestra.run.smithi114:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d69f13d827952bb07bb9b7973bca14f97d764d6f shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid aa240572-0159-11ef-bc93-c7b262605968 -- ceph auth get-or-create client.0 mon 'allow *' osd 'allow *' mds 'allow *' mgr 'allow *' 2024-04-23T10:24:51.656 INFO:teuthology.orchestra.run.smithi114.stderr:Inferring config /var/lib/ceph/aa240572-0159-11ef-bc93-c7b262605968/mon.a/config 2024-04-23T10:24:51.742 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 23 10:24:51 smithi119 ceph-mon[32608]: mgrmap e17: x(active, since 7m), standbys: z, y 2024-04-23T10:24:51.742 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 23 10:24:51 smithi119 ceph-mon[32608]: from='client.? 172.21.15.114:0/3392172770' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-23T10:24:51.742 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 23 10:24:51 smithi119 ceph-mon[32608]: from='client.? 172.21.15.114:0/541711716' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-23T10:24:51.742 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 23 10:24:51 smithi119 ceph-mon[32608]: from='client.14790 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-04-23T10:24:51.754 INFO:journalctl@ceph.mon.a.smithi114.stdout:Apr 23 10:24:51 smithi114 ceph-mon[29255]: mgrmap e17: x(active, since 7m), standbys: z, y 2024-04-23T10:24:51.754 INFO:journalctl@ceph.mon.a.smithi114.stdout:Apr 23 10:24:51 smithi114 ceph-mon[29255]: from='client.? 172.21.15.114:0/3392172770' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-23T10:24:51.754 INFO:journalctl@ceph.mon.a.smithi114.stdout:Apr 23 10:24:51 smithi114 ceph-mon[29255]: from='client.? 172.21.15.114:0/541711716' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-23T10:24:51.754 INFO:journalctl@ceph.mon.a.smithi114.stdout:Apr 23 10:24:51 smithi114 ceph-mon[29255]: from='client.14790 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-04-23T10:24:51.828 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 23 10:24:51 smithi204 ceph-mon[32649]: mgrmap e17: x(active, since 7m), standbys: z, y 2024-04-23T10:24:51.828 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 23 10:24:51 smithi204 ceph-mon[32649]: from='client.? 172.21.15.114:0/3392172770' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-23T10:24:51.828 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 23 10:24:51 smithi204 ceph-mon[32649]: from='client.? 172.21.15.114:0/541711716' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-23T10:24:51.828 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 23 10:24:51 smithi204 ceph-mon[32649]: from='client.14790 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-04-23T10:24:52.742 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 23 10:24:52 smithi119 ceph-mon[32608]: from='client.14796 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-04-23T10:24:52.742 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 23 10:24:52 smithi119 ceph-mon[32608]: pgmap v290: 97 pgs: 97 active+clean; 582 KiB data, 370 MiB used, 1.0 TiB / 1.0 TiB avail; 895 B/s rd, 5.1 KiB/s wr, 16 op/s 2024-04-23T10:24:52.754 INFO:journalctl@ceph.mon.a.smithi114.stdout:Apr 23 10:24:52 smithi114 ceph-mon[29255]: from='client.14796 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-04-23T10:24:52.754 INFO:journalctl@ceph.mon.a.smithi114.stdout:Apr 23 10:24:52 smithi114 ceph-mon[29255]: pgmap v290: 97 pgs: 97 active+clean; 582 KiB data, 370 MiB used, 1.0 TiB / 1.0 TiB avail; 895 B/s rd, 5.1 KiB/s wr, 16 op/s 2024-04-23T10:24:52.828 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 23 10:24:52 smithi204 ceph-mon[32649]: from='client.14796 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-04-23T10:24:52.828 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 23 10:24:52 smithi204 ceph-mon[32649]: pgmap v290: 97 pgs: 97 active+clean; 582 KiB data, 370 MiB used, 1.0 TiB / 1.0 TiB avail; 895 B/s rd, 5.1 KiB/s wr, 16 op/s 2024-04-23T10:24:53.349 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:24:53.347+0000 7fa10cc79700 1 -- 172.21.15.114:0/3361737240 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fa1081019c0 msgr2=0x7fa108101de0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T10:24:53.349 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:24:53.347+0000 7fa10cc79700 1 --2- 172.21.15.114:0/3361737240 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fa1081019c0 0x7fa108101de0 secure :-1 s=READY pgs=242 cs=0 l=1 rev1=1 crypto rx=0x7fa0f80099c0 tx=0x7fa0f8009cd0 comp rx=0 tx=0).stop 2024-04-23T10:24:53.349 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:24:53.348+0000 7fa10cc79700 1 -- 172.21.15.114:0/3361737240 shutdown_connections 2024-04-23T10:24:53.349 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:24:53.348+0000 7fa10cc79700 1 --2- 172.21.15.114:0/3361737240 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fa108103560 0x7fa108109e10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:24:53.349 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:24:53.348+0000 7fa10cc79700 1 --2- 172.21.15.114:0/3361737240 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fa1081019c0 0x7fa108101de0 unknown :-1 s=CLOSED pgs=242 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:24:53.349 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:24:53.348+0000 7fa10cc79700 1 --2- 172.21.15.114:0/3361737240 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fa108102bc0 0x7fa108103020 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:24:53.349 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:24:53.348+0000 7fa10cc79700 1 -- 172.21.15.114:0/3361737240 >> 172.21.15.114:0/3361737240 conn(0x7fa1080fcf60 msgr2=0x7fa1080ff3a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T10:24:53.349 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:24:53.348+0000 7fa10cc79700 1 -- 172.21.15.114:0/3361737240 shutdown_connections 2024-04-23T10:24:53.350 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:24:53.348+0000 7fa10cc79700 1 -- 172.21.15.114:0/3361737240 wait complete. 2024-04-23T10:24:53.350 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:24:53.348+0000 7fa10cc79700 1 Processor -- start 2024-04-23T10:24:53.350 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:24:53.348+0000 7fa10cc79700 1 -- start start 2024-04-23T10:24:53.350 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:24:53.348+0000 7fa10cc79700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fa1081019c0 0x7fa10806b280 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T10:24:53.350 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:24:53.348+0000 7fa10cc79700 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fa108102bc0 0x7fa10819d890 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T10:24:53.350 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:24:53.349+0000 7fa10cc79700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fa108103560 0x7fa10819fed0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T10:24:53.350 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:24:53.349+0000 7fa10cc79700 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x7fa10810d4b0 con 0x7fa1081019c0 2024-04-23T10:24:53.350 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:24:53.349+0000 7fa10cc79700 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 -- 0x7fa10810d330 con 0x7fa108103560 2024-04-23T10:24:53.350 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:24:53.349+0000 7fa10cc79700 1 -- --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_getmap magic: 0 -- 0x7fa10810d630 con 0x7fa108102bc0 2024-04-23T10:24:53.350 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:24:53.349+0000 7fa10659c700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fa1081019c0 0x7fa10806b280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T10:24:53.350 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:24:53.349+0000 7fa10659c700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fa1081019c0 0x7fa10806b280 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.114:43516/0 (socket says 172.21.15.114:43516) 2024-04-23T10:24:53.350 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:24:53.349+0000 7fa10659c700 1 -- 172.21.15.114:0/3569351440 learned_addr learned my addr 172.21.15.114:0/3569351440 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-04-23T10:24:53.351 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:24:53.349+0000 7fa106d9d700 1 --2- 172.21.15.114:0/3569351440 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fa108103560 0x7fa10819fed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T10:24:53.351 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:24:53.349+0000 7fa105d9b700 1 --2- 172.21.15.114:0/3569351440 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fa108102bc0 0x7fa10819d890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T10:24:53.351 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:24:53.349+0000 7fa10659c700 1 -- 172.21.15.114:0/3569351440 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fa108102bc0 msgr2=0x7fa10819d890 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T10:24:53.351 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:24:53.349+0000 7fa10659c700 1 --2- 172.21.15.114:0/3569351440 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fa108102bc0 0x7fa10819d890 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:24:53.351 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:24:53.349+0000 7fa10659c700 1 -- 172.21.15.114:0/3569351440 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fa108103560 msgr2=0x7fa10819fed0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T10:24:53.351 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:24:53.349+0000 7fa10659c700 1 --2- 172.21.15.114:0/3569351440 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fa108103560 0x7fa10819fed0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:24:53.351 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:24:53.349+0000 7fa10659c700 1 -- 172.21.15.114:0/3569351440 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa1081a07a0 con 0x7fa1081019c0 2024-04-23T10:24:53.351 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:24:53.349+0000 7fa106d9d700 1 --2- 172.21.15.114:0/3569351440 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fa108103560 0x7fa10819fed0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-23T10:24:53.351 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:24:53.349+0000 7fa10659c700 1 --2- 172.21.15.114:0/3569351440 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fa1081019c0 0x7fa10806b280 secure :-1 s=READY pgs=243 cs=0 l=1 rev1=1 crypto rx=0x7fa0f8009f50 tx=0x7fa0f8003a10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T10:24:53.351 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:24:53.349+0000 7fa0f77fe700 1 -- 172.21.15.114:0/3569351440 <== mon.0 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa0f8027070 con 0x7fa1081019c0 2024-04-23T10:24:53.351 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:24:53.349+0000 7fa10cc79700 1 -- 172.21.15.114:0/3569351440 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fa1081a0a30 con 0x7fa1081019c0 2024-04-23T10:24:53.351 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:24:53.349+0000 7fa0f77fe700 1 -- 172.21.15.114:0/3569351440 <== mon.0 v2:172.21.15.114:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fa0f8022470 con 0x7fa1081019c0 2024-04-23T10:24:53.351 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:24:53.349+0000 7fa0f77fe700 1 -- 172.21.15.114:0/3569351440 <== mon.0 v2:172.21.15.114:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa0f800f6c0 con 0x7fa1081019c0 2024-04-23T10:24:53.352 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:24:53.349+0000 7fa10cc79700 1 -- 172.21.15.114:0/3569351440 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fa1081a0f60 con 0x7fa1081019c0 2024-04-23T10:24:53.352 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:24:53.350+0000 7fa10cc79700 1 -- 172.21.15.114:0/3569351440 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fa108050f80 con 0x7fa1081019c0 2024-04-23T10:24:53.352 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:24:53.351+0000 7fa0f77fe700 1 -- 172.21.15.114:0/3569351440 <== mon.0 v2:172.21.15.114:3300/0 4 ==== mgrmap(e 17) ==== 138698+0+0 (secure 0 0 0) 0x7fa0f802a070 con 0x7fa1081019c0 2024-04-23T10:24:53.352 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:24:53.352+0000 7fa0f77fe700 1 --2- 172.21.15.114:0/3569351440 >> [v2:172.21.15.114:6800/465333808,v1:172.21.15.114:6801/465333808] conn(0x7fa0f00a3d80 0x7fa0f00a6240 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T10:24:53.352 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:24:53.352+0000 7fa105d9b700 1 --2- 172.21.15.114:0/3569351440 >> [v2:172.21.15.114:6800/465333808,v1:172.21.15.114:6801/465333808] conn(0x7fa0f00a3d80 0x7fa0f00a6240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T10:24:53.353 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:24:53.352+0000 7fa0f77fe700 1 -- 172.21.15.114:0/3569351440 <== mon.0 v2:172.21.15.114:3300/0 5 ==== osd_map(87..87 src has 1..87) ==== 7979+0+0 (secure 0 0 0) 0x7fa0f80d13d0 con 0x7fa1081019c0 2024-04-23T10:24:53.353 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:24:53.352+0000 7fa105d9b700 1 --2- 172.21.15.114:0/3569351440 >> [v2:172.21.15.114:6800/465333808,v1:172.21.15.114:6801/465333808] conn(0x7fa0f00a3d80 0x7fa0f00a6240 secure :-1 s=READY pgs=180 cs=0 l=1 rev1=1 crypto rx=0x7fa10806b800 tx=0x7fa0fc005d50 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T10:24:53.355 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:24:53.356+0000 7fa0f77fe700 1 -- 172.21.15.114:0/3569351440 <== mon.0 v2:172.21.15.114:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7fa0f8013070 con 0x7fa1081019c0 2024-04-23T10:24:53.635 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:24:53.636+0000 7fa10cc79700 1 -- 172.21.15.114:0/3569351440 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_command({"prefix": "auth get-or-create", "entity": "client.0", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]} v 0) -- 0x7fa10802e390 con 0x7fa1081019c0 2024-04-23T10:24:53.638 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:24:53.639+0000 7fa0f77fe700 1 -- 172.21.15.114:0/3569351440 <== mon.0 v2:172.21.15.114:3300/0 7 ==== mon_command_ack([{"prefix": "auth get-or-create", "entity": "client.0", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]=0 v31) ==== 170+0+59 (secure 0 0 0) 0x7fa0f809de90 con 0x7fa1081019c0 2024-04-23T10:24:53.638 INFO:teuthology.orchestra.run.smithi114.stdout:[client.0] 2024-04-23T10:24:53.638 INFO:teuthology.orchestra.run.smithi114.stdout: key = AQB1jCdmM1ALJhAA2BUwz/Nh4VI0QFJiNPQH7A== 2024-04-23T10:24:53.640 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:24:53.641+0000 7fa10cc79700 1 -- 172.21.15.114:0/3569351440 >> [v2:172.21.15.114:6800/465333808,v1:172.21.15.114:6801/465333808] conn(0x7fa0f00a3d80 msgr2=0x7fa0f00a6240 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T10:24:53.640 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:24:53.641+0000 7fa10cc79700 1 --2- 172.21.15.114:0/3569351440 >> [v2:172.21.15.114:6800/465333808,v1:172.21.15.114:6801/465333808] conn(0x7fa0f00a3d80 0x7fa0f00a6240 secure :-1 s=READY pgs=180 cs=0 l=1 rev1=1 crypto rx=0x7fa10806b800 tx=0x7fa0fc005d50 comp rx=0 tx=0).stop 2024-04-23T10:24:53.641 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:24:53.641+0000 7fa10cc79700 1 -- 172.21.15.114:0/3569351440 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fa1081019c0 msgr2=0x7fa10806b280 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T10:24:53.641 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:24:53.641+0000 7fa10cc79700 1 --2- 172.21.15.114:0/3569351440 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fa1081019c0 0x7fa10806b280 secure :-1 s=READY pgs=243 cs=0 l=1 rev1=1 crypto rx=0x7fa0f8009f50 tx=0x7fa0f8003a10 comp rx=0 tx=0).stop 2024-04-23T10:24:53.641 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:24:53.641+0000 7fa10cc79700 1 -- 172.21.15.114:0/3569351440 shutdown_connections 2024-04-23T10:24:53.641 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:24:53.641+0000 7fa10cc79700 1 --2- 172.21.15.114:0/3569351440 >> [v2:172.21.15.114:6800/465333808,v1:172.21.15.114:6801/465333808] conn(0x7fa0f00a3d80 0x7fa0f00a6240 unknown :-1 s=CLOSED pgs=180 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:24:53.641 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:24:53.641+0000 7fa10cc79700 1 --2- 172.21.15.114:0/3569351440 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fa108103560 0x7fa10819fed0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:24:53.641 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:24:53.641+0000 7fa10cc79700 1 --2- 172.21.15.114:0/3569351440 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fa1081019c0 0x7fa10806b280 unknown :-1 s=CLOSED pgs=243 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:24:53.641 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:24:53.641+0000 7fa10cc79700 1 --2- 172.21.15.114:0/3569351440 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fa108102bc0 0x7fa10819d890 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:24:53.641 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:24:53.641+0000 7fa10cc79700 1 -- 172.21.15.114:0/3569351440 >> 172.21.15.114:0/3569351440 conn(0x7fa1080fcf60 msgr2=0x7fa108105ec0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T10:24:53.641 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:24:53.641+0000 7fa10cc79700 1 -- 172.21.15.114:0/3569351440 shutdown_connections 2024-04-23T10:24:53.641 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:24:53.641+0000 7fa10cc79700 1 -- 172.21.15.114:0/3569351440 wait complete. 2024-04-23T10:24:53.992 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 23 10:24:53 smithi119 ceph-mon[32608]: from='client.? 172.21.15.114:0/3569351440' entity='client.admin' cmd=[{"prefix": "auth get-or-create", "entity": "client.0", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]: dispatch 2024-04-23T10:24:53.992 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 23 10:24:53 smithi119 ceph-mon[32608]: from='client.? 172.21.15.114:0/3569351440' entity='client.admin' cmd='[{"prefix": "auth get-or-create", "entity": "client.0", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]': finished 2024-04-23T10:24:54.004 INFO:journalctl@ceph.mon.a.smithi114.stdout:Apr 23 10:24:53 smithi114 ceph-mon[29255]: from='client.? 172.21.15.114:0/3569351440' entity='client.admin' cmd=[{"prefix": "auth get-or-create", "entity": "client.0", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]: dispatch 2024-04-23T10:24:54.004 INFO:journalctl@ceph.mon.a.smithi114.stdout:Apr 23 10:24:53 smithi114 ceph-mon[29255]: from='client.? 172.21.15.114:0/3569351440' entity='client.admin' cmd='[{"prefix": "auth get-or-create", "entity": "client.0", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]': finished 2024-04-23T10:24:54.078 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 23 10:24:53 smithi204 ceph-mon[32649]: from='client.? 172.21.15.114:0/3569351440' entity='client.admin' cmd=[{"prefix": "auth get-or-create", "entity": "client.0", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]: dispatch 2024-04-23T10:24:54.078 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 23 10:24:53 smithi204 ceph-mon[32649]: from='client.? 172.21.15.114:0/3569351440' entity='client.admin' cmd='[{"prefix": "auth get-or-create", "entity": "client.0", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]': finished 2024-04-23T10:24:54.409 DEBUG:teuthology.orchestra.run.smithi114:> set -ex 2024-04-23T10:24:54.409 DEBUG:teuthology.orchestra.run.smithi114:> sudo dd of=/etc/ceph/ceph.client.0.keyring 2024-04-23T10:24:54.409 DEBUG:teuthology.orchestra.run.smithi114:> sudo chmod 0644 /etc/ceph/ceph.client.0.keyring 2024-04-23T10:24:54.445 INFO:tasks.ceph:Waiting until ceph daemons up and pgs clean... 2024-04-23T10:24:54.445 INFO:tasks.cephadm.ceph_manager.ceph:waiting for mgr available 2024-04-23T10:24:54.445 DEBUG:teuthology.orchestra.run.smithi114:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d69f13d827952bb07bb9b7973bca14f97d764d6f shell --fsid aa240572-0159-11ef-bc93-c7b262605968 -- ceph mgr dump --format=json 2024-04-23T10:24:54.722 INFO:teuthology.orchestra.run.smithi114.stderr:Inferring config /var/lib/ceph/aa240572-0159-11ef-bc93-c7b262605968/mon.a/config 2024-04-23T10:24:54.953 INFO:journalctl@ceph.mon.a.smithi114.stdout:Apr 23 10:24:54 smithi114 ceph-mon[29255]: pgmap v291: 97 pgs: 97 active+clean; 618 KiB data, 370 MiB used, 1.0 TiB / 1.0 TiB avail; 7.2 KiB/s rd, 8.3 KiB/s wr, 19 op/s 2024-04-23T10:24:54.991 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 23 10:24:54 smithi119 ceph-mon[32608]: pgmap v291: 97 pgs: 97 active+clean; 618 KiB data, 370 MiB used, 1.0 TiB / 1.0 TiB avail; 7.2 KiB/s rd, 8.3 KiB/s wr, 19 op/s 2024-04-23T10:24:55.077 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 23 10:24:54 smithi204 ceph-mon[32649]: pgmap v291: 97 pgs: 97 active+clean; 618 KiB data, 370 MiB used, 1.0 TiB / 1.0 TiB avail; 7.2 KiB/s rd, 8.3 KiB/s wr, 19 op/s 2024-04-23T10:24:56.435 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:24:56.434+0000 7fd9fafc3700 1 -- 172.21.15.114:0/4133751977 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fd9f4102330 msgr2=0x7fd9f410e7f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T10:24:56.436 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:24:56.434+0000 7fd9fafc3700 1 --2- 172.21.15.114:0/4133751977 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fd9f4102330 0x7fd9f410e7f0 secure :-1 s=READY pgs=244 cs=0 l=1 rev1=1 crypto rx=0x7fd9ec00b0a0 tx=0x7fd9ec00b3b0 comp rx=0 tx=0).stop 2024-04-23T10:24:56.436 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:24:56.435+0000 7fd9fafc3700 1 -- 172.21.15.114:0/4133751977 shutdown_connections 2024-04-23T10:24:56.436 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:24:56.435+0000 7fd9fafc3700 1 --2- 172.21.15.114:0/4133751977 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fd9f4102330 0x7fd9f410e7f0 unknown :-1 s=CLOSED pgs=244 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:24:56.436 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:24:56.435+0000 7fd9fafc3700 1 --2- 172.21.15.114:0/4133751977 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fd9f4107920 0x7fd9f4107d00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:24:56.436 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:24:56.435+0000 7fd9fafc3700 1 --2- 172.21.15.114:0/4133751977 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fd9f4101970 0x7fd9f4101df0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:24:56.436 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:24:56.435+0000 7fd9fafc3700 1 -- 172.21.15.114:0/4133751977 >> 172.21.15.114:0/4133751977 conn(0x7fd9f4078910 msgr2=0x7fd9f4078d20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T10:24:56.436 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:24:56.435+0000 7fd9fafc3700 1 -- 172.21.15.114:0/4133751977 shutdown_connections 2024-04-23T10:24:56.436 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:24:56.435+0000 7fd9fafc3700 1 -- 172.21.15.114:0/4133751977 wait complete. 2024-04-23T10:24:56.436 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:24:56.435+0000 7fd9fafc3700 1 Processor -- start 2024-04-23T10:24:56.436 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:24:56.435+0000 7fd9fafc3700 1 -- start start 2024-04-23T10:24:56.436 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:24:56.435+0000 7fd9fafc3700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fd9f4101970 0x7fd9f4199ef0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T10:24:56.436 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:24:56.435+0000 7fd9fafc3700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fd9f4102330 0x7fd9f419a430 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T10:24:56.436 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:24:56.435+0000 7fd9fafc3700 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fd9f4107920 0x7fd9f41a4f70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T10:24:56.436 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:24:56.435+0000 7fd9fafc3700 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x7fd9f4111b30 con 0x7fd9f4101970 2024-04-23T10:24:56.437 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:24:56.435+0000 7fd9fafc3700 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 -- 0x7fd9f41119b0 con 0x7fd9f4102330 2024-04-23T10:24:56.437 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:24:56.435+0000 7fd9fafc3700 1 -- --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_getmap magic: 0 -- 0x7fd9f4111cb0 con 0x7fd9f4107920 2024-04-23T10:24:56.437 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:24:56.435+0000 7fd9f8d5f700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fd9f4101970 0x7fd9f4199ef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T10:24:56.437 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:24:56.436+0000 7fd9f8d5f700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fd9f4101970 0x7fd9f4199ef0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.114:36248/0 (socket says 172.21.15.114:36248) 2024-04-23T10:24:56.437 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:24:56.436+0000 7fd9f8d5f700 1 -- 172.21.15.114:0/3077385510 learned_addr learned my addr 172.21.15.114:0/3077385510 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-04-23T10:24:56.437 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:24:56.436+0000 7fd9f3fff700 1 --2- 172.21.15.114:0/3077385510 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fd9f4102330 0x7fd9f419a430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T10:24:56.437 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:24:56.436+0000 7fd9f9560700 1 --2- 172.21.15.114:0/3077385510 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fd9f4107920 0x7fd9f41a4f70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T10:24:56.437 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:24:56.436+0000 7fd9f8d5f700 1 -- 172.21.15.114:0/3077385510 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fd9f4107920 msgr2=0x7fd9f41a4f70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T10:24:56.437 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:24:56.436+0000 7fd9f8d5f700 1 --2- 172.21.15.114:0/3077385510 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fd9f4107920 0x7fd9f41a4f70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:24:56.437 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:24:56.436+0000 7fd9f8d5f700 1 -- 172.21.15.114:0/3077385510 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fd9f4102330 msgr2=0x7fd9f419a430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T10:24:56.437 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:24:56.436+0000 7fd9f8d5f700 1 --2- 172.21.15.114:0/3077385510 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fd9f4102330 0x7fd9f419a430 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:24:56.437 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:24:56.436+0000 7fd9f8d5f700 1 -- 172.21.15.114:0/3077385510 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd9f41a54b0 con 0x7fd9f4101970 2024-04-23T10:24:56.437 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:24:56.436+0000 7fd9f8d5f700 1 --2- 172.21.15.114:0/3077385510 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fd9f4101970 0x7fd9f4199ef0 secure :-1 s=READY pgs=245 cs=0 l=1 rev1=1 crypto rx=0x7fd9e800d980 tx=0x7fd9e800dc90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T10:24:56.437 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:24:56.436+0000 7fd9f1ffb700 1 -- 172.21.15.114:0/3077385510 <== mon.0 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd9e8017070 con 0x7fd9f4101970 2024-04-23T10:24:56.438 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:24:56.436+0000 7fd9f1ffb700 1 -- 172.21.15.114:0/3077385510 <== mon.0 v2:172.21.15.114:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fd9e8005020 con 0x7fd9f4101970 2024-04-23T10:24:56.438 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:24:56.436+0000 7fd9fafc3700 1 -- 172.21.15.114:0/3077385510 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fd9f41a5740 con 0x7fd9f4101970 2024-04-23T10:24:56.438 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:24:56.436+0000 7fd9f1ffb700 1 -- 172.21.15.114:0/3077385510 <== mon.0 v2:172.21.15.114:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd9e8009b50 con 0x7fd9f4101970 2024-04-23T10:24:56.438 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:24:56.436+0000 7fd9fafc3700 1 -- 172.21.15.114:0/3077385510 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fd9f41a5c50 con 0x7fd9f4101970 2024-04-23T10:24:56.439 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:24:56.437+0000 7fd9fafc3700 1 -- 172.21.15.114:0/3077385510 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fd9f4069730 con 0x7fd9f4101970 2024-04-23T10:24:56.439 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:24:56.438+0000 7fd9f1ffb700 1 -- 172.21.15.114:0/3077385510 <== mon.0 v2:172.21.15.114:3300/0 4 ==== mgrmap(e 17) ==== 138698+0+0 (secure 0 0 0) 0x7fd9e80051d0 con 0x7fd9f4101970 2024-04-23T10:24:56.439 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:24:56.439+0000 7fd9f1ffb700 1 --2- 172.21.15.114:0/3077385510 >> [v2:172.21.15.114:6800/465333808,v1:172.21.15.114:6801/465333808] conn(0x7fd9e40a3d50 0x7fd9e40a6210 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T10:24:56.439 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:24:56.439+0000 7fd9f3fff700 1 --2- 172.21.15.114:0/3077385510 >> [v2:172.21.15.114:6800/465333808,v1:172.21.15.114:6801/465333808] conn(0x7fd9e40a3d50 0x7fd9e40a6210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T10:24:56.439 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:24:56.439+0000 7fd9f1ffb700 1 -- 172.21.15.114:0/3077385510 <== mon.0 v2:172.21.15.114:3300/0 5 ==== osd_map(87..87 src has 1..87) ==== 7979+0+0 (secure 0 0 0) 0x7fd9e80d0240 con 0x7fd9f4101970 2024-04-23T10:24:56.439 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:24:56.439+0000 7fd9f3fff700 1 --2- 172.21.15.114:0/3077385510 >> [v2:172.21.15.114:6800/465333808,v1:172.21.15.114:6801/465333808] conn(0x7fd9e40a3d50 0x7fd9e40a6210 secure :-1 s=READY pgs=181 cs=0 l=1 rev1=1 crypto rx=0x7fd9f4078ea0 tx=0x7fd9e000a300 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T10:24:56.442 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:24:56.443+0000 7fd9f1ffb700 1 -- 172.21.15.114:0/3077385510 <== mon.0 v2:172.21.15.114:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7fd9e8020020 con 0x7fd9f4101970 2024-04-23T10:24:56.700 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:24:56.701+0000 7fd9fafc3700 1 -- 172.21.15.114:0/3077385510 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_command({"prefix": "mgr dump", "format": "json"} v 0) -- 0x7fd9f402df80 con 0x7fd9f4101970 2024-04-23T10:24:56.705 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:24:56.707+0000 7fd9f1ffb700 1 -- 172.21.15.114:0/3077385510 <== mon.0 v2:172.21.15.114:3300/0 7 ==== mon_command_ack([{"prefix": "mgr dump", "format": "json"}]=0 v17) ==== 74+0+265002 (secure 0 0 0) 0x7fd9e809cc70 con 0x7fd9f4101970 2024-04-23T10:24:56.709 INFO:teuthology.orchestra.run.smithi114.stdout: 2024-04-23T10:24:56.711 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:24:56.709+0000 7fd9fafc3700 1 -- 172.21.15.114:0/3077385510 >> [v2:172.21.15.114:6800/465333808,v1:172.21.15.114:6801/465333808] conn(0x7fd9e40a3d50 msgr2=0x7fd9e40a6210 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T10:24:56.711 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:24:56.709+0000 7fd9fafc3700 1 --2- 172.21.15.114:0/3077385510 >> [v2:172.21.15.114:6800/465333808,v1:172.21.15.114:6801/465333808] conn(0x7fd9e40a3d50 0x7fd9e40a6210 secure :-1 s=READY pgs=181 cs=0 l=1 rev1=1 crypto rx=0x7fd9f4078ea0 tx=0x7fd9e000a300 comp rx=0 tx=0).stop 2024-04-23T10:24:56.711 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:24:56.709+0000 7fd9fafc3700 1 -- 172.21.15.114:0/3077385510 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fd9f4101970 msgr2=0x7fd9f4199ef0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T10:24:56.711 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:24:56.709+0000 7fd9fafc3700 1 --2- 172.21.15.114:0/3077385510 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fd9f4101970 0x7fd9f4199ef0 secure :-1 s=READY pgs=245 cs=0 l=1 rev1=1 crypto rx=0x7fd9e800d980 tx=0x7fd9e800dc90 comp rx=0 tx=0).stop 2024-04-23T10:24:56.711 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:24:56.709+0000 7fd9fafc3700 1 -- 172.21.15.114:0/3077385510 shutdown_connections 2024-04-23T10:24:56.711 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:24:56.709+0000 7fd9fafc3700 1 --2- 172.21.15.114:0/3077385510 >> [v2:172.21.15.114:6800/465333808,v1:172.21.15.114:6801/465333808] conn(0x7fd9e40a3d50 0x7fd9e40a6210 unknown :-1 s=CLOSED pgs=181 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:24:56.711 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:24:56.709+0000 7fd9fafc3700 1 --2- 172.21.15.114:0/3077385510 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fd9f4107920 0x7fd9f41a4f70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:24:56.711 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:24:56.709+0000 7fd9fafc3700 1 --2- 172.21.15.114:0/3077385510 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fd9f4101970 0x7fd9f4199ef0 unknown :-1 s=CLOSED pgs=245 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:24:56.711 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:24:56.709+0000 7fd9fafc3700 1 --2- 172.21.15.114:0/3077385510 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fd9f4102330 0x7fd9f419a430 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:24:56.711 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:24:56.709+0000 7fd9fafc3700 1 -- 172.21.15.114:0/3077385510 >> 172.21.15.114:0/3077385510 conn(0x7fd9f4078910 msgr2=0x7fd9f40ff940 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T10:24:56.711 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:24:56.710+0000 7fd9fafc3700 1 -- 172.21.15.114:0/3077385510 shutdown_connections 2024-04-23T10:24:56.711 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:24:56.710+0000 7fd9fafc3700 1 -- 172.21.15.114:0/3077385510 wait complete. 2024-04-23T10:24:57.328 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 23 10:24:57 smithi204 ceph-mon[32649]: pgmap v292: 97 pgs: 97 active+clean; 618 KiB data, 370 MiB used, 1.0 TiB / 1.0 TiB avail; 6.5 KiB/s rd, 7.5 KiB/s wr, 17 op/s 2024-04-23T10:24:57.328 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 23 10:24:57 smithi204 ceph-mon[32649]: from='client.? 172.21.15.114:0/3077385510' entity='client.admin' cmd=[{"prefix": "mgr dump", "format": "json"}]: dispatch 2024-04-23T10:24:57.337 INFO:journalctl@ceph.mon.a.smithi114.stdout:Apr 23 10:24:57 smithi114 ceph-mon[29255]: pgmap v292: 97 pgs: 97 active+clean; 618 KiB data, 370 MiB used, 1.0 TiB / 1.0 TiB avail; 6.5 KiB/s rd, 7.5 KiB/s wr, 17 op/s 2024-04-23T10:24:57.338 INFO:journalctl@ceph.mon.a.smithi114.stdout:Apr 23 10:24:57 smithi114 ceph-mon[29255]: from='client.? 172.21.15.114:0/3077385510' entity='client.admin' cmd=[{"prefix": "mgr dump", "format": "json"}]: dispatch 2024-04-23T10:24:57.364 INFO:teuthology.orchestra.run.smithi114.stdout:{"epoch":17,"flags":0,"active_gid":14152,"active_name":"x","active_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6800","nonce":465333808},{"type":"v1","addr":"172.21.15.114:6801","nonce":465333808}]},"active_addr":"172.21.15.114:6801/465333808","active_change":"2024-04-23T10:17:45.874711+0000","active_mgr_features":4540701547738038271,"available":true,"standbys":[{"gid":24109,"name":"z","mgr_features":4540701547738038271,"available_modules":[{"name":"alerts","can_run":true,"error_string":"","module_options":{"interval":{"name":"interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"How frequently to reexamine health status","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"smtp_destination":{"name":"smtp_destination","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Email address to send alerts to","long_desc":"","tags":[],"see_also":[]},"smtp_from_name":{"name":"smtp_from_name","type":"str","level":"advanced","flags":1,"default_value":"Ceph","min":"","max":"","enum_allowed":[],"desc":"Email From: name","long_desc":"","tags":[],"see_also":[]},"smtp_host":{"name":"smtp_host","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_password":{"name":"smtp_password","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Password to authenticate with","long_desc":"","tags":[],"see_also":[]},"smtp_port":{"name":"smtp_port","type":"int","level":"advanced","flags":1,"default_value":"465","min":"","max":"","enum_allowed":[],"desc":"SMTP port","long_desc":"","tags":[],"see_also":[]},"smtp_sender":{"name":"smtp_sender","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP envelope sender","long_desc":"","tags":[],"see_also":[]},"smtp_ssl":{"name":"smtp_ssl","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Use SSL to connect to SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_user":{"name":"smtp_user","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"User to authenticate as","long_desc":"","tags":[],"see_also":[]}}},{"name":"balancer","can_run":true,"error_string":"","module_options":{"active":{"name":"active","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"automatically balance PGs across cluster","long_desc":"","tags":[],"see_also":[]},"begin_time":{"name":"begin_time","type":"str","level":"advanced","flags":1,"default_value":"0000","min":"","max":"","enum_allowed":[],"desc":"beginning time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"begin_weekday":{"name":"begin_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"6","enum_allowed":[],"desc":"Restrict automatic balancing to this day of the week or later","long_desc":"0 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"crush_compat_max_iterations":{"name":"crush_compat_max_iterations","type":"uint","level":"advanced","flags":1,"default_value":"25","min":"1","max":"250","enum_allowed":[],"desc":"maximum number of iterations to attempt optimization","long_desc":"","tags":[],"see_also":[]},"crush_compat_metrics":{"name":"crush_compat_metrics","type":"str","level":"advanced","flags":1,"default_value":"pgs,objects,bytes","min":"","max":"","enum_allowed":[],"desc":"metrics with which to calculate OSD utilization","long_desc":"Value is a list of one or more of \"pgs\", \"objects\", or \"bytes\", and indicates which metrics to use to balance utilization.","tags":[],"see_also":[]},"crush_compat_step":{"name":"crush_compat_step","type":"float","level":"advanced","flags":1,"default_value":"0.5","min":"0.001","max":"0.999","enum_allowed":[],"desc":"aggressiveness of optimization","long_desc":".99 is very aggressive, .01 is less aggressive","tags":[],"see_also":[]},"end_time":{"name":"end_time","type":"str","level":"advanced","flags":1,"default_value":"2359","min":"","max":"","enum_allowed":[],"desc":"ending time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"end_weekday":{"name":"end_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"6","enum_allowed":[],"desc":"Restrict automatic balancing to days of the week earlier than this","long_desc":"0 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_score":{"name":"min_score","type":"float","level":"advanced","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"minimum score, below which no optimization is attempted","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":1,"default_value":"upmap","min":"","max":"","enum_allowed":["crush-compat","none","read","upmap","upmap-read"],"desc":"Balancer mode","long_desc":"","tags":[],"see_also":[]},"pool_ids":{"name":"pool_ids","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"pools which the automatic balancing will be limited to","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and attempt optimization","long_desc":"","tags":[],"see_also":[]},"upmap_max_deviation":{"name":"upmap_max_deviation","type":"int","level":"advanced","flags":1,"default_value":"5","min":"1","max":"","enum_allowed":[],"desc":"deviation below which no optimization is attempted","long_desc":"If the number of PGs are within this count then no optimization is attempted","tags":[],"see_also":[]},"upmap_max_optimizations":{"name":"upmap_max_optimizations","type":"uint","level":"advanced","flags":1,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"maximum upmap optimizations to make per attempt","long_desc":"","tags":[],"see_also":[]}}},{"name":"cephadm","can_run":true,"error_string":"","module_options":{"agent_down_multiplier":{"name":"agent_down_multiplier","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"","max":"","enum_allowed":[],"desc":"Multiplied by agent refresh rate to calculate how long agent must not report before being marked down","long_desc":"","tags":[],"see_also":[]},"agent_refresh_rate":{"name":"agent_refresh_rate","type":"secs","level":"advanced","flags":0,"default_value":"20","min":"","max":"","enum_allowed":[],"desc":"How often agent on each host will try to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"agent_starting_port":{"name":"agent_starting_port","type":"int","level":"advanced","flags":0,"default_value":"4721","min":"","max":"","enum_allowed":[],"desc":"First port agent will try to bind to (will also try up to next 1000 subsequent ports if blocked)","long_desc":"","tags":[],"see_also":[]},"allow_ptrace":{"name":"allow_ptrace","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow SYS_PTRACE capability on ceph containers","long_desc":"The SYS_PTRACE capability is needed to attach to a process with gdb or strace. Enabling this options can allow debugging daemons that encounter problems at runtime.","tags":[],"see_also":[]},"autotune_interval":{"name":"autotune_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to autotune daemon memory","long_desc":"","tags":[],"see_also":[]},"autotune_memory_target_ratio":{"name":"autotune_memory_target_ratio","type":"float","level":"advanced","flags":0,"default_value":"0.7","min":"","max":"","enum_allowed":[],"desc":"ratio of total system memory to divide amongst autotuned daemons","long_desc":"","tags":[],"see_also":[]},"cephadm_log_destination":{"name":"cephadm_log_destination","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":["file","file,syslog","syslog"],"desc":"Destination for cephadm command's persistent logging","long_desc":"","tags":[],"see_also":[]},"cgroups_split":{"name":"cgroups_split","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Pass --cgroups=split when cephadm creates containers (currently podman only)","long_desc":"","tags":[],"see_also":[]},"config_checks_enabled":{"name":"config_checks_enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable or disable the cephadm configuration analysis","long_desc":"","tags":[],"see_also":[]},"config_dashboard":{"name":"config_dashboard","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"manage configs like API endpoints in Dashboard.","long_desc":"","tags":[],"see_also":[]},"container_image_alertmanager":{"name":"container_image_alertmanager","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/alertmanager:v0.25.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_base":{"name":"container_image_base","type":"str","level":"advanced","flags":1,"default_value":"quay.io/ceph/ceph","min":"","max":"","enum_allowed":[],"desc":"Container image name, without the tag","long_desc":"","tags":[],"see_also":[]},"container_image_elasticsearch":{"name":"container_image_elasticsearch","type":"str","level":"advanced","flags":0,"default_value":"quay.io/omrizeneva/elasticsearch:6.8.23","min":"","max":"","enum_allowed":[],"desc":"elasticsearch container image","long_desc":"","tags":[],"see_also":[]},"container_image_grafana":{"name":"container_image_grafana","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/grafana:9.4.12","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_haproxy":{"name":"container_image_haproxy","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/haproxy:2.3","min":"","max":"","enum_allowed":[],"desc":"HAproxy container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_agent":{"name":"container_image_jaeger_agent","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-agent:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger agent container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_collector":{"name":"container_image_jaeger_collector","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-collector:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger collector container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_query":{"name":"container_image_jaeger_query","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-query:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger query container image","long_desc":"","tags":[],"see_also":[]},"container_image_keepalived":{"name":"container_image_keepalived","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/keepalived:2.2.4","min":"","max":"","enum_allowed":[],"desc":"Keepalived container image","long_desc":"","tags":[],"see_also":[]},"container_image_loki":{"name":"container_image_loki","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/loki:2.4.0","min":"","max":"","enum_allowed":[],"desc":"Loki container image","long_desc":"","tags":[],"see_also":[]},"container_image_node_exporter":{"name":"container_image_node_exporter","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/node-exporter:v1.5.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_nvmeof":{"name":"container_image_nvmeof","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/nvmeof:1.2.1","min":"","max":"","enum_allowed":[],"desc":"Nvme-of container image","long_desc":"","tags":[],"see_also":[]},"container_image_prometheus":{"name":"container_image_prometheus","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/prometheus:v2.43.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_promtail":{"name":"container_image_promtail","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/promtail:2.4.0","min":"","max":"","enum_allowed":[],"desc":"Promtail container image","long_desc":"","tags":[],"see_also":[]},"container_image_samba":{"name":"container_image_samba","type":"str","level":"advanced","flags":0,"default_value":"quay.io/samba.org/samba-server:devbuilds-centos-amd64","min":"","max":"","enum_allowed":[],"desc":"Samba/SMB container image","long_desc":"","tags":[],"see_also":[]},"container_image_snmp_gateway":{"name":"container_image_snmp_gateway","type":"str","level":"advanced","flags":0,"default_value":"docker.io/maxwo/snmp-notifier:v1.2.1","min":"","max":"","enum_allowed":[],"desc":"SNMP Gateway container image","long_desc":"","tags":[],"see_also":[]},"container_init":{"name":"container_init","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Run podman/docker with `--init`","long_desc":"","tags":[],"see_also":[]},"daemon_cache_timeout":{"name":"daemon_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"seconds to cache service (daemon) inventory","long_desc":"","tags":[],"see_also":[]},"default_cephadm_command_timeout":{"name":"default_cephadm_command_timeout","type":"int","level":"advanced","flags":0,"default_value":"900","min":"","max":"","enum_allowed":[],"desc":"Default timeout applied to cephadm commands run directly on the host (in seconds)","long_desc":"","tags":[],"see_also":[]},"default_registry":{"name":"default_registry","type":"str","level":"advanced","flags":0,"default_value":"docker.io","min":"","max":"","enum_allowed":[],"desc":"Search-registry to which we should normalize unqualified image names. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"device_cache_timeout":{"name":"device_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"1800","min":"","max":"","enum_allowed":[],"desc":"seconds to cache device inventory","long_desc":"","tags":[],"see_also":[]},"device_enhanced_scan":{"name":"device_enhanced_scan","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use libstoragemgmt during device scans","long_desc":"","tags":[],"see_also":[]},"facts_cache_timeout":{"name":"facts_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"seconds to cache host facts data","long_desc":"","tags":[],"see_also":[]},"grafana_dashboards_path":{"name":"grafana_dashboards_path","type":"str","level":"advanced","flags":0,"default_value":"/etc/grafana/dashboards/ceph-dashboard/","min":"","max":"","enum_allowed":[],"desc":"location of dashboards to include in grafana deployments","long_desc":"","tags":[],"see_also":[]},"host_check_interval":{"name":"host_check_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to perform a host check","long_desc":"","tags":[],"see_also":[]},"hw_monitoring":{"name":"hw_monitoring","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Deploy hw monitoring daemon on every host.","long_desc":"","tags":[],"see_also":[]},"inventory_list_all":{"name":"inventory_list_all","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Whether ceph-volume inventory should report more devices (mostly mappers (LVs / mpaths), partitions...)","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_refresh_metadata":{"name":"log_refresh_metadata","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Log all refresh metadata. Includes daemon, device, and host info collected regularly. Only has effect if logging at debug level","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"log to the \"cephadm\" cluster log channel\"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf":{"name":"manage_etc_ceph_ceph_conf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Manage and own /etc/ceph/ceph.conf on the hosts.","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf_hosts":{"name":"manage_etc_ceph_ceph_conf_hosts","type":"str","level":"advanced","flags":0,"default_value":"*","min":"","max":"","enum_allowed":[],"desc":"PlacementSpec describing on which hosts to manage /etc/ceph/ceph.conf","long_desc":"","tags":[],"see_also":[]},"max_count_per_host":{"name":"max_count_per_host","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of daemons per service per host","long_desc":"","tags":[],"see_also":[]},"max_osd_draining_count":{"name":"max_osd_draining_count","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of osds that will be drained simultaneously when osds are removed","long_desc":"","tags":[],"see_also":[]},"migration_current":{"name":"migration_current","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"internal - do not modify","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":0,"default_value":"root","min":"","max":"","enum_allowed":["cephadm-package","root"],"desc":"mode for remote execution of cephadm","long_desc":"","tags":[],"see_also":[]},"oob_default_addr":{"name":"oob_default_addr","type":"str","level":"advanced","flags":0,"default_value":"169.254.1.1","min":"","max":"","enum_allowed":[],"desc":"Default address for RedFish API (oob management).","long_desc":"","tags":[],"see_also":[]},"prometheus_alerts_path":{"name":"prometheus_alerts_path","type":"str","level":"advanced","flags":0,"default_value":"/etc/prometheus/ceph/ceph_default_alerts.yml","min":"","max":"","enum_allowed":[],"desc":"location of alerts to include in prometheus deployments","long_desc":"","tags":[],"see_also":[]},"registry_insecure":{"name":"registry_insecure","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Registry is to be considered insecure (no TLS available). Only for development purposes.","long_desc":"","tags":[],"see_also":[]},"registry_password":{"name":"registry_password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository password. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"registry_url":{"name":"registry_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Registry url for login purposes. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"registry_username":{"name":"registry_username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository username. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"secure_monitoring_stack":{"name":"secure_monitoring_stack","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable TLS security for all the monitoring stack daemons","long_desc":"","tags":[],"see_also":[]},"service_discovery_port":{"name":"service_discovery_port","type":"int","level":"advanced","flags":0,"default_value":"8765","min":"","max":"","enum_allowed":[],"desc":"cephadm service discovery port","long_desc":"","tags":[],"see_also":[]},"ssh_config_file":{"name":"ssh_config_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"customized SSH config file to connect to managed hosts","long_desc":"","tags":[],"see_also":[]},"use_agent":{"name":"use_agent","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use cephadm agent on each host to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"use_repo_digest":{"name":"use_repo_digest","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Automatically convert image tags to image digest. Make sure all daemons use the same image","long_desc":"","tags":[],"see_also":[]},"warn_on_failed_host_check":{"name":"warn_on_failed_host_check","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if the host check fails","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_daemons":{"name":"warn_on_stray_daemons","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected that are not managed by cephadm","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_hosts":{"name":"warn_on_stray_hosts","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected on a host that is not managed by cephadm","long_desc":"","tags":[],"see_also":[]}}},{"name":"crash","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"retain_interval":{"name":"retain_interval","type":"secs","level":"advanced","flags":1,"default_value":"31536000","min":"","max":"","enum_allowed":[],"desc":"how long to retain crashes before pruning them","long_desc":"","tags":[],"see_also":[]},"warn_recent_interval":{"name":"warn_recent_interval","type":"secs","level":"advanced","flags":1,"default_value":"1209600","min":"","max":"","enum_allowed":[],"desc":"time interval in which to warn about recent crashes","long_desc":"","tags":[],"see_also":[]}}},{"name":"dashboard","can_run":true,"error_string":"","module_options":{"ACCOUNT_LOCKOUT_ATTEMPTS":{"name":"ACCOUNT_LOCKOUT_ATTEMPTS","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_HOST":{"name":"ALERTMANAGER_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_SSL_VERIFY":{"name":"ALERTMANAGER_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_ENABLED":{"name":"AUDIT_API_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_LOG_PAYLOAD":{"name":"AUDIT_API_LOG_PAYLOAD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ENABLE_BROWSABLE_API":{"name":"ENABLE_BROWSABLE_API","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_CEPHFS":{"name":"FEATURE_TOGGLE_CEPHFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_DASHBOARD":{"name":"FEATURE_TOGGLE_DASHBOARD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_ISCSI":{"name":"FEATURE_TOGGLE_ISCSI","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_MIRRORING":{"name":"FEATURE_TOGGLE_MIRRORING","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_NFS":{"name":"FEATURE_TOGGLE_NFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RBD":{"name":"FEATURE_TOGGLE_RBD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RGW":{"name":"FEATURE_TOGGLE_RGW","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE":{"name":"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_PASSWORD":{"name":"GRAFANA_API_PASSWORD","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_SSL_VERIFY":{"name":"GRAFANA_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_URL":{"name":"GRAFANA_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_USERNAME":{"name":"GRAFANA_API_USERNAME","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_FRONTEND_API_URL":{"name":"GRAFANA_FRONTEND_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_UPDATE_DASHBOARDS":{"name":"GRAFANA_UPDATE_DASHBOARDS","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISCSI_API_SSL_VERIFICATION":{"name":"ISCSI_API_SSL_VERIFICATION","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISSUE_TRACKER_API_KEY":{"name":"ISSUE_TRACKER_API_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"MANAGED_BY_CLUSTERS":{"name":"MANAGED_BY_CLUSTERS","type":"str","level":"advanced","flags":0,"default_value":"[]","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"MULTICLUSTER_CONFIG":{"name":"MULTICLUSTER_CONFIG","type":"str","level":"advanced","flags":0,"default_value":"{}","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_HOST":{"name":"PROMETHEUS_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_SSL_VERIFY":{"name":"PROMETHEUS_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_COMPLEXITY_ENABLED":{"name":"PWD_POLICY_CHECK_COMPLEXITY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED":{"name":"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_LENGTH_ENABLED":{"name":"PWD_POLICY_CHECK_LENGTH_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_OLDPWD_ENABLED":{"name":"PWD_POLICY_CHECK_OLDPWD_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_USERNAME_ENABLED":{"name":"PWD_POLICY_CHECK_USERNAME_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_ENABLED":{"name":"PWD_POLICY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_EXCLUSION_LIST":{"name":"PWD_POLICY_EXCLUSION_LIST","type":"str","level":"advanced","flags":0,"default_value":"osd,host,dashboard,pool,block,nfs,ceph,monitors,gateway,logs,crush,maps","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_COMPLEXITY":{"name":"PWD_POLICY_MIN_COMPLEXITY","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_LENGTH":{"name":"PWD_POLICY_MIN_LENGTH","type":"int","level":"advanced","flags":0,"default_value":"8","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"REST_REQUESTS_TIMEOUT":{"name":"REST_REQUESTS_TIMEOUT","type":"int","level":"advanced","flags":0,"default_value":"45","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ACCESS_KEY":{"name":"RGW_API_ACCESS_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ADMIN_RESOURCE":{"name":"RGW_API_ADMIN_RESOURCE","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SECRET_KEY":{"name":"RGW_API_SECRET_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SSL_VERIFY":{"name":"RGW_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"UNSAFE_TLS_v1_2":{"name":"UNSAFE_TLS_v1_2","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_SPAN":{"name":"USER_PWD_EXPIRATION_SPAN","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_1":{"name":"USER_PWD_EXPIRATION_WARNING_1","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_2":{"name":"USER_PWD_EXPIRATION_WARNING_2","type":"int","level":"advanced","flags":0,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"cross_origin_url":{"name":"cross_origin_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"crt_file":{"name":"crt_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"debug":{"name":"debug","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable/disable debug options","long_desc":"","tags":[],"see_also":[]},"jwt_token_ttl":{"name":"jwt_token_ttl","type":"int","level":"advanced","flags":0,"default_value":"28800","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"motd":{"name":"motd","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"The message of the day","long_desc":"","tags":[],"see_also":[]},"redirect_resolve_ip_addr":{"name":"redirect_resolve_ip_addr","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":0,"default_value":"8080","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl_server_port":{"name":"ssl_server_port","type":"int","level":"advanced","flags":0,"default_value":"8443","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":0,"default_value":"redirect","min":"","max":"","enum_allowed":["error","redirect"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":0,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url_prefix":{"name":"url_prefix","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"devicehealth","can_run":true,"error_string":"","module_options":{"enable_monitoring":{"name":"enable_monitoring","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"monitor device health metrics","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mark_out_threshold":{"name":"mark_out_threshold","type":"secs","level":"advanced","flags":1,"default_value":"2419200","min":"","max":"","enum_allowed":[],"desc":"automatically mark OSD if it may fail before this long","long_desc":"","tags":[],"see_also":[]},"pool_name":{"name":"pool_name","type":"str","level":"advanced","flags":1,"default_value":"device_health_metrics","min":"","max":"","enum_allowed":[],"desc":"name of pool in which to store device health metrics","long_desc":"","tags":[],"see_also":[]},"retention_period":{"name":"retention_period","type":"secs","level":"advanced","flags":1,"default_value":"15552000","min":"","max":"","enum_allowed":[],"desc":"how long to retain device health metrics","long_desc":"","tags":[],"see_also":[]},"scrape_frequency":{"name":"scrape_frequency","type":"secs","level":"advanced","flags":1,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"how frequently to scrape device health metrics","long_desc":"","tags":[],"see_also":[]},"self_heal":{"name":"self_heal","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"preemptively heal cluster around devices that may fail","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and check device health","long_desc":"","tags":[],"see_also":[]},"warn_threshold":{"name":"warn_threshold","type":"secs","level":"advanced","flags":1,"default_value":"7257600","min":"","max":"","enum_allowed":[],"desc":"raise health warning if OSD may fail before this long","long_desc":"","tags":[],"see_also":[]}}},{"name":"diskprediction_local","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predict_interval":{"name":"predict_interval","type":"str","level":"advanced","flags":0,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predictor_model":{"name":"predictor_model","type":"str","level":"advanced","flags":0,"default_value":"prophetstor","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"str","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"influx","can_run":false,"error_string":"influxdb python module not found","module_options":{"batch_size":{"name":"batch_size","type":"int","level":"advanced","flags":0,"default_value":"5000","min":"","max":"","enum_allowed":[],"desc":"How big batches of data points should be when sending to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"database":{"name":"database","type":"str","level":"advanced","flags":0,"default_value":"ceph","min":"","max":"","enum_allowed":[],"desc":"InfluxDB database name. You will need to create this database and grant write privileges to the configured username or the username must have admin privileges to create it.","long_desc":"","tags":[],"see_also":[]},"hostname":{"name":"hostname","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server hostname","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"30","min":"5","max":"","enum_allowed":[],"desc":"Time between reports to InfluxDB. Default 30 seconds.","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"password":{"name":"password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"password of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"port":{"name":"port","type":"int","level":"advanced","flags":0,"default_value":"8086","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server port","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"str","level":"advanced","flags":0,"default_value":"false","min":"","max":"","enum_allowed":[],"desc":"Use https connection for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]},"threads":{"name":"threads","type":"int","level":"advanced","flags":0,"default_value":"5","min":"1","max":"32","enum_allowed":[],"desc":"How many worker threads should be spawned for sending data to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"username":{"name":"username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"username of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"verify_ssl":{"name":"verify_ssl","type":"str","level":"advanced","flags":0,"default_value":"true","min":"","max":"","enum_allowed":[],"desc":"Verify https cert for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]}}},{"name":"insights","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"iostat","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"k8sevents","can_run":true,"error_string":"","module_options":{"ceph_event_retention_days":{"name":"ceph_event_retention_days","type":"int","level":"advanced","flags":0,"default_value":"7","min":"","max":"","enum_allowed":[],"desc":"Days to hold ceph event information within local cache","long_desc":"","tags":[],"see_also":[]},"config_check_secs":{"name":"config_check_secs","type":"int","level":"advanced","flags":0,"default_value":"10","min":"10","max":"","enum_allowed":[],"desc":"interval (secs) to check for cluster configuration changes","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"localpool","can_run":true,"error_string":"","module_options":{"failure_domain":{"name":"failure_domain","type":"str","level":"advanced","flags":1,"default_value":"host","min":"","max":"","enum_allowed":[],"desc":"failure domain for any created local pool","long_desc":"what failure domain we should separate data replicas across.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_size":{"name":"min_size","type":"int","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"default min_size for any created local pool","long_desc":"value to set min_size to (unchanged from Ceph's default if this option is not set)","tags":[],"see_also":[]},"num_rep":{"name":"num_rep","type":"int","level":"advanced","flags":1,"default_value":"3","min":"","max":"","enum_allowed":[],"desc":"default replica count for any created local pool","long_desc":"","tags":[],"see_also":[]},"pg_num":{"name":"pg_num","type":"int","level":"advanced","flags":1,"default_value":"128","min":"","max":"","enum_allowed":[],"desc":"default pg_num for any created local pool","long_desc":"","tags":[],"see_also":[]},"prefix":{"name":"prefix","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"name prefix for any created local pool","long_desc":"","tags":[],"see_also":[]},"subtree":{"name":"subtree","type":"str","level":"advanced","flags":1,"default_value":"rack","min":"","max":"","enum_allowed":[],"desc":"CRUSH level for which to create a local pool","long_desc":"which CRUSH subtree type the module should create a pool for.","tags":[],"see_also":[]}}},{"name":"mds_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"mirroring","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"nfs","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"orchestrator","can_run":true,"error_string":"","module_options":{"fail_fs":{"name":"fail_fs","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Fail filesystem for rapid multi-rank mds upgrade","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"orchestrator":{"name":"orchestrator","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["cephadm","rook","test_orchestrator"],"desc":"Orchestrator backend","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_perf_query","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"pg_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"threshold":{"name":"threshold","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"1.0","max":"","enum_allowed":[],"desc":"scaling threshold","long_desc":"The factor by which the `NEW PG_NUM` must vary from the current`PG_NUM` before being accepted. Cannot be less than 1.0","tags":[],"see_also":[]}}},{"name":"progress","can_run":true,"error_string":"","module_options":{"allow_pg_recovery_event":{"name":"allow_pg_recovery_event","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow the module to show pg recovery progress","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_completed_events":{"name":"max_completed_events","type":"int","level":"advanced","flags":1,"default_value":"50","min":"","max":"","enum_allowed":[],"desc":"number of past completed events to remember","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"how long the module is going to sleep","long_desc":"","tags":[],"see_also":[]}}},{"name":"prometheus","can_run":true,"error_string":"","module_options":{"cache":{"name":"cache","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"exclude_perf_counters":{"name":"exclude_perf_counters","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Do not include perf-counters in the metrics output","long_desc":"Gathering perf-counters from a single Prometheus exporter can degrade ceph-mgr performance, especially in large clusters. Instead, Ceph-exporter daemons are now used by default for perf-counter gathering. This should only be disabled when no ceph-exporters are deployed.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools":{"name":"rbd_stats_pools","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools_refresh_interval":{"name":"rbd_stats_pools_refresh_interval","type":"int","level":"advanced","flags":0,"default_value":"300","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"scrape_interval":{"name":"scrape_interval","type":"float","level":"advanced","flags":0,"default_value":"15.0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"the IPv4 or IPv6 address on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":1,"default_value":"9283","min":"","max":"","enum_allowed":[],"desc":"the port on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"stale_cache_strategy":{"name":"stale_cache_strategy","type":"str","level":"advanced","flags":0,"default_value":"log","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":1,"default_value":"default","min":"","max":"","enum_allowed":["default","error"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":1,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rbd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_snap_create":{"name":"max_concurrent_snap_create","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mirror_snapshot_schedule":{"name":"mirror_snapshot_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"trash_purge_schedule":{"name":"trash_purge_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"restful","can_run":true,"error_string":"","module_options":{"enable_auth":{"name":"enable_auth","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rgw","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rook","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"storage_class":{"name":"storage_class","type":"str","level":"advanced","flags":0,"default_value":"local","min":"","max":"","enum_allowed":[],"desc":"storage class name for LSO-discovered PVs","long_desc":"","tags":[],"see_also":[]}}},{"name":"selftest","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption1":{"name":"roption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption2":{"name":"roption2","type":"str","level":"advanced","flags":0,"default_value":"xyz","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption1":{"name":"rwoption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption2":{"name":"rwoption2","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption3":{"name":"rwoption3","type":"float","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption4":{"name":"rwoption4","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption5":{"name":"rwoption5","type":"bool","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption6":{"name":"rwoption6","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption7":{"name":"rwoption7","type":"int","level":"advanced","flags":0,"default_value":"","min":"1","max":"42","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testkey":{"name":"testkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testlkey":{"name":"testlkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testnewline":{"name":"testnewline","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"snap_schedule","can_run":true,"error_string":"","module_options":{"allow_m_granularity":{"name":"allow_m_granularity","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow minute scheduled snapshots","long_desc":"","tags":[],"see_also":[]},"dump_on_update":{"name":"dump_on_update","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"dump database to debug log on update","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"stats","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"status","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telegraf","can_run":true,"error_string":"","module_options":{"address":{"name":"address","type":"str","level":"advanced","flags":0,"default_value":"unixgram:///tmp/telegraf.sock","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"15","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telemetry","can_run":true,"error_string":"","module_options":{"channel_basic":{"name":"channel_basic","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share basic cluster information (size, version)","long_desc":"","tags":[],"see_also":[]},"channel_crash":{"name":"channel_crash","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share metadata about Ceph daemon crashes (version, stack straces, etc)","long_desc":"","tags":[],"see_also":[]},"channel_device":{"name":"channel_device","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share device health metrics (e.g., SMART data, minus potentially identifying info like serial numbers)","long_desc":"","tags":[],"see_also":[]},"channel_ident":{"name":"channel_ident","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share a user-provided description and/or contact email for the cluster","long_desc":"","tags":[],"see_also":[]},"channel_perf":{"name":"channel_perf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share various performance metrics of a cluster","long_desc":"","tags":[],"see_also":[]},"contact":{"name":"contact","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"description":{"name":"description","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"device_url":{"name":"device_url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/device","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"int","level":"advanced","flags":0,"default_value":"24","min":"8","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"last_opt_revision":{"name":"last_opt_revision","type":"int","level":"advanced","flags":0,"default_value":"1","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard":{"name":"leaderboard","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard_description":{"name":"leaderboard_description","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"organization":{"name":"organization","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"proxy":{"name":"proxy","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url":{"name":"url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/report","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"test_orchestrator","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"volumes","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_clones":{"name":"max_concurrent_clones","type":"int","level":"advanced","flags":0,"default_value":"4","min":"","max":"","enum_allowed":[],"desc":"Number of asynchronous cloner threads","long_desc":"","tags":[],"see_also":[]},"periodic_async_work":{"name":"periodic_async_work","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Periodically check for async work","long_desc":"","tags":[],"see_also":[]},"snapshot_clone_delay":{"name":"snapshot_clone_delay","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"Delay clone begin operation by snapshot_clone_delay seconds","long_desc":"","tags":[],"see_also":[]},"snapshot_clone_no_wait":{"name":"snapshot_clone_no_wait","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Reject subvolume clone request when cloner threads are busy","long_desc":"","tags":[],"see_also":[]}}},{"name":"zabbix","can_run":true,"error_string":"","module_options":{"discovery_interval":{"name":"discovery_interval","type":"uint","level":"advanced","flags":0,"default_value":"100","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"identifier":{"name":"identifier","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_host":{"name":"zabbix_host","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_port":{"name":"zabbix_port","type":"int","level":"advanced","flags":0,"default_value":"10051","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_sender":{"name":"zabbix_sender","type":"str","level":"advanced","flags":0,"default_value":"/usr/bin/zabbix_sender","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}}]},{"gid":24116,"name":"y","mgr_features":4540701547738038271,"available_modules":[{"name":"alerts","can_run":true,"error_string":"","module_options":{"interval":{"name":"interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"How frequently to reexamine health status","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"smtp_destination":{"name":"smtp_destination","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Email address to send alerts to","long_desc":"","tags":[],"see_also":[]},"smtp_from_name":{"name":"smtp_from_name","type":"str","level":"advanced","flags":1,"default_value":"Ceph","min":"","max":"","enum_allowed":[],"desc":"Email From: name","long_desc":"","tags":[],"see_also":[]},"smtp_host":{"name":"smtp_host","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_password":{"name":"smtp_password","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Password to authenticate with","long_desc":"","tags":[],"see_also":[]},"smtp_port":{"name":"smtp_port","type":"int","level":"advanced","flags":1,"default_value":"465","min":"","max":"","enum_allowed":[],"desc":"SMTP port","long_desc":"","tags":[],"see_also":[]},"smtp_sender":{"name":"smtp_sender","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP envelope sender","long_desc":"","tags":[],"see_also":[]},"smtp_ssl":{"name":"smtp_ssl","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Use SSL to connect to SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_user":{"name":"smtp_user","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"User to authenticate as","long_desc":"","tags":[],"see_also":[]}}},{"name":"balancer","can_run":true,"error_string":"","module_options":{"active":{"name":"active","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"automatically balance PGs across cluster","long_desc":"","tags":[],"see_also":[]},"begin_time":{"name":"begin_time","type":"str","level":"advanced","flags":1,"default_value":"0000","min":"","max":"","enum_allowed":[],"desc":"beginning time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"begin_weekday":{"name":"begin_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"6","enum_allowed":[],"desc":"Restrict automatic balancing to this day of the week or later","long_desc":"0 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"crush_compat_max_iterations":{"name":"crush_compat_max_iterations","type":"uint","level":"advanced","flags":1,"default_value":"25","min":"1","max":"250","enum_allowed":[],"desc":"maximum number of iterations to attempt optimization","long_desc":"","tags":[],"see_also":[]},"crush_compat_metrics":{"name":"crush_compat_metrics","type":"str","level":"advanced","flags":1,"default_value":"pgs,objects,bytes","min":"","max":"","enum_allowed":[],"desc":"metrics with which to calculate OSD utilization","long_desc":"Value is a list of one or more of \"pgs\", \"objects\", or \"bytes\", and indicates which metrics to use to balance utilization.","tags":[],"see_also":[]},"crush_compat_step":{"name":"crush_compat_step","type":"float","level":"advanced","flags":1,"default_value":"0.5","min":"0.001","max":"0.999","enum_allowed":[],"desc":"aggressiveness of optimization","long_desc":".99 is very aggressive, .01 is less aggressive","tags":[],"see_also":[]},"end_time":{"name":"end_time","type":"str","level":"advanced","flags":1,"default_value":"2359","min":"","max":"","enum_allowed":[],"desc":"ending time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"end_weekday":{"name":"end_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"6","enum_allowed":[],"desc":"Restrict automatic balancing to days of the week earlier than this","long_desc":"0 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_score":{"name":"min_score","type":"float","level":"advanced","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"minimum score, below which no optimization is attempted","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":1,"default_value":"upmap","min":"","max":"","enum_allowed":["crush-compat","none","read","upmap","upmap-read"],"desc":"Balancer mode","long_desc":"","tags":[],"see_also":[]},"pool_ids":{"name":"pool_ids","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"pools which the automatic balancing will be limited to","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and attempt optimization","long_desc":"","tags":[],"see_also":[]},"upmap_max_deviation":{"name":"upmap_max_deviation","type":"int","level":"advanced","flags":1,"default_value":"5","min":"1","max":"","enum_allowed":[],"desc":"deviation below which no optimization is attempted","long_desc":"If the number of PGs are within this count then no optimization is attempted","tags":[],"see_also":[]},"upmap_max_optimizations":{"name":"upmap_max_optimizations","type":"uint","level":"advanced","flags":1,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"maximum upmap optimizations to make per attempt","long_desc":"","tags":[],"see_also":[]}}},{"name":"cephadm","can_run":true,"error_string":"","module_options":{"agent_down_multiplier":{"name":"agent_down_multiplier","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"","max":"","enum_allowed":[],"desc":"Multiplied by agent refresh rate to calculate how long agent must not report before being marked down","long_desc":"","tags":[],"see_also":[]},"agent_refresh_rate":{"name":"agent_refresh_rate","type":"secs","level":"advanced","flags":0,"default_value":"20","min":"","max":"","enum_allowed":[],"desc":"How often agent on each host will try to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"agent_starting_port":{"name":"agent_starting_port","type":"int","level":"advanced","flags":0,"default_value":"4721","min":"","max":"","enum_allowed":[],"desc":"First port agent will try to bind to (will also try up to next 1000 subsequent ports if blocked)","long_desc":"","tags":[],"see_also":[]},"allow_ptrace":{"name":"allow_ptrace","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow SYS_PTRACE capability on ceph containers","long_desc":"The SYS_PTRACE capability is needed to attach to a process with gdb or strace. Enabling this options can allow debugging daemons that encounter problems at runtime.","tags":[],"see_also":[]},"autotune_interval":{"name":"autotune_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to autotune daemon memory","long_desc":"","tags":[],"see_also":[]},"autotune_memory_target_ratio":{"name":"autotune_memory_target_ratio","type":"float","level":"advanced","flags":0,"default_value":"0.7","min":"","max":"","enum_allowed":[],"desc":"ratio of total system memory to divide amongst autotuned daemons","long_desc":"","tags":[],"see_also":[]},"cephadm_log_destination":{"name":"cephadm_log_destination","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":["file","file,syslog","syslog"],"desc":"Destination for cephadm command's persistent logging","long_desc":"","tags":[],"see_also":[]},"cgroups_split":{"name":"cgroups_split","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Pass --cgroups=split when cephadm creates containers (currently podman only)","long_desc":"","tags":[],"see_also":[]},"config_checks_enabled":{"name":"config_checks_enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable or disable the cephadm configuration analysis","long_desc":"","tags":[],"see_also":[]},"config_dashboard":{"name":"config_dashboard","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"manage configs like API endpoints in Dashboard.","long_desc":"","tags":[],"see_also":[]},"container_image_alertmanager":{"name":"container_image_alertmanager","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/alertmanager:v0.25.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_base":{"name":"container_image_base","type":"str","level":"advanced","flags":1,"default_value":"quay.io/ceph/ceph","min":"","max":"","enum_allowed":[],"desc":"Container image name, without the tag","long_desc":"","tags":[],"see_also":[]},"container_image_elasticsearch":{"name":"container_image_elasticsearch","type":"str","level":"advanced","flags":0,"default_value":"quay.io/omrizeneva/elasticsearch:6.8.23","min":"","max":"","enum_allowed":[],"desc":"elasticsearch container image","long_desc":"","tags":[],"see_also":[]},"container_image_grafana":{"name":"container_image_grafana","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/grafana:9.4.12","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_haproxy":{"name":"container_image_haproxy","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/haproxy:2.3","min":"","max":"","enum_allowed":[],"desc":"HAproxy container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_agent":{"name":"container_image_jaeger_agent","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-agent:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger agent container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_collector":{"name":"container_image_jaeger_collector","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-collector:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger collector container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_query":{"name":"container_image_jaeger_query","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-query:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger query container image","long_desc":"","tags":[],"see_also":[]},"container_image_keepalived":{"name":"container_image_keepalived","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/keepalived:2.2.4","min":"","max":"","enum_allowed":[],"desc":"Keepalived container image","long_desc":"","tags":[],"see_also":[]},"container_image_loki":{"name":"container_image_loki","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/loki:2.4.0","min":"","max":"","enum_allowed":[],"desc":"Loki container image","long_desc":"","tags":[],"see_also":[]},"container_image_node_exporter":{"name":"container_image_node_exporter","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/node-exporter:v1.5.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_nvmeof":{"name":"container_image_nvmeof","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/nvmeof:1.2.1","min":"","max":"","enum_allowed":[],"desc":"Nvme-of container image","long_desc":"","tags":[],"see_also":[]},"container_image_prometheus":{"name":"container_image_prometheus","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/prometheus:v2.43.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_promtail":{"name":"container_image_promtail","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/promtail:2.4.0","min":"","max":"","enum_allowed":[],"desc":"Promtail container image","long_desc":"","tags":[],"see_also":[]},"container_image_samba":{"name":"container_image_samba","type":"str","level":"advanced","flags":0,"default_value":"quay.io/samba.org/samba-server:devbuilds-centos-amd64","min":"","max":"","enum_allowed":[],"desc":"Samba/SMB container image","long_desc":"","tags":[],"see_also":[]},"container_image_snmp_gateway":{"name":"container_image_snmp_gateway","type":"str","level":"advanced","flags":0,"default_value":"docker.io/maxwo/snmp-notifier:v1.2.1","min":"","max":"","enum_allowed":[],"desc":"SNMP Gateway container image","long_desc":"","tags":[],"see_also":[]},"container_init":{"name":"container_init","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Run podman/docker with `--init`","long_desc":"","tags":[],"see_also":[]},"daemon_cache_timeout":{"name":"daemon_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"seconds to cache service (daemon) inventory","long_desc":"","tags":[],"see_also":[]},"default_cephadm_command_timeout":{"name":"default_cephadm_command_timeout","type":"int","level":"advanced","flags":0,"default_value":"900","min":"","max":"","enum_allowed":[],"desc":"Default timeout applied to cephadm commands run directly on the host (in seconds)","long_desc":"","tags":[],"see_also":[]},"default_registry":{"name":"default_registry","type":"str","level":"advanced","flags":0,"default_value":"docker.io","min":"","max":"","enum_allowed":[],"desc":"Search-registry to which we should normalize unqualified image names. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"device_cache_timeout":{"name":"device_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"1800","min":"","max":"","enum_allowed":[],"desc":"seconds to cache device inventory","long_desc":"","tags":[],"see_also":[]},"device_enhanced_scan":{"name":"device_enhanced_scan","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use libstoragemgmt during device scans","long_desc":"","tags":[],"see_also":[]},"facts_cache_timeout":{"name":"facts_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"seconds to cache host facts data","long_desc":"","tags":[],"see_also":[]},"grafana_dashboards_path":{"name":"grafana_dashboards_path","type":"str","level":"advanced","flags":0,"default_value":"/etc/grafana/dashboards/ceph-dashboard/","min":"","max":"","enum_allowed":[],"desc":"location of dashboards to include in grafana deployments","long_desc":"","tags":[],"see_also":[]},"host_check_interval":{"name":"host_check_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to perform a host check","long_desc":"","tags":[],"see_also":[]},"hw_monitoring":{"name":"hw_monitoring","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Deploy hw monitoring daemon on every host.","long_desc":"","tags":[],"see_also":[]},"inventory_list_all":{"name":"inventory_list_all","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Whether ceph-volume inventory should report more devices (mostly mappers (LVs / mpaths), partitions...)","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_refresh_metadata":{"name":"log_refresh_metadata","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Log all refresh metadata. Includes daemon, device, and host info collected regularly. Only has effect if logging at debug level","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"log to the \"cephadm\" cluster log channel\"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf":{"name":"manage_etc_ceph_ceph_conf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Manage and own /etc/ceph/ceph.conf on the hosts.","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf_hosts":{"name":"manage_etc_ceph_ceph_conf_hosts","type":"str","level":"advanced","flags":0,"default_value":"*","min":"","max":"","enum_allowed":[],"desc":"PlacementSpec describing on which hosts to manage /etc/ceph/ceph.conf","long_desc":"","tags":[],"see_also":[]},"max_count_per_host":{"name":"max_count_per_host","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of daemons per service per host","long_desc":"","tags":[],"see_also":[]},"max_osd_draining_count":{"name":"max_osd_draining_count","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of osds that will be drained simultaneously when osds are removed","long_desc":"","tags":[],"see_also":[]},"migration_current":{"name":"migration_current","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"internal - do not modify","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":0,"default_value":"root","min":"","max":"","enum_allowed":["cephadm-package","root"],"desc":"mode for remote execution of cephadm","long_desc":"","tags":[],"see_also":[]},"oob_default_addr":{"name":"oob_default_addr","type":"str","level":"advanced","flags":0,"default_value":"169.254.1.1","min":"","max":"","enum_allowed":[],"desc":"Default address for RedFish API (oob management).","long_desc":"","tags":[],"see_also":[]},"prometheus_alerts_path":{"name":"prometheus_alerts_path","type":"str","level":"advanced","flags":0,"default_value":"/etc/prometheus/ceph/ceph_default_alerts.yml","min":"","max":"","enum_allowed":[],"desc":"location of alerts to include in prometheus deployments","long_desc":"","tags":[],"see_also":[]},"registry_insecure":{"name":"registry_insecure","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Registry is to be considered insecure (no TLS available). Only for development purposes.","long_desc":"","tags":[],"see_also":[]},"registry_password":{"name":"registry_password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository password. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"registry_url":{"name":"registry_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Registry url for login purposes. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"registry_username":{"name":"registry_username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository username. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"secure_monitoring_stack":{"name":"secure_monitoring_stack","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable TLS security for all the monitoring stack daemons","long_desc":"","tags":[],"see_also":[]},"service_discovery_port":{"name":"service_discovery_port","type":"int","level":"advanced","flags":0,"default_value":"8765","min":"","max":"","enum_allowed":[],"desc":"cephadm service discovery port","long_desc":"","tags":[],"see_also":[]},"ssh_config_file":{"name":"ssh_config_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"customized SSH config file to connect to managed hosts","long_desc":"","tags":[],"see_also":[]},"use_agent":{"name":"use_agent","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use cephadm agent on each host to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"use_repo_digest":{"name":"use_repo_digest","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Automatically convert image tags to image digest. Make sure all daemons use the same image","long_desc":"","tags":[],"see_also":[]},"warn_on_failed_host_check":{"name":"warn_on_failed_host_check","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if the host check fails","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_daemons":{"name":"warn_on_stray_daemons","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected that are not managed by cephadm","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_hosts":{"name":"warn_on_stray_hosts","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected on a host that is not managed by cephadm","long_desc":"","tags":[],"see_also":[]}}},{"name":"crash","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"retain_interval":{"name":"retain_interval","type":"secs","level":"advanced","flags":1,"default_value":"31536000","min":"","max":"","enum_allowed":[],"desc":"how long to retain crashes before pruning them","long_desc":"","tags":[],"see_also":[]},"warn_recent_interval":{"name":"warn_recent_interval","type":"secs","level":"advanced","flags":1,"default_value":"1209600","min":"","max":"","enum_allowed":[],"desc":"time interval in which to warn about recent crashes","long_desc":"","tags":[],"see_also":[]}}},{"name":"dashboard","can_run":true,"error_string":"","module_options":{"ACCOUNT_LOCKOUT_ATTEMPTS":{"name":"ACCOUNT_LOCKOUT_ATTEMPTS","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_HOST":{"name":"ALERTMANAGER_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_SSL_VERIFY":{"name":"ALERTMANAGER_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_ENABLED":{"name":"AUDIT_API_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_LOG_PAYLOAD":{"name":"AUDIT_API_LOG_PAYLOAD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ENABLE_BROWSABLE_API":{"name":"ENABLE_BROWSABLE_API","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_CEPHFS":{"name":"FEATURE_TOGGLE_CEPHFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_DASHBOARD":{"name":"FEATURE_TOGGLE_DASHBOARD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_ISCSI":{"name":"FEATURE_TOGGLE_ISCSI","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_MIRRORING":{"name":"FEATURE_TOGGLE_MIRRORING","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_NFS":{"name":"FEATURE_TOGGLE_NFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RBD":{"name":"FEATURE_TOGGLE_RBD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RGW":{"name":"FEATURE_TOGGLE_RGW","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE":{"name":"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_PASSWORD":{"name":"GRAFANA_API_PASSWORD","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_SSL_VERIFY":{"name":"GRAFANA_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_URL":{"name":"GRAFANA_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_USERNAME":{"name":"GRAFANA_API_USERNAME","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_FRONTEND_API_URL":{"name":"GRAFANA_FRONTEND_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_UPDATE_DASHBOARDS":{"name":"GRAFANA_UPDATE_DASHBOARDS","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISCSI_API_SSL_VERIFICATION":{"name":"ISCSI_API_SSL_VERIFICATION","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISSUE_TRACKER_API_KEY":{"name":"ISSUE_TRACKER_API_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"MANAGED_BY_CLUSTERS":{"name":"MANAGED_BY_CLUSTERS","type":"str","level":"advanced","flags":0,"default_value":"[]","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"MULTICLUSTER_CONFIG":{"name":"MULTICLUSTER_CONFIG","type":"str","level":"advanced","flags":0,"default_value":"{}","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_HOST":{"name":"PROMETHEUS_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_SSL_VERIFY":{"name":"PROMETHEUS_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_COMPLEXITY_ENABLED":{"name":"PWD_POLICY_CHECK_COMPLEXITY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED":{"name":"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_LENGTH_ENABLED":{"name":"PWD_POLICY_CHECK_LENGTH_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_OLDPWD_ENABLED":{"name":"PWD_POLICY_CHECK_OLDPWD_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_USERNAME_ENABLED":{"name":"PWD_POLICY_CHECK_USERNAME_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_ENABLED":{"name":"PWD_POLICY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_EXCLUSION_LIST":{"name":"PWD_POLICY_EXCLUSION_LIST","type":"str","level":"advanced","flags":0,"default_value":"osd,host,dashboard,pool,block,nfs,ceph,monitors,gateway,logs,crush,maps","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_COMPLEXITY":{"name":"PWD_POLICY_MIN_COMPLEXITY","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_LENGTH":{"name":"PWD_POLICY_MIN_LENGTH","type":"int","level":"advanced","flags":0,"default_value":"8","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"REST_REQUESTS_TIMEOUT":{"name":"REST_REQUESTS_TIMEOUT","type":"int","level":"advanced","flags":0,"default_value":"45","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ACCESS_KEY":{"name":"RGW_API_ACCESS_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ADMIN_RESOURCE":{"name":"RGW_API_ADMIN_RESOURCE","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SECRET_KEY":{"name":"RGW_API_SECRET_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SSL_VERIFY":{"name":"RGW_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"UNSAFE_TLS_v1_2":{"name":"UNSAFE_TLS_v1_2","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_SPAN":{"name":"USER_PWD_EXPIRATION_SPAN","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_1":{"name":"USER_PWD_EXPIRATION_WARNING_1","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_2":{"name":"USER_PWD_EXPIRATION_WARNING_2","type":"int","level":"advanced","flags":0,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"cross_origin_url":{"name":"cross_origin_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"crt_file":{"name":"crt_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"debug":{"name":"debug","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable/disable debug options","long_desc":"","tags":[],"see_also":[]},"jwt_token_ttl":{"name":"jwt_token_ttl","type":"int","level":"advanced","flags":0,"default_value":"28800","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"motd":{"name":"motd","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"The message of the day","long_desc":"","tags":[],"see_also":[]},"redirect_resolve_ip_addr":{"name":"redirect_resolve_ip_addr","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":0,"default_value":"8080","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl_server_port":{"name":"ssl_server_port","type":"int","level":"advanced","flags":0,"default_value":"8443","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":0,"default_value":"redirect","min":"","max":"","enum_allowed":["error","redirect"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":0,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url_prefix":{"name":"url_prefix","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"devicehealth","can_run":true,"error_string":"","module_options":{"enable_monitoring":{"name":"enable_monitoring","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"monitor device health metrics","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mark_out_threshold":{"name":"mark_out_threshold","type":"secs","level":"advanced","flags":1,"default_value":"2419200","min":"","max":"","enum_allowed":[],"desc":"automatically mark OSD if it may fail before this long","long_desc":"","tags":[],"see_also":[]},"pool_name":{"name":"pool_name","type":"str","level":"advanced","flags":1,"default_value":"device_health_metrics","min":"","max":"","enum_allowed":[],"desc":"name of pool in which to store device health metrics","long_desc":"","tags":[],"see_also":[]},"retention_period":{"name":"retention_period","type":"secs","level":"advanced","flags":1,"default_value":"15552000","min":"","max":"","enum_allowed":[],"desc":"how long to retain device health metrics","long_desc":"","tags":[],"see_also":[]},"scrape_frequency":{"name":"scrape_frequency","type":"secs","level":"advanced","flags":1,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"how frequently to scrape device health metrics","long_desc":"","tags":[],"see_also":[]},"self_heal":{"name":"self_heal","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"preemptively heal cluster around devices that may fail","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and check device health","long_desc":"","tags":[],"see_also":[]},"warn_threshold":{"name":"warn_threshold","type":"secs","level":"advanced","flags":1,"default_value":"7257600","min":"","max":"","enum_allowed":[],"desc":"raise health warning if OSD may fail before this long","long_desc":"","tags":[],"see_also":[]}}},{"name":"diskprediction_local","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predict_interval":{"name":"predict_interval","type":"str","level":"advanced","flags":0,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predictor_model":{"name":"predictor_model","type":"str","level":"advanced","flags":0,"default_value":"prophetstor","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"str","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"influx","can_run":false,"error_string":"influxdb python module not found","module_options":{"batch_size":{"name":"batch_size","type":"int","level":"advanced","flags":0,"default_value":"5000","min":"","max":"","enum_allowed":[],"desc":"How big batches of data points should be when sending to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"database":{"name":"database","type":"str","level":"advanced","flags":0,"default_value":"ceph","min":"","max":"","enum_allowed":[],"desc":"InfluxDB database name. You will need to create this database and grant write privileges to the configured username or the username must have admin privileges to create it.","long_desc":"","tags":[],"see_also":[]},"hostname":{"name":"hostname","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server hostname","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"30","min":"5","max":"","enum_allowed":[],"desc":"Time between reports to InfluxDB. Default 30 seconds.","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"password":{"name":"password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"password of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"port":{"name":"port","type":"int","level":"advanced","flags":0,"default_value":"8086","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server port","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"str","level":"advanced","flags":0,"default_value":"false","min":"","max":"","enum_allowed":[],"desc":"Use https connection for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]},"threads":{"name":"threads","type":"int","level":"advanced","flags":0,"default_value":"5","min":"1","max":"32","enum_allowed":[],"desc":"How many worker threads should be spawned for sending data to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"username":{"name":"username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"username of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"verify_ssl":{"name":"verify_ssl","type":"str","level":"advanced","flags":0,"default_value":"true","min":"","max":"","enum_allowed":[],"desc":"Verify https cert for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]}}},{"name":"insights","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"iostat","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"k8sevents","can_run":true,"error_string":"","module_options":{"ceph_event_retention_days":{"name":"ceph_event_retention_days","type":"int","level":"advanced","flags":0,"default_value":"7","min":"","max":"","enum_allowed":[],"desc":"Days to hold ceph event information within local cache","long_desc":"","tags":[],"see_also":[]},"config_check_secs":{"name":"config_check_secs","type":"int","level":"advanced","flags":0,"default_value":"10","min":"10","max":"","enum_allowed":[],"desc":"interval (secs) to check for cluster configuration changes","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"localpool","can_run":true,"error_string":"","module_options":{"failure_domain":{"name":"failure_domain","type":"str","level":"advanced","flags":1,"default_value":"host","min":"","max":"","enum_allowed":[],"desc":"failure domain for any created local pool","long_desc":"what failure domain we should separate data replicas across.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_size":{"name":"min_size","type":"int","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"default min_size for any created local pool","long_desc":"value to set min_size to (unchanged from Ceph's default if this option is not set)","tags":[],"see_also":[]},"num_rep":{"name":"num_rep","type":"int","level":"advanced","flags":1,"default_value":"3","min":"","max":"","enum_allowed":[],"desc":"default replica count for any created local pool","long_desc":"","tags":[],"see_also":[]},"pg_num":{"name":"pg_num","type":"int","level":"advanced","flags":1,"default_value":"128","min":"","max":"","enum_allowed":[],"desc":"default pg_num for any created local pool","long_desc":"","tags":[],"see_also":[]},"prefix":{"name":"prefix","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"name prefix for any created local pool","long_desc":"","tags":[],"see_also":[]},"subtree":{"name":"subtree","type":"str","level":"advanced","flags":1,"default_value":"rack","min":"","max":"","enum_allowed":[],"desc":"CRUSH level for which to create a local pool","long_desc":"which CRUSH subtree type the module should create a pool for.","tags":[],"see_also":[]}}},{"name":"mds_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"mirroring","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"nfs","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"orchestrator","can_run":true,"error_string":"","module_options":{"fail_fs":{"name":"fail_fs","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Fail filesystem for rapid multi-rank mds upgrade","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"orchestrator":{"name":"orchestrator","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["cephadm","rook","test_orchestrator"],"desc":"Orchestrator backend","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_perf_query","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"pg_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"threshold":{"name":"threshold","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"1.0","max":"","enum_allowed":[],"desc":"scaling threshold","long_desc":"The factor by which the `NEW PG_NUM` must vary from the current`PG_NUM` before being accepted. Cannot be less than 1.0","tags":[],"see_also":[]}}},{"name":"progress","can_run":true,"error_string":"","module_options":{"allow_pg_recovery_event":{"name":"allow_pg_recovery_event","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow the module to show pg recovery progress","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_completed_events":{"name":"max_completed_events","type":"int","level":"advanced","flags":1,"default_value":"50","min":"","max":"","enum_allowed":[],"desc":"number of past completed events to remember","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"how long the module is going to sleep","long_desc":"","tags":[],"see_also":[]}}},{"name":"prometheus","can_run":true,"error_string":"","module_options":{"cache":{"name":"cache","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"exclude_perf_counters":{"name":"exclude_perf_counters","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Do not include perf-counters in the metrics output","long_desc":"Gathering perf-counters from a single Prometheus exporter can degrade ceph-mgr performance, especially in large clusters. Instead, Ceph-exporter daemons are now used by default for perf-counter gathering. This should only be disabled when no ceph-exporters are deployed.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools":{"name":"rbd_stats_pools","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools_refresh_interval":{"name":"rbd_stats_pools_refresh_interval","type":"int","level":"advanced","flags":0,"default_value":"300","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"scrape_interval":{"name":"scrape_interval","type":"float","level":"advanced","flags":0,"default_value":"15.0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"the IPv4 or IPv6 address on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":1,"default_value":"9283","min":"","max":"","enum_allowed":[],"desc":"the port on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"stale_cache_strategy":{"name":"stale_cache_strategy","type":"str","level":"advanced","flags":0,"default_value":"log","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":1,"default_value":"default","min":"","max":"","enum_allowed":["default","error"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":1,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rbd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_snap_create":{"name":"max_concurrent_snap_create","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mirror_snapshot_schedule":{"name":"mirror_snapshot_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"trash_purge_schedule":{"name":"trash_purge_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"restful","can_run":true,"error_string":"","module_options":{"enable_auth":{"name":"enable_auth","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rgw","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rook","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"storage_class":{"name":"storage_class","type":"str","level":"advanced","flags":0,"default_value":"local","min":"","max":"","enum_allowed":[],"desc":"storage class name for LSO-discovered PVs","long_desc":"","tags":[],"see_also":[]}}},{"name":"selftest","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption1":{"name":"roption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption2":{"name":"roption2","type":"str","level":"advanced","flags":0,"default_value":"xyz","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption1":{"name":"rwoption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption2":{"name":"rwoption2","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption3":{"name":"rwoption3","type":"float","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption4":{"name":"rwoption4","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption5":{"name":"rwoption5","type":"bool","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption6":{"name":"rwoption6","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption7":{"name":"rwoption7","type":"int","level":"advanced","flags":0,"default_value":"","min":"1","max":"42","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testkey":{"name":"testkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testlkey":{"name":"testlkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testnewline":{"name":"testnewline","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"snap_schedule","can_run":true,"error_string":"","module_options":{"allow_m_granularity":{"name":"allow_m_granularity","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow minute scheduled snapshots","long_desc":"","tags":[],"see_also":[]},"dump_on_update":{"name":"dump_on_update","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"dump database to debug log on update","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"stats","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"status","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telegraf","can_run":true,"error_string":"","module_options":{"address":{"name":"address","type":"str","level":"advanced","flags":0,"default_value":"unixgram:///tmp/telegraf.sock","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"15","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telemetry","can_run":true,"error_string":"","module_options":{"channel_basic":{"name":"channel_basic","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share basic cluster information (size, version)","long_desc":"","tags":[],"see_also":[]},"channel_crash":{"name":"channel_crash","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share metadata about Ceph daemon crashes (version, stack straces, etc)","long_desc":"","tags":[],"see_also":[]},"channel_device":{"name":"channel_device","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share device health metrics (e.g., SMART data, minus potentially identifying info like serial numbers)","long_desc":"","tags":[],"see_also":[]},"channel_ident":{"name":"channel_ident","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share a user-provided description and/or contact email for the cluster","long_desc":"","tags":[],"see_also":[]},"channel_perf":{"name":"channel_perf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share various performance metrics of a cluster","long_desc":"","tags":[],"see_also":[]},"contact":{"name":"contact","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"description":{"name":"description","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"device_url":{"name":"device_url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/device","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"int","level":"advanced","flags":0,"default_value":"24","min":"8","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"last_opt_revision":{"name":"last_opt_revision","type":"int","level":"advanced","flags":0,"default_value":"1","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard":{"name":"leaderboard","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard_description":{"name":"leaderboard_description","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"organization":{"name":"organization","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"proxy":{"name":"proxy","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url":{"name":"url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/report","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"test_orchestrator","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"volumes","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_clones":{"name":"max_concurrent_clones","type":"int","level":"advanced","flags":0,"default_value":"4","min":"","max":"","enum_allowed":[],"desc":"Number of asynchronous cloner threads","long_desc":"","tags":[],"see_also":[]},"periodic_async_work":{"name":"periodic_async_work","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Periodically check for async work","long_desc":"","tags":[],"see_also":[]},"snapshot_clone_delay":{"name":"snapshot_clone_delay","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"Delay clone begin operation by snapshot_clone_delay seconds","long_desc":"","tags":[],"see_also":[]},"snapshot_clone_no_wait":{"name":"snapshot_clone_no_wait","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Reject subvolume clone request when cloner threads are busy","long_desc":"","tags":[],"see_also":[]}}},{"name":"zabbix","can_run":true,"error_string":"","module_options":{"discovery_interval":{"name":"discovery_interval","type":"uint","level":"advanced","flags":0,"default_value":"100","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"identifier":{"name":"identifier","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_host":{"name":"zabbix_host","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_port":{"name":"zabbix_port","type":"int","level":"advanced","flags":0,"default_value":"10051","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_sender":{"name":"zabbix_sender","type":"str","level":"advanced","flags":0,"default_value":"/usr/bin/zabbix_sender","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}}]}],"modules":["cephadm","dashboard","iostat","nfs","restful"],"available_modules":[{"name":"alerts","can_run":true,"error_string":"","module_options":{"interval":{"name":"interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"How frequently to reexamine health status","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"smtp_destination":{"name":"smtp_destination","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Email address to send alerts to","long_desc":"","tags":[],"see_also":[]},"smtp_from_name":{"name":"smtp_from_name","type":"str","level":"advanced","flags":1,"default_value":"Ceph","min":"","max":"","enum_allowed":[],"desc":"Email From: name","long_desc":"","tags":[],"see_also":[]},"smtp_host":{"name":"smtp_host","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_password":{"name":"smtp_password","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Password to authenticate with","long_desc":"","tags":[],"see_also":[]},"smtp_port":{"name":"smtp_port","type":"int","level":"advanced","flags":1,"default_value":"465","min":"","max":"","enum_allowed":[],"desc":"SMTP port","long_desc":"","tags":[],"see_also":[]},"smtp_sender":{"name":"smtp_sender","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP envelope sender","long_desc":"","tags":[],"see_also":[]},"smtp_ssl":{"name":"smtp_ssl","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Use SSL to connect to SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_user":{"name":"smtp_user","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"User to authenticate as","long_desc":"","tags":[],"see_also":[]}}},{"name":"balancer","can_run":true,"error_string":"","module_options":{"active":{"name":"active","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"automatically balance PGs across cluster","long_desc":"","tags":[],"see_also":[]},"begin_time":{"name":"begin_time","type":"str","level":"advanced","flags":1,"default_value":"0000","min":"","max":"","enum_allowed":[],"desc":"beginning time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"begin_weekday":{"name":"begin_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"6","enum_allowed":[],"desc":"Restrict automatic balancing to this day of the week or later","long_desc":"0 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"crush_compat_max_iterations":{"name":"crush_compat_max_iterations","type":"uint","level":"advanced","flags":1,"default_value":"25","min":"1","max":"250","enum_allowed":[],"desc":"maximum number of iterations to attempt optimization","long_desc":"","tags":[],"see_also":[]},"crush_compat_metrics":{"name":"crush_compat_metrics","type":"str","level":"advanced","flags":1,"default_value":"pgs,objects,bytes","min":"","max":"","enum_allowed":[],"desc":"metrics with which to calculate OSD utilization","long_desc":"Value is a list of one or more of \"pgs\", \"objects\", or \"bytes\", and indicates which metrics to use to balance utilization.","tags":[],"see_also":[]},"crush_compat_step":{"name":"crush_compat_step","type":"float","level":"advanced","flags":1,"default_value":"0.5","min":"0.001","max":"0.999","enum_allowed":[],"desc":"aggressiveness of optimization","long_desc":".99 is very aggressive, .01 is less aggressive","tags":[],"see_also":[]},"end_time":{"name":"end_time","type":"str","level":"advanced","flags":1,"default_value":"2359","min":"","max":"","enum_allowed":[],"desc":"ending time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"end_weekday":{"name":"end_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"6","enum_allowed":[],"desc":"Restrict automatic balancing to days of the week earlier than this","long_desc":"0 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_score":{"name":"min_score","type":"float","level":"advanced","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"minimum score, below which no optimization is attempted","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":1,"default_value":"upmap","min":"","max":"","enum_allowed":["crush-compat","none","read","upmap","upmap-read"],"desc":"Balancer mode","long_desc":"","tags":[],"see_also":[]},"pool_ids":{"name":"pool_ids","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"pools which the automatic balancing will be limited to","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and attempt optimization","long_desc":"","tags":[],"see_also":[]},"upmap_max_deviation":{"name":"upmap_max_deviation","type":"int","level":"advanced","flags":1,"default_value":"5","min":"1","max":"","enum_allowed":[],"desc":"deviation below which no optimization is attempted","long_desc":"If the number of PGs are within this count then no optimization is attempted","tags":[],"see_also":[]},"upmap_max_optimizations":{"name":"upmap_max_optimizations","type":"uint","level":"advanced","flags":1,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"maximum upmap optimizations to make per attempt","long_desc":"","tags":[],"see_also":[]}}},{"name":"cephadm","can_run":true,"error_string":"","module_options":{"agent_down_multiplier":{"name":"agent_down_multiplier","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"","max":"","enum_allowed":[],"desc":"Multiplied by agent refresh rate to calculate how long agent must not report before being marked down","long_desc":"","tags":[],"see_also":[]},"agent_refresh_rate":{"name":"agent_refresh_rate","type":"secs","level":"advanced","flags":0,"default_value":"20","min":"","max":"","enum_allowed":[],"desc":"How often agent on each host will try to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"agent_starting_port":{"name":"agent_starting_port","type":"int","level":"advanced","flags":0,"default_value":"4721","min":"","max":"","enum_allowed":[],"desc":"First port agent will try to bind to (will also try up to next 1000 subsequent ports if blocked)","long_desc":"","tags":[],"see_also":[]},"allow_ptrace":{"name":"allow_ptrace","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow SYS_PTRACE capability on ceph containers","long_desc":"The SYS_PTRACE capability is needed to attach to a process with gdb or strace. Enabling this options can allow debugging daemons that encounter problems at runtime.","tags":[],"see_also":[]},"autotune_interval":{"name":"autotune_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to autotune daemon memory","long_desc":"","tags":[],"see_also":[]},"autotune_memory_target_ratio":{"name":"autotune_memory_target_ratio","type":"float","level":"advanced","flags":0,"default_value":"0.7","min":"","max":"","enum_allowed":[],"desc":"ratio of total system memory to divide amongst autotuned daemons","long_desc":"","tags":[],"see_also":[]},"cephadm_log_destination":{"name":"cephadm_log_destination","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":["file","file,syslog","syslog"],"desc":"Destination for cephadm command's persistent logging","long_desc":"","tags":[],"see_also":[]},"cgroups_split":{"name":"cgroups_split","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Pass --cgroups=split when cephadm creates containers (currently podman only)","long_desc":"","tags":[],"see_also":[]},"config_checks_enabled":{"name":"config_checks_enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable or disable the cephadm configuration analysis","long_desc":"","tags":[],"see_also":[]},"config_dashboard":{"name":"config_dashboard","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"manage configs like API endpoints in Dashboard.","long_desc":"","tags":[],"see_also":[]},"container_image_alertmanager":{"name":"container_image_alertmanager","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/alertmanager:v0.25.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_base":{"name":"container_image_base","type":"str","level":"advanced","flags":1,"default_value":"quay.io/ceph/ceph","min":"","max":"","enum_allowed":[],"desc":"Container image name, without the tag","long_desc":"","tags":[],"see_also":[]},"container_image_elasticsearch":{"name":"container_image_elasticsearch","type":"str","level":"advanced","flags":0,"default_value":"quay.io/omrizeneva/elasticsearch:6.8.23","min":"","max":"","enum_allowed":[],"desc":"elasticsearch container image","long_desc":"","tags":[],"see_also":[]},"container_image_grafana":{"name":"container_image_grafana","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/grafana:9.4.12","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_haproxy":{"name":"container_image_haproxy","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/haproxy:2.3","min":"","max":"","enum_allowed":[],"desc":"HAproxy container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_agent":{"name":"container_image_jaeger_agent","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-agent:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger agent container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_collector":{"name":"container_image_jaeger_collector","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-collector:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger collector container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_query":{"name":"container_image_jaeger_query","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-query:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger query container image","long_desc":"","tags":[],"see_also":[]},"container_image_keepalived":{"name":"container_image_keepalived","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/keepalived:2.2.4","min":"","max":"","enum_allowed":[],"desc":"Keepalived container image","long_desc":"","tags":[],"see_also":[]},"container_image_loki":{"name":"container_image_loki","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/loki:2.4.0","min":"","max":"","enum_allowed":[],"desc":"Loki container image","long_desc":"","tags":[],"see_also":[]},"container_image_node_exporter":{"name":"container_image_node_exporter","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/node-exporter:v1.5.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_nvmeof":{"name":"container_image_nvmeof","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/nvmeof:1.2.1","min":"","max":"","enum_allowed":[],"desc":"Nvme-of container image","long_desc":"","tags":[],"see_also":[]},"container_image_prometheus":{"name":"container_image_prometheus","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/prometheus:v2.43.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_promtail":{"name":"container_image_promtail","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/promtail:2.4.0","min":"","max":"","enum_allowed":[],"desc":"Promtail container image","long_desc":"","tags":[],"see_also":[]},"container_image_samba":{"name":"container_image_samba","type":"str","level":"advanced","flags":0,"default_value":"quay.io/samba.org/samba-server:devbuilds-centos-amd64","min":"","max":"","enum_allowed":[],"desc":"Samba/SMB container image","long_desc":"","tags":[],"see_also":[]},"container_image_snmp_gateway":{"name":"container_image_snmp_gateway","type":"str","level":"advanced","flags":0,"default_value":"docker.io/maxwo/snmp-notifier:v1.2.1","min":"","max":"","enum_allowed":[],"desc":"SNMP Gateway container image","long_desc":"","tags":[],"see_also":[]},"container_init":{"name":"container_init","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Run podman/docker with `--init`","long_desc":"","tags":[],"see_also":[]},"daemon_cache_timeout":{"name":"daemon_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"seconds to cache service (daemon) inventory","long_desc":"","tags":[],"see_also":[]},"default_cephadm_command_timeout":{"name":"default_cephadm_command_timeout","type":"int","level":"advanced","flags":0,"default_value":"900","min":"","max":"","enum_allowed":[],"desc":"Default timeout applied to cephadm commands run directly on the host (in seconds)","long_desc":"","tags":[],"see_also":[]},"default_registry":{"name":"default_registry","type":"str","level":"advanced","flags":0,"default_value":"docker.io","min":"","max":"","enum_allowed":[],"desc":"Search-registry to which we should normalize unqualified image names. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"device_cache_timeout":{"name":"device_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"1800","min":"","max":"","enum_allowed":[],"desc":"seconds to cache device inventory","long_desc":"","tags":[],"see_also":[]},"device_enhanced_scan":{"name":"device_enhanced_scan","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use libstoragemgmt during device scans","long_desc":"","tags":[],"see_also":[]},"facts_cache_timeout":{"name":"facts_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"seconds to cache host facts data","long_desc":"","tags":[],"see_also":[]},"grafana_dashboards_path":{"name":"grafana_dashboards_path","type":"str","level":"advanced","flags":0,"default_value":"/etc/grafana/dashboards/ceph-dashboard/","min":"","max":"","enum_allowed":[],"desc":"location of dashboards to include in grafana deployments","long_desc":"","tags":[],"see_also":[]},"host_check_interval":{"name":"host_check_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to perform a host check","long_desc":"","tags":[],"see_also":[]},"hw_monitoring":{"name":"hw_monitoring","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Deploy hw monitoring daemon on every host.","long_desc":"","tags":[],"see_also":[]},"inventory_list_all":{"name":"inventory_list_all","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Whether ceph-volume inventory should report more devices (mostly mappers (LVs / mpaths), partitions...)","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_refresh_metadata":{"name":"log_refresh_metadata","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Log all refresh metadata. Includes daemon, device, and host info collected regularly. Only has effect if logging at debug level","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"log to the \"cephadm\" cluster log channel\"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf":{"name":"manage_etc_ceph_ceph_conf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Manage and own /etc/ceph/ceph.conf on the hosts.","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf_hosts":{"name":"manage_etc_ceph_ceph_conf_hosts","type":"str","level":"advanced","flags":0,"default_value":"*","min":"","max":"","enum_allowed":[],"desc":"PlacementSpec describing on which hosts to manage /etc/ceph/ceph.conf","long_desc":"","tags":[],"see_also":[]},"max_count_per_host":{"name":"max_count_per_host","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of daemons per service per host","long_desc":"","tags":[],"see_also":[]},"max_osd_draining_count":{"name":"max_osd_draining_count","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of osds that will be drained simultaneously when osds are removed","long_desc":"","tags":[],"see_also":[]},"migration_current":{"name":"migration_current","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"internal - do not modify","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":0,"default_value":"root","min":"","max":"","enum_allowed":["cephadm-package","root"],"desc":"mode for remote execution of cephadm","long_desc":"","tags":[],"see_also":[]},"oob_default_addr":{"name":"oob_default_addr","type":"str","level":"advanced","flags":0,"default_value":"169.254.1.1","min":"","max":"","enum_allowed":[],"desc":"Default address for RedFish API (oob management).","long_desc":"","tags":[],"see_also":[]},"prometheus_alerts_path":{"name":"prometheus_alerts_path","type":"str","level":"advanced","flags":0,"default_value":"/etc/prometheus/ceph/ceph_default_alerts.yml","min":"","max":"","enum_allowed":[],"desc":"location of alerts to include in prometheus deployments","long_desc":"","tags":[],"see_also":[]},"registry_insecure":{"name":"registry_insecure","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Registry is to be considered insecure (no TLS available). Only for development purposes.","long_desc":"","tags":[],"see_also":[]},"registry_password":{"name":"registry_password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository password. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"registry_url":{"name":"registry_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Registry url for login purposes. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"registry_username":{"name":"registry_username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository username. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"secure_monitoring_stack":{"name":"secure_monitoring_stack","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable TLS security for all the monitoring stack daemons","long_desc":"","tags":[],"see_also":[]},"service_discovery_port":{"name":"service_discovery_port","type":"int","level":"advanced","flags":0,"default_value":"8765","min":"","max":"","enum_allowed":[],"desc":"cephadm service discovery port","long_desc":"","tags":[],"see_also":[]},"ssh_config_file":{"name":"ssh_config_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"customized SSH config file to connect to managed hosts","long_desc":"","tags":[],"see_also":[]},"use_agent":{"name":"use_agent","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use cephadm agent on each host to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"use_repo_digest":{"name":"use_repo_digest","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Automatically convert image tags to image digest. Make sure all daemons use the same image","long_desc":"","tags":[],"see_also":[]},"warn_on_failed_host_check":{"name":"warn_on_failed_host_check","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if the host check fails","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_daemons":{"name":"warn_on_stray_daemons","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected that are not managed by cephadm","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_hosts":{"name":"warn_on_stray_hosts","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected on a host that is not managed by cephadm","long_desc":"","tags":[],"see_also":[]}}},{"name":"crash","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"retain_interval":{"name":"retain_interval","type":"secs","level":"advanced","flags":1,"default_value":"31536000","min":"","max":"","enum_allowed":[],"desc":"how long to retain crashes before pruning them","long_desc":"","tags":[],"see_also":[]},"warn_recent_interval":{"name":"warn_recent_interval","type":"secs","level":"advanced","flags":1,"default_value":"1209600","min":"","max":"","enum_allowed":[],"desc":"time interval in which to warn about recent crashes","long_desc":"","tags":[],"see_also":[]}}},{"name":"dashboard","can_run":true,"error_string":"","module_options":{"ACCOUNT_LOCKOUT_ATTEMPTS":{"name":"ACCOUNT_LOCKOUT_ATTEMPTS","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_HOST":{"name":"ALERTMANAGER_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_SSL_VERIFY":{"name":"ALERTMANAGER_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_ENABLED":{"name":"AUDIT_API_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_LOG_PAYLOAD":{"name":"AUDIT_API_LOG_PAYLOAD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ENABLE_BROWSABLE_API":{"name":"ENABLE_BROWSABLE_API","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_CEPHFS":{"name":"FEATURE_TOGGLE_CEPHFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_DASHBOARD":{"name":"FEATURE_TOGGLE_DASHBOARD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_ISCSI":{"name":"FEATURE_TOGGLE_ISCSI","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_MIRRORING":{"name":"FEATURE_TOGGLE_MIRRORING","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_NFS":{"name":"FEATURE_TOGGLE_NFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RBD":{"name":"FEATURE_TOGGLE_RBD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RGW":{"name":"FEATURE_TOGGLE_RGW","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE":{"name":"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_PASSWORD":{"name":"GRAFANA_API_PASSWORD","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_SSL_VERIFY":{"name":"GRAFANA_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_URL":{"name":"GRAFANA_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_USERNAME":{"name":"GRAFANA_API_USERNAME","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_FRONTEND_API_URL":{"name":"GRAFANA_FRONTEND_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_UPDATE_DASHBOARDS":{"name":"GRAFANA_UPDATE_DASHBOARDS","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISCSI_API_SSL_VERIFICATION":{"name":"ISCSI_API_SSL_VERIFICATION","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISSUE_TRACKER_API_KEY":{"name":"ISSUE_TRACKER_API_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"MANAGED_BY_CLUSTERS":{"name":"MANAGED_BY_CLUSTERS","type":"str","level":"advanced","flags":0,"default_value":"[]","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"MULTICLUSTER_CONFIG":{"name":"MULTICLUSTER_CONFIG","type":"str","level":"advanced","flags":0,"default_value":"{}","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_HOST":{"name":"PROMETHEUS_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_SSL_VERIFY":{"name":"PROMETHEUS_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_COMPLEXITY_ENABLED":{"name":"PWD_POLICY_CHECK_COMPLEXITY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED":{"name":"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_LENGTH_ENABLED":{"name":"PWD_POLICY_CHECK_LENGTH_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_OLDPWD_ENABLED":{"name":"PWD_POLICY_CHECK_OLDPWD_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_USERNAME_ENABLED":{"name":"PWD_POLICY_CHECK_USERNAME_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_ENABLED":{"name":"PWD_POLICY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_EXCLUSION_LIST":{"name":"PWD_POLICY_EXCLUSION_LIST","type":"str","level":"advanced","flags":0,"default_value":"osd,host,dashboard,pool,block,nfs,ceph,monitors,gateway,logs,crush,maps","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_COMPLEXITY":{"name":"PWD_POLICY_MIN_COMPLEXITY","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_LENGTH":{"name":"PWD_POLICY_MIN_LENGTH","type":"int","level":"advanced","flags":0,"default_value":"8","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"REST_REQUESTS_TIMEOUT":{"name":"REST_REQUESTS_TIMEOUT","type":"int","level":"advanced","flags":0,"default_value":"45","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ACCESS_KEY":{"name":"RGW_API_ACCESS_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ADMIN_RESOURCE":{"name":"RGW_API_ADMIN_RESOURCE","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SECRET_KEY":{"name":"RGW_API_SECRET_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SSL_VERIFY":{"name":"RGW_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"UNSAFE_TLS_v1_2":{"name":"UNSAFE_TLS_v1_2","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_SPAN":{"name":"USER_PWD_EXPIRATION_SPAN","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_1":{"name":"USER_PWD_EXPIRATION_WARNING_1","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_2":{"name":"USER_PWD_EXPIRATION_WARNING_2","type":"int","level":"advanced","flags":0,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"cross_origin_url":{"name":"cross_origin_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"crt_file":{"name":"crt_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"debug":{"name":"debug","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable/disable debug options","long_desc":"","tags":[],"see_also":[]},"jwt_token_ttl":{"name":"jwt_token_ttl","type":"int","level":"advanced","flags":0,"default_value":"28800","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"motd":{"name":"motd","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"The message of the day","long_desc":"","tags":[],"see_also":[]},"redirect_resolve_ip_addr":{"name":"redirect_resolve_ip_addr","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":0,"default_value":"8080","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl_server_port":{"name":"ssl_server_port","type":"int","level":"advanced","flags":0,"default_value":"8443","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":0,"default_value":"redirect","min":"","max":"","enum_allowed":["error","redirect"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":0,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url_prefix":{"name":"url_prefix","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"devicehealth","can_run":true,"error_string":"","module_options":{"enable_monitoring":{"name":"enable_monitoring","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"monitor device health metrics","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mark_out_threshold":{"name":"mark_out_threshold","type":"secs","level":"advanced","flags":1,"default_value":"2419200","min":"","max":"","enum_allowed":[],"desc":"automatically mark OSD if it may fail before this long","long_desc":"","tags":[],"see_also":[]},"pool_name":{"name":"pool_name","type":"str","level":"advanced","flags":1,"default_value":"device_health_metrics","min":"","max":"","enum_allowed":[],"desc":"name of pool in which to store device health metrics","long_desc":"","tags":[],"see_also":[]},"retention_period":{"name":"retention_period","type":"secs","level":"advanced","flags":1,"default_value":"15552000","min":"","max":"","enum_allowed":[],"desc":"how long to retain device health metrics","long_desc":"","tags":[],"see_also":[]},"scrape_frequency":{"name":"scrape_frequency","type":"secs","level":"advanced","flags":1,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"how frequently to scrape device health metrics","long_desc":"","tags":[],"see_also":[]},"self_heal":{"name":"self_heal","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"preemptively heal cluster around devices that may fail","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and check device health","long_desc":"","tags":[],"see_also":[]},"warn_threshold":{"name":"warn_threshold","type":"secs","level":"advanced","flags":1,"default_value":"7257600","min":"","max":"","enum_allowed":[],"desc":"raise health warning if OSD may fail before this long","long_desc":"","tags":[],"see_also":[]}}},{"name":"diskprediction_local","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predict_interval":{"name":"predict_interval","type":"str","level":"advanced","flags":0,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predictor_model":{"name":"predictor_model","type":"str","level":"advanced","flags":0,"default_value":"prophetstor","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"str","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"influx","can_run":false,"error_string":"influxdb python module not found","module_options":{"batch_size":{"name":"batch_size","type":"int","level":"advanced","flags":0,"default_value":"5000","min":"","max":"","enum_allowed":[],"desc":"How big batches of data points should be when sending to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"database":{"name":"database","type":"str","level":"advanced","flags":0,"default_value":"ceph","min":"","max":"","enum_allowed":[],"desc":"InfluxDB database name. You will need to create this database and grant write privileges to the configured username or the username must have admin privileges to create it.","long_desc":"","tags":[],"see_also":[]},"hostname":{"name":"hostname","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server hostname","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"30","min":"5","max":"","enum_allowed":[],"desc":"Time between reports to InfluxDB. Default 30 seconds.","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"password":{"name":"password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"password of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"port":{"name":"port","type":"int","level":"advanced","flags":0,"default_value":"8086","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server port","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"str","level":"advanced","flags":0,"default_value":"false","min":"","max":"","enum_allowed":[],"desc":"Use https connection for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]},"threads":{"name":"threads","type":"int","level":"advanced","flags":0,"default_value":"5","min":"1","max":"32","enum_allowed":[],"desc":"How many worker threads should be spawned for sending data to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"username":{"name":"username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"username of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"verify_ssl":{"name":"verify_ssl","type":"str","level":"advanced","flags":0,"default_value":"true","min":"","max":"","enum_allowed":[],"desc":"Verify https cert for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]}}},{"name":"insights","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"iostat","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"k8sevents","can_run":true,"error_string":"","module_options":{"ceph_event_retention_days":{"name":"ceph_event_retention_days","type":"int","level":"advanced","flags":0,"default_value":"7","min":"","max":"","enum_allowed":[],"desc":"Days to hold ceph event information within local cache","long_desc":"","tags":[],"see_also":[]},"config_check_secs":{"name":"config_check_secs","type":"int","level":"advanced","flags":0,"default_value":"10","min":"10","max":"","enum_allowed":[],"desc":"interval (secs) to check for cluster configuration changes","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"localpool","can_run":true,"error_string":"","module_options":{"failure_domain":{"name":"failure_domain","type":"str","level":"advanced","flags":1,"default_value":"host","min":"","max":"","enum_allowed":[],"desc":"failure domain for any created local pool","long_desc":"what failure domain we should separate data replicas across.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_size":{"name":"min_size","type":"int","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"default min_size for any created local pool","long_desc":"value to set min_size to (unchanged from Ceph's default if this option is not set)","tags":[],"see_also":[]},"num_rep":{"name":"num_rep","type":"int","level":"advanced","flags":1,"default_value":"3","min":"","max":"","enum_allowed":[],"desc":"default replica count for any created local pool","long_desc":"","tags":[],"see_also":[]},"pg_num":{"name":"pg_num","type":"int","level":"advanced","flags":1,"default_value":"128","min":"","max":"","enum_allowed":[],"desc":"default pg_num for any created local pool","long_desc":"","tags":[],"see_also":[]},"prefix":{"name":"prefix","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"name prefix for any created local pool","long_desc":"","tags":[],"see_also":[]},"subtree":{"name":"subtree","type":"str","level":"advanced","flags":1,"default_value":"rack","min":"","max":"","enum_allowed":[],"desc":"CRUSH level for which to create a local pool","long_desc":"which CRUSH subtree type the module should create a pool for.","tags":[],"see_also":[]}}},{"name":"mds_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"mirroring","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"nfs","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"orchestrator","can_run":true,"error_string":"","module_options":{"fail_fs":{"name":"fail_fs","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Fail filesystem for rapid multi-rank mds upgrade","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"orchestrator":{"name":"orchestrator","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["cephadm","rook","test_orchestrator"],"desc":"Orchestrator backend","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_perf_query","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"pg_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"threshold":{"name":"threshold","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"1.0","max":"","enum_allowed":[],"desc":"scaling threshold","long_desc":"The factor by which the `NEW PG_NUM` must vary from the current`PG_NUM` before being accepted. Cannot be less than 1.0","tags":[],"see_also":[]}}},{"name":"progress","can_run":true,"error_string":"","module_options":{"allow_pg_recovery_event":{"name":"allow_pg_recovery_event","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow the module to show pg recovery progress","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_completed_events":{"name":"max_completed_events","type":"int","level":"advanced","flags":1,"default_value":"50","min":"","max":"","enum_allowed":[],"desc":"number of past completed events to remember","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"how long the module is going to sleep","long_desc":"","tags":[],"see_also":[]}}},{"name":"prometheus","can_run":true,"error_string":"","module_options":{"cache":{"name":"cache","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"exclude_perf_counters":{"name":"exclude_perf_counters","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Do not include perf-counters in the metrics output","long_desc":"Gathering perf-counters from a single Prometheus exporter can degrade ceph-mgr performance, especially in large clusters. Instead, Ceph-exporter daemons are now used by default for perf-counter gathering. This should only be disabled when no ceph-exporters are deployed.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools":{"name":"rbd_stats_pools","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools_refresh_interval":{"name":"rbd_stats_pools_refresh_interval","type":"int","level":"advanced","flags":0,"default_value":"300","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"scrape_interval":{"name":"scrape_interval","type":"float","level":"advanced","flags":0,"default_value":"15.0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"the IPv4 or IPv6 address on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":1,"default_value":"9283","min":"","max":"","enum_allowed":[],"desc":"the port on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"stale_cache_strategy":{"name":"stale_cache_strategy","type":"str","level":"advanced","flags":0,"default_value":"log","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":1,"default_value":"default","min":"","max":"","enum_allowed":["default","error"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":1,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rbd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_snap_create":{"name":"max_concurrent_snap_create","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mirror_snapshot_schedule":{"name":"mirror_snapshot_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"trash_purge_schedule":{"name":"trash_purge_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"restful","can_run":true,"error_string":"","module_options":{"enable_auth":{"name":"enable_auth","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rgw","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rook","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"storage_class":{"name":"storage_class","type":"str","level":"advanced","flags":0,"default_value":"local","min":"","max":"","enum_allowed":[],"desc":"storage class name for LSO-discovered PVs","long_desc":"","tags":[],"see_also":[]}}},{"name":"selftest","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption1":{"name":"roption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption2":{"name":"roption2","type":"str","level":"advanced","flags":0,"default_value":"xyz","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption1":{"name":"rwoption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption2":{"name":"rwoption2","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption3":{"name":"rwoption3","type":"float","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption4":{"name":"rwoption4","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption5":{"name":"rwoption5","type":"bool","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption6":{"name":"rwoption6","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption7":{"name":"rwoption7","type":"int","level":"advanced","flags":0,"default_value":"","min":"1","max":"42","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testkey":{"name":"testkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testlkey":{"name":"testlkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testnewline":{"name":"testnewline","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"snap_schedule","can_run":true,"error_string":"","module_options":{"allow_m_granularity":{"name":"allow_m_granularity","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow minute scheduled snapshots","long_desc":"","tags":[],"see_also":[]},"dump_on_update":{"name":"dump_on_update","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"dump database to debug log on update","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"stats","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"status","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telegraf","can_run":true,"error_string":"","module_options":{"address":{"name":"address","type":"str","level":"advanced","flags":0,"default_value":"unixgram:///tmp/telegraf.sock","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"15","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telemetry","can_run":true,"error_string":"","module_options":{"channel_basic":{"name":"channel_basic","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share basic cluster information (size, version)","long_desc":"","tags":[],"see_also":[]},"channel_crash":{"name":"channel_crash","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share metadata about Ceph daemon crashes (version, stack straces, etc)","long_desc":"","tags":[],"see_also":[]},"channel_device":{"name":"channel_device","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share device health metrics (e.g., SMART data, minus potentially identifying info like serial numbers)","long_desc":"","tags":[],"see_also":[]},"channel_ident":{"name":"channel_ident","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share a user-provided description and/or contact email for the cluster","long_desc":"","tags":[],"see_also":[]},"channel_perf":{"name":"channel_perf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share various performance metrics of a cluster","long_desc":"","tags":[],"see_also":[]},"contact":{"name":"contact","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"description":{"name":"description","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"device_url":{"name":"device_url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/device","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"int","level":"advanced","flags":0,"default_value":"24","min":"8","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"last_opt_revision":{"name":"last_opt_revision","type":"int","level":"advanced","flags":0,"default_value":"1","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard":{"name":"leaderboard","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard_description":{"name":"leaderboard_description","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"organization":{"name":"organization","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"proxy":{"name":"proxy","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url":{"name":"url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/report","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"test_orchestrator","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"volumes","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_clones":{"name":"max_concurrent_clones","type":"int","level":"advanced","flags":0,"default_value":"4","min":"","max":"","enum_allowed":[],"desc":"Number of asynchronous cloner threads","long_desc":"","tags":[],"see_also":[]},"periodic_async_work":{"name":"periodic_async_work","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Periodically check for async work","long_desc":"","tags":[],"see_also":[]},"snapshot_clone_delay":{"name":"snapshot_clone_delay","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"Delay clone begin operation by snapshot_clone_delay seconds","long_desc":"","tags":[],"see_also":[]},"snapshot_clone_no_wait":{"name":"snapshot_clone_no_wait","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Reject subvolume clone request when cloner threads are busy","long_desc":"","tags":[],"see_also":[]}}},{"name":"zabbix","can_run":true,"error_string":"","module_options":{"discovery_interval":{"name":"discovery_interval","type":"uint","level":"advanced","flags":0,"default_value":"100","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"identifier":{"name":"identifier","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_host":{"name":"zabbix_host","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_port":{"name":"zabbix_port","type":"int","level":"advanced","flags":0,"default_value":"10051","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_sender":{"name":"zabbix_sender","type":"str","level":"advanced","flags":0,"default_value":"/usr/bin/zabbix_sender","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}}],"services":{"dashboard":"https://172.21.15.114:8443/"},"always_on_modules":{"octopus":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"pacific":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"quincy":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"reef":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"squid":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"]},"last_failure_osd_epoch":3,"active_clients":[{"name":"devicehealth","addrvec":[{"type":"v2","addr":"172.21.15.114:0","nonce":480720790}]},{"name":"libcephsqlite","addrvec":[{"type":"v2","addr":"172.21.15.114:0","nonce":1787801099}]},{"name":"rbd_support","addrvec":[{"type":"v2","addr":"172.21.15.114:0","nonce":637004095}]},{"name":"volumes","addrvec":[{"type":"v2","addr":"172.21.15.114:0","nonce":2713201622}]},{"name":"volumes","addrvec":[{"type":"v2","addr":"172.21.15.114:0","nonce":2685210857}]}]} 2024-04-23T10:24:57.370 INFO:tasks.cephadm.ceph_manager.ceph:mgr available! 2024-04-23T10:24:57.370 INFO:tasks.cephadm.ceph_manager.ceph:waiting for all up 2024-04-23T10:24:57.371 DEBUG:teuthology.orchestra.run.smithi114:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d69f13d827952bb07bb9b7973bca14f97d764d6f shell --fsid aa240572-0159-11ef-bc93-c7b262605968 -- ceph osd dump --format=json 2024-04-23T10:24:57.492 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 23 10:24:57 smithi119 ceph-mon[32608]: pgmap v292: 97 pgs: 97 active+clean; 618 KiB data, 370 MiB used, 1.0 TiB / 1.0 TiB avail; 6.5 KiB/s rd, 7.5 KiB/s wr, 17 op/s 2024-04-23T10:24:57.492 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 23 10:24:57 smithi119 ceph-mon[32608]: from='client.? 172.21.15.114:0/3077385510' entity='client.admin' cmd=[{"prefix": "mgr dump", "format": "json"}]: dispatch 2024-04-23T10:24:57.609 INFO:teuthology.orchestra.run.smithi114.stderr:Inferring config /var/lib/ceph/aa240572-0159-11ef-bc93-c7b262605968/mon.a/config 2024-04-23T10:24:59.064 INFO:journalctl@ceph.mon.a.smithi114.stdout:Apr 23 10:24:59 smithi114 ceph-mon[29255]: pgmap v293: 97 pgs: 97 active+clean; 623 KiB data, 370 MiB used, 1.0 TiB / 1.0 TiB avail; 7.2 KiB/s rd, 7.4 KiB/s wr, 18 op/s 2024-04-23T10:24:59.083 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:24:59.084+0000 7fa3390fe700 1 -- 172.21.15.114:0/2137461507 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fa334103450 msgr2=0x7fa33410f8b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T10:24:59.083 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:24:59.084+0000 7fa3390fe700 1 --2- 172.21.15.114:0/2137461507 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fa334103450 0x7fa33410f8b0 secure :-1 s=READY pgs=246 cs=0 l=1 rev1=1 crypto rx=0x7fa32c00b0a0 tx=0x7fa32c00b3b0 comp rx=0 tx=0).stop 2024-04-23T10:24:59.083 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:24:59.084+0000 7fa3390fe700 1 -- 172.21.15.114:0/2137461507 shutdown_connections 2024-04-23T10:24:59.083 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:24:59.084+0000 7fa3390fe700 1 --2- 172.21.15.114:0/2137461507 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fa334108a50 0x7fa334108e30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:24:59.083 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:24:59.084+0000 7fa3390fe700 1 --2- 172.21.15.114:0/2137461507 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fa334103450 0x7fa33410f8b0 unknown :-1 s=CLOSED pgs=246 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:24:59.083 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:24:59.084+0000 7fa3390fe700 1 --2- 172.21.15.114:0/2137461507 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fa334102a90 0x7fa334102f10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:24:59.084 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:24:59.084+0000 7fa3390fe700 1 -- 172.21.15.114:0/2137461507 >> 172.21.15.114:0/2137461507 conn(0x7fa334079ba0 msgr2=0x7fa334079fb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T10:24:59.084 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:24:59.084+0000 7fa3390fe700 1 -- 172.21.15.114:0/2137461507 shutdown_connections 2024-04-23T10:24:59.084 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:24:59.084+0000 7fa3390fe700 1 -- 172.21.15.114:0/2137461507 wait complete. 2024-04-23T10:24:59.085 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:24:59.085+0000 7fa3390fe700 1 Processor -- start 2024-04-23T10:24:59.086 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:24:59.085+0000 7fa3390fe700 1 -- start start 2024-04-23T10:24:59.086 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:24:59.085+0000 7fa3390fe700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fa334102a90 0x7fa334199130 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T10:24:59.086 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:24:59.085+0000 7fa3390fe700 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fa334103450 0x7fa334199670 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T10:24:59.086 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:24:59.085+0000 7fa3390fe700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fa334108a50 0x7fa33419d9b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T10:24:59.086 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:24:59.085+0000 7fa3390fe700 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x7fa334112c20 con 0x7fa334102a90 2024-04-23T10:24:59.086 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:24:59.085+0000 7fa3390fe700 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 -- 0x7fa334112aa0 con 0x7fa334108a50 2024-04-23T10:24:59.086 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:24:59.085+0000 7fa3390fe700 1 -- --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_getmap magic: 0 -- 0x7fa334112da0 con 0x7fa334103450 2024-04-23T10:24:59.086 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:24:59.085+0000 7fa332d9d700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fa334102a90 0x7fa334199130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T10:24:59.086 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:24:59.085+0000 7fa332d9d700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fa334102a90 0x7fa334199130 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.114:36270/0 (socket says 172.21.15.114:36270) 2024-04-23T10:24:59.086 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:24:59.085+0000 7fa332d9d700 1 -- 172.21.15.114:0/209898835 learned_addr learned my addr 172.21.15.114:0/209898835 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-04-23T10:24:59.086 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:24:59.085+0000 7fa33259c700 1 --2- 172.21.15.114:0/209898835 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fa334103450 0x7fa334199670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T10:24:59.086 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:24:59.086+0000 7fa33359e700 1 --2- 172.21.15.114:0/209898835 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fa334108a50 0x7fa33419d9b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T10:24:59.086 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:24:59.086+0000 7fa332d9d700 1 -- 172.21.15.114:0/209898835 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fa334103450 msgr2=0x7fa334199670 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T10:24:59.087 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:24:59.086+0000 7fa332d9d700 1 --2- 172.21.15.114:0/209898835 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fa334103450 0x7fa334199670 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:24:59.087 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:24:59.086+0000 7fa332d9d700 1 -- 172.21.15.114:0/209898835 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fa334108a50 msgr2=0x7fa33419d9b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T10:24:59.087 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:24:59.086+0000 7fa332d9d700 1 --2- 172.21.15.114:0/209898835 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fa334108a50 0x7fa33419d9b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:24:59.087 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:24:59.086+0000 7fa332d9d700 1 -- 172.21.15.114:0/209898835 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa33419e0f0 con 0x7fa334102a90 2024-04-23T10:24:59.087 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:24:59.086+0000 7fa332d9d700 1 --2- 172.21.15.114:0/209898835 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fa334102a90 0x7fa334199130 secure :-1 s=READY pgs=247 cs=0 l=1 rev1=1 crypto rx=0x7fa32400d980 tx=0x7fa32400dc90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T10:24:59.087 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:24:59.086+0000 7fa31bfff700 1 -- 172.21.15.114:0/209898835 <== mon.0 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa324016070 con 0x7fa334102a90 2024-04-23T10:24:59.087 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:24:59.086+0000 7fa3390fe700 1 -- 172.21.15.114:0/209898835 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fa33419e3e0 con 0x7fa334102a90 2024-04-23T10:24:59.087 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:24:59.086+0000 7fa31bfff700 1 -- 172.21.15.114:0/209898835 <== mon.0 v2:172.21.15.114:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fa324004830 con 0x7fa334102a90 2024-04-23T10:24:59.087 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:24:59.086+0000 7fa31bfff700 1 -- 172.21.15.114:0/209898835 <== mon.0 v2:172.21.15.114:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa324005540 con 0x7fa334102a90 2024-04-23T10:24:59.087 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:24:59.086+0000 7fa3390fe700 1 -- 172.21.15.114:0/209898835 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fa3341a5c60 con 0x7fa334102a90 2024-04-23T10:24:59.088 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:24:59.087+0000 7fa3390fe700 1 -- 172.21.15.114:0/209898835 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fa334069730 con 0x7fa334102a90 2024-04-23T10:24:59.088 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:24:59.088+0000 7fa31bfff700 1 -- 172.21.15.114:0/209898835 <== mon.0 v2:172.21.15.114:3300/0 4 ==== mgrmap(e 17) ==== 138698+0+0 (secure 0 0 0) 0x7fa324004220 con 0x7fa334102a90 2024-04-23T10:24:59.088 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:24:59.089+0000 7fa31bfff700 1 --2- 172.21.15.114:0/209898835 >> [v2:172.21.15.114:6800/465333808,v1:172.21.15.114:6801/465333808] conn(0x7fa31c0a3cd0 0x7fa31c0a6190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T10:24:59.088 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:24:59.089+0000 7fa33259c700 1 --2- 172.21.15.114:0/209898835 >> [v2:172.21.15.114:6800/465333808,v1:172.21.15.114:6801/465333808] conn(0x7fa31c0a3cd0 0x7fa31c0a6190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T10:24:59.088 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:24:59.089+0000 7fa31bfff700 1 -- 172.21.15.114:0/209898835 <== mon.0 v2:172.21.15.114:3300/0 5 ==== osd_map(87..87 src has 1..87) ==== 7979+0+0 (secure 0 0 0) 0x7fa3240d01e0 con 0x7fa334102a90 2024-04-23T10:24:59.088 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:24:59.089+0000 7fa33259c700 1 --2- 172.21.15.114:0/209898835 >> [v2:172.21.15.114:6800/465333808,v1:172.21.15.114:6801/465333808] conn(0x7fa31c0a3cd0 0x7fa31c0a6190 secure :-1 s=READY pgs=182 cs=0 l=1 rev1=1 crypto rx=0x7fa33407a180 tx=0x7fa32800b3c0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T10:24:59.092 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:24:59.093+0000 7fa31bfff700 1 -- 172.21.15.114:0/209898835 <== mon.0 v2:172.21.15.114:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7fa32401f020 con 0x7fa334102a90 2024-04-23T10:24:59.291 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:24:59.292+0000 7fa3390fe700 1 -- 172.21.15.114:0/209898835 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7fa33402df80 con 0x7fa334102a90 2024-04-23T10:24:59.292 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:24:59.293+0000 7fa31bfff700 1 -- 172.21.15.114:0/209898835 <== mon.0 v2:172.21.15.114:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v87) ==== 74+0+22291 (secure 0 0 0) 0x7fa324022350 con 0x7fa334102a90 2024-04-23T10:24:59.293 INFO:teuthology.orchestra.run.smithi114.stdout: 2024-04-23T10:24:59.293 INFO:teuthology.orchestra.run.smithi114.stdout:{"epoch":87,"fsid":"aa240572-0159-11ef-bc93-c7b262605968","created":"2024-04-23T10:16:53.351136+0000","modified":"2024-04-23T10:24:45.032245+0000","last_up_change":"2024-04-23T10:23:54.153269+0000","last_in_change":"2024-04-23T10:23:32.963085+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":26,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-04-23T10:20:11.970297+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":12,"score_stable":12,"optimal_score":0.25,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-04-23T10:24:41.218540+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"86","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":2.25,"score_stable":2.25,"optimal_score":1,"raw_score_acting":2.25,"raw_score_stable":2.25,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-04-23T10:24:42.341799+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"87","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"fd64adde-9445-45a4-a171-ef656e2c5709","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":85,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6802","nonce":736366878},{"type":"v1","addr":"172.21.15.114:6803","nonce":736366878}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6804","nonce":736366878},{"type":"v1","addr":"172.21.15.114:6805","nonce":736366878}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6808","nonce":736366878},{"type":"v1","addr":"172.21.15.114:6809","nonce":736366878}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6806","nonce":736366878},{"type":"v1","addr":"172.21.15.114:6807","nonce":736366878}]},"public_addr":"172.21.15.114:6803/736366878","cluster_addr":"172.21.15.114:6805/736366878","heartbeat_back_addr":"172.21.15.114:6809/736366878","heartbeat_front_addr":"172.21.15.114:6807/736366878","state":["exists","up"]},{"osd":1,"uuid":"e279a1e6-2702-472a-930a-e08ece76a861","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":85,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6800","nonce":1133540779},{"type":"v1","addr":"172.21.15.119:6801","nonce":1133540779}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6802","nonce":1133540779},{"type":"v1","addr":"172.21.15.119:6803","nonce":1133540779}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6806","nonce":1133540779},{"type":"v1","addr":"172.21.15.119:6807","nonce":1133540779}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6804","nonce":1133540779},{"type":"v1","addr":"172.21.15.119:6805","nonce":1133540779}]},"public_addr":"172.21.15.119:6801/1133540779","cluster_addr":"172.21.15.119:6803/1133540779","heartbeat_back_addr":"172.21.15.119:6807/1133540779","heartbeat_front_addr":"172.21.15.119:6805/1133540779","state":["exists","up"]},{"osd":2,"uuid":"f4c9f2b4-ff49-4e14-89cd-b477feeb9382","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":85,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6800","nonce":799652511},{"type":"v1","addr":"172.21.15.204:6801","nonce":799652511}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6802","nonce":799652511},{"type":"v1","addr":"172.21.15.204:6803","nonce":799652511}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6806","nonce":799652511},{"type":"v1","addr":"172.21.15.204:6807","nonce":799652511}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6804","nonce":799652511},{"type":"v1","addr":"172.21.15.204:6805","nonce":799652511}]},"public_addr":"172.21.15.204:6801/799652511","cluster_addr":"172.21.15.204:6803/799652511","heartbeat_back_addr":"172.21.15.204:6807/799652511","heartbeat_front_addr":"172.21.15.204:6805/799652511","state":["exists","up"]},{"osd":3,"uuid":"52d2a566-e71c-4bb3-8d6c-5abdd43177c2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":25,"up_thru":85,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6810","nonce":4275595091},{"type":"v1","addr":"172.21.15.114:6811","nonce":4275595091}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6812","nonce":4275595091},{"type":"v1","addr":"172.21.15.114:6813","nonce":4275595091}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6816","nonce":4275595091},{"type":"v1","addr":"172.21.15.114:6817","nonce":4275595091}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6814","nonce":4275595091},{"type":"v1","addr":"172.21.15.114:6815","nonce":4275595091}]},"public_addr":"172.21.15.114:6811/4275595091","cluster_addr":"172.21.15.114:6813/4275595091","heartbeat_back_addr":"172.21.15.114:6817/4275595091","heartbeat_front_addr":"172.21.15.114:6815/4275595091","state":["exists","up"]},{"osd":4,"uuid":"d911a7c1-4bdc-4d28-b8a6-73896a6e7de6","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":33,"up_thru":85,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6808","nonce":1611007172},{"type":"v1","addr":"172.21.15.119:6809","nonce":1611007172}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6810","nonce":1611007172},{"type":"v1","addr":"172.21.15.119:6811","nonce":1611007172}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6814","nonce":1611007172},{"type":"v1","addr":"172.21.15.119:6815","nonce":1611007172}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6812","nonce":1611007172},{"type":"v1","addr":"172.21.15.119:6813","nonce":1611007172}]},"public_addr":"172.21.15.119:6809/1611007172","cluster_addr":"172.21.15.119:6811/1611007172","heartbeat_back_addr":"172.21.15.119:6815/1611007172","heartbeat_front_addr":"172.21.15.119:6813/1611007172","state":["exists","up"]},{"osd":5,"uuid":"f7590e1a-e93c-4c29-b15f-38ee52d1f921","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":39,"up_thru":85,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6808","nonce":1427563312},{"type":"v1","addr":"172.21.15.204:6809","nonce":1427563312}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6810","nonce":1427563312},{"type":"v1","addr":"172.21.15.204:6811","nonce":1427563312}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6814","nonce":1427563312},{"type":"v1","addr":"172.21.15.204:6815","nonce":1427563312}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6812","nonce":1427563312},{"type":"v1","addr":"172.21.15.204:6813","nonce":1427563312}]},"public_addr":"172.21.15.204:6809/1427563312","cluster_addr":"172.21.15.204:6811/1427563312","heartbeat_back_addr":"172.21.15.204:6815/1427563312","heartbeat_front_addr":"172.21.15.204:6813/1427563312","state":["exists","up"]},{"osd":6,"uuid":"534d20e8-3e54-4db8-a845-6119f49c7a78","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":47,"up_thru":85,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6818","nonce":4072912544},{"type":"v1","addr":"172.21.15.114:6819","nonce":4072912544}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6820","nonce":4072912544},{"type":"v1","addr":"172.21.15.114:6821","nonce":4072912544}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6824","nonce":4072912544},{"type":"v1","addr":"172.21.15.114:6825","nonce":4072912544}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6822","nonce":4072912544},{"type":"v1","addr":"172.21.15.114:6823","nonce":4072912544}]},"public_addr":"172.21.15.114:6819/4072912544","cluster_addr":"172.21.15.114:6821/4072912544","heartbeat_back_addr":"172.21.15.114:6825/4072912544","heartbeat_front_addr":"172.21.15.114:6823/4072912544","state":["exists","up"]},{"osd":7,"uuid":"f0ec22b8-1b48-4859-9d65-b00403aff3ad","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":53,"up_thru":85,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6816","nonce":2237458793},{"type":"v1","addr":"172.21.15.119:6817","nonce":2237458793}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6818","nonce":2237458793},{"type":"v1","addr":"172.21.15.119:6819","nonce":2237458793}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6822","nonce":2237458793},{"type":"v1","addr":"172.21.15.119:6823","nonce":2237458793}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6820","nonce":2237458793},{"type":"v1","addr":"172.21.15.119:6821","nonce":2237458793}]},"public_addr":"172.21.15.119:6817/2237458793","cluster_addr":"172.21.15.119:6819/2237458793","heartbeat_back_addr":"172.21.15.119:6823/2237458793","heartbeat_front_addr":"172.21.15.119:6821/2237458793","state":["exists","up"]},{"osd":8,"uuid":"3ac89686-6d76-4cef-9089-ac29e4b3bcac","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":62,"up_thru":85,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6816","nonce":1426318231},{"type":"v1","addr":"172.21.15.204:6817","nonce":1426318231}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6818","nonce":1426318231},{"type":"v1","addr":"172.21.15.204:6819","nonce":1426318231}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6822","nonce":1426318231},{"type":"v1","addr":"172.21.15.204:6823","nonce":1426318231}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6820","nonce":1426318231},{"type":"v1","addr":"172.21.15.204:6821","nonce":1426318231}]},"public_addr":"172.21.15.204:6817/1426318231","cluster_addr":"172.21.15.204:6819/1426318231","heartbeat_back_addr":"172.21.15.204:6823/1426318231","heartbeat_front_addr":"172.21.15.204:6821/1426318231","state":["exists","up"]},{"osd":9,"uuid":"a80ffe4d-1e98-460f-b046-c4754285cc5a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":69,"up_thru":85,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6826","nonce":2750635721},{"type":"v1","addr":"172.21.15.114:6827","nonce":2750635721}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6828","nonce":2750635721},{"type":"v1","addr":"172.21.15.114:6829","nonce":2750635721}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6832","nonce":2750635721},{"type":"v1","addr":"172.21.15.114:6833","nonce":2750635721}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6830","nonce":2750635721},{"type":"v1","addr":"172.21.15.114:6831","nonce":2750635721}]},"public_addr":"172.21.15.114:6827/2750635721","cluster_addr":"172.21.15.114:6829/2750635721","heartbeat_back_addr":"172.21.15.114:6833/2750635721","heartbeat_front_addr":"172.21.15.114:6831/2750635721","state":["exists","up"]},{"osd":10,"uuid":"cd75a579-bf12-4600-ac09-a8bfce772ced","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":74,"up_thru":85,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6824","nonce":1400187701},{"type":"v1","addr":"172.21.15.119:6825","nonce":1400187701}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6826","nonce":1400187701},{"type":"v1","addr":"172.21.15.119:6827","nonce":1400187701}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6830","nonce":1400187701},{"type":"v1","addr":"172.21.15.119:6831","nonce":1400187701}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6828","nonce":1400187701},{"type":"v1","addr":"172.21.15.119:6829","nonce":1400187701}]},"public_addr":"172.21.15.119:6825/1400187701","cluster_addr":"172.21.15.119:6827/1400187701","heartbeat_back_addr":"172.21.15.119:6831/1400187701","heartbeat_front_addr":"172.21.15.119:6829/1400187701","state":["exists","up"]},{"osd":11,"uuid":"53f037a5-3ec0-483a-80a1-dfee923a68e5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":82,"up_thru":85,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6824","nonce":2407298490},{"type":"v1","addr":"172.21.15.204:6825","nonce":2407298490}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6826","nonce":2407298490},{"type":"v1","addr":"172.21.15.204:6827","nonce":2407298490}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6830","nonce":2407298490},{"type":"v1","addr":"172.21.15.204:6831","nonce":2407298490}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6828","nonce":2407298490},{"type":"v1","addr":"172.21.15.204:6829","nonce":2407298490}]},"public_addr":"172.21.15.204:6825/2407298490","cluster_addr":"172.21.15.204:6827/2407298490","heartbeat_back_addr":"172.21.15.204:6831/2407298490","heartbeat_front_addr":"172.21.15.204:6829/2407298490","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-23T10:19:18.401147+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-23T10:19:42.611299+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-23T10:20:07.277367+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-23T10:20:31.208368+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-23T10:20:55.866225+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-23T10:21:19.961237+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-23T10:21:44.480783+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-23T10:22:09.173641+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-23T10:22:34.030328+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-23T10:22:59.652293+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-23T10:23:25.659673+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-23T10:23:51.168171+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.114:6801/1604678557":"2024-04-24T10:17:45.874609+0000","172.21.15.114:0/4025385854":"2024-04-24T10:17:45.874609+0000","172.21.15.114:0/2228273469":"2024-04-24T10:17:19.910348+0000","172.21.15.114:6800/2831530838":"2024-04-24T10:17:19.910348+0000","172.21.15.114:0/1333447450":"2024-04-24T10:17:45.874609+0000","172.21.15.114:0/2233026840":"2024-04-24T10:17:19.910348+0000","172.21.15.114:0/1902281545":"2024-04-24T10:17:19.910348+0000","172.21.15.114:6801/2831530838":"2024-04-24T10:17:19.910348+0000","172.21.15.114:6800/1604678557":"2024-04-24T10:17:45.874609+0000","172.21.15.114:0/1623553811":"2024-04-24T10:17:45.874609+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-23T10:24:59.295 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:24:59.296+0000 7fa3390fe700 1 -- 172.21.15.114:0/209898835 >> [v2:172.21.15.114:6800/465333808,v1:172.21.15.114:6801/465333808] conn(0x7fa31c0a3cd0 msgr2=0x7fa31c0a6190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T10:24:59.295 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:24:59.296+0000 7fa3390fe700 1 --2- 172.21.15.114:0/209898835 >> [v2:172.21.15.114:6800/465333808,v1:172.21.15.114:6801/465333808] conn(0x7fa31c0a3cd0 0x7fa31c0a6190 secure :-1 s=READY pgs=182 cs=0 l=1 rev1=1 crypto rx=0x7fa33407a180 tx=0x7fa32800b3c0 comp rx=0 tx=0).stop 2024-04-23T10:24:59.295 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:24:59.296+0000 7fa3390fe700 1 -- 172.21.15.114:0/209898835 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fa334102a90 msgr2=0x7fa334199130 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T10:24:59.295 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:24:59.296+0000 7fa3390fe700 1 --2- 172.21.15.114:0/209898835 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fa334102a90 0x7fa334199130 secure :-1 s=READY pgs=247 cs=0 l=1 rev1=1 crypto rx=0x7fa32400d980 tx=0x7fa32400dc90 comp rx=0 tx=0).stop 2024-04-23T10:24:59.295 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:24:59.296+0000 7fa3390fe700 1 -- 172.21.15.114:0/209898835 shutdown_connections 2024-04-23T10:24:59.295 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:24:59.296+0000 7fa3390fe700 1 --2- 172.21.15.114:0/209898835 >> [v2:172.21.15.114:6800/465333808,v1:172.21.15.114:6801/465333808] conn(0x7fa31c0a3cd0 0x7fa31c0a6190 unknown :-1 s=CLOSED pgs=182 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:24:59.295 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:24:59.296+0000 7fa3390fe700 1 --2- 172.21.15.114:0/209898835 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fa334108a50 0x7fa33419d9b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:24:59.295 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:24:59.296+0000 7fa3390fe700 1 --2- 172.21.15.114:0/209898835 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fa334102a90 0x7fa334199130 unknown :-1 s=CLOSED pgs=247 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:24:59.295 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:24:59.296+0000 7fa3390fe700 1 --2- 172.21.15.114:0/209898835 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fa334103450 0x7fa334199670 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:24:59.295 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:24:59.296+0000 7fa3390fe700 1 -- 172.21.15.114:0/209898835 >> 172.21.15.114:0/209898835 conn(0x7fa334079ba0 msgr2=0x7fa334101740 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T10:24:59.296 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:24:59.296+0000 7fa3390fe700 1 -- 172.21.15.114:0/209898835 shutdown_connections 2024-04-23T10:24:59.296 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:24:59.296+0000 7fa3390fe700 1 -- 172.21.15.114:0/209898835 wait complete. 2024-04-23T10:24:59.328 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 23 10:24:59 smithi204 ceph-mon[32649]: pgmap v293: 97 pgs: 97 active+clean; 623 KiB data, 370 MiB used, 1.0 TiB / 1.0 TiB avail; 7.2 KiB/s rd, 7.4 KiB/s wr, 18 op/s 2024-04-23T10:24:59.492 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 23 10:24:59 smithi119 ceph-mon[32608]: pgmap v293: 97 pgs: 97 active+clean; 623 KiB data, 370 MiB used, 1.0 TiB / 1.0 TiB avail; 7.2 KiB/s rd, 7.4 KiB/s wr, 18 op/s 2024-04-23T10:24:59.966 INFO:tasks.cephadm.ceph_manager.ceph:all up! 2024-04-23T10:24:59.967 DEBUG:teuthology.orchestra.run.smithi114:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d69f13d827952bb07bb9b7973bca14f97d764d6f shell --fsid aa240572-0159-11ef-bc93-c7b262605968 -- ceph osd dump --format=json 2024-04-23T10:25:00.212 INFO:teuthology.orchestra.run.smithi114.stderr:Inferring config /var/lib/ceph/aa240572-0159-11ef-bc93-c7b262605968/mon.a/config 2024-04-23T10:25:00.254 INFO:journalctl@ceph.mon.a.smithi114.stdout:Apr 23 10:25:00 smithi114 ceph-mon[29255]: from='client.? 172.21.15.114:0/209898835' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-23T10:25:00.328 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 23 10:25:00 smithi204 ceph-mon[32649]: from='client.? 172.21.15.114:0/209898835' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-23T10:25:00.492 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 23 10:25:00 smithi119 ceph-mon[32608]: from='client.? 172.21.15.114:0/209898835' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-23T10:25:01.328 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 23 10:25:01 smithi204 ceph-mon[32649]: pgmap v294: 97 pgs: 97 active+clean; 623 KiB data, 370 MiB used, 1.0 TiB / 1.0 TiB avail; 7.7 KiB/s rd, 5.7 KiB/s wr, 16 op/s 2024-04-23T10:25:01.333 INFO:journalctl@ceph.mon.a.smithi114.stdout:Apr 23 10:25:01 smithi114 ceph-mon[29255]: pgmap v294: 97 pgs: 97 active+clean; 623 KiB data, 370 MiB used, 1.0 TiB / 1.0 TiB avail; 7.7 KiB/s rd, 5.7 KiB/s wr, 16 op/s 2024-04-23T10:25:01.492 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 23 10:25:01 smithi119 ceph-mon[32608]: pgmap v294: 97 pgs: 97 active+clean; 623 KiB data, 370 MiB used, 1.0 TiB / 1.0 TiB avail; 7.7 KiB/s rd, 5.7 KiB/s wr, 16 op/s 2024-04-23T10:25:01.864 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:25:01.863+0000 7f734f8d3700 1 -- 172.21.15.114:0/1027669685 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f7348102b00 msgr2=0x7f7348102f80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T10:25:01.864 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:25:01.863+0000 7f734f8d3700 1 --2- 172.21.15.114:0/1027669685 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f7348102b00 0x7f7348102f80 secure :-1 s=READY pgs=248 cs=0 l=1 rev1=1 crypto rx=0x7f7338009990 tx=0x7f7338009ca0 comp rx=0 tx=0).stop 2024-04-23T10:25:01.864 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:25:01.863+0000 7f734f8d3700 1 -- 172.21.15.114:0/1027669685 shutdown_connections 2024-04-23T10:25:01.864 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:25:01.863+0000 7f734f8d3700 1 --2- 172.21.15.114:0/1027669685 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f7348108a10 0x7f7348108df0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:25:01.864 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:25:01.863+0000 7f734f8d3700 1 --2- 172.21.15.114:0/1027669685 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f73481034c0 0x7f734810f8e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:25:01.865 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:25:01.863+0000 7f734f8d3700 1 --2- 172.21.15.114:0/1027669685 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f7348102b00 0x7f7348102f80 unknown :-1 s=CLOSED pgs=248 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:25:01.865 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:25:01.863+0000 7f734f8d3700 1 -- 172.21.15.114:0/1027669685 >> 172.21.15.114:0/1027669685 conn(0x7f7348079c00 msgr2=0x7f734807a010 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T10:25:01.865 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:25:01.863+0000 7f734f8d3700 1 -- 172.21.15.114:0/1027669685 shutdown_connections 2024-04-23T10:25:01.865 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:25:01.863+0000 7f734f8d3700 1 -- 172.21.15.114:0/1027669685 wait complete. 2024-04-23T10:25:01.865 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:25:01.864+0000 7f734f8d3700 1 Processor -- start 2024-04-23T10:25:01.865 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:25:01.864+0000 7f734f8d3700 1 -- start start 2024-04-23T10:25:01.865 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:25:01.864+0000 7f734f8d3700 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f7348102b00 0x7f7348074d40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T10:25:01.865 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:25:01.864+0000 7f734f8d3700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f73481034c0 0x7f7348075280 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T10:25:01.865 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:25:01.864+0000 7f734f8d3700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f7348108a10 0x7f73480757c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T10:25:01.865 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:25:01.864+0000 7f734f8d3700 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x7f7348112cb0 con 0x7f73481034c0 2024-04-23T10:25:01.865 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:25:01.864+0000 7f734f8d3700 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 -- 0x7f7348112b30 con 0x7f7348108a10 2024-04-23T10:25:01.865 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:25:01.864+0000 7f734f8d3700 1 -- --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_getmap magic: 0 -- 0x7f7348112e30 con 0x7f7348102b00 2024-04-23T10:25:01.865 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:25:01.864+0000 7f734ce6e700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f73481034c0 0x7f7348075280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T10:25:01.865 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:25:01.864+0000 7f734ce6e700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f73481034c0 0x7f7348075280 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.114:36290/0 (socket says 172.21.15.114:36290) 2024-04-23T10:25:01.865 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:25:01.864+0000 7f734ce6e700 1 -- 172.21.15.114:0/3578709985 learned_addr learned my addr 172.21.15.114:0/3578709985 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-04-23T10:25:01.866 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:25:01.864+0000 7f734d66f700 1 --2- 172.21.15.114:0/3578709985 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f7348102b00 0x7f7348074d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T10:25:01.866 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:25:01.864+0000 7f734de70700 1 --2- 172.21.15.114:0/3578709985 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f7348108a10 0x7f73480757c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T10:25:01.866 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:25:01.864+0000 7f734ce6e700 1 -- 172.21.15.114:0/3578709985 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f7348102b00 msgr2=0x7f7348074d40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T10:25:01.866 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:25:01.865+0000 7f734ce6e700 1 --2- 172.21.15.114:0/3578709985 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f7348102b00 0x7f7348074d40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:25:01.866 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:25:01.865+0000 7f734ce6e700 1 -- 172.21.15.114:0/3578709985 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f7348108a10 msgr2=0x7f73480757c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T10:25:01.866 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:25:01.865+0000 7f734ce6e700 1 --2- 172.21.15.114:0/3578709985 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f7348108a10 0x7f73480757c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:25:01.866 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:25:01.865+0000 7f734ce6e700 1 -- 172.21.15.114:0/3578709985 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f73481a7790 con 0x7f73481034c0 2024-04-23T10:25:01.866 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:25:01.865+0000 7f734de70700 1 --2- 172.21.15.114:0/3578709985 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f7348108a10 0x7f73480757c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-23T10:25:01.866 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:25:01.865+0000 7f734d66f700 1 --2- 172.21.15.114:0/3578709985 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f7348102b00 0x7f7348074d40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-23T10:25:01.866 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:25:01.865+0000 7f734ce6e700 1 --2- 172.21.15.114:0/3578709985 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f73481034c0 0x7f7348075280 secure :-1 s=READY pgs=249 cs=0 l=1 rev1=1 crypto rx=0x7f7338000c00 tx=0x7f73380047b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T10:25:01.866 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:25:01.865+0000 7f733e7fc700 1 -- 172.21.15.114:0/3578709985 <== mon.0 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7338027070 con 0x7f73481034c0 2024-04-23T10:25:01.866 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:25:01.865+0000 7f733e7fc700 1 -- 172.21.15.114:0/3578709985 <== mon.0 v2:172.21.15.114:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f73380041d0 con 0x7f73481034c0 2024-04-23T10:25:01.866 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:25:01.865+0000 7f733e7fc700 1 -- 172.21.15.114:0/3578709985 <== mon.0 v2:172.21.15.114:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f733800f460 con 0x7f73481034c0 2024-04-23T10:25:01.866 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:25:01.865+0000 7f734f8d3700 1 -- 172.21.15.114:0/3578709985 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f73481a7a20 con 0x7f73481034c0 2024-04-23T10:25:01.866 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:25:01.865+0000 7f734f8d3700 1 -- 172.21.15.114:0/3578709985 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f73481a7e20 con 0x7f73481034c0 2024-04-23T10:25:01.867 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:25:01.866+0000 7f734f8d3700 1 -- 172.21.15.114:0/3578709985 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f7348069730 con 0x7f73481034c0 2024-04-23T10:25:01.867 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:25:01.867+0000 7f733e7fc700 1 -- 172.21.15.114:0/3578709985 <== mon.0 v2:172.21.15.114:3300/0 4 ==== mgrmap(e 17) ==== 138698+0+0 (secure 0 0 0) 0x7f733802a070 con 0x7f73481034c0 2024-04-23T10:25:01.867 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:25:01.867+0000 7f733e7fc700 1 --2- 172.21.15.114:0/3578709985 >> [v2:172.21.15.114:6800/465333808,v1:172.21.15.114:6801/465333808] conn(0x7f73340a3d50 0x7f73340a6210 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T10:25:01.867 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:25:01.868+0000 7f734d66f700 1 --2- 172.21.15.114:0/3578709985 >> [v2:172.21.15.114:6800/465333808,v1:172.21.15.114:6801/465333808] conn(0x7f73340a3d50 0x7f73340a6210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T10:25:01.867 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:25:01.868+0000 7f733e7fc700 1 -- 172.21.15.114:0/3578709985 <== mon.0 v2:172.21.15.114:3300/0 5 ==== osd_map(87..87 src has 1..87) ==== 7979+0+0 (secure 0 0 0) 0x7f73380d1770 con 0x7f73481034c0 2024-04-23T10:25:01.868 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:25:01.868+0000 7f734d66f700 1 --2- 172.21.15.114:0/3578709985 >> [v2:172.21.15.114:6800/465333808,v1:172.21.15.114:6801/465333808] conn(0x7f73340a3d50 0x7f73340a6210 secure :-1 s=READY pgs=183 cs=0 l=1 rev1=1 crypto rx=0x7f7344007900 tx=0x7f7344008040 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T10:25:01.870 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:25:01.871+0000 7f733e7fc700 1 -- 172.21.15.114:0/3578709985 <== mon.0 v2:172.21.15.114:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f7338031070 con 0x7f73481034c0 2024-04-23T10:25:02.074 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:25:02.075+0000 7f734f8d3700 1 -- 172.21.15.114:0/3578709985 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f734802df80 con 0x7f73481034c0 2024-04-23T10:25:02.075 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:25:02.076+0000 7f733e7fc700 1 -- 172.21.15.114:0/3578709985 <== mon.0 v2:172.21.15.114:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v87) ==== 74+0+22291 (secure 0 0 0) 0x7f7338013070 con 0x7f73481034c0 2024-04-23T10:25:02.075 INFO:teuthology.orchestra.run.smithi114.stdout: 2024-04-23T10:25:02.076 INFO:teuthology.orchestra.run.smithi114.stdout:{"epoch":87,"fsid":"aa240572-0159-11ef-bc93-c7b262605968","created":"2024-04-23T10:16:53.351136+0000","modified":"2024-04-23T10:24:45.032245+0000","last_up_change":"2024-04-23T10:23:54.153269+0000","last_in_change":"2024-04-23T10:23:32.963085+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":26,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-04-23T10:20:11.970297+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":12,"score_stable":12,"optimal_score":0.25,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-04-23T10:24:41.218540+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"86","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":2.25,"score_stable":2.25,"optimal_score":1,"raw_score_acting":2.25,"raw_score_stable":2.25,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-04-23T10:24:42.341799+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"87","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"fd64adde-9445-45a4-a171-ef656e2c5709","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":85,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6802","nonce":736366878},{"type":"v1","addr":"172.21.15.114:6803","nonce":736366878}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6804","nonce":736366878},{"type":"v1","addr":"172.21.15.114:6805","nonce":736366878}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6808","nonce":736366878},{"type":"v1","addr":"172.21.15.114:6809","nonce":736366878}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6806","nonce":736366878},{"type":"v1","addr":"172.21.15.114:6807","nonce":736366878}]},"public_addr":"172.21.15.114:6803/736366878","cluster_addr":"172.21.15.114:6805/736366878","heartbeat_back_addr":"172.21.15.114:6809/736366878","heartbeat_front_addr":"172.21.15.114:6807/736366878","state":["exists","up"]},{"osd":1,"uuid":"e279a1e6-2702-472a-930a-e08ece76a861","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":85,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6800","nonce":1133540779},{"type":"v1","addr":"172.21.15.119:6801","nonce":1133540779}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6802","nonce":1133540779},{"type":"v1","addr":"172.21.15.119:6803","nonce":1133540779}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6806","nonce":1133540779},{"type":"v1","addr":"172.21.15.119:6807","nonce":1133540779}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6804","nonce":1133540779},{"type":"v1","addr":"172.21.15.119:6805","nonce":1133540779}]},"public_addr":"172.21.15.119:6801/1133540779","cluster_addr":"172.21.15.119:6803/1133540779","heartbeat_back_addr":"172.21.15.119:6807/1133540779","heartbeat_front_addr":"172.21.15.119:6805/1133540779","state":["exists","up"]},{"osd":2,"uuid":"f4c9f2b4-ff49-4e14-89cd-b477feeb9382","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":85,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6800","nonce":799652511},{"type":"v1","addr":"172.21.15.204:6801","nonce":799652511}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6802","nonce":799652511},{"type":"v1","addr":"172.21.15.204:6803","nonce":799652511}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6806","nonce":799652511},{"type":"v1","addr":"172.21.15.204:6807","nonce":799652511}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6804","nonce":799652511},{"type":"v1","addr":"172.21.15.204:6805","nonce":799652511}]},"public_addr":"172.21.15.204:6801/799652511","cluster_addr":"172.21.15.204:6803/799652511","heartbeat_back_addr":"172.21.15.204:6807/799652511","heartbeat_front_addr":"172.21.15.204:6805/799652511","state":["exists","up"]},{"osd":3,"uuid":"52d2a566-e71c-4bb3-8d6c-5abdd43177c2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":25,"up_thru":85,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6810","nonce":4275595091},{"type":"v1","addr":"172.21.15.114:6811","nonce":4275595091}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6812","nonce":4275595091},{"type":"v1","addr":"172.21.15.114:6813","nonce":4275595091}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6816","nonce":4275595091},{"type":"v1","addr":"172.21.15.114:6817","nonce":4275595091}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6814","nonce":4275595091},{"type":"v1","addr":"172.21.15.114:6815","nonce":4275595091}]},"public_addr":"172.21.15.114:6811/4275595091","cluster_addr":"172.21.15.114:6813/4275595091","heartbeat_back_addr":"172.21.15.114:6817/4275595091","heartbeat_front_addr":"172.21.15.114:6815/4275595091","state":["exists","up"]},{"osd":4,"uuid":"d911a7c1-4bdc-4d28-b8a6-73896a6e7de6","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":33,"up_thru":85,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6808","nonce":1611007172},{"type":"v1","addr":"172.21.15.119:6809","nonce":1611007172}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6810","nonce":1611007172},{"type":"v1","addr":"172.21.15.119:6811","nonce":1611007172}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6814","nonce":1611007172},{"type":"v1","addr":"172.21.15.119:6815","nonce":1611007172}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6812","nonce":1611007172},{"type":"v1","addr":"172.21.15.119:6813","nonce":1611007172}]},"public_addr":"172.21.15.119:6809/1611007172","cluster_addr":"172.21.15.119:6811/1611007172","heartbeat_back_addr":"172.21.15.119:6815/1611007172","heartbeat_front_addr":"172.21.15.119:6813/1611007172","state":["exists","up"]},{"osd":5,"uuid":"f7590e1a-e93c-4c29-b15f-38ee52d1f921","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":39,"up_thru":85,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6808","nonce":1427563312},{"type":"v1","addr":"172.21.15.204:6809","nonce":1427563312}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6810","nonce":1427563312},{"type":"v1","addr":"172.21.15.204:6811","nonce":1427563312}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6814","nonce":1427563312},{"type":"v1","addr":"172.21.15.204:6815","nonce":1427563312}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6812","nonce":1427563312},{"type":"v1","addr":"172.21.15.204:6813","nonce":1427563312}]},"public_addr":"172.21.15.204:6809/1427563312","cluster_addr":"172.21.15.204:6811/1427563312","heartbeat_back_addr":"172.21.15.204:6815/1427563312","heartbeat_front_addr":"172.21.15.204:6813/1427563312","state":["exists","up"]},{"osd":6,"uuid":"534d20e8-3e54-4db8-a845-6119f49c7a78","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":47,"up_thru":85,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6818","nonce":4072912544},{"type":"v1","addr":"172.21.15.114:6819","nonce":4072912544}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6820","nonce":4072912544},{"type":"v1","addr":"172.21.15.114:6821","nonce":4072912544}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6824","nonce":4072912544},{"type":"v1","addr":"172.21.15.114:6825","nonce":4072912544}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6822","nonce":4072912544},{"type":"v1","addr":"172.21.15.114:6823","nonce":4072912544}]},"public_addr":"172.21.15.114:6819/4072912544","cluster_addr":"172.21.15.114:6821/4072912544","heartbeat_back_addr":"172.21.15.114:6825/4072912544","heartbeat_front_addr":"172.21.15.114:6823/4072912544","state":["exists","up"]},{"osd":7,"uuid":"f0ec22b8-1b48-4859-9d65-b00403aff3ad","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":53,"up_thru":85,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6816","nonce":2237458793},{"type":"v1","addr":"172.21.15.119:6817","nonce":2237458793}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6818","nonce":2237458793},{"type":"v1","addr":"172.21.15.119:6819","nonce":2237458793}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6822","nonce":2237458793},{"type":"v1","addr":"172.21.15.119:6823","nonce":2237458793}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6820","nonce":2237458793},{"type":"v1","addr":"172.21.15.119:6821","nonce":2237458793}]},"public_addr":"172.21.15.119:6817/2237458793","cluster_addr":"172.21.15.119:6819/2237458793","heartbeat_back_addr":"172.21.15.119:6823/2237458793","heartbeat_front_addr":"172.21.15.119:6821/2237458793","state":["exists","up"]},{"osd":8,"uuid":"3ac89686-6d76-4cef-9089-ac29e4b3bcac","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":62,"up_thru":85,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6816","nonce":1426318231},{"type":"v1","addr":"172.21.15.204:6817","nonce":1426318231}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6818","nonce":1426318231},{"type":"v1","addr":"172.21.15.204:6819","nonce":1426318231}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6822","nonce":1426318231},{"type":"v1","addr":"172.21.15.204:6823","nonce":1426318231}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6820","nonce":1426318231},{"type":"v1","addr":"172.21.15.204:6821","nonce":1426318231}]},"public_addr":"172.21.15.204:6817/1426318231","cluster_addr":"172.21.15.204:6819/1426318231","heartbeat_back_addr":"172.21.15.204:6823/1426318231","heartbeat_front_addr":"172.21.15.204:6821/1426318231","state":["exists","up"]},{"osd":9,"uuid":"a80ffe4d-1e98-460f-b046-c4754285cc5a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":69,"up_thru":85,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6826","nonce":2750635721},{"type":"v1","addr":"172.21.15.114:6827","nonce":2750635721}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6828","nonce":2750635721},{"type":"v1","addr":"172.21.15.114:6829","nonce":2750635721}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6832","nonce":2750635721},{"type":"v1","addr":"172.21.15.114:6833","nonce":2750635721}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6830","nonce":2750635721},{"type":"v1","addr":"172.21.15.114:6831","nonce":2750635721}]},"public_addr":"172.21.15.114:6827/2750635721","cluster_addr":"172.21.15.114:6829/2750635721","heartbeat_back_addr":"172.21.15.114:6833/2750635721","heartbeat_front_addr":"172.21.15.114:6831/2750635721","state":["exists","up"]},{"osd":10,"uuid":"cd75a579-bf12-4600-ac09-a8bfce772ced","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":74,"up_thru":85,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6824","nonce":1400187701},{"type":"v1","addr":"172.21.15.119:6825","nonce":1400187701}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6826","nonce":1400187701},{"type":"v1","addr":"172.21.15.119:6827","nonce":1400187701}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6830","nonce":1400187701},{"type":"v1","addr":"172.21.15.119:6831","nonce":1400187701}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6828","nonce":1400187701},{"type":"v1","addr":"172.21.15.119:6829","nonce":1400187701}]},"public_addr":"172.21.15.119:6825/1400187701","cluster_addr":"172.21.15.119:6827/1400187701","heartbeat_back_addr":"172.21.15.119:6831/1400187701","heartbeat_front_addr":"172.21.15.119:6829/1400187701","state":["exists","up"]},{"osd":11,"uuid":"53f037a5-3ec0-483a-80a1-dfee923a68e5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":82,"up_thru":85,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6824","nonce":2407298490},{"type":"v1","addr":"172.21.15.204:6825","nonce":2407298490}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6826","nonce":2407298490},{"type":"v1","addr":"172.21.15.204:6827","nonce":2407298490}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6830","nonce":2407298490},{"type":"v1","addr":"172.21.15.204:6831","nonce":2407298490}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6828","nonce":2407298490},{"type":"v1","addr":"172.21.15.204:6829","nonce":2407298490}]},"public_addr":"172.21.15.204:6825/2407298490","cluster_addr":"172.21.15.204:6827/2407298490","heartbeat_back_addr":"172.21.15.204:6831/2407298490","heartbeat_front_addr":"172.21.15.204:6829/2407298490","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-23T10:19:18.401147+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-23T10:19:42.611299+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-23T10:20:07.277367+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-23T10:20:31.208368+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-23T10:20:55.866225+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-23T10:21:19.961237+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-23T10:21:44.480783+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-23T10:22:09.173641+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-23T10:22:34.030328+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-23T10:22:59.652293+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-23T10:23:25.659673+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-23T10:23:51.168171+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.114:6801/1604678557":"2024-04-24T10:17:45.874609+0000","172.21.15.114:0/4025385854":"2024-04-24T10:17:45.874609+0000","172.21.15.114:0/2228273469":"2024-04-24T10:17:19.910348+0000","172.21.15.114:6800/2831530838":"2024-04-24T10:17:19.910348+0000","172.21.15.114:0/1333447450":"2024-04-24T10:17:45.874609+0000","172.21.15.114:0/2233026840":"2024-04-24T10:17:19.910348+0000","172.21.15.114:0/1902281545":"2024-04-24T10:17:19.910348+0000","172.21.15.114:6801/2831530838":"2024-04-24T10:17:19.910348+0000","172.21.15.114:6800/1604678557":"2024-04-24T10:17:45.874609+0000","172.21.15.114:0/1623553811":"2024-04-24T10:17:45.874609+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-23T10:25:02.078 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:25:02.078+0000 7f734f8d3700 1 -- 172.21.15.114:0/3578709985 >> [v2:172.21.15.114:6800/465333808,v1:172.21.15.114:6801/465333808] conn(0x7f73340a3d50 msgr2=0x7f73340a6210 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T10:25:02.078 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:25:02.078+0000 7f734f8d3700 1 --2- 172.21.15.114:0/3578709985 >> [v2:172.21.15.114:6800/465333808,v1:172.21.15.114:6801/465333808] conn(0x7f73340a3d50 0x7f73340a6210 secure :-1 s=READY pgs=183 cs=0 l=1 rev1=1 crypto rx=0x7f7344007900 tx=0x7f7344008040 comp rx=0 tx=0).stop 2024-04-23T10:25:02.078 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:25:02.078+0000 7f734f8d3700 1 -- 172.21.15.114:0/3578709985 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f73481034c0 msgr2=0x7f7348075280 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T10:25:02.078 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:25:02.078+0000 7f734f8d3700 1 --2- 172.21.15.114:0/3578709985 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f73481034c0 0x7f7348075280 secure :-1 s=READY pgs=249 cs=0 l=1 rev1=1 crypto rx=0x7f7338000c00 tx=0x7f73380047b0 comp rx=0 tx=0).stop 2024-04-23T10:25:02.078 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:25:02.079+0000 7f734f8d3700 1 -- 172.21.15.114:0/3578709985 shutdown_connections 2024-04-23T10:25:02.078 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:25:02.079+0000 7f734f8d3700 1 --2- 172.21.15.114:0/3578709985 >> [v2:172.21.15.114:6800/465333808,v1:172.21.15.114:6801/465333808] conn(0x7f73340a3d50 0x7f73340a6210 unknown :-1 s=CLOSED pgs=183 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:25:02.078 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:25:02.079+0000 7f734f8d3700 1 --2- 172.21.15.114:0/3578709985 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f7348108a10 0x7f73480757c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:25:02.078 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:25:02.079+0000 7f734f8d3700 1 --2- 172.21.15.114:0/3578709985 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f7348102b00 0x7f7348074d40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:25:02.078 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:25:02.079+0000 7f734f8d3700 1 --2- 172.21.15.114:0/3578709985 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f73481034c0 0x7f7348075280 unknown :-1 s=CLOSED pgs=249 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:25:02.078 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:25:02.079+0000 7f734f8d3700 1 -- 172.21.15.114:0/3578709985 >> 172.21.15.114:0/3578709985 conn(0x7f7348079c00 msgr2=0x7f7348100c50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T10:25:02.078 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:25:02.079+0000 7f734f8d3700 1 -- 172.21.15.114:0/3578709985 shutdown_connections 2024-04-23T10:25:02.079 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:25:02.079+0000 7f734f8d3700 1 -- 172.21.15.114:0/3578709985 wait complete. 2024-04-23T10:25:02.745 DEBUG:teuthology.orchestra.run.smithi114:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d69f13d827952bb07bb9b7973bca14f97d764d6f shell --fsid aa240572-0159-11ef-bc93-c7b262605968 -- ceph tell osd.0 flush_pg_stats 2024-04-23T10:25:02.746 DEBUG:teuthology.orchestra.run.smithi114:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d69f13d827952bb07bb9b7973bca14f97d764d6f shell --fsid aa240572-0159-11ef-bc93-c7b262605968 -- ceph tell osd.1 flush_pg_stats 2024-04-23T10:25:02.746 DEBUG:teuthology.orchestra.run.smithi114:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d69f13d827952bb07bb9b7973bca14f97d764d6f shell --fsid aa240572-0159-11ef-bc93-c7b262605968 -- ceph tell osd.2 flush_pg_stats 2024-04-23T10:25:02.747 DEBUG:teuthology.orchestra.run.smithi114:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d69f13d827952bb07bb9b7973bca14f97d764d6f shell --fsid aa240572-0159-11ef-bc93-c7b262605968 -- ceph tell osd.3 flush_pg_stats 2024-04-23T10:25:02.747 DEBUG:teuthology.orchestra.run.smithi114:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d69f13d827952bb07bb9b7973bca14f97d764d6f shell --fsid aa240572-0159-11ef-bc93-c7b262605968 -- ceph tell osd.4 flush_pg_stats 2024-04-23T10:25:02.748 DEBUG:teuthology.orchestra.run.smithi114:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d69f13d827952bb07bb9b7973bca14f97d764d6f shell --fsid aa240572-0159-11ef-bc93-c7b262605968 -- ceph tell osd.5 flush_pg_stats 2024-04-23T10:25:02.748 DEBUG:teuthology.orchestra.run.smithi114:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d69f13d827952bb07bb9b7973bca14f97d764d6f shell --fsid aa240572-0159-11ef-bc93-c7b262605968 -- ceph tell osd.6 flush_pg_stats 2024-04-23T10:25:02.749 DEBUG:teuthology.orchestra.run.smithi114:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d69f13d827952bb07bb9b7973bca14f97d764d6f shell --fsid aa240572-0159-11ef-bc93-c7b262605968 -- ceph tell osd.7 flush_pg_stats 2024-04-23T10:25:02.749 DEBUG:teuthology.orchestra.run.smithi114:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d69f13d827952bb07bb9b7973bca14f97d764d6f shell --fsid aa240572-0159-11ef-bc93-c7b262605968 -- ceph tell osd.8 flush_pg_stats 2024-04-23T10:25:02.750 DEBUG:teuthology.orchestra.run.smithi114:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d69f13d827952bb07bb9b7973bca14f97d764d6f shell --fsid aa240572-0159-11ef-bc93-c7b262605968 -- ceph tell osd.9 flush_pg_stats 2024-04-23T10:25:02.750 DEBUG:teuthology.orchestra.run.smithi114:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d69f13d827952bb07bb9b7973bca14f97d764d6f shell --fsid aa240572-0159-11ef-bc93-c7b262605968 -- ceph tell osd.10 flush_pg_stats 2024-04-23T10:25:02.751 DEBUG:teuthology.orchestra.run.smithi114:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d69f13d827952bb07bb9b7973bca14f97d764d6f shell --fsid aa240572-0159-11ef-bc93-c7b262605968 -- ceph tell osd.11 flush_pg_stats 2024-04-23T10:25:03.328 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 23 10:25:03 smithi204 ceph-mon[32649]: pgmap v295: 97 pgs: 97 active+clean; 623 KiB data, 370 MiB used, 1.0 TiB / 1.0 TiB avail; 7.3 KiB/s rd, 4.2 KiB/s wr, 10 op/s 2024-04-23T10:25:03.328 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 23 10:25:03 smithi204 ceph-mon[32649]: from='client.? 172.21.15.114:0/3578709985' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-23T10:25:03.492 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 23 10:25:03 smithi119 ceph-mon[32608]: pgmap v295: 97 pgs: 97 active+clean; 623 KiB data, 370 MiB used, 1.0 TiB / 1.0 TiB avail; 7.3 KiB/s rd, 4.2 KiB/s wr, 10 op/s 2024-04-23T10:25:03.492 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 23 10:25:03 smithi119 ceph-mon[32608]: from='client.? 172.21.15.114:0/3578709985' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-23T10:25:03.504 INFO:journalctl@ceph.mon.a.smithi114.stdout:Apr 23 10:25:03 smithi114 ceph-mon[29255]: pgmap v295: 97 pgs: 97 active+clean; 623 KiB data, 370 MiB used, 1.0 TiB / 1.0 TiB avail; 7.3 KiB/s rd, 4.2 KiB/s wr, 10 op/s 2024-04-23T10:25:03.504 INFO:journalctl@ceph.mon.a.smithi114.stdout:Apr 23 10:25:03 smithi114 ceph-mon[29255]: from='client.? 172.21.15.114:0/3578709985' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-23T10:25:03.572 INFO:teuthology.orchestra.run.smithi114.stderr:Inferring config /var/lib/ceph/aa240572-0159-11ef-bc93-c7b262605968/mon.a/config 2024-04-23T10:25:03.573 INFO:teuthology.orchestra.run.smithi114.stderr:Inferring config /var/lib/ceph/aa240572-0159-11ef-bc93-c7b262605968/mon.a/config 2024-04-23T10:25:03.574 INFO:teuthology.orchestra.run.smithi114.stderr:Inferring config /var/lib/ceph/aa240572-0159-11ef-bc93-c7b262605968/mon.a/config 2024-04-23T10:25:03.574 INFO:teuthology.orchestra.run.smithi114.stderr:Inferring config /var/lib/ceph/aa240572-0159-11ef-bc93-c7b262605968/mon.a/config 2024-04-23T10:25:03.575 INFO:teuthology.orchestra.run.smithi114.stderr:Inferring config /var/lib/ceph/aa240572-0159-11ef-bc93-c7b262605968/mon.a/config 2024-04-23T10:25:03.576 INFO:teuthology.orchestra.run.smithi114.stderr:Inferring config /var/lib/ceph/aa240572-0159-11ef-bc93-c7b262605968/mon.a/config 2024-04-23T10:25:03.576 INFO:teuthology.orchestra.run.smithi114.stderr:Inferring config /var/lib/ceph/aa240572-0159-11ef-bc93-c7b262605968/mon.a/config 2024-04-23T10:25:03.581 INFO:teuthology.orchestra.run.smithi114.stderr:Inferring config /var/lib/ceph/aa240572-0159-11ef-bc93-c7b262605968/mon.a/config 2024-04-23T10:25:03.584 INFO:teuthology.orchestra.run.smithi114.stderr:Inferring config /var/lib/ceph/aa240572-0159-11ef-bc93-c7b262605968/mon.a/config 2024-04-23T10:25:03.587 INFO:teuthology.orchestra.run.smithi114.stderr:Inferring config /var/lib/ceph/aa240572-0159-11ef-bc93-c7b262605968/mon.a/config 2024-04-23T10:25:03.588 INFO:teuthology.orchestra.run.smithi114.stderr:Inferring config /var/lib/ceph/aa240572-0159-11ef-bc93-c7b262605968/mon.a/config 2024-04-23T10:25:05.254 INFO:journalctl@ceph.mon.a.smithi114.stdout:Apr 23 10:25:05 smithi114 ceph-mon[29255]: pgmap v296: 97 pgs: 97 active+clean; 636 KiB data, 370 MiB used, 1.0 TiB / 1.0 TiB avail; 9.4 KiB/s rd, 5.2 KiB/s wr, 11 op/s 2024-04-23T10:25:05.328 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 23 10:25:05 smithi204 ceph-mon[32649]: pgmap v296: 97 pgs: 97 active+clean; 636 KiB data, 370 MiB used, 1.0 TiB / 1.0 TiB avail; 9.4 KiB/s rd, 5.2 KiB/s wr, 11 op/s 2024-04-23T10:25:05.491 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 23 10:25:05 smithi119 ceph-mon[32608]: pgmap v296: 97 pgs: 97 active+clean; 636 KiB data, 370 MiB used, 1.0 TiB / 1.0 TiB avail; 9.4 KiB/s rd, 5.2 KiB/s wr, 11 op/s 2024-04-23T10:25:07.176 INFO:journalctl@ceph.mon.a.smithi114.stdout:Apr 23 10:25:07 smithi114 ceph-mon[29255]: pgmap v297: 97 pgs: 97 active+clean; 636 KiB data, 370 MiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 1.8 KiB/s wr, 6 op/s 2024-04-23T10:25:07.328 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 23 10:25:07 smithi204 ceph-mon[32649]: pgmap v297: 97 pgs: 97 active+clean; 636 KiB data, 370 MiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 1.8 KiB/s wr, 6 op/s 2024-04-23T10:25:07.476 INFO:teuthology.orchestra.run.smithi114.stderr:Inferring config /var/lib/ceph/aa240572-0159-11ef-bc93-c7b262605968/mon.a/config 2024-04-23T10:25:07.492 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 23 10:25:07 smithi119 ceph-mon[32608]: pgmap v297: 97 pgs: 97 active+clean; 636 KiB data, 370 MiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 1.8 KiB/s wr, 6 op/s 2024-04-23T10:25:09.240 INFO:journalctl@ceph.mon.a.smithi114.stdout:Apr 23 10:25:09 smithi114 ceph-mon[29255]: pgmap v298: 97 pgs: 97 active+clean; 636 KiB data, 370 MiB used, 1.0 TiB / 1.0 TiB avail; 4.9 KiB/s rd, 1.8 KiB/s wr, 7 op/s 2024-04-23T10:25:09.328 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 23 10:25:09 smithi204 ceph-mon[32649]: pgmap v298: 97 pgs: 97 active+clean; 636 KiB data, 370 MiB used, 1.0 TiB / 1.0 TiB avail; 4.9 KiB/s rd, 1.8 KiB/s wr, 7 op/s 2024-04-23T10:25:09.492 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 23 10:25:09 smithi119 ceph-mon[32608]: pgmap v298: 97 pgs: 97 active+clean; 636 KiB data, 370 MiB used, 1.0 TiB / 1.0 TiB avail; 4.9 KiB/s rd, 1.8 KiB/s wr, 7 op/s 2024-04-23T10:25:11.184 INFO:journalctl@ceph.mon.a.smithi114.stdout:Apr 23 10:25:11 smithi114 ceph-mon[29255]: pgmap v299: 97 pgs: 97 active+clean; 636 KiB data, 370 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 1.2 KiB/s wr, 6 op/s 2024-04-23T10:25:11.328 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 23 10:25:11 smithi204 ceph-mon[32649]: pgmap v299: 97 pgs: 97 active+clean; 636 KiB data, 370 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 1.2 KiB/s wr, 6 op/s 2024-04-23T10:25:11.492 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 23 10:25:11 smithi119 ceph-mon[32608]: pgmap v299: 97 pgs: 97 active+clean; 636 KiB data, 370 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 1.2 KiB/s wr, 6 op/s 2024-04-23T10:25:13.182 INFO:journalctl@ceph.mon.a.smithi114.stdout:Apr 23 10:25:13 smithi114 ceph-mon[29255]: pgmap v300: 97 pgs: 97 active+clean; 636 KiB data, 370 MiB used, 1.0 TiB / 1.0 TiB avail; 3.6 KiB/s rd, 1.2 KiB/s wr, 5 op/s 2024-04-23T10:25:13.328 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 23 10:25:13 smithi204 ceph-mon[32649]: pgmap v300: 97 pgs: 97 active+clean; 636 KiB data, 370 MiB used, 1.0 TiB / 1.0 TiB avail; 3.6 KiB/s rd, 1.2 KiB/s wr, 5 op/s 2024-04-23T10:25:13.492 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 23 10:25:13 smithi119 ceph-mon[32608]: pgmap v300: 97 pgs: 97 active+clean; 636 KiB data, 370 MiB used, 1.0 TiB / 1.0 TiB avail; 3.6 KiB/s rd, 1.2 KiB/s wr, 5 op/s 2024-04-23T10:25:14.680 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:25:14.678+0000 7f2514398700 1 -- 172.21.15.114:0/1994256337 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f250c101210 msgr2=0x7f250c1015f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T10:25:14.680 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:25:14.678+0000 7f2514398700 1 --2- 172.21.15.114:0/1994256337 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f250c101210 0x7f250c1015f0 secure :-1 s=READY pgs=250 cs=0 l=1 rev1=1 crypto rx=0x7f25080099c0 tx=0x7f2508009cd0 comp rx=0 tx=0).stop 2024-04-23T10:25:14.680 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:25:14.678+0000 7f2514398700 1 -- 172.21.15.114:0/1994256337 shutdown_connections 2024-04-23T10:25:14.680 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:25:14.678+0000 7f2514398700 1 --2- 172.21.15.114:0/1994256337 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f250c107e30 0x7f250c10f9b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:25:14.680 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:25:14.678+0000 7f2514398700 1 --2- 172.21.15.114:0/1994256337 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f250c101210 0x7f250c1015f0 unknown :-1 s=CLOSED pgs=250 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:25:14.681 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:25:14.678+0000 7f2514398700 1 --2- 172.21.15.114:0/1994256337 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f250c101bc0 0x7f250c107700 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:25:14.681 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:25:14.678+0000 7f2514398700 1 -- 172.21.15.114:0/1994256337 >> 172.21.15.114:0/1994256337 conn(0x7f250c077f40 msgr2=0x7f250c0ff2c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T10:25:14.681 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:25:14.678+0000 7f2514398700 1 -- 172.21.15.114:0/1994256337 shutdown_connections 2024-04-23T10:25:14.681 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:25:14.678+0000 7f2514398700 1 -- 172.21.15.114:0/1994256337 wait complete. 2024-04-23T10:25:14.681 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:25:14.679+0000 7f2514398700 1 Processor -- start 2024-04-23T10:25:14.681 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:25:14.679+0000 7f2514398700 1 -- start start 2024-04-23T10:25:14.681 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:25:14.679+0000 7f2514398700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f250c101210 0x7f250c199160 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T10:25:14.681 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:25:14.679+0000 7f2514398700 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f250c101bc0 0x7f250c1996a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T10:25:14.681 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:25:14.679+0000 7f2514398700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f250c107e30 0x7f250c19da30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T10:25:14.681 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:25:14.679+0000 7f2514398700 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x7f250c112c70 con 0x7f250c101210 2024-04-23T10:25:14.681 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:25:14.679+0000 7f2514398700 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 -- 0x7f250c112af0 con 0x7f250c107e30 2024-04-23T10:25:14.681 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:25:14.679+0000 7f2514398700 1 -- --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_getmap magic: 0 -- 0x7f250c112df0 con 0x7f250c101bc0 2024-04-23T10:25:14.681 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:25:14.679+0000 7f2512134700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f250c101210 0x7f250c199160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T10:25:14.681 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:25:14.679+0000 7f2512134700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f250c101210 0x7f250c199160 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.114:54020/0 (socket says 172.21.15.114:54020) 2024-04-23T10:25:14.682 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:25:14.679+0000 7f2512134700 1 -- 172.21.15.114:0/1126870207 learned_addr learned my addr 172.21.15.114:0/1126870207 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-04-23T10:25:14.682 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:25:14.679+0000 7f2512935700 1 --2- 172.21.15.114:0/1126870207 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f250c107e30 0x7f250c19da30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T10:25:14.682 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:25:14.679+0000 7f2511933700 1 --2- 172.21.15.114:0/1126870207 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f250c101bc0 0x7f250c1996a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T10:25:14.682 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:25:14.679+0000 7f2512134700 1 -- 172.21.15.114:0/1126870207 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f250c101bc0 msgr2=0x7f250c1996a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T10:25:14.682 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:25:14.679+0000 7f2512134700 1 --2- 172.21.15.114:0/1126870207 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f250c101bc0 0x7f250c1996a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:25:14.682 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:25:14.679+0000 7f2512134700 1 -- 172.21.15.114:0/1126870207 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f250c107e30 msgr2=0x7f250c19da30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T10:25:14.682 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:25:14.679+0000 7f2512134700 1 --2- 172.21.15.114:0/1126870207 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f250c107e30 0x7f250c19da30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:25:14.682 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:25:14.679+0000 7f2512134700 1 -- 172.21.15.114:0/1126870207 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f250c19e170 con 0x7f250c101210 2024-04-23T10:25:14.682 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:25:14.679+0000 7f2512134700 1 --2- 172.21.15.114:0/1126870207 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f250c101210 0x7f250c199160 secure :-1 s=READY pgs=251 cs=0 l=1 rev1=1 crypto rx=0x7f250800b580 tx=0x7f25080047b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T10:25:14.682 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:25:14.680+0000 7f25037fe700 1 -- 172.21.15.114:0/1126870207 <== mon.0 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2508027070 con 0x7f250c101210 2024-04-23T10:25:14.682 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:25:14.680+0000 7f2514398700 1 -- 172.21.15.114:0/1126870207 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f250c19e400 con 0x7f250c101210 2024-04-23T10:25:14.682 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:25:14.680+0000 7f25037fe700 1 -- 172.21.15.114:0/1126870207 <== mon.0 v2:172.21.15.114:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f2508003f20 con 0x7f250c101210 2024-04-23T10:25:14.682 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:25:14.680+0000 7f25037fe700 1 -- 172.21.15.114:0/1126870207 <== mon.0 v2:172.21.15.114:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2508022d70 con 0x7f250c101210 2024-04-23T10:25:14.682 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:25:14.680+0000 7f2514398700 1 -- 172.21.15.114:0/1126870207 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f250c1a5ce0 con 0x7f250c101210 2024-04-23T10:25:14.682 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:25:14.680+0000 7f2514398700 1 -- 172.21.15.114:0/1126870207 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_get_version(what=osdmap handle=1) -- 0x7f250c0631d0 con 0x7f250c101210 2024-04-23T10:25:14.684 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:25:14.682+0000 7f25037fe700 1 -- 172.21.15.114:0/1126870207 <== mon.0 v2:172.21.15.114:3300/0 4 ==== mgrmap(e 17) ==== 138698+0+0 (secure 0 0 0) 0x7f2508022500 con 0x7f250c101210 2024-04-23T10:25:14.684 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:25:14.682+0000 7f25037fe700 1 --2- 172.21.15.114:0/1126870207 >> [v2:172.21.15.114:6800/465333808,v1:172.21.15.114:6801/465333808] conn(0x7f24f80a3cd0 0x7f24f80a6190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T10:25:14.684 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:25:14.682+0000 7f2511933700 1 --2- 172.21.15.114:0/1126870207 >> [v2:172.21.15.114:6800/465333808,v1:172.21.15.114:6801/465333808] conn(0x7f24f80a3cd0 0x7f24f80a6190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T10:25:14.684 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:25:14.682+0000 7f25037fe700 1 -- 172.21.15.114:0/1126870207 <== mon.0 v2:172.21.15.114:3300/0 5 ==== osd_map(87..87 src has 1..87) ==== 7979+0+0 (secure 0 0 0) 0x7f25080d1760 con 0x7f250c101210 2024-04-23T10:25:14.684 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:25:14.682+0000 7f25037fe700 1 --2- 172.21.15.114:0/1126870207 >> [v2:172.21.15.119:6824/1400187701,v1:172.21.15.119:6825/1400187701] conn(0x7f24f80ad4b0 0x7f24f80af8d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T10:25:14.684 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:25:14.682+0000 7f25037fe700 1 -- 172.21.15.114:0/1126870207 --> [v2:172.21.15.119:6824/1400187701,v1:172.21.15.119:6825/1400187701] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f2508019e80 con 0x7f24f80ad4b0 2024-04-23T10:25:14.684 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:25:14.682+0000 7f25037fe700 1 -- 172.21.15.114:0/1126870207 <== mon.0 v2:172.21.15.114:3300/0 6 ==== mon_get_version_reply(handle=1 version=87) ==== 24+0+0 (secure 0 0 0) 0x7f25080d1c40 con 0x7f250c101210 2024-04-23T10:25:14.684 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:25:14.683+0000 7f2511933700 1 --2- 172.21.15.114:0/1126870207 >> [v2:172.21.15.114:6800/465333808,v1:172.21.15.114:6801/465333808] conn(0x7f24f80a3cd0 0x7f24f80a6190 secure :-1 s=READY pgs=184 cs=0 l=1 rev1=1 crypto rx=0x7f24fc005950 tx=0x7f24fc0058e0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T10:25:14.684 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:25:14.683+0000 7f2512935700 1 --2- 172.21.15.114:0/1126870207 >> [v2:172.21.15.119:6824/1400187701,v1:172.21.15.119:6825/1400187701] conn(0x7f24f80ad4b0 0x7f24f80af8d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T10:25:14.684 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:25:14.683+0000 7f2512935700 1 --2- 172.21.15.114:0/1126870207 >> [v2:172.21.15.119:6824/1400187701,v1:172.21.15.119:6825/1400187701] conn(0x7f24f80ad4b0 0x7f24f80af8d0 crc :-1 s=READY pgs=12 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.10 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T10:25:14.685 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:25:14.684+0000 7f25037fe700 1 -- 172.21.15.114:0/1126870207 <== osd.10 v2:172.21.15.119:6824/1400187701 1 ==== command_reply(tid 1: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7f2508019e80 con 0x7f24f80ad4b0 2024-04-23T10:25:14.700 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:25:14.700+0000 7f2514398700 1 -- 172.21.15.114:0/1126870207 --> [v2:172.21.15.119:6824/1400187701,v1:172.21.15.119:6825/1400187701] -- command(tid 2: {"prefix": "flush_pg_stats"}) -- 0x7f250c063650 con 0x7f24f80ad4b0 2024-04-23T10:25:14.700 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:25:14.700+0000 7f25037fe700 1 -- 172.21.15.114:0/1126870207 <== osd.10 v2:172.21.15.119:6824/1400187701 2 ==== command_reply(tid 2: 0 ) ==== 8+0+12 (crc 0 0 0) 0x7f250c063650 con 0x7f24f80ad4b0 2024-04-23T10:25:14.701 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:25:14.701+0000 7f2514398700 1 -- 172.21.15.114:0/1126870207 >> [v2:172.21.15.119:6824/1400187701,v1:172.21.15.119:6825/1400187701] conn(0x7f24f80ad4b0 msgr2=0x7f24f80af8d0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T10:25:14.701 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:25:14.701+0000 7f2514398700 1 --2- 172.21.15.114:0/1126870207 >> [v2:172.21.15.119:6824/1400187701,v1:172.21.15.119:6825/1400187701] conn(0x7f24f80ad4b0 0x7f24f80af8d0 crc :-1 s=READY pgs=12 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:25:14.701 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:25:14.701+0000 7f2514398700 1 -- 172.21.15.114:0/1126870207 >> [v2:172.21.15.114:6800/465333808,v1:172.21.15.114:6801/465333808] conn(0x7f24f80a3cd0 msgr2=0x7f24f80a6190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T10:25:14.701 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:25:14.701+0000 7f2514398700 1 --2- 172.21.15.114:0/1126870207 >> [v2:172.21.15.114:6800/465333808,v1:172.21.15.114:6801/465333808] conn(0x7f24f80a3cd0 0x7f24f80a6190 secure :-1 s=READY pgs=184 cs=0 l=1 rev1=1 crypto rx=0x7f24fc005950 tx=0x7f24fc0058e0 comp rx=0 tx=0).stop 2024-04-23T10:25:14.701 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:25:14.701+0000 7f2514398700 1 -- 172.21.15.114:0/1126870207 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f250c101210 msgr2=0x7f250c199160 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T10:25:14.702 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:25:14.701+0000 7f2514398700 1 --2- 172.21.15.114:0/1126870207 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f250c101210 0x7f250c199160 secure :-1 s=READY pgs=251 cs=0 l=1 rev1=1 crypto rx=0x7f250800b580 tx=0x7f25080047b0 comp rx=0 tx=0).stop 2024-04-23T10:25:14.702 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:25:14.701+0000 7f2512935700 1 -- 172.21.15.114:0/1126870207 reap_dead start 2024-04-23T10:25:14.702 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:25:14.701+0000 7f2514398700 1 -- 172.21.15.114:0/1126870207 shutdown_connections 2024-04-23T10:25:14.702 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:25:14.701+0000 7f2514398700 1 -- 172.21.15.114:0/1126870207 >> 172.21.15.114:0/1126870207 conn(0x7f250c077f40 msgr2=0x7f250c0ff2c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T10:25:14.702 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:25:14.701+0000 7f2514398700 1 -- 172.21.15.114:0/1126870207 shutdown_connections 2024-04-23T10:25:14.702 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:25:14.701+0000 7f2514398700 1 -- 172.21.15.114:0/1126870207 wait complete. 2024-04-23T10:25:15.328 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 23 10:25:15 smithi204 ceph-mon[32649]: pgmap v301: 97 pgs: 97 active+clean; 639 KiB data, 370 MiB used, 1.0 TiB / 1.0 TiB avail; 5.2 KiB/s rd, 1.7 KiB/s wr, 8 op/s 2024-04-23T10:25:15.492 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 23 10:25:15 smithi119 ceph-mon[32608]: pgmap v301: 97 pgs: 97 active+clean; 639 KiB data, 370 MiB used, 1.0 TiB / 1.0 TiB avail; 5.2 KiB/s rd, 1.7 KiB/s wr, 8 op/s 2024-04-23T10:25:15.504 INFO:journalctl@ceph.mon.a.smithi114.stdout:Apr 23 10:25:15 smithi114 ceph-mon[29255]: pgmap v301: 97 pgs: 97 active+clean; 639 KiB data, 370 MiB used, 1.0 TiB / 1.0 TiB avail; 5.2 KiB/s rd, 1.7 KiB/s wr, 8 op/s 2024-04-23T10:25:16.283 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:25:16.284+0000 7fedaa877700 1 -- 172.21.15.114:0/1872197117 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7feda41019d0 msgr2=0x7feda410f910 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T10:25:16.283 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:25:16.284+0000 7fedaa877700 1 --2- 172.21.15.114:0/1872197117 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7feda41019d0 0x7feda410f910 secure :-1 s=READY pgs=254 cs=0 l=1 rev1=1 crypto rx=0x7fed9c00b0a0 tx=0x7fed9c00b3b0 comp rx=0 tx=0).stop 2024-04-23T10:25:16.283 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:25:16.284+0000 7fedaa877700 1 -- 172.21.15.114:0/1872197117 shutdown_connections 2024-04-23T10:25:16.283 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:25:16.284+0000 7fedaa877700 1 --2- 172.21.15.114:0/1872197117 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7feda41019d0 0x7feda410f910 unknown :-1 s=CLOSED pgs=254 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:25:16.283 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:25:16.284+0000 7fedaa877700 1 --2- 172.21.15.114:0/1872197117 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7feda4108ad0 0x7feda4108eb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:25:16.283 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:25:16.284+0000 7fedaa877700 1 --2- 172.21.15.114:0/1872197117 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7feda4101030 0x7feda4101490 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:25:16.284 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:25:16.284+0000 7fedaa877700 1 -- 172.21.15.114:0/1872197117 >> 172.21.15.114:0/1872197117 conn(0x7feda4077fb0 msgr2=0x7feda40ff360 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T10:25:16.284 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:25:16.284+0000 7fedaa877700 1 -- 172.21.15.114:0/1872197117 shutdown_connections 2024-04-23T10:25:16.284 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:25:16.284+0000 7fedaa877700 1 -- 172.21.15.114:0/1872197117 wait complete. 2024-04-23T10:25:16.286 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:25:16.285+0000 7fedaa877700 1 Processor -- start 2024-04-23T10:25:16.286 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:25:16.285+0000 7fedaa877700 1 -- start start 2024-04-23T10:25:16.286 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:25:16.285+0000 7fedaa877700 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7feda4101030 0x7feda4102210 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T10:25:16.286 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:25:16.285+0000 7fedaa877700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7feda41019d0 0x7feda4102750 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T10:25:16.286 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:25:16.285+0000 7fedaa877700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7feda4108ad0 0x7feda4102c90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T10:25:16.286 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:25:16.285+0000 7fedaa877700 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x7feda4112c40 con 0x7feda4108ad0 2024-04-23T10:25:16.286 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:25:16.285+0000 7fedaa877700 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 -- 0x7feda4112ac0 con 0x7feda41019d0 2024-04-23T10:25:16.286 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:25:16.285+0000 7fedaa877700 1 -- --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_getmap magic: 0 -- 0x7feda4112dc0 con 0x7feda4101030 2024-04-23T10:25:16.286 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:25:16.285+0000 7feda8e14700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7feda4108ad0 0x7feda4102c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T10:25:16.286 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:25:16.285+0000 7feda8e14700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7feda4108ad0 0x7feda4102c90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.114:36242/0 (socket says 172.21.15.114:36242) 2024-04-23T10:25:16.286 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:25:16.285+0000 7feda8e14700 1 -- 172.21.15.114:0/3299783344 learned_addr learned my addr 172.21.15.114:0/3299783344 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-04-23T10:25:16.287 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:25:16.285+0000 7feda37fe700 1 --2- 172.21.15.114:0/3299783344 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7feda41019d0 0x7feda4102750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T10:25:16.287 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:25:16.285+0000 7feda3fff700 1 --2- 172.21.15.114:0/3299783344 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7feda4101030 0x7feda4102210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T10:25:16.287 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:25:16.286+0000 7feda8e14700 1 -- 172.21.15.114:0/3299783344 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7feda4101030 msgr2=0x7feda4102210 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T10:25:16.287 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:25:16.286+0000 7feda8e14700 1 --2- 172.21.15.114:0/3299783344 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7feda4101030 0x7feda4102210 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:25:16.287 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:25:16.286+0000 7feda8e14700 1 -- 172.21.15.114:0/3299783344 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7feda41019d0 msgr2=0x7feda4102750 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T10:25:16.287 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:25:16.286+0000 7feda8e14700 1 --2- 172.21.15.114:0/3299783344 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7feda41019d0 0x7feda4102750 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:25:16.287 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:25:16.286+0000 7feda8e14700 1 -- 172.21.15.114:0/3299783344 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7feda41a7660 con 0x7feda4108ad0 2024-04-23T10:25:16.287 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:25:16.286+0000 7feda3fff700 1 --2- 172.21.15.114:0/3299783344 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7feda4101030 0x7feda4102210 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-23T10:25:16.287 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:25:16.286+0000 7feda8e14700 1 --2- 172.21.15.114:0/3299783344 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7feda4108ad0 0x7feda4102c90 secure :-1 s=READY pgs=255 cs=0 l=1 rev1=1 crypto rx=0x7fed9c0062a0 tx=0x7fed9c004e20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T10:25:16.287 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:25:16.286+0000 7feda17fa700 1 -- 172.21.15.114:0/3299783344 <== mon.0 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fed9c021070 con 0x7feda4108ad0 2024-04-23T10:25:16.287 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:25:16.286+0000 7fedaa877700 1 -- 172.21.15.114:0/3299783344 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7feda41a78f0 con 0x7feda4108ad0 2024-04-23T10:25:16.287 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:25:16.286+0000 7feda17fa700 1 -- 172.21.15.114:0/3299783344 <== mon.0 v2:172.21.15.114:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fed9c007c80 con 0x7feda4108ad0 2024-04-23T10:25:16.287 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:25:16.286+0000 7feda17fa700 1 -- 172.21.15.114:0/3299783344 <== mon.0 v2:172.21.15.114:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fed9c01d450 con 0x7feda4108ad0 2024-04-23T10:25:16.287 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:25:16.286+0000 7fedaa877700 1 -- 172.21.15.114:0/3299783344 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7feda41a7df0 con 0x7feda4108ad0 2024-04-23T10:25:16.287 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:25:16.287+0000 7fedaa877700 1 -- 172.21.15.114:0/3299783344 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_get_version(what=osdmap handle=1) -- 0x7feda4063150 con 0x7feda4108ad0 2024-04-23T10:25:16.289 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:25:16.288+0000 7feda17fa700 1 -- 172.21.15.114:0/3299783344 <== mon.0 v2:172.21.15.114:3300/0 4 ==== mgrmap(e 17) ==== 138698+0+0 (secure 0 0 0) 0x7fed9c004000 con 0x7feda4108ad0 2024-04-23T10:25:16.289 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:25:16.289+0000 7feda17fa700 1 --2- 172.21.15.114:0/3299783344 >> [v2:172.21.15.114:6800/465333808,v1:172.21.15.114:6801/465333808] conn(0x7fed8c0a3cd0 0x7fed8c0a6190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T10:25:16.289 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:25:16.289+0000 7feda3fff700 1 --2- 172.21.15.114:0/3299783344 >> [v2:172.21.15.114:6800/465333808,v1:172.21.15.114:6801/465333808] conn(0x7fed8c0a3cd0 0x7fed8c0a6190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T10:25:16.289 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:25:16.289+0000 7feda17fa700 1 -- 172.21.15.114:0/3299783344 <== mon.0 v2:172.21.15.114:3300/0 5 ==== osd_map(87..87 src has 1..87) ==== 7979+0+0 (secure 0 0 0) 0x7fed9c00f020 con 0x7feda4108ad0 2024-04-23T10:25:16.289 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:25:16.289+0000 7feda17fa700 1 --2- 172.21.15.114:0/3299783344 >> [v2:172.21.15.114:6818/4072912544,v1:172.21.15.114:6819/4072912544] conn(0x7fed8c0ad4b0 0x7fed8c0af8d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T10:25:16.289 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:25:16.289+0000 7feda17fa700 1 -- 172.21.15.114:0/3299783344 --> [v2:172.21.15.114:6818/4072912544,v1:172.21.15.114:6819/4072912544] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7fed9c02ee50 con 0x7fed8c0ad4b0 2024-04-23T10:25:16.289 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:25:16.289+0000 7feda17fa700 1 -- 172.21.15.114:0/3299783344 <== mon.0 v2:172.21.15.114:3300/0 6 ==== mon_get_version_reply(handle=1 version=87) ==== 24+0+0 (secure 0 0 0) 0x7fed9c0d21c0 con 0x7feda4108ad0 2024-04-23T10:25:16.289 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:25:16.289+0000 7feda37fe700 1 --2- 172.21.15.114:0/3299783344 >> [v2:172.21.15.114:6818/4072912544,v1:172.21.15.114:6819/4072912544] conn(0x7fed8c0ad4b0 0x7fed8c0af8d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T10:25:16.289 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:25:16.289+0000 7feda3fff700 1 --2- 172.21.15.114:0/3299783344 >> [v2:172.21.15.114:6800/465333808,v1:172.21.15.114:6801/465333808] conn(0x7fed8c0a3cd0 0x7fed8c0a6190 secure :-1 s=READY pgs=186 cs=0 l=1 rev1=1 crypto rx=0x7fed940099d0 tx=0x7fed94008040 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T10:25:16.289 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:25:16.289+0000 7feda37fe700 1 --2- 172.21.15.114:0/3299783344 >> [v2:172.21.15.114:6818/4072912544,v1:172.21.15.114:6819/4072912544] conn(0x7fed8c0ad4b0 0x7fed8c0af8d0 crc :-1 s=READY pgs=12 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.6 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T10:25:16.289 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:25:16.290+0000 7feda17fa700 1 -- 172.21.15.114:0/3299783344 <== osd.6 v2:172.21.15.114:6818/4072912544 1 ==== command_reply(tid 1: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7fed9c02ee50 con 0x7fed8c0ad4b0 2024-04-23T10:25:16.305 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:25:16.306+0000 7fedaa877700 1 -- 172.21.15.114:0/3299783344 --> [v2:172.21.15.114:6818/4072912544,v1:172.21.15.114:6819/4072912544] -- command(tid 2: {"prefix": "flush_pg_stats"}) -- 0x7feda40635d0 con 0x7fed8c0ad4b0 2024-04-23T10:25:16.306 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:25:16.307+0000 7feda17fa700 1 -- 172.21.15.114:0/3299783344 <== osd.6 v2:172.21.15.114:6818/4072912544 2 ==== command_reply(tid 2: 0 ) ==== 8+0+12 (crc 0 0 0) 0x7feda40635d0 con 0x7fed8c0ad4b0 2024-04-23T10:25:16.307 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:25:16.307+0000 7fedaa877700 1 -- 172.21.15.114:0/3299783344 >> [v2:172.21.15.114:6818/4072912544,v1:172.21.15.114:6819/4072912544] conn(0x7fed8c0ad4b0 msgr2=0x7fed8c0af8d0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T10:25:16.307 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:25:16.307+0000 7fedaa877700 1 --2- 172.21.15.114:0/3299783344 >> [v2:172.21.15.114:6818/4072912544,v1:172.21.15.114:6819/4072912544] conn(0x7fed8c0ad4b0 0x7fed8c0af8d0 crc :-1 s=READY pgs=12 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:25:16.307 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:25:16.307+0000 7fedaa877700 1 -- 172.21.15.114:0/3299783344 >> [v2:172.21.15.114:6800/465333808,v1:172.21.15.114:6801/465333808] conn(0x7fed8c0a3cd0 msgr2=0x7fed8c0a6190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T10:25:16.307 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:25:16.307+0000 7fedaa877700 1 --2- 172.21.15.114:0/3299783344 >> [v2:172.21.15.114:6800/465333808,v1:172.21.15.114:6801/465333808] conn(0x7fed8c0a3cd0 0x7fed8c0a6190 secure :-1 s=READY pgs=186 cs=0 l=1 rev1=1 crypto rx=0x7fed940099d0 tx=0x7fed94008040 comp rx=0 tx=0).stop 2024-04-23T10:25:16.307 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:25:16.307+0000 7fedaa877700 1 -- 172.21.15.114:0/3299783344 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7feda4108ad0 msgr2=0x7feda4102c90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T10:25:16.307 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:25:16.307+0000 7fedaa877700 1 --2- 172.21.15.114:0/3299783344 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7feda4108ad0 0x7feda4102c90 secure :-1 s=READY pgs=255 cs=0 l=1 rev1=1 crypto rx=0x7fed9c0062a0 tx=0x7fed9c004e20 comp rx=0 tx=0).stop 2024-04-23T10:25:16.307 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:25:16.307+0000 7feda8e14700 1 -- 172.21.15.114:0/3299783344 reap_dead start 2024-04-23T10:25:16.307 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:25:16.307+0000 7fedaa877700 1 -- 172.21.15.114:0/3299783344 shutdown_connections 2024-04-23T10:25:16.307 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:25:16.307+0000 7fedaa877700 1 -- 172.21.15.114:0/3299783344 >> 172.21.15.114:0/3299783344 conn(0x7feda4077fb0 msgr2=0x7feda40ff880 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T10:25:16.307 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:25:16.308+0000 7fedaa877700 1 -- 172.21.15.114:0/3299783344 shutdown_connections 2024-04-23T10:25:16.307 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:25:16.308+0000 7fedaa877700 1 -- 172.21.15.114:0/3299783344 wait complete. 2024-04-23T10:25:16.464 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:25:15.240+0000 7f86477e5700 1 -- 172.21.15.114:0/974731717 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f8640101da0 msgr2=0x7f864010f810 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T10:25:16.464 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:25:15.240+0000 7f86477e5700 1 --2- 172.21.15.114:0/974731717 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f8640101da0 0x7f864010f810 secure :-1 s=READY pgs=252 cs=0 l=1 rev1=1 crypto rx=0x7f863800b0a0 tx=0x7f863800b3b0 comp rx=0 tx=0).stop 2024-04-23T10:25:16.464 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:25:15.240+0000 7f86477e5700 1 -- 172.21.15.114:0/974731717 shutdown_connections 2024-04-23T10:25:16.464 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:25:15.240+0000 7f86477e5700 1 --2- 172.21.15.114:0/974731717 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f8640107b00 0x7f8640107ee0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:25:16.464 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:25:15.240+0000 7f86477e5700 1 --2- 172.21.15.114:0/974731717 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f8640101da0 0x7f864010f810 unknown :-1 s=CLOSED pgs=252 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:25:16.464 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:25:15.240+0000 7f86477e5700 1 --2- 172.21.15.114:0/974731717 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f86401013e0 0x7f8640101860 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:25:16.464 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:25:15.240+0000 7f86477e5700 1 -- 172.21.15.114:0/974731717 >> 172.21.15.114:0/974731717 conn(0x7f8640076c60 msgr2=0x7f8640077070 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T10:25:16.464 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:25:15.240+0000 7f86477e5700 1 -- 172.21.15.114:0/974731717 shutdown_connections 2024-04-23T10:25:16.464 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:25:15.240+0000 7f86477e5700 1 -- 172.21.15.114:0/974731717 wait complete. 2024-04-23T10:25:16.465 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:25:15.241+0000 7f86477e5700 1 Processor -- start 2024-04-23T10:25:16.465 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:25:15.241+0000 7f86477e5700 1 -- start start 2024-04-23T10:25:16.465 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:25:15.241+0000 7f86477e5700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f86401013e0 0x7f8640199260 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T10:25:16.465 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:25:15.241+0000 7f86477e5700 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f8640101da0 0x7f86401997a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T10:25:16.465 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:25:15.241+0000 7f86477e5700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f8640107b00 0x7f864019dae0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T10:25:16.465 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:25:15.241+0000 7f86477e5700 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x7f8640112d10 con 0x7f8640107b00 2024-04-23T10:25:16.465 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:25:15.241+0000 7f86477e5700 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 -- 0x7f8640112b90 con 0x7f86401013e0 2024-04-23T10:25:16.465 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:25:15.241+0000 7f86477e5700 1 -- --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_getmap magic: 0 -- 0x7f8640112e90 con 0x7f8640101da0 2024-04-23T10:25:16.465 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:25:15.241+0000 7f8645d82700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f8640107b00 0x7f864019dae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T10:25:16.465 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:25:15.241+0000 7f8645d82700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f8640107b00 0x7f864019dae0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.114:54044/0 (socket says 172.21.15.114:54044) 2024-04-23T10:25:16.465 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:25:15.241+0000 7f8645d82700 1 -- 172.21.15.114:0/691269890 learned_addr learned my addr 172.21.15.114:0/691269890 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-04-23T10:25:16.465 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:25:15.241+0000 7f8645581700 1 --2- 172.21.15.114:0/691269890 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f86401013e0 0x7f8640199260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T10:25:16.465 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:25:15.241+0000 7f8645d82700 1 -- 172.21.15.114:0/691269890 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f8640101da0 msgr2=0x7f86401997a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T10:25:16.465 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:25:15.241+0000 7f8644d80700 1 --2- 172.21.15.114:0/691269890 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f8640101da0 0x7f86401997a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T10:25:16.466 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:25:15.241+0000 7f8645d82700 1 --2- 172.21.15.114:0/691269890 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f8640101da0 0x7f86401997a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:25:16.466 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:25:15.241+0000 7f8645d82700 1 -- 172.21.15.114:0/691269890 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f86401013e0 msgr2=0x7f8640199260 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T10:25:16.466 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:25:15.241+0000 7f8645d82700 1 --2- 172.21.15.114:0/691269890 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f86401013e0 0x7f8640199260 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:25:16.466 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:25:15.241+0000 7f8645d82700 1 -- 172.21.15.114:0/691269890 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f864019e220 con 0x7f8640107b00 2024-04-23T10:25:16.466 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:25:15.241+0000 7f8644d80700 1 --2- 172.21.15.114:0/691269890 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f8640101da0 0x7f86401997a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-23T10:25:16.466 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:25:15.242+0000 7f8645d82700 1 --2- 172.21.15.114:0/691269890 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f8640107b00 0x7f864019dae0 secure :-1 s=READY pgs=253 cs=0 l=1 rev1=1 crypto rx=0x7f86380062a0 tx=0x7f8638004ed0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T10:25:16.466 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:25:15.242+0000 7f86367fc700 1 -- 172.21.15.114:0/691269890 <== mon.0 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8638021070 con 0x7f8640107b00 2024-04-23T10:25:16.466 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:25:15.242+0000 7f86367fc700 1 -- 172.21.15.114:0/691269890 <== mon.0 v2:172.21.15.114:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f8638007d00 con 0x7f8640107b00 2024-04-23T10:25:16.466 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:25:15.242+0000 7f86477e5700 1 -- 172.21.15.114:0/691269890 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f864019e4b0 con 0x7f8640107b00 2024-04-23T10:25:16.466 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:25:15.242+0000 7f86367fc700 1 -- 172.21.15.114:0/691269890 <== mon.0 v2:172.21.15.114:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f863801d450 con 0x7f8640107b00 2024-04-23T10:25:16.467 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:25:15.242+0000 7f86477e5700 1 -- 172.21.15.114:0/691269890 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f86401a5d90 con 0x7f8640107b00 2024-04-23T10:25:16.467 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:25:15.243+0000 7f86477e5700 1 -- 172.21.15.114:0/691269890 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_get_version(what=osdmap handle=1) -- 0x7f86400631d0 con 0x7f8640107b00 2024-04-23T10:25:16.467 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:25:15.244+0000 7f86367fc700 1 -- 172.21.15.114:0/691269890 <== mon.0 v2:172.21.15.114:3300/0 4 ==== mgrmap(e 17) ==== 138698+0+0 (secure 0 0 0) 0x7f8638004000 con 0x7f8640107b00 2024-04-23T10:25:16.467 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:25:15.244+0000 7f86367fc700 1 --2- 172.21.15.114:0/691269890 >> [v2:172.21.15.114:6800/465333808,v1:172.21.15.114:6801/465333808] conn(0x7f862c0a3cd0 0x7f862c0a6190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T10:25:16.467 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:25:15.245+0000 7f8645581700 1 --2- 172.21.15.114:0/691269890 >> [v2:172.21.15.114:6800/465333808,v1:172.21.15.114:6801/465333808] conn(0x7f862c0a3cd0 0x7f862c0a6190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T10:25:16.467 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:25:15.245+0000 7f86367fc700 1 -- 172.21.15.114:0/691269890 <== mon.0 v2:172.21.15.114:3300/0 5 ==== osd_map(87..87 src has 1..87) ==== 7979+0+0 (secure 0 0 0) 0x7f863800f020 con 0x7f8640107b00 2024-04-23T10:25:16.467 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:25:15.245+0000 7f86367fc700 1 --2- 172.21.15.114:0/691269890 >> [v2:172.21.15.119:6800/1133540779,v1:172.21.15.119:6801/1133540779] conn(0x7f862c0ad4b0 0x7f862c0af8d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T10:25:16.467 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:25:15.245+0000 7f86367fc700 1 -- 172.21.15.114:0/691269890 --> [v2:172.21.15.119:6800/1133540779,v1:172.21.15.119:6801/1133540779] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f86380020b0 con 0x7f862c0ad4b0 2024-04-23T10:25:16.467 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:25:15.245+0000 7f86367fc700 1 -- 172.21.15.114:0/691269890 <== mon.0 v2:172.21.15.114:3300/0 6 ==== mon_get_version_reply(handle=1 version=87) ==== 24+0+0 (secure 0 0 0) 0x7f86380d2200 con 0x7f8640107b00 2024-04-23T10:25:16.467 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:25:15.245+0000 7f8645581700 1 --2- 172.21.15.114:0/691269890 >> [v2:172.21.15.114:6800/465333808,v1:172.21.15.114:6801/465333808] conn(0x7f862c0a3cd0 0x7f862c0a6190 secure :-1 s=READY pgs=185 cs=0 l=1 rev1=1 crypto rx=0x7f863c006fd0 tx=0x7f863c008040 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T10:25:16.467 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:25:15.245+0000 7f8644d80700 1 --2- 172.21.15.114:0/691269890 >> [v2:172.21.15.119:6800/1133540779,v1:172.21.15.119:6801/1133540779] conn(0x7f862c0ad4b0 0x7f862c0af8d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T10:25:16.467 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:25:15.245+0000 7f8644d80700 1 --2- 172.21.15.114:0/691269890 >> [v2:172.21.15.119:6800/1133540779,v1:172.21.15.119:6801/1133540779] conn(0x7f862c0ad4b0 0x7f862c0af8d0 crc :-1 s=READY pgs=11 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T10:25:16.467 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:25:15.246+0000 7f86367fc700 1 -- 172.21.15.114:0/691269890 <== osd.1 v2:172.21.15.119:6800/1133540779 1 ==== command_reply(tid 1: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7f86380020b0 con 0x7f862c0ad4b0 2024-04-23T10:25:16.468 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:25:15.263+0000 7f86477e5700 1 -- 172.21.15.114:0/691269890 --> [v2:172.21.15.119:6800/1133540779,v1:172.21.15.119:6801/1133540779] -- command(tid 2: {"prefix": "flush_pg_stats"}) -- 0x7f8640063650 con 0x7f862c0ad4b0 2024-04-23T10:25:16.468 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:25:15.263+0000 7f86367fc700 1 -- 172.21.15.114:0/691269890 <== osd.1 v2:172.21.15.119:6800/1133540779 2 ==== command_reply(tid 2: 0 ) ==== 8+0+11 (crc 0 0 0) 0x7f8640063650 con 0x7f862c0ad4b0 2024-04-23T10:25:16.468 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:25:15.263+0000 7f86477e5700 1 -- 172.21.15.114:0/691269890 >> [v2:172.21.15.119:6800/1133540779,v1:172.21.15.119:6801/1133540779] conn(0x7f862c0ad4b0 msgr2=0x7f862c0af8d0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T10:25:16.468 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:25:15.263+0000 7f86477e5700 1 --2- 172.21.15.114:0/691269890 >> [v2:172.21.15.119:6800/1133540779,v1:172.21.15.119:6801/1133540779] conn(0x7f862c0ad4b0 0x7f862c0af8d0 crc :-1 s=READY pgs=11 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:25:16.468 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:25:15.264+0000 7f86477e5700 1 -- 172.21.15.114:0/691269890 >> [v2:172.21.15.114:6800/465333808,v1:172.21.15.114:6801/465333808] conn(0x7f862c0a3cd0 msgr2=0x7f862c0a6190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T10:25:16.468 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:25:15.264+0000 7f86477e5700 1 --2- 172.21.15.114:0/691269890 >> [v2:172.21.15.114:6800/465333808,v1:172.21.15.114:6801/465333808] conn(0x7f862c0a3cd0 0x7f862c0a6190 secure :-1 s=READY pgs=185 cs=0 l=1 rev1=1 crypto rx=0x7f863c006fd0 tx=0x7f863c008040 comp rx=0 tx=0).stop 2024-04-23T10:25:16.468 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:25:15.264+0000 7f86477e5700 1 -- 172.21.15.114:0/691269890 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f8640107b00 msgr2=0x7f864019dae0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T10:25:16.468 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:25:15.264+0000 7f86477e5700 1 --2- 172.21.15.114:0/691269890 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f8640107b00 0x7f864019dae0 secure :-1 s=READY pgs=253 cs=0 l=1 rev1=1 crypto rx=0x7f86380062a0 tx=0x7f8638004ed0 comp rx=0 tx=0).stop 2024-04-23T10:25:16.468 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:25:15.264+0000 7f8645d82700 1 -- 172.21.15.114:0/691269890 reap_dead start 2024-04-23T10:25:16.468 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:25:15.264+0000 7f86477e5700 1 -- 172.21.15.114:0/691269890 shutdown_connections 2024-04-23T10:25:16.468 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:25:15.264+0000 7f86477e5700 1 -- 172.21.15.114:0/691269890 >> 172.21.15.114:0/691269890 conn(0x7f8640076c60 msgr2=0x7f86401000c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T10:25:16.468 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:25:15.264+0000 7f86477e5700 1 -- 172.21.15.114:0/691269890 shutdown_connections 2024-04-23T10:25:16.468 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:25:15.264+0000 7f86477e5700 1 -- 172.21.15.114:0/691269890 wait complete. 2024-04-23T10:25:16.883 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:25:16.863+0000 7fcfd8292700 1 -- 172.21.15.114:0/211570077 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fcfd01022f0 msgr2=0x7fcfd010e7e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T10:25:16.883 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:25:16.863+0000 7fcfd8292700 1 --2- 172.21.15.114:0/211570077 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fcfd01022f0 0x7fcfd010e7e0 secure :-1 s=READY pgs=256 cs=0 l=1 rev1=1 crypto rx=0x7fcfc800b0a0 tx=0x7fcfc800b3b0 comp rx=0 tx=0).stop 2024-04-23T10:25:16.883 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:25:16.863+0000 7fcfd8292700 1 -- 172.21.15.114:0/211570077 shutdown_connections 2024-04-23T10:25:16.883 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:25:16.863+0000 7fcfd8292700 1 --2- 172.21.15.114:0/211570077 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fcfd01022f0 0x7fcfd010e7e0 unknown :-1 s=CLOSED pgs=256 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:25:16.884 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:25:16.863+0000 7fcfd8292700 1 --2- 172.21.15.114:0/211570077 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fcfd0107980 0x7fcfd0107d60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:25:16.884 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:25:16.863+0000 7fcfd8292700 1 --2- 172.21.15.114:0/211570077 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fcfd0101930 0x7fcfd0101db0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:25:16.884 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:25:16.863+0000 7fcfd8292700 1 -- 172.21.15.114:0/211570077 >> 172.21.15.114:0/211570077 conn(0x7fcfd00788b0 msgr2=0x7fcfd0078cc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T10:25:16.884 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:25:16.863+0000 7fcfd8292700 1 -- 172.21.15.114:0/211570077 shutdown_connections 2024-04-23T10:25:16.884 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:25:16.863+0000 7fcfd8292700 1 -- 172.21.15.114:0/211570077 wait complete. 2024-04-23T10:25:16.884 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:25:16.864+0000 7fcfd8292700 1 Processor -- start 2024-04-23T10:25:16.884 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:25:16.864+0000 7fcfd8292700 1 -- start start 2024-04-23T10:25:16.884 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:25:16.864+0000 7fcfd8292700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fcfd0101930 0x7fcfd01034e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T10:25:16.884 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:25:16.864+0000 7fcfd8292700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fcfd01022f0 0x7fcfd0103a20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T10:25:16.884 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:25:16.864+0000 7fcfd8292700 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fcfd0107980 0x7fcfd0103f60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T10:25:16.884 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:25:16.864+0000 7fcfd8292700 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x7fcfd0111b60 con 0x7fcfd0101930 2024-04-23T10:25:16.884 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:25:16.864+0000 7fcfd8292700 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 -- 0x7fcfd01119e0 con 0x7fcfd01022f0 2024-04-23T10:25:16.884 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:25:16.864+0000 7fcfd8292700 1 -- --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_getmap magic: 0 -- 0x7fcfd0111ce0 con 0x7fcfd0107980 2024-04-23T10:25:16.884 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:25:16.864+0000 7fcfd602e700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fcfd0101930 0x7fcfd01034e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T10:25:16.885 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:25:16.864+0000 7fcfd602e700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fcfd0101930 0x7fcfd01034e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.114:36258/0 (socket says 172.21.15.114:36258) 2024-04-23T10:25:16.885 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:25:16.864+0000 7fcfd602e700 1 -- 172.21.15.114:0/1331307751 learned_addr learned my addr 172.21.15.114:0/1331307751 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-04-23T10:25:16.885 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:25:16.864+0000 7fcfd682f700 1 --2- 172.21.15.114:0/1331307751 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fcfd0107980 0x7fcfd0103f60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T10:25:16.885 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:25:16.864+0000 7fcfd582d700 1 --2- 172.21.15.114:0/1331307751 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fcfd01022f0 0x7fcfd0103a20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T10:25:16.885 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:25:16.865+0000 7fcfd602e700 1 -- 172.21.15.114:0/1331307751 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fcfd0107980 msgr2=0x7fcfd0103f60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T10:25:16.885 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:25:16.865+0000 7fcfd602e700 1 --2- 172.21.15.114:0/1331307751 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fcfd0107980 0x7fcfd0103f60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:25:16.885 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:25:16.865+0000 7fcfd602e700 1 -- 172.21.15.114:0/1331307751 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fcfd01022f0 msgr2=0x7fcfd0103a20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T10:25:16.885 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:25:16.865+0000 7fcfd602e700 1 --2- 172.21.15.114:0/1331307751 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fcfd01022f0 0x7fcfd0103a20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:25:16.885 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:25:16.865+0000 7fcfd602e700 1 -- 172.21.15.114:0/1331307751 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fcfd01a7670 con 0x7fcfd0101930 2024-04-23T10:25:16.885 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:25:16.865+0000 7fcfd582d700 1 --2- 172.21.15.114:0/1331307751 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fcfd01022f0 0x7fcfd0103a20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-23T10:25:16.885 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:25:16.865+0000 7fcfd602e700 1 --2- 172.21.15.114:0/1331307751 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fcfd0101930 0x7fcfd01034e0 secure :-1 s=READY pgs=257 cs=0 l=1 rev1=1 crypto rx=0x7fcfcc00c970 tx=0x7fcfcc00cc80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T10:25:16.885 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:25:16.865+0000 7fcfc77fe700 1 -- 172.21.15.114:0/1331307751 <== mon.0 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcfcc007aa0 con 0x7fcfd0101930 2024-04-23T10:25:16.886 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:25:16.865+0000 7fcfd8292700 1 -- 172.21.15.114:0/1331307751 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fcfd01a7960 con 0x7fcfd0101930 2024-04-23T10:25:16.886 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:25:16.865+0000 7fcfc77fe700 1 -- 172.21.15.114:0/1331307751 <== mon.0 v2:172.21.15.114:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fcfcc004ce0 con 0x7fcfd0101930 2024-04-23T10:25:16.886 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:25:16.865+0000 7fcfc77fe700 1 -- 172.21.15.114:0/1331307751 <== mon.0 v2:172.21.15.114:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcfcc005510 con 0x7fcfd0101930 2024-04-23T10:25:16.886 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:25:16.865+0000 7fcfd8292700 1 -- 172.21.15.114:0/1331307751 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fcfd01a7ea0 con 0x7fcfd0101930 2024-04-23T10:25:16.886 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:25:16.866+0000 7fcfd8292700 1 -- 172.21.15.114:0/1331307751 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_get_version(what=osdmap handle=1) -- 0x7fcfd0063110 con 0x7fcfd0101930 2024-04-23T10:25:16.886 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:25:16.867+0000 7fcfc77fe700 1 -- 172.21.15.114:0/1331307751 <== mon.0 v2:172.21.15.114:3300/0 4 ==== mgrmap(e 17) ==== 138698+0+0 (secure 0 0 0) 0x7fcfcc007c40 con 0x7fcfd0101930 2024-04-23T10:25:16.886 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:25:16.868+0000 7fcfc77fe700 1 --2- 172.21.15.114:0/1331307751 >> [v2:172.21.15.114:6800/465333808,v1:172.21.15.114:6801/465333808] conn(0x7fcfbc0a3cd0 0x7fcfbc0a6190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T10:25:16.886 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:25:16.868+0000 7fcfd582d700 1 --2- 172.21.15.114:0/1331307751 >> [v2:172.21.15.114:6800/465333808,v1:172.21.15.114:6801/465333808] conn(0x7fcfbc0a3cd0 0x7fcfbc0a6190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T10:25:16.886 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:25:16.868+0000 7fcfc77fe700 1 -- 172.21.15.114:0/1331307751 <== mon.0 v2:172.21.15.114:3300/0 5 ==== osd_map(87..87 src has 1..87) ==== 7979+0+0 (secure 0 0 0) 0x7fcfcc0d05d0 con 0x7fcfd0101930 2024-04-23T10:25:16.886 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:25:16.868+0000 7fcfc77fe700 1 --2- 172.21.15.114:0/1331307751 >> [v2:172.21.15.114:6810/4275595091,v1:172.21.15.114:6811/4275595091] conn(0x7fcfbc0ad4b0 0x7fcfbc0af8d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T10:25:16.886 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:25:16.868+0000 7fcfc77fe700 1 -- 172.21.15.114:0/1331307751 --> [v2:172.21.15.114:6810/4275595091,v1:172.21.15.114:6811/4275595091] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7fcfcc00eec0 con 0x7fcfbc0ad4b0 2024-04-23T10:25:16.886 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:25:16.868+0000 7fcfc77fe700 1 -- 172.21.15.114:0/1331307751 <== mon.0 v2:172.21.15.114:3300/0 6 ==== mon_get_version_reply(handle=1 version=87) ==== 24+0+0 (secure 0 0 0) 0x7fcfcc0d0ab0 con 0x7fcfd0101930 2024-04-23T10:25:16.886 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:25:16.868+0000 7fcfd682f700 1 --2- 172.21.15.114:0/1331307751 >> [v2:172.21.15.114:6810/4275595091,v1:172.21.15.114:6811/4275595091] conn(0x7fcfbc0ad4b0 0x7fcfbc0af8d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T10:25:16.886 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:25:16.868+0000 7fcfd582d700 1 --2- 172.21.15.114:0/1331307751 >> [v2:172.21.15.114:6800/465333808,v1:172.21.15.114:6801/465333808] conn(0x7fcfbc0a3cd0 0x7fcfbc0a6190 secure :-1 s=READY pgs=187 cs=0 l=1 rev1=1 crypto rx=0x7fcfd0078ff0 tx=0x7fcfc0008040 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T10:25:16.886 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:25:16.869+0000 7fcfd682f700 1 --2- 172.21.15.114:0/1331307751 >> [v2:172.21.15.114:6810/4275595091,v1:172.21.15.114:6811/4275595091] conn(0x7fcfbc0ad4b0 0x7fcfbc0af8d0 crc :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.3 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T10:25:16.887 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:25:16.869+0000 7fcfc77fe700 1 -- 172.21.15.114:0/1331307751 <== osd.3 v2:172.21.15.114:6810/4275595091 1 ==== command_reply(tid 1: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7fcfcc00eec0 con 0x7fcfbc0ad4b0 2024-04-23T10:25:16.887 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:25:16.886+0000 7fcfd8292700 1 -- 172.21.15.114:0/1331307751 --> [v2:172.21.15.114:6810/4275595091,v1:172.21.15.114:6811/4275595091] -- command(tid 2: {"prefix": "flush_pg_stats"}) -- 0x7fcfd0063590 con 0x7fcfbc0ad4b0 2024-04-23T10:25:16.887 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:25:16.886+0000 7fcfc77fe700 1 -- 172.21.15.114:0/1331307751 <== osd.3 v2:172.21.15.114:6810/4275595091 2 ==== command_reply(tid 2: 0 ) ==== 8+0+12 (crc 0 0 0) 0x7fcfd0063590 con 0x7fcfbc0ad4b0 2024-04-23T10:25:16.888 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:25:16.886+0000 7fcfd8292700 1 -- 172.21.15.114:0/1331307751 >> [v2:172.21.15.114:6810/4275595091,v1:172.21.15.114:6811/4275595091] conn(0x7fcfbc0ad4b0 msgr2=0x7fcfbc0af8d0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T10:25:16.888 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:25:16.886+0000 7fcfd8292700 1 --2- 172.21.15.114:0/1331307751 >> [v2:172.21.15.114:6810/4275595091,v1:172.21.15.114:6811/4275595091] conn(0x7fcfbc0ad4b0 0x7fcfbc0af8d0 crc :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:25:16.888 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:25:16.887+0000 7fcfd8292700 1 -- 172.21.15.114:0/1331307751 >> [v2:172.21.15.114:6800/465333808,v1:172.21.15.114:6801/465333808] conn(0x7fcfbc0a3cd0 msgr2=0x7fcfbc0a6190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T10:25:16.888 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:25:16.887+0000 7fcfd8292700 1 --2- 172.21.15.114:0/1331307751 >> [v2:172.21.15.114:6800/465333808,v1:172.21.15.114:6801/465333808] conn(0x7fcfbc0a3cd0 0x7fcfbc0a6190 secure :-1 s=READY pgs=187 cs=0 l=1 rev1=1 crypto rx=0x7fcfd0078ff0 tx=0x7fcfc0008040 comp rx=0 tx=0).stop 2024-04-23T10:25:16.888 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:25:16.887+0000 7fcfd8292700 1 -- 172.21.15.114:0/1331307751 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fcfd0101930 msgr2=0x7fcfd01034e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T10:25:16.888 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:25:16.887+0000 7fcfd8292700 1 --2- 172.21.15.114:0/1331307751 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fcfd0101930 0x7fcfd01034e0 secure :-1 s=READY pgs=257 cs=0 l=1 rev1=1 crypto rx=0x7fcfcc00c970 tx=0x7fcfcc00cc80 comp rx=0 tx=0).stop 2024-04-23T10:25:16.888 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:25:16.887+0000 7fcfd682f700 1 -- 172.21.15.114:0/1331307751 reap_dead start 2024-04-23T10:25:16.888 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:25:16.887+0000 7fcfd8292700 1 -- 172.21.15.114:0/1331307751 shutdown_connections 2024-04-23T10:25:16.888 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:25:16.887+0000 7fcfd8292700 1 -- 172.21.15.114:0/1331307751 >> 172.21.15.114:0/1331307751 conn(0x7fcfd00788b0 msgr2=0x7fcfd006b330 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T10:25:16.888 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:25:16.887+0000 7fcfd8292700 1 -- 172.21.15.114:0/1331307751 shutdown_connections 2024-04-23T10:25:16.888 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:25:16.887+0000 7fcfd8292700 1 -- 172.21.15.114:0/1331307751 wait complete. 2024-04-23T10:25:17.204 INFO:journalctl@ceph.mon.a.smithi114.stdout:Apr 23 10:25:17 smithi114 ceph-mon[29255]: pgmap v302: 97 pgs: 97 active+clean; 639 KiB data, 370 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 426 B/s wr, 6 op/s 2024-04-23T10:25:17.328 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 23 10:25:17 smithi204 ceph-mon[32649]: pgmap v302: 97 pgs: 97 active+clean; 639 KiB data, 370 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 426 B/s wr, 6 op/s 2024-04-23T10:25:17.492 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 23 10:25:17 smithi119 ceph-mon[32608]: pgmap v302: 97 pgs: 97 active+clean; 639 KiB data, 370 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 426 B/s wr, 6 op/s 2024-04-23T10:25:17.550 INFO:teuthology.orchestra.run.smithi114.stdout:55834574917 2024-04-23T10:25:17.551 DEBUG:teuthology.orchestra.run.smithi114:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d69f13d827952bb07bb9b7973bca14f97d764d6f shell --fsid aa240572-0159-11ef-bc93-c7b262605968 -- ceph osd last-stat-seq osd.1 2024-04-23T10:25:19.115 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:25:19.114+0000 7ff8c5940700 1 -- 172.21.15.114:0/604073115 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7ff8c01034b0 msgr2=0x7ff8c010f8e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T10:25:19.115 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:25:19.114+0000 7ff8c5940700 1 --2- 172.21.15.114:0/604073115 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7ff8c01034b0 0x7ff8c010f8e0 secure :-1 s=READY pgs=258 cs=0 l=1 rev1=1 crypto rx=0x7ff8b8009990 tx=0x7ff8b8009ca0 comp rx=0 tx=0).stop 2024-04-23T10:25:19.115 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:25:19.114+0000 7ff8c5940700 1 -- 172.21.15.114:0/604073115 shutdown_connections 2024-04-23T10:25:19.115 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:25:19.114+0000 7ff8c5940700 1 --2- 172.21.15.114:0/604073115 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7ff8c01034b0 0x7ff8c010f8e0 unknown :-1 s=CLOSED pgs=258 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:25:19.115 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:25:19.114+0000 7ff8c5940700 1 --2- 172.21.15.114:0/604073115 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7ff8c0108a80 0x7ff8c0108e60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:25:19.115 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:25:19.114+0000 7ff8c5940700 1 --2- 172.21.15.114:0/604073115 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7ff8c0102af0 0x7ff8c0102f70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:25:19.115 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:25:19.114+0000 7ff8c5940700 1 -- 172.21.15.114:0/604073115 >> 172.21.15.114:0/604073115 conn(0x7ff8c0079ad0 msgr2=0x7ff8c0079ee0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T10:25:19.116 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:25:19.114+0000 7ff8c5940700 1 -- 172.21.15.114:0/604073115 shutdown_connections 2024-04-23T10:25:19.116 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:25:19.114+0000 7ff8c5940700 1 -- 172.21.15.114:0/604073115 wait complete. 2024-04-23T10:25:19.116 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:25:19.115+0000 7ff8c5940700 1 Processor -- start 2024-04-23T10:25:19.116 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:25:19.115+0000 7ff8c5940700 1 -- start start 2024-04-23T10:25:19.116 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:25:19.115+0000 7ff8c5940700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7ff8c0102af0 0x7ff8c0074d40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T10:25:19.116 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:25:19.115+0000 7ff8c5940700 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7ff8c01034b0 0x7ff8c0075280 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T10:25:19.116 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:25:19.115+0000 7ff8c5940700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7ff8c0108a80 0x7ff8c00757c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T10:25:19.116 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:25:19.115+0000 7ff8c5940700 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x7ff8c0112c30 con 0x7ff8c0102af0 2024-04-23T10:25:19.116 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:25:19.115+0000 7ff8c5940700 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 -- 0x7ff8c0112ab0 con 0x7ff8c0108a80 2024-04-23T10:25:19.116 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:25:19.115+0000 7ff8c5940700 1 -- --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_getmap magic: 0 -- 0x7ff8c0112db0 con 0x7ff8c01034b0 2024-04-23T10:25:19.116 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:25:19.115+0000 7ff8beffd700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7ff8c0102af0 0x7ff8c0074d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T10:25:19.116 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:25:19.115+0000 7ff8beffd700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7ff8c0102af0 0x7ff8c0074d40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.114:36272/0 (socket says 172.21.15.114:36272) 2024-04-23T10:25:19.116 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:25:19.115+0000 7ff8beffd700 1 -- 172.21.15.114:0/1908005862 learned_addr learned my addr 172.21.15.114:0/1908005862 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-04-23T10:25:19.116 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:25:19.115+0000 7ff8be7fc700 1 --2- 172.21.15.114:0/1908005862 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7ff8c01034b0 0x7ff8c0075280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T10:25:19.116 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:25:19.115+0000 7ff8bf7fe700 1 --2- 172.21.15.114:0/1908005862 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7ff8c0108a80 0x7ff8c00757c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T10:25:19.117 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:25:19.115+0000 7ff8beffd700 1 -- 172.21.15.114:0/1908005862 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7ff8c01034b0 msgr2=0x7ff8c0075280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T10:25:19.117 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:25:19.115+0000 7ff8beffd700 1 --2- 172.21.15.114:0/1908005862 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7ff8c01034b0 0x7ff8c0075280 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:25:19.117 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:25:19.115+0000 7ff8beffd700 1 -- 172.21.15.114:0/1908005862 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7ff8c0108a80 msgr2=0x7ff8c00757c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T10:25:19.117 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:25:19.115+0000 7ff8beffd700 1 --2- 172.21.15.114:0/1908005862 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7ff8c0108a80 0x7ff8c00757c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:25:19.117 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:25:19.115+0000 7ff8beffd700 1 -- 172.21.15.114:0/1908005862 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff8c01a7640 con 0x7ff8c0102af0 2024-04-23T10:25:19.117 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:25:19.115+0000 7ff8bf7fe700 1 --2- 172.21.15.114:0/1908005862 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7ff8c0108a80 0x7ff8c00757c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-23T10:25:19.117 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:25:19.116+0000 7ff8beffd700 1 --2- 172.21.15.114:0/1908005862 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7ff8c0102af0 0x7ff8c0074d40 secure :-1 s=READY pgs=259 cs=0 l=1 rev1=1 crypto rx=0x7ff8b0007f10 tx=0x7ff8b000d3b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T10:25:19.117 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:25:19.116+0000 7ff8c493e700 1 -- 172.21.15.114:0/1908005862 <== mon.0 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff8b000dbd0 con 0x7ff8c0102af0 2024-04-23T10:25:19.117 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:25:19.116+0000 7ff8c5940700 1 -- 172.21.15.114:0/1908005862 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7ff8c01a7930 con 0x7ff8c0102af0 2024-04-23T10:25:19.117 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:25:19.116+0000 7ff8c493e700 1 -- 172.21.15.114:0/1908005862 <== mon.0 v2:172.21.15.114:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7ff8b000dd70 con 0x7ff8c0102af0 2024-04-23T10:25:19.117 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:25:19.116+0000 7ff8c493e700 1 -- 172.21.15.114:0/1908005862 <== mon.0 v2:172.21.15.114:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff8b0005a10 con 0x7ff8c0102af0 2024-04-23T10:25:19.117 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:25:19.116+0000 7ff8c5940700 1 -- 172.21.15.114:0/1908005862 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7ff8c01a7e20 con 0x7ff8c0102af0 2024-04-23T10:25:19.119 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:25:19.117+0000 7ff8c5940700 1 -- 172.21.15.114:0/1908005862 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_get_version(what=osdmap handle=1) -- 0x7ff8c0063110 con 0x7ff8c0102af0 2024-04-23T10:25:19.119 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:25:19.118+0000 7ff8c493e700 1 -- 172.21.15.114:0/1908005862 <== mon.0 v2:172.21.15.114:3300/0 4 ==== mgrmap(e 17) ==== 138698+0+0 (secure 0 0 0) 0x7ff8b001a020 con 0x7ff8c0102af0 2024-04-23T10:25:19.119 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:25:19.118+0000 7ff8c493e700 1 --2- 172.21.15.114:0/1908005862 >> [v2:172.21.15.114:6800/465333808,v1:172.21.15.114:6801/465333808] conn(0x7ff8ac0a3cd0 0x7ff8ac0a6190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T10:25:19.119 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:25:19.118+0000 7ff8be7fc700 1 --2- 172.21.15.114:0/1908005862 >> [v2:172.21.15.114:6800/465333808,v1:172.21.15.114:6801/465333808] conn(0x7ff8ac0a3cd0 0x7ff8ac0a6190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T10:25:19.119 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:25:19.118+0000 7ff8c493e700 1 -- 172.21.15.114:0/1908005862 <== mon.0 v2:172.21.15.114:3300/0 5 ==== osd_map(87..87 src has 1..87) ==== 7979+0+0 (secure 0 0 0) 0x7ff8b00d0220 con 0x7ff8c0102af0 2024-04-23T10:25:19.119 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:25:19.119+0000 7ff8c493e700 1 --2- 172.21.15.114:0/1908005862 >> [v2:172.21.15.204:6816/1426318231,v1:172.21.15.204:6817/1426318231] conn(0x7ff8ac0ad4b0 0x7ff8ac0af8d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T10:25:19.119 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:25:19.119+0000 7ff8c493e700 1 -- 172.21.15.114:0/1908005862 --> [v2:172.21.15.204:6816/1426318231,v1:172.21.15.204:6817/1426318231] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7ff8b0017e70 con 0x7ff8ac0ad4b0 2024-04-23T10:25:19.119 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:25:19.119+0000 7ff8c493e700 1 -- 172.21.15.114:0/1908005862 <== mon.0 v2:172.21.15.114:3300/0 6 ==== mon_get_version_reply(handle=1 version=87) ==== 24+0+0 (secure 0 0 0) 0x7ff8b00d0700 con 0x7ff8c0102af0 2024-04-23T10:25:19.119 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:25:19.119+0000 7ff8be7fc700 1 --2- 172.21.15.114:0/1908005862 >> [v2:172.21.15.114:6800/465333808,v1:172.21.15.114:6801/465333808] conn(0x7ff8ac0a3cd0 0x7ff8ac0a6190 secure :-1 s=READY pgs=188 cs=0 l=1 rev1=1 crypto rx=0x7ff8c007a0b0 tx=0x7ff8a8009260 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T10:25:19.119 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:25:19.119+0000 7ff8bf7fe700 1 --2- 172.21.15.114:0/1908005862 >> [v2:172.21.15.204:6816/1426318231,v1:172.21.15.204:6817/1426318231] conn(0x7ff8ac0ad4b0 0x7ff8ac0af8d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T10:25:19.119 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:25:19.119+0000 7ff8bf7fe700 1 --2- 172.21.15.114:0/1908005862 >> [v2:172.21.15.204:6816/1426318231,v1:172.21.15.204:6817/1426318231] conn(0x7ff8ac0ad4b0 0x7ff8ac0af8d0 crc :-1 s=READY pgs=14 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.8 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T10:25:19.119 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:25:19.120+0000 7ff8c493e700 1 -- 172.21.15.114:0/1908005862 <== osd.8 v2:172.21.15.204:6816/1426318231 1 ==== command_reply(tid 1: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7ff8b0017e70 con 0x7ff8ac0ad4b0 2024-04-23T10:25:19.135 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:25:19.136+0000 7ff8c5940700 1 -- 172.21.15.114:0/1908005862 --> [v2:172.21.15.204:6816/1426318231,v1:172.21.15.204:6817/1426318231] -- command(tid 2: {"prefix": "flush_pg_stats"}) -- 0x7ff8c0063590 con 0x7ff8ac0ad4b0 2024-04-23T10:25:19.135 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:25:19.137+0000 7ff8c493e700 1 -- 172.21.15.114:0/1908005862 <== osd.8 v2:172.21.15.204:6816/1426318231 2 ==== command_reply(tid 2: 0 ) ==== 8+0+12 (crc 0 0 0) 0x7ff8c0063590 con 0x7ff8ac0ad4b0 2024-04-23T10:25:19.136 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:25:19.137+0000 7ff8c5940700 1 -- 172.21.15.114:0/1908005862 >> [v2:172.21.15.204:6816/1426318231,v1:172.21.15.204:6817/1426318231] conn(0x7ff8ac0ad4b0 msgr2=0x7ff8ac0af8d0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T10:25:19.136 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:25:19.137+0000 7ff8c5940700 1 --2- 172.21.15.114:0/1908005862 >> [v2:172.21.15.204:6816/1426318231,v1:172.21.15.204:6817/1426318231] conn(0x7ff8ac0ad4b0 0x7ff8ac0af8d0 crc :-1 s=READY pgs=14 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:25:19.136 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:25:19.137+0000 7ff8c5940700 1 -- 172.21.15.114:0/1908005862 >> [v2:172.21.15.114:6800/465333808,v1:172.21.15.114:6801/465333808] conn(0x7ff8ac0a3cd0 msgr2=0x7ff8ac0a6190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T10:25:19.137 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:25:19.137+0000 7ff8c5940700 1 --2- 172.21.15.114:0/1908005862 >> [v2:172.21.15.114:6800/465333808,v1:172.21.15.114:6801/465333808] conn(0x7ff8ac0a3cd0 0x7ff8ac0a6190 secure :-1 s=READY pgs=188 cs=0 l=1 rev1=1 crypto rx=0x7ff8c007a0b0 tx=0x7ff8a8009260 comp rx=0 tx=0).stop 2024-04-23T10:25:19.137 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:25:19.137+0000 7ff8c5940700 1 -- 172.21.15.114:0/1908005862 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7ff8c0102af0 msgr2=0x7ff8c0074d40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T10:25:19.137 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:25:19.137+0000 7ff8c5940700 1 --2- 172.21.15.114:0/1908005862 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7ff8c0102af0 0x7ff8c0074d40 secure :-1 s=READY pgs=259 cs=0 l=1 rev1=1 crypto rx=0x7ff8b0007f10 tx=0x7ff8b000d3b0 comp rx=0 tx=0).stop 2024-04-23T10:25:19.137 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:25:19.137+0000 7ff8bf7fe700 1 -- 172.21.15.114:0/1908005862 reap_dead start 2024-04-23T10:25:19.137 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:25:19.137+0000 7ff8c5940700 1 -- 172.21.15.114:0/1908005862 shutdown_connections 2024-04-23T10:25:19.137 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:25:19.137+0000 7ff8c5940700 1 -- 172.21.15.114:0/1908005862 >> 172.21.15.114:0/1908005862 conn(0x7ff8c0079ad0 msgr2=0x7ff8c0101790 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T10:25:19.137 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:25:19.137+0000 7ff8c5940700 1 -- 172.21.15.114:0/1908005862 shutdown_connections 2024-04-23T10:25:19.137 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:25:19.137+0000 7ff8c5940700 1 -- 172.21.15.114:0/1908005862 wait complete. 2024-04-23T10:25:19.328 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 23 10:25:19 smithi204 ceph-mon[32649]: pgmap v303: 97 pgs: 97 active+clean; 639 KiB data, 370 MiB used, 1.0 TiB / 1.0 TiB avail; 3.9 KiB/s rd, 426 B/s wr, 7 op/s 2024-04-23T10:25:19.492 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 23 10:25:19 smithi119 ceph-mon[32608]: pgmap v303: 97 pgs: 97 active+clean; 639 KiB data, 370 MiB used, 1.0 TiB / 1.0 TiB avail; 3.9 KiB/s rd, 426 B/s wr, 7 op/s 2024-04-23T10:25:19.504 INFO:journalctl@ceph.mon.a.smithi114.stdout:Apr 23 10:25:19 smithi114 ceph-mon[29255]: pgmap v303: 97 pgs: 97 active+clean; 639 KiB data, 370 MiB used, 1.0 TiB / 1.0 TiB avail; 3.9 KiB/s rd, 426 B/s wr, 7 op/s 2024-04-23T10:25:19.559 INFO:teuthology.orchestra.run.smithi114.stdout:317827579928 2024-04-23T10:25:19.560 DEBUG:teuthology.orchestra.run.smithi114:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d69f13d827952bb07bb9b7973bca14f97d764d6f shell --fsid aa240572-0159-11ef-bc93-c7b262605968 -- ceph osd last-stat-seq osd.10 2024-04-23T10:25:21.254 INFO:journalctl@ceph.mon.a.smithi114.stdout:Apr 23 10:25:21 smithi114 ceph-mon[29255]: pgmap v304: 97 pgs: 97 active+clean; 639 KiB data, 370 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 426 B/s wr, 6 op/s 2024-04-23T10:25:21.328 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 23 10:25:21 smithi204 ceph-mon[32649]: pgmap v304: 97 pgs: 97 active+clean; 639 KiB data, 370 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 426 B/s wr, 6 op/s 2024-04-23T10:25:21.492 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 23 10:25:21 smithi119 ceph-mon[32608]: pgmap v304: 97 pgs: 97 active+clean; 639 KiB data, 370 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 426 B/s wr, 6 op/s 2024-04-23T10:25:22.173 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:25:22.171+0000 7f2d323ed700 1 -- 172.21.15.114:0/2173158143 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f2d2c06a750 msgr2=0x7f2d2c06abb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T10:25:22.173 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:25:22.171+0000 7f2d323ed700 1 --2- 172.21.15.114:0/2173158143 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f2d2c06a750 0x7f2d2c06abb0 secure :-1 s=READY pgs=262 cs=0 l=1 rev1=1 crypto rx=0x7f2d20009990 tx=0x7f2d20009ca0 comp rx=0 tx=0).stop 2024-04-23T10:25:22.173 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:25:22.171+0000 7f2d323ed700 1 -- 172.21.15.114:0/2173158143 shutdown_connections 2024-04-23T10:25:22.173 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:25:22.171+0000 7f2d323ed700 1 --2- 172.21.15.114:0/2173158143 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f2d2c108a30 0x7f2d2c108e10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:25:22.173 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:25:22.171+0000 7f2d323ed700 1 --2- 172.21.15.114:0/2173158143 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f2d2c06b0f0 0x7f2d2c10f900 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:25:22.173 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:25:22.171+0000 7f2d323ed700 1 --2- 172.21.15.114:0/2173158143 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f2d2c06a750 0x7f2d2c06abb0 unknown :-1 s=CLOSED pgs=262 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:25:22.174 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:25:22.171+0000 7f2d323ed700 1 -- 172.21.15.114:0/2173158143 >> 172.21.15.114:0/2173158143 conn(0x7f2d2c0788f0 msgr2=0x7f2d2c078d00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T10:25:22.174 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:25:22.171+0000 7f2d323ed700 1 -- 172.21.15.114:0/2173158143 shutdown_connections 2024-04-23T10:25:22.174 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:25:22.171+0000 7f2d323ed700 1 -- 172.21.15.114:0/2173158143 wait complete. 2024-04-23T10:25:22.174 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:25:22.171+0000 7f2d323ed700 1 Processor -- start 2024-04-23T10:25:22.174 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:25:22.171+0000 7f2d323ed700 1 -- start start 2024-04-23T10:25:22.174 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:25:22.172+0000 7f2d323ed700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f2d2c06a750 0x7f2d2c199280 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T10:25:22.174 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:25:22.172+0000 7f2d323ed700 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f2d2c06b0f0 0x7f2d2c1997c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T10:25:22.174 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:25:22.172+0000 7f2d323ed700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f2d2c108a30 0x7f2d2c19db50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T10:25:22.174 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:25:22.172+0000 7f2d323ed700 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x7f2d2c112cb0 con 0x7f2d2c108a30 2024-04-23T10:25:22.174 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:25:22.172+0000 7f2d323ed700 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 -- 0x7f2d2c112b30 con 0x7f2d2c06a750 2024-04-23T10:25:22.174 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:25:22.172+0000 7f2d323ed700 1 -- --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_getmap magic: 0 -- 0x7f2d2c112e30 con 0x7f2d2c06b0f0 2024-04-23T10:25:22.174 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:25:22.172+0000 7f2d3098a700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f2d2c108a30 0x7f2d2c19db50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T10:25:22.175 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:25:22.172+0000 7f2d3098a700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f2d2c108a30 0x7f2d2c19db50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.114:36300/0 (socket says 172.21.15.114:36300) 2024-04-23T10:25:22.175 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:25:22.172+0000 7f2d3098a700 1 -- 172.21.15.114:0/523557766 learned_addr learned my addr 172.21.15.114:0/523557766 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-04-23T10:25:22.175 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:25:22.172+0000 7f2d2b7fe700 1 --2- 172.21.15.114:0/523557766 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f2d2c06b0f0 0x7f2d2c1997c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T10:25:22.175 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:25:22.172+0000 7f2d2bfff700 1 --2- 172.21.15.114:0/523557766 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f2d2c06a750 0x7f2d2c199280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T10:25:22.175 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:25:22.172+0000 7f2d3098a700 1 -- 172.21.15.114:0/523557766 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f2d2c06b0f0 msgr2=0x7f2d2c1997c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T10:25:22.175 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:25:22.172+0000 7f2d3098a700 1 --2- 172.21.15.114:0/523557766 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f2d2c06b0f0 0x7f2d2c1997c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:25:22.175 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:25:22.172+0000 7f2d3098a700 1 -- 172.21.15.114:0/523557766 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f2d2c06a750 msgr2=0x7f2d2c199280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T10:25:22.175 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:25:22.172+0000 7f2d3098a700 1 --2- 172.21.15.114:0/523557766 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f2d2c06a750 0x7f2d2c199280 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:25:22.175 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:25:22.172+0000 7f2d3098a700 1 -- 172.21.15.114:0/523557766 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f2d2c19e290 con 0x7f2d2c108a30 2024-04-23T10:25:22.175 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:25:22.172+0000 7f2d3098a700 1 --2- 172.21.15.114:0/523557766 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f2d2c108a30 0x7f2d2c19db50 secure :-1 s=READY pgs=263 cs=0 l=1 rev1=1 crypto rx=0x7f2d2400b970 tx=0x7f2d2400bc80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T10:25:22.175 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:25:22.172+0000 7f2d297fa700 1 -- 172.21.15.114:0/523557766 <== mon.0 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2d2400d860 con 0x7f2d2c108a30 2024-04-23T10:25:22.175 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:25:22.172+0000 7f2d323ed700 1 -- 172.21.15.114:0/523557766 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f2d2c19e580 con 0x7f2d2c108a30 2024-04-23T10:25:22.175 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:25:22.172+0000 7f2d297fa700 1 -- 172.21.15.114:0/523557766 <== mon.0 v2:172.21.15.114:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f2d2400e430 con 0x7f2d2c108a30 2024-04-23T10:25:22.175 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:25:22.172+0000 7f2d297fa700 1 -- 172.21.15.114:0/523557766 <== mon.0 v2:172.21.15.114:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2d24017660 con 0x7f2d2c108a30 2024-04-23T10:25:22.175 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:25:22.172+0000 7f2d323ed700 1 -- 172.21.15.114:0/523557766 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f2d2c1a5e00 con 0x7f2d2c108a30 2024-04-23T10:25:22.176 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:25:22.174+0000 7f2d323ed700 1 -- 172.21.15.114:0/523557766 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_get_version(what=osdmap handle=1) -- 0x7f2d2c0631d0 con 0x7f2d2c108a30 2024-04-23T10:25:22.177 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:25:22.175+0000 7f2d297fa700 1 -- 172.21.15.114:0/523557766 <== mon.0 v2:172.21.15.114:3300/0 4 ==== mgrmap(e 17) ==== 138698+0+0 (secure 0 0 0) 0x7f2d2400da00 con 0x7f2d2c108a30 2024-04-23T10:25:22.177 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:25:22.175+0000 7f2d297fa700 1 --2- 172.21.15.114:0/523557766 >> [v2:172.21.15.114:6800/465333808,v1:172.21.15.114:6801/465333808] conn(0x7f2d140a3cd0 0x7f2d140a6190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T10:25:22.177 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:25:22.176+0000 7f2d2bfff700 1 --2- 172.21.15.114:0/523557766 >> [v2:172.21.15.114:6800/465333808,v1:172.21.15.114:6801/465333808] conn(0x7f2d140a3cd0 0x7f2d140a6190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T10:25:22.177 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:25:22.176+0000 7f2d297fa700 1 -- 172.21.15.114:0/523557766 <== mon.0 v2:172.21.15.114:3300/0 5 ==== osd_map(87..87 src has 1..87) ==== 7979+0+0 (secure 0 0 0) 0x7f2d240d00c0 con 0x7f2d2c108a30 2024-04-23T10:25:22.177 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:25:22.176+0000 7f2d297fa700 1 --2- 172.21.15.114:0/523557766 >> [v2:172.21.15.114:6802/736366878,v1:172.21.15.114:6803/736366878] conn(0x7f2d140ad4b0 0x7f2d140af8d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T10:25:22.177 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:25:22.176+0000 7f2d297fa700 1 -- 172.21.15.114:0/523557766 --> [v2:172.21.15.114:6802/736366878,v1:172.21.15.114:6803/736366878] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f2d24010ec0 con 0x7f2d140ad4b0 2024-04-23T10:25:22.177 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:25:22.176+0000 7f2d297fa700 1 -- 172.21.15.114:0/523557766 <== mon.0 v2:172.21.15.114:3300/0 6 ==== mon_get_version_reply(handle=1 version=87) ==== 24+0+0 (secure 0 0 0) 0x7f2d240d7050 con 0x7f2d2c108a30 2024-04-23T10:25:22.177 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:25:22.176+0000 7f2d2b7fe700 1 --2- 172.21.15.114:0/523557766 >> [v2:172.21.15.114:6802/736366878,v1:172.21.15.114:6803/736366878] conn(0x7f2d140ad4b0 0x7f2d140af8d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T10:25:22.177 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:25:22.176+0000 7f2d2bfff700 1 --2- 172.21.15.114:0/523557766 >> [v2:172.21.15.114:6800/465333808,v1:172.21.15.114:6801/465333808] conn(0x7f2d140a3cd0 0x7f2d140a6190 secure :-1 s=READY pgs=190 cs=0 l=1 rev1=1 crypto rx=0x7f2d1c009a10 tx=0x7f2d1c009190 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T10:25:22.177 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:25:22.176+0000 7f2d2b7fe700 1 --2- 172.21.15.114:0/523557766 >> [v2:172.21.15.114:6802/736366878,v1:172.21.15.114:6803/736366878] conn(0x7f2d140ad4b0 0x7f2d140af8d0 crc :-1 s=READY pgs=9 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T10:25:22.177 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:25:22.177+0000 7f2d297fa700 1 -- 172.21.15.114:0/523557766 <== osd.0 v2:172.21.15.114:6802/736366878 1 ==== command_reply(tid 1: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7f2d24010ec0 con 0x7f2d140ad4b0 2024-04-23T10:25:22.192 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:25:22.194+0000 7f2d323ed700 1 -- 172.21.15.114:0/523557766 --> [v2:172.21.15.114:6802/736366878,v1:172.21.15.114:6803/736366878] -- command(tid 2: {"prefix": "flush_pg_stats"}) -- 0x7f2d2c063650 con 0x7f2d140ad4b0 2024-04-23T10:25:22.193 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:25:22.194+0000 7f2d297fa700 1 -- 172.21.15.114:0/523557766 <== osd.0 v2:172.21.15.114:6802/736366878 2 ==== command_reply(tid 2: 0 ) ==== 8+0+11 (crc 0 0 0) 0x7f2d2c063650 con 0x7f2d140ad4b0 2024-04-23T10:25:22.193 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:25:22.194+0000 7f2d323ed700 1 -- 172.21.15.114:0/523557766 >> [v2:172.21.15.114:6802/736366878,v1:172.21.15.114:6803/736366878] conn(0x7f2d140ad4b0 msgr2=0x7f2d140af8d0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T10:25:22.194 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:25:22.194+0000 7f2d323ed700 1 --2- 172.21.15.114:0/523557766 >> [v2:172.21.15.114:6802/736366878,v1:172.21.15.114:6803/736366878] conn(0x7f2d140ad4b0 0x7f2d140af8d0 crc :-1 s=READY pgs=9 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:25:22.194 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:25:22.194+0000 7f2d323ed700 1 -- 172.21.15.114:0/523557766 >> [v2:172.21.15.114:6800/465333808,v1:172.21.15.114:6801/465333808] conn(0x7f2d140a3cd0 msgr2=0x7f2d140a6190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T10:25:22.194 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:25:22.194+0000 7f2d323ed700 1 --2- 172.21.15.114:0/523557766 >> [v2:172.21.15.114:6800/465333808,v1:172.21.15.114:6801/465333808] conn(0x7f2d140a3cd0 0x7f2d140a6190 secure :-1 s=READY pgs=190 cs=0 l=1 rev1=1 crypto rx=0x7f2d1c009a10 tx=0x7f2d1c009190 comp rx=0 tx=0).stop 2024-04-23T10:25:22.194 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:25:22.194+0000 7f2d323ed700 1 -- 172.21.15.114:0/523557766 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f2d2c108a30 msgr2=0x7f2d2c19db50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T10:25:22.194 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:25:22.194+0000 7f2d323ed700 1 --2- 172.21.15.114:0/523557766 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f2d2c108a30 0x7f2d2c19db50 secure :-1 s=READY pgs=263 cs=0 l=1 rev1=1 crypto rx=0x7f2d2400b970 tx=0x7f2d2400bc80 comp rx=0 tx=0).stop 2024-04-23T10:25:22.194 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:25:22.194+0000 7f2d3098a700 1 -- 172.21.15.114:0/523557766 reap_dead start 2024-04-23T10:25:22.194 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:25:22.194+0000 7f2d323ed700 1 -- 172.21.15.114:0/523557766 shutdown_connections 2024-04-23T10:25:22.194 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:25:22.194+0000 7f2d323ed700 1 -- 172.21.15.114:0/523557766 >> 172.21.15.114:0/523557766 conn(0x7f2d2c0788f0 msgr2=0x7f2d2c0ffb50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T10:25:22.194 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:25:22.194+0000 7f2d323ed700 1 -- 172.21.15.114:0/523557766 shutdown_connections 2024-04-23T10:25:22.194 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:25:22.194+0000 7f2d323ed700 1 -- 172.21.15.114:0/523557766 wait complete. 2024-04-23T10:25:22.609 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:25:22.385+0000 7f196fb92700 1 -- 172.21.15.114:0/911504141 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f19681023a0 msgr2=0x7f1968102780 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T10:25:22.609 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:25:22.385+0000 7f196fb92700 1 --2- 172.21.15.114:0/911504141 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f19681023a0 0x7f1968102780 secure :-1 s=READY pgs=264 cs=0 l=1 rev1=1 crypto rx=0x7f19640099c0 tx=0x7f1964009cd0 comp rx=0 tx=0).stop 2024-04-23T10:25:22.609 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:25:22.385+0000 7f196fb92700 1 -- 172.21.15.114:0/911504141 shutdown_connections 2024-04-23T10:25:22.609 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:25:22.385+0000 7f196fb92700 1 --2- 172.21.15.114:0/911504141 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f1968106cc0 0x7f196810e8d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:25:22.609 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:25:22.385+0000 7f196fb92700 1 --2- 172.21.15.114:0/911504141 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f19681023a0 0x7f1968102780 unknown :-1 s=CLOSED pgs=264 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:25:22.609 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:25:22.385+0000 7f196fb92700 1 --2- 172.21.15.114:0/911504141 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f1968102d50 0x7f19681065e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:25:22.609 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:25:22.385+0000 7f196fb92700 1 -- 172.21.15.114:0/911504141 >> 172.21.15.114:0/911504141 conn(0x7f1968077fb0 msgr2=0x7f19680ff360 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T10:25:22.609 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:25:22.385+0000 7f196fb92700 1 -- 172.21.15.114:0/911504141 shutdown_connections 2024-04-23T10:25:22.610 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:25:22.385+0000 7f196fb92700 1 -- 172.21.15.114:0/911504141 wait complete. 2024-04-23T10:25:22.610 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:25:22.385+0000 7f196fb92700 1 Processor -- start 2024-04-23T10:25:22.610 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:25:22.386+0000 7f196fb92700 1 -- start start 2024-04-23T10:25:22.610 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:25:22.386+0000 7f196fb92700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f19681023a0 0x7f1968199ff0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T10:25:22.610 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:25:22.386+0000 7f196fb92700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f1968102d50 0x7f196819a530 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T10:25:22.610 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:25:22.386+0000 7f196fb92700 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f1968106cc0 0x7f19681a5070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T10:25:22.610 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:25:22.386+0000 7f196fb92700 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x7f1968111bb0 con 0x7f19681023a0 2024-04-23T10:25:22.610 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:25:22.386+0000 7f196fb92700 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 -- 0x7f1968111a30 con 0x7f1968102d50 2024-04-23T10:25:22.610 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:25:22.386+0000 7f196fb92700 1 -- --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_getmap magic: 0 -- 0x7f1968111d30 con 0x7f1968106cc0 2024-04-23T10:25:22.610 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:25:22.386+0000 7f196d92e700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f19681023a0 0x7f1968199ff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T10:25:22.610 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:25:22.386+0000 7f196d92e700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f19681023a0 0x7f1968199ff0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.114:36318/0 (socket says 172.21.15.114:36318) 2024-04-23T10:25:22.610 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:25:22.386+0000 7f196d92e700 1 -- 172.21.15.114:0/1832178810 learned_addr learned my addr 172.21.15.114:0/1832178810 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-04-23T10:25:22.610 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:25:22.386+0000 7f196e12f700 1 --2- 172.21.15.114:0/1832178810 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f1968106cc0 0x7f19681a5070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T10:25:22.610 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:25:22.386+0000 7f196d92e700 1 -- 172.21.15.114:0/1832178810 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f1968106cc0 msgr2=0x7f19681a5070 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T10:25:22.611 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:25:22.386+0000 7f196d12d700 1 --2- 172.21.15.114:0/1832178810 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f1968102d50 0x7f196819a530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T10:25:22.611 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:25:22.386+0000 7f196d92e700 1 --2- 172.21.15.114:0/1832178810 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f1968106cc0 0x7f19681a5070 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:25:22.611 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:25:22.386+0000 7f196d92e700 1 -- 172.21.15.114:0/1832178810 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f1968102d50 msgr2=0x7f196819a530 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T10:25:22.611 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:25:22.386+0000 7f196d92e700 1 --2- 172.21.15.114:0/1832178810 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f1968102d50 0x7f196819a530 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:25:22.611 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:25:22.386+0000 7f196d92e700 1 -- 172.21.15.114:0/1832178810 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f19681a55b0 con 0x7f19681023a0 2024-04-23T10:25:22.611 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:25:22.386+0000 7f196e12f700 1 --2- 172.21.15.114:0/1832178810 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f1968106cc0 0x7f19681a5070 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-23T10:25:22.611 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:25:22.386+0000 7f196d12d700 1 --2- 172.21.15.114:0/1832178810 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f1968102d50 0x7f196819a530 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-23T10:25:22.611 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:25:22.386+0000 7f196d92e700 1 --2- 172.21.15.114:0/1832178810 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f19681023a0 0x7f1968199ff0 secure :-1 s=READY pgs=265 cs=0 l=1 rev1=1 crypto rx=0x7f196400b580 tx=0x7f19640047b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T10:25:22.611 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:25:22.386+0000 7f195effd700 1 -- 172.21.15.114:0/1832178810 <== mon.0 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1964027070 con 0x7f19681023a0 2024-04-23T10:25:22.611 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:25:22.386+0000 7f195effd700 1 -- 172.21.15.114:0/1832178810 <== mon.0 v2:172.21.15.114:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f1964004020 con 0x7f19681023a0 2024-04-23T10:25:22.611 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:25:22.386+0000 7f195effd700 1 -- 172.21.15.114:0/1832178810 <== mon.0 v2:172.21.15.114:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1964022e30 con 0x7f19681023a0 2024-04-23T10:25:22.611 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:25:22.386+0000 7f196fb92700 1 -- 172.21.15.114:0/1832178810 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f19681a5780 con 0x7f19681023a0 2024-04-23T10:25:22.611 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:25:22.387+0000 7f196fb92700 1 -- 172.21.15.114:0/1832178810 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f19681a5bd0 con 0x7f19681023a0 2024-04-23T10:25:22.611 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:25:22.387+0000 7f196fb92700 1 -- 172.21.15.114:0/1832178810 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_get_version(what=osdmap handle=1) -- 0x7f1968063150 con 0x7f19681023a0 2024-04-23T10:25:22.612 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:25:22.388+0000 7f195effd700 1 -- 172.21.15.114:0/1832178810 <== mon.0 v2:172.21.15.114:3300/0 4 ==== mgrmap(e 17) ==== 138698+0+0 (secure 0 0 0) 0x7f1964022590 con 0x7f19681023a0 2024-04-23T10:25:22.612 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:25:22.389+0000 7f195effd700 1 --2- 172.21.15.114:0/1832178810 >> [v2:172.21.15.114:6800/465333808,v1:172.21.15.114:6801/465333808] conn(0x7f19540a3cd0 0x7f19540a6190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T10:25:22.612 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:25:22.389+0000 7f196d12d700 1 --2- 172.21.15.114:0/1832178810 >> [v2:172.21.15.114:6800/465333808,v1:172.21.15.114:6801/465333808] conn(0x7f19540a3cd0 0x7f19540a6190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T10:25:22.612 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:25:22.389+0000 7f195effd700 1 -- 172.21.15.114:0/1832178810 <== mon.0 v2:172.21.15.114:3300/0 5 ==== osd_map(87..87 src has 1..87) ==== 7979+0+0 (secure 0 0 0) 0x7f19640d1780 con 0x7f19681023a0 2024-04-23T10:25:22.612 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:25:22.389+0000 7f195effd700 1 --2- 172.21.15.114:0/1832178810 >> [v2:172.21.15.204:6824/2407298490,v1:172.21.15.204:6825/2407298490] conn(0x7f19540ad4b0 0x7f19540af8d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T10:25:22.612 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:25:22.389+0000 7f195effd700 1 -- 172.21.15.114:0/1832178810 --> [v2:172.21.15.204:6824/2407298490,v1:172.21.15.204:6825/2407298490] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f1964019e80 con 0x7f19540ad4b0 2024-04-23T10:25:22.612 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:25:22.389+0000 7f195effd700 1 -- 172.21.15.114:0/1832178810 <== mon.0 v2:172.21.15.114:3300/0 6 ==== mon_get_version_reply(handle=1 version=87) ==== 24+0+0 (secure 0 0 0) 0x7f19640d1c60 con 0x7f19681023a0 2024-04-23T10:25:22.612 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:25:22.389+0000 7f196d12d700 1 --2- 172.21.15.114:0/1832178810 >> [v2:172.21.15.114:6800/465333808,v1:172.21.15.114:6801/465333808] conn(0x7f19540a3cd0 0x7f19540a6190 secure :-1 s=READY pgs=191 cs=0 l=1 rev1=1 crypto rx=0x7f1958005fd0 tx=0x7f1958005dc0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T10:25:22.612 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:25:22.390+0000 7f196e12f700 1 --2- 172.21.15.114:0/1832178810 >> [v2:172.21.15.204:6824/2407298490,v1:172.21.15.204:6825/2407298490] conn(0x7f19540ad4b0 0x7f19540af8d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T10:25:22.613 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:25:22.390+0000 7f196e12f700 1 --2- 172.21.15.114:0/1832178810 >> [v2:172.21.15.204:6824/2407298490,v1:172.21.15.204:6825/2407298490] conn(0x7f19540ad4b0 0x7f19540af8d0 crc :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.11 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T10:25:22.613 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:25:22.391+0000 7f195effd700 1 -- 172.21.15.114:0/1832178810 <== osd.11 v2:172.21.15.204:6824/2407298490 1 ==== command_reply(tid 1: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7f1964019e80 con 0x7f19540ad4b0 2024-04-23T10:25:22.613 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:25:22.407+0000 7f196fb92700 1 -- 172.21.15.114:0/1832178810 --> [v2:172.21.15.204:6824/2407298490,v1:172.21.15.204:6825/2407298490] -- command(tid 2: {"prefix": "flush_pg_stats"}) -- 0x7f1968063550 con 0x7f19540ad4b0 2024-04-23T10:25:22.613 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:25:22.408+0000 7f195effd700 1 -- 172.21.15.114:0/1832178810 <== osd.11 v2:172.21.15.204:6824/2407298490 2 ==== command_reply(tid 2: 0 ) ==== 8+0+12 (crc 0 0 0) 0x7f1968063550 con 0x7f19540ad4b0 2024-04-23T10:25:22.613 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:25:22.408+0000 7f196fb92700 1 -- 172.21.15.114:0/1832178810 >> [v2:172.21.15.204:6824/2407298490,v1:172.21.15.204:6825/2407298490] conn(0x7f19540ad4b0 msgr2=0x7f19540af8d0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T10:25:22.613 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:25:22.408+0000 7f196fb92700 1 --2- 172.21.15.114:0/1832178810 >> [v2:172.21.15.204:6824/2407298490,v1:172.21.15.204:6825/2407298490] conn(0x7f19540ad4b0 0x7f19540af8d0 crc :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:25:22.613 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:25:22.408+0000 7f196fb92700 1 -- 172.21.15.114:0/1832178810 >> [v2:172.21.15.114:6800/465333808,v1:172.21.15.114:6801/465333808] conn(0x7f19540a3cd0 msgr2=0x7f19540a6190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T10:25:22.613 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:25:22.408+0000 7f196fb92700 1 --2- 172.21.15.114:0/1832178810 >> [v2:172.21.15.114:6800/465333808,v1:172.21.15.114:6801/465333808] conn(0x7f19540a3cd0 0x7f19540a6190 secure :-1 s=READY pgs=191 cs=0 l=1 rev1=1 crypto rx=0x7f1958005fd0 tx=0x7f1958005dc0 comp rx=0 tx=0).stop 2024-04-23T10:25:22.613 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:25:22.408+0000 7f196fb92700 1 -- 172.21.15.114:0/1832178810 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f19681023a0 msgr2=0x7f1968199ff0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T10:25:22.613 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:25:22.408+0000 7f196fb92700 1 --2- 172.21.15.114:0/1832178810 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f19681023a0 0x7f1968199ff0 secure :-1 s=READY pgs=265 cs=0 l=1 rev1=1 crypto rx=0x7f196400b580 tx=0x7f19640047b0 comp rx=0 tx=0).stop 2024-04-23T10:25:22.613 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:25:22.408+0000 7f196e12f700 1 -- 172.21.15.114:0/1832178810 reap_dead start 2024-04-23T10:25:22.613 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:25:22.408+0000 7f196fb92700 1 -- 172.21.15.114:0/1832178810 shutdown_connections 2024-04-23T10:25:22.613 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:25:22.408+0000 7f196fb92700 1 -- 172.21.15.114:0/1832178810 >> 172.21.15.114:0/1832178810 conn(0x7f1968077fb0 msgr2=0x7f19680ff360 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T10:25:22.613 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:25:22.408+0000 7f196fb92700 1 -- 172.21.15.114:0/1832178810 shutdown_connections 2024-04-23T10:25:22.613 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:25:22.408+0000 7f196fb92700 1 -- 172.21.15.114:0/1832178810 wait complete. 2024-04-23T10:25:23.254 INFO:journalctl@ceph.mon.a.smithi114.stdout:Apr 23 10:25:23 smithi114 ceph-mon[29255]: pgmap v305: 97 pgs: 97 active+clean; 639 KiB data, 370 MiB used, 1.0 TiB / 1.0 TiB avail; 2.8 KiB/s rd, 426 B/s wr, 5 op/s 2024-04-23T10:25:23.328 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 23 10:25:23 smithi204 ceph-mon[32649]: pgmap v305: 97 pgs: 97 active+clean; 639 KiB data, 370 MiB used, 1.0 TiB / 1.0 TiB avail; 2.8 KiB/s rd, 426 B/s wr, 5 op/s 2024-04-23T10:25:23.492 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 23 10:25:23 smithi119 ceph-mon[32608]: pgmap v305: 97 pgs: 97 active+clean; 639 KiB data, 370 MiB used, 1.0 TiB / 1.0 TiB avail; 2.8 KiB/s rd, 426 B/s wr, 5 op/s 2024-04-23T10:25:23.532 INFO:teuthology.orchestra.run.smithi114.stdout:201863462958 2024-04-23T10:25:23.533 DEBUG:teuthology.orchestra.run.smithi114:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d69f13d827952bb07bb9b7973bca14f97d764d6f shell --fsid aa240572-0159-11ef-bc93-c7b262605968 -- ceph osd last-stat-seq osd.6 2024-04-23T10:25:23.713 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:25:23.714+0000 7f9b7cd16700 1 -- 172.21.15.114:0/362154521 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f9b78102a90 msgr2=0x7f9b78102f10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T10:25:23.714 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:25:23.714+0000 7f9b7cd16700 1 --2- 172.21.15.114:0/362154521 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f9b78102a90 0x7f9b78102f10 secure :-1 s=READY pgs=266 cs=0 l=1 rev1=1 crypto rx=0x7f9b6c009990 tx=0x7f9b6c009ca0 comp rx=0 tx=0).stop 2024-04-23T10:25:23.714 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:25:23.714+0000 7f9b7cd16700 1 -- 172.21.15.114:0/362154521 shutdown_connections 2024-04-23T10:25:23.714 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:25:23.714+0000 7f9b7cd16700 1 --2- 172.21.15.114:0/362154521 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f9b78103450 0x7f9b7810f8b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:25:23.714 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:25:23.714+0000 7f9b7cd16700 1 --2- 172.21.15.114:0/362154521 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f9b78108a50 0x7f9b78108e30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:25:23.714 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:25:23.714+0000 7f9b7cd16700 1 --2- 172.21.15.114:0/362154521 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f9b78102a90 0x7f9b78102f10 unknown :-1 s=CLOSED pgs=266 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:25:23.714 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:25:23.714+0000 7f9b7cd16700 1 -- 172.21.15.114:0/362154521 >> 172.21.15.114:0/362154521 conn(0x7f9b78079ba0 msgr2=0x7f9b78079fb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T10:25:23.714 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:25:23.714+0000 7f9b7cd16700 1 -- 172.21.15.114:0/362154521 shutdown_connections 2024-04-23T10:25:23.714 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:25:23.714+0000 7f9b7cd16700 1 -- 172.21.15.114:0/362154521 wait complete. 2024-04-23T10:25:23.716 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:25:23.715+0000 7f9b7cd16700 1 Processor -- start 2024-04-23T10:25:23.716 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:25:23.715+0000 7f9b7cd16700 1 -- start start 2024-04-23T10:25:23.716 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:25:23.716+0000 7f9b7cd16700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f9b78102a90 0x7f9b781990f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T10:25:23.716 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:25:23.716+0000 7f9b7cd16700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f9b78103450 0x7f9b78199630 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T10:25:23.716 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:25:23.716+0000 7f9b7cd16700 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f9b78108a50 0x7f9b7819d9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T10:25:23.716 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:25:23.716+0000 7f9b7cd16700 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x7f9b78112c60 con 0x7f9b78103450 2024-04-23T10:25:23.716 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:25:23.716+0000 7f9b7cd16700 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 -- 0x7f9b78112ae0 con 0x7f9b78102a90 2024-04-23T10:25:23.716 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:25:23.716+0000 7f9b7cd16700 1 -- --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_getmap magic: 0 -- 0x7f9b78112de0 con 0x7f9b78108a50 2024-04-23T10:25:23.717 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:25:23.716+0000 7f9b75d9b700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f9b78103450 0x7f9b78199630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T10:25:23.717 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:25:23.716+0000 7f9b75d9b700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f9b78103450 0x7f9b78199630 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.114:36338/0 (socket says 172.21.15.114:36338) 2024-04-23T10:25:23.717 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:25:23.716+0000 7f9b75d9b700 1 -- 172.21.15.114:0/4227406836 learned_addr learned my addr 172.21.15.114:0/4227406836 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-04-23T10:25:23.717 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:25:23.716+0000 7f9b76d9d700 1 --2- 172.21.15.114:0/4227406836 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f9b78108a50 0x7f9b7819d9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T10:25:23.717 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:25:23.716+0000 7f9b7659c700 1 --2- 172.21.15.114:0/4227406836 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f9b78102a90 0x7f9b781990f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T10:25:23.717 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:25:23.716+0000 7f9b75d9b700 1 -- 172.21.15.114:0/4227406836 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f9b78108a50 msgr2=0x7f9b7819d9c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T10:25:23.717 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:25:23.716+0000 7f9b75d9b700 1 --2- 172.21.15.114:0/4227406836 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f9b78108a50 0x7f9b7819d9c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:25:23.717 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:25:23.716+0000 7f9b75d9b700 1 -- 172.21.15.114:0/4227406836 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f9b78102a90 msgr2=0x7f9b781990f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T10:25:23.717 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:25:23.716+0000 7f9b75d9b700 1 --2- 172.21.15.114:0/4227406836 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f9b78102a90 0x7f9b781990f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:25:23.717 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:25:23.716+0000 7f9b75d9b700 1 -- 172.21.15.114:0/4227406836 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f9b7819e100 con 0x7f9b78103450 2024-04-23T10:25:23.717 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:25:23.716+0000 7f9b76d9d700 1 --2- 172.21.15.114:0/4227406836 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f9b78108a50 0x7f9b7819d9c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-23T10:25:23.717 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:25:23.716+0000 7f9b75d9b700 1 --2- 172.21.15.114:0/4227406836 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f9b78103450 0x7f9b78199630 secure :-1 s=READY pgs=267 cs=0 l=1 rev1=1 crypto rx=0x7f9b6c000c00 tx=0x7f9b6c0047b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T10:25:23.717 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:25:23.716+0000 7f9b677fe700 1 -- 172.21.15.114:0/4227406836 <== mon.0 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9b6c027070 con 0x7f9b78103450 2024-04-23T10:25:23.717 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:25:23.717+0000 7f9b7cd16700 1 -- 172.21.15.114:0/4227406836 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f9b7819e390 con 0x7f9b78103450 2024-04-23T10:25:23.717 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:25:23.717+0000 7f9b677fe700 1 -- 172.21.15.114:0/4227406836 <== mon.0 v2:172.21.15.114:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f9b6c0041d0 con 0x7f9b78103450 2024-04-23T10:25:23.718 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:25:23.717+0000 7f9b677fe700 1 -- 172.21.15.114:0/4227406836 <== mon.0 v2:172.21.15.114:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9b6c00f460 con 0x7f9b78103450 2024-04-23T10:25:23.718 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:25:23.717+0000 7f9b7cd16700 1 -- 172.21.15.114:0/4227406836 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f9b781a5c70 con 0x7f9b78103450 2024-04-23T10:25:23.719 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:25:23.717+0000 7f9b7cd16700 1 -- 172.21.15.114:0/4227406836 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_get_version(what=osdmap handle=1) -- 0x7f9b78063150 con 0x7f9b78103450 2024-04-23T10:25:23.719 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:25:23.719+0000 7f9b677fe700 1 -- 172.21.15.114:0/4227406836 <== mon.0 v2:172.21.15.114:3300/0 4 ==== mgrmap(e 17) ==== 138698+0+0 (secure 0 0 0) 0x7f9b6c02a070 con 0x7f9b78103450 2024-04-23T10:25:23.719 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:25:23.719+0000 7f9b677fe700 1 --2- 172.21.15.114:0/4227406836 >> [v2:172.21.15.114:6800/465333808,v1:172.21.15.114:6801/465333808] conn(0x7f9b600a3cd0 0x7f9b600a6190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T10:25:23.719 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:25:23.719+0000 7f9b7659c700 1 --2- 172.21.15.114:0/4227406836 >> [v2:172.21.15.114:6800/465333808,v1:172.21.15.114:6801/465333808] conn(0x7f9b600a3cd0 0x7f9b600a6190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T10:25:23.719 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:25:23.720+0000 7f9b677fe700 1 -- 172.21.15.114:0/4227406836 <== mon.0 v2:172.21.15.114:3300/0 5 ==== osd_map(87..87 src has 1..87) ==== 7979+0+0 (secure 0 0 0) 0x7f9b6c0d1ab0 con 0x7f9b78103450 2024-04-23T10:25:23.719 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:25:23.720+0000 7f9b677fe700 1 --2- 172.21.15.114:0/4227406836 >> [v2:172.21.15.119:6816/2237458793,v1:172.21.15.119:6817/2237458793] conn(0x7f9b600ad4b0 0x7f9b600af8d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T10:25:23.719 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:25:23.720+0000 7f9b677fe700 1 -- 172.21.15.114:0/4227406836 --> [v2:172.21.15.119:6816/2237458793,v1:172.21.15.119:6817/2237458793] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f9b6c0043d0 con 0x7f9b600ad4b0 2024-04-23T10:25:23.719 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:25:23.720+0000 7f9b677fe700 1 -- 172.21.15.114:0/4227406836 <== mon.0 v2:172.21.15.114:3300/0 6 ==== mon_get_version_reply(handle=1 version=87) ==== 24+0+0 (secure 0 0 0) 0x7f9b6c0d1f90 con 0x7f9b78103450 2024-04-23T10:25:23.719 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:25:23.720+0000 7f9b76d9d700 1 --2- 172.21.15.114:0/4227406836 >> [v2:172.21.15.119:6816/2237458793,v1:172.21.15.119:6817/2237458793] conn(0x7f9b600ad4b0 0x7f9b600af8d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T10:25:23.719 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:25:23.720+0000 7f9b7659c700 1 --2- 172.21.15.114:0/4227406836 >> [v2:172.21.15.114:6800/465333808,v1:172.21.15.114:6801/465333808] conn(0x7f9b600a3cd0 0x7f9b600a6190 secure :-1 s=READY pgs=192 cs=0 l=1 rev1=1 crypto rx=0x7f9b68006fd0 tx=0x7f9b68008040 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T10:25:23.720 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:25:23.720+0000 7f9b76d9d700 1 --2- 172.21.15.114:0/4227406836 >> [v2:172.21.15.119:6816/2237458793,v1:172.21.15.119:6817/2237458793] conn(0x7f9b600ad4b0 0x7f9b600af8d0 crc :-1 s=READY pgs=11 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.7 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T10:25:23.720 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:25:23.721+0000 7f9b677fe700 1 -- 172.21.15.114:0/4227406836 <== osd.7 v2:172.21.15.119:6816/2237458793 1 ==== command_reply(tid 1: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7f9b6c0043d0 con 0x7f9b600ad4b0 2024-04-23T10:25:23.747 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:25:23.748+0000 7f9b7cd16700 1 -- 172.21.15.114:0/4227406836 --> [v2:172.21.15.119:6816/2237458793,v1:172.21.15.119:6817/2237458793] -- command(tid 2: {"prefix": "flush_pg_stats"}) -- 0x7f9b78063550 con 0x7f9b600ad4b0 2024-04-23T10:25:23.748 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:25:23.749+0000 7f9b677fe700 1 -- 172.21.15.114:0/4227406836 <== osd.7 v2:172.21.15.119:6816/2237458793 2 ==== command_reply(tid 2: 0 ) ==== 8+0+12 (crc 0 0 0) 0x7f9b78063550 con 0x7f9b600ad4b0 2024-04-23T10:25:23.749 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:25:23.749+0000 7f9b7cd16700 1 -- 172.21.15.114:0/4227406836 >> [v2:172.21.15.119:6816/2237458793,v1:172.21.15.119:6817/2237458793] conn(0x7f9b600ad4b0 msgr2=0x7f9b600af8d0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T10:25:23.749 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:25:23.749+0000 7f9b7cd16700 1 --2- 172.21.15.114:0/4227406836 >> [v2:172.21.15.119:6816/2237458793,v1:172.21.15.119:6817/2237458793] conn(0x7f9b600ad4b0 0x7f9b600af8d0 crc :-1 s=READY pgs=11 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:25:23.749 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:25:23.750+0000 7f9b7cd16700 1 -- 172.21.15.114:0/4227406836 >> [v2:172.21.15.114:6800/465333808,v1:172.21.15.114:6801/465333808] conn(0x7f9b600a3cd0 msgr2=0x7f9b600a6190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T10:25:23.749 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:25:23.750+0000 7f9b7cd16700 1 --2- 172.21.15.114:0/4227406836 >> [v2:172.21.15.114:6800/465333808,v1:172.21.15.114:6801/465333808] conn(0x7f9b600a3cd0 0x7f9b600a6190 secure :-1 s=READY pgs=192 cs=0 l=1 rev1=1 crypto rx=0x7f9b68006fd0 tx=0x7f9b68008040 comp rx=0 tx=0).stop 2024-04-23T10:25:23.749 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:25:23.750+0000 7f9b7cd16700 1 -- 172.21.15.114:0/4227406836 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f9b78103450 msgr2=0x7f9b78199630 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T10:25:23.749 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:25:23.750+0000 7f9b7cd16700 1 --2- 172.21.15.114:0/4227406836 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f9b78103450 0x7f9b78199630 secure :-1 s=READY pgs=267 cs=0 l=1 rev1=1 crypto rx=0x7f9b6c000c00 tx=0x7f9b6c0047b0 comp rx=0 tx=0).stop 2024-04-23T10:25:23.749 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:25:23.750+0000 7f9b76d9d700 1 -- 172.21.15.114:0/4227406836 reap_dead start 2024-04-23T10:25:23.749 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:25:23.750+0000 7f9b7cd16700 1 -- 172.21.15.114:0/4227406836 shutdown_connections 2024-04-23T10:25:23.750 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:25:23.750+0000 7f9b7cd16700 1 -- 172.21.15.114:0/4227406836 >> 172.21.15.114:0/4227406836 conn(0x7f9b78079ba0 msgr2=0x7f9b78101870 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T10:25:23.750 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:25:23.750+0000 7f9b7cd16700 1 -- 172.21.15.114:0/4227406836 shutdown_connections 2024-04-23T10:25:23.750 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:25:23.750+0000 7f9b7cd16700 1 -- 172.21.15.114:0/4227406836 wait complete. 2024-04-23T10:25:24.532 INFO:teuthology.orchestra.run.smithi114.stdout:107374182458 2024-04-23T10:25:24.532 DEBUG:teuthology.orchestra.run.smithi114:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d69f13d827952bb07bb9b7973bca14f97d764d6f shell --fsid aa240572-0159-11ef-bc93-c7b262605968 -- ceph osd last-stat-seq osd.3 2024-04-23T10:25:24.761 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:25:24.762+0000 7f9427a60700 1 -- 172.21.15.114:0/1054497663 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f9420102a90 msgr2=0x7f9420102f10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T10:25:24.761 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:25:24.762+0000 7f9427a60700 1 --2- 172.21.15.114:0/1054497663 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f9420102a90 0x7f9420102f10 secure :-1 s=READY pgs=268 cs=0 l=1 rev1=1 crypto rx=0x7f941c009990 tx=0x7f941c009ca0 comp rx=0 tx=0).stop 2024-04-23T10:25:24.761 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:25:24.762+0000 7f9427a60700 1 -- 172.21.15.114:0/1054497663 shutdown_connections 2024-04-23T10:25:24.761 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:25:24.762+0000 7f9427a60700 1 --2- 172.21.15.114:0/1054497663 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f9420108a50 0x7f9420108e30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:25:24.761 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:25:24.762+0000 7f9427a60700 1 --2- 172.21.15.114:0/1054497663 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f9420103450 0x7f942010f8b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:25:24.761 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:25:24.762+0000 7f9427a60700 1 --2- 172.21.15.114:0/1054497663 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f9420102a90 0x7f9420102f10 unknown :-1 s=CLOSED pgs=268 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:25:24.761 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:25:24.762+0000 7f9427a60700 1 -- 172.21.15.114:0/1054497663 >> 172.21.15.114:0/1054497663 conn(0x7f9420079ba0 msgr2=0x7f9420079fb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T10:25:24.761 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:25:24.762+0000 7f9427a60700 1 -- 172.21.15.114:0/1054497663 shutdown_connections 2024-04-23T10:25:24.762 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:25:24.762+0000 7f9427a60700 1 -- 172.21.15.114:0/1054497663 wait complete. 2024-04-23T10:25:24.763 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:25:24.762+0000 7f9427a60700 1 Processor -- start 2024-04-23T10:25:24.763 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:25:24.762+0000 7f9427a60700 1 -- start start 2024-04-23T10:25:24.764 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:25:24.763+0000 7f9427a60700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f9420102a90 0x7f942019b460 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T10:25:24.764 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:25:24.763+0000 7f9427a60700 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f9420103450 0x7f942019b9a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T10:25:24.764 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:25:24.763+0000 7f9427a60700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f9420108a50 0x7f942019fd30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T10:25:24.764 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:25:24.763+0000 7f9427a60700 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x7f9420112ce0 con 0x7f9420102a90 2024-04-23T10:25:24.764 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:25:24.763+0000 7f9427a60700 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 -- 0x7f9420112b60 con 0x7f9420108a50 2024-04-23T10:25:24.764 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:25:24.763+0000 7f9427a60700 1 -- --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_getmap magic: 0 -- 0x7f9420112e60 con 0x7f9420103450 2024-04-23T10:25:24.764 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:25:24.763+0000 7f94257fc700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f9420102a90 0x7f942019b460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T10:25:24.764 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:25:24.763+0000 7f94257fc700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f9420102a90 0x7f942019b460 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.114:36358/0 (socket says 172.21.15.114:36358) 2024-04-23T10:25:24.764 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:25:24.763+0000 7f94257fc700 1 -- 172.21.15.114:0/3542100873 learned_addr learned my addr 172.21.15.114:0/3542100873 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-04-23T10:25:24.764 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:25:24.763+0000 7f9425ffd700 1 --2- 172.21.15.114:0/3542100873 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f9420108a50 0x7f942019fd30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T10:25:24.764 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:25:24.763+0000 7f9424ffb700 1 --2- 172.21.15.114:0/3542100873 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f9420103450 0x7f942019b9a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T10:25:24.764 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:25:24.763+0000 7f94257fc700 1 -- 172.21.15.114:0/3542100873 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f9420103450 msgr2=0x7f942019b9a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T10:25:24.764 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:25:24.763+0000 7f94257fc700 1 --2- 172.21.15.114:0/3542100873 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f9420103450 0x7f942019b9a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:25:24.764 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:25:24.763+0000 7f94257fc700 1 -- 172.21.15.114:0/3542100873 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f9420108a50 msgr2=0x7f942019fd30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T10:25:24.764 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:25:24.763+0000 7f94257fc700 1 --2- 172.21.15.114:0/3542100873 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f9420108a50 0x7f942019fd30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:25:24.765 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:25:24.763+0000 7f94257fc700 1 -- 172.21.15.114:0/3542100873 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f94201a0470 con 0x7f9420102a90 2024-04-23T10:25:24.765 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:25:24.763+0000 7f94257fc700 1 --2- 172.21.15.114:0/3542100873 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f9420102a90 0x7f942019b460 secure :-1 s=READY pgs=269 cs=0 l=1 rev1=1 crypto rx=0x7f941000b530 tx=0x7f941000b8f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T10:25:24.765 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:25:24.764+0000 7f94167fc700 1 -- 172.21.15.114:0/3542100873 <== mon.0 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9410015020 con 0x7f9420102a90 2024-04-23T10:25:24.765 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:25:24.764+0000 7f9427a60700 1 -- 172.21.15.114:0/3542100873 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f94201a0760 con 0x7f9420102a90 2024-04-23T10:25:24.765 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:25:24.764+0000 7f94167fc700 1 -- 172.21.15.114:0/3542100873 <== mon.0 v2:172.21.15.114:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f9410011ce0 con 0x7f9420102a90 2024-04-23T10:25:24.765 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:25:24.764+0000 7f94167fc700 1 -- 172.21.15.114:0/3542100873 <== mon.0 v2:172.21.15.114:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9410010d20 con 0x7f9420102a90 2024-04-23T10:25:24.765 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:25:24.764+0000 7f9427a60700 1 -- 172.21.15.114:0/3542100873 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f94201a7fe0 con 0x7f9420102a90 2024-04-23T10:25:24.766 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:25:24.765+0000 7f9427a60700 1 -- 172.21.15.114:0/3542100873 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_get_version(what=osdmap handle=1) -- 0x7f9420063150 con 0x7f9420102a90 2024-04-23T10:25:24.766 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:25:24.766+0000 7f94167fc700 1 -- 172.21.15.114:0/3542100873 <== mon.0 v2:172.21.15.114:3300/0 4 ==== mgrmap(e 17) ==== 138698+0+0 (secure 0 0 0) 0x7f9410018020 con 0x7f9420102a90 2024-04-23T10:25:24.766 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:25:24.766+0000 7f94167fc700 1 --2- 172.21.15.114:0/3542100873 >> [v2:172.21.15.114:6800/465333808,v1:172.21.15.114:6801/465333808] conn(0x7f940c0a3cd0 0x7f940c0a6190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T10:25:24.766 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:25:24.766+0000 7f9424ffb700 1 --2- 172.21.15.114:0/3542100873 >> [v2:172.21.15.114:6800/465333808,v1:172.21.15.114:6801/465333808] conn(0x7f940c0a3cd0 0x7f940c0a6190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T10:25:24.766 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:25:24.767+0000 7f94167fc700 1 -- 172.21.15.114:0/3542100873 <== mon.0 v2:172.21.15.114:3300/0 5 ==== osd_map(87..87 src has 1..87) ==== 7979+0+0 (secure 0 0 0) 0x7f94100cfad0 con 0x7f9420102a90 2024-04-23T10:25:24.766 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:25:24.767+0000 7f94167fc700 1 --2- 172.21.15.114:0/3542100873 >> [v2:172.21.15.204:6800/799652511,v1:172.21.15.204:6801/799652511] conn(0x7f940c0ad4b0 0x7f940c0af8d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T10:25:24.766 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:25:24.767+0000 7f94167fc700 1 -- 172.21.15.114:0/3542100873 --> [v2:172.21.15.204:6800/799652511,v1:172.21.15.204:6801/799652511] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f940c0aff80 con 0x7f940c0ad4b0 2024-04-23T10:25:24.767 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:25:24.767+0000 7f94167fc700 1 -- 172.21.15.114:0/3542100873 <== mon.0 v2:172.21.15.114:3300/0 6 ==== mon_get_version_reply(handle=1 version=87) ==== 24+0+0 (secure 0 0 0) 0x7f94100d4240 con 0x7f9420102a90 2024-04-23T10:25:24.767 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:25:24.767+0000 7f9424ffb700 1 --2- 172.21.15.114:0/3542100873 >> [v2:172.21.15.114:6800/465333808,v1:172.21.15.114:6801/465333808] conn(0x7f940c0a3cd0 0x7f940c0a6190 secure :-1 s=READY pgs=193 cs=0 l=1 rev1=1 crypto rx=0x7f941c006010 tx=0x7f941c01a040 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T10:25:24.767 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:25:24.767+0000 7f9425ffd700 1 --2- 172.21.15.114:0/3542100873 >> [v2:172.21.15.204:6800/799652511,v1:172.21.15.204:6801/799652511] conn(0x7f940c0ad4b0 0x7f940c0af8d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T10:25:24.767 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:25:24.767+0000 7f9425ffd700 1 --2- 172.21.15.114:0/3542100873 >> [v2:172.21.15.204:6800/799652511,v1:172.21.15.204:6801/799652511] conn(0x7f940c0ad4b0 0x7f940c0af8d0 crc :-1 s=READY pgs=14 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T10:25:24.767 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:25:24.768+0000 7f94167fc700 1 -- 172.21.15.114:0/3542100873 <== osd.2 v2:172.21.15.204:6800/799652511 1 ==== command_reply(tid 1: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7f940c0aff80 con 0x7f940c0ad4b0 2024-04-23T10:25:24.783 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:25:24.784+0000 7f9427a60700 1 -- 172.21.15.114:0/3542100873 --> [v2:172.21.15.204:6800/799652511,v1:172.21.15.204:6801/799652511] -- command(tid 2: {"prefix": "flush_pg_stats"}) -- 0x7f94200635d0 con 0x7f940c0ad4b0 2024-04-23T10:25:24.783 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:25:24.784+0000 7f94167fc700 1 -- 172.21.15.114:0/3542100873 <== osd.2 v2:172.21.15.204:6800/799652511 2 ==== command_reply(tid 2: 0 ) ==== 8+0+11 (crc 0 0 0) 0x7f94200635d0 con 0x7f940c0ad4b0 2024-04-23T10:25:24.784 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:25:24.785+0000 7f9427a60700 1 -- 172.21.15.114:0/3542100873 >> [v2:172.21.15.204:6800/799652511,v1:172.21.15.204:6801/799652511] conn(0x7f940c0ad4b0 msgr2=0x7f940c0af8d0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T10:25:24.784 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:25:24.785+0000 7f9427a60700 1 --2- 172.21.15.114:0/3542100873 >> [v2:172.21.15.204:6800/799652511,v1:172.21.15.204:6801/799652511] conn(0x7f940c0ad4b0 0x7f940c0af8d0 crc :-1 s=READY pgs=14 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:25:24.784 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:25:24.785+0000 7f9427a60700 1 -- 172.21.15.114:0/3542100873 >> [v2:172.21.15.114:6800/465333808,v1:172.21.15.114:6801/465333808] conn(0x7f940c0a3cd0 msgr2=0x7f940c0a6190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T10:25:24.784 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:25:24.785+0000 7f9427a60700 1 --2- 172.21.15.114:0/3542100873 >> [v2:172.21.15.114:6800/465333808,v1:172.21.15.114:6801/465333808] conn(0x7f940c0a3cd0 0x7f940c0a6190 secure :-1 s=READY pgs=193 cs=0 l=1 rev1=1 crypto rx=0x7f941c006010 tx=0x7f941c01a040 comp rx=0 tx=0).stop 2024-04-23T10:25:24.784 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:25:24.785+0000 7f9427a60700 1 -- 172.21.15.114:0/3542100873 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f9420102a90 msgr2=0x7f942019b460 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T10:25:24.784 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:25:24.785+0000 7f9427a60700 1 --2- 172.21.15.114:0/3542100873 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f9420102a90 0x7f942019b460 secure :-1 s=READY pgs=269 cs=0 l=1 rev1=1 crypto rx=0x7f941000b530 tx=0x7f941000b8f0 comp rx=0 tx=0).stop 2024-04-23T10:25:24.784 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:25:24.785+0000 7f9425ffd700 1 -- 172.21.15.114:0/3542100873 reap_dead start 2024-04-23T10:25:24.785 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:25:24.785+0000 7f9427a60700 1 -- 172.21.15.114:0/3542100873 shutdown_connections 2024-04-23T10:25:24.785 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:25:24.785+0000 7f9427a60700 1 -- 172.21.15.114:0/3542100873 >> 172.21.15.114:0/3542100873 conn(0x7f9420079ba0 msgr2=0x7f942010df70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T10:25:24.785 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:25:24.785+0000 7f9427a60700 1 -- 172.21.15.114:0/3542100873 shutdown_connections 2024-04-23T10:25:24.785 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:25:24.785+0000 7f9427a60700 1 -- 172.21.15.114:0/3542100873 wait complete. 2024-04-23T10:25:25.097 INFO:teuthology.orchestra.run.smithi114.stdout:266287972389 2024-04-23T10:25:25.098 DEBUG:teuthology.orchestra.run.smithi114:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d69f13d827952bb07bb9b7973bca14f97d764d6f shell --fsid aa240572-0159-11ef-bc93-c7b262605968 -- ceph osd last-stat-seq osd.8 2024-04-23T10:25:25.324 INFO:journalctl@ceph.mon.a.smithi114.stdout:Apr 23 10:25:25 smithi114 ceph-mon[29255]: pgmap v306: 97 pgs: 97 active+clean; 641 KiB data, 370 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 597 B/s wr, 7 op/s 2024-04-23T10:25:25.492 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 23 10:25:25 smithi119 ceph-mon[32608]: pgmap v306: 97 pgs: 97 active+clean; 641 KiB data, 370 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 597 B/s wr, 7 op/s 2024-04-23T10:25:25.539 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:25:25.325+0000 7f5f59be3700 1 -- 172.21.15.114:0/290616154 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f5f54108a30 msgr2=0x7f5f54108e10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T10:25:25.539 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:25:25.325+0000 7f5f59be3700 1 --2- 172.21.15.114:0/290616154 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f5f54108a30 0x7f5f54108e10 secure :-1 s=READY pgs=270 cs=0 l=1 rev1=1 crypto rx=0x7f5f44009990 tx=0x7f5f44009ca0 comp rx=0 tx=0).stop 2024-04-23T10:25:25.539 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:25:25.325+0000 7f5f59be3700 1 -- 172.21.15.114:0/290616154 shutdown_connections 2024-04-23T10:25:25.539 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:25:25.325+0000 7f5f59be3700 1 --2- 172.21.15.114:0/290616154 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f5f5406b0f0 0x7f5f5410f900 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:25:25.539 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:25:25.325+0000 7f5f59be3700 1 --2- 172.21.15.114:0/290616154 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f5f54108a30 0x7f5f54108e10 unknown :-1 s=CLOSED pgs=270 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:25:25.539 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:25:25.325+0000 7f5f59be3700 1 --2- 172.21.15.114:0/290616154 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f5f5406a750 0x7f5f5406abb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:25:25.539 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:25:25.325+0000 7f5f59be3700 1 -- 172.21.15.114:0/290616154 >> 172.21.15.114:0/290616154 conn(0x7f5f540788f0 msgr2=0x7f5f54078d00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T10:25:25.539 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:25:25.325+0000 7f5f59be3700 1 -- 172.21.15.114:0/290616154 shutdown_connections 2024-04-23T10:25:25.539 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:25:25.325+0000 7f5f59be3700 1 -- 172.21.15.114:0/290616154 wait complete. 2024-04-23T10:25:25.540 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:25:25.326+0000 7f5f59be3700 1 Processor -- start 2024-04-23T10:25:25.540 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:25:25.326+0000 7f5f59be3700 1 -- start start 2024-04-23T10:25:25.540 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:25:25.327+0000 7f5f59be3700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f5f5406a750 0x7f5f54199140 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T10:25:25.540 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:25:25.327+0000 7f5f59be3700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f5f5406b0f0 0x7f5f54199680 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T10:25:25.540 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:25:25.327+0000 7f5f59be3700 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f5f54108a30 0x7f5f5419da10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T10:25:25.540 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:25:25.327+0000 7f5f59be3700 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x7f5f54112c30 con 0x7f5f5406b0f0 2024-04-23T10:25:25.540 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:25:25.327+0000 7f5f59be3700 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 -- 0x7f5f54112ab0 con 0x7f5f5406a750 2024-04-23T10:25:25.540 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:25:25.327+0000 7f5f59be3700 1 -- --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_getmap magic: 0 -- 0x7f5f54112db0 con 0x7f5f54108a30 2024-04-23T10:25:25.540 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:25:25.327+0000 7f5f52ffd700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f5f5406b0f0 0x7f5f54199680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T10:25:25.540 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:25:25.327+0000 7f5f52ffd700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f5f5406b0f0 0x7f5f54199680 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.114:36386/0 (socket says 172.21.15.114:36386) 2024-04-23T10:25:25.540 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:25:25.327+0000 7f5f52ffd700 1 -- 172.21.15.114:0/1825561570 learned_addr learned my addr 172.21.15.114:0/1825561570 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-04-23T10:25:25.540 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:25:25.327+0000 7f5f53fff700 1 --2- 172.21.15.114:0/1825561570 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f5f54108a30 0x7f5f5419da10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T10:25:25.540 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:25:25.327+0000 7f5f537fe700 1 --2- 172.21.15.114:0/1825561570 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f5f5406a750 0x7f5f54199140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T10:25:25.540 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:25:25.327+0000 7f5f52ffd700 1 -- 172.21.15.114:0/1825561570 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f5f54108a30 msgr2=0x7f5f5419da10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T10:25:25.541 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:25:25.327+0000 7f5f52ffd700 1 --2- 172.21.15.114:0/1825561570 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f5f54108a30 0x7f5f5419da10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:25:25.541 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:25:25.327+0000 7f5f52ffd700 1 -- 172.21.15.114:0/1825561570 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f5f5406a750 msgr2=0x7f5f54199140 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T10:25:25.541 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:25:25.327+0000 7f5f52ffd700 1 --2- 172.21.15.114:0/1825561570 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f5f5406a750 0x7f5f54199140 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:25:25.541 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:25:25.327+0000 7f5f52ffd700 1 -- 172.21.15.114:0/1825561570 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f5f5419e150 con 0x7f5f5406b0f0 2024-04-23T10:25:25.541 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:25:25.327+0000 7f5f53fff700 1 --2- 172.21.15.114:0/1825561570 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f5f54108a30 0x7f5f5419da10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-23T10:25:25.541 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:25:25.327+0000 7f5f537fe700 1 --2- 172.21.15.114:0/1825561570 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f5f5406a750 0x7f5f54199140 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-23T10:25:25.541 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:25:25.327+0000 7f5f52ffd700 1 --2- 172.21.15.114:0/1825561570 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f5f5406b0f0 0x7f5f54199680 secure :-1 s=READY pgs=271 cs=0 l=1 rev1=1 crypto rx=0x7f5f4800b530 tx=0x7f5f4800b8f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T10:25:25.541 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:25:25.327+0000 7f5f50ff9700 1 -- 172.21.15.114:0/1825561570 <== mon.0 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5f48015020 con 0x7f5f5406b0f0 2024-04-23T10:25:25.541 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:25:25.328+0000 7f5f59be3700 1 -- 172.21.15.114:0/1825561570 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f5f5419e440 con 0x7f5f5406b0f0 2024-04-23T10:25:25.541 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:25:25.328+0000 7f5f50ff9700 1 -- 172.21.15.114:0/1825561570 <== mon.0 v2:172.21.15.114:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f5f48011ce0 con 0x7f5f5406b0f0 2024-04-23T10:25:25.541 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:25:25.328+0000 7f5f50ff9700 1 -- 172.21.15.114:0/1825561570 <== mon.0 v2:172.21.15.114:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5f48010d20 con 0x7f5f5406b0f0 2024-04-23T10:25:25.541 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:25:25.328+0000 7f5f59be3700 1 -- 172.21.15.114:0/1825561570 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f5f541a5cc0 con 0x7f5f5406b0f0 2024-04-23T10:25:25.541 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:25:25.328+0000 7f5f59be3700 1 -- 172.21.15.114:0/1825561570 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_get_version(what=osdmap handle=1) -- 0x7f5f540631d0 con 0x7f5f5406b0f0 2024-04-23T10:25:25.541 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:25:25.330+0000 7f5f50ff9700 1 -- 172.21.15.114:0/1825561570 <== mon.0 v2:172.21.15.114:3300/0 4 ==== mgrmap(e 17) ==== 138698+0+0 (secure 0 0 0) 0x7f5f48018020 con 0x7f5f5406b0f0 2024-04-23T10:25:25.541 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:25:25.330+0000 7f5f50ff9700 1 --2- 172.21.15.114:0/1825561570 >> [v2:172.21.15.114:6800/465333808,v1:172.21.15.114:6801/465333808] conn(0x7f5f3c0a3cd0 0x7f5f3c0a6190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T10:25:25.542 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:25:25.330+0000 7f5f537fe700 1 --2- 172.21.15.114:0/1825561570 >> [v2:172.21.15.114:6800/465333808,v1:172.21.15.114:6801/465333808] conn(0x7f5f3c0a3cd0 0x7f5f3c0a6190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T10:25:25.542 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:25:25.330+0000 7f5f50ff9700 1 -- 172.21.15.114:0/1825561570 <== mon.0 v2:172.21.15.114:3300/0 5 ==== osd_map(87..87 src has 1..87) ==== 7979+0+0 (secure 0 0 0) 0x7f5f480d0260 con 0x7f5f5406b0f0 2024-04-23T10:25:25.542 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:25:25.330+0000 7f5f50ff9700 1 --2- 172.21.15.114:0/1825561570 >> [v2:172.21.15.114:6826/2750635721,v1:172.21.15.114:6827/2750635721] conn(0x7f5f3c0ad4b0 0x7f5f3c0af8d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T10:25:25.542 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:25:25.330+0000 7f5f50ff9700 1 -- 172.21.15.114:0/1825561570 --> [v2:172.21.15.114:6826/2750635721,v1:172.21.15.114:6827/2750635721] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f5f48010ec0 con 0x7f5f3c0ad4b0 2024-04-23T10:25:25.542 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:25:25.330+0000 7f5f50ff9700 1 -- 172.21.15.114:0/1825561570 <== mon.0 v2:172.21.15.114:3300/0 6 ==== mon_get_version_reply(handle=1 version=87) ==== 24+0+0 (secure 0 0 0) 0x7f5f480d0740 con 0x7f5f5406b0f0 2024-04-23T10:25:25.542 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:25:25.331+0000 7f5f53fff700 1 --2- 172.21.15.114:0/1825561570 >> [v2:172.21.15.114:6826/2750635721,v1:172.21.15.114:6827/2750635721] conn(0x7f5f3c0ad4b0 0x7f5f3c0af8d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T10:25:25.542 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:25:25.331+0000 7f5f537fe700 1 --2- 172.21.15.114:0/1825561570 >> [v2:172.21.15.114:6800/465333808,v1:172.21.15.114:6801/465333808] conn(0x7f5f3c0a3cd0 0x7f5f3c0a6190 secure :-1 s=READY pgs=194 cs=0 l=1 rev1=1 crypto rx=0x7f5f44006010 tx=0x7f5f4400b540 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T10:25:25.542 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:25:25.331+0000 7f5f53fff700 1 --2- 172.21.15.114:0/1825561570 >> [v2:172.21.15.114:6826/2750635721,v1:172.21.15.114:6827/2750635721] conn(0x7f5f3c0ad4b0 0x7f5f3c0af8d0 crc :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.9 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T10:25:25.542 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:25:25.332+0000 7f5f50ff9700 1 -- 172.21.15.114:0/1825561570 <== osd.9 v2:172.21.15.114:6826/2750635721 1 ==== command_reply(tid 1: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7f5f48010ec0 con 0x7f5f3c0ad4b0 2024-04-23T10:25:25.543 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:25:25.347+0000 7f5f59be3700 1 -- 172.21.15.114:0/1825561570 --> [v2:172.21.15.114:6826/2750635721,v1:172.21.15.114:6827/2750635721] -- command(tid 2: {"prefix": "flush_pg_stats"}) -- 0x7f5f54063650 con 0x7f5f3c0ad4b0 2024-04-23T10:25:25.543 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:25:25.348+0000 7f5f50ff9700 1 -- 172.21.15.114:0/1825561570 <== osd.9 v2:172.21.15.114:6826/2750635721 2 ==== command_reply(tid 2: 0 ) ==== 8+0+12 (crc 0 0 0) 0x7f5f54063650 con 0x7f5f3c0ad4b0 2024-04-23T10:25:25.543 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:25:25.348+0000 7f5f59be3700 1 -- 172.21.15.114:0/1825561570 >> [v2:172.21.15.114:6826/2750635721,v1:172.21.15.114:6827/2750635721] conn(0x7f5f3c0ad4b0 msgr2=0x7f5f3c0af8d0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T10:25:25.543 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:25:25.348+0000 7f5f59be3700 1 --2- 172.21.15.114:0/1825561570 >> [v2:172.21.15.114:6826/2750635721,v1:172.21.15.114:6827/2750635721] conn(0x7f5f3c0ad4b0 0x7f5f3c0af8d0 crc :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:25:25.543 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:25:25.348+0000 7f5f59be3700 1 -- 172.21.15.114:0/1825561570 >> [v2:172.21.15.114:6800/465333808,v1:172.21.15.114:6801/465333808] conn(0x7f5f3c0a3cd0 msgr2=0x7f5f3c0a6190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T10:25:25.543 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:25:25.348+0000 7f5f59be3700 1 --2- 172.21.15.114:0/1825561570 >> [v2:172.21.15.114:6800/465333808,v1:172.21.15.114:6801/465333808] conn(0x7f5f3c0a3cd0 0x7f5f3c0a6190 secure :-1 s=READY pgs=194 cs=0 l=1 rev1=1 crypto rx=0x7f5f44006010 tx=0x7f5f4400b540 comp rx=0 tx=0).stop 2024-04-23T10:25:25.543 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:25:25.348+0000 7f5f59be3700 1 -- 172.21.15.114:0/1825561570 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f5f5406b0f0 msgr2=0x7f5f54199680 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T10:25:25.543 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:25:25.348+0000 7f5f59be3700 1 --2- 172.21.15.114:0/1825561570 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f5f5406b0f0 0x7f5f54199680 secure :-1 s=READY pgs=271 cs=0 l=1 rev1=1 crypto rx=0x7f5f4800b530 tx=0x7f5f4800b8f0 comp rx=0 tx=0).stop 2024-04-23T10:25:25.543 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:25:25.348+0000 7f5f53fff700 1 -- 172.21.15.114:0/1825561570 reap_dead start 2024-04-23T10:25:25.543 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:25:25.348+0000 7f5f59be3700 1 -- 172.21.15.114:0/1825561570 shutdown_connections 2024-04-23T10:25:25.543 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:25:25.348+0000 7f5f59be3700 1 -- 172.21.15.114:0/1825561570 >> 172.21.15.114:0/1825561570 conn(0x7f5f540788f0 msgr2=0x7f5f540ffa90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T10:25:25.543 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:25:25.348+0000 7f5f59be3700 1 -- 172.21.15.114:0/1825561570 shutdown_connections 2024-04-23T10:25:25.543 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:25:25.348+0000 7f5f59be3700 1 -- 172.21.15.114:0/1825561570 wait complete. 2024-04-23T10:25:25.578 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 23 10:25:25 smithi204 ceph-mon[32649]: pgmap v306: 97 pgs: 97 active+clean; 641 KiB data, 370 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 597 B/s wr, 7 op/s 2024-04-23T10:25:26.246 INFO:teuthology.orchestra.run.smithi114.stdout:296352743456 2024-04-23T10:25:26.247 DEBUG:teuthology.orchestra.run.smithi114:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d69f13d827952bb07bb9b7973bca14f97d764d6f shell --fsid aa240572-0159-11ef-bc93-c7b262605968 -- ceph osd last-stat-seq osd.9 2024-04-23T10:25:26.870 INFO:teuthology.orchestra.run.smithi114.stdout:227633266730 2024-04-23T10:25:26.870 DEBUG:teuthology.orchestra.run.smithi114:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d69f13d827952bb07bb9b7973bca14f97d764d6f shell --fsid aa240572-0159-11ef-bc93-c7b262605968 -- ceph osd last-stat-seq osd.7 2024-04-23T10:25:27.254 INFO:journalctl@ceph.mon.a.smithi114.stdout:Apr 23 10:25:27 smithi114 ceph-mon[29255]: pgmap v307: 97 pgs: 97 active+clean; 641 KiB data, 370 MiB used, 1.0 TiB / 1.0 TiB avail; 2.7 KiB/s rd, 170 B/s wr, 5 op/s 2024-04-23T10:25:27.379 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 23 10:25:27 smithi204 ceph-mon[32649]: pgmap v307: 97 pgs: 97 active+clean; 641 KiB data, 370 MiB used, 1.0 TiB / 1.0 TiB avail; 2.7 KiB/s rd, 170 B/s wr, 5 op/s 2024-04-23T10:25:27.425 INFO:teuthology.orchestra.run.smithi114.stdout:77309411395 2024-04-23T10:25:27.426 DEBUG:teuthology.orchestra.run.smithi114:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d69f13d827952bb07bb9b7973bca14f97d764d6f shell --fsid aa240572-0159-11ef-bc93-c7b262605968 -- ceph osd last-stat-seq osd.2 2024-04-23T10:25:27.491 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 23 10:25:27 smithi119 ceph-mon[32608]: pgmap v307: 97 pgs: 97 active+clean; 641 KiB data, 370 MiB used, 1.0 TiB / 1.0 TiB avail; 2.7 KiB/s rd, 170 B/s wr, 5 op/s 2024-04-23T10:25:27.808 INFO:teuthology.orchestra.run.smithi114.stdout:34359738444 2024-04-23T10:25:27.808 DEBUG:teuthology.orchestra.run.smithi114:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d69f13d827952bb07bb9b7973bca14f97d764d6f shell --fsid aa240572-0159-11ef-bc93-c7b262605968 -- ceph osd last-stat-seq osd.0 2024-04-23T10:25:27.892 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:25:21.960+0000 7fa3400b8700 1 -- 172.21.15.114:0/1355691384 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fa338105480 msgr2=0x7fa338105910 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T10:25:27.892 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:25:21.960+0000 7fa3400b8700 1 --2- 172.21.15.114:0/1355691384 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fa338105480 0x7fa338105910 secure :-1 s=READY pgs=260 cs=0 l=1 rev1=1 crypto rx=0x7fa33000b0a0 tx=0x7fa33000b3b0 comp rx=0 tx=0).stop 2024-04-23T10:25:27.892 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:25:21.960+0000 7fa3400b8700 1 -- 172.21.15.114:0/1355691384 shutdown_connections 2024-04-23T10:25:27.892 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:25:21.960+0000 7fa3400b8700 1 --2- 172.21.15.114:0/1355691384 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fa338105480 0x7fa338105910 unknown :-1 s=CLOSED pgs=260 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:25:27.893 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:25:21.960+0000 7fa3400b8700 1 --2- 172.21.15.114:0/1355691384 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fa338107970 0x7fa33806a750 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:25:27.893 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:25:21.960+0000 7fa3400b8700 1 --2- 172.21.15.114:0/1355691384 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fa33806ac90 0x7fa33806b0f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:25:27.893 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:25:21.960+0000 7fa3400b8700 1 -- 172.21.15.114:0/1355691384 >> 172.21.15.114:0/1355691384 conn(0x7fa3380788f0 msgr2=0x7fa338078d00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T10:25:27.893 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:25:21.960+0000 7fa3400b8700 1 -- 172.21.15.114:0/1355691384 shutdown_connections 2024-04-23T10:25:27.893 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:25:21.960+0000 7fa3400b8700 1 -- 172.21.15.114:0/1355691384 wait complete. 2024-04-23T10:25:27.893 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:25:21.960+0000 7fa3400b8700 1 Processor -- start 2024-04-23T10:25:27.893 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:25:21.960+0000 7fa3400b8700 1 -- start start 2024-04-23T10:25:27.893 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:25:21.961+0000 7fa3400b8700 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fa33806ac90 0x7fa33819f640 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T10:25:27.893 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:25:21.961+0000 7fa3400b8700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fa338105480 0x7fa33819fb80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T10:25:27.893 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:25:21.961+0000 7fa3400b8700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fa338107970 0x7fa3381a3f10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T10:25:27.893 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:25:21.961+0000 7fa3400b8700 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x7fa3381191b0 con 0x7fa338105480 2024-04-23T10:25:27.893 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:25:21.961+0000 7fa3400b8700 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 -- 0x7fa338119030 con 0x7fa338107970 2024-04-23T10:25:27.893 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:25:21.961+0000 7fa3400b8700 1 -- --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_getmap magic: 0 -- 0x7fa338119330 con 0x7fa33806ac90 2024-04-23T10:25:27.893 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:25:21.961+0000 7fa33d653700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fa338105480 0x7fa33819fb80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T10:25:27.893 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:25:21.961+0000 7fa33d653700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fa338105480 0x7fa33819fb80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.114:36288/0 (socket says 172.21.15.114:36288) 2024-04-23T10:25:27.894 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:25:21.961+0000 7fa33d653700 1 -- 172.21.15.114:0/3808184479 learned_addr learned my addr 172.21.15.114:0/3808184479 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-04-23T10:25:27.894 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:25:21.961+0000 7fa33de54700 1 --2- 172.21.15.114:0/3808184479 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fa33806ac90 0x7fa33819f640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T10:25:27.894 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:25:21.961+0000 7fa33e655700 1 --2- 172.21.15.114:0/3808184479 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fa338107970 0x7fa3381a3f10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T10:25:27.894 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:25:21.961+0000 7fa33d653700 1 -- 172.21.15.114:0/3808184479 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fa33806ac90 msgr2=0x7fa33819f640 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T10:25:27.894 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:25:21.961+0000 7fa33d653700 1 --2- 172.21.15.114:0/3808184479 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fa33806ac90 0x7fa33819f640 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:25:27.894 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:25:21.961+0000 7fa33d653700 1 -- 172.21.15.114:0/3808184479 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fa338107970 msgr2=0x7fa3381a3f10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T10:25:27.894 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:25:21.961+0000 7fa33d653700 1 --2- 172.21.15.114:0/3808184479 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fa338107970 0x7fa3381a3f10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:25:27.894 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:25:21.961+0000 7fa33d653700 1 -- 172.21.15.114:0/3808184479 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa3381a4650 con 0x7fa338105480 2024-04-23T10:25:27.895 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:25:21.961+0000 7fa33e655700 1 --2- 172.21.15.114:0/3808184479 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fa338107970 0x7fa3381a3f10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-23T10:25:27.895 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:25:21.961+0000 7fa33d653700 1 --2- 172.21.15.114:0/3808184479 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fa338105480 0x7fa33819fb80 secure :-1 s=READY pgs=261 cs=0 l=1 rev1=1 crypto rx=0x7fa32800d950 tx=0x7fa32800dc60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T10:25:27.895 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:25:21.961+0000 7fa32effd700 1 -- 172.21.15.114:0/3808184479 <== mon.0 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa328016070 con 0x7fa338105480 2024-04-23T10:25:27.895 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:25:21.961+0000 7fa32effd700 1 -- 172.21.15.114:0/3808184479 <== mon.0 v2:172.21.15.114:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fa328009df0 con 0x7fa338105480 2024-04-23T10:25:27.895 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:25:21.961+0000 7fa3400b8700 1 -- 172.21.15.114:0/3808184479 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fa3381a4940 con 0x7fa338105480 2024-04-23T10:25:27.895 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:25:21.961+0000 7fa32effd700 1 -- 172.21.15.114:0/3808184479 <== mon.0 v2:172.21.15.114:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa32800f3c0 con 0x7fa338105480 2024-04-23T10:25:27.895 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:25:21.961+0000 7fa3400b8700 1 -- 172.21.15.114:0/3808184479 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fa3381ac1c0 con 0x7fa338105480 2024-04-23T10:25:27.895 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:25:21.962+0000 7fa3400b8700 1 -- 172.21.15.114:0/3808184479 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_get_version(what=osdmap handle=1) -- 0x7fa338104ec0 con 0x7fa338105480 2024-04-23T10:25:27.895 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:25:21.963+0000 7fa32effd700 1 -- 172.21.15.114:0/3808184479 <== mon.0 v2:172.21.15.114:3300/0 4 ==== mgrmap(e 17) ==== 138698+0+0 (secure 0 0 0) 0x7fa328019070 con 0x7fa338105480 2024-04-23T10:25:27.895 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:25:21.964+0000 7fa32effd700 1 --2- 172.21.15.114:0/3808184479 >> [v2:172.21.15.114:6800/465333808,v1:172.21.15.114:6801/465333808] conn(0x7fa3240a3cd0 0x7fa3240a6190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T10:25:27.895 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:25:21.964+0000 7fa33de54700 1 --2- 172.21.15.114:0/3808184479 >> [v2:172.21.15.114:6800/465333808,v1:172.21.15.114:6801/465333808] conn(0x7fa3240a3cd0 0x7fa3240a6190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T10:25:27.895 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:25:21.964+0000 7fa32effd700 1 -- 172.21.15.114:0/3808184479 <== mon.0 v2:172.21.15.114:3300/0 5 ==== osd_map(87..87 src has 1..87) ==== 7979+0+0 (secure 0 0 0) 0x7fa3280d0490 con 0x7fa338105480 2024-04-23T10:25:27.895 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:25:21.964+0000 7fa32effd700 1 --2- 172.21.15.114:0/3808184479 >> [v2:172.21.15.204:6808/1427563312,v1:172.21.15.204:6809/1427563312] conn(0x7fa3240ad4b0 0x7fa3240af8d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T10:25:27.895 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:25:21.964+0000 7fa32effd700 1 -- 172.21.15.114:0/3808184479 --> [v2:172.21.15.204:6808/1427563312,v1:172.21.15.204:6809/1427563312] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7fa3240aff80 con 0x7fa3240ad4b0 2024-04-23T10:25:27.896 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:25:21.964+0000 7fa32effd700 1 -- 172.21.15.114:0/3808184479 <== mon.0 v2:172.21.15.114:3300/0 6 ==== mon_get_version_reply(handle=1 version=87) ==== 24+0+0 (secure 0 0 0) 0x7fa3280d0970 con 0x7fa338105480 2024-04-23T10:25:27.896 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:25:21.964+0000 7fa33de54700 1 --2- 172.21.15.114:0/3808184479 >> [v2:172.21.15.114:6800/465333808,v1:172.21.15.114:6801/465333808] conn(0x7fa3240a3cd0 0x7fa3240a6190 secure :-1 s=READY pgs=189 cs=0 l=1 rev1=1 crypto rx=0x7fa334005950 tx=0x7fa33400a300 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T10:25:27.896 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:25:21.964+0000 7fa33e655700 1 --2- 172.21.15.114:0/3808184479 >> [v2:172.21.15.204:6808/1427563312,v1:172.21.15.204:6809/1427563312] conn(0x7fa3240ad4b0 0x7fa3240af8d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T10:25:27.896 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:25:21.965+0000 7fa33e655700 1 --2- 172.21.15.114:0/3808184479 >> [v2:172.21.15.204:6808/1427563312,v1:172.21.15.204:6809/1427563312] conn(0x7fa3240ad4b0 0x7fa3240af8d0 crc :-1 s=READY pgs=12 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.5 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T10:25:27.896 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:25:21.966+0000 7fa32effd700 1 -- 172.21.15.114:0/3808184479 <== osd.5 v2:172.21.15.204:6808/1427563312 1 ==== command_reply(tid 1: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7fa3240aff80 con 0x7fa3240ad4b0 2024-04-23T10:25:27.896 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:25:21.982+0000 7fa3400b8700 1 -- 172.21.15.114:0/3808184479 --> [v2:172.21.15.204:6808/1427563312,v1:172.21.15.204:6809/1427563312] -- command(tid 2: {"prefix": "flush_pg_stats"}) -- 0x7fa338101f60 con 0x7fa3240ad4b0 2024-04-23T10:25:27.896 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:25:21.982+0000 7fa32effd700 1 -- 172.21.15.114:0/3808184479 <== osd.5 v2:172.21.15.204:6808/1427563312 2 ==== command_reply(tid 2: 0 ) ==== 8+0+12 (crc 0 0 0) 0x7fa338101f60 con 0x7fa3240ad4b0 2024-04-23T10:25:27.896 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:25:21.982+0000 7fa3400b8700 1 -- 172.21.15.114:0/3808184479 >> [v2:172.21.15.204:6808/1427563312,v1:172.21.15.204:6809/1427563312] conn(0x7fa3240ad4b0 msgr2=0x7fa3240af8d0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T10:25:27.896 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:25:21.982+0000 7fa3400b8700 1 --2- 172.21.15.114:0/3808184479 >> [v2:172.21.15.204:6808/1427563312,v1:172.21.15.204:6809/1427563312] conn(0x7fa3240ad4b0 0x7fa3240af8d0 crc :-1 s=READY pgs=12 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:25:27.896 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:25:21.983+0000 7fa3400b8700 1 -- 172.21.15.114:0/3808184479 >> [v2:172.21.15.114:6800/465333808,v1:172.21.15.114:6801/465333808] conn(0x7fa3240a3cd0 msgr2=0x7fa3240a6190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T10:25:27.896 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:25:21.983+0000 7fa3400b8700 1 --2- 172.21.15.114:0/3808184479 >> [v2:172.21.15.114:6800/465333808,v1:172.21.15.114:6801/465333808] conn(0x7fa3240a3cd0 0x7fa3240a6190 secure :-1 s=READY pgs=189 cs=0 l=1 rev1=1 crypto rx=0x7fa334005950 tx=0x7fa33400a300 comp rx=0 tx=0).stop 2024-04-23T10:25:27.896 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:25:21.983+0000 7fa3400b8700 1 -- 172.21.15.114:0/3808184479 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fa338105480 msgr2=0x7fa33819fb80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T10:25:27.896 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:25:21.983+0000 7fa3400b8700 1 --2- 172.21.15.114:0/3808184479 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fa338105480 0x7fa33819fb80 secure :-1 s=READY pgs=261 cs=0 l=1 rev1=1 crypto rx=0x7fa32800d950 tx=0x7fa32800dc60 comp rx=0 tx=0).stop 2024-04-23T10:25:27.896 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:25:21.983+0000 7fa33e655700 1 -- 172.21.15.114:0/3808184479 reap_dead start 2024-04-23T10:25:27.896 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:25:21.983+0000 7fa3400b8700 1 -- 172.21.15.114:0/3808184479 shutdown_connections 2024-04-23T10:25:27.897 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:25:21.983+0000 7fa3400b8700 1 -- 172.21.15.114:0/3808184479 >> 172.21.15.114:0/3808184479 conn(0x7fa3380788f0 msgr2=0x7fa338104460 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T10:25:27.897 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:25:21.983+0000 7fa3400b8700 1 -- 172.21.15.114:0/3808184479 shutdown_connections 2024-04-23T10:25:27.897 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:25:21.983+0000 7fa3400b8700 1 -- 172.21.15.114:0/3808184479 wait complete. 2024-04-23T10:25:28.439 INFO:teuthology.orchestra.run.smithi114.stdout:167503724596 2024-04-23T10:25:28.439 DEBUG:teuthology.orchestra.run.smithi114:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d69f13d827952bb07bb9b7973bca14f97d764d6f shell --fsid aa240572-0159-11ef-bc93-c7b262605968 -- ceph osd last-stat-seq osd.5 2024-04-23T10:25:28.522 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:25:26.485+0000 7f3c9ebc4700 1 -- 172.21.15.114:0/874912163 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f3c98101940 msgr2=0x7f3c98101dc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T10:25:28.522 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:25:26.485+0000 7f3c9ebc4700 1 --2- 172.21.15.114:0/874912163 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f3c98101940 0x7f3c98101dc0 secure :-1 s=READY pgs=272 cs=0 l=1 rev1=1 crypto rx=0x7f3c84009990 tx=0x7f3c84009ca0 comp rx=0 tx=0).stop 2024-04-23T10:25:28.522 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:25:26.485+0000 7f3c9ebc4700 1 -- 172.21.15.114:0/874912163 shutdown_connections 2024-04-23T10:25:28.522 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:25:26.485+0000 7f3c9ebc4700 1 --2- 172.21.15.114:0/874912163 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f3c98107990 0x7f3c98107d70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:25:28.522 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:25:26.485+0000 7f3c9ebc4700 1 --2- 172.21.15.114:0/874912163 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f3c98102300 0x7f3c9810e7f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:25:28.522 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:25:26.485+0000 7f3c9ebc4700 1 --2- 172.21.15.114:0/874912163 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f3c98101940 0x7f3c98101dc0 unknown :-1 s=CLOSED pgs=272 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:25:28.522 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:25:26.485+0000 7f3c9ebc4700 1 -- 172.21.15.114:0/874912163 >> 172.21.15.114:0/874912163 conn(0x7f3c980788f0 msgr2=0x7f3c98078d00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T10:25:28.522 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:25:26.485+0000 7f3c9ebc4700 1 -- 172.21.15.114:0/874912163 shutdown_connections 2024-04-23T10:25:28.522 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:25:26.485+0000 7f3c9ebc4700 1 -- 172.21.15.114:0/874912163 wait complete. 2024-04-23T10:25:28.522 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:25:26.486+0000 7f3c9ebc4700 1 Processor -- start 2024-04-23T10:25:28.522 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:25:26.486+0000 7f3c9ebc4700 1 -- start start 2024-04-23T10:25:28.522 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:25:26.486+0000 7f3c9ebc4700 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f3c98101940 0x7f3c98198120 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T10:25:28.522 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:25:26.486+0000 7f3c9ebc4700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f3c98102300 0x7f3c98198660 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T10:25:28.523 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:25:26.486+0000 7f3c9ebc4700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f3c98107990 0x7f3c9819c9f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T10:25:28.523 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:25:26.486+0000 7f3c9ebc4700 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x7f3c98111bd0 con 0x7f3c98107990 2024-04-23T10:25:28.523 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:25:26.486+0000 7f3c9ebc4700 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 -- 0x7f3c98111a50 con 0x7f3c98102300 2024-04-23T10:25:28.523 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:25:26.486+0000 7f3c9ebc4700 1 -- --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_getmap magic: 0 -- 0x7f3c98111d50 con 0x7f3c98101940 2024-04-23T10:25:28.523 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:25:26.486+0000 7f3c9d161700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f3c98107990 0x7f3c9819c9f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T10:25:28.523 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:25:26.486+0000 7f3c9d161700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f3c98107990 0x7f3c9819c9f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.114:38518/0 (socket says 172.21.15.114:38518) 2024-04-23T10:25:28.523 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:25:26.487+0000 7f3c9d161700 1 -- 172.21.15.114:0/1179295947 learned_addr learned my addr 172.21.15.114:0/1179295947 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-04-23T10:25:28.523 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:25:26.487+0000 7f3c9c960700 1 --2- 172.21.15.114:0/1179295947 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f3c98101940 0x7f3c98198120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T10:25:28.523 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:25:26.487+0000 7f3c9d161700 1 -- 172.21.15.114:0/1179295947 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f3c98101940 msgr2=0x7f3c98198120 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T10:25:28.523 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:25:26.487+0000 7f3c9d161700 1 --2- 172.21.15.114:0/1179295947 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f3c98101940 0x7f3c98198120 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:25:28.523 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:25:26.487+0000 7f3c9d161700 1 -- 172.21.15.114:0/1179295947 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f3c98102300 msgr2=0x7f3c98198660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T10:25:28.524 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:25:26.487+0000 7f3c9d161700 1 --2- 172.21.15.114:0/1179295947 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f3c98102300 0x7f3c98198660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:25:28.524 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:25:26.487+0000 7f3c9d161700 1 -- 172.21.15.114:0/1179295947 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f3c9819d130 con 0x7f3c98107990 2024-04-23T10:25:28.524 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:25:26.487+0000 7f3c9d161700 1 --2- 172.21.15.114:0/1179295947 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f3c98107990 0x7f3c9819c9f0 secure :-1 s=READY pgs=273 cs=0 l=1 rev1=1 crypto rx=0x7f3c9000b970 tx=0x7f3c9000bd30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T10:25:28.524 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:25:26.487+0000 7f3c95ffb700 1 -- 172.21.15.114:0/1179295947 <== mon.0 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3c90019070 con 0x7f3c98107990 2024-04-23T10:25:28.524 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:25:26.487+0000 7f3c9ebc4700 1 -- 172.21.15.114:0/1179295947 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f3c9819d420 con 0x7f3c98107990 2024-04-23T10:25:28.524 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:25:26.487+0000 7f3c95ffb700 1 -- 172.21.15.114:0/1179295947 <== mon.0 v2:172.21.15.114:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f3c90014440 con 0x7f3c98107990 2024-04-23T10:25:28.524 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:25:26.487+0000 7f3c95ffb700 1 -- 172.21.15.114:0/1179295947 <== mon.0 v2:172.21.15.114:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3c9000ce60 con 0x7f3c98107990 2024-04-23T10:25:28.524 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:25:26.487+0000 7f3c9ebc4700 1 -- 172.21.15.114:0/1179295947 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f3c981a4ca0 con 0x7f3c98107990 2024-04-23T10:25:28.524 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:25:26.488+0000 7f3c9ebc4700 1 -- 172.21.15.114:0/1179295947 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_get_version(what=osdmap handle=1) -- 0x7f3c980631d0 con 0x7f3c98107990 2024-04-23T10:25:28.524 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:25:26.489+0000 7f3c95ffb700 1 -- 172.21.15.114:0/1179295947 <== mon.0 v2:172.21.15.114:3300/0 4 ==== mgrmap(e 17) ==== 138698+0+0 (secure 0 0 0) 0x7f3c9001c070 con 0x7f3c98107990 2024-04-23T10:25:28.524 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:25:26.490+0000 7f3c95ffb700 1 --2- 172.21.15.114:0/1179295947 >> [v2:172.21.15.114:6800/465333808,v1:172.21.15.114:6801/465333808] conn(0x7f3c880a3cd0 0x7f3c880a6190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T10:25:28.524 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:25:26.490+0000 7f3c9c960700 1 --2- 172.21.15.114:0/1179295947 >> [v2:172.21.15.114:6800/465333808,v1:172.21.15.114:6801/465333808] conn(0x7f3c880a3cd0 0x7f3c880a6190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T10:25:28.524 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:25:26.490+0000 7f3c95ffb700 1 -- 172.21.15.114:0/1179295947 <== mon.0 v2:172.21.15.114:3300/0 5 ==== osd_map(87..87 src has 1..87) ==== 7979+0+0 (secure 0 0 0) 0x7f3c900cf720 con 0x7f3c98107990 2024-04-23T10:25:28.524 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:25:26.490+0000 7f3c95ffb700 1 --2- 172.21.15.114:0/1179295947 >> [v2:172.21.15.119:6808/1611007172,v1:172.21.15.119:6809/1611007172] conn(0x7f3c880ad4b0 0x7f3c880af8d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T10:25:28.524 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:25:26.490+0000 7f3c95ffb700 1 -- 172.21.15.114:0/1179295947 --> [v2:172.21.15.119:6808/1611007172,v1:172.21.15.119:6809/1611007172] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f3c880aff80 con 0x7f3c880ad4b0 2024-04-23T10:25:28.525 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:25:26.490+0000 7f3c95ffb700 1 -- 172.21.15.114:0/1179295947 <== mon.0 v2:172.21.15.114:3300/0 6 ==== mon_get_version_reply(handle=1 version=87) ==== 24+0+0 (secure 0 0 0) 0x7f3c900d6050 con 0x7f3c98107990 2024-04-23T10:25:28.525 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:25:26.490+0000 7f3c9c960700 1 --2- 172.21.15.114:0/1179295947 >> [v2:172.21.15.114:6800/465333808,v1:172.21.15.114:6801/465333808] conn(0x7f3c880a3cd0 0x7f3c880a6190 secure :-1 s=READY pgs=195 cs=0 l=1 rev1=1 crypto rx=0x7f3c8c006fd0 tx=0x7f3c8c008040 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T10:25:28.525 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:25:26.490+0000 7f3c97fff700 1 --2- 172.21.15.114:0/1179295947 >> [v2:172.21.15.119:6808/1611007172,v1:172.21.15.119:6809/1611007172] conn(0x7f3c880ad4b0 0x7f3c880af8d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T10:25:28.525 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:25:26.491+0000 7f3c97fff700 1 --2- 172.21.15.114:0/1179295947 >> [v2:172.21.15.119:6808/1611007172,v1:172.21.15.119:6809/1611007172] conn(0x7f3c880ad4b0 0x7f3c880af8d0 crc :-1 s=READY pgs=11 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.4 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T10:25:28.525 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:25:26.492+0000 7f3c95ffb700 1 -- 172.21.15.114:0/1179295947 <== osd.4 v2:172.21.15.119:6808/1611007172 1 ==== command_reply(tid 1: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7f3c880aff80 con 0x7f3c880ad4b0 2024-04-23T10:25:28.525 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:25:26.508+0000 7f3c9ebc4700 1 -- 172.21.15.114:0/1179295947 --> [v2:172.21.15.119:6808/1611007172,v1:172.21.15.119:6809/1611007172] -- command(tid 2: {"prefix": "flush_pg_stats"}) -- 0x7f3c98063650 con 0x7f3c880ad4b0 2024-04-23T10:25:28.525 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:25:26.508+0000 7f3c95ffb700 1 -- 172.21.15.114:0/1179295947 <== osd.4 v2:172.21.15.119:6808/1611007172 2 ==== command_reply(tid 2: 0 ) ==== 8+0+12 (crc 0 0 0) 0x7f3c98063650 con 0x7f3c880ad4b0 2024-04-23T10:25:28.525 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:25:26.509+0000 7f3c9ebc4700 1 -- 172.21.15.114:0/1179295947 >> [v2:172.21.15.119:6808/1611007172,v1:172.21.15.119:6809/1611007172] conn(0x7f3c880ad4b0 msgr2=0x7f3c880af8d0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T10:25:28.525 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:25:26.509+0000 7f3c9ebc4700 1 --2- 172.21.15.114:0/1179295947 >> [v2:172.21.15.119:6808/1611007172,v1:172.21.15.119:6809/1611007172] conn(0x7f3c880ad4b0 0x7f3c880af8d0 crc :-1 s=READY pgs=11 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:25:28.525 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:25:26.509+0000 7f3c9ebc4700 1 -- 172.21.15.114:0/1179295947 >> [v2:172.21.15.114:6800/465333808,v1:172.21.15.114:6801/465333808] conn(0x7f3c880a3cd0 msgr2=0x7f3c880a6190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T10:25:28.525 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:25:26.509+0000 7f3c9ebc4700 1 --2- 172.21.15.114:0/1179295947 >> [v2:172.21.15.114:6800/465333808,v1:172.21.15.114:6801/465333808] conn(0x7f3c880a3cd0 0x7f3c880a6190 secure :-1 s=READY pgs=195 cs=0 l=1 rev1=1 crypto rx=0x7f3c8c006fd0 tx=0x7f3c8c008040 comp rx=0 tx=0).stop 2024-04-23T10:25:28.525 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:25:26.509+0000 7f3c9ebc4700 1 -- 172.21.15.114:0/1179295947 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f3c98107990 msgr2=0x7f3c9819c9f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T10:25:28.525 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:25:26.509+0000 7f3c9ebc4700 1 --2- 172.21.15.114:0/1179295947 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f3c98107990 0x7f3c9819c9f0 secure :-1 s=READY pgs=273 cs=0 l=1 rev1=1 crypto rx=0x7f3c9000b970 tx=0x7f3c9000bd30 comp rx=0 tx=0).stop 2024-04-23T10:25:28.525 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:25:26.509+0000 7f3c9d161700 1 -- 172.21.15.114:0/1179295947 reap_dead start 2024-04-23T10:25:28.525 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:25:26.509+0000 7f3c9ebc4700 1 -- 172.21.15.114:0/1179295947 shutdown_connections 2024-04-23T10:25:28.526 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:25:26.509+0000 7f3c9ebc4700 1 -- 172.21.15.114:0/1179295947 >> 172.21.15.114:0/1179295947 conn(0x7f3c980788f0 msgr2=0x7f3c980ffb50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T10:25:28.526 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:25:26.509+0000 7f3c9ebc4700 1 -- 172.21.15.114:0/1179295947 shutdown_connections 2024-04-23T10:25:28.526 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:25:26.509+0000 7f3c9ebc4700 1 -- 172.21.15.114:0/1179295947 wait complete. 2024-04-23T10:25:29.030 INFO:teuthology.orchestra.run.smithi114.stdout:141733920824 2024-04-23T10:25:29.030 DEBUG:teuthology.orchestra.run.smithi114:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d69f13d827952bb07bb9b7973bca14f97d764d6f shell --fsid aa240572-0159-11ef-bc93-c7b262605968 -- ceph osd last-stat-seq osd.4 2024-04-23T10:25:29.492 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 23 10:25:29 smithi119 ceph-mon[32608]: pgmap v308: 97 pgs: 97 active+clean; 641 KiB data, 370 MiB used, 1.0 TiB / 1.0 TiB avail; 3.0 KiB/s rd, 170 B/s wr, 6 op/s 2024-04-23T10:25:29.504 INFO:journalctl@ceph.mon.a.smithi114.stdout:Apr 23 10:25:29 smithi114 ceph-mon[29255]: pgmap v308: 97 pgs: 97 active+clean; 641 KiB data, 370 MiB used, 1.0 TiB / 1.0 TiB avail; 3.0 KiB/s rd, 170 B/s wr, 6 op/s 2024-04-23T10:25:29.542 INFO:teuthology.orchestra.run.smithi114.stdout:352187318294 2024-04-23T10:25:29.543 DEBUG:teuthology.orchestra.run.smithi114:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d69f13d827952bb07bb9b7973bca14f97d764d6f shell --fsid aa240572-0159-11ef-bc93-c7b262605968 -- ceph osd last-stat-seq osd.11 2024-04-23T10:25:29.577 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 23 10:25:29 smithi204 ceph-mon[32649]: pgmap v308: 97 pgs: 97 active+clean; 641 KiB data, 370 MiB used, 1.0 TiB / 1.0 TiB avail; 3.0 KiB/s rd, 170 B/s wr, 6 op/s 2024-04-23T10:25:29.650 INFO:teuthology.orchestra.run.smithi114.stderr:Inferring config /var/lib/ceph/aa240572-0159-11ef-bc93-c7b262605968/mon.a/config 2024-04-23T10:25:29.652 INFO:teuthology.orchestra.run.smithi114.stderr:Inferring config /var/lib/ceph/aa240572-0159-11ef-bc93-c7b262605968/mon.a/config 2024-04-23T10:25:29.655 INFO:teuthology.orchestra.run.smithi114.stderr:Inferring config /var/lib/ceph/aa240572-0159-11ef-bc93-c7b262605968/mon.a/config 2024-04-23T10:25:29.655 INFO:teuthology.orchestra.run.smithi114.stderr:Inferring config /var/lib/ceph/aa240572-0159-11ef-bc93-c7b262605968/mon.a/config 2024-04-23T10:25:29.655 INFO:teuthology.orchestra.run.smithi114.stderr:Inferring config /var/lib/ceph/aa240572-0159-11ef-bc93-c7b262605968/mon.a/config 2024-04-23T10:25:31.142 INFO:teuthology.orchestra.run.smithi114.stderr:Inferring config /var/lib/ceph/aa240572-0159-11ef-bc93-c7b262605968/mon.a/config 2024-04-23T10:25:31.142 INFO:teuthology.orchestra.run.smithi114.stderr:Inferring config /var/lib/ceph/aa240572-0159-11ef-bc93-c7b262605968/mon.a/config 2024-04-23T10:25:31.222 INFO:journalctl@ceph.mon.a.smithi114.stdout:Apr 23 10:25:31 smithi114 ceph-mon[29255]: pgmap v309: 97 pgs: 97 active+clean; 641 KiB data, 370 MiB used, 1.0 TiB / 1.0 TiB avail; 2.8 KiB/s rd, 170 B/s wr, 5 op/s 2024-04-23T10:25:31.492 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 23 10:25:31 smithi119 ceph-mon[32608]: pgmap v309: 97 pgs: 97 active+clean; 641 KiB data, 370 MiB used, 1.0 TiB / 1.0 TiB avail; 2.8 KiB/s rd, 170 B/s wr, 5 op/s 2024-04-23T10:25:31.578 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 23 10:25:31 smithi204 ceph-mon[32649]: pgmap v309: 97 pgs: 97 active+clean; 641 KiB data, 370 MiB used, 1.0 TiB / 1.0 TiB avail; 2.8 KiB/s rd, 170 B/s wr, 5 op/s 2024-04-23T10:25:33.254 INFO:journalctl@ceph.mon.a.smithi114.stdout:Apr 23 10:25:33 smithi114 ceph-mon[29255]: pgmap v310: 97 pgs: 97 active+clean; 641 KiB data, 370 MiB used, 1.0 TiB / 1.0 TiB avail; 2.7 KiB/s rd, 170 B/s wr, 5 op/s 2024-04-23T10:25:33.492 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 23 10:25:33 smithi119 ceph-mon[32608]: pgmap v310: 97 pgs: 97 active+clean; 641 KiB data, 370 MiB used, 1.0 TiB / 1.0 TiB avail; 2.7 KiB/s rd, 170 B/s wr, 5 op/s 2024-04-23T10:25:33.578 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 23 10:25:33 smithi204 ceph-mon[32649]: pgmap v310: 97 pgs: 97 active+clean; 641 KiB data, 370 MiB used, 1.0 TiB / 1.0 TiB avail; 2.7 KiB/s rd, 170 B/s wr, 5 op/s 2024-04-23T10:25:35.492 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 23 10:25:35 smithi119 ceph-mon[32608]: pgmap v311: 97 pgs: 97 active+clean; 641 KiB data, 370 MiB used, 1.0 TiB / 1.0 TiB avail; 3.9 KiB/s rd, 170 B/s wr, 7 op/s 2024-04-23T10:25:35.504 INFO:journalctl@ceph.mon.a.smithi114.stdout:Apr 23 10:25:35 smithi114 ceph-mon[29255]: pgmap v311: 97 pgs: 97 active+clean; 641 KiB data, 370 MiB used, 1.0 TiB / 1.0 TiB avail; 3.9 KiB/s rd, 170 B/s wr, 7 op/s 2024-04-23T10:25:35.578 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 23 10:25:35 smithi204 ceph-mon[32649]: pgmap v311: 97 pgs: 97 active+clean; 641 KiB data, 370 MiB used, 1.0 TiB / 1.0 TiB avail; 3.9 KiB/s rd, 170 B/s wr, 7 op/s 2024-04-23T10:25:36.664 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:25:36.665+0000 7f1086316700 1 -- 172.21.15.114:0/1371626511 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f1080102ac0 msgr2=0x7f1080102f40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T10:25:36.665 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:25:36.665+0000 7f1086316700 1 --2- 172.21.15.114:0/1371626511 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f1080102ac0 0x7f1080102f40 secure :-1 s=READY pgs=274 cs=0 l=1 rev1=1 crypto rx=0x7f1074009990 tx=0x7f1074009ca0 comp rx=0 tx=0).stop 2024-04-23T10:25:36.665 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:25:36.665+0000 7f1086316700 1 -- 172.21.15.114:0/1371626511 shutdown_connections 2024-04-23T10:25:36.665 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:25:36.665+0000 7f1086316700 1 --2- 172.21.15.114:0/1371626511 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f1080108a70 0x7f1080108e50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:25:36.665 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:25:36.665+0000 7f1086316700 1 --2- 172.21.15.114:0/1371626511 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f1080103480 0x7f108010f8d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:25:36.665 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:25:36.665+0000 7f1086316700 1 --2- 172.21.15.114:0/1371626511 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f1080102ac0 0x7f1080102f40 unknown :-1 s=CLOSED pgs=274 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:25:36.665 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:25:36.665+0000 7f1086316700 1 -- 172.21.15.114:0/1371626511 >> 172.21.15.114:0/1371626511 conn(0x7f1080079bb0 msgr2=0x7f1080079fc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T10:25:36.665 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:25:36.665+0000 7f1086316700 1 -- 172.21.15.114:0/1371626511 shutdown_connections 2024-04-23T10:25:36.665 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:25:36.666+0000 7f1086316700 1 -- 172.21.15.114:0/1371626511 wait complete. 2024-04-23T10:25:36.667 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:25:36.666+0000 7f1086316700 1 Processor -- start 2024-04-23T10:25:36.667 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:25:36.666+0000 7f1086316700 1 -- start start 2024-04-23T10:25:36.667 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:25:36.666+0000 7f1086316700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f1080102ac0 0x7f1080199220 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T10:25:36.667 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:25:36.666+0000 7f1086316700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f1080103480 0x7f1080199760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T10:25:36.667 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:25:36.666+0000 7f1086316700 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f1080108a70 0x7f108019daf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T10:25:36.667 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:25:36.666+0000 7f1086316700 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x7f1080112c40 con 0x7f1080103480 2024-04-23T10:25:36.667 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:25:36.666+0000 7f1086316700 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 -- 0x7f1080112ac0 con 0x7f1080102ac0 2024-04-23T10:25:36.668 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:25:36.666+0000 7f1086316700 1 -- --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_getmap magic: 0 -- 0x7f1080112dc0 con 0x7f1080108a70 2024-04-23T10:25:36.668 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:25:36.667+0000 7f107f7fe700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f1080103480 0x7f1080199760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T10:25:36.668 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:25:36.667+0000 7f107f7fe700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f1080103480 0x7f1080199760 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.114:50788/0 (socket says 172.21.15.114:50788) 2024-04-23T10:25:36.668 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:25:36.667+0000 7f107f7fe700 1 -- 172.21.15.114:0/2570387136 learned_addr learned my addr 172.21.15.114:0/2570387136 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-04-23T10:25:36.668 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:25:36.667+0000 7f10848b3700 1 --2- 172.21.15.114:0/2570387136 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f1080108a70 0x7f108019daf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T10:25:36.668 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:25:36.667+0000 7f107ffff700 1 --2- 172.21.15.114:0/2570387136 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f1080102ac0 0x7f1080199220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T10:25:36.668 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:25:36.667+0000 7f107f7fe700 1 -- 172.21.15.114:0/2570387136 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f1080108a70 msgr2=0x7f108019daf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T10:25:36.668 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:25:36.667+0000 7f107f7fe700 1 --2- 172.21.15.114:0/2570387136 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f1080108a70 0x7f108019daf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:25:36.668 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:25:36.667+0000 7f107f7fe700 1 -- 172.21.15.114:0/2570387136 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f1080102ac0 msgr2=0x7f1080199220 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T10:25:36.668 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:25:36.667+0000 7f107f7fe700 1 --2- 172.21.15.114:0/2570387136 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f1080102ac0 0x7f1080199220 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:25:36.668 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:25:36.667+0000 7f107f7fe700 1 -- 172.21.15.114:0/2570387136 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f108019e230 con 0x7f1080103480 2024-04-23T10:25:36.668 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:25:36.667+0000 7f10848b3700 1 --2- 172.21.15.114:0/2570387136 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f1080108a70 0x7f108019daf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-23T10:25:36.668 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:25:36.667+0000 7f107f7fe700 1 --2- 172.21.15.114:0/2570387136 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f1080103480 0x7f1080199760 secure :-1 s=READY pgs=275 cs=0 l=1 rev1=1 crypto rx=0x7f1074000c00 tx=0x7f10740047b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T10:25:36.668 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:25:36.667+0000 7f107d7fa700 1 -- 172.21.15.114:0/2570387136 <== mon.0 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1074027070 con 0x7f1080103480 2024-04-23T10:25:36.668 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:25:36.667+0000 7f1086316700 1 -- 172.21.15.114:0/2570387136 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f108019e4c0 con 0x7f1080103480 2024-04-23T10:25:36.668 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:25:36.667+0000 7f107d7fa700 1 -- 172.21.15.114:0/2570387136 <== mon.0 v2:172.21.15.114:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f10740041d0 con 0x7f1080103480 2024-04-23T10:25:36.669 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:25:36.667+0000 7f107d7fa700 1 -- 172.21.15.114:0/2570387136 <== mon.0 v2:172.21.15.114:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f107400f460 con 0x7f1080103480 2024-04-23T10:25:36.669 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:25:36.667+0000 7f1086316700 1 -- 172.21.15.114:0/2570387136 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f10801a5da0 con 0x7f1080103480 2024-04-23T10:25:36.669 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:25:36.668+0000 7f1086316700 1 -- 172.21.15.114:0/2570387136 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f1080069730 con 0x7f1080103480 2024-04-23T10:25:36.669 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:25:36.669+0000 7f107d7fa700 1 -- 172.21.15.114:0/2570387136 <== mon.0 v2:172.21.15.114:3300/0 4 ==== mgrmap(e 17) ==== 138698+0+0 (secure 0 0 0) 0x7f107402a070 con 0x7f1080103480 2024-04-23T10:25:36.669 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:25:36.670+0000 7f107d7fa700 1 --2- 172.21.15.114:0/2570387136 >> [v2:172.21.15.114:6800/465333808,v1:172.21.15.114:6801/465333808] conn(0x7f10680a3cd0 0x7f10680a6190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T10:25:36.670 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:25:36.670+0000 7f107ffff700 1 --2- 172.21.15.114:0/2570387136 >> [v2:172.21.15.114:6800/465333808,v1:172.21.15.114:6801/465333808] conn(0x7f10680a3cd0 0x7f10680a6190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T10:25:36.670 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:25:36.670+0000 7f107d7fa700 1 -- 172.21.15.114:0/2570387136 <== mon.0 v2:172.21.15.114:3300/0 5 ==== osd_map(87..87 src has 1..87) ==== 7979+0+0 (secure 0 0 0) 0x7f10740d13e0 con 0x7f1080103480 2024-04-23T10:25:36.670 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:25:36.670+0000 7f107ffff700 1 --2- 172.21.15.114:0/2570387136 >> [v2:172.21.15.114:6800/465333808,v1:172.21.15.114:6801/465333808] conn(0x7f10680a3cd0 0x7f10680a6190 secure :-1 s=READY pgs=196 cs=0 l=1 rev1=1 crypto rx=0x7f1070005fd0 tx=0x7f1070005db0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T10:25:36.673 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:25:36.674+0000 7f107d7fa700 1 -- 172.21.15.114:0/2570387136 <== mon.0 v2:172.21.15.114:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f107405eff0 con 0x7f1080103480 2024-04-23T10:25:36.880 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:25:36.881+0000 7f1086316700 1 -- 172.21.15.114:0/2570387136 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 10} v 0) -- 0x7f108002df80 con 0x7f1080103480 2024-04-23T10:25:36.880 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:25:36.881+0000 7f107d7fa700 1 -- 172.21.15.114:0/2570387136 <== mon.0 v2:172.21.15.114:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 10}]=0 v0) ==== 75+0+13 (secure 0 0 0) 0x7f1074013070 con 0x7f1080103480 2024-04-23T10:25:36.880 INFO:teuthology.orchestra.run.smithi114.stdout:317827579932 2024-04-23T10:25:36.883 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:25:36.883+0000 7f1086316700 1 -- 172.21.15.114:0/2570387136 >> [v2:172.21.15.114:6800/465333808,v1:172.21.15.114:6801/465333808] conn(0x7f10680a3cd0 msgr2=0x7f10680a6190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T10:25:36.883 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:25:36.883+0000 7f1086316700 1 --2- 172.21.15.114:0/2570387136 >> [v2:172.21.15.114:6800/465333808,v1:172.21.15.114:6801/465333808] conn(0x7f10680a3cd0 0x7f10680a6190 secure :-1 s=READY pgs=196 cs=0 l=1 rev1=1 crypto rx=0x7f1070005fd0 tx=0x7f1070005db0 comp rx=0 tx=0).stop 2024-04-23T10:25:36.883 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:25:36.883+0000 7f1086316700 1 -- 172.21.15.114:0/2570387136 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f1080103480 msgr2=0x7f1080199760 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T10:25:36.883 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:25:36.883+0000 7f1086316700 1 --2- 172.21.15.114:0/2570387136 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f1080103480 0x7f1080199760 secure :-1 s=READY pgs=275 cs=0 l=1 rev1=1 crypto rx=0x7f1074000c00 tx=0x7f10740047b0 comp rx=0 tx=0).stop 2024-04-23T10:25:36.883 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:25:36.884+0000 7f1086316700 1 -- 172.21.15.114:0/2570387136 shutdown_connections 2024-04-23T10:25:36.883 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:25:36.884+0000 7f1086316700 1 --2- 172.21.15.114:0/2570387136 >> [v2:172.21.15.114:6800/465333808,v1:172.21.15.114:6801/465333808] conn(0x7f10680a3cd0 0x7f10680a6190 unknown :-1 s=CLOSED pgs=196 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:25:36.883 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:25:36.884+0000 7f1086316700 1 --2- 172.21.15.114:0/2570387136 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f1080102ac0 0x7f1080199220 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:25:36.883 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:25:36.884+0000 7f1086316700 1 --2- 172.21.15.114:0/2570387136 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f1080108a70 0x7f108019daf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:25:36.883 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:25:36.884+0000 7f1086316700 1 --2- 172.21.15.114:0/2570387136 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f1080103480 0x7f1080199760 unknown :-1 s=CLOSED pgs=275 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:25:36.883 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:25:36.884+0000 7f1086316700 1 -- 172.21.15.114:0/2570387136 >> 172.21.15.114:0/2570387136 conn(0x7f1080079bb0 msgr2=0x7f1080101920 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T10:25:36.883 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:25:36.884+0000 7f1086316700 1 -- 172.21.15.114:0/2570387136 shutdown_connections 2024-04-23T10:25:36.883 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:25:36.884+0000 7f1086316700 1 -- 172.21.15.114:0/2570387136 wait complete. 2024-04-23T10:25:37.174 INFO:journalctl@ceph.mon.a.smithi114.stdout:Apr 23 10:25:37 smithi114 ceph-mon[29255]: pgmap v312: 97 pgs: 97 active+clean; 641 KiB data, 370 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-23T10:25:37.175 INFO:journalctl@ceph.mon.a.smithi114.stdout:Apr 23 10:25:37 smithi114 ceph-mon[29255]: from='client.? 172.21.15.114:0/2570387136' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 10}]: dispatch 2024-04-23T10:25:37.217 INFO:teuthology.orchestra.run.smithi114.stderr:Inferring config /var/lib/ceph/aa240572-0159-11ef-bc93-c7b262605968/mon.a/config 2024-04-23T10:25:37.217 INFO:teuthology.orchestra.run.smithi114.stderr:Inferring config /var/lib/ceph/aa240572-0159-11ef-bc93-c7b262605968/mon.a/config 2024-04-23T10:25:37.492 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 23 10:25:37 smithi119 ceph-mon[32608]: pgmap v312: 97 pgs: 97 active+clean; 641 KiB data, 370 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-23T10:25:37.492 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 23 10:25:37 smithi119 ceph-mon[32608]: from='client.? 172.21.15.114:0/2570387136' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 10}]: dispatch 2024-04-23T10:25:37.578 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 23 10:25:37 smithi204 ceph-mon[32649]: pgmap v312: 97 pgs: 97 active+clean; 641 KiB data, 370 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-23T10:25:37.578 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 23 10:25:37 smithi204 ceph-mon[32649]: from='client.? 172.21.15.114:0/2570387136' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 10}]: dispatch 2024-04-23T10:25:38.062 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:25:38.061+0000 7f2c4e027700 1 -- 172.21.15.114:0/1522234924 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f2c48103720 msgr2=0x7f2c4810f990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T10:25:38.062 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:25:38.061+0000 7f2c4e027700 1 --2- 172.21.15.114:0/1522234924 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f2c48103720 0x7f2c4810f990 secure :-1 s=READY pgs=276 cs=0 l=1 rev1=1 crypto rx=0x7f2c4000b0a0 tx=0x7f2c4000b3b0 comp rx=0 tx=0).stop 2024-04-23T10:25:38.062 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:25:38.062+0000 7f2c4e027700 1 -- 172.21.15.114:0/1522234924 shutdown_connections 2024-04-23T10:25:38.062 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:25:38.062+0000 7f2c4e027700 1 --2- 172.21.15.114:0/1522234924 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f2c48103720 0x7f2c4810f990 unknown :-1 s=CLOSED pgs=276 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:25:38.062 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:25:38.062+0000 7f2c4e027700 1 --2- 172.21.15.114:0/1522234924 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f2c48108bc0 0x7f2c48108fa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:25:38.062 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:25:38.062+0000 7f2c4e027700 1 --2- 172.21.15.114:0/1522234924 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f2c48102d60 0x7f2c481031e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:25:38.062 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:25:38.062+0000 7f2c4e027700 1 -- 172.21.15.114:0/1522234924 >> 172.21.15.114:0/1522234924 conn(0x7f2c4807a9d0 msgr2=0x7f2c4807ade0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T10:25:38.063 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:25:38.062+0000 7f2c4e027700 1 -- 172.21.15.114:0/1522234924 shutdown_connections 2024-04-23T10:25:38.063 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:25:38.062+0000 7f2c4e027700 1 -- 172.21.15.114:0/1522234924 wait complete. 2024-04-23T10:25:38.063 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:25:38.062+0000 7f2c4e027700 1 Processor -- start 2024-04-23T10:25:38.063 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:25:38.062+0000 7f2c4e027700 1 -- start start 2024-04-23T10:25:38.063 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:25:38.063+0000 7f2c4e027700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f2c48102d60 0x7f2c48074d40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T10:25:38.063 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:25:38.063+0000 7f2c4e027700 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f2c48103720 0x7f2c48075280 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T10:25:38.063 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:25:38.063+0000 7f2c4e027700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f2c48108bc0 0x7f2c480757c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T10:25:38.063 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:25:38.063+0000 7f2c4e027700 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x7f2c48112cc0 con 0x7f2c48108bc0 2024-04-23T10:25:38.063 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:25:38.063+0000 7f2c4e027700 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 -- 0x7f2c48112b40 con 0x7f2c48102d60 2024-04-23T10:25:38.063 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:25:38.063+0000 7f2c4e027700 1 -- --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_getmap magic: 0 -- 0x7f2c48112e40 con 0x7f2c48103720 2024-04-23T10:25:38.063 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:25:38.063+0000 7f2c47fff700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f2c48108bc0 0x7f2c480757c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T10:25:38.063 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:25:38.063+0000 7f2c47fff700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f2c48108bc0 0x7f2c480757c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.114:50802/0 (socket says 172.21.15.114:50802) 2024-04-23T10:25:38.063 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:25:38.063+0000 7f2c47fff700 1 -- 172.21.15.114:0/1919621417 learned_addr learned my addr 172.21.15.114:0/1919621417 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-04-23T10:25:38.063 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:25:38.063+0000 7f2c46ffd700 1 --2- 172.21.15.114:0/1919621417 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f2c48103720 0x7f2c48075280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T10:25:38.064 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:25:38.063+0000 7f2c47fff700 1 -- 172.21.15.114:0/1919621417 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f2c48103720 msgr2=0x7f2c48075280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T10:25:38.064 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:25:38.063+0000 7f2c47fff700 1 --2- 172.21.15.114:0/1919621417 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f2c48103720 0x7f2c48075280 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:25:38.064 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:25:38.063+0000 7f2c47fff700 1 -- 172.21.15.114:0/1919621417 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f2c48102d60 msgr2=0x7f2c48074d40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T10:25:38.064 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:25:38.063+0000 7f2c47fff700 1 --2- 172.21.15.114:0/1919621417 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f2c48102d60 0x7f2c48074d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:25:38.064 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:25:38.063+0000 7f2c47fff700 1 -- 172.21.15.114:0/1919621417 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f2c481a7570 con 0x7f2c48108bc0 2024-04-23T10:25:38.064 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:25:38.063+0000 7f2c47fff700 1 --2- 172.21.15.114:0/1919621417 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f2c48108bc0 0x7f2c480757c0 secure :-1 s=READY pgs=277 cs=0 l=1 rev1=1 crypto rx=0x7f2c4000b830 tx=0x7f2c40004ce0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T10:25:38.065 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:25:38.063+0000 7f2c44ff9700 1 -- 172.21.15.114:0/1919621417 <== mon.0 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2c40021070 con 0x7f2c48108bc0 2024-04-23T10:25:38.065 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:25:38.064+0000 7f2c4e027700 1 -- 172.21.15.114:0/1919621417 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f2c481a7800 con 0x7f2c48108bc0 2024-04-23T10:25:38.065 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:25:38.064+0000 7f2c4e027700 1 -- 172.21.15.114:0/1919621417 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f2c481a7d00 con 0x7f2c48108bc0 2024-04-23T10:25:38.065 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:25:38.064+0000 7f2c44ff9700 1 -- 172.21.15.114:0/1919621417 <== mon.0 v2:172.21.15.114:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f2c40007e30 con 0x7f2c48108bc0 2024-04-23T10:25:38.065 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:25:38.064+0000 7f2c44ff9700 1 -- 172.21.15.114:0/1919621417 <== mon.0 v2:172.21.15.114:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2c40004730 con 0x7f2c48108bc0 2024-04-23T10:25:38.065 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:25:38.065+0000 7f2c4e027700 1 -- 172.21.15.114:0/1919621417 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f2c34005330 con 0x7f2c48108bc0 2024-04-23T10:25:38.065 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:25:38.066+0000 7f2c44ff9700 1 -- 172.21.15.114:0/1919621417 <== mon.0 v2:172.21.15.114:3300/0 4 ==== mgrmap(e 17) ==== 138698+0+0 (secure 0 0 0) 0x7f2c40003c00 con 0x7f2c48108bc0 2024-04-23T10:25:38.066 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:25:38.066+0000 7f2c44ff9700 1 --2- 172.21.15.114:0/1919621417 >> [v2:172.21.15.114:6800/465333808,v1:172.21.15.114:6801/465333808] conn(0x7f2c300a3d50 0x7f2c300a6210 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T10:25:38.066 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:25:38.067+0000 7f2c477fe700 1 --2- 172.21.15.114:0/1919621417 >> [v2:172.21.15.114:6800/465333808,v1:172.21.15.114:6801/465333808] conn(0x7f2c300a3d50 0x7f2c300a6210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T10:25:38.066 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:25:38.067+0000 7f2c44ff9700 1 -- 172.21.15.114:0/1919621417 <== mon.0 v2:172.21.15.114:3300/0 5 ==== osd_map(87..87 src has 1..87) ==== 7979+0+0 (secure 0 0 0) 0x7f2c4000f020 con 0x7f2c48108bc0 2024-04-23T10:25:38.066 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:25:38.067+0000 7f2c477fe700 1 --2- 172.21.15.114:0/1919621417 >> [v2:172.21.15.114:6800/465333808,v1:172.21.15.114:6801/465333808] conn(0x7f2c300a3d50 0x7f2c300a6210 secure :-1 s=READY pgs=197 cs=0 l=1 rev1=1 crypto rx=0x7f2c3c0059d0 tx=0x7f2c3c005cb0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T10:25:38.069 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:25:38.070+0000 7f2c44ff9700 1 -- 172.21.15.114:0/1919621417 <== mon.0 v2:172.21.15.114:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f2c4002a020 con 0x7f2c48108bc0 2024-04-23T10:25:38.278 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:25:38.278+0000 7f2c4e027700 1 -- 172.21.15.114:0/1919621417 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 6} v 0) -- 0x7f2c340058f0 con 0x7f2c48108bc0 2024-04-23T10:25:38.278 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:25:38.279+0000 7f2c44ff9700 1 -- 172.21.15.114:0/1919621417 <== mon.0 v2:172.21.15.114:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 6}]=0 v0) ==== 74+0+13 (secure 0 0 0) 0x7f2c4009e590 con 0x7f2c48108bc0 2024-04-23T10:25:38.278 INFO:teuthology.orchestra.run.smithi114.stdout:201863462962 2024-04-23T10:25:38.280 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:25:38.281+0000 7f2c4e027700 1 -- 172.21.15.114:0/1919621417 >> [v2:172.21.15.114:6800/465333808,v1:172.21.15.114:6801/465333808] conn(0x7f2c300a3d50 msgr2=0x7f2c300a6210 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T10:25:38.281 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:25:38.281+0000 7f2c4e027700 1 --2- 172.21.15.114:0/1919621417 >> [v2:172.21.15.114:6800/465333808,v1:172.21.15.114:6801/465333808] conn(0x7f2c300a3d50 0x7f2c300a6210 secure :-1 s=READY pgs=197 cs=0 l=1 rev1=1 crypto rx=0x7f2c3c0059d0 tx=0x7f2c3c005cb0 comp rx=0 tx=0).stop 2024-04-23T10:25:38.281 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:25:38.281+0000 7f2c4e027700 1 -- 172.21.15.114:0/1919621417 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f2c48108bc0 msgr2=0x7f2c480757c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T10:25:38.281 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:25:38.281+0000 7f2c4e027700 1 --2- 172.21.15.114:0/1919621417 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f2c48108bc0 0x7f2c480757c0 secure :-1 s=READY pgs=277 cs=0 l=1 rev1=1 crypto rx=0x7f2c4000b830 tx=0x7f2c40004ce0 comp rx=0 tx=0).stop 2024-04-23T10:25:38.281 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:25:38.281+0000 7f2c4e027700 1 -- 172.21.15.114:0/1919621417 shutdown_connections 2024-04-23T10:25:38.281 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:25:38.281+0000 7f2c4e027700 1 --2- 172.21.15.114:0/1919621417 >> [v2:172.21.15.114:6800/465333808,v1:172.21.15.114:6801/465333808] conn(0x7f2c300a3d50 0x7f2c300a6210 unknown :-1 s=CLOSED pgs=197 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:25:38.281 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:25:38.281+0000 7f2c4e027700 1 --2- 172.21.15.114:0/1919621417 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f2c48102d60 0x7f2c48074d40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:25:38.281 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:25:38.281+0000 7f2c4e027700 1 --2- 172.21.15.114:0/1919621417 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f2c48108bc0 0x7f2c480757c0 unknown :-1 s=CLOSED pgs=277 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:25:38.281 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:25:38.281+0000 7f2c4e027700 1 --2- 172.21.15.114:0/1919621417 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f2c48103720 0x7f2c48075280 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:25:38.281 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:25:38.281+0000 7f2c4e027700 1 -- 172.21.15.114:0/1919621417 >> 172.21.15.114:0/1919621417 conn(0x7f2c4807a9d0 msgr2=0x7f2c481017b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T10:25:38.281 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:25:38.281+0000 7f2c4e027700 1 -- 172.21.15.114:0/1919621417 shutdown_connections 2024-04-23T10:25:38.281 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:25:38.282+0000 7f2c4e027700 1 -- 172.21.15.114:0/1919621417 wait complete. 2024-04-23T10:25:39.254 INFO:journalctl@ceph.mon.a.smithi114.stdout:Apr 23 10:25:39 smithi114 ceph-mon[29255]: pgmap v313: 97 pgs: 97 active+clean; 641 KiB data, 370 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-04-23T10:25:39.255 INFO:journalctl@ceph.mon.a.smithi114.stdout:Apr 23 10:25:39 smithi114 ceph-mon[29255]: from='client.? 172.21.15.114:0/1919621417' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 6}]: dispatch 2024-04-23T10:25:39.492 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 23 10:25:39 smithi119 ceph-mon[32608]: pgmap v313: 97 pgs: 97 active+clean; 641 KiB data, 370 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-04-23T10:25:39.492 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 23 10:25:39 smithi119 ceph-mon[32608]: from='client.? 172.21.15.114:0/1919621417' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 6}]: dispatch 2024-04-23T10:25:39.578 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 23 10:25:39 smithi204 ceph-mon[32649]: pgmap v313: 97 pgs: 97 active+clean; 641 KiB data, 370 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-04-23T10:25:39.578 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 23 10:25:39 smithi204 ceph-mon[32649]: from='client.? 172.21.15.114:0/1919621417' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 6}]: dispatch 2024-04-23T10:25:39.877 INFO:teuthology.orchestra.run.smithi114.stderr:Inferring config /var/lib/ceph/aa240572-0159-11ef-bc93-c7b262605968/mon.a/config 2024-04-23T10:25:39.887 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:25:39.887+0000 7f668f09c700 1 -- 172.21.15.114:0/3546499251 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f66881019d0 msgr2=0x7f668810f910 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T10:25:39.887 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:25:39.887+0000 7f668f09c700 1 --2- 172.21.15.114:0/3546499251 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f66881019d0 0x7f668810f910 secure :-1 s=READY pgs=278 cs=0 l=1 rev1=1 crypto rx=0x7f668000b0a0 tx=0x7f668000b3b0 comp rx=0 tx=0).stop 2024-04-23T10:25:39.887 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:25:39.887+0000 7f668f09c700 1 -- 172.21.15.114:0/3546499251 shutdown_connections 2024-04-23T10:25:39.887 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:25:39.887+0000 7f668f09c700 1 --2- 172.21.15.114:0/3546499251 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f66881019d0 0x7f668810f910 unknown :-1 s=CLOSED pgs=278 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:25:39.887 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:25:39.887+0000 7f668f09c700 1 --2- 172.21.15.114:0/3546499251 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f6688108ad0 0x7f6688108eb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:25:39.887 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:25:39.887+0000 7f668f09c700 1 --2- 172.21.15.114:0/3546499251 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f6688101030 0x7f6688101490 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:25:39.887 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:25:39.887+0000 7f668f09c700 1 -- 172.21.15.114:0/3546499251 >> 172.21.15.114:0/3546499251 conn(0x7f6688077fb0 msgr2=0x7f66880ff360 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T10:25:39.887 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:25:39.888+0000 7f668f09c700 1 -- 172.21.15.114:0/3546499251 shutdown_connections 2024-04-23T10:25:39.887 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:25:39.888+0000 7f668f09c700 1 -- 172.21.15.114:0/3546499251 wait complete. 2024-04-23T10:25:39.889 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:25:39.888+0000 7f668f09c700 1 Processor -- start 2024-04-23T10:25:39.889 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:25:39.888+0000 7f668f09c700 1 -- start start 2024-04-23T10:25:39.889 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:25:39.889+0000 7f668f09c700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f6688101030 0x7f66881990b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T10:25:39.889 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:25:39.889+0000 7f668f09c700 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f66881019d0 0x7f66881995f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T10:25:39.889 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:25:39.889+0000 7f668f09c700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f6688108ad0 0x7f668819d980 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T10:25:39.890 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:25:39.889+0000 7f668f09c700 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x7f6688112c20 con 0x7f6688108ad0 2024-04-23T10:25:39.890 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:25:39.889+0000 7f668f09c700 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 -- 0x7f6688112aa0 con 0x7f6688101030 2024-04-23T10:25:39.890 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:25:39.889+0000 7f668f09c700 1 -- --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_getmap magic: 0 -- 0x7f6688112da0 con 0x7f66881019d0 2024-04-23T10:25:39.890 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:25:39.889+0000 7f668d639700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f6688108ad0 0x7f668819d980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T10:25:39.890 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:25:39.889+0000 7f668ce38700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f6688101030 0x7f66881990b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T10:25:39.890 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:25:39.889+0000 7f668d639700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f6688108ad0 0x7f668819d980 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.114:50824/0 (socket says 172.21.15.114:50824) 2024-04-23T10:25:39.890 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:25:39.889+0000 7f668ce38700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f6688101030 0x7f66881990b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.119:3300/0 says I am v2:172.21.15.114:58826/0 (socket says 172.21.15.114:58826) 2024-04-23T10:25:39.890 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:25:39.889+0000 7f668d639700 1 -- 172.21.15.114:0/1894551276 learned_addr learned my addr 172.21.15.114:0/1894551276 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-04-23T10:25:39.890 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:25:39.889+0000 7f6687fff700 1 --2- 172.21.15.114:0/1894551276 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f66881019d0 0x7f66881995f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T10:25:39.890 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:25:39.889+0000 7f668d639700 1 -- 172.21.15.114:0/1894551276 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f66881019d0 msgr2=0x7f66881995f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T10:25:39.890 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:25:39.889+0000 7f668d639700 1 --2- 172.21.15.114:0/1894551276 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f66881019d0 0x7f66881995f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:25:39.890 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:25:39.889+0000 7f668d639700 1 -- 172.21.15.114:0/1894551276 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f6688101030 msgr2=0x7f66881990b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T10:25:39.890 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:25:39.889+0000 7f668d639700 1 --2- 172.21.15.114:0/1894551276 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f6688101030 0x7f66881990b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:25:39.891 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:25:39.889+0000 7f668d639700 1 -- 172.21.15.114:0/1894551276 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f668819e0c0 con 0x7f6688108ad0 2024-04-23T10:25:39.891 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:25:39.890+0000 7f668d639700 1 --2- 172.21.15.114:0/1894551276 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f6688108ad0 0x7f668819d980 secure :-1 s=READY pgs=279 cs=0 l=1 rev1=1 crypto rx=0x7f66800062a0 tx=0x7f6680004c80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T10:25:39.891 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:25:39.890+0000 7f6685ffb700 1 -- 172.21.15.114:0/1894551276 <== mon.0 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6680021070 con 0x7f6688108ad0 2024-04-23T10:25:39.891 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:25:39.890+0000 7f6685ffb700 1 -- 172.21.15.114:0/1894551276 <== mon.0 v2:172.21.15.114:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f6680007e30 con 0x7f6688108ad0 2024-04-23T10:25:39.891 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:25:39.890+0000 7f668f09c700 1 -- 172.21.15.114:0/1894551276 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f668819e350 con 0x7f6688108ad0 2024-04-23T10:25:39.891 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:25:39.890+0000 7f6685ffb700 1 -- 172.21.15.114:0/1894551276 <== mon.0 v2:172.21.15.114:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6680004780 con 0x7f6688108ad0 2024-04-23T10:25:39.891 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:25:39.890+0000 7f668f09c700 1 -- 172.21.15.114:0/1894551276 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f66881a5c30 con 0x7f6688108ad0 2024-04-23T10:25:39.891 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:25:39.891+0000 7f668f09c700 1 -- 172.21.15.114:0/1894551276 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f6688104350 con 0x7f6688108ad0 2024-04-23T10:25:39.891 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:25:39.892+0000 7f6685ffb700 1 -- 172.21.15.114:0/1894551276 <== mon.0 v2:172.21.15.114:3300/0 4 ==== mgrmap(e 17) ==== 138698+0+0 (secure 0 0 0) 0x7f6680007980 con 0x7f6688108ad0 2024-04-23T10:25:39.892 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:25:39.893+0000 7f6685ffb700 1 --2- 172.21.15.114:0/1894551276 >> [v2:172.21.15.114:6800/465333808,v1:172.21.15.114:6801/465333808] conn(0x7f66780a3cd0 0x7f66780a6190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T10:25:39.892 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:25:39.893+0000 7f6685ffb700 1 -- 172.21.15.114:0/1894551276 <== mon.0 v2:172.21.15.114:3300/0 5 ==== osd_map(87..87 src has 1..87) ==== 7979+0+0 (secure 0 0 0) 0x7f66800d1340 con 0x7f6688108ad0 2024-04-23T10:25:39.892 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:25:39.893+0000 7f668ce38700 1 --2- 172.21.15.114:0/1894551276 >> [v2:172.21.15.114:6800/465333808,v1:172.21.15.114:6801/465333808] conn(0x7f66780a3cd0 0x7f66780a6190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T10:25:39.892 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:25:39.894+0000 7f668ce38700 1 --2- 172.21.15.114:0/1894551276 >> [v2:172.21.15.114:6800/465333808,v1:172.21.15.114:6801/465333808] conn(0x7f66780a3cd0 0x7f66780a6190 secure :-1 s=READY pgs=198 cs=0 l=1 rev1=1 crypto rx=0x7f667c009be0 tx=0x7f667c009260 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T10:25:39.897 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:25:39.898+0000 7f6685ffb700 1 -- 172.21.15.114:0/1894551276 <== mon.0 v2:172.21.15.114:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f6680004000 con 0x7f6688108ad0 2024-04-23T10:25:40.106 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:25:40.107+0000 7f668f09c700 1 -- 172.21.15.114:0/1894551276 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 8} v 0) -- 0x7f668802df80 con 0x7f6688108ad0 2024-04-23T10:25:40.107 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:25:40.107+0000 7f6685ffb700 1 -- 172.21.15.114:0/1894551276 <== mon.0 v2:172.21.15.114:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 8}]=0 v0) ==== 74+0+13 (secure 0 0 0) 0x7f6680003c00 con 0x7f6688108ad0 2024-04-23T10:25:40.107 INFO:teuthology.orchestra.run.smithi114.stdout:266287972393 2024-04-23T10:25:40.109 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:25:40.110+0000 7f668f09c700 1 -- 172.21.15.114:0/1894551276 >> [v2:172.21.15.114:6800/465333808,v1:172.21.15.114:6801/465333808] conn(0x7f66780a3cd0 msgr2=0x7f66780a6190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T10:25:40.110 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:25:40.110+0000 7f668f09c700 1 --2- 172.21.15.114:0/1894551276 >> [v2:172.21.15.114:6800/465333808,v1:172.21.15.114:6801/465333808] conn(0x7f66780a3cd0 0x7f66780a6190 secure :-1 s=READY pgs=198 cs=0 l=1 rev1=1 crypto rx=0x7f667c009be0 tx=0x7f667c009260 comp rx=0 tx=0).stop 2024-04-23T10:25:40.110 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:25:40.110+0000 7f668f09c700 1 -- 172.21.15.114:0/1894551276 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f6688108ad0 msgr2=0x7f668819d980 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T10:25:40.110 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:25:40.110+0000 7f668f09c700 1 --2- 172.21.15.114:0/1894551276 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f6688108ad0 0x7f668819d980 secure :-1 s=READY pgs=279 cs=0 l=1 rev1=1 crypto rx=0x7f66800062a0 tx=0x7f6680004c80 comp rx=0 tx=0).stop 2024-04-23T10:25:40.110 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:25:40.110+0000 7f668f09c700 1 -- 172.21.15.114:0/1894551276 shutdown_connections 2024-04-23T10:25:40.110 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:25:40.110+0000 7f668f09c700 1 --2- 172.21.15.114:0/1894551276 >> [v2:172.21.15.114:6800/465333808,v1:172.21.15.114:6801/465333808] conn(0x7f66780a3cd0 0x7f66780a6190 unknown :-1 s=CLOSED pgs=198 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:25:40.110 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:25:40.110+0000 7f668f09c700 1 --2- 172.21.15.114:0/1894551276 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f6688101030 0x7f66881990b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:25:40.110 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:25:40.110+0000 7f668f09c700 1 --2- 172.21.15.114:0/1894551276 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f6688108ad0 0x7f668819d980 unknown :-1 s=CLOSED pgs=279 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:25:40.110 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:25:40.110+0000 7f668f09c700 1 --2- 172.21.15.114:0/1894551276 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f66881019d0 0x7f66881995f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:25:40.110 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:25:40.110+0000 7f668f09c700 1 -- 172.21.15.114:0/1894551276 >> 172.21.15.114:0/1894551276 conn(0x7f6688077fb0 msgr2=0x7f66880ff860 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T10:25:40.110 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:25:40.110+0000 7f668f09c700 1 -- 172.21.15.114:0/1894551276 shutdown_connections 2024-04-23T10:25:40.111 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:25:40.110+0000 7f668f09c700 1 -- 172.21.15.114:0/1894551276 wait complete. 2024-04-23T10:25:40.492 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 23 10:25:40 smithi119 ceph-mon[32608]: pgmap v314: 97 pgs: 97 active+clean; 641 KiB data, 370 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-04-23T10:25:40.492 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 23 10:25:40 smithi119 ceph-mon[32608]: from='client.? 172.21.15.114:0/1894551276' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 8}]: dispatch 2024-04-23T10:25:40.504 INFO:journalctl@ceph.mon.a.smithi114.stdout:Apr 23 10:25:40 smithi114 ceph-mon[29255]: pgmap v314: 97 pgs: 97 active+clean; 641 KiB data, 370 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-04-23T10:25:40.504 INFO:journalctl@ceph.mon.a.smithi114.stdout:Apr 23 10:25:40 smithi114 ceph-mon[29255]: from='client.? 172.21.15.114:0/1894551276' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 8}]: dispatch 2024-04-23T10:25:40.578 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 23 10:25:40 smithi204 ceph-mon[32649]: pgmap v314: 97 pgs: 97 active+clean; 641 KiB data, 370 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-04-23T10:25:40.578 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 23 10:25:40 smithi204 ceph-mon[32649]: from='client.? 172.21.15.114:0/1894551276' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 8}]: dispatch 2024-04-23T10:25:41.339 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:25:41.338+0000 7f1aa3807700 1 -- 172.21.15.114:0/3585235113 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f1a9c1022f0 msgr2=0x7f1a9c10e7e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T10:25:41.340 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:25:41.338+0000 7f1aa3807700 1 --2- 172.21.15.114:0/3585235113 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f1a9c1022f0 0x7f1a9c10e7e0 secure :-1 s=READY pgs=282 cs=0 l=1 rev1=1 crypto rx=0x7f1a9400b0a0 tx=0x7f1a9400b3b0 comp rx=0 tx=0).stop 2024-04-23T10:25:41.340 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:25:41.338+0000 7f1aa3807700 1 -- 172.21.15.114:0/3585235113 shutdown_connections 2024-04-23T10:25:41.340 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:25:41.338+0000 7f1aa3807700 1 --2- 172.21.15.114:0/3585235113 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f1a9c107980 0x7f1a9c107d60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:25:41.340 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:25:41.338+0000 7f1aa3807700 1 --2- 172.21.15.114:0/3585235113 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f1a9c1022f0 0x7f1a9c10e7e0 unknown :-1 s=CLOSED pgs=282 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:25:41.340 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:25:41.338+0000 7f1aa3807700 1 --2- 172.21.15.114:0/3585235113 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f1a9c101930 0x7f1a9c101db0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:25:41.340 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:25:41.338+0000 7f1aa3807700 1 -- 172.21.15.114:0/3585235113 >> 172.21.15.114:0/3585235113 conn(0x7f1a9c0788b0 msgr2=0x7f1a9c078cc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T10:25:41.340 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:25:41.338+0000 7f1aa3807700 1 -- 172.21.15.114:0/3585235113 shutdown_connections 2024-04-23T10:25:41.340 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:25:41.338+0000 7f1aa3807700 1 -- 172.21.15.114:0/3585235113 wait complete. 2024-04-23T10:25:41.341 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:25:41.339+0000 7f1aa3807700 1 Processor -- start 2024-04-23T10:25:41.341 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:25:41.339+0000 7f1aa3807700 1 -- start start 2024-04-23T10:25:41.341 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:25:41.339+0000 7f1aa3807700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f1a9c101930 0x7f1a9c199f70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T10:25:41.341 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:25:41.339+0000 7f1aa3807700 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f1a9c1022f0 0x7f1a9c19a4b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T10:25:41.341 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:25:41.339+0000 7f1aa3807700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f1a9c107980 0x7f1a9c1a4ff0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T10:25:41.341 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:25:41.339+0000 7f1aa3807700 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x7f1a9c111ad0 con 0x7f1a9c101930 2024-04-23T10:25:41.341 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:25:41.339+0000 7f1aa3807700 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 -- 0x7f1a9c111950 con 0x7f1a9c107980 2024-04-23T10:25:41.341 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:25:41.339+0000 7f1aa3807700 1 -- --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_getmap magic: 0 -- 0x7f1a9c111c50 con 0x7f1a9c1022f0 2024-04-23T10:25:41.341 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:25:41.339+0000 7f1aa15a3700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f1a9c101930 0x7f1a9c199f70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T10:25:41.341 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:25:41.339+0000 7f1aa15a3700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f1a9c101930 0x7f1a9c199f70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.114:50866/0 (socket says 172.21.15.114:50866) 2024-04-23T10:25:41.342 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:25:41.339+0000 7f1aa1da4700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f1a9c107980 0x7f1a9c1a4ff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T10:25:41.342 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:25:41.339+0000 7f1aa15a3700 1 -- 172.21.15.114:0/2940402328 learned_addr learned my addr 172.21.15.114:0/2940402328 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-04-23T10:25:41.342 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:25:41.340+0000 7f1aa0da2700 1 --2- 172.21.15.114:0/2940402328 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f1a9c1022f0 0x7f1a9c19a4b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T10:25:41.342 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:25:41.340+0000 7f1aa15a3700 1 -- 172.21.15.114:0/2940402328 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f1a9c1022f0 msgr2=0x7f1a9c19a4b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T10:25:41.342 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:25:41.340+0000 7f1aa15a3700 1 --2- 172.21.15.114:0/2940402328 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f1a9c1022f0 0x7f1a9c19a4b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:25:41.342 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:25:41.340+0000 7f1aa15a3700 1 -- 172.21.15.114:0/2940402328 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f1a9c107980 msgr2=0x7f1a9c1a4ff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T10:25:41.342 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:25:41.340+0000 7f1aa15a3700 1 --2- 172.21.15.114:0/2940402328 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f1a9c107980 0x7f1a9c1a4ff0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:25:41.342 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:25:41.340+0000 7f1aa15a3700 1 -- 172.21.15.114:0/2940402328 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f1a9c1a5530 con 0x7f1a9c101930 2024-04-23T10:25:41.342 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:25:41.340+0000 7f1aa1da4700 1 --2- 172.21.15.114:0/2940402328 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f1a9c107980 0x7f1a9c1a4ff0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-23T10:25:41.342 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:25:41.340+0000 7f1aa15a3700 1 --2- 172.21.15.114:0/2940402328 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f1a9c101930 0x7f1a9c199f70 secure :-1 s=READY pgs=283 cs=0 l=1 rev1=1 crypto rx=0x7f1a9800d980 tx=0x7f1a9800dc90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T10:25:41.342 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:25:41.340+0000 7f1a927fc700 1 -- 172.21.15.114:0/2940402328 <== mon.0 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1a98016070 con 0x7f1a9c101930 2024-04-23T10:25:41.343 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:25:41.340+0000 7f1aa3807700 1 -- 172.21.15.114:0/2940402328 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f1a9c197d60 con 0x7f1a9c101930 2024-04-23T10:25:41.343 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:25:41.340+0000 7f1a927fc700 1 -- 172.21.15.114:0/2940402328 <== mon.0 v2:172.21.15.114:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f1a98004830 con 0x7f1a9c101930 2024-04-23T10:25:41.343 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:25:41.340+0000 7f1a927fc700 1 -- 172.21.15.114:0/2940402328 <== mon.0 v2:172.21.15.114:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1a98005540 con 0x7f1a9c101930 2024-04-23T10:25:41.343 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:25:41.340+0000 7f1aa3807700 1 -- 172.21.15.114:0/2940402328 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f1a9c1982a0 con 0x7f1a9c101930 2024-04-23T10:25:41.344 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:25:41.341+0000 7f1aa3807700 1 -- 172.21.15.114:0/2940402328 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f1a9c069730 con 0x7f1a9c101930 2024-04-23T10:25:41.344 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:25:41.343+0000 7f1a927fc700 1 -- 172.21.15.114:0/2940402328 <== mon.0 v2:172.21.15.114:3300/0 4 ==== mgrmap(e 17) ==== 138698+0+0 (secure 0 0 0) 0x7f1a98004220 con 0x7f1a9c101930 2024-04-23T10:25:41.344 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:25:41.343+0000 7f1a927fc700 1 --2- 172.21.15.114:0/2940402328 >> [v2:172.21.15.114:6800/465333808,v1:172.21.15.114:6801/465333808] conn(0x7f1a880a3cd0 0x7f1a880a6190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T10:25:41.344 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:25:41.343+0000 7f1aa0da2700 1 --2- 172.21.15.114:0/2940402328 >> [v2:172.21.15.114:6800/465333808,v1:172.21.15.114:6801/465333808] conn(0x7f1a880a3cd0 0x7f1a880a6190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T10:25:41.344 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:25:41.343+0000 7f1a927fc700 1 -- 172.21.15.114:0/2940402328 <== mon.0 v2:172.21.15.114:3300/0 5 ==== osd_map(87..87 src has 1..87) ==== 7979+0+0 (secure 0 0 0) 0x7f1a980d05f0 con 0x7f1a9c101930 2024-04-23T10:25:41.344 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:25:41.344+0000 7f1aa0da2700 1 --2- 172.21.15.114:0/2940402328 >> [v2:172.21.15.114:6800/465333808,v1:172.21.15.114:6801/465333808] conn(0x7f1a880a3cd0 0x7f1a880a6190 secure :-1 s=READY pgs=200 cs=0 l=1 rev1=1 crypto rx=0x7f1a9c078eb0 tx=0x7f1a8c00a380 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T10:25:41.350 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:25:41.351+0000 7f1a927fc700 1 -- 172.21.15.114:0/2940402328 <== mon.0 v2:172.21.15.114:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f1a9801f020 con 0x7f1a9c101930 2024-04-23T10:25:41.561 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:25:41.561+0000 7f1aa3807700 1 -- 172.21.15.114:0/2940402328 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 0} v 0) -- 0x7f1a9c02df80 con 0x7f1a9c101930 2024-04-23T10:25:41.561 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:25:41.562+0000 7f1a927fc700 1 -- 172.21.15.114:0/2940402328 <== mon.0 v2:172.21.15.114:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 0}]=0 v0) ==== 74+0+12 (secure 0 0 0) 0x7f1a9809d020 con 0x7f1a9c101930 2024-04-23T10:25:41.561 INFO:teuthology.orchestra.run.smithi114.stdout:34359738448 2024-04-23T10:25:41.563 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:25:41.564+0000 7f1aa3807700 1 -- 172.21.15.114:0/2940402328 >> [v2:172.21.15.114:6800/465333808,v1:172.21.15.114:6801/465333808] conn(0x7f1a880a3cd0 msgr2=0x7f1a880a6190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T10:25:41.564 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:25:41.564+0000 7f1aa3807700 1 --2- 172.21.15.114:0/2940402328 >> [v2:172.21.15.114:6800/465333808,v1:172.21.15.114:6801/465333808] conn(0x7f1a880a3cd0 0x7f1a880a6190 secure :-1 s=READY pgs=200 cs=0 l=1 rev1=1 crypto rx=0x7f1a9c078eb0 tx=0x7f1a8c00a380 comp rx=0 tx=0).stop 2024-04-23T10:25:41.564 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:25:41.564+0000 7f1aa3807700 1 -- 172.21.15.114:0/2940402328 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f1a9c101930 msgr2=0x7f1a9c199f70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T10:25:41.564 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:25:41.564+0000 7f1aa3807700 1 --2- 172.21.15.114:0/2940402328 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f1a9c101930 0x7f1a9c199f70 secure :-1 s=READY pgs=283 cs=0 l=1 rev1=1 crypto rx=0x7f1a9800d980 tx=0x7f1a9800dc90 comp rx=0 tx=0).stop 2024-04-23T10:25:41.564 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:25:41.564+0000 7f1aa3807700 1 -- 172.21.15.114:0/2940402328 shutdown_connections 2024-04-23T10:25:41.564 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:25:41.564+0000 7f1aa3807700 1 --2- 172.21.15.114:0/2940402328 >> [v2:172.21.15.114:6800/465333808,v1:172.21.15.114:6801/465333808] conn(0x7f1a880a3cd0 0x7f1a880a6190 unknown :-1 s=CLOSED pgs=200 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:25:41.564 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:25:41.564+0000 7f1aa3807700 1 --2- 172.21.15.114:0/2940402328 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f1a9c107980 0x7f1a9c1a4ff0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:25:41.564 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:25:41.564+0000 7f1aa3807700 1 --2- 172.21.15.114:0/2940402328 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f1a9c101930 0x7f1a9c199f70 unknown :-1 s=CLOSED pgs=283 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:25:41.564 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:25:41.564+0000 7f1aa3807700 1 --2- 172.21.15.114:0/2940402328 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f1a9c1022f0 0x7f1a9c19a4b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:25:41.564 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:25:41.564+0000 7f1aa3807700 1 -- 172.21.15.114:0/2940402328 >> 172.21.15.114:0/2940402328 conn(0x7f1a9c0788b0 msgr2=0x7f1a9c0ffa20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T10:25:41.564 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:25:41.564+0000 7f1aa3807700 1 -- 172.21.15.114:0/2940402328 shutdown_connections 2024-04-23T10:25:41.564 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:25:41.565+0000 7f1aa3807700 1 -- 172.21.15.114:0/2940402328 wait complete. 2024-04-23T10:25:41.703 INFO:journalctl@ceph.mon.a.smithi114.stdout:Apr 23 10:25:41 smithi114 ceph-mon[29255]: from='client.? 172.21.15.114:0/1030411522' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 9}]: dispatch 2024-04-23T10:25:41.828 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 23 10:25:41 smithi204 ceph-mon[32649]: from='client.? 172.21.15.114:0/1030411522' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 9}]: dispatch 2024-04-23T10:25:41.992 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 23 10:25:41 smithi119 ceph-mon[32608]: from='client.? 172.21.15.114:0/1030411522' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 9}]: dispatch 2024-04-23T10:25:42.669 INFO:journalctl@ceph.mon.a.smithi114.stdout:Apr 23 10:25:42 smithi114 ceph-mon[29255]: from='client.? 172.21.15.114:0/2940402328' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 0}]: dispatch 2024-04-23T10:25:42.669 INFO:journalctl@ceph.mon.a.smithi114.stdout:Apr 23 10:25:42 smithi114 ceph-mon[29255]: pgmap v315: 97 pgs: 97 active+clean; 641 KiB data, 370 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-23T10:25:42.769 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:25:42.770+0000 7f60348a3700 1 -- 172.21.15.114:0/374624238 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f6030108ad0 msgr2=0x7f6030108eb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T10:25:42.769 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:25:42.770+0000 7f60348a3700 1 --2- 172.21.15.114:0/374624238 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f6030108ad0 0x7f6030108eb0 secure :-1 s=READY pgs=284 cs=0 l=1 rev1=1 crypto rx=0x7f602800b0a0 tx=0x7f602800b3b0 comp rx=0 tx=0).stop 2024-04-23T10:25:42.769 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:25:42.770+0000 7f60348a3700 1 -- 172.21.15.114:0/374624238 shutdown_connections 2024-04-23T10:25:42.769 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:25:42.770+0000 7f60348a3700 1 --2- 172.21.15.114:0/374624238 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f60301019d0 0x7f603010f910 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:25:42.770 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:25:42.770+0000 7f60348a3700 1 --2- 172.21.15.114:0/374624238 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f6030108ad0 0x7f6030108eb0 unknown :-1 s=CLOSED pgs=284 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:25:42.770 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:25:42.770+0000 7f60348a3700 1 --2- 172.21.15.114:0/374624238 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f6030101030 0x7f6030101490 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:25:42.770 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:25:42.770+0000 7f60348a3700 1 -- 172.21.15.114:0/374624238 >> 172.21.15.114:0/374624238 conn(0x7f6030077fb0 msgr2=0x7f60300ff360 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T10:25:42.770 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:25:42.770+0000 7f60348a3700 1 -- 172.21.15.114:0/374624238 shutdown_connections 2024-04-23T10:25:42.770 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:25:42.770+0000 7f60348a3700 1 -- 172.21.15.114:0/374624238 wait complete. 2024-04-23T10:25:42.772 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:25:42.771+0000 7f60348a3700 1 Processor -- start 2024-04-23T10:25:42.772 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:25:42.771+0000 7f60348a3700 1 -- start start 2024-04-23T10:25:42.772 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:25:42.771+0000 7f60348a3700 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f6030101030 0x7f603019b310 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T10:25:42.772 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:25:42.771+0000 7f60348a3700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f60301019d0 0x7f603019b850 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T10:25:42.772 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:25:42.771+0000 7f60348a3700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f6030108ad0 0x7f603019fbe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T10:25:42.772 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:25:42.771+0000 7f60348a3700 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x7f6030112c40 con 0x7f6030108ad0 2024-04-23T10:25:42.772 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:25:42.771+0000 7f60348a3700 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 -- 0x7f6030112ac0 con 0x7f60301019d0 2024-04-23T10:25:42.772 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:25:42.771+0000 7f60348a3700 1 -- --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_getmap magic: 0 -- 0x7f6030112dc0 con 0x7f6030101030 2024-04-23T10:25:42.772 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:25:42.771+0000 7f602ed9d700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f6030108ad0 0x7f603019fbe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T10:25:42.773 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:25:42.771+0000 7f602ed9d700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f6030108ad0 0x7f603019fbe0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.114:50884/0 (socket says 172.21.15.114:50884) 2024-04-23T10:25:42.773 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:25:42.771+0000 7f602ed9d700 1 -- 172.21.15.114:0/500770910 learned_addr learned my addr 172.21.15.114:0/500770910 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-04-23T10:25:42.773 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:25:42.771+0000 7f602e59c700 1 --2- 172.21.15.114:0/500770910 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f6030101030 0x7f603019b310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T10:25:42.773 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:25:42.771+0000 7f602dd9b700 1 --2- 172.21.15.114:0/500770910 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f60301019d0 0x7f603019b850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T10:25:42.773 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:25:42.771+0000 7f602ed9d700 1 -- 172.21.15.114:0/500770910 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f6030101030 msgr2=0x7f603019b310 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T10:25:42.773 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:25:42.771+0000 7f602ed9d700 1 --2- 172.21.15.114:0/500770910 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f6030101030 0x7f603019b310 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:25:42.773 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:25:42.771+0000 7f602ed9d700 1 -- 172.21.15.114:0/500770910 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f60301019d0 msgr2=0x7f603019b850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T10:25:42.773 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:25:42.771+0000 7f602ed9d700 1 --2- 172.21.15.114:0/500770910 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f60301019d0 0x7f603019b850 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:25:42.773 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:25:42.771+0000 7f602ed9d700 1 -- 172.21.15.114:0/500770910 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f60301a0320 con 0x7f6030108ad0 2024-04-23T10:25:42.773 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:25:42.772+0000 7f602ed9d700 1 --2- 172.21.15.114:0/500770910 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f6030108ad0 0x7f603019fbe0 secure :-1 s=READY pgs=285 cs=0 l=1 rev1=1 crypto rx=0x7f602000b550 tx=0x7f602000b860 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T10:25:42.773 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:25:42.772+0000 7f601f7fe700 1 -- 172.21.15.114:0/500770910 <== mon.0 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6020015020 con 0x7f6030108ad0 2024-04-23T10:25:42.773 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:25:42.772+0000 7f601f7fe700 1 -- 172.21.15.114:0/500770910 <== mon.0 v2:172.21.15.114:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f6020011ce0 con 0x7f6030108ad0 2024-04-23T10:25:42.773 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:25:42.772+0000 7f60348a3700 1 -- 172.21.15.114:0/500770910 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f60301a0610 con 0x7f6030108ad0 2024-04-23T10:25:42.773 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:25:42.772+0000 7f601f7fe700 1 -- 172.21.15.114:0/500770910 <== mon.0 v2:172.21.15.114:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6020010c90 con 0x7f6030108ad0 2024-04-23T10:25:42.773 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:25:42.772+0000 7f60348a3700 1 -- 172.21.15.114:0/500770910 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f60301a7e90 con 0x7f6030108ad0 2024-04-23T10:25:42.774 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:25:42.773+0000 7f60348a3700 1 -- 172.21.15.114:0/500770910 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f6030104350 con 0x7f6030108ad0 2024-04-23T10:25:42.774 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:25:42.774+0000 7f601f7fe700 1 -- 172.21.15.114:0/500770910 <== mon.0 v2:172.21.15.114:3300/0 4 ==== mgrmap(e 17) ==== 138698+0+0 (secure 0 0 0) 0x7f6020018020 con 0x7f6030108ad0 2024-04-23T10:25:42.774 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:25:42.775+0000 7f601f7fe700 1 --2- 172.21.15.114:0/500770910 >> [v2:172.21.15.114:6800/465333808,v1:172.21.15.114:6801/465333808] conn(0x7f60180a3cd0 0x7f60180a6190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T10:25:42.774 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:25:42.775+0000 7f602e59c700 1 --2- 172.21.15.114:0/500770910 >> [v2:172.21.15.114:6800/465333808,v1:172.21.15.114:6801/465333808] conn(0x7f60180a3cd0 0x7f60180a6190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T10:25:42.775 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:25:42.775+0000 7f601f7fe700 1 -- 172.21.15.114:0/500770910 <== mon.0 v2:172.21.15.114:3300/0 5 ==== osd_map(87..87 src has 1..87) ==== 7979+0+0 (secure 0 0 0) 0x7f60200cfe30 con 0x7f6030108ad0 2024-04-23T10:25:42.775 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:25:42.775+0000 7f602e59c700 1 --2- 172.21.15.114:0/500770910 >> [v2:172.21.15.114:6800/465333808,v1:172.21.15.114:6801/465333808] conn(0x7f60180a3cd0 0x7f60180a6190 secure :-1 s=READY pgs=201 cs=0 l=1 rev1=1 crypto rx=0x7f6028009e60 tx=0x7f602800b7b0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T10:25:42.777 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:25:42.778+0000 7f601f7fe700 1 -- 172.21.15.114:0/500770910 <== mon.0 v2:172.21.15.114:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f602001f020 con 0x7f6030108ad0 2024-04-23T10:25:42.828 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 23 10:25:42 smithi204 ceph-mon[32649]: from='client.? 172.21.15.114:0/2940402328' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 0}]: dispatch 2024-04-23T10:25:42.828 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 23 10:25:42 smithi204 ceph-mon[32649]: pgmap v315: 97 pgs: 97 active+clean; 641 KiB data, 370 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-23T10:25:42.860 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:25:41.253+0000 7ff701296700 1 -- 172.21.15.114:0/4138502153 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7ff6fc108aa0 msgr2=0x7ff6fc108e80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T10:25:42.861 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:25:41.253+0000 7ff701296700 1 --2- 172.21.15.114:0/4138502153 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7ff6fc108aa0 0x7ff6fc108e80 secure :-1 s=READY pgs=280 cs=0 l=1 rev1=1 crypto rx=0x7ff6ec009990 tx=0x7ff6ec009ca0 comp rx=0 tx=0).stop 2024-04-23T10:25:42.861 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:25:41.253+0000 7ff701296700 1 -- 172.21.15.114:0/4138502153 shutdown_connections 2024-04-23T10:25:42.861 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:25:41.253+0000 7ff701296700 1 --2- 172.21.15.114:0/4138502153 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7ff6fc103490 0x7ff6fc10f970 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:25:42.861 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:25:41.253+0000 7ff701296700 1 --2- 172.21.15.114:0/4138502153 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7ff6fc108aa0 0x7ff6fc108e80 unknown :-1 s=CLOSED pgs=280 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:25:42.861 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:25:41.253+0000 7ff701296700 1 --2- 172.21.15.114:0/4138502153 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7ff6fc102ad0 0x7ff6fc102f50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:25:42.861 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:25:41.253+0000 7ff701296700 1 -- 172.21.15.114:0/4138502153 >> 172.21.15.114:0/4138502153 conn(0x7ff6fc079c30 msgr2=0x7ff6fc07a040 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T10:25:42.861 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:25:41.254+0000 7ff701296700 1 -- 172.21.15.114:0/4138502153 shutdown_connections 2024-04-23T10:25:42.861 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:25:41.254+0000 7ff701296700 1 -- 172.21.15.114:0/4138502153 wait complete. 2024-04-23T10:25:42.861 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:25:41.254+0000 7ff701296700 1 Processor -- start 2024-04-23T10:25:42.861 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:25:41.254+0000 7ff701296700 1 -- start start 2024-04-23T10:25:42.861 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:25:41.254+0000 7ff701296700 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7ff6fc102ad0 0x7ff6fc074d40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T10:25:42.861 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:25:41.254+0000 7ff701296700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7ff6fc103490 0x7ff6fc075280 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T10:25:42.861 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:25:41.254+0000 7ff701296700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7ff6fc108aa0 0x7ff6fc0757c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T10:25:42.861 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:25:41.255+0000 7ff701296700 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x7ff6fc112d40 con 0x7ff6fc103490 2024-04-23T10:25:42.862 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:25:41.255+0000 7ff701296700 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 -- 0x7ff6fc112bc0 con 0x7ff6fc108aa0 2024-04-23T10:25:42.862 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:25:41.255+0000 7ff701296700 1 -- --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_getmap magic: 0 -- 0x7ff6fc112ec0 con 0x7ff6fc102ad0 2024-04-23T10:25:42.862 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:25:41.255+0000 7ff6fa7fc700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7ff6fc103490 0x7ff6fc075280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T10:25:42.862 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:25:41.255+0000 7ff6fa7fc700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7ff6fc103490 0x7ff6fc075280 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.114:50844/0 (socket says 172.21.15.114:50844) 2024-04-23T10:25:42.862 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:25:41.255+0000 7ff6fa7fc700 1 -- 172.21.15.114:0/1030411522 learned_addr learned my addr 172.21.15.114:0/1030411522 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-04-23T10:25:42.862 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:25:41.255+0000 7ff6faffd700 1 --2- 172.21.15.114:0/1030411522 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7ff6fc102ad0 0x7ff6fc074d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T10:25:42.862 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:25:41.255+0000 7ff6fb7fe700 1 --2- 172.21.15.114:0/1030411522 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7ff6fc108aa0 0x7ff6fc0757c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T10:25:42.862 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:25:41.255+0000 7ff6fa7fc700 1 -- 172.21.15.114:0/1030411522 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7ff6fc102ad0 msgr2=0x7ff6fc074d40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T10:25:42.862 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:25:41.255+0000 7ff6fa7fc700 1 --2- 172.21.15.114:0/1030411522 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7ff6fc102ad0 0x7ff6fc074d40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:25:42.862 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:25:41.255+0000 7ff6fa7fc700 1 -- 172.21.15.114:0/1030411522 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7ff6fc108aa0 msgr2=0x7ff6fc0757c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T10:25:42.862 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:25:41.255+0000 7ff6fa7fc700 1 --2- 172.21.15.114:0/1030411522 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7ff6fc108aa0 0x7ff6fc0757c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:25:42.862 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:25:41.255+0000 7ff6fa7fc700 1 -- 172.21.15.114:0/1030411522 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff6fc1a7760 con 0x7ff6fc103490 2024-04-23T10:25:42.863 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:25:41.255+0000 7ff6fb7fe700 1 --2- 172.21.15.114:0/1030411522 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7ff6fc108aa0 0x7ff6fc0757c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-23T10:25:42.863 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:25:41.255+0000 7ff6faffd700 1 --2- 172.21.15.114:0/1030411522 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7ff6fc102ad0 0x7ff6fc074d40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-23T10:25:42.863 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:25:41.255+0000 7ff6fa7fc700 1 --2- 172.21.15.114:0/1030411522 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7ff6fc103490 0x7ff6fc075280 secure :-1 s=READY pgs=281 cs=0 l=1 rev1=1 crypto rx=0x7ff6f000b530 tx=0x7ff6f000b8f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T10:25:42.863 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:25:41.255+0000 7ff6e3fff700 1 -- 172.21.15.114:0/1030411522 <== mon.0 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff6f0015020 con 0x7ff6fc103490 2024-04-23T10:25:42.863 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:25:41.255+0000 7ff6e3fff700 1 -- 172.21.15.114:0/1030411522 <== mon.0 v2:172.21.15.114:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7ff6f0011ce0 con 0x7ff6fc103490 2024-04-23T10:25:42.863 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:25:41.255+0000 7ff701296700 1 -- 172.21.15.114:0/1030411522 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7ff6fc1a7a50 con 0x7ff6fc103490 2024-04-23T10:25:42.863 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:25:41.255+0000 7ff6e3fff700 1 -- 172.21.15.114:0/1030411522 <== mon.0 v2:172.21.15.114:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff6f0010d20 con 0x7ff6fc103490 2024-04-23T10:25:42.863 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:25:41.255+0000 7ff701296700 1 -- 172.21.15.114:0/1030411522 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7ff6fc1a7f90 con 0x7ff6fc103490 2024-04-23T10:25:42.863 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:25:41.256+0000 7ff701296700 1 -- 172.21.15.114:0/1030411522 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7ff6fc10b0a0 con 0x7ff6fc103490 2024-04-23T10:25:42.863 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:25:41.257+0000 7ff6e3fff700 1 -- 172.21.15.114:0/1030411522 <== mon.0 v2:172.21.15.114:3300/0 4 ==== mgrmap(e 17) ==== 138698+0+0 (secure 0 0 0) 0x7ff6f0018020 con 0x7ff6fc103490 2024-04-23T10:25:42.863 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:25:41.258+0000 7ff6e3fff700 1 --2- 172.21.15.114:0/1030411522 >> [v2:172.21.15.114:6800/465333808,v1:172.21.15.114:6801/465333808] conn(0x7ff6e40a3d50 0x7ff6e40a6210 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T10:25:42.863 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:25:41.258+0000 7ff6faffd700 1 --2- 172.21.15.114:0/1030411522 >> [v2:172.21.15.114:6800/465333808,v1:172.21.15.114:6801/465333808] conn(0x7ff6e40a3d50 0x7ff6e40a6210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T10:25:42.864 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:25:41.258+0000 7ff6e3fff700 1 -- 172.21.15.114:0/1030411522 <== mon.0 v2:172.21.15.114:3300/0 5 ==== osd_map(87..87 src has 1..87) ==== 7979+0+0 (secure 0 0 0) 0x7ff6f00cfad0 con 0x7ff6fc103490 2024-04-23T10:25:42.864 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:25:41.258+0000 7ff6faffd700 1 --2- 172.21.15.114:0/1030411522 >> [v2:172.21.15.114:6800/465333808,v1:172.21.15.114:6801/465333808] conn(0x7ff6e40a3d50 0x7ff6e40a6210 secure :-1 s=READY pgs=199 cs=0 l=1 rev1=1 crypto rx=0x7ff6ec006010 tx=0x7ff6ec00b540 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T10:25:42.864 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:25:41.263+0000 7ff6e3fff700 1 -- 172.21.15.114:0/1030411522 <== mon.0 v2:172.21.15.114:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7ff6f001f020 con 0x7ff6fc103490 2024-04-23T10:25:42.864 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:25:41.478+0000 7ff701296700 1 -- 172.21.15.114:0/1030411522 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 9} v 0) -- 0x7ff6fc10aa80 con 0x7ff6fc103490 2024-04-23T10:25:42.864 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:25:41.478+0000 7ff6e3fff700 1 -- 172.21.15.114:0/1030411522 <== mon.0 v2:172.21.15.114:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 9}]=0 v0) ==== 74+0+13 (secure 0 0 0) 0x7ff6f009c610 con 0x7ff6fc103490 2024-04-23T10:25:42.864 INFO:teuthology.orchestra.run.smithi114.stdout:296352743459 2024-04-23T10:25:42.864 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:25:41.480+0000 7ff701296700 1 -- 172.21.15.114:0/1030411522 >> [v2:172.21.15.114:6800/465333808,v1:172.21.15.114:6801/465333808] conn(0x7ff6e40a3d50 msgr2=0x7ff6e40a6210 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T10:25:42.864 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:25:41.480+0000 7ff701296700 1 --2- 172.21.15.114:0/1030411522 >> [v2:172.21.15.114:6800/465333808,v1:172.21.15.114:6801/465333808] conn(0x7ff6e40a3d50 0x7ff6e40a6210 secure :-1 s=READY pgs=199 cs=0 l=1 rev1=1 crypto rx=0x7ff6ec006010 tx=0x7ff6ec00b540 comp rx=0 tx=0).stop 2024-04-23T10:25:42.864 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:25:41.480+0000 7ff701296700 1 -- 172.21.15.114:0/1030411522 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7ff6fc103490 msgr2=0x7ff6fc075280 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T10:25:42.865 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:25:41.480+0000 7ff701296700 1 --2- 172.21.15.114:0/1030411522 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7ff6fc103490 0x7ff6fc075280 secure :-1 s=READY pgs=281 cs=0 l=1 rev1=1 crypto rx=0x7ff6f000b530 tx=0x7ff6f000b8f0 comp rx=0 tx=0).stop 2024-04-23T10:25:42.865 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:25:41.480+0000 7ff701296700 1 -- 172.21.15.114:0/1030411522 shutdown_connections 2024-04-23T10:25:42.865 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:25:41.480+0000 7ff701296700 1 --2- 172.21.15.114:0/1030411522 >> [v2:172.21.15.114:6800/465333808,v1:172.21.15.114:6801/465333808] conn(0x7ff6e40a3d50 0x7ff6e40a6210 unknown :-1 s=CLOSED pgs=199 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:25:42.865 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:25:41.480+0000 7ff701296700 1 --2- 172.21.15.114:0/1030411522 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7ff6fc108aa0 0x7ff6fc0757c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:25:42.865 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:25:41.480+0000 7ff701296700 1 --2- 172.21.15.114:0/1030411522 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7ff6fc102ad0 0x7ff6fc074d40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:25:42.865 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:25:41.480+0000 7ff701296700 1 --2- 172.21.15.114:0/1030411522 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7ff6fc103490 0x7ff6fc075280 unknown :-1 s=CLOSED pgs=281 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:25:42.865 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:25:41.480+0000 7ff701296700 1 -- 172.21.15.114:0/1030411522 >> 172.21.15.114:0/1030411522 conn(0x7ff6fc079c30 msgr2=0x7ff6fc100b00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T10:25:42.865 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:25:41.481+0000 7ff701296700 1 -- 172.21.15.114:0/1030411522 shutdown_connections 2024-04-23T10:25:42.865 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:25:41.481+0000 7ff701296700 1 -- 172.21.15.114:0/1030411522 wait complete. 2024-04-23T10:25:42.982 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:25:42.983+0000 7f60348a3700 1 -- 172.21.15.114:0/500770910 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 4} v 0) -- 0x7f6030069730 con 0x7f6030108ad0 2024-04-23T10:25:42.982 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:25:42.984+0000 7f601f7fe700 1 -- 172.21.15.114:0/500770910 <== mon.0 v2:172.21.15.114:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 4}]=0 v0) ==== 74+0+13 (secure 0 0 0) 0x7f6030069730 con 0x7f6030108ad0 2024-04-23T10:25:42.983 INFO:teuthology.orchestra.run.smithi114.stdout:141733920827 2024-04-23T10:25:42.985 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:25:42.986+0000 7f60348a3700 1 -- 172.21.15.114:0/500770910 >> [v2:172.21.15.114:6800/465333808,v1:172.21.15.114:6801/465333808] conn(0x7f60180a3cd0 msgr2=0x7f60180a6190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T10:25:42.986 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:25:42.986+0000 7f60348a3700 1 --2- 172.21.15.114:0/500770910 >> [v2:172.21.15.114:6800/465333808,v1:172.21.15.114:6801/465333808] conn(0x7f60180a3cd0 0x7f60180a6190 secure :-1 s=READY pgs=201 cs=0 l=1 rev1=1 crypto rx=0x7f6028009e60 tx=0x7f602800b7b0 comp rx=0 tx=0).stop 2024-04-23T10:25:42.986 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:25:42.986+0000 7f60348a3700 1 -- 172.21.15.114:0/500770910 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f6030108ad0 msgr2=0x7f603019fbe0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T10:25:42.986 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:25:42.986+0000 7f60348a3700 1 --2- 172.21.15.114:0/500770910 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f6030108ad0 0x7f603019fbe0 secure :-1 s=READY pgs=285 cs=0 l=1 rev1=1 crypto rx=0x7f602000b550 tx=0x7f602000b860 comp rx=0 tx=0).stop 2024-04-23T10:25:42.986 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:25:42.986+0000 7f60348a3700 1 -- 172.21.15.114:0/500770910 shutdown_connections 2024-04-23T10:25:42.986 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:25:42.986+0000 7f60348a3700 1 --2- 172.21.15.114:0/500770910 >> [v2:172.21.15.114:6800/465333808,v1:172.21.15.114:6801/465333808] conn(0x7f60180a3cd0 0x7f60180a6190 unknown :-1 s=CLOSED pgs=201 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:25:42.986 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:25:42.986+0000 7f60348a3700 1 --2- 172.21.15.114:0/500770910 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f6030108ad0 0x7f603019fbe0 unknown :-1 s=CLOSED pgs=285 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:25:42.986 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:25:42.986+0000 7f60348a3700 1 --2- 172.21.15.114:0/500770910 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f6030101030 0x7f603019b310 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:25:42.986 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:25:42.986+0000 7f60348a3700 1 --2- 172.21.15.114:0/500770910 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f60301019d0 0x7f603019b850 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:25:42.986 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:25:42.986+0000 7f60348a3700 1 -- 172.21.15.114:0/500770910 >> 172.21.15.114:0/500770910 conn(0x7f6030077fb0 msgr2=0x7f60300ff570 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T10:25:42.986 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:25:42.986+0000 7f60348a3700 1 -- 172.21.15.114:0/500770910 shutdown_connections 2024-04-23T10:25:42.986 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:25:42.986+0000 7f60348a3700 1 -- 172.21.15.114:0/500770910 wait complete. 2024-04-23T10:25:42.991 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 23 10:25:42 smithi119 ceph-mon[32608]: from='client.? 172.21.15.114:0/2940402328' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 0}]: dispatch 2024-04-23T10:25:42.991 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 23 10:25:42 smithi119 ceph-mon[32608]: pgmap v315: 97 pgs: 97 active+clean; 641 KiB data, 370 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-23T10:25:43.528 INFO:tasks.cephadm.ceph_manager.ceph:need seq 296352743456 got 296352743459 for osd.9 2024-04-23T10:25:43.528 DEBUG:teuthology.parallel:result is None 2024-04-23T10:25:43.754 INFO:journalctl@ceph.mon.a.smithi114.stdout:Apr 23 10:25:43 smithi114 ceph-mon[29255]: from='mgr.14152 172.21.15.114:0/1529706138' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-23T10:25:43.754 INFO:journalctl@ceph.mon.a.smithi114.stdout:Apr 23 10:25:43 smithi114 ceph-mon[29255]: from='client.? 172.21.15.114:0/500770910' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 4}]: dispatch 2024-04-23T10:25:43.827 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 23 10:25:43 smithi204 ceph-mon[32649]: from='mgr.14152 172.21.15.114:0/1529706138' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-23T10:25:43.827 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 23 10:25:43 smithi204 ceph-mon[32649]: from='client.? 172.21.15.114:0/500770910' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 4}]: dispatch 2024-04-23T10:25:43.858 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 23 10:25:43 smithi119 ceph-mon[32608]: from='mgr.14152 172.21.15.114:0/1529706138' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-23T10:25:43.858 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 23 10:25:43 smithi119 ceph-mon[32608]: from='client.? 172.21.15.114:0/500770910' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 4}]: dispatch 2024-04-23T10:25:44.279 INFO:teuthology.orchestra.run.smithi114.stderr:Inferring config /var/lib/ceph/aa240572-0159-11ef-bc93-c7b262605968/mon.a/config 2024-04-23T10:25:44.667 INFO:tasks.cephadm.ceph_manager.ceph:need seq 201863462958 got 201863462962 for osd.6 2024-04-23T10:25:44.667 DEBUG:teuthology.parallel:result is None 2024-04-23T10:25:44.749 INFO:journalctl@ceph.mon.a.smithi114.stdout:Apr 23 10:25:44 smithi114 ceph-mon[29255]: pgmap v316: 97 pgs: 97 active+clean; 641 KiB data, 370 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-04-23T10:25:44.828 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 23 10:25:44 smithi204 ceph-mon[32649]: pgmap v316: 97 pgs: 97 active+clean; 641 KiB data, 370 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-04-23T10:25:44.991 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 23 10:25:44 smithi119 ceph-mon[32608]: pgmap v316: 97 pgs: 97 active+clean; 641 KiB data, 370 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-04-23T10:25:45.255 INFO:tasks.cephadm.ceph_manager.ceph:need seq 317827579928 got 317827579932 for osd.10 2024-04-23T10:25:45.255 DEBUG:teuthology.parallel:result is None 2024-04-23T10:25:45.458 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:25:45.459+0000 7fcde3b17700 1 -- 172.21.15.114:0/3494436203 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fcddc102330 msgr2=0x7fcddc10e7f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T10:25:45.459 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:25:45.459+0000 7fcde3b17700 1 --2- 172.21.15.114:0/3494436203 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fcddc102330 0x7fcddc10e7f0 secure :-1 s=READY pgs=286 cs=0 l=1 rev1=1 crypto rx=0x7fcdd400b0a0 tx=0x7fcdd400b3b0 comp rx=0 tx=0).stop 2024-04-23T10:25:45.459 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:25:45.459+0000 7fcde3b17700 1 -- 172.21.15.114:0/3494436203 shutdown_connections 2024-04-23T10:25:45.459 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:25:45.459+0000 7fcde3b17700 1 --2- 172.21.15.114:0/3494436203 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fcddc102330 0x7fcddc10e7f0 unknown :-1 s=CLOSED pgs=286 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:25:45.459 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:25:45.459+0000 7fcde3b17700 1 --2- 172.21.15.114:0/3494436203 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fcddc107920 0x7fcddc107d00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:25:45.459 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:25:45.459+0000 7fcde3b17700 1 --2- 172.21.15.114:0/3494436203 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fcddc101970 0x7fcddc101df0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:25:45.459 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:25:45.459+0000 7fcde3b17700 1 -- 172.21.15.114:0/3494436203 >> 172.21.15.114:0/3494436203 conn(0x7fcddc078910 msgr2=0x7fcddc078d20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T10:25:45.459 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:25:45.460+0000 7fcde3b17700 1 -- 172.21.15.114:0/3494436203 shutdown_connections 2024-04-23T10:25:45.459 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:25:45.460+0000 7fcde3b17700 1 -- 172.21.15.114:0/3494436203 wait complete. 2024-04-23T10:25:45.461 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:25:45.460+0000 7fcde3b17700 1 Processor -- start 2024-04-23T10:25:45.461 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:25:45.460+0000 7fcde3b17700 1 -- start start 2024-04-23T10:25:45.461 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:25:45.461+0000 7fcde3b17700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fcddc101970 0x7fcddc199f60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T10:25:45.461 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:25:45.461+0000 7fcde3b17700 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fcddc102330 0x7fcddc19a4a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T10:25:45.461 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:25:45.461+0000 7fcde3b17700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fcddc107920 0x7fcddc1a4fe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T10:25:45.462 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:25:45.461+0000 7fcde3b17700 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x7fcddc111b60 con 0x7fcddc107920 2024-04-23T10:25:45.462 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:25:45.461+0000 7fcde3b17700 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 -- 0x7fcddc1119e0 con 0x7fcddc101970 2024-04-23T10:25:45.462 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:25:45.461+0000 7fcde3b17700 1 -- --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_getmap magic: 0 -- 0x7fcddc111ce0 con 0x7fcddc102330 2024-04-23T10:25:45.462 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:25:45.461+0000 7fcde20b4700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fcddc107920 0x7fcddc1a4fe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T10:25:45.462 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:25:45.461+0000 7fcde20b4700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fcddc107920 0x7fcddc1a4fe0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.114:50900/0 (socket says 172.21.15.114:50900) 2024-04-23T10:25:45.462 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:25:45.461+0000 7fcde20b4700 1 -- 172.21.15.114:0/1554184828 learned_addr learned my addr 172.21.15.114:0/1554184828 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-04-23T10:25:45.462 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:25:45.461+0000 7fcde10b2700 1 --2- 172.21.15.114:0/1554184828 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fcddc102330 0x7fcddc19a4a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T10:25:45.462 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:25:45.461+0000 7fcde18b3700 1 --2- 172.21.15.114:0/1554184828 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fcddc101970 0x7fcddc199f60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T10:25:45.462 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:25:45.461+0000 7fcde20b4700 1 -- 172.21.15.114:0/1554184828 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fcddc102330 msgr2=0x7fcddc19a4a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T10:25:45.462 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:25:45.461+0000 7fcde20b4700 1 --2- 172.21.15.114:0/1554184828 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fcddc102330 0x7fcddc19a4a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:25:45.462 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:25:45.461+0000 7fcde20b4700 1 -- 172.21.15.114:0/1554184828 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fcddc101970 msgr2=0x7fcddc199f60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T10:25:45.462 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:25:45.461+0000 7fcde20b4700 1 --2- 172.21.15.114:0/1554184828 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fcddc101970 0x7fcddc199f60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:25:45.462 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:25:45.461+0000 7fcde20b4700 1 -- 172.21.15.114:0/1554184828 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fcddc1a5520 con 0x7fcddc107920 2024-04-23T10:25:45.463 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:25:45.461+0000 7fcde20b4700 1 --2- 172.21.15.114:0/1554184828 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fcddc107920 0x7fcddc1a4fe0 secure :-1 s=READY pgs=287 cs=0 l=1 rev1=1 crypto rx=0x7fcdd40062a0 tx=0x7fcdd4004e20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T10:25:45.463 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:25:45.461+0000 7fcdd2ffd700 1 -- 172.21.15.114:0/1554184828 <== mon.0 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcdd4021070 con 0x7fcddc107920 2024-04-23T10:25:45.463 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:25:45.462+0000 7fcde3b17700 1 -- 172.21.15.114:0/1554184828 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fcddc1a56f0 con 0x7fcddc107920 2024-04-23T10:25:45.463 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:25:45.462+0000 7fcdd2ffd700 1 -- 172.21.15.114:0/1554184828 <== mon.0 v2:172.21.15.114:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fcdd4007c80 con 0x7fcddc107920 2024-04-23T10:25:45.463 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:25:45.462+0000 7fcdd2ffd700 1 -- 172.21.15.114:0/1554184828 <== mon.0 v2:172.21.15.114:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcdd401d450 con 0x7fcddc107920 2024-04-23T10:25:45.463 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:25:45.462+0000 7fcde3b17700 1 -- 172.21.15.114:0/1554184828 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fcddc1a5ba0 con 0x7fcddc107920 2024-04-23T10:25:45.463 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:25:45.462+0000 7fcde3b17700 1 -- 172.21.15.114:0/1554184828 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fcddc069730 con 0x7fcddc107920 2024-04-23T10:25:45.463 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:25:45.464+0000 7fcdd2ffd700 1 -- 172.21.15.114:0/1554184828 <== mon.0 v2:172.21.15.114:3300/0 4 ==== mgrmap(e 17) ==== 138698+0+0 (secure 0 0 0) 0x7fcdd4004000 con 0x7fcddc107920 2024-04-23T10:25:45.464 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:25:45.465+0000 7fcdd2ffd700 1 --2- 172.21.15.114:0/1554184828 >> [v2:172.21.15.114:6800/465333808,v1:172.21.15.114:6801/465333808] conn(0x7fcdc80a3cd0 0x7fcdc80a6190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T10:25:45.464 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:25:45.465+0000 7fcde18b3700 1 --2- 172.21.15.114:0/1554184828 >> [v2:172.21.15.114:6800/465333808,v1:172.21.15.114:6801/465333808] conn(0x7fcdc80a3cd0 0x7fcdc80a6190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T10:25:45.464 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:25:45.465+0000 7fcdd2ffd700 1 -- 172.21.15.114:0/1554184828 <== mon.0 v2:172.21.15.114:3300/0 5 ==== osd_map(87..87 src has 1..87) ==== 7979+0+0 (secure 0 0 0) 0x7fcdd400f020 con 0x7fcddc107920 2024-04-23T10:25:45.464 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:25:45.465+0000 7fcde18b3700 1 --2- 172.21.15.114:0/1554184828 >> [v2:172.21.15.114:6800/465333808,v1:172.21.15.114:6801/465333808] conn(0x7fcdc80a3cd0 0x7fcdc80a6190 secure :-1 s=READY pgs=202 cs=0 l=1 rev1=1 crypto rx=0x7fcdd8006fd0 tx=0x7fcdd8008040 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T10:25:45.468 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:25:45.470+0000 7fcdd2ffd700 1 -- 172.21.15.114:0/1554184828 <== mon.0 v2:172.21.15.114:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7fcdd402a070 con 0x7fcddc107920 2024-04-23T10:25:45.676 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:25:45.677+0000 7fcde3b17700 1 -- 172.21.15.114:0/1554184828 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 1} v 0) -- 0x7fcddc02e140 con 0x7fcddc107920 2024-04-23T10:25:45.676 INFO:journalctl@ceph.mon.a.smithi114.stdout:Apr 23 10:25:45 smithi114 ceph-mon[29255]: from='mgr.14152 172.21.15.114:0/1529706138' entity='mgr.x' 2024-04-23T10:25:45.676 INFO:journalctl@ceph.mon.a.smithi114.stdout:Apr 23 10:25:45 smithi114 ceph-mon[29255]: from='mgr.14152 172.21.15.114:0/1529706138' entity='mgr.x' 2024-04-23T10:25:45.676 INFO:journalctl@ceph.mon.a.smithi114.stdout:Apr 23 10:25:45 smithi114 ceph-mon[29255]: from='mgr.14152 172.21.15.114:0/1529706138' entity='mgr.x' 2024-04-23T10:25:45.676 INFO:journalctl@ceph.mon.a.smithi114.stdout:Apr 23 10:25:45 smithi114 ceph-mon[29255]: from='mgr.14152 172.21.15.114:0/1529706138' entity='mgr.x' 2024-04-23T10:25:45.677 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:25:45.677+0000 7fcdd2ffd700 1 -- 172.21.15.114:0/1554184828 <== mon.0 v2:172.21.15.114:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 1}]=0 v0) ==== 74+0+12 (secure 0 0 0) 0x7fcdd409e720 con 0x7fcddc107920 2024-04-23T10:25:45.677 INFO:teuthology.orchestra.run.smithi114.stdout:55834574923 2024-04-23T10:25:45.678 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:25:45.679+0000 7fcde3b17700 1 -- 172.21.15.114:0/1554184828 >> [v2:172.21.15.114:6800/465333808,v1:172.21.15.114:6801/465333808] conn(0x7fcdc80a3cd0 msgr2=0x7fcdc80a6190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T10:25:45.678 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:25:45.679+0000 7fcde3b17700 1 --2- 172.21.15.114:0/1554184828 >> [v2:172.21.15.114:6800/465333808,v1:172.21.15.114:6801/465333808] conn(0x7fcdc80a3cd0 0x7fcdc80a6190 secure :-1 s=READY pgs=202 cs=0 l=1 rev1=1 crypto rx=0x7fcdd8006fd0 tx=0x7fcdd8008040 comp rx=0 tx=0).stop 2024-04-23T10:25:45.679 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:25:45.679+0000 7fcde3b17700 1 -- 172.21.15.114:0/1554184828 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fcddc107920 msgr2=0x7fcddc1a4fe0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T10:25:45.679 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:25:45.679+0000 7fcde3b17700 1 --2- 172.21.15.114:0/1554184828 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fcddc107920 0x7fcddc1a4fe0 secure :-1 s=READY pgs=287 cs=0 l=1 rev1=1 crypto rx=0x7fcdd40062a0 tx=0x7fcdd4004e20 comp rx=0 tx=0).stop 2024-04-23T10:25:45.679 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:25:45.679+0000 7fcde3b17700 1 -- 172.21.15.114:0/1554184828 shutdown_connections 2024-04-23T10:25:45.679 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:25:45.679+0000 7fcde3b17700 1 --2- 172.21.15.114:0/1554184828 >> [v2:172.21.15.114:6800/465333808,v1:172.21.15.114:6801/465333808] conn(0x7fcdc80a3cd0 0x7fcdc80a6190 unknown :-1 s=CLOSED pgs=202 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:25:45.679 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:25:45.679+0000 7fcde3b17700 1 --2- 172.21.15.114:0/1554184828 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fcddc101970 0x7fcddc199f60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:25:45.679 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:25:45.679+0000 7fcde3b17700 1 --2- 172.21.15.114:0/1554184828 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fcddc107920 0x7fcddc1a4fe0 unknown :-1 s=CLOSED pgs=287 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:25:45.679 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:25:45.679+0000 7fcde3b17700 1 --2- 172.21.15.114:0/1554184828 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fcddc102330 0x7fcddc19a4a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:25:45.679 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:25:45.679+0000 7fcde3b17700 1 -- 172.21.15.114:0/1554184828 >> 172.21.15.114:0/1554184828 conn(0x7fcddc078910 msgr2=0x7fcddc0ff980 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T10:25:45.679 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:25:45.679+0000 7fcde3b17700 1 -- 172.21.15.114:0/1554184828 shutdown_connections 2024-04-23T10:25:45.679 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:25:45.679+0000 7fcde3b17700 1 -- 172.21.15.114:0/1554184828 wait complete. 2024-04-23T10:25:45.828 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 23 10:25:45 smithi204 ceph-mon[32649]: from='mgr.14152 172.21.15.114:0/1529706138' entity='mgr.x' 2024-04-23T10:25:45.828 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 23 10:25:45 smithi204 ceph-mon[32649]: from='mgr.14152 172.21.15.114:0/1529706138' entity='mgr.x' 2024-04-23T10:25:45.828 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 23 10:25:45 smithi204 ceph-mon[32649]: from='mgr.14152 172.21.15.114:0/1529706138' entity='mgr.x' 2024-04-23T10:25:45.828 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 23 10:25:45 smithi204 ceph-mon[32649]: from='mgr.14152 172.21.15.114:0/1529706138' entity='mgr.x' 2024-04-23T10:25:45.991 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 23 10:25:45 smithi119 ceph-mon[32608]: from='mgr.14152 172.21.15.114:0/1529706138' entity='mgr.x' 2024-04-23T10:25:45.992 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 23 10:25:45 smithi119 ceph-mon[32608]: from='mgr.14152 172.21.15.114:0/1529706138' entity='mgr.x' 2024-04-23T10:25:45.992 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 23 10:25:45 smithi119 ceph-mon[32608]: from='mgr.14152 172.21.15.114:0/1529706138' entity='mgr.x' 2024-04-23T10:25:45.992 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 23 10:25:45 smithi119 ceph-mon[32608]: from='mgr.14152 172.21.15.114:0/1529706138' entity='mgr.x' 2024-04-23T10:25:46.566 INFO:tasks.cephadm.ceph_manager.ceph:need seq 141733920824 got 141733920827 for osd.4 2024-04-23T10:25:46.566 DEBUG:teuthology.parallel:result is None 2024-04-23T10:25:46.754 INFO:journalctl@ceph.mon.a.smithi114.stdout:Apr 23 10:25:46 smithi114 ceph-mon[29255]: from='client.? 172.21.15.114:0/1554184828' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 1}]: dispatch 2024-04-23T10:25:46.754 INFO:journalctl@ceph.mon.a.smithi114.stdout:Apr 23 10:25:46 smithi114 ceph-mon[29255]: from='mgr.14152 172.21.15.114:0/1529706138' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.e", "id": [2, 4]}]: dispatch 2024-04-23T10:25:46.754 INFO:journalctl@ceph.mon.a.smithi114.stdout:Apr 23 10:25:46 smithi114 ceph-mon[29255]: from='mgr.14152 172.21.15.114:0/1529706138' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.1d", "id": [8, 11]}]: dispatch 2024-04-23T10:25:46.754 INFO:journalctl@ceph.mon.a.smithi114.stdout:Apr 23 10:25:46 smithi114 ceph-mon[29255]: from='mgr.14152 172.21.15.114:0/1529706138' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.e", "id": [5, 9]}]: dispatch 2024-04-23T10:25:46.754 INFO:journalctl@ceph.mon.a.smithi114.stdout:Apr 23 10:25:46 smithi114 ceph-mon[29255]: from='mgr.14152 172.21.15.114:0/1529706138' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.12", "id": [3, 9]}]: dispatch 2024-04-23T10:25:46.755 INFO:journalctl@ceph.mon.a.smithi114.stdout:Apr 23 10:25:46 smithi114 ceph-mon[29255]: from='mgr.14152 172.21.15.114:0/1529706138' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.17", "id": [4, 6]}]: dispatch 2024-04-23T10:25:46.755 INFO:journalctl@ceph.mon.a.smithi114.stdout:Apr 23 10:25:46 smithi114 ceph-mon[29255]: from='mgr.14152 172.21.15.114:0/1529706138' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.23", "id": [4, 7]}]: dispatch 2024-04-23T10:25:46.755 INFO:journalctl@ceph.mon.a.smithi114.stdout:Apr 23 10:25:46 smithi114 ceph-mon[29255]: from='mgr.14152 172.21.15.114:0/1529706138' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.27", "id": [4, 6]}]: dispatch 2024-04-23T10:25:46.755 INFO:journalctl@ceph.mon.a.smithi114.stdout:Apr 23 10:25:46 smithi114 ceph-mon[29255]: from='mgr.14152 172.21.15.114:0/1529706138' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.31", "id": [4, 8]}]: dispatch 2024-04-23T10:25:46.755 INFO:journalctl@ceph.mon.a.smithi114.stdout:Apr 23 10:25:46 smithi114 ceph-mon[29255]: from='mgr.14152 172.21.15.114:0/1529706138' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.38", "id": [3, 9]}]: dispatch 2024-04-23T10:25:46.755 INFO:journalctl@ceph.mon.a.smithi114.stdout:Apr 23 10:25:46 smithi114 ceph-mon[29255]: from='mgr.14152 172.21.15.114:0/1529706138' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.39", "id": [5, 8]}]: dispatch 2024-04-23T10:25:46.755 INFO:journalctl@ceph.mon.a.smithi114.stdout:Apr 23 10:25:46 smithi114 ceph-mon[29255]: pgmap v317: 97 pgs: 97 active+clean; 641 KiB data, 370 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-23T10:25:46.828 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 23 10:25:46 smithi204 ceph-mon[32649]: from='client.? 172.21.15.114:0/1554184828' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 1}]: dispatch 2024-04-23T10:25:46.828 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 23 10:25:46 smithi204 ceph-mon[32649]: from='mgr.14152 172.21.15.114:0/1529706138' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.e", "id": [2, 4]}]: dispatch 2024-04-23T10:25:46.828 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 23 10:25:46 smithi204 ceph-mon[32649]: from='mgr.14152 172.21.15.114:0/1529706138' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.1d", "id": [8, 11]}]: dispatch 2024-04-23T10:25:46.828 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 23 10:25:46 smithi204 ceph-mon[32649]: from='mgr.14152 172.21.15.114:0/1529706138' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.e", "id": [5, 9]}]: dispatch 2024-04-23T10:25:46.828 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 23 10:25:46 smithi204 ceph-mon[32649]: from='mgr.14152 172.21.15.114:0/1529706138' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.12", "id": [3, 9]}]: dispatch 2024-04-23T10:25:46.828 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 23 10:25:46 smithi204 ceph-mon[32649]: from='mgr.14152 172.21.15.114:0/1529706138' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.17", "id": [4, 6]}]: dispatch 2024-04-23T10:25:46.828 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 23 10:25:46 smithi204 ceph-mon[32649]: from='mgr.14152 172.21.15.114:0/1529706138' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.23", "id": [4, 7]}]: dispatch 2024-04-23T10:25:46.828 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 23 10:25:46 smithi204 ceph-mon[32649]: from='mgr.14152 172.21.15.114:0/1529706138' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.27", "id": [4, 6]}]: dispatch 2024-04-23T10:25:46.828 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 23 10:25:46 smithi204 ceph-mon[32649]: from='mgr.14152 172.21.15.114:0/1529706138' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.31", "id": [4, 8]}]: dispatch 2024-04-23T10:25:46.828 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 23 10:25:46 smithi204 ceph-mon[32649]: from='mgr.14152 172.21.15.114:0/1529706138' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.38", "id": [3, 9]}]: dispatch 2024-04-23T10:25:46.828 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 23 10:25:46 smithi204 ceph-mon[32649]: from='mgr.14152 172.21.15.114:0/1529706138' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.39", "id": [5, 8]}]: dispatch 2024-04-23T10:25:46.828 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 23 10:25:46 smithi204 ceph-mon[32649]: pgmap v317: 97 pgs: 97 active+clean; 641 KiB data, 370 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-23T10:25:46.892 INFO:tasks.cephadm.ceph_manager.ceph:need seq 266287972389 got 266287972393 for osd.8 2024-04-23T10:25:46.892 DEBUG:teuthology.parallel:result is None 2024-04-23T10:25:46.992 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 23 10:25:46 smithi119 ceph-mon[32608]: from='client.? 172.21.15.114:0/1554184828' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 1}]: dispatch 2024-04-23T10:25:46.992 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 23 10:25:46 smithi119 ceph-mon[32608]: from='mgr.14152 172.21.15.114:0/1529706138' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.e", "id": [2, 4]}]: dispatch 2024-04-23T10:25:46.992 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 23 10:25:46 smithi119 ceph-mon[32608]: from='mgr.14152 172.21.15.114:0/1529706138' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.1d", "id": [8, 11]}]: dispatch 2024-04-23T10:25:46.992 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 23 10:25:46 smithi119 ceph-mon[32608]: from='mgr.14152 172.21.15.114:0/1529706138' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.e", "id": [5, 9]}]: dispatch 2024-04-23T10:25:46.992 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 23 10:25:46 smithi119 ceph-mon[32608]: from='mgr.14152 172.21.15.114:0/1529706138' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.12", "id": [3, 9]}]: dispatch 2024-04-23T10:25:46.992 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 23 10:25:46 smithi119 ceph-mon[32608]: from='mgr.14152 172.21.15.114:0/1529706138' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.17", "id": [4, 6]}]: dispatch 2024-04-23T10:25:46.992 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 23 10:25:46 smithi119 ceph-mon[32608]: from='mgr.14152 172.21.15.114:0/1529706138' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.23", "id": [4, 7]}]: dispatch 2024-04-23T10:25:46.992 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 23 10:25:46 smithi119 ceph-mon[32608]: from='mgr.14152 172.21.15.114:0/1529706138' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.27", "id": [4, 6]}]: dispatch 2024-04-23T10:25:46.992 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 23 10:25:46 smithi119 ceph-mon[32608]: from='mgr.14152 172.21.15.114:0/1529706138' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.31", "id": [4, 8]}]: dispatch 2024-04-23T10:25:46.992 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 23 10:25:46 smithi119 ceph-mon[32608]: from='mgr.14152 172.21.15.114:0/1529706138' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.38", "id": [3, 9]}]: dispatch 2024-04-23T10:25:46.993 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 23 10:25:46 smithi119 ceph-mon[32608]: from='mgr.14152 172.21.15.114:0/1529706138' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.39", "id": [5, 8]}]: dispatch 2024-04-23T10:25:46.993 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 23 10:25:46 smithi119 ceph-mon[32608]: pgmap v317: 97 pgs: 97 active+clean; 641 KiB data, 370 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-23T10:25:47.274 INFO:teuthology.orchestra.run.smithi114.stderr:Inferring config /var/lib/ceph/aa240572-0159-11ef-bc93-c7b262605968/mon.a/config 2024-04-23T10:25:47.301 INFO:tasks.cephadm.ceph_manager.ceph:need seq 34359738444 got 34359738448 for osd.0 2024-04-23T10:25:47.301 DEBUG:teuthology.parallel:result is None 2024-04-23T10:25:47.828 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 23 10:25:47 smithi204 ceph-mon[32649]: from='mgr.14152 172.21.15.114:0/1529706138' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.e", "id": [2, 4]}]': finished 2024-04-23T10:25:47.828 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 23 10:25:47 smithi204 ceph-mon[32649]: from='mgr.14152 172.21.15.114:0/1529706138' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.1d", "id": [8, 11]}]': finished 2024-04-23T10:25:47.828 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 23 10:25:47 smithi204 ceph-mon[32649]: from='mgr.14152 172.21.15.114:0/1529706138' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.e", "id": [5, 9]}]': finished 2024-04-23T10:25:47.828 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 23 10:25:47 smithi204 ceph-mon[32649]: from='mgr.14152 172.21.15.114:0/1529706138' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.12", "id": [3, 9]}]': finished 2024-04-23T10:25:47.828 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 23 10:25:47 smithi204 ceph-mon[32649]: from='mgr.14152 172.21.15.114:0/1529706138' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.17", "id": [4, 6]}]': finished 2024-04-23T10:25:47.828 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 23 10:25:47 smithi204 ceph-mon[32649]: from='mgr.14152 172.21.15.114:0/1529706138' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.23", "id": [4, 7]}]': finished 2024-04-23T10:25:47.828 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 23 10:25:47 smithi204 ceph-mon[32649]: from='mgr.14152 172.21.15.114:0/1529706138' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.27", "id": [4, 6]}]': finished 2024-04-23T10:25:47.828 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 23 10:25:47 smithi204 ceph-mon[32649]: from='mgr.14152 172.21.15.114:0/1529706138' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.31", "id": [4, 8]}]': finished 2024-04-23T10:25:47.828 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 23 10:25:47 smithi204 ceph-mon[32649]: from='mgr.14152 172.21.15.114:0/1529706138' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.38", "id": [3, 9]}]': finished 2024-04-23T10:25:47.828 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 23 10:25:47 smithi204 ceph-mon[32649]: from='mgr.14152 172.21.15.114:0/1529706138' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.39", "id": [5, 8]}]': finished 2024-04-23T10:25:47.828 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 23 10:25:47 smithi204 ceph-mon[32649]: osdmap e88: 12 total, 12 up, 12 in 2024-04-23T10:25:47.832 INFO:journalctl@ceph.mon.a.smithi114.stdout:Apr 23 10:25:47 smithi114 ceph-mon[29255]: from='mgr.14152 172.21.15.114:0/1529706138' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.e", "id": [2, 4]}]': finished 2024-04-23T10:25:47.832 INFO:journalctl@ceph.mon.a.smithi114.stdout:Apr 23 10:25:47 smithi114 ceph-mon[29255]: from='mgr.14152 172.21.15.114:0/1529706138' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.1d", "id": [8, 11]}]': finished 2024-04-23T10:25:47.832 INFO:journalctl@ceph.mon.a.smithi114.stdout:Apr 23 10:25:47 smithi114 ceph-mon[29255]: from='mgr.14152 172.21.15.114:0/1529706138' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.e", "id": [5, 9]}]': finished 2024-04-23T10:25:47.832 INFO:journalctl@ceph.mon.a.smithi114.stdout:Apr 23 10:25:47 smithi114 ceph-mon[29255]: from='mgr.14152 172.21.15.114:0/1529706138' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.12", "id": [3, 9]}]': finished 2024-04-23T10:25:47.832 INFO:journalctl@ceph.mon.a.smithi114.stdout:Apr 23 10:25:47 smithi114 ceph-mon[29255]: from='mgr.14152 172.21.15.114:0/1529706138' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.17", "id": [4, 6]}]': finished 2024-04-23T10:25:47.832 INFO:journalctl@ceph.mon.a.smithi114.stdout:Apr 23 10:25:47 smithi114 ceph-mon[29255]: from='mgr.14152 172.21.15.114:0/1529706138' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.23", "id": [4, 7]}]': finished 2024-04-23T10:25:47.832 INFO:journalctl@ceph.mon.a.smithi114.stdout:Apr 23 10:25:47 smithi114 ceph-mon[29255]: from='mgr.14152 172.21.15.114:0/1529706138' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.27", "id": [4, 6]}]': finished 2024-04-23T10:25:47.832 INFO:journalctl@ceph.mon.a.smithi114.stdout:Apr 23 10:25:47 smithi114 ceph-mon[29255]: from='mgr.14152 172.21.15.114:0/1529706138' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.31", "id": [4, 8]}]': finished 2024-04-23T10:25:47.832 INFO:journalctl@ceph.mon.a.smithi114.stdout:Apr 23 10:25:47 smithi114 ceph-mon[29255]: from='mgr.14152 172.21.15.114:0/1529706138' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.38", "id": [3, 9]}]': finished 2024-04-23T10:25:47.832 INFO:journalctl@ceph.mon.a.smithi114.stdout:Apr 23 10:25:47 smithi114 ceph-mon[29255]: from='mgr.14152 172.21.15.114:0/1529706138' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.39", "id": [5, 8]}]': finished 2024-04-23T10:25:47.832 INFO:journalctl@ceph.mon.a.smithi114.stdout:Apr 23 10:25:47 smithi114 ceph-mon[29255]: osdmap e88: 12 total, 12 up, 12 in 2024-04-23T10:25:47.992 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 23 10:25:47 smithi119 ceph-mon[32608]: from='mgr.14152 172.21.15.114:0/1529706138' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.e", "id": [2, 4]}]': finished 2024-04-23T10:25:47.992 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 23 10:25:47 smithi119 ceph-mon[32608]: from='mgr.14152 172.21.15.114:0/1529706138' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.1d", "id": [8, 11]}]': finished 2024-04-23T10:25:47.992 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 23 10:25:47 smithi119 ceph-mon[32608]: from='mgr.14152 172.21.15.114:0/1529706138' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.e", "id": [5, 9]}]': finished 2024-04-23T10:25:47.992 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 23 10:25:47 smithi119 ceph-mon[32608]: from='mgr.14152 172.21.15.114:0/1529706138' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.12", "id": [3, 9]}]': finished 2024-04-23T10:25:47.992 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 23 10:25:47 smithi119 ceph-mon[32608]: from='mgr.14152 172.21.15.114:0/1529706138' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.17", "id": [4, 6]}]': finished 2024-04-23T10:25:47.992 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 23 10:25:47 smithi119 ceph-mon[32608]: from='mgr.14152 172.21.15.114:0/1529706138' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.23", "id": [4, 7]}]': finished 2024-04-23T10:25:47.992 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 23 10:25:47 smithi119 ceph-mon[32608]: from='mgr.14152 172.21.15.114:0/1529706138' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.27", "id": [4, 6]}]': finished 2024-04-23T10:25:47.992 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 23 10:25:47 smithi119 ceph-mon[32608]: from='mgr.14152 172.21.15.114:0/1529706138' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.31", "id": [4, 8]}]': finished 2024-04-23T10:25:47.992 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 23 10:25:47 smithi119 ceph-mon[32608]: from='mgr.14152 172.21.15.114:0/1529706138' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.38", "id": [3, 9]}]': finished 2024-04-23T10:25:47.992 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 23 10:25:47 smithi119 ceph-mon[32608]: from='mgr.14152 172.21.15.114:0/1529706138' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.39", "id": [5, 8]}]': finished 2024-04-23T10:25:47.992 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 23 10:25:47 smithi119 ceph-mon[32608]: osdmap e88: 12 total, 12 up, 12 in 2024-04-23T10:25:48.514 INFO:tasks.cephadm.ceph_manager.ceph:need seq 55834574917 got 55834574923 for osd.1 2024-04-23T10:25:48.514 DEBUG:teuthology.parallel:result is None 2024-04-23T10:25:48.754 INFO:journalctl@ceph.mon.a.smithi114.stdout:Apr 23 10:25:48 smithi114 ceph-mon[29255]: osdmap e89: 12 total, 12 up, 12 in 2024-04-23T10:25:48.754 INFO:journalctl@ceph.mon.a.smithi114.stdout:Apr 23 10:25:48 smithi114 ceph-mon[29255]: from='client.? 172.21.15.114:0/3856192211' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 3}]: dispatch 2024-04-23T10:25:48.754 INFO:journalctl@ceph.mon.a.smithi114.stdout:Apr 23 10:25:48 smithi114 ceph-mon[29255]: pgmap v320: 97 pgs: 5 peering, 92 active+clean; 641 KiB data, 370 MiB used, 1.0 TiB / 1.0 TiB avail; 3.1 KiB/s rd, 6 op/s 2024-04-23T10:25:48.992 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 23 10:25:48 smithi119 ceph-mon[32608]: osdmap e89: 12 total, 12 up, 12 in 2024-04-23T10:25:48.992 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 23 10:25:48 smithi119 ceph-mon[32608]: from='client.? 172.21.15.114:0/3856192211' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 3}]: dispatch 2024-04-23T10:25:48.992 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 23 10:25:48 smithi119 ceph-mon[32608]: pgmap v320: 97 pgs: 5 peering, 92 active+clean; 641 KiB data, 370 MiB used, 1.0 TiB / 1.0 TiB avail; 3.1 KiB/s rd, 6 op/s 2024-04-23T10:25:49.078 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 23 10:25:48 smithi204 ceph-mon[32649]: osdmap e89: 12 total, 12 up, 12 in 2024-04-23T10:25:49.078 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 23 10:25:48 smithi204 ceph-mon[32649]: from='client.? 172.21.15.114:0/3856192211' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 3}]: dispatch 2024-04-23T10:25:49.078 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 23 10:25:48 smithi204 ceph-mon[32649]: pgmap v320: 97 pgs: 5 peering, 92 active+clean; 641 KiB data, 370 MiB used, 1.0 TiB / 1.0 TiB avail; 3.1 KiB/s rd, 6 op/s 2024-04-23T10:25:49.653 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:25:47.736+0000 7f924b9eb700 1 -- 172.21.15.114:0/1576750055 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f9244108a50 msgr2=0x7f9244108e30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T10:25:49.653 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:25:47.736+0000 7f924b9eb700 1 --2- 172.21.15.114:0/1576750055 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f9244108a50 0x7f9244108e30 secure :-1 s=READY pgs=288 cs=0 l=1 rev1=1 crypto rx=0x7f9240009990 tx=0x7f9240009ca0 comp rx=0 tx=0).stop 2024-04-23T10:25:49.653 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:25:47.737+0000 7f924b9eb700 1 -- 172.21.15.114:0/1576750055 shutdown_connections 2024-04-23T10:25:49.653 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:25:47.737+0000 7f924b9eb700 1 --2- 172.21.15.114:0/1576750055 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f9244103450 0x7f924410f8b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:25:49.653 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:25:47.737+0000 7f924b9eb700 1 --2- 172.21.15.114:0/1576750055 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f9244108a50 0x7f9244108e30 unknown :-1 s=CLOSED pgs=288 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:25:49.653 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:25:47.737+0000 7f924b9eb700 1 --2- 172.21.15.114:0/1576750055 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f9244102a90 0x7f9244102f10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:25:49.653 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:25:47.737+0000 7f924b9eb700 1 -- 172.21.15.114:0/1576750055 >> 172.21.15.114:0/1576750055 conn(0x7f9244079ba0 msgr2=0x7f9244079fb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T10:25:49.653 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:25:47.737+0000 7f924b9eb700 1 -- 172.21.15.114:0/1576750055 shutdown_connections 2024-04-23T10:25:49.653 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:25:47.737+0000 7f924b9eb700 1 -- 172.21.15.114:0/1576750055 wait complete. 2024-04-23T10:25:49.654 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:25:47.737+0000 7f924b9eb700 1 Processor -- start 2024-04-23T10:25:49.654 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:25:47.737+0000 7f924b9eb700 1 -- start start 2024-04-23T10:25:49.654 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:25:47.737+0000 7f924b9eb700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f9244102a90 0x7f92441990f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T10:25:49.654 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:25:47.737+0000 7f924b9eb700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f9244103450 0x7f9244199630 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T10:25:49.654 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:25:47.737+0000 7f924b9eb700 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f9244108a50 0x7f924419d970 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T10:25:49.654 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:25:47.737+0000 7f924b9eb700 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x7f9244112c60 con 0x7f9244103450 2024-04-23T10:25:49.654 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:25:47.737+0000 7f924b9eb700 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 -- 0x7f9244112ae0 con 0x7f9244102a90 2024-04-23T10:25:49.654 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:25:47.737+0000 7f924b9eb700 1 -- --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_getmap magic: 0 -- 0x7f9244112de0 con 0x7f9244108a50 2024-04-23T10:25:49.654 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:25:47.738+0000 7f9248f86700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f9244103450 0x7f9244199630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T10:25:49.654 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:25:47.738+0000 7f9248f86700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f9244103450 0x7f9244199630 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.114:58436/0 (socket says 172.21.15.114:58436) 2024-04-23T10:25:49.654 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:25:47.738+0000 7f9248f86700 1 -- 172.21.15.114:0/3856192211 learned_addr learned my addr 172.21.15.114:0/3856192211 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-04-23T10:25:49.654 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:25:47.738+0000 7f9249787700 1 --2- 172.21.15.114:0/3856192211 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f9244102a90 0x7f92441990f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T10:25:49.654 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:25:47.738+0000 7f9249f88700 1 --2- 172.21.15.114:0/3856192211 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f9244108a50 0x7f924419d970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T10:25:49.655 INFO:teuthology.orchestra.run.smithi114.stdout:107374182464 2024-04-23T10:25:49.655 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:25:47.738+0000 7f9248f86700 1 -- 172.21.15.114:0/3856192211 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f9244108a50 msgr2=0x7f924419d970 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T10:25:49.655 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:25:47.738+0000 7f9248f86700 1 --2- 172.21.15.114:0/3856192211 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f9244108a50 0x7f924419d970 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:25:49.655 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:25:47.738+0000 7f9248f86700 1 -- 172.21.15.114:0/3856192211 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f9244102a90 msgr2=0x7f92441990f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T10:25:49.655 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:25:47.738+0000 7f9248f86700 1 --2- 172.21.15.114:0/3856192211 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f9244102a90 0x7f92441990f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:25:49.655 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:25:47.738+0000 7f9248f86700 1 -- 172.21.15.114:0/3856192211 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f924419e0b0 con 0x7f9244103450 2024-04-23T10:25:49.655 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:25:47.738+0000 7f9249f88700 1 --2- 172.21.15.114:0/3856192211 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f9244108a50 0x7f924419d970 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-23T10:25:49.655 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:25:47.738+0000 7f9249787700 1 --2- 172.21.15.114:0/3856192211 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f9244102a90 0x7f92441990f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-23T10:25:49.655 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:25:47.738+0000 7f9248f86700 1 --2- 172.21.15.114:0/3856192211 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f9244103450 0x7f9244199630 secure :-1 s=READY pgs=289 cs=0 l=1 rev1=1 crypto rx=0x7f923400c9b0 tx=0x7f923400cd70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T10:25:49.656 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:25:47.738+0000 7f923a7fc700 1 -- 172.21.15.114:0/3856192211 <== mon.0 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9234007b00 con 0x7f9244103450 2024-04-23T10:25:49.656 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:25:47.738+0000 7f924b9eb700 1 -- 172.21.15.114:0/3856192211 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f924419e3a0 con 0x7f9244103450 2024-04-23T10:25:49.656 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:25:47.738+0000 7f923a7fc700 1 -- 172.21.15.114:0/3856192211 <== mon.0 v2:172.21.15.114:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f9234007ca0 con 0x7f9244103450 2024-04-23T10:25:49.656 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:25:47.738+0000 7f923a7fc700 1 -- 172.21.15.114:0/3856192211 <== mon.0 v2:172.21.15.114:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9234018730 con 0x7f9244103450 2024-04-23T10:25:49.656 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:25:47.738+0000 7f924b9eb700 1 -- 172.21.15.114:0/3856192211 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f92441a5c20 con 0x7f9244103450 2024-04-23T10:25:49.656 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:25:47.739+0000 7f924b9eb700 1 -- 172.21.15.114:0/3856192211 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f9244069730 con 0x7f9244103450 2024-04-23T10:25:49.656 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:25:47.741+0000 7f923a7fc700 1 -- 172.21.15.114:0/3856192211 <== mon.0 v2:172.21.15.114:3300/0 4 ==== mgrmap(e 17) ==== 138698+0+0 (secure 0 0 0) 0x7f923400f450 con 0x7f9244103450 2024-04-23T10:25:49.656 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:25:47.741+0000 7f923a7fc700 1 --2- 172.21.15.114:0/3856192211 >> [v2:172.21.15.114:6800/465333808,v1:172.21.15.114:6801/465333808] conn(0x7f92300a3cd0 0x7f92300a6190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T10:25:49.656 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:25:47.741+0000 7f9249787700 1 --2- 172.21.15.114:0/3856192211 >> [v2:172.21.15.114:6800/465333808,v1:172.21.15.114:6801/465333808] conn(0x7f92300a3cd0 0x7f92300a6190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T10:25:49.656 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:25:47.741+0000 7f923a7fc700 1 -- 172.21.15.114:0/3856192211 <== mon.0 v2:172.21.15.114:3300/0 5 ==== osd_map(89..89 src has 1..89) ==== 8269+0+0 (secure 0 0 0) 0x7f92340d06b0 con 0x7f9244103450 2024-04-23T10:25:49.656 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:25:47.742+0000 7f9249787700 1 --2- 172.21.15.114:0/3856192211 >> [v2:172.21.15.114:6800/465333808,v1:172.21.15.114:6801/465333808] conn(0x7f92300a3cd0 0x7f92300a6190 secure :-1 s=READY pgs=203 cs=0 l=1 rev1=1 crypto rx=0x7f924000b5c0 tx=0x7f9240005fb0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T10:25:49.656 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:25:47.746+0000 7f923a7fc700 1 -- 172.21.15.114:0/3856192211 <== mon.0 v2:172.21.15.114:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f9234014030 con 0x7f9244103450 2024-04-23T10:25:49.656 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:25:47.952+0000 7f924b9eb700 1 -- 172.21.15.114:0/3856192211 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 3} v 0) -- 0x7f924402df80 con 0x7f9244103450 2024-04-23T10:25:49.656 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:25:47.952+0000 7f923a7fc700 1 -- 172.21.15.114:0/3856192211 <== mon.0 v2:172.21.15.114:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 3}]=0 v0) ==== 74+0+13 (secure 0 0 0) 0x7f923409ceb0 con 0x7f9244103450 2024-04-23T10:25:49.656 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:25:47.954+0000 7f924b9eb700 1 -- 172.21.15.114:0/3856192211 >> [v2:172.21.15.114:6800/465333808,v1:172.21.15.114:6801/465333808] conn(0x7f92300a3cd0 msgr2=0x7f92300a6190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T10:25:49.657 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:25:47.954+0000 7f924b9eb700 1 --2- 172.21.15.114:0/3856192211 >> [v2:172.21.15.114:6800/465333808,v1:172.21.15.114:6801/465333808] conn(0x7f92300a3cd0 0x7f92300a6190 secure :-1 s=READY pgs=203 cs=0 l=1 rev1=1 crypto rx=0x7f924000b5c0 tx=0x7f9240005fb0 comp rx=0 tx=0).stop 2024-04-23T10:25:49.657 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:25:47.954+0000 7f924b9eb700 1 -- 172.21.15.114:0/3856192211 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f9244103450 msgr2=0x7f9244199630 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T10:25:49.657 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:25:47.954+0000 7f924b9eb700 1 --2- 172.21.15.114:0/3856192211 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f9244103450 0x7f9244199630 secure :-1 s=READY pgs=289 cs=0 l=1 rev1=1 crypto rx=0x7f923400c9b0 tx=0x7f923400cd70 comp rx=0 tx=0).stop 2024-04-23T10:25:49.657 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:25:47.955+0000 7f924b9eb700 1 -- 172.21.15.114:0/3856192211 shutdown_connections 2024-04-23T10:25:49.657 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:25:47.955+0000 7f924b9eb700 1 --2- 172.21.15.114:0/3856192211 >> [v2:172.21.15.114:6800/465333808,v1:172.21.15.114:6801/465333808] conn(0x7f92300a3cd0 0x7f92300a6190 unknown :-1 s=CLOSED pgs=203 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:25:49.657 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:25:47.955+0000 7f924b9eb700 1 --2- 172.21.15.114:0/3856192211 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f9244102a90 0x7f92441990f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:25:49.657 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:25:47.955+0000 7f924b9eb700 1 --2- 172.21.15.114:0/3856192211 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f9244108a50 0x7f924419d970 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:25:49.657 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:25:47.955+0000 7f924b9eb700 1 --2- 172.21.15.114:0/3856192211 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f9244103450 0x7f9244199630 unknown :-1 s=CLOSED pgs=289 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:25:49.657 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:25:47.955+0000 7f924b9eb700 1 -- 172.21.15.114:0/3856192211 >> 172.21.15.114:0/3856192211 conn(0x7f9244079ba0 msgr2=0x7f92441017f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T10:25:49.657 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:25:47.955+0000 7f924b9eb700 1 -- 172.21.15.114:0/3856192211 shutdown_connections 2024-04-23T10:25:49.657 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:25:47.955+0000 7f924b9eb700 1 -- 172.21.15.114:0/3856192211 wait complete. 2024-04-23T10:25:49.939 INFO:journalctl@ceph.mon.a.smithi114.stdout:Apr 23 10:25:49 smithi114 ceph-mon[29255]: Health check failed: Reduced data availability: 5 pgs peering (PG_AVAILABILITY) 2024-04-23T10:25:49.992 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 23 10:25:49 smithi119 ceph-mon[32608]: Health check failed: Reduced data availability: 5 pgs peering (PG_AVAILABILITY) 2024-04-23T10:25:50.078 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 23 10:25:49 smithi204 ceph-mon[32649]: Health check failed: Reduced data availability: 5 pgs peering (PG_AVAILABILITY) 2024-04-23T10:25:50.277 INFO:tasks.cephadm.ceph_manager.ceph:need seq 107374182458 got 107374182464 for osd.3 2024-04-23T10:25:50.277 DEBUG:teuthology.parallel:result is None 2024-04-23T10:25:50.911 INFO:journalctl@ceph.mon.a.smithi114.stdout:Apr 23 10:25:50 smithi114 ceph-mon[29255]: pgmap v321: 97 pgs: 5 peering, 92 active+clean; 641 KiB data, 370 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s; 0 B/s, 0 objects/s recovering 2024-04-23T10:25:50.992 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 23 10:25:50 smithi119 ceph-mon[32608]: pgmap v321: 97 pgs: 5 peering, 92 active+clean; 641 KiB data, 370 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s; 0 B/s, 0 objects/s recovering 2024-04-23T10:25:51.078 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 23 10:25:50 smithi204 ceph-mon[32649]: pgmap v321: 97 pgs: 5 peering, 92 active+clean; 641 KiB data, 370 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s; 0 B/s, 0 objects/s recovering 2024-04-23T10:25:51.788 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:25:51.788+0000 7f4c1cb27700 1 -- 172.21.15.114:0/359460962 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f4c18103450 msgr2=0x7f4c1810f8b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T10:25:51.788 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:25:51.788+0000 7f4c1cb27700 1 --2- 172.21.15.114:0/359460962 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f4c18103450 0x7f4c1810f8b0 secure :-1 s=READY pgs=290 cs=0 l=1 rev1=1 crypto rx=0x7f4c1000b0a0 tx=0x7f4c1000b3b0 comp rx=0 tx=0).stop 2024-04-23T10:25:51.788 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:25:51.788+0000 7f4c1cb27700 1 -- 172.21.15.114:0/359460962 shutdown_connections 2024-04-23T10:25:51.788 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:25:51.788+0000 7f4c1cb27700 1 --2- 172.21.15.114:0/359460962 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f4c18103450 0x7f4c1810f8b0 unknown :-1 s=CLOSED pgs=290 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:25:51.788 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:25:51.788+0000 7f4c1cb27700 1 --2- 172.21.15.114:0/359460962 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f4c18108a50 0x7f4c18108e30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:25:51.788 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:25:51.789+0000 7f4c1cb27700 1 --2- 172.21.15.114:0/359460962 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f4c18102a90 0x7f4c18102f10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:25:51.788 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:25:51.789+0000 7f4c1cb27700 1 -- 172.21.15.114:0/359460962 >> 172.21.15.114:0/359460962 conn(0x7f4c18079ba0 msgr2=0x7f4c18079fb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T10:25:51.788 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:25:51.789+0000 7f4c1cb27700 1 -- 172.21.15.114:0/359460962 shutdown_connections 2024-04-23T10:25:51.788 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:25:51.789+0000 7f4c1cb27700 1 -- 172.21.15.114:0/359460962 wait complete. 2024-04-23T10:25:51.790 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:25:51.789+0000 7f4c1cb27700 1 Processor -- start 2024-04-23T10:25:51.790 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:25:51.789+0000 7f4c1cb27700 1 -- start start 2024-04-23T10:25:51.790 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:25:51.789+0000 7f4c1cb27700 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f4c18102a90 0x7f4c181990b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T10:25:51.790 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:25:51.790+0000 7f4c1cb27700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f4c18103450 0x7f4c181995f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T10:25:51.790 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:25:51.790+0000 7f4c1cb27700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f4c18108a50 0x7f4c1819d980 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T10:25:51.790 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:25:51.790+0000 7f4c1cb27700 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x7f4c18112c00 con 0x7f4c18108a50 2024-04-23T10:25:51.791 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:25:51.790+0000 7f4c1cb27700 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 -- 0x7f4c18112a80 con 0x7f4c18103450 2024-04-23T10:25:51.791 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:25:51.790+0000 7f4c1cb27700 1 -- --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_getmap magic: 0 -- 0x7f4c18112d80 con 0x7f4c18102a90 2024-04-23T10:25:51.791 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:25:51.790+0000 7f4c16d9d700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f4c18108a50 0x7f4c1819d980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T10:25:51.791 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:25:51.790+0000 7f4c16d9d700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f4c18108a50 0x7f4c1819d980 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.114:58444/0 (socket says 172.21.15.114:58444) 2024-04-23T10:25:51.791 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:25:51.790+0000 7f4c16d9d700 1 -- 172.21.15.114:0/32258099 learned_addr learned my addr 172.21.15.114:0/32258099 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-04-23T10:25:51.791 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:25:51.790+0000 7f4c1659c700 1 --2- 172.21.15.114:0/32258099 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f4c18102a90 0x7f4c181990b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T10:25:51.791 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:25:51.790+0000 7f4c15d9b700 1 --2- 172.21.15.114:0/32258099 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f4c18103450 0x7f4c181995f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T10:25:51.791 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:25:51.790+0000 7f4c16d9d700 1 -- 172.21.15.114:0/32258099 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f4c18102a90 msgr2=0x7f4c181990b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T10:25:51.791 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:25:51.790+0000 7f4c16d9d700 1 --2- 172.21.15.114:0/32258099 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f4c18102a90 0x7f4c181990b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:25:51.791 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:25:51.790+0000 7f4c16d9d700 1 -- 172.21.15.114:0/32258099 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f4c18103450 msgr2=0x7f4c181995f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T10:25:51.791 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:25:51.790+0000 7f4c16d9d700 1 --2- 172.21.15.114:0/32258099 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f4c18103450 0x7f4c181995f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:25:51.791 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:25:51.790+0000 7f4c16d9d700 1 -- 172.21.15.114:0/32258099 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f4c1819e0c0 con 0x7f4c18108a50 2024-04-23T10:25:51.791 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:25:51.790+0000 7f4c15d9b700 1 --2- 172.21.15.114:0/32258099 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f4c18103450 0x7f4c181995f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-23T10:25:51.792 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:25:51.790+0000 7f4c16d9d700 1 --2- 172.21.15.114:0/32258099 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f4c18108a50 0x7f4c1819d980 secure :-1 s=READY pgs=291 cs=0 l=1 rev1=1 crypto rx=0x7f4c1000b830 tx=0x7f4c10004ce0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T10:25:51.792 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:25:51.790+0000 7f4c077fe700 1 -- 172.21.15.114:0/32258099 <== mon.0 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4c10021070 con 0x7f4c18108a50 2024-04-23T10:25:51.792 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:25:51.790+0000 7f4c077fe700 1 -- 172.21.15.114:0/32258099 <== mon.0 v2:172.21.15.114:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f4c10007e30 con 0x7f4c18108a50 2024-04-23T10:25:51.792 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:25:51.790+0000 7f4c1cb27700 1 -- 172.21.15.114:0/32258099 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f4c1819e350 con 0x7f4c18108a50 2024-04-23T10:25:51.792 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:25:51.790+0000 7f4c077fe700 1 -- 172.21.15.114:0/32258099 <== mon.0 v2:172.21.15.114:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4c100047d0 con 0x7f4c18108a50 2024-04-23T10:25:51.792 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:25:51.790+0000 7f4c1cb27700 1 -- 172.21.15.114:0/32258099 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f4c181a5c30 con 0x7f4c18108a50 2024-04-23T10:25:51.792 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:25:51.791+0000 7f4c1cb27700 1 -- 172.21.15.114:0/32258099 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f4c18069730 con 0x7f4c18108a50 2024-04-23T10:25:51.793 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:25:51.792+0000 7f4c077fe700 1 -- 172.21.15.114:0/32258099 <== mon.0 v2:172.21.15.114:3300/0 4 ==== mgrmap(e 17) ==== 138698+0+0 (secure 0 0 0) 0x7f4c10007980 con 0x7f4c18108a50 2024-04-23T10:25:51.793 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:25:51.793+0000 7f4c077fe700 1 --2- 172.21.15.114:0/32258099 >> [v2:172.21.15.114:6800/465333808,v1:172.21.15.114:6801/465333808] conn(0x7f4c000a3cd0 0x7f4c000a6190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T10:25:51.793 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:25:51.793+0000 7f4c1659c700 1 --2- 172.21.15.114:0/32258099 >> [v2:172.21.15.114:6800/465333808,v1:172.21.15.114:6801/465333808] conn(0x7f4c000a3cd0 0x7f4c000a6190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T10:25:51.793 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:25:51.793+0000 7f4c077fe700 1 -- 172.21.15.114:0/32258099 <== mon.0 v2:172.21.15.114:3300/0 5 ==== osd_map(89..89 src has 1..89) ==== 8269+0+0 (secure 0 0 0) 0x7f4c1000f020 con 0x7f4c18108a50 2024-04-23T10:25:51.793 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:25:51.794+0000 7f4c1659c700 1 --2- 172.21.15.114:0/32258099 >> [v2:172.21.15.114:6800/465333808,v1:172.21.15.114:6801/465333808] conn(0x7f4c000a3cd0 0x7f4c000a6190 secure :-1 s=READY pgs=204 cs=0 l=1 rev1=1 crypto rx=0x7f4c080045b0 tx=0x7f4c0800b3c0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T10:25:51.797 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:25:51.798+0000 7f4c077fe700 1 -- 172.21.15.114:0/32258099 <== mon.0 v2:172.21.15.114:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f4c1002a020 con 0x7f4c18108a50 2024-04-23T10:25:52.002 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:25:52.003+0000 7f4c1cb27700 1 -- 172.21.15.114:0/32258099 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 11} v 0) -- 0x7f4c1802df80 con 0x7f4c18108a50 2024-04-23T10:25:52.002 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:25:52.003+0000 7f4c077fe700 1 -- 172.21.15.114:0/32258099 <== mon.0 v2:172.21.15.114:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 11}]=0 v0) ==== 75+0+13 (secure 0 0 0) 0x7f4c1009e400 con 0x7f4c18108a50 2024-04-23T10:25:52.002 INFO:teuthology.orchestra.run.smithi114.stdout:352187318299 2024-04-23T10:25:52.005 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:25:52.006+0000 7f4c1cb27700 1 -- 172.21.15.114:0/32258099 >> [v2:172.21.15.114:6800/465333808,v1:172.21.15.114:6801/465333808] conn(0x7f4c000a3cd0 msgr2=0x7f4c000a6190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T10:25:52.005 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:25:52.006+0000 7f4c1cb27700 1 --2- 172.21.15.114:0/32258099 >> [v2:172.21.15.114:6800/465333808,v1:172.21.15.114:6801/465333808] conn(0x7f4c000a3cd0 0x7f4c000a6190 secure :-1 s=READY pgs=204 cs=0 l=1 rev1=1 crypto rx=0x7f4c080045b0 tx=0x7f4c0800b3c0 comp rx=0 tx=0).stop 2024-04-23T10:25:52.005 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:25:52.006+0000 7f4c1cb27700 1 -- 172.21.15.114:0/32258099 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f4c18108a50 msgr2=0x7f4c1819d980 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T10:25:52.005 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:25:52.006+0000 7f4c1cb27700 1 --2- 172.21.15.114:0/32258099 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f4c18108a50 0x7f4c1819d980 secure :-1 s=READY pgs=291 cs=0 l=1 rev1=1 crypto rx=0x7f4c1000b830 tx=0x7f4c10004ce0 comp rx=0 tx=0).stop 2024-04-23T10:25:52.005 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:25:52.006+0000 7f4c1cb27700 1 -- 172.21.15.114:0/32258099 shutdown_connections 2024-04-23T10:25:52.006 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:25:52.006+0000 7f4c1cb27700 1 --2- 172.21.15.114:0/32258099 >> [v2:172.21.15.114:6800/465333808,v1:172.21.15.114:6801/465333808] conn(0x7f4c000a3cd0 0x7f4c000a6190 unknown :-1 s=CLOSED pgs=204 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:25:52.006 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:25:52.006+0000 7f4c1cb27700 1 --2- 172.21.15.114:0/32258099 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f4c18108a50 0x7f4c1819d980 unknown :-1 s=CLOSED pgs=291 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:25:52.006 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:25:52.006+0000 7f4c1cb27700 1 --2- 172.21.15.114:0/32258099 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f4c18102a90 0x7f4c181990b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:25:52.006 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:25:52.006+0000 7f4c1cb27700 1 --2- 172.21.15.114:0/32258099 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f4c18103450 0x7f4c181995f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:25:52.006 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:25:52.006+0000 7f4c1cb27700 1 -- 172.21.15.114:0/32258099 >> 172.21.15.114:0/32258099 conn(0x7f4c18079ba0 msgr2=0x7f4c18101850 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T10:25:52.006 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:25:52.006+0000 7f4c1cb27700 1 -- 172.21.15.114:0/32258099 shutdown_connections 2024-04-23T10:25:52.006 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:25:52.006+0000 7f4c1cb27700 1 -- 172.21.15.114:0/32258099 wait complete. 2024-04-23T10:25:52.022 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:25:52.004+0000 7f2bd3ac8700 1 -- 172.21.15.114:0/1164079084 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f2bcc1034b0 msgr2=0x7f2bcc10f8e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T10:25:52.022 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:25:52.004+0000 7f2bd3ac8700 1 --2- 172.21.15.114:0/1164079084 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f2bcc1034b0 0x7f2bcc10f8e0 secure :-1 s=READY pgs=292 cs=0 l=1 rev1=1 crypto rx=0x7f2bc400b0a0 tx=0x7f2bc400b3b0 comp rx=0 tx=0).stop 2024-04-23T10:25:52.022 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:25:52.005+0000 7f2bd3ac8700 1 -- 172.21.15.114:0/1164079084 shutdown_connections 2024-04-23T10:25:52.022 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:25:52.005+0000 7f2bd3ac8700 1 --2- 172.21.15.114:0/1164079084 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f2bcc1034b0 0x7f2bcc10f8e0 unknown :-1 s=CLOSED pgs=292 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:25:52.023 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:25:52.005+0000 7f2bd3ac8700 1 --2- 172.21.15.114:0/1164079084 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f2bcc108a80 0x7f2bcc108e60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:25:52.023 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:25:52.005+0000 7f2bd3ac8700 1 --2- 172.21.15.114:0/1164079084 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f2bcc102af0 0x7f2bcc102f70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:25:52.023 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:25:52.005+0000 7f2bd3ac8700 1 -- 172.21.15.114:0/1164079084 >> 172.21.15.114:0/1164079084 conn(0x7f2bcc079b20 msgr2=0x7f2bcc079f30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T10:25:52.023 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:25:52.005+0000 7f2bd3ac8700 1 -- 172.21.15.114:0/1164079084 shutdown_connections 2024-04-23T10:25:52.023 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:25:52.005+0000 7f2bd3ac8700 1 -- 172.21.15.114:0/1164079084 wait complete. 2024-04-23T10:25:52.023 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:25:52.005+0000 7f2bd3ac8700 1 Processor -- start 2024-04-23T10:25:52.023 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:25:52.006+0000 7f2bd3ac8700 1 -- start start 2024-04-23T10:25:52.023 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:25:52.006+0000 7f2bd3ac8700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f2bcc102af0 0x7f2bcc074d40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T10:25:52.023 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:25:52.006+0000 7f2bd3ac8700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f2bcc1034b0 0x7f2bcc075280 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T10:25:52.023 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:25:52.006+0000 7f2bd3ac8700 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f2bcc108a80 0x7f2bcc0757c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T10:25:52.023 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:25:52.006+0000 7f2bd3ac8700 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x7f2bcc112c10 con 0x7f2bcc102af0 2024-04-23T10:25:52.023 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:25:52.006+0000 7f2bd3ac8700 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 -- 0x7f2bcc112a90 con 0x7f2bcc1034b0 2024-04-23T10:25:52.023 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:25:52.006+0000 7f2bd3ac8700 1 -- --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_getmap magic: 0 -- 0x7f2bcc112d90 con 0x7f2bcc108a80 2024-04-23T10:25:52.023 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:25:52.006+0000 7f2bd1864700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f2bcc102af0 0x7f2bcc074d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T10:25:52.023 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:25:52.006+0000 7f2bd1864700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f2bcc102af0 0x7f2bcc074d40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.114:58456/0 (socket says 172.21.15.114:58456) 2024-04-23T10:25:52.024 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:25:52.006+0000 7f2bd1864700 1 -- 172.21.15.114:0/79632769 learned_addr learned my addr 172.21.15.114:0/79632769 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-04-23T10:25:52.024 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:25:52.006+0000 7f2bd1063700 1 --2- 172.21.15.114:0/79632769 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f2bcc1034b0 0x7f2bcc075280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T10:25:52.024 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:25:52.006+0000 7f2bd2065700 1 --2- 172.21.15.114:0/79632769 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f2bcc108a80 0x7f2bcc0757c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T10:25:52.024 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:25:52.006+0000 7f2bd1864700 1 -- 172.21.15.114:0/79632769 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f2bcc108a80 msgr2=0x7f2bcc0757c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T10:25:52.024 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:25:52.006+0000 7f2bd1864700 1 --2- 172.21.15.114:0/79632769 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f2bcc108a80 0x7f2bcc0757c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:25:52.024 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:25:52.006+0000 7f2bd1864700 1 -- 172.21.15.114:0/79632769 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f2bcc1034b0 msgr2=0x7f2bcc075280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T10:25:52.024 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:25:52.006+0000 7f2bd1864700 1 --2- 172.21.15.114:0/79632769 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f2bcc1034b0 0x7f2bcc075280 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:25:52.024 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:25:52.006+0000 7f2bd1864700 1 -- 172.21.15.114:0/79632769 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f2bcc1a7550 con 0x7f2bcc102af0 2024-04-23T10:25:52.024 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:25:52.006+0000 7f2bd2065700 1 --2- 172.21.15.114:0/79632769 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f2bcc108a80 0x7f2bcc0757c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-23T10:25:52.024 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:25:52.007+0000 7f2bd1864700 1 --2- 172.21.15.114:0/79632769 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f2bcc102af0 0x7f2bcc074d40 secure :-1 s=READY pgs=293 cs=0 l=1 rev1=1 crypto rx=0x7f2bbc00c9b0 tx=0x7f2bbc00ccc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T10:25:52.024 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:25:52.007+0000 7f2bc2ffd700 1 -- 172.21.15.114:0/79632769 <== mon.0 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2bbc007b00 con 0x7f2bcc102af0 2024-04-23T10:25:52.024 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:25:52.007+0000 7f2bc2ffd700 1 -- 172.21.15.114:0/79632769 <== mon.0 v2:172.21.15.114:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f2bbc007ca0 con 0x7f2bcc102af0 2024-04-23T10:25:52.024 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:25:52.007+0000 7f2bd3ac8700 1 -- 172.21.15.114:0/79632769 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f2bcc1a7840 con 0x7f2bcc102af0 2024-04-23T10:25:52.024 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:25:52.007+0000 7f2bc2ffd700 1 -- 172.21.15.114:0/79632769 <== mon.0 v2:172.21.15.114:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2bbc018660 con 0x7f2bcc102af0 2024-04-23T10:25:52.024 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:25:52.007+0000 7f2bd3ac8700 1 -- 172.21.15.114:0/79632769 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f2bcc1a7d80 con 0x7f2bcc102af0 2024-04-23T10:25:52.025 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:25:52.008+0000 7f2bd3ac8700 1 -- 172.21.15.114:0/79632769 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f2bcc069730 con 0x7f2bcc102af0 2024-04-23T10:25:52.025 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:25:52.009+0000 7f2bc2ffd700 1 -- 172.21.15.114:0/79632769 <== mon.0 v2:172.21.15.114:3300/0 4 ==== mgrmap(e 17) ==== 138698+0+0 (secure 0 0 0) 0x7f2bbc00f450 con 0x7f2bcc102af0 2024-04-23T10:25:52.025 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:25:52.010+0000 7f2bc2ffd700 1 --2- 172.21.15.114:0/79632769 >> [v2:172.21.15.114:6800/465333808,v1:172.21.15.114:6801/465333808] conn(0x7f2bb80a3cd0 0x7f2bb80a6190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T10:25:52.025 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:25:52.010+0000 7f2bd1063700 1 --2- 172.21.15.114:0/79632769 >> [v2:172.21.15.114:6800/465333808,v1:172.21.15.114:6801/465333808] conn(0x7f2bb80a3cd0 0x7f2bb80a6190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T10:25:52.025 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:25:52.010+0000 7f2bc2ffd700 1 -- 172.21.15.114:0/79632769 <== mon.0 v2:172.21.15.114:3300/0 5 ==== osd_map(89..89 src has 1..89) ==== 8269+0+0 (secure 0 0 0) 0x7f2bbc0d07b0 con 0x7f2bcc102af0 2024-04-23T10:25:52.025 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:25:52.010+0000 7f2bd1063700 1 --2- 172.21.15.114:0/79632769 >> [v2:172.21.15.114:6800/465333808,v1:172.21.15.114:6801/465333808] conn(0x7f2bb80a3cd0 0x7f2bb80a6190 secure :-1 s=READY pgs=205 cs=0 l=1 rev1=1 crypto rx=0x7f2bcc07a190 tx=0x7f2bc800b410 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T10:25:52.025 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:25:52.017+0000 7f2bc2ffd700 1 -- 172.21.15.114:0/79632769 <== mon.0 v2:172.21.15.114:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f2bbc014030 con 0x7f2bcc102af0 2024-04-23T10:25:52.223 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:25:52.224+0000 7f2bd3ac8700 1 -- 172.21.15.114:0/79632769 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 5} v 0) -- 0x7f2bcc02df80 con 0x7f2bcc102af0 2024-04-23T10:25:52.223 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:25:52.224+0000 7f2bc2ffd700 1 -- 172.21.15.114:0/79632769 <== mon.0 v2:172.21.15.114:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 5}]=0 v0) ==== 74+0+13 (secure 0 0 0) 0x7f2bbc09d0a0 con 0x7f2bcc102af0 2024-04-23T10:25:52.223 INFO:teuthology.orchestra.run.smithi114.stdout:167503724602 2024-04-23T10:25:52.226 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:25:52.227+0000 7f2bd3ac8700 1 -- 172.21.15.114:0/79632769 >> [v2:172.21.15.114:6800/465333808,v1:172.21.15.114:6801/465333808] conn(0x7f2bb80a3cd0 msgr2=0x7f2bb80a6190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T10:25:52.226 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:25:52.227+0000 7f2bd3ac8700 1 --2- 172.21.15.114:0/79632769 >> [v2:172.21.15.114:6800/465333808,v1:172.21.15.114:6801/465333808] conn(0x7f2bb80a3cd0 0x7f2bb80a6190 secure :-1 s=READY pgs=205 cs=0 l=1 rev1=1 crypto rx=0x7f2bcc07a190 tx=0x7f2bc800b410 comp rx=0 tx=0).stop 2024-04-23T10:25:52.226 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:25:52.227+0000 7f2bd3ac8700 1 -- 172.21.15.114:0/79632769 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f2bcc102af0 msgr2=0x7f2bcc074d40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T10:25:52.226 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:25:52.227+0000 7f2bd3ac8700 1 --2- 172.21.15.114:0/79632769 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f2bcc102af0 0x7f2bcc074d40 secure :-1 s=READY pgs=293 cs=0 l=1 rev1=1 crypto rx=0x7f2bbc00c9b0 tx=0x7f2bbc00ccc0 comp rx=0 tx=0).stop 2024-04-23T10:25:52.226 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:25:52.227+0000 7f2bd3ac8700 1 -- 172.21.15.114:0/79632769 shutdown_connections 2024-04-23T10:25:52.226 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:25:52.227+0000 7f2bd3ac8700 1 --2- 172.21.15.114:0/79632769 >> [v2:172.21.15.114:6800/465333808,v1:172.21.15.114:6801/465333808] conn(0x7f2bb80a3cd0 0x7f2bb80a6190 unknown :-1 s=CLOSED pgs=205 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:25:52.226 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:25:52.227+0000 7f2bd3ac8700 1 --2- 172.21.15.114:0/79632769 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f2bcc108a80 0x7f2bcc0757c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:25:52.226 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:25:52.227+0000 7f2bd3ac8700 1 --2- 172.21.15.114:0/79632769 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f2bcc102af0 0x7f2bcc074d40 unknown :-1 s=CLOSED pgs=293 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:25:52.226 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:25:52.227+0000 7f2bd3ac8700 1 --2- 172.21.15.114:0/79632769 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f2bcc1034b0 0x7f2bcc075280 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:25:52.226 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:25:52.227+0000 7f2bd3ac8700 1 -- 172.21.15.114:0/79632769 >> 172.21.15.114:0/79632769 conn(0x7f2bcc079b20 msgr2=0x7f2bcc101780 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T10:25:52.226 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:25:52.227+0000 7f2bd3ac8700 1 -- 172.21.15.114:0/79632769 shutdown_connections 2024-04-23T10:25:52.226 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:25:52.227+0000 7f2bd3ac8700 1 -- 172.21.15.114:0/79632769 wait complete. 2024-04-23T10:25:52.254 INFO:journalctl@ceph.mon.a.smithi114.stdout:Apr 23 10:25:52 smithi114 ceph-mon[29255]: from='client.? 172.21.15.114:0/32258099' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 11}]: dispatch 2024-04-23T10:25:52.328 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 23 10:25:52 smithi204 ceph-mon[32649]: from='client.? 172.21.15.114:0/32258099' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 11}]: dispatch 2024-04-23T10:25:52.492 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 23 10:25:52 smithi119 ceph-mon[32608]: from='client.? 172.21.15.114:0/32258099' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 11}]: dispatch 2024-04-23T10:25:52.997 INFO:tasks.cephadm.ceph_manager.ceph:need seq 167503724596 got 167503724602 for osd.5 2024-04-23T10:25:52.997 DEBUG:teuthology.parallel:result is None 2024-04-23T10:25:53.225 INFO:journalctl@ceph.mon.a.smithi114.stdout:Apr 23 10:25:53 smithi114 ceph-mon[29255]: pgmap v322: 97 pgs: 4 peering, 93 active+clean; 641 KiB data, 370 MiB used, 1.0 TiB / 1.0 TiB avail; 1.9 KiB/s rd, 3 op/s; 0 B/s, 1 keys/s, 0 objects/s recovering 2024-04-23T10:25:53.225 INFO:journalctl@ceph.mon.a.smithi114.stdout:Apr 23 10:25:53 smithi114 ceph-mon[29255]: from='client.? 172.21.15.114:0/79632769' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 5}]: dispatch 2024-04-23T10:25:53.261 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:25:53.262+0000 7f58b2113700 1 -- 172.21.15.114:0/2697131077 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f58ac06bae0 msgr2=0x7f58ac10f930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T10:25:53.261 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:25:53.262+0000 7f58b2113700 1 --2- 172.21.15.114:0/2697131077 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f58ac06bae0 0x7f58ac10f930 secure :-1 s=READY pgs=294 cs=0 l=1 rev1=1 crypto rx=0x7f58a400b0a0 tx=0x7f58a400b3b0 comp rx=0 tx=0).stop 2024-04-23T10:25:53.262 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:25:53.262+0000 7f58b2113700 1 -- 172.21.15.114:0/2697131077 shutdown_connections 2024-04-23T10:25:53.262 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:25:53.262+0000 7f58b2113700 1 --2- 172.21.15.114:0/2697131077 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f58ac108a60 0x7f58ac108e40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:25:53.262 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:25:53.262+0000 7f58b2113700 1 --2- 172.21.15.114:0/2697131077 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f58ac06bae0 0x7f58ac10f930 unknown :-1 s=CLOSED pgs=294 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:25:53.262 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:25:53.262+0000 7f58b2113700 1 --2- 172.21.15.114:0/2697131077 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f58ac06b140 0x7f58ac06b5a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:25:53.262 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:25:53.262+0000 7f58b2113700 1 -- 172.21.15.114:0/2697131077 >> 172.21.15.114:0/2697131077 conn(0x7f58ac078980 msgr2=0x7f58ac078d90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T10:25:53.262 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:25:53.262+0000 7f58b2113700 1 -- 172.21.15.114:0/2697131077 shutdown_connections 2024-04-23T10:25:53.262 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:25:53.262+0000 7f58b2113700 1 -- 172.21.15.114:0/2697131077 wait complete. 2024-04-23T10:25:53.264 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:25:53.263+0000 7f58b2113700 1 Processor -- start 2024-04-23T10:25:53.264 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:25:53.263+0000 7f58b2113700 1 -- start start 2024-04-23T10:25:53.264 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:25:53.263+0000 7f58b2113700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f58ac06b140 0x7f58ac101a40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T10:25:53.264 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:25:53.263+0000 7f58b2113700 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f58ac06bae0 0x7f58ac101f80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T10:25:53.264 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:25:53.263+0000 7f58b2113700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f58ac108a60 0x7f58ac1024c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T10:25:53.264 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:25:53.263+0000 7f58b2113700 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x7f58ac112c40 con 0x7f58ac108a60 2024-04-23T10:25:53.264 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:25:53.263+0000 7f58b2113700 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 -- 0x7f58ac112ac0 con 0x7f58ac06b140 2024-04-23T10:25:53.264 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:25:53.263+0000 7f58b2113700 1 -- --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_getmap magic: 0 -- 0x7f58ac112dc0 con 0x7f58ac06bae0 2024-04-23T10:25:53.264 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:25:53.264+0000 7f58abfff700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f58ac108a60 0x7f58ac1024c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T10:25:53.264 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:25:53.264+0000 7f58ab7fe700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f58ac06b140 0x7f58ac101a40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T10:25:53.264 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:25:53.264+0000 7f58abfff700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f58ac108a60 0x7f58ac1024c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.114:58478/0 (socket says 172.21.15.114:58478) 2024-04-23T10:25:53.264 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:25:53.264+0000 7f58abfff700 1 -- 172.21.15.114:0/2372415693 learned_addr learned my addr 172.21.15.114:0/2372415693 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-04-23T10:25:53.264 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:25:53.264+0000 7f58aaffd700 1 --2- 172.21.15.114:0/2372415693 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f58ac06bae0 0x7f58ac101f80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T10:25:53.265 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:25:53.264+0000 7f58abfff700 1 -- 172.21.15.114:0/2372415693 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f58ac06bae0 msgr2=0x7f58ac101f80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T10:25:53.265 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:25:53.264+0000 7f58abfff700 1 --2- 172.21.15.114:0/2372415693 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f58ac06bae0 0x7f58ac101f80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:25:53.265 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:25:53.264+0000 7f58abfff700 1 -- 172.21.15.114:0/2372415693 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f58ac06b140 msgr2=0x7f58ac101a40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T10:25:53.265 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:25:53.264+0000 7f58abfff700 1 --2- 172.21.15.114:0/2372415693 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f58ac06b140 0x7f58ac101a40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:25:53.265 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:25:53.264+0000 7f58abfff700 1 -- 172.21.15.114:0/2372415693 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f58ac1a7620 con 0x7f58ac108a60 2024-04-23T10:25:53.265 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:25:53.264+0000 7f58ab7fe700 1 --2- 172.21.15.114:0/2372415693 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f58ac06b140 0x7f58ac101a40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-23T10:25:53.265 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:25:53.264+0000 7f58abfff700 1 --2- 172.21.15.114:0/2372415693 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f58ac108a60 0x7f58ac1024c0 secure :-1 s=READY pgs=295 cs=0 l=1 rev1=1 crypto rx=0x7f58a40062a0 tx=0x7f58a4004c80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T10:25:53.265 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:25:53.264+0000 7f58a8ff9700 1 -- 172.21.15.114:0/2372415693 <== mon.0 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f58a4021070 con 0x7f58ac108a60 2024-04-23T10:25:53.265 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:25:53.264+0000 7f58a8ff9700 1 -- 172.21.15.114:0/2372415693 <== mon.0 v2:172.21.15.114:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f58a4007e30 con 0x7f58ac108a60 2024-04-23T10:25:53.265 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:25:53.264+0000 7f58a8ff9700 1 -- 172.21.15.114:0/2372415693 <== mon.0 v2:172.21.15.114:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f58a40046d0 con 0x7f58ac108a60 2024-04-23T10:25:53.265 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:25:53.264+0000 7f58b2113700 1 -- 172.21.15.114:0/2372415693 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f58ac1a78b0 con 0x7f58ac108a60 2024-04-23T10:25:53.265 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:25:53.264+0000 7f58b2113700 1 -- 172.21.15.114:0/2372415693 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f58ac1a7d60 con 0x7f58ac108a60 2024-04-23T10:25:53.266 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:25:53.265+0000 7f58b2113700 1 -- 172.21.15.114:0/2372415693 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f58ac1041a0 con 0x7f58ac108a60 2024-04-23T10:25:53.266 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:25:53.267+0000 7f58a8ff9700 1 -- 172.21.15.114:0/2372415693 <== mon.0 v2:172.21.15.114:3300/0 4 ==== mgrmap(e 17) ==== 138698+0+0 (secure 0 0 0) 0x7f58a4007980 con 0x7f58ac108a60 2024-04-23T10:25:53.266 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:25:53.267+0000 7f58a8ff9700 1 --2- 172.21.15.114:0/2372415693 >> [v2:172.21.15.114:6800/465333808,v1:172.21.15.114:6801/465333808] conn(0x7f58940a3d50 0x7f58940a6210 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T10:25:53.266 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:25:53.267+0000 7f58a8ff9700 1 -- 172.21.15.114:0/2372415693 <== mon.0 v2:172.21.15.114:3300/0 5 ==== osd_map(89..89 src has 1..89) ==== 8269+0+0 (secure 0 0 0) 0x7f58a400f020 con 0x7f58ac108a60 2024-04-23T10:25:53.267 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:25:53.268+0000 7f58ab7fe700 1 --2- 172.21.15.114:0/2372415693 >> [v2:172.21.15.114:6800/465333808,v1:172.21.15.114:6801/465333808] conn(0x7f58940a3d50 0x7f58940a6210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T10:25:53.267 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:25:53.268+0000 7f58ab7fe700 1 --2- 172.21.15.114:0/2372415693 >> [v2:172.21.15.114:6800/465333808,v1:172.21.15.114:6801/465333808] conn(0x7f58940a3d50 0x7f58940a6210 secure :-1 s=READY pgs=206 cs=0 l=1 rev1=1 crypto rx=0x7f589c005950 tx=0x7f589c00b410 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T10:25:53.271 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:25:53.272+0000 7f58a8ff9700 1 -- 172.21.15.114:0/2372415693 <== mon.0 v2:172.21.15.114:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f58a402a020 con 0x7f58ac108a60 2024-04-23T10:25:53.328 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 23 10:25:53 smithi204 ceph-mon[32649]: pgmap v322: 97 pgs: 4 peering, 93 active+clean; 641 KiB data, 370 MiB used, 1.0 TiB / 1.0 TiB avail; 1.9 KiB/s rd, 3 op/s; 0 B/s, 1 keys/s, 0 objects/s recovering 2024-04-23T10:25:53.328 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 23 10:25:53 smithi204 ceph-mon[32649]: from='client.? 172.21.15.114:0/79632769' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 5}]: dispatch 2024-04-23T10:25:53.474 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:25:53.475+0000 7f58b2113700 1 -- 172.21.15.114:0/2372415693 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 2} v 0) -- 0x7f58ac02e140 con 0x7f58ac108a60 2024-04-23T10:25:53.474 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:25:53.475+0000 7f58a8ff9700 1 -- 172.21.15.114:0/2372415693 <== mon.0 v2:172.21.15.114:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 2}]=0 v0) ==== 74+0+12 (secure 0 0 0) 0x7f58a4088090 con 0x7f58ac108a60 2024-04-23T10:25:53.474 INFO:teuthology.orchestra.run.smithi114.stdout:77309411401 2024-04-23T10:25:53.476 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:25:53.477+0000 7f58b2113700 1 -- 172.21.15.114:0/2372415693 >> [v2:172.21.15.114:6800/465333808,v1:172.21.15.114:6801/465333808] conn(0x7f58940a3d50 msgr2=0x7f58940a6210 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T10:25:53.477 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:25:53.477+0000 7f58b2113700 1 --2- 172.21.15.114:0/2372415693 >> [v2:172.21.15.114:6800/465333808,v1:172.21.15.114:6801/465333808] conn(0x7f58940a3d50 0x7f58940a6210 secure :-1 s=READY pgs=206 cs=0 l=1 rev1=1 crypto rx=0x7f589c005950 tx=0x7f589c00b410 comp rx=0 tx=0).stop 2024-04-23T10:25:53.477 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:25:53.477+0000 7f58b2113700 1 -- 172.21.15.114:0/2372415693 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f58ac108a60 msgr2=0x7f58ac1024c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T10:25:53.477 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:25:53.477+0000 7f58b2113700 1 --2- 172.21.15.114:0/2372415693 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f58ac108a60 0x7f58ac1024c0 secure :-1 s=READY pgs=295 cs=0 l=1 rev1=1 crypto rx=0x7f58a40062a0 tx=0x7f58a4004c80 comp rx=0 tx=0).stop 2024-04-23T10:25:53.477 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:25:53.478+0000 7f58b2113700 1 -- 172.21.15.114:0/2372415693 shutdown_connections 2024-04-23T10:25:53.477 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:25:53.478+0000 7f58b2113700 1 --2- 172.21.15.114:0/2372415693 >> [v2:172.21.15.114:6800/465333808,v1:172.21.15.114:6801/465333808] conn(0x7f58940a3d50 0x7f58940a6210 unknown :-1 s=CLOSED pgs=206 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:25:53.477 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:25:53.478+0000 7f58b2113700 1 --2- 172.21.15.114:0/2372415693 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f58ac06b140 0x7f58ac101a40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:25:53.477 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:25:53.478+0000 7f58b2113700 1 --2- 172.21.15.114:0/2372415693 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f58ac108a60 0x7f58ac1024c0 unknown :-1 s=CLOSED pgs=295 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:25:53.477 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:25:53.478+0000 7f58b2113700 1 --2- 172.21.15.114:0/2372415693 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f58ac06bae0 0x7f58ac101f80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:25:53.477 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:25:53.478+0000 7f58b2113700 1 -- 172.21.15.114:0/2372415693 >> 172.21.15.114:0/2372415693 conn(0x7f58ac078980 msgr2=0x7f58ac0ffb20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T10:25:53.477 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:25:53.478+0000 7f58b2113700 1 -- 172.21.15.114:0/2372415693 shutdown_connections 2024-04-23T10:25:53.477 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:25:53.478+0000 7f58b2113700 1 -- 172.21.15.114:0/2372415693 wait complete. 2024-04-23T10:25:53.492 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 23 10:25:53 smithi119 ceph-mon[32608]: pgmap v322: 97 pgs: 4 peering, 93 active+clean; 641 KiB data, 370 MiB used, 1.0 TiB / 1.0 TiB avail; 1.9 KiB/s rd, 3 op/s; 0 B/s, 1 keys/s, 0 objects/s recovering 2024-04-23T10:25:53.492 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 23 10:25:53 smithi119 ceph-mon[32608]: from='client.? 172.21.15.114:0/79632769' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 5}]: dispatch 2024-04-23T10:25:53.648 INFO:tasks.cephadm.ceph_manager.ceph:need seq 352187318294 got 352187318299 for osd.11 2024-04-23T10:25:53.648 DEBUG:teuthology.parallel:result is None 2024-04-23T10:25:54.099 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:25:54.093+0000 7fd217fa4700 1 -- 172.21.15.114:0/1400250889 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fd210102330 msgr2=0x7fd21010e7f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T10:25:54.099 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:25:54.093+0000 7fd217fa4700 1 --2- 172.21.15.114:0/1400250889 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fd210102330 0x7fd21010e7f0 secure :-1 s=READY pgs=296 cs=0 l=1 rev1=1 crypto rx=0x7fd20800b0a0 tx=0x7fd20800b3b0 comp rx=0 tx=0).stop 2024-04-23T10:25:54.099 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:25:54.093+0000 7fd217fa4700 1 -- 172.21.15.114:0/1400250889 shutdown_connections 2024-04-23T10:25:54.100 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:25:54.093+0000 7fd217fa4700 1 --2- 172.21.15.114:0/1400250889 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fd210107920 0x7fd210107d00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:25:54.100 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:25:54.093+0000 7fd217fa4700 1 --2- 172.21.15.114:0/1400250889 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fd210102330 0x7fd21010e7f0 unknown :-1 s=CLOSED pgs=296 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:25:54.100 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:25:54.093+0000 7fd217fa4700 1 --2- 172.21.15.114:0/1400250889 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fd210101970 0x7fd210101df0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:25:54.100 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:25:54.093+0000 7fd217fa4700 1 -- 172.21.15.114:0/1400250889 >> 172.21.15.114:0/1400250889 conn(0x7fd2100779b0 msgr2=0x7fd210077dc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T10:25:54.100 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:25:54.093+0000 7fd217fa4700 1 -- 172.21.15.114:0/1400250889 shutdown_connections 2024-04-23T10:25:54.100 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:25:54.093+0000 7fd217fa4700 1 -- 172.21.15.114:0/1400250889 wait complete. 2024-04-23T10:25:54.100 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:25:54.094+0000 7fd217fa4700 1 Processor -- start 2024-04-23T10:25:54.100 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:25:54.094+0000 7fd217fa4700 1 -- start start 2024-04-23T10:25:54.100 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:25:54.094+0000 7fd217fa4700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fd210101970 0x7fd210199ee0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T10:25:54.100 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:25:54.094+0000 7fd217fa4700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fd210102330 0x7fd21019a420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T10:25:54.100 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:25:54.094+0000 7fd217fa4700 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fd210107920 0x7fd2101a4f60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T10:25:54.100 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:25:54.094+0000 7fd217fa4700 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x7fd210111b00 con 0x7fd210102330 2024-04-23T10:25:54.100 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:25:54.094+0000 7fd217fa4700 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 -- 0x7fd210111980 con 0x7fd210101970 2024-04-23T10:25:54.101 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:25:54.094+0000 7fd217fa4700 1 -- --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_getmap magic: 0 -- 0x7fd210111c80 con 0x7fd210107920 2024-04-23T10:25:54.101 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:25:54.094+0000 7fd21553f700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fd210102330 0x7fd21019a420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T10:25:54.101 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:25:54.094+0000 7fd21553f700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fd210102330 0x7fd21019a420 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.114:58498/0 (socket says 172.21.15.114:58498) 2024-04-23T10:25:54.101 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:25:54.094+0000 7fd21553f700 1 -- 172.21.15.114:0/2591050375 learned_addr learned my addr 172.21.15.114:0/2591050375 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-04-23T10:25:54.101 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:25:54.094+0000 7fd216541700 1 --2- 172.21.15.114:0/2591050375 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fd210107920 0x7fd2101a4f60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T10:25:54.101 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:25:54.094+0000 7fd215d40700 1 --2- 172.21.15.114:0/2591050375 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fd210101970 0x7fd210199ee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T10:25:54.101 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:25:54.094+0000 7fd21553f700 1 -- 172.21.15.114:0/2591050375 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fd210107920 msgr2=0x7fd2101a4f60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T10:25:54.101 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:25:54.094+0000 7fd21553f700 1 --2- 172.21.15.114:0/2591050375 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fd210107920 0x7fd2101a4f60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:25:54.101 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:25:54.095+0000 7fd21553f700 1 -- 172.21.15.114:0/2591050375 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fd210101970 msgr2=0x7fd210199ee0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T10:25:54.101 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:25:54.095+0000 7fd21553f700 1 --2- 172.21.15.114:0/2591050375 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fd210101970 0x7fd210199ee0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:25:54.101 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:25:54.095+0000 7fd21553f700 1 -- 172.21.15.114:0/2591050375 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd2101a54a0 con 0x7fd210102330 2024-04-23T10:25:54.101 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:25:54.095+0000 7fd215d40700 1 --2- 172.21.15.114:0/2591050375 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fd210101970 0x7fd210199ee0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-23T10:25:54.101 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:25:54.095+0000 7fd21553f700 1 --2- 172.21.15.114:0/2591050375 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fd210102330 0x7fd21019a420 secure :-1 s=READY pgs=297 cs=0 l=1 rev1=1 crypto rx=0x7fd20000d950 tx=0x7fd20000dd10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T10:25:54.102 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:25:54.095+0000 7fd206ffd700 1 -- 172.21.15.114:0/2591050375 <== mon.0 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd200016070 con 0x7fd210102330 2024-04-23T10:25:54.102 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:25:54.095+0000 7fd217fa4700 1 -- 172.21.15.114:0/2591050375 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fd2101a56d0 con 0x7fd210102330 2024-04-23T10:25:54.102 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:25:54.095+0000 7fd206ffd700 1 -- 172.21.15.114:0/2591050375 <== mon.0 v2:172.21.15.114:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fd200010460 con 0x7fd210102330 2024-04-23T10:25:54.102 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:25:54.095+0000 7fd206ffd700 1 -- 172.21.15.114:0/2591050375 <== mon.0 v2:172.21.15.114:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd20000f620 con 0x7fd210102330 2024-04-23T10:25:54.102 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:25:54.095+0000 7fd217fa4700 1 -- 172.21.15.114:0/2591050375 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fd2101a5c30 con 0x7fd210102330 2024-04-23T10:25:54.102 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:25:54.096+0000 7fd217fa4700 1 -- 172.21.15.114:0/2591050375 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fd210103060 con 0x7fd210102330 2024-04-23T10:25:54.102 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:25:54.097+0000 7fd206ffd700 1 -- 172.21.15.114:0/2591050375 <== mon.0 v2:172.21.15.114:3300/0 4 ==== mgrmap(e 17) ==== 138698+0+0 (secure 0 0 0) 0x7fd200019070 con 0x7fd210102330 2024-04-23T10:25:54.102 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:25:54.097+0000 7fd206ffd700 1 --2- 172.21.15.114:0/2591050375 >> [v2:172.21.15.114:6800/465333808,v1:172.21.15.114:6801/465333808] conn(0x7fd1fc0a3cd0 0x7fd1fc0a6190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T10:25:54.102 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:25:54.098+0000 7fd215d40700 1 --2- 172.21.15.114:0/2591050375 >> [v2:172.21.15.114:6800/465333808,v1:172.21.15.114:6801/465333808] conn(0x7fd1fc0a3cd0 0x7fd1fc0a6190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T10:25:54.102 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:25:54.098+0000 7fd206ffd700 1 -- 172.21.15.114:0/2591050375 <== mon.0 v2:172.21.15.114:3300/0 5 ==== osd_map(89..89 src has 1..89) ==== 8269+0+0 (secure 0 0 0) 0x7fd2000d0240 con 0x7fd210102330 2024-04-23T10:25:54.102 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:25:54.098+0000 7fd215d40700 1 --2- 172.21.15.114:0/2591050375 >> [v2:172.21.15.114:6800/465333808,v1:172.21.15.114:6801/465333808] conn(0x7fd1fc0a3cd0 0x7fd1fc0a6190 secure :-1 s=READY pgs=207 cs=0 l=1 rev1=1 crypto rx=0x7fd20c009b50 tx=0x7fd20c009550 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T10:25:54.103 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:25:54.101+0000 7fd206ffd700 1 -- 172.21.15.114:0/2591050375 <== mon.0 v2:172.21.15.114:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7fd20009caa0 con 0x7fd210102330 2024-04-23T10:25:54.308 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:25:54.309+0000 7fd217fa4700 1 -- 172.21.15.114:0/2591050375 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 7} v 0) -- 0x7fd21002df80 con 0x7fd210102330 2024-04-23T10:25:54.308 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:25:54.309+0000 7fd206ffd700 1 -- 172.21.15.114:0/2591050375 <== mon.0 v2:172.21.15.114:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 7}]=0 v0) ==== 74+0+13 (secure 0 0 0) 0x7fd20009c3a0 con 0x7fd210102330 2024-04-23T10:25:54.308 INFO:teuthology.orchestra.run.smithi114.stdout:227633266736 2024-04-23T10:25:54.310 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:25:54.311+0000 7fd217fa4700 1 -- 172.21.15.114:0/2591050375 >> [v2:172.21.15.114:6800/465333808,v1:172.21.15.114:6801/465333808] conn(0x7fd1fc0a3cd0 msgr2=0x7fd1fc0a6190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T10:25:54.310 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:25:54.311+0000 7fd217fa4700 1 --2- 172.21.15.114:0/2591050375 >> [v2:172.21.15.114:6800/465333808,v1:172.21.15.114:6801/465333808] conn(0x7fd1fc0a3cd0 0x7fd1fc0a6190 secure :-1 s=READY pgs=207 cs=0 l=1 rev1=1 crypto rx=0x7fd20c009b50 tx=0x7fd20c009550 comp rx=0 tx=0).stop 2024-04-23T10:25:54.311 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:25:54.311+0000 7fd217fa4700 1 -- 172.21.15.114:0/2591050375 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fd210102330 msgr2=0x7fd21019a420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T10:25:54.311 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:25:54.311+0000 7fd217fa4700 1 --2- 172.21.15.114:0/2591050375 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fd210102330 0x7fd21019a420 secure :-1 s=READY pgs=297 cs=0 l=1 rev1=1 crypto rx=0x7fd20000d950 tx=0x7fd20000dd10 comp rx=0 tx=0).stop 2024-04-23T10:25:54.311 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:25:54.311+0000 7fd217fa4700 1 -- 172.21.15.114:0/2591050375 shutdown_connections 2024-04-23T10:25:54.311 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:25:54.311+0000 7fd217fa4700 1 --2- 172.21.15.114:0/2591050375 >> [v2:172.21.15.114:6800/465333808,v1:172.21.15.114:6801/465333808] conn(0x7fd1fc0a3cd0 0x7fd1fc0a6190 unknown :-1 s=CLOSED pgs=207 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:25:54.311 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:25:54.311+0000 7fd217fa4700 1 --2- 172.21.15.114:0/2591050375 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fd210101970 0x7fd210199ee0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:25:54.311 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:25:54.311+0000 7fd217fa4700 1 --2- 172.21.15.114:0/2591050375 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fd210107920 0x7fd2101a4f60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:25:54.311 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:25:54.311+0000 7fd217fa4700 1 --2- 172.21.15.114:0/2591050375 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fd210102330 0x7fd21019a420 unknown :-1 s=CLOSED pgs=297 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:25:54.311 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:25:54.311+0000 7fd217fa4700 1 -- 172.21.15.114:0/2591050375 >> 172.21.15.114:0/2591050375 conn(0x7fd2100779b0 msgr2=0x7fd2100ffa20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T10:25:54.311 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:25:54.311+0000 7fd217fa4700 1 -- 172.21.15.114:0/2591050375 shutdown_connections 2024-04-23T10:25:54.311 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:25:54.311+0000 7fd217fa4700 1 -- 172.21.15.114:0/2591050375 wait complete. 2024-04-23T10:25:54.321 INFO:journalctl@ceph.mon.a.smithi114.stdout:Apr 23 10:25:54 smithi114 ceph-mon[29255]: from='client.? 172.21.15.114:0/2372415693' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 2}]: dispatch 2024-04-23T10:25:54.328 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 23 10:25:54 smithi204 ceph-mon[32649]: from='client.? 172.21.15.114:0/2372415693' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 2}]: dispatch 2024-04-23T10:25:54.492 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 23 10:25:54 smithi119 ceph-mon[32608]: from='client.? 172.21.15.114:0/2372415693' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 2}]: dispatch 2024-04-23T10:25:54.680 INFO:tasks.cephadm.ceph_manager.ceph:need seq 77309411395 got 77309411401 for osd.2 2024-04-23T10:25:54.680 DEBUG:teuthology.parallel:result is None 2024-04-23T10:25:55.254 INFO:journalctl@ceph.mon.a.smithi114.stdout:Apr 23 10:25:55 smithi114 ceph-mon[29255]: pgmap v323: 97 pgs: 97 active+clean; 641 KiB data, 371 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s; 0 B/s, 1 keys/s, 0 objects/s recovering 2024-04-23T10:25:55.254 INFO:journalctl@ceph.mon.a.smithi114.stdout:Apr 23 10:25:55 smithi114 ceph-mon[29255]: Health check cleared: PG_AVAILABILITY (was: Reduced data availability: 4 pgs peering) 2024-04-23T10:25:55.254 INFO:journalctl@ceph.mon.a.smithi114.stdout:Apr 23 10:25:55 smithi114 ceph-mon[29255]: Cluster is now healthy 2024-04-23T10:25:55.254 INFO:journalctl@ceph.mon.a.smithi114.stdout:Apr 23 10:25:55 smithi114 ceph-mon[29255]: from='client.? 172.21.15.114:0/2591050375' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 7}]: dispatch 2024-04-23T10:25:55.328 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 23 10:25:55 smithi204 ceph-mon[32649]: pgmap v323: 97 pgs: 97 active+clean; 641 KiB data, 371 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s; 0 B/s, 1 keys/s, 0 objects/s recovering 2024-04-23T10:25:55.328 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 23 10:25:55 smithi204 ceph-mon[32649]: Health check cleared: PG_AVAILABILITY (was: Reduced data availability: 4 pgs peering) 2024-04-23T10:25:55.328 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 23 10:25:55 smithi204 ceph-mon[32649]: Cluster is now healthy 2024-04-23T10:25:55.328 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 23 10:25:55 smithi204 ceph-mon[32649]: from='client.? 172.21.15.114:0/2591050375' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 7}]: dispatch 2024-04-23T10:25:55.492 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 23 10:25:55 smithi119 ceph-mon[32608]: pgmap v323: 97 pgs: 97 active+clean; 641 KiB data, 371 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s; 0 B/s, 1 keys/s, 0 objects/s recovering 2024-04-23T10:25:55.492 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 23 10:25:55 smithi119 ceph-mon[32608]: Health check cleared: PG_AVAILABILITY (was: Reduced data availability: 4 pgs peering) 2024-04-23T10:25:55.492 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 23 10:25:55 smithi119 ceph-mon[32608]: Cluster is now healthy 2024-04-23T10:25:55.492 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 23 10:25:55 smithi119 ceph-mon[32608]: from='client.? 172.21.15.114:0/2591050375' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 7}]: dispatch 2024-04-23T10:25:55.965 INFO:tasks.cephadm.ceph_manager.ceph:need seq 227633266730 got 227633266736 for osd.7 2024-04-23T10:25:55.965 DEBUG:teuthology.parallel:result is None 2024-04-23T10:25:55.966 INFO:tasks.cephadm.ceph_manager.ceph:waiting for clean 2024-04-23T10:25:55.966 DEBUG:teuthology.orchestra.run.smithi114:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d69f13d827952bb07bb9b7973bca14f97d764d6f shell --fsid aa240572-0159-11ef-bc93-c7b262605968 -- ceph pg dump --format=json 2024-04-23T10:25:56.243 INFO:teuthology.orchestra.run.smithi114.stderr:Inferring config /var/lib/ceph/aa240572-0159-11ef-bc93-c7b262605968/mon.a/config 2024-04-23T10:25:57.328 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 23 10:25:57 smithi204 ceph-mon[32649]: pgmap v324: 97 pgs: 97 active+clean; 641 KiB data, 371 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 6 op/s; 0 B/s, 1 keys/s, 0 objects/s recovering 2024-04-23T10:25:57.350 INFO:journalctl@ceph.mon.a.smithi114.stdout:Apr 23 10:25:57 smithi114 ceph-mon[29255]: pgmap v324: 97 pgs: 97 active+clean; 641 KiB data, 371 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 6 op/s; 0 B/s, 1 keys/s, 0 objects/s recovering 2024-04-23T10:25:57.492 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 23 10:25:57 smithi119 ceph-mon[32608]: pgmap v324: 97 pgs: 97 active+clean; 641 KiB data, 371 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 6 op/s; 0 B/s, 1 keys/s, 0 objects/s recovering 2024-04-23T10:25:57.879 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:25:57.879+0000 7f98c09c6700 1 -- 172.21.15.114:0/2250887251 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f98bc102330 msgr2=0x7f98bc10e7f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T10:25:57.879 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:25:57.879+0000 7f98c09c6700 1 --2- 172.21.15.114:0/2250887251 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f98bc102330 0x7f98bc10e7f0 secure :-1 s=READY pgs=298 cs=0 l=1 rev1=1 crypto rx=0x7f98b400b0a0 tx=0x7f98b400b3b0 comp rx=0 tx=0).stop 2024-04-23T10:25:57.879 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:25:57.879+0000 7f98c09c6700 1 -- 172.21.15.114:0/2250887251 shutdown_connections 2024-04-23T10:25:57.879 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:25:57.879+0000 7f98c09c6700 1 --2- 172.21.15.114:0/2250887251 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f98bc107920 0x7f98bc107d00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:25:57.879 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:25:57.879+0000 7f98c09c6700 1 --2- 172.21.15.114:0/2250887251 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f98bc102330 0x7f98bc10e7f0 unknown :-1 s=CLOSED pgs=298 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:25:57.879 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:25:57.879+0000 7f98c09c6700 1 --2- 172.21.15.114:0/2250887251 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f98bc101970 0x7f98bc101df0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:25:57.879 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:25:57.879+0000 7f98c09c6700 1 -- 172.21.15.114:0/2250887251 >> 172.21.15.114:0/2250887251 conn(0x7f98bc078910 msgr2=0x7f98bc078d20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T10:25:57.879 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:25:57.879+0000 7f98c09c6700 1 -- 172.21.15.114:0/2250887251 shutdown_connections 2024-04-23T10:25:57.879 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:25:57.879+0000 7f98c09c6700 1 -- 172.21.15.114:0/2250887251 wait complete. 2024-04-23T10:25:57.881 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:25:57.879+0000 7f98c09c6700 1 Processor -- start 2024-04-23T10:25:57.882 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:25:57.879+0000 7f98c09c6700 1 -- start start 2024-04-23T10:25:57.882 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:25:57.880+0000 7f98c09c6700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f98bc101970 0x7f98bc1991e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T10:25:57.882 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:25:57.880+0000 7f98c09c6700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f98bc102330 0x7f98bc199720 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T10:25:57.882 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:25:57.880+0000 7f98c09c6700 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f98bc107920 0x7f98bc19da60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T10:25:57.882 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:25:57.880+0000 7f98c09c6700 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x7f98bc111b60 con 0x7f98bc101970 2024-04-23T10:25:57.882 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:25:57.880+0000 7f98c09c6700 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 -- 0x7f98bc1119e0 con 0x7f98bc102330 2024-04-23T10:25:57.882 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:25:57.880+0000 7f98c09c6700 1 -- --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_getmap magic: 0 -- 0x7f98bc111ce0 con 0x7f98bc107920 2024-04-23T10:25:57.882 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:25:57.880+0000 7f98ba59c700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f98bc101970 0x7f98bc1991e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T10:25:57.882 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:25:57.880+0000 7f98ba59c700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f98bc101970 0x7f98bc1991e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.114:34114/0 (socket says 172.21.15.114:34114) 2024-04-23T10:25:57.882 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:25:57.880+0000 7f98ba59c700 1 -- 172.21.15.114:0/536294666 learned_addr learned my addr 172.21.15.114:0/536294666 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-04-23T10:25:57.883 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:25:57.880+0000 7f98bad9d700 1 --2- 172.21.15.114:0/536294666 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f98bc107920 0x7f98bc19da60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T10:25:57.883 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:25:57.880+0000 7f98b9d9b700 1 --2- 172.21.15.114:0/536294666 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f98bc102330 0x7f98bc199720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T10:25:57.883 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:25:57.880+0000 7f98ba59c700 1 -- 172.21.15.114:0/536294666 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f98bc107920 msgr2=0x7f98bc19da60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T10:25:57.883 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:25:57.880+0000 7f98ba59c700 1 --2- 172.21.15.114:0/536294666 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f98bc107920 0x7f98bc19da60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:25:57.883 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:25:57.880+0000 7f98ba59c700 1 -- 172.21.15.114:0/536294666 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f98bc102330 msgr2=0x7f98bc199720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T10:25:57.883 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:25:57.880+0000 7f98ba59c700 1 --2- 172.21.15.114:0/536294666 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f98bc102330 0x7f98bc199720 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:25:57.883 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:25:57.880+0000 7f98ba59c700 1 -- 172.21.15.114:0/536294666 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f98bc19e1a0 con 0x7f98bc101970 2024-04-23T10:25:57.883 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:25:57.880+0000 7f98bad9d700 1 --2- 172.21.15.114:0/536294666 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f98bc107920 0x7f98bc19da60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-23T10:25:57.883 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:25:57.880+0000 7f98ba59c700 1 --2- 172.21.15.114:0/536294666 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f98bc101970 0x7f98bc1991e0 secure :-1 s=READY pgs=299 cs=0 l=1 rev1=1 crypto rx=0x7f98ac00b530 tx=0x7f98ac00b840 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T10:25:57.883 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:25:57.880+0000 7f98ab7fe700 1 -- 172.21.15.114:0/536294666 <== mon.0 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f98ac015020 con 0x7f98bc101970 2024-04-23T10:25:57.883 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:25:57.880+0000 7f98ab7fe700 1 -- 172.21.15.114:0/536294666 <== mon.0 v2:172.21.15.114:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f98ac011ce0 con 0x7f98bc101970 2024-04-23T10:25:57.883 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:25:57.880+0000 7f98c09c6700 1 -- 172.21.15.114:0/536294666 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f98bc19e490 con 0x7f98bc101970 2024-04-23T10:25:57.884 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:25:57.880+0000 7f98ab7fe700 1 -- 172.21.15.114:0/536294666 <== mon.0 v2:172.21.15.114:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f98ac010e40 con 0x7f98bc101970 2024-04-23T10:25:57.884 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:25:57.881+0000 7f98c09c6700 1 -- 172.21.15.114:0/536294666 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f98bc1a5d10 con 0x7f98bc101970 2024-04-23T10:25:57.884 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:25:57.881+0000 7f98c09c6700 1 -- 172.21.15.114:0/536294666 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f98bc069730 con 0x7f98bc101970 2024-04-23T10:25:57.884 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:25:57.883+0000 7f98ab7fe700 1 -- 172.21.15.114:0/536294666 <== mon.0 v2:172.21.15.114:3300/0 4 ==== mgrmap(e 17) ==== 138698+0+0 (secure 0 0 0) 0x7f98ac018020 con 0x7f98bc101970 2024-04-23T10:25:57.884 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:25:57.883+0000 7f98ab7fe700 1 --2- 172.21.15.114:0/536294666 >> [v2:172.21.15.114:6800/465333808,v1:172.21.15.114:6801/465333808] conn(0x7f98a40a3cd0 0x7f98a40a6190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T10:25:57.885 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:25:57.883+0000 7f98b9d9b700 1 --2- 172.21.15.114:0/536294666 >> [v2:172.21.15.114:6800/465333808,v1:172.21.15.114:6801/465333808] conn(0x7f98a40a3cd0 0x7f98a40a6190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T10:25:57.885 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:25:57.883+0000 7f98ab7fe700 1 -- 172.21.15.114:0/536294666 <== mon.0 v2:172.21.15.114:3300/0 5 ==== osd_map(89..89 src has 1..89) ==== 8269+0+0 (secure 0 0 0) 0x7f98ac0d0250 con 0x7f98bc101970 2024-04-23T10:25:57.885 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:25:57.884+0000 7f98b9d9b700 1 --2- 172.21.15.114:0/536294666 >> [v2:172.21.15.114:6800/465333808,v1:172.21.15.114:6801/465333808] conn(0x7f98a40a3cd0 0x7f98a40a6190 secure :-1 s=READY pgs=208 cs=0 l=1 rev1=1 crypto rx=0x7f98b0005fd0 tx=0x7f98b0005f60 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T10:25:57.887 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:25:57.887+0000 7f98ab7fe700 1 -- 172.21.15.114:0/536294666 <== mon.0 v2:172.21.15.114:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f98ac01f020 con 0x7f98bc101970 2024-04-23T10:25:58.118 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:25:58.118+0000 7f98c09c6700 1 -- 172.21.15.114:0/536294666 --> [v2:172.21.15.114:6800/465333808,v1:172.21.15.114:6801/465333808] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) -- 0x7f98bc19a4f0 con 0x7f98a40a3cd0 2024-04-23T10:25:58.126 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:25:58.126+0000 7f98ab7fe700 1 -- 172.21.15.114:0/536294666 <== mgr.14152 v2:172.21.15.114:6800/465333808 1 ==== mgr_command_reply(tid 0: 0 dumped all) ==== 18+0+258714 (secure 0 0 0) 0x7f98bc19a4f0 con 0x7f98a40a3cd0 2024-04-23T10:25:58.130 INFO:teuthology.orchestra.run.smithi114.stdout: 2024-04-23T10:25:58.132 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:25:58.129+0000 7f98c09c6700 1 -- 172.21.15.114:0/536294666 >> [v2:172.21.15.114:6800/465333808,v1:172.21.15.114:6801/465333808] conn(0x7f98a40a3cd0 msgr2=0x7f98a40a6190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T10:25:58.132 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:25:58.129+0000 7f98c09c6700 1 --2- 172.21.15.114:0/536294666 >> [v2:172.21.15.114:6800/465333808,v1:172.21.15.114:6801/465333808] conn(0x7f98a40a3cd0 0x7f98a40a6190 secure :-1 s=READY pgs=208 cs=0 l=1 rev1=1 crypto rx=0x7f98b0005fd0 tx=0x7f98b0005f60 comp rx=0 tx=0).stop 2024-04-23T10:25:58.132 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:25:58.129+0000 7f98c09c6700 1 -- 172.21.15.114:0/536294666 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f98bc101970 msgr2=0x7f98bc1991e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T10:25:58.132 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:25:58.129+0000 7f98c09c6700 1 --2- 172.21.15.114:0/536294666 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f98bc101970 0x7f98bc1991e0 secure :-1 s=READY pgs=299 cs=0 l=1 rev1=1 crypto rx=0x7f98ac00b530 tx=0x7f98ac00b840 comp rx=0 tx=0).stop 2024-04-23T10:25:58.132 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:25:58.129+0000 7f98c09c6700 1 -- 172.21.15.114:0/536294666 shutdown_connections 2024-04-23T10:25:58.132 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:25:58.129+0000 7f98c09c6700 1 --2- 172.21.15.114:0/536294666 >> [v2:172.21.15.114:6800/465333808,v1:172.21.15.114:6801/465333808] conn(0x7f98a40a3cd0 0x7f98a40a6190 unknown :-1 s=CLOSED pgs=208 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:25:58.132 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:25:58.129+0000 7f98c09c6700 1 --2- 172.21.15.114:0/536294666 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f98bc107920 0x7f98bc19da60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:25:58.132 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:25:58.129+0000 7f98c09c6700 1 --2- 172.21.15.114:0/536294666 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f98bc101970 0x7f98bc1991e0 unknown :-1 s=CLOSED pgs=299 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:25:58.132 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:25:58.129+0000 7f98c09c6700 1 --2- 172.21.15.114:0/536294666 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f98bc102330 0x7f98bc199720 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:25:58.133 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:25:58.129+0000 7f98c09c6700 1 -- 172.21.15.114:0/536294666 >> 172.21.15.114:0/536294666 conn(0x7f98bc078910 msgr2=0x7f98bc06b570 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T10:25:58.133 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:25:58.129+0000 7f98c09c6700 1 -- 172.21.15.114:0/536294666 shutdown_connections 2024-04-23T10:25:58.133 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:25:58.129+0000 7f98c09c6700 1 -- 172.21.15.114:0/536294666 wait complete. 2024-04-23T10:25:58.133 INFO:teuthology.orchestra.run.smithi114.stderr:dumped all 2024-04-23T10:25:58.702 INFO:teuthology.orchestra.run.smithi114.stdout:{"pg_ready":true,"pg_map":{"version":325,"stamp":"2024-04-23T10:25:57.974268+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":656040,"num_objects":64,"num_object_clones":0,"num_object_copies":192,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":64,"num_whiteouts":0,"num_read":572,"num_read_kb":405,"num_write":340,"num_write_kb":3049,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":7,"num_bytes_recovered":1180736,"num_keys_recovered":10,"num_objects_omap":40,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":275,"ondisk_log_size":275,"up":291,"acting":291,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":291,"num_osds":12,"num_per_pool_osds":12,"num_per_pool_omap_osds":12,"kb":1124941824,"kb_used":379416,"kb_used_data":13548,"kb_used_omap":18,"kb_used_meta":365677,"kb_avail":1124562408,"statfs":{"total":1151940427776,"available":1151551905792,"internally_reserved":0,"allocated":13873152,"data_stored":7548408,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":19142,"internal_metadata":374453562},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":65,"num_read_kb":30,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":10,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"10.394800"},"pg_stats":[{"pgid":"3.3f","version":"0'0","reported_seq":17,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-23T10:25:47.584614+0000","last_change":"2024-04-23T10:24:44.051465+0000","last_active":"2024-04-23T10:25:47.584614+0000","last_peered":"2024-04-23T10:25:47.584614+0000","last_clean":"2024-04-23T10:25:47.584614+0000","last_became_active":"2024-04-23T10:24:44.050863+0000","last_became_peered":"2024-04-23T10:24:44.050863+0000","last_unstale":"2024-04-23T10:25:47.584614+0000","last_undegraded":"2024-04-23T10:25:47.584614+0000","last_fullsized":"2024-04-23T10:25:47.584614+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T10:24:43.025324+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T10:24:43.025324+0000","last_clean_scrub_stamp":"2024-04-23T10:24:43.025324+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T15:11:12.508458+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,1,0],"acting":[11,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.3e","version":"0'0","reported_seq":17,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-23T10:25:47.583513+0000","last_change":"2024-04-23T10:24:44.050739+0000","last_active":"2024-04-23T10:25:47.583513+0000","last_peered":"2024-04-23T10:25:47.583513+0000","last_clean":"2024-04-23T10:25:47.583513+0000","last_became_active":"2024-04-23T10:24:44.050430+0000","last_became_peered":"2024-04-23T10:24:44.050430+0000","last_unstale":"2024-04-23T10:25:47.583513+0000","last_undegraded":"2024-04-23T10:25:47.583513+0000","last_fullsized":"2024-04-23T10:25:47.583513+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T10:24:43.025324+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T10:24:43.025324+0000","last_clean_scrub_stamp":"2024-04-23T10:24:43.025324+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T17:36:25.983147+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,11,10],"acting":[0,11,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3d","version":"0'0","reported_seq":17,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-23T10:25:47.583328+0000","last_change":"2024-04-23T10:24:44.054805+0000","last_active":"2024-04-23T10:25:47.583328+0000","last_peered":"2024-04-23T10:25:47.583328+0000","last_clean":"2024-04-23T10:25:47.583328+0000","last_became_active":"2024-04-23T10:24:44.054423+0000","last_became_peered":"2024-04-23T10:24:44.054423+0000","last_unstale":"2024-04-23T10:25:47.583328+0000","last_undegraded":"2024-04-23T10:25:47.583328+0000","last_fullsized":"2024-04-23T10:25:47.583328+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T10:24:43.025324+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T10:24:43.025324+0000","last_clean_scrub_stamp":"2024-04-23T10:24:43.025324+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T21:34:26.793624+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,1],"acting":[0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3c","version":"0'0","reported_seq":17,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-23T10:25:47.720418+0000","last_change":"2024-04-23T10:24:44.055966+0000","last_active":"2024-04-23T10:25:47.720418+0000","last_peered":"2024-04-23T10:25:47.720418+0000","last_clean":"2024-04-23T10:25:47.720418+0000","last_became_active":"2024-04-23T10:24:44.055028+0000","last_became_peered":"2024-04-23T10:24:44.055028+0000","last_unstale":"2024-04-23T10:25:47.720418+0000","last_undegraded":"2024-04-23T10:25:47.720418+0000","last_fullsized":"2024-04-23T10:25:47.720418+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T10:24:43.025324+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T10:24:43.025324+0000","last_clean_scrub_stamp":"2024-04-23T10:24:43.025324+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T19:41:04.594025+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,8,2],"acting":[4,8,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.3b","version":"0'0","reported_seq":17,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-23T10:25:47.587937+0000","last_change":"2024-04-23T10:24:44.046954+0000","last_active":"2024-04-23T10:25:47.587937+0000","last_peered":"2024-04-23T10:25:47.587937+0000","last_clean":"2024-04-23T10:25:47.587937+0000","last_became_active":"2024-04-23T10:24:44.046631+0000","last_became_peered":"2024-04-23T10:24:44.046631+0000","last_unstale":"2024-04-23T10:25:47.587937+0000","last_undegraded":"2024-04-23T10:25:47.587937+0000","last_fullsized":"2024-04-23T10:25:47.587937+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T10:24:43.025324+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T10:24:43.025324+0000","last_clean_scrub_stamp":"2024-04-23T10:24:43.025324+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T15:04:26.878894+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,3,6],"acting":[8,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.3a","version":"0'0","reported_seq":17,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-23T10:25:47.720134+0000","last_change":"2024-04-23T10:24:44.051550+0000","last_active":"2024-04-23T10:25:47.720134+0000","last_peered":"2024-04-23T10:25:47.720134+0000","last_clean":"2024-04-23T10:25:47.720134+0000","last_became_active":"2024-04-23T10:24:44.050469+0000","last_became_peered":"2024-04-23T10:24:44.050469+0000","last_unstale":"2024-04-23T10:25:47.720134+0000","last_undegraded":"2024-04-23T10:25:47.720134+0000","last_fullsized":"2024-04-23T10:25:47.720134+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T10:24:43.025324+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T10:24:43.025324+0000","last_clean_scrub_stamp":"2024-04-23T10:24:43.025324+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T21:13:42.469252+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,9,4],"acting":[10,9,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.39","version":"0'0","reported_seq":23,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-23T10:25:47.588865+0000","last_change":"2024-04-23T10:25:47.588865+0000","last_active":"2024-04-23T10:25:47.588865+0000","last_peered":"2024-04-23T10:25:47.588865+0000","last_clean":"2024-04-23T10:25:47.588865+0000","last_became_active":"2024-04-23T10:25:47.588571+0000","last_became_peered":"2024-04-23T10:25:47.588571+0000","last_unstale":"2024-04-23T10:25:47.588865+0000","last_undegraded":"2024-04-23T10:25:47.588865+0000","last_fullsized":"2024-04-23T10:25:47.588865+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T10:24:43.025324+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T10:24:43.025324+0000","last_clean_scrub_stamp":"2024-04-23T10:24:43.025324+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T17:14:00.026999+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,9,8],"acting":[7,9,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.38","version":"0'0","reported_seq":11,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-23T10:25:47.722416+0000","last_change":"2024-04-23T10:25:47.722416+0000","last_active":"2024-04-23T10:25:47.722416+0000","last_peered":"2024-04-23T10:25:47.722416+0000","last_clean":"2024-04-23T10:25:47.722416+0000","last_became_active":"2024-04-23T10:25:47.722015+0000","last_became_peered":"2024-04-23T10:25:47.722015+0000","last_unstale":"2024-04-23T10:25:47.722416+0000","last_undegraded":"2024-04-23T10:25:47.722416+0000","last_fullsized":"2024-04-23T10:25:47.722416+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T10:24:43.025324+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T10:24:43.025324+0000","last_clean_scrub_stamp":"2024-04-23T10:24:43.025324+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T17:02:57.881164+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,5,10],"acting":[9,5,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.37","version":"0'0","reported_seq":17,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-23T10:25:47.720708+0000","last_change":"2024-04-23T10:24:44.055261+0000","last_active":"2024-04-23T10:25:47.720708+0000","last_peered":"2024-04-23T10:25:47.720708+0000","last_clean":"2024-04-23T10:25:47.720708+0000","last_became_active":"2024-04-23T10:24:44.054405+0000","last_became_peered":"2024-04-23T10:24:44.054405+0000","last_unstale":"2024-04-23T10:25:47.720708+0000","last_undegraded":"2024-04-23T10:25:47.720708+0000","last_fullsized":"2024-04-23T10:25:47.720708+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T10:24:43.025324+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T10:24:43.025324+0000","last_clean_scrub_stamp":"2024-04-23T10:24:43.025324+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T19:30:30.306739+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,11],"acting":[1,5,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.36","version":"0'0","reported_seq":17,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-23T10:25:47.720053+0000","last_change":"2024-04-23T10:24:44.055854+0000","last_active":"2024-04-23T10:25:47.720053+0000","last_peered":"2024-04-23T10:25:47.720053+0000","last_clean":"2024-04-23T10:25:47.720053+0000","last_became_active":"2024-04-23T10:24:44.055275+0000","last_became_peered":"2024-04-23T10:24:44.055275+0000","last_unstale":"2024-04-23T10:25:47.720053+0000","last_undegraded":"2024-04-23T10:25:47.720053+0000","last_fullsized":"2024-04-23T10:25:47.720053+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T10:24:43.025324+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T10:24:43.025324+0000","last_clean_scrub_stamp":"2024-04-23T10:24:43.025324+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T21:48:25.934878+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,2,4],"acting":[10,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.35","version":"0'0","reported_seq":17,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-23T10:25:47.582659+0000","last_change":"2024-04-23T10:24:44.061314+0000","last_active":"2024-04-23T10:25:47.582659+0000","last_peered":"2024-04-23T10:25:47.582659+0000","last_clean":"2024-04-23T10:25:47.582659+0000","last_became_active":"2024-04-23T10:24:44.060981+0000","last_became_peered":"2024-04-23T10:24:44.060981+0000","last_unstale":"2024-04-23T10:25:47.582659+0000","last_undegraded":"2024-04-23T10:25:47.582659+0000","last_fullsized":"2024-04-23T10:25:47.582659+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T10:24:43.025324+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T10:24:43.025324+0000","last_clean_scrub_stamp":"2024-04-23T10:24:43.025324+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T11:48:08.387575+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,8,1],"acting":[7,8,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.34","version":"0'0","reported_seq":17,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-23T10:25:47.719901+0000","last_change":"2024-04-23T10:24:44.055335+0000","last_active":"2024-04-23T10:25:47.719901+0000","last_peered":"2024-04-23T10:25:47.719901+0000","last_clean":"2024-04-23T10:25:47.719901+0000","last_became_active":"2024-04-23T10:24:44.051257+0000","last_became_peered":"2024-04-23T10:24:44.051257+0000","last_unstale":"2024-04-23T10:25:47.719901+0000","last_undegraded":"2024-04-23T10:25:47.719901+0000","last_fullsized":"2024-04-23T10:25:47.719901+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T10:24:43.025324+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T10:24:43.025324+0000","last_clean_scrub_stamp":"2024-04-23T10:24:43.025324+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T11:57:45.999751+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,10,8],"acting":[4,10,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.33","version":"0'0","reported_seq":17,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-23T10:25:47.584677+0000","last_change":"2024-04-23T10:24:44.052078+0000","last_active":"2024-04-23T10:25:47.584677+0000","last_peered":"2024-04-23T10:25:47.584677+0000","last_clean":"2024-04-23T10:25:47.584677+0000","last_became_active":"2024-04-23T10:24:44.051747+0000","last_became_peered":"2024-04-23T10:24:44.051747+0000","last_unstale":"2024-04-23T10:25:47.584677+0000","last_undegraded":"2024-04-23T10:25:47.584677+0000","last_fullsized":"2024-04-23T10:25:47.584677+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T10:24:43.025324+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T10:24:43.025324+0000","last_clean_scrub_stamp":"2024-04-23T10:24:43.025324+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T19:14:16.495967+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,10,5],"acting":[6,10,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.32","version":"0'0","reported_seq":17,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-23T10:25:47.583123+0000","last_change":"2024-04-23T10:24:44.051994+0000","last_active":"2024-04-23T10:25:47.583123+0000","last_peered":"2024-04-23T10:25:47.583123+0000","last_clean":"2024-04-23T10:25:47.583123+0000","last_became_active":"2024-04-23T10:24:44.051638+0000","last_became_peered":"2024-04-23T10:24:44.051638+0000","last_unstale":"2024-04-23T10:25:47.583123+0000","last_undegraded":"2024-04-23T10:25:47.583123+0000","last_fullsized":"2024-04-23T10:25:47.583123+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T10:24:43.025324+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T10:24:43.025324+0000","last_clean_scrub_stamp":"2024-04-23T10:24:43.025324+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T14:28:30.447198+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,8,10],"acting":[0,8,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.31","version":"0'0","reported_seq":23,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-23T10:25:47.589538+0000","last_change":"2024-04-23T10:25:47.589538+0000","last_active":"2024-04-23T10:25:47.589538+0000","last_peered":"2024-04-23T10:25:47.589538+0000","last_clean":"2024-04-23T10:25:47.589538+0000","last_became_active":"2024-04-23T10:25:47.589172+0000","last_became_peered":"2024-04-23T10:25:47.589172+0000","last_unstale":"2024-04-23T10:25:47.589538+0000","last_undegraded":"2024-04-23T10:25:47.589538+0000","last_fullsized":"2024-04-23T10:25:47.589538+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T10:24:43.025324+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T10:24:43.025324+0000","last_clean_scrub_stamp":"2024-04-23T10:24:43.025324+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T19:47:34.267992+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,8,2],"acting":[0,8,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.30","version":"0'0","reported_seq":17,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-23T10:25:47.719565+0000","last_change":"2024-04-23T10:24:44.057154+0000","last_active":"2024-04-23T10:25:47.719565+0000","last_peered":"2024-04-23T10:25:47.719565+0000","last_clean":"2024-04-23T10:25:47.719565+0000","last_became_active":"2024-04-23T10:24:44.056836+0000","last_became_peered":"2024-04-23T10:24:44.056836+0000","last_unstale":"2024-04-23T10:25:47.719565+0000","last_undegraded":"2024-04-23T10:25:47.719565+0000","last_fullsized":"2024-04-23T10:25:47.719565+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T10:24:43.025324+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T10:24:43.025324+0000","last_clean_scrub_stamp":"2024-04-23T10:24:43.025324+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T13:48:59.107635+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,0],"acting":[1,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.2f","version":"0'0","reported_seq":17,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-23T10:25:47.584741+0000","last_change":"2024-04-23T10:24:44.051175+0000","last_active":"2024-04-23T10:25:47.584741+0000","last_peered":"2024-04-23T10:25:47.584741+0000","last_clean":"2024-04-23T10:25:47.584741+0000","last_became_active":"2024-04-23T10:24:44.050402+0000","last_became_peered":"2024-04-23T10:24:44.050402+0000","last_unstale":"2024-04-23T10:25:47.584741+0000","last_undegraded":"2024-04-23T10:25:47.584741+0000","last_fullsized":"2024-04-23T10:25:47.584741+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T10:24:43.025324+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T10:24:43.025324+0000","last_clean_scrub_stamp":"2024-04-23T10:24:43.025324+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T15:01:26.919701+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,1,0],"acting":[11,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.2e","version":"0'0","reported_seq":17,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-23T10:25:47.580911+0000","last_change":"2024-04-23T10:24:44.053390+0000","last_active":"2024-04-23T10:25:47.580911+0000","last_peered":"2024-04-23T10:25:47.580911+0000","last_clean":"2024-04-23T10:25:47.580911+0000","last_became_active":"2024-04-23T10:24:44.052970+0000","last_became_peered":"2024-04-23T10:24:44.052970+0000","last_unstale":"2024-04-23T10:25:47.580911+0000","last_undegraded":"2024-04-23T10:25:47.580911+0000","last_fullsized":"2024-04-23T10:25:47.580911+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T10:24:43.025324+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T10:24:43.025324+0000","last_clean_scrub_stamp":"2024-04-23T10:24:43.025324+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T13:27:55.704328+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,1,10],"acting":[9,1,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.2d","version":"0'0","reported_seq":17,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-23T10:25:47.582595+0000","last_change":"2024-04-23T10:24:44.049307+0000","last_active":"2024-04-23T10:25:47.582595+0000","last_peered":"2024-04-23T10:25:47.582595+0000","last_clean":"2024-04-23T10:25:47.582595+0000","last_became_active":"2024-04-23T10:24:44.048981+0000","last_became_peered":"2024-04-23T10:24:44.048981+0000","last_unstale":"2024-04-23T10:25:47.582595+0000","last_undegraded":"2024-04-23T10:25:47.582595+0000","last_fullsized":"2024-04-23T10:25:47.582595+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T10:24:43.025324+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T10:24:43.025324+0000","last_clean_scrub_stamp":"2024-04-23T10:24:43.025324+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T12:42:21.804591+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,6],"acting":[2,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.1f","version":"87'9","reported_seq":29,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-23T10:25:47.580889+0000","last_change":"2024-04-23T10:24:43.045804+0000","last_active":"2024-04-23T10:25:47.580889+0000","last_peered":"2024-04-23T10:25:47.580889+0000","last_clean":"2024-04-23T10:25:47.580889+0000","last_became_active":"2024-04-23T10:24:43.045135+0000","last_became_peered":"2024-04-23T10:24:43.045135+0000","last_unstale":"2024-04-23T10:25:47.580889+0000","last_undegraded":"2024-04-23T10:25:47.580889+0000","last_fullsized":"2024-04-23T10:25:47.580889+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T10:24:42.023127+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T10:24:42.023127+0000","last_clean_scrub_stamp":"2024-04-23T10:24:42.023127+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T20:27:28.421644+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,7,8],"acting":[9,7,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.1e","version":"0'0","reported_seq":17,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-23T10:25:47.586405+0000","last_change":"2024-04-23T10:24:44.055008+0000","last_active":"2024-04-23T10:25:47.586405+0000","last_peered":"2024-04-23T10:25:47.586405+0000","last_clean":"2024-04-23T10:25:47.586405+0000","last_became_active":"2024-04-23T10:24:44.054682+0000","last_became_peered":"2024-04-23T10:24:44.054682+0000","last_unstale":"2024-04-23T10:25:47.586405+0000","last_undegraded":"2024-04-23T10:25:47.586405+0000","last_fullsized":"2024-04-23T10:25:47.586405+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T10:24:43.025324+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T10:24:43.025324+0000","last_clean_scrub_stamp":"2024-04-23T10:24:43.025324+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T15:49:58.394958+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,5],"acting":[3,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.1e","version":"87'2","reported_seq":21,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-23T10:25:47.586522+0000","last_change":"2024-04-23T10:24:43.052303+0000","last_active":"2024-04-23T10:25:47.586522+0000","last_peered":"2024-04-23T10:25:47.586522+0000","last_clean":"2024-04-23T10:25:47.586522+0000","last_became_active":"2024-04-23T10:24:43.052010+0000","last_became_peered":"2024-04-23T10:24:43.052010+0000","last_unstale":"2024-04-23T10:25:47.586522+0000","last_undegraded":"2024-04-23T10:25:47.586522+0000","last_fullsized":"2024-04-23T10:25:47.586522+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T10:24:42.023127+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T10:24:42.023127+0000","last_clean_scrub_stamp":"2024-04-23T10:24:42.023127+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T13:03:15.440610+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,5],"acting":[3,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1f","version":"0'0","reported_seq":17,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-23T10:25:47.583624+0000","last_change":"2024-04-23T10:24:44.050749+0000","last_active":"2024-04-23T10:25:47.583624+0000","last_peered":"2024-04-23T10:25:47.583624+0000","last_clean":"2024-04-23T10:25:47.583624+0000","last_became_active":"2024-04-23T10:24:44.049780+0000","last_became_peered":"2024-04-23T10:24:44.049780+0000","last_unstale":"2024-04-23T10:25:47.583624+0000","last_undegraded":"2024-04-23T10:25:47.583624+0000","last_fullsized":"2024-04-23T10:25:47.583624+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T10:24:43.025324+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T10:24:43.025324+0000","last_clean_scrub_stamp":"2024-04-23T10:24:43.025324+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T16:32:37.963776+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,9],"acting":[0,1,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1d","version":"87'4","reported_seq":17,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-23T10:25:47.592399+0000","last_change":"2024-04-23T10:25:47.591939+0000","last_active":"2024-04-23T10:25:47.592399+0000","last_peered":"2024-04-23T10:25:47.592399+0000","last_clean":"2024-04-23T10:25:47.592399+0000","last_became_active":"2024-04-23T10:25:47.586626+0000","last_became_peered":"2024-04-23T10:25:47.586626+0000","last_unstale":"2024-04-23T10:25:47.592399+0000","last_undegraded":"2024-04-23T10:25:47.592399+0000","last_fullsized":"2024-04-23T10:25:47.592399+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T10:24:42.023127+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T10:24:42.023127+0000","last_clean_scrub_stamp":"2024-04-23T10:24:42.023127+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T20:55:38.216483+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":10,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,2,7],"acting":[11,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.1c","version":"0'0","reported_seq":17,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-23T10:25:47.720068+0000","last_change":"2024-04-23T10:24:44.051606+0000","last_active":"2024-04-23T10:25:47.720068+0000","last_peered":"2024-04-23T10:25:47.720068+0000","last_clean":"2024-04-23T10:25:47.720068+0000","last_became_active":"2024-04-23T10:24:44.051191+0000","last_became_peered":"2024-04-23T10:24:44.051191+0000","last_unstale":"2024-04-23T10:25:47.720068+0000","last_undegraded":"2024-04-23T10:25:47.720068+0000","last_fullsized":"2024-04-23T10:25:47.720068+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T10:24:43.025324+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T10:24:43.025324+0000","last_clean_scrub_stamp":"2024-04-23T10:24:43.025324+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T16:21:51.706066+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,10,6],"acting":[1,10,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1c","version":"86'2","reported_seq":21,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-23T10:25:47.588287+0000","last_change":"2024-04-23T10:24:43.048594+0000","last_active":"2024-04-23T10:25:47.588287+0000","last_peered":"2024-04-23T10:25:47.588287+0000","last_clean":"2024-04-23T10:25:47.588287+0000","last_became_active":"2024-04-23T10:24:43.048276+0000","last_became_peered":"2024-04-23T10:24:43.048276+0000","last_unstale":"2024-04-23T10:25:47.588287+0000","last_undegraded":"2024-04-23T10:25:47.588287+0000","last_fullsized":"2024-04-23T10:25:47.588287+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T10:24:42.023127+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T10:24:42.023127+0000","last_clean_scrub_stamp":"2024-04-23T10:24:42.023127+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T12:20:40.196674+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,1,5],"acting":[8,1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.1d","version":"0'0","reported_seq":17,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-23T10:25:47.720397+0000","last_change":"2024-04-23T10:24:44.050970+0000","last_active":"2024-04-23T10:25:47.720397+0000","last_peered":"2024-04-23T10:25:47.720397+0000","last_clean":"2024-04-23T10:25:47.720397+0000","last_became_active":"2024-04-23T10:24:44.050640+0000","last_became_peered":"2024-04-23T10:24:44.050640+0000","last_unstale":"2024-04-23T10:25:47.720397+0000","last_undegraded":"2024-04-23T10:25:47.720397+0000","last_fullsized":"2024-04-23T10:25:47.720397+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T10:24:43.025324+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T10:24:43.025324+0000","last_clean_scrub_stamp":"2024-04-23T10:24:43.025324+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T21:03:24.570290+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,1],"acting":[5,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.1b","version":"87'6","reported_seq":94,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-23T10:25:52.417262+0000","last_change":"2024-04-23T10:24:43.051742+0000","last_active":"2024-04-23T10:25:52.417262+0000","last_peered":"2024-04-23T10:25:52.417262+0000","last_clean":"2024-04-23T10:25:52.417262+0000","last_became_active":"2024-04-23T10:24:43.051420+0000","last_became_peered":"2024-04-23T10:24:43.051420+0000","last_unstale":"2024-04-23T10:25:52.417262+0000","last_undegraded":"2024-04-23T10:25:52.417262+0000","last_fullsized":"2024-04-23T10:25:52.417262+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T10:24:42.023127+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T10:24:42.023127+0000","last_clean_scrub_stamp":"2024-04-23T10:24:42.023127+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T17:54:37.182731+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1013,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":69,"num_read_kb":3,"num_write":3,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,7],"acting":[5,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.1a","version":"0'0","reported_seq":17,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-23T10:25:47.587950+0000","last_change":"2024-04-23T10:24:44.049314+0000","last_active":"2024-04-23T10:25:47.587950+0000","last_peered":"2024-04-23T10:25:47.587950+0000","last_clean":"2024-04-23T10:25:47.587950+0000","last_became_active":"2024-04-23T10:24:44.048352+0000","last_became_peered":"2024-04-23T10:24:44.048352+0000","last_unstale":"2024-04-23T10:25:47.587950+0000","last_undegraded":"2024-04-23T10:25:47.587950+0000","last_fullsized":"2024-04-23T10:25:47.587950+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T10:24:43.025324+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T10:24:43.025324+0000","last_clean_scrub_stamp":"2024-04-23T10:24:43.025324+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T16:57:29.957716+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,9,5],"acting":[8,9,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"2.1a","version":"87'3","reported_seq":22,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-23T10:25:47.720011+0000","last_change":"2024-04-23T10:24:43.051362+0000","last_active":"2024-04-23T10:25:47.720011+0000","last_peered":"2024-04-23T10:25:47.720011+0000","last_clean":"2024-04-23T10:25:47.720011+0000","last_became_active":"2024-04-23T10:24:43.051045+0000","last_became_peered":"2024-04-23T10:24:43.051045+0000","last_unstale":"2024-04-23T10:25:47.720011+0000","last_undegraded":"2024-04-23T10:25:47.720011+0000","last_fullsized":"2024-04-23T10:25:47.720011+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T10:24:42.023127+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T10:24:42.023127+0000","last_clean_scrub_stamp":"2024-04-23T10:24:42.023127+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T15:40:53.618281+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,8,6],"acting":[10,8,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.1b","version":"0'0","reported_seq":17,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-23T10:25:47.720118+0000","last_change":"2024-04-23T10:24:44.040323+0000","last_active":"2024-04-23T10:25:47.720118+0000","last_peered":"2024-04-23T10:25:47.720118+0000","last_clean":"2024-04-23T10:25:47.720118+0000","last_became_active":"2024-04-23T10:24:44.039995+0000","last_became_peered":"2024-04-23T10:24:44.039995+0000","last_unstale":"2024-04-23T10:25:47.720118+0000","last_undegraded":"2024-04-23T10:25:47.720118+0000","last_fullsized":"2024-04-23T10:25:47.720118+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T10:24:43.025324+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T10:24:43.025324+0000","last_clean_scrub_stamp":"2024-04-23T10:24:43.025324+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T16:38:48.756776+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,3],"acting":[5,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.19","version":"87'2","reported_seq":21,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-23T10:25:47.584155+0000","last_change":"2024-04-23T10:24:43.048754+0000","last_active":"2024-04-23T10:25:47.584155+0000","last_peered":"2024-04-23T10:25:47.584155+0000","last_clean":"2024-04-23T10:25:47.584155+0000","last_became_active":"2024-04-23T10:24:43.047962+0000","last_became_peered":"2024-04-23T10:24:43.047962+0000","last_unstale":"2024-04-23T10:25:47.584155+0000","last_undegraded":"2024-04-23T10:25:47.584155+0000","last_fullsized":"2024-04-23T10:25:47.584155+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T10:24:42.023127+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T10:24:42.023127+0000","last_clean_scrub_stamp":"2024-04-23T10:24:42.023127+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T21:08:38.487185+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,9],"acting":[6,4,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.18","version":"0'0","reported_seq":17,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-23T10:25:47.585662+0000","last_change":"2024-04-23T10:24:44.045022+0000","last_active":"2024-04-23T10:25:47.585662+0000","last_peered":"2024-04-23T10:25:47.585662+0000","last_clean":"2024-04-23T10:25:47.585662+0000","last_became_active":"2024-04-23T10:24:44.044590+0000","last_became_peered":"2024-04-23T10:24:44.044590+0000","last_unstale":"2024-04-23T10:25:47.585662+0000","last_undegraded":"2024-04-23T10:25:47.585662+0000","last_fullsized":"2024-04-23T10:25:47.585662+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T10:24:43.025324+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T10:24:43.025324+0000","last_clean_scrub_stamp":"2024-04-23T10:24:43.025324+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T13:20:49.892013+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,11,6],"acting":[3,11,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.18","version":"87'1","reported_seq":20,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-23T10:25:47.720010+0000","last_change":"2024-04-23T10:24:43.050170+0000","last_active":"2024-04-23T10:25:47.720010+0000","last_peered":"2024-04-23T10:25:47.720010+0000","last_clean":"2024-04-23T10:25:47.720010+0000","last_became_active":"2024-04-23T10:24:43.049070+0000","last_became_peered":"2024-04-23T10:24:43.049070+0000","last_unstale":"2024-04-23T10:25:47.720010+0000","last_undegraded":"2024-04-23T10:25:47.720010+0000","last_fullsized":"2024-04-23T10:25:47.720010+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T10:24:42.023127+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T10:24:42.023127+0000","last_clean_scrub_stamp":"2024-04-23T10:24:42.023127+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T19:34:13.983628+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,7],"acting":[5,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.19","version":"0'0","reported_seq":17,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-23T10:25:47.580673+0000","last_change":"2024-04-23T10:24:44.047165+0000","last_active":"2024-04-23T10:25:47.580673+0000","last_peered":"2024-04-23T10:25:47.580673+0000","last_clean":"2024-04-23T10:25:47.580673+0000","last_became_active":"2024-04-23T10:24:44.046776+0000","last_became_peered":"2024-04-23T10:24:44.046776+0000","last_unstale":"2024-04-23T10:25:47.580673+0000","last_undegraded":"2024-04-23T10:25:47.580673+0000","last_fullsized":"2024-04-23T10:25:47.580673+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T10:24:43.025324+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T10:24:43.025324+0000","last_clean_scrub_stamp":"2024-04-23T10:24:43.025324+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T18:11:57.181747+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,3,10],"acting":[9,3,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"2.17","version":"87'3","reported_seq":91,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-23T10:25:55.421165+0000","last_change":"2024-04-23T10:24:43.049785+0000","last_active":"2024-04-23T10:25:55.421165+0000","last_peered":"2024-04-23T10:25:55.421165+0000","last_clean":"2024-04-23T10:25:55.421165+0000","last_became_active":"2024-04-23T10:24:43.049402+0000","last_became_peered":"2024-04-23T10:24:43.049402+0000","last_unstale":"2024-04-23T10:25:55.421165+0000","last_undegraded":"2024-04-23T10:25:55.421165+0000","last_fullsized":"2024-04-23T10:25:55.421165+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T10:24:42.023127+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T10:24:42.023127+0000","last_clean_scrub_stamp":"2024-04-23T10:24:42.023127+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T14:20:13.774082+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":69,"num_read_kb":69,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,5,9],"acting":[10,5,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.16","version":"0'0","reported_seq":17,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-23T10:25:47.720748+0000","last_change":"2024-04-23T10:24:44.051768+0000","last_active":"2024-04-23T10:25:47.720748+0000","last_peered":"2024-04-23T10:25:47.720748+0000","last_clean":"2024-04-23T10:25:47.720748+0000","last_became_active":"2024-04-23T10:24:44.051228+0000","last_became_peered":"2024-04-23T10:24:44.051228+0000","last_unstale":"2024-04-23T10:25:47.720748+0000","last_undegraded":"2024-04-23T10:25:47.720748+0000","last_fullsized":"2024-04-23T10:25:47.720748+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T10:24:43.025324+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T10:24:43.025324+0000","last_clean_scrub_stamp":"2024-04-23T10:24:43.025324+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T14:09:47.649083+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,10,9],"acting":[1,10,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.16","version":"87'2","reported_seq":21,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-23T10:25:47.582738+0000","last_change":"2024-04-23T10:24:43.043944+0000","last_active":"2024-04-23T10:25:47.582738+0000","last_peered":"2024-04-23T10:25:47.582738+0000","last_clean":"2024-04-23T10:25:47.582738+0000","last_became_active":"2024-04-23T10:24:43.043648+0000","last_became_peered":"2024-04-23T10:24:43.043648+0000","last_unstale":"2024-04-23T10:25:47.582738+0000","last_undegraded":"2024-04-23T10:25:47.582738+0000","last_fullsized":"2024-04-23T10:25:47.582738+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T10:24:42.023127+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T10:24:42.023127+0000","last_clean_scrub_stamp":"2024-04-23T10:24:42.023127+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T13:41:15.724662+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,2],"acting":[7,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.17","version":"0'0","reported_seq":23,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-23T10:25:47.721534+0000","last_change":"2024-04-23T10:25:47.721534+0000","last_active":"2024-04-23T10:25:47.721534+0000","last_peered":"2024-04-23T10:25:47.721534+0000","last_clean":"2024-04-23T10:25:47.721534+0000","last_became_active":"2024-04-23T10:25:47.721130+0000","last_became_peered":"2024-04-23T10:25:47.721130+0000","last_unstale":"2024-04-23T10:25:47.721534+0000","last_undegraded":"2024-04-23T10:25:47.721534+0000","last_fullsized":"2024-04-23T10:25:47.721534+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T10:24:43.025324+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T10:24:43.025324+0000","last_clean_scrub_stamp":"2024-04-23T10:24:43.025324+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T21:21:26.537486+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,5],"acting":[0,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.15","version":"87'4","reported_seq":24,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-23T10:25:47.720578+0000","last_change":"2024-04-23T10:24:43.048517+0000","last_active":"2024-04-23T10:25:47.720578+0000","last_peered":"2024-04-23T10:25:47.720578+0000","last_clean":"2024-04-23T10:25:47.720578+0000","last_became_active":"2024-04-23T10:24:43.048183+0000","last_became_peered":"2024-04-23T10:24:43.048183+0000","last_unstale":"2024-04-23T10:25:47.720578+0000","last_undegraded":"2024-04-23T10:25:47.720578+0000","last_fullsized":"2024-04-23T10:25:47.720578+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T10:24:42.023127+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T10:24:42.023127+0000","last_clean_scrub_stamp":"2024-04-23T10:24:42.023127+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T20:50:15.017833+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":672,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":5,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,10],"acting":[5,0,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.14","version":"0'0","reported_seq":17,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-23T10:25:47.720284+0000","last_change":"2024-04-23T10:24:44.055598+0000","last_active":"2024-04-23T10:25:47.720284+0000","last_peered":"2024-04-23T10:25:47.720284+0000","last_clean":"2024-04-23T10:25:47.720284+0000","last_became_active":"2024-04-23T10:24:44.053787+0000","last_became_peered":"2024-04-23T10:24:44.053787+0000","last_unstale":"2024-04-23T10:25:47.720284+0000","last_undegraded":"2024-04-23T10:25:47.720284+0000","last_fullsized":"2024-04-23T10:25:47.720284+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T10:24:43.025324+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T10:24:43.025324+0000","last_clean_scrub_stamp":"2024-04-23T10:24:43.025324+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T20:04:21.603306+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,8,4],"acting":[1,8,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.14","version":"87'28","reported_seq":193,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-23T10:25:56.434612+0000","last_change":"2024-04-23T10:24:43.044601+0000","last_active":"2024-04-23T10:25:56.434612+0000","last_peered":"2024-04-23T10:25:56.434612+0000","last_clean":"2024-04-23T10:25:56.434612+0000","last_became_active":"2024-04-23T10:24:43.044238+0000","last_became_peered":"2024-04-23T10:24:43.044238+0000","last_unstale":"2024-04-23T10:25:56.434612+0000","last_undegraded":"2024-04-23T10:25:56.434612+0000","last_fullsized":"2024-04-23T10:25:56.434612+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T10:24:42.023127+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T10:24:42.023127+0000","last_clean_scrub_stamp":"2024-04-23T10:24:42.023127+0000","objects_scrubbed":0,"log_size":28,"log_dups_size":0,"ondisk_log_size":28,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T12:53:26.275771+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":61046,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":146,"num_read_kb":132,"num_write":29,"num_write_kb":69,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,1,2],"acting":[10,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.15","version":"0'0","reported_seq":17,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-23T10:25:47.588308+0000","last_change":"2024-04-23T10:24:44.051375+0000","last_active":"2024-04-23T10:25:47.588308+0000","last_peered":"2024-04-23T10:25:47.588308+0000","last_clean":"2024-04-23T10:25:47.588308+0000","last_became_active":"2024-04-23T10:24:44.050924+0000","last_became_peered":"2024-04-23T10:24:44.050924+0000","last_unstale":"2024-04-23T10:25:47.588308+0000","last_undegraded":"2024-04-23T10:25:47.588308+0000","last_fullsized":"2024-04-23T10:25:47.588308+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T10:24:43.025324+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T10:24:43.025324+0000","last_clean_scrub_stamp":"2024-04-23T10:24:43.025324+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T21:50:57.534760+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,11,7],"acting":[8,11,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"2.13","version":"87'6","reported_seq":26,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-23T10:25:47.582395+0000","last_change":"2024-04-23T10:24:43.044457+0000","last_active":"2024-04-23T10:25:47.582395+0000","last_peered":"2024-04-23T10:25:47.582395+0000","last_clean":"2024-04-23T10:25:47.582395+0000","last_became_active":"2024-04-23T10:24:43.043956+0000","last_became_peered":"2024-04-23T10:24:43.043956+0000","last_unstale":"2024-04-23T10:25:47.582395+0000","last_undegraded":"2024-04-23T10:25:47.582395+0000","last_fullsized":"2024-04-23T10:25:47.582395+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T10:24:42.023127+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T10:24:42.023127+0000","last_clean_scrub_stamp":"2024-04-23T10:24:42.023127+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T20:16:08.186729+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":9,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,10,3],"acting":[2,10,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.12","version":"0'0","reported_seq":23,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-23T10:25:47.587240+0000","last_change":"2024-04-23T10:25:47.587240+0000","last_active":"2024-04-23T10:25:47.587240+0000","last_peered":"2024-04-23T10:25:47.587240+0000","last_clean":"2024-04-23T10:25:47.587240+0000","last_became_active":"2024-04-23T10:25:47.586659+0000","last_became_peered":"2024-04-23T10:25:47.586659+0000","last_unstale":"2024-04-23T10:25:47.587240+0000","last_undegraded":"2024-04-23T10:25:47.587240+0000","last_fullsized":"2024-04-23T10:25:47.587240+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T10:24:43.025324+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T10:24:43.025324+0000","last_clean_scrub_stamp":"2024-04-23T10:24:43.025324+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T19:55:03.256183+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,9],"acting":[6,7,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.12","version":"87'3","reported_seq":22,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-23T10:25:47.719998+0000","last_change":"2024-04-23T10:24:43.052181+0000","last_active":"2024-04-23T10:25:47.719998+0000","last_peered":"2024-04-23T10:25:47.719998+0000","last_clean":"2024-04-23T10:25:47.719998+0000","last_became_active":"2024-04-23T10:24:43.051793+0000","last_became_peered":"2024-04-23T10:24:43.051793+0000","last_unstale":"2024-04-23T10:25:47.719998+0000","last_undegraded":"2024-04-23T10:25:47.719998+0000","last_fullsized":"2024-04-23T10:25:47.719998+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T10:24:42.023127+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T10:24:42.023127+0000","last_clean_scrub_stamp":"2024-04-23T10:24:42.023127+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T20:57:59.015115+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,7],"acting":[5,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.13","version":"0'0","reported_seq":17,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-23T10:25:47.582606+0000","last_change":"2024-04-23T10:24:44.062689+0000","last_active":"2024-04-23T10:25:47.582606+0000","last_peered":"2024-04-23T10:25:47.582606+0000","last_clean":"2024-04-23T10:25:47.582606+0000","last_became_active":"2024-04-23T10:24:44.062331+0000","last_became_peered":"2024-04-23T10:24:44.062331+0000","last_unstale":"2024-04-23T10:25:47.582606+0000","last_undegraded":"2024-04-23T10:25:47.582606+0000","last_fullsized":"2024-04-23T10:25:47.582606+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T10:24:43.025324+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T10:24:43.025324+0000","last_clean_scrub_stamp":"2024-04-23T10:24:43.025324+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T14:06:04.989156+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,6],"acting":[7,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.11","version":"87'4","reported_seq":96,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-23T10:25:56.423831+0000","last_change":"2024-04-23T10:24:43.043901+0000","last_active":"2024-04-23T10:25:56.423831+0000","last_peered":"2024-04-23T10:25:56.423831+0000","last_clean":"2024-04-23T10:25:56.423831+0000","last_became_active":"2024-04-23T10:24:43.043598+0000","last_became_peered":"2024-04-23T10:24:43.043598+0000","last_unstale":"2024-04-23T10:25:56.423831+0000","last_undegraded":"2024-04-23T10:25:56.423831+0000","last_fullsized":"2024-04-23T10:25:56.423831+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T10:24:42.023127+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T10:24:42.023127+0000","last_clean_scrub_stamp":"2024-04-23T10:24:42.023127+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T14:00:06.338084+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":979,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":74,"num_read_kb":4,"num_write":3,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,10],"acting":[4,7,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.10","version":"0'0","reported_seq":17,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-23T10:25:47.582241+0000","last_change":"2024-04-23T10:24:44.058773+0000","last_active":"2024-04-23T10:25:47.582241+0000","last_peered":"2024-04-23T10:25:47.582241+0000","last_clean":"2024-04-23T10:25:47.582241+0000","last_became_active":"2024-04-23T10:24:44.058420+0000","last_became_peered":"2024-04-23T10:24:44.058420+0000","last_unstale":"2024-04-23T10:25:47.582241+0000","last_undegraded":"2024-04-23T10:25:47.582241+0000","last_fullsized":"2024-04-23T10:25:47.582241+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T10:24:43.025324+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T10:24:43.025324+0000","last_clean_scrub_stamp":"2024-04-23T10:24:43.025324+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T12:44:30.023047+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,10],"acting":[7,5,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.10","version":"87'15","reported_seq":35,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-23T10:25:47.587839+0000","last_change":"2024-04-23T10:24:43.044700+0000","last_active":"2024-04-23T10:25:47.587839+0000","last_peered":"2024-04-23T10:25:47.587839+0000","last_clean":"2024-04-23T10:25:47.587839+0000","last_became_active":"2024-04-23T10:24:43.044394+0000","last_became_peered":"2024-04-23T10:24:43.044394+0000","last_unstale":"2024-04-23T10:25:47.587839+0000","last_undegraded":"2024-04-23T10:25:47.587839+0000","last_fullsized":"2024-04-23T10:25:47.587839+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T10:24:42.023127+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T10:24:42.023127+0000","last_clean_scrub_stamp":"2024-04-23T10:24:42.023127+0000","objects_scrubbed":0,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T15:16:41.367517+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":112,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":20,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":6,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,0,2],"acting":[8,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.11","version":"0'0","reported_seq":17,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-23T10:25:47.719971+0000","last_change":"2024-04-23T10:24:44.049739+0000","last_active":"2024-04-23T10:25:47.719971+0000","last_peered":"2024-04-23T10:25:47.719971+0000","last_clean":"2024-04-23T10:25:47.719971+0000","last_became_active":"2024-04-23T10:24:44.049414+0000","last_became_peered":"2024-04-23T10:24:44.049414+0000","last_unstale":"2024-04-23T10:25:47.719971+0000","last_undegraded":"2024-04-23T10:25:47.719971+0000","last_fullsized":"2024-04-23T10:25:47.719971+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T10:24:43.025324+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T10:24:43.025324+0000","last_clean_scrub_stamp":"2024-04-23T10:24:43.025324+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T12:51:02.645259+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,1],"acting":[5,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.f","version":"87'2","reported_seq":22,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-23T10:25:47.588324+0000","last_change":"2024-04-23T10:24:43.048884+0000","last_active":"2024-04-23T10:25:47.588324+0000","last_peered":"2024-04-23T10:25:47.588324+0000","last_clean":"2024-04-23T10:25:47.588324+0000","last_became_active":"2024-04-23T10:24:43.048583+0000","last_became_peered":"2024-04-23T10:24:43.048583+0000","last_unstale":"2024-04-23T10:25:47.588324+0000","last_undegraded":"2024-04-23T10:25:47.588324+0000","last_fullsized":"2024-04-23T10:25:47.588324+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T10:24:42.023127+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T10:24:42.023127+0000","last_clean_scrub_stamp":"2024-04-23T10:24:42.023127+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T17:55:57.869700+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,5,4],"acting":[8,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.e","version":"0'0","reported_seq":23,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-23T10:25:47.722078+0000","last_change":"2024-04-23T10:25:47.722078+0000","last_active":"2024-04-23T10:25:47.722078+0000","last_peered":"2024-04-23T10:25:47.722078+0000","last_clean":"2024-04-23T10:25:47.722078+0000","last_became_active":"2024-04-23T10:25:47.721730+0000","last_became_peered":"2024-04-23T10:25:47.721730+0000","last_unstale":"2024-04-23T10:25:47.722078+0000","last_undegraded":"2024-04-23T10:25:47.722078+0000","last_fullsized":"2024-04-23T10:25:47.722078+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T10:24:43.025324+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T10:24:43.025324+0000","last_clean_scrub_stamp":"2024-04-23T10:24:43.025324+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T18:18:40.916492+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,9,4],"acting":[11,9,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":19,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-23T10:25:47.580777+0000","last_change":"2024-04-23T10:24:43.044375+0000","last_active":"2024-04-23T10:25:47.580777+0000","last_peered":"2024-04-23T10:25:47.580777+0000","last_clean":"2024-04-23T10:25:47.580777+0000","last_became_active":"2024-04-23T10:24:43.043914+0000","last_became_peered":"2024-04-23T10:24:43.043914+0000","last_unstale":"2024-04-23T10:25:47.580777+0000","last_undegraded":"2024-04-23T10:25:47.580777+0000","last_fullsized":"2024-04-23T10:25:47.580777+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T10:24:42.023127+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T10:24:42.023127+0000","last_clean_scrub_stamp":"2024-04-23T10:24:42.023127+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T16:22:44.165156+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,2,0],"acting":[9,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.5","version":"87'1","reported_seq":18,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-23T10:25:47.720416+0000","last_change":"2024-04-23T10:24:44.057272+0000","last_active":"2024-04-23T10:25:47.720416+0000","last_peered":"2024-04-23T10:25:47.720416+0000","last_clean":"2024-04-23T10:25:47.720416+0000","last_became_active":"2024-04-23T10:24:44.056968+0000","last_became_peered":"2024-04-23T10:24:44.056968+0000","last_unstale":"2024-04-23T10:25:47.720416+0000","last_undegraded":"2024-04-23T10:25:47.720416+0000","last_fullsized":"2024-04-23T10:25:47.720416+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T10:24:43.025324+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T10:24:43.025324+0000","last_clean_scrub_stamp":"2024-04-23T10:24:43.025324+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T11:10:09.021845+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":117,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,3,6],"acting":[10,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"2.2","version":"87'1","reported_seq":20,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-23T10:25:47.582434+0000","last_change":"2024-04-23T10:24:43.051822+0000","last_active":"2024-04-23T10:25:47.582434+0000","last_peered":"2024-04-23T10:25:47.582434+0000","last_clean":"2024-04-23T10:25:47.582434+0000","last_became_active":"2024-04-23T10:24:43.051491+0000","last_became_peered":"2024-04-23T10:24:43.051491+0000","last_unstale":"2024-04-23T10:25:47.582434+0000","last_undegraded":"2024-04-23T10:25:47.582434+0000","last_fullsized":"2024-04-23T10:25:47.582434+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T10:24:42.023127+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T10:24:42.023127+0000","last_clean_scrub_stamp":"2024-04-23T10:24:42.023127+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T13:49:23.109064+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,1],"acting":[7,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.3","version":"0'0","reported_seq":17,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-23T10:25:47.720861+0000","last_change":"2024-04-23T10:24:44.053604+0000","last_active":"2024-04-23T10:25:47.720861+0000","last_peered":"2024-04-23T10:25:47.720861+0000","last_clean":"2024-04-23T10:25:47.720861+0000","last_became_active":"2024-04-23T10:24:44.053147+0000","last_became_peered":"2024-04-23T10:24:44.053147+0000","last_unstale":"2024-04-23T10:25:47.720861+0000","last_undegraded":"2024-04-23T10:25:47.720861+0000","last_fullsized":"2024-04-23T10:25:47.720861+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T10:24:43.025324+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T10:24:43.025324+0000","last_clean_scrub_stamp":"2024-04-23T10:24:43.025324+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T21:58:38.981282+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,4,0],"acting":[10,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.0","version":"0'0","reported_seq":17,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-23T10:25:47.720397+0000","last_change":"2024-04-23T10:24:44.052564+0000","last_active":"2024-04-23T10:25:47.720397+0000","last_peered":"2024-04-23T10:25:47.720397+0000","last_clean":"2024-04-23T10:25:47.720397+0000","last_became_active":"2024-04-23T10:24:44.052201+0000","last_became_peered":"2024-04-23T10:24:44.052201+0000","last_unstale":"2024-04-23T10:25:47.720397+0000","last_undegraded":"2024-04-23T10:25:47.720397+0000","last_fullsized":"2024-04-23T10:25:47.720397+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T10:24:43.025324+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T10:24:43.025324+0000","last_clean_scrub_stamp":"2024-04-23T10:24:43.025324+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T16:25:54.725347+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,4],"acting":[5,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.1","version":"87'4","reported_seq":88,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-23T10:25:51.415763+0000","last_change":"2024-04-23T10:24:43.049067+0000","last_active":"2024-04-23T10:25:51.415763+0000","last_peered":"2024-04-23T10:25:51.415763+0000","last_clean":"2024-04-23T10:25:51.415763+0000","last_became_active":"2024-04-23T10:24:43.048400+0000","last_became_peered":"2024-04-23T10:24:43.048400+0000","last_unstale":"2024-04-23T10:25:51.415763+0000","last_undegraded":"2024-04-23T10:25:51.415763+0000","last_fullsized":"2024-04-23T10:25:51.415763+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T10:24:42.023127+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T10:24:42.023127+0000","last_clean_scrub_stamp":"2024-04-23T10:24:42.023127+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T12:17:02.439054+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":65,"num_read_kb":65,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,0],"acting":[6,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.2","version":"0'0","reported_seq":17,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-23T10:25:47.586431+0000","last_change":"2024-04-23T10:24:44.059114+0000","last_active":"2024-04-23T10:25:47.586431+0000","last_peered":"2024-04-23T10:25:47.586431+0000","last_clean":"2024-04-23T10:25:47.586431+0000","last_became_active":"2024-04-23T10:24:44.058618+0000","last_became_peered":"2024-04-23T10:24:44.058618+0000","last_unstale":"2024-04-23T10:25:47.586431+0000","last_undegraded":"2024-04-23T10:25:47.586431+0000","last_fullsized":"2024-04-23T10:25:47.586431+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T10:24:43.025324+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T10:24:43.025324+0000","last_clean_scrub_stamp":"2024-04-23T10:24:43.025324+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T20:18:09.707062+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,1],"acting":[3,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.3","version":"87'3","reported_seq":24,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-23T10:25:47.719834+0000","last_change":"2024-04-23T10:24:43.039725+0000","last_active":"2024-04-23T10:25:47.719834+0000","last_peered":"2024-04-23T10:25:47.719834+0000","last_clean":"2024-04-23T10:25:47.719834+0000","last_became_active":"2024-04-23T10:24:43.039187+0000","last_became_peered":"2024-04-23T10:24:43.039187+0000","last_unstale":"2024-04-23T10:25:47.719834+0000","last_undegraded":"2024-04-23T10:25:47.719834+0000","last_fullsized":"2024-04-23T10:25:47.719834+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T10:24:42.023127+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T10:24:42.023127+0000","last_clean_scrub_stamp":"2024-04-23T10:24:42.023127+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T17:20:50.844878+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":3,"num_read_kb":3,"num_write":4,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,10,9],"acting":[4,10,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.0","version":"22'137","reported_seq":350,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-23T10:25:47.582389+0000","last_change":"2024-04-23T10:23:31.092860+0000","last_active":"2024-04-23T10:25:47.582389+0000","last_peered":"2024-04-23T10:25:47.582389+0000","last_clean":"2024-04-23T10:25:47.582389+0000","last_became_active":"2024-04-23T10:23:31.092357+0000","last_became_peered":"2024-04-23T10:23:31.092357+0000","last_unstale":"2024-04-23T10:25:47.582389+0000","last_undegraded":"2024-04-23T10:25:47.582389+0000","last_fullsized":"2024-04-23T10:25:47.582389+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T10:20:12.201233+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T10:20:12.201233+0000","last_clean_scrub_stamp":"2024-04-23T10:20:12.201233+0000","objects_scrubbed":0,"log_size":137,"log_dups_size":0,"ondisk_log_size":137,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T21:28:56.804053+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":136,"num_read_kb":119,"num_write":181,"num_write_kb":2938,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":1180736,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,10],"acting":[7,5,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1","version":"0'0","reported_seq":17,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-23T10:25:47.583932+0000","last_change":"2024-04-23T10:24:44.058393+0000","last_active":"2024-04-23T10:25:47.583932+0000","last_peered":"2024-04-23T10:25:47.583932+0000","last_clean":"2024-04-23T10:25:47.583932+0000","last_became_active":"2024-04-23T10:24:44.058075+0000","last_became_peered":"2024-04-23T10:24:44.058075+0000","last_unstale":"2024-04-23T10:25:47.583932+0000","last_undegraded":"2024-04-23T10:25:47.583932+0000","last_fullsized":"2024-04-23T10:25:47.583932+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T10:24:43.025324+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T10:24:43.025324+0000","last_clean_scrub_stamp":"2024-04-23T10:24:43.025324+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T10:32:05.693311+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,2],"acting":[6,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.0","version":"87'3","reported_seq":22,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-23T10:25:47.582491+0000","last_change":"2024-04-23T10:24:43.048678+0000","last_active":"2024-04-23T10:25:47.582491+0000","last_peered":"2024-04-23T10:25:47.582491+0000","last_clean":"2024-04-23T10:25:47.582491+0000","last_became_active":"2024-04-23T10:24:43.048341+0000","last_became_peered":"2024-04-23T10:24:43.048341+0000","last_unstale":"2024-04-23T10:25:47.582491+0000","last_undegraded":"2024-04-23T10:25:47.582491+0000","last_fullsized":"2024-04-23T10:25:47.582491+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T10:24:42.023127+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T10:24:42.023127+0000","last_clean_scrub_stamp":"2024-04-23T10:24:42.023127+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T12:24:33.705325+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,8],"acting":[7,5,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.5","version":"87'1","reported_seq":21,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-23T10:25:47.588426+0000","last_change":"2024-04-23T10:24:43.046105+0000","last_active":"2024-04-23T10:25:47.588426+0000","last_peered":"2024-04-23T10:25:47.588426+0000","last_clean":"2024-04-23T10:25:47.588426+0000","last_became_active":"2024-04-23T10:24:43.045804+0000","last_became_peered":"2024-04-23T10:24:43.045804+0000","last_unstale":"2024-04-23T10:25:47.588426+0000","last_undegraded":"2024-04-23T10:25:47.588426+0000","last_fullsized":"2024-04-23T10:25:47.588426+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T10:24:42.023127+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T10:24:42.023127+0000","last_clean_scrub_stamp":"2024-04-23T10:24:42.023127+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T22:23:35.927468+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,0,9],"acting":[8,0,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.4","version":"0'0","reported_seq":17,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-23T10:25:47.584146+0000","last_change":"2024-04-23T10:24:44.059731+0000","last_active":"2024-04-23T10:25:47.584146+0000","last_peered":"2024-04-23T10:25:47.584146+0000","last_clean":"2024-04-23T10:25:47.584146+0000","last_became_active":"2024-04-23T10:24:44.059352+0000","last_became_peered":"2024-04-23T10:24:44.059352+0000","last_unstale":"2024-04-23T10:25:47.584146+0000","last_undegraded":"2024-04-23T10:25:47.584146+0000","last_fullsized":"2024-04-23T10:25:47.584146+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T10:24:43.025324+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T10:24:43.025324+0000","last_clean_scrub_stamp":"2024-04-23T10:24:43.025324+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T16:36:08.985444+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2,9],"acting":[6,2,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.6","version":"87'8","reported_seq":27,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-23T10:25:47.580989+0000","last_change":"2024-04-23T10:24:43.044527+0000","last_active":"2024-04-23T10:25:47.580989+0000","last_peered":"2024-04-23T10:25:47.580989+0000","last_clean":"2024-04-23T10:25:47.580989+0000","last_became_active":"2024-04-23T10:24:43.044212+0000","last_became_peered":"2024-04-23T10:24:43.044212+0000","last_unstale":"2024-04-23T10:25:47.580989+0000","last_undegraded":"2024-04-23T10:25:47.580989+0000","last_fullsized":"2024-04-23T10:25:47.580989+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T10:24:42.023127+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T10:24:42.023127+0000","last_clean_scrub_stamp":"2024-04-23T10:24:42.023127+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T19:39:46.369132+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":14,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,8,3],"acting":[9,8,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.7","version":"0'0","reported_seq":17,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-23T10:25:47.586588+0000","last_change":"2024-04-23T10:24:44.055920+0000","last_active":"2024-04-23T10:25:47.586588+0000","last_peered":"2024-04-23T10:25:47.586588+0000","last_clean":"2024-04-23T10:25:47.586588+0000","last_became_active":"2024-04-23T10:24:44.055411+0000","last_became_peered":"2024-04-23T10:24:44.055411+0000","last_unstale":"2024-04-23T10:25:47.586588+0000","last_undegraded":"2024-04-23T10:25:47.586588+0000","last_fullsized":"2024-04-23T10:25:47.586588+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T10:24:43.025324+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T10:24:43.025324+0000","last_clean_scrub_stamp":"2024-04-23T10:24:43.025324+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T14:43:10.446790+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,11,7],"acting":[3,11,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.7","version":"87'4","reported_seq":23,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-23T10:25:47.720935+0000","last_change":"2024-04-23T10:24:43.047989+0000","last_active":"2024-04-23T10:25:47.720935+0000","last_peered":"2024-04-23T10:25:47.720935+0000","last_clean":"2024-04-23T10:25:47.720935+0000","last_became_active":"2024-04-23T10:24:43.047625+0000","last_became_peered":"2024-04-23T10:24:43.047625+0000","last_unstale":"2024-04-23T10:25:47.720935+0000","last_undegraded":"2024-04-23T10:25:47.720935+0000","last_fullsized":"2024-04-23T10:25:47.720935+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T10:24:42.023127+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T10:24:42.023127+0000","last_clean_scrub_stamp":"2024-04-23T10:24:42.023127+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T13:50:16.647176+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,11,6],"acting":[4,11,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.6","version":"0'0","reported_seq":17,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-23T10:25:47.583380+0000","last_change":"2024-04-23T10:24:44.049214+0000","last_active":"2024-04-23T10:25:47.583380+0000","last_peered":"2024-04-23T10:25:47.583380+0000","last_clean":"2024-04-23T10:25:47.583380+0000","last_became_active":"2024-04-23T10:24:44.048869+0000","last_became_peered":"2024-04-23T10:24:44.048869+0000","last_unstale":"2024-04-23T10:25:47.583380+0000","last_undegraded":"2024-04-23T10:25:47.583380+0000","last_fullsized":"2024-04-23T10:25:47.583380+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T10:24:43.025324+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T10:24:43.025324+0000","last_clean_scrub_stamp":"2024-04-23T10:24:43.025324+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T18:17:09.772428+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,4],"acting":[0,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.8","version":"87'2","reported_seq":21,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-23T10:25:47.719683+0000","last_change":"2024-04-23T10:24:43.046189+0000","last_active":"2024-04-23T10:25:47.719683+0000","last_peered":"2024-04-23T10:25:47.719683+0000","last_clean":"2024-04-23T10:25:47.719683+0000","last_became_active":"2024-04-23T10:24:43.045848+0000","last_became_peered":"2024-04-23T10:24:43.045848+0000","last_unstale":"2024-04-23T10:25:47.719683+0000","last_undegraded":"2024-04-23T10:25:47.719683+0000","last_fullsized":"2024-04-23T10:25:47.719683+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T10:24:42.023127+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T10:24:42.023127+0000","last_clean_scrub_stamp":"2024-04-23T10:24:42.023127+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T22:23:39.180377+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,0],"acting":[1,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.9","version":"0'0","reported_seq":17,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-23T10:25:47.588246+0000","last_change":"2024-04-23T10:24:44.048415+0000","last_active":"2024-04-23T10:25:47.588246+0000","last_peered":"2024-04-23T10:25:47.588246+0000","last_clean":"2024-04-23T10:25:47.588246+0000","last_became_active":"2024-04-23T10:24:44.048030+0000","last_became_peered":"2024-04-23T10:24:44.048030+0000","last_unstale":"2024-04-23T10:25:47.588246+0000","last_undegraded":"2024-04-23T10:25:47.588246+0000","last_fullsized":"2024-04-23T10:25:47.588246+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T10:24:43.025324+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T10:24:43.025324+0000","last_clean_scrub_stamp":"2024-04-23T10:24:43.025324+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T14:42:09.920927+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,11,10],"acting":[8,11,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"2.9","version":"87'3","reported_seq":22,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-23T10:25:47.585862+0000","last_change":"2024-04-23T10:24:43.041525+0000","last_active":"2024-04-23T10:25:47.585862+0000","last_peered":"2024-04-23T10:25:47.585862+0000","last_clean":"2024-04-23T10:25:47.585862+0000","last_became_active":"2024-04-23T10:24:43.041200+0000","last_became_peered":"2024-04-23T10:24:43.041200+0000","last_unstale":"2024-04-23T10:25:47.585862+0000","last_undegraded":"2024-04-23T10:25:47.585862+0000","last_fullsized":"2024-04-23T10:25:47.585862+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T10:24:42.023127+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T10:24:42.023127+0000","last_clean_scrub_stamp":"2024-04-23T10:24:42.023127+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T16:45:06.846991+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,9,10],"acting":[3,9,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.8","version":"0'0","reported_seq":17,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-23T10:25:47.584376+0000","last_change":"2024-04-23T10:24:44.055341+0000","last_active":"2024-04-23T10:25:47.584376+0000","last_peered":"2024-04-23T10:25:47.584376+0000","last_clean":"2024-04-23T10:25:47.584376+0000","last_became_active":"2024-04-23T10:24:44.054216+0000","last_became_peered":"2024-04-23T10:24:44.054216+0000","last_unstale":"2024-04-23T10:25:47.584376+0000","last_undegraded":"2024-04-23T10:25:47.584376+0000","last_fullsized":"2024-04-23T10:25:47.584376+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T10:24:43.025324+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T10:24:43.025324+0000","last_clean_scrub_stamp":"2024-04-23T10:24:43.025324+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T21:31:07.753283+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,0,2],"acting":[11,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"2.a","version":"87'1","reported_seq":20,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-23T10:25:47.582426+0000","last_change":"2024-04-23T10:24:43.045400+0000","last_active":"2024-04-23T10:25:47.582426+0000","last_peered":"2024-04-23T10:25:47.582426+0000","last_clean":"2024-04-23T10:25:47.582426+0000","last_became_active":"2024-04-23T10:24:43.044677+0000","last_became_peered":"2024-04-23T10:24:43.044677+0000","last_unstale":"2024-04-23T10:25:47.582426+0000","last_undegraded":"2024-04-23T10:25:47.582426+0000","last_fullsized":"2024-04-23T10:25:47.582426+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T10:24:42.023127+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T10:24:42.023127+0000","last_clean_scrub_stamp":"2024-04-23T10:24:42.023127+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T19:06:19.700132+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,8],"acting":[2,3,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.b","version":"0'0","reported_seq":17,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-23T10:25:47.586118+0000","last_change":"2024-04-23T10:24:44.049968+0000","last_active":"2024-04-23T10:25:47.586118+0000","last_peered":"2024-04-23T10:25:47.586118+0000","last_clean":"2024-04-23T10:25:47.586118+0000","last_became_active":"2024-04-23T10:24:44.049621+0000","last_became_peered":"2024-04-23T10:24:44.049621+0000","last_unstale":"2024-04-23T10:25:47.586118+0000","last_undegraded":"2024-04-23T10:25:47.586118+0000","last_fullsized":"2024-04-23T10:25:47.586118+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T10:24:43.025324+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T10:24:43.025324+0000","last_clean_scrub_stamp":"2024-04-23T10:24:43.025324+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T16:23:43.082831+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,10],"acting":[3,0,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.b","version":"87'4","reported_seq":24,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-23T10:25:47.587989+0000","last_change":"2024-04-23T10:24:43.041828+0000","last_active":"2024-04-23T10:25:47.587989+0000","last_peered":"2024-04-23T10:25:47.587989+0000","last_clean":"2024-04-23T10:25:47.587989+0000","last_became_active":"2024-04-23T10:24:43.041480+0000","last_became_peered":"2024-04-23T10:24:43.041480+0000","last_unstale":"2024-04-23T10:25:47.587989+0000","last_undegraded":"2024-04-23T10:25:47.587989+0000","last_fullsized":"2024-04-23T10:25:47.587989+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T10:24:42.023127+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T10:24:42.023127+0000","last_clean_scrub_stamp":"2024-04-23T10:24:42.023127+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T14:43:03.203971+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":2,"num_read_kb":2,"num_write":4,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,2,7],"acting":[8,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.a","version":"0'0","reported_seq":17,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-23T10:25:47.582759+0000","last_change":"2024-04-23T10:24:44.062103+0000","last_active":"2024-04-23T10:25:47.582759+0000","last_peered":"2024-04-23T10:25:47.582759+0000","last_clean":"2024-04-23T10:25:47.582759+0000","last_became_active":"2024-04-23T10:24:44.061794+0000","last_became_peered":"2024-04-23T10:24:44.061794+0000","last_unstale":"2024-04-23T10:25:47.582759+0000","last_undegraded":"2024-04-23T10:25:47.582759+0000","last_fullsized":"2024-04-23T10:25:47.582759+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T10:24:43.025324+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T10:24:43.025324+0000","last_clean_scrub_stamp":"2024-04-23T10:24:43.025324+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T10:45:30.130047+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,6],"acting":[7,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.c","version":"87'3","reported_seq":23,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-23T10:25:47.582717+0000","last_change":"2024-04-23T10:24:43.045702+0000","last_active":"2024-04-23T10:25:47.582717+0000","last_peered":"2024-04-23T10:25:47.582717+0000","last_clean":"2024-04-23T10:25:47.582717+0000","last_became_active":"2024-04-23T10:24:43.045258+0000","last_became_peered":"2024-04-23T10:24:43.045258+0000","last_unstale":"2024-04-23T10:25:47.582717+0000","last_undegraded":"2024-04-23T10:25:47.582717+0000","last_fullsized":"2024-04-23T10:25:47.582717+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T10:24:42.023127+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T10:24:42.023127+0000","last_clean_scrub_stamp":"2024-04-23T10:24:42.023127+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T12:54:30.965080+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,11,0],"acting":[2,11,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.d","version":"0'0","reported_seq":17,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-23T10:25:47.582779+0000","last_change":"2024-04-23T10:24:44.061118+0000","last_active":"2024-04-23T10:25:47.582779+0000","last_peered":"2024-04-23T10:25:47.582779+0000","last_clean":"2024-04-23T10:25:47.582779+0000","last_became_active":"2024-04-23T10:24:44.060743+0000","last_became_peered":"2024-04-23T10:24:44.060743+0000","last_unstale":"2024-04-23T10:25:47.582779+0000","last_undegraded":"2024-04-23T10:25:47.582779+0000","last_fullsized":"2024-04-23T10:25:47.582779+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T10:24:43.025324+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T10:24:43.025324+0000","last_clean_scrub_stamp":"2024-04-23T10:24:43.025324+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T17:17:46.731121+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,11],"acting":[7,1,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.d","version":"86'1","reported_seq":21,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-23T10:25:47.720362+0000","last_change":"2024-04-23T10:24:43.052519+0000","last_active":"2024-04-23T10:25:47.720362+0000","last_peered":"2024-04-23T10:25:47.720362+0000","last_clean":"2024-04-23T10:25:47.720362+0000","last_became_active":"2024-04-23T10:24:43.052187+0000","last_became_peered":"2024-04-23T10:24:43.052187+0000","last_unstale":"2024-04-23T10:25:47.720362+0000","last_undegraded":"2024-04-23T10:25:47.720362+0000","last_fullsized":"2024-04-23T10:25:47.720362+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T10:24:42.023127+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T10:24:42.023127+0000","last_clean_scrub_stamp":"2024-04-23T10:24:42.023127+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T15:51:52.839409+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,10,3],"acting":[5,10,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.c","version":"87'1","reported_seq":18,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-23T10:25:47.582550+0000","last_change":"2024-04-23T10:24:44.053086+0000","last_active":"2024-04-23T10:25:47.582550+0000","last_peered":"2024-04-23T10:25:47.582550+0000","last_clean":"2024-04-23T10:25:47.582550+0000","last_became_active":"2024-04-23T10:24:44.052687+0000","last_became_peered":"2024-04-23T10:24:44.052687+0000","last_unstale":"2024-04-23T10:25:47.582550+0000","last_undegraded":"2024-04-23T10:25:47.582550+0000","last_fullsized":"2024-04-23T10:25:47.582550+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T10:24:43.025324+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T10:24:43.025324+0000","last_clean_scrub_stamp":"2024-04-23T10:24:43.025324+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T14:35:44.958742+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":117,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,5],"acting":[7,3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.e","version":"86'2","reported_seq":31,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-23T10:25:47.724936+0000","last_change":"2024-04-23T10:25:47.724936+0000","last_active":"2024-04-23T10:25:47.724936+0000","last_peered":"2024-04-23T10:25:47.724936+0000","last_clean":"2024-04-23T10:25:47.724936+0000","last_became_active":"2024-04-23T10:25:47.722182+0000","last_became_peered":"2024-04-23T10:25:47.722182+0000","last_unstale":"2024-04-23T10:25:47.724936+0000","last_undegraded":"2024-04-23T10:25:47.724936+0000","last_fullsized":"2024-04-23T10:25:47.724936+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T10:24:42.023127+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T10:24:42.023127+0000","last_clean_scrub_stamp":"2024-04-23T10:24:42.023127+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T19:18:42.762648+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,4],"acting":[6,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.f","version":"0'0","reported_seq":17,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-23T10:25:47.582923+0000","last_change":"2024-04-23T10:24:44.049543+0000","last_active":"2024-04-23T10:25:47.582923+0000","last_peered":"2024-04-23T10:25:47.582923+0000","last_clean":"2024-04-23T10:25:47.582923+0000","last_became_active":"2024-04-23T10:24:44.049198+0000","last_became_peered":"2024-04-23T10:24:44.049198+0000","last_unstale":"2024-04-23T10:25:47.582923+0000","last_undegraded":"2024-04-23T10:25:47.582923+0000","last_fullsized":"2024-04-23T10:25:47.582923+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T10:24:43.025324+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T10:24:43.025324+0000","last_clean_scrub_stamp":"2024-04-23T10:24:43.025324+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T12:25:15.974267+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,11,8],"acting":[7,11,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.20","version":"0'0","reported_seq":17,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-23T10:25:47.584458+0000","last_change":"2024-04-23T10:24:44.054440+0000","last_active":"2024-04-23T10:25:47.584458+0000","last_peered":"2024-04-23T10:25:47.584458+0000","last_clean":"2024-04-23T10:25:47.584458+0000","last_became_active":"2024-04-23T10:24:44.053951+0000","last_became_peered":"2024-04-23T10:24:44.053951+0000","last_unstale":"2024-04-23T10:25:47.584458+0000","last_undegraded":"2024-04-23T10:25:47.584458+0000","last_fullsized":"2024-04-23T10:25:47.584458+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T10:24:43.025324+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T10:24:43.025324+0000","last_clean_scrub_stamp":"2024-04-23T10:24:43.025324+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T22:20:10.452712+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,2,5],"acting":[11,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.21","version":"0'0","reported_seq":17,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-23T10:25:47.582419+0000","last_change":"2024-04-23T10:24:44.059015+0000","last_active":"2024-04-23T10:25:47.582419+0000","last_peered":"2024-04-23T10:25:47.582419+0000","last_clean":"2024-04-23T10:25:47.582419+0000","last_became_active":"2024-04-23T10:24:44.058691+0000","last_became_peered":"2024-04-23T10:24:44.058691+0000","last_unstale":"2024-04-23T10:25:47.582419+0000","last_undegraded":"2024-04-23T10:25:47.582419+0000","last_fullsized":"2024-04-23T10:25:47.582419+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T10:24:43.025324+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T10:24:43.025324+0000","last_clean_scrub_stamp":"2024-04-23T10:24:43.025324+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T14:57:48.153094+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,4],"acting":[2,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.22","version":"0'0","reported_seq":17,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-23T10:25:47.582200+0000","last_change":"2024-04-23T10:24:44.055534+0000","last_active":"2024-04-23T10:25:47.582200+0000","last_peered":"2024-04-23T10:25:47.582200+0000","last_clean":"2024-04-23T10:25:47.582200+0000","last_became_active":"2024-04-23T10:24:44.055181+0000","last_became_peered":"2024-04-23T10:24:44.055181+0000","last_unstale":"2024-04-23T10:25:47.582200+0000","last_undegraded":"2024-04-23T10:25:47.582200+0000","last_fullsized":"2024-04-23T10:25:47.582200+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T10:24:43.025324+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T10:24:43.025324+0000","last_clean_scrub_stamp":"2024-04-23T10:24:43.025324+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T13:56:33.878232+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.23","version":"0'0","reported_seq":23,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-23T10:25:47.587380+0000","last_change":"2024-04-23T10:25:47.587380+0000","last_active":"2024-04-23T10:25:47.587380+0000","last_peered":"2024-04-23T10:25:47.587380+0000","last_clean":"2024-04-23T10:25:47.587380+0000","last_became_active":"2024-04-23T10:25:47.586893+0000","last_became_peered":"2024-04-23T10:25:47.586893+0000","last_unstale":"2024-04-23T10:25:47.587380+0000","last_undegraded":"2024-04-23T10:25:47.587380+0000","last_fullsized":"2024-04-23T10:25:47.587380+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T10:24:43.025324+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T10:24:43.025324+0000","last_clean_scrub_stamp":"2024-04-23T10:24:43.025324+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T10:51:34.390594+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,7],"acting":[0,6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.24","version":"0'0","reported_seq":17,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-23T10:25:47.584018+0000","last_change":"2024-04-23T10:24:44.050741+0000","last_active":"2024-04-23T10:25:47.584018+0000","last_peered":"2024-04-23T10:25:47.584018+0000","last_clean":"2024-04-23T10:25:47.584018+0000","last_became_active":"2024-04-23T10:24:44.050398+0000","last_became_peered":"2024-04-23T10:24:44.050398+0000","last_unstale":"2024-04-23T10:25:47.584018+0000","last_undegraded":"2024-04-23T10:25:47.584018+0000","last_fullsized":"2024-04-23T10:25:47.584018+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T10:24:43.025324+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T10:24:43.025324+0000","last_clean_scrub_stamp":"2024-04-23T10:24:43.025324+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T10:49:07.146673+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,0],"acting":[6,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.25","version":"0'0","reported_seq":17,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-23T10:25:47.585333+0000","last_change":"2024-04-23T10:24:44.048933+0000","last_active":"2024-04-23T10:25:47.585333+0000","last_peered":"2024-04-23T10:25:47.585333+0000","last_clean":"2024-04-23T10:25:47.585333+0000","last_became_active":"2024-04-23T10:24:44.048553+0000","last_became_peered":"2024-04-23T10:24:44.048553+0000","last_unstale":"2024-04-23T10:25:47.585333+0000","last_undegraded":"2024-04-23T10:25:47.585333+0000","last_fullsized":"2024-04-23T10:25:47.585333+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T10:24:43.025324+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T10:24:43.025324+0000","last_clean_scrub_stamp":"2024-04-23T10:24:43.025324+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T10:40:24.569906+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,8,5],"acting":[11,8,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.26","version":"0'0","reported_seq":17,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-23T10:25:47.582825+0000","last_change":"2024-04-23T10:24:44.051895+0000","last_active":"2024-04-23T10:25:47.582825+0000","last_peered":"2024-04-23T10:25:47.582825+0000","last_clean":"2024-04-23T10:25:47.582825+0000","last_became_active":"2024-04-23T10:24:44.051005+0000","last_became_peered":"2024-04-23T10:24:44.051005+0000","last_unstale":"2024-04-23T10:25:47.582825+0000","last_undegraded":"2024-04-23T10:25:47.582825+0000","last_fullsized":"2024-04-23T10:25:47.582825+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T10:24:43.025324+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T10:24:43.025324+0000","last_clean_scrub_stamp":"2024-04-23T10:24:43.025324+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T18:13:05.227365+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,8,1],"acting":[2,8,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.27","version":"0'0","reported_seq":23,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-23T10:25:47.587504+0000","last_change":"2024-04-23T10:25:47.587504+0000","last_active":"2024-04-23T10:25:47.587504+0000","last_peered":"2024-04-23T10:25:47.587504+0000","last_clean":"2024-04-23T10:25:47.587504+0000","last_became_active":"2024-04-23T10:25:47.587157+0000","last_became_peered":"2024-04-23T10:25:47.587157+0000","last_unstale":"2024-04-23T10:25:47.587504+0000","last_undegraded":"2024-04-23T10:25:47.587504+0000","last_fullsized":"2024-04-23T10:25:47.587504+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T10:24:43.025324+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T10:24:43.025324+0000","last_clean_scrub_stamp":"2024-04-23T10:24:43.025324+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T18:46:25.090568+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,2],"acting":[3,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.28","version":"0'0","reported_seq":17,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-23T10:25:47.720199+0000","last_change":"2024-04-23T10:24:44.044158+0000","last_active":"2024-04-23T10:25:47.720199+0000","last_peered":"2024-04-23T10:25:47.720199+0000","last_clean":"2024-04-23T10:25:47.720199+0000","last_became_active":"2024-04-23T10:24:44.043829+0000","last_became_peered":"2024-04-23T10:24:44.043829+0000","last_unstale":"2024-04-23T10:25:47.720199+0000","last_undegraded":"2024-04-23T10:25:47.720199+0000","last_fullsized":"2024-04-23T10:25:47.720199+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T10:24:43.025324+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T10:24:43.025324+0000","last_clean_scrub_stamp":"2024-04-23T10:24:43.025324+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T15:10:08.081831+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,9,3],"acting":[5,9,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.29","version":"0'0","reported_seq":17,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-23T10:25:47.585761+0000","last_change":"2024-04-23T10:24:44.051267+0000","last_active":"2024-04-23T10:25:47.585761+0000","last_peered":"2024-04-23T10:25:47.585761+0000","last_clean":"2024-04-23T10:25:47.585761+0000","last_became_active":"2024-04-23T10:24:44.050911+0000","last_became_peered":"2024-04-23T10:24:44.050911+0000","last_unstale":"2024-04-23T10:25:47.585761+0000","last_undegraded":"2024-04-23T10:25:47.585761+0000","last_fullsized":"2024-04-23T10:25:47.585761+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T10:24:43.025324+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T10:24:43.025324+0000","last_clean_scrub_stamp":"2024-04-23T10:24:43.025324+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T18:09:28.871427+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,11],"acting":[3,2,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.2a","version":"0'0","reported_seq":17,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-23T10:25:47.720038+0000","last_change":"2024-04-23T10:24:44.057653+0000","last_active":"2024-04-23T10:25:47.720038+0000","last_peered":"2024-04-23T10:25:47.720038+0000","last_clean":"2024-04-23T10:25:47.720038+0000","last_became_active":"2024-04-23T10:24:44.056627+0000","last_became_peered":"2024-04-23T10:24:44.056627+0000","last_unstale":"2024-04-23T10:25:47.720038+0000","last_undegraded":"2024-04-23T10:25:47.720038+0000","last_fullsized":"2024-04-23T10:25:47.720038+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T10:24:43.025324+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T10:24:43.025324+0000","last_clean_scrub_stamp":"2024-04-23T10:24:43.025324+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T18:29:21.480187+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,3],"acting":[1,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.2b","version":"0'0","reported_seq":17,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-23T10:25:47.584646+0000","last_change":"2024-04-23T10:24:44.059735+0000","last_active":"2024-04-23T10:25:47.584646+0000","last_peered":"2024-04-23T10:25:47.584646+0000","last_clean":"2024-04-23T10:25:47.584646+0000","last_became_active":"2024-04-23T10:24:44.059052+0000","last_became_peered":"2024-04-23T10:24:44.059052+0000","last_unstale":"2024-04-23T10:25:47.584646+0000","last_undegraded":"2024-04-23T10:25:47.584646+0000","last_fullsized":"2024-04-23T10:25:47.584646+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T10:24:43.025324+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T10:24:43.025324+0000","last_clean_scrub_stamp":"2024-04-23T10:24:43.025324+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T14:29:25.794655+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,0],"acting":[6,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.2c","version":"0'0","reported_seq":17,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-23T10:25:47.584627+0000","last_change":"2024-04-23T10:24:44.051831+0000","last_active":"2024-04-23T10:25:47.584627+0000","last_peered":"2024-04-23T10:25:47.584627+0000","last_clean":"2024-04-23T10:25:47.584627+0000","last_became_active":"2024-04-23T10:24:44.051480+0000","last_became_peered":"2024-04-23T10:24:44.051480+0000","last_unstale":"2024-04-23T10:25:47.584627+0000","last_undegraded":"2024-04-23T10:25:47.584627+0000","last_fullsized":"2024-04-23T10:25:47.584627+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T10:24:43.025324+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T10:24:43.025324+0000","last_clean_scrub_stamp":"2024-04-23T10:24:43.025324+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T19:20:55.181393+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,4,8],"acting":[11,4,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]}],"pool_stats":[{"poolid":3,"num_pg":64,"stat_sum":{"num_bytes":234,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":702,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":2,"ondisk_log_size":2,"up":192,"acting":192,"num_store_stats":12},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":136,"num_read_kb":119,"num_write":181,"num_write_kb":2938,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":1180736,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":4751360,"data_stored":4722944,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":137,"ondisk_log_size":137,"up":3,"acting":3,"num_store_stats":8},{"poolid":2,"num_pg":32,"stat_sum":{"num_bytes":65438,"num_objects":60,"num_object_clones":0,"num_object_copies":180,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":60,"num_whiteouts":0,"num_read":436,"num_read_kb":286,"num_write":157,"num_write_kb":109,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":10,"num_objects_omap":40,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":417792,"data_stored":196314,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":136,"ondisk_log_size":136,"up":96,"acting":96,"num_store_stats":12}],"osd_stats":[{"osd":11,"up_from":82,"seq":352187318301,"num_pgs":19,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32132,"kb_used_data":952,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93713020,"statfs":{"total":95995035648,"available":95962132480,"internally_reserved":0,"allocated":974848,"data_stored":465606,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1596,"internal_metadata":31914436},"hb_peers":[0,1,2,4,5,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":10,"up_from":74,"seq":317827579937,"num_pgs":26,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28656,"kb_used_data":1632,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93716496,"statfs":{"total":95995035648,"available":95965691904,"internally_reserved":0,"allocated":1671168,"data_stored":1118524,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1600,"internal_metadata":27654592},"hb_peers":[0,1,2,3,4,5,6,8,9,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":9,"up_from":69,"seq":296352743463,"num_pgs":20,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":27988,"kb_used_data":968,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717164,"statfs":{"total":95995035648,"available":95966375936,"internally_reserved":0,"allocated":991232,"data_stored":465842,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1600,"internal_metadata":27654592},"hb_peers":[0,1,2,3,4,5,6,7,8,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":8,"up_from":62,"seq":266287972396,"num_pgs":25,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32152,"kb_used_data":968,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93713000,"statfs":{"total":95995035648,"available":95962112000,"internally_reserved":0,"allocated":991232,"data_stored":465762,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,2,3,4,5,6,7,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":7,"up_from":53,"seq":227633266737,"num_pgs":26,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":36888,"kb_used_data":1548,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":93708264,"statfs":{"total":95995035648,"available":95957262336,"internally_reserved":0,"allocated":1585152,"data_stored":1058083,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":36174282},"hb_peers":[1,2,3,4,5,6,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":6,"up_from":47,"seq":201863462966,"num_pgs":22,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32140,"kb_used_data":956,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93713012,"statfs":{"total":95995035648,"available":95962124288,"internally_reserved":0,"allocated":978944,"data_stored":465231,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":31914438},"hb_peers":[0,1,2,3,4,5,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":13,"seq":55834574926,"num_pgs":24,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28036,"kb_used_data":1016,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717116,"statfs":{"total":95995035648,"available":95966326784,"internally_reserved":0,"allocated":1040384,"data_stored":526070,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,2,3,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":8,"seq":34359738452,"num_pgs":25,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32160,"kb_used_data":976,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93712992,"statfs":{"total":95995035648,"available":95962103808,"internally_reserved":0,"allocated":999424,"data_stored":466502,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":31914438},"hb_peers":[1,2,3,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":18,"seq":77309411402,"num_pgs":27,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":36384,"kb_used_data":1040,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":93708768,"statfs":{"total":95995035648,"available":95957778432,"internally_reserved":0,"allocated":1064960,"data_stored":527811,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":36174277},"hb_peers":[0,1,3,4,5,6,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":25,"seq":107374182466,"num_pgs":26,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":27996,"kb_used_data":972,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717156,"statfs":{"total":95995035648,"available":95966367744,"internally_reserved":0,"allocated":995328,"data_stored":465542,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[0,1,2,4,5,6,7,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":4,"up_from":33,"seq":141733920831,"num_pgs":23,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":27980,"kb_used_data":960,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717172,"statfs":{"total":95995035648,"available":95966384128,"internally_reserved":0,"allocated":983040,"data_stored":466059,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[1,2,3,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":39,"seq":167503724603,"num_pgs":28,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":36904,"kb_used_data":1560,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":93708248,"statfs":{"total":95995035648,"available":95957245952,"internally_reserved":0,"allocated":1597440,"data_stored":1057376,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":36174277},"hb_peers":[0,1,2,3,4,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":28672,"data_stored":1478,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":69632,"data_stored":61046,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":94208,"data_stored":62787,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":284,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":1035,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":28672,"data_stored":1867,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":90,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":2574,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":738,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":818,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":102400,"data_stored":63015,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":582,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":234,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":117,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":117,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":117,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":117,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-04-23T10:25:58.707 DEBUG:teuthology.orchestra.run.smithi114:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d69f13d827952bb07bb9b7973bca14f97d764d6f shell --fsid aa240572-0159-11ef-bc93-c7b262605968 -- ceph pg dump --format=json 2024-04-23T10:25:58.952 INFO:teuthology.orchestra.run.smithi114.stderr:Inferring config /var/lib/ceph/aa240572-0159-11ef-bc93-c7b262605968/mon.a/config 2024-04-23T10:25:59.319 INFO:journalctl@ceph.mon.a.smithi114.stdout:Apr 23 10:25:59 smithi114 ceph-mon[29255]: pgmap v325: 97 pgs: 97 active+clean; 641 KiB data, 371 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s; 0 B/s, 0 keys/s, 0 objects/s recovering 2024-04-23T10:25:59.319 INFO:journalctl@ceph.mon.a.smithi114.stdout:Apr 23 10:25:59 smithi114 ceph-mon[29255]: from='mgr.14152 172.21.15.114:0/1529706138' entity='mgr.x' 2024-04-23T10:25:59.319 INFO:journalctl@ceph.mon.a.smithi114.stdout:Apr 23 10:25:59 smithi114 ceph-mon[29255]: from='mgr.14152 172.21.15.114:0/1529706138' entity='mgr.x' 2024-04-23T10:25:59.320 INFO:journalctl@ceph.mon.a.smithi114.stdout:Apr 23 10:25:59 smithi114 ceph-mon[29255]: from='mgr.14152 172.21.15.114:0/1529706138' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-23T10:25:59.320 INFO:journalctl@ceph.mon.a.smithi114.stdout:Apr 23 10:25:59 smithi114 ceph-mon[29255]: from='mgr.14152 172.21.15.114:0/1529706138' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-23T10:25:59.320 INFO:journalctl@ceph.mon.a.smithi114.stdout:Apr 23 10:25:59 smithi114 ceph-mon[29255]: from='mgr.14152 172.21.15.114:0/1529706138' entity='mgr.x' 2024-04-23T10:25:59.327 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 23 10:25:59 smithi204 ceph-mon[32649]: pgmap v325: 97 pgs: 97 active+clean; 641 KiB data, 371 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s; 0 B/s, 0 keys/s, 0 objects/s recovering 2024-04-23T10:25:59.328 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 23 10:25:59 smithi204 ceph-mon[32649]: from='mgr.14152 172.21.15.114:0/1529706138' entity='mgr.x' 2024-04-23T10:25:59.328 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 23 10:25:59 smithi204 ceph-mon[32649]: from='mgr.14152 172.21.15.114:0/1529706138' entity='mgr.x' 2024-04-23T10:25:59.328 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 23 10:25:59 smithi204 ceph-mon[32649]: from='mgr.14152 172.21.15.114:0/1529706138' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-23T10:25:59.328 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 23 10:25:59 smithi204 ceph-mon[32649]: from='mgr.14152 172.21.15.114:0/1529706138' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-23T10:25:59.328 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 23 10:25:59 smithi204 ceph-mon[32649]: from='mgr.14152 172.21.15.114:0/1529706138' entity='mgr.x' 2024-04-23T10:25:59.492 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 23 10:25:59 smithi119 ceph-mon[32608]: pgmap v325: 97 pgs: 97 active+clean; 641 KiB data, 371 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s; 0 B/s, 0 keys/s, 0 objects/s recovering 2024-04-23T10:25:59.492 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 23 10:25:59 smithi119 ceph-mon[32608]: from='mgr.14152 172.21.15.114:0/1529706138' entity='mgr.x' 2024-04-23T10:25:59.492 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 23 10:25:59 smithi119 ceph-mon[32608]: from='mgr.14152 172.21.15.114:0/1529706138' entity='mgr.x' 2024-04-23T10:25:59.492 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 23 10:25:59 smithi119 ceph-mon[32608]: from='mgr.14152 172.21.15.114:0/1529706138' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-23T10:25:59.492 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 23 10:25:59 smithi119 ceph-mon[32608]: from='mgr.14152 172.21.15.114:0/1529706138' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-23T10:25:59.492 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 23 10:25:59 smithi119 ceph-mon[32608]: from='mgr.14152 172.21.15.114:0/1529706138' entity='mgr.x' 2024-04-23T10:26:00.056 INFO:journalctl@ceph.mon.a.smithi114.stdout:Apr 23 10:26:00 smithi114 ceph-mon[29255]: from='client.14970 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-04-23T10:26:00.328 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 23 10:26:00 smithi204 ceph-mon[32649]: from='client.14970 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-04-23T10:26:00.491 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 23 10:26:00 smithi119 ceph-mon[32608]: from='client.14970 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-04-23T10:26:00.629 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:00.628+0000 7f524f156700 1 -- 172.21.15.114:0/62791092 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f5248102b00 msgr2=0x7f5248102f80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T10:26:00.629 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:00.628+0000 7f524f156700 1 --2- 172.21.15.114:0/62791092 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f5248102b00 0x7f5248102f80 secure :-1 s=READY pgs=300 cs=0 l=1 rev1=1 crypto rx=0x7f5234009990 tx=0x7f5234009ca0 comp rx=0 tx=0).stop 2024-04-23T10:26:00.629 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:00.628+0000 7f524f156700 1 -- 172.21.15.114:0/62791092 shutdown_connections 2024-04-23T10:26:00.629 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:00.628+0000 7f524f156700 1 --2- 172.21.15.114:0/62791092 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f5248108a10 0x7f5248108df0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:26:00.629 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:00.628+0000 7f524f156700 1 --2- 172.21.15.114:0/62791092 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f52481034c0 0x7f524810f8e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:26:00.629 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:00.628+0000 7f524f156700 1 --2- 172.21.15.114:0/62791092 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f5248102b00 0x7f5248102f80 unknown :-1 s=CLOSED pgs=300 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:26:00.629 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:00.628+0000 7f524f156700 1 -- 172.21.15.114:0/62791092 >> 172.21.15.114:0/62791092 conn(0x7f5248079c00 msgr2=0x7f524807a010 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T10:26:00.630 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:00.628+0000 7f524f156700 1 -- 172.21.15.114:0/62791092 shutdown_connections 2024-04-23T10:26:00.630 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:00.628+0000 7f524f156700 1 -- 172.21.15.114:0/62791092 wait complete. 2024-04-23T10:26:00.630 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:00.629+0000 7f524f156700 1 Processor -- start 2024-04-23T10:26:00.630 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:00.629+0000 7f524f156700 1 -- start start 2024-04-23T10:26:00.630 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:00.629+0000 7f524f156700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f5248102b00 0x7f52481990e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T10:26:00.630 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:00.629+0000 7f524f156700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f52481034c0 0x7f5248199620 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T10:26:00.630 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:00.629+0000 7f524f156700 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f5248108a10 0x7f524819d9b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T10:26:00.630 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:00.629+0000 7f524f156700 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x7f5248112c50 con 0x7f5248102b00 2024-04-23T10:26:00.630 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:00.629+0000 7f524f156700 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 -- 0x7f5248112ad0 con 0x7f52481034c0 2024-04-23T10:26:00.630 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:00.629+0000 7f524f156700 1 -- --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_getmap magic: 0 -- 0x7f5248112dd0 con 0x7f5248108a10 2024-04-23T10:26:00.630 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:00.629+0000 7f524cef2700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f5248102b00 0x7f52481990e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T10:26:00.630 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:00.629+0000 7f524cef2700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f5248102b00 0x7f52481990e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.114:34134/0 (socket says 172.21.15.114:34134) 2024-04-23T10:26:00.630 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:00.629+0000 7f524cef2700 1 -- 172.21.15.114:0/3223781144 learned_addr learned my addr 172.21.15.114:0/3223781144 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-04-23T10:26:00.631 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:00.629+0000 7f5247fff700 1 --2- 172.21.15.114:0/3223781144 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f52481034c0 0x7f5248199620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T10:26:00.631 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:00.629+0000 7f524cef2700 1 -- 172.21.15.114:0/3223781144 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f5248108a10 msgr2=0x7f524819d9b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T10:26:00.631 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:00.629+0000 7f524cef2700 1 --2- 172.21.15.114:0/3223781144 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f5248108a10 0x7f524819d9b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:26:00.631 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:00.629+0000 7f524cef2700 1 -- 172.21.15.114:0/3223781144 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f52481034c0 msgr2=0x7f5248199620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T10:26:00.631 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:00.629+0000 7f524cef2700 1 --2- 172.21.15.114:0/3223781144 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f52481034c0 0x7f5248199620 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:26:00.631 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:00.629+0000 7f524cef2700 1 -- 172.21.15.114:0/3223781144 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f524819e0f0 con 0x7f5248102b00 2024-04-23T10:26:00.631 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:00.630+0000 7f524cef2700 1 --2- 172.21.15.114:0/3223781144 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f5248102b00 0x7f52481990e0 secure :-1 s=READY pgs=301 cs=0 l=1 rev1=1 crypto rx=0x7f523c00c920 tx=0x7f523c00cc30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T10:26:00.631 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:00.630+0000 7f5245ffb700 1 -- 172.21.15.114:0/3223781144 <== mon.0 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f523c007b00 con 0x7f5248102b00 2024-04-23T10:26:00.631 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:00.630+0000 7f524f156700 1 -- 172.21.15.114:0/3223781144 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f524819e3e0 con 0x7f5248102b00 2024-04-23T10:26:00.631 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:00.630+0000 7f5245ffb700 1 -- 172.21.15.114:0/3223781144 <== mon.0 v2:172.21.15.114:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f523c004ce0 con 0x7f5248102b00 2024-04-23T10:26:00.631 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:00.630+0000 7f5245ffb700 1 -- 172.21.15.114:0/3223781144 <== mon.0 v2:172.21.15.114:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f523c005500 con 0x7f5248102b00 2024-04-23T10:26:00.633 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:00.631+0000 7f524f156700 1 -- 172.21.15.114:0/3223781144 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f52481a5c60 con 0x7f5248102b00 2024-04-23T10:26:00.633 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:00.631+0000 7f524f156700 1 -- 172.21.15.114:0/3223781144 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f5248069730 con 0x7f5248102b00 2024-04-23T10:26:00.633 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:00.632+0000 7f5245ffb700 1 -- 172.21.15.114:0/3223781144 <== mon.0 v2:172.21.15.114:3300/0 4 ==== mgrmap(e 17) ==== 138698+0+0 (secure 0 0 0) 0x7f523c007ca0 con 0x7f5248102b00 2024-04-23T10:26:00.633 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:00.633+0000 7f5245ffb700 1 --2- 172.21.15.114:0/3223781144 >> [v2:172.21.15.114:6800/465333808,v1:172.21.15.114:6801/465333808] conn(0x7f52380a3cd0 0x7f52380a6190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T10:26:00.633 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:00.633+0000 7f5247fff700 1 --2- 172.21.15.114:0/3223781144 >> [v2:172.21.15.114:6800/465333808,v1:172.21.15.114:6801/465333808] conn(0x7f52380a3cd0 0x7f52380a6190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T10:26:00.633 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:00.633+0000 7f5245ffb700 1 -- 172.21.15.114:0/3223781144 <== mon.0 v2:172.21.15.114:3300/0 5 ==== osd_map(89..89 src has 1..89) ==== 8269+0+0 (secure 0 0 0) 0x7f523c0cfd80 con 0x7f5248102b00 2024-04-23T10:26:00.633 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:00.633+0000 7f5247fff700 1 --2- 172.21.15.114:0/3223781144 >> [v2:172.21.15.114:6800/465333808,v1:172.21.15.114:6801/465333808] conn(0x7f52380a3cd0 0x7f52380a6190 secure :-1 s=READY pgs=209 cs=0 l=1 rev1=1 crypto rx=0x7f52340098e0 tx=0x7f5234005960 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T10:26:00.636 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:00.637+0000 7f5245ffb700 1 -- 172.21.15.114:0/3223781144 <== mon.0 v2:172.21.15.114:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f523c014030 con 0x7f5248102b00 2024-04-23T10:26:00.831 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:00.832+0000 7f524f156700 1 -- 172.21.15.114:0/3223781144 --> [v2:172.21.15.114:6800/465333808,v1:172.21.15.114:6801/465333808] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) -- 0x7f5248062e80 con 0x7f52380a3cd0 2024-04-23T10:26:00.838 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:00.839+0000 7f5245ffb700 1 -- 172.21.15.114:0/3223781144 <== mgr.14152 v2:172.21.15.114:6800/465333808 1 ==== mgr_command_reply(tid 0: 0 dumped all) ==== 18+0+258714 (secure 0 0 0) 0x7f5248062e80 con 0x7f52380a3cd0 2024-04-23T10:26:00.842 INFO:teuthology.orchestra.run.smithi114.stdout: 2024-04-23T10:26:00.844 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:00.841+0000 7f524f156700 1 -- 172.21.15.114:0/3223781144 >> [v2:172.21.15.114:6800/465333808,v1:172.21.15.114:6801/465333808] conn(0x7f52380a3cd0 msgr2=0x7f52380a6190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T10:26:00.844 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:00.841+0000 7f524f156700 1 --2- 172.21.15.114:0/3223781144 >> [v2:172.21.15.114:6800/465333808,v1:172.21.15.114:6801/465333808] conn(0x7f52380a3cd0 0x7f52380a6190 secure :-1 s=READY pgs=209 cs=0 l=1 rev1=1 crypto rx=0x7f52340098e0 tx=0x7f5234005960 comp rx=0 tx=0).stop 2024-04-23T10:26:00.844 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:00.841+0000 7f524f156700 1 -- 172.21.15.114:0/3223781144 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f5248102b00 msgr2=0x7f52481990e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T10:26:00.844 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:00.841+0000 7f524f156700 1 --2- 172.21.15.114:0/3223781144 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f5248102b00 0x7f52481990e0 secure :-1 s=READY pgs=301 cs=0 l=1 rev1=1 crypto rx=0x7f523c00c920 tx=0x7f523c00cc30 comp rx=0 tx=0).stop 2024-04-23T10:26:00.844 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:00.842+0000 7f524f156700 1 -- 172.21.15.114:0/3223781144 shutdown_connections 2024-04-23T10:26:00.844 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:00.842+0000 7f524f156700 1 --2- 172.21.15.114:0/3223781144 >> [v2:172.21.15.114:6800/465333808,v1:172.21.15.114:6801/465333808] conn(0x7f52380a3cd0 0x7f52380a6190 unknown :-1 s=CLOSED pgs=209 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:26:00.844 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:00.842+0000 7f524f156700 1 --2- 172.21.15.114:0/3223781144 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f5248108a10 0x7f524819d9b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:26:00.844 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:00.842+0000 7f524f156700 1 --2- 172.21.15.114:0/3223781144 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f5248102b00 0x7f52481990e0 unknown :-1 s=CLOSED pgs=301 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:26:00.844 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:00.842+0000 7f524f156700 1 --2- 172.21.15.114:0/3223781144 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f52481034c0 0x7f5248199620 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:26:00.844 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:00.842+0000 7f524f156700 1 -- 172.21.15.114:0/3223781144 >> 172.21.15.114:0/3223781144 conn(0x7f5248079c00 msgr2=0x7f5248100cb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T10:26:00.844 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:00.842+0000 7f524f156700 1 -- 172.21.15.114:0/3223781144 shutdown_connections 2024-04-23T10:26:00.845 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:00.842+0000 7f524f156700 1 -- 172.21.15.114:0/3223781144 wait complete. 2024-04-23T10:26:00.845 INFO:teuthology.orchestra.run.smithi114.stderr:dumped all 2024-04-23T10:26:01.328 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 23 10:26:01 smithi204 ceph-mon[32649]: pgmap v326: 97 pgs: 97 active+clean; 641 KiB data, 371 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s; 0 B/s, 0 keys/s, 0 objects/s recovering 2024-04-23T10:26:01.449 INFO:journalctl@ceph.mon.a.smithi114.stdout:Apr 23 10:26:01 smithi114 ceph-mon[29255]: pgmap v326: 97 pgs: 97 active+clean; 641 KiB data, 371 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s; 0 B/s, 0 keys/s, 0 objects/s recovering 2024-04-23T10:26:01.477 INFO:teuthology.orchestra.run.smithi114.stdout:{"pg_ready":true,"pg_map":{"version":326,"stamp":"2024-04-23T10:25:59.974834+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":656040,"num_objects":64,"num_object_clones":0,"num_object_copies":192,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":64,"num_whiteouts":0,"num_read":582,"num_read_kb":410,"num_write":340,"num_write_kb":3049,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":7,"num_bytes_recovered":1180736,"num_keys_recovered":10,"num_objects_omap":40,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":275,"ondisk_log_size":275,"up":291,"acting":291,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":291,"num_osds":12,"num_per_pool_osds":12,"num_per_pool_omap_osds":12,"kb":1124941824,"kb_used":379416,"kb_used_data":13548,"kb_used_omap":18,"kb_used_meta":365677,"kb_avail":1124562408,"statfs":{"total":1151940427776,"available":1151551905792,"internally_reserved":0,"allocated":13873152,"data_stored":7548408,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":19142,"internal_metadata":374453562},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":70,"num_read_kb":35,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":10,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.003188"},"pg_stats":[{"pgid":"3.3f","version":"0'0","reported_seq":17,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-23T10:25:47.584614+0000","last_change":"2024-04-23T10:24:44.051465+0000","last_active":"2024-04-23T10:25:47.584614+0000","last_peered":"2024-04-23T10:25:47.584614+0000","last_clean":"2024-04-23T10:25:47.584614+0000","last_became_active":"2024-04-23T10:24:44.050863+0000","last_became_peered":"2024-04-23T10:24:44.050863+0000","last_unstale":"2024-04-23T10:25:47.584614+0000","last_undegraded":"2024-04-23T10:25:47.584614+0000","last_fullsized":"2024-04-23T10:25:47.584614+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T10:24:43.025324+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T10:24:43.025324+0000","last_clean_scrub_stamp":"2024-04-23T10:24:43.025324+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T15:11:12.508458+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,1,0],"acting":[11,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.3e","version":"0'0","reported_seq":17,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-23T10:25:47.583513+0000","last_change":"2024-04-23T10:24:44.050739+0000","last_active":"2024-04-23T10:25:47.583513+0000","last_peered":"2024-04-23T10:25:47.583513+0000","last_clean":"2024-04-23T10:25:47.583513+0000","last_became_active":"2024-04-23T10:24:44.050430+0000","last_became_peered":"2024-04-23T10:24:44.050430+0000","last_unstale":"2024-04-23T10:25:47.583513+0000","last_undegraded":"2024-04-23T10:25:47.583513+0000","last_fullsized":"2024-04-23T10:25:47.583513+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T10:24:43.025324+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T10:24:43.025324+0000","last_clean_scrub_stamp":"2024-04-23T10:24:43.025324+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T17:36:25.983147+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,11,10],"acting":[0,11,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3d","version":"0'0","reported_seq":17,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-23T10:25:47.583328+0000","last_change":"2024-04-23T10:24:44.054805+0000","last_active":"2024-04-23T10:25:47.583328+0000","last_peered":"2024-04-23T10:25:47.583328+0000","last_clean":"2024-04-23T10:25:47.583328+0000","last_became_active":"2024-04-23T10:24:44.054423+0000","last_became_peered":"2024-04-23T10:24:44.054423+0000","last_unstale":"2024-04-23T10:25:47.583328+0000","last_undegraded":"2024-04-23T10:25:47.583328+0000","last_fullsized":"2024-04-23T10:25:47.583328+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T10:24:43.025324+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T10:24:43.025324+0000","last_clean_scrub_stamp":"2024-04-23T10:24:43.025324+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T21:34:26.793624+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,1],"acting":[0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3c","version":"0'0","reported_seq":17,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-23T10:25:47.720418+0000","last_change":"2024-04-23T10:24:44.055966+0000","last_active":"2024-04-23T10:25:47.720418+0000","last_peered":"2024-04-23T10:25:47.720418+0000","last_clean":"2024-04-23T10:25:47.720418+0000","last_became_active":"2024-04-23T10:24:44.055028+0000","last_became_peered":"2024-04-23T10:24:44.055028+0000","last_unstale":"2024-04-23T10:25:47.720418+0000","last_undegraded":"2024-04-23T10:25:47.720418+0000","last_fullsized":"2024-04-23T10:25:47.720418+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T10:24:43.025324+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T10:24:43.025324+0000","last_clean_scrub_stamp":"2024-04-23T10:24:43.025324+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T19:41:04.594025+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,8,2],"acting":[4,8,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.3b","version":"0'0","reported_seq":17,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-23T10:25:47.587937+0000","last_change":"2024-04-23T10:24:44.046954+0000","last_active":"2024-04-23T10:25:47.587937+0000","last_peered":"2024-04-23T10:25:47.587937+0000","last_clean":"2024-04-23T10:25:47.587937+0000","last_became_active":"2024-04-23T10:24:44.046631+0000","last_became_peered":"2024-04-23T10:24:44.046631+0000","last_unstale":"2024-04-23T10:25:47.587937+0000","last_undegraded":"2024-04-23T10:25:47.587937+0000","last_fullsized":"2024-04-23T10:25:47.587937+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T10:24:43.025324+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T10:24:43.025324+0000","last_clean_scrub_stamp":"2024-04-23T10:24:43.025324+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T15:04:26.878894+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,3,6],"acting":[8,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.3a","version":"0'0","reported_seq":17,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-23T10:25:47.720134+0000","last_change":"2024-04-23T10:24:44.051550+0000","last_active":"2024-04-23T10:25:47.720134+0000","last_peered":"2024-04-23T10:25:47.720134+0000","last_clean":"2024-04-23T10:25:47.720134+0000","last_became_active":"2024-04-23T10:24:44.050469+0000","last_became_peered":"2024-04-23T10:24:44.050469+0000","last_unstale":"2024-04-23T10:25:47.720134+0000","last_undegraded":"2024-04-23T10:25:47.720134+0000","last_fullsized":"2024-04-23T10:25:47.720134+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T10:24:43.025324+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T10:24:43.025324+0000","last_clean_scrub_stamp":"2024-04-23T10:24:43.025324+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T21:13:42.469252+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,9,4],"acting":[10,9,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.39","version":"0'0","reported_seq":23,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-23T10:25:47.588865+0000","last_change":"2024-04-23T10:25:47.588865+0000","last_active":"2024-04-23T10:25:47.588865+0000","last_peered":"2024-04-23T10:25:47.588865+0000","last_clean":"2024-04-23T10:25:47.588865+0000","last_became_active":"2024-04-23T10:25:47.588571+0000","last_became_peered":"2024-04-23T10:25:47.588571+0000","last_unstale":"2024-04-23T10:25:47.588865+0000","last_undegraded":"2024-04-23T10:25:47.588865+0000","last_fullsized":"2024-04-23T10:25:47.588865+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T10:24:43.025324+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T10:24:43.025324+0000","last_clean_scrub_stamp":"2024-04-23T10:24:43.025324+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T17:14:00.026999+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,9,8],"acting":[7,9,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.38","version":"0'0","reported_seq":11,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-23T10:25:47.722416+0000","last_change":"2024-04-23T10:25:47.722416+0000","last_active":"2024-04-23T10:25:47.722416+0000","last_peered":"2024-04-23T10:25:47.722416+0000","last_clean":"2024-04-23T10:25:47.722416+0000","last_became_active":"2024-04-23T10:25:47.722015+0000","last_became_peered":"2024-04-23T10:25:47.722015+0000","last_unstale":"2024-04-23T10:25:47.722416+0000","last_undegraded":"2024-04-23T10:25:47.722416+0000","last_fullsized":"2024-04-23T10:25:47.722416+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T10:24:43.025324+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T10:24:43.025324+0000","last_clean_scrub_stamp":"2024-04-23T10:24:43.025324+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T17:02:57.881164+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,5,10],"acting":[9,5,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.37","version":"0'0","reported_seq":17,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-23T10:25:47.720708+0000","last_change":"2024-04-23T10:24:44.055261+0000","last_active":"2024-04-23T10:25:47.720708+0000","last_peered":"2024-04-23T10:25:47.720708+0000","last_clean":"2024-04-23T10:25:47.720708+0000","last_became_active":"2024-04-23T10:24:44.054405+0000","last_became_peered":"2024-04-23T10:24:44.054405+0000","last_unstale":"2024-04-23T10:25:47.720708+0000","last_undegraded":"2024-04-23T10:25:47.720708+0000","last_fullsized":"2024-04-23T10:25:47.720708+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T10:24:43.025324+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T10:24:43.025324+0000","last_clean_scrub_stamp":"2024-04-23T10:24:43.025324+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T19:30:30.306739+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,11],"acting":[1,5,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.36","version":"0'0","reported_seq":17,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-23T10:25:47.720053+0000","last_change":"2024-04-23T10:24:44.055854+0000","last_active":"2024-04-23T10:25:47.720053+0000","last_peered":"2024-04-23T10:25:47.720053+0000","last_clean":"2024-04-23T10:25:47.720053+0000","last_became_active":"2024-04-23T10:24:44.055275+0000","last_became_peered":"2024-04-23T10:24:44.055275+0000","last_unstale":"2024-04-23T10:25:47.720053+0000","last_undegraded":"2024-04-23T10:25:47.720053+0000","last_fullsized":"2024-04-23T10:25:47.720053+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T10:24:43.025324+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T10:24:43.025324+0000","last_clean_scrub_stamp":"2024-04-23T10:24:43.025324+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T21:48:25.934878+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,2,4],"acting":[10,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.35","version":"0'0","reported_seq":17,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-23T10:25:47.582659+0000","last_change":"2024-04-23T10:24:44.061314+0000","last_active":"2024-04-23T10:25:47.582659+0000","last_peered":"2024-04-23T10:25:47.582659+0000","last_clean":"2024-04-23T10:25:47.582659+0000","last_became_active":"2024-04-23T10:24:44.060981+0000","last_became_peered":"2024-04-23T10:24:44.060981+0000","last_unstale":"2024-04-23T10:25:47.582659+0000","last_undegraded":"2024-04-23T10:25:47.582659+0000","last_fullsized":"2024-04-23T10:25:47.582659+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T10:24:43.025324+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T10:24:43.025324+0000","last_clean_scrub_stamp":"2024-04-23T10:24:43.025324+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T11:48:08.387575+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,8,1],"acting":[7,8,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.34","version":"0'0","reported_seq":17,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-23T10:25:47.719901+0000","last_change":"2024-04-23T10:24:44.055335+0000","last_active":"2024-04-23T10:25:47.719901+0000","last_peered":"2024-04-23T10:25:47.719901+0000","last_clean":"2024-04-23T10:25:47.719901+0000","last_became_active":"2024-04-23T10:24:44.051257+0000","last_became_peered":"2024-04-23T10:24:44.051257+0000","last_unstale":"2024-04-23T10:25:47.719901+0000","last_undegraded":"2024-04-23T10:25:47.719901+0000","last_fullsized":"2024-04-23T10:25:47.719901+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T10:24:43.025324+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T10:24:43.025324+0000","last_clean_scrub_stamp":"2024-04-23T10:24:43.025324+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T11:57:45.999751+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,10,8],"acting":[4,10,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.33","version":"0'0","reported_seq":17,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-23T10:25:47.584677+0000","last_change":"2024-04-23T10:24:44.052078+0000","last_active":"2024-04-23T10:25:47.584677+0000","last_peered":"2024-04-23T10:25:47.584677+0000","last_clean":"2024-04-23T10:25:47.584677+0000","last_became_active":"2024-04-23T10:24:44.051747+0000","last_became_peered":"2024-04-23T10:24:44.051747+0000","last_unstale":"2024-04-23T10:25:47.584677+0000","last_undegraded":"2024-04-23T10:25:47.584677+0000","last_fullsized":"2024-04-23T10:25:47.584677+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T10:24:43.025324+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T10:24:43.025324+0000","last_clean_scrub_stamp":"2024-04-23T10:24:43.025324+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T19:14:16.495967+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,10,5],"acting":[6,10,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.32","version":"0'0","reported_seq":17,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-23T10:25:47.583123+0000","last_change":"2024-04-23T10:24:44.051994+0000","last_active":"2024-04-23T10:25:47.583123+0000","last_peered":"2024-04-23T10:25:47.583123+0000","last_clean":"2024-04-23T10:25:47.583123+0000","last_became_active":"2024-04-23T10:24:44.051638+0000","last_became_peered":"2024-04-23T10:24:44.051638+0000","last_unstale":"2024-04-23T10:25:47.583123+0000","last_undegraded":"2024-04-23T10:25:47.583123+0000","last_fullsized":"2024-04-23T10:25:47.583123+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T10:24:43.025324+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T10:24:43.025324+0000","last_clean_scrub_stamp":"2024-04-23T10:24:43.025324+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T14:28:30.447198+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,8,10],"acting":[0,8,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.31","version":"0'0","reported_seq":23,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-23T10:25:47.589538+0000","last_change":"2024-04-23T10:25:47.589538+0000","last_active":"2024-04-23T10:25:47.589538+0000","last_peered":"2024-04-23T10:25:47.589538+0000","last_clean":"2024-04-23T10:25:47.589538+0000","last_became_active":"2024-04-23T10:25:47.589172+0000","last_became_peered":"2024-04-23T10:25:47.589172+0000","last_unstale":"2024-04-23T10:25:47.589538+0000","last_undegraded":"2024-04-23T10:25:47.589538+0000","last_fullsized":"2024-04-23T10:25:47.589538+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T10:24:43.025324+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T10:24:43.025324+0000","last_clean_scrub_stamp":"2024-04-23T10:24:43.025324+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T19:47:34.267992+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,8,2],"acting":[0,8,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.30","version":"0'0","reported_seq":17,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-23T10:25:47.719565+0000","last_change":"2024-04-23T10:24:44.057154+0000","last_active":"2024-04-23T10:25:47.719565+0000","last_peered":"2024-04-23T10:25:47.719565+0000","last_clean":"2024-04-23T10:25:47.719565+0000","last_became_active":"2024-04-23T10:24:44.056836+0000","last_became_peered":"2024-04-23T10:24:44.056836+0000","last_unstale":"2024-04-23T10:25:47.719565+0000","last_undegraded":"2024-04-23T10:25:47.719565+0000","last_fullsized":"2024-04-23T10:25:47.719565+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T10:24:43.025324+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T10:24:43.025324+0000","last_clean_scrub_stamp":"2024-04-23T10:24:43.025324+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T13:48:59.107635+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,0],"acting":[1,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.2f","version":"0'0","reported_seq":17,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-23T10:25:47.584741+0000","last_change":"2024-04-23T10:24:44.051175+0000","last_active":"2024-04-23T10:25:47.584741+0000","last_peered":"2024-04-23T10:25:47.584741+0000","last_clean":"2024-04-23T10:25:47.584741+0000","last_became_active":"2024-04-23T10:24:44.050402+0000","last_became_peered":"2024-04-23T10:24:44.050402+0000","last_unstale":"2024-04-23T10:25:47.584741+0000","last_undegraded":"2024-04-23T10:25:47.584741+0000","last_fullsized":"2024-04-23T10:25:47.584741+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T10:24:43.025324+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T10:24:43.025324+0000","last_clean_scrub_stamp":"2024-04-23T10:24:43.025324+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T15:01:26.919701+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,1,0],"acting":[11,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.2e","version":"0'0","reported_seq":17,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-23T10:25:47.580911+0000","last_change":"2024-04-23T10:24:44.053390+0000","last_active":"2024-04-23T10:25:47.580911+0000","last_peered":"2024-04-23T10:25:47.580911+0000","last_clean":"2024-04-23T10:25:47.580911+0000","last_became_active":"2024-04-23T10:24:44.052970+0000","last_became_peered":"2024-04-23T10:24:44.052970+0000","last_unstale":"2024-04-23T10:25:47.580911+0000","last_undegraded":"2024-04-23T10:25:47.580911+0000","last_fullsized":"2024-04-23T10:25:47.580911+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T10:24:43.025324+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T10:24:43.025324+0000","last_clean_scrub_stamp":"2024-04-23T10:24:43.025324+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T13:27:55.704328+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,1,10],"acting":[9,1,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.2d","version":"0'0","reported_seq":17,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-23T10:25:47.582595+0000","last_change":"2024-04-23T10:24:44.049307+0000","last_active":"2024-04-23T10:25:47.582595+0000","last_peered":"2024-04-23T10:25:47.582595+0000","last_clean":"2024-04-23T10:25:47.582595+0000","last_became_active":"2024-04-23T10:24:44.048981+0000","last_became_peered":"2024-04-23T10:24:44.048981+0000","last_unstale":"2024-04-23T10:25:47.582595+0000","last_undegraded":"2024-04-23T10:25:47.582595+0000","last_fullsized":"2024-04-23T10:25:47.582595+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T10:24:43.025324+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T10:24:43.025324+0000","last_clean_scrub_stamp":"2024-04-23T10:24:43.025324+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T12:42:21.804591+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,6],"acting":[2,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.1f","version":"87'9","reported_seq":29,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-23T10:25:47.580889+0000","last_change":"2024-04-23T10:24:43.045804+0000","last_active":"2024-04-23T10:25:47.580889+0000","last_peered":"2024-04-23T10:25:47.580889+0000","last_clean":"2024-04-23T10:25:47.580889+0000","last_became_active":"2024-04-23T10:24:43.045135+0000","last_became_peered":"2024-04-23T10:24:43.045135+0000","last_unstale":"2024-04-23T10:25:47.580889+0000","last_undegraded":"2024-04-23T10:25:47.580889+0000","last_fullsized":"2024-04-23T10:25:47.580889+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T10:24:42.023127+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T10:24:42.023127+0000","last_clean_scrub_stamp":"2024-04-23T10:24:42.023127+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T20:27:28.421644+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,7,8],"acting":[9,7,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.1e","version":"0'0","reported_seq":17,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-23T10:25:47.586405+0000","last_change":"2024-04-23T10:24:44.055008+0000","last_active":"2024-04-23T10:25:47.586405+0000","last_peered":"2024-04-23T10:25:47.586405+0000","last_clean":"2024-04-23T10:25:47.586405+0000","last_became_active":"2024-04-23T10:24:44.054682+0000","last_became_peered":"2024-04-23T10:24:44.054682+0000","last_unstale":"2024-04-23T10:25:47.586405+0000","last_undegraded":"2024-04-23T10:25:47.586405+0000","last_fullsized":"2024-04-23T10:25:47.586405+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T10:24:43.025324+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T10:24:43.025324+0000","last_clean_scrub_stamp":"2024-04-23T10:24:43.025324+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T15:49:58.394958+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,5],"acting":[3,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.1e","version":"87'2","reported_seq":21,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-23T10:25:47.586522+0000","last_change":"2024-04-23T10:24:43.052303+0000","last_active":"2024-04-23T10:25:47.586522+0000","last_peered":"2024-04-23T10:25:47.586522+0000","last_clean":"2024-04-23T10:25:47.586522+0000","last_became_active":"2024-04-23T10:24:43.052010+0000","last_became_peered":"2024-04-23T10:24:43.052010+0000","last_unstale":"2024-04-23T10:25:47.586522+0000","last_undegraded":"2024-04-23T10:25:47.586522+0000","last_fullsized":"2024-04-23T10:25:47.586522+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T10:24:42.023127+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T10:24:42.023127+0000","last_clean_scrub_stamp":"2024-04-23T10:24:42.023127+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T13:03:15.440610+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,5],"acting":[3,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1f","version":"0'0","reported_seq":17,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-23T10:25:47.583624+0000","last_change":"2024-04-23T10:24:44.050749+0000","last_active":"2024-04-23T10:25:47.583624+0000","last_peered":"2024-04-23T10:25:47.583624+0000","last_clean":"2024-04-23T10:25:47.583624+0000","last_became_active":"2024-04-23T10:24:44.049780+0000","last_became_peered":"2024-04-23T10:24:44.049780+0000","last_unstale":"2024-04-23T10:25:47.583624+0000","last_undegraded":"2024-04-23T10:25:47.583624+0000","last_fullsized":"2024-04-23T10:25:47.583624+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T10:24:43.025324+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T10:24:43.025324+0000","last_clean_scrub_stamp":"2024-04-23T10:24:43.025324+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T16:32:37.963776+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,9],"acting":[0,1,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1d","version":"87'4","reported_seq":17,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-23T10:25:47.592399+0000","last_change":"2024-04-23T10:25:47.591939+0000","last_active":"2024-04-23T10:25:47.592399+0000","last_peered":"2024-04-23T10:25:47.592399+0000","last_clean":"2024-04-23T10:25:47.592399+0000","last_became_active":"2024-04-23T10:25:47.586626+0000","last_became_peered":"2024-04-23T10:25:47.586626+0000","last_unstale":"2024-04-23T10:25:47.592399+0000","last_undegraded":"2024-04-23T10:25:47.592399+0000","last_fullsized":"2024-04-23T10:25:47.592399+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T10:24:42.023127+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T10:24:42.023127+0000","last_clean_scrub_stamp":"2024-04-23T10:24:42.023127+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T20:55:38.216483+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":10,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,2,7],"acting":[11,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.1c","version":"0'0","reported_seq":17,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-23T10:25:47.720068+0000","last_change":"2024-04-23T10:24:44.051606+0000","last_active":"2024-04-23T10:25:47.720068+0000","last_peered":"2024-04-23T10:25:47.720068+0000","last_clean":"2024-04-23T10:25:47.720068+0000","last_became_active":"2024-04-23T10:24:44.051191+0000","last_became_peered":"2024-04-23T10:24:44.051191+0000","last_unstale":"2024-04-23T10:25:47.720068+0000","last_undegraded":"2024-04-23T10:25:47.720068+0000","last_fullsized":"2024-04-23T10:25:47.720068+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T10:24:43.025324+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T10:24:43.025324+0000","last_clean_scrub_stamp":"2024-04-23T10:24:43.025324+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T16:21:51.706066+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,10,6],"acting":[1,10,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1c","version":"86'2","reported_seq":21,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-23T10:25:47.588287+0000","last_change":"2024-04-23T10:24:43.048594+0000","last_active":"2024-04-23T10:25:47.588287+0000","last_peered":"2024-04-23T10:25:47.588287+0000","last_clean":"2024-04-23T10:25:47.588287+0000","last_became_active":"2024-04-23T10:24:43.048276+0000","last_became_peered":"2024-04-23T10:24:43.048276+0000","last_unstale":"2024-04-23T10:25:47.588287+0000","last_undegraded":"2024-04-23T10:25:47.588287+0000","last_fullsized":"2024-04-23T10:25:47.588287+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T10:24:42.023127+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T10:24:42.023127+0000","last_clean_scrub_stamp":"2024-04-23T10:24:42.023127+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T12:20:40.196674+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,1,5],"acting":[8,1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.1d","version":"0'0","reported_seq":17,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-23T10:25:47.720397+0000","last_change":"2024-04-23T10:24:44.050970+0000","last_active":"2024-04-23T10:25:47.720397+0000","last_peered":"2024-04-23T10:25:47.720397+0000","last_clean":"2024-04-23T10:25:47.720397+0000","last_became_active":"2024-04-23T10:24:44.050640+0000","last_became_peered":"2024-04-23T10:24:44.050640+0000","last_unstale":"2024-04-23T10:25:47.720397+0000","last_undegraded":"2024-04-23T10:25:47.720397+0000","last_fullsized":"2024-04-23T10:25:47.720397+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T10:24:43.025324+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T10:24:43.025324+0000","last_clean_scrub_stamp":"2024-04-23T10:24:43.025324+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T21:03:24.570290+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,1],"acting":[5,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.1b","version":"87'6","reported_seq":99,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-23T10:25:57.424848+0000","last_change":"2024-04-23T10:24:43.051742+0000","last_active":"2024-04-23T10:25:57.424848+0000","last_peered":"2024-04-23T10:25:57.424848+0000","last_clean":"2024-04-23T10:25:57.424848+0000","last_became_active":"2024-04-23T10:24:43.051420+0000","last_became_peered":"2024-04-23T10:24:43.051420+0000","last_unstale":"2024-04-23T10:25:57.424848+0000","last_undegraded":"2024-04-23T10:25:57.424848+0000","last_fullsized":"2024-04-23T10:25:57.424848+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T10:24:42.023127+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T10:24:42.023127+0000","last_clean_scrub_stamp":"2024-04-23T10:24:42.023127+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T17:54:37.182731+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1013,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":74,"num_read_kb":3,"num_write":3,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,7],"acting":[5,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.1a","version":"0'0","reported_seq":17,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-23T10:25:47.587950+0000","last_change":"2024-04-23T10:24:44.049314+0000","last_active":"2024-04-23T10:25:47.587950+0000","last_peered":"2024-04-23T10:25:47.587950+0000","last_clean":"2024-04-23T10:25:47.587950+0000","last_became_active":"2024-04-23T10:24:44.048352+0000","last_became_peered":"2024-04-23T10:24:44.048352+0000","last_unstale":"2024-04-23T10:25:47.587950+0000","last_undegraded":"2024-04-23T10:25:47.587950+0000","last_fullsized":"2024-04-23T10:25:47.587950+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T10:24:43.025324+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T10:24:43.025324+0000","last_clean_scrub_stamp":"2024-04-23T10:24:43.025324+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T16:57:29.957716+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,9,5],"acting":[8,9,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"2.1a","version":"87'3","reported_seq":22,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-23T10:25:47.720011+0000","last_change":"2024-04-23T10:24:43.051362+0000","last_active":"2024-04-23T10:25:47.720011+0000","last_peered":"2024-04-23T10:25:47.720011+0000","last_clean":"2024-04-23T10:25:47.720011+0000","last_became_active":"2024-04-23T10:24:43.051045+0000","last_became_peered":"2024-04-23T10:24:43.051045+0000","last_unstale":"2024-04-23T10:25:47.720011+0000","last_undegraded":"2024-04-23T10:25:47.720011+0000","last_fullsized":"2024-04-23T10:25:47.720011+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T10:24:42.023127+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T10:24:42.023127+0000","last_clean_scrub_stamp":"2024-04-23T10:24:42.023127+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T15:40:53.618281+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,8,6],"acting":[10,8,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.1b","version":"0'0","reported_seq":17,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-23T10:25:47.720118+0000","last_change":"2024-04-23T10:24:44.040323+0000","last_active":"2024-04-23T10:25:47.720118+0000","last_peered":"2024-04-23T10:25:47.720118+0000","last_clean":"2024-04-23T10:25:47.720118+0000","last_became_active":"2024-04-23T10:24:44.039995+0000","last_became_peered":"2024-04-23T10:24:44.039995+0000","last_unstale":"2024-04-23T10:25:47.720118+0000","last_undegraded":"2024-04-23T10:25:47.720118+0000","last_fullsized":"2024-04-23T10:25:47.720118+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T10:24:43.025324+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T10:24:43.025324+0000","last_clean_scrub_stamp":"2024-04-23T10:24:43.025324+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T16:38:48.756776+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,3],"acting":[5,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.19","version":"87'2","reported_seq":21,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-23T10:25:47.584155+0000","last_change":"2024-04-23T10:24:43.048754+0000","last_active":"2024-04-23T10:25:47.584155+0000","last_peered":"2024-04-23T10:25:47.584155+0000","last_clean":"2024-04-23T10:25:47.584155+0000","last_became_active":"2024-04-23T10:24:43.047962+0000","last_became_peered":"2024-04-23T10:24:43.047962+0000","last_unstale":"2024-04-23T10:25:47.584155+0000","last_undegraded":"2024-04-23T10:25:47.584155+0000","last_fullsized":"2024-04-23T10:25:47.584155+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T10:24:42.023127+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T10:24:42.023127+0000","last_clean_scrub_stamp":"2024-04-23T10:24:42.023127+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T21:08:38.487185+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,9],"acting":[6,4,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.18","version":"0'0","reported_seq":17,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-23T10:25:47.585662+0000","last_change":"2024-04-23T10:24:44.045022+0000","last_active":"2024-04-23T10:25:47.585662+0000","last_peered":"2024-04-23T10:25:47.585662+0000","last_clean":"2024-04-23T10:25:47.585662+0000","last_became_active":"2024-04-23T10:24:44.044590+0000","last_became_peered":"2024-04-23T10:24:44.044590+0000","last_unstale":"2024-04-23T10:25:47.585662+0000","last_undegraded":"2024-04-23T10:25:47.585662+0000","last_fullsized":"2024-04-23T10:25:47.585662+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T10:24:43.025324+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T10:24:43.025324+0000","last_clean_scrub_stamp":"2024-04-23T10:24:43.025324+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T13:20:49.892013+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,11,6],"acting":[3,11,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.18","version":"87'1","reported_seq":20,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-23T10:25:47.720010+0000","last_change":"2024-04-23T10:24:43.050170+0000","last_active":"2024-04-23T10:25:47.720010+0000","last_peered":"2024-04-23T10:25:47.720010+0000","last_clean":"2024-04-23T10:25:47.720010+0000","last_became_active":"2024-04-23T10:24:43.049070+0000","last_became_peered":"2024-04-23T10:24:43.049070+0000","last_unstale":"2024-04-23T10:25:47.720010+0000","last_undegraded":"2024-04-23T10:25:47.720010+0000","last_fullsized":"2024-04-23T10:25:47.720010+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T10:24:42.023127+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T10:24:42.023127+0000","last_clean_scrub_stamp":"2024-04-23T10:24:42.023127+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T19:34:13.983628+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,7],"acting":[5,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.19","version":"0'0","reported_seq":17,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-23T10:25:47.580673+0000","last_change":"2024-04-23T10:24:44.047165+0000","last_active":"2024-04-23T10:25:47.580673+0000","last_peered":"2024-04-23T10:25:47.580673+0000","last_clean":"2024-04-23T10:25:47.580673+0000","last_became_active":"2024-04-23T10:24:44.046776+0000","last_became_peered":"2024-04-23T10:24:44.046776+0000","last_unstale":"2024-04-23T10:25:47.580673+0000","last_undegraded":"2024-04-23T10:25:47.580673+0000","last_fullsized":"2024-04-23T10:25:47.580673+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T10:24:43.025324+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T10:24:43.025324+0000","last_clean_scrub_stamp":"2024-04-23T10:24:43.025324+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T18:11:57.181747+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,3,10],"acting":[9,3,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"2.17","version":"87'3","reported_seq":91,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-23T10:25:55.421165+0000","last_change":"2024-04-23T10:24:43.049785+0000","last_active":"2024-04-23T10:25:55.421165+0000","last_peered":"2024-04-23T10:25:55.421165+0000","last_clean":"2024-04-23T10:25:55.421165+0000","last_became_active":"2024-04-23T10:24:43.049402+0000","last_became_peered":"2024-04-23T10:24:43.049402+0000","last_unstale":"2024-04-23T10:25:55.421165+0000","last_undegraded":"2024-04-23T10:25:55.421165+0000","last_fullsized":"2024-04-23T10:25:55.421165+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T10:24:42.023127+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T10:24:42.023127+0000","last_clean_scrub_stamp":"2024-04-23T10:24:42.023127+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T14:20:13.774082+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":69,"num_read_kb":69,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,5,9],"acting":[10,5,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.16","version":"0'0","reported_seq":17,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-23T10:25:47.720748+0000","last_change":"2024-04-23T10:24:44.051768+0000","last_active":"2024-04-23T10:25:47.720748+0000","last_peered":"2024-04-23T10:25:47.720748+0000","last_clean":"2024-04-23T10:25:47.720748+0000","last_became_active":"2024-04-23T10:24:44.051228+0000","last_became_peered":"2024-04-23T10:24:44.051228+0000","last_unstale":"2024-04-23T10:25:47.720748+0000","last_undegraded":"2024-04-23T10:25:47.720748+0000","last_fullsized":"2024-04-23T10:25:47.720748+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T10:24:43.025324+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T10:24:43.025324+0000","last_clean_scrub_stamp":"2024-04-23T10:24:43.025324+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T14:09:47.649083+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,10,9],"acting":[1,10,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.16","version":"87'2","reported_seq":21,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-23T10:25:47.582738+0000","last_change":"2024-04-23T10:24:43.043944+0000","last_active":"2024-04-23T10:25:47.582738+0000","last_peered":"2024-04-23T10:25:47.582738+0000","last_clean":"2024-04-23T10:25:47.582738+0000","last_became_active":"2024-04-23T10:24:43.043648+0000","last_became_peered":"2024-04-23T10:24:43.043648+0000","last_unstale":"2024-04-23T10:25:47.582738+0000","last_undegraded":"2024-04-23T10:25:47.582738+0000","last_fullsized":"2024-04-23T10:25:47.582738+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T10:24:42.023127+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T10:24:42.023127+0000","last_clean_scrub_stamp":"2024-04-23T10:24:42.023127+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T13:41:15.724662+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,2],"acting":[7,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.17","version":"0'0","reported_seq":23,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-23T10:25:47.721534+0000","last_change":"2024-04-23T10:25:47.721534+0000","last_active":"2024-04-23T10:25:47.721534+0000","last_peered":"2024-04-23T10:25:47.721534+0000","last_clean":"2024-04-23T10:25:47.721534+0000","last_became_active":"2024-04-23T10:25:47.721130+0000","last_became_peered":"2024-04-23T10:25:47.721130+0000","last_unstale":"2024-04-23T10:25:47.721534+0000","last_undegraded":"2024-04-23T10:25:47.721534+0000","last_fullsized":"2024-04-23T10:25:47.721534+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T10:24:43.025324+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T10:24:43.025324+0000","last_clean_scrub_stamp":"2024-04-23T10:24:43.025324+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T21:21:26.537486+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,5],"acting":[0,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.15","version":"87'4","reported_seq":24,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-23T10:25:47.720578+0000","last_change":"2024-04-23T10:24:43.048517+0000","last_active":"2024-04-23T10:25:47.720578+0000","last_peered":"2024-04-23T10:25:47.720578+0000","last_clean":"2024-04-23T10:25:47.720578+0000","last_became_active":"2024-04-23T10:24:43.048183+0000","last_became_peered":"2024-04-23T10:24:43.048183+0000","last_unstale":"2024-04-23T10:25:47.720578+0000","last_undegraded":"2024-04-23T10:25:47.720578+0000","last_fullsized":"2024-04-23T10:25:47.720578+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T10:24:42.023127+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T10:24:42.023127+0000","last_clean_scrub_stamp":"2024-04-23T10:24:42.023127+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T20:50:15.017833+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":672,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":5,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,10],"acting":[5,0,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.14","version":"0'0","reported_seq":17,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-23T10:25:47.720284+0000","last_change":"2024-04-23T10:24:44.055598+0000","last_active":"2024-04-23T10:25:47.720284+0000","last_peered":"2024-04-23T10:25:47.720284+0000","last_clean":"2024-04-23T10:25:47.720284+0000","last_became_active":"2024-04-23T10:24:44.053787+0000","last_became_peered":"2024-04-23T10:24:44.053787+0000","last_unstale":"2024-04-23T10:25:47.720284+0000","last_undegraded":"2024-04-23T10:25:47.720284+0000","last_fullsized":"2024-04-23T10:25:47.720284+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T10:24:43.025324+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T10:24:43.025324+0000","last_clean_scrub_stamp":"2024-04-23T10:24:43.025324+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T20:04:21.603306+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,8,4],"acting":[1,8,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.14","version":"87'28","reported_seq":193,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-23T10:25:56.434612+0000","last_change":"2024-04-23T10:24:43.044601+0000","last_active":"2024-04-23T10:25:56.434612+0000","last_peered":"2024-04-23T10:25:56.434612+0000","last_clean":"2024-04-23T10:25:56.434612+0000","last_became_active":"2024-04-23T10:24:43.044238+0000","last_became_peered":"2024-04-23T10:24:43.044238+0000","last_unstale":"2024-04-23T10:25:56.434612+0000","last_undegraded":"2024-04-23T10:25:56.434612+0000","last_fullsized":"2024-04-23T10:25:56.434612+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T10:24:42.023127+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T10:24:42.023127+0000","last_clean_scrub_stamp":"2024-04-23T10:24:42.023127+0000","objects_scrubbed":0,"log_size":28,"log_dups_size":0,"ondisk_log_size":28,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T12:53:26.275771+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":61046,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":146,"num_read_kb":132,"num_write":29,"num_write_kb":69,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,1,2],"acting":[10,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.15","version":"0'0","reported_seq":17,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-23T10:25:47.588308+0000","last_change":"2024-04-23T10:24:44.051375+0000","last_active":"2024-04-23T10:25:47.588308+0000","last_peered":"2024-04-23T10:25:47.588308+0000","last_clean":"2024-04-23T10:25:47.588308+0000","last_became_active":"2024-04-23T10:24:44.050924+0000","last_became_peered":"2024-04-23T10:24:44.050924+0000","last_unstale":"2024-04-23T10:25:47.588308+0000","last_undegraded":"2024-04-23T10:25:47.588308+0000","last_fullsized":"2024-04-23T10:25:47.588308+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T10:24:43.025324+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T10:24:43.025324+0000","last_clean_scrub_stamp":"2024-04-23T10:24:43.025324+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T21:50:57.534760+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,11,7],"acting":[8,11,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"2.13","version":"87'6","reported_seq":26,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-23T10:25:47.582395+0000","last_change":"2024-04-23T10:24:43.044457+0000","last_active":"2024-04-23T10:25:47.582395+0000","last_peered":"2024-04-23T10:25:47.582395+0000","last_clean":"2024-04-23T10:25:47.582395+0000","last_became_active":"2024-04-23T10:24:43.043956+0000","last_became_peered":"2024-04-23T10:24:43.043956+0000","last_unstale":"2024-04-23T10:25:47.582395+0000","last_undegraded":"2024-04-23T10:25:47.582395+0000","last_fullsized":"2024-04-23T10:25:47.582395+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T10:24:42.023127+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T10:24:42.023127+0000","last_clean_scrub_stamp":"2024-04-23T10:24:42.023127+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T20:16:08.186729+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":9,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,10,3],"acting":[2,10,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.12","version":"0'0","reported_seq":23,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-23T10:25:47.587240+0000","last_change":"2024-04-23T10:25:47.587240+0000","last_active":"2024-04-23T10:25:47.587240+0000","last_peered":"2024-04-23T10:25:47.587240+0000","last_clean":"2024-04-23T10:25:47.587240+0000","last_became_active":"2024-04-23T10:25:47.586659+0000","last_became_peered":"2024-04-23T10:25:47.586659+0000","last_unstale":"2024-04-23T10:25:47.587240+0000","last_undegraded":"2024-04-23T10:25:47.587240+0000","last_fullsized":"2024-04-23T10:25:47.587240+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T10:24:43.025324+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T10:24:43.025324+0000","last_clean_scrub_stamp":"2024-04-23T10:24:43.025324+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T19:55:03.256183+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,9],"acting":[6,7,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.12","version":"87'3","reported_seq":22,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-23T10:25:47.719998+0000","last_change":"2024-04-23T10:24:43.052181+0000","last_active":"2024-04-23T10:25:47.719998+0000","last_peered":"2024-04-23T10:25:47.719998+0000","last_clean":"2024-04-23T10:25:47.719998+0000","last_became_active":"2024-04-23T10:24:43.051793+0000","last_became_peered":"2024-04-23T10:24:43.051793+0000","last_unstale":"2024-04-23T10:25:47.719998+0000","last_undegraded":"2024-04-23T10:25:47.719998+0000","last_fullsized":"2024-04-23T10:25:47.719998+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T10:24:42.023127+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T10:24:42.023127+0000","last_clean_scrub_stamp":"2024-04-23T10:24:42.023127+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T20:57:59.015115+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,7],"acting":[5,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.13","version":"0'0","reported_seq":17,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-23T10:25:47.582606+0000","last_change":"2024-04-23T10:24:44.062689+0000","last_active":"2024-04-23T10:25:47.582606+0000","last_peered":"2024-04-23T10:25:47.582606+0000","last_clean":"2024-04-23T10:25:47.582606+0000","last_became_active":"2024-04-23T10:24:44.062331+0000","last_became_peered":"2024-04-23T10:24:44.062331+0000","last_unstale":"2024-04-23T10:25:47.582606+0000","last_undegraded":"2024-04-23T10:25:47.582606+0000","last_fullsized":"2024-04-23T10:25:47.582606+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T10:24:43.025324+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T10:24:43.025324+0000","last_clean_scrub_stamp":"2024-04-23T10:24:43.025324+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T14:06:04.989156+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,6],"acting":[7,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.11","version":"87'4","reported_seq":96,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-23T10:25:56.423831+0000","last_change":"2024-04-23T10:24:43.043901+0000","last_active":"2024-04-23T10:25:56.423831+0000","last_peered":"2024-04-23T10:25:56.423831+0000","last_clean":"2024-04-23T10:25:56.423831+0000","last_became_active":"2024-04-23T10:24:43.043598+0000","last_became_peered":"2024-04-23T10:24:43.043598+0000","last_unstale":"2024-04-23T10:25:56.423831+0000","last_undegraded":"2024-04-23T10:25:56.423831+0000","last_fullsized":"2024-04-23T10:25:56.423831+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T10:24:42.023127+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T10:24:42.023127+0000","last_clean_scrub_stamp":"2024-04-23T10:24:42.023127+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T14:00:06.338084+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":979,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":74,"num_read_kb":4,"num_write":3,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,10],"acting":[4,7,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.10","version":"0'0","reported_seq":17,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-23T10:25:47.582241+0000","last_change":"2024-04-23T10:24:44.058773+0000","last_active":"2024-04-23T10:25:47.582241+0000","last_peered":"2024-04-23T10:25:47.582241+0000","last_clean":"2024-04-23T10:25:47.582241+0000","last_became_active":"2024-04-23T10:24:44.058420+0000","last_became_peered":"2024-04-23T10:24:44.058420+0000","last_unstale":"2024-04-23T10:25:47.582241+0000","last_undegraded":"2024-04-23T10:25:47.582241+0000","last_fullsized":"2024-04-23T10:25:47.582241+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T10:24:43.025324+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T10:24:43.025324+0000","last_clean_scrub_stamp":"2024-04-23T10:24:43.025324+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T12:44:30.023047+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,10],"acting":[7,5,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.10","version":"87'15","reported_seq":35,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-23T10:25:47.587839+0000","last_change":"2024-04-23T10:24:43.044700+0000","last_active":"2024-04-23T10:25:47.587839+0000","last_peered":"2024-04-23T10:25:47.587839+0000","last_clean":"2024-04-23T10:25:47.587839+0000","last_became_active":"2024-04-23T10:24:43.044394+0000","last_became_peered":"2024-04-23T10:24:43.044394+0000","last_unstale":"2024-04-23T10:25:47.587839+0000","last_undegraded":"2024-04-23T10:25:47.587839+0000","last_fullsized":"2024-04-23T10:25:47.587839+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T10:24:42.023127+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T10:24:42.023127+0000","last_clean_scrub_stamp":"2024-04-23T10:24:42.023127+0000","objects_scrubbed":0,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T15:16:41.367517+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":112,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":20,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":6,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,0,2],"acting":[8,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.11","version":"0'0","reported_seq":17,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-23T10:25:47.719971+0000","last_change":"2024-04-23T10:24:44.049739+0000","last_active":"2024-04-23T10:25:47.719971+0000","last_peered":"2024-04-23T10:25:47.719971+0000","last_clean":"2024-04-23T10:25:47.719971+0000","last_became_active":"2024-04-23T10:24:44.049414+0000","last_became_peered":"2024-04-23T10:24:44.049414+0000","last_unstale":"2024-04-23T10:25:47.719971+0000","last_undegraded":"2024-04-23T10:25:47.719971+0000","last_fullsized":"2024-04-23T10:25:47.719971+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T10:24:43.025324+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T10:24:43.025324+0000","last_clean_scrub_stamp":"2024-04-23T10:24:43.025324+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T12:51:02.645259+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,1],"acting":[5,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.f","version":"87'2","reported_seq":22,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-23T10:25:47.588324+0000","last_change":"2024-04-23T10:24:43.048884+0000","last_active":"2024-04-23T10:25:47.588324+0000","last_peered":"2024-04-23T10:25:47.588324+0000","last_clean":"2024-04-23T10:25:47.588324+0000","last_became_active":"2024-04-23T10:24:43.048583+0000","last_became_peered":"2024-04-23T10:24:43.048583+0000","last_unstale":"2024-04-23T10:25:47.588324+0000","last_undegraded":"2024-04-23T10:25:47.588324+0000","last_fullsized":"2024-04-23T10:25:47.588324+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T10:24:42.023127+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T10:24:42.023127+0000","last_clean_scrub_stamp":"2024-04-23T10:24:42.023127+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T17:55:57.869700+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,5,4],"acting":[8,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.e","version":"0'0","reported_seq":23,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-23T10:25:47.722078+0000","last_change":"2024-04-23T10:25:47.722078+0000","last_active":"2024-04-23T10:25:47.722078+0000","last_peered":"2024-04-23T10:25:47.722078+0000","last_clean":"2024-04-23T10:25:47.722078+0000","last_became_active":"2024-04-23T10:25:47.721730+0000","last_became_peered":"2024-04-23T10:25:47.721730+0000","last_unstale":"2024-04-23T10:25:47.722078+0000","last_undegraded":"2024-04-23T10:25:47.722078+0000","last_fullsized":"2024-04-23T10:25:47.722078+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T10:24:43.025324+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T10:24:43.025324+0000","last_clean_scrub_stamp":"2024-04-23T10:24:43.025324+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T18:18:40.916492+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,9,4],"acting":[11,9,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":19,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-23T10:25:47.580777+0000","last_change":"2024-04-23T10:24:43.044375+0000","last_active":"2024-04-23T10:25:47.580777+0000","last_peered":"2024-04-23T10:25:47.580777+0000","last_clean":"2024-04-23T10:25:47.580777+0000","last_became_active":"2024-04-23T10:24:43.043914+0000","last_became_peered":"2024-04-23T10:24:43.043914+0000","last_unstale":"2024-04-23T10:25:47.580777+0000","last_undegraded":"2024-04-23T10:25:47.580777+0000","last_fullsized":"2024-04-23T10:25:47.580777+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T10:24:42.023127+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T10:24:42.023127+0000","last_clean_scrub_stamp":"2024-04-23T10:24:42.023127+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T16:22:44.165156+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,2,0],"acting":[9,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.5","version":"87'1","reported_seq":18,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-23T10:25:47.720416+0000","last_change":"2024-04-23T10:24:44.057272+0000","last_active":"2024-04-23T10:25:47.720416+0000","last_peered":"2024-04-23T10:25:47.720416+0000","last_clean":"2024-04-23T10:25:47.720416+0000","last_became_active":"2024-04-23T10:24:44.056968+0000","last_became_peered":"2024-04-23T10:24:44.056968+0000","last_unstale":"2024-04-23T10:25:47.720416+0000","last_undegraded":"2024-04-23T10:25:47.720416+0000","last_fullsized":"2024-04-23T10:25:47.720416+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T10:24:43.025324+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T10:24:43.025324+0000","last_clean_scrub_stamp":"2024-04-23T10:24:43.025324+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T11:10:09.021845+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":117,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,3,6],"acting":[10,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"2.2","version":"87'1","reported_seq":20,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-23T10:25:47.582434+0000","last_change":"2024-04-23T10:24:43.051822+0000","last_active":"2024-04-23T10:25:47.582434+0000","last_peered":"2024-04-23T10:25:47.582434+0000","last_clean":"2024-04-23T10:25:47.582434+0000","last_became_active":"2024-04-23T10:24:43.051491+0000","last_became_peered":"2024-04-23T10:24:43.051491+0000","last_unstale":"2024-04-23T10:25:47.582434+0000","last_undegraded":"2024-04-23T10:25:47.582434+0000","last_fullsized":"2024-04-23T10:25:47.582434+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T10:24:42.023127+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T10:24:42.023127+0000","last_clean_scrub_stamp":"2024-04-23T10:24:42.023127+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T13:49:23.109064+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,1],"acting":[7,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.3","version":"0'0","reported_seq":17,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-23T10:25:47.720861+0000","last_change":"2024-04-23T10:24:44.053604+0000","last_active":"2024-04-23T10:25:47.720861+0000","last_peered":"2024-04-23T10:25:47.720861+0000","last_clean":"2024-04-23T10:25:47.720861+0000","last_became_active":"2024-04-23T10:24:44.053147+0000","last_became_peered":"2024-04-23T10:24:44.053147+0000","last_unstale":"2024-04-23T10:25:47.720861+0000","last_undegraded":"2024-04-23T10:25:47.720861+0000","last_fullsized":"2024-04-23T10:25:47.720861+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T10:24:43.025324+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T10:24:43.025324+0000","last_clean_scrub_stamp":"2024-04-23T10:24:43.025324+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T21:58:38.981282+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,4,0],"acting":[10,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.0","version":"0'0","reported_seq":17,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-23T10:25:47.720397+0000","last_change":"2024-04-23T10:24:44.052564+0000","last_active":"2024-04-23T10:25:47.720397+0000","last_peered":"2024-04-23T10:25:47.720397+0000","last_clean":"2024-04-23T10:25:47.720397+0000","last_became_active":"2024-04-23T10:24:44.052201+0000","last_became_peered":"2024-04-23T10:24:44.052201+0000","last_unstale":"2024-04-23T10:25:47.720397+0000","last_undegraded":"2024-04-23T10:25:47.720397+0000","last_fullsized":"2024-04-23T10:25:47.720397+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T10:24:43.025324+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T10:24:43.025324+0000","last_clean_scrub_stamp":"2024-04-23T10:24:43.025324+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T16:25:54.725347+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,4],"acting":[5,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.1","version":"87'4","reported_seq":93,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-23T10:25:56.423143+0000","last_change":"2024-04-23T10:24:43.049067+0000","last_active":"2024-04-23T10:25:56.423143+0000","last_peered":"2024-04-23T10:25:56.423143+0000","last_clean":"2024-04-23T10:25:56.423143+0000","last_became_active":"2024-04-23T10:24:43.048400+0000","last_became_peered":"2024-04-23T10:24:43.048400+0000","last_unstale":"2024-04-23T10:25:56.423143+0000","last_undegraded":"2024-04-23T10:25:56.423143+0000","last_fullsized":"2024-04-23T10:25:56.423143+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T10:24:42.023127+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T10:24:42.023127+0000","last_clean_scrub_stamp":"2024-04-23T10:24:42.023127+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T12:17:02.439054+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":70,"num_read_kb":70,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,0],"acting":[6,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.2","version":"0'0","reported_seq":17,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-23T10:25:47.586431+0000","last_change":"2024-04-23T10:24:44.059114+0000","last_active":"2024-04-23T10:25:47.586431+0000","last_peered":"2024-04-23T10:25:47.586431+0000","last_clean":"2024-04-23T10:25:47.586431+0000","last_became_active":"2024-04-23T10:24:44.058618+0000","last_became_peered":"2024-04-23T10:24:44.058618+0000","last_unstale":"2024-04-23T10:25:47.586431+0000","last_undegraded":"2024-04-23T10:25:47.586431+0000","last_fullsized":"2024-04-23T10:25:47.586431+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T10:24:43.025324+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T10:24:43.025324+0000","last_clean_scrub_stamp":"2024-04-23T10:24:43.025324+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T20:18:09.707062+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,1],"acting":[3,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.3","version":"87'3","reported_seq":24,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-23T10:25:47.719834+0000","last_change":"2024-04-23T10:24:43.039725+0000","last_active":"2024-04-23T10:25:47.719834+0000","last_peered":"2024-04-23T10:25:47.719834+0000","last_clean":"2024-04-23T10:25:47.719834+0000","last_became_active":"2024-04-23T10:24:43.039187+0000","last_became_peered":"2024-04-23T10:24:43.039187+0000","last_unstale":"2024-04-23T10:25:47.719834+0000","last_undegraded":"2024-04-23T10:25:47.719834+0000","last_fullsized":"2024-04-23T10:25:47.719834+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T10:24:42.023127+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T10:24:42.023127+0000","last_clean_scrub_stamp":"2024-04-23T10:24:42.023127+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T17:20:50.844878+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":3,"num_read_kb":3,"num_write":4,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,10,9],"acting":[4,10,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.0","version":"22'137","reported_seq":350,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-23T10:25:47.582389+0000","last_change":"2024-04-23T10:23:31.092860+0000","last_active":"2024-04-23T10:25:47.582389+0000","last_peered":"2024-04-23T10:25:47.582389+0000","last_clean":"2024-04-23T10:25:47.582389+0000","last_became_active":"2024-04-23T10:23:31.092357+0000","last_became_peered":"2024-04-23T10:23:31.092357+0000","last_unstale":"2024-04-23T10:25:47.582389+0000","last_undegraded":"2024-04-23T10:25:47.582389+0000","last_fullsized":"2024-04-23T10:25:47.582389+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T10:20:12.201233+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T10:20:12.201233+0000","last_clean_scrub_stamp":"2024-04-23T10:20:12.201233+0000","objects_scrubbed":0,"log_size":137,"log_dups_size":0,"ondisk_log_size":137,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T21:28:56.804053+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":136,"num_read_kb":119,"num_write":181,"num_write_kb":2938,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":1180736,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,10],"acting":[7,5,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1","version":"0'0","reported_seq":17,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-23T10:25:47.583932+0000","last_change":"2024-04-23T10:24:44.058393+0000","last_active":"2024-04-23T10:25:47.583932+0000","last_peered":"2024-04-23T10:25:47.583932+0000","last_clean":"2024-04-23T10:25:47.583932+0000","last_became_active":"2024-04-23T10:24:44.058075+0000","last_became_peered":"2024-04-23T10:24:44.058075+0000","last_unstale":"2024-04-23T10:25:47.583932+0000","last_undegraded":"2024-04-23T10:25:47.583932+0000","last_fullsized":"2024-04-23T10:25:47.583932+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T10:24:43.025324+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T10:24:43.025324+0000","last_clean_scrub_stamp":"2024-04-23T10:24:43.025324+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T10:32:05.693311+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,2],"acting":[6,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.0","version":"87'3","reported_seq":22,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-23T10:25:47.582491+0000","last_change":"2024-04-23T10:24:43.048678+0000","last_active":"2024-04-23T10:25:47.582491+0000","last_peered":"2024-04-23T10:25:47.582491+0000","last_clean":"2024-04-23T10:25:47.582491+0000","last_became_active":"2024-04-23T10:24:43.048341+0000","last_became_peered":"2024-04-23T10:24:43.048341+0000","last_unstale":"2024-04-23T10:25:47.582491+0000","last_undegraded":"2024-04-23T10:25:47.582491+0000","last_fullsized":"2024-04-23T10:25:47.582491+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T10:24:42.023127+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T10:24:42.023127+0000","last_clean_scrub_stamp":"2024-04-23T10:24:42.023127+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T12:24:33.705325+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,8],"acting":[7,5,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.5","version":"87'1","reported_seq":21,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-23T10:25:47.588426+0000","last_change":"2024-04-23T10:24:43.046105+0000","last_active":"2024-04-23T10:25:47.588426+0000","last_peered":"2024-04-23T10:25:47.588426+0000","last_clean":"2024-04-23T10:25:47.588426+0000","last_became_active":"2024-04-23T10:24:43.045804+0000","last_became_peered":"2024-04-23T10:24:43.045804+0000","last_unstale":"2024-04-23T10:25:47.588426+0000","last_undegraded":"2024-04-23T10:25:47.588426+0000","last_fullsized":"2024-04-23T10:25:47.588426+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T10:24:42.023127+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T10:24:42.023127+0000","last_clean_scrub_stamp":"2024-04-23T10:24:42.023127+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T22:23:35.927468+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,0,9],"acting":[8,0,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.4","version":"0'0","reported_seq":17,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-23T10:25:47.584146+0000","last_change":"2024-04-23T10:24:44.059731+0000","last_active":"2024-04-23T10:25:47.584146+0000","last_peered":"2024-04-23T10:25:47.584146+0000","last_clean":"2024-04-23T10:25:47.584146+0000","last_became_active":"2024-04-23T10:24:44.059352+0000","last_became_peered":"2024-04-23T10:24:44.059352+0000","last_unstale":"2024-04-23T10:25:47.584146+0000","last_undegraded":"2024-04-23T10:25:47.584146+0000","last_fullsized":"2024-04-23T10:25:47.584146+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T10:24:43.025324+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T10:24:43.025324+0000","last_clean_scrub_stamp":"2024-04-23T10:24:43.025324+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T16:36:08.985444+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2,9],"acting":[6,2,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.6","version":"87'8","reported_seq":27,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-23T10:25:47.580989+0000","last_change":"2024-04-23T10:24:43.044527+0000","last_active":"2024-04-23T10:25:47.580989+0000","last_peered":"2024-04-23T10:25:47.580989+0000","last_clean":"2024-04-23T10:25:47.580989+0000","last_became_active":"2024-04-23T10:24:43.044212+0000","last_became_peered":"2024-04-23T10:24:43.044212+0000","last_unstale":"2024-04-23T10:25:47.580989+0000","last_undegraded":"2024-04-23T10:25:47.580989+0000","last_fullsized":"2024-04-23T10:25:47.580989+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T10:24:42.023127+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T10:24:42.023127+0000","last_clean_scrub_stamp":"2024-04-23T10:24:42.023127+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T19:39:46.369132+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":14,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,8,3],"acting":[9,8,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.7","version":"0'0","reported_seq":17,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-23T10:25:47.586588+0000","last_change":"2024-04-23T10:24:44.055920+0000","last_active":"2024-04-23T10:25:47.586588+0000","last_peered":"2024-04-23T10:25:47.586588+0000","last_clean":"2024-04-23T10:25:47.586588+0000","last_became_active":"2024-04-23T10:24:44.055411+0000","last_became_peered":"2024-04-23T10:24:44.055411+0000","last_unstale":"2024-04-23T10:25:47.586588+0000","last_undegraded":"2024-04-23T10:25:47.586588+0000","last_fullsized":"2024-04-23T10:25:47.586588+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T10:24:43.025324+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T10:24:43.025324+0000","last_clean_scrub_stamp":"2024-04-23T10:24:43.025324+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T14:43:10.446790+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,11,7],"acting":[3,11,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.7","version":"87'4","reported_seq":23,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-23T10:25:47.720935+0000","last_change":"2024-04-23T10:24:43.047989+0000","last_active":"2024-04-23T10:25:47.720935+0000","last_peered":"2024-04-23T10:25:47.720935+0000","last_clean":"2024-04-23T10:25:47.720935+0000","last_became_active":"2024-04-23T10:24:43.047625+0000","last_became_peered":"2024-04-23T10:24:43.047625+0000","last_unstale":"2024-04-23T10:25:47.720935+0000","last_undegraded":"2024-04-23T10:25:47.720935+0000","last_fullsized":"2024-04-23T10:25:47.720935+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T10:24:42.023127+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T10:24:42.023127+0000","last_clean_scrub_stamp":"2024-04-23T10:24:42.023127+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T13:50:16.647176+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,11,6],"acting":[4,11,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.6","version":"0'0","reported_seq":17,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-23T10:25:47.583380+0000","last_change":"2024-04-23T10:24:44.049214+0000","last_active":"2024-04-23T10:25:47.583380+0000","last_peered":"2024-04-23T10:25:47.583380+0000","last_clean":"2024-04-23T10:25:47.583380+0000","last_became_active":"2024-04-23T10:24:44.048869+0000","last_became_peered":"2024-04-23T10:24:44.048869+0000","last_unstale":"2024-04-23T10:25:47.583380+0000","last_undegraded":"2024-04-23T10:25:47.583380+0000","last_fullsized":"2024-04-23T10:25:47.583380+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T10:24:43.025324+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T10:24:43.025324+0000","last_clean_scrub_stamp":"2024-04-23T10:24:43.025324+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T18:17:09.772428+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,4],"acting":[0,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.8","version":"87'2","reported_seq":21,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-23T10:25:47.719683+0000","last_change":"2024-04-23T10:24:43.046189+0000","last_active":"2024-04-23T10:25:47.719683+0000","last_peered":"2024-04-23T10:25:47.719683+0000","last_clean":"2024-04-23T10:25:47.719683+0000","last_became_active":"2024-04-23T10:24:43.045848+0000","last_became_peered":"2024-04-23T10:24:43.045848+0000","last_unstale":"2024-04-23T10:25:47.719683+0000","last_undegraded":"2024-04-23T10:25:47.719683+0000","last_fullsized":"2024-04-23T10:25:47.719683+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T10:24:42.023127+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T10:24:42.023127+0000","last_clean_scrub_stamp":"2024-04-23T10:24:42.023127+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T22:23:39.180377+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,0],"acting":[1,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.9","version":"0'0","reported_seq":17,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-23T10:25:47.588246+0000","last_change":"2024-04-23T10:24:44.048415+0000","last_active":"2024-04-23T10:25:47.588246+0000","last_peered":"2024-04-23T10:25:47.588246+0000","last_clean":"2024-04-23T10:25:47.588246+0000","last_became_active":"2024-04-23T10:24:44.048030+0000","last_became_peered":"2024-04-23T10:24:44.048030+0000","last_unstale":"2024-04-23T10:25:47.588246+0000","last_undegraded":"2024-04-23T10:25:47.588246+0000","last_fullsized":"2024-04-23T10:25:47.588246+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T10:24:43.025324+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T10:24:43.025324+0000","last_clean_scrub_stamp":"2024-04-23T10:24:43.025324+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T14:42:09.920927+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,11,10],"acting":[8,11,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"2.9","version":"87'3","reported_seq":22,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-23T10:25:47.585862+0000","last_change":"2024-04-23T10:24:43.041525+0000","last_active":"2024-04-23T10:25:47.585862+0000","last_peered":"2024-04-23T10:25:47.585862+0000","last_clean":"2024-04-23T10:25:47.585862+0000","last_became_active":"2024-04-23T10:24:43.041200+0000","last_became_peered":"2024-04-23T10:24:43.041200+0000","last_unstale":"2024-04-23T10:25:47.585862+0000","last_undegraded":"2024-04-23T10:25:47.585862+0000","last_fullsized":"2024-04-23T10:25:47.585862+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T10:24:42.023127+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T10:24:42.023127+0000","last_clean_scrub_stamp":"2024-04-23T10:24:42.023127+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T16:45:06.846991+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,9,10],"acting":[3,9,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.8","version":"0'0","reported_seq":17,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-23T10:25:47.584376+0000","last_change":"2024-04-23T10:24:44.055341+0000","last_active":"2024-04-23T10:25:47.584376+0000","last_peered":"2024-04-23T10:25:47.584376+0000","last_clean":"2024-04-23T10:25:47.584376+0000","last_became_active":"2024-04-23T10:24:44.054216+0000","last_became_peered":"2024-04-23T10:24:44.054216+0000","last_unstale":"2024-04-23T10:25:47.584376+0000","last_undegraded":"2024-04-23T10:25:47.584376+0000","last_fullsized":"2024-04-23T10:25:47.584376+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T10:24:43.025324+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T10:24:43.025324+0000","last_clean_scrub_stamp":"2024-04-23T10:24:43.025324+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T21:31:07.753283+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,0,2],"acting":[11,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"2.a","version":"87'1","reported_seq":20,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-23T10:25:47.582426+0000","last_change":"2024-04-23T10:24:43.045400+0000","last_active":"2024-04-23T10:25:47.582426+0000","last_peered":"2024-04-23T10:25:47.582426+0000","last_clean":"2024-04-23T10:25:47.582426+0000","last_became_active":"2024-04-23T10:24:43.044677+0000","last_became_peered":"2024-04-23T10:24:43.044677+0000","last_unstale":"2024-04-23T10:25:47.582426+0000","last_undegraded":"2024-04-23T10:25:47.582426+0000","last_fullsized":"2024-04-23T10:25:47.582426+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T10:24:42.023127+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T10:24:42.023127+0000","last_clean_scrub_stamp":"2024-04-23T10:24:42.023127+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T19:06:19.700132+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,8],"acting":[2,3,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.b","version":"0'0","reported_seq":17,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-23T10:25:47.586118+0000","last_change":"2024-04-23T10:24:44.049968+0000","last_active":"2024-04-23T10:25:47.586118+0000","last_peered":"2024-04-23T10:25:47.586118+0000","last_clean":"2024-04-23T10:25:47.586118+0000","last_became_active":"2024-04-23T10:24:44.049621+0000","last_became_peered":"2024-04-23T10:24:44.049621+0000","last_unstale":"2024-04-23T10:25:47.586118+0000","last_undegraded":"2024-04-23T10:25:47.586118+0000","last_fullsized":"2024-04-23T10:25:47.586118+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T10:24:43.025324+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T10:24:43.025324+0000","last_clean_scrub_stamp":"2024-04-23T10:24:43.025324+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T16:23:43.082831+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,10],"acting":[3,0,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.b","version":"87'4","reported_seq":24,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-23T10:25:47.587989+0000","last_change":"2024-04-23T10:24:43.041828+0000","last_active":"2024-04-23T10:25:47.587989+0000","last_peered":"2024-04-23T10:25:47.587989+0000","last_clean":"2024-04-23T10:25:47.587989+0000","last_became_active":"2024-04-23T10:24:43.041480+0000","last_became_peered":"2024-04-23T10:24:43.041480+0000","last_unstale":"2024-04-23T10:25:47.587989+0000","last_undegraded":"2024-04-23T10:25:47.587989+0000","last_fullsized":"2024-04-23T10:25:47.587989+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T10:24:42.023127+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T10:24:42.023127+0000","last_clean_scrub_stamp":"2024-04-23T10:24:42.023127+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T14:43:03.203971+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":2,"num_read_kb":2,"num_write":4,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,2,7],"acting":[8,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.a","version":"0'0","reported_seq":17,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-23T10:25:47.582759+0000","last_change":"2024-04-23T10:24:44.062103+0000","last_active":"2024-04-23T10:25:47.582759+0000","last_peered":"2024-04-23T10:25:47.582759+0000","last_clean":"2024-04-23T10:25:47.582759+0000","last_became_active":"2024-04-23T10:24:44.061794+0000","last_became_peered":"2024-04-23T10:24:44.061794+0000","last_unstale":"2024-04-23T10:25:47.582759+0000","last_undegraded":"2024-04-23T10:25:47.582759+0000","last_fullsized":"2024-04-23T10:25:47.582759+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T10:24:43.025324+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T10:24:43.025324+0000","last_clean_scrub_stamp":"2024-04-23T10:24:43.025324+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T10:45:30.130047+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,6],"acting":[7,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.c","version":"87'3","reported_seq":23,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-23T10:25:47.582717+0000","last_change":"2024-04-23T10:24:43.045702+0000","last_active":"2024-04-23T10:25:47.582717+0000","last_peered":"2024-04-23T10:25:47.582717+0000","last_clean":"2024-04-23T10:25:47.582717+0000","last_became_active":"2024-04-23T10:24:43.045258+0000","last_became_peered":"2024-04-23T10:24:43.045258+0000","last_unstale":"2024-04-23T10:25:47.582717+0000","last_undegraded":"2024-04-23T10:25:47.582717+0000","last_fullsized":"2024-04-23T10:25:47.582717+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T10:24:42.023127+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T10:24:42.023127+0000","last_clean_scrub_stamp":"2024-04-23T10:24:42.023127+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T12:54:30.965080+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,11,0],"acting":[2,11,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.d","version":"0'0","reported_seq":17,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-23T10:25:47.582779+0000","last_change":"2024-04-23T10:24:44.061118+0000","last_active":"2024-04-23T10:25:47.582779+0000","last_peered":"2024-04-23T10:25:47.582779+0000","last_clean":"2024-04-23T10:25:47.582779+0000","last_became_active":"2024-04-23T10:24:44.060743+0000","last_became_peered":"2024-04-23T10:24:44.060743+0000","last_unstale":"2024-04-23T10:25:47.582779+0000","last_undegraded":"2024-04-23T10:25:47.582779+0000","last_fullsized":"2024-04-23T10:25:47.582779+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T10:24:43.025324+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T10:24:43.025324+0000","last_clean_scrub_stamp":"2024-04-23T10:24:43.025324+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T17:17:46.731121+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,11],"acting":[7,1,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.d","version":"86'1","reported_seq":21,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-23T10:25:47.720362+0000","last_change":"2024-04-23T10:24:43.052519+0000","last_active":"2024-04-23T10:25:47.720362+0000","last_peered":"2024-04-23T10:25:47.720362+0000","last_clean":"2024-04-23T10:25:47.720362+0000","last_became_active":"2024-04-23T10:24:43.052187+0000","last_became_peered":"2024-04-23T10:24:43.052187+0000","last_unstale":"2024-04-23T10:25:47.720362+0000","last_undegraded":"2024-04-23T10:25:47.720362+0000","last_fullsized":"2024-04-23T10:25:47.720362+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T10:24:42.023127+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T10:24:42.023127+0000","last_clean_scrub_stamp":"2024-04-23T10:24:42.023127+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T15:51:52.839409+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,10,3],"acting":[5,10,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.c","version":"87'1","reported_seq":18,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-23T10:25:47.582550+0000","last_change":"2024-04-23T10:24:44.053086+0000","last_active":"2024-04-23T10:25:47.582550+0000","last_peered":"2024-04-23T10:25:47.582550+0000","last_clean":"2024-04-23T10:25:47.582550+0000","last_became_active":"2024-04-23T10:24:44.052687+0000","last_became_peered":"2024-04-23T10:24:44.052687+0000","last_unstale":"2024-04-23T10:25:47.582550+0000","last_undegraded":"2024-04-23T10:25:47.582550+0000","last_fullsized":"2024-04-23T10:25:47.582550+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T10:24:43.025324+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T10:24:43.025324+0000","last_clean_scrub_stamp":"2024-04-23T10:24:43.025324+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T14:35:44.958742+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":117,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,5],"acting":[7,3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.e","version":"86'2","reported_seq":31,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-23T10:25:47.724936+0000","last_change":"2024-04-23T10:25:47.724936+0000","last_active":"2024-04-23T10:25:47.724936+0000","last_peered":"2024-04-23T10:25:47.724936+0000","last_clean":"2024-04-23T10:25:47.724936+0000","last_became_active":"2024-04-23T10:25:47.722182+0000","last_became_peered":"2024-04-23T10:25:47.722182+0000","last_unstale":"2024-04-23T10:25:47.724936+0000","last_undegraded":"2024-04-23T10:25:47.724936+0000","last_fullsized":"2024-04-23T10:25:47.724936+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T10:24:42.023127+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T10:24:42.023127+0000","last_clean_scrub_stamp":"2024-04-23T10:24:42.023127+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T19:18:42.762648+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,4],"acting":[6,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.f","version":"0'0","reported_seq":17,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-23T10:25:47.582923+0000","last_change":"2024-04-23T10:24:44.049543+0000","last_active":"2024-04-23T10:25:47.582923+0000","last_peered":"2024-04-23T10:25:47.582923+0000","last_clean":"2024-04-23T10:25:47.582923+0000","last_became_active":"2024-04-23T10:24:44.049198+0000","last_became_peered":"2024-04-23T10:24:44.049198+0000","last_unstale":"2024-04-23T10:25:47.582923+0000","last_undegraded":"2024-04-23T10:25:47.582923+0000","last_fullsized":"2024-04-23T10:25:47.582923+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T10:24:43.025324+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T10:24:43.025324+0000","last_clean_scrub_stamp":"2024-04-23T10:24:43.025324+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T12:25:15.974267+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,11,8],"acting":[7,11,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.20","version":"0'0","reported_seq":17,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-23T10:25:47.584458+0000","last_change":"2024-04-23T10:24:44.054440+0000","last_active":"2024-04-23T10:25:47.584458+0000","last_peered":"2024-04-23T10:25:47.584458+0000","last_clean":"2024-04-23T10:25:47.584458+0000","last_became_active":"2024-04-23T10:24:44.053951+0000","last_became_peered":"2024-04-23T10:24:44.053951+0000","last_unstale":"2024-04-23T10:25:47.584458+0000","last_undegraded":"2024-04-23T10:25:47.584458+0000","last_fullsized":"2024-04-23T10:25:47.584458+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T10:24:43.025324+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T10:24:43.025324+0000","last_clean_scrub_stamp":"2024-04-23T10:24:43.025324+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T22:20:10.452712+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,2,5],"acting":[11,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.21","version":"0'0","reported_seq":17,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-23T10:25:47.582419+0000","last_change":"2024-04-23T10:24:44.059015+0000","last_active":"2024-04-23T10:25:47.582419+0000","last_peered":"2024-04-23T10:25:47.582419+0000","last_clean":"2024-04-23T10:25:47.582419+0000","last_became_active":"2024-04-23T10:24:44.058691+0000","last_became_peered":"2024-04-23T10:24:44.058691+0000","last_unstale":"2024-04-23T10:25:47.582419+0000","last_undegraded":"2024-04-23T10:25:47.582419+0000","last_fullsized":"2024-04-23T10:25:47.582419+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T10:24:43.025324+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T10:24:43.025324+0000","last_clean_scrub_stamp":"2024-04-23T10:24:43.025324+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T14:57:48.153094+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,4],"acting":[2,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.22","version":"0'0","reported_seq":17,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-23T10:25:47.582200+0000","last_change":"2024-04-23T10:24:44.055534+0000","last_active":"2024-04-23T10:25:47.582200+0000","last_peered":"2024-04-23T10:25:47.582200+0000","last_clean":"2024-04-23T10:25:47.582200+0000","last_became_active":"2024-04-23T10:24:44.055181+0000","last_became_peered":"2024-04-23T10:24:44.055181+0000","last_unstale":"2024-04-23T10:25:47.582200+0000","last_undegraded":"2024-04-23T10:25:47.582200+0000","last_fullsized":"2024-04-23T10:25:47.582200+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T10:24:43.025324+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T10:24:43.025324+0000","last_clean_scrub_stamp":"2024-04-23T10:24:43.025324+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T13:56:33.878232+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.23","version":"0'0","reported_seq":23,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-23T10:25:47.587380+0000","last_change":"2024-04-23T10:25:47.587380+0000","last_active":"2024-04-23T10:25:47.587380+0000","last_peered":"2024-04-23T10:25:47.587380+0000","last_clean":"2024-04-23T10:25:47.587380+0000","last_became_active":"2024-04-23T10:25:47.586893+0000","last_became_peered":"2024-04-23T10:25:47.586893+0000","last_unstale":"2024-04-23T10:25:47.587380+0000","last_undegraded":"2024-04-23T10:25:47.587380+0000","last_fullsized":"2024-04-23T10:25:47.587380+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T10:24:43.025324+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T10:24:43.025324+0000","last_clean_scrub_stamp":"2024-04-23T10:24:43.025324+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T10:51:34.390594+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,7],"acting":[0,6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.24","version":"0'0","reported_seq":17,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-23T10:25:47.584018+0000","last_change":"2024-04-23T10:24:44.050741+0000","last_active":"2024-04-23T10:25:47.584018+0000","last_peered":"2024-04-23T10:25:47.584018+0000","last_clean":"2024-04-23T10:25:47.584018+0000","last_became_active":"2024-04-23T10:24:44.050398+0000","last_became_peered":"2024-04-23T10:24:44.050398+0000","last_unstale":"2024-04-23T10:25:47.584018+0000","last_undegraded":"2024-04-23T10:25:47.584018+0000","last_fullsized":"2024-04-23T10:25:47.584018+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T10:24:43.025324+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T10:24:43.025324+0000","last_clean_scrub_stamp":"2024-04-23T10:24:43.025324+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T10:49:07.146673+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,0],"acting":[6,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.25","version":"0'0","reported_seq":17,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-23T10:25:47.585333+0000","last_change":"2024-04-23T10:24:44.048933+0000","last_active":"2024-04-23T10:25:47.585333+0000","last_peered":"2024-04-23T10:25:47.585333+0000","last_clean":"2024-04-23T10:25:47.585333+0000","last_became_active":"2024-04-23T10:24:44.048553+0000","last_became_peered":"2024-04-23T10:24:44.048553+0000","last_unstale":"2024-04-23T10:25:47.585333+0000","last_undegraded":"2024-04-23T10:25:47.585333+0000","last_fullsized":"2024-04-23T10:25:47.585333+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T10:24:43.025324+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T10:24:43.025324+0000","last_clean_scrub_stamp":"2024-04-23T10:24:43.025324+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T10:40:24.569906+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,8,5],"acting":[11,8,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.26","version":"0'0","reported_seq":17,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-23T10:25:47.582825+0000","last_change":"2024-04-23T10:24:44.051895+0000","last_active":"2024-04-23T10:25:47.582825+0000","last_peered":"2024-04-23T10:25:47.582825+0000","last_clean":"2024-04-23T10:25:47.582825+0000","last_became_active":"2024-04-23T10:24:44.051005+0000","last_became_peered":"2024-04-23T10:24:44.051005+0000","last_unstale":"2024-04-23T10:25:47.582825+0000","last_undegraded":"2024-04-23T10:25:47.582825+0000","last_fullsized":"2024-04-23T10:25:47.582825+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T10:24:43.025324+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T10:24:43.025324+0000","last_clean_scrub_stamp":"2024-04-23T10:24:43.025324+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T18:13:05.227365+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,8,1],"acting":[2,8,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.27","version":"0'0","reported_seq":23,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-23T10:25:47.587504+0000","last_change":"2024-04-23T10:25:47.587504+0000","last_active":"2024-04-23T10:25:47.587504+0000","last_peered":"2024-04-23T10:25:47.587504+0000","last_clean":"2024-04-23T10:25:47.587504+0000","last_became_active":"2024-04-23T10:25:47.587157+0000","last_became_peered":"2024-04-23T10:25:47.587157+0000","last_unstale":"2024-04-23T10:25:47.587504+0000","last_undegraded":"2024-04-23T10:25:47.587504+0000","last_fullsized":"2024-04-23T10:25:47.587504+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T10:24:43.025324+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T10:24:43.025324+0000","last_clean_scrub_stamp":"2024-04-23T10:24:43.025324+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T18:46:25.090568+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,2],"acting":[3,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.28","version":"0'0","reported_seq":17,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-23T10:25:47.720199+0000","last_change":"2024-04-23T10:24:44.044158+0000","last_active":"2024-04-23T10:25:47.720199+0000","last_peered":"2024-04-23T10:25:47.720199+0000","last_clean":"2024-04-23T10:25:47.720199+0000","last_became_active":"2024-04-23T10:24:44.043829+0000","last_became_peered":"2024-04-23T10:24:44.043829+0000","last_unstale":"2024-04-23T10:25:47.720199+0000","last_undegraded":"2024-04-23T10:25:47.720199+0000","last_fullsized":"2024-04-23T10:25:47.720199+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T10:24:43.025324+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T10:24:43.025324+0000","last_clean_scrub_stamp":"2024-04-23T10:24:43.025324+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T15:10:08.081831+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,9,3],"acting":[5,9,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.29","version":"0'0","reported_seq":17,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-23T10:25:47.585761+0000","last_change":"2024-04-23T10:24:44.051267+0000","last_active":"2024-04-23T10:25:47.585761+0000","last_peered":"2024-04-23T10:25:47.585761+0000","last_clean":"2024-04-23T10:25:47.585761+0000","last_became_active":"2024-04-23T10:24:44.050911+0000","last_became_peered":"2024-04-23T10:24:44.050911+0000","last_unstale":"2024-04-23T10:25:47.585761+0000","last_undegraded":"2024-04-23T10:25:47.585761+0000","last_fullsized":"2024-04-23T10:25:47.585761+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T10:24:43.025324+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T10:24:43.025324+0000","last_clean_scrub_stamp":"2024-04-23T10:24:43.025324+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T18:09:28.871427+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,11],"acting":[3,2,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.2a","version":"0'0","reported_seq":17,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-23T10:25:47.720038+0000","last_change":"2024-04-23T10:24:44.057653+0000","last_active":"2024-04-23T10:25:47.720038+0000","last_peered":"2024-04-23T10:25:47.720038+0000","last_clean":"2024-04-23T10:25:47.720038+0000","last_became_active":"2024-04-23T10:24:44.056627+0000","last_became_peered":"2024-04-23T10:24:44.056627+0000","last_unstale":"2024-04-23T10:25:47.720038+0000","last_undegraded":"2024-04-23T10:25:47.720038+0000","last_fullsized":"2024-04-23T10:25:47.720038+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T10:24:43.025324+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T10:24:43.025324+0000","last_clean_scrub_stamp":"2024-04-23T10:24:43.025324+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T18:29:21.480187+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,3],"acting":[1,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.2b","version":"0'0","reported_seq":17,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-23T10:25:47.584646+0000","last_change":"2024-04-23T10:24:44.059735+0000","last_active":"2024-04-23T10:25:47.584646+0000","last_peered":"2024-04-23T10:25:47.584646+0000","last_clean":"2024-04-23T10:25:47.584646+0000","last_became_active":"2024-04-23T10:24:44.059052+0000","last_became_peered":"2024-04-23T10:24:44.059052+0000","last_unstale":"2024-04-23T10:25:47.584646+0000","last_undegraded":"2024-04-23T10:25:47.584646+0000","last_fullsized":"2024-04-23T10:25:47.584646+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T10:24:43.025324+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T10:24:43.025324+0000","last_clean_scrub_stamp":"2024-04-23T10:24:43.025324+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T14:29:25.794655+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,0],"acting":[6,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.2c","version":"0'0","reported_seq":17,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-23T10:25:47.584627+0000","last_change":"2024-04-23T10:24:44.051831+0000","last_active":"2024-04-23T10:25:47.584627+0000","last_peered":"2024-04-23T10:25:47.584627+0000","last_clean":"2024-04-23T10:25:47.584627+0000","last_became_active":"2024-04-23T10:24:44.051480+0000","last_became_peered":"2024-04-23T10:24:44.051480+0000","last_unstale":"2024-04-23T10:25:47.584627+0000","last_undegraded":"2024-04-23T10:25:47.584627+0000","last_fullsized":"2024-04-23T10:25:47.584627+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T10:24:43.025324+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T10:24:43.025324+0000","last_clean_scrub_stamp":"2024-04-23T10:24:43.025324+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T19:20:55.181393+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,4,8],"acting":[11,4,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]}],"pool_stats":[{"poolid":3,"num_pg":64,"stat_sum":{"num_bytes":234,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":702,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":2,"ondisk_log_size":2,"up":192,"acting":192,"num_store_stats":12},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":136,"num_read_kb":119,"num_write":181,"num_write_kb":2938,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":1180736,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":4751360,"data_stored":4722944,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":137,"ondisk_log_size":137,"up":3,"acting":3,"num_store_stats":8},{"poolid":2,"num_pg":32,"stat_sum":{"num_bytes":65438,"num_objects":60,"num_object_clones":0,"num_object_copies":180,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":60,"num_whiteouts":0,"num_read":446,"num_read_kb":291,"num_write":157,"num_write_kb":109,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":10,"num_objects_omap":40,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":417792,"data_stored":196314,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":136,"ondisk_log_size":136,"up":96,"acting":96,"num_store_stats":12}],"osd_stats":[{"osd":11,"up_from":82,"seq":352187318301,"num_pgs":19,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32132,"kb_used_data":952,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93713020,"statfs":{"total":95995035648,"available":95962132480,"internally_reserved":0,"allocated":974848,"data_stored":465606,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1596,"internal_metadata":31914436},"hb_peers":[0,1,2,4,5,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":10,"up_from":74,"seq":317827579937,"num_pgs":26,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28656,"kb_used_data":1632,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93716496,"statfs":{"total":95995035648,"available":95965691904,"internally_reserved":0,"allocated":1671168,"data_stored":1118524,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1600,"internal_metadata":27654592},"hb_peers":[0,1,2,3,4,5,6,8,9,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":9,"up_from":69,"seq":296352743463,"num_pgs":20,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":27988,"kb_used_data":968,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717164,"statfs":{"total":95995035648,"available":95966375936,"internally_reserved":0,"allocated":991232,"data_stored":465842,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1600,"internal_metadata":27654592},"hb_peers":[0,1,2,3,4,5,6,7,8,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":8,"up_from":62,"seq":266287972397,"num_pgs":25,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32152,"kb_used_data":968,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93713000,"statfs":{"total":95995035648,"available":95962112000,"internally_reserved":0,"allocated":991232,"data_stored":465762,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,2,3,4,5,6,7,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":7,"up_from":53,"seq":227633266737,"num_pgs":26,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":36888,"kb_used_data":1548,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":93708264,"statfs":{"total":95995035648,"available":95957262336,"internally_reserved":0,"allocated":1585152,"data_stored":1058083,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":36174282},"hb_peers":[1,2,3,4,5,6,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":6,"up_from":47,"seq":201863462967,"num_pgs":22,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32140,"kb_used_data":956,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93713012,"statfs":{"total":95995035648,"available":95962124288,"internally_reserved":0,"allocated":978944,"data_stored":465231,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":31914438},"hb_peers":[0,1,2,3,4,5,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":13,"seq":55834574926,"num_pgs":24,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28036,"kb_used_data":1016,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717116,"statfs":{"total":95995035648,"available":95966326784,"internally_reserved":0,"allocated":1040384,"data_stored":526070,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,2,3,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":8,"seq":34359738452,"num_pgs":25,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32160,"kb_used_data":976,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93712992,"statfs":{"total":95995035648,"available":95962103808,"internally_reserved":0,"allocated":999424,"data_stored":466502,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":31914438},"hb_peers":[1,2,3,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":18,"seq":77309411402,"num_pgs":27,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":36384,"kb_used_data":1040,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":93708768,"statfs":{"total":95995035648,"available":95957778432,"internally_reserved":0,"allocated":1064960,"data_stored":527811,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":36174277},"hb_peers":[0,1,3,4,5,6,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":25,"seq":107374182467,"num_pgs":26,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":27996,"kb_used_data":972,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717156,"statfs":{"total":95995035648,"available":95966367744,"internally_reserved":0,"allocated":995328,"data_stored":465542,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[0,1,2,4,5,6,7,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":4,"up_from":33,"seq":141733920831,"num_pgs":23,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":27980,"kb_used_data":960,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717172,"statfs":{"total":95995035648,"available":95966384128,"internally_reserved":0,"allocated":983040,"data_stored":466059,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[1,2,3,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":39,"seq":167503724604,"num_pgs":28,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":36904,"kb_used_data":1560,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":93708248,"statfs":{"total":95995035648,"available":95957245952,"internally_reserved":0,"allocated":1597440,"data_stored":1057376,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":36174277},"hb_peers":[0,1,2,3,4,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":28672,"data_stored":1478,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":69632,"data_stored":61046,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":94208,"data_stored":62787,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":284,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":1035,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":28672,"data_stored":1867,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":90,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":2574,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":738,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":818,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":102400,"data_stored":63015,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":582,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":234,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":117,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":117,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":117,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":117,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-04-23T10:26:01.480 INFO:tasks.cephadm.ceph_manager.ceph:clean! 2024-04-23T10:26:01.481 INFO:tasks.ceph:Waiting until ceph cluster ceph is healthy... 2024-04-23T10:26:01.481 INFO:tasks.cephadm.ceph_manager.ceph:wait_until_healthy 2024-04-23T10:26:01.481 DEBUG:teuthology.orchestra.run.smithi114:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d69f13d827952bb07bb9b7973bca14f97d764d6f shell --fsid aa240572-0159-11ef-bc93-c7b262605968 -- ceph health --format=json 2024-04-23T10:26:01.492 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 23 10:26:01 smithi119 ceph-mon[32608]: pgmap v326: 97 pgs: 97 active+clean; 641 KiB data, 371 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s; 0 B/s, 0 keys/s, 0 objects/s recovering 2024-04-23T10:26:01.721 INFO:teuthology.orchestra.run.smithi114.stderr:Inferring config /var/lib/ceph/aa240572-0159-11ef-bc93-c7b262605968/mon.a/config 2024-04-23T10:26:02.144 INFO:journalctl@ceph.mon.a.smithi114.stdout:Apr 23 10:26:02 smithi114 ceph-mon[29255]: from='client.14976 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-04-23T10:26:02.328 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 23 10:26:02 smithi204 ceph-mon[32649]: from='client.14976 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-04-23T10:26:02.492 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 23 10:26:02 smithi119 ceph-mon[32608]: from='client.14976 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-04-23T10:26:03.217 INFO:journalctl@ceph.mon.a.smithi114.stdout:Apr 23 10:26:03 smithi114 ceph-mon[29255]: pgmap v327: 97 pgs: 97 active+clean; 641 KiB data, 371 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s; 0 B/s, 0 keys/s, 0 objects/s recovering 2024-04-23T10:26:03.328 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 23 10:26:03 smithi204 ceph-mon[32649]: pgmap v327: 97 pgs: 97 active+clean; 641 KiB data, 371 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s; 0 B/s, 0 keys/s, 0 objects/s recovering 2024-04-23T10:26:03.461 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:03.460+0000 7fe827c83700 1 -- 172.21.15.114:0/1730133517 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fe820106420 msgr2=0x7fe82010dff0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T10:26:03.461 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:03.460+0000 7fe827c83700 1 --2- 172.21.15.114:0/1730133517 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fe820106420 0x7fe82010dff0 secure :-1 s=READY pgs=302 cs=0 l=1 rev1=1 crypto rx=0x7fe81800b0d0 tx=0x7fe81800b3e0 comp rx=0 tx=0).stop 2024-04-23T10:26:03.461 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:03.460+0000 7fe827c83700 1 -- 172.21.15.114:0/1730133517 shutdown_connections 2024-04-23T10:26:03.461 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:03.460+0000 7fe827c83700 1 --2- 172.21.15.114:0/1730133517 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fe820101340 0x7fe820101720 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:26:03.461 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:03.460+0000 7fe827c83700 1 --2- 172.21.15.114:0/1730133517 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fe820106420 0x7fe82010dff0 unknown :-1 s=CLOSED pgs=302 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:26:03.462 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:03.460+0000 7fe827c83700 1 --2- 172.21.15.114:0/1730133517 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fe820101cf0 0x7fe820105d40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:26:03.462 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:03.460+0000 7fe827c83700 1 -- 172.21.15.114:0/1730133517 >> 172.21.15.114:0/1730133517 conn(0x7fe820077f40 msgr2=0x7fe8200ff2c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T10:26:03.462 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:03.460+0000 7fe827c83700 1 -- 172.21.15.114:0/1730133517 shutdown_connections 2024-04-23T10:26:03.462 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:03.460+0000 7fe827c83700 1 -- 172.21.15.114:0/1730133517 wait complete. 2024-04-23T10:26:03.462 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:03.460+0000 7fe827c83700 1 Processor -- start 2024-04-23T10:26:03.462 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:03.460+0000 7fe827c83700 1 -- start start 2024-04-23T10:26:03.462 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:03.460+0000 7fe827c83700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fe820101340 0x7fe8201991a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T10:26:03.462 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:03.460+0000 7fe827c83700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fe820101cf0 0x7fe8201996e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T10:26:03.462 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:03.460+0000 7fe827c83700 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fe820106420 0x7fe82019da20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T10:26:03.462 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:03.461+0000 7fe827c83700 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x7fe820110aa0 con 0x7fe820101340 2024-04-23T10:26:03.462 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:03.461+0000 7fe827c83700 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 -- 0x7fe820110920 con 0x7fe820101cf0 2024-04-23T10:26:03.462 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:03.461+0000 7fe827c83700 1 -- --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_getmap magic: 0 -- 0x7fe820110c20 con 0x7fe820106420 2024-04-23T10:26:03.462 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:03.461+0000 7fe825a1f700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fe820101340 0x7fe8201991a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T10:26:03.463 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:03.461+0000 7fe825a1f700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fe820101340 0x7fe8201991a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.114:34154/0 (socket says 172.21.15.114:34154) 2024-04-23T10:26:03.463 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:03.461+0000 7fe825a1f700 1 -- 172.21.15.114:0/4288863675 learned_addr learned my addr 172.21.15.114:0/4288863675 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-04-23T10:26:03.463 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:03.461+0000 7fe82521e700 1 --2- 172.21.15.114:0/4288863675 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fe820101cf0 0x7fe8201996e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T10:26:03.463 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:03.461+0000 7fe826220700 1 --2- 172.21.15.114:0/4288863675 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fe820106420 0x7fe82019da20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T10:26:03.463 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:03.461+0000 7fe825a1f700 1 -- 172.21.15.114:0/4288863675 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fe820106420 msgr2=0x7fe82019da20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T10:26:03.463 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:03.461+0000 7fe825a1f700 1 --2- 172.21.15.114:0/4288863675 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fe820106420 0x7fe82019da20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:26:03.463 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:03.461+0000 7fe825a1f700 1 -- 172.21.15.114:0/4288863675 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fe820101cf0 msgr2=0x7fe8201996e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T10:26:03.463 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:03.461+0000 7fe825a1f700 1 --2- 172.21.15.114:0/4288863675 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fe820101cf0 0x7fe8201996e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:26:03.463 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:03.461+0000 7fe825a1f700 1 -- 172.21.15.114:0/4288863675 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe82019e160 con 0x7fe820101340 2024-04-23T10:26:03.463 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:03.461+0000 7fe825a1f700 1 --2- 172.21.15.114:0/4288863675 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fe820101340 0x7fe8201991a0 secure :-1 s=READY pgs=303 cs=0 l=1 rev1=1 crypto rx=0x7fe81c00ddc0 tx=0x7fe81c00a530 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T10:26:03.463 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:03.461+0000 7fe816ffd700 1 -- 172.21.15.114:0/4288863675 <== mon.0 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe81c01b070 con 0x7fe820101340 2024-04-23T10:26:03.463 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:03.461+0000 7fe816ffd700 1 -- 172.21.15.114:0/4288863675 <== mon.0 v2:172.21.15.114:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fe81c00ac90 con 0x7fe820101340 2024-04-23T10:26:03.464 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:03.461+0000 7fe827c83700 1 -- 172.21.15.114:0/4288863675 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fe82019e450 con 0x7fe820101340 2024-04-23T10:26:03.464 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:03.461+0000 7fe816ffd700 1 -- 172.21.15.114:0/4288863675 <== mon.0 v2:172.21.15.114:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe81c0166a0 con 0x7fe820101340 2024-04-23T10:26:03.464 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:03.461+0000 7fe827c83700 1 -- 172.21.15.114:0/4288863675 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fe8201a5cd0 con 0x7fe820101340 2024-04-23T10:26:03.464 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:03.462+0000 7fe827c83700 1 -- 172.21.15.114:0/4288863675 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fe820069730 con 0x7fe820101340 2024-04-23T10:26:03.464 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:03.464+0000 7fe816ffd700 1 -- 172.21.15.114:0/4288863675 <== mon.0 v2:172.21.15.114:3300/0 4 ==== mgrmap(e 17) ==== 138698+0+0 (secure 0 0 0) 0x7fe81c01e070 con 0x7fe820101340 2024-04-23T10:26:03.465 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:03.465+0000 7fe816ffd700 1 --2- 172.21.15.114:0/4288863675 >> [v2:172.21.15.114:6800/465333808,v1:172.21.15.114:6801/465333808] conn(0x7fe80c0a3cd0 0x7fe80c0a6190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T10:26:03.465 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:03.465+0000 7fe82521e700 1 --2- 172.21.15.114:0/4288863675 >> [v2:172.21.15.114:6800/465333808,v1:172.21.15.114:6801/465333808] conn(0x7fe80c0a3cd0 0x7fe80c0a6190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T10:26:03.465 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:03.465+0000 7fe816ffd700 1 -- 172.21.15.114:0/4288863675 <== mon.0 v2:172.21.15.114:3300/0 5 ==== osd_map(89..89 src has 1..89) ==== 8269+0+0 (secure 0 0 0) 0x7fe81c0cffc0 con 0x7fe820101340 2024-04-23T10:26:03.465 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:03.465+0000 7fe82521e700 1 --2- 172.21.15.114:0/4288863675 >> [v2:172.21.15.114:6800/465333808,v1:172.21.15.114:6801/465333808] conn(0x7fe80c0a3cd0 0x7fe80c0a6190 secure :-1 s=READY pgs=210 cs=0 l=1 rev1=1 crypto rx=0x7fe820104050 tx=0x7fe810009190 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T10:26:03.467 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:03.469+0000 7fe816ffd700 1 -- 172.21.15.114:0/4288863675 <== mon.0 v2:172.21.15.114:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7fe81c09c930 con 0x7fe820101340 2024-04-23T10:26:03.491 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 23 10:26:03 smithi119 ceph-mon[32608]: pgmap v327: 97 pgs: 97 active+clean; 641 KiB data, 371 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s; 0 B/s, 0 keys/s, 0 objects/s recovering 2024-04-23T10:26:03.710 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:03.711+0000 7fe827c83700 1 -- 172.21.15.114:0/4288863675 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_command({"prefix": "health", "format": "json"} v 0) -- 0x7fe82019a4e0 con 0x7fe820101340 2024-04-23T10:26:03.710 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:03.711+0000 7fe816ffd700 1 -- 172.21.15.114:0/4288863675 <== mon.0 v2:172.21.15.114:3300/0 7 ==== mon_command_ack([{"prefix": "health", "format": "json"}]=0 v0) ==== 72+0+46 (secure 0 0 0) 0x7fe82019a4e0 con 0x7fe820101340 2024-04-23T10:26:03.710 INFO:teuthology.orchestra.run.smithi114.stdout: 2024-04-23T10:26:03.710 INFO:teuthology.orchestra.run.smithi114.stdout:{"status":"HEALTH_OK","checks":{},"mutes":[]} 2024-04-23T10:26:03.712 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:03.713+0000 7fe827c83700 1 -- 172.21.15.114:0/4288863675 >> [v2:172.21.15.114:6800/465333808,v1:172.21.15.114:6801/465333808] conn(0x7fe80c0a3cd0 msgr2=0x7fe80c0a6190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T10:26:03.712 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:03.713+0000 7fe827c83700 1 --2- 172.21.15.114:0/4288863675 >> [v2:172.21.15.114:6800/465333808,v1:172.21.15.114:6801/465333808] conn(0x7fe80c0a3cd0 0x7fe80c0a6190 secure :-1 s=READY pgs=210 cs=0 l=1 rev1=1 crypto rx=0x7fe820104050 tx=0x7fe810009190 comp rx=0 tx=0).stop 2024-04-23T10:26:03.712 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:03.713+0000 7fe827c83700 1 -- 172.21.15.114:0/4288863675 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fe820101340 msgr2=0x7fe8201991a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T10:26:03.712 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:03.713+0000 7fe827c83700 1 --2- 172.21.15.114:0/4288863675 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fe820101340 0x7fe8201991a0 secure :-1 s=READY pgs=303 cs=0 l=1 rev1=1 crypto rx=0x7fe81c00ddc0 tx=0x7fe81c00a530 comp rx=0 tx=0).stop 2024-04-23T10:26:03.713 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:03.713+0000 7fe827c83700 1 -- 172.21.15.114:0/4288863675 shutdown_connections 2024-04-23T10:26:03.713 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:03.713+0000 7fe827c83700 1 --2- 172.21.15.114:0/4288863675 >> [v2:172.21.15.114:6800/465333808,v1:172.21.15.114:6801/465333808] conn(0x7fe80c0a3cd0 0x7fe80c0a6190 unknown :-1 s=CLOSED pgs=210 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:26:03.713 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:03.713+0000 7fe827c83700 1 --2- 172.21.15.114:0/4288863675 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fe820106420 0x7fe82019da20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:26:03.713 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:03.713+0000 7fe827c83700 1 --2- 172.21.15.114:0/4288863675 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fe820101340 0x7fe8201991a0 unknown :-1 s=CLOSED pgs=303 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:26:03.713 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:03.713+0000 7fe827c83700 1 --2- 172.21.15.114:0/4288863675 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fe820101cf0 0x7fe8201996e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:26:03.713 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:03.713+0000 7fe827c83700 1 -- 172.21.15.114:0/4288863675 >> 172.21.15.114:0/4288863675 conn(0x7fe820077f40 msgr2=0x7fe8200ff2c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T10:26:03.713 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:03.713+0000 7fe827c83700 1 -- 172.21.15.114:0/4288863675 shutdown_connections 2024-04-23T10:26:03.713 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:03.713+0000 7fe827c83700 1 -- 172.21.15.114:0/4288863675 wait complete. 2024-04-23T10:26:04.315 INFO:journalctl@ceph.mon.a.smithi114.stdout:Apr 23 10:26:04 smithi114 ceph-mon[29255]: from='client.? 172.21.15.114:0/4288863675' entity='client.admin' cmd=[{"prefix": "health", "format": "json"}]: dispatch 2024-04-23T10:26:04.328 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 23 10:26:04 smithi204 ceph-mon[32649]: from='client.? 172.21.15.114:0/4288863675' entity='client.admin' cmd=[{"prefix": "health", "format": "json"}]: dispatch 2024-04-23T10:26:04.342 INFO:tasks.cephadm.ceph_manager.ceph:wait_until_healthy done 2024-04-23T10:26:04.343 DEBUG:teuthology.orchestra.run.smithi114:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-04-23T10:26:04.446 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:04.445+0000 7f33af7cd640 1 -- 172.21.15.114:0/4134332193 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f33a8100e90 msgr2=0x7f33a81012b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T10:26:04.447 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:04.445+0000 7f33af7cd640 1 --2- 172.21.15.114:0/4134332193 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f33a8100e90 0x7f33a81012b0 secure :-1 s=READY pgs=304 cs=0 l=1 rev1=1 crypto rx=0x7f3398009900 tx=0x7f339802a410 comp rx=0 tx=0).stop 2024-04-23T10:26:04.447 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:04.445+0000 7f33af7cd640 1 -- 172.21.15.114:0/4134332193 shutdown_connections 2024-04-23T10:26:04.447 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:04.445+0000 7f33af7cd640 1 --2- 172.21.15.114:0/4134332193 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f33a81029a0 0x7f33a81092c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:26:04.447 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:04.445+0000 7f33af7cd640 1 --2- 172.21.15.114:0/4134332193 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f33a8101fe0 0x7f33a8102460 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:26:04.447 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:04.445+0000 7f33af7cd640 1 --2- 172.21.15.114:0/4134332193 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f33a8100e90 0x7f33a81012b0 unknown :-1 s=CLOSED pgs=304 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:26:04.447 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:04.445+0000 7f33af7cd640 1 -- 172.21.15.114:0/4134332193 >> 172.21.15.114:0/4134332193 conn(0x7f33a80fc400 msgr2=0x7f33a80fe840 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T10:26:04.447 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:04.445+0000 7f33af7cd640 1 -- 172.21.15.114:0/4134332193 shutdown_connections 2024-04-23T10:26:04.447 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:04.445+0000 7f33af7cd640 1 -- 172.21.15.114:0/4134332193 wait complete. 2024-04-23T10:26:04.447 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:04.445+0000 7f33af7cd640 1 Processor -- start 2024-04-23T10:26:04.447 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:04.445+0000 7f33af7cd640 1 -- start start 2024-04-23T10:26:04.448 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:04.446+0000 7f33af7cd640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f33a8100e90 0x7f33a8069c70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T10:26:04.448 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:04.446+0000 7f33af7cd640 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f33a8101fe0 0x7f33a819cfa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T10:26:04.448 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:04.446+0000 7f33af7cd640 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f33a81029a0 0x7f33a819f5e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T10:26:04.448 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:04.446+0000 7f33af7cd640 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x7f33a810ca40 con 0x7f33a8100e90 2024-04-23T10:26:04.448 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:04.446+0000 7f33af7cd640 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 -- 0x7f33a810c8c0 con 0x7f33a8101fe0 2024-04-23T10:26:04.448 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:04.446+0000 7f33af7cd640 1 -- --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_getmap magic: 0 -- 0x7f33a810cbc0 con 0x7f33a81029a0 2024-04-23T10:26:04.448 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:04.446+0000 7f33ad569640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f33a8100e90 0x7f33a8069c70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T10:26:04.448 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:04.446+0000 7f33ad569640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f33a8100e90 0x7f33a8069c70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.114:34170/0 (socket says 172.21.15.114:34170) 2024-04-23T10:26:04.448 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:04.446+0000 7f33ad569640 1 -- 172.21.15.114:0/1520892417 learned_addr learned my addr 172.21.15.114:0/1520892417 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-04-23T10:26:04.448 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:04.446+0000 7f33add6a640 1 --2- 172.21.15.114:0/1520892417 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f33a81029a0 0x7f33a819f5e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T10:26:04.448 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:04.446+0000 7f33ad569640 1 -- 172.21.15.114:0/1520892417 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f33a81029a0 msgr2=0x7f33a819f5e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T10:26:04.448 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:04.446+0000 7f33ad569640 1 --2- 172.21.15.114:0/1520892417 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f33a81029a0 0x7f33a819f5e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:26:04.449 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:04.446+0000 7f33ad569640 1 -- 172.21.15.114:0/1520892417 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f33a8101fe0 msgr2=0x7f33a819cfa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T10:26:04.449 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:04.446+0000 7f33ad569640 1 --2- 172.21.15.114:0/1520892417 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f33a8101fe0 0x7f33a819cfa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:26:04.449 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:04.446+0000 7f33ad569640 1 -- 172.21.15.114:0/1520892417 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f33a819fea0 con 0x7f33a8100e90 2024-04-23T10:26:04.449 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:04.446+0000 7f33ad569640 1 --2- 172.21.15.114:0/1520892417 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f33a8100e90 0x7f33a8069c70 secure :-1 s=READY pgs=305 cs=0 l=1 rev1=1 crypto rx=0x7f3398005940 tx=0x7f3398004430 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T10:26:04.449 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:04.446+0000 7f339e7fc640 1 -- 172.21.15.114:0/1520892417 <== mon.0 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f339803f030 con 0x7f33a8100e90 2024-04-23T10:26:04.449 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:04.446+0000 7f339e7fc640 1 -- 172.21.15.114:0/1520892417 <== mon.0 v2:172.21.15.114:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f3398003cb0 con 0x7f33a8100e90 2024-04-23T10:26:04.449 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:04.446+0000 7f33af7cd640 1 -- 172.21.15.114:0/1520892417 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f33a81a0130 con 0x7f33a8100e90 2024-04-23T10:26:04.449 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:04.446+0000 7f339e7fc640 1 -- 172.21.15.114:0/1520892417 <== mon.0 v2:172.21.15.114:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3398035070 con 0x7f33a8100e90 2024-04-23T10:26:04.449 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:04.446+0000 7f33af7cd640 1 -- 172.21.15.114:0/1520892417 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f33a81a06a0 con 0x7f33a8100e90 2024-04-23T10:26:04.449 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:04.447+0000 7f33af7cd640 1 -- 172.21.15.114:0/1520892417 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f33a81081c0 con 0x7f33a8100e90 2024-04-23T10:26:04.450 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:04.448+0000 7f339e7fc640 1 -- 172.21.15.114:0/1520892417 <== mon.0 v2:172.21.15.114:3300/0 4 ==== mgrmap(e 17) ==== 138698+0+0 (secure 0 0 0) 0x7f3398042030 con 0x7f33a8100e90 2024-04-23T10:26:04.450 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:04.449+0000 7f339e7fc640 1 --2- 172.21.15.114:0/1520892417 >> [v2:172.21.15.114:6800/465333808,v1:172.21.15.114:6801/465333808] conn(0x7f33940a3c40 0x7f33940a6100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T10:26:04.450 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:04.449+0000 7f33acd68640 1 --2- 172.21.15.114:0/1520892417 >> [v2:172.21.15.114:6800/465333808,v1:172.21.15.114:6801/465333808] conn(0x7f33940a3c40 0x7f33940a6100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T10:26:04.450 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:04.449+0000 7f339e7fc640 1 -- 172.21.15.114:0/1520892417 <== mon.0 v2:172.21.15.114:3300/0 5 ==== osd_map(89..89 src has 1..89) ==== 8269+0+0 (secure 0 0 0) 0x7f33980f0fb0 con 0x7f33a8100e90 2024-04-23T10:26:04.450 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:04.449+0000 7f33acd68640 1 --2- 172.21.15.114:0/1520892417 >> [v2:172.21.15.114:6800/465333808,v1:172.21.15.114:6801/465333808] conn(0x7f33940a3c40 0x7f33940a6100 secure :-1 s=READY pgs=211 cs=0 l=1 rev1=1 crypto rx=0x7f33a4005fd0 tx=0x7f33a400b410 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T10:26:04.452 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:04.453+0000 7f339e7fc640 1 -- 172.21.15.114:0/1520892417 <== mon.0 v2:172.21.15.114:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f3398003e60 con 0x7f33a8100e90 2024-04-23T10:26:04.492 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 23 10:26:04 smithi119 ceph-mon[32608]: from='client.? 172.21.15.114:0/4288863675' entity='client.admin' cmd=[{"prefix": "health", "format": "json"}]: dispatch 2024-04-23T10:26:04.631 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:04.631+0000 7f33af7cd640 1 -- 172.21.15.114:0/1520892417 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f33a802d830 con 0x7f33a8100e90 2024-04-23T10:26:04.632 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:04.633+0000 7f339e7fc640 1 -- 172.21.15.114:0/1520892417 <== mon.0 v2:172.21.15.114:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v89) ==== 74+0+22818 (secure 0 0 0) 0x7f33980bd810 con 0x7f33a8100e90 2024-04-23T10:26:04.632 INFO:teuthology.orchestra.run.smithi114.stdout: 2024-04-23T10:26:04.632 INFO:teuthology.orchestra.run.smithi114.stdout:{"epoch":89,"fsid":"aa240572-0159-11ef-bc93-c7b262605968","created":"2024-04-23T10:16:53.351136+0000","modified":"2024-04-23T10:25:47.573901+0000","last_up_change":"2024-04-23T10:23:54.153269+0000","last_in_change":"2024-04-23T10:23:32.963085+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":26,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-04-23T10:20:11.970297+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":12,"score_stable":12,"optimal_score":0.25,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-04-23T10:24:41.218540+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"86","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.8799999952316284,"score_stable":1.8799999952316284,"optimal_score":1,"raw_score_acting":1.8799999952316284,"raw_score_stable":1.8799999952316284,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-04-23T10:24:42.341799+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"87","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"fd64adde-9445-45a4-a171-ef656e2c5709","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":88,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6802","nonce":736366878},{"type":"v1","addr":"172.21.15.114:6803","nonce":736366878}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6804","nonce":736366878},{"type":"v1","addr":"172.21.15.114:6805","nonce":736366878}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6808","nonce":736366878},{"type":"v1","addr":"172.21.15.114:6809","nonce":736366878}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6806","nonce":736366878},{"type":"v1","addr":"172.21.15.114:6807","nonce":736366878}]},"public_addr":"172.21.15.114:6803/736366878","cluster_addr":"172.21.15.114:6805/736366878","heartbeat_back_addr":"172.21.15.114:6809/736366878","heartbeat_front_addr":"172.21.15.114:6807/736366878","state":["exists","up"]},{"osd":1,"uuid":"e279a1e6-2702-472a-930a-e08ece76a861","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":85,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6800","nonce":1133540779},{"type":"v1","addr":"172.21.15.119:6801","nonce":1133540779}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6802","nonce":1133540779},{"type":"v1","addr":"172.21.15.119:6803","nonce":1133540779}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6806","nonce":1133540779},{"type":"v1","addr":"172.21.15.119:6807","nonce":1133540779}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6804","nonce":1133540779},{"type":"v1","addr":"172.21.15.119:6805","nonce":1133540779}]},"public_addr":"172.21.15.119:6801/1133540779","cluster_addr":"172.21.15.119:6803/1133540779","heartbeat_back_addr":"172.21.15.119:6807/1133540779","heartbeat_front_addr":"172.21.15.119:6805/1133540779","state":["exists","up"]},{"osd":2,"uuid":"f4c9f2b4-ff49-4e14-89cd-b477feeb9382","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":85,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6800","nonce":799652511},{"type":"v1","addr":"172.21.15.204:6801","nonce":799652511}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6802","nonce":799652511},{"type":"v1","addr":"172.21.15.204:6803","nonce":799652511}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6806","nonce":799652511},{"type":"v1","addr":"172.21.15.204:6807","nonce":799652511}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6804","nonce":799652511},{"type":"v1","addr":"172.21.15.204:6805","nonce":799652511}]},"public_addr":"172.21.15.204:6801/799652511","cluster_addr":"172.21.15.204:6803/799652511","heartbeat_back_addr":"172.21.15.204:6807/799652511","heartbeat_front_addr":"172.21.15.204:6805/799652511","state":["exists","up"]},{"osd":3,"uuid":"52d2a566-e71c-4bb3-8d6c-5abdd43177c2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":25,"up_thru":88,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6810","nonce":4275595091},{"type":"v1","addr":"172.21.15.114:6811","nonce":4275595091}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6812","nonce":4275595091},{"type":"v1","addr":"172.21.15.114:6813","nonce":4275595091}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6816","nonce":4275595091},{"type":"v1","addr":"172.21.15.114:6817","nonce":4275595091}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6814","nonce":4275595091},{"type":"v1","addr":"172.21.15.114:6815","nonce":4275595091}]},"public_addr":"172.21.15.114:6811/4275595091","cluster_addr":"172.21.15.114:6813/4275595091","heartbeat_back_addr":"172.21.15.114:6817/4275595091","heartbeat_front_addr":"172.21.15.114:6815/4275595091","state":["exists","up"]},{"osd":4,"uuid":"d911a7c1-4bdc-4d28-b8a6-73896a6e7de6","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":33,"up_thru":85,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6808","nonce":1611007172},{"type":"v1","addr":"172.21.15.119:6809","nonce":1611007172}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6810","nonce":1611007172},{"type":"v1","addr":"172.21.15.119:6811","nonce":1611007172}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6814","nonce":1611007172},{"type":"v1","addr":"172.21.15.119:6815","nonce":1611007172}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6812","nonce":1611007172},{"type":"v1","addr":"172.21.15.119:6813","nonce":1611007172}]},"public_addr":"172.21.15.119:6809/1611007172","cluster_addr":"172.21.15.119:6811/1611007172","heartbeat_back_addr":"172.21.15.119:6815/1611007172","heartbeat_front_addr":"172.21.15.119:6813/1611007172","state":["exists","up"]},{"osd":5,"uuid":"f7590e1a-e93c-4c29-b15f-38ee52d1f921","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":39,"up_thru":85,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6808","nonce":1427563312},{"type":"v1","addr":"172.21.15.204:6809","nonce":1427563312}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6810","nonce":1427563312},{"type":"v1","addr":"172.21.15.204:6811","nonce":1427563312}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6814","nonce":1427563312},{"type":"v1","addr":"172.21.15.204:6815","nonce":1427563312}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6812","nonce":1427563312},{"type":"v1","addr":"172.21.15.204:6813","nonce":1427563312}]},"public_addr":"172.21.15.204:6809/1427563312","cluster_addr":"172.21.15.204:6811/1427563312","heartbeat_back_addr":"172.21.15.204:6815/1427563312","heartbeat_front_addr":"172.21.15.204:6813/1427563312","state":["exists","up"]},{"osd":6,"uuid":"534d20e8-3e54-4db8-a845-6119f49c7a78","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":47,"up_thru":88,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6818","nonce":4072912544},{"type":"v1","addr":"172.21.15.114:6819","nonce":4072912544}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6820","nonce":4072912544},{"type":"v1","addr":"172.21.15.114:6821","nonce":4072912544}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6824","nonce":4072912544},{"type":"v1","addr":"172.21.15.114:6825","nonce":4072912544}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6822","nonce":4072912544},{"type":"v1","addr":"172.21.15.114:6823","nonce":4072912544}]},"public_addr":"172.21.15.114:6819/4072912544","cluster_addr":"172.21.15.114:6821/4072912544","heartbeat_back_addr":"172.21.15.114:6825/4072912544","heartbeat_front_addr":"172.21.15.114:6823/4072912544","state":["exists","up"]},{"osd":7,"uuid":"f0ec22b8-1b48-4859-9d65-b00403aff3ad","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":53,"up_thru":88,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6816","nonce":2237458793},{"type":"v1","addr":"172.21.15.119:6817","nonce":2237458793}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6818","nonce":2237458793},{"type":"v1","addr":"172.21.15.119:6819","nonce":2237458793}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6822","nonce":2237458793},{"type":"v1","addr":"172.21.15.119:6823","nonce":2237458793}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6820","nonce":2237458793},{"type":"v1","addr":"172.21.15.119:6821","nonce":2237458793}]},"public_addr":"172.21.15.119:6817/2237458793","cluster_addr":"172.21.15.119:6819/2237458793","heartbeat_back_addr":"172.21.15.119:6823/2237458793","heartbeat_front_addr":"172.21.15.119:6821/2237458793","state":["exists","up"]},{"osd":8,"uuid":"3ac89686-6d76-4cef-9089-ac29e4b3bcac","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":62,"up_thru":85,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6816","nonce":1426318231},{"type":"v1","addr":"172.21.15.204:6817","nonce":1426318231}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6818","nonce":1426318231},{"type":"v1","addr":"172.21.15.204:6819","nonce":1426318231}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6822","nonce":1426318231},{"type":"v1","addr":"172.21.15.204:6823","nonce":1426318231}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6820","nonce":1426318231},{"type":"v1","addr":"172.21.15.204:6821","nonce":1426318231}]},"public_addr":"172.21.15.204:6817/1426318231","cluster_addr":"172.21.15.204:6819/1426318231","heartbeat_back_addr":"172.21.15.204:6823/1426318231","heartbeat_front_addr":"172.21.15.204:6821/1426318231","state":["exists","up"]},{"osd":9,"uuid":"a80ffe4d-1e98-460f-b046-c4754285cc5a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":69,"up_thru":88,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6826","nonce":2750635721},{"type":"v1","addr":"172.21.15.114:6827","nonce":2750635721}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6828","nonce":2750635721},{"type":"v1","addr":"172.21.15.114:6829","nonce":2750635721}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6832","nonce":2750635721},{"type":"v1","addr":"172.21.15.114:6833","nonce":2750635721}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6830","nonce":2750635721},{"type":"v1","addr":"172.21.15.114:6831","nonce":2750635721}]},"public_addr":"172.21.15.114:6827/2750635721","cluster_addr":"172.21.15.114:6829/2750635721","heartbeat_back_addr":"172.21.15.114:6833/2750635721","heartbeat_front_addr":"172.21.15.114:6831/2750635721","state":["exists","up"]},{"osd":10,"uuid":"cd75a579-bf12-4600-ac09-a8bfce772ced","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":74,"up_thru":85,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6824","nonce":1400187701},{"type":"v1","addr":"172.21.15.119:6825","nonce":1400187701}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6826","nonce":1400187701},{"type":"v1","addr":"172.21.15.119:6827","nonce":1400187701}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6830","nonce":1400187701},{"type":"v1","addr":"172.21.15.119:6831","nonce":1400187701}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6828","nonce":1400187701},{"type":"v1","addr":"172.21.15.119:6829","nonce":1400187701}]},"public_addr":"172.21.15.119:6825/1400187701","cluster_addr":"172.21.15.119:6827/1400187701","heartbeat_back_addr":"172.21.15.119:6831/1400187701","heartbeat_front_addr":"172.21.15.119:6829/1400187701","state":["exists","up"]},{"osd":11,"uuid":"53f037a5-3ec0-483a-80a1-dfee923a68e5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":82,"up_thru":88,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6824","nonce":2407298490},{"type":"v1","addr":"172.21.15.204:6825","nonce":2407298490}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6826","nonce":2407298490},{"type":"v1","addr":"172.21.15.204:6827","nonce":2407298490}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6830","nonce":2407298490},{"type":"v1","addr":"172.21.15.204:6831","nonce":2407298490}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6828","nonce":2407298490},{"type":"v1","addr":"172.21.15.204:6829","nonce":2407298490}]},"public_addr":"172.21.15.204:6825/2407298490","cluster_addr":"172.21.15.204:6827/2407298490","heartbeat_back_addr":"172.21.15.204:6831/2407298490","heartbeat_front_addr":"172.21.15.204:6829/2407298490","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-23T10:19:18.401147+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-23T10:19:42.611299+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-23T10:20:07.277367+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-23T10:20:31.208368+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-23T10:20:55.866225+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-23T10:21:19.961237+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-23T10:21:44.480783+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-23T10:22:09.173641+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-23T10:22:34.030328+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-23T10:22:59.652293+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-23T10:23:25.659673+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-23T10:23:51.168171+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"2.e","mappings":[{"from":2,"to":4}]},{"pgid":"2.1d","mappings":[{"from":8,"to":11}]},{"pgid":"3.e","mappings":[{"from":5,"to":9}]},{"pgid":"3.12","mappings":[{"from":3,"to":9}]},{"pgid":"3.17","mappings":[{"from":4,"to":6}]},{"pgid":"3.23","mappings":[{"from":4,"to":7}]},{"pgid":"3.27","mappings":[{"from":4,"to":6}]},{"pgid":"3.31","mappings":[{"from":4,"to":8}]},{"pgid":"3.38","mappings":[{"from":3,"to":9}]},{"pgid":"3.39","mappings":[{"from":5,"to":8}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.114:6801/1604678557":"2024-04-24T10:17:45.874609+0000","172.21.15.114:0/4025385854":"2024-04-24T10:17:45.874609+0000","172.21.15.114:0/2228273469":"2024-04-24T10:17:19.910348+0000","172.21.15.114:6800/2831530838":"2024-04-24T10:17:19.910348+0000","172.21.15.114:0/1333447450":"2024-04-24T10:17:45.874609+0000","172.21.15.114:0/2233026840":"2024-04-24T10:17:19.910348+0000","172.21.15.114:0/1902281545":"2024-04-24T10:17:19.910348+0000","172.21.15.114:6801/2831530838":"2024-04-24T10:17:19.910348+0000","172.21.15.114:6800/1604678557":"2024-04-24T10:17:45.874609+0000","172.21.15.114:0/1623553811":"2024-04-24T10:17:45.874609+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-23T10:26:04.634 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:04.635+0000 7f33af7cd640 1 -- 172.21.15.114:0/1520892417 >> [v2:172.21.15.114:6800/465333808,v1:172.21.15.114:6801/465333808] conn(0x7f33940a3c40 msgr2=0x7f33940a6100 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T10:26:04.635 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:04.635+0000 7f33af7cd640 1 --2- 172.21.15.114:0/1520892417 >> [v2:172.21.15.114:6800/465333808,v1:172.21.15.114:6801/465333808] conn(0x7f33940a3c40 0x7f33940a6100 secure :-1 s=READY pgs=211 cs=0 l=1 rev1=1 crypto rx=0x7f33a4005fd0 tx=0x7f33a400b410 comp rx=0 tx=0).stop 2024-04-23T10:26:04.635 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:04.635+0000 7f33af7cd640 1 -- 172.21.15.114:0/1520892417 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f33a8100e90 msgr2=0x7f33a8069c70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T10:26:04.635 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:04.635+0000 7f33af7cd640 1 --2- 172.21.15.114:0/1520892417 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f33a8100e90 0x7f33a8069c70 secure :-1 s=READY pgs=305 cs=0 l=1 rev1=1 crypto rx=0x7f3398005940 tx=0x7f3398004430 comp rx=0 tx=0).stop 2024-04-23T10:26:04.635 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:04.635+0000 7f33af7cd640 1 -- 172.21.15.114:0/1520892417 shutdown_connections 2024-04-23T10:26:04.635 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:04.635+0000 7f33af7cd640 1 --2- 172.21.15.114:0/1520892417 >> [v2:172.21.15.114:6800/465333808,v1:172.21.15.114:6801/465333808] conn(0x7f33940a3c40 0x7f33940a6100 unknown :-1 s=CLOSED pgs=211 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:26:04.635 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:04.635+0000 7f33af7cd640 1 --2- 172.21.15.114:0/1520892417 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f33a81029a0 0x7f33a819f5e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:26:04.635 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:04.635+0000 7f33af7cd640 1 --2- 172.21.15.114:0/1520892417 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f33a8101fe0 0x7f33a819cfa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:26:04.635 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:04.635+0000 7f33af7cd640 1 --2- 172.21.15.114:0/1520892417 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f33a8100e90 0x7f33a8069c70 unknown :-1 s=CLOSED pgs=305 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:26:04.635 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:04.635+0000 7f33af7cd640 1 -- 172.21.15.114:0/1520892417 >> 172.21.15.114:0/1520892417 conn(0x7f33a80fc400 msgr2=0x7f33a8105390 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T10:26:04.635 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:04.635+0000 7f33af7cd640 1 -- 172.21.15.114:0/1520892417 shutdown_connections 2024-04-23T10:26:04.635 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:04.635+0000 7f33af7cd640 1 -- 172.21.15.114:0/1520892417 wait complete. 2024-04-23T10:26:04.646 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-04-23T10:20:11.970297+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '22', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 12, 'score_stable': 12, 'optimal_score': 0.25, 'raw_score_acting': 3, 'raw_score_stable': 3, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 2, 'pool_name': 'cephfs_metadata', 'create_time': '2024-04-23T10:24:41.218540+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '86', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.8799999952316284, 'score_stable': 1.8799999952316284, 'optimal_score': 1, 'raw_score_acting': 1.8799999952316284, 'raw_score_stable': 1.8799999952316284, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 3, 'pool_name': 'cephfs_data', 'create_time': '2024-04-23T10:24:42.341799+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '87', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-04-23T10:26:04.646 DEBUG:teuthology.orchestra.run.smithi114:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-04-23T10:26:04.750 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:04.749+0000 7fe8a5719640 1 -- 172.21.15.114:0/1098177913 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fe8a0103b60 msgr2=0x7fe8a010a3f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T10:26:04.750 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:04.749+0000 7fe8a5719640 1 --2- 172.21.15.114:0/1098177913 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fe8a0103b60 0x7fe8a010a3f0 secure :-1 s=READY pgs=306 cs=0 l=1 rev1=1 crypto rx=0x7fe898009900 tx=0x7fe89802a410 comp rx=0 tx=0).stop 2024-04-23T10:26:04.750 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:04.749+0000 7fe8a5719640 1 -- 172.21.15.114:0/1098177913 shutdown_connections 2024-04-23T10:26:04.751 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:04.749+0000 7fe8a5719640 1 --2- 172.21.15.114:0/1098177913 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fe8a0103b60 0x7fe8a010a3f0 unknown :-1 s=CLOSED pgs=306 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:26:04.751 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:04.749+0000 7fe8a5719640 1 --2- 172.21.15.114:0/1098177913 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fe8a01031a0 0x7fe8a0103620 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:26:04.751 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:04.749+0000 7fe8a5719640 1 --2- 172.21.15.114:0/1098177913 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fe8a0101fb0 0x7fe8a01023d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:26:04.751 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:04.749+0000 7fe8a5719640 1 -- 172.21.15.114:0/1098177913 >> 172.21.15.114:0/1098177913 conn(0x7fe8a00fd520 msgr2=0x7fe8a00ff960 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T10:26:04.751 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:04.749+0000 7fe8a5719640 1 -- 172.21.15.114:0/1098177913 shutdown_connections 2024-04-23T10:26:04.751 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:04.749+0000 7fe8a5719640 1 -- 172.21.15.114:0/1098177913 wait complete. 2024-04-23T10:26:04.751 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:04.749+0000 7fe8a5719640 1 Processor -- start 2024-04-23T10:26:04.751 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:04.749+0000 7fe8a5719640 1 -- start start 2024-04-23T10:26:04.751 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:04.750+0000 7fe8a5719640 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fe8a0101fb0 0x7fe8a0198200 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T10:26:04.751 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:04.750+0000 7fe8a5719640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fe8a01031a0 0x7fe8a0198740 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T10:26:04.751 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:04.750+0000 7fe8a5719640 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fe8a0103b60 0x7fe8a019f7b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T10:26:04.751 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:04.750+0000 7fe8a5719640 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x7fe8a010db80 con 0x7fe8a01031a0 2024-04-23T10:26:04.751 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:04.750+0000 7fe8a5719640 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 -- 0x7fe8a010da00 con 0x7fe8a0101fb0 2024-04-23T10:26:04.751 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:04.750+0000 7fe8a5719640 1 -- --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_getmap magic: 0 -- 0x7fe8a010dd00 con 0x7fe8a0103b60 2024-04-23T10:26:04.752 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:04.750+0000 7fe89e7fc640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fe8a01031a0 0x7fe8a0198740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T10:26:04.752 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:04.750+0000 7fe89e7fc640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fe8a01031a0 0x7fe8a0198740 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.114:34180/0 (socket says 172.21.15.114:34180) 2024-04-23T10:26:04.752 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:04.750+0000 7fe89e7fc640 1 -- 172.21.15.114:0/668246424 learned_addr learned my addr 172.21.15.114:0/668246424 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-04-23T10:26:04.752 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:04.750+0000 7fe89effd640 1 --2- 172.21.15.114:0/668246424 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fe8a0101fb0 0x7fe8a0198200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T10:26:04.752 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:04.750+0000 7fe89e7fc640 1 -- 172.21.15.114:0/668246424 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fe8a0103b60 msgr2=0x7fe8a019f7b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T10:26:04.752 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:04.750+0000 7fe89e7fc640 1 --2- 172.21.15.114:0/668246424 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fe8a0103b60 0x7fe8a019f7b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:26:04.752 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:04.750+0000 7fe89e7fc640 1 -- 172.21.15.114:0/668246424 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fe8a0101fb0 msgr2=0x7fe8a0198200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T10:26:04.752 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:04.750+0000 7fe89e7fc640 1 --2- 172.21.15.114:0/668246424 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fe8a0101fb0 0x7fe8a0198200 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:26:04.752 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:04.750+0000 7fe89e7fc640 1 -- 172.21.15.114:0/668246424 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe8a019fe20 con 0x7fe8a01031a0 2024-04-23T10:26:04.752 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:04.750+0000 7fe89e7fc640 1 --2- 172.21.15.114:0/668246424 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fe8a01031a0 0x7fe8a0198740 secure :-1 s=READY pgs=307 cs=0 l=1 rev1=1 crypto rx=0x7fe89400b4a0 tx=0x7fe89400b960 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T10:26:04.752 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:04.750+0000 7fe887fff640 1 -- 172.21.15.114:0/668246424 <== mon.0 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe894014070 con 0x7fe8a01031a0 2024-04-23T10:26:04.752 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:04.750+0000 7fe887fff640 1 -- 172.21.15.114:0/668246424 <== mon.0 v2:172.21.15.114:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fe894010ac0 con 0x7fe8a01031a0 2024-04-23T10:26:04.753 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:04.750+0000 7fe887fff640 1 -- 172.21.15.114:0/668246424 <== mon.0 v2:172.21.15.114:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe894017c30 con 0x7fe8a01031a0 2024-04-23T10:26:04.753 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:04.750+0000 7fe8a5719640 1 -- 172.21.15.114:0/668246424 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fe8a01a0110 con 0x7fe8a01031a0 2024-04-23T10:26:04.753 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:04.750+0000 7fe8a5719640 1 -- 172.21.15.114:0/668246424 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fe8a01a06e0 con 0x7fe8a01031a0 2024-04-23T10:26:04.753 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:04.751+0000 7fe8a5719640 1 -- 172.21.15.114:0/668246424 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fe8a01092f0 con 0x7fe8a01031a0 2024-04-23T10:26:04.754 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:04.753+0000 7fe887fff640 1 -- 172.21.15.114:0/668246424 <== mon.0 v2:172.21.15.114:3300/0 4 ==== mgrmap(e 17) ==== 138698+0+0 (secure 0 0 0) 0x7fe894017070 con 0x7fe8a01031a0 2024-04-23T10:26:04.754 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:04.753+0000 7fe887fff640 1 --2- 172.21.15.114:0/668246424 >> [v2:172.21.15.114:6800/465333808,v1:172.21.15.114:6801/465333808] conn(0x7fe8880a3c40 0x7fe8880a6100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T10:26:04.754 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:04.754+0000 7fe89effd640 1 --2- 172.21.15.114:0/668246424 >> [v2:172.21.15.114:6800/465333808,v1:172.21.15.114:6801/465333808] conn(0x7fe8880a3c40 0x7fe8880a6100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T10:26:04.754 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:04.754+0000 7fe887fff640 1 -- 172.21.15.114:0/668246424 <== mon.0 v2:172.21.15.114:3300/0 5 ==== osd_map(89..89 src has 1..89) ==== 8269+0+0 (secure 0 0 0) 0x7fe8940cfce0 con 0x7fe8a01031a0 2024-04-23T10:26:04.754 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:04.754+0000 7fe89effd640 1 --2- 172.21.15.114:0/668246424 >> [v2:172.21.15.114:6800/465333808,v1:172.21.15.114:6801/465333808] conn(0x7fe8880a3c40 0x7fe8880a6100 secure :-1 s=READY pgs=212 cs=0 l=1 rev1=1 crypto rx=0x7fe8a0103000 tx=0x7fe890008230 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T10:26:04.756 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:04.757+0000 7fe887fff640 1 -- 172.21.15.114:0/668246424 <== mon.0 v2:172.21.15.114:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7fe89409c5f0 con 0x7fe8a01031a0 2024-04-23T10:26:04.933 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:04.934+0000 7fe8a5719640 1 -- 172.21.15.114:0/668246424 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) -- 0x7fe8a002d850 con 0x7fe8a01031a0 2024-04-23T10:26:04.933 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:04.935+0000 7fe887fff640 1 -- 172.21.15.114:0/668246424 <== mon.0 v2:172.21.15.114:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v89) ==== 93+0+10 (secure 0 0 0) 0x7fe89409bef0 con 0x7fe8a01031a0 2024-04-23T10:26:04.933 INFO:teuthology.orchestra.run.smithi114.stdout:pg_num: 1 2024-04-23T10:26:04.936 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:04.937+0000 7fe8a5719640 1 -- 172.21.15.114:0/668246424 >> [v2:172.21.15.114:6800/465333808,v1:172.21.15.114:6801/465333808] conn(0x7fe8880a3c40 msgr2=0x7fe8880a6100 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T10:26:04.936 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:04.937+0000 7fe8a5719640 1 --2- 172.21.15.114:0/668246424 >> [v2:172.21.15.114:6800/465333808,v1:172.21.15.114:6801/465333808] conn(0x7fe8880a3c40 0x7fe8880a6100 secure :-1 s=READY pgs=212 cs=0 l=1 rev1=1 crypto rx=0x7fe8a0103000 tx=0x7fe890008230 comp rx=0 tx=0).stop 2024-04-23T10:26:04.936 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:04.937+0000 7fe8a5719640 1 -- 172.21.15.114:0/668246424 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fe8a01031a0 msgr2=0x7fe8a0198740 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T10:26:04.936 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:04.937+0000 7fe8a5719640 1 --2- 172.21.15.114:0/668246424 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fe8a01031a0 0x7fe8a0198740 secure :-1 s=READY pgs=307 cs=0 l=1 rev1=1 crypto rx=0x7fe89400b4a0 tx=0x7fe89400b960 comp rx=0 tx=0).stop 2024-04-23T10:26:04.936 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:04.937+0000 7fe8a5719640 1 -- 172.21.15.114:0/668246424 shutdown_connections 2024-04-23T10:26:04.936 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:04.937+0000 7fe8a5719640 1 --2- 172.21.15.114:0/668246424 >> [v2:172.21.15.114:6800/465333808,v1:172.21.15.114:6801/465333808] conn(0x7fe8880a3c40 0x7fe8880a6100 unknown :-1 s=CLOSED pgs=212 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:26:04.936 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:04.937+0000 7fe8a5719640 1 --2- 172.21.15.114:0/668246424 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fe8a0103b60 0x7fe8a019f7b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:26:04.936 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:04.937+0000 7fe8a5719640 1 --2- 172.21.15.114:0/668246424 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fe8a01031a0 0x7fe8a0198740 unknown :-1 s=CLOSED pgs=307 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:26:04.937 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:04.937+0000 7fe8a5719640 1 --2- 172.21.15.114:0/668246424 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fe8a0101fb0 0x7fe8a0198200 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:26:04.937 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:04.937+0000 7fe8a5719640 1 -- 172.21.15.114:0/668246424 >> 172.21.15.114:0/668246424 conn(0x7fe8a00fd520 msgr2=0x7fe8a01064c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T10:26:04.937 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:04.937+0000 7fe8a5719640 1 -- 172.21.15.114:0/668246424 shutdown_connections 2024-04-23T10:26:04.937 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:04.937+0000 7fe8a5719640 1 -- 172.21.15.114:0/668246424 wait complete. 2024-04-23T10:26:04.947 DEBUG:teuthology.orchestra.run.smithi114:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2024-04-23T10:26:05.051 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:05.049+0000 7fec1b32d640 1 -- 172.21.15.114:0/3080656030 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fec14101fc0 msgr2=0x7fec14102440 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T10:26:05.051 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:05.049+0000 7fec1b32d640 1 --2- 172.21.15.114:0/3080656030 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fec14101fc0 0x7fec14102440 secure :-1 s=READY pgs=308 cs=0 l=1 rev1=1 crypto rx=0x7fec04009900 tx=0x7fec0402a410 comp rx=0 tx=0).stop 2024-04-23T10:26:05.052 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:05.049+0000 7fec1b32d640 1 -- 172.21.15.114:0/3080656030 shutdown_connections 2024-04-23T10:26:05.052 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:05.049+0000 7fec1b32d640 1 --2- 172.21.15.114:0/3080656030 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fec14102980 0x7fec141092a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:26:05.052 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:05.049+0000 7fec1b32d640 1 --2- 172.21.15.114:0/3080656030 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fec14101fc0 0x7fec14102440 unknown :-1 s=CLOSED pgs=308 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:26:05.052 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:05.049+0000 7fec1b32d640 1 --2- 172.21.15.114:0/3080656030 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fec14100e70 0x7fec14101290 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:26:05.052 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:05.049+0000 7fec1b32d640 1 -- 172.21.15.114:0/3080656030 >> 172.21.15.114:0/3080656030 conn(0x7fec140fc3e0 msgr2=0x7fec140fe820 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T10:26:05.052 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:05.049+0000 7fec1b32d640 1 -- 172.21.15.114:0/3080656030 shutdown_connections 2024-04-23T10:26:05.052 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:05.049+0000 7fec1b32d640 1 -- 172.21.15.114:0/3080656030 wait complete. 2024-04-23T10:26:05.052 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:05.050+0000 7fec1b32d640 1 Processor -- start 2024-04-23T10:26:05.052 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:05.050+0000 7fec1b32d640 1 -- start start 2024-04-23T10:26:05.052 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:05.050+0000 7fec1b32d640 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fec14100e70 0x7fec14197230 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T10:26:05.052 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:05.050+0000 7fec1b32d640 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fec14101fc0 0x7fec14197770 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T10:26:05.052 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:05.050+0000 7fec1b32d640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fec14102980 0x7fec1419e7e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T10:26:05.053 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:05.050+0000 7fec1b32d640 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x7fec1410caa0 con 0x7fec14102980 2024-04-23T10:26:05.053 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:05.050+0000 7fec1b32d640 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 -- 0x7fec1410c920 con 0x7fec14100e70 2024-04-23T10:26:05.053 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:05.050+0000 7fec1b32d640 1 -- --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_getmap magic: 0 -- 0x7fec1410cc20 con 0x7fec14101fc0 2024-04-23T10:26:05.053 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:05.050+0000 7fec198ca640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fec14102980 0x7fec1419e7e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T10:26:05.053 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:05.050+0000 7fec198ca640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fec14102980 0x7fec1419e7e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.114:34192/0 (socket says 172.21.15.114:34192) 2024-04-23T10:26:05.053 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:05.050+0000 7fec198ca640 1 -- 172.21.15.114:0/3883860348 learned_addr learned my addr 172.21.15.114:0/3883860348 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-04-23T10:26:05.053 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:05.050+0000 7fec190c9640 1 --2- 172.21.15.114:0/3883860348 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fec14100e70 0x7fec14197230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T10:26:05.053 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:05.050+0000 7fec188c8640 1 --2- 172.21.15.114:0/3883860348 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fec14101fc0 0x7fec14197770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T10:26:05.053 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:05.050+0000 7fec198ca640 1 -- 172.21.15.114:0/3883860348 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fec14101fc0 msgr2=0x7fec14197770 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T10:26:05.053 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:05.050+0000 7fec198ca640 1 --2- 172.21.15.114:0/3883860348 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fec14101fc0 0x7fec14197770 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:26:05.053 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:05.050+0000 7fec198ca640 1 -- 172.21.15.114:0/3883860348 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fec14100e70 msgr2=0x7fec14197230 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T10:26:05.053 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:05.050+0000 7fec198ca640 1 --2- 172.21.15.114:0/3883860348 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fec14100e70 0x7fec14197230 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:26:05.053 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:05.050+0000 7fec198ca640 1 -- 172.21.15.114:0/3883860348 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fec1419ee50 con 0x7fec14102980 2024-04-23T10:26:05.054 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:05.050+0000 7fec190c9640 1 --2- 172.21.15.114:0/3883860348 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fec14100e70 0x7fec14197230 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-23T10:26:05.054 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:05.051+0000 7fec198ca640 1 --2- 172.21.15.114:0/3883860348 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fec14102980 0x7fec1419e7e0 secure :-1 s=READY pgs=309 cs=0 l=1 rev1=1 crypto rx=0x7fec0c00db90 tx=0x7fec0c00dc80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T10:26:05.054 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:05.051+0000 7fec0a7fc640 1 -- 172.21.15.114:0/3883860348 <== mon.0 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fec0c00e980 con 0x7fec14102980 2024-04-23T10:26:05.054 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:05.051+0000 7fec1b32d640 1 -- 172.21.15.114:0/3883860348 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fec140690e0 con 0x7fec14102980 2024-04-23T10:26:05.054 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:05.051+0000 7fec0a7fc640 1 -- 172.21.15.114:0/3883860348 <== mon.0 v2:172.21.15.114:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fec0c009580 con 0x7fec14102980 2024-04-23T10:26:05.054 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:05.051+0000 7fec0a7fc640 1 -- 172.21.15.114:0/3883860348 <== mon.0 v2:172.21.15.114:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fec0c016650 con 0x7fec14102980 2024-04-23T10:26:05.054 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:05.051+0000 7fec1b32d640 1 -- 172.21.15.114:0/3883860348 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fec140696b0 con 0x7fec14102980 2024-04-23T10:26:05.054 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:05.052+0000 7fec1b32d640 1 -- 172.21.15.114:0/3883860348 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fec14068d70 con 0x7fec14102980 2024-04-23T10:26:05.055 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:05.053+0000 7fec0a7fc640 1 -- 172.21.15.114:0/3883860348 <== mon.0 v2:172.21.15.114:3300/0 4 ==== mgrmap(e 17) ==== 138698+0+0 (secure 0 0 0) 0x7fec0c009730 con 0x7fec14102980 2024-04-23T10:26:05.055 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:05.054+0000 7fec0a7fc640 1 --2- 172.21.15.114:0/3883860348 >> [v2:172.21.15.114:6800/465333808,v1:172.21.15.114:6801/465333808] conn(0x7fec000a3c40 0x7fec000a6100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T10:26:05.055 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:05.054+0000 7fec190c9640 1 --2- 172.21.15.114:0/3883860348 >> [v2:172.21.15.114:6800/465333808,v1:172.21.15.114:6801/465333808] conn(0x7fec000a3c40 0x7fec000a6100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T10:26:05.055 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:05.054+0000 7fec0a7fc640 1 -- 172.21.15.114:0/3883860348 <== mon.0 v2:172.21.15.114:3300/0 5 ==== osd_map(89..89 src has 1..89) ==== 8269+0+0 (secure 0 0 0) 0x7fec0c0d1bd0 con 0x7fec14102980 2024-04-23T10:26:05.055 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:05.054+0000 7fec190c9640 1 --2- 172.21.15.114:0/3883860348 >> [v2:172.21.15.114:6800/465333808,v1:172.21.15.114:6801/465333808] conn(0x7fec000a3c40 0x7fec000a6100 secure :-1 s=READY pgs=213 cs=0 l=1 rev1=1 crypto rx=0x7fec10009930 tx=0x7fec10008040 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T10:26:05.057 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:05.058+0000 7fec0a7fc640 1 -- 172.21.15.114:0/3883860348 <== mon.0 v2:172.21.15.114:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7fec0c09e540 con 0x7fec14102980 2024-04-23T10:26:05.232 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:05.233+0000 7fec1b32d640 1 -- 172.21.15.114:0/3883860348 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) -- 0x7fec1419f080 con 0x7fec14102980 2024-04-23T10:26:05.233 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:05.234+0000 7fec0a7fc640 1 -- 172.21.15.114:0/3883860348 <== mon.0 v2:172.21.15.114:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v89) ==== 104+0+11 (secure 0 0 0) 0x7fec0c09de40 con 0x7fec14102980 2024-04-23T10:26:05.233 INFO:teuthology.orchestra.run.smithi114.stdout:pg_num: 32 2024-04-23T10:26:05.235 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:05.236+0000 7fec1b32d640 1 -- 172.21.15.114:0/3883860348 >> [v2:172.21.15.114:6800/465333808,v1:172.21.15.114:6801/465333808] conn(0x7fec000a3c40 msgr2=0x7fec000a6100 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T10:26:05.235 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:05.236+0000 7fec1b32d640 1 --2- 172.21.15.114:0/3883860348 >> [v2:172.21.15.114:6800/465333808,v1:172.21.15.114:6801/465333808] conn(0x7fec000a3c40 0x7fec000a6100 secure :-1 s=READY pgs=213 cs=0 l=1 rev1=1 crypto rx=0x7fec10009930 tx=0x7fec10008040 comp rx=0 tx=0).stop 2024-04-23T10:26:05.235 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:05.236+0000 7fec1b32d640 1 -- 172.21.15.114:0/3883860348 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fec14102980 msgr2=0x7fec1419e7e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T10:26:05.235 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:05.236+0000 7fec1b32d640 1 --2- 172.21.15.114:0/3883860348 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fec14102980 0x7fec1419e7e0 secure :-1 s=READY pgs=309 cs=0 l=1 rev1=1 crypto rx=0x7fec0c00db90 tx=0x7fec0c00dc80 comp rx=0 tx=0).stop 2024-04-23T10:26:05.235 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:05.236+0000 7fec1b32d640 1 -- 172.21.15.114:0/3883860348 shutdown_connections 2024-04-23T10:26:05.235 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:05.236+0000 7fec1b32d640 1 --2- 172.21.15.114:0/3883860348 >> [v2:172.21.15.114:6800/465333808,v1:172.21.15.114:6801/465333808] conn(0x7fec000a3c40 0x7fec000a6100 unknown :-1 s=CLOSED pgs=213 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:26:05.235 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:05.236+0000 7fec1b32d640 1 --2- 172.21.15.114:0/3883860348 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fec14102980 0x7fec1419e7e0 unknown :-1 s=CLOSED pgs=309 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:26:05.235 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:05.236+0000 7fec1b32d640 1 --2- 172.21.15.114:0/3883860348 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fec14101fc0 0x7fec14197770 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:26:05.236 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:05.236+0000 7fec1b32d640 1 --2- 172.21.15.114:0/3883860348 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fec14100e70 0x7fec14197230 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:26:05.236 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:05.236+0000 7fec1b32d640 1 -- 172.21.15.114:0/3883860348 >> 172.21.15.114:0/3883860348 conn(0x7fec140fc3e0 msgr2=0x7fec141054c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T10:26:05.236 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:05.236+0000 7fec1b32d640 1 -- 172.21.15.114:0/3883860348 shutdown_connections 2024-04-23T10:26:05.236 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:05.236+0000 7fec1b32d640 1 -- 172.21.15.114:0/3883860348 wait complete. 2024-04-23T10:26:05.245 INFO:journalctl@ceph.mon.a.smithi114.stdout:Apr 23 10:26:05 smithi114 ceph-mon[29255]: pgmap v328: 97 pgs: 97 active+clean; 641 KiB data, 371 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-04-23T10:26:05.245 INFO:journalctl@ceph.mon.a.smithi114.stdout:Apr 23 10:26:05 smithi114 ceph-mon[29255]: from='client.? 172.21.15.114:0/1520892417' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-23T10:26:05.245 INFO:journalctl@ceph.mon.a.smithi114.stdout:Apr 23 10:26:05 smithi114 ceph-mon[29255]: from='client.? 172.21.15.114:0/668246424' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-04-23T10:26:05.246 DEBUG:teuthology.orchestra.run.smithi114:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2024-04-23T10:26:05.328 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 23 10:26:05 smithi204 ceph-mon[32649]: pgmap v328: 97 pgs: 97 active+clean; 641 KiB data, 371 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-04-23T10:26:05.328 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 23 10:26:05 smithi204 ceph-mon[32649]: from='client.? 172.21.15.114:0/1520892417' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-23T10:26:05.328 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 23 10:26:05 smithi204 ceph-mon[32649]: from='client.? 172.21.15.114:0/668246424' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-04-23T10:26:05.348 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:05.347+0000 7f012c7ad640 1 -- 172.21.15.114:0/4263468412 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f0124107cd0 msgr2=0x7f012410a0c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T10:26:05.349 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:05.347+0000 7f012c7ad640 1 --2- 172.21.15.114:0/4263468412 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f0124107cd0 0x7f012410a0c0 secure :-1 s=READY pgs=310 cs=0 l=1 rev1=1 crypto rx=0x7f011c009ca0 tx=0x7f011c02a8c0 comp rx=0 tx=0).stop 2024-04-23T10:26:05.349 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:05.347+0000 7f012c7ad640 1 -- 172.21.15.114:0/4263468412 shutdown_connections 2024-04-23T10:26:05.349 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:05.347+0000 7f012c7ad640 1 --2- 172.21.15.114:0/4263468412 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f0124107cd0 0x7f012410a0c0 unknown :-1 s=CLOSED pgs=310 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:26:05.349 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:05.347+0000 7f012c7ad640 1 --2- 172.21.15.114:0/4263468412 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f01241053a0 0x7f0124107790 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:26:05.349 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:05.347+0000 7f012c7ad640 1 --2- 172.21.15.114:0/4263468412 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f0124102a70 0x7f0124104e60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:26:05.349 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:05.347+0000 7f012c7ad640 1 -- 172.21.15.114:0/4263468412 >> 172.21.15.114:0/4263468412 conn(0x7f01240fc400 msgr2=0x7f01240fe840 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T10:26:05.349 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:05.347+0000 7f012c7ad640 1 -- 172.21.15.114:0/4263468412 shutdown_connections 2024-04-23T10:26:05.349 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:05.348+0000 7f012c7ad640 1 -- 172.21.15.114:0/4263468412 wait complete. 2024-04-23T10:26:05.349 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:05.348+0000 7f012c7ad640 1 Processor -- start 2024-04-23T10:26:05.349 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:05.348+0000 7f012c7ad640 1 -- start start 2024-04-23T10:26:05.349 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:05.348+0000 7f012c7ad640 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f0124102a70 0x7f0124198250 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T10:26:05.349 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:05.348+0000 7f012c7ad640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f01241053a0 0x7f0124198790 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T10:26:05.349 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:05.348+0000 7f012c7ad640 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f0124107cd0 0x7f012419f800 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T10:26:05.350 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:05.348+0000 7f012c7ad640 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x7f012410db50 con 0x7f01241053a0 2024-04-23T10:26:05.350 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:05.348+0000 7f012c7ad640 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 -- 0x7f012410d9d0 con 0x7f0124107cd0 2024-04-23T10:26:05.350 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:05.348+0000 7f012c7ad640 1 -- --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_getmap magic: 0 -- 0x7f012410dcd0 con 0x7f0124102a70 2024-04-23T10:26:05.350 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:05.348+0000 7f0129d48640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f01241053a0 0x7f0124198790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T10:26:05.350 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:05.348+0000 7f0129d48640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f01241053a0 0x7f0124198790 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.114:34218/0 (socket says 172.21.15.114:34218) 2024-04-23T10:26:05.350 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:05.348+0000 7f0129d48640 1 -- 172.21.15.114:0/3878244073 learned_addr learned my addr 172.21.15.114:0/3878244073 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-04-23T10:26:05.350 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:05.348+0000 7f012a549640 1 --2- 172.21.15.114:0/3878244073 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f0124102a70 0x7f0124198250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T10:26:05.350 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:05.348+0000 7f012ad4a640 1 --2- 172.21.15.114:0/3878244073 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f0124107cd0 0x7f012419f800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T10:26:05.350 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:05.348+0000 7f0129d48640 1 -- 172.21.15.114:0/3878244073 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f0124102a70 msgr2=0x7f0124198250 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T10:26:05.350 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:05.349+0000 7f0129d48640 1 --2- 172.21.15.114:0/3878244073 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f0124102a70 0x7f0124198250 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:26:05.350 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:05.349+0000 7f0129d48640 1 -- 172.21.15.114:0/3878244073 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f0124107cd0 msgr2=0x7f012419f800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T10:26:05.350 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:05.349+0000 7f0129d48640 1 --2- 172.21.15.114:0/3878244073 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f0124107cd0 0x7f012419f800 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:26:05.351 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:05.349+0000 7f0129d48640 1 -- 172.21.15.114:0/3878244073 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f012419fe70 con 0x7f01241053a0 2024-04-23T10:26:05.351 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:05.349+0000 7f0129d48640 1 --2- 172.21.15.114:0/3878244073 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f01241053a0 0x7f0124198790 secure :-1 s=READY pgs=311 cs=0 l=1 rev1=1 crypto rx=0x7f012000d740 tx=0x7f012000dc00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T10:26:05.351 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:05.349+0000 7f011b7fe640 1 -- 172.21.15.114:0/3878244073 <== mon.0 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0120017070 con 0x7f01241053a0 2024-04-23T10:26:05.351 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:05.349+0000 7f011b7fe640 1 -- 172.21.15.114:0/3878244073 <== mon.0 v2:172.21.15.114:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f012000de50 con 0x7f01241053a0 2024-04-23T10:26:05.351 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:05.349+0000 7f012c7ad640 1 -- 172.21.15.114:0/3878244073 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f01241a0160 con 0x7f01241053a0 2024-04-23T10:26:05.351 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:05.349+0000 7f011b7fe640 1 -- 172.21.15.114:0/3878244073 <== mon.0 v2:172.21.15.114:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0120004ce0 con 0x7f01241053a0 2024-04-23T10:26:05.351 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:05.349+0000 7f012c7ad640 1 -- 172.21.15.114:0/3878244073 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f01241a06e0 con 0x7f01241053a0 2024-04-23T10:26:05.351 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:05.350+0000 7f012c7ad640 1 -- 172.21.15.114:0/3878244073 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f01240fdef0 con 0x7f01241053a0 2024-04-23T10:26:05.352 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:05.351+0000 7f011b7fe640 1 -- 172.21.15.114:0/3878244073 <== mon.0 v2:172.21.15.114:3300/0 4 ==== mgrmap(e 17) ==== 138698+0+0 (secure 0 0 0) 0x7f0120004e80 con 0x7f01241053a0 2024-04-23T10:26:05.352 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:05.351+0000 7f011b7fe640 1 --2- 172.21.15.114:0/3878244073 >> [v2:172.21.15.114:6800/465333808,v1:172.21.15.114:6801/465333808] conn(0x7f01100a3c40 0x7f01100a6100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T10:26:05.352 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:05.351+0000 7f012a549640 1 --2- 172.21.15.114:0/3878244073 >> [v2:172.21.15.114:6800/465333808,v1:172.21.15.114:6801/465333808] conn(0x7f01100a3c40 0x7f01100a6100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T10:26:05.352 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:05.352+0000 7f011b7fe640 1 -- 172.21.15.114:0/3878244073 <== mon.0 v2:172.21.15.114:3300/0 5 ==== osd_map(89..89 src has 1..89) ==== 8269+0+0 (secure 0 0 0) 0x7f01200d0000 con 0x7f01241053a0 2024-04-23T10:26:05.352 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:05.352+0000 7f012a549640 1 --2- 172.21.15.114:0/3878244073 >> [v2:172.21.15.114:6800/465333808,v1:172.21.15.114:6801/465333808] conn(0x7f01100a3c40 0x7f01100a6100 secure :-1 s=READY pgs=214 cs=0 l=1 rev1=1 crypto rx=0x7f0114009890 tx=0x7f0114008040 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T10:26:05.355 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:05.356+0000 7f011b7fe640 1 -- 172.21.15.114:0/3878244073 <== mon.0 v2:172.21.15.114:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f012009c860 con 0x7f01241053a0 2024-04-23T10:26:05.492 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 23 10:26:05 smithi119 ceph-mon[32608]: pgmap v328: 97 pgs: 97 active+clean; 641 KiB data, 371 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-04-23T10:26:05.492 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 23 10:26:05 smithi119 ceph-mon[32608]: from='client.? 172.21.15.114:0/1520892417' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-23T10:26:05.492 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 23 10:26:05 smithi119 ceph-mon[32608]: from='client.? 172.21.15.114:0/668246424' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-04-23T10:26:05.532 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:05.533+0000 7f012c7ad640 1 -- 172.21.15.114:0/3878244073 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) -- 0x7f012402d810 con 0x7f01241053a0 2024-04-23T10:26:05.532 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:05.534+0000 7f011b7fe640 1 -- 172.21.15.114:0/3878244073 <== mon.0 v2:172.21.15.114:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v89) ==== 100+0+11 (secure 0 0 0) 0x7f012009c160 con 0x7f01241053a0 2024-04-23T10:26:05.532 INFO:teuthology.orchestra.run.smithi114.stdout:pg_num: 64 2024-04-23T10:26:05.535 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:05.535+0000 7f012c7ad640 1 -- 172.21.15.114:0/3878244073 >> [v2:172.21.15.114:6800/465333808,v1:172.21.15.114:6801/465333808] conn(0x7f01100a3c40 msgr2=0x7f01100a6100 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T10:26:05.535 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:05.535+0000 7f012c7ad640 1 --2- 172.21.15.114:0/3878244073 >> [v2:172.21.15.114:6800/465333808,v1:172.21.15.114:6801/465333808] conn(0x7f01100a3c40 0x7f01100a6100 secure :-1 s=READY pgs=214 cs=0 l=1 rev1=1 crypto rx=0x7f0114009890 tx=0x7f0114008040 comp rx=0 tx=0).stop 2024-04-23T10:26:05.535 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:05.535+0000 7f012c7ad640 1 -- 172.21.15.114:0/3878244073 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f01241053a0 msgr2=0x7f0124198790 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T10:26:05.535 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:05.535+0000 7f012c7ad640 1 --2- 172.21.15.114:0/3878244073 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f01241053a0 0x7f0124198790 secure :-1 s=READY pgs=311 cs=0 l=1 rev1=1 crypto rx=0x7f012000d740 tx=0x7f012000dc00 comp rx=0 tx=0).stop 2024-04-23T10:26:05.535 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:05.535+0000 7f012c7ad640 1 -- 172.21.15.114:0/3878244073 shutdown_connections 2024-04-23T10:26:05.535 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:05.535+0000 7f012c7ad640 1 --2- 172.21.15.114:0/3878244073 >> [v2:172.21.15.114:6800/465333808,v1:172.21.15.114:6801/465333808] conn(0x7f01100a3c40 0x7f01100a6100 unknown :-1 s=CLOSED pgs=214 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:26:05.535 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:05.535+0000 7f012c7ad640 1 --2- 172.21.15.114:0/3878244073 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f0124107cd0 0x7f012419f800 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:26:05.535 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:05.535+0000 7f012c7ad640 1 --2- 172.21.15.114:0/3878244073 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f01241053a0 0x7f0124198790 unknown :-1 s=CLOSED pgs=311 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:26:05.535 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:05.535+0000 7f012c7ad640 1 --2- 172.21.15.114:0/3878244073 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f0124102a70 0x7f0124198250 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:26:05.535 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:05.535+0000 7f012c7ad640 1 -- 172.21.15.114:0/3878244073 >> 172.21.15.114:0/3878244073 conn(0x7f01240fc400 msgr2=0x7f0124103310 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T10:26:05.535 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:05.536+0000 7f012c7ad640 1 -- 172.21.15.114:0/3878244073 shutdown_connections 2024-04-23T10:26:05.536 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:05.536+0000 7f012c7ad640 1 -- 172.21.15.114:0/3878244073 wait complete. 2024-04-23T10:26:05.546 DEBUG:teuthology.orchestra.run.smithi114:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-23T10:26:05.647 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:05.645+0000 7fadd1f29640 1 -- 172.21.15.114:0/862023515 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fadcc1029a0 msgr2=0x7fadcc1092c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T10:26:05.647 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:05.645+0000 7fadd1f29640 1 --2- 172.21.15.114:0/862023515 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fadcc1029a0 0x7fadcc1092c0 secure :-1 s=READY pgs=312 cs=0 l=1 rev1=1 crypto rx=0x7fadc4009ca0 tx=0x7fadc402a8c0 comp rx=0 tx=0).stop 2024-04-23T10:26:05.647 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:05.645+0000 7fadd1f29640 1 -- 172.21.15.114:0/862023515 shutdown_connections 2024-04-23T10:26:05.647 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:05.645+0000 7fadd1f29640 1 --2- 172.21.15.114:0/862023515 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fadcc1029a0 0x7fadcc1092c0 unknown :-1 s=CLOSED pgs=312 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:26:05.647 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:05.645+0000 7fadd1f29640 1 --2- 172.21.15.114:0/862023515 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fadcc101fe0 0x7fadcc102460 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:26:05.647 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:05.645+0000 7fadd1f29640 1 --2- 172.21.15.114:0/862023515 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fadcc100e90 0x7fadcc1012b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:26:05.647 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:05.645+0000 7fadd1f29640 1 -- 172.21.15.114:0/862023515 >> 172.21.15.114:0/862023515 conn(0x7fadcc0fc400 msgr2=0x7fadcc0fe840 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T10:26:05.647 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:05.646+0000 7fadd1f29640 1 -- 172.21.15.114:0/862023515 shutdown_connections 2024-04-23T10:26:05.647 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:05.646+0000 7fadd1f29640 1 -- 172.21.15.114:0/862023515 wait complete. 2024-04-23T10:26:05.647 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:05.646+0000 7fadd1f29640 1 Processor -- start 2024-04-23T10:26:05.647 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:05.646+0000 7fadd1f29640 1 -- start start 2024-04-23T10:26:05.648 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:05.646+0000 7fadd1f29640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fadcc100e90 0x7fadcc197040 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T10:26:05.648 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:05.646+0000 7fadd1f29640 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fadcc101fe0 0x7fadcc197580 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T10:26:05.648 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:05.646+0000 7fadd1f29640 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fadcc1029a0 0x7fadcc19e5f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T10:26:05.648 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:05.646+0000 7fadd1f29640 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x7fadcc10c9d0 con 0x7fadcc100e90 2024-04-23T10:26:05.648 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:05.646+0000 7fadd1f29640 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 -- 0x7fadcc10c850 con 0x7fadcc1029a0 2024-04-23T10:26:05.648 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:05.646+0000 7fadd1f29640 1 -- --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_getmap magic: 0 -- 0x7fadcc10cb50 con 0x7fadcc101fe0 2024-04-23T10:26:05.648 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:05.646+0000 7fadcb7fe640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fadcc100e90 0x7fadcc197040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T10:26:05.648 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:05.647+0000 7fadcb7fe640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fadcc100e90 0x7fadcc197040 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.114:57960/0 (socket says 172.21.15.114:57960) 2024-04-23T10:26:05.648 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:05.647+0000 7fadcb7fe640 1 -- 172.21.15.114:0/553273662 learned_addr learned my addr 172.21.15.114:0/553273662 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-04-23T10:26:05.648 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:05.647+0000 7fadcbfff640 1 --2- 172.21.15.114:0/553273662 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fadcc1029a0 0x7fadcc19e5f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T10:26:05.648 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:05.647+0000 7fadcaffd640 1 --2- 172.21.15.114:0/553273662 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fadcc101fe0 0x7fadcc197580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T10:26:05.648 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:05.647+0000 7fadcb7fe640 1 -- 172.21.15.114:0/553273662 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fadcc101fe0 msgr2=0x7fadcc197580 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T10:26:05.648 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:05.647+0000 7fadcb7fe640 1 --2- 172.21.15.114:0/553273662 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fadcc101fe0 0x7fadcc197580 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:26:05.648 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:05.647+0000 7fadcb7fe640 1 -- 172.21.15.114:0/553273662 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fadcc1029a0 msgr2=0x7fadcc19e5f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T10:26:05.648 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:05.647+0000 7fadcb7fe640 1 --2- 172.21.15.114:0/553273662 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fadcc1029a0 0x7fadcc19e5f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:26:05.649 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:05.647+0000 7fadcb7fe640 1 -- 172.21.15.114:0/553273662 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fadcc19ec60 con 0x7fadcc100e90 2024-04-23T10:26:05.649 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:05.647+0000 7fadcbfff640 1 --2- 172.21.15.114:0/553273662 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fadcc1029a0 0x7fadcc19e5f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-23T10:26:05.649 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:05.647+0000 7fadcb7fe640 1 --2- 172.21.15.114:0/553273662 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fadcc100e90 0x7fadcc197040 secure :-1 s=READY pgs=313 cs=0 l=1 rev1=1 crypto rx=0x7fadc000d950 tx=0x7fadc000de10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T10:26:05.649 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:05.647+0000 7fadc8ff9640 1 -- 172.21.15.114:0/553273662 <== mon.0 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fadc0017030 con 0x7fadcc100e90 2024-04-23T10:26:05.649 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:05.647+0000 7fadc8ff9640 1 -- 172.21.15.114:0/553273662 <== mon.0 v2:172.21.15.114:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fadc0004020 con 0x7fadcc100e90 2024-04-23T10:26:05.649 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:05.647+0000 7fadd1f29640 1 -- 172.21.15.114:0/553273662 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fadcc19ef50 con 0x7fadcc100e90 2024-04-23T10:26:05.649 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:05.647+0000 7fadc8ff9640 1 -- 172.21.15.114:0/553273662 <== mon.0 v2:172.21.15.114:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fadc00047a0 con 0x7fadcc100e90 2024-04-23T10:26:05.649 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:05.647+0000 7fadd1f29640 1 -- 172.21.15.114:0/553273662 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fadcc19f3b0 con 0x7fadcc100e90 2024-04-23T10:26:05.649 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:05.648+0000 7fadd1f29640 1 -- 172.21.15.114:0/553273662 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fadcc1081a0 con 0x7fadcc100e90 2024-04-23T10:26:05.650 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:05.649+0000 7fadc8ff9640 1 -- 172.21.15.114:0/553273662 <== mon.0 v2:172.21.15.114:3300/0 4 ==== mgrmap(e 17) ==== 138698+0+0 (secure 0 0 0) 0x7fadc0009a30 con 0x7fadcc100e90 2024-04-23T10:26:05.650 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:05.650+0000 7fadc8ff9640 1 --2- 172.21.15.114:0/553273662 >> [v2:172.21.15.114:6800/465333808,v1:172.21.15.114:6801/465333808] conn(0x7fadb40a3c40 0x7fadb40a6100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T10:26:05.650 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:05.650+0000 7fadcaffd640 1 --2- 172.21.15.114:0/553273662 >> [v2:172.21.15.114:6800/465333808,v1:172.21.15.114:6801/465333808] conn(0x7fadb40a3c40 0x7fadb40a6100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T10:26:05.650 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:05.650+0000 7fadc8ff9640 1 -- 172.21.15.114:0/553273662 <== mon.0 v2:172.21.15.114:3300/0 5 ==== osd_map(89..89 src has 1..89) ==== 8269+0+0 (secure 0 0 0) 0x7fadc00d0150 con 0x7fadcc100e90 2024-04-23T10:26:05.650 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:05.650+0000 7fadcaffd640 1 --2- 172.21.15.114:0/553273662 >> [v2:172.21.15.114:6800/465333808,v1:172.21.15.114:6801/465333808] conn(0x7fadb40a3c40 0x7fadb40a6100 secure :-1 s=READY pgs=215 cs=0 l=1 rev1=1 crypto rx=0x7fadbc005e00 tx=0x7fadbc005c50 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T10:26:05.653 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:05.654+0000 7fadc8ff9640 1 -- 172.21.15.114:0/553273662 <== mon.0 v2:172.21.15.114:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7fadc009c9b0 con 0x7fadcc100e90 2024-04-23T10:26:05.868 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:05.870+0000 7fadd1f29640 1 -- 172.21.15.114:0/553273662 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fadcc02d5c0 con 0x7fadcc100e90 2024-04-23T10:26:05.869 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:05.870+0000 7fadc8ff9640 1 -- 172.21.15.114:0/553273662 <== mon.0 v2:172.21.15.114:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 20 v20) ==== 94+0+11884 (secure 0 0 0) 0x7fadc0096e20 con 0x7fadcc100e90 2024-04-23T10:26:05.870 INFO:teuthology.orchestra.run.smithi114.stdout: 2024-04-23T10:26:05.870 INFO:teuthology.orchestra.run.smithi114.stdout:{"epoch":20,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14547,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.114:6835/663493580","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6834","nonce":663493580},{"type":"v1","addr":"172.21.15.114:6835","nonce":663493580}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14565,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.114:6837/1439252154","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6836","nonce":1439252154},{"type":"v1","addr":"172.21.15.114:6837","nonce":1439252154}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5},{"gid":14634,"name":"g","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.114:6839/3849796939","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6838","nonce":3849796939},{"type":"v1","addr":"172.21.15.114:6839","nonce":3849796939}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":11},{"gid":14646,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.114:6841/3544523891","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6840","nonce":3544523891},{"type":"v1","addr":"172.21.15.114:6841","nonce":3544523891}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":12},{"gid":24385,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.204:6833/3866561754","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6832","nonce":3866561754},{"type":"v1","addr":"172.21.15.204:6833","nonce":3866561754}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2},{"gid":24427,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.204:6835/2533793172","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6834","nonce":2533793172},{"type":"v1","addr":"172.21.15.204:6835","nonce":2533793172}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":7}],"filesystems":[{"mdsmap":{"epoch":20,"flags":50,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-04-23T10:24:43.383158+0000","modified":"2024-04-23T10:24:48.315128+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":3,"in":[0,1,2],"up":{"mds_0":24545,"mds_1":24509,"mds_2":24479},"failed":[],"damaged":[],"stopped":[],"info":{"gid_24439":{"gid":24439,"name":"c","rank":2,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.204:6837/513822179","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6836","nonce":513822179},{"type":"v1","addr":"172.21.15.204:6837","nonce":513822179}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24443":{"gid":24443,"name":"k","rank":1,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.119:6833/783244660","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6832","nonce":783244660},{"type":"v1","addr":"172.21.15.119:6833","nonce":783244660}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24463":{"gid":24463,"name":"i","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.204:6839/3222658893","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6838","nonce":3222658893},{"type":"v1","addr":"172.21.15.204:6839","nonce":3222658893}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24479":{"gid":24479,"name":"e","rank":2,"incarnation":19,"state":"up:active","state_seq":9,"addr":"172.21.15.119:6835/2033916709","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6834","nonce":2033916709},{"type":"v1","addr":"172.21.15.119:6835","nonce":2033916709}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24509":{"gid":24509,"name":"h","rank":1,"incarnation":17,"state":"up:active","state_seq":6,"addr":"172.21.15.119:6837/1872363128","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6836","nonce":1872363128},{"type":"v1","addr":"172.21.15.119:6837","nonce":1872363128}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24545":{"gid":24545,"name":"b","rank":0,"incarnation":14,"state":"up:active","state_seq":3,"addr":"172.21.15.119:6839/937297340","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6838","nonce":937297340},{"type":"v1","addr":"172.21.15.119:6839","nonce":937297340}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":24545,"qdb_cluster":[24509,24545,24479]},"id":1}]} 2024-04-23T10:26:05.870 INFO:teuthology.orchestra.run.smithi114.stderr:dumped fsmap epoch 20 2024-04-23T10:26:05.872 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:05.872+0000 7fadd1f29640 1 -- 172.21.15.114:0/553273662 >> [v2:172.21.15.114:6800/465333808,v1:172.21.15.114:6801/465333808] conn(0x7fadb40a3c40 msgr2=0x7fadb40a6100 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T10:26:05.872 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:05.872+0000 7fadd1f29640 1 --2- 172.21.15.114:0/553273662 >> [v2:172.21.15.114:6800/465333808,v1:172.21.15.114:6801/465333808] conn(0x7fadb40a3c40 0x7fadb40a6100 secure :-1 s=READY pgs=215 cs=0 l=1 rev1=1 crypto rx=0x7fadbc005e00 tx=0x7fadbc005c50 comp rx=0 tx=0).stop 2024-04-23T10:26:05.872 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:05.872+0000 7fadd1f29640 1 -- 172.21.15.114:0/553273662 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fadcc100e90 msgr2=0x7fadcc197040 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T10:26:05.872 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:05.872+0000 7fadd1f29640 1 --2- 172.21.15.114:0/553273662 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fadcc100e90 0x7fadcc197040 secure :-1 s=READY pgs=313 cs=0 l=1 rev1=1 crypto rx=0x7fadc000d950 tx=0x7fadc000de10 comp rx=0 tx=0).stop 2024-04-23T10:26:05.872 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:05.872+0000 7fadd1f29640 1 -- 172.21.15.114:0/553273662 shutdown_connections 2024-04-23T10:26:05.872 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:05.872+0000 7fadd1f29640 1 --2- 172.21.15.114:0/553273662 >> [v2:172.21.15.114:6800/465333808,v1:172.21.15.114:6801/465333808] conn(0x7fadb40a3c40 0x7fadb40a6100 unknown :-1 s=CLOSED pgs=215 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:26:05.872 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:05.872+0000 7fadd1f29640 1 --2- 172.21.15.114:0/553273662 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fadcc1029a0 0x7fadcc19e5f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:26:05.872 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:05.872+0000 7fadd1f29640 1 --2- 172.21.15.114:0/553273662 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fadcc101fe0 0x7fadcc197580 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:26:05.872 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:05.872+0000 7fadd1f29640 1 --2- 172.21.15.114:0/553273662 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fadcc100e90 0x7fadcc197040 unknown :-1 s=CLOSED pgs=313 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:26:05.872 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:05.872+0000 7fadd1f29640 1 -- 172.21.15.114:0/553273662 >> 172.21.15.114:0/553273662 conn(0x7fadcc0fc400 msgr2=0x7fadcc105370 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T10:26:05.873 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:05.873+0000 7fadd1f29640 1 -- 172.21.15.114:0/553273662 shutdown_connections 2024-04-23T10:26:05.873 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:05.873+0000 7fadd1f29640 1 -- 172.21.15.114:0/553273662 wait complete. 2024-04-23T10:26:05.883 DEBUG:teuthology.orchestra.run.smithi114:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-04-23T10:26:05.983 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:05.982+0000 7f3d1419b640 1 -- 172.21.15.114:0/1599066639 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f3d0c1029b0 msgr2=0x7f3d0c1092d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T10:26:05.983 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:05.982+0000 7f3d1419b640 1 --2- 172.21.15.114:0/1599066639 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f3d0c1029b0 0x7f3d0c1092d0 secure :-1 s=READY pgs=314 cs=0 l=1 rev1=1 crypto rx=0x7f3d04009ca0 tx=0x7f3d0402a8c0 comp rx=0 tx=0).stop 2024-04-23T10:26:05.983 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:05.982+0000 7f3d1419b640 1 -- 172.21.15.114:0/1599066639 shutdown_connections 2024-04-23T10:26:05.983 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:05.982+0000 7f3d1419b640 1 --2- 172.21.15.114:0/1599066639 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f3d0c1029b0 0x7f3d0c1092d0 unknown :-1 s=CLOSED pgs=314 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:26:05.983 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:05.982+0000 7f3d1419b640 1 --2- 172.21.15.114:0/1599066639 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f3d0c101ff0 0x7f3d0c102470 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:26:05.983 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:05.982+0000 7f3d1419b640 1 --2- 172.21.15.114:0/1599066639 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f3d0c100ea0 0x7f3d0c1012c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:26:05.983 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:05.982+0000 7f3d1419b640 1 -- 172.21.15.114:0/1599066639 >> 172.21.15.114:0/1599066639 conn(0x7f3d0c0fc430 msgr2=0x7f3d0c0fe850 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T10:26:05.983 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:05.982+0000 7f3d1419b640 1 -- 172.21.15.114:0/1599066639 shutdown_connections 2024-04-23T10:26:05.983 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:05.982+0000 7f3d1419b640 1 -- 172.21.15.114:0/1599066639 wait complete. 2024-04-23T10:26:05.983 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:05.983+0000 7f3d1419b640 1 Processor -- start 2024-04-23T10:26:05.983 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:05.983+0000 7f3d1419b640 1 -- start start 2024-04-23T10:26:05.984 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:05.983+0000 7f3d1419b640 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f3d0c100ea0 0x7f3d0c1981a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T10:26:05.984 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:05.983+0000 7f3d1419b640 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f3d0c101ff0 0x7f3d0c1986e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T10:26:05.984 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:05.983+0000 7f3d1419b640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f3d0c1029b0 0x7f3d0c19f750 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T10:26:05.984 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:05.983+0000 7f3d1419b640 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x7f3d0c10ca00 con 0x7f3d0c1029b0 2024-04-23T10:26:05.984 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:05.983+0000 7f3d1419b640 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 -- 0x7f3d0c10c880 con 0x7f3d0c100ea0 2024-04-23T10:26:05.984 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:05.983+0000 7f3d1419b640 1 -- --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_getmap magic: 0 -- 0x7f3d0c10cb80 con 0x7f3d0c101ff0 2024-04-23T10:26:05.984 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:05.983+0000 7f3d12738640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f3d0c1029b0 0x7f3d0c19f750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T10:26:05.984 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:05.983+0000 7f3d12738640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f3d0c1029b0 0x7f3d0c19f750 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.114:57970/0 (socket says 172.21.15.114:57970) 2024-04-23T10:26:05.984 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:05.983+0000 7f3d12738640 1 -- 172.21.15.114:0/847073594 learned_addr learned my addr 172.21.15.114:0/847073594 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-04-23T10:26:05.984 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:05.983+0000 7f3d11f37640 1 --2- 172.21.15.114:0/847073594 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f3d0c100ea0 0x7f3d0c1981a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T10:26:05.984 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:05.983+0000 7f3d12738640 1 -- 172.21.15.114:0/847073594 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f3d0c101ff0 msgr2=0x7f3d0c1986e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T10:26:05.984 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:05.983+0000 7f3d12738640 1 --2- 172.21.15.114:0/847073594 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f3d0c101ff0 0x7f3d0c1986e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:26:05.984 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:05.983+0000 7f3d12738640 1 -- 172.21.15.114:0/847073594 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f3d0c100ea0 msgr2=0x7f3d0c1981a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T10:26:05.984 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:05.983+0000 7f3d12738640 1 --2- 172.21.15.114:0/847073594 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f3d0c100ea0 0x7f3d0c1981a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:26:05.984 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:05.983+0000 7f3d12738640 1 -- 172.21.15.114:0/847073594 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f3d0c19fdc0 con 0x7f3d0c1029b0 2024-04-23T10:26:05.985 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:05.983+0000 7f3d12738640 1 --2- 172.21.15.114:0/847073594 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f3d0c1029b0 0x7f3d0c19f750 secure :-1 s=READY pgs=315 cs=0 l=1 rev1=1 crypto rx=0x7f3d04031a20 tx=0x7f3d04003c20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T10:26:05.985 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:05.984+0000 7f3d02ffd640 1 -- 172.21.15.114:0/847073594 <== mon.0 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3d04004240 con 0x7f3d0c1029b0 2024-04-23T10:26:05.985 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:05.984+0000 7f3d1419b640 1 -- 172.21.15.114:0/847073594 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f3d0c1a0050 con 0x7f3d0c1029b0 2024-04-23T10:26:05.985 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:05.984+0000 7f3d02ffd640 1 -- 172.21.15.114:0/847073594 <== mon.0 v2:172.21.15.114:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f3d04003e10 con 0x7f3d0c1029b0 2024-04-23T10:26:05.985 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:05.984+0000 7f3d02ffd640 1 -- 172.21.15.114:0/847073594 <== mon.0 v2:172.21.15.114:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3d040305c0 con 0x7f3d0c1029b0 2024-04-23T10:26:05.985 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:05.984+0000 7f3d1419b640 1 -- 172.21.15.114:0/847073594 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f3d0c1a05a0 con 0x7f3d0c1029b0 2024-04-23T10:26:05.986 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:05.985+0000 7f3d1419b640 1 -- 172.21.15.114:0/847073594 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f3d0c1a0800 con 0x7f3d0c1029b0 2024-04-23T10:26:05.986 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:05.986+0000 7f3d02ffd640 1 -- 172.21.15.114:0/847073594 <== mon.0 v2:172.21.15.114:3300/0 4 ==== mgrmap(e 17) ==== 138698+0+0 (secure 0 0 0) 0x7f3d0402f660 con 0x7f3d0c1029b0 2024-04-23T10:26:05.986 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:05.986+0000 7f3d02ffd640 1 --2- 172.21.15.114:0/847073594 >> [v2:172.21.15.114:6800/465333808,v1:172.21.15.114:6801/465333808] conn(0x7f3cf80a3c40 0x7f3cf80a6100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T10:26:05.986 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:05.986+0000 7f3d11f37640 1 --2- 172.21.15.114:0/847073594 >> [v2:172.21.15.114:6800/465333808,v1:172.21.15.114:6801/465333808] conn(0x7f3cf80a3c40 0x7f3cf80a6100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T10:26:05.986 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:05.986+0000 7f3d02ffd640 1 -- 172.21.15.114:0/847073594 <== mon.0 v2:172.21.15.114:3300/0 5 ==== osd_map(89..89 src has 1..89) ==== 8269+0+0 (secure 0 0 0) 0x7f3d040f1d70 con 0x7f3d0c1029b0 2024-04-23T10:26:05.986 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:05.987+0000 7f3d11f37640 1 --2- 172.21.15.114:0/847073594 >> [v2:172.21.15.114:6800/465333808,v1:172.21.15.114:6801/465333808] conn(0x7f3cf80a3c40 0x7f3cf80a6100 secure :-1 s=READY pgs=216 cs=0 l=1 rev1=1 crypto rx=0x7f3d080058c0 tx=0x7f3d0800a250 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T10:26:05.990 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:05.991+0000 7f3d02ffd640 1 -- 172.21.15.114:0/847073594 <== mon.0 v2:172.21.15.114:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f3d040be6e0 con 0x7f3d0c1029b0 2024-04-23T10:26:06.174 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:06.176+0000 7f3d1419b640 1 -- 172.21.15.114:0/847073594 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f3d0c1a0b90 con 0x7f3d0c1029b0 2024-04-23T10:26:06.176 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:06.177+0000 7f3d02ffd640 1 -- 172.21.15.114:0/847073594 <== mon.0 v2:172.21.15.114:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v89) ==== 74+0+22818 (secure 0 0 0) 0x7f3d040bdfe0 con 0x7f3d0c1029b0 2024-04-23T10:26:06.176 INFO:teuthology.orchestra.run.smithi114.stdout: 2024-04-23T10:26:06.176 INFO:teuthology.orchestra.run.smithi114.stdout:{"epoch":89,"fsid":"aa240572-0159-11ef-bc93-c7b262605968","created":"2024-04-23T10:16:53.351136+0000","modified":"2024-04-23T10:25:47.573901+0000","last_up_change":"2024-04-23T10:23:54.153269+0000","last_in_change":"2024-04-23T10:23:32.963085+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":26,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-04-23T10:20:11.970297+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":12,"score_stable":12,"optimal_score":0.25,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-04-23T10:24:41.218540+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"86","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.8799999952316284,"score_stable":1.8799999952316284,"optimal_score":1,"raw_score_acting":1.8799999952316284,"raw_score_stable":1.8799999952316284,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-04-23T10:24:42.341799+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"87","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"fd64adde-9445-45a4-a171-ef656e2c5709","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":88,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6802","nonce":736366878},{"type":"v1","addr":"172.21.15.114:6803","nonce":736366878}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6804","nonce":736366878},{"type":"v1","addr":"172.21.15.114:6805","nonce":736366878}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6808","nonce":736366878},{"type":"v1","addr":"172.21.15.114:6809","nonce":736366878}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6806","nonce":736366878},{"type":"v1","addr":"172.21.15.114:6807","nonce":736366878}]},"public_addr":"172.21.15.114:6803/736366878","cluster_addr":"172.21.15.114:6805/736366878","heartbeat_back_addr":"172.21.15.114:6809/736366878","heartbeat_front_addr":"172.21.15.114:6807/736366878","state":["exists","up"]},{"osd":1,"uuid":"e279a1e6-2702-472a-930a-e08ece76a861","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":85,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6800","nonce":1133540779},{"type":"v1","addr":"172.21.15.119:6801","nonce":1133540779}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6802","nonce":1133540779},{"type":"v1","addr":"172.21.15.119:6803","nonce":1133540779}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6806","nonce":1133540779},{"type":"v1","addr":"172.21.15.119:6807","nonce":1133540779}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6804","nonce":1133540779},{"type":"v1","addr":"172.21.15.119:6805","nonce":1133540779}]},"public_addr":"172.21.15.119:6801/1133540779","cluster_addr":"172.21.15.119:6803/1133540779","heartbeat_back_addr":"172.21.15.119:6807/1133540779","heartbeat_front_addr":"172.21.15.119:6805/1133540779","state":["exists","up"]},{"osd":2,"uuid":"f4c9f2b4-ff49-4e14-89cd-b477feeb9382","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":85,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6800","nonce":799652511},{"type":"v1","addr":"172.21.15.204:6801","nonce":799652511}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6802","nonce":799652511},{"type":"v1","addr":"172.21.15.204:6803","nonce":799652511}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6806","nonce":799652511},{"type":"v1","addr":"172.21.15.204:6807","nonce":799652511}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6804","nonce":799652511},{"type":"v1","addr":"172.21.15.204:6805","nonce":799652511}]},"public_addr":"172.21.15.204:6801/799652511","cluster_addr":"172.21.15.204:6803/799652511","heartbeat_back_addr":"172.21.15.204:6807/799652511","heartbeat_front_addr":"172.21.15.204:6805/799652511","state":["exists","up"]},{"osd":3,"uuid":"52d2a566-e71c-4bb3-8d6c-5abdd43177c2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":25,"up_thru":88,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6810","nonce":4275595091},{"type":"v1","addr":"172.21.15.114:6811","nonce":4275595091}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6812","nonce":4275595091},{"type":"v1","addr":"172.21.15.114:6813","nonce":4275595091}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6816","nonce":4275595091},{"type":"v1","addr":"172.21.15.114:6817","nonce":4275595091}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6814","nonce":4275595091},{"type":"v1","addr":"172.21.15.114:6815","nonce":4275595091}]},"public_addr":"172.21.15.114:6811/4275595091","cluster_addr":"172.21.15.114:6813/4275595091","heartbeat_back_addr":"172.21.15.114:6817/4275595091","heartbeat_front_addr":"172.21.15.114:6815/4275595091","state":["exists","up"]},{"osd":4,"uuid":"d911a7c1-4bdc-4d28-b8a6-73896a6e7de6","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":33,"up_thru":85,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6808","nonce":1611007172},{"type":"v1","addr":"172.21.15.119:6809","nonce":1611007172}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6810","nonce":1611007172},{"type":"v1","addr":"172.21.15.119:6811","nonce":1611007172}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6814","nonce":1611007172},{"type":"v1","addr":"172.21.15.119:6815","nonce":1611007172}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6812","nonce":1611007172},{"type":"v1","addr":"172.21.15.119:6813","nonce":1611007172}]},"public_addr":"172.21.15.119:6809/1611007172","cluster_addr":"172.21.15.119:6811/1611007172","heartbeat_back_addr":"172.21.15.119:6815/1611007172","heartbeat_front_addr":"172.21.15.119:6813/1611007172","state":["exists","up"]},{"osd":5,"uuid":"f7590e1a-e93c-4c29-b15f-38ee52d1f921","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":39,"up_thru":85,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6808","nonce":1427563312},{"type":"v1","addr":"172.21.15.204:6809","nonce":1427563312}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6810","nonce":1427563312},{"type":"v1","addr":"172.21.15.204:6811","nonce":1427563312}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6814","nonce":1427563312},{"type":"v1","addr":"172.21.15.204:6815","nonce":1427563312}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6812","nonce":1427563312},{"type":"v1","addr":"172.21.15.204:6813","nonce":1427563312}]},"public_addr":"172.21.15.204:6809/1427563312","cluster_addr":"172.21.15.204:6811/1427563312","heartbeat_back_addr":"172.21.15.204:6815/1427563312","heartbeat_front_addr":"172.21.15.204:6813/1427563312","state":["exists","up"]},{"osd":6,"uuid":"534d20e8-3e54-4db8-a845-6119f49c7a78","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":47,"up_thru":88,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6818","nonce":4072912544},{"type":"v1","addr":"172.21.15.114:6819","nonce":4072912544}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6820","nonce":4072912544},{"type":"v1","addr":"172.21.15.114:6821","nonce":4072912544}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6824","nonce":4072912544},{"type":"v1","addr":"172.21.15.114:6825","nonce":4072912544}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6822","nonce":4072912544},{"type":"v1","addr":"172.21.15.114:6823","nonce":4072912544}]},"public_addr":"172.21.15.114:6819/4072912544","cluster_addr":"172.21.15.114:6821/4072912544","heartbeat_back_addr":"172.21.15.114:6825/4072912544","heartbeat_front_addr":"172.21.15.114:6823/4072912544","state":["exists","up"]},{"osd":7,"uuid":"f0ec22b8-1b48-4859-9d65-b00403aff3ad","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":53,"up_thru":88,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6816","nonce":2237458793},{"type":"v1","addr":"172.21.15.119:6817","nonce":2237458793}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6818","nonce":2237458793},{"type":"v1","addr":"172.21.15.119:6819","nonce":2237458793}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6822","nonce":2237458793},{"type":"v1","addr":"172.21.15.119:6823","nonce":2237458793}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6820","nonce":2237458793},{"type":"v1","addr":"172.21.15.119:6821","nonce":2237458793}]},"public_addr":"172.21.15.119:6817/2237458793","cluster_addr":"172.21.15.119:6819/2237458793","heartbeat_back_addr":"172.21.15.119:6823/2237458793","heartbeat_front_addr":"172.21.15.119:6821/2237458793","state":["exists","up"]},{"osd":8,"uuid":"3ac89686-6d76-4cef-9089-ac29e4b3bcac","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":62,"up_thru":85,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6816","nonce":1426318231},{"type":"v1","addr":"172.21.15.204:6817","nonce":1426318231}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6818","nonce":1426318231},{"type":"v1","addr":"172.21.15.204:6819","nonce":1426318231}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6822","nonce":1426318231},{"type":"v1","addr":"172.21.15.204:6823","nonce":1426318231}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6820","nonce":1426318231},{"type":"v1","addr":"172.21.15.204:6821","nonce":1426318231}]},"public_addr":"172.21.15.204:6817/1426318231","cluster_addr":"172.21.15.204:6819/1426318231","heartbeat_back_addr":"172.21.15.204:6823/1426318231","heartbeat_front_addr":"172.21.15.204:6821/1426318231","state":["exists","up"]},{"osd":9,"uuid":"a80ffe4d-1e98-460f-b046-c4754285cc5a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":69,"up_thru":88,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6826","nonce":2750635721},{"type":"v1","addr":"172.21.15.114:6827","nonce":2750635721}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6828","nonce":2750635721},{"type":"v1","addr":"172.21.15.114:6829","nonce":2750635721}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6832","nonce":2750635721},{"type":"v1","addr":"172.21.15.114:6833","nonce":2750635721}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6830","nonce":2750635721},{"type":"v1","addr":"172.21.15.114:6831","nonce":2750635721}]},"public_addr":"172.21.15.114:6827/2750635721","cluster_addr":"172.21.15.114:6829/2750635721","heartbeat_back_addr":"172.21.15.114:6833/2750635721","heartbeat_front_addr":"172.21.15.114:6831/2750635721","state":["exists","up"]},{"osd":10,"uuid":"cd75a579-bf12-4600-ac09-a8bfce772ced","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":74,"up_thru":85,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6824","nonce":1400187701},{"type":"v1","addr":"172.21.15.119:6825","nonce":1400187701}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6826","nonce":1400187701},{"type":"v1","addr":"172.21.15.119:6827","nonce":1400187701}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6830","nonce":1400187701},{"type":"v1","addr":"172.21.15.119:6831","nonce":1400187701}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6828","nonce":1400187701},{"type":"v1","addr":"172.21.15.119:6829","nonce":1400187701}]},"public_addr":"172.21.15.119:6825/1400187701","cluster_addr":"172.21.15.119:6827/1400187701","heartbeat_back_addr":"172.21.15.119:6831/1400187701","heartbeat_front_addr":"172.21.15.119:6829/1400187701","state":["exists","up"]},{"osd":11,"uuid":"53f037a5-3ec0-483a-80a1-dfee923a68e5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":82,"up_thru":88,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6824","nonce":2407298490},{"type":"v1","addr":"172.21.15.204:6825","nonce":2407298490}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6826","nonce":2407298490},{"type":"v1","addr":"172.21.15.204:6827","nonce":2407298490}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6830","nonce":2407298490},{"type":"v1","addr":"172.21.15.204:6831","nonce":2407298490}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6828","nonce":2407298490},{"type":"v1","addr":"172.21.15.204:6829","nonce":2407298490}]},"public_addr":"172.21.15.204:6825/2407298490","cluster_addr":"172.21.15.204:6827/2407298490","heartbeat_back_addr":"172.21.15.204:6831/2407298490","heartbeat_front_addr":"172.21.15.204:6829/2407298490","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-23T10:19:18.401147+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-23T10:19:42.611299+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-23T10:20:07.277367+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-23T10:20:31.208368+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-23T10:20:55.866225+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-23T10:21:19.961237+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-23T10:21:44.480783+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-23T10:22:09.173641+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-23T10:22:34.030328+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-23T10:22:59.652293+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-23T10:23:25.659673+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-23T10:23:51.168171+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"2.e","mappings":[{"from":2,"to":4}]},{"pgid":"2.1d","mappings":[{"from":8,"to":11}]},{"pgid":"3.e","mappings":[{"from":5,"to":9}]},{"pgid":"3.12","mappings":[{"from":3,"to":9}]},{"pgid":"3.17","mappings":[{"from":4,"to":6}]},{"pgid":"3.23","mappings":[{"from":4,"to":7}]},{"pgid":"3.27","mappings":[{"from":4,"to":6}]},{"pgid":"3.31","mappings":[{"from":4,"to":8}]},{"pgid":"3.38","mappings":[{"from":3,"to":9}]},{"pgid":"3.39","mappings":[{"from":5,"to":8}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.114:6801/1604678557":"2024-04-24T10:17:45.874609+0000","172.21.15.114:0/4025385854":"2024-04-24T10:17:45.874609+0000","172.21.15.114:0/2228273469":"2024-04-24T10:17:19.910348+0000","172.21.15.114:6800/2831530838":"2024-04-24T10:17:19.910348+0000","172.21.15.114:0/1333447450":"2024-04-24T10:17:45.874609+0000","172.21.15.114:0/2233026840":"2024-04-24T10:17:19.910348+0000","172.21.15.114:0/1902281545":"2024-04-24T10:17:19.910348+0000","172.21.15.114:6801/2831530838":"2024-04-24T10:17:19.910348+0000","172.21.15.114:6800/1604678557":"2024-04-24T10:17:45.874609+0000","172.21.15.114:0/1623553811":"2024-04-24T10:17:45.874609+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-23T10:26:06.178 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:06.179+0000 7f3d1419b640 1 -- 172.21.15.114:0/847073594 >> [v2:172.21.15.114:6800/465333808,v1:172.21.15.114:6801/465333808] conn(0x7f3cf80a3c40 msgr2=0x7f3cf80a6100 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T10:26:06.178 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:06.179+0000 7f3d1419b640 1 --2- 172.21.15.114:0/847073594 >> [v2:172.21.15.114:6800/465333808,v1:172.21.15.114:6801/465333808] conn(0x7f3cf80a3c40 0x7f3cf80a6100 secure :-1 s=READY pgs=216 cs=0 l=1 rev1=1 crypto rx=0x7f3d080058c0 tx=0x7f3d0800a250 comp rx=0 tx=0).stop 2024-04-23T10:26:06.179 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:06.179+0000 7f3d1419b640 1 -- 172.21.15.114:0/847073594 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f3d0c1029b0 msgr2=0x7f3d0c19f750 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T10:26:06.179 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:06.179+0000 7f3d1419b640 1 --2- 172.21.15.114:0/847073594 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f3d0c1029b0 0x7f3d0c19f750 secure :-1 s=READY pgs=315 cs=0 l=1 rev1=1 crypto rx=0x7f3d04031a20 tx=0x7f3d04003c20 comp rx=0 tx=0).stop 2024-04-23T10:26:06.179 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:06.179+0000 7f3d1419b640 1 -- 172.21.15.114:0/847073594 shutdown_connections 2024-04-23T10:26:06.179 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:06.179+0000 7f3d1419b640 1 --2- 172.21.15.114:0/847073594 >> [v2:172.21.15.114:6800/465333808,v1:172.21.15.114:6801/465333808] conn(0x7f3cf80a3c40 0x7f3cf80a6100 unknown :-1 s=CLOSED pgs=216 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:26:06.179 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:06.179+0000 7f3d1419b640 1 --2- 172.21.15.114:0/847073594 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f3d0c1029b0 0x7f3d0c19f750 unknown :-1 s=CLOSED pgs=315 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:26:06.179 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:06.179+0000 7f3d1419b640 1 --2- 172.21.15.114:0/847073594 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f3d0c101ff0 0x7f3d0c1986e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:26:06.179 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:06.179+0000 7f3d1419b640 1 --2- 172.21.15.114:0/847073594 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f3d0c100ea0 0x7f3d0c1981a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:26:06.179 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:06.179+0000 7f3d1419b640 1 -- 172.21.15.114:0/847073594 >> 172.21.15.114:0/847073594 conn(0x7f3d0c0fc430 msgr2=0x7f3d0c0fdf10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T10:26:06.179 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:06.179+0000 7f3d1419b640 1 -- 172.21.15.114:0/847073594 shutdown_connections 2024-04-23T10:26:06.179 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:06.179+0000 7f3d1419b640 1 -- 172.21.15.114:0/847073594 wait complete. 2024-04-23T10:26:06.188 INFO:journalctl@ceph.mon.a.smithi114.stdout:Apr 23 10:26:06 smithi114 ceph-mon[29255]: from='client.? 172.21.15.114:0/3883860348' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2024-04-23T10:26:06.188 INFO:journalctl@ceph.mon.a.smithi114.stdout:Apr 23 10:26:06 smithi114 ceph-mon[29255]: from='client.? 172.21.15.114:0/3878244073' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2024-04-23T10:26:06.188 INFO:journalctl@ceph.mon.a.smithi114.stdout:Apr 23 10:26:06 smithi114 ceph-mon[29255]: from='client.? 172.21.15.114:0/553273662' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-23T10:26:06.189 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-04-23T10:20:11.970297+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '22', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 12, 'score_stable': 12, 'optimal_score': 0.25, 'raw_score_acting': 3, 'raw_score_stable': 3, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 2, 'pool_name': 'cephfs_metadata', 'create_time': '2024-04-23T10:24:41.218540+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '86', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.8799999952316284, 'score_stable': 1.8799999952316284, 'optimal_score': 1, 'raw_score_acting': 1.8799999952316284, 'raw_score_stable': 1.8799999952316284, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 3, 'pool_name': 'cephfs_data', 'create_time': '2024-04-23T10:24:42.341799+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '87', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-04-23T10:26:06.190 DEBUG:teuthology.orchestra.run.smithi114:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-04-23T10:26:06.292 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:06.291+0000 7ff3def29640 1 -- 172.21.15.114:0/339234233 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7ff3d8105340 msgr2=0x7ff3d8107730 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T10:26:06.293 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:06.291+0000 7ff3def29640 1 --2- 172.21.15.114:0/339234233 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7ff3d8105340 0x7ff3d8107730 secure :-1 s=READY pgs=316 cs=0 l=1 rev1=1 crypto rx=0x7ff3cc009900 tx=0x7ff3cc02a410 comp rx=0 tx=0).stop 2024-04-23T10:26:06.293 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:06.291+0000 7ff3def29640 1 -- 172.21.15.114:0/339234233 shutdown_connections 2024-04-23T10:26:06.293 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:06.291+0000 7ff3def29640 1 --2- 172.21.15.114:0/339234233 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7ff3d8107c70 0x7ff3d810a0c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:26:06.293 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:06.291+0000 7ff3def29640 1 --2- 172.21.15.114:0/339234233 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7ff3d8105340 0x7ff3d8107730 unknown :-1 s=CLOSED pgs=316 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:26:06.293 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:06.291+0000 7ff3def29640 1 --2- 172.21.15.114:0/339234233 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7ff3d8102a10 0x7ff3d8104e00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:26:06.293 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:06.291+0000 7ff3def29640 1 -- 172.21.15.114:0/339234233 >> 172.21.15.114:0/339234233 conn(0x7ff3d80fc400 msgr2=0x7ff3d80fe840 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T10:26:06.293 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:06.291+0000 7ff3def29640 1 -- 172.21.15.114:0/339234233 shutdown_connections 2024-04-23T10:26:06.293 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:06.291+0000 7ff3def29640 1 -- 172.21.15.114:0/339234233 wait complete. 2024-04-23T10:26:06.293 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:06.292+0000 7ff3def29640 1 Processor -- start 2024-04-23T10:26:06.293 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:06.292+0000 7ff3def29640 1 -- start start 2024-04-23T10:26:06.293 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:06.292+0000 7ff3def29640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7ff3d8102a10 0x7ff3d81983b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T10:26:06.293 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:06.292+0000 7ff3def29640 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7ff3d8105340 0x7ff3d81988f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T10:26:06.294 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:06.292+0000 7ff3def29640 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7ff3d8107c70 0x7ff3d819f960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T10:26:06.294 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:06.292+0000 7ff3def29640 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x7ff3d810dbe0 con 0x7ff3d8102a10 2024-04-23T10:26:06.294 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:06.292+0000 7ff3def29640 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 -- 0x7ff3d810da60 con 0x7ff3d8107c70 2024-04-23T10:26:06.294 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:06.292+0000 7ff3def29640 1 -- --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_getmap magic: 0 -- 0x7ff3d810dd60 con 0x7ff3d8105340 2024-04-23T10:26:06.294 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:06.292+0000 7ff3dccc5640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7ff3d8102a10 0x7ff3d81983b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T10:26:06.294 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:06.292+0000 7ff3dccc5640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7ff3d8102a10 0x7ff3d81983b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.114:57998/0 (socket says 172.21.15.114:57998) 2024-04-23T10:26:06.294 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:06.292+0000 7ff3dccc5640 1 -- 172.21.15.114:0/1366471159 learned_addr learned my addr 172.21.15.114:0/1366471159 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-04-23T10:26:06.294 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:06.292+0000 7ff3dd4c6640 1 --2- 172.21.15.114:0/1366471159 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7ff3d8107c70 0x7ff3d819f960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T10:26:06.294 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:06.292+0000 7ff3d7fff640 1 --2- 172.21.15.114:0/1366471159 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7ff3d8105340 0x7ff3d81988f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T10:26:06.294 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:06.292+0000 7ff3dccc5640 1 -- 172.21.15.114:0/1366471159 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7ff3d8105340 msgr2=0x7ff3d81988f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T10:26:06.294 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:06.292+0000 7ff3dccc5640 1 --2- 172.21.15.114:0/1366471159 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7ff3d8105340 0x7ff3d81988f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:26:06.294 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:06.292+0000 7ff3dccc5640 1 -- 172.21.15.114:0/1366471159 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7ff3d8107c70 msgr2=0x7ff3d819f960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T10:26:06.294 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:06.292+0000 7ff3dccc5640 1 --2- 172.21.15.114:0/1366471159 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7ff3d8107c70 0x7ff3d819f960 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:26:06.295 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:06.292+0000 7ff3dccc5640 1 -- 172.21.15.114:0/1366471159 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff3d819ffd0 con 0x7ff3d8102a10 2024-04-23T10:26:06.295 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:06.292+0000 7ff3d7fff640 1 --2- 172.21.15.114:0/1366471159 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7ff3d8105340 0x7ff3d81988f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-23T10:26:06.295 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:06.292+0000 7ff3dccc5640 1 --2- 172.21.15.114:0/1366471159 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7ff3d8102a10 0x7ff3d81983b0 secure :-1 s=READY pgs=317 cs=0 l=1 rev1=1 crypto rx=0x7ff3c800b4a0 tx=0x7ff3c800b960 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T10:26:06.295 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:06.293+0000 7ff3d5ffb640 1 -- 172.21.15.114:0/1366471159 <== mon.0 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff3c8014070 con 0x7ff3d8102a10 2024-04-23T10:26:06.295 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:06.293+0000 7ff3d5ffb640 1 -- 172.21.15.114:0/1366471159 <== mon.0 v2:172.21.15.114:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7ff3c8010ac0 con 0x7ff3d8102a10 2024-04-23T10:26:06.295 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:06.293+0000 7ff3def29640 1 -- 172.21.15.114:0/1366471159 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7ff3d81a02c0 con 0x7ff3d8102a10 2024-04-23T10:26:06.295 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:06.293+0000 7ff3d5ffb640 1 -- 172.21.15.114:0/1366471159 <== mon.0 v2:172.21.15.114:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff3c8017c30 con 0x7ff3d8102a10 2024-04-23T10:26:06.295 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:06.293+0000 7ff3def29640 1 -- 172.21.15.114:0/1366471159 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7ff3d81a0860 con 0x7ff3d8102a10 2024-04-23T10:26:06.295 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:06.294+0000 7ff3def29640 1 -- 172.21.15.114:0/1366471159 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7ff3d80fdef0 con 0x7ff3d8102a10 2024-04-23T10:26:06.296 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:06.295+0000 7ff3d5ffb640 1 -- 172.21.15.114:0/1366471159 <== mon.0 v2:172.21.15.114:3300/0 4 ==== mgrmap(e 17) ==== 138698+0+0 (secure 0 0 0) 0x7ff3c8017070 con 0x7ff3d8102a10 2024-04-23T10:26:06.296 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:06.295+0000 7ff3d5ffb640 1 --2- 172.21.15.114:0/1366471159 >> [v2:172.21.15.114:6800/465333808,v1:172.21.15.114:6801/465333808] conn(0x7ff3c00a3c40 0x7ff3c00a6100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T10:26:06.296 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:06.295+0000 7ff3d7fff640 1 --2- 172.21.15.114:0/1366471159 >> [v2:172.21.15.114:6800/465333808,v1:172.21.15.114:6801/465333808] conn(0x7ff3c00a3c40 0x7ff3c00a6100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T10:26:06.296 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:06.296+0000 7ff3d5ffb640 1 -- 172.21.15.114:0/1366471159 <== mon.0 v2:172.21.15.114:3300/0 5 ==== osd_map(89..89 src has 1..89) ==== 8269+0+0 (secure 0 0 0) 0x7ff3c80cfb80 con 0x7ff3d8102a10 2024-04-23T10:26:06.296 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:06.296+0000 7ff3d7fff640 1 --2- 172.21.15.114:0/1366471159 >> [v2:172.21.15.114:6800/465333808,v1:172.21.15.114:6801/465333808] conn(0x7ff3c00a3c40 0x7ff3c00a6100 secure :-1 s=READY pgs=217 cs=0 l=1 rev1=1 crypto rx=0x7ff3cc006010 tx=0x7ff3cc0314a0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T10:26:06.298 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:06.300+0000 7ff3d5ffb640 1 -- 172.21.15.114:0/1366471159 <== mon.0 v2:172.21.15.114:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7ff3c809c4f0 con 0x7ff3d8102a10 2024-04-23T10:26:06.328 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 23 10:26:06 smithi204 ceph-mon[32649]: from='client.? 172.21.15.114:0/3883860348' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2024-04-23T10:26:06.328 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 23 10:26:06 smithi204 ceph-mon[32649]: from='client.? 172.21.15.114:0/3878244073' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2024-04-23T10:26:06.328 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 23 10:26:06 smithi204 ceph-mon[32649]: from='client.? 172.21.15.114:0/553273662' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-23T10:26:06.476 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:06.477+0000 7ff3def29640 1 -- 172.21.15.114:0/1366471159 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) -- 0x7ff3d8062b50 con 0x7ff3d8102a10 2024-04-23T10:26:06.477 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:06.478+0000 7ff3d5ffb640 1 -- 172.21.15.114:0/1366471159 <== mon.0 v2:172.21.15.114:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v89) ==== 93+0+10 (secure 0 0 0) 0x7ff3c809bdf0 con 0x7ff3d8102a10 2024-04-23T10:26:06.477 INFO:teuthology.orchestra.run.smithi114.stdout:pg_num: 1 2024-04-23T10:26:06.479 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:06.480+0000 7ff3def29640 1 -- 172.21.15.114:0/1366471159 >> [v2:172.21.15.114:6800/465333808,v1:172.21.15.114:6801/465333808] conn(0x7ff3c00a3c40 msgr2=0x7ff3c00a6100 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T10:26:06.479 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:06.480+0000 7ff3def29640 1 --2- 172.21.15.114:0/1366471159 >> [v2:172.21.15.114:6800/465333808,v1:172.21.15.114:6801/465333808] conn(0x7ff3c00a3c40 0x7ff3c00a6100 secure :-1 s=READY pgs=217 cs=0 l=1 rev1=1 crypto rx=0x7ff3cc006010 tx=0x7ff3cc0314a0 comp rx=0 tx=0).stop 2024-04-23T10:26:06.479 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:06.480+0000 7ff3def29640 1 -- 172.21.15.114:0/1366471159 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7ff3d8102a10 msgr2=0x7ff3d81983b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T10:26:06.479 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:06.480+0000 7ff3def29640 1 --2- 172.21.15.114:0/1366471159 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7ff3d8102a10 0x7ff3d81983b0 secure :-1 s=READY pgs=317 cs=0 l=1 rev1=1 crypto rx=0x7ff3c800b4a0 tx=0x7ff3c800b960 comp rx=0 tx=0).stop 2024-04-23T10:26:06.479 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:06.480+0000 7ff3def29640 1 -- 172.21.15.114:0/1366471159 shutdown_connections 2024-04-23T10:26:06.479 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:06.480+0000 7ff3def29640 1 --2- 172.21.15.114:0/1366471159 >> [v2:172.21.15.114:6800/465333808,v1:172.21.15.114:6801/465333808] conn(0x7ff3c00a3c40 0x7ff3c00a6100 unknown :-1 s=CLOSED pgs=217 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:26:06.479 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:06.480+0000 7ff3def29640 1 --2- 172.21.15.114:0/1366471159 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7ff3d8107c70 0x7ff3d819f960 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:26:06.479 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:06.480+0000 7ff3def29640 1 --2- 172.21.15.114:0/1366471159 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7ff3d8105340 0x7ff3d81988f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:26:06.479 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:06.480+0000 7ff3def29640 1 --2- 172.21.15.114:0/1366471159 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7ff3d8102a10 0x7ff3d81983b0 unknown :-1 s=CLOSED pgs=317 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:26:06.479 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:06.480+0000 7ff3def29640 1 -- 172.21.15.114:0/1366471159 >> 172.21.15.114:0/1366471159 conn(0x7ff3d80fc400 msgr2=0x7ff3d81034f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T10:26:06.479 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:06.480+0000 7ff3def29640 1 -- 172.21.15.114:0/1366471159 shutdown_connections 2024-04-23T10:26:06.480 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:06.480+0000 7ff3def29640 1 -- 172.21.15.114:0/1366471159 wait complete. 2024-04-23T10:26:06.490 DEBUG:teuthology.orchestra.run.smithi114:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2024-04-23T10:26:06.492 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 23 10:26:06 smithi119 ceph-mon[32608]: from='client.? 172.21.15.114:0/3883860348' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2024-04-23T10:26:06.492 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 23 10:26:06 smithi119 ceph-mon[32608]: from='client.? 172.21.15.114:0/3878244073' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2024-04-23T10:26:06.492 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 23 10:26:06 smithi119 ceph-mon[32608]: from='client.? 172.21.15.114:0/553273662' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-23T10:26:06.594 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:06.592+0000 7f218cac2640 1 -- 172.21.15.114:0/304696020 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f2188101fc0 msgr2=0x7f2188102440 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T10:26:06.594 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:06.592+0000 7f218cac2640 1 --2- 172.21.15.114:0/304696020 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f2188101fc0 0x7f2188102440 secure :-1 s=READY pgs=318 cs=0 l=1 rev1=1 crypto rx=0x7f217c009900 tx=0x7f217c02a410 comp rx=0 tx=0).stop 2024-04-23T10:26:06.594 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:06.592+0000 7f218cac2640 1 -- 172.21.15.114:0/304696020 shutdown_connections 2024-04-23T10:26:06.594 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:06.592+0000 7f218cac2640 1 --2- 172.21.15.114:0/304696020 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f2188102980 0x7f21881092a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:26:06.594 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:06.592+0000 7f218cac2640 1 --2- 172.21.15.114:0/304696020 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f2188101fc0 0x7f2188102440 unknown :-1 s=CLOSED pgs=318 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:26:06.594 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:06.592+0000 7f218cac2640 1 --2- 172.21.15.114:0/304696020 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f2188100e70 0x7f2188101290 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:26:06.594 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:06.592+0000 7f218cac2640 1 -- 172.21.15.114:0/304696020 >> 172.21.15.114:0/304696020 conn(0x7f21880fc420 msgr2=0x7f21880fe840 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T10:26:06.594 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:06.592+0000 7f218cac2640 1 -- 172.21.15.114:0/304696020 shutdown_connections 2024-04-23T10:26:06.594 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:06.592+0000 7f218cac2640 1 -- 172.21.15.114:0/304696020 wait complete. 2024-04-23T10:26:06.594 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:06.593+0000 7f218cac2640 1 Processor -- start 2024-04-23T10:26:06.594 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:06.593+0000 7f218cac2640 1 -- start start 2024-04-23T10:26:06.594 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:06.593+0000 7f218cac2640 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f2188100e70 0x7f218806b120 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T10:26:06.595 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:06.593+0000 7f218cac2640 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f2188101fc0 0x7f218819cf70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T10:26:06.595 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:06.593+0000 7f218cac2640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f2188102980 0x7f218819f590 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T10:26:06.595 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:06.593+0000 7f218cac2640 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x7f218810ca20 con 0x7f2188102980 2024-04-23T10:26:06.595 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:06.593+0000 7f218cac2640 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 -- 0x7f218810c8a0 con 0x7f2188100e70 2024-04-23T10:26:06.595 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:06.593+0000 7f218cac2640 1 -- --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_getmap magic: 0 -- 0x7f218810cba0 con 0x7f2188101fc0 2024-04-23T10:26:06.595 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:06.593+0000 7f2186d9d640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f2188102980 0x7f218819f590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T10:26:06.595 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:06.593+0000 7f2186d9d640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f2188102980 0x7f218819f590 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.114:58014/0 (socket says 172.21.15.114:58014) 2024-04-23T10:26:06.595 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:06.593+0000 7f2186d9d640 1 -- 172.21.15.114:0/3929921621 learned_addr learned my addr 172.21.15.114:0/3929921621 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-04-23T10:26:06.595 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:06.593+0000 7f218659c640 1 --2- 172.21.15.114:0/3929921621 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f2188100e70 0x7f218806b120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T10:26:06.595 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:06.593+0000 7f2185d9b640 1 --2- 172.21.15.114:0/3929921621 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f2188101fc0 0x7f218819cf70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T10:26:06.595 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:06.593+0000 7f2186d9d640 1 -- 172.21.15.114:0/3929921621 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f2188101fc0 msgr2=0x7f218819cf70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T10:26:06.595 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:06.593+0000 7f2186d9d640 1 --2- 172.21.15.114:0/3929921621 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f2188101fc0 0x7f218819cf70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:26:06.595 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:06.593+0000 7f2186d9d640 1 -- 172.21.15.114:0/3929921621 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f2188100e70 msgr2=0x7f218806b120 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T10:26:06.595 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:06.593+0000 7f2186d9d640 1 --2- 172.21.15.114:0/3929921621 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f2188100e70 0x7f218806b120 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:26:06.595 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:06.593+0000 7f2186d9d640 1 -- 172.21.15.114:0/3929921621 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f218819fea0 con 0x7f2188102980 2024-04-23T10:26:06.596 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:06.593+0000 7f2185d9b640 1 --2- 172.21.15.114:0/3929921621 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f2188101fc0 0x7f218819cf70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-23T10:26:06.596 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:06.593+0000 7f2186d9d640 1 --2- 172.21.15.114:0/3929921621 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f2188102980 0x7f218819f590 secure :-1 s=READY pgs=319 cs=0 l=1 rev1=1 crypto rx=0x7f218000d750 tx=0x7f218000dc10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T10:26:06.596 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:06.594+0000 7f21777fe640 1 -- 172.21.15.114:0/3929921621 <== mon.0 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2180017070 con 0x7f2188102980 2024-04-23T10:26:06.596 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:06.594+0000 7f21777fe640 1 -- 172.21.15.114:0/3929921621 <== mon.0 v2:172.21.15.114:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f21800054d0 con 0x7f2188102980 2024-04-23T10:26:06.596 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:06.594+0000 7f218cac2640 1 -- 172.21.15.114:0/3929921621 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f21881a0130 con 0x7f2188102980 2024-04-23T10:26:06.596 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:06.594+0000 7f21777fe640 1 -- 172.21.15.114:0/3929921621 <== mon.0 v2:172.21.15.114:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2180004c60 con 0x7f2188102980 2024-04-23T10:26:06.596 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:06.594+0000 7f218cac2640 1 -- 172.21.15.114:0/3929921621 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f21881a06b0 con 0x7f2188102980 2024-04-23T10:26:06.596 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:06.595+0000 7f218cac2640 1 -- 172.21.15.114:0/3929921621 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f2188108250 con 0x7f2188102980 2024-04-23T10:26:06.597 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:06.596+0000 7f21777fe640 1 -- 172.21.15.114:0/3929921621 <== mon.0 v2:172.21.15.114:3300/0 4 ==== mgrmap(e 17) ==== 138698+0+0 (secure 0 0 0) 0x7f2180009bd0 con 0x7f2188102980 2024-04-23T10:26:06.597 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:06.596+0000 7f21777fe640 1 --2- 172.21.15.114:0/3929921621 >> [v2:172.21.15.114:6800/465333808,v1:172.21.15.114:6801/465333808] conn(0x7f21700a3c40 0x7f21700a6100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T10:26:06.597 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:06.596+0000 7f218659c640 1 --2- 172.21.15.114:0/3929921621 >> [v2:172.21.15.114:6800/465333808,v1:172.21.15.114:6801/465333808] conn(0x7f21700a3c40 0x7f21700a6100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T10:26:06.597 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:06.597+0000 7f21777fe640 1 -- 172.21.15.114:0/3929921621 <== mon.0 v2:172.21.15.114:3300/0 5 ==== osd_map(89..89 src has 1..89) ==== 8269+0+0 (secure 0 0 0) 0x7f21800d03b0 con 0x7f2188102980 2024-04-23T10:26:06.597 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:06.597+0000 7f218659c640 1 --2- 172.21.15.114:0/3929921621 >> [v2:172.21.15.114:6800/465333808,v1:172.21.15.114:6801/465333808] conn(0x7f21700a3c40 0x7f21700a6100 secure :-1 s=READY pgs=218 cs=0 l=1 rev1=1 crypto rx=0x7f217800a400 tx=0x7f217800d040 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T10:26:06.599 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:06.600+0000 7f21777fe640 1 -- 172.21.15.114:0/3929921621 <== mon.0 v2:172.21.15.114:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f218009cc10 con 0x7f2188102980 2024-04-23T10:26:06.778 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:06.779+0000 7f218cac2640 1 -- 172.21.15.114:0/3929921621 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) -- 0x7f218802d7c0 con 0x7f2188102980 2024-04-23T10:26:06.778 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:06.780+0000 7f21777fe640 1 -- 172.21.15.114:0/3929921621 <== mon.0 v2:172.21.15.114:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v89) ==== 104+0+11 (secure 0 0 0) 0x7f218009c510 con 0x7f2188102980 2024-04-23T10:26:06.779 INFO:teuthology.orchestra.run.smithi114.stdout:pg_num: 32 2024-04-23T10:26:06.781 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:06.782+0000 7f218cac2640 1 -- 172.21.15.114:0/3929921621 >> [v2:172.21.15.114:6800/465333808,v1:172.21.15.114:6801/465333808] conn(0x7f21700a3c40 msgr2=0x7f21700a6100 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T10:26:06.781 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:06.782+0000 7f218cac2640 1 --2- 172.21.15.114:0/3929921621 >> [v2:172.21.15.114:6800/465333808,v1:172.21.15.114:6801/465333808] conn(0x7f21700a3c40 0x7f21700a6100 secure :-1 s=READY pgs=218 cs=0 l=1 rev1=1 crypto rx=0x7f217800a400 tx=0x7f217800d040 comp rx=0 tx=0).stop 2024-04-23T10:26:06.781 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:06.782+0000 7f218cac2640 1 -- 172.21.15.114:0/3929921621 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f2188102980 msgr2=0x7f218819f590 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T10:26:06.781 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:06.782+0000 7f218cac2640 1 --2- 172.21.15.114:0/3929921621 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f2188102980 0x7f218819f590 secure :-1 s=READY pgs=319 cs=0 l=1 rev1=1 crypto rx=0x7f218000d750 tx=0x7f218000dc10 comp rx=0 tx=0).stop 2024-04-23T10:26:06.781 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:06.782+0000 7f218cac2640 1 -- 172.21.15.114:0/3929921621 shutdown_connections 2024-04-23T10:26:06.781 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:06.782+0000 7f218cac2640 1 --2- 172.21.15.114:0/3929921621 >> [v2:172.21.15.114:6800/465333808,v1:172.21.15.114:6801/465333808] conn(0x7f21700a3c40 0x7f21700a6100 unknown :-1 s=CLOSED pgs=218 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:26:06.781 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:06.782+0000 7f218cac2640 1 --2- 172.21.15.114:0/3929921621 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f2188102980 0x7f218819f590 unknown :-1 s=CLOSED pgs=319 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:26:06.781 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:06.782+0000 7f218cac2640 1 --2- 172.21.15.114:0/3929921621 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f2188101fc0 0x7f218819cf70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:26:06.781 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:06.782+0000 7f218cac2640 1 --2- 172.21.15.114:0/3929921621 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f2188100e70 0x7f218806b120 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:26:06.781 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:06.782+0000 7f218cac2640 1 -- 172.21.15.114:0/3929921621 >> 172.21.15.114:0/3929921621 conn(0x7f21880fc420 msgr2=0x7f2188105420 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T10:26:06.782 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:06.782+0000 7f218cac2640 1 -- 172.21.15.114:0/3929921621 shutdown_connections 2024-04-23T10:26:06.782 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:06.782+0000 7f218cac2640 1 -- 172.21.15.114:0/3929921621 wait complete. 2024-04-23T10:26:06.792 DEBUG:teuthology.orchestra.run.smithi114:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2024-04-23T10:26:06.894 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:06.893+0000 7f7fe215d640 1 -- 172.21.15.114:0/3166347187 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f7fdc105400 msgr2=0x7f7fdc1077f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T10:26:06.895 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:06.893+0000 7f7fe215d640 1 --2- 172.21.15.114:0/3166347187 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f7fdc105400 0x7f7fdc1077f0 secure :-1 s=READY pgs=320 cs=0 l=1 rev1=1 crypto rx=0x7f7fd0009900 tx=0x7f7fd002a410 comp rx=0 tx=0).stop 2024-04-23T10:26:06.895 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:06.893+0000 7f7fe215d640 1 -- 172.21.15.114:0/3166347187 shutdown_connections 2024-04-23T10:26:06.895 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:06.893+0000 7f7fe215d640 1 --2- 172.21.15.114:0/3166347187 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f7fdc107d30 0x7f7fdc10a120 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:26:06.895 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:06.893+0000 7f7fe215d640 1 --2- 172.21.15.114:0/3166347187 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f7fdc105400 0x7f7fdc1077f0 unknown :-1 s=CLOSED pgs=320 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:26:06.895 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:06.893+0000 7f7fe215d640 1 --2- 172.21.15.114:0/3166347187 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f7fdc102ad0 0x7f7fdc104ec0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:26:06.895 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:06.893+0000 7f7fe215d640 1 -- 172.21.15.114:0/3166347187 >> 172.21.15.114:0/3166347187 conn(0x7f7fdc0fc420 msgr2=0x7f7fdc0fe860 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T10:26:06.895 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:06.893+0000 7f7fe215d640 1 -- 172.21.15.114:0/3166347187 shutdown_connections 2024-04-23T10:26:06.895 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:06.894+0000 7f7fe215d640 1 -- 172.21.15.114:0/3166347187 wait complete. 2024-04-23T10:26:06.895 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:06.894+0000 7f7fe215d640 1 Processor -- start 2024-04-23T10:26:06.895 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:06.894+0000 7f7fe215d640 1 -- start start 2024-04-23T10:26:06.895 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:06.894+0000 7f7fe215d640 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f7fdc102ad0 0x7f7fdc198380 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T10:26:06.895 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:06.894+0000 7f7fe215d640 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f7fdc105400 0x7f7fdc1988c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T10:26:06.895 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:06.894+0000 7f7fe215d640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f7fdc107d30 0x7f7fdc19f930 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T10:26:06.895 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:06.894+0000 7f7fe215d640 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x7f7fdc10dc00 con 0x7f7fdc107d30 2024-04-23T10:26:06.896 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:06.894+0000 7f7fe215d640 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 -- 0x7f7fdc10da80 con 0x7f7fdc102ad0 2024-04-23T10:26:06.896 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:06.894+0000 7f7fe215d640 1 -- --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_getmap magic: 0 -- 0x7f7fdc10dd80 con 0x7f7fdc105400 2024-04-23T10:26:06.896 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:06.894+0000 7f7fdbfff640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f7fdc107d30 0x7f7fdc19f930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T10:26:06.896 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:06.894+0000 7f7fdbfff640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f7fdc107d30 0x7f7fdc19f930 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.114:58020/0 (socket says 172.21.15.114:58020) 2024-04-23T10:26:06.896 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:06.894+0000 7f7fdbfff640 1 -- 172.21.15.114:0/3533348400 learned_addr learned my addr 172.21.15.114:0/3533348400 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-04-23T10:26:06.896 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:06.895+0000 7f7fdaffd640 1 --2- 172.21.15.114:0/3533348400 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f7fdc105400 0x7f7fdc1988c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T10:26:06.896 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:06.895+0000 7f7fdb7fe640 1 --2- 172.21.15.114:0/3533348400 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f7fdc102ad0 0x7f7fdc198380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T10:26:06.896 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:06.895+0000 7f7fdbfff640 1 -- 172.21.15.114:0/3533348400 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f7fdc105400 msgr2=0x7f7fdc1988c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T10:26:06.896 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:06.895+0000 7f7fdbfff640 1 --2- 172.21.15.114:0/3533348400 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f7fdc105400 0x7f7fdc1988c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:26:06.896 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:06.895+0000 7f7fdbfff640 1 -- 172.21.15.114:0/3533348400 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f7fdc102ad0 msgr2=0x7f7fdc198380 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T10:26:06.896 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:06.895+0000 7f7fdbfff640 1 --2- 172.21.15.114:0/3533348400 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f7fdc102ad0 0x7f7fdc198380 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:26:06.896 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:06.895+0000 7f7fdbfff640 1 -- 172.21.15.114:0/3533348400 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f7fdc19ffa0 con 0x7f7fdc107d30 2024-04-23T10:26:06.896 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:06.895+0000 7f7fdbfff640 1 --2- 172.21.15.114:0/3533348400 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f7fdc107d30 0x7f7fdc19f930 secure :-1 s=READY pgs=321 cs=0 l=1 rev1=1 crypto rx=0x7f7fd400b970 tx=0x7f7fd400be30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T10:26:06.896 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:06.895+0000 7f7fd8ff9640 1 -- 172.21.15.114:0/3533348400 <== mon.0 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7fd4019070 con 0x7f7fdc107d30 2024-04-23T10:26:06.897 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:06.895+0000 7f7fd8ff9640 1 -- 172.21.15.114:0/3533348400 <== mon.0 v2:172.21.15.114:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f7fd4014440 con 0x7f7fdc107d30 2024-04-23T10:26:06.897 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:06.895+0000 7f7fe215d640 1 -- 172.21.15.114:0/3533348400 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f7fdc1a0290 con 0x7f7fdc107d30 2024-04-23T10:26:06.897 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:06.895+0000 7f7fd8ff9640 1 -- 172.21.15.114:0/3533348400 <== mon.0 v2:172.21.15.114:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7fd40075d0 con 0x7f7fdc107d30 2024-04-23T10:26:06.897 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:06.895+0000 7f7fe215d640 1 -- 172.21.15.114:0/3533348400 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f7fdc1a07e0 con 0x7f7fdc107d30 2024-04-23T10:26:06.898 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:06.896+0000 7f7fe215d640 1 -- 172.21.15.114:0/3533348400 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f7fdc0fdf10 con 0x7f7fdc107d30 2024-04-23T10:26:06.898 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:06.897+0000 7f7fd8ff9640 1 -- 172.21.15.114:0/3533348400 <== mon.0 v2:172.21.15.114:3300/0 4 ==== mgrmap(e 17) ==== 138698+0+0 (secure 0 0 0) 0x7f7fd401c070 con 0x7f7fdc107d30 2024-04-23T10:26:06.898 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:06.898+0000 7f7fd8ff9640 1 --2- 172.21.15.114:0/3533348400 >> [v2:172.21.15.114:6800/465333808,v1:172.21.15.114:6801/465333808] conn(0x7f7fc40a3c40 0x7f7fc40a6100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T10:26:06.898 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:06.898+0000 7f7fdb7fe640 1 --2- 172.21.15.114:0/3533348400 >> [v2:172.21.15.114:6800/465333808,v1:172.21.15.114:6801/465333808] conn(0x7f7fc40a3c40 0x7f7fc40a6100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T10:26:06.898 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:06.898+0000 7f7fd8ff9640 1 -- 172.21.15.114:0/3533348400 <== mon.0 v2:172.21.15.114:3300/0 5 ==== osd_map(89..89 src has 1..89) ==== 8269+0+0 (secure 0 0 0) 0x7f7fd40cfa20 con 0x7f7fdc107d30 2024-04-23T10:26:06.898 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:06.898+0000 7f7fdb7fe640 1 --2- 172.21.15.114:0/3533348400 >> [v2:172.21.15.114:6800/465333808,v1:172.21.15.114:6801/465333808] conn(0x7f7fc40a3c40 0x7f7fc40a6100 secure :-1 s=READY pgs=219 cs=0 l=1 rev1=1 crypto rx=0x7f7fcc00a810 tx=0x7f7fcc008040 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T10:26:06.902 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:06.903+0000 7f7fd8ff9640 1 -- 172.21.15.114:0/3533348400 <== mon.0 v2:172.21.15.114:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f7fd409bd40 con 0x7f7fdc107d30 2024-04-23T10:26:07.083 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:07.084+0000 7f7fe215d640 1 -- 172.21.15.114:0/3533348400 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) -- 0x7f7fdc062b50 con 0x7f7fdc107d30 2024-04-23T10:26:07.084 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:07.085+0000 7f7fd8ff9640 1 -- 172.21.15.114:0/3533348400 <== mon.0 v2:172.21.15.114:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v89) ==== 100+0+11 (secure 0 0 0) 0x7f7fd40a0040 con 0x7f7fdc107d30 2024-04-23T10:26:07.084 INFO:teuthology.orchestra.run.smithi114.stdout:pg_num: 64 2024-04-23T10:26:07.086 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:07.087+0000 7f7fe215d640 1 -- 172.21.15.114:0/3533348400 >> [v2:172.21.15.114:6800/465333808,v1:172.21.15.114:6801/465333808] conn(0x7f7fc40a3c40 msgr2=0x7f7fc40a6100 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T10:26:07.086 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:07.087+0000 7f7fe215d640 1 --2- 172.21.15.114:0/3533348400 >> [v2:172.21.15.114:6800/465333808,v1:172.21.15.114:6801/465333808] conn(0x7f7fc40a3c40 0x7f7fc40a6100 secure :-1 s=READY pgs=219 cs=0 l=1 rev1=1 crypto rx=0x7f7fcc00a810 tx=0x7f7fcc008040 comp rx=0 tx=0).stop 2024-04-23T10:26:07.086 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:07.087+0000 7f7fe215d640 1 -- 172.21.15.114:0/3533348400 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f7fdc107d30 msgr2=0x7f7fdc19f930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T10:26:07.086 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:07.087+0000 7f7fe215d640 1 --2- 172.21.15.114:0/3533348400 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f7fdc107d30 0x7f7fdc19f930 secure :-1 s=READY pgs=321 cs=0 l=1 rev1=1 crypto rx=0x7f7fd400b970 tx=0x7f7fd400be30 comp rx=0 tx=0).stop 2024-04-23T10:26:07.086 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:07.087+0000 7f7fe215d640 1 -- 172.21.15.114:0/3533348400 shutdown_connections 2024-04-23T10:26:07.086 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:07.087+0000 7f7fe215d640 1 --2- 172.21.15.114:0/3533348400 >> [v2:172.21.15.114:6800/465333808,v1:172.21.15.114:6801/465333808] conn(0x7f7fc40a3c40 0x7f7fc40a6100 unknown :-1 s=CLOSED pgs=219 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:26:07.086 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:07.087+0000 7f7fe215d640 1 --2- 172.21.15.114:0/3533348400 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f7fdc107d30 0x7f7fdc19f930 unknown :-1 s=CLOSED pgs=321 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:26:07.086 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:07.087+0000 7f7fe215d640 1 --2- 172.21.15.114:0/3533348400 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f7fdc105400 0x7f7fdc1988c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:26:07.087 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:07.087+0000 7f7fe215d640 1 --2- 172.21.15.114:0/3533348400 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f7fdc102ad0 0x7f7fdc198380 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:26:07.087 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:07.087+0000 7f7fe215d640 1 -- 172.21.15.114:0/3533348400 >> 172.21.15.114:0/3533348400 conn(0x7f7fdc0fc420 msgr2=0x7f7fdc104960 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T10:26:07.087 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:07.087+0000 7f7fe215d640 1 -- 172.21.15.114:0/3533348400 shutdown_connections 2024-04-23T10:26:07.087 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:07.087+0000 7f7fe215d640 1 -- 172.21.15.114:0/3533348400 wait complete. 2024-04-23T10:26:07.097 DEBUG:teuthology.orchestra.run.smithi114:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-23T10:26:07.195 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:07.196+0000 7f5fd8235640 1 -- 172.21.15.114:0/3488474455 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f5fd01031b0 msgr2=0x7f5fd0103630 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T10:26:07.195 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:07.196+0000 7f5fd8235640 1 --2- 172.21.15.114:0/3488474455 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f5fd01031b0 0x7f5fd0103630 secure :-1 s=READY pgs=322 cs=0 l=1 rev1=1 crypto rx=0x7f5fc0009900 tx=0x7f5fc002a410 comp rx=0 tx=0).stop 2024-04-23T10:26:07.195 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:07.196+0000 7f5fd8235640 1 -- 172.21.15.114:0/3488474455 shutdown_connections 2024-04-23T10:26:07.195 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:07.196+0000 7f5fd8235640 1 --2- 172.21.15.114:0/3488474455 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f5fd0103b70 0x7f5fd010a400 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:26:07.195 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:07.196+0000 7f5fd8235640 1 --2- 172.21.15.114:0/3488474455 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f5fd01031b0 0x7f5fd0103630 unknown :-1 s=CLOSED pgs=322 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:26:07.196 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:07.196+0000 7f5fd8235640 1 --2- 172.21.15.114:0/3488474455 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f5fd0101fc0 0x7f5fd01023e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:26:07.196 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:07.196+0000 7f5fd8235640 1 -- 172.21.15.114:0/3488474455 >> 172.21.15.114:0/3488474455 conn(0x7f5fd00fd530 msgr2=0x7f5fd00ff970 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T10:26:07.196 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:07.196+0000 7f5fd8235640 1 -- 172.21.15.114:0/3488474455 shutdown_connections 2024-04-23T10:26:07.196 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:07.196+0000 7f5fd8235640 1 -- 172.21.15.114:0/3488474455 wait complete. 2024-04-23T10:26:07.198 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:07.197+0000 7f5fd8235640 1 Processor -- start 2024-04-23T10:26:07.198 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:07.197+0000 7f5fd8235640 1 -- start start 2024-04-23T10:26:07.198 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:07.197+0000 7f5fd8235640 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f5fd0101fc0 0x7f5fd01980d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T10:26:07.198 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:07.197+0000 7f5fd8235640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f5fd01031b0 0x7f5fd0198610 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T10:26:07.198 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:07.197+0000 7f5fd8235640 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f5fd0103b70 0x7f5fd019f680 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T10:26:07.198 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:07.197+0000 7f5fd8235640 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x7f5fd010db00 con 0x7f5fd01031b0 2024-04-23T10:26:07.198 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:07.197+0000 7f5fd8235640 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 -- 0x7f5fd010d980 con 0x7f5fd0101fc0 2024-04-23T10:26:07.198 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:07.197+0000 7f5fd8235640 1 -- --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_getmap magic: 0 -- 0x7f5fd010dc80 con 0x7f5fd0103b70 2024-04-23T10:26:07.199 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:07.197+0000 7f5fd57d0640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f5fd01031b0 0x7f5fd0198610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T10:26:07.199 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:07.197+0000 7f5fd57d0640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f5fd01031b0 0x7f5fd0198610 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.114:58038/0 (socket says 172.21.15.114:58038) 2024-04-23T10:26:07.199 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:07.197+0000 7f5fd57d0640 1 -- 172.21.15.114:0/440960242 learned_addr learned my addr 172.21.15.114:0/440960242 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-04-23T10:26:07.199 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:07.197+0000 7f5fd5fd1640 1 --2- 172.21.15.114:0/440960242 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f5fd0101fc0 0x7f5fd01980d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T10:26:07.199 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:07.197+0000 7f5fd57d0640 1 -- 172.21.15.114:0/440960242 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f5fd0103b70 msgr2=0x7f5fd019f680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T10:26:07.199 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:07.197+0000 7f5fd57d0640 1 --2- 172.21.15.114:0/440960242 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f5fd0103b70 0x7f5fd019f680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:26:07.199 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:07.197+0000 7f5fd57d0640 1 -- 172.21.15.114:0/440960242 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f5fd0101fc0 msgr2=0x7f5fd01980d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T10:26:07.199 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:07.197+0000 7f5fd57d0640 1 --2- 172.21.15.114:0/440960242 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f5fd0101fc0 0x7f5fd01980d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:26:07.199 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:07.197+0000 7f5fd57d0640 1 -- 172.21.15.114:0/440960242 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f5fd019fcf0 con 0x7f5fd01031b0 2024-04-23T10:26:07.199 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:07.197+0000 7f5fd5fd1640 1 --2- 172.21.15.114:0/440960242 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f5fd0101fc0 0x7f5fd01980d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-23T10:26:07.199 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:07.197+0000 7f5fd57d0640 1 --2- 172.21.15.114:0/440960242 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f5fd01031b0 0x7f5fd0198610 secure :-1 s=READY pgs=323 cs=0 l=1 rev1=1 crypto rx=0x7f5fc0031500 tx=0x7f5fc00043c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T10:26:07.199 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:07.198+0000 7f5fc6ffd640 1 -- 172.21.15.114:0/440960242 <== mon.0 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5fc003f030 con 0x7f5fd01031b0 2024-04-23T10:26:07.199 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:07.198+0000 7f5fc6ffd640 1 -- 172.21.15.114:0/440960242 <== mon.0 v2:172.21.15.114:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f5fc0003800 con 0x7f5fd01031b0 2024-04-23T10:26:07.199 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:07.198+0000 7f5fd8235640 1 -- 172.21.15.114:0/440960242 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f5fd019ff80 con 0x7f5fd01031b0 2024-04-23T10:26:07.200 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:07.198+0000 7f5fc6ffd640 1 -- 172.21.15.114:0/440960242 <== mon.0 v2:172.21.15.114:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5fc0035070 con 0x7f5fd01031b0 2024-04-23T10:26:07.200 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:07.198+0000 7f5fd8235640 1 -- 172.21.15.114:0/440960242 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f5fd01a0440 con 0x7f5fd01031b0 2024-04-23T10:26:07.200 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:07.199+0000 7f5fd8235640 1 -- 172.21.15.114:0/440960242 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f5fd0109280 con 0x7f5fd01031b0 2024-04-23T10:26:07.200 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:07.200+0000 7f5fc6ffd640 1 -- 172.21.15.114:0/440960242 <== mon.0 v2:172.21.15.114:3300/0 4 ==== mgrmap(e 17) ==== 138698+0+0 (secure 0 0 0) 0x7f5fc00039b0 con 0x7f5fd01031b0 2024-04-23T10:26:07.201 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:07.200+0000 7f5fc6ffd640 1 --2- 172.21.15.114:0/440960242 >> [v2:172.21.15.114:6800/465333808,v1:172.21.15.114:6801/465333808] conn(0x7f5fbc0a3cc0 0x7f5fbc0a6180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T10:26:07.201 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:07.200+0000 7f5fd5fd1640 1 --2- 172.21.15.114:0/440960242 >> [v2:172.21.15.114:6800/465333808,v1:172.21.15.114:6801/465333808] conn(0x7f5fbc0a3cc0 0x7f5fbc0a6180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T10:26:07.201 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:07.200+0000 7f5fc6ffd640 1 -- 172.21.15.114:0/440960242 <== mon.0 v2:172.21.15.114:3300/0 5 ==== osd_map(89..89 src has 1..89) ==== 8269+0+0 (secure 0 0 0) 0x7f5fc00f0ff0 con 0x7f5fd01031b0 2024-04-23T10:26:07.201 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:07.201+0000 7f5fd5fd1640 1 --2- 172.21.15.114:0/440960242 >> [v2:172.21.15.114:6800/465333808,v1:172.21.15.114:6801/465333808] conn(0x7f5fbc0a3cc0 0x7f5fbc0a6180 secure :-1 s=READY pgs=220 cs=0 l=1 rev1=1 crypto rx=0x7f5fd0103010 tx=0x7f5fcc005db0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T10:26:07.203 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:07.205+0000 7f5fc6ffd640 1 -- 172.21.15.114:0/440960242 <== mon.0 v2:172.21.15.114:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f5fc00bd850 con 0x7f5fd01031b0 2024-04-23T10:26:07.328 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 23 10:26:07 smithi204 ceph-mon[32649]: pgmap v329: 97 pgs: 97 active+clean; 641 KiB data, 371 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-23T10:26:07.328 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 23 10:26:07 smithi204 ceph-mon[32649]: from='client.? 172.21.15.114:0/847073594' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-23T10:26:07.328 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 23 10:26:07 smithi204 ceph-mon[32649]: from='client.? 172.21.15.114:0/1366471159' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-04-23T10:26:07.328 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 23 10:26:07 smithi204 ceph-mon[32649]: from='client.? 172.21.15.114:0/3929921621' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2024-04-23T10:26:07.419 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:07.421+0000 7f5fd8235640 1 -- 172.21.15.114:0/440960242 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f5fd002d830 con 0x7f5fd01031b0 2024-04-23T10:26:07.420 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:07.421+0000 7f5fc6ffd640 1 -- 172.21.15.114:0/440960242 <== mon.0 v2:172.21.15.114:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 20 v20) ==== 94+0+11884 (secure 0 0 0) 0x7f5fc00bd150 con 0x7f5fd01031b0 2024-04-23T10:26:07.420 INFO:teuthology.orchestra.run.smithi114.stdout: 2024-04-23T10:26:07.420 INFO:teuthology.orchestra.run.smithi114.stdout:{"epoch":20,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14547,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.114:6835/663493580","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6834","nonce":663493580},{"type":"v1","addr":"172.21.15.114:6835","nonce":663493580}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14565,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.114:6837/1439252154","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6836","nonce":1439252154},{"type":"v1","addr":"172.21.15.114:6837","nonce":1439252154}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5},{"gid":14634,"name":"g","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.114:6839/3849796939","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6838","nonce":3849796939},{"type":"v1","addr":"172.21.15.114:6839","nonce":3849796939}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":11},{"gid":14646,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.114:6841/3544523891","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6840","nonce":3544523891},{"type":"v1","addr":"172.21.15.114:6841","nonce":3544523891}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":12},{"gid":24385,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.204:6833/3866561754","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6832","nonce":3866561754},{"type":"v1","addr":"172.21.15.204:6833","nonce":3866561754}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2},{"gid":24427,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.204:6835/2533793172","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6834","nonce":2533793172},{"type":"v1","addr":"172.21.15.204:6835","nonce":2533793172}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":7}],"filesystems":[{"mdsmap":{"epoch":20,"flags":50,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-04-23T10:24:43.383158+0000","modified":"2024-04-23T10:24:48.315128+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":3,"in":[0,1,2],"up":{"mds_0":24545,"mds_1":24509,"mds_2":24479},"failed":[],"damaged":[],"stopped":[],"info":{"gid_24439":{"gid":24439,"name":"c","rank":2,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.204:6837/513822179","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6836","nonce":513822179},{"type":"v1","addr":"172.21.15.204:6837","nonce":513822179}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24443":{"gid":24443,"name":"k","rank":1,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.119:6833/783244660","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6832","nonce":783244660},{"type":"v1","addr":"172.21.15.119:6833","nonce":783244660}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24463":{"gid":24463,"name":"i","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.204:6839/3222658893","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6838","nonce":3222658893},{"type":"v1","addr":"172.21.15.204:6839","nonce":3222658893}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24479":{"gid":24479,"name":"e","rank":2,"incarnation":19,"state":"up:active","state_seq":9,"addr":"172.21.15.119:6835/2033916709","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6834","nonce":2033916709},{"type":"v1","addr":"172.21.15.119:6835","nonce":2033916709}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24509":{"gid":24509,"name":"h","rank":1,"incarnation":17,"state":"up:active","state_seq":6,"addr":"172.21.15.119:6837/1872363128","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6836","nonce":1872363128},{"type":"v1","addr":"172.21.15.119:6837","nonce":1872363128}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24545":{"gid":24545,"name":"b","rank":0,"incarnation":14,"state":"up:active","state_seq":3,"addr":"172.21.15.119:6839/937297340","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6838","nonce":937297340},{"type":"v1","addr":"172.21.15.119:6839","nonce":937297340}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":24545,"qdb_cluster":[24509,24545,24479]},"id":1}]} 2024-04-23T10:26:07.421 INFO:teuthology.orchestra.run.smithi114.stderr:dumped fsmap epoch 20 2024-04-23T10:26:07.422 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:07.423+0000 7f5fd8235640 1 -- 172.21.15.114:0/440960242 >> [v2:172.21.15.114:6800/465333808,v1:172.21.15.114:6801/465333808] conn(0x7f5fbc0a3cc0 msgr2=0x7f5fbc0a6180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T10:26:07.423 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:07.423+0000 7f5fd8235640 1 --2- 172.21.15.114:0/440960242 >> [v2:172.21.15.114:6800/465333808,v1:172.21.15.114:6801/465333808] conn(0x7f5fbc0a3cc0 0x7f5fbc0a6180 secure :-1 s=READY pgs=220 cs=0 l=1 rev1=1 crypto rx=0x7f5fd0103010 tx=0x7f5fcc005db0 comp rx=0 tx=0).stop 2024-04-23T10:26:07.423 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:07.423+0000 7f5fd8235640 1 -- 172.21.15.114:0/440960242 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f5fd01031b0 msgr2=0x7f5fd0198610 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T10:26:07.423 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:07.423+0000 7f5fd8235640 1 --2- 172.21.15.114:0/440960242 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f5fd01031b0 0x7f5fd0198610 secure :-1 s=READY pgs=323 cs=0 l=1 rev1=1 crypto rx=0x7f5fc0031500 tx=0x7f5fc00043c0 comp rx=0 tx=0).stop 2024-04-23T10:26:07.423 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:07.423+0000 7f5fd8235640 1 -- 172.21.15.114:0/440960242 shutdown_connections 2024-04-23T10:26:07.423 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:07.423+0000 7f5fd8235640 1 --2- 172.21.15.114:0/440960242 >> [v2:172.21.15.114:6800/465333808,v1:172.21.15.114:6801/465333808] conn(0x7f5fbc0a3cc0 0x7f5fbc0a6180 unknown :-1 s=CLOSED pgs=220 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:26:07.423 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:07.423+0000 7f5fd8235640 1 --2- 172.21.15.114:0/440960242 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f5fd0103b70 0x7f5fd019f680 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:26:07.423 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:07.423+0000 7f5fd8235640 1 --2- 172.21.15.114:0/440960242 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f5fd01031b0 0x7f5fd0198610 unknown :-1 s=CLOSED pgs=323 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:26:07.423 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:07.423+0000 7f5fd8235640 1 --2- 172.21.15.114:0/440960242 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f5fd0101fc0 0x7f5fd01980d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:26:07.423 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:07.423+0000 7f5fd8235640 1 -- 172.21.15.114:0/440960242 >> 172.21.15.114:0/440960242 conn(0x7f5fd00fd530 msgr2=0x7f5fd0106450 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T10:26:07.423 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:07.423+0000 7f5fd8235640 1 -- 172.21.15.114:0/440960242 shutdown_connections 2024-04-23T10:26:07.423 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:07.423+0000 7f5fd8235640 1 -- 172.21.15.114:0/440960242 wait complete. 2024-04-23T10:26:07.432 INFO:journalctl@ceph.mon.a.smithi114.stdout:Apr 23 10:26:07 smithi114 ceph-mon[29255]: pgmap v329: 97 pgs: 97 active+clean; 641 KiB data, 371 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-23T10:26:07.433 INFO:journalctl@ceph.mon.a.smithi114.stdout:Apr 23 10:26:07 smithi114 ceph-mon[29255]: from='client.? 172.21.15.114:0/847073594' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-23T10:26:07.433 INFO:journalctl@ceph.mon.a.smithi114.stdout:Apr 23 10:26:07 smithi114 ceph-mon[29255]: from='client.? 172.21.15.114:0/1366471159' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-04-23T10:26:07.433 INFO:journalctl@ceph.mon.a.smithi114.stdout:Apr 23 10:26:07 smithi114 ceph-mon[29255]: from='client.? 172.21.15.114:0/3929921621' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2024-04-23T10:26:07.434 DEBUG:teuthology.orchestra.run.smithi114:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-04-23T10:26:07.491 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 23 10:26:07 smithi119 ceph-mon[32608]: pgmap v329: 97 pgs: 97 active+clean; 641 KiB data, 371 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-23T10:26:07.492 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 23 10:26:07 smithi119 ceph-mon[32608]: from='client.? 172.21.15.114:0/847073594' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-23T10:26:07.492 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 23 10:26:07 smithi119 ceph-mon[32608]: from='client.? 172.21.15.114:0/1366471159' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-04-23T10:26:07.492 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 23 10:26:07 smithi119 ceph-mon[32608]: from='client.? 172.21.15.114:0/3929921621' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2024-04-23T10:26:07.532 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:07.533+0000 7f3b4687c640 1 -- 172.21.15.114:0/2924404053 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f3b40101fa0 msgr2=0x7f3b401023c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T10:26:07.532 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:07.533+0000 7f3b4687c640 1 --2- 172.21.15.114:0/2924404053 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f3b40101fa0 0x7f3b401023c0 secure :-1 s=READY pgs=324 cs=0 l=1 rev1=1 crypto rx=0x7f3b30009900 tx=0x7f3b3002a410 comp rx=0 tx=0).stop 2024-04-23T10:26:07.532 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:07.533+0000 7f3b4687c640 1 -- 172.21.15.114:0/2924404053 shutdown_connections 2024-04-23T10:26:07.532 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:07.533+0000 7f3b4687c640 1 --2- 172.21.15.114:0/2924404053 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f3b40103b50 0x7f3b4010a3e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:26:07.532 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:07.533+0000 7f3b4687c640 1 --2- 172.21.15.114:0/2924404053 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f3b40103190 0x7f3b40103610 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:26:07.533 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:07.533+0000 7f3b4687c640 1 --2- 172.21.15.114:0/2924404053 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f3b40101fa0 0x7f3b401023c0 unknown :-1 s=CLOSED pgs=324 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:26:07.533 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:07.533+0000 7f3b4687c640 1 -- 172.21.15.114:0/2924404053 >> 172.21.15.114:0/2924404053 conn(0x7f3b400fd530 msgr2=0x7f3b400ff950 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T10:26:07.533 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:07.533+0000 7f3b4687c640 1 -- 172.21.15.114:0/2924404053 shutdown_connections 2024-04-23T10:26:07.533 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:07.533+0000 7f3b4687c640 1 -- 172.21.15.114:0/2924404053 wait complete. 2024-04-23T10:26:07.533 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:07.533+0000 7f3b4687c640 1 Processor -- start 2024-04-23T10:26:07.533 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:07.533+0000 7f3b4687c640 1 -- start start 2024-04-23T10:26:07.535 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:07.534+0000 7f3b4687c640 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f3b40101fa0 0x7f3b40198130 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T10:26:07.535 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:07.534+0000 7f3b4687c640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f3b40103190 0x7f3b40198670 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T10:26:07.535 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:07.534+0000 7f3b4687c640 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f3b40103b50 0x7f3b4019f6e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T10:26:07.536 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:07.534+0000 7f3b4687c640 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x7f3b4010dae0 con 0x7f3b40103190 2024-04-23T10:26:07.536 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:07.534+0000 7f3b4687c640 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 -- 0x7f3b4010d960 con 0x7f3b40101fa0 2024-04-23T10:26:07.536 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:07.534+0000 7f3b4687c640 1 -- --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_getmap magic: 0 -- 0x7f3b4010dc60 con 0x7f3b40103b50 2024-04-23T10:26:07.536 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:07.534+0000 7f3b3f7fe640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f3b40103190 0x7f3b40198670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T10:26:07.536 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:07.534+0000 7f3b3f7fe640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f3b40103190 0x7f3b40198670 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.114:58056/0 (socket says 172.21.15.114:58056) 2024-04-23T10:26:07.536 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:07.534+0000 7f3b3f7fe640 1 -- 172.21.15.114:0/872498497 learned_addr learned my addr 172.21.15.114:0/872498497 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-04-23T10:26:07.536 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:07.534+0000 7f3b3ffff640 1 --2- 172.21.15.114:0/872498497 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f3b40101fa0 0x7f3b40198130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T10:26:07.536 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:07.534+0000 7f3b44e19640 1 --2- 172.21.15.114:0/872498497 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f3b40103b50 0x7f3b4019f6e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T10:26:07.536 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:07.534+0000 7f3b3f7fe640 1 -- 172.21.15.114:0/872498497 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f3b40103b50 msgr2=0x7f3b4019f6e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T10:26:07.536 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:07.534+0000 7f3b3f7fe640 1 --2- 172.21.15.114:0/872498497 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f3b40103b50 0x7f3b4019f6e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:26:07.536 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:07.534+0000 7f3b3f7fe640 1 -- 172.21.15.114:0/872498497 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f3b40101fa0 msgr2=0x7f3b40198130 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T10:26:07.536 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:07.534+0000 7f3b3f7fe640 1 --2- 172.21.15.114:0/872498497 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f3b40101fa0 0x7f3b40198130 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:26:07.536 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:07.534+0000 7f3b3f7fe640 1 -- 172.21.15.114:0/872498497 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f3b4019fd50 con 0x7f3b40103190 2024-04-23T10:26:07.536 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:07.534+0000 7f3b44e19640 1 --2- 172.21.15.114:0/872498497 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f3b40103b50 0x7f3b4019f6e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-23T10:26:07.537 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:07.534+0000 7f3b3f7fe640 1 --2- 172.21.15.114:0/872498497 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f3b40103190 0x7f3b40198670 secure :-1 s=READY pgs=325 cs=0 l=1 rev1=1 crypto rx=0x7f3b3400c920 tx=0x7f3b3400cde0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T10:26:07.537 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:07.534+0000 7f3b3d7fa640 1 -- 172.21.15.114:0/872498497 <== mon.0 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3b34007c70 con 0x7f3b40103190 2024-04-23T10:26:07.537 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:07.534+0000 7f3b4687c640 1 -- 172.21.15.114:0/872498497 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f3b4019ffe0 con 0x7f3b40103190 2024-04-23T10:26:07.537 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:07.534+0000 7f3b3d7fa640 1 -- 172.21.15.114:0/872498497 <== mon.0 v2:172.21.15.114:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f3b34007e10 con 0x7f3b40103190 2024-04-23T10:26:07.537 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:07.534+0000 7f3b3d7fa640 1 -- 172.21.15.114:0/872498497 <== mon.0 v2:172.21.15.114:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3b34018820 con 0x7f3b40103190 2024-04-23T10:26:07.537 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:07.534+0000 7f3b4687c640 1 -- 172.21.15.114:0/872498497 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f3b401a0550 con 0x7f3b40103190 2024-04-23T10:26:07.537 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:07.535+0000 7f3b4687c640 1 -- 172.21.15.114:0/872498497 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f3b401092c0 con 0x7f3b40103190 2024-04-23T10:26:07.537 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:07.537+0000 7f3b3d7fa640 1 -- 172.21.15.114:0/872498497 <== mon.0 v2:172.21.15.114:3300/0 4 ==== mgrmap(e 17) ==== 138698+0+0 (secure 0 0 0) 0x7f3b3400f450 con 0x7f3b40103190 2024-04-23T10:26:07.538 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:07.537+0000 7f3b3d7fa640 1 --2- 172.21.15.114:0/872498497 >> [v2:172.21.15.114:6800/465333808,v1:172.21.15.114:6801/465333808] conn(0x7f3b280a3c40 0x7f3b280a6100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T10:26:07.538 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:07.538+0000 7f3b3ffff640 1 --2- 172.21.15.114:0/872498497 >> [v2:172.21.15.114:6800/465333808,v1:172.21.15.114:6801/465333808] conn(0x7f3b280a3c40 0x7f3b280a6100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T10:26:07.538 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:07.538+0000 7f3b3d7fa640 1 -- 172.21.15.114:0/872498497 <== mon.0 v2:172.21.15.114:3300/0 5 ==== osd_map(89..89 src has 1..89) ==== 8269+0+0 (secure 0 0 0) 0x7f3b340d0940 con 0x7f3b40103190 2024-04-23T10:26:07.538 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:07.538+0000 7f3b3ffff640 1 --2- 172.21.15.114:0/872498497 >> [v2:172.21.15.114:6800/465333808,v1:172.21.15.114:6801/465333808] conn(0x7f3b280a3c40 0x7f3b280a6100 secure :-1 s=READY pgs=221 cs=0 l=1 rev1=1 crypto rx=0x7f3b30009230 tx=0x7f3b300314a0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T10:26:07.540 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:07.541+0000 7f3b3d7fa640 1 -- 172.21.15.114:0/872498497 <== mon.0 v2:172.21.15.114:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f3b34014030 con 0x7f3b40103190 2024-04-23T10:26:07.717 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:07.719+0000 7f3b4687c640 1 -- 172.21.15.114:0/872498497 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f3b4002d7c0 con 0x7f3b40103190 2024-04-23T10:26:07.719 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:07.720+0000 7f3b3d7fa640 1 -- 172.21.15.114:0/872498497 <== mon.0 v2:172.21.15.114:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v89) ==== 74+0+22818 (secure 0 0 0) 0x7f3b3409d230 con 0x7f3b40103190 2024-04-23T10:26:07.719 INFO:teuthology.orchestra.run.smithi114.stdout: 2024-04-23T10:26:07.719 INFO:teuthology.orchestra.run.smithi114.stdout:{"epoch":89,"fsid":"aa240572-0159-11ef-bc93-c7b262605968","created":"2024-04-23T10:16:53.351136+0000","modified":"2024-04-23T10:25:47.573901+0000","last_up_change":"2024-04-23T10:23:54.153269+0000","last_in_change":"2024-04-23T10:23:32.963085+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":26,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-04-23T10:20:11.970297+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":12,"score_stable":12,"optimal_score":0.25,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-04-23T10:24:41.218540+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"86","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.8799999952316284,"score_stable":1.8799999952316284,"optimal_score":1,"raw_score_acting":1.8799999952316284,"raw_score_stable":1.8799999952316284,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-04-23T10:24:42.341799+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"87","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"fd64adde-9445-45a4-a171-ef656e2c5709","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":88,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6802","nonce":736366878},{"type":"v1","addr":"172.21.15.114:6803","nonce":736366878}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6804","nonce":736366878},{"type":"v1","addr":"172.21.15.114:6805","nonce":736366878}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6808","nonce":736366878},{"type":"v1","addr":"172.21.15.114:6809","nonce":736366878}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6806","nonce":736366878},{"type":"v1","addr":"172.21.15.114:6807","nonce":736366878}]},"public_addr":"172.21.15.114:6803/736366878","cluster_addr":"172.21.15.114:6805/736366878","heartbeat_back_addr":"172.21.15.114:6809/736366878","heartbeat_front_addr":"172.21.15.114:6807/736366878","state":["exists","up"]},{"osd":1,"uuid":"e279a1e6-2702-472a-930a-e08ece76a861","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":85,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6800","nonce":1133540779},{"type":"v1","addr":"172.21.15.119:6801","nonce":1133540779}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6802","nonce":1133540779},{"type":"v1","addr":"172.21.15.119:6803","nonce":1133540779}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6806","nonce":1133540779},{"type":"v1","addr":"172.21.15.119:6807","nonce":1133540779}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6804","nonce":1133540779},{"type":"v1","addr":"172.21.15.119:6805","nonce":1133540779}]},"public_addr":"172.21.15.119:6801/1133540779","cluster_addr":"172.21.15.119:6803/1133540779","heartbeat_back_addr":"172.21.15.119:6807/1133540779","heartbeat_front_addr":"172.21.15.119:6805/1133540779","state":["exists","up"]},{"osd":2,"uuid":"f4c9f2b4-ff49-4e14-89cd-b477feeb9382","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":85,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6800","nonce":799652511},{"type":"v1","addr":"172.21.15.204:6801","nonce":799652511}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6802","nonce":799652511},{"type":"v1","addr":"172.21.15.204:6803","nonce":799652511}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6806","nonce":799652511},{"type":"v1","addr":"172.21.15.204:6807","nonce":799652511}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6804","nonce":799652511},{"type":"v1","addr":"172.21.15.204:6805","nonce":799652511}]},"public_addr":"172.21.15.204:6801/799652511","cluster_addr":"172.21.15.204:6803/799652511","heartbeat_back_addr":"172.21.15.204:6807/799652511","heartbeat_front_addr":"172.21.15.204:6805/799652511","state":["exists","up"]},{"osd":3,"uuid":"52d2a566-e71c-4bb3-8d6c-5abdd43177c2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":25,"up_thru":88,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6810","nonce":4275595091},{"type":"v1","addr":"172.21.15.114:6811","nonce":4275595091}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6812","nonce":4275595091},{"type":"v1","addr":"172.21.15.114:6813","nonce":4275595091}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6816","nonce":4275595091},{"type":"v1","addr":"172.21.15.114:6817","nonce":4275595091}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6814","nonce":4275595091},{"type":"v1","addr":"172.21.15.114:6815","nonce":4275595091}]},"public_addr":"172.21.15.114:6811/4275595091","cluster_addr":"172.21.15.114:6813/4275595091","heartbeat_back_addr":"172.21.15.114:6817/4275595091","heartbeat_front_addr":"172.21.15.114:6815/4275595091","state":["exists","up"]},{"osd":4,"uuid":"d911a7c1-4bdc-4d28-b8a6-73896a6e7de6","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":33,"up_thru":85,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6808","nonce":1611007172},{"type":"v1","addr":"172.21.15.119:6809","nonce":1611007172}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6810","nonce":1611007172},{"type":"v1","addr":"172.21.15.119:6811","nonce":1611007172}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6814","nonce":1611007172},{"type":"v1","addr":"172.21.15.119:6815","nonce":1611007172}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6812","nonce":1611007172},{"type":"v1","addr":"172.21.15.119:6813","nonce":1611007172}]},"public_addr":"172.21.15.119:6809/1611007172","cluster_addr":"172.21.15.119:6811/1611007172","heartbeat_back_addr":"172.21.15.119:6815/1611007172","heartbeat_front_addr":"172.21.15.119:6813/1611007172","state":["exists","up"]},{"osd":5,"uuid":"f7590e1a-e93c-4c29-b15f-38ee52d1f921","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":39,"up_thru":85,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6808","nonce":1427563312},{"type":"v1","addr":"172.21.15.204:6809","nonce":1427563312}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6810","nonce":1427563312},{"type":"v1","addr":"172.21.15.204:6811","nonce":1427563312}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6814","nonce":1427563312},{"type":"v1","addr":"172.21.15.204:6815","nonce":1427563312}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6812","nonce":1427563312},{"type":"v1","addr":"172.21.15.204:6813","nonce":1427563312}]},"public_addr":"172.21.15.204:6809/1427563312","cluster_addr":"172.21.15.204:6811/1427563312","heartbeat_back_addr":"172.21.15.204:6815/1427563312","heartbeat_front_addr":"172.21.15.204:6813/1427563312","state":["exists","up"]},{"osd":6,"uuid":"534d20e8-3e54-4db8-a845-6119f49c7a78","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":47,"up_thru":88,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6818","nonce":4072912544},{"type":"v1","addr":"172.21.15.114:6819","nonce":4072912544}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6820","nonce":4072912544},{"type":"v1","addr":"172.21.15.114:6821","nonce":4072912544}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6824","nonce":4072912544},{"type":"v1","addr":"172.21.15.114:6825","nonce":4072912544}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6822","nonce":4072912544},{"type":"v1","addr":"172.21.15.114:6823","nonce":4072912544}]},"public_addr":"172.21.15.114:6819/4072912544","cluster_addr":"172.21.15.114:6821/4072912544","heartbeat_back_addr":"172.21.15.114:6825/4072912544","heartbeat_front_addr":"172.21.15.114:6823/4072912544","state":["exists","up"]},{"osd":7,"uuid":"f0ec22b8-1b48-4859-9d65-b00403aff3ad","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":53,"up_thru":88,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6816","nonce":2237458793},{"type":"v1","addr":"172.21.15.119:6817","nonce":2237458793}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6818","nonce":2237458793},{"type":"v1","addr":"172.21.15.119:6819","nonce":2237458793}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6822","nonce":2237458793},{"type":"v1","addr":"172.21.15.119:6823","nonce":2237458793}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6820","nonce":2237458793},{"type":"v1","addr":"172.21.15.119:6821","nonce":2237458793}]},"public_addr":"172.21.15.119:6817/2237458793","cluster_addr":"172.21.15.119:6819/2237458793","heartbeat_back_addr":"172.21.15.119:6823/2237458793","heartbeat_front_addr":"172.21.15.119:6821/2237458793","state":["exists","up"]},{"osd":8,"uuid":"3ac89686-6d76-4cef-9089-ac29e4b3bcac","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":62,"up_thru":85,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6816","nonce":1426318231},{"type":"v1","addr":"172.21.15.204:6817","nonce":1426318231}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6818","nonce":1426318231},{"type":"v1","addr":"172.21.15.204:6819","nonce":1426318231}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6822","nonce":1426318231},{"type":"v1","addr":"172.21.15.204:6823","nonce":1426318231}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6820","nonce":1426318231},{"type":"v1","addr":"172.21.15.204:6821","nonce":1426318231}]},"public_addr":"172.21.15.204:6817/1426318231","cluster_addr":"172.21.15.204:6819/1426318231","heartbeat_back_addr":"172.21.15.204:6823/1426318231","heartbeat_front_addr":"172.21.15.204:6821/1426318231","state":["exists","up"]},{"osd":9,"uuid":"a80ffe4d-1e98-460f-b046-c4754285cc5a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":69,"up_thru":88,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6826","nonce":2750635721},{"type":"v1","addr":"172.21.15.114:6827","nonce":2750635721}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6828","nonce":2750635721},{"type":"v1","addr":"172.21.15.114:6829","nonce":2750635721}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6832","nonce":2750635721},{"type":"v1","addr":"172.21.15.114:6833","nonce":2750635721}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6830","nonce":2750635721},{"type":"v1","addr":"172.21.15.114:6831","nonce":2750635721}]},"public_addr":"172.21.15.114:6827/2750635721","cluster_addr":"172.21.15.114:6829/2750635721","heartbeat_back_addr":"172.21.15.114:6833/2750635721","heartbeat_front_addr":"172.21.15.114:6831/2750635721","state":["exists","up"]},{"osd":10,"uuid":"cd75a579-bf12-4600-ac09-a8bfce772ced","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":74,"up_thru":85,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6824","nonce":1400187701},{"type":"v1","addr":"172.21.15.119:6825","nonce":1400187701}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6826","nonce":1400187701},{"type":"v1","addr":"172.21.15.119:6827","nonce":1400187701}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6830","nonce":1400187701},{"type":"v1","addr":"172.21.15.119:6831","nonce":1400187701}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6828","nonce":1400187701},{"type":"v1","addr":"172.21.15.119:6829","nonce":1400187701}]},"public_addr":"172.21.15.119:6825/1400187701","cluster_addr":"172.21.15.119:6827/1400187701","heartbeat_back_addr":"172.21.15.119:6831/1400187701","heartbeat_front_addr":"172.21.15.119:6829/1400187701","state":["exists","up"]},{"osd":11,"uuid":"53f037a5-3ec0-483a-80a1-dfee923a68e5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":82,"up_thru":88,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6824","nonce":2407298490},{"type":"v1","addr":"172.21.15.204:6825","nonce":2407298490}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6826","nonce":2407298490},{"type":"v1","addr":"172.21.15.204:6827","nonce":2407298490}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6830","nonce":2407298490},{"type":"v1","addr":"172.21.15.204:6831","nonce":2407298490}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6828","nonce":2407298490},{"type":"v1","addr":"172.21.15.204:6829","nonce":2407298490}]},"public_addr":"172.21.15.204:6825/2407298490","cluster_addr":"172.21.15.204:6827/2407298490","heartbeat_back_addr":"172.21.15.204:6831/2407298490","heartbeat_front_addr":"172.21.15.204:6829/2407298490","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-23T10:19:18.401147+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-23T10:19:42.611299+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-23T10:20:07.277367+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-23T10:20:31.208368+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-23T10:20:55.866225+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-23T10:21:19.961237+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-23T10:21:44.480783+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-23T10:22:09.173641+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-23T10:22:34.030328+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-23T10:22:59.652293+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-23T10:23:25.659673+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-23T10:23:51.168171+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"2.e","mappings":[{"from":2,"to":4}]},{"pgid":"2.1d","mappings":[{"from":8,"to":11}]},{"pgid":"3.e","mappings":[{"from":5,"to":9}]},{"pgid":"3.12","mappings":[{"from":3,"to":9}]},{"pgid":"3.17","mappings":[{"from":4,"to":6}]},{"pgid":"3.23","mappings":[{"from":4,"to":7}]},{"pgid":"3.27","mappings":[{"from":4,"to":6}]},{"pgid":"3.31","mappings":[{"from":4,"to":8}]},{"pgid":"3.38","mappings":[{"from":3,"to":9}]},{"pgid":"3.39","mappings":[{"from":5,"to":8}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.114:6801/1604678557":"2024-04-24T10:17:45.874609+0000","172.21.15.114:0/4025385854":"2024-04-24T10:17:45.874609+0000","172.21.15.114:0/2228273469":"2024-04-24T10:17:19.910348+0000","172.21.15.114:6800/2831530838":"2024-04-24T10:17:19.910348+0000","172.21.15.114:0/1333447450":"2024-04-24T10:17:45.874609+0000","172.21.15.114:0/2233026840":"2024-04-24T10:17:19.910348+0000","172.21.15.114:0/1902281545":"2024-04-24T10:17:19.910348+0000","172.21.15.114:6801/2831530838":"2024-04-24T10:17:19.910348+0000","172.21.15.114:6800/1604678557":"2024-04-24T10:17:45.874609+0000","172.21.15.114:0/1623553811":"2024-04-24T10:17:45.874609+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-23T10:26:07.721 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:07.722+0000 7f3b4687c640 1 -- 172.21.15.114:0/872498497 >> [v2:172.21.15.114:6800/465333808,v1:172.21.15.114:6801/465333808] conn(0x7f3b280a3c40 msgr2=0x7f3b280a6100 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T10:26:07.721 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:07.722+0000 7f3b4687c640 1 --2- 172.21.15.114:0/872498497 >> [v2:172.21.15.114:6800/465333808,v1:172.21.15.114:6801/465333808] conn(0x7f3b280a3c40 0x7f3b280a6100 secure :-1 s=READY pgs=221 cs=0 l=1 rev1=1 crypto rx=0x7f3b30009230 tx=0x7f3b300314a0 comp rx=0 tx=0).stop 2024-04-23T10:26:07.722 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:07.722+0000 7f3b4687c640 1 -- 172.21.15.114:0/872498497 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f3b40103190 msgr2=0x7f3b40198670 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T10:26:07.722 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:07.722+0000 7f3b4687c640 1 --2- 172.21.15.114:0/872498497 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f3b40103190 0x7f3b40198670 secure :-1 s=READY pgs=325 cs=0 l=1 rev1=1 crypto rx=0x7f3b3400c920 tx=0x7f3b3400cde0 comp rx=0 tx=0).stop 2024-04-23T10:26:07.722 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:07.722+0000 7f3b4687c640 1 -- 172.21.15.114:0/872498497 shutdown_connections 2024-04-23T10:26:07.722 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:07.722+0000 7f3b4687c640 1 --2- 172.21.15.114:0/872498497 >> [v2:172.21.15.114:6800/465333808,v1:172.21.15.114:6801/465333808] conn(0x7f3b280a3c40 0x7f3b280a6100 unknown :-1 s=CLOSED pgs=221 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:26:07.722 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:07.722+0000 7f3b4687c640 1 --2- 172.21.15.114:0/872498497 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f3b40103b50 0x7f3b4019f6e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:26:07.722 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:07.722+0000 7f3b4687c640 1 --2- 172.21.15.114:0/872498497 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f3b40103190 0x7f3b40198670 unknown :-1 s=CLOSED pgs=325 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:26:07.722 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:07.722+0000 7f3b4687c640 1 --2- 172.21.15.114:0/872498497 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f3b40101fa0 0x7f3b40198130 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:26:07.722 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:07.722+0000 7f3b4687c640 1 -- 172.21.15.114:0/872498497 >> 172.21.15.114:0/872498497 conn(0x7f3b400fd530 msgr2=0x7f3b40106490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T10:26:07.722 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:07.722+0000 7f3b4687c640 1 -- 172.21.15.114:0/872498497 shutdown_connections 2024-04-23T10:26:07.722 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:07.722+0000 7f3b4687c640 1 -- 172.21.15.114:0/872498497 wait complete. 2024-04-23T10:26:07.733 DEBUG:teuthology.orchestra.run.smithi114:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-23T10:26:07.832 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:07.831+0000 7ff105119640 1 -- 172.21.15.114:0/98188572 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7ff100103c00 msgr2=0x7ff10010a490 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T10:26:07.832 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:07.831+0000 7ff105119640 1 --2- 172.21.15.114:0/98188572 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7ff100103c00 0x7ff10010a490 secure :-1 s=READY pgs=326 cs=0 l=1 rev1=1 crypto rx=0x7ff0f8009cd0 tx=0x7ff0f802a8e0 comp rx=0 tx=0).stop 2024-04-23T10:26:07.832 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:07.831+0000 7ff105119640 1 -- 172.21.15.114:0/98188572 shutdown_connections 2024-04-23T10:26:07.832 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:07.831+0000 7ff105119640 1 --2- 172.21.15.114:0/98188572 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7ff100103c00 0x7ff10010a490 unknown :-1 s=CLOSED pgs=326 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:26:07.832 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:07.831+0000 7ff105119640 1 --2- 172.21.15.114:0/98188572 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7ff100103240 0x7ff1001036c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:26:07.833 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:07.831+0000 7ff105119640 1 --2- 172.21.15.114:0/98188572 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7ff100100270 0x7ff100100690 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:26:07.833 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:07.831+0000 7ff105119640 1 -- 172.21.15.114:0/98188572 >> 172.21.15.114:0/98188572 conn(0x7ff1000fbb80 msgr2=0x7ff1000fdfe0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T10:26:07.833 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:07.831+0000 7ff105119640 1 -- 172.21.15.114:0/98188572 shutdown_connections 2024-04-23T10:26:07.833 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:07.831+0000 7ff105119640 1 -- 172.21.15.114:0/98188572 wait complete. 2024-04-23T10:26:07.833 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:07.832+0000 7ff105119640 1 Processor -- start 2024-04-23T10:26:07.833 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:07.832+0000 7ff105119640 1 -- start start 2024-04-23T10:26:07.833 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:07.832+0000 7ff105119640 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7ff100100270 0x7ff1001981c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T10:26:07.833 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:07.832+0000 7ff105119640 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7ff100103240 0x7ff100198700 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T10:26:07.833 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:07.832+0000 7ff105119640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7ff100103c00 0x7ff10019f720 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T10:26:07.833 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:07.832+0000 7ff105119640 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x7ff10010db10 con 0x7ff100103c00 2024-04-23T10:26:07.833 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:07.832+0000 7ff105119640 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 -- 0x7ff10010d990 con 0x7ff100100270 2024-04-23T10:26:07.833 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:07.832+0000 7ff105119640 1 -- --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_getmap magic: 0 -- 0x7ff10010dc90 con 0x7ff100103240 2024-04-23T10:26:07.833 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:07.832+0000 7ff0ff59e640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7ff100103c00 0x7ff10019f720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T10:26:07.833 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:07.832+0000 7ff0ff59e640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7ff100103c00 0x7ff10019f720 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.114:58084/0 (socket says 172.21.15.114:58084) 2024-04-23T10:26:07.833 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:07.832+0000 7ff0ff59e640 1 -- 172.21.15.114:0/103719779 learned_addr learned my addr 172.21.15.114:0/103719779 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-04-23T10:26:07.834 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:07.832+0000 7ff0f7fff640 1 --2- 172.21.15.114:0/103719779 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7ff100103240 0x7ff100198700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T10:26:07.834 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:07.832+0000 7ff0fed9d640 1 --2- 172.21.15.114:0/103719779 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7ff100100270 0x7ff1001981c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T10:26:07.834 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:07.832+0000 7ff0ff59e640 1 -- 172.21.15.114:0/103719779 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7ff100103240 msgr2=0x7ff100198700 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T10:26:07.834 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:07.832+0000 7ff0ff59e640 1 --2- 172.21.15.114:0/103719779 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7ff100103240 0x7ff100198700 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:26:07.834 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:07.832+0000 7ff0ff59e640 1 -- 172.21.15.114:0/103719779 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7ff100100270 msgr2=0x7ff1001981c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T10:26:07.834 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:07.832+0000 7ff0ff59e640 1 --2- 172.21.15.114:0/103719779 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7ff100100270 0x7ff1001981c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:26:07.834 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:07.832+0000 7ff0ff59e640 1 -- 172.21.15.114:0/103719779 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff10019fd90 con 0x7ff100103c00 2024-04-23T10:26:07.834 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:07.832+0000 7ff0f7fff640 1 --2- 172.21.15.114:0/103719779 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7ff100103240 0x7ff100198700 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-23T10:26:07.834 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:07.832+0000 7ff0fed9d640 1 --2- 172.21.15.114:0/103719779 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7ff100100270 0x7ff1001981c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-23T10:26:07.834 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:07.832+0000 7ff0ff59e640 1 --2- 172.21.15.114:0/103719779 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7ff100103c00 0x7ff10019f720 secure :-1 s=READY pgs=327 cs=0 l=1 rev1=1 crypto rx=0x7ff0f80318b0 tx=0x7ff0f8003ba0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T10:26:07.834 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:07.833+0000 7ff0fcd99640 1 -- 172.21.15.114:0/103719779 <== mon.0 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff0f8004240 con 0x7ff100103c00 2024-04-23T10:26:07.834 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:07.833+0000 7ff0fcd99640 1 -- 172.21.15.114:0/103719779 <== mon.0 v2:172.21.15.114:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7ff0f80305a0 con 0x7ff100103c00 2024-04-23T10:26:07.834 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:07.833+0000 7ff0fcd99640 1 -- 172.21.15.114:0/103719779 <== mon.0 v2:172.21.15.114:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff0f80466e0 con 0x7ff100103c00 2024-04-23T10:26:07.834 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:07.833+0000 7ff105119640 1 -- 172.21.15.114:0/103719779 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7ff1001a0020 con 0x7ff100103c00 2024-04-23T10:26:07.834 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:07.833+0000 7ff105119640 1 -- 172.21.15.114:0/103719779 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7ff1001a0570 con 0x7ff100103c00 2024-04-23T10:26:07.835 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:07.834+0000 7ff105119640 1 -- 172.21.15.114:0/103719779 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7ff100109410 con 0x7ff100103c00 2024-04-23T10:26:07.836 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:07.835+0000 7ff0fcd99640 1 -- 172.21.15.114:0/103719779 <== mon.0 v2:172.21.15.114:3300/0 4 ==== mgrmap(e 17) ==== 138698+0+0 (secure 0 0 0) 0x7ff0f8030750 con 0x7ff100103c00 2024-04-23T10:26:07.836 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:07.835+0000 7ff0fcd99640 1 --2- 172.21.15.114:0/103719779 >> [v2:172.21.15.114:6800/465333808,v1:172.21.15.114:6801/465333808] conn(0x7ff0ec0a3c40 0x7ff0ec0a6100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T10:26:07.836 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:07.835+0000 7ff0fed9d640 1 --2- 172.21.15.114:0/103719779 >> [v2:172.21.15.114:6800/465333808,v1:172.21.15.114:6801/465333808] conn(0x7ff0ec0a3c40 0x7ff0ec0a6100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T10:26:07.836 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:07.835+0000 7ff0fcd99640 1 -- 172.21.15.114:0/103719779 <== mon.0 v2:172.21.15.114:3300/0 5 ==== osd_map(89..89 src has 1..89) ==== 8269+0+0 (secure 0 0 0) 0x7ff0f802e030 con 0x7ff100103c00 2024-04-23T10:26:07.836 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:07.836+0000 7ff0fed9d640 1 --2- 172.21.15.114:0/103719779 >> [v2:172.21.15.114:6800/465333808,v1:172.21.15.114:6801/465333808] conn(0x7ff0ec0a3c40 0x7ff0ec0a6100 secure :-1 s=READY pgs=222 cs=0 l=1 rev1=1 crypto rx=0x7ff10006a390 tx=0x7ff0f0005f90 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T10:26:07.838 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:07.840+0000 7ff0fcd99640 1 -- 172.21.15.114:0/103719779 <== mon.0 v2:172.21.15.114:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7ff0f804b060 con 0x7ff100103c00 2024-04-23T10:26:08.057 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:08.059+0000 7ff105119640 1 -- 172.21.15.114:0/103719779 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7ff100062b50 con 0x7ff100103c00 2024-04-23T10:26:08.058 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:08.059+0000 7ff0fcd99640 1 -- 172.21.15.114:0/103719779 <== mon.0 v2:172.21.15.114:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 20 v20) ==== 94+0+11884 (secure 0 0 0) 0x7ff0f80be420 con 0x7ff100103c00 2024-04-23T10:26:08.058 INFO:teuthology.orchestra.run.smithi114.stdout: 2024-04-23T10:26:08.059 INFO:teuthology.orchestra.run.smithi114.stdout:{"epoch":20,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14547,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.114:6835/663493580","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6834","nonce":663493580},{"type":"v1","addr":"172.21.15.114:6835","nonce":663493580}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14565,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.114:6837/1439252154","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6836","nonce":1439252154},{"type":"v1","addr":"172.21.15.114:6837","nonce":1439252154}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5},{"gid":14634,"name":"g","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.114:6839/3849796939","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6838","nonce":3849796939},{"type":"v1","addr":"172.21.15.114:6839","nonce":3849796939}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":11},{"gid":14646,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.114:6841/3544523891","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6840","nonce":3544523891},{"type":"v1","addr":"172.21.15.114:6841","nonce":3544523891}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":12},{"gid":24385,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.204:6833/3866561754","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6832","nonce":3866561754},{"type":"v1","addr":"172.21.15.204:6833","nonce":3866561754}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2},{"gid":24427,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.204:6835/2533793172","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6834","nonce":2533793172},{"type":"v1","addr":"172.21.15.204:6835","nonce":2533793172}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":7}],"filesystems":[{"mdsmap":{"epoch":20,"flags":50,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-04-23T10:24:43.383158+0000","modified":"2024-04-23T10:24:48.315128+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":3,"in":[0,1,2],"up":{"mds_0":24545,"mds_1":24509,"mds_2":24479},"failed":[],"damaged":[],"stopped":[],"info":{"gid_24439":{"gid":24439,"name":"c","rank":2,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.204:6837/513822179","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6836","nonce":513822179},{"type":"v1","addr":"172.21.15.204:6837","nonce":513822179}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24443":{"gid":24443,"name":"k","rank":1,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.119:6833/783244660","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6832","nonce":783244660},{"type":"v1","addr":"172.21.15.119:6833","nonce":783244660}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24463":{"gid":24463,"name":"i","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.204:6839/3222658893","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6838","nonce":3222658893},{"type":"v1","addr":"172.21.15.204:6839","nonce":3222658893}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24479":{"gid":24479,"name":"e","rank":2,"incarnation":19,"state":"up:active","state_seq":9,"addr":"172.21.15.119:6835/2033916709","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6834","nonce":2033916709},{"type":"v1","addr":"172.21.15.119:6835","nonce":2033916709}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24509":{"gid":24509,"name":"h","rank":1,"incarnation":17,"state":"up:active","state_seq":6,"addr":"172.21.15.119:6837/1872363128","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6836","nonce":1872363128},{"type":"v1","addr":"172.21.15.119:6837","nonce":1872363128}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24545":{"gid":24545,"name":"b","rank":0,"incarnation":14,"state":"up:active","state_seq":3,"addr":"172.21.15.119:6839/937297340","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6838","nonce":937297340},{"type":"v1","addr":"172.21.15.119:6839","nonce":937297340}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":24545,"qdb_cluster":[24509,24545,24479]},"id":1}]} 2024-04-23T10:26:08.059 INFO:teuthology.orchestra.run.smithi114.stderr:dumped fsmap epoch 20 2024-04-23T10:26:08.061 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:08.061+0000 7ff105119640 1 -- 172.21.15.114:0/103719779 >> [v2:172.21.15.114:6800/465333808,v1:172.21.15.114:6801/465333808] conn(0x7ff0ec0a3c40 msgr2=0x7ff0ec0a6100 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T10:26:08.061 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:08.061+0000 7ff105119640 1 --2- 172.21.15.114:0/103719779 >> [v2:172.21.15.114:6800/465333808,v1:172.21.15.114:6801/465333808] conn(0x7ff0ec0a3c40 0x7ff0ec0a6100 secure :-1 s=READY pgs=222 cs=0 l=1 rev1=1 crypto rx=0x7ff10006a390 tx=0x7ff0f0005f90 comp rx=0 tx=0).stop 2024-04-23T10:26:08.061 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:08.061+0000 7ff105119640 1 -- 172.21.15.114:0/103719779 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7ff100103c00 msgr2=0x7ff10019f720 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T10:26:08.061 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:08.061+0000 7ff105119640 1 --2- 172.21.15.114:0/103719779 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7ff100103c00 0x7ff10019f720 secure :-1 s=READY pgs=327 cs=0 l=1 rev1=1 crypto rx=0x7ff0f80318b0 tx=0x7ff0f8003ba0 comp rx=0 tx=0).stop 2024-04-23T10:26:08.061 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:08.061+0000 7ff105119640 1 -- 172.21.15.114:0/103719779 shutdown_connections 2024-04-23T10:26:08.061 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:08.061+0000 7ff105119640 1 --2- 172.21.15.114:0/103719779 >> [v2:172.21.15.114:6800/465333808,v1:172.21.15.114:6801/465333808] conn(0x7ff0ec0a3c40 0x7ff0ec0a6100 unknown :-1 s=CLOSED pgs=222 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:26:08.061 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:08.061+0000 7ff105119640 1 --2- 172.21.15.114:0/103719779 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7ff100103c00 0x7ff10019f720 unknown :-1 s=CLOSED pgs=327 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:26:08.061 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:08.061+0000 7ff105119640 1 --2- 172.21.15.114:0/103719779 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7ff100103240 0x7ff100198700 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:26:08.061 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:08.061+0000 7ff105119640 1 --2- 172.21.15.114:0/103719779 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7ff100100270 0x7ff1001981c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:26:08.062 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:08.062+0000 7ff105119640 1 -- 172.21.15.114:0/103719779 >> 172.21.15.114:0/103719779 conn(0x7ff1000fbb80 msgr2=0x7ff1001065e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T10:26:08.062 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:08.062+0000 7ff105119640 1 -- 172.21.15.114:0/103719779 shutdown_connections 2024-04-23T10:26:08.062 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:08.062+0000 7ff105119640 1 -- 172.21.15.114:0/103719779 wait complete. 2024-04-23T10:26:08.072 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 20, 'flags': 50, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': True, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 32, 'explicitly_allowed_features': 32, 'created': '2024-04-23T10:24:43.383158+0000', 'modified': '2024-04-23T10:24:48.315128+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 300, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}, 'max_mds': 3, 'in': [0, 1, 2], 'up': {'mds_0': 24545, 'mds_1': 24509, 'mds_2': 24479}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_24439': {'gid': 24439, 'name': 'c', 'rank': 2, 'incarnation': 0, 'state': 'up:standby-replay', 'state_seq': 1, 'addr': '172.21.15.204:6837/513822179', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.204:6836', 'nonce': 513822179}, {'type': 'v1', 'addr': '172.21.15.204:6837', 'nonce': 513822179}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_24443': {'gid': 24443, 'name': 'k', 'rank': 1, 'incarnation': 0, 'state': 'up:standby-replay', 'state_seq': 1, 'addr': '172.21.15.119:6833/783244660', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.119:6832', 'nonce': 783244660}, {'type': 'v1', 'addr': '172.21.15.119:6833', 'nonce': 783244660}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_24463': {'gid': 24463, 'name': 'i', 'rank': 0, 'incarnation': 0, 'state': 'up:standby-replay', 'state_seq': 1, 'addr': '172.21.15.204:6839/3222658893', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.204:6838', 'nonce': 3222658893}, {'type': 'v1', 'addr': '172.21.15.204:6839', 'nonce': 3222658893}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_24479': {'gid': 24479, 'name': 'e', 'rank': 2, 'incarnation': 19, 'state': 'up:active', 'state_seq': 9, 'addr': '172.21.15.119:6835/2033916709', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.119:6834', 'nonce': 2033916709}, {'type': 'v1', 'addr': '172.21.15.119:6835', 'nonce': 2033916709}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_24509': {'gid': 24509, 'name': 'h', 'rank': 1, 'incarnation': 17, 'state': 'up:active', 'state_seq': 6, 'addr': '172.21.15.119:6837/1872363128', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.119:6836', 'nonce': 1872363128}, {'type': 'v1', 'addr': '172.21.15.119:6837', 'nonce': 1872363128}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_24545': {'gid': 24545, 'name': 'b', 'rank': 0, 'incarnation': 14, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.15.119:6839/937297340', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.119:6838', 'nonce': 937297340}, {'type': 'v1', 'addr': '172.21.15.119:6839', 'nonce': 937297340}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}}, 'data_pools': [3], 'metadata_pool': 2, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 24545, 'qdb_cluster': [24509, 24545, 24479]} 2024-04-23T10:26:08.072 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: 3/3 2024-04-23T10:26:08.072 DEBUG:teuthology.orchestra.run.smithi114:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.e status 2024-04-23T10:26:08.174 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:08.173+0000 7f9800664640 1 -- 172.21.15.114:0/2532046083 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f97f8109e40 msgr2=0x7f97f810c2d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T10:26:08.174 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:08.173+0000 7f9800664640 1 --2- 172.21.15.114:0/2532046083 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f97f8109e40 0x7f97f810c2d0 secure :-1 s=READY pgs=328 cs=0 l=1 rev1=1 crypto rx=0x7f97f0009ca0 tx=0x7f97f002a950 comp rx=0 tx=0).stop 2024-04-23T10:26:08.174 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:08.173+0000 7f9800664640 1 -- 172.21.15.114:0/2532046083 shutdown_connections 2024-04-23T10:26:08.174 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:08.173+0000 7f9800664640 1 --2- 172.21.15.114:0/2532046083 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f97f8109e40 0x7f97f810c2d0 unknown :-1 s=CLOSED pgs=328 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:26:08.174 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:08.173+0000 7f9800664640 1 --2- 172.21.15.114:0/2532046083 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f97f8074930 0x7f97f8074d90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:26:08.174 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:08.173+0000 7f9800664640 1 --2- 172.21.15.114:0/2532046083 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f97f8075f10 0x7f97f8074360 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:26:08.174 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:08.173+0000 7f9800664640 1 -- 172.21.15.114:0/2532046083 >> 172.21.15.114:0/2532046083 conn(0x7f97f80fd500 msgr2=0x7f97f80ff940 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T10:26:08.174 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:08.173+0000 7f9800664640 1 -- 172.21.15.114:0/2532046083 shutdown_connections 2024-04-23T10:26:08.174 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:08.173+0000 7f9800664640 1 -- 172.21.15.114:0/2532046083 wait complete. 2024-04-23T10:26:08.175 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:08.173+0000 7f9800664640 1 Processor -- start 2024-04-23T10:26:08.175 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:08.173+0000 7f9800664640 1 -- start start 2024-04-23T10:26:08.175 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:08.174+0000 7f9800664640 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f97f8074930 0x7f97f819a350 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T10:26:08.175 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:08.174+0000 7f9800664640 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f97f8075f10 0x7f97f819a890 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T10:26:08.175 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:08.174+0000 7f9800664640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f97f8109e40 0x7f97f81a1900 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T10:26:08.175 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:08.174+0000 7f9800664640 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x7f97f810fce0 con 0x7f97f8109e40 2024-04-23T10:26:08.175 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:08.174+0000 7f9800664640 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 -- 0x7f97f810fb60 con 0x7f97f8074930 2024-04-23T10:26:08.175 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:08.174+0000 7f9800664640 1 -- --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_getmap magic: 0 -- 0x7f97f810fe60 con 0x7f97f8075f10 2024-04-23T10:26:08.175 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:08.174+0000 7f97fec01640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f97f8109e40 0x7f97f81a1900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T10:26:08.175 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:08.174+0000 7f97fec01640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f97f8109e40 0x7f97f81a1900 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.114:58100/0 (socket says 172.21.15.114:58100) 2024-04-23T10:26:08.175 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:08.174+0000 7f97fec01640 1 -- 172.21.15.114:0/3869408247 learned_addr learned my addr 172.21.15.114:0/3869408247 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-04-23T10:26:08.175 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:08.174+0000 7f97fe400640 1 --2- 172.21.15.114:0/3869408247 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f97f8074930 0x7f97f819a350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T10:26:08.175 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:08.174+0000 7f97fdbff640 1 --2- 172.21.15.114:0/3869408247 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f97f8075f10 0x7f97f819a890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T10:26:08.175 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:08.174+0000 7f97fec01640 1 -- 172.21.15.114:0/3869408247 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f97f8075f10 msgr2=0x7f97f819a890 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T10:26:08.176 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:08.174+0000 7f97fec01640 1 --2- 172.21.15.114:0/3869408247 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f97f8075f10 0x7f97f819a890 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:26:08.176 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:08.174+0000 7f97fec01640 1 -- 172.21.15.114:0/3869408247 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f97f8074930 msgr2=0x7f97f819a350 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T10:26:08.176 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:08.174+0000 7f97fec01640 1 --2- 172.21.15.114:0/3869408247 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f97f8074930 0x7f97f819a350 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:26:08.176 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:08.174+0000 7f97fec01640 1 -- 172.21.15.114:0/3869408247 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f97f81a1f70 con 0x7f97f8109e40 2024-04-23T10:26:08.176 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:08.174+0000 7f97fec01640 1 --2- 172.21.15.114:0/3869408247 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f97f8109e40 0x7f97f81a1900 secure :-1 s=READY pgs=329 cs=0 l=1 rev1=1 crypto rx=0x7f97f0006010 tx=0x7f97f002fc20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T10:26:08.176 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:08.174+0000 7f97f77fe640 1 -- 172.21.15.114:0/3869408247 <== mon.0 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f97f0034040 con 0x7f97f8109e40 2024-04-23T10:26:08.176 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:08.174+0000 7f9800664640 1 -- 172.21.15.114:0/3869408247 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f97f81a2200 con 0x7f97f8109e40 2024-04-23T10:26:08.176 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:08.174+0000 7f97f77fe640 1 -- 172.21.15.114:0/3869408247 <== mon.0 v2:172.21.15.114:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f97f00308f0 con 0x7f97f8109e40 2024-04-23T10:26:08.176 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:08.174+0000 7f97f77fe640 1 -- 172.21.15.114:0/3869408247 <== mon.0 v2:172.21.15.114:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f97f0039450 con 0x7f97f8109e40 2024-04-23T10:26:08.176 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:08.174+0000 7f9800664640 1 -- 172.21.15.114:0/3869408247 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f97f81a2740 con 0x7f97f8109e40 2024-04-23T10:26:08.183 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:08.176+0000 7f97f77fe640 1 -- 172.21.15.114:0/3869408247 <== mon.0 v2:172.21.15.114:3300/0 4 ==== mgrmap(e 17) ==== 138698+0+0 (secure 0 0 0) 0x7f97f0037020 con 0x7f97f8109e40 2024-04-23T10:26:08.183 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:08.177+0000 7f980337b740 1 Processor -- start 2024-04-23T10:26:08.183 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:08.177+0000 7f980337b740 1 -- start start 2024-04-23T10:26:08.183 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:08.177+0000 7f980337b740 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55e2d9860b80 0x55e2d9860fa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T10:26:08.183 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:08.177+0000 7f980337b740 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x55e2d9861d50 0x55e2d98621d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T10:26:08.183 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:08.177+0000 7f980337b740 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x55e2d9862710 0x55e2d9868fa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T10:26:08.183 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:08.177+0000 7f980337b740 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x55e2d97c5c50 con 0x55e2d9860b80 2024-04-23T10:26:08.184 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:08.177+0000 7f980337b740 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 -- 0x55e2d9762830 con 0x55e2d9861d50 2024-04-23T10:26:08.184 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:08.177+0000 7f980337b740 1 -- --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_getmap magic: 0 -- 0x55e2d97c1c10 con 0x55e2d9862710 2024-04-23T10:26:08.184 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:08.177+0000 7f97f77fe640 1 --2- 172.21.15.114:0/3869408247 >> [v2:172.21.15.114:6800/465333808,v1:172.21.15.114:6801/465333808] conn(0x7f97e00a3c40 0x7f97e00a6100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T10:26:08.184 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:08.177+0000 7f97fdbff640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55e2d9860b80 0x55e2d9860fa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T10:26:08.184 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:08.177+0000 7f97fdbff640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55e2d9860b80 0x55e2d9860fa0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.114:58108/0 (socket says 172.21.15.114:58108) 2024-04-23T10:26:08.184 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:08.177+0000 7f97fdbff640 1 -- 172.21.15.114:0/1228493906 learned_addr learned my addr 172.21.15.114:0/1228493906 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-04-23T10:26:08.184 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:08.177+0000 7f97fe400640 1 --2- 172.21.15.114:0/1228493906 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x55e2d9861d50 0x55e2d98621d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T10:26:08.184 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:08.177+0000 7f97fec01640 1 --2- 172.21.15.114:0/3869408247 >> [v2:172.21.15.114:6800/465333808,v1:172.21.15.114:6801/465333808] conn(0x7f97e00a3c40 0x7f97e00a6100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T10:26:08.184 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:08.178+0000 7f97fdbff640 1 --2- 172.21.15.114:0/1228493906 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x55e2d9862710 0x55e2d9868fa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T10:26:08.184 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:08.178+0000 7f97f77fe640 1 -- 172.21.15.114:0/3869408247 <== mon.0 v2:172.21.15.114:3300/0 5 ==== osd_map(89..89 src has 1..89) ==== 8269+0+0 (secure 0 0 0) 0x7f97f00f1580 con 0x7f97f8109e40 2024-04-23T10:26:08.184 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:08.178+0000 7f97fdbff640 1 -- 172.21.15.114:0/1228493906 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x55e2d9862710 msgr2=0x55e2d9868fa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T10:26:08.184 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:08.178+0000 7f97fdbff640 1 --2- 172.21.15.114:0/1228493906 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x55e2d9862710 0x55e2d9868fa0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:26:08.184 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:08.178+0000 7f97fdbff640 1 -- 172.21.15.114:0/1228493906 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x55e2d9861d50 msgr2=0x55e2d98621d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T10:26:08.184 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:08.178+0000 7f97fdbff640 1 --2- 172.21.15.114:0/1228493906 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x55e2d9861d50 0x55e2d98621d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:26:08.185 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:08.178+0000 7f97fdbff640 1 -- 172.21.15.114:0/1228493906 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55e2d94d82d0 con 0x55e2d9860b80 2024-04-23T10:26:08.185 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:08.178+0000 7f97fdbff640 1 --2- 172.21.15.114:0/1228493906 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55e2d9860b80 0x55e2d9860fa0 secure :-1 s=READY pgs=330 cs=0 l=1 rev1=1 crypto rx=0x7f97ec009dd0 tx=0x7f97ec0097e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c7c182d7c3e7762e server_cookie=0 in_seq=0 out_seq=0 2024-04-23T10:26:08.185 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:08.178+0000 7f97fec01640 1 --2- 172.21.15.114:0/3869408247 >> [v2:172.21.15.114:6800/465333808,v1:172.21.15.114:6801/465333808] conn(0x7f97e00a3c40 0x7f97e00a6100 secure :-1 s=READY pgs=223 cs=0 l=1 rev1=1 crypto rx=0x7f97f00bd580 tx=0x7f97f008c280 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T10:26:08.185 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:08.178+0000 7f97f4bfe640 1 -- 172.21.15.114:0/1228493906 <== mon.0 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f97ec004030 con 0x55e2d9860b80 2024-04-23T10:26:08.185 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:08.178+0000 7f97f4bfe640 1 -- 172.21.15.114:0/1228493906 <== mon.0 v2:172.21.15.114:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f97ec0035f0 con 0x55e2d9860b80 2024-04-23T10:26:08.185 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:08.178+0000 7f97f4bfe640 1 -- 172.21.15.114:0/1228493906 <== mon.0 v2:172.21.15.114:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f97ec012690 con 0x55e2d9860b80 2024-04-23T10:26:08.185 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:08.179+0000 7f980337b740 1 -- 172.21.15.114:0/1228493906 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55e2d9860b80 msgr2=0x55e2d9860fa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T10:26:08.185 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:08.179+0000 7f980337b740 1 --2- 172.21.15.114:0/1228493906 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55e2d9860b80 0x55e2d9860fa0 secure :-1 s=READY pgs=330 cs=0 l=1 rev1=1 crypto rx=0x7f97ec009dd0 tx=0x7f97ec0097e0 comp rx=0 tx=0).stop 2024-04-23T10:26:08.185 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:08.179+0000 7f980337b740 1 -- 172.21.15.114:0/1228493906 shutdown_connections 2024-04-23T10:26:08.185 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:08.179+0000 7f980337b740 1 --2- 172.21.15.114:0/1228493906 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x55e2d9862710 0x55e2d9868fa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:26:08.185 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:08.179+0000 7f980337b740 1 --2- 172.21.15.114:0/1228493906 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x55e2d9861d50 0x55e2d98621d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:26:08.185 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:08.179+0000 7f980337b740 1 --2- 172.21.15.114:0/1228493906 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55e2d9860b80 0x55e2d9860fa0 unknown :-1 s=CLOSED pgs=330 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:26:08.185 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:08.179+0000 7f980337b740 1 -- 172.21.15.114:0/1228493906 >> 172.21.15.114:0/1228493906 conn(0x55e2d985c070 msgr2=0x55e2d985c470 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T10:26:08.185 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:08.179+0000 7f980337b740 1 -- 172.21.15.114:0/1228493906 shutdown_connections 2024-04-23T10:26:08.186 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:08.179+0000 7f980337b740 1 -- 172.21.15.114:0/1228493906 wait complete. 2024-04-23T10:26:08.186 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:08.179+0000 7f980337b740 1 -- start start 2024-04-23T10:26:08.186 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:08.179+0000 7f980337b740 1 Processor -- start 2024-04-23T10:26:08.186 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:08.179+0000 7f980337b740 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x55e2d9860b80 0x55e2d986afd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T10:26:08.186 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:08.179+0000 7f980337b740 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55e2d9861d50 0x55e2d986b510 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T10:26:08.186 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:08.179+0000 7f980337b740 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x55e2d9862710 0x55e2d9872550 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T10:26:08.186 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:08.179+0000 7f980337b740 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x55e2d9869720 con 0x55e2d9861d50 2024-04-23T10:26:08.186 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:08.179+0000 7f980337b740 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 -- 0x55e2d9760470 con 0x55e2d9862710 2024-04-23T10:26:08.186 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:08.179+0000 7f980337b740 1 -- --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_getmap magic: 0 -- 0x55e2d98698a0 con 0x55e2d9860b80 2024-04-23T10:26:08.186 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:08.179+0000 7f980337b740 10 client.0 _ll_register_callbacks cb 0x55e2d9785f30 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-23T10:26:08.186 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:08.179+0000 7f97fe400640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55e2d9861d50 0x55e2d986b510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T10:26:08.186 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:08.179+0000 7f97fe400640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55e2d9861d50 0x55e2d986b510 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.114:58116/0 (socket says 172.21.15.114:58116) 2024-04-23T10:26:08.186 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:08.180+0000 7f97fe400640 1 -- 172.21.15.114:0/218424096 learned_addr learned my addr 172.21.15.114:0/218424096 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-04-23T10:26:08.187 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:08.180+0000 7f97fdbff640 1 --2- 172.21.15.114:0/218424096 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x55e2d9862710 0x55e2d9872550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T10:26:08.187 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:08.180+0000 7f97fdbff640 1 --2- 172.21.15.114:0/218424096 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x55e2d9860b80 0x55e2d986afd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T10:26:08.187 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:08.180+0000 7f97fe400640 1 -- 172.21.15.114:0/218424096 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x55e2d9860b80 msgr2=0x55e2d986afd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T10:26:08.187 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:08.180+0000 7f97fe400640 1 --2- 172.21.15.114:0/218424096 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x55e2d9860b80 0x55e2d986afd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:26:08.187 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:08.180+0000 7f97fe400640 1 -- 172.21.15.114:0/218424096 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x55e2d9862710 msgr2=0x55e2d9872550 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T10:26:08.187 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:08.180+0000 7f97fe400640 1 --2- 172.21.15.114:0/218424096 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x55e2d9862710 0x55e2d9872550 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:26:08.187 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:08.180+0000 7f97fe400640 1 -- 172.21.15.114:0/218424096 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55e2d95a26b0 con 0x55e2d9861d50 2024-04-23T10:26:08.187 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:08.180+0000 7f97fdbff640 1 --2- 172.21.15.114:0/218424096 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x55e2d9860b80 0x55e2d986afd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-23T10:26:08.187 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:08.180+0000 7f97fe400640 1 --2- 172.21.15.114:0/218424096 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55e2d9861d50 0x55e2d986b510 secure :-1 s=READY pgs=331 cs=0 l=1 rev1=1 crypto rx=0x7f97e8012740 tx=0x7f97e8012c00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=16951dffd57504d1 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T10:26:08.187 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:08.180+0000 7f97ddffb640 10 client.0 ms_handle_connect on v2:172.21.15.114:3300/0 2024-04-23T10:26:08.187 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:08.180+0000 7f97ddffb640 1 -- 172.21.15.114:0/218424096 <== mon.0 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f97e800f8c0 con 0x55e2d9861d50 2024-04-23T10:26:08.187 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:08.180+0000 7f97ddffb640 1 -- 172.21.15.114:0/218424096 <== mon.0 v2:172.21.15.114:3300/0 2 ==== osd_map(89..89 src has 1..89) ==== 8269+0+0 (secure 0 0 0) 0x7f97e800fa60 con 0x55e2d9861d50 2024-04-23T10:26:08.187 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:08.180+0000 7f980337b740 1 -- 172.21.15.114:0/218424096 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f97ec0035f0 con 0x55e2d9861d50 2024-04-23T10:26:08.187 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:08.180+0000 7f97ddffb640 1 -- 172.21.15.114:0/218424096 <== mon.0 v2:172.21.15.114:3300/0 3 ==== mon_get_version_reply(handle=1 version=20) ==== 24+0+0 (secure 0 0 0) 0x7f97e800fc60 con 0x55e2d9861d50 2024-04-23T10:26:08.187 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:08.180+0000 7f980337b740 10 client.15066 fetch_fsmap learned FSMap version 20 2024-04-23T10:26:08.188 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:08.180+0000 7f980337b740 1 -- 172.21.15.114:0/218424096 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({config=0+,fsmap=20,monmap=4+}) -- 0x55e2d9875990 con 0x55e2d9861d50 2024-04-23T10:26:08.188 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:08.180+0000 7f97ddffb640 1 -- 172.21.15.114:0/218424096 <== mon.0 v2:172.21.15.114:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f97ec0035f0 con 0x55e2d9861d50 2024-04-23T10:26:08.188 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:08.180+0000 7f97ddffb640 1 -- 172.21.15.114:0/218424096 <== mon.0 v2:172.21.15.114:3300/0 5 ==== fsmap(e 20) ==== 8338+0+0 (secure 0 0 0) 0x7f97e8026420 con 0x55e2d9861d50 2024-04-23T10:26:08.188 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:08.181+0000 7f980337b740 10 client.15066 fetch_fsmap finished waiting for FSMap version 20 2024-04-23T10:26:08.188 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:08.181+0000 7f980337b740 10 client.15066 resolve_mds: resolved name 'e' to daemon mds.e 2024-04-23T10:26:08.188 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:08.181+0000 7f980337b740 20 client.15066 populate_metadata read hostname 'smithi114' 2024-04-23T10:26:08.188 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:08.181+0000 7f980337b740 1 --2- 172.21.15.114:0/218424096 >> [v2:172.21.15.119:6834/2033916709,v1:172.21.15.119:6835/2033916709] conn(0x55e2d9878010 0x55e2d9879480 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T10:26:08.188 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:08.181+0000 7f980337b740 4 client.15066 mds_command: new command op to 24479 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-23T10:26:08.188 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:08.181+0000 7f980337b740 1 -- 172.21.15.114:0/218424096 --> [v2:172.21.15.119:6834/2033916709,v1:172.21.15.119:6835/2033916709] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55e2d97d3670 con 0x55e2d9878010 2024-04-23T10:26:08.188 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:08.181+0000 7f97fdbff640 1 --2- 172.21.15.114:0/218424096 >> [v2:172.21.15.119:6834/2033916709,v1:172.21.15.119:6835/2033916709] conn(0x55e2d9878010 0x55e2d9879480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T10:26:08.188 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:08.181+0000 7f97fdbff640 1 --2- 172.21.15.114:0/218424096 >> [v2:172.21.15.119:6834/2033916709,v1:172.21.15.119:6835/2033916709] conn(0x55e2d9878010 0x55e2d9879480 crc :-1 s=READY pgs=15 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.2 client_cookie=e983596fb6bfaa64 server_cookie=e0e7c4149c58e442 in_seq=0 out_seq=0 2024-04-23T10:26:08.188 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:08.181+0000 7f97ddffb640 10 client.15066 ms_handle_connect on v2:172.21.15.119:6834/2033916709 2024-04-23T10:26:08.188 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:08.182+0000 7f97ddffb640 1 -- 172.21.15.114:0/218424096 <== mds.2 v2:172.21.15.119:6834/2033916709 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x55e2d97d3670 con 0x55e2d9878010 2024-04-23T10:26:08.188 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:08.182+0000 7f97ddffb640 10 client.15066 handle_command_reply: tid=0 2024-04-23T10:26:08.188 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:08.182+0000 7f980337b740 1 client.15066 shutdown 2024-04-23T10:26:08.189 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:08.182+0000 7f980337b740 1 -- 172.21.15.114:0/218424096 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55e2d9861d50 msgr2=0x55e2d986b510 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T10:26:08.189 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:08.182+0000 7f980337b740 1 --2- 172.21.15.114:0/218424096 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55e2d9861d50 0x55e2d986b510 secure :-1 s=READY pgs=331 cs=0 l=1 rev1=1 crypto rx=0x7f97e8012740 tx=0x7f97e8012c00 comp rx=0 tx=0).stop 2024-04-23T10:26:08.189 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:08.182+0000 7f980337b740 1 -- 172.21.15.114:0/218424096 shutdown_connections 2024-04-23T10:26:08.189 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:08.182+0000 7f980337b740 1 --2- 172.21.15.114:0/218424096 >> [v2:172.21.15.119:6834/2033916709,v1:172.21.15.119:6835/2033916709] conn(0x55e2d9878010 0x55e2d9879480 crc :-1 s=READY pgs=15 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:26:08.189 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:08.182+0000 7f980337b740 1 --2- 172.21.15.114:0/218424096 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x55e2d9862710 0x55e2d9872550 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:26:08.189 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:08.182+0000 7f97ddffb640 0 client.15066 ms_handle_reset on v2:172.21.15.119:6834/2033916709 2024-04-23T10:26:08.189 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:08.182+0000 7f980337b740 1 --2- 172.21.15.114:0/218424096 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55e2d9861d50 0x55e2d986b510 unknown :-1 s=CLOSED pgs=331 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:26:08.189 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:08.182+0000 7f980337b740 1 --2- 172.21.15.114:0/218424096 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x55e2d9860b80 0x55e2d986afd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:26:08.189 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:08.182+0000 7f980337b740 1 -- 172.21.15.114:0/218424096 >> 172.21.15.114:0/218424096 conn(0x55e2d985c070 msgr2=0x55e2d985b1d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T10:26:08.189 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:08.182+0000 7f980337b740 1 -- 172.21.15.114:0/218424096 shutdown_connections 2024-04-23T10:26:08.189 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:08.182+0000 7f980337b740 1 -- 172.21.15.114:0/218424096 wait complete. 2024-04-23T10:26:08.189 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:08.182+0000 7f980337b740 20 client.15066 trim_cache size 0 max 16384 2024-04-23T10:26:08.204 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:08.200+0000 7f980337b740 1 Processor -- start 2024-04-23T10:26:08.204 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:08.200+0000 7f980337b740 1 -- start start 2024-04-23T10:26:08.204 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:08.200+0000 7f980337b740 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55e2d9860b80 0x55e2d985ef50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T10:26:08.205 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:08.200+0000 7f980337b740 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x55e2d9861d50 0x55e2d9872510 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T10:26:08.205 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:08.200+0000 7f980337b740 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x55e2d9862710 0x55e2d985f490 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T10:26:08.205 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:08.200+0000 7f980337b740 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x55e2d98777c0 con 0x55e2d9860b80 2024-04-23T10:26:08.205 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:08.200+0000 7f980337b740 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 -- 0x55e2d9877e90 con 0x55e2d9861d50 2024-04-23T10:26:08.205 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:08.200+0000 7f980337b740 1 -- --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_getmap magic: 0 -- 0x55e2d9878560 con 0x55e2d9862710 2024-04-23T10:26:08.205 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:08.200+0000 7f97fdbff640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55e2d9860b80 0x55e2d985ef50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T10:26:08.205 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:08.200+0000 7f97fdbff640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55e2d9860b80 0x55e2d985ef50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.114:58130/0 (socket says 172.21.15.114:58130) 2024-04-23T10:26:08.205 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:08.200+0000 7f97fdbff640 1 -- 172.21.15.114:0/1883741841 learned_addr learned my addr 172.21.15.114:0/1883741841 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-04-23T10:26:08.205 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:08.200+0000 7f97fe400640 1 --2- 172.21.15.114:0/1883741841 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x55e2d9861d50 0x55e2d9872510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T10:26:08.205 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:08.200+0000 7f97fdbff640 1 -- 172.21.15.114:0/1883741841 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x55e2d9862710 msgr2=0x55e2d985f490 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T10:26:08.205 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:08.200+0000 7f97fdbff640 1 --2- 172.21.15.114:0/1883741841 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x55e2d9862710 0x55e2d985f490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:26:08.205 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:08.200+0000 7f97fdbff640 1 -- 172.21.15.114:0/1883741841 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x55e2d9861d50 msgr2=0x55e2d9872510 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T10:26:08.205 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:08.200+0000 7f97fdbff640 1 --2- 172.21.15.114:0/1883741841 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x55e2d9861d50 0x55e2d9872510 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:26:08.205 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:08.200+0000 7f97fdbff640 1 -- 172.21.15.114:0/1883741841 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55e2d985cd40 con 0x55e2d9860b80 2024-04-23T10:26:08.206 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:08.200+0000 7f97fe400640 1 --2- 172.21.15.114:0/1883741841 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x55e2d9861d50 0x55e2d9872510 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-23T10:26:08.206 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:08.201+0000 7f97fdbff640 1 --2- 172.21.15.114:0/1883741841 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55e2d9860b80 0x55e2d985ef50 secure :-1 s=READY pgs=332 cs=0 l=1 rev1=1 crypto rx=0x7f97ec009dd0 tx=0x7f97ec003d50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e0aac23bd2c45049 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T10:26:08.206 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:08.201+0000 7f97dffff640 1 -- 172.21.15.114:0/1883741841 <== mon.0 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f97ec030070 con 0x55e2d9860b80 2024-04-23T10:26:08.206 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:08.201+0000 7f97dffff640 1 -- 172.21.15.114:0/1883741841 <== mon.0 v2:172.21.15.114:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f97ec003e50 con 0x55e2d9860b80 2024-04-23T10:26:08.206 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:08.201+0000 7f97dffff640 1 -- 172.21.15.114:0/1883741841 <== mon.0 v2:172.21.15.114:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f97ec02b7b0 con 0x55e2d9860b80 2024-04-23T10:26:08.206 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:08.201+0000 7f980337b740 1 -- 172.21.15.114:0/1883741841 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55e2d9860b80 msgr2=0x55e2d985ef50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T10:26:08.206 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:08.201+0000 7f980337b740 1 --2- 172.21.15.114:0/1883741841 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55e2d9860b80 0x55e2d985ef50 secure :-1 s=READY pgs=332 cs=0 l=1 rev1=1 crypto rx=0x7f97ec009dd0 tx=0x7f97ec003d50 comp rx=0 tx=0).stop 2024-04-23T10:26:08.206 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:08.201+0000 7f980337b740 1 -- 172.21.15.114:0/1883741841 shutdown_connections 2024-04-23T10:26:08.206 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:08.201+0000 7f980337b740 1 --2- 172.21.15.114:0/1883741841 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x55e2d9862710 0x55e2d985f490 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:26:08.206 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:08.201+0000 7f980337b740 1 --2- 172.21.15.114:0/1883741841 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x55e2d9861d50 0x55e2d9872510 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:26:08.206 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:08.201+0000 7f980337b740 1 --2- 172.21.15.114:0/1883741841 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55e2d9860b80 0x55e2d985ef50 unknown :-1 s=CLOSED pgs=332 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:26:08.206 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:08.201+0000 7f980337b740 1 -- 172.21.15.114:0/1883741841 >> 172.21.15.114:0/1883741841 conn(0x55e2d985c070 msgr2=0x55e2d98570b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T10:26:08.206 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:08.201+0000 7f980337b740 1 -- 172.21.15.114:0/1883741841 shutdown_connections 2024-04-23T10:26:08.207 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:08.201+0000 7f980337b740 1 -- 172.21.15.114:0/1883741841 wait complete. 2024-04-23T10:26:08.207 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:08.202+0000 7f980337b740 1 -- start start 2024-04-23T10:26:08.207 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:08.202+0000 7f980337b740 1 Processor -- start 2024-04-23T10:26:08.207 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:08.202+0000 7f980337b740 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55e2d9860b80 0x55e2d986dd10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T10:26:08.207 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:08.202+0000 7f980337b740 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x55e2d9861d50 0x55e2d986e250 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T10:26:08.207 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:08.202+0000 7f980337b740 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x55e2d9862710 0x55e2d9871270 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T10:26:08.207 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:08.202+0000 7f980337b740 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x55e2d9879300 con 0x55e2d9860b80 2024-04-23T10:26:08.207 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:08.202+0000 7f980337b740 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 -- 0x55e2d9878c30 con 0x55e2d9861d50 2024-04-23T10:26:08.207 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:08.202+0000 7f980337b740 1 -- --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_getmap magic: 0 -- 0x55e2d98799d0 con 0x55e2d9862710 2024-04-23T10:26:08.207 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:08.202+0000 7f980337b740 10 client.0 _ll_register_callbacks cb 0x7f97e8009930 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-23T10:26:08.207 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:08.202+0000 7f97fdbff640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55e2d9860b80 0x55e2d986dd10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T10:26:08.207 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:08.202+0000 7f97fdbff640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55e2d9860b80 0x55e2d986dd10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.114:58142/0 (socket says 172.21.15.114:58142) 2024-04-23T10:26:08.207 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:08.202+0000 7f97fdbff640 1 -- 172.21.15.114:0/2458904929 learned_addr learned my addr 172.21.15.114:0/2458904929 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-04-23T10:26:08.207 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:08.202+0000 7f97fe400640 1 --2- 172.21.15.114:0/2458904929 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x55e2d9861d50 0x55e2d986e250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T10:26:08.208 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:08.202+0000 7f97fdbff640 1 --2- 172.21.15.114:0/2458904929 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x55e2d9862710 0x55e2d9871270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T10:26:08.208 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:08.202+0000 7f97fdbff640 1 -- 172.21.15.114:0/2458904929 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x55e2d9862710 msgr2=0x55e2d9871270 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T10:26:08.208 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:08.202+0000 7f97fdbff640 1 --2- 172.21.15.114:0/2458904929 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x55e2d9862710 0x55e2d9871270 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:26:08.208 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:08.202+0000 7f97fdbff640 1 -- 172.21.15.114:0/2458904929 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x55e2d9861d50 msgr2=0x55e2d986e250 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T10:26:08.208 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:08.202+0000 7f97fdbff640 1 --2- 172.21.15.114:0/2458904929 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x55e2d9861d50 0x55e2d986e250 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:26:08.208 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:08.202+0000 7f97fdbff640 1 -- 172.21.15.114:0/2458904929 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55e2d9871860 con 0x55e2d9860b80 2024-04-23T10:26:08.208 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:08.202+0000 7f97fdbff640 1 --2- 172.21.15.114:0/2458904929 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55e2d9860b80 0x55e2d986dd10 secure :-1 s=READY pgs=333 cs=0 l=1 rev1=1 crypto rx=0x7f97ec009da0 tx=0x7f97ec005f90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=498a7f47212b718e server_cookie=0 in_seq=0 out_seq=0 2024-04-23T10:26:08.208 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:08.202+0000 7f97ddffb640 10 client.0 ms_handle_connect on v2:172.21.15.114:3300/0 2024-04-23T10:26:08.208 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:08.203+0000 7f97ddffb640 1 -- 172.21.15.114:0/2458904929 <== mon.0 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f97ec030040 con 0x55e2d9860b80 2024-04-23T10:26:08.208 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:08.203+0000 7f97ddffb640 1 -- 172.21.15.114:0/2458904929 <== mon.0 v2:172.21.15.114:3300/0 2 ==== osd_map(89..89 src has 1..89) ==== 8269+0+0 (secure 0 0 0) 0x7f97ec005270 con 0x55e2d9860b80 2024-04-23T10:26:08.208 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:08.203+0000 7f980337b740 1 -- 172.21.15.114:0/2458904929 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f97ec003e50 con 0x55e2d9860b80 2024-04-23T10:26:08.208 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:08.203+0000 7f97ddffb640 1 -- 172.21.15.114:0/2458904929 <== mon.0 v2:172.21.15.114:3300/0 3 ==== mon_get_version_reply(handle=1 version=20) ==== 24+0+0 (secure 0 0 0) 0x7f97ec005470 con 0x55e2d9860b80 2024-04-23T10:26:08.208 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:08.203+0000 7f980337b740 10 client.15072 fetch_fsmap learned FSMap version 20 2024-04-23T10:26:08.208 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:08.203+0000 7f980337b740 1 -- 172.21.15.114:0/2458904929 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({config=0+,fsmap=20,monmap=4+}) -- 0x55e2d98de080 con 0x55e2d9860b80 2024-04-23T10:26:08.209 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:08.203+0000 7f97ddffb640 1 -- 172.21.15.114:0/2458904929 <== mon.0 v2:172.21.15.114:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f97ec003e50 con 0x55e2d9860b80 2024-04-23T10:26:08.209 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:08.203+0000 7f97ddffb640 1 -- 172.21.15.114:0/2458904929 <== mon.0 v2:172.21.15.114:3300/0 5 ==== fsmap(e 20) ==== 8338+0+0 (secure 0 0 0) 0x7f97ec00f040 con 0x55e2d9860b80 2024-04-23T10:26:08.209 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:08.203+0000 7f980337b740 10 client.15072 fetch_fsmap finished waiting for FSMap version 20 2024-04-23T10:26:08.209 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:08.203+0000 7f980337b740 10 client.15072 resolve_mds: resolved name 'e' to daemon mds.e 2024-04-23T10:26:08.209 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:08.203+0000 7f980337b740 20 client.15072 populate_metadata read hostname 'smithi114' 2024-04-23T10:26:08.209 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:08.203+0000 7f980337b740 1 --2- 172.21.15.114:0/2458904929 >> [v2:172.21.15.119:6834/2033916709,v1:172.21.15.119:6835/2033916709] conn(0x55e2d98e2510 0x55e2d98e1220 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T10:26:08.209 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:08.203+0000 7f980337b740 4 client.15072 mds_command: new command op to 24479 tid=0 multi_id=0 [{"prefix": "status"}] 2024-04-23T10:26:08.209 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:08.203+0000 7f980337b740 1 -- 172.21.15.114:0/2458904929 --> [v2:172.21.15.119:6834/2033916709,v1:172.21.15.119:6835/2033916709] -- command(tid 0: {"prefix": "status"}) -- 0x55e2d97cd4e0 con 0x55e2d98e2510 2024-04-23T10:26:08.209 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:08.203+0000 7f97fe400640 1 --2- 172.21.15.114:0/2458904929 >> [v2:172.21.15.119:6834/2033916709,v1:172.21.15.119:6835/2033916709] conn(0x55e2d98e2510 0x55e2d98e1220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T10:26:08.209 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:08.204+0000 7f97fe400640 1 --2- 172.21.15.114:0/2458904929 >> [v2:172.21.15.119:6834/2033916709,v1:172.21.15.119:6835/2033916709] conn(0x55e2d98e2510 0x55e2d98e1220 crc :-1 s=READY pgs=16 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.2 client_cookie=b6c8e1ad218f8bb0 server_cookie=f4bd4026968c0bce in_seq=0 out_seq=0 2024-04-23T10:26:08.209 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:08.204+0000 7f97ddffb640 10 client.15072 ms_handle_connect on v2:172.21.15.119:6834/2033916709 2024-04-23T10:26:08.209 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:08.204+0000 7f97ddffb640 1 -- 172.21.15.114:0/2458904929 <== mds.2 v2:172.21.15.119:6834/2033916709 1 ==== command_reply(tid 0: 0 ) ==== 8+0+327 (crc 0 0 0) 0x55e2d97cd4e0 con 0x55e2d98e2510 2024-04-23T10:26:08.209 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:08.204+0000 7f97ddffb640 10 client.15072 handle_command_reply: tid=0 2024-04-23T10:26:08.209 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:08.204+0000 7f980337b740 1 client.15072 shutdown 2024-04-23T10:26:08.209 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:08.204+0000 7f980337b740 1 -- 172.21.15.114:0/2458904929 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55e2d9860b80 msgr2=0x55e2d986dd10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T10:26:08.209 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:08.204+0000 7f980337b740 1 --2- 172.21.15.114:0/2458904929 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55e2d9860b80 0x55e2d986dd10 secure :-1 s=READY pgs=333 cs=0 l=1 rev1=1 crypto rx=0x7f97ec009da0 tx=0x7f97ec005f90 comp rx=0 tx=0).stop 2024-04-23T10:26:08.210 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:08.204+0000 7f980337b740 1 -- 172.21.15.114:0/2458904929 shutdown_connections 2024-04-23T10:26:08.210 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:08.204+0000 7f980337b740 1 --2- 172.21.15.114:0/2458904929 >> [v2:172.21.15.119:6834/2033916709,v1:172.21.15.119:6835/2033916709] conn(0x55e2d98e2510 0x55e2d98e1220 crc :-1 s=READY pgs=16 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:26:08.210 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:08.204+0000 7f980337b740 1 --2- 172.21.15.114:0/2458904929 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x55e2d9862710 0x55e2d9871270 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:26:08.210 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:08.204+0000 7f980337b740 1 --2- 172.21.15.114:0/2458904929 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x55e2d9861d50 0x55e2d986e250 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:26:08.210 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:08.204+0000 7f97ddffb640 0 client.15072 ms_handle_reset on v2:172.21.15.119:6834/2033916709 2024-04-23T10:26:08.210 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:08.204+0000 7f980337b740 1 --2- 172.21.15.114:0/2458904929 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55e2d9860b80 0x55e2d986dd10 unknown :-1 s=CLOSED pgs=333 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:26:08.210 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:08.204+0000 7f980337b740 1 -- 172.21.15.114:0/2458904929 >> 172.21.15.114:0/2458904929 conn(0x55e2d985c070 msgr2=0x55e2d985a940 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T10:26:08.210 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:08.204+0000 7f980337b740 1 -- 172.21.15.114:0/2458904929 shutdown_connections 2024-04-23T10:26:08.210 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:08.204+0000 7f980337b740 1 -- 172.21.15.114:0/2458904929 wait complete. 2024-04-23T10:26:08.210 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:08.204+0000 7f980337b740 20 client.15072 trim_cache size 0 max 16384 2024-04-23T10:26:08.211 INFO:teuthology.orchestra.run.smithi114.stdout:{ 2024-04-23T10:26:08.211 INFO:teuthology.orchestra.run.smithi114.stdout: "cluster_fsid": "aa240572-0159-11ef-bc93-c7b262605968", 2024-04-23T10:26:08.211 INFO:teuthology.orchestra.run.smithi114.stdout: "whoami": 2, 2024-04-23T10:26:08.211 INFO:teuthology.orchestra.run.smithi114.stdout: "id": 24479, 2024-04-23T10:26:08.211 INFO:teuthology.orchestra.run.smithi114.stdout: "want_state": "up:active", 2024-04-23T10:26:08.211 INFO:teuthology.orchestra.run.smithi114.stdout: "state": "up:active", 2024-04-23T10:26:08.211 INFO:teuthology.orchestra.run.smithi114.stdout: "fs_name": "cephfs", 2024-04-23T10:26:08.211 INFO:teuthology.orchestra.run.smithi114.stdout: "rank_uptime": 80.859869281000002, 2024-04-23T10:26:08.211 INFO:teuthology.orchestra.run.smithi114.stdout: "mdsmap_epoch": 20, 2024-04-23T10:26:08.211 INFO:teuthology.orchestra.run.smithi114.stdout: "osdmap_epoch": 89, 2024-04-23T10:26:08.211 INFO:teuthology.orchestra.run.smithi114.stdout: "osdmap_epoch_barrier": 87, 2024-04-23T10:26:08.211 INFO:teuthology.orchestra.run.smithi114.stdout: "uptime": 109.063084186 2024-04-23T10:26:08.211 INFO:teuthology.orchestra.run.smithi114.stdout:} 2024-04-23T10:26:08.212 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:08.204+0000 7f9800664640 1 -- 172.21.15.114:0/3869408247 >> [v2:172.21.15.114:6800/465333808,v1:172.21.15.114:6801/465333808] conn(0x7f97e00a3c40 msgr2=0x7f97e00a6100 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T10:26:08.212 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:08.204+0000 7f9800664640 1 --2- 172.21.15.114:0/3869408247 >> [v2:172.21.15.114:6800/465333808,v1:172.21.15.114:6801/465333808] conn(0x7f97e00a3c40 0x7f97e00a6100 secure :-1 s=READY pgs=223 cs=0 l=1 rev1=1 crypto rx=0x7f97f00bd580 tx=0x7f97f008c280 comp rx=0 tx=0).stop 2024-04-23T10:26:08.212 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:08.204+0000 7f9800664640 1 -- 172.21.15.114:0/3869408247 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f97f8109e40 msgr2=0x7f97f81a1900 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T10:26:08.212 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:08.205+0000 7f9800664640 1 --2- 172.21.15.114:0/3869408247 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f97f8109e40 0x7f97f81a1900 secure :-1 s=READY pgs=329 cs=0 l=1 rev1=1 crypto rx=0x7f97f0006010 tx=0x7f97f002fc20 comp rx=0 tx=0).stop 2024-04-23T10:26:08.212 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:08.205+0000 7f9800664640 1 -- 172.21.15.114:0/3869408247 shutdown_connections 2024-04-23T10:26:08.213 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:08.205+0000 7f9800664640 1 --2- 172.21.15.114:0/3869408247 >> [v2:172.21.15.114:6800/465333808,v1:172.21.15.114:6801/465333808] conn(0x7f97e00a3c40 0x7f97e00a6100 unknown :-1 s=CLOSED pgs=223 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:26:08.213 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:08.205+0000 7f9800664640 1 --2- 172.21.15.114:0/3869408247 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f97f8109e40 0x7f97f81a1900 unknown :-1 s=CLOSED pgs=329 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:26:08.213 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:08.205+0000 7f9800664640 1 --2- 172.21.15.114:0/3869408247 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f97f8075f10 0x7f97f819a890 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:26:08.213 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:08.205+0000 7f9800664640 1 --2- 172.21.15.114:0/3869408247 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f97f8074930 0x7f97f819a350 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:26:08.213 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:08.205+0000 7f9800664640 1 -- 172.21.15.114:0/3869408247 >> 172.21.15.114:0/3869408247 conn(0x7f97f80fd500 msgr2=0x7f97f8105dd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T10:26:08.213 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:08.205+0000 7f9800664640 1 -- 172.21.15.114:0/3869408247 shutdown_connections 2024-04-23T10:26:08.213 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:08.205+0000 7f9800664640 1 -- 172.21.15.114:0/3869408247 wait complete. 2024-04-23T10:26:08.214 DEBUG:teuthology.orchestra.run.smithi114:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.h status 2024-04-23T10:26:08.313 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:08.314+0000 7f72acb4a640 1 -- 172.21.15.114:0/3660434367 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f72a8101fa0 msgr2=0x7f72a81023c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T10:26:08.313 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:08.314+0000 7f72acb4a640 1 --2- 172.21.15.114:0/3660434367 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f72a8101fa0 0x7f72a81023c0 secure :-1 s=READY pgs=334 cs=0 l=1 rev1=1 crypto rx=0x7f7298009900 tx=0x7f729802a410 comp rx=0 tx=0).stop 2024-04-23T10:26:08.313 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:08.314+0000 7f72acb4a640 1 -- 172.21.15.114:0/3660434367 shutdown_connections 2024-04-23T10:26:08.313 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:08.314+0000 7f72acb4a640 1 --2- 172.21.15.114:0/3660434367 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f72a8103b50 0x7f72a810a3e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:26:08.313 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:08.314+0000 7f72acb4a640 1 --2- 172.21.15.114:0/3660434367 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f72a8103190 0x7f72a8103610 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:26:08.313 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:08.314+0000 7f72acb4a640 1 --2- 172.21.15.114:0/3660434367 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f72a8101fa0 0x7f72a81023c0 unknown :-1 s=CLOSED pgs=334 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:26:08.313 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:08.314+0000 7f72acb4a640 1 -- 172.21.15.114:0/3660434367 >> 172.21.15.114:0/3660434367 conn(0x7f72a80fd530 msgr2=0x7f72a80ff950 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T10:26:08.314 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:08.314+0000 7f72acb4a640 1 -- 172.21.15.114:0/3660434367 shutdown_connections 2024-04-23T10:26:08.314 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:08.314+0000 7f72acb4a640 1 -- 172.21.15.114:0/3660434367 wait complete. 2024-04-23T10:26:08.316 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:08.315+0000 7f72acb4a640 1 Processor -- start 2024-04-23T10:26:08.316 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:08.315+0000 7f72acb4a640 1 -- start start 2024-04-23T10:26:08.316 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:08.315+0000 7f72acb4a640 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f72a8101fa0 0x7f72a81981f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T10:26:08.316 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:08.315+0000 7f72acb4a640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f72a8103190 0x7f72a8198730 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T10:26:08.316 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:08.315+0000 7f72acb4a640 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f72a8103b50 0x7f72a819f7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T10:26:08.316 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:08.315+0000 7f72acb4a640 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x7f72a810dae0 con 0x7f72a8103190 2024-04-23T10:26:08.316 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:08.315+0000 7f72acb4a640 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 -- 0x7f72a810d960 con 0x7f72a8103b50 2024-04-23T10:26:08.316 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:08.315+0000 7f72acb4a640 1 -- --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_getmap magic: 0 -- 0x7f72a810dc60 con 0x7f72a8101fa0 2024-04-23T10:26:08.316 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:08.315+0000 7f72a5d9b640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f72a8103190 0x7f72a8198730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T10:26:08.316 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:08.315+0000 7f72a5d9b640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f72a8103190 0x7f72a8198730 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.114:58162/0 (socket says 172.21.15.114:58162) 2024-04-23T10:26:08.316 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:08.315+0000 7f72a5d9b640 1 -- 172.21.15.114:0/4103576177 learned_addr learned my addr 172.21.15.114:0/4103576177 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-04-23T10:26:08.316 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:08.315+0000 7f72a659c640 1 --2- 172.21.15.114:0/4103576177 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f72a8101fa0 0x7f72a81981f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T10:26:08.316 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:08.315+0000 7f72a6d9d640 1 --2- 172.21.15.114:0/4103576177 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f72a8103b50 0x7f72a819f7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T10:26:08.316 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:08.315+0000 7f72a5d9b640 1 -- 172.21.15.114:0/4103576177 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f72a8101fa0 msgr2=0x7f72a81981f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T10:26:08.317 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:08.315+0000 7f72a5d9b640 1 --2- 172.21.15.114:0/4103576177 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f72a8101fa0 0x7f72a81981f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:26:08.317 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:08.315+0000 7f72a5d9b640 1 -- 172.21.15.114:0/4103576177 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f72a8103b50 msgr2=0x7f72a819f7a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T10:26:08.317 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:08.315+0000 7f72a5d9b640 1 --2- 172.21.15.114:0/4103576177 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f72a8103b50 0x7f72a819f7a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:26:08.317 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:08.315+0000 7f72a5d9b640 1 -- 172.21.15.114:0/4103576177 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f72a819fe10 con 0x7f72a8103190 2024-04-23T10:26:08.317 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:08.315+0000 7f72a5d9b640 1 --2- 172.21.15.114:0/4103576177 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f72a8103190 0x7f72a8198730 secure :-1 s=READY pgs=335 cs=0 l=1 rev1=1 crypto rx=0x7f729000d770 tx=0x7f729000dc30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T10:26:08.317 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:08.315+0000 7f729f7fe640 1 -- 172.21.15.114:0/4103576177 <== mon.0 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7290017070 con 0x7f72a8103190 2024-04-23T10:26:08.317 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:08.316+0000 7f729f7fe640 1 -- 172.21.15.114:0/4103576177 <== mon.0 v2:172.21.15.114:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f72900054d0 con 0x7f72a8103190 2024-04-23T10:26:08.317 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:08.316+0000 7f72acb4a640 1 -- 172.21.15.114:0/4103576177 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f72a81a0100 con 0x7f72a8103190 2024-04-23T10:26:08.317 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:08.316+0000 7f729f7fe640 1 -- 172.21.15.114:0/4103576177 <== mon.0 v2:172.21.15.114:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7290004c30 con 0x7f72a8103190 2024-04-23T10:26:08.317 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:08.316+0000 7f72acb4a640 1 -- 172.21.15.114:0/4103576177 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f72a81a0600 con 0x7f72a8103190 2024-04-23T10:26:08.323 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:08.318+0000 7f729f7fe640 1 -- 172.21.15.114:0/4103576177 <== mon.0 v2:172.21.15.114:3300/0 4 ==== mgrmap(e 17) ==== 138698+0+0 (secure 0 0 0) 0x7f7290004e10 con 0x7f72a8103190 2024-04-23T10:26:08.323 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:08.318+0000 7f72af85c740 1 Processor -- start 2024-04-23T10:26:08.323 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:08.318+0000 7f72af85c740 1 -- start start 2024-04-23T10:26:08.324 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:08.318+0000 7f72af85c740 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x55845be28f90 0x55845be293b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T10:26:08.324 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:08.318+0000 7f72af85c740 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x55845be2a160 0x55845be2a5e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T10:26:08.324 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:08.318+0000 7f72af85c740 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55845be2ab20 0x55845be313b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T10:26:08.324 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:08.318+0000 7f72af85c740 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x55845bd89c50 con 0x55845be2ab20 2024-04-23T10:26:08.324 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:08.318+0000 7f72af85c740 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 -- 0x55845bd26830 con 0x55845be2a160 2024-04-23T10:26:08.324 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:08.318+0000 7f72af85c740 1 -- --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_getmap magic: 0 -- 0x55845bd85c10 con 0x55845be28f90 2024-04-23T10:26:08.324 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:08.318+0000 7f72a5d9b640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55845be2ab20 0x55845be313b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T10:26:08.324 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:08.318+0000 7f72a5d9b640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55845be2ab20 0x55845be313b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.114:58170/0 (socket says 172.21.15.114:58170) 2024-04-23T10:26:08.324 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:08.318+0000 7f72a5d9b640 1 -- 172.21.15.114:0/2226427644 learned_addr learned my addr 172.21.15.114:0/2226427644 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-04-23T10:26:08.324 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:08.318+0000 7f729f7fe640 1 --2- 172.21.15.114:0/4103576177 >> [v2:172.21.15.114:6800/465333808,v1:172.21.15.114:6801/465333808] conn(0x7f72940a3c40 0x7f72940a6100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T10:26:08.324 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:08.319+0000 7f72a659c640 1 --2- 172.21.15.114:0/2226427644 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x55845be2a160 0x55845be2a5e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T10:26:08.324 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:08.319+0000 7f72a6d9d640 1 --2- 172.21.15.114:0/2226427644 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x55845be28f90 0x55845be293b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T10:26:08.324 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:08.319+0000 7f72a6d9d640 1 --2- 172.21.15.114:0/4103576177 >> [v2:172.21.15.114:6800/465333808,v1:172.21.15.114:6801/465333808] conn(0x7f72940a3c40 0x7f72940a6100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T10:26:08.324 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:08.319+0000 7f72a5d9b640 1 -- 172.21.15.114:0/2226427644 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x55845be28f90 msgr2=0x55845be293b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T10:26:08.324 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:08.319+0000 7f72a5d9b640 1 --2- 172.21.15.114:0/2226427644 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x55845be28f90 0x55845be293b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:26:08.325 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:08.319+0000 7f72a5d9b640 1 -- 172.21.15.114:0/2226427644 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x55845be2a160 msgr2=0x55845be2a5e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T10:26:08.325 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:08.319+0000 7f72a5d9b640 1 --2- 172.21.15.114:0/2226427644 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x55845be2a160 0x55845be2a5e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:26:08.325 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:08.319+0000 7f72a5d9b640 1 -- 172.21.15.114:0/2226427644 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55845ba9c2d0 con 0x55845be2ab20 2024-04-23T10:26:08.325 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:08.319+0000 7f72a659c640 1 --2- 172.21.15.114:0/2226427644 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x55845be2a160 0x55845be2a5e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-23T10:26:08.325 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:08.319+0000 7f729f7fe640 1 -- 172.21.15.114:0/4103576177 <== mon.0 v2:172.21.15.114:3300/0 5 ==== osd_map(89..89 src has 1..89) ==== 8269+0+0 (secure 0 0 0) 0x7f72900cffc0 con 0x7f72a8103190 2024-04-23T10:26:08.325 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:08.319+0000 7f72a5d9b640 1 --2- 172.21.15.114:0/2226427644 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55845be2ab20 0x55845be313b0 secure :-1 s=READY pgs=336 cs=0 l=1 rev1=1 crypto rx=0x7f72900cd8c0 tx=0x7f729009be40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3ac287a807a6ee22 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T10:26:08.325 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:08.319+0000 7f72a6d9d640 1 --2- 172.21.15.114:0/4103576177 >> [v2:172.21.15.114:6800/465333808,v1:172.21.15.114:6801/465333808] conn(0x7f72940a3c40 0x7f72940a6100 secure :-1 s=READY pgs=224 cs=0 l=1 rev1=1 crypto rx=0x7f72a0005c40 tx=0x7f72a0009500 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T10:26:08.325 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:08.319+0000 7f729cff9640 1 -- 172.21.15.114:0/2226427644 <== mon.0 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7290020030 con 0x55845be2ab20 2024-04-23T10:26:08.325 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:08.319+0000 7f729cff9640 1 -- 172.21.15.114:0/2226427644 <== mon.0 v2:172.21.15.114:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f72900a4d90 con 0x55845be2ab20 2024-04-23T10:26:08.325 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:08.319+0000 7f729cff9640 1 -- 172.21.15.114:0/2226427644 <== mon.0 v2:172.21.15.114:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f72900cdba0 con 0x55845be2ab20 2024-04-23T10:26:08.325 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:08.320+0000 7f72af85c740 1 -- 172.21.15.114:0/2226427644 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55845be2ab20 msgr2=0x55845be313b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T10:26:08.325 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:08.320+0000 7f72af85c740 1 --2- 172.21.15.114:0/2226427644 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55845be2ab20 0x55845be313b0 secure :-1 s=READY pgs=336 cs=0 l=1 rev1=1 crypto rx=0x7f72900cd8c0 tx=0x7f729009be40 comp rx=0 tx=0).stop 2024-04-23T10:26:08.325 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:08.320+0000 7f72af85c740 1 -- 172.21.15.114:0/2226427644 shutdown_connections 2024-04-23T10:26:08.325 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:08.320+0000 7f72af85c740 1 --2- 172.21.15.114:0/2226427644 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55845be2ab20 0x55845be313b0 unknown :-1 s=CLOSED pgs=336 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:26:08.325 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:08.320+0000 7f72af85c740 1 --2- 172.21.15.114:0/2226427644 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x55845be2a160 0x55845be2a5e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:26:08.326 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:08.320+0000 7f72af85c740 1 --2- 172.21.15.114:0/2226427644 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x55845be28f90 0x55845be293b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:26:08.326 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:08.320+0000 7f72af85c740 1 -- 172.21.15.114:0/2226427644 >> 172.21.15.114:0/2226427644 conn(0x55845be20070 msgr2=0x55845be20470 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T10:26:08.326 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:08.320+0000 7f72af85c740 1 -- 172.21.15.114:0/2226427644 shutdown_connections 2024-04-23T10:26:08.326 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:08.320+0000 7f72af85c740 1 -- 172.21.15.114:0/2226427644 wait complete. 2024-04-23T10:26:08.326 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:08.320+0000 7f72af85c740 1 -- start start 2024-04-23T10:26:08.326 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:08.320+0000 7f72af85c740 1 Processor -- start 2024-04-23T10:26:08.326 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:08.320+0000 7f72af85c740 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x55845be28f90 0x55845be33320 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T10:26:08.326 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:08.320+0000 7f72af85c740 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55845be2a160 0x55845be33860 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T10:26:08.326 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:08.320+0000 7f72af85c740 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x55845be2ab20 0x55845be3a8a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T10:26:08.326 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:08.321+0000 7f72af85c740 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x55845be31a70 con 0x55845be2a160 2024-04-23T10:26:08.326 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:08.321+0000 7f72af85c740 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 -- 0x55845bd24470 con 0x55845be2ab20 2024-04-23T10:26:08.326 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:08.321+0000 7f72af85c740 1 -- --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_getmap magic: 0 -- 0x55845be31bf0 con 0x55845be28f90 2024-04-23T10:26:08.326 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:08.321+0000 7f72af85c740 10 client.0 _ll_register_callbacks cb 0x55845bd49f30 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-23T10:26:08.326 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:08.321+0000 7f72a5d9b640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55845be2a160 0x55845be33860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T10:26:08.327 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:08.321+0000 7f72a5d9b640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55845be2a160 0x55845be33860 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.114:58180/0 (socket says 172.21.15.114:58180) 2024-04-23T10:26:08.327 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:08.321+0000 7f72a5d9b640 1 -- 172.21.15.114:0/3672356447 learned_addr learned my addr 172.21.15.114:0/3672356447 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-04-23T10:26:08.327 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:08.321+0000 7f72a5d9b640 1 -- 172.21.15.114:0/3672356447 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x55845be28f90 msgr2=0x55845be33320 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T10:26:08.327 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:08.321+0000 7f72a6d9d640 1 --2- 172.21.15.114:0/3672356447 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x55845be2ab20 0x55845be3a8a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T10:26:08.327 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:08.321+0000 7f72a5d9b640 1 --2- 172.21.15.114:0/3672356447 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x55845be28f90 0x55845be33320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:26:08.327 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:08.321+0000 7f72a5d9b640 1 -- 172.21.15.114:0/3672356447 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x55845be2ab20 msgr2=0x55845be3a8a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T10:26:08.327 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:08.321+0000 7f72a5d9b640 1 --2- 172.21.15.114:0/3672356447 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x55845be2ab20 0x55845be3a8a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:26:08.327 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:08.321+0000 7f72a5d9b640 1 -- 172.21.15.114:0/3672356447 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55845bb666b0 con 0x55845be2a160 2024-04-23T10:26:08.327 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:08.321+0000 7f72a6d9d640 1 --2- 172.21.15.114:0/3672356447 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x55845be2ab20 0x55845be3a8a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-23T10:26:08.327 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:08.321+0000 7f72a5d9b640 1 --2- 172.21.15.114:0/3672356447 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55845be2a160 0x55845be33860 secure :-1 s=READY pgs=337 cs=0 l=1 rev1=1 crypto rx=0x7f7290012040 tx=0x7f729009c1d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8653993e1b975b40 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T10:26:08.327 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:08.321+0000 7f728dffb640 10 client.0 ms_handle_connect on v2:172.21.15.114:3300/0 2024-04-23T10:26:08.327 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:08.321+0000 7f728dffb640 1 -- 172.21.15.114:0/3672356447 <== mon.0 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7290020070 con 0x55845be2a160 2024-04-23T10:26:08.327 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:08.321+0000 7f728dffb640 1 -- 172.21.15.114:0/3672356447 <== mon.0 v2:172.21.15.114:3300/0 2 ==== osd_map(89..89 src has 1..89) ==== 8269+0+0 (secure 0 0 0) 0x7f72900a00d0 con 0x55845be2a160 2024-04-23T10:26:08.327 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:08.321+0000 7f72af85c740 1 -- 172.21.15.114:0/3672356447 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f72900a4d90 con 0x55845be2a160 2024-04-23T10:26:08.327 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:08.321+0000 7f728dffb640 1 -- 172.21.15.114:0/3672356447 <== mon.0 v2:172.21.15.114:3300/0 3 ==== mon_get_version_reply(handle=1 version=20) ==== 24+0+0 (secure 0 0 0) 0x7f72900a02d0 con 0x55845be2a160 2024-04-23T10:26:08.328 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:08.321+0000 7f72af85c740 10 client.15084 fetch_fsmap learned FSMap version 20 2024-04-23T10:26:08.328 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:08.321+0000 7f72af85c740 1 -- 172.21.15.114:0/3672356447 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({config=0+,fsmap=20,monmap=4+}) -- 0x55845be3dc10 con 0x55845be2a160 2024-04-23T10:26:08.328 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:08.321+0000 7f728dffb640 1 -- 172.21.15.114:0/3672356447 <== mon.0 v2:172.21.15.114:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f72900a4d90 con 0x55845be2a160 2024-04-23T10:26:08.328 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:08.321+0000 7f728dffb640 1 -- 172.21.15.114:0/3672356447 <== mon.0 v2:172.21.15.114:3300/0 5 ==== fsmap(e 20) ==== 8338+0+0 (secure 0 0 0) 0x7f7290097f20 con 0x55845be2a160 2024-04-23T10:26:08.328 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:08.322+0000 7f72af85c740 10 client.15084 fetch_fsmap finished waiting for FSMap version 20 2024-04-23T10:26:08.328 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:08.322+0000 7f72af85c740 10 client.15084 resolve_mds: resolved name 'h' to daemon mds.h 2024-04-23T10:26:08.328 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:08.322+0000 7f72af85c740 20 client.15084 populate_metadata read hostname 'smithi114' 2024-04-23T10:26:08.328 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:08.322+0000 7f72af85c740 1 --2- 172.21.15.114:0/3672356447 >> [v2:172.21.15.119:6836/1872363128,v1:172.21.15.119:6837/1872363128] conn(0x55845be3fdb0 0x55845be41220 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T10:26:08.328 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:08.322+0000 7f72af85c740 4 client.15084 mds_command: new command op to 24509 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-23T10:26:08.328 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:08.322+0000 7f72af85c740 1 -- 172.21.15.114:0/3672356447 --> [v2:172.21.15.119:6836/1872363128,v1:172.21.15.119:6837/1872363128] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55845bd97670 con 0x55845be3fdb0 2024-04-23T10:26:08.328 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:08.322+0000 7f72a659c640 1 --2- 172.21.15.114:0/3672356447 >> [v2:172.21.15.119:6836/1872363128,v1:172.21.15.119:6837/1872363128] conn(0x55845be3fdb0 0x55845be41220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T10:26:08.328 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:08.322+0000 7f728dffb640 10 client.15084 ms_handle_connect on v2:172.21.15.119:6836/1872363128 2024-04-23T10:26:08.328 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:08.322+0000 7f72a659c640 1 --2- 172.21.15.114:0/3672356447 >> [v2:172.21.15.119:6836/1872363128,v1:172.21.15.119:6837/1872363128] conn(0x55845be3fdb0 0x55845be41220 crc :-1 s=READY pgs=16 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.1 client_cookie=8052090092542ad0 server_cookie=f3b0ae7f5a617b75 in_seq=0 out_seq=0 2024-04-23T10:26:08.328 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:08.323+0000 7f728dffb640 1 -- 172.21.15.114:0/3672356447 <== mds.1 v2:172.21.15.119:6836/1872363128 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x55845bd97670 con 0x55845be3fdb0 2024-04-23T10:26:08.328 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:08.323+0000 7f728dffb640 10 client.15084 handle_command_reply: tid=0 2024-04-23T10:26:08.329 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:08.323+0000 7f72af85c740 1 client.15084 shutdown 2024-04-23T10:26:08.329 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:08.323+0000 7f72af85c740 1 -- 172.21.15.114:0/3672356447 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55845be2a160 msgr2=0x55845be33860 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T10:26:08.329 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:08.323+0000 7f72af85c740 1 --2- 172.21.15.114:0/3672356447 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55845be2a160 0x55845be33860 secure :-1 s=READY pgs=337 cs=0 l=1 rev1=1 crypto rx=0x7f7290012040 tx=0x7f729009c1d0 comp rx=0 tx=0).stop 2024-04-23T10:26:08.329 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:08.323+0000 7f72af85c740 1 -- 172.21.15.114:0/3672356447 shutdown_connections 2024-04-23T10:26:08.329 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:08.323+0000 7f72af85c740 1 --2- 172.21.15.114:0/3672356447 >> [v2:172.21.15.119:6836/1872363128,v1:172.21.15.119:6837/1872363128] conn(0x55845be3fdb0 0x55845be41220 crc :-1 s=READY pgs=16 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:26:08.329 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:08.323+0000 7f72af85c740 1 --2- 172.21.15.114:0/3672356447 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x55845be2ab20 0x55845be3a8a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:26:08.329 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:08.323+0000 7f728dffb640 0 client.15084 ms_handle_reset on v2:172.21.15.119:6836/1872363128 2024-04-23T10:26:08.329 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:08.323+0000 7f72af85c740 1 --2- 172.21.15.114:0/3672356447 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55845be2a160 0x55845be33860 unknown :-1 s=CLOSED pgs=337 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:26:08.329 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:08.323+0000 7f72af85c740 1 --2- 172.21.15.114:0/3672356447 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x55845be28f90 0x55845be33320 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:26:08.329 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:08.323+0000 7f72af85c740 1 -- 172.21.15.114:0/3672356447 >> 172.21.15.114:0/3672356447 conn(0x55845be20070 msgr2=0x55845be1f1d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T10:26:08.329 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:08.323+0000 7f72af85c740 1 -- 172.21.15.114:0/3672356447 shutdown_connections 2024-04-23T10:26:08.329 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:08.323+0000 7f72af85c740 1 -- 172.21.15.114:0/3672356447 wait complete. 2024-04-23T10:26:08.329 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:08.323+0000 7f72af85c740 20 client.15084 trim_cache size 0 max 16384 2024-04-23T10:26:08.330 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 23 10:26:08 smithi204 ceph-mon[32649]: from='client.? 172.21.15.114:0/3533348400' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2024-04-23T10:26:08.330 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 23 10:26:08 smithi204 ceph-mon[32649]: from='client.? 172.21.15.114:0/440960242' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-23T10:26:08.330 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 23 10:26:08 smithi204 ceph-mon[32649]: from='client.? 172.21.15.114:0/872498497' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-23T10:26:08.346 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:08.341+0000 7f72af85c740 1 Processor -- start 2024-04-23T10:26:08.346 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:08.341+0000 7f72af85c740 1 -- start start 2024-04-23T10:26:08.346 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:08.341+0000 7f72af85c740 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55845be28f90 0x55845be1f050 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T10:26:08.346 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:08.341+0000 7f72af85c740 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x55845be2a160 0x55845be1f590 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T10:26:08.346 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:08.341+0000 7f72af85c740 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x55845be2ab20 0x55845be3a860 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T10:26:08.346 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:08.341+0000 7f72af85c740 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x55845be3f560 con 0x55845be28f90 2024-04-23T10:26:08.346 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:08.341+0000 7f72af85c740 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 -- 0x55845be3fc30 con 0x55845be2ab20 2024-04-23T10:26:08.346 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:08.341+0000 7f72af85c740 1 -- --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_getmap magic: 0 -- 0x55845be40300 con 0x55845be2a160 2024-04-23T10:26:08.346 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:08.341+0000 7f72a659c640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55845be28f90 0x55845be1f050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T10:26:08.346 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:08.341+0000 7f72a659c640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55845be28f90 0x55845be1f050 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.114:58190/0 (socket says 172.21.15.114:58190) 2024-04-23T10:26:08.346 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:08.341+0000 7f72a659c640 1 -- 172.21.15.114:0/1433987340 learned_addr learned my addr 172.21.15.114:0/1433987340 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-04-23T10:26:08.346 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:08.342+0000 7f72a5d9b640 1 --2- 172.21.15.114:0/1433987340 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x55845be2a160 0x55845be1f590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T10:26:08.347 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:08.342+0000 7f72a6d9d640 1 --2- 172.21.15.114:0/1433987340 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x55845be2ab20 0x55845be3a860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T10:26:08.347 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:08.342+0000 7f72a659c640 1 -- 172.21.15.114:0/1433987340 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x55845be2a160 msgr2=0x55845be1f590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T10:26:08.347 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:08.342+0000 7f72a659c640 1 --2- 172.21.15.114:0/1433987340 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x55845be2a160 0x55845be1f590 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:26:08.347 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:08.342+0000 7f72a659c640 1 -- 172.21.15.114:0/1433987340 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x55845be2ab20 msgr2=0x55845be3a860 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T10:26:08.347 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:08.342+0000 7f72a659c640 1 --2- 172.21.15.114:0/1433987340 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x55845be2ab20 0x55845be3a860 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:26:08.347 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:08.342+0000 7f72a659c640 1 -- 172.21.15.114:0/1433987340 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55845be344a0 con 0x55845be28f90 2024-04-23T10:26:08.347 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:08.342+0000 7f72a6d9d640 1 --2- 172.21.15.114:0/1433987340 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x55845be2ab20 0x55845be3a860 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-23T10:26:08.347 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:08.342+0000 7f72a659c640 1 --2- 172.21.15.114:0/1433987340 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55845be28f90 0x55845be1f050 secure :-1 s=READY pgs=338 cs=0 l=1 rev1=1 crypto rx=0x7f7298009230 tx=0x7f7298000bc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7366d0cc7892bf3b server_cookie=0 in_seq=0 out_seq=0 2024-04-23T10:26:08.347 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:08.342+0000 7f728ffff640 1 -- 172.21.15.114:0/1433987340 <== mon.0 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7298035070 con 0x55845be28f90 2024-04-23T10:26:08.347 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:08.342+0000 7f728ffff640 1 -- 172.21.15.114:0/1433987340 <== mon.0 v2:172.21.15.114:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f7298044dc0 con 0x55845be28f90 2024-04-23T10:26:08.347 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:08.342+0000 7f728ffff640 1 -- 172.21.15.114:0/1433987340 <== mon.0 v2:172.21.15.114:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f72980445e0 con 0x55845be28f90 2024-04-23T10:26:08.347 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:08.342+0000 7f72af85c740 1 -- 172.21.15.114:0/1433987340 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55845be28f90 msgr2=0x55845be1f050 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T10:26:08.347 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:08.342+0000 7f72af85c740 1 --2- 172.21.15.114:0/1433987340 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55845be28f90 0x55845be1f050 secure :-1 s=READY pgs=338 cs=0 l=1 rev1=1 crypto rx=0x7f7298009230 tx=0x7f7298000bc0 comp rx=0 tx=0).stop 2024-04-23T10:26:08.347 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:08.342+0000 7f72af85c740 1 -- 172.21.15.114:0/1433987340 shutdown_connections 2024-04-23T10:26:08.348 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:08.343+0000 7f72af85c740 1 --2- 172.21.15.114:0/1433987340 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x55845be2ab20 0x55845be3a860 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:26:08.348 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:08.343+0000 7f72af85c740 1 --2- 172.21.15.114:0/1433987340 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x55845be2a160 0x55845be1f590 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:26:08.348 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:08.343+0000 7f72af85c740 1 --2- 172.21.15.114:0/1433987340 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55845be28f90 0x55845be1f050 unknown :-1 s=CLOSED pgs=338 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:26:08.348 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:08.343+0000 7f72af85c740 1 -- 172.21.15.114:0/1433987340 >> 172.21.15.114:0/1433987340 conn(0x55845be20070 msgr2=0x55845be1e940 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T10:26:08.348 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:08.343+0000 7f72af85c740 1 -- 172.21.15.114:0/1433987340 shutdown_connections 2024-04-23T10:26:08.348 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:08.343+0000 7f72af85c740 1 -- 172.21.15.114:0/1433987340 wait complete. 2024-04-23T10:26:08.348 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:08.343+0000 7f72af85c740 1 -- start start 2024-04-23T10:26:08.348 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:08.343+0000 7f72af85c740 1 Processor -- start 2024-04-23T10:26:08.348 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:08.343+0000 7f72af85c740 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55845be28f90 0x55845be36410 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T10:26:08.348 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:08.343+0000 7f72af85c740 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x55845be2a160 0x55845be36950 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T10:26:08.348 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:08.343+0000 7f72af85c740 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x55845be2ab20 0x55845be39970 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T10:26:08.348 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:08.343+0000 7f72af85c740 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x55845be410a0 con 0x55845be28f90 2024-04-23T10:26:08.348 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:08.343+0000 7f72af85c740 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 -- 0x55845be409d0 con 0x55845be2a160 2024-04-23T10:26:08.348 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:08.343+0000 7f72af85c740 1 -- --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_getmap magic: 0 -- 0x55845be41770 con 0x55845be2ab20 2024-04-23T10:26:08.348 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:08.343+0000 7f72af85c740 10 client.0 _ll_register_callbacks cb 0x7f729009bd10 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-23T10:26:08.349 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:08.343+0000 7f72a659c640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55845be28f90 0x55845be36410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T10:26:08.349 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:08.343+0000 7f72a659c640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55845be28f90 0x55845be36410 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.114:58194/0 (socket says 172.21.15.114:58194) 2024-04-23T10:26:08.349 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:08.343+0000 7f72a659c640 1 -- 172.21.15.114:0/1043299427 learned_addr learned my addr 172.21.15.114:0/1043299427 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-04-23T10:26:08.349 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:08.343+0000 7f72a6d9d640 1 --2- 172.21.15.114:0/1043299427 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x55845be2ab20 0x55845be39970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T10:26:08.349 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:08.343+0000 7f72a659c640 1 -- 172.21.15.114:0/1043299427 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x55845be2ab20 msgr2=0x55845be39970 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T10:26:08.349 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:08.343+0000 7f72a659c640 1 --2- 172.21.15.114:0/1043299427 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x55845be2ab20 0x55845be39970 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:26:08.349 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:08.343+0000 7f72a5d9b640 1 --2- 172.21.15.114:0/1043299427 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x55845be2a160 0x55845be36950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T10:26:08.349 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:08.343+0000 7f72a659c640 1 -- 172.21.15.114:0/1043299427 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x55845be2a160 msgr2=0x55845be36950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T10:26:08.349 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:08.343+0000 7f72a659c640 1 --2- 172.21.15.114:0/1043299427 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x55845be2a160 0x55845be36950 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:26:08.349 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:08.343+0000 7f72a659c640 1 -- 172.21.15.114:0/1043299427 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55845be39f60 con 0x55845be28f90 2024-04-23T10:26:08.349 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:08.343+0000 7f72a5d9b640 1 --2- 172.21.15.114:0/1043299427 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x55845be2a160 0x55845be36950 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-23T10:26:08.349 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:08.344+0000 7f72a659c640 1 --2- 172.21.15.114:0/1043299427 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55845be28f90 0x55845be36410 secure :-1 s=READY pgs=339 cs=0 l=1 rev1=1 crypto rx=0x7f72980096c0 tx=0x7f7298031ca0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=baa97ffab17f3c37 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T10:26:08.349 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:08.344+0000 7f728dffb640 10 client.0 ms_handle_connect on v2:172.21.15.114:3300/0 2024-04-23T10:26:08.349 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:08.344+0000 7f728dffb640 1 -- 172.21.15.114:0/1043299427 <== mon.0 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7298052070 con 0x55845be28f90 2024-04-23T10:26:08.349 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:08.344+0000 7f728dffb640 1 -- 172.21.15.114:0/1043299427 <== mon.0 v2:172.21.15.114:3300/0 2 ==== osd_map(89..89 src has 1..89) ==== 8269+0+0 (secure 0 0 0) 0x7f7298044580 con 0x55845be28f90 2024-04-23T10:26:08.350 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:08.344+0000 7f72af85c740 1 -- 172.21.15.114:0/1043299427 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f7298044dc0 con 0x55845be28f90 2024-04-23T10:26:08.350 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:08.344+0000 7f728dffb640 1 -- 172.21.15.114:0/1043299427 <== mon.0 v2:172.21.15.114:3300/0 3 ==== mon_get_version_reply(handle=1 version=20) ==== 24+0+0 (secure 0 0 0) 0x7f729805e050 con 0x55845be28f90 2024-04-23T10:26:08.350 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:08.344+0000 7f72af85c740 10 client.15090 fetch_fsmap learned FSMap version 20 2024-04-23T10:26:08.350 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:08.344+0000 7f72af85c740 1 -- 172.21.15.114:0/1043299427 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({config=0+,fsmap=20,monmap=4+}) -- 0x55845bea6110 con 0x55845be28f90 2024-04-23T10:26:08.350 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:08.344+0000 7f728dffb640 1 -- 172.21.15.114:0/1043299427 <== mon.0 v2:172.21.15.114:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f7298044dc0 con 0x55845be28f90 2024-04-23T10:26:08.350 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:08.344+0000 7f728dffb640 1 -- 172.21.15.114:0/1043299427 <== mon.0 v2:172.21.15.114:3300/0 5 ==== fsmap(e 20) ==== 8338+0+0 (secure 0 0 0) 0x7f729804e880 con 0x55845be28f90 2024-04-23T10:26:08.350 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:08.344+0000 7f72af85c740 10 client.15090 fetch_fsmap finished waiting for FSMap version 20 2024-04-23T10:26:08.350 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:08.344+0000 7f72af85c740 10 client.15090 resolve_mds: resolved name 'h' to daemon mds.h 2024-04-23T10:26:08.350 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:08.344+0000 7f72af85c740 20 client.15090 populate_metadata read hostname 'smithi114' 2024-04-23T10:26:08.350 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:08.344+0000 7f72af85c740 1 --2- 172.21.15.114:0/1043299427 >> [v2:172.21.15.119:6836/1872363128,v1:172.21.15.119:6837/1872363128] conn(0x55845beaa8a0 0x55845bea6ce0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T10:26:08.350 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:08.344+0000 7f72af85c740 4 client.15090 mds_command: new command op to 24509 tid=0 multi_id=0 [{"prefix": "status"}] 2024-04-23T10:26:08.350 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:08.344+0000 7f72af85c740 1 -- 172.21.15.114:0/1043299427 --> [v2:172.21.15.119:6836/1872363128,v1:172.21.15.119:6837/1872363128] -- command(tid 0: {"prefix": "status"}) -- 0x55845bd914e0 con 0x55845beaa8a0 2024-04-23T10:26:08.350 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:08.344+0000 7f72a5d9b640 1 --2- 172.21.15.114:0/1043299427 >> [v2:172.21.15.119:6836/1872363128,v1:172.21.15.119:6837/1872363128] conn(0x55845beaa8a0 0x55845bea6ce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T10:26:08.351 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:08.345+0000 7f72a5d9b640 1 --2- 172.21.15.114:0/1043299427 >> [v2:172.21.15.119:6836/1872363128,v1:172.21.15.119:6837/1872363128] conn(0x55845beaa8a0 0x55845bea6ce0 crc :-1 s=READY pgs=17 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.1 client_cookie=cb26de5a1d07b4e3 server_cookie=1f3a7d4d75074296 in_seq=0 out_seq=0 2024-04-23T10:26:08.351 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:08.345+0000 7f728dffb640 10 client.15090 ms_handle_connect on v2:172.21.15.119:6836/1872363128 2024-04-23T10:26:08.351 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:08.345+0000 7f728dffb640 1 -- 172.21.15.114:0/1043299427 <== mds.1 v2:172.21.15.119:6836/1872363128 1 ==== command_reply(tid 0: 0 ) ==== 8+0+327 (crc 0 0 0) 0x55845bd914e0 con 0x55845beaa8a0 2024-04-23T10:26:08.351 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:08.345+0000 7f728dffb640 10 client.15090 handle_command_reply: tid=0 2024-04-23T10:26:08.351 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:08.345+0000 7f72af85c740 1 client.15090 shutdown 2024-04-23T10:26:08.351 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:08.345+0000 7f72af85c740 1 -- 172.21.15.114:0/1043299427 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55845be28f90 msgr2=0x55845be36410 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T10:26:08.351 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:08.345+0000 7f72af85c740 1 --2- 172.21.15.114:0/1043299427 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55845be28f90 0x55845be36410 secure :-1 s=READY pgs=339 cs=0 l=1 rev1=1 crypto rx=0x7f72980096c0 tx=0x7f7298031ca0 comp rx=0 tx=0).stop 2024-04-23T10:26:08.351 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:08.345+0000 7f72af85c740 1 -- 172.21.15.114:0/1043299427 shutdown_connections 2024-04-23T10:26:08.351 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:08.345+0000 7f72af85c740 1 --2- 172.21.15.114:0/1043299427 >> [v2:172.21.15.119:6836/1872363128,v1:172.21.15.119:6837/1872363128] conn(0x55845beaa8a0 0x55845bea6ce0 crc :-1 s=READY pgs=17 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:26:08.351 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:08.345+0000 7f72af85c740 1 --2- 172.21.15.114:0/1043299427 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x55845be2ab20 0x55845be39970 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:26:08.351 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:08.345+0000 7f728dffb640 0 client.15090 ms_handle_reset on v2:172.21.15.119:6836/1872363128 2024-04-23T10:26:08.351 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:08.345+0000 7f72af85c740 1 --2- 172.21.15.114:0/1043299427 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x55845be2a160 0x55845be36950 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:26:08.351 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:08.345+0000 7f72af85c740 1 --2- 172.21.15.114:0/1043299427 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55845be28f90 0x55845be36410 unknown :-1 s=CLOSED pgs=339 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:26:08.351 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:08.345+0000 7f72af85c740 1 -- 172.21.15.114:0/1043299427 >> 172.21.15.114:0/1043299427 conn(0x55845be20070 msgr2=0x55845be1f1d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T10:26:08.351 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:08.345+0000 7f72af85c740 1 -- 172.21.15.114:0/1043299427 shutdown_connections 2024-04-23T10:26:08.352 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:08.345+0000 7f72af85c740 1 -- 172.21.15.114:0/1043299427 wait complete. 2024-04-23T10:26:08.352 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:08.345+0000 7f72af85c740 20 client.15090 trim_cache size 0 max 16384 2024-04-23T10:26:08.352 INFO:teuthology.orchestra.run.smithi114.stdout:{ 2024-04-23T10:26:08.352 INFO:teuthology.orchestra.run.smithi114.stdout: "cluster_fsid": "aa240572-0159-11ef-bc93-c7b262605968", 2024-04-23T10:26:08.352 INFO:teuthology.orchestra.run.smithi114.stdout: "whoami": 1, 2024-04-23T10:26:08.352 INFO:teuthology.orchestra.run.smithi114.stdout: "id": 24509, 2024-04-23T10:26:08.352 INFO:teuthology.orchestra.run.smithi114.stdout: "want_state": "up:active", 2024-04-23T10:26:08.352 INFO:teuthology.orchestra.run.smithi114.stdout: "state": "up:active", 2024-04-23T10:26:08.352 INFO:teuthology.orchestra.run.smithi114.stdout: "fs_name": "cephfs", 2024-04-23T10:26:08.352 INFO:teuthology.orchestra.run.smithi114.stdout: "rank_uptime": 82.008749968999993, 2024-04-23T10:26:08.352 INFO:teuthology.orchestra.run.smithi114.stdout: "mdsmap_epoch": 20, 2024-04-23T10:26:08.352 INFO:teuthology.orchestra.run.smithi114.stdout: "osdmap_epoch": 89, 2024-04-23T10:26:08.353 INFO:teuthology.orchestra.run.smithi114.stdout: "osdmap_epoch_barrier": 87, 2024-04-23T10:26:08.353 INFO:teuthology.orchestra.run.smithi114.stdout: "uptime": 100.941828235 2024-04-23T10:26:08.353 INFO:teuthology.orchestra.run.smithi114.stdout:} 2024-04-23T10:26:08.353 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:08.346+0000 7f72acb4a640 1 -- 172.21.15.114:0/4103576177 >> [v2:172.21.15.114:6800/465333808,v1:172.21.15.114:6801/465333808] conn(0x7f72940a3c40 msgr2=0x7f72940a6100 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T10:26:08.353 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:08.346+0000 7f72acb4a640 1 --2- 172.21.15.114:0/4103576177 >> [v2:172.21.15.114:6800/465333808,v1:172.21.15.114:6801/465333808] conn(0x7f72940a3c40 0x7f72940a6100 secure :-1 s=READY pgs=224 cs=0 l=1 rev1=1 crypto rx=0x7f72a0005c40 tx=0x7f72a0009500 comp rx=0 tx=0).stop 2024-04-23T10:26:08.354 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:08.346+0000 7f72acb4a640 1 -- 172.21.15.114:0/4103576177 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f72a8103190 msgr2=0x7f72a8198730 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T10:26:08.354 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:08.346+0000 7f72acb4a640 1 --2- 172.21.15.114:0/4103576177 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f72a8103190 0x7f72a8198730 secure :-1 s=READY pgs=335 cs=0 l=1 rev1=1 crypto rx=0x7f729000d770 tx=0x7f729000dc30 comp rx=0 tx=0).stop 2024-04-23T10:26:08.354 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:08.346+0000 7f72acb4a640 1 -- 172.21.15.114:0/4103576177 shutdown_connections 2024-04-23T10:26:08.354 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:08.346+0000 7f72acb4a640 1 --2- 172.21.15.114:0/4103576177 >> [v2:172.21.15.114:6800/465333808,v1:172.21.15.114:6801/465333808] conn(0x7f72940a3c40 0x7f72940a6100 unknown :-1 s=CLOSED pgs=224 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:26:08.354 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:08.346+0000 7f72acb4a640 1 --2- 172.21.15.114:0/4103576177 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f72a8103b50 0x7f72a819f7a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:26:08.354 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:08.346+0000 7f72acb4a640 1 --2- 172.21.15.114:0/4103576177 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f72a8103190 0x7f72a8198730 unknown :-1 s=CLOSED pgs=335 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:26:08.354 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:08.346+0000 7f72acb4a640 1 --2- 172.21.15.114:0/4103576177 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f72a8101fa0 0x7f72a81981f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:26:08.354 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:08.346+0000 7f72acb4a640 1 -- 172.21.15.114:0/4103576177 >> 172.21.15.114:0/4103576177 conn(0x7f72a80fd530 msgr2=0x7f72a81065d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T10:26:08.354 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:08.346+0000 7f72acb4a640 1 -- 172.21.15.114:0/4103576177 shutdown_connections 2024-04-23T10:26:08.354 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:08.346+0000 7f72acb4a640 1 -- 172.21.15.114:0/4103576177 wait complete. 2024-04-23T10:26:08.355 INFO:journalctl@ceph.mon.a.smithi114.stdout:Apr 23 10:26:08 smithi114 ceph-mon[29255]: from='client.? 172.21.15.114:0/3533348400' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2024-04-23T10:26:08.355 INFO:journalctl@ceph.mon.a.smithi114.stdout:Apr 23 10:26:08 smithi114 ceph-mon[29255]: from='client.? 172.21.15.114:0/440960242' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-23T10:26:08.355 INFO:journalctl@ceph.mon.a.smithi114.stdout:Apr 23 10:26:08 smithi114 ceph-mon[29255]: from='client.? 172.21.15.114:0/872498497' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-23T10:26:08.355 DEBUG:teuthology.orchestra.run.smithi114:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.b status 2024-04-23T10:26:08.454 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:08.455+0000 7fa0e2474640 1 -- 172.21.15.114:0/3607460835 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fa0dc103180 msgr2=0x7fa0dc103600 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T10:26:08.454 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:08.455+0000 7fa0e2474640 1 --2- 172.21.15.114:0/3607460835 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fa0dc103180 0x7fa0dc103600 secure :-1 s=READY pgs=340 cs=0 l=1 rev1=1 crypto rx=0x7fa0cc009900 tx=0x7fa0cc02a410 comp rx=0 tx=0).stop 2024-04-23T10:26:08.454 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:08.455+0000 7fa0e2474640 1 -- 172.21.15.114:0/3607460835 shutdown_connections 2024-04-23T10:26:08.454 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:08.455+0000 7fa0e2474640 1 --2- 172.21.15.114:0/3607460835 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fa0dc103b40 0x7fa0dc10a3d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:26:08.454 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:08.455+0000 7fa0e2474640 1 --2- 172.21.15.114:0/3607460835 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fa0dc103180 0x7fa0dc103600 unknown :-1 s=CLOSED pgs=340 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:26:08.454 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:08.455+0000 7fa0e2474640 1 --2- 172.21.15.114:0/3607460835 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fa0dc101f90 0x7fa0dc1023b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:26:08.454 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:08.455+0000 7fa0e2474640 1 -- 172.21.15.114:0/3607460835 >> 172.21.15.114:0/3607460835 conn(0x7fa0dc0fd500 msgr2=0x7fa0dc0ff940 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T10:26:08.454 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:08.455+0000 7fa0e2474640 1 -- 172.21.15.114:0/3607460835 shutdown_connections 2024-04-23T10:26:08.455 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:08.455+0000 7fa0e2474640 1 -- 172.21.15.114:0/3607460835 wait complete. 2024-04-23T10:26:08.455 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:08.455+0000 7fa0e2474640 1 Processor -- start 2024-04-23T10:26:08.455 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:08.455+0000 7fa0e2474640 1 -- start start 2024-04-23T10:26:08.456 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:08.456+0000 7fa0e2474640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fa0dc101f90 0x7fa0dc075d00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T10:26:08.457 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:08.456+0000 7fa0e2474640 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fa0dc103180 0x7fa0dc074340 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T10:26:08.457 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:08.456+0000 7fa0e2474640 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fa0dc103b40 0x7fa0dc074880 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T10:26:08.457 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:08.456+0000 7fa0e2474640 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x7fa0dc10dad0 con 0x7fa0dc101f90 2024-04-23T10:26:08.457 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:08.456+0000 7fa0e2474640 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 -- 0x7fa0dc10d950 con 0x7fa0dc103180 2024-04-23T10:26:08.457 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:08.456+0000 7fa0e2474640 1 -- --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_getmap magic: 0 -- 0x7fa0dc10dc50 con 0x7fa0dc103b40 2024-04-23T10:26:08.457 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:08.456+0000 7fa0dbfff640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fa0dc101f90 0x7fa0dc075d00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T10:26:08.457 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:08.456+0000 7fa0dbfff640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fa0dc101f90 0x7fa0dc075d00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.114:58204/0 (socket says 172.21.15.114:58204) 2024-04-23T10:26:08.457 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:08.456+0000 7fa0dbfff640 1 -- 172.21.15.114:0/3741647453 learned_addr learned my addr 172.21.15.114:0/3741647453 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-04-23T10:26:08.457 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:08.456+0000 7fa0db7fe640 1 --2- 172.21.15.114:0/3741647453 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fa0dc103180 0x7fa0dc074340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T10:26:08.457 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:08.456+0000 7fa0dbfff640 1 -- 172.21.15.114:0/3741647453 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fa0dc103b40 msgr2=0x7fa0dc074880 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T10:26:08.457 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:08.456+0000 7fa0dbfff640 1 --2- 172.21.15.114:0/3741647453 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fa0dc103b40 0x7fa0dc074880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:26:08.457 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:08.456+0000 7fa0dbfff640 1 -- 172.21.15.114:0/3741647453 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fa0dc103180 msgr2=0x7fa0dc074340 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T10:26:08.457 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:08.456+0000 7fa0dbfff640 1 --2- 172.21.15.114:0/3741647453 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fa0dc103180 0x7fa0dc074340 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:26:08.458 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:08.456+0000 7fa0dbfff640 1 -- 172.21.15.114:0/3741647453 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa0dc075110 con 0x7fa0dc101f90 2024-04-23T10:26:08.458 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:08.456+0000 7fa0dbfff640 1 --2- 172.21.15.114:0/3741647453 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fa0dc101f90 0x7fa0dc075d00 secure :-1 s=READY pgs=341 cs=0 l=1 rev1=1 crypto rx=0x7fa0d000e9c0 tx=0x7fa0d000ee80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T10:26:08.458 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:08.456+0000 7fa0d97fa640 1 -- 172.21.15.114:0/3741647453 <== mon.0 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa0d000cd80 con 0x7fa0dc101f90 2024-04-23T10:26:08.458 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:08.456+0000 7fa0d97fa640 1 -- 172.21.15.114:0/3741647453 <== mon.0 v2:172.21.15.114:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fa0d00035f0 con 0x7fa0dc101f90 2024-04-23T10:26:08.458 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:08.456+0000 7fa0d97fa640 1 -- 172.21.15.114:0/3741647453 <== mon.0 v2:172.21.15.114:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa0d0010690 con 0x7fa0dc101f90 2024-04-23T10:26:08.458 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:08.456+0000 7fa0e2474640 1 -- 172.21.15.114:0/3741647453 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fa0dc1a22a0 con 0x7fa0dc101f90 2024-04-23T10:26:08.458 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:08.456+0000 7fa0e2474640 1 -- 172.21.15.114:0/3741647453 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fa0dc1a27f0 con 0x7fa0dc101f90 2024-04-23T10:26:08.464 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:08.459+0000 7fa0d97fa640 1 -- 172.21.15.114:0/3741647453 <== mon.0 v2:172.21.15.114:3300/0 4 ==== mgrmap(e 17) ==== 138698+0+0 (secure 0 0 0) 0x7fa0d00037a0 con 0x7fa0dc101f90 2024-04-23T10:26:08.464 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:08.459+0000 7fa0e5182740 1 Processor -- start 2024-04-23T10:26:08.465 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:08.459+0000 7fa0e5182740 1 -- start start 2024-04-23T10:26:08.465 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:08.459+0000 7fa0e5182740 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55fb89689b80 0x55fb89689fa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T10:26:08.465 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:08.459+0000 7fa0e5182740 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x55fb8968ad50 0x55fb8968b1d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T10:26:08.465 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:08.459+0000 7fa0e5182740 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x55fb8968b710 0x55fb89691fa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T10:26:08.465 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:08.459+0000 7fa0e5182740 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x55fb895eec50 con 0x55fb89689b80 2024-04-23T10:26:08.465 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:08.459+0000 7fa0e5182740 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 -- 0x55fb8958b830 con 0x55fb8968ad50 2024-04-23T10:26:08.465 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:08.459+0000 7fa0e5182740 1 -- --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_getmap magic: 0 -- 0x55fb895eac10 con 0x55fb8968b710 2024-04-23T10:26:08.465 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:08.459+0000 7fa0e0a11640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55fb89689b80 0x55fb89689fa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T10:26:08.465 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:08.459+0000 7fa0e0a11640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55fb89689b80 0x55fb89689fa0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.114:58208/0 (socket says 172.21.15.114:58208) 2024-04-23T10:26:08.465 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:08.459+0000 7fa0e0a11640 1 -- 172.21.15.114:0/211804633 learned_addr learned my addr 172.21.15.114:0/211804633 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-04-23T10:26:08.465 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:08.460+0000 7fa0db7fe640 1 --2- 172.21.15.114:0/211804633 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x55fb8968b710 0x55fb89691fa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T10:26:08.465 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:08.460+0000 7fa0d97fa640 1 --2- 172.21.15.114:0/3741647453 >> [v2:172.21.15.114:6800/465333808,v1:172.21.15.114:6801/465333808] conn(0x7fa0c40a3c40 0x7fa0c40a6100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T10:26:08.465 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:08.460+0000 7fa0dbfff640 1 --2- 172.21.15.114:0/211804633 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x55fb8968ad50 0x55fb8968b1d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T10:26:08.465 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:08.460+0000 7fa0e0a11640 1 -- 172.21.15.114:0/211804633 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x55fb8968b710 msgr2=0x55fb89691fa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T10:26:08.465 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:08.460+0000 7fa0e0a11640 1 --2- 172.21.15.114:0/211804633 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x55fb8968b710 0x55fb89691fa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:26:08.465 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:08.460+0000 7fa0e0a11640 1 -- 172.21.15.114:0/211804633 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x55fb8968ad50 msgr2=0x55fb8968b1d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T10:26:08.466 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:08.460+0000 7fa0e0a11640 1 --2- 172.21.15.114:0/211804633 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x55fb8968ad50 0x55fb8968b1d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:26:08.466 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:08.460+0000 7fa0e0a11640 1 -- 172.21.15.114:0/211804633 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55fb893012d0 con 0x55fb89689b80 2024-04-23T10:26:08.466 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:08.460+0000 7fa0dbfff640 1 --2- 172.21.15.114:0/211804633 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x55fb8968ad50 0x55fb8968b1d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-23T10:26:08.466 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:08.460+0000 7fa0d97fa640 1 -- 172.21.15.114:0/3741647453 <== mon.0 v2:172.21.15.114:3300/0 5 ==== osd_map(89..89 src has 1..89) ==== 8269+0+0 (secure 0 0 0) 0x7fa0d00d0ae0 con 0x7fa0dc101f90 2024-04-23T10:26:08.466 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:08.460+0000 7fa0e0a11640 1 --2- 172.21.15.114:0/211804633 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55fb89689b80 0x55fb89689fa0 secure :-1 s=READY pgs=342 cs=0 l=1 rev1=1 crypto rx=0x7fa0d4009ea0 tx=0x7fa0d4009f90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e4cbe6fa7a26b0bc server_cookie=0 in_seq=0 out_seq=0 2024-04-23T10:26:08.466 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:08.460+0000 7fa0e0a11640 1 --2- 172.21.15.114:0/3741647453 >> [v2:172.21.15.114:6800/465333808,v1:172.21.15.114:6801/465333808] conn(0x7fa0c40a3c40 0x7fa0c40a6100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T10:26:08.466 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:08.460+0000 7fa0c27fc640 1 -- 172.21.15.114:0/211804633 <== mon.0 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa0d4004d60 con 0x55fb89689b80 2024-04-23T10:26:08.466 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:08.460+0000 7fa0c27fc640 1 -- 172.21.15.114:0/211804633 <== mon.0 v2:172.21.15.114:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fa0d4005650 con 0x55fb89689b80 2024-04-23T10:26:08.466 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:08.460+0000 7fa0c27fc640 1 -- 172.21.15.114:0/211804633 <== mon.0 v2:172.21.15.114:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa0d4010d60 con 0x55fb89689b80 2024-04-23T10:26:08.466 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:08.461+0000 7fa0e5182740 1 -- 172.21.15.114:0/211804633 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55fb89689b80 msgr2=0x55fb89689fa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T10:26:08.466 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:08.461+0000 7fa0e0a11640 1 --2- 172.21.15.114:0/3741647453 >> [v2:172.21.15.114:6800/465333808,v1:172.21.15.114:6801/465333808] conn(0x7fa0c40a3c40 0x7fa0c40a6100 secure :-1 s=READY pgs=225 cs=0 l=1 rev1=1 crypto rx=0x7fa0d4019000 tx=0x7fa0d4018950 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T10:26:08.466 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:08.461+0000 7fa0e5182740 1 --2- 172.21.15.114:0/211804633 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55fb89689b80 0x55fb89689fa0 secure :-1 s=READY pgs=342 cs=0 l=1 rev1=1 crypto rx=0x7fa0d4009ea0 tx=0x7fa0d4009f90 comp rx=0 tx=0).stop 2024-04-23T10:26:08.466 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:08.461+0000 7fa0e5182740 1 -- 172.21.15.114:0/211804633 shutdown_connections 2024-04-23T10:26:08.466 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:08.461+0000 7fa0e5182740 1 --2- 172.21.15.114:0/211804633 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x55fb8968b710 0x55fb89691fa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:26:08.467 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:08.461+0000 7fa0e5182740 1 --2- 172.21.15.114:0/211804633 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x55fb8968ad50 0x55fb8968b1d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:26:08.467 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:08.461+0000 7fa0e5182740 1 --2- 172.21.15.114:0/211804633 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55fb89689b80 0x55fb89689fa0 unknown :-1 s=CLOSED pgs=342 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:26:08.467 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:08.461+0000 7fa0e5182740 1 -- 172.21.15.114:0/211804633 >> 172.21.15.114:0/211804633 conn(0x55fb89685070 msgr2=0x55fb89685470 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T10:26:08.467 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:08.461+0000 7fa0e5182740 1 -- 172.21.15.114:0/211804633 shutdown_connections 2024-04-23T10:26:08.467 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:08.461+0000 7fa0e5182740 1 -- 172.21.15.114:0/211804633 wait complete. 2024-04-23T10:26:08.467 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:08.461+0000 7fa0e5182740 1 -- start start 2024-04-23T10:26:08.467 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:08.461+0000 7fa0e5182740 1 Processor -- start 2024-04-23T10:26:08.467 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:08.462+0000 7fa0e5182740 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55fb89689b80 0x55fb89693f40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T10:26:08.467 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:08.462+0000 7fa0e5182740 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x55fb8968ad50 0x55fb89694480 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T10:26:08.467 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:08.462+0000 7fa0e5182740 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x55fb8968b710 0x55fb8969b4c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T10:26:08.467 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:08.462+0000 7fa0e5182740 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x55fb89692720 con 0x55fb89689b80 2024-04-23T10:26:08.467 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:08.462+0000 7fa0e5182740 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 -- 0x55fb89589470 con 0x55fb8968ad50 2024-04-23T10:26:08.467 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:08.462+0000 7fa0e5182740 1 -- --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_getmap magic: 0 -- 0x55fb896928a0 con 0x55fb8968b710 2024-04-23T10:26:08.467 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:08.462+0000 7fa0e5182740 10 client.0 _ll_register_callbacks cb 0x55fb895aef30 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-23T10:26:08.467 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:08.462+0000 7fa0dbfff640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55fb89689b80 0x55fb89693f40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T10:26:08.467 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:08.462+0000 7fa0dbfff640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55fb89689b80 0x55fb89693f40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.114:58218/0 (socket says 172.21.15.114:58218) 2024-04-23T10:26:08.468 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:08.462+0000 7fa0dbfff640 1 -- 172.21.15.114:0/4290468217 learned_addr learned my addr 172.21.15.114:0/4290468217 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-04-23T10:26:08.468 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:08.462+0000 7fa0e0a11640 1 --2- 172.21.15.114:0/4290468217 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x55fb8968b710 0x55fb8969b4c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T10:26:08.468 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:08.462+0000 7fa0dbfff640 1 -- 172.21.15.114:0/4290468217 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x55fb8968b710 msgr2=0x55fb8969b4c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T10:26:08.468 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:08.462+0000 7fa0dbfff640 1 --2- 172.21.15.114:0/4290468217 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x55fb8968b710 0x55fb8969b4c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:26:08.468 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:08.462+0000 7fa0dbfff640 1 -- 172.21.15.114:0/4290468217 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x55fb8968ad50 msgr2=0x55fb89694480 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T10:26:08.468 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:08.462+0000 7fa0dbfff640 1 --2- 172.21.15.114:0/4290468217 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x55fb8968ad50 0x55fb89694480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:26:08.468 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:08.462+0000 7fa0dbfff640 1 -- 172.21.15.114:0/4290468217 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55fb893cb6b0 con 0x55fb89689b80 2024-04-23T10:26:08.468 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:08.462+0000 7fa0dbfff640 1 --2- 172.21.15.114:0/4290468217 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55fb89689b80 0x55fb89693f40 secure :-1 s=READY pgs=343 cs=0 l=1 rev1=1 crypto rx=0x7fa0d0000c00 tx=0x7fa0d009c660 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3aeb4e3f710fdc97 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T10:26:08.468 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:08.462+0000 7fa0bb7fe640 10 client.0 ms_handle_connect on v2:172.21.15.114:3300/0 2024-04-23T10:26:08.468 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:08.462+0000 7fa0bb7fe640 1 -- 172.21.15.114:0/4290468217 <== mon.0 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa0d00a0ff0 con 0x55fb89689b80 2024-04-23T10:26:08.468 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:08.462+0000 7fa0bb7fe640 1 -- 172.21.15.114:0/4290468217 <== mon.0 v2:172.21.15.114:3300/0 2 ==== osd_map(89..89 src has 1..89) ==== 8269+0+0 (secure 0 0 0) 0x7fa0d0087040 con 0x55fb89689b80 2024-04-23T10:26:08.468 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:08.462+0000 7fa0e5182740 1 -- 172.21.15.114:0/4290468217 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fa0d4005650 con 0x55fb89689b80 2024-04-23T10:26:08.468 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:08.462+0000 7fa0bb7fe640 1 -- 172.21.15.114:0/4290468217 <== mon.0 v2:172.21.15.114:3300/0 3 ==== mon_get_version_reply(handle=1 version=20) ==== 24+0+0 (secure 0 0 0) 0x7fa0d00a58b0 con 0x55fb89689b80 2024-04-23T10:26:08.468 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:08.462+0000 7fa0e5182740 10 client.15102 fetch_fsmap learned FSMap version 20 2024-04-23T10:26:08.468 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:08.462+0000 7fa0e5182740 1 -- 172.21.15.114:0/4290468217 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({config=0+,fsmap=20,monmap=4+}) -- 0x55fb8969e870 con 0x55fb89689b80 2024-04-23T10:26:08.468 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:08.462+0000 7fa0bb7fe640 1 -- 172.21.15.114:0/4290468217 <== mon.0 v2:172.21.15.114:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fa0d4005650 con 0x55fb89689b80 2024-04-23T10:26:08.469 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:08.463+0000 7fa0bb7fe640 1 -- 172.21.15.114:0/4290468217 <== mon.0 v2:172.21.15.114:3300/0 5 ==== fsmap(e 20) ==== 8338+0+0 (secure 0 0 0) 0x7fa0d00a5420 con 0x55fb89689b80 2024-04-23T10:26:08.469 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:08.463+0000 7fa0e5182740 10 client.15102 fetch_fsmap finished waiting for FSMap version 20 2024-04-23T10:26:08.469 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:08.463+0000 7fa0e5182740 10 client.15102 resolve_mds: resolved name 'b' to daemon mds.b 2024-04-23T10:26:08.469 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:08.463+0000 7fa0e5182740 20 client.15102 populate_metadata read hostname 'smithi114' 2024-04-23T10:26:08.469 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:08.463+0000 7fa0e5182740 1 --2- 172.21.15.114:0/4290468217 >> [v2:172.21.15.119:6838/937297340,v1:172.21.15.119:6839/937297340] conn(0x55fb896a0ed0 0x55fb896a2340 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T10:26:08.469 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:08.463+0000 7fa0e5182740 4 client.15102 mds_command: new command op to 24545 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-23T10:26:08.469 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:08.463+0000 7fa0e5182740 1 -- 172.21.15.114:0/4290468217 --> [v2:172.21.15.119:6838/937297340,v1:172.21.15.119:6839/937297340] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55fb895fc670 con 0x55fb896a0ed0 2024-04-23T10:26:08.469 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:08.463+0000 7fa0db7fe640 1 --2- 172.21.15.114:0/4290468217 >> [v2:172.21.15.119:6838/937297340,v1:172.21.15.119:6839/937297340] conn(0x55fb896a0ed0 0x55fb896a2340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T10:26:08.469 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:08.464+0000 7fa0db7fe640 1 --2- 172.21.15.114:0/4290468217 >> [v2:172.21.15.119:6838/937297340,v1:172.21.15.119:6839/937297340] conn(0x55fb896a0ed0 0x55fb896a2340 crc :-1 s=READY pgs=19 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=94b70953a6b0d0b9 server_cookie=695db0a7e67cbce3 in_seq=0 out_seq=0 2024-04-23T10:26:08.469 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:08.464+0000 7fa0bb7fe640 10 client.15102 ms_handle_connect on v2:172.21.15.119:6838/937297340 2024-04-23T10:26:08.469 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:08.465+0000 7fa0bb7fe640 1 -- 172.21.15.114:0/4290468217 <== mds.0 v2:172.21.15.119:6838/937297340 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x55fb895fc670 con 0x55fb896a0ed0 2024-04-23T10:26:08.469 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:08.465+0000 7fa0bb7fe640 10 client.15102 handle_command_reply: tid=0 2024-04-23T10:26:08.469 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:08.465+0000 7fa0e5182740 1 client.15102 shutdown 2024-04-23T10:26:08.469 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:08.465+0000 7fa0e5182740 1 -- 172.21.15.114:0/4290468217 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55fb89689b80 msgr2=0x55fb89693f40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T10:26:08.470 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:08.465+0000 7fa0e5182740 1 --2- 172.21.15.114:0/4290468217 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55fb89689b80 0x55fb89693f40 secure :-1 s=READY pgs=343 cs=0 l=1 rev1=1 crypto rx=0x7fa0d0000c00 tx=0x7fa0d009c660 comp rx=0 tx=0).stop 2024-04-23T10:26:08.470 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:08.465+0000 7fa0e5182740 1 -- 172.21.15.114:0/4290468217 shutdown_connections 2024-04-23T10:26:08.470 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:08.465+0000 7fa0e5182740 1 --2- 172.21.15.114:0/4290468217 >> [v2:172.21.15.119:6838/937297340,v1:172.21.15.119:6839/937297340] conn(0x55fb896a0ed0 0x55fb896a2340 crc :-1 s=READY pgs=19 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:26:08.470 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:08.465+0000 7fa0e5182740 1 --2- 172.21.15.114:0/4290468217 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x55fb8968b710 0x55fb8969b4c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:26:08.470 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:08.465+0000 7fa0bb7fe640 0 client.15102 ms_handle_reset on v2:172.21.15.119:6838/937297340 2024-04-23T10:26:08.470 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:08.465+0000 7fa0e5182740 1 --2- 172.21.15.114:0/4290468217 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x55fb8968ad50 0x55fb89694480 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:26:08.470 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:08.465+0000 7fa0e5182740 1 --2- 172.21.15.114:0/4290468217 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55fb89689b80 0x55fb89693f40 unknown :-1 s=CLOSED pgs=343 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:26:08.470 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:08.465+0000 7fa0e5182740 1 -- 172.21.15.114:0/4290468217 >> 172.21.15.114:0/4290468217 conn(0x55fb89685070 msgr2=0x55fb896841d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T10:26:08.470 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:08.465+0000 7fa0e5182740 1 -- 172.21.15.114:0/4290468217 shutdown_connections 2024-04-23T10:26:08.470 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:08.465+0000 7fa0e5182740 1 -- 172.21.15.114:0/4290468217 wait complete. 2024-04-23T10:26:08.470 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:08.465+0000 7fa0e5182740 20 client.15102 trim_cache size 0 max 16384 2024-04-23T10:26:08.487 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:08.483+0000 7fa0e5182740 1 Processor -- start 2024-04-23T10:26:08.487 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:08.483+0000 7fa0e5182740 1 -- start start 2024-04-23T10:26:08.487 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:08.483+0000 7fa0e5182740 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x55fb89689b80 0x55fb89683940 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T10:26:08.487 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:08.483+0000 7fa0e5182740 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55fb8968ad50 0x55fb8969b480 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T10:26:08.487 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:08.483+0000 7fa0e5182740 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x55fb8968b710 0x55fb89683e80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T10:26:08.487 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:08.483+0000 7fa0e5182740 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x55fb896a0680 con 0x55fb8968ad50 2024-04-23T10:26:08.487 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:08.483+0000 7fa0e5182740 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 -- 0x55fb896a0d50 con 0x55fb89689b80 2024-04-23T10:26:08.487 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:08.483+0000 7fa0e5182740 1 -- --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_getmap magic: 0 -- 0x55fb896a1420 con 0x55fb8968b710 2024-04-23T10:26:08.488 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:08.483+0000 7fa0db7fe640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55fb8968ad50 0x55fb8969b480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T10:26:08.488 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:08.483+0000 7fa0db7fe640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55fb8968ad50 0x55fb8969b480 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.114:58222/0 (socket says 172.21.15.114:58222) 2024-04-23T10:26:08.488 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:08.483+0000 7fa0db7fe640 1 -- 172.21.15.114:0/2447605024 learned_addr learned my addr 172.21.15.114:0/2447605024 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-04-23T10:26:08.488 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:08.483+0000 7fa0db7fe640 1 -- 172.21.15.114:0/2447605024 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x55fb8968b710 msgr2=0x55fb89683e80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T10:26:08.488 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:08.483+0000 7fa0db7fe640 1 --2- 172.21.15.114:0/2447605024 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x55fb8968b710 0x55fb89683e80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:26:08.488 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:08.483+0000 7fa0dbfff640 1 --2- 172.21.15.114:0/2447605024 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x55fb89689b80 0x55fb89683940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T10:26:08.488 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:08.483+0000 7fa0db7fe640 1 -- 172.21.15.114:0/2447605024 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x55fb89689b80 msgr2=0x55fb89683940 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T10:26:08.488 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:08.483+0000 7fa0db7fe640 1 --2- 172.21.15.114:0/2447605024 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x55fb89689b80 0x55fb89683940 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:26:08.488 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:08.483+0000 7fa0db7fe640 1 -- 172.21.15.114:0/2447605024 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55fb89691a40 con 0x55fb8968ad50 2024-04-23T10:26:08.488 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:08.483+0000 7fa0db7fe640 1 --2- 172.21.15.114:0/2447605024 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55fb8968ad50 0x55fb8969b480 secure :-1 s=READY pgs=344 cs=0 l=1 rev1=1 crypto rx=0x7fa0cc02a930 tx=0x7fa0cc009600 comp rx=0 tx=0).ready entity=mon.0 client_cookie=698d62c45c6aced1 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T10:26:08.488 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:08.484+0000 7fa0c1ffb640 1 -- 172.21.15.114:0/2447605024 <== mon.0 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa0cc049070 con 0x55fb8968ad50 2024-04-23T10:26:08.488 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:08.484+0000 7fa0c1ffb640 1 -- 172.21.15.114:0/2447605024 <== mon.0 v2:172.21.15.114:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fa0cc004040 con 0x55fb8968ad50 2024-04-23T10:26:08.488 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:08.484+0000 7fa0c1ffb640 1 -- 172.21.15.114:0/2447605024 <== mon.0 v2:172.21.15.114:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa0cc035040 con 0x55fb8968ad50 2024-04-23T10:26:08.488 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:08.484+0000 7fa0e5182740 1 -- 172.21.15.114:0/2447605024 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55fb8968ad50 msgr2=0x55fb8969b480 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T10:26:08.488 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:08.484+0000 7fa0e5182740 1 --2- 172.21.15.114:0/2447605024 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55fb8968ad50 0x55fb8969b480 secure :-1 s=READY pgs=344 cs=0 l=1 rev1=1 crypto rx=0x7fa0cc02a930 tx=0x7fa0cc009600 comp rx=0 tx=0).stop 2024-04-23T10:26:08.489 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:08.484+0000 7fa0e5182740 1 -- 172.21.15.114:0/2447605024 shutdown_connections 2024-04-23T10:26:08.489 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:08.484+0000 7fa0e5182740 1 --2- 172.21.15.114:0/2447605024 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x55fb8968b710 0x55fb89683e80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:26:08.489 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:08.484+0000 7fa0e5182740 1 --2- 172.21.15.114:0/2447605024 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55fb8968ad50 0x55fb8969b480 unknown :-1 s=CLOSED pgs=344 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:26:08.489 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:08.484+0000 7fa0e5182740 1 --2- 172.21.15.114:0/2447605024 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x55fb89689b80 0x55fb89683940 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:26:08.489 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:08.484+0000 7fa0e5182740 1 -- 172.21.15.114:0/2447605024 >> 172.21.15.114:0/2447605024 conn(0x55fb89685070 msgr2=0x55fb8968e750 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T10:26:08.489 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:08.484+0000 7fa0e5182740 1 -- 172.21.15.114:0/2447605024 shutdown_connections 2024-04-23T10:26:08.489 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:08.484+0000 7fa0e5182740 1 -- 172.21.15.114:0/2447605024 wait complete. 2024-04-23T10:26:08.489 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:08.485+0000 7fa0e5182740 1 -- start start 2024-04-23T10:26:08.489 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:08.485+0000 7fa0e5182740 1 Processor -- start 2024-04-23T10:26:08.489 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:08.485+0000 7fa0e5182740 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x55fb89689b80 0x55fb89697220 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T10:26:08.489 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:08.485+0000 7fa0e5182740 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x55fb8968ad50 0x55fb89697760 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T10:26:08.489 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:08.485+0000 7fa0e5182740 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55fb8968b710 0x55fb8969a780 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T10:26:08.489 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:08.485+0000 7fa0e5182740 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x55fb896a21c0 con 0x55fb8968b710 2024-04-23T10:26:08.489 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:08.485+0000 7fa0e5182740 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 -- 0x55fb896a1af0 con 0x55fb89689b80 2024-04-23T10:26:08.489 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:08.485+0000 7fa0e5182740 1 -- --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_getmap magic: 0 -- 0x55fb896a2890 con 0x55fb8968ad50 2024-04-23T10:26:08.490 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:08.485+0000 7fa0e5182740 10 client.0 _ll_register_callbacks cb 0x7fa0d0003d90 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-23T10:26:08.490 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:08.485+0000 7fa0e0a11640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55fb8968b710 0x55fb8969a780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T10:26:08.490 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:08.485+0000 7fa0e0a11640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55fb8968b710 0x55fb8969a780 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.114:58238/0 (socket says 172.21.15.114:58238) 2024-04-23T10:26:08.490 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:08.485+0000 7fa0e0a11640 1 -- 172.21.15.114:0/3170683979 learned_addr learned my addr 172.21.15.114:0/3170683979 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-04-23T10:26:08.490 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:08.485+0000 7fa0db7fe640 1 --2- 172.21.15.114:0/3170683979 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x55fb8968ad50 0x55fb89697760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T10:26:08.490 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:08.485+0000 7fa0dbfff640 1 --2- 172.21.15.114:0/3170683979 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x55fb89689b80 0x55fb89697220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T10:26:08.490 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:08.485+0000 7fa0e0a11640 1 -- 172.21.15.114:0/3170683979 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x55fb8968ad50 msgr2=0x55fb89697760 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T10:26:08.490 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:08.485+0000 7fa0e0a11640 1 --2- 172.21.15.114:0/3170683979 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x55fb8968ad50 0x55fb89697760 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:26:08.490 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:08.485+0000 7fa0e0a11640 1 -- 172.21.15.114:0/3170683979 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x55fb89689b80 msgr2=0x55fb89697220 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T10:26:08.490 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:08.485+0000 7fa0e0a11640 1 --2- 172.21.15.114:0/3170683979 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x55fb89689b80 0x55fb89697220 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:26:08.490 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:08.485+0000 7fa0e0a11640 1 -- 172.21.15.114:0/3170683979 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55fb8969ad70 con 0x55fb8968b710 2024-04-23T10:26:08.490 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:08.485+0000 7fa0e0a11640 1 --2- 172.21.15.114:0/3170683979 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55fb8968b710 0x55fb8969a780 secure :-1 s=READY pgs=345 cs=0 l=1 rev1=1 crypto rx=0x7fa0d4009e70 tx=0x7fa0d4013000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=be1b53226f737faa server_cookie=0 in_seq=0 out_seq=0 2024-04-23T10:26:08.490 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:08.485+0000 7fa0bb7fe640 10 client.0 ms_handle_connect on v2:172.21.15.114:3300/0 2024-04-23T10:26:08.490 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:08.486+0000 7fa0bb7fe640 1 -- 172.21.15.114:0/3170683979 <== mon.0 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa0d4034020 con 0x55fb8968b710 2024-04-23T10:26:08.490 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:08.486+0000 7fa0bb7fe640 1 -- 172.21.15.114:0/3170683979 <== mon.0 v2:172.21.15.114:3300/0 2 ==== osd_map(89..89 src has 1..89) ==== 8269+0+0 (secure 0 0 0) 0x7fa0d4005250 con 0x55fb8968b710 2024-04-23T10:26:08.491 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:08.486+0000 7fa0e5182740 1 -- 172.21.15.114:0/3170683979 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fa0cc004040 con 0x55fb8968b710 2024-04-23T10:26:08.491 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:08.486+0000 7fa0bb7fe640 1 -- 172.21.15.114:0/3170683979 <== mon.0 v2:172.21.15.114:3300/0 3 ==== mon_get_version_reply(handle=1 version=20) ==== 24+0+0 (secure 0 0 0) 0x7fa0d403f050 con 0x55fb8968b710 2024-04-23T10:26:08.491 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:08.486+0000 7fa0e5182740 10 client.15108 fetch_fsmap learned FSMap version 20 2024-04-23T10:26:08.491 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:08.486+0000 7fa0e5182740 1 -- 172.21.15.114:0/3170683979 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({config=0+,fsmap=20,monmap=4+}) -- 0x55fb89707550 con 0x55fb8968b710 2024-04-23T10:26:08.491 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:08.486+0000 7fa0bb7fe640 1 -- 172.21.15.114:0/3170683979 <== mon.0 v2:172.21.15.114:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fa0cc004040 con 0x55fb8968b710 2024-04-23T10:26:08.491 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:08.486+0000 7fa0bb7fe640 1 -- 172.21.15.114:0/3170683979 <== mon.0 v2:172.21.15.114:3300/0 5 ==== fsmap(e 20) ==== 8338+0+0 (secure 0 0 0) 0x7fa0d4004040 con 0x55fb8968b710 2024-04-23T10:26:08.491 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:08.486+0000 7fa0e5182740 10 client.15108 fetch_fsmap finished waiting for FSMap version 20 2024-04-23T10:26:08.491 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:08.486+0000 7fa0e5182740 10 client.15108 resolve_mds: resolved name 'b' to daemon mds.b 2024-04-23T10:26:08.491 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:08.486+0000 7fa0e5182740 20 client.15108 populate_metadata read hostname 'smithi114' 2024-04-23T10:26:08.491 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:08.486+0000 7fa0e5182740 1 --2- 172.21.15.114:0/3170683979 >> [v2:172.21.15.119:6838/937297340,v1:172.21.15.119:6839/937297340] conn(0x55fb896a0ed0 0x55fb8970ab70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T10:26:08.491 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:08.486+0000 7fa0e5182740 4 client.15108 mds_command: new command op to 24545 tid=0 multi_id=0 [{"prefix": "status"}] 2024-04-23T10:26:08.491 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:08.486+0000 7fa0e5182740 1 -- 172.21.15.114:0/3170683979 --> [v2:172.21.15.119:6838/937297340,v1:172.21.15.119:6839/937297340] -- command(tid 0: {"prefix": "status"}) -- 0x55fb895f64e0 con 0x55fb896a0ed0 2024-04-23T10:26:08.491 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:08.487+0000 7fa0dbfff640 1 --2- 172.21.15.114:0/3170683979 >> [v2:172.21.15.119:6838/937297340,v1:172.21.15.119:6839/937297340] conn(0x55fb896a0ed0 0x55fb8970ab70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T10:26:08.491 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:08.487+0000 7fa0bb7fe640 10 client.15108 ms_handle_connect on v2:172.21.15.119:6838/937297340 2024-04-23T10:26:08.491 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:08.487+0000 7fa0dbfff640 1 --2- 172.21.15.114:0/3170683979 >> [v2:172.21.15.119:6838/937297340,v1:172.21.15.119:6839/937297340] conn(0x55fb896a0ed0 0x55fb8970ab70 crc :-1 s=READY pgs=20 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=16a93156a127df66 server_cookie=4c9c453f9c9cd809 in_seq=0 out_seq=0 2024-04-23T10:26:08.492 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:08.487+0000 7fa0bb7fe640 1 -- 172.21.15.114:0/3170683979 <== mds.0 v2:172.21.15.119:6838/937297340 1 ==== command_reply(tid 0: 0 ) ==== 8+0+332 (crc 0 0 0) 0x55fb895f64e0 con 0x55fb896a0ed0 2024-04-23T10:26:08.492 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:08.487+0000 7fa0bb7fe640 10 client.15108 handle_command_reply: tid=0 2024-04-23T10:26:08.492 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:08.487+0000 7fa0e5182740 1 client.15108 shutdown 2024-04-23T10:26:08.492 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:08.487+0000 7fa0e5182740 1 -- 172.21.15.114:0/3170683979 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55fb8968b710 msgr2=0x55fb8969a780 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T10:26:08.492 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:08.487+0000 7fa0e5182740 1 --2- 172.21.15.114:0/3170683979 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55fb8968b710 0x55fb8969a780 secure :-1 s=READY pgs=345 cs=0 l=1 rev1=1 crypto rx=0x7fa0d4009e70 tx=0x7fa0d4013000 comp rx=0 tx=0).stop 2024-04-23T10:26:08.492 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:08.487+0000 7fa0e5182740 1 -- 172.21.15.114:0/3170683979 shutdown_connections 2024-04-23T10:26:08.492 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:08.487+0000 7fa0e5182740 1 --2- 172.21.15.114:0/3170683979 >> [v2:172.21.15.119:6838/937297340,v1:172.21.15.119:6839/937297340] conn(0x55fb896a0ed0 0x55fb8970ab70 crc :-1 s=READY pgs=20 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:26:08.492 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:08.487+0000 7fa0e5182740 1 --2- 172.21.15.114:0/3170683979 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55fb8968b710 0x55fb8969a780 unknown :-1 s=CLOSED pgs=345 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:26:08.492 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:08.487+0000 7fa0e5182740 1 --2- 172.21.15.114:0/3170683979 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x55fb8968ad50 0x55fb89697760 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:26:08.492 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:08.487+0000 7fa0bb7fe640 0 client.15108 ms_handle_reset on v2:172.21.15.119:6838/937297340 2024-04-23T10:26:08.492 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:08.487+0000 7fa0e5182740 1 --2- 172.21.15.114:0/3170683979 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x55fb89689b80 0x55fb89697220 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:26:08.492 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:08.487+0000 7fa0e5182740 1 -- 172.21.15.114:0/3170683979 >> 172.21.15.114:0/3170683979 conn(0x55fb89685070 msgr2=0x55fb89687d50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T10:26:08.492 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:08.487+0000 7fa0e5182740 1 -- 172.21.15.114:0/3170683979 shutdown_connections 2024-04-23T10:26:08.492 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:08.487+0000 7fa0e5182740 1 -- 172.21.15.114:0/3170683979 wait complete. 2024-04-23T10:26:08.492 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:08.488+0000 7fa0e5182740 20 client.15108 trim_cache size 0 max 16384 2024-04-23T10:26:08.493 INFO:teuthology.orchestra.run.smithi114.stdout:{ 2024-04-23T10:26:08.493 INFO:teuthology.orchestra.run.smithi114.stdout: "cluster_fsid": "aa240572-0159-11ef-bc93-c7b262605968", 2024-04-23T10:26:08.493 INFO:teuthology.orchestra.run.smithi114.stdout: "whoami": 0, 2024-04-23T10:26:08.493 INFO:teuthology.orchestra.run.smithi114.stdout: "id": 24545, 2024-04-23T10:26:08.493 INFO:teuthology.orchestra.run.smithi114.stdout: "want_state": "up:active", 2024-04-23T10:26:08.493 INFO:teuthology.orchestra.run.smithi114.stdout: "state": "up:active", 2024-04-23T10:26:08.493 INFO:teuthology.orchestra.run.smithi114.stdout: "fs_name": "cephfs", 2024-04-23T10:26:08.493 INFO:teuthology.orchestra.run.smithi114.stdout: "rank_uptime": 84.169753275999994, 2024-04-23T10:26:08.493 INFO:teuthology.orchestra.run.smithi114.stdout: "mdsmap_epoch": 20, 2024-04-23T10:26:08.493 INFO:teuthology.orchestra.run.smithi114.stdout: "osdmap_epoch": 89, 2024-04-23T10:26:08.493 INFO:teuthology.orchestra.run.smithi114.stdout: "osdmap_epoch_barrier": 87, 2024-04-23T10:26:08.493 INFO:teuthology.orchestra.run.smithi114.stdout: "uptime": 90.075694216000002 2024-04-23T10:26:08.493 INFO:teuthology.orchestra.run.smithi114.stdout:} 2024-04-23T10:26:08.494 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:08.488+0000 7fa0e2474640 1 -- 172.21.15.114:0/3741647453 >> [v2:172.21.15.114:6800/465333808,v1:172.21.15.114:6801/465333808] conn(0x7fa0c40a3c40 msgr2=0x7fa0c40a6100 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T10:26:08.494 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:08.488+0000 7fa0e2474640 1 --2- 172.21.15.114:0/3741647453 >> [v2:172.21.15.114:6800/465333808,v1:172.21.15.114:6801/465333808] conn(0x7fa0c40a3c40 0x7fa0c40a6100 secure :-1 s=READY pgs=225 cs=0 l=1 rev1=1 crypto rx=0x7fa0d4019000 tx=0x7fa0d4018950 comp rx=0 tx=0).stop 2024-04-23T10:26:08.494 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:08.488+0000 7fa0e2474640 1 -- 172.21.15.114:0/3741647453 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fa0dc101f90 msgr2=0x7fa0dc075d00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T10:26:08.494 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:08.488+0000 7fa0e2474640 1 --2- 172.21.15.114:0/3741647453 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fa0dc101f90 0x7fa0dc075d00 secure :-1 s=READY pgs=341 cs=0 l=1 rev1=1 crypto rx=0x7fa0d000e9c0 tx=0x7fa0d000ee80 comp rx=0 tx=0).stop 2024-04-23T10:26:08.495 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:08.488+0000 7fa0e2474640 1 -- 172.21.15.114:0/3741647453 shutdown_connections 2024-04-23T10:26:08.495 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:08.488+0000 7fa0e2474640 1 --2- 172.21.15.114:0/3741647453 >> [v2:172.21.15.114:6800/465333808,v1:172.21.15.114:6801/465333808] conn(0x7fa0c40a3c40 0x7fa0c40a6100 unknown :-1 s=CLOSED pgs=225 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:26:08.495 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:08.488+0000 7fa0e2474640 1 --2- 172.21.15.114:0/3741647453 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fa0dc103b40 0x7fa0dc074880 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:26:08.495 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:08.488+0000 7fa0e2474640 1 --2- 172.21.15.114:0/3741647453 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fa0dc103180 0x7fa0dc074340 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:26:08.495 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:08.488+0000 7fa0e2474640 1 --2- 172.21.15.114:0/3741647453 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fa0dc101f90 0x7fa0dc075d00 unknown :-1 s=CLOSED pgs=341 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:26:08.495 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:08.488+0000 7fa0e2474640 1 -- 172.21.15.114:0/3741647453 >> 172.21.15.114:0/3741647453 conn(0x7fa0dc0fd500 msgr2=0x7fa0dc106480 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T10:26:08.495 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:08.488+0000 7fa0e2474640 1 -- 172.21.15.114:0/3741647453 shutdown_connections 2024-04-23T10:26:08.495 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:08.488+0000 7fa0e2474640 1 -- 172.21.15.114:0/3741647453 wait complete. 2024-04-23T10:26:08.495 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 23 10:26:08 smithi119 ceph-mon[32608]: from='client.? 172.21.15.114:0/3533348400' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2024-04-23T10:26:08.495 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 23 10:26:08 smithi119 ceph-mon[32608]: from='client.? 172.21.15.114:0/440960242' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-23T10:26:08.495 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 23 10:26:08 smithi119 ceph-mon[32608]: from='client.? 172.21.15.114:0/872498497' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-23T10:26:08.498 INFO:tasks.cephadm:Setup complete, yielding 2024-04-23T10:26:08.498 INFO:teuthology.run_tasks:Running task cephadm.shell... 2024-04-23T10:26:08.506 INFO:tasks.cephadm:Running commands on role mon.a host ubuntu@smithi114.front.sepia.ceph.com 2024-04-23T10:26:08.507 DEBUG:teuthology.orchestra.run.smithi114:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d69f13d827952bb07bb9b7973bca14f97d764d6f shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid aa240572-0159-11ef-bc93-c7b262605968 -- bash -c 'ceph orch status' 2024-04-23T10:26:08.742 INFO:teuthology.orchestra.run.smithi114.stderr:Inferring config /var/lib/ceph/aa240572-0159-11ef-bc93-c7b262605968/mon.a/config 2024-04-23T10:26:09.161 INFO:journalctl@ceph.mon.a.smithi114.stdout:Apr 23 10:26:09 smithi114 ceph-mon[29255]: pgmap v330: 97 pgs: 97 active+clean; 641 KiB data, 371 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-04-23T10:26:09.162 INFO:journalctl@ceph.mon.a.smithi114.stdout:Apr 23 10:26:09 smithi114 ceph-mon[29255]: from='client.? 172.21.15.114:0/103719779' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-23T10:26:09.328 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 23 10:26:09 smithi204 ceph-mon[32649]: pgmap v330: 97 pgs: 97 active+clean; 641 KiB data, 371 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-04-23T10:26:09.328 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 23 10:26:09 smithi204 ceph-mon[32649]: from='client.? 172.21.15.114:0/103719779' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-23T10:26:09.492 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 23 10:26:09 smithi119 ceph-mon[32608]: pgmap v330: 97 pgs: 97 active+clean; 641 KiB data, 371 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-04-23T10:26:09.492 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 23 10:26:09 smithi119 ceph-mon[32608]: from='client.? 172.21.15.114:0/103719779' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-23T10:26:10.243 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:10.244+0000 7f995684c700 1 -- 172.21.15.114:0/629174712 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f9950103d70 msgr2=0x7f99501041d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T10:26:10.243 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:10.244+0000 7f995684c700 1 --2- 172.21.15.114:0/629174712 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f9950103d70 0x7f99501041d0 secure :-1 s=READY pgs=346 cs=0 l=1 rev1=1 crypto rx=0x7f9944009990 tx=0x7f9944009ca0 comp rx=0 tx=0).stop 2024-04-23T10:26:10.244 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:10.244+0000 7f995684c700 1 -- 172.21.15.114:0/629174712 shutdown_connections 2024-04-23T10:26:10.244 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:10.244+0000 7f995684c700 1 --2- 172.21.15.114:0/629174712 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f9950104710 0x7f995010afc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:26:10.244 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:10.244+0000 7f995684c700 1 --2- 172.21.15.114:0/629174712 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f9950100d90 0x7f99501011b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:26:10.244 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:10.244+0000 7f995684c700 1 --2- 172.21.15.114:0/629174712 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f9950103d70 0x7f99501041d0 unknown :-1 s=CLOSED pgs=346 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:26:10.244 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:10.244+0000 7f995684c700 1 -- 172.21.15.114:0/629174712 >> 172.21.15.114:0/629174712 conn(0x7f99500fc6f0 msgr2=0x7f99500feb30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T10:26:10.244 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:10.244+0000 7f995684c700 1 -- 172.21.15.114:0/629174712 shutdown_connections 2024-04-23T10:26:10.244 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:10.244+0000 7f995684c700 1 -- 172.21.15.114:0/629174712 wait complete. 2024-04-23T10:26:10.246 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:10.245+0000 7f995684c700 1 Processor -- start 2024-04-23T10:26:10.246 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:10.245+0000 7f995684c700 1 -- start start 2024-04-23T10:26:10.246 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:10.245+0000 7f995684c700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f9950100d90 0x7f9950198b30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T10:26:10.246 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:10.245+0000 7f995684c700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f9950103d70 0x7f9950199070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T10:26:10.246 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:10.245+0000 7f995684c700 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f9950104710 0x7f99501a00f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T10:26:10.246 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:10.245+0000 7f995684c700 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x7f995010e550 con 0x7f9950103d70 2024-04-23T10:26:10.246 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:10.245+0000 7f995684c700 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 -- 0x7f995010e3d0 con 0x7f9950100d90 2024-04-23T10:26:10.247 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:10.245+0000 7f995684c700 1 -- --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_getmap magic: 0 -- 0x7f995010e6d0 con 0x7f9950104710 2024-04-23T10:26:10.247 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:10.245+0000 7f994f7fe700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f9950103d70 0x7f9950199070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T10:26:10.247 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:10.246+0000 7f994f7fe700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f9950103d70 0x7f9950199070 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.114:58268/0 (socket says 172.21.15.114:58268) 2024-04-23T10:26:10.247 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:10.246+0000 7f994f7fe700 1 -- 172.21.15.114:0/3657370540 learned_addr learned my addr 172.21.15.114:0/3657370540 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-04-23T10:26:10.247 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:10.246+0000 7f994ffff700 1 --2- 172.21.15.114:0/3657370540 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f9950100d90 0x7f9950198b30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T10:26:10.247 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:10.246+0000 7f9954de9700 1 --2- 172.21.15.114:0/3657370540 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f9950104710 0x7f99501a00f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T10:26:10.247 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:10.246+0000 7f994f7fe700 1 -- 172.21.15.114:0/3657370540 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f9950104710 msgr2=0x7f99501a00f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T10:26:10.247 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:10.246+0000 7f994f7fe700 1 --2- 172.21.15.114:0/3657370540 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f9950104710 0x7f99501a00f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:26:10.247 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:10.246+0000 7f994f7fe700 1 -- 172.21.15.114:0/3657370540 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f9950100d90 msgr2=0x7f9950198b30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T10:26:10.247 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:10.246+0000 7f994f7fe700 1 --2- 172.21.15.114:0/3657370540 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f9950100d90 0x7f9950198b30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:26:10.247 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:10.246+0000 7f994f7fe700 1 -- 172.21.15.114:0/3657370540 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f99501a0790 con 0x7f9950103d70 2024-04-23T10:26:10.247 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:10.246+0000 7f9954de9700 1 --2- 172.21.15.114:0/3657370540 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f9950104710 0x7f99501a00f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-23T10:26:10.247 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:10.246+0000 7f994f7fe700 1 --2- 172.21.15.114:0/3657370540 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f9950103d70 0x7f9950199070 secure :-1 s=READY pgs=347 cs=0 l=1 rev1=1 crypto rx=0x7f99440036f0 tx=0x7f99440047b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T10:26:10.247 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:10.246+0000 7f994d7fa700 1 -- 172.21.15.114:0/3657370540 <== mon.0 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9944027070 con 0x7f9950103d70 2024-04-23T10:26:10.247 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:10.246+0000 7f994d7fa700 1 -- 172.21.15.114:0/3657370540 <== mon.0 v2:172.21.15.114:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f9944004080 con 0x7f9950103d70 2024-04-23T10:26:10.247 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:10.246+0000 7f994d7fa700 1 -- 172.21.15.114:0/3657370540 <== mon.0 v2:172.21.15.114:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9944022d50 con 0x7f9950103d70 2024-04-23T10:26:10.248 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:10.246+0000 7f995684c700 1 -- 172.21.15.114:0/3657370540 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f99501a0a20 con 0x7f9950103d70 2024-04-23T10:26:10.248 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:10.246+0000 7f995684c700 1 -- 172.21.15.114:0/3657370540 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f99501a0e70 con 0x7f9950103d70 2024-04-23T10:26:10.248 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:10.247+0000 7f995684c700 1 -- 172.21.15.114:0/3657370540 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f9950069730 con 0x7f9950103d70 2024-04-23T10:26:10.248 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:10.248+0000 7f994d7fa700 1 -- 172.21.15.114:0/3657370540 <== mon.0 v2:172.21.15.114:3300/0 4 ==== mgrmap(e 17) ==== 138698+0+0 (secure 0 0 0) 0x7f994402a070 con 0x7f9950103d70 2024-04-23T10:26:10.249 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:10.249+0000 7f994d7fa700 1 --2- 172.21.15.114:0/3657370540 >> [v2:172.21.15.114:6800/465333808,v1:172.21.15.114:6801/465333808] conn(0x7f99380a3cd0 0x7f99380a6190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T10:26:10.249 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:10.249+0000 7f994ffff700 1 --2- 172.21.15.114:0/3657370540 >> [v2:172.21.15.114:6800/465333808,v1:172.21.15.114:6801/465333808] conn(0x7f99380a3cd0 0x7f99380a6190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T10:26:10.249 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:10.249+0000 7f994d7fa700 1 -- 172.21.15.114:0/3657370540 <== mon.0 v2:172.21.15.114:3300/0 5 ==== osd_map(89..89 src has 1..89) ==== 8269+0+0 (secure 0 0 0) 0x7f99440d1540 con 0x7f9950103d70 2024-04-23T10:26:10.249 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:10.250+0000 7f994ffff700 1 --2- 172.21.15.114:0/3657370540 >> [v2:172.21.15.114:6800/465333808,v1:172.21.15.114:6801/465333808] conn(0x7f99380a3cd0 0x7f99380a6190 secure :-1 s=READY pgs=226 cs=0 l=1 rev1=1 crypto rx=0x7f995006afb0 tx=0x7f994000b410 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T10:26:10.252 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:10.254+0000 7f994d7fa700 1 -- 172.21.15.114:0/3657370540 <== mon.0 v2:172.21.15.114:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f994405eff0 con 0x7f9950103d70 2024-04-23T10:26:10.467 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:10.468+0000 7f995684c700 1 -- 172.21.15.114:0/3657370540 --> [v2:172.21.15.114:6800/465333808,v1:172.21.15.114:6801/465333808] -- mgr_command(tid 0: {"prefix": "orch status", "target": ["mon-mgr", ""]}) -- 0x7f9950062e80 con 0x7f99380a3cd0 2024-04-23T10:26:10.468 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:10.469+0000 7f994d7fa700 1 -- 172.21.15.114:0/3657370540 <== mgr.14152 v2:172.21.15.114:6800/465333808 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+43 (secure 0 0 0) 0x7f9950062e80 con 0x7f99380a3cd0 2024-04-23T10:26:10.468 INFO:teuthology.orchestra.run.smithi114.stdout:Backend: cephadm 2024-04-23T10:26:10.468 INFO:teuthology.orchestra.run.smithi114.stdout:Available: Yes 2024-04-23T10:26:10.468 INFO:teuthology.orchestra.run.smithi114.stdout:Paused: No 2024-04-23T10:26:10.471 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:10.471+0000 7f995684c700 1 -- 172.21.15.114:0/3657370540 >> [v2:172.21.15.114:6800/465333808,v1:172.21.15.114:6801/465333808] conn(0x7f99380a3cd0 msgr2=0x7f99380a6190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T10:26:10.471 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:10.471+0000 7f995684c700 1 --2- 172.21.15.114:0/3657370540 >> [v2:172.21.15.114:6800/465333808,v1:172.21.15.114:6801/465333808] conn(0x7f99380a3cd0 0x7f99380a6190 secure :-1 s=READY pgs=226 cs=0 l=1 rev1=1 crypto rx=0x7f995006afb0 tx=0x7f994000b410 comp rx=0 tx=0).stop 2024-04-23T10:26:10.471 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:10.472+0000 7f995684c700 1 -- 172.21.15.114:0/3657370540 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f9950103d70 msgr2=0x7f9950199070 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T10:26:10.471 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:10.472+0000 7f995684c700 1 --2- 172.21.15.114:0/3657370540 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f9950103d70 0x7f9950199070 secure :-1 s=READY pgs=347 cs=0 l=1 rev1=1 crypto rx=0x7f99440036f0 tx=0x7f99440047b0 comp rx=0 tx=0).stop 2024-04-23T10:26:10.471 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:10.472+0000 7f995684c700 1 -- 172.21.15.114:0/3657370540 shutdown_connections 2024-04-23T10:26:10.471 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:10.472+0000 7f995684c700 1 --2- 172.21.15.114:0/3657370540 >> [v2:172.21.15.114:6800/465333808,v1:172.21.15.114:6801/465333808] conn(0x7f99380a3cd0 0x7f99380a6190 unknown :-1 s=CLOSED pgs=226 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:26:10.471 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:10.472+0000 7f995684c700 1 --2- 172.21.15.114:0/3657370540 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f9950100d90 0x7f9950198b30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:26:10.471 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:10.472+0000 7f995684c700 1 --2- 172.21.15.114:0/3657370540 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f9950104710 0x7f99501a00f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:26:10.471 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:10.472+0000 7f995684c700 1 --2- 172.21.15.114:0/3657370540 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f9950103d70 0x7f9950199070 unknown :-1 s=CLOSED pgs=347 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:26:10.471 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:10.472+0000 7f995684c700 1 -- 172.21.15.114:0/3657370540 >> 172.21.15.114:0/3657370540 conn(0x7f99500fc6f0 msgr2=0x7f9950107070 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T10:26:10.471 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:10.472+0000 7f995684c700 1 -- 172.21.15.114:0/3657370540 shutdown_connections 2024-04-23T10:26:10.472 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:10.472+0000 7f995684c700 1 -- 172.21.15.114:0/3657370540 wait complete. 2024-04-23T10:26:11.060 DEBUG:teuthology.orchestra.run.smithi114:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d69f13d827952bb07bb9b7973bca14f97d764d6f shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid aa240572-0159-11ef-bc93-c7b262605968 -- bash -c 'ceph orch ps' 2024-04-23T10:26:11.254 INFO:journalctl@ceph.mon.a.smithi114.stdout:Apr 23 10:26:11 smithi114 ceph-mon[29255]: pgmap v331: 97 pgs: 97 active+clean; 641 KiB data, 371 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-04-23T10:26:11.307 INFO:teuthology.orchestra.run.smithi114.stderr:Inferring config /var/lib/ceph/aa240572-0159-11ef-bc93-c7b262605968/mon.a/config 2024-04-23T10:26:11.327 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 23 10:26:11 smithi204 ceph-mon[32649]: pgmap v331: 97 pgs: 97 active+clean; 641 KiB data, 371 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-04-23T10:26:11.492 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 23 10:26:11 smithi119 ceph-mon[32608]: pgmap v331: 97 pgs: 97 active+clean; 641 KiB data, 371 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-04-23T10:26:12.238 INFO:journalctl@ceph.mon.a.smithi114.stdout:Apr 23 10:26:12 smithi114 ceph-mon[29255]: from='client.15114 -' entity='client.admin' cmd=[{"prefix": "orch status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-23T10:26:12.328 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 23 10:26:12 smithi204 ceph-mon[32649]: from='client.15114 -' entity='client.admin' cmd=[{"prefix": "orch status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-23T10:26:12.492 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 23 10:26:12 smithi119 ceph-mon[32608]: from='client.15114 -' entity='client.admin' cmd=[{"prefix": "orch status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-23T10:26:13.044 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:13.045+0000 7fce960dd700 1 -- 172.21.15.114:0/2424483394 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fce90103550 msgr2=0x7fce90109e00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T10:26:13.044 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:13.045+0000 7fce960dd700 1 --2- 172.21.15.114:0/2424483394 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fce90103550 0x7fce90109e00 secure :-1 s=READY pgs=348 cs=0 l=1 rev1=1 crypto rx=0x7fce8800b0d0 tx=0x7fce8800b3e0 comp rx=0 tx=0).stop 2024-04-23T10:26:13.045 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:13.045+0000 7fce960dd700 1 -- 172.21.15.114:0/2424483394 shutdown_connections 2024-04-23T10:26:13.045 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:13.045+0000 7fce960dd700 1 --2- 172.21.15.114:0/2424483394 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fce90103550 0x7fce90109e00 unknown :-1 s=CLOSED pgs=348 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:26:13.045 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:13.045+0000 7fce960dd700 1 --2- 172.21.15.114:0/2424483394 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fce901019b0 0x7fce90101dd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:26:13.045 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:13.045+0000 7fce960dd700 1 --2- 172.21.15.114:0/2424483394 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fce90102bb0 0x7fce90103010 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:26:13.045 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:13.045+0000 7fce960dd700 1 -- 172.21.15.114:0/2424483394 >> 172.21.15.114:0/2424483394 conn(0x7fce900fcf70 msgr2=0x7fce900ff390 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T10:26:13.045 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:13.045+0000 7fce960dd700 1 -- 172.21.15.114:0/2424483394 shutdown_connections 2024-04-23T10:26:13.045 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:13.045+0000 7fce960dd700 1 -- 172.21.15.114:0/2424483394 wait complete. 2024-04-23T10:26:13.047 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:13.046+0000 7fce960dd700 1 Processor -- start 2024-04-23T10:26:13.047 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:13.046+0000 7fce960dd700 1 -- start start 2024-04-23T10:26:13.047 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:13.046+0000 7fce960dd700 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fce901019b0 0x7fce90197ad0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T10:26:13.047 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:13.046+0000 7fce960dd700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fce90102bb0 0x7fce90198010 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T10:26:13.047 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:13.046+0000 7fce960dd700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fce90103550 0x7fce9019f090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T10:26:13.047 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:13.046+0000 7fce960dd700 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x7fce9010d450 con 0x7fce90102bb0 2024-04-23T10:26:13.047 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:13.046+0000 7fce960dd700 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 -- 0x7fce9010d2d0 con 0x7fce90103550 2024-04-23T10:26:13.047 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:13.046+0000 7fce960dd700 1 -- --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_getmap magic: 0 -- 0x7fce9010d5d0 con 0x7fce901019b0 2024-04-23T10:26:13.047 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:13.047+0000 7fce8effd700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fce90102bb0 0x7fce90198010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T10:26:13.047 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:13.047+0000 7fce8effd700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fce90102bb0 0x7fce90198010 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.114:58276/0 (socket says 172.21.15.114:58276) 2024-04-23T10:26:13.047 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:13.047+0000 7fce8effd700 1 -- 172.21.15.114:0/3320537489 learned_addr learned my addr 172.21.15.114:0/3320537489 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-04-23T10:26:13.047 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:13.047+0000 7fce8f7fe700 1 --2- 172.21.15.114:0/3320537489 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fce901019b0 0x7fce90197ad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T10:26:13.047 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:13.047+0000 7fce8ffff700 1 --2- 172.21.15.114:0/3320537489 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fce90103550 0x7fce9019f090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T10:26:13.047 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:13.047+0000 7fce8effd700 1 -- 172.21.15.114:0/3320537489 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fce901019b0 msgr2=0x7fce90197ad0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T10:26:13.048 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:13.047+0000 7fce8effd700 1 --2- 172.21.15.114:0/3320537489 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fce901019b0 0x7fce90197ad0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:26:13.048 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:13.047+0000 7fce8effd700 1 -- 172.21.15.114:0/3320537489 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fce90103550 msgr2=0x7fce9019f090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T10:26:13.048 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:13.047+0000 7fce8effd700 1 --2- 172.21.15.114:0/3320537489 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fce90103550 0x7fce9019f090 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:26:13.048 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:13.047+0000 7fce8effd700 1 -- 172.21.15.114:0/3320537489 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fce9019f730 con 0x7fce90102bb0 2024-04-23T10:26:13.048 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:13.047+0000 7fce8effd700 1 --2- 172.21.15.114:0/3320537489 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fce90102bb0 0x7fce90198010 secure :-1 s=READY pgs=349 cs=0 l=1 rev1=1 crypto rx=0x7fce800099a0 tx=0x7fce80009cb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T10:26:13.048 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:13.047+0000 7fce8cff9700 1 -- 172.21.15.114:0/3320537489 <== mon.0 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fce8001b070 con 0x7fce90102bb0 2024-04-23T10:26:13.048 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:13.047+0000 7fce8cff9700 1 -- 172.21.15.114:0/3320537489 <== mon.0 v2:172.21.15.114:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fce8000d260 con 0x7fce90102bb0 2024-04-23T10:26:13.048 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:13.047+0000 7fce960dd700 1 -- 172.21.15.114:0/3320537489 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fce9006b1f0 con 0x7fce90102bb0 2024-04-23T10:26:13.048 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:13.047+0000 7fce8cff9700 1 -- 172.21.15.114:0/3320537489 <== mon.0 v2:172.21.15.114:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fce8000c9c0 con 0x7fce90102bb0 2024-04-23T10:26:13.048 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:13.047+0000 7fce960dd700 1 -- 172.21.15.114:0/3320537489 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fce9006b730 con 0x7fce90102bb0 2024-04-23T10:26:13.049 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:13.048+0000 7fce960dd700 1 -- 172.21.15.114:0/3320537489 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fce90069730 con 0x7fce90102bb0 2024-04-23T10:26:13.049 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:13.049+0000 7fce8cff9700 1 -- 172.21.15.114:0/3320537489 <== mon.0 v2:172.21.15.114:3300/0 4 ==== mgrmap(e 17) ==== 138698+0+0 (secure 0 0 0) 0x7fce8001e070 con 0x7fce90102bb0 2024-04-23T10:26:13.049 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:13.050+0000 7fce8cff9700 1 --2- 172.21.15.114:0/3320537489 >> [v2:172.21.15.114:6800/465333808,v1:172.21.15.114:6801/465333808] conn(0x7fce780a3cd0 0x7fce780a6190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T10:26:13.049 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:13.050+0000 7fce8f7fe700 1 --2- 172.21.15.114:0/3320537489 >> [v2:172.21.15.114:6800/465333808,v1:172.21.15.114:6801/465333808] conn(0x7fce780a3cd0 0x7fce780a6190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T10:26:13.049 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:13.050+0000 7fce8cff9700 1 -- 172.21.15.114:0/3320537489 <== mon.0 v2:172.21.15.114:3300/0 5 ==== osd_map(89..89 src has 1..89) ==== 8269+0+0 (secure 0 0 0) 0x7fce800d1320 con 0x7fce90102bb0 2024-04-23T10:26:13.050 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:13.050+0000 7fce8f7fe700 1 --2- 172.21.15.114:0/3320537489 >> [v2:172.21.15.114:6800/465333808,v1:172.21.15.114:6801/465333808] conn(0x7fce780a3cd0 0x7fce780a6190 secure :-1 s=READY pgs=227 cs=0 l=1 rev1=1 crypto rx=0x7fce90102a10 tx=0x7fce8400b410 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T10:26:13.052 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:13.054+0000 7fce8cff9700 1 -- 172.21.15.114:0/3320537489 <== mon.0 v2:172.21.15.114:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7fce8009db80 con 0x7fce90102bb0 2024-04-23T10:26:13.254 INFO:journalctl@ceph.mon.a.smithi114.stdout:Apr 23 10:26:13 smithi114 ceph-mon[29255]: pgmap v332: 97 pgs: 97 active+clean; 641 KiB data, 371 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-23T10:26:13.264 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:13.265+0000 7fce960dd700 1 -- 172.21.15.114:0/3320537489 --> [v2:172.21.15.114:6800/465333808,v1:172.21.15.114:6801/465333808] -- mgr_command(tid 0: {"prefix": "orch ps", "target": ["mon-mgr", ""]}) -- 0x7fce9006b990 con 0x7fce780a3cd0 2024-04-23T10:26:13.278 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:13.279+0000 7fce8cff9700 1 -- 172.21.15.114:0/3320537489 <== mgr.14152 v2:172.21.15.114:6800/465333808 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+4185 (secure 0 0 0) 0x7fce9006b990 con 0x7fce780a3cd0 2024-04-23T10:26:13.278 INFO:teuthology.orchestra.run.smithi114.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-04-23T10:26:13.278 INFO:teuthology.orchestra.run.smithi114.stdout:mds.a smithi114 running (97s) 15s ago 97s 16.1M - 19.0.0-3156-gd69f13d8 da4b004c09e8 b4bc3a391217 2024-04-23T10:26:13.278 INFO:teuthology.orchestra.run.smithi114.stdout:mds.b smithi119 running (95s) 28s ago 94s 26.0M - 19.0.0-3156-gd69f13d8 da4b004c09e8 c3c180699097 2024-04-23T10:26:13.278 INFO:teuthology.orchestra.run.smithi114.stdout:mds.c smithi204 running (109s) 28s ago 108s 16.8M - 19.0.0-3156-gd69f13d8 da4b004c09e8 084acf2b16c7 2024-04-23T10:26:13.278 INFO:teuthology.orchestra.run.smithi114.stdout:mds.d smithi114 running (117s) 15s ago 116s 13.8M - 19.0.0-3156-gd69f13d8 da4b004c09e8 6fd229fdfcca 2024-04-23T10:26:13.278 INFO:teuthology.orchestra.run.smithi114.stdout:mds.e smithi119 running (114s) 28s ago 114s 20.4M - 19.0.0-3156-gd69f13d8 da4b004c09e8 d5e5a1bdb2a7 2024-04-23T10:26:13.278 INFO:teuthology.orchestra.run.smithi114.stdout:mds.f smithi204 running (111s) 28s ago 111s 13.9M - 19.0.0-3156-gd69f13d8 da4b004c09e8 4fd9d2764c4f 2024-04-23T10:26:13.278 INFO:teuthology.orchestra.run.smithi114.stdout:mds.g smithi114 running (100s) 15s ago 100s 16.3M - 19.0.0-3156-gd69f13d8 da4b004c09e8 9c754624738d 2024-04-23T10:26:13.278 INFO:teuthology.orchestra.run.smithi114.stdout:mds.h smithi119 running (106s) 28s ago 105s 18.1M - 19.0.0-3156-gd69f13d8 da4b004c09e8 97a4ae8c5002 2024-04-23T10:26:13.279 INFO:teuthology.orchestra.run.smithi114.stdout:mds.i smithi204 running (103s) 28s ago 103s 20.1M - 19.0.0-3156-gd69f13d8 da4b004c09e8 6e31504a857c 2024-04-23T10:26:13.279 INFO:teuthology.orchestra.run.smithi114.stdout:mds.j smithi114 running (2m) 15s ago 119s 16.3M - 19.0.0-3156-gd69f13d8 da4b004c09e8 383afeb57c86 2024-04-23T10:26:13.279 INFO:teuthology.orchestra.run.smithi114.stdout:mds.k smithi119 running (2m) 28s ago 2m 18.4M - 19.0.0-3156-gd69f13d8 da4b004c09e8 ef9a53e050a9 2024-04-23T10:26:13.279 INFO:teuthology.orchestra.run.smithi114.stdout:mds.l smithi204 running (2m) 28s ago 2m 16.2M - 19.0.0-3156-gd69f13d8 da4b004c09e8 ffce6fd32893 2024-04-23T10:26:13.279 INFO:teuthology.orchestra.run.smithi114.stdout:mgr.x smithi114 *:9283,8765 running (9m) 15s ago 9m 515M - 19.0.0-3156-gd69f13d8 da4b004c09e8 df0cd55890f9 2024-04-23T10:26:13.279 INFO:teuthology.orchestra.run.smithi114.stdout:mgr.y smithi119 *:8443,8765 running (7m) 28s ago 7m 453M - 19.0.0-3156-gd69f13d8 da4b004c09e8 c8a099050ba0 2024-04-23T10:26:13.279 INFO:teuthology.orchestra.run.smithi114.stdout:mgr.z smithi204 *:8443,8765 running (7m) 28s ago 7m 455M - 19.0.0-3156-gd69f13d8 da4b004c09e8 8a04d6bd3ee9 2024-04-23T10:26:13.279 INFO:teuthology.orchestra.run.smithi114.stdout:mon.a smithi114 running (9m) 15s ago 9m 53.4M 2048M 19.0.0-3156-gd69f13d8 da4b004c09e8 3c977ab8b867 2024-04-23T10:26:13.279 INFO:teuthology.orchestra.run.smithi114.stdout:mon.b smithi119 running (7m) 28s ago 7m 42.7M 2048M 19.0.0-3156-gd69f13d8 da4b004c09e8 903c3f3bd1b4 2024-04-23T10:26:13.279 INFO:teuthology.orchestra.run.smithi114.stdout:mon.c smithi204 running (7m) 28s ago 7m 39.3M 2048M 19.0.0-3156-gd69f13d8 da4b004c09e8 adbe74a807a5 2024-04-23T10:26:13.279 INFO:teuthology.orchestra.run.smithi114.stdout:osd.0 smithi114 running (7m) 15s ago 6m 56.7M 4272M 19.0.0-3156-gd69f13d8 da4b004c09e8 69855b5983de 2024-04-23T10:26:13.279 INFO:teuthology.orchestra.run.smithi114.stdout:osd.1 smithi119 running (6m) 28s ago 6m 48.7M 4272M 19.0.0-3156-gd69f13d8 da4b004c09e8 fb83cfbb8ef0 2024-04-23T10:26:13.279 INFO:teuthology.orchestra.run.smithi114.stdout:osd.2 smithi204 running (6m) 28s ago 6m 58.1M 4272M 19.0.0-3156-gd69f13d8 da4b004c09e8 b5133a0335f7 2024-04-23T10:26:13.279 INFO:teuthology.orchestra.run.smithi114.stdout:osd.3 smithi114 running (5m) 15s ago 5m 47.2M 4272M 19.0.0-3156-gd69f13d8 da4b004c09e8 2139eacb0b2d 2024-04-23T10:26:13.279 INFO:teuthology.orchestra.run.smithi114.stdout:osd.4 smithi119 running (5m) 28s ago 5m 45.9M 4272M 19.0.0-3156-gd69f13d8 da4b004c09e8 350ca9fa13c1 2024-04-23T10:26:13.279 INFO:teuthology.orchestra.run.smithi114.stdout:osd.5 smithi204 running (4m) 28s ago 4m 54.8M 4272M 19.0.0-3156-gd69f13d8 da4b004c09e8 9ba3dca2bfcc 2024-04-23T10:26:13.280 INFO:teuthology.orchestra.run.smithi114.stdout:osd.6 smithi114 running (4m) 15s ago 4m 52.9M 4272M 19.0.0-3156-gd69f13d8 da4b004c09e8 8d9b1fa93580 2024-04-23T10:26:13.280 INFO:teuthology.orchestra.run.smithi114.stdout:osd.7 smithi119 running (4m) 28s ago 4m 54.0M 4272M 19.0.0-3156-gd69f13d8 da4b004c09e8 1891cc449f68 2024-04-23T10:26:13.280 INFO:teuthology.orchestra.run.smithi114.stdout:osd.8 smithi204 running (3m) 28s ago 3m 55.0M 4272M 19.0.0-3156-gd69f13d8 da4b004c09e8 91762dd15696 2024-04-23T10:26:13.280 INFO:teuthology.orchestra.run.smithi114.stdout:osd.9 smithi114 running (3m) 15s ago 3m 43.2M 4272M 19.0.0-3156-gd69f13d8 da4b004c09e8 4c9e36e99c22 2024-04-23T10:26:13.280 INFO:teuthology.orchestra.run.smithi114.stdout:osd.10 smithi119 running (2m) 28s ago 2m 47.3M 4272M 19.0.0-3156-gd69f13d8 da4b004c09e8 44ce77aa1dad 2024-04-23T10:26:13.280 INFO:teuthology.orchestra.run.smithi114.stdout:osd.11 smithi204 running (2m) 28s ago 2m 51.9M 4272M 19.0.0-3156-gd69f13d8 da4b004c09e8 683e1a039db5 2024-04-23T10:26:13.281 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:13.281+0000 7fce960dd700 1 -- 172.21.15.114:0/3320537489 >> [v2:172.21.15.114:6800/465333808,v1:172.21.15.114:6801/465333808] conn(0x7fce780a3cd0 msgr2=0x7fce780a6190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T10:26:13.281 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:13.281+0000 7fce960dd700 1 --2- 172.21.15.114:0/3320537489 >> [v2:172.21.15.114:6800/465333808,v1:172.21.15.114:6801/465333808] conn(0x7fce780a3cd0 0x7fce780a6190 secure :-1 s=READY pgs=227 cs=0 l=1 rev1=1 crypto rx=0x7fce90102a10 tx=0x7fce8400b410 comp rx=0 tx=0).stop 2024-04-23T10:26:13.281 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:13.281+0000 7fce960dd700 1 -- 172.21.15.114:0/3320537489 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fce90102bb0 msgr2=0x7fce90198010 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T10:26:13.281 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:13.281+0000 7fce960dd700 1 --2- 172.21.15.114:0/3320537489 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fce90102bb0 0x7fce90198010 secure :-1 s=READY pgs=349 cs=0 l=1 rev1=1 crypto rx=0x7fce800099a0 tx=0x7fce80009cb0 comp rx=0 tx=0).stop 2024-04-23T10:26:13.281 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:13.281+0000 7fce960dd700 1 -- 172.21.15.114:0/3320537489 shutdown_connections 2024-04-23T10:26:13.281 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:13.281+0000 7fce960dd700 1 --2- 172.21.15.114:0/3320537489 >> [v2:172.21.15.114:6800/465333808,v1:172.21.15.114:6801/465333808] conn(0x7fce780a3cd0 0x7fce780a6190 unknown :-1 s=CLOSED pgs=227 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:26:13.281 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:13.281+0000 7fce960dd700 1 --2- 172.21.15.114:0/3320537489 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fce90103550 0x7fce9019f090 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:26:13.281 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:13.281+0000 7fce960dd700 1 --2- 172.21.15.114:0/3320537489 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fce901019b0 0x7fce90197ad0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:26:13.281 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:13.281+0000 7fce960dd700 1 --2- 172.21.15.114:0/3320537489 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fce90102bb0 0x7fce90198010 unknown :-1 s=CLOSED pgs=349 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:26:13.281 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:13.281+0000 7fce960dd700 1 -- 172.21.15.114:0/3320537489 >> 172.21.15.114:0/3320537489 conn(0x7fce900fcf70 msgr2=0x7fce90105f20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T10:26:13.281 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:13.281+0000 7fce960dd700 1 -- 172.21.15.114:0/3320537489 shutdown_connections 2024-04-23T10:26:13.281 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:13.281+0000 7fce960dd700 1 -- 172.21.15.114:0/3320537489 wait complete. 2024-04-23T10:26:13.327 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 23 10:26:13 smithi204 ceph-mon[32649]: pgmap v332: 97 pgs: 97 active+clean; 641 KiB data, 371 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-23T10:26:13.492 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 23 10:26:13 smithi119 ceph-mon[32608]: pgmap v332: 97 pgs: 97 active+clean; 641 KiB data, 371 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-23T10:26:13.931 DEBUG:teuthology.orchestra.run.smithi114:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d69f13d827952bb07bb9b7973bca14f97d764d6f shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid aa240572-0159-11ef-bc93-c7b262605968 -- bash -c 'ceph orch ls' 2024-04-23T10:26:14.171 INFO:teuthology.orchestra.run.smithi114.stderr:Inferring config /var/lib/ceph/aa240572-0159-11ef-bc93-c7b262605968/mon.a/config 2024-04-23T10:26:15.328 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 23 10:26:15 smithi204 ceph-mon[32649]: from='client.15120 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-04-23T10:26:15.328 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 23 10:26:15 smithi204 ceph-mon[32649]: pgmap v333: 97 pgs: 97 active+clean; 641 KiB data, 371 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-04-23T10:26:15.426 INFO:journalctl@ceph.mon.a.smithi114.stdout:Apr 23 10:26:15 smithi114 ceph-mon[29255]: from='client.15120 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-04-23T10:26:15.426 INFO:journalctl@ceph.mon.a.smithi114.stdout:Apr 23 10:26:15 smithi114 ceph-mon[29255]: pgmap v333: 97 pgs: 97 active+clean; 641 KiB data, 371 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-04-23T10:26:15.492 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 23 10:26:15 smithi119 ceph-mon[32608]: from='client.15120 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-04-23T10:26:15.492 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 23 10:26:15 smithi119 ceph-mon[32608]: pgmap v333: 97 pgs: 97 active+clean; 641 KiB data, 371 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-04-23T10:26:15.874 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:15.873+0000 7f0e40a7e700 1 -- 172.21.15.114:0/1829122285 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f0e3c1046e0 msgr2=0x7f0e3c10af90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T10:26:15.874 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:15.873+0000 7f0e40a7e700 1 --2- 172.21.15.114:0/1829122285 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f0e3c1046e0 0x7f0e3c10af90 secure :-1 s=READY pgs=350 cs=0 l=1 rev1=1 crypto rx=0x7f0e3400b0a0 tx=0x7f0e3400b3b0 comp rx=0 tx=0).stop 2024-04-23T10:26:15.874 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:15.873+0000 7f0e40a7e700 1 -- 172.21.15.114:0/1829122285 shutdown_connections 2024-04-23T10:26:15.874 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:15.873+0000 7f0e40a7e700 1 --2- 172.21.15.114:0/1829122285 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f0e3c100d60 0x7f0e3c101180 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:26:15.875 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:15.873+0000 7f0e40a7e700 1 --2- 172.21.15.114:0/1829122285 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f0e3c1046e0 0x7f0e3c10af90 unknown :-1 s=CLOSED pgs=350 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:26:15.875 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:15.873+0000 7f0e40a7e700 1 --2- 172.21.15.114:0/1829122285 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f0e3c103d40 0x7f0e3c1041a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:26:15.875 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:15.873+0000 7f0e40a7e700 1 -- 172.21.15.114:0/1829122285 >> 172.21.15.114:0/1829122285 conn(0x7f0e3c0fc6a0 msgr2=0x7f0e3c0feb00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T10:26:15.875 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:15.873+0000 7f0e40a7e700 1 -- 172.21.15.114:0/1829122285 shutdown_connections 2024-04-23T10:26:15.875 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:15.873+0000 7f0e40a7e700 1 -- 172.21.15.114:0/1829122285 wait complete. 2024-04-23T10:26:15.875 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:15.874+0000 7f0e40a7e700 1 Processor -- start 2024-04-23T10:26:15.875 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:15.874+0000 7f0e40a7e700 1 -- start start 2024-04-23T10:26:15.875 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:15.874+0000 7f0e40a7e700 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f0e3c100d60 0x7f0e3c198cf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T10:26:15.875 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:15.874+0000 7f0e40a7e700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f0e3c103d40 0x7f0e3c199230 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T10:26:15.875 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:15.874+0000 7f0e40a7e700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f0e3c1046e0 0x7f0e3c1a0270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T10:26:15.875 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:15.874+0000 7f0e40a7e700 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x7f0e3c10e620 con 0x7f0e3c1046e0 2024-04-23T10:26:15.875 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:15.874+0000 7f0e40a7e700 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 -- 0x7f0e3c10e4a0 con 0x7f0e3c103d40 2024-04-23T10:26:15.875 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:15.874+0000 7f0e40a7e700 1 -- --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_getmap magic: 0 -- 0x7f0e3c10e7a0 con 0x7f0e3c100d60 2024-04-23T10:26:15.875 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:15.875+0000 7f0e3ad9d700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f0e3c1046e0 0x7f0e3c1a0270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T10:26:15.875 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:15.875+0000 7f0e3ad9d700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f0e3c1046e0 0x7f0e3c1a0270 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.114:58600/0 (socket says 172.21.15.114:58600) 2024-04-23T10:26:15.876 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:15.875+0000 7f0e3ad9d700 1 -- 172.21.15.114:0/2630075686 learned_addr learned my addr 172.21.15.114:0/2630075686 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-04-23T10:26:15.876 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:15.875+0000 7f0e3a59c700 1 --2- 172.21.15.114:0/2630075686 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f0e3c100d60 0x7f0e3c198cf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T10:26:15.876 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:15.875+0000 7f0e3ad9d700 1 -- 172.21.15.114:0/2630075686 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f0e3c100d60 msgr2=0x7f0e3c198cf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T10:26:15.876 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:15.875+0000 7f0e3ad9d700 1 --2- 172.21.15.114:0/2630075686 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f0e3c100d60 0x7f0e3c198cf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:26:15.876 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:15.875+0000 7f0e39d9b700 1 --2- 172.21.15.114:0/2630075686 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f0e3c103d40 0x7f0e3c199230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T10:26:15.876 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:15.875+0000 7f0e3ad9d700 1 -- 172.21.15.114:0/2630075686 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f0e3c103d40 msgr2=0x7f0e3c199230 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T10:26:15.876 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:15.875+0000 7f0e3ad9d700 1 --2- 172.21.15.114:0/2630075686 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f0e3c103d40 0x7f0e3c199230 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:26:15.876 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:15.875+0000 7f0e3ad9d700 1 -- 172.21.15.114:0/2630075686 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f0e3c1a0940 con 0x7f0e3c1046e0 2024-04-23T10:26:15.876 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:15.875+0000 7f0e3ad9d700 1 --2- 172.21.15.114:0/2630075686 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f0e3c1046e0 0x7f0e3c1a0270 secure :-1 s=READY pgs=351 cs=0 l=1 rev1=1 crypto rx=0x7f0e3400ced0 tx=0x7f0e3400cfb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T10:26:15.876 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:15.875+0000 7f0e2b7fe700 1 -- 172.21.15.114:0/2630075686 <== mon.0 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0e34021070 con 0x7f0e3c1046e0 2024-04-23T10:26:15.876 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:15.875+0000 7f0e40a7e700 1 -- 172.21.15.114:0/2630075686 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f0e3c1a0bd0 con 0x7f0e3c1046e0 2024-04-23T10:26:15.876 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:15.875+0000 7f0e2b7fe700 1 -- 172.21.15.114:0/2630075686 <== mon.0 v2:172.21.15.114:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f0e3401d900 con 0x7f0e3c1046e0 2024-04-23T10:26:15.876 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:15.875+0000 7f0e2b7fe700 1 -- 172.21.15.114:0/2630075686 <== mon.0 v2:172.21.15.114:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0e34005420 con 0x7f0e3c1046e0 2024-04-23T10:26:15.876 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:15.875+0000 7f0e40a7e700 1 -- 172.21.15.114:0/2630075686 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f0e3c1a1110 con 0x7f0e3c1046e0 2024-04-23T10:26:15.877 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:15.876+0000 7f0e40a7e700 1 -- 172.21.15.114:0/2630075686 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f0e3c069730 con 0x7f0e3c1046e0 2024-04-23T10:26:15.877 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:15.877+0000 7f0e2b7fe700 1 -- 172.21.15.114:0/2630075686 <== mon.0 v2:172.21.15.114:3300/0 4 ==== mgrmap(e 17) ==== 138698+0+0 (secure 0 0 0) 0x7f0e34024070 con 0x7f0e3c1046e0 2024-04-23T10:26:15.877 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:15.878+0000 7f0e2b7fe700 1 --2- 172.21.15.114:0/2630075686 >> [v2:172.21.15.114:6800/465333808,v1:172.21.15.114:6801/465333808] conn(0x7f0e240a3cd0 0x7f0e240a6190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T10:26:15.877 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:15.878+0000 7f0e3a59c700 1 --2- 172.21.15.114:0/2630075686 >> [v2:172.21.15.114:6800/465333808,v1:172.21.15.114:6801/465333808] conn(0x7f0e240a3cd0 0x7f0e240a6190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T10:26:15.877 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:15.878+0000 7f0e2b7fe700 1 -- 172.21.15.114:0/2630075686 <== mon.0 v2:172.21.15.114:3300/0 5 ==== osd_map(89..89 src has 1..89) ==== 8269+0+0 (secure 0 0 0) 0x7f0e340d1900 con 0x7f0e3c1046e0 2024-04-23T10:26:15.878 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:15.878+0000 7f0e3a59c700 1 --2- 172.21.15.114:0/2630075686 >> [v2:172.21.15.114:6800/465333808,v1:172.21.15.114:6801/465333808] conn(0x7f0e240a3cd0 0x7f0e240a6190 secure :-1 s=READY pgs=228 cs=0 l=1 rev1=1 crypto rx=0x7f0e30006fd0 tx=0x7f0e30008040 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T10:26:15.881 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:15.882+0000 7f0e2b7fe700 1 -- 172.21.15.114:0/2630075686 <== mon.0 v2:172.21.15.114:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f0e3409e1f0 con 0x7f0e3c1046e0 2024-04-23T10:26:16.075 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:16.075+0000 7f0e2b7fe700 1 -- 172.21.15.114:0/2630075686 <== mon.0 v2:172.21.15.114:3300/0 7 ==== mgrmap(e 18) ==== 138647+0+0 (secure 0 0 0) 0x7f0e3409a160 con 0x7f0e3c1046e0 2024-04-23T10:26:16.122 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:16.124+0000 7f0e40a7e700 1 -- 172.21.15.114:0/2630075686 --> [v2:172.21.15.114:6800/465333808,v1:172.21.15.114:6801/465333808] -- mgr_command(tid 0: {"prefix": "orch ls", "target": ["mon-mgr", ""]}) -- 0x7f0e3c062e80 con 0x7f0e240a3cd0 2024-04-23T10:26:16.128 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:16.130+0000 7f0e2b7fe700 1 -- 172.21.15.114:0/2630075686 <== mgr.14152 v2:172.21.15.114:6800/465333808 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+980 (secure 0 0 0) 0x7f0e3c062e80 con 0x7f0e240a3cd0 2024-04-23T10:26:16.129 INFO:teuthology.orchestra.run.smithi114.stdout:NAME PORTS RUNNING REFRESHED AGE PLACEMENT 2024-04-23T10:26:16.129 INFO:teuthology.orchestra.run.smithi114.stdout:mds.all 12/12 31s ago 2m smithi114=a;smithi114=d;smithi114=g;smithi114=j;smithi119=b;smithi119=e;smithi119=h;smithi119=k;smithi204=c;smithi204=f;smithi204=i;smithi204=l;count:12 2024-04-23T10:26:16.129 INFO:teuthology.orchestra.run.smithi114.stdout:mgr 3/3 31s ago 7m smithi114=x;smithi119=y;smithi204=z;count:3 2024-04-23T10:26:16.129 INFO:teuthology.orchestra.run.smithi114.stdout:mon 3/3 31s ago 7m smithi114:172.21.15.114=a;smithi119:172.21.15.119=b;smithi204:172.21.15.204=c;count:3 2024-04-23T10:26:16.129 INFO:teuthology.orchestra.run.smithi114.stdout:osd 12 31s ago - 2024-04-23T10:26:16.131 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:16.132+0000 7f0e40a7e700 1 -- 172.21.15.114:0/2630075686 >> [v2:172.21.15.114:6800/465333808,v1:172.21.15.114:6801/465333808] conn(0x7f0e240a3cd0 msgr2=0x7f0e240a6190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T10:26:16.131 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:16.132+0000 7f0e40a7e700 1 --2- 172.21.15.114:0/2630075686 >> [v2:172.21.15.114:6800/465333808,v1:172.21.15.114:6801/465333808] conn(0x7f0e240a3cd0 0x7f0e240a6190 secure :-1 s=READY pgs=228 cs=0 l=1 rev1=1 crypto rx=0x7f0e30006fd0 tx=0x7f0e30008040 comp rx=0 tx=0).stop 2024-04-23T10:26:16.131 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:16.132+0000 7f0e40a7e700 1 -- 172.21.15.114:0/2630075686 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f0e3c1046e0 msgr2=0x7f0e3c1a0270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T10:26:16.131 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:16.132+0000 7f0e40a7e700 1 --2- 172.21.15.114:0/2630075686 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f0e3c1046e0 0x7f0e3c1a0270 secure :-1 s=READY pgs=351 cs=0 l=1 rev1=1 crypto rx=0x7f0e3400ced0 tx=0x7f0e3400cfb0 comp rx=0 tx=0).stop 2024-04-23T10:26:16.131 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:16.132+0000 7f0e40a7e700 1 -- 172.21.15.114:0/2630075686 shutdown_connections 2024-04-23T10:26:16.131 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:16.132+0000 7f0e40a7e700 1 --2- 172.21.15.114:0/2630075686 >> [v2:172.21.15.114:6800/465333808,v1:172.21.15.114:6801/465333808] conn(0x7f0e240a3cd0 0x7f0e240a6190 unknown :-1 s=CLOSED pgs=228 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:26:16.131 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:16.132+0000 7f0e40a7e700 1 --2- 172.21.15.114:0/2630075686 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f0e3c1046e0 0x7f0e3c1a0270 unknown :-1 s=CLOSED pgs=351 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:26:16.131 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:16.132+0000 7f0e40a7e700 1 --2- 172.21.15.114:0/2630075686 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f0e3c100d60 0x7f0e3c198cf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:26:16.131 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:16.132+0000 7f0e40a7e700 1 --2- 172.21.15.114:0/2630075686 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f0e3c103d40 0x7f0e3c199230 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:26:16.131 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:16.132+0000 7f0e40a7e700 1 -- 172.21.15.114:0/2630075686 >> 172.21.15.114:0/2630075686 conn(0x7f0e3c0fc6a0 msgr2=0x7f0e3c107160 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T10:26:16.131 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:16.132+0000 7f0e40a7e700 1 -- 172.21.15.114:0/2630075686 shutdown_connections 2024-04-23T10:26:16.132 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:16.132+0000 7f0e40a7e700 1 -- 172.21.15.114:0/2630075686 wait complete. 2024-04-23T10:26:16.805 DEBUG:teuthology.orchestra.run.smithi114:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d69f13d827952bb07bb9b7973bca14f97d764d6f shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid aa240572-0159-11ef-bc93-c7b262605968 -- bash -c 'ceph orch host ls' 2024-04-23T10:26:17.045 INFO:teuthology.orchestra.run.smithi114.stderr:Inferring config /var/lib/ceph/aa240572-0159-11ef-bc93-c7b262605968/mon.a/config 2024-04-23T10:26:17.320 INFO:journalctl@ceph.mon.a.smithi114.stdout:Apr 23 10:26:17 smithi114 ceph-mon[29255]: pgmap v334: 97 pgs: 97 active+clean; 641 KiB data, 371 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-23T10:26:17.321 INFO:journalctl@ceph.mon.a.smithi114.stdout:Apr 23 10:26:17 smithi114 ceph-mon[29255]: mgrmap e18: x(active, since 8m), standbys: z, y 2024-04-23T10:26:17.328 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 23 10:26:17 smithi204 ceph-mon[32649]: pgmap v334: 97 pgs: 97 active+clean; 641 KiB data, 371 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-23T10:26:17.328 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 23 10:26:17 smithi204 ceph-mon[32649]: mgrmap e18: x(active, since 8m), standbys: z, y 2024-04-23T10:26:17.492 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 23 10:26:17 smithi119 ceph-mon[32608]: pgmap v334: 97 pgs: 97 active+clean; 641 KiB data, 371 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-23T10:26:17.492 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 23 10:26:17 smithi119 ceph-mon[32608]: mgrmap e18: x(active, since 8m), standbys: z, y 2024-04-23T10:26:18.168 INFO:journalctl@ceph.mon.a.smithi114.stdout:Apr 23 10:26:18 smithi114 ceph-mon[29255]: from='client.15126 -' entity='client.admin' cmd=[{"prefix": "orch ls", "target": ["mon-mgr", ""]}]: dispatch 2024-04-23T10:26:18.327 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 23 10:26:18 smithi204 ceph-mon[32649]: from='client.15126 -' entity='client.admin' cmd=[{"prefix": "orch ls", "target": ["mon-mgr", ""]}]: dispatch 2024-04-23T10:26:18.491 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 23 10:26:18 smithi119 ceph-mon[32608]: from='client.15126 -' entity='client.admin' cmd=[{"prefix": "orch ls", "target": ["mon-mgr", ""]}]: dispatch 2024-04-23T10:26:18.667 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:18.666+0000 7f9eb282b700 1 -- 172.21.15.114:0/1120422168 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f9eac1019b0 msgr2=0x7f9eac101dd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T10:26:18.667 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:18.666+0000 7f9eb282b700 1 --2- 172.21.15.114:0/1120422168 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f9eac1019b0 0x7f9eac101dd0 secure :-1 s=READY pgs=352 cs=0 l=1 rev1=1 crypto rx=0x7f9e940099c0 tx=0x7f9e94009cd0 comp rx=0 tx=0).stop 2024-04-23T10:26:18.667 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:18.666+0000 7f9eb282b700 1 -- 172.21.15.114:0/1120422168 shutdown_connections 2024-04-23T10:26:18.667 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:18.666+0000 7f9eb282b700 1 --2- 172.21.15.114:0/1120422168 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f9eac103550 0x7f9eac109e00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:26:18.667 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:18.666+0000 7f9eb282b700 1 --2- 172.21.15.114:0/1120422168 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f9eac1019b0 0x7f9eac101dd0 unknown :-1 s=CLOSED pgs=352 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:26:18.667 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:18.666+0000 7f9eb282b700 1 --2- 172.21.15.114:0/1120422168 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f9eac102bb0 0x7f9eac103010 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:26:18.668 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:18.666+0000 7f9eb282b700 1 -- 172.21.15.114:0/1120422168 >> 172.21.15.114:0/1120422168 conn(0x7f9eac0fcf70 msgr2=0x7f9eac0ff390 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T10:26:18.668 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:18.666+0000 7f9eb282b700 1 -- 172.21.15.114:0/1120422168 shutdown_connections 2024-04-23T10:26:18.668 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:18.666+0000 7f9eb282b700 1 -- 172.21.15.114:0/1120422168 wait complete. 2024-04-23T10:26:18.668 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:18.667+0000 7f9eb282b700 1 Processor -- start 2024-04-23T10:26:18.668 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:18.667+0000 7f9eb282b700 1 -- start start 2024-04-23T10:26:18.668 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:18.667+0000 7f9eb282b700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f9eac1019b0 0x7f9eac198d30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T10:26:18.668 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:18.667+0000 7f9eb282b700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f9eac102bb0 0x7f9eac199270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T10:26:18.668 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:18.667+0000 7f9eb282b700 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f9eac103550 0x7f9eac1a02b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T10:26:18.668 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:18.667+0000 7f9eb282b700 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x7f9eac10e5c0 con 0x7f9eac102bb0 2024-04-23T10:26:18.668 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:18.667+0000 7f9eb282b700 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 -- 0x7f9eac10e440 con 0x7f9eac1019b0 2024-04-23T10:26:18.668 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:18.667+0000 7f9eb282b700 1 -- --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_getmap magic: 0 -- 0x7f9eac10e740 con 0x7f9eac103550 2024-04-23T10:26:18.668 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:18.667+0000 7f9eab7fe700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f9eac102bb0 0x7f9eac199270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T10:26:18.668 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:18.667+0000 7f9eab7fe700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f9eac102bb0 0x7f9eac199270 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.114:58616/0 (socket says 172.21.15.114:58616) 2024-04-23T10:26:18.668 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:18.667+0000 7f9eab7fe700 1 -- 172.21.15.114:0/3200807905 learned_addr learned my addr 172.21.15.114:0/3200807905 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-04-23T10:26:18.668 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:18.667+0000 7f9eabfff700 1 --2- 172.21.15.114:0/3200807905 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f9eac1019b0 0x7f9eac198d30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T10:26:18.669 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:18.667+0000 7f9eb0dc8700 1 --2- 172.21.15.114:0/3200807905 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f9eac103550 0x7f9eac1a02b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T10:26:18.669 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:18.668+0000 7f9eab7fe700 1 -- 172.21.15.114:0/3200807905 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f9eac103550 msgr2=0x7f9eac1a02b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T10:26:18.669 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:18.668+0000 7f9eab7fe700 1 --2- 172.21.15.114:0/3200807905 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f9eac103550 0x7f9eac1a02b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:26:18.669 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:18.668+0000 7f9eab7fe700 1 -- 172.21.15.114:0/3200807905 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f9eac1019b0 msgr2=0x7f9eac198d30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T10:26:18.669 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:18.668+0000 7f9eab7fe700 1 --2- 172.21.15.114:0/3200807905 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f9eac1019b0 0x7f9eac198d30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:26:18.669 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:18.668+0000 7f9eab7fe700 1 -- 172.21.15.114:0/3200807905 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f9eac1a0980 con 0x7f9eac102bb0 2024-04-23T10:26:18.669 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:18.668+0000 7f9eb0dc8700 1 --2- 172.21.15.114:0/3200807905 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f9eac103550 0x7f9eac1a02b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-23T10:26:18.669 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:18.668+0000 7f9eab7fe700 1 --2- 172.21.15.114:0/3200807905 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f9eac102bb0 0x7f9eac199270 secure :-1 s=READY pgs=353 cs=0 l=1 rev1=1 crypto rx=0x7f9e9c00d980 tx=0x7f9e9c00dc90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T10:26:18.669 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:18.668+0000 7f9ea97fa700 1 -- 172.21.15.114:0/3200807905 <== mon.0 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9e9c017070 con 0x7f9eac102bb0 2024-04-23T10:26:18.669 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:18.668+0000 7f9ea97fa700 1 -- 172.21.15.114:0/3200807905 <== mon.0 v2:172.21.15.114:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f9e9c005020 con 0x7f9eac102bb0 2024-04-23T10:26:18.669 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:18.668+0000 7f9eb282b700 1 -- 172.21.15.114:0/3200807905 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f9eac1a0c70 con 0x7f9eac102bb0 2024-04-23T10:26:18.669 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:18.668+0000 7f9ea97fa700 1 -- 172.21.15.114:0/3200807905 <== mon.0 v2:172.21.15.114:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9e9c009b50 con 0x7f9eac102bb0 2024-04-23T10:26:18.669 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:18.668+0000 7f9eb282b700 1 -- 172.21.15.114:0/3200807905 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f9eac1a1130 con 0x7f9eac102bb0 2024-04-23T10:26:18.670 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:18.669+0000 7f9eb282b700 1 -- 172.21.15.114:0/3200807905 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f9eac069730 con 0x7f9eac102bb0 2024-04-23T10:26:18.670 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:18.670+0000 7f9ea97fa700 1 -- 172.21.15.114:0/3200807905 <== mon.0 v2:172.21.15.114:3300/0 4 ==== mgrmap(e 18) ==== 138647+0+0 (secure 0 0 0) 0x7f9e9c0051d0 con 0x7f9eac102bb0 2024-04-23T10:26:18.670 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:18.670+0000 7f9ea97fa700 1 --2- 172.21.15.114:0/3200807905 >> [v2:172.21.15.114:6800/465333808,v1:172.21.15.114:6801/465333808] conn(0x7f9e980a3c40 0x7f9e980a6100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T10:26:18.670 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:18.671+0000 7f9eabfff700 1 --2- 172.21.15.114:0/3200807905 >> [v2:172.21.15.114:6800/465333808,v1:172.21.15.114:6801/465333808] conn(0x7f9e980a3c40 0x7f9e980a6100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T10:26:18.670 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:18.671+0000 7f9ea97fa700 1 -- 172.21.15.114:0/3200807905 <== mon.0 v2:172.21.15.114:3300/0 5 ==== osd_map(89..89 src has 1..89) ==== 8269+0+0 (secure 0 0 0) 0x7f9e9c0d06d0 con 0x7f9eac102bb0 2024-04-23T10:26:18.671 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:18.671+0000 7f9eabfff700 1 --2- 172.21.15.114:0/3200807905 >> [v2:172.21.15.114:6800/465333808,v1:172.21.15.114:6801/465333808] conn(0x7f9e980a3c40 0x7f9e980a6100 secure :-1 s=READY pgs=229 cs=0 l=1 rev1=1 crypto rx=0x7f9e94006010 tx=0x7f9e94003700 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T10:26:18.673 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:18.674+0000 7f9ea97fa700 1 -- 172.21.15.114:0/3200807905 <== mon.0 v2:172.21.15.114:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f9e9c020020 con 0x7f9eac102bb0 2024-04-23T10:26:18.880 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:18.881+0000 7f9eb282b700 1 -- 172.21.15.114:0/3200807905 --> [v2:172.21.15.114:6800/465333808,v1:172.21.15.114:6801/465333808] -- mgr_command(tid 0: {"prefix": "orch host ls", "target": ["mon-mgr", ""]}) -- 0x7f9eac062e80 con 0x7f9e980a3c40 2024-04-23T10:26:18.881 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:18.882+0000 7f9ea97fa700 1 -- 172.21.15.114:0/3200807905 <== mgr.14152 v2:172.21.15.114:6800/465333808 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+191 (secure 0 0 0) 0x7f9eac062e80 con 0x7f9e980a3c40 2024-04-23T10:26:18.881 INFO:teuthology.orchestra.run.smithi114.stdout:HOST ADDR LABELS STATUS 2024-04-23T10:26:18.881 INFO:teuthology.orchestra.run.smithi114.stdout:smithi114 172.21.15.114 2024-04-23T10:26:18.882 INFO:teuthology.orchestra.run.smithi114.stdout:smithi119 172.21.15.119 2024-04-23T10:26:18.882 INFO:teuthology.orchestra.run.smithi114.stdout:smithi204 172.21.15.204 2024-04-23T10:26:18.882 INFO:teuthology.orchestra.run.smithi114.stdout:3 hosts in cluster 2024-04-23T10:26:18.884 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:18.885+0000 7f9eb282b700 1 -- 172.21.15.114:0/3200807905 >> [v2:172.21.15.114:6800/465333808,v1:172.21.15.114:6801/465333808] conn(0x7f9e980a3c40 msgr2=0x7f9e980a6100 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T10:26:18.884 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:18.885+0000 7f9eb282b700 1 --2- 172.21.15.114:0/3200807905 >> [v2:172.21.15.114:6800/465333808,v1:172.21.15.114:6801/465333808] conn(0x7f9e980a3c40 0x7f9e980a6100 secure :-1 s=READY pgs=229 cs=0 l=1 rev1=1 crypto rx=0x7f9e94006010 tx=0x7f9e94003700 comp rx=0 tx=0).stop 2024-04-23T10:26:18.884 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:18.885+0000 7f9eb282b700 1 -- 172.21.15.114:0/3200807905 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f9eac102bb0 msgr2=0x7f9eac199270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T10:26:18.884 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:18.885+0000 7f9eb282b700 1 --2- 172.21.15.114:0/3200807905 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f9eac102bb0 0x7f9eac199270 secure :-1 s=READY pgs=353 cs=0 l=1 rev1=1 crypto rx=0x7f9e9c00d980 tx=0x7f9e9c00dc90 comp rx=0 tx=0).stop 2024-04-23T10:26:18.884 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:18.885+0000 7f9eb282b700 1 -- 172.21.15.114:0/3200807905 shutdown_connections 2024-04-23T10:26:18.885 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:18.885+0000 7f9eb282b700 1 --2- 172.21.15.114:0/3200807905 >> [v2:172.21.15.114:6800/465333808,v1:172.21.15.114:6801/465333808] conn(0x7f9e980a3c40 0x7f9e980a6100 unknown :-1 s=CLOSED pgs=229 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:26:18.885 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:18.885+0000 7f9eb282b700 1 --2- 172.21.15.114:0/3200807905 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f9eac1019b0 0x7f9eac198d30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:26:18.885 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:18.885+0000 7f9eb282b700 1 --2- 172.21.15.114:0/3200807905 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f9eac103550 0x7f9eac1a02b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:26:18.885 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:18.885+0000 7f9eb282b700 1 --2- 172.21.15.114:0/3200807905 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f9eac102bb0 0x7f9eac199270 unknown :-1 s=CLOSED pgs=353 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:26:18.885 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:18.885+0000 7f9eb282b700 1 -- 172.21.15.114:0/3200807905 >> 172.21.15.114:0/3200807905 conn(0x7f9eac0fcf70 msgr2=0x7f9eac106070 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T10:26:18.885 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:18.885+0000 7f9eb282b700 1 -- 172.21.15.114:0/3200807905 shutdown_connections 2024-04-23T10:26:18.885 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:18.885+0000 7f9eb282b700 1 -- 172.21.15.114:0/3200807905 wait complete. 2024-04-23T10:26:19.254 INFO:journalctl@ceph.mon.a.smithi114.stdout:Apr 23 10:26:19 smithi114 ceph-mon[29255]: pgmap v335: 97 pgs: 97 active+clean; 641 KiB data, 375 MiB used, 1.0 TiB / 1.0 TiB avail; 3.5 KiB/s rd, 426 B/s wr, 7 op/s 2024-04-23T10:26:19.328 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 23 10:26:19 smithi204 ceph-mon[32649]: pgmap v335: 97 pgs: 97 active+clean; 641 KiB data, 375 MiB used, 1.0 TiB / 1.0 TiB avail; 3.5 KiB/s rd, 426 B/s wr, 7 op/s 2024-04-23T10:26:19.492 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 23 10:26:19 smithi119 ceph-mon[32608]: pgmap v335: 97 pgs: 97 active+clean; 641 KiB data, 375 MiB used, 1.0 TiB / 1.0 TiB avail; 3.5 KiB/s rd, 426 B/s wr, 7 op/s 2024-04-23T10:26:19.493 DEBUG:teuthology.orchestra.run.smithi114:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d69f13d827952bb07bb9b7973bca14f97d764d6f shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid aa240572-0159-11ef-bc93-c7b262605968 -- bash -c 'ceph orch device ls' 2024-04-23T10:26:19.731 INFO:teuthology.orchestra.run.smithi114.stderr:Inferring config /var/lib/ceph/aa240572-0159-11ef-bc93-c7b262605968/mon.a/config 2024-04-23T10:26:20.171 INFO:journalctl@ceph.mon.a.smithi114.stdout:Apr 23 10:26:20 smithi114 ceph-mon[29255]: from='client.15132 -' entity='client.admin' cmd=[{"prefix": "orch host ls", "target": ["mon-mgr", ""]}]: dispatch 2024-04-23T10:26:20.491 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 23 10:26:20 smithi119 ceph-mon[32608]: from='client.15132 -' entity='client.admin' cmd=[{"prefix": "orch host ls", "target": ["mon-mgr", ""]}]: dispatch 2024-04-23T10:26:20.578 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 23 10:26:20 smithi204 ceph-mon[32649]: from='client.15132 -' entity='client.admin' cmd=[{"prefix": "orch host ls", "target": ["mon-mgr", ""]}]: dispatch 2024-04-23T10:26:21.232 INFO:journalctl@ceph.mon.a.smithi114.stdout:Apr 23 10:26:21 smithi114 ceph-mon[29255]: pgmap v336: 97 pgs: 97 active+clean; 641 KiB data, 375 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 597 B/s wr, 6 op/s 2024-04-23T10:26:21.481 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:21.482+0000 7fc46b34a700 1 -- 172.21.15.114:0/1822469563 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fc4641019b0 msgr2=0x7fc464101dd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T10:26:21.481 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:21.482+0000 7fc46b34a700 1 --2- 172.21.15.114:0/1822469563 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fc4641019b0 0x7fc464101dd0 secure :-1 s=READY pgs=354 cs=0 l=1 rev1=1 crypto rx=0x7fc46000b0d0 tx=0x7fc46000b3e0 comp rx=0 tx=0).stop 2024-04-23T10:26:21.482 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:21.482+0000 7fc46b34a700 1 -- 172.21.15.114:0/1822469563 shutdown_connections 2024-04-23T10:26:21.482 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:21.482+0000 7fc46b34a700 1 --2- 172.21.15.114:0/1822469563 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fc464103550 0x7fc464109e00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:26:21.482 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:21.482+0000 7fc46b34a700 1 --2- 172.21.15.114:0/1822469563 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fc4641019b0 0x7fc464101dd0 unknown :-1 s=CLOSED pgs=354 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:26:21.482 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:21.482+0000 7fc46b34a700 1 --2- 172.21.15.114:0/1822469563 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fc464102bb0 0x7fc464103010 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:26:21.482 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:21.482+0000 7fc46b34a700 1 -- 172.21.15.114:0/1822469563 >> 172.21.15.114:0/1822469563 conn(0x7fc4640fcf70 msgr2=0x7fc4640ff390 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T10:26:21.482 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:21.482+0000 7fc46b34a700 1 -- 172.21.15.114:0/1822469563 shutdown_connections 2024-04-23T10:26:21.482 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:21.482+0000 7fc46b34a700 1 -- 172.21.15.114:0/1822469563 wait complete. 2024-04-23T10:26:21.484 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:21.483+0000 7fc46b34a700 1 Processor -- start 2024-04-23T10:26:21.484 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:21.483+0000 7fc46b34a700 1 -- start start 2024-04-23T10:26:21.484 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:21.483+0000 7fc46b34a700 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fc4641019b0 0x7fc464198d40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T10:26:21.484 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:21.483+0000 7fc46b34a700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fc464102bb0 0x7fc464199280 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T10:26:21.484 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:21.483+0000 7fc46b34a700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fc464103550 0x7fc4641a0270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T10:26:21.484 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:21.483+0000 7fc46b34a700 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x7fc46410d4a0 con 0x7fc464102bb0 2024-04-23T10:26:21.484 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:21.483+0000 7fc46b34a700 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 -- 0x7fc46410d320 con 0x7fc464103550 2024-04-23T10:26:21.484 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:21.483+0000 7fc46b34a700 1 -- --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_getmap magic: 0 -- 0x7fc46410d620 con 0x7fc4641019b0 2024-04-23T10:26:21.484 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:21.483+0000 7fc4688e5700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fc464102bb0 0x7fc464199280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T10:26:21.485 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:21.483+0000 7fc4688e5700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fc464102bb0 0x7fc464199280 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.114:58646/0 (socket says 172.21.15.114:58646) 2024-04-23T10:26:21.485 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:21.483+0000 7fc4688e5700 1 -- 172.21.15.114:0/1686211012 learned_addr learned my addr 172.21.15.114:0/1686211012 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-04-23T10:26:21.485 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:21.483+0000 7fc4690e6700 1 --2- 172.21.15.114:0/1686211012 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fc4641019b0 0x7fc464198d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T10:26:21.485 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:21.483+0000 7fc4698e7700 1 --2- 172.21.15.114:0/1686211012 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fc464103550 0x7fc4641a0270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T10:26:21.485 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:21.483+0000 7fc4688e5700 1 -- 172.21.15.114:0/1686211012 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fc4641019b0 msgr2=0x7fc464198d40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T10:26:21.485 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:21.483+0000 7fc4688e5700 1 --2- 172.21.15.114:0/1686211012 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fc4641019b0 0x7fc464198d40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:26:21.485 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:21.483+0000 7fc4688e5700 1 -- 172.21.15.114:0/1686211012 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fc464103550 msgr2=0x7fc4641a0270 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T10:26:21.485 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:21.484+0000 7fc4688e5700 1 --2- 172.21.15.114:0/1686211012 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fc464103550 0x7fc4641a0270 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:26:21.485 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:21.484+0000 7fc4688e5700 1 -- 172.21.15.114:0/1686211012 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc4641a0940 con 0x7fc464102bb0 2024-04-23T10:26:21.485 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:21.484+0000 7fc4698e7700 1 --2- 172.21.15.114:0/1686211012 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fc464103550 0x7fc4641a0270 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-23T10:26:21.485 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:21.484+0000 7fc4688e5700 1 --2- 172.21.15.114:0/1686211012 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fc464102bb0 0x7fc464199280 secure :-1 s=READY pgs=355 cs=0 l=1 rev1=1 crypto rx=0x7fc45400b530 tx=0x7fc45400b8f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T10:26:21.485 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:21.484+0000 7fc45a7fc700 1 -- 172.21.15.114:0/1686211012 <== mon.0 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc454015020 con 0x7fc464102bb0 2024-04-23T10:26:21.485 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:21.484+0000 7fc45a7fc700 1 -- 172.21.15.114:0/1686211012 <== mon.0 v2:172.21.15.114:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fc454011ce0 con 0x7fc464102bb0 2024-04-23T10:26:21.485 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:21.484+0000 7fc45a7fc700 1 -- 172.21.15.114:0/1686211012 <== mon.0 v2:172.21.15.114:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc454010d20 con 0x7fc464102bb0 2024-04-23T10:26:21.485 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:21.484+0000 7fc46b34a700 1 -- 172.21.15.114:0/1686211012 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fc4641a0c30 con 0x7fc464102bb0 2024-04-23T10:26:21.485 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:21.484+0000 7fc46b34a700 1 -- 172.21.15.114:0/1686211012 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fc4641a1120 con 0x7fc464102bb0 2024-04-23T10:26:21.485 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:21.485+0000 7fc46b34a700 1 -- 172.21.15.114:0/1686211012 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fc464069730 con 0x7fc464102bb0 2024-04-23T10:26:21.486 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:21.487+0000 7fc45a7fc700 1 -- 172.21.15.114:0/1686211012 <== mon.0 v2:172.21.15.114:3300/0 4 ==== mgrmap(e 18) ==== 138647+0+0 (secure 0 0 0) 0x7fc454018020 con 0x7fc464102bb0 2024-04-23T10:26:21.486 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:21.487+0000 7fc45a7fc700 1 --2- 172.21.15.114:0/1686211012 >> [v2:172.21.15.114:6800/465333808,v1:172.21.15.114:6801/465333808] conn(0x7fc4500a3c40 0x7fc4500a6100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T10:26:21.486 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:21.487+0000 7fc4690e6700 1 --2- 172.21.15.114:0/1686211012 >> [v2:172.21.15.114:6800/465333808,v1:172.21.15.114:6801/465333808] conn(0x7fc4500a3c40 0x7fc4500a6100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T10:26:21.486 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:21.488+0000 7fc45a7fc700 1 -- 172.21.15.114:0/1686211012 <== mon.0 v2:172.21.15.114:3300/0 5 ==== osd_map(89..89 src has 1..89) ==== 8269+0+0 (secure 0 0 0) 0x7fc4540d0220 con 0x7fc464102bb0 2024-04-23T10:26:21.487 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:21.488+0000 7fc4690e6700 1 --2- 172.21.15.114:0/1686211012 >> [v2:172.21.15.114:6800/465333808,v1:172.21.15.114:6801/465333808] conn(0x7fc4500a3c40 0x7fc4500a6100 secure :-1 s=READY pgs=230 cs=0 l=1 rev1=1 crypto rx=0x7fc46419a1d0 tx=0x7fc46000b040 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T10:26:21.490 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:21.491+0000 7fc45a7fc700 1 -- 172.21.15.114:0/1686211012 <== mon.0 v2:172.21.15.114:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7fc45401f020 con 0x7fc464102bb0 2024-04-23T10:26:21.492 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 23 10:26:21 smithi119 ceph-mon[32608]: pgmap v336: 97 pgs: 97 active+clean; 641 KiB data, 375 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 597 B/s wr, 6 op/s 2024-04-23T10:26:21.577 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 23 10:26:21 smithi204 ceph-mon[32649]: pgmap v336: 97 pgs: 97 active+clean; 641 KiB data, 375 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 597 B/s wr, 6 op/s 2024-04-23T10:26:21.697 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:21.699+0000 7fc46b34a700 1 -- 172.21.15.114:0/1686211012 --> [v2:172.21.15.114:6800/465333808,v1:172.21.15.114:6801/465333808] -- mgr_command(tid 0: {"prefix": "orch device ls", "target": ["mon-mgr", ""]}) -- 0x7fc46406b8a0 con 0x7fc4500a3c40 2024-04-23T10:26:21.699 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:21.700+0000 7fc45a7fc700 1 -- 172.21.15.114:0/1686211012 <== mgr.14152 v2:172.21.15.114:6800/465333808 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+532 (secure 0 0 0) 0x7fc46406b8a0 con 0x7fc4500a3c40 2024-04-23T10:26:21.699 INFO:teuthology.orchestra.run.smithi114.stdout:HOST PATH TYPE DEVICE ID SIZE AVAILABLE REFRESHED REJECT REASONS 2024-04-23T10:26:21.700 INFO:teuthology.orchestra.run.smithi114.stdout:smithi114 /dev/nvme0n1 ssd INTEL_SSDPEDMD400G4_PHFT620400WB400BGN 372G No 3m ago Has a FileSystem, LVM detected 2024-04-23T10:26:21.700 INFO:teuthology.orchestra.run.smithi114.stdout:smithi119 /dev/nvme0n1 ssd INTEL_SSDPEDMD400G4_PHFT620400Y1400BGN 372G No 2m ago Has a FileSystem, LVM detected 2024-04-23T10:26:21.700 INFO:teuthology.orchestra.run.smithi114.stdout:smithi204 /dev/nvme0n1 ssd INTEL_SSDPEDMD400G4_CVFT623300GD400BGN 372G No 2m ago Has a FileSystem, LVM detected 2024-04-23T10:26:21.702 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:21.702+0000 7fc46b34a700 1 -- 172.21.15.114:0/1686211012 >> [v2:172.21.15.114:6800/465333808,v1:172.21.15.114:6801/465333808] conn(0x7fc4500a3c40 msgr2=0x7fc4500a6100 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T10:26:21.702 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:21.702+0000 7fc46b34a700 1 --2- 172.21.15.114:0/1686211012 >> [v2:172.21.15.114:6800/465333808,v1:172.21.15.114:6801/465333808] conn(0x7fc4500a3c40 0x7fc4500a6100 secure :-1 s=READY pgs=230 cs=0 l=1 rev1=1 crypto rx=0x7fc46419a1d0 tx=0x7fc46000b040 comp rx=0 tx=0).stop 2024-04-23T10:26:21.702 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:21.702+0000 7fc46b34a700 1 -- 172.21.15.114:0/1686211012 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fc464102bb0 msgr2=0x7fc464199280 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T10:26:21.702 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:21.702+0000 7fc46b34a700 1 --2- 172.21.15.114:0/1686211012 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fc464102bb0 0x7fc464199280 secure :-1 s=READY pgs=355 cs=0 l=1 rev1=1 crypto rx=0x7fc45400b530 tx=0x7fc45400b8f0 comp rx=0 tx=0).stop 2024-04-23T10:26:21.702 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:21.702+0000 7fc46b34a700 1 -- 172.21.15.114:0/1686211012 shutdown_connections 2024-04-23T10:26:21.702 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:21.702+0000 7fc46b34a700 1 --2- 172.21.15.114:0/1686211012 >> [v2:172.21.15.114:6800/465333808,v1:172.21.15.114:6801/465333808] conn(0x7fc4500a3c40 0x7fc4500a6100 unknown :-1 s=CLOSED pgs=230 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:26:21.702 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:21.702+0000 7fc46b34a700 1 --2- 172.21.15.114:0/1686211012 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fc464103550 0x7fc4641a0270 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:26:21.702 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:21.702+0000 7fc46b34a700 1 --2- 172.21.15.114:0/1686211012 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fc4641019b0 0x7fc464198d40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:26:21.702 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:21.702+0000 7fc46b34a700 1 --2- 172.21.15.114:0/1686211012 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fc464102bb0 0x7fc464199280 unknown :-1 s=CLOSED pgs=355 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:26:21.702 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:21.702+0000 7fc46b34a700 1 -- 172.21.15.114:0/1686211012 >> 172.21.15.114:0/1686211012 conn(0x7fc4640fcf70 msgr2=0x7fc46406b190 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T10:26:21.702 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:21.703+0000 7fc46b34a700 1 -- 172.21.15.114:0/1686211012 shutdown_connections 2024-04-23T10:26:21.703 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:21.703+0000 7fc46b34a700 1 -- 172.21.15.114:0/1686211012 wait complete. 2024-04-23T10:26:22.510 INFO:teuthology.run_tasks:Running task cephadm.shell... 2024-04-23T10:26:22.519 INFO:tasks.cephadm:Running commands on role mon.a host ubuntu@smithi114.front.sepia.ceph.com 2024-04-23T10:26:22.519 DEBUG:teuthology.orchestra.run.smithi114:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d69f13d827952bb07bb9b7973bca14f97d764d6f shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid aa240572-0159-11ef-bc93-c7b262605968 -- bash -c 'ceph fs dump' 2024-04-23T10:26:22.765 INFO:teuthology.orchestra.run.smithi114.stderr:Inferring config /var/lib/ceph/aa240572-0159-11ef-bc93-c7b262605968/mon.a/config 2024-04-23T10:26:23.204 INFO:journalctl@ceph.mon.a.smithi114.stdout:Apr 23 10:26:23 smithi114 ceph-mon[29255]: from='client.15138 -' entity='client.admin' cmd=[{"prefix": "orch device ls", "target": ["mon-mgr", ""]}]: dispatch 2024-04-23T10:26:23.204 INFO:journalctl@ceph.mon.a.smithi114.stdout:Apr 23 10:26:23 smithi114 ceph-mon[29255]: pgmap v337: 97 pgs: 97 active+clean; 641 KiB data, 375 MiB used, 1.0 TiB / 1.0 TiB avail; 2.7 KiB/s rd, 597 B/s wr, 5 op/s 2024-04-23T10:26:23.492 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 23 10:26:23 smithi119 ceph-mon[32608]: from='client.15138 -' entity='client.admin' cmd=[{"prefix": "orch device ls", "target": ["mon-mgr", ""]}]: dispatch 2024-04-23T10:26:23.492 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 23 10:26:23 smithi119 ceph-mon[32608]: pgmap v337: 97 pgs: 97 active+clean; 641 KiB data, 375 MiB used, 1.0 TiB / 1.0 TiB avail; 2.7 KiB/s rd, 597 B/s wr, 5 op/s 2024-04-23T10:26:23.577 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 23 10:26:23 smithi204 ceph-mon[32649]: from='client.15138 -' entity='client.admin' cmd=[{"prefix": "orch device ls", "target": ["mon-mgr", ""]}]: dispatch 2024-04-23T10:26:23.578 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 23 10:26:23 smithi204 ceph-mon[32649]: pgmap v337: 97 pgs: 97 active+clean; 641 KiB data, 375 MiB used, 1.0 TiB / 1.0 TiB avail; 2.7 KiB/s rd, 597 B/s wr, 5 op/s 2024-04-23T10:26:24.413 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:24.412+0000 7fd4ca9dc700 1 -- 172.21.15.114:0/1139931887 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fd4c4104660 msgr2=0x7fd4c410af10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T10:26:24.413 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:24.412+0000 7fd4ca9dc700 1 --2- 172.21.15.114:0/1139931887 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fd4c4104660 0x7fd4c410af10 secure :-1 s=READY pgs=356 cs=0 l=1 rev1=1 crypto rx=0x7fd4bc00b0d0 tx=0x7fd4bc00b3e0 comp rx=0 tx=0).stop 2024-04-23T10:26:24.413 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:24.412+0000 7fd4ca9dc700 1 -- 172.21.15.114:0/1139931887 shutdown_connections 2024-04-23T10:26:24.413 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:24.412+0000 7fd4ca9dc700 1 --2- 172.21.15.114:0/1139931887 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fd4c4102ac0 0x7fd4c4102ee0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:26:24.413 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:24.412+0000 7fd4ca9dc700 1 --2- 172.21.15.114:0/1139931887 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fd4c4104660 0x7fd4c410af10 unknown :-1 s=CLOSED pgs=356 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:26:24.414 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:24.412+0000 7fd4ca9dc700 1 --2- 172.21.15.114:0/1139931887 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fd4c4103cc0 0x7fd4c4104120 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:26:24.414 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:24.412+0000 7fd4ca9dc700 1 -- 172.21.15.114:0/1139931887 >> 172.21.15.114:0/1139931887 conn(0x7fd4c40fe060 msgr2=0x7fd4c41004a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T10:26:24.414 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:24.412+0000 7fd4ca9dc700 1 -- 172.21.15.114:0/1139931887 shutdown_connections 2024-04-23T10:26:24.414 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:24.412+0000 7fd4ca9dc700 1 -- 172.21.15.114:0/1139931887 wait complete. 2024-04-23T10:26:24.414 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:24.413+0000 7fd4ca9dc700 1 Processor -- start 2024-04-23T10:26:24.414 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:24.413+0000 7fd4ca9dc700 1 -- start start 2024-04-23T10:26:24.414 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:24.413+0000 7fd4ca9dc700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fd4c4102ac0 0x7fd4c419ad40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T10:26:24.414 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:24.413+0000 7fd4ca9dc700 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fd4c4103cc0 0x7fd4c419b280 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T10:26:24.414 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:24.413+0000 7fd4ca9dc700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fd4c4104660 0x7fd4c41a22c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T10:26:24.414 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:24.413+0000 7fd4ca9dc700 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x7fd4c410e5b0 con 0x7fd4c4102ac0 2024-04-23T10:26:24.414 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:24.413+0000 7fd4ca9dc700 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 -- 0x7fd4c410e430 con 0x7fd4c4104660 2024-04-23T10:26:24.414 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:24.413+0000 7fd4ca9dc700 1 -- --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_getmap magic: 0 -- 0x7fd4c410e730 con 0x7fd4c4103cc0 2024-04-23T10:26:24.415 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:24.413+0000 7fd4c3fff700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fd4c4102ac0 0x7fd4c419ad40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T10:26:24.415 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:24.413+0000 7fd4c3fff700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fd4c4102ac0 0x7fd4c419ad40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.114:58674/0 (socket says 172.21.15.114:58674) 2024-04-23T10:26:24.415 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:24.413+0000 7fd4c3fff700 1 -- 172.21.15.114:0/3152054937 learned_addr learned my addr 172.21.15.114:0/3152054937 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-04-23T10:26:24.415 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:24.413+0000 7fd4c8f79700 1 --2- 172.21.15.114:0/3152054937 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fd4c4104660 0x7fd4c41a22c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T10:26:24.415 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:24.413+0000 7fd4bbfff700 1 --2- 172.21.15.114:0/3152054937 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fd4c4103cc0 0x7fd4c419b280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T10:26:24.415 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:24.413+0000 7fd4c3fff700 1 -- 172.21.15.114:0/3152054937 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fd4c4103cc0 msgr2=0x7fd4c419b280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T10:26:24.415 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:24.413+0000 7fd4c3fff700 1 --2- 172.21.15.114:0/3152054937 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fd4c4103cc0 0x7fd4c419b280 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:26:24.415 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:24.413+0000 7fd4c3fff700 1 -- 172.21.15.114:0/3152054937 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fd4c4104660 msgr2=0x7fd4c41a22c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T10:26:24.415 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:24.413+0000 7fd4c3fff700 1 --2- 172.21.15.114:0/3152054937 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fd4c4104660 0x7fd4c41a22c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:26:24.415 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:24.413+0000 7fd4c3fff700 1 -- 172.21.15.114:0/3152054937 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd4c41a2990 con 0x7fd4c4102ac0 2024-04-23T10:26:24.415 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:24.413+0000 7fd4c8f79700 1 --2- 172.21.15.114:0/3152054937 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fd4c4104660 0x7fd4c41a22c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-23T10:26:24.415 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:24.414+0000 7fd4c3fff700 1 --2- 172.21.15.114:0/3152054937 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fd4c4102ac0 0x7fd4c419ad40 secure :-1 s=READY pgs=357 cs=0 l=1 rev1=1 crypto rx=0x7fd4b400c990 tx=0x7fd4b400cd50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T10:26:24.415 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:24.414+0000 7fd4c1ffb700 1 -- 172.21.15.114:0/3152054937 <== mon.0 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd4b4007b00 con 0x7fd4c4102ac0 2024-04-23T10:26:24.415 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:24.414+0000 7fd4ca9dc700 1 -- 172.21.15.114:0/3152054937 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fd4c41a2c80 con 0x7fd4c4102ac0 2024-04-23T10:26:24.416 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:24.414+0000 7fd4c1ffb700 1 -- 172.21.15.114:0/3152054937 <== mon.0 v2:172.21.15.114:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fd4b4007ca0 con 0x7fd4c4102ac0 2024-04-23T10:26:24.416 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:24.414+0000 7fd4c1ffb700 1 -- 172.21.15.114:0/3152054937 <== mon.0 v2:172.21.15.114:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd4b4018660 con 0x7fd4c4102ac0 2024-04-23T10:26:24.416 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:24.414+0000 7fd4ca9dc700 1 -- 172.21.15.114:0/3152054937 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fd4c41a31c0 con 0x7fd4c4102ac0 2024-04-23T10:26:24.417 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:24.415+0000 7fd4ca9dc700 1 -- 172.21.15.114:0/3152054937 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fd4c4069730 con 0x7fd4c4102ac0 2024-04-23T10:26:24.417 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:24.416+0000 7fd4c1ffb700 1 -- 172.21.15.114:0/3152054937 <== mon.0 v2:172.21.15.114:3300/0 4 ==== mgrmap(e 18) ==== 138647+0+0 (secure 0 0 0) 0x7fd4b400f450 con 0x7fd4c4102ac0 2024-04-23T10:26:24.417 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:24.416+0000 7fd4c1ffb700 1 --2- 172.21.15.114:0/3152054937 >> [v2:172.21.15.114:6800/465333808,v1:172.21.15.114:6801/465333808] conn(0x7fd4b00a3c40 0x7fd4b00a6100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T10:26:24.417 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:24.416+0000 7fd4bbfff700 1 --2- 172.21.15.114:0/3152054937 >> [v2:172.21.15.114:6800/465333808,v1:172.21.15.114:6801/465333808] conn(0x7fd4b00a3c40 0x7fd4b00a6100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T10:26:24.417 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:24.417+0000 7fd4c1ffb700 1 -- 172.21.15.114:0/3152054937 <== mon.0 v2:172.21.15.114:3300/0 5 ==== osd_map(89..89 src has 1..89) ==== 8269+0+0 (secure 0 0 0) 0x7fd4b40d0430 con 0x7fd4c4102ac0 2024-04-23T10:26:24.417 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:24.417+0000 7fd4bbfff700 1 --2- 172.21.15.114:0/3152054937 >> [v2:172.21.15.114:6800/465333808,v1:172.21.15.114:6801/465333808] conn(0x7fd4b00a3c40 0x7fd4b00a6100 secure :-1 s=READY pgs=231 cs=0 l=1 rev1=1 crypto rx=0x7fd4ac006fd0 tx=0x7fd4ac008040 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T10:26:24.420 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:24.421+0000 7fd4c1ffb700 1 -- 172.21.15.114:0/3152054937 <== mon.0 v2:172.21.15.114:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7fd4b4014030 con 0x7fd4c4102ac0 2024-04-23T10:26:24.661 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:24.662+0000 7fd4ca9dc700 1 -- 172.21.15.114:0/3152054937 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_command({"prefix": "fs dump"} v 0) -- 0x7fd4c41a3bf0 con 0x7fd4c4102ac0 2024-04-23T10:26:24.661 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:24.662+0000 7fd4c1ffb700 1 -- 172.21.15.114:0/3152054937 <== mon.0 v2:172.21.15.114:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump"}]=0 dumped fsmap epoch 20 v20) ==== 76+0+3058 (secure 0 0 0) 0x7fd4b409cdf0 con 0x7fd4c4102ac0 2024-04-23T10:26:24.661 INFO:teuthology.orchestra.run.smithi114.stdout:e20 2024-04-23T10:26:24.662 INFO:teuthology.orchestra.run.smithi114.stdout:enable_multiple, ever_enabled_multiple: 1,1 2024-04-23T10:26:24.662 INFO:teuthology.orchestra.run.smithi114.stdout:default compat: compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,8=no anchor table,9=file layout v2,10=snaprealm v2,11=minor log segments,12=quiesce subvolumes} 2024-04-23T10:26:24.662 INFO:teuthology.orchestra.run.smithi114.stdout:legacy client fscid: 1 2024-04-23T10:26:24.662 INFO:teuthology.orchestra.run.smithi114.stdout: 2024-04-23T10:26:24.662 INFO:teuthology.orchestra.run.smithi114.stdout:Filesystem 'cephfs' (1) 2024-04-23T10:26:24.662 INFO:teuthology.orchestra.run.smithi114.stdout:fs_name cephfs 2024-04-23T10:26:24.662 INFO:teuthology.orchestra.run.smithi114.stdout:epoch 20 2024-04-23T10:26:24.662 INFO:teuthology.orchestra.run.smithi114.stdout:flags 32 joinable allow_snaps allow_multimds_snaps allow_standby_replay 2024-04-23T10:26:24.662 INFO:teuthology.orchestra.run.smithi114.stdout:created 2024-04-23T10:24:43.383158+0000 2024-04-23T10:26:24.662 INFO:teuthology.orchestra.run.smithi114.stdout:modified 2024-04-23T10:24:48.315128+0000 2024-04-23T10:26:24.662 INFO:teuthology.orchestra.run.smithi114.stdout:tableserver 0 2024-04-23T10:26:24.662 INFO:teuthology.orchestra.run.smithi114.stdout:root 0 2024-04-23T10:26:24.662 INFO:teuthology.orchestra.run.smithi114.stdout:session_timeout 300 2024-04-23T10:26:24.662 INFO:teuthology.orchestra.run.smithi114.stdout:session_autoclose 300 2024-04-23T10:26:24.663 INFO:teuthology.orchestra.run.smithi114.stdout:max_file_size 1099511627776 2024-04-23T10:26:24.663 INFO:teuthology.orchestra.run.smithi114.stdout:max_xattr_size 65536 2024-04-23T10:26:24.663 INFO:teuthology.orchestra.run.smithi114.stdout:required_client_features {} 2024-04-23T10:26:24.663 INFO:teuthology.orchestra.run.smithi114.stdout:last_failure 0 2024-04-23T10:26:24.663 INFO:teuthology.orchestra.run.smithi114.stdout:last_failure_osd_epoch 0 2024-04-23T10:26:24.663 INFO:teuthology.orchestra.run.smithi114.stdout:compat compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,7=mds uses inline data,8=no anchor table,9=file layout v2,10=snaprealm v2,11=minor log segments,12=quiesce subvolumes} 2024-04-23T10:26:24.663 INFO:teuthology.orchestra.run.smithi114.stdout:max_mds 3 2024-04-23T10:26:24.663 INFO:teuthology.orchestra.run.smithi114.stdout:in 0,1,2 2024-04-23T10:26:24.663 INFO:teuthology.orchestra.run.smithi114.stdout:up {0=24545,1=24509,2=24479} 2024-04-23T10:26:24.663 INFO:teuthology.orchestra.run.smithi114.stdout:failed 2024-04-23T10:26:24.663 INFO:teuthology.orchestra.run.smithi114.stdout:damaged 2024-04-23T10:26:24.663 INFO:teuthology.orchestra.run.smithi114.stdout:stopped 2024-04-23T10:26:24.663 INFO:teuthology.orchestra.run.smithi114.stdout:data_pools [3] 2024-04-23T10:26:24.663 INFO:teuthology.orchestra.run.smithi114.stdout:metadata_pool 2 2024-04-23T10:26:24.663 INFO:teuthology.orchestra.run.smithi114.stdout:inline_data disabled 2024-04-23T10:26:24.663 INFO:teuthology.orchestra.run.smithi114.stdout:balancer 2024-04-23T10:26:24.664 INFO:teuthology.orchestra.run.smithi114.stdout:bal_rank_mask -1 2024-04-23T10:26:24.664 INFO:teuthology.orchestra.run.smithi114.stdout:standby_count_wanted 0 2024-04-23T10:26:24.664 INFO:teuthology.orchestra.run.smithi114.stdout:qdb_cluster leader: 24545 members: 24509,24545,24479 2024-04-23T10:26:24.664 INFO:teuthology.orchestra.run.smithi114.stdout:[mds.b{0:24545} state up:active seq 3 addr [v2:172.21.15.119:6838/937297340,v1:172.21.15.119:6839/937297340] compat {c=[1],r=[1],i=[1fff]}] 2024-04-23T10:26:24.664 INFO:teuthology.orchestra.run.smithi114.stdout:[mds.i{0:24463} state up:standby-replay seq 1 addr [v2:172.21.15.204:6838/3222658893,v1:172.21.15.204:6839/3222658893] compat {c=[1],r=[1],i=[1fff]}] 2024-04-23T10:26:24.664 INFO:teuthology.orchestra.run.smithi114.stdout:[mds.h{1:24509} state up:active seq 6 addr [v2:172.21.15.119:6836/1872363128,v1:172.21.15.119:6837/1872363128] compat {c=[1],r=[1],i=[1fff]}] 2024-04-23T10:26:24.664 INFO:teuthology.orchestra.run.smithi114.stdout:[mds.k{1:24443} state up:standby-replay seq 1 addr [v2:172.21.15.119:6832/783244660,v1:172.21.15.119:6833/783244660] compat {c=[1],r=[1],i=[1fff]}] 2024-04-23T10:26:24.664 INFO:teuthology.orchestra.run.smithi114.stdout:[mds.e{2:24479} state up:active seq 9 addr [v2:172.21.15.119:6834/2033916709,v1:172.21.15.119:6835/2033916709] compat {c=[1],r=[1],i=[1fff]}] 2024-04-23T10:26:24.664 INFO:teuthology.orchestra.run.smithi114.stdout:[mds.c{2:24439} state up:standby-replay seq 1 addr [v2:172.21.15.204:6836/513822179,v1:172.21.15.204:6837/513822179] compat {c=[1],r=[1],i=[1fff]}] 2024-04-23T10:26:24.664 INFO:teuthology.orchestra.run.smithi114.stdout: 2024-04-23T10:26:24.664 INFO:teuthology.orchestra.run.smithi114.stdout: 2024-04-23T10:26:24.664 INFO:teuthology.orchestra.run.smithi114.stdout:Standby daemons: 2024-04-23T10:26:24.664 INFO:teuthology.orchestra.run.smithi114.stdout: 2024-04-23T10:26:24.664 INFO:teuthology.orchestra.run.smithi114.stdout:[mds.j{-1:14547} state up:standby seq 1 addr [v2:172.21.15.114:6834/663493580,v1:172.21.15.114:6835/663493580] compat {c=[1],r=[1],i=[1fff]}] 2024-04-23T10:26:24.664 INFO:teuthology.orchestra.run.smithi114.stdout:[mds.d{-1:14565} state up:standby seq 1 addr [v2:172.21.15.114:6836/1439252154,v1:172.21.15.114:6837/1439252154] compat {c=[1],r=[1],i=[1fff]}] 2024-04-23T10:26:24.664 INFO:teuthology.orchestra.run.smithi114.stdout:[mds.g{-1:14634} state up:standby seq 1 addr [v2:172.21.15.114:6838/3849796939,v1:172.21.15.114:6839/3849796939] compat {c=[1],r=[1],i=[1fff]}] 2024-04-23T10:26:24.665 INFO:teuthology.orchestra.run.smithi114.stdout:[mds.a{-1:14646} state up:standby seq 1 addr [v2:172.21.15.114:6840/3544523891,v1:172.21.15.114:6841/3544523891] compat {c=[1],r=[1],i=[1fff]}] 2024-04-23T10:26:24.665 INFO:teuthology.orchestra.run.smithi114.stdout:[mds.l{-1:24385} state up:standby seq 1 addr [v2:172.21.15.204:6832/3866561754,v1:172.21.15.204:6833/3866561754] compat {c=[1],r=[1],i=[1fff]}] 2024-04-23T10:26:24.665 INFO:teuthology.orchestra.run.smithi114.stdout:[mds.f{-1:24427} state up:standby seq 1 addr [v2:172.21.15.204:6834/2533793172,v1:172.21.15.204:6835/2533793172] compat {c=[1],r=[1],i=[1fff]}] 2024-04-23T10:26:24.665 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:24.664+0000 7fd4ca9dc700 1 -- 172.21.15.114:0/3152054937 >> [v2:172.21.15.114:6800/465333808,v1:172.21.15.114:6801/465333808] conn(0x7fd4b00a3c40 msgr2=0x7fd4b00a6100 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T10:26:24.665 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:24.664+0000 7fd4ca9dc700 1 --2- 172.21.15.114:0/3152054937 >> [v2:172.21.15.114:6800/465333808,v1:172.21.15.114:6801/465333808] conn(0x7fd4b00a3c40 0x7fd4b00a6100 secure :-1 s=READY pgs=231 cs=0 l=1 rev1=1 crypto rx=0x7fd4ac006fd0 tx=0x7fd4ac008040 comp rx=0 tx=0).stop 2024-04-23T10:26:24.665 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:24.664+0000 7fd4ca9dc700 1 -- 172.21.15.114:0/3152054937 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fd4c4102ac0 msgr2=0x7fd4c419ad40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T10:26:24.666 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:24.664+0000 7fd4ca9dc700 1 --2- 172.21.15.114:0/3152054937 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fd4c4102ac0 0x7fd4c419ad40 secure :-1 s=READY pgs=357 cs=0 l=1 rev1=1 crypto rx=0x7fd4b400c990 tx=0x7fd4b400cd50 comp rx=0 tx=0).stop 2024-04-23T10:26:24.666 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:24.664+0000 7fd4ca9dc700 1 -- 172.21.15.114:0/3152054937 shutdown_connections 2024-04-23T10:26:24.666 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:24.664+0000 7fd4ca9dc700 1 --2- 172.21.15.114:0/3152054937 >> [v2:172.21.15.114:6800/465333808,v1:172.21.15.114:6801/465333808] conn(0x7fd4b00a3c40 0x7fd4b00a6100 unknown :-1 s=CLOSED pgs=231 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:26:24.666 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:24.664+0000 7fd4ca9dc700 1 --2- 172.21.15.114:0/3152054937 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fd4c4104660 0x7fd4c41a22c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:26:24.666 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:24.664+0000 7fd4ca9dc700 1 --2- 172.21.15.114:0/3152054937 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fd4c4102ac0 0x7fd4c419ad40 unknown :-1 s=CLOSED pgs=357 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:26:24.666 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:24.664+0000 7fd4ca9dc700 1 --2- 172.21.15.114:0/3152054937 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fd4c4103cc0 0x7fd4c419b280 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:26:24.666 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:24.664+0000 7fd4ca9dc700 1 -- 172.21.15.114:0/3152054937 >> 172.21.15.114:0/3152054937 conn(0x7fd4c40fe060 msgr2=0x7fd4c4074c70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T10:26:24.666 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:24.664+0000 7fd4ca9dc700 1 -- 172.21.15.114:0/3152054937 shutdown_connections 2024-04-23T10:26:24.666 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:24.664+0000 7fd4ca9dc700 1 -- 172.21.15.114:0/3152054937 wait complete. 2024-04-23T10:26:24.666 INFO:teuthology.orchestra.run.smithi114.stderr:dumped fsmap epoch 20 2024-04-23T10:26:25.254 INFO:journalctl@ceph.mon.a.smithi114.stdout:Apr 23 10:26:25 smithi114 ceph-mon[29255]: pgmap v338: 97 pgs: 97 active+clean; 641 KiB data, 375 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 597 B/s wr, 8 op/s 2024-04-23T10:26:25.254 INFO:journalctl@ceph.mon.a.smithi114.stdout:Apr 23 10:26:25 smithi114 ceph-mon[29255]: from='client.? 172.21.15.114:0/3152054937' entity='client.admin' cmd=[{"prefix": "fs dump"}]: dispatch 2024-04-23T10:26:25.421 DEBUG:teuthology.orchestra.run.smithi114:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d69f13d827952bb07bb9b7973bca14f97d764d6f shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid aa240572-0159-11ef-bc93-c7b262605968 -- bash -c 'ceph osd dump' 2024-04-23T10:26:25.492 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 23 10:26:25 smithi119 ceph-mon[32608]: pgmap v338: 97 pgs: 97 active+clean; 641 KiB data, 375 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 597 B/s wr, 8 op/s 2024-04-23T10:26:25.492 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 23 10:26:25 smithi119 ceph-mon[32608]: from='client.? 172.21.15.114:0/3152054937' entity='client.admin' cmd=[{"prefix": "fs dump"}]: dispatch 2024-04-23T10:26:25.578 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 23 10:26:25 smithi204 ceph-mon[32649]: pgmap v338: 97 pgs: 97 active+clean; 641 KiB data, 375 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 597 B/s wr, 8 op/s 2024-04-23T10:26:25.578 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 23 10:26:25 smithi204 ceph-mon[32649]: from='client.? 172.21.15.114:0/3152054937' entity='client.admin' cmd=[{"prefix": "fs dump"}]: dispatch 2024-04-23T10:26:25.660 INFO:teuthology.orchestra.run.smithi114.stderr:Inferring config /var/lib/ceph/aa240572-0159-11ef-bc93-c7b262605968/mon.a/config 2024-04-23T10:26:27.269 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:27.269+0000 7f20e0e70700 1 -- 172.21.15.114:0/2868063257 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f20dc105ed0 msgr2=0x7f20dc1082c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T10:26:27.269 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:27.270+0000 7f20e0e70700 1 --2- 172.21.15.114:0/2868063257 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f20dc105ed0 0x7f20dc1082c0 secure :-1 s=READY pgs=358 cs=0 l=1 rev1=1 crypto rx=0x7f20cc009990 tx=0x7f20cc009ca0 comp rx=0 tx=0).stop 2024-04-23T10:26:27.269 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:27.270+0000 7f20e0e70700 1 -- 172.21.15.114:0/2868063257 shutdown_connections 2024-04-23T10:26:27.269 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:27.270+0000 7f20e0e70700 1 --2- 172.21.15.114:0/2868063257 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f20dc108800 0x7f20dc10ac10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:26:27.269 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:27.270+0000 7f20e0e70700 1 --2- 172.21.15.114:0/2868063257 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f20dc1035a0 0x7f20dc105990 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:26:27.269 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:27.270+0000 7f20e0e70700 1 --2- 172.21.15.114:0/2868063257 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f20dc105ed0 0x7f20dc1082c0 unknown :-1 s=CLOSED pgs=358 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:26:27.269 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:27.270+0000 7f20e0e70700 1 -- 172.21.15.114:0/2868063257 >> 172.21.15.114:0/2868063257 conn(0x7f20dc0fcf70 msgr2=0x7f20dc0ff390 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T10:26:27.269 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:27.270+0000 7f20e0e70700 1 -- 172.21.15.114:0/2868063257 shutdown_connections 2024-04-23T10:26:27.269 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:27.270+0000 7f20e0e70700 1 -- 172.21.15.114:0/2868063257 wait complete. 2024-04-23T10:26:27.271 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:27.270+0000 7f20e0e70700 1 Processor -- start 2024-04-23T10:26:27.271 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:27.270+0000 7f20e0e70700 1 -- start start 2024-04-23T10:26:27.271 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:27.271+0000 7f20e0e70700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f20dc1035a0 0x7f20dc198c70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T10:26:27.271 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:27.271+0000 7f20e0e70700 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f20dc105ed0 0x7f20dc1991b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T10:26:27.272 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:27.271+0000 7f20e0e70700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f20dc108800 0x7f20dc1a0220 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T10:26:27.272 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:27.271+0000 7f20e0e70700 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x7f20dc10e560 con 0x7f20dc1035a0 2024-04-23T10:26:27.272 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:27.271+0000 7f20e0e70700 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 -- 0x7f20dc10e3e0 con 0x7f20dc108800 2024-04-23T10:26:27.272 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:27.271+0000 7f20e0e70700 1 -- --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_getmap magic: 0 -- 0x7f20dc10e6e0 con 0x7f20dc105ed0 2024-04-23T10:26:27.272 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:27.271+0000 7f20da59c700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f20dc1035a0 0x7f20dc198c70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T10:26:27.272 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:27.271+0000 7f20da59c700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f20dc1035a0 0x7f20dc198c70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.114:58232/0 (socket says 172.21.15.114:58232) 2024-04-23T10:26:27.272 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:27.271+0000 7f20da59c700 1 -- 172.21.15.114:0/197534427 learned_addr learned my addr 172.21.15.114:0/197534427 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-04-23T10:26:27.272 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:27.271+0000 7f20dad9d700 1 --2- 172.21.15.114:0/197534427 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f20dc108800 0x7f20dc1a0220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T10:26:27.272 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:27.271+0000 7f20d9d9b700 1 --2- 172.21.15.114:0/197534427 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f20dc105ed0 0x7f20dc1991b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T10:26:27.272 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:27.271+0000 7f20da59c700 1 -- 172.21.15.114:0/197534427 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f20dc105ed0 msgr2=0x7f20dc1991b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T10:26:27.272 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:27.271+0000 7f20da59c700 1 --2- 172.21.15.114:0/197534427 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f20dc105ed0 0x7f20dc1991b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:26:27.272 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:27.271+0000 7f20da59c700 1 -- 172.21.15.114:0/197534427 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f20dc108800 msgr2=0x7f20dc1a0220 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T10:26:27.272 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:27.271+0000 7f20da59c700 1 --2- 172.21.15.114:0/197534427 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f20dc108800 0x7f20dc1a0220 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:26:27.272 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:27.271+0000 7f20da59c700 1 -- 172.21.15.114:0/197534427 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f20dc1a08f0 con 0x7f20dc1035a0 2024-04-23T10:26:27.272 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:27.271+0000 7f20d9d9b700 1 --2- 172.21.15.114:0/197534427 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f20dc105ed0 0x7f20dc1991b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-23T10:26:27.272 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:27.271+0000 7f20da59c700 1 --2- 172.21.15.114:0/197534427 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f20dc1035a0 0x7f20dc198c70 secure :-1 s=READY pgs=359 cs=0 l=1 rev1=1 crypto rx=0x7f20d000d980 tx=0x7f20d000dc90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T10:26:27.273 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:27.271+0000 7f20cb7fe700 1 -- 172.21.15.114:0/197534427 <== mon.0 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f20d0017070 con 0x7f20dc1035a0 2024-04-23T10:26:27.273 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:27.271+0000 7f20cb7fe700 1 -- 172.21.15.114:0/197534427 <== mon.0 v2:172.21.15.114:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f20d0005020 con 0x7f20dc1035a0 2024-04-23T10:26:27.273 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:27.271+0000 7f20cb7fe700 1 -- 172.21.15.114:0/197534427 <== mon.0 v2:172.21.15.114:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f20d0009b50 con 0x7f20dc1035a0 2024-04-23T10:26:27.273 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:27.271+0000 7f20e0e70700 1 -- 172.21.15.114:0/197534427 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f20dc1a0be0 con 0x7f20dc1035a0 2024-04-23T10:26:27.273 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:27.271+0000 7f20e0e70700 1 -- 172.21.15.114:0/197534427 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f20dc1a1120 con 0x7f20dc1035a0 2024-04-23T10:26:27.273 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:27.272+0000 7f20e0e70700 1 -- 172.21.15.114:0/197534427 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f20dc069730 con 0x7f20dc1035a0 2024-04-23T10:26:27.274 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:27.274+0000 7f20cb7fe700 1 -- 172.21.15.114:0/197534427 <== mon.0 v2:172.21.15.114:3300/0 4 ==== mgrmap(e 18) ==== 138647+0+0 (secure 0 0 0) 0x7f20d00051d0 con 0x7f20dc1035a0 2024-04-23T10:26:27.274 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:27.274+0000 7f20cb7fe700 1 --2- 172.21.15.114:0/197534427 >> [v2:172.21.15.114:6800/465333808,v1:172.21.15.114:6801/465333808] conn(0x7f20c40a3cc0 0x7f20c40a6180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T10:26:27.274 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:27.274+0000 7f20d9d9b700 1 --2- 172.21.15.114:0/197534427 >> [v2:172.21.15.114:6800/465333808,v1:172.21.15.114:6801/465333808] conn(0x7f20c40a3cc0 0x7f20c40a6180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T10:26:27.274 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:27.274+0000 7f20cb7fe700 1 -- 172.21.15.114:0/197534427 <== mon.0 v2:172.21.15.114:3300/0 5 ==== osd_map(89..89 src has 1..89) ==== 8269+0+0 (secure 0 0 0) 0x7f20d00d06a0 con 0x7f20dc1035a0 2024-04-23T10:26:27.274 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:27.275+0000 7f20d9d9b700 1 --2- 172.21.15.114:0/197534427 >> [v2:172.21.15.114:6800/465333808,v1:172.21.15.114:6801/465333808] conn(0x7f20c40a3cc0 0x7f20c40a6180 secure :-1 s=READY pgs=232 cs=0 l=1 rev1=1 crypto rx=0x7f20cc005cd0 tx=0x7f20cc005c20 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T10:26:27.276 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:27.278+0000 7f20cb7fe700 1 -- 172.21.15.114:0/197534427 <== mon.0 v2:172.21.15.114:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f20d0020020 con 0x7f20dc1035a0 2024-04-23T10:26:27.380 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 23 10:26:27 smithi204 ceph-mon[32649]: pgmap v339: 97 pgs: 97 active+clean; 641 KiB data, 375 MiB used, 1.0 TiB / 1.0 TiB avail; 2.7 KiB/s rd, 597 B/s wr, 5 op/s 2024-04-23T10:26:27.473 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:27.474+0000 7f20e0e70700 1 -- 172.21.15.114:0/197534427 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_command({"prefix": "osd dump"} v 0) -- 0x7f20dc02e000 con 0x7f20dc1035a0 2024-04-23T10:26:27.473 INFO:journalctl@ceph.mon.a.smithi114.stdout:Apr 23 10:26:27 smithi114 ceph-mon[29255]: pgmap v339: 97 pgs: 97 active+clean; 641 KiB data, 375 MiB used, 1.0 TiB / 1.0 TiB avail; 2.7 KiB/s rd, 597 B/s wr, 5 op/s 2024-04-23T10:26:27.474 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:27.475+0000 7f20cb7fe700 1 -- 172.21.15.114:0/197534427 <== mon.0 v2:172.21.15.114:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump"}]=0 v89) ==== 56+0+5367 (secure 0 0 0) 0x7f20d009d0e0 con 0x7f20dc1035a0 2024-04-23T10:26:27.474 INFO:teuthology.orchestra.run.smithi114.stdout:epoch 89 2024-04-23T10:26:27.475 INFO:teuthology.orchestra.run.smithi114.stdout:fsid aa240572-0159-11ef-bc93-c7b262605968 2024-04-23T10:26:27.475 INFO:teuthology.orchestra.run.smithi114.stdout:created 2024-04-23T10:16:53.351136+0000 2024-04-23T10:26:27.475 INFO:teuthology.orchestra.run.smithi114.stdout:modified 2024-04-23T10:25:47.573901+0000 2024-04-23T10:26:27.475 INFO:teuthology.orchestra.run.smithi114.stdout:flags sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit 2024-04-23T10:26:27.475 INFO:teuthology.orchestra.run.smithi114.stdout:crush_version 26 2024-04-23T10:26:27.475 INFO:teuthology.orchestra.run.smithi114.stdout:full_ratio 0.95 2024-04-23T10:26:27.475 INFO:teuthology.orchestra.run.smithi114.stdout:backfillfull_ratio 0.9 2024-04-23T10:26:27.475 INFO:teuthology.orchestra.run.smithi114.stdout:nearfull_ratio 0.85 2024-04-23T10:26:27.475 INFO:teuthology.orchestra.run.smithi114.stdout:require_min_compat_client luminous 2024-04-23T10:26:27.475 INFO:teuthology.orchestra.run.smithi114.stdout:min_compat_client luminous 2024-04-23T10:26:27.475 INFO:teuthology.orchestra.run.smithi114.stdout:require_osd_release squid 2024-04-23T10:26:27.475 INFO:teuthology.orchestra.run.smithi114.stdout:stretch_mode_enabled false 2024-04-23T10:26:27.475 INFO:teuthology.orchestra.run.smithi114.stdout:pool 1 '.mgr' replicated size 3 min_size 2 crush_rule 0 object_hash rjenkins pg_num 1 pgp_num 1 autoscale_mode off last_change 22 flags hashpspool stripe_width 0 pg_num_max 32 pg_num_min 1 application mgr read_balance_score 12.00 2024-04-23T10:26:27.475 INFO:teuthology.orchestra.run.smithi114.stdout:pool 2 'cephfs_metadata' replicated size 3 min_size 2 crush_rule 0 object_hash rjenkins pg_num 32 pgp_num 32 autoscale_mode off last_change 86 flags hashpspool stripe_width 0 pg_autoscale_bias 4 pg_num_min 16 recovery_priority 5 application cephfs read_balance_score 1.88 2024-04-23T10:26:27.476 INFO:teuthology.orchestra.run.smithi114.stdout:pool 3 'cephfs_data' replicated size 3 min_size 2 crush_rule 0 object_hash rjenkins pg_num 64 pgp_num 64 autoscale_mode off last_change 87 flags hashpspool stripe_width 0 pg_num_min 64 target_size_ratio 0.9 application cephfs read_balance_score 1.50 2024-04-23T10:26:27.476 INFO:teuthology.orchestra.run.smithi114.stdout:max_osd 12 2024-04-23T10:26:27.476 INFO:teuthology.orchestra.run.smithi114.stdout:osd.0 up in weight 1 up_from 8 up_thru 88 down_at 0 last_clean_interval [0,0) [v2:172.21.15.114:6802/736366878,v1:172.21.15.114:6803/736366878] [v2:172.21.15.114:6804/736366878,v1:172.21.15.114:6805/736366878] exists,up fd64adde-9445-45a4-a171-ef656e2c5709 2024-04-23T10:26:27.476 INFO:teuthology.orchestra.run.smithi114.stdout:osd.1 up in weight 1 up_from 13 up_thru 85 down_at 0 last_clean_interval [0,0) [v2:172.21.15.119:6800/1133540779,v1:172.21.15.119:6801/1133540779] [v2:172.21.15.119:6802/1133540779,v1:172.21.15.119:6803/1133540779] exists,up e279a1e6-2702-472a-930a-e08ece76a861 2024-04-23T10:26:27.476 INFO:teuthology.orchestra.run.smithi114.stdout:osd.2 up in weight 1 up_from 18 up_thru 85 down_at 0 last_clean_interval [0,0) [v2:172.21.15.204:6800/799652511,v1:172.21.15.204:6801/799652511] [v2:172.21.15.204:6802/799652511,v1:172.21.15.204:6803/799652511] exists,up f4c9f2b4-ff49-4e14-89cd-b477feeb9382 2024-04-23T10:26:27.476 INFO:teuthology.orchestra.run.smithi114.stdout:osd.3 up in weight 1 up_from 25 up_thru 88 down_at 0 last_clean_interval [0,0) [v2:172.21.15.114:6810/4275595091,v1:172.21.15.114:6811/4275595091] [v2:172.21.15.114:6812/4275595091,v1:172.21.15.114:6813/4275595091] exists,up 52d2a566-e71c-4bb3-8d6c-5abdd43177c2 2024-04-23T10:26:27.476 INFO:teuthology.orchestra.run.smithi114.stdout:osd.4 up in weight 1 up_from 33 up_thru 85 down_at 0 last_clean_interval [0,0) [v2:172.21.15.119:6808/1611007172,v1:172.21.15.119:6809/1611007172] [v2:172.21.15.119:6810/1611007172,v1:172.21.15.119:6811/1611007172] exists,up d911a7c1-4bdc-4d28-b8a6-73896a6e7de6 2024-04-23T10:26:27.476 INFO:teuthology.orchestra.run.smithi114.stdout:osd.5 up in weight 1 up_from 39 up_thru 85 down_at 0 last_clean_interval [0,0) [v2:172.21.15.204:6808/1427563312,v1:172.21.15.204:6809/1427563312] [v2:172.21.15.204:6810/1427563312,v1:172.21.15.204:6811/1427563312] exists,up f7590e1a-e93c-4c29-b15f-38ee52d1f921 2024-04-23T10:26:27.476 INFO:teuthology.orchestra.run.smithi114.stdout:osd.6 up in weight 1 up_from 47 up_thru 88 down_at 0 last_clean_interval [0,0) [v2:172.21.15.114:6818/4072912544,v1:172.21.15.114:6819/4072912544] [v2:172.21.15.114:6820/4072912544,v1:172.21.15.114:6821/4072912544] exists,up 534d20e8-3e54-4db8-a845-6119f49c7a78 2024-04-23T10:26:27.476 INFO:teuthology.orchestra.run.smithi114.stdout:osd.7 up in weight 1 up_from 53 up_thru 88 down_at 0 last_clean_interval [0,0) [v2:172.21.15.119:6816/2237458793,v1:172.21.15.119:6817/2237458793] [v2:172.21.15.119:6818/2237458793,v1:172.21.15.119:6819/2237458793] exists,up f0ec22b8-1b48-4859-9d65-b00403aff3ad 2024-04-23T10:26:27.476 INFO:teuthology.orchestra.run.smithi114.stdout:osd.8 up in weight 1 up_from 62 up_thru 85 down_at 0 last_clean_interval [0,0) [v2:172.21.15.204:6816/1426318231,v1:172.21.15.204:6817/1426318231] [v2:172.21.15.204:6818/1426318231,v1:172.21.15.204:6819/1426318231] exists,up 3ac89686-6d76-4cef-9089-ac29e4b3bcac 2024-04-23T10:26:27.476 INFO:teuthology.orchestra.run.smithi114.stdout:osd.9 up in weight 1 up_from 69 up_thru 88 down_at 0 last_clean_interval [0,0) [v2:172.21.15.114:6826/2750635721,v1:172.21.15.114:6827/2750635721] [v2:172.21.15.114:6828/2750635721,v1:172.21.15.114:6829/2750635721] exists,up a80ffe4d-1e98-460f-b046-c4754285cc5a 2024-04-23T10:26:27.476 INFO:teuthology.orchestra.run.smithi114.stdout:osd.10 up in weight 1 up_from 74 up_thru 85 down_at 0 last_clean_interval [0,0) [v2:172.21.15.119:6824/1400187701,v1:172.21.15.119:6825/1400187701] [v2:172.21.15.119:6826/1400187701,v1:172.21.15.119:6827/1400187701] exists,up cd75a579-bf12-4600-ac09-a8bfce772ced 2024-04-23T10:26:27.476 INFO:teuthology.orchestra.run.smithi114.stdout:osd.11 up in weight 1 up_from 82 up_thru 88 down_at 0 last_clean_interval [0,0) [v2:172.21.15.204:6824/2407298490,v1:172.21.15.204:6825/2407298490] [v2:172.21.15.204:6826/2407298490,v1:172.21.15.204:6827/2407298490] exists,up 53f037a5-3ec0-483a-80a1-dfee923a68e5 2024-04-23T10:26:27.476 INFO:teuthology.orchestra.run.smithi114.stdout:pg_upmap_items 2.e [2,4] 2024-04-23T10:26:27.476 INFO:teuthology.orchestra.run.smithi114.stdout:pg_upmap_items 2.1d [8,11] 2024-04-23T10:26:27.477 INFO:teuthology.orchestra.run.smithi114.stdout:pg_upmap_items 3.e [5,9] 2024-04-23T10:26:27.477 INFO:teuthology.orchestra.run.smithi114.stdout:pg_upmap_items 3.12 [3,9] 2024-04-23T10:26:27.477 INFO:teuthology.orchestra.run.smithi114.stdout:pg_upmap_items 3.17 [4,6] 2024-04-23T10:26:27.477 INFO:teuthology.orchestra.run.smithi114.stdout:pg_upmap_items 3.23 [4,7] 2024-04-23T10:26:27.477 INFO:teuthology.orchestra.run.smithi114.stdout:pg_upmap_items 3.27 [4,6] 2024-04-23T10:26:27.477 INFO:teuthology.orchestra.run.smithi114.stdout:pg_upmap_items 3.31 [4,8] 2024-04-23T10:26:27.477 INFO:teuthology.orchestra.run.smithi114.stdout:pg_upmap_items 3.38 [3,9] 2024-04-23T10:26:27.477 INFO:teuthology.orchestra.run.smithi114.stdout:pg_upmap_items 3.39 [5,8] 2024-04-23T10:26:27.477 INFO:teuthology.orchestra.run.smithi114.stdout:blocklist 172.21.15.114:6801/1604678557 expires 2024-04-24T10:17:45.874609+0000 2024-04-23T10:26:27.477 INFO:teuthology.orchestra.run.smithi114.stdout:blocklist 172.21.15.114:0/4025385854 expires 2024-04-24T10:17:45.874609+0000 2024-04-23T10:26:27.477 INFO:teuthology.orchestra.run.smithi114.stdout:blocklist 172.21.15.114:0/2228273469 expires 2024-04-24T10:17:19.910348+0000 2024-04-23T10:26:27.477 INFO:teuthology.orchestra.run.smithi114.stdout:blocklist 172.21.15.114:6800/2831530838 expires 2024-04-24T10:17:19.910348+0000 2024-04-23T10:26:27.477 INFO:teuthology.orchestra.run.smithi114.stdout:blocklist 172.21.15.114:0/1333447450 expires 2024-04-24T10:17:45.874609+0000 2024-04-23T10:26:27.477 INFO:teuthology.orchestra.run.smithi114.stdout:blocklist 172.21.15.114:0/2233026840 expires 2024-04-24T10:17:19.910348+0000 2024-04-23T10:26:27.477 INFO:teuthology.orchestra.run.smithi114.stdout:blocklist 172.21.15.114:0/1902281545 expires 2024-04-24T10:17:19.910348+0000 2024-04-23T10:26:27.478 INFO:teuthology.orchestra.run.smithi114.stdout:blocklist 172.21.15.114:6801/2831530838 expires 2024-04-24T10:17:19.910348+0000 2024-04-23T10:26:27.478 INFO:teuthology.orchestra.run.smithi114.stdout:blocklist 172.21.15.114:6800/1604678557 expires 2024-04-24T10:17:45.874609+0000 2024-04-23T10:26:27.478 INFO:teuthology.orchestra.run.smithi114.stdout:blocklist 172.21.15.114:0/1623553811 expires 2024-04-24T10:17:45.874609+0000 2024-04-23T10:26:27.478 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:27.477+0000 7f20e0e70700 1 -- 172.21.15.114:0/197534427 >> [v2:172.21.15.114:6800/465333808,v1:172.21.15.114:6801/465333808] conn(0x7f20c40a3cc0 msgr2=0x7f20c40a6180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T10:26:27.478 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:27.477+0000 7f20e0e70700 1 --2- 172.21.15.114:0/197534427 >> [v2:172.21.15.114:6800/465333808,v1:172.21.15.114:6801/465333808] conn(0x7f20c40a3cc0 0x7f20c40a6180 secure :-1 s=READY pgs=232 cs=0 l=1 rev1=1 crypto rx=0x7f20cc005cd0 tx=0x7f20cc005c20 comp rx=0 tx=0).stop 2024-04-23T10:26:27.479 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:27.477+0000 7f20e0e70700 1 -- 172.21.15.114:0/197534427 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f20dc1035a0 msgr2=0x7f20dc198c70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T10:26:27.479 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:27.477+0000 7f20e0e70700 1 --2- 172.21.15.114:0/197534427 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f20dc1035a0 0x7f20dc198c70 secure :-1 s=READY pgs=359 cs=0 l=1 rev1=1 crypto rx=0x7f20d000d980 tx=0x7f20d000dc90 comp rx=0 tx=0).stop 2024-04-23T10:26:27.479 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:27.478+0000 7f20e0e70700 1 -- 172.21.15.114:0/197534427 shutdown_connections 2024-04-23T10:26:27.479 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:27.478+0000 7f20e0e70700 1 --2- 172.21.15.114:0/197534427 >> [v2:172.21.15.114:6800/465333808,v1:172.21.15.114:6801/465333808] conn(0x7f20c40a3cc0 0x7f20c40a6180 unknown :-1 s=CLOSED pgs=232 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:26:27.479 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:27.478+0000 7f20e0e70700 1 --2- 172.21.15.114:0/197534427 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f20dc108800 0x7f20dc1a0220 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:26:27.479 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:27.478+0000 7f20e0e70700 1 --2- 172.21.15.114:0/197534427 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f20dc1035a0 0x7f20dc198c70 unknown :-1 s=CLOSED pgs=359 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:26:27.479 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:27.478+0000 7f20e0e70700 1 --2- 172.21.15.114:0/197534427 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f20dc105ed0 0x7f20dc1991b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:26:27.479 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:27.478+0000 7f20e0e70700 1 -- 172.21.15.114:0/197534427 >> 172.21.15.114:0/197534427 conn(0x7f20dc0fcf70 msgr2=0x7f20dc103ee0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T10:26:27.479 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:27.478+0000 7f20e0e70700 1 -- 172.21.15.114:0/197534427 shutdown_connections 2024-04-23T10:26:27.479 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:27.478+0000 7f20e0e70700 1 -- 172.21.15.114:0/197534427 wait complete. 2024-04-23T10:26:27.492 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 23 10:26:27 smithi119 ceph-mon[32608]: pgmap v339: 97 pgs: 97 active+clean; 641 KiB data, 375 MiB used, 1.0 TiB / 1.0 TiB avail; 2.7 KiB/s rd, 597 B/s wr, 5 op/s 2024-04-23T10:26:28.140 INFO:teuthology.run_tasks:Running task fs.ready... 2024-04-23T10:26:28.150 DEBUG:teuthology.orchestra.run.smithi114:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-04-23T10:26:28.250 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:28.251+0000 7f67a6a31640 1 -- 172.21.15.114:0/1533159592 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f67a01029b0 msgr2=0x7f67a01092d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T10:26:28.250 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:28.251+0000 7f67a6a31640 1 --2- 172.21.15.114:0/1533159592 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f67a01029b0 0x7f67a01092d0 secure :-1 s=READY pgs=360 cs=0 l=1 rev1=1 crypto rx=0x7f6798009ca0 tx=0x7f679802a8c0 comp rx=0 tx=0).stop 2024-04-23T10:26:28.250 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:28.251+0000 7f67a6a31640 1 -- 172.21.15.114:0/1533159592 shutdown_connections 2024-04-23T10:26:28.250 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:28.251+0000 7f67a6a31640 1 --2- 172.21.15.114:0/1533159592 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f67a01029b0 0x7f67a01092d0 unknown :-1 s=CLOSED pgs=360 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:26:28.250 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:28.251+0000 7f67a6a31640 1 --2- 172.21.15.114:0/1533159592 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f67a0101ff0 0x7f67a0102470 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:26:28.250 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:28.251+0000 7f67a6a31640 1 --2- 172.21.15.114:0/1533159592 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f67a0100ea0 0x7f67a01012c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:26:28.250 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:28.251+0000 7f67a6a31640 1 -- 172.21.15.114:0/1533159592 >> 172.21.15.114:0/1533159592 conn(0x7f67a00fc430 msgr2=0x7f67a00fe850 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T10:26:28.251 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:28.251+0000 7f67a6a31640 1 -- 172.21.15.114:0/1533159592 shutdown_connections 2024-04-23T10:26:28.251 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:28.251+0000 7f67a6a31640 1 -- 172.21.15.114:0/1533159592 wait complete. 2024-04-23T10:26:28.253 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:28.251+0000 7f67a6a31640 1 Processor -- start 2024-04-23T10:26:28.253 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:28.251+0000 7f67a6a31640 1 -- start start 2024-04-23T10:26:28.253 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:28.252+0000 7f67a6a31640 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f67a0100ea0 0x7f67a010b840 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T10:26:28.253 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:28.252+0000 7f67a6a31640 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f67a0101ff0 0x7f67a0109e80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T10:26:28.253 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:28.252+0000 7f67a6a31640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f67a01029b0 0x7f67a010a3c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T10:26:28.253 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:28.252+0000 7f67a6a31640 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x7f67a010db40 con 0x7f67a01029b0 2024-04-23T10:26:28.253 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:28.252+0000 7f67a6a31640 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 -- 0x7f67a010d9c0 con 0x7f67a0101ff0 2024-04-23T10:26:28.253 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:28.252+0000 7f67a6a31640 1 -- --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_getmap magic: 0 -- 0x7f67a010dcc0 con 0x7f67a0100ea0 2024-04-23T10:26:28.253 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:28.252+0000 7f67a4fce640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f67a01029b0 0x7f67a010a3c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T10:26:28.253 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:28.252+0000 7f67a4fce640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f67a01029b0 0x7f67a010a3c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.114:58262/0 (socket says 172.21.15.114:58262) 2024-04-23T10:26:28.253 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:28.252+0000 7f67a4fce640 1 -- 172.21.15.114:0/2101001458 learned_addr learned my addr 172.21.15.114:0/2101001458 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-04-23T10:26:28.253 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:28.252+0000 7f679f7fe640 1 --2- 172.21.15.114:0/2101001458 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f67a0101ff0 0x7f67a0109e80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T10:26:28.254 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:28.252+0000 7f679ffff640 1 --2- 172.21.15.114:0/2101001458 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f67a0100ea0 0x7f67a010b840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T10:26:28.254 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:28.252+0000 7f67a4fce640 1 -- 172.21.15.114:0/2101001458 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f67a0100ea0 msgr2=0x7f67a010b840 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T10:26:28.254 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:28.252+0000 7f67a4fce640 1 --2- 172.21.15.114:0/2101001458 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f67a0100ea0 0x7f67a010b840 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:26:28.254 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:28.252+0000 7f67a4fce640 1 -- 172.21.15.114:0/2101001458 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f67a0101ff0 msgr2=0x7f67a0109e80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T10:26:28.254 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:28.252+0000 7f67a4fce640 1 --2- 172.21.15.114:0/2101001458 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f67a0101ff0 0x7f67a0109e80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:26:28.254 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:28.252+0000 7f67a4fce640 1 -- 172.21.15.114:0/2101001458 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f67a010ac50 con 0x7f67a01029b0 2024-04-23T10:26:28.254 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:28.252+0000 7f67a4fce640 1 --2- 172.21.15.114:0/2101001458 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f67a01029b0 0x7f67a010a3c0 secure :-1 s=READY pgs=361 cs=0 l=1 rev1=1 crypto rx=0x7f67980097b0 tx=0x7f6798003c20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T10:26:28.254 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:28.252+0000 7f679d7fa640 1 -- 172.21.15.114:0/2101001458 <== mon.0 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6798004240 con 0x7f67a01029b0 2024-04-23T10:26:28.254 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:28.253+0000 7f679d7fa640 1 -- 172.21.15.114:0/2101001458 <== mon.0 v2:172.21.15.114:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f67980317c0 con 0x7f67a01029b0 2024-04-23T10:26:28.254 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:28.253+0000 7f679d7fa640 1 -- 172.21.15.114:0/2101001458 <== mon.0 v2:172.21.15.114:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f679802fd80 con 0x7f67a01029b0 2024-04-23T10:26:28.254 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:28.253+0000 7f67a6a31640 1 -- 172.21.15.114:0/2101001458 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f67a01a22a0 con 0x7f67a01029b0 2024-04-23T10:26:28.254 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:28.253+0000 7f67a6a31640 1 -- 172.21.15.114:0/2101001458 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f67a01a27f0 con 0x7f67a01029b0 2024-04-23T10:26:28.254 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:28.253+0000 7f67a6a31640 1 -- 172.21.15.114:0/2101001458 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f67a01081b0 con 0x7f67a01029b0 2024-04-23T10:26:28.255 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:28.255+0000 7f679d7fa640 1 -- 172.21.15.114:0/2101001458 <== mon.0 v2:172.21.15.114:3300/0 4 ==== mgrmap(e 18) ==== 138647+0+0 (secure 0 0 0) 0x7f67980043e0 con 0x7f67a01029b0 2024-04-23T10:26:28.255 INFO:journalctl@ceph.mon.a.smithi114.stdout:Apr 23 10:26:28 smithi114 ceph-mon[29255]: from='client.? 172.21.15.114:0/197534427' entity='client.admin' cmd=[{"prefix": "osd dump"}]: dispatch 2024-04-23T10:26:28.256 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:28.255+0000 7f679d7fa640 1 --2- 172.21.15.114:0/2101001458 >> [v2:172.21.15.114:6800/465333808,v1:172.21.15.114:6801/465333808] conn(0x7f67880a3bb0 0x7f67880a6070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T10:26:28.256 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:28.255+0000 7f679ffff640 1 --2- 172.21.15.114:0/2101001458 >> [v2:172.21.15.114:6800/465333808,v1:172.21.15.114:6801/465333808] conn(0x7f67880a3bb0 0x7f67880a6070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T10:26:28.256 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:28.255+0000 7f679d7fa640 1 -- 172.21.15.114:0/2101001458 <== mon.0 v2:172.21.15.114:3300/0 5 ==== osd_map(89..89 src has 1..89) ==== 8269+0+0 (secure 0 0 0) 0x7f67980f1be0 con 0x7f67a01029b0 2024-04-23T10:26:28.256 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:28.256+0000 7f679ffff640 1 --2- 172.21.15.114:0/2101001458 >> [v2:172.21.15.114:6800/465333808,v1:172.21.15.114:6801/465333808] conn(0x7f67880a3bb0 0x7f67880a6070 secure :-1 s=READY pgs=233 cs=0 l=1 rev1=1 crypto rx=0x7f67940058c0 tx=0x7f679400a250 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T10:26:28.258 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:28.260+0000 7f679d7fa640 1 -- 172.21.15.114:0/2101001458 <== mon.0 v2:172.21.15.114:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f67980be6c0 con 0x7f67a01029b0 2024-04-23T10:26:28.439 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:28.440+0000 7f67a6a31640 1 -- 172.21.15.114:0/2101001458 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f67a0062b50 con 0x7f67a01029b0 2024-04-23T10:26:28.440 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:28.441+0000 7f679d7fa640 1 -- 172.21.15.114:0/2101001458 <== mon.0 v2:172.21.15.114:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v89) ==== 74+0+22818 (secure 0 0 0) 0x7f67980bdfc0 con 0x7f67a01029b0 2024-04-23T10:26:28.441 INFO:teuthology.orchestra.run.smithi114.stdout: 2024-04-23T10:26:28.441 INFO:teuthology.orchestra.run.smithi114.stdout:{"epoch":89,"fsid":"aa240572-0159-11ef-bc93-c7b262605968","created":"2024-04-23T10:16:53.351136+0000","modified":"2024-04-23T10:25:47.573901+0000","last_up_change":"2024-04-23T10:23:54.153269+0000","last_in_change":"2024-04-23T10:23:32.963085+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":26,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-04-23T10:20:11.970297+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":12,"score_stable":12,"optimal_score":0.25,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-04-23T10:24:41.218540+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"86","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.8799999952316284,"score_stable":1.8799999952316284,"optimal_score":1,"raw_score_acting":1.8799999952316284,"raw_score_stable":1.8799999952316284,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-04-23T10:24:42.341799+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"87","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"fd64adde-9445-45a4-a171-ef656e2c5709","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":88,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6802","nonce":736366878},{"type":"v1","addr":"172.21.15.114:6803","nonce":736366878}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6804","nonce":736366878},{"type":"v1","addr":"172.21.15.114:6805","nonce":736366878}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6808","nonce":736366878},{"type":"v1","addr":"172.21.15.114:6809","nonce":736366878}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6806","nonce":736366878},{"type":"v1","addr":"172.21.15.114:6807","nonce":736366878}]},"public_addr":"172.21.15.114:6803/736366878","cluster_addr":"172.21.15.114:6805/736366878","heartbeat_back_addr":"172.21.15.114:6809/736366878","heartbeat_front_addr":"172.21.15.114:6807/736366878","state":["exists","up"]},{"osd":1,"uuid":"e279a1e6-2702-472a-930a-e08ece76a861","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":85,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6800","nonce":1133540779},{"type":"v1","addr":"172.21.15.119:6801","nonce":1133540779}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6802","nonce":1133540779},{"type":"v1","addr":"172.21.15.119:6803","nonce":1133540779}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6806","nonce":1133540779},{"type":"v1","addr":"172.21.15.119:6807","nonce":1133540779}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6804","nonce":1133540779},{"type":"v1","addr":"172.21.15.119:6805","nonce":1133540779}]},"public_addr":"172.21.15.119:6801/1133540779","cluster_addr":"172.21.15.119:6803/1133540779","heartbeat_back_addr":"172.21.15.119:6807/1133540779","heartbeat_front_addr":"172.21.15.119:6805/1133540779","state":["exists","up"]},{"osd":2,"uuid":"f4c9f2b4-ff49-4e14-89cd-b477feeb9382","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":85,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6800","nonce":799652511},{"type":"v1","addr":"172.21.15.204:6801","nonce":799652511}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6802","nonce":799652511},{"type":"v1","addr":"172.21.15.204:6803","nonce":799652511}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6806","nonce":799652511},{"type":"v1","addr":"172.21.15.204:6807","nonce":799652511}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6804","nonce":799652511},{"type":"v1","addr":"172.21.15.204:6805","nonce":799652511}]},"public_addr":"172.21.15.204:6801/799652511","cluster_addr":"172.21.15.204:6803/799652511","heartbeat_back_addr":"172.21.15.204:6807/799652511","heartbeat_front_addr":"172.21.15.204:6805/799652511","state":["exists","up"]},{"osd":3,"uuid":"52d2a566-e71c-4bb3-8d6c-5abdd43177c2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":25,"up_thru":88,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6810","nonce":4275595091},{"type":"v1","addr":"172.21.15.114:6811","nonce":4275595091}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6812","nonce":4275595091},{"type":"v1","addr":"172.21.15.114:6813","nonce":4275595091}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6816","nonce":4275595091},{"type":"v1","addr":"172.21.15.114:6817","nonce":4275595091}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6814","nonce":4275595091},{"type":"v1","addr":"172.21.15.114:6815","nonce":4275595091}]},"public_addr":"172.21.15.114:6811/4275595091","cluster_addr":"172.21.15.114:6813/4275595091","heartbeat_back_addr":"172.21.15.114:6817/4275595091","heartbeat_front_addr":"172.21.15.114:6815/4275595091","state":["exists","up"]},{"osd":4,"uuid":"d911a7c1-4bdc-4d28-b8a6-73896a6e7de6","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":33,"up_thru":85,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6808","nonce":1611007172},{"type":"v1","addr":"172.21.15.119:6809","nonce":1611007172}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6810","nonce":1611007172},{"type":"v1","addr":"172.21.15.119:6811","nonce":1611007172}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6814","nonce":1611007172},{"type":"v1","addr":"172.21.15.119:6815","nonce":1611007172}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6812","nonce":1611007172},{"type":"v1","addr":"172.21.15.119:6813","nonce":1611007172}]},"public_addr":"172.21.15.119:6809/1611007172","cluster_addr":"172.21.15.119:6811/1611007172","heartbeat_back_addr":"172.21.15.119:6815/1611007172","heartbeat_front_addr":"172.21.15.119:6813/1611007172","state":["exists","up"]},{"osd":5,"uuid":"f7590e1a-e93c-4c29-b15f-38ee52d1f921","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":39,"up_thru":85,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6808","nonce":1427563312},{"type":"v1","addr":"172.21.15.204:6809","nonce":1427563312}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6810","nonce":1427563312},{"type":"v1","addr":"172.21.15.204:6811","nonce":1427563312}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6814","nonce":1427563312},{"type":"v1","addr":"172.21.15.204:6815","nonce":1427563312}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6812","nonce":1427563312},{"type":"v1","addr":"172.21.15.204:6813","nonce":1427563312}]},"public_addr":"172.21.15.204:6809/1427563312","cluster_addr":"172.21.15.204:6811/1427563312","heartbeat_back_addr":"172.21.15.204:6815/1427563312","heartbeat_front_addr":"172.21.15.204:6813/1427563312","state":["exists","up"]},{"osd":6,"uuid":"534d20e8-3e54-4db8-a845-6119f49c7a78","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":47,"up_thru":88,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6818","nonce":4072912544},{"type":"v1","addr":"172.21.15.114:6819","nonce":4072912544}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6820","nonce":4072912544},{"type":"v1","addr":"172.21.15.114:6821","nonce":4072912544}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6824","nonce":4072912544},{"type":"v1","addr":"172.21.15.114:6825","nonce":4072912544}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6822","nonce":4072912544},{"type":"v1","addr":"172.21.15.114:6823","nonce":4072912544}]},"public_addr":"172.21.15.114:6819/4072912544","cluster_addr":"172.21.15.114:6821/4072912544","heartbeat_back_addr":"172.21.15.114:6825/4072912544","heartbeat_front_addr":"172.21.15.114:6823/4072912544","state":["exists","up"]},{"osd":7,"uuid":"f0ec22b8-1b48-4859-9d65-b00403aff3ad","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":53,"up_thru":88,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6816","nonce":2237458793},{"type":"v1","addr":"172.21.15.119:6817","nonce":2237458793}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6818","nonce":2237458793},{"type":"v1","addr":"172.21.15.119:6819","nonce":2237458793}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6822","nonce":2237458793},{"type":"v1","addr":"172.21.15.119:6823","nonce":2237458793}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6820","nonce":2237458793},{"type":"v1","addr":"172.21.15.119:6821","nonce":2237458793}]},"public_addr":"172.21.15.119:6817/2237458793","cluster_addr":"172.21.15.119:6819/2237458793","heartbeat_back_addr":"172.21.15.119:6823/2237458793","heartbeat_front_addr":"172.21.15.119:6821/2237458793","state":["exists","up"]},{"osd":8,"uuid":"3ac89686-6d76-4cef-9089-ac29e4b3bcac","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":62,"up_thru":85,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6816","nonce":1426318231},{"type":"v1","addr":"172.21.15.204:6817","nonce":1426318231}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6818","nonce":1426318231},{"type":"v1","addr":"172.21.15.204:6819","nonce":1426318231}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6822","nonce":1426318231},{"type":"v1","addr":"172.21.15.204:6823","nonce":1426318231}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6820","nonce":1426318231},{"type":"v1","addr":"172.21.15.204:6821","nonce":1426318231}]},"public_addr":"172.21.15.204:6817/1426318231","cluster_addr":"172.21.15.204:6819/1426318231","heartbeat_back_addr":"172.21.15.204:6823/1426318231","heartbeat_front_addr":"172.21.15.204:6821/1426318231","state":["exists","up"]},{"osd":9,"uuid":"a80ffe4d-1e98-460f-b046-c4754285cc5a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":69,"up_thru":88,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6826","nonce":2750635721},{"type":"v1","addr":"172.21.15.114:6827","nonce":2750635721}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6828","nonce":2750635721},{"type":"v1","addr":"172.21.15.114:6829","nonce":2750635721}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6832","nonce":2750635721},{"type":"v1","addr":"172.21.15.114:6833","nonce":2750635721}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6830","nonce":2750635721},{"type":"v1","addr":"172.21.15.114:6831","nonce":2750635721}]},"public_addr":"172.21.15.114:6827/2750635721","cluster_addr":"172.21.15.114:6829/2750635721","heartbeat_back_addr":"172.21.15.114:6833/2750635721","heartbeat_front_addr":"172.21.15.114:6831/2750635721","state":["exists","up"]},{"osd":10,"uuid":"cd75a579-bf12-4600-ac09-a8bfce772ced","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":74,"up_thru":85,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6824","nonce":1400187701},{"type":"v1","addr":"172.21.15.119:6825","nonce":1400187701}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6826","nonce":1400187701},{"type":"v1","addr":"172.21.15.119:6827","nonce":1400187701}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6830","nonce":1400187701},{"type":"v1","addr":"172.21.15.119:6831","nonce":1400187701}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6828","nonce":1400187701},{"type":"v1","addr":"172.21.15.119:6829","nonce":1400187701}]},"public_addr":"172.21.15.119:6825/1400187701","cluster_addr":"172.21.15.119:6827/1400187701","heartbeat_back_addr":"172.21.15.119:6831/1400187701","heartbeat_front_addr":"172.21.15.119:6829/1400187701","state":["exists","up"]},{"osd":11,"uuid":"53f037a5-3ec0-483a-80a1-dfee923a68e5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":82,"up_thru":88,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6824","nonce":2407298490},{"type":"v1","addr":"172.21.15.204:6825","nonce":2407298490}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6826","nonce":2407298490},{"type":"v1","addr":"172.21.15.204:6827","nonce":2407298490}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6830","nonce":2407298490},{"type":"v1","addr":"172.21.15.204:6831","nonce":2407298490}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6828","nonce":2407298490},{"type":"v1","addr":"172.21.15.204:6829","nonce":2407298490}]},"public_addr":"172.21.15.204:6825/2407298490","cluster_addr":"172.21.15.204:6827/2407298490","heartbeat_back_addr":"172.21.15.204:6831/2407298490","heartbeat_front_addr":"172.21.15.204:6829/2407298490","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-23T10:19:18.401147+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-23T10:19:42.611299+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-23T10:20:07.277367+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-23T10:20:31.208368+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-23T10:20:55.866225+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-23T10:21:19.961237+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-23T10:21:44.480783+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-23T10:22:09.173641+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-23T10:22:34.030328+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-23T10:22:59.652293+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-23T10:23:25.659673+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-23T10:23:51.168171+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"2.e","mappings":[{"from":2,"to":4}]},{"pgid":"2.1d","mappings":[{"from":8,"to":11}]},{"pgid":"3.e","mappings":[{"from":5,"to":9}]},{"pgid":"3.12","mappings":[{"from":3,"to":9}]},{"pgid":"3.17","mappings":[{"from":4,"to":6}]},{"pgid":"3.23","mappings":[{"from":4,"to":7}]},{"pgid":"3.27","mappings":[{"from":4,"to":6}]},{"pgid":"3.31","mappings":[{"from":4,"to":8}]},{"pgid":"3.38","mappings":[{"from":3,"to":9}]},{"pgid":"3.39","mappings":[{"from":5,"to":8}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.114:6801/1604678557":"2024-04-24T10:17:45.874609+0000","172.21.15.114:0/4025385854":"2024-04-24T10:17:45.874609+0000","172.21.15.114:0/2228273469":"2024-04-24T10:17:19.910348+0000","172.21.15.114:6800/2831530838":"2024-04-24T10:17:19.910348+0000","172.21.15.114:0/1333447450":"2024-04-24T10:17:45.874609+0000","172.21.15.114:0/2233026840":"2024-04-24T10:17:19.910348+0000","172.21.15.114:0/1902281545":"2024-04-24T10:17:19.910348+0000","172.21.15.114:6801/2831530838":"2024-04-24T10:17:19.910348+0000","172.21.15.114:6800/1604678557":"2024-04-24T10:17:45.874609+0000","172.21.15.114:0/1623553811":"2024-04-24T10:17:45.874609+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-23T10:26:28.443 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:28.444+0000 7f67a6a31640 1 -- 172.21.15.114:0/2101001458 >> [v2:172.21.15.114:6800/465333808,v1:172.21.15.114:6801/465333808] conn(0x7f67880a3bb0 msgr2=0x7f67880a6070 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T10:26:28.443 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:28.444+0000 7f67a6a31640 1 --2- 172.21.15.114:0/2101001458 >> [v2:172.21.15.114:6800/465333808,v1:172.21.15.114:6801/465333808] conn(0x7f67880a3bb0 0x7f67880a6070 secure :-1 s=READY pgs=233 cs=0 l=1 rev1=1 crypto rx=0x7f67940058c0 tx=0x7f679400a250 comp rx=0 tx=0).stop 2024-04-23T10:26:28.443 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:28.444+0000 7f67a6a31640 1 -- 172.21.15.114:0/2101001458 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f67a01029b0 msgr2=0x7f67a010a3c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T10:26:28.443 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:28.444+0000 7f67a6a31640 1 --2- 172.21.15.114:0/2101001458 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f67a01029b0 0x7f67a010a3c0 secure :-1 s=READY pgs=361 cs=0 l=1 rev1=1 crypto rx=0x7f67980097b0 tx=0x7f6798003c20 comp rx=0 tx=0).stop 2024-04-23T10:26:28.443 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:28.444+0000 7f67a6a31640 1 -- 172.21.15.114:0/2101001458 shutdown_connections 2024-04-23T10:26:28.443 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:28.444+0000 7f67a6a31640 1 --2- 172.21.15.114:0/2101001458 >> [v2:172.21.15.114:6800/465333808,v1:172.21.15.114:6801/465333808] conn(0x7f67880a3bb0 0x7f67880a6070 unknown :-1 s=CLOSED pgs=233 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:26:28.443 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:28.444+0000 7f67a6a31640 1 --2- 172.21.15.114:0/2101001458 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f67a01029b0 0x7f67a010a3c0 unknown :-1 s=CLOSED pgs=361 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:26:28.443 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:28.444+0000 7f67a6a31640 1 --2- 172.21.15.114:0/2101001458 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f67a0101ff0 0x7f67a0109e80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:26:28.443 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:28.444+0000 7f67a6a31640 1 --2- 172.21.15.114:0/2101001458 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f67a0100ea0 0x7f67a010b840 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:26:28.443 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:28.444+0000 7f67a6a31640 1 -- 172.21.15.114:0/2101001458 >> 172.21.15.114:0/2101001458 conn(0x7f67a00fc430 msgr2=0x7f67a0105380 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T10:26:28.443 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:28.444+0000 7f67a6a31640 1 -- 172.21.15.114:0/2101001458 shutdown_connections 2024-04-23T10:26:28.443 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:28.444+0000 7f67a6a31640 1 -- 172.21.15.114:0/2101001458 wait complete. 2024-04-23T10:26:28.454 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-04-23T10:20:11.970297+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '22', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 12, 'score_stable': 12, 'optimal_score': 0.25, 'raw_score_acting': 3, 'raw_score_stable': 3, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 2, 'pool_name': 'cephfs_metadata', 'create_time': '2024-04-23T10:24:41.218540+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '86', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.8799999952316284, 'score_stable': 1.8799999952316284, 'optimal_score': 1, 'raw_score_acting': 1.8799999952316284, 'raw_score_stable': 1.8799999952316284, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 3, 'pool_name': 'cephfs_data', 'create_time': '2024-04-23T10:24:42.341799+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '87', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-04-23T10:26:28.455 DEBUG:teuthology.orchestra.run.smithi114:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-04-23T10:26:28.492 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 23 10:26:28 smithi119 ceph-mon[32608]: from='client.? 172.21.15.114:0/197534427' entity='client.admin' cmd=[{"prefix": "osd dump"}]: dispatch 2024-04-23T10:26:28.558 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:28.557+0000 7fdb5799b640 1 -- 172.21.15.114:0/2804005460 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fdb50101fa0 msgr2=0x7fdb501023c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T10:26:28.558 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:28.557+0000 7fdb5799b640 1 --2- 172.21.15.114:0/2804005460 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fdb50101fa0 0x7fdb501023c0 secure :-1 s=READY pgs=362 cs=0 l=1 rev1=1 crypto rx=0x7fdb4c009900 tx=0x7fdb4c02a410 comp rx=0 tx=0).stop 2024-04-23T10:26:28.558 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:28.557+0000 7fdb5799b640 1 -- 172.21.15.114:0/2804005460 shutdown_connections 2024-04-23T10:26:28.558 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:28.557+0000 7fdb5799b640 1 --2- 172.21.15.114:0/2804005460 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fdb50103b50 0x7fdb5010a3e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:26:28.558 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:28.557+0000 7fdb5799b640 1 --2- 172.21.15.114:0/2804005460 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fdb50103190 0x7fdb50103610 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:26:28.558 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:28.557+0000 7fdb5799b640 1 --2- 172.21.15.114:0/2804005460 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fdb50101fa0 0x7fdb501023c0 unknown :-1 s=CLOSED pgs=362 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:26:28.558 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:28.557+0000 7fdb5799b640 1 -- 172.21.15.114:0/2804005460 >> 172.21.15.114:0/2804005460 conn(0x7fdb500fd530 msgr2=0x7fdb500ff950 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T10:26:28.558 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:28.557+0000 7fdb5799b640 1 -- 172.21.15.114:0/2804005460 shutdown_connections 2024-04-23T10:26:28.559 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:28.557+0000 7fdb5799b640 1 -- 172.21.15.114:0/2804005460 wait complete. 2024-04-23T10:26:28.559 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:28.558+0000 7fdb5799b640 1 Processor -- start 2024-04-23T10:26:28.559 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:28.558+0000 7fdb5799b640 1 -- start start 2024-04-23T10:26:28.559 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:28.558+0000 7fdb5799b640 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fdb50101fa0 0x7fdb50198260 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T10:26:28.559 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:28.558+0000 7fdb5799b640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fdb50103190 0x7fdb501987a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T10:26:28.559 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:28.558+0000 7fdb5799b640 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fdb50103b50 0x7fdb5019f810 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T10:26:28.559 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:28.558+0000 7fdb5799b640 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x7fdb5010daa0 con 0x7fdb50103190 2024-04-23T10:26:28.559 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:28.558+0000 7fdb5799b640 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 -- 0x7fdb5010d920 con 0x7fdb50103b50 2024-04-23T10:26:28.559 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:28.558+0000 7fdb5799b640 1 -- --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_getmap magic: 0 -- 0x7fdb5010dc20 con 0x7fdb50101fa0 2024-04-23T10:26:28.559 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:28.558+0000 7fdb54f36640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fdb50103190 0x7fdb501987a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T10:26:28.559 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:28.558+0000 7fdb54f36640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fdb50103190 0x7fdb501987a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.114:58280/0 (socket says 172.21.15.114:58280) 2024-04-23T10:26:28.559 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:28.558+0000 7fdb54f36640 1 -- 172.21.15.114:0/1563956103 learned_addr learned my addr 172.21.15.114:0/1563956103 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-04-23T10:26:28.559 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:28.558+0000 7fdb55737640 1 --2- 172.21.15.114:0/1563956103 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fdb50101fa0 0x7fdb50198260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T10:26:28.559 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:28.558+0000 7fdb54f36640 1 -- 172.21.15.114:0/1563956103 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fdb50101fa0 msgr2=0x7fdb50198260 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T10:26:28.559 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:28.558+0000 7fdb54f36640 1 --2- 172.21.15.114:0/1563956103 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fdb50101fa0 0x7fdb50198260 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:26:28.559 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:28.558+0000 7fdb54f36640 1 -- 172.21.15.114:0/1563956103 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fdb50103b50 msgr2=0x7fdb5019f810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T10:26:28.560 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:28.558+0000 7fdb55f38640 1 --2- 172.21.15.114:0/1563956103 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fdb50103b50 0x7fdb5019f810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T10:26:28.560 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:28.558+0000 7fdb54f36640 1 --2- 172.21.15.114:0/1563956103 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fdb50103b50 0x7fdb5019f810 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:26:28.560 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:28.558+0000 7fdb54f36640 1 -- 172.21.15.114:0/1563956103 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fdb5019fe80 con 0x7fdb50103190 2024-04-23T10:26:28.560 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:28.558+0000 7fdb55737640 1 --2- 172.21.15.114:0/1563956103 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fdb50101fa0 0x7fdb50198260 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-23T10:26:28.560 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:28.558+0000 7fdb55f38640 1 --2- 172.21.15.114:0/1563956103 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fdb50103b50 0x7fdb5019f810 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-23T10:26:28.560 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:28.558+0000 7fdb54f36640 1 --2- 172.21.15.114:0/1563956103 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fdb50103190 0x7fdb501987a0 secure :-1 s=READY pgs=363 cs=0 l=1 rev1=1 crypto rx=0x7fdb4000c890 tx=0x7fdb4000cd50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T10:26:28.560 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:28.558+0000 7fdb467fc640 1 -- 172.21.15.114:0/1563956103 <== mon.0 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdb40004220 con 0x7fdb50103190 2024-04-23T10:26:28.560 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:28.558+0000 7fdb5799b640 1 -- 172.21.15.114:0/1563956103 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fdb501a0170 con 0x7fdb50103190 2024-04-23T10:26:28.560 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:28.558+0000 7fdb467fc640 1 -- 172.21.15.114:0/1563956103 <== mon.0 v2:172.21.15.114:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fdb400043c0 con 0x7fdb50103190 2024-04-23T10:26:28.560 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:28.558+0000 7fdb467fc640 1 -- 172.21.15.114:0/1563956103 <== mon.0 v2:172.21.15.114:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdb40004a00 con 0x7fdb50103190 2024-04-23T10:26:28.560 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:28.558+0000 7fdb5799b640 1 -- 172.21.15.114:0/1563956103 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fdb501a0710 con 0x7fdb50103190 2024-04-23T10:26:28.560 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:28.559+0000 7fdb5799b640 1 -- 172.21.15.114:0/1563956103 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fdb50109430 con 0x7fdb50103190 2024-04-23T10:26:28.561 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:28.560+0000 7fdb467fc640 1 -- 172.21.15.114:0/1563956103 <== mon.0 v2:172.21.15.114:3300/0 4 ==== mgrmap(e 18) ==== 138647+0+0 (secure 0 0 0) 0x7fdb400079b0 con 0x7fdb50103190 2024-04-23T10:26:28.561 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:28.561+0000 7fdb467fc640 1 --2- 172.21.15.114:0/1563956103 >> [v2:172.21.15.114:6800/465333808,v1:172.21.15.114:6801/465333808] conn(0x7fdb3c0a3c30 0x7fdb3c0a60f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T10:26:28.561 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:28.561+0000 7fdb55737640 1 --2- 172.21.15.114:0/1563956103 >> [v2:172.21.15.114:6800/465333808,v1:172.21.15.114:6801/465333808] conn(0x7fdb3c0a3c30 0x7fdb3c0a60f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T10:26:28.561 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:28.561+0000 7fdb467fc640 1 -- 172.21.15.114:0/1563956103 <== mon.0 v2:172.21.15.114:3300/0 5 ==== osd_map(89..89 src has 1..89) ==== 8269+0+0 (secure 0 0 0) 0x7fdb400d0190 con 0x7fdb50103190 2024-04-23T10:26:28.561 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:28.561+0000 7fdb55737640 1 --2- 172.21.15.114:0/1563956103 >> [v2:172.21.15.114:6800/465333808,v1:172.21.15.114:6801/465333808] conn(0x7fdb3c0a3c30 0x7fdb3c0a60f0 secure :-1 s=READY pgs=234 cs=0 l=1 rev1=1 crypto rx=0x7fdb4c006010 tx=0x7fdb4c000bc0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T10:26:28.564 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:28.565+0000 7fdb467fc640 1 -- 172.21.15.114:0/1563956103 <== mon.0 v2:172.21.15.114:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7fdb40014030 con 0x7fdb50103190 2024-04-23T10:26:28.577 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 23 10:26:28 smithi204 ceph-mon[32649]: from='client.? 172.21.15.114:0/197534427' entity='client.admin' cmd=[{"prefix": "osd dump"}]: dispatch 2024-04-23T10:26:28.740 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:28.742+0000 7fdb5799b640 1 -- 172.21.15.114:0/1563956103 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) -- 0x7fdb50062b80 con 0x7fdb50103190 2024-04-23T10:26:28.741 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:28.742+0000 7fdb467fc640 1 -- 172.21.15.114:0/1563956103 <== mon.0 v2:172.21.15.114:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v89) ==== 93+0+10 (secure 0 0 0) 0x7fdb4009cb50 con 0x7fdb50103190 2024-04-23T10:26:28.741 INFO:teuthology.orchestra.run.smithi114.stdout:pg_num: 1 2024-04-23T10:26:28.743 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:28.744+0000 7fdb5799b640 1 -- 172.21.15.114:0/1563956103 >> [v2:172.21.15.114:6800/465333808,v1:172.21.15.114:6801/465333808] conn(0x7fdb3c0a3c30 msgr2=0x7fdb3c0a60f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T10:26:28.743 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:28.744+0000 7fdb5799b640 1 --2- 172.21.15.114:0/1563956103 >> [v2:172.21.15.114:6800/465333808,v1:172.21.15.114:6801/465333808] conn(0x7fdb3c0a3c30 0x7fdb3c0a60f0 secure :-1 s=READY pgs=234 cs=0 l=1 rev1=1 crypto rx=0x7fdb4c006010 tx=0x7fdb4c000bc0 comp rx=0 tx=0).stop 2024-04-23T10:26:28.743 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:28.744+0000 7fdb5799b640 1 -- 172.21.15.114:0/1563956103 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fdb50103190 msgr2=0x7fdb501987a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T10:26:28.743 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:28.744+0000 7fdb5799b640 1 --2- 172.21.15.114:0/1563956103 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fdb50103190 0x7fdb501987a0 secure :-1 s=READY pgs=363 cs=0 l=1 rev1=1 crypto rx=0x7fdb4000c890 tx=0x7fdb4000cd50 comp rx=0 tx=0).stop 2024-04-23T10:26:28.744 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:28.744+0000 7fdb5799b640 1 -- 172.21.15.114:0/1563956103 shutdown_connections 2024-04-23T10:26:28.744 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:28.744+0000 7fdb5799b640 1 --2- 172.21.15.114:0/1563956103 >> [v2:172.21.15.114:6800/465333808,v1:172.21.15.114:6801/465333808] conn(0x7fdb3c0a3c30 0x7fdb3c0a60f0 unknown :-1 s=CLOSED pgs=234 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:26:28.744 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:28.744+0000 7fdb5799b640 1 --2- 172.21.15.114:0/1563956103 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fdb50103b50 0x7fdb5019f810 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:26:28.744 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:28.744+0000 7fdb5799b640 1 --2- 172.21.15.114:0/1563956103 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fdb50103190 0x7fdb501987a0 unknown :-1 s=CLOSED pgs=363 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:26:28.744 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:28.744+0000 7fdb5799b640 1 --2- 172.21.15.114:0/1563956103 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fdb50101fa0 0x7fdb50198260 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:26:28.744 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:28.744+0000 7fdb5799b640 1 -- 172.21.15.114:0/1563956103 >> 172.21.15.114:0/1563956103 conn(0x7fdb500fd530 msgr2=0x7fdb50106600 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T10:26:28.744 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:28.744+0000 7fdb5799b640 1 -- 172.21.15.114:0/1563956103 shutdown_connections 2024-04-23T10:26:28.744 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:28.744+0000 7fdb5799b640 1 -- 172.21.15.114:0/1563956103 wait complete. 2024-04-23T10:26:28.754 DEBUG:teuthology.orchestra.run.smithi114:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2024-04-23T10:26:28.856 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:28.856+0000 7fe0796cf640 1 -- 172.21.15.114:0/2000368187 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fe074100ea0 msgr2=0x7fe0741012c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T10:26:28.857 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:28.856+0000 7fe0796cf640 1 --2- 172.21.15.114:0/2000368187 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fe074100ea0 0x7fe0741012c0 secure :-1 s=READY pgs=364 cs=0 l=1 rev1=1 crypto rx=0x7fe064009900 tx=0x7fe06402a410 comp rx=0 tx=0).stop 2024-04-23T10:26:28.857 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:28.856+0000 7fe0796cf640 1 -- 172.21.15.114:0/2000368187 shutdown_connections 2024-04-23T10:26:28.857 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:28.856+0000 7fe0796cf640 1 --2- 172.21.15.114:0/2000368187 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fe0741029b0 0x7fe0741092d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:26:28.857 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:28.856+0000 7fe0796cf640 1 --2- 172.21.15.114:0/2000368187 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fe074101ff0 0x7fe074102470 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:26:28.857 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:28.856+0000 7fe0796cf640 1 --2- 172.21.15.114:0/2000368187 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fe074100ea0 0x7fe0741012c0 unknown :-1 s=CLOSED pgs=364 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:26:28.857 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:28.856+0000 7fe0796cf640 1 -- 172.21.15.114:0/2000368187 >> 172.21.15.114:0/2000368187 conn(0x7fe0740fc430 msgr2=0x7fe0740fe850 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T10:26:28.857 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:28.856+0000 7fe0796cf640 1 -- 172.21.15.114:0/2000368187 shutdown_connections 2024-04-23T10:26:28.857 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:28.856+0000 7fe0796cf640 1 -- 172.21.15.114:0/2000368187 wait complete. 2024-04-23T10:26:28.857 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:28.856+0000 7fe0796cf640 1 Processor -- start 2024-04-23T10:26:28.857 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:28.857+0000 7fe0796cf640 1 -- start start 2024-04-23T10:26:28.857 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:28.857+0000 7fe0796cf640 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fe074100ea0 0x7fe074198180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T10:26:28.857 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:28.857+0000 7fe0796cf640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fe074101ff0 0x7fe0741986c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T10:26:28.857 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:28.857+0000 7fe0796cf640 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fe0741029b0 0x7fe07419f730 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T10:26:28.857 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:28.857+0000 7fe0796cf640 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x7fe07410c9d0 con 0x7fe074101ff0 2024-04-23T10:26:28.857 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:28.857+0000 7fe0796cf640 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 -- 0x7fe07410c850 con 0x7fe0741029b0 2024-04-23T10:26:28.858 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:28.857+0000 7fe0796cf640 1 -- --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_getmap magic: 0 -- 0x7fe07410cb50 con 0x7fe074100ea0 2024-04-23T10:26:28.858 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:28.857+0000 7fe0727fc640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fe074101ff0 0x7fe0741986c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T10:26:28.858 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:28.857+0000 7fe0727fc640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fe074101ff0 0x7fe0741986c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.114:58304/0 (socket says 172.21.15.114:58304) 2024-04-23T10:26:28.858 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:28.857+0000 7fe0727fc640 1 -- 172.21.15.114:0/2570275516 learned_addr learned my addr 172.21.15.114:0/2570275516 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-04-23T10:26:28.858 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:28.857+0000 7fe072ffd640 1 --2- 172.21.15.114:0/2570275516 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fe074100ea0 0x7fe074198180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T10:26:28.858 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:28.857+0000 7fe0737fe640 1 --2- 172.21.15.114:0/2570275516 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fe0741029b0 0x7fe07419f730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T10:26:28.858 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:28.857+0000 7fe0727fc640 1 -- 172.21.15.114:0/2570275516 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fe074100ea0 msgr2=0x7fe074198180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T10:26:28.858 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:28.857+0000 7fe0727fc640 1 --2- 172.21.15.114:0/2570275516 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fe074100ea0 0x7fe074198180 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:26:28.858 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:28.857+0000 7fe0727fc640 1 -- 172.21.15.114:0/2570275516 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fe0741029b0 msgr2=0x7fe07419f730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T10:26:28.858 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:28.857+0000 7fe0727fc640 1 --2- 172.21.15.114:0/2570275516 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fe0741029b0 0x7fe07419f730 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:26:28.858 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:28.857+0000 7fe0727fc640 1 -- 172.21.15.114:0/2570275516 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe07419fda0 con 0x7fe074101ff0 2024-04-23T10:26:28.858 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:28.857+0000 7fe0727fc640 1 --2- 172.21.15.114:0/2570275516 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fe074101ff0 0x7fe0741986c0 secure :-1 s=READY pgs=365 cs=0 l=1 rev1=1 crypto rx=0x7fe06800c970 tx=0x7fe06800ce30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T10:26:28.858 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:28.857+0000 7fe05bfff640 1 -- 172.21.15.114:0/2570275516 <== mon.0 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe068004220 con 0x7fe074101ff0 2024-04-23T10:26:28.858 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:28.857+0000 7fe0796cf640 1 -- 172.21.15.114:0/2570275516 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fe0741a0090 con 0x7fe074101ff0 2024-04-23T10:26:28.858 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:28.857+0000 7fe05bfff640 1 -- 172.21.15.114:0/2570275516 <== mon.0 v2:172.21.15.114:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fe0680043c0 con 0x7fe074101ff0 2024-04-23T10:26:28.858 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:28.857+0000 7fe05bfff640 1 -- 172.21.15.114:0/2570275516 <== mon.0 v2:172.21.15.114:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe06800f450 con 0x7fe074101ff0 2024-04-23T10:26:28.859 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:28.858+0000 7fe0796cf640 1 -- 172.21.15.114:0/2570275516 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fe0741a05e0 con 0x7fe074101ff0 2024-04-23T10:26:28.859 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:28.858+0000 7fe0796cf640 1 -- 172.21.15.114:0/2570275516 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fe0741a08a0 con 0x7fe074101ff0 2024-04-23T10:26:28.860 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:28.859+0000 7fe05bfff640 1 -- 172.21.15.114:0/2570275516 <== mon.0 v2:172.21.15.114:3300/0 4 ==== mgrmap(e 18) ==== 138647+0+0 (secure 0 0 0) 0x7fe068003610 con 0x7fe074101ff0 2024-04-23T10:26:28.860 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:28.860+0000 7fe05bfff640 1 --2- 172.21.15.114:0/2570275516 >> [v2:172.21.15.114:6800/465333808,v1:172.21.15.114:6801/465333808] conn(0x7fe05c0a3bb0 0x7fe05c0a6070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T10:26:28.860 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:28.860+0000 7fe072ffd640 1 --2- 172.21.15.114:0/2570275516 >> [v2:172.21.15.114:6800/465333808,v1:172.21.15.114:6801/465333808] conn(0x7fe05c0a3bb0 0x7fe05c0a6070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T10:26:28.860 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:28.860+0000 7fe05bfff640 1 -- 172.21.15.114:0/2570275516 <== mon.0 v2:172.21.15.114:3300/0 5 ==== osd_map(89..89 src has 1..89) ==== 8269+0+0 (secure 0 0 0) 0x7fe0680d01f0 con 0x7fe074101ff0 2024-04-23T10:26:28.860 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:28.860+0000 7fe072ffd640 1 --2- 172.21.15.114:0/2570275516 >> [v2:172.21.15.114:6800/465333808,v1:172.21.15.114:6801/465333808] conn(0x7fe05c0a3bb0 0x7fe05c0a6070 secure :-1 s=READY pgs=235 cs=0 l=1 rev1=1 crypto rx=0x7fe064000c00 tx=0x7fe0640314a0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T10:26:28.863 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:28.864+0000 7fe05bfff640 1 -- 172.21.15.114:0/2570275516 <== mon.0 v2:172.21.15.114:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7fe068014030 con 0x7fe074101ff0 2024-04-23T10:26:29.040 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:29.041+0000 7fe0796cf640 1 -- 172.21.15.114:0/2570275516 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) -- 0x7fe0741a0a70 con 0x7fe074101ff0 2024-04-23T10:26:29.041 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:29.042+0000 7fe05bfff640 1 -- 172.21.15.114:0/2570275516 <== mon.0 v2:172.21.15.114:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v89) ==== 104+0+11 (secure 0 0 0) 0x7fe06809c560 con 0x7fe074101ff0 2024-04-23T10:26:29.041 INFO:teuthology.orchestra.run.smithi114.stdout:pg_num: 32 2024-04-23T10:26:29.043 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:29.044+0000 7fe0796cf640 1 -- 172.21.15.114:0/2570275516 >> [v2:172.21.15.114:6800/465333808,v1:172.21.15.114:6801/465333808] conn(0x7fe05c0a3bb0 msgr2=0x7fe05c0a6070 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T10:26:29.043 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:29.044+0000 7fe0796cf640 1 --2- 172.21.15.114:0/2570275516 >> [v2:172.21.15.114:6800/465333808,v1:172.21.15.114:6801/465333808] conn(0x7fe05c0a3bb0 0x7fe05c0a6070 secure :-1 s=READY pgs=235 cs=0 l=1 rev1=1 crypto rx=0x7fe064000c00 tx=0x7fe0640314a0 comp rx=0 tx=0).stop 2024-04-23T10:26:29.043 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:29.044+0000 7fe0796cf640 1 -- 172.21.15.114:0/2570275516 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fe074101ff0 msgr2=0x7fe0741986c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T10:26:29.043 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:29.044+0000 7fe0796cf640 1 --2- 172.21.15.114:0/2570275516 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fe074101ff0 0x7fe0741986c0 secure :-1 s=READY pgs=365 cs=0 l=1 rev1=1 crypto rx=0x7fe06800c970 tx=0x7fe06800ce30 comp rx=0 tx=0).stop 2024-04-23T10:26:29.043 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:29.044+0000 7fe0796cf640 1 -- 172.21.15.114:0/2570275516 shutdown_connections 2024-04-23T10:26:29.043 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:29.044+0000 7fe0796cf640 1 --2- 172.21.15.114:0/2570275516 >> [v2:172.21.15.114:6800/465333808,v1:172.21.15.114:6801/465333808] conn(0x7fe05c0a3bb0 0x7fe05c0a6070 unknown :-1 s=CLOSED pgs=235 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:26:29.043 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:29.044+0000 7fe0796cf640 1 --2- 172.21.15.114:0/2570275516 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fe0741029b0 0x7fe07419f730 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:26:29.043 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:29.044+0000 7fe0796cf640 1 --2- 172.21.15.114:0/2570275516 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fe074101ff0 0x7fe0741986c0 unknown :-1 s=CLOSED pgs=365 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:26:29.043 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:29.044+0000 7fe0796cf640 1 --2- 172.21.15.114:0/2570275516 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fe074100ea0 0x7fe074198180 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:26:29.043 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:29.044+0000 7fe0796cf640 1 -- 172.21.15.114:0/2570275516 >> 172.21.15.114:0/2570275516 conn(0x7fe0740fc430 msgr2=0x7fe07406a590 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T10:26:29.044 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:29.044+0000 7fe0796cf640 1 -- 172.21.15.114:0/2570275516 shutdown_connections 2024-04-23T10:26:29.044 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:29.044+0000 7fe0796cf640 1 -- 172.21.15.114:0/2570275516 wait complete. 2024-04-23T10:26:29.054 DEBUG:teuthology.orchestra.run.smithi114:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2024-04-23T10:26:29.157 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:29.157+0000 7f36b7f0f640 1 -- 172.21.15.114:0/291908730 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f36b0102040 msgr2=0x7f36b01024c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T10:26:29.158 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:29.157+0000 7f36b7f0f640 1 --2- 172.21.15.114:0/291908730 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f36b0102040 0x7f36b01024c0 secure :-1 s=READY pgs=366 cs=0 l=1 rev1=1 crypto rx=0x7f36a4009900 tx=0x7f36a402a410 comp rx=0 tx=0).stop 2024-04-23T10:26:29.158 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:29.157+0000 7f36b7f0f640 1 -- 172.21.15.114:0/291908730 shutdown_connections 2024-04-23T10:26:29.158 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:29.157+0000 7f36b7f0f640 1 --2- 172.21.15.114:0/291908730 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f36b0102a00 0x7f36b0109320 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:26:29.158 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:29.157+0000 7f36b7f0f640 1 --2- 172.21.15.114:0/291908730 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f36b0102040 0x7f36b01024c0 unknown :-1 s=CLOSED pgs=366 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:26:29.158 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:29.157+0000 7f36b7f0f640 1 --2- 172.21.15.114:0/291908730 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f36b0100e60 0x7f36b0101280 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:26:29.158 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:29.157+0000 7f36b7f0f640 1 -- 172.21.15.114:0/291908730 >> 172.21.15.114:0/291908730 conn(0x7f36b00fc440 msgr2=0x7f36b00fe860 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T10:26:29.158 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:29.157+0000 7f36b7f0f640 1 -- 172.21.15.114:0/291908730 shutdown_connections 2024-04-23T10:26:29.158 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:29.157+0000 7f36b7f0f640 1 -- 172.21.15.114:0/291908730 wait complete. 2024-04-23T10:26:29.158 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:29.157+0000 7f36b7f0f640 1 Processor -- start 2024-04-23T10:26:29.158 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:29.157+0000 7f36b7f0f640 1 -- start start 2024-04-23T10:26:29.158 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:29.158+0000 7f36b7f0f640 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f36b0100e60 0x7f36b006b0b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T10:26:29.158 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:29.158+0000 7f36b7f0f640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f36b0102040 0x7f36b019cf30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T10:26:29.159 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:29.158+0000 7f36b7f0f640 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f36b0102a00 0x7f36b019f570 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T10:26:29.159 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:29.158+0000 7f36b7f0f640 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x7f36b010ca50 con 0x7f36b0102040 2024-04-23T10:26:29.159 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:29.158+0000 7f36b7f0f640 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 -- 0x7f36b010c8d0 con 0x7f36b0102a00 2024-04-23T10:26:29.159 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:29.158+0000 7f36b7f0f640 1 -- --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_getmap magic: 0 -- 0x7f36b010cbd0 con 0x7f36b0100e60 2024-04-23T10:26:29.159 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:29.158+0000 7f36b54aa640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f36b0102040 0x7f36b019cf30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T10:26:29.159 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:29.158+0000 7f36b54aa640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f36b0102040 0x7f36b019cf30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.114:58312/0 (socket says 172.21.15.114:58312) 2024-04-23T10:26:29.159 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:29.158+0000 7f36b54aa640 1 -- 172.21.15.114:0/2539925690 learned_addr learned my addr 172.21.15.114:0/2539925690 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-04-23T10:26:29.159 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:29.158+0000 7f36b54aa640 1 -- 172.21.15.114:0/2539925690 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f36b0100e60 msgr2=0x7f36b006b0b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T10:26:29.159 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:29.158+0000 7f36b54aa640 1 --2- 172.21.15.114:0/2539925690 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f36b0100e60 0x7f36b006b0b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:26:29.159 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:29.158+0000 7f36b54aa640 1 -- 172.21.15.114:0/2539925690 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f36b0102a00 msgr2=0x7f36b019f570 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T10:26:29.159 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:29.158+0000 7f36b54aa640 1 --2- 172.21.15.114:0/2539925690 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f36b0102a00 0x7f36b019f570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:26:29.159 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:29.158+0000 7f36b54aa640 1 -- 172.21.15.114:0/2539925690 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f36b019fe30 con 0x7f36b0102040 2024-04-23T10:26:29.159 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:29.158+0000 7f36b54aa640 1 --2- 172.21.15.114:0/2539925690 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f36b0102040 0x7f36b019cf30 secure :-1 s=READY pgs=367 cs=0 l=1 rev1=1 crypto rx=0x7f36a4005ab0 tx=0x7f36a4003cd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T10:26:29.159 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:29.158+0000 7f36aeffd640 1 -- 172.21.15.114:0/2539925690 <== mon.0 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f36a403f070 con 0x7f36b0102040 2024-04-23T10:26:29.159 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:29.158+0000 7f36aeffd640 1 -- 172.21.15.114:0/2539925690 <== mon.0 v2:172.21.15.114:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f36a403b460 con 0x7f36b0102040 2024-04-23T10:26:29.160 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:29.158+0000 7f36b7f0f640 1 -- 172.21.15.114:0/2539925690 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f36b01a00c0 con 0x7f36b0102040 2024-04-23T10:26:29.160 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:29.158+0000 7f36aeffd640 1 -- 172.21.15.114:0/2539925690 <== mon.0 v2:172.21.15.114:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f36a4035030 con 0x7f36b0102040 2024-04-23T10:26:29.160 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:29.158+0000 7f36b7f0f640 1 -- 172.21.15.114:0/2539925690 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f36b01a05b0 con 0x7f36b0102040 2024-04-23T10:26:29.160 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:29.159+0000 7f36b7f0f640 1 -- 172.21.15.114:0/2539925690 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f36b0108200 con 0x7f36b0102040 2024-04-23T10:26:29.161 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:29.160+0000 7f36aeffd640 1 -- 172.21.15.114:0/2539925690 <== mon.0 v2:172.21.15.114:3300/0 4 ==== mgrmap(e 18) ==== 138647+0+0 (secure 0 0 0) 0x7f36a4042070 con 0x7f36b0102040 2024-04-23T10:26:29.161 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:29.161+0000 7f36aeffd640 1 --2- 172.21.15.114:0/2539925690 >> [v2:172.21.15.114:6800/465333808,v1:172.21.15.114:6801/465333808] conn(0x7f36980a3bb0 0x7f36980a6070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T10:26:29.161 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:29.161+0000 7f36b5cab640 1 --2- 172.21.15.114:0/2539925690 >> [v2:172.21.15.114:6800/465333808,v1:172.21.15.114:6801/465333808] conn(0x7f36980a3bb0 0x7f36980a6070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T10:26:29.161 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:29.161+0000 7f36aeffd640 1 -- 172.21.15.114:0/2539925690 <== mon.0 v2:172.21.15.114:3300/0 5 ==== osd_map(89..89 src has 1..89) ==== 8269+0+0 (secure 0 0 0) 0x7f36a40f11d0 con 0x7f36b0102040 2024-04-23T10:26:29.161 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:29.161+0000 7f36b5cab640 1 --2- 172.21.15.114:0/2539925690 >> [v2:172.21.15.114:6800/465333808,v1:172.21.15.114:6801/465333808] conn(0x7f36980a3bb0 0x7f36980a6070 secure :-1 s=READY pgs=236 cs=0 l=1 rev1=1 crypto rx=0x7f36b0101bb0 tx=0x7f36a80093c0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T10:26:29.164 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:29.165+0000 7f36aeffd640 1 -- 172.21.15.114:0/2539925690 <== mon.0 v2:172.21.15.114:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f36a40bdb70 con 0x7f36b0102040 2024-04-23T10:26:29.342 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:29.344+0000 7f36b7f0f640 1 -- 172.21.15.114:0/2539925690 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) -- 0x7f36b002d5c0 con 0x7f36b0102040 2024-04-23T10:26:29.343 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:29.344+0000 7f36aeffd640 1 -- 172.21.15.114:0/2539925690 <== mon.0 v2:172.21.15.114:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v89) ==== 100+0+11 (secure 0 0 0) 0x7f36a40bd470 con 0x7f36b0102040 2024-04-23T10:26:29.343 INFO:teuthology.orchestra.run.smithi114.stdout:pg_num: 64 2024-04-23T10:26:29.345 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:29.346+0000 7f36b7f0f640 1 -- 172.21.15.114:0/2539925690 >> [v2:172.21.15.114:6800/465333808,v1:172.21.15.114:6801/465333808] conn(0x7f36980a3bb0 msgr2=0x7f36980a6070 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T10:26:29.345 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:29.346+0000 7f36b7f0f640 1 --2- 172.21.15.114:0/2539925690 >> [v2:172.21.15.114:6800/465333808,v1:172.21.15.114:6801/465333808] conn(0x7f36980a3bb0 0x7f36980a6070 secure :-1 s=READY pgs=236 cs=0 l=1 rev1=1 crypto rx=0x7f36b0101bb0 tx=0x7f36a80093c0 comp rx=0 tx=0).stop 2024-04-23T10:26:29.345 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:29.346+0000 7f36b7f0f640 1 -- 172.21.15.114:0/2539925690 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f36b0102040 msgr2=0x7f36b019cf30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T10:26:29.345 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:29.346+0000 7f36b7f0f640 1 --2- 172.21.15.114:0/2539925690 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f36b0102040 0x7f36b019cf30 secure :-1 s=READY pgs=367 cs=0 l=1 rev1=1 crypto rx=0x7f36a4005ab0 tx=0x7f36a4003cd0 comp rx=0 tx=0).stop 2024-04-23T10:26:29.345 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:29.346+0000 7f36b7f0f640 1 -- 172.21.15.114:0/2539925690 shutdown_connections 2024-04-23T10:26:29.346 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:29.346+0000 7f36b7f0f640 1 --2- 172.21.15.114:0/2539925690 >> [v2:172.21.15.114:6800/465333808,v1:172.21.15.114:6801/465333808] conn(0x7f36980a3bb0 0x7f36980a6070 unknown :-1 s=CLOSED pgs=236 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:26:29.346 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:29.346+0000 7f36b7f0f640 1 --2- 172.21.15.114:0/2539925690 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f36b0102a00 0x7f36b019f570 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:26:29.346 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:29.346+0000 7f36b7f0f640 1 --2- 172.21.15.114:0/2539925690 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f36b0102040 0x7f36b019cf30 unknown :-1 s=CLOSED pgs=367 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:26:29.346 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:29.346+0000 7f36b7f0f640 1 --2- 172.21.15.114:0/2539925690 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f36b0100e60 0x7f36b006b0b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:26:29.346 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:29.346+0000 7f36b7f0f640 1 -- 172.21.15.114:0/2539925690 >> 172.21.15.114:0/2539925690 conn(0x7f36b00fc440 msgr2=0x7f36b01053d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T10:26:29.346 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:29.346+0000 7f36b7f0f640 1 -- 172.21.15.114:0/2539925690 shutdown_connections 2024-04-23T10:26:29.346 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:29.346+0000 7f36b7f0f640 1 -- 172.21.15.114:0/2539925690 wait complete. 2024-04-23T10:26:29.355 INFO:journalctl@ceph.mon.a.smithi114.stdout:Apr 23 10:26:29 smithi114 ceph-mon[29255]: pgmap v340: 97 pgs: 97 active+clean; 641 KiB data, 375 MiB used, 1.0 TiB / 1.0 TiB avail; 3.6 KiB/s rd, 597 B/s wr, 7 op/s 2024-04-23T10:26:29.355 INFO:journalctl@ceph.mon.a.smithi114.stdout:Apr 23 10:26:29 smithi114 ceph-mon[29255]: from='client.? 172.21.15.114:0/2101001458' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-23T10:26:29.355 INFO:journalctl@ceph.mon.a.smithi114.stdout:Apr 23 10:26:29 smithi114 ceph-mon[29255]: from='client.? 172.21.15.114:0/1563956103' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-04-23T10:26:29.355 INFO:journalctl@ceph.mon.a.smithi114.stdout:Apr 23 10:26:29 smithi114 ceph-mon[29255]: from='client.? 172.21.15.114:0/2570275516' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2024-04-23T10:26:29.356 DEBUG:teuthology.orchestra.run.smithi114:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-23T10:26:29.455 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:29.454+0000 7fe5cbe4e640 1 -- 172.21.15.114:0/2281338457 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fe5c4100800 msgr2=0x7fe5c4102c30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T10:26:29.455 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:29.454+0000 7fe5cbe4e640 1 --2- 172.21.15.114:0/2281338457 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fe5c4100800 0x7fe5c4102c30 secure :-1 s=READY pgs=368 cs=0 l=1 rev1=1 crypto rx=0x7fe5c0009900 tx=0x7fe5c002a410 comp rx=0 tx=0).stop 2024-04-23T10:26:29.455 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:29.454+0000 7fe5cbe4e640 1 -- 172.21.15.114:0/2281338457 shutdown_connections 2024-04-23T10:26:29.455 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:29.454+0000 7fe5cbe4e640 1 --2- 172.21.15.114:0/2281338457 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fe5c4105b20 0x7fe5c4105fa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:26:29.455 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:29.454+0000 7fe5cbe4e640 1 --2- 172.21.15.114:0/2281338457 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fe5c4103170 0x7fe5c41055e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:26:29.455 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:29.454+0000 7fe5cbe4e640 1 --2- 172.21.15.114:0/2281338457 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fe5c4100800 0x7fe5c4102c30 unknown :-1 s=CLOSED pgs=368 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:26:29.456 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:29.454+0000 7fe5cbe4e640 1 -- 172.21.15.114:0/2281338457 >> 172.21.15.114:0/2281338457 conn(0x7fe5c40fc400 msgr2=0x7fe5c40fe840 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T10:26:29.456 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:29.454+0000 7fe5cbe4e640 1 -- 172.21.15.114:0/2281338457 shutdown_connections 2024-04-23T10:26:29.456 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:29.454+0000 7fe5cbe4e640 1 -- 172.21.15.114:0/2281338457 wait complete. 2024-04-23T10:26:29.456 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:29.455+0000 7fe5cbe4e640 1 Processor -- start 2024-04-23T10:26:29.456 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:29.455+0000 7fe5cbe4e640 1 -- start start 2024-04-23T10:26:29.456 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:29.455+0000 7fe5cbe4e640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fe5c4100800 0x7fe5c419e730 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T10:26:29.456 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:29.455+0000 7fe5cbe4e640 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fe5c4103170 0x7fe5c419ec70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T10:26:29.456 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:29.455+0000 7fe5cbe4e640 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fe5c4105b20 0x7fe5c41a5c70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T10:26:29.456 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:29.455+0000 7fe5cbe4e640 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x7fe5c4114080 con 0x7fe5c4100800 2024-04-23T10:26:29.456 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:29.455+0000 7fe5cbe4e640 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 -- 0x7fe5c4113f00 con 0x7fe5c4103170 2024-04-23T10:26:29.456 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:29.455+0000 7fe5cbe4e640 1 -- --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_getmap magic: 0 -- 0x7fe5c4114200 con 0x7fe5c4105b20 2024-04-23T10:26:29.456 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:29.455+0000 7fe5c9bea640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fe5c4100800 0x7fe5c419e730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T10:26:29.456 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:29.455+0000 7fe5c9bea640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fe5c4100800 0x7fe5c419e730 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.114:58320/0 (socket says 172.21.15.114:58320) 2024-04-23T10:26:29.457 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:29.455+0000 7fe5c9bea640 1 -- 172.21.15.114:0/2847980599 learned_addr learned my addr 172.21.15.114:0/2847980599 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-04-23T10:26:29.457 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:29.455+0000 7fe5c93e9640 1 --2- 172.21.15.114:0/2847980599 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fe5c4103170 0x7fe5c419ec70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T10:26:29.457 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:29.455+0000 7fe5c9bea640 1 -- 172.21.15.114:0/2847980599 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fe5c4105b20 msgr2=0x7fe5c41a5c70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T10:26:29.457 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:29.455+0000 7fe5c9bea640 1 --2- 172.21.15.114:0/2847980599 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fe5c4105b20 0x7fe5c41a5c70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:26:29.457 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:29.455+0000 7fe5c9bea640 1 -- 172.21.15.114:0/2847980599 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fe5c4103170 msgr2=0x7fe5c419ec70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T10:26:29.457 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:29.455+0000 7fe5c9bea640 1 --2- 172.21.15.114:0/2847980599 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fe5c4103170 0x7fe5c419ec70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:26:29.457 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:29.455+0000 7fe5c9bea640 1 -- 172.21.15.114:0/2847980599 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe5c41a62e0 con 0x7fe5c4100800 2024-04-23T10:26:29.457 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:29.456+0000 7fe5c9bea640 1 --2- 172.21.15.114:0/2847980599 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fe5c4100800 0x7fe5c419e730 secure :-1 s=READY pgs=369 cs=0 l=1 rev1=1 crypto rx=0x7fe5c0031500 tx=0x7fe5c00043c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T10:26:29.457 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:29.456+0000 7fe5baffd640 1 -- 172.21.15.114:0/2847980599 <== mon.0 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe5c003f030 con 0x7fe5c4100800 2024-04-23T10:26:29.457 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:29.456+0000 7fe5baffd640 1 -- 172.21.15.114:0/2847980599 <== mon.0 v2:172.21.15.114:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fe5c0003800 con 0x7fe5c4100800 2024-04-23T10:26:29.457 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:29.456+0000 7fe5baffd640 1 -- 172.21.15.114:0/2847980599 <== mon.0 v2:172.21.15.114:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe5c0035070 con 0x7fe5c4100800 2024-04-23T10:26:29.457 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:29.456+0000 7fe5cbe4e640 1 -- 172.21.15.114:0/2847980599 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fe5c41a6570 con 0x7fe5c4100800 2024-04-23T10:26:29.457 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:29.456+0000 7fe5cbe4e640 1 -- 172.21.15.114:0/2847980599 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fe5c41a6ab0 con 0x7fe5c4100800 2024-04-23T10:26:29.458 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:29.457+0000 7fe5cbe4e640 1 -- 172.21.15.114:0/2847980599 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fe5c40fdef0 con 0x7fe5c4100800 2024-04-23T10:26:29.458 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:29.458+0000 7fe5baffd640 1 -- 172.21.15.114:0/2847980599 <== mon.0 v2:172.21.15.114:3300/0 4 ==== mgrmap(e 18) ==== 138647+0+0 (secure 0 0 0) 0x7fe5c00039b0 con 0x7fe5c4100800 2024-04-23T10:26:29.458 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:29.458+0000 7fe5baffd640 1 --2- 172.21.15.114:0/2847980599 >> [v2:172.21.15.114:6800/465333808,v1:172.21.15.114:6801/465333808] conn(0x7fe5b00a3c30 0x7fe5b00a60f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T10:26:29.459 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:29.458+0000 7fe5c93e9640 1 --2- 172.21.15.114:0/2847980599 >> [v2:172.21.15.114:6800/465333808,v1:172.21.15.114:6801/465333808] conn(0x7fe5b00a3c30 0x7fe5b00a60f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T10:26:29.459 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:29.458+0000 7fe5baffd640 1 -- 172.21.15.114:0/2847980599 <== mon.0 v2:172.21.15.114:3300/0 5 ==== osd_map(89..89 src has 1..89) ==== 8269+0+0 (secure 0 0 0) 0x7fe5c00f0db0 con 0x7fe5c4100800 2024-04-23T10:26:29.459 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:29.459+0000 7fe5c93e9640 1 --2- 172.21.15.114:0/2847980599 >> [v2:172.21.15.114:6800/465333808,v1:172.21.15.114:6801/465333808] conn(0x7fe5b00a3c30 0x7fe5b00a60f0 secure :-1 s=READY pgs=237 cs=0 l=1 rev1=1 crypto rx=0x7fe5c419fbd0 tx=0x7fe5b400a2d0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T10:26:29.461 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:29.463+0000 7fe5baffd640 1 -- 172.21.15.114:0/2847980599 <== mon.0 v2:172.21.15.114:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7fe5c00bd770 con 0x7fe5c4100800 2024-04-23T10:26:29.492 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 23 10:26:29 smithi119 ceph-mon[32608]: pgmap v340: 97 pgs: 97 active+clean; 641 KiB data, 375 MiB used, 1.0 TiB / 1.0 TiB avail; 3.6 KiB/s rd, 597 B/s wr, 7 op/s 2024-04-23T10:26:29.492 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 23 10:26:29 smithi119 ceph-mon[32608]: from='client.? 172.21.15.114:0/2101001458' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-23T10:26:29.492 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 23 10:26:29 smithi119 ceph-mon[32608]: from='client.? 172.21.15.114:0/1563956103' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-04-23T10:26:29.492 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 23 10:26:29 smithi119 ceph-mon[32608]: from='client.? 172.21.15.114:0/2570275516' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2024-04-23T10:26:29.578 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 23 10:26:29 smithi204 ceph-mon[32649]: pgmap v340: 97 pgs: 97 active+clean; 641 KiB data, 375 MiB used, 1.0 TiB / 1.0 TiB avail; 3.6 KiB/s rd, 597 B/s wr, 7 op/s 2024-04-23T10:26:29.578 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 23 10:26:29 smithi204 ceph-mon[32649]: from='client.? 172.21.15.114:0/2101001458' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-23T10:26:29.578 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 23 10:26:29 smithi204 ceph-mon[32649]: from='client.? 172.21.15.114:0/1563956103' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-04-23T10:26:29.578 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 23 10:26:29 smithi204 ceph-mon[32649]: from='client.? 172.21.15.114:0/2570275516' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2024-04-23T10:26:29.679 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:29.681+0000 7fe5cbe4e640 1 -- 172.21.15.114:0/2847980599 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fe5c4062b50 con 0x7fe5c4100800 2024-04-23T10:26:29.680 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:29.682+0000 7fe5baffd640 1 -- 172.21.15.114:0/2847980599 <== mon.0 v2:172.21.15.114:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 20 v20) ==== 94+0+11884 (secure 0 0 0) 0x7fe5c00bd070 con 0x7fe5c4100800 2024-04-23T10:26:29.680 INFO:teuthology.orchestra.run.smithi114.stdout: 2024-04-23T10:26:29.681 INFO:teuthology.orchestra.run.smithi114.stdout:{"epoch":20,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14547,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.114:6835/663493580","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6834","nonce":663493580},{"type":"v1","addr":"172.21.15.114:6835","nonce":663493580}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14565,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.114:6837/1439252154","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6836","nonce":1439252154},{"type":"v1","addr":"172.21.15.114:6837","nonce":1439252154}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5},{"gid":14634,"name":"g","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.114:6839/3849796939","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6838","nonce":3849796939},{"type":"v1","addr":"172.21.15.114:6839","nonce":3849796939}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":11},{"gid":14646,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.114:6841/3544523891","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6840","nonce":3544523891},{"type":"v1","addr":"172.21.15.114:6841","nonce":3544523891}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":12},{"gid":24385,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.204:6833/3866561754","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6832","nonce":3866561754},{"type":"v1","addr":"172.21.15.204:6833","nonce":3866561754}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2},{"gid":24427,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.204:6835/2533793172","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6834","nonce":2533793172},{"type":"v1","addr":"172.21.15.204:6835","nonce":2533793172}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":7}],"filesystems":[{"mdsmap":{"epoch":20,"flags":50,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-04-23T10:24:43.383158+0000","modified":"2024-04-23T10:24:48.315128+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":3,"in":[0,1,2],"up":{"mds_0":24545,"mds_1":24509,"mds_2":24479},"failed":[],"damaged":[],"stopped":[],"info":{"gid_24439":{"gid":24439,"name":"c","rank":2,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.204:6837/513822179","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6836","nonce":513822179},{"type":"v1","addr":"172.21.15.204:6837","nonce":513822179}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24443":{"gid":24443,"name":"k","rank":1,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.119:6833/783244660","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6832","nonce":783244660},{"type":"v1","addr":"172.21.15.119:6833","nonce":783244660}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24463":{"gid":24463,"name":"i","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.204:6839/3222658893","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6838","nonce":3222658893},{"type":"v1","addr":"172.21.15.204:6839","nonce":3222658893}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24479":{"gid":24479,"name":"e","rank":2,"incarnation":19,"state":"up:active","state_seq":9,"addr":"172.21.15.119:6835/2033916709","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6834","nonce":2033916709},{"type":"v1","addr":"172.21.15.119:6835","nonce":2033916709}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24509":{"gid":24509,"name":"h","rank":1,"incarnation":17,"state":"up:active","state_seq":6,"addr":"172.21.15.119:6837/1872363128","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6836","nonce":1872363128},{"type":"v1","addr":"172.21.15.119:6837","nonce":1872363128}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24545":{"gid":24545,"name":"b","rank":0,"incarnation":14,"state":"up:active","state_seq":3,"addr":"172.21.15.119:6839/937297340","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6838","nonce":937297340},{"type":"v1","addr":"172.21.15.119:6839","nonce":937297340}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":24545,"qdb_cluster":[24509,24545,24479]},"id":1}]} 2024-04-23T10:26:29.681 INFO:teuthology.orchestra.run.smithi114.stderr:dumped fsmap epoch 20 2024-04-23T10:26:29.683 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:29.683+0000 7fe5cbe4e640 1 -- 172.21.15.114:0/2847980599 >> [v2:172.21.15.114:6800/465333808,v1:172.21.15.114:6801/465333808] conn(0x7fe5b00a3c30 msgr2=0x7fe5b00a60f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T10:26:29.683 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:29.683+0000 7fe5cbe4e640 1 --2- 172.21.15.114:0/2847980599 >> [v2:172.21.15.114:6800/465333808,v1:172.21.15.114:6801/465333808] conn(0x7fe5b00a3c30 0x7fe5b00a60f0 secure :-1 s=READY pgs=237 cs=0 l=1 rev1=1 crypto rx=0x7fe5c419fbd0 tx=0x7fe5b400a2d0 comp rx=0 tx=0).stop 2024-04-23T10:26:29.683 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:29.683+0000 7fe5cbe4e640 1 -- 172.21.15.114:0/2847980599 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fe5c4100800 msgr2=0x7fe5c419e730 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T10:26:29.683 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:29.683+0000 7fe5cbe4e640 1 --2- 172.21.15.114:0/2847980599 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fe5c4100800 0x7fe5c419e730 secure :-1 s=READY pgs=369 cs=0 l=1 rev1=1 crypto rx=0x7fe5c0031500 tx=0x7fe5c00043c0 comp rx=0 tx=0).stop 2024-04-23T10:26:29.683 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:29.684+0000 7fe5cbe4e640 1 -- 172.21.15.114:0/2847980599 shutdown_connections 2024-04-23T10:26:29.683 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:29.684+0000 7fe5cbe4e640 1 --2- 172.21.15.114:0/2847980599 >> [v2:172.21.15.114:6800/465333808,v1:172.21.15.114:6801/465333808] conn(0x7fe5b00a3c30 0x7fe5b00a60f0 unknown :-1 s=CLOSED pgs=237 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:26:29.683 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:29.684+0000 7fe5cbe4e640 1 --2- 172.21.15.114:0/2847980599 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fe5c4105b20 0x7fe5c41a5c70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:26:29.683 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:29.684+0000 7fe5cbe4e640 1 --2- 172.21.15.114:0/2847980599 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fe5c4103170 0x7fe5c419ec70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:26:29.683 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:29.684+0000 7fe5cbe4e640 1 --2- 172.21.15.114:0/2847980599 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fe5c4100800 0x7fe5c419e730 unknown :-1 s=CLOSED pgs=369 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:26:29.683 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:29.684+0000 7fe5cbe4e640 1 -- 172.21.15.114:0/2847980599 >> 172.21.15.114:0/2847980599 conn(0x7fe5c40fc400 msgr2=0x7fe5c4103b90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T10:26:29.683 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:29.684+0000 7fe5cbe4e640 1 -- 172.21.15.114:0/2847980599 shutdown_connections 2024-04-23T10:26:29.683 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:29.684+0000 7fe5cbe4e640 1 -- 172.21.15.114:0/2847980599 wait complete. 2024-04-23T10:26:29.694 DEBUG:teuthology.orchestra.run.smithi114:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-04-23T10:26:29.793 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:29.792+0000 7f0943d30640 1 -- 172.21.15.114:0/70735459 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f093c102040 msgr2=0x7f093c1024c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T10:26:29.793 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:29.792+0000 7f0943d30640 1 --2- 172.21.15.114:0/70735459 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f093c102040 0x7f093c1024c0 secure :-1 s=READY pgs=370 cs=0 l=1 rev1=1 crypto rx=0x7f092c009900 tx=0x7f092c02a410 comp rx=0 tx=0).stop 2024-04-23T10:26:29.793 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:29.792+0000 7f0943d30640 1 -- 172.21.15.114:0/70735459 shutdown_connections 2024-04-23T10:26:29.793 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:29.792+0000 7f0943d30640 1 --2- 172.21.15.114:0/70735459 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f093c102a00 0x7f093c109320 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:26:29.793 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:29.792+0000 7f0943d30640 1 --2- 172.21.15.114:0/70735459 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f093c102040 0x7f093c1024c0 unknown :-1 s=CLOSED pgs=370 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:26:29.793 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:29.792+0000 7f0943d30640 1 --2- 172.21.15.114:0/70735459 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f093c100e60 0x7f093c101280 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:26:29.793 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:29.792+0000 7f0943d30640 1 -- 172.21.15.114:0/70735459 >> 172.21.15.114:0/70735459 conn(0x7f093c0fc440 msgr2=0x7f093c0fe860 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T10:26:29.793 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:29.792+0000 7f0943d30640 1 -- 172.21.15.114:0/70735459 shutdown_connections 2024-04-23T10:26:29.794 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:29.792+0000 7f0943d30640 1 -- 172.21.15.114:0/70735459 wait complete. 2024-04-23T10:26:29.794 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:29.793+0000 7f0943d30640 1 Processor -- start 2024-04-23T10:26:29.794 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:29.793+0000 7f0943d30640 1 -- start start 2024-04-23T10:26:29.794 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:29.793+0000 7f0943d30640 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f093c100e60 0x7f093c19a400 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T10:26:29.794 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:29.793+0000 7f0943d30640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f093c102040 0x7f093c19a940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T10:26:29.794 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:29.793+0000 7f0943d30640 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f093c102a00 0x7f093c1a19b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T10:26:29.794 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:29.793+0000 7f0943d30640 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x7f093c10ca10 con 0x7f093c102040 2024-04-23T10:26:29.794 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:29.793+0000 7f0943d30640 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 -- 0x7f093c10c890 con 0x7f093c100e60 2024-04-23T10:26:29.794 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:29.793+0000 7f0943d30640 1 -- --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_getmap magic: 0 -- 0x7f093c10cb90 con 0x7f093c102a00 2024-04-23T10:26:29.794 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:29.793+0000 7f09412cb640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f093c102040 0x7f093c19a940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T10:26:29.794 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:29.793+0000 7f09412cb640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f093c102040 0x7f093c19a940 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.114:58326/0 (socket says 172.21.15.114:58326) 2024-04-23T10:26:29.794 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:29.793+0000 7f09412cb640 1 -- 172.21.15.114:0/1893883747 learned_addr learned my addr 172.21.15.114:0/1893883747 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-04-23T10:26:29.794 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:29.793+0000 7f0941acc640 1 --2- 172.21.15.114:0/1893883747 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f093c100e60 0x7f093c19a400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T10:26:29.794 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:29.793+0000 7f09422cd640 1 --2- 172.21.15.114:0/1893883747 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f093c102a00 0x7f093c1a19b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T10:26:29.794 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:29.793+0000 7f09412cb640 1 -- 172.21.15.114:0/1893883747 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f093c102a00 msgr2=0x7f093c1a19b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T10:26:29.795 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:29.793+0000 7f09412cb640 1 --2- 172.21.15.114:0/1893883747 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f093c102a00 0x7f093c1a19b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:26:29.795 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:29.793+0000 7f09412cb640 1 -- 172.21.15.114:0/1893883747 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f093c100e60 msgr2=0x7f093c19a400 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T10:26:29.795 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:29.793+0000 7f09412cb640 1 --2- 172.21.15.114:0/1893883747 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f093c100e60 0x7f093c19a400 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:26:29.795 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:29.793+0000 7f09412cb640 1 -- 172.21.15.114:0/1893883747 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f093c1a2020 con 0x7f093c102040 2024-04-23T10:26:29.795 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:29.793+0000 7f0941acc640 1 --2- 172.21.15.114:0/1893883747 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f093c100e60 0x7f093c19a400 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-23T10:26:29.795 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:29.793+0000 7f09412cb640 1 --2- 172.21.15.114:0/1893883747 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f093c102040 0x7f093c19a940 secure :-1 s=READY pgs=371 cs=0 l=1 rev1=1 crypto rx=0x7f092c005940 tx=0x7f092c004430 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T10:26:29.795 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:29.793+0000 7f0932ffd640 1 -- 172.21.15.114:0/1893883747 <== mon.0 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f092c03f030 con 0x7f093c102040 2024-04-23T10:26:29.795 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:29.793+0000 7f0943d30640 1 -- 172.21.15.114:0/1893883747 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f093c1a22b0 con 0x7f093c102040 2024-04-23T10:26:29.795 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:29.793+0000 7f0932ffd640 1 -- 172.21.15.114:0/1893883747 <== mon.0 v2:172.21.15.114:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f092c003cb0 con 0x7f093c102040 2024-04-23T10:26:29.795 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:29.793+0000 7f0932ffd640 1 -- 172.21.15.114:0/1893883747 <== mon.0 v2:172.21.15.114:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f092c035070 con 0x7f093c102040 2024-04-23T10:26:29.795 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:29.794+0000 7f0943d30640 1 -- 172.21.15.114:0/1893883747 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f093c1a27f0 con 0x7f093c102040 2024-04-23T10:26:29.795 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:29.794+0000 7f0943d30640 1 -- 172.21.15.114:0/1893883747 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f093c1a2a50 con 0x7f093c102040 2024-04-23T10:26:29.796 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:29.795+0000 7f0932ffd640 1 -- 172.21.15.114:0/1893883747 <== mon.0 v2:172.21.15.114:3300/0 4 ==== mgrmap(e 18) ==== 138647+0+0 (secure 0 0 0) 0x7f092c042030 con 0x7f093c102040 2024-04-23T10:26:29.796 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:29.796+0000 7f0932ffd640 1 --2- 172.21.15.114:0/1893883747 >> [v2:172.21.15.114:6800/465333808,v1:172.21.15.114:6801/465333808] conn(0x7f09280a3bb0 0x7f09280a6070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T10:26:29.796 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:29.796+0000 7f0941acc640 1 --2- 172.21.15.114:0/1893883747 >> [v2:172.21.15.114:6800/465333808,v1:172.21.15.114:6801/465333808] conn(0x7f09280a3bb0 0x7f09280a6070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T10:26:29.796 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:29.796+0000 7f0932ffd640 1 -- 172.21.15.114:0/1893883747 <== mon.0 v2:172.21.15.114:3300/0 5 ==== osd_map(89..89 src has 1..89) ==== 8269+0+0 (secure 0 0 0) 0x7f092c0f0e20 con 0x7f093c102040 2024-04-23T10:26:29.796 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:29.796+0000 7f0941acc640 1 --2- 172.21.15.114:0/1893883747 >> [v2:172.21.15.114:6800/465333808,v1:172.21.15.114:6801/465333808] conn(0x7f09280a3bb0 0x7f09280a6070 secure :-1 s=READY pgs=238 cs=0 l=1 rev1=1 crypto rx=0x7f09380058c0 tx=0x7f0938005850 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T10:26:29.799 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:29.800+0000 7f0932ffd640 1 -- 172.21.15.114:0/1893883747 <== mon.0 v2:172.21.15.114:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f092c003e60 con 0x7f093c102040 2024-04-23T10:26:29.976 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:29.977+0000 7f0943d30640 1 -- 172.21.15.114:0/1893883747 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f093c1a3120 con 0x7f093c102040 2024-04-23T10:26:29.978 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:29.979+0000 7f0932ffd640 1 -- 172.21.15.114:0/1893883747 <== mon.0 v2:172.21.15.114:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v89) ==== 74+0+22818 (secure 0 0 0) 0x7f092c0bd7e0 con 0x7f093c102040 2024-04-23T10:26:29.978 INFO:teuthology.orchestra.run.smithi114.stdout: 2024-04-23T10:26:29.978 INFO:teuthology.orchestra.run.smithi114.stdout:{"epoch":89,"fsid":"aa240572-0159-11ef-bc93-c7b262605968","created":"2024-04-23T10:16:53.351136+0000","modified":"2024-04-23T10:25:47.573901+0000","last_up_change":"2024-04-23T10:23:54.153269+0000","last_in_change":"2024-04-23T10:23:32.963085+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":26,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-04-23T10:20:11.970297+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":12,"score_stable":12,"optimal_score":0.25,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-04-23T10:24:41.218540+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"86","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.8799999952316284,"score_stable":1.8799999952316284,"optimal_score":1,"raw_score_acting":1.8799999952316284,"raw_score_stable":1.8799999952316284,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-04-23T10:24:42.341799+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"87","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"fd64adde-9445-45a4-a171-ef656e2c5709","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":88,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6802","nonce":736366878},{"type":"v1","addr":"172.21.15.114:6803","nonce":736366878}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6804","nonce":736366878},{"type":"v1","addr":"172.21.15.114:6805","nonce":736366878}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6808","nonce":736366878},{"type":"v1","addr":"172.21.15.114:6809","nonce":736366878}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6806","nonce":736366878},{"type":"v1","addr":"172.21.15.114:6807","nonce":736366878}]},"public_addr":"172.21.15.114:6803/736366878","cluster_addr":"172.21.15.114:6805/736366878","heartbeat_back_addr":"172.21.15.114:6809/736366878","heartbeat_front_addr":"172.21.15.114:6807/736366878","state":["exists","up"]},{"osd":1,"uuid":"e279a1e6-2702-472a-930a-e08ece76a861","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":85,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6800","nonce":1133540779},{"type":"v1","addr":"172.21.15.119:6801","nonce":1133540779}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6802","nonce":1133540779},{"type":"v1","addr":"172.21.15.119:6803","nonce":1133540779}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6806","nonce":1133540779},{"type":"v1","addr":"172.21.15.119:6807","nonce":1133540779}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6804","nonce":1133540779},{"type":"v1","addr":"172.21.15.119:6805","nonce":1133540779}]},"public_addr":"172.21.15.119:6801/1133540779","cluster_addr":"172.21.15.119:6803/1133540779","heartbeat_back_addr":"172.21.15.119:6807/1133540779","heartbeat_front_addr":"172.21.15.119:6805/1133540779","state":["exists","up"]},{"osd":2,"uuid":"f4c9f2b4-ff49-4e14-89cd-b477feeb9382","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":85,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6800","nonce":799652511},{"type":"v1","addr":"172.21.15.204:6801","nonce":799652511}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6802","nonce":799652511},{"type":"v1","addr":"172.21.15.204:6803","nonce":799652511}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6806","nonce":799652511},{"type":"v1","addr":"172.21.15.204:6807","nonce":799652511}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6804","nonce":799652511},{"type":"v1","addr":"172.21.15.204:6805","nonce":799652511}]},"public_addr":"172.21.15.204:6801/799652511","cluster_addr":"172.21.15.204:6803/799652511","heartbeat_back_addr":"172.21.15.204:6807/799652511","heartbeat_front_addr":"172.21.15.204:6805/799652511","state":["exists","up"]},{"osd":3,"uuid":"52d2a566-e71c-4bb3-8d6c-5abdd43177c2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":25,"up_thru":88,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6810","nonce":4275595091},{"type":"v1","addr":"172.21.15.114:6811","nonce":4275595091}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6812","nonce":4275595091},{"type":"v1","addr":"172.21.15.114:6813","nonce":4275595091}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6816","nonce":4275595091},{"type":"v1","addr":"172.21.15.114:6817","nonce":4275595091}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6814","nonce":4275595091},{"type":"v1","addr":"172.21.15.114:6815","nonce":4275595091}]},"public_addr":"172.21.15.114:6811/4275595091","cluster_addr":"172.21.15.114:6813/4275595091","heartbeat_back_addr":"172.21.15.114:6817/4275595091","heartbeat_front_addr":"172.21.15.114:6815/4275595091","state":["exists","up"]},{"osd":4,"uuid":"d911a7c1-4bdc-4d28-b8a6-73896a6e7de6","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":33,"up_thru":85,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6808","nonce":1611007172},{"type":"v1","addr":"172.21.15.119:6809","nonce":1611007172}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6810","nonce":1611007172},{"type":"v1","addr":"172.21.15.119:6811","nonce":1611007172}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6814","nonce":1611007172},{"type":"v1","addr":"172.21.15.119:6815","nonce":1611007172}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6812","nonce":1611007172},{"type":"v1","addr":"172.21.15.119:6813","nonce":1611007172}]},"public_addr":"172.21.15.119:6809/1611007172","cluster_addr":"172.21.15.119:6811/1611007172","heartbeat_back_addr":"172.21.15.119:6815/1611007172","heartbeat_front_addr":"172.21.15.119:6813/1611007172","state":["exists","up"]},{"osd":5,"uuid":"f7590e1a-e93c-4c29-b15f-38ee52d1f921","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":39,"up_thru":85,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6808","nonce":1427563312},{"type":"v1","addr":"172.21.15.204:6809","nonce":1427563312}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6810","nonce":1427563312},{"type":"v1","addr":"172.21.15.204:6811","nonce":1427563312}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6814","nonce":1427563312},{"type":"v1","addr":"172.21.15.204:6815","nonce":1427563312}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6812","nonce":1427563312},{"type":"v1","addr":"172.21.15.204:6813","nonce":1427563312}]},"public_addr":"172.21.15.204:6809/1427563312","cluster_addr":"172.21.15.204:6811/1427563312","heartbeat_back_addr":"172.21.15.204:6815/1427563312","heartbeat_front_addr":"172.21.15.204:6813/1427563312","state":["exists","up"]},{"osd":6,"uuid":"534d20e8-3e54-4db8-a845-6119f49c7a78","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":47,"up_thru":88,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6818","nonce":4072912544},{"type":"v1","addr":"172.21.15.114:6819","nonce":4072912544}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6820","nonce":4072912544},{"type":"v1","addr":"172.21.15.114:6821","nonce":4072912544}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6824","nonce":4072912544},{"type":"v1","addr":"172.21.15.114:6825","nonce":4072912544}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6822","nonce":4072912544},{"type":"v1","addr":"172.21.15.114:6823","nonce":4072912544}]},"public_addr":"172.21.15.114:6819/4072912544","cluster_addr":"172.21.15.114:6821/4072912544","heartbeat_back_addr":"172.21.15.114:6825/4072912544","heartbeat_front_addr":"172.21.15.114:6823/4072912544","state":["exists","up"]},{"osd":7,"uuid":"f0ec22b8-1b48-4859-9d65-b00403aff3ad","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":53,"up_thru":88,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6816","nonce":2237458793},{"type":"v1","addr":"172.21.15.119:6817","nonce":2237458793}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6818","nonce":2237458793},{"type":"v1","addr":"172.21.15.119:6819","nonce":2237458793}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6822","nonce":2237458793},{"type":"v1","addr":"172.21.15.119:6823","nonce":2237458793}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6820","nonce":2237458793},{"type":"v1","addr":"172.21.15.119:6821","nonce":2237458793}]},"public_addr":"172.21.15.119:6817/2237458793","cluster_addr":"172.21.15.119:6819/2237458793","heartbeat_back_addr":"172.21.15.119:6823/2237458793","heartbeat_front_addr":"172.21.15.119:6821/2237458793","state":["exists","up"]},{"osd":8,"uuid":"3ac89686-6d76-4cef-9089-ac29e4b3bcac","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":62,"up_thru":85,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6816","nonce":1426318231},{"type":"v1","addr":"172.21.15.204:6817","nonce":1426318231}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6818","nonce":1426318231},{"type":"v1","addr":"172.21.15.204:6819","nonce":1426318231}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6822","nonce":1426318231},{"type":"v1","addr":"172.21.15.204:6823","nonce":1426318231}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6820","nonce":1426318231},{"type":"v1","addr":"172.21.15.204:6821","nonce":1426318231}]},"public_addr":"172.21.15.204:6817/1426318231","cluster_addr":"172.21.15.204:6819/1426318231","heartbeat_back_addr":"172.21.15.204:6823/1426318231","heartbeat_front_addr":"172.21.15.204:6821/1426318231","state":["exists","up"]},{"osd":9,"uuid":"a80ffe4d-1e98-460f-b046-c4754285cc5a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":69,"up_thru":88,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6826","nonce":2750635721},{"type":"v1","addr":"172.21.15.114:6827","nonce":2750635721}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6828","nonce":2750635721},{"type":"v1","addr":"172.21.15.114:6829","nonce":2750635721}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6832","nonce":2750635721},{"type":"v1","addr":"172.21.15.114:6833","nonce":2750635721}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6830","nonce":2750635721},{"type":"v1","addr":"172.21.15.114:6831","nonce":2750635721}]},"public_addr":"172.21.15.114:6827/2750635721","cluster_addr":"172.21.15.114:6829/2750635721","heartbeat_back_addr":"172.21.15.114:6833/2750635721","heartbeat_front_addr":"172.21.15.114:6831/2750635721","state":["exists","up"]},{"osd":10,"uuid":"cd75a579-bf12-4600-ac09-a8bfce772ced","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":74,"up_thru":85,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6824","nonce":1400187701},{"type":"v1","addr":"172.21.15.119:6825","nonce":1400187701}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6826","nonce":1400187701},{"type":"v1","addr":"172.21.15.119:6827","nonce":1400187701}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6830","nonce":1400187701},{"type":"v1","addr":"172.21.15.119:6831","nonce":1400187701}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6828","nonce":1400187701},{"type":"v1","addr":"172.21.15.119:6829","nonce":1400187701}]},"public_addr":"172.21.15.119:6825/1400187701","cluster_addr":"172.21.15.119:6827/1400187701","heartbeat_back_addr":"172.21.15.119:6831/1400187701","heartbeat_front_addr":"172.21.15.119:6829/1400187701","state":["exists","up"]},{"osd":11,"uuid":"53f037a5-3ec0-483a-80a1-dfee923a68e5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":82,"up_thru":88,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6824","nonce":2407298490},{"type":"v1","addr":"172.21.15.204:6825","nonce":2407298490}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6826","nonce":2407298490},{"type":"v1","addr":"172.21.15.204:6827","nonce":2407298490}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6830","nonce":2407298490},{"type":"v1","addr":"172.21.15.204:6831","nonce":2407298490}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6828","nonce":2407298490},{"type":"v1","addr":"172.21.15.204:6829","nonce":2407298490}]},"public_addr":"172.21.15.204:6825/2407298490","cluster_addr":"172.21.15.204:6827/2407298490","heartbeat_back_addr":"172.21.15.204:6831/2407298490","heartbeat_front_addr":"172.21.15.204:6829/2407298490","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-23T10:19:18.401147+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-23T10:19:42.611299+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-23T10:20:07.277367+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-23T10:20:31.208368+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-23T10:20:55.866225+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-23T10:21:19.961237+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-23T10:21:44.480783+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-23T10:22:09.173641+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-23T10:22:34.030328+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-23T10:22:59.652293+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-23T10:23:25.659673+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-23T10:23:51.168171+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"2.e","mappings":[{"from":2,"to":4}]},{"pgid":"2.1d","mappings":[{"from":8,"to":11}]},{"pgid":"3.e","mappings":[{"from":5,"to":9}]},{"pgid":"3.12","mappings":[{"from":3,"to":9}]},{"pgid":"3.17","mappings":[{"from":4,"to":6}]},{"pgid":"3.23","mappings":[{"from":4,"to":7}]},{"pgid":"3.27","mappings":[{"from":4,"to":6}]},{"pgid":"3.31","mappings":[{"from":4,"to":8}]},{"pgid":"3.38","mappings":[{"from":3,"to":9}]},{"pgid":"3.39","mappings":[{"from":5,"to":8}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.114:6801/1604678557":"2024-04-24T10:17:45.874609+0000","172.21.15.114:0/4025385854":"2024-04-24T10:17:45.874609+0000","172.21.15.114:0/2228273469":"2024-04-24T10:17:19.910348+0000","172.21.15.114:6800/2831530838":"2024-04-24T10:17:19.910348+0000","172.21.15.114:0/1333447450":"2024-04-24T10:17:45.874609+0000","172.21.15.114:0/2233026840":"2024-04-24T10:17:19.910348+0000","172.21.15.114:0/1902281545":"2024-04-24T10:17:19.910348+0000","172.21.15.114:6801/2831530838":"2024-04-24T10:17:19.910348+0000","172.21.15.114:6800/1604678557":"2024-04-24T10:17:45.874609+0000","172.21.15.114:0/1623553811":"2024-04-24T10:17:45.874609+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-23T10:26:29.980 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:29.981+0000 7f0943d30640 1 -- 172.21.15.114:0/1893883747 >> [v2:172.21.15.114:6800/465333808,v1:172.21.15.114:6801/465333808] conn(0x7f09280a3bb0 msgr2=0x7f09280a6070 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T10:26:29.980 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:29.981+0000 7f0943d30640 1 --2- 172.21.15.114:0/1893883747 >> [v2:172.21.15.114:6800/465333808,v1:172.21.15.114:6801/465333808] conn(0x7f09280a3bb0 0x7f09280a6070 secure :-1 s=READY pgs=238 cs=0 l=1 rev1=1 crypto rx=0x7f09380058c0 tx=0x7f0938005850 comp rx=0 tx=0).stop 2024-04-23T10:26:29.980 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:29.981+0000 7f0943d30640 1 -- 172.21.15.114:0/1893883747 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f093c102040 msgr2=0x7f093c19a940 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T10:26:29.980 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:29.981+0000 7f0943d30640 1 --2- 172.21.15.114:0/1893883747 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f093c102040 0x7f093c19a940 secure :-1 s=READY pgs=371 cs=0 l=1 rev1=1 crypto rx=0x7f092c005940 tx=0x7f092c004430 comp rx=0 tx=0).stop 2024-04-23T10:26:29.980 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:29.981+0000 7f0943d30640 1 -- 172.21.15.114:0/1893883747 shutdown_connections 2024-04-23T10:26:29.981 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:29.981+0000 7f0943d30640 1 --2- 172.21.15.114:0/1893883747 >> [v2:172.21.15.114:6800/465333808,v1:172.21.15.114:6801/465333808] conn(0x7f09280a3bb0 0x7f09280a6070 unknown :-1 s=CLOSED pgs=238 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:26:29.981 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:29.981+0000 7f0943d30640 1 --2- 172.21.15.114:0/1893883747 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f093c102a00 0x7f093c1a19b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:26:29.981 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:29.981+0000 7f0943d30640 1 --2- 172.21.15.114:0/1893883747 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f093c102040 0x7f093c19a940 unknown :-1 s=CLOSED pgs=371 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:26:29.981 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:29.981+0000 7f0943d30640 1 --2- 172.21.15.114:0/1893883747 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f093c100e60 0x7f093c19a400 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:26:29.981 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:29.981+0000 7f0943d30640 1 -- 172.21.15.114:0/1893883747 >> 172.21.15.114:0/1893883747 conn(0x7f093c0fc440 msgr2=0x7f093c06a5f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T10:26:29.981 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:29.981+0000 7f0943d30640 1 -- 172.21.15.114:0/1893883747 shutdown_connections 2024-04-23T10:26:29.981 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:29.981+0000 7f0943d30640 1 -- 172.21.15.114:0/1893883747 wait complete. 2024-04-23T10:26:29.992 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-04-23T10:20:11.970297+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '22', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 12, 'score_stable': 12, 'optimal_score': 0.25, 'raw_score_acting': 3, 'raw_score_stable': 3, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 2, 'pool_name': 'cephfs_metadata', 'create_time': '2024-04-23T10:24:41.218540+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '86', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.8799999952316284, 'score_stable': 1.8799999952316284, 'optimal_score': 1, 'raw_score_acting': 1.8799999952316284, 'raw_score_stable': 1.8799999952316284, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 3, 'pool_name': 'cephfs_data', 'create_time': '2024-04-23T10:24:42.341799+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '87', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-04-23T10:26:29.992 DEBUG:teuthology.orchestra.run.smithi114:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-04-23T10:26:30.095 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:30.094+0000 7f4fda033640 1 -- 172.21.15.114:0/2895452224 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f4fd4101fe0 msgr2=0x7f4fd4102460 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T10:26:30.095 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:30.094+0000 7f4fda033640 1 --2- 172.21.15.114:0/2895452224 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f4fd4101fe0 0x7f4fd4102460 secure :-1 s=READY pgs=372 cs=0 l=1 rev1=1 crypto rx=0x7f4fc4009900 tx=0x7f4fc402a410 comp rx=0 tx=0).stop 2024-04-23T10:26:30.095 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:30.094+0000 7f4fda033640 1 -- 172.21.15.114:0/2895452224 shutdown_connections 2024-04-23T10:26:30.095 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:30.094+0000 7f4fda033640 1 --2- 172.21.15.114:0/2895452224 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f4fd41029a0 0x7f4fd41092c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:26:30.095 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:30.094+0000 7f4fda033640 1 --2- 172.21.15.114:0/2895452224 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f4fd4101fe0 0x7f4fd4102460 unknown :-1 s=CLOSED pgs=372 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:26:30.095 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:30.094+0000 7f4fda033640 1 --2- 172.21.15.114:0/2895452224 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f4fd4100e90 0x7f4fd41012b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:26:30.095 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:30.094+0000 7f4fda033640 1 -- 172.21.15.114:0/2895452224 >> 172.21.15.114:0/2895452224 conn(0x7f4fd40fc400 msgr2=0x7f4fd40fe840 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T10:26:30.095 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:30.094+0000 7f4fda033640 1 -- 172.21.15.114:0/2895452224 shutdown_connections 2024-04-23T10:26:30.095 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:30.094+0000 7f4fda033640 1 -- 172.21.15.114:0/2895452224 wait complete. 2024-04-23T10:26:30.096 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:30.094+0000 7f4fda033640 1 Processor -- start 2024-04-23T10:26:30.096 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:30.094+0000 7f4fda033640 1 -- start start 2024-04-23T10:26:30.096 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:30.094+0000 7f4fda033640 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f4fd4100e90 0x7f4fd4069c50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T10:26:30.096 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:30.094+0000 7f4fda033640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f4fd4101fe0 0x7f4fd419cf00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T10:26:30.096 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:30.094+0000 7f4fda033640 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f4fd41029a0 0x7f4fd419f520 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T10:26:30.096 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:30.094+0000 7f4fda033640 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x7f4fd410c9d0 con 0x7f4fd4101fe0 2024-04-23T10:26:30.096 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:30.094+0000 7f4fda033640 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 -- 0x7f4fd410c850 con 0x7f4fd41029a0 2024-04-23T10:26:30.096 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:30.094+0000 7f4fda033640 1 -- --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_getmap magic: 0 -- 0x7f4fd410cb50 con 0x7f4fd4100e90 2024-04-23T10:26:30.096 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:30.095+0000 7f4fd2ffd640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f4fd4101fe0 0x7f4fd419cf00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T10:26:30.096 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:30.095+0000 7f4fd2ffd640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f4fd4101fe0 0x7f4fd419cf00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.114:58354/0 (socket says 172.21.15.114:58354) 2024-04-23T10:26:30.096 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:30.095+0000 7f4fd2ffd640 1 -- 172.21.15.114:0/1129920789 learned_addr learned my addr 172.21.15.114:0/1129920789 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-04-23T10:26:30.096 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:30.095+0000 7f4fd37fe640 1 --2- 172.21.15.114:0/1129920789 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f4fd4100e90 0x7f4fd4069c50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T10:26:30.096 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:30.095+0000 7f4fd3fff640 1 --2- 172.21.15.114:0/1129920789 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f4fd41029a0 0x7f4fd419f520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T10:26:30.096 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:30.095+0000 7f4fd2ffd640 1 -- 172.21.15.114:0/1129920789 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f4fd4100e90 msgr2=0x7f4fd4069c50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T10:26:30.096 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:30.095+0000 7f4fd2ffd640 1 --2- 172.21.15.114:0/1129920789 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f4fd4100e90 0x7f4fd4069c50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:26:30.097 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:30.095+0000 7f4fd2ffd640 1 -- 172.21.15.114:0/1129920789 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f4fd41029a0 msgr2=0x7f4fd419f520 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T10:26:30.097 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:30.095+0000 7f4fd2ffd640 1 --2- 172.21.15.114:0/1129920789 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f4fd41029a0 0x7f4fd419f520 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:26:30.097 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:30.095+0000 7f4fd2ffd640 1 -- 172.21.15.114:0/1129920789 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f4fd419fe30 con 0x7f4fd4101fe0 2024-04-23T10:26:30.097 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:30.095+0000 7f4fd3fff640 1 --2- 172.21.15.114:0/1129920789 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f4fd41029a0 0x7f4fd419f520 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-23T10:26:30.097 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:30.095+0000 7f4fd2ffd640 1 --2- 172.21.15.114:0/1129920789 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f4fd4101fe0 0x7f4fd419cf00 secure :-1 s=READY pgs=373 cs=0 l=1 rev1=1 crypto rx=0x7f4fc4006010 tx=0x7f4fc40042f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T10:26:30.097 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:30.095+0000 7f4fd0ff9640 1 -- 172.21.15.114:0/1129920789 <== mon.0 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4fc403f030 con 0x7f4fd4101fe0 2024-04-23T10:26:30.097 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:30.095+0000 7f4fd0ff9640 1 -- 172.21.15.114:0/1129920789 <== mon.0 v2:172.21.15.114:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f4fc40318f0 con 0x7f4fd4101fe0 2024-04-23T10:26:30.097 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:30.095+0000 7f4fda033640 1 -- 172.21.15.114:0/1129920789 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f4fd41a0060 con 0x7f4fd4101fe0 2024-04-23T10:26:30.097 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:30.095+0000 7f4fd0ff9640 1 -- 172.21.15.114:0/1129920789 <== mon.0 v2:172.21.15.114:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4fc4035070 con 0x7f4fd4101fe0 2024-04-23T10:26:30.097 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:30.095+0000 7f4fda033640 1 -- 172.21.15.114:0/1129920789 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f4fd41a0550 con 0x7f4fd4101fe0 2024-04-23T10:26:30.097 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:30.096+0000 7f4fda033640 1 -- 172.21.15.114:0/1129920789 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f4fd41081c0 con 0x7f4fd4101fe0 2024-04-23T10:26:30.098 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:30.097+0000 7f4fd0ff9640 1 -- 172.21.15.114:0/1129920789 <== mon.0 v2:172.21.15.114:3300/0 4 ==== mgrmap(e 18) ==== 138647+0+0 (secure 0 0 0) 0x7f4fc4042030 con 0x7f4fd4101fe0 2024-04-23T10:26:30.098 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:30.098+0000 7f4fd0ff9640 1 --2- 172.21.15.114:0/1129920789 >> [v2:172.21.15.114:6800/465333808,v1:172.21.15.114:6801/465333808] conn(0x7f4fbc0a3bb0 0x7f4fbc0a6070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T10:26:30.098 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:30.098+0000 7f4fd37fe640 1 --2- 172.21.15.114:0/1129920789 >> [v2:172.21.15.114:6800/465333808,v1:172.21.15.114:6801/465333808] conn(0x7f4fbc0a3bb0 0x7f4fbc0a6070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T10:26:30.098 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:30.098+0000 7f4fd0ff9640 1 -- 172.21.15.114:0/1129920789 <== mon.0 v2:172.21.15.114:3300/0 5 ==== osd_map(89..89 src has 1..89) ==== 8269+0+0 (secure 0 0 0) 0x7f4fc40f0980 con 0x7f4fd4101fe0 2024-04-23T10:26:30.098 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:30.098+0000 7f4fd37fe640 1 --2- 172.21.15.114:0/1129920789 >> [v2:172.21.15.114:6800/465333808,v1:172.21.15.114:6801/465333808] conn(0x7f4fbc0a3bb0 0x7f4fbc0a6070 secure :-1 s=READY pgs=239 cs=0 l=1 rev1=1 crypto rx=0x7f4fc8009bd0 tx=0x7f4fc8005f90 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T10:26:30.101 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:30.103+0000 7f4fd0ff9640 1 -- 172.21.15.114:0/1129920789 <== mon.0 v2:172.21.15.114:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f4fc40bd340 con 0x7f4fd4101fe0 2024-04-23T10:26:30.254 INFO:journalctl@ceph.mon.a.smithi114.stdout:Apr 23 10:26:30 smithi114 ceph-mon[29255]: from='client.? 172.21.15.114:0/2539925690' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2024-04-23T10:26:30.254 INFO:journalctl@ceph.mon.a.smithi114.stdout:Apr 23 10:26:30 smithi114 ceph-mon[29255]: from='client.? 172.21.15.114:0/2847980599' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-23T10:26:30.254 INFO:journalctl@ceph.mon.a.smithi114.stdout:Apr 23 10:26:30 smithi114 ceph-mon[29255]: from='client.? 172.21.15.114:0/1893883747' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-23T10:26:30.279 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:30.280+0000 7f4fda033640 1 -- 172.21.15.114:0/1129920789 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) -- 0x7f4fd4062b50 con 0x7f4fd4101fe0 2024-04-23T10:26:30.280 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:30.281+0000 7f4fd0ff9640 1 -- 172.21.15.114:0/1129920789 <== mon.0 v2:172.21.15.114:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v89) ==== 93+0+10 (secure 0 0 0) 0x7f4fc40bcc40 con 0x7f4fd4101fe0 2024-04-23T10:26:30.280 INFO:teuthology.orchestra.run.smithi114.stdout:pg_num: 1 2024-04-23T10:26:30.282 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:30.283+0000 7f4fda033640 1 -- 172.21.15.114:0/1129920789 >> [v2:172.21.15.114:6800/465333808,v1:172.21.15.114:6801/465333808] conn(0x7f4fbc0a3bb0 msgr2=0x7f4fbc0a6070 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T10:26:30.282 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:30.283+0000 7f4fda033640 1 --2- 172.21.15.114:0/1129920789 >> [v2:172.21.15.114:6800/465333808,v1:172.21.15.114:6801/465333808] conn(0x7f4fbc0a3bb0 0x7f4fbc0a6070 secure :-1 s=READY pgs=239 cs=0 l=1 rev1=1 crypto rx=0x7f4fc8009bd0 tx=0x7f4fc8005f90 comp rx=0 tx=0).stop 2024-04-23T10:26:30.282 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:30.283+0000 7f4fda033640 1 -- 172.21.15.114:0/1129920789 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f4fd4101fe0 msgr2=0x7f4fd419cf00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T10:26:30.282 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:30.283+0000 7f4fda033640 1 --2- 172.21.15.114:0/1129920789 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f4fd4101fe0 0x7f4fd419cf00 secure :-1 s=READY pgs=373 cs=0 l=1 rev1=1 crypto rx=0x7f4fc4006010 tx=0x7f4fc40042f0 comp rx=0 tx=0).stop 2024-04-23T10:26:30.282 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:30.283+0000 7f4fda033640 1 -- 172.21.15.114:0/1129920789 shutdown_connections 2024-04-23T10:26:30.282 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:30.283+0000 7f4fda033640 1 --2- 172.21.15.114:0/1129920789 >> [v2:172.21.15.114:6800/465333808,v1:172.21.15.114:6801/465333808] conn(0x7f4fbc0a3bb0 0x7f4fbc0a6070 unknown :-1 s=CLOSED pgs=239 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:26:30.282 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:30.283+0000 7f4fda033640 1 --2- 172.21.15.114:0/1129920789 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f4fd41029a0 0x7f4fd419f520 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:26:30.282 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:30.283+0000 7f4fda033640 1 --2- 172.21.15.114:0/1129920789 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f4fd4101fe0 0x7f4fd419cf00 unknown :-1 s=CLOSED pgs=373 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:26:30.282 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:30.283+0000 7f4fda033640 1 --2- 172.21.15.114:0/1129920789 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f4fd4100e90 0x7f4fd4069c50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:26:30.282 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:30.283+0000 7f4fda033640 1 -- 172.21.15.114:0/1129920789 >> 172.21.15.114:0/1129920789 conn(0x7f4fd40fc400 msgr2=0x7f4fd4105390 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T10:26:30.282 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:30.283+0000 7f4fda033640 1 -- 172.21.15.114:0/1129920789 shutdown_connections 2024-04-23T10:26:30.282 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:30.283+0000 7f4fda033640 1 -- 172.21.15.114:0/1129920789 wait complete. 2024-04-23T10:26:30.293 DEBUG:teuthology.orchestra.run.smithi114:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2024-04-23T10:26:30.395 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:30.394+0000 7f74a2a72640 1 -- 172.21.15.114:0/3297701277 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f749c101fe0 msgr2=0x7f749c102460 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T10:26:30.395 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:30.394+0000 7f74a2a72640 1 --2- 172.21.15.114:0/3297701277 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f749c101fe0 0x7f749c102460 secure :-1 s=READY pgs=374 cs=0 l=1 rev1=1 crypto rx=0x7f7488009900 tx=0x7f748802a410 comp rx=0 tx=0).stop 2024-04-23T10:26:30.395 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:30.394+0000 7f74a2a72640 1 -- 172.21.15.114:0/3297701277 shutdown_connections 2024-04-23T10:26:30.395 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:30.394+0000 7f74a2a72640 1 --2- 172.21.15.114:0/3297701277 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f749c1029a0 0x7f749c1092c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:26:30.395 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:30.394+0000 7f74a2a72640 1 --2- 172.21.15.114:0/3297701277 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f749c101fe0 0x7f749c102460 unknown :-1 s=CLOSED pgs=374 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:26:30.395 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:30.394+0000 7f74a2a72640 1 --2- 172.21.15.114:0/3297701277 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f749c100e90 0x7f749c1012b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:26:30.395 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:30.394+0000 7f74a2a72640 1 -- 172.21.15.114:0/3297701277 >> 172.21.15.114:0/3297701277 conn(0x7f749c0fc400 msgr2=0x7f749c0fe840 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T10:26:30.395 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:30.394+0000 7f74a2a72640 1 -- 172.21.15.114:0/3297701277 shutdown_connections 2024-04-23T10:26:30.395 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:30.394+0000 7f74a2a72640 1 -- 172.21.15.114:0/3297701277 wait complete. 2024-04-23T10:26:30.395 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:30.394+0000 7f74a2a72640 1 Processor -- start 2024-04-23T10:26:30.395 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:30.394+0000 7f74a2a72640 1 -- start start 2024-04-23T10:26:30.395 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:30.395+0000 7f74a2a72640 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f749c100e90 0x7f749c06b180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T10:26:30.396 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:30.395+0000 7f74a2a72640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f749c101fe0 0x7f749c19cf30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T10:26:30.396 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:30.395+0000 7f74a2a72640 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f749c1029a0 0x7f749c19f5a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T10:26:30.396 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:30.395+0000 7f74a2a72640 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x7f749c10ca40 con 0x7f749c101fe0 2024-04-23T10:26:30.396 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:30.395+0000 7f74a2a72640 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 -- 0x7f749c10c8c0 con 0x7f749c100e90 2024-04-23T10:26:30.396 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:30.395+0000 7f74a2a72640 1 -- --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_getmap magic: 0 -- 0x7f749c10cbc0 con 0x7f749c1029a0 2024-04-23T10:26:30.396 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:30.395+0000 7f749bfff640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f749c101fe0 0x7f749c19cf30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T10:26:30.396 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:30.395+0000 7f749bfff640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f749c101fe0 0x7f749c19cf30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.114:58366/0 (socket says 172.21.15.114:58366) 2024-04-23T10:26:30.396 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:30.395+0000 7f749bfff640 1 -- 172.21.15.114:0/3529908104 learned_addr learned my addr 172.21.15.114:0/3529908104 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-04-23T10:26:30.396 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:30.395+0000 7f74a080e640 1 --2- 172.21.15.114:0/3529908104 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f749c100e90 0x7f749c06b180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T10:26:30.396 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:30.395+0000 7f74a100f640 1 --2- 172.21.15.114:0/3529908104 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f749c1029a0 0x7f749c19f5a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T10:26:30.396 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:30.395+0000 7f749bfff640 1 -- 172.21.15.114:0/3529908104 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f749c1029a0 msgr2=0x7f749c19f5a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T10:26:30.396 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:30.395+0000 7f749bfff640 1 --2- 172.21.15.114:0/3529908104 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f749c1029a0 0x7f749c19f5a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:26:30.396 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:30.395+0000 7f749bfff640 1 -- 172.21.15.114:0/3529908104 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f749c100e90 msgr2=0x7f749c06b180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T10:26:30.396 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:30.395+0000 7f749bfff640 1 --2- 172.21.15.114:0/3529908104 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f749c100e90 0x7f749c06b180 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:26:30.396 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:30.395+0000 7f749bfff640 1 -- 172.21.15.114:0/3529908104 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f749c19fe60 con 0x7f749c101fe0 2024-04-23T10:26:30.396 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:30.395+0000 7f749bfff640 1 --2- 172.21.15.114:0/3529908104 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f749c101fe0 0x7f749c19cf30 secure :-1 s=READY pgs=375 cs=0 l=1 rev1=1 crypto rx=0x7f7488006010 tx=0x7f74880042f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T10:26:30.397 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:30.395+0000 7f7499ffb640 1 -- 172.21.15.114:0/3529908104 <== mon.0 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f748803f030 con 0x7f749c101fe0 2024-04-23T10:26:30.397 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:30.395+0000 7f7499ffb640 1 -- 172.21.15.114:0/3529908104 <== mon.0 v2:172.21.15.114:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f7488003cb0 con 0x7f749c101fe0 2024-04-23T10:26:30.397 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:30.395+0000 7f74a2a72640 1 -- 172.21.15.114:0/3529908104 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f749c1a00f0 con 0x7f749c101fe0 2024-04-23T10:26:30.397 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:30.395+0000 7f7499ffb640 1 -- 172.21.15.114:0/3529908104 <== mon.0 v2:172.21.15.114:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7488035070 con 0x7f749c101fe0 2024-04-23T10:26:30.397 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:30.396+0000 7f74a2a72640 1 -- 172.21.15.114:0/3529908104 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f749c1a06b0 con 0x7f749c101fe0 2024-04-23T10:26:30.398 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:30.397+0000 7f74a2a72640 1 -- 172.21.15.114:0/3529908104 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f749c108270 con 0x7f749c101fe0 2024-04-23T10:26:30.398 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:30.398+0000 7f7499ffb640 1 -- 172.21.15.114:0/3529908104 <== mon.0 v2:172.21.15.114:3300/0 4 ==== mgrmap(e 18) ==== 138647+0+0 (secure 0 0 0) 0x7f7488042030 con 0x7f749c101fe0 2024-04-23T10:26:30.398 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:30.398+0000 7f7499ffb640 1 --2- 172.21.15.114:0/3529908104 >> [v2:172.21.15.114:6800/465333808,v1:172.21.15.114:6801/465333808] conn(0x7f748c0a3c30 0x7f748c0a60f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T10:26:30.398 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:30.398+0000 7f74a080e640 1 --2- 172.21.15.114:0/3529908104 >> [v2:172.21.15.114:6800/465333808,v1:172.21.15.114:6801/465333808] conn(0x7f748c0a3c30 0x7f748c0a60f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T10:26:30.398 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:30.398+0000 7f7499ffb640 1 -- 172.21.15.114:0/3529908104 <== mon.0 v2:172.21.15.114:3300/0 5 ==== osd_map(89..89 src has 1..89) ==== 8269+0+0 (secure 0 0 0) 0x7f74880f09d0 con 0x7f749c101fe0 2024-04-23T10:26:30.398 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:30.399+0000 7f74a080e640 1 --2- 172.21.15.114:0/3529908104 >> [v2:172.21.15.114:6800/465333808,v1:172.21.15.114:6801/465333808] conn(0x7f748c0a3c30 0x7f748c0a60f0 secure :-1 s=READY pgs=240 cs=0 l=1 rev1=1 crypto rx=0x7f7490009990 tx=0x7f7490009110 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T10:26:30.401 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:30.403+0000 7f7499ffb640 1 -- 172.21.15.114:0/3529908104 <== mon.0 v2:172.21.15.114:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f74880bd390 con 0x7f749c101fe0 2024-04-23T10:26:30.491 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 23 10:26:30 smithi119 ceph-mon[32608]: from='client.? 172.21.15.114:0/2539925690' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2024-04-23T10:26:30.492 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 23 10:26:30 smithi119 ceph-mon[32608]: from='client.? 172.21.15.114:0/2847980599' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-23T10:26:30.492 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 23 10:26:30 smithi119 ceph-mon[32608]: from='client.? 172.21.15.114:0/1893883747' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-23T10:26:30.578 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 23 10:26:30 smithi204 ceph-mon[32649]: from='client.? 172.21.15.114:0/2539925690' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2024-04-23T10:26:30.578 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 23 10:26:30 smithi204 ceph-mon[32649]: from='client.? 172.21.15.114:0/2847980599' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-23T10:26:30.578 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 23 10:26:30 smithi204 ceph-mon[32649]: from='client.? 172.21.15.114:0/1893883747' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-23T10:26:30.579 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:30.580+0000 7f74a2a72640 1 -- 172.21.15.114:0/3529908104 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) -- 0x7f749c062b50 con 0x7f749c101fe0 2024-04-23T10:26:30.579 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:30.580+0000 7f7499ffb640 1 -- 172.21.15.114:0/3529908104 <== mon.0 v2:172.21.15.114:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v89) ==== 104+0+11 (secure 0 0 0) 0x7f74880bcc90 con 0x7f749c101fe0 2024-04-23T10:26:30.579 INFO:teuthology.orchestra.run.smithi114.stdout:pg_num: 32 2024-04-23T10:26:30.582 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:30.582+0000 7f74a2a72640 1 -- 172.21.15.114:0/3529908104 >> [v2:172.21.15.114:6800/465333808,v1:172.21.15.114:6801/465333808] conn(0x7f748c0a3c30 msgr2=0x7f748c0a60f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T10:26:30.582 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:30.583+0000 7f74a2a72640 1 --2- 172.21.15.114:0/3529908104 >> [v2:172.21.15.114:6800/465333808,v1:172.21.15.114:6801/465333808] conn(0x7f748c0a3c30 0x7f748c0a60f0 secure :-1 s=READY pgs=240 cs=0 l=1 rev1=1 crypto rx=0x7f7490009990 tx=0x7f7490009110 comp rx=0 tx=0).stop 2024-04-23T10:26:30.582 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:30.583+0000 7f74a2a72640 1 -- 172.21.15.114:0/3529908104 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f749c101fe0 msgr2=0x7f749c19cf30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T10:26:30.582 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:30.583+0000 7f74a2a72640 1 --2- 172.21.15.114:0/3529908104 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f749c101fe0 0x7f749c19cf30 secure :-1 s=READY pgs=375 cs=0 l=1 rev1=1 crypto rx=0x7f7488006010 tx=0x7f74880042f0 comp rx=0 tx=0).stop 2024-04-23T10:26:30.582 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:30.583+0000 7f74a2a72640 1 -- 172.21.15.114:0/3529908104 shutdown_connections 2024-04-23T10:26:30.582 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:30.583+0000 7f74a2a72640 1 --2- 172.21.15.114:0/3529908104 >> [v2:172.21.15.114:6800/465333808,v1:172.21.15.114:6801/465333808] conn(0x7f748c0a3c30 0x7f748c0a60f0 unknown :-1 s=CLOSED pgs=240 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:26:30.582 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:30.583+0000 7f74a2a72640 1 --2- 172.21.15.114:0/3529908104 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f749c1029a0 0x7f749c19f5a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:26:30.582 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:30.583+0000 7f74a2a72640 1 --2- 172.21.15.114:0/3529908104 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f749c101fe0 0x7f749c19cf30 unknown :-1 s=CLOSED pgs=375 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:26:30.582 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:30.583+0000 7f74a2a72640 1 --2- 172.21.15.114:0/3529908104 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f749c100e90 0x7f749c06b180 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:26:30.582 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:30.583+0000 7f74a2a72640 1 -- 172.21.15.114:0/3529908104 >> 172.21.15.114:0/3529908104 conn(0x7f749c0fc400 msgr2=0x7f749c105440 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T10:26:30.582 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:30.583+0000 7f74a2a72640 1 -- 172.21.15.114:0/3529908104 shutdown_connections 2024-04-23T10:26:30.582 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:30.583+0000 7f74a2a72640 1 -- 172.21.15.114:0/3529908104 wait complete. 2024-04-23T10:26:30.593 DEBUG:teuthology.orchestra.run.smithi114:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2024-04-23T10:26:30.695 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:30.694+0000 7f06c06d9640 1 -- 172.21.15.114:0/1759999423 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f06b8102a30 msgr2=0x7f06b8104e20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T10:26:30.696 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:30.694+0000 7f06c06d9640 1 --2- 172.21.15.114:0/1759999423 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f06b8102a30 0x7f06b8104e20 secure :-1 s=READY pgs=376 cs=0 l=1 rev1=1 crypto rx=0x7f06b4009900 tx=0x7f06b402a410 comp rx=0 tx=0).stop 2024-04-23T10:26:30.696 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:30.695+0000 7f06c06d9640 1 -- 172.21.15.114:0/1759999423 shutdown_connections 2024-04-23T10:26:30.696 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:30.695+0000 7f06c06d9640 1 --2- 172.21.15.114:0/1759999423 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f06b8107c90 0x7f06b810a0e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:26:30.696 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:30.695+0000 7f06c06d9640 1 --2- 172.21.15.114:0/1759999423 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f06b8105360 0x7f06b8107750 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:26:30.696 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:30.695+0000 7f06c06d9640 1 --2- 172.21.15.114:0/1759999423 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f06b8102a30 0x7f06b8104e20 unknown :-1 s=CLOSED pgs=376 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:26:30.696 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:30.695+0000 7f06c06d9640 1 -- 172.21.15.114:0/1759999423 >> 172.21.15.114:0/1759999423 conn(0x7f06b80fc440 msgr2=0x7f06b80fe860 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T10:26:30.696 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:30.695+0000 7f06c06d9640 1 -- 172.21.15.114:0/1759999423 shutdown_connections 2024-04-23T10:26:30.696 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:30.695+0000 7f06c06d9640 1 -- 172.21.15.114:0/1759999423 wait complete. 2024-04-23T10:26:30.696 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:30.695+0000 7f06c06d9640 1 Processor -- start 2024-04-23T10:26:30.696 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:30.695+0000 7f06c06d9640 1 -- start start 2024-04-23T10:26:30.696 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:30.695+0000 7f06c06d9640 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f06b8102a30 0x7f06b8198270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T10:26:30.696 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:30.695+0000 7f06c06d9640 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f06b8105360 0x7f06b81987b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T10:26:30.696 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:30.695+0000 7f06c06d9640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f06b8107c90 0x7f06b819f7d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T10:26:30.697 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:30.695+0000 7f06c06d9640 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x7f06b810daf0 con 0x7f06b8107c90 2024-04-23T10:26:30.697 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:30.695+0000 7f06c06d9640 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 -- 0x7f06b810d970 con 0x7f06b8102a30 2024-04-23T10:26:30.697 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:30.695+0000 7f06c06d9640 1 -- --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_getmap magic: 0 -- 0x7f06b810dc70 con 0x7f06b8105360 2024-04-23T10:26:30.697 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:30.695+0000 7f06bec76640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f06b8107c90 0x7f06b819f7d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T10:26:30.697 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:30.695+0000 7f06bec76640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f06b8107c90 0x7f06b819f7d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.114:58376/0 (socket says 172.21.15.114:58376) 2024-04-23T10:26:30.697 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:30.695+0000 7f06bec76640 1 -- 172.21.15.114:0/3222327275 learned_addr learned my addr 172.21.15.114:0/3222327275 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-04-23T10:26:30.697 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:30.695+0000 7f06bdc74640 1 --2- 172.21.15.114:0/3222327275 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f06b8105360 0x7f06b81987b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T10:26:30.697 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:30.696+0000 7f06be475640 1 --2- 172.21.15.114:0/3222327275 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f06b8102a30 0x7f06b8198270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T10:26:30.697 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:30.696+0000 7f06bec76640 1 -- 172.21.15.114:0/3222327275 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f06b8105360 msgr2=0x7f06b81987b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T10:26:30.697 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:30.696+0000 7f06bec76640 1 --2- 172.21.15.114:0/3222327275 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f06b8105360 0x7f06b81987b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:26:30.697 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:30.696+0000 7f06bec76640 1 -- 172.21.15.114:0/3222327275 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f06b8102a30 msgr2=0x7f06b8198270 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T10:26:30.697 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:30.696+0000 7f06bec76640 1 --2- 172.21.15.114:0/3222327275 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f06b8102a30 0x7f06b8198270 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:26:30.697 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:30.696+0000 7f06bec76640 1 -- 172.21.15.114:0/3222327275 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f06b819fe40 con 0x7f06b8107c90 2024-04-23T10:26:30.697 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:30.696+0000 7f06bdc74640 1 --2- 172.21.15.114:0/3222327275 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f06b8105360 0x7f06b81987b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-23T10:26:30.697 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:30.696+0000 7f06be475640 1 --2- 172.21.15.114:0/3222327275 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f06b8102a30 0x7f06b8198270 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-23T10:26:30.697 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:30.696+0000 7f06bec76640 1 --2- 172.21.15.114:0/3222327275 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f06b8107c90 0x7f06b819f7d0 secure :-1 s=READY pgs=377 cs=0 l=1 rev1=1 crypto rx=0x7f06b000eb10 tx=0x7f06b000c2d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T10:26:30.698 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:30.696+0000 7f06af7fe640 1 -- 172.21.15.114:0/3222327275 <== mon.0 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f06b0004b20 con 0x7f06b8107c90 2024-04-23T10:26:30.698 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:30.696+0000 7f06af7fe640 1 -- 172.21.15.114:0/3222327275 <== mon.0 v2:172.21.15.114:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f06b0004cc0 con 0x7f06b8107c90 2024-04-23T10:26:30.698 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:30.696+0000 7f06c06d9640 1 -- 172.21.15.114:0/3222327275 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f06b81a0130 con 0x7f06b8107c90 2024-04-23T10:26:30.698 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:30.696+0000 7f06af7fe640 1 -- 172.21.15.114:0/3222327275 <== mon.0 v2:172.21.15.114:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f06b0010670 con 0x7f06b8107c90 2024-04-23T10:26:30.698 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:30.696+0000 7f06c06d9640 1 -- 172.21.15.114:0/3222327275 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f06b81a0630 con 0x7f06b8107c90 2024-04-23T10:26:30.699 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:30.697+0000 7f06c06d9640 1 -- 172.21.15.114:0/3222327275 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f06b80fdf30 con 0x7f06b8107c90 2024-04-23T10:26:30.699 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:30.698+0000 7f06af7fe640 1 -- 172.21.15.114:0/3222327275 <== mon.0 v2:172.21.15.114:3300/0 4 ==== mgrmap(e 18) ==== 138647+0+0 (secure 0 0 0) 0x7f06b0005020 con 0x7f06b8107c90 2024-04-23T10:26:30.699 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:30.699+0000 7f06af7fe640 1 --2- 172.21.15.114:0/3222327275 >> [v2:172.21.15.114:6800/465333808,v1:172.21.15.114:6801/465333808] conn(0x7f06a40a3bb0 0x7f06a40a6070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T10:26:30.699 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:30.699+0000 7f06be475640 1 --2- 172.21.15.114:0/3222327275 >> [v2:172.21.15.114:6800/465333808,v1:172.21.15.114:6801/465333808] conn(0x7f06a40a3bb0 0x7f06a40a6070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T10:26:30.699 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:30.699+0000 7f06af7fe640 1 -- 172.21.15.114:0/3222327275 <== mon.0 v2:172.21.15.114:3300/0 5 ==== osd_map(89..89 src has 1..89) ==== 8269+0+0 (secure 0 0 0) 0x7f06b00d0a80 con 0x7f06b8107c90 2024-04-23T10:26:30.699 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:30.699+0000 7f06be475640 1 --2- 172.21.15.114:0/3222327275 >> [v2:172.21.15.114:6800/465333808,v1:172.21.15.114:6801/465333808] conn(0x7f06a40a3bb0 0x7f06a40a6070 secure :-1 s=READY pgs=241 cs=0 l=1 rev1=1 crypto rx=0x7f06b4006010 tx=0x7f06b4000bc0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T10:26:30.702 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:30.703+0000 7f06af7fe640 1 -- 172.21.15.114:0/3222327275 <== mon.0 v2:172.21.15.114:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f06b009d440 con 0x7f06b8107c90 2024-04-23T10:26:30.878 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:30.879+0000 7f06c06d9640 1 -- 172.21.15.114:0/3222327275 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) -- 0x7f06b8062b50 con 0x7f06b8107c90 2024-04-23T10:26:30.878 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:30.880+0000 7f06af7fe640 1 -- 172.21.15.114:0/3222327275 <== mon.0 v2:172.21.15.114:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v89) ==== 100+0+11 (secure 0 0 0) 0x7f06b009cd40 con 0x7f06b8107c90 2024-04-23T10:26:30.879 INFO:teuthology.orchestra.run.smithi114.stdout:pg_num: 64 2024-04-23T10:26:30.881 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:30.881+0000 7f06c06d9640 1 -- 172.21.15.114:0/3222327275 >> [v2:172.21.15.114:6800/465333808,v1:172.21.15.114:6801/465333808] conn(0x7f06a40a3bb0 msgr2=0x7f06a40a6070 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T10:26:30.881 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:30.881+0000 7f06c06d9640 1 --2- 172.21.15.114:0/3222327275 >> [v2:172.21.15.114:6800/465333808,v1:172.21.15.114:6801/465333808] conn(0x7f06a40a3bb0 0x7f06a40a6070 secure :-1 s=READY pgs=241 cs=0 l=1 rev1=1 crypto rx=0x7f06b4006010 tx=0x7f06b4000bc0 comp rx=0 tx=0).stop 2024-04-23T10:26:30.881 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:30.881+0000 7f06c06d9640 1 -- 172.21.15.114:0/3222327275 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f06b8107c90 msgr2=0x7f06b819f7d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T10:26:30.881 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:30.881+0000 7f06c06d9640 1 --2- 172.21.15.114:0/3222327275 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f06b8107c90 0x7f06b819f7d0 secure :-1 s=READY pgs=377 cs=0 l=1 rev1=1 crypto rx=0x7f06b000eb10 tx=0x7f06b000c2d0 comp rx=0 tx=0).stop 2024-04-23T10:26:30.881 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:30.881+0000 7f06c06d9640 1 -- 172.21.15.114:0/3222327275 shutdown_connections 2024-04-23T10:26:30.881 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:30.881+0000 7f06c06d9640 1 --2- 172.21.15.114:0/3222327275 >> [v2:172.21.15.114:6800/465333808,v1:172.21.15.114:6801/465333808] conn(0x7f06a40a3bb0 0x7f06a40a6070 unknown :-1 s=CLOSED pgs=241 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:26:30.881 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:30.881+0000 7f06c06d9640 1 --2- 172.21.15.114:0/3222327275 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f06b8107c90 0x7f06b819f7d0 unknown :-1 s=CLOSED pgs=377 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:26:30.881 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:30.881+0000 7f06c06d9640 1 --2- 172.21.15.114:0/3222327275 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f06b8105360 0x7f06b81987b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:26:30.881 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:30.881+0000 7f06c06d9640 1 --2- 172.21.15.114:0/3222327275 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f06b8102a30 0x7f06b8198270 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:26:30.881 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:30.881+0000 7f06c06d9640 1 -- 172.21.15.114:0/3222327275 >> 172.21.15.114:0/3222327275 conn(0x7f06b80fc440 msgr2=0x7f06b81034f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T10:26:30.881 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:30.881+0000 7f06c06d9640 1 -- 172.21.15.114:0/3222327275 shutdown_connections 2024-04-23T10:26:30.881 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:30.881+0000 7f06c06d9640 1 -- 172.21.15.114:0/3222327275 wait complete. 2024-04-23T10:26:30.892 DEBUG:teuthology.orchestra.run.smithi114:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-23T10:26:30.991 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:30.990+0000 7fe3965e8640 1 -- 172.21.15.114:0/50005159 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fe390100290 msgr2=0x7fe3901006b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T10:26:30.992 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:30.990+0000 7fe3965e8640 1 --2- 172.21.15.114:0/50005159 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fe390100290 0x7fe3901006b0 secure :-1 s=READY pgs=378 cs=0 l=1 rev1=1 crypto rx=0x7fe380009930 tx=0x7fe38002a420 comp rx=0 tx=0).stop 2024-04-23T10:26:30.992 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:30.990+0000 7fe3965e8640 1 -- 172.21.15.114:0/50005159 shutdown_connections 2024-04-23T10:26:30.992 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:30.990+0000 7fe3965e8640 1 --2- 172.21.15.114:0/50005159 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fe390103cc0 0x7fe39010a550 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:26:30.992 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:30.990+0000 7fe3965e8640 1 --2- 172.21.15.114:0/50005159 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fe390103300 0x7fe390103780 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:26:30.992 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:30.990+0000 7fe3965e8640 1 --2- 172.21.15.114:0/50005159 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fe390100290 0x7fe3901006b0 unknown :-1 s=CLOSED pgs=378 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:26:30.992 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:30.990+0000 7fe3965e8640 1 -- 172.21.15.114:0/50005159 >> 172.21.15.114:0/50005159 conn(0x7fe3900fbbc0 msgr2=0x7fe3900fdfe0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T10:26:30.992 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:30.990+0000 7fe3965e8640 1 -- 172.21.15.114:0/50005159 shutdown_connections 2024-04-23T10:26:30.992 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:30.990+0000 7fe3965e8640 1 -- 172.21.15.114:0/50005159 wait complete. 2024-04-23T10:26:30.992 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:30.991+0000 7fe3965e8640 1 Processor -- start 2024-04-23T10:26:30.992 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:30.991+0000 7fe3965e8640 1 -- start start 2024-04-23T10:26:30.992 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:30.991+0000 7fe3965e8640 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fe390100290 0x7fe390198270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T10:26:30.992 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:30.991+0000 7fe3965e8640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fe390103300 0x7fe3901987b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T10:26:30.993 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:30.991+0000 7fe3965e8640 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fe390103cc0 0x7fe39019f820 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T10:26:30.993 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:30.991+0000 7fe3965e8640 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x7fe39010db40 con 0x7fe390103300 2024-04-23T10:26:30.993 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:30.991+0000 7fe3965e8640 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 -- 0x7fe39010d9c0 con 0x7fe390100290 2024-04-23T10:26:30.993 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:30.991+0000 7fe3965e8640 1 -- --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_getmap magic: 0 -- 0x7fe39010dcc0 con 0x7fe390103cc0 2024-04-23T10:26:30.993 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:30.991+0000 7fe38f7fe640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fe390103300 0x7fe3901987b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T10:26:30.993 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:30.991+0000 7fe38f7fe640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fe390103300 0x7fe3901987b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.114:58390/0 (socket says 172.21.15.114:58390) 2024-04-23T10:26:30.993 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:30.991+0000 7fe38f7fe640 1 -- 172.21.15.114:0/4045900068 learned_addr learned my addr 172.21.15.114:0/4045900068 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-04-23T10:26:30.993 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:30.991+0000 7fe38ffff640 1 --2- 172.21.15.114:0/4045900068 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fe390100290 0x7fe390198270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T10:26:30.993 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:30.991+0000 7fe38f7fe640 1 -- 172.21.15.114:0/4045900068 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fe390103cc0 msgr2=0x7fe39019f820 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T10:26:30.993 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:30.991+0000 7fe38f7fe640 1 --2- 172.21.15.114:0/4045900068 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fe390103cc0 0x7fe39019f820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:26:30.993 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:30.991+0000 7fe38f7fe640 1 -- 172.21.15.114:0/4045900068 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fe390100290 msgr2=0x7fe390198270 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T10:26:30.993 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:30.991+0000 7fe38f7fe640 1 --2- 172.21.15.114:0/4045900068 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fe390100290 0x7fe390198270 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:26:30.993 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:30.991+0000 7fe38f7fe640 1 -- 172.21.15.114:0/4045900068 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe39019fe90 con 0x7fe390103300 2024-04-23T10:26:30.993 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:30.991+0000 7fe38ffff640 1 --2- 172.21.15.114:0/4045900068 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fe390100290 0x7fe390198270 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-23T10:26:30.994 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:30.991+0000 7fe38f7fe640 1 --2- 172.21.15.114:0/4045900068 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fe390103300 0x7fe3901987b0 secure :-1 s=READY pgs=379 cs=0 l=1 rev1=1 crypto rx=0x7fe38400b4a0 tx=0x7fe38400b960 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T10:26:30.994 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:30.992+0000 7fe38d7fa640 1 -- 172.21.15.114:0/4045900068 <== mon.0 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe384014070 con 0x7fe390103300 2024-04-23T10:26:30.994 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:30.992+0000 7fe38d7fa640 1 -- 172.21.15.114:0/4045900068 <== mon.0 v2:172.21.15.114:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fe384010ac0 con 0x7fe390103300 2024-04-23T10:26:30.994 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:30.992+0000 7fe3965e8640 1 -- 172.21.15.114:0/4045900068 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fe3901a0180 con 0x7fe390103300 2024-04-23T10:26:30.994 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:30.992+0000 7fe38d7fa640 1 -- 172.21.15.114:0/4045900068 <== mon.0 v2:172.21.15.114:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe384017c30 con 0x7fe390103300 2024-04-23T10:26:30.994 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:30.992+0000 7fe3965e8640 1 -- 172.21.15.114:0/4045900068 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fe3901a0750 con 0x7fe390103300 2024-04-23T10:26:30.994 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:30.993+0000 7fe3965e8640 1 -- 172.21.15.114:0/4045900068 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fe390109500 con 0x7fe390103300 2024-04-23T10:26:30.995 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:30.994+0000 7fe38d7fa640 1 -- 172.21.15.114:0/4045900068 <== mon.0 v2:172.21.15.114:3300/0 4 ==== mgrmap(e 18) ==== 138647+0+0 (secure 0 0 0) 0x7fe384017070 con 0x7fe390103300 2024-04-23T10:26:30.995 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:30.994+0000 7fe38d7fa640 1 --2- 172.21.15.114:0/4045900068 >> [v2:172.21.15.114:6800/465333808,v1:172.21.15.114:6801/465333808] conn(0x7fe3780a3bb0 0x7fe3780a6070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T10:26:30.995 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:30.994+0000 7fe38ffff640 1 --2- 172.21.15.114:0/4045900068 >> [v2:172.21.15.114:6800/465333808,v1:172.21.15.114:6801/465333808] conn(0x7fe3780a3bb0 0x7fe3780a6070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T10:26:30.995 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:30.994+0000 7fe38d7fa640 1 -- 172.21.15.114:0/4045900068 <== mon.0 v2:172.21.15.114:3300/0 5 ==== osd_map(89..89 src has 1..89) ==== 8269+0+0 (secure 0 0 0) 0x7fe3840cfc00 con 0x7fe390103300 2024-04-23T10:26:30.995 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:30.995+0000 7fe38ffff640 1 --2- 172.21.15.114:0/4045900068 >> [v2:172.21.15.114:6800/465333808,v1:172.21.15.114:6801/465333808] conn(0x7fe3780a3bb0 0x7fe3780a6070 secure :-1 s=READY pgs=242 cs=0 l=1 rev1=1 crypto rx=0x7fe380006010 tx=0x7fe380000bc0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T10:26:30.997 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:30.998+0000 7fe38d7fa640 1 -- 172.21.15.114:0/4045900068 <== mon.0 v2:172.21.15.114:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7fe38409c5c0 con 0x7fe390103300 2024-04-23T10:26:31.214 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:31.215+0000 7fe3965e8640 1 -- 172.21.15.114:0/4045900068 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fe39002d830 con 0x7fe390103300 2024-04-23T10:26:31.214 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:31.216+0000 7fe38d7fa640 1 -- 172.21.15.114:0/4045900068 <== mon.0 v2:172.21.15.114:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 20 v20) ==== 94+0+11884 (secure 0 0 0) 0x7fe38409bec0 con 0x7fe390103300 2024-04-23T10:26:31.215 INFO:teuthology.orchestra.run.smithi114.stdout: 2024-04-23T10:26:31.215 INFO:teuthology.orchestra.run.smithi114.stdout:{"epoch":20,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14547,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.114:6835/663493580","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6834","nonce":663493580},{"type":"v1","addr":"172.21.15.114:6835","nonce":663493580}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14565,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.114:6837/1439252154","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6836","nonce":1439252154},{"type":"v1","addr":"172.21.15.114:6837","nonce":1439252154}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5},{"gid":14634,"name":"g","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.114:6839/3849796939","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6838","nonce":3849796939},{"type":"v1","addr":"172.21.15.114:6839","nonce":3849796939}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":11},{"gid":14646,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.114:6841/3544523891","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6840","nonce":3544523891},{"type":"v1","addr":"172.21.15.114:6841","nonce":3544523891}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":12},{"gid":24385,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.204:6833/3866561754","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6832","nonce":3866561754},{"type":"v1","addr":"172.21.15.204:6833","nonce":3866561754}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2},{"gid":24427,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.204:6835/2533793172","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6834","nonce":2533793172},{"type":"v1","addr":"172.21.15.204:6835","nonce":2533793172}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":7}],"filesystems":[{"mdsmap":{"epoch":20,"flags":50,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-04-23T10:24:43.383158+0000","modified":"2024-04-23T10:24:48.315128+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":3,"in":[0,1,2],"up":{"mds_0":24545,"mds_1":24509,"mds_2":24479},"failed":[],"damaged":[],"stopped":[],"info":{"gid_24439":{"gid":24439,"name":"c","rank":2,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.204:6837/513822179","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6836","nonce":513822179},{"type":"v1","addr":"172.21.15.204:6837","nonce":513822179}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24443":{"gid":24443,"name":"k","rank":1,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.119:6833/783244660","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6832","nonce":783244660},{"type":"v1","addr":"172.21.15.119:6833","nonce":783244660}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24463":{"gid":24463,"name":"i","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.204:6839/3222658893","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6838","nonce":3222658893},{"type":"v1","addr":"172.21.15.204:6839","nonce":3222658893}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24479":{"gid":24479,"name":"e","rank":2,"incarnation":19,"state":"up:active","state_seq":9,"addr":"172.21.15.119:6835/2033916709","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6834","nonce":2033916709},{"type":"v1","addr":"172.21.15.119:6835","nonce":2033916709}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24509":{"gid":24509,"name":"h","rank":1,"incarnation":17,"state":"up:active","state_seq":6,"addr":"172.21.15.119:6837/1872363128","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6836","nonce":1872363128},{"type":"v1","addr":"172.21.15.119:6837","nonce":1872363128}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24545":{"gid":24545,"name":"b","rank":0,"incarnation":14,"state":"up:active","state_seq":3,"addr":"172.21.15.119:6839/937297340","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6838","nonce":937297340},{"type":"v1","addr":"172.21.15.119:6839","nonce":937297340}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":24545,"qdb_cluster":[24509,24545,24479]},"id":1}]} 2024-04-23T10:26:31.215 INFO:teuthology.orchestra.run.smithi114.stderr:dumped fsmap epoch 20 2024-04-23T10:26:31.217 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:31.218+0000 7fe3965e8640 1 -- 172.21.15.114:0/4045900068 >> [v2:172.21.15.114:6800/465333808,v1:172.21.15.114:6801/465333808] conn(0x7fe3780a3bb0 msgr2=0x7fe3780a6070 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T10:26:31.217 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:31.218+0000 7fe3965e8640 1 --2- 172.21.15.114:0/4045900068 >> [v2:172.21.15.114:6800/465333808,v1:172.21.15.114:6801/465333808] conn(0x7fe3780a3bb0 0x7fe3780a6070 secure :-1 s=READY pgs=242 cs=0 l=1 rev1=1 crypto rx=0x7fe380006010 tx=0x7fe380000bc0 comp rx=0 tx=0).stop 2024-04-23T10:26:31.217 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:31.218+0000 7fe3965e8640 1 -- 172.21.15.114:0/4045900068 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fe390103300 msgr2=0x7fe3901987b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T10:26:31.217 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:31.218+0000 7fe3965e8640 1 --2- 172.21.15.114:0/4045900068 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fe390103300 0x7fe3901987b0 secure :-1 s=READY pgs=379 cs=0 l=1 rev1=1 crypto rx=0x7fe38400b4a0 tx=0x7fe38400b960 comp rx=0 tx=0).stop 2024-04-23T10:26:31.217 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:31.218+0000 7fe3965e8640 1 -- 172.21.15.114:0/4045900068 shutdown_connections 2024-04-23T10:26:31.217 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:31.218+0000 7fe3965e8640 1 --2- 172.21.15.114:0/4045900068 >> [v2:172.21.15.114:6800/465333808,v1:172.21.15.114:6801/465333808] conn(0x7fe3780a3bb0 0x7fe3780a6070 unknown :-1 s=CLOSED pgs=242 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:26:31.217 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:31.218+0000 7fe3965e8640 1 --2- 172.21.15.114:0/4045900068 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fe390103cc0 0x7fe39019f820 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:26:31.217 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:31.218+0000 7fe3965e8640 1 --2- 172.21.15.114:0/4045900068 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fe390103300 0x7fe3901987b0 unknown :-1 s=CLOSED pgs=379 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:26:31.217 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:31.218+0000 7fe3965e8640 1 --2- 172.21.15.114:0/4045900068 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fe390100290 0x7fe390198270 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:26:31.217 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:31.218+0000 7fe3965e8640 1 -- 172.21.15.114:0/4045900068 >> 172.21.15.114:0/4045900068 conn(0x7fe3900fbbc0 msgr2=0x7fe3901066d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T10:26:31.217 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:31.218+0000 7fe3965e8640 1 -- 172.21.15.114:0/4045900068 shutdown_connections 2024-04-23T10:26:31.218 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:31.218+0000 7fe3965e8640 1 -- 172.21.15.114:0/4045900068 wait complete. 2024-04-23T10:26:31.227 INFO:journalctl@ceph.mon.a.smithi114.stdout:Apr 23 10:26:31 smithi114 ceph-mon[29255]: pgmap v341: 97 pgs: 97 active+clean; 641 KiB data, 375 MiB used, 1.0 TiB / 1.0 TiB avail; 3.0 KiB/s rd, 170 B/s wr, 6 op/s 2024-04-23T10:26:31.227 INFO:journalctl@ceph.mon.a.smithi114.stdout:Apr 23 10:26:31 smithi114 ceph-mon[29255]: from='client.? 172.21.15.114:0/1129920789' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-04-23T10:26:31.227 INFO:journalctl@ceph.mon.a.smithi114.stdout:Apr 23 10:26:31 smithi114 ceph-mon[29255]: from='client.? 172.21.15.114:0/3529908104' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2024-04-23T10:26:31.227 INFO:journalctl@ceph.mon.a.smithi114.stdout:Apr 23 10:26:31 smithi114 ceph-mon[29255]: from='client.? 172.21.15.114:0/3222327275' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2024-04-23T10:26:31.228 DEBUG:teuthology.orchestra.run.smithi114:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-04-23T10:26:31.328 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:31.327+0000 7f6fa8319640 1 -- 172.21.15.114:0/2185894725 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f6fa0101fa0 msgr2=0x7f6fa01023c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T10:26:31.328 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:31.327+0000 7f6fa8319640 1 --2- 172.21.15.114:0/2185894725 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f6fa0101fa0 0x7f6fa01023c0 secure :-1 s=READY pgs=380 cs=0 l=1 rev1=1 crypto rx=0x7f6f9c009900 tx=0x7f6f9c02a410 comp rx=0 tx=0).stop 2024-04-23T10:26:31.328 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:31.327+0000 7f6fa8319640 1 -- 172.21.15.114:0/2185894725 shutdown_connections 2024-04-23T10:26:31.328 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:31.327+0000 7f6fa8319640 1 --2- 172.21.15.114:0/2185894725 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f6fa0103b50 0x7f6fa010a3e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:26:31.328 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:31.327+0000 7f6fa8319640 1 --2- 172.21.15.114:0/2185894725 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f6fa0103190 0x7f6fa0103610 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:26:31.328 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:31.327+0000 7f6fa8319640 1 --2- 172.21.15.114:0/2185894725 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f6fa0101fa0 0x7f6fa01023c0 unknown :-1 s=CLOSED pgs=380 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:26:31.328 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:31.327+0000 7f6fa8319640 1 -- 172.21.15.114:0/2185894725 >> 172.21.15.114:0/2185894725 conn(0x7f6fa00fd550 msgr2=0x7f6fa00ff970 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T10:26:31.328 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:31.327+0000 7f6fa8319640 1 -- 172.21.15.114:0/2185894725 shutdown_connections 2024-04-23T10:26:31.329 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:31.327+0000 7f6fa8319640 1 -- 172.21.15.114:0/2185894725 wait complete. 2024-04-23T10:26:31.329 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:31.327+0000 7f6fa8319640 1 Processor -- start 2024-04-23T10:26:31.329 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:31.327+0000 7f6fa8319640 1 -- start start 2024-04-23T10:26:31.329 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:31.327+0000 7f6fa8319640 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f6fa0101fa0 0x7f6fa0198210 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T10:26:31.329 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:31.327+0000 7f6fa8319640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f6fa0103190 0x7f6fa0198750 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T10:26:31.329 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:31.327+0000 7f6fa8319640 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f6fa0103b50 0x7f6fa019f7c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T10:26:31.329 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:31.327+0000 7f6fa8319640 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x7f6fa010db20 con 0x7f6fa0103190 2024-04-23T10:26:31.329 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:31.328+0000 7f6fa8319640 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 -- 0x7f6fa010d9a0 con 0x7f6fa0103b50 2024-04-23T10:26:31.329 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:31.328+0000 7f6fa8319640 1 -- --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_getmap magic: 0 -- 0x7f6fa010dca0 con 0x7f6fa0101fa0 2024-04-23T10:26:31.329 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:31.328+0000 7f6fa58b4640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f6fa0103190 0x7f6fa0198750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T10:26:31.329 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:31.328+0000 7f6fa58b4640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f6fa0103190 0x7f6fa0198750 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.114:58416/0 (socket says 172.21.15.114:58416) 2024-04-23T10:26:31.329 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:31.328+0000 7f6fa58b4640 1 -- 172.21.15.114:0/3605219822 learned_addr learned my addr 172.21.15.114:0/3605219822 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-04-23T10:26:31.329 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:31.328+0000 7f6fa58b4640 1 -- 172.21.15.114:0/3605219822 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f6fa0101fa0 msgr2=0x7f6fa0198210 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T10:26:31.329 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:31.328+0000 7f6fa58b4640 1 --2- 172.21.15.114:0/3605219822 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f6fa0101fa0 0x7f6fa0198210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:26:31.330 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:31.328+0000 7f6fa58b4640 1 -- 172.21.15.114:0/3605219822 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f6fa0103b50 msgr2=0x7f6fa019f7c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T10:26:31.330 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:31.328+0000 7f6fa58b4640 1 --2- 172.21.15.114:0/3605219822 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f6fa0103b50 0x7f6fa019f7c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:26:31.330 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:31.328+0000 7f6fa58b4640 1 -- 172.21.15.114:0/3605219822 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f6fa019fe30 con 0x7f6fa0103190 2024-04-23T10:26:31.330 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:31.328+0000 7f6fa58b4640 1 --2- 172.21.15.114:0/3605219822 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f6fa0103190 0x7f6fa0198750 secure :-1 s=READY pgs=381 cs=0 l=1 rev1=1 crypto rx=0x7f6f9000b4a0 tx=0x7f6f9000b960 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T10:26:31.330 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:31.328+0000 7f6f977fe640 1 -- 172.21.15.114:0/3605219822 <== mon.0 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6f90014070 con 0x7f6fa0103190 2024-04-23T10:26:31.330 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:31.328+0000 7f6f977fe640 1 -- 172.21.15.114:0/3605219822 <== mon.0 v2:172.21.15.114:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f6f90010ac0 con 0x7f6fa0103190 2024-04-23T10:26:31.330 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:31.328+0000 7f6fa8319640 1 -- 172.21.15.114:0/3605219822 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f6fa01a0120 con 0x7f6fa0103190 2024-04-23T10:26:31.330 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:31.328+0000 7f6f977fe640 1 -- 172.21.15.114:0/3605219822 <== mon.0 v2:172.21.15.114:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6f90017c30 con 0x7f6fa0103190 2024-04-23T10:26:31.330 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:31.328+0000 7f6fa8319640 1 -- 172.21.15.114:0/3605219822 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f6fa01a06f0 con 0x7f6fa0103190 2024-04-23T10:26:31.330 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:31.329+0000 7f6fa8319640 1 -- 172.21.15.114:0/3605219822 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f6fa01092c0 con 0x7f6fa0103190 2024-04-23T10:26:31.331 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:31.330+0000 7f6f977fe640 1 -- 172.21.15.114:0/3605219822 <== mon.0 v2:172.21.15.114:3300/0 4 ==== mgrmap(e 18) ==== 138647+0+0 (secure 0 0 0) 0x7f6f90017070 con 0x7f6fa0103190 2024-04-23T10:26:31.331 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:31.331+0000 7f6f977fe640 1 --2- 172.21.15.114:0/3605219822 >> [v2:172.21.15.114:6800/465333808,v1:172.21.15.114:6801/465333808] conn(0x7f6f8c0a3bb0 0x7f6f8c0a6070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T10:26:31.331 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:31.331+0000 7f6fa60b5640 1 --2- 172.21.15.114:0/3605219822 >> [v2:172.21.15.114:6800/465333808,v1:172.21.15.114:6801/465333808] conn(0x7f6f8c0a3bb0 0x7f6f8c0a6070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T10:26:31.331 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:31.331+0000 7f6f977fe640 1 -- 172.21.15.114:0/3605219822 <== mon.0 v2:172.21.15.114:3300/0 5 ==== osd_map(89..89 src has 1..89) ==== 8269+0+0 (secure 0 0 0) 0x7f6f900cfc00 con 0x7f6fa0103190 2024-04-23T10:26:31.331 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:31.331+0000 7f6fa60b5640 1 --2- 172.21.15.114:0/3605219822 >> [v2:172.21.15.114:6800/465333808,v1:172.21.15.114:6801/465333808] conn(0x7f6f8c0a3bb0 0x7f6f8c0a6070 secure :-1 s=READY pgs=243 cs=0 l=1 rev1=1 crypto rx=0x7f6f9c0096c0 tx=0x7f6f9c000bc0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T10:26:31.333 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:31.335+0000 7f6f977fe640 1 -- 172.21.15.114:0/3605219822 <== mon.0 v2:172.21.15.114:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f6f9009c5c0 con 0x7f6fa0103190 2024-04-23T10:26:31.492 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 23 10:26:31 smithi119 ceph-mon[32608]: pgmap v341: 97 pgs: 97 active+clean; 641 KiB data, 375 MiB used, 1.0 TiB / 1.0 TiB avail; 3.0 KiB/s rd, 170 B/s wr, 6 op/s 2024-04-23T10:26:31.492 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 23 10:26:31 smithi119 ceph-mon[32608]: from='client.? 172.21.15.114:0/1129920789' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-04-23T10:26:31.492 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 23 10:26:31 smithi119 ceph-mon[32608]: from='client.? 172.21.15.114:0/3529908104' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2024-04-23T10:26:31.492 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 23 10:26:31 smithi119 ceph-mon[32608]: from='client.? 172.21.15.114:0/3222327275' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2024-04-23T10:26:31.513 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:31.514+0000 7f6fa8319640 1 -- 172.21.15.114:0/3605219822 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f6fa002d870 con 0x7f6fa0103190 2024-04-23T10:26:31.514 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:31.515+0000 7f6f977fe640 1 -- 172.21.15.114:0/3605219822 <== mon.0 v2:172.21.15.114:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v89) ==== 74+0+22818 (secure 0 0 0) 0x7f6f9009bec0 con 0x7f6fa0103190 2024-04-23T10:26:31.515 INFO:teuthology.orchestra.run.smithi114.stdout: 2024-04-23T10:26:31.515 INFO:teuthology.orchestra.run.smithi114.stdout:{"epoch":89,"fsid":"aa240572-0159-11ef-bc93-c7b262605968","created":"2024-04-23T10:16:53.351136+0000","modified":"2024-04-23T10:25:47.573901+0000","last_up_change":"2024-04-23T10:23:54.153269+0000","last_in_change":"2024-04-23T10:23:32.963085+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":26,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-04-23T10:20:11.970297+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":12,"score_stable":12,"optimal_score":0.25,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-04-23T10:24:41.218540+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"86","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.8799999952316284,"score_stable":1.8799999952316284,"optimal_score":1,"raw_score_acting":1.8799999952316284,"raw_score_stable":1.8799999952316284,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-04-23T10:24:42.341799+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"87","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"fd64adde-9445-45a4-a171-ef656e2c5709","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":88,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6802","nonce":736366878},{"type":"v1","addr":"172.21.15.114:6803","nonce":736366878}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6804","nonce":736366878},{"type":"v1","addr":"172.21.15.114:6805","nonce":736366878}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6808","nonce":736366878},{"type":"v1","addr":"172.21.15.114:6809","nonce":736366878}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6806","nonce":736366878},{"type":"v1","addr":"172.21.15.114:6807","nonce":736366878}]},"public_addr":"172.21.15.114:6803/736366878","cluster_addr":"172.21.15.114:6805/736366878","heartbeat_back_addr":"172.21.15.114:6809/736366878","heartbeat_front_addr":"172.21.15.114:6807/736366878","state":["exists","up"]},{"osd":1,"uuid":"e279a1e6-2702-472a-930a-e08ece76a861","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":85,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6800","nonce":1133540779},{"type":"v1","addr":"172.21.15.119:6801","nonce":1133540779}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6802","nonce":1133540779},{"type":"v1","addr":"172.21.15.119:6803","nonce":1133540779}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6806","nonce":1133540779},{"type":"v1","addr":"172.21.15.119:6807","nonce":1133540779}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6804","nonce":1133540779},{"type":"v1","addr":"172.21.15.119:6805","nonce":1133540779}]},"public_addr":"172.21.15.119:6801/1133540779","cluster_addr":"172.21.15.119:6803/1133540779","heartbeat_back_addr":"172.21.15.119:6807/1133540779","heartbeat_front_addr":"172.21.15.119:6805/1133540779","state":["exists","up"]},{"osd":2,"uuid":"f4c9f2b4-ff49-4e14-89cd-b477feeb9382","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":85,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6800","nonce":799652511},{"type":"v1","addr":"172.21.15.204:6801","nonce":799652511}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6802","nonce":799652511},{"type":"v1","addr":"172.21.15.204:6803","nonce":799652511}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6806","nonce":799652511},{"type":"v1","addr":"172.21.15.204:6807","nonce":799652511}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6804","nonce":799652511},{"type":"v1","addr":"172.21.15.204:6805","nonce":799652511}]},"public_addr":"172.21.15.204:6801/799652511","cluster_addr":"172.21.15.204:6803/799652511","heartbeat_back_addr":"172.21.15.204:6807/799652511","heartbeat_front_addr":"172.21.15.204:6805/799652511","state":["exists","up"]},{"osd":3,"uuid":"52d2a566-e71c-4bb3-8d6c-5abdd43177c2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":25,"up_thru":88,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6810","nonce":4275595091},{"type":"v1","addr":"172.21.15.114:6811","nonce":4275595091}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6812","nonce":4275595091},{"type":"v1","addr":"172.21.15.114:6813","nonce":4275595091}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6816","nonce":4275595091},{"type":"v1","addr":"172.21.15.114:6817","nonce":4275595091}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6814","nonce":4275595091},{"type":"v1","addr":"172.21.15.114:6815","nonce":4275595091}]},"public_addr":"172.21.15.114:6811/4275595091","cluster_addr":"172.21.15.114:6813/4275595091","heartbeat_back_addr":"172.21.15.114:6817/4275595091","heartbeat_front_addr":"172.21.15.114:6815/4275595091","state":["exists","up"]},{"osd":4,"uuid":"d911a7c1-4bdc-4d28-b8a6-73896a6e7de6","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":33,"up_thru":85,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6808","nonce":1611007172},{"type":"v1","addr":"172.21.15.119:6809","nonce":1611007172}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6810","nonce":1611007172},{"type":"v1","addr":"172.21.15.119:6811","nonce":1611007172}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6814","nonce":1611007172},{"type":"v1","addr":"172.21.15.119:6815","nonce":1611007172}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6812","nonce":1611007172},{"type":"v1","addr":"172.21.15.119:6813","nonce":1611007172}]},"public_addr":"172.21.15.119:6809/1611007172","cluster_addr":"172.21.15.119:6811/1611007172","heartbeat_back_addr":"172.21.15.119:6815/1611007172","heartbeat_front_addr":"172.21.15.119:6813/1611007172","state":["exists","up"]},{"osd":5,"uuid":"f7590e1a-e93c-4c29-b15f-38ee52d1f921","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":39,"up_thru":85,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6808","nonce":1427563312},{"type":"v1","addr":"172.21.15.204:6809","nonce":1427563312}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6810","nonce":1427563312},{"type":"v1","addr":"172.21.15.204:6811","nonce":1427563312}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6814","nonce":1427563312},{"type":"v1","addr":"172.21.15.204:6815","nonce":1427563312}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6812","nonce":1427563312},{"type":"v1","addr":"172.21.15.204:6813","nonce":1427563312}]},"public_addr":"172.21.15.204:6809/1427563312","cluster_addr":"172.21.15.204:6811/1427563312","heartbeat_back_addr":"172.21.15.204:6815/1427563312","heartbeat_front_addr":"172.21.15.204:6813/1427563312","state":["exists","up"]},{"osd":6,"uuid":"534d20e8-3e54-4db8-a845-6119f49c7a78","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":47,"up_thru":88,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6818","nonce":4072912544},{"type":"v1","addr":"172.21.15.114:6819","nonce":4072912544}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6820","nonce":4072912544},{"type":"v1","addr":"172.21.15.114:6821","nonce":4072912544}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6824","nonce":4072912544},{"type":"v1","addr":"172.21.15.114:6825","nonce":4072912544}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6822","nonce":4072912544},{"type":"v1","addr":"172.21.15.114:6823","nonce":4072912544}]},"public_addr":"172.21.15.114:6819/4072912544","cluster_addr":"172.21.15.114:6821/4072912544","heartbeat_back_addr":"172.21.15.114:6825/4072912544","heartbeat_front_addr":"172.21.15.114:6823/4072912544","state":["exists","up"]},{"osd":7,"uuid":"f0ec22b8-1b48-4859-9d65-b00403aff3ad","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":53,"up_thru":88,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6816","nonce":2237458793},{"type":"v1","addr":"172.21.15.119:6817","nonce":2237458793}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6818","nonce":2237458793},{"type":"v1","addr":"172.21.15.119:6819","nonce":2237458793}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6822","nonce":2237458793},{"type":"v1","addr":"172.21.15.119:6823","nonce":2237458793}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6820","nonce":2237458793},{"type":"v1","addr":"172.21.15.119:6821","nonce":2237458793}]},"public_addr":"172.21.15.119:6817/2237458793","cluster_addr":"172.21.15.119:6819/2237458793","heartbeat_back_addr":"172.21.15.119:6823/2237458793","heartbeat_front_addr":"172.21.15.119:6821/2237458793","state":["exists","up"]},{"osd":8,"uuid":"3ac89686-6d76-4cef-9089-ac29e4b3bcac","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":62,"up_thru":85,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6816","nonce":1426318231},{"type":"v1","addr":"172.21.15.204:6817","nonce":1426318231}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6818","nonce":1426318231},{"type":"v1","addr":"172.21.15.204:6819","nonce":1426318231}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6822","nonce":1426318231},{"type":"v1","addr":"172.21.15.204:6823","nonce":1426318231}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6820","nonce":1426318231},{"type":"v1","addr":"172.21.15.204:6821","nonce":1426318231}]},"public_addr":"172.21.15.204:6817/1426318231","cluster_addr":"172.21.15.204:6819/1426318231","heartbeat_back_addr":"172.21.15.204:6823/1426318231","heartbeat_front_addr":"172.21.15.204:6821/1426318231","state":["exists","up"]},{"osd":9,"uuid":"a80ffe4d-1e98-460f-b046-c4754285cc5a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":69,"up_thru":88,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6826","nonce":2750635721},{"type":"v1","addr":"172.21.15.114:6827","nonce":2750635721}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6828","nonce":2750635721},{"type":"v1","addr":"172.21.15.114:6829","nonce":2750635721}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6832","nonce":2750635721},{"type":"v1","addr":"172.21.15.114:6833","nonce":2750635721}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6830","nonce":2750635721},{"type":"v1","addr":"172.21.15.114:6831","nonce":2750635721}]},"public_addr":"172.21.15.114:6827/2750635721","cluster_addr":"172.21.15.114:6829/2750635721","heartbeat_back_addr":"172.21.15.114:6833/2750635721","heartbeat_front_addr":"172.21.15.114:6831/2750635721","state":["exists","up"]},{"osd":10,"uuid":"cd75a579-bf12-4600-ac09-a8bfce772ced","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":74,"up_thru":85,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6824","nonce":1400187701},{"type":"v1","addr":"172.21.15.119:6825","nonce":1400187701}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6826","nonce":1400187701},{"type":"v1","addr":"172.21.15.119:6827","nonce":1400187701}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6830","nonce":1400187701},{"type":"v1","addr":"172.21.15.119:6831","nonce":1400187701}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6828","nonce":1400187701},{"type":"v1","addr":"172.21.15.119:6829","nonce":1400187701}]},"public_addr":"172.21.15.119:6825/1400187701","cluster_addr":"172.21.15.119:6827/1400187701","heartbeat_back_addr":"172.21.15.119:6831/1400187701","heartbeat_front_addr":"172.21.15.119:6829/1400187701","state":["exists","up"]},{"osd":11,"uuid":"53f037a5-3ec0-483a-80a1-dfee923a68e5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":82,"up_thru":88,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6824","nonce":2407298490},{"type":"v1","addr":"172.21.15.204:6825","nonce":2407298490}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6826","nonce":2407298490},{"type":"v1","addr":"172.21.15.204:6827","nonce":2407298490}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6830","nonce":2407298490},{"type":"v1","addr":"172.21.15.204:6831","nonce":2407298490}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6828","nonce":2407298490},{"type":"v1","addr":"172.21.15.204:6829","nonce":2407298490}]},"public_addr":"172.21.15.204:6825/2407298490","cluster_addr":"172.21.15.204:6827/2407298490","heartbeat_back_addr":"172.21.15.204:6831/2407298490","heartbeat_front_addr":"172.21.15.204:6829/2407298490","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-23T10:19:18.401147+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-23T10:19:42.611299+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-23T10:20:07.277367+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-23T10:20:31.208368+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-23T10:20:55.866225+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-23T10:21:19.961237+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-23T10:21:44.480783+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-23T10:22:09.173641+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-23T10:22:34.030328+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-23T10:22:59.652293+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-23T10:23:25.659673+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-23T10:23:51.168171+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"2.e","mappings":[{"from":2,"to":4}]},{"pgid":"2.1d","mappings":[{"from":8,"to":11}]},{"pgid":"3.e","mappings":[{"from":5,"to":9}]},{"pgid":"3.12","mappings":[{"from":3,"to":9}]},{"pgid":"3.17","mappings":[{"from":4,"to":6}]},{"pgid":"3.23","mappings":[{"from":4,"to":7}]},{"pgid":"3.27","mappings":[{"from":4,"to":6}]},{"pgid":"3.31","mappings":[{"from":4,"to":8}]},{"pgid":"3.38","mappings":[{"from":3,"to":9}]},{"pgid":"3.39","mappings":[{"from":5,"to":8}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.114:6801/1604678557":"2024-04-24T10:17:45.874609+0000","172.21.15.114:0/4025385854":"2024-04-24T10:17:45.874609+0000","172.21.15.114:0/2228273469":"2024-04-24T10:17:19.910348+0000","172.21.15.114:6800/2831530838":"2024-04-24T10:17:19.910348+0000","172.21.15.114:0/1333447450":"2024-04-24T10:17:45.874609+0000","172.21.15.114:0/2233026840":"2024-04-24T10:17:19.910348+0000","172.21.15.114:0/1902281545":"2024-04-24T10:17:19.910348+0000","172.21.15.114:6801/2831530838":"2024-04-24T10:17:19.910348+0000","172.21.15.114:6800/1604678557":"2024-04-24T10:17:45.874609+0000","172.21.15.114:0/1623553811":"2024-04-24T10:17:45.874609+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-23T10:26:31.517 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:31.518+0000 7f6fa8319640 1 -- 172.21.15.114:0/3605219822 >> [v2:172.21.15.114:6800/465333808,v1:172.21.15.114:6801/465333808] conn(0x7f6f8c0a3bb0 msgr2=0x7f6f8c0a6070 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T10:26:31.517 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:31.518+0000 7f6fa8319640 1 --2- 172.21.15.114:0/3605219822 >> [v2:172.21.15.114:6800/465333808,v1:172.21.15.114:6801/465333808] conn(0x7f6f8c0a3bb0 0x7f6f8c0a6070 secure :-1 s=READY pgs=243 cs=0 l=1 rev1=1 crypto rx=0x7f6f9c0096c0 tx=0x7f6f9c000bc0 comp rx=0 tx=0).stop 2024-04-23T10:26:31.517 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:31.518+0000 7f6fa8319640 1 -- 172.21.15.114:0/3605219822 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f6fa0103190 msgr2=0x7f6fa0198750 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T10:26:31.517 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:31.518+0000 7f6fa8319640 1 --2- 172.21.15.114:0/3605219822 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f6fa0103190 0x7f6fa0198750 secure :-1 s=READY pgs=381 cs=0 l=1 rev1=1 crypto rx=0x7f6f9000b4a0 tx=0x7f6f9000b960 comp rx=0 tx=0).stop 2024-04-23T10:26:31.517 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:31.518+0000 7f6fa8319640 1 -- 172.21.15.114:0/3605219822 shutdown_connections 2024-04-23T10:26:31.517 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:31.518+0000 7f6fa8319640 1 --2- 172.21.15.114:0/3605219822 >> [v2:172.21.15.114:6800/465333808,v1:172.21.15.114:6801/465333808] conn(0x7f6f8c0a3bb0 0x7f6f8c0a6070 unknown :-1 s=CLOSED pgs=243 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:26:31.517 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:31.518+0000 7f6fa8319640 1 --2- 172.21.15.114:0/3605219822 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f6fa0103b50 0x7f6fa019f7c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:26:31.517 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:31.518+0000 7f6fa8319640 1 --2- 172.21.15.114:0/3605219822 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f6fa0103190 0x7f6fa0198750 unknown :-1 s=CLOSED pgs=381 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:26:31.517 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:31.518+0000 7f6fa8319640 1 --2- 172.21.15.114:0/3605219822 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f6fa0101fa0 0x7f6fa0198210 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:26:31.517 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:31.518+0000 7f6fa8319640 1 -- 172.21.15.114:0/3605219822 >> 172.21.15.114:0/3605219822 conn(0x7f6fa00fd550 msgr2=0x7f6fa0106490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T10:26:31.517 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:31.518+0000 7f6fa8319640 1 -- 172.21.15.114:0/3605219822 shutdown_connections 2024-04-23T10:26:31.517 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:31.518+0000 7f6fa8319640 1 -- 172.21.15.114:0/3605219822 wait complete. 2024-04-23T10:26:31.528 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 20, 'flags': 50, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': True, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 32, 'explicitly_allowed_features': 32, 'created': '2024-04-23T10:24:43.383158+0000', 'modified': '2024-04-23T10:24:48.315128+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 300, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}, 'max_mds': 3, 'in': [0, 1, 2], 'up': {'mds_0': 24545, 'mds_1': 24509, 'mds_2': 24479}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_24439': {'gid': 24439, 'name': 'c', 'rank': 2, 'incarnation': 0, 'state': 'up:standby-replay', 'state_seq': 1, 'addr': '172.21.15.204:6837/513822179', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.204:6836', 'nonce': 513822179}, {'type': 'v1', 'addr': '172.21.15.204:6837', 'nonce': 513822179}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_24443': {'gid': 24443, 'name': 'k', 'rank': 1, 'incarnation': 0, 'state': 'up:standby-replay', 'state_seq': 1, 'addr': '172.21.15.119:6833/783244660', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.119:6832', 'nonce': 783244660}, {'type': 'v1', 'addr': '172.21.15.119:6833', 'nonce': 783244660}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_24463': {'gid': 24463, 'name': 'i', 'rank': 0, 'incarnation': 0, 'state': 'up:standby-replay', 'state_seq': 1, 'addr': '172.21.15.204:6839/3222658893', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.204:6838', 'nonce': 3222658893}, {'type': 'v1', 'addr': '172.21.15.204:6839', 'nonce': 3222658893}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_24479': {'gid': 24479, 'name': 'e', 'rank': 2, 'incarnation': 19, 'state': 'up:active', 'state_seq': 9, 'addr': '172.21.15.119:6835/2033916709', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.119:6834', 'nonce': 2033916709}, {'type': 'v1', 'addr': '172.21.15.119:6835', 'nonce': 2033916709}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_24509': {'gid': 24509, 'name': 'h', 'rank': 1, 'incarnation': 17, 'state': 'up:active', 'state_seq': 6, 'addr': '172.21.15.119:6837/1872363128', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.119:6836', 'nonce': 1872363128}, {'type': 'v1', 'addr': '172.21.15.119:6837', 'nonce': 1872363128}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_24545': {'gid': 24545, 'name': 'b', 'rank': 0, 'incarnation': 14, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.15.119:6839/937297340', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.119:6838', 'nonce': 937297340}, {'type': 'v1', 'addr': '172.21.15.119:6839', 'nonce': 937297340}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}}, 'data_pools': [3], 'metadata_pool': 2, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 24545, 'qdb_cluster': [24509, 24545, 24479]} 2024-04-23T10:26:31.529 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: 3/3 2024-04-23T10:26:31.529 DEBUG:teuthology.orchestra.run.smithi114:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.e status 2024-04-23T10:26:31.578 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 23 10:26:31 smithi204 ceph-mon[32649]: pgmap v341: 97 pgs: 97 active+clean; 641 KiB data, 375 MiB used, 1.0 TiB / 1.0 TiB avail; 3.0 KiB/s rd, 170 B/s wr, 6 op/s 2024-04-23T10:26:31.578 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 23 10:26:31 smithi204 ceph-mon[32649]: from='client.? 172.21.15.114:0/1129920789' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-04-23T10:26:31.578 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 23 10:26:31 smithi204 ceph-mon[32649]: from='client.? 172.21.15.114:0/3529908104' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2024-04-23T10:26:31.578 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 23 10:26:31 smithi204 ceph-mon[32649]: from='client.? 172.21.15.114:0/3222327275' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2024-04-23T10:26:31.628 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:31.628+0000 7f253f708640 1 -- 172.21.15.114:0/3944537853 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f2538102080 msgr2=0x7f2538102500 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T10:26:31.629 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:31.628+0000 7f253f708640 1 --2- 172.21.15.114:0/3944537853 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f2538102080 0x7f2538102500 secure :-1 s=READY pgs=382 cs=0 l=1 rev1=1 crypto rx=0x7f2534009900 tx=0x7f253402a410 comp rx=0 tx=0).stop 2024-04-23T10:26:31.629 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:31.628+0000 7f253f708640 1 -- 172.21.15.114:0/3944537853 shutdown_connections 2024-04-23T10:26:31.629 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:31.628+0000 7f253f708640 1 --2- 172.21.15.114:0/3944537853 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f2538102a40 0x7f2538109360 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:26:31.629 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:31.628+0000 7f253f708640 1 --2- 172.21.15.114:0/3944537853 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f2538102080 0x7f2538102500 unknown :-1 s=CLOSED pgs=382 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:26:31.629 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:31.628+0000 7f253f708640 1 --2- 172.21.15.114:0/3944537853 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f2538100f30 0x7f2538101350 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:26:31.629 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:31.628+0000 7f253f708640 1 -- 172.21.15.114:0/3944537853 >> 172.21.15.114:0/3944537853 conn(0x7f25380fc4a0 msgr2=0x7f25380fe8e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T10:26:31.629 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:31.628+0000 7f253f708640 1 -- 172.21.15.114:0/3944537853 shutdown_connections 2024-04-23T10:26:31.629 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:31.628+0000 7f253f708640 1 -- 172.21.15.114:0/3944537853 wait complete. 2024-04-23T10:26:31.629 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:31.628+0000 7f253f708640 1 Processor -- start 2024-04-23T10:26:31.629 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:31.628+0000 7f253f708640 1 -- start start 2024-04-23T10:26:31.629 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:31.628+0000 7f253f708640 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f2538100f30 0x7f253806b160 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T10:26:31.629 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:31.629+0000 7f253f708640 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f2538102080 0x7f253819cf40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T10:26:31.629 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:31.629+0000 7f253f708640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f2538102a40 0x7f253819f5b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T10:26:31.629 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:31.629+0000 7f253f708640 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x7f253810cae0 con 0x7f2538102a40 2024-04-23T10:26:31.630 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:31.629+0000 7f253f708640 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 -- 0x7f253810c960 con 0x7f2538100f30 2024-04-23T10:26:31.630 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:31.629+0000 7f253f708640 1 -- --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_getmap magic: 0 -- 0x7f253810cc60 con 0x7f2538102080 2024-04-23T10:26:31.630 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:31.629+0000 7f253dca5640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f2538102a40 0x7f253819f5b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T10:26:31.630 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:31.629+0000 7f253dca5640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f2538102a40 0x7f253819f5b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.114:58432/0 (socket says 172.21.15.114:58432) 2024-04-23T10:26:31.630 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:31.629+0000 7f253dca5640 1 -- 172.21.15.114:0/2227654281 learned_addr learned my addr 172.21.15.114:0/2227654281 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-04-23T10:26:31.630 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:31.629+0000 7f253cca3640 1 --2- 172.21.15.114:0/2227654281 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f2538102080 0x7f253819cf40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T10:26:31.630 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:31.629+0000 7f253dca5640 1 -- 172.21.15.114:0/2227654281 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f2538102080 msgr2=0x7f253819cf40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T10:26:31.630 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:31.629+0000 7f253dca5640 1 --2- 172.21.15.114:0/2227654281 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f2538102080 0x7f253819cf40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:26:31.630 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:31.629+0000 7f253dca5640 1 -- 172.21.15.114:0/2227654281 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f2538100f30 msgr2=0x7f253806b160 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T10:26:31.630 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:31.629+0000 7f253dca5640 1 --2- 172.21.15.114:0/2227654281 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f2538100f30 0x7f253806b160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:26:31.630 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:31.629+0000 7f253dca5640 1 -- 172.21.15.114:0/2227654281 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f253819fe70 con 0x7f2538102a40 2024-04-23T10:26:31.630 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:31.629+0000 7f253dca5640 1 --2- 172.21.15.114:0/2227654281 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f2538102a40 0x7f253819f5b0 secure :-1 s=READY pgs=383 cs=0 l=1 rev1=1 crypto rx=0x7f253000cce0 tx=0x7f253000a620 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T10:26:31.630 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:31.629+0000 7f252a7fc640 1 -- 172.21.15.114:0/2227654281 <== mon.0 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f253000f450 con 0x7f2538102a40 2024-04-23T10:26:31.631 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:31.629+0000 7f252a7fc640 1 -- 172.21.15.114:0/2227654281 <== mon.0 v2:172.21.15.114:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f253000faa0 con 0x7f2538102a40 2024-04-23T10:26:31.631 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:31.629+0000 7f253f708640 1 -- 172.21.15.114:0/2227654281 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f25381a0160 con 0x7f2538102a40 2024-04-23T10:26:31.631 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:31.629+0000 7f252a7fc640 1 -- 172.21.15.114:0/2227654281 <== mon.0 v2:172.21.15.114:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2530018b70 con 0x7f2538102a40 2024-04-23T10:26:31.631 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:31.629+0000 7f253f708640 1 -- 172.21.15.114:0/2227654281 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f25381a0630 con 0x7f2538102a40 2024-04-23T10:26:31.637 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:31.631+0000 7f252a7fc640 1 -- 172.21.15.114:0/2227654281 <== mon.0 v2:172.21.15.114:3300/0 4 ==== mgrmap(e 18) ==== 138647+0+0 (secure 0 0 0) 0x7f2530018d10 con 0x7f2538102a40 2024-04-23T10:26:31.638 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:31.632+0000 7f2542428740 1 Processor -- start 2024-04-23T10:26:31.638 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:31.632+0000 7f2542428740 1 -- start start 2024-04-23T10:26:31.638 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:31.632+0000 7f2542428740 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55f5de898b80 0x55f5de898fa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T10:26:31.638 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:31.632+0000 7f2542428740 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x55f5de899d50 0x55f5de89a1d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T10:26:31.638 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:31.632+0000 7f2542428740 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x55f5de89a710 0x55f5de8a0fa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T10:26:31.638 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:31.632+0000 7f252a7fc640 1 --2- 172.21.15.114:0/2227654281 >> [v2:172.21.15.114:6800/465333808,v1:172.21.15.114:6801/465333808] conn(0x7f25240a3bb0 0x7f25240a6070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T10:26:31.638 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:31.632+0000 7f2542428740 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x55f5de7fdc50 con 0x55f5de898b80 2024-04-23T10:26:31.638 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:31.632+0000 7f2542428740 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 -- 0x55f5de79a830 con 0x55f5de89a710 2024-04-23T10:26:31.638 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:31.632+0000 7f2542428740 1 -- --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_getmap magic: 0 -- 0x55f5de7f9c10 con 0x55f5de899d50 2024-04-23T10:26:31.638 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:31.632+0000 7f253cca3640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55f5de898b80 0x55f5de898fa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T10:26:31.638 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:31.632+0000 7f253cca3640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55f5de898b80 0x55f5de898fa0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.114:58448/0 (socket says 172.21.15.114:58448) 2024-04-23T10:26:31.638 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:31.632+0000 7f253cca3640 1 -- 172.21.15.114:0/2920597128 learned_addr learned my addr 172.21.15.114:0/2920597128 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-04-23T10:26:31.639 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:31.632+0000 7f253dca5640 1 --2- 172.21.15.114:0/2227654281 >> [v2:172.21.15.114:6800/465333808,v1:172.21.15.114:6801/465333808] conn(0x7f25240a3bb0 0x7f25240a6070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T10:26:31.639 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:31.632+0000 7f252a7fc640 1 -- 172.21.15.114:0/2227654281 <== mon.0 v2:172.21.15.114:3300/0 5 ==== osd_map(89..89 src has 1..89) ==== 8269+0+0 (secure 0 0 0) 0x7f25300d0910 con 0x7f2538102a40 2024-04-23T10:26:31.639 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:31.632+0000 7f253d4a4640 1 --2- 172.21.15.114:0/2920597128 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x55f5de899d50 0x55f5de89a1d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T10:26:31.639 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:31.632+0000 7f253cca3640 1 --2- 172.21.15.114:0/2920597128 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x55f5de89a710 0x55f5de8a0fa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T10:26:31.639 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:31.632+0000 7f253cca3640 1 -- 172.21.15.114:0/2920597128 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x55f5de899d50 msgr2=0x55f5de89a1d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T10:26:31.639 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:31.632+0000 7f253cca3640 1 --2- 172.21.15.114:0/2920597128 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x55f5de899d50 0x55f5de89a1d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:26:31.639 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:31.632+0000 7f253cca3640 1 -- 172.21.15.114:0/2920597128 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x55f5de89a710 msgr2=0x55f5de8a0fa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T10:26:31.639 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:31.632+0000 7f253cca3640 1 --2- 172.21.15.114:0/2920597128 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x55f5de89a710 0x55f5de8a0fa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:26:31.639 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:31.632+0000 7f253cca3640 1 -- 172.21.15.114:0/2920597128 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55f5de5102d0 con 0x55f5de898b80 2024-04-23T10:26:31.639 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:31.632+0000 7f253dca5640 1 --2- 172.21.15.114:0/2227654281 >> [v2:172.21.15.114:6800/465333808,v1:172.21.15.114:6801/465333808] conn(0x7f25240a3bb0 0x7f25240a6070 secure :-1 s=READY pgs=244 cs=0 l=1 rev1=1 crypto rx=0x7f25300a5310 tx=0x7f2530067220 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T10:26:31.639 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:31.632+0000 7f253cca3640 1 --2- 172.21.15.114:0/2920597128 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55f5de898b80 0x55f5de898fa0 secure :-1 s=READY pgs=384 cs=0 l=1 rev1=1 crypto rx=0x7f2534031500 tx=0x7f25340058b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=669b29c68f3c585f server_cookie=0 in_seq=0 out_seq=0 2024-04-23T10:26:31.639 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:31.633+0000 7f25237fe640 1 -- 172.21.15.114:0/2920597128 <== mon.0 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2534044020 con 0x55f5de898b80 2024-04-23T10:26:31.639 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:31.633+0000 7f25237fe640 1 -- 172.21.15.114:0/2920597128 <== mon.0 v2:172.21.15.114:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f25340035f0 con 0x55f5de898b80 2024-04-23T10:26:31.639 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:31.633+0000 7f25237fe640 1 -- 172.21.15.114:0/2920597128 <== mon.0 v2:172.21.15.114:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2534035070 con 0x55f5de898b80 2024-04-23T10:26:31.640 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:31.633+0000 7f2542428740 1 -- 172.21.15.114:0/2920597128 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55f5de898b80 msgr2=0x55f5de898fa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T10:26:31.640 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:31.633+0000 7f2542428740 1 --2- 172.21.15.114:0/2920597128 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55f5de898b80 0x55f5de898fa0 secure :-1 s=READY pgs=384 cs=0 l=1 rev1=1 crypto rx=0x7f2534031500 tx=0x7f25340058b0 comp rx=0 tx=0).stop 2024-04-23T10:26:31.640 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:31.633+0000 7f2542428740 1 -- 172.21.15.114:0/2920597128 shutdown_connections 2024-04-23T10:26:31.640 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:31.633+0000 7f2542428740 1 --2- 172.21.15.114:0/2920597128 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x55f5de89a710 0x55f5de8a0fa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:26:31.640 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:31.633+0000 7f2542428740 1 --2- 172.21.15.114:0/2920597128 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x55f5de899d50 0x55f5de89a1d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:26:31.640 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:31.633+0000 7f2542428740 1 --2- 172.21.15.114:0/2920597128 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55f5de898b80 0x55f5de898fa0 unknown :-1 s=CLOSED pgs=384 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:26:31.640 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:31.633+0000 7f2542428740 1 -- 172.21.15.114:0/2920597128 >> 172.21.15.114:0/2920597128 conn(0x55f5de894070 msgr2=0x55f5de894470 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T10:26:31.640 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:31.633+0000 7f2542428740 1 -- 172.21.15.114:0/2920597128 shutdown_connections 2024-04-23T10:26:31.640 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:31.633+0000 7f2542428740 1 -- 172.21.15.114:0/2920597128 wait complete. 2024-04-23T10:26:31.640 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:31.634+0000 7f2542428740 1 -- start start 2024-04-23T10:26:31.640 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:31.634+0000 7f2542428740 1 Processor -- start 2024-04-23T10:26:31.640 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:31.634+0000 7f2542428740 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x55f5de898b80 0x55f5de8a3000 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T10:26:31.640 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:31.634+0000 7f2542428740 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x55f5de899d50 0x55f5de8a3540 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T10:26:31.640 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:31.634+0000 7f2542428740 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55f5de89a710 0x55f5de8aa580 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T10:26:31.641 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:31.634+0000 7f2542428740 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x55f5de8a16c0 con 0x55f5de89a710 2024-04-23T10:26:31.641 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:31.634+0000 7f2542428740 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 -- 0x55f5de798470 con 0x55f5de898b80 2024-04-23T10:26:31.641 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:31.634+0000 7f2542428740 1 -- --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_getmap magic: 0 -- 0x55f5de8a1840 con 0x55f5de899d50 2024-04-23T10:26:31.641 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:31.634+0000 7f2542428740 10 client.0 _ll_register_callbacks cb 0x55f5de7bdf30 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-23T10:26:31.641 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:31.634+0000 7f253cca3640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55f5de89a710 0x55f5de8aa580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T10:26:31.641 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:31.634+0000 7f253cca3640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55f5de89a710 0x55f5de8aa580 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.114:58464/0 (socket says 172.21.15.114:58464) 2024-04-23T10:26:31.641 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:31.634+0000 7f253cca3640 1 -- 172.21.15.114:0/2895399546 learned_addr learned my addr 172.21.15.114:0/2895399546 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-04-23T10:26:31.641 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:31.634+0000 7f253d4a4640 1 --2- 172.21.15.114:0/2895399546 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x55f5de899d50 0x55f5de8a3540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T10:26:31.641 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:31.634+0000 7f253cca3640 1 --2- 172.21.15.114:0/2895399546 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x55f5de898b80 0x55f5de8a3000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T10:26:31.641 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:31.634+0000 7f253cca3640 1 -- 172.21.15.114:0/2895399546 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x55f5de899d50 msgr2=0x55f5de8a3540 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T10:26:31.641 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:31.634+0000 7f253cca3640 1 --2- 172.21.15.114:0/2895399546 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x55f5de899d50 0x55f5de8a3540 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:26:31.641 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:31.635+0000 7f253cca3640 1 -- 172.21.15.114:0/2895399546 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x55f5de898b80 msgr2=0x55f5de8a3000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T10:26:31.641 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:31.635+0000 7f253cca3640 1 --2- 172.21.15.114:0/2895399546 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x55f5de898b80 0x55f5de8a3000 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:26:31.641 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:31.635+0000 7f253cca3640 1 -- 172.21.15.114:0/2895399546 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55f5de5da6b0 con 0x55f5de89a710 2024-04-23T10:26:31.642 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:31.635+0000 7f253cca3640 1 --2- 172.21.15.114:0/2895399546 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55f5de89a710 0x55f5de8aa580 secure :-1 s=READY pgs=385 cs=0 l=1 rev1=1 crypto rx=0x7f2534005940 tx=0x7f2534040040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=97fb3ecdffc5f8fc server_cookie=0 in_seq=0 out_seq=0 2024-04-23T10:26:31.642 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:31.635+0000 7f2520ff9640 10 client.0 ms_handle_connect on v2:172.21.15.114:3300/0 2024-04-23T10:26:31.642 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:31.635+0000 7f2520ff9640 1 -- 172.21.15.114:0/2895399546 <== mon.0 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2534035070 con 0x55f5de89a710 2024-04-23T10:26:31.642 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:31.635+0000 7f2520ff9640 1 -- 172.21.15.114:0/2895399546 <== mon.0 v2:172.21.15.114:3300/0 2 ==== osd_map(89..89 src has 1..89) ==== 8269+0+0 (secure 0 0 0) 0x7f2534003f70 con 0x55f5de89a710 2024-04-23T10:26:31.642 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:31.635+0000 7f2542428740 1 -- 172.21.15.114:0/2895399546 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f25340035f0 con 0x55f5de89a710 2024-04-23T10:26:31.642 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:31.635+0000 7f2520ff9640 1 -- 172.21.15.114:0/2895399546 <== mon.0 v2:172.21.15.114:3300/0 3 ==== mon_get_version_reply(handle=1 version=20) ==== 24+0+0 (secure 0 0 0) 0x7f2534044070 con 0x55f5de89a710 2024-04-23T10:26:31.642 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:31.635+0000 7f2542428740 10 client.15228 fetch_fsmap learned FSMap version 20 2024-04-23T10:26:31.642 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:31.635+0000 7f2542428740 1 -- 172.21.15.114:0/2895399546 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({config=0+,fsmap=20,monmap=4+}) -- 0x55f5de8ad8f0 con 0x55f5de89a710 2024-04-23T10:26:31.642 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:31.635+0000 7f2520ff9640 1 -- 172.21.15.114:0/2895399546 <== mon.0 v2:172.21.15.114:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f25340035f0 con 0x55f5de89a710 2024-04-23T10:26:31.642 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:31.635+0000 7f2520ff9640 1 -- 172.21.15.114:0/2895399546 <== mon.0 v2:172.21.15.114:3300/0 5 ==== fsmap(e 20) ==== 8338+0+0 (secure 0 0 0) 0x7f2534055b30 con 0x55f5de89a710 2024-04-23T10:26:31.642 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:31.635+0000 7f2542428740 10 client.15228 fetch_fsmap finished waiting for FSMap version 20 2024-04-23T10:26:31.642 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:31.635+0000 7f2542428740 10 client.15228 resolve_mds: resolved name 'e' to daemon mds.e 2024-04-23T10:26:31.642 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:31.635+0000 7f2542428740 20 client.15228 populate_metadata read hostname 'smithi114' 2024-04-23T10:26:31.643 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:31.635+0000 7f2542428740 1 --2- 172.21.15.114:0/2895399546 >> [v2:172.21.15.119:6834/2033916709,v1:172.21.15.119:6835/2033916709] conn(0x55f5de8afa60 0x55f5de8b0ed0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T10:26:31.643 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:31.636+0000 7f2542428740 4 client.15228 mds_command: new command op to 24479 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-23T10:26:31.643 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:31.636+0000 7f2542428740 1 -- 172.21.15.114:0/2895399546 --> [v2:172.21.15.119:6834/2033916709,v1:172.21.15.119:6835/2033916709] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55f5de80b670 con 0x55f5de8afa60 2024-04-23T10:26:31.643 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:31.636+0000 7f253d4a4640 1 --2- 172.21.15.114:0/2895399546 >> [v2:172.21.15.119:6834/2033916709,v1:172.21.15.119:6835/2033916709] conn(0x55f5de8afa60 0x55f5de8b0ed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T10:26:31.643 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:31.636+0000 7f2520ff9640 10 client.15228 ms_handle_connect on v2:172.21.15.119:6834/2033916709 2024-04-23T10:26:31.643 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:31.636+0000 7f253d4a4640 1 --2- 172.21.15.114:0/2895399546 >> [v2:172.21.15.119:6834/2033916709,v1:172.21.15.119:6835/2033916709] conn(0x55f5de8afa60 0x55f5de8b0ed0 crc :-1 s=READY pgs=17 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.2 client_cookie=ec273faf58f2fb9a server_cookie=76e1053f82bcc50a in_seq=0 out_seq=0 2024-04-23T10:26:31.643 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:31.637+0000 7f2520ff9640 1 -- 172.21.15.114:0/2895399546 <== mds.2 v2:172.21.15.119:6834/2033916709 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x55f5de80b670 con 0x55f5de8afa60 2024-04-23T10:26:31.643 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:31.637+0000 7f2520ff9640 10 client.15228 handle_command_reply: tid=0 2024-04-23T10:26:31.643 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:31.637+0000 7f2542428740 1 client.15228 shutdown 2024-04-23T10:26:31.643 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:31.637+0000 7f2542428740 1 -- 172.21.15.114:0/2895399546 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55f5de89a710 msgr2=0x55f5de8aa580 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T10:26:31.643 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:31.637+0000 7f2542428740 1 --2- 172.21.15.114:0/2895399546 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55f5de89a710 0x55f5de8aa580 secure :-1 s=READY pgs=385 cs=0 l=1 rev1=1 crypto rx=0x7f2534005940 tx=0x7f2534040040 comp rx=0 tx=0).stop 2024-04-23T10:26:31.643 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:31.637+0000 7f2542428740 1 -- 172.21.15.114:0/2895399546 shutdown_connections 2024-04-23T10:26:31.643 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:31.637+0000 7f2542428740 1 --2- 172.21.15.114:0/2895399546 >> [v2:172.21.15.119:6834/2033916709,v1:172.21.15.119:6835/2033916709] conn(0x55f5de8afa60 0x55f5de8b0ed0 crc :-1 s=READY pgs=17 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:26:31.643 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:31.637+0000 7f2542428740 1 --2- 172.21.15.114:0/2895399546 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55f5de89a710 0x55f5de8aa580 unknown :-1 s=CLOSED pgs=385 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:26:31.643 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:31.637+0000 7f2542428740 1 --2- 172.21.15.114:0/2895399546 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x55f5de899d50 0x55f5de8a3540 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:26:31.644 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:31.637+0000 7f2520ff9640 0 client.15228 ms_handle_reset on v2:172.21.15.119:6834/2033916709 2024-04-23T10:26:31.644 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:31.637+0000 7f2542428740 1 --2- 172.21.15.114:0/2895399546 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x55f5de898b80 0x55f5de8a3000 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:26:31.644 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:31.637+0000 7f2542428740 1 -- 172.21.15.114:0/2895399546 >> 172.21.15.114:0/2895399546 conn(0x55f5de894070 msgr2=0x55f5de8931d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T10:26:31.644 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:31.637+0000 7f2542428740 1 -- 172.21.15.114:0/2895399546 shutdown_connections 2024-04-23T10:26:31.644 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:31.637+0000 7f2542428740 1 -- 172.21.15.114:0/2895399546 wait complete. 2024-04-23T10:26:31.644 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:31.637+0000 7f2542428740 20 client.15228 trim_cache size 0 max 16384 2024-04-23T10:26:31.660 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:31.655+0000 7f2542428740 1 Processor -- start 2024-04-23T10:26:31.660 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:31.655+0000 7f2542428740 1 -- start start 2024-04-23T10:26:31.660 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:31.655+0000 7f2542428740 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x55f5de898b80 0x55f5de8aafd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T10:26:31.660 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:31.655+0000 7f2542428740 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x55f5de899d50 0x55f5de89cd50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T10:26:31.660 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:31.655+0000 7f2542428740 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55f5de89a710 0x55f5de88f0b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T10:26:31.660 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:31.655+0000 7f2542428740 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x55f5de8af210 con 0x55f5de89a710 2024-04-23T10:26:31.660 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:31.655+0000 7f2542428740 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 -- 0x55f5de8af8e0 con 0x55f5de898b80 2024-04-23T10:26:31.660 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:31.655+0000 7f2542428740 1 -- --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_getmap magic: 0 -- 0x55f5de8affb0 con 0x55f5de899d50 2024-04-23T10:26:31.660 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:31.655+0000 7f253cca3640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55f5de89a710 0x55f5de88f0b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T10:26:31.661 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:31.655+0000 7f253cca3640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55f5de89a710 0x55f5de88f0b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.114:58472/0 (socket says 172.21.15.114:58472) 2024-04-23T10:26:31.661 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:31.655+0000 7f253cca3640 1 -- 172.21.15.114:0/3327409255 learned_addr learned my addr 172.21.15.114:0/3327409255 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-04-23T10:26:31.661 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:31.656+0000 7f253d4a4640 1 --2- 172.21.15.114:0/3327409255 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x55f5de899d50 0x55f5de89cd50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T10:26:31.661 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:31.656+0000 7f253cca3640 1 -- 172.21.15.114:0/3327409255 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x55f5de899d50 msgr2=0x55f5de89cd50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T10:26:31.661 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:31.656+0000 7f253cca3640 1 --2- 172.21.15.114:0/3327409255 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x55f5de899d50 0x55f5de89cd50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:26:31.661 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:31.656+0000 7f253cca3640 1 -- 172.21.15.114:0/3327409255 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x55f5de898b80 msgr2=0x55f5de8aafd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T10:26:31.661 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:31.656+0000 7f253cca3640 1 --2- 172.21.15.114:0/3327409255 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x55f5de898b80 0x55f5de8aafd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:26:31.661 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:31.656+0000 7f253cca3640 1 -- 172.21.15.114:0/3327409255 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55f5de8997d0 con 0x55f5de89a710 2024-04-23T10:26:31.661 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:31.656+0000 7f253d4a4640 1 --2- 172.21.15.114:0/3327409255 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x55f5de899d50 0x55f5de89cd50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-23T10:26:31.661 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:31.656+0000 7f253cca3640 1 --2- 172.21.15.114:0/3327409255 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55f5de89a710 0x55f5de88f0b0 secure :-1 s=READY pgs=386 cs=0 l=1 rev1=1 crypto rx=0x7f2534005c90 tx=0x7f2534005cc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=33f793f4b647dc6d server_cookie=0 in_seq=0 out_seq=0 2024-04-23T10:26:31.661 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:31.656+0000 7f2522ffd640 1 -- 172.21.15.114:0/3327409255 <== mon.0 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f253404d070 con 0x55f5de89a710 2024-04-23T10:26:31.661 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:31.656+0000 7f2522ffd640 1 -- 172.21.15.114:0/3327409255 <== mon.0 v2:172.21.15.114:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f25340035f0 con 0x55f5de89a710 2024-04-23T10:26:31.661 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:31.656+0000 7f2522ffd640 1 -- 172.21.15.114:0/3327409255 <== mon.0 v2:172.21.15.114:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2534035040 con 0x55f5de89a710 2024-04-23T10:26:31.661 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:31.656+0000 7f2542428740 1 -- 172.21.15.114:0/3327409255 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55f5de89a710 msgr2=0x55f5de88f0b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T10:26:31.661 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:31.656+0000 7f2542428740 1 --2- 172.21.15.114:0/3327409255 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55f5de89a710 0x55f5de88f0b0 secure :-1 s=READY pgs=386 cs=0 l=1 rev1=1 crypto rx=0x7f2534005c90 tx=0x7f2534005cc0 comp rx=0 tx=0).stop 2024-04-23T10:26:31.662 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:31.657+0000 7f2542428740 1 -- 172.21.15.114:0/3327409255 shutdown_connections 2024-04-23T10:26:31.662 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:31.657+0000 7f2542428740 1 --2- 172.21.15.114:0/3327409255 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55f5de89a710 0x55f5de88f0b0 unknown :-1 s=CLOSED pgs=386 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:26:31.662 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:31.657+0000 7f2542428740 1 --2- 172.21.15.114:0/3327409255 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x55f5de899d50 0x55f5de89cd50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:26:31.662 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:31.657+0000 7f2542428740 1 --2- 172.21.15.114:0/3327409255 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x55f5de898b80 0x55f5de8aafd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:26:31.662 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:31.657+0000 7f2542428740 1 -- 172.21.15.114:0/3327409255 >> 172.21.15.114:0/3327409255 conn(0x55f5de894070 msgr2=0x55f5de894450 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T10:26:31.662 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:31.657+0000 7f2542428740 1 -- 172.21.15.114:0/3327409255 shutdown_connections 2024-04-23T10:26:31.662 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:31.657+0000 7f2542428740 1 -- 172.21.15.114:0/3327409255 wait complete. 2024-04-23T10:26:31.662 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:31.657+0000 7f2542428740 1 -- start start 2024-04-23T10:26:31.662 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:31.657+0000 7f2542428740 1 Processor -- start 2024-04-23T10:26:31.662 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:31.657+0000 7f2542428740 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x55f5de898b80 0x55f5de89e870 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T10:26:31.662 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:31.657+0000 7f2542428740 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55f5de899d50 0x55f5de89edb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T10:26:31.662 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:31.657+0000 7f2542428740 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x55f5de89a710 0x55f5de89f2f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T10:26:31.662 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:31.657+0000 7f2542428740 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x55f5de8b0d50 con 0x55f5de899d50 2024-04-23T10:26:31.662 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:31.657+0000 7f2542428740 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 -- 0x55f5de8b0680 con 0x55f5de89a710 2024-04-23T10:26:31.662 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:31.657+0000 7f2542428740 1 -- --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_getmap magic: 0 -- 0x55f5de8b1420 con 0x55f5de898b80 2024-04-23T10:26:31.663 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:31.657+0000 7f2542428740 10 client.0 _ll_register_callbacks cb 0x7f252c006f30 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-23T10:26:31.663 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:31.657+0000 7f253d4a4640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55f5de899d50 0x55f5de89edb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T10:26:31.663 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:31.657+0000 7f253d4a4640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55f5de899d50 0x55f5de89edb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.114:58474/0 (socket says 172.21.15.114:58474) 2024-04-23T10:26:31.663 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:31.657+0000 7f253d4a4640 1 -- 172.21.15.114:0/2478556689 learned_addr learned my addr 172.21.15.114:0/2478556689 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-04-23T10:26:31.663 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:31.657+0000 7f253cca3640 1 --2- 172.21.15.114:0/2478556689 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x55f5de898b80 0x55f5de89e870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T10:26:31.663 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:31.658+0000 7f253d4a4640 1 -- 172.21.15.114:0/2478556689 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x55f5de898b80 msgr2=0x55f5de89e870 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T10:26:31.663 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:31.658+0000 7f253d4a4640 1 --2- 172.21.15.114:0/2478556689 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x55f5de898b80 0x55f5de89e870 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:26:31.663 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:31.658+0000 7f253d4a4640 1 -- 172.21.15.114:0/2478556689 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x55f5de89a710 msgr2=0x55f5de89f2f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T10:26:31.663 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:31.658+0000 7f253d4a4640 1 --2- 172.21.15.114:0/2478556689 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x55f5de89a710 0x55f5de89f2f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:26:31.663 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:31.658+0000 7f253d4a4640 1 -- 172.21.15.114:0/2478556689 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55f5de89f960 con 0x55f5de899d50 2024-04-23T10:26:31.663 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:31.658+0000 7f253cca3640 1 --2- 172.21.15.114:0/2478556689 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x55f5de898b80 0x55f5de89e870 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-23T10:26:31.663 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:31.658+0000 7f253d4a4640 1 --2- 172.21.15.114:0/2478556689 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55f5de899d50 0x55f5de89edb0 secure :-1 s=READY pgs=387 cs=0 l=1 rev1=1 crypto rx=0x7f252c009800 tx=0x7f252c009f00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5b68820966c1d79b server_cookie=0 in_seq=0 out_seq=0 2024-04-23T10:26:31.663 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:31.658+0000 7f2520ff9640 10 client.0 ms_handle_connect on v2:172.21.15.114:3300/0 2024-04-23T10:26:31.664 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:31.658+0000 7f2520ff9640 1 -- 172.21.15.114:0/2478556689 <== mon.0 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f252c018a10 con 0x55f5de899d50 2024-04-23T10:26:31.664 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:31.658+0000 7f2520ff9640 1 -- 172.21.15.114:0/2478556689 <== mon.0 v2:172.21.15.114:3300/0 2 ==== osd_map(89..89 src has 1..89) ==== 8269+0+0 (secure 0 0 0) 0x7f252c00f040 con 0x55f5de899d50 2024-04-23T10:26:31.664 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:31.658+0000 7f2542428740 1 -- 172.21.15.114:0/2478556689 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f25340035f0 con 0x55f5de899d50 2024-04-23T10:26:31.664 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:31.658+0000 7f2520ff9640 1 -- 172.21.15.114:0/2478556689 <== mon.0 v2:172.21.15.114:3300/0 3 ==== mon_get_version_reply(handle=1 version=20) ==== 24+0+0 (secure 0 0 0) 0x7f252c018bb0 con 0x55f5de899d50 2024-04-23T10:26:31.664 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:31.658+0000 7f2542428740 10 client.15234 fetch_fsmap learned FSMap version 20 2024-04-23T10:26:31.664 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:31.658+0000 7f2542428740 1 -- 172.21.15.114:0/2478556689 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({config=0+,fsmap=20,monmap=4+}) -- 0x55f5de919e90 con 0x55f5de899d50 2024-04-23T10:26:31.664 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:31.658+0000 7f2520ff9640 1 -- 172.21.15.114:0/2478556689 <== mon.0 v2:172.21.15.114:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f25340035f0 con 0x55f5de899d50 2024-04-23T10:26:31.664 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:31.658+0000 7f2520ff9640 1 -- 172.21.15.114:0/2478556689 <== mon.0 v2:172.21.15.114:3300/0 5 ==== fsmap(e 20) ==== 8338+0+0 (secure 0 0 0) 0x7f252c02c420 con 0x55f5de899d50 2024-04-23T10:26:31.664 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:31.658+0000 7f2542428740 10 client.15234 fetch_fsmap finished waiting for FSMap version 20 2024-04-23T10:26:31.664 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:31.658+0000 7f2542428740 10 client.15234 resolve_mds: resolved name 'e' to daemon mds.e 2024-04-23T10:26:31.664 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:31.658+0000 7f2542428740 20 client.15234 populate_metadata read hostname 'smithi114' 2024-04-23T10:26:31.664 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:31.658+0000 7f2542428740 1 --2- 172.21.15.114:0/2478556689 >> [v2:172.21.15.119:6834/2033916709,v1:172.21.15.119:6835/2033916709] conn(0x55f5de91ea20 0x55f5de91d730 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T10:26:31.664 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:31.658+0000 7f2542428740 4 client.15234 mds_command: new command op to 24479 tid=0 multi_id=0 [{"prefix": "status"}] 2024-04-23T10:26:31.664 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:31.658+0000 7f2542428740 1 -- 172.21.15.114:0/2478556689 --> [v2:172.21.15.119:6834/2033916709,v1:172.21.15.119:6835/2033916709] -- command(tid 0: {"prefix": "status"}) -- 0x55f5de8054e0 con 0x55f5de91ea20 2024-04-23T10:26:31.664 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:31.659+0000 7f253cca3640 1 --2- 172.21.15.114:0/2478556689 >> [v2:172.21.15.119:6834/2033916709,v1:172.21.15.119:6835/2033916709] conn(0x55f5de91ea20 0x55f5de91d730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T10:26:31.664 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:31.659+0000 7f2520ff9640 10 client.15234 ms_handle_connect on v2:172.21.15.119:6834/2033916709 2024-04-23T10:26:31.665 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:31.659+0000 7f253cca3640 1 --2- 172.21.15.114:0/2478556689 >> [v2:172.21.15.119:6834/2033916709,v1:172.21.15.119:6835/2033916709] conn(0x55f5de91ea20 0x55f5de91d730 crc :-1 s=READY pgs=18 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.2 client_cookie=9063f49c63132a64 server_cookie=c1fe90dbd7b9896d in_seq=0 out_seq=0 2024-04-23T10:26:31.665 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:31.659+0000 7f2520ff9640 1 -- 172.21.15.114:0/2478556689 <== mds.2 v2:172.21.15.119:6834/2033916709 1 ==== command_reply(tid 0: 0 ) ==== 8+0+322 (crc 0 0 0) 0x55f5de8054e0 con 0x55f5de91ea20 2024-04-23T10:26:31.665 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:31.659+0000 7f2520ff9640 10 client.15234 handle_command_reply: tid=0 2024-04-23T10:26:31.665 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:31.659+0000 7f2542428740 1 client.15234 shutdown 2024-04-23T10:26:31.665 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:31.659+0000 7f2542428740 1 -- 172.21.15.114:0/2478556689 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55f5de899d50 msgr2=0x55f5de89edb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T10:26:31.665 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:31.659+0000 7f2542428740 1 --2- 172.21.15.114:0/2478556689 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55f5de899d50 0x55f5de89edb0 secure :-1 s=READY pgs=387 cs=0 l=1 rev1=1 crypto rx=0x7f252c009800 tx=0x7f252c009f00 comp rx=0 tx=0).stop 2024-04-23T10:26:31.665 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:31.659+0000 7f2542428740 1 -- 172.21.15.114:0/2478556689 shutdown_connections 2024-04-23T10:26:31.665 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:31.659+0000 7f2542428740 1 --2- 172.21.15.114:0/2478556689 >> [v2:172.21.15.119:6834/2033916709,v1:172.21.15.119:6835/2033916709] conn(0x55f5de91ea20 0x55f5de91d730 crc :-1 s=READY pgs=18 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:26:31.665 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:31.659+0000 7f2542428740 1 --2- 172.21.15.114:0/2478556689 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x55f5de89a710 0x55f5de89f2f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:26:31.665 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:31.659+0000 7f2542428740 1 --2- 172.21.15.114:0/2478556689 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55f5de899d50 0x55f5de89edb0 unknown :-1 s=CLOSED pgs=387 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:26:31.665 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:31.659+0000 7f2520ff9640 0 client.15234 ms_handle_reset on v2:172.21.15.119:6834/2033916709 2024-04-23T10:26:31.665 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:31.659+0000 7f2542428740 1 --2- 172.21.15.114:0/2478556689 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x55f5de898b80 0x55f5de89e870 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:26:31.665 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:31.659+0000 7f2542428740 1 -- 172.21.15.114:0/2478556689 >> 172.21.15.114:0/2478556689 conn(0x55f5de894070 msgr2=0x55f5de88f0b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T10:26:31.665 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:31.660+0000 7f2542428740 1 -- 172.21.15.114:0/2478556689 shutdown_connections 2024-04-23T10:26:31.665 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:31.660+0000 7f2542428740 1 -- 172.21.15.114:0/2478556689 wait complete. 2024-04-23T10:26:31.665 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:31.660+0000 7f2542428740 20 client.15234 trim_cache size 0 max 16384 2024-04-23T10:26:31.666 INFO:teuthology.orchestra.run.smithi114.stdout:{ 2024-04-23T10:26:31.666 INFO:teuthology.orchestra.run.smithi114.stdout: "cluster_fsid": "aa240572-0159-11ef-bc93-c7b262605968", 2024-04-23T10:26:31.666 INFO:teuthology.orchestra.run.smithi114.stdout: "whoami": 2, 2024-04-23T10:26:31.666 INFO:teuthology.orchestra.run.smithi114.stdout: "id": 24479, 2024-04-23T10:26:31.666 INFO:teuthology.orchestra.run.smithi114.stdout: "want_state": "up:active", 2024-04-23T10:26:31.666 INFO:teuthology.orchestra.run.smithi114.stdout: "state": "up:active", 2024-04-23T10:26:31.666 INFO:teuthology.orchestra.run.smithi114.stdout: "fs_name": "cephfs", 2024-04-23T10:26:31.666 INFO:teuthology.orchestra.run.smithi114.stdout: "rank_uptime": 104.315253309, 2024-04-23T10:26:31.666 INFO:teuthology.orchestra.run.smithi114.stdout: "mdsmap_epoch": 20, 2024-04-23T10:26:31.666 INFO:teuthology.orchestra.run.smithi114.stdout: "osdmap_epoch": 89, 2024-04-23T10:26:31.666 INFO:teuthology.orchestra.run.smithi114.stdout: "osdmap_epoch_barrier": 87, 2024-04-23T10:26:31.666 INFO:teuthology.orchestra.run.smithi114.stdout: "uptime": 132.518464583 2024-04-23T10:26:31.667 INFO:teuthology.orchestra.run.smithi114.stdout:} 2024-04-23T10:26:31.667 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:31.660+0000 7f253f708640 1 -- 172.21.15.114:0/2227654281 >> [v2:172.21.15.114:6800/465333808,v1:172.21.15.114:6801/465333808] conn(0x7f25240a3bb0 msgr2=0x7f25240a6070 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T10:26:31.667 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:31.660+0000 7f253f708640 1 --2- 172.21.15.114:0/2227654281 >> [v2:172.21.15.114:6800/465333808,v1:172.21.15.114:6801/465333808] conn(0x7f25240a3bb0 0x7f25240a6070 secure :-1 s=READY pgs=244 cs=0 l=1 rev1=1 crypto rx=0x7f25300a5310 tx=0x7f2530067220 comp rx=0 tx=0).stop 2024-04-23T10:26:31.667 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:31.660+0000 7f253f708640 1 -- 172.21.15.114:0/2227654281 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f2538102a40 msgr2=0x7f253819f5b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T10:26:31.667 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:31.660+0000 7f253f708640 1 --2- 172.21.15.114:0/2227654281 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f2538102a40 0x7f253819f5b0 secure :-1 s=READY pgs=383 cs=0 l=1 rev1=1 crypto rx=0x7f253000cce0 tx=0x7f253000a620 comp rx=0 tx=0).stop 2024-04-23T10:26:31.668 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:31.660+0000 7f253f708640 1 -- 172.21.15.114:0/2227654281 shutdown_connections 2024-04-23T10:26:31.668 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:31.660+0000 7f253f708640 1 --2- 172.21.15.114:0/2227654281 >> [v2:172.21.15.114:6800/465333808,v1:172.21.15.114:6801/465333808] conn(0x7f25240a3bb0 0x7f25240a6070 unknown :-1 s=CLOSED pgs=244 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:26:31.668 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:31.660+0000 7f253f708640 1 --2- 172.21.15.114:0/2227654281 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f2538102a40 0x7f253819f5b0 unknown :-1 s=CLOSED pgs=383 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:26:31.668 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:31.660+0000 7f253f708640 1 --2- 172.21.15.114:0/2227654281 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f2538102080 0x7f253819cf40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:26:31.668 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:31.660+0000 7f253f708640 1 --2- 172.21.15.114:0/2227654281 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f2538100f30 0x7f253806b160 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:26:31.668 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:31.660+0000 7f253f708640 1 -- 172.21.15.114:0/2227654281 >> 172.21.15.114:0/2227654281 conn(0x7f25380fc4a0 msgr2=0x7f2538105410 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T10:26:31.668 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:31.660+0000 7f253f708640 1 -- 172.21.15.114:0/2227654281 shutdown_connections 2024-04-23T10:26:31.668 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:31.660+0000 7f253f708640 1 -- 172.21.15.114:0/2227654281 wait complete. 2024-04-23T10:26:31.670 DEBUG:teuthology.orchestra.run.smithi114:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.h status 2024-04-23T10:26:31.770 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:31.769+0000 7fb7143ba640 1 -- 172.21.15.114:0/3454529559 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fb70c100e60 msgr2=0x7fb70c101280 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T10:26:31.770 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:31.769+0000 7fb7143ba640 1 --2- 172.21.15.114:0/3454529559 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fb70c100e60 0x7fb70c101280 secure :-1 s=READY pgs=388 cs=0 l=1 rev1=1 crypto rx=0x7fb704009ca0 tx=0x7fb70402a8c0 comp rx=0 tx=0).stop 2024-04-23T10:26:31.770 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:31.769+0000 7fb7143ba640 1 -- 172.21.15.114:0/3454529559 shutdown_connections 2024-04-23T10:26:31.771 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:31.769+0000 7fb7143ba640 1 --2- 172.21.15.114:0/3454529559 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fb70c102a00 0x7fb70c109320 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:26:31.771 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:31.769+0000 7fb7143ba640 1 --2- 172.21.15.114:0/3454529559 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fb70c102040 0x7fb70c1024c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:26:31.771 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:31.769+0000 7fb7143ba640 1 --2- 172.21.15.114:0/3454529559 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fb70c100e60 0x7fb70c101280 unknown :-1 s=CLOSED pgs=388 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:26:31.771 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:31.769+0000 7fb7143ba640 1 -- 172.21.15.114:0/3454529559 >> 172.21.15.114:0/3454529559 conn(0x7fb70c0fc440 msgr2=0x7fb70c0fe860 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T10:26:31.771 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:31.769+0000 7fb7143ba640 1 -- 172.21.15.114:0/3454529559 shutdown_connections 2024-04-23T10:26:31.771 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:31.769+0000 7fb7143ba640 1 -- 172.21.15.114:0/3454529559 wait complete. 2024-04-23T10:26:31.771 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:31.770+0000 7fb7143ba640 1 Processor -- start 2024-04-23T10:26:31.771 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:31.770+0000 7fb7143ba640 1 -- start start 2024-04-23T10:26:31.771 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:31.770+0000 7fb7143ba640 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fb70c100e60 0x7fb70c1981f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T10:26:31.771 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:31.770+0000 7fb7143ba640 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fb70c102040 0x7fb70c198730 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T10:26:31.771 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:31.770+0000 7fb7143ba640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fb70c102a00 0x7fb70c19f7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T10:26:31.771 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:31.770+0000 7fb7143ba640 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x7fb70c10c9f0 con 0x7fb70c102a00 2024-04-23T10:26:31.771 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:31.770+0000 7fb7143ba640 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 -- 0x7fb70c10c870 con 0x7fb70c102040 2024-04-23T10:26:31.772 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:31.770+0000 7fb7143ba640 1 -- --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_getmap magic: 0 -- 0x7fb70c10cb70 con 0x7fb70c100e60 2024-04-23T10:26:31.772 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:31.770+0000 7fb712957640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fb70c102a00 0x7fb70c19f7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T10:26:31.772 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:31.770+0000 7fb712957640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fb70c102a00 0x7fb70c19f7a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.114:58500/0 (socket says 172.21.15.114:58500) 2024-04-23T10:26:31.772 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:31.770+0000 7fb712957640 1 -- 172.21.15.114:0/2515183953 learned_addr learned my addr 172.21.15.114:0/2515183953 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-04-23T10:26:31.772 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:31.770+0000 7fb712156640 1 --2- 172.21.15.114:0/2515183953 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fb70c100e60 0x7fb70c1981f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T10:26:31.772 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:31.770+0000 7fb711955640 1 --2- 172.21.15.114:0/2515183953 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fb70c102040 0x7fb70c198730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T10:26:31.772 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:31.770+0000 7fb712957640 1 -- 172.21.15.114:0/2515183953 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fb70c100e60 msgr2=0x7fb70c1981f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T10:26:31.772 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:31.770+0000 7fb712957640 1 --2- 172.21.15.114:0/2515183953 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fb70c100e60 0x7fb70c1981f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:26:31.772 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:31.770+0000 7fb712957640 1 -- 172.21.15.114:0/2515183953 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fb70c102040 msgr2=0x7fb70c198730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T10:26:31.772 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:31.770+0000 7fb712957640 1 --2- 172.21.15.114:0/2515183953 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fb70c102040 0x7fb70c198730 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:26:31.773 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:31.770+0000 7fb712957640 1 -- 172.21.15.114:0/2515183953 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb70c19fe10 con 0x7fb70c102a00 2024-04-23T10:26:31.773 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:31.771+0000 7fb712957640 1 --2- 172.21.15.114:0/2515183953 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fb70c102a00 0x7fb70c19f7a0 secure :-1 s=READY pgs=389 cs=0 l=1 rev1=1 crypto rx=0x7fb70800d750 tx=0x7fb70800dc10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T10:26:31.773 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:31.771+0000 7fb7037fe640 1 -- 172.21.15.114:0/2515183953 <== mon.0 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb708017070 con 0x7fb70c102a00 2024-04-23T10:26:31.773 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:31.771+0000 7fb7037fe640 1 -- 172.21.15.114:0/2515183953 <== mon.0 v2:172.21.15.114:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fb7080054d0 con 0x7fb70c102a00 2024-04-23T10:26:31.773 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:31.771+0000 7fb7037fe640 1 -- 172.21.15.114:0/2515183953 <== mon.0 v2:172.21.15.114:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb708004c60 con 0x7fb70c102a00 2024-04-23T10:26:31.773 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:31.771+0000 7fb7143ba640 1 -- 172.21.15.114:0/2515183953 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fb70c1a0100 con 0x7fb70c102a00 2024-04-23T10:26:31.773 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:31.771+0000 7fb7143ba640 1 -- 172.21.15.114:0/2515183953 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fb70c1a0650 con 0x7fb70c102a00 2024-04-23T10:26:31.780 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:31.773+0000 7fb7037fe640 1 -- 172.21.15.114:0/2515183953 <== mon.0 v2:172.21.15.114:3300/0 4 ==== mgrmap(e 18) ==== 138647+0+0 (secure 0 0 0) 0x7fb708009bd0 con 0x7fb70c102a00 2024-04-23T10:26:31.780 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:31.773+0000 7fb7171a8740 1 Processor -- start 2024-04-23T10:26:31.780 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:31.774+0000 7fb7171a8740 1 -- start start 2024-04-23T10:26:31.780 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:31.774+0000 7fb7171a8740 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x55570e666b80 0x55570e666fa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T10:26:31.780 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:31.774+0000 7fb7171a8740 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55570e667d50 0x55570e6681d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T10:26:31.780 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:31.774+0000 7fb7037fe640 1 --2- 172.21.15.114:0/2515183953 >> [v2:172.21.15.114:6800/465333808,v1:172.21.15.114:6801/465333808] conn(0x7fb6f80a3c30 0x7fb6f80a60f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T10:26:31.780 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:31.774+0000 7fb7171a8740 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x55570e668710 0x55570e66efa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T10:26:31.780 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:31.774+0000 7fb7171a8740 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x55570e5cbc50 con 0x55570e667d50 2024-04-23T10:26:31.780 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:31.774+0000 7fb7171a8740 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 -- 0x55570e568830 con 0x55570e668710 2024-04-23T10:26:31.780 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:31.774+0000 7fb7171a8740 1 -- --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_getmap magic: 0 -- 0x55570e5c7c10 con 0x55570e666b80 2024-04-23T10:26:31.780 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:31.774+0000 7fb712156640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55570e667d50 0x55570e6681d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T10:26:31.780 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:31.774+0000 7fb711955640 1 --2- 172.21.15.114:0/2515183953 >> [v2:172.21.15.114:6800/465333808,v1:172.21.15.114:6801/465333808] conn(0x7fb6f80a3c30 0x7fb6f80a60f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T10:26:31.781 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:31.774+0000 7fb712156640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55570e667d50 0x55570e6681d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.114:58514/0 (socket says 172.21.15.114:58514) 2024-04-23T10:26:31.781 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:31.774+0000 7fb712156640 1 -- 172.21.15.114:0/3789595251 learned_addr learned my addr 172.21.15.114:0/3789595251 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-04-23T10:26:31.781 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:31.774+0000 7fb712957640 1 --2- 172.21.15.114:0/3789595251 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x55570e668710 0x55570e66efa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T10:26:31.781 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:31.774+0000 7fb7037fe640 1 -- 172.21.15.114:0/2515183953 <== mon.0 v2:172.21.15.114:3300/0 5 ==== osd_map(89..89 src has 1..89) ==== 8269+0+0 (secure 0 0 0) 0x7fb7080cfda0 con 0x7fb70c102a00 2024-04-23T10:26:31.781 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:31.774+0000 7fb711955640 1 --2- 172.21.15.114:0/3789595251 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x55570e666b80 0x55570e666fa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T10:26:31.781 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:31.774+0000 7fb712156640 1 -- 172.21.15.114:0/3789595251 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x55570e666b80 msgr2=0x55570e666fa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T10:26:31.781 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:31.774+0000 7fb712156640 1 --2- 172.21.15.114:0/3789595251 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x55570e666b80 0x55570e666fa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:26:31.781 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:31.774+0000 7fb712156640 1 -- 172.21.15.114:0/3789595251 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x55570e668710 msgr2=0x55570e66efa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T10:26:31.781 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:31.774+0000 7fb712156640 1 --2- 172.21.15.114:0/3789595251 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x55570e668710 0x55570e66efa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:26:31.781 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:31.774+0000 7fb712156640 1 -- 172.21.15.114:0/3789595251 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55570e2de2d0 con 0x55570e667d50 2024-04-23T10:26:31.781 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:31.774+0000 7fb711955640 1 --2- 172.21.15.114:0/2515183953 >> [v2:172.21.15.114:6800/465333808,v1:172.21.15.114:6801/465333808] conn(0x7fb6f80a3c30 0x7fb6f80a60f0 secure :-1 s=READY pgs=245 cs=0 l=1 rev1=1 crypto rx=0x7fb70c199700 tx=0x7fb6fc00cbe0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T10:26:31.781 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:31.774+0000 7fb712156640 1 --2- 172.21.15.114:0/3789595251 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55570e667d50 0x55570e6681d0 secure :-1 s=READY pgs=390 cs=0 l=1 rev1=1 crypto rx=0x7fb704009ca0 tx=0x7fb704005f70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=aa2a093ab2e25625 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T10:26:31.781 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:31.774+0000 7fb700bfe640 1 -- 172.21.15.114:0/3789595251 <== mon.0 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb704031650 con 0x55570e667d50 2024-04-23T10:26:31.781 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:31.775+0000 7fb700bfe640 1 -- 172.21.15.114:0/3789595251 <== mon.0 v2:172.21.15.114:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fb704036030 con 0x55570e667d50 2024-04-23T10:26:31.781 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:31.775+0000 7fb700bfe640 1 -- 172.21.15.114:0/3789595251 <== mon.0 v2:172.21.15.114:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb7040044e0 con 0x55570e667d50 2024-04-23T10:26:31.781 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:31.775+0000 7fb7171a8740 1 -- 172.21.15.114:0/3789595251 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55570e667d50 msgr2=0x55570e6681d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T10:26:31.782 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:31.775+0000 7fb7171a8740 1 --2- 172.21.15.114:0/3789595251 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55570e667d50 0x55570e6681d0 secure :-1 s=READY pgs=390 cs=0 l=1 rev1=1 crypto rx=0x7fb704009ca0 tx=0x7fb704005f70 comp rx=0 tx=0).stop 2024-04-23T10:26:31.782 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:31.775+0000 7fb7171a8740 1 -- 172.21.15.114:0/3789595251 shutdown_connections 2024-04-23T10:26:31.782 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:31.775+0000 7fb7171a8740 1 --2- 172.21.15.114:0/3789595251 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x55570e668710 0x55570e66efa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:26:31.782 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:31.775+0000 7fb7171a8740 1 --2- 172.21.15.114:0/3789595251 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55570e667d50 0x55570e6681d0 unknown :-1 s=CLOSED pgs=390 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:26:31.782 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:31.775+0000 7fb7171a8740 1 --2- 172.21.15.114:0/3789595251 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x55570e666b80 0x55570e666fa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:26:31.782 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:31.775+0000 7fb7171a8740 1 -- 172.21.15.114:0/3789595251 >> 172.21.15.114:0/3789595251 conn(0x55570e662070 msgr2=0x55570e662470 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T10:26:31.782 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:31.775+0000 7fb7171a8740 1 -- 172.21.15.114:0/3789595251 shutdown_connections 2024-04-23T10:26:31.782 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:31.775+0000 7fb7171a8740 1 -- 172.21.15.114:0/3789595251 wait complete. 2024-04-23T10:26:31.782 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:31.776+0000 7fb7171a8740 1 -- start start 2024-04-23T10:26:31.782 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:31.776+0000 7fb7171a8740 1 Processor -- start 2024-04-23T10:26:31.782 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:31.776+0000 7fb7171a8740 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x55570e666b80 0x55570e670d40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T10:26:31.782 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:31.776+0000 7fb7171a8740 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x55570e667d50 0x55570e671280 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T10:26:31.782 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:31.776+0000 7fb7171a8740 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55570e668710 0x55570e6782c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T10:26:31.782 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:31.776+0000 7fb7171a8740 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x55570e66f540 con 0x55570e668710 2024-04-23T10:26:31.782 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:31.776+0000 7fb7171a8740 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 -- 0x55570e566470 con 0x55570e666b80 2024-04-23T10:26:31.783 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:31.776+0000 7fb7171a8740 1 -- --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_getmap magic: 0 -- 0x55570e66f6c0 con 0x55570e667d50 2024-04-23T10:26:31.783 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:31.776+0000 7fb7171a8740 10 client.0 _ll_register_callbacks cb 0x55570e58bf30 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-23T10:26:31.783 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:31.776+0000 7fb712957640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55570e668710 0x55570e6782c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T10:26:31.783 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:31.776+0000 7fb712957640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55570e668710 0x55570e6782c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.114:58526/0 (socket says 172.21.15.114:58526) 2024-04-23T10:26:31.783 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:31.776+0000 7fb712957640 1 -- 172.21.15.114:0/2762830015 learned_addr learned my addr 172.21.15.114:0/2762830015 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-04-23T10:26:31.783 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:31.776+0000 7fb711955640 1 --2- 172.21.15.114:0/2762830015 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x55570e667d50 0x55570e671280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T10:26:31.783 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:31.776+0000 7fb712156640 1 --2- 172.21.15.114:0/2762830015 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x55570e666b80 0x55570e670d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T10:26:31.783 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:31.776+0000 7fb712957640 1 -- 172.21.15.114:0/2762830015 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x55570e667d50 msgr2=0x55570e671280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T10:26:31.783 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:31.776+0000 7fb712957640 1 --2- 172.21.15.114:0/2762830015 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x55570e667d50 0x55570e671280 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:26:31.783 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:31.776+0000 7fb712957640 1 -- 172.21.15.114:0/2762830015 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x55570e666b80 msgr2=0x55570e670d40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T10:26:31.783 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:31.776+0000 7fb712957640 1 --2- 172.21.15.114:0/2762830015 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x55570e666b80 0x55570e670d40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:26:31.783 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:31.776+0000 7fb712957640 1 -- 172.21.15.114:0/2762830015 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55570e3a86b0 con 0x55570e668710 2024-04-23T10:26:31.783 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:31.776+0000 7fb711955640 1 --2- 172.21.15.114:0/2762830015 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x55570e667d50 0x55570e671280 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-23T10:26:31.783 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:31.776+0000 7fb712957640 1 --2- 172.21.15.114:0/2762830015 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55570e668710 0x55570e6782c0 secure :-1 s=READY pgs=391 cs=0 l=1 rev1=1 crypto rx=0x7fb708097ce0 tx=0x7fb70809bc30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d33d736481d331b8 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T10:26:31.783 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:31.776+0000 7fb6f5ffb640 10 client.0 ms_handle_connect on v2:172.21.15.114:3300/0 2024-04-23T10:26:31.783 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:31.777+0000 7fb6f5ffb640 1 -- 172.21.15.114:0/2762830015 <== mon.0 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb708020040 con 0x55570e668710 2024-04-23T10:26:31.784 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:31.777+0000 7fb6f5ffb640 1 -- 172.21.15.114:0/2762830015 <== mon.0 v2:172.21.15.114:3300/0 2 ==== osd_map(89..89 src has 1..89) ==== 8269+0+0 (secure 0 0 0) 0x7fb708098670 con 0x55570e668710 2024-04-23T10:26:31.784 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:31.777+0000 7fb7171a8740 1 -- 172.21.15.114:0/2762830015 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fb704036030 con 0x55570e668710 2024-04-23T10:26:31.784 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:31.777+0000 7fb6f5ffb640 1 -- 172.21.15.114:0/2762830015 <== mon.0 v2:172.21.15.114:3300/0 3 ==== mon_get_version_reply(handle=1 version=20) ==== 24+0+0 (secure 0 0 0) 0x7fb708005680 con 0x55570e668710 2024-04-23T10:26:31.784 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:31.777+0000 7fb7171a8740 10 client.15246 fetch_fsmap learned FSMap version 20 2024-04-23T10:26:31.784 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:31.777+0000 7fb7171a8740 1 -- 172.21.15.114:0/2762830015 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({config=0+,fsmap=20,monmap=4+}) -- 0x55570e67b700 con 0x55570e668710 2024-04-23T10:26:31.784 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:31.777+0000 7fb6f5ffb640 1 -- 172.21.15.114:0/2762830015 <== mon.0 v2:172.21.15.114:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fb704036030 con 0x55570e668710 2024-04-23T10:26:31.784 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:31.777+0000 7fb6f5ffb640 1 -- 172.21.15.114:0/2762830015 <== mon.0 v2:172.21.15.114:3300/0 5 ==== fsmap(e 20) ==== 8338+0+0 (secure 0 0 0) 0x7fb7080cdce0 con 0x55570e668710 2024-04-23T10:26:31.784 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:31.777+0000 7fb7171a8740 10 client.15246 fetch_fsmap finished waiting for FSMap version 20 2024-04-23T10:26:31.784 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:31.777+0000 7fb7171a8740 10 client.15246 resolve_mds: resolved name 'h' to daemon mds.h 2024-04-23T10:26:31.784 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:31.777+0000 7fb7171a8740 20 client.15246 populate_metadata read hostname 'smithi114' 2024-04-23T10:26:31.784 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:31.777+0000 7fb7171a8740 1 --2- 172.21.15.114:0/2762830015 >> [v2:172.21.15.119:6836/1872363128,v1:172.21.15.119:6837/1872363128] conn(0x55570e67ddb0 0x55570e67f220 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T10:26:31.784 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:31.777+0000 7fb7171a8740 4 client.15246 mds_command: new command op to 24509 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-23T10:26:31.784 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:31.777+0000 7fb7171a8740 1 -- 172.21.15.114:0/2762830015 --> [v2:172.21.15.119:6836/1872363128,v1:172.21.15.119:6837/1872363128] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55570e5d9670 con 0x55570e67ddb0 2024-04-23T10:26:31.784 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:31.777+0000 7fb712156640 1 --2- 172.21.15.114:0/2762830015 >> [v2:172.21.15.119:6836/1872363128,v1:172.21.15.119:6837/1872363128] conn(0x55570e67ddb0 0x55570e67f220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T10:26:31.785 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:31.778+0000 7fb6f5ffb640 10 client.15246 ms_handle_connect on v2:172.21.15.119:6836/1872363128 2024-04-23T10:26:31.785 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:31.778+0000 7fb712156640 1 --2- 172.21.15.114:0/2762830015 >> [v2:172.21.15.119:6836/1872363128,v1:172.21.15.119:6837/1872363128] conn(0x55570e67ddb0 0x55570e67f220 crc :-1 s=READY pgs=18 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.1 client_cookie=c81908c626bd38f9 server_cookie=5ae0cf5e4f1ab0a2 in_seq=0 out_seq=0 2024-04-23T10:26:31.785 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:31.778+0000 7fb6f5ffb640 1 -- 172.21.15.114:0/2762830015 <== mds.1 v2:172.21.15.119:6836/1872363128 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x55570e5d9670 con 0x55570e67ddb0 2024-04-23T10:26:31.785 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:31.778+0000 7fb6f5ffb640 10 client.15246 handle_command_reply: tid=0 2024-04-23T10:26:31.785 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:31.778+0000 7fb7171a8740 1 client.15246 shutdown 2024-04-23T10:26:31.785 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:31.779+0000 7fb7171a8740 1 -- 172.21.15.114:0/2762830015 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55570e668710 msgr2=0x55570e6782c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T10:26:31.785 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:31.779+0000 7fb7171a8740 1 --2- 172.21.15.114:0/2762830015 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55570e668710 0x55570e6782c0 secure :-1 s=READY pgs=391 cs=0 l=1 rev1=1 crypto rx=0x7fb708097ce0 tx=0x7fb70809bc30 comp rx=0 tx=0).stop 2024-04-23T10:26:31.785 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:31.779+0000 7fb7171a8740 1 -- 172.21.15.114:0/2762830015 shutdown_connections 2024-04-23T10:26:31.785 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:31.779+0000 7fb7171a8740 1 --2- 172.21.15.114:0/2762830015 >> [v2:172.21.15.119:6836/1872363128,v1:172.21.15.119:6837/1872363128] conn(0x55570e67ddb0 0x55570e67f220 crc :-1 s=READY pgs=18 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:26:31.785 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:31.779+0000 7fb7171a8740 1 --2- 172.21.15.114:0/2762830015 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55570e668710 0x55570e6782c0 unknown :-1 s=CLOSED pgs=391 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:26:31.785 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:31.779+0000 7fb6f5ffb640 0 client.15246 ms_handle_reset on v2:172.21.15.119:6836/1872363128 2024-04-23T10:26:31.785 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:31.779+0000 7fb7171a8740 1 --2- 172.21.15.114:0/2762830015 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x55570e667d50 0x55570e671280 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:26:31.785 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:31.779+0000 7fb7171a8740 1 --2- 172.21.15.114:0/2762830015 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x55570e666b80 0x55570e670d40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:26:31.785 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:31.779+0000 7fb7171a8740 1 -- 172.21.15.114:0/2762830015 >> 172.21.15.114:0/2762830015 conn(0x55570e662070 msgr2=0x55570e6611d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T10:26:31.785 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:31.779+0000 7fb7171a8740 1 -- 172.21.15.114:0/2762830015 shutdown_connections 2024-04-23T10:26:31.785 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:31.779+0000 7fb7171a8740 1 -- 172.21.15.114:0/2762830015 wait complete. 2024-04-23T10:26:31.785 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:31.779+0000 7fb7171a8740 20 client.15246 trim_cache size 0 max 16384 2024-04-23T10:26:31.801 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:31.796+0000 7fb7171a8740 1 Processor -- start 2024-04-23T10:26:31.801 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:31.797+0000 7fb7171a8740 1 -- start start 2024-04-23T10:26:31.801 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:31.797+0000 7fb7171a8740 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x55570e666b80 0x55570e65ce70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T10:26:31.802 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:31.797+0000 7fb7171a8740 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55570e667d50 0x55570e65d3b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T10:26:31.802 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:31.797+0000 7fb7171a8740 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x55570e668710 0x55570e678280 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T10:26:31.802 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:31.797+0000 7fb7171a8740 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x55570e67d560 con 0x55570e667d50 2024-04-23T10:26:31.802 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:31.797+0000 7fb7171a8740 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 -- 0x55570e67dc30 con 0x55570e666b80 2024-04-23T10:26:31.802 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:31.797+0000 7fb7171a8740 1 -- --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_getmap magic: 0 -- 0x55570e67e300 con 0x55570e668710 2024-04-23T10:26:31.802 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:31.797+0000 7fb711955640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55570e667d50 0x55570e65d3b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T10:26:31.802 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:31.797+0000 7fb711955640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55570e667d50 0x55570e65d3b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.114:58542/0 (socket says 172.21.15.114:58542) 2024-04-23T10:26:31.802 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:31.797+0000 7fb711955640 1 -- 172.21.15.114:0/2979349070 learned_addr learned my addr 172.21.15.114:0/2979349070 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-04-23T10:26:31.802 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:31.797+0000 7fb711955640 1 -- 172.21.15.114:0/2979349070 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x55570e668710 msgr2=0x55570e678280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T10:26:31.802 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:31.797+0000 7fb711955640 1 --2- 172.21.15.114:0/2979349070 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x55570e668710 0x55570e678280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:26:31.802 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:31.797+0000 7fb711955640 1 -- 172.21.15.114:0/2979349070 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x55570e666b80 msgr2=0x55570e65ce70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T10:26:31.802 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:31.797+0000 7fb711955640 1 --2- 172.21.15.114:0/2979349070 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x55570e666b80 0x55570e65ce70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:26:31.802 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:31.797+0000 7fb711955640 1 -- 172.21.15.114:0/2979349070 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55570e661ea0 con 0x55570e667d50 2024-04-23T10:26:31.802 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:31.797+0000 7fb711955640 1 --2- 172.21.15.114:0/2979349070 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55570e667d50 0x55570e65d3b0 secure :-1 s=READY pgs=392 cs=0 l=1 rev1=1 crypto rx=0x7fb6fc0143d0 tx=0x7fb6fc004730 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f16d401331620845 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T10:26:31.802 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:31.797+0000 7fb6f7fff640 1 -- 172.21.15.114:0/2979349070 <== mon.0 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb6fc026070 con 0x55570e667d50 2024-04-23T10:26:31.803 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:31.797+0000 7fb6f7fff640 1 -- 172.21.15.114:0/2979349070 <== mon.0 v2:172.21.15.114:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fb6fc012e00 con 0x55570e667d50 2024-04-23T10:26:31.803 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:31.797+0000 7fb6f7fff640 1 -- 172.21.15.114:0/2979349070 <== mon.0 v2:172.21.15.114:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb6fc0047a0 con 0x55570e667d50 2024-04-23T10:26:31.803 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:31.798+0000 7fb7171a8740 1 -- 172.21.15.114:0/2979349070 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55570e667d50 msgr2=0x55570e65d3b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T10:26:31.803 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:31.798+0000 7fb7171a8740 1 --2- 172.21.15.114:0/2979349070 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55570e667d50 0x55570e65d3b0 secure :-1 s=READY pgs=392 cs=0 l=1 rev1=1 crypto rx=0x7fb6fc0143d0 tx=0x7fb6fc004730 comp rx=0 tx=0).stop 2024-04-23T10:26:31.803 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:31.798+0000 7fb7171a8740 1 -- 172.21.15.114:0/2979349070 shutdown_connections 2024-04-23T10:26:31.803 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:31.798+0000 7fb7171a8740 1 --2- 172.21.15.114:0/2979349070 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x55570e668710 0x55570e678280 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:26:31.803 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:31.798+0000 7fb7171a8740 1 --2- 172.21.15.114:0/2979349070 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55570e667d50 0x55570e65d3b0 unknown :-1 s=CLOSED pgs=392 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:26:31.803 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:31.798+0000 7fb7171a8740 1 --2- 172.21.15.114:0/2979349070 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x55570e666b80 0x55570e65ce70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:26:31.803 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:31.798+0000 7fb7171a8740 1 -- 172.21.15.114:0/2979349070 >> 172.21.15.114:0/2979349070 conn(0x55570e662070 msgr2=0x55570e670d40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T10:26:31.803 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:31.798+0000 7fb7171a8740 1 -- 172.21.15.114:0/2979349070 shutdown_connections 2024-04-23T10:26:31.803 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:31.798+0000 7fb7171a8740 1 -- 172.21.15.114:0/2979349070 wait complete. 2024-04-23T10:26:31.803 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:31.798+0000 7fb7171a8740 1 -- start start 2024-04-23T10:26:31.803 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:31.798+0000 7fb7171a8740 1 Processor -- start 2024-04-23T10:26:31.803 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:31.798+0000 7fb7171a8740 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x55570e666b80 0x55570e673530 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T10:26:31.804 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:31.798+0000 7fb7171a8740 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55570e667d50 0x55570e673a70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T10:26:31.804 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:31.798+0000 7fb7171a8740 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x55570e668710 0x55570e676a90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T10:26:31.804 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:31.798+0000 7fb7171a8740 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x55570e67f0a0 con 0x55570e667d50 2024-04-23T10:26:31.804 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:31.798+0000 7fb7171a8740 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 -- 0x55570e67e9d0 con 0x55570e666b80 2024-04-23T10:26:31.804 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:31.798+0000 7fb7171a8740 1 -- --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_getmap magic: 0 -- 0x55570e67f770 con 0x55570e668710 2024-04-23T10:26:31.804 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:31.799+0000 7fb7171a8740 10 client.0 _ll_register_callbacks cb 0x7fb70809bea0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-23T10:26:31.804 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:31.799+0000 7fb711955640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55570e667d50 0x55570e673a70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T10:26:31.804 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:31.799+0000 7fb711955640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55570e667d50 0x55570e673a70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.114:58546/0 (socket says 172.21.15.114:58546) 2024-04-23T10:26:31.804 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:31.799+0000 7fb711955640 1 -- 172.21.15.114:0/838266158 learned_addr learned my addr 172.21.15.114:0/838266158 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-04-23T10:26:31.804 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:31.799+0000 7fb712156640 1 --2- 172.21.15.114:0/838266158 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x55570e666b80 0x55570e673530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T10:26:31.804 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:31.799+0000 7fb712957640 1 --2- 172.21.15.114:0/838266158 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x55570e668710 0x55570e676a90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T10:26:31.804 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:31.799+0000 7fb711955640 1 -- 172.21.15.114:0/838266158 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x55570e668710 msgr2=0x55570e676a90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T10:26:31.804 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:31.799+0000 7fb711955640 1 --2- 172.21.15.114:0/838266158 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x55570e668710 0x55570e676a90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:26:31.804 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:31.799+0000 7fb711955640 1 -- 172.21.15.114:0/838266158 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x55570e666b80 msgr2=0x55570e673530 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T10:26:31.804 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:31.799+0000 7fb711955640 1 --2- 172.21.15.114:0/838266158 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x55570e666b80 0x55570e673530 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:26:31.804 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:31.799+0000 7fb711955640 1 -- 172.21.15.114:0/838266158 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55570e677080 con 0x55570e667d50 2024-04-23T10:26:31.805 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:31.799+0000 7fb712957640 1 --2- 172.21.15.114:0/838266158 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x55570e668710 0x55570e676a90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-23T10:26:31.805 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:31.799+0000 7fb712156640 1 --2- 172.21.15.114:0/838266158 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x55570e666b80 0x55570e673530 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-23T10:26:31.805 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:31.799+0000 7fb711955640 1 --2- 172.21.15.114:0/838266158 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55570e667d50 0x55570e673a70 secure :-1 s=READY pgs=393 cs=0 l=1 rev1=1 crypto rx=0x7fb6fc014500 tx=0x7fb6fc011fb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d6765c278f624664 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T10:26:31.805 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:31.799+0000 7fb6f5ffb640 10 client.0 ms_handle_connect on v2:172.21.15.114:3300/0 2024-04-23T10:26:31.805 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:31.799+0000 7fb6f5ffb640 1 -- 172.21.15.114:0/838266158 <== mon.0 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb6fc038060 con 0x55570e667d50 2024-04-23T10:26:31.805 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:31.799+0000 7fb6f5ffb640 1 -- 172.21.15.114:0/838266158 <== mon.0 v2:172.21.15.114:3300/0 2 ==== osd_map(89..89 src has 1..89) ==== 8269+0+0 (secure 0 0 0) 0x7fb6fc012950 con 0x55570e667d50 2024-04-23T10:26:31.805 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:31.799+0000 7fb7171a8740 1 -- 172.21.15.114:0/838266158 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fb6fc012e00 con 0x55570e667d50 2024-04-23T10:26:31.805 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:31.799+0000 7fb6f5ffb640 1 -- 172.21.15.114:0/838266158 <== mon.0 v2:172.21.15.114:3300/0 3 ==== mon_get_version_reply(handle=1 version=20) ==== 24+0+0 (secure 0 0 0) 0x7fb6fc026070 con 0x55570e667d50 2024-04-23T10:26:31.805 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:31.799+0000 7fb7171a8740 10 client.15252 fetch_fsmap learned FSMap version 20 2024-04-23T10:26:31.805 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:31.799+0000 7fb7171a8740 1 -- 172.21.15.114:0/838266158 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({config=0+,fsmap=20,monmap=4+}) -- 0x55570e6783b0 con 0x55570e667d50 2024-04-23T10:26:31.805 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:31.799+0000 7fb6f5ffb640 1 -- 172.21.15.114:0/838266158 <== mon.0 v2:172.21.15.114:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fb6fc012e00 con 0x55570e667d50 2024-04-23T10:26:31.805 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:31.800+0000 7fb6f5ffb640 1 -- 172.21.15.114:0/838266158 <== mon.0 v2:172.21.15.114:3300/0 5 ==== fsmap(e 20) ==== 8338+0+0 (secure 0 0 0) 0x7fb6fc022440 con 0x55570e667d50 2024-04-23T10:26:31.806 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:31.800+0000 7fb7171a8740 10 client.15252 fetch_fsmap finished waiting for FSMap version 20 2024-04-23T10:26:31.806 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:31.800+0000 7fb7171a8740 10 client.15252 resolve_mds: resolved name 'h' to daemon mds.h 2024-04-23T10:26:31.806 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:31.800+0000 7fb7171a8740 20 client.15252 populate_metadata read hostname 'smithi114' 2024-04-23T10:26:31.806 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:31.800+0000 7fb7171a8740 1 --2- 172.21.15.114:0/838266158 >> [v2:172.21.15.119:6836/1872363128,v1:172.21.15.119:6837/1872363128] conn(0x55570e6e81d0 0x55570e6e6ee0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T10:26:31.806 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:31.800+0000 7fb7171a8740 4 client.15252 mds_command: new command op to 24509 tid=0 multi_id=0 [{"prefix": "status"}] 2024-04-23T10:26:31.806 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:31.800+0000 7fb7171a8740 1 -- 172.21.15.114:0/838266158 --> [v2:172.21.15.119:6836/1872363128,v1:172.21.15.119:6837/1872363128] -- command(tid 0: {"prefix": "status"}) -- 0x55570e5d34e0 con 0x55570e6e81d0 2024-04-23T10:26:31.806 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:31.800+0000 7fb712156640 1 --2- 172.21.15.114:0/838266158 >> [v2:172.21.15.119:6836/1872363128,v1:172.21.15.119:6837/1872363128] conn(0x55570e6e81d0 0x55570e6e6ee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T10:26:31.806 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:31.800+0000 7fb6f5ffb640 10 client.15252 ms_handle_connect on v2:172.21.15.119:6836/1872363128 2024-04-23T10:26:31.806 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:31.800+0000 7fb712156640 1 --2- 172.21.15.114:0/838266158 >> [v2:172.21.15.119:6836/1872363128,v1:172.21.15.119:6837/1872363128] conn(0x55570e6e81d0 0x55570e6e6ee0 crc :-1 s=READY pgs=19 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.1 client_cookie=ef6fc0a465e5a4ca server_cookie=c68b226f96da5ee0 in_seq=0 out_seq=0 2024-04-23T10:26:31.806 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:31.800+0000 7fb6f5ffb640 1 -- 172.21.15.114:0/838266158 <== mds.1 v2:172.21.15.119:6836/1872363128 1 ==== command_reply(tid 0: 0 ) ==== 8+0+327 (crc 0 0 0) 0x55570e5d34e0 con 0x55570e6e81d0 2024-04-23T10:26:31.806 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:31.800+0000 7fb6f5ffb640 10 client.15252 handle_command_reply: tid=0 2024-04-23T10:26:31.806 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:31.800+0000 7fb7171a8740 1 client.15252 shutdown 2024-04-23T10:26:31.806 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:31.801+0000 7fb7171a8740 1 -- 172.21.15.114:0/838266158 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55570e667d50 msgr2=0x55570e673a70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T10:26:31.806 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:31.801+0000 7fb7171a8740 1 --2- 172.21.15.114:0/838266158 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55570e667d50 0x55570e673a70 secure :-1 s=READY pgs=393 cs=0 l=1 rev1=1 crypto rx=0x7fb6fc014500 tx=0x7fb6fc011fb0 comp rx=0 tx=0).stop 2024-04-23T10:26:31.807 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:31.801+0000 7fb7171a8740 1 -- 172.21.15.114:0/838266158 shutdown_connections 2024-04-23T10:26:31.807 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:31.801+0000 7fb7171a8740 1 --2- 172.21.15.114:0/838266158 >> [v2:172.21.15.119:6836/1872363128,v1:172.21.15.119:6837/1872363128] conn(0x55570e6e81d0 0x55570e6e6ee0 crc :-1 s=READY pgs=19 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:26:31.807 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:31.801+0000 7fb7171a8740 1 --2- 172.21.15.114:0/838266158 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x55570e668710 0x55570e676a90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:26:31.807 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:31.801+0000 7fb7171a8740 1 --2- 172.21.15.114:0/838266158 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55570e667d50 0x55570e673a70 unknown :-1 s=CLOSED pgs=393 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:26:31.807 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:31.801+0000 7fb7171a8740 1 --2- 172.21.15.114:0/838266158 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x55570e666b80 0x55570e673530 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:26:31.807 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:31.801+0000 7fb6f5ffb640 0 client.15252 ms_handle_reset on v2:172.21.15.119:6836/1872363128 2024-04-23T10:26:31.807 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:31.801+0000 7fb7171a8740 1 -- 172.21.15.114:0/838266158 >> 172.21.15.114:0/838266158 conn(0x55570e662070 msgr2=0x55570e670d40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T10:26:31.807 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:31.801+0000 7fb7171a8740 1 -- 172.21.15.114:0/838266158 shutdown_connections 2024-04-23T10:26:31.807 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:31.801+0000 7fb7171a8740 1 -- 172.21.15.114:0/838266158 wait complete. 2024-04-23T10:26:31.807 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:31.801+0000 7fb7171a8740 20 client.15252 trim_cache size 0 max 16384 2024-04-23T10:26:31.808 INFO:teuthology.orchestra.run.smithi114.stdout:{ 2024-04-23T10:26:31.808 INFO:teuthology.orchestra.run.smithi114.stdout: "cluster_fsid": "aa240572-0159-11ef-bc93-c7b262605968", 2024-04-23T10:26:31.808 INFO:teuthology.orchestra.run.smithi114.stdout: "whoami": 1, 2024-04-23T10:26:31.808 INFO:teuthology.orchestra.run.smithi114.stdout: "id": 24509, 2024-04-23T10:26:31.808 INFO:teuthology.orchestra.run.smithi114.stdout: "want_state": "up:active", 2024-04-23T10:26:31.808 INFO:teuthology.orchestra.run.smithi114.stdout: "state": "up:active", 2024-04-23T10:26:31.808 INFO:teuthology.orchestra.run.smithi114.stdout: "fs_name": "cephfs", 2024-04-23T10:26:31.808 INFO:teuthology.orchestra.run.smithi114.stdout: "rank_uptime": 105.46413984500001, 2024-04-23T10:26:31.808 INFO:teuthology.orchestra.run.smithi114.stdout: "mdsmap_epoch": 20, 2024-04-23T10:26:31.808 INFO:teuthology.orchestra.run.smithi114.stdout: "osdmap_epoch": 89, 2024-04-23T10:26:31.808 INFO:teuthology.orchestra.run.smithi114.stdout: "osdmap_epoch_barrier": 87, 2024-04-23T10:26:31.808 INFO:teuthology.orchestra.run.smithi114.stdout: "uptime": 124.397205978 2024-04-23T10:26:31.808 INFO:teuthology.orchestra.run.smithi114.stdout:} 2024-04-23T10:26:31.809 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:31.801+0000 7fb7143ba640 1 -- 172.21.15.114:0/2515183953 >> [v2:172.21.15.114:6800/465333808,v1:172.21.15.114:6801/465333808] conn(0x7fb6f80a3c30 msgr2=0x7fb6f80a60f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T10:26:31.809 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:31.801+0000 7fb7143ba640 1 --2- 172.21.15.114:0/2515183953 >> [v2:172.21.15.114:6800/465333808,v1:172.21.15.114:6801/465333808] conn(0x7fb6f80a3c30 0x7fb6f80a60f0 secure :-1 s=READY pgs=245 cs=0 l=1 rev1=1 crypto rx=0x7fb70c199700 tx=0x7fb6fc00cbe0 comp rx=0 tx=0).stop 2024-04-23T10:26:31.809 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:31.801+0000 7fb7143ba640 1 -- 172.21.15.114:0/2515183953 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fb70c102a00 msgr2=0x7fb70c19f7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T10:26:31.809 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:31.801+0000 7fb7143ba640 1 --2- 172.21.15.114:0/2515183953 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fb70c102a00 0x7fb70c19f7a0 secure :-1 s=READY pgs=389 cs=0 l=1 rev1=1 crypto rx=0x7fb70800d750 tx=0x7fb70800dc10 comp rx=0 tx=0).stop 2024-04-23T10:26:31.809 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:31.801+0000 7fb7143ba640 1 -- 172.21.15.114:0/2515183953 shutdown_connections 2024-04-23T10:26:31.810 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:31.801+0000 7fb7143ba640 1 --2- 172.21.15.114:0/2515183953 >> [v2:172.21.15.114:6800/465333808,v1:172.21.15.114:6801/465333808] conn(0x7fb6f80a3c30 0x7fb6f80a60f0 unknown :-1 s=CLOSED pgs=245 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:26:31.810 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:31.801+0000 7fb7143ba640 1 --2- 172.21.15.114:0/2515183953 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fb70c102a00 0x7fb70c19f7a0 unknown :-1 s=CLOSED pgs=389 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:26:31.810 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:31.801+0000 7fb7143ba640 1 --2- 172.21.15.114:0/2515183953 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fb70c102040 0x7fb70c198730 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:26:31.810 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:31.801+0000 7fb7143ba640 1 --2- 172.21.15.114:0/2515183953 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fb70c100e60 0x7fb70c1981f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:26:31.810 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:31.801+0000 7fb7143ba640 1 -- 172.21.15.114:0/2515183953 >> 172.21.15.114:0/2515183953 conn(0x7fb70c0fc440 msgr2=0x7fb70c06a570 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T10:26:31.810 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:31.801+0000 7fb7143ba640 1 -- 172.21.15.114:0/2515183953 shutdown_connections 2024-04-23T10:26:31.810 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:31.801+0000 7fb7143ba640 1 -- 172.21.15.114:0/2515183953 wait complete. 2024-04-23T10:26:31.811 DEBUG:teuthology.orchestra.run.smithi114:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.b status 2024-04-23T10:26:31.912 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:31.910+0000 7ff91d679640 1 -- 172.21.15.114:0/1639840675 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7ff918105360 msgr2=0x7ff918107750 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T10:26:31.912 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:31.910+0000 7ff91d679640 1 --2- 172.21.15.114:0/1639840675 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7ff918105360 0x7ff918107750 secure :-1 s=READY pgs=394 cs=0 l=1 rev1=1 crypto rx=0x7ff90c009900 tx=0x7ff90c02a410 comp rx=0 tx=0).stop 2024-04-23T10:26:31.912 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:31.910+0000 7ff91d679640 1 -- 172.21.15.114:0/1639840675 shutdown_connections 2024-04-23T10:26:31.912 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:31.910+0000 7ff91d679640 1 --2- 172.21.15.114:0/1639840675 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7ff918107c90 0x7ff91810a0e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:26:31.912 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:31.910+0000 7ff91d679640 1 --2- 172.21.15.114:0/1639840675 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7ff918105360 0x7ff918107750 unknown :-1 s=CLOSED pgs=394 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:26:31.912 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:31.910+0000 7ff91d679640 1 --2- 172.21.15.114:0/1639840675 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7ff918102a30 0x7ff918104e20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:26:31.912 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:31.910+0000 7ff91d679640 1 -- 172.21.15.114:0/1639840675 >> 172.21.15.114:0/1639840675 conn(0x7ff9180fc440 msgr2=0x7ff9180fe860 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T10:26:31.912 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:31.911+0000 7ff91d679640 1 -- 172.21.15.114:0/1639840675 shutdown_connections 2024-04-23T10:26:31.912 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:31.911+0000 7ff91d679640 1 -- 172.21.15.114:0/1639840675 wait complete. 2024-04-23T10:26:31.913 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:31.911+0000 7ff91d679640 1 Processor -- start 2024-04-23T10:26:31.913 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:31.911+0000 7ff91d679640 1 -- start start 2024-04-23T10:26:31.913 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:31.911+0000 7ff91d679640 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7ff918102a30 0x7ff9181981e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T10:26:31.913 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:31.911+0000 7ff91d679640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7ff918105360 0x7ff918198720 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T10:26:31.913 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:31.911+0000 7ff91d679640 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7ff918107c90 0x7ff91819f790 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T10:26:31.913 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:31.911+0000 7ff91d679640 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x7ff91810daf0 con 0x7ff918105360 2024-04-23T10:26:31.913 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:31.911+0000 7ff91d679640 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 -- 0x7ff91810d970 con 0x7ff918102a30 2024-04-23T10:26:31.913 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:31.911+0000 7ff91d679640 1 -- --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_getmap magic: 0 -- 0x7ff91810dc70 con 0x7ff918107c90 2024-04-23T10:26:31.913 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:31.911+0000 7ff9167fc640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7ff918105360 0x7ff918198720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T10:26:31.914 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:31.911+0000 7ff9167fc640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7ff918105360 0x7ff918198720 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.114:58570/0 (socket says 172.21.15.114:58570) 2024-04-23T10:26:31.914 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:31.911+0000 7ff9167fc640 1 -- 172.21.15.114:0/3773376827 learned_addr learned my addr 172.21.15.114:0/3773376827 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-04-23T10:26:31.914 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:31.912+0000 7ff9177fe640 1 --2- 172.21.15.114:0/3773376827 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7ff918107c90 0x7ff91819f790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T10:26:31.914 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:31.912+0000 7ff916ffd640 1 --2- 172.21.15.114:0/3773376827 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7ff918102a30 0x7ff9181981e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T10:26:31.914 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:31.912+0000 7ff9167fc640 1 -- 172.21.15.114:0/3773376827 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7ff918107c90 msgr2=0x7ff91819f790 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T10:26:31.914 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:31.912+0000 7ff9167fc640 1 --2- 172.21.15.114:0/3773376827 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7ff918107c90 0x7ff91819f790 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:26:31.914 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:31.912+0000 7ff9167fc640 1 -- 172.21.15.114:0/3773376827 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7ff918102a30 msgr2=0x7ff9181981e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T10:26:31.914 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:31.912+0000 7ff9167fc640 1 --2- 172.21.15.114:0/3773376827 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7ff918102a30 0x7ff9181981e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:26:31.914 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:31.912+0000 7ff9167fc640 1 -- 172.21.15.114:0/3773376827 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff91819fe00 con 0x7ff918105360 2024-04-23T10:26:31.914 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:31.912+0000 7ff916ffd640 1 --2- 172.21.15.114:0/3773376827 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7ff918102a30 0x7ff9181981e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-23T10:26:31.915 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:31.912+0000 7ff9167fc640 1 --2- 172.21.15.114:0/3773376827 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7ff918105360 0x7ff918198720 secure :-1 s=READY pgs=395 cs=0 l=1 rev1=1 crypto rx=0x7ff90c006010 tx=0x7ff90c004430 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T10:26:31.915 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:31.912+0000 7ff8fffff640 1 -- 172.21.15.114:0/3773376827 <== mon.0 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff90c03f030 con 0x7ff918105360 2024-04-23T10:26:31.915 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:31.912+0000 7ff8fffff640 1 -- 172.21.15.114:0/3773376827 <== mon.0 v2:172.21.15.114:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7ff90c003cb0 con 0x7ff918105360 2024-04-23T10:26:31.915 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:31.912+0000 7ff91d679640 1 -- 172.21.15.114:0/3773376827 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7ff9181a0090 con 0x7ff918105360 2024-04-23T10:26:31.915 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:31.912+0000 7ff8fffff640 1 -- 172.21.15.114:0/3773376827 <== mon.0 v2:172.21.15.114:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff90c035070 con 0x7ff918105360 2024-04-23T10:26:31.915 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:31.912+0000 7ff91d679640 1 -- 172.21.15.114:0/3773376827 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7ff9181a0580 con 0x7ff918105360 2024-04-23T10:26:31.923 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:31.914+0000 7ff8fffff640 1 -- 172.21.15.114:0/3773376827 <== mon.0 v2:172.21.15.114:3300/0 4 ==== mgrmap(e 18) ==== 138647+0+0 (secure 0 0 0) 0x7ff90c042030 con 0x7ff918105360 2024-04-23T10:26:31.923 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:31.914+0000 7ff92038d740 1 Processor -- start 2024-04-23T10:26:31.924 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:31.914+0000 7ff92038d740 1 -- start start 2024-04-23T10:26:31.924 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:31.914+0000 7ff92038d740 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x55bcd71a1f90 0x55bcd71a23b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T10:26:31.924 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:31.915+0000 7ff92038d740 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x55bcd71a3160 0x55bcd71a35e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T10:26:31.924 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:31.915+0000 7ff92038d740 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55bcd71a3b20 0x55bcd71aa3b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T10:26:31.924 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:31.915+0000 7ff92038d740 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x55bcd7102c50 con 0x55bcd71a3b20 2024-04-23T10:26:31.924 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:31.915+0000 7ff92038d740 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 -- 0x55bcd709f830 con 0x55bcd71a3160 2024-04-23T10:26:31.924 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:31.915+0000 7ff92038d740 1 -- --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_getmap magic: 0 -- 0x55bcd70fec10 con 0x55bcd71a1f90 2024-04-23T10:26:31.924 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:31.915+0000 7ff9167fc640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55bcd71a3b20 0x55bcd71aa3b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T10:26:31.924 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:31.915+0000 7ff9167fc640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55bcd71a3b20 0x55bcd71aa3b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.114:58584/0 (socket says 172.21.15.114:58584) 2024-04-23T10:26:31.924 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:31.915+0000 7ff9167fc640 1 -- 172.21.15.114:0/762083308 learned_addr learned my addr 172.21.15.114:0/762083308 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-04-23T10:26:31.924 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:31.915+0000 7ff916ffd640 1 --2- 172.21.15.114:0/762083308 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x55bcd71a3160 0x55bcd71a35e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T10:26:31.924 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:31.915+0000 7ff9177fe640 1 --2- 172.21.15.114:0/762083308 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x55bcd71a1f90 0x55bcd71a23b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T10:26:31.924 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:31.915+0000 7ff9167fc640 1 -- 172.21.15.114:0/762083308 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x55bcd71a1f90 msgr2=0x55bcd71a23b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T10:26:31.924 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:31.915+0000 7ff9167fc640 1 --2- 172.21.15.114:0/762083308 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x55bcd71a1f90 0x55bcd71a23b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:26:31.924 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:31.915+0000 7ff9167fc640 1 -- 172.21.15.114:0/762083308 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x55bcd71a3160 msgr2=0x55bcd71a35e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T10:26:31.925 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:31.915+0000 7ff9167fc640 1 --2- 172.21.15.114:0/762083308 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x55bcd71a3160 0x55bcd71a35e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:26:31.925 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:31.915+0000 7ff8fffff640 1 --2- 172.21.15.114:0/3773376827 >> [v2:172.21.15.114:6800/465333808,v1:172.21.15.114:6801/465333808] conn(0x7ff9000a3bb0 0x7ff9000a6070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T10:26:31.925 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:31.915+0000 7ff9167fc640 1 -- 172.21.15.114:0/762083308 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55bcd6e152d0 con 0x55bcd71a3b20 2024-04-23T10:26:31.925 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:31.915+0000 7ff9177fe640 1 --2- 172.21.15.114:0/3773376827 >> [v2:172.21.15.114:6800/465333808,v1:172.21.15.114:6801/465333808] conn(0x7ff9000a3bb0 0x7ff9000a6070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T10:26:31.925 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:31.915+0000 7ff9167fc640 1 --2- 172.21.15.114:0/762083308 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55bcd71a3b20 0x55bcd71aa3b0 secure :-1 s=READY pgs=396 cs=0 l=1 rev1=1 crypto rx=0x7ff90c0f4cd0 tx=0x7ff90c0b9090 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9940231f1e85ae59 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T10:26:31.925 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:31.915+0000 7ff8fffff640 1 -- 172.21.15.114:0/3773376827 <== mon.0 v2:172.21.15.114:3300/0 5 ==== osd_map(89..89 src has 1..89) ==== 8269+0+0 (secure 0 0 0) 0x7ff90c0f0b30 con 0x7ff918105360 2024-04-23T10:26:31.925 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:31.915+0000 7ff8fd7fa640 1 -- 172.21.15.114:0/762083308 <== mon.0 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff90c0c0070 con 0x55bcd71a3b20 2024-04-23T10:26:31.925 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:31.915+0000 7ff8fd7fa640 1 -- 172.21.15.114:0/762083308 <== mon.0 v2:172.21.15.114:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7ff90c0baef0 con 0x55bcd71a3b20 2024-04-23T10:26:31.925 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:31.915+0000 7ff8fd7fa640 1 -- 172.21.15.114:0/762083308 <== mon.0 v2:172.21.15.114:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff90c0bc930 con 0x55bcd71a3b20 2024-04-23T10:26:31.925 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:31.916+0000 7ff92038d740 1 -- 172.21.15.114:0/762083308 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55bcd71a3b20 msgr2=0x55bcd71aa3b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T10:26:31.925 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:31.916+0000 7ff9177fe640 1 --2- 172.21.15.114:0/3773376827 >> [v2:172.21.15.114:6800/465333808,v1:172.21.15.114:6801/465333808] conn(0x7ff9000a3bb0 0x7ff9000a6070 secure :-1 s=READY pgs=246 cs=0 l=1 rev1=1 crypto rx=0x55bcd71a3060 tx=0x7ff910005f60 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T10:26:31.925 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:31.916+0000 7ff92038d740 1 --2- 172.21.15.114:0/762083308 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55bcd71a3b20 0x55bcd71aa3b0 secure :-1 s=READY pgs=396 cs=0 l=1 rev1=1 crypto rx=0x7ff90c0f4cd0 tx=0x7ff90c0b9090 comp rx=0 tx=0).stop 2024-04-23T10:26:31.925 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:31.916+0000 7ff92038d740 1 -- 172.21.15.114:0/762083308 shutdown_connections 2024-04-23T10:26:31.925 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:31.916+0000 7ff92038d740 1 --2- 172.21.15.114:0/762083308 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55bcd71a3b20 0x55bcd71aa3b0 unknown :-1 s=CLOSED pgs=396 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:26:31.925 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:31.916+0000 7ff92038d740 1 --2- 172.21.15.114:0/762083308 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x55bcd71a3160 0x55bcd71a35e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:26:31.925 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:31.916+0000 7ff92038d740 1 --2- 172.21.15.114:0/762083308 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x55bcd71a1f90 0x55bcd71a23b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:26:31.926 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:31.916+0000 7ff92038d740 1 -- 172.21.15.114:0/762083308 >> 172.21.15.114:0/762083308 conn(0x55bcd7199070 msgr2=0x55bcd7199470 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T10:26:31.926 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:31.916+0000 7ff92038d740 1 -- 172.21.15.114:0/762083308 shutdown_connections 2024-04-23T10:26:31.926 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:31.916+0000 7ff92038d740 1 -- 172.21.15.114:0/762083308 wait complete. 2024-04-23T10:26:31.926 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:31.916+0000 7ff92038d740 1 -- start start 2024-04-23T10:26:31.926 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:31.916+0000 7ff92038d740 1 Processor -- start 2024-04-23T10:26:31.926 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:31.917+0000 7ff92038d740 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x55bcd71a1f90 0x55bcd71ac1c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T10:26:31.926 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:31.917+0000 7ff92038d740 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x55bcd71a3160 0x55bcd71ac700 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T10:26:31.926 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:31.917+0000 7ff92038d740 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55bcd71a3b20 0x55bcd71b3740 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T10:26:31.926 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:31.917+0000 7ff92038d740 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x55bcd71aa8f0 con 0x55bcd71a3b20 2024-04-23T10:26:31.926 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:31.917+0000 7ff92038d740 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 -- 0x55bcd709d470 con 0x55bcd71a3160 2024-04-23T10:26:31.926 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:31.917+0000 7ff92038d740 1 -- --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_getmap magic: 0 -- 0x55bcd71aaa70 con 0x55bcd71a1f90 2024-04-23T10:26:31.926 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:31.917+0000 7ff92038d740 10 client.0 _ll_register_callbacks cb 0x55bcd70c2f30 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-23T10:26:31.926 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:31.917+0000 7ff9177fe640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55bcd71a3b20 0x55bcd71b3740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T10:26:31.926 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:31.917+0000 7ff9177fe640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55bcd71a3b20 0x55bcd71b3740 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.114:58588/0 (socket says 172.21.15.114:58588) 2024-04-23T10:26:31.927 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:31.917+0000 7ff9177fe640 1 -- 172.21.15.114:0/1557336723 learned_addr learned my addr 172.21.15.114:0/1557336723 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-04-23T10:26:31.927 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:31.917+0000 7ff9167fc640 1 --2- 172.21.15.114:0/1557336723 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x55bcd71a3160 0x55bcd71ac700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T10:26:31.927 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:31.917+0000 7ff916ffd640 1 --2- 172.21.15.114:0/1557336723 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x55bcd71a1f90 0x55bcd71ac1c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T10:26:31.927 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:31.917+0000 7ff9177fe640 1 -- 172.21.15.114:0/1557336723 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x55bcd71a1f90 msgr2=0x55bcd71ac1c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T10:26:31.927 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:31.917+0000 7ff9177fe640 1 --2- 172.21.15.114:0/1557336723 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x55bcd71a1f90 0x55bcd71ac1c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:26:31.927 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:31.917+0000 7ff9177fe640 1 -- 172.21.15.114:0/1557336723 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x55bcd71a3160 msgr2=0x55bcd71ac700 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T10:26:31.927 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:31.917+0000 7ff9177fe640 1 --2- 172.21.15.114:0/1557336723 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x55bcd71a3160 0x55bcd71ac700 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:26:31.927 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:31.917+0000 7ff9177fe640 1 -- 172.21.15.114:0/1557336723 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55bcd6edf6b0 con 0x55bcd71a3b20 2024-04-23T10:26:31.927 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:31.917+0000 7ff916ffd640 1 --2- 172.21.15.114:0/1557336723 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x55bcd71a1f90 0x55bcd71ac1c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-23T10:26:31.927 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:31.917+0000 7ff9177fe640 1 --2- 172.21.15.114:0/1557336723 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55bcd71a3b20 0x55bcd71b3740 secure :-1 s=READY pgs=397 cs=0 l=1 rev1=1 crypto rx=0x7ff910005dd0 tx=0x7ff9100046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3c91ae800e6c499 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T10:26:31.927 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:31.917+0000 7ff8faff5640 10 client.0 ms_handle_connect on v2:172.21.15.114:3300/0 2024-04-23T10:26:31.927 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:31.917+0000 7ff8faff5640 1 -- 172.21.15.114:0/1557336723 <== mon.0 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff91002a060 con 0x55bcd71a3b20 2024-04-23T10:26:31.927 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:31.917+0000 7ff8faff5640 1 -- 172.21.15.114:0/1557336723 <== mon.0 v2:172.21.15.114:3300/0 2 ==== osd_map(89..89 src has 1..89) ==== 8269+0+0 (secure 0 0 0) 0x7ff91001db10 con 0x55bcd71a3b20 2024-04-23T10:26:31.927 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:31.917+0000 7ff92038d740 1 -- 172.21.15.114:0/1557336723 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7ff90c0baef0 con 0x55bcd71a3b20 2024-04-23T10:26:31.927 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:31.917+0000 7ff8faff5640 1 -- 172.21.15.114:0/1557336723 <== mon.0 v2:172.21.15.114:3300/0 3 ==== mon_get_version_reply(handle=1 version=20) ==== 24+0+0 (secure 0 0 0) 0x7ff91001dd10 con 0x55bcd71a3b20 2024-04-23T10:26:31.927 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:31.917+0000 7ff92038d740 10 client.15264 fetch_fsmap learned FSMap version 20 2024-04-23T10:26:31.928 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:31.917+0000 7ff92038d740 1 -- 172.21.15.114:0/1557336723 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({config=0+,fsmap=20,monmap=4+}) -- 0x55bcd71b6b80 con 0x55bcd71a3b20 2024-04-23T10:26:31.928 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:31.918+0000 7ff8faff5640 1 -- 172.21.15.114:0/1557336723 <== mon.0 v2:172.21.15.114:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7ff90c0baef0 con 0x55bcd71a3b20 2024-04-23T10:26:31.928 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:31.918+0000 7ff8faff5640 1 -- 172.21.15.114:0/1557336723 <== mon.0 v2:172.21.15.114:3300/0 5 ==== fsmap(e 20) ==== 8338+0+0 (secure 0 0 0) 0x7ff910019040 con 0x55bcd71a3b20 2024-04-23T10:26:31.928 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:31.918+0000 7ff92038d740 10 client.15264 fetch_fsmap finished waiting for FSMap version 20 2024-04-23T10:26:31.928 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:31.918+0000 7ff92038d740 10 client.15264 resolve_mds: resolved name 'b' to daemon mds.b 2024-04-23T10:26:31.928 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:31.918+0000 7ff92038d740 20 client.15264 populate_metadata read hostname 'smithi114' 2024-04-23T10:26:31.928 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:31.918+0000 7ff92038d740 1 --2- 172.21.15.114:0/1557336723 >> [v2:172.21.15.119:6838/937297340,v1:172.21.15.119:6839/937297340] conn(0x55bcd71b9230 0x55bcd71ba6a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T10:26:31.928 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:31.918+0000 7ff92038d740 4 client.15264 mds_command: new command op to 24545 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-23T10:26:31.928 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:31.918+0000 7ff92038d740 1 -- 172.21.15.114:0/1557336723 --> [v2:172.21.15.119:6838/937297340,v1:172.21.15.119:6839/937297340] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55bcd7110670 con 0x55bcd71b9230 2024-04-23T10:26:31.928 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:31.918+0000 7ff916ffd640 1 --2- 172.21.15.114:0/1557336723 >> [v2:172.21.15.119:6838/937297340,v1:172.21.15.119:6839/937297340] conn(0x55bcd71b9230 0x55bcd71ba6a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T10:26:31.928 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:31.918+0000 7ff916ffd640 1 --2- 172.21.15.114:0/1557336723 >> [v2:172.21.15.119:6838/937297340,v1:172.21.15.119:6839/937297340] conn(0x55bcd71b9230 0x55bcd71ba6a0 crc :-1 s=READY pgs=25 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=b1927dc2cab5e6b4 server_cookie=7ab4595241d8cf75 in_seq=0 out_seq=0 2024-04-23T10:26:31.928 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:31.918+0000 7ff8faff5640 10 client.15264 ms_handle_connect on v2:172.21.15.119:6838/937297340 2024-04-23T10:26:31.928 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:31.919+0000 7ff8faff5640 1 -- 172.21.15.114:0/1557336723 <== mds.0 v2:172.21.15.119:6838/937297340 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x55bcd7110670 con 0x55bcd71b9230 2024-04-23T10:26:31.928 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:31.919+0000 7ff8faff5640 10 client.15264 handle_command_reply: tid=0 2024-04-23T10:26:31.929 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:31.919+0000 7ff92038d740 1 client.15264 shutdown 2024-04-23T10:26:31.929 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:31.919+0000 7ff92038d740 1 -- 172.21.15.114:0/1557336723 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55bcd71a3b20 msgr2=0x55bcd71b3740 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T10:26:31.929 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:31.919+0000 7ff92038d740 1 --2- 172.21.15.114:0/1557336723 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55bcd71a3b20 0x55bcd71b3740 secure :-1 s=READY pgs=397 cs=0 l=1 rev1=1 crypto rx=0x7ff910005dd0 tx=0x7ff9100046c0 comp rx=0 tx=0).stop 2024-04-23T10:26:31.929 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:31.919+0000 7ff92038d740 1 -- 172.21.15.114:0/1557336723 shutdown_connections 2024-04-23T10:26:31.929 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:31.919+0000 7ff92038d740 1 --2- 172.21.15.114:0/1557336723 >> [v2:172.21.15.119:6838/937297340,v1:172.21.15.119:6839/937297340] conn(0x55bcd71b9230 0x55bcd71ba6a0 crc :-1 s=READY pgs=25 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:26:31.929 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:31.919+0000 7ff92038d740 1 --2- 172.21.15.114:0/1557336723 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55bcd71a3b20 0x55bcd71b3740 unknown :-1 s=CLOSED pgs=397 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:26:31.929 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:31.919+0000 7ff8faff5640 0 client.15264 ms_handle_reset on v2:172.21.15.119:6838/937297340 2024-04-23T10:26:31.929 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:31.919+0000 7ff92038d740 1 --2- 172.21.15.114:0/1557336723 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x55bcd71a3160 0x55bcd71ac700 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:26:31.929 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:31.919+0000 7ff92038d740 1 --2- 172.21.15.114:0/1557336723 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x55bcd71a1f90 0x55bcd71ac1c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:26:31.929 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:31.919+0000 7ff92038d740 1 -- 172.21.15.114:0/1557336723 >> 172.21.15.114:0/1557336723 conn(0x55bcd7199070 msgr2=0x55bcd71981d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T10:26:31.929 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:31.919+0000 7ff92038d740 1 -- 172.21.15.114:0/1557336723 shutdown_connections 2024-04-23T10:26:31.929 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:31.919+0000 7ff92038d740 1 -- 172.21.15.114:0/1557336723 wait complete. 2024-04-23T10:26:31.929 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:31.919+0000 7ff92038d740 20 client.15264 trim_cache size 0 max 16384 2024-04-23T10:26:31.942 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:31.937+0000 7ff92038d740 1 Processor -- start 2024-04-23T10:26:31.942 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:31.937+0000 7ff92038d740 1 -- start start 2024-04-23T10:26:31.942 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:31.938+0000 7ff92038d740 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x55bcd71a1f90 0x55bcd71a3540 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T10:26:31.942 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:31.938+0000 7ff92038d740 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x55bcd71a3160 0x55bcd71b3660 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T10:26:31.942 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:31.938+0000 7ff92038d740 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55bcd71a3b20 0x55bcd71a6ca0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T10:26:31.942 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:31.938+0000 7ff92038d740 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x55bcd71b89e0 con 0x55bcd71a3b20 2024-04-23T10:26:31.942 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:31.938+0000 7ff92038d740 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 -- 0x55bcd71b90b0 con 0x55bcd71a1f90 2024-04-23T10:26:31.943 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:31.938+0000 7ff92038d740 1 -- --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_getmap magic: 0 -- 0x55bcd71b9780 con 0x55bcd71a3160 2024-04-23T10:26:31.943 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:31.938+0000 7ff9177fe640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55bcd71a3b20 0x55bcd71a6ca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T10:26:31.943 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:31.938+0000 7ff9177fe640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55bcd71a3b20 0x55bcd71a6ca0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.114:58592/0 (socket says 172.21.15.114:58592) 2024-04-23T10:26:31.943 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:31.938+0000 7ff9177fe640 1 -- 172.21.15.114:0/1750070939 learned_addr learned my addr 172.21.15.114:0/1750070939 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-04-23T10:26:31.943 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:31.938+0000 7ff9177fe640 1 -- 172.21.15.114:0/1750070939 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x55bcd71a3160 msgr2=0x55bcd71b3660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T10:26:31.943 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:31.938+0000 7ff9177fe640 1 --2- 172.21.15.114:0/1750070939 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x55bcd71a3160 0x55bcd71b3660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:26:31.943 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:31.938+0000 7ff9177fe640 1 -- 172.21.15.114:0/1750070939 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x55bcd71a1f90 msgr2=0x55bcd71a3540 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T10:26:31.943 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:31.938+0000 7ff9177fe640 1 --2- 172.21.15.114:0/1750070939 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x55bcd71a1f90 0x55bcd71a3540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:26:31.943 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:31.938+0000 7ff9177fe640 1 -- 172.21.15.114:0/1750070939 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55bcd71ba9d0 con 0x55bcd71a3b20 2024-04-23T10:26:31.943 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:31.938+0000 7ff9177fe640 1 --2- 172.21.15.114:0/1750070939 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55bcd71a3b20 0x55bcd71a6ca0 secure :-1 s=READY pgs=398 cs=0 l=1 rev1=1 crypto rx=0x7ff91000a550 tx=0x7ff9100107a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=464e346a576eb35a server_cookie=0 in_seq=0 out_seq=0 2024-04-23T10:26:31.943 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:31.938+0000 7ff8fcff9640 1 -- 172.21.15.114:0/1750070939 <== mon.0 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff91001db10 con 0x55bcd71a3b20 2024-04-23T10:26:31.943 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:31.938+0000 7ff8fcff9640 1 -- 172.21.15.114:0/1750070939 <== mon.0 v2:172.21.15.114:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7ff910025750 con 0x55bcd71a3b20 2024-04-23T10:26:31.943 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:31.938+0000 7ff8fcff9640 1 -- 172.21.15.114:0/1750070939 <== mon.0 v2:172.21.15.114:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff910004730 con 0x55bcd71a3b20 2024-04-23T10:26:31.943 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:31.939+0000 7ff92038d740 1 -- 172.21.15.114:0/1750070939 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55bcd71a3b20 msgr2=0x55bcd71a6ca0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T10:26:31.943 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:31.939+0000 7ff92038d740 1 --2- 172.21.15.114:0/1750070939 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55bcd71a3b20 0x55bcd71a6ca0 secure :-1 s=READY pgs=398 cs=0 l=1 rev1=1 crypto rx=0x7ff91000a550 tx=0x7ff9100107a0 comp rx=0 tx=0).stop 2024-04-23T10:26:31.943 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:31.939+0000 7ff92038d740 1 -- 172.21.15.114:0/1750070939 shutdown_connections 2024-04-23T10:26:31.944 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:31.939+0000 7ff92038d740 1 --2- 172.21.15.114:0/1750070939 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55bcd71a3b20 0x55bcd71a6ca0 unknown :-1 s=CLOSED pgs=398 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:26:31.944 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:31.939+0000 7ff92038d740 1 --2- 172.21.15.114:0/1750070939 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x55bcd71a3160 0x55bcd71b3660 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:26:31.944 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:31.939+0000 7ff92038d740 1 --2- 172.21.15.114:0/1750070939 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x55bcd71a1f90 0x55bcd71a3540 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:26:31.944 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:31.939+0000 7ff92038d740 1 -- 172.21.15.114:0/1750070939 >> 172.21.15.114:0/1750070939 conn(0x55bcd7199070 msgr2=0x55bcd71940b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T10:26:31.944 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:31.939+0000 7ff92038d740 1 -- 172.21.15.114:0/1750070939 shutdown_connections 2024-04-23T10:26:31.944 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:31.939+0000 7ff92038d740 1 -- 172.21.15.114:0/1750070939 wait complete. 2024-04-23T10:26:31.944 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:31.939+0000 7ff92038d740 1 -- start start 2024-04-23T10:26:31.944 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:31.939+0000 7ff92038d740 1 Processor -- start 2024-04-23T10:26:31.944 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:31.939+0000 7ff92038d740 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x55bcd71a1f90 0x55bcd71af270 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T10:26:31.944 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:31.939+0000 7ff92038d740 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55bcd71a3160 0x55bcd71af7b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T10:26:31.944 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:31.939+0000 7ff92038d740 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x55bcd71a3b20 0x55bcd71b27d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T10:26:31.944 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:31.939+0000 7ff92038d740 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x55bcd71ba520 con 0x55bcd71a3160 2024-04-23T10:26:31.944 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:31.939+0000 7ff92038d740 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 -- 0x55bcd71b9e50 con 0x55bcd71a3b20 2024-04-23T10:26:31.944 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:31.939+0000 7ff92038d740 1 -- --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_getmap magic: 0 -- 0x55bcd71babf0 con 0x55bcd71a1f90 2024-04-23T10:26:31.944 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:31.939+0000 7ff92038d740 10 client.0 _ll_register_callbacks cb 0x7ff910005ac0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-23T10:26:31.944 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:31.939+0000 7ff9167fc640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55bcd71a3160 0x55bcd71af7b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T10:26:31.944 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:31.940+0000 7ff9167fc640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55bcd71a3160 0x55bcd71af7b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.114:58602/0 (socket says 172.21.15.114:58602) 2024-04-23T10:26:31.945 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:31.940+0000 7ff9167fc640 1 -- 172.21.15.114:0/2991830219 learned_addr learned my addr 172.21.15.114:0/2991830219 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-04-23T10:26:31.945 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:31.940+0000 7ff9177fe640 1 --2- 172.21.15.114:0/2991830219 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x55bcd71a3b20 0x55bcd71b27d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T10:26:31.945 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:31.940+0000 7ff916ffd640 1 --2- 172.21.15.114:0/2991830219 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x55bcd71a1f90 0x55bcd71af270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T10:26:31.945 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:31.940+0000 7ff9167fc640 1 -- 172.21.15.114:0/2991830219 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x55bcd71a1f90 msgr2=0x55bcd71af270 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T10:26:31.945 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:31.940+0000 7ff9167fc640 1 --2- 172.21.15.114:0/2991830219 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x55bcd71a1f90 0x55bcd71af270 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:26:31.945 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:31.940+0000 7ff9167fc640 1 -- 172.21.15.114:0/2991830219 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x55bcd71a3b20 msgr2=0x55bcd71b27d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T10:26:31.945 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:31.940+0000 7ff9167fc640 1 --2- 172.21.15.114:0/2991830219 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x55bcd71a3b20 0x55bcd71b27d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:26:31.945 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:31.940+0000 7ff9167fc640 1 -- 172.21.15.114:0/2991830219 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55bcd71b2dc0 con 0x55bcd71a3160 2024-04-23T10:26:31.945 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:31.940+0000 7ff9177fe640 1 --2- 172.21.15.114:0/2991830219 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x55bcd71a3b20 0x55bcd71b27d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-23T10:26:31.945 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:31.940+0000 7ff9167fc640 1 --2- 172.21.15.114:0/2991830219 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55bcd71a3160 0x55bcd71af7b0 secure :-1 s=READY pgs=399 cs=0 l=1 rev1=1 crypto rx=0x7ff90c0f4c20 tx=0x7ff90c032f60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e68fdf16c4c2bea3 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T10:26:31.945 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:31.940+0000 7ff8faff5640 10 client.0 ms_handle_connect on v2:172.21.15.114:3300/0 2024-04-23T10:26:31.945 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:31.940+0000 7ff8faff5640 1 -- 172.21.15.114:0/2991830219 <== mon.0 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff90c0c0040 con 0x55bcd71a3160 2024-04-23T10:26:31.945 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:31.940+0000 7ff8faff5640 1 -- 172.21.15.114:0/2991830219 <== mon.0 v2:172.21.15.114:3300/0 2 ==== osd_map(89..89 src has 1..89) ==== 8269+0+0 (secure 0 0 0) 0x7ff90c0f43e0 con 0x55bcd71a3160 2024-04-23T10:26:31.945 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:31.940+0000 7ff92038d740 1 -- 172.21.15.114:0/2991830219 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7ff910025750 con 0x55bcd71a3160 2024-04-23T10:26:31.946 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:31.940+0000 7ff8faff5640 1 -- 172.21.15.114:0/2991830219 <== mon.0 v2:172.21.15.114:3300/0 3 ==== mon_get_version_reply(handle=1 version=20) ==== 24+0+0 (secure 0 0 0) 0x7ff90c049040 con 0x55bcd71a3160 2024-04-23T10:26:31.946 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:31.940+0000 7ff92038d740 10 client.15270 fetch_fsmap learned FSMap version 20 2024-04-23T10:26:31.946 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:31.940+0000 7ff92038d740 1 -- 172.21.15.114:0/2991830219 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({config=0+,fsmap=20,monmap=4+}) -- 0x55bcd721f6f0 con 0x55bcd71a3160 2024-04-23T10:26:31.946 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:31.940+0000 7ff8faff5640 1 -- 172.21.15.114:0/2991830219 <== mon.0 v2:172.21.15.114:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7ff910025750 con 0x55bcd71a3160 2024-04-23T10:26:31.946 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:31.940+0000 7ff8faff5640 1 -- 172.21.15.114:0/2991830219 <== mon.0 v2:172.21.15.114:3300/0 5 ==== fsmap(e 20) ==== 8338+0+0 (secure 0 0 0) 0x7ff90c0b9460 con 0x55bcd71a3160 2024-04-23T10:26:31.946 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:31.940+0000 7ff92038d740 10 client.15270 fetch_fsmap finished waiting for FSMap version 20 2024-04-23T10:26:31.946 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:31.940+0000 7ff92038d740 10 client.15270 resolve_mds: resolved name 'b' to daemon mds.b 2024-04-23T10:26:31.946 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:31.940+0000 7ff92038d740 20 client.15270 populate_metadata read hostname 'smithi114' 2024-04-23T10:26:31.946 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:31.941+0000 7ff92038d740 1 --2- 172.21.15.114:0/2991830219 >> [v2:172.21.15.119:6838/937297340,v1:172.21.15.119:6839/937297340] conn(0x55bcd7223eb0 0x55bcd72202f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T10:26:31.946 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:31.941+0000 7ff92038d740 4 client.15270 mds_command: new command op to 24545 tid=0 multi_id=0 [{"prefix": "status"}] 2024-04-23T10:26:31.946 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:31.941+0000 7ff92038d740 1 -- 172.21.15.114:0/2991830219 --> [v2:172.21.15.119:6838/937297340,v1:172.21.15.119:6839/937297340] -- command(tid 0: {"prefix": "status"}) -- 0x55bcd710a4e0 con 0x55bcd7223eb0 2024-04-23T10:26:31.946 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:31.941+0000 7ff916ffd640 1 --2- 172.21.15.114:0/2991830219 >> [v2:172.21.15.119:6838/937297340,v1:172.21.15.119:6839/937297340] conn(0x55bcd7223eb0 0x55bcd72202f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T10:26:31.946 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:31.941+0000 7ff8faff5640 10 client.15270 ms_handle_connect on v2:172.21.15.119:6838/937297340 2024-04-23T10:26:31.946 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:31.941+0000 7ff916ffd640 1 --2- 172.21.15.114:0/2991830219 >> [v2:172.21.15.119:6838/937297340,v1:172.21.15.119:6839/937297340] conn(0x55bcd7223eb0 0x55bcd72202f0 crc :-1 s=READY pgs=26 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=11bff3ea5b5065b server_cookie=c75f2b9154d127e8 in_seq=0 out_seq=0 2024-04-23T10:26:31.947 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:31.941+0000 7ff8faff5640 1 -- 172.21.15.114:0/2991830219 <== mds.0 v2:172.21.15.119:6838/937297340 1 ==== command_reply(tid 0: 0 ) ==== 8+0+327 (crc 0 0 0) 0x55bcd710a4e0 con 0x55bcd7223eb0 2024-04-23T10:26:31.947 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:31.941+0000 7ff8faff5640 10 client.15270 handle_command_reply: tid=0 2024-04-23T10:26:31.947 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:31.941+0000 7ff92038d740 1 client.15270 shutdown 2024-04-23T10:26:31.947 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:31.941+0000 7ff92038d740 1 -- 172.21.15.114:0/2991830219 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55bcd71a3160 msgr2=0x55bcd71af7b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T10:26:31.947 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:31.941+0000 7ff92038d740 1 --2- 172.21.15.114:0/2991830219 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55bcd71a3160 0x55bcd71af7b0 secure :-1 s=READY pgs=399 cs=0 l=1 rev1=1 crypto rx=0x7ff90c0f4c20 tx=0x7ff90c032f60 comp rx=0 tx=0).stop 2024-04-23T10:26:31.947 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:31.941+0000 7ff92038d740 1 -- 172.21.15.114:0/2991830219 shutdown_connections 2024-04-23T10:26:31.947 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:31.941+0000 7ff92038d740 1 --2- 172.21.15.114:0/2991830219 >> [v2:172.21.15.119:6838/937297340,v1:172.21.15.119:6839/937297340] conn(0x55bcd7223eb0 0x55bcd72202f0 crc :-1 s=READY pgs=26 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:26:31.947 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:31.941+0000 7ff92038d740 1 --2- 172.21.15.114:0/2991830219 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x55bcd71a3b20 0x55bcd71b27d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:26:31.947 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:31.941+0000 7ff92038d740 1 --2- 172.21.15.114:0/2991830219 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55bcd71a3160 0x55bcd71af7b0 unknown :-1 s=CLOSED pgs=399 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:26:31.947 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:31.941+0000 7ff8faff5640 0 client.15270 ms_handle_reset on v2:172.21.15.119:6838/937297340 2024-04-23T10:26:31.947 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:31.941+0000 7ff92038d740 1 --2- 172.21.15.114:0/2991830219 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x55bcd71a1f90 0x55bcd71af270 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:26:31.947 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:31.941+0000 7ff92038d740 1 -- 172.21.15.114:0/2991830219 >> 172.21.15.114:0/2991830219 conn(0x55bcd7199070 msgr2=0x55bcd71a6ca0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T10:26:31.947 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:31.942+0000 7ff92038d740 1 -- 172.21.15.114:0/2991830219 shutdown_connections 2024-04-23T10:26:31.947 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:31.942+0000 7ff92038d740 1 -- 172.21.15.114:0/2991830219 wait complete. 2024-04-23T10:26:31.947 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:31.942+0000 7ff92038d740 20 client.15270 trim_cache size 0 max 16384 2024-04-23T10:26:31.948 INFO:teuthology.orchestra.run.smithi114.stdout:{ 2024-04-23T10:26:31.948 INFO:teuthology.orchestra.run.smithi114.stdout: "cluster_fsid": "aa240572-0159-11ef-bc93-c7b262605968", 2024-04-23T10:26:31.948 INFO:teuthology.orchestra.run.smithi114.stdout: "whoami": 0, 2024-04-23T10:26:31.948 INFO:teuthology.orchestra.run.smithi114.stdout: "id": 24545, 2024-04-23T10:26:31.948 INFO:teuthology.orchestra.run.smithi114.stdout: "want_state": "up:active", 2024-04-23T10:26:31.948 INFO:teuthology.orchestra.run.smithi114.stdout: "state": "up:active", 2024-04-23T10:26:31.948 INFO:teuthology.orchestra.run.smithi114.stdout: "fs_name": "cephfs", 2024-04-23T10:26:31.948 INFO:teuthology.orchestra.run.smithi114.stdout: "rank_uptime": 107.62385781899999, 2024-04-23T10:26:31.948 INFO:teuthology.orchestra.run.smithi114.stdout: "mdsmap_epoch": 20, 2024-04-23T10:26:31.948 INFO:teuthology.orchestra.run.smithi114.stdout: "osdmap_epoch": 89, 2024-04-23T10:26:31.948 INFO:teuthology.orchestra.run.smithi114.stdout: "osdmap_epoch_barrier": 87, 2024-04-23T10:26:31.948 INFO:teuthology.orchestra.run.smithi114.stdout: "uptime": 113.529798423 2024-04-23T10:26:31.948 INFO:teuthology.orchestra.run.smithi114.stdout:} 2024-04-23T10:26:31.949 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:31.942+0000 7ff91d679640 1 -- 172.21.15.114:0/3773376827 >> [v2:172.21.15.114:6800/465333808,v1:172.21.15.114:6801/465333808] conn(0x7ff9000a3bb0 msgr2=0x7ff9000a6070 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T10:26:31.949 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:31.942+0000 7ff91d679640 1 --2- 172.21.15.114:0/3773376827 >> [v2:172.21.15.114:6800/465333808,v1:172.21.15.114:6801/465333808] conn(0x7ff9000a3bb0 0x7ff9000a6070 secure :-1 s=READY pgs=246 cs=0 l=1 rev1=1 crypto rx=0x55bcd71a3060 tx=0x7ff910005f60 comp rx=0 tx=0).stop 2024-04-23T10:26:31.949 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:31.942+0000 7ff91d679640 1 -- 172.21.15.114:0/3773376827 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7ff918105360 msgr2=0x7ff918198720 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T10:26:31.949 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:31.942+0000 7ff91d679640 1 --2- 172.21.15.114:0/3773376827 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7ff918105360 0x7ff918198720 secure :-1 s=READY pgs=395 cs=0 l=1 rev1=1 crypto rx=0x7ff90c006010 tx=0x7ff90c004430 comp rx=0 tx=0).stop 2024-04-23T10:26:31.950 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:31.942+0000 7ff91d679640 1 -- 172.21.15.114:0/3773376827 shutdown_connections 2024-04-23T10:26:31.950 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:31.942+0000 7ff91d679640 1 --2- 172.21.15.114:0/3773376827 >> [v2:172.21.15.114:6800/465333808,v1:172.21.15.114:6801/465333808] conn(0x7ff9000a3bb0 0x7ff9000a6070 unknown :-1 s=CLOSED pgs=246 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:26:31.950 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:31.942+0000 7ff91d679640 1 --2- 172.21.15.114:0/3773376827 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7ff918107c90 0x7ff91819f790 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:26:31.950 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:31.942+0000 7ff91d679640 1 --2- 172.21.15.114:0/3773376827 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7ff918105360 0x7ff918198720 unknown :-1 s=CLOSED pgs=395 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:26:31.950 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:31.942+0000 7ff91d679640 1 --2- 172.21.15.114:0/3773376827 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7ff918102a30 0x7ff9181981e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:26:31.950 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:31.942+0000 7ff91d679640 1 -- 172.21.15.114:0/3773376827 >> 172.21.15.114:0/3773376827 conn(0x7ff9180fc440 msgr2=0x7ff918103430 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T10:26:31.950 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:31.942+0000 7ff91d679640 1 -- 172.21.15.114:0/3773376827 shutdown_connections 2024-04-23T10:26:31.950 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:31.942+0000 7ff91d679640 1 -- 172.21.15.114:0/3773376827 wait complete. 2024-04-23T10:26:31.952 INFO:teuthology.run_tasks:Running task sequential... 2024-04-23T10:26:31.961 INFO:teuthology.task.sequential:In sequential, running task sequential... 2024-04-23T10:26:31.961 INFO:teuthology.task.sequential:In sequential, running task print... 2024-04-23T10:26:31.961 INFO:teuthology.task.print:Enabling mgr modules 2024-04-23T10:26:31.961 INFO:teuthology.task.sequential:In sequential, running task exec... 2024-04-23T10:26:31.962 INFO:teuthology.task.exec:Executing custom commands... 2024-04-23T10:26:31.962 INFO:teuthology.task.exec:Running commands on role mon.a host ubuntu@smithi114.front.sepia.ceph.com 2024-04-23T10:26:31.963 DEBUG:teuthology.orchestra.run.smithi114:> sudo TESTDIR=/home/ubuntu/cephtest bash -c 'ceph mgr module enable snap_schedule' 2024-04-23T10:26:32.063 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:32.062+0000 7f0e01bbf640 1 -- 172.21.15.114:0/66306069 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f0dfc101fb0 msgr2=0x7f0dfc1023d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T10:26:32.063 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:32.062+0000 7f0e01bbf640 1 --2- 172.21.15.114:0/66306069 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f0dfc101fb0 0x7f0dfc1023d0 secure :-1 s=READY pgs=400 cs=0 l=1 rev1=1 crypto rx=0x7f0dec009900 tx=0x7f0dec02a410 comp rx=0 tx=0).stop 2024-04-23T10:26:32.063 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:32.062+0000 7f0e01bbf640 1 -- 172.21.15.114:0/66306069 shutdown_connections 2024-04-23T10:26:32.063 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:32.062+0000 7f0e01bbf640 1 --2- 172.21.15.114:0/66306069 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f0dfc103b60 0x7f0dfc10a3f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:26:32.063 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:32.062+0000 7f0e01bbf640 1 --2- 172.21.15.114:0/66306069 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f0dfc1031a0 0x7f0dfc103620 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:26:32.063 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:32.062+0000 7f0e01bbf640 1 --2- 172.21.15.114:0/66306069 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f0dfc101fb0 0x7f0dfc1023d0 unknown :-1 s=CLOSED pgs=400 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:26:32.063 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:32.062+0000 7f0e01bbf640 1 -- 172.21.15.114:0/66306069 >> 172.21.15.114:0/66306069 conn(0x7f0dfc0fd540 msgr2=0x7f0dfc0ff960 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T10:26:32.063 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:32.062+0000 7f0e01bbf640 1 -- 172.21.15.114:0/66306069 shutdown_connections 2024-04-23T10:26:32.063 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:32.062+0000 7f0e01bbf640 1 -- 172.21.15.114:0/66306069 wait complete. 2024-04-23T10:26:32.064 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:32.062+0000 7f0e01bbf640 1 Processor -- start 2024-04-23T10:26:32.064 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:32.062+0000 7f0e01bbf640 1 -- start start 2024-04-23T10:26:32.064 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:32.062+0000 7f0e01bbf640 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f0dfc101fb0 0x7f0dfc1982b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T10:26:32.064 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:32.063+0000 7f0e01bbf640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f0dfc1031a0 0x7f0dfc1987f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T10:26:32.064 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:32.063+0000 7f0e01bbf640 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f0dfc103b60 0x7f0dfc19f860 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T10:26:32.064 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:32.063+0000 7f0e01bbf640 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x7f0dfc10dbb0 con 0x7f0dfc1031a0 2024-04-23T10:26:32.064 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:32.063+0000 7f0e01bbf640 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 -- 0x7f0dfc10da30 con 0x7f0dfc101fb0 2024-04-23T10:26:32.064 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:32.063+0000 7f0e01bbf640 1 -- --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_getmap magic: 0 -- 0x7f0dfc10dd30 con 0x7f0dfc103b60 2024-04-23T10:26:32.064 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:32.063+0000 7f0dfaffd640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f0dfc1031a0 0x7f0dfc1987f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T10:26:32.064 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:32.063+0000 7f0dfaffd640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f0dfc1031a0 0x7f0dfc1987f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.114:58624/0 (socket says 172.21.15.114:58624) 2024-04-23T10:26:32.064 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:32.063+0000 7f0dfaffd640 1 -- 172.21.15.114:0/1878244572 learned_addr learned my addr 172.21.15.114:0/1878244572 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-04-23T10:26:32.064 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:32.063+0000 7f0dfb7fe640 1 --2- 172.21.15.114:0/1878244572 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f0dfc101fb0 0x7f0dfc1982b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T10:26:32.064 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:32.063+0000 7f0dfaffd640 1 -- 172.21.15.114:0/1878244572 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f0dfc103b60 msgr2=0x7f0dfc19f860 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T10:26:32.064 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:32.063+0000 7f0dfaffd640 1 --2- 172.21.15.114:0/1878244572 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f0dfc103b60 0x7f0dfc19f860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:26:32.064 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:32.063+0000 7f0dfaffd640 1 -- 172.21.15.114:0/1878244572 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f0dfc101fb0 msgr2=0x7f0dfc1982b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T10:26:32.065 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:32.063+0000 7f0dfaffd640 1 --2- 172.21.15.114:0/1878244572 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f0dfc101fb0 0x7f0dfc1982b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:26:32.065 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:32.063+0000 7f0dfaffd640 1 -- 172.21.15.114:0/1878244572 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f0dfc19fed0 con 0x7f0dfc1031a0 2024-04-23T10:26:32.065 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:32.063+0000 7f0dfb7fe640 1 --2- 172.21.15.114:0/1878244572 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f0dfc101fb0 0x7f0dfc1982b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-23T10:26:32.065 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:32.063+0000 7f0dfaffd640 1 --2- 172.21.15.114:0/1878244572 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f0dfc1031a0 0x7f0dfc1987f0 secure :-1 s=READY pgs=401 cs=0 l=1 rev1=1 crypto rx=0x7f0df000b4a0 tx=0x7f0df000b960 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T10:26:32.065 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:32.063+0000 7f0df8ff9640 1 -- 172.21.15.114:0/1878244572 <== mon.0 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0df0014070 con 0x7f0dfc1031a0 2024-04-23T10:26:32.065 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:32.063+0000 7f0df8ff9640 1 -- 172.21.15.114:0/1878244572 <== mon.0 v2:172.21.15.114:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f0df0010ac0 con 0x7f0dfc1031a0 2024-04-23T10:26:32.065 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:32.063+0000 7f0e01bbf640 1 -- 172.21.15.114:0/1878244572 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f0dfc1a01c0 con 0x7f0dfc1031a0 2024-04-23T10:26:32.065 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:32.063+0000 7f0df8ff9640 1 -- 172.21.15.114:0/1878244572 <== mon.0 v2:172.21.15.114:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0df0017c30 con 0x7f0dfc1031a0 2024-04-23T10:26:32.065 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:32.063+0000 7f0e01bbf640 1 -- 172.21.15.114:0/1878244572 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f0dfc1a0790 con 0x7f0dfc1031a0 2024-04-23T10:26:32.065 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:32.064+0000 7f0e01bbf640 1 -- 172.21.15.114:0/1878244572 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f0dfc109320 con 0x7f0dfc1031a0 2024-04-23T10:26:32.066 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:32.065+0000 7f0df8ff9640 1 -- 172.21.15.114:0/1878244572 <== mon.0 v2:172.21.15.114:3300/0 4 ==== mgrmap(e 18) ==== 138647+0+0 (secure 0 0 0) 0x7f0df0017070 con 0x7f0dfc1031a0 2024-04-23T10:26:32.066 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:32.066+0000 7f0df8ff9640 1 --2- 172.21.15.114:0/1878244572 >> [v2:172.21.15.114:6800/465333808,v1:172.21.15.114:6801/465333808] conn(0x7f0de40a3c50 0x7f0de40a6110 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T10:26:32.066 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:32.066+0000 7f0dfb7fe640 1 --2- 172.21.15.114:0/1878244572 >> [v2:172.21.15.114:6800/465333808,v1:172.21.15.114:6801/465333808] conn(0x7f0de40a3c50 0x7f0de40a6110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T10:26:32.066 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:32.066+0000 7f0df8ff9640 1 -- 172.21.15.114:0/1878244572 <== mon.0 v2:172.21.15.114:3300/0 5 ==== osd_map(89..89 src has 1..89) ==== 8269+0+0 (secure 0 0 0) 0x7f0df00cfc00 con 0x7f0dfc1031a0 2024-04-23T10:26:32.066 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:32.066+0000 7f0dfb7fe640 1 --2- 172.21.15.114:0/1878244572 >> [v2:172.21.15.114:6800/465333808,v1:172.21.15.114:6801/465333808] conn(0x7f0de40a3c50 0x7f0de40a6110 secure :-1 s=READY pgs=247 cs=0 l=1 rev1=1 crypto rx=0x7f0dec000c00 tx=0x7f0dec0314a0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T10:26:32.068 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:32.070+0000 7f0df8ff9640 1 -- 172.21.15.114:0/1878244572 <== mon.0 v2:172.21.15.114:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f0df009c5c0 con 0x7f0dfc1031a0 2024-04-23T10:26:32.295 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:32.296+0000 7f0e01bbf640 1 -- 172.21.15.114:0/1878244572 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_command({"prefix": "mgr module enable", "module": "snap_schedule"} v 0) -- 0x7f0dfc02d810 con 0x7f0dfc1031a0 2024-04-23T10:26:32.492 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 23 10:26:32 smithi119 ceph-mon[32608]: from='client.? 172.21.15.114:0/4045900068' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-23T10:26:32.492 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 23 10:26:32 smithi119 ceph-mon[32608]: from='client.? 172.21.15.114:0/3605219822' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-23T10:26:32.504 INFO:journalctl@ceph.mon.a.smithi114.stdout:Apr 23 10:26:32 smithi114 ceph-mon[29255]: from='client.? 172.21.15.114:0/4045900068' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-23T10:26:32.504 INFO:journalctl@ceph.mon.a.smithi114.stdout:Apr 23 10:26:32 smithi114 ceph-mon[29255]: from='client.? 172.21.15.114:0/3605219822' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-23T10:26:32.577 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 23 10:26:32 smithi204 ceph-mon[32649]: from='client.? 172.21.15.114:0/4045900068' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-23T10:26:32.578 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 23 10:26:32 smithi204 ceph-mon[32649]: from='client.? 172.21.15.114:0/3605219822' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-23T10:26:33.119 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:33.120+0000 7f0df8ff9640 1 -- 172.21.15.114:0/1878244572 <== mon.0 v2:172.21.15.114:3300/0 7 ==== mon_command_ack([{"prefix": "mgr module enable", "module": "snap_schedule"}]=0 v19) ==== 92+0+0 (secure 0 0 0) 0x7f0df009bec0 con 0x7f0dfc1031a0 2024-04-23T10:26:33.122 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:33.123+0000 7f0e01bbf640 1 -- 172.21.15.114:0/1878244572 >> [v2:172.21.15.114:6800/465333808,v1:172.21.15.114:6801/465333808] conn(0x7f0de40a3c50 msgr2=0x7f0de40a6110 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T10:26:33.122 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:33.123+0000 7f0e01bbf640 1 --2- 172.21.15.114:0/1878244572 >> [v2:172.21.15.114:6800/465333808,v1:172.21.15.114:6801/465333808] conn(0x7f0de40a3c50 0x7f0de40a6110 secure :-1 s=READY pgs=247 cs=0 l=1 rev1=1 crypto rx=0x7f0dec000c00 tx=0x7f0dec0314a0 comp rx=0 tx=0).stop 2024-04-23T10:26:33.122 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:33.123+0000 7f0e01bbf640 1 -- 172.21.15.114:0/1878244572 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f0dfc1031a0 msgr2=0x7f0dfc1987f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T10:26:33.122 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:33.123+0000 7f0e01bbf640 1 --2- 172.21.15.114:0/1878244572 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f0dfc1031a0 0x7f0dfc1987f0 secure :-1 s=READY pgs=401 cs=0 l=1 rev1=1 crypto rx=0x7f0df000b4a0 tx=0x7f0df000b960 comp rx=0 tx=0).stop 2024-04-23T10:26:33.122 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:33.123+0000 7f0e01bbf640 1 -- 172.21.15.114:0/1878244572 shutdown_connections 2024-04-23T10:26:33.123 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:33.123+0000 7f0e01bbf640 1 --2- 172.21.15.114:0/1878244572 >> [v2:172.21.15.114:6800/465333808,v1:172.21.15.114:6801/465333808] conn(0x7f0de40a3c50 0x7f0de40a6110 unknown :-1 s=CLOSED pgs=247 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:26:33.123 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:33.123+0000 7f0e01bbf640 1 --2- 172.21.15.114:0/1878244572 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f0dfc103b60 0x7f0dfc19f860 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:26:33.123 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:33.123+0000 7f0e01bbf640 1 --2- 172.21.15.114:0/1878244572 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f0dfc1031a0 0x7f0dfc1987f0 unknown :-1 s=CLOSED pgs=401 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:26:33.123 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:33.123+0000 7f0e01bbf640 1 --2- 172.21.15.114:0/1878244572 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f0dfc101fb0 0x7f0dfc1982b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:26:33.123 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:33.123+0000 7f0e01bbf640 1 -- 172.21.15.114:0/1878244572 >> 172.21.15.114:0/1878244572 conn(0x7f0dfc0fd540 msgr2=0x7f0dfc1064f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T10:26:33.123 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:33.123+0000 7f0e01bbf640 1 -- 172.21.15.114:0/1878244572 shutdown_connections 2024-04-23T10:26:33.123 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:33.123+0000 7f0e01bbf640 1 -- 172.21.15.114:0/1878244572 wait complete. 2024-04-23T10:26:33.140 DEBUG:teuthology.orchestra.run.smithi114:> sudo TESTDIR=/home/ubuntu/cephtest bash -c 'ceph config set mgr mgr/snap_schedule/allow_m_granularity true' 2024-04-23T10:26:33.277 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:33.275+0000 7f2adf2a7640 1 -- 172.21.15.114:0/2218426730 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f2ad8102040 msgr2=0x7f2ad81024c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T10:26:33.277 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:33.275+0000 7f2adf2a7640 1 --2- 172.21.15.114:0/2218426730 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f2ad8102040 0x7f2ad81024c0 secure :-1 s=READY pgs=404 cs=0 l=1 rev1=1 crypto rx=0x7f2ac8009900 tx=0x7f2ac802a410 comp rx=0 tx=0).stop 2024-04-23T10:26:33.277 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:33.275+0000 7f2adf2a7640 1 -- 172.21.15.114:0/2218426730 shutdown_connections 2024-04-23T10:26:33.277 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:33.275+0000 7f2adf2a7640 1 --2- 172.21.15.114:0/2218426730 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f2ad8102a00 0x7f2ad8109320 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:26:33.277 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:33.275+0000 7f2adf2a7640 1 --2- 172.21.15.114:0/2218426730 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f2ad8102040 0x7f2ad81024c0 unknown :-1 s=CLOSED pgs=404 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:26:33.277 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:33.275+0000 7f2adf2a7640 1 --2- 172.21.15.114:0/2218426730 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f2ad8100e60 0x7f2ad8101280 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:26:33.277 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:33.275+0000 7f2adf2a7640 1 -- 172.21.15.114:0/2218426730 >> 172.21.15.114:0/2218426730 conn(0x7f2ad80fc420 msgr2=0x7f2ad80fe860 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T10:26:33.277 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:33.276+0000 7f2adf2a7640 1 -- 172.21.15.114:0/2218426730 shutdown_connections 2024-04-23T10:26:33.277 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:33.276+0000 7f2adf2a7640 1 -- 172.21.15.114:0/2218426730 wait complete. 2024-04-23T10:26:33.277 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:33.276+0000 7f2adf2a7640 1 Processor -- start 2024-04-23T10:26:33.277 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:33.276+0000 7f2adf2a7640 1 -- start start 2024-04-23T10:26:33.278 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:33.276+0000 7f2adf2a7640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f2ad8100e60 0x7f2ad8104e70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T10:26:33.278 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:33.276+0000 7f2adf2a7640 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f2ad8102040 0x7f2ad81034b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T10:26:33.278 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:33.276+0000 7f2adf2a7640 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f2ad8102a00 0x7f2ad81039f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T10:26:33.278 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:33.276+0000 7f2adf2a7640 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x7f2ad810c9f0 con 0x7f2ad8100e60 2024-04-23T10:26:33.278 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:33.276+0000 7f2adf2a7640 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 -- 0x7f2ad810c870 con 0x7f2ad8102a00 2024-04-23T10:26:33.278 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:33.276+0000 7f2adf2a7640 1 -- --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_getmap magic: 0 -- 0x7f2ad810cb70 con 0x7f2ad8102040 2024-04-23T10:26:33.278 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:33.276+0000 7f2add043640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f2ad8100e60 0x7f2ad8104e70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T10:26:33.278 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:33.276+0000 7f2add043640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f2ad8100e60 0x7f2ad8104e70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.114:58666/0 (socket says 172.21.15.114:58666) 2024-04-23T10:26:33.278 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:33.276+0000 7f2add043640 1 -- 172.21.15.114:0/3862220856 learned_addr learned my addr 172.21.15.114:0/3862220856 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-04-23T10:26:33.278 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:33.276+0000 7f2adc842640 1 --2- 172.21.15.114:0/3862220856 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f2ad8102040 0x7f2ad81034b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T10:26:33.278 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:33.276+0000 7f2add844640 1 --2- 172.21.15.114:0/3862220856 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f2ad8102a00 0x7f2ad81039f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T10:26:33.278 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:33.277+0000 7f2add043640 1 -- 172.21.15.114:0/3862220856 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f2ad8102040 msgr2=0x7f2ad81034b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T10:26:33.278 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:33.277+0000 7f2add043640 1 --2- 172.21.15.114:0/3862220856 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f2ad8102040 0x7f2ad81034b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:26:33.278 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:33.277+0000 7f2add043640 1 -- 172.21.15.114:0/3862220856 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f2ad8102a00 msgr2=0x7f2ad81039f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T10:26:33.279 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:33.277+0000 7f2add043640 1 --2- 172.21.15.114:0/3862220856 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f2ad8102a00 0x7f2ad81039f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:26:33.279 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:33.277+0000 7f2add043640 1 -- 172.21.15.114:0/3862220856 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f2ad8104280 con 0x7f2ad8100e60 2024-04-23T10:26:33.279 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:33.277+0000 7f2add043640 1 --2- 172.21.15.114:0/3862220856 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f2ad8100e60 0x7f2ad8104e70 secure :-1 s=READY pgs=405 cs=0 l=1 rev1=1 crypto rx=0x7f2ad000e9c0 tx=0x7f2ad000ee80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T10:26:33.279 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:33.277+0000 7f2ace7fc640 1 -- 172.21.15.114:0/3862220856 <== mon.0 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2ad000cde0 con 0x7f2ad8100e60 2024-04-23T10:26:33.279 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:33.277+0000 7f2ace7fc640 1 -- 172.21.15.114:0/3862220856 <== mon.0 v2:172.21.15.114:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f2ad00108e0 con 0x7f2ad8100e60 2024-04-23T10:26:33.279 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:33.277+0000 7f2adf2a7640 1 -- 172.21.15.114:0/3862220856 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f2ad81a2280 con 0x7f2ad8100e60 2024-04-23T10:26:33.279 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:33.277+0000 7f2ace7fc640 1 -- 172.21.15.114:0/3862220856 <== mon.0 v2:172.21.15.114:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2ad0018960 con 0x7f2ad8100e60 2024-04-23T10:26:33.279 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:33.277+0000 7f2adf2a7640 1 -- 172.21.15.114:0/3862220856 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f2ad81a27f0 con 0x7f2ad8100e60 2024-04-23T10:26:33.279 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:33.278+0000 7f2adf2a7640 1 -- 172.21.15.114:0/3862220856 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f2ad81a2ab0 con 0x7f2ad8100e60 2024-04-23T10:26:33.280 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:33.279+0000 7f2ace7fc640 1 -- 172.21.15.114:0/3862220856 <== mon.0 v2:172.21.15.114:3300/0 4 ==== mgrmap(e 19) ==== 138664+0+0 (secure 0 0 0) 0x7f2ad0010a90 con 0x7f2ad8100e60 2024-04-23T10:26:33.280 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:33.280+0000 7f2ace7fc640 1 --2- 172.21.15.114:0/3862220856 >> [v2:172.21.15.114:6800/465333808,v1:172.21.15.114:6801/465333808] conn(0x7f2ac40a3d20 0x7f2ac40a61e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T10:26:33.280 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:33.280+0000 7f2adc842640 1 -- 172.21.15.114:0/3862220856 >> [v2:172.21.15.114:6800/465333808,v1:172.21.15.114:6801/465333808] conn(0x7f2ac40a3d20 msgr2=0x7f2ac40a61e0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.114:6800/465333808 2024-04-23T10:26:33.280 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:33.280+0000 7f2ace7fc640 1 -- 172.21.15.114:0/3862220856 <== mon.0 v2:172.21.15.114:3300/0 5 ==== osd_map(89..89 src has 1..89) ==== 8269+0+0 (secure 0 0 0) 0x7f2ad00d08a0 con 0x7f2ad8100e60 2024-04-23T10:26:33.280 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:33.280+0000 7f2adc842640 1 --2- 172.21.15.114:0/3862220856 >> [v2:172.21.15.114:6800/465333808,v1:172.21.15.114:6801/465333808] conn(0x7f2ac40a3d20 0x7f2ac40a61e0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-04-23T10:26:33.283 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:33.285+0000 7f2ace7fc640 1 -- 172.21.15.114:0/3862220856 <== mon.0 v2:172.21.15.114:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f2ad0053ff0 con 0x7f2ad8100e60 2024-04-23T10:26:33.468 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:33.469+0000 7f2adf2a7640 1 -- 172.21.15.114:0/3862220856 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_command([{prefix=config set, name=mgr/snap_schedule/allow_m_granularity}] v 0) -- 0x7f2ad81a3160 con 0x7f2ad8100e60 2024-04-23T10:26:33.472 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:33.473+0000 7f2ace7fc640 1 -- 172.21.15.114:0/3862220856 <== mon.0 v2:172.21.15.114:3300/0 7 ==== mon_command_ack([{prefix=config set, name=mgr/snap_schedule/allow_m_granularity}]=0 v30)=0 v30) ==== 138+0+0 (secure 0 0 0) 0x7f2ad009d260 con 0x7f2ad8100e60 2024-04-23T10:26:33.474 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:33.475+0000 7f2adf2a7640 1 -- 172.21.15.114:0/3862220856 >> [v2:172.21.15.114:6800/465333808,v1:172.21.15.114:6801/465333808] conn(0x7f2ac40a3d20 msgr2=0x7f2ac40a61e0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-04-23T10:26:33.475 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:33.475+0000 7f2adf2a7640 1 --2- 172.21.15.114:0/3862220856 >> [v2:172.21.15.114:6800/465333808,v1:172.21.15.114:6801/465333808] conn(0x7f2ac40a3d20 0x7f2ac40a61e0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:26:33.475 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:33.475+0000 7f2adf2a7640 1 -- 172.21.15.114:0/3862220856 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f2ad8100e60 msgr2=0x7f2ad8104e70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T10:26:33.475 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:33.475+0000 7f2adf2a7640 1 --2- 172.21.15.114:0/3862220856 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f2ad8100e60 0x7f2ad8104e70 secure :-1 s=READY pgs=405 cs=0 l=1 rev1=1 crypto rx=0x7f2ad000e9c0 tx=0x7f2ad000ee80 comp rx=0 tx=0).stop 2024-04-23T10:26:33.475 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:33.475+0000 7f2adf2a7640 1 -- 172.21.15.114:0/3862220856 shutdown_connections 2024-04-23T10:26:33.475 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:33.475+0000 7f2adf2a7640 1 --2- 172.21.15.114:0/3862220856 >> [v2:172.21.15.114:6800/465333808,v1:172.21.15.114:6801/465333808] conn(0x7f2ac40a3d20 0x7f2ac40a61e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:26:33.475 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:33.475+0000 7f2adf2a7640 1 --2- 172.21.15.114:0/3862220856 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f2ad8102a00 0x7f2ad81039f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:26:33.475 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:33.475+0000 7f2adf2a7640 1 --2- 172.21.15.114:0/3862220856 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f2ad8102040 0x7f2ad81034b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:26:33.475 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:33.475+0000 7f2adf2a7640 1 --2- 172.21.15.114:0/3862220856 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f2ad8100e60 0x7f2ad8104e70 unknown :-1 s=CLOSED pgs=405 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:26:33.475 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:33.475+0000 7f2adf2a7640 1 -- 172.21.15.114:0/3862220856 >> 172.21.15.114:0/3862220856 conn(0x7f2ad80fc420 msgr2=0x7f2ad80fded0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T10:26:33.475 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:33.475+0000 7f2adf2a7640 1 -- 172.21.15.114:0/3862220856 shutdown_connections 2024-04-23T10:26:33.475 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:33.475+0000 7f2adf2a7640 1 -- 172.21.15.114:0/3862220856 wait complete. 2024-04-23T10:26:33.485 INFO:journalctl@ceph.mon.a.smithi114.stdout:Apr 23 10:26:33 smithi114 ceph-mon[29255]: pgmap v342: 97 pgs: 97 active+clean; 641 KiB data, 375 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-23T10:26:33.485 INFO:journalctl@ceph.mon.a.smithi114.stdout:Apr 23 10:26:33 smithi114 ceph-mon[29255]: from='client.? 172.21.15.114:0/1878244572' entity='client.admin' cmd=[{"prefix": "mgr module enable", "module": "snap_schedule"}]: dispatch 2024-04-23T10:26:33.485 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Apr 23 10:26:33 smithi114 ceph-aa240572-0159-11ef-bc93-c7b262605968-mgr-x[29490]: ignoring --setuser ceph since I am not root 2024-04-23T10:26:33.485 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Apr 23 10:26:33 smithi114 ceph-aa240572-0159-11ef-bc93-c7b262605968-mgr-x[29490]: ignoring --setgroup ceph since I am not root 2024-04-23T10:26:33.485 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Apr 23 10:26:33 smithi114 ceph-aa240572-0159-11ef-bc93-c7b262605968-mgr-x[29490]: 2024-04-23T10:26:33.335+0000 7f21d5288200 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2024-04-23T10:26:33.486 DEBUG:teuthology.orchestra.run.smithi114:> sudo TESTDIR=/home/ubuntu/cephtest bash -c 'ceph config set mgr mgr/snap_schedule/dump_on_update true' 2024-04-23T10:26:33.487 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 23 10:26:33 smithi204 ceph-mon[32649]: pgmap v342: 97 pgs: 97 active+clean; 641 KiB data, 375 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-23T10:26:33.487 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 23 10:26:33 smithi204 ceph-mon[32649]: from='client.? 172.21.15.114:0/1878244572' entity='client.admin' cmd=[{"prefix": "mgr module enable", "module": "snap_schedule"}]: dispatch 2024-04-23T10:26:33.487 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 23 10:26:33 smithi119 ceph-mon[32608]: pgmap v342: 97 pgs: 97 active+clean; 641 KiB data, 375 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-23T10:26:33.487 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 23 10:26:33 smithi119 ceph-mon[32608]: from='client.? 172.21.15.114:0/1878244572' entity='client.admin' cmd=[{"prefix": "mgr module enable", "module": "snap_schedule"}]: dispatch 2024-04-23T10:26:33.488 INFO:journalctl@ceph.mgr.z.smithi204.stdout:Apr 23 10:26:33 smithi204 ceph-aa240572-0159-11ef-bc93-c7b262605968-mgr-z[34493]: ignoring --setuser ceph since I am not root 2024-04-23T10:26:33.488 INFO:journalctl@ceph.mgr.z.smithi204.stdout:Apr 23 10:26:33 smithi204 ceph-aa240572-0159-11ef-bc93-c7b262605968-mgr-z[34493]: ignoring --setgroup ceph since I am not root 2024-04-23T10:26:33.488 INFO:journalctl@ceph.mgr.z.smithi204.stdout:Apr 23 10:26:33 smithi204 ceph-mgr[34497]: -- 172.21.15.204:0/146798356 <== mon.1 v2:172.21.15.204:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 194+0+0 (secure 0 0 0) 0x5654c4daa5a0 con 0x5654c4d7ec00 2024-04-23T10:26:33.488 INFO:journalctl@ceph.mgr.z.smithi204.stdout:Apr 23 10:26:33 smithi204 ceph-aa240572-0159-11ef-bc93-c7b262605968-mgr-z[34493]: 2024-04-23T10:26:33.303+0000 7f39c4768200 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2024-04-23T10:26:33.488 INFO:journalctl@ceph.mgr.z.smithi204.stdout:Apr 23 10:26:33 smithi204 ceph-aa240572-0159-11ef-bc93-c7b262605968-mgr-z[34493]: 2024-04-23T10:26:33.488+0000 7f39c4768200 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2024-04-23T10:26:33.488 INFO:journalctl@ceph.mgr.y.smithi119.stdout:Apr 23 10:26:33 smithi119 ceph-aa240572-0159-11ef-bc93-c7b262605968-mgr-y[34019]: ignoring --setuser ceph since I am not root 2024-04-23T10:26:33.488 INFO:journalctl@ceph.mgr.y.smithi119.stdout:Apr 23 10:26:33 smithi119 ceph-aa240572-0159-11ef-bc93-c7b262605968-mgr-y[34019]: ignoring --setgroup ceph since I am not root 2024-04-23T10:26:33.488 INFO:journalctl@ceph.mgr.y.smithi119.stdout:Apr 23 10:26:33 smithi119 ceph-mgr[34023]: -- 172.21.15.119:0/1047845048 <== mon.1 v2:172.21.15.204:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 194+0+0 (secure 0 0 0) 0x55c34eb865a0 con 0x55c34eb5ac00 2024-04-23T10:26:33.488 INFO:journalctl@ceph.mgr.y.smithi119.stdout:Apr 23 10:26:33 smithi119 ceph-aa240572-0159-11ef-bc93-c7b262605968-mgr-y[34019]: 2024-04-23T10:26:33.302+0000 7f0f8532c200 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2024-04-23T10:26:33.488 INFO:journalctl@ceph.mgr.y.smithi119.stdout:Apr 23 10:26:33 smithi119 ceph-aa240572-0159-11ef-bc93-c7b262605968-mgr-y[34019]: 2024-04-23T10:26:33.487+0000 7f0f8532c200 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2024-04-23T10:26:33.592 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:33.590+0000 7f393c507640 1 -- 172.21.15.114:0/3598798675 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f3934102a00 msgr2=0x7f3934109320 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T10:26:33.592 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:33.590+0000 7f393c507640 1 --2- 172.21.15.114:0/3598798675 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f3934102a00 0x7f3934109320 secure :-1 s=READY pgs=406 cs=0 l=1 rev1=1 crypto rx=0x7f392c009ca0 tx=0x7f392c02a8c0 comp rx=0 tx=0).stop 2024-04-23T10:26:33.592 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:33.591+0000 7f393c507640 1 -- 172.21.15.114:0/3598798675 shutdown_connections 2024-04-23T10:26:33.592 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:33.591+0000 7f393c507640 1 --2- 172.21.15.114:0/3598798675 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f3934102a00 0x7f3934109320 unknown :-1 s=CLOSED pgs=406 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:26:33.592 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:33.591+0000 7f393c507640 1 --2- 172.21.15.114:0/3598798675 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f3934102040 0x7f39341024c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:26:33.592 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:33.591+0000 7f393c507640 1 --2- 172.21.15.114:0/3598798675 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f3934100e60 0x7f3934101280 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:26:33.592 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:33.591+0000 7f393c507640 1 -- 172.21.15.114:0/3598798675 >> 172.21.15.114:0/3598798675 conn(0x7f39340fc420 msgr2=0x7f39340fe860 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T10:26:33.593 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:33.591+0000 7f393c507640 1 -- 172.21.15.114:0/3598798675 shutdown_connections 2024-04-23T10:26:33.593 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:33.591+0000 7f393c507640 1 -- 172.21.15.114:0/3598798675 wait complete. 2024-04-23T10:26:33.593 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:33.591+0000 7f393c507640 1 Processor -- start 2024-04-23T10:26:33.593 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:33.591+0000 7f393c507640 1 -- start start 2024-04-23T10:26:33.593 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:33.591+0000 7f393c507640 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f3934100e60 0x7f3934196fe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T10:26:33.593 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:33.591+0000 7f393c507640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f3934102040 0x7f3934197520 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T10:26:33.593 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:33.591+0000 7f393c507640 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f3934102a00 0x7f393419e590 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T10:26:33.593 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:33.591+0000 7f393c507640 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x7f393410c9f0 con 0x7f3934102040 2024-04-23T10:26:33.593 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:33.591+0000 7f393c507640 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 -- 0x7f393410c870 con 0x7f3934102a00 2024-04-23T10:26:33.593 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:33.591+0000 7f393c507640 1 -- --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_getmap magic: 0 -- 0x7f393410cb70 con 0x7f3934100e60 2024-04-23T10:26:33.593 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:33.591+0000 7f3939aa2640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f3934102040 0x7f3934197520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T10:26:33.593 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:33.592+0000 7f3939aa2640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f3934102040 0x7f3934197520 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.114:58686/0 (socket says 172.21.15.114:58686) 2024-04-23T10:26:33.593 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:33.592+0000 7f3939aa2640 1 -- 172.21.15.114:0/2681667025 learned_addr learned my addr 172.21.15.114:0/2681667025 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-04-23T10:26:33.593 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:33.592+0000 7f393a2a3640 1 --2- 172.21.15.114:0/2681667025 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f3934100e60 0x7f3934196fe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T10:26:33.594 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:33.592+0000 7f393aaa4640 1 --2- 172.21.15.114:0/2681667025 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f3934102a00 0x7f393419e590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T10:26:33.594 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:33.592+0000 7f3939aa2640 1 -- 172.21.15.114:0/2681667025 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f3934100e60 msgr2=0x7f3934196fe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T10:26:33.594 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:33.592+0000 7f3939aa2640 1 --2- 172.21.15.114:0/2681667025 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f3934100e60 0x7f3934196fe0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:26:33.594 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:33.592+0000 7f3939aa2640 1 -- 172.21.15.114:0/2681667025 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f3934102a00 msgr2=0x7f393419e590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T10:26:33.594 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:33.592+0000 7f3939aa2640 1 --2- 172.21.15.114:0/2681667025 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f3934102a00 0x7f393419e590 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:26:33.594 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:33.592+0000 7f3939aa2640 1 -- 172.21.15.114:0/2681667025 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f393419ec00 con 0x7f3934102040 2024-04-23T10:26:33.594 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:33.592+0000 7f3939aa2640 1 --2- 172.21.15.114:0/2681667025 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f3934102040 0x7f3934197520 secure :-1 s=READY pgs=407 cs=0 l=1 rev1=1 crypto rx=0x7f393000d950 tx=0x7f393000de10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T10:26:33.594 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:33.592+0000 7f392b7fe640 1 -- 172.21.15.114:0/2681667025 <== mon.0 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3930017030 con 0x7f3934102040 2024-04-23T10:26:33.594 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:33.592+0000 7f393c507640 1 -- 172.21.15.114:0/2681667025 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f393419eef0 con 0x7f3934102040 2024-04-23T10:26:33.594 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:33.592+0000 7f392b7fe640 1 -- 172.21.15.114:0/2681667025 <== mon.0 v2:172.21.15.114:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f3930004020 con 0x7f3934102040 2024-04-23T10:26:33.594 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:33.592+0000 7f392b7fe640 1 -- 172.21.15.114:0/2681667025 <== mon.0 v2:172.21.15.114:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f39300047a0 con 0x7f3934102040 2024-04-23T10:26:33.594 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:33.592+0000 7f393c507640 1 -- 172.21.15.114:0/2681667025 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f393419f3b0 con 0x7f3934102040 2024-04-23T10:26:33.594 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:33.593+0000 7f393c507640 1 -- 172.21.15.114:0/2681667025 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f3934108200 con 0x7f3934102040 2024-04-23T10:26:33.595 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:33.595+0000 7f392b7fe640 1 -- 172.21.15.114:0/2681667025 <== mon.0 v2:172.21.15.114:3300/0 4 ==== mgrmap(e 19) ==== 138664+0+0 (secure 0 0 0) 0x7f3930009a30 con 0x7f3934102040 2024-04-23T10:26:33.595 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:33.596+0000 7f392b7fe640 1 --2- 172.21.15.114:0/2681667025 >> [v2:172.21.15.114:6800/465333808,v1:172.21.15.114:6801/465333808] conn(0x7f39200a3ca0 0x7f39200a6160 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T10:26:33.595 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:33.596+0000 7f393a2a3640 1 -- 172.21.15.114:0/2681667025 >> [v2:172.21.15.114:6800/465333808,v1:172.21.15.114:6801/465333808] conn(0x7f39200a3ca0 msgr2=0x7f39200a6160 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.114:6800/465333808 2024-04-23T10:26:33.595 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:33.596+0000 7f392b7fe640 1 -- 172.21.15.114:0/2681667025 <== mon.0 v2:172.21.15.114:3300/0 5 ==== osd_map(89..89 src has 1..89) ==== 8269+0+0 (secure 0 0 0) 0x7f39300cfdd0 con 0x7f3934102040 2024-04-23T10:26:33.595 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:33.596+0000 7f393a2a3640 1 --2- 172.21.15.114:0/2681667025 >> [v2:172.21.15.114:6800/465333808,v1:172.21.15.114:6801/465333808] conn(0x7f39200a3ca0 0x7f39200a6160 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-04-23T10:26:33.599 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:33.600+0000 7f392b7fe640 1 -- 172.21.15.114:0/2681667025 <== mon.0 v2:172.21.15.114:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f393009c790 con 0x7f3934102040 2024-04-23T10:26:33.742 INFO:journalctl@ceph.mgr.y.smithi119.stdout:Apr 23 10:26:33 smithi119 ceph-aa240572-0159-11ef-bc93-c7b262605968-mgr-y[34019]: 2024-04-23T10:26:33.589+0000 7f0f8532c200 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2024-04-23T10:26:33.754 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Apr 23 10:26:33 smithi114 ceph-aa240572-0159-11ef-bc93-c7b262605968-mgr-x[29490]: 2024-04-23T10:26:33.525+0000 7f21d5288200 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2024-04-23T10:26:33.754 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Apr 23 10:26:33 smithi114 ceph-aa240572-0159-11ef-bc93-c7b262605968-mgr-x[29490]: 2024-04-23T10:26:33.635+0000 7f21d5288200 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2024-04-23T10:26:33.785 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:33.787+0000 7f393c507640 1 -- 172.21.15.114:0/2681667025 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_command([{prefix=config set, name=mgr/snap_schedule/dump_on_update}] v 0) -- 0x7f3934062c60 con 0x7f3934102040 2024-04-23T10:26:33.789 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:33.790+0000 7f392b7fe640 1 -- 172.21.15.114:0/2681667025 <== mon.0 v2:172.21.15.114:3300/0 7 ==== mon_command_ack([{prefix=config set, name=mgr/snap_schedule/dump_on_update}]=0 v31)=0 v31) ==== 133+0+0 (secure 0 0 0) 0x7f393009c090 con 0x7f3934102040 2024-04-23T10:26:33.792 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:33.793+0000 7f393c507640 1 -- 172.21.15.114:0/2681667025 >> [v2:172.21.15.114:6800/465333808,v1:172.21.15.114:6801/465333808] conn(0x7f39200a3ca0 msgr2=0x7f39200a6160 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-04-23T10:26:33.792 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:33.793+0000 7f393c507640 1 --2- 172.21.15.114:0/2681667025 >> [v2:172.21.15.114:6800/465333808,v1:172.21.15.114:6801/465333808] conn(0x7f39200a3ca0 0x7f39200a6160 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:26:33.792 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:33.793+0000 7f393c507640 1 -- 172.21.15.114:0/2681667025 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f3934102040 msgr2=0x7f3934197520 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T10:26:33.792 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:33.793+0000 7f393c507640 1 --2- 172.21.15.114:0/2681667025 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f3934102040 0x7f3934197520 secure :-1 s=READY pgs=407 cs=0 l=1 rev1=1 crypto rx=0x7f393000d950 tx=0x7f393000de10 comp rx=0 tx=0).stop 2024-04-23T10:26:33.792 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:33.793+0000 7f393c507640 1 -- 172.21.15.114:0/2681667025 shutdown_connections 2024-04-23T10:26:33.792 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:33.793+0000 7f393c507640 1 --2- 172.21.15.114:0/2681667025 >> [v2:172.21.15.114:6800/465333808,v1:172.21.15.114:6801/465333808] conn(0x7f39200a3ca0 0x7f39200a6160 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:26:33.792 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:33.793+0000 7f393c507640 1 --2- 172.21.15.114:0/2681667025 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f3934102a00 0x7f393419e590 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:26:33.792 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:33.793+0000 7f393c507640 1 --2- 172.21.15.114:0/2681667025 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f3934102040 0x7f3934197520 unknown :-1 s=CLOSED pgs=407 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:26:33.792 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:33.793+0000 7f393c507640 1 --2- 172.21.15.114:0/2681667025 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f3934100e60 0x7f3934196fe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:26:33.793 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:33.793+0000 7f393c507640 1 -- 172.21.15.114:0/2681667025 >> 172.21.15.114:0/2681667025 conn(0x7f39340fc420 msgr2=0x7f39341053d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T10:26:33.793 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:33.793+0000 7f393c507640 1 -- 172.21.15.114:0/2681667025 shutdown_connections 2024-04-23T10:26:33.793 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:33.793+0000 7f393c507640 1 -- 172.21.15.114:0/2681667025 wait complete. 2024-04-23T10:26:33.803 INFO:teuthology.run_tasks:Running task kclient... 2024-04-23T10:26:33.826 INFO:tasks.kclient:Mounting kernel clients... 2024-04-23T10:26:33.826 INFO:tasks.kclient:config is {'client.0': {}} 2024-04-23T10:26:33.826 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-23T10:26:33.826 DEBUG:teuthology.orchestra.run.smithi114:> ip netns list 2024-04-23T10:26:33.828 INFO:journalctl@ceph.mgr.z.smithi204.stdout:Apr 23 10:26:33 smithi204 ceph-aa240572-0159-11ef-bc93-c7b262605968-mgr-z[34493]: 2024-04-23T10:26:33.590+0000 7f39c4768200 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2024-04-23T10:26:33.842 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-23T10:26:33.843 DEBUG:teuthology.orchestra.run.smithi114:> sudo ip link delete ceph-brx 2024-04-23T10:26:33.908 INFO:teuthology.orchestra.run.smithi114.stderr:Cannot find device "ceph-brx" 2024-04-23T10:26:33.909 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-23T10:26:33.910 INFO:tasks.kclient:client.0 config is {'mntopts': ['ms_mode=crc', 'nowsync'], 'syntax': 'v1', 'mount_subvol_num': 1} 2024-04-23T10:26:33.910 INFO:tasks.cephfs.mount:cephfs_mntpt = None 2024-04-23T10:26:33.910 INFO:tasks.cephfs.mount:hostfs_mntpt = /home/ubuntu/cephtest/mnt.0 2024-04-23T10:26:33.910 DEBUG:teuthology.orchestra.run.smithi114:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d69f13d827952bb07bb9b7973bca14f97d764d6f shell --fsid aa240572-0159-11ef-bc93-c7b262605968 -- ceph fs ls 2024-04-23T10:26:34.120 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Apr 23 10:26:33 smithi114 ceph-aa240572-0159-11ef-bc93-c7b262605968-mgr-x[29490]: 2024-04-23T10:26:33.956+0000 7f21d5288200 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2024-04-23T10:26:34.120 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Apr 23 10:26:34 smithi114 ceph-aa240572-0159-11ef-bc93-c7b262605968-mgr-x[29490]: 2024-04-23T10:26:34.044+0000 7f21d5288200 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2024-04-23T10:26:34.199 INFO:teuthology.orchestra.run.smithi114.stderr:Inferring config /var/lib/ceph/aa240572-0159-11ef-bc93-c7b262605968/mon.a/config 2024-04-23T10:26:34.241 INFO:journalctl@ceph.mgr.y.smithi119.stdout:Apr 23 10:26:33 smithi119 ceph-aa240572-0159-11ef-bc93-c7b262605968-mgr-y[34019]: 2024-04-23T10:26:33.896+0000 7f0f8532c200 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2024-04-23T10:26:34.242 INFO:journalctl@ceph.mgr.y.smithi119.stdout:Apr 23 10:26:33 smithi119 ceph-aa240572-0159-11ef-bc93-c7b262605968-mgr-y[34019]: 2024-04-23T10:26:33.980+0000 7f0f8532c200 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2024-04-23T10:26:34.242 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 23 10:26:34 smithi119 ceph-mon[32608]: from='client.? 172.21.15.114:0/1878244572' entity='client.admin' cmd='[{"prefix": "mgr module enable", "module": "snap_schedule"}]': finished 2024-04-23T10:26:34.242 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 23 10:26:34 smithi119 ceph-mon[32608]: mgrmap e19: x(active, since 8m), standbys: z, y 2024-04-23T10:26:34.242 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 23 10:26:34 smithi119 ceph-mon[32608]: from='client.? 172.21.15.114:0/3862220856' entity='client.admin' 2024-04-23T10:26:34.242 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 23 10:26:34 smithi119 ceph-mon[32608]: from='client.? 172.21.15.114:0/2681667025' entity='client.admin' 2024-04-23T10:26:34.327 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 23 10:26:34 smithi204 ceph-mon[32649]: from='client.? 172.21.15.114:0/1878244572' entity='client.admin' cmd='[{"prefix": "mgr module enable", "module": "snap_schedule"}]': finished 2024-04-23T10:26:34.328 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 23 10:26:34 smithi204 ceph-mon[32649]: mgrmap e19: x(active, since 8m), standbys: z, y 2024-04-23T10:26:34.328 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 23 10:26:34 smithi204 ceph-mon[32649]: from='client.? 172.21.15.114:0/3862220856' entity='client.admin' 2024-04-23T10:26:34.328 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 23 10:26:34 smithi204 ceph-mon[32649]: from='client.? 172.21.15.114:0/2681667025' entity='client.admin' 2024-04-23T10:26:34.328 INFO:journalctl@ceph.mgr.z.smithi204.stdout:Apr 23 10:26:33 smithi204 ceph-aa240572-0159-11ef-bc93-c7b262605968-mgr-z[34493]: 2024-04-23T10:26:33.898+0000 7f39c4768200 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2024-04-23T10:26:34.328 INFO:journalctl@ceph.mgr.z.smithi204.stdout:Apr 23 10:26:33 smithi204 ceph-aa240572-0159-11ef-bc93-c7b262605968-mgr-z[34493]: 2024-04-23T10:26:33.983+0000 7f39c4768200 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2024-04-23T10:26:34.492 INFO:journalctl@ceph.mon.a.smithi114.stdout:Apr 23 10:26:34 smithi114 ceph-mon[29255]: from='client.? 172.21.15.114:0/1878244572' entity='client.admin' cmd='[{"prefix": "mgr module enable", "module": "snap_schedule"}]': finished 2024-04-23T10:26:34.492 INFO:journalctl@ceph.mon.a.smithi114.stdout:Apr 23 10:26:34 smithi114 ceph-mon[29255]: mgrmap e19: x(active, since 8m), standbys: z, y 2024-04-23T10:26:34.492 INFO:journalctl@ceph.mon.a.smithi114.stdout:Apr 23 10:26:34 smithi114 ceph-mon[29255]: from='client.? 172.21.15.114:0/3862220856' entity='client.admin' 2024-04-23T10:26:34.492 INFO:journalctl@ceph.mon.a.smithi114.stdout:Apr 23 10:26:34 smithi114 ceph-mon[29255]: from='client.? 172.21.15.114:0/2681667025' entity='client.admin' 2024-04-23T10:26:35.492 INFO:journalctl@ceph.mgr.y.smithi119.stdout:Apr 23 10:26:35 smithi119 ceph-aa240572-0159-11ef-bc93-c7b262605968-mgr-y[34019]: 2024-04-23T10:26:35.154+0000 7f0f8532c200 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2024-04-23T10:26:35.492 INFO:journalctl@ceph.mgr.y.smithi119.stdout:Apr 23 10:26:35 smithi119 ceph-aa240572-0159-11ef-bc93-c7b262605968-mgr-y[34019]: 2024-04-23T10:26:35.325+0000 7f0f8532c200 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2024-04-23T10:26:35.506 INFO:journalctl@ceph.mgr.z.smithi204.stdout:Apr 23 10:26:35 smithi204 ceph-aa240572-0159-11ef-bc93-c7b262605968-mgr-z[34493]: 2024-04-23T10:26:35.164+0000 7f39c4768200 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2024-04-23T10:26:35.506 INFO:journalctl@ceph.mgr.z.smithi204.stdout:Apr 23 10:26:35 smithi204 ceph-aa240572-0159-11ef-bc93-c7b262605968-mgr-z[34493]: 2024-04-23T10:26:35.336+0000 7f39c4768200 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2024-04-23T10:26:35.506 INFO:journalctl@ceph.mgr.z.smithi204.stdout:Apr 23 10:26:35 smithi204 ceph-aa240572-0159-11ef-bc93-c7b262605968-mgr-z[34493]: 2024-04-23T10:26:35.507+0000 7f39c4768200 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2024-04-23T10:26:35.624 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Apr 23 10:26:35 smithi114 ceph-aa240572-0159-11ef-bc93-c7b262605968-mgr-x[29490]: 2024-04-23T10:26:35.269+0000 7f21d5288200 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2024-04-23T10:26:35.624 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Apr 23 10:26:35 smithi114 ceph-aa240572-0159-11ef-bc93-c7b262605968-mgr-x[29490]: 2024-04-23T10:26:35.443+0000 7f21d5288200 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2024-04-23T10:26:35.625 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Apr 23 10:26:35 smithi114 ceph-aa240572-0159-11ef-bc93-c7b262605968-mgr-x[29490]: 2024-04-23T10:26:35.624+0000 7f21d5288200 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2024-04-23T10:26:35.784 INFO:journalctl@ceph.mgr.y.smithi119.stdout:Apr 23 10:26:35 smithi119 ceph-aa240572-0159-11ef-bc93-c7b262605968-mgr-y[34019]: 2024-04-23T10:26:35.496+0000 7f0f8532c200 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2024-04-23T10:26:36.013 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:36.013+0000 7f91130b0700 1 -- 172.21.15.114:0/1363893045 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f910c1026e0 msgr2=0x7f910c10e770 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T10:26:36.014 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:36.013+0000 7f91130b0700 1 --2- 172.21.15.114:0/1363893045 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f910c1026e0 0x7f910c10e770 secure :-1 s=READY pgs=408 cs=0 l=1 rev1=1 crypto rx=0x7f910400b0a0 tx=0x7f910400b3b0 comp rx=0 tx=0).stop 2024-04-23T10:26:36.014 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:36.013+0000 7f91130b0700 1 -- 172.21.15.114:0/1363893045 shutdown_connections 2024-04-23T10:26:36.014 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:36.013+0000 7f91130b0700 1 --2- 172.21.15.114:0/1363893045 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f910c106990 0x7f910c106d70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:26:36.014 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:36.013+0000 7f91130b0700 1 --2- 172.21.15.114:0/1363893045 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f910c1026e0 0x7f910c10e770 unknown :-1 s=CLOSED pgs=408 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:26:36.014 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:36.013+0000 7f91130b0700 1 --2- 172.21.15.114:0/1363893045 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f910c101d20 0x7f910c1021a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:26:36.014 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:36.013+0000 7f91130b0700 1 -- 172.21.15.114:0/1363893045 >> 172.21.15.114:0/1363893045 conn(0x7f910c0788f0 msgr2=0x7f910c078d00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T10:26:36.014 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:36.013+0000 7f91130b0700 1 -- 172.21.15.114:0/1363893045 shutdown_connections 2024-04-23T10:26:36.014 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:36.013+0000 7f91130b0700 1 -- 172.21.15.114:0/1363893045 wait complete. 2024-04-23T10:26:36.014 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:36.013+0000 7f91130b0700 1 Processor -- start 2024-04-23T10:26:36.014 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:36.014+0000 7f91130b0700 1 -- start start 2024-04-23T10:26:36.014 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:36.014+0000 7f91130b0700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f910c101d20 0x7f910c19b2b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T10:26:36.014 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:36.014+0000 7f91130b0700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f910c1026e0 0x7f910c19b7f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T10:26:36.015 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:36.014+0000 7f91130b0700 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f910c106990 0x7f910c19fb80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T10:26:36.015 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:36.014+0000 7f91130b0700 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x7f910c111ae0 con 0x7f910c101d20 2024-04-23T10:26:36.015 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:36.014+0000 7f91130b0700 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 -- 0x7f910c111960 con 0x7f910c1026e0 2024-04-23T10:26:36.015 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:36.014+0000 7f91130b0700 1 -- --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_getmap magic: 0 -- 0x7f910c111c60 con 0x7f910c106990 2024-04-23T10:26:36.015 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:36.014+0000 7f9110e4c700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f910c101d20 0x7f910c19b2b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T10:26:36.015 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:36.014+0000 7f9110e4c700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f910c101d20 0x7f910c19b2b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.114:40540/0 (socket says 172.21.15.114:40540) 2024-04-23T10:26:36.015 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:36.014+0000 7f9110e4c700 1 -- 172.21.15.114:0/2423038962 learned_addr learned my addr 172.21.15.114:0/2423038962 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-04-23T10:26:36.015 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:36.014+0000 7f910bfff700 1 --2- 172.21.15.114:0/2423038962 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f910c1026e0 0x7f910c19b7f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T10:26:36.015 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:36.014+0000 7f911164d700 1 --2- 172.21.15.114:0/2423038962 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f910c106990 0x7f910c19fb80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T10:26:36.015 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:36.014+0000 7f9110e4c700 1 -- 172.21.15.114:0/2423038962 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f910c106990 msgr2=0x7f910c19fb80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T10:26:36.015 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:36.014+0000 7f9110e4c700 1 --2- 172.21.15.114:0/2423038962 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f910c106990 0x7f910c19fb80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:26:36.015 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:36.014+0000 7f9110e4c700 1 -- 172.21.15.114:0/2423038962 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f910c1026e0 msgr2=0x7f910c19b7f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T10:26:36.015 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:36.014+0000 7f9110e4c700 1 --2- 172.21.15.114:0/2423038962 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f910c1026e0 0x7f910c19b7f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:26:36.015 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:36.014+0000 7f9110e4c700 1 -- 172.21.15.114:0/2423038962 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f910c1a02c0 con 0x7f910c101d20 2024-04-23T10:26:36.015 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:36.014+0000 7f9110e4c700 1 --2- 172.21.15.114:0/2423038962 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f910c101d20 0x7f910c19b2b0 secure :-1 s=READY pgs=409 cs=0 l=1 rev1=1 crypto rx=0x7f910000d950 tx=0x7f910000dc60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T10:26:36.015 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:36.014+0000 7f9109ffb700 1 -- 172.21.15.114:0/2423038962 <== mon.0 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9100016070 con 0x7f910c101d20 2024-04-23T10:26:36.016 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:36.015+0000 7f9109ffb700 1 -- 172.21.15.114:0/2423038962 <== mon.0 v2:172.21.15.114:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f910000fd40 con 0x7f910c101d20 2024-04-23T10:26:36.016 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:36.015+0000 7f9109ffb700 1 -- 172.21.15.114:0/2423038962 <== mon.0 v2:172.21.15.114:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f910000e3b0 con 0x7f910c101d20 2024-04-23T10:26:36.016 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:36.015+0000 7f91130b0700 1 -- 172.21.15.114:0/2423038962 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f910c1a05b0 con 0x7f910c101d20 2024-04-23T10:26:36.016 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:36.015+0000 7f91130b0700 1 -- 172.21.15.114:0/2423038962 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f910c1a7e30 con 0x7f910c101d20 2024-04-23T10:26:36.016 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:36.016+0000 7f91130b0700 1 -- 172.21.15.114:0/2423038962 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f910c103080 con 0x7f910c101d20 2024-04-23T10:26:36.016 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:36.017+0000 7f9109ffb700 1 -- 172.21.15.114:0/2423038962 <== mon.0 v2:172.21.15.114:3300/0 4 ==== mgrmap(e 19) ==== 138664+0+0 (secure 0 0 0) 0x7f9100019070 con 0x7f910c101d20 2024-04-23T10:26:36.017 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:36.018+0000 7f9109ffb700 1 --2- 172.21.15.114:0/2423038962 >> [v2:172.21.15.114:6800/465333808,v1:172.21.15.114:6801/465333808] conn(0x7f90fc0a3d10 0x7f90fc0a61d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T10:26:36.017 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:36.018+0000 7f910bfff700 1 -- 172.21.15.114:0/2423038962 >> [v2:172.21.15.114:6800/465333808,v1:172.21.15.114:6801/465333808] conn(0x7f90fc0a3d10 msgr2=0x7f90fc0a61d0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.114:6800/465333808 2024-04-23T10:26:36.017 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:36.018+0000 7f9109ffb700 1 -- 172.21.15.114:0/2423038962 <== mon.0 v2:172.21.15.114:3300/0 5 ==== osd_map(89..89 src has 1..89) ==== 8269+0+0 (secure 0 0 0) 0x7f91000cfe30 con 0x7f910c101d20 2024-04-23T10:26:36.017 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:36.018+0000 7f910bfff700 1 --2- 172.21.15.114:0/2423038962 >> [v2:172.21.15.114:6800/465333808,v1:172.21.15.114:6801/465333808] conn(0x7f90fc0a3d10 0x7f90fc0a61d0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-04-23T10:26:36.022 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:36.023+0000 7f9109ffb700 1 -- 172.21.15.114:0/2423038962 <== mon.0 v2:172.21.15.114:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f910009c7f0 con 0x7f910c101d20 2024-04-23T10:26:36.078 INFO:journalctl@ceph.mgr.z.smithi204.stdout:Apr 23 10:26:35 smithi204 ceph-aa240572-0159-11ef-bc93-c7b262605968-mgr-z[34493]: 2024-04-23T10:26:35.796+0000 7f39c4768200 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2024-04-23T10:26:36.078 INFO:journalctl@ceph.mgr.z.smithi204.stdout:Apr 23 10:26:35 smithi204 ceph-aa240572-0159-11ef-bc93-c7b262605968-mgr-z[34493]: 2024-04-23T10:26:35.888+0000 7f39c4768200 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2024-04-23T10:26:36.080 INFO:journalctl@ceph.mgr.y.smithi119.stdout:Apr 23 10:26:35 smithi119 ceph-aa240572-0159-11ef-bc93-c7b262605968-mgr-y[34019]: 2024-04-23T10:26:35.784+0000 7f0f8532c200 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2024-04-23T10:26:36.080 INFO:journalctl@ceph.mgr.y.smithi119.stdout:Apr 23 10:26:35 smithi119 ceph-aa240572-0159-11ef-bc93-c7b262605968-mgr-y[34019]: 2024-04-23T10:26:35.876+0000 7f0f8532c200 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2024-04-23T10:26:36.218 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:36.219+0000 7f910bfff700 1 -- 172.21.15.114:0/2423038962 >> [v2:172.21.15.114:6800/465333808,v1:172.21.15.114:6801/465333808] conn(0x7f90fc0a3d10 msgr2=0x7f90fc0a61d0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.114:6800/465333808 2024-04-23T10:26:36.218 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:36.219+0000 7f910bfff700 1 --2- 172.21.15.114:0/2423038962 >> [v2:172.21.15.114:6800/465333808,v1:172.21.15.114:6801/465333808] conn(0x7f90fc0a3d10 0x7f90fc0a61d0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-04-23T10:26:36.218 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Apr 23 10:26:35 smithi114 ceph-aa240572-0159-11ef-bc93-c7b262605968-mgr-x[29490]: 2024-04-23T10:26:35.922+0000 7f21d5288200 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2024-04-23T10:26:36.218 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Apr 23 10:26:36 smithi114 ceph-aa240572-0159-11ef-bc93-c7b262605968-mgr-x[29490]: 2024-04-23T10:26:36.022+0000 7f21d5288200 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2024-04-23T10:26:36.278 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:36.279+0000 7f91130b0700 1 -- 172.21.15.114:0/2423038962 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_command({"prefix": "fs ls"} v 0) -- 0x7f910c1a8090 con 0x7f910c101d20 2024-04-23T10:26:36.278 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:36.279+0000 7f9109ffb700 1 -- 172.21.15.114:0/2423038962 <== mon.0 v2:172.21.15.114:3300/0 7 ==== mon_command_ack([{"prefix": "fs ls"}]=0 v20) ==== 53+0+73 (secure 0 0 0) 0x7f910009c0f0 con 0x7f910c101d20 2024-04-23T10:26:36.278 INFO:teuthology.orchestra.run.smithi114.stdout:name: cephfs, metadata pool: cephfs_metadata, data pools: [cephfs_data ] 2024-04-23T10:26:36.280 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:36.281+0000 7f91130b0700 1 -- 172.21.15.114:0/2423038962 >> [v2:172.21.15.114:6800/465333808,v1:172.21.15.114:6801/465333808] conn(0x7f90fc0a3d10 msgr2=0x7f90fc0a61d0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-04-23T10:26:36.280 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:36.281+0000 7f91130b0700 1 --2- 172.21.15.114:0/2423038962 >> [v2:172.21.15.114:6800/465333808,v1:172.21.15.114:6801/465333808] conn(0x7f90fc0a3d10 0x7f90fc0a61d0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:26:36.280 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:36.281+0000 7f91130b0700 1 -- 172.21.15.114:0/2423038962 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f910c101d20 msgr2=0x7f910c19b2b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T10:26:36.281 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:36.281+0000 7f91130b0700 1 --2- 172.21.15.114:0/2423038962 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f910c101d20 0x7f910c19b2b0 secure :-1 s=READY pgs=409 cs=0 l=1 rev1=1 crypto rx=0x7f910000d950 tx=0x7f910000dc60 comp rx=0 tx=0).stop 2024-04-23T10:26:36.281 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:36.281+0000 7f91130b0700 1 -- 172.21.15.114:0/2423038962 shutdown_connections 2024-04-23T10:26:36.281 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:36.281+0000 7f91130b0700 1 --2- 172.21.15.114:0/2423038962 >> [v2:172.21.15.114:6800/465333808,v1:172.21.15.114:6801/465333808] conn(0x7f90fc0a3d10 0x7f90fc0a61d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:26:36.281 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:36.281+0000 7f91130b0700 1 --2- 172.21.15.114:0/2423038962 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f910c106990 0x7f910c19fb80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:26:36.281 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:36.281+0000 7f91130b0700 1 --2- 172.21.15.114:0/2423038962 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f910c101d20 0x7f910c19b2b0 unknown :-1 s=CLOSED pgs=409 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:26:36.281 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:36.281+0000 7f91130b0700 1 --2- 172.21.15.114:0/2423038962 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f910c1026e0 0x7f910c19b7f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:26:36.281 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:36.281+0000 7f91130b0700 1 -- 172.21.15.114:0/2423038962 >> 172.21.15.114:0/2423038962 conn(0x7f910c0788f0 msgr2=0x7f910c06b270 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T10:26:36.281 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:36.282+0000 7f91130b0700 1 -- 172.21.15.114:0/2423038962 shutdown_connections 2024-04-23T10:26:36.281 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:36.282+0000 7f91130b0700 1 -- 172.21.15.114:0/2423038962 wait complete. 2024-04-23T10:26:36.492 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 23 10:26:36 smithi119 ceph-mon[32608]: from='client.? 172.21.15.114:0/2423038962' entity='client.admin' cmd=[{"prefix": "fs ls"}]: dispatch 2024-04-23T10:26:36.492 INFO:journalctl@ceph.mgr.y.smithi119.stdout:Apr 23 10:26:36 smithi119 ceph-aa240572-0159-11ef-bc93-c7b262605968-mgr-y[34019]: 2024-04-23T10:26:36.081+0000 7f0f8532c200 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2024-04-23T10:26:36.492 INFO:journalctl@ceph.mgr.y.smithi119.stdout:Apr 23 10:26:36 smithi119 ceph-aa240572-0159-11ef-bc93-c7b262605968-mgr-y[34019]: 2024-04-23T10:26:36.273+0000 7f0f8532c200 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2024-04-23T10:26:36.504 INFO:journalctl@ceph.mon.a.smithi114.stdout:Apr 23 10:26:36 smithi114 ceph-mon[29255]: from='client.? 172.21.15.114:0/2423038962' entity='client.admin' cmd=[{"prefix": "fs ls"}]: dispatch 2024-04-23T10:26:36.504 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Apr 23 10:26:36 smithi114 ceph-aa240572-0159-11ef-bc93-c7b262605968-mgr-x[29490]: 2024-04-23T10:26:36.234+0000 7f21d5288200 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2024-04-23T10:26:36.504 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Apr 23 10:26:36 smithi114 ceph-aa240572-0159-11ef-bc93-c7b262605968-mgr-x[29490]: 2024-04-23T10:26:36.441+0000 7f21d5288200 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2024-04-23T10:26:36.578 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 23 10:26:36 smithi204 ceph-mon[32649]: from='client.? 172.21.15.114:0/2423038962' entity='client.admin' cmd=[{"prefix": "fs ls"}]: dispatch 2024-04-23T10:26:36.578 INFO:journalctl@ceph.mgr.z.smithi204.stdout:Apr 23 10:26:36 smithi204 ceph-aa240572-0159-11ef-bc93-c7b262605968-mgr-z[34493]: 2024-04-23T10:26:36.095+0000 7f39c4768200 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2024-04-23T10:26:36.578 INFO:journalctl@ceph.mgr.z.smithi204.stdout:Apr 23 10:26:36 smithi204 ceph-aa240572-0159-11ef-bc93-c7b262605968-mgr-z[34493]: 2024-04-23T10:26:36.289+0000 7f39c4768200 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2024-04-23T10:26:37.004 INFO:tasks.cephfs.mount:Mounting default Ceph FS; just confirmed its presence on cluster 2024-04-23T10:26:37.004 INFO:tasks.cephfs.mount:Mounting Ceph FS. Following are details of mount; remember "None" represents Python type None - 2024-04-23T10:26:37.004 INFO:tasks.cephfs.mount:self.client_remote.hostname = smithi114.front.sepia.ceph.com 2024-04-23T10:26:37.005 INFO:tasks.cephfs.mount:self.client.name = client.0 2024-04-23T10:26:37.005 INFO:tasks.cephfs.mount:self.hostfs_mntpt = /home/ubuntu/cephtest/mnt.0 2024-04-23T10:26:37.005 INFO:tasks.cephfs.mount:self.cephfs_name = None 2024-04-23T10:26:37.005 INFO:tasks.cephfs.mount:self.cephfs_mntpt = None 2024-04-23T10:26:37.005 INFO:tasks.cephfs.mount:self.client_keyring_path = None 2024-04-23T10:26:37.005 INFO:tasks.cephfs.mount:Setting the 'None' netns for '/home/ubuntu/cephtest/mnt.0' 2024-04-23T10:26:37.005 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-23T10:26:37.005 DEBUG:teuthology.orchestra.run.smithi114:> ip addr 2024-04-23T10:26:37.021 INFO:teuthology.orchestra.run.smithi114.stdout:1: lo: mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000 2024-04-23T10:26:37.021 INFO:teuthology.orchestra.run.smithi114.stdout: link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00 2024-04-23T10:26:37.021 INFO:teuthology.orchestra.run.smithi114.stdout: inet 127.0.0.1/8 scope host lo 2024-04-23T10:26:37.021 INFO:teuthology.orchestra.run.smithi114.stdout: valid_lft forever preferred_lft forever 2024-04-23T10:26:37.021 INFO:teuthology.orchestra.run.smithi114.stdout: inet6 ::1/128 scope host 2024-04-23T10:26:37.021 INFO:teuthology.orchestra.run.smithi114.stdout: valid_lft forever preferred_lft forever 2024-04-23T10:26:37.021 INFO:teuthology.orchestra.run.smithi114.stdout:2: eno1: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-04-23T10:26:37.021 INFO:teuthology.orchestra.run.smithi114.stdout: link/ether 0c:c4:7a:d8:a8:92 brd ff:ff:ff:ff:ff:ff 2024-04-23T10:26:37.022 INFO:teuthology.orchestra.run.smithi114.stdout: altname enp7s0f0 2024-04-23T10:26:37.022 INFO:teuthology.orchestra.run.smithi114.stdout:3: eno2: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-04-23T10:26:37.022 INFO:teuthology.orchestra.run.smithi114.stdout: link/ether 0c:c4:7a:d8:a8:93 brd ff:ff:ff:ff:ff:ff 2024-04-23T10:26:37.022 INFO:teuthology.orchestra.run.smithi114.stdout: altname enp7s0f1 2024-04-23T10:26:37.022 INFO:teuthology.orchestra.run.smithi114.stdout:4: enp3s0f0: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-04-23T10:26:37.022 INFO:teuthology.orchestra.run.smithi114.stdout: link/ether 0c:c4:7a:8f:cb:62 brd ff:ff:ff:ff:ff:ff 2024-04-23T10:26:37.022 INFO:teuthology.orchestra.run.smithi114.stdout:5: enp3s0f1: mtu 1500 qdisc mq state UP group default qlen 1000 2024-04-23T10:26:37.022 INFO:teuthology.orchestra.run.smithi114.stdout: link/ether 0c:c4:7a:8f:cb:63 brd ff:ff:ff:ff:ff:ff 2024-04-23T10:26:37.022 INFO:teuthology.orchestra.run.smithi114.stdout: inet 172.21.15.114/20 brd 172.21.15.255 scope global dynamic noprefixroute enp3s0f1 2024-04-23T10:26:37.022 INFO:teuthology.orchestra.run.smithi114.stdout: valid_lft 41480sec preferred_lft 41480sec 2024-04-23T10:26:37.022 INFO:teuthology.orchestra.run.smithi114.stdout: inet6 fe80::db0f:356f:35d2:e591/64 scope link noprefixroute 2024-04-23T10:26:37.022 INFO:teuthology.orchestra.run.smithi114.stdout: valid_lft forever preferred_lft forever 2024-04-23T10:26:37.023 INFO:tasks.cephfs.mount:Setuping the 'ceph-brx' with 192.168.255.254/16 2024-04-23T10:26:37.023 DEBUG:teuthology.orchestra.run.smithi114:> (cd / && exec stdin-killer --timeout=300 -- bash -c ' 2024-04-23T10:26:37.023 DEBUG:teuthology.orchestra.run.smithi114:> set -e 2024-04-23T10:26:37.023 DEBUG:teuthology.orchestra.run.smithi114:> sudo ip link add name ceph-brx type bridge 2024-04-23T10:26:37.023 DEBUG:teuthology.orchestra.run.smithi114:> sudo ip addr flush dev ceph-brx 2024-04-23T10:26:37.023 DEBUG:teuthology.orchestra.run.smithi114:> sudo ip link set ceph-brx up 2024-04-23T10:26:37.023 DEBUG:teuthology.orchestra.run.smithi114:> sudo ip addr add 192.168.255.254/16 brd 192.168.255.255 dev ceph-brx 2024-04-23T10:26:37.023 DEBUG:teuthology.orchestra.run.smithi114:> ') 2024-04-23T10:26:37.111 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:37 stdin-killer INFO: expiration expected; waiting 300 seconds for command to complete 2024-04-23T10:26:37.201 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:37 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-04-23T10:26:37.208 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-23T10:26:37.208 DEBUG:teuthology.orchestra.run.smithi114:> echo 1 | sudo tee /proc/sys/net/ipv4/ip_forward 2024-04-23T10:26:37.230 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Apr 23 10:26:37 smithi114 ceph-aa240572-0159-11ef-bc93-c7b262605968-mgr-x[29490]: 2024-04-23T10:26:37.055+0000 7f21d5288200 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2024-04-23T10:26:37.232 INFO:teuthology.orchestra.run.smithi114.stdout:1 2024-04-23T10:26:37.234 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-23T10:26:37.234 DEBUG:teuthology.orchestra.run.smithi114:> route 2024-04-23T10:26:37.242 INFO:journalctl@ceph.mgr.y.smithi119.stdout:Apr 23 10:26:36 smithi119 ceph-aa240572-0159-11ef-bc93-c7b262605968-mgr-y[34019]: 2024-04-23T10:26:36.868+0000 7f0f8532c200 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2024-04-23T10:26:37.299 INFO:teuthology.orchestra.run.smithi114.stdout:Kernel IP routing table 2024-04-23T10:26:37.299 INFO:teuthology.orchestra.run.smithi114.stdout:Destination Gateway Genmask Flags Metric Ref Use Iface 2024-04-23T10:26:37.300 INFO:teuthology.orchestra.run.smithi114.stdout:default _gateway 0.0.0.0 UG 100 0 0 enp3s0f1 2024-04-23T10:26:37.300 INFO:teuthology.orchestra.run.smithi114.stdout:172.21.0.0 0.0.0.0 255.255.240.0 U 100 0 0 enp3s0f1 2024-04-23T10:26:37.300 INFO:teuthology.orchestra.run.smithi114.stdout:192.168.0.0 0.0.0.0 255.255.0.0 U 0 0 0 ceph-brx 2024-04-23T10:26:37.301 DEBUG:teuthology.orchestra.run.smithi114:> (cd / && exec stdin-killer --timeout=300 -- bash -c ' 2024-04-23T10:26:37.301 DEBUG:teuthology.orchestra.run.smithi114:> set -e 2024-04-23T10:26:37.301 DEBUG:teuthology.orchestra.run.smithi114:> sudo iptables -A FORWARD -o enp3s0f1 -i ceph-brx -j ACCEPT 2024-04-23T10:26:37.301 DEBUG:teuthology.orchestra.run.smithi114:> sudo iptables -A FORWARD -i enp3s0f1 -o ceph-brx -j ACCEPT 2024-04-23T10:26:37.301 DEBUG:teuthology.orchestra.run.smithi114:> sudo iptables -t nat -A POSTROUTING -s 192.168.255.254/16 -o enp3s0f1 -j MASQUERADE 2024-04-23T10:26:37.302 DEBUG:teuthology.orchestra.run.smithi114:> ') 2024-04-23T10:26:37.315 INFO:journalctl@ceph.mgr.z.smithi204.stdout:Apr 23 10:26:36 smithi204 ceph-aa240572-0159-11ef-bc93-c7b262605968-mgr-z[34493]: 2024-04-23T10:26:36.892+0000 7f39c4768200 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2024-04-23T10:26:37.388 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:37 stdin-killer INFO: expiration expected; waiting 300 seconds for command to complete 2024-04-23T10:26:37.557 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:37 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-04-23T10:26:37.563 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-23T10:26:37.563 DEBUG:teuthology.orchestra.run.smithi114:> ip netns list 2024-04-23T10:26:37.578 INFO:journalctl@ceph.mgr.z.smithi204.stdout:Apr 23 10:26:37 smithi204 ceph-aa240572-0159-11ef-bc93-c7b262605968-mgr-z[34493]: 2024-04-23T10:26:37.316+0000 7f39c4768200 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2024-04-23T10:26:37.578 INFO:journalctl@ceph.mgr.z.smithi204.stdout:Apr 23 10:26:37 smithi204 ceph-aa240572-0159-11ef-bc93-c7b262605968-mgr-z[34493]: 2024-04-23T10:26:37.413+0000 7f39c4768200 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2024-04-23T10:26:37.579 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-23T10:26:37.579 DEBUG:teuthology.orchestra.run.smithi114:> ip netns list-id 2024-04-23T10:26:37.585 INFO:journalctl@ceph.mgr.y.smithi119.stdout:Apr 23 10:26:37 smithi119 ceph-aa240572-0159-11ef-bc93-c7b262605968-mgr-y[34019]: 2024-04-23T10:26:37.289+0000 7f0f8532c200 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2024-04-23T10:26:37.585 INFO:journalctl@ceph.mgr.y.smithi119.stdout:Apr 23 10:26:37 smithi119 ceph-aa240572-0159-11ef-bc93-c7b262605968-mgr-y[34019]: 2024-04-23T10:26:37.385+0000 7f0f8532c200 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2024-04-23T10:26:37.634 DEBUG:teuthology.orchestra.run.smithi114:> (cd / && exec stdin-killer --timeout=300 -- bash -c ' 2024-04-23T10:26:37.634 DEBUG:teuthology.orchestra.run.smithi114:> set -e 2024-04-23T10:26:37.634 DEBUG:teuthology.orchestra.run.smithi114:> sudo ip netns add ceph-ns--home-ubuntu-cephtest-mnt.0 2024-04-23T10:26:37.634 DEBUG:teuthology.orchestra.run.smithi114:> sudo ip netns set ceph-ns--home-ubuntu-cephtest-mnt.0 0 2024-04-23T10:26:37.634 DEBUG:teuthology.orchestra.run.smithi114:> ') 2024-04-23T10:26:37.723 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:37 stdin-killer INFO: expiration expected; waiting 300 seconds for command to complete 2024-04-23T10:26:37.741 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Apr 23 10:26:37 smithi114 ceph-aa240572-0159-11ef-bc93-c7b262605968-mgr-x[29490]: 2024-04-23T10:26:37.510+0000 7f21d5288200 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2024-04-23T10:26:37.742 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Apr 23 10:26:37 smithi114 ceph-aa240572-0159-11ef-bc93-c7b262605968-mgr-x[29490]: 2024-04-23T10:26:37.614+0000 7f21d5288200 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2024-04-23T10:26:37.745 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:37 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-04-23T10:26:37.754 INFO:tasks.cephfs.mount:Setuping the netns 'ceph-ns--home-ubuntu-cephtest-mnt.0' with 192.168.0.1/16 2024-04-23T10:26:37.754 DEBUG:teuthology.orchestra.run.smithi114:> (cd / && exec stdin-killer --timeout=300 -- bash -c ' 2024-04-23T10:26:37.754 DEBUG:teuthology.orchestra.run.smithi114:> set -e 2024-04-23T10:26:37.754 DEBUG:teuthology.orchestra.run.smithi114:> sudo ip link add veth0 netns ceph-ns--home-ubuntu-cephtest-mnt.0 type veth peer name brx.0 2024-04-23T10:26:37.754 DEBUG:teuthology.orchestra.run.smithi114:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.0 ip addr add 192.168.0.1/16 brd 192.168.255.255 dev veth0 2024-04-23T10:26:37.754 DEBUG:teuthology.orchestra.run.smithi114:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.0 ip link set veth0 up 2024-04-23T10:26:37.754 DEBUG:teuthology.orchestra.run.smithi114:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.0 ip link set lo up 2024-04-23T10:26:37.754 DEBUG:teuthology.orchestra.run.smithi114:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.0 ip route add default via 192.168.255.254 2024-04-23T10:26:37.755 DEBUG:teuthology.orchestra.run.smithi114:> ') 2024-04-23T10:26:37.803 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:37 stdin-killer INFO: expiration expected; waiting 300 seconds for command to complete 2024-04-23T10:26:37.889 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:37 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-04-23T10:26:37.895 DEBUG:teuthology.orchestra.run.smithi114:> (cd / && exec stdin-killer --timeout=300 -- bash -c ' 2024-04-23T10:26:37.895 DEBUG:teuthology.orchestra.run.smithi114:> set -e 2024-04-23T10:26:37.896 DEBUG:teuthology.orchestra.run.smithi114:> sudo ip link set brx.0 up 2024-04-23T10:26:37.896 DEBUG:teuthology.orchestra.run.smithi114:> sudo ip link set dev brx.0 master ceph-brx 2024-04-23T10:26:37.896 DEBUG:teuthology.orchestra.run.smithi114:> ') 2024-04-23T10:26:37.944 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:37 stdin-killer INFO: expiration expected; waiting 300 seconds for command to complete 2024-04-23T10:26:37.965 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:37 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-04-23T10:26:37.972 INFO:teuthology.orchestra.run:Running command with timeout 60 2024-04-23T10:26:37.972 DEBUG:teuthology.orchestra.run.smithi114:> mkdir -p -v /home/ubuntu/cephtest/mnt.0 2024-04-23T10:26:37.987 INFO:teuthology.orchestra.run.smithi114.stdout:mkdir: created directory '/home/ubuntu/cephtest/mnt.0' 2024-04-23T10:26:37.987 INFO:teuthology.orchestra.run:Running command with timeout 60 2024-04-23T10:26:37.987 DEBUG:teuthology.orchestra.run.smithi114:> chmod 0000 /home/ubuntu/cephtest/mnt.0 2024-04-23T10:26:37.991 INFO:journalctl@ceph.mgr.y.smithi119.stdout:Apr 23 10:26:37 smithi119 ceph-aa240572-0159-11ef-bc93-c7b262605968-mgr-y[34019]: 2024-04-23T10:26:37.586+0000 7f0f8532c200 -1 mgr[py] Module rgw has missing NOTIFY_TYPES member 2024-04-23T10:26:38.004 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Apr 23 10:26:37 smithi114 ceph-aa240572-0159-11ef-bc93-c7b262605968-mgr-x[29490]: 2024-04-23T10:26:37.836+0000 7f21d5288200 -1 mgr[py] Module rgw has missing NOTIFY_TYPES member 2024-04-23T10:26:38.018 WARNING:tasks.cephfs.mount:You cannot specify both: cephfs_mntpt and mount_subvol_num 2024-04-23T10:26:38.018 INFO:tasks.cephfs.mount:Mounting subvol 1 for now 2024-04-23T10:26:38.018 INFO:tasks.cephfs.kernel_mount:mounting using device: :/volumes/qa/sv_1/b206794e-7622-4ab8-8943-f3ae4e65b7a3 2024-04-23T10:26:38.018 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-23T10:26:38.018 DEBUG:teuthology.orchestra.run.smithi114:> sudo nsenter --net=/var/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage /bin/mount -t ceph :/volumes/qa/sv_1/b206794e-7622-4ab8-8943-f3ae4e65b7a3 /home/ubuntu/cephtest/mnt.0 -v -o norequire_active_mds,conf=/etc/ceph/ceph.conf,norbytes,name=0,mds_namespace=cephfs,ms_mode=crc,nowsync 2024-04-23T10:26:38.078 INFO:journalctl@ceph.mgr.z.smithi204.stdout:Apr 23 10:26:37 smithi204 ceph-aa240572-0159-11ef-bc93-c7b262605968-mgr-z[34493]: 2024-04-23T10:26:37.614+0000 7f39c4768200 -1 mgr[py] Module rgw has missing NOTIFY_TYPES member 2024-04-23T10:26:38.111 INFO:teuthology.orchestra.run.smithi114.stdout:parsing options: rw,norequire_active_mds,conf=/etc/ceph/ceph.conf,norbytes,name=0,mds_namespace=cephfs,ms_mode=crc,nowsync 2024-04-23T10:26:38.111 INFO:teuthology.orchestra.run.smithi114.stdout:mount.ceph: options "norequire_active_mds,norbytes,name=0,mds_namespace=cephfs,ms_mode=crc,nowsync". 2024-04-23T10:26:38.111 INFO:teuthology.orchestra.run.smithi114.stdout:invalid new device string format 2024-04-23T10:26:38.237 INFO:teuthology.orchestra.run.smithi114.stdout:parsing options: rw,norequire_active_mds,conf=/etc/ceph/ceph.conf,norbytes,name=0,mds_namespace=cephfs,ms_mode=crc,nowsync 2024-04-23T10:26:38.237 INFO:teuthology.orchestra.run.smithi114.stdout:mount.ceph: options "norequire_active_mds,norbytes,name=0,mds_namespace=cephfs,ms_mode=crc,nowsync". 2024-04-23T10:26:38.237 INFO:teuthology.orchestra.run.smithi114.stdout:invalid new device string format 2024-04-23T10:26:38.237 INFO:teuthology.orchestra.run.smithi114.stdout:mount.ceph: resolved to: "172.21.15.114:3300,172.21.15.119:3300,172.21.15.204:3300" 2024-04-23T10:26:38.237 INFO:teuthology.orchestra.run.smithi114.stdout:mount.ceph: trying mount with old device syntax: 172.21.15.114:3300,172.21.15.119:3300,172.21.15.204:3300:/volumes/qa/sv_1/b206794e-7622-4ab8-8943-f3ae4e65b7a3 2024-04-23T10:26:38.237 INFO:teuthology.orchestra.run.smithi114.stdout:mount.ceph: options "norequire_active_mds,norbytes,name=0,mds_namespace=cephfs,ms_mode=crc,nowsync,key=0,fsid=aa240572-0159-11ef-bc93-c7b262605968" will pass to kernel 2024-04-23T10:26:38.238 INFO:teuthology.orchestra.run.smithi114.stdout:mount: /home/ubuntu/cephtest/mnt.0 does not contain SELinux labels. 2024-04-23T10:26:38.238 INFO:teuthology.orchestra.run.smithi114.stdout: You just mounted a file system that supports labels which does not 2024-04-23T10:26:38.238 INFO:teuthology.orchestra.run.smithi114.stdout: contain labels, onto an SELinux box. It is likely that confined 2024-04-23T10:26:38.239 INFO:teuthology.orchestra.run.smithi114.stdout: applications will generate AVC messages and not be allowed access to 2024-04-23T10:26:38.239 INFO:teuthology.orchestra.run.smithi114.stdout: this file system. For more details see restorecon(8) and mount(8). 2024-04-23T10:26:38.240 INFO:tasks.cephfs.kernel_mount:mount command passed 2024-04-23T10:26:38.240 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-23T10:26:38.240 DEBUG:teuthology.orchestra.run.smithi114:> sudo chmod 1777 /home/ubuntu/cephtest/mnt.0 2024-04-23T10:26:38.265 DEBUG:teuthology.orchestra.run.smithi114:> (cd /home/ubuntu/cephtest/mnt.0 && exec stdin-killer --timeout=900 -- bash -c 'getfattr --only-values -n ceph.client_id .') 2024-04-23T10:26:38.356 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:38 stdin-killer INFO: expiration expected; waiting 900 seconds for command to complete 2024-04-23T10:26:38.358 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:38 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-04-23T10:26:38.364 INFO:teuthology.orchestra.run.smithi114.stdout:client15312 2024-04-23T10:26:38.364 DEBUG:teuthology.orchestra.run.smithi114:> set -ex 2024-04-23T10:26:38.364 DEBUG:teuthology.orchestra.run.smithi114:> dd if=/proc/self/mounts of=/dev/stdout 2024-04-23T10:26:38.380 DEBUG:teuthology.orchestra.run.smithi114:> (cd /home/ubuntu/cephtest/mnt.0 && exec stdin-killer --timeout=900 -- bash -c 'getfattr --only-values -n ceph.client_id .') 2024-04-23T10:26:38.439 INFO:journalctl@ceph.mgr.y.smithi119.stdout:Apr 23 10:26:38 smithi119 ceph-aa240572-0159-11ef-bc93-c7b262605968-mgr-y[34019]: 2024-04-23T10:26:38.104+0000 7f0f8532c200 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2024-04-23T10:26:38.439 INFO:journalctl@ceph.mgr.y.smithi119.stdout:Apr 23 10:26:38 smithi119 ceph-aa240572-0159-11ef-bc93-c7b262605968-mgr-y[34019]: 2024-04-23T10:26:38.195+0000 7f0f8532c200 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2024-04-23T10:26:38.469 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:38 stdin-killer INFO: expiration expected; waiting 900 seconds for command to complete 2024-04-23T10:26:38.471 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:38 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-04-23T10:26:38.473 INFO:journalctl@ceph.mgr.z.smithi204.stdout:Apr 23 10:26:38 smithi204 ceph-aa240572-0159-11ef-bc93-c7b262605968-mgr-z[34493]: 2024-04-23T10:26:38.139+0000 7f39c4768200 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2024-04-23T10:26:38.473 INFO:journalctl@ceph.mgr.z.smithi204.stdout:Apr 23 10:26:38 smithi204 ceph-aa240572-0159-11ef-bc93-c7b262605968-mgr-z[34493]: 2024-04-23T10:26:38.232+0000 7f39c4768200 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2024-04-23T10:26:38.477 INFO:teuthology.orchestra.run.smithi114.stdout:client15312 2024-04-23T10:26:38.477 DEBUG:teuthology.orchestra.run.smithi114:> (cd /home/ubuntu/cephtest/mnt.0 && exec stdin-killer --timeout=900 -- bash -c 'getfattr --only-values -n ceph.client_id .') 2024-04-23T10:26:38.525 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:38 stdin-killer INFO: expiration expected; waiting 900 seconds for command to complete 2024-04-23T10:26:38.527 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:38 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-04-23T10:26:38.533 INFO:teuthology.orchestra.run.smithi114.stdout:client15312 2024-04-23T10:26:38.533 DEBUG:teuthology.orchestra.run.smithi114:> (cd /home/ubuntu/cephtest/mnt.0 && exec stdin-killer --timeout=300 -- bash -c 'sudo dd if=/sys/kernel/debug/ceph/aa240572-0159-11ef-bc93-c7b262605968.client15312/status') 2024-04-23T10:26:38.582 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:38 stdin-killer INFO: expiration expected; waiting 300 seconds for command to complete 2024-04-23T10:26:38.592 INFO:teuthology.orchestra.run.smithi114.stdout:instance: client.15312 (3)192.168.0.1:0/923372514 2024-04-23T10:26:38.592 INFO:teuthology.orchestra.run.smithi114.stdout:blocklisted: false 2024-04-23T10:26:38.592 INFO:teuthology.orchestra.run.smithi114.stderr:0+1 records in 2024-04-23T10:26:38.592 INFO:teuthology.orchestra.run.smithi114.stderr:0+1 records out 2024-04-23T10:26:38.592 INFO:teuthology.orchestra.run.smithi114.stderr:69 bytes copied, 2.2242e-05 s, 3.1 MB/s 2024-04-23T10:26:38.592 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:38 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-04-23T10:26:38.599 INFO:teuthology.run_tasks:Running task exec... 2024-04-23T10:26:38.608 INFO:teuthology.task.exec:Executing custom commands... 2024-04-23T10:26:38.609 INFO:teuthology.task.exec:Running commands on role mon.a host ubuntu@smithi114.front.sepia.ceph.com 2024-04-23T10:26:38.609 DEBUG:teuthology.orchestra.run.smithi114:> sudo TESTDIR=/home/ubuntu/cephtest bash -c 'ceph fs set cephfs balance_automate true' 2024-04-23T10:26:38.713 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:38.712+0000 7f44e2a62640 1 -- 172.21.15.114:0/995026720 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f44dc10e090 msgr2=0x7f44dc10e4b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T10:26:38.714 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:38.712+0000 7f44e2a62640 1 --2- 172.21.15.114:0/995026720 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f44dc10e090 0x7f44dc10e4b0 secure :-1 s=READY pgs=411 cs=0 l=1 rev1=1 crypto rx=0x7f44cc009900 tx=0x7f44cc02a410 comp rx=0 tx=0).stop 2024-04-23T10:26:38.714 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:38.712+0000 7f44e2a62640 1 -- 172.21.15.114:0/995026720 shutdown_connections 2024-04-23T10:26:38.714 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:38.712+0000 7f44e2a62640 1 --2- 172.21.15.114:0/995026720 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f44dc10cf80 0x7f44dc114df0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:26:38.714 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:38.712+0000 7f44e2a62640 1 --2- 172.21.15.114:0/995026720 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f44dc10c5c0 0x7f44dc10ca40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:26:38.714 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:38.712+0000 7f44e2a62640 1 --2- 172.21.15.114:0/995026720 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f44dc10e090 0x7f44dc10e4b0 unknown :-1 s=CLOSED pgs=411 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:26:38.714 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:38.712+0000 7f44e2a62640 1 -- 172.21.15.114:0/995026720 >> 172.21.15.114:0/995026720 conn(0x7f44dc06cb70 msgr2=0x7f44dc06cfa0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T10:26:38.714 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:38.712+0000 7f44e2a62640 1 -- 172.21.15.114:0/995026720 shutdown_connections 2024-04-23T10:26:38.714 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:38.712+0000 7f44e2a62640 1 -- 172.21.15.114:0/995026720 wait complete. 2024-04-23T10:26:38.714 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:38.713+0000 7f44e2a62640 1 Processor -- start 2024-04-23T10:26:38.714 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:38.713+0000 7f44e2a62640 1 -- start start 2024-04-23T10:26:38.714 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:38.713+0000 7f44e2a62640 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f44dc10c5c0 0x7f44dc1a2c90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T10:26:38.714 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:38.713+0000 7f44e2a62640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f44dc10cf80 0x7f44dc1a31d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T10:26:38.714 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:38.713+0000 7f44e2a62640 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f44dc10e090 0x7f44dc1aa230 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T10:26:38.715 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:38.713+0000 7f44e2a62640 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x7f44dc1185d0 con 0x7f44dc10cf80 2024-04-23T10:26:38.715 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:38.713+0000 7f44e2a62640 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 -- 0x7f44dc118450 con 0x7f44dc10c5c0 2024-04-23T10:26:38.715 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:38.713+0000 7f44e2a62640 1 -- --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_getmap magic: 0 -- 0x7f44dc118750 con 0x7f44dc10e090 2024-04-23T10:26:38.715 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:38.713+0000 7f44db7fe640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f44dc10cf80 0x7f44dc1a31d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T10:26:38.715 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:38.713+0000 7f44db7fe640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f44dc10cf80 0x7f44dc1a31d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.114:40564/0 (socket says 172.21.15.114:40564) 2024-04-23T10:26:38.715 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:38.713+0000 7f44db7fe640 1 -- 172.21.15.114:0/3918374212 learned_addr learned my addr 172.21.15.114:0/3918374212 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-04-23T10:26:38.715 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:38.713+0000 7f44dbfff640 1 --2- 172.21.15.114:0/3918374212 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f44dc10c5c0 0x7f44dc1a2c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T10:26:38.715 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:38.713+0000 7f44e0fff640 1 --2- 172.21.15.114:0/3918374212 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f44dc10e090 0x7f44dc1aa230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T10:26:38.715 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:38.714+0000 7f44db7fe640 1 -- 172.21.15.114:0/3918374212 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f44dc10e090 msgr2=0x7f44dc1aa230 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T10:26:38.715 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:38.714+0000 7f44db7fe640 1 --2- 172.21.15.114:0/3918374212 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f44dc10e090 0x7f44dc1aa230 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:26:38.715 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:38.714+0000 7f44db7fe640 1 -- 172.21.15.114:0/3918374212 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f44dc10c5c0 msgr2=0x7f44dc1a2c90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T10:26:38.715 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:38.714+0000 7f44db7fe640 1 --2- 172.21.15.114:0/3918374212 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f44dc10c5c0 0x7f44dc1a2c90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:26:38.715 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:38.714+0000 7f44db7fe640 1 -- 172.21.15.114:0/3918374212 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f44dc1aa8d0 con 0x7f44dc10cf80 2024-04-23T10:26:38.715 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:38.714+0000 7f44db7fe640 1 --2- 172.21.15.114:0/3918374212 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f44dc10cf80 0x7f44dc1a31d0 secure :-1 s=READY pgs=412 cs=0 l=1 rev1=1 crypto rx=0x7f44d000e9f0 tx=0x7f44d000edf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T10:26:38.716 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:38.714+0000 7f44d97fa640 1 -- 172.21.15.114:0/3918374212 <== mon.0 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f44d000cd80 con 0x7f44dc10cf80 2024-04-23T10:26:38.716 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:38.714+0000 7f44d97fa640 1 -- 172.21.15.114:0/3918374212 <== mon.0 v2:172.21.15.114:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f44d00054d0 con 0x7f44dc10cf80 2024-04-23T10:26:38.716 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:38.714+0000 7f44e2a62640 1 -- 172.21.15.114:0/3918374212 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f44dc1aabc0 con 0x7f44dc10cf80 2024-04-23T10:26:38.716 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:38.714+0000 7f44d97fa640 1 -- 172.21.15.114:0/3918374212 <== mon.0 v2:172.21.15.114:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f44d0010690 con 0x7f44dc10cf80 2024-04-23T10:26:38.716 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:38.714+0000 7f44e2a62640 1 -- 172.21.15.114:0/3918374212 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f44dc1ab0c0 con 0x7f44dc10cf80 2024-04-23T10:26:38.717 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:38.715+0000 7f44e2a62640 1 -- 172.21.15.114:0/3918374212 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f44dc04fce0 con 0x7f44dc10cf80 2024-04-23T10:26:38.717 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:38.717+0000 7f44d97fa640 1 -- 172.21.15.114:0/3918374212 <== mon.0 v2:172.21.15.114:3300/0 4 ==== mgrmap(e 19) ==== 138664+0+0 (secure 0 0 0) 0x7f44d0004e30 con 0x7f44dc10cf80 2024-04-23T10:26:38.717 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:38.718+0000 7f44d97fa640 1 --2- 172.21.15.114:0/3918374212 >> [v2:172.21.15.114:6800/465333808,v1:172.21.15.114:6801/465333808] conn(0x7f44c40a3ca0 0x7f44c40a6160 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T10:26:38.717 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:38.718+0000 7f44dbfff640 1 -- 172.21.15.114:0/3918374212 >> [v2:172.21.15.114:6800/465333808,v1:172.21.15.114:6801/465333808] conn(0x7f44c40a3ca0 msgr2=0x7f44c40a6160 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.114:6800/465333808 2024-04-23T10:26:38.717 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:38.718+0000 7f44d97fa640 1 -- 172.21.15.114:0/3918374212 <== mon.0 v2:172.21.15.114:3300/0 5 ==== osd_map(89..89 src has 1..89) ==== 8269+0+0 (secure 0 0 0) 0x7f44d00d0a60 con 0x7f44dc10cf80 2024-04-23T10:26:38.717 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:38.718+0000 7f44dbfff640 1 --2- 172.21.15.114:0/3918374212 >> [v2:172.21.15.114:6800/465333808,v1:172.21.15.114:6801/465333808] conn(0x7f44c40a3ca0 0x7f44c40a6160 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-04-23T10:26:38.719 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:38.720+0000 7f44d97fa640 1 -- 172.21.15.114:0/3918374212 <== mon.0 v2:172.21.15.114:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f44d0053ff0 con 0x7f44dc10cf80 2024-04-23T10:26:38.741 INFO:journalctl@ceph.mgr.y.smithi119.stdout:Apr 23 10:26:38 smithi119 ceph-aa240572-0159-11ef-bc93-c7b262605968-mgr-y[34019]: 2024-04-23T10:26:38.439+0000 7f0f8532c200 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2024-04-23T10:26:38.741 INFO:journalctl@ceph.mgr.y.smithi119.stdout:Apr 23 10:26:38 smithi119 ceph-aa240572-0159-11ef-bc93-c7b262605968-mgr-y[34019]: 2024-04-23T10:26:38.572+0000 7f0f8532c200 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2024-04-23T10:26:38.742 INFO:journalctl@ceph.mgr.y.smithi119.stdout:Apr 23 10:26:38 smithi119 ceph-aa240572-0159-11ef-bc93-c7b262605968-mgr-y[34019]: 2024-04-23T10:26:38.667+0000 7f0f8532c200 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2024-04-23T10:26:38.754 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Apr 23 10:26:38 smithi114 ceph-aa240572-0159-11ef-bc93-c7b262605968-mgr-x[29490]: 2024-04-23T10:26:38.391+0000 7f21d5288200 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2024-04-23T10:26:38.754 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Apr 23 10:26:38 smithi114 ceph-aa240572-0159-11ef-bc93-c7b262605968-mgr-x[29490]: 2024-04-23T10:26:38.492+0000 7f21d5288200 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2024-04-23T10:26:38.817 INFO:journalctl@ceph.mgr.z.smithi204.stdout:Apr 23 10:26:38 smithi204 ceph-aa240572-0159-11ef-bc93-c7b262605968-mgr-z[34493]: 2024-04-23T10:26:38.474+0000 7f39c4768200 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2024-04-23T10:26:38.817 INFO:journalctl@ceph.mgr.z.smithi204.stdout:Apr 23 10:26:38 smithi204 ceph-aa240572-0159-11ef-bc93-c7b262605968-mgr-z[34493]: 2024-04-23T10:26:38.607+0000 7f39c4768200 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2024-04-23T10:26:38.818 INFO:journalctl@ceph.mgr.z.smithi204.stdout:Apr 23 10:26:38 smithi204 ceph-aa240572-0159-11ef-bc93-c7b262605968-mgr-z[34493]: 2024-04-23T10:26:38.702+0000 7f39c4768200 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2024-04-23T10:26:38.818 INFO:journalctl@ceph.mgr.z.smithi204.stdout:Apr 23 10:26:38 smithi204 ceph-aa240572-0159-11ef-bc93-c7b262605968-mgr-z[34493]: 2024-04-23T10:26:38.818+0000 7f39c4768200 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2024-04-23T10:26:38.918 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:38.919+0000 7f44dbfff640 1 -- 172.21.15.114:0/3918374212 >> [v2:172.21.15.114:6800/465333808,v1:172.21.15.114:6801/465333808] conn(0x7f44c40a3ca0 msgr2=0x7f44c40a6160 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.114:6800/465333808 2024-04-23T10:26:38.918 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:38.919+0000 7f44dbfff640 1 --2- 172.21.15.114:0/3918374212 >> [v2:172.21.15.114:6800/465333808,v1:172.21.15.114:6801/465333808] conn(0x7f44c40a3ca0 0x7f44c40a6160 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-04-23T10:26:38.942 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:38.944+0000 7f44e2a62640 1 -- 172.21.15.114:0/3918374212 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_command({"prefix": "fs set", "fs_name": "cephfs", "var": "balance_automate", "val": "true"} v 0) -- 0x7f44dc113cd0 con 0x7f44dc10cf80 2024-04-23T10:26:38.994 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:38.995+0000 7f44d97fa640 1 -- 172.21.15.114:0/3918374212 <== mon.0 v2:172.21.15.114:3300/0 7 ==== mon_command_ack([{"prefix": "fs set", "fs_name": "cephfs", "var": "balance_automate", "val": "true"}]=0 v21) ==== 117+0+0 (secure 0 0 0) 0x7f44d009d420 con 0x7f44dc10cf80 2024-04-23T10:26:38.997 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:38.998+0000 7f44e2a62640 1 -- 172.21.15.114:0/3918374212 >> [v2:172.21.15.114:6800/465333808,v1:172.21.15.114:6801/465333808] conn(0x7f44c40a3ca0 msgr2=0x7f44c40a6160 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-04-23T10:26:38.997 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:38.998+0000 7f44e2a62640 1 --2- 172.21.15.114:0/3918374212 >> [v2:172.21.15.114:6800/465333808,v1:172.21.15.114:6801/465333808] conn(0x7f44c40a3ca0 0x7f44c40a6160 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:26:38.997 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:38.998+0000 7f44e2a62640 1 -- 172.21.15.114:0/3918374212 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f44dc10cf80 msgr2=0x7f44dc1a31d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T10:26:38.997 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:38.998+0000 7f44e2a62640 1 --2- 172.21.15.114:0/3918374212 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f44dc10cf80 0x7f44dc1a31d0 secure :-1 s=READY pgs=412 cs=0 l=1 rev1=1 crypto rx=0x7f44d000e9f0 tx=0x7f44d000edf0 comp rx=0 tx=0).stop 2024-04-23T10:26:38.997 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:38.998+0000 7f44e2a62640 1 -- 172.21.15.114:0/3918374212 shutdown_connections 2024-04-23T10:26:38.997 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:38.998+0000 7f44e2a62640 1 --2- 172.21.15.114:0/3918374212 >> [v2:172.21.15.114:6800/465333808,v1:172.21.15.114:6801/465333808] conn(0x7f44c40a3ca0 0x7f44c40a6160 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:26:38.998 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:38.998+0000 7f44e2a62640 1 --2- 172.21.15.114:0/3918374212 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f44dc10e090 0x7f44dc1aa230 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:26:38.998 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:38.998+0000 7f44e2a62640 1 --2- 172.21.15.114:0/3918374212 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f44dc10cf80 0x7f44dc1a31d0 unknown :-1 s=CLOSED pgs=412 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:26:38.998 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:38.998+0000 7f44e2a62640 1 --2- 172.21.15.114:0/3918374212 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f44dc10c5c0 0x7f44dc1a2c90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:26:38.998 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:38.998+0000 7f44e2a62640 1 -- 172.21.15.114:0/3918374212 >> 172.21.15.114:0/3918374212 conn(0x7f44dc06cb70 msgr2=0x7f44dc110ea0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T10:26:38.998 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:38.998+0000 7f44e2a62640 1 -- 172.21.15.114:0/3918374212 shutdown_connections 2024-04-23T10:26:38.998 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:38.998+0000 7f44e2a62640 1 -- 172.21.15.114:0/3918374212 wait complete. 2024-04-23T10:26:39.007 INFO:journalctl@ceph.mon.a.smithi114.stdout:Apr 23 10:26:39 smithi114 ceph-mon[29255]: from='client.? 172.21.15.114:0/3918374212' entity='client.admin' cmd=[{"prefix": "fs set", "fs_name": "cephfs", "var": "balance_automate", "val": "true"}]: dispatch 2024-04-23T10:26:39.008 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Apr 23 10:26:38 smithi114 ceph-aa240572-0159-11ef-bc93-c7b262605968-mgr-x[29490]: 2024-04-23T10:26:38.758+0000 7f21d5288200 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2024-04-23T10:26:39.008 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Apr 23 10:26:38 smithi114 ceph-aa240572-0159-11ef-bc93-c7b262605968-mgr-x[29490]: 2024-04-23T10:26:38.897+0000 7f21d5288200 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2024-04-23T10:26:39.008 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Apr 23 10:26:38 smithi114 ceph-aa240572-0159-11ef-bc93-c7b262605968-mgr-x[29490]: 2024-04-23T10:26:38.997+0000 7f21d5288200 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2024-04-23T10:26:39.009 INFO:teuthology.run_tasks:Running task check-counter... 2024-04-23T10:26:39.019 INFO:teuthology.run_tasks:Running task exec... 2024-04-23T10:26:39.027 INFO:teuthology.task.exec:Executing custom commands... 2024-04-23T10:26:39.028 INFO:teuthology.task.exec:Running commands on role mon.a host ubuntu@smithi114.front.sepia.ceph.com 2024-04-23T10:26:39.028 DEBUG:teuthology.orchestra.run.smithi114:> sudo TESTDIR=/home/ubuntu/cephtest bash -c 'ceph fs snap-schedule add --fs=cephfs --path=/ --snap_schedule=1m' 2024-04-23T10:26:39.077 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 23 10:26:39 smithi204 ceph-mon[32649]: from='client.? 172.21.15.114:0/3918374212' entity='client.admin' cmd=[{"prefix": "fs set", "fs_name": "cephfs", "var": "balance_automate", "val": "true"}]: dispatch 2024-04-23T10:26:39.083 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 23 10:26:39 smithi119 ceph-mon[32608]: from='client.? 172.21.15.114:0/3918374212' entity='client.admin' cmd=[{"prefix": "fs set", "fs_name": "cephfs", "var": "balance_automate", "val": "true"}]: dispatch 2024-04-23T10:26:39.083 INFO:journalctl@ceph.mgr.y.smithi119.stdout:Apr 23 10:26:38 smithi119 ceph-aa240572-0159-11ef-bc93-c7b262605968-mgr-y[34019]: 2024-04-23T10:26:38.784+0000 7f0f8532c200 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2024-04-23T10:26:39.134 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:39.133+0000 7fda9cccf640 1 -- 172.21.15.114:0/3831335176 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fda98107c90 msgr2=0x7fda9810a0e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T10:26:39.134 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:39.133+0000 7fda9cccf640 1 --2- 172.21.15.114:0/3831335176 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fda98107c90 0x7fda9810a0e0 secure :-1 s=READY pgs=413 cs=0 l=1 rev1=1 crypto rx=0x7fda90009ca0 tx=0x7fda9002a8c0 comp rx=0 tx=0).stop 2024-04-23T10:26:39.134 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:39.133+0000 7fda9cccf640 1 -- 172.21.15.114:0/3831335176 shutdown_connections 2024-04-23T10:26:39.134 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:39.133+0000 7fda9cccf640 1 --2- 172.21.15.114:0/3831335176 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fda98107c90 0x7fda9810a0e0 unknown :-1 s=CLOSED pgs=413 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:26:39.134 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:39.133+0000 7fda9cccf640 1 --2- 172.21.15.114:0/3831335176 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fda98105360 0x7fda98107750 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:26:39.134 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:39.133+0000 7fda9cccf640 1 --2- 172.21.15.114:0/3831335176 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fda98102a30 0x7fda98104e20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:26:39.134 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:39.133+0000 7fda9cccf640 1 -- 172.21.15.114:0/3831335176 >> 172.21.15.114:0/3831335176 conn(0x7fda980fc420 msgr2=0x7fda980fe860 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T10:26:39.134 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:39.133+0000 7fda9cccf640 1 -- 172.21.15.114:0/3831335176 shutdown_connections 2024-04-23T10:26:39.135 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:39.133+0000 7fda9cccf640 1 -- 172.21.15.114:0/3831335176 wait complete. 2024-04-23T10:26:39.135 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:39.134+0000 7fda9cccf640 1 Processor -- start 2024-04-23T10:26:39.135 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:39.134+0000 7fda9cccf640 1 -- start start 2024-04-23T10:26:39.135 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:39.134+0000 7fda9cccf640 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fda98102a30 0x7fda98198120 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T10:26:39.135 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:39.134+0000 7fda9cccf640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fda98105360 0x7fda98198660 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T10:26:39.135 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:39.134+0000 7fda9cccf640 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fda98107c90 0x7fda9819f6d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T10:26:39.135 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:39.134+0000 7fda9cccf640 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x7fda9810db40 con 0x7fda98105360 2024-04-23T10:26:39.135 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:39.134+0000 7fda9cccf640 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 -- 0x7fda9810d9c0 con 0x7fda98102a30 2024-04-23T10:26:39.135 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:39.134+0000 7fda9cccf640 1 -- --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_getmap magic: 0 -- 0x7fda9810dcc0 con 0x7fda98107c90 2024-04-23T10:26:39.135 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:39.134+0000 7fda95d9b640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fda98105360 0x7fda98198660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T10:26:39.135 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:39.134+0000 7fda95d9b640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fda98105360 0x7fda98198660 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.114:40586/0 (socket says 172.21.15.114:40586) 2024-04-23T10:26:39.135 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:39.134+0000 7fda95d9b640 1 -- 172.21.15.114:0/163687756 learned_addr learned my addr 172.21.15.114:0/163687756 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-04-23T10:26:39.135 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:39.134+0000 7fda9659c640 1 --2- 172.21.15.114:0/163687756 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fda98102a30 0x7fda98198120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T10:26:39.136 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:39.134+0000 7fda96d9d640 1 --2- 172.21.15.114:0/163687756 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fda98107c90 0x7fda9819f6d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T10:26:39.136 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:39.134+0000 7fda95d9b640 1 -- 172.21.15.114:0/163687756 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fda98107c90 msgr2=0x7fda9819f6d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T10:26:39.136 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:39.134+0000 7fda95d9b640 1 --2- 172.21.15.114:0/163687756 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fda98107c90 0x7fda9819f6d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:26:39.136 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:39.134+0000 7fda95d9b640 1 -- 172.21.15.114:0/163687756 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fda98102a30 msgr2=0x7fda98198120 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T10:26:39.136 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:39.134+0000 7fda95d9b640 1 --2- 172.21.15.114:0/163687756 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fda98102a30 0x7fda98198120 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:26:39.136 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:39.134+0000 7fda95d9b640 1 -- 172.21.15.114:0/163687756 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fda9819fd40 con 0x7fda98105360 2024-04-23T10:26:39.136 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:39.134+0000 7fda95d9b640 1 --2- 172.21.15.114:0/163687756 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fda98105360 0x7fda98198660 secure :-1 s=READY pgs=414 cs=0 l=1 rev1=1 crypto rx=0x7fda8800e9f0 tx=0x7fda8800eeb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T10:26:39.136 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:39.134+0000 7fda877fe640 1 -- 172.21.15.114:0/163687756 <== mon.0 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fda8800ccc0 con 0x7fda98105360 2024-04-23T10:26:39.136 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:39.135+0000 7fda9cccf640 1 -- 172.21.15.114:0/163687756 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fda9819ffd0 con 0x7fda98105360 2024-04-23T10:26:39.136 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:39.135+0000 7fda877fe640 1 -- 172.21.15.114:0/163687756 <== mon.0 v2:172.21.15.114:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fda880054d0 con 0x7fda98105360 2024-04-23T10:26:39.136 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:39.135+0000 7fda877fe640 1 -- 172.21.15.114:0/163687756 <== mon.0 v2:172.21.15.114:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fda88010430 con 0x7fda98105360 2024-04-23T10:26:39.136 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:39.135+0000 7fda9cccf640 1 -- 172.21.15.114:0/163687756 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fda981a04f0 con 0x7fda98105360 2024-04-23T10:26:39.137 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:39.135+0000 7fda9cccf640 1 -- 172.21.15.114:0/163687756 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fda980fdf10 con 0x7fda98105360 2024-04-23T10:26:39.137 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:39.137+0000 7fda877fe640 1 -- 172.21.15.114:0/163687756 <== mon.0 v2:172.21.15.114:3300/0 4 ==== mgrmap(e 19) ==== 138664+0+0 (secure 0 0 0) 0x7fda880046c0 con 0x7fda98105360 2024-04-23T10:26:39.137 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:39.137+0000 7fda877fe640 1 --2- 172.21.15.114:0/163687756 >> [v2:172.21.15.114:6800/465333808,v1:172.21.15.114:6801/465333808] conn(0x7fda800a3ca0 0x7fda800a6160 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T10:26:39.137 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:39.137+0000 7fda9659c640 1 -- 172.21.15.114:0/163687756 >> [v2:172.21.15.114:6800/465333808,v1:172.21.15.114:6801/465333808] conn(0x7fda800a3ca0 msgr2=0x7fda800a6160 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.114:6800/465333808 2024-04-23T10:26:39.138 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:39.137+0000 7fda877fe640 1 -- 172.21.15.114:0/163687756 <== mon.0 v2:172.21.15.114:3300/0 5 ==== osd_map(89..89 src has 1..89) ==== 8269+0+0 (secure 0 0 0) 0x7fda880d0950 con 0x7fda98105360 2024-04-23T10:26:39.138 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:39.138+0000 7fda9659c640 1 --2- 172.21.15.114:0/163687756 >> [v2:172.21.15.114:6800/465333808,v1:172.21.15.114:6801/465333808] conn(0x7fda800a3ca0 0x7fda800a6160 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-04-23T10:26:39.140 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:39.141+0000 7fda877fe640 1 -- 172.21.15.114:0/163687756 <== mon.0 v2:172.21.15.114:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7fda88053ff0 con 0x7fda98105360 2024-04-23T10:26:39.338 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:39.339+0000 7fda9659c640 1 -- 172.21.15.114:0/163687756 >> [v2:172.21.15.114:6800/465333808,v1:172.21.15.114:6801/465333808] conn(0x7fda800a3ca0 msgr2=0x7fda800a6160 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.114:6800/465333808 2024-04-23T10:26:39.338 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:39.339+0000 7fda9659c640 1 --2- 172.21.15.114:0/163687756 >> [v2:172.21.15.114:6800/465333808,v1:172.21.15.114:6801/465333808] conn(0x7fda800a3ca0 0x7fda800a6160 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-04-23T10:26:39.346 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:39.347+0000 7fda9cccf640 1 -- 172.21.15.114:0/163687756 --> [v2:172.21.15.114:6800/465333808,v1:172.21.15.114:6801/465333808] -- mgr_command(tid 0: {"prefix": "fs snap-schedule add", "fs": "cephfs", "path": "/", "snap_schedule": "1m", "target": ["mon-mgr", ""]}) -- 0x7fda98062390 con 0x7fda800a3ca0 2024-04-23T10:26:39.434 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Apr 23 10:26:39 smithi114 ceph-aa240572-0159-11ef-bc93-c7b262605968-mgr-x[29490]: 2024-04-23T10:26:39.123+0000 7f21d5288200 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2024-04-23T10:26:39.434 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Apr 23 10:26:39 smithi114 ceph-aa240572-0159-11ef-bc93-c7b262605968-mgr-x[29490]: 2024-04-23T10:26:39.434+0000 7f21d5288200 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2024-04-23T10:26:39.491 INFO:journalctl@ceph.mgr.y.smithi119.stdout:Apr 23 10:26:39 smithi119 ceph-aa240572-0159-11ef-bc93-c7b262605968-mgr-y[34019]: 2024-04-23T10:26:39.084+0000 7f0f8532c200 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2024-04-23T10:26:39.578 INFO:journalctl@ceph.mgr.z.smithi204.stdout:Apr 23 10:26:39 smithi204 ceph-aa240572-0159-11ef-bc93-c7b262605968-mgr-z[34493]: 2024-04-23T10:26:39.122+0000 7f39c4768200 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2024-04-23T10:26:39.738 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:39.740+0000 7fda9659c640 1 -- 172.21.15.114:0/163687756 >> [v2:172.21.15.114:6800/465333808,v1:172.21.15.114:6801/465333808] conn(0x7fda800a3ca0 msgr2=0x7fda800a6160 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.114:6800/465333808 2024-04-23T10:26:39.739 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:39.740+0000 7fda9659c640 1 --2- 172.21.15.114:0/163687756 >> [v2:172.21.15.114:6800/465333808,v1:172.21.15.114:6801/465333808] conn(0x7fda800a3ca0 0x7fda800a6160 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2024-04-23T10:26:39.992 INFO:journalctl@ceph.mgr.y.smithi119.stdout:Apr 23 10:26:39 smithi119 ceph-aa240572-0159-11ef-bc93-c7b262605968-mgr-y[34019]: 2024-04-23T10:26:39.686+0000 7f0f8532c200 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2024-04-23T10:26:39.992 INFO:journalctl@ceph.mgr.y.smithi119.stdout:Apr 23 10:26:39 smithi119 ceph-aa240572-0159-11ef-bc93-c7b262605968-mgr-y[34019]: 2024-04-23T10:26:39.899+0000 7f0f8532c200 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2024-04-23T10:26:40.000 INFO:journalctl@ceph.mgr.z.smithi204.stdout:Apr 23 10:26:39 smithi204 ceph-aa240572-0159-11ef-bc93-c7b262605968-mgr-z[34493]: 2024-04-23T10:26:39.724+0000 7f39c4768200 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2024-04-23T10:26:40.000 INFO:journalctl@ceph.mgr.z.smithi204.stdout:Apr 23 10:26:39 smithi204 ceph-aa240572-0159-11ef-bc93-c7b262605968-mgr-z[34493]: 2024-04-23T10:26:39.941+0000 7f39c4768200 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2024-04-23T10:26:40.010 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:40.012+0000 7fda877fe640 1 -- 172.21.15.114:0/163687756 <== mon.0 v2:172.21.15.114:3300/0 7 ==== mgrmap(e 20) ==== 138664+0+0 (secure 0 0 0) 0x7fda88099280 con 0x7fda98105360 2024-04-23T10:26:40.254 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Apr 23 10:26:40 smithi114 ceph-aa240572-0159-11ef-bc93-c7b262605968-mgr-x[29490]: 2024-04-23T10:26:40.046+0000 7f21d5288200 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2024-04-23T10:26:40.254 INFO:journalctl@ceph.mon.a.smithi114.stdout:Apr 23 10:26:40 smithi114 ceph-mon[29255]: from='client.? 172.21.15.114:0/3918374212' entity='client.admin' cmd='[{"prefix": "fs set", "fs_name": "cephfs", "var": "balance_automate", "val": "true"}]': finished 2024-04-23T10:26:40.254 INFO:journalctl@ceph.mon.a.smithi114.stdout:Apr 23 10:26:40 smithi114 ceph-mon[29255]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=e=up:active} 3 up:standby-replay 6 up:standby 2024-04-23T10:26:40.255 INFO:journalctl@ceph.mon.a.smithi114.stdout:Apr 23 10:26:40 smithi114 ceph-mon[29255]: Standby manager daemon y restarted 2024-04-23T10:26:40.255 INFO:journalctl@ceph.mon.a.smithi114.stdout:Apr 23 10:26:40 smithi114 ceph-mon[29255]: Standby manager daemon y started 2024-04-23T10:26:40.255 INFO:journalctl@ceph.mon.a.smithi114.stdout:Apr 23 10:26:40 smithi114 ceph-mon[29255]: from='mgr.? 172.21.15.119:0/652968524' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/y/crt"}]: dispatch 2024-04-23T10:26:40.255 INFO:journalctl@ceph.mon.a.smithi114.stdout:Apr 23 10:26:40 smithi114 ceph-mon[29255]: from='mgr.? 172.21.15.119:0/652968524' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-04-23T10:26:40.255 INFO:journalctl@ceph.mon.a.smithi114.stdout:Apr 23 10:26:40 smithi114 ceph-mon[29255]: from='mgr.? 172.21.15.119:0/652968524' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/y/key"}]: dispatch 2024-04-23T10:26:40.255 INFO:journalctl@ceph.mon.a.smithi114.stdout:Apr 23 10:26:40 smithi114 ceph-mon[29255]: from='mgr.? 172.21.15.119:0/652968524' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-04-23T10:26:40.255 INFO:journalctl@ceph.mon.a.smithi114.stdout:Apr 23 10:26:40 smithi114 ceph-mon[29255]: Standby manager daemon z restarted 2024-04-23T10:26:40.255 INFO:journalctl@ceph.mon.a.smithi114.stdout:Apr 23 10:26:40 smithi114 ceph-mon[29255]: Standby manager daemon z started 2024-04-23T10:26:40.255 INFO:journalctl@ceph.mon.a.smithi114.stdout:Apr 23 10:26:40 smithi114 ceph-mon[29255]: from='mgr.? 172.21.15.204:0/266345556' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/z/crt"}]: dispatch 2024-04-23T10:26:40.255 INFO:journalctl@ceph.mon.a.smithi114.stdout:Apr 23 10:26:40 smithi114 ceph-mon[29255]: from='mgr.? 172.21.15.204:0/266345556' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-04-23T10:26:40.255 INFO:journalctl@ceph.mon.a.smithi114.stdout:Apr 23 10:26:40 smithi114 ceph-mon[29255]: from='mgr.? 172.21.15.204:0/266345556' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/z/key"}]: dispatch 2024-04-23T10:26:40.255 INFO:journalctl@ceph.mon.a.smithi114.stdout:Apr 23 10:26:40 smithi114 ceph-mon[29255]: from='mgr.? 172.21.15.204:0/266345556' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-04-23T10:26:40.281 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:40.282+0000 7fda877fe640 1 -- 172.21.15.114:0/163687756 <== mon.0 v2:172.21.15.114:3300/0 8 ==== mgrmap(e 21) ==== 138331+0+0 (secure 0 0 0) 0x7fda8809ce70 con 0x7fda98105360 2024-04-23T10:26:40.281 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:40.282+0000 7fda877fe640 1 -- 172.21.15.114:0/163687756 >> [v2:172.21.15.114:6800/465333808,v1:172.21.15.114:6801/465333808] conn(0x7fda800a3ca0 msgr2=0x7fda800a6160 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-04-23T10:26:40.281 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:40.282+0000 7fda877fe640 1 --2- 172.21.15.114:0/163687756 >> [v2:172.21.15.114:6800/465333808,v1:172.21.15.114:6801/465333808] conn(0x7fda800a3ca0 0x7fda800a6160 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:26:40.328 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 23 10:26:40 smithi204 ceph-mon[32649]: from='client.? 172.21.15.114:0/3918374212' entity='client.admin' cmd='[{"prefix": "fs set", "fs_name": "cephfs", "var": "balance_automate", "val": "true"}]': finished 2024-04-23T10:26:40.328 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 23 10:26:40 smithi204 ceph-mon[32649]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=e=up:active} 3 up:standby-replay 6 up:standby 2024-04-23T10:26:40.328 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 23 10:26:40 smithi204 ceph-mon[32649]: Standby manager daemon y restarted 2024-04-23T10:26:40.328 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 23 10:26:40 smithi204 ceph-mon[32649]: Standby manager daemon y started 2024-04-23T10:26:40.328 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 23 10:26:40 smithi204 ceph-mon[32649]: from='mgr.? 172.21.15.119:0/652968524' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/y/crt"}]: dispatch 2024-04-23T10:26:40.328 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 23 10:26:40 smithi204 ceph-mon[32649]: from='mgr.? 172.21.15.119:0/652968524' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-04-23T10:26:40.328 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 23 10:26:40 smithi204 ceph-mon[32649]: from='mgr.? 172.21.15.119:0/652968524' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/y/key"}]: dispatch 2024-04-23T10:26:40.328 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 23 10:26:40 smithi204 ceph-mon[32649]: from='mgr.? 172.21.15.119:0/652968524' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-04-23T10:26:40.329 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 23 10:26:40 smithi204 ceph-mon[32649]: Standby manager daemon z restarted 2024-04-23T10:26:40.329 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 23 10:26:40 smithi204 ceph-mon[32649]: Standby manager daemon z started 2024-04-23T10:26:40.329 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 23 10:26:40 smithi204 ceph-mon[32649]: from='mgr.? 172.21.15.204:0/266345556' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/z/crt"}]: dispatch 2024-04-23T10:26:40.329 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 23 10:26:40 smithi204 ceph-mon[32649]: from='mgr.? 172.21.15.204:0/266345556' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-04-23T10:26:40.329 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 23 10:26:40 smithi204 ceph-mon[32649]: from='mgr.? 172.21.15.204:0/266345556' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/z/key"}]: dispatch 2024-04-23T10:26:40.329 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 23 10:26:40 smithi204 ceph-mon[32649]: from='mgr.? 172.21.15.204:0/266345556' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-04-23T10:26:40.412 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 23 10:26:40 smithi119 ceph-mon[32608]: from='client.? 172.21.15.114:0/3918374212' entity='client.admin' cmd='[{"prefix": "fs set", "fs_name": "cephfs", "var": "balance_automate", "val": "true"}]': finished 2024-04-23T10:26:40.412 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 23 10:26:40 smithi119 ceph-mon[32608]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=e=up:active} 3 up:standby-replay 6 up:standby 2024-04-23T10:26:40.412 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 23 10:26:40 smithi119 ceph-mon[32608]: Standby manager daemon y restarted 2024-04-23T10:26:40.412 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 23 10:26:40 smithi119 ceph-mon[32608]: Standby manager daemon y started 2024-04-23T10:26:40.412 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 23 10:26:40 smithi119 ceph-mon[32608]: from='mgr.? 172.21.15.119:0/652968524' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/y/crt"}]: dispatch 2024-04-23T10:26:40.413 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 23 10:26:40 smithi119 ceph-mon[32608]: from='mgr.? 172.21.15.119:0/652968524' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-04-23T10:26:40.413 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 23 10:26:40 smithi119 ceph-mon[32608]: from='mgr.? 172.21.15.119:0/652968524' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/y/key"}]: dispatch 2024-04-23T10:26:40.413 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 23 10:26:40 smithi119 ceph-mon[32608]: from='mgr.? 172.21.15.119:0/652968524' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-04-23T10:26:40.413 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 23 10:26:40 smithi119 ceph-mon[32608]: Standby manager daemon z restarted 2024-04-23T10:26:40.413 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 23 10:26:40 smithi119 ceph-mon[32608]: Standby manager daemon z started 2024-04-23T10:26:40.413 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 23 10:26:40 smithi119 ceph-mon[32608]: from='mgr.? 172.21.15.204:0/266345556' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/z/crt"}]: dispatch 2024-04-23T10:26:40.413 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 23 10:26:40 smithi119 ceph-mon[32608]: from='mgr.? 172.21.15.204:0/266345556' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-04-23T10:26:40.413 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 23 10:26:40 smithi119 ceph-mon[32608]: from='mgr.? 172.21.15.204:0/266345556' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/z/key"}]: dispatch 2024-04-23T10:26:40.413 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 23 10:26:40 smithi119 ceph-mon[32608]: from='mgr.? 172.21.15.204:0/266345556' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-04-23T10:26:40.522 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Apr 23 10:26:40 smithi114 ceph-aa240572-0159-11ef-bc93-c7b262605968-mgr-x[29490]: 2024-04-23T10:26:40.270+0000 7f21d5288200 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2024-04-23T10:26:40.522 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Apr 23 10:26:40 smithi114 ceph-aa240572-0159-11ef-bc93-c7b262605968-mgr-x[29490]: 2024-04-23T10:26:40.515+0000 7f2137c02700 -1 client.0 error registering admin socket command: (17) File exists 2024-04-23T10:26:40.522 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Apr 23 10:26:40 smithi114 ceph-aa240572-0159-11ef-bc93-c7b262605968-mgr-x[29490]: 2024-04-23T10:26:40.515+0000 7f2137c02700 -1 client.0 error registering admin socket command: (17) File exists 2024-04-23T10:26:40.522 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Apr 23 10:26:40 smithi114 ceph-aa240572-0159-11ef-bc93-c7b262605968-mgr-x[29490]: 2024-04-23T10:26:40.515+0000 7f2137c02700 -1 client.0 error registering admin socket command: (17) File exists 2024-04-23T10:26:40.522 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Apr 23 10:26:40 smithi114 ceph-aa240572-0159-11ef-bc93-c7b262605968-mgr-x[29490]: 2024-04-23T10:26:40.515+0000 7f2137c02700 -1 client.0 error registering admin socket command: (17) File exists 2024-04-23T10:26:40.522 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Apr 23 10:26:40 smithi114 ceph-aa240572-0159-11ef-bc93-c7b262605968-mgr-x[29490]: 2024-04-23T10:26:40.515+0000 7f2137c02700 -1 client.0 error registering admin socket command: (17) File exists 2024-04-23T10:26:40.522 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Apr 23 10:26:40 smithi114 ceph-aa240572-0159-11ef-bc93-c7b262605968-mgr-x[29490]: 2024-04-23T10:26:40.515+0000 7f2134bfc700 -1 client.0 error registering admin socket command: (17) File exists 2024-04-23T10:26:40.523 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Apr 23 10:26:40 smithi114 ceph-aa240572-0159-11ef-bc93-c7b262605968-mgr-x[29490]: 2024-04-23T10:26:40.515+0000 7f2134bfc700 -1 client.0 error registering admin socket command: (17) File exists 2024-04-23T10:26:40.523 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Apr 23 10:26:40 smithi114 ceph-aa240572-0159-11ef-bc93-c7b262605968-mgr-x[29490]: 2024-04-23T10:26:40.515+0000 7f2134bfc700 -1 client.0 error registering admin socket command: (17) File exists 2024-04-23T10:26:40.523 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Apr 23 10:26:40 smithi114 ceph-aa240572-0159-11ef-bc93-c7b262605968-mgr-x[29490]: 2024-04-23T10:26:40.515+0000 7f2134bfc700 -1 client.0 error registering admin socket command: (17) File exists 2024-04-23T10:26:40.523 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Apr 23 10:26:40 smithi114 ceph-aa240572-0159-11ef-bc93-c7b262605968-mgr-x[29490]: 2024-04-23T10:26:40.515+0000 7f2134bfc700 -1 client.0 error registering admin socket command: (17) File exists 2024-04-23T10:26:41.289 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:41.290+0000 7fda877fe640 1 -- 172.21.15.114:0/163687756 <== mon.0 v2:172.21.15.114:3300/0 9 ==== mgrmap(e 22) ==== 138458+0+0 (secure 0 0 0) 0x7fda880ce480 con 0x7fda98105360 2024-04-23T10:26:41.290 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:41.291+0000 7fda877fe640 1 --2- 172.21.15.114:0/163687756 >> [v2:172.21.15.114:6800/1395693466,v1:172.21.15.114:6801/1395693466] conn(0x7fda80035480 0x7fda80035860 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T10:26:41.290 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:41.291+0000 7fda877fe640 1 -- 172.21.15.114:0/163687756 --> [v2:172.21.15.114:6800/1395693466,v1:172.21.15.114:6801/1395693466] -- mgr_command(tid 0: {"prefix": "fs snap-schedule add", "fs": "cephfs", "path": "/", "snap_schedule": "1m", "target": ["mon-mgr", ""]}) -- 0x7fda8801cec0 con 0x7fda80035480 2024-04-23T10:26:41.292 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:41.293+0000 7fda9659c640 1 --2- 172.21.15.114:0/163687756 >> [v2:172.21.15.114:6800/1395693466,v1:172.21.15.114:6801/1395693466] conn(0x7fda80035480 0x7fda80035860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T10:26:41.294 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:41.295+0000 7fda9659c640 1 --2- 172.21.15.114:0/163687756 >> [v2:172.21.15.114:6800/1395693466,v1:172.21.15.114:6801/1395693466] conn(0x7fda80035480 0x7fda80035860 secure :-1 s=READY pgs=30 cs=0 l=1 rev1=1 crypto rx=0x7fda8c0097f0 tx=0x7fda8c005f90 comp rx=0 tx=0).ready entity=mgr.15291 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T10:26:41.328 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 23 10:26:41 smithi204 ceph-mon[32649]: mgrmap e20: x(active, since 8m), standbys: z, y 2024-04-23T10:26:41.328 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 23 10:26:41 smithi204 ceph-mon[32649]: Active manager daemon x restarted 2024-04-23T10:26:41.328 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 23 10:26:41 smithi204 ceph-mon[32649]: Activating manager daemon x 2024-04-23T10:26:41.328 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 23 10:26:41 smithi204 ceph-mon[32649]: osdmap e90: 12 total, 12 up, 12 in 2024-04-23T10:26:41.328 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 23 10:26:41 smithi204 ceph-mon[32649]: mgrmap e21: x(active, starting, since 0.0122461s), standbys: z, y 2024-04-23T10:26:41.328 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 23 10:26:41 smithi204 ceph-mon[32649]: from='mgr.15291 172.21.15.114:0/1095789187' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-04-23T10:26:41.329 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 23 10:26:41 smithi204 ceph-mon[32649]: from='mgr.15291 172.21.15.114:0/1095789187' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-04-23T10:26:41.329 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 23 10:26:41 smithi204 ceph-mon[32649]: from='mgr.15291 172.21.15.114:0/1095789187' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-04-23T10:26:41.329 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 23 10:26:41 smithi204 ceph-mon[32649]: from='mgr.15291 172.21.15.114:0/1095789187' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "j"}]: dispatch 2024-04-23T10:26:41.329 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 23 10:26:41 smithi204 ceph-mon[32649]: from='mgr.15291 172.21.15.114:0/1095789187' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "d"}]: dispatch 2024-04-23T10:26:41.329 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 23 10:26:41 smithi204 ceph-mon[32649]: from='mgr.15291 172.21.15.114:0/1095789187' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "g"}]: dispatch 2024-04-23T10:26:41.329 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 23 10:26:41 smithi204 ceph-mon[32649]: from='mgr.15291 172.21.15.114:0/1095789187' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "a"}]: dispatch 2024-04-23T10:26:41.329 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 23 10:26:41 smithi204 ceph-mon[32649]: from='mgr.15291 172.21.15.114:0/1095789187' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "l"}]: dispatch 2024-04-23T10:26:41.329 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 23 10:26:41 smithi204 ceph-mon[32649]: from='mgr.15291 172.21.15.114:0/1095789187' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "f"}]: dispatch 2024-04-23T10:26:41.329 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 23 10:26:41 smithi204 ceph-mon[32649]: from='mgr.15291 172.21.15.114:0/1095789187' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "c"}]: dispatch 2024-04-23T10:26:41.329 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 23 10:26:41 smithi204 ceph-mon[32649]: from='mgr.15291 172.21.15.114:0/1095789187' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "k"}]: dispatch 2024-04-23T10:26:41.329 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 23 10:26:41 smithi204 ceph-mon[32649]: from='mgr.15291 172.21.15.114:0/1095789187' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "i"}]: dispatch 2024-04-23T10:26:41.329 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 23 10:26:41 smithi204 ceph-mon[32649]: from='mgr.15291 172.21.15.114:0/1095789187' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "e"}]: dispatch 2024-04-23T10:26:41.329 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 23 10:26:41 smithi204 ceph-mon[32649]: from='mgr.15291 172.21.15.114:0/1095789187' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "h"}]: dispatch 2024-04-23T10:26:41.330 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 23 10:26:41 smithi204 ceph-mon[32649]: from='mgr.15291 172.21.15.114:0/1095789187' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "b"}]: dispatch 2024-04-23T10:26:41.330 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 23 10:26:41 smithi204 ceph-mon[32649]: from='mgr.15291 172.21.15.114:0/1095789187' entity='mgr.x' cmd=[{"prefix": "mgr metadata", "who": "x", "id": "x"}]: dispatch 2024-04-23T10:26:41.330 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 23 10:26:41 smithi204 ceph-mon[32649]: from='mgr.15291 172.21.15.114:0/1095789187' entity='mgr.x' cmd=[{"prefix": "mgr metadata", "who": "z", "id": "z"}]: dispatch 2024-04-23T10:26:41.330 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 23 10:26:41 smithi204 ceph-mon[32649]: from='mgr.15291 172.21.15.114:0/1095789187' entity='mgr.x' cmd=[{"prefix": "mgr metadata", "who": "y", "id": "y"}]: dispatch 2024-04-23T10:26:41.330 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 23 10:26:41 smithi204 ceph-mon[32649]: from='mgr.15291 172.21.15.114:0/1095789187' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-04-23T10:26:41.330 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 23 10:26:41 smithi204 ceph-mon[32649]: from='mgr.15291 172.21.15.114:0/1095789187' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-04-23T10:26:41.330 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 23 10:26:41 smithi204 ceph-mon[32649]: from='mgr.15291 172.21.15.114:0/1095789187' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-04-23T10:26:41.330 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 23 10:26:41 smithi204 ceph-mon[32649]: from='mgr.15291 172.21.15.114:0/1095789187' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-04-23T10:26:41.330 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 23 10:26:41 smithi204 ceph-mon[32649]: from='mgr.15291 172.21.15.114:0/1095789187' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-04-23T10:26:41.330 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 23 10:26:41 smithi204 ceph-mon[32649]: from='mgr.15291 172.21.15.114:0/1095789187' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-04-23T10:26:41.330 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 23 10:26:41 smithi204 ceph-mon[32649]: from='mgr.15291 172.21.15.114:0/1095789187' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-04-23T10:26:41.330 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 23 10:26:41 smithi204 ceph-mon[32649]: from='mgr.15291 172.21.15.114:0/1095789187' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-04-23T10:26:41.331 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 23 10:26:41 smithi204 ceph-mon[32649]: from='mgr.15291 172.21.15.114:0/1095789187' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 8}]: dispatch 2024-04-23T10:26:41.331 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 23 10:26:41 smithi204 ceph-mon[32649]: from='mgr.15291 172.21.15.114:0/1095789187' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 9}]: dispatch 2024-04-23T10:26:41.331 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 23 10:26:41 smithi204 ceph-mon[32649]: from='mgr.15291 172.21.15.114:0/1095789187' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 10}]: dispatch 2024-04-23T10:26:41.331 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 23 10:26:41 smithi204 ceph-mon[32649]: from='mgr.15291 172.21.15.114:0/1095789187' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 11}]: dispatch 2024-04-23T10:26:41.331 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 23 10:26:41 smithi204 ceph-mon[32649]: from='mgr.15291 172.21.15.114:0/1095789187' entity='mgr.x' cmd=[{"prefix": "mds metadata"}]: dispatch 2024-04-23T10:26:41.331 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 23 10:26:41 smithi204 ceph-mon[32649]: from='mgr.15291 172.21.15.114:0/1095789187' entity='mgr.x' cmd=[{"prefix": "osd metadata"}]: dispatch 2024-04-23T10:26:41.331 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 23 10:26:41 smithi204 ceph-mon[32649]: from='mgr.15291 172.21.15.114:0/1095789187' entity='mgr.x' cmd=[{"prefix": "mon metadata"}]: dispatch 2024-04-23T10:26:41.331 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 23 10:26:41 smithi204 ceph-mon[32649]: Manager daemon x is now available 2024-04-23T10:26:41.331 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 23 10:26:41 smithi204 ceph-mon[32649]: from='mgr.15291 172.21.15.114:0/1095789187' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-23T10:26:41.331 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 23 10:26:41 smithi204 ceph-mon[32649]: from='mgr.15291 172.21.15.114:0/1095789187' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2024-04-23T10:26:41.331 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 23 10:26:41 smithi204 ceph-mon[32649]: from='mgr.15291 172.21.15.114:0/1095789187' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2024-04-23T10:26:41.358 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:41.359+0000 7fda877fe640 1 -- 172.21.15.114:0/163687756 <== mgr.15291 v2:172.21.15.114:6800/1395693466 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+24 (secure 0 0 0) 0x7fda8801cec0 con 0x7fda80035480 2024-04-23T10:26:41.358 INFO:teuthology.orchestra.run.smithi114.stdout:Schedule set for path / 2024-04-23T10:26:41.360 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:41.361+0000 7fda9cccf640 1 -- 172.21.15.114:0/163687756 >> [v2:172.21.15.114:6800/1395693466,v1:172.21.15.114:6801/1395693466] conn(0x7fda80035480 msgr2=0x7fda80035860 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T10:26:41.361 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:41.361+0000 7fda9cccf640 1 --2- 172.21.15.114:0/163687756 >> [v2:172.21.15.114:6800/1395693466,v1:172.21.15.114:6801/1395693466] conn(0x7fda80035480 0x7fda80035860 secure :-1 s=READY pgs=30 cs=0 l=1 rev1=1 crypto rx=0x7fda8c0097f0 tx=0x7fda8c005f90 comp rx=0 tx=0).stop 2024-04-23T10:26:41.361 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:41.361+0000 7fda9cccf640 1 -- 172.21.15.114:0/163687756 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fda98105360 msgr2=0x7fda98198660 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T10:26:41.361 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:41.361+0000 7fda9cccf640 1 --2- 172.21.15.114:0/163687756 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fda98105360 0x7fda98198660 secure :-1 s=READY pgs=414 cs=0 l=1 rev1=1 crypto rx=0x7fda8800e9f0 tx=0x7fda8800eeb0 comp rx=0 tx=0).stop 2024-04-23T10:26:41.361 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:41.361+0000 7fda96d9d640 1 -- 172.21.15.114:0/163687756 reap_dead start 2024-04-23T10:26:41.361 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:41.361+0000 7fda9cccf640 1 -- 172.21.15.114:0/163687756 shutdown_connections 2024-04-23T10:26:41.361 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:41.361+0000 7fda9cccf640 1 -- 172.21.15.114:0/163687756 >> 172.21.15.114:0/163687756 conn(0x7fda980fc420 msgr2=0x7fda98103450 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T10:26:41.361 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:41.361+0000 7fda9cccf640 1 -- 172.21.15.114:0/163687756 shutdown_connections 2024-04-23T10:26:41.361 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:41.361+0000 7fda9cccf640 1 -- 172.21.15.114:0/163687756 wait complete. 2024-04-23T10:26:41.371 INFO:journalctl@ceph.mon.a.smithi114.stdout:Apr 23 10:26:41 smithi114 ceph-mon[29255]: mgrmap e20: x(active, since 8m), standbys: z, y 2024-04-23T10:26:41.371 INFO:journalctl@ceph.mon.a.smithi114.stdout:Apr 23 10:26:41 smithi114 ceph-mon[29255]: Active manager daemon x restarted 2024-04-23T10:26:41.371 INFO:journalctl@ceph.mon.a.smithi114.stdout:Apr 23 10:26:41 smithi114 ceph-mon[29255]: Activating manager daemon x 2024-04-23T10:26:41.371 INFO:journalctl@ceph.mon.a.smithi114.stdout:Apr 23 10:26:41 smithi114 ceph-mon[29255]: osdmap e90: 12 total, 12 up, 12 in 2024-04-23T10:26:41.371 INFO:journalctl@ceph.mon.a.smithi114.stdout:Apr 23 10:26:41 smithi114 ceph-mon[29255]: mgrmap e21: x(active, starting, since 0.0122461s), standbys: z, y 2024-04-23T10:26:41.371 INFO:journalctl@ceph.mon.a.smithi114.stdout:Apr 23 10:26:41 smithi114 ceph-mon[29255]: from='mgr.15291 172.21.15.114:0/1095789187' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-04-23T10:26:41.371 INFO:journalctl@ceph.mon.a.smithi114.stdout:Apr 23 10:26:41 smithi114 ceph-mon[29255]: from='mgr.15291 172.21.15.114:0/1095789187' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-04-23T10:26:41.371 INFO:journalctl@ceph.mon.a.smithi114.stdout:Apr 23 10:26:41 smithi114 ceph-mon[29255]: from='mgr.15291 172.21.15.114:0/1095789187' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-04-23T10:26:41.371 INFO:journalctl@ceph.mon.a.smithi114.stdout:Apr 23 10:26:41 smithi114 ceph-mon[29255]: from='mgr.15291 172.21.15.114:0/1095789187' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "j"}]: dispatch 2024-04-23T10:26:41.371 INFO:journalctl@ceph.mon.a.smithi114.stdout:Apr 23 10:26:41 smithi114 ceph-mon[29255]: from='mgr.15291 172.21.15.114:0/1095789187' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "d"}]: dispatch 2024-04-23T10:26:41.371 INFO:journalctl@ceph.mon.a.smithi114.stdout:Apr 23 10:26:41 smithi114 ceph-mon[29255]: from='mgr.15291 172.21.15.114:0/1095789187' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "g"}]: dispatch 2024-04-23T10:26:41.372 INFO:journalctl@ceph.mon.a.smithi114.stdout:Apr 23 10:26:41 smithi114 ceph-mon[29255]: from='mgr.15291 172.21.15.114:0/1095789187' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "a"}]: dispatch 2024-04-23T10:26:41.372 INFO:journalctl@ceph.mon.a.smithi114.stdout:Apr 23 10:26:41 smithi114 ceph-mon[29255]: from='mgr.15291 172.21.15.114:0/1095789187' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "l"}]: dispatch 2024-04-23T10:26:41.372 INFO:journalctl@ceph.mon.a.smithi114.stdout:Apr 23 10:26:41 smithi114 ceph-mon[29255]: from='mgr.15291 172.21.15.114:0/1095789187' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "f"}]: dispatch 2024-04-23T10:26:41.372 INFO:journalctl@ceph.mon.a.smithi114.stdout:Apr 23 10:26:41 smithi114 ceph-mon[29255]: from='mgr.15291 172.21.15.114:0/1095789187' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "c"}]: dispatch 2024-04-23T10:26:41.372 INFO:journalctl@ceph.mon.a.smithi114.stdout:Apr 23 10:26:41 smithi114 ceph-mon[29255]: from='mgr.15291 172.21.15.114:0/1095789187' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "k"}]: dispatch 2024-04-23T10:26:41.372 INFO:journalctl@ceph.mon.a.smithi114.stdout:Apr 23 10:26:41 smithi114 ceph-mon[29255]: from='mgr.15291 172.21.15.114:0/1095789187' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "i"}]: dispatch 2024-04-23T10:26:41.372 INFO:journalctl@ceph.mon.a.smithi114.stdout:Apr 23 10:26:41 smithi114 ceph-mon[29255]: from='mgr.15291 172.21.15.114:0/1095789187' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "e"}]: dispatch 2024-04-23T10:26:41.372 INFO:journalctl@ceph.mon.a.smithi114.stdout:Apr 23 10:26:41 smithi114 ceph-mon[29255]: from='mgr.15291 172.21.15.114:0/1095789187' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "h"}]: dispatch 2024-04-23T10:26:41.372 INFO:journalctl@ceph.mon.a.smithi114.stdout:Apr 23 10:26:41 smithi114 ceph-mon[29255]: from='mgr.15291 172.21.15.114:0/1095789187' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "b"}]: dispatch 2024-04-23T10:26:41.372 INFO:journalctl@ceph.mon.a.smithi114.stdout:Apr 23 10:26:41 smithi114 ceph-mon[29255]: from='mgr.15291 172.21.15.114:0/1095789187' entity='mgr.x' cmd=[{"prefix": "mgr metadata", "who": "x", "id": "x"}]: dispatch 2024-04-23T10:26:41.372 INFO:journalctl@ceph.mon.a.smithi114.stdout:Apr 23 10:26:41 smithi114 ceph-mon[29255]: from='mgr.15291 172.21.15.114:0/1095789187' entity='mgr.x' cmd=[{"prefix": "mgr metadata", "who": "z", "id": "z"}]: dispatch 2024-04-23T10:26:41.373 INFO:journalctl@ceph.mon.a.smithi114.stdout:Apr 23 10:26:41 smithi114 ceph-mon[29255]: from='mgr.15291 172.21.15.114:0/1095789187' entity='mgr.x' cmd=[{"prefix": "mgr metadata", "who": "y", "id": "y"}]: dispatch 2024-04-23T10:26:41.373 INFO:journalctl@ceph.mon.a.smithi114.stdout:Apr 23 10:26:41 smithi114 ceph-mon[29255]: from='mgr.15291 172.21.15.114:0/1095789187' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-04-23T10:26:41.373 INFO:journalctl@ceph.mon.a.smithi114.stdout:Apr 23 10:26:41 smithi114 ceph-mon[29255]: from='mgr.15291 172.21.15.114:0/1095789187' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-04-23T10:26:41.373 INFO:journalctl@ceph.mon.a.smithi114.stdout:Apr 23 10:26:41 smithi114 ceph-mon[29255]: from='mgr.15291 172.21.15.114:0/1095789187' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-04-23T10:26:41.373 INFO:journalctl@ceph.mon.a.smithi114.stdout:Apr 23 10:26:41 smithi114 ceph-mon[29255]: from='mgr.15291 172.21.15.114:0/1095789187' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-04-23T10:26:41.373 INFO:journalctl@ceph.mon.a.smithi114.stdout:Apr 23 10:26:41 smithi114 ceph-mon[29255]: from='mgr.15291 172.21.15.114:0/1095789187' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-04-23T10:26:41.373 INFO:journalctl@ceph.mon.a.smithi114.stdout:Apr 23 10:26:41 smithi114 ceph-mon[29255]: from='mgr.15291 172.21.15.114:0/1095789187' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-04-23T10:26:41.373 INFO:journalctl@ceph.mon.a.smithi114.stdout:Apr 23 10:26:41 smithi114 ceph-mon[29255]: from='mgr.15291 172.21.15.114:0/1095789187' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-04-23T10:26:41.373 INFO:journalctl@ceph.mon.a.smithi114.stdout:Apr 23 10:26:41 smithi114 ceph-mon[29255]: from='mgr.15291 172.21.15.114:0/1095789187' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-04-23T10:26:41.373 INFO:journalctl@ceph.mon.a.smithi114.stdout:Apr 23 10:26:41 smithi114 ceph-mon[29255]: from='mgr.15291 172.21.15.114:0/1095789187' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 8}]: dispatch 2024-04-23T10:26:41.373 INFO:journalctl@ceph.mon.a.smithi114.stdout:Apr 23 10:26:41 smithi114 ceph-mon[29255]: from='mgr.15291 172.21.15.114:0/1095789187' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 9}]: dispatch 2024-04-23T10:26:41.373 INFO:journalctl@ceph.mon.a.smithi114.stdout:Apr 23 10:26:41 smithi114 ceph-mon[29255]: from='mgr.15291 172.21.15.114:0/1095789187' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 10}]: dispatch 2024-04-23T10:26:41.373 INFO:journalctl@ceph.mon.a.smithi114.stdout:Apr 23 10:26:41 smithi114 ceph-mon[29255]: from='mgr.15291 172.21.15.114:0/1095789187' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 11}]: dispatch 2024-04-23T10:26:41.374 INFO:journalctl@ceph.mon.a.smithi114.stdout:Apr 23 10:26:41 smithi114 ceph-mon[29255]: from='mgr.15291 172.21.15.114:0/1095789187' entity='mgr.x' cmd=[{"prefix": "mds metadata"}]: dispatch 2024-04-23T10:26:41.374 INFO:journalctl@ceph.mon.a.smithi114.stdout:Apr 23 10:26:41 smithi114 ceph-mon[29255]: from='mgr.15291 172.21.15.114:0/1095789187' entity='mgr.x' cmd=[{"prefix": "osd metadata"}]: dispatch 2024-04-23T10:26:41.374 INFO:journalctl@ceph.mon.a.smithi114.stdout:Apr 23 10:26:41 smithi114 ceph-mon[29255]: from='mgr.15291 172.21.15.114:0/1095789187' entity='mgr.x' cmd=[{"prefix": "mon metadata"}]: dispatch 2024-04-23T10:26:41.374 INFO:journalctl@ceph.mon.a.smithi114.stdout:Apr 23 10:26:41 smithi114 ceph-mon[29255]: Manager daemon x is now available 2024-04-23T10:26:41.374 INFO:journalctl@ceph.mon.a.smithi114.stdout:Apr 23 10:26:41 smithi114 ceph-mon[29255]: from='mgr.15291 172.21.15.114:0/1095789187' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-23T10:26:41.374 INFO:journalctl@ceph.mon.a.smithi114.stdout:Apr 23 10:26:41 smithi114 ceph-mon[29255]: from='mgr.15291 172.21.15.114:0/1095789187' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2024-04-23T10:26:41.374 INFO:journalctl@ceph.mon.a.smithi114.stdout:Apr 23 10:26:41 smithi114 ceph-mon[29255]: from='mgr.15291 172.21.15.114:0/1095789187' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2024-04-23T10:26:41.375 DEBUG:teuthology.orchestra.run.smithi114:> sudo TESTDIR=/home/ubuntu/cephtest bash -c 'ceph fs snap-schedule retention add --fs=cephfs --path=/ --retention-spec-or-period=6m3h' 2024-04-23T10:26:41.481 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:41.479+0000 7f4c34b37640 1 -- 172.21.15.114:0/1748669748 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f4c30107d30 msgr2=0x7f4c3010a180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T10:26:41.481 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:41.479+0000 7f4c34b37640 1 --2- 172.21.15.114:0/1748669748 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f4c30107d30 0x7f4c3010a180 secure :-1 s=READY pgs=426 cs=0 l=1 rev1=1 crypto rx=0x7f4c20009900 tx=0x7f4c2002a3f0 comp rx=0 tx=0).stop 2024-04-23T10:26:41.481 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:41.479+0000 7f4c34b37640 1 -- 172.21.15.114:0/1748669748 shutdown_connections 2024-04-23T10:26:41.481 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:41.479+0000 7f4c34b37640 1 --2- 172.21.15.114:0/1748669748 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f4c30107d30 0x7f4c3010a180 unknown :-1 s=CLOSED pgs=426 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:26:41.481 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:41.479+0000 7f4c34b37640 1 --2- 172.21.15.114:0/1748669748 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f4c30105400 0x7f4c301077f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:26:41.481 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:41.479+0000 7f4c34b37640 1 --2- 172.21.15.114:0/1748669748 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f4c30102ad0 0x7f4c30104ec0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:26:41.481 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:41.479+0000 7f4c34b37640 1 -- 172.21.15.114:0/1748669748 >> 172.21.15.114:0/1748669748 conn(0x7f4c300fc4c0 msgr2=0x7f4c300fe900 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T10:26:41.481 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:41.479+0000 7f4c34b37640 1 -- 172.21.15.114:0/1748669748 shutdown_connections 2024-04-23T10:26:41.482 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:41.479+0000 7f4c34b37640 1 -- 172.21.15.114:0/1748669748 wait complete. 2024-04-23T10:26:41.482 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:41.480+0000 7f4c34b37640 1 Processor -- start 2024-04-23T10:26:41.482 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:41.480+0000 7f4c34b37640 1 -- start start 2024-04-23T10:26:41.482 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:41.480+0000 7f4c34b37640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f4c30102ad0 0x7f4c30198050 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T10:26:41.482 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:41.480+0000 7f4c34b37640 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f4c30105400 0x7f4c30198590 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T10:26:41.482 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:41.480+0000 7f4c34b37640 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f4c30107d30 0x7f4c3019f600 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T10:26:41.482 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:41.480+0000 7f4c34b37640 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x7f4c3010db20 con 0x7f4c30102ad0 2024-04-23T10:26:41.482 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:41.480+0000 7f4c34b37640 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 -- 0x7f4c3010d9a0 con 0x7f4c30107d30 2024-04-23T10:26:41.482 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:41.480+0000 7f4c34b37640 1 -- --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_getmap magic: 0 -- 0x7f4c3010dca0 con 0x7f4c30105400 2024-04-23T10:26:41.482 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:41.480+0000 7f4c2e59c640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f4c30102ad0 0x7f4c30198050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T10:26:41.482 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:41.480+0000 7f4c2e59c640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f4c30102ad0 0x7f4c30198050 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.114:40728/0 (socket says 172.21.15.114:40728) 2024-04-23T10:26:41.483 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:41.480+0000 7f4c2e59c640 1 -- 172.21.15.114:0/1657370017 learned_addr learned my addr 172.21.15.114:0/1657370017 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-04-23T10:26:41.483 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:41.480+0000 7f4c2dd9b640 1 --2- 172.21.15.114:0/1657370017 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f4c30105400 0x7f4c30198590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T10:26:41.483 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:41.480+0000 7f4c2ed9d640 1 --2- 172.21.15.114:0/1657370017 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f4c30107d30 0x7f4c3019f600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T10:26:41.483 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:41.480+0000 7f4c2e59c640 1 -- 172.21.15.114:0/1657370017 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f4c30105400 msgr2=0x7f4c30198590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T10:26:41.483 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:41.480+0000 7f4c2e59c640 1 --2- 172.21.15.114:0/1657370017 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f4c30105400 0x7f4c30198590 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:26:41.483 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:41.480+0000 7f4c2e59c640 1 -- 172.21.15.114:0/1657370017 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f4c30107d30 msgr2=0x7f4c3019f600 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T10:26:41.483 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:41.480+0000 7f4c2e59c640 1 --2- 172.21.15.114:0/1657370017 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f4c30107d30 0x7f4c3019f600 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:26:41.483 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:41.480+0000 7f4c2e59c640 1 -- 172.21.15.114:0/1657370017 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f4c3019fc70 con 0x7f4c30102ad0 2024-04-23T10:26:41.483 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:41.480+0000 7f4c2ed9d640 1 --2- 172.21.15.114:0/1657370017 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f4c30107d30 0x7f4c3019f600 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-23T10:26:41.483 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:41.480+0000 7f4c2e59c640 1 --2- 172.21.15.114:0/1657370017 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f4c30102ad0 0x7f4c30198050 secure :-1 s=READY pgs=427 cs=0 l=1 rev1=1 crypto rx=0x7f4c2800c920 tx=0x7f4c2800cde0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T10:26:41.483 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:41.480+0000 7f4c1f7fe640 1 -- 172.21.15.114:0/1657370017 <== mon.0 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4c28007c70 con 0x7f4c30102ad0 2024-04-23T10:26:41.483 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:41.481+0000 7f4c1f7fe640 1 -- 172.21.15.114:0/1657370017 <== mon.0 v2:172.21.15.114:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f4c28007e10 con 0x7f4c30102ad0 2024-04-23T10:26:41.483 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:41.481+0000 7f4c34b37640 1 -- 172.21.15.114:0/1657370017 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f4c3019ff60 con 0x7f4c30102ad0 2024-04-23T10:26:41.484 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:41.481+0000 7f4c1f7fe640 1 -- 172.21.15.114:0/1657370017 <== mon.0 v2:172.21.15.114:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4c28018820 con 0x7f4c30102ad0 2024-04-23T10:26:41.484 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:41.481+0000 7f4c34b37640 1 -- 172.21.15.114:0/1657370017 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f4c301a03c0 con 0x7f4c30102ad0 2024-04-23T10:26:41.484 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:41.482+0000 7f4c34b37640 1 -- 172.21.15.114:0/1657370017 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f4c300fdfb0 con 0x7f4c30102ad0 2024-04-23T10:26:41.485 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:41.483+0000 7f4c1f7fe640 1 -- 172.21.15.114:0/1657370017 <== mon.0 v2:172.21.15.114:3300/0 4 ==== mgrmap(e 22) ==== 138458+0+0 (secure 0 0 0) 0x7f4c2800f450 con 0x7f4c30102ad0 2024-04-23T10:26:41.485 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:41.483+0000 7f4c1f7fe640 1 --2- 172.21.15.114:0/1657370017 >> [v2:172.21.15.114:6800/1395693466,v1:172.21.15.114:6801/1395693466] conn(0x7f4c180a3a60 0x7f4c180a5f20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T10:26:41.485 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:41.484+0000 7f4c2dd9b640 1 --2- 172.21.15.114:0/1657370017 >> [v2:172.21.15.114:6800/1395693466,v1:172.21.15.114:6801/1395693466] conn(0x7f4c180a3a60 0x7f4c180a5f20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T10:26:41.485 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:41.484+0000 7f4c1f7fe640 1 -- 172.21.15.114:0/1657370017 <== mon.0 v2:172.21.15.114:3300/0 5 ==== osd_map(90..90 src has 1..90) ==== 8527+0+0 (secure 0 0 0) 0x7f4c280d0230 con 0x7f4c30102ad0 2024-04-23T10:26:41.485 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:41.484+0000 7f4c2dd9b640 1 --2- 172.21.15.114:0/1657370017 >> [v2:172.21.15.114:6800/1395693466,v1:172.21.15.114:6801/1395693466] conn(0x7f4c180a3a60 0x7f4c180a5f20 secure :-1 s=READY pgs=36 cs=0 l=1 rev1=1 crypto rx=0x7f4c30199560 tx=0x7f4c24009680 comp rx=0 tx=0).ready entity=mgr.15291 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T10:26:41.487 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:41.488+0000 7f4c1f7fe640 1 -- 172.21.15.114:0/1657370017 <== mon.0 v2:172.21.15.114:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f4c28014030 con 0x7f4c30102ad0 2024-04-23T10:26:41.492 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 23 10:26:41 smithi119 ceph-mon[32608]: mgrmap e20: x(active, since 8m), standbys: z, y 2024-04-23T10:26:41.492 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 23 10:26:41 smithi119 ceph-mon[32608]: Active manager daemon x restarted 2024-04-23T10:26:41.492 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 23 10:26:41 smithi119 ceph-mon[32608]: Activating manager daemon x 2024-04-23T10:26:41.492 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 23 10:26:41 smithi119 ceph-mon[32608]: osdmap e90: 12 total, 12 up, 12 in 2024-04-23T10:26:41.492 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 23 10:26:41 smithi119 ceph-mon[32608]: mgrmap e21: x(active, starting, since 0.0122461s), standbys: z, y 2024-04-23T10:26:41.492 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 23 10:26:41 smithi119 ceph-mon[32608]: from='mgr.15291 172.21.15.114:0/1095789187' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-04-23T10:26:41.493 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 23 10:26:41 smithi119 ceph-mon[32608]: from='mgr.15291 172.21.15.114:0/1095789187' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-04-23T10:26:41.493 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 23 10:26:41 smithi119 ceph-mon[32608]: from='mgr.15291 172.21.15.114:0/1095789187' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-04-23T10:26:41.493 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 23 10:26:41 smithi119 ceph-mon[32608]: from='mgr.15291 172.21.15.114:0/1095789187' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "j"}]: dispatch 2024-04-23T10:26:41.493 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 23 10:26:41 smithi119 ceph-mon[32608]: from='mgr.15291 172.21.15.114:0/1095789187' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "d"}]: dispatch 2024-04-23T10:26:41.493 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 23 10:26:41 smithi119 ceph-mon[32608]: from='mgr.15291 172.21.15.114:0/1095789187' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "g"}]: dispatch 2024-04-23T10:26:41.493 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 23 10:26:41 smithi119 ceph-mon[32608]: from='mgr.15291 172.21.15.114:0/1095789187' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "a"}]: dispatch 2024-04-23T10:26:41.493 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 23 10:26:41 smithi119 ceph-mon[32608]: from='mgr.15291 172.21.15.114:0/1095789187' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "l"}]: dispatch 2024-04-23T10:26:41.493 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 23 10:26:41 smithi119 ceph-mon[32608]: from='mgr.15291 172.21.15.114:0/1095789187' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "f"}]: dispatch 2024-04-23T10:26:41.493 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 23 10:26:41 smithi119 ceph-mon[32608]: from='mgr.15291 172.21.15.114:0/1095789187' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "c"}]: dispatch 2024-04-23T10:26:41.493 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 23 10:26:41 smithi119 ceph-mon[32608]: from='mgr.15291 172.21.15.114:0/1095789187' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "k"}]: dispatch 2024-04-23T10:26:41.493 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 23 10:26:41 smithi119 ceph-mon[32608]: from='mgr.15291 172.21.15.114:0/1095789187' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "i"}]: dispatch 2024-04-23T10:26:41.494 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 23 10:26:41 smithi119 ceph-mon[32608]: from='mgr.15291 172.21.15.114:0/1095789187' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "e"}]: dispatch 2024-04-23T10:26:41.494 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 23 10:26:41 smithi119 ceph-mon[32608]: from='mgr.15291 172.21.15.114:0/1095789187' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "h"}]: dispatch 2024-04-23T10:26:41.494 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 23 10:26:41 smithi119 ceph-mon[32608]: from='mgr.15291 172.21.15.114:0/1095789187' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "b"}]: dispatch 2024-04-23T10:26:41.494 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 23 10:26:41 smithi119 ceph-mon[32608]: from='mgr.15291 172.21.15.114:0/1095789187' entity='mgr.x' cmd=[{"prefix": "mgr metadata", "who": "x", "id": "x"}]: dispatch 2024-04-23T10:26:41.494 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 23 10:26:41 smithi119 ceph-mon[32608]: from='mgr.15291 172.21.15.114:0/1095789187' entity='mgr.x' cmd=[{"prefix": "mgr metadata", "who": "z", "id": "z"}]: dispatch 2024-04-23T10:26:41.494 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 23 10:26:41 smithi119 ceph-mon[32608]: from='mgr.15291 172.21.15.114:0/1095789187' entity='mgr.x' cmd=[{"prefix": "mgr metadata", "who": "y", "id": "y"}]: dispatch 2024-04-23T10:26:41.494 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 23 10:26:41 smithi119 ceph-mon[32608]: from='mgr.15291 172.21.15.114:0/1095789187' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-04-23T10:26:41.494 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 23 10:26:41 smithi119 ceph-mon[32608]: from='mgr.15291 172.21.15.114:0/1095789187' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-04-23T10:26:41.494 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 23 10:26:41 smithi119 ceph-mon[32608]: from='mgr.15291 172.21.15.114:0/1095789187' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-04-23T10:26:41.494 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 23 10:26:41 smithi119 ceph-mon[32608]: from='mgr.15291 172.21.15.114:0/1095789187' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-04-23T10:26:41.494 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 23 10:26:41 smithi119 ceph-mon[32608]: from='mgr.15291 172.21.15.114:0/1095789187' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-04-23T10:26:41.495 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 23 10:26:41 smithi119 ceph-mon[32608]: from='mgr.15291 172.21.15.114:0/1095789187' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-04-23T10:26:41.495 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 23 10:26:41 smithi119 ceph-mon[32608]: from='mgr.15291 172.21.15.114:0/1095789187' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-04-23T10:26:41.495 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 23 10:26:41 smithi119 ceph-mon[32608]: from='mgr.15291 172.21.15.114:0/1095789187' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-04-23T10:26:41.495 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 23 10:26:41 smithi119 ceph-mon[32608]: from='mgr.15291 172.21.15.114:0/1095789187' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 8}]: dispatch 2024-04-23T10:26:41.495 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 23 10:26:41 smithi119 ceph-mon[32608]: from='mgr.15291 172.21.15.114:0/1095789187' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 9}]: dispatch 2024-04-23T10:26:41.495 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 23 10:26:41 smithi119 ceph-mon[32608]: from='mgr.15291 172.21.15.114:0/1095789187' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 10}]: dispatch 2024-04-23T10:26:41.495 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 23 10:26:41 smithi119 ceph-mon[32608]: from='mgr.15291 172.21.15.114:0/1095789187' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 11}]: dispatch 2024-04-23T10:26:41.495 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 23 10:26:41 smithi119 ceph-mon[32608]: from='mgr.15291 172.21.15.114:0/1095789187' entity='mgr.x' cmd=[{"prefix": "mds metadata"}]: dispatch 2024-04-23T10:26:41.495 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 23 10:26:41 smithi119 ceph-mon[32608]: from='mgr.15291 172.21.15.114:0/1095789187' entity='mgr.x' cmd=[{"prefix": "osd metadata"}]: dispatch 2024-04-23T10:26:41.495 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 23 10:26:41 smithi119 ceph-mon[32608]: from='mgr.15291 172.21.15.114:0/1095789187' entity='mgr.x' cmd=[{"prefix": "mon metadata"}]: dispatch 2024-04-23T10:26:41.495 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 23 10:26:41 smithi119 ceph-mon[32608]: Manager daemon x is now available 2024-04-23T10:26:41.495 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 23 10:26:41 smithi119 ceph-mon[32608]: from='mgr.15291 172.21.15.114:0/1095789187' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-23T10:26:41.495 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 23 10:26:41 smithi119 ceph-mon[32608]: from='mgr.15291 172.21.15.114:0/1095789187' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2024-04-23T10:26:41.496 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 23 10:26:41 smithi119 ceph-mon[32608]: from='mgr.15291 172.21.15.114:0/1095789187' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2024-04-23T10:26:41.697 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:41.697+0000 7f4c34b37640 1 -- 172.21.15.114:0/1657370017 --> [v2:172.21.15.114:6800/1395693466,v1:172.21.15.114:6801/1395693466] -- mgr_command(tid 0: {"prefix": "fs snap-schedule retention add", "fs": "cephfs", "path": "/", "retention_spec_or_period": "6m3h", "target": ["mon-mgr", ""]}) -- 0x7f4c30062390 con 0x7f4c180a3a60 2024-04-23T10:26:41.751 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:41.751+0000 7f4c1f7fe640 1 -- 172.21.15.114:0/1657370017 <== mgr.15291 v2:172.21.15.114:6800/1395693466 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+26 (secure 0 0 0) 0x7f4c30062390 con 0x7f4c180a3a60 2024-04-23T10:26:41.751 INFO:teuthology.orchestra.run.smithi114.stdout:Retention added to path / 2024-04-23T10:26:41.753 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:41.753+0000 7f4c34b37640 1 -- 172.21.15.114:0/1657370017 >> [v2:172.21.15.114:6800/1395693466,v1:172.21.15.114:6801/1395693466] conn(0x7f4c180a3a60 msgr2=0x7f4c180a5f20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T10:26:41.753 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:41.753+0000 7f4c34b37640 1 --2- 172.21.15.114:0/1657370017 >> [v2:172.21.15.114:6800/1395693466,v1:172.21.15.114:6801/1395693466] conn(0x7f4c180a3a60 0x7f4c180a5f20 secure :-1 s=READY pgs=36 cs=0 l=1 rev1=1 crypto rx=0x7f4c30199560 tx=0x7f4c24009680 comp rx=0 tx=0).stop 2024-04-23T10:26:41.754 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:41.753+0000 7f4c34b37640 1 -- 172.21.15.114:0/1657370017 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f4c30102ad0 msgr2=0x7f4c30198050 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T10:26:41.754 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:41.753+0000 7f4c34b37640 1 --2- 172.21.15.114:0/1657370017 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f4c30102ad0 0x7f4c30198050 secure :-1 s=READY pgs=427 cs=0 l=1 rev1=1 crypto rx=0x7f4c2800c920 tx=0x7f4c2800cde0 comp rx=0 tx=0).stop 2024-04-23T10:26:41.754 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:41.753+0000 7f4c34b37640 1 -- 172.21.15.114:0/1657370017 shutdown_connections 2024-04-23T10:26:41.754 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:41.753+0000 7f4c34b37640 1 --2- 172.21.15.114:0/1657370017 >> [v2:172.21.15.114:6800/1395693466,v1:172.21.15.114:6801/1395693466] conn(0x7f4c180a3a60 0x7f4c180a5f20 unknown :-1 s=CLOSED pgs=36 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:26:41.754 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:41.753+0000 7f4c34b37640 1 --2- 172.21.15.114:0/1657370017 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f4c30107d30 0x7f4c3019f600 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:26:41.754 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:41.753+0000 7f4c34b37640 1 --2- 172.21.15.114:0/1657370017 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f4c30105400 0x7f4c30198590 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:26:41.754 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:41.753+0000 7f4c34b37640 1 --2- 172.21.15.114:0/1657370017 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f4c30102ad0 0x7f4c30198050 unknown :-1 s=CLOSED pgs=427 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:26:41.754 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:41.753+0000 7f4c34b37640 1 -- 172.21.15.114:0/1657370017 >> 172.21.15.114:0/1657370017 conn(0x7f4c300fc4c0 msgr2=0x7f4c30103410 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T10:26:41.754 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:41.753+0000 7f4c34b37640 1 -- 172.21.15.114:0/1657370017 shutdown_connections 2024-04-23T10:26:41.754 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:41.753+0000 7f4c34b37640 1 -- 172.21.15.114:0/1657370017 wait complete. 2024-04-23T10:26:41.765 DEBUG:teuthology.orchestra.run.smithi114:> sudo TESTDIR=/home/ubuntu/cephtest bash -c 'ceph fs snap-schedule status --fs=cephfs --path=/' 2024-04-23T10:26:41.866 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:41.866+0000 7f0ce93bf640 1 -- 172.21.15.114:0/439360485 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f0ce4102040 msgr2=0x7f0ce41024c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T10:26:41.866 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:41.866+0000 7f0ce93bf640 1 --2- 172.21.15.114:0/439360485 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f0ce4102040 0x7f0ce41024c0 secure :-1 s=READY pgs=428 cs=0 l=1 rev1=1 crypto rx=0x7f0cd8009900 tx=0x7f0cd802a410 comp rx=0 tx=0).stop 2024-04-23T10:26:41.866 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:41.866+0000 7f0ce93bf640 1 -- 172.21.15.114:0/439360485 shutdown_connections 2024-04-23T10:26:41.866 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:41.866+0000 7f0ce93bf640 1 --2- 172.21.15.114:0/439360485 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f0ce4102a00 0x7f0ce4109320 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:26:41.866 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:41.866+0000 7f0ce93bf640 1 --2- 172.21.15.114:0/439360485 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f0ce4102040 0x7f0ce41024c0 unknown :-1 s=CLOSED pgs=428 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:26:41.866 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:41.866+0000 7f0ce93bf640 1 --2- 172.21.15.114:0/439360485 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f0ce4100e60 0x7f0ce4101280 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:26:41.866 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:41.866+0000 7f0ce93bf640 1 -- 172.21.15.114:0/439360485 >> 172.21.15.114:0/439360485 conn(0x7f0ce40fc420 msgr2=0x7f0ce40fe860 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T10:26:41.866 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:41.866+0000 7f0ce93bf640 1 -- 172.21.15.114:0/439360485 shutdown_connections 2024-04-23T10:26:41.867 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:41.866+0000 7f0ce93bf640 1 -- 172.21.15.114:0/439360485 wait complete. 2024-04-23T10:26:41.867 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:41.866+0000 7f0ce93bf640 1 Processor -- start 2024-04-23T10:26:41.868 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:41.867+0000 7f0ce93bf640 1 -- start start 2024-04-23T10:26:41.869 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:41.867+0000 7f0ce93bf640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f0ce4100e60 0x7f0ce4197090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T10:26:41.869 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:41.867+0000 7f0ce93bf640 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f0ce4102040 0x7f0ce41975d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T10:26:41.869 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:41.867+0000 7f0ce93bf640 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f0ce4102a00 0x7f0ce419e640 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T10:26:41.869 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:41.867+0000 7f0ce93bf640 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x7f0ce410ca10 con 0x7f0ce4100e60 2024-04-23T10:26:41.869 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:41.867+0000 7f0ce93bf640 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 -- 0x7f0ce410c890 con 0x7f0ce4102040 2024-04-23T10:26:41.869 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:41.867+0000 7f0ce93bf640 1 -- --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_getmap magic: 0 -- 0x7f0ce410cb90 con 0x7f0ce4102a00 2024-04-23T10:26:41.869 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:41.867+0000 7f0ce2ffd640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f0ce4100e60 0x7f0ce4197090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T10:26:41.869 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:41.867+0000 7f0ce2ffd640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f0ce4100e60 0x7f0ce4197090 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.114:40734/0 (socket says 172.21.15.114:40734) 2024-04-23T10:26:41.869 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:41.867+0000 7f0ce2ffd640 1 -- 172.21.15.114:0/11428640 learned_addr learned my addr 172.21.15.114:0/11428640 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-04-23T10:26:41.869 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:41.867+0000 7f0ce37fe640 1 --2- 172.21.15.114:0/11428640 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f0ce4102a00 0x7f0ce419e640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T10:26:41.869 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:41.867+0000 7f0ce2ffd640 1 -- 172.21.15.114:0/11428640 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f0ce4102a00 msgr2=0x7f0ce419e640 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T10:26:41.869 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:41.867+0000 7f0ce27fc640 1 --2- 172.21.15.114:0/11428640 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f0ce4102040 0x7f0ce41975d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T10:26:41.869 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:41.867+0000 7f0ce2ffd640 1 --2- 172.21.15.114:0/11428640 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f0ce4102a00 0x7f0ce419e640 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:26:41.869 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:41.867+0000 7f0ce2ffd640 1 -- 172.21.15.114:0/11428640 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f0ce4102040 msgr2=0x7f0ce41975d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T10:26:41.870 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:41.867+0000 7f0ce2ffd640 1 --2- 172.21.15.114:0/11428640 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f0ce4102040 0x7f0ce41975d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:26:41.870 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:41.867+0000 7f0ce2ffd640 1 -- 172.21.15.114:0/11428640 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f0ce419ecb0 con 0x7f0ce4100e60 2024-04-23T10:26:41.870 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:41.867+0000 7f0ce37fe640 1 --2- 172.21.15.114:0/11428640 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f0ce4102a00 0x7f0ce419e640 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-23T10:26:41.870 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:41.867+0000 7f0ce27fc640 1 --2- 172.21.15.114:0/11428640 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f0ce4102040 0x7f0ce41975d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-23T10:26:41.870 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:41.867+0000 7f0ce2ffd640 1 --2- 172.21.15.114:0/11428640 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f0ce4100e60 0x7f0ce4197090 secure :-1 s=READY pgs=429 cs=0 l=1 rev1=1 crypto rx=0x7f0cd400d950 tx=0x7f0cd400de10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T10:26:41.870 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:41.867+0000 7f0ccbfff640 1 -- 172.21.15.114:0/11428640 <== mon.0 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0cd4017070 con 0x7f0ce4100e60 2024-04-23T10:26:41.870 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:41.867+0000 7f0ccbfff640 1 -- 172.21.15.114:0/11428640 <== mon.0 v2:172.21.15.114:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f0cd4009c30 con 0x7f0ce4100e60 2024-04-23T10:26:41.870 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:41.867+0000 7f0ce93bf640 1 -- 172.21.15.114:0/11428640 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f0ce419efa0 con 0x7f0ce4100e60 2024-04-23T10:26:41.870 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:41.867+0000 7f0ccbfff640 1 -- 172.21.15.114:0/11428640 <== mon.0 v2:172.21.15.114:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0cd400f620 con 0x7f0ce4100e60 2024-04-23T10:26:41.870 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:41.868+0000 7f0ce93bf640 1 -- 172.21.15.114:0/11428640 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f0ce419f400 con 0x7f0ce4100e60 2024-04-23T10:26:41.870 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:41.869+0000 7f0ce93bf640 1 -- 172.21.15.114:0/11428640 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f0ce4108220 con 0x7f0ce4100e60 2024-04-23T10:26:41.871 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:41.870+0000 7f0ccbfff640 1 -- 172.21.15.114:0/11428640 <== mon.0 v2:172.21.15.114:3300/0 4 ==== mgrmap(e 22) ==== 138458+0+0 (secure 0 0 0) 0x7f0cd401a070 con 0x7f0ce4100e60 2024-04-23T10:26:41.871 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:41.870+0000 7f0ccbfff640 1 --2- 172.21.15.114:0/11428640 >> [v2:172.21.15.114:6800/1395693466,v1:172.21.15.114:6801/1395693466] conn(0x7f0ccc0a3a60 0x7f0ccc0a5f20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T10:26:41.871 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:41.871+0000 7f0ce27fc640 1 --2- 172.21.15.114:0/11428640 >> [v2:172.21.15.114:6800/1395693466,v1:172.21.15.114:6801/1395693466] conn(0x7f0ccc0a3a60 0x7f0ccc0a5f20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T10:26:41.871 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:41.871+0000 7f0ccbfff640 1 -- 172.21.15.114:0/11428640 <== mon.0 v2:172.21.15.114:3300/0 5 ==== osd_map(90..90 src has 1..90) ==== 8527+0+0 (secure 0 0 0) 0x7f0cd40cfbf0 con 0x7f0ce4100e60 2024-04-23T10:26:41.871 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:41.871+0000 7f0ce27fc640 1 --2- 172.21.15.114:0/11428640 >> [v2:172.21.15.114:6800/1395693466,v1:172.21.15.114:6801/1395693466] conn(0x7f0ccc0a3a60 0x7f0ccc0a5f20 secure :-1 s=READY pgs=37 cs=0 l=1 rev1=1 crypto rx=0x7f0cd8006010 tx=0x7f0cd8000bc0 comp rx=0 tx=0).ready entity=mgr.15291 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T10:26:41.874 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:41.874+0000 7f0ccbfff640 1 -- 172.21.15.114:0/11428640 <== mon.0 v2:172.21.15.114:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f0cd4009de0 con 0x7f0ce4100e60 2024-04-23T10:26:42.077 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 23 10:26:42 smithi204 ceph-mon[32649]: [23/Apr/2024:10:26:40] ENGINE Bus STARTING 2024-04-23T10:26:42.078 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 23 10:26:42 smithi204 ceph-mon[32649]: [23/Apr/2024:10:26:40] ENGINE Serving on http://172.21.15.114:8765 2024-04-23T10:26:42.078 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 23 10:26:42 smithi204 ceph-mon[32649]: [23/Apr/2024:10:26:41] ENGINE Serving on https://172.21.15.114:7150 2024-04-23T10:26:42.078 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 23 10:26:42 smithi204 ceph-mon[32649]: [23/Apr/2024:10:26:41] ENGINE Bus STARTED 2024-04-23T10:26:42.078 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 23 10:26:42 smithi204 ceph-mon[32649]: mgrmap e22: x(active, since 1.02192s), standbys: z, y 2024-04-23T10:26:42.094 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:42.094+0000 7f0ce93bf640 1 -- 172.21.15.114:0/11428640 --> [v2:172.21.15.114:6800/1395693466,v1:172.21.15.114:6801/1395693466] -- mgr_command(tid 0: {"prefix": "fs snap-schedule status", "fs": "cephfs", "path": "/", "target": ["mon-mgr", ""]}) -- 0x7f0ce4062390 con 0x7f0ccc0a3a60 2024-04-23T10:26:42.102 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:42.102+0000 7f0ccbfff640 1 -- 172.21.15.114:0/11428640 <== mgr.15291 v2:172.21.15.114:6800/1395693466 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+298 (secure 0 0 0) 0x7f0ce4062390 con 0x7f0ccc0a3a60 2024-04-23T10:26:42.102 INFO:teuthology.orchestra.run.smithi114.stdout:{"fs": "cephfs", "subvol": null, "group": null, "path": "/", "rel_path": "/", "schedule": "1m", "retention": {"m": 6, "h": 3}, "start": "2024-04-23T00:00:00", "created": "2024-04-23T10:26:41", "first": null, "last": null, "last_pruned": null, "created_count": 0, "pruned_count": 0, "active": true} 2024-04-23T10:26:42.104 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:42.104+0000 7f0ce93bf640 1 -- 172.21.15.114:0/11428640 >> [v2:172.21.15.114:6800/1395693466,v1:172.21.15.114:6801/1395693466] conn(0x7f0ccc0a3a60 msgr2=0x7f0ccc0a5f20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T10:26:42.104 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:42.104+0000 7f0ce93bf640 1 --2- 172.21.15.114:0/11428640 >> [v2:172.21.15.114:6800/1395693466,v1:172.21.15.114:6801/1395693466] conn(0x7f0ccc0a3a60 0x7f0ccc0a5f20 secure :-1 s=READY pgs=37 cs=0 l=1 rev1=1 crypto rx=0x7f0cd8006010 tx=0x7f0cd8000bc0 comp rx=0 tx=0).stop 2024-04-23T10:26:42.104 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:42.104+0000 7f0ce93bf640 1 -- 172.21.15.114:0/11428640 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f0ce4100e60 msgr2=0x7f0ce4197090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T10:26:42.105 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:42.104+0000 7f0ce93bf640 1 --2- 172.21.15.114:0/11428640 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f0ce4100e60 0x7f0ce4197090 secure :-1 s=READY pgs=429 cs=0 l=1 rev1=1 crypto rx=0x7f0cd400d950 tx=0x7f0cd400de10 comp rx=0 tx=0).stop 2024-04-23T10:26:42.105 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:42.104+0000 7f0ce93bf640 1 -- 172.21.15.114:0/11428640 shutdown_connections 2024-04-23T10:26:42.105 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:42.104+0000 7f0ce93bf640 1 --2- 172.21.15.114:0/11428640 >> [v2:172.21.15.114:6800/1395693466,v1:172.21.15.114:6801/1395693466] conn(0x7f0ccc0a3a60 0x7f0ccc0a5f20 unknown :-1 s=CLOSED pgs=37 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:26:42.105 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:42.104+0000 7f0ce93bf640 1 --2- 172.21.15.114:0/11428640 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f0ce4102a00 0x7f0ce419e640 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:26:42.105 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:42.104+0000 7f0ce93bf640 1 --2- 172.21.15.114:0/11428640 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f0ce4102040 0x7f0ce41975d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:26:42.105 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:42.104+0000 7f0ce93bf640 1 --2- 172.21.15.114:0/11428640 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f0ce4100e60 0x7f0ce4197090 unknown :-1 s=CLOSED pgs=429 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:26:42.105 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:42.104+0000 7f0ce93bf640 1 -- 172.21.15.114:0/11428640 >> 172.21.15.114:0/11428640 conn(0x7f0ce40fc420 msgr2=0x7f0ce41053f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T10:26:42.105 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:42.104+0000 7f0ce93bf640 1 -- 172.21.15.114:0/11428640 shutdown_connections 2024-04-23T10:26:42.105 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:42.104+0000 7f0ce93bf640 1 -- 172.21.15.114:0/11428640 wait complete. 2024-04-23T10:26:42.116 DEBUG:teuthology.orchestra.run.smithi114:> sudo TESTDIR=/home/ubuntu/cephtest bash -c 'ceph fs snap-schedule list --fs=cephfs --path=/ --recursive=true' 2024-04-23T10:26:42.225 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:42.223+0000 7fbc0db94640 1 -- 172.21.15.114:0/531078909 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fbc081098b0 msgr2=0x7fbc08110290 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T10:26:42.225 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:42.223+0000 7fbc0db94640 1 --2- 172.21.15.114:0/531078909 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fbc081098b0 0x7fbc08110290 secure :-1 s=READY pgs=430 cs=0 l=1 rev1=1 crypto rx=0x7fbbfc009900 tx=0x7fbbfc02a410 comp rx=0 tx=0).stop 2024-04-23T10:26:42.225 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:42.223+0000 7fbc0db94640 1 -- 172.21.15.114:0/531078909 shutdown_connections 2024-04-23T10:26:42.225 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:42.223+0000 7fbc0db94640 1 --2- 172.21.15.114:0/531078909 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fbc081107d0 0x7fbc08112bc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:26:42.225 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:42.223+0000 7fbc0db94640 1 --2- 172.21.15.114:0/531078909 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fbc081098b0 0x7fbc08110290 unknown :-1 s=CLOSED pgs=430 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:26:42.225 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:42.223+0000 7fbc0db94640 1 --2- 172.21.15.114:0/531078909 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fbc08108f50 0x7fbc08109370 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:26:42.225 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:42.223+0000 7fbc0db94640 1 -- 172.21.15.114:0/531078909 >> 172.21.15.114:0/531078909 conn(0x7fbc0806efc0 msgr2=0x7fbc08071400 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T10:26:42.225 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:42.223+0000 7fbc0db94640 1 -- 172.21.15.114:0/531078909 shutdown_connections 2024-04-23T10:26:42.225 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:42.223+0000 7fbc0db94640 1 -- 172.21.15.114:0/531078909 wait complete. 2024-04-23T10:26:42.225 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:42.223+0000 7fbc0db94640 1 Processor -- start 2024-04-23T10:26:42.225 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:42.223+0000 7fbc0db94640 1 -- start start 2024-04-23T10:26:42.225 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:42.224+0000 7fbc0db94640 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fbc08108f50 0x7fbc081a6ed0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T10:26:42.226 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:42.224+0000 7fbc0db94640 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fbc081098b0 0x7fbc081a7410 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T10:26:42.226 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:42.224+0000 7fbc0db94640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fbc081107d0 0x7fbc081ae480 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T10:26:42.226 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:42.224+0000 7fbc0db94640 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x7fbc0810b380 con 0x7fbc081107d0 2024-04-23T10:26:42.226 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:42.224+0000 7fbc0db94640 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 -- 0x7fbc0810b200 con 0x7fbc08108f50 2024-04-23T10:26:42.226 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:42.224+0000 7fbc0db94640 1 -- --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_getmap magic: 0 -- 0x7fbc0810b500 con 0x7fbc081098b0 2024-04-23T10:26:42.226 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:42.224+0000 7fbc07fff640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fbc081107d0 0x7fbc081ae480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T10:26:42.226 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:42.224+0000 7fbc07fff640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fbc081107d0 0x7fbc081ae480 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.114:40756/0 (socket says 172.21.15.114:40756) 2024-04-23T10:26:42.226 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:42.224+0000 7fbc07fff640 1 -- 172.21.15.114:0/693771542 learned_addr learned my addr 172.21.15.114:0/693771542 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-04-23T10:26:42.226 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:42.224+0000 7fbc077fe640 1 --2- 172.21.15.114:0/693771542 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fbc08108f50 0x7fbc081a6ed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T10:26:42.226 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:42.224+0000 7fbc06ffd640 1 --2- 172.21.15.114:0/693771542 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fbc081098b0 0x7fbc081a7410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T10:26:42.226 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:42.224+0000 7fbc07fff640 1 -- 172.21.15.114:0/693771542 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fbc081098b0 msgr2=0x7fbc081a7410 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T10:26:42.226 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:42.224+0000 7fbc07fff640 1 --2- 172.21.15.114:0/693771542 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fbc081098b0 0x7fbc081a7410 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:26:42.226 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:42.224+0000 7fbc07fff640 1 -- 172.21.15.114:0/693771542 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fbc08108f50 msgr2=0x7fbc081a6ed0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T10:26:42.226 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:42.224+0000 7fbc07fff640 1 --2- 172.21.15.114:0/693771542 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fbc08108f50 0x7fbc081a6ed0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:26:42.226 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:42.224+0000 7fbc07fff640 1 -- 172.21.15.114:0/693771542 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fbc081aeaf0 con 0x7fbc081107d0 2024-04-23T10:26:42.227 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:42.224+0000 7fbc077fe640 1 --2- 172.21.15.114:0/693771542 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fbc08108f50 0x7fbc081a6ed0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-23T10:26:42.227 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:42.224+0000 7fbc07fff640 1 --2- 172.21.15.114:0/693771542 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fbc081107d0 0x7fbc081ae480 secure :-1 s=READY pgs=431 cs=0 l=1 rev1=1 crypto rx=0x7fbc0000eb10 tx=0x7fbc0000c2d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T10:26:42.227 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:42.224+0000 7fbc04ff9640 1 -- 172.21.15.114:0/693771542 <== mon.0 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbc00004b20 con 0x7fbc081107d0 2024-04-23T10:26:42.227 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:42.225+0000 7fbc0db94640 1 -- 172.21.15.114:0/693771542 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fbc081aede0 con 0x7fbc081107d0 2024-04-23T10:26:42.227 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:42.225+0000 7fbc04ff9640 1 -- 172.21.15.114:0/693771542 <== mon.0 v2:172.21.15.114:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fbc00004cc0 con 0x7fbc081107d0 2024-04-23T10:26:42.227 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:42.225+0000 7fbc04ff9640 1 -- 172.21.15.114:0/693771542 <== mon.0 v2:172.21.15.114:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbc00010670 con 0x7fbc081107d0 2024-04-23T10:26:42.227 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:42.225+0000 7fbc0db94640 1 -- 172.21.15.114:0/693771542 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fbc081af250 con 0x7fbc081107d0 2024-04-23T10:26:42.228 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:42.226+0000 7fbc0db94640 1 -- 172.21.15.114:0/693771542 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fbc08118610 con 0x7fbc081107d0 2024-04-23T10:26:42.228 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:42.227+0000 7fbc04ff9640 1 -- 172.21.15.114:0/693771542 <== mon.0 v2:172.21.15.114:3300/0 4 ==== mgrmap(e 22) ==== 138458+0+0 (secure 0 0 0) 0x7fbc00005020 con 0x7fbc081107d0 2024-04-23T10:26:42.228 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:42.228+0000 7fbc04ff9640 1 --2- 172.21.15.114:0/693771542 >> [v2:172.21.15.114:6800/1395693466,v1:172.21.15.114:6801/1395693466] conn(0x7fbbf00a3a60 0x7fbbf00a5f20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T10:26:42.228 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:42.228+0000 7fbc04ff9640 1 -- 172.21.15.114:0/693771542 <== mon.0 v2:172.21.15.114:3300/0 5 ==== osd_map(90..90 src has 1..90) ==== 8527+0+0 (secure 0 0 0) 0x7fbc000d0680 con 0x7fbc081107d0 2024-04-23T10:26:42.228 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:42.228+0000 7fbc077fe640 1 --2- 172.21.15.114:0/693771542 >> [v2:172.21.15.114:6800/1395693466,v1:172.21.15.114:6801/1395693466] conn(0x7fbbf00a3a60 0x7fbbf00a5f20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T10:26:42.228 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:42.229+0000 7fbc077fe640 1 --2- 172.21.15.114:0/693771542 >> [v2:172.21.15.114:6800/1395693466,v1:172.21.15.114:6801/1395693466] conn(0x7fbbf00a3a60 0x7fbbf00a5f20 secure :-1 s=READY pgs=38 cs=0 l=1 rev1=1 crypto rx=0x7fbbf8009fc0 tx=0x7fbbf8005f90 comp rx=0 tx=0).ready entity=mgr.15291 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T10:26:42.233 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:42.233+0000 7fbc04ff9640 1 -- 172.21.15.114:0/693771542 <== mon.0 v2:172.21.15.114:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7fbc0009d230 con 0x7fbc081107d0 2024-04-23T10:26:42.471 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:42.471+0000 7fbc0db94640 1 -- 172.21.15.114:0/693771542 --> [v2:172.21.15.114:6800/1395693466,v1:172.21.15.114:6801/1395693466] -- mgr_command(tid 0: {"prefix": "fs snap-schedule list", "fs": "cephfs", "path": "/", "recursive": true, "target": ["mon-mgr", ""]}) -- 0x7fbc08062390 con 0x7fbbf00a3a60 2024-04-23T10:26:42.478 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:42.478+0000 7fbc04ff9640 1 -- 172.21.15.114:0/693771542 <== mgr.15291 v2:172.21.15.114:6800/1395693466 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+10 (secure 0 0 0) 0x7fbc08062390 con 0x7fbbf00a3a60 2024-04-23T10:26:42.478 INFO:teuthology.orchestra.run.smithi114.stdout:/ 1m 6m3h 2024-04-23T10:26:42.480 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:42.480+0000 7fbc0db94640 1 -- 172.21.15.114:0/693771542 >> [v2:172.21.15.114:6800/1395693466,v1:172.21.15.114:6801/1395693466] conn(0x7fbbf00a3a60 msgr2=0x7fbbf00a5f20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T10:26:42.480 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:42.480+0000 7fbc0db94640 1 --2- 172.21.15.114:0/693771542 >> [v2:172.21.15.114:6800/1395693466,v1:172.21.15.114:6801/1395693466] conn(0x7fbbf00a3a60 0x7fbbf00a5f20 secure :-1 s=READY pgs=38 cs=0 l=1 rev1=1 crypto rx=0x7fbbf8009fc0 tx=0x7fbbf8005f90 comp rx=0 tx=0).stop 2024-04-23T10:26:42.480 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:42.480+0000 7fbc0db94640 1 -- 172.21.15.114:0/693771542 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fbc081107d0 msgr2=0x7fbc081ae480 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T10:26:42.480 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:42.480+0000 7fbc0db94640 1 --2- 172.21.15.114:0/693771542 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fbc081107d0 0x7fbc081ae480 secure :-1 s=READY pgs=431 cs=0 l=1 rev1=1 crypto rx=0x7fbc0000eb10 tx=0x7fbc0000c2d0 comp rx=0 tx=0).stop 2024-04-23T10:26:42.480 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:42.480+0000 7fbc0db94640 1 -- 172.21.15.114:0/693771542 shutdown_connections 2024-04-23T10:26:42.480 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:42.480+0000 7fbc0db94640 1 --2- 172.21.15.114:0/693771542 >> [v2:172.21.15.114:6800/1395693466,v1:172.21.15.114:6801/1395693466] conn(0x7fbbf00a3a60 0x7fbbf00a5f20 unknown :-1 s=CLOSED pgs=38 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:26:42.481 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:42.480+0000 7fbc0db94640 1 --2- 172.21.15.114:0/693771542 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fbc081107d0 0x7fbc081ae480 unknown :-1 s=CLOSED pgs=431 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:26:42.481 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:42.480+0000 7fbc0db94640 1 --2- 172.21.15.114:0/693771542 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fbc081098b0 0x7fbc081a7410 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:26:42.481 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:42.480+0000 7fbc0db94640 1 --2- 172.21.15.114:0/693771542 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fbc08108f50 0x7fbc081a6ed0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:26:42.481 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:42.480+0000 7fbc0db94640 1 -- 172.21.15.114:0/693771542 >> 172.21.15.114:0/693771542 conn(0x7fbc0806efc0 msgr2=0x7fbc08070cc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T10:26:42.481 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:42.480+0000 7fbc0db94640 1 -- 172.21.15.114:0/693771542 shutdown_connections 2024-04-23T10:26:42.481 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:42.480+0000 7fbc0db94640 1 -- 172.21.15.114:0/693771542 wait complete. 2024-04-23T10:26:42.491 INFO:journalctl@ceph.mon.a.smithi114.stdout:Apr 23 10:26:42 smithi114 ceph-mon[29255]: [23/Apr/2024:10:26:40] ENGINE Bus STARTING 2024-04-23T10:26:42.491 INFO:journalctl@ceph.mon.a.smithi114.stdout:Apr 23 10:26:42 smithi114 ceph-mon[29255]: [23/Apr/2024:10:26:40] ENGINE Serving on http://172.21.15.114:8765 2024-04-23T10:26:42.491 INFO:journalctl@ceph.mon.a.smithi114.stdout:Apr 23 10:26:42 smithi114 ceph-mon[29255]: [23/Apr/2024:10:26:41] ENGINE Serving on https://172.21.15.114:7150 2024-04-23T10:26:42.491 INFO:journalctl@ceph.mon.a.smithi114.stdout:Apr 23 10:26:42 smithi114 ceph-mon[29255]: [23/Apr/2024:10:26:41] ENGINE Bus STARTED 2024-04-23T10:26:42.491 INFO:journalctl@ceph.mon.a.smithi114.stdout:Apr 23 10:26:42 smithi114 ceph-mon[29255]: mgrmap e22: x(active, since 1.02192s), standbys: z, y 2024-04-23T10:26:42.492 DEBUG:teuthology.orchestra.run.smithi114:> sudo TESTDIR=/home/ubuntu/cephtest bash -c 'date +%s > START_TIME' 2024-04-23T10:26:42.492 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 23 10:26:42 smithi119 ceph-mon[32608]: [23/Apr/2024:10:26:40] ENGINE Bus STARTING 2024-04-23T10:26:42.492 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 23 10:26:42 smithi119 ceph-mon[32608]: [23/Apr/2024:10:26:40] ENGINE Serving on http://172.21.15.114:8765 2024-04-23T10:26:42.492 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 23 10:26:42 smithi119 ceph-mon[32608]: [23/Apr/2024:10:26:41] ENGINE Serving on https://172.21.15.114:7150 2024-04-23T10:26:42.492 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 23 10:26:42 smithi119 ceph-mon[32608]: [23/Apr/2024:10:26:41] ENGINE Bus STARTED 2024-04-23T10:26:42.492 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 23 10:26:42 smithi119 ceph-mon[32608]: mgrmap e22: x(active, since 1.02192s), standbys: z, y 2024-04-23T10:26:42.516 INFO:teuthology.run_tasks:Running task full_sequential_finally... 2024-04-23T10:26:42.526 INFO:teuthology.run_tasks:Running task background_exec... 2024-04-23T10:26:42.535 INFO:teuthology.task.background_exec:Running background command on role mon.a host ubuntu@smithi114.front.sepia.ceph.com 2024-04-23T10:26:42.535 DEBUG:teuthology.orchestra.run.smithi114:> sudo TESTDIR=/home/ubuntu/cephtest daemon-helper kill --kill-group bash -c 'while sleep 13; do ceph tell mds.cephfs:0 flush journal; done' 2024-04-23T10:26:42.560 INFO:teuthology.run_tasks:Running task quiescer... 2024-04-23T10:26:42.571 DEBUG:teuthology.orchestra.run.smithi114:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-04-23T10:26:42.672 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:42.670+0000 7f6592c6b640 1 -- 172.21.15.114:0/3277251159 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f658c100e90 msgr2=0x7f658c1012b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T10:26:42.672 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:42.670+0000 7f6592c6b640 1 --2- 172.21.15.114:0/3277251159 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f658c100e90 0x7f658c1012b0 secure :-1 s=READY pgs=432 cs=0 l=1 rev1=1 crypto rx=0x7f6584009ca0 tx=0x7f658402a8c0 comp rx=0 tx=0).stop 2024-04-23T10:26:42.673 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:42.670+0000 7f6592c6b640 1 -- 172.21.15.114:0/3277251159 shutdown_connections 2024-04-23T10:26:42.673 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:42.670+0000 7f6592c6b640 1 --2- 172.21.15.114:0/3277251159 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f658c1029a0 0x7f658c1092c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:26:42.673 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:42.670+0000 7f6592c6b640 1 --2- 172.21.15.114:0/3277251159 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f658c101fe0 0x7f658c102460 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:26:42.673 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:42.670+0000 7f6592c6b640 1 --2- 172.21.15.114:0/3277251159 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f658c100e90 0x7f658c1012b0 unknown :-1 s=CLOSED pgs=432 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:26:42.673 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:42.670+0000 7f6592c6b640 1 -- 172.21.15.114:0/3277251159 >> 172.21.15.114:0/3277251159 conn(0x7f658c0fc400 msgr2=0x7f658c0fe840 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T10:26:42.673 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:42.670+0000 7f6592c6b640 1 -- 172.21.15.114:0/3277251159 shutdown_connections 2024-04-23T10:26:42.673 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:42.670+0000 7f6592c6b640 1 -- 172.21.15.114:0/3277251159 wait complete. 2024-04-23T10:26:42.673 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:42.671+0000 7f6592c6b640 1 Processor -- start 2024-04-23T10:26:42.673 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:42.671+0000 7f6592c6b640 1 -- start start 2024-04-23T10:26:42.673 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:42.671+0000 7f6592c6b640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f658c100e90 0x7f658c19a410 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T10:26:42.673 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:42.671+0000 7f6592c6b640 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f658c101fe0 0x7f658c19a950 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T10:26:42.673 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:42.671+0000 7f6592c6b640 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f658c1029a0 0x7f658c1a19c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T10:26:42.673 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:42.671+0000 7f6592c6b640 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x7f658c10c9e0 con 0x7f658c100e90 2024-04-23T10:26:42.673 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:42.671+0000 7f6592c6b640 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 -- 0x7f658c10c860 con 0x7f658c1029a0 2024-04-23T10:26:42.674 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:42.671+0000 7f6592c6b640 1 -- --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_getmap magic: 0 -- 0x7f658c10cb60 con 0x7f658c101fe0 2024-04-23T10:26:42.674 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:42.671+0000 7f6590a07640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f658c100e90 0x7f658c19a410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T10:26:42.674 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:42.671+0000 7f6590a07640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f658c100e90 0x7f658c19a410 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.114:40794/0 (socket says 172.21.15.114:40794) 2024-04-23T10:26:42.674 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:42.671+0000 7f6590a07640 1 -- 172.21.15.114:0/856911319 learned_addr learned my addr 172.21.15.114:0/856911319 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-04-23T10:26:42.674 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:42.671+0000 7f658bfff640 1 --2- 172.21.15.114:0/856911319 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f658c101fe0 0x7f658c19a950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T10:26:42.674 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:42.671+0000 7f6591208640 1 --2- 172.21.15.114:0/856911319 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f658c1029a0 0x7f658c1a19c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T10:26:42.674 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:42.671+0000 7f6590a07640 1 -- 172.21.15.114:0/856911319 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f658c101fe0 msgr2=0x7f658c19a950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T10:26:42.674 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:42.671+0000 7f6590a07640 1 --2- 172.21.15.114:0/856911319 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f658c101fe0 0x7f658c19a950 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:26:42.674 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:42.671+0000 7f6590a07640 1 -- 172.21.15.114:0/856911319 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f658c1029a0 msgr2=0x7f658c1a19c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T10:26:42.674 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:42.671+0000 7f6590a07640 1 --2- 172.21.15.114:0/856911319 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f658c1029a0 0x7f658c1a19c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:26:42.674 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:42.671+0000 7f6590a07640 1 -- 172.21.15.114:0/856911319 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f658c1a2030 con 0x7f658c100e90 2024-04-23T10:26:42.674 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:42.672+0000 7f6590a07640 1 --2- 172.21.15.114:0/856911319 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f658c100e90 0x7f658c19a410 secure :-1 s=READY pgs=433 cs=0 l=1 rev1=1 crypto rx=0x7f6584006010 tx=0x7f65840317a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T10:26:42.674 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:42.672+0000 7f6589ffb640 1 -- 172.21.15.114:0/856911319 <== mon.0 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6584003c30 con 0x7f658c100e90 2024-04-23T10:26:42.675 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:42.672+0000 7f6589ffb640 1 -- 172.21.15.114:0/856911319 <== mon.0 v2:172.21.15.114:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f6584003dd0 con 0x7f658c100e90 2024-04-23T10:26:42.675 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:42.672+0000 7f6592c6b640 1 -- 172.21.15.114:0/856911319 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f658c1a22c0 con 0x7f658c100e90 2024-04-23T10:26:42.675 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:42.672+0000 7f6589ffb640 1 -- 172.21.15.114:0/856911319 <== mon.0 v2:172.21.15.114:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f658402fc20 con 0x7f658c100e90 2024-04-23T10:26:42.675 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:42.672+0000 7f6592c6b640 1 -- 172.21.15.114:0/856911319 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f658c1a2810 con 0x7f658c100e90 2024-04-23T10:26:42.676 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:42.673+0000 7f6592c6b640 1 -- 172.21.15.114:0/856911319 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f658c1a2a70 con 0x7f658c100e90 2024-04-23T10:26:42.676 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:42.674+0000 7f6589ffb640 1 -- 172.21.15.114:0/856911319 <== mon.0 v2:172.21.15.114:3300/0 4 ==== mgrmap(e 22) ==== 138458+0+0 (secure 0 0 0) 0x7f65840046a0 con 0x7f658c100e90 2024-04-23T10:26:42.676 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:42.675+0000 7f6589ffb640 1 --2- 172.21.15.114:0/856911319 >> [v2:172.21.15.114:6800/1395693466,v1:172.21.15.114:6801/1395693466] conn(0x7f65780a39b0 0x7f65780a5e70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T10:26:42.676 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:42.675+0000 7f658bfff640 1 --2- 172.21.15.114:0/856911319 >> [v2:172.21.15.114:6800/1395693466,v1:172.21.15.114:6801/1395693466] conn(0x7f65780a39b0 0x7f65780a5e70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T10:26:42.676 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:42.675+0000 7f6589ffb640 1 -- 172.21.15.114:0/856911319 <== mon.0 v2:172.21.15.114:3300/0 5 ==== osd_map(90..90 src has 1..90) ==== 8527+0+0 (secure 0 0 0) 0x7f65840f15f0 con 0x7f658c100e90 2024-04-23T10:26:42.676 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:42.676+0000 7f658bfff640 1 --2- 172.21.15.114:0/856911319 >> [v2:172.21.15.114:6800/1395693466,v1:172.21.15.114:6801/1395693466] conn(0x7f65780a39b0 0x7f65780a5e70 secure :-1 s=READY pgs=40 cs=0 l=1 rev1=1 crypto rx=0x7f658c19b920 tx=0x7f6574005f90 comp rx=0 tx=0).ready entity=mgr.15291 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T10:26:42.679 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:42.681+0000 7f6589ffb640 1 -- 172.21.15.114:0/856911319 <== mon.0 v2:172.21.15.114:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f65840be230 con 0x7f658c100e90 2024-04-23T10:26:42.865 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:42.866+0000 7f6592c6b640 1 -- 172.21.15.114:0/856911319 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f658c1a2c40 con 0x7f658c100e90 2024-04-23T10:26:42.867 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:42.868+0000 7f6589ffb640 1 -- 172.21.15.114:0/856911319 <== mon.0 v2:172.21.15.114:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v90) ==== 74+0+23198 (secure 0 0 0) 0x7f65840bdb30 con 0x7f658c100e90 2024-04-23T10:26:42.867 INFO:teuthology.orchestra.run.smithi114.stdout: 2024-04-23T10:26:42.867 INFO:teuthology.orchestra.run.smithi114.stdout:{"epoch":90,"fsid":"aa240572-0159-11ef-bc93-c7b262605968","created":"2024-04-23T10:16:53.351136+0000","modified":"2024-04-23T10:26:40.272696+0000","last_up_change":"2024-04-23T10:23:54.153269+0000","last_in_change":"2024-04-23T10:23:32.963085+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":26,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-04-23T10:20:11.970297+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":12,"score_stable":12,"optimal_score":0.25,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-04-23T10:24:41.218540+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"86","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.8799999952316284,"score_stable":1.8799999952316284,"optimal_score":1,"raw_score_acting":1.8799999952316284,"raw_score_stable":1.8799999952316284,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-04-23T10:24:42.341799+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"87","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"fd64adde-9445-45a4-a171-ef656e2c5709","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":88,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6802","nonce":736366878},{"type":"v1","addr":"172.21.15.114:6803","nonce":736366878}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6804","nonce":736366878},{"type":"v1","addr":"172.21.15.114:6805","nonce":736366878}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6808","nonce":736366878},{"type":"v1","addr":"172.21.15.114:6809","nonce":736366878}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6806","nonce":736366878},{"type":"v1","addr":"172.21.15.114:6807","nonce":736366878}]},"public_addr":"172.21.15.114:6803/736366878","cluster_addr":"172.21.15.114:6805/736366878","heartbeat_back_addr":"172.21.15.114:6809/736366878","heartbeat_front_addr":"172.21.15.114:6807/736366878","state":["exists","up"]},{"osd":1,"uuid":"e279a1e6-2702-472a-930a-e08ece76a861","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":85,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6800","nonce":1133540779},{"type":"v1","addr":"172.21.15.119:6801","nonce":1133540779}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6802","nonce":1133540779},{"type":"v1","addr":"172.21.15.119:6803","nonce":1133540779}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6806","nonce":1133540779},{"type":"v1","addr":"172.21.15.119:6807","nonce":1133540779}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6804","nonce":1133540779},{"type":"v1","addr":"172.21.15.119:6805","nonce":1133540779}]},"public_addr":"172.21.15.119:6801/1133540779","cluster_addr":"172.21.15.119:6803/1133540779","heartbeat_back_addr":"172.21.15.119:6807/1133540779","heartbeat_front_addr":"172.21.15.119:6805/1133540779","state":["exists","up"]},{"osd":2,"uuid":"f4c9f2b4-ff49-4e14-89cd-b477feeb9382","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":85,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6800","nonce":799652511},{"type":"v1","addr":"172.21.15.204:6801","nonce":799652511}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6802","nonce":799652511},{"type":"v1","addr":"172.21.15.204:6803","nonce":799652511}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6806","nonce":799652511},{"type":"v1","addr":"172.21.15.204:6807","nonce":799652511}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6804","nonce":799652511},{"type":"v1","addr":"172.21.15.204:6805","nonce":799652511}]},"public_addr":"172.21.15.204:6801/799652511","cluster_addr":"172.21.15.204:6803/799652511","heartbeat_back_addr":"172.21.15.204:6807/799652511","heartbeat_front_addr":"172.21.15.204:6805/799652511","state":["exists","up"]},{"osd":3,"uuid":"52d2a566-e71c-4bb3-8d6c-5abdd43177c2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":25,"up_thru":88,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6810","nonce":4275595091},{"type":"v1","addr":"172.21.15.114:6811","nonce":4275595091}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6812","nonce":4275595091},{"type":"v1","addr":"172.21.15.114:6813","nonce":4275595091}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6816","nonce":4275595091},{"type":"v1","addr":"172.21.15.114:6817","nonce":4275595091}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6814","nonce":4275595091},{"type":"v1","addr":"172.21.15.114:6815","nonce":4275595091}]},"public_addr":"172.21.15.114:6811/4275595091","cluster_addr":"172.21.15.114:6813/4275595091","heartbeat_back_addr":"172.21.15.114:6817/4275595091","heartbeat_front_addr":"172.21.15.114:6815/4275595091","state":["exists","up"]},{"osd":4,"uuid":"d911a7c1-4bdc-4d28-b8a6-73896a6e7de6","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":33,"up_thru":85,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6808","nonce":1611007172},{"type":"v1","addr":"172.21.15.119:6809","nonce":1611007172}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6810","nonce":1611007172},{"type":"v1","addr":"172.21.15.119:6811","nonce":1611007172}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6814","nonce":1611007172},{"type":"v1","addr":"172.21.15.119:6815","nonce":1611007172}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6812","nonce":1611007172},{"type":"v1","addr":"172.21.15.119:6813","nonce":1611007172}]},"public_addr":"172.21.15.119:6809/1611007172","cluster_addr":"172.21.15.119:6811/1611007172","heartbeat_back_addr":"172.21.15.119:6815/1611007172","heartbeat_front_addr":"172.21.15.119:6813/1611007172","state":["exists","up"]},{"osd":5,"uuid":"f7590e1a-e93c-4c29-b15f-38ee52d1f921","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":39,"up_thru":85,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6808","nonce":1427563312},{"type":"v1","addr":"172.21.15.204:6809","nonce":1427563312}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6810","nonce":1427563312},{"type":"v1","addr":"172.21.15.204:6811","nonce":1427563312}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6814","nonce":1427563312},{"type":"v1","addr":"172.21.15.204:6815","nonce":1427563312}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6812","nonce":1427563312},{"type":"v1","addr":"172.21.15.204:6813","nonce":1427563312}]},"public_addr":"172.21.15.204:6809/1427563312","cluster_addr":"172.21.15.204:6811/1427563312","heartbeat_back_addr":"172.21.15.204:6815/1427563312","heartbeat_front_addr":"172.21.15.204:6813/1427563312","state":["exists","up"]},{"osd":6,"uuid":"534d20e8-3e54-4db8-a845-6119f49c7a78","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":47,"up_thru":88,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6818","nonce":4072912544},{"type":"v1","addr":"172.21.15.114:6819","nonce":4072912544}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6820","nonce":4072912544},{"type":"v1","addr":"172.21.15.114:6821","nonce":4072912544}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6824","nonce":4072912544},{"type":"v1","addr":"172.21.15.114:6825","nonce":4072912544}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6822","nonce":4072912544},{"type":"v1","addr":"172.21.15.114:6823","nonce":4072912544}]},"public_addr":"172.21.15.114:6819/4072912544","cluster_addr":"172.21.15.114:6821/4072912544","heartbeat_back_addr":"172.21.15.114:6825/4072912544","heartbeat_front_addr":"172.21.15.114:6823/4072912544","state":["exists","up"]},{"osd":7,"uuid":"f0ec22b8-1b48-4859-9d65-b00403aff3ad","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":53,"up_thru":88,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6816","nonce":2237458793},{"type":"v1","addr":"172.21.15.119:6817","nonce":2237458793}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6818","nonce":2237458793},{"type":"v1","addr":"172.21.15.119:6819","nonce":2237458793}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6822","nonce":2237458793},{"type":"v1","addr":"172.21.15.119:6823","nonce":2237458793}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6820","nonce":2237458793},{"type":"v1","addr":"172.21.15.119:6821","nonce":2237458793}]},"public_addr":"172.21.15.119:6817/2237458793","cluster_addr":"172.21.15.119:6819/2237458793","heartbeat_back_addr":"172.21.15.119:6823/2237458793","heartbeat_front_addr":"172.21.15.119:6821/2237458793","state":["exists","up"]},{"osd":8,"uuid":"3ac89686-6d76-4cef-9089-ac29e4b3bcac","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":62,"up_thru":85,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6816","nonce":1426318231},{"type":"v1","addr":"172.21.15.204:6817","nonce":1426318231}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6818","nonce":1426318231},{"type":"v1","addr":"172.21.15.204:6819","nonce":1426318231}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6822","nonce":1426318231},{"type":"v1","addr":"172.21.15.204:6823","nonce":1426318231}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6820","nonce":1426318231},{"type":"v1","addr":"172.21.15.204:6821","nonce":1426318231}]},"public_addr":"172.21.15.204:6817/1426318231","cluster_addr":"172.21.15.204:6819/1426318231","heartbeat_back_addr":"172.21.15.204:6823/1426318231","heartbeat_front_addr":"172.21.15.204:6821/1426318231","state":["exists","up"]},{"osd":9,"uuid":"a80ffe4d-1e98-460f-b046-c4754285cc5a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":69,"up_thru":88,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6826","nonce":2750635721},{"type":"v1","addr":"172.21.15.114:6827","nonce":2750635721}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6828","nonce":2750635721},{"type":"v1","addr":"172.21.15.114:6829","nonce":2750635721}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6832","nonce":2750635721},{"type":"v1","addr":"172.21.15.114:6833","nonce":2750635721}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6830","nonce":2750635721},{"type":"v1","addr":"172.21.15.114:6831","nonce":2750635721}]},"public_addr":"172.21.15.114:6827/2750635721","cluster_addr":"172.21.15.114:6829/2750635721","heartbeat_back_addr":"172.21.15.114:6833/2750635721","heartbeat_front_addr":"172.21.15.114:6831/2750635721","state":["exists","up"]},{"osd":10,"uuid":"cd75a579-bf12-4600-ac09-a8bfce772ced","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":74,"up_thru":85,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6824","nonce":1400187701},{"type":"v1","addr":"172.21.15.119:6825","nonce":1400187701}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6826","nonce":1400187701},{"type":"v1","addr":"172.21.15.119:6827","nonce":1400187701}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6830","nonce":1400187701},{"type":"v1","addr":"172.21.15.119:6831","nonce":1400187701}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6828","nonce":1400187701},{"type":"v1","addr":"172.21.15.119:6829","nonce":1400187701}]},"public_addr":"172.21.15.119:6825/1400187701","cluster_addr":"172.21.15.119:6827/1400187701","heartbeat_back_addr":"172.21.15.119:6831/1400187701","heartbeat_front_addr":"172.21.15.119:6829/1400187701","state":["exists","up"]},{"osd":11,"uuid":"53f037a5-3ec0-483a-80a1-dfee923a68e5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":82,"up_thru":88,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6824","nonce":2407298490},{"type":"v1","addr":"172.21.15.204:6825","nonce":2407298490}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6826","nonce":2407298490},{"type":"v1","addr":"172.21.15.204:6827","nonce":2407298490}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6830","nonce":2407298490},{"type":"v1","addr":"172.21.15.204:6831","nonce":2407298490}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6828","nonce":2407298490},{"type":"v1","addr":"172.21.15.204:6829","nonce":2407298490}]},"public_addr":"172.21.15.204:6825/2407298490","cluster_addr":"172.21.15.204:6827/2407298490","heartbeat_back_addr":"172.21.15.204:6831/2407298490","heartbeat_front_addr":"172.21.15.204:6829/2407298490","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-23T10:19:18.401147+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-23T10:19:42.611299+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-23T10:20:07.277367+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-23T10:20:31.208368+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-23T10:20:55.866225+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-23T10:21:19.961237+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-23T10:21:44.480783+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-23T10:22:09.173641+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-23T10:22:34.030328+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-23T10:22:59.652293+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-23T10:23:25.659673+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-23T10:23:51.168171+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"2.e","mappings":[{"from":2,"to":4}]},{"pgid":"2.1d","mappings":[{"from":8,"to":11}]},{"pgid":"3.e","mappings":[{"from":5,"to":9}]},{"pgid":"3.12","mappings":[{"from":3,"to":9}]},{"pgid":"3.17","mappings":[{"from":4,"to":6}]},{"pgid":"3.23","mappings":[{"from":4,"to":7}]},{"pgid":"3.27","mappings":[{"from":4,"to":6}]},{"pgid":"3.31","mappings":[{"from":4,"to":8}]},{"pgid":"3.38","mappings":[{"from":3,"to":9}]},{"pgid":"3.39","mappings":[{"from":5,"to":8}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.114:0/480720790":"2024-04-24T10:26:40.272662+0000","172.21.15.114:6801/465333808":"2024-04-24T10:26:40.272662+0000","172.21.15.114:6800/465333808":"2024-04-24T10:26:40.272662+0000","172.21.15.114:0/1787801099":"2024-04-24T10:26:40.272662+0000","172.21.15.114:6801/1604678557":"2024-04-24T10:17:45.874609+0000","172.21.15.114:0/4025385854":"2024-04-24T10:17:45.874609+0000","172.21.15.114:0/2228273469":"2024-04-24T10:17:19.910348+0000","172.21.15.114:0/2713201622":"2024-04-24T10:26:40.272662+0000","172.21.15.114:6800/2831530838":"2024-04-24T10:17:19.910348+0000","172.21.15.114:0/637004095":"2024-04-24T10:26:40.272662+0000","172.21.15.114:0/1333447450":"2024-04-24T10:17:45.874609+0000","172.21.15.114:0/2233026840":"2024-04-24T10:17:19.910348+0000","172.21.15.114:0/1902281545":"2024-04-24T10:17:19.910348+0000","172.21.15.114:6801/2831530838":"2024-04-24T10:17:19.910348+0000","172.21.15.114:6800/1604678557":"2024-04-24T10:17:45.874609+0000","172.21.15.114:0/1623553811":"2024-04-24T10:17:45.874609+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-23T10:26:42.869 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:42.870+0000 7f6592c6b640 1 -- 172.21.15.114:0/856911319 >> [v2:172.21.15.114:6800/1395693466,v1:172.21.15.114:6801/1395693466] conn(0x7f65780a39b0 msgr2=0x7f65780a5e70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T10:26:42.869 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:42.870+0000 7f6592c6b640 1 --2- 172.21.15.114:0/856911319 >> [v2:172.21.15.114:6800/1395693466,v1:172.21.15.114:6801/1395693466] conn(0x7f65780a39b0 0x7f65780a5e70 secure :-1 s=READY pgs=40 cs=0 l=1 rev1=1 crypto rx=0x7f658c19b920 tx=0x7f6574005f90 comp rx=0 tx=0).stop 2024-04-23T10:26:42.869 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:42.870+0000 7f6592c6b640 1 -- 172.21.15.114:0/856911319 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f658c100e90 msgr2=0x7f658c19a410 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T10:26:42.869 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:42.870+0000 7f6592c6b640 1 --2- 172.21.15.114:0/856911319 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f658c100e90 0x7f658c19a410 secure :-1 s=READY pgs=433 cs=0 l=1 rev1=1 crypto rx=0x7f6584006010 tx=0x7f65840317a0 comp rx=0 tx=0).stop 2024-04-23T10:26:42.869 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:42.870+0000 7f6592c6b640 1 -- 172.21.15.114:0/856911319 shutdown_connections 2024-04-23T10:26:42.870 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:42.870+0000 7f6592c6b640 1 --2- 172.21.15.114:0/856911319 >> [v2:172.21.15.114:6800/1395693466,v1:172.21.15.114:6801/1395693466] conn(0x7f65780a39b0 0x7f65780a5e70 unknown :-1 s=CLOSED pgs=40 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:26:42.870 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:42.870+0000 7f6592c6b640 1 --2- 172.21.15.114:0/856911319 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f658c1029a0 0x7f658c1a19c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:26:42.870 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:42.870+0000 7f6592c6b640 1 --2- 172.21.15.114:0/856911319 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f658c101fe0 0x7f658c19a950 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:26:42.870 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:42.870+0000 7f6592c6b640 1 --2- 172.21.15.114:0/856911319 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f658c100e90 0x7f658c19a410 unknown :-1 s=CLOSED pgs=433 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:26:42.870 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:42.870+0000 7f6592c6b640 1 -- 172.21.15.114:0/856911319 >> 172.21.15.114:0/856911319 conn(0x7f658c0fc400 msgr2=0x7f658c0fdf30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T10:26:42.870 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:42.870+0000 7f6592c6b640 1 -- 172.21.15.114:0/856911319 shutdown_connections 2024-04-23T10:26:42.870 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:42.870+0000 7f6592c6b640 1 -- 172.21.15.114:0/856911319 wait complete. 2024-04-23T10:26:42.880 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-04-23T10:20:11.970297+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '22', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 12, 'score_stable': 12, 'optimal_score': 0.25, 'raw_score_acting': 3, 'raw_score_stable': 3, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 2, 'pool_name': 'cephfs_metadata', 'create_time': '2024-04-23T10:24:41.218540+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '86', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.8799999952316284, 'score_stable': 1.8799999952316284, 'optimal_score': 1, 'raw_score_acting': 1.8799999952316284, 'raw_score_stable': 1.8799999952316284, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 3, 'pool_name': 'cephfs_data', 'create_time': '2024-04-23T10:24:42.341799+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '87', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-04-23T10:26:42.881 DEBUG:teuthology.orchestra.run.smithi114:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-04-23T10:26:42.985 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:42.985+0000 7fe6bcea7640 1 -- 172.21.15.114:0/1105090849 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fe6b8107c80 msgr2=0x7fe6b810a0d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T10:26:42.985 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:42.985+0000 7fe6bcea7640 1 --2- 172.21.15.114:0/1105090849 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fe6b8107c80 0x7fe6b810a0d0 secure :-1 s=READY pgs=434 cs=0 l=1 rev1=1 crypto rx=0x7fe6b0009ca0 tx=0x7fe6b002a8c0 comp rx=0 tx=0).stop 2024-04-23T10:26:42.985 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:42.986+0000 7fe6bcea7640 1 -- 172.21.15.114:0/1105090849 shutdown_connections 2024-04-23T10:26:42.985 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:42.986+0000 7fe6bcea7640 1 --2- 172.21.15.114:0/1105090849 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fe6b8107c80 0x7fe6b810a0d0 unknown :-1 s=CLOSED pgs=434 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:26:42.985 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:42.986+0000 7fe6bcea7640 1 --2- 172.21.15.114:0/1105090849 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fe6b8105350 0x7fe6b8107740 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:26:42.985 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:42.986+0000 7fe6bcea7640 1 --2- 172.21.15.114:0/1105090849 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fe6b8102a20 0x7fe6b8104e10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:26:42.985 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:42.986+0000 7fe6bcea7640 1 -- 172.21.15.114:0/1105090849 >> 172.21.15.114:0/1105090849 conn(0x7fe6b80fc410 msgr2=0x7fe6b80fe850 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T10:26:42.985 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:42.986+0000 7fe6bcea7640 1 -- 172.21.15.114:0/1105090849 shutdown_connections 2024-04-23T10:26:42.985 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:42.986+0000 7fe6bcea7640 1 -- 172.21.15.114:0/1105090849 wait complete. 2024-04-23T10:26:42.988 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:42.987+0000 7fe6bcea7640 1 Processor -- start 2024-04-23T10:26:42.988 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:42.987+0000 7fe6bcea7640 1 -- start start 2024-04-23T10:26:42.988 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:42.987+0000 7fe6bcea7640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fe6b8102a20 0x7fe6b81981c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T10:26:42.988 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:42.987+0000 7fe6bcea7640 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fe6b8105350 0x7fe6b8198700 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T10:26:42.988 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:42.987+0000 7fe6bcea7640 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fe6b8107c80 0x7fe6b819f770 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T10:26:42.988 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:42.987+0000 7fe6bcea7640 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x7fe6b810dae0 con 0x7fe6b8102a20 2024-04-23T10:26:42.988 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:42.987+0000 7fe6bcea7640 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 -- 0x7fe6b810d960 con 0x7fe6b8107c80 2024-04-23T10:26:42.988 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:42.987+0000 7fe6bcea7640 1 -- --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_getmap magic: 0 -- 0x7fe6b810dc60 con 0x7fe6b8105350 2024-04-23T10:26:42.988 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:42.987+0000 7fe6b659c640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fe6b8102a20 0x7fe6b81981c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T10:26:42.988 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:42.987+0000 7fe6b659c640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fe6b8102a20 0x7fe6b81981c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.114:40810/0 (socket says 172.21.15.114:40810) 2024-04-23T10:26:42.988 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:42.987+0000 7fe6b659c640 1 -- 172.21.15.114:0/2108459556 learned_addr learned my addr 172.21.15.114:0/2108459556 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-04-23T10:26:42.989 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:42.987+0000 7fe6b6d9d640 1 --2- 172.21.15.114:0/2108459556 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fe6b8107c80 0x7fe6b819f770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T10:26:42.989 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:42.987+0000 7fe6b5d9b640 1 --2- 172.21.15.114:0/2108459556 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fe6b8105350 0x7fe6b8198700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T10:26:42.989 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:42.987+0000 7fe6b659c640 1 -- 172.21.15.114:0/2108459556 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fe6b8105350 msgr2=0x7fe6b8198700 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T10:26:42.989 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:42.987+0000 7fe6b659c640 1 --2- 172.21.15.114:0/2108459556 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fe6b8105350 0x7fe6b8198700 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:26:42.989 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:42.987+0000 7fe6b659c640 1 -- 172.21.15.114:0/2108459556 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fe6b8107c80 msgr2=0x7fe6b819f770 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T10:26:42.989 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:42.987+0000 7fe6b659c640 1 --2- 172.21.15.114:0/2108459556 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fe6b8107c80 0x7fe6b819f770 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:26:42.989 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:42.987+0000 7fe6b659c640 1 -- 172.21.15.114:0/2108459556 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe6b819fde0 con 0x7fe6b8102a20 2024-04-23T10:26:42.989 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:42.988+0000 7fe6b659c640 1 --2- 172.21.15.114:0/2108459556 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fe6b8102a20 0x7fe6b81981c0 secure :-1 s=READY pgs=435 cs=0 l=1 rev1=1 crypto rx=0x7fe6a800d950 tx=0x7fe6a800de10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T10:26:42.989 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:42.988+0000 7fe6a77fe640 1 -- 172.21.15.114:0/2108459556 <== mon.0 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe6a8017030 con 0x7fe6b8102a20 2024-04-23T10:26:42.989 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:42.988+0000 7fe6a77fe640 1 -- 172.21.15.114:0/2108459556 <== mon.0 v2:172.21.15.114:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fe6a8004020 con 0x7fe6b8102a20 2024-04-23T10:26:42.989 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:42.988+0000 7fe6a77fe640 1 -- 172.21.15.114:0/2108459556 <== mon.0 v2:172.21.15.114:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe6a80047a0 con 0x7fe6b8102a20 2024-04-23T10:26:42.989 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:42.988+0000 7fe6bcea7640 1 -- 172.21.15.114:0/2108459556 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fe6b81a00d0 con 0x7fe6b8102a20 2024-04-23T10:26:42.989 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:42.988+0000 7fe6bcea7640 1 -- 172.21.15.114:0/2108459556 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fe6b81a05d0 con 0x7fe6b8102a20 2024-04-23T10:26:42.989 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:42.989+0000 7fe6bcea7640 1 -- 172.21.15.114:0/2108459556 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fe6b80fdf00 con 0x7fe6b8102a20 2024-04-23T10:26:42.990 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:42.990+0000 7fe6a77fe640 1 -- 172.21.15.114:0/2108459556 <== mon.0 v2:172.21.15.114:3300/0 4 ==== mgrmap(e 22) ==== 138458+0+0 (secure 0 0 0) 0x7fe6a8009a30 con 0x7fe6b8102a20 2024-04-23T10:26:42.990 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:42.991+0000 7fe6a77fe640 1 --2- 172.21.15.114:0/2108459556 >> [v2:172.21.15.114:6800/1395693466,v1:172.21.15.114:6801/1395693466] conn(0x7fe6a00a39b0 0x7fe6a00a5e70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T10:26:42.990 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:42.991+0000 7fe6b5d9b640 1 --2- 172.21.15.114:0/2108459556 >> [v2:172.21.15.114:6800/1395693466,v1:172.21.15.114:6801/1395693466] conn(0x7fe6a00a39b0 0x7fe6a00a5e70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T10:26:42.990 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:42.991+0000 7fe6a77fe640 1 -- 172.21.15.114:0/2108459556 <== mon.0 v2:172.21.15.114:3300/0 5 ==== osd_map(90..90 src has 1..90) ==== 8527+0+0 (secure 0 0 0) 0x7fe6a80d01a0 con 0x7fe6b8102a20 2024-04-23T10:26:42.991 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:42.992+0000 7fe6b5d9b640 1 --2- 172.21.15.114:0/2108459556 >> [v2:172.21.15.114:6800/1395693466,v1:172.21.15.114:6801/1395693466] conn(0x7fe6a00a39b0 0x7fe6a00a5e70 secure :-1 s=READY pgs=41 cs=0 l=1 rev1=1 crypto rx=0x7fe6b81996d0 tx=0x7fe6ac00b390 comp rx=0 tx=0).ready entity=mgr.15291 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T10:26:42.996 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:42.998+0000 7fe6a77fe640 1 -- 172.21.15.114:0/2108459556 <== mon.0 v2:172.21.15.114:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7fe6a809cd50 con 0x7fe6b8102a20 2024-04-23T10:26:43.024 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:43.024+0000 7fe6a77fe640 1 -- 172.21.15.114:0/2108459556 <== mon.0 v2:172.21.15.114:3300/0 7 ==== mgrmap(e 23) ==== 138767+0+0 (secure 0 0 0) 0x7fe6a8098cc0 con 0x7fe6b8102a20 2024-04-23T10:26:43.195 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:43.196+0000 7fe6bcea7640 1 -- 172.21.15.114:0/2108459556 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) -- 0x7fe6b802d870 con 0x7fe6b8102a20 2024-04-23T10:26:43.195 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:43.197+0000 7fe6a77fe640 1 -- 172.21.15.114:0/2108459556 <== mon.0 v2:172.21.15.114:3300/0 8 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v90) ==== 93+0+10 (secure 0 0 0) 0x7fe6a8095b80 con 0x7fe6b8102a20 2024-04-23T10:26:43.196 INFO:teuthology.orchestra.run.smithi114.stdout:pg_num: 1 2024-04-23T10:26:43.199 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:43.200+0000 7fe6bcea7640 1 -- 172.21.15.114:0/2108459556 >> [v2:172.21.15.114:6800/1395693466,v1:172.21.15.114:6801/1395693466] conn(0x7fe6a00a39b0 msgr2=0x7fe6a00a5e70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T10:26:43.199 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:43.200+0000 7fe6bcea7640 1 --2- 172.21.15.114:0/2108459556 >> [v2:172.21.15.114:6800/1395693466,v1:172.21.15.114:6801/1395693466] conn(0x7fe6a00a39b0 0x7fe6a00a5e70 secure :-1 s=READY pgs=41 cs=0 l=1 rev1=1 crypto rx=0x7fe6b81996d0 tx=0x7fe6ac00b390 comp rx=0 tx=0).stop 2024-04-23T10:26:43.199 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:43.200+0000 7fe6bcea7640 1 -- 172.21.15.114:0/2108459556 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fe6b8102a20 msgr2=0x7fe6b81981c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T10:26:43.199 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:43.200+0000 7fe6bcea7640 1 --2- 172.21.15.114:0/2108459556 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fe6b8102a20 0x7fe6b81981c0 secure :-1 s=READY pgs=435 cs=0 l=1 rev1=1 crypto rx=0x7fe6a800d950 tx=0x7fe6a800de10 comp rx=0 tx=0).stop 2024-04-23T10:26:43.199 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:43.200+0000 7fe6bcea7640 1 -- 172.21.15.114:0/2108459556 shutdown_connections 2024-04-23T10:26:43.199 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:43.200+0000 7fe6bcea7640 1 --2- 172.21.15.114:0/2108459556 >> [v2:172.21.15.114:6800/1395693466,v1:172.21.15.114:6801/1395693466] conn(0x7fe6a00a39b0 0x7fe6a00a5e70 unknown :-1 s=CLOSED pgs=41 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:26:43.199 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:43.200+0000 7fe6bcea7640 1 --2- 172.21.15.114:0/2108459556 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fe6b8107c80 0x7fe6b819f770 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:26:43.199 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:43.200+0000 7fe6bcea7640 1 --2- 172.21.15.114:0/2108459556 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fe6b8105350 0x7fe6b8198700 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:26:43.200 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:43.200+0000 7fe6bcea7640 1 --2- 172.21.15.114:0/2108459556 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fe6b8102a20 0x7fe6b81981c0 unknown :-1 s=CLOSED pgs=435 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:26:43.200 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:43.200+0000 7fe6bcea7640 1 -- 172.21.15.114:0/2108459556 >> 172.21.15.114:0/2108459556 conn(0x7fe6b80fc410 msgr2=0x7fe6b8103460 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T10:26:43.200 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:43.200+0000 7fe6bcea7640 1 -- 172.21.15.114:0/2108459556 shutdown_connections 2024-04-23T10:26:43.200 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:43.200+0000 7fe6bcea7640 1 -- 172.21.15.114:0/2108459556 wait complete. 2024-04-23T10:26:43.213 INFO:journalctl@ceph.mon.a.smithi114.stdout:Apr 23 10:26:43 smithi114 ceph-mon[29255]: from='client.15324 -' entity='client.admin' cmd=[{"prefix": "fs snap-schedule add", "fs": "cephfs", "path": "/", "snap_schedule": "1m", "target": ["mon-mgr", ""]}]: dispatch 2024-04-23T10:26:43.213 INFO:journalctl@ceph.mon.a.smithi114.stdout:Apr 23 10:26:43 smithi114 ceph-mon[29255]: pgmap v3: 97 pgs: 97 active+clean; 1.1 MiB data, 375 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-23T10:26:43.213 INFO:journalctl@ceph.mon.a.smithi114.stdout:Apr 23 10:26:43 smithi114 ceph-mon[29255]: from='client.15366 -' entity='client.admin' cmd=[{"prefix": "fs snap-schedule retention add", "fs": "cephfs", "path": "/", "retention_spec_or_period": "6m3h", "target": ["mon-mgr", ""]}]: dispatch 2024-04-23T10:26:43.213 INFO:journalctl@ceph.mon.a.smithi114.stdout:Apr 23 10:26:43 smithi114 ceph-mon[29255]: from='client.15372 -' entity='client.admin' cmd=[{"prefix": "fs snap-schedule status", "fs": "cephfs", "path": "/", "target": ["mon-mgr", ""]}]: dispatch 2024-04-23T10:26:43.213 INFO:journalctl@ceph.mon.a.smithi114.stdout:Apr 23 10:26:43 smithi114 ceph-mon[29255]: from='mgr.15291 172.21.15.114:0/1095789187' entity='mgr.x' 2024-04-23T10:26:43.213 INFO:journalctl@ceph.mon.a.smithi114.stdout:Apr 23 10:26:43 smithi114 ceph-mon[29255]: from='mgr.15291 172.21.15.114:0/1095789187' entity='mgr.x' 2024-04-23T10:26:43.214 INFO:journalctl@ceph.mon.a.smithi114.stdout:Apr 23 10:26:43 smithi114 ceph-mon[29255]: from='mgr.15291 172.21.15.114:0/1095789187' entity='mgr.x' 2024-04-23T10:26:43.214 INFO:journalctl@ceph.mon.a.smithi114.stdout:Apr 23 10:26:43 smithi114 ceph-mon[29255]: from='mgr.15291 172.21.15.114:0/1095789187' entity='mgr.x' 2024-04-23T10:26:43.214 INFO:journalctl@ceph.mon.a.smithi114.stdout:Apr 23 10:26:43 smithi114 ceph-mon[29255]: from='mgr.15291 172.21.15.114:0/1095789187' entity='mgr.x' 2024-04-23T10:26:43.214 INFO:journalctl@ceph.mon.a.smithi114.stdout:Apr 23 10:26:43 smithi114 ceph-mon[29255]: from='mgr.15291 172.21.15.114:0/1095789187' entity='mgr.x' 2024-04-23T10:26:43.214 INFO:journalctl@ceph.mon.a.smithi114.stdout:Apr 23 10:26:43 smithi114 ceph-mon[29255]: from='client.? 172.21.15.114:0/856911319' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-23T10:26:43.215 DEBUG:teuthology.orchestra.run.smithi114:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2024-04-23T10:26:43.319 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:43.320+0000 7fcf90ea7640 1 -- 172.21.15.114:0/3199223442 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fcf8c107c50 msgr2=0x7fcf8c10a0a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T10:26:43.319 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:43.320+0000 7fcf90ea7640 1 --2- 172.21.15.114:0/3199223442 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fcf8c107c50 0x7fcf8c10a0a0 secure :-1 s=READY pgs=436 cs=0 l=1 rev1=1 crypto rx=0x7fcf84009ca0 tx=0x7fcf8402a8c0 comp rx=0 tx=0).stop 2024-04-23T10:26:43.320 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:43.320+0000 7fcf90ea7640 1 -- 172.21.15.114:0/3199223442 shutdown_connections 2024-04-23T10:26:43.320 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:43.320+0000 7fcf90ea7640 1 --2- 172.21.15.114:0/3199223442 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fcf8c107c50 0x7fcf8c10a0a0 unknown :-1 s=CLOSED pgs=436 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:26:43.320 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:43.320+0000 7fcf90ea7640 1 --2- 172.21.15.114:0/3199223442 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fcf8c105320 0x7fcf8c107710 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:26:43.320 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:43.320+0000 7fcf90ea7640 1 --2- 172.21.15.114:0/3199223442 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fcf8c1029f0 0x7fcf8c104de0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:26:43.320 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:43.320+0000 7fcf90ea7640 1 -- 172.21.15.114:0/3199223442 >> 172.21.15.114:0/3199223442 conn(0x7fcf8c0fc3e0 msgr2=0x7fcf8c0fe820 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T10:26:43.320 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:43.321+0000 7fcf90ea7640 1 -- 172.21.15.114:0/3199223442 shutdown_connections 2024-04-23T10:26:43.320 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:43.321+0000 7fcf90ea7640 1 -- 172.21.15.114:0/3199223442 wait complete. 2024-04-23T10:26:43.322 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:43.321+0000 7fcf90ea7640 1 Processor -- start 2024-04-23T10:26:43.322 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:43.321+0000 7fcf90ea7640 1 -- start start 2024-04-23T10:26:43.322 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:43.321+0000 7fcf90ea7640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fcf8c1029f0 0x7fcf8c198140 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T10:26:43.322 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:43.321+0000 7fcf90ea7640 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fcf8c105320 0x7fcf8c198680 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T10:26:43.322 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:43.321+0000 7fcf90ea7640 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fcf8c107c50 0x7fcf8c19f6f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T10:26:43.322 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:43.321+0000 7fcf90ea7640 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x7fcf8c10dac0 con 0x7fcf8c1029f0 2024-04-23T10:26:43.322 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:43.321+0000 7fcf90ea7640 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 -- 0x7fcf8c10d940 con 0x7fcf8c107c50 2024-04-23T10:26:43.322 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:43.321+0000 7fcf90ea7640 1 -- --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_getmap magic: 0 -- 0x7fcf8c10dc40 con 0x7fcf8c105320 2024-04-23T10:26:43.322 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:43.322+0000 7fcf8a59c640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fcf8c1029f0 0x7fcf8c198140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T10:26:43.322 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:43.322+0000 7fcf8ad9d640 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fcf8c107c50 0x7fcf8c19f6f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T10:26:43.322 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:43.322+0000 7fcf8a59c640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fcf8c1029f0 0x7fcf8c198140 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.114:40834/0 (socket says 172.21.15.114:40834) 2024-04-23T10:26:43.322 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:43.322+0000 7fcf8ad9d640 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fcf8c107c50 0x7fcf8c19f6f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.119:3300/0 says I am v2:172.21.15.114:58894/0 (socket says 172.21.15.114:58894) 2024-04-23T10:26:43.323 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:43.322+0000 7fcf8a59c640 1 -- 172.21.15.114:0/1382747356 learned_addr learned my addr 172.21.15.114:0/1382747356 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-04-23T10:26:43.323 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:43.322+0000 7fcf89d9b640 1 --2- 172.21.15.114:0/1382747356 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fcf8c105320 0x7fcf8c198680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T10:26:43.323 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:43.322+0000 7fcf8a59c640 1 -- 172.21.15.114:0/1382747356 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fcf8c105320 msgr2=0x7fcf8c198680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T10:26:43.323 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:43.322+0000 7fcf8a59c640 1 --2- 172.21.15.114:0/1382747356 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fcf8c105320 0x7fcf8c198680 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:26:43.323 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:43.322+0000 7fcf8a59c640 1 -- 172.21.15.114:0/1382747356 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fcf8c107c50 msgr2=0x7fcf8c19f6f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T10:26:43.323 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:43.322+0000 7fcf8a59c640 1 --2- 172.21.15.114:0/1382747356 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fcf8c107c50 0x7fcf8c19f6f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:26:43.323 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:43.322+0000 7fcf8a59c640 1 -- 172.21.15.114:0/1382747356 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fcf8c19fd60 con 0x7fcf8c1029f0 2024-04-23T10:26:43.323 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:43.322+0000 7fcf8a59c640 1 --2- 172.21.15.114:0/1382747356 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fcf8c1029f0 0x7fcf8c198140 secure :-1 s=READY pgs=437 cs=0 l=1 rev1=1 crypto rx=0x7fcf8000e9f0 tx=0x7fcf8000edf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T10:26:43.323 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:43.322+0000 7fcf7b7fe640 1 -- 172.21.15.114:0/1382747356 <== mon.0 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcf8000cd80 con 0x7fcf8c1029f0 2024-04-23T10:26:43.323 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:43.322+0000 7fcf7b7fe640 1 -- 172.21.15.114:0/1382747356 <== mon.0 v2:172.21.15.114:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fcf800054d0 con 0x7fcf8c1029f0 2024-04-23T10:26:43.323 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:43.322+0000 7fcf90ea7640 1 -- 172.21.15.114:0/1382747356 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fcf8c19fff0 con 0x7fcf8c1029f0 2024-04-23T10:26:43.323 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:43.322+0000 7fcf7b7fe640 1 -- 172.21.15.114:0/1382747356 <== mon.0 v2:172.21.15.114:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcf80010690 con 0x7fcf8c1029f0 2024-04-23T10:26:43.323 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:43.322+0000 7fcf90ea7640 1 -- 172.21.15.114:0/1382747356 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fcf8c1a0460 con 0x7fcf8c1029f0 2024-04-23T10:26:43.324 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:43.323+0000 7fcf90ea7640 1 -- 172.21.15.114:0/1382747356 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fcf8c0fded0 con 0x7fcf8c1029f0 2024-04-23T10:26:43.324 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:43.324+0000 7fcf7b7fe640 1 -- 172.21.15.114:0/1382747356 <== mon.0 v2:172.21.15.114:3300/0 4 ==== mgrmap(e 23) ==== 138767+0+0 (secure 0 0 0) 0x7fcf80004e90 con 0x7fcf8c1029f0 2024-04-23T10:26:43.324 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:43.325+0000 7fcf7b7fe640 1 --2- 172.21.15.114:0/1382747356 >> [v2:172.21.15.114:6800/1395693466,v1:172.21.15.114:6801/1395693466] conn(0x7fcf740a3d20 0x7fcf740a61e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T10:26:43.324 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:43.325+0000 7fcf89d9b640 1 --2- 172.21.15.114:0/1382747356 >> [v2:172.21.15.114:6800/1395693466,v1:172.21.15.114:6801/1395693466] conn(0x7fcf740a3d20 0x7fcf740a61e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T10:26:43.324 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:43.325+0000 7fcf7b7fe640 1 -- 172.21.15.114:0/1382747356 <== mon.0 v2:172.21.15.114:3300/0 5 ==== osd_map(90..90 src has 1..90) ==== 8527+0+0 (secure 0 0 0) 0x7fcf800d0be0 con 0x7fcf8c1029f0 2024-04-23T10:26:43.324 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:43.325+0000 7fcf89d9b640 1 --2- 172.21.15.114:0/1382747356 >> [v2:172.21.15.114:6800/1395693466,v1:172.21.15.114:6801/1395693466] conn(0x7fcf740a3d20 0x7fcf740a61e0 secure :-1 s=READY pgs=42 cs=0 l=1 rev1=1 crypto rx=0x7fcf8c199650 tx=0x7fcf7c005f90 comp rx=0 tx=0).ready entity=mgr.15291 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T10:26:43.328 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 23 10:26:43 smithi204 ceph-mon[32649]: from='client.15324 -' entity='client.admin' cmd=[{"prefix": "fs snap-schedule add", "fs": "cephfs", "path": "/", "snap_schedule": "1m", "target": ["mon-mgr", ""]}]: dispatch 2024-04-23T10:26:43.328 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 23 10:26:43 smithi204 ceph-mon[32649]: pgmap v3: 97 pgs: 97 active+clean; 1.1 MiB data, 375 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-23T10:26:43.328 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 23 10:26:43 smithi204 ceph-mon[32649]: from='client.15366 -' entity='client.admin' cmd=[{"prefix": "fs snap-schedule retention add", "fs": "cephfs", "path": "/", "retention_spec_or_period": "6m3h", "target": ["mon-mgr", ""]}]: dispatch 2024-04-23T10:26:43.328 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 23 10:26:43 smithi204 ceph-mon[32649]: from='client.15372 -' entity='client.admin' cmd=[{"prefix": "fs snap-schedule status", "fs": "cephfs", "path": "/", "target": ["mon-mgr", ""]}]: dispatch 2024-04-23T10:26:43.328 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 23 10:26:43 smithi204 ceph-mon[32649]: from='mgr.15291 172.21.15.114:0/1095789187' entity='mgr.x' 2024-04-23T10:26:43.328 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 23 10:26:43 smithi204 ceph-mon[32649]: from='mgr.15291 172.21.15.114:0/1095789187' entity='mgr.x' 2024-04-23T10:26:43.328 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 23 10:26:43 smithi204 ceph-mon[32649]: from='mgr.15291 172.21.15.114:0/1095789187' entity='mgr.x' 2024-04-23T10:26:43.328 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 23 10:26:43 smithi204 ceph-mon[32649]: from='mgr.15291 172.21.15.114:0/1095789187' entity='mgr.x' 2024-04-23T10:26:43.328 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 23 10:26:43 smithi204 ceph-mon[32649]: from='mgr.15291 172.21.15.114:0/1095789187' entity='mgr.x' 2024-04-23T10:26:43.328 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 23 10:26:43 smithi204 ceph-mon[32649]: from='mgr.15291 172.21.15.114:0/1095789187' entity='mgr.x' 2024-04-23T10:26:43.328 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 23 10:26:43 smithi204 ceph-mon[32649]: from='client.? 172.21.15.114:0/856911319' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-23T10:26:43.329 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:43.330+0000 7fcf7b7fe640 1 -- 172.21.15.114:0/1382747356 <== mon.0 v2:172.21.15.114:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7fcf8009d280 con 0x7fcf8c1029f0 2024-04-23T10:26:43.491 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 23 10:26:43 smithi119 ceph-mon[32608]: from='client.15324 -' entity='client.admin' cmd=[{"prefix": "fs snap-schedule add", "fs": "cephfs", "path": "/", "snap_schedule": "1m", "target": ["mon-mgr", ""]}]: dispatch 2024-04-23T10:26:43.491 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 23 10:26:43 smithi119 ceph-mon[32608]: pgmap v3: 97 pgs: 97 active+clean; 1.1 MiB data, 375 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-23T10:26:43.492 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 23 10:26:43 smithi119 ceph-mon[32608]: from='client.15366 -' entity='client.admin' cmd=[{"prefix": "fs snap-schedule retention add", "fs": "cephfs", "path": "/", "retention_spec_or_period": "6m3h", "target": ["mon-mgr", ""]}]: dispatch 2024-04-23T10:26:43.492 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 23 10:26:43 smithi119 ceph-mon[32608]: from='client.15372 -' entity='client.admin' cmd=[{"prefix": "fs snap-schedule status", "fs": "cephfs", "path": "/", "target": ["mon-mgr", ""]}]: dispatch 2024-04-23T10:26:43.492 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 23 10:26:43 smithi119 ceph-mon[32608]: from='mgr.15291 172.21.15.114:0/1095789187' entity='mgr.x' 2024-04-23T10:26:43.492 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 23 10:26:43 smithi119 ceph-mon[32608]: from='mgr.15291 172.21.15.114:0/1095789187' entity='mgr.x' 2024-04-23T10:26:43.492 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 23 10:26:43 smithi119 ceph-mon[32608]: from='mgr.15291 172.21.15.114:0/1095789187' entity='mgr.x' 2024-04-23T10:26:43.492 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 23 10:26:43 smithi119 ceph-mon[32608]: from='mgr.15291 172.21.15.114:0/1095789187' entity='mgr.x' 2024-04-23T10:26:43.492 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 23 10:26:43 smithi119 ceph-mon[32608]: from='mgr.15291 172.21.15.114:0/1095789187' entity='mgr.x' 2024-04-23T10:26:43.492 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 23 10:26:43 smithi119 ceph-mon[32608]: from='mgr.15291 172.21.15.114:0/1095789187' entity='mgr.x' 2024-04-23T10:26:43.492 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 23 10:26:43 smithi119 ceph-mon[32608]: from='client.? 172.21.15.114:0/856911319' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-23T10:26:43.510 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:43.511+0000 7fcf90ea7640 1 -- 172.21.15.114:0/1382747356 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) -- 0x7fcf8c062b50 con 0x7fcf8c1029f0 2024-04-23T10:26:43.510 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:43.512+0000 7fcf7b7fe640 1 -- 172.21.15.114:0/1382747356 <== mon.0 v2:172.21.15.114:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v90) ==== 104+0+11 (secure 0 0 0) 0x7fcf8009cb80 con 0x7fcf8c1029f0 2024-04-23T10:26:43.510 INFO:teuthology.orchestra.run.smithi114.stdout:pg_num: 32 2024-04-23T10:26:43.513 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:43.513+0000 7fcf90ea7640 1 -- 172.21.15.114:0/1382747356 >> [v2:172.21.15.114:6800/1395693466,v1:172.21.15.114:6801/1395693466] conn(0x7fcf740a3d20 msgr2=0x7fcf740a61e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T10:26:43.513 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:43.513+0000 7fcf90ea7640 1 --2- 172.21.15.114:0/1382747356 >> [v2:172.21.15.114:6800/1395693466,v1:172.21.15.114:6801/1395693466] conn(0x7fcf740a3d20 0x7fcf740a61e0 secure :-1 s=READY pgs=42 cs=0 l=1 rev1=1 crypto rx=0x7fcf8c199650 tx=0x7fcf7c005f90 comp rx=0 tx=0).stop 2024-04-23T10:26:43.513 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:43.514+0000 7fcf90ea7640 1 -- 172.21.15.114:0/1382747356 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fcf8c1029f0 msgr2=0x7fcf8c198140 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T10:26:43.513 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:43.514+0000 7fcf90ea7640 1 --2- 172.21.15.114:0/1382747356 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fcf8c1029f0 0x7fcf8c198140 secure :-1 s=READY pgs=437 cs=0 l=1 rev1=1 crypto rx=0x7fcf8000e9f0 tx=0x7fcf8000edf0 comp rx=0 tx=0).stop 2024-04-23T10:26:43.513 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:43.514+0000 7fcf90ea7640 1 -- 172.21.15.114:0/1382747356 shutdown_connections 2024-04-23T10:26:43.513 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:43.514+0000 7fcf90ea7640 1 --2- 172.21.15.114:0/1382747356 >> [v2:172.21.15.114:6800/1395693466,v1:172.21.15.114:6801/1395693466] conn(0x7fcf740a3d20 0x7fcf740a61e0 unknown :-1 s=CLOSED pgs=42 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:26:43.513 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:43.514+0000 7fcf90ea7640 1 --2- 172.21.15.114:0/1382747356 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fcf8c107c50 0x7fcf8c19f6f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:26:43.513 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:43.514+0000 7fcf90ea7640 1 --2- 172.21.15.114:0/1382747356 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fcf8c105320 0x7fcf8c198680 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:26:43.513 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:43.514+0000 7fcf90ea7640 1 --2- 172.21.15.114:0/1382747356 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fcf8c1029f0 0x7fcf8c198140 unknown :-1 s=CLOSED pgs=437 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:26:43.513 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:43.514+0000 7fcf90ea7640 1 -- 172.21.15.114:0/1382747356 >> 172.21.15.114:0/1382747356 conn(0x7fcf8c0fc3e0 msgr2=0x7fcf8c103330 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T10:26:43.513 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:43.514+0000 7fcf90ea7640 1 -- 172.21.15.114:0/1382747356 shutdown_connections 2024-04-23T10:26:43.513 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:43.514+0000 7fcf90ea7640 1 -- 172.21.15.114:0/1382747356 wait complete. 2024-04-23T10:26:43.525 DEBUG:teuthology.orchestra.run.smithi114:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2024-04-23T10:26:43.638 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:43.638+0000 7f603e031640 1 -- 172.21.15.114:0/762875891 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f6038107c70 msgr2=0x7f603810a0c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T10:26:43.638 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:43.638+0000 7f603e031640 1 --2- 172.21.15.114:0/762875891 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f6038107c70 0x7f603810a0c0 secure :-1 s=READY pgs=438 cs=0 l=1 rev1=1 crypto rx=0x7f6030009ca0 tx=0x7f603002a8c0 comp rx=0 tx=0).stop 2024-04-23T10:26:43.638 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:43.638+0000 7f603e031640 1 -- 172.21.15.114:0/762875891 shutdown_connections 2024-04-23T10:26:43.638 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:43.638+0000 7f603e031640 1 --2- 172.21.15.114:0/762875891 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f6038107c70 0x7f603810a0c0 unknown :-1 s=CLOSED pgs=438 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:26:43.638 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:43.638+0000 7f603e031640 1 --2- 172.21.15.114:0/762875891 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f6038105340 0x7f6038107730 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:26:43.638 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:43.638+0000 7f603e031640 1 --2- 172.21.15.114:0/762875891 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f6038102a10 0x7f6038104e00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:26:43.638 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:43.638+0000 7f603e031640 1 -- 172.21.15.114:0/762875891 >> 172.21.15.114:0/762875891 conn(0x7f60380fc400 msgr2=0x7f60380fe840 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T10:26:43.638 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:43.639+0000 7f603e031640 1 -- 172.21.15.114:0/762875891 shutdown_connections 2024-04-23T10:26:43.638 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:43.639+0000 7f603e031640 1 -- 172.21.15.114:0/762875891 wait complete. 2024-04-23T10:26:43.638 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:43.639+0000 7f603e031640 1 Processor -- start 2024-04-23T10:26:43.640 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:43.639+0000 7f603e031640 1 -- start start 2024-04-23T10:26:43.640 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:43.639+0000 7f603e031640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f6038102a10 0x7f60381981a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T10:26:43.641 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:43.639+0000 7f603e031640 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f6038105340 0x7f60381986e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T10:26:43.641 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:43.639+0000 7f603e031640 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f6038107c70 0x7f603819f750 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T10:26:43.641 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:43.639+0000 7f603e031640 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x7f603810db30 con 0x7f6038102a10 2024-04-23T10:26:43.641 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:43.639+0000 7f603e031640 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 -- 0x7f603810d9b0 con 0x7f6038107c70 2024-04-23T10:26:43.641 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:43.639+0000 7f603e031640 1 -- --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_getmap magic: 0 -- 0x7f603810dcb0 con 0x7f6038105340 2024-04-23T10:26:43.641 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:43.640+0000 7f60377fe640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f6038102a10 0x7f60381981a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T10:26:43.641 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:43.640+0000 7f60377fe640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f6038102a10 0x7f60381981a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.114:40850/0 (socket says 172.21.15.114:40850) 2024-04-23T10:26:43.641 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:43.640+0000 7f60377fe640 1 -- 172.21.15.114:0/2318849018 learned_addr learned my addr 172.21.15.114:0/2318849018 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-04-23T10:26:43.641 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:43.640+0000 7f6036ffd640 1 --2- 172.21.15.114:0/2318849018 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f6038105340 0x7f60381986e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T10:26:43.641 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:43.640+0000 7f6037fff640 1 --2- 172.21.15.114:0/2318849018 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f6038107c70 0x7f603819f750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T10:26:43.641 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:43.640+0000 7f6036ffd640 1 -- 172.21.15.114:0/2318849018 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f6038107c70 msgr2=0x7f603819f750 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T10:26:43.641 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:43.640+0000 7f6036ffd640 1 --2- 172.21.15.114:0/2318849018 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f6038107c70 0x7f603819f750 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:26:43.641 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:43.640+0000 7f6036ffd640 1 -- 172.21.15.114:0/2318849018 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f6038102a10 msgr2=0x7f60381981a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T10:26:43.641 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:43.640+0000 7f6036ffd640 1 --2- 172.21.15.114:0/2318849018 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f6038102a10 0x7f60381981a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:26:43.641 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:43.640+0000 7f6036ffd640 1 -- 172.21.15.114:0/2318849018 --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f603819fdc0 con 0x7f6038105340 2024-04-23T10:26:43.642 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:43.640+0000 7f60377fe640 1 --2- 172.21.15.114:0/2318849018 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f6038102a10 0x7f60381981a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-23T10:26:43.642 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:43.640+0000 7f6036ffd640 1 --2- 172.21.15.114:0/2318849018 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f6038105340 0x7f60381986e0 secure :-1 s=READY pgs=61 cs=0 l=1 rev1=1 crypto rx=0x7f602800e9c0 tx=0x7f602800ee80 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T10:26:43.642 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:43.640+0000 7f6034ff9640 1 -- 172.21.15.114:0/2318849018 <== mon.1 v2:172.21.15.204:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f602800ccc0 con 0x7f6038105340 2024-04-23T10:26:43.642 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:43.640+0000 7f6034ff9640 1 -- 172.21.15.114:0/2318849018 <== mon.1 v2:172.21.15.204:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f60280108e0 con 0x7f6038105340 2024-04-23T10:26:43.642 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:43.640+0000 7f603e031640 1 -- 172.21.15.114:0/2318849018 --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f60381a00b0 con 0x7f6038105340 2024-04-23T10:26:43.642 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:43.640+0000 7f6034ff9640 1 -- 172.21.15.114:0/2318849018 <== mon.1 v2:172.21.15.204:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f60280187c0 con 0x7f6038105340 2024-04-23T10:26:43.642 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:43.640+0000 7f603e031640 1 -- 172.21.15.114:0/2318849018 --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f60381a0600 con 0x7f6038105340 2024-04-23T10:26:43.642 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:43.642+0000 7f603e031640 1 -- 172.21.15.114:0/2318849018 --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f60380fdef0 con 0x7f6038105340 2024-04-23T10:26:43.643 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:43.642+0000 7f6034ff9640 1 -- 172.21.15.114:0/2318849018 <== mon.1 v2:172.21.15.204:3300/0 4 ==== mgrmap(e 23) ==== 138767+0+0 (secure 0 0 0) 0x7f6028010430 con 0x7f6038105340 2024-04-23T10:26:43.643 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:43.643+0000 7f6034ff9640 1 --2- 172.21.15.114:0/2318849018 >> [v2:172.21.15.114:6800/1395693466,v1:172.21.15.114:6801/1395693466] conn(0x7f60200a3d20 0x7f60200a61e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T10:26:43.643 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:43.643+0000 7f60377fe640 1 --2- 172.21.15.114:0/2318849018 >> [v2:172.21.15.114:6800/1395693466,v1:172.21.15.114:6801/1395693466] conn(0x7f60200a3d20 0x7f60200a61e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T10:26:43.643 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:43.643+0000 7f6034ff9640 1 -- 172.21.15.114:0/2318849018 <== mon.1 v2:172.21.15.204:3300/0 5 ==== osd_map(90..90 src has 1..90) ==== 8527+0+0 (secure 0 0 0) 0x7f60280d0ad0 con 0x7f6038105340 2024-04-23T10:26:43.643 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:43.643+0000 7f60377fe640 1 --2- 172.21.15.114:0/2318849018 >> [v2:172.21.15.114:6800/1395693466,v1:172.21.15.114:6801/1395693466] conn(0x7f60200a3d20 0x7f60200a61e0 secure :-1 s=READY pgs=43 cs=0 l=1 rev1=1 crypto rx=0x7f602c011500 tx=0x7f602c005f90 comp rx=0 tx=0).ready entity=mgr.15291 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T10:26:43.646 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:43.648+0000 7f6034ff9640 1 -- 172.21.15.114:0/2318849018 <== mon.1 v2:172.21.15.204:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f602809cb20 con 0x7f6038105340 2024-04-23T10:26:43.837 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:43.838+0000 7f603e031640 1 -- 172.21.15.114:0/2318849018 --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) -- 0x7f603802d770 con 0x7f6038105340 2024-04-23T10:26:43.837 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:43.838+0000 7f6034ff9640 1 -- 172.21.15.114:0/2318849018 <== mon.1 v2:172.21.15.204:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v90) ==== 100+0+11 (secure 0 0 0) 0x7f60280a0e20 con 0x7f6038105340 2024-04-23T10:26:43.837 INFO:teuthology.orchestra.run.smithi114.stdout:pg_num: 64 2024-04-23T10:26:43.840 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:43.840+0000 7f603e031640 1 -- 172.21.15.114:0/2318849018 >> [v2:172.21.15.114:6800/1395693466,v1:172.21.15.114:6801/1395693466] conn(0x7f60200a3d20 msgr2=0x7f60200a61e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T10:26:43.840 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:43.840+0000 7f603e031640 1 --2- 172.21.15.114:0/2318849018 >> [v2:172.21.15.114:6800/1395693466,v1:172.21.15.114:6801/1395693466] conn(0x7f60200a3d20 0x7f60200a61e0 secure :-1 s=READY pgs=43 cs=0 l=1 rev1=1 crypto rx=0x7f602c011500 tx=0x7f602c005f90 comp rx=0 tx=0).stop 2024-04-23T10:26:43.840 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:43.840+0000 7f603e031640 1 -- 172.21.15.114:0/2318849018 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f6038105340 msgr2=0x7f60381986e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T10:26:43.840 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:43.840+0000 7f603e031640 1 --2- 172.21.15.114:0/2318849018 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f6038105340 0x7f60381986e0 secure :-1 s=READY pgs=61 cs=0 l=1 rev1=1 crypto rx=0x7f602800e9c0 tx=0x7f602800ee80 comp rx=0 tx=0).stop 2024-04-23T10:26:43.840 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:43.841+0000 7f603e031640 1 -- 172.21.15.114:0/2318849018 shutdown_connections 2024-04-23T10:26:43.840 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:43.841+0000 7f603e031640 1 --2- 172.21.15.114:0/2318849018 >> [v2:172.21.15.114:6800/1395693466,v1:172.21.15.114:6801/1395693466] conn(0x7f60200a3d20 0x7f60200a61e0 unknown :-1 s=CLOSED pgs=43 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:26:43.840 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:43.841+0000 7f603e031640 1 --2- 172.21.15.114:0/2318849018 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f6038107c70 0x7f603819f750 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:26:43.840 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:43.841+0000 7f603e031640 1 --2- 172.21.15.114:0/2318849018 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f6038105340 0x7f60381986e0 unknown :-1 s=CLOSED pgs=61 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:26:43.840 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:43.841+0000 7f603e031640 1 --2- 172.21.15.114:0/2318849018 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f6038102a10 0x7f60381981a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:26:43.840 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:43.841+0000 7f603e031640 1 -- 172.21.15.114:0/2318849018 >> 172.21.15.114:0/2318849018 conn(0x7f60380fc400 msgr2=0x7f6038103350 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T10:26:43.840 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:43.841+0000 7f603e031640 1 -- 172.21.15.114:0/2318849018 shutdown_connections 2024-04-23T10:26:43.841 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:43.841+0000 7f603e031640 1 -- 172.21.15.114:0/2318849018 wait complete. 2024-04-23T10:26:43.851 DEBUG:teuthology.orchestra.run.smithi114:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-04-23T10:26:43.953 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:43.954+0000 7f53a6e79640 1 -- 172.21.15.114:0/616135942 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f53a0105340 msgr2=0x7f53a0107730 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T10:26:43.953 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:43.954+0000 7f53a6e79640 1 --2- 172.21.15.114:0/616135942 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f53a0105340 0x7f53a0107730 secure :-1 s=READY pgs=439 cs=0 l=1 rev1=1 crypto rx=0x7f538c009900 tx=0x7f538c02a410 comp rx=0 tx=0).stop 2024-04-23T10:26:43.953 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:43.954+0000 7f53a6e79640 1 -- 172.21.15.114:0/616135942 shutdown_connections 2024-04-23T10:26:43.953 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:43.954+0000 7f53a6e79640 1 --2- 172.21.15.114:0/616135942 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f53a0107c70 0x7f53a010a0c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:26:43.953 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:43.954+0000 7f53a6e79640 1 --2- 172.21.15.114:0/616135942 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f53a0105340 0x7f53a0107730 unknown :-1 s=CLOSED pgs=439 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:26:43.953 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:43.954+0000 7f53a6e79640 1 --2- 172.21.15.114:0/616135942 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f53a0102a10 0x7f53a0104e00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:26:43.953 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:43.954+0000 7f53a6e79640 1 -- 172.21.15.114:0/616135942 >> 172.21.15.114:0/616135942 conn(0x7f53a00fc400 msgr2=0x7f53a00fe840 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T10:26:43.953 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:43.954+0000 7f53a6e79640 1 -- 172.21.15.114:0/616135942 shutdown_connections 2024-04-23T10:26:43.953 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:43.954+0000 7f53a6e79640 1 -- 172.21.15.114:0/616135942 wait complete. 2024-04-23T10:26:43.955 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:43.954+0000 7f53a6e79640 1 Processor -- start 2024-04-23T10:26:43.955 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:43.954+0000 7f53a6e79640 1 -- start start 2024-04-23T10:26:43.955 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:43.955+0000 7f53a6e79640 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f53a0102a10 0x7f53a01981e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T10:26:43.956 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:43.955+0000 7f53a6e79640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f53a0105340 0x7f53a0198720 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T10:26:43.956 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:43.955+0000 7f53a6e79640 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f53a0107c70 0x7f53a019f790 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T10:26:43.956 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:43.955+0000 7f53a6e79640 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x7f53a010dab0 con 0x7f53a0105340 2024-04-23T10:26:43.956 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:43.955+0000 7f53a6e79640 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 -- 0x7f53a010d930 con 0x7f53a0107c70 2024-04-23T10:26:43.956 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:43.955+0000 7f53a6e79640 1 -- --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_getmap magic: 0 -- 0x7f53a010dc30 con 0x7f53a0102a10 2024-04-23T10:26:43.956 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:43.955+0000 7f539ffff640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f53a0105340 0x7f53a0198720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T10:26:43.956 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:43.955+0000 7f539ffff640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f53a0105340 0x7f53a0198720 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.114:40872/0 (socket says 172.21.15.114:40872) 2024-04-23T10:26:43.956 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:43.955+0000 7f539ffff640 1 -- 172.21.15.114:0/2609718743 learned_addr learned my addr 172.21.15.114:0/2609718743 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-04-23T10:26:43.956 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:43.955+0000 7f53a5416640 1 --2- 172.21.15.114:0/2609718743 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f53a0107c70 0x7f53a019f790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T10:26:43.956 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:43.955+0000 7f53a4c15640 1 --2- 172.21.15.114:0/2609718743 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f53a0102a10 0x7f53a01981e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T10:26:43.956 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:43.955+0000 7f539ffff640 1 -- 172.21.15.114:0/2609718743 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f53a0102a10 msgr2=0x7f53a01981e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T10:26:43.956 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:43.955+0000 7f539ffff640 1 --2- 172.21.15.114:0/2609718743 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f53a0102a10 0x7f53a01981e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:26:43.956 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:43.955+0000 7f539ffff640 1 -- 172.21.15.114:0/2609718743 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f53a0107c70 msgr2=0x7f53a019f790 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T10:26:43.956 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:43.955+0000 7f539ffff640 1 --2- 172.21.15.114:0/2609718743 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f53a0107c70 0x7f53a019f790 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:26:43.957 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:43.955+0000 7f539ffff640 1 -- 172.21.15.114:0/2609718743 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f53a019fe00 con 0x7f53a0105340 2024-04-23T10:26:43.957 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:43.955+0000 7f539ffff640 1 --2- 172.21.15.114:0/2609718743 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f53a0105340 0x7f53a0198720 secure :-1 s=READY pgs=440 cs=0 l=1 rev1=1 crypto rx=0x7f538c006010 tx=0x7f538c0042f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T10:26:43.957 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:43.955+0000 7f539dffb640 1 -- 172.21.15.114:0/2609718743 <== mon.0 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f538c03f030 con 0x7f53a0105340 2024-04-23T10:26:43.957 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:43.956+0000 7f53a6e79640 1 -- 172.21.15.114:0/2609718743 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f53a01a0090 con 0x7f53a0105340 2024-04-23T10:26:43.957 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:43.956+0000 7f539dffb640 1 -- 172.21.15.114:0/2609718743 <== mon.0 v2:172.21.15.114:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f538c031910 con 0x7f53a0105340 2024-04-23T10:26:43.957 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:43.956+0000 7f539dffb640 1 -- 172.21.15.114:0/2609718743 <== mon.0 v2:172.21.15.114:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f538c035070 con 0x7f53a0105340 2024-04-23T10:26:43.957 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:43.956+0000 7f53a6e79640 1 -- 172.21.15.114:0/2609718743 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f53a01a0580 con 0x7f53a0105340 2024-04-23T10:26:43.958 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:43.957+0000 7f53a6e79640 1 -- 172.21.15.114:0/2609718743 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f53a00fdef0 con 0x7f53a0105340 2024-04-23T10:26:43.958 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:43.958+0000 7f539dffb640 1 -- 172.21.15.114:0/2609718743 <== mon.0 v2:172.21.15.114:3300/0 4 ==== mgrmap(e 23) ==== 138767+0+0 (secure 0 0 0) 0x7f538c042030 con 0x7f53a0105340 2024-04-23T10:26:43.958 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:43.958+0000 7f539dffb640 1 --2- 172.21.15.114:0/2609718743 >> [v2:172.21.15.114:6800/1395693466,v1:172.21.15.114:6801/1395693466] conn(0x7f53900a3d20 0x7f53900a61e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T10:26:43.958 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:43.958+0000 7f53a4c15640 1 --2- 172.21.15.114:0/2609718743 >> [v2:172.21.15.114:6800/1395693466,v1:172.21.15.114:6801/1395693466] conn(0x7f53900a3d20 0x7f53900a61e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T10:26:43.958 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:43.958+0000 7f539dffb640 1 -- 172.21.15.114:0/2609718743 <== mon.0 v2:172.21.15.114:3300/0 5 ==== osd_map(90..90 src has 1..90) ==== 8527+0+0 (secure 0 0 0) 0x7f538c0f0da0 con 0x7f53a0105340 2024-04-23T10:26:43.958 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:43.959+0000 7f53a4c15640 1 --2- 172.21.15.114:0/2609718743 >> [v2:172.21.15.114:6800/1395693466,v1:172.21.15.114:6801/1395693466] conn(0x7f53900a3d20 0x7f53900a61e0 secure :-1 s=READY pgs=44 cs=0 l=1 rev1=1 crypto rx=0x7f5394009b60 tx=0x7f5394005f90 comp rx=0 tx=0).ready entity=mgr.15291 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T10:26:43.962 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:43.963+0000 7f539dffb640 1 -- 172.21.15.114:0/2609718743 <== mon.0 v2:172.21.15.114:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f538c0bd440 con 0x7f53a0105340 2024-04-23T10:26:44.155 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:44.156+0000 7f53a6e79640 1 -- 172.21.15.114:0/2609718743 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f53a002d770 con 0x7f53a0105340 2024-04-23T10:26:44.157 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:44.158+0000 7f539dffb640 1 -- 172.21.15.114:0/2609718743 <== mon.0 v2:172.21.15.114:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v90) ==== 74+0+23198 (secure 0 0 0) 0x7f538c003e60 con 0x7f53a0105340 2024-04-23T10:26:44.157 INFO:teuthology.orchestra.run.smithi114.stdout: 2024-04-23T10:26:44.157 INFO:teuthology.orchestra.run.smithi114.stdout:{"epoch":90,"fsid":"aa240572-0159-11ef-bc93-c7b262605968","created":"2024-04-23T10:16:53.351136+0000","modified":"2024-04-23T10:26:40.272696+0000","last_up_change":"2024-04-23T10:23:54.153269+0000","last_in_change":"2024-04-23T10:23:32.963085+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":26,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-04-23T10:20:11.970297+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":12,"score_stable":12,"optimal_score":0.25,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-04-23T10:24:41.218540+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"86","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.8799999952316284,"score_stable":1.8799999952316284,"optimal_score":1,"raw_score_acting":1.8799999952316284,"raw_score_stable":1.8799999952316284,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-04-23T10:24:42.341799+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"87","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"fd64adde-9445-45a4-a171-ef656e2c5709","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":88,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6802","nonce":736366878},{"type":"v1","addr":"172.21.15.114:6803","nonce":736366878}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6804","nonce":736366878},{"type":"v1","addr":"172.21.15.114:6805","nonce":736366878}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6808","nonce":736366878},{"type":"v1","addr":"172.21.15.114:6809","nonce":736366878}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6806","nonce":736366878},{"type":"v1","addr":"172.21.15.114:6807","nonce":736366878}]},"public_addr":"172.21.15.114:6803/736366878","cluster_addr":"172.21.15.114:6805/736366878","heartbeat_back_addr":"172.21.15.114:6809/736366878","heartbeat_front_addr":"172.21.15.114:6807/736366878","state":["exists","up"]},{"osd":1,"uuid":"e279a1e6-2702-472a-930a-e08ece76a861","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":85,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6800","nonce":1133540779},{"type":"v1","addr":"172.21.15.119:6801","nonce":1133540779}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6802","nonce":1133540779},{"type":"v1","addr":"172.21.15.119:6803","nonce":1133540779}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6806","nonce":1133540779},{"type":"v1","addr":"172.21.15.119:6807","nonce":1133540779}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6804","nonce":1133540779},{"type":"v1","addr":"172.21.15.119:6805","nonce":1133540779}]},"public_addr":"172.21.15.119:6801/1133540779","cluster_addr":"172.21.15.119:6803/1133540779","heartbeat_back_addr":"172.21.15.119:6807/1133540779","heartbeat_front_addr":"172.21.15.119:6805/1133540779","state":["exists","up"]},{"osd":2,"uuid":"f4c9f2b4-ff49-4e14-89cd-b477feeb9382","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":85,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6800","nonce":799652511},{"type":"v1","addr":"172.21.15.204:6801","nonce":799652511}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6802","nonce":799652511},{"type":"v1","addr":"172.21.15.204:6803","nonce":799652511}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6806","nonce":799652511},{"type":"v1","addr":"172.21.15.204:6807","nonce":799652511}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6804","nonce":799652511},{"type":"v1","addr":"172.21.15.204:6805","nonce":799652511}]},"public_addr":"172.21.15.204:6801/799652511","cluster_addr":"172.21.15.204:6803/799652511","heartbeat_back_addr":"172.21.15.204:6807/799652511","heartbeat_front_addr":"172.21.15.204:6805/799652511","state":["exists","up"]},{"osd":3,"uuid":"52d2a566-e71c-4bb3-8d6c-5abdd43177c2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":25,"up_thru":88,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6810","nonce":4275595091},{"type":"v1","addr":"172.21.15.114:6811","nonce":4275595091}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6812","nonce":4275595091},{"type":"v1","addr":"172.21.15.114:6813","nonce":4275595091}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6816","nonce":4275595091},{"type":"v1","addr":"172.21.15.114:6817","nonce":4275595091}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6814","nonce":4275595091},{"type":"v1","addr":"172.21.15.114:6815","nonce":4275595091}]},"public_addr":"172.21.15.114:6811/4275595091","cluster_addr":"172.21.15.114:6813/4275595091","heartbeat_back_addr":"172.21.15.114:6817/4275595091","heartbeat_front_addr":"172.21.15.114:6815/4275595091","state":["exists","up"]},{"osd":4,"uuid":"d911a7c1-4bdc-4d28-b8a6-73896a6e7de6","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":33,"up_thru":85,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6808","nonce":1611007172},{"type":"v1","addr":"172.21.15.119:6809","nonce":1611007172}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6810","nonce":1611007172},{"type":"v1","addr":"172.21.15.119:6811","nonce":1611007172}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6814","nonce":1611007172},{"type":"v1","addr":"172.21.15.119:6815","nonce":1611007172}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6812","nonce":1611007172},{"type":"v1","addr":"172.21.15.119:6813","nonce":1611007172}]},"public_addr":"172.21.15.119:6809/1611007172","cluster_addr":"172.21.15.119:6811/1611007172","heartbeat_back_addr":"172.21.15.119:6815/1611007172","heartbeat_front_addr":"172.21.15.119:6813/1611007172","state":["exists","up"]},{"osd":5,"uuid":"f7590e1a-e93c-4c29-b15f-38ee52d1f921","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":39,"up_thru":85,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6808","nonce":1427563312},{"type":"v1","addr":"172.21.15.204:6809","nonce":1427563312}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6810","nonce":1427563312},{"type":"v1","addr":"172.21.15.204:6811","nonce":1427563312}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6814","nonce":1427563312},{"type":"v1","addr":"172.21.15.204:6815","nonce":1427563312}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6812","nonce":1427563312},{"type":"v1","addr":"172.21.15.204:6813","nonce":1427563312}]},"public_addr":"172.21.15.204:6809/1427563312","cluster_addr":"172.21.15.204:6811/1427563312","heartbeat_back_addr":"172.21.15.204:6815/1427563312","heartbeat_front_addr":"172.21.15.204:6813/1427563312","state":["exists","up"]},{"osd":6,"uuid":"534d20e8-3e54-4db8-a845-6119f49c7a78","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":47,"up_thru":88,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6818","nonce":4072912544},{"type":"v1","addr":"172.21.15.114:6819","nonce":4072912544}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6820","nonce":4072912544},{"type":"v1","addr":"172.21.15.114:6821","nonce":4072912544}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6824","nonce":4072912544},{"type":"v1","addr":"172.21.15.114:6825","nonce":4072912544}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6822","nonce":4072912544},{"type":"v1","addr":"172.21.15.114:6823","nonce":4072912544}]},"public_addr":"172.21.15.114:6819/4072912544","cluster_addr":"172.21.15.114:6821/4072912544","heartbeat_back_addr":"172.21.15.114:6825/4072912544","heartbeat_front_addr":"172.21.15.114:6823/4072912544","state":["exists","up"]},{"osd":7,"uuid":"f0ec22b8-1b48-4859-9d65-b00403aff3ad","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":53,"up_thru":88,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6816","nonce":2237458793},{"type":"v1","addr":"172.21.15.119:6817","nonce":2237458793}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6818","nonce":2237458793},{"type":"v1","addr":"172.21.15.119:6819","nonce":2237458793}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6822","nonce":2237458793},{"type":"v1","addr":"172.21.15.119:6823","nonce":2237458793}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6820","nonce":2237458793},{"type":"v1","addr":"172.21.15.119:6821","nonce":2237458793}]},"public_addr":"172.21.15.119:6817/2237458793","cluster_addr":"172.21.15.119:6819/2237458793","heartbeat_back_addr":"172.21.15.119:6823/2237458793","heartbeat_front_addr":"172.21.15.119:6821/2237458793","state":["exists","up"]},{"osd":8,"uuid":"3ac89686-6d76-4cef-9089-ac29e4b3bcac","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":62,"up_thru":85,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6816","nonce":1426318231},{"type":"v1","addr":"172.21.15.204:6817","nonce":1426318231}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6818","nonce":1426318231},{"type":"v1","addr":"172.21.15.204:6819","nonce":1426318231}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6822","nonce":1426318231},{"type":"v1","addr":"172.21.15.204:6823","nonce":1426318231}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6820","nonce":1426318231},{"type":"v1","addr":"172.21.15.204:6821","nonce":1426318231}]},"public_addr":"172.21.15.204:6817/1426318231","cluster_addr":"172.21.15.204:6819/1426318231","heartbeat_back_addr":"172.21.15.204:6823/1426318231","heartbeat_front_addr":"172.21.15.204:6821/1426318231","state":["exists","up"]},{"osd":9,"uuid":"a80ffe4d-1e98-460f-b046-c4754285cc5a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":69,"up_thru":88,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6826","nonce":2750635721},{"type":"v1","addr":"172.21.15.114:6827","nonce":2750635721}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6828","nonce":2750635721},{"type":"v1","addr":"172.21.15.114:6829","nonce":2750635721}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6832","nonce":2750635721},{"type":"v1","addr":"172.21.15.114:6833","nonce":2750635721}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6830","nonce":2750635721},{"type":"v1","addr":"172.21.15.114:6831","nonce":2750635721}]},"public_addr":"172.21.15.114:6827/2750635721","cluster_addr":"172.21.15.114:6829/2750635721","heartbeat_back_addr":"172.21.15.114:6833/2750635721","heartbeat_front_addr":"172.21.15.114:6831/2750635721","state":["exists","up"]},{"osd":10,"uuid":"cd75a579-bf12-4600-ac09-a8bfce772ced","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":74,"up_thru":85,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6824","nonce":1400187701},{"type":"v1","addr":"172.21.15.119:6825","nonce":1400187701}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6826","nonce":1400187701},{"type":"v1","addr":"172.21.15.119:6827","nonce":1400187701}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6830","nonce":1400187701},{"type":"v1","addr":"172.21.15.119:6831","nonce":1400187701}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6828","nonce":1400187701},{"type":"v1","addr":"172.21.15.119:6829","nonce":1400187701}]},"public_addr":"172.21.15.119:6825/1400187701","cluster_addr":"172.21.15.119:6827/1400187701","heartbeat_back_addr":"172.21.15.119:6831/1400187701","heartbeat_front_addr":"172.21.15.119:6829/1400187701","state":["exists","up"]},{"osd":11,"uuid":"53f037a5-3ec0-483a-80a1-dfee923a68e5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":82,"up_thru":88,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6824","nonce":2407298490},{"type":"v1","addr":"172.21.15.204:6825","nonce":2407298490}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6826","nonce":2407298490},{"type":"v1","addr":"172.21.15.204:6827","nonce":2407298490}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6830","nonce":2407298490},{"type":"v1","addr":"172.21.15.204:6831","nonce":2407298490}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6828","nonce":2407298490},{"type":"v1","addr":"172.21.15.204:6829","nonce":2407298490}]},"public_addr":"172.21.15.204:6825/2407298490","cluster_addr":"172.21.15.204:6827/2407298490","heartbeat_back_addr":"172.21.15.204:6831/2407298490","heartbeat_front_addr":"172.21.15.204:6829/2407298490","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-23T10:19:18.401147+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-23T10:19:42.611299+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-23T10:20:07.277367+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-23T10:20:31.208368+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-23T10:20:55.866225+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-23T10:21:19.961237+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-23T10:21:44.480783+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-23T10:22:09.173641+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-23T10:22:34.030328+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-23T10:22:59.652293+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-23T10:23:25.659673+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-23T10:23:51.168171+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"2.e","mappings":[{"from":2,"to":4}]},{"pgid":"2.1d","mappings":[{"from":8,"to":11}]},{"pgid":"3.e","mappings":[{"from":5,"to":9}]},{"pgid":"3.12","mappings":[{"from":3,"to":9}]},{"pgid":"3.17","mappings":[{"from":4,"to":6}]},{"pgid":"3.23","mappings":[{"from":4,"to":7}]},{"pgid":"3.27","mappings":[{"from":4,"to":6}]},{"pgid":"3.31","mappings":[{"from":4,"to":8}]},{"pgid":"3.38","mappings":[{"from":3,"to":9}]},{"pgid":"3.39","mappings":[{"from":5,"to":8}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.114:0/480720790":"2024-04-24T10:26:40.272662+0000","172.21.15.114:6801/465333808":"2024-04-24T10:26:40.272662+0000","172.21.15.114:6800/465333808":"2024-04-24T10:26:40.272662+0000","172.21.15.114:0/1787801099":"2024-04-24T10:26:40.272662+0000","172.21.15.114:6801/1604678557":"2024-04-24T10:17:45.874609+0000","172.21.15.114:0/4025385854":"2024-04-24T10:17:45.874609+0000","172.21.15.114:0/2228273469":"2024-04-24T10:17:19.910348+0000","172.21.15.114:0/2713201622":"2024-04-24T10:26:40.272662+0000","172.21.15.114:6800/2831530838":"2024-04-24T10:17:19.910348+0000","172.21.15.114:0/637004095":"2024-04-24T10:26:40.272662+0000","172.21.15.114:0/1333447450":"2024-04-24T10:17:45.874609+0000","172.21.15.114:0/2233026840":"2024-04-24T10:17:19.910348+0000","172.21.15.114:0/1902281545":"2024-04-24T10:17:19.910348+0000","172.21.15.114:6801/2831530838":"2024-04-24T10:17:19.910348+0000","172.21.15.114:6800/1604678557":"2024-04-24T10:17:45.874609+0000","172.21.15.114:0/1623553811":"2024-04-24T10:17:45.874609+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-23T10:26:44.159 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:44.160+0000 7f53a6e79640 1 -- 172.21.15.114:0/2609718743 >> [v2:172.21.15.114:6800/1395693466,v1:172.21.15.114:6801/1395693466] conn(0x7f53900a3d20 msgr2=0x7f53900a61e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T10:26:44.159 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:44.160+0000 7f53a6e79640 1 --2- 172.21.15.114:0/2609718743 >> [v2:172.21.15.114:6800/1395693466,v1:172.21.15.114:6801/1395693466] conn(0x7f53900a3d20 0x7f53900a61e0 secure :-1 s=READY pgs=44 cs=0 l=1 rev1=1 crypto rx=0x7f5394009b60 tx=0x7f5394005f90 comp rx=0 tx=0).stop 2024-04-23T10:26:44.159 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:44.160+0000 7f53a6e79640 1 -- 172.21.15.114:0/2609718743 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f53a0105340 msgr2=0x7f53a0198720 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T10:26:44.159 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:44.160+0000 7f53a6e79640 1 --2- 172.21.15.114:0/2609718743 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f53a0105340 0x7f53a0198720 secure :-1 s=READY pgs=440 cs=0 l=1 rev1=1 crypto rx=0x7f538c006010 tx=0x7f538c0042f0 comp rx=0 tx=0).stop 2024-04-23T10:26:44.160 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:44.160+0000 7f53a6e79640 1 -- 172.21.15.114:0/2609718743 shutdown_connections 2024-04-23T10:26:44.160 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:44.160+0000 7f53a6e79640 1 --2- 172.21.15.114:0/2609718743 >> [v2:172.21.15.114:6800/1395693466,v1:172.21.15.114:6801/1395693466] conn(0x7f53900a3d20 0x7f53900a61e0 unknown :-1 s=CLOSED pgs=44 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:26:44.160 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:44.160+0000 7f53a6e79640 1 --2- 172.21.15.114:0/2609718743 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f53a0107c70 0x7f53a019f790 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:26:44.160 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:44.160+0000 7f53a6e79640 1 --2- 172.21.15.114:0/2609718743 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f53a0105340 0x7f53a0198720 unknown :-1 s=CLOSED pgs=440 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:26:44.160 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:44.160+0000 7f53a6e79640 1 --2- 172.21.15.114:0/2609718743 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f53a0102a10 0x7f53a01981e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:26:44.160 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:44.160+0000 7f53a6e79640 1 -- 172.21.15.114:0/2609718743 >> 172.21.15.114:0/2609718743 conn(0x7f53a00fc400 msgr2=0x7f53a01034f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T10:26:44.160 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:44.160+0000 7f53a6e79640 1 -- 172.21.15.114:0/2609718743 shutdown_connections 2024-04-23T10:26:44.160 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:44.160+0000 7f53a6e79640 1 -- 172.21.15.114:0/2609718743 wait complete. 2024-04-23T10:26:44.170 INFO:journalctl@ceph.mon.a.smithi114.stdout:Apr 23 10:26:44 smithi114 ceph-mon[29255]: pgmap v4: 97 pgs: 97 active+clean; 1.1 MiB data, 375 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-23T10:26:44.170 INFO:journalctl@ceph.mon.a.smithi114.stdout:Apr 23 10:26:44 smithi114 ceph-mon[29255]: from='client.15378 -' entity='client.admin' cmd=[{"prefix": "fs snap-schedule list", "fs": "cephfs", "path": "/", "recursive": true, "target": ["mon-mgr", ""]}]: dispatch 2024-04-23T10:26:44.170 INFO:journalctl@ceph.mon.a.smithi114.stdout:Apr 23 10:26:44 smithi114 ceph-mon[29255]: mgrmap e23: x(active, since 2s), standbys: z, y 2024-04-23T10:26:44.170 INFO:journalctl@ceph.mon.a.smithi114.stdout:Apr 23 10:26:44 smithi114 ceph-mon[29255]: from='client.? 172.21.15.114:0/2108459556' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-04-23T10:26:44.170 INFO:journalctl@ceph.mon.a.smithi114.stdout:Apr 23 10:26:44 smithi114 ceph-mon[29255]: from='client.? 172.21.15.114:0/1382747356' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2024-04-23T10:26:44.170 INFO:journalctl@ceph.mon.a.smithi114.stdout:Apr 23 10:26:44 smithi114 ceph-mon[29255]: from='mgr.15291 172.21.15.114:0/1095789187' entity='mgr.x' 2024-04-23T10:26:44.170 INFO:journalctl@ceph.mon.a.smithi114.stdout:Apr 23 10:26:44 smithi114 ceph-mon[29255]: from='mgr.15291 172.21.15.114:0/1095789187' entity='mgr.x' 2024-04-23T10:26:44.170 INFO:journalctl@ceph.mon.a.smithi114.stdout:Apr 23 10:26:44 smithi114 ceph-mon[29255]: from='mgr.15291 172.21.15.114:0/1095789187' entity='mgr.x' 2024-04-23T10:26:44.170 INFO:journalctl@ceph.mon.a.smithi114.stdout:Apr 23 10:26:44 smithi114 ceph-mon[29255]: from='mgr.15291 172.21.15.114:0/1095789187' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-04-23T10:26:44.170 INFO:journalctl@ceph.mon.a.smithi114.stdout:Apr 23 10:26:44 smithi114 ceph-mon[29255]: from='mgr.15291 172.21.15.114:0/1095789187' entity='mgr.x' 2024-04-23T10:26:44.170 INFO:journalctl@ceph.mon.a.smithi114.stdout:Apr 23 10:26:44 smithi114 ceph-mon[29255]: from='mgr.15291 172.21.15.114:0/1095789187' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-04-23T10:26:44.171 INFO:journalctl@ceph.mon.a.smithi114.stdout:Apr 23 10:26:44 smithi114 ceph-mon[29255]: from='mgr.15291 172.21.15.114:0/1095789187' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-04-23T10:26:44.171 INFO:journalctl@ceph.mon.a.smithi114.stdout:Apr 23 10:26:44 smithi114 ceph-mon[29255]: from='mgr.15291 172.21.15.114:0/1095789187' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-04-23T10:26:44.171 INFO:journalctl@ceph.mon.a.smithi114.stdout:Apr 23 10:26:44 smithi114 ceph-mon[29255]: from='mgr.15291 172.21.15.114:0/1095789187' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-04-23T10:26:44.171 INFO:journalctl@ceph.mon.a.smithi114.stdout:Apr 23 10:26:44 smithi114 ceph-mon[29255]: from='mgr.15291 172.21.15.114:0/1095789187' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2024-04-23T10:26:44.171 INFO:journalctl@ceph.mon.a.smithi114.stdout:Apr 23 10:26:44 smithi114 ceph-mon[29255]: from='mgr.15291 172.21.15.114:0/1095789187' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2024-04-23T10:26:44.171 INFO:journalctl@ceph.mon.a.smithi114.stdout:Apr 23 10:26:44 smithi114 ceph-mon[29255]: from='mgr.15291 172.21.15.114:0/1095789187' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2024-04-23T10:26:44.171 INFO:journalctl@ceph.mon.a.smithi114.stdout:Apr 23 10:26:44 smithi114 ceph-mon[29255]: from='mgr.15291 172.21.15.114:0/1095789187' entity='mgr.x' 2024-04-23T10:26:44.171 INFO:journalctl@ceph.mon.a.smithi114.stdout:Apr 23 10:26:44 smithi114 ceph-mon[29255]: from='mgr.15291 172.21.15.114:0/1095789187' entity='mgr.x' 2024-04-23T10:26:44.171 INFO:journalctl@ceph.mon.a.smithi114.stdout:Apr 23 10:26:44 smithi114 ceph-mon[29255]: from='mgr.15291 172.21.15.114:0/1095789187' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-04-23T10:26:44.171 INFO:journalctl@ceph.mon.a.smithi114.stdout:Apr 23 10:26:44 smithi114 ceph-mon[29255]: from='mgr.15291 172.21.15.114:0/1095789187' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-04-23T10:26:44.171 INFO:journalctl@ceph.mon.a.smithi114.stdout:Apr 23 10:26:44 smithi114 ceph-mon[29255]: from='mgr.15291 172.21.15.114:0/1095789187' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-04-23T10:26:44.171 INFO:journalctl@ceph.mon.a.smithi114.stdout:Apr 23 10:26:44 smithi114 ceph-mon[29255]: from='mgr.15291 172.21.15.114:0/1095789187' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"}]: dispatch 2024-04-23T10:26:44.171 INFO:journalctl@ceph.mon.a.smithi114.stdout:Apr 23 10:26:44 smithi114 ceph-mon[29255]: from='mgr.15291 172.21.15.114:0/1095789187' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-23T10:26:44.171 INFO:journalctl@ceph.mon.a.smithi114.stdout:Apr 23 10:26:44 smithi114 ceph-mon[29255]: from='mgr.15291 172.21.15.114:0/1095789187' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-23T10:26:44.172 INFO:journalctl@ceph.mon.a.smithi114.stdout:Apr 23 10:26:44 smithi114 ceph-mon[29255]: from='client.? 172.21.15.114:0/2318849018' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2024-04-23T10:26:44.173 INFO:tasks.quiescer.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-04-23T10:20:11.970297+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '22', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 12, 'score_stable': 12, 'optimal_score': 0.25, 'raw_score_acting': 3, 'raw_score_stable': 3, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 2, 'pool_name': 'cephfs_metadata', 'create_time': '2024-04-23T10:24:41.218540+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '86', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.8799999952316284, 'score_stable': 1.8799999952316284, 'optimal_score': 1, 'raw_score_acting': 1.8799999952316284, 'raw_score_stable': 1.8799999952316284, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 3, 'pool_name': 'cephfs_data', 'create_time': '2024-04-23T10:24:42.341799+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '87', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-04-23T10:26:44.173 DEBUG:teuthology.orchestra.run.smithi114:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-04-23T10:26:44.279 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:44.280+0000 7f803a867640 1 -- 172.21.15.114:0/2052441726 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f8034100e60 msgr2=0x7f8034101280 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T10:26:44.280 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:44.280+0000 7f803a867640 1 --2- 172.21.15.114:0/2052441726 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f8034100e60 0x7f8034101280 secure :-1 s=READY pgs=56 cs=0 l=1 rev1=1 crypto rx=0x7f8024009900 tx=0x7f802402a410 comp rx=0 tx=0).stop 2024-04-23T10:26:44.280 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:44.280+0000 7f803a867640 1 -- 172.21.15.114:0/2052441726 shutdown_connections 2024-04-23T10:26:44.280 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:44.280+0000 7f803a867640 1 --2- 172.21.15.114:0/2052441726 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f8034102a00 0x7f8034109320 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:26:44.280 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:44.280+0000 7f803a867640 1 --2- 172.21.15.114:0/2052441726 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f8034102040 0x7f80341024c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:26:44.280 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:44.280+0000 7f803a867640 1 --2- 172.21.15.114:0/2052441726 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f8034100e60 0x7f8034101280 unknown :-1 s=CLOSED pgs=56 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:26:44.280 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:44.280+0000 7f803a867640 1 -- 172.21.15.114:0/2052441726 >> 172.21.15.114:0/2052441726 conn(0x7f80340fc440 msgr2=0x7f80340fe860 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T10:26:44.280 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:44.280+0000 7f803a867640 1 -- 172.21.15.114:0/2052441726 shutdown_connections 2024-04-23T10:26:44.280 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:44.280+0000 7f803a867640 1 -- 172.21.15.114:0/2052441726 wait complete. 2024-04-23T10:26:44.280 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:44.281+0000 7f803a867640 1 Processor -- start 2024-04-23T10:26:44.280 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:44.281+0000 7f803a867640 1 -- start start 2024-04-23T10:26:44.282 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:44.281+0000 7f803a867640 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f8034100e60 0x7f80341982a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T10:26:44.282 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:44.281+0000 7f803a867640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f8034102040 0x7f80341987e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T10:26:44.282 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:44.281+0000 7f803a867640 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f8034102a00 0x7f803419f850 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T10:26:44.282 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:44.281+0000 7f803a867640 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x7f803410ca20 con 0x7f8034102040 2024-04-23T10:26:44.282 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:44.281+0000 7f803a867640 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 -- 0x7f803410c8a0 con 0x7f8034100e60 2024-04-23T10:26:44.282 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:44.281+0000 7f803a867640 1 -- --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_getmap magic: 0 -- 0x7f803410cba0 con 0x7f8034102a00 2024-04-23T10:26:44.282 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:44.281+0000 7f80337fe640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f8034102040 0x7f80341987e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T10:26:44.282 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:44.281+0000 7f8033fff640 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f8034100e60 0x7f80341982a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T10:26:44.282 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:44.281+0000 7f80337fe640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f8034102040 0x7f80341987e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.114:40892/0 (socket says 172.21.15.114:40892) 2024-04-23T10:26:44.282 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:44.281+0000 7f8038e04640 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f8034102a00 0x7f803419f850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T10:26:44.282 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:44.281+0000 7f80337fe640 1 -- 172.21.15.114:0/2324190627 learned_addr learned my addr 172.21.15.114:0/2324190627 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-04-23T10:26:44.282 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:44.281+0000 7f8033fff640 1 --2- 172.21.15.114:0/2324190627 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f8034100e60 0x7f80341982a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.119:3300/0 says I am v2:172.21.15.114:58952/0 (socket says 172.21.15.114:58952) 2024-04-23T10:26:44.283 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:44.282+0000 7f8033fff640 1 -- 172.21.15.114:0/2324190627 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f8034102a00 msgr2=0x7f803419f850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T10:26:44.283 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:44.282+0000 7f8033fff640 1 --2- 172.21.15.114:0/2324190627 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f8034102a00 0x7f803419f850 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:26:44.283 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:44.282+0000 7f8033fff640 1 -- 172.21.15.114:0/2324190627 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f8034102040 msgr2=0x7f80341987e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T10:26:44.283 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:44.282+0000 7f8033fff640 1 --2- 172.21.15.114:0/2324190627 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f8034102040 0x7f80341987e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:26:44.283 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:44.282+0000 7f8033fff640 1 -- 172.21.15.114:0/2324190627 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f803419fec0 con 0x7f8034100e60 2024-04-23T10:26:44.283 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:44.282+0000 7f8033fff640 1 --2- 172.21.15.114:0/2324190627 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f8034100e60 0x7f80341982a0 secure :-1 s=READY pgs=57 cs=0 l=1 rev1=1 crypto rx=0x7f802402a930 tx=0x7f8024003c10 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T10:26:44.283 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:44.282+0000 7f80317fa640 1 -- 172.21.15.114:0/2324190627 <== mon.2 v2:172.21.15.119:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f802403f070 con 0x7f8034100e60 2024-04-23T10:26:44.283 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:44.282+0000 7f80317fa640 1 -- 172.21.15.114:0/2324190627 <== mon.2 v2:172.21.15.119:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f8024031c60 con 0x7f8034100e60 2024-04-23T10:26:44.283 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:44.282+0000 7f803a867640 1 -- 172.21.15.114:0/2324190627 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f80341a0150 con 0x7f8034100e60 2024-04-23T10:26:44.283 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:44.282+0000 7f80317fa640 1 -- 172.21.15.114:0/2324190627 <== mon.2 v2:172.21.15.119:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8024035030 con 0x7f8034100e60 2024-04-23T10:26:44.283 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:44.282+0000 7f803a867640 1 -- 172.21.15.114:0/2324190627 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f80341a0610 con 0x7f8034100e60 2024-04-23T10:26:44.283 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:44.283+0000 7f803a867640 1 -- 172.21.15.114:0/2324190627 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f80341a0870 con 0x7f8034100e60 2024-04-23T10:26:44.284 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:44.284+0000 7f80317fa640 1 -- 172.21.15.114:0/2324190627 <== mon.2 v2:172.21.15.119:3300/0 4 ==== mgrmap(e 23) ==== 138767+0+0 (secure 0 0 0) 0x7f8024042070 con 0x7f8034100e60 2024-04-23T10:26:44.284 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:44.284+0000 7f80317fa640 1 --2- 172.21.15.114:0/2324190627 >> [v2:172.21.15.114:6800/1395693466,v1:172.21.15.114:6801/1395693466] conn(0x7f801c0a3d20 0x7f801c0a61e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T10:26:44.284 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:44.285+0000 7f80317fa640 1 -- 172.21.15.114:0/2324190627 <== mon.2 v2:172.21.15.119:3300/0 5 ==== osd_map(90..90 src has 1..90) ==== 8527+0+0 (secure 0 0 0) 0x7f80240f13b0 con 0x7f8034100e60 2024-04-23T10:26:44.284 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:44.285+0000 7f80337fe640 1 --2- 172.21.15.114:0/2324190627 >> [v2:172.21.15.114:6800/1395693466,v1:172.21.15.114:6801/1395693466] conn(0x7f801c0a3d20 0x7f801c0a61e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T10:26:44.284 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:44.285+0000 7f80337fe640 1 --2- 172.21.15.114:0/2324190627 >> [v2:172.21.15.114:6800/1395693466,v1:172.21.15.114:6801/1395693466] conn(0x7f801c0a3d20 0x7f801c0a61e0 secure :-1 s=READY pgs=45 cs=0 l=1 rev1=1 crypto rx=0x7f80341997b0 tx=0x7f8028005f90 comp rx=0 tx=0).ready entity=mgr.15291 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T10:26:44.288 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:44.289+0000 7f80317fa640 1 -- 172.21.15.114:0/2324190627 <== mon.2 v2:172.21.15.119:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f80240bd3e0 con 0x7f8034100e60 2024-04-23T10:26:44.328 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 23 10:26:44 smithi204 ceph-mon[32649]: pgmap v4: 97 pgs: 97 active+clean; 1.1 MiB data, 375 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-23T10:26:44.328 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 23 10:26:44 smithi204 ceph-mon[32649]: from='client.15378 -' entity='client.admin' cmd=[{"prefix": "fs snap-schedule list", "fs": "cephfs", "path": "/", "recursive": true, "target": ["mon-mgr", ""]}]: dispatch 2024-04-23T10:26:44.328 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 23 10:26:44 smithi204 ceph-mon[32649]: mgrmap e23: x(active, since 2s), standbys: z, y 2024-04-23T10:26:44.328 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 23 10:26:44 smithi204 ceph-mon[32649]: from='client.? 172.21.15.114:0/2108459556' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-04-23T10:26:44.328 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 23 10:26:44 smithi204 ceph-mon[32649]: from='client.? 172.21.15.114:0/1382747356' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2024-04-23T10:26:44.328 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 23 10:26:44 smithi204 ceph-mon[32649]: from='mgr.15291 172.21.15.114:0/1095789187' entity='mgr.x' 2024-04-23T10:26:44.328 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 23 10:26:44 smithi204 ceph-mon[32649]: from='mgr.15291 172.21.15.114:0/1095789187' entity='mgr.x' 2024-04-23T10:26:44.328 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 23 10:26:44 smithi204 ceph-mon[32649]: from='mgr.15291 172.21.15.114:0/1095789187' entity='mgr.x' 2024-04-23T10:26:44.328 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 23 10:26:44 smithi204 ceph-mon[32649]: from='mgr.15291 172.21.15.114:0/1095789187' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-04-23T10:26:44.328 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 23 10:26:44 smithi204 ceph-mon[32649]: from='mgr.15291 172.21.15.114:0/1095789187' entity='mgr.x' 2024-04-23T10:26:44.328 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 23 10:26:44 smithi204 ceph-mon[32649]: from='mgr.15291 172.21.15.114:0/1095789187' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-04-23T10:26:44.328 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 23 10:26:44 smithi204 ceph-mon[32649]: from='mgr.15291 172.21.15.114:0/1095789187' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-04-23T10:26:44.328 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 23 10:26:44 smithi204 ceph-mon[32649]: from='mgr.15291 172.21.15.114:0/1095789187' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-04-23T10:26:44.329 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 23 10:26:44 smithi204 ceph-mon[32649]: from='mgr.15291 172.21.15.114:0/1095789187' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-04-23T10:26:44.329 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 23 10:26:44 smithi204 ceph-mon[32649]: from='mgr.15291 172.21.15.114:0/1095789187' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2024-04-23T10:26:44.329 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 23 10:26:44 smithi204 ceph-mon[32649]: from='mgr.15291 172.21.15.114:0/1095789187' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2024-04-23T10:26:44.329 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 23 10:26:44 smithi204 ceph-mon[32649]: from='mgr.15291 172.21.15.114:0/1095789187' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2024-04-23T10:26:44.329 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 23 10:26:44 smithi204 ceph-mon[32649]: from='mgr.15291 172.21.15.114:0/1095789187' entity='mgr.x' 2024-04-23T10:26:44.329 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 23 10:26:44 smithi204 ceph-mon[32649]: from='mgr.15291 172.21.15.114:0/1095789187' entity='mgr.x' 2024-04-23T10:26:44.329 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 23 10:26:44 smithi204 ceph-mon[32649]: from='mgr.15291 172.21.15.114:0/1095789187' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-04-23T10:26:44.329 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 23 10:26:44 smithi204 ceph-mon[32649]: from='mgr.15291 172.21.15.114:0/1095789187' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-04-23T10:26:44.329 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 23 10:26:44 smithi204 ceph-mon[32649]: from='mgr.15291 172.21.15.114:0/1095789187' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-04-23T10:26:44.329 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 23 10:26:44 smithi204 ceph-mon[32649]: from='mgr.15291 172.21.15.114:0/1095789187' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"}]: dispatch 2024-04-23T10:26:44.329 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 23 10:26:44 smithi204 ceph-mon[32649]: from='mgr.15291 172.21.15.114:0/1095789187' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-23T10:26:44.329 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 23 10:26:44 smithi204 ceph-mon[32649]: from='mgr.15291 172.21.15.114:0/1095789187' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-23T10:26:44.329 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 23 10:26:44 smithi204 ceph-mon[32649]: from='client.? 172.21.15.114:0/2318849018' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2024-04-23T10:26:44.477 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:44.478+0000 7f803a867640 1 -- 172.21.15.114:0/2324190627 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) -- 0x7f80341a0a40 con 0x7f8034100e60 2024-04-23T10:26:44.477 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:44.479+0000 7f80317fa640 1 -- 172.21.15.114:0/2324190627 <== mon.2 v2:172.21.15.119:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v90) ==== 93+0+10 (secure 0 0 0) 0x7f80240c16e0 con 0x7f8034100e60 2024-04-23T10:26:44.478 INFO:teuthology.orchestra.run.smithi114.stdout:pg_num: 1 2024-04-23T10:26:44.480 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:44.481+0000 7f803a867640 1 -- 172.21.15.114:0/2324190627 >> [v2:172.21.15.114:6800/1395693466,v1:172.21.15.114:6801/1395693466] conn(0x7f801c0a3d20 msgr2=0x7f801c0a61e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T10:26:44.480 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:44.481+0000 7f803a867640 1 --2- 172.21.15.114:0/2324190627 >> [v2:172.21.15.114:6800/1395693466,v1:172.21.15.114:6801/1395693466] conn(0x7f801c0a3d20 0x7f801c0a61e0 secure :-1 s=READY pgs=45 cs=0 l=1 rev1=1 crypto rx=0x7f80341997b0 tx=0x7f8028005f90 comp rx=0 tx=0).stop 2024-04-23T10:26:44.480 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:44.481+0000 7f803a867640 1 -- 172.21.15.114:0/2324190627 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f8034100e60 msgr2=0x7f80341982a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T10:26:44.480 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:44.481+0000 7f803a867640 1 --2- 172.21.15.114:0/2324190627 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f8034100e60 0x7f80341982a0 secure :-1 s=READY pgs=57 cs=0 l=1 rev1=1 crypto rx=0x7f802402a930 tx=0x7f8024003c10 comp rx=0 tx=0).stop 2024-04-23T10:26:44.480 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:44.481+0000 7f803a867640 1 -- 172.21.15.114:0/2324190627 shutdown_connections 2024-04-23T10:26:44.481 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:44.481+0000 7f803a867640 1 --2- 172.21.15.114:0/2324190627 >> [v2:172.21.15.114:6800/1395693466,v1:172.21.15.114:6801/1395693466] conn(0x7f801c0a3d20 0x7f801c0a61e0 unknown :-1 s=CLOSED pgs=45 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:26:44.481 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:44.481+0000 7f803a867640 1 --2- 172.21.15.114:0/2324190627 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f8034102a00 0x7f803419f850 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:26:44.481 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:44.481+0000 7f803a867640 1 --2- 172.21.15.114:0/2324190627 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f8034102040 0x7f80341987e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:26:44.481 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:44.481+0000 7f803a867640 1 --2- 172.21.15.114:0/2324190627 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f8034100e60 0x7f80341982a0 unknown :-1 s=CLOSED pgs=57 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:26:44.481 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:44.481+0000 7f803a867640 1 -- 172.21.15.114:0/2324190627 >> 172.21.15.114:0/2324190627 conn(0x7f80340fc440 msgr2=0x7f803406a2f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T10:26:44.481 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:44.481+0000 7f803a867640 1 -- 172.21.15.114:0/2324190627 shutdown_connections 2024-04-23T10:26:44.481 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:44.481+0000 7f803a867640 1 -- 172.21.15.114:0/2324190627 wait complete. 2024-04-23T10:26:44.491 DEBUG:teuthology.orchestra.run.smithi114:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2024-04-23T10:26:44.492 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 23 10:26:44 smithi119 ceph-mon[32608]: pgmap v4: 97 pgs: 97 active+clean; 1.1 MiB data, 375 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-23T10:26:44.492 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 23 10:26:44 smithi119 ceph-mon[32608]: from='client.15378 -' entity='client.admin' cmd=[{"prefix": "fs snap-schedule list", "fs": "cephfs", "path": "/", "recursive": true, "target": ["mon-mgr", ""]}]: dispatch 2024-04-23T10:26:44.492 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 23 10:26:44 smithi119 ceph-mon[32608]: mgrmap e23: x(active, since 2s), standbys: z, y 2024-04-23T10:26:44.492 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 23 10:26:44 smithi119 ceph-mon[32608]: from='client.? 172.21.15.114:0/2108459556' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-04-23T10:26:44.492 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 23 10:26:44 smithi119 ceph-mon[32608]: from='client.? 172.21.15.114:0/1382747356' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2024-04-23T10:26:44.492 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 23 10:26:44 smithi119 ceph-mon[32608]: from='mgr.15291 172.21.15.114:0/1095789187' entity='mgr.x' 2024-04-23T10:26:44.492 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 23 10:26:44 smithi119 ceph-mon[32608]: from='mgr.15291 172.21.15.114:0/1095789187' entity='mgr.x' 2024-04-23T10:26:44.493 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 23 10:26:44 smithi119 ceph-mon[32608]: from='mgr.15291 172.21.15.114:0/1095789187' entity='mgr.x' 2024-04-23T10:26:44.493 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 23 10:26:44 smithi119 ceph-mon[32608]: from='mgr.15291 172.21.15.114:0/1095789187' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-04-23T10:26:44.493 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 23 10:26:44 smithi119 ceph-mon[32608]: from='mgr.15291 172.21.15.114:0/1095789187' entity='mgr.x' 2024-04-23T10:26:44.493 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 23 10:26:44 smithi119 ceph-mon[32608]: from='mgr.15291 172.21.15.114:0/1095789187' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-04-23T10:26:44.493 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 23 10:26:44 smithi119 ceph-mon[32608]: from='mgr.15291 172.21.15.114:0/1095789187' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-04-23T10:26:44.493 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 23 10:26:44 smithi119 ceph-mon[32608]: from='mgr.15291 172.21.15.114:0/1095789187' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-04-23T10:26:44.493 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 23 10:26:44 smithi119 ceph-mon[32608]: from='mgr.15291 172.21.15.114:0/1095789187' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-04-23T10:26:44.493 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 23 10:26:44 smithi119 ceph-mon[32608]: from='mgr.15291 172.21.15.114:0/1095789187' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2024-04-23T10:26:44.493 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 23 10:26:44 smithi119 ceph-mon[32608]: from='mgr.15291 172.21.15.114:0/1095789187' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2024-04-23T10:26:44.493 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 23 10:26:44 smithi119 ceph-mon[32608]: from='mgr.15291 172.21.15.114:0/1095789187' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2024-04-23T10:26:44.493 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 23 10:26:44 smithi119 ceph-mon[32608]: from='mgr.15291 172.21.15.114:0/1095789187' entity='mgr.x' 2024-04-23T10:26:44.493 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 23 10:26:44 smithi119 ceph-mon[32608]: from='mgr.15291 172.21.15.114:0/1095789187' entity='mgr.x' 2024-04-23T10:26:44.493 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 23 10:26:44 smithi119 ceph-mon[32608]: from='mgr.15291 172.21.15.114:0/1095789187' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-04-23T10:26:44.493 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 23 10:26:44 smithi119 ceph-mon[32608]: from='mgr.15291 172.21.15.114:0/1095789187' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-04-23T10:26:44.493 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 23 10:26:44 smithi119 ceph-mon[32608]: from='mgr.15291 172.21.15.114:0/1095789187' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-04-23T10:26:44.494 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 23 10:26:44 smithi119 ceph-mon[32608]: from='mgr.15291 172.21.15.114:0/1095789187' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"}]: dispatch 2024-04-23T10:26:44.494 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 23 10:26:44 smithi119 ceph-mon[32608]: from='mgr.15291 172.21.15.114:0/1095789187' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-23T10:26:44.494 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 23 10:26:44 smithi119 ceph-mon[32608]: from='mgr.15291 172.21.15.114:0/1095789187' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-23T10:26:44.494 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 23 10:26:44 smithi119 ceph-mon[32608]: from='client.? 172.21.15.114:0/2318849018' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2024-04-23T10:26:44.613 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:44.613+0000 7f9f34d76640 1 -- 172.21.15.114:0/3714417748 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f9f30102b30 msgr2=0x7f9f30106b70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T10:26:44.613 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:44.613+0000 7f9f34d76640 1 --2- 172.21.15.114:0/3714417748 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f9f30102b30 0x7f9f30106b70 secure :-1 s=READY pgs=441 cs=0 l=1 rev1=1 crypto rx=0x7f9f24009900 tx=0x7f9f2402a410 comp rx=0 tx=0).stop 2024-04-23T10:26:44.613 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:44.614+0000 7f9f34d76640 1 -- 172.21.15.114:0/3714417748 shutdown_connections 2024-04-23T10:26:44.613 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:44.614+0000 7f9f34d76640 1 --2- 172.21.15.114:0/3714417748 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f9f30107250 0x7f9f3010ee20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:26:44.613 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:44.614+0000 7f9f34d76640 1 --2- 172.21.15.114:0/3714417748 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f9f30102b30 0x7f9f30106b70 unknown :-1 s=CLOSED pgs=441 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:26:44.613 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:44.614+0000 7f9f34d76640 1 --2- 172.21.15.114:0/3714417748 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f9f30102180 0x7f9f30102560 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:26:44.613 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:44.614+0000 7f9f34d76640 1 -- 172.21.15.114:0/3714417748 >> 172.21.15.114:0/3714417748 conn(0x7f9f300fda20 msgr2=0x7f9f300ffe40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T10:26:44.613 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:44.614+0000 7f9f34d76640 1 -- 172.21.15.114:0/3714417748 shutdown_connections 2024-04-23T10:26:44.613 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:44.614+0000 7f9f34d76640 1 -- 172.21.15.114:0/3714417748 wait complete. 2024-04-23T10:26:44.615 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:44.614+0000 7f9f34d76640 1 Processor -- start 2024-04-23T10:26:44.615 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:44.614+0000 7f9f34d76640 1 -- start start 2024-04-23T10:26:44.615 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:44.615+0000 7f9f34d76640 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f9f30102180 0x7f9f30198790 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T10:26:44.615 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:44.615+0000 7f9f34d76640 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f9f30107250 0x7f9f30198cd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T10:26:44.615 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:44.615+0000 7f9f34d76640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f9f3019d000 0x7f9f3019d4b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T10:26:44.616 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:44.615+0000 7f9f34d76640 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x7f9f30112190 con 0x7f9f3019d000 2024-04-23T10:26:44.616 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:44.615+0000 7f9f34d76640 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 -- 0x7f9f30112010 con 0x7f9f30107250 2024-04-23T10:26:44.616 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:44.615+0000 7f9f34d76640 1 -- --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_getmap magic: 0 -- 0x7f9f30112310 con 0x7f9f30102180 2024-04-23T10:26:44.616 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:44.615+0000 7f9f2ed9d640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f9f3019d000 0x7f9f3019d4b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T10:26:44.616 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:44.615+0000 7f9f2ed9d640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f9f3019d000 0x7f9f3019d4b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.114:40904/0 (socket says 172.21.15.114:40904) 2024-04-23T10:26:44.616 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:44.615+0000 7f9f2ed9d640 1 -- 172.21.15.114:0/2624372018 learned_addr learned my addr 172.21.15.114:0/2624372018 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-04-23T10:26:44.616 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:44.615+0000 7f9f2e59c640 1 --2- 172.21.15.114:0/2624372018 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f9f30102180 0x7f9f30198790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T10:26:44.616 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:44.615+0000 7f9f2dd9b640 1 --2- 172.21.15.114:0/2624372018 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f9f30107250 0x7f9f30198cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T10:26:44.616 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:44.615+0000 7f9f2ed9d640 1 -- 172.21.15.114:0/2624372018 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f9f30102180 msgr2=0x7f9f30198790 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T10:26:44.616 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:44.615+0000 7f9f2ed9d640 1 --2- 172.21.15.114:0/2624372018 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f9f30102180 0x7f9f30198790 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:26:44.616 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:44.615+0000 7f9f2ed9d640 1 -- 172.21.15.114:0/2624372018 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f9f30107250 msgr2=0x7f9f30198cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T10:26:44.616 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:44.615+0000 7f9f2ed9d640 1 --2- 172.21.15.114:0/2624372018 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f9f30107250 0x7f9f30198cd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:26:44.616 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:44.615+0000 7f9f2ed9d640 1 -- 172.21.15.114:0/2624372018 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f9f3019daf0 con 0x7f9f3019d000 2024-04-23T10:26:44.616 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:44.615+0000 7f9f2dd9b640 1 --2- 172.21.15.114:0/2624372018 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f9f30107250 0x7f9f30198cd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-23T10:26:44.616 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:44.615+0000 7f9f2ed9d640 1 --2- 172.21.15.114:0/2624372018 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f9f3019d000 0x7f9f3019d4b0 secure :-1 s=READY pgs=442 cs=0 l=1 rev1=1 crypto rx=0x7f9f2000eb10 tx=0x7f9f2000c2d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T10:26:44.617 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:44.615+0000 7f9f1f7fe640 1 -- 172.21.15.114:0/2624372018 <== mon.0 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9f20004b20 con 0x7f9f3019d000 2024-04-23T10:26:44.617 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:44.615+0000 7f9f1f7fe640 1 -- 172.21.15.114:0/2624372018 <== mon.0 v2:172.21.15.114:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f9f20004cc0 con 0x7f9f3019d000 2024-04-23T10:26:44.617 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:44.615+0000 7f9f34d76640 1 -- 172.21.15.114:0/2624372018 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f9f3019dde0 con 0x7f9f3019d000 2024-04-23T10:26:44.617 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:44.615+0000 7f9f1f7fe640 1 -- 172.21.15.114:0/2624372018 <== mon.0 v2:172.21.15.114:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9f20010670 con 0x7f9f3019d000 2024-04-23T10:26:44.617 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:44.615+0000 7f9f34d76640 1 -- 172.21.15.114:0/2624372018 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f9f301a55a0 con 0x7f9f3019d000 2024-04-23T10:26:44.617 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:44.617+0000 7f9f34d76640 1 -- 172.21.15.114:0/2624372018 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f9f30068d70 con 0x7f9f3019d000 2024-04-23T10:26:44.618 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:44.617+0000 7f9f1f7fe640 1 -- 172.21.15.114:0/2624372018 <== mon.0 v2:172.21.15.114:3300/0 4 ==== mgrmap(e 23) ==== 138767+0+0 (secure 0 0 0) 0x7f9f20005020 con 0x7f9f3019d000 2024-04-23T10:26:44.618 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:44.618+0000 7f9f1f7fe640 1 --2- 172.21.15.114:0/2624372018 >> [v2:172.21.15.114:6800/1395693466,v1:172.21.15.114:6801/1395693466] conn(0x7f9f180a3d20 0x7f9f180a61e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T10:26:44.618 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:44.618+0000 7f9f2e59c640 1 --2- 172.21.15.114:0/2624372018 >> [v2:172.21.15.114:6800/1395693466,v1:172.21.15.114:6801/1395693466] conn(0x7f9f180a3d20 0x7f9f180a61e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T10:26:44.618 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:44.618+0000 7f9f1f7fe640 1 -- 172.21.15.114:0/2624372018 <== mon.0 v2:172.21.15.114:3300/0 5 ==== osd_map(90..90 src has 1..90) ==== 8527+0+0 (secure 0 0 0) 0x7f9f200d0d50 con 0x7f9f3019d000 2024-04-23T10:26:44.618 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:44.619+0000 7f9f2e59c640 1 --2- 172.21.15.114:0/2624372018 >> [v2:172.21.15.114:6800/1395693466,v1:172.21.15.114:6801/1395693466] conn(0x7f9f180a3d20 0x7f9f180a61e0 secure :-1 s=READY pgs=46 cs=0 l=1 rev1=1 crypto rx=0x7f9f280095f0 tx=0x7f9f28005f90 comp rx=0 tx=0).ready entity=mgr.15291 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T10:26:44.621 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:44.622+0000 7f9f1f7fe640 1 -- 172.21.15.114:0/2624372018 <== mon.0 v2:172.21.15.114:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f9f200d4030 con 0x7f9f3019d000 2024-04-23T10:26:44.810 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:44.812+0000 7f9f34d76640 1 -- 172.21.15.114:0/2624372018 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) -- 0x7f9f3010a4e0 con 0x7f9f3019d000 2024-04-23T10:26:44.811 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:44.812+0000 7f9f1f7fe640 1 -- 172.21.15.114:0/2624372018 <== mon.0 v2:172.21.15.114:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v90) ==== 104+0+11 (secure 0 0 0) 0x7f9f2009d3f0 con 0x7f9f3019d000 2024-04-23T10:26:44.811 INFO:teuthology.orchestra.run.smithi114.stdout:pg_num: 32 2024-04-23T10:26:44.813 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:44.814+0000 7f9f34d76640 1 -- 172.21.15.114:0/2624372018 >> [v2:172.21.15.114:6800/1395693466,v1:172.21.15.114:6801/1395693466] conn(0x7f9f180a3d20 msgr2=0x7f9f180a61e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T10:26:44.813 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:44.814+0000 7f9f34d76640 1 --2- 172.21.15.114:0/2624372018 >> [v2:172.21.15.114:6800/1395693466,v1:172.21.15.114:6801/1395693466] conn(0x7f9f180a3d20 0x7f9f180a61e0 secure :-1 s=READY pgs=46 cs=0 l=1 rev1=1 crypto rx=0x7f9f280095f0 tx=0x7f9f28005f90 comp rx=0 tx=0).stop 2024-04-23T10:26:44.813 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:44.814+0000 7f9f34d76640 1 -- 172.21.15.114:0/2624372018 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f9f3019d000 msgr2=0x7f9f3019d4b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T10:26:44.813 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:44.814+0000 7f9f34d76640 1 --2- 172.21.15.114:0/2624372018 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f9f3019d000 0x7f9f3019d4b0 secure :-1 s=READY pgs=442 cs=0 l=1 rev1=1 crypto rx=0x7f9f2000eb10 tx=0x7f9f2000c2d0 comp rx=0 tx=0).stop 2024-04-23T10:26:44.813 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:44.814+0000 7f9f34d76640 1 -- 172.21.15.114:0/2624372018 shutdown_connections 2024-04-23T10:26:44.813 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:44.814+0000 7f9f34d76640 1 --2- 172.21.15.114:0/2624372018 >> [v2:172.21.15.114:6800/1395693466,v1:172.21.15.114:6801/1395693466] conn(0x7f9f180a3d20 0x7f9f180a61e0 unknown :-1 s=CLOSED pgs=46 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:26:44.813 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:44.814+0000 7f9f34d76640 1 --2- 172.21.15.114:0/2624372018 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f9f3019d000 0x7f9f3019d4b0 unknown :-1 s=CLOSED pgs=442 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:26:44.814 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:44.814+0000 7f9f34d76640 1 --2- 172.21.15.114:0/2624372018 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f9f30107250 0x7f9f30198cd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:26:44.814 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:44.814+0000 7f9f34d76640 1 --2- 172.21.15.114:0/2624372018 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f9f30102180 0x7f9f30198790 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:26:44.814 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:44.814+0000 7f9f34d76640 1 -- 172.21.15.114:0/2624372018 >> 172.21.15.114:0/2624372018 conn(0x7f9f300fda20 msgr2=0x7f9f300fee50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T10:26:44.814 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:44.814+0000 7f9f34d76640 1 -- 172.21.15.114:0/2624372018 shutdown_connections 2024-04-23T10:26:44.814 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:44.814+0000 7f9f34d76640 1 -- 172.21.15.114:0/2624372018 wait complete. 2024-04-23T10:26:44.824 DEBUG:teuthology.orchestra.run.smithi114:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2024-04-23T10:26:44.927 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:44.928+0000 7fec5dccf640 1 -- 172.21.15.114:0/1376712302 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fec5806a820 msgr2=0x7fec5806ac00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T10:26:44.927 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:44.928+0000 7fec5dccf640 1 --2- 172.21.15.114:0/1376712302 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fec5806a820 0x7fec5806ac00 secure :-1 s=READY pgs=443 cs=0 l=1 rev1=1 crypto rx=0x7fec4c009900 tx=0x7fec4c02a410 comp rx=0 tx=0).stop 2024-04-23T10:26:44.927 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:44.928+0000 7fec5dccf640 1 -- 172.21.15.114:0/1376712302 shutdown_connections 2024-04-23T10:26:44.927 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:44.928+0000 7fec5dccf640 1 --2- 172.21.15.114:0/1376712302 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fec58107270 0x7fec5810ee40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:26:44.927 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:44.928+0000 7fec5dccf640 1 --2- 172.21.15.114:0/1376712302 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fec5806b1d0 0x7fec58106b90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:26:44.927 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:44.928+0000 7fec5dccf640 1 --2- 172.21.15.114:0/1376712302 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fec5806a820 0x7fec5806ac00 unknown :-1 s=CLOSED pgs=443 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:26:44.928 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:44.928+0000 7fec5dccf640 1 -- 172.21.15.114:0/1376712302 >> 172.21.15.114:0/1376712302 conn(0x7fec58077eb0 msgr2=0x7fec580782c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T10:26:44.928 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:44.928+0000 7fec5dccf640 1 -- 172.21.15.114:0/1376712302 shutdown_connections 2024-04-23T10:26:44.928 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:44.928+0000 7fec5dccf640 1 -- 172.21.15.114:0/1376712302 wait complete. 2024-04-23T10:26:44.930 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:44.929+0000 7fec5dccf640 1 Processor -- start 2024-04-23T10:26:44.930 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:44.929+0000 7fec5dccf640 1 -- start start 2024-04-23T10:26:44.930 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:44.929+0000 7fec5dccf640 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fec5806a820 0x7fec5819e410 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T10:26:44.930 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:44.929+0000 7fec5dccf640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fec58107270 0x7fec5819e950 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T10:26:44.930 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:44.929+0000 7fec5dccf640 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fec581985c0 0x7fec58198a70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T10:26:44.930 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:44.929+0000 7fec5dccf640 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x7fec581121e0 con 0x7fec58107270 2024-04-23T10:26:44.930 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:44.929+0000 7fec5dccf640 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 -- 0x7fec58112060 con 0x7fec581985c0 2024-04-23T10:26:44.930 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:44.929+0000 7fec5dccf640 1 -- --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_getmap magic: 0 -- 0x7fec58112360 con 0x7fec5806a820 2024-04-23T10:26:44.930 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:44.929+0000 7fec56ffd640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fec58107270 0x7fec5819e950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T10:26:44.931 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:44.930+0000 7fec577fe640 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fec5806a820 0x7fec5819e410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T10:26:44.931 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:44.930+0000 7fec577fe640 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fec5806a820 0x7fec5819e410 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.204:3300/0 says I am v2:172.21.15.114:37640/0 (socket says 172.21.15.114:37640) 2024-04-23T10:26:44.931 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:44.930+0000 7fec56ffd640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fec58107270 0x7fec5819e950 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.114:40920/0 (socket says 172.21.15.114:40920) 2024-04-23T10:26:44.931 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:44.930+0000 7fec577fe640 1 -- 172.21.15.114:0/3698241698 learned_addr learned my addr 172.21.15.114:0/3698241698 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-04-23T10:26:44.931 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:44.930+0000 7fec57fff640 1 --2- 172.21.15.114:0/3698241698 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fec581985c0 0x7fec58198a70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T10:26:44.931 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:44.930+0000 7fec577fe640 1 -- 172.21.15.114:0/3698241698 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fec581985c0 msgr2=0x7fec58198a70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T10:26:44.931 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:44.930+0000 7fec577fe640 1 --2- 172.21.15.114:0/3698241698 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fec581985c0 0x7fec58198a70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:26:44.931 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:44.930+0000 7fec577fe640 1 -- 172.21.15.114:0/3698241698 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fec58107270 msgr2=0x7fec5819e950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T10:26:44.931 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:44.930+0000 7fec577fe640 1 --2- 172.21.15.114:0/3698241698 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fec58107270 0x7fec5819e950 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:26:44.931 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:44.930+0000 7fec577fe640 1 -- 172.21.15.114:0/3698241698 --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fec581990e0 con 0x7fec5806a820 2024-04-23T10:26:44.931 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:44.930+0000 7fec56ffd640 1 --2- 172.21.15.114:0/3698241698 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fec58107270 0x7fec5819e950 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-23T10:26:44.931 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:44.930+0000 7fec577fe640 1 --2- 172.21.15.114:0/3698241698 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fec5806a820 0x7fec5819e410 secure :-1 s=READY pgs=62 cs=0 l=1 rev1=1 crypto rx=0x7fec4c005940 tx=0x7fec4c004270 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T10:26:44.931 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:44.930+0000 7fec54ff9640 1 -- 172.21.15.114:0/3698241698 <== mon.1 v2:172.21.15.204:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fec4c03f030 con 0x7fec5806a820 2024-04-23T10:26:44.931 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:44.930+0000 7fec54ff9640 1 -- 172.21.15.114:0/3698241698 <== mon.1 v2:172.21.15.204:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fec4c003aa0 con 0x7fec5806a820 2024-04-23T10:26:44.931 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:44.930+0000 7fec5dccf640 1 -- 172.21.15.114:0/3698241698 --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fec58199370 con 0x7fec5806a820 2024-04-23T10:26:44.931 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:44.930+0000 7fec54ff9640 1 -- 172.21.15.114:0/3698241698 <== mon.1 v2:172.21.15.204:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fec4c035070 con 0x7fec5806a820 2024-04-23T10:26:44.932 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:44.930+0000 7fec5dccf640 1 -- 172.21.15.114:0/3698241698 --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fec581a5560 con 0x7fec5806a820 2024-04-23T10:26:44.932 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:44.931+0000 7fec5dccf640 1 -- 172.21.15.114:0/3698241698 --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fec580691c0 con 0x7fec5806a820 2024-04-23T10:26:44.932 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:44.932+0000 7fec54ff9640 1 -- 172.21.15.114:0/3698241698 <== mon.1 v2:172.21.15.204:3300/0 4 ==== mgrmap(e 23) ==== 138767+0+0 (secure 0 0 0) 0x7fec4c042030 con 0x7fec5806a820 2024-04-23T10:26:44.932 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:44.933+0000 7fec54ff9640 1 --2- 172.21.15.114:0/3698241698 >> [v2:172.21.15.114:6800/1395693466,v1:172.21.15.114:6801/1395693466] conn(0x7fec400a3d20 0x7fec400a61e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T10:26:44.933 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:44.933+0000 7fec56ffd640 1 --2- 172.21.15.114:0/3698241698 >> [v2:172.21.15.114:6800/1395693466,v1:172.21.15.114:6801/1395693466] conn(0x7fec400a3d20 0x7fec400a61e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T10:26:44.933 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:44.933+0000 7fec54ff9640 1 -- 172.21.15.114:0/3698241698 <== mon.1 v2:172.21.15.204:3300/0 5 ==== osd_map(90..90 src has 1..90) ==== 8527+0+0 (secure 0 0 0) 0x7fec4c0f0f40 con 0x7fec5806a820 2024-04-23T10:26:44.933 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:44.933+0000 7fec56ffd640 1 --2- 172.21.15.114:0/3698241698 >> [v2:172.21.15.114:6800/1395693466,v1:172.21.15.114:6801/1395693466] conn(0x7fec400a3d20 0x7fec400a61e0 secure :-1 s=READY pgs=47 cs=0 l=1 rev1=1 crypto rx=0x7fec581009c0 tx=0x7fec4800b390 comp rx=0 tx=0).ready entity=mgr.15291 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T10:26:44.936 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:44.937+0000 7fec54ff9640 1 -- 172.21.15.114:0/3698241698 <== mon.1 v2:172.21.15.204:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7fec4c0bd580 con 0x7fec5806a820 2024-04-23T10:26:45.031 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:45.032+0000 7fec54ff9640 1 -- 172.21.15.114:0/3698241698 <== mon.1 v2:172.21.15.204:3300/0 7 ==== mgrmap(e 24) ==== 138823+0+0 (secure 0 0 0) 0x7fec4c0b94f0 con 0x7fec5806a820 2024-04-23T10:26:45.120 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:45.121+0000 7fec5dccf640 1 -- 172.21.15.114:0/3698241698 --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) -- 0x7fec5806b1d0 con 0x7fec5806a820 2024-04-23T10:26:45.120 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:45.122+0000 7fec54ff9640 1 -- 172.21.15.114:0/3698241698 <== mon.1 v2:172.21.15.204:3300/0 8 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v90) ==== 100+0+11 (secure 0 0 0) 0x7fec4c0ae5d0 con 0x7fec5806a820 2024-04-23T10:26:45.120 INFO:teuthology.orchestra.run.smithi114.stdout:pg_num: 64 2024-04-23T10:26:45.122 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:45.123+0000 7fec5dccf640 1 -- 172.21.15.114:0/3698241698 >> [v2:172.21.15.114:6800/1395693466,v1:172.21.15.114:6801/1395693466] conn(0x7fec400a3d20 msgr2=0x7fec400a61e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T10:26:45.123 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:45.123+0000 7fec5dccf640 1 --2- 172.21.15.114:0/3698241698 >> [v2:172.21.15.114:6800/1395693466,v1:172.21.15.114:6801/1395693466] conn(0x7fec400a3d20 0x7fec400a61e0 secure :-1 s=READY pgs=47 cs=0 l=1 rev1=1 crypto rx=0x7fec581009c0 tx=0x7fec4800b390 comp rx=0 tx=0).stop 2024-04-23T10:26:45.123 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:45.123+0000 7fec5dccf640 1 -- 172.21.15.114:0/3698241698 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fec5806a820 msgr2=0x7fec5819e410 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T10:26:45.123 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:45.123+0000 7fec5dccf640 1 --2- 172.21.15.114:0/3698241698 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fec5806a820 0x7fec5819e410 secure :-1 s=READY pgs=62 cs=0 l=1 rev1=1 crypto rx=0x7fec4c005940 tx=0x7fec4c004270 comp rx=0 tx=0).stop 2024-04-23T10:26:45.123 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:45.124+0000 7fec5dccf640 1 -- 172.21.15.114:0/3698241698 shutdown_connections 2024-04-23T10:26:45.123 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:45.124+0000 7fec5dccf640 1 --2- 172.21.15.114:0/3698241698 >> [v2:172.21.15.114:6800/1395693466,v1:172.21.15.114:6801/1395693466] conn(0x7fec400a3d20 0x7fec400a61e0 unknown :-1 s=CLOSED pgs=47 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:26:45.123 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:45.124+0000 7fec5dccf640 1 --2- 172.21.15.114:0/3698241698 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fec581985c0 0x7fec58198a70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:26:45.123 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:45.124+0000 7fec5dccf640 1 --2- 172.21.15.114:0/3698241698 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fec58107270 0x7fec5819e950 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:26:45.123 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:45.124+0000 7fec5dccf640 1 --2- 172.21.15.114:0/3698241698 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fec5806a820 0x7fec5819e410 unknown :-1 s=CLOSED pgs=62 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:26:45.123 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:45.124+0000 7fec5dccf640 1 -- 172.21.15.114:0/3698241698 >> 172.21.15.114:0/3698241698 conn(0x7fec58077eb0 msgr2=0x7fec580ff0b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T10:26:45.123 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:45.124+0000 7fec5dccf640 1 -- 172.21.15.114:0/3698241698 shutdown_connections 2024-04-23T10:26:45.123 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:45.124+0000 7fec5dccf640 1 -- 172.21.15.114:0/3698241698 wait complete. 2024-04-23T10:26:45.133 INFO:journalctl@ceph.mon.a.smithi114.stdout:Apr 23 10:26:45 smithi114 ceph-mon[29255]: Adjusting osd_memory_target on smithi119 to 176.2M 2024-04-23T10:26:45.133 INFO:journalctl@ceph.mon.a.smithi114.stdout:Apr 23 10:26:45 smithi114 ceph-mon[29255]: Unable to set osd_memory_target on smithi119 to 184770252: error parsing value: Value '184770252' is below minimum 939524096 2024-04-23T10:26:45.133 INFO:journalctl@ceph.mon.a.smithi114.stdout:Apr 23 10:26:45 smithi114 ceph-mon[29255]: Adjusting osd_memory_target on smithi204 to 176.2M 2024-04-23T10:26:45.133 INFO:journalctl@ceph.mon.a.smithi114.stdout:Apr 23 10:26:45 smithi114 ceph-mon[29255]: Unable to set osd_memory_target on smithi204 to 184759500: error parsing value: Value '184759500' is below minimum 939524096 2024-04-23T10:26:45.133 INFO:journalctl@ceph.mon.a.smithi114.stdout:Apr 23 10:26:45 smithi114 ceph-mon[29255]: Adjusting osd_memory_target on smithi114 to 176.2M 2024-04-23T10:26:45.133 INFO:journalctl@ceph.mon.a.smithi114.stdout:Apr 23 10:26:45 smithi114 ceph-mon[29255]: Unable to set osd_memory_target on smithi114 to 184769536: error parsing value: Value '184769536' is below minimum 939524096 2024-04-23T10:26:45.133 INFO:journalctl@ceph.mon.a.smithi114.stdout:Apr 23 10:26:45 smithi114 ceph-mon[29255]: Updating smithi114:/etc/ceph/ceph.conf 2024-04-23T10:26:45.134 INFO:journalctl@ceph.mon.a.smithi114.stdout:Apr 23 10:26:45 smithi114 ceph-mon[29255]: Updating smithi119:/etc/ceph/ceph.conf 2024-04-23T10:26:45.134 INFO:journalctl@ceph.mon.a.smithi114.stdout:Apr 23 10:26:45 smithi114 ceph-mon[29255]: Updating smithi204:/etc/ceph/ceph.conf 2024-04-23T10:26:45.134 INFO:journalctl@ceph.mon.a.smithi114.stdout:Apr 23 10:26:45 smithi114 ceph-mon[29255]: Updating smithi204:/var/lib/ceph/aa240572-0159-11ef-bc93-c7b262605968/config/ceph.conf 2024-04-23T10:26:45.134 INFO:journalctl@ceph.mon.a.smithi114.stdout:Apr 23 10:26:45 smithi114 ceph-mon[29255]: Updating smithi114:/var/lib/ceph/aa240572-0159-11ef-bc93-c7b262605968/config/ceph.conf 2024-04-23T10:26:45.134 INFO:journalctl@ceph.mon.a.smithi114.stdout:Apr 23 10:26:45 smithi114 ceph-mon[29255]: Updating smithi119:/var/lib/ceph/aa240572-0159-11ef-bc93-c7b262605968/config/ceph.conf 2024-04-23T10:26:45.134 INFO:journalctl@ceph.mon.a.smithi114.stdout:Apr 23 10:26:45 smithi114 ceph-mon[29255]: from='client.? 172.21.15.114:0/2609718743' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-23T10:26:45.134 INFO:journalctl@ceph.mon.a.smithi114.stdout:Apr 23 10:26:45 smithi114 ceph-mon[29255]: from='client.? 172.21.15.114:0/2324190627' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-04-23T10:26:45.134 INFO:journalctl@ceph.mon.a.smithi114.stdout:Apr 23 10:26:45 smithi114 ceph-mon[29255]: from='mgr.15291 172.21.15.114:0/1095789187' entity='mgr.x' 2024-04-23T10:26:45.134 INFO:journalctl@ceph.mon.a.smithi114.stdout:Apr 23 10:26:45 smithi114 ceph-mon[29255]: from='mgr.15291 172.21.15.114:0/1095789187' entity='mgr.x' 2024-04-23T10:26:45.134 INFO:journalctl@ceph.mon.a.smithi114.stdout:Apr 23 10:26:45 smithi114 ceph-mon[29255]: from='mgr.15291 172.21.15.114:0/1095789187' entity='mgr.x' 2024-04-23T10:26:45.134 INFO:journalctl@ceph.mon.a.smithi114.stdout:Apr 23 10:26:45 smithi114 ceph-mon[29255]: from='mgr.15291 172.21.15.114:0/1095789187' entity='mgr.x' 2024-04-23T10:26:45.134 INFO:journalctl@ceph.mon.a.smithi114.stdout:Apr 23 10:26:45 smithi114 ceph-mon[29255]: from='mgr.15291 172.21.15.114:0/1095789187' entity='mgr.x' 2024-04-23T10:26:45.134 INFO:journalctl@ceph.mon.a.smithi114.stdout:Apr 23 10:26:45 smithi114 ceph-mon[29255]: from='mgr.15291 172.21.15.114:0/1095789187' entity='mgr.x' 2024-04-23T10:26:45.134 INFO:journalctl@ceph.mon.a.smithi114.stdout:Apr 23 10:26:45 smithi114 ceph-mon[29255]: from='mgr.15291 172.21.15.114:0/1095789187' entity='mgr.x' 2024-04-23T10:26:45.134 INFO:journalctl@ceph.mon.a.smithi114.stdout:Apr 23 10:26:45 smithi114 ceph-mon[29255]: from='client.? 172.21.15.114:0/2624372018' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2024-04-23T10:26:45.135 INFO:journalctl@ceph.mon.a.smithi114.stdout:Apr 23 10:26:45 smithi114 ceph-mon[29255]: from='mgr.15291 172.21.15.114:0/1095789187' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-23T10:26:45.135 INFO:journalctl@ceph.mon.a.smithi114.stdout:Apr 23 10:26:45 smithi114 ceph-mon[29255]: from='mgr.15291 172.21.15.114:0/1095789187' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-23T10:26:45.135 INFO:journalctl@ceph.mon.a.smithi114.stdout:Apr 23 10:26:45 smithi114 ceph-mon[29255]: from='mgr.15291 172.21.15.114:0/1095789187' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-23T10:26:45.135 INFO:journalctl@ceph.mon.a.smithi114.stdout:Apr 23 10:26:45 smithi114 ceph-mon[29255]: from='mgr.15291 172.21.15.114:0/1095789187' entity='mgr.x' 2024-04-23T10:26:45.135 INFO:tasks.quiescer.ceph_manager:waiting for clean 2024-04-23T10:26:45.136 DEBUG:teuthology.orchestra.run.smithi114:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-04-23T10:26:45.239 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:45.238+0000 7f0f7def5640 1 -- 172.21.15.114:0/1749827608 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f0f78106170 msgr2=0x7f0f7810dd40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T10:26:45.239 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:45.238+0000 7f0f7def5640 1 --2- 172.21.15.114:0/1749827608 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f0f78106170 0x7f0f7810dd40 secure :-1 s=READY pgs=444 cs=0 l=1 rev1=1 crypto rx=0x7f0f70009ca0 tx=0x7f0f7002a8c0 comp rx=0 tx=0).stop 2024-04-23T10:26:45.239 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:45.238+0000 7f0f7def5640 1 -- 172.21.15.114:0/1749827608 shutdown_connections 2024-04-23T10:26:45.239 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:45.238+0000 7f0f7def5640 1 --2- 172.21.15.114:0/1749827608 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f0f78106170 0x7f0f7810dd40 unknown :-1 s=CLOSED pgs=444 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:26:45.239 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:45.238+0000 7f0f7def5640 1 --2- 172.21.15.114:0/1749827608 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f0f78101a50 0x7f0f78105a90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:26:45.239 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:45.238+0000 7f0f7def5640 1 --2- 172.21.15.114:0/1749827608 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f0f781010a0 0x7f0f78101480 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:26:45.239 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:45.238+0000 7f0f7def5640 1 -- 172.21.15.114:0/1749827608 >> 172.21.15.114:0/1749827608 conn(0x7f0f78077f50 msgr2=0x7f0f78078360 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T10:26:45.239 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:45.238+0000 7f0f7def5640 1 -- 172.21.15.114:0/1749827608 shutdown_connections 2024-04-23T10:26:45.239 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:45.238+0000 7f0f7def5640 1 -- 172.21.15.114:0/1749827608 wait complete. 2024-04-23T10:26:45.239 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:45.238+0000 7f0f7def5640 1 Processor -- start 2024-04-23T10:26:45.240 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:45.239+0000 7f0f7def5640 1 -- start start 2024-04-23T10:26:45.240 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:45.239+0000 7f0f7def5640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f0f781010a0 0x7f0f78199580 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T10:26:45.240 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:45.239+0000 7f0f7def5640 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f0f78106170 0x7f0f78199ac0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T10:26:45.240 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:45.239+0000 7f0f7def5640 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f0f7819a050 0x7f0f781a4600 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T10:26:45.240 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:45.239+0000 7f0f7def5640 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x7f0f781110c0 con 0x7f0f781010a0 2024-04-23T10:26:45.240 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:45.239+0000 7f0f7def5640 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 -- 0x7f0f78110f40 con 0x7f0f78106170 2024-04-23T10:26:45.240 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:45.239+0000 7f0f7def5640 1 -- --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_getmap magic: 0 -- 0x7f0f78111240 con 0x7f0f7819a050 2024-04-23T10:26:45.240 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:45.239+0000 7f0f777fe640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f0f781010a0 0x7f0f78199580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T10:26:45.240 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:45.239+0000 7f0f777fe640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f0f781010a0 0x7f0f78199580 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.114:40936/0 (socket says 172.21.15.114:40936) 2024-04-23T10:26:45.240 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:45.239+0000 7f0f777fe640 1 -- 172.21.15.114:0/2565221777 learned_addr learned my addr 172.21.15.114:0/2565221777 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-04-23T10:26:45.240 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:45.239+0000 7f0f77fff640 1 --2- 172.21.15.114:0/2565221777 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f0f7819a050 0x7f0f781a4600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T10:26:45.240 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:45.239+0000 7f0f76ffd640 1 --2- 172.21.15.114:0/2565221777 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f0f78106170 0x7f0f78199ac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T10:26:45.240 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:45.239+0000 7f0f777fe640 1 -- 172.21.15.114:0/2565221777 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f0f7819a050 msgr2=0x7f0f781a4600 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T10:26:45.240 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:45.239+0000 7f0f777fe640 1 --2- 172.21.15.114:0/2565221777 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f0f7819a050 0x7f0f781a4600 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:26:45.240 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:45.239+0000 7f0f777fe640 1 -- 172.21.15.114:0/2565221777 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f0f78106170 msgr2=0x7f0f78199ac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T10:26:45.241 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:45.239+0000 7f0f777fe640 1 --2- 172.21.15.114:0/2565221777 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f0f78106170 0x7f0f78199ac0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:26:45.241 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:45.239+0000 7f0f777fe640 1 -- 172.21.15.114:0/2565221777 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f0f781a4dd0 con 0x7f0f781010a0 2024-04-23T10:26:45.241 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:45.239+0000 7f0f76ffd640 1 --2- 172.21.15.114:0/2565221777 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f0f78106170 0x7f0f78199ac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-23T10:26:45.241 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:45.239+0000 7f0f77fff640 1 --2- 172.21.15.114:0/2565221777 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f0f7819a050 0x7f0f781a4600 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-23T10:26:45.241 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:45.239+0000 7f0f777fe640 1 --2- 172.21.15.114:0/2565221777 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f0f781010a0 0x7f0f78199580 secure :-1 s=READY pgs=445 cs=0 l=1 rev1=1 crypto rx=0x7f0f6800d950 tx=0x7f0f6800de10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T10:26:45.241 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:45.239+0000 7f0f74ff9640 1 -- 172.21.15.114:0/2565221777 <== mon.0 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0f68017070 con 0x7f0f781010a0 2024-04-23T10:26:45.241 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:45.240+0000 7f0f7def5640 1 -- 172.21.15.114:0/2565221777 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f0f781a50c0 con 0x7f0f781010a0 2024-04-23T10:26:45.241 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:45.240+0000 7f0f74ff9640 1 -- 172.21.15.114:0/2565221777 <== mon.0 v2:172.21.15.114:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f0f68009c30 con 0x7f0f781010a0 2024-04-23T10:26:45.241 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:45.240+0000 7f0f74ff9640 1 -- 172.21.15.114:0/2565221777 <== mon.0 v2:172.21.15.114:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0f6800f620 con 0x7f0f781010a0 2024-04-23T10:26:45.241 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:45.240+0000 7f0f7def5640 1 -- 172.21.15.114:0/2565221777 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f0f781a5660 con 0x7f0f781010a0 2024-04-23T10:26:45.242 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:45.241+0000 7f0f7def5640 1 -- 172.21.15.114:0/2565221777 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f0f78068d70 con 0x7f0f781010a0 2024-04-23T10:26:45.242 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:45.241+0000 7f0f74ff9640 1 -- 172.21.15.114:0/2565221777 <== mon.0 v2:172.21.15.114:3300/0 4 ==== mgrmap(e 24) ==== 138823+0+0 (secure 0 0 0) 0x7f0f68010460 con 0x7f0f781010a0 2024-04-23T10:26:45.242 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:45.242+0000 7f0f74ff9640 1 --2- 172.21.15.114:0/2565221777 >> [v2:172.21.15.114:6800/1395693466,v1:172.21.15.114:6801/1395693466] conn(0x7f0f600a3db0 0x7f0f600a6270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T10:26:45.242 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:45.242+0000 7f0f76ffd640 1 --2- 172.21.15.114:0/2565221777 >> [v2:172.21.15.114:6800/1395693466,v1:172.21.15.114:6801/1395693466] conn(0x7f0f600a3db0 0x7f0f600a6270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T10:26:45.242 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:45.242+0000 7f0f74ff9640 1 -- 172.21.15.114:0/2565221777 <== mon.0 v2:172.21.15.114:3300/0 5 ==== osd_map(90..90 src has 1..90) ==== 8527+0+0 (secure 0 0 0) 0x7f0f680d06c0 con 0x7f0f781010a0 2024-04-23T10:26:45.242 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:45.242+0000 7f0f76ffd640 1 --2- 172.21.15.114:0/2565221777 >> [v2:172.21.15.114:6800/1395693466,v1:172.21.15.114:6801/1395693466] conn(0x7f0f600a3db0 0x7f0f600a6270 secure :-1 s=READY pgs=48 cs=0 l=1 rev1=1 crypto rx=0x7f0f78069c30 tx=0x7f0f6c005f90 comp rx=0 tx=0).ready entity=mgr.15291 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T10:26:45.250 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:45.251+0000 7f0f74ff9640 1 -- 172.21.15.114:0/2565221777 <== mon.0 v2:172.21.15.114:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f0f6809cbf0 con 0x7f0f781010a0 2024-04-23T10:26:45.328 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 23 10:26:45 smithi204 ceph-mon[32649]: Adjusting osd_memory_target on smithi119 to 176.2M 2024-04-23T10:26:45.328 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 23 10:26:45 smithi204 ceph-mon[32649]: Unable to set osd_memory_target on smithi119 to 184770252: error parsing value: Value '184770252' is below minimum 939524096 2024-04-23T10:26:45.328 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 23 10:26:45 smithi204 ceph-mon[32649]: Adjusting osd_memory_target on smithi204 to 176.2M 2024-04-23T10:26:45.328 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 23 10:26:45 smithi204 ceph-mon[32649]: Unable to set osd_memory_target on smithi204 to 184759500: error parsing value: Value '184759500' is below minimum 939524096 2024-04-23T10:26:45.328 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 23 10:26:45 smithi204 ceph-mon[32649]: Adjusting osd_memory_target on smithi114 to 176.2M 2024-04-23T10:26:45.328 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 23 10:26:45 smithi204 ceph-mon[32649]: Unable to set osd_memory_target on smithi114 to 184769536: error parsing value: Value '184769536' is below minimum 939524096 2024-04-23T10:26:45.328 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 23 10:26:45 smithi204 ceph-mon[32649]: Updating smithi114:/etc/ceph/ceph.conf 2024-04-23T10:26:45.328 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 23 10:26:45 smithi204 ceph-mon[32649]: Updating smithi119:/etc/ceph/ceph.conf 2024-04-23T10:26:45.328 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 23 10:26:45 smithi204 ceph-mon[32649]: Updating smithi204:/etc/ceph/ceph.conf 2024-04-23T10:26:45.328 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 23 10:26:45 smithi204 ceph-mon[32649]: Updating smithi204:/var/lib/ceph/aa240572-0159-11ef-bc93-c7b262605968/config/ceph.conf 2024-04-23T10:26:45.328 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 23 10:26:45 smithi204 ceph-mon[32649]: Updating smithi114:/var/lib/ceph/aa240572-0159-11ef-bc93-c7b262605968/config/ceph.conf 2024-04-23T10:26:45.328 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 23 10:26:45 smithi204 ceph-mon[32649]: Updating smithi119:/var/lib/ceph/aa240572-0159-11ef-bc93-c7b262605968/config/ceph.conf 2024-04-23T10:26:45.329 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 23 10:26:45 smithi204 ceph-mon[32649]: from='client.? 172.21.15.114:0/2609718743' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-23T10:26:45.329 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 23 10:26:45 smithi204 ceph-mon[32649]: from='client.? 172.21.15.114:0/2324190627' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-04-23T10:26:45.329 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 23 10:26:45 smithi204 ceph-mon[32649]: from='mgr.15291 172.21.15.114:0/1095789187' entity='mgr.x' 2024-04-23T10:26:45.329 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 23 10:26:45 smithi204 ceph-mon[32649]: from='mgr.15291 172.21.15.114:0/1095789187' entity='mgr.x' 2024-04-23T10:26:45.329 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 23 10:26:45 smithi204 ceph-mon[32649]: from='mgr.15291 172.21.15.114:0/1095789187' entity='mgr.x' 2024-04-23T10:26:45.329 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 23 10:26:45 smithi204 ceph-mon[32649]: from='mgr.15291 172.21.15.114:0/1095789187' entity='mgr.x' 2024-04-23T10:26:45.329 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 23 10:26:45 smithi204 ceph-mon[32649]: from='mgr.15291 172.21.15.114:0/1095789187' entity='mgr.x' 2024-04-23T10:26:45.329 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 23 10:26:45 smithi204 ceph-mon[32649]: from='mgr.15291 172.21.15.114:0/1095789187' entity='mgr.x' 2024-04-23T10:26:45.329 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 23 10:26:45 smithi204 ceph-mon[32649]: from='mgr.15291 172.21.15.114:0/1095789187' entity='mgr.x' 2024-04-23T10:26:45.329 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 23 10:26:45 smithi204 ceph-mon[32649]: from='client.? 172.21.15.114:0/2624372018' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2024-04-23T10:26:45.329 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 23 10:26:45 smithi204 ceph-mon[32649]: from='mgr.15291 172.21.15.114:0/1095789187' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-23T10:26:45.329 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 23 10:26:45 smithi204 ceph-mon[32649]: from='mgr.15291 172.21.15.114:0/1095789187' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-23T10:26:45.329 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 23 10:26:45 smithi204 ceph-mon[32649]: from='mgr.15291 172.21.15.114:0/1095789187' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-23T10:26:45.329 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 23 10:26:45 smithi204 ceph-mon[32649]: from='mgr.15291 172.21.15.114:0/1095789187' entity='mgr.x' 2024-04-23T10:26:45.432 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:45.433+0000 7f0f7def5640 1 -- 172.21.15.114:0/2565221777 --> [v2:172.21.15.114:6800/1395693466,v1:172.21.15.114:6801/1395693466] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) -- 0x7f0f78101a50 con 0x7f0f600a3db0 2024-04-23T10:26:45.439 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:45.441+0000 7f0f74ff9640 1 -- 172.21.15.114:0/2565221777 <== mgr.15291 v2:172.21.15.114:6800/1395693466 1 ==== mgr_command_reply(tid 0: 0 dumped all) ==== 18+0+257730 (secure 0 0 0) 0x7f0f78101a50 con 0x7f0f600a3db0 2024-04-23T10:26:45.442 INFO:teuthology.orchestra.run.smithi114.stdout: 2024-04-23T10:26:45.444 INFO:teuthology.orchestra.run.smithi114.stderr:dumped all 2024-04-23T10:26:45.445 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:45.444+0000 7f0f7def5640 1 -- 172.21.15.114:0/2565221777 >> [v2:172.21.15.114:6800/1395693466,v1:172.21.15.114:6801/1395693466] conn(0x7f0f600a3db0 msgr2=0x7f0f600a6270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T10:26:45.445 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:45.444+0000 7f0f7def5640 1 --2- 172.21.15.114:0/2565221777 >> [v2:172.21.15.114:6800/1395693466,v1:172.21.15.114:6801/1395693466] conn(0x7f0f600a3db0 0x7f0f600a6270 secure :-1 s=READY pgs=48 cs=0 l=1 rev1=1 crypto rx=0x7f0f78069c30 tx=0x7f0f6c005f90 comp rx=0 tx=0).stop 2024-04-23T10:26:45.446 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:45.444+0000 7f0f7def5640 1 -- 172.21.15.114:0/2565221777 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f0f781010a0 msgr2=0x7f0f78199580 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T10:26:45.446 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:45.444+0000 7f0f7def5640 1 --2- 172.21.15.114:0/2565221777 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f0f781010a0 0x7f0f78199580 secure :-1 s=READY pgs=445 cs=0 l=1 rev1=1 crypto rx=0x7f0f6800d950 tx=0x7f0f6800de10 comp rx=0 tx=0).stop 2024-04-23T10:26:45.446 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:45.444+0000 7f0f7def5640 1 -- 172.21.15.114:0/2565221777 shutdown_connections 2024-04-23T10:26:45.446 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:45.444+0000 7f0f7def5640 1 --2- 172.21.15.114:0/2565221777 >> [v2:172.21.15.114:6800/1395693466,v1:172.21.15.114:6801/1395693466] conn(0x7f0f600a3db0 0x7f0f600a6270 unknown :-1 s=CLOSED pgs=48 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:26:45.446 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:45.444+0000 7f0f7def5640 1 --2- 172.21.15.114:0/2565221777 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f0f7819a050 0x7f0f781a4600 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:26:45.446 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:45.444+0000 7f0f7def5640 1 --2- 172.21.15.114:0/2565221777 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f0f78106170 0x7f0f78199ac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:26:45.446 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:45.444+0000 7f0f7def5640 1 --2- 172.21.15.114:0/2565221777 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f0f781010a0 0x7f0f78199580 unknown :-1 s=CLOSED pgs=445 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:26:45.446 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:45.444+0000 7f0f7def5640 1 -- 172.21.15.114:0/2565221777 >> 172.21.15.114:0/2565221777 conn(0x7f0f78077f50 msgr2=0x7f0f780fef30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T10:26:45.446 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:45.444+0000 7f0f7def5640 1 -- 172.21.15.114:0/2565221777 shutdown_connections 2024-04-23T10:26:45.446 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:45.444+0000 7f0f7def5640 1 -- 172.21.15.114:0/2565221777 wait complete. 2024-04-23T10:26:45.454 INFO:teuthology.orchestra.run.smithi114.stdout:{"pg_ready":true,"pg_map":{"version":5,"stamp":"2024-04-23T10:26:44.290805+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":1188632,"num_objects":66,"num_object_clones":0,"num_object_copies":198,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":66,"num_whiteouts":0,"num_read":938,"num_read_kb":629,"num_write":468,"num_write_kb":4377,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":7,"num_bytes_recovered":1180736,"num_keys_recovered":10,"num_objects_omap":40,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":865,"ondisk_log_size":865,"up":291,"acting":291,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":291,"num_osds":12,"num_per_pool_osds":12,"num_per_pool_omap_osds":12,"kb":1124941824,"kb_used":384428,"kb_used_data":14440,"kb_used_omap":18,"kb_used_meta":369837,"kb_avail":1124557396,"statfs":{"total":1151940427776,"available":1151546773504,"internally_reserved":0,"allocated":14786560,"data_stored":8381446,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":19142,"internal_metadata":378713402},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"2.989676"},"pg_stats":[{"pgid":"3.3f","version":"89'8","reported_seq":27,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-23T10:26:40.460272+0000","last_change":"2024-04-23T10:24:44.051465+0000","last_active":"2024-04-23T10:26:40.460272+0000","last_peered":"2024-04-23T10:26:40.460272+0000","last_clean":"2024-04-23T10:26:40.460272+0000","last_became_active":"2024-04-23T10:24:44.050863+0000","last_became_peered":"2024-04-23T10:24:44.050863+0000","last_unstale":"2024-04-23T10:26:40.460272+0000","last_undegraded":"2024-04-23T10:26:40.460272+0000","last_fullsized":"2024-04-23T10:26:40.460272+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T10:24:43.025324+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T10:24:43.025324+0000","last_clean_scrub_stamp":"2024-04-23T10:24:43.025324+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T15:11:12.508458+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,1,0],"acting":[11,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.3e","version":"89'4","reported_seq":23,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-23T10:26:40.459824+0000","last_change":"2024-04-23T10:24:44.050739+0000","last_active":"2024-04-23T10:26:40.459824+0000","last_peered":"2024-04-23T10:26:40.459824+0000","last_clean":"2024-04-23T10:26:40.459824+0000","last_became_active":"2024-04-23T10:24:44.050430+0000","last_became_peered":"2024-04-23T10:24:44.050430+0000","last_unstale":"2024-04-23T10:26:40.459824+0000","last_undegraded":"2024-04-23T10:26:40.459824+0000","last_fullsized":"2024-04-23T10:26:40.459824+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T10:24:43.025324+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T10:24:43.025324+0000","last_clean_scrub_stamp":"2024-04-23T10:24:43.025324+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T17:36:25.983147+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,11,10],"acting":[0,11,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3d","version":"89'7","reported_seq":26,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-23T10:26:40.460358+0000","last_change":"2024-04-23T10:24:44.054805+0000","last_active":"2024-04-23T10:26:40.460358+0000","last_peered":"2024-04-23T10:26:40.460358+0000","last_clean":"2024-04-23T10:26:40.460358+0000","last_became_active":"2024-04-23T10:24:44.054423+0000","last_became_peered":"2024-04-23T10:24:44.054423+0000","last_unstale":"2024-04-23T10:26:40.460358+0000","last_undegraded":"2024-04-23T10:26:40.460358+0000","last_fullsized":"2024-04-23T10:26:40.460358+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T10:24:43.025324+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T10:24:43.025324+0000","last_clean_scrub_stamp":"2024-04-23T10:24:43.025324+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T21:34:26.793624+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,1],"acting":[0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3c","version":"89'8","reported_seq":27,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-23T10:26:40.459685+0000","last_change":"2024-04-23T10:24:44.055966+0000","last_active":"2024-04-23T10:26:40.459685+0000","last_peered":"2024-04-23T10:26:40.459685+0000","last_clean":"2024-04-23T10:26:40.459685+0000","last_became_active":"2024-04-23T10:24:44.055028+0000","last_became_peered":"2024-04-23T10:24:44.055028+0000","last_unstale":"2024-04-23T10:26:40.459685+0000","last_undegraded":"2024-04-23T10:26:40.459685+0000","last_fullsized":"2024-04-23T10:26:40.459685+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T10:24:43.025324+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T10:24:43.025324+0000","last_clean_scrub_stamp":"2024-04-23T10:24:43.025324+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T19:41:04.594025+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,8,2],"acting":[4,8,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.3b","version":"89'4","reported_seq":23,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-23T10:26:40.373641+0000","last_change":"2024-04-23T10:24:44.046954+0000","last_active":"2024-04-23T10:26:40.373641+0000","last_peered":"2024-04-23T10:26:40.373641+0000","last_clean":"2024-04-23T10:26:40.373641+0000","last_became_active":"2024-04-23T10:24:44.046631+0000","last_became_peered":"2024-04-23T10:24:44.046631+0000","last_unstale":"2024-04-23T10:26:40.373641+0000","last_undegraded":"2024-04-23T10:26:40.373641+0000","last_fullsized":"2024-04-23T10:26:40.373641+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T10:24:43.025324+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T10:24:43.025324+0000","last_clean_scrub_stamp":"2024-04-23T10:24:43.025324+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T15:04:26.878894+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,3,6],"acting":[8,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.3a","version":"89'6","reported_seq":25,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-23T10:26:40.381329+0000","last_change":"2024-04-23T10:24:44.051550+0000","last_active":"2024-04-23T10:26:40.381329+0000","last_peered":"2024-04-23T10:26:40.381329+0000","last_clean":"2024-04-23T10:26:40.381329+0000","last_became_active":"2024-04-23T10:24:44.050469+0000","last_became_peered":"2024-04-23T10:24:44.050469+0000","last_unstale":"2024-04-23T10:26:40.381329+0000","last_undegraded":"2024-04-23T10:26:40.381329+0000","last_fullsized":"2024-04-23T10:26:40.381329+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T10:24:43.025324+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T10:24:43.025324+0000","last_clean_scrub_stamp":"2024-04-23T10:24:43.025324+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T21:13:42.469252+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,9,4],"acting":[10,9,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.39","version":"89'5","reported_seq":30,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-23T10:26:40.521664+0000","last_change":"2024-04-23T10:25:47.588865+0000","last_active":"2024-04-23T10:26:40.521664+0000","last_peered":"2024-04-23T10:26:40.521664+0000","last_clean":"2024-04-23T10:26:40.521664+0000","last_became_active":"2024-04-23T10:25:47.588571+0000","last_became_peered":"2024-04-23T10:25:47.588571+0000","last_unstale":"2024-04-23T10:26:40.521664+0000","last_undegraded":"2024-04-23T10:26:40.521664+0000","last_fullsized":"2024-04-23T10:26:40.521664+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T10:24:43.025324+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T10:24:43.025324+0000","last_clean_scrub_stamp":"2024-04-23T10:24:43.025324+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T17:14:00.026999+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,9,8],"acting":[7,9,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.38","version":"89'9","reported_seq":22,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-23T10:26:40.371622+0000","last_change":"2024-04-23T10:25:47.722416+0000","last_active":"2024-04-23T10:26:40.371622+0000","last_peered":"2024-04-23T10:26:40.371622+0000","last_clean":"2024-04-23T10:26:40.371622+0000","last_became_active":"2024-04-23T10:25:47.722015+0000","last_became_peered":"2024-04-23T10:25:47.722015+0000","last_unstale":"2024-04-23T10:26:40.371622+0000","last_undegraded":"2024-04-23T10:26:40.371622+0000","last_fullsized":"2024-04-23T10:26:40.371622+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T10:24:43.025324+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T10:24:43.025324+0000","last_clean_scrub_stamp":"2024-04-23T10:24:43.025324+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T17:02:57.881164+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,5,10],"acting":[9,5,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.37","version":"89'11","reported_seq":30,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-23T10:26:40.281426+0000","last_change":"2024-04-23T10:24:44.055261+0000","last_active":"2024-04-23T10:26:40.281426+0000","last_peered":"2024-04-23T10:26:40.281426+0000","last_clean":"2024-04-23T10:26:40.281426+0000","last_became_active":"2024-04-23T10:24:44.054405+0000","last_became_peered":"2024-04-23T10:24:44.054405+0000","last_unstale":"2024-04-23T10:26:40.281426+0000","last_undegraded":"2024-04-23T10:26:40.281426+0000","last_fullsized":"2024-04-23T10:26:40.281426+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T10:24:43.025324+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T10:24:43.025324+0000","last_clean_scrub_stamp":"2024-04-23T10:24:43.025324+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T19:30:30.306739+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,11],"acting":[1,5,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.36","version":"89'16","reported_seq":35,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-23T10:26:40.381657+0000","last_change":"2024-04-23T10:24:44.055854+0000","last_active":"2024-04-23T10:26:40.381657+0000","last_peered":"2024-04-23T10:26:40.381657+0000","last_clean":"2024-04-23T10:26:40.381657+0000","last_became_active":"2024-04-23T10:24:44.055275+0000","last_became_peered":"2024-04-23T10:24:44.055275+0000","last_unstale":"2024-04-23T10:26:40.381657+0000","last_undegraded":"2024-04-23T10:26:40.381657+0000","last_fullsized":"2024-04-23T10:26:40.381657+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T10:24:43.025324+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T10:24:43.025324+0000","last_clean_scrub_stamp":"2024-04-23T10:24:43.025324+0000","objects_scrubbed":0,"log_size":16,"log_dups_size":0,"ondisk_log_size":16,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T21:48:25.934878+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,2,4],"acting":[10,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.35","version":"89'6","reported_seq":25,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-23T10:26:40.522323+0000","last_change":"2024-04-23T10:24:44.061314+0000","last_active":"2024-04-23T10:26:40.522323+0000","last_peered":"2024-04-23T10:26:40.522323+0000","last_clean":"2024-04-23T10:26:40.522323+0000","last_became_active":"2024-04-23T10:24:44.060981+0000","last_became_peered":"2024-04-23T10:24:44.060981+0000","last_unstale":"2024-04-23T10:26:40.522323+0000","last_undegraded":"2024-04-23T10:26:40.522323+0000","last_fullsized":"2024-04-23T10:26:40.522323+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T10:24:43.025324+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T10:24:43.025324+0000","last_clean_scrub_stamp":"2024-04-23T10:24:43.025324+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T11:48:08.387575+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,8,1],"acting":[7,8,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.34","version":"89'7","reported_seq":26,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-23T10:26:40.459648+0000","last_change":"2024-04-23T10:24:44.055335+0000","last_active":"2024-04-23T10:26:40.459648+0000","last_peered":"2024-04-23T10:26:40.459648+0000","last_clean":"2024-04-23T10:26:40.459648+0000","last_became_active":"2024-04-23T10:24:44.051257+0000","last_became_peered":"2024-04-23T10:24:44.051257+0000","last_unstale":"2024-04-23T10:26:40.459648+0000","last_undegraded":"2024-04-23T10:26:40.459648+0000","last_fullsized":"2024-04-23T10:26:40.459648+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T10:24:43.025324+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T10:24:43.025324+0000","last_clean_scrub_stamp":"2024-04-23T10:24:43.025324+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T11:57:45.999751+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,10,8],"acting":[4,10,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.33","version":"89'10","reported_seq":29,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-23T10:26:40.278008+0000","last_change":"2024-04-23T10:24:44.052078+0000","last_active":"2024-04-23T10:26:40.278008+0000","last_peered":"2024-04-23T10:26:40.278008+0000","last_clean":"2024-04-23T10:26:40.278008+0000","last_became_active":"2024-04-23T10:24:44.051747+0000","last_became_peered":"2024-04-23T10:24:44.051747+0000","last_unstale":"2024-04-23T10:26:40.278008+0000","last_undegraded":"2024-04-23T10:26:40.278008+0000","last_fullsized":"2024-04-23T10:26:40.278008+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T10:24:43.025324+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T10:24:43.025324+0000","last_clean_scrub_stamp":"2024-04-23T10:24:43.025324+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T19:14:16.495967+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,10,5],"acting":[6,10,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.32","version":"89'7","reported_seq":26,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-23T10:26:40.459810+0000","last_change":"2024-04-23T10:24:44.051994+0000","last_active":"2024-04-23T10:26:40.459810+0000","last_peered":"2024-04-23T10:26:40.459810+0000","last_clean":"2024-04-23T10:26:40.459810+0000","last_became_active":"2024-04-23T10:24:44.051638+0000","last_became_peered":"2024-04-23T10:24:44.051638+0000","last_unstale":"2024-04-23T10:26:40.459810+0000","last_undegraded":"2024-04-23T10:26:40.459810+0000","last_fullsized":"2024-04-23T10:26:40.459810+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T10:24:43.025324+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T10:24:43.025324+0000","last_clean_scrub_stamp":"2024-04-23T10:24:43.025324+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T14:28:30.447198+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,8,10],"acting":[0,8,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.31","version":"89'7","reported_seq":32,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-23T10:26:40.459693+0000","last_change":"2024-04-23T10:25:47.589538+0000","last_active":"2024-04-23T10:26:40.459693+0000","last_peered":"2024-04-23T10:26:40.459693+0000","last_clean":"2024-04-23T10:26:40.459693+0000","last_became_active":"2024-04-23T10:25:47.589172+0000","last_became_peered":"2024-04-23T10:25:47.589172+0000","last_unstale":"2024-04-23T10:26:40.459693+0000","last_undegraded":"2024-04-23T10:26:40.459693+0000","last_fullsized":"2024-04-23T10:26:40.459693+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T10:24:43.025324+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T10:24:43.025324+0000","last_clean_scrub_stamp":"2024-04-23T10:24:43.025324+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T19:47:34.267992+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,8,2],"acting":[0,8,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.30","version":"89'9","reported_seq":28,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-23T10:26:40.280931+0000","last_change":"2024-04-23T10:24:44.057154+0000","last_active":"2024-04-23T10:26:40.280931+0000","last_peered":"2024-04-23T10:26:40.280931+0000","last_clean":"2024-04-23T10:26:40.280931+0000","last_became_active":"2024-04-23T10:24:44.056836+0000","last_became_peered":"2024-04-23T10:24:44.056836+0000","last_unstale":"2024-04-23T10:26:40.280931+0000","last_undegraded":"2024-04-23T10:26:40.280931+0000","last_fullsized":"2024-04-23T10:26:40.280931+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T10:24:43.025324+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T10:24:43.025324+0000","last_clean_scrub_stamp":"2024-04-23T10:24:43.025324+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T13:48:59.107635+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,0],"acting":[1,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.2f","version":"89'8","reported_seq":27,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-23T10:26:40.460422+0000","last_change":"2024-04-23T10:24:44.051175+0000","last_active":"2024-04-23T10:26:40.460422+0000","last_peered":"2024-04-23T10:26:40.460422+0000","last_clean":"2024-04-23T10:26:40.460422+0000","last_became_active":"2024-04-23T10:24:44.050402+0000","last_became_peered":"2024-04-23T10:24:44.050402+0000","last_unstale":"2024-04-23T10:26:40.460422+0000","last_undegraded":"2024-04-23T10:26:40.460422+0000","last_fullsized":"2024-04-23T10:26:40.460422+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T10:24:43.025324+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T10:24:43.025324+0000","last_clean_scrub_stamp":"2024-04-23T10:24:43.025324+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T15:01:26.919701+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,1,0],"acting":[11,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.2e","version":"89'10","reported_seq":29,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-23T10:26:40.371997+0000","last_change":"2024-04-23T10:24:44.053390+0000","last_active":"2024-04-23T10:26:40.371997+0000","last_peered":"2024-04-23T10:26:40.371997+0000","last_clean":"2024-04-23T10:26:40.371997+0000","last_became_active":"2024-04-23T10:24:44.052970+0000","last_became_peered":"2024-04-23T10:24:44.052970+0000","last_unstale":"2024-04-23T10:26:40.371997+0000","last_undegraded":"2024-04-23T10:26:40.371997+0000","last_fullsized":"2024-04-23T10:26:40.371997+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T10:24:43.025324+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T10:24:43.025324+0000","last_clean_scrub_stamp":"2024-04-23T10:24:43.025324+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T13:27:55.704328+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,1,10],"acting":[9,1,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.2d","version":"89'7","reported_seq":26,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-23T10:26:40.460284+0000","last_change":"2024-04-23T10:24:44.049307+0000","last_active":"2024-04-23T10:26:40.460284+0000","last_peered":"2024-04-23T10:26:40.460284+0000","last_clean":"2024-04-23T10:26:40.460284+0000","last_became_active":"2024-04-23T10:24:44.048981+0000","last_became_peered":"2024-04-23T10:24:44.048981+0000","last_unstale":"2024-04-23T10:26:40.460284+0000","last_undegraded":"2024-04-23T10:26:40.460284+0000","last_fullsized":"2024-04-23T10:26:40.460284+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T10:24:43.025324+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T10:24:43.025324+0000","last_clean_scrub_stamp":"2024-04-23T10:24:43.025324+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T12:42:21.804591+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,6],"acting":[2,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.1f","version":"87'9","reported_seq":31,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-23T10:26:40.371868+0000","last_change":"2024-04-23T10:24:43.045804+0000","last_active":"2024-04-23T10:26:40.371868+0000","last_peered":"2024-04-23T10:26:40.371868+0000","last_clean":"2024-04-23T10:26:40.371868+0000","last_became_active":"2024-04-23T10:24:43.045135+0000","last_became_peered":"2024-04-23T10:24:43.045135+0000","last_unstale":"2024-04-23T10:26:40.371868+0000","last_undegraded":"2024-04-23T10:26:40.371868+0000","last_fullsized":"2024-04-23T10:26:40.371868+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T10:24:42.023127+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T10:24:42.023127+0000","last_clean_scrub_stamp":"2024-04-23T10:24:42.023127+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T20:27:28.421644+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,7,8],"acting":[9,7,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.1e","version":"89'5","reported_seq":24,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-23T10:26:40.374223+0000","last_change":"2024-04-23T10:24:44.055008+0000","last_active":"2024-04-23T10:26:40.374223+0000","last_peered":"2024-04-23T10:26:40.374223+0000","last_clean":"2024-04-23T10:26:40.374223+0000","last_became_active":"2024-04-23T10:24:44.054682+0000","last_became_peered":"2024-04-23T10:24:44.054682+0000","last_unstale":"2024-04-23T10:26:40.374223+0000","last_undegraded":"2024-04-23T10:26:40.374223+0000","last_fullsized":"2024-04-23T10:26:40.374223+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T10:24:43.025324+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T10:24:43.025324+0000","last_clean_scrub_stamp":"2024-04-23T10:24:43.025324+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T15:49:58.394958+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,5],"acting":[3,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.1e","version":"87'2","reported_seq":23,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-23T10:26:40.374327+0000","last_change":"2024-04-23T10:24:43.052303+0000","last_active":"2024-04-23T10:26:40.374327+0000","last_peered":"2024-04-23T10:26:40.374327+0000","last_clean":"2024-04-23T10:26:40.374327+0000","last_became_active":"2024-04-23T10:24:43.052010+0000","last_became_peered":"2024-04-23T10:24:43.052010+0000","last_unstale":"2024-04-23T10:26:40.374327+0000","last_undegraded":"2024-04-23T10:26:40.374327+0000","last_fullsized":"2024-04-23T10:26:40.374327+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T10:24:42.023127+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T10:24:42.023127+0000","last_clean_scrub_stamp":"2024-04-23T10:24:42.023127+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T13:03:15.440610+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,5],"acting":[3,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1f","version":"89'6","reported_seq":25,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-23T10:26:40.460053+0000","last_change":"2024-04-23T10:24:44.050749+0000","last_active":"2024-04-23T10:26:40.460053+0000","last_peered":"2024-04-23T10:26:40.460053+0000","last_clean":"2024-04-23T10:26:40.460053+0000","last_became_active":"2024-04-23T10:24:44.049780+0000","last_became_peered":"2024-04-23T10:24:44.049780+0000","last_unstale":"2024-04-23T10:26:40.460053+0000","last_undegraded":"2024-04-23T10:26:40.460053+0000","last_fullsized":"2024-04-23T10:26:40.460053+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T10:24:43.025324+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T10:24:43.025324+0000","last_clean_scrub_stamp":"2024-04-23T10:24:43.025324+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T16:32:37.963776+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,9],"acting":[0,1,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1d","version":"87'4","reported_seq":19,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-23T10:26:40.460147+0000","last_change":"2024-04-23T10:25:47.591939+0000","last_active":"2024-04-23T10:26:40.460147+0000","last_peered":"2024-04-23T10:26:40.460147+0000","last_clean":"2024-04-23T10:26:40.460147+0000","last_became_active":"2024-04-23T10:25:47.586626+0000","last_became_peered":"2024-04-23T10:25:47.586626+0000","last_unstale":"2024-04-23T10:26:40.460147+0000","last_undegraded":"2024-04-23T10:26:40.460147+0000","last_fullsized":"2024-04-23T10:26:40.460147+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T10:24:42.023127+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T10:24:42.023127+0000","last_clean_scrub_stamp":"2024-04-23T10:24:42.023127+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T20:55:38.216483+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":10,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,2,7],"acting":[11,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.1c","version":"89'9","reported_seq":28,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-23T10:26:40.281020+0000","last_change":"2024-04-23T10:24:44.051606+0000","last_active":"2024-04-23T10:26:40.281020+0000","last_peered":"2024-04-23T10:26:40.281020+0000","last_clean":"2024-04-23T10:26:40.281020+0000","last_became_active":"2024-04-23T10:24:44.051191+0000","last_became_peered":"2024-04-23T10:24:44.051191+0000","last_unstale":"2024-04-23T10:26:40.281020+0000","last_undegraded":"2024-04-23T10:26:40.281020+0000","last_fullsized":"2024-04-23T10:26:40.281020+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T10:24:43.025324+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T10:24:43.025324+0000","last_clean_scrub_stamp":"2024-04-23T10:24:43.025324+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T16:21:51.706066+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,10,6],"acting":[1,10,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1c","version":"86'2","reported_seq":23,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-23T10:26:40.373937+0000","last_change":"2024-04-23T10:24:43.048594+0000","last_active":"2024-04-23T10:26:40.373937+0000","last_peered":"2024-04-23T10:26:40.373937+0000","last_clean":"2024-04-23T10:26:40.373937+0000","last_became_active":"2024-04-23T10:24:43.048276+0000","last_became_peered":"2024-04-23T10:24:43.048276+0000","last_unstale":"2024-04-23T10:26:40.373937+0000","last_undegraded":"2024-04-23T10:26:40.373937+0000","last_fullsized":"2024-04-23T10:26:40.373937+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T10:24:42.023127+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T10:24:42.023127+0000","last_clean_scrub_stamp":"2024-04-23T10:24:42.023127+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T12:20:40.196674+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,1,5],"acting":[8,1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.1d","version":"89'15","reported_seq":34,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-23T10:26:40.282049+0000","last_change":"2024-04-23T10:24:44.050970+0000","last_active":"2024-04-23T10:26:40.282049+0000","last_peered":"2024-04-23T10:26:40.282049+0000","last_clean":"2024-04-23T10:26:40.282049+0000","last_became_active":"2024-04-23T10:24:44.050640+0000","last_became_peered":"2024-04-23T10:24:44.050640+0000","last_unstale":"2024-04-23T10:26:40.282049+0000","last_undegraded":"2024-04-23T10:26:40.282049+0000","last_fullsized":"2024-04-23T10:26:40.282049+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T10:24:43.025324+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T10:24:43.025324+0000","last_clean_scrub_stamp":"2024-04-23T10:24:43.025324+0000","objects_scrubbed":0,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T21:03:24.570290+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,1],"acting":[5,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.1b","version":"90'8","reported_seq":146,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-23T10:26:40.486313+0000","last_change":"2024-04-23T10:24:43.051742+0000","last_active":"2024-04-23T10:26:40.486313+0000","last_peered":"2024-04-23T10:26:40.486313+0000","last_clean":"2024-04-23T10:26:40.486313+0000","last_became_active":"2024-04-23T10:24:43.051420+0000","last_became_peered":"2024-04-23T10:24:43.051420+0000","last_unstale":"2024-04-23T10:26:40.486313+0000","last_undegraded":"2024-04-23T10:26:40.486313+0000","last_fullsized":"2024-04-23T10:26:40.486313+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T10:24:42.023127+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T10:24:42.023127+0000","last_clean_scrub_stamp":"2024-04-23T10:24:42.023127+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T17:54:37.182731+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1545,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":117,"num_read_kb":4,"num_write":5,"num_write_kb":5,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,7],"acting":[5,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.1a","version":"89'7","reported_seq":26,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-23T10:26:40.373718+0000","last_change":"2024-04-23T10:24:44.049314+0000","last_active":"2024-04-23T10:26:40.373718+0000","last_peered":"2024-04-23T10:26:40.373718+0000","last_clean":"2024-04-23T10:26:40.373718+0000","last_became_active":"2024-04-23T10:24:44.048352+0000","last_became_peered":"2024-04-23T10:24:44.048352+0000","last_unstale":"2024-04-23T10:26:40.373718+0000","last_undegraded":"2024-04-23T10:26:40.373718+0000","last_fullsized":"2024-04-23T10:26:40.373718+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T10:24:43.025324+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T10:24:43.025324+0000","last_clean_scrub_stamp":"2024-04-23T10:24:43.025324+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T16:57:29.957716+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,9,5],"acting":[8,9,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"2.1a","version":"87'3","reported_seq":24,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-23T10:26:40.381286+0000","last_change":"2024-04-23T10:24:43.051362+0000","last_active":"2024-04-23T10:26:40.381286+0000","last_peered":"2024-04-23T10:26:40.381286+0000","last_clean":"2024-04-23T10:26:40.381286+0000","last_became_active":"2024-04-23T10:24:43.051045+0000","last_became_peered":"2024-04-23T10:24:43.051045+0000","last_unstale":"2024-04-23T10:26:40.381286+0000","last_undegraded":"2024-04-23T10:26:40.381286+0000","last_fullsized":"2024-04-23T10:26:40.381286+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T10:24:42.023127+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T10:24:42.023127+0000","last_clean_scrub_stamp":"2024-04-23T10:24:42.023127+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T15:40:53.618281+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,8,6],"acting":[10,8,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.1b","version":"89'8","reported_seq":27,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-23T10:26:40.281974+0000","last_change":"2024-04-23T10:24:44.040323+0000","last_active":"2024-04-23T10:26:40.281974+0000","last_peered":"2024-04-23T10:26:40.281974+0000","last_clean":"2024-04-23T10:26:40.281974+0000","last_became_active":"2024-04-23T10:24:44.039995+0000","last_became_peered":"2024-04-23T10:24:44.039995+0000","last_unstale":"2024-04-23T10:26:40.281974+0000","last_undegraded":"2024-04-23T10:26:40.281974+0000","last_fullsized":"2024-04-23T10:26:40.281974+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T10:24:43.025324+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T10:24:43.025324+0000","last_clean_scrub_stamp":"2024-04-23T10:24:43.025324+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T16:38:48.756776+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,3],"acting":[5,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.19","version":"87'2","reported_seq":23,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-23T10:26:40.277848+0000","last_change":"2024-04-23T10:24:43.048754+0000","last_active":"2024-04-23T10:26:40.277848+0000","last_peered":"2024-04-23T10:26:40.277848+0000","last_clean":"2024-04-23T10:26:40.277848+0000","last_became_active":"2024-04-23T10:24:43.047962+0000","last_became_peered":"2024-04-23T10:24:43.047962+0000","last_unstale":"2024-04-23T10:26:40.277848+0000","last_undegraded":"2024-04-23T10:26:40.277848+0000","last_fullsized":"2024-04-23T10:26:40.277848+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T10:24:42.023127+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T10:24:42.023127+0000","last_clean_scrub_stamp":"2024-04-23T10:24:42.023127+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T21:08:38.487185+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,9],"acting":[6,4,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.18","version":"89'8","reported_seq":27,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-23T10:26:40.373688+0000","last_change":"2024-04-23T10:24:44.045022+0000","last_active":"2024-04-23T10:26:40.373688+0000","last_peered":"2024-04-23T10:26:40.373688+0000","last_clean":"2024-04-23T10:26:40.373688+0000","last_became_active":"2024-04-23T10:24:44.044590+0000","last_became_peered":"2024-04-23T10:24:44.044590+0000","last_unstale":"2024-04-23T10:26:40.373688+0000","last_undegraded":"2024-04-23T10:26:40.373688+0000","last_fullsized":"2024-04-23T10:26:40.373688+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T10:24:43.025324+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T10:24:43.025324+0000","last_clean_scrub_stamp":"2024-04-23T10:24:43.025324+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T13:20:49.892013+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,11,6],"acting":[3,11,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.18","version":"87'1","reported_seq":22,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-23T10:26:40.281503+0000","last_change":"2024-04-23T10:24:43.050170+0000","last_active":"2024-04-23T10:26:40.281503+0000","last_peered":"2024-04-23T10:26:40.281503+0000","last_clean":"2024-04-23T10:26:40.281503+0000","last_became_active":"2024-04-23T10:24:43.049070+0000","last_became_peered":"2024-04-23T10:24:43.049070+0000","last_unstale":"2024-04-23T10:26:40.281503+0000","last_undegraded":"2024-04-23T10:26:40.281503+0000","last_fullsized":"2024-04-23T10:26:40.281503+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T10:24:42.023127+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T10:24:42.023127+0000","last_clean_scrub_stamp":"2024-04-23T10:24:42.023127+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T19:34:13.983628+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,7],"acting":[5,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.19","version":"89'11","reported_seq":30,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-23T10:26:40.371665+0000","last_change":"2024-04-23T10:24:44.047165+0000","last_active":"2024-04-23T10:26:40.371665+0000","last_peered":"2024-04-23T10:26:40.371665+0000","last_clean":"2024-04-23T10:26:40.371665+0000","last_became_active":"2024-04-23T10:24:44.046776+0000","last_became_peered":"2024-04-23T10:24:44.046776+0000","last_unstale":"2024-04-23T10:26:40.371665+0000","last_undegraded":"2024-04-23T10:26:40.371665+0000","last_fullsized":"2024-04-23T10:26:40.371665+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T10:24:43.025324+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T10:24:43.025324+0000","last_clean_scrub_stamp":"2024-04-23T10:24:43.025324+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T18:11:57.181747+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,3,10],"acting":[9,3,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"2.17","version":"90'5","reported_seq":139,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-23T10:26:40.485905+0000","last_change":"2024-04-23T10:24:43.049785+0000","last_active":"2024-04-23T10:26:40.485905+0000","last_peered":"2024-04-23T10:26:40.485905+0000","last_clean":"2024-04-23T10:26:40.485905+0000","last_became_active":"2024-04-23T10:24:43.049402+0000","last_became_peered":"2024-04-23T10:24:43.049402+0000","last_unstale":"2024-04-23T10:26:40.485905+0000","last_undegraded":"2024-04-23T10:26:40.485905+0000","last_fullsized":"2024-04-23T10:26:40.485905+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T10:24:42.023127+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T10:24:42.023127+0000","last_clean_scrub_stamp":"2024-04-23T10:24:42.023127+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T14:20:13.774082+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":113,"num_read_kb":113,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,5,9],"acting":[10,5,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.16","version":"89'10","reported_seq":29,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-23T10:26:40.281357+0000","last_change":"2024-04-23T10:24:44.051768+0000","last_active":"2024-04-23T10:26:40.281357+0000","last_peered":"2024-04-23T10:26:40.281357+0000","last_clean":"2024-04-23T10:26:40.281357+0000","last_became_active":"2024-04-23T10:24:44.051228+0000","last_became_peered":"2024-04-23T10:24:44.051228+0000","last_unstale":"2024-04-23T10:26:40.281357+0000","last_undegraded":"2024-04-23T10:26:40.281357+0000","last_fullsized":"2024-04-23T10:26:40.281357+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T10:24:43.025324+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T10:24:43.025324+0000","last_clean_scrub_stamp":"2024-04-23T10:24:43.025324+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T14:09:47.649083+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,10,9],"acting":[1,10,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.16","version":"87'2","reported_seq":23,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-23T10:26:40.522354+0000","last_change":"2024-04-23T10:24:43.043944+0000","last_active":"2024-04-23T10:26:40.522354+0000","last_peered":"2024-04-23T10:26:40.522354+0000","last_clean":"2024-04-23T10:26:40.522354+0000","last_became_active":"2024-04-23T10:24:43.043648+0000","last_became_peered":"2024-04-23T10:24:43.043648+0000","last_unstale":"2024-04-23T10:26:40.522354+0000","last_undegraded":"2024-04-23T10:26:40.522354+0000","last_fullsized":"2024-04-23T10:26:40.522354+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T10:24:42.023127+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T10:24:42.023127+0000","last_clean_scrub_stamp":"2024-04-23T10:24:42.023127+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T13:41:15.724662+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,2],"acting":[7,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.17","version":"89'9","reported_seq":34,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-23T10:26:40.460012+0000","last_change":"2024-04-23T10:25:47.721534+0000","last_active":"2024-04-23T10:26:40.460012+0000","last_peered":"2024-04-23T10:26:40.460012+0000","last_clean":"2024-04-23T10:26:40.460012+0000","last_became_active":"2024-04-23T10:25:47.721130+0000","last_became_peered":"2024-04-23T10:25:47.721130+0000","last_unstale":"2024-04-23T10:26:40.460012+0000","last_undegraded":"2024-04-23T10:26:40.460012+0000","last_fullsized":"2024-04-23T10:26:40.460012+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T10:24:43.025324+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T10:24:43.025324+0000","last_clean_scrub_stamp":"2024-04-23T10:24:43.025324+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T21:21:26.537486+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,5],"acting":[0,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.15","version":"87'4","reported_seq":26,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-23T10:26:40.282238+0000","last_change":"2024-04-23T10:24:43.048517+0000","last_active":"2024-04-23T10:26:40.282238+0000","last_peered":"2024-04-23T10:26:40.282238+0000","last_clean":"2024-04-23T10:26:40.282238+0000","last_became_active":"2024-04-23T10:24:43.048183+0000","last_became_peered":"2024-04-23T10:24:43.048183+0000","last_unstale":"2024-04-23T10:26:40.282238+0000","last_undegraded":"2024-04-23T10:26:40.282238+0000","last_fullsized":"2024-04-23T10:26:40.282238+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T10:24:42.023127+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T10:24:42.023127+0000","last_clean_scrub_stamp":"2024-04-23T10:24:42.023127+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T20:50:15.017833+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":672,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":5,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,10],"acting":[5,0,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.14","version":"89'11","reported_seq":30,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-23T10:26:40.281189+0000","last_change":"2024-04-23T10:24:44.055598+0000","last_active":"2024-04-23T10:26:40.281189+0000","last_peered":"2024-04-23T10:26:40.281189+0000","last_clean":"2024-04-23T10:26:40.281189+0000","last_became_active":"2024-04-23T10:24:44.053787+0000","last_became_peered":"2024-04-23T10:24:44.053787+0000","last_unstale":"2024-04-23T10:26:40.281189+0000","last_undegraded":"2024-04-23T10:26:40.281189+0000","last_fullsized":"2024-04-23T10:26:40.281189+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T10:24:43.025324+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T10:24:43.025324+0000","last_clean_scrub_stamp":"2024-04-23T10:24:43.025324+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T20:04:21.603306+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,8,4],"acting":[1,8,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.14","version":"90'36","reported_seq":293,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-23T10:26:40.520887+0000","last_change":"2024-04-23T10:24:43.044601+0000","last_active":"2024-04-23T10:26:40.520887+0000","last_peered":"2024-04-23T10:26:40.520887+0000","last_clean":"2024-04-23T10:26:40.520887+0000","last_became_active":"2024-04-23T10:24:43.044238+0000","last_became_peered":"2024-04-23T10:24:43.044238+0000","last_unstale":"2024-04-23T10:26:40.520887+0000","last_undegraded":"2024-04-23T10:26:40.520887+0000","last_fullsized":"2024-04-23T10:26:40.520887+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T10:24:42.023127+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T10:24:42.023127+0000","last_clean_scrub_stamp":"2024-04-23T10:24:42.023127+0000","objects_scrubbed":0,"log_size":36,"log_dups_size":0,"ondisk_log_size":36,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T12:53:26.275771+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":67758,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":236,"num_read_kb":183,"num_write":37,"num_write_kb":82,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,1,2],"acting":[10,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.15","version":"89'6","reported_seq":25,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-23T10:26:40.373730+0000","last_change":"2024-04-23T10:24:44.051375+0000","last_active":"2024-04-23T10:26:40.373730+0000","last_peered":"2024-04-23T10:26:40.373730+0000","last_clean":"2024-04-23T10:26:40.373730+0000","last_became_active":"2024-04-23T10:24:44.050924+0000","last_became_peered":"2024-04-23T10:24:44.050924+0000","last_unstale":"2024-04-23T10:26:40.373730+0000","last_undegraded":"2024-04-23T10:26:40.373730+0000","last_fullsized":"2024-04-23T10:26:40.373730+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T10:24:43.025324+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T10:24:43.025324+0000","last_clean_scrub_stamp":"2024-04-23T10:24:43.025324+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T21:50:57.534760+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,11,7],"acting":[8,11,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"2.13","version":"87'6","reported_seq":28,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-23T10:26:40.459793+0000","last_change":"2024-04-23T10:24:43.044457+0000","last_active":"2024-04-23T10:26:40.459793+0000","last_peered":"2024-04-23T10:26:40.459793+0000","last_clean":"2024-04-23T10:26:40.459793+0000","last_became_active":"2024-04-23T10:24:43.043956+0000","last_became_peered":"2024-04-23T10:24:43.043956+0000","last_unstale":"2024-04-23T10:26:40.459793+0000","last_undegraded":"2024-04-23T10:26:40.459793+0000","last_fullsized":"2024-04-23T10:26:40.459793+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T10:24:42.023127+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T10:24:42.023127+0000","last_clean_scrub_stamp":"2024-04-23T10:24:42.023127+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T20:16:08.186729+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":9,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,10,3],"acting":[2,10,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.12","version":"89'9","reported_seq":34,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-23T10:26:40.277806+0000","last_change":"2024-04-23T10:25:47.587240+0000","last_active":"2024-04-23T10:26:40.277806+0000","last_peered":"2024-04-23T10:26:40.277806+0000","last_clean":"2024-04-23T10:26:40.277806+0000","last_became_active":"2024-04-23T10:25:47.586659+0000","last_became_peered":"2024-04-23T10:25:47.586659+0000","last_unstale":"2024-04-23T10:26:40.277806+0000","last_undegraded":"2024-04-23T10:26:40.277806+0000","last_fullsized":"2024-04-23T10:26:40.277806+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T10:24:43.025324+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T10:24:43.025324+0000","last_clean_scrub_stamp":"2024-04-23T10:24:43.025324+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T19:55:03.256183+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,9],"acting":[6,7,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.12","version":"87'3","reported_seq":24,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-23T10:26:40.281768+0000","last_change":"2024-04-23T10:24:43.052181+0000","last_active":"2024-04-23T10:26:40.281768+0000","last_peered":"2024-04-23T10:26:40.281768+0000","last_clean":"2024-04-23T10:26:40.281768+0000","last_became_active":"2024-04-23T10:24:43.051793+0000","last_became_peered":"2024-04-23T10:24:43.051793+0000","last_unstale":"2024-04-23T10:26:40.281768+0000","last_undegraded":"2024-04-23T10:26:40.281768+0000","last_fullsized":"2024-04-23T10:26:40.281768+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T10:24:42.023127+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T10:24:42.023127+0000","last_clean_scrub_stamp":"2024-04-23T10:24:42.023127+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T20:57:59.015115+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,7],"acting":[5,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.13","version":"89'15","reported_seq":34,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-23T10:26:40.522012+0000","last_change":"2024-04-23T10:24:44.062689+0000","last_active":"2024-04-23T10:26:40.522012+0000","last_peered":"2024-04-23T10:26:40.522012+0000","last_clean":"2024-04-23T10:26:40.522012+0000","last_became_active":"2024-04-23T10:24:44.062331+0000","last_became_peered":"2024-04-23T10:24:44.062331+0000","last_unstale":"2024-04-23T10:26:40.522012+0000","last_undegraded":"2024-04-23T10:26:40.522012+0000","last_fullsized":"2024-04-23T10:26:40.522012+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T10:24:43.025324+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T10:24:43.025324+0000","last_clean_scrub_stamp":"2024-04-23T10:24:43.025324+0000","objects_scrubbed":0,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T14:06:04.989156+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,6],"acting":[7,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.11","version":"90'6","reported_seq":144,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-23T10:26:40.486457+0000","last_change":"2024-04-23T10:24:43.043901+0000","last_active":"2024-04-23T10:26:40.486457+0000","last_peered":"2024-04-23T10:26:40.486457+0000","last_clean":"2024-04-23T10:26:40.486457+0000","last_became_active":"2024-04-23T10:24:43.043598+0000","last_became_peered":"2024-04-23T10:24:43.043598+0000","last_unstale":"2024-04-23T10:26:40.486457+0000","last_undegraded":"2024-04-23T10:26:40.486457+0000","last_fullsized":"2024-04-23T10:26:40.486457+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T10:24:42.023127+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T10:24:42.023127+0000","last_clean_scrub_stamp":"2024-04-23T10:24:42.023127+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T14:00:06.338084+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1511,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":118,"num_read_kb":5,"num_write":5,"num_write_kb":4,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,10],"acting":[4,7,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.10","version":"89'6","reported_seq":25,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-23T10:26:40.521595+0000","last_change":"2024-04-23T10:24:44.058773+0000","last_active":"2024-04-23T10:26:40.521595+0000","last_peered":"2024-04-23T10:26:40.521595+0000","last_clean":"2024-04-23T10:26:40.521595+0000","last_became_active":"2024-04-23T10:24:44.058420+0000","last_became_peered":"2024-04-23T10:24:44.058420+0000","last_unstale":"2024-04-23T10:26:40.521595+0000","last_undegraded":"2024-04-23T10:26:40.521595+0000","last_fullsized":"2024-04-23T10:26:40.521595+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T10:24:43.025324+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T10:24:43.025324+0000","last_clean_scrub_stamp":"2024-04-23T10:24:43.025324+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T12:44:30.023047+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,10],"acting":[7,5,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.10","version":"87'15","reported_seq":37,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-23T10:26:40.373552+0000","last_change":"2024-04-23T10:24:43.044700+0000","last_active":"2024-04-23T10:26:40.373552+0000","last_peered":"2024-04-23T10:26:40.373552+0000","last_clean":"2024-04-23T10:26:40.373552+0000","last_became_active":"2024-04-23T10:24:43.044394+0000","last_became_peered":"2024-04-23T10:24:43.044394+0000","last_unstale":"2024-04-23T10:26:40.373552+0000","last_undegraded":"2024-04-23T10:26:40.373552+0000","last_fullsized":"2024-04-23T10:26:40.373552+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T10:24:42.023127+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T10:24:42.023127+0000","last_clean_scrub_stamp":"2024-04-23T10:24:42.023127+0000","objects_scrubbed":0,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T15:16:41.367517+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":112,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":20,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":6,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,0,2],"acting":[8,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.11","version":"89'6","reported_seq":25,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-23T10:26:40.281571+0000","last_change":"2024-04-23T10:24:44.049739+0000","last_active":"2024-04-23T10:26:40.281571+0000","last_peered":"2024-04-23T10:26:40.281571+0000","last_clean":"2024-04-23T10:26:40.281571+0000","last_became_active":"2024-04-23T10:24:44.049414+0000","last_became_peered":"2024-04-23T10:24:44.049414+0000","last_unstale":"2024-04-23T10:26:40.281571+0000","last_undegraded":"2024-04-23T10:26:40.281571+0000","last_fullsized":"2024-04-23T10:26:40.281571+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T10:24:43.025324+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T10:24:43.025324+0000","last_clean_scrub_stamp":"2024-04-23T10:24:43.025324+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T12:51:02.645259+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,1],"acting":[5,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.f","version":"87'2","reported_seq":24,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-23T10:26:40.373920+0000","last_change":"2024-04-23T10:24:43.048884+0000","last_active":"2024-04-23T10:26:40.373920+0000","last_peered":"2024-04-23T10:26:40.373920+0000","last_clean":"2024-04-23T10:26:40.373920+0000","last_became_active":"2024-04-23T10:24:43.048583+0000","last_became_peered":"2024-04-23T10:24:43.048583+0000","last_unstale":"2024-04-23T10:26:40.373920+0000","last_undegraded":"2024-04-23T10:26:40.373920+0000","last_fullsized":"2024-04-23T10:26:40.373920+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T10:24:42.023127+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T10:24:42.023127+0000","last_clean_scrub_stamp":"2024-04-23T10:24:42.023127+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T17:55:57.869700+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,5,4],"acting":[8,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.e","version":"89'5","reported_seq":30,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-23T10:26:40.460256+0000","last_change":"2024-04-23T10:25:47.722078+0000","last_active":"2024-04-23T10:26:40.460256+0000","last_peered":"2024-04-23T10:26:40.460256+0000","last_clean":"2024-04-23T10:26:40.460256+0000","last_became_active":"2024-04-23T10:25:47.721730+0000","last_became_peered":"2024-04-23T10:25:47.721730+0000","last_unstale":"2024-04-23T10:26:40.460256+0000","last_undegraded":"2024-04-23T10:26:40.460256+0000","last_fullsized":"2024-04-23T10:26:40.460256+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T10:24:43.025324+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T10:24:43.025324+0000","last_clean_scrub_stamp":"2024-04-23T10:24:43.025324+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T18:18:40.916492+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,9,4],"acting":[11,9,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":21,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-23T10:26:40.371847+0000","last_change":"2024-04-23T10:24:43.044375+0000","last_active":"2024-04-23T10:26:40.371847+0000","last_peered":"2024-04-23T10:26:40.371847+0000","last_clean":"2024-04-23T10:26:40.371847+0000","last_became_active":"2024-04-23T10:24:43.043914+0000","last_became_peered":"2024-04-23T10:24:43.043914+0000","last_unstale":"2024-04-23T10:26:40.371847+0000","last_undegraded":"2024-04-23T10:26:40.371847+0000","last_fullsized":"2024-04-23T10:26:40.371847+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T10:24:42.023127+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T10:24:42.023127+0000","last_clean_scrub_stamp":"2024-04-23T10:24:42.023127+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T16:22:44.165156+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,2,0],"acting":[9,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.5","version":"89'5","reported_seq":24,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-23T10:26:40.381571+0000","last_change":"2024-04-23T10:24:44.057272+0000","last_active":"2024-04-23T10:26:40.381571+0000","last_peered":"2024-04-23T10:26:40.381571+0000","last_clean":"2024-04-23T10:26:40.381571+0000","last_became_active":"2024-04-23T10:24:44.056968+0000","last_became_peered":"2024-04-23T10:24:44.056968+0000","last_unstale":"2024-04-23T10:26:40.381571+0000","last_undegraded":"2024-04-23T10:26:40.381571+0000","last_fullsized":"2024-04-23T10:26:40.381571+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T10:24:43.025324+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T10:24:43.025324+0000","last_clean_scrub_stamp":"2024-04-23T10:24:43.025324+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T11:10:09.021845+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":117,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,3,6],"acting":[10,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"2.2","version":"87'1","reported_seq":22,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-23T10:26:40.521864+0000","last_change":"2024-04-23T10:24:43.051822+0000","last_active":"2024-04-23T10:26:40.521864+0000","last_peered":"2024-04-23T10:26:40.521864+0000","last_clean":"2024-04-23T10:26:40.521864+0000","last_became_active":"2024-04-23T10:24:43.051491+0000","last_became_peered":"2024-04-23T10:24:43.051491+0000","last_unstale":"2024-04-23T10:26:40.521864+0000","last_undegraded":"2024-04-23T10:26:40.521864+0000","last_fullsized":"2024-04-23T10:26:40.521864+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T10:24:42.023127+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T10:24:42.023127+0000","last_clean_scrub_stamp":"2024-04-23T10:24:42.023127+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T13:49:23.109064+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,1],"acting":[7,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.3","version":"89'7","reported_seq":26,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-23T10:26:40.381469+0000","last_change":"2024-04-23T10:24:44.053604+0000","last_active":"2024-04-23T10:26:40.381469+0000","last_peered":"2024-04-23T10:26:40.381469+0000","last_clean":"2024-04-23T10:26:40.381469+0000","last_became_active":"2024-04-23T10:24:44.053147+0000","last_became_peered":"2024-04-23T10:24:44.053147+0000","last_unstale":"2024-04-23T10:26:40.381469+0000","last_undegraded":"2024-04-23T10:26:40.381469+0000","last_fullsized":"2024-04-23T10:26:40.381469+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T10:24:43.025324+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T10:24:43.025324+0000","last_clean_scrub_stamp":"2024-04-23T10:24:43.025324+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T21:58:38.981282+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,4,0],"acting":[10,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.0","version":"89'8","reported_seq":27,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-23T10:26:40.281729+0000","last_change":"2024-04-23T10:24:44.052564+0000","last_active":"2024-04-23T10:26:40.281729+0000","last_peered":"2024-04-23T10:26:40.281729+0000","last_clean":"2024-04-23T10:26:40.281729+0000","last_became_active":"2024-04-23T10:24:44.052201+0000","last_became_peered":"2024-04-23T10:24:44.052201+0000","last_unstale":"2024-04-23T10:26:40.281729+0000","last_undegraded":"2024-04-23T10:26:40.281729+0000","last_fullsized":"2024-04-23T10:26:40.281729+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T10:24:43.025324+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T10:24:43.025324+0000","last_clean_scrub_stamp":"2024-04-23T10:24:43.025324+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T16:25:54.725347+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,4],"acting":[5,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.1","version":"90'6","reported_seq":140,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-23T10:26:40.486009+0000","last_change":"2024-04-23T10:24:43.049067+0000","last_active":"2024-04-23T10:26:40.486009+0000","last_peered":"2024-04-23T10:26:40.486009+0000","last_clean":"2024-04-23T10:26:40.486009+0000","last_became_active":"2024-04-23T10:24:43.048400+0000","last_became_peered":"2024-04-23T10:24:43.048400+0000","last_unstale":"2024-04-23T10:26:40.486009+0000","last_undegraded":"2024-04-23T10:26:40.486009+0000","last_fullsized":"2024-04-23T10:26:40.486009+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T10:24:42.023127+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T10:24:42.023127+0000","last_clean_scrub_stamp":"2024-04-23T10:24:42.023127+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T12:17:02.439054+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":113,"num_read_kb":113,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,0],"acting":[6,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.2","version":"89'8","reported_seq":27,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-23T10:26:40.373993+0000","last_change":"2024-04-23T10:24:44.059114+0000","last_active":"2024-04-23T10:26:40.373993+0000","last_peered":"2024-04-23T10:26:40.373993+0000","last_clean":"2024-04-23T10:26:40.373993+0000","last_became_active":"2024-04-23T10:24:44.058618+0000","last_became_peered":"2024-04-23T10:24:44.058618+0000","last_unstale":"2024-04-23T10:26:40.373993+0000","last_undegraded":"2024-04-23T10:26:40.373993+0000","last_fullsized":"2024-04-23T10:26:40.373993+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T10:24:43.025324+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T10:24:43.025324+0000","last_clean_scrub_stamp":"2024-04-23T10:24:43.025324+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T20:18:09.707062+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,1],"acting":[3,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.3","version":"87'3","reported_seq":26,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-23T10:26:40.459688+0000","last_change":"2024-04-23T10:24:43.039725+0000","last_active":"2024-04-23T10:26:40.459688+0000","last_peered":"2024-04-23T10:26:40.459688+0000","last_clean":"2024-04-23T10:26:40.459688+0000","last_became_active":"2024-04-23T10:24:43.039187+0000","last_became_peered":"2024-04-23T10:24:43.039187+0000","last_unstale":"2024-04-23T10:26:40.459688+0000","last_undegraded":"2024-04-23T10:26:40.459688+0000","last_fullsized":"2024-04-23T10:26:40.459688+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T10:24:42.023127+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T10:24:42.023127+0000","last_clean_scrub_stamp":"2024-04-23T10:24:42.023127+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T17:20:50.844878+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":3,"num_read_kb":3,"num_write":4,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,10,9],"acting":[4,10,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.0","version":"22'137","reported_seq":353,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-23T10:26:40.521666+0000","last_change":"2024-04-23T10:23:31.092860+0000","last_active":"2024-04-23T10:26:40.521666+0000","last_peered":"2024-04-23T10:26:40.521666+0000","last_clean":"2024-04-23T10:26:40.521666+0000","last_became_active":"2024-04-23T10:23:31.092357+0000","last_became_peered":"2024-04-23T10:23:31.092357+0000","last_unstale":"2024-04-23T10:26:40.521666+0000","last_undegraded":"2024-04-23T10:26:40.521666+0000","last_fullsized":"2024-04-23T10:26:40.521666+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T10:20:12.201233+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T10:20:12.201233+0000","last_clean_scrub_stamp":"2024-04-23T10:20:12.201233+0000","objects_scrubbed":0,"log_size":137,"log_dups_size":0,"ondisk_log_size":137,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T21:28:56.804053+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":136,"num_read_kb":119,"num_write":181,"num_write_kb":2938,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":1180736,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,10],"acting":[7,5,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1","version":"89'5","reported_seq":24,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-23T10:26:40.277697+0000","last_change":"2024-04-23T10:24:44.058393+0000","last_active":"2024-04-23T10:26:40.277697+0000","last_peered":"2024-04-23T10:26:40.277697+0000","last_clean":"2024-04-23T10:26:40.277697+0000","last_became_active":"2024-04-23T10:24:44.058075+0000","last_became_peered":"2024-04-23T10:24:44.058075+0000","last_unstale":"2024-04-23T10:26:40.277697+0000","last_undegraded":"2024-04-23T10:26:40.277697+0000","last_fullsized":"2024-04-23T10:26:40.277697+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T10:24:43.025324+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T10:24:43.025324+0000","last_clean_scrub_stamp":"2024-04-23T10:24:43.025324+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T10:32:05.693311+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,2],"acting":[6,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.0","version":"87'3","reported_seq":24,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-23T10:26:40.521796+0000","last_change":"2024-04-23T10:24:43.048678+0000","last_active":"2024-04-23T10:26:40.521796+0000","last_peered":"2024-04-23T10:26:40.521796+0000","last_clean":"2024-04-23T10:26:40.521796+0000","last_became_active":"2024-04-23T10:24:43.048341+0000","last_became_peered":"2024-04-23T10:24:43.048341+0000","last_unstale":"2024-04-23T10:26:40.521796+0000","last_undegraded":"2024-04-23T10:26:40.521796+0000","last_fullsized":"2024-04-23T10:26:40.521796+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T10:24:42.023127+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T10:24:42.023127+0000","last_clean_scrub_stamp":"2024-04-23T10:24:42.023127+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T12:24:33.705325+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,8],"acting":[7,5,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.5","version":"87'1","reported_seq":23,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-23T10:26:40.373795+0000","last_change":"2024-04-23T10:24:43.046105+0000","last_active":"2024-04-23T10:26:40.373795+0000","last_peered":"2024-04-23T10:26:40.373795+0000","last_clean":"2024-04-23T10:26:40.373795+0000","last_became_active":"2024-04-23T10:24:43.045804+0000","last_became_peered":"2024-04-23T10:24:43.045804+0000","last_unstale":"2024-04-23T10:26:40.373795+0000","last_undegraded":"2024-04-23T10:26:40.373795+0000","last_fullsized":"2024-04-23T10:26:40.373795+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T10:24:42.023127+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T10:24:42.023127+0000","last_clean_scrub_stamp":"2024-04-23T10:24:42.023127+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T22:23:35.927468+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,0,9],"acting":[8,0,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.4","version":"89'9","reported_seq":28,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-23T10:26:40.278062+0000","last_change":"2024-04-23T10:24:44.059731+0000","last_active":"2024-04-23T10:26:40.278062+0000","last_peered":"2024-04-23T10:26:40.278062+0000","last_clean":"2024-04-23T10:26:40.278062+0000","last_became_active":"2024-04-23T10:24:44.059352+0000","last_became_peered":"2024-04-23T10:24:44.059352+0000","last_unstale":"2024-04-23T10:26:40.278062+0000","last_undegraded":"2024-04-23T10:26:40.278062+0000","last_fullsized":"2024-04-23T10:26:40.278062+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T10:24:43.025324+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T10:24:43.025324+0000","last_clean_scrub_stamp":"2024-04-23T10:24:43.025324+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T16:36:08.985444+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2,9],"acting":[6,2,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.6","version":"90'44","reported_seq":88,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-23T10:26:40.483019+0000","last_change":"2024-04-23T10:24:43.044527+0000","last_active":"2024-04-23T10:26:40.483019+0000","last_peered":"2024-04-23T10:26:40.483019+0000","last_clean":"2024-04-23T10:26:40.483019+0000","last_became_active":"2024-04-23T10:24:43.044212+0000","last_became_peered":"2024-04-23T10:24:43.044212+0000","last_unstale":"2024-04-23T10:26:40.483019+0000","last_undegraded":"2024-04-23T10:26:40.483019+0000","last_fullsized":"2024-04-23T10:26:40.483019+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T10:24:42.023127+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T10:24:42.023127+0000","last_clean_scrub_stamp":"2024-04-23T10:24:42.023127+0000","objects_scrubbed":0,"log_size":44,"log_dups_size":0,"ondisk_log_size":44,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T19:39:46.369132+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":393216,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":82,"num_read_kb":70,"num_write":78,"num_write_kb":836,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,8,3],"acting":[9,8,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.7","version":"89'5","reported_seq":24,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-23T10:26:40.374352+0000","last_change":"2024-04-23T10:24:44.055920+0000","last_active":"2024-04-23T10:26:40.374352+0000","last_peered":"2024-04-23T10:26:40.374352+0000","last_clean":"2024-04-23T10:26:40.374352+0000","last_became_active":"2024-04-23T10:24:44.055411+0000","last_became_peered":"2024-04-23T10:24:44.055411+0000","last_unstale":"2024-04-23T10:26:40.374352+0000","last_undegraded":"2024-04-23T10:26:40.374352+0000","last_fullsized":"2024-04-23T10:26:40.374352+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T10:24:43.025324+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T10:24:43.025324+0000","last_clean_scrub_stamp":"2024-04-23T10:24:43.025324+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T14:43:10.446790+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,11,7],"acting":[3,11,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.7","version":"87'4","reported_seq":25,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-23T10:26:40.459655+0000","last_change":"2024-04-23T10:24:43.047989+0000","last_active":"2024-04-23T10:26:40.459655+0000","last_peered":"2024-04-23T10:26:40.459655+0000","last_clean":"2024-04-23T10:26:40.459655+0000","last_became_active":"2024-04-23T10:24:43.047625+0000","last_became_peered":"2024-04-23T10:24:43.047625+0000","last_unstale":"2024-04-23T10:26:40.459655+0000","last_undegraded":"2024-04-23T10:26:40.459655+0000","last_fullsized":"2024-04-23T10:26:40.459655+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T10:24:42.023127+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T10:24:42.023127+0000","last_clean_scrub_stamp":"2024-04-23T10:24:42.023127+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T13:50:16.647176+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,11,6],"acting":[4,11,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.6","version":"89'7","reported_seq":26,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-23T10:26:40.459895+0000","last_change":"2024-04-23T10:24:44.049214+0000","last_active":"2024-04-23T10:26:40.459895+0000","last_peered":"2024-04-23T10:26:40.459895+0000","last_clean":"2024-04-23T10:26:40.459895+0000","last_became_active":"2024-04-23T10:24:44.048869+0000","last_became_peered":"2024-04-23T10:24:44.048869+0000","last_unstale":"2024-04-23T10:26:40.459895+0000","last_undegraded":"2024-04-23T10:26:40.459895+0000","last_fullsized":"2024-04-23T10:26:40.459895+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T10:24:43.025324+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T10:24:43.025324+0000","last_clean_scrub_stamp":"2024-04-23T10:24:43.025324+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T18:17:09.772428+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,4],"acting":[0,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.8","version":"87'2","reported_seq":23,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-23T10:26:40.281014+0000","last_change":"2024-04-23T10:24:43.046189+0000","last_active":"2024-04-23T10:26:40.281014+0000","last_peered":"2024-04-23T10:26:40.281014+0000","last_clean":"2024-04-23T10:26:40.281014+0000","last_became_active":"2024-04-23T10:24:43.045848+0000","last_became_peered":"2024-04-23T10:24:43.045848+0000","last_unstale":"2024-04-23T10:26:40.281014+0000","last_undegraded":"2024-04-23T10:26:40.281014+0000","last_fullsized":"2024-04-23T10:26:40.281014+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T10:24:42.023127+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T10:24:42.023127+0000","last_clean_scrub_stamp":"2024-04-23T10:24:42.023127+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T22:23:39.180377+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,0],"acting":[1,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.9","version":"89'2","reported_seq":21,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-23T10:26:40.373557+0000","last_change":"2024-04-23T10:24:44.048415+0000","last_active":"2024-04-23T10:26:40.373557+0000","last_peered":"2024-04-23T10:26:40.373557+0000","last_clean":"2024-04-23T10:26:40.373557+0000","last_became_active":"2024-04-23T10:24:44.048030+0000","last_became_peered":"2024-04-23T10:24:44.048030+0000","last_unstale":"2024-04-23T10:26:40.373557+0000","last_undegraded":"2024-04-23T10:26:40.373557+0000","last_fullsized":"2024-04-23T10:26:40.373557+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T10:24:43.025324+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T10:24:43.025324+0000","last_clean_scrub_stamp":"2024-04-23T10:24:43.025324+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T14:42:09.920927+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,11,10],"acting":[8,11,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"2.9","version":"87'3","reported_seq":24,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-23T10:26:40.373745+0000","last_change":"2024-04-23T10:24:43.041525+0000","last_active":"2024-04-23T10:26:40.373745+0000","last_peered":"2024-04-23T10:26:40.373745+0000","last_clean":"2024-04-23T10:26:40.373745+0000","last_became_active":"2024-04-23T10:24:43.041200+0000","last_became_peered":"2024-04-23T10:24:43.041200+0000","last_unstale":"2024-04-23T10:26:40.373745+0000","last_undegraded":"2024-04-23T10:26:40.373745+0000","last_fullsized":"2024-04-23T10:26:40.373745+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T10:24:42.023127+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T10:24:42.023127+0000","last_clean_scrub_stamp":"2024-04-23T10:24:42.023127+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T16:45:06.846991+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,9,10],"acting":[3,9,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.8","version":"89'9","reported_seq":28,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-23T10:26:40.460025+0000","last_change":"2024-04-23T10:24:44.055341+0000","last_active":"2024-04-23T10:26:40.460025+0000","last_peered":"2024-04-23T10:26:40.460025+0000","last_clean":"2024-04-23T10:26:40.460025+0000","last_became_active":"2024-04-23T10:24:44.054216+0000","last_became_peered":"2024-04-23T10:24:44.054216+0000","last_unstale":"2024-04-23T10:26:40.460025+0000","last_undegraded":"2024-04-23T10:26:40.460025+0000","last_fullsized":"2024-04-23T10:26:40.460025+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T10:24:43.025324+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T10:24:43.025324+0000","last_clean_scrub_stamp":"2024-04-23T10:24:43.025324+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T21:31:07.753283+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,0,2],"acting":[11,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"2.a","version":"87'1","reported_seq":22,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-23T10:26:40.459791+0000","last_change":"2024-04-23T10:24:43.045400+0000","last_active":"2024-04-23T10:26:40.459791+0000","last_peered":"2024-04-23T10:26:40.459791+0000","last_clean":"2024-04-23T10:26:40.459791+0000","last_became_active":"2024-04-23T10:24:43.044677+0000","last_became_peered":"2024-04-23T10:24:43.044677+0000","last_unstale":"2024-04-23T10:26:40.459791+0000","last_undegraded":"2024-04-23T10:26:40.459791+0000","last_fullsized":"2024-04-23T10:26:40.459791+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T10:24:42.023127+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T10:24:42.023127+0000","last_clean_scrub_stamp":"2024-04-23T10:24:42.023127+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T19:06:19.700132+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,8],"acting":[2,3,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.b","version":"89'8","reported_seq":27,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-23T10:26:40.374065+0000","last_change":"2024-04-23T10:24:44.049968+0000","last_active":"2024-04-23T10:26:40.374065+0000","last_peered":"2024-04-23T10:26:40.374065+0000","last_clean":"2024-04-23T10:26:40.374065+0000","last_became_active":"2024-04-23T10:24:44.049621+0000","last_became_peered":"2024-04-23T10:24:44.049621+0000","last_unstale":"2024-04-23T10:26:40.374065+0000","last_undegraded":"2024-04-23T10:26:40.374065+0000","last_fullsized":"2024-04-23T10:26:40.374065+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T10:24:43.025324+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T10:24:43.025324+0000","last_clean_scrub_stamp":"2024-04-23T10:24:43.025324+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T16:23:43.082831+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,10],"acting":[3,0,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.b","version":"87'4","reported_seq":26,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-23T10:26:40.373780+0000","last_change":"2024-04-23T10:24:43.041828+0000","last_active":"2024-04-23T10:26:40.373780+0000","last_peered":"2024-04-23T10:26:40.373780+0000","last_clean":"2024-04-23T10:26:40.373780+0000","last_became_active":"2024-04-23T10:24:43.041480+0000","last_became_peered":"2024-04-23T10:24:43.041480+0000","last_unstale":"2024-04-23T10:26:40.373780+0000","last_undegraded":"2024-04-23T10:26:40.373780+0000","last_fullsized":"2024-04-23T10:26:40.373780+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T10:24:42.023127+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T10:24:42.023127+0000","last_clean_scrub_stamp":"2024-04-23T10:24:42.023127+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T14:43:03.203971+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":2,"num_read_kb":2,"num_write":4,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,2,7],"acting":[8,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.a","version":"89'5","reported_seq":24,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-23T10:26:40.522120+0000","last_change":"2024-04-23T10:24:44.062103+0000","last_active":"2024-04-23T10:26:40.522120+0000","last_peered":"2024-04-23T10:26:40.522120+0000","last_clean":"2024-04-23T10:26:40.522120+0000","last_became_active":"2024-04-23T10:24:44.061794+0000","last_became_peered":"2024-04-23T10:24:44.061794+0000","last_unstale":"2024-04-23T10:26:40.522120+0000","last_undegraded":"2024-04-23T10:26:40.522120+0000","last_fullsized":"2024-04-23T10:26:40.522120+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T10:24:43.025324+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T10:24:43.025324+0000","last_clean_scrub_stamp":"2024-04-23T10:24:43.025324+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T10:45:30.130047+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,6],"acting":[7,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.c","version":"87'3","reported_seq":25,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-23T10:26:40.460001+0000","last_change":"2024-04-23T10:24:43.045702+0000","last_active":"2024-04-23T10:26:40.460001+0000","last_peered":"2024-04-23T10:26:40.460001+0000","last_clean":"2024-04-23T10:26:40.460001+0000","last_became_active":"2024-04-23T10:24:43.045258+0000","last_became_peered":"2024-04-23T10:24:43.045258+0000","last_unstale":"2024-04-23T10:26:40.460001+0000","last_undegraded":"2024-04-23T10:26:40.460001+0000","last_fullsized":"2024-04-23T10:26:40.460001+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T10:24:42.023127+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T10:24:42.023127+0000","last_clean_scrub_stamp":"2024-04-23T10:24:42.023127+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T12:54:30.965080+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,11,0],"acting":[2,11,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.d","version":"89'7","reported_seq":26,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-23T10:26:40.522390+0000","last_change":"2024-04-23T10:24:44.061118+0000","last_active":"2024-04-23T10:26:40.522390+0000","last_peered":"2024-04-23T10:26:40.522390+0000","last_clean":"2024-04-23T10:26:40.522390+0000","last_became_active":"2024-04-23T10:24:44.060743+0000","last_became_peered":"2024-04-23T10:24:44.060743+0000","last_unstale":"2024-04-23T10:26:40.522390+0000","last_undegraded":"2024-04-23T10:26:40.522390+0000","last_fullsized":"2024-04-23T10:26:40.522390+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T10:24:43.025324+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T10:24:43.025324+0000","last_clean_scrub_stamp":"2024-04-23T10:24:43.025324+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T17:17:46.731121+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,11],"acting":[7,1,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.d","version":"90'39","reported_seq":68,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-23T10:26:40.480483+0000","last_change":"2024-04-23T10:24:43.052519+0000","last_active":"2024-04-23T10:26:40.480483+0000","last_peered":"2024-04-23T10:26:40.480483+0000","last_clean":"2024-04-23T10:26:40.480483+0000","last_became_active":"2024-04-23T10:24:43.052187+0000","last_became_peered":"2024-04-23T10:24:43.052187+0000","last_unstale":"2024-04-23T10:26:40.480483+0000","last_undegraded":"2024-04-23T10:26:40.480483+0000","last_fullsized":"2024-04-23T10:26:40.480483+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T10:24:42.023127+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T10:24:42.023127+0000","last_clean_scrub_stamp":"2024-04-23T10:24:42.023127+0000","objects_scrubbed":0,"log_size":39,"log_dups_size":0,"ondisk_log_size":39,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T15:51:52.839409+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":131670,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":11,"num_read_kb":10,"num_write":49,"num_write_kb":475,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,10,3],"acting":[5,10,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.c","version":"89'7","reported_seq":26,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-23T10:26:40.521835+0000","last_change":"2024-04-23T10:24:44.053086+0000","last_active":"2024-04-23T10:26:40.521835+0000","last_peered":"2024-04-23T10:26:40.521835+0000","last_clean":"2024-04-23T10:26:40.521835+0000","last_became_active":"2024-04-23T10:24:44.052687+0000","last_became_peered":"2024-04-23T10:24:44.052687+0000","last_unstale":"2024-04-23T10:26:40.521835+0000","last_undegraded":"2024-04-23T10:26:40.521835+0000","last_fullsized":"2024-04-23T10:26:40.521835+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T10:24:43.025324+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T10:24:43.025324+0000","last_clean_scrub_stamp":"2024-04-23T10:24:43.025324+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T14:35:44.958742+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":117,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,5],"acting":[7,3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.e","version":"86'2","reported_seq":33,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-23T10:26:40.278094+0000","last_change":"2024-04-23T10:25:47.724936+0000","last_active":"2024-04-23T10:26:40.278094+0000","last_peered":"2024-04-23T10:26:40.278094+0000","last_clean":"2024-04-23T10:26:40.278094+0000","last_became_active":"2024-04-23T10:25:47.722182+0000","last_became_peered":"2024-04-23T10:25:47.722182+0000","last_unstale":"2024-04-23T10:26:40.278094+0000","last_undegraded":"2024-04-23T10:26:40.278094+0000","last_fullsized":"2024-04-23T10:26:40.278094+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T10:24:42.023127+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T10:24:42.023127+0000","last_clean_scrub_stamp":"2024-04-23T10:24:42.023127+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T19:18:42.762648+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,4],"acting":[6,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.f","version":"89'9","reported_seq":28,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-23T10:26:40.522498+0000","last_change":"2024-04-23T10:24:44.049543+0000","last_active":"2024-04-23T10:26:40.522498+0000","last_peered":"2024-04-23T10:26:40.522498+0000","last_clean":"2024-04-23T10:26:40.522498+0000","last_became_active":"2024-04-23T10:24:44.049198+0000","last_became_peered":"2024-04-23T10:24:44.049198+0000","last_unstale":"2024-04-23T10:26:40.522498+0000","last_undegraded":"2024-04-23T10:26:40.522498+0000","last_fullsized":"2024-04-23T10:26:40.522498+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T10:24:43.025324+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T10:24:43.025324+0000","last_clean_scrub_stamp":"2024-04-23T10:24:43.025324+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T12:25:15.974267+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,11,8],"acting":[7,11,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.20","version":"89'9","reported_seq":28,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-23T10:26:40.460095+0000","last_change":"2024-04-23T10:24:44.054440+0000","last_active":"2024-04-23T10:26:40.460095+0000","last_peered":"2024-04-23T10:26:40.460095+0000","last_clean":"2024-04-23T10:26:40.460095+0000","last_became_active":"2024-04-23T10:24:44.053951+0000","last_became_peered":"2024-04-23T10:24:44.053951+0000","last_unstale":"2024-04-23T10:26:40.460095+0000","last_undegraded":"2024-04-23T10:26:40.460095+0000","last_fullsized":"2024-04-23T10:26:40.460095+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T10:24:43.025324+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T10:24:43.025324+0000","last_clean_scrub_stamp":"2024-04-23T10:24:43.025324+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T22:20:10.452712+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,2,5],"acting":[11,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.21","version":"89'12","reported_seq":31,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-23T10:26:40.459759+0000","last_change":"2024-04-23T10:24:44.059015+0000","last_active":"2024-04-23T10:26:40.459759+0000","last_peered":"2024-04-23T10:26:40.459759+0000","last_clean":"2024-04-23T10:26:40.459759+0000","last_became_active":"2024-04-23T10:24:44.058691+0000","last_became_peered":"2024-04-23T10:24:44.058691+0000","last_unstale":"2024-04-23T10:26:40.459759+0000","last_undegraded":"2024-04-23T10:26:40.459759+0000","last_fullsized":"2024-04-23T10:26:40.459759+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T10:24:43.025324+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T10:24:43.025324+0000","last_clean_scrub_stamp":"2024-04-23T10:24:43.025324+0000","objects_scrubbed":0,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T14:57:48.153094+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,4],"acting":[2,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.22","version":"89'8","reported_seq":27,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-23T10:26:40.459657+0000","last_change":"2024-04-23T10:24:44.055534+0000","last_active":"2024-04-23T10:26:40.459657+0000","last_peered":"2024-04-23T10:26:40.459657+0000","last_clean":"2024-04-23T10:26:40.459657+0000","last_became_active":"2024-04-23T10:24:44.055181+0000","last_became_peered":"2024-04-23T10:24:44.055181+0000","last_unstale":"2024-04-23T10:26:40.459657+0000","last_undegraded":"2024-04-23T10:26:40.459657+0000","last_fullsized":"2024-04-23T10:26:40.459657+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T10:24:43.025324+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T10:24:43.025324+0000","last_clean_scrub_stamp":"2024-04-23T10:24:43.025324+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T13:56:33.878232+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.23","version":"89'7","reported_seq":32,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-23T10:26:40.460085+0000","last_change":"2024-04-23T10:25:47.587380+0000","last_active":"2024-04-23T10:26:40.460085+0000","last_peered":"2024-04-23T10:26:40.460085+0000","last_clean":"2024-04-23T10:26:40.460085+0000","last_became_active":"2024-04-23T10:25:47.586893+0000","last_became_peered":"2024-04-23T10:25:47.586893+0000","last_unstale":"2024-04-23T10:26:40.460085+0000","last_undegraded":"2024-04-23T10:26:40.460085+0000","last_fullsized":"2024-04-23T10:26:40.460085+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T10:24:43.025324+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T10:24:43.025324+0000","last_clean_scrub_stamp":"2024-04-23T10:24:43.025324+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T10:51:34.390594+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,7],"acting":[0,6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.24","version":"89'4","reported_seq":23,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-23T10:26:40.278003+0000","last_change":"2024-04-23T10:24:44.050741+0000","last_active":"2024-04-23T10:26:40.278003+0000","last_peered":"2024-04-23T10:26:40.278003+0000","last_clean":"2024-04-23T10:26:40.278003+0000","last_became_active":"2024-04-23T10:24:44.050398+0000","last_became_peered":"2024-04-23T10:24:44.050398+0000","last_unstale":"2024-04-23T10:26:40.278003+0000","last_undegraded":"2024-04-23T10:26:40.278003+0000","last_fullsized":"2024-04-23T10:26:40.278003+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T10:24:43.025324+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T10:24:43.025324+0000","last_clean_scrub_stamp":"2024-04-23T10:24:43.025324+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T10:49:07.146673+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,0],"acting":[6,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.25","version":"89'6","reported_seq":25,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-23T10:26:40.460199+0000","last_change":"2024-04-23T10:24:44.048933+0000","last_active":"2024-04-23T10:26:40.460199+0000","last_peered":"2024-04-23T10:26:40.460199+0000","last_clean":"2024-04-23T10:26:40.460199+0000","last_became_active":"2024-04-23T10:24:44.048553+0000","last_became_peered":"2024-04-23T10:24:44.048553+0000","last_unstale":"2024-04-23T10:26:40.460199+0000","last_undegraded":"2024-04-23T10:26:40.460199+0000","last_fullsized":"2024-04-23T10:26:40.460199+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T10:24:43.025324+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T10:24:43.025324+0000","last_clean_scrub_stamp":"2024-04-23T10:24:43.025324+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T10:40:24.569906+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,8,5],"acting":[11,8,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.26","version":"89'15","reported_seq":34,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-23T10:26:40.460160+0000","last_change":"2024-04-23T10:24:44.051895+0000","last_active":"2024-04-23T10:26:40.460160+0000","last_peered":"2024-04-23T10:26:40.460160+0000","last_clean":"2024-04-23T10:26:40.460160+0000","last_became_active":"2024-04-23T10:24:44.051005+0000","last_became_peered":"2024-04-23T10:24:44.051005+0000","last_unstale":"2024-04-23T10:26:40.460160+0000","last_undegraded":"2024-04-23T10:26:40.460160+0000","last_fullsized":"2024-04-23T10:26:40.460160+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T10:24:43.025324+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T10:24:43.025324+0000","last_clean_scrub_stamp":"2024-04-23T10:24:43.025324+0000","objects_scrubbed":0,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T18:13:05.227365+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,8,1],"acting":[2,8,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.27","version":"89'2","reported_seq":27,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-23T10:26:40.374327+0000","last_change":"2024-04-23T10:25:47.587504+0000","last_active":"2024-04-23T10:26:40.374327+0000","last_peered":"2024-04-23T10:26:40.374327+0000","last_clean":"2024-04-23T10:26:40.374327+0000","last_became_active":"2024-04-23T10:25:47.587157+0000","last_became_peered":"2024-04-23T10:25:47.587157+0000","last_unstale":"2024-04-23T10:26:40.374327+0000","last_undegraded":"2024-04-23T10:26:40.374327+0000","last_fullsized":"2024-04-23T10:26:40.374327+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T10:24:43.025324+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T10:24:43.025324+0000","last_clean_scrub_stamp":"2024-04-23T10:24:43.025324+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T18:46:25.090568+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,2],"acting":[3,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.28","version":"89'6","reported_seq":25,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-23T10:26:40.281576+0000","last_change":"2024-04-23T10:24:44.044158+0000","last_active":"2024-04-23T10:26:40.281576+0000","last_peered":"2024-04-23T10:26:40.281576+0000","last_clean":"2024-04-23T10:26:40.281576+0000","last_became_active":"2024-04-23T10:24:44.043829+0000","last_became_peered":"2024-04-23T10:24:44.043829+0000","last_unstale":"2024-04-23T10:26:40.281576+0000","last_undegraded":"2024-04-23T10:26:40.281576+0000","last_fullsized":"2024-04-23T10:26:40.281576+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T10:24:43.025324+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T10:24:43.025324+0000","last_clean_scrub_stamp":"2024-04-23T10:24:43.025324+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T15:10:08.081831+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,9,3],"acting":[5,9,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.29","version":"89'8","reported_seq":27,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-23T10:26:40.373723+0000","last_change":"2024-04-23T10:24:44.051267+0000","last_active":"2024-04-23T10:26:40.373723+0000","last_peered":"2024-04-23T10:26:40.373723+0000","last_clean":"2024-04-23T10:26:40.373723+0000","last_became_active":"2024-04-23T10:24:44.050911+0000","last_became_peered":"2024-04-23T10:24:44.050911+0000","last_unstale":"2024-04-23T10:26:40.373723+0000","last_undegraded":"2024-04-23T10:26:40.373723+0000","last_fullsized":"2024-04-23T10:26:40.373723+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T10:24:43.025324+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T10:24:43.025324+0000","last_clean_scrub_stamp":"2024-04-23T10:24:43.025324+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T18:09:28.871427+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,11],"acting":[3,2,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.2a","version":"89'10","reported_seq":29,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-23T10:26:40.281065+0000","last_change":"2024-04-23T10:24:44.057653+0000","last_active":"2024-04-23T10:26:40.281065+0000","last_peered":"2024-04-23T10:26:40.281065+0000","last_clean":"2024-04-23T10:26:40.281065+0000","last_became_active":"2024-04-23T10:24:44.056627+0000","last_became_peered":"2024-04-23T10:24:44.056627+0000","last_unstale":"2024-04-23T10:26:40.281065+0000","last_undegraded":"2024-04-23T10:26:40.281065+0000","last_fullsized":"2024-04-23T10:26:40.281065+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T10:24:43.025324+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T10:24:43.025324+0000","last_clean_scrub_stamp":"2024-04-23T10:24:43.025324+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T18:29:21.480187+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,3],"acting":[1,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.2b","version":"89'11","reported_seq":30,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-23T10:26:40.278095+0000","last_change":"2024-04-23T10:24:44.059735+0000","last_active":"2024-04-23T10:26:40.278095+0000","last_peered":"2024-04-23T10:26:40.278095+0000","last_clean":"2024-04-23T10:26:40.278095+0000","last_became_active":"2024-04-23T10:24:44.059052+0000","last_became_peered":"2024-04-23T10:24:44.059052+0000","last_unstale":"2024-04-23T10:26:40.278095+0000","last_undegraded":"2024-04-23T10:26:40.278095+0000","last_fullsized":"2024-04-23T10:26:40.278095+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T10:24:43.025324+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T10:24:43.025324+0000","last_clean_scrub_stamp":"2024-04-23T10:24:43.025324+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T14:29:25.794655+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,0],"acting":[6,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.2c","version":"89'8","reported_seq":27,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-23T10:26:40.459891+0000","last_change":"2024-04-23T10:24:44.051831+0000","last_active":"2024-04-23T10:26:40.459891+0000","last_peered":"2024-04-23T10:26:40.459891+0000","last_clean":"2024-04-23T10:26:40.459891+0000","last_became_active":"2024-04-23T10:24:44.051480+0000","last_became_peered":"2024-04-23T10:24:44.051480+0000","last_unstale":"2024-04-23T10:26:40.459891+0000","last_undegraded":"2024-04-23T10:26:40.459891+0000","last_fullsized":"2024-04-23T10:26:40.459891+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T10:24:43.025324+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T10:24:43.025324+0000","last_clean_scrub_stamp":"2024-04-23T10:24:43.025324+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T19:20:55.181393+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,4,8],"acting":[11,4,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]}],"pool_stats":[{"poolid":3,"num_pg":64,"stat_sum":{"num_bytes":234,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":702,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":502,"ondisk_log_size":502,"up":192,"acting":192,"num_store_stats":12},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":136,"num_read_kb":119,"num_write":181,"num_write_kb":2938,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":1180736,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1781760,"data_stored":1771104,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":137,"ondisk_log_size":137,"up":3,"acting":3,"num_store_stats":3},{"poolid":2,"num_pg":32,"stat_sum":{"num_bytes":598030,"num_objects":62,"num_object_clones":0,"num_object_copies":186,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":62,"num_whiteouts":0,"num_read":802,"num_read_kb":510,"num_write":285,"num_write_kb":1437,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":10,"num_objects_omap":40,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":2027520,"data_stored":1794090,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":226,"ondisk_log_size":226,"up":96,"acting":96,"num_store_stats":12}],"osd_stats":[{"osd":11,"up_from":82,"seq":352187318309,"num_pgs":19,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32148,"kb_used_data":968,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93713004,"statfs":{"total":95995035648,"available":95962116096,"internally_reserved":0,"allocated":991232,"data_stored":474591,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1596,"internal_metadata":31914436},"hb_peers":[0,1,2,4,5,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":10,"up_from":74,"seq":317827579945,"num_pgs":26,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28656,"kb_used_data":1640,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93716496,"statfs":{"total":95995035648,"available":95965691904,"internally_reserved":0,"allocated":1679360,"data_stored":1124595,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1600,"internal_metadata":27654592},"hb_peers":[0,1,2,3,4,5,6,8,9,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":9,"up_from":69,"seq":296352743471,"num_pgs":20,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28196,"kb_used_data":1176,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93716956,"statfs":{"total":95995035648,"available":95966162944,"internally_reserved":0,"allocated":1204224,"data_stored":671435,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1600,"internal_metadata":27654592},"hb_peers":[0,1,2,3,4,5,6,7,8,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":8,"up_from":62,"seq":266287972405,"num_pgs":25,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32548,"kb_used_data":1368,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93712604,"statfs":{"total":95995035648,"available":95961706496,"internally_reserved":0,"allocated":1400832,"data_stored":867963,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,2,3,4,5,6,7,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":7,"up_from":53,"seq":227633266745,"num_pgs":26,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":36904,"kb_used_data":1564,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":93708248,"statfs":{"total":95995035648,"available":95957245952,"internally_reserved":0,"allocated":1601536,"data_stored":1068132,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":36174282},"hb_peers":[1,2,3,4,5,6,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":6,"up_from":47,"seq":201863462975,"num_pgs":22,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32152,"kb_used_data":972,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93713000,"statfs":{"total":95995035648,"available":95962112000,"internally_reserved":0,"allocated":995328,"data_stored":474216,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":31914438},"hb_peers":[0,1,2,3,4,5,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":13,"seq":55834574934,"num_pgs":24,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28060,"kb_used_data":1040,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717092,"statfs":{"total":95995035648,"available":95966302208,"internally_reserved":0,"allocated":1064960,"data_stored":541767,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,2,3,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":8,"seq":34359738460,"num_pgs":25,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":36332,"kb_used_data":992,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":93708820,"statfs":{"total":95995035648,"available":95957831680,"internally_reserved":0,"allocated":1015808,"data_stored":475487,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":36174278},"hb_peers":[1,2,3,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":18,"seq":77309411410,"num_pgs":27,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":36404,"kb_used_data":1064,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":93708748,"statfs":{"total":95995035648,"available":95957757952,"internally_reserved":0,"allocated":1089536,"data_stored":544040,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":36174277},"hb_peers":[0,1,3,4,5,6,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":25,"seq":107374182474,"num_pgs":26,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":27988,"kb_used_data":972,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717164,"statfs":{"total":95995035648,"available":95966375936,"internally_reserved":0,"allocated":995328,"data_stored":465542,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[0,1,2,4,5,6,7,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":4,"up_from":33,"seq":141733920839,"num_pgs":23,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":27996,"kb_used_data":976,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717156,"statfs":{"total":95995035648,"available":95966367744,"internally_reserved":0,"allocated":999424,"data_stored":475185,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[1,2,3,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":39,"seq":167503724611,"num_pgs":28,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":37044,"kb_used_data":1708,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":93708108,"statfs":{"total":95995035648,"available":95957102592,"internally_reserved":0,"allocated":1748992,"data_stored":1198493,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":36174277},"hb_peers":[0,1,2,3,4,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":28672,"data_stored":1478,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":77824,"data_stored":67758,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":102400,"data_stored":70031,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":544768,"data_stored":525100,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":1567,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":163840,"data_stored":133999,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":90,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":3638,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":413696,"data_stored":393954,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":413696,"data_stored":394034,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":245760,"data_stored":201859,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":582,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":234,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":117,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":117,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":117,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":117,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-04-23T10:26:45.458 DEBUG:teuthology.orchestra.run.smithi114:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-04-23T10:26:45.492 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 23 10:26:45 smithi119 ceph-mon[32608]: Adjusting osd_memory_target on smithi119 to 176.2M 2024-04-23T10:26:45.492 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 23 10:26:45 smithi119 ceph-mon[32608]: Unable to set osd_memory_target on smithi119 to 184770252: error parsing value: Value '184770252' is below minimum 939524096 2024-04-23T10:26:45.492 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 23 10:26:45 smithi119 ceph-mon[32608]: Adjusting osd_memory_target on smithi204 to 176.2M 2024-04-23T10:26:45.492 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 23 10:26:45 smithi119 ceph-mon[32608]: Unable to set osd_memory_target on smithi204 to 184759500: error parsing value: Value '184759500' is below minimum 939524096 2024-04-23T10:26:45.492 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 23 10:26:45 smithi119 ceph-mon[32608]: Adjusting osd_memory_target on smithi114 to 176.2M 2024-04-23T10:26:45.492 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 23 10:26:45 smithi119 ceph-mon[32608]: Unable to set osd_memory_target on smithi114 to 184769536: error parsing value: Value '184769536' is below minimum 939524096 2024-04-23T10:26:45.492 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 23 10:26:45 smithi119 ceph-mon[32608]: Updating smithi114:/etc/ceph/ceph.conf 2024-04-23T10:26:45.492 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 23 10:26:45 smithi119 ceph-mon[32608]: Updating smithi119:/etc/ceph/ceph.conf 2024-04-23T10:26:45.492 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 23 10:26:45 smithi119 ceph-mon[32608]: Updating smithi204:/etc/ceph/ceph.conf 2024-04-23T10:26:45.493 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 23 10:26:45 smithi119 ceph-mon[32608]: Updating smithi204:/var/lib/ceph/aa240572-0159-11ef-bc93-c7b262605968/config/ceph.conf 2024-04-23T10:26:45.493 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 23 10:26:45 smithi119 ceph-mon[32608]: Updating smithi114:/var/lib/ceph/aa240572-0159-11ef-bc93-c7b262605968/config/ceph.conf 2024-04-23T10:26:45.493 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 23 10:26:45 smithi119 ceph-mon[32608]: Updating smithi119:/var/lib/ceph/aa240572-0159-11ef-bc93-c7b262605968/config/ceph.conf 2024-04-23T10:26:45.493 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 23 10:26:45 smithi119 ceph-mon[32608]: from='client.? 172.21.15.114:0/2609718743' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-23T10:26:45.493 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 23 10:26:45 smithi119 ceph-mon[32608]: from='client.? 172.21.15.114:0/2324190627' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-04-23T10:26:45.493 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 23 10:26:45 smithi119 ceph-mon[32608]: from='mgr.15291 172.21.15.114:0/1095789187' entity='mgr.x' 2024-04-23T10:26:45.493 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 23 10:26:45 smithi119 ceph-mon[32608]: from='mgr.15291 172.21.15.114:0/1095789187' entity='mgr.x' 2024-04-23T10:26:45.493 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 23 10:26:45 smithi119 ceph-mon[32608]: from='mgr.15291 172.21.15.114:0/1095789187' entity='mgr.x' 2024-04-23T10:26:45.493 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 23 10:26:45 smithi119 ceph-mon[32608]: from='mgr.15291 172.21.15.114:0/1095789187' entity='mgr.x' 2024-04-23T10:26:45.493 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 23 10:26:45 smithi119 ceph-mon[32608]: from='mgr.15291 172.21.15.114:0/1095789187' entity='mgr.x' 2024-04-23T10:26:45.493 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 23 10:26:45 smithi119 ceph-mon[32608]: from='mgr.15291 172.21.15.114:0/1095789187' entity='mgr.x' 2024-04-23T10:26:45.493 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 23 10:26:45 smithi119 ceph-mon[32608]: from='mgr.15291 172.21.15.114:0/1095789187' entity='mgr.x' 2024-04-23T10:26:45.493 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 23 10:26:45 smithi119 ceph-mon[32608]: from='client.? 172.21.15.114:0/2624372018' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2024-04-23T10:26:45.493 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 23 10:26:45 smithi119 ceph-mon[32608]: from='mgr.15291 172.21.15.114:0/1095789187' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-23T10:26:45.493 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 23 10:26:45 smithi119 ceph-mon[32608]: from='mgr.15291 172.21.15.114:0/1095789187' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-23T10:26:45.494 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 23 10:26:45 smithi119 ceph-mon[32608]: from='mgr.15291 172.21.15.114:0/1095789187' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-23T10:26:45.494 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 23 10:26:45 smithi119 ceph-mon[32608]: from='mgr.15291 172.21.15.114:0/1095789187' entity='mgr.x' 2024-04-23T10:26:45.596 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:45.597+0000 7f4f7bb5d640 1 -- 172.21.15.114:0/886844209 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f4f7410c520 msgr2=0x7f4f7410e9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T10:26:45.596 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:45.597+0000 7f4f7bb5d640 1 --2- 172.21.15.114:0/886844209 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f4f7410c520 0x7f4f7410e9c0 secure :-1 s=READY pgs=446 cs=0 l=1 rev1=1 crypto rx=0x7f4f6c009ca0 tx=0x7f4f6c02a9e0 comp rx=0 tx=0).stop 2024-04-23T10:26:45.596 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:45.597+0000 7f4f7bb5d640 1 -- 172.21.15.114:0/886844209 shutdown_connections 2024-04-23T10:26:45.596 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:45.597+0000 7f4f7bb5d640 1 --2- 172.21.15.114:0/886844209 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f4f7410c520 0x7f4f7410e9c0 unknown :-1 s=CLOSED pgs=446 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:26:45.596 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:45.597+0000 7f4f7bb5d640 1 --2- 172.21.15.114:0/886844209 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f4f74103b00 0x7f4f7410bfe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:26:45.596 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:45.597+0000 7f4f7bb5d640 1 --2- 172.21.15.114:0/886844209 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f4f741031e0 0x7f4f741035c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:26:45.596 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:45.597+0000 7f4f7bb5d640 1 -- 172.21.15.114:0/886844209 >> 172.21.15.114:0/886844209 conn(0x7f4f74077f50 msgr2=0x7f4f74078360 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T10:26:45.596 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:45.597+0000 7f4f7bb5d640 1 -- 172.21.15.114:0/886844209 shutdown_connections 2024-04-23T10:26:45.596 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:45.597+0000 7f4f7bb5d640 1 -- 172.21.15.114:0/886844209 wait complete. 2024-04-23T10:26:45.599 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:45.598+0000 7f4f7bb5d640 1 Processor -- start 2024-04-23T10:26:45.599 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:45.598+0000 7f4f7bb5d640 1 -- start start 2024-04-23T10:26:45.599 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:45.598+0000 7f4f7bb5d640 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f4f741031e0 0x7f4f741986d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T10:26:45.599 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:45.598+0000 7f4f7bb5d640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f4f74103b00 0x7f4f74198c10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T10:26:45.599 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:45.598+0000 7f4f7bb5d640 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f4f7410c520 0x7f4f7419cf90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T10:26:45.599 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:45.598+0000 7f4f7bb5d640 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x7f4f741121b0 con 0x7f4f74103b00 2024-04-23T10:26:45.599 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:45.598+0000 7f4f7bb5d640 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 -- 0x7f4f74112030 con 0x7f4f741031e0 2024-04-23T10:26:45.599 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:45.598+0000 7f4f7bb5d640 1 -- --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_getmap magic: 0 -- 0x7f4f74112330 con 0x7f4f7410c520 2024-04-23T10:26:45.599 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:45.598+0000 7f4f790f8640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f4f74103b00 0x7f4f74198c10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T10:26:45.599 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:45.598+0000 7f4f790f8640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f4f74103b00 0x7f4f74198c10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.114:51862/0 (socket says 172.21.15.114:51862) 2024-04-23T10:26:45.599 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:45.598+0000 7f4f790f8640 1 -- 172.21.15.114:0/4271360291 learned_addr learned my addr 172.21.15.114:0/4271360291 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-04-23T10:26:45.600 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:45.598+0000 7f4f7a0fa640 1 --2- 172.21.15.114:0/4271360291 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f4f7410c520 0x7f4f7419cf90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T10:26:45.600 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:45.598+0000 7f4f798f9640 1 --2- 172.21.15.114:0/4271360291 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f4f741031e0 0x7f4f741986d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T10:26:45.600 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:45.598+0000 7f4f790f8640 1 -- 172.21.15.114:0/4271360291 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f4f7410c520 msgr2=0x7f4f7419cf90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T10:26:45.600 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:45.598+0000 7f4f790f8640 1 --2- 172.21.15.114:0/4271360291 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f4f7410c520 0x7f4f7419cf90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:26:45.600 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:45.598+0000 7f4f790f8640 1 -- 172.21.15.114:0/4271360291 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f4f741031e0 msgr2=0x7f4f741986d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T10:26:45.600 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:45.598+0000 7f4f790f8640 1 --2- 172.21.15.114:0/4271360291 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f4f741031e0 0x7f4f741986d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:26:45.600 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:45.598+0000 7f4f790f8640 1 -- 172.21.15.114:0/4271360291 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f4f7419d6c0 con 0x7f4f74103b00 2024-04-23T10:26:45.600 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:45.598+0000 7f4f798f9640 1 --2- 172.21.15.114:0/4271360291 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f4f741031e0 0x7f4f741986d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-23T10:26:45.600 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:45.598+0000 7f4f790f8640 1 --2- 172.21.15.114:0/4271360291 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f4f74103b00 0x7f4f74198c10 secure :-1 s=READY pgs=447 cs=0 l=1 rev1=1 crypto rx=0x7f4f7000e9f0 tx=0x7f4f7000edf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T10:26:45.600 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:45.598+0000 7f4f6affd640 1 -- 172.21.15.114:0/4271360291 <== mon.0 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4f7000cd80 con 0x7f4f74103b00 2024-04-23T10:26:45.600 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:45.599+0000 7f4f7bb5d640 1 -- 172.21.15.114:0/4271360291 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f4f7419d9b0 con 0x7f4f74103b00 2024-04-23T10:26:45.600 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:45.599+0000 7f4f6affd640 1 -- 172.21.15.114:0/4271360291 <== mon.0 v2:172.21.15.114:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f4f700054d0 con 0x7f4f74103b00 2024-04-23T10:26:45.601 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:45.599+0000 7f4f6affd640 1 -- 172.21.15.114:0/4271360291 <== mon.0 v2:172.21.15.114:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4f70010690 con 0x7f4f74103b00 2024-04-23T10:26:45.601 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:45.599+0000 7f4f7bb5d640 1 -- 172.21.15.114:0/4271360291 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f4f741a5240 con 0x7f4f74103b00 2024-04-23T10:26:45.601 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:45.600+0000 7f4f7bb5d640 1 -- 172.21.15.114:0/4271360291 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f4f74068d70 con 0x7f4f74103b00 2024-04-23T10:26:45.601 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:45.601+0000 7f4f6affd640 1 -- 172.21.15.114:0/4271360291 <== mon.0 v2:172.21.15.114:3300/0 4 ==== mgrmap(e 24) ==== 138823+0+0 (secure 0 0 0) 0x7f4f70004e90 con 0x7f4f74103b00 2024-04-23T10:26:45.601 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:45.601+0000 7f4f6affd640 1 --2- 172.21.15.114:0/4271360291 >> [v2:172.21.15.114:6800/1395693466,v1:172.21.15.114:6801/1395693466] conn(0x7f4f600a3db0 0x7f4f600a6270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T10:26:45.602 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:45.602+0000 7f4f798f9640 1 --2- 172.21.15.114:0/4271360291 >> [v2:172.21.15.114:6800/1395693466,v1:172.21.15.114:6801/1395693466] conn(0x7f4f600a3db0 0x7f4f600a6270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T10:26:45.602 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:45.602+0000 7f4f6affd640 1 -- 172.21.15.114:0/4271360291 <== mon.0 v2:172.21.15.114:3300/0 5 ==== osd_map(90..90 src has 1..90) ==== 8527+0+0 (secure 0 0 0) 0x7f4f700d0dd0 con 0x7f4f74103b00 2024-04-23T10:26:45.602 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:45.602+0000 7f4f798f9640 1 --2- 172.21.15.114:0/4271360291 >> [v2:172.21.15.114:6800/1395693466,v1:172.21.15.114:6801/1395693466] conn(0x7f4f600a3db0 0x7f4f600a6270 secure :-1 s=READY pgs=49 cs=0 l=1 rev1=1 crypto rx=0x7f4f640058c0 tx=0x7f4f64005850 comp rx=0 tx=0).ready entity=mgr.15291 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T10:26:45.605 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:45.606+0000 7f4f6affd640 1 -- 172.21.15.114:0/4271360291 <== mon.0 v2:172.21.15.114:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f4f7009d300 con 0x7f4f74103b00 2024-04-23T10:26:45.783 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:45.785+0000 7f4f7bb5d640 1 -- 172.21.15.114:0/4271360291 --> [v2:172.21.15.114:6800/1395693466,v1:172.21.15.114:6801/1395693466] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) -- 0x7f4f74062390 con 0x7f4f600a3db0 2024-04-23T10:26:45.790 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:45.792+0000 7f4f6affd640 1 -- 172.21.15.114:0/4271360291 <== mgr.15291 v2:172.21.15.114:6800/1395693466 1 ==== mgr_command_reply(tid 0: 0 dumped all) ==== 18+0+257730 (secure 0 0 0) 0x7f4f74062390 con 0x7f4f600a3db0 2024-04-23T10:26:45.793 INFO:teuthology.orchestra.run.smithi114.stdout: 2024-04-23T10:26:45.794 INFO:teuthology.orchestra.run.smithi114.stderr:dumped all 2024-04-23T10:26:45.796 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:45.795+0000 7f4f7bb5d640 1 -- 172.21.15.114:0/4271360291 >> [v2:172.21.15.114:6800/1395693466,v1:172.21.15.114:6801/1395693466] conn(0x7f4f600a3db0 msgr2=0x7f4f600a6270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T10:26:45.796 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:45.795+0000 7f4f7bb5d640 1 --2- 172.21.15.114:0/4271360291 >> [v2:172.21.15.114:6800/1395693466,v1:172.21.15.114:6801/1395693466] conn(0x7f4f600a3db0 0x7f4f600a6270 secure :-1 s=READY pgs=49 cs=0 l=1 rev1=1 crypto rx=0x7f4f640058c0 tx=0x7f4f64005850 comp rx=0 tx=0).stop 2024-04-23T10:26:45.796 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:45.795+0000 7f4f7bb5d640 1 -- 172.21.15.114:0/4271360291 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f4f74103b00 msgr2=0x7f4f74198c10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T10:26:45.796 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:45.795+0000 7f4f7bb5d640 1 --2- 172.21.15.114:0/4271360291 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f4f74103b00 0x7f4f74198c10 secure :-1 s=READY pgs=447 cs=0 l=1 rev1=1 crypto rx=0x7f4f7000e9f0 tx=0x7f4f7000edf0 comp rx=0 tx=0).stop 2024-04-23T10:26:45.796 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:45.795+0000 7f4f7bb5d640 1 -- 172.21.15.114:0/4271360291 shutdown_connections 2024-04-23T10:26:45.796 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:45.795+0000 7f4f7bb5d640 1 --2- 172.21.15.114:0/4271360291 >> [v2:172.21.15.114:6800/1395693466,v1:172.21.15.114:6801/1395693466] conn(0x7f4f600a3db0 0x7f4f600a6270 unknown :-1 s=CLOSED pgs=49 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:26:45.796 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:45.795+0000 7f4f7bb5d640 1 --2- 172.21.15.114:0/4271360291 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f4f7410c520 0x7f4f7419cf90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:26:45.796 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:45.795+0000 7f4f7bb5d640 1 --2- 172.21.15.114:0/4271360291 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f4f74103b00 0x7f4f74198c10 unknown :-1 s=CLOSED pgs=447 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:26:45.796 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:45.795+0000 7f4f7bb5d640 1 --2- 172.21.15.114:0/4271360291 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f4f741031e0 0x7f4f741986d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:26:45.796 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:45.795+0000 7f4f7bb5d640 1 -- 172.21.15.114:0/4271360291 >> 172.21.15.114:0/4271360291 conn(0x7f4f74077f50 msgr2=0x7f4f740fdf40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T10:26:45.796 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:45.795+0000 7f4f7bb5d640 1 -- 172.21.15.114:0/4271360291 shutdown_connections 2024-04-23T10:26:45.796 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:45.795+0000 7f4f7bb5d640 1 -- 172.21.15.114:0/4271360291 wait complete. 2024-04-23T10:26:45.805 INFO:teuthology.orchestra.run.smithi114.stdout:{"pg_ready":true,"pg_map":{"version":5,"stamp":"2024-04-23T10:26:44.290805+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":1188632,"num_objects":66,"num_object_clones":0,"num_object_copies":198,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":66,"num_whiteouts":0,"num_read":938,"num_read_kb":629,"num_write":468,"num_write_kb":4377,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":7,"num_bytes_recovered":1180736,"num_keys_recovered":10,"num_objects_omap":40,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":865,"ondisk_log_size":865,"up":291,"acting":291,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":291,"num_osds":12,"num_per_pool_osds":12,"num_per_pool_omap_osds":12,"kb":1124941824,"kb_used":384428,"kb_used_data":14440,"kb_used_omap":18,"kb_used_meta":369837,"kb_avail":1124557396,"statfs":{"total":1151940427776,"available":1151546773504,"internally_reserved":0,"allocated":14786560,"data_stored":8381446,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":19142,"internal_metadata":378713402},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"2.989676"},"pg_stats":[{"pgid":"3.3f","version":"89'8","reported_seq":27,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-23T10:26:40.460272+0000","last_change":"2024-04-23T10:24:44.051465+0000","last_active":"2024-04-23T10:26:40.460272+0000","last_peered":"2024-04-23T10:26:40.460272+0000","last_clean":"2024-04-23T10:26:40.460272+0000","last_became_active":"2024-04-23T10:24:44.050863+0000","last_became_peered":"2024-04-23T10:24:44.050863+0000","last_unstale":"2024-04-23T10:26:40.460272+0000","last_undegraded":"2024-04-23T10:26:40.460272+0000","last_fullsized":"2024-04-23T10:26:40.460272+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T10:24:43.025324+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T10:24:43.025324+0000","last_clean_scrub_stamp":"2024-04-23T10:24:43.025324+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T15:11:12.508458+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,1,0],"acting":[11,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.3e","version":"89'4","reported_seq":23,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-23T10:26:40.459824+0000","last_change":"2024-04-23T10:24:44.050739+0000","last_active":"2024-04-23T10:26:40.459824+0000","last_peered":"2024-04-23T10:26:40.459824+0000","last_clean":"2024-04-23T10:26:40.459824+0000","last_became_active":"2024-04-23T10:24:44.050430+0000","last_became_peered":"2024-04-23T10:24:44.050430+0000","last_unstale":"2024-04-23T10:26:40.459824+0000","last_undegraded":"2024-04-23T10:26:40.459824+0000","last_fullsized":"2024-04-23T10:26:40.459824+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T10:24:43.025324+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T10:24:43.025324+0000","last_clean_scrub_stamp":"2024-04-23T10:24:43.025324+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T17:36:25.983147+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,11,10],"acting":[0,11,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3d","version":"89'7","reported_seq":26,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-23T10:26:40.460358+0000","last_change":"2024-04-23T10:24:44.054805+0000","last_active":"2024-04-23T10:26:40.460358+0000","last_peered":"2024-04-23T10:26:40.460358+0000","last_clean":"2024-04-23T10:26:40.460358+0000","last_became_active":"2024-04-23T10:24:44.054423+0000","last_became_peered":"2024-04-23T10:24:44.054423+0000","last_unstale":"2024-04-23T10:26:40.460358+0000","last_undegraded":"2024-04-23T10:26:40.460358+0000","last_fullsized":"2024-04-23T10:26:40.460358+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T10:24:43.025324+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T10:24:43.025324+0000","last_clean_scrub_stamp":"2024-04-23T10:24:43.025324+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T21:34:26.793624+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,1],"acting":[0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3c","version":"89'8","reported_seq":27,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-23T10:26:40.459685+0000","last_change":"2024-04-23T10:24:44.055966+0000","last_active":"2024-04-23T10:26:40.459685+0000","last_peered":"2024-04-23T10:26:40.459685+0000","last_clean":"2024-04-23T10:26:40.459685+0000","last_became_active":"2024-04-23T10:24:44.055028+0000","last_became_peered":"2024-04-23T10:24:44.055028+0000","last_unstale":"2024-04-23T10:26:40.459685+0000","last_undegraded":"2024-04-23T10:26:40.459685+0000","last_fullsized":"2024-04-23T10:26:40.459685+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T10:24:43.025324+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T10:24:43.025324+0000","last_clean_scrub_stamp":"2024-04-23T10:24:43.025324+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T19:41:04.594025+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,8,2],"acting":[4,8,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.3b","version":"89'4","reported_seq":23,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-23T10:26:40.373641+0000","last_change":"2024-04-23T10:24:44.046954+0000","last_active":"2024-04-23T10:26:40.373641+0000","last_peered":"2024-04-23T10:26:40.373641+0000","last_clean":"2024-04-23T10:26:40.373641+0000","last_became_active":"2024-04-23T10:24:44.046631+0000","last_became_peered":"2024-04-23T10:24:44.046631+0000","last_unstale":"2024-04-23T10:26:40.373641+0000","last_undegraded":"2024-04-23T10:26:40.373641+0000","last_fullsized":"2024-04-23T10:26:40.373641+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T10:24:43.025324+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T10:24:43.025324+0000","last_clean_scrub_stamp":"2024-04-23T10:24:43.025324+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T15:04:26.878894+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,3,6],"acting":[8,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.3a","version":"89'6","reported_seq":25,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-23T10:26:40.381329+0000","last_change":"2024-04-23T10:24:44.051550+0000","last_active":"2024-04-23T10:26:40.381329+0000","last_peered":"2024-04-23T10:26:40.381329+0000","last_clean":"2024-04-23T10:26:40.381329+0000","last_became_active":"2024-04-23T10:24:44.050469+0000","last_became_peered":"2024-04-23T10:24:44.050469+0000","last_unstale":"2024-04-23T10:26:40.381329+0000","last_undegraded":"2024-04-23T10:26:40.381329+0000","last_fullsized":"2024-04-23T10:26:40.381329+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T10:24:43.025324+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T10:24:43.025324+0000","last_clean_scrub_stamp":"2024-04-23T10:24:43.025324+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T21:13:42.469252+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,9,4],"acting":[10,9,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.39","version":"89'5","reported_seq":30,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-23T10:26:40.521664+0000","last_change":"2024-04-23T10:25:47.588865+0000","last_active":"2024-04-23T10:26:40.521664+0000","last_peered":"2024-04-23T10:26:40.521664+0000","last_clean":"2024-04-23T10:26:40.521664+0000","last_became_active":"2024-04-23T10:25:47.588571+0000","last_became_peered":"2024-04-23T10:25:47.588571+0000","last_unstale":"2024-04-23T10:26:40.521664+0000","last_undegraded":"2024-04-23T10:26:40.521664+0000","last_fullsized":"2024-04-23T10:26:40.521664+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T10:24:43.025324+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T10:24:43.025324+0000","last_clean_scrub_stamp":"2024-04-23T10:24:43.025324+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T17:14:00.026999+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,9,8],"acting":[7,9,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.38","version":"89'9","reported_seq":22,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-23T10:26:40.371622+0000","last_change":"2024-04-23T10:25:47.722416+0000","last_active":"2024-04-23T10:26:40.371622+0000","last_peered":"2024-04-23T10:26:40.371622+0000","last_clean":"2024-04-23T10:26:40.371622+0000","last_became_active":"2024-04-23T10:25:47.722015+0000","last_became_peered":"2024-04-23T10:25:47.722015+0000","last_unstale":"2024-04-23T10:26:40.371622+0000","last_undegraded":"2024-04-23T10:26:40.371622+0000","last_fullsized":"2024-04-23T10:26:40.371622+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T10:24:43.025324+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T10:24:43.025324+0000","last_clean_scrub_stamp":"2024-04-23T10:24:43.025324+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T17:02:57.881164+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,5,10],"acting":[9,5,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.37","version":"89'11","reported_seq":30,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-23T10:26:40.281426+0000","last_change":"2024-04-23T10:24:44.055261+0000","last_active":"2024-04-23T10:26:40.281426+0000","last_peered":"2024-04-23T10:26:40.281426+0000","last_clean":"2024-04-23T10:26:40.281426+0000","last_became_active":"2024-04-23T10:24:44.054405+0000","last_became_peered":"2024-04-23T10:24:44.054405+0000","last_unstale":"2024-04-23T10:26:40.281426+0000","last_undegraded":"2024-04-23T10:26:40.281426+0000","last_fullsized":"2024-04-23T10:26:40.281426+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T10:24:43.025324+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T10:24:43.025324+0000","last_clean_scrub_stamp":"2024-04-23T10:24:43.025324+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T19:30:30.306739+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,11],"acting":[1,5,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.36","version":"89'16","reported_seq":35,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-23T10:26:40.381657+0000","last_change":"2024-04-23T10:24:44.055854+0000","last_active":"2024-04-23T10:26:40.381657+0000","last_peered":"2024-04-23T10:26:40.381657+0000","last_clean":"2024-04-23T10:26:40.381657+0000","last_became_active":"2024-04-23T10:24:44.055275+0000","last_became_peered":"2024-04-23T10:24:44.055275+0000","last_unstale":"2024-04-23T10:26:40.381657+0000","last_undegraded":"2024-04-23T10:26:40.381657+0000","last_fullsized":"2024-04-23T10:26:40.381657+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T10:24:43.025324+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T10:24:43.025324+0000","last_clean_scrub_stamp":"2024-04-23T10:24:43.025324+0000","objects_scrubbed":0,"log_size":16,"log_dups_size":0,"ondisk_log_size":16,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T21:48:25.934878+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,2,4],"acting":[10,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.35","version":"89'6","reported_seq":25,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-23T10:26:40.522323+0000","last_change":"2024-04-23T10:24:44.061314+0000","last_active":"2024-04-23T10:26:40.522323+0000","last_peered":"2024-04-23T10:26:40.522323+0000","last_clean":"2024-04-23T10:26:40.522323+0000","last_became_active":"2024-04-23T10:24:44.060981+0000","last_became_peered":"2024-04-23T10:24:44.060981+0000","last_unstale":"2024-04-23T10:26:40.522323+0000","last_undegraded":"2024-04-23T10:26:40.522323+0000","last_fullsized":"2024-04-23T10:26:40.522323+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T10:24:43.025324+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T10:24:43.025324+0000","last_clean_scrub_stamp":"2024-04-23T10:24:43.025324+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T11:48:08.387575+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,8,1],"acting":[7,8,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.34","version":"89'7","reported_seq":26,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-23T10:26:40.459648+0000","last_change":"2024-04-23T10:24:44.055335+0000","last_active":"2024-04-23T10:26:40.459648+0000","last_peered":"2024-04-23T10:26:40.459648+0000","last_clean":"2024-04-23T10:26:40.459648+0000","last_became_active":"2024-04-23T10:24:44.051257+0000","last_became_peered":"2024-04-23T10:24:44.051257+0000","last_unstale":"2024-04-23T10:26:40.459648+0000","last_undegraded":"2024-04-23T10:26:40.459648+0000","last_fullsized":"2024-04-23T10:26:40.459648+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T10:24:43.025324+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T10:24:43.025324+0000","last_clean_scrub_stamp":"2024-04-23T10:24:43.025324+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T11:57:45.999751+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,10,8],"acting":[4,10,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.33","version":"89'10","reported_seq":29,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-23T10:26:40.278008+0000","last_change":"2024-04-23T10:24:44.052078+0000","last_active":"2024-04-23T10:26:40.278008+0000","last_peered":"2024-04-23T10:26:40.278008+0000","last_clean":"2024-04-23T10:26:40.278008+0000","last_became_active":"2024-04-23T10:24:44.051747+0000","last_became_peered":"2024-04-23T10:24:44.051747+0000","last_unstale":"2024-04-23T10:26:40.278008+0000","last_undegraded":"2024-04-23T10:26:40.278008+0000","last_fullsized":"2024-04-23T10:26:40.278008+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T10:24:43.025324+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T10:24:43.025324+0000","last_clean_scrub_stamp":"2024-04-23T10:24:43.025324+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T19:14:16.495967+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,10,5],"acting":[6,10,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.32","version":"89'7","reported_seq":26,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-23T10:26:40.459810+0000","last_change":"2024-04-23T10:24:44.051994+0000","last_active":"2024-04-23T10:26:40.459810+0000","last_peered":"2024-04-23T10:26:40.459810+0000","last_clean":"2024-04-23T10:26:40.459810+0000","last_became_active":"2024-04-23T10:24:44.051638+0000","last_became_peered":"2024-04-23T10:24:44.051638+0000","last_unstale":"2024-04-23T10:26:40.459810+0000","last_undegraded":"2024-04-23T10:26:40.459810+0000","last_fullsized":"2024-04-23T10:26:40.459810+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T10:24:43.025324+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T10:24:43.025324+0000","last_clean_scrub_stamp":"2024-04-23T10:24:43.025324+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T14:28:30.447198+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,8,10],"acting":[0,8,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.31","version":"89'7","reported_seq":32,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-23T10:26:40.459693+0000","last_change":"2024-04-23T10:25:47.589538+0000","last_active":"2024-04-23T10:26:40.459693+0000","last_peered":"2024-04-23T10:26:40.459693+0000","last_clean":"2024-04-23T10:26:40.459693+0000","last_became_active":"2024-04-23T10:25:47.589172+0000","last_became_peered":"2024-04-23T10:25:47.589172+0000","last_unstale":"2024-04-23T10:26:40.459693+0000","last_undegraded":"2024-04-23T10:26:40.459693+0000","last_fullsized":"2024-04-23T10:26:40.459693+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T10:24:43.025324+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T10:24:43.025324+0000","last_clean_scrub_stamp":"2024-04-23T10:24:43.025324+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T19:47:34.267992+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,8,2],"acting":[0,8,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.30","version":"89'9","reported_seq":28,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-23T10:26:40.280931+0000","last_change":"2024-04-23T10:24:44.057154+0000","last_active":"2024-04-23T10:26:40.280931+0000","last_peered":"2024-04-23T10:26:40.280931+0000","last_clean":"2024-04-23T10:26:40.280931+0000","last_became_active":"2024-04-23T10:24:44.056836+0000","last_became_peered":"2024-04-23T10:24:44.056836+0000","last_unstale":"2024-04-23T10:26:40.280931+0000","last_undegraded":"2024-04-23T10:26:40.280931+0000","last_fullsized":"2024-04-23T10:26:40.280931+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T10:24:43.025324+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T10:24:43.025324+0000","last_clean_scrub_stamp":"2024-04-23T10:24:43.025324+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T13:48:59.107635+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,0],"acting":[1,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.2f","version":"89'8","reported_seq":27,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-23T10:26:40.460422+0000","last_change":"2024-04-23T10:24:44.051175+0000","last_active":"2024-04-23T10:26:40.460422+0000","last_peered":"2024-04-23T10:26:40.460422+0000","last_clean":"2024-04-23T10:26:40.460422+0000","last_became_active":"2024-04-23T10:24:44.050402+0000","last_became_peered":"2024-04-23T10:24:44.050402+0000","last_unstale":"2024-04-23T10:26:40.460422+0000","last_undegraded":"2024-04-23T10:26:40.460422+0000","last_fullsized":"2024-04-23T10:26:40.460422+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T10:24:43.025324+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T10:24:43.025324+0000","last_clean_scrub_stamp":"2024-04-23T10:24:43.025324+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T15:01:26.919701+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,1,0],"acting":[11,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.2e","version":"89'10","reported_seq":29,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-23T10:26:40.371997+0000","last_change":"2024-04-23T10:24:44.053390+0000","last_active":"2024-04-23T10:26:40.371997+0000","last_peered":"2024-04-23T10:26:40.371997+0000","last_clean":"2024-04-23T10:26:40.371997+0000","last_became_active":"2024-04-23T10:24:44.052970+0000","last_became_peered":"2024-04-23T10:24:44.052970+0000","last_unstale":"2024-04-23T10:26:40.371997+0000","last_undegraded":"2024-04-23T10:26:40.371997+0000","last_fullsized":"2024-04-23T10:26:40.371997+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T10:24:43.025324+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T10:24:43.025324+0000","last_clean_scrub_stamp":"2024-04-23T10:24:43.025324+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T13:27:55.704328+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,1,10],"acting":[9,1,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.2d","version":"89'7","reported_seq":26,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-23T10:26:40.460284+0000","last_change":"2024-04-23T10:24:44.049307+0000","last_active":"2024-04-23T10:26:40.460284+0000","last_peered":"2024-04-23T10:26:40.460284+0000","last_clean":"2024-04-23T10:26:40.460284+0000","last_became_active":"2024-04-23T10:24:44.048981+0000","last_became_peered":"2024-04-23T10:24:44.048981+0000","last_unstale":"2024-04-23T10:26:40.460284+0000","last_undegraded":"2024-04-23T10:26:40.460284+0000","last_fullsized":"2024-04-23T10:26:40.460284+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T10:24:43.025324+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T10:24:43.025324+0000","last_clean_scrub_stamp":"2024-04-23T10:24:43.025324+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T12:42:21.804591+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,6],"acting":[2,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.1f","version":"87'9","reported_seq":31,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-23T10:26:40.371868+0000","last_change":"2024-04-23T10:24:43.045804+0000","last_active":"2024-04-23T10:26:40.371868+0000","last_peered":"2024-04-23T10:26:40.371868+0000","last_clean":"2024-04-23T10:26:40.371868+0000","last_became_active":"2024-04-23T10:24:43.045135+0000","last_became_peered":"2024-04-23T10:24:43.045135+0000","last_unstale":"2024-04-23T10:26:40.371868+0000","last_undegraded":"2024-04-23T10:26:40.371868+0000","last_fullsized":"2024-04-23T10:26:40.371868+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T10:24:42.023127+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T10:24:42.023127+0000","last_clean_scrub_stamp":"2024-04-23T10:24:42.023127+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T20:27:28.421644+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,7,8],"acting":[9,7,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.1e","version":"89'5","reported_seq":24,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-23T10:26:40.374223+0000","last_change":"2024-04-23T10:24:44.055008+0000","last_active":"2024-04-23T10:26:40.374223+0000","last_peered":"2024-04-23T10:26:40.374223+0000","last_clean":"2024-04-23T10:26:40.374223+0000","last_became_active":"2024-04-23T10:24:44.054682+0000","last_became_peered":"2024-04-23T10:24:44.054682+0000","last_unstale":"2024-04-23T10:26:40.374223+0000","last_undegraded":"2024-04-23T10:26:40.374223+0000","last_fullsized":"2024-04-23T10:26:40.374223+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T10:24:43.025324+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T10:24:43.025324+0000","last_clean_scrub_stamp":"2024-04-23T10:24:43.025324+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T15:49:58.394958+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,5],"acting":[3,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.1e","version":"87'2","reported_seq":23,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-23T10:26:40.374327+0000","last_change":"2024-04-23T10:24:43.052303+0000","last_active":"2024-04-23T10:26:40.374327+0000","last_peered":"2024-04-23T10:26:40.374327+0000","last_clean":"2024-04-23T10:26:40.374327+0000","last_became_active":"2024-04-23T10:24:43.052010+0000","last_became_peered":"2024-04-23T10:24:43.052010+0000","last_unstale":"2024-04-23T10:26:40.374327+0000","last_undegraded":"2024-04-23T10:26:40.374327+0000","last_fullsized":"2024-04-23T10:26:40.374327+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T10:24:42.023127+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T10:24:42.023127+0000","last_clean_scrub_stamp":"2024-04-23T10:24:42.023127+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T13:03:15.440610+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,5],"acting":[3,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1f","version":"89'6","reported_seq":25,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-23T10:26:40.460053+0000","last_change":"2024-04-23T10:24:44.050749+0000","last_active":"2024-04-23T10:26:40.460053+0000","last_peered":"2024-04-23T10:26:40.460053+0000","last_clean":"2024-04-23T10:26:40.460053+0000","last_became_active":"2024-04-23T10:24:44.049780+0000","last_became_peered":"2024-04-23T10:24:44.049780+0000","last_unstale":"2024-04-23T10:26:40.460053+0000","last_undegraded":"2024-04-23T10:26:40.460053+0000","last_fullsized":"2024-04-23T10:26:40.460053+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T10:24:43.025324+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T10:24:43.025324+0000","last_clean_scrub_stamp":"2024-04-23T10:24:43.025324+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T16:32:37.963776+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,9],"acting":[0,1,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1d","version":"87'4","reported_seq":19,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-23T10:26:40.460147+0000","last_change":"2024-04-23T10:25:47.591939+0000","last_active":"2024-04-23T10:26:40.460147+0000","last_peered":"2024-04-23T10:26:40.460147+0000","last_clean":"2024-04-23T10:26:40.460147+0000","last_became_active":"2024-04-23T10:25:47.586626+0000","last_became_peered":"2024-04-23T10:25:47.586626+0000","last_unstale":"2024-04-23T10:26:40.460147+0000","last_undegraded":"2024-04-23T10:26:40.460147+0000","last_fullsized":"2024-04-23T10:26:40.460147+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T10:24:42.023127+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T10:24:42.023127+0000","last_clean_scrub_stamp":"2024-04-23T10:24:42.023127+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T20:55:38.216483+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":10,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,2,7],"acting":[11,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.1c","version":"89'9","reported_seq":28,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-23T10:26:40.281020+0000","last_change":"2024-04-23T10:24:44.051606+0000","last_active":"2024-04-23T10:26:40.281020+0000","last_peered":"2024-04-23T10:26:40.281020+0000","last_clean":"2024-04-23T10:26:40.281020+0000","last_became_active":"2024-04-23T10:24:44.051191+0000","last_became_peered":"2024-04-23T10:24:44.051191+0000","last_unstale":"2024-04-23T10:26:40.281020+0000","last_undegraded":"2024-04-23T10:26:40.281020+0000","last_fullsized":"2024-04-23T10:26:40.281020+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T10:24:43.025324+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T10:24:43.025324+0000","last_clean_scrub_stamp":"2024-04-23T10:24:43.025324+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T16:21:51.706066+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,10,6],"acting":[1,10,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1c","version":"86'2","reported_seq":23,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-23T10:26:40.373937+0000","last_change":"2024-04-23T10:24:43.048594+0000","last_active":"2024-04-23T10:26:40.373937+0000","last_peered":"2024-04-23T10:26:40.373937+0000","last_clean":"2024-04-23T10:26:40.373937+0000","last_became_active":"2024-04-23T10:24:43.048276+0000","last_became_peered":"2024-04-23T10:24:43.048276+0000","last_unstale":"2024-04-23T10:26:40.373937+0000","last_undegraded":"2024-04-23T10:26:40.373937+0000","last_fullsized":"2024-04-23T10:26:40.373937+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T10:24:42.023127+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T10:24:42.023127+0000","last_clean_scrub_stamp":"2024-04-23T10:24:42.023127+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T12:20:40.196674+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,1,5],"acting":[8,1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.1d","version":"89'15","reported_seq":34,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-23T10:26:40.282049+0000","last_change":"2024-04-23T10:24:44.050970+0000","last_active":"2024-04-23T10:26:40.282049+0000","last_peered":"2024-04-23T10:26:40.282049+0000","last_clean":"2024-04-23T10:26:40.282049+0000","last_became_active":"2024-04-23T10:24:44.050640+0000","last_became_peered":"2024-04-23T10:24:44.050640+0000","last_unstale":"2024-04-23T10:26:40.282049+0000","last_undegraded":"2024-04-23T10:26:40.282049+0000","last_fullsized":"2024-04-23T10:26:40.282049+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T10:24:43.025324+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T10:24:43.025324+0000","last_clean_scrub_stamp":"2024-04-23T10:24:43.025324+0000","objects_scrubbed":0,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T21:03:24.570290+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,1],"acting":[5,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.1b","version":"90'8","reported_seq":146,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-23T10:26:40.486313+0000","last_change":"2024-04-23T10:24:43.051742+0000","last_active":"2024-04-23T10:26:40.486313+0000","last_peered":"2024-04-23T10:26:40.486313+0000","last_clean":"2024-04-23T10:26:40.486313+0000","last_became_active":"2024-04-23T10:24:43.051420+0000","last_became_peered":"2024-04-23T10:24:43.051420+0000","last_unstale":"2024-04-23T10:26:40.486313+0000","last_undegraded":"2024-04-23T10:26:40.486313+0000","last_fullsized":"2024-04-23T10:26:40.486313+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T10:24:42.023127+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T10:24:42.023127+0000","last_clean_scrub_stamp":"2024-04-23T10:24:42.023127+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T17:54:37.182731+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1545,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":117,"num_read_kb":4,"num_write":5,"num_write_kb":5,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,7],"acting":[5,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.1a","version":"89'7","reported_seq":26,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-23T10:26:40.373718+0000","last_change":"2024-04-23T10:24:44.049314+0000","last_active":"2024-04-23T10:26:40.373718+0000","last_peered":"2024-04-23T10:26:40.373718+0000","last_clean":"2024-04-23T10:26:40.373718+0000","last_became_active":"2024-04-23T10:24:44.048352+0000","last_became_peered":"2024-04-23T10:24:44.048352+0000","last_unstale":"2024-04-23T10:26:40.373718+0000","last_undegraded":"2024-04-23T10:26:40.373718+0000","last_fullsized":"2024-04-23T10:26:40.373718+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T10:24:43.025324+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T10:24:43.025324+0000","last_clean_scrub_stamp":"2024-04-23T10:24:43.025324+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T16:57:29.957716+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,9,5],"acting":[8,9,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"2.1a","version":"87'3","reported_seq":24,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-23T10:26:40.381286+0000","last_change":"2024-04-23T10:24:43.051362+0000","last_active":"2024-04-23T10:26:40.381286+0000","last_peered":"2024-04-23T10:26:40.381286+0000","last_clean":"2024-04-23T10:26:40.381286+0000","last_became_active":"2024-04-23T10:24:43.051045+0000","last_became_peered":"2024-04-23T10:24:43.051045+0000","last_unstale":"2024-04-23T10:26:40.381286+0000","last_undegraded":"2024-04-23T10:26:40.381286+0000","last_fullsized":"2024-04-23T10:26:40.381286+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T10:24:42.023127+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T10:24:42.023127+0000","last_clean_scrub_stamp":"2024-04-23T10:24:42.023127+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T15:40:53.618281+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,8,6],"acting":[10,8,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.1b","version":"89'8","reported_seq":27,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-23T10:26:40.281974+0000","last_change":"2024-04-23T10:24:44.040323+0000","last_active":"2024-04-23T10:26:40.281974+0000","last_peered":"2024-04-23T10:26:40.281974+0000","last_clean":"2024-04-23T10:26:40.281974+0000","last_became_active":"2024-04-23T10:24:44.039995+0000","last_became_peered":"2024-04-23T10:24:44.039995+0000","last_unstale":"2024-04-23T10:26:40.281974+0000","last_undegraded":"2024-04-23T10:26:40.281974+0000","last_fullsized":"2024-04-23T10:26:40.281974+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T10:24:43.025324+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T10:24:43.025324+0000","last_clean_scrub_stamp":"2024-04-23T10:24:43.025324+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T16:38:48.756776+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,3],"acting":[5,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.19","version":"87'2","reported_seq":23,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-23T10:26:40.277848+0000","last_change":"2024-04-23T10:24:43.048754+0000","last_active":"2024-04-23T10:26:40.277848+0000","last_peered":"2024-04-23T10:26:40.277848+0000","last_clean":"2024-04-23T10:26:40.277848+0000","last_became_active":"2024-04-23T10:24:43.047962+0000","last_became_peered":"2024-04-23T10:24:43.047962+0000","last_unstale":"2024-04-23T10:26:40.277848+0000","last_undegraded":"2024-04-23T10:26:40.277848+0000","last_fullsized":"2024-04-23T10:26:40.277848+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T10:24:42.023127+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T10:24:42.023127+0000","last_clean_scrub_stamp":"2024-04-23T10:24:42.023127+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T21:08:38.487185+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,9],"acting":[6,4,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.18","version":"89'8","reported_seq":27,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-23T10:26:40.373688+0000","last_change":"2024-04-23T10:24:44.045022+0000","last_active":"2024-04-23T10:26:40.373688+0000","last_peered":"2024-04-23T10:26:40.373688+0000","last_clean":"2024-04-23T10:26:40.373688+0000","last_became_active":"2024-04-23T10:24:44.044590+0000","last_became_peered":"2024-04-23T10:24:44.044590+0000","last_unstale":"2024-04-23T10:26:40.373688+0000","last_undegraded":"2024-04-23T10:26:40.373688+0000","last_fullsized":"2024-04-23T10:26:40.373688+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T10:24:43.025324+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T10:24:43.025324+0000","last_clean_scrub_stamp":"2024-04-23T10:24:43.025324+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T13:20:49.892013+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,11,6],"acting":[3,11,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.18","version":"87'1","reported_seq":22,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-23T10:26:40.281503+0000","last_change":"2024-04-23T10:24:43.050170+0000","last_active":"2024-04-23T10:26:40.281503+0000","last_peered":"2024-04-23T10:26:40.281503+0000","last_clean":"2024-04-23T10:26:40.281503+0000","last_became_active":"2024-04-23T10:24:43.049070+0000","last_became_peered":"2024-04-23T10:24:43.049070+0000","last_unstale":"2024-04-23T10:26:40.281503+0000","last_undegraded":"2024-04-23T10:26:40.281503+0000","last_fullsized":"2024-04-23T10:26:40.281503+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T10:24:42.023127+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T10:24:42.023127+0000","last_clean_scrub_stamp":"2024-04-23T10:24:42.023127+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T19:34:13.983628+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,7],"acting":[5,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.19","version":"89'11","reported_seq":30,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-23T10:26:40.371665+0000","last_change":"2024-04-23T10:24:44.047165+0000","last_active":"2024-04-23T10:26:40.371665+0000","last_peered":"2024-04-23T10:26:40.371665+0000","last_clean":"2024-04-23T10:26:40.371665+0000","last_became_active":"2024-04-23T10:24:44.046776+0000","last_became_peered":"2024-04-23T10:24:44.046776+0000","last_unstale":"2024-04-23T10:26:40.371665+0000","last_undegraded":"2024-04-23T10:26:40.371665+0000","last_fullsized":"2024-04-23T10:26:40.371665+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T10:24:43.025324+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T10:24:43.025324+0000","last_clean_scrub_stamp":"2024-04-23T10:24:43.025324+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T18:11:57.181747+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,3,10],"acting":[9,3,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"2.17","version":"90'5","reported_seq":139,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-23T10:26:40.485905+0000","last_change":"2024-04-23T10:24:43.049785+0000","last_active":"2024-04-23T10:26:40.485905+0000","last_peered":"2024-04-23T10:26:40.485905+0000","last_clean":"2024-04-23T10:26:40.485905+0000","last_became_active":"2024-04-23T10:24:43.049402+0000","last_became_peered":"2024-04-23T10:24:43.049402+0000","last_unstale":"2024-04-23T10:26:40.485905+0000","last_undegraded":"2024-04-23T10:26:40.485905+0000","last_fullsized":"2024-04-23T10:26:40.485905+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T10:24:42.023127+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T10:24:42.023127+0000","last_clean_scrub_stamp":"2024-04-23T10:24:42.023127+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T14:20:13.774082+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":113,"num_read_kb":113,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,5,9],"acting":[10,5,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.16","version":"89'10","reported_seq":29,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-23T10:26:40.281357+0000","last_change":"2024-04-23T10:24:44.051768+0000","last_active":"2024-04-23T10:26:40.281357+0000","last_peered":"2024-04-23T10:26:40.281357+0000","last_clean":"2024-04-23T10:26:40.281357+0000","last_became_active":"2024-04-23T10:24:44.051228+0000","last_became_peered":"2024-04-23T10:24:44.051228+0000","last_unstale":"2024-04-23T10:26:40.281357+0000","last_undegraded":"2024-04-23T10:26:40.281357+0000","last_fullsized":"2024-04-23T10:26:40.281357+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T10:24:43.025324+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T10:24:43.025324+0000","last_clean_scrub_stamp":"2024-04-23T10:24:43.025324+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T14:09:47.649083+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,10,9],"acting":[1,10,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.16","version":"87'2","reported_seq":23,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-23T10:26:40.522354+0000","last_change":"2024-04-23T10:24:43.043944+0000","last_active":"2024-04-23T10:26:40.522354+0000","last_peered":"2024-04-23T10:26:40.522354+0000","last_clean":"2024-04-23T10:26:40.522354+0000","last_became_active":"2024-04-23T10:24:43.043648+0000","last_became_peered":"2024-04-23T10:24:43.043648+0000","last_unstale":"2024-04-23T10:26:40.522354+0000","last_undegraded":"2024-04-23T10:26:40.522354+0000","last_fullsized":"2024-04-23T10:26:40.522354+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T10:24:42.023127+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T10:24:42.023127+0000","last_clean_scrub_stamp":"2024-04-23T10:24:42.023127+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T13:41:15.724662+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,2],"acting":[7,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.17","version":"89'9","reported_seq":34,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-23T10:26:40.460012+0000","last_change":"2024-04-23T10:25:47.721534+0000","last_active":"2024-04-23T10:26:40.460012+0000","last_peered":"2024-04-23T10:26:40.460012+0000","last_clean":"2024-04-23T10:26:40.460012+0000","last_became_active":"2024-04-23T10:25:47.721130+0000","last_became_peered":"2024-04-23T10:25:47.721130+0000","last_unstale":"2024-04-23T10:26:40.460012+0000","last_undegraded":"2024-04-23T10:26:40.460012+0000","last_fullsized":"2024-04-23T10:26:40.460012+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T10:24:43.025324+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T10:24:43.025324+0000","last_clean_scrub_stamp":"2024-04-23T10:24:43.025324+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T21:21:26.537486+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,5],"acting":[0,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.15","version":"87'4","reported_seq":26,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-23T10:26:40.282238+0000","last_change":"2024-04-23T10:24:43.048517+0000","last_active":"2024-04-23T10:26:40.282238+0000","last_peered":"2024-04-23T10:26:40.282238+0000","last_clean":"2024-04-23T10:26:40.282238+0000","last_became_active":"2024-04-23T10:24:43.048183+0000","last_became_peered":"2024-04-23T10:24:43.048183+0000","last_unstale":"2024-04-23T10:26:40.282238+0000","last_undegraded":"2024-04-23T10:26:40.282238+0000","last_fullsized":"2024-04-23T10:26:40.282238+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T10:24:42.023127+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T10:24:42.023127+0000","last_clean_scrub_stamp":"2024-04-23T10:24:42.023127+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T20:50:15.017833+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":672,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":5,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,10],"acting":[5,0,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.14","version":"89'11","reported_seq":30,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-23T10:26:40.281189+0000","last_change":"2024-04-23T10:24:44.055598+0000","last_active":"2024-04-23T10:26:40.281189+0000","last_peered":"2024-04-23T10:26:40.281189+0000","last_clean":"2024-04-23T10:26:40.281189+0000","last_became_active":"2024-04-23T10:24:44.053787+0000","last_became_peered":"2024-04-23T10:24:44.053787+0000","last_unstale":"2024-04-23T10:26:40.281189+0000","last_undegraded":"2024-04-23T10:26:40.281189+0000","last_fullsized":"2024-04-23T10:26:40.281189+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T10:24:43.025324+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T10:24:43.025324+0000","last_clean_scrub_stamp":"2024-04-23T10:24:43.025324+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T20:04:21.603306+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,8,4],"acting":[1,8,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.14","version":"90'36","reported_seq":293,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-23T10:26:40.520887+0000","last_change":"2024-04-23T10:24:43.044601+0000","last_active":"2024-04-23T10:26:40.520887+0000","last_peered":"2024-04-23T10:26:40.520887+0000","last_clean":"2024-04-23T10:26:40.520887+0000","last_became_active":"2024-04-23T10:24:43.044238+0000","last_became_peered":"2024-04-23T10:24:43.044238+0000","last_unstale":"2024-04-23T10:26:40.520887+0000","last_undegraded":"2024-04-23T10:26:40.520887+0000","last_fullsized":"2024-04-23T10:26:40.520887+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T10:24:42.023127+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T10:24:42.023127+0000","last_clean_scrub_stamp":"2024-04-23T10:24:42.023127+0000","objects_scrubbed":0,"log_size":36,"log_dups_size":0,"ondisk_log_size":36,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T12:53:26.275771+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":67758,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":236,"num_read_kb":183,"num_write":37,"num_write_kb":82,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,1,2],"acting":[10,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.15","version":"89'6","reported_seq":25,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-23T10:26:40.373730+0000","last_change":"2024-04-23T10:24:44.051375+0000","last_active":"2024-04-23T10:26:40.373730+0000","last_peered":"2024-04-23T10:26:40.373730+0000","last_clean":"2024-04-23T10:26:40.373730+0000","last_became_active":"2024-04-23T10:24:44.050924+0000","last_became_peered":"2024-04-23T10:24:44.050924+0000","last_unstale":"2024-04-23T10:26:40.373730+0000","last_undegraded":"2024-04-23T10:26:40.373730+0000","last_fullsized":"2024-04-23T10:26:40.373730+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T10:24:43.025324+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T10:24:43.025324+0000","last_clean_scrub_stamp":"2024-04-23T10:24:43.025324+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T21:50:57.534760+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,11,7],"acting":[8,11,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"2.13","version":"87'6","reported_seq":28,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-23T10:26:40.459793+0000","last_change":"2024-04-23T10:24:43.044457+0000","last_active":"2024-04-23T10:26:40.459793+0000","last_peered":"2024-04-23T10:26:40.459793+0000","last_clean":"2024-04-23T10:26:40.459793+0000","last_became_active":"2024-04-23T10:24:43.043956+0000","last_became_peered":"2024-04-23T10:24:43.043956+0000","last_unstale":"2024-04-23T10:26:40.459793+0000","last_undegraded":"2024-04-23T10:26:40.459793+0000","last_fullsized":"2024-04-23T10:26:40.459793+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T10:24:42.023127+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T10:24:42.023127+0000","last_clean_scrub_stamp":"2024-04-23T10:24:42.023127+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T20:16:08.186729+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":9,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,10,3],"acting":[2,10,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.12","version":"89'9","reported_seq":34,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-23T10:26:40.277806+0000","last_change":"2024-04-23T10:25:47.587240+0000","last_active":"2024-04-23T10:26:40.277806+0000","last_peered":"2024-04-23T10:26:40.277806+0000","last_clean":"2024-04-23T10:26:40.277806+0000","last_became_active":"2024-04-23T10:25:47.586659+0000","last_became_peered":"2024-04-23T10:25:47.586659+0000","last_unstale":"2024-04-23T10:26:40.277806+0000","last_undegraded":"2024-04-23T10:26:40.277806+0000","last_fullsized":"2024-04-23T10:26:40.277806+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T10:24:43.025324+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T10:24:43.025324+0000","last_clean_scrub_stamp":"2024-04-23T10:24:43.025324+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T19:55:03.256183+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,9],"acting":[6,7,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.12","version":"87'3","reported_seq":24,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-23T10:26:40.281768+0000","last_change":"2024-04-23T10:24:43.052181+0000","last_active":"2024-04-23T10:26:40.281768+0000","last_peered":"2024-04-23T10:26:40.281768+0000","last_clean":"2024-04-23T10:26:40.281768+0000","last_became_active":"2024-04-23T10:24:43.051793+0000","last_became_peered":"2024-04-23T10:24:43.051793+0000","last_unstale":"2024-04-23T10:26:40.281768+0000","last_undegraded":"2024-04-23T10:26:40.281768+0000","last_fullsized":"2024-04-23T10:26:40.281768+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T10:24:42.023127+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T10:24:42.023127+0000","last_clean_scrub_stamp":"2024-04-23T10:24:42.023127+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T20:57:59.015115+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,7],"acting":[5,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.13","version":"89'15","reported_seq":34,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-23T10:26:40.522012+0000","last_change":"2024-04-23T10:24:44.062689+0000","last_active":"2024-04-23T10:26:40.522012+0000","last_peered":"2024-04-23T10:26:40.522012+0000","last_clean":"2024-04-23T10:26:40.522012+0000","last_became_active":"2024-04-23T10:24:44.062331+0000","last_became_peered":"2024-04-23T10:24:44.062331+0000","last_unstale":"2024-04-23T10:26:40.522012+0000","last_undegraded":"2024-04-23T10:26:40.522012+0000","last_fullsized":"2024-04-23T10:26:40.522012+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T10:24:43.025324+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T10:24:43.025324+0000","last_clean_scrub_stamp":"2024-04-23T10:24:43.025324+0000","objects_scrubbed":0,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T14:06:04.989156+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,6],"acting":[7,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.11","version":"90'6","reported_seq":144,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-23T10:26:40.486457+0000","last_change":"2024-04-23T10:24:43.043901+0000","last_active":"2024-04-23T10:26:40.486457+0000","last_peered":"2024-04-23T10:26:40.486457+0000","last_clean":"2024-04-23T10:26:40.486457+0000","last_became_active":"2024-04-23T10:24:43.043598+0000","last_became_peered":"2024-04-23T10:24:43.043598+0000","last_unstale":"2024-04-23T10:26:40.486457+0000","last_undegraded":"2024-04-23T10:26:40.486457+0000","last_fullsized":"2024-04-23T10:26:40.486457+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T10:24:42.023127+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T10:24:42.023127+0000","last_clean_scrub_stamp":"2024-04-23T10:24:42.023127+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T14:00:06.338084+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1511,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":118,"num_read_kb":5,"num_write":5,"num_write_kb":4,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,10],"acting":[4,7,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.10","version":"89'6","reported_seq":25,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-23T10:26:40.521595+0000","last_change":"2024-04-23T10:24:44.058773+0000","last_active":"2024-04-23T10:26:40.521595+0000","last_peered":"2024-04-23T10:26:40.521595+0000","last_clean":"2024-04-23T10:26:40.521595+0000","last_became_active":"2024-04-23T10:24:44.058420+0000","last_became_peered":"2024-04-23T10:24:44.058420+0000","last_unstale":"2024-04-23T10:26:40.521595+0000","last_undegraded":"2024-04-23T10:26:40.521595+0000","last_fullsized":"2024-04-23T10:26:40.521595+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T10:24:43.025324+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T10:24:43.025324+0000","last_clean_scrub_stamp":"2024-04-23T10:24:43.025324+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T12:44:30.023047+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,10],"acting":[7,5,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.10","version":"87'15","reported_seq":37,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-23T10:26:40.373552+0000","last_change":"2024-04-23T10:24:43.044700+0000","last_active":"2024-04-23T10:26:40.373552+0000","last_peered":"2024-04-23T10:26:40.373552+0000","last_clean":"2024-04-23T10:26:40.373552+0000","last_became_active":"2024-04-23T10:24:43.044394+0000","last_became_peered":"2024-04-23T10:24:43.044394+0000","last_unstale":"2024-04-23T10:26:40.373552+0000","last_undegraded":"2024-04-23T10:26:40.373552+0000","last_fullsized":"2024-04-23T10:26:40.373552+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T10:24:42.023127+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T10:24:42.023127+0000","last_clean_scrub_stamp":"2024-04-23T10:24:42.023127+0000","objects_scrubbed":0,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T15:16:41.367517+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":112,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":20,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":6,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,0,2],"acting":[8,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.11","version":"89'6","reported_seq":25,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-23T10:26:40.281571+0000","last_change":"2024-04-23T10:24:44.049739+0000","last_active":"2024-04-23T10:26:40.281571+0000","last_peered":"2024-04-23T10:26:40.281571+0000","last_clean":"2024-04-23T10:26:40.281571+0000","last_became_active":"2024-04-23T10:24:44.049414+0000","last_became_peered":"2024-04-23T10:24:44.049414+0000","last_unstale":"2024-04-23T10:26:40.281571+0000","last_undegraded":"2024-04-23T10:26:40.281571+0000","last_fullsized":"2024-04-23T10:26:40.281571+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T10:24:43.025324+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T10:24:43.025324+0000","last_clean_scrub_stamp":"2024-04-23T10:24:43.025324+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T12:51:02.645259+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,1],"acting":[5,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.f","version":"87'2","reported_seq":24,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-23T10:26:40.373920+0000","last_change":"2024-04-23T10:24:43.048884+0000","last_active":"2024-04-23T10:26:40.373920+0000","last_peered":"2024-04-23T10:26:40.373920+0000","last_clean":"2024-04-23T10:26:40.373920+0000","last_became_active":"2024-04-23T10:24:43.048583+0000","last_became_peered":"2024-04-23T10:24:43.048583+0000","last_unstale":"2024-04-23T10:26:40.373920+0000","last_undegraded":"2024-04-23T10:26:40.373920+0000","last_fullsized":"2024-04-23T10:26:40.373920+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T10:24:42.023127+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T10:24:42.023127+0000","last_clean_scrub_stamp":"2024-04-23T10:24:42.023127+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T17:55:57.869700+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,5,4],"acting":[8,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.e","version":"89'5","reported_seq":30,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-23T10:26:40.460256+0000","last_change":"2024-04-23T10:25:47.722078+0000","last_active":"2024-04-23T10:26:40.460256+0000","last_peered":"2024-04-23T10:26:40.460256+0000","last_clean":"2024-04-23T10:26:40.460256+0000","last_became_active":"2024-04-23T10:25:47.721730+0000","last_became_peered":"2024-04-23T10:25:47.721730+0000","last_unstale":"2024-04-23T10:26:40.460256+0000","last_undegraded":"2024-04-23T10:26:40.460256+0000","last_fullsized":"2024-04-23T10:26:40.460256+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T10:24:43.025324+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T10:24:43.025324+0000","last_clean_scrub_stamp":"2024-04-23T10:24:43.025324+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T18:18:40.916492+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,9,4],"acting":[11,9,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":21,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-23T10:26:40.371847+0000","last_change":"2024-04-23T10:24:43.044375+0000","last_active":"2024-04-23T10:26:40.371847+0000","last_peered":"2024-04-23T10:26:40.371847+0000","last_clean":"2024-04-23T10:26:40.371847+0000","last_became_active":"2024-04-23T10:24:43.043914+0000","last_became_peered":"2024-04-23T10:24:43.043914+0000","last_unstale":"2024-04-23T10:26:40.371847+0000","last_undegraded":"2024-04-23T10:26:40.371847+0000","last_fullsized":"2024-04-23T10:26:40.371847+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T10:24:42.023127+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T10:24:42.023127+0000","last_clean_scrub_stamp":"2024-04-23T10:24:42.023127+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T16:22:44.165156+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,2,0],"acting":[9,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.5","version":"89'5","reported_seq":24,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-23T10:26:40.381571+0000","last_change":"2024-04-23T10:24:44.057272+0000","last_active":"2024-04-23T10:26:40.381571+0000","last_peered":"2024-04-23T10:26:40.381571+0000","last_clean":"2024-04-23T10:26:40.381571+0000","last_became_active":"2024-04-23T10:24:44.056968+0000","last_became_peered":"2024-04-23T10:24:44.056968+0000","last_unstale":"2024-04-23T10:26:40.381571+0000","last_undegraded":"2024-04-23T10:26:40.381571+0000","last_fullsized":"2024-04-23T10:26:40.381571+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T10:24:43.025324+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T10:24:43.025324+0000","last_clean_scrub_stamp":"2024-04-23T10:24:43.025324+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T11:10:09.021845+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":117,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,3,6],"acting":[10,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"2.2","version":"87'1","reported_seq":22,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-23T10:26:40.521864+0000","last_change":"2024-04-23T10:24:43.051822+0000","last_active":"2024-04-23T10:26:40.521864+0000","last_peered":"2024-04-23T10:26:40.521864+0000","last_clean":"2024-04-23T10:26:40.521864+0000","last_became_active":"2024-04-23T10:24:43.051491+0000","last_became_peered":"2024-04-23T10:24:43.051491+0000","last_unstale":"2024-04-23T10:26:40.521864+0000","last_undegraded":"2024-04-23T10:26:40.521864+0000","last_fullsized":"2024-04-23T10:26:40.521864+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T10:24:42.023127+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T10:24:42.023127+0000","last_clean_scrub_stamp":"2024-04-23T10:24:42.023127+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T13:49:23.109064+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,1],"acting":[7,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.3","version":"89'7","reported_seq":26,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-23T10:26:40.381469+0000","last_change":"2024-04-23T10:24:44.053604+0000","last_active":"2024-04-23T10:26:40.381469+0000","last_peered":"2024-04-23T10:26:40.381469+0000","last_clean":"2024-04-23T10:26:40.381469+0000","last_became_active":"2024-04-23T10:24:44.053147+0000","last_became_peered":"2024-04-23T10:24:44.053147+0000","last_unstale":"2024-04-23T10:26:40.381469+0000","last_undegraded":"2024-04-23T10:26:40.381469+0000","last_fullsized":"2024-04-23T10:26:40.381469+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T10:24:43.025324+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T10:24:43.025324+0000","last_clean_scrub_stamp":"2024-04-23T10:24:43.025324+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T21:58:38.981282+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,4,0],"acting":[10,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.0","version":"89'8","reported_seq":27,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-23T10:26:40.281729+0000","last_change":"2024-04-23T10:24:44.052564+0000","last_active":"2024-04-23T10:26:40.281729+0000","last_peered":"2024-04-23T10:26:40.281729+0000","last_clean":"2024-04-23T10:26:40.281729+0000","last_became_active":"2024-04-23T10:24:44.052201+0000","last_became_peered":"2024-04-23T10:24:44.052201+0000","last_unstale":"2024-04-23T10:26:40.281729+0000","last_undegraded":"2024-04-23T10:26:40.281729+0000","last_fullsized":"2024-04-23T10:26:40.281729+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T10:24:43.025324+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T10:24:43.025324+0000","last_clean_scrub_stamp":"2024-04-23T10:24:43.025324+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T16:25:54.725347+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,4],"acting":[5,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.1","version":"90'6","reported_seq":140,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-23T10:26:40.486009+0000","last_change":"2024-04-23T10:24:43.049067+0000","last_active":"2024-04-23T10:26:40.486009+0000","last_peered":"2024-04-23T10:26:40.486009+0000","last_clean":"2024-04-23T10:26:40.486009+0000","last_became_active":"2024-04-23T10:24:43.048400+0000","last_became_peered":"2024-04-23T10:24:43.048400+0000","last_unstale":"2024-04-23T10:26:40.486009+0000","last_undegraded":"2024-04-23T10:26:40.486009+0000","last_fullsized":"2024-04-23T10:26:40.486009+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T10:24:42.023127+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T10:24:42.023127+0000","last_clean_scrub_stamp":"2024-04-23T10:24:42.023127+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T12:17:02.439054+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":113,"num_read_kb":113,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,0],"acting":[6,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.2","version":"89'8","reported_seq":27,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-23T10:26:40.373993+0000","last_change":"2024-04-23T10:24:44.059114+0000","last_active":"2024-04-23T10:26:40.373993+0000","last_peered":"2024-04-23T10:26:40.373993+0000","last_clean":"2024-04-23T10:26:40.373993+0000","last_became_active":"2024-04-23T10:24:44.058618+0000","last_became_peered":"2024-04-23T10:24:44.058618+0000","last_unstale":"2024-04-23T10:26:40.373993+0000","last_undegraded":"2024-04-23T10:26:40.373993+0000","last_fullsized":"2024-04-23T10:26:40.373993+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T10:24:43.025324+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T10:24:43.025324+0000","last_clean_scrub_stamp":"2024-04-23T10:24:43.025324+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T20:18:09.707062+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,1],"acting":[3,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.3","version":"87'3","reported_seq":26,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-23T10:26:40.459688+0000","last_change":"2024-04-23T10:24:43.039725+0000","last_active":"2024-04-23T10:26:40.459688+0000","last_peered":"2024-04-23T10:26:40.459688+0000","last_clean":"2024-04-23T10:26:40.459688+0000","last_became_active":"2024-04-23T10:24:43.039187+0000","last_became_peered":"2024-04-23T10:24:43.039187+0000","last_unstale":"2024-04-23T10:26:40.459688+0000","last_undegraded":"2024-04-23T10:26:40.459688+0000","last_fullsized":"2024-04-23T10:26:40.459688+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T10:24:42.023127+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T10:24:42.023127+0000","last_clean_scrub_stamp":"2024-04-23T10:24:42.023127+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T17:20:50.844878+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":3,"num_read_kb":3,"num_write":4,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,10,9],"acting":[4,10,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.0","version":"22'137","reported_seq":353,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-23T10:26:40.521666+0000","last_change":"2024-04-23T10:23:31.092860+0000","last_active":"2024-04-23T10:26:40.521666+0000","last_peered":"2024-04-23T10:26:40.521666+0000","last_clean":"2024-04-23T10:26:40.521666+0000","last_became_active":"2024-04-23T10:23:31.092357+0000","last_became_peered":"2024-04-23T10:23:31.092357+0000","last_unstale":"2024-04-23T10:26:40.521666+0000","last_undegraded":"2024-04-23T10:26:40.521666+0000","last_fullsized":"2024-04-23T10:26:40.521666+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T10:20:12.201233+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T10:20:12.201233+0000","last_clean_scrub_stamp":"2024-04-23T10:20:12.201233+0000","objects_scrubbed":0,"log_size":137,"log_dups_size":0,"ondisk_log_size":137,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T21:28:56.804053+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":136,"num_read_kb":119,"num_write":181,"num_write_kb":2938,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":1180736,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,10],"acting":[7,5,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1","version":"89'5","reported_seq":24,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-23T10:26:40.277697+0000","last_change":"2024-04-23T10:24:44.058393+0000","last_active":"2024-04-23T10:26:40.277697+0000","last_peered":"2024-04-23T10:26:40.277697+0000","last_clean":"2024-04-23T10:26:40.277697+0000","last_became_active":"2024-04-23T10:24:44.058075+0000","last_became_peered":"2024-04-23T10:24:44.058075+0000","last_unstale":"2024-04-23T10:26:40.277697+0000","last_undegraded":"2024-04-23T10:26:40.277697+0000","last_fullsized":"2024-04-23T10:26:40.277697+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T10:24:43.025324+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T10:24:43.025324+0000","last_clean_scrub_stamp":"2024-04-23T10:24:43.025324+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T10:32:05.693311+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,2],"acting":[6,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.0","version":"87'3","reported_seq":24,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-23T10:26:40.521796+0000","last_change":"2024-04-23T10:24:43.048678+0000","last_active":"2024-04-23T10:26:40.521796+0000","last_peered":"2024-04-23T10:26:40.521796+0000","last_clean":"2024-04-23T10:26:40.521796+0000","last_became_active":"2024-04-23T10:24:43.048341+0000","last_became_peered":"2024-04-23T10:24:43.048341+0000","last_unstale":"2024-04-23T10:26:40.521796+0000","last_undegraded":"2024-04-23T10:26:40.521796+0000","last_fullsized":"2024-04-23T10:26:40.521796+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T10:24:42.023127+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T10:24:42.023127+0000","last_clean_scrub_stamp":"2024-04-23T10:24:42.023127+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T12:24:33.705325+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,8],"acting":[7,5,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.5","version":"87'1","reported_seq":23,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-23T10:26:40.373795+0000","last_change":"2024-04-23T10:24:43.046105+0000","last_active":"2024-04-23T10:26:40.373795+0000","last_peered":"2024-04-23T10:26:40.373795+0000","last_clean":"2024-04-23T10:26:40.373795+0000","last_became_active":"2024-04-23T10:24:43.045804+0000","last_became_peered":"2024-04-23T10:24:43.045804+0000","last_unstale":"2024-04-23T10:26:40.373795+0000","last_undegraded":"2024-04-23T10:26:40.373795+0000","last_fullsized":"2024-04-23T10:26:40.373795+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T10:24:42.023127+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T10:24:42.023127+0000","last_clean_scrub_stamp":"2024-04-23T10:24:42.023127+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T22:23:35.927468+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,0,9],"acting":[8,0,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.4","version":"89'9","reported_seq":28,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-23T10:26:40.278062+0000","last_change":"2024-04-23T10:24:44.059731+0000","last_active":"2024-04-23T10:26:40.278062+0000","last_peered":"2024-04-23T10:26:40.278062+0000","last_clean":"2024-04-23T10:26:40.278062+0000","last_became_active":"2024-04-23T10:24:44.059352+0000","last_became_peered":"2024-04-23T10:24:44.059352+0000","last_unstale":"2024-04-23T10:26:40.278062+0000","last_undegraded":"2024-04-23T10:26:40.278062+0000","last_fullsized":"2024-04-23T10:26:40.278062+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T10:24:43.025324+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T10:24:43.025324+0000","last_clean_scrub_stamp":"2024-04-23T10:24:43.025324+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T16:36:08.985444+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2,9],"acting":[6,2,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.6","version":"90'44","reported_seq":88,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-23T10:26:40.483019+0000","last_change":"2024-04-23T10:24:43.044527+0000","last_active":"2024-04-23T10:26:40.483019+0000","last_peered":"2024-04-23T10:26:40.483019+0000","last_clean":"2024-04-23T10:26:40.483019+0000","last_became_active":"2024-04-23T10:24:43.044212+0000","last_became_peered":"2024-04-23T10:24:43.044212+0000","last_unstale":"2024-04-23T10:26:40.483019+0000","last_undegraded":"2024-04-23T10:26:40.483019+0000","last_fullsized":"2024-04-23T10:26:40.483019+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T10:24:42.023127+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T10:24:42.023127+0000","last_clean_scrub_stamp":"2024-04-23T10:24:42.023127+0000","objects_scrubbed":0,"log_size":44,"log_dups_size":0,"ondisk_log_size":44,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T19:39:46.369132+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":393216,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":82,"num_read_kb":70,"num_write":78,"num_write_kb":836,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,8,3],"acting":[9,8,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.7","version":"89'5","reported_seq":24,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-23T10:26:40.374352+0000","last_change":"2024-04-23T10:24:44.055920+0000","last_active":"2024-04-23T10:26:40.374352+0000","last_peered":"2024-04-23T10:26:40.374352+0000","last_clean":"2024-04-23T10:26:40.374352+0000","last_became_active":"2024-04-23T10:24:44.055411+0000","last_became_peered":"2024-04-23T10:24:44.055411+0000","last_unstale":"2024-04-23T10:26:40.374352+0000","last_undegraded":"2024-04-23T10:26:40.374352+0000","last_fullsized":"2024-04-23T10:26:40.374352+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T10:24:43.025324+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T10:24:43.025324+0000","last_clean_scrub_stamp":"2024-04-23T10:24:43.025324+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T14:43:10.446790+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,11,7],"acting":[3,11,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.7","version":"87'4","reported_seq":25,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-23T10:26:40.459655+0000","last_change":"2024-04-23T10:24:43.047989+0000","last_active":"2024-04-23T10:26:40.459655+0000","last_peered":"2024-04-23T10:26:40.459655+0000","last_clean":"2024-04-23T10:26:40.459655+0000","last_became_active":"2024-04-23T10:24:43.047625+0000","last_became_peered":"2024-04-23T10:24:43.047625+0000","last_unstale":"2024-04-23T10:26:40.459655+0000","last_undegraded":"2024-04-23T10:26:40.459655+0000","last_fullsized":"2024-04-23T10:26:40.459655+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T10:24:42.023127+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T10:24:42.023127+0000","last_clean_scrub_stamp":"2024-04-23T10:24:42.023127+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T13:50:16.647176+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,11,6],"acting":[4,11,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.6","version":"89'7","reported_seq":26,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-23T10:26:40.459895+0000","last_change":"2024-04-23T10:24:44.049214+0000","last_active":"2024-04-23T10:26:40.459895+0000","last_peered":"2024-04-23T10:26:40.459895+0000","last_clean":"2024-04-23T10:26:40.459895+0000","last_became_active":"2024-04-23T10:24:44.048869+0000","last_became_peered":"2024-04-23T10:24:44.048869+0000","last_unstale":"2024-04-23T10:26:40.459895+0000","last_undegraded":"2024-04-23T10:26:40.459895+0000","last_fullsized":"2024-04-23T10:26:40.459895+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T10:24:43.025324+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T10:24:43.025324+0000","last_clean_scrub_stamp":"2024-04-23T10:24:43.025324+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T18:17:09.772428+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,4],"acting":[0,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.8","version":"87'2","reported_seq":23,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-23T10:26:40.281014+0000","last_change":"2024-04-23T10:24:43.046189+0000","last_active":"2024-04-23T10:26:40.281014+0000","last_peered":"2024-04-23T10:26:40.281014+0000","last_clean":"2024-04-23T10:26:40.281014+0000","last_became_active":"2024-04-23T10:24:43.045848+0000","last_became_peered":"2024-04-23T10:24:43.045848+0000","last_unstale":"2024-04-23T10:26:40.281014+0000","last_undegraded":"2024-04-23T10:26:40.281014+0000","last_fullsized":"2024-04-23T10:26:40.281014+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T10:24:42.023127+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T10:24:42.023127+0000","last_clean_scrub_stamp":"2024-04-23T10:24:42.023127+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T22:23:39.180377+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,0],"acting":[1,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.9","version":"89'2","reported_seq":21,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-23T10:26:40.373557+0000","last_change":"2024-04-23T10:24:44.048415+0000","last_active":"2024-04-23T10:26:40.373557+0000","last_peered":"2024-04-23T10:26:40.373557+0000","last_clean":"2024-04-23T10:26:40.373557+0000","last_became_active":"2024-04-23T10:24:44.048030+0000","last_became_peered":"2024-04-23T10:24:44.048030+0000","last_unstale":"2024-04-23T10:26:40.373557+0000","last_undegraded":"2024-04-23T10:26:40.373557+0000","last_fullsized":"2024-04-23T10:26:40.373557+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T10:24:43.025324+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T10:24:43.025324+0000","last_clean_scrub_stamp":"2024-04-23T10:24:43.025324+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T14:42:09.920927+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,11,10],"acting":[8,11,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"2.9","version":"87'3","reported_seq":24,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-23T10:26:40.373745+0000","last_change":"2024-04-23T10:24:43.041525+0000","last_active":"2024-04-23T10:26:40.373745+0000","last_peered":"2024-04-23T10:26:40.373745+0000","last_clean":"2024-04-23T10:26:40.373745+0000","last_became_active":"2024-04-23T10:24:43.041200+0000","last_became_peered":"2024-04-23T10:24:43.041200+0000","last_unstale":"2024-04-23T10:26:40.373745+0000","last_undegraded":"2024-04-23T10:26:40.373745+0000","last_fullsized":"2024-04-23T10:26:40.373745+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T10:24:42.023127+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T10:24:42.023127+0000","last_clean_scrub_stamp":"2024-04-23T10:24:42.023127+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T16:45:06.846991+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,9,10],"acting":[3,9,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.8","version":"89'9","reported_seq":28,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-23T10:26:40.460025+0000","last_change":"2024-04-23T10:24:44.055341+0000","last_active":"2024-04-23T10:26:40.460025+0000","last_peered":"2024-04-23T10:26:40.460025+0000","last_clean":"2024-04-23T10:26:40.460025+0000","last_became_active":"2024-04-23T10:24:44.054216+0000","last_became_peered":"2024-04-23T10:24:44.054216+0000","last_unstale":"2024-04-23T10:26:40.460025+0000","last_undegraded":"2024-04-23T10:26:40.460025+0000","last_fullsized":"2024-04-23T10:26:40.460025+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T10:24:43.025324+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T10:24:43.025324+0000","last_clean_scrub_stamp":"2024-04-23T10:24:43.025324+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T21:31:07.753283+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,0,2],"acting":[11,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"2.a","version":"87'1","reported_seq":22,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-23T10:26:40.459791+0000","last_change":"2024-04-23T10:24:43.045400+0000","last_active":"2024-04-23T10:26:40.459791+0000","last_peered":"2024-04-23T10:26:40.459791+0000","last_clean":"2024-04-23T10:26:40.459791+0000","last_became_active":"2024-04-23T10:24:43.044677+0000","last_became_peered":"2024-04-23T10:24:43.044677+0000","last_unstale":"2024-04-23T10:26:40.459791+0000","last_undegraded":"2024-04-23T10:26:40.459791+0000","last_fullsized":"2024-04-23T10:26:40.459791+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T10:24:42.023127+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T10:24:42.023127+0000","last_clean_scrub_stamp":"2024-04-23T10:24:42.023127+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T19:06:19.700132+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,8],"acting":[2,3,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.b","version":"89'8","reported_seq":27,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-23T10:26:40.374065+0000","last_change":"2024-04-23T10:24:44.049968+0000","last_active":"2024-04-23T10:26:40.374065+0000","last_peered":"2024-04-23T10:26:40.374065+0000","last_clean":"2024-04-23T10:26:40.374065+0000","last_became_active":"2024-04-23T10:24:44.049621+0000","last_became_peered":"2024-04-23T10:24:44.049621+0000","last_unstale":"2024-04-23T10:26:40.374065+0000","last_undegraded":"2024-04-23T10:26:40.374065+0000","last_fullsized":"2024-04-23T10:26:40.374065+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T10:24:43.025324+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T10:24:43.025324+0000","last_clean_scrub_stamp":"2024-04-23T10:24:43.025324+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T16:23:43.082831+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,10],"acting":[3,0,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.b","version":"87'4","reported_seq":26,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-23T10:26:40.373780+0000","last_change":"2024-04-23T10:24:43.041828+0000","last_active":"2024-04-23T10:26:40.373780+0000","last_peered":"2024-04-23T10:26:40.373780+0000","last_clean":"2024-04-23T10:26:40.373780+0000","last_became_active":"2024-04-23T10:24:43.041480+0000","last_became_peered":"2024-04-23T10:24:43.041480+0000","last_unstale":"2024-04-23T10:26:40.373780+0000","last_undegraded":"2024-04-23T10:26:40.373780+0000","last_fullsized":"2024-04-23T10:26:40.373780+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T10:24:42.023127+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T10:24:42.023127+0000","last_clean_scrub_stamp":"2024-04-23T10:24:42.023127+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T14:43:03.203971+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":2,"num_read_kb":2,"num_write":4,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,2,7],"acting":[8,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.a","version":"89'5","reported_seq":24,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-23T10:26:40.522120+0000","last_change":"2024-04-23T10:24:44.062103+0000","last_active":"2024-04-23T10:26:40.522120+0000","last_peered":"2024-04-23T10:26:40.522120+0000","last_clean":"2024-04-23T10:26:40.522120+0000","last_became_active":"2024-04-23T10:24:44.061794+0000","last_became_peered":"2024-04-23T10:24:44.061794+0000","last_unstale":"2024-04-23T10:26:40.522120+0000","last_undegraded":"2024-04-23T10:26:40.522120+0000","last_fullsized":"2024-04-23T10:26:40.522120+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T10:24:43.025324+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T10:24:43.025324+0000","last_clean_scrub_stamp":"2024-04-23T10:24:43.025324+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T10:45:30.130047+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,6],"acting":[7,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.c","version":"87'3","reported_seq":25,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-23T10:26:40.460001+0000","last_change":"2024-04-23T10:24:43.045702+0000","last_active":"2024-04-23T10:26:40.460001+0000","last_peered":"2024-04-23T10:26:40.460001+0000","last_clean":"2024-04-23T10:26:40.460001+0000","last_became_active":"2024-04-23T10:24:43.045258+0000","last_became_peered":"2024-04-23T10:24:43.045258+0000","last_unstale":"2024-04-23T10:26:40.460001+0000","last_undegraded":"2024-04-23T10:26:40.460001+0000","last_fullsized":"2024-04-23T10:26:40.460001+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T10:24:42.023127+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T10:24:42.023127+0000","last_clean_scrub_stamp":"2024-04-23T10:24:42.023127+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T12:54:30.965080+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,11,0],"acting":[2,11,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.d","version":"89'7","reported_seq":26,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-23T10:26:40.522390+0000","last_change":"2024-04-23T10:24:44.061118+0000","last_active":"2024-04-23T10:26:40.522390+0000","last_peered":"2024-04-23T10:26:40.522390+0000","last_clean":"2024-04-23T10:26:40.522390+0000","last_became_active":"2024-04-23T10:24:44.060743+0000","last_became_peered":"2024-04-23T10:24:44.060743+0000","last_unstale":"2024-04-23T10:26:40.522390+0000","last_undegraded":"2024-04-23T10:26:40.522390+0000","last_fullsized":"2024-04-23T10:26:40.522390+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T10:24:43.025324+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T10:24:43.025324+0000","last_clean_scrub_stamp":"2024-04-23T10:24:43.025324+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T17:17:46.731121+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,11],"acting":[7,1,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.d","version":"90'39","reported_seq":68,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-23T10:26:40.480483+0000","last_change":"2024-04-23T10:24:43.052519+0000","last_active":"2024-04-23T10:26:40.480483+0000","last_peered":"2024-04-23T10:26:40.480483+0000","last_clean":"2024-04-23T10:26:40.480483+0000","last_became_active":"2024-04-23T10:24:43.052187+0000","last_became_peered":"2024-04-23T10:24:43.052187+0000","last_unstale":"2024-04-23T10:26:40.480483+0000","last_undegraded":"2024-04-23T10:26:40.480483+0000","last_fullsized":"2024-04-23T10:26:40.480483+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T10:24:42.023127+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T10:24:42.023127+0000","last_clean_scrub_stamp":"2024-04-23T10:24:42.023127+0000","objects_scrubbed":0,"log_size":39,"log_dups_size":0,"ondisk_log_size":39,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T15:51:52.839409+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":131670,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":11,"num_read_kb":10,"num_write":49,"num_write_kb":475,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,10,3],"acting":[5,10,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.c","version":"89'7","reported_seq":26,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-23T10:26:40.521835+0000","last_change":"2024-04-23T10:24:44.053086+0000","last_active":"2024-04-23T10:26:40.521835+0000","last_peered":"2024-04-23T10:26:40.521835+0000","last_clean":"2024-04-23T10:26:40.521835+0000","last_became_active":"2024-04-23T10:24:44.052687+0000","last_became_peered":"2024-04-23T10:24:44.052687+0000","last_unstale":"2024-04-23T10:26:40.521835+0000","last_undegraded":"2024-04-23T10:26:40.521835+0000","last_fullsized":"2024-04-23T10:26:40.521835+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T10:24:43.025324+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T10:24:43.025324+0000","last_clean_scrub_stamp":"2024-04-23T10:24:43.025324+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T14:35:44.958742+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":117,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,5],"acting":[7,3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.e","version":"86'2","reported_seq":33,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-23T10:26:40.278094+0000","last_change":"2024-04-23T10:25:47.724936+0000","last_active":"2024-04-23T10:26:40.278094+0000","last_peered":"2024-04-23T10:26:40.278094+0000","last_clean":"2024-04-23T10:26:40.278094+0000","last_became_active":"2024-04-23T10:25:47.722182+0000","last_became_peered":"2024-04-23T10:25:47.722182+0000","last_unstale":"2024-04-23T10:26:40.278094+0000","last_undegraded":"2024-04-23T10:26:40.278094+0000","last_fullsized":"2024-04-23T10:26:40.278094+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T10:24:42.023127+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T10:24:42.023127+0000","last_clean_scrub_stamp":"2024-04-23T10:24:42.023127+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T19:18:42.762648+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,4],"acting":[6,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.f","version":"89'9","reported_seq":28,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-23T10:26:40.522498+0000","last_change":"2024-04-23T10:24:44.049543+0000","last_active":"2024-04-23T10:26:40.522498+0000","last_peered":"2024-04-23T10:26:40.522498+0000","last_clean":"2024-04-23T10:26:40.522498+0000","last_became_active":"2024-04-23T10:24:44.049198+0000","last_became_peered":"2024-04-23T10:24:44.049198+0000","last_unstale":"2024-04-23T10:26:40.522498+0000","last_undegraded":"2024-04-23T10:26:40.522498+0000","last_fullsized":"2024-04-23T10:26:40.522498+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T10:24:43.025324+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T10:24:43.025324+0000","last_clean_scrub_stamp":"2024-04-23T10:24:43.025324+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T12:25:15.974267+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,11,8],"acting":[7,11,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.20","version":"89'9","reported_seq":28,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-23T10:26:40.460095+0000","last_change":"2024-04-23T10:24:44.054440+0000","last_active":"2024-04-23T10:26:40.460095+0000","last_peered":"2024-04-23T10:26:40.460095+0000","last_clean":"2024-04-23T10:26:40.460095+0000","last_became_active":"2024-04-23T10:24:44.053951+0000","last_became_peered":"2024-04-23T10:24:44.053951+0000","last_unstale":"2024-04-23T10:26:40.460095+0000","last_undegraded":"2024-04-23T10:26:40.460095+0000","last_fullsized":"2024-04-23T10:26:40.460095+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T10:24:43.025324+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T10:24:43.025324+0000","last_clean_scrub_stamp":"2024-04-23T10:24:43.025324+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T22:20:10.452712+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,2,5],"acting":[11,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.21","version":"89'12","reported_seq":31,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-23T10:26:40.459759+0000","last_change":"2024-04-23T10:24:44.059015+0000","last_active":"2024-04-23T10:26:40.459759+0000","last_peered":"2024-04-23T10:26:40.459759+0000","last_clean":"2024-04-23T10:26:40.459759+0000","last_became_active":"2024-04-23T10:24:44.058691+0000","last_became_peered":"2024-04-23T10:24:44.058691+0000","last_unstale":"2024-04-23T10:26:40.459759+0000","last_undegraded":"2024-04-23T10:26:40.459759+0000","last_fullsized":"2024-04-23T10:26:40.459759+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T10:24:43.025324+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T10:24:43.025324+0000","last_clean_scrub_stamp":"2024-04-23T10:24:43.025324+0000","objects_scrubbed":0,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T14:57:48.153094+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,4],"acting":[2,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.22","version":"89'8","reported_seq":27,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-23T10:26:40.459657+0000","last_change":"2024-04-23T10:24:44.055534+0000","last_active":"2024-04-23T10:26:40.459657+0000","last_peered":"2024-04-23T10:26:40.459657+0000","last_clean":"2024-04-23T10:26:40.459657+0000","last_became_active":"2024-04-23T10:24:44.055181+0000","last_became_peered":"2024-04-23T10:24:44.055181+0000","last_unstale":"2024-04-23T10:26:40.459657+0000","last_undegraded":"2024-04-23T10:26:40.459657+0000","last_fullsized":"2024-04-23T10:26:40.459657+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T10:24:43.025324+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T10:24:43.025324+0000","last_clean_scrub_stamp":"2024-04-23T10:24:43.025324+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T13:56:33.878232+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.23","version":"89'7","reported_seq":32,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-23T10:26:40.460085+0000","last_change":"2024-04-23T10:25:47.587380+0000","last_active":"2024-04-23T10:26:40.460085+0000","last_peered":"2024-04-23T10:26:40.460085+0000","last_clean":"2024-04-23T10:26:40.460085+0000","last_became_active":"2024-04-23T10:25:47.586893+0000","last_became_peered":"2024-04-23T10:25:47.586893+0000","last_unstale":"2024-04-23T10:26:40.460085+0000","last_undegraded":"2024-04-23T10:26:40.460085+0000","last_fullsized":"2024-04-23T10:26:40.460085+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T10:24:43.025324+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T10:24:43.025324+0000","last_clean_scrub_stamp":"2024-04-23T10:24:43.025324+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T10:51:34.390594+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,7],"acting":[0,6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.24","version":"89'4","reported_seq":23,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-23T10:26:40.278003+0000","last_change":"2024-04-23T10:24:44.050741+0000","last_active":"2024-04-23T10:26:40.278003+0000","last_peered":"2024-04-23T10:26:40.278003+0000","last_clean":"2024-04-23T10:26:40.278003+0000","last_became_active":"2024-04-23T10:24:44.050398+0000","last_became_peered":"2024-04-23T10:24:44.050398+0000","last_unstale":"2024-04-23T10:26:40.278003+0000","last_undegraded":"2024-04-23T10:26:40.278003+0000","last_fullsized":"2024-04-23T10:26:40.278003+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T10:24:43.025324+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T10:24:43.025324+0000","last_clean_scrub_stamp":"2024-04-23T10:24:43.025324+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T10:49:07.146673+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,0],"acting":[6,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.25","version":"89'6","reported_seq":25,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-23T10:26:40.460199+0000","last_change":"2024-04-23T10:24:44.048933+0000","last_active":"2024-04-23T10:26:40.460199+0000","last_peered":"2024-04-23T10:26:40.460199+0000","last_clean":"2024-04-23T10:26:40.460199+0000","last_became_active":"2024-04-23T10:24:44.048553+0000","last_became_peered":"2024-04-23T10:24:44.048553+0000","last_unstale":"2024-04-23T10:26:40.460199+0000","last_undegraded":"2024-04-23T10:26:40.460199+0000","last_fullsized":"2024-04-23T10:26:40.460199+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T10:24:43.025324+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T10:24:43.025324+0000","last_clean_scrub_stamp":"2024-04-23T10:24:43.025324+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T10:40:24.569906+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,8,5],"acting":[11,8,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.26","version":"89'15","reported_seq":34,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-23T10:26:40.460160+0000","last_change":"2024-04-23T10:24:44.051895+0000","last_active":"2024-04-23T10:26:40.460160+0000","last_peered":"2024-04-23T10:26:40.460160+0000","last_clean":"2024-04-23T10:26:40.460160+0000","last_became_active":"2024-04-23T10:24:44.051005+0000","last_became_peered":"2024-04-23T10:24:44.051005+0000","last_unstale":"2024-04-23T10:26:40.460160+0000","last_undegraded":"2024-04-23T10:26:40.460160+0000","last_fullsized":"2024-04-23T10:26:40.460160+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T10:24:43.025324+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T10:24:43.025324+0000","last_clean_scrub_stamp":"2024-04-23T10:24:43.025324+0000","objects_scrubbed":0,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T18:13:05.227365+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,8,1],"acting":[2,8,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.27","version":"89'2","reported_seq":27,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-23T10:26:40.374327+0000","last_change":"2024-04-23T10:25:47.587504+0000","last_active":"2024-04-23T10:26:40.374327+0000","last_peered":"2024-04-23T10:26:40.374327+0000","last_clean":"2024-04-23T10:26:40.374327+0000","last_became_active":"2024-04-23T10:25:47.587157+0000","last_became_peered":"2024-04-23T10:25:47.587157+0000","last_unstale":"2024-04-23T10:26:40.374327+0000","last_undegraded":"2024-04-23T10:26:40.374327+0000","last_fullsized":"2024-04-23T10:26:40.374327+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T10:24:43.025324+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T10:24:43.025324+0000","last_clean_scrub_stamp":"2024-04-23T10:24:43.025324+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T18:46:25.090568+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,2],"acting":[3,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.28","version":"89'6","reported_seq":25,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-23T10:26:40.281576+0000","last_change":"2024-04-23T10:24:44.044158+0000","last_active":"2024-04-23T10:26:40.281576+0000","last_peered":"2024-04-23T10:26:40.281576+0000","last_clean":"2024-04-23T10:26:40.281576+0000","last_became_active":"2024-04-23T10:24:44.043829+0000","last_became_peered":"2024-04-23T10:24:44.043829+0000","last_unstale":"2024-04-23T10:26:40.281576+0000","last_undegraded":"2024-04-23T10:26:40.281576+0000","last_fullsized":"2024-04-23T10:26:40.281576+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T10:24:43.025324+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T10:24:43.025324+0000","last_clean_scrub_stamp":"2024-04-23T10:24:43.025324+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T15:10:08.081831+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,9,3],"acting":[5,9,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.29","version":"89'8","reported_seq":27,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-23T10:26:40.373723+0000","last_change":"2024-04-23T10:24:44.051267+0000","last_active":"2024-04-23T10:26:40.373723+0000","last_peered":"2024-04-23T10:26:40.373723+0000","last_clean":"2024-04-23T10:26:40.373723+0000","last_became_active":"2024-04-23T10:24:44.050911+0000","last_became_peered":"2024-04-23T10:24:44.050911+0000","last_unstale":"2024-04-23T10:26:40.373723+0000","last_undegraded":"2024-04-23T10:26:40.373723+0000","last_fullsized":"2024-04-23T10:26:40.373723+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T10:24:43.025324+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T10:24:43.025324+0000","last_clean_scrub_stamp":"2024-04-23T10:24:43.025324+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T18:09:28.871427+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,11],"acting":[3,2,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.2a","version":"89'10","reported_seq":29,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-23T10:26:40.281065+0000","last_change":"2024-04-23T10:24:44.057653+0000","last_active":"2024-04-23T10:26:40.281065+0000","last_peered":"2024-04-23T10:26:40.281065+0000","last_clean":"2024-04-23T10:26:40.281065+0000","last_became_active":"2024-04-23T10:24:44.056627+0000","last_became_peered":"2024-04-23T10:24:44.056627+0000","last_unstale":"2024-04-23T10:26:40.281065+0000","last_undegraded":"2024-04-23T10:26:40.281065+0000","last_fullsized":"2024-04-23T10:26:40.281065+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T10:24:43.025324+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T10:24:43.025324+0000","last_clean_scrub_stamp":"2024-04-23T10:24:43.025324+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T18:29:21.480187+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,3],"acting":[1,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.2b","version":"89'11","reported_seq":30,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-23T10:26:40.278095+0000","last_change":"2024-04-23T10:24:44.059735+0000","last_active":"2024-04-23T10:26:40.278095+0000","last_peered":"2024-04-23T10:26:40.278095+0000","last_clean":"2024-04-23T10:26:40.278095+0000","last_became_active":"2024-04-23T10:24:44.059052+0000","last_became_peered":"2024-04-23T10:24:44.059052+0000","last_unstale":"2024-04-23T10:26:40.278095+0000","last_undegraded":"2024-04-23T10:26:40.278095+0000","last_fullsized":"2024-04-23T10:26:40.278095+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T10:24:43.025324+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T10:24:43.025324+0000","last_clean_scrub_stamp":"2024-04-23T10:24:43.025324+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T14:29:25.794655+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,0],"acting":[6,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.2c","version":"89'8","reported_seq":27,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-23T10:26:40.459891+0000","last_change":"2024-04-23T10:24:44.051831+0000","last_active":"2024-04-23T10:26:40.459891+0000","last_peered":"2024-04-23T10:26:40.459891+0000","last_clean":"2024-04-23T10:26:40.459891+0000","last_became_active":"2024-04-23T10:24:44.051480+0000","last_became_peered":"2024-04-23T10:24:44.051480+0000","last_unstale":"2024-04-23T10:26:40.459891+0000","last_undegraded":"2024-04-23T10:26:40.459891+0000","last_fullsized":"2024-04-23T10:26:40.459891+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T10:24:43.025324+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T10:24:43.025324+0000","last_clean_scrub_stamp":"2024-04-23T10:24:43.025324+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T19:20:55.181393+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,4,8],"acting":[11,4,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]}],"pool_stats":[{"poolid":3,"num_pg":64,"stat_sum":{"num_bytes":234,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":702,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":502,"ondisk_log_size":502,"up":192,"acting":192,"num_store_stats":12},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":136,"num_read_kb":119,"num_write":181,"num_write_kb":2938,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":1180736,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1781760,"data_stored":1771104,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":137,"ondisk_log_size":137,"up":3,"acting":3,"num_store_stats":3},{"poolid":2,"num_pg":32,"stat_sum":{"num_bytes":598030,"num_objects":62,"num_object_clones":0,"num_object_copies":186,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":62,"num_whiteouts":0,"num_read":802,"num_read_kb":510,"num_write":285,"num_write_kb":1437,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":10,"num_objects_omap":40,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":2027520,"data_stored":1794090,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":226,"ondisk_log_size":226,"up":96,"acting":96,"num_store_stats":12}],"osd_stats":[{"osd":11,"up_from":82,"seq":352187318309,"num_pgs":19,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32148,"kb_used_data":968,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93713004,"statfs":{"total":95995035648,"available":95962116096,"internally_reserved":0,"allocated":991232,"data_stored":474591,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1596,"internal_metadata":31914436},"hb_peers":[0,1,2,4,5,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":10,"up_from":74,"seq":317827579945,"num_pgs":26,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28656,"kb_used_data":1640,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93716496,"statfs":{"total":95995035648,"available":95965691904,"internally_reserved":0,"allocated":1679360,"data_stored":1124595,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1600,"internal_metadata":27654592},"hb_peers":[0,1,2,3,4,5,6,8,9,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":9,"up_from":69,"seq":296352743471,"num_pgs":20,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28196,"kb_used_data":1176,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93716956,"statfs":{"total":95995035648,"available":95966162944,"internally_reserved":0,"allocated":1204224,"data_stored":671435,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1600,"internal_metadata":27654592},"hb_peers":[0,1,2,3,4,5,6,7,8,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":8,"up_from":62,"seq":266287972405,"num_pgs":25,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32548,"kb_used_data":1368,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93712604,"statfs":{"total":95995035648,"available":95961706496,"internally_reserved":0,"allocated":1400832,"data_stored":867963,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,2,3,4,5,6,7,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":7,"up_from":53,"seq":227633266745,"num_pgs":26,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":36904,"kb_used_data":1564,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":93708248,"statfs":{"total":95995035648,"available":95957245952,"internally_reserved":0,"allocated":1601536,"data_stored":1068132,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":36174282},"hb_peers":[1,2,3,4,5,6,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":6,"up_from":47,"seq":201863462975,"num_pgs":22,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32152,"kb_used_data":972,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93713000,"statfs":{"total":95995035648,"available":95962112000,"internally_reserved":0,"allocated":995328,"data_stored":474216,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":31914438},"hb_peers":[0,1,2,3,4,5,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":13,"seq":55834574934,"num_pgs":24,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28060,"kb_used_data":1040,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717092,"statfs":{"total":95995035648,"available":95966302208,"internally_reserved":0,"allocated":1064960,"data_stored":541767,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,2,3,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":8,"seq":34359738460,"num_pgs":25,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":36332,"kb_used_data":992,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":93708820,"statfs":{"total":95995035648,"available":95957831680,"internally_reserved":0,"allocated":1015808,"data_stored":475487,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":36174278},"hb_peers":[1,2,3,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":18,"seq":77309411410,"num_pgs":27,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":36404,"kb_used_data":1064,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":93708748,"statfs":{"total":95995035648,"available":95957757952,"internally_reserved":0,"allocated":1089536,"data_stored":544040,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":36174277},"hb_peers":[0,1,3,4,5,6,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":25,"seq":107374182474,"num_pgs":26,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":27988,"kb_used_data":972,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717164,"statfs":{"total":95995035648,"available":95966375936,"internally_reserved":0,"allocated":995328,"data_stored":465542,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[0,1,2,4,5,6,7,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":4,"up_from":33,"seq":141733920839,"num_pgs":23,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":27996,"kb_used_data":976,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717156,"statfs":{"total":95995035648,"available":95966367744,"internally_reserved":0,"allocated":999424,"data_stored":475185,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[1,2,3,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":39,"seq":167503724611,"num_pgs":28,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":37044,"kb_used_data":1708,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":93708108,"statfs":{"total":95995035648,"available":95957102592,"internally_reserved":0,"allocated":1748992,"data_stored":1198493,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":36174277},"hb_peers":[0,1,2,3,4,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":28672,"data_stored":1478,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":77824,"data_stored":67758,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":102400,"data_stored":70031,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":544768,"data_stored":525100,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":1567,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":163840,"data_stored":133999,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":90,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":3638,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":413696,"data_stored":393954,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":413696,"data_stored":394034,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":245760,"data_stored":201859,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":582,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":234,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":117,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":117,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":117,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":117,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-04-23T10:26:45.808 INFO:tasks.quiescer.ceph_manager:clean! 2024-04-23T10:26:45.809 DEBUG:teuthology.orchestra.run.smithi114:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-04-23T10:26:45.948 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:45.947+0000 7fed0a337640 1 -- 172.21.15.114:0/3941172935 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fed04101a50 msgr2=0x7fed04105a90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T10:26:45.948 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:45.947+0000 7fed0a337640 1 --2- 172.21.15.114:0/3941172935 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fed04101a50 0x7fed04105a90 secure :-1 s=READY pgs=448 cs=0 l=1 rev1=1 crypto rx=0x7fecf8009900 tx=0x7fecf802a410 comp rx=0 tx=0).stop 2024-04-23T10:26:45.948 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:45.947+0000 7fed0a337640 1 -- 172.21.15.114:0/3941172935 shutdown_connections 2024-04-23T10:26:45.948 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:45.947+0000 7fed0a337640 1 --2- 172.21.15.114:0/3941172935 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fed04106170 0x7fed0410dd40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:26:45.948 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:45.947+0000 7fed0a337640 1 --2- 172.21.15.114:0/3941172935 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fed04101a50 0x7fed04105a90 unknown :-1 s=CLOSED pgs=448 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:26:45.949 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:45.947+0000 7fed0a337640 1 --2- 172.21.15.114:0/3941172935 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fed041010a0 0x7fed04101480 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:26:45.949 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:45.947+0000 7fed0a337640 1 -- 172.21.15.114:0/3941172935 >> 172.21.15.114:0/3941172935 conn(0x7fed04077f50 msgr2=0x7fed04078360 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T10:26:45.949 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:45.947+0000 7fed0a337640 1 -- 172.21.15.114:0/3941172935 shutdown_connections 2024-04-23T10:26:45.949 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:45.947+0000 7fed0a337640 1 -- 172.21.15.114:0/3941172935 wait complete. 2024-04-23T10:26:45.949 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:45.948+0000 7fed0a337640 1 Processor -- start 2024-04-23T10:26:45.949 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:45.948+0000 7fed0a337640 1 -- start start 2024-04-23T10:26:45.949 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:45.948+0000 7fed0a337640 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fed041010a0 0x7fed041995f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T10:26:45.949 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:45.948+0000 7fed0a337640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fed04106170 0x7fed04199b30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T10:26:45.949 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:45.948+0000 7fed0a337640 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fed0419a0c0 0x7fed041a4670 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T10:26:45.949 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:45.948+0000 7fed0a337640 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x7fed041110b0 con 0x7fed04106170 2024-04-23T10:26:45.949 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:45.948+0000 7fed0a337640 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 -- 0x7fed04110f30 con 0x7fed0419a0c0 2024-04-23T10:26:45.949 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:45.948+0000 7fed0a337640 1 -- --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_getmap magic: 0 -- 0x7fed04111230 con 0x7fed041010a0 2024-04-23T10:26:45.949 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:45.948+0000 7fed037fe640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fed04106170 0x7fed04199b30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T10:26:45.949 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:45.948+0000 7fed037fe640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fed04106170 0x7fed04199b30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.114:51884/0 (socket says 172.21.15.114:51884) 2024-04-23T10:26:45.950 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:45.948+0000 7fed037fe640 1 -- 172.21.15.114:0/2287845770 learned_addr learned my addr 172.21.15.114:0/2287845770 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-04-23T10:26:45.950 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:45.948+0000 7fed03fff640 1 --2- 172.21.15.114:0/2287845770 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fed041010a0 0x7fed041995f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T10:26:45.950 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:45.948+0000 7fed088d4640 1 --2- 172.21.15.114:0/2287845770 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fed0419a0c0 0x7fed041a4670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T10:26:45.950 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:45.948+0000 7fed037fe640 1 -- 172.21.15.114:0/2287845770 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fed041010a0 msgr2=0x7fed041995f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T10:26:45.950 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:45.948+0000 7fed037fe640 1 --2- 172.21.15.114:0/2287845770 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fed041010a0 0x7fed041995f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:26:45.950 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:45.948+0000 7fed037fe640 1 -- 172.21.15.114:0/2287845770 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fed0419a0c0 msgr2=0x7fed041a4670 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T10:26:45.950 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:45.948+0000 7fed037fe640 1 --2- 172.21.15.114:0/2287845770 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fed0419a0c0 0x7fed041a4670 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:26:45.950 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:45.948+0000 7fed037fe640 1 -- 172.21.15.114:0/2287845770 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fed041a4e40 con 0x7fed04106170 2024-04-23T10:26:45.950 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:45.948+0000 7fed03fff640 1 --2- 172.21.15.114:0/2287845770 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fed041010a0 0x7fed041995f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-23T10:26:45.950 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:45.949+0000 7fed037fe640 1 --2- 172.21.15.114:0/2287845770 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fed04106170 0x7fed04199b30 secure :-1 s=READY pgs=449 cs=0 l=1 rev1=1 crypto rx=0x7fecf8006010 tx=0x7fecf8004060 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T10:26:45.950 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:45.949+0000 7fed017fa640 1 -- 172.21.15.114:0/2287845770 <== mon.0 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fecf803f030 con 0x7fed04106170 2024-04-23T10:26:45.950 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:45.949+0000 7fed0a337640 1 -- 172.21.15.114:0/2287845770 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fed041a50d0 con 0x7fed04106170 2024-04-23T10:26:45.950 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:45.949+0000 7fed017fa640 1 -- 172.21.15.114:0/2287845770 <== mon.0 v2:172.21.15.114:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fecf8003aa0 con 0x7fed04106170 2024-04-23T10:26:45.951 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:45.949+0000 7fed017fa640 1 -- 172.21.15.114:0/2287845770 <== mon.0 v2:172.21.15.114:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fecf8035070 con 0x7fed04106170 2024-04-23T10:26:45.951 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:45.949+0000 7fed0a337640 1 -- 172.21.15.114:0/2287845770 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fed041a55c0 con 0x7fed04106170 2024-04-23T10:26:45.951 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:45.950+0000 7fed0a337640 1 -- 172.21.15.114:0/2287845770 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fed04068d70 con 0x7fed04106170 2024-04-23T10:26:45.952 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:45.951+0000 7fed017fa640 1 -- 172.21.15.114:0/2287845770 <== mon.0 v2:172.21.15.114:3300/0 4 ==== mgrmap(e 24) ==== 138823+0+0 (secure 0 0 0) 0x7fecf8042030 con 0x7fed04106170 2024-04-23T10:26:45.952 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:45.952+0000 7fed017fa640 1 --2- 172.21.15.114:0/2287845770 >> [v2:172.21.15.114:6800/1395693466,v1:172.21.15.114:6801/1395693466] conn(0x7fecec0a3db0 0x7fecec0a6270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T10:26:45.952 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:45.952+0000 7fed03fff640 1 --2- 172.21.15.114:0/2287845770 >> [v2:172.21.15.114:6800/1395693466,v1:172.21.15.114:6801/1395693466] conn(0x7fecec0a3db0 0x7fecec0a6270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T10:26:45.952 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:45.952+0000 7fed017fa640 1 -- 172.21.15.114:0/2287845770 <== mon.0 v2:172.21.15.114:3300/0 5 ==== osd_map(90..90 src has 1..90) ==== 8527+0+0 (secure 0 0 0) 0x7fecf80f0ea0 con 0x7fed04106170 2024-04-23T10:26:45.952 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:45.952+0000 7fed03fff640 1 --2- 172.21.15.114:0/2287845770 >> [v2:172.21.15.114:6800/1395693466,v1:172.21.15.114:6801/1395693466] conn(0x7fecec0a3db0 0x7fecec0a6270 secure :-1 s=READY pgs=50 cs=0 l=1 rev1=1 crypto rx=0x7fecf4009c20 tx=0x7fecf4005f90 comp rx=0 tx=0).ready entity=mgr.15291 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T10:26:45.955 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:45.956+0000 7fed017fa640 1 -- 172.21.15.114:0/2287845770 <== mon.0 v2:172.21.15.114:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7fecf80bd3d0 con 0x7fed04106170 2024-04-23T10:26:46.131 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:46.133+0000 7fed0a337640 1 -- 172.21.15.114:0/2287845770 --> [v2:172.21.15.114:6800/1395693466,v1:172.21.15.114:6801/1395693466] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) -- 0x7fed04109410 con 0x7fecec0a3db0 2024-04-23T10:26:46.138 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:46.140+0000 7fed017fa640 1 -- 172.21.15.114:0/2287845770 <== mgr.15291 v2:172.21.15.114:6800/1395693466 1 ==== mgr_command_reply(tid 0: 0 dumped all) ==== 18+0+257730 (secure 0 0 0) 0x7fecf4009950 con 0x7fecec0a3db0 2024-04-23T10:26:46.141 INFO:teuthology.orchestra.run.smithi114.stdout: 2024-04-23T10:26:46.143 INFO:teuthology.orchestra.run.smithi114.stderr:dumped all 2024-04-23T10:26:46.144 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:46.143+0000 7fed0a337640 1 -- 172.21.15.114:0/2287845770 >> [v2:172.21.15.114:6800/1395693466,v1:172.21.15.114:6801/1395693466] conn(0x7fecec0a3db0 msgr2=0x7fecec0a6270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T10:26:46.144 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:46.143+0000 7fed0a337640 1 --2- 172.21.15.114:0/2287845770 >> [v2:172.21.15.114:6800/1395693466,v1:172.21.15.114:6801/1395693466] conn(0x7fecec0a3db0 0x7fecec0a6270 secure :-1 s=READY pgs=50 cs=0 l=1 rev1=1 crypto rx=0x7fecf4009c20 tx=0x7fecf4005f90 comp rx=0 tx=0).stop 2024-04-23T10:26:46.144 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:46.143+0000 7fed0a337640 1 -- 172.21.15.114:0/2287845770 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fed04106170 msgr2=0x7fed04199b30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T10:26:46.144 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:46.143+0000 7fed0a337640 1 --2- 172.21.15.114:0/2287845770 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fed04106170 0x7fed04199b30 secure :-1 s=READY pgs=449 cs=0 l=1 rev1=1 crypto rx=0x7fecf8006010 tx=0x7fecf8004060 comp rx=0 tx=0).stop 2024-04-23T10:26:46.144 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:46.143+0000 7fed0a337640 1 -- 172.21.15.114:0/2287845770 shutdown_connections 2024-04-23T10:26:46.145 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:46.143+0000 7fed0a337640 1 --2- 172.21.15.114:0/2287845770 >> [v2:172.21.15.114:6800/1395693466,v1:172.21.15.114:6801/1395693466] conn(0x7fecec0a3db0 0x7fecec0a6270 unknown :-1 s=CLOSED pgs=50 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:26:46.145 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:46.143+0000 7fed0a337640 1 --2- 172.21.15.114:0/2287845770 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fed0419a0c0 0x7fed041a4670 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:26:46.145 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:46.143+0000 7fed0a337640 1 --2- 172.21.15.114:0/2287845770 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fed04106170 0x7fed04199b30 unknown :-1 s=CLOSED pgs=449 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:26:46.145 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:46.143+0000 7fed0a337640 1 --2- 172.21.15.114:0/2287845770 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fed041010a0 0x7fed041995f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:26:46.145 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:46.143+0000 7fed0a337640 1 -- 172.21.15.114:0/2287845770 >> 172.21.15.114:0/2287845770 conn(0x7fed04077f50 msgr2=0x7fed040ff150 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T10:26:46.145 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:46.143+0000 7fed0a337640 1 -- 172.21.15.114:0/2287845770 shutdown_connections 2024-04-23T10:26:46.145 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:46.143+0000 7fed0a337640 1 -- 172.21.15.114:0/2287845770 wait complete. 2024-04-23T10:26:46.152 INFO:journalctl@ceph.mon.a.smithi114.stdout:Apr 23 10:26:46 smithi114 ceph-mon[29255]: Updating smithi204:/etc/ceph/ceph.client.admin.keyring 2024-04-23T10:26:46.152 INFO:journalctl@ceph.mon.a.smithi114.stdout:Apr 23 10:26:46 smithi114 ceph-mon[29255]: Updating smithi114:/etc/ceph/ceph.client.admin.keyring 2024-04-23T10:26:46.152 INFO:journalctl@ceph.mon.a.smithi114.stdout:Apr 23 10:26:46 smithi114 ceph-mon[29255]: Updating smithi119:/etc/ceph/ceph.client.admin.keyring 2024-04-23T10:26:46.152 INFO:journalctl@ceph.mon.a.smithi114.stdout:Apr 23 10:26:46 smithi114 ceph-mon[29255]: pgmap v5: 97 pgs: 97 active+clean; 1.1 MiB data, 375 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-23T10:26:46.152 INFO:journalctl@ceph.mon.a.smithi114.stdout:Apr 23 10:26:46 smithi114 ceph-mon[29255]: Updating smithi204:/var/lib/ceph/aa240572-0159-11ef-bc93-c7b262605968/config/ceph.client.admin.keyring 2024-04-23T10:26:46.152 INFO:journalctl@ceph.mon.a.smithi114.stdout:Apr 23 10:26:46 smithi114 ceph-mon[29255]: Updating smithi114:/var/lib/ceph/aa240572-0159-11ef-bc93-c7b262605968/config/ceph.client.admin.keyring 2024-04-23T10:26:46.152 INFO:journalctl@ceph.mon.a.smithi114.stdout:Apr 23 10:26:46 smithi114 ceph-mon[29255]: Updating smithi119:/var/lib/ceph/aa240572-0159-11ef-bc93-c7b262605968/config/ceph.client.admin.keyring 2024-04-23T10:26:46.152 INFO:journalctl@ceph.mon.a.smithi114.stdout:Apr 23 10:26:46 smithi114 ceph-mon[29255]: mgrmap e24: x(active, since 4s), standbys: z, y 2024-04-23T10:26:46.152 INFO:journalctl@ceph.mon.a.smithi114.stdout:Apr 23 10:26:46 smithi114 ceph-mon[29255]: from='client.? 172.21.15.114:0/3698241698' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2024-04-23T10:26:46.153 INFO:teuthology.orchestra.run.smithi114.stdout:{"pg_ready":true,"pg_map":{"version":5,"stamp":"2024-04-23T10:26:44.290805+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":1188632,"num_objects":66,"num_object_clones":0,"num_object_copies":198,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":66,"num_whiteouts":0,"num_read":938,"num_read_kb":629,"num_write":468,"num_write_kb":4377,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":7,"num_bytes_recovered":1180736,"num_keys_recovered":10,"num_objects_omap":40,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":865,"ondisk_log_size":865,"up":291,"acting":291,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":291,"num_osds":12,"num_per_pool_osds":12,"num_per_pool_omap_osds":12,"kb":1124941824,"kb_used":384428,"kb_used_data":14440,"kb_used_omap":18,"kb_used_meta":369837,"kb_avail":1124557396,"statfs":{"total":1151940427776,"available":1151546773504,"internally_reserved":0,"allocated":14786560,"data_stored":8381446,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":19142,"internal_metadata":378713402},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"2.989676"},"pg_stats":[{"pgid":"3.3f","version":"89'8","reported_seq":27,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-23T10:26:40.460272+0000","last_change":"2024-04-23T10:24:44.051465+0000","last_active":"2024-04-23T10:26:40.460272+0000","last_peered":"2024-04-23T10:26:40.460272+0000","last_clean":"2024-04-23T10:26:40.460272+0000","last_became_active":"2024-04-23T10:24:44.050863+0000","last_became_peered":"2024-04-23T10:24:44.050863+0000","last_unstale":"2024-04-23T10:26:40.460272+0000","last_undegraded":"2024-04-23T10:26:40.460272+0000","last_fullsized":"2024-04-23T10:26:40.460272+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T10:24:43.025324+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T10:24:43.025324+0000","last_clean_scrub_stamp":"2024-04-23T10:24:43.025324+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T15:11:12.508458+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,1,0],"acting":[11,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.3e","version":"89'4","reported_seq":23,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-23T10:26:40.459824+0000","last_change":"2024-04-23T10:24:44.050739+0000","last_active":"2024-04-23T10:26:40.459824+0000","last_peered":"2024-04-23T10:26:40.459824+0000","last_clean":"2024-04-23T10:26:40.459824+0000","last_became_active":"2024-04-23T10:24:44.050430+0000","last_became_peered":"2024-04-23T10:24:44.050430+0000","last_unstale":"2024-04-23T10:26:40.459824+0000","last_undegraded":"2024-04-23T10:26:40.459824+0000","last_fullsized":"2024-04-23T10:26:40.459824+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T10:24:43.025324+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T10:24:43.025324+0000","last_clean_scrub_stamp":"2024-04-23T10:24:43.025324+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T17:36:25.983147+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,11,10],"acting":[0,11,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3d","version":"89'7","reported_seq":26,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-23T10:26:40.460358+0000","last_change":"2024-04-23T10:24:44.054805+0000","last_active":"2024-04-23T10:26:40.460358+0000","last_peered":"2024-04-23T10:26:40.460358+0000","last_clean":"2024-04-23T10:26:40.460358+0000","last_became_active":"2024-04-23T10:24:44.054423+0000","last_became_peered":"2024-04-23T10:24:44.054423+0000","last_unstale":"2024-04-23T10:26:40.460358+0000","last_undegraded":"2024-04-23T10:26:40.460358+0000","last_fullsized":"2024-04-23T10:26:40.460358+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T10:24:43.025324+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T10:24:43.025324+0000","last_clean_scrub_stamp":"2024-04-23T10:24:43.025324+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T21:34:26.793624+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,1],"acting":[0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3c","version":"89'8","reported_seq":27,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-23T10:26:40.459685+0000","last_change":"2024-04-23T10:24:44.055966+0000","last_active":"2024-04-23T10:26:40.459685+0000","last_peered":"2024-04-23T10:26:40.459685+0000","last_clean":"2024-04-23T10:26:40.459685+0000","last_became_active":"2024-04-23T10:24:44.055028+0000","last_became_peered":"2024-04-23T10:24:44.055028+0000","last_unstale":"2024-04-23T10:26:40.459685+0000","last_undegraded":"2024-04-23T10:26:40.459685+0000","last_fullsized":"2024-04-23T10:26:40.459685+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T10:24:43.025324+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T10:24:43.025324+0000","last_clean_scrub_stamp":"2024-04-23T10:24:43.025324+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T19:41:04.594025+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,8,2],"acting":[4,8,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.3b","version":"89'4","reported_seq":23,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-23T10:26:40.373641+0000","last_change":"2024-04-23T10:24:44.046954+0000","last_active":"2024-04-23T10:26:40.373641+0000","last_peered":"2024-04-23T10:26:40.373641+0000","last_clean":"2024-04-23T10:26:40.373641+0000","last_became_active":"2024-04-23T10:24:44.046631+0000","last_became_peered":"2024-04-23T10:24:44.046631+0000","last_unstale":"2024-04-23T10:26:40.373641+0000","last_undegraded":"2024-04-23T10:26:40.373641+0000","last_fullsized":"2024-04-23T10:26:40.373641+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T10:24:43.025324+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T10:24:43.025324+0000","last_clean_scrub_stamp":"2024-04-23T10:24:43.025324+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T15:04:26.878894+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,3,6],"acting":[8,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.3a","version":"89'6","reported_seq":25,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-23T10:26:40.381329+0000","last_change":"2024-04-23T10:24:44.051550+0000","last_active":"2024-04-23T10:26:40.381329+0000","last_peered":"2024-04-23T10:26:40.381329+0000","last_clean":"2024-04-23T10:26:40.381329+0000","last_became_active":"2024-04-23T10:24:44.050469+0000","last_became_peered":"2024-04-23T10:24:44.050469+0000","last_unstale":"2024-04-23T10:26:40.381329+0000","last_undegraded":"2024-04-23T10:26:40.381329+0000","last_fullsized":"2024-04-23T10:26:40.381329+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T10:24:43.025324+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T10:24:43.025324+0000","last_clean_scrub_stamp":"2024-04-23T10:24:43.025324+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T21:13:42.469252+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,9,4],"acting":[10,9,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.39","version":"89'5","reported_seq":30,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-23T10:26:40.521664+0000","last_change":"2024-04-23T10:25:47.588865+0000","last_active":"2024-04-23T10:26:40.521664+0000","last_peered":"2024-04-23T10:26:40.521664+0000","last_clean":"2024-04-23T10:26:40.521664+0000","last_became_active":"2024-04-23T10:25:47.588571+0000","last_became_peered":"2024-04-23T10:25:47.588571+0000","last_unstale":"2024-04-23T10:26:40.521664+0000","last_undegraded":"2024-04-23T10:26:40.521664+0000","last_fullsized":"2024-04-23T10:26:40.521664+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T10:24:43.025324+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T10:24:43.025324+0000","last_clean_scrub_stamp":"2024-04-23T10:24:43.025324+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T17:14:00.026999+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,9,8],"acting":[7,9,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.38","version":"89'9","reported_seq":22,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-23T10:26:40.371622+0000","last_change":"2024-04-23T10:25:47.722416+0000","last_active":"2024-04-23T10:26:40.371622+0000","last_peered":"2024-04-23T10:26:40.371622+0000","last_clean":"2024-04-23T10:26:40.371622+0000","last_became_active":"2024-04-23T10:25:47.722015+0000","last_became_peered":"2024-04-23T10:25:47.722015+0000","last_unstale":"2024-04-23T10:26:40.371622+0000","last_undegraded":"2024-04-23T10:26:40.371622+0000","last_fullsized":"2024-04-23T10:26:40.371622+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T10:24:43.025324+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T10:24:43.025324+0000","last_clean_scrub_stamp":"2024-04-23T10:24:43.025324+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T17:02:57.881164+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,5,10],"acting":[9,5,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.37","version":"89'11","reported_seq":30,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-23T10:26:40.281426+0000","last_change":"2024-04-23T10:24:44.055261+0000","last_active":"2024-04-23T10:26:40.281426+0000","last_peered":"2024-04-23T10:26:40.281426+0000","last_clean":"2024-04-23T10:26:40.281426+0000","last_became_active":"2024-04-23T10:24:44.054405+0000","last_became_peered":"2024-04-23T10:24:44.054405+0000","last_unstale":"2024-04-23T10:26:40.281426+0000","last_undegraded":"2024-04-23T10:26:40.281426+0000","last_fullsized":"2024-04-23T10:26:40.281426+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T10:24:43.025324+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T10:24:43.025324+0000","last_clean_scrub_stamp":"2024-04-23T10:24:43.025324+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T19:30:30.306739+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,11],"acting":[1,5,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.36","version":"89'16","reported_seq":35,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-23T10:26:40.381657+0000","last_change":"2024-04-23T10:24:44.055854+0000","last_active":"2024-04-23T10:26:40.381657+0000","last_peered":"2024-04-23T10:26:40.381657+0000","last_clean":"2024-04-23T10:26:40.381657+0000","last_became_active":"2024-04-23T10:24:44.055275+0000","last_became_peered":"2024-04-23T10:24:44.055275+0000","last_unstale":"2024-04-23T10:26:40.381657+0000","last_undegraded":"2024-04-23T10:26:40.381657+0000","last_fullsized":"2024-04-23T10:26:40.381657+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T10:24:43.025324+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T10:24:43.025324+0000","last_clean_scrub_stamp":"2024-04-23T10:24:43.025324+0000","objects_scrubbed":0,"log_size":16,"log_dups_size":0,"ondisk_log_size":16,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T21:48:25.934878+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,2,4],"acting":[10,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.35","version":"89'6","reported_seq":25,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-23T10:26:40.522323+0000","last_change":"2024-04-23T10:24:44.061314+0000","last_active":"2024-04-23T10:26:40.522323+0000","last_peered":"2024-04-23T10:26:40.522323+0000","last_clean":"2024-04-23T10:26:40.522323+0000","last_became_active":"2024-04-23T10:24:44.060981+0000","last_became_peered":"2024-04-23T10:24:44.060981+0000","last_unstale":"2024-04-23T10:26:40.522323+0000","last_undegraded":"2024-04-23T10:26:40.522323+0000","last_fullsized":"2024-04-23T10:26:40.522323+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T10:24:43.025324+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T10:24:43.025324+0000","last_clean_scrub_stamp":"2024-04-23T10:24:43.025324+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T11:48:08.387575+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,8,1],"acting":[7,8,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.34","version":"89'7","reported_seq":26,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-23T10:26:40.459648+0000","last_change":"2024-04-23T10:24:44.055335+0000","last_active":"2024-04-23T10:26:40.459648+0000","last_peered":"2024-04-23T10:26:40.459648+0000","last_clean":"2024-04-23T10:26:40.459648+0000","last_became_active":"2024-04-23T10:24:44.051257+0000","last_became_peered":"2024-04-23T10:24:44.051257+0000","last_unstale":"2024-04-23T10:26:40.459648+0000","last_undegraded":"2024-04-23T10:26:40.459648+0000","last_fullsized":"2024-04-23T10:26:40.459648+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T10:24:43.025324+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T10:24:43.025324+0000","last_clean_scrub_stamp":"2024-04-23T10:24:43.025324+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T11:57:45.999751+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,10,8],"acting":[4,10,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.33","version":"89'10","reported_seq":29,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-23T10:26:40.278008+0000","last_change":"2024-04-23T10:24:44.052078+0000","last_active":"2024-04-23T10:26:40.278008+0000","last_peered":"2024-04-23T10:26:40.278008+0000","last_clean":"2024-04-23T10:26:40.278008+0000","last_became_active":"2024-04-23T10:24:44.051747+0000","last_became_peered":"2024-04-23T10:24:44.051747+0000","last_unstale":"2024-04-23T10:26:40.278008+0000","last_undegraded":"2024-04-23T10:26:40.278008+0000","last_fullsized":"2024-04-23T10:26:40.278008+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T10:24:43.025324+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T10:24:43.025324+0000","last_clean_scrub_stamp":"2024-04-23T10:24:43.025324+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T19:14:16.495967+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,10,5],"acting":[6,10,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.32","version":"89'7","reported_seq":26,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-23T10:26:40.459810+0000","last_change":"2024-04-23T10:24:44.051994+0000","last_active":"2024-04-23T10:26:40.459810+0000","last_peered":"2024-04-23T10:26:40.459810+0000","last_clean":"2024-04-23T10:26:40.459810+0000","last_became_active":"2024-04-23T10:24:44.051638+0000","last_became_peered":"2024-04-23T10:24:44.051638+0000","last_unstale":"2024-04-23T10:26:40.459810+0000","last_undegraded":"2024-04-23T10:26:40.459810+0000","last_fullsized":"2024-04-23T10:26:40.459810+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T10:24:43.025324+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T10:24:43.025324+0000","last_clean_scrub_stamp":"2024-04-23T10:24:43.025324+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T14:28:30.447198+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,8,10],"acting":[0,8,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.31","version":"89'7","reported_seq":32,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-23T10:26:40.459693+0000","last_change":"2024-04-23T10:25:47.589538+0000","last_active":"2024-04-23T10:26:40.459693+0000","last_peered":"2024-04-23T10:26:40.459693+0000","last_clean":"2024-04-23T10:26:40.459693+0000","last_became_active":"2024-04-23T10:25:47.589172+0000","last_became_peered":"2024-04-23T10:25:47.589172+0000","last_unstale":"2024-04-23T10:26:40.459693+0000","last_undegraded":"2024-04-23T10:26:40.459693+0000","last_fullsized":"2024-04-23T10:26:40.459693+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T10:24:43.025324+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T10:24:43.025324+0000","last_clean_scrub_stamp":"2024-04-23T10:24:43.025324+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T19:47:34.267992+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,8,2],"acting":[0,8,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.30","version":"89'9","reported_seq":28,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-23T10:26:40.280931+0000","last_change":"2024-04-23T10:24:44.057154+0000","last_active":"2024-04-23T10:26:40.280931+0000","last_peered":"2024-04-23T10:26:40.280931+0000","last_clean":"2024-04-23T10:26:40.280931+0000","last_became_active":"2024-04-23T10:24:44.056836+0000","last_became_peered":"2024-04-23T10:24:44.056836+0000","last_unstale":"2024-04-23T10:26:40.280931+0000","last_undegraded":"2024-04-23T10:26:40.280931+0000","last_fullsized":"2024-04-23T10:26:40.280931+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T10:24:43.025324+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T10:24:43.025324+0000","last_clean_scrub_stamp":"2024-04-23T10:24:43.025324+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T13:48:59.107635+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,0],"acting":[1,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.2f","version":"89'8","reported_seq":27,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-23T10:26:40.460422+0000","last_change":"2024-04-23T10:24:44.051175+0000","last_active":"2024-04-23T10:26:40.460422+0000","last_peered":"2024-04-23T10:26:40.460422+0000","last_clean":"2024-04-23T10:26:40.460422+0000","last_became_active":"2024-04-23T10:24:44.050402+0000","last_became_peered":"2024-04-23T10:24:44.050402+0000","last_unstale":"2024-04-23T10:26:40.460422+0000","last_undegraded":"2024-04-23T10:26:40.460422+0000","last_fullsized":"2024-04-23T10:26:40.460422+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T10:24:43.025324+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T10:24:43.025324+0000","last_clean_scrub_stamp":"2024-04-23T10:24:43.025324+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T15:01:26.919701+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,1,0],"acting":[11,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.2e","version":"89'10","reported_seq":29,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-23T10:26:40.371997+0000","last_change":"2024-04-23T10:24:44.053390+0000","last_active":"2024-04-23T10:26:40.371997+0000","last_peered":"2024-04-23T10:26:40.371997+0000","last_clean":"2024-04-23T10:26:40.371997+0000","last_became_active":"2024-04-23T10:24:44.052970+0000","last_became_peered":"2024-04-23T10:24:44.052970+0000","last_unstale":"2024-04-23T10:26:40.371997+0000","last_undegraded":"2024-04-23T10:26:40.371997+0000","last_fullsized":"2024-04-23T10:26:40.371997+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T10:24:43.025324+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T10:24:43.025324+0000","last_clean_scrub_stamp":"2024-04-23T10:24:43.025324+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T13:27:55.704328+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,1,10],"acting":[9,1,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.2d","version":"89'7","reported_seq":26,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-23T10:26:40.460284+0000","last_change":"2024-04-23T10:24:44.049307+0000","last_active":"2024-04-23T10:26:40.460284+0000","last_peered":"2024-04-23T10:26:40.460284+0000","last_clean":"2024-04-23T10:26:40.460284+0000","last_became_active":"2024-04-23T10:24:44.048981+0000","last_became_peered":"2024-04-23T10:24:44.048981+0000","last_unstale":"2024-04-23T10:26:40.460284+0000","last_undegraded":"2024-04-23T10:26:40.460284+0000","last_fullsized":"2024-04-23T10:26:40.460284+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T10:24:43.025324+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T10:24:43.025324+0000","last_clean_scrub_stamp":"2024-04-23T10:24:43.025324+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T12:42:21.804591+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,6],"acting":[2,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.1f","version":"87'9","reported_seq":31,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-23T10:26:40.371868+0000","last_change":"2024-04-23T10:24:43.045804+0000","last_active":"2024-04-23T10:26:40.371868+0000","last_peered":"2024-04-23T10:26:40.371868+0000","last_clean":"2024-04-23T10:26:40.371868+0000","last_became_active":"2024-04-23T10:24:43.045135+0000","last_became_peered":"2024-04-23T10:24:43.045135+0000","last_unstale":"2024-04-23T10:26:40.371868+0000","last_undegraded":"2024-04-23T10:26:40.371868+0000","last_fullsized":"2024-04-23T10:26:40.371868+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T10:24:42.023127+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T10:24:42.023127+0000","last_clean_scrub_stamp":"2024-04-23T10:24:42.023127+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T20:27:28.421644+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,7,8],"acting":[9,7,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.1e","version":"89'5","reported_seq":24,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-23T10:26:40.374223+0000","last_change":"2024-04-23T10:24:44.055008+0000","last_active":"2024-04-23T10:26:40.374223+0000","last_peered":"2024-04-23T10:26:40.374223+0000","last_clean":"2024-04-23T10:26:40.374223+0000","last_became_active":"2024-04-23T10:24:44.054682+0000","last_became_peered":"2024-04-23T10:24:44.054682+0000","last_unstale":"2024-04-23T10:26:40.374223+0000","last_undegraded":"2024-04-23T10:26:40.374223+0000","last_fullsized":"2024-04-23T10:26:40.374223+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T10:24:43.025324+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T10:24:43.025324+0000","last_clean_scrub_stamp":"2024-04-23T10:24:43.025324+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T15:49:58.394958+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,5],"acting":[3,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.1e","version":"87'2","reported_seq":23,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-23T10:26:40.374327+0000","last_change":"2024-04-23T10:24:43.052303+0000","last_active":"2024-04-23T10:26:40.374327+0000","last_peered":"2024-04-23T10:26:40.374327+0000","last_clean":"2024-04-23T10:26:40.374327+0000","last_became_active":"2024-04-23T10:24:43.052010+0000","last_became_peered":"2024-04-23T10:24:43.052010+0000","last_unstale":"2024-04-23T10:26:40.374327+0000","last_undegraded":"2024-04-23T10:26:40.374327+0000","last_fullsized":"2024-04-23T10:26:40.374327+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T10:24:42.023127+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T10:24:42.023127+0000","last_clean_scrub_stamp":"2024-04-23T10:24:42.023127+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T13:03:15.440610+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,5],"acting":[3,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1f","version":"89'6","reported_seq":25,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-23T10:26:40.460053+0000","last_change":"2024-04-23T10:24:44.050749+0000","last_active":"2024-04-23T10:26:40.460053+0000","last_peered":"2024-04-23T10:26:40.460053+0000","last_clean":"2024-04-23T10:26:40.460053+0000","last_became_active":"2024-04-23T10:24:44.049780+0000","last_became_peered":"2024-04-23T10:24:44.049780+0000","last_unstale":"2024-04-23T10:26:40.460053+0000","last_undegraded":"2024-04-23T10:26:40.460053+0000","last_fullsized":"2024-04-23T10:26:40.460053+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T10:24:43.025324+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T10:24:43.025324+0000","last_clean_scrub_stamp":"2024-04-23T10:24:43.025324+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T16:32:37.963776+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,9],"acting":[0,1,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1d","version":"87'4","reported_seq":19,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-23T10:26:40.460147+0000","last_change":"2024-04-23T10:25:47.591939+0000","last_active":"2024-04-23T10:26:40.460147+0000","last_peered":"2024-04-23T10:26:40.460147+0000","last_clean":"2024-04-23T10:26:40.460147+0000","last_became_active":"2024-04-23T10:25:47.586626+0000","last_became_peered":"2024-04-23T10:25:47.586626+0000","last_unstale":"2024-04-23T10:26:40.460147+0000","last_undegraded":"2024-04-23T10:26:40.460147+0000","last_fullsized":"2024-04-23T10:26:40.460147+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T10:24:42.023127+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T10:24:42.023127+0000","last_clean_scrub_stamp":"2024-04-23T10:24:42.023127+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T20:55:38.216483+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":10,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,2,7],"acting":[11,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.1c","version":"89'9","reported_seq":28,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-23T10:26:40.281020+0000","last_change":"2024-04-23T10:24:44.051606+0000","last_active":"2024-04-23T10:26:40.281020+0000","last_peered":"2024-04-23T10:26:40.281020+0000","last_clean":"2024-04-23T10:26:40.281020+0000","last_became_active":"2024-04-23T10:24:44.051191+0000","last_became_peered":"2024-04-23T10:24:44.051191+0000","last_unstale":"2024-04-23T10:26:40.281020+0000","last_undegraded":"2024-04-23T10:26:40.281020+0000","last_fullsized":"2024-04-23T10:26:40.281020+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T10:24:43.025324+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T10:24:43.025324+0000","last_clean_scrub_stamp":"2024-04-23T10:24:43.025324+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T16:21:51.706066+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,10,6],"acting":[1,10,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1c","version":"86'2","reported_seq":23,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-23T10:26:40.373937+0000","last_change":"2024-04-23T10:24:43.048594+0000","last_active":"2024-04-23T10:26:40.373937+0000","last_peered":"2024-04-23T10:26:40.373937+0000","last_clean":"2024-04-23T10:26:40.373937+0000","last_became_active":"2024-04-23T10:24:43.048276+0000","last_became_peered":"2024-04-23T10:24:43.048276+0000","last_unstale":"2024-04-23T10:26:40.373937+0000","last_undegraded":"2024-04-23T10:26:40.373937+0000","last_fullsized":"2024-04-23T10:26:40.373937+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T10:24:42.023127+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T10:24:42.023127+0000","last_clean_scrub_stamp":"2024-04-23T10:24:42.023127+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T12:20:40.196674+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,1,5],"acting":[8,1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.1d","version":"89'15","reported_seq":34,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-23T10:26:40.282049+0000","last_change":"2024-04-23T10:24:44.050970+0000","last_active":"2024-04-23T10:26:40.282049+0000","last_peered":"2024-04-23T10:26:40.282049+0000","last_clean":"2024-04-23T10:26:40.282049+0000","last_became_active":"2024-04-23T10:24:44.050640+0000","last_became_peered":"2024-04-23T10:24:44.050640+0000","last_unstale":"2024-04-23T10:26:40.282049+0000","last_undegraded":"2024-04-23T10:26:40.282049+0000","last_fullsized":"2024-04-23T10:26:40.282049+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T10:24:43.025324+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T10:24:43.025324+0000","last_clean_scrub_stamp":"2024-04-23T10:24:43.025324+0000","objects_scrubbed":0,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T21:03:24.570290+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,1],"acting":[5,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.1b","version":"90'8","reported_seq":146,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-23T10:26:40.486313+0000","last_change":"2024-04-23T10:24:43.051742+0000","last_active":"2024-04-23T10:26:40.486313+0000","last_peered":"2024-04-23T10:26:40.486313+0000","last_clean":"2024-04-23T10:26:40.486313+0000","last_became_active":"2024-04-23T10:24:43.051420+0000","last_became_peered":"2024-04-23T10:24:43.051420+0000","last_unstale":"2024-04-23T10:26:40.486313+0000","last_undegraded":"2024-04-23T10:26:40.486313+0000","last_fullsized":"2024-04-23T10:26:40.486313+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T10:24:42.023127+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T10:24:42.023127+0000","last_clean_scrub_stamp":"2024-04-23T10:24:42.023127+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T17:54:37.182731+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1545,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":117,"num_read_kb":4,"num_write":5,"num_write_kb":5,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,7],"acting":[5,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.1a","version":"89'7","reported_seq":26,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-23T10:26:40.373718+0000","last_change":"2024-04-23T10:24:44.049314+0000","last_active":"2024-04-23T10:26:40.373718+0000","last_peered":"2024-04-23T10:26:40.373718+0000","last_clean":"2024-04-23T10:26:40.373718+0000","last_became_active":"2024-04-23T10:24:44.048352+0000","last_became_peered":"2024-04-23T10:24:44.048352+0000","last_unstale":"2024-04-23T10:26:40.373718+0000","last_undegraded":"2024-04-23T10:26:40.373718+0000","last_fullsized":"2024-04-23T10:26:40.373718+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T10:24:43.025324+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T10:24:43.025324+0000","last_clean_scrub_stamp":"2024-04-23T10:24:43.025324+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T16:57:29.957716+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,9,5],"acting":[8,9,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"2.1a","version":"87'3","reported_seq":24,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-23T10:26:40.381286+0000","last_change":"2024-04-23T10:24:43.051362+0000","last_active":"2024-04-23T10:26:40.381286+0000","last_peered":"2024-04-23T10:26:40.381286+0000","last_clean":"2024-04-23T10:26:40.381286+0000","last_became_active":"2024-04-23T10:24:43.051045+0000","last_became_peered":"2024-04-23T10:24:43.051045+0000","last_unstale":"2024-04-23T10:26:40.381286+0000","last_undegraded":"2024-04-23T10:26:40.381286+0000","last_fullsized":"2024-04-23T10:26:40.381286+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T10:24:42.023127+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T10:24:42.023127+0000","last_clean_scrub_stamp":"2024-04-23T10:24:42.023127+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T15:40:53.618281+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,8,6],"acting":[10,8,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.1b","version":"89'8","reported_seq":27,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-23T10:26:40.281974+0000","last_change":"2024-04-23T10:24:44.040323+0000","last_active":"2024-04-23T10:26:40.281974+0000","last_peered":"2024-04-23T10:26:40.281974+0000","last_clean":"2024-04-23T10:26:40.281974+0000","last_became_active":"2024-04-23T10:24:44.039995+0000","last_became_peered":"2024-04-23T10:24:44.039995+0000","last_unstale":"2024-04-23T10:26:40.281974+0000","last_undegraded":"2024-04-23T10:26:40.281974+0000","last_fullsized":"2024-04-23T10:26:40.281974+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T10:24:43.025324+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T10:24:43.025324+0000","last_clean_scrub_stamp":"2024-04-23T10:24:43.025324+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T16:38:48.756776+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,3],"acting":[5,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.19","version":"87'2","reported_seq":23,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-23T10:26:40.277848+0000","last_change":"2024-04-23T10:24:43.048754+0000","last_active":"2024-04-23T10:26:40.277848+0000","last_peered":"2024-04-23T10:26:40.277848+0000","last_clean":"2024-04-23T10:26:40.277848+0000","last_became_active":"2024-04-23T10:24:43.047962+0000","last_became_peered":"2024-04-23T10:24:43.047962+0000","last_unstale":"2024-04-23T10:26:40.277848+0000","last_undegraded":"2024-04-23T10:26:40.277848+0000","last_fullsized":"2024-04-23T10:26:40.277848+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T10:24:42.023127+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T10:24:42.023127+0000","last_clean_scrub_stamp":"2024-04-23T10:24:42.023127+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T21:08:38.487185+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,9],"acting":[6,4,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.18","version":"89'8","reported_seq":27,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-23T10:26:40.373688+0000","last_change":"2024-04-23T10:24:44.045022+0000","last_active":"2024-04-23T10:26:40.373688+0000","last_peered":"2024-04-23T10:26:40.373688+0000","last_clean":"2024-04-23T10:26:40.373688+0000","last_became_active":"2024-04-23T10:24:44.044590+0000","last_became_peered":"2024-04-23T10:24:44.044590+0000","last_unstale":"2024-04-23T10:26:40.373688+0000","last_undegraded":"2024-04-23T10:26:40.373688+0000","last_fullsized":"2024-04-23T10:26:40.373688+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T10:24:43.025324+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T10:24:43.025324+0000","last_clean_scrub_stamp":"2024-04-23T10:24:43.025324+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T13:20:49.892013+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,11,6],"acting":[3,11,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.18","version":"87'1","reported_seq":22,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-23T10:26:40.281503+0000","last_change":"2024-04-23T10:24:43.050170+0000","last_active":"2024-04-23T10:26:40.281503+0000","last_peered":"2024-04-23T10:26:40.281503+0000","last_clean":"2024-04-23T10:26:40.281503+0000","last_became_active":"2024-04-23T10:24:43.049070+0000","last_became_peered":"2024-04-23T10:24:43.049070+0000","last_unstale":"2024-04-23T10:26:40.281503+0000","last_undegraded":"2024-04-23T10:26:40.281503+0000","last_fullsized":"2024-04-23T10:26:40.281503+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T10:24:42.023127+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T10:24:42.023127+0000","last_clean_scrub_stamp":"2024-04-23T10:24:42.023127+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T19:34:13.983628+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,7],"acting":[5,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.19","version":"89'11","reported_seq":30,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-23T10:26:40.371665+0000","last_change":"2024-04-23T10:24:44.047165+0000","last_active":"2024-04-23T10:26:40.371665+0000","last_peered":"2024-04-23T10:26:40.371665+0000","last_clean":"2024-04-23T10:26:40.371665+0000","last_became_active":"2024-04-23T10:24:44.046776+0000","last_became_peered":"2024-04-23T10:24:44.046776+0000","last_unstale":"2024-04-23T10:26:40.371665+0000","last_undegraded":"2024-04-23T10:26:40.371665+0000","last_fullsized":"2024-04-23T10:26:40.371665+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T10:24:43.025324+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T10:24:43.025324+0000","last_clean_scrub_stamp":"2024-04-23T10:24:43.025324+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T18:11:57.181747+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,3,10],"acting":[9,3,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"2.17","version":"90'5","reported_seq":139,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-23T10:26:40.485905+0000","last_change":"2024-04-23T10:24:43.049785+0000","last_active":"2024-04-23T10:26:40.485905+0000","last_peered":"2024-04-23T10:26:40.485905+0000","last_clean":"2024-04-23T10:26:40.485905+0000","last_became_active":"2024-04-23T10:24:43.049402+0000","last_became_peered":"2024-04-23T10:24:43.049402+0000","last_unstale":"2024-04-23T10:26:40.485905+0000","last_undegraded":"2024-04-23T10:26:40.485905+0000","last_fullsized":"2024-04-23T10:26:40.485905+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T10:24:42.023127+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T10:24:42.023127+0000","last_clean_scrub_stamp":"2024-04-23T10:24:42.023127+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T14:20:13.774082+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":113,"num_read_kb":113,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,5,9],"acting":[10,5,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.16","version":"89'10","reported_seq":29,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-23T10:26:40.281357+0000","last_change":"2024-04-23T10:24:44.051768+0000","last_active":"2024-04-23T10:26:40.281357+0000","last_peered":"2024-04-23T10:26:40.281357+0000","last_clean":"2024-04-23T10:26:40.281357+0000","last_became_active":"2024-04-23T10:24:44.051228+0000","last_became_peered":"2024-04-23T10:24:44.051228+0000","last_unstale":"2024-04-23T10:26:40.281357+0000","last_undegraded":"2024-04-23T10:26:40.281357+0000","last_fullsized":"2024-04-23T10:26:40.281357+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T10:24:43.025324+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T10:24:43.025324+0000","last_clean_scrub_stamp":"2024-04-23T10:24:43.025324+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T14:09:47.649083+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,10,9],"acting":[1,10,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.16","version":"87'2","reported_seq":23,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-23T10:26:40.522354+0000","last_change":"2024-04-23T10:24:43.043944+0000","last_active":"2024-04-23T10:26:40.522354+0000","last_peered":"2024-04-23T10:26:40.522354+0000","last_clean":"2024-04-23T10:26:40.522354+0000","last_became_active":"2024-04-23T10:24:43.043648+0000","last_became_peered":"2024-04-23T10:24:43.043648+0000","last_unstale":"2024-04-23T10:26:40.522354+0000","last_undegraded":"2024-04-23T10:26:40.522354+0000","last_fullsized":"2024-04-23T10:26:40.522354+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T10:24:42.023127+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T10:24:42.023127+0000","last_clean_scrub_stamp":"2024-04-23T10:24:42.023127+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T13:41:15.724662+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,2],"acting":[7,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.17","version":"89'9","reported_seq":34,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-23T10:26:40.460012+0000","last_change":"2024-04-23T10:25:47.721534+0000","last_active":"2024-04-23T10:26:40.460012+0000","last_peered":"2024-04-23T10:26:40.460012+0000","last_clean":"2024-04-23T10:26:40.460012+0000","last_became_active":"2024-04-23T10:25:47.721130+0000","last_became_peered":"2024-04-23T10:25:47.721130+0000","last_unstale":"2024-04-23T10:26:40.460012+0000","last_undegraded":"2024-04-23T10:26:40.460012+0000","last_fullsized":"2024-04-23T10:26:40.460012+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T10:24:43.025324+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T10:24:43.025324+0000","last_clean_scrub_stamp":"2024-04-23T10:24:43.025324+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T21:21:26.537486+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,5],"acting":[0,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.15","version":"87'4","reported_seq":26,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-23T10:26:40.282238+0000","last_change":"2024-04-23T10:24:43.048517+0000","last_active":"2024-04-23T10:26:40.282238+0000","last_peered":"2024-04-23T10:26:40.282238+0000","last_clean":"2024-04-23T10:26:40.282238+0000","last_became_active":"2024-04-23T10:24:43.048183+0000","last_became_peered":"2024-04-23T10:24:43.048183+0000","last_unstale":"2024-04-23T10:26:40.282238+0000","last_undegraded":"2024-04-23T10:26:40.282238+0000","last_fullsized":"2024-04-23T10:26:40.282238+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T10:24:42.023127+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T10:24:42.023127+0000","last_clean_scrub_stamp":"2024-04-23T10:24:42.023127+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T20:50:15.017833+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":672,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":5,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,10],"acting":[5,0,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.14","version":"89'11","reported_seq":30,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-23T10:26:40.281189+0000","last_change":"2024-04-23T10:24:44.055598+0000","last_active":"2024-04-23T10:26:40.281189+0000","last_peered":"2024-04-23T10:26:40.281189+0000","last_clean":"2024-04-23T10:26:40.281189+0000","last_became_active":"2024-04-23T10:24:44.053787+0000","last_became_peered":"2024-04-23T10:24:44.053787+0000","last_unstale":"2024-04-23T10:26:40.281189+0000","last_undegraded":"2024-04-23T10:26:40.281189+0000","last_fullsized":"2024-04-23T10:26:40.281189+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T10:24:43.025324+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T10:24:43.025324+0000","last_clean_scrub_stamp":"2024-04-23T10:24:43.025324+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T20:04:21.603306+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,8,4],"acting":[1,8,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.14","version":"90'36","reported_seq":293,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-23T10:26:40.520887+0000","last_change":"2024-04-23T10:24:43.044601+0000","last_active":"2024-04-23T10:26:40.520887+0000","last_peered":"2024-04-23T10:26:40.520887+0000","last_clean":"2024-04-23T10:26:40.520887+0000","last_became_active":"2024-04-23T10:24:43.044238+0000","last_became_peered":"2024-04-23T10:24:43.044238+0000","last_unstale":"2024-04-23T10:26:40.520887+0000","last_undegraded":"2024-04-23T10:26:40.520887+0000","last_fullsized":"2024-04-23T10:26:40.520887+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T10:24:42.023127+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T10:24:42.023127+0000","last_clean_scrub_stamp":"2024-04-23T10:24:42.023127+0000","objects_scrubbed":0,"log_size":36,"log_dups_size":0,"ondisk_log_size":36,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T12:53:26.275771+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":67758,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":236,"num_read_kb":183,"num_write":37,"num_write_kb":82,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,1,2],"acting":[10,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.15","version":"89'6","reported_seq":25,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-23T10:26:40.373730+0000","last_change":"2024-04-23T10:24:44.051375+0000","last_active":"2024-04-23T10:26:40.373730+0000","last_peered":"2024-04-23T10:26:40.373730+0000","last_clean":"2024-04-23T10:26:40.373730+0000","last_became_active":"2024-04-23T10:24:44.050924+0000","last_became_peered":"2024-04-23T10:24:44.050924+0000","last_unstale":"2024-04-23T10:26:40.373730+0000","last_undegraded":"2024-04-23T10:26:40.373730+0000","last_fullsized":"2024-04-23T10:26:40.373730+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T10:24:43.025324+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T10:24:43.025324+0000","last_clean_scrub_stamp":"2024-04-23T10:24:43.025324+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T21:50:57.534760+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,11,7],"acting":[8,11,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"2.13","version":"87'6","reported_seq":28,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-23T10:26:40.459793+0000","last_change":"2024-04-23T10:24:43.044457+0000","last_active":"2024-04-23T10:26:40.459793+0000","last_peered":"2024-04-23T10:26:40.459793+0000","last_clean":"2024-04-23T10:26:40.459793+0000","last_became_active":"2024-04-23T10:24:43.043956+0000","last_became_peered":"2024-04-23T10:24:43.043956+0000","last_unstale":"2024-04-23T10:26:40.459793+0000","last_undegraded":"2024-04-23T10:26:40.459793+0000","last_fullsized":"2024-04-23T10:26:40.459793+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T10:24:42.023127+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T10:24:42.023127+0000","last_clean_scrub_stamp":"2024-04-23T10:24:42.023127+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T20:16:08.186729+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":9,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,10,3],"acting":[2,10,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.12","version":"89'9","reported_seq":34,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-23T10:26:40.277806+0000","last_change":"2024-04-23T10:25:47.587240+0000","last_active":"2024-04-23T10:26:40.277806+0000","last_peered":"2024-04-23T10:26:40.277806+0000","last_clean":"2024-04-23T10:26:40.277806+0000","last_became_active":"2024-04-23T10:25:47.586659+0000","last_became_peered":"2024-04-23T10:25:47.586659+0000","last_unstale":"2024-04-23T10:26:40.277806+0000","last_undegraded":"2024-04-23T10:26:40.277806+0000","last_fullsized":"2024-04-23T10:26:40.277806+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T10:24:43.025324+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T10:24:43.025324+0000","last_clean_scrub_stamp":"2024-04-23T10:24:43.025324+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T19:55:03.256183+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,9],"acting":[6,7,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.12","version":"87'3","reported_seq":24,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-23T10:26:40.281768+0000","last_change":"2024-04-23T10:24:43.052181+0000","last_active":"2024-04-23T10:26:40.281768+0000","last_peered":"2024-04-23T10:26:40.281768+0000","last_clean":"2024-04-23T10:26:40.281768+0000","last_became_active":"2024-04-23T10:24:43.051793+0000","last_became_peered":"2024-04-23T10:24:43.051793+0000","last_unstale":"2024-04-23T10:26:40.281768+0000","last_undegraded":"2024-04-23T10:26:40.281768+0000","last_fullsized":"2024-04-23T10:26:40.281768+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T10:24:42.023127+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T10:24:42.023127+0000","last_clean_scrub_stamp":"2024-04-23T10:24:42.023127+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T20:57:59.015115+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,7],"acting":[5,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.13","version":"89'15","reported_seq":34,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-23T10:26:40.522012+0000","last_change":"2024-04-23T10:24:44.062689+0000","last_active":"2024-04-23T10:26:40.522012+0000","last_peered":"2024-04-23T10:26:40.522012+0000","last_clean":"2024-04-23T10:26:40.522012+0000","last_became_active":"2024-04-23T10:24:44.062331+0000","last_became_peered":"2024-04-23T10:24:44.062331+0000","last_unstale":"2024-04-23T10:26:40.522012+0000","last_undegraded":"2024-04-23T10:26:40.522012+0000","last_fullsized":"2024-04-23T10:26:40.522012+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T10:24:43.025324+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T10:24:43.025324+0000","last_clean_scrub_stamp":"2024-04-23T10:24:43.025324+0000","objects_scrubbed":0,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T14:06:04.989156+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,6],"acting":[7,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.11","version":"90'6","reported_seq":144,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-23T10:26:40.486457+0000","last_change":"2024-04-23T10:24:43.043901+0000","last_active":"2024-04-23T10:26:40.486457+0000","last_peered":"2024-04-23T10:26:40.486457+0000","last_clean":"2024-04-23T10:26:40.486457+0000","last_became_active":"2024-04-23T10:24:43.043598+0000","last_became_peered":"2024-04-23T10:24:43.043598+0000","last_unstale":"2024-04-23T10:26:40.486457+0000","last_undegraded":"2024-04-23T10:26:40.486457+0000","last_fullsized":"2024-04-23T10:26:40.486457+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T10:24:42.023127+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T10:24:42.023127+0000","last_clean_scrub_stamp":"2024-04-23T10:24:42.023127+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T14:00:06.338084+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1511,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":118,"num_read_kb":5,"num_write":5,"num_write_kb":4,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,10],"acting":[4,7,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.10","version":"89'6","reported_seq":25,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-23T10:26:40.521595+0000","last_change":"2024-04-23T10:24:44.058773+0000","last_active":"2024-04-23T10:26:40.521595+0000","last_peered":"2024-04-23T10:26:40.521595+0000","last_clean":"2024-04-23T10:26:40.521595+0000","last_became_active":"2024-04-23T10:24:44.058420+0000","last_became_peered":"2024-04-23T10:24:44.058420+0000","last_unstale":"2024-04-23T10:26:40.521595+0000","last_undegraded":"2024-04-23T10:26:40.521595+0000","last_fullsized":"2024-04-23T10:26:40.521595+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T10:24:43.025324+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T10:24:43.025324+0000","last_clean_scrub_stamp":"2024-04-23T10:24:43.025324+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T12:44:30.023047+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,10],"acting":[7,5,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.10","version":"87'15","reported_seq":37,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-23T10:26:40.373552+0000","last_change":"2024-04-23T10:24:43.044700+0000","last_active":"2024-04-23T10:26:40.373552+0000","last_peered":"2024-04-23T10:26:40.373552+0000","last_clean":"2024-04-23T10:26:40.373552+0000","last_became_active":"2024-04-23T10:24:43.044394+0000","last_became_peered":"2024-04-23T10:24:43.044394+0000","last_unstale":"2024-04-23T10:26:40.373552+0000","last_undegraded":"2024-04-23T10:26:40.373552+0000","last_fullsized":"2024-04-23T10:26:40.373552+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T10:24:42.023127+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T10:24:42.023127+0000","last_clean_scrub_stamp":"2024-04-23T10:24:42.023127+0000","objects_scrubbed":0,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T15:16:41.367517+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":112,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":20,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":6,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,0,2],"acting":[8,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.11","version":"89'6","reported_seq":25,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-23T10:26:40.281571+0000","last_change":"2024-04-23T10:24:44.049739+0000","last_active":"2024-04-23T10:26:40.281571+0000","last_peered":"2024-04-23T10:26:40.281571+0000","last_clean":"2024-04-23T10:26:40.281571+0000","last_became_active":"2024-04-23T10:24:44.049414+0000","last_became_peered":"2024-04-23T10:24:44.049414+0000","last_unstale":"2024-04-23T10:26:40.281571+0000","last_undegraded":"2024-04-23T10:26:40.281571+0000","last_fullsized":"2024-04-23T10:26:40.281571+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T10:24:43.025324+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T10:24:43.025324+0000","last_clean_scrub_stamp":"2024-04-23T10:24:43.025324+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T12:51:02.645259+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,1],"acting":[5,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.f","version":"87'2","reported_seq":24,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-23T10:26:40.373920+0000","last_change":"2024-04-23T10:24:43.048884+0000","last_active":"2024-04-23T10:26:40.373920+0000","last_peered":"2024-04-23T10:26:40.373920+0000","last_clean":"2024-04-23T10:26:40.373920+0000","last_became_active":"2024-04-23T10:24:43.048583+0000","last_became_peered":"2024-04-23T10:24:43.048583+0000","last_unstale":"2024-04-23T10:26:40.373920+0000","last_undegraded":"2024-04-23T10:26:40.373920+0000","last_fullsized":"2024-04-23T10:26:40.373920+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T10:24:42.023127+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T10:24:42.023127+0000","last_clean_scrub_stamp":"2024-04-23T10:24:42.023127+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T17:55:57.869700+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,5,4],"acting":[8,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.e","version":"89'5","reported_seq":30,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-23T10:26:40.460256+0000","last_change":"2024-04-23T10:25:47.722078+0000","last_active":"2024-04-23T10:26:40.460256+0000","last_peered":"2024-04-23T10:26:40.460256+0000","last_clean":"2024-04-23T10:26:40.460256+0000","last_became_active":"2024-04-23T10:25:47.721730+0000","last_became_peered":"2024-04-23T10:25:47.721730+0000","last_unstale":"2024-04-23T10:26:40.460256+0000","last_undegraded":"2024-04-23T10:26:40.460256+0000","last_fullsized":"2024-04-23T10:26:40.460256+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T10:24:43.025324+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T10:24:43.025324+0000","last_clean_scrub_stamp":"2024-04-23T10:24:43.025324+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T18:18:40.916492+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,9,4],"acting":[11,9,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":21,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-23T10:26:40.371847+0000","last_change":"2024-04-23T10:24:43.044375+0000","last_active":"2024-04-23T10:26:40.371847+0000","last_peered":"2024-04-23T10:26:40.371847+0000","last_clean":"2024-04-23T10:26:40.371847+0000","last_became_active":"2024-04-23T10:24:43.043914+0000","last_became_peered":"2024-04-23T10:24:43.043914+0000","last_unstale":"2024-04-23T10:26:40.371847+0000","last_undegraded":"2024-04-23T10:26:40.371847+0000","last_fullsized":"2024-04-23T10:26:40.371847+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T10:24:42.023127+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T10:24:42.023127+0000","last_clean_scrub_stamp":"2024-04-23T10:24:42.023127+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T16:22:44.165156+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,2,0],"acting":[9,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.5","version":"89'5","reported_seq":24,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-23T10:26:40.381571+0000","last_change":"2024-04-23T10:24:44.057272+0000","last_active":"2024-04-23T10:26:40.381571+0000","last_peered":"2024-04-23T10:26:40.381571+0000","last_clean":"2024-04-23T10:26:40.381571+0000","last_became_active":"2024-04-23T10:24:44.056968+0000","last_became_peered":"2024-04-23T10:24:44.056968+0000","last_unstale":"2024-04-23T10:26:40.381571+0000","last_undegraded":"2024-04-23T10:26:40.381571+0000","last_fullsized":"2024-04-23T10:26:40.381571+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T10:24:43.025324+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T10:24:43.025324+0000","last_clean_scrub_stamp":"2024-04-23T10:24:43.025324+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T11:10:09.021845+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":117,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,3,6],"acting":[10,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"2.2","version":"87'1","reported_seq":22,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-23T10:26:40.521864+0000","last_change":"2024-04-23T10:24:43.051822+0000","last_active":"2024-04-23T10:26:40.521864+0000","last_peered":"2024-04-23T10:26:40.521864+0000","last_clean":"2024-04-23T10:26:40.521864+0000","last_became_active":"2024-04-23T10:24:43.051491+0000","last_became_peered":"2024-04-23T10:24:43.051491+0000","last_unstale":"2024-04-23T10:26:40.521864+0000","last_undegraded":"2024-04-23T10:26:40.521864+0000","last_fullsized":"2024-04-23T10:26:40.521864+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T10:24:42.023127+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T10:24:42.023127+0000","last_clean_scrub_stamp":"2024-04-23T10:24:42.023127+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T13:49:23.109064+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,1],"acting":[7,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.3","version":"89'7","reported_seq":26,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-23T10:26:40.381469+0000","last_change":"2024-04-23T10:24:44.053604+0000","last_active":"2024-04-23T10:26:40.381469+0000","last_peered":"2024-04-23T10:26:40.381469+0000","last_clean":"2024-04-23T10:26:40.381469+0000","last_became_active":"2024-04-23T10:24:44.053147+0000","last_became_peered":"2024-04-23T10:24:44.053147+0000","last_unstale":"2024-04-23T10:26:40.381469+0000","last_undegraded":"2024-04-23T10:26:40.381469+0000","last_fullsized":"2024-04-23T10:26:40.381469+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T10:24:43.025324+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T10:24:43.025324+0000","last_clean_scrub_stamp":"2024-04-23T10:24:43.025324+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T21:58:38.981282+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,4,0],"acting":[10,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.0","version":"89'8","reported_seq":27,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-23T10:26:40.281729+0000","last_change":"2024-04-23T10:24:44.052564+0000","last_active":"2024-04-23T10:26:40.281729+0000","last_peered":"2024-04-23T10:26:40.281729+0000","last_clean":"2024-04-23T10:26:40.281729+0000","last_became_active":"2024-04-23T10:24:44.052201+0000","last_became_peered":"2024-04-23T10:24:44.052201+0000","last_unstale":"2024-04-23T10:26:40.281729+0000","last_undegraded":"2024-04-23T10:26:40.281729+0000","last_fullsized":"2024-04-23T10:26:40.281729+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T10:24:43.025324+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T10:24:43.025324+0000","last_clean_scrub_stamp":"2024-04-23T10:24:43.025324+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T16:25:54.725347+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,4],"acting":[5,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.1","version":"90'6","reported_seq":140,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-23T10:26:40.486009+0000","last_change":"2024-04-23T10:24:43.049067+0000","last_active":"2024-04-23T10:26:40.486009+0000","last_peered":"2024-04-23T10:26:40.486009+0000","last_clean":"2024-04-23T10:26:40.486009+0000","last_became_active":"2024-04-23T10:24:43.048400+0000","last_became_peered":"2024-04-23T10:24:43.048400+0000","last_unstale":"2024-04-23T10:26:40.486009+0000","last_undegraded":"2024-04-23T10:26:40.486009+0000","last_fullsized":"2024-04-23T10:26:40.486009+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T10:24:42.023127+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T10:24:42.023127+0000","last_clean_scrub_stamp":"2024-04-23T10:24:42.023127+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T12:17:02.439054+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":113,"num_read_kb":113,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,0],"acting":[6,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.2","version":"89'8","reported_seq":27,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-23T10:26:40.373993+0000","last_change":"2024-04-23T10:24:44.059114+0000","last_active":"2024-04-23T10:26:40.373993+0000","last_peered":"2024-04-23T10:26:40.373993+0000","last_clean":"2024-04-23T10:26:40.373993+0000","last_became_active":"2024-04-23T10:24:44.058618+0000","last_became_peered":"2024-04-23T10:24:44.058618+0000","last_unstale":"2024-04-23T10:26:40.373993+0000","last_undegraded":"2024-04-23T10:26:40.373993+0000","last_fullsized":"2024-04-23T10:26:40.373993+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T10:24:43.025324+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T10:24:43.025324+0000","last_clean_scrub_stamp":"2024-04-23T10:24:43.025324+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T20:18:09.707062+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,1],"acting":[3,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.3","version":"87'3","reported_seq":26,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-23T10:26:40.459688+0000","last_change":"2024-04-23T10:24:43.039725+0000","last_active":"2024-04-23T10:26:40.459688+0000","last_peered":"2024-04-23T10:26:40.459688+0000","last_clean":"2024-04-23T10:26:40.459688+0000","last_became_active":"2024-04-23T10:24:43.039187+0000","last_became_peered":"2024-04-23T10:24:43.039187+0000","last_unstale":"2024-04-23T10:26:40.459688+0000","last_undegraded":"2024-04-23T10:26:40.459688+0000","last_fullsized":"2024-04-23T10:26:40.459688+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T10:24:42.023127+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T10:24:42.023127+0000","last_clean_scrub_stamp":"2024-04-23T10:24:42.023127+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T17:20:50.844878+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":3,"num_read_kb":3,"num_write":4,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,10,9],"acting":[4,10,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.0","version":"22'137","reported_seq":353,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-23T10:26:40.521666+0000","last_change":"2024-04-23T10:23:31.092860+0000","last_active":"2024-04-23T10:26:40.521666+0000","last_peered":"2024-04-23T10:26:40.521666+0000","last_clean":"2024-04-23T10:26:40.521666+0000","last_became_active":"2024-04-23T10:23:31.092357+0000","last_became_peered":"2024-04-23T10:23:31.092357+0000","last_unstale":"2024-04-23T10:26:40.521666+0000","last_undegraded":"2024-04-23T10:26:40.521666+0000","last_fullsized":"2024-04-23T10:26:40.521666+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T10:20:12.201233+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T10:20:12.201233+0000","last_clean_scrub_stamp":"2024-04-23T10:20:12.201233+0000","objects_scrubbed":0,"log_size":137,"log_dups_size":0,"ondisk_log_size":137,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T21:28:56.804053+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":136,"num_read_kb":119,"num_write":181,"num_write_kb":2938,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":1180736,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,10],"acting":[7,5,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1","version":"89'5","reported_seq":24,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-23T10:26:40.277697+0000","last_change":"2024-04-23T10:24:44.058393+0000","last_active":"2024-04-23T10:26:40.277697+0000","last_peered":"2024-04-23T10:26:40.277697+0000","last_clean":"2024-04-23T10:26:40.277697+0000","last_became_active":"2024-04-23T10:24:44.058075+0000","last_became_peered":"2024-04-23T10:24:44.058075+0000","last_unstale":"2024-04-23T10:26:40.277697+0000","last_undegraded":"2024-04-23T10:26:40.277697+0000","last_fullsized":"2024-04-23T10:26:40.277697+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T10:24:43.025324+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T10:24:43.025324+0000","last_clean_scrub_stamp":"2024-04-23T10:24:43.025324+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T10:32:05.693311+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,2],"acting":[6,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.0","version":"87'3","reported_seq":24,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-23T10:26:40.521796+0000","last_change":"2024-04-23T10:24:43.048678+0000","last_active":"2024-04-23T10:26:40.521796+0000","last_peered":"2024-04-23T10:26:40.521796+0000","last_clean":"2024-04-23T10:26:40.521796+0000","last_became_active":"2024-04-23T10:24:43.048341+0000","last_became_peered":"2024-04-23T10:24:43.048341+0000","last_unstale":"2024-04-23T10:26:40.521796+0000","last_undegraded":"2024-04-23T10:26:40.521796+0000","last_fullsized":"2024-04-23T10:26:40.521796+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T10:24:42.023127+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T10:24:42.023127+0000","last_clean_scrub_stamp":"2024-04-23T10:24:42.023127+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T12:24:33.705325+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,8],"acting":[7,5,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.5","version":"87'1","reported_seq":23,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-23T10:26:40.373795+0000","last_change":"2024-04-23T10:24:43.046105+0000","last_active":"2024-04-23T10:26:40.373795+0000","last_peered":"2024-04-23T10:26:40.373795+0000","last_clean":"2024-04-23T10:26:40.373795+0000","last_became_active":"2024-04-23T10:24:43.045804+0000","last_became_peered":"2024-04-23T10:24:43.045804+0000","last_unstale":"2024-04-23T10:26:40.373795+0000","last_undegraded":"2024-04-23T10:26:40.373795+0000","last_fullsized":"2024-04-23T10:26:40.373795+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T10:24:42.023127+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T10:24:42.023127+0000","last_clean_scrub_stamp":"2024-04-23T10:24:42.023127+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T22:23:35.927468+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,0,9],"acting":[8,0,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.4","version":"89'9","reported_seq":28,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-23T10:26:40.278062+0000","last_change":"2024-04-23T10:24:44.059731+0000","last_active":"2024-04-23T10:26:40.278062+0000","last_peered":"2024-04-23T10:26:40.278062+0000","last_clean":"2024-04-23T10:26:40.278062+0000","last_became_active":"2024-04-23T10:24:44.059352+0000","last_became_peered":"2024-04-23T10:24:44.059352+0000","last_unstale":"2024-04-23T10:26:40.278062+0000","last_undegraded":"2024-04-23T10:26:40.278062+0000","last_fullsized":"2024-04-23T10:26:40.278062+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T10:24:43.025324+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T10:24:43.025324+0000","last_clean_scrub_stamp":"2024-04-23T10:24:43.025324+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T16:36:08.985444+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2,9],"acting":[6,2,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.6","version":"90'44","reported_seq":88,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-23T10:26:40.483019+0000","last_change":"2024-04-23T10:24:43.044527+0000","last_active":"2024-04-23T10:26:40.483019+0000","last_peered":"2024-04-23T10:26:40.483019+0000","last_clean":"2024-04-23T10:26:40.483019+0000","last_became_active":"2024-04-23T10:24:43.044212+0000","last_became_peered":"2024-04-23T10:24:43.044212+0000","last_unstale":"2024-04-23T10:26:40.483019+0000","last_undegraded":"2024-04-23T10:26:40.483019+0000","last_fullsized":"2024-04-23T10:26:40.483019+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T10:24:42.023127+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T10:24:42.023127+0000","last_clean_scrub_stamp":"2024-04-23T10:24:42.023127+0000","objects_scrubbed":0,"log_size":44,"log_dups_size":0,"ondisk_log_size":44,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T19:39:46.369132+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":393216,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":82,"num_read_kb":70,"num_write":78,"num_write_kb":836,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,8,3],"acting":[9,8,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.7","version":"89'5","reported_seq":24,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-23T10:26:40.374352+0000","last_change":"2024-04-23T10:24:44.055920+0000","last_active":"2024-04-23T10:26:40.374352+0000","last_peered":"2024-04-23T10:26:40.374352+0000","last_clean":"2024-04-23T10:26:40.374352+0000","last_became_active":"2024-04-23T10:24:44.055411+0000","last_became_peered":"2024-04-23T10:24:44.055411+0000","last_unstale":"2024-04-23T10:26:40.374352+0000","last_undegraded":"2024-04-23T10:26:40.374352+0000","last_fullsized":"2024-04-23T10:26:40.374352+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T10:24:43.025324+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T10:24:43.025324+0000","last_clean_scrub_stamp":"2024-04-23T10:24:43.025324+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T14:43:10.446790+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,11,7],"acting":[3,11,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.7","version":"87'4","reported_seq":25,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-23T10:26:40.459655+0000","last_change":"2024-04-23T10:24:43.047989+0000","last_active":"2024-04-23T10:26:40.459655+0000","last_peered":"2024-04-23T10:26:40.459655+0000","last_clean":"2024-04-23T10:26:40.459655+0000","last_became_active":"2024-04-23T10:24:43.047625+0000","last_became_peered":"2024-04-23T10:24:43.047625+0000","last_unstale":"2024-04-23T10:26:40.459655+0000","last_undegraded":"2024-04-23T10:26:40.459655+0000","last_fullsized":"2024-04-23T10:26:40.459655+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T10:24:42.023127+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T10:24:42.023127+0000","last_clean_scrub_stamp":"2024-04-23T10:24:42.023127+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T13:50:16.647176+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,11,6],"acting":[4,11,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.6","version":"89'7","reported_seq":26,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-23T10:26:40.459895+0000","last_change":"2024-04-23T10:24:44.049214+0000","last_active":"2024-04-23T10:26:40.459895+0000","last_peered":"2024-04-23T10:26:40.459895+0000","last_clean":"2024-04-23T10:26:40.459895+0000","last_became_active":"2024-04-23T10:24:44.048869+0000","last_became_peered":"2024-04-23T10:24:44.048869+0000","last_unstale":"2024-04-23T10:26:40.459895+0000","last_undegraded":"2024-04-23T10:26:40.459895+0000","last_fullsized":"2024-04-23T10:26:40.459895+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T10:24:43.025324+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T10:24:43.025324+0000","last_clean_scrub_stamp":"2024-04-23T10:24:43.025324+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T18:17:09.772428+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,4],"acting":[0,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.8","version":"87'2","reported_seq":23,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-23T10:26:40.281014+0000","last_change":"2024-04-23T10:24:43.046189+0000","last_active":"2024-04-23T10:26:40.281014+0000","last_peered":"2024-04-23T10:26:40.281014+0000","last_clean":"2024-04-23T10:26:40.281014+0000","last_became_active":"2024-04-23T10:24:43.045848+0000","last_became_peered":"2024-04-23T10:24:43.045848+0000","last_unstale":"2024-04-23T10:26:40.281014+0000","last_undegraded":"2024-04-23T10:26:40.281014+0000","last_fullsized":"2024-04-23T10:26:40.281014+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T10:24:42.023127+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T10:24:42.023127+0000","last_clean_scrub_stamp":"2024-04-23T10:24:42.023127+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T22:23:39.180377+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,0],"acting":[1,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.9","version":"89'2","reported_seq":21,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-23T10:26:40.373557+0000","last_change":"2024-04-23T10:24:44.048415+0000","last_active":"2024-04-23T10:26:40.373557+0000","last_peered":"2024-04-23T10:26:40.373557+0000","last_clean":"2024-04-23T10:26:40.373557+0000","last_became_active":"2024-04-23T10:24:44.048030+0000","last_became_peered":"2024-04-23T10:24:44.048030+0000","last_unstale":"2024-04-23T10:26:40.373557+0000","last_undegraded":"2024-04-23T10:26:40.373557+0000","last_fullsized":"2024-04-23T10:26:40.373557+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T10:24:43.025324+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T10:24:43.025324+0000","last_clean_scrub_stamp":"2024-04-23T10:24:43.025324+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T14:42:09.920927+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,11,10],"acting":[8,11,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"2.9","version":"87'3","reported_seq":24,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-23T10:26:40.373745+0000","last_change":"2024-04-23T10:24:43.041525+0000","last_active":"2024-04-23T10:26:40.373745+0000","last_peered":"2024-04-23T10:26:40.373745+0000","last_clean":"2024-04-23T10:26:40.373745+0000","last_became_active":"2024-04-23T10:24:43.041200+0000","last_became_peered":"2024-04-23T10:24:43.041200+0000","last_unstale":"2024-04-23T10:26:40.373745+0000","last_undegraded":"2024-04-23T10:26:40.373745+0000","last_fullsized":"2024-04-23T10:26:40.373745+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T10:24:42.023127+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T10:24:42.023127+0000","last_clean_scrub_stamp":"2024-04-23T10:24:42.023127+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T16:45:06.846991+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,9,10],"acting":[3,9,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.8","version":"89'9","reported_seq":28,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-23T10:26:40.460025+0000","last_change":"2024-04-23T10:24:44.055341+0000","last_active":"2024-04-23T10:26:40.460025+0000","last_peered":"2024-04-23T10:26:40.460025+0000","last_clean":"2024-04-23T10:26:40.460025+0000","last_became_active":"2024-04-23T10:24:44.054216+0000","last_became_peered":"2024-04-23T10:24:44.054216+0000","last_unstale":"2024-04-23T10:26:40.460025+0000","last_undegraded":"2024-04-23T10:26:40.460025+0000","last_fullsized":"2024-04-23T10:26:40.460025+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T10:24:43.025324+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T10:24:43.025324+0000","last_clean_scrub_stamp":"2024-04-23T10:24:43.025324+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T21:31:07.753283+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,0,2],"acting":[11,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"2.a","version":"87'1","reported_seq":22,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-23T10:26:40.459791+0000","last_change":"2024-04-23T10:24:43.045400+0000","last_active":"2024-04-23T10:26:40.459791+0000","last_peered":"2024-04-23T10:26:40.459791+0000","last_clean":"2024-04-23T10:26:40.459791+0000","last_became_active":"2024-04-23T10:24:43.044677+0000","last_became_peered":"2024-04-23T10:24:43.044677+0000","last_unstale":"2024-04-23T10:26:40.459791+0000","last_undegraded":"2024-04-23T10:26:40.459791+0000","last_fullsized":"2024-04-23T10:26:40.459791+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T10:24:42.023127+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T10:24:42.023127+0000","last_clean_scrub_stamp":"2024-04-23T10:24:42.023127+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T19:06:19.700132+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,8],"acting":[2,3,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.b","version":"89'8","reported_seq":27,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-23T10:26:40.374065+0000","last_change":"2024-04-23T10:24:44.049968+0000","last_active":"2024-04-23T10:26:40.374065+0000","last_peered":"2024-04-23T10:26:40.374065+0000","last_clean":"2024-04-23T10:26:40.374065+0000","last_became_active":"2024-04-23T10:24:44.049621+0000","last_became_peered":"2024-04-23T10:24:44.049621+0000","last_unstale":"2024-04-23T10:26:40.374065+0000","last_undegraded":"2024-04-23T10:26:40.374065+0000","last_fullsized":"2024-04-23T10:26:40.374065+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T10:24:43.025324+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T10:24:43.025324+0000","last_clean_scrub_stamp":"2024-04-23T10:24:43.025324+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T16:23:43.082831+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,10],"acting":[3,0,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.b","version":"87'4","reported_seq":26,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-23T10:26:40.373780+0000","last_change":"2024-04-23T10:24:43.041828+0000","last_active":"2024-04-23T10:26:40.373780+0000","last_peered":"2024-04-23T10:26:40.373780+0000","last_clean":"2024-04-23T10:26:40.373780+0000","last_became_active":"2024-04-23T10:24:43.041480+0000","last_became_peered":"2024-04-23T10:24:43.041480+0000","last_unstale":"2024-04-23T10:26:40.373780+0000","last_undegraded":"2024-04-23T10:26:40.373780+0000","last_fullsized":"2024-04-23T10:26:40.373780+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T10:24:42.023127+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T10:24:42.023127+0000","last_clean_scrub_stamp":"2024-04-23T10:24:42.023127+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T14:43:03.203971+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":2,"num_read_kb":2,"num_write":4,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,2,7],"acting":[8,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.a","version":"89'5","reported_seq":24,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-23T10:26:40.522120+0000","last_change":"2024-04-23T10:24:44.062103+0000","last_active":"2024-04-23T10:26:40.522120+0000","last_peered":"2024-04-23T10:26:40.522120+0000","last_clean":"2024-04-23T10:26:40.522120+0000","last_became_active":"2024-04-23T10:24:44.061794+0000","last_became_peered":"2024-04-23T10:24:44.061794+0000","last_unstale":"2024-04-23T10:26:40.522120+0000","last_undegraded":"2024-04-23T10:26:40.522120+0000","last_fullsized":"2024-04-23T10:26:40.522120+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T10:24:43.025324+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T10:24:43.025324+0000","last_clean_scrub_stamp":"2024-04-23T10:24:43.025324+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T10:45:30.130047+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,6],"acting":[7,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.c","version":"87'3","reported_seq":25,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-23T10:26:40.460001+0000","last_change":"2024-04-23T10:24:43.045702+0000","last_active":"2024-04-23T10:26:40.460001+0000","last_peered":"2024-04-23T10:26:40.460001+0000","last_clean":"2024-04-23T10:26:40.460001+0000","last_became_active":"2024-04-23T10:24:43.045258+0000","last_became_peered":"2024-04-23T10:24:43.045258+0000","last_unstale":"2024-04-23T10:26:40.460001+0000","last_undegraded":"2024-04-23T10:26:40.460001+0000","last_fullsized":"2024-04-23T10:26:40.460001+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T10:24:42.023127+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T10:24:42.023127+0000","last_clean_scrub_stamp":"2024-04-23T10:24:42.023127+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T12:54:30.965080+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,11,0],"acting":[2,11,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.d","version":"89'7","reported_seq":26,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-23T10:26:40.522390+0000","last_change":"2024-04-23T10:24:44.061118+0000","last_active":"2024-04-23T10:26:40.522390+0000","last_peered":"2024-04-23T10:26:40.522390+0000","last_clean":"2024-04-23T10:26:40.522390+0000","last_became_active":"2024-04-23T10:24:44.060743+0000","last_became_peered":"2024-04-23T10:24:44.060743+0000","last_unstale":"2024-04-23T10:26:40.522390+0000","last_undegraded":"2024-04-23T10:26:40.522390+0000","last_fullsized":"2024-04-23T10:26:40.522390+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T10:24:43.025324+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T10:24:43.025324+0000","last_clean_scrub_stamp":"2024-04-23T10:24:43.025324+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T17:17:46.731121+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,11],"acting":[7,1,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.d","version":"90'39","reported_seq":68,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-23T10:26:40.480483+0000","last_change":"2024-04-23T10:24:43.052519+0000","last_active":"2024-04-23T10:26:40.480483+0000","last_peered":"2024-04-23T10:26:40.480483+0000","last_clean":"2024-04-23T10:26:40.480483+0000","last_became_active":"2024-04-23T10:24:43.052187+0000","last_became_peered":"2024-04-23T10:24:43.052187+0000","last_unstale":"2024-04-23T10:26:40.480483+0000","last_undegraded":"2024-04-23T10:26:40.480483+0000","last_fullsized":"2024-04-23T10:26:40.480483+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T10:24:42.023127+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T10:24:42.023127+0000","last_clean_scrub_stamp":"2024-04-23T10:24:42.023127+0000","objects_scrubbed":0,"log_size":39,"log_dups_size":0,"ondisk_log_size":39,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T15:51:52.839409+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":131670,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":11,"num_read_kb":10,"num_write":49,"num_write_kb":475,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,10,3],"acting":[5,10,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.c","version":"89'7","reported_seq":26,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-23T10:26:40.521835+0000","last_change":"2024-04-23T10:24:44.053086+0000","last_active":"2024-04-23T10:26:40.521835+0000","last_peered":"2024-04-23T10:26:40.521835+0000","last_clean":"2024-04-23T10:26:40.521835+0000","last_became_active":"2024-04-23T10:24:44.052687+0000","last_became_peered":"2024-04-23T10:24:44.052687+0000","last_unstale":"2024-04-23T10:26:40.521835+0000","last_undegraded":"2024-04-23T10:26:40.521835+0000","last_fullsized":"2024-04-23T10:26:40.521835+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T10:24:43.025324+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T10:24:43.025324+0000","last_clean_scrub_stamp":"2024-04-23T10:24:43.025324+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T14:35:44.958742+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":117,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,5],"acting":[7,3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.e","version":"86'2","reported_seq":33,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-23T10:26:40.278094+0000","last_change":"2024-04-23T10:25:47.724936+0000","last_active":"2024-04-23T10:26:40.278094+0000","last_peered":"2024-04-23T10:26:40.278094+0000","last_clean":"2024-04-23T10:26:40.278094+0000","last_became_active":"2024-04-23T10:25:47.722182+0000","last_became_peered":"2024-04-23T10:25:47.722182+0000","last_unstale":"2024-04-23T10:26:40.278094+0000","last_undegraded":"2024-04-23T10:26:40.278094+0000","last_fullsized":"2024-04-23T10:26:40.278094+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T10:24:42.023127+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T10:24:42.023127+0000","last_clean_scrub_stamp":"2024-04-23T10:24:42.023127+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T19:18:42.762648+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,4],"acting":[6,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.f","version":"89'9","reported_seq":28,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-23T10:26:40.522498+0000","last_change":"2024-04-23T10:24:44.049543+0000","last_active":"2024-04-23T10:26:40.522498+0000","last_peered":"2024-04-23T10:26:40.522498+0000","last_clean":"2024-04-23T10:26:40.522498+0000","last_became_active":"2024-04-23T10:24:44.049198+0000","last_became_peered":"2024-04-23T10:24:44.049198+0000","last_unstale":"2024-04-23T10:26:40.522498+0000","last_undegraded":"2024-04-23T10:26:40.522498+0000","last_fullsized":"2024-04-23T10:26:40.522498+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T10:24:43.025324+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T10:24:43.025324+0000","last_clean_scrub_stamp":"2024-04-23T10:24:43.025324+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T12:25:15.974267+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,11,8],"acting":[7,11,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.20","version":"89'9","reported_seq":28,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-23T10:26:40.460095+0000","last_change":"2024-04-23T10:24:44.054440+0000","last_active":"2024-04-23T10:26:40.460095+0000","last_peered":"2024-04-23T10:26:40.460095+0000","last_clean":"2024-04-23T10:26:40.460095+0000","last_became_active":"2024-04-23T10:24:44.053951+0000","last_became_peered":"2024-04-23T10:24:44.053951+0000","last_unstale":"2024-04-23T10:26:40.460095+0000","last_undegraded":"2024-04-23T10:26:40.460095+0000","last_fullsized":"2024-04-23T10:26:40.460095+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T10:24:43.025324+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T10:24:43.025324+0000","last_clean_scrub_stamp":"2024-04-23T10:24:43.025324+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T22:20:10.452712+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,2,5],"acting":[11,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.21","version":"89'12","reported_seq":31,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-23T10:26:40.459759+0000","last_change":"2024-04-23T10:24:44.059015+0000","last_active":"2024-04-23T10:26:40.459759+0000","last_peered":"2024-04-23T10:26:40.459759+0000","last_clean":"2024-04-23T10:26:40.459759+0000","last_became_active":"2024-04-23T10:24:44.058691+0000","last_became_peered":"2024-04-23T10:24:44.058691+0000","last_unstale":"2024-04-23T10:26:40.459759+0000","last_undegraded":"2024-04-23T10:26:40.459759+0000","last_fullsized":"2024-04-23T10:26:40.459759+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T10:24:43.025324+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T10:24:43.025324+0000","last_clean_scrub_stamp":"2024-04-23T10:24:43.025324+0000","objects_scrubbed":0,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T14:57:48.153094+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,4],"acting":[2,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.22","version":"89'8","reported_seq":27,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-23T10:26:40.459657+0000","last_change":"2024-04-23T10:24:44.055534+0000","last_active":"2024-04-23T10:26:40.459657+0000","last_peered":"2024-04-23T10:26:40.459657+0000","last_clean":"2024-04-23T10:26:40.459657+0000","last_became_active":"2024-04-23T10:24:44.055181+0000","last_became_peered":"2024-04-23T10:24:44.055181+0000","last_unstale":"2024-04-23T10:26:40.459657+0000","last_undegraded":"2024-04-23T10:26:40.459657+0000","last_fullsized":"2024-04-23T10:26:40.459657+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T10:24:43.025324+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T10:24:43.025324+0000","last_clean_scrub_stamp":"2024-04-23T10:24:43.025324+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T13:56:33.878232+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.23","version":"89'7","reported_seq":32,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-23T10:26:40.460085+0000","last_change":"2024-04-23T10:25:47.587380+0000","last_active":"2024-04-23T10:26:40.460085+0000","last_peered":"2024-04-23T10:26:40.460085+0000","last_clean":"2024-04-23T10:26:40.460085+0000","last_became_active":"2024-04-23T10:25:47.586893+0000","last_became_peered":"2024-04-23T10:25:47.586893+0000","last_unstale":"2024-04-23T10:26:40.460085+0000","last_undegraded":"2024-04-23T10:26:40.460085+0000","last_fullsized":"2024-04-23T10:26:40.460085+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T10:24:43.025324+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T10:24:43.025324+0000","last_clean_scrub_stamp":"2024-04-23T10:24:43.025324+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T10:51:34.390594+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,7],"acting":[0,6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.24","version":"89'4","reported_seq":23,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-23T10:26:40.278003+0000","last_change":"2024-04-23T10:24:44.050741+0000","last_active":"2024-04-23T10:26:40.278003+0000","last_peered":"2024-04-23T10:26:40.278003+0000","last_clean":"2024-04-23T10:26:40.278003+0000","last_became_active":"2024-04-23T10:24:44.050398+0000","last_became_peered":"2024-04-23T10:24:44.050398+0000","last_unstale":"2024-04-23T10:26:40.278003+0000","last_undegraded":"2024-04-23T10:26:40.278003+0000","last_fullsized":"2024-04-23T10:26:40.278003+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T10:24:43.025324+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T10:24:43.025324+0000","last_clean_scrub_stamp":"2024-04-23T10:24:43.025324+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T10:49:07.146673+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,0],"acting":[6,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.25","version":"89'6","reported_seq":25,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-23T10:26:40.460199+0000","last_change":"2024-04-23T10:24:44.048933+0000","last_active":"2024-04-23T10:26:40.460199+0000","last_peered":"2024-04-23T10:26:40.460199+0000","last_clean":"2024-04-23T10:26:40.460199+0000","last_became_active":"2024-04-23T10:24:44.048553+0000","last_became_peered":"2024-04-23T10:24:44.048553+0000","last_unstale":"2024-04-23T10:26:40.460199+0000","last_undegraded":"2024-04-23T10:26:40.460199+0000","last_fullsized":"2024-04-23T10:26:40.460199+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T10:24:43.025324+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T10:24:43.025324+0000","last_clean_scrub_stamp":"2024-04-23T10:24:43.025324+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T10:40:24.569906+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,8,5],"acting":[11,8,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.26","version":"89'15","reported_seq":34,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-23T10:26:40.460160+0000","last_change":"2024-04-23T10:24:44.051895+0000","last_active":"2024-04-23T10:26:40.460160+0000","last_peered":"2024-04-23T10:26:40.460160+0000","last_clean":"2024-04-23T10:26:40.460160+0000","last_became_active":"2024-04-23T10:24:44.051005+0000","last_became_peered":"2024-04-23T10:24:44.051005+0000","last_unstale":"2024-04-23T10:26:40.460160+0000","last_undegraded":"2024-04-23T10:26:40.460160+0000","last_fullsized":"2024-04-23T10:26:40.460160+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T10:24:43.025324+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T10:24:43.025324+0000","last_clean_scrub_stamp":"2024-04-23T10:24:43.025324+0000","objects_scrubbed":0,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T18:13:05.227365+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,8,1],"acting":[2,8,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.27","version":"89'2","reported_seq":27,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-23T10:26:40.374327+0000","last_change":"2024-04-23T10:25:47.587504+0000","last_active":"2024-04-23T10:26:40.374327+0000","last_peered":"2024-04-23T10:26:40.374327+0000","last_clean":"2024-04-23T10:26:40.374327+0000","last_became_active":"2024-04-23T10:25:47.587157+0000","last_became_peered":"2024-04-23T10:25:47.587157+0000","last_unstale":"2024-04-23T10:26:40.374327+0000","last_undegraded":"2024-04-23T10:26:40.374327+0000","last_fullsized":"2024-04-23T10:26:40.374327+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T10:24:43.025324+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T10:24:43.025324+0000","last_clean_scrub_stamp":"2024-04-23T10:24:43.025324+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T18:46:25.090568+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,2],"acting":[3,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.28","version":"89'6","reported_seq":25,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-23T10:26:40.281576+0000","last_change":"2024-04-23T10:24:44.044158+0000","last_active":"2024-04-23T10:26:40.281576+0000","last_peered":"2024-04-23T10:26:40.281576+0000","last_clean":"2024-04-23T10:26:40.281576+0000","last_became_active":"2024-04-23T10:24:44.043829+0000","last_became_peered":"2024-04-23T10:24:44.043829+0000","last_unstale":"2024-04-23T10:26:40.281576+0000","last_undegraded":"2024-04-23T10:26:40.281576+0000","last_fullsized":"2024-04-23T10:26:40.281576+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T10:24:43.025324+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T10:24:43.025324+0000","last_clean_scrub_stamp":"2024-04-23T10:24:43.025324+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T15:10:08.081831+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,9,3],"acting":[5,9,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.29","version":"89'8","reported_seq":27,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-23T10:26:40.373723+0000","last_change":"2024-04-23T10:24:44.051267+0000","last_active":"2024-04-23T10:26:40.373723+0000","last_peered":"2024-04-23T10:26:40.373723+0000","last_clean":"2024-04-23T10:26:40.373723+0000","last_became_active":"2024-04-23T10:24:44.050911+0000","last_became_peered":"2024-04-23T10:24:44.050911+0000","last_unstale":"2024-04-23T10:26:40.373723+0000","last_undegraded":"2024-04-23T10:26:40.373723+0000","last_fullsized":"2024-04-23T10:26:40.373723+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T10:24:43.025324+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T10:24:43.025324+0000","last_clean_scrub_stamp":"2024-04-23T10:24:43.025324+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T18:09:28.871427+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,11],"acting":[3,2,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.2a","version":"89'10","reported_seq":29,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-23T10:26:40.281065+0000","last_change":"2024-04-23T10:24:44.057653+0000","last_active":"2024-04-23T10:26:40.281065+0000","last_peered":"2024-04-23T10:26:40.281065+0000","last_clean":"2024-04-23T10:26:40.281065+0000","last_became_active":"2024-04-23T10:24:44.056627+0000","last_became_peered":"2024-04-23T10:24:44.056627+0000","last_unstale":"2024-04-23T10:26:40.281065+0000","last_undegraded":"2024-04-23T10:26:40.281065+0000","last_fullsized":"2024-04-23T10:26:40.281065+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T10:24:43.025324+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T10:24:43.025324+0000","last_clean_scrub_stamp":"2024-04-23T10:24:43.025324+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T18:29:21.480187+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,3],"acting":[1,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.2b","version":"89'11","reported_seq":30,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-23T10:26:40.278095+0000","last_change":"2024-04-23T10:24:44.059735+0000","last_active":"2024-04-23T10:26:40.278095+0000","last_peered":"2024-04-23T10:26:40.278095+0000","last_clean":"2024-04-23T10:26:40.278095+0000","last_became_active":"2024-04-23T10:24:44.059052+0000","last_became_peered":"2024-04-23T10:24:44.059052+0000","last_unstale":"2024-04-23T10:26:40.278095+0000","last_undegraded":"2024-04-23T10:26:40.278095+0000","last_fullsized":"2024-04-23T10:26:40.278095+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T10:24:43.025324+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T10:24:43.025324+0000","last_clean_scrub_stamp":"2024-04-23T10:24:43.025324+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T14:29:25.794655+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,0],"acting":[6,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.2c","version":"89'8","reported_seq":27,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-23T10:26:40.459891+0000","last_change":"2024-04-23T10:24:44.051831+0000","last_active":"2024-04-23T10:26:40.459891+0000","last_peered":"2024-04-23T10:26:40.459891+0000","last_clean":"2024-04-23T10:26:40.459891+0000","last_became_active":"2024-04-23T10:24:44.051480+0000","last_became_peered":"2024-04-23T10:24:44.051480+0000","last_unstale":"2024-04-23T10:26:40.459891+0000","last_undegraded":"2024-04-23T10:26:40.459891+0000","last_fullsized":"2024-04-23T10:26:40.459891+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T10:24:43.025324+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T10:24:43.025324+0000","last_clean_scrub_stamp":"2024-04-23T10:24:43.025324+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T19:20:55.181393+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,4,8],"acting":[11,4,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]}],"pool_stats":[{"poolid":3,"num_pg":64,"stat_sum":{"num_bytes":234,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":702,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":502,"ondisk_log_size":502,"up":192,"acting":192,"num_store_stats":12},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":136,"num_read_kb":119,"num_write":181,"num_write_kb":2938,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":1180736,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1781760,"data_stored":1771104,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":137,"ondisk_log_size":137,"up":3,"acting":3,"num_store_stats":3},{"poolid":2,"num_pg":32,"stat_sum":{"num_bytes":598030,"num_objects":62,"num_object_clones":0,"num_object_copies":186,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":62,"num_whiteouts":0,"num_read":802,"num_read_kb":510,"num_write":285,"num_write_kb":1437,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":10,"num_objects_omap":40,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":2027520,"data_stored":1794090,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":226,"ondisk_log_size":226,"up":96,"acting":96,"num_store_stats":12}],"osd_stats":[{"osd":11,"up_from":82,"seq":352187318309,"num_pgs":19,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32148,"kb_used_data":968,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93713004,"statfs":{"total":95995035648,"available":95962116096,"internally_reserved":0,"allocated":991232,"data_stored":474591,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1596,"internal_metadata":31914436},"hb_peers":[0,1,2,4,5,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":10,"up_from":74,"seq":317827579945,"num_pgs":26,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28656,"kb_used_data":1640,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93716496,"statfs":{"total":95995035648,"available":95965691904,"internally_reserved":0,"allocated":1679360,"data_stored":1124595,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1600,"internal_metadata":27654592},"hb_peers":[0,1,2,3,4,5,6,8,9,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":9,"up_from":69,"seq":296352743471,"num_pgs":20,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28196,"kb_used_data":1176,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93716956,"statfs":{"total":95995035648,"available":95966162944,"internally_reserved":0,"allocated":1204224,"data_stored":671435,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1600,"internal_metadata":27654592},"hb_peers":[0,1,2,3,4,5,6,7,8,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":8,"up_from":62,"seq":266287972405,"num_pgs":25,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32548,"kb_used_data":1368,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93712604,"statfs":{"total":95995035648,"available":95961706496,"internally_reserved":0,"allocated":1400832,"data_stored":867963,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,2,3,4,5,6,7,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":7,"up_from":53,"seq":227633266745,"num_pgs":26,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":36904,"kb_used_data":1564,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":93708248,"statfs":{"total":95995035648,"available":95957245952,"internally_reserved":0,"allocated":1601536,"data_stored":1068132,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":36174282},"hb_peers":[1,2,3,4,5,6,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":6,"up_from":47,"seq":201863462975,"num_pgs":22,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32152,"kb_used_data":972,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93713000,"statfs":{"total":95995035648,"available":95962112000,"internally_reserved":0,"allocated":995328,"data_stored":474216,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":31914438},"hb_peers":[0,1,2,3,4,5,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":13,"seq":55834574934,"num_pgs":24,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28060,"kb_used_data":1040,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717092,"statfs":{"total":95995035648,"available":95966302208,"internally_reserved":0,"allocated":1064960,"data_stored":541767,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,2,3,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":8,"seq":34359738460,"num_pgs":25,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":36332,"kb_used_data":992,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":93708820,"statfs":{"total":95995035648,"available":95957831680,"internally_reserved":0,"allocated":1015808,"data_stored":475487,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":36174278},"hb_peers":[1,2,3,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":18,"seq":77309411410,"num_pgs":27,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":36404,"kb_used_data":1064,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":93708748,"statfs":{"total":95995035648,"available":95957757952,"internally_reserved":0,"allocated":1089536,"data_stored":544040,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":36174277},"hb_peers":[0,1,3,4,5,6,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":25,"seq":107374182474,"num_pgs":26,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":27988,"kb_used_data":972,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717164,"statfs":{"total":95995035648,"available":95966375936,"internally_reserved":0,"allocated":995328,"data_stored":465542,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[0,1,2,4,5,6,7,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":4,"up_from":33,"seq":141733920839,"num_pgs":23,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":27996,"kb_used_data":976,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717156,"statfs":{"total":95995035648,"available":95966367744,"internally_reserved":0,"allocated":999424,"data_stored":475185,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[1,2,3,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":39,"seq":167503724611,"num_pgs":28,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":37044,"kb_used_data":1708,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":93708108,"statfs":{"total":95995035648,"available":95957102592,"internally_reserved":0,"allocated":1748992,"data_stored":1198493,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":36174277},"hb_peers":[0,1,2,3,4,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":28672,"data_stored":1478,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":77824,"data_stored":67758,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":102400,"data_stored":70031,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":544768,"data_stored":525100,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":1567,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":163840,"data_stored":133999,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":90,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":3638,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":413696,"data_stored":393954,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":413696,"data_stored":394034,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":245760,"data_stored":201859,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":582,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":234,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":117,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":117,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":117,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":117,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-04-23T10:26:46.157 DEBUG:teuthology.orchestra.run.smithi114:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-23T10:26:46.294 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:46.293+0000 7fd850968640 1 -- 172.21.15.114:0/1840861600 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fd84c101a50 msgr2=0x7fd84c105a90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T10:26:46.294 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:46.293+0000 7fd850968640 1 --2- 172.21.15.114:0/1840861600 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fd84c101a50 0x7fd84c105a90 secure :-1 s=READY pgs=450 cs=0 l=1 rev1=1 crypto rx=0x7fd840009900 tx=0x7fd84002a410 comp rx=0 tx=0).stop 2024-04-23T10:26:46.294 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:46.293+0000 7fd850968640 1 -- 172.21.15.114:0/1840861600 shutdown_connections 2024-04-23T10:26:46.294 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:46.293+0000 7fd850968640 1 --2- 172.21.15.114:0/1840861600 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fd84c106170 0x7fd84c10dd40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:26:46.295 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:46.293+0000 7fd850968640 1 --2- 172.21.15.114:0/1840861600 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fd84c101a50 0x7fd84c105a90 unknown :-1 s=CLOSED pgs=450 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:26:46.295 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:46.293+0000 7fd850968640 1 --2- 172.21.15.114:0/1840861600 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fd84c1010a0 0x7fd84c101480 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:26:46.295 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:46.293+0000 7fd850968640 1 -- 172.21.15.114:0/1840861600 >> 172.21.15.114:0/1840861600 conn(0x7fd84c077f50 msgr2=0x7fd84c078360 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T10:26:46.295 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:46.293+0000 7fd850968640 1 -- 172.21.15.114:0/1840861600 shutdown_connections 2024-04-23T10:26:46.295 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:46.293+0000 7fd850968640 1 -- 172.21.15.114:0/1840861600 wait complete. 2024-04-23T10:26:46.295 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:46.294+0000 7fd850968640 1 Processor -- start 2024-04-23T10:26:46.295 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:46.294+0000 7fd850968640 1 -- start start 2024-04-23T10:26:46.295 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:46.294+0000 7fd850968640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fd84c1010a0 0x7fd84c199590 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T10:26:46.295 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:46.294+0000 7fd850968640 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fd84c106170 0x7fd84c199ad0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T10:26:46.295 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:46.294+0000 7fd850968640 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fd84c19a060 0x7fd84c1a4610 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T10:26:46.295 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:46.294+0000 7fd850968640 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x7fd84c1110b0 con 0x7fd84c1010a0 2024-04-23T10:26:46.295 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:46.294+0000 7fd850968640 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 -- 0x7fd84c110f30 con 0x7fd84c106170 2024-04-23T10:26:46.295 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:46.294+0000 7fd850968640 1 -- --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_getmap magic: 0 -- 0x7fd84c111230 con 0x7fd84c19a060 2024-04-23T10:26:46.296 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:46.294+0000 7fd84a59c640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fd84c1010a0 0x7fd84c199590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T10:26:46.296 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:46.294+0000 7fd84a59c640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fd84c1010a0 0x7fd84c199590 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.114:51910/0 (socket says 172.21.15.114:51910) 2024-04-23T10:26:46.296 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:46.294+0000 7fd84a59c640 1 -- 172.21.15.114:0/3878333711 learned_addr learned my addr 172.21.15.114:0/3878333711 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-04-23T10:26:46.296 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:46.294+0000 7fd84ad9d640 1 --2- 172.21.15.114:0/3878333711 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fd84c19a060 0x7fd84c1a4610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T10:26:46.296 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:46.295+0000 7fd849d9b640 1 --2- 172.21.15.114:0/3878333711 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fd84c106170 0x7fd84c199ad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T10:26:46.296 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:46.295+0000 7fd84a59c640 1 -- 172.21.15.114:0/3878333711 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fd84c19a060 msgr2=0x7fd84c1a4610 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T10:26:46.296 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:46.295+0000 7fd84a59c640 1 --2- 172.21.15.114:0/3878333711 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fd84c19a060 0x7fd84c1a4610 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:26:46.296 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:46.295+0000 7fd84a59c640 1 -- 172.21.15.114:0/3878333711 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fd84c106170 msgr2=0x7fd84c199ad0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T10:26:46.296 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:46.295+0000 7fd84a59c640 1 --2- 172.21.15.114:0/3878333711 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fd84c106170 0x7fd84c199ad0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:26:46.296 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:46.295+0000 7fd84a59c640 1 -- 172.21.15.114:0/3878333711 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd84c1a4de0 con 0x7fd84c1010a0 2024-04-23T10:26:46.296 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:46.295+0000 7fd84ad9d640 1 --2- 172.21.15.114:0/3878333711 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fd84c19a060 0x7fd84c1a4610 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-23T10:26:46.296 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:46.295+0000 7fd849d9b640 1 --2- 172.21.15.114:0/3878333711 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fd84c106170 0x7fd84c199ad0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-23T10:26:46.296 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:46.295+0000 7fd84a59c640 1 --2- 172.21.15.114:0/3878333711 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fd84c1010a0 0x7fd84c199590 secure :-1 s=READY pgs=451 cs=0 l=1 rev1=1 crypto rx=0x7fd83c00e9f0 tx=0x7fd83c00edf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T10:26:46.297 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:46.295+0000 7fd83b7fe640 1 -- 172.21.15.114:0/3878333711 <== mon.0 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd83c00cd80 con 0x7fd84c1010a0 2024-04-23T10:26:46.297 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:46.295+0000 7fd850968640 1 -- 172.21.15.114:0/3878333711 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fd84c1a50d0 con 0x7fd84c1010a0 2024-04-23T10:26:46.297 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:46.295+0000 7fd850968640 1 -- 172.21.15.114:0/3878333711 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fd84c1a55f0 con 0x7fd84c1010a0 2024-04-23T10:26:46.297 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:46.295+0000 7fd83b7fe640 1 -- 172.21.15.114:0/3878333711 <== mon.0 v2:172.21.15.114:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fd83c0054d0 con 0x7fd84c1010a0 2024-04-23T10:26:46.298 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:46.295+0000 7fd83b7fe640 1 -- 172.21.15.114:0/3878333711 <== mon.0 v2:172.21.15.114:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd83c010690 con 0x7fd84c1010a0 2024-04-23T10:26:46.298 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:46.297+0000 7fd850968640 1 -- 172.21.15.114:0/3878333711 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fd82c005290 con 0x7fd84c1010a0 2024-04-23T10:26:46.298 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:46.297+0000 7fd83b7fe640 1 -- 172.21.15.114:0/3878333711 <== mon.0 v2:172.21.15.114:3300/0 4 ==== mgrmap(e 24) ==== 138823+0+0 (secure 0 0 0) 0x7fd83c0046c0 con 0x7fd84c1010a0 2024-04-23T10:26:46.298 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:46.298+0000 7fd83b7fe640 1 --2- 172.21.15.114:0/3878333711 >> [v2:172.21.15.114:6800/1395693466,v1:172.21.15.114:6801/1395693466] conn(0x7fd8340a3e30 0x7fd8340a62f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T10:26:46.298 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:46.298+0000 7fd849d9b640 1 --2- 172.21.15.114:0/3878333711 >> [v2:172.21.15.114:6800/1395693466,v1:172.21.15.114:6801/1395693466] conn(0x7fd8340a3e30 0x7fd8340a62f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T10:26:46.298 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:46.298+0000 7fd83b7fe640 1 -- 172.21.15.114:0/3878333711 <== mon.0 v2:172.21.15.114:3300/0 5 ==== osd_map(90..90 src has 1..90) ==== 8527+0+0 (secure 0 0 0) 0x7fd83c0d0ca0 con 0x7fd84c1010a0 2024-04-23T10:26:46.298 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:46.299+0000 7fd849d9b640 1 --2- 172.21.15.114:0/3878333711 >> [v2:172.21.15.114:6800/1395693466,v1:172.21.15.114:6801/1395693466] conn(0x7fd8340a3e30 0x7fd8340a62f0 secure :-1 s=READY pgs=51 cs=0 l=1 rev1=1 crypto rx=0x7fd840000c00 tx=0x7fd840009610 comp rx=0 tx=0).ready entity=mgr.15291 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T10:26:46.301 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:46.302+0000 7fd83b7fe640 1 -- 172.21.15.114:0/3878333711 <== mon.0 v2:172.21.15.114:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7fd83c09d1d0 con 0x7fd84c1010a0 2024-04-23T10:26:46.328 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 23 10:26:46 smithi204 ceph-mon[32649]: Updating smithi204:/etc/ceph/ceph.client.admin.keyring 2024-04-23T10:26:46.328 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 23 10:26:46 smithi204 ceph-mon[32649]: Updating smithi114:/etc/ceph/ceph.client.admin.keyring 2024-04-23T10:26:46.328 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 23 10:26:46 smithi204 ceph-mon[32649]: Updating smithi119:/etc/ceph/ceph.client.admin.keyring 2024-04-23T10:26:46.328 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 23 10:26:46 smithi204 ceph-mon[32649]: pgmap v5: 97 pgs: 97 active+clean; 1.1 MiB data, 375 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-23T10:26:46.328 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 23 10:26:46 smithi204 ceph-mon[32649]: Updating smithi204:/var/lib/ceph/aa240572-0159-11ef-bc93-c7b262605968/config/ceph.client.admin.keyring 2024-04-23T10:26:46.328 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 23 10:26:46 smithi204 ceph-mon[32649]: Updating smithi114:/var/lib/ceph/aa240572-0159-11ef-bc93-c7b262605968/config/ceph.client.admin.keyring 2024-04-23T10:26:46.328 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 23 10:26:46 smithi204 ceph-mon[32649]: Updating smithi119:/var/lib/ceph/aa240572-0159-11ef-bc93-c7b262605968/config/ceph.client.admin.keyring 2024-04-23T10:26:46.328 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 23 10:26:46 smithi204 ceph-mon[32649]: mgrmap e24: x(active, since 4s), standbys: z, y 2024-04-23T10:26:46.328 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 23 10:26:46 smithi204 ceph-mon[32649]: from='client.? 172.21.15.114:0/3698241698' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2024-04-23T10:26:46.492 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 23 10:26:46 smithi119 ceph-mon[32608]: Updating smithi204:/etc/ceph/ceph.client.admin.keyring 2024-04-23T10:26:46.492 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 23 10:26:46 smithi119 ceph-mon[32608]: Updating smithi114:/etc/ceph/ceph.client.admin.keyring 2024-04-23T10:26:46.492 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 23 10:26:46 smithi119 ceph-mon[32608]: Updating smithi119:/etc/ceph/ceph.client.admin.keyring 2024-04-23T10:26:46.492 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 23 10:26:46 smithi119 ceph-mon[32608]: pgmap v5: 97 pgs: 97 active+clean; 1.1 MiB data, 375 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-23T10:26:46.492 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 23 10:26:46 smithi119 ceph-mon[32608]: Updating smithi204:/var/lib/ceph/aa240572-0159-11ef-bc93-c7b262605968/config/ceph.client.admin.keyring 2024-04-23T10:26:46.492 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 23 10:26:46 smithi119 ceph-mon[32608]: Updating smithi114:/var/lib/ceph/aa240572-0159-11ef-bc93-c7b262605968/config/ceph.client.admin.keyring 2024-04-23T10:26:46.492 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 23 10:26:46 smithi119 ceph-mon[32608]: Updating smithi119:/var/lib/ceph/aa240572-0159-11ef-bc93-c7b262605968/config/ceph.client.admin.keyring 2024-04-23T10:26:46.492 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 23 10:26:46 smithi119 ceph-mon[32608]: mgrmap e24: x(active, since 4s), standbys: z, y 2024-04-23T10:26:46.492 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 23 10:26:46 smithi119 ceph-mon[32608]: from='client.? 172.21.15.114:0/3698241698' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2024-04-23T10:26:46.520 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:46.521+0000 7fd850968640 1 -- 172.21.15.114:0/3878333711 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fd82c005850 con 0x7fd84c1010a0 2024-04-23T10:26:46.520 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:46.521+0000 7fd83b7fe640 1 -- 172.21.15.114:0/3878333711 <== mon.0 v2:172.21.15.114:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 21 v21) ==== 94+0+11884 (secure 0 0 0) 0x7fd83c09cad0 con 0x7fd84c1010a0 2024-04-23T10:26:46.521 INFO:teuthology.orchestra.run.smithi114.stdout: 2024-04-23T10:26:46.521 INFO:teuthology.orchestra.run.smithi114.stdout:{"epoch":21,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14547,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.114:6835/663493580","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6834","nonce":663493580},{"type":"v1","addr":"172.21.15.114:6835","nonce":663493580}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14565,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.114:6837/1439252154","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6836","nonce":1439252154},{"type":"v1","addr":"172.21.15.114:6837","nonce":1439252154}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5},{"gid":14634,"name":"g","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.114:6839/3849796939","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6838","nonce":3849796939},{"type":"v1","addr":"172.21.15.114:6839","nonce":3849796939}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":11},{"gid":14646,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.114:6841/3544523891","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6840","nonce":3544523891},{"type":"v1","addr":"172.21.15.114:6841","nonce":3544523891}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":12},{"gid":24385,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.204:6833/3866561754","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6832","nonce":3866561754},{"type":"v1","addr":"172.21.15.204:6833","nonce":3866561754}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2},{"gid":24427,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.204:6835/2533793172","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6834","nonce":2533793172},{"type":"v1","addr":"172.21.15.204:6835","nonce":2533793172}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":7}],"filesystems":[{"mdsmap":{"epoch":21,"flags":306,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":true},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-04-23T10:24:43.383158+0000","modified":"2024-04-23T10:26:38.945346+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":3,"in":[0,1,2],"up":{"mds_0":24545,"mds_1":24509,"mds_2":24479},"failed":[],"damaged":[],"stopped":[],"info":{"gid_24439":{"gid":24439,"name":"c","rank":2,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.204:6837/513822179","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6836","nonce":513822179},{"type":"v1","addr":"172.21.15.204:6837","nonce":513822179}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24443":{"gid":24443,"name":"k","rank":1,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.119:6833/783244660","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6832","nonce":783244660},{"type":"v1","addr":"172.21.15.119:6833","nonce":783244660}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24463":{"gid":24463,"name":"i","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.204:6839/3222658893","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6838","nonce":3222658893},{"type":"v1","addr":"172.21.15.204:6839","nonce":3222658893}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24479":{"gid":24479,"name":"e","rank":2,"incarnation":19,"state":"up:active","state_seq":9,"addr":"172.21.15.119:6835/2033916709","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6834","nonce":2033916709},{"type":"v1","addr":"172.21.15.119:6835","nonce":2033916709}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24509":{"gid":24509,"name":"h","rank":1,"incarnation":17,"state":"up:active","state_seq":6,"addr":"172.21.15.119:6837/1872363128","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6836","nonce":1872363128},{"type":"v1","addr":"172.21.15.119:6837","nonce":1872363128}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24545":{"gid":24545,"name":"b","rank":0,"incarnation":14,"state":"up:active","state_seq":3,"addr":"172.21.15.119:6839/937297340","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6838","nonce":937297340},{"type":"v1","addr":"172.21.15.119:6839","nonce":937297340}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":24545,"qdb_cluster":[24479,24509,24545]},"id":1}]} 2024-04-23T10:26:46.521 INFO:teuthology.orchestra.run.smithi114.stderr:dumped fsmap epoch 21 2024-04-23T10:26:46.523 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:46.523+0000 7fd850968640 1 -- 172.21.15.114:0/3878333711 >> [v2:172.21.15.114:6800/1395693466,v1:172.21.15.114:6801/1395693466] conn(0x7fd8340a3e30 msgr2=0x7fd8340a62f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T10:26:46.523 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:46.523+0000 7fd850968640 1 --2- 172.21.15.114:0/3878333711 >> [v2:172.21.15.114:6800/1395693466,v1:172.21.15.114:6801/1395693466] conn(0x7fd8340a3e30 0x7fd8340a62f0 secure :-1 s=READY pgs=51 cs=0 l=1 rev1=1 crypto rx=0x7fd840000c00 tx=0x7fd840009610 comp rx=0 tx=0).stop 2024-04-23T10:26:46.523 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:46.523+0000 7fd850968640 1 -- 172.21.15.114:0/3878333711 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fd84c1010a0 msgr2=0x7fd84c199590 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T10:26:46.523 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:46.523+0000 7fd850968640 1 --2- 172.21.15.114:0/3878333711 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fd84c1010a0 0x7fd84c199590 secure :-1 s=READY pgs=451 cs=0 l=1 rev1=1 crypto rx=0x7fd83c00e9f0 tx=0x7fd83c00edf0 comp rx=0 tx=0).stop 2024-04-23T10:26:46.523 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:46.524+0000 7fd850968640 1 -- 172.21.15.114:0/3878333711 shutdown_connections 2024-04-23T10:26:46.523 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:46.524+0000 7fd850968640 1 --2- 172.21.15.114:0/3878333711 >> [v2:172.21.15.114:6800/1395693466,v1:172.21.15.114:6801/1395693466] conn(0x7fd8340a3e30 0x7fd8340a62f0 unknown :-1 s=CLOSED pgs=51 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:26:46.523 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:46.524+0000 7fd850968640 1 --2- 172.21.15.114:0/3878333711 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fd84c19a060 0x7fd84c1a4610 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:26:46.523 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:46.524+0000 7fd850968640 1 --2- 172.21.15.114:0/3878333711 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fd84c106170 0x7fd84c199ad0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:26:46.523 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:46.524+0000 7fd850968640 1 --2- 172.21.15.114:0/3878333711 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fd84c1010a0 0x7fd84c199590 unknown :-1 s=CLOSED pgs=451 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:26:46.523 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:46.524+0000 7fd850968640 1 -- 172.21.15.114:0/3878333711 >> 172.21.15.114:0/3878333711 conn(0x7fd84c077f50 msgr2=0x7fd84c0ff090 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T10:26:46.523 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:46.524+0000 7fd850968640 1 -- 172.21.15.114:0/3878333711 shutdown_connections 2024-04-23T10:26:46.523 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:46.524+0000 7fd850968640 1 -- 172.21.15.114:0/3878333711 wait complete. 2024-04-23T10:26:46.534 DEBUG:teuthology.orchestra.run.smithi114:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-04-23T10:26:46.635 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:46.633+0000 7f68b19a2640 1 -- 172.21.15.114:0/2794560598 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f68ac106170 msgr2=0x7f68ac10dd40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T10:26:46.635 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:46.633+0000 7f68b19a2640 1 --2- 172.21.15.114:0/2794560598 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f68ac106170 0x7f68ac10dd40 secure :-1 s=READY pgs=452 cs=0 l=1 rev1=1 crypto rx=0x7f689c009900 tx=0x7f689c02a410 comp rx=0 tx=0).stop 2024-04-23T10:26:46.636 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:46.634+0000 7f68b19a2640 1 -- 172.21.15.114:0/2794560598 shutdown_connections 2024-04-23T10:26:46.636 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:46.634+0000 7f68b19a2640 1 --2- 172.21.15.114:0/2794560598 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f68ac106170 0x7f68ac10dd40 unknown :-1 s=CLOSED pgs=452 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:26:46.636 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:46.634+0000 7f68b19a2640 1 --2- 172.21.15.114:0/2794560598 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f68ac101a50 0x7f68ac105a90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:26:46.636 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:46.634+0000 7f68b19a2640 1 --2- 172.21.15.114:0/2794560598 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f68ac1010a0 0x7f68ac101480 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:26:46.636 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:46.634+0000 7f68b19a2640 1 -- 172.21.15.114:0/2794560598 >> 172.21.15.114:0/2794560598 conn(0x7f68ac077f50 msgr2=0x7f68ac078360 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T10:26:46.636 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:46.634+0000 7f68b19a2640 1 -- 172.21.15.114:0/2794560598 shutdown_connections 2024-04-23T10:26:46.636 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:46.634+0000 7f68b19a2640 1 -- 172.21.15.114:0/2794560598 wait complete. 2024-04-23T10:26:46.636 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:46.634+0000 7f68b19a2640 1 Processor -- start 2024-04-23T10:26:46.636 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:46.634+0000 7f68b19a2640 1 -- start start 2024-04-23T10:26:46.636 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:46.635+0000 7f68b19a2640 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f68ac1010a0 0x7f68ac1986f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T10:26:46.636 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:46.635+0000 7f68b19a2640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f68ac106170 0x7f68ac198c30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T10:26:46.636 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:46.635+0000 7f68b19a2640 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f68ac19cfb0 0x7f68ac19d460 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T10:26:46.636 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:46.635+0000 7f68b19a2640 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x7f68ac1121f0 con 0x7f68ac106170 2024-04-23T10:26:46.637 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:46.635+0000 7f68b19a2640 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 -- 0x7f68ac112070 con 0x7f68ac1010a0 2024-04-23T10:26:46.637 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:46.635+0000 7f68b19a2640 1 -- --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_getmap magic: 0 -- 0x7f68ac112370 con 0x7f68ac19cfb0 2024-04-23T10:26:46.637 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:46.635+0000 7f68aa7fc640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f68ac106170 0x7f68ac198c30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T10:26:46.637 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:46.635+0000 7f68aa7fc640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f68ac106170 0x7f68ac198c30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.114:51926/0 (socket says 172.21.15.114:51926) 2024-04-23T10:26:46.637 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:46.635+0000 7f68aa7fc640 1 -- 172.21.15.114:0/526715715 learned_addr learned my addr 172.21.15.114:0/526715715 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-04-23T10:26:46.637 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:46.635+0000 7f68aaffd640 1 --2- 172.21.15.114:0/526715715 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f68ac1010a0 0x7f68ac1986f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T10:26:46.637 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:46.635+0000 7f68ab7fe640 1 --2- 172.21.15.114:0/526715715 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f68ac19cfb0 0x7f68ac19d460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T10:26:46.637 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:46.635+0000 7f68aa7fc640 1 -- 172.21.15.114:0/526715715 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f68ac19cfb0 msgr2=0x7f68ac19d460 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T10:26:46.637 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:46.635+0000 7f68aa7fc640 1 --2- 172.21.15.114:0/526715715 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f68ac19cfb0 0x7f68ac19d460 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:26:46.637 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:46.635+0000 7f68aa7fc640 1 -- 172.21.15.114:0/526715715 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f68ac1010a0 msgr2=0x7f68ac1986f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T10:26:46.637 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:46.635+0000 7f68aa7fc640 1 --2- 172.21.15.114:0/526715715 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f68ac1010a0 0x7f68ac1986f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:26:46.637 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:46.635+0000 7f68aa7fc640 1 -- 172.21.15.114:0/526715715 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f68ac19daa0 con 0x7f68ac106170 2024-04-23T10:26:46.638 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:46.635+0000 7f68aaffd640 1 --2- 172.21.15.114:0/526715715 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f68ac1010a0 0x7f68ac1986f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-23T10:26:46.638 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:46.635+0000 7f68aa7fc640 1 --2- 172.21.15.114:0/526715715 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f68ac106170 0x7f68ac198c30 secure :-1 s=READY pgs=453 cs=0 l=1 rev1=1 crypto rx=0x7f689400d770 tx=0x7f689400dc30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T10:26:46.638 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:46.635+0000 7f68b09a0640 1 -- 172.21.15.114:0/526715715 <== mon.0 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6894017070 con 0x7f68ac106170 2024-04-23T10:26:46.638 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:46.635+0000 7f68b09a0640 1 -- 172.21.15.114:0/526715715 <== mon.0 v2:172.21.15.114:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f6894009c30 con 0x7f68ac106170 2024-04-23T10:26:46.638 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:46.635+0000 7f68b19a2640 1 -- 172.21.15.114:0/526715715 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f68ac19dd90 con 0x7f68ac106170 2024-04-23T10:26:46.638 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:46.635+0000 7f68b09a0640 1 -- 172.21.15.114:0/526715715 <== mon.0 v2:172.21.15.114:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f689400f620 con 0x7f68ac106170 2024-04-23T10:26:46.638 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:46.635+0000 7f68b19a2640 1 -- 172.21.15.114:0/526715715 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f68ac1a55d0 con 0x7f68ac106170 2024-04-23T10:26:46.638 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:46.636+0000 7f68b19a2640 1 -- 172.21.15.114:0/526715715 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f68ac068d70 con 0x7f68ac106170 2024-04-23T10:26:46.639 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:46.637+0000 7f68b09a0640 1 -- 172.21.15.114:0/526715715 <== mon.0 v2:172.21.15.114:3300/0 4 ==== mgrmap(e 24) ==== 138823+0+0 (secure 0 0 0) 0x7f689401a070 con 0x7f68ac106170 2024-04-23T10:26:46.639 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:46.638+0000 7f68b09a0640 1 --2- 172.21.15.114:0/526715715 >> [v2:172.21.15.114:6800/1395693466,v1:172.21.15.114:6801/1395693466] conn(0x7f68980a3e30 0x7f68980a62f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T10:26:46.639 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:46.638+0000 7f68aaffd640 1 --2- 172.21.15.114:0/526715715 >> [v2:172.21.15.114:6800/1395693466,v1:172.21.15.114:6801/1395693466] conn(0x7f68980a3e30 0x7f68980a62f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T10:26:46.639 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:46.638+0000 7f68b09a0640 1 -- 172.21.15.114:0/526715715 <== mon.0 v2:172.21.15.114:3300/0 5 ==== osd_map(90..90 src has 1..90) ==== 8527+0+0 (secure 0 0 0) 0x7f68940d0450 con 0x7f68ac106170 2024-04-23T10:26:46.639 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:46.638+0000 7f68aaffd640 1 --2- 172.21.15.114:0/526715715 >> [v2:172.21.15.114:6800/1395693466,v1:172.21.15.114:6801/1395693466] conn(0x7f68980a3e30 0x7f68980a62f0 secure :-1 s=READY pgs=52 cs=0 l=1 rev1=1 crypto rx=0x7f68a4005fd0 tx=0x7f68a4009500 comp rx=0 tx=0).ready entity=mgr.15291 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T10:26:46.642 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:46.643+0000 7f68b09a0640 1 -- 172.21.15.114:0/526715715 <== mon.0 v2:172.21.15.114:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f6894009de0 con 0x7f68ac106170 2024-04-23T10:26:46.819 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:46.820+0000 7f68b19a2640 1 -- 172.21.15.114:0/526715715 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f68ac101a50 con 0x7f68ac106170 2024-04-23T10:26:46.820 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:46.821+0000 7f68b09a0640 1 -- 172.21.15.114:0/526715715 <== mon.0 v2:172.21.15.114:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v90) ==== 74+0+23198 (secure 0 0 0) 0x7f689409c980 con 0x7f68ac106170 2024-04-23T10:26:46.821 INFO:teuthology.orchestra.run.smithi114.stdout: 2024-04-23T10:26:46.821 INFO:teuthology.orchestra.run.smithi114.stdout:{"epoch":90,"fsid":"aa240572-0159-11ef-bc93-c7b262605968","created":"2024-04-23T10:16:53.351136+0000","modified":"2024-04-23T10:26:40.272696+0000","last_up_change":"2024-04-23T10:23:54.153269+0000","last_in_change":"2024-04-23T10:23:32.963085+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":26,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-04-23T10:20:11.970297+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":12,"score_stable":12,"optimal_score":0.25,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-04-23T10:24:41.218540+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"86","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.8799999952316284,"score_stable":1.8799999952316284,"optimal_score":1,"raw_score_acting":1.8799999952316284,"raw_score_stable":1.8799999952316284,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-04-23T10:24:42.341799+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"87","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"fd64adde-9445-45a4-a171-ef656e2c5709","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":88,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6802","nonce":736366878},{"type":"v1","addr":"172.21.15.114:6803","nonce":736366878}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6804","nonce":736366878},{"type":"v1","addr":"172.21.15.114:6805","nonce":736366878}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6808","nonce":736366878},{"type":"v1","addr":"172.21.15.114:6809","nonce":736366878}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6806","nonce":736366878},{"type":"v1","addr":"172.21.15.114:6807","nonce":736366878}]},"public_addr":"172.21.15.114:6803/736366878","cluster_addr":"172.21.15.114:6805/736366878","heartbeat_back_addr":"172.21.15.114:6809/736366878","heartbeat_front_addr":"172.21.15.114:6807/736366878","state":["exists","up"]},{"osd":1,"uuid":"e279a1e6-2702-472a-930a-e08ece76a861","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":85,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6800","nonce":1133540779},{"type":"v1","addr":"172.21.15.119:6801","nonce":1133540779}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6802","nonce":1133540779},{"type":"v1","addr":"172.21.15.119:6803","nonce":1133540779}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6806","nonce":1133540779},{"type":"v1","addr":"172.21.15.119:6807","nonce":1133540779}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6804","nonce":1133540779},{"type":"v1","addr":"172.21.15.119:6805","nonce":1133540779}]},"public_addr":"172.21.15.119:6801/1133540779","cluster_addr":"172.21.15.119:6803/1133540779","heartbeat_back_addr":"172.21.15.119:6807/1133540779","heartbeat_front_addr":"172.21.15.119:6805/1133540779","state":["exists","up"]},{"osd":2,"uuid":"f4c9f2b4-ff49-4e14-89cd-b477feeb9382","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":85,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6800","nonce":799652511},{"type":"v1","addr":"172.21.15.204:6801","nonce":799652511}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6802","nonce":799652511},{"type":"v1","addr":"172.21.15.204:6803","nonce":799652511}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6806","nonce":799652511},{"type":"v1","addr":"172.21.15.204:6807","nonce":799652511}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6804","nonce":799652511},{"type":"v1","addr":"172.21.15.204:6805","nonce":799652511}]},"public_addr":"172.21.15.204:6801/799652511","cluster_addr":"172.21.15.204:6803/799652511","heartbeat_back_addr":"172.21.15.204:6807/799652511","heartbeat_front_addr":"172.21.15.204:6805/799652511","state":["exists","up"]},{"osd":3,"uuid":"52d2a566-e71c-4bb3-8d6c-5abdd43177c2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":25,"up_thru":88,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6810","nonce":4275595091},{"type":"v1","addr":"172.21.15.114:6811","nonce":4275595091}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6812","nonce":4275595091},{"type":"v1","addr":"172.21.15.114:6813","nonce":4275595091}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6816","nonce":4275595091},{"type":"v1","addr":"172.21.15.114:6817","nonce":4275595091}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6814","nonce":4275595091},{"type":"v1","addr":"172.21.15.114:6815","nonce":4275595091}]},"public_addr":"172.21.15.114:6811/4275595091","cluster_addr":"172.21.15.114:6813/4275595091","heartbeat_back_addr":"172.21.15.114:6817/4275595091","heartbeat_front_addr":"172.21.15.114:6815/4275595091","state":["exists","up"]},{"osd":4,"uuid":"d911a7c1-4bdc-4d28-b8a6-73896a6e7de6","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":33,"up_thru":85,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6808","nonce":1611007172},{"type":"v1","addr":"172.21.15.119:6809","nonce":1611007172}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6810","nonce":1611007172},{"type":"v1","addr":"172.21.15.119:6811","nonce":1611007172}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6814","nonce":1611007172},{"type":"v1","addr":"172.21.15.119:6815","nonce":1611007172}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6812","nonce":1611007172},{"type":"v1","addr":"172.21.15.119:6813","nonce":1611007172}]},"public_addr":"172.21.15.119:6809/1611007172","cluster_addr":"172.21.15.119:6811/1611007172","heartbeat_back_addr":"172.21.15.119:6815/1611007172","heartbeat_front_addr":"172.21.15.119:6813/1611007172","state":["exists","up"]},{"osd":5,"uuid":"f7590e1a-e93c-4c29-b15f-38ee52d1f921","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":39,"up_thru":85,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6808","nonce":1427563312},{"type":"v1","addr":"172.21.15.204:6809","nonce":1427563312}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6810","nonce":1427563312},{"type":"v1","addr":"172.21.15.204:6811","nonce":1427563312}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6814","nonce":1427563312},{"type":"v1","addr":"172.21.15.204:6815","nonce":1427563312}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6812","nonce":1427563312},{"type":"v1","addr":"172.21.15.204:6813","nonce":1427563312}]},"public_addr":"172.21.15.204:6809/1427563312","cluster_addr":"172.21.15.204:6811/1427563312","heartbeat_back_addr":"172.21.15.204:6815/1427563312","heartbeat_front_addr":"172.21.15.204:6813/1427563312","state":["exists","up"]},{"osd":6,"uuid":"534d20e8-3e54-4db8-a845-6119f49c7a78","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":47,"up_thru":88,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6818","nonce":4072912544},{"type":"v1","addr":"172.21.15.114:6819","nonce":4072912544}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6820","nonce":4072912544},{"type":"v1","addr":"172.21.15.114:6821","nonce":4072912544}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6824","nonce":4072912544},{"type":"v1","addr":"172.21.15.114:6825","nonce":4072912544}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6822","nonce":4072912544},{"type":"v1","addr":"172.21.15.114:6823","nonce":4072912544}]},"public_addr":"172.21.15.114:6819/4072912544","cluster_addr":"172.21.15.114:6821/4072912544","heartbeat_back_addr":"172.21.15.114:6825/4072912544","heartbeat_front_addr":"172.21.15.114:6823/4072912544","state":["exists","up"]},{"osd":7,"uuid":"f0ec22b8-1b48-4859-9d65-b00403aff3ad","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":53,"up_thru":88,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6816","nonce":2237458793},{"type":"v1","addr":"172.21.15.119:6817","nonce":2237458793}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6818","nonce":2237458793},{"type":"v1","addr":"172.21.15.119:6819","nonce":2237458793}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6822","nonce":2237458793},{"type":"v1","addr":"172.21.15.119:6823","nonce":2237458793}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6820","nonce":2237458793},{"type":"v1","addr":"172.21.15.119:6821","nonce":2237458793}]},"public_addr":"172.21.15.119:6817/2237458793","cluster_addr":"172.21.15.119:6819/2237458793","heartbeat_back_addr":"172.21.15.119:6823/2237458793","heartbeat_front_addr":"172.21.15.119:6821/2237458793","state":["exists","up"]},{"osd":8,"uuid":"3ac89686-6d76-4cef-9089-ac29e4b3bcac","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":62,"up_thru":85,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6816","nonce":1426318231},{"type":"v1","addr":"172.21.15.204:6817","nonce":1426318231}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6818","nonce":1426318231},{"type":"v1","addr":"172.21.15.204:6819","nonce":1426318231}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6822","nonce":1426318231},{"type":"v1","addr":"172.21.15.204:6823","nonce":1426318231}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6820","nonce":1426318231},{"type":"v1","addr":"172.21.15.204:6821","nonce":1426318231}]},"public_addr":"172.21.15.204:6817/1426318231","cluster_addr":"172.21.15.204:6819/1426318231","heartbeat_back_addr":"172.21.15.204:6823/1426318231","heartbeat_front_addr":"172.21.15.204:6821/1426318231","state":["exists","up"]},{"osd":9,"uuid":"a80ffe4d-1e98-460f-b046-c4754285cc5a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":69,"up_thru":88,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6826","nonce":2750635721},{"type":"v1","addr":"172.21.15.114:6827","nonce":2750635721}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6828","nonce":2750635721},{"type":"v1","addr":"172.21.15.114:6829","nonce":2750635721}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6832","nonce":2750635721},{"type":"v1","addr":"172.21.15.114:6833","nonce":2750635721}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6830","nonce":2750635721},{"type":"v1","addr":"172.21.15.114:6831","nonce":2750635721}]},"public_addr":"172.21.15.114:6827/2750635721","cluster_addr":"172.21.15.114:6829/2750635721","heartbeat_back_addr":"172.21.15.114:6833/2750635721","heartbeat_front_addr":"172.21.15.114:6831/2750635721","state":["exists","up"]},{"osd":10,"uuid":"cd75a579-bf12-4600-ac09-a8bfce772ced","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":74,"up_thru":85,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6824","nonce":1400187701},{"type":"v1","addr":"172.21.15.119:6825","nonce":1400187701}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6826","nonce":1400187701},{"type":"v1","addr":"172.21.15.119:6827","nonce":1400187701}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6830","nonce":1400187701},{"type":"v1","addr":"172.21.15.119:6831","nonce":1400187701}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6828","nonce":1400187701},{"type":"v1","addr":"172.21.15.119:6829","nonce":1400187701}]},"public_addr":"172.21.15.119:6825/1400187701","cluster_addr":"172.21.15.119:6827/1400187701","heartbeat_back_addr":"172.21.15.119:6831/1400187701","heartbeat_front_addr":"172.21.15.119:6829/1400187701","state":["exists","up"]},{"osd":11,"uuid":"53f037a5-3ec0-483a-80a1-dfee923a68e5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":82,"up_thru":88,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6824","nonce":2407298490},{"type":"v1","addr":"172.21.15.204:6825","nonce":2407298490}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6826","nonce":2407298490},{"type":"v1","addr":"172.21.15.204:6827","nonce":2407298490}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6830","nonce":2407298490},{"type":"v1","addr":"172.21.15.204:6831","nonce":2407298490}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6828","nonce":2407298490},{"type":"v1","addr":"172.21.15.204:6829","nonce":2407298490}]},"public_addr":"172.21.15.204:6825/2407298490","cluster_addr":"172.21.15.204:6827/2407298490","heartbeat_back_addr":"172.21.15.204:6831/2407298490","heartbeat_front_addr":"172.21.15.204:6829/2407298490","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-23T10:19:18.401147+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-23T10:19:42.611299+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-23T10:20:07.277367+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-23T10:20:31.208368+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-23T10:20:55.866225+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-23T10:21:19.961237+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-23T10:21:44.480783+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-23T10:22:09.173641+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-23T10:22:34.030328+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-23T10:22:59.652293+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-23T10:23:25.659673+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-23T10:23:51.168171+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"2.e","mappings":[{"from":2,"to":4}]},{"pgid":"2.1d","mappings":[{"from":8,"to":11}]},{"pgid":"3.e","mappings":[{"from":5,"to":9}]},{"pgid":"3.12","mappings":[{"from":3,"to":9}]},{"pgid":"3.17","mappings":[{"from":4,"to":6}]},{"pgid":"3.23","mappings":[{"from":4,"to":7}]},{"pgid":"3.27","mappings":[{"from":4,"to":6}]},{"pgid":"3.31","mappings":[{"from":4,"to":8}]},{"pgid":"3.38","mappings":[{"from":3,"to":9}]},{"pgid":"3.39","mappings":[{"from":5,"to":8}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.114:0/480720790":"2024-04-24T10:26:40.272662+0000","172.21.15.114:6801/465333808":"2024-04-24T10:26:40.272662+0000","172.21.15.114:6800/465333808":"2024-04-24T10:26:40.272662+0000","172.21.15.114:0/1787801099":"2024-04-24T10:26:40.272662+0000","172.21.15.114:6801/1604678557":"2024-04-24T10:17:45.874609+0000","172.21.15.114:0/4025385854":"2024-04-24T10:17:45.874609+0000","172.21.15.114:0/2228273469":"2024-04-24T10:17:19.910348+0000","172.21.15.114:0/2713201622":"2024-04-24T10:26:40.272662+0000","172.21.15.114:6800/2831530838":"2024-04-24T10:17:19.910348+0000","172.21.15.114:0/637004095":"2024-04-24T10:26:40.272662+0000","172.21.15.114:0/1333447450":"2024-04-24T10:17:45.874609+0000","172.21.15.114:0/2233026840":"2024-04-24T10:17:19.910348+0000","172.21.15.114:0/1902281545":"2024-04-24T10:17:19.910348+0000","172.21.15.114:6801/2831530838":"2024-04-24T10:17:19.910348+0000","172.21.15.114:6800/1604678557":"2024-04-24T10:17:45.874609+0000","172.21.15.114:0/1623553811":"2024-04-24T10:17:45.874609+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-23T10:26:46.823 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:46.823+0000 7f68b19a2640 1 -- 172.21.15.114:0/526715715 >> [v2:172.21.15.114:6800/1395693466,v1:172.21.15.114:6801/1395693466] conn(0x7f68980a3e30 msgr2=0x7f68980a62f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T10:26:46.823 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:46.823+0000 7f68b19a2640 1 --2- 172.21.15.114:0/526715715 >> [v2:172.21.15.114:6800/1395693466,v1:172.21.15.114:6801/1395693466] conn(0x7f68980a3e30 0x7f68980a62f0 secure :-1 s=READY pgs=52 cs=0 l=1 rev1=1 crypto rx=0x7f68a4005fd0 tx=0x7f68a4009500 comp rx=0 tx=0).stop 2024-04-23T10:26:46.823 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:46.823+0000 7f68b19a2640 1 -- 172.21.15.114:0/526715715 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f68ac106170 msgr2=0x7f68ac198c30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T10:26:46.823 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:46.823+0000 7f68b19a2640 1 --2- 172.21.15.114:0/526715715 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f68ac106170 0x7f68ac198c30 secure :-1 s=READY pgs=453 cs=0 l=1 rev1=1 crypto rx=0x7f689400d770 tx=0x7f689400dc30 comp rx=0 tx=0).stop 2024-04-23T10:26:46.823 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:46.824+0000 7f68b19a2640 1 -- 172.21.15.114:0/526715715 shutdown_connections 2024-04-23T10:26:46.823 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:46.824+0000 7f68b19a2640 1 --2- 172.21.15.114:0/526715715 >> [v2:172.21.15.114:6800/1395693466,v1:172.21.15.114:6801/1395693466] conn(0x7f68980a3e30 0x7f68980a62f0 unknown :-1 s=CLOSED pgs=52 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:26:46.823 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:46.824+0000 7f68b19a2640 1 --2- 172.21.15.114:0/526715715 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f68ac19cfb0 0x7f68ac19d460 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:26:46.823 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:46.824+0000 7f68b19a2640 1 --2- 172.21.15.114:0/526715715 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f68ac106170 0x7f68ac198c30 unknown :-1 s=CLOSED pgs=453 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:26:46.823 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:46.824+0000 7f68b19a2640 1 --2- 172.21.15.114:0/526715715 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f68ac1010a0 0x7f68ac1986f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:26:46.823 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:46.824+0000 7f68b19a2640 1 -- 172.21.15.114:0/526715715 >> 172.21.15.114:0/526715715 conn(0x7f68ac077f50 msgr2=0x7f68ac0fef30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T10:26:46.823 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:46.824+0000 7f68b19a2640 1 -- 172.21.15.114:0/526715715 shutdown_connections 2024-04-23T10:26:46.823 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:46.824+0000 7f68b19a2640 1 -- 172.21.15.114:0/526715715 wait complete. 2024-04-23T10:26:46.834 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-04-23T10:20:11.970297+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '22', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 12, 'score_stable': 12, 'optimal_score': 0.25, 'raw_score_acting': 3, 'raw_score_stable': 3, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 2, 'pool_name': 'cephfs_metadata', 'create_time': '2024-04-23T10:24:41.218540+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '86', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.8799999952316284, 'score_stable': 1.8799999952316284, 'optimal_score': 1, 'raw_score_acting': 1.8799999952316284, 'raw_score_stable': 1.8799999952316284, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 3, 'pool_name': 'cephfs_data', 'create_time': '2024-04-23T10:24:42.341799+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '87', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-04-23T10:26:46.834 DEBUG:teuthology.orchestra.run.smithi114:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-04-23T10:26:46.936 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:46.936+0000 7f025d75d640 1 -- 172.21.15.114:0/2972417687 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f0258107260 msgr2=0x7f025810ee30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T10:26:46.936 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:46.936+0000 7f025d75d640 1 --2- 172.21.15.114:0/2972417687 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f0258107260 0x7f025810ee30 secure :-1 s=READY pgs=454 cs=0 l=1 rev1=1 crypto rx=0x7f0250009ca0 tx=0x7f025002a8c0 comp rx=0 tx=0).stop 2024-04-23T10:26:46.936 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:46.937+0000 7f025d75d640 1 -- 172.21.15.114:0/2972417687 shutdown_connections 2024-04-23T10:26:46.936 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:46.937+0000 7f025d75d640 1 --2- 172.21.15.114:0/2972417687 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f0258107260 0x7f025810ee30 unknown :-1 s=CLOSED pgs=454 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:26:46.936 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:46.937+0000 7f025d75d640 1 --2- 172.21.15.114:0/2972417687 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f0258102b40 0x7f0258106b80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:26:46.936 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:46.937+0000 7f025d75d640 1 --2- 172.21.15.114:0/2972417687 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f0258102190 0x7f0258102570 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:26:46.936 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:46.937+0000 7f025d75d640 1 -- 172.21.15.114:0/2972417687 >> 172.21.15.114:0/2972417687 conn(0x7f02580793a0 msgr2=0x7f02580797b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T10:26:46.936 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:46.937+0000 7f025d75d640 1 -- 172.21.15.114:0/2972417687 shutdown_connections 2024-04-23T10:26:46.936 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:46.937+0000 7f025d75d640 1 -- 172.21.15.114:0/2972417687 wait complete. 2024-04-23T10:26:46.936 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:46.937+0000 7f025d75d640 1 Processor -- start 2024-04-23T10:26:46.936 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:46.937+0000 7f025d75d640 1 -- start start 2024-04-23T10:26:46.938 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:46.937+0000 7f025d75d640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f0258102190 0x7f02581986f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T10:26:46.939 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:46.937+0000 7f025d75d640 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f0258107260 0x7f0258198c30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T10:26:46.939 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:46.937+0000 7f025d75d640 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f025819cf60 0x7f025819d410 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T10:26:46.939 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:46.937+0000 7f025d75d640 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x7f02581121b0 con 0x7f0258102190 2024-04-23T10:26:46.939 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:46.937+0000 7f025d75d640 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 -- 0x7f0258112030 con 0x7f025819cf60 2024-04-23T10:26:46.939 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:46.937+0000 7f025d75d640 1 -- --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_getmap magic: 0 -- 0x7f0258112330 con 0x7f0258107260 2024-04-23T10:26:46.939 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:46.938+0000 7f0256ffd640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f0258102190 0x7f02581986f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T10:26:46.939 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:46.938+0000 7f0256ffd640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f0258102190 0x7f02581986f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.114:51954/0 (socket says 172.21.15.114:51954) 2024-04-23T10:26:46.939 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:46.938+0000 7f0256ffd640 1 -- 172.21.15.114:0/3757603754 learned_addr learned my addr 172.21.15.114:0/3757603754 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-04-23T10:26:46.939 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:46.938+0000 7f02567fc640 1 --2- 172.21.15.114:0/3757603754 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f0258107260 0x7f0258198c30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T10:26:46.939 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:46.938+0000 7f02577fe640 1 --2- 172.21.15.114:0/3757603754 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f025819cf60 0x7f025819d410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T10:26:46.939 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:46.938+0000 7f0256ffd640 1 -- 172.21.15.114:0/3757603754 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f0258107260 msgr2=0x7f0258198c30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T10:26:46.939 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:46.938+0000 7f0256ffd640 1 --2- 172.21.15.114:0/3757603754 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f0258107260 0x7f0258198c30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:26:46.939 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:46.938+0000 7f0256ffd640 1 -- 172.21.15.114:0/3757603754 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f025819cf60 msgr2=0x7f025819d410 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T10:26:46.939 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:46.938+0000 7f0256ffd640 1 --2- 172.21.15.114:0/3757603754 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f025819cf60 0x7f025819d410 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:26:46.939 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:46.938+0000 7f0256ffd640 1 -- 172.21.15.114:0/3757603754 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f025819da50 con 0x7f0258102190 2024-04-23T10:26:46.940 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:46.938+0000 7f02567fc640 1 --2- 172.21.15.114:0/3757603754 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f0258107260 0x7f0258198c30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-23T10:26:46.940 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:46.938+0000 7f0256ffd640 1 --2- 172.21.15.114:0/3757603754 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f0258102190 0x7f02581986f0 secure :-1 s=READY pgs=455 cs=0 l=1 rev1=1 crypto rx=0x7f024800d950 tx=0x7f024800de10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T10:26:46.940 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:46.938+0000 7f023ffff640 1 -- 172.21.15.114:0/3757603754 <== mon.0 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0248017030 con 0x7f0258102190 2024-04-23T10:26:46.940 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:46.938+0000 7f023ffff640 1 -- 172.21.15.114:0/3757603754 <== mon.0 v2:172.21.15.114:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f0248004020 con 0x7f0258102190 2024-04-23T10:26:46.940 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:46.938+0000 7f025d75d640 1 -- 172.21.15.114:0/3757603754 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f025819dd40 con 0x7f0258102190 2024-04-23T10:26:46.940 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:46.938+0000 7f023ffff640 1 -- 172.21.15.114:0/3757603754 <== mon.0 v2:172.21.15.114:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f02480047a0 con 0x7f0258102190 2024-04-23T10:26:46.940 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:46.938+0000 7f025d75d640 1 -- 172.21.15.114:0/3757603754 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f02581a5530 con 0x7f0258102190 2024-04-23T10:26:46.940 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:46.939+0000 7f025d75d640 1 -- 172.21.15.114:0/3757603754 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f02580691c0 con 0x7f0258102190 2024-04-23T10:26:46.941 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:46.940+0000 7f023ffff640 1 -- 172.21.15.114:0/3757603754 <== mon.0 v2:172.21.15.114:3300/0 4 ==== mgrmap(e 24) ==== 138823+0+0 (secure 0 0 0) 0x7f0248009a30 con 0x7f0258102190 2024-04-23T10:26:46.941 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:46.941+0000 7f023ffff640 1 --2- 172.21.15.114:0/3757603754 >> [v2:172.21.15.114:6800/1395693466,v1:172.21.15.114:6801/1395693466] conn(0x7f02400a3db0 0x7f02400a6270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T10:26:46.941 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:46.941+0000 7f02567fc640 1 --2- 172.21.15.114:0/3757603754 >> [v2:172.21.15.114:6800/1395693466,v1:172.21.15.114:6801/1395693466] conn(0x7f02400a3db0 0x7f02400a6270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T10:26:46.941 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:46.941+0000 7f023ffff640 1 -- 172.21.15.114:0/3757603754 <== mon.0 v2:172.21.15.114:3300/0 5 ==== osd_map(90..90 src has 1..90) ==== 8527+0+0 (secure 0 0 0) 0x7f02480d04b0 con 0x7f0258102190 2024-04-23T10:26:46.941 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:46.942+0000 7f02567fc640 1 --2- 172.21.15.114:0/3757603754 >> [v2:172.21.15.114:6800/1395693466,v1:172.21.15.114:6801/1395693466] conn(0x7f02400a3db0 0x7f02400a6270 secure :-1 s=READY pgs=53 cs=0 l=1 rev1=1 crypto rx=0x7f024c005e10 tx=0x7f024c005da0 comp rx=0 tx=0).ready entity=mgr.15291 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T10:26:46.944 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:46.946+0000 7f023ffff640 1 -- 172.21.15.114:0/3757603754 <== mon.0 v2:172.21.15.114:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f024809c9e0 con 0x7f0258102190 2024-04-23T10:26:47.123 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:47.124+0000 7f025d75d640 1 -- 172.21.15.114:0/3757603754 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) -- 0x7f0258102b40 con 0x7f0258102190 2024-04-23T10:26:47.123 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:47.124+0000 7f023ffff640 1 -- 172.21.15.114:0/3757603754 <== mon.0 v2:172.21.15.114:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v90) ==== 93+0+10 (secure 0 0 0) 0x7f024809c2e0 con 0x7f0258102190 2024-04-23T10:26:47.123 INFO:teuthology.orchestra.run.smithi114.stdout:pg_num: 1 2024-04-23T10:26:47.125 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:47.126+0000 7f025d75d640 1 -- 172.21.15.114:0/3757603754 >> [v2:172.21.15.114:6800/1395693466,v1:172.21.15.114:6801/1395693466] conn(0x7f02400a3db0 msgr2=0x7f02400a6270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T10:26:47.125 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:47.126+0000 7f025d75d640 1 --2- 172.21.15.114:0/3757603754 >> [v2:172.21.15.114:6800/1395693466,v1:172.21.15.114:6801/1395693466] conn(0x7f02400a3db0 0x7f02400a6270 secure :-1 s=READY pgs=53 cs=0 l=1 rev1=1 crypto rx=0x7f024c005e10 tx=0x7f024c005da0 comp rx=0 tx=0).stop 2024-04-23T10:26:47.125 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:47.126+0000 7f025d75d640 1 -- 172.21.15.114:0/3757603754 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f0258102190 msgr2=0x7f02581986f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T10:26:47.126 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:47.126+0000 7f025d75d640 1 --2- 172.21.15.114:0/3757603754 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f0258102190 0x7f02581986f0 secure :-1 s=READY pgs=455 cs=0 l=1 rev1=1 crypto rx=0x7f024800d950 tx=0x7f024800de10 comp rx=0 tx=0).stop 2024-04-23T10:26:47.126 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:47.126+0000 7f025d75d640 1 -- 172.21.15.114:0/3757603754 shutdown_connections 2024-04-23T10:26:47.126 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:47.126+0000 7f025d75d640 1 --2- 172.21.15.114:0/3757603754 >> [v2:172.21.15.114:6800/1395693466,v1:172.21.15.114:6801/1395693466] conn(0x7f02400a3db0 0x7f02400a6270 unknown :-1 s=CLOSED pgs=53 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:26:47.126 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:47.126+0000 7f025d75d640 1 --2- 172.21.15.114:0/3757603754 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f025819cf60 0x7f025819d410 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:26:47.126 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:47.126+0000 7f025d75d640 1 --2- 172.21.15.114:0/3757603754 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f0258107260 0x7f0258198c30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:26:47.126 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:47.126+0000 7f025d75d640 1 --2- 172.21.15.114:0/3757603754 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f0258102190 0x7f02581986f0 unknown :-1 s=CLOSED pgs=455 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:26:47.126 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:47.126+0000 7f025d75d640 1 -- 172.21.15.114:0/3757603754 >> 172.21.15.114:0/3757603754 conn(0x7f02580793a0 msgr2=0x7f02580fff00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T10:26:47.126 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:47.126+0000 7f025d75d640 1 -- 172.21.15.114:0/3757603754 shutdown_connections 2024-04-23T10:26:47.126 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:47.126+0000 7f025d75d640 1 -- 172.21.15.114:0/3757603754 wait complete. 2024-04-23T10:26:47.135 INFO:journalctl@ceph.mon.a.smithi114.stdout:Apr 23 10:26:47 smithi114 ceph-mon[29255]: from='client.15438 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-04-23T10:26:47.135 INFO:journalctl@ceph.mon.a.smithi114.stdout:Apr 23 10:26:47 smithi114 ceph-mon[29255]: from='client.15444 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-04-23T10:26:47.135 INFO:journalctl@ceph.mon.a.smithi114.stdout:Apr 23 10:26:47 smithi114 ceph-mon[29255]: from='client.15450 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-04-23T10:26:47.135 INFO:journalctl@ceph.mon.a.smithi114.stdout:Apr 23 10:26:47 smithi114 ceph-mon[29255]: from='client.? 172.21.15.114:0/3878333711' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-23T10:26:47.135 INFO:journalctl@ceph.mon.a.smithi114.stdout:Apr 23 10:26:47 smithi114 ceph-mon[29255]: from='client.? 172.21.15.114:0/526715715' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-23T10:26:47.136 DEBUG:teuthology.orchestra.run.smithi114:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2024-04-23T10:26:47.237 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:47.238+0000 7f5d54a2c640 1 -- 172.21.15.114:0/2653780073 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f5d50106180 msgr2=0x7f5d5010dd50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T10:26:47.237 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:47.238+0000 7f5d54a2c640 1 --2- 172.21.15.114:0/2653780073 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f5d50106180 0x7f5d5010dd50 secure :-1 s=READY pgs=456 cs=0 l=1 rev1=1 crypto rx=0x7f5d48009ca0 tx=0x7f5d4802a8c0 comp rx=0 tx=0).stop 2024-04-23T10:26:47.237 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:47.238+0000 7f5d54a2c640 1 -- 172.21.15.114:0/2653780073 shutdown_connections 2024-04-23T10:26:47.237 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:47.238+0000 7f5d54a2c640 1 --2- 172.21.15.114:0/2653780073 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f5d50106180 0x7f5d5010dd50 unknown :-1 s=CLOSED pgs=456 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:26:47.237 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:47.238+0000 7f5d54a2c640 1 --2- 172.21.15.114:0/2653780073 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f5d50101a60 0x7f5d50105aa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:26:47.237 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:47.238+0000 7f5d54a2c640 1 --2- 172.21.15.114:0/2653780073 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f5d501010b0 0x7f5d50101490 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:26:47.237 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:47.238+0000 7f5d54a2c640 1 -- 172.21.15.114:0/2653780073 >> 172.21.15.114:0/2653780073 conn(0x7f5d500771d0 msgr2=0x7f5d500775e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T10:26:47.238 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:47.238+0000 7f5d54a2c640 1 -- 172.21.15.114:0/2653780073 shutdown_connections 2024-04-23T10:26:47.238 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:47.238+0000 7f5d54a2c640 1 -- 172.21.15.114:0/2653780073 wait complete. 2024-04-23T10:26:47.240 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:47.239+0000 7f5d54a2c640 1 Processor -- start 2024-04-23T10:26:47.240 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:47.239+0000 7f5d54a2c640 1 -- start start 2024-04-23T10:26:47.240 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:47.239+0000 7f5d54a2c640 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f5d501010b0 0x7f5d501986e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T10:26:47.240 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:47.239+0000 7f5d54a2c640 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f5d50106180 0x7f5d50198c20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T10:26:47.240 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:47.239+0000 7f5d54a2c640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f5d5019cf50 0x7f5d5019d400 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T10:26:47.240 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:47.239+0000 7f5d54a2c640 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x7f5d50112200 con 0x7f5d5019cf50 2024-04-23T10:26:47.240 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:47.239+0000 7f5d54a2c640 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 -- 0x7f5d50112080 con 0x7f5d50106180 2024-04-23T10:26:47.240 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:47.239+0000 7f5d54a2c640 1 -- --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_getmap magic: 0 -- 0x7f5d50112380 con 0x7f5d501010b0 2024-04-23T10:26:47.240 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:47.239+0000 7f5d4ed9d640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f5d5019cf50 0x7f5d5019d400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T10:26:47.240 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:47.239+0000 7f5d4ed9d640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f5d5019cf50 0x7f5d5019d400 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.114:51978/0 (socket says 172.21.15.114:51978) 2024-04-23T10:26:47.240 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:47.239+0000 7f5d4ed9d640 1 -- 172.21.15.114:0/1714696800 learned_addr learned my addr 172.21.15.114:0/1714696800 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-04-23T10:26:47.240 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:47.239+0000 7f5d47fff640 1 --2- 172.21.15.114:0/1714696800 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f5d50106180 0x7f5d50198c20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T10:26:47.241 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:47.239+0000 7f5d4e59c640 1 --2- 172.21.15.114:0/1714696800 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f5d501010b0 0x7f5d501986e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T10:26:47.241 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:47.239+0000 7f5d4ed9d640 1 -- 172.21.15.114:0/1714696800 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f5d501010b0 msgr2=0x7f5d501986e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T10:26:47.241 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:47.239+0000 7f5d4ed9d640 1 --2- 172.21.15.114:0/1714696800 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f5d501010b0 0x7f5d501986e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:26:47.241 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:47.239+0000 7f5d4ed9d640 1 -- 172.21.15.114:0/1714696800 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f5d50106180 msgr2=0x7f5d50198c20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T10:26:47.241 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:47.239+0000 7f5d4ed9d640 1 --2- 172.21.15.114:0/1714696800 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f5d50106180 0x7f5d50198c20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:26:47.241 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:47.239+0000 7f5d4ed9d640 1 -- 172.21.15.114:0/1714696800 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f5d5019da40 con 0x7f5d5019cf50 2024-04-23T10:26:47.241 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:47.239+0000 7f5d4ed9d640 1 --2- 172.21.15.114:0/1714696800 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f5d5019cf50 0x7f5d5019d400 secure :-1 s=READY pgs=457 cs=0 l=1 rev1=1 crypto rx=0x7f5d48005cc0 tx=0x7f5d4802f380 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T10:26:47.241 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:47.240+0000 7f5d477fe640 1 -- 172.21.15.114:0/1714696800 <== mon.0 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5d48003bf0 con 0x7f5d5019cf50 2024-04-23T10:26:47.241 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:47.240+0000 7f5d477fe640 1 -- 172.21.15.114:0/1714696800 <== mon.0 v2:172.21.15.114:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f5d48003d90 con 0x7f5d5019cf50 2024-04-23T10:26:47.241 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:47.240+0000 7f5d477fe640 1 -- 172.21.15.114:0/1714696800 <== mon.0 v2:172.21.15.114:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5d4802fbc0 con 0x7f5d5019cf50 2024-04-23T10:26:47.241 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:47.240+0000 7f5d54a2c640 1 -- 172.21.15.114:0/1714696800 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f5d5019dcd0 con 0x7f5d5019cf50 2024-04-23T10:26:47.241 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:47.240+0000 7f5d54a2c640 1 -- 172.21.15.114:0/1714696800 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f5d501a54f0 con 0x7f5d5019cf50 2024-04-23T10:26:47.241 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:47.241+0000 7f5d54a2c640 1 -- 172.21.15.114:0/1714696800 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f5d500691c0 con 0x7f5d5019cf50 2024-04-23T10:26:47.242 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:47.242+0000 7f5d477fe640 1 -- 172.21.15.114:0/1714696800 <== mon.0 v2:172.21.15.114:3300/0 4 ==== mgrmap(e 24) ==== 138823+0+0 (secure 0 0 0) 0x7f5d4802fd60 con 0x7f5d5019cf50 2024-04-23T10:26:47.242 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:47.242+0000 7f5d477fe640 1 --2- 172.21.15.114:0/1714696800 >> [v2:172.21.15.114:6800/1395693466,v1:172.21.15.114:6801/1395693466] conn(0x7f5d3c0a3db0 0x7f5d3c0a6270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T10:26:47.242 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:47.243+0000 7f5d477fe640 1 -- 172.21.15.114:0/1714696800 <== mon.0 v2:172.21.15.114:3300/0 5 ==== osd_map(90..90 src has 1..90) ==== 8527+0+0 (secure 0 0 0) 0x7f5d480f1f00 con 0x7f5d5019cf50 2024-04-23T10:26:47.242 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:47.243+0000 7f5d4e59c640 1 --2- 172.21.15.114:0/1714696800 >> [v2:172.21.15.114:6800/1395693466,v1:172.21.15.114:6801/1395693466] conn(0x7f5d3c0a3db0 0x7f5d3c0a6270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T10:26:47.242 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:47.243+0000 7f5d4e59c640 1 --2- 172.21.15.114:0/1714696800 >> [v2:172.21.15.114:6800/1395693466,v1:172.21.15.114:6801/1395693466] conn(0x7f5d3c0a3db0 0x7f5d3c0a6270 secure :-1 s=READY pgs=54 cs=0 l=1 rev1=1 crypto rx=0x7f5d400058c0 tx=0x7f5d4000a2d0 comp rx=0 tx=0).ready entity=mgr.15291 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T10:26:47.246 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:47.247+0000 7f5d477fe640 1 -- 172.21.15.114:0/1714696800 <== mon.0 v2:172.21.15.114:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f5d48004610 con 0x7f5d5019cf50 2024-04-23T10:26:47.328 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 23 10:26:47 smithi204 ceph-mon[32649]: from='client.15438 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-04-23T10:26:47.328 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 23 10:26:47 smithi204 ceph-mon[32649]: from='client.15444 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-04-23T10:26:47.328 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 23 10:26:47 smithi204 ceph-mon[32649]: from='client.15450 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-04-23T10:26:47.328 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 23 10:26:47 smithi204 ceph-mon[32649]: from='client.? 172.21.15.114:0/3878333711' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-23T10:26:47.328 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 23 10:26:47 smithi204 ceph-mon[32649]: from='client.? 172.21.15.114:0/526715715' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-23T10:26:47.423 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:47.425+0000 7f5d54a2c640 1 -- 172.21.15.114:0/1714696800 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) -- 0x7f5d50109470 con 0x7f5d5019cf50 2024-04-23T10:26:47.424 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:47.425+0000 7f5d477fe640 1 -- 172.21.15.114:0/1714696800 <== mon.0 v2:172.21.15.114:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v90) ==== 104+0+11 (secure 0 0 0) 0x7f5d480098b0 con 0x7f5d5019cf50 2024-04-23T10:26:47.424 INFO:teuthology.orchestra.run.smithi114.stdout:pg_num: 32 2024-04-23T10:26:47.426 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:47.427+0000 7f5d54a2c640 1 -- 172.21.15.114:0/1714696800 >> [v2:172.21.15.114:6800/1395693466,v1:172.21.15.114:6801/1395693466] conn(0x7f5d3c0a3db0 msgr2=0x7f5d3c0a6270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T10:26:47.426 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:47.427+0000 7f5d54a2c640 1 --2- 172.21.15.114:0/1714696800 >> [v2:172.21.15.114:6800/1395693466,v1:172.21.15.114:6801/1395693466] conn(0x7f5d3c0a3db0 0x7f5d3c0a6270 secure :-1 s=READY pgs=54 cs=0 l=1 rev1=1 crypto rx=0x7f5d400058c0 tx=0x7f5d4000a2d0 comp rx=0 tx=0).stop 2024-04-23T10:26:47.427 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:47.427+0000 7f5d54a2c640 1 -- 172.21.15.114:0/1714696800 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f5d5019cf50 msgr2=0x7f5d5019d400 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T10:26:47.427 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:47.427+0000 7f5d54a2c640 1 --2- 172.21.15.114:0/1714696800 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f5d5019cf50 0x7f5d5019d400 secure :-1 s=READY pgs=457 cs=0 l=1 rev1=1 crypto rx=0x7f5d48005cc0 tx=0x7f5d4802f380 comp rx=0 tx=0).stop 2024-04-23T10:26:47.427 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:47.427+0000 7f5d54a2c640 1 -- 172.21.15.114:0/1714696800 shutdown_connections 2024-04-23T10:26:47.427 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:47.427+0000 7f5d54a2c640 1 --2- 172.21.15.114:0/1714696800 >> [v2:172.21.15.114:6800/1395693466,v1:172.21.15.114:6801/1395693466] conn(0x7f5d3c0a3db0 0x7f5d3c0a6270 unknown :-1 s=CLOSED pgs=54 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:26:47.427 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:47.427+0000 7f5d54a2c640 1 --2- 172.21.15.114:0/1714696800 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f5d5019cf50 0x7f5d5019d400 unknown :-1 s=CLOSED pgs=457 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:26:47.427 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:47.427+0000 7f5d54a2c640 1 --2- 172.21.15.114:0/1714696800 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f5d50106180 0x7f5d50198c20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:26:47.427 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:47.427+0000 7f5d54a2c640 1 --2- 172.21.15.114:0/1714696800 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f5d501010b0 0x7f5d501986e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:26:47.427 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:47.427+0000 7f5d54a2c640 1 -- 172.21.15.114:0/1714696800 >> 172.21.15.114:0/1714696800 conn(0x7f5d500771d0 msgr2=0x7f5d500fecc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T10:26:47.427 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:47.427+0000 7f5d54a2c640 1 -- 172.21.15.114:0/1714696800 shutdown_connections 2024-04-23T10:26:47.427 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:47.427+0000 7f5d54a2c640 1 -- 172.21.15.114:0/1714696800 wait complete. 2024-04-23T10:26:47.437 DEBUG:teuthology.orchestra.run.smithi114:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2024-04-23T10:26:47.492 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 23 10:26:47 smithi119 ceph-mon[32608]: from='client.15438 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-04-23T10:26:47.492 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 23 10:26:47 smithi119 ceph-mon[32608]: from='client.15444 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-04-23T10:26:47.492 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 23 10:26:47 smithi119 ceph-mon[32608]: from='client.15450 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-04-23T10:26:47.492 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 23 10:26:47 smithi119 ceph-mon[32608]: from='client.? 172.21.15.114:0/3878333711' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-23T10:26:47.492 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 23 10:26:47 smithi119 ceph-mon[32608]: from='client.? 172.21.15.114:0/526715715' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-23T10:26:47.540 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:47.539+0000 7fbd67d30640 1 -- 172.21.15.114:0/1032701237 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fbd601010a0 msgr2=0x7fbd60101480 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T10:26:47.540 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:47.539+0000 7fbd67d30640 1 --2- 172.21.15.114:0/1032701237 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fbd601010a0 0x7fbd60101480 secure :-1 s=READY pgs=458 cs=0 l=1 rev1=1 crypto rx=0x7fbd5c009900 tx=0x7fbd5c02a410 comp rx=0 tx=0).stop 2024-04-23T10:26:47.540 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:47.539+0000 7fbd67d30640 1 -- 172.21.15.114:0/1032701237 shutdown_connections 2024-04-23T10:26:47.540 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:47.539+0000 7fbd67d30640 1 --2- 172.21.15.114:0/1032701237 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fbd60106170 0x7fbd6010dd40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:26:47.540 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:47.539+0000 7fbd67d30640 1 --2- 172.21.15.114:0/1032701237 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fbd60101a50 0x7fbd60105a90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:26:47.540 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:47.539+0000 7fbd67d30640 1 --2- 172.21.15.114:0/1032701237 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fbd601010a0 0x7fbd60101480 unknown :-1 s=CLOSED pgs=458 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:26:47.540 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:47.539+0000 7fbd67d30640 1 -- 172.21.15.114:0/1032701237 >> 172.21.15.114:0/1032701237 conn(0x7fbd60077eb0 msgr2=0x7fbd600782c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T10:26:47.541 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:47.539+0000 7fbd67d30640 1 -- 172.21.15.114:0/1032701237 shutdown_connections 2024-04-23T10:26:47.541 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:47.539+0000 7fbd67d30640 1 -- 172.21.15.114:0/1032701237 wait complete. 2024-04-23T10:26:47.541 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:47.540+0000 7fbd67d30640 1 Processor -- start 2024-04-23T10:26:47.541 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:47.540+0000 7fbd67d30640 1 -- start start 2024-04-23T10:26:47.541 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:47.540+0000 7fbd67d30640 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fbd601010a0 0x7fbd60198760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T10:26:47.541 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:47.540+0000 7fbd67d30640 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fbd60106170 0x7fbd60198ca0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T10:26:47.541 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:47.540+0000 7fbd67d30640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fbd6019d020 0x7fbd6019d4d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T10:26:47.541 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:47.540+0000 7fbd67d30640 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x7fbd601110d0 con 0x7fbd6019d020 2024-04-23T10:26:47.541 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:47.540+0000 7fbd67d30640 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 -- 0x7fbd60110f50 con 0x7fbd601010a0 2024-04-23T10:26:47.541 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:47.540+0000 7fbd67d30640 1 -- --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_getmap magic: 0 -- 0x7fbd60111250 con 0x7fbd60106170 2024-04-23T10:26:47.541 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:47.540+0000 7fbd662cd640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fbd6019d020 0x7fbd6019d4d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T10:26:47.541 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:47.540+0000 7fbd662cd640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fbd6019d020 0x7fbd6019d4d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.114:52000/0 (socket says 172.21.15.114:52000) 2024-04-23T10:26:47.541 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:47.540+0000 7fbd662cd640 1 -- 172.21.15.114:0/866259966 learned_addr learned my addr 172.21.15.114:0/866259966 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-04-23T10:26:47.541 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:47.540+0000 7fbd652cb640 1 --2- 172.21.15.114:0/866259966 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fbd60106170 0x7fbd60198ca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T10:26:47.541 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:47.540+0000 7fbd65acc640 1 --2- 172.21.15.114:0/866259966 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fbd601010a0 0x7fbd60198760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T10:26:47.541 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:47.540+0000 7fbd662cd640 1 -- 172.21.15.114:0/866259966 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fbd60106170 msgr2=0x7fbd60198ca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T10:26:47.542 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:47.540+0000 7fbd662cd640 1 --2- 172.21.15.114:0/866259966 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fbd60106170 0x7fbd60198ca0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:26:47.542 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:47.540+0000 7fbd662cd640 1 -- 172.21.15.114:0/866259966 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fbd601010a0 msgr2=0x7fbd60198760 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T10:26:47.542 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:47.540+0000 7fbd662cd640 1 --2- 172.21.15.114:0/866259966 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fbd601010a0 0x7fbd60198760 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:26:47.542 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:47.540+0000 7fbd662cd640 1 -- 172.21.15.114:0/866259966 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fbd6019db10 con 0x7fbd6019d020 2024-04-23T10:26:47.542 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:47.540+0000 7fbd65acc640 1 --2- 172.21.15.114:0/866259966 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fbd601010a0 0x7fbd60198760 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-23T10:26:47.542 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:47.541+0000 7fbd662cd640 1 --2- 172.21.15.114:0/866259966 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fbd6019d020 0x7fbd6019d4d0 secure :-1 s=READY pgs=459 cs=0 l=1 rev1=1 crypto rx=0x7fbd5800eb10 tx=0x7fbd5800c2d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T10:26:47.542 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:47.541+0000 7fbd56ffd640 1 -- 172.21.15.114:0/866259966 <== mon.0 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbd58004b20 con 0x7fbd6019d020 2024-04-23T10:26:47.542 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:47.541+0000 7fbd56ffd640 1 -- 172.21.15.114:0/866259966 <== mon.0 v2:172.21.15.114:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fbd58004cc0 con 0x7fbd6019d020 2024-04-23T10:26:47.542 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:47.541+0000 7fbd67d30640 1 -- 172.21.15.114:0/866259966 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fbd6019de00 con 0x7fbd6019d020 2024-04-23T10:26:47.542 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:47.541+0000 7fbd56ffd640 1 -- 172.21.15.114:0/866259966 <== mon.0 v2:172.21.15.114:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbd58010670 con 0x7fbd6019d020 2024-04-23T10:26:47.542 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:47.541+0000 7fbd67d30640 1 -- 172.21.15.114:0/866259966 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fbd601a55f0 con 0x7fbd6019d020 2024-04-23T10:26:47.543 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:47.542+0000 7fbd67d30640 1 -- 172.21.15.114:0/866259966 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fbd600691c0 con 0x7fbd6019d020 2024-04-23T10:26:47.543 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:47.543+0000 7fbd56ffd640 1 -- 172.21.15.114:0/866259966 <== mon.0 v2:172.21.15.114:3300/0 4 ==== mgrmap(e 24) ==== 138823+0+0 (secure 0 0 0) 0x7fbd58005020 con 0x7fbd6019d020 2024-04-23T10:26:47.543 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:47.543+0000 7fbd56ffd640 1 --2- 172.21.15.114:0/866259966 >> [v2:172.21.15.114:6800/1395693466,v1:172.21.15.114:6801/1395693466] conn(0x7fbd4c0a3e30 0x7fbd4c0a62f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T10:26:47.543 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:47.543+0000 7fbd56ffd640 1 -- 172.21.15.114:0/866259966 <== mon.0 v2:172.21.15.114:3300/0 5 ==== osd_map(90..90 src has 1..90) ==== 8527+0+0 (secure 0 0 0) 0x7fbd580d0ef0 con 0x7fbd6019d020 2024-04-23T10:26:47.543 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:47.543+0000 7fbd65acc640 1 --2- 172.21.15.114:0/866259966 >> [v2:172.21.15.114:6800/1395693466,v1:172.21.15.114:6801/1395693466] conn(0x7fbd4c0a3e30 0x7fbd4c0a62f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T10:26:47.543 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:47.544+0000 7fbd65acc640 1 --2- 172.21.15.114:0/866259966 >> [v2:172.21.15.114:6800/1395693466,v1:172.21.15.114:6801/1395693466] conn(0x7fbd4c0a3e30 0x7fbd4c0a62f0 secure :-1 s=READY pgs=55 cs=0 l=1 rev1=1 crypto rx=0x7fbd5c031440 tx=0x7fbd5c005a20 comp rx=0 tx=0).ready entity=mgr.15291 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T10:26:47.546 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:47.548+0000 7fbd56ffd640 1 -- 172.21.15.114:0/866259966 <== mon.0 v2:172.21.15.114:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7fbd5809d420 con 0x7fbd6019d020 2024-04-23T10:26:47.726 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:47.727+0000 7fbd67d30640 1 -- 172.21.15.114:0/866259966 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) -- 0x7fbd60101a50 con 0x7fbd6019d020 2024-04-23T10:26:47.727 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:47.728+0000 7fbd56ffd640 1 -- 172.21.15.114:0/866259966 <== mon.0 v2:172.21.15.114:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v90) ==== 100+0+11 (secure 0 0 0) 0x7fbd5809cd20 con 0x7fbd6019d020 2024-04-23T10:26:47.727 INFO:teuthology.orchestra.run.smithi114.stdout:pg_num: 64 2024-04-23T10:26:47.729 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:47.730+0000 7fbd67d30640 1 -- 172.21.15.114:0/866259966 >> [v2:172.21.15.114:6800/1395693466,v1:172.21.15.114:6801/1395693466] conn(0x7fbd4c0a3e30 msgr2=0x7fbd4c0a62f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T10:26:47.729 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:47.730+0000 7fbd67d30640 1 --2- 172.21.15.114:0/866259966 >> [v2:172.21.15.114:6800/1395693466,v1:172.21.15.114:6801/1395693466] conn(0x7fbd4c0a3e30 0x7fbd4c0a62f0 secure :-1 s=READY pgs=55 cs=0 l=1 rev1=1 crypto rx=0x7fbd5c031440 tx=0x7fbd5c005a20 comp rx=0 tx=0).stop 2024-04-23T10:26:47.729 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:47.730+0000 7fbd67d30640 1 -- 172.21.15.114:0/866259966 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fbd6019d020 msgr2=0x7fbd6019d4d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T10:26:47.729 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:47.730+0000 7fbd67d30640 1 --2- 172.21.15.114:0/866259966 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fbd6019d020 0x7fbd6019d4d0 secure :-1 s=READY pgs=459 cs=0 l=1 rev1=1 crypto rx=0x7fbd5800eb10 tx=0x7fbd5800c2d0 comp rx=0 tx=0).stop 2024-04-23T10:26:47.729 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:47.730+0000 7fbd67d30640 1 -- 172.21.15.114:0/866259966 shutdown_connections 2024-04-23T10:26:47.729 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:47.730+0000 7fbd67d30640 1 --2- 172.21.15.114:0/866259966 >> [v2:172.21.15.114:6800/1395693466,v1:172.21.15.114:6801/1395693466] conn(0x7fbd4c0a3e30 0x7fbd4c0a62f0 unknown :-1 s=CLOSED pgs=55 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:26:47.729 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:47.730+0000 7fbd67d30640 1 --2- 172.21.15.114:0/866259966 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fbd6019d020 0x7fbd6019d4d0 unknown :-1 s=CLOSED pgs=459 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:26:47.729 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:47.730+0000 7fbd67d30640 1 --2- 172.21.15.114:0/866259966 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fbd60106170 0x7fbd60198ca0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:26:47.730 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:47.730+0000 7fbd67d30640 1 --2- 172.21.15.114:0/866259966 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fbd601010a0 0x7fbd60198760 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:26:47.730 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:47.730+0000 7fbd67d30640 1 -- 172.21.15.114:0/866259966 >> 172.21.15.114:0/866259966 conn(0x7fbd60077eb0 msgr2=0x7fbd6006a990 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T10:26:47.730 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:47.730+0000 7fbd67d30640 1 -- 172.21.15.114:0/866259966 shutdown_connections 2024-04-23T10:26:47.730 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:47.730+0000 7fbd67d30640 1 -- 172.21.15.114:0/866259966 wait complete. 2024-04-23T10:26:47.739 DEBUG:teuthology.orchestra.run.smithi114:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-23T10:26:47.840 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:47.838+0000 7f8d4c02c640 1 -- 172.21.15.114:0/3277950042 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f8d44101090 msgr2=0x7f8d44101470 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T10:26:47.840 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:47.838+0000 7f8d4c02c640 1 --2- 172.21.15.114:0/3277950042 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f8d44101090 0x7f8d44101470 secure :-1 s=READY pgs=460 cs=0 l=1 rev1=1 crypto rx=0x7f8d40009900 tx=0x7f8d4002a410 comp rx=0 tx=0).stop 2024-04-23T10:26:47.840 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:47.838+0000 7f8d4c02c640 1 -- 172.21.15.114:0/3277950042 shutdown_connections 2024-04-23T10:26:47.840 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:47.838+0000 7f8d4c02c640 1 --2- 172.21.15.114:0/3277950042 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f8d44106160 0x7f8d4410dd30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:26:47.840 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:47.838+0000 7f8d4c02c640 1 --2- 172.21.15.114:0/3277950042 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f8d44101a40 0x7f8d44105a80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:26:47.840 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:47.838+0000 7f8d4c02c640 1 --2- 172.21.15.114:0/3277950042 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f8d44101090 0x7f8d44101470 unknown :-1 s=CLOSED pgs=460 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:26:47.840 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:47.838+0000 7f8d4c02c640 1 -- 172.21.15.114:0/3277950042 >> 172.21.15.114:0/3277950042 conn(0x7f8d440771d0 msgr2=0x7f8d440775e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T10:26:47.840 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:47.838+0000 7f8d4c02c640 1 -- 172.21.15.114:0/3277950042 shutdown_connections 2024-04-23T10:26:47.840 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:47.838+0000 7f8d4c02c640 1 -- 172.21.15.114:0/3277950042 wait complete. 2024-04-23T10:26:47.840 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:47.839+0000 7f8d4c02c640 1 Processor -- start 2024-04-23T10:26:47.840 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:47.839+0000 7f8d4c02c640 1 -- start start 2024-04-23T10:26:47.841 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:47.839+0000 7f8d4c02c640 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f8d44101090 0x7f8d4410e6c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T10:26:47.841 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:47.839+0000 7f8d4c02c640 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f8d44106160 0x7f8d4410ec00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T10:26:47.841 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:47.839+0000 7f8d4c02c640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f8d4410f140 0x7f8d441a68b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T10:26:47.841 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:47.839+0000 7f8d4c02c640 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x7f8d441121e0 con 0x7f8d4410f140 2024-04-23T10:26:47.841 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:47.839+0000 7f8d4c02c640 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 -- 0x7f8d44112060 con 0x7f8d44101090 2024-04-23T10:26:47.841 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:47.839+0000 7f8d4c02c640 1 -- --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_getmap magic: 0 -- 0x7f8d44112360 con 0x7f8d44106160 2024-04-23T10:26:47.841 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:47.839+0000 7f8d4a5c9640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f8d4410f140 0x7f8d441a68b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T10:26:47.841 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:47.839+0000 7f8d4a5c9640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f8d4410f140 0x7f8d441a68b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.114:52014/0 (socket says 172.21.15.114:52014) 2024-04-23T10:26:47.841 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:47.839+0000 7f8d4a5c9640 1 -- 172.21.15.114:0/2867739893 learned_addr learned my addr 172.21.15.114:0/2867739893 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-04-23T10:26:47.841 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:47.839+0000 7f8d49dc8640 1 --2- 172.21.15.114:0/2867739893 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f8d44101090 0x7f8d4410e6c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T10:26:47.841 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:47.839+0000 7f8d495c7640 1 --2- 172.21.15.114:0/2867739893 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f8d44106160 0x7f8d4410ec00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T10:26:47.841 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:47.839+0000 7f8d4a5c9640 1 -- 172.21.15.114:0/2867739893 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f8d44106160 msgr2=0x7f8d4410ec00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T10:26:47.841 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:47.839+0000 7f8d4a5c9640 1 --2- 172.21.15.114:0/2867739893 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f8d44106160 0x7f8d4410ec00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:26:47.841 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:47.839+0000 7f8d4a5c9640 1 -- 172.21.15.114:0/2867739893 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f8d44101090 msgr2=0x7f8d4410e6c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T10:26:47.842 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:47.839+0000 7f8d4a5c9640 1 --2- 172.21.15.114:0/2867739893 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f8d44101090 0x7f8d4410e6c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:26:47.842 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:47.839+0000 7f8d4a5c9640 1 -- 172.21.15.114:0/2867739893 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f8d441a6fb0 con 0x7f8d4410f140 2024-04-23T10:26:47.842 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:47.839+0000 7f8d495c7640 1 --2- 172.21.15.114:0/2867739893 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f8d44106160 0x7f8d4410ec00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-23T10:26:47.842 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:47.840+0000 7f8d4a5c9640 1 --2- 172.21.15.114:0/2867739893 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f8d4410f140 0x7f8d441a68b0 secure :-1 s=READY pgs=461 cs=0 l=1 rev1=1 crypto rx=0x7f8d3c00cce0 tx=0x7f8d3c00a620 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T10:26:47.842 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:47.840+0000 7f8d3affd640 1 -- 172.21.15.114:0/2867739893 <== mon.0 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8d3c00f450 con 0x7f8d4410f140 2024-04-23T10:26:47.842 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:47.840+0000 7f8d3affd640 1 -- 172.21.15.114:0/2867739893 <== mon.0 v2:172.21.15.114:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f8d3c00faa0 con 0x7f8d4410f140 2024-04-23T10:26:47.842 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:47.840+0000 7f8d3affd640 1 -- 172.21.15.114:0/2867739893 <== mon.0 v2:172.21.15.114:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8d3c018b70 con 0x7f8d4410f140 2024-04-23T10:26:47.842 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:47.840+0000 7f8d4c02c640 1 -- 172.21.15.114:0/2867739893 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f8d441a72a0 con 0x7f8d4410f140 2024-04-23T10:26:47.842 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:47.840+0000 7f8d4c02c640 1 -- 172.21.15.114:0/2867739893 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f8d441a77a0 con 0x7f8d4410f140 2024-04-23T10:26:47.842 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:47.841+0000 7f8d4c02c640 1 -- 172.21.15.114:0/2867739893 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f8d440691c0 con 0x7f8d4410f140 2024-04-23T10:26:47.843 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:47.842+0000 7f8d3affd640 1 -- 172.21.15.114:0/2867739893 <== mon.0 v2:172.21.15.114:3300/0 4 ==== mgrmap(e 24) ==== 138823+0+0 (secure 0 0 0) 0x7f8d3c018d10 con 0x7f8d4410f140 2024-04-23T10:26:47.843 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:47.842+0000 7f8d3affd640 1 --2- 172.21.15.114:0/2867739893 >> [v2:172.21.15.114:6800/1395693466,v1:172.21.15.114:6801/1395693466] conn(0x7f8d300a3de0 0x7f8d300a62a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T10:26:47.843 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:47.842+0000 7f8d49dc8640 1 --2- 172.21.15.114:0/2867739893 >> [v2:172.21.15.114:6800/1395693466,v1:172.21.15.114:6801/1395693466] conn(0x7f8d300a3de0 0x7f8d300a62a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T10:26:47.843 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:47.842+0000 7f8d3affd640 1 -- 172.21.15.114:0/2867739893 <== mon.0 v2:172.21.15.114:3300/0 5 ==== osd_map(90..90 src has 1..90) ==== 8527+0+0 (secure 0 0 0) 0x7f8d3c0d0df0 con 0x7f8d4410f140 2024-04-23T10:26:47.843 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:47.843+0000 7f8d49dc8640 1 --2- 172.21.15.114:0/2867739893 >> [v2:172.21.15.114:6800/1395693466,v1:172.21.15.114:6801/1395693466] conn(0x7f8d300a3de0 0x7f8d300a62a0 secure :-1 s=READY pgs=56 cs=0 l=1 rev1=1 crypto rx=0x7f8d40031440 tx=0x7f8d400095f0 comp rx=0 tx=0).ready entity=mgr.15291 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T10:26:47.846 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:47.847+0000 7f8d3affd640 1 -- 172.21.15.114:0/2867739893 <== mon.0 v2:172.21.15.114:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f8d3c014030 con 0x7f8d4410f140 2024-04-23T10:26:48.062 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:48.063+0000 7f8d4c02c640 1 -- 172.21.15.114:0/2867739893 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f8d44101a40 con 0x7f8d4410f140 2024-04-23T10:26:48.063 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:48.064+0000 7f8d3affd640 1 -- 172.21.15.114:0/2867739893 <== mon.0 v2:172.21.15.114:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 21 v21) ==== 94+0+11884 (secure 0 0 0) 0x7f8d3c09d320 con 0x7f8d4410f140 2024-04-23T10:26:48.063 INFO:teuthology.orchestra.run.smithi114.stdout: 2024-04-23T10:26:48.063 INFO:teuthology.orchestra.run.smithi114.stdout:{"epoch":21,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14547,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.114:6835/663493580","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6834","nonce":663493580},{"type":"v1","addr":"172.21.15.114:6835","nonce":663493580}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14565,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.114:6837/1439252154","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6836","nonce":1439252154},{"type":"v1","addr":"172.21.15.114:6837","nonce":1439252154}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5},{"gid":14634,"name":"g","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.114:6839/3849796939","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6838","nonce":3849796939},{"type":"v1","addr":"172.21.15.114:6839","nonce":3849796939}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":11},{"gid":14646,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.114:6841/3544523891","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6840","nonce":3544523891},{"type":"v1","addr":"172.21.15.114:6841","nonce":3544523891}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":12},{"gid":24385,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.204:6833/3866561754","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6832","nonce":3866561754},{"type":"v1","addr":"172.21.15.204:6833","nonce":3866561754}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2},{"gid":24427,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.204:6835/2533793172","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6834","nonce":2533793172},{"type":"v1","addr":"172.21.15.204:6835","nonce":2533793172}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":7}],"filesystems":[{"mdsmap":{"epoch":21,"flags":306,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":true},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-04-23T10:24:43.383158+0000","modified":"2024-04-23T10:26:38.945346+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":3,"in":[0,1,2],"up":{"mds_0":24545,"mds_1":24509,"mds_2":24479},"failed":[],"damaged":[],"stopped":[],"info":{"gid_24439":{"gid":24439,"name":"c","rank":2,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.204:6837/513822179","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6836","nonce":513822179},{"type":"v1","addr":"172.21.15.204:6837","nonce":513822179}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24443":{"gid":24443,"name":"k","rank":1,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.119:6833/783244660","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6832","nonce":783244660},{"type":"v1","addr":"172.21.15.119:6833","nonce":783244660}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24463":{"gid":24463,"name":"i","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.204:6839/3222658893","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6838","nonce":3222658893},{"type":"v1","addr":"172.21.15.204:6839","nonce":3222658893}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24479":{"gid":24479,"name":"e","rank":2,"incarnation":19,"state":"up:active","state_seq":9,"addr":"172.21.15.119:6835/2033916709","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6834","nonce":2033916709},{"type":"v1","addr":"172.21.15.119:6835","nonce":2033916709}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24509":{"gid":24509,"name":"h","rank":1,"incarnation":17,"state":"up:active","state_seq":6,"addr":"172.21.15.119:6837/1872363128","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6836","nonce":1872363128},{"type":"v1","addr":"172.21.15.119:6837","nonce":1872363128}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24545":{"gid":24545,"name":"b","rank":0,"incarnation":14,"state":"up:active","state_seq":3,"addr":"172.21.15.119:6839/937297340","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6838","nonce":937297340},{"type":"v1","addr":"172.21.15.119:6839","nonce":937297340}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":24545,"qdb_cluster":[24479,24509,24545]},"id":1}]} 2024-04-23T10:26:48.063 INFO:teuthology.orchestra.run.smithi114.stderr:dumped fsmap epoch 21 2024-04-23T10:26:48.065 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:48.066+0000 7f8d4c02c640 1 -- 172.21.15.114:0/2867739893 >> [v2:172.21.15.114:6800/1395693466,v1:172.21.15.114:6801/1395693466] conn(0x7f8d300a3de0 msgr2=0x7f8d300a62a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T10:26:48.065 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:48.066+0000 7f8d4c02c640 1 --2- 172.21.15.114:0/2867739893 >> [v2:172.21.15.114:6800/1395693466,v1:172.21.15.114:6801/1395693466] conn(0x7f8d300a3de0 0x7f8d300a62a0 secure :-1 s=READY pgs=56 cs=0 l=1 rev1=1 crypto rx=0x7f8d40031440 tx=0x7f8d400095f0 comp rx=0 tx=0).stop 2024-04-23T10:26:48.065 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:48.066+0000 7f8d4c02c640 1 -- 172.21.15.114:0/2867739893 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f8d4410f140 msgr2=0x7f8d441a68b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T10:26:48.065 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:48.066+0000 7f8d4c02c640 1 --2- 172.21.15.114:0/2867739893 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f8d4410f140 0x7f8d441a68b0 secure :-1 s=READY pgs=461 cs=0 l=1 rev1=1 crypto rx=0x7f8d3c00cce0 tx=0x7f8d3c00a620 comp rx=0 tx=0).stop 2024-04-23T10:26:48.065 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:48.066+0000 7f8d4c02c640 1 -- 172.21.15.114:0/2867739893 shutdown_connections 2024-04-23T10:26:48.065 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:48.066+0000 7f8d4c02c640 1 --2- 172.21.15.114:0/2867739893 >> [v2:172.21.15.114:6800/1395693466,v1:172.21.15.114:6801/1395693466] conn(0x7f8d300a3de0 0x7f8d300a62a0 unknown :-1 s=CLOSED pgs=56 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:26:48.066 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:48.066+0000 7f8d4c02c640 1 --2- 172.21.15.114:0/2867739893 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f8d4410f140 0x7f8d441a68b0 unknown :-1 s=CLOSED pgs=461 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:26:48.066 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:48.066+0000 7f8d4c02c640 1 --2- 172.21.15.114:0/2867739893 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f8d44106160 0x7f8d4410ec00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:26:48.066 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:48.066+0000 7f8d4c02c640 1 --2- 172.21.15.114:0/2867739893 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f8d44101090 0x7f8d4410e6c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:26:48.066 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:48.066+0000 7f8d4c02c640 1 -- 172.21.15.114:0/2867739893 >> 172.21.15.114:0/2867739893 conn(0x7f8d440771d0 msgr2=0x7f8d440feda0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T10:26:48.066 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:48.066+0000 7f8d4c02c640 1 -- 172.21.15.114:0/2867739893 shutdown_connections 2024-04-23T10:26:48.066 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:48.066+0000 7f8d4c02c640 1 -- 172.21.15.114:0/2867739893 wait complete. 2024-04-23T10:26:48.076 DEBUG:teuthology.orchestra.run.smithi114:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-04-23T10:26:48.176 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:48.174+0000 7f6398fad640 1 -- 172.21.15.114:0/2026301482 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f6394106170 msgr2=0x7f639410dd40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T10:26:48.176 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:48.174+0000 7f6398fad640 1 --2- 172.21.15.114:0/2026301482 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f6394106170 0x7f639410dd40 secure :-1 s=READY pgs=462 cs=0 l=1 rev1=1 crypto rx=0x7f638c009ca0 tx=0x7f638c02a8c0 comp rx=0 tx=0).stop 2024-04-23T10:26:48.176 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:48.174+0000 7f6398fad640 1 -- 172.21.15.114:0/2026301482 shutdown_connections 2024-04-23T10:26:48.176 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:48.174+0000 7f6398fad640 1 --2- 172.21.15.114:0/2026301482 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f6394106170 0x7f639410dd40 unknown :-1 s=CLOSED pgs=462 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:26:48.176 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:48.175+0000 7f6398fad640 1 --2- 172.21.15.114:0/2026301482 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f6394101a50 0x7f6394105a90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:26:48.176 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:48.175+0000 7f6398fad640 1 --2- 172.21.15.114:0/2026301482 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f63941010a0 0x7f6394101480 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:26:48.176 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:48.175+0000 7f6398fad640 1 -- 172.21.15.114:0/2026301482 >> 172.21.15.114:0/2026301482 conn(0x7f6394077f50 msgr2=0x7f6394078360 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T10:26:48.176 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:48.175+0000 7f6398fad640 1 -- 172.21.15.114:0/2026301482 shutdown_connections 2024-04-23T10:26:48.176 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:48.175+0000 7f6398fad640 1 -- 172.21.15.114:0/2026301482 wait complete. 2024-04-23T10:26:48.176 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:48.175+0000 7f6398fad640 1 Processor -- start 2024-04-23T10:26:48.176 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:48.175+0000 7f6398fad640 1 -- start start 2024-04-23T10:26:48.176 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:48.175+0000 7f6398fad640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f63941010a0 0x7f63941994e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T10:26:48.176 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:48.175+0000 7f6398fad640 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f6394106170 0x7f6394199a20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T10:26:48.177 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:48.175+0000 7f6398fad640 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f639419a000 0x7f63941a4560 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T10:26:48.177 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:48.175+0000 7f6398fad640 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x7f63941110e0 con 0x7f63941010a0 2024-04-23T10:26:48.177 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:48.175+0000 7f6398fad640 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 -- 0x7f6394110f60 con 0x7f6394106170 2024-04-23T10:26:48.177 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:48.175+0000 7f6398fad640 1 -- --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_getmap magic: 0 -- 0x7f6394111260 con 0x7f639419a000 2024-04-23T10:26:48.177 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:48.176+0000 7f639259c640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f63941010a0 0x7f63941994e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T10:26:48.177 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:48.176+0000 7f639259c640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f63941010a0 0x7f63941994e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.114:52032/0 (socket says 172.21.15.114:52032) 2024-04-23T10:26:48.177 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:48.176+0000 7f639259c640 1 -- 172.21.15.114:0/1272188247 learned_addr learned my addr 172.21.15.114:0/1272188247 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-04-23T10:26:48.177 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:48.176+0000 7f638bfff640 1 --2- 172.21.15.114:0/1272188247 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f6394106170 0x7f6394199a20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T10:26:48.177 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:48.176+0000 7f6392d9d640 1 --2- 172.21.15.114:0/1272188247 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f639419a000 0x7f63941a4560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T10:26:48.177 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:48.176+0000 7f639259c640 1 -- 172.21.15.114:0/1272188247 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f639419a000 msgr2=0x7f63941a4560 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T10:26:48.177 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:48.176+0000 7f639259c640 1 --2- 172.21.15.114:0/1272188247 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f639419a000 0x7f63941a4560 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:26:48.177 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:48.176+0000 7f639259c640 1 -- 172.21.15.114:0/1272188247 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f6394106170 msgr2=0x7f6394199a20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T10:26:48.177 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:48.176+0000 7f639259c640 1 --2- 172.21.15.114:0/1272188247 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f6394106170 0x7f6394199a20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:26:48.177 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:48.176+0000 7f639259c640 1 -- 172.21.15.114:0/1272188247 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f63941a4d80 con 0x7f63941010a0 2024-04-23T10:26:48.177 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:48.176+0000 7f638bfff640 1 --2- 172.21.15.114:0/1272188247 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f6394106170 0x7f6394199a20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-23T10:26:48.178 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:48.176+0000 7f639259c640 1 --2- 172.21.15.114:0/1272188247 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f63941010a0 0x7f63941994e0 secure :-1 s=READY pgs=463 cs=0 l=1 rev1=1 crypto rx=0x7f638400e9f0 tx=0x7f638400eeb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T10:26:48.178 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:48.176+0000 7f638b7fe640 1 -- 172.21.15.114:0/1272188247 <== mon.0 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f638400cd80 con 0x7f63941010a0 2024-04-23T10:26:48.178 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:48.176+0000 7f638b7fe640 1 -- 172.21.15.114:0/1272188247 <== mon.0 v2:172.21.15.114:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f63840108e0 con 0x7f63941010a0 2024-04-23T10:26:48.178 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:48.176+0000 7f638b7fe640 1 -- 172.21.15.114:0/1272188247 <== mon.0 v2:172.21.15.114:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6384018980 con 0x7f63941010a0 2024-04-23T10:26:48.178 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:48.176+0000 7f6398fad640 1 -- 172.21.15.114:0/1272188247 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f63941a5070 con 0x7f63941010a0 2024-04-23T10:26:48.178 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:48.176+0000 7f6398fad640 1 -- 172.21.15.114:0/1272188247 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f63941a55c0 con 0x7f63941010a0 2024-04-23T10:26:48.179 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:48.177+0000 7f6398fad640 1 -- 172.21.15.114:0/1272188247 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f6394068d70 con 0x7f63941010a0 2024-04-23T10:26:48.179 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:48.178+0000 7f638b7fe640 1 -- 172.21.15.114:0/1272188247 <== mon.0 v2:172.21.15.114:3300/0 4 ==== mgrmap(e 24) ==== 138823+0+0 (secure 0 0 0) 0x7f6384010430 con 0x7f63941010a0 2024-04-23T10:26:48.179 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:48.179+0000 7f638b7fe640 1 --2- 172.21.15.114:0/1272188247 >> [v2:172.21.15.114:6800/1395693466,v1:172.21.15.114:6801/1395693466] conn(0x7f63800a3db0 0x7f63800a6270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T10:26:48.179 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:48.179+0000 7f638bfff640 1 --2- 172.21.15.114:0/1272188247 >> [v2:172.21.15.114:6800/1395693466,v1:172.21.15.114:6801/1395693466] conn(0x7f63800a3db0 0x7f63800a6270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T10:26:48.179 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:48.179+0000 7f638b7fe640 1 -- 172.21.15.114:0/1272188247 <== mon.0 v2:172.21.15.114:3300/0 5 ==== osd_map(90..90 src has 1..90) ==== 8527+0+0 (secure 0 0 0) 0x7f63840d0de0 con 0x7f63941010a0 2024-04-23T10:26:48.179 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:48.179+0000 7f638bfff640 1 --2- 172.21.15.114:0/1272188247 >> [v2:172.21.15.114:6800/1395693466,v1:172.21.15.114:6801/1395693466] conn(0x7f63800a3db0 0x7f63800a6270 secure :-1 s=READY pgs=57 cs=0 l=1 rev1=1 crypto rx=0x7f637c005fd0 tx=0x7f637c0094d0 comp rx=0 tx=0).ready entity=mgr.15291 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T10:26:48.182 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:48.184+0000 7f638b7fe640 1 -- 172.21.15.114:0/1272188247 <== mon.0 v2:172.21.15.114:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f638409d310 con 0x7f63941010a0 2024-04-23T10:26:48.328 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 23 10:26:48 smithi204 ceph-mon[32649]: pgmap v6: 97 pgs: 97 active+clean; 1.1 MiB data, 376 MiB used, 1.0 TiB / 1.0 TiB avail; 41 KiB/s rd, 90 KiB/s wr, 66 op/s 2024-04-23T10:26:48.328 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 23 10:26:48 smithi204 ceph-mon[32649]: from='client.? 172.21.15.114:0/3757603754' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-04-23T10:26:48.328 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 23 10:26:48 smithi204 ceph-mon[32649]: from='client.? 172.21.15.114:0/1714696800' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2024-04-23T10:26:48.328 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 23 10:26:48 smithi204 ceph-mon[32649]: from='client.? 172.21.15.114:0/866259966' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2024-04-23T10:26:48.365 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:48.366+0000 7f6398fad640 1 -- 172.21.15.114:0/1272188247 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f63941093a0 con 0x7f63941010a0 2024-04-23T10:26:48.366 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:48.368+0000 7f638b7fe640 1 -- 172.21.15.114:0/1272188247 <== mon.0 v2:172.21.15.114:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v90) ==== 74+0+23198 (secure 0 0 0) 0x7f638409cc10 con 0x7f63941010a0 2024-04-23T10:26:48.367 INFO:teuthology.orchestra.run.smithi114.stdout: 2024-04-23T10:26:48.367 INFO:teuthology.orchestra.run.smithi114.stdout:{"epoch":90,"fsid":"aa240572-0159-11ef-bc93-c7b262605968","created":"2024-04-23T10:16:53.351136+0000","modified":"2024-04-23T10:26:40.272696+0000","last_up_change":"2024-04-23T10:23:54.153269+0000","last_in_change":"2024-04-23T10:23:32.963085+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":26,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-04-23T10:20:11.970297+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":12,"score_stable":12,"optimal_score":0.25,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-04-23T10:24:41.218540+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"86","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.8799999952316284,"score_stable":1.8799999952316284,"optimal_score":1,"raw_score_acting":1.8799999952316284,"raw_score_stable":1.8799999952316284,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-04-23T10:24:42.341799+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"87","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"fd64adde-9445-45a4-a171-ef656e2c5709","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":88,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6802","nonce":736366878},{"type":"v1","addr":"172.21.15.114:6803","nonce":736366878}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6804","nonce":736366878},{"type":"v1","addr":"172.21.15.114:6805","nonce":736366878}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6808","nonce":736366878},{"type":"v1","addr":"172.21.15.114:6809","nonce":736366878}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6806","nonce":736366878},{"type":"v1","addr":"172.21.15.114:6807","nonce":736366878}]},"public_addr":"172.21.15.114:6803/736366878","cluster_addr":"172.21.15.114:6805/736366878","heartbeat_back_addr":"172.21.15.114:6809/736366878","heartbeat_front_addr":"172.21.15.114:6807/736366878","state":["exists","up"]},{"osd":1,"uuid":"e279a1e6-2702-472a-930a-e08ece76a861","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":85,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6800","nonce":1133540779},{"type":"v1","addr":"172.21.15.119:6801","nonce":1133540779}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6802","nonce":1133540779},{"type":"v1","addr":"172.21.15.119:6803","nonce":1133540779}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6806","nonce":1133540779},{"type":"v1","addr":"172.21.15.119:6807","nonce":1133540779}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6804","nonce":1133540779},{"type":"v1","addr":"172.21.15.119:6805","nonce":1133540779}]},"public_addr":"172.21.15.119:6801/1133540779","cluster_addr":"172.21.15.119:6803/1133540779","heartbeat_back_addr":"172.21.15.119:6807/1133540779","heartbeat_front_addr":"172.21.15.119:6805/1133540779","state":["exists","up"]},{"osd":2,"uuid":"f4c9f2b4-ff49-4e14-89cd-b477feeb9382","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":85,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6800","nonce":799652511},{"type":"v1","addr":"172.21.15.204:6801","nonce":799652511}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6802","nonce":799652511},{"type":"v1","addr":"172.21.15.204:6803","nonce":799652511}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6806","nonce":799652511},{"type":"v1","addr":"172.21.15.204:6807","nonce":799652511}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6804","nonce":799652511},{"type":"v1","addr":"172.21.15.204:6805","nonce":799652511}]},"public_addr":"172.21.15.204:6801/799652511","cluster_addr":"172.21.15.204:6803/799652511","heartbeat_back_addr":"172.21.15.204:6807/799652511","heartbeat_front_addr":"172.21.15.204:6805/799652511","state":["exists","up"]},{"osd":3,"uuid":"52d2a566-e71c-4bb3-8d6c-5abdd43177c2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":25,"up_thru":88,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6810","nonce":4275595091},{"type":"v1","addr":"172.21.15.114:6811","nonce":4275595091}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6812","nonce":4275595091},{"type":"v1","addr":"172.21.15.114:6813","nonce":4275595091}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6816","nonce":4275595091},{"type":"v1","addr":"172.21.15.114:6817","nonce":4275595091}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6814","nonce":4275595091},{"type":"v1","addr":"172.21.15.114:6815","nonce":4275595091}]},"public_addr":"172.21.15.114:6811/4275595091","cluster_addr":"172.21.15.114:6813/4275595091","heartbeat_back_addr":"172.21.15.114:6817/4275595091","heartbeat_front_addr":"172.21.15.114:6815/4275595091","state":["exists","up"]},{"osd":4,"uuid":"d911a7c1-4bdc-4d28-b8a6-73896a6e7de6","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":33,"up_thru":85,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6808","nonce":1611007172},{"type":"v1","addr":"172.21.15.119:6809","nonce":1611007172}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6810","nonce":1611007172},{"type":"v1","addr":"172.21.15.119:6811","nonce":1611007172}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6814","nonce":1611007172},{"type":"v1","addr":"172.21.15.119:6815","nonce":1611007172}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6812","nonce":1611007172},{"type":"v1","addr":"172.21.15.119:6813","nonce":1611007172}]},"public_addr":"172.21.15.119:6809/1611007172","cluster_addr":"172.21.15.119:6811/1611007172","heartbeat_back_addr":"172.21.15.119:6815/1611007172","heartbeat_front_addr":"172.21.15.119:6813/1611007172","state":["exists","up"]},{"osd":5,"uuid":"f7590e1a-e93c-4c29-b15f-38ee52d1f921","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":39,"up_thru":85,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6808","nonce":1427563312},{"type":"v1","addr":"172.21.15.204:6809","nonce":1427563312}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6810","nonce":1427563312},{"type":"v1","addr":"172.21.15.204:6811","nonce":1427563312}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6814","nonce":1427563312},{"type":"v1","addr":"172.21.15.204:6815","nonce":1427563312}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6812","nonce":1427563312},{"type":"v1","addr":"172.21.15.204:6813","nonce":1427563312}]},"public_addr":"172.21.15.204:6809/1427563312","cluster_addr":"172.21.15.204:6811/1427563312","heartbeat_back_addr":"172.21.15.204:6815/1427563312","heartbeat_front_addr":"172.21.15.204:6813/1427563312","state":["exists","up"]},{"osd":6,"uuid":"534d20e8-3e54-4db8-a845-6119f49c7a78","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":47,"up_thru":88,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6818","nonce":4072912544},{"type":"v1","addr":"172.21.15.114:6819","nonce":4072912544}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6820","nonce":4072912544},{"type":"v1","addr":"172.21.15.114:6821","nonce":4072912544}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6824","nonce":4072912544},{"type":"v1","addr":"172.21.15.114:6825","nonce":4072912544}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6822","nonce":4072912544},{"type":"v1","addr":"172.21.15.114:6823","nonce":4072912544}]},"public_addr":"172.21.15.114:6819/4072912544","cluster_addr":"172.21.15.114:6821/4072912544","heartbeat_back_addr":"172.21.15.114:6825/4072912544","heartbeat_front_addr":"172.21.15.114:6823/4072912544","state":["exists","up"]},{"osd":7,"uuid":"f0ec22b8-1b48-4859-9d65-b00403aff3ad","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":53,"up_thru":88,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6816","nonce":2237458793},{"type":"v1","addr":"172.21.15.119:6817","nonce":2237458793}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6818","nonce":2237458793},{"type":"v1","addr":"172.21.15.119:6819","nonce":2237458793}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6822","nonce":2237458793},{"type":"v1","addr":"172.21.15.119:6823","nonce":2237458793}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6820","nonce":2237458793},{"type":"v1","addr":"172.21.15.119:6821","nonce":2237458793}]},"public_addr":"172.21.15.119:6817/2237458793","cluster_addr":"172.21.15.119:6819/2237458793","heartbeat_back_addr":"172.21.15.119:6823/2237458793","heartbeat_front_addr":"172.21.15.119:6821/2237458793","state":["exists","up"]},{"osd":8,"uuid":"3ac89686-6d76-4cef-9089-ac29e4b3bcac","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":62,"up_thru":85,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6816","nonce":1426318231},{"type":"v1","addr":"172.21.15.204:6817","nonce":1426318231}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6818","nonce":1426318231},{"type":"v1","addr":"172.21.15.204:6819","nonce":1426318231}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6822","nonce":1426318231},{"type":"v1","addr":"172.21.15.204:6823","nonce":1426318231}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6820","nonce":1426318231},{"type":"v1","addr":"172.21.15.204:6821","nonce":1426318231}]},"public_addr":"172.21.15.204:6817/1426318231","cluster_addr":"172.21.15.204:6819/1426318231","heartbeat_back_addr":"172.21.15.204:6823/1426318231","heartbeat_front_addr":"172.21.15.204:6821/1426318231","state":["exists","up"]},{"osd":9,"uuid":"a80ffe4d-1e98-460f-b046-c4754285cc5a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":69,"up_thru":88,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6826","nonce":2750635721},{"type":"v1","addr":"172.21.15.114:6827","nonce":2750635721}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6828","nonce":2750635721},{"type":"v1","addr":"172.21.15.114:6829","nonce":2750635721}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6832","nonce":2750635721},{"type":"v1","addr":"172.21.15.114:6833","nonce":2750635721}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6830","nonce":2750635721},{"type":"v1","addr":"172.21.15.114:6831","nonce":2750635721}]},"public_addr":"172.21.15.114:6827/2750635721","cluster_addr":"172.21.15.114:6829/2750635721","heartbeat_back_addr":"172.21.15.114:6833/2750635721","heartbeat_front_addr":"172.21.15.114:6831/2750635721","state":["exists","up"]},{"osd":10,"uuid":"cd75a579-bf12-4600-ac09-a8bfce772ced","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":74,"up_thru":85,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6824","nonce":1400187701},{"type":"v1","addr":"172.21.15.119:6825","nonce":1400187701}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6826","nonce":1400187701},{"type":"v1","addr":"172.21.15.119:6827","nonce":1400187701}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6830","nonce":1400187701},{"type":"v1","addr":"172.21.15.119:6831","nonce":1400187701}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6828","nonce":1400187701},{"type":"v1","addr":"172.21.15.119:6829","nonce":1400187701}]},"public_addr":"172.21.15.119:6825/1400187701","cluster_addr":"172.21.15.119:6827/1400187701","heartbeat_back_addr":"172.21.15.119:6831/1400187701","heartbeat_front_addr":"172.21.15.119:6829/1400187701","state":["exists","up"]},{"osd":11,"uuid":"53f037a5-3ec0-483a-80a1-dfee923a68e5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":82,"up_thru":88,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6824","nonce":2407298490},{"type":"v1","addr":"172.21.15.204:6825","nonce":2407298490}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6826","nonce":2407298490},{"type":"v1","addr":"172.21.15.204:6827","nonce":2407298490}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6830","nonce":2407298490},{"type":"v1","addr":"172.21.15.204:6831","nonce":2407298490}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6828","nonce":2407298490},{"type":"v1","addr":"172.21.15.204:6829","nonce":2407298490}]},"public_addr":"172.21.15.204:6825/2407298490","cluster_addr":"172.21.15.204:6827/2407298490","heartbeat_back_addr":"172.21.15.204:6831/2407298490","heartbeat_front_addr":"172.21.15.204:6829/2407298490","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-23T10:19:18.401147+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-23T10:19:42.611299+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-23T10:20:07.277367+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-23T10:20:31.208368+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-23T10:20:55.866225+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-23T10:21:19.961237+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-23T10:21:44.480783+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-23T10:22:09.173641+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-23T10:22:34.030328+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-23T10:22:59.652293+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-23T10:23:25.659673+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-23T10:23:51.168171+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"2.e","mappings":[{"from":2,"to":4}]},{"pgid":"2.1d","mappings":[{"from":8,"to":11}]},{"pgid":"3.e","mappings":[{"from":5,"to":9}]},{"pgid":"3.12","mappings":[{"from":3,"to":9}]},{"pgid":"3.17","mappings":[{"from":4,"to":6}]},{"pgid":"3.23","mappings":[{"from":4,"to":7}]},{"pgid":"3.27","mappings":[{"from":4,"to":6}]},{"pgid":"3.31","mappings":[{"from":4,"to":8}]},{"pgid":"3.38","mappings":[{"from":3,"to":9}]},{"pgid":"3.39","mappings":[{"from":5,"to":8}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.114:0/480720790":"2024-04-24T10:26:40.272662+0000","172.21.15.114:6801/465333808":"2024-04-24T10:26:40.272662+0000","172.21.15.114:6800/465333808":"2024-04-24T10:26:40.272662+0000","172.21.15.114:0/1787801099":"2024-04-24T10:26:40.272662+0000","172.21.15.114:6801/1604678557":"2024-04-24T10:17:45.874609+0000","172.21.15.114:0/4025385854":"2024-04-24T10:17:45.874609+0000","172.21.15.114:0/2228273469":"2024-04-24T10:17:19.910348+0000","172.21.15.114:0/2713201622":"2024-04-24T10:26:40.272662+0000","172.21.15.114:6800/2831530838":"2024-04-24T10:17:19.910348+0000","172.21.15.114:0/637004095":"2024-04-24T10:26:40.272662+0000","172.21.15.114:0/1333447450":"2024-04-24T10:17:45.874609+0000","172.21.15.114:0/2233026840":"2024-04-24T10:17:19.910348+0000","172.21.15.114:0/1902281545":"2024-04-24T10:17:19.910348+0000","172.21.15.114:6801/2831530838":"2024-04-24T10:17:19.910348+0000","172.21.15.114:6800/1604678557":"2024-04-24T10:17:45.874609+0000","172.21.15.114:0/1623553811":"2024-04-24T10:17:45.874609+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-23T10:26:48.369 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:48.370+0000 7f6398fad640 1 -- 172.21.15.114:0/1272188247 >> [v2:172.21.15.114:6800/1395693466,v1:172.21.15.114:6801/1395693466] conn(0x7f63800a3db0 msgr2=0x7f63800a6270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T10:26:48.369 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:48.370+0000 7f6398fad640 1 --2- 172.21.15.114:0/1272188247 >> [v2:172.21.15.114:6800/1395693466,v1:172.21.15.114:6801/1395693466] conn(0x7f63800a3db0 0x7f63800a6270 secure :-1 s=READY pgs=57 cs=0 l=1 rev1=1 crypto rx=0x7f637c005fd0 tx=0x7f637c0094d0 comp rx=0 tx=0).stop 2024-04-23T10:26:48.369 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:48.370+0000 7f6398fad640 1 -- 172.21.15.114:0/1272188247 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f63941010a0 msgr2=0x7f63941994e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T10:26:48.369 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:48.370+0000 7f6398fad640 1 --2- 172.21.15.114:0/1272188247 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f63941010a0 0x7f63941994e0 secure :-1 s=READY pgs=463 cs=0 l=1 rev1=1 crypto rx=0x7f638400e9f0 tx=0x7f638400eeb0 comp rx=0 tx=0).stop 2024-04-23T10:26:48.369 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:48.370+0000 7f6398fad640 1 -- 172.21.15.114:0/1272188247 shutdown_connections 2024-04-23T10:26:48.369 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:48.370+0000 7f6398fad640 1 --2- 172.21.15.114:0/1272188247 >> [v2:172.21.15.114:6800/1395693466,v1:172.21.15.114:6801/1395693466] conn(0x7f63800a3db0 0x7f63800a6270 unknown :-1 s=CLOSED pgs=57 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:26:48.369 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:48.370+0000 7f6398fad640 1 --2- 172.21.15.114:0/1272188247 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f639419a000 0x7f63941a4560 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:26:48.369 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:48.370+0000 7f6398fad640 1 --2- 172.21.15.114:0/1272188247 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f6394106170 0x7f6394199a20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:26:48.369 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:48.370+0000 7f6398fad640 1 --2- 172.21.15.114:0/1272188247 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f63941010a0 0x7f63941994e0 unknown :-1 s=CLOSED pgs=463 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:26:48.369 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:48.370+0000 7f6398fad640 1 -- 172.21.15.114:0/1272188247 >> 172.21.15.114:0/1272188247 conn(0x7f6394077f50 msgr2=0x7f63940fefa0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T10:26:48.370 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:48.370+0000 7f6398fad640 1 -- 172.21.15.114:0/1272188247 shutdown_connections 2024-04-23T10:26:48.370 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:48.370+0000 7f6398fad640 1 -- 172.21.15.114:0/1272188247 wait complete. 2024-04-23T10:26:48.379 INFO:journalctl@ceph.mon.a.smithi114.stdout:Apr 23 10:26:48 smithi114 ceph-mon[29255]: pgmap v6: 97 pgs: 97 active+clean; 1.1 MiB data, 376 MiB used, 1.0 TiB / 1.0 TiB avail; 41 KiB/s rd, 90 KiB/s wr, 66 op/s 2024-04-23T10:26:48.379 INFO:journalctl@ceph.mon.a.smithi114.stdout:Apr 23 10:26:48 smithi114 ceph-mon[29255]: from='client.? 172.21.15.114:0/3757603754' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-04-23T10:26:48.379 INFO:journalctl@ceph.mon.a.smithi114.stdout:Apr 23 10:26:48 smithi114 ceph-mon[29255]: from='client.? 172.21.15.114:0/1714696800' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2024-04-23T10:26:48.379 INFO:journalctl@ceph.mon.a.smithi114.stdout:Apr 23 10:26:48 smithi114 ceph-mon[29255]: from='client.? 172.21.15.114:0/866259966' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2024-04-23T10:26:48.380 INFO:tasks.quiescer.fs.[cephfs]:Initializing Quiescer with random seed 93370 2024-04-23T10:26:48.380 DEBUG:tasks.quiescer:Yielding 2024-04-23T10:26:48.380 INFO:teuthology.run_tasks:Running task workunit... 2024-04-23T10:26:48.392 INFO:tasks.workunit:Pulling workunits from ref 0a6c3ed699031b80a2b419e7e795368719871394 2024-04-23T10:26:48.393 INFO:tasks.workunit:Making a separate scratch dir for every client... 2024-04-23T10:26:48.393 INFO:tasks.workunit:timeout=3h 2024-04-23T10:26:48.393 INFO:tasks.workunit:cleanup=True 2024-04-23T10:26:48.393 DEBUG:teuthology.orchestra.run.smithi114:> stat -- /home/ubuntu/cephtest/mnt.0 2024-04-23T10:26:48.393 DEBUG:teuthology.orchestra.run.smithi114:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-23T10:26:48.408 INFO:teuthology.orchestra.run.smithi114.stdout: File: /home/ubuntu/cephtest/mnt.0 2024-04-23T10:26:48.408 INFO:teuthology.orchestra.run.smithi114.stdout: Size: 0 Blocks: 0 IO Block: 65536 directory 2024-04-23T10:26:48.408 INFO:teuthology.orchestra.run.smithi114.stdout:Device: 73h/115d Inode: 1099511628283 Links: 2 2024-04-23T10:26:48.409 INFO:teuthology.orchestra.run.smithi114.stdout:Access: (1777/drwxrwxrwt) Uid: ( 0/ root) Gid: ( 0/ root) 2024-04-23T10:26:48.409 INFO:teuthology.orchestra.run.smithi114.stdout:Context: system_u:object_r:unlabeled_t:s0 2024-04-23T10:26:48.409 INFO:teuthology.orchestra.run.smithi114.stdout:Access: 2024-04-23 10:24:49.716406371 +0000 2024-04-23T10:26:48.409 INFO:teuthology.orchestra.run.smithi114.stdout:Modify: 2024-04-23 10:24:49.716406371 +0000 2024-04-23T10:26:48.409 INFO:teuthology.orchestra.run.smithi114.stdout:Change: 2024-04-23 10:26:38.264075039 +0000 2024-04-23T10:26:48.409 INFO:teuthology.orchestra.run.smithi114.stdout: Birth: 2024-04-23 10:24:49.716406371 +0000 2024-04-23T10:26:48.409 INFO:tasks.workunit:Did not need to create dir /home/ubuntu/cephtest/mnt.0 2024-04-23T10:26:48.409 DEBUG:teuthology.orchestra.run.smithi114:> cd -- /home/ubuntu/cephtest/mnt.0 && sudo install -d -m 0755 --owner=ubuntu -- client.0 2024-04-23T10:26:48.480 DEBUG:teuthology.orchestra.run.smithi114:> rm -rf /home/ubuntu/cephtest/clone.client.0 && git clone https://github.com/ceph/ceph /home/ubuntu/cephtest/clone.client.0 && cd /home/ubuntu/cephtest/clone.client.0 && git checkout 0a6c3ed699031b80a2b419e7e795368719871394 2024-04-23T10:26:48.492 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 23 10:26:48 smithi119 ceph-mon[32608]: pgmap v6: 97 pgs: 97 active+clean; 1.1 MiB data, 376 MiB used, 1.0 TiB / 1.0 TiB avail; 41 KiB/s rd, 90 KiB/s wr, 66 op/s 2024-04-23T10:26:48.492 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 23 10:26:48 smithi119 ceph-mon[32608]: from='client.? 172.21.15.114:0/3757603754' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-04-23T10:26:48.492 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 23 10:26:48 smithi119 ceph-mon[32608]: from='client.? 172.21.15.114:0/1714696800' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2024-04-23T10:26:48.492 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 23 10:26:48 smithi119 ceph-mon[32608]: from='client.? 172.21.15.114:0/866259966' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2024-04-23T10:26:48.496 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:48.495+0000 7fda11976640 1 -- 172.21.15.114:0/1184511065 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fda0c1031b0 msgr2=0x7fda0c103590 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T10:26:48.496 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:48.495+0000 7fda11976640 1 --2- 172.21.15.114:0/1184511065 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fda0c1031b0 0x7fda0c103590 secure :-1 s=READY pgs=464 cs=0 l=1 rev1=1 crypto rx=0x7fda04009900 tx=0x7fda0402a480 comp rx=0 tx=0).stop 2024-04-23T10:26:48.496 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:48.495+0000 7fda11976640 1 -- 172.21.15.114:0/1184511065 shutdown_connections 2024-04-23T10:26:48.496 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:48.495+0000 7fda11976640 1 --2- 172.21.15.114:0/1184511065 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fda0c10c580 0x7fda0c10e9b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:26:48.496 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:48.495+0000 7fda11976640 1 --2- 172.21.15.114:0/1184511065 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fda0c103ad0 0x7fda0c10bfb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:26:48.496 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:48.495+0000 7fda11976640 1 --2- 172.21.15.114:0/1184511065 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fda0c1031b0 0x7fda0c103590 unknown :-1 s=CLOSED pgs=464 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:26:48.496 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:48.495+0000 7fda11976640 1 -- 172.21.15.114:0/1184511065 >> 172.21.15.114:0/1184511065 conn(0x7fda0c0771d0 msgr2=0x7fda0c0775e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T10:26:48.496 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:48.495+0000 7fda11976640 1 -- 172.21.15.114:0/1184511065 shutdown_connections 2024-04-23T10:26:48.496 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:48.495+0000 7fda11976640 1 -- 172.21.15.114:0/1184511065 wait complete. 2024-04-23T10:26:48.496 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:48.496+0000 7fda11976640 1 Processor -- start 2024-04-23T10:26:48.496 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:48.496+0000 7fda11976640 1 -- start start 2024-04-23T10:26:48.497 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:48.496+0000 7fda11976640 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fda0c1031b0 0x7fda0c198750 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T10:26:48.497 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:48.496+0000 7fda11976640 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fda0c103ad0 0x7fda0c198c90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T10:26:48.497 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:48.496+0000 7fda11976640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fda0c10c580 0x7fda0c19d010 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T10:26:48.497 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:48.496+0000 7fda11976640 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x7fda0c1121d0 con 0x7fda0c10c580 2024-04-23T10:26:48.497 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:48.496+0000 7fda11976640 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 -- 0x7fda0c112050 con 0x7fda0c103ad0 2024-04-23T10:26:48.497 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:48.496+0000 7fda11976640 1 -- --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_getmap magic: 0 -- 0x7fda0c112350 con 0x7fda0c1031b0 2024-04-23T10:26:48.497 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:48.497+0000 7fda0b7fe640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fda0c10c580 0x7fda0c19d010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T10:26:48.497 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:48.497+0000 7fda0b7fe640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fda0c10c580 0x7fda0c19d010 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.114:52040/0 (socket says 172.21.15.114:52040) 2024-04-23T10:26:48.497 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:48.497+0000 7fda0b7fe640 1 -- 172.21.15.114:0/3486804119 learned_addr learned my addr 172.21.15.114:0/3486804119 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-04-23T10:26:48.497 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:48.497+0000 7fda0affd640 1 --2- 172.21.15.114:0/3486804119 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fda0c1031b0 0x7fda0c198750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T10:26:48.497 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:48.497+0000 7fda0a7fc640 1 --2- 172.21.15.114:0/3486804119 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fda0c103ad0 0x7fda0c198c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T10:26:48.498 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:48.497+0000 7fda0b7fe640 1 -- 172.21.15.114:0/3486804119 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fda0c1031b0 msgr2=0x7fda0c198750 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T10:26:48.498 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:48.497+0000 7fda0b7fe640 1 --2- 172.21.15.114:0/3486804119 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fda0c1031b0 0x7fda0c198750 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:26:48.498 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:48.497+0000 7fda0b7fe640 1 -- 172.21.15.114:0/3486804119 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fda0c103ad0 msgr2=0x7fda0c198c90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T10:26:48.498 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:48.497+0000 7fda0b7fe640 1 --2- 172.21.15.114:0/3486804119 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fda0c103ad0 0x7fda0c198c90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:26:48.499 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:48.497+0000 7fda0b7fe640 1 -- 172.21.15.114:0/3486804119 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fda0c19d740 con 0x7fda0c10c580 2024-04-23T10:26:48.499 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:48.497+0000 7fda0b7fe640 1 --2- 172.21.15.114:0/3486804119 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fda0c10c580 0x7fda0c19d010 secure :-1 s=READY pgs=465 cs=0 l=1 rev1=1 crypto rx=0x7fd9fc00d720 tx=0x7fd9fc00dbe0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T10:26:48.499 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:48.497+0000 7fda10974640 1 -- 172.21.15.114:0/3486804119 <== mon.0 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd9fc017070 con 0x7fda0c10c580 2024-04-23T10:26:48.499 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:48.497+0000 7fda11976640 1 -- 172.21.15.114:0/3486804119 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fda0c19da30 con 0x7fda0c10c580 2024-04-23T10:26:48.499 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:48.497+0000 7fda10974640 1 -- 172.21.15.114:0/3486804119 <== mon.0 v2:172.21.15.114:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fd9fc00de30 con 0x7fda0c10c580 2024-04-23T10:26:48.499 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:48.497+0000 7fda10974640 1 -- 172.21.15.114:0/3486804119 <== mon.0 v2:172.21.15.114:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd9fc004d00 con 0x7fda0c10c580 2024-04-23T10:26:48.499 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:48.497+0000 7fda11976640 1 -- 172.21.15.114:0/3486804119 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fda0c1a52c0 con 0x7fda0c10c580 2024-04-23T10:26:48.499 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:48.498+0000 7fda11976640 1 -- 172.21.15.114:0/3486804119 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fda0c0691c0 con 0x7fda0c10c580 2024-04-23T10:26:48.499 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:48.499+0000 7fda10974640 1 -- 172.21.15.114:0/3486804119 <== mon.0 v2:172.21.15.114:3300/0 4 ==== mgrmap(e 24) ==== 138823+0+0 (secure 0 0 0) 0x7fd9fc004ea0 con 0x7fda0c10c580 2024-04-23T10:26:48.500 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:48.500+0000 7fda10974640 1 --2- 172.21.15.114:0/3486804119 >> [v2:172.21.15.114:6800/1395693466,v1:172.21.15.114:6801/1395693466] conn(0x7fd9f40a3db0 0x7fd9f40a6270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T10:26:48.500 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:48.500+0000 7fda0affd640 1 --2- 172.21.15.114:0/3486804119 >> [v2:172.21.15.114:6800/1395693466,v1:172.21.15.114:6801/1395693466] conn(0x7fd9f40a3db0 0x7fd9f40a6270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T10:26:48.500 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:48.500+0000 7fda10974640 1 -- 172.21.15.114:0/3486804119 <== mon.0 v2:172.21.15.114:3300/0 5 ==== osd_map(90..90 src has 1..90) ==== 8527+0+0 (secure 0 0 0) 0x7fd9fc0d0570 con 0x7fda0c10c580 2024-04-23T10:26:48.500 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:48.501+0000 7fda0affd640 1 --2- 172.21.15.114:0/3486804119 >> [v2:172.21.15.114:6800/1395693466,v1:172.21.15.114:6801/1395693466] conn(0x7fd9f40a3db0 0x7fd9f40a6270 secure :-1 s=READY pgs=58 cs=0 l=1 rev1=1 crypto rx=0x7fda04009230 tx=0x7fda04000bc0 comp rx=0 tx=0).ready entity=mgr.15291 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T10:26:48.503 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:48.504+0000 7fda10974640 1 -- 172.21.15.114:0/3486804119 <== mon.0 v2:172.21.15.114:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7fd9fc09c450 con 0x7fda0c10c580 2024-04-23T10:26:48.509 INFO:tasks.workunit.client.0.smithi114.stderr:Cloning into '/home/ubuntu/cephtest/clone.client.0'... 2024-04-23T10:26:48.720 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:48.721+0000 7fda11976640 1 -- 172.21.15.114:0/3486804119 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fda0c02d5c0 con 0x7fda0c10c580 2024-04-23T10:26:48.720 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:48.721+0000 7fda10974640 1 -- 172.21.15.114:0/3486804119 <== mon.0 v2:172.21.15.114:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 21 v21) ==== 94+0+11884 (secure 0 0 0) 0x7fd9fc0a0750 con 0x7fda0c10c580 2024-04-23T10:26:48.721 INFO:teuthology.orchestra.run.smithi114.stdout: 2024-04-23T10:26:48.721 INFO:teuthology.orchestra.run.smithi114.stdout:{"epoch":21,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14547,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.114:6835/663493580","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6834","nonce":663493580},{"type":"v1","addr":"172.21.15.114:6835","nonce":663493580}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14565,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.114:6837/1439252154","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6836","nonce":1439252154},{"type":"v1","addr":"172.21.15.114:6837","nonce":1439252154}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5},{"gid":14634,"name":"g","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.114:6839/3849796939","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6838","nonce":3849796939},{"type":"v1","addr":"172.21.15.114:6839","nonce":3849796939}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":11},{"gid":14646,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.114:6841/3544523891","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6840","nonce":3544523891},{"type":"v1","addr":"172.21.15.114:6841","nonce":3544523891}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":12},{"gid":24385,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.204:6833/3866561754","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6832","nonce":3866561754},{"type":"v1","addr":"172.21.15.204:6833","nonce":3866561754}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2},{"gid":24427,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.204:6835/2533793172","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6834","nonce":2533793172},{"type":"v1","addr":"172.21.15.204:6835","nonce":2533793172}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":7}],"filesystems":[{"mdsmap":{"epoch":21,"flags":306,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":true},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-04-23T10:24:43.383158+0000","modified":"2024-04-23T10:26:38.945346+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":3,"in":[0,1,2],"up":{"mds_0":24545,"mds_1":24509,"mds_2":24479},"failed":[],"damaged":[],"stopped":[],"info":{"gid_24439":{"gid":24439,"name":"c","rank":2,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.204:6837/513822179","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6836","nonce":513822179},{"type":"v1","addr":"172.21.15.204:6837","nonce":513822179}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24443":{"gid":24443,"name":"k","rank":1,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.119:6833/783244660","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6832","nonce":783244660},{"type":"v1","addr":"172.21.15.119:6833","nonce":783244660}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24463":{"gid":24463,"name":"i","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.204:6839/3222658893","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6838","nonce":3222658893},{"type":"v1","addr":"172.21.15.204:6839","nonce":3222658893}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24479":{"gid":24479,"name":"e","rank":2,"incarnation":19,"state":"up:active","state_seq":9,"addr":"172.21.15.119:6835/2033916709","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6834","nonce":2033916709},{"type":"v1","addr":"172.21.15.119:6835","nonce":2033916709}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24509":{"gid":24509,"name":"h","rank":1,"incarnation":17,"state":"up:active","state_seq":6,"addr":"172.21.15.119:6837/1872363128","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6836","nonce":1872363128},{"type":"v1","addr":"172.21.15.119:6837","nonce":1872363128}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24545":{"gid":24545,"name":"b","rank":0,"incarnation":14,"state":"up:active","state_seq":3,"addr":"172.21.15.119:6839/937297340","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6838","nonce":937297340},{"type":"v1","addr":"172.21.15.119:6839","nonce":937297340}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":24545,"qdb_cluster":[24479,24509,24545]},"id":1}]} 2024-04-23T10:26:48.721 INFO:teuthology.orchestra.run.smithi114.stderr:dumped fsmap epoch 21 2024-04-23T10:26:48.723 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:48.723+0000 7fda11976640 1 -- 172.21.15.114:0/3486804119 >> [v2:172.21.15.114:6800/1395693466,v1:172.21.15.114:6801/1395693466] conn(0x7fd9f40a3db0 msgr2=0x7fd9f40a6270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T10:26:48.723 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:48.723+0000 7fda11976640 1 --2- 172.21.15.114:0/3486804119 >> [v2:172.21.15.114:6800/1395693466,v1:172.21.15.114:6801/1395693466] conn(0x7fd9f40a3db0 0x7fd9f40a6270 secure :-1 s=READY pgs=58 cs=0 l=1 rev1=1 crypto rx=0x7fda04009230 tx=0x7fda04000bc0 comp rx=0 tx=0).stop 2024-04-23T10:26:48.723 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:48.723+0000 7fda11976640 1 -- 172.21.15.114:0/3486804119 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fda0c10c580 msgr2=0x7fda0c19d010 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T10:26:48.723 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:48.723+0000 7fda11976640 1 --2- 172.21.15.114:0/3486804119 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fda0c10c580 0x7fda0c19d010 secure :-1 s=READY pgs=465 cs=0 l=1 rev1=1 crypto rx=0x7fd9fc00d720 tx=0x7fd9fc00dbe0 comp rx=0 tx=0).stop 2024-04-23T10:26:48.723 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:48.723+0000 7fda11976640 1 -- 172.21.15.114:0/3486804119 shutdown_connections 2024-04-23T10:26:48.723 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:48.723+0000 7fda11976640 1 --2- 172.21.15.114:0/3486804119 >> [v2:172.21.15.114:6800/1395693466,v1:172.21.15.114:6801/1395693466] conn(0x7fd9f40a3db0 0x7fd9f40a6270 unknown :-1 s=CLOSED pgs=58 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:26:48.723 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:48.723+0000 7fda11976640 1 --2- 172.21.15.114:0/3486804119 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fda0c10c580 0x7fda0c19d010 unknown :-1 s=CLOSED pgs=465 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:26:48.723 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:48.723+0000 7fda11976640 1 --2- 172.21.15.114:0/3486804119 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fda0c103ad0 0x7fda0c198c90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:26:48.723 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:48.723+0000 7fda11976640 1 --2- 172.21.15.114:0/3486804119 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fda0c1031b0 0x7fda0c198750 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:26:48.723 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:48.723+0000 7fda11976640 1 -- 172.21.15.114:0/3486804119 >> 172.21.15.114:0/3486804119 conn(0x7fda0c0771d0 msgr2=0x7fda0c0fde10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T10:26:48.723 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:48.724+0000 7fda11976640 1 -- 172.21.15.114:0/3486804119 shutdown_connections 2024-04-23T10:26:48.723 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:48.724+0000 7fda11976640 1 -- 172.21.15.114:0/3486804119 wait complete. 2024-04-23T10:26:48.734 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 21, 'flags': 306, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': True, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': True}, 'ever_allowed_features': 32, 'explicitly_allowed_features': 32, 'created': '2024-04-23T10:24:43.383158+0000', 'modified': '2024-04-23T10:26:38.945346+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 300, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}, 'max_mds': 3, 'in': [0, 1, 2], 'up': {'mds_0': 24545, 'mds_1': 24509, 'mds_2': 24479}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_24439': {'gid': 24439, 'name': 'c', 'rank': 2, 'incarnation': 0, 'state': 'up:standby-replay', 'state_seq': 1, 'addr': '172.21.15.204:6837/513822179', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.204:6836', 'nonce': 513822179}, {'type': 'v1', 'addr': '172.21.15.204:6837', 'nonce': 513822179}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_24443': {'gid': 24443, 'name': 'k', 'rank': 1, 'incarnation': 0, 'state': 'up:standby-replay', 'state_seq': 1, 'addr': '172.21.15.119:6833/783244660', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.119:6832', 'nonce': 783244660}, {'type': 'v1', 'addr': '172.21.15.119:6833', 'nonce': 783244660}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_24463': {'gid': 24463, 'name': 'i', 'rank': 0, 'incarnation': 0, 'state': 'up:standby-replay', 'state_seq': 1, 'addr': '172.21.15.204:6839/3222658893', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.204:6838', 'nonce': 3222658893}, {'type': 'v1', 'addr': '172.21.15.204:6839', 'nonce': 3222658893}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_24479': {'gid': 24479, 'name': 'e', 'rank': 2, 'incarnation': 19, 'state': 'up:active', 'state_seq': 9, 'addr': '172.21.15.119:6835/2033916709', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.119:6834', 'nonce': 2033916709}, {'type': 'v1', 'addr': '172.21.15.119:6835', 'nonce': 2033916709}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_24509': {'gid': 24509, 'name': 'h', 'rank': 1, 'incarnation': 17, 'state': 'up:active', 'state_seq': 6, 'addr': '172.21.15.119:6837/1872363128', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.119:6836', 'nonce': 1872363128}, {'type': 'v1', 'addr': '172.21.15.119:6837', 'nonce': 1872363128}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_24545': {'gid': 24545, 'name': 'b', 'rank': 0, 'incarnation': 14, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.15.119:6839/937297340', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.119:6838', 'nonce': 937297340}, {'type': 'v1', 'addr': '172.21.15.119:6839', 'nonce': 937297340}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}}, 'data_pools': [3], 'metadata_pool': 2, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 24545, 'qdb_cluster': [24479, 24509, 24545]} 2024-04-23T10:26:48.734 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: 3/3 2024-04-23T10:26:48.734 DEBUG:teuthology.orchestra.run.smithi114:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.e status 2024-04-23T10:26:48.876 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:48.874+0000 7f837d9e5640 1 -- 172.21.15.114:0/1312970232 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f8378106150 msgr2=0x7f837810dd20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T10:26:48.876 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:48.874+0000 7f837d9e5640 1 --2- 172.21.15.114:0/1312970232 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f8378106150 0x7f837810dd20 secure :-1 s=READY pgs=466 cs=0 l=1 rev1=1 crypto rx=0x7f8370009ca0 tx=0x7f837002a8c0 comp rx=0 tx=0).stop 2024-04-23T10:26:48.876 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:48.874+0000 7f837d9e5640 1 -- 172.21.15.114:0/1312970232 shutdown_connections 2024-04-23T10:26:48.876 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:48.874+0000 7f837d9e5640 1 --2- 172.21.15.114:0/1312970232 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f8378106150 0x7f837810dd20 unknown :-1 s=CLOSED pgs=466 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:26:48.876 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:48.874+0000 7f837d9e5640 1 --2- 172.21.15.114:0/1312970232 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f8378101a30 0x7f8378105a70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:26:48.876 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:48.874+0000 7f837d9e5640 1 --2- 172.21.15.114:0/1312970232 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f8378101080 0x7f8378101460 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:26:48.876 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:48.874+0000 7f837d9e5640 1 -- 172.21.15.114:0/1312970232 >> 172.21.15.114:0/1312970232 conn(0x7f83780771d0 msgr2=0x7f83780775e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T10:26:48.876 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:48.874+0000 7f837d9e5640 1 -- 172.21.15.114:0/1312970232 shutdown_connections 2024-04-23T10:26:48.876 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:48.874+0000 7f837d9e5640 1 -- 172.21.15.114:0/1312970232 wait complete. 2024-04-23T10:26:48.876 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:48.875+0000 7f837d9e5640 1 Processor -- start 2024-04-23T10:26:48.877 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:48.875+0000 7f837d9e5640 1 -- start start 2024-04-23T10:26:48.877 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:48.875+0000 7f837d9e5640 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f8378101080 0x7f8378198730 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T10:26:48.877 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:48.875+0000 7f837d9e5640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f8378106150 0x7f8378198c70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T10:26:48.877 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:48.875+0000 7f837d9e5640 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f837819cff0 0x7f837819d4a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T10:26:48.877 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:48.875+0000 7f837d9e5640 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x7f83781110c0 con 0x7f8378106150 2024-04-23T10:26:48.877 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:48.875+0000 7f837d9e5640 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 -- 0x7f8378110f40 con 0x7f8378101080 2024-04-23T10:26:48.877 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:48.875+0000 7f837d9e5640 1 -- --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_getmap magic: 0 -- 0x7f8378111240 con 0x7f837819cff0 2024-04-23T10:26:48.877 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:48.875+0000 7f836ffff640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f8378106150 0x7f8378198c70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T10:26:48.877 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:48.875+0000 7f836ffff640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f8378106150 0x7f8378198c70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.114:52050/0 (socket says 172.21.15.114:52050) 2024-04-23T10:26:48.877 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:48.875+0000 7f836ffff640 1 -- 172.21.15.114:0/917495656 learned_addr learned my addr 172.21.15.114:0/917495656 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-04-23T10:26:48.877 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:48.875+0000 7f83777fe640 1 --2- 172.21.15.114:0/917495656 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f837819cff0 0x7f837819d4a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T10:26:48.877 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:48.875+0000 7f8376ffd640 1 --2- 172.21.15.114:0/917495656 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f8378101080 0x7f8378198730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T10:26:48.878 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:48.875+0000 7f836ffff640 1 -- 172.21.15.114:0/917495656 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f837819cff0 msgr2=0x7f837819d4a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T10:26:48.878 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:48.875+0000 7f836ffff640 1 --2- 172.21.15.114:0/917495656 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f837819cff0 0x7f837819d4a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:26:48.878 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:48.875+0000 7f836ffff640 1 -- 172.21.15.114:0/917495656 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f8378101080 msgr2=0x7f8378198730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T10:26:48.878 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:48.875+0000 7f836ffff640 1 --2- 172.21.15.114:0/917495656 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f8378101080 0x7f8378198730 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:26:48.878 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:48.875+0000 7f836ffff640 1 -- 172.21.15.114:0/917495656 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f837819dae0 con 0x7f8378106150 2024-04-23T10:26:48.878 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:48.875+0000 7f83777fe640 1 --2- 172.21.15.114:0/917495656 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f837819cff0 0x7f837819d4a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-23T10:26:48.878 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:48.875+0000 7f836ffff640 1 --2- 172.21.15.114:0/917495656 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f8378106150 0x7f8378198c70 secure :-1 s=READY pgs=467 cs=0 l=1 rev1=1 crypto rx=0x7f836000d740 tx=0x7f836000dc00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T10:26:48.878 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:48.876+0000 7f837c9e3640 1 -- 172.21.15.114:0/917495656 <== mon.0 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8360017070 con 0x7f8378106150 2024-04-23T10:26:48.878 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:48.876+0000 7f837c9e3640 1 -- 172.21.15.114:0/917495656 <== mon.0 v2:172.21.15.114:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f836000de50 con 0x7f8378106150 2024-04-23T10:26:48.878 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:48.876+0000 7f837c9e3640 1 -- 172.21.15.114:0/917495656 <== mon.0 v2:172.21.15.114:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8360004ce0 con 0x7f8378106150 2024-04-23T10:26:48.878 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:48.876+0000 7f837d9e5640 1 -- 172.21.15.114:0/917495656 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f837819ddd0 con 0x7f8378106150 2024-04-23T10:26:48.879 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:48.876+0000 7f837d9e5640 1 -- 172.21.15.114:0/917495656 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f83781a5570 con 0x7f8378106150 2024-04-23T10:26:48.886 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:48.878+0000 7f837c9e3640 1 -- 172.21.15.114:0/917495656 <== mon.0 v2:172.21.15.114:3300/0 4 ==== mgrmap(e 24) ==== 138823+0+0 (secure 0 0 0) 0x7f8360004e80 con 0x7f8378106150 2024-04-23T10:26:48.887 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:48.878+0000 7f83807d9740 1 Processor -- start 2024-04-23T10:26:48.887 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:48.878+0000 7f83807d9740 1 -- start start 2024-04-23T10:26:48.887 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:48.878+0000 7f83807d9740 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x557c22c66db0 0x557c22c67190 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T10:26:48.887 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:48.878+0000 7f83807d9740 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x557c22c67760 0x557c22c71800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T10:26:48.887 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:48.878+0000 7f83807d9740 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x557c22c71dd0 0x557c22c74220 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T10:26:48.887 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:48.878+0000 7f83807d9740 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x557c22bccc50 con 0x557c22c67760 2024-04-23T10:26:48.887 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:48.878+0000 7f83807d9740 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 -- 0x557c22b69830 con 0x557c22c71dd0 2024-04-23T10:26:48.887 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:48.878+0000 7f83807d9740 1 -- --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_getmap magic: 0 -- 0x557c22bc8c10 con 0x557c22c66db0 2024-04-23T10:26:48.887 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:48.878+0000 7f837c9e3640 1 --2- 172.21.15.114:0/917495656 >> [v2:172.21.15.114:6800/1395693466,v1:172.21.15.114:6801/1395693466] conn(0x7f83640a3db0 0x7f83640a6270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T10:26:48.887 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:48.878+0000 7f8376ffd640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x557c22c67760 0x557c22c71800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T10:26:48.887 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:48.878+0000 7f8376ffd640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x557c22c67760 0x557c22c71800 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.114:52052/0 (socket says 172.21.15.114:52052) 2024-04-23T10:26:48.887 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:48.879+0000 7f8376ffd640 1 -- 172.21.15.114:0/1098581843 learned_addr learned my addr 172.21.15.114:0/1098581843 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-04-23T10:26:48.888 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:48.879+0000 7f83777fe640 1 --2- 172.21.15.114:0/1098581843 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x557c22c66db0 0x557c22c67190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T10:26:48.888 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:48.879+0000 7f83777fe640 1 --2- 172.21.15.114:0/917495656 >> [v2:172.21.15.114:6800/1395693466,v1:172.21.15.114:6801/1395693466] conn(0x7f83640a3db0 0x7f83640a6270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T10:26:48.888 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:48.879+0000 7f837c9e3640 1 -- 172.21.15.114:0/917495656 <== mon.0 v2:172.21.15.114:3300/0 5 ==== osd_map(90..90 src has 1..90) ==== 8527+0+0 (secure 0 0 0) 0x7f83600d0380 con 0x7f8378106150 2024-04-23T10:26:48.888 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:48.879+0000 7f836ffff640 1 --2- 172.21.15.114:0/1098581843 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x557c22c71dd0 0x557c22c74220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T10:26:48.888 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:48.879+0000 7f8376ffd640 1 -- 172.21.15.114:0/1098581843 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x557c22c66db0 msgr2=0x557c22c67190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T10:26:48.888 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:48.879+0000 7f8376ffd640 1 --2- 172.21.15.114:0/1098581843 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x557c22c66db0 0x557c22c67190 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:26:48.888 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:48.879+0000 7f8376ffd640 1 -- 172.21.15.114:0/1098581843 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x557c22c71dd0 msgr2=0x557c22c74220 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T10:26:48.888 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:48.879+0000 7f8376ffd640 1 --2- 172.21.15.114:0/1098581843 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x557c22c71dd0 0x557c22c74220 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:26:48.888 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:48.879+0000 7f8376ffd640 1 -- 172.21.15.114:0/1098581843 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x557c228df2d0 con 0x557c22c67760 2024-04-23T10:26:48.888 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:48.879+0000 7f836ffff640 1 --2- 172.21.15.114:0/1098581843 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x557c22c71dd0 0x557c22c74220 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-23T10:26:48.888 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:48.879+0000 7f8376ffd640 1 --2- 172.21.15.114:0/1098581843 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x557c22c67760 0x557c22c71800 secure :-1 s=READY pgs=468 cs=0 l=1 rev1=1 crypto rx=0x7f836800cc60 tx=0x7f836800e720 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d78131a567c64a1 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T10:26:48.888 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:48.879+0000 7f83777fe640 1 --2- 172.21.15.114:0/917495656 >> [v2:172.21.15.114:6800/1395693466,v1:172.21.15.114:6801/1395693466] conn(0x7f83640a3db0 0x7f83640a6270 secure :-1 s=READY pgs=59 cs=0 l=1 rev1=1 crypto rx=0x7f8370009430 tx=0x7f837002a8c0 comp rx=0 tx=0).ready entity=mgr.15291 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T10:26:48.888 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:48.879+0000 7f836dffb640 1 -- 172.21.15.114:0/1098581843 <== mon.0 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8368014070 con 0x557c22c67760 2024-04-23T10:26:48.888 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:48.879+0000 7f836dffb640 1 -- 172.21.15.114:0/1098581843 <== mon.0 v2:172.21.15.114:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f83680047a0 con 0x557c22c67760 2024-04-23T10:26:48.889 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:48.879+0000 7f836dffb640 1 -- 172.21.15.114:0/1098581843 <== mon.0 v2:172.21.15.114:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8368010690 con 0x557c22c67760 2024-04-23T10:26:48.889 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:48.880+0000 7f83807d9740 1 -- 172.21.15.114:0/1098581843 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x557c22c67760 msgr2=0x557c22c71800 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T10:26:48.889 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:48.880+0000 7f83807d9740 1 --2- 172.21.15.114:0/1098581843 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x557c22c67760 0x557c22c71800 secure :-1 s=READY pgs=468 cs=0 l=1 rev1=1 crypto rx=0x7f836800cc60 tx=0x7f836800e720 comp rx=0 tx=0).stop 2024-04-23T10:26:48.889 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:48.880+0000 7f83807d9740 1 -- 172.21.15.114:0/1098581843 shutdown_connections 2024-04-23T10:26:48.889 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:48.880+0000 7f83807d9740 1 --2- 172.21.15.114:0/1098581843 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x557c22c71dd0 0x557c22c74220 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:26:48.889 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:48.880+0000 7f83807d9740 1 --2- 172.21.15.114:0/1098581843 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x557c22c67760 0x557c22c71800 unknown :-1 s=CLOSED pgs=468 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:26:48.889 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:48.880+0000 7f83807d9740 1 --2- 172.21.15.114:0/1098581843 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x557c22c66db0 0x557c22c67190 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:26:48.889 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:48.880+0000 7f83807d9740 1 -- 172.21.15.114:0/1098581843 >> 172.21.15.114:0/1098581843 conn(0x557c22c63070 msgr2=0x557c22c63450 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T10:26:48.889 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:48.880+0000 7f83807d9740 1 -- 172.21.15.114:0/1098581843 shutdown_connections 2024-04-23T10:26:48.889 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:48.880+0000 7f83807d9740 1 -- 172.21.15.114:0/1098581843 wait complete. 2024-04-23T10:26:48.889 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:48.880+0000 7f83807d9740 1 -- start start 2024-04-23T10:26:48.889 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:48.880+0000 7f83807d9740 1 Processor -- start 2024-04-23T10:26:48.890 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:48.881+0000 7f83807d9740 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x557c22c66db0 0x557c22c6ec80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T10:26:48.890 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:48.881+0000 7f83807d9740 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x557c22c67760 0x557c22c6f1c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T10:26:48.890 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:48.881+0000 7f83807d9740 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x557c22c71dd0 0x557c22c6f700 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T10:26:48.890 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:48.881+0000 7f83807d9740 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x557c22c74d40 con 0x557c22c67760 2024-04-23T10:26:48.890 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:48.881+0000 7f83807d9740 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 -- 0x557c22b67470 con 0x557c22c71dd0 2024-04-23T10:26:48.890 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:48.881+0000 7f83807d9740 1 -- --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_getmap magic: 0 -- 0x557c22c74ec0 con 0x557c22c66db0 2024-04-23T10:26:48.890 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:48.881+0000 7f83807d9740 10 client.0 _ll_register_callbacks cb 0x557c22b8cf30 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-23T10:26:48.890 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:48.881+0000 7f836ffff640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x557c22c67760 0x557c22c6f1c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T10:26:48.890 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:48.881+0000 7f836ffff640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x557c22c67760 0x557c22c6f1c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.114:52068/0 (socket says 172.21.15.114:52068) 2024-04-23T10:26:48.890 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:48.881+0000 7f836ffff640 1 -- 172.21.15.114:0/1941317649 learned_addr learned my addr 172.21.15.114:0/1941317649 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-04-23T10:26:48.891 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:48.881+0000 7f836ffff640 1 -- 172.21.15.114:0/1941317649 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x557c22c66db0 msgr2=0x557c22c6ec80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T10:26:48.891 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:48.881+0000 7f8376ffd640 1 --2- 172.21.15.114:0/1941317649 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x557c22c66db0 0x557c22c6ec80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T10:26:48.891 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:48.881+0000 7f836ffff640 1 --2- 172.21.15.114:0/1941317649 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x557c22c66db0 0x557c22c6ec80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:26:48.891 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:48.881+0000 7f836ffff640 1 -- 172.21.15.114:0/1941317649 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x557c22c71dd0 msgr2=0x557c22c6f700 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T10:26:48.891 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:48.881+0000 7f836ffff640 1 --2- 172.21.15.114:0/1941317649 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x557c22c71dd0 0x557c22c6f700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:26:48.891 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:48.881+0000 7f836ffff640 1 -- 172.21.15.114:0/1941317649 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({osdmap=0}) -- 0x557c229a96b0 con 0x557c22c67760 2024-04-23T10:26:48.891 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:48.881+0000 7f8376ffd640 1 --2- 172.21.15.114:0/1941317649 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x557c22c66db0 0x557c22c6ec80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-23T10:26:48.891 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:48.881+0000 7f836ffff640 1 --2- 172.21.15.114:0/1941317649 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x557c22c67760 0x557c22c6f1c0 secure :-1 s=READY pgs=469 cs=0 l=1 rev1=1 crypto rx=0x7f8360098b80 tx=0x7f836009c8b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f3c0c00998002b69 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T10:26:48.891 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:48.881+0000 7f8356ffd640 10 client.0 ms_handle_connect on v2:172.21.15.114:3300/0 2024-04-23T10:26:48.891 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:48.881+0000 7f8356ffd640 1 -- 172.21.15.114:0/1941317649 <== mon.0 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8360020040 con 0x557c22c67760 2024-04-23T10:26:48.892 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:48.881+0000 7f8356ffd640 1 -- 172.21.15.114:0/1941317649 <== mon.0 v2:172.21.15.114:3300/0 2 ==== osd_map(90..90 src has 1..90) ==== 8527+0+0 (secure 0 0 0) 0x7f83600a0170 con 0x557c22c67760 2024-04-23T10:26:48.892 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:48.881+0000 7f83807d9740 1 -- 172.21.15.114:0/1941317649 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f83680047a0 con 0x557c22c67760 2024-04-23T10:26:48.892 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:48.881+0000 7f8356ffd640 1 -- 172.21.15.114:0/1941317649 <== mon.0 v2:172.21.15.114:3300/0 3 ==== mon_get_version_reply(handle=1 version=21) ==== 24+0+0 (secure 0 0 0) 0x7f83600a0560 con 0x557c22c67760 2024-04-23T10:26:48.892 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:48.881+0000 7f83807d9740 10 client.15510 fetch_fsmap learned FSMap version 21 2024-04-23T10:26:48.892 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:48.882+0000 7f83807d9740 1 -- 172.21.15.114:0/1941317649 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({config=0+,fsmap=21,monmap=4+}) -- 0x557c22c80fe0 con 0x557c22c67760 2024-04-23T10:26:48.892 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:48.882+0000 7f8356ffd640 1 -- 172.21.15.114:0/1941317649 <== mon.0 v2:172.21.15.114:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f83680047a0 con 0x557c22c67760 2024-04-23T10:26:48.892 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:48.882+0000 7f8356ffd640 1 -- 172.21.15.114:0/1941317649 <== mon.0 v2:172.21.15.114:3300/0 5 ==== fsmap(e 21) ==== 8338+0+0 (secure 0 0 0) 0x7f8360065020 con 0x557c22c67760 2024-04-23T10:26:48.892 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:48.882+0000 7f83807d9740 10 client.15510 fetch_fsmap finished waiting for FSMap version 21 2024-04-23T10:26:48.892 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:48.882+0000 7f83807d9740 10 client.15510 resolve_mds: resolved name 'e' to daemon mds.e 2024-04-23T10:26:48.892 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:48.882+0000 7f83807d9740 20 client.15510 populate_metadata read hostname 'smithi114' 2024-04-23T10:26:48.892 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:48.882+0000 7f83807d9740 1 --2- 172.21.15.114:0/1941317649 >> [v2:172.21.15.119:6834/2033916709,v1:172.21.15.119:6835/2033916709] conn(0x557c22c836d0 0x557c22c84b40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T10:26:48.892 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:48.882+0000 7f83807d9740 4 client.15510 mds_command: new command op to 24479 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-23T10:26:48.892 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:48.882+0000 7f83807d9740 1 -- 172.21.15.114:0/1941317649 --> [v2:172.21.15.119:6834/2033916709,v1:172.21.15.119:6835/2033916709] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x557c22bda670 con 0x557c22c836d0 2024-04-23T10:26:48.893 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:48.882+0000 7f8376ffd640 1 --2- 172.21.15.114:0/1941317649 >> [v2:172.21.15.119:6834/2033916709,v1:172.21.15.119:6835/2033916709] conn(0x557c22c836d0 0x557c22c84b40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T10:26:48.893 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:48.883+0000 7f8376ffd640 1 --2- 172.21.15.114:0/1941317649 >> [v2:172.21.15.119:6834/2033916709,v1:172.21.15.119:6835/2033916709] conn(0x557c22c836d0 0x557c22c84b40 crc :-1 s=READY pgs=29 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.2 client_cookie=2a18b811d156d809 server_cookie=d3070df99c34e49 in_seq=0 out_seq=0 2024-04-23T10:26:48.893 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:48.883+0000 7f8356ffd640 10 client.15510 ms_handle_connect on v2:172.21.15.119:6834/2033916709 2024-04-23T10:26:48.893 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:48.883+0000 7f8356ffd640 1 -- 172.21.15.114:0/1941317649 <== mds.2 v2:172.21.15.119:6834/2033916709 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x557c22bda670 con 0x557c22c836d0 2024-04-23T10:26:48.893 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:48.883+0000 7f8356ffd640 10 client.15510 handle_command_reply: tid=0 2024-04-23T10:26:48.893 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:48.883+0000 7f83807d9740 1 client.15510 shutdown 2024-04-23T10:26:48.893 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:48.883+0000 7f83807d9740 1 -- 172.21.15.114:0/1941317649 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x557c22c67760 msgr2=0x557c22c6f1c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T10:26:48.893 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:48.883+0000 7f83807d9740 1 --2- 172.21.15.114:0/1941317649 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x557c22c67760 0x557c22c6f1c0 secure :-1 s=READY pgs=469 cs=0 l=1 rev1=1 crypto rx=0x7f8360098b80 tx=0x7f836009c8b0 comp rx=0 tx=0).stop 2024-04-23T10:26:48.893 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:48.883+0000 7f83807d9740 1 -- 172.21.15.114:0/1941317649 shutdown_connections 2024-04-23T10:26:48.893 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:48.883+0000 7f83807d9740 1 --2- 172.21.15.114:0/1941317649 >> [v2:172.21.15.119:6834/2033916709,v1:172.21.15.119:6835/2033916709] conn(0x557c22c836d0 0x557c22c84b40 crc :-1 s=READY pgs=29 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:26:48.893 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:48.883+0000 7f83807d9740 1 --2- 172.21.15.114:0/1941317649 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x557c22c71dd0 0x557c22c6f700 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:26:48.893 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:48.883+0000 7f8356ffd640 0 client.15510 ms_handle_reset on v2:172.21.15.119:6834/2033916709 2024-04-23T10:26:48.893 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:48.883+0000 7f83807d9740 1 --2- 172.21.15.114:0/1941317649 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x557c22c67760 0x557c22c6f1c0 unknown :-1 s=CLOSED pgs=469 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:26:48.893 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:48.883+0000 7f83807d9740 1 --2- 172.21.15.114:0/1941317649 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x557c22c66db0 0x557c22c6ec80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:26:48.894 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:48.884+0000 7f83807d9740 1 -- 172.21.15.114:0/1941317649 >> 172.21.15.114:0/1941317649 conn(0x557c22c63070 msgr2=0x557c22c68ee0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T10:26:48.894 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:48.884+0000 7f83807d9740 1 -- 172.21.15.114:0/1941317649 shutdown_connections 2024-04-23T10:26:48.894 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:48.884+0000 7f83807d9740 1 -- 172.21.15.114:0/1941317649 wait complete. 2024-04-23T10:26:48.894 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:48.884+0000 7f83807d9740 20 client.15510 trim_cache size 0 max 16384 2024-04-23T10:26:48.906 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:48.901+0000 7f83807d9740 1 Processor -- start 2024-04-23T10:26:48.906 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:48.901+0000 7f83807d9740 1 -- start start 2024-04-23T10:26:48.906 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:48.901+0000 7f83807d9740 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x557c22c66db0 0x557c22c6f520 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T10:26:48.906 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:48.901+0000 7f83807d9740 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x557c22c67760 0x557c22c693b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T10:26:48.906 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:48.901+0000 7f83807d9740 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x557c22c71dd0 0x557c22c6bf00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T10:26:48.906 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:48.901+0000 7f83807d9740 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x557c22c82e80 con 0x557c22c66db0 2024-04-23T10:26:48.906 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:48.902+0000 7f83807d9740 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 -- 0x557c22c83550 con 0x557c22c67760 2024-04-23T10:26:48.906 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:48.902+0000 7f83807d9740 1 -- --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_getmap magic: 0 -- 0x557c22c83c20 con 0x557c22c71dd0 2024-04-23T10:26:48.907 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:48.902+0000 7f8376ffd640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x557c22c66db0 0x557c22c6f520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T10:26:48.907 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:48.902+0000 7f8376ffd640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x557c22c66db0 0x557c22c6f520 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.114:52084/0 (socket says 172.21.15.114:52084) 2024-04-23T10:26:48.907 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:48.902+0000 7f8376ffd640 1 -- 172.21.15.114:0/309965171 learned_addr learned my addr 172.21.15.114:0/309965171 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-04-23T10:26:48.907 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:48.902+0000 7f836ffff640 1 --2- 172.21.15.114:0/309965171 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x557c22c67760 0x557c22c693b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T10:26:48.907 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:48.902+0000 7f8376ffd640 1 -- 172.21.15.114:0/309965171 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x557c22c71dd0 msgr2=0x557c22c6bf00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T10:26:48.907 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:48.902+0000 7f8376ffd640 1 --2- 172.21.15.114:0/309965171 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x557c22c71dd0 0x557c22c6bf00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:26:48.907 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:48.902+0000 7f8376ffd640 1 -- 172.21.15.114:0/309965171 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x557c22c67760 msgr2=0x557c22c693b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T10:26:48.907 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:48.902+0000 7f8376ffd640 1 --2- 172.21.15.114:0/309965171 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x557c22c67760 0x557c22c693b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:26:48.907 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:48.902+0000 7f8376ffd640 1 -- 172.21.15.114:0/309965171 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x557c22c62ea0 con 0x557c22c66db0 2024-04-23T10:26:48.907 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:48.902+0000 7f836ffff640 1 --2- 172.21.15.114:0/309965171 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x557c22c67760 0x557c22c693b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-23T10:26:48.907 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:48.902+0000 7f8376ffd640 1 --2- 172.21.15.114:0/309965171 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x557c22c66db0 0x557c22c6f520 secure :-1 s=READY pgs=470 cs=0 l=1 rev1=1 crypto rx=0x7f836800f010 tx=0x7f836800c920 comp rx=0 tx=0).ready entity=mon.0 client_cookie=fb638d3c9c13b0d server_cookie=0 in_seq=0 out_seq=0 2024-04-23T10:26:48.907 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:48.902+0000 7f836d7fa640 1 -- 172.21.15.114:0/309965171 <== mon.0 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8368014040 con 0x557c22c66db0 2024-04-23T10:26:48.907 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:48.902+0000 7f836d7fa640 1 -- 172.21.15.114:0/309965171 <== mon.0 v2:172.21.15.114:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f836801a2c0 con 0x557c22c66db0 2024-04-23T10:26:48.907 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:48.902+0000 7f836d7fa640 1 -- 172.21.15.114:0/309965171 <== mon.0 v2:172.21.15.114:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8368003d10 con 0x557c22c66db0 2024-04-23T10:26:48.907 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:48.903+0000 7f83807d9740 1 -- 172.21.15.114:0/309965171 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x557c22c66db0 msgr2=0x557c22c6f520 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T10:26:48.908 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:48.903+0000 7f83807d9740 1 --2- 172.21.15.114:0/309965171 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x557c22c66db0 0x557c22c6f520 secure :-1 s=READY pgs=470 cs=0 l=1 rev1=1 crypto rx=0x7f836800f010 tx=0x7f836800c920 comp rx=0 tx=0).stop 2024-04-23T10:26:48.908 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:48.903+0000 7f83807d9740 1 -- 172.21.15.114:0/309965171 shutdown_connections 2024-04-23T10:26:48.908 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:48.903+0000 7f83807d9740 1 --2- 172.21.15.114:0/309965171 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x557c22c71dd0 0x557c22c6bf00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:26:48.908 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:48.903+0000 7f83807d9740 1 --2- 172.21.15.114:0/309965171 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x557c22c67760 0x557c22c693b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:26:48.908 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:48.903+0000 7f83807d9740 1 --2- 172.21.15.114:0/309965171 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x557c22c66db0 0x557c22c6f520 unknown :-1 s=CLOSED pgs=470 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:26:48.908 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:48.903+0000 7f83807d9740 1 -- 172.21.15.114:0/309965171 >> 172.21.15.114:0/309965171 conn(0x557c22c63070 msgr2=0x557c22c6d550 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T10:26:48.908 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:48.903+0000 7f83807d9740 1 -- 172.21.15.114:0/309965171 shutdown_connections 2024-04-23T10:26:48.908 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:48.903+0000 7f83807d9740 1 -- 172.21.15.114:0/309965171 wait complete. 2024-04-23T10:26:48.908 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:48.903+0000 7f83807d9740 1 -- start start 2024-04-23T10:26:48.908 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:48.903+0000 7f83807d9740 1 Processor -- start 2024-04-23T10:26:48.908 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:48.903+0000 7f83807d9740 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x557c22c66db0 0x557c22c8d1b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T10:26:48.908 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:48.903+0000 7f83807d9740 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x557c22c67760 0x557c22c6b6e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T10:26:48.908 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:48.903+0000 7f83807d9740 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x557c22c71dd0 0x557c22c6bc20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T10:26:48.908 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:48.903+0000 7f83807d9740 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x557c22c849c0 con 0x557c22c67760 2024-04-23T10:26:48.908 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:48.903+0000 7f83807d9740 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 -- 0x557c22c842f0 con 0x557c22c66db0 2024-04-23T10:26:48.908 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:48.903+0000 7f83807d9740 1 -- --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_getmap magic: 0 -- 0x557c22c85090 con 0x557c22c71dd0 2024-04-23T10:26:48.909 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:48.904+0000 7f83807d9740 10 client.0 _ll_register_callbacks cb 0x7f83600cd8a0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-23T10:26:48.909 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:48.904+0000 7f836ffff640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x557c22c67760 0x557c22c6b6e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T10:26:48.909 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:48.904+0000 7f836ffff640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x557c22c67760 0x557c22c6b6e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.114:52088/0 (socket says 172.21.15.114:52088) 2024-04-23T10:26:48.909 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:48.904+0000 7f836ffff640 1 -- 172.21.15.114:0/1679603141 learned_addr learned my addr 172.21.15.114:0/1679603141 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-04-23T10:26:48.909 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:48.904+0000 7f8376ffd640 1 --2- 172.21.15.114:0/1679603141 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x557c22c66db0 0x557c22c8d1b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T10:26:48.909 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:48.904+0000 7f83777fe640 1 --2- 172.21.15.114:0/1679603141 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x557c22c71dd0 0x557c22c6bc20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T10:26:48.909 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:48.904+0000 7f836ffff640 1 -- 172.21.15.114:0/1679603141 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x557c22c71dd0 msgr2=0x557c22c6bc20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T10:26:48.909 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:48.904+0000 7f836ffff640 1 --2- 172.21.15.114:0/1679603141 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x557c22c71dd0 0x557c22c6bc20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:26:48.909 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:48.904+0000 7f836ffff640 1 -- 172.21.15.114:0/1679603141 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x557c22c66db0 msgr2=0x557c22c8d1b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T10:26:48.909 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:48.904+0000 7f836ffff640 1 --2- 172.21.15.114:0/1679603141 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x557c22c66db0 0x557c22c8d1b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:26:48.909 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:48.904+0000 7f836ffff640 1 -- 172.21.15.114:0/1679603141 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({osdmap=0}) -- 0x557c22c6c400 con 0x557c22c67760 2024-04-23T10:26:48.909 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:48.904+0000 7f836ffff640 1 --2- 172.21.15.114:0/1679603141 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x557c22c67760 0x557c22c6b6e0 secure :-1 s=READY pgs=471 cs=0 l=1 rev1=1 crypto rx=0x7f83600cdaf0 tx=0x7f8360098a30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6bdc0962c669d4da server_cookie=0 in_seq=0 out_seq=0 2024-04-23T10:26:48.909 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:48.904+0000 7f8356ffd640 10 client.0 ms_handle_connect on v2:172.21.15.114:3300/0 2024-04-23T10:26:48.909 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:48.904+0000 7f8356ffd640 1 -- 172.21.15.114:0/1679603141 <== mon.0 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8360020070 con 0x557c22c67760 2024-04-23T10:26:48.909 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:48.904+0000 7f8356ffd640 1 -- 172.21.15.114:0/1679603141 <== mon.0 v2:172.21.15.114:3300/0 2 ==== osd_map(90..90 src has 1..90) ==== 8527+0+0 (secure 0 0 0) 0x7f836009b600 con 0x557c22c67760 2024-04-23T10:26:48.910 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:48.904+0000 7f83807d9740 1 -- 172.21.15.114:0/1679603141 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f836801a2c0 con 0x557c22c67760 2024-04-23T10:26:48.910 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:48.904+0000 7f8356ffd640 1 -- 172.21.15.114:0/1679603141 <== mon.0 v2:172.21.15.114:3300/0 3 ==== mon_get_version_reply(handle=1 version=21) ==== 24+0+0 (secure 0 0 0) 0x7f836009bcf0 con 0x557c22c67760 2024-04-23T10:26:48.910 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:48.904+0000 7f83807d9740 10 client.15516 fetch_fsmap learned FSMap version 21 2024-04-23T10:26:48.910 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:48.904+0000 7f83807d9740 1 -- 172.21.15.114:0/1679603141 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({config=0+,fsmap=21,monmap=4+}) -- 0x557c22ce9680 con 0x557c22c67760 2024-04-23T10:26:48.910 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:48.904+0000 7f8356ffd640 1 -- 172.21.15.114:0/1679603141 <== mon.0 v2:172.21.15.114:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f836801a2c0 con 0x557c22c67760 2024-04-23T10:26:48.910 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:48.904+0000 7f8356ffd640 1 -- 172.21.15.114:0/1679603141 <== mon.0 v2:172.21.15.114:3300/0 5 ==== fsmap(e 21) ==== 8338+0+0 (secure 0 0 0) 0x7f8360065070 con 0x557c22c67760 2024-04-23T10:26:48.910 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:48.905+0000 7f83807d9740 10 client.15516 fetch_fsmap finished waiting for FSMap version 21 2024-04-23T10:26:48.910 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:48.905+0000 7f83807d9740 10 client.15516 resolve_mds: resolved name 'e' to daemon mds.e 2024-04-23T10:26:48.910 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:48.905+0000 7f83807d9740 20 client.15516 populate_metadata read hostname 'smithi114' 2024-04-23T10:26:48.910 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:48.905+0000 7f83807d9740 1 --2- 172.21.15.114:0/1679603141 >> [v2:172.21.15.119:6834/2033916709,v1:172.21.15.119:6835/2033916709] conn(0x557c22cee1c0 0x557c22ceced0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T10:26:48.910 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:48.905+0000 7f83807d9740 4 client.15516 mds_command: new command op to 24479 tid=0 multi_id=0 [{"prefix": "status"}] 2024-04-23T10:26:48.910 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:48.905+0000 7f83807d9740 1 -- 172.21.15.114:0/1679603141 --> [v2:172.21.15.119:6834/2033916709,v1:172.21.15.119:6835/2033916709] -- command(tid 0: {"prefix": "status"}) -- 0x557c22bd44e0 con 0x557c22cee1c0 2024-04-23T10:26:48.910 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:48.905+0000 7f8376ffd640 1 --2- 172.21.15.114:0/1679603141 >> [v2:172.21.15.119:6834/2033916709,v1:172.21.15.119:6835/2033916709] conn(0x557c22cee1c0 0x557c22ceced0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T10:26:48.910 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:48.905+0000 7f8356ffd640 10 client.15516 ms_handle_connect on v2:172.21.15.119:6834/2033916709 2024-04-23T10:26:48.910 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:48.905+0000 7f8376ffd640 1 --2- 172.21.15.114:0/1679603141 >> [v2:172.21.15.119:6834/2033916709,v1:172.21.15.119:6835/2033916709] conn(0x557c22cee1c0 0x557c22ceced0 crc :-1 s=READY pgs=30 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.2 client_cookie=492d0ed5b59edbdf server_cookie=33fd6f97375d80ec in_seq=0 out_seq=0 2024-04-23T10:26:48.911 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:48.905+0000 7f8356ffd640 1 -- 172.21.15.114:0/1679603141 <== mds.2 v2:172.21.15.119:6834/2033916709 1 ==== command_reply(tid 0: 0 ) ==== 8+0+327 (crc 0 0 0) 0x557c22bd44e0 con 0x557c22cee1c0 2024-04-23T10:26:48.911 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:48.905+0000 7f8356ffd640 10 client.15516 handle_command_reply: tid=0 2024-04-23T10:26:48.911 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:48.905+0000 7f83807d9740 1 client.15516 shutdown 2024-04-23T10:26:48.911 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:48.906+0000 7f83807d9740 1 -- 172.21.15.114:0/1679603141 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x557c22c67760 msgr2=0x557c22c6b6e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T10:26:48.911 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:48.906+0000 7f83807d9740 1 --2- 172.21.15.114:0/1679603141 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x557c22c67760 0x557c22c6b6e0 secure :-1 s=READY pgs=471 cs=0 l=1 rev1=1 crypto rx=0x7f83600cdaf0 tx=0x7f8360098a30 comp rx=0 tx=0).stop 2024-04-23T10:26:48.911 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:48.906+0000 7f83807d9740 1 -- 172.21.15.114:0/1679603141 shutdown_connections 2024-04-23T10:26:48.911 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:48.906+0000 7f83807d9740 1 --2- 172.21.15.114:0/1679603141 >> [v2:172.21.15.119:6834/2033916709,v1:172.21.15.119:6835/2033916709] conn(0x557c22cee1c0 0x557c22ceced0 crc :-1 s=READY pgs=30 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:26:48.911 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:48.906+0000 7f83807d9740 1 --2- 172.21.15.114:0/1679603141 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x557c22c71dd0 0x557c22c6bc20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:26:48.911 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:48.906+0000 7f8356ffd640 0 client.15516 ms_handle_reset on v2:172.21.15.119:6834/2033916709 2024-04-23T10:26:48.911 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:48.906+0000 7f83807d9740 1 --2- 172.21.15.114:0/1679603141 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x557c22c67760 0x557c22c6b6e0 unknown :-1 s=CLOSED pgs=471 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:26:48.911 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:48.906+0000 7f83807d9740 1 --2- 172.21.15.114:0/1679603141 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x557c22c66db0 0x557c22c8d1b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:26:48.911 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:48.906+0000 7f83807d9740 1 -- 172.21.15.114:0/1679603141 >> 172.21.15.114:0/1679603141 conn(0x557c22c63070 msgr2=0x557c22c61940 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T10:26:48.911 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:48.906+0000 7f83807d9740 1 -- 172.21.15.114:0/1679603141 shutdown_connections 2024-04-23T10:26:48.911 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:48.906+0000 7f83807d9740 1 -- 172.21.15.114:0/1679603141 wait complete. 2024-04-23T10:26:48.911 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:48.906+0000 7f83807d9740 20 client.15516 trim_cache size 0 max 16384 2024-04-23T10:26:48.912 INFO:teuthology.orchestra.run.smithi114.stdout:{ 2024-04-23T10:26:48.912 INFO:teuthology.orchestra.run.smithi114.stdout: "cluster_fsid": "aa240572-0159-11ef-bc93-c7b262605968", 2024-04-23T10:26:48.912 INFO:teuthology.orchestra.run.smithi114.stdout: "whoami": 2, 2024-04-23T10:26:48.912 INFO:teuthology.orchestra.run.smithi114.stdout: "id": 24479, 2024-04-23T10:26:48.912 INFO:teuthology.orchestra.run.smithi114.stdout: "want_state": "up:active", 2024-04-23T10:26:48.912 INFO:teuthology.orchestra.run.smithi114.stdout: "state": "up:active", 2024-04-23T10:26:48.912 INFO:teuthology.orchestra.run.smithi114.stdout: "fs_name": "cephfs", 2024-04-23T10:26:48.912 INFO:teuthology.orchestra.run.smithi114.stdout: "rank_uptime": 121.561397645, 2024-04-23T10:26:48.912 INFO:teuthology.orchestra.run.smithi114.stdout: "mdsmap_epoch": 21, 2024-04-23T10:26:48.912 INFO:teuthology.orchestra.run.smithi114.stdout: "osdmap_epoch": 90, 2024-04-23T10:26:48.912 INFO:teuthology.orchestra.run.smithi114.stdout: "osdmap_epoch_barrier": 89, 2024-04-23T10:26:48.912 INFO:teuthology.orchestra.run.smithi114.stdout: "uptime": 149.76460894499999 2024-04-23T10:26:48.912 INFO:teuthology.orchestra.run.smithi114.stdout:} 2024-04-23T10:26:48.913 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:48.906+0000 7f837d9e5640 1 -- 172.21.15.114:0/917495656 >> [v2:172.21.15.114:6800/1395693466,v1:172.21.15.114:6801/1395693466] conn(0x7f83640a3db0 msgr2=0x7f83640a6270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T10:26:48.913 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:48.906+0000 7f837d9e5640 1 --2- 172.21.15.114:0/917495656 >> [v2:172.21.15.114:6800/1395693466,v1:172.21.15.114:6801/1395693466] conn(0x7f83640a3db0 0x7f83640a6270 secure :-1 s=READY pgs=59 cs=0 l=1 rev1=1 crypto rx=0x7f8370009430 tx=0x7f837002a8c0 comp rx=0 tx=0).stop 2024-04-23T10:26:48.913 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:48.906+0000 7f837d9e5640 1 -- 172.21.15.114:0/917495656 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f8378106150 msgr2=0x7f8378198c70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T10:26:48.913 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:48.906+0000 7f837d9e5640 1 --2- 172.21.15.114:0/917495656 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f8378106150 0x7f8378198c70 secure :-1 s=READY pgs=467 cs=0 l=1 rev1=1 crypto rx=0x7f836000d740 tx=0x7f836000dc00 comp rx=0 tx=0).stop 2024-04-23T10:26:48.913 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:48.906+0000 7f837d9e5640 1 -- 172.21.15.114:0/917495656 shutdown_connections 2024-04-23T10:26:48.913 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:48.906+0000 7f837d9e5640 1 --2- 172.21.15.114:0/917495656 >> [v2:172.21.15.114:6800/1395693466,v1:172.21.15.114:6801/1395693466] conn(0x7f83640a3db0 0x7f83640a6270 unknown :-1 s=CLOSED pgs=59 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:26:48.913 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:48.906+0000 7f837d9e5640 1 --2- 172.21.15.114:0/917495656 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f837819cff0 0x7f837819d4a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:26:48.914 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:48.906+0000 7f837d9e5640 1 --2- 172.21.15.114:0/917495656 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f8378106150 0x7f8378198c70 unknown :-1 s=CLOSED pgs=467 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:26:48.914 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:48.906+0000 7f837d9e5640 1 --2- 172.21.15.114:0/917495656 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f8378101080 0x7f8378198730 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:26:48.914 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:48.906+0000 7f837d9e5640 1 -- 172.21.15.114:0/917495656 >> 172.21.15.114:0/917495656 conn(0x7f83780771d0 msgr2=0x7f8378104d90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T10:26:48.914 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:48.906+0000 7f837d9e5640 1 -- 172.21.15.114:0/917495656 shutdown_connections 2024-04-23T10:26:48.914 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:48.906+0000 7f837d9e5640 1 -- 172.21.15.114:0/917495656 wait complete. 2024-04-23T10:26:48.916 DEBUG:teuthology.orchestra.run.smithi114:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.h status 2024-04-23T10:26:49.016 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:49.016+0000 7f2a83101640 1 -- 172.21.15.114:0/3437739925 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f2a7c105ee0 msgr2=0x7f2a7c1062c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T10:26:49.017 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:49.016+0000 7f2a83101640 1 --2- 172.21.15.114:0/3437739925 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f2a7c105ee0 0x7f2a7c1062c0 secure :-1 s=READY pgs=472 cs=0 l=1 rev1=1 crypto rx=0x7f2a78009900 tx=0x7f2a7802a400 comp rx=0 tx=0).stop 2024-04-23T10:26:49.017 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:49.016+0000 7f2a83101640 1 -- 172.21.15.114:0/3437739925 shutdown_connections 2024-04-23T10:26:49.017 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:49.016+0000 7f2a83101640 1 --2- 172.21.15.114:0/3437739925 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f2a7c101b60 0x7f2a7c10dc90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:26:49.017 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:49.016+0000 7f2a83101640 1 --2- 172.21.15.114:0/3437739925 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f2a7c1011a0 0x7f2a7c101620 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:26:49.017 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:49.016+0000 7f2a83101640 1 --2- 172.21.15.114:0/3437739925 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f2a7c105ee0 0x7f2a7c1062c0 unknown :-1 s=CLOSED pgs=472 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:26:49.017 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:49.016+0000 7f2a83101640 1 -- 172.21.15.114:0/3437739925 >> 172.21.15.114:0/3437739925 conn(0x7f2a7c076c10 msgr2=0x7f2a7c077020 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T10:26:49.017 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:49.016+0000 7f2a83101640 1 -- 172.21.15.114:0/3437739925 shutdown_connections 2024-04-23T10:26:49.017 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:49.016+0000 7f2a83101640 1 -- 172.21.15.114:0/3437739925 wait complete. 2024-04-23T10:26:49.017 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:49.016+0000 7f2a83101640 1 Processor -- start 2024-04-23T10:26:49.017 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:49.016+0000 7f2a83101640 1 -- start start 2024-04-23T10:26:49.017 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:49.017+0000 7f2a83101640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f2a7c1011a0 0x7f2a7c198790 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T10:26:49.017 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:49.017+0000 7f2a83101640 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f2a7c101b60 0x7f2a7c198cd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T10:26:49.017 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:49.017+0000 7f2a83101640 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f2a7c105ee0 0x7f2a7c19bfa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T10:26:49.017 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:49.017+0000 7f2a83101640 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x7f2a7c112270 con 0x7f2a7c1011a0 2024-04-23T10:26:49.018 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:49.017+0000 7f2a83101640 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 -- 0x7f2a7c1120f0 con 0x7f2a7c101b60 2024-04-23T10:26:49.018 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:49.017+0000 7f2a83101640 1 -- --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_getmap magic: 0 -- 0x7f2a7c1123f0 con 0x7f2a7c105ee0 2024-04-23T10:26:49.018 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:49.017+0000 7f2a80e9d640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f2a7c1011a0 0x7f2a7c198790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T10:26:49.018 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:49.017+0000 7f2a80e9d640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f2a7c1011a0 0x7f2a7c198790 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.114:52112/0 (socket says 172.21.15.114:52112) 2024-04-23T10:26:49.018 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:49.017+0000 7f2a80e9d640 1 -- 172.21.15.114:0/2450587483 learned_addr learned my addr 172.21.15.114:0/2450587483 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-04-23T10:26:49.018 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:49.017+0000 7f2a80e9d640 1 -- 172.21.15.114:0/2450587483 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f2a7c105ee0 msgr2=0x7f2a7c19bfa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T10:26:49.018 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:49.017+0000 7f2a80e9d640 1 --2- 172.21.15.114:0/2450587483 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f2a7c105ee0 0x7f2a7c19bfa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:26:49.018 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:49.017+0000 7f2a80e9d640 1 -- 172.21.15.114:0/2450587483 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f2a7c101b60 msgr2=0x7f2a7c198cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T10:26:49.018 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:49.017+0000 7f2a80e9d640 1 --2- 172.21.15.114:0/2450587483 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f2a7c101b60 0x7f2a7c198cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:26:49.018 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:49.017+0000 7f2a80e9d640 1 -- 172.21.15.114:0/2450587483 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f2a7c19c6d0 con 0x7f2a7c1011a0 2024-04-23T10:26:49.018 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:49.017+0000 7f2a80e9d640 1 --2- 172.21.15.114:0/2450587483 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f2a7c1011a0 0x7f2a7c198790 secure :-1 s=READY pgs=473 cs=0 l=1 rev1=1 crypto rx=0x7f2a78005940 tx=0x7f2a780042f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T10:26:49.018 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:49.017+0000 7f2a71ffb640 1 -- 172.21.15.114:0/2450587483 <== mon.0 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2a7803f030 con 0x7f2a7c1011a0 2024-04-23T10:26:49.018 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:49.017+0000 7f2a71ffb640 1 -- 172.21.15.114:0/2450587483 <== mon.0 v2:172.21.15.114:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f2a78003cb0 con 0x7f2a7c1011a0 2024-04-23T10:26:49.018 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:49.017+0000 7f2a83101640 1 -- 172.21.15.114:0/2450587483 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f2a7c19c960 con 0x7f2a7c1011a0 2024-04-23T10:26:49.018 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:49.017+0000 7f2a71ffb640 1 -- 172.21.15.114:0/2450587483 <== mon.0 v2:172.21.15.114:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2a78035070 con 0x7f2a7c1011a0 2024-04-23T10:26:49.018 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:49.018+0000 7f2a83101640 1 -- 172.21.15.114:0/2450587483 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f2a7c1a5330 con 0x7f2a7c1011a0 2024-04-23T10:26:49.021 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:49.020+0000 7f2a85e18740 1 Processor -- start 2024-04-23T10:26:49.021 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:49.020+0000 7f2a71ffb640 1 -- 172.21.15.114:0/2450587483 <== mon.0 v2:172.21.15.114:3300/0 4 ==== mgrmap(e 24) ==== 138823+0+0 (secure 0 0 0) 0x7f2a78042030 con 0x7f2a7c1011a0 2024-04-23T10:26:49.021 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:49.020+0000 7f2a85e18740 1 -- start start 2024-04-23T10:26:49.021 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:49.021+0000 7f2a85e18740 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x555a33353db0 0x555a33354190 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T10:26:49.021 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:49.021+0000 7f2a85e18740 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x555a33354760 0x555a3335e800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T10:26:49.021 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:49.021+0000 7f2a85e18740 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x555a3335edd0 0x555a33361220 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T10:26:49.021 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:49.021+0000 7f2a85e18740 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x555a332b9c50 con 0x555a33353db0 2024-04-23T10:26:49.022 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:49.021+0000 7f2a85e18740 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 -- 0x555a33256830 con 0x555a3335edd0 2024-04-23T10:26:49.022 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:49.021+0000 7f2a85e18740 1 -- --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_getmap magic: 0 -- 0x555a332b5c10 con 0x555a33354760 2024-04-23T10:26:49.022 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:49.021+0000 7f2a8169e640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x555a33353db0 0x555a33354190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T10:26:49.022 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:49.021+0000 7f2a8169e640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x555a33353db0 0x555a33354190 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.114:52128/0 (socket says 172.21.15.114:52128) 2024-04-23T10:26:49.022 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:49.021+0000 7f2a8169e640 1 -- 172.21.15.114:0/1043512702 learned_addr learned my addr 172.21.15.114:0/1043512702 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-04-23T10:26:49.022 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:49.021+0000 7f2a73fff640 1 --2- 172.21.15.114:0/1043512702 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x555a3335edd0 0x555a33361220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T10:26:49.022 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:49.021+0000 7f2a80e9d640 1 --2- 172.21.15.114:0/1043512702 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x555a33354760 0x555a3335e800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T10:26:49.022 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:49.021+0000 7f2a71ffb640 1 --2- 172.21.15.114:0/2450587483 >> [v2:172.21.15.114:6800/1395693466,v1:172.21.15.114:6801/1395693466] conn(0x7f2a640a3db0 0x7f2a640a6270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T10:26:49.022 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:49.021+0000 7f2a8169e640 1 -- 172.21.15.114:0/1043512702 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x555a33354760 msgr2=0x555a3335e800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T10:26:49.022 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:49.021+0000 7f2a8169e640 1 --2- 172.21.15.114:0/1043512702 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x555a33354760 0x555a3335e800 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:26:49.022 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:49.021+0000 7f2a8169e640 1 -- 172.21.15.114:0/1043512702 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x555a3335edd0 msgr2=0x555a33361220 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T10:26:49.022 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:49.021+0000 7f2a8169e640 1 --2- 172.21.15.114:0/1043512702 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x555a3335edd0 0x555a33361220 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:26:49.022 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:49.021+0000 7f2a8169e640 1 -- 172.21.15.114:0/1043512702 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x555a32fcc2d0 con 0x555a33353db0 2024-04-23T10:26:49.022 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:49.021+0000 7f2a71ffb640 1 -- 172.21.15.114:0/2450587483 <== mon.0 v2:172.21.15.114:3300/0 5 ==== osd_map(90..90 src has 1..90) ==== 8527+0+0 (secure 0 0 0) 0x7f2a780f12f0 con 0x7f2a7c1011a0 2024-04-23T10:26:49.022 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:49.021+0000 7f2a8169e640 1 --2- 172.21.15.114:0/1043512702 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x555a33353db0 0x555a33354190 secure :-1 s=READY pgs=474 cs=0 l=1 rev1=1 crypto rx=0x7f2a74009d10 tx=0x7f2a7400e6c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=dc1c6f2b2d1ed758 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T10:26:49.023 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:49.021+0000 7f2a8169e640 1 --2- 172.21.15.114:0/2450587483 >> [v2:172.21.15.114:6800/1395693466,v1:172.21.15.114:6801/1395693466] conn(0x7f2a640a3db0 0x7f2a640a6270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T10:26:49.023 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:49.021+0000 7f2a62ffd640 1 -- 172.21.15.114:0/1043512702 <== mon.0 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2a74016070 con 0x555a33353db0 2024-04-23T10:26:49.023 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:49.021+0000 7f2a62ffd640 1 -- 172.21.15.114:0/1043512702 <== mon.0 v2:172.21.15.114:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f2a74012cf0 con 0x555a33353db0 2024-04-23T10:26:49.023 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:49.021+0000 7f2a62ffd640 1 -- 172.21.15.114:0/1043512702 <== mon.0 v2:172.21.15.114:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2a740103d0 con 0x555a33353db0 2024-04-23T10:26:49.023 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:49.022+0000 7f2a85e18740 1 -- 172.21.15.114:0/1043512702 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x555a33353db0 msgr2=0x555a33354190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T10:26:49.023 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:49.022+0000 7f2a8169e640 1 --2- 172.21.15.114:0/2450587483 >> [v2:172.21.15.114:6800/1395693466,v1:172.21.15.114:6801/1395693466] conn(0x7f2a640a3db0 0x7f2a640a6270 secure :-1 s=READY pgs=60 cs=0 l=1 rev1=1 crypto rx=0x7f2a74018000 tx=0x7f2a74010af0 comp rx=0 tx=0).ready entity=mgr.15291 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T10:26:49.023 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:49.022+0000 7f2a85e18740 1 --2- 172.21.15.114:0/1043512702 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x555a33353db0 0x555a33354190 secure :-1 s=READY pgs=474 cs=0 l=1 rev1=1 crypto rx=0x7f2a74009d10 tx=0x7f2a7400e6c0 comp rx=0 tx=0).stop 2024-04-23T10:26:49.023 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:49.022+0000 7f2a85e18740 1 -- 172.21.15.114:0/1043512702 shutdown_connections 2024-04-23T10:26:49.023 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:49.022+0000 7f2a85e18740 1 --2- 172.21.15.114:0/1043512702 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x555a3335edd0 0x555a33361220 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:26:49.023 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:49.022+0000 7f2a85e18740 1 --2- 172.21.15.114:0/1043512702 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x555a33354760 0x555a3335e800 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:26:49.023 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:49.022+0000 7f2a85e18740 1 --2- 172.21.15.114:0/1043512702 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x555a33353db0 0x555a33354190 unknown :-1 s=CLOSED pgs=474 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:26:49.023 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:49.022+0000 7f2a85e18740 1 -- 172.21.15.114:0/1043512702 >> 172.21.15.114:0/1043512702 conn(0x555a33350070 msgr2=0x555a33350450 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T10:26:49.023 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:49.022+0000 7f2a85e18740 1 -- 172.21.15.114:0/1043512702 shutdown_connections 2024-04-23T10:26:49.023 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:49.022+0000 7f2a85e18740 1 -- 172.21.15.114:0/1043512702 wait complete. 2024-04-23T10:26:49.023 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:49.022+0000 7f2a85e18740 1 -- start start 2024-04-23T10:26:49.023 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:49.023+0000 7f2a85e18740 1 Processor -- start 2024-04-23T10:26:49.027 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:49.023+0000 7f2a85e18740 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x555a33353db0 0x555a3335bb90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T10:26:49.027 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:49.023+0000 7f2a85e18740 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x555a33354760 0x555a3335c0d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T10:26:49.027 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:49.023+0000 7f2a85e18740 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x555a3335edd0 0x555a3335c610 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T10:26:49.027 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:49.023+0000 7f2a85e18740 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x555a33361d40 con 0x555a33354760 2024-04-23T10:26:49.027 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:49.023+0000 7f2a85e18740 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 -- 0x555a33254470 con 0x555a33353db0 2024-04-23T10:26:49.027 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:49.023+0000 7f2a85e18740 1 -- --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_getmap magic: 0 -- 0x555a33361ec0 con 0x555a3335edd0 2024-04-23T10:26:49.027 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:49.023+0000 7f2a85e18740 10 client.0 _ll_register_callbacks cb 0x555a33279f30 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-23T10:26:49.028 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:49.023+0000 7f2a73fff640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x555a33354760 0x555a3335c0d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T10:26:49.028 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:49.023+0000 7f2a73fff640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x555a33354760 0x555a3335c0d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.114:52140/0 (socket says 172.21.15.114:52140) 2024-04-23T10:26:49.028 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:49.023+0000 7f2a73fff640 1 -- 172.21.15.114:0/260652592 learned_addr learned my addr 172.21.15.114:0/260652592 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-04-23T10:26:49.028 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:49.023+0000 7f2a80e9d640 1 --2- 172.21.15.114:0/260652592 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x555a33353db0 0x555a3335bb90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T10:26:49.028 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:49.023+0000 7f2a73fff640 1 -- 172.21.15.114:0/260652592 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x555a3335edd0 msgr2=0x555a3335c610 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T10:26:49.028 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:49.023+0000 7f2a73fff640 1 --2- 172.21.15.114:0/260652592 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x555a3335edd0 0x555a3335c610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:26:49.028 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:49.023+0000 7f2a73fff640 1 -- 172.21.15.114:0/260652592 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x555a33353db0 msgr2=0x555a3335bb90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T10:26:49.028 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:49.023+0000 7f2a73fff640 1 --2- 172.21.15.114:0/260652592 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x555a33353db0 0x555a3335bb90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:26:49.028 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:49.023+0000 7f2a73fff640 1 -- 172.21.15.114:0/260652592 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({osdmap=0}) -- 0x555a330966b0 con 0x555a33354760 2024-04-23T10:26:49.028 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:49.023+0000 7f2a73fff640 1 --2- 172.21.15.114:0/260652592 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x555a33354760 0x555a3335c0d0 secure :-1 s=READY pgs=475 cs=0 l=1 rev1=1 crypto rx=0x7f2a6c00bd80 tx=0x7f2a6c00f730 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3f7168865f45cec1 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T10:26:49.028 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:49.023+0000 7f2a5bfff640 10 client.0 ms_handle_connect on v2:172.21.15.114:3300/0 2024-04-23T10:26:49.028 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:49.024+0000 7f2a5bfff640 1 -- 172.21.15.114:0/260652592 <== mon.0 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2a6c015070 con 0x555a33354760 2024-04-23T10:26:49.028 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:49.024+0000 7f2a5bfff640 1 -- 172.21.15.114:0/260652592 <== mon.0 v2:172.21.15.114:3300/0 2 ==== osd_map(90..90 src has 1..90) ==== 8527+0+0 (secure 0 0 0) 0x7f2a6c011650 con 0x555a33354760 2024-04-23T10:26:49.028 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:49.024+0000 7f2a85e18740 1 -- 172.21.15.114:0/260652592 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f2a74012cf0 con 0x555a33354760 2024-04-23T10:26:49.028 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:49.024+0000 7f2a5bfff640 1 -- 172.21.15.114:0/260652592 <== mon.0 v2:172.21.15.114:3300/0 3 ==== mon_get_version_reply(handle=1 version=21) ==== 24+0+0 (secure 0 0 0) 0x7f2a6c011850 con 0x555a33354760 2024-04-23T10:26:49.028 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:49.024+0000 7f2a85e18740 10 client.15528 fetch_fsmap learned FSMap version 21 2024-04-23T10:26:49.029 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:49.024+0000 7f2a85e18740 1 -- 172.21.15.114:0/260652592 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({config=0+,fsmap=21,monmap=4+}) -- 0x555a3336def0 con 0x555a33354760 2024-04-23T10:26:49.029 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:49.024+0000 7f2a5bfff640 1 -- 172.21.15.114:0/260652592 <== mon.0 v2:172.21.15.114:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f2a74012cf0 con 0x555a33354760 2024-04-23T10:26:49.029 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:49.024+0000 7f2a5bfff640 1 -- 172.21.15.114:0/260652592 <== mon.0 v2:172.21.15.114:3300/0 5 ==== fsmap(e 21) ==== 8338+0+0 (secure 0 0 0) 0x7f2a6c01abc0 con 0x555a33354760 2024-04-23T10:26:49.029 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:49.024+0000 7f2a85e18740 10 client.15528 fetch_fsmap finished waiting for FSMap version 21 2024-04-23T10:26:49.029 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:49.024+0000 7f2a85e18740 10 client.15528 resolve_mds: resolved name 'h' to daemon mds.h 2024-04-23T10:26:49.029 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:49.024+0000 7f2a85e18740 20 client.15528 populate_metadata read hostname 'smithi114' 2024-04-23T10:26:49.029 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:49.024+0000 7f2a85e18740 1 --2- 172.21.15.114:0/260652592 >> [v2:172.21.15.119:6836/1872363128,v1:172.21.15.119:6837/1872363128] conn(0x555a33370580 0x555a333719f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T10:26:49.029 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:49.024+0000 7f2a85e18740 4 client.15528 mds_command: new command op to 24509 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-23T10:26:49.029 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:49.024+0000 7f2a85e18740 1 -- 172.21.15.114:0/260652592 --> [v2:172.21.15.119:6836/1872363128,v1:172.21.15.119:6837/1872363128] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x555a332c7670 con 0x555a33370580 2024-04-23T10:26:49.029 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:49.024+0000 7f2a80e9d640 1 --2- 172.21.15.114:0/260652592 >> [v2:172.21.15.119:6836/1872363128,v1:172.21.15.119:6837/1872363128] conn(0x555a33370580 0x555a333719f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T10:26:49.029 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:49.025+0000 7f2a80e9d640 1 --2- 172.21.15.114:0/260652592 >> [v2:172.21.15.119:6836/1872363128,v1:172.21.15.119:6837/1872363128] conn(0x555a33370580 0x555a333719f0 crc :-1 s=READY pgs=30 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.1 client_cookie=f87fbe86b36ff90f server_cookie=2f1f4ce2236492f6 in_seq=0 out_seq=0 2024-04-23T10:26:49.029 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:49.025+0000 7f2a5bfff640 10 client.15528 ms_handle_connect on v2:172.21.15.119:6836/1872363128 2024-04-23T10:26:49.029 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:49.025+0000 7f2a5bfff640 1 -- 172.21.15.114:0/260652592 <== mds.1 v2:172.21.15.119:6836/1872363128 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x555a332c7670 con 0x555a33370580 2024-04-23T10:26:49.029 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:49.025+0000 7f2a5bfff640 10 client.15528 handle_command_reply: tid=0 2024-04-23T10:26:49.029 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:49.026+0000 7f2a85e18740 1 client.15528 shutdown 2024-04-23T10:26:49.029 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:49.026+0000 7f2a85e18740 1 -- 172.21.15.114:0/260652592 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x555a33354760 msgr2=0x555a3335c0d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T10:26:49.030 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:49.026+0000 7f2a85e18740 1 --2- 172.21.15.114:0/260652592 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x555a33354760 0x555a3335c0d0 secure :-1 s=READY pgs=475 cs=0 l=1 rev1=1 crypto rx=0x7f2a6c00bd80 tx=0x7f2a6c00f730 comp rx=0 tx=0).stop 2024-04-23T10:26:49.030 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:49.026+0000 7f2a85e18740 1 -- 172.21.15.114:0/260652592 shutdown_connections 2024-04-23T10:26:49.030 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:49.026+0000 7f2a85e18740 1 --2- 172.21.15.114:0/260652592 >> [v2:172.21.15.119:6836/1872363128,v1:172.21.15.119:6837/1872363128] conn(0x555a33370580 0x555a333719f0 crc :-1 s=READY pgs=30 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:26:49.030 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:49.026+0000 7f2a85e18740 1 --2- 172.21.15.114:0/260652592 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x555a3335edd0 0x555a3335c610 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:26:49.030 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:49.026+0000 7f2a85e18740 1 --2- 172.21.15.114:0/260652592 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x555a33354760 0x555a3335c0d0 unknown :-1 s=CLOSED pgs=475 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:26:49.030 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:49.026+0000 7f2a85e18740 1 --2- 172.21.15.114:0/260652592 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x555a33353db0 0x555a3335bb90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:26:49.030 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:49.026+0000 7f2a5bfff640 0 client.15528 ms_handle_reset on v2:172.21.15.119:6836/1872363128 2024-04-23T10:26:49.030 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:49.026+0000 7f2a85e18740 1 -- 172.21.15.114:0/260652592 >> 172.21.15.114:0/260652592 conn(0x555a33350070 msgr2=0x555a33355ee0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T10:26:49.030 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:49.026+0000 7f2a85e18740 1 -- 172.21.15.114:0/260652592 shutdown_connections 2024-04-23T10:26:49.030 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:49.026+0000 7f2a85e18740 1 -- 172.21.15.114:0/260652592 wait complete. 2024-04-23T10:26:49.030 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:49.026+0000 7f2a85e18740 20 client.15528 trim_cache size 0 max 16384 2024-04-23T10:26:49.048 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:49.044+0000 7f2a85e18740 1 Processor -- start 2024-04-23T10:26:49.049 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:49.044+0000 7f2a85e18740 1 -- start start 2024-04-23T10:26:49.049 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:49.044+0000 7f2a85e18740 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x555a33353db0 0x555a33356200 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T10:26:49.049 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:49.044+0000 7f2a85e18740 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x555a33354760 0x555a3335c430 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T10:26:49.049 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:49.044+0000 7f2a85e18740 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x555a3335edd0 0x555a33358df0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T10:26:49.049 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:49.044+0000 7f2a85e18740 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x555a3336fd30 con 0x555a33353db0 2024-04-23T10:26:49.049 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:49.044+0000 7f2a85e18740 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 -- 0x555a33370400 con 0x555a33354760 2024-04-23T10:26:49.049 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:49.044+0000 7f2a85e18740 1 -- --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_getmap magic: 0 -- 0x555a33370ad0 con 0x555a3335edd0 2024-04-23T10:26:49.049 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:49.045+0000 7f2a80e9d640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x555a33353db0 0x555a33356200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T10:26:49.049 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:49.045+0000 7f2a80e9d640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x555a33353db0 0x555a33356200 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.114:52142/0 (socket says 172.21.15.114:52142) 2024-04-23T10:26:49.049 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:49.045+0000 7f2a80e9d640 1 -- 172.21.15.114:0/2583588869 learned_addr learned my addr 172.21.15.114:0/2583588869 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-04-23T10:26:49.049 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:49.045+0000 7f2a8169e640 1 --2- 172.21.15.114:0/2583588869 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x555a3335edd0 0x555a33358df0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T10:26:49.049 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:49.045+0000 7f2a73fff640 1 --2- 172.21.15.114:0/2583588869 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x555a33354760 0x555a3335c430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T10:26:49.049 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:49.045+0000 7f2a80e9d640 1 -- 172.21.15.114:0/2583588869 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x555a3335edd0 msgr2=0x555a33358df0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T10:26:49.049 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:49.045+0000 7f2a80e9d640 1 --2- 172.21.15.114:0/2583588869 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x555a3335edd0 0x555a33358df0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:26:49.050 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:49.045+0000 7f2a80e9d640 1 -- 172.21.15.114:0/2583588869 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x555a33354760 msgr2=0x555a3335c430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T10:26:49.050 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:49.045+0000 7f2a80e9d640 1 --2- 172.21.15.114:0/2583588869 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x555a33354760 0x555a3335c430 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:26:49.050 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:49.045+0000 7f2a80e9d640 1 -- 172.21.15.114:0/2583588869 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x555a3335eb60 con 0x555a33353db0 2024-04-23T10:26:49.050 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:49.045+0000 7f2a73fff640 1 --2- 172.21.15.114:0/2583588869 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x555a33354760 0x555a3335c430 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-23T10:26:49.050 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:49.045+0000 7f2a80e9d640 1 --2- 172.21.15.114:0/2583588869 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x555a33353db0 0x555a33356200 secure :-1 s=READY pgs=476 cs=0 l=1 rev1=1 crypto rx=0x7f2a78006010 tx=0x7f2a7802a400 comp rx=0 tx=0).ready entity=mon.0 client_cookie=77c4c32a4ec0ece2 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T10:26:49.050 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:49.045+0000 7f2a627fc640 1 -- 172.21.15.114:0/2583588869 <== mon.0 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2a78049070 con 0x555a33353db0 2024-04-23T10:26:49.050 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:49.045+0000 7f2a627fc640 1 -- 172.21.15.114:0/2583588869 <== mon.0 v2:172.21.15.114:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f2a780bd820 con 0x555a33353db0 2024-04-23T10:26:49.050 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:49.045+0000 7f2a627fc640 1 -- 172.21.15.114:0/2583588869 <== mon.0 v2:172.21.15.114:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2a780b9790 con 0x555a33353db0 2024-04-23T10:26:49.050 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:49.046+0000 7f2a85e18740 1 -- 172.21.15.114:0/2583588869 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x555a33353db0 msgr2=0x555a33356200 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T10:26:49.050 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:49.046+0000 7f2a85e18740 1 --2- 172.21.15.114:0/2583588869 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x555a33353db0 0x555a33356200 secure :-1 s=READY pgs=476 cs=0 l=1 rev1=1 crypto rx=0x7f2a78006010 tx=0x7f2a7802a400 comp rx=0 tx=0).stop 2024-04-23T10:26:49.050 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:49.046+0000 7f2a85e18740 1 -- 172.21.15.114:0/2583588869 shutdown_connections 2024-04-23T10:26:49.050 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:49.046+0000 7f2a85e18740 1 --2- 172.21.15.114:0/2583588869 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x555a3335edd0 0x555a33358df0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:26:49.050 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:49.046+0000 7f2a85e18740 1 --2- 172.21.15.114:0/2583588869 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x555a33354760 0x555a3335c430 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:26:49.051 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:49.046+0000 7f2a85e18740 1 --2- 172.21.15.114:0/2583588869 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x555a33353db0 0x555a33356200 unknown :-1 s=CLOSED pgs=476 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:26:49.051 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:49.046+0000 7f2a85e18740 1 -- 172.21.15.114:0/2583588869 >> 172.21.15.114:0/2583588869 conn(0x555a33350070 msgr2=0x555a33356e20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T10:26:49.051 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:49.046+0000 7f2a85e18740 1 -- 172.21.15.114:0/2583588869 shutdown_connections 2024-04-23T10:26:49.051 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:49.046+0000 7f2a85e18740 1 -- 172.21.15.114:0/2583588869 wait complete. 2024-04-23T10:26:49.051 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:49.046+0000 7f2a85e18740 1 -- start start 2024-04-23T10:26:49.051 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:49.046+0000 7f2a85e18740 1 Processor -- start 2024-04-23T10:26:49.051 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:49.047+0000 7f2a85e18740 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x555a33353db0 0x555a33373930 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T10:26:49.051 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:49.047+0000 7f2a85e18740 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x555a33354760 0x555a33373e70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T10:26:49.051 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:49.047+0000 7f2a85e18740 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x555a3335edd0 0x555a333743b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T10:26:49.051 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:49.047+0000 7f2a85e18740 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x555a33371870 con 0x555a33353db0 2024-04-23T10:26:49.051 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:49.047+0000 7f2a85e18740 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 -- 0x555a333711a0 con 0x555a3335edd0 2024-04-23T10:26:49.051 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:49.047+0000 7f2a85e18740 1 -- --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_getmap magic: 0 -- 0x555a33371f40 con 0x555a33354760 2024-04-23T10:26:49.051 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:49.047+0000 7f2a85e18740 10 client.0 _ll_register_callbacks cb 0x7f2a6c00b3a0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-23T10:26:49.051 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:49.047+0000 7f2a80e9d640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x555a33353db0 0x555a33373930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T10:26:49.051 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:49.047+0000 7f2a80e9d640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x555a33353db0 0x555a33373930 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.114:52154/0 (socket says 172.21.15.114:52154) 2024-04-23T10:26:49.051 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:49.047+0000 7f2a80e9d640 1 -- 172.21.15.114:0/602210874 learned_addr learned my addr 172.21.15.114:0/602210874 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-04-23T10:26:49.052 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:49.047+0000 7f2a8169e640 1 --2- 172.21.15.114:0/602210874 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x555a3335edd0 0x555a333743b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T10:26:49.052 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:49.047+0000 7f2a73fff640 1 --2- 172.21.15.114:0/602210874 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x555a33354760 0x555a33373e70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T10:26:49.052 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:49.047+0000 7f2a80e9d640 1 -- 172.21.15.114:0/602210874 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x555a33354760 msgr2=0x555a33373e70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T10:26:49.052 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:49.047+0000 7f2a80e9d640 1 --2- 172.21.15.114:0/602210874 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x555a33354760 0x555a33373e70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:26:49.052 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:49.047+0000 7f2a80e9d640 1 -- 172.21.15.114:0/602210874 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x555a3335edd0 msgr2=0x555a333743b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T10:26:49.052 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:49.047+0000 7f2a80e9d640 1 --2- 172.21.15.114:0/602210874 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x555a3335edd0 0x555a333743b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:26:49.052 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:49.047+0000 7f2a80e9d640 1 -- 172.21.15.114:0/602210874 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({osdmap=0}) -- 0x555a33378390 con 0x555a33353db0 2024-04-23T10:26:49.052 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:49.047+0000 7f2a80e9d640 1 --2- 172.21.15.114:0/602210874 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x555a33353db0 0x555a33373930 secure :-1 s=READY pgs=477 cs=0 l=1 rev1=1 crypto rx=0x7f2a780ef040 tx=0x7f2a78032f60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=393441c7c6c519ca server_cookie=0 in_seq=0 out_seq=0 2024-04-23T10:26:49.052 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:49.047+0000 7f2a5bfff640 10 client.0 ms_handle_connect on v2:172.21.15.114:3300/0 2024-04-23T10:26:49.052 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:49.047+0000 7f2a5bfff640 1 -- 172.21.15.114:0/602210874 <== mon.0 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2a78049040 con 0x555a33353db0 2024-04-23T10:26:49.052 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:49.047+0000 7f2a5bfff640 1 -- 172.21.15.114:0/602210874 <== mon.0 v2:172.21.15.114:3300/0 2 ==== osd_map(90..90 src has 1..90) ==== 8527+0+0 (secure 0 0 0) 0x7f2a780c1160 con 0x555a33353db0 2024-04-23T10:26:49.052 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:49.047+0000 7f2a85e18740 1 -- 172.21.15.114:0/602210874 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f2a780bd820 con 0x555a33353db0 2024-04-23T10:26:49.052 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:49.047+0000 7f2a5bfff640 1 -- 172.21.15.114:0/602210874 <== mon.0 v2:172.21.15.114:3300/0 3 ==== mon_get_version_reply(handle=1 version=21) ==== 24+0+0 (secure 0 0 0) 0x7f2a780b9790 con 0x555a33353db0 2024-04-23T10:26:49.052 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:49.047+0000 7f2a85e18740 10 client.15534 fetch_fsmap learned FSMap version 21 2024-04-23T10:26:49.052 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:49.047+0000 7f2a85e18740 1 -- 172.21.15.114:0/602210874 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({config=0+,fsmap=21,monmap=4+}) -- 0x555a333d69f0 con 0x555a33353db0 2024-04-23T10:26:49.052 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:49.048+0000 7f2a5bfff640 1 -- 172.21.15.114:0/602210874 <== mon.0 v2:172.21.15.114:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f2a780bd820 con 0x555a33353db0 2024-04-23T10:26:49.053 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:49.048+0000 7f2a5bfff640 1 -- 172.21.15.114:0/602210874 <== mon.0 v2:172.21.15.114:3300/0 5 ==== fsmap(e 21) ==== 8338+0+0 (secure 0 0 0) 0x7f2a780bd380 con 0x555a33353db0 2024-04-23T10:26:49.053 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:49.048+0000 7f2a85e18740 10 client.15534 fetch_fsmap finished waiting for FSMap version 21 2024-04-23T10:26:49.053 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:49.048+0000 7f2a85e18740 10 client.15534 resolve_mds: resolved name 'h' to daemon mds.h 2024-04-23T10:26:49.053 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:49.048+0000 7f2a85e18740 20 client.15534 populate_metadata read hostname 'smithi114' 2024-04-23T10:26:49.053 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:49.048+0000 7f2a85e18740 1 --2- 172.21.15.114:0/602210874 >> [v2:172.21.15.119:6836/1872363128,v1:172.21.15.119:6837/1872363128] conn(0x555a333db440 0x555a333da150 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T10:26:49.053 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:49.048+0000 7f2a85e18740 4 client.15534 mds_command: new command op to 24509 tid=0 multi_id=0 [{"prefix": "status"}] 2024-04-23T10:26:49.053 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:49.048+0000 7f2a85e18740 1 -- 172.21.15.114:0/602210874 --> [v2:172.21.15.119:6836/1872363128,v1:172.21.15.119:6837/1872363128] -- command(tid 0: {"prefix": "status"}) -- 0x555a332c14e0 con 0x555a333db440 2024-04-23T10:26:49.053 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:49.048+0000 7f2a73fff640 1 --2- 172.21.15.114:0/602210874 >> [v2:172.21.15.119:6836/1872363128,v1:172.21.15.119:6837/1872363128] conn(0x555a333db440 0x555a333da150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T10:26:49.053 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:49.048+0000 7f2a5bfff640 10 client.15534 ms_handle_connect on v2:172.21.15.119:6836/1872363128 2024-04-23T10:26:49.053 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:49.048+0000 7f2a73fff640 1 --2- 172.21.15.114:0/602210874 >> [v2:172.21.15.119:6836/1872363128,v1:172.21.15.119:6837/1872363128] conn(0x555a333db440 0x555a333da150 crc :-1 s=READY pgs=31 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.1 client_cookie=72293d0d9a2b4bbf server_cookie=f310c86fd96f0bd6 in_seq=0 out_seq=0 2024-04-23T10:26:49.053 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:49.049+0000 7f2a5bfff640 1 -- 172.21.15.114:0/602210874 <== mds.1 v2:172.21.15.119:6836/1872363128 1 ==== command_reply(tid 0: 0 ) ==== 8+0+322 (crc 0 0 0) 0x555a332c14e0 con 0x555a333db440 2024-04-23T10:26:49.053 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:49.049+0000 7f2a5bfff640 10 client.15534 handle_command_reply: tid=0 2024-04-23T10:26:49.053 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:49.049+0000 7f2a85e18740 1 client.15534 shutdown 2024-04-23T10:26:49.053 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:49.049+0000 7f2a85e18740 1 -- 172.21.15.114:0/602210874 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x555a33353db0 msgr2=0x555a33373930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T10:26:49.053 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:49.049+0000 7f2a85e18740 1 --2- 172.21.15.114:0/602210874 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x555a33353db0 0x555a33373930 secure :-1 s=READY pgs=477 cs=0 l=1 rev1=1 crypto rx=0x7f2a780ef040 tx=0x7f2a78032f60 comp rx=0 tx=0).stop 2024-04-23T10:26:49.054 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:49.049+0000 7f2a85e18740 1 -- 172.21.15.114:0/602210874 shutdown_connections 2024-04-23T10:26:49.054 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:49.049+0000 7f2a85e18740 1 --2- 172.21.15.114:0/602210874 >> [v2:172.21.15.119:6836/1872363128,v1:172.21.15.119:6837/1872363128] conn(0x555a333db440 0x555a333da150 crc :-1 s=READY pgs=31 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:26:49.054 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:49.049+0000 7f2a85e18740 1 --2- 172.21.15.114:0/602210874 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x555a3335edd0 0x555a333743b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:26:49.054 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:49.049+0000 7f2a85e18740 1 --2- 172.21.15.114:0/602210874 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x555a33354760 0x555a33373e70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:26:49.054 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:49.049+0000 7f2a5bfff640 0 client.15534 ms_handle_reset on v2:172.21.15.119:6836/1872363128 2024-04-23T10:26:49.054 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:49.049+0000 7f2a85e18740 1 --2- 172.21.15.114:0/602210874 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x555a33353db0 0x555a33373930 unknown :-1 s=CLOSED pgs=477 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:26:49.054 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:49.049+0000 7f2a85e18740 1 -- 172.21.15.114:0/602210874 >> 172.21.15.114:0/602210874 conn(0x555a33350070 msgr2=0x555a3336a110 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T10:26:49.054 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:49.049+0000 7f2a85e18740 1 -- 172.21.15.114:0/602210874 shutdown_connections 2024-04-23T10:26:49.054 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:49.049+0000 7f2a85e18740 1 -- 172.21.15.114:0/602210874 wait complete. 2024-04-23T10:26:49.054 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:49.049+0000 7f2a85e18740 20 client.15534 trim_cache size 0 max 16384 2024-04-23T10:26:49.054 INFO:teuthology.orchestra.run.smithi114.stdout:{ 2024-04-23T10:26:49.054 INFO:teuthology.orchestra.run.smithi114.stdout: "cluster_fsid": "aa240572-0159-11ef-bc93-c7b262605968", 2024-04-23T10:26:49.054 INFO:teuthology.orchestra.run.smithi114.stdout: "whoami": 1, 2024-04-23T10:26:49.055 INFO:teuthology.orchestra.run.smithi114.stdout: "id": 24509, 2024-04-23T10:26:49.055 INFO:teuthology.orchestra.run.smithi114.stdout: "want_state": "up:active", 2024-04-23T10:26:49.055 INFO:teuthology.orchestra.run.smithi114.stdout: "state": "up:active", 2024-04-23T10:26:49.055 INFO:teuthology.orchestra.run.smithi114.stdout: "fs_name": "cephfs", 2024-04-23T10:26:49.055 INFO:teuthology.orchestra.run.smithi114.stdout: "rank_uptime": 122.712302638, 2024-04-23T10:26:49.055 INFO:teuthology.orchestra.run.smithi114.stdout: "mdsmap_epoch": 21, 2024-04-23T10:26:49.055 INFO:teuthology.orchestra.run.smithi114.stdout: "osdmap_epoch": 90, 2024-04-23T10:26:49.055 INFO:teuthology.orchestra.run.smithi114.stdout: "osdmap_epoch_barrier": 89, 2024-04-23T10:26:49.055 INFO:teuthology.orchestra.run.smithi114.stdout: "uptime": 141.645369991 2024-04-23T10:26:49.055 INFO:teuthology.orchestra.run.smithi114.stdout:} 2024-04-23T10:26:49.056 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:49.049+0000 7f2a83101640 1 -- 172.21.15.114:0/2450587483 >> [v2:172.21.15.114:6800/1395693466,v1:172.21.15.114:6801/1395693466] conn(0x7f2a640a3db0 msgr2=0x7f2a640a6270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T10:26:49.056 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:49.049+0000 7f2a83101640 1 --2- 172.21.15.114:0/2450587483 >> [v2:172.21.15.114:6800/1395693466,v1:172.21.15.114:6801/1395693466] conn(0x7f2a640a3db0 0x7f2a640a6270 secure :-1 s=READY pgs=60 cs=0 l=1 rev1=1 crypto rx=0x7f2a74018000 tx=0x7f2a74010af0 comp rx=0 tx=0).stop 2024-04-23T10:26:49.056 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:49.049+0000 7f2a83101640 1 -- 172.21.15.114:0/2450587483 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f2a7c1011a0 msgr2=0x7f2a7c198790 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T10:26:49.056 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:49.049+0000 7f2a83101640 1 --2- 172.21.15.114:0/2450587483 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f2a7c1011a0 0x7f2a7c198790 secure :-1 s=READY pgs=473 cs=0 l=1 rev1=1 crypto rx=0x7f2a78005940 tx=0x7f2a780042f0 comp rx=0 tx=0).stop 2024-04-23T10:26:49.056 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:49.049+0000 7f2a83101640 1 -- 172.21.15.114:0/2450587483 shutdown_connections 2024-04-23T10:26:49.056 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:49.050+0000 7f2a83101640 1 --2- 172.21.15.114:0/2450587483 >> [v2:172.21.15.114:6800/1395693466,v1:172.21.15.114:6801/1395693466] conn(0x7f2a640a3db0 0x7f2a640a6270 unknown :-1 s=CLOSED pgs=60 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:26:49.056 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:49.050+0000 7f2a83101640 1 --2- 172.21.15.114:0/2450587483 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f2a7c105ee0 0x7f2a7c19bfa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:26:49.056 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:49.050+0000 7f2a83101640 1 --2- 172.21.15.114:0/2450587483 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f2a7c101b60 0x7f2a7c198cd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:26:49.056 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:49.050+0000 7f2a83101640 1 --2- 172.21.15.114:0/2450587483 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f2a7c1011a0 0x7f2a7c198790 unknown :-1 s=CLOSED pgs=473 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:26:49.056 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:49.050+0000 7f2a83101640 1 -- 172.21.15.114:0/2450587483 >> 172.21.15.114:0/2450587483 conn(0x7f2a7c076c10 msgr2=0x7f2a7c0fedc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T10:26:49.056 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:49.050+0000 7f2a83101640 1 -- 172.21.15.114:0/2450587483 shutdown_connections 2024-04-23T10:26:49.057 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:49.050+0000 7f2a83101640 1 -- 172.21.15.114:0/2450587483 wait complete. 2024-04-23T10:26:49.059 DEBUG:teuthology.orchestra.run.smithi114:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.b status 2024-04-23T10:26:49.158 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:49.159+0000 7fed171a2640 1 -- 172.21.15.114:0/242028499 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fed10106150 msgr2=0x7fed1010dd20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T10:26:49.158 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:49.159+0000 7fed171a2640 1 --2- 172.21.15.114:0/242028499 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fed10106150 0x7fed1010dd20 secure :-1 s=READY pgs=478 cs=0 l=1 rev1=1 crypto rx=0x7fed08009ca0 tx=0x7fed0802a8c0 comp rx=0 tx=0).stop 2024-04-23T10:26:49.158 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:49.159+0000 7fed171a2640 1 -- 172.21.15.114:0/242028499 shutdown_connections 2024-04-23T10:26:49.158 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:49.159+0000 7fed171a2640 1 --2- 172.21.15.114:0/242028499 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fed10106150 0x7fed1010dd20 unknown :-1 s=CLOSED pgs=478 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:26:49.158 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:49.159+0000 7fed171a2640 1 --2- 172.21.15.114:0/242028499 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fed10101a30 0x7fed10105a70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:26:49.158 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:49.159+0000 7fed171a2640 1 --2- 172.21.15.114:0/242028499 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fed10101080 0x7fed10101460 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:26:49.158 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:49.159+0000 7fed171a2640 1 -- 172.21.15.114:0/242028499 >> 172.21.15.114:0/242028499 conn(0x7fed100771d0 msgr2=0x7fed100775e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T10:26:49.158 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:49.159+0000 7fed171a2640 1 -- 172.21.15.114:0/242028499 shutdown_connections 2024-04-23T10:26:49.159 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:49.159+0000 7fed171a2640 1 -- 172.21.15.114:0/242028499 wait complete. 2024-04-23T10:26:49.161 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:49.160+0000 7fed171a2640 1 Processor -- start 2024-04-23T10:26:49.161 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:49.160+0000 7fed171a2640 1 -- start start 2024-04-23T10:26:49.161 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:49.160+0000 7fed171a2640 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fed10101080 0x7fed10198670 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T10:26:49.161 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:49.160+0000 7fed171a2640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fed10106150 0x7fed10198bb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T10:26:49.161 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:49.160+0000 7fed171a2640 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fed1019cf30 0x7fed1019d3e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T10:26:49.161 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:49.160+0000 7fed171a2640 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x7fed101110a0 con 0x7fed10106150 2024-04-23T10:26:49.161 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:49.160+0000 7fed171a2640 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 -- 0x7fed10110f20 con 0x7fed10101080 2024-04-23T10:26:49.161 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:49.160+0000 7fed171a2640 1 -- --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_getmap magic: 0 -- 0x7fed10111220 con 0x7fed1019cf30 2024-04-23T10:26:49.161 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:49.160+0000 7fed0ffff640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fed10106150 0x7fed10198bb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T10:26:49.161 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:49.160+0000 7fed0ffff640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fed10106150 0x7fed10198bb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.114:52170/0 (socket says 172.21.15.114:52170) 2024-04-23T10:26:49.161 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:49.160+0000 7fed0ffff640 1 -- 172.21.15.114:0/2982426946 learned_addr learned my addr 172.21.15.114:0/2982426946 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-04-23T10:26:49.161 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:49.160+0000 7fed14f3e640 1 --2- 172.21.15.114:0/2982426946 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fed10101080 0x7fed10198670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T10:26:49.161 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:49.160+0000 7fed0ffff640 1 -- 172.21.15.114:0/2982426946 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fed1019cf30 msgr2=0x7fed1019d3e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T10:26:49.162 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:49.160+0000 7fed1573f640 1 --2- 172.21.15.114:0/2982426946 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fed1019cf30 0x7fed1019d3e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T10:26:49.162 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:49.160+0000 7fed0ffff640 1 --2- 172.21.15.114:0/2982426946 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fed1019cf30 0x7fed1019d3e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:26:49.162 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:49.160+0000 7fed0ffff640 1 -- 172.21.15.114:0/2982426946 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fed10101080 msgr2=0x7fed10198670 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T10:26:49.162 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:49.160+0000 7fed0ffff640 1 --2- 172.21.15.114:0/2982426946 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fed10101080 0x7fed10198670 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:26:49.162 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:49.160+0000 7fed0ffff640 1 -- 172.21.15.114:0/2982426946 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fed1019da20 con 0x7fed10106150 2024-04-23T10:26:49.162 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:49.160+0000 7fed14f3e640 1 --2- 172.21.15.114:0/2982426946 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fed10101080 0x7fed10198670 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-23T10:26:49.162 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:49.160+0000 7fed0ffff640 1 --2- 172.21.15.114:0/2982426946 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fed10106150 0x7fed10198bb0 secure :-1 s=READY pgs=479 cs=0 l=1 rev1=1 crypto rx=0x7fed0400c970 tx=0x7fed0400ce30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T10:26:49.162 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:49.160+0000 7fed0dffb640 1 -- 172.21.15.114:0/2982426946 <== mon.0 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fed04014070 con 0x7fed10106150 2024-04-23T10:26:49.162 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:49.160+0000 7fed0dffb640 1 -- 172.21.15.114:0/2982426946 <== mon.0 v2:172.21.15.114:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fed04004440 con 0x7fed10106150 2024-04-23T10:26:49.162 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:49.160+0000 7fed171a2640 1 -- 172.21.15.114:0/2982426946 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fed1019dd10 con 0x7fed10106150 2024-04-23T10:26:49.162 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:49.160+0000 7fed0dffb640 1 -- 172.21.15.114:0/2982426946 <== mon.0 v2:172.21.15.114:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fed04003e50 con 0x7fed10106150 2024-04-23T10:26:49.162 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:49.161+0000 7fed171a2640 1 -- 172.21.15.114:0/2982426946 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fed101a5500 con 0x7fed10106150 2024-04-23T10:26:49.168 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:49.163+0000 7fed0dffb640 1 -- 172.21.15.114:0/2982426946 <== mon.0 v2:172.21.15.114:3300/0 4 ==== mgrmap(e 24) ==== 138823+0+0 (secure 0 0 0) 0x7fed04017070 con 0x7fed10106150 2024-04-23T10:26:49.168 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:49.163+0000 7fed19f96740 1 Processor -- start 2024-04-23T10:26:49.169 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:49.163+0000 7fed19f96740 1 -- start start 2024-04-23T10:26:49.169 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:49.164+0000 7fed0dffb640 1 --2- 172.21.15.114:0/2982426946 >> [v2:172.21.15.114:6800/1395693466,v1:172.21.15.114:6801/1395693466] conn(0x7fecf80a3f50 0x7fecf80a6410 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T10:26:49.169 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:49.164+0000 7fed19f96740 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x5647d67a0db0 0x5647d67a1190 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T10:26:49.169 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:49.164+0000 7fed19f96740 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x5647d67a1760 0x5647d67ab800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T10:26:49.169 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:49.164+0000 7fed19f96740 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x5647d67abdd0 0x5647d67ae220 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T10:26:49.169 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:49.164+0000 7fed19f96740 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x5647d6706c50 con 0x5647d67a1760 2024-04-23T10:26:49.169 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:49.164+0000 7fed19f96740 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 -- 0x5647d66a3830 con 0x5647d67abdd0 2024-04-23T10:26:49.169 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:49.164+0000 7fed19f96740 1 -- --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_getmap magic: 0 -- 0x5647d6702c10 con 0x5647d67a0db0 2024-04-23T10:26:49.169 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:49.164+0000 7fed1573f640 1 --2- 172.21.15.114:0/2982426946 >> [v2:172.21.15.114:6800/1395693466,v1:172.21.15.114:6801/1395693466] conn(0x7fecf80a3f50 0x7fecf80a6410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T10:26:49.169 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:49.164+0000 7fed0ffff640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x5647d67a1760 0x5647d67ab800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T10:26:49.169 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:49.164+0000 7fed0ffff640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x5647d67a1760 0x5647d67ab800 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.114:52174/0 (socket says 172.21.15.114:52174) 2024-04-23T10:26:49.169 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:49.164+0000 7fed0ffff640 1 -- 172.21.15.114:0/3937280713 learned_addr learned my addr 172.21.15.114:0/3937280713 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-04-23T10:26:49.169 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:49.164+0000 7fed0dffb640 1 -- 172.21.15.114:0/2982426946 <== mon.0 v2:172.21.15.114:3300/0 5 ==== osd_map(90..90 src has 1..90) ==== 8527+0+0 (secure 0 0 0) 0x7fed040d05b0 con 0x7fed10106150 2024-04-23T10:26:49.169 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:49.164+0000 7fed14f3e640 1 --2- 172.21.15.114:0/3937280713 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x5647d67a0db0 0x5647d67a1190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T10:26:49.169 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:49.164+0000 7fed1573f640 1 --2- 172.21.15.114:0/3937280713 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x5647d67abdd0 0x5647d67ae220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T10:26:49.170 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:49.164+0000 7fed0ffff640 1 -- 172.21.15.114:0/3937280713 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x5647d67a0db0 msgr2=0x5647d67a1190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T10:26:49.170 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:49.164+0000 7fed0ffff640 1 --2- 172.21.15.114:0/3937280713 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x5647d67a0db0 0x5647d67a1190 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:26:49.170 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:49.164+0000 7fed0ffff640 1 -- 172.21.15.114:0/3937280713 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x5647d67abdd0 msgr2=0x5647d67ae220 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T10:26:49.170 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:49.164+0000 7fed0ffff640 1 --2- 172.21.15.114:0/3937280713 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x5647d67abdd0 0x5647d67ae220 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:26:49.170 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:49.164+0000 7fed0ffff640 1 -- 172.21.15.114:0/3937280713 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5647d64192d0 con 0x5647d67a1760 2024-04-23T10:26:49.170 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:49.164+0000 7fed14f3e640 1 --2- 172.21.15.114:0/3937280713 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x5647d67a0db0 0x5647d67a1190 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-23T10:26:49.170 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:49.164+0000 7fed1573f640 1 --2- 172.21.15.114:0/3937280713 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x5647d67abdd0 0x5647d67ae220 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-23T10:26:49.170 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:49.164+0000 7fed0ffff640 1 --2- 172.21.15.114:0/3937280713 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x5647d67a1760 0x5647d67ab800 secure :-1 s=READY pgs=480 cs=0 l=1 rev1=1 crypto rx=0x7fed04098db0 tx=0x7fed040045f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ac0303a343be33bf server_cookie=0 in_seq=0 out_seq=0 2024-04-23T10:26:49.170 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:49.164+0000 7fed1573f640 1 --2- 172.21.15.114:0/2982426946 >> [v2:172.21.15.114:6800/1395693466,v1:172.21.15.114:6801/1395693466] conn(0x7fecf80a3f50 0x7fecf80a6410 secure :-1 s=READY pgs=61 cs=0 l=1 rev1=1 crypto rx=0x7fed08009ca0 tx=0x7fed080098b0 comp rx=0 tx=0).ready entity=mgr.15291 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T10:26:49.170 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:49.164+0000 7fecf6ffd640 1 -- 172.21.15.114:0/3937280713 <== mon.0 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fed040b5d40 con 0x5647d67a1760 2024-04-23T10:26:49.170 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:49.164+0000 7fecf6ffd640 1 -- 172.21.15.114:0/3937280713 <== mon.0 v2:172.21.15.114:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fed0409cae0 con 0x5647d67a1760 2024-04-23T10:26:49.170 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:49.164+0000 7fecf6ffd640 1 -- 172.21.15.114:0/3937280713 <== mon.0 v2:172.21.15.114:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fed040b5b60 con 0x5647d67a1760 2024-04-23T10:26:49.170 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:49.165+0000 7fed19f96740 1 -- 172.21.15.114:0/3937280713 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x5647d67a1760 msgr2=0x5647d67ab800 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T10:26:49.170 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:49.165+0000 7fed19f96740 1 --2- 172.21.15.114:0/3937280713 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x5647d67a1760 0x5647d67ab800 secure :-1 s=READY pgs=480 cs=0 l=1 rev1=1 crypto rx=0x7fed04098db0 tx=0x7fed040045f0 comp rx=0 tx=0).stop 2024-04-23T10:26:49.170 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:49.165+0000 7fed19f96740 1 -- 172.21.15.114:0/3937280713 shutdown_connections 2024-04-23T10:26:49.171 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:49.165+0000 7fed19f96740 1 --2- 172.21.15.114:0/3937280713 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x5647d67abdd0 0x5647d67ae220 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:26:49.171 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:49.165+0000 7fed19f96740 1 --2- 172.21.15.114:0/3937280713 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x5647d67a1760 0x5647d67ab800 unknown :-1 s=CLOSED pgs=480 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:26:49.171 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:49.165+0000 7fed19f96740 1 --2- 172.21.15.114:0/3937280713 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x5647d67a0db0 0x5647d67a1190 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:26:49.171 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:49.165+0000 7fed19f96740 1 -- 172.21.15.114:0/3937280713 >> 172.21.15.114:0/3937280713 conn(0x5647d679d070 msgr2=0x5647d679d450 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T10:26:49.171 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:49.165+0000 7fed19f96740 1 -- 172.21.15.114:0/3937280713 shutdown_connections 2024-04-23T10:26:49.171 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:49.165+0000 7fed19f96740 1 -- 172.21.15.114:0/3937280713 wait complete. 2024-04-23T10:26:49.171 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:49.165+0000 7fed19f96740 1 -- start start 2024-04-23T10:26:49.171 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:49.165+0000 7fed19f96740 1 Processor -- start 2024-04-23T10:26:49.171 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:49.166+0000 7fed19f96740 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x5647d67a0db0 0x5647d67a8ca0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T10:26:49.171 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:49.166+0000 7fed19f96740 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x5647d67a1760 0x5647d67a91e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T10:26:49.171 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:49.166+0000 7fed19f96740 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x5647d67abdd0 0x5647d67a9720 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T10:26:49.171 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:49.166+0000 7fed19f96740 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x5647d67aece0 con 0x5647d67a0db0 2024-04-23T10:26:49.171 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:49.166+0000 7fed19f96740 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 -- 0x5647d66a1470 con 0x5647d67a1760 2024-04-23T10:26:49.171 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:49.166+0000 7fed19f96740 1 -- --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_getmap magic: 0 -- 0x5647d67aee60 con 0x5647d67abdd0 2024-04-23T10:26:49.171 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:49.166+0000 7fed19f96740 10 client.0 _ll_register_callbacks cb 0x5647d66c6f30 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-23T10:26:49.171 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:49.166+0000 7fed14f3e640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x5647d67a0db0 0x5647d67a8ca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T10:26:49.172 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:49.166+0000 7fed14f3e640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x5647d67a0db0 0x5647d67a8ca0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.114:52188/0 (socket says 172.21.15.114:52188) 2024-04-23T10:26:49.172 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:49.166+0000 7fed14f3e640 1 -- 172.21.15.114:0/3642634482 learned_addr learned my addr 172.21.15.114:0/3642634482 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-04-23T10:26:49.172 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:49.166+0000 7fed0ffff640 1 --2- 172.21.15.114:0/3642634482 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x5647d67a1760 0x5647d67a91e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T10:26:49.172 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:49.166+0000 7fed14f3e640 1 -- 172.21.15.114:0/3642634482 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x5647d67abdd0 msgr2=0x5647d67a9720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T10:26:49.172 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:49.166+0000 7fed14f3e640 1 --2- 172.21.15.114:0/3642634482 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x5647d67abdd0 0x5647d67a9720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:26:49.172 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:49.166+0000 7fed14f3e640 1 -- 172.21.15.114:0/3642634482 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x5647d67a1760 msgr2=0x5647d67a91e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T10:26:49.172 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:49.166+0000 7fed14f3e640 1 --2- 172.21.15.114:0/3642634482 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x5647d67a1760 0x5647d67a91e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:26:49.172 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:49.166+0000 7fed14f3e640 1 -- 172.21.15.114:0/3642634482 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5647d64e36b0 con 0x5647d67a0db0 2024-04-23T10:26:49.172 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:49.166+0000 7fed0ffff640 1 --2- 172.21.15.114:0/3642634482 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x5647d67a1760 0x5647d67a91e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-23T10:26:49.172 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:49.166+0000 7fed14f3e640 1 --2- 172.21.15.114:0/3642634482 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x5647d67a0db0 0x5647d67a8ca0 secure :-1 s=READY pgs=481 cs=0 l=1 rev1=1 crypto rx=0x7fed000106a0 tx=0x7fed00010b60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1875a0de4352b68f server_cookie=0 in_seq=0 out_seq=0 2024-04-23T10:26:49.172 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:49.166+0000 7feceffff640 10 client.0 ms_handle_connect on v2:172.21.15.114:3300/0 2024-04-23T10:26:49.172 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:49.167+0000 7feceffff640 1 -- 172.21.15.114:0/3642634482 <== mon.0 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fed00010ea0 con 0x5647d67a0db0 2024-04-23T10:26:49.172 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:49.167+0000 7feceffff640 1 -- 172.21.15.114:0/3642634482 <== mon.0 v2:172.21.15.114:3300/0 2 ==== osd_map(90..90 src has 1..90) ==== 8527+0+0 (secure 0 0 0) 0x7fed0000ead0 con 0x5647d67a0db0 2024-04-23T10:26:49.172 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:49.167+0000 7fed19f96740 1 -- 172.21.15.114:0/3642634482 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fed0409cae0 con 0x5647d67a0db0 2024-04-23T10:26:49.172 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:49.167+0000 7feceffff640 1 -- 172.21.15.114:0/3642634482 <== mon.0 v2:172.21.15.114:3300/0 3 ==== mon_get_version_reply(handle=1 version=21) ==== 24+0+0 (secure 0 0 0) 0x7fed0000ecd0 con 0x5647d67a0db0 2024-04-23T10:26:49.173 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:49.167+0000 7fed19f96740 10 client.15546 fetch_fsmap learned FSMap version 21 2024-04-23T10:26:49.173 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:49.167+0000 7fed19f96740 1 -- 172.21.15.114:0/3642634482 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({config=0+,fsmap=21,monmap=4+}) -- 0x5647d67baf10 con 0x5647d67a0db0 2024-04-23T10:26:49.173 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:49.167+0000 7feceffff640 1 -- 172.21.15.114:0/3642634482 <== mon.0 v2:172.21.15.114:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fed0409cae0 con 0x5647d67a0db0 2024-04-23T10:26:49.173 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:49.167+0000 7feceffff640 1 -- 172.21.15.114:0/3642634482 <== mon.0 v2:172.21.15.114:3300/0 5 ==== fsmap(e 21) ==== 8338+0+0 (secure 0 0 0) 0x7fed0000f040 con 0x5647d67a0db0 2024-04-23T10:26:49.173 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:49.167+0000 7fed19f96740 10 client.15546 fetch_fsmap finished waiting for FSMap version 21 2024-04-23T10:26:49.173 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:49.167+0000 7fed19f96740 10 client.15546 resolve_mds: resolved name 'b' to daemon mds.b 2024-04-23T10:26:49.173 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:49.167+0000 7fed19f96740 20 client.15546 populate_metadata read hostname 'smithi114' 2024-04-23T10:26:49.173 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:49.167+0000 7fed19f96740 1 --2- 172.21.15.114:0/3642634482 >> [v2:172.21.15.119:6838/937297340,v1:172.21.15.119:6839/937297340] conn(0x5647d67bd600 0x5647d67bea70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T10:26:49.173 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:49.167+0000 7fed19f96740 4 client.15546 mds_command: new command op to 24545 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-23T10:26:49.173 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:49.167+0000 7fed19f96740 1 -- 172.21.15.114:0/3642634482 --> [v2:172.21.15.119:6838/937297340,v1:172.21.15.119:6839/937297340] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x5647d6714670 con 0x5647d67bd600 2024-04-23T10:26:49.173 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:49.168+0000 7fed0ffff640 1 --2- 172.21.15.114:0/3642634482 >> [v2:172.21.15.119:6838/937297340,v1:172.21.15.119:6839/937297340] conn(0x5647d67bd600 0x5647d67bea70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T10:26:49.173 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:49.168+0000 7feceffff640 10 client.15546 ms_handle_connect on v2:172.21.15.119:6838/937297340 2024-04-23T10:26:49.173 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:49.168+0000 7fed0ffff640 1 --2- 172.21.15.114:0/3642634482 >> [v2:172.21.15.119:6838/937297340,v1:172.21.15.119:6839/937297340] conn(0x5647d67bd600 0x5647d67bea70 crc :-1 s=READY pgs=37 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=d35c63dd5fd29fd2 server_cookie=a6a347efb90511d9 in_seq=0 out_seq=0 2024-04-23T10:26:49.173 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:49.169+0000 7feceffff640 1 -- 172.21.15.114:0/3642634482 <== mds.0 v2:172.21.15.119:6838/937297340 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x5647d6714670 con 0x5647d67bd600 2024-04-23T10:26:49.173 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:49.169+0000 7feceffff640 10 client.15546 handle_command_reply: tid=0 2024-04-23T10:26:49.173 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:49.169+0000 7fed19f96740 1 client.15546 shutdown 2024-04-23T10:26:49.174 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:49.169+0000 7fed19f96740 1 -- 172.21.15.114:0/3642634482 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x5647d67a0db0 msgr2=0x5647d67a8ca0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T10:26:49.174 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:49.169+0000 7fed19f96740 1 --2- 172.21.15.114:0/3642634482 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x5647d67a0db0 0x5647d67a8ca0 secure :-1 s=READY pgs=481 cs=0 l=1 rev1=1 crypto rx=0x7fed000106a0 tx=0x7fed00010b60 comp rx=0 tx=0).stop 2024-04-23T10:26:49.174 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:49.169+0000 7fed19f96740 1 -- 172.21.15.114:0/3642634482 shutdown_connections 2024-04-23T10:26:49.174 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:49.169+0000 7fed19f96740 1 --2- 172.21.15.114:0/3642634482 >> [v2:172.21.15.119:6838/937297340,v1:172.21.15.119:6839/937297340] conn(0x5647d67bd600 0x5647d67bea70 crc :-1 s=READY pgs=37 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:26:49.174 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:49.169+0000 7fed19f96740 1 --2- 172.21.15.114:0/3642634482 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x5647d67abdd0 0x5647d67a9720 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:26:49.174 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:49.169+0000 7feceffff640 0 client.15546 ms_handle_reset on v2:172.21.15.119:6838/937297340 2024-04-23T10:26:49.174 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:49.169+0000 7fed19f96740 1 --2- 172.21.15.114:0/3642634482 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x5647d67a1760 0x5647d67a91e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:26:49.174 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:49.169+0000 7fed19f96740 1 --2- 172.21.15.114:0/3642634482 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x5647d67a0db0 0x5647d67a8ca0 unknown :-1 s=CLOSED pgs=481 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:26:49.174 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:49.169+0000 7fed19f96740 1 -- 172.21.15.114:0/3642634482 >> 172.21.15.114:0/3642634482 conn(0x5647d679d070 msgr2=0x5647d67a2ee0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T10:26:49.174 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:49.169+0000 7fed19f96740 1 -- 172.21.15.114:0/3642634482 shutdown_connections 2024-04-23T10:26:49.174 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:49.169+0000 7fed19f96740 1 -- 172.21.15.114:0/3642634482 wait complete. 2024-04-23T10:26:49.174 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:49.169+0000 7fed19f96740 20 client.15546 trim_cache size 0 max 16384 2024-04-23T10:26:49.191 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:49.187+0000 7fed19f96740 1 Processor -- start 2024-04-23T10:26:49.191 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:49.187+0000 7fed19f96740 1 -- start start 2024-04-23T10:26:49.191 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:49.187+0000 7fed19f96740 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x5647d67a0db0 0x5647d679c950 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T10:26:49.191 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:49.187+0000 7fed19f96740 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x5647d67a1760 0x5647d679d450 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T10:26:49.192 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:49.187+0000 7fed19f96740 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x5647d67abdd0 0x5647d67a35f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T10:26:49.192 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:49.187+0000 7fed19f96740 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x5647d67bcdb0 con 0x5647d67a0db0 2024-04-23T10:26:49.192 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:49.187+0000 7fed19f96740 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 -- 0x5647d67bd480 con 0x5647d67a1760 2024-04-23T10:26:49.192 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:49.187+0000 7fed19f96740 1 -- --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_getmap magic: 0 -- 0x5647d67bdb50 con 0x5647d67abdd0 2024-04-23T10:26:49.192 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:49.187+0000 7fed14f3e640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x5647d67a0db0 0x5647d679c950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T10:26:49.192 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:49.187+0000 7fed14f3e640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x5647d67a0db0 0x5647d679c950 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.114:52196/0 (socket says 172.21.15.114:52196) 2024-04-23T10:26:49.192 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:49.187+0000 7fed14f3e640 1 -- 172.21.15.114:0/1602836920 learned_addr learned my addr 172.21.15.114:0/1602836920 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-04-23T10:26:49.192 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:49.187+0000 7fed0ffff640 1 --2- 172.21.15.114:0/1602836920 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x5647d67a1760 0x5647d679d450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T10:26:49.192 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:49.187+0000 7fed14f3e640 1 -- 172.21.15.114:0/1602836920 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x5647d67abdd0 msgr2=0x5647d67a35f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T10:26:49.192 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:49.187+0000 7fed14f3e640 1 --2- 172.21.15.114:0/1602836920 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x5647d67abdd0 0x5647d67a35f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:26:49.192 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:49.187+0000 7fed14f3e640 1 -- 172.21.15.114:0/1602836920 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x5647d67a1760 msgr2=0x5647d679d450 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T10:26:49.192 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:49.187+0000 7fed14f3e640 1 --2- 172.21.15.114:0/1602836920 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x5647d67a1760 0x5647d679d450 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:26:49.192 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:49.187+0000 7fed14f3e640 1 -- 172.21.15.114:0/1602836920 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5647d67a3b30 con 0x5647d67a0db0 2024-04-23T10:26:49.192 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:49.187+0000 7fed0ffff640 1 --2- 172.21.15.114:0/1602836920 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x5647d67a1760 0x5647d679d450 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-23T10:26:49.192 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:49.188+0000 7fed14f3e640 1 --2- 172.21.15.114:0/1602836920 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x5647d67a0db0 0x5647d679c950 secure :-1 s=READY pgs=482 cs=0 l=1 rev1=1 crypto rx=0x7fed0000b520 tx=0x7fed00005730 comp rx=0 tx=0).ready entity=mon.0 client_cookie=75789eb72e178849 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T10:26:49.193 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:49.188+0000 7fecf67fc640 1 -- 172.21.15.114:0/1602836920 <== mon.0 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fed0000bd70 con 0x5647d67a0db0 2024-04-23T10:26:49.193 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:49.188+0000 7fecf67fc640 1 -- 172.21.15.114:0/1602836920 <== mon.0 v2:172.21.15.114:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fed000100d0 con 0x5647d67a0db0 2024-04-23T10:26:49.193 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:49.188+0000 7fecf67fc640 1 -- 172.21.15.114:0/1602836920 <== mon.0 v2:172.21.15.114:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fed00023c90 con 0x5647d67a0db0 2024-04-23T10:26:49.193 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:49.188+0000 7fed19f96740 1 -- 172.21.15.114:0/1602836920 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x5647d67a0db0 msgr2=0x5647d679c950 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T10:26:49.193 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:49.188+0000 7fed19f96740 1 --2- 172.21.15.114:0/1602836920 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x5647d67a0db0 0x5647d679c950 secure :-1 s=READY pgs=482 cs=0 l=1 rev1=1 crypto rx=0x7fed0000b520 tx=0x7fed00005730 comp rx=0 tx=0).stop 2024-04-23T10:26:49.193 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:49.188+0000 7fed19f96740 1 -- 172.21.15.114:0/1602836920 shutdown_connections 2024-04-23T10:26:49.193 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:49.188+0000 7fed19f96740 1 --2- 172.21.15.114:0/1602836920 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x5647d67abdd0 0x5647d67a35f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:26:49.193 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:49.188+0000 7fed19f96740 1 --2- 172.21.15.114:0/1602836920 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x5647d67a1760 0x5647d679d450 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:26:49.193 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:49.188+0000 7fed19f96740 1 --2- 172.21.15.114:0/1602836920 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x5647d67a0db0 0x5647d679c950 unknown :-1 s=CLOSED pgs=482 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:26:49.193 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:49.188+0000 7fed19f96740 1 -- 172.21.15.114:0/1602836920 >> 172.21.15.114:0/1602836920 conn(0x5647d679d070 msgr2=0x5647d67b70b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T10:26:49.193 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:49.188+0000 7fed19f96740 1 -- 172.21.15.114:0/1602836920 shutdown_connections 2024-04-23T10:26:49.193 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:49.188+0000 7fed19f96740 1 -- 172.21.15.114:0/1602836920 wait complete. 2024-04-23T10:26:49.193 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:49.189+0000 7fed19f96740 1 -- start start 2024-04-23T10:26:49.193 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:49.189+0000 7fed19f96740 1 Processor -- start 2024-04-23T10:26:49.193 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:49.189+0000 7fed19f96740 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x5647d67a0db0 0x5647d67c6f30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T10:26:49.193 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:49.189+0000 7fed19f96740 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x5647d67a1760 0x5647d67a5530 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T10:26:49.193 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:49.189+0000 7fed19f96740 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x5647d67abdd0 0x5647d67a5a70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T10:26:49.194 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:49.189+0000 7fed19f96740 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x5647d67be8f0 con 0x5647d67a1760 2024-04-23T10:26:49.194 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:49.189+0000 7fed19f96740 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 -- 0x5647d67be220 con 0x5647d67a0db0 2024-04-23T10:26:49.194 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:49.189+0000 7fed19f96740 1 -- --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_getmap magic: 0 -- 0x5647d67befc0 con 0x5647d67abdd0 2024-04-23T10:26:49.194 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:49.189+0000 7fed19f96740 10 client.0 _ll_register_callbacks cb 0x7fed0000b400 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-23T10:26:49.194 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:49.189+0000 7fed0ffff640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x5647d67a1760 0x5647d67a5530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T10:26:49.194 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:49.189+0000 7fed0ffff640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x5647d67a1760 0x5647d67a5530 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.114:52200/0 (socket says 172.21.15.114:52200) 2024-04-23T10:26:49.194 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:49.189+0000 7fed0ffff640 1 -- 172.21.15.114:0/1207415804 learned_addr learned my addr 172.21.15.114:0/1207415804 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-04-23T10:26:49.194 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:49.189+0000 7fed14f3e640 1 --2- 172.21.15.114:0/1207415804 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x5647d67a0db0 0x5647d67c6f30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T10:26:49.194 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:49.189+0000 7fed1573f640 1 --2- 172.21.15.114:0/1207415804 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x5647d67abdd0 0x5647d67a5a70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T10:26:49.194 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:49.189+0000 7fed0ffff640 1 -- 172.21.15.114:0/1207415804 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x5647d67abdd0 msgr2=0x5647d67a5a70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T10:26:49.194 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:49.189+0000 7fed0ffff640 1 --2- 172.21.15.114:0/1207415804 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x5647d67abdd0 0x5647d67a5a70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:26:49.194 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:49.189+0000 7fed0ffff640 1 -- 172.21.15.114:0/1207415804 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x5647d67a0db0 msgr2=0x5647d67c6f30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T10:26:49.194 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:49.189+0000 7fed0ffff640 1 --2- 172.21.15.114:0/1207415804 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x5647d67a0db0 0x5647d67c6f30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:26:49.194 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:49.189+0000 7fed0ffff640 1 -- 172.21.15.114:0/1207415804 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5647d67a6060 con 0x5647d67a1760 2024-04-23T10:26:49.194 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:49.189+0000 7fed1573f640 1 --2- 172.21.15.114:0/1207415804 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x5647d67abdd0 0x5647d67a5a70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-23T10:26:49.195 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:49.190+0000 7fed0ffff640 1 --2- 172.21.15.114:0/1207415804 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x5647d67a1760 0x5647d67a5530 secure :-1 s=READY pgs=483 cs=0 l=1 rev1=1 crypto rx=0x7fed0409c370 tx=0x7fed040a08d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=58d37163602612f9 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T10:26:49.195 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:49.190+0000 7feceffff640 10 client.0 ms_handle_connect on v2:172.21.15.114:3300/0 2024-04-23T10:26:49.195 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:49.190+0000 7feceffff640 1 -- 172.21.15.114:0/1207415804 <== mon.0 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fed04085020 con 0x5647d67a1760 2024-04-23T10:26:49.195 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:49.190+0000 7feceffff640 1 -- 172.21.15.114:0/1207415804 <== mon.0 v2:172.21.15.114:3300/0 2 ==== osd_map(90..90 src has 1..90) ==== 8527+0+0 (secure 0 0 0) 0x7fed04098a50 con 0x5647d67a1760 2024-04-23T10:26:49.195 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:49.190+0000 7fed19f96740 1 -- 172.21.15.114:0/1207415804 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fed000100d0 con 0x5647d67a1760 2024-04-23T10:26:49.195 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:49.190+0000 7feceffff640 1 -- 172.21.15.114:0/1207415804 <== mon.0 v2:172.21.15.114:3300/0 3 ==== mon_get_version_reply(handle=1 version=21) ==== 24+0+0 (secure 0 0 0) 0x7fed04098680 con 0x5647d67a1760 2024-04-23T10:26:49.195 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:49.190+0000 7fed19f96740 10 client.15552 fetch_fsmap learned FSMap version 21 2024-04-23T10:26:49.195 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:49.190+0000 7fed19f96740 1 -- 172.21.15.114:0/1207415804 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({config=0+,fsmap=21,monmap=4+}) -- 0x5647d67c6aa0 con 0x5647d67a1760 2024-04-23T10:26:49.195 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:49.190+0000 7feceffff640 1 -- 172.21.15.114:0/1207415804 <== mon.0 v2:172.21.15.114:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fed000100d0 con 0x5647d67a1760 2024-04-23T10:26:49.195 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:49.190+0000 7feceffff640 1 -- 172.21.15.114:0/1207415804 <== mon.0 v2:172.21.15.114:3300/0 5 ==== fsmap(e 21) ==== 8338+0+0 (secure 0 0 0) 0x7fed040baa30 con 0x5647d67a1760 2024-04-23T10:26:49.195 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:49.190+0000 7fed19f96740 10 client.15552 fetch_fsmap finished waiting for FSMap version 21 2024-04-23T10:26:49.195 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:49.190+0000 7fed19f96740 10 client.15552 resolve_mds: resolved name 'b' to daemon mds.b 2024-04-23T10:26:49.195 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:49.190+0000 7fed19f96740 20 client.15552 populate_metadata read hostname 'smithi114' 2024-04-23T10:26:49.195 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:49.190+0000 7fed19f96740 1 --2- 172.21.15.114:0/1207415804 >> [v2:172.21.15.119:6838/937297340,v1:172.21.15.119:6839/937297340] conn(0x5647d6827b00 0x5647d6826810 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T10:26:49.195 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:49.190+0000 7fed19f96740 4 client.15552 mds_command: new command op to 24545 tid=0 multi_id=0 [{"prefix": "status"}] 2024-04-23T10:26:49.195 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:49.190+0000 7fed19f96740 1 -- 172.21.15.114:0/1207415804 --> [v2:172.21.15.119:6838/937297340,v1:172.21.15.119:6839/937297340] -- command(tid 0: {"prefix": "status"}) -- 0x5647d670e4e0 con 0x5647d6827b00 2024-04-23T10:26:49.196 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:49.191+0000 7fed14f3e640 1 --2- 172.21.15.114:0/1207415804 >> [v2:172.21.15.119:6838/937297340,v1:172.21.15.119:6839/937297340] conn(0x5647d6827b00 0x5647d6826810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T10:26:49.196 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:49.191+0000 7fed14f3e640 1 --2- 172.21.15.114:0/1207415804 >> [v2:172.21.15.119:6838/937297340,v1:172.21.15.119:6839/937297340] conn(0x5647d6827b00 0x5647d6826810 crc :-1 s=READY pgs=38 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=e77b1271e4076a88 server_cookie=71b9abd2629a1f3f in_seq=0 out_seq=0 2024-04-23T10:26:49.196 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:49.191+0000 7feceffff640 10 client.15552 ms_handle_connect on v2:172.21.15.119:6838/937297340 2024-04-23T10:26:49.196 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:49.191+0000 7feceffff640 1 -- 172.21.15.114:0/1207415804 <== mds.0 v2:172.21.15.119:6838/937297340 1 ==== command_reply(tid 0: 0 ) ==== 8+0+322 (crc 0 0 0) 0x5647d670e4e0 con 0x5647d6827b00 2024-04-23T10:26:49.196 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:49.191+0000 7feceffff640 10 client.15552 handle_command_reply: tid=0 2024-04-23T10:26:49.196 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:49.191+0000 7fed19f96740 1 client.15552 shutdown 2024-04-23T10:26:49.196 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:49.191+0000 7fed19f96740 1 -- 172.21.15.114:0/1207415804 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x5647d67a1760 msgr2=0x5647d67a5530 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T10:26:49.196 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:49.191+0000 7fed19f96740 1 --2- 172.21.15.114:0/1207415804 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x5647d67a1760 0x5647d67a5530 secure :-1 s=READY pgs=483 cs=0 l=1 rev1=1 crypto rx=0x7fed0409c370 tx=0x7fed040a08d0 comp rx=0 tx=0).stop 2024-04-23T10:26:49.196 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:49.191+0000 7fed19f96740 1 -- 172.21.15.114:0/1207415804 shutdown_connections 2024-04-23T10:26:49.196 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:49.191+0000 7fed19f96740 1 --2- 172.21.15.114:0/1207415804 >> [v2:172.21.15.119:6838/937297340,v1:172.21.15.119:6839/937297340] conn(0x5647d6827b00 0x5647d6826810 crc :-1 s=READY pgs=38 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:26:49.196 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:49.191+0000 7fed19f96740 1 --2- 172.21.15.114:0/1207415804 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x5647d67abdd0 0x5647d67a5a70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:26:49.196 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:49.191+0000 7fed19f96740 1 --2- 172.21.15.114:0/1207415804 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x5647d67a1760 0x5647d67a5530 unknown :-1 s=CLOSED pgs=483 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:26:49.196 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:49.191+0000 7feceffff640 0 client.15552 ms_handle_reset on v2:172.21.15.119:6838/937297340 2024-04-23T10:26:49.196 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:49.191+0000 7fed19f96740 1 --2- 172.21.15.114:0/1207415804 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x5647d67a0db0 0x5647d67c6f30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:26:49.196 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:49.191+0000 7fed19f96740 1 -- 172.21.15.114:0/1207415804 >> 172.21.15.114:0/1207415804 conn(0x5647d679d070 msgr2=0x5647d67a7cf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T10:26:49.196 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:49.191+0000 7fed19f96740 1 -- 172.21.15.114:0/1207415804 shutdown_connections 2024-04-23T10:26:49.197 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:49.191+0000 7fed19f96740 1 -- 172.21.15.114:0/1207415804 wait complete. 2024-04-23T10:26:49.197 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:49.192+0000 7fed19f96740 20 client.15552 trim_cache size 0 max 16384 2024-04-23T10:26:49.197 INFO:teuthology.orchestra.run.smithi114.stdout:{ 2024-04-23T10:26:49.197 INFO:teuthology.orchestra.run.smithi114.stdout: "cluster_fsid": "aa240572-0159-11ef-bc93-c7b262605968", 2024-04-23T10:26:49.197 INFO:teuthology.orchestra.run.smithi114.stdout: "whoami": 0, 2024-04-23T10:26:49.197 INFO:teuthology.orchestra.run.smithi114.stdout: "id": 24545, 2024-04-23T10:26:49.197 INFO:teuthology.orchestra.run.smithi114.stdout: "want_state": "up:active", 2024-04-23T10:26:49.197 INFO:teuthology.orchestra.run.smithi114.stdout: "state": "up:active", 2024-04-23T10:26:49.197 INFO:teuthology.orchestra.run.smithi114.stdout: "fs_name": "cephfs", 2024-04-23T10:26:49.197 INFO:teuthology.orchestra.run.smithi114.stdout: "rank_uptime": 124.873733241, 2024-04-23T10:26:49.197 INFO:teuthology.orchestra.run.smithi114.stdout: "mdsmap_epoch": 21, 2024-04-23T10:26:49.197 INFO:teuthology.orchestra.run.smithi114.stdout: "osdmap_epoch": 90, 2024-04-23T10:26:49.198 INFO:teuthology.orchestra.run.smithi114.stdout: "osdmap_epoch_barrier": 89, 2024-04-23T10:26:49.198 INFO:teuthology.orchestra.run.smithi114.stdout: "uptime": 130.779674419 2024-04-23T10:26:49.198 INFO:teuthology.orchestra.run.smithi114.stdout:} 2024-04-23T10:26:49.198 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:49.192+0000 7fed171a2640 1 -- 172.21.15.114:0/2982426946 >> [v2:172.21.15.114:6800/1395693466,v1:172.21.15.114:6801/1395693466] conn(0x7fecf80a3f50 msgr2=0x7fecf80a6410 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T10:26:49.199 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:49.192+0000 7fed171a2640 1 --2- 172.21.15.114:0/2982426946 >> [v2:172.21.15.114:6800/1395693466,v1:172.21.15.114:6801/1395693466] conn(0x7fecf80a3f50 0x7fecf80a6410 secure :-1 s=READY pgs=61 cs=0 l=1 rev1=1 crypto rx=0x7fed08009ca0 tx=0x7fed080098b0 comp rx=0 tx=0).stop 2024-04-23T10:26:49.199 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:49.192+0000 7fed171a2640 1 -- 172.21.15.114:0/2982426946 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fed10106150 msgr2=0x7fed10198bb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T10:26:49.199 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:49.192+0000 7fed171a2640 1 --2- 172.21.15.114:0/2982426946 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fed10106150 0x7fed10198bb0 secure :-1 s=READY pgs=479 cs=0 l=1 rev1=1 crypto rx=0x7fed0400c970 tx=0x7fed0400ce30 comp rx=0 tx=0).stop 2024-04-23T10:26:49.199 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:49.192+0000 7fed171a2640 1 -- 172.21.15.114:0/2982426946 shutdown_connections 2024-04-23T10:26:49.199 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:49.192+0000 7fed171a2640 1 --2- 172.21.15.114:0/2982426946 >> [v2:172.21.15.114:6800/1395693466,v1:172.21.15.114:6801/1395693466] conn(0x7fecf80a3f50 0x7fecf80a6410 unknown :-1 s=CLOSED pgs=61 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:26:49.199 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:49.192+0000 7fed171a2640 1 --2- 172.21.15.114:0/2982426946 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fed1019cf30 0x7fed1019d3e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:26:49.199 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:49.192+0000 7fed171a2640 1 --2- 172.21.15.114:0/2982426946 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fed10106150 0x7fed10198bb0 unknown :-1 s=CLOSED pgs=479 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:26:49.199 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:49.192+0000 7fed171a2640 1 --2- 172.21.15.114:0/2982426946 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fed10101080 0x7fed10198670 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:26:49.199 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:49.192+0000 7fed171a2640 1 -- 172.21.15.114:0/2982426946 >> 172.21.15.114:0/2982426946 conn(0x7fed100771d0 msgr2=0x7fed1006a840 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T10:26:49.199 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:49.192+0000 7fed171a2640 1 -- 172.21.15.114:0/2982426946 shutdown_connections 2024-04-23T10:26:49.199 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:26:49.192+0000 7fed171a2640 1 -- 172.21.15.114:0/2982426946 wait complete. 2024-04-23T10:26:49.202 INFO:tasks.quiescer:Ready to start quiesce thrashing; initial delay: 15 sec 2024-04-23T10:26:49.328 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 23 10:26:49 smithi204 ceph-mon[32649]: from='client.? 172.21.15.114:0/2867739893' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-23T10:26:49.328 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 23 10:26:49 smithi204 ceph-mon[32649]: from='client.? 172.21.15.114:0/1272188247' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-23T10:26:49.328 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 23 10:26:49 smithi204 ceph-mon[32649]: from='client.? 172.21.15.114:0/3486804119' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-23T10:26:49.491 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 23 10:26:49 smithi119 ceph-mon[32608]: from='client.? 172.21.15.114:0/2867739893' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-23T10:26:49.492 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 23 10:26:49 smithi119 ceph-mon[32608]: from='client.? 172.21.15.114:0/1272188247' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-23T10:26:49.492 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 23 10:26:49 smithi119 ceph-mon[32608]: from='client.? 172.21.15.114:0/3486804119' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-23T10:26:49.504 INFO:journalctl@ceph.mon.a.smithi114.stdout:Apr 23 10:26:49 smithi114 ceph-mon[29255]: from='client.? 172.21.15.114:0/2867739893' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-23T10:26:49.504 INFO:journalctl@ceph.mon.a.smithi114.stdout:Apr 23 10:26:49 smithi114 ceph-mon[29255]: from='client.? 172.21.15.114:0/1272188247' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-23T10:26:49.504 INFO:journalctl@ceph.mon.a.smithi114.stdout:Apr 23 10:26:49 smithi114 ceph-mon[29255]: from='client.? 172.21.15.114:0/3486804119' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-23T10:26:50.328 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 23 10:26:50 smithi204 ceph-mon[32649]: pgmap v7: 97 pgs: 97 active+clean; 1.3 MiB data, 377 MiB used, 1.0 TiB / 1.0 TiB avail; 66 KiB/s rd, 131 KiB/s wr, 76 op/s 2024-04-23T10:26:50.492 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 23 10:26:50 smithi119 ceph-mon[32608]: pgmap v7: 97 pgs: 97 active+clean; 1.3 MiB data, 377 MiB used, 1.0 TiB / 1.0 TiB avail; 66 KiB/s rd, 131 KiB/s wr, 76 op/s 2024-04-23T10:26:50.504 INFO:journalctl@ceph.mon.a.smithi114.stdout:Apr 23 10:26:50 smithi114 ceph-mon[29255]: pgmap v7: 97 pgs: 97 active+clean; 1.3 MiB data, 377 MiB used, 1.0 TiB / 1.0 TiB avail; 66 KiB/s rd, 131 KiB/s wr, 76 op/s 2024-04-23T10:26:52.328 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 23 10:26:52 smithi204 ceph-mon[32649]: pgmap v8: 97 pgs: 97 active+clean; 1.3 MiB data, 377 MiB used, 1.0 TiB / 1.0 TiB avail; 51 KiB/s rd, 102 KiB/s wr, 59 op/s 2024-04-23T10:26:52.492 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 23 10:26:52 smithi119 ceph-mon[32608]: pgmap v8: 97 pgs: 97 active+clean; 1.3 MiB data, 377 MiB used, 1.0 TiB / 1.0 TiB avail; 51 KiB/s rd, 102 KiB/s wr, 59 op/s 2024-04-23T10:26:52.504 INFO:journalctl@ceph.mon.a.smithi114.stdout:Apr 23 10:26:52 smithi114 ceph-mon[29255]: pgmap v8: 97 pgs: 97 active+clean; 1.3 MiB data, 377 MiB used, 1.0 TiB / 1.0 TiB avail; 51 KiB/s rd, 102 KiB/s wr, 59 op/s 2024-04-23T10:26:54.328 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 23 10:26:54 smithi204 ceph-mon[32649]: pgmap v9: 97 pgs: 97 active+clean; 1.3 MiB data, 377 MiB used, 1.0 TiB / 1.0 TiB avail; 44 KiB/s rd, 84 KiB/s wr, 52 op/s 2024-04-23T10:26:54.492 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 23 10:26:54 smithi119 ceph-mon[32608]: pgmap v9: 97 pgs: 97 active+clean; 1.3 MiB data, 377 MiB used, 1.0 TiB / 1.0 TiB avail; 44 KiB/s rd, 84 KiB/s wr, 52 op/s 2024-04-23T10:26:54.503 INFO:journalctl@ceph.mon.a.smithi114.stdout:Apr 23 10:26:54 smithi114 ceph-mon[29255]: pgmap v9: 97 pgs: 97 active+clean; 1.3 MiB data, 377 MiB used, 1.0 TiB / 1.0 TiB avail; 44 KiB/s rd, 84 KiB/s wr, 52 op/s 2024-04-23T10:26:55.692 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:26:55.693+0000 7f55326a7640 1 -- 172.21.15.114:0/314148139 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f552c101a50 msgr2=0x7f552c105a90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T10:26:55.693 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:26:55.693+0000 7f55326a7640 1 --2- 172.21.15.114:0/314148139 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f552c101a50 0x7f552c105a90 secure :-1 s=READY pgs=484 cs=0 l=1 rev1=1 crypto rx=0x7f5520009900 tx=0x7f552002a410 comp rx=0 tx=0).stop 2024-04-23T10:26:55.693 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:26:55.693+0000 7f55326a7640 1 -- 172.21.15.114:0/314148139 shutdown_connections 2024-04-23T10:26:55.693 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:26:55.693+0000 7f55326a7640 1 --2- 172.21.15.114:0/314148139 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f552c106170 0x7f552c10dd40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:26:55.693 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:26:55.693+0000 7f55326a7640 1 --2- 172.21.15.114:0/314148139 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f552c101a50 0x7f552c105a90 unknown :-1 s=CLOSED pgs=484 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:26:55.693 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:26:55.693+0000 7f55326a7640 1 --2- 172.21.15.114:0/314148139 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f552c1010a0 0x7f552c101480 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:26:55.693 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:26:55.693+0000 7f55326a7640 1 -- 172.21.15.114:0/314148139 >> 172.21.15.114:0/314148139 conn(0x7f552c0fc920 msgr2=0x7f552c0fed40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T10:26:55.693 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:26:55.693+0000 7f55326a7640 1 -- 172.21.15.114:0/314148139 shutdown_connections 2024-04-23T10:26:55.693 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:26:55.693+0000 7f55326a7640 1 -- 172.21.15.114:0/314148139 wait complete. 2024-04-23T10:26:55.693 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:26:55.694+0000 7f55326a7640 1 Processor -- start 2024-04-23T10:26:55.693 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:26:55.694+0000 7f55326a7640 1 -- start start 2024-04-23T10:26:55.695 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:26:55.694+0000 7f55326a7640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f552c1010a0 0x7f552c1986d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T10:26:55.695 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:26:55.694+0000 7f55326a7640 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f552c106170 0x7f552c198c10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T10:26:55.695 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:26:55.694+0000 7f55326a7640 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f552c19cf90 0x7f552c19d440 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T10:26:55.695 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:26:55.694+0000 7f55326a7640 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x7f552c1110b0 con 0x7f552c1010a0 2024-04-23T10:26:55.695 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:26:55.694+0000 7f55326a7640 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 -- 0x7f552c110f30 con 0x7f552c19cf90 2024-04-23T10:26:55.695 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:26:55.694+0000 7f55326a7640 1 -- --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_getmap magic: 0 -- 0x7f552c111230 con 0x7f552c106170 2024-04-23T10:26:55.695 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:26:55.694+0000 7f552bfff640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f552c1010a0 0x7f552c1986d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T10:26:55.695 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:26:55.694+0000 7f552bfff640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f552c1010a0 0x7f552c1986d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.114:38904/0 (socket says 172.21.15.114:38904) 2024-04-23T10:26:55.695 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:26:55.694+0000 7f552bfff640 1 -- 172.21.15.114:0/769441257 learned_addr learned my addr 172.21.15.114:0/769441257 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-04-23T10:26:55.696 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:26:55.694+0000 7f552b7fe640 1 --2- 172.21.15.114:0/769441257 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f552c106170 0x7f552c198c10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T10:26:55.696 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:26:55.694+0000 7f5530c44640 1 --2- 172.21.15.114:0/769441257 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f552c19cf90 0x7f552c19d440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T10:26:55.696 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:26:55.694+0000 7f552bfff640 1 -- 172.21.15.114:0/769441257 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f552c106170 msgr2=0x7f552c198c10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T10:26:55.696 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:26:55.694+0000 7f552bfff640 1 --2- 172.21.15.114:0/769441257 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f552c106170 0x7f552c198c10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:26:55.696 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:26:55.694+0000 7f552bfff640 1 -- 172.21.15.114:0/769441257 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f552c19cf90 msgr2=0x7f552c19d440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T10:26:55.696 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:26:55.694+0000 7f552bfff640 1 --2- 172.21.15.114:0/769441257 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f552c19cf90 0x7f552c19d440 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:26:55.696 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:26:55.694+0000 7f552bfff640 1 -- 172.21.15.114:0/769441257 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f552c19da80 con 0x7f552c1010a0 2024-04-23T10:26:55.696 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:26:55.694+0000 7f5530c44640 1 --2- 172.21.15.114:0/769441257 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f552c19cf90 0x7f552c19d440 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-23T10:26:55.696 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:26:55.695+0000 7f552bfff640 1 --2- 172.21.15.114:0/769441257 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f552c1010a0 0x7f552c1986d0 secure :-1 s=READY pgs=485 cs=0 l=1 rev1=1 crypto rx=0x7f551c00e9f0 tx=0x7f551c00eeb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T10:26:55.696 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:26:55.695+0000 7f55297fa640 1 -- 172.21.15.114:0/769441257 <== mon.0 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f551c00cd80 con 0x7f552c1010a0 2024-04-23T10:26:55.696 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:26:55.695+0000 7f55326a7640 1 -- 172.21.15.114:0/769441257 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f552c19dd70 con 0x7f552c1010a0 2024-04-23T10:26:55.696 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:26:55.695+0000 7f55297fa640 1 -- 172.21.15.114:0/769441257 <== mon.0 v2:172.21.15.114:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f551c0108e0 con 0x7f552c1010a0 2024-04-23T10:26:55.696 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:26:55.695+0000 7f55297fa640 1 -- 172.21.15.114:0/769441257 <== mon.0 v2:172.21.15.114:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f551c018980 con 0x7f552c1010a0 2024-04-23T10:26:55.696 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:26:55.695+0000 7f55326a7640 1 -- 172.21.15.114:0/769441257 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f552c1a5530 con 0x7f552c1010a0 2024-04-23T10:26:55.702 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:26:55.697+0000 7f55297fa640 1 -- 172.21.15.114:0/769441257 <== mon.0 v2:172.21.15.114:3300/0 4 ==== mgrmap(e 24) ==== 138823+0+0 (secure 0 0 0) 0x7f551c010430 con 0x7f552c1010a0 2024-04-23T10:26:55.702 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:26:55.697+0000 7f55353c0740 1 Processor -- start 2024-04-23T10:26:55.703 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:26:55.698+0000 7f55353c0740 1 -- start start 2024-04-23T10:26:55.703 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:26:55.698+0000 7f55297fa640 1 --2- 172.21.15.114:0/769441257 >> [v2:172.21.15.114:6800/1395693466,v1:172.21.15.114:6801/1395693466] conn(0x7f55140a3e60 0x7f55140a6320 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T10:26:55.703 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:26:55.698+0000 7f55353c0740 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x561dde6e6620 0x561dde6e6a00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T10:26:55.703 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:26:55.698+0000 7f55353c0740 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x561dde6e6f40 0x561dde6f12c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T10:26:55.703 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:26:55.698+0000 7f55353c0740 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x561dde6f1800 0x561dde6f3bf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T10:26:55.703 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:26:55.698+0000 7f55297fa640 1 -- 172.21.15.114:0/769441257 <== mon.0 v2:172.21.15.114:3300/0 5 ==== osd_map(90..90 src has 1..90) ==== 8527+0+0 (secure 0 0 0) 0x7f551c0d0d30 con 0x7f552c1010a0 2024-04-23T10:26:55.703 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:26:55.698+0000 7f55353c0740 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x561dde64b970 con 0x561dde6e6f40 2024-04-23T10:26:55.703 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:26:55.698+0000 7f55353c0740 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 -- 0x561dde5e84a0 con 0x561dde6f1800 2024-04-23T10:26:55.703 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:26:55.698+0000 7f55353c0740 1 -- --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_getmap magic: 0 -- 0x561dde647930 con 0x561dde6e6620 2024-04-23T10:26:55.703 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:26:55.698+0000 7f5530c44640 1 --2- 172.21.15.114:0/769441257 >> [v2:172.21.15.114:6800/1395693466,v1:172.21.15.114:6801/1395693466] conn(0x7f55140a3e60 0x7f55140a6320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T10:26:55.703 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:26:55.698+0000 7f552b7fe640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x561dde6e6f40 0x561dde6f12c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T10:26:55.703 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:26:55.698+0000 7f552b7fe640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x561dde6e6f40 0x561dde6f12c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.114:38920/0 (socket says 172.21.15.114:38920) 2024-04-23T10:26:55.703 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:26:55.698+0000 7f552b7fe640 1 -- 172.21.15.114:0/3625985179 learned_addr learned my addr 172.21.15.114:0/3625985179 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-04-23T10:26:55.703 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:26:55.698+0000 7f552bfff640 1 --2- 172.21.15.114:0/3625985179 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x561dde6e6620 0x561dde6e6a00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T10:26:55.704 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:26:55.698+0000 7f552b7fe640 1 -- 172.21.15.114:0/3625985179 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x561dde6e6620 msgr2=0x561dde6e6a00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T10:26:55.704 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:26:55.698+0000 7f552b7fe640 1 --2- 172.21.15.114:0/3625985179 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x561dde6e6620 0x561dde6e6a00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:26:55.704 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:26:55.698+0000 7f552b7fe640 1 -- 172.21.15.114:0/3625985179 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x561dde6f1800 msgr2=0x561dde6f3bf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T10:26:55.704 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:26:55.698+0000 7f552b7fe640 1 --2- 172.21.15.114:0/3625985179 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x561dde6f1800 0x561dde6f3bf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:26:55.704 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:26:55.698+0000 7f552b7fe640 1 -- 172.21.15.114:0/3625985179 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x561dde35e210 con 0x561dde6e6f40 2024-04-23T10:26:55.704 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:26:55.698+0000 7f5530c44640 1 --2- 172.21.15.114:0/769441257 >> [v2:172.21.15.114:6800/1395693466,v1:172.21.15.114:6801/1395693466] conn(0x7f55140a3e60 0x7f55140a6320 secure :-1 s=READY pgs=62 cs=0 l=1 rev1=1 crypto rx=0x7f55240092f0 tx=0x7f5524010740 comp rx=0 tx=0).ready entity=mgr.15291 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T10:26:55.704 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:26:55.698+0000 7f552b7fe640 1 --2- 172.21.15.114:0/3625985179 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x561dde6e6f40 0x561dde6f12c0 secure :-1 s=READY pgs=486 cs=0 l=1 rev1=1 crypto rx=0x7f5520009900 tx=0x7f5520032b20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=11f1063705da44f4 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T10:26:55.704 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:26:55.699+0000 7f55127fc640 1 -- 172.21.15.114:0/3625985179 <== mon.0 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5520040070 con 0x561dde6e6f40 2024-04-23T10:26:55.704 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:26:55.699+0000 7f55127fc640 1 -- 172.21.15.114:0/3625985179 <== mon.0 v2:172.21.15.114:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f5520004030 con 0x561dde6e6f40 2024-04-23T10:26:55.704 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:26:55.699+0000 7f55127fc640 1 -- 172.21.15.114:0/3625985179 <== mon.0 v2:172.21.15.114:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5520035040 con 0x561dde6e6f40 2024-04-23T10:26:55.704 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:26:55.699+0000 7f55353c0740 1 -- 172.21.15.114:0/3625985179 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x561dde6e6f40 msgr2=0x561dde6f12c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T10:26:55.704 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:26:55.699+0000 7f55353c0740 1 --2- 172.21.15.114:0/3625985179 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x561dde6e6f40 0x561dde6f12c0 secure :-1 s=READY pgs=486 cs=0 l=1 rev1=1 crypto rx=0x7f5520009900 tx=0x7f5520032b20 comp rx=0 tx=0).stop 2024-04-23T10:26:55.704 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:26:55.699+0000 7f55353c0740 1 -- 172.21.15.114:0/3625985179 shutdown_connections 2024-04-23T10:26:55.704 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:26:55.699+0000 7f55353c0740 1 --2- 172.21.15.114:0/3625985179 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x561dde6f1800 0x561dde6f3bf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:26:55.705 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:26:55.699+0000 7f55353c0740 1 --2- 172.21.15.114:0/3625985179 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x561dde6e6f40 0x561dde6f12c0 unknown :-1 s=CLOSED pgs=486 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:26:55.705 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:26:55.699+0000 7f55353c0740 1 --2- 172.21.15.114:0/3625985179 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x561dde6e6620 0x561dde6e6a00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:26:55.705 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:26:55.699+0000 7f55353c0740 1 -- 172.21.15.114:0/3625985179 >> 172.21.15.114:0/3625985179 conn(0x561dde6e2b30 msgr2=0x561dde6e2f10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T10:26:55.705 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:26:55.699+0000 7f55353c0740 1 -- 172.21.15.114:0/3625985179 shutdown_connections 2024-04-23T10:26:55.705 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:26:55.699+0000 7f55353c0740 1 -- 172.21.15.114:0/3625985179 wait complete. 2024-04-23T10:26:55.705 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:26:55.700+0000 7f55353c0740 1 -- start start 2024-04-23T10:26:55.705 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:26:55.700+0000 7f55353c0740 1 Processor -- start 2024-04-23T10:26:55.705 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:26:55.700+0000 7f55353c0740 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x561dde6e6620 0x561dde6f1230 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T10:26:55.705 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:26:55.700+0000 7f55353c0740 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x561dde6e6f40 0x561dde6ee520 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T10:26:55.705 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:26:55.700+0000 7f55353c0740 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x561dde6f1800 0x561dde6eec60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T10:26:55.705 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:26:55.700+0000 7f55353c0740 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x561dde6f43d0 con 0x561dde6e6620 2024-04-23T10:26:55.705 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:26:55.700+0000 7f55353c0740 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 -- 0x561dde5e60b0 con 0x561dde6e6f40 2024-04-23T10:26:55.705 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:26:55.700+0000 7f55353c0740 1 -- --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_getmap magic: 0 -- 0x561dde6f4550 con 0x561dde6f1800 2024-04-23T10:26:55.705 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:26:55.700+0000 7f55353c0740 10 client.0 _ll_register_callbacks cb 0x561dde5d3d30 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-23T10:26:55.705 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:26:55.700+0000 7f552bfff640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x561dde6e6620 0x561dde6f1230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T10:26:55.706 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:26:55.700+0000 7f552bfff640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x561dde6e6620 0x561dde6f1230 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.114:38934/0 (socket says 172.21.15.114:38934) 2024-04-23T10:26:55.706 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:26:55.700+0000 7f552bfff640 1 -- 172.21.15.114:0/779590417 learned_addr learned my addr 172.21.15.114:0/779590417 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-04-23T10:26:55.706 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:26:55.700+0000 7f5530c44640 1 --2- 172.21.15.114:0/779590417 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x561dde6f1800 0x561dde6eec60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T10:26:55.706 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:26:55.700+0000 7f552b7fe640 1 --2- 172.21.15.114:0/779590417 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x561dde6e6f40 0x561dde6ee520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T10:26:55.706 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:26:55.701+0000 7f552bfff640 1 -- 172.21.15.114:0/779590417 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x561dde6f1800 msgr2=0x561dde6eec60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T10:26:55.706 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:26:55.701+0000 7f552bfff640 1 --2- 172.21.15.114:0/779590417 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x561dde6f1800 0x561dde6eec60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:26:55.706 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:26:55.701+0000 7f552bfff640 1 -- 172.21.15.114:0/779590417 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x561dde6e6f40 msgr2=0x561dde6ee520 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T10:26:55.706 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:26:55.701+0000 7f552bfff640 1 --2- 172.21.15.114:0/779590417 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x561dde6e6f40 0x561dde6ee520 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:26:55.706 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:26:55.701+0000 7f552bfff640 1 -- 172.21.15.114:0/779590417 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({osdmap=0}) -- 0x561dde42a620 con 0x561dde6e6620 2024-04-23T10:26:55.706 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:26:55.701+0000 7f5530c44640 1 --2- 172.21.15.114:0/779590417 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x561dde6f1800 0x561dde6eec60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-23T10:26:55.706 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:26:55.701+0000 7f552bfff640 1 --2- 172.21.15.114:0/779590417 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x561dde6e6620 0x561dde6f1230 secure :-1 s=READY pgs=487 cs=0 l=1 rev1=1 crypto rx=0x7f551c00c6a0 tx=0x7f551c010860 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8173e95f3bef62ca server_cookie=0 in_seq=0 out_seq=0 2024-04-23T10:26:55.706 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:26:55.701+0000 7f550b7fe640 10 client.0 ms_handle_connect on v2:172.21.15.114:3300/0 2024-04-23T10:26:55.706 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:26:55.701+0000 7f550b7fe640 1 -- 172.21.15.114:0/779590417 <== mon.0 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f551c0a57d0 con 0x561dde6e6620 2024-04-23T10:26:55.706 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:26:55.701+0000 7f550b7fe640 1 -- 172.21.15.114:0/779590417 <== mon.0 v2:172.21.15.114:3300/0 2 ==== osd_map(90..90 src has 1..90) ==== 8527+0+0 (secure 0 0 0) 0x7f551c0ce670 con 0x561dde6e6620 2024-04-23T10:26:55.707 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:26:55.701+0000 7f55353c0740 1 -- 172.21.15.114:0/779590417 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f5520004030 con 0x561dde6e6620 2024-04-23T10:26:55.707 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:26:55.701+0000 7f550b7fe640 1 -- 172.21.15.114:0/779590417 <== mon.0 v2:172.21.15.114:3300/0 3 ==== mon_get_version_reply(handle=1 version=21) ==== 24+0+0 (secure 0 0 0) 0x7f551c0ce870 con 0x561dde6e6620 2024-04-23T10:26:55.707 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:26:55.701+0000 7f55353c0740 10 client.15564 fetch_fsmap learned FSMap version 21 2024-04-23T10:26:55.707 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:26:55.701+0000 7f55353c0740 1 -- 172.21.15.114:0/779590417 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({config=0+,fsmap=21,monmap=4+}) -- 0x561dde7003b0 con 0x561dde6e6620 2024-04-23T10:26:55.707 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:26:55.701+0000 7f550b7fe640 1 -- 172.21.15.114:0/779590417 <== mon.0 v2:172.21.15.114:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f5520004030 con 0x561dde6e6620 2024-04-23T10:26:55.707 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:26:55.701+0000 7f550b7fe640 1 -- 172.21.15.114:0/779590417 <== mon.0 v2:172.21.15.114:3300/0 5 ==== fsmap(e 21) ==== 8338+0+0 (secure 0 0 0) 0x7f551c0988b0 con 0x561dde6e6620 2024-04-23T10:26:55.707 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:26:55.701+0000 7f55353c0740 10 client.15564 fetch_fsmap finished waiting for FSMap version 21 2024-04-23T10:26:55.707 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:26:55.701+0000 7f55353c0740 10 client.15564 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-04-23T10:26:55.707 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:26:55.701+0000 7f55353c0740 20 client.15564 populate_metadata read hostname 'smithi114' 2024-04-23T10:26:55.707 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:26:55.701+0000 7f55353c0740 1 --2- 172.21.15.114:0/779590417 >> [v2:172.21.15.119:6838/937297340,v1:172.21.15.119:6839/937297340] conn(0x561dde701870 0x561dde703c60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T10:26:55.708 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:26:55.702+0000 7f55353c0740 4 client.15564 mds_command: new command op to 24545 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-23T10:26:55.708 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:26:55.702+0000 7f55353c0740 1 -- 172.21.15.114:0/779590417 --> [v2:172.21.15.119:6838/937297340,v1:172.21.15.119:6839/937297340] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x561dde659390 con 0x561dde701870 2024-04-23T10:26:55.708 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:26:55.702+0000 7f552b7fe640 1 --2- 172.21.15.114:0/779590417 >> [v2:172.21.15.119:6838/937297340,v1:172.21.15.119:6839/937297340] conn(0x561dde701870 0x561dde703c60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T10:26:55.708 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:26:55.702+0000 7f552b7fe640 1 --2- 172.21.15.114:0/779590417 >> [v2:172.21.15.119:6838/937297340,v1:172.21.15.119:6839/937297340] conn(0x561dde701870 0x561dde703c60 crc :-1 s=READY pgs=39 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=f4b6927f29c122dc server_cookie=b107356ddc6f7e93 in_seq=0 out_seq=0 2024-04-23T10:26:55.708 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:26:55.702+0000 7f550b7fe640 10 client.15564 ms_handle_connect on v2:172.21.15.119:6838/937297340 2024-04-23T10:26:55.708 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:26:55.703+0000 7f550b7fe640 1 -- 172.21.15.114:0/779590417 <== mds.0 v2:172.21.15.119:6838/937297340 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x561dde659390 con 0x561dde701870 2024-04-23T10:26:55.708 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:26:55.703+0000 7f550b7fe640 10 client.15564 handle_command_reply: tid=0 2024-04-23T10:26:55.708 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:26:55.703+0000 7f55353c0740 1 client.15564 shutdown 2024-04-23T10:26:55.708 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:26:55.703+0000 7f55353c0740 1 -- 172.21.15.114:0/779590417 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x561dde6e6620 msgr2=0x561dde6f1230 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T10:26:55.708 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:26:55.703+0000 7f55353c0740 1 --2- 172.21.15.114:0/779590417 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x561dde6e6620 0x561dde6f1230 secure :-1 s=READY pgs=487 cs=0 l=1 rev1=1 crypto rx=0x7f551c00c6a0 tx=0x7f551c010860 comp rx=0 tx=0).stop 2024-04-23T10:26:55.708 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:26:55.703+0000 7f55353c0740 1 -- 172.21.15.114:0/779590417 shutdown_connections 2024-04-23T10:26:55.708 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:26:55.703+0000 7f55353c0740 1 --2- 172.21.15.114:0/779590417 >> [v2:172.21.15.119:6838/937297340,v1:172.21.15.119:6839/937297340] conn(0x561dde701870 0x561dde703c60 crc :-1 s=READY pgs=39 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:26:55.708 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:26:55.703+0000 7f55353c0740 1 --2- 172.21.15.114:0/779590417 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x561dde6f1800 0x561dde6eec60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:26:55.708 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:26:55.703+0000 7f550b7fe640 0 client.15564 ms_handle_reset on v2:172.21.15.119:6838/937297340 2024-04-23T10:26:55.708 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:26:55.703+0000 7f55353c0740 1 --2- 172.21.15.114:0/779590417 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x561dde6e6f40 0x561dde6ee520 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:26:55.709 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:26:55.703+0000 7f55353c0740 1 --2- 172.21.15.114:0/779590417 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x561dde6e6620 0x561dde6f1230 unknown :-1 s=CLOSED pgs=487 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:26:55.709 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:26:55.703+0000 7f55353c0740 1 -- 172.21.15.114:0/779590417 >> 172.21.15.114:0/779590417 conn(0x561dde6e2b30 msgr2=0x561dde6e8860 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T10:26:55.709 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:26:55.703+0000 7f55353c0740 1 -- 172.21.15.114:0/779590417 shutdown_connections 2024-04-23T10:26:55.709 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:26:55.703+0000 7f55353c0740 1 -- 172.21.15.114:0/779590417 wait complete. 2024-04-23T10:26:55.709 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:26:55.703+0000 7f55353c0740 20 client.15564 trim_cache size 0 max 16384 2024-04-23T10:26:55.719 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:26:55.719+0000 7f55353c0740 1 Processor -- start 2024-04-23T10:26:55.719 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:26:55.719+0000 7f55353c0740 1 -- start start 2024-04-23T10:26:55.719 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:26:55.719+0000 7f55353c0740 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x561dde6e6620 0x561dde6e2430 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T10:26:55.719 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:26:55.719+0000 7f55353c0740 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x561dde6e6f40 0x561dde6f0d60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T10:26:55.720 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:26:55.719+0000 7f55353c0740 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x561dde6f1800 0x561dde6f12a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T10:26:55.720 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:26:55.719+0000 7f55353c0740 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x7f552c1b0470 con 0x561dde6f1800 2024-04-23T10:26:55.720 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:26:55.719+0000 7f55353c0740 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 -- 0x7f552c1add40 con 0x561dde6e6f40 2024-04-23T10:26:55.720 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:26:55.719+0000 7f55353c0740 1 -- --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_getmap magic: 0 -- 0x7f552c1abb90 con 0x561dde6e6620 2024-04-23T10:26:55.720 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:26:55.719+0000 7f5530c44640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x561dde6f1800 0x561dde6f12a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T10:26:55.720 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:26:55.719+0000 7f5530c44640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x561dde6f1800 0x561dde6f12a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.114:38938/0 (socket says 172.21.15.114:38938) 2024-04-23T10:26:55.720 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:26:55.719+0000 7f5530c44640 1 -- 172.21.15.114:0/4140795271 learned_addr learned my addr 172.21.15.114:0/4140795271 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-04-23T10:26:55.720 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:26:55.719+0000 7f552b7fe640 1 --2- 172.21.15.114:0/4140795271 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x561dde6e6f40 0x561dde6f0d60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T10:26:55.720 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:26:55.719+0000 7f552bfff640 1 --2- 172.21.15.114:0/4140795271 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x561dde6e6620 0x561dde6e2430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T10:26:55.720 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:26:55.719+0000 7f5530c44640 1 -- 172.21.15.114:0/4140795271 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x561dde6e6620 msgr2=0x561dde6e2430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T10:26:55.720 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:26:55.719+0000 7f5530c44640 1 --2- 172.21.15.114:0/4140795271 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x561dde6e6620 0x561dde6e2430 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:26:55.720 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:26:55.719+0000 7f5530c44640 1 -- 172.21.15.114:0/4140795271 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x561dde6e6f40 msgr2=0x561dde6f0d60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T10:26:55.720 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:26:55.719+0000 7f5530c44640 1 --2- 172.21.15.114:0/4140795271 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x561dde6e6f40 0x561dde6f0d60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:26:55.720 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:26:55.719+0000 7f5530c44640 1 -- 172.21.15.114:0/4140795271 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x561dde6de500 con 0x561dde6f1800 2024-04-23T10:26:55.720 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:26:55.719+0000 7f552b7fe640 1 --2- 172.21.15.114:0/4140795271 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x561dde6e6f40 0x561dde6f0d60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-23T10:26:55.721 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:26:55.719+0000 7f5530c44640 1 --2- 172.21.15.114:0/4140795271 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x561dde6f1800 0x561dde6f12a0 secure :-1 s=READY pgs=488 cs=0 l=1 rev1=1 crypto rx=0x7f55240127c0 tx=0x7f5524015000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=429755c1b0a21956 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T10:26:55.721 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:26:55.720+0000 7f5511ffb640 1 -- 172.21.15.114:0/4140795271 <== mon.0 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5524011960 con 0x561dde6f1800 2024-04-23T10:26:55.721 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:26:55.720+0000 7f5511ffb640 1 -- 172.21.15.114:0/4140795271 <== mon.0 v2:172.21.15.114:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f5524017040 con 0x561dde6f1800 2024-04-23T10:26:55.721 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:26:55.720+0000 7f5511ffb640 1 -- 172.21.15.114:0/4140795271 <== mon.0 v2:172.21.15.114:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5524021df0 con 0x561dde6f1800 2024-04-23T10:26:55.721 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:26:55.720+0000 7f55353c0740 1 -- 172.21.15.114:0/4140795271 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x561dde6f1800 msgr2=0x561dde6f12a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T10:26:55.721 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:26:55.720+0000 7f55353c0740 1 --2- 172.21.15.114:0/4140795271 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x561dde6f1800 0x561dde6f12a0 secure :-1 s=READY pgs=488 cs=0 l=1 rev1=1 crypto rx=0x7f55240127c0 tx=0x7f5524015000 comp rx=0 tx=0).stop 2024-04-23T10:26:55.721 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:26:55.720+0000 7f55353c0740 1 -- 172.21.15.114:0/4140795271 shutdown_connections 2024-04-23T10:26:55.721 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:26:55.720+0000 7f55353c0740 1 --2- 172.21.15.114:0/4140795271 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x561dde6f1800 0x561dde6f12a0 unknown :-1 s=CLOSED pgs=488 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:26:55.721 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:26:55.720+0000 7f55353c0740 1 --2- 172.21.15.114:0/4140795271 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x561dde6e6f40 0x561dde6f0d60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:26:55.721 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:26:55.720+0000 7f55353c0740 1 --2- 172.21.15.114:0/4140795271 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x561dde6e6620 0x561dde6e2430 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:26:55.721 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:26:55.720+0000 7f55353c0740 1 -- 172.21.15.114:0/4140795271 >> 172.21.15.114:0/4140795271 conn(0x561dde6e2b30 msgr2=0x561dde6e99d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T10:26:55.721 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:26:55.720+0000 7f55353c0740 1 -- 172.21.15.114:0/4140795271 shutdown_connections 2024-04-23T10:26:55.721 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:26:55.720+0000 7f55353c0740 1 -- 172.21.15.114:0/4140795271 wait complete. 2024-04-23T10:26:55.721 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:26:55.721+0000 7f55353c0740 1 -- start start 2024-04-23T10:26:55.724 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:26:55.721+0000 7f55353c0740 1 Processor -- start 2024-04-23T10:26:55.724 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:26:55.721+0000 7f55353c0740 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x561dde6e6620 0x561dde6eb540 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T10:26:55.724 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:26:55.721+0000 7f55353c0740 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x561dde6e6f40 0x561dde6eba80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T10:26:55.724 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:26:55.721+0000 7f55353c0740 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x561dde6f1800 0x561dde766dc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T10:26:55.725 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:26:55.721+0000 7f55353c0740 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x7f552c1ac200 con 0x561dde6e6f40 2024-04-23T10:26:55.725 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:26:55.721+0000 7f55353c0740 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 -- 0x7f552c1ad670 con 0x561dde6f1800 2024-04-23T10:26:55.725 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:26:55.721+0000 7f55353c0740 1 -- --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_getmap magic: 0 -- 0x7f552c1ac8d0 con 0x561dde6e6620 2024-04-23T10:26:55.725 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:26:55.721+0000 7f55353c0740 10 client.0 _ll_register_callbacks cb 0x7f551c0ce250 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-23T10:26:55.725 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:26:55.721+0000 7f552b7fe640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x561dde6e6f40 0x561dde6eba80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T10:26:55.725 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:26:55.721+0000 7f552b7fe640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x561dde6e6f40 0x561dde6eba80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.114:38954/0 (socket says 172.21.15.114:38954) 2024-04-23T10:26:55.725 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:26:55.721+0000 7f552b7fe640 1 -- 172.21.15.114:0/3957344963 learned_addr learned my addr 172.21.15.114:0/3957344963 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-04-23T10:26:55.725 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:26:55.721+0000 7f552bfff640 1 --2- 172.21.15.114:0/3957344963 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x561dde6e6620 0x561dde6eb540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T10:26:55.725 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:26:55.722+0000 7f5530c44640 1 --2- 172.21.15.114:0/3957344963 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x561dde6f1800 0x561dde766dc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T10:26:55.725 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:26:55.722+0000 7f552b7fe640 1 -- 172.21.15.114:0/3957344963 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x561dde6e6620 msgr2=0x561dde6eb540 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T10:26:55.725 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:26:55.722+0000 7f552b7fe640 1 --2- 172.21.15.114:0/3957344963 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x561dde6e6620 0x561dde6eb540 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:26:55.725 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:26:55.722+0000 7f552b7fe640 1 -- 172.21.15.114:0/3957344963 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x561dde6f1800 msgr2=0x561dde766dc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T10:26:55.725 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:26:55.722+0000 7f552b7fe640 1 --2- 172.21.15.114:0/3957344963 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x561dde6f1800 0x561dde766dc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:26:55.725 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:26:55.722+0000 7f552b7fe640 1 -- 172.21.15.114:0/3957344963 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({osdmap=0}) -- 0x561dde767300 con 0x561dde6e6f40 2024-04-23T10:26:55.726 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:26:55.722+0000 7f5530c44640 1 --2- 172.21.15.114:0/3957344963 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x561dde6f1800 0x561dde766dc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-23T10:26:55.726 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:26:55.722+0000 7f552b7fe640 1 --2- 172.21.15.114:0/3957344963 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x561dde6e6f40 0x561dde6eba80 secure :-1 s=READY pgs=489 cs=0 l=1 rev1=1 crypto rx=0x561dde6e7bc0 tx=0x7f552002ab10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a175764dc783e33 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T10:26:55.726 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:26:55.722+0000 7f550b7fe640 10 client.0 ms_handle_connect on v2:172.21.15.114:3300/0 2024-04-23T10:26:55.726 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:26:55.722+0000 7f550b7fe640 1 -- 172.21.15.114:0/3957344963 <== mon.0 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5520054060 con 0x561dde6e6f40 2024-04-23T10:26:55.726 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:26:55.722+0000 7f550b7fe640 1 -- 172.21.15.114:0/3957344963 <== mon.0 v2:172.21.15.114:3300/0 2 ==== osd_map(90..90 src has 1..90) ==== 8527+0+0 (secure 0 0 0) 0x7f5520004040 con 0x561dde6e6f40 2024-04-23T10:26:55.726 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:26:55.722+0000 7f55353c0740 1 -- 172.21.15.114:0/3957344963 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f5524017040 con 0x561dde6e6f40 2024-04-23T10:26:55.726 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:26:55.722+0000 7f550b7fe640 1 -- 172.21.15.114:0/3957344963 <== mon.0 v2:172.21.15.114:3300/0 3 ==== mon_get_version_reply(handle=1 version=21) ==== 24+0+0 (secure 0 0 0) 0x7f5520040070 con 0x561dde6e6f40 2024-04-23T10:26:55.726 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:26:55.722+0000 7f55353c0740 10 client.15570 fetch_fsmap learned FSMap version 21 2024-04-23T10:26:55.726 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:26:55.722+0000 7f55353c0740 1 -- 172.21.15.114:0/3957344963 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({config=0+,fsmap=21,monmap=4+}) -- 0x561dde768470 con 0x561dde6e6f40 2024-04-23T10:26:55.726 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:26:55.722+0000 7f550b7fe640 1 -- 172.21.15.114:0/3957344963 <== mon.0 v2:172.21.15.114:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f5524017040 con 0x561dde6e6f40 2024-04-23T10:26:55.726 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:26:55.722+0000 7f550b7fe640 1 -- 172.21.15.114:0/3957344963 <== mon.0 v2:172.21.15.114:3300/0 5 ==== fsmap(e 21) ==== 8338+0+0 (secure 0 0 0) 0x7f5520005040 con 0x561dde6e6f40 2024-04-23T10:26:55.726 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:26:55.722+0000 7f55353c0740 10 client.15570 fetch_fsmap finished waiting for FSMap version 21 2024-04-23T10:26:55.726 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:26:55.722+0000 7f55353c0740 10 client.15570 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-04-23T10:26:55.726 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:26:55.722+0000 7f55353c0740 20 client.15570 populate_metadata read hostname 'smithi114' 2024-04-23T10:26:55.727 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:26:55.722+0000 7f55353c0740 1 --2- 172.21.15.114:0/3957344963 >> [v2:172.21.15.119:6838/937297340,v1:172.21.15.119:6839/937297340] conn(0x561dde768850 0x561dde76ac70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T10:26:55.727 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:26:55.722+0000 7f55353c0740 4 client.15570 mds_command: new command op to 24545 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-04-23T10:26:55.727 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:26:55.723+0000 7f55353c0740 1 -- 172.21.15.114:0/3957344963 --> [v2:172.21.15.119:6838/937297340,v1:172.21.15.119:6839/937297340] -- command(tid 0: {"prefix": "flush journal"}) -- 0x561dde653200 con 0x561dde768850 2024-04-23T10:26:55.727 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:26:55.723+0000 7f552bfff640 1 --2- 172.21.15.114:0/3957344963 >> [v2:172.21.15.119:6838/937297340,v1:172.21.15.119:6839/937297340] conn(0x561dde768850 0x561dde76ac70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T10:26:55.727 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:26:55.723+0000 7f552bfff640 1 --2- 172.21.15.114:0/3957344963 >> [v2:172.21.15.119:6838/937297340,v1:172.21.15.119:6839/937297340] conn(0x561dde768850 0x561dde76ac70 crc :-1 s=READY pgs=40 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=c3493409e8cd400c server_cookie=e146550d9bde379d in_seq=0 out_seq=0 2024-04-23T10:26:55.727 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:26:55.723+0000 7f550b7fe640 10 client.15570 ms_handle_connect on v2:172.21.15.119:6838/937297340 2024-04-23T10:26:55.734 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:26:55.734+0000 7f550b7fe640 1 -- 172.21.15.114:0/3957344963 <== mds.0 v2:172.21.15.119:6838/937297340 1 ==== command_reply(tid 0: 0 ) ==== 8+0+44 (crc 0 0 0) 0x561dde653200 con 0x561dde768850 2024-04-23T10:26:55.734 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:26:55.735+0000 7f550b7fe640 10 client.15570 handle_command_reply: tid=0 2024-04-23T10:26:55.734 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:26:55.735+0000 7f55353c0740 1 client.15570 shutdown 2024-04-23T10:26:55.734 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:26:55.735+0000 7f55353c0740 1 -- 172.21.15.114:0/3957344963 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x561dde6e6f40 msgr2=0x561dde6eba80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T10:26:55.734 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:26:55.735+0000 7f55353c0740 1 --2- 172.21.15.114:0/3957344963 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x561dde6e6f40 0x561dde6eba80 secure :-1 s=READY pgs=489 cs=0 l=1 rev1=1 crypto rx=0x561dde6e7bc0 tx=0x7f552002ab10 comp rx=0 tx=0).stop 2024-04-23T10:26:55.735 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:26:55.735+0000 7f55353c0740 1 -- 172.21.15.114:0/3957344963 shutdown_connections 2024-04-23T10:26:55.735 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:26:55.735+0000 7f55353c0740 1 --2- 172.21.15.114:0/3957344963 >> [v2:172.21.15.119:6838/937297340,v1:172.21.15.119:6839/937297340] conn(0x561dde768850 0x561dde76ac70 crc :-1 s=READY pgs=40 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:26:55.735 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:26:55.735+0000 7f55353c0740 1 --2- 172.21.15.114:0/3957344963 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x561dde6f1800 0x561dde766dc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:26:55.735 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:26:55.735+0000 7f55353c0740 1 --2- 172.21.15.114:0/3957344963 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x561dde6e6f40 0x561dde6eba80 unknown :-1 s=CLOSED pgs=489 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:26:55.735 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:26:55.735+0000 7f550b7fe640 0 client.15570 ms_handle_reset on v2:172.21.15.119:6838/937297340 2024-04-23T10:26:55.735 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:26:55.735+0000 7f55353c0740 1 --2- 172.21.15.114:0/3957344963 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x561dde6e6620 0x561dde6eb540 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:26:55.735 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:26:55.735+0000 7f55353c0740 1 -- 172.21.15.114:0/3957344963 >> 172.21.15.114:0/3957344963 conn(0x561dde6e2b30 msgr2=0x561dde6e7fd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T10:26:55.735 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:26:55.735+0000 7f55353c0740 1 -- 172.21.15.114:0/3957344963 shutdown_connections 2024-04-23T10:26:55.735 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:26:55.735+0000 7f55353c0740 1 -- 172.21.15.114:0/3957344963 wait complete. 2024-04-23T10:26:55.735 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:26:55.735+0000 7f55353c0740 20 client.15570 trim_cache size 0 max 16384 2024-04-23T10:26:55.735 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stdout:{ 2024-04-23T10:26:55.736 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stdout: "message": "", 2024-04-23T10:26:55.736 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stdout: "return_code": 0 2024-04-23T10:26:55.736 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stdout:} 2024-04-23T10:26:55.737 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:26:55.735+0000 7f55326a7640 1 -- 172.21.15.114:0/769441257 >> [v2:172.21.15.114:6800/1395693466,v1:172.21.15.114:6801/1395693466] conn(0x7f55140a3e60 msgr2=0x7f55140a6320 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T10:26:55.737 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:26:55.735+0000 7f55326a7640 1 --2- 172.21.15.114:0/769441257 >> [v2:172.21.15.114:6800/1395693466,v1:172.21.15.114:6801/1395693466] conn(0x7f55140a3e60 0x7f55140a6320 secure :-1 s=READY pgs=62 cs=0 l=1 rev1=1 crypto rx=0x7f55240092f0 tx=0x7f5524010740 comp rx=0 tx=0).stop 2024-04-23T10:26:55.737 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:26:55.736+0000 7f55326a7640 1 -- 172.21.15.114:0/769441257 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f552c1010a0 msgr2=0x7f552c1986d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T10:26:55.737 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:26:55.736+0000 7f55326a7640 1 --2- 172.21.15.114:0/769441257 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f552c1010a0 0x7f552c1986d0 secure :-1 s=READY pgs=485 cs=0 l=1 rev1=1 crypto rx=0x7f551c00e9f0 tx=0x7f551c00eeb0 comp rx=0 tx=0).stop 2024-04-23T10:26:55.737 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:26:55.736+0000 7f55326a7640 1 -- 172.21.15.114:0/769441257 shutdown_connections 2024-04-23T10:26:55.737 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:26:55.736+0000 7f55326a7640 1 --2- 172.21.15.114:0/769441257 >> [v2:172.21.15.114:6800/1395693466,v1:172.21.15.114:6801/1395693466] conn(0x7f55140a3e60 0x7f55140a6320 unknown :-1 s=CLOSED pgs=62 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:26:55.737 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:26:55.736+0000 7f55326a7640 1 --2- 172.21.15.114:0/769441257 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f552c19cf90 0x7f552c19d440 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:26:55.737 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:26:55.736+0000 7f55326a7640 1 --2- 172.21.15.114:0/769441257 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f552c106170 0x7f552c198c10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:26:55.737 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:26:55.736+0000 7f55326a7640 1 --2- 172.21.15.114:0/769441257 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f552c1010a0 0x7f552c1986d0 unknown :-1 s=CLOSED pgs=485 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:26:55.737 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:26:55.736+0000 7f55326a7640 1 -- 172.21.15.114:0/769441257 >> 172.21.15.114:0/769441257 conn(0x7f552c0fc920 msgr2=0x7f552c06aa30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T10:26:55.737 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:26:55.736+0000 7f55326a7640 1 -- 172.21.15.114:0/769441257 shutdown_connections 2024-04-23T10:26:55.738 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:26:55.736+0000 7f55326a7640 1 -- 172.21.15.114:0/769441257 wait complete. 2024-04-23T10:26:56.254 INFO:journalctl@ceph.mon.a.smithi114.stdout:Apr 23 10:26:56 smithi114 ceph-mon[29255]: pgmap v10: 97 pgs: 97 active+clean; 1.3 MiB data, 377 MiB used, 1.0 TiB / 1.0 TiB avail; 40 KiB/s rd, 77 KiB/s wr, 47 op/s 2024-04-23T10:26:56.328 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 23 10:26:56 smithi204 ceph-mon[32649]: pgmap v10: 97 pgs: 97 active+clean; 1.3 MiB data, 377 MiB used, 1.0 TiB / 1.0 TiB avail; 40 KiB/s rd, 77 KiB/s wr, 47 op/s 2024-04-23T10:26:56.492 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 23 10:26:56 smithi119 ceph-mon[32608]: pgmap v10: 97 pgs: 97 active+clean; 1.3 MiB data, 377 MiB used, 1.0 TiB / 1.0 TiB avail; 40 KiB/s rd, 77 KiB/s wr, 47 op/s 2024-04-23T10:26:58.328 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 23 10:26:58 smithi204 ceph-mon[32649]: pgmap v11: 97 pgs: 97 active+clean; 1.3 MiB data, 377 MiB used, 1.0 TiB / 1.0 TiB avail; 40 KiB/s rd, 77 KiB/s wr, 49 op/s 2024-04-23T10:26:58.492 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 23 10:26:58 smithi119 ceph-mon[32608]: pgmap v11: 97 pgs: 97 active+clean; 1.3 MiB data, 377 MiB used, 1.0 TiB / 1.0 TiB avail; 40 KiB/s rd, 77 KiB/s wr, 49 op/s 2024-04-23T10:26:58.503 INFO:journalctl@ceph.mon.a.smithi114.stdout:Apr 23 10:26:58 smithi114 ceph-mon[29255]: pgmap v11: 97 pgs: 97 active+clean; 1.3 MiB data, 377 MiB used, 1.0 TiB / 1.0 TiB avail; 40 KiB/s rd, 77 KiB/s wr, 49 op/s 2024-04-23T10:27:00.328 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 23 10:27:00 smithi204 ceph-mon[32649]: pgmap v12: 97 pgs: 97 active+clean; 1.3 MiB data, 377 MiB used, 1.0 TiB / 1.0 TiB avail; 24 KiB/s rd, 41 KiB/s wr, 27 op/s 2024-04-23T10:27:00.492 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 23 10:27:00 smithi119 ceph-mon[32608]: pgmap v12: 97 pgs: 97 active+clean; 1.3 MiB data, 377 MiB used, 1.0 TiB / 1.0 TiB avail; 24 KiB/s rd, 41 KiB/s wr, 27 op/s 2024-04-23T10:27:00.504 INFO:journalctl@ceph.mon.a.smithi114.stdout:Apr 23 10:27:00 smithi114 ceph-mon[29255]: pgmap v12: 97 pgs: 97 active+clean; 1.3 MiB data, 377 MiB used, 1.0 TiB / 1.0 TiB avail; 24 KiB/s rd, 41 KiB/s wr, 27 op/s 2024-04-23T10:27:01.254 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Apr 23 10:27:00 smithi114 ceph-aa240572-0159-11ef-bc93-c7b262605968-mgr-x[29490]: 2024-04-23T10:27:00.762+0000 7f211573e700 -1 client.0 error registering admin socket command: (17) File exists 2024-04-23T10:27:01.255 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Apr 23 10:27:00 smithi114 ceph-aa240572-0159-11ef-bc93-c7b262605968-mgr-x[29490]: 2024-04-23T10:27:00.762+0000 7f211573e700 -1 client.0 error registering admin socket command: (17) File exists 2024-04-23T10:27:01.255 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Apr 23 10:27:00 smithi114 ceph-aa240572-0159-11ef-bc93-c7b262605968-mgr-x[29490]: 2024-04-23T10:27:00.762+0000 7f211573e700 -1 client.0 error registering admin socket command: (17) File exists 2024-04-23T10:27:01.255 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Apr 23 10:27:00 smithi114 ceph-aa240572-0159-11ef-bc93-c7b262605968-mgr-x[29490]: 2024-04-23T10:27:00.762+0000 7f211573e700 -1 client.0 error registering admin socket command: (17) File exists 2024-04-23T10:27:01.255 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Apr 23 10:27:00 smithi114 ceph-aa240572-0159-11ef-bc93-c7b262605968-mgr-x[29490]: 2024-04-23T10:27:00.762+0000 7f211573e700 -1 client.0 error registering admin socket command: (17) File exists 2024-04-23T10:27:02.328 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 23 10:27:02 smithi204 ceph-mon[32649]: pgmap v13: 97 pgs: 97 active+clean; 1.3 MiB data, 377 MiB used, 1.0 TiB / 1.0 TiB avail; 3.1 KiB/s rd, 2.3 KiB/s wr, 9 op/s 2024-04-23T10:27:02.492 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 23 10:27:02 smithi119 ceph-mon[32608]: pgmap v13: 97 pgs: 97 active+clean; 1.3 MiB data, 377 MiB used, 1.0 TiB / 1.0 TiB avail; 3.1 KiB/s rd, 2.3 KiB/s wr, 9 op/s 2024-04-23T10:27:02.504 INFO:journalctl@ceph.mon.a.smithi114.stdout:Apr 23 10:27:02 smithi114 ceph-mon[29255]: pgmap v13: 97 pgs: 97 active+clean; 1.3 MiB data, 377 MiB used, 1.0 TiB / 1.0 TiB avail; 3.1 KiB/s rd, 2.3 KiB/s wr, 9 op/s 2024-04-23T10:27:04.202 INFO:tasks.quiescer.fs.[cephfs]:Total duration (45.4) is longer than `split_if_longer` (35.0), will split into two consecutive quiesces 2024-04-23T10:27:04.203 DEBUG:tasks.quiescer.fs.[cephfs]:Going to quiesce for duration: 22.7 2024-04-23T10:27:04.203 DEBUG:teuthology.orchestra.run.smithi114:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-23T10:27:04.308 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:27:04.307+0000 7fddaed29640 1 -- 172.21.15.114:0/1315835661 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fdda8106170 msgr2=0x7fdda810dd40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T10:27:04.309 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:27:04.307+0000 7fddaed29640 1 --2- 172.21.15.114:0/1315835661 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fdda8106170 0x7fdda810dd40 secure :-1 s=READY pgs=492 cs=0 l=1 rev1=1 crypto rx=0x7fdd98009900 tx=0x7fdd9802a410 comp rx=0 tx=0).stop 2024-04-23T10:27:04.309 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:27:04.307+0000 7fddaed29640 1 -- 172.21.15.114:0/1315835661 shutdown_connections 2024-04-23T10:27:04.309 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:27:04.307+0000 7fddaed29640 1 --2- 172.21.15.114:0/1315835661 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fdda8106170 0x7fdda810dd40 unknown :-1 s=CLOSED pgs=492 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:27:04.309 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:27:04.307+0000 7fddaed29640 1 --2- 172.21.15.114:0/1315835661 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fdda8101a50 0x7fdda8105a90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:27:04.309 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:27:04.307+0000 7fddaed29640 1 --2- 172.21.15.114:0/1315835661 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fdda81010a0 0x7fdda8101480 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:27:04.309 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:27:04.307+0000 7fddaed29640 1 -- 172.21.15.114:0/1315835661 >> 172.21.15.114:0/1315835661 conn(0x7fdda8077f50 msgr2=0x7fdda8078360 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T10:27:04.309 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:27:04.307+0000 7fddaed29640 1 -- 172.21.15.114:0/1315835661 shutdown_connections 2024-04-23T10:27:04.309 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:27:04.307+0000 7fddaed29640 1 -- 172.21.15.114:0/1315835661 wait complete. 2024-04-23T10:27:04.309 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:27:04.308+0000 7fddaed29640 1 Processor -- start 2024-04-23T10:27:04.309 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:27:04.308+0000 7fddaed29640 1 -- start start 2024-04-23T10:27:04.309 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:27:04.308+0000 7fddaed29640 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fdda81010a0 0x7fdda81986c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T10:27:04.309 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:27:04.308+0000 7fddaed29640 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fdda8106170 0x7fdda8198c00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T10:27:04.309 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:27:04.308+0000 7fddaed29640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fdda819cf80 0x7fdda819d430 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T10:27:04.310 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:27:04.308+0000 7fddaed29640 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x7fdda81110e0 con 0x7fdda819cf80 2024-04-23T10:27:04.310 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:27:04.308+0000 7fddaed29640 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 -- 0x7fdda8110f60 con 0x7fdda81010a0 2024-04-23T10:27:04.310 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:27:04.308+0000 7fddaed29640 1 -- --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_getmap magic: 0 -- 0x7fdda8111260 con 0x7fdda8106170 2024-04-23T10:27:04.310 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:27:04.308+0000 7fddad2c6640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fdda819cf80 0x7fdda819d430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T10:27:04.310 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:27:04.308+0000 7fddad2c6640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fdda819cf80 0x7fdda819d430 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.114:38994/0 (socket says 172.21.15.114:38994) 2024-04-23T10:27:04.310 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:27:04.308+0000 7fddad2c6640 1 -- 172.21.15.114:0/540177368 learned_addr learned my addr 172.21.15.114:0/540177368 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-04-23T10:27:04.310 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:27:04.308+0000 7fdda7fff640 1 --2- 172.21.15.114:0/540177368 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fdda8106170 0x7fdda8198c00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T10:27:04.310 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:27:04.309+0000 7fddacac5640 1 --2- 172.21.15.114:0/540177368 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fdda81010a0 0x7fdda81986c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T10:27:04.310 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:27:04.309+0000 7fddad2c6640 1 -- 172.21.15.114:0/540177368 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fdda8106170 msgr2=0x7fdda8198c00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T10:27:04.310 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:27:04.309+0000 7fddad2c6640 1 --2- 172.21.15.114:0/540177368 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fdda8106170 0x7fdda8198c00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:27:04.310 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:27:04.309+0000 7fddad2c6640 1 -- 172.21.15.114:0/540177368 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fdda81010a0 msgr2=0x7fdda81986c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T10:27:04.310 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:27:04.309+0000 7fddad2c6640 1 --2- 172.21.15.114:0/540177368 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fdda81010a0 0x7fdda81986c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:27:04.310 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:27:04.309+0000 7fddad2c6640 1 -- 172.21.15.114:0/540177368 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fdda819da70 con 0x7fdda819cf80 2024-04-23T10:27:04.311 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:27:04.309+0000 7fddad2c6640 1 --2- 172.21.15.114:0/540177368 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fdda819cf80 0x7fdda819d430 secure :-1 s=READY pgs=493 cs=0 l=1 rev1=1 crypto rx=0x7fdd98004770 tx=0x7fdd98004270 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T10:27:04.311 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:27:04.309+0000 7fdda5ffb640 1 -- 172.21.15.114:0/540177368 <== mon.0 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdd9803f030 con 0x7fdda819cf80 2024-04-23T10:27:04.311 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:27:04.309+0000 7fdda5ffb640 1 -- 172.21.15.114:0/540177368 <== mon.0 v2:172.21.15.114:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fdd98003aa0 con 0x7fdda819cf80 2024-04-23T10:27:04.311 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:27:04.309+0000 7fddaed29640 1 -- 172.21.15.114:0/540177368 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fdda819dd00 con 0x7fdda819cf80 2024-04-23T10:27:04.311 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:27:04.309+0000 7fdda5ffb640 1 -- 172.21.15.114:0/540177368 <== mon.0 v2:172.21.15.114:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdd98035070 con 0x7fdda819cf80 2024-04-23T10:27:04.311 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:27:04.309+0000 7fddaed29640 1 -- 172.21.15.114:0/540177368 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fdda81a54c0 con 0x7fdda819cf80 2024-04-23T10:27:04.312 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:27:04.310+0000 7fddaed29640 1 -- 172.21.15.114:0/540177368 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fdda8068d70 con 0x7fdda819cf80 2024-04-23T10:27:04.312 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:27:04.311+0000 7fdda5ffb640 1 -- 172.21.15.114:0/540177368 <== mon.0 v2:172.21.15.114:3300/0 4 ==== mgrmap(e 25) ==== 138880+0+0 (secure 0 0 0) 0x7fdd98042030 con 0x7fdda819cf80 2024-04-23T10:27:04.312 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:27:04.312+0000 7fdda5ffb640 1 --2- 172.21.15.114:0/540177368 >> [v2:172.21.15.114:6800/1395693466,v1:172.21.15.114:6801/1395693466] conn(0x7fdd940a3e40 0x7fdd940a6300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T10:27:04.312 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:27:04.312+0000 7fddacac5640 1 --2- 172.21.15.114:0/540177368 >> [v2:172.21.15.114:6800/1395693466,v1:172.21.15.114:6801/1395693466] conn(0x7fdd940a3e40 0x7fdd940a6300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T10:27:04.312 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:27:04.312+0000 7fdda5ffb640 1 -- 172.21.15.114:0/540177368 <== mon.0 v2:172.21.15.114:3300/0 5 ==== osd_map(90..90 src has 1..90) ==== 8527+0+0 (secure 0 0 0) 0x7fdd980f0f20 con 0x7fdda819cf80 2024-04-23T10:27:04.312 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:27:04.312+0000 7fddacac5640 1 --2- 172.21.15.114:0/540177368 >> [v2:172.21.15.114:6800/1395693466,v1:172.21.15.114:6801/1395693466] conn(0x7fdd940a3e40 0x7fdd940a6300 secure :-1 s=READY pgs=63 cs=0 l=1 rev1=1 crypto rx=0x7fdda0005fd0 tx=0x7fdda00098b0 comp rx=0 tx=0).ready entity=mgr.15291 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T10:27:04.315 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:27:04.317+0000 7fdda5ffb640 1 -- 172.21.15.114:0/540177368 <== mon.0 v2:172.21.15.114:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7fdd980bd440 con 0x7fdda819cf80 2024-04-23T10:27:04.327 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 23 10:27:04 smithi204 ceph-mon[32649]: pgmap v14: 97 pgs: 97 active+clean; 1.3 MiB data, 377 MiB used, 1.0 TiB / 1.0 TiB avail; 14 KiB/s rd, 25 KiB/s wr, 27 op/s 2024-04-23T10:27:04.328 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 23 10:27:04 smithi204 ceph-mon[32649]: mgrmap e25: x(active, since 22s), standbys: z, y 2024-04-23T10:27:04.492 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 23 10:27:04 smithi119 ceph-mon[32608]: pgmap v14: 97 pgs: 97 active+clean; 1.3 MiB data, 377 MiB used, 1.0 TiB / 1.0 TiB avail; 14 KiB/s rd, 25 KiB/s wr, 27 op/s 2024-04-23T10:27:04.492 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 23 10:27:04 smithi119 ceph-mon[32608]: mgrmap e25: x(active, since 22s), standbys: z, y 2024-04-23T10:27:04.504 INFO:journalctl@ceph.mon.a.smithi114.stdout:Apr 23 10:27:04 smithi114 ceph-mon[29255]: pgmap v14: 97 pgs: 97 active+clean; 1.3 MiB data, 377 MiB used, 1.0 TiB / 1.0 TiB avail; 14 KiB/s rd, 25 KiB/s wr, 27 op/s 2024-04-23T10:27:04.504 INFO:journalctl@ceph.mon.a.smithi114.stdout:Apr 23 10:27:04 smithi114 ceph-mon[29255]: mgrmap e25: x(active, since 22s), standbys: z, y 2024-04-23T10:27:04.580 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:27:04.581+0000 7fddaed29640 1 -- 172.21.15.114:0/540177368 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fdda8101a50 con 0x7fdda819cf80 2024-04-23T10:27:04.581 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:27:04.582+0000 7fdda5ffb640 1 -- 172.21.15.114:0/540177368 <== mon.0 v2:172.21.15.114:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 21 v21) ==== 94+0+11884 (secure 0 0 0) 0x7fdd980bcd40 con 0x7fdda819cf80 2024-04-23T10:27:04.581 INFO:teuthology.orchestra.run.smithi114.stdout: 2024-04-23T10:27:04.581 INFO:teuthology.orchestra.run.smithi114.stdout:{"epoch":21,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14547,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.114:6835/663493580","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6834","nonce":663493580},{"type":"v1","addr":"172.21.15.114:6835","nonce":663493580}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14565,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.114:6837/1439252154","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6836","nonce":1439252154},{"type":"v1","addr":"172.21.15.114:6837","nonce":1439252154}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5},{"gid":14634,"name":"g","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.114:6839/3849796939","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6838","nonce":3849796939},{"type":"v1","addr":"172.21.15.114:6839","nonce":3849796939}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":11},{"gid":14646,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.114:6841/3544523891","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6840","nonce":3544523891},{"type":"v1","addr":"172.21.15.114:6841","nonce":3544523891}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":12},{"gid":24385,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.204:6833/3866561754","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6832","nonce":3866561754},{"type":"v1","addr":"172.21.15.204:6833","nonce":3866561754}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2},{"gid":24427,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.204:6835/2533793172","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6834","nonce":2533793172},{"type":"v1","addr":"172.21.15.204:6835","nonce":2533793172}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":7}],"filesystems":[{"mdsmap":{"epoch":21,"flags":306,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":true},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-04-23T10:24:43.383158+0000","modified":"2024-04-23T10:26:38.945346+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":3,"in":[0,1,2],"up":{"mds_0":24545,"mds_1":24509,"mds_2":24479},"failed":[],"damaged":[],"stopped":[],"info":{"gid_24439":{"gid":24439,"name":"c","rank":2,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.204:6837/513822179","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6836","nonce":513822179},{"type":"v1","addr":"172.21.15.204:6837","nonce":513822179}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24443":{"gid":24443,"name":"k","rank":1,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.119:6833/783244660","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6832","nonce":783244660},{"type":"v1","addr":"172.21.15.119:6833","nonce":783244660}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24463":{"gid":24463,"name":"i","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.204:6839/3222658893","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6838","nonce":3222658893},{"type":"v1","addr":"172.21.15.204:6839","nonce":3222658893}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24479":{"gid":24479,"name":"e","rank":2,"incarnation":19,"state":"up:active","state_seq":9,"addr":"172.21.15.119:6835/2033916709","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6834","nonce":2033916709},{"type":"v1","addr":"172.21.15.119:6835","nonce":2033916709}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24509":{"gid":24509,"name":"h","rank":1,"incarnation":17,"state":"up:active","state_seq":6,"addr":"172.21.15.119:6837/1872363128","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6836","nonce":1872363128},{"type":"v1","addr":"172.21.15.119:6837","nonce":1872363128}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24545":{"gid":24545,"name":"b","rank":0,"incarnation":14,"state":"up:active","state_seq":3,"addr":"172.21.15.119:6839/937297340","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6838","nonce":937297340},{"type":"v1","addr":"172.21.15.119:6839","nonce":937297340}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":24545,"qdb_cluster":[24479,24509,24545]},"id":1}]} 2024-04-23T10:27:04.581 INFO:teuthology.orchestra.run.smithi114.stderr:dumped fsmap epoch 21 2024-04-23T10:27:04.583 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:27:04.584+0000 7fddaed29640 1 -- 172.21.15.114:0/540177368 >> [v2:172.21.15.114:6800/1395693466,v1:172.21.15.114:6801/1395693466] conn(0x7fdd940a3e40 msgr2=0x7fdd940a6300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T10:27:04.583 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:27:04.584+0000 7fddaed29640 1 --2- 172.21.15.114:0/540177368 >> [v2:172.21.15.114:6800/1395693466,v1:172.21.15.114:6801/1395693466] conn(0x7fdd940a3e40 0x7fdd940a6300 secure :-1 s=READY pgs=63 cs=0 l=1 rev1=1 crypto rx=0x7fdda0005fd0 tx=0x7fdda00098b0 comp rx=0 tx=0).stop 2024-04-23T10:27:04.584 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:27:04.584+0000 7fddaed29640 1 -- 172.21.15.114:0/540177368 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fdda819cf80 msgr2=0x7fdda819d430 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T10:27:04.584 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:27:04.584+0000 7fddaed29640 1 --2- 172.21.15.114:0/540177368 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fdda819cf80 0x7fdda819d430 secure :-1 s=READY pgs=493 cs=0 l=1 rev1=1 crypto rx=0x7fdd98004770 tx=0x7fdd98004270 comp rx=0 tx=0).stop 2024-04-23T10:27:04.584 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:27:04.584+0000 7fddaed29640 1 -- 172.21.15.114:0/540177368 shutdown_connections 2024-04-23T10:27:04.584 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:27:04.584+0000 7fddaed29640 1 --2- 172.21.15.114:0/540177368 >> [v2:172.21.15.114:6800/1395693466,v1:172.21.15.114:6801/1395693466] conn(0x7fdd940a3e40 0x7fdd940a6300 unknown :-1 s=CLOSED pgs=63 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:27:04.584 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:27:04.584+0000 7fddaed29640 1 --2- 172.21.15.114:0/540177368 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fdda819cf80 0x7fdda819d430 unknown :-1 s=CLOSED pgs=493 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:27:04.584 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:27:04.584+0000 7fddaed29640 1 --2- 172.21.15.114:0/540177368 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fdda8106170 0x7fdda8198c00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:27:04.584 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:27:04.584+0000 7fddaed29640 1 --2- 172.21.15.114:0/540177368 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fdda81010a0 0x7fdda81986c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:27:04.584 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:27:04.584+0000 7fddaed29640 1 -- 172.21.15.114:0/540177368 >> 172.21.15.114:0/540177368 conn(0x7fdda8077f50 msgr2=0x7fdda8069440 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T10:27:04.584 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:27:04.584+0000 7fddaed29640 1 -- 172.21.15.114:0/540177368 shutdown_connections 2024-04-23T10:27:04.584 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:27:04.584+0000 7fddaed29640 1 -- 172.21.15.114:0/540177368 wait complete. 2024-04-23T10:27:04.595 DEBUG:tasks.quiescer.fs.[cephfs]:Running ceph command: 'tell mds.24545 quiesce db / --timeout 60 --expiration 82.7 --await-for 42.0' 2024-04-23T10:27:04.595 DEBUG:teuthology.orchestra.run.smithi114:> timeout 75 sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.24545 quiesce db / --timeout 60 --expiration 82.7 --await-for 42.0 2024-04-23T10:27:04.712 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:27:04.710+0000 7eff653f9640 1 -- 172.21.15.114:0/4015382179 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7eff60102b40 msgr2=0x7eff60106b80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T10:27:04.712 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:27:04.710+0000 7eff653f9640 1 --2- 172.21.15.114:0/4015382179 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7eff60102b40 0x7eff60106b80 secure :-1 s=READY pgs=494 cs=0 l=1 rev1=1 crypto rx=0x7eff50009900 tx=0x7eff5002a410 comp rx=0 tx=0).stop 2024-04-23T10:27:04.712 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:27:04.711+0000 7eff653f9640 1 -- 172.21.15.114:0/4015382179 shutdown_connections 2024-04-23T10:27:04.712 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:27:04.711+0000 7eff653f9640 1 --2- 172.21.15.114:0/4015382179 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7eff60107260 0x7eff6010ee30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:27:04.712 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:27:04.711+0000 7eff653f9640 1 --2- 172.21.15.114:0/4015382179 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7eff60102b40 0x7eff60106b80 unknown :-1 s=CLOSED pgs=494 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:27:04.712 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:27:04.711+0000 7eff653f9640 1 --2- 172.21.15.114:0/4015382179 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7eff60102190 0x7eff60102570 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:27:04.712 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:27:04.711+0000 7eff653f9640 1 -- 172.21.15.114:0/4015382179 >> 172.21.15.114:0/4015382179 conn(0x7eff60079310 msgr2=0x7eff60079720 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T10:27:04.712 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:27:04.711+0000 7eff653f9640 1 -- 172.21.15.114:0/4015382179 shutdown_connections 2024-04-23T10:27:04.712 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:27:04.711+0000 7eff653f9640 1 -- 172.21.15.114:0/4015382179 wait complete. 2024-04-23T10:27:04.712 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:27:04.711+0000 7eff653f9640 1 Processor -- start 2024-04-23T10:27:04.712 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:27:04.711+0000 7eff653f9640 1 -- start start 2024-04-23T10:27:04.713 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:27:04.712+0000 7eff653f9640 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7eff60102190 0x7eff601986f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T10:27:04.713 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:27:04.712+0000 7eff653f9640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7eff60107260 0x7eff60198c30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T10:27:04.713 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:27:04.712+0000 7eff653f9640 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7eff6019cfb0 0x7eff6019d460 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T10:27:04.713 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:27:04.712+0000 7eff653f9640 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x7eff601121d0 con 0x7eff60107260 2024-04-23T10:27:04.713 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:27:04.712+0000 7eff653f9640 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 -- 0x7eff60112050 con 0x7eff60102190 2024-04-23T10:27:04.713 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:27:04.712+0000 7eff653f9640 1 -- --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_getmap magic: 0 -- 0x7eff60112350 con 0x7eff6019cfb0 2024-04-23T10:27:04.713 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:27:04.712+0000 7eff5e7fc640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7eff60107260 0x7eff60198c30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T10:27:04.713 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:27:04.712+0000 7eff5e7fc640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7eff60107260 0x7eff60198c30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.114:39022/0 (socket says 172.21.15.114:39022) 2024-04-23T10:27:04.713 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:27:04.712+0000 7eff5e7fc640 1 -- 172.21.15.114:0/2168538502 learned_addr learned my addr 172.21.15.114:0/2168538502 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-04-23T10:27:04.713 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:27:04.712+0000 7eff5effd640 1 --2- 172.21.15.114:0/2168538502 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7eff60102190 0x7eff601986f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T10:27:04.713 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:27:04.712+0000 7eff5f7fe640 1 --2- 172.21.15.114:0/2168538502 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7eff6019cfb0 0x7eff6019d460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T10:27:04.713 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:27:04.712+0000 7eff5e7fc640 1 -- 172.21.15.114:0/2168538502 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7eff6019cfb0 msgr2=0x7eff6019d460 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T10:27:04.713 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:27:04.712+0000 7eff5e7fc640 1 --2- 172.21.15.114:0/2168538502 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7eff6019cfb0 0x7eff6019d460 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:27:04.713 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:27:04.712+0000 7eff5e7fc640 1 -- 172.21.15.114:0/2168538502 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7eff60102190 msgr2=0x7eff601986f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T10:27:04.713 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:27:04.712+0000 7eff5e7fc640 1 --2- 172.21.15.114:0/2168538502 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7eff60102190 0x7eff601986f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:27:04.714 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:27:04.712+0000 7eff5e7fc640 1 -- 172.21.15.114:0/2168538502 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7eff6019daa0 con 0x7eff60107260 2024-04-23T10:27:04.714 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:27:04.712+0000 7eff5f7fe640 1 --2- 172.21.15.114:0/2168538502 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7eff6019cfb0 0x7eff6019d460 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-23T10:27:04.714 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:27:04.712+0000 7eff5effd640 1 --2- 172.21.15.114:0/2168538502 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7eff60102190 0x7eff601986f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-23T10:27:04.714 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:27:04.712+0000 7eff5e7fc640 1 --2- 172.21.15.114:0/2168538502 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7eff60107260 0x7eff60198c30 secure :-1 s=READY pgs=495 cs=0 l=1 rev1=1 crypto rx=0x7eff50005ab0 tx=0x7eff50003cd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T10:27:04.714 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:27:04.712+0000 7eff47fff640 1 -- 172.21.15.114:0/2168538502 <== mon.0 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7eff5003f070 con 0x7eff60107260 2024-04-23T10:27:04.714 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:27:04.712+0000 7eff653f9640 1 -- 172.21.15.114:0/2168538502 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7eff6019dd30 con 0x7eff60107260 2024-04-23T10:27:04.714 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:27:04.712+0000 7eff47fff640 1 -- 172.21.15.114:0/2168538502 <== mon.0 v2:172.21.15.114:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7eff50031d60 con 0x7eff60107260 2024-04-23T10:27:04.714 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:27:04.712+0000 7eff47fff640 1 -- 172.21.15.114:0/2168538502 <== mon.0 v2:172.21.15.114:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7eff50035030 con 0x7eff60107260 2024-04-23T10:27:04.714 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:27:04.712+0000 7eff653f9640 1 -- 172.21.15.114:0/2168538502 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7eff601a5550 con 0x7eff60107260 2024-04-23T10:27:04.720 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:27:04.715+0000 7eff681df740 1 Processor -- start 2024-04-23T10:27:04.720 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:27:04.715+0000 7eff681df740 1 -- start start 2024-04-23T10:27:04.720 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:27:04.715+0000 7eff47fff640 1 -- 172.21.15.114:0/2168538502 <== mon.0 v2:172.21.15.114:3300/0 4 ==== mgrmap(e 25) ==== 138880+0+0 (secure 0 0 0) 0x7eff50042070 con 0x7eff60107260 2024-04-23T10:27:04.721 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:27:04.715+0000 7eff681df740 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x55a4fc7160a0 0x55a4fc716480 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T10:27:04.721 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:27:04.715+0000 7eff681df740 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x55a4fc7169c0 0x55a4fc720b50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T10:27:04.721 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:27:04.715+0000 7eff681df740 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55a4fc721090 0x55a4fc723480 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T10:27:04.721 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:27:04.715+0000 7eff681df740 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x55a4fc67b370 con 0x55a4fc721090 2024-04-23T10:27:04.721 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:27:04.715+0000 7eff681df740 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 -- 0x55a4fc6182e0 con 0x55a4fc7160a0 2024-04-23T10:27:04.721 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:27:04.715+0000 7eff681df740 1 -- --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_getmap magic: 0 -- 0x55a4fc677330 con 0x55a4fc7169c0 2024-04-23T10:27:04.721 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:27:04.715+0000 7eff5e7fc640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55a4fc721090 0x55a4fc723480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T10:27:04.721 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:27:04.715+0000 7eff5e7fc640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55a4fc721090 0x55a4fc723480 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.114:39038/0 (socket says 172.21.15.114:39038) 2024-04-23T10:27:04.721 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:27:04.715+0000 7eff5e7fc640 1 -- 172.21.15.114:0/4289548151 learned_addr learned my addr 172.21.15.114:0/4289548151 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-04-23T10:27:04.721 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:27:04.715+0000 7eff5f7fe640 1 --2- 172.21.15.114:0/4289548151 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x55a4fc7160a0 0x55a4fc716480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T10:27:04.721 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:27:04.715+0000 7eff5effd640 1 --2- 172.21.15.114:0/4289548151 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x55a4fc7169c0 0x55a4fc720b50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T10:27:04.721 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:27:04.716+0000 7eff5e7fc640 1 -- 172.21.15.114:0/4289548151 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x55a4fc7169c0 msgr2=0x55a4fc720b50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T10:27:04.721 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:27:04.716+0000 7eff5e7fc640 1 --2- 172.21.15.114:0/4289548151 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x55a4fc7169c0 0x55a4fc720b50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:27:04.721 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:27:04.716+0000 7eff5e7fc640 1 -- 172.21.15.114:0/4289548151 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x55a4fc7160a0 msgr2=0x55a4fc716480 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T10:27:04.721 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:27:04.716+0000 7eff5e7fc640 1 --2- 172.21.15.114:0/4289548151 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x55a4fc7160a0 0x55a4fc716480 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:27:04.722 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:27:04.716+0000 7eff5e7fc640 1 -- 172.21.15.114:0/4289548151 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55a4fc459d80 con 0x55a4fc721090 2024-04-23T10:27:04.722 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:27:04.716+0000 7eff47fff640 1 --2- 172.21.15.114:0/2168538502 >> [v2:172.21.15.114:6800/1395693466,v1:172.21.15.114:6801/1395693466] conn(0x7eff480a3e40 0x7eff480a6300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T10:27:04.722 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:27:04.716+0000 7eff5f7fe640 1 --2- 172.21.15.114:0/2168538502 >> [v2:172.21.15.114:6800/1395693466,v1:172.21.15.114:6801/1395693466] conn(0x7eff480a3e40 0x7eff480a6300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T10:27:04.722 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:27:04.716+0000 7eff47fff640 1 -- 172.21.15.114:0/2168538502 <== mon.0 v2:172.21.15.114:3300/0 5 ==== osd_map(90..90 src has 1..90) ==== 8527+0+0 (secure 0 0 0) 0x7eff500f1250 con 0x7eff60107260 2024-04-23T10:27:04.722 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:27:04.716+0000 7eff5e7fc640 1 --2- 172.21.15.114:0/4289548151 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55a4fc721090 0x55a4fc723480 secure :-1 s=READY pgs=496 cs=0 l=1 rev1=1 crypto rx=0x7eff500ee930 tx=0x7eff500f9580 comp rx=0 tx=0).ready entity=mon.0 client_cookie=53fb43c9da55606 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T10:27:04.722 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:27:04.716+0000 7eff457fa640 1 -- 172.21.15.114:0/4289548151 <== mon.0 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7eff50049020 con 0x55a4fc721090 2024-04-23T10:27:04.722 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:27:04.716+0000 7eff457fa640 1 -- 172.21.15.114:0/4289548151 <== mon.0 v2:172.21.15.114:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7eff500bd770 con 0x55a4fc721090 2024-04-23T10:27:04.722 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:27:04.716+0000 7eff457fa640 1 -- 172.21.15.114:0/4289548151 <== mon.0 v2:172.21.15.114:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7eff500bce40 con 0x55a4fc721090 2024-04-23T10:27:04.722 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:27:04.717+0000 7eff681df740 1 -- 172.21.15.114:0/4289548151 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55a4fc721090 msgr2=0x55a4fc723480 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T10:27:04.722 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:27:04.717+0000 7eff681df740 1 --2- 172.21.15.114:0/4289548151 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55a4fc721090 0x55a4fc723480 secure :-1 s=READY pgs=496 cs=0 l=1 rev1=1 crypto rx=0x7eff500ee930 tx=0x7eff500f9580 comp rx=0 tx=0).stop 2024-04-23T10:27:04.722 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:27:04.717+0000 7eff5f7fe640 1 --2- 172.21.15.114:0/2168538502 >> [v2:172.21.15.114:6800/1395693466,v1:172.21.15.114:6801/1395693466] conn(0x7eff480a3e40 0x7eff480a6300 secure :-1 s=READY pgs=64 cs=0 l=1 rev1=1 crypto rx=0x7eff58005cc0 tx=0x7eff58008620 comp rx=0 tx=0).ready entity=mgr.15291 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T10:27:04.722 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:27:04.717+0000 7eff681df740 1 -- 172.21.15.114:0/4289548151 shutdown_connections 2024-04-23T10:27:04.722 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:27:04.717+0000 7eff681df740 1 --2- 172.21.15.114:0/4289548151 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55a4fc721090 0x55a4fc723480 unknown :-1 s=CLOSED pgs=496 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:27:04.722 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:27:04.717+0000 7eff681df740 1 --2- 172.21.15.114:0/4289548151 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x55a4fc7169c0 0x55a4fc720b50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:27:04.722 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:27:04.717+0000 7eff681df740 1 --2- 172.21.15.114:0/4289548151 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x55a4fc7160a0 0x55a4fc716480 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:27:04.723 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:27:04.717+0000 7eff681df740 1 -- 172.21.15.114:0/4289548151 >> 172.21.15.114:0/4289548151 conn(0x55a4fc7123c0 msgr2=0x55a4fc7127a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T10:27:04.723 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:27:04.717+0000 7eff681df740 1 -- 172.21.15.114:0/4289548151 shutdown_connections 2024-04-23T10:27:04.723 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:27:04.717+0000 7eff681df740 1 -- 172.21.15.114:0/4289548151 wait complete. 2024-04-23T10:27:04.723 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:27:04.717+0000 7eff681df740 1 -- start start 2024-04-23T10:27:04.723 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:27:04.717+0000 7eff681df740 1 Processor -- start 2024-04-23T10:27:04.723 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:27:04.717+0000 7eff681df740 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55a4fc7160a0 0x55a4fc71dfa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T10:27:04.723 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:27:04.718+0000 7eff681df740 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x55a4fc7169c0 0x55a4fc71e4e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T10:27:04.723 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:27:04.718+0000 7eff681df740 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x55a4fc721090 0x55a4fc71ea20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T10:27:04.723 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:27:04.718+0000 7eff681df740 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x55a4fc723c40 con 0x55a4fc7160a0 2024-04-23T10:27:04.723 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:27:04.718+0000 7eff681df740 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 -- 0x55a4fc615fb0 con 0x55a4fc721090 2024-04-23T10:27:04.723 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:27:04.718+0000 7eff681df740 1 -- --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_getmap magic: 0 -- 0x55a4fc723dc0 con 0x55a4fc7169c0 2024-04-23T10:27:04.723 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:27:04.718+0000 7eff5effd640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55a4fc7160a0 0x55a4fc71dfa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T10:27:04.723 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:27:04.718+0000 7eff681df740 10 client.0 _ll_register_callbacks cb 0x55a4fc6159a0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-23T10:27:04.723 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:27:04.718+0000 7eff5effd640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55a4fc7160a0 0x55a4fc71dfa0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.114:39048/0 (socket says 172.21.15.114:39048) 2024-04-23T10:27:04.723 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:27:04.718+0000 7eff5effd640 1 -- 172.21.15.114:0/801836491 learned_addr learned my addr 172.21.15.114:0/801836491 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-04-23T10:27:04.724 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:27:04.718+0000 7eff5f7fe640 1 --2- 172.21.15.114:0/801836491 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x55a4fc721090 0x55a4fc71ea20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T10:27:04.724 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:27:04.718+0000 7eff5e7fc640 1 --2- 172.21.15.114:0/801836491 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x55a4fc7169c0 0x55a4fc71e4e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T10:27:04.724 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:27:04.718+0000 7eff5effd640 1 -- 172.21.15.114:0/801836491 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x55a4fc7169c0 msgr2=0x55a4fc71e4e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T10:27:04.724 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:27:04.718+0000 7eff5effd640 1 --2- 172.21.15.114:0/801836491 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x55a4fc7169c0 0x55a4fc71e4e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:27:04.724 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:27:04.718+0000 7eff5effd640 1 -- 172.21.15.114:0/801836491 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x55a4fc721090 msgr2=0x55a4fc71ea20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T10:27:04.724 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:27:04.718+0000 7eff5effd640 1 --2- 172.21.15.114:0/801836491 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x55a4fc721090 0x55a4fc71ea20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:27:04.724 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:27:04.718+0000 7eff5effd640 1 -- 172.21.15.114:0/801836491 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55a4fc4ba510 con 0x55a4fc7160a0 2024-04-23T10:27:04.724 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:27:04.718+0000 7eff5f7fe640 1 --2- 172.21.15.114:0/801836491 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x55a4fc721090 0x55a4fc71ea20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-23T10:27:04.724 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:27:04.718+0000 7eff5e7fc640 1 --2- 172.21.15.114:0/801836491 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x55a4fc7169c0 0x55a4fc71e4e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-23T10:27:04.724 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:27:04.718+0000 7eff5effd640 1 --2- 172.21.15.114:0/801836491 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55a4fc7160a0 0x55a4fc71dfa0 secure :-1 s=READY pgs=497 cs=0 l=1 rev1=1 crypto rx=0x7eff5400bb70 tx=0x7eff540039c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=77fc5f2953a781d1 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T10:27:04.724 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:27:04.718+0000 7eff42ff5640 10 client.0 ms_handle_connect on v2:172.21.15.114:3300/0 2024-04-23T10:27:04.724 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:27:04.718+0000 7eff42ff5640 1 -- 172.21.15.114:0/801836491 <== mon.0 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7eff54005220 con 0x55a4fc7160a0 2024-04-23T10:27:04.724 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:27:04.718+0000 7eff42ff5640 1 -- 172.21.15.114:0/801836491 <== mon.0 v2:172.21.15.114:3300/0 2 ==== osd_map(90..90 src has 1..90) ==== 8527+0+0 (secure 0 0 0) 0x7eff540053c0 con 0x55a4fc7160a0 2024-04-23T10:27:04.724 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:27:04.718+0000 7eff681df740 1 -- 172.21.15.114:0/801836491 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7eff500bd770 con 0x55a4fc7160a0 2024-04-23T10:27:04.724 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:27:04.718+0000 7eff42ff5640 1 -- 172.21.15.114:0/801836491 <== mon.0 v2:172.21.15.114:3300/0 3 ==== mon_get_version_reply(handle=1 version=21) ==== 24+0+0 (secure 0 0 0) 0x7eff540055c0 con 0x55a4fc7160a0 2024-04-23T10:27:04.724 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:27:04.718+0000 7eff681df740 10 client.15594 fetch_fsmap learned FSMap version 21 2024-04-23T10:27:04.725 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:27:04.718+0000 7eff681df740 1 -- 172.21.15.114:0/801836491 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({config=0+,fsmap=21,monmap=4+}) -- 0x55a4fc34d670 con 0x55a4fc7160a0 2024-04-23T10:27:04.725 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:27:04.719+0000 7eff42ff5640 1 -- 172.21.15.114:0/801836491 <== mon.0 v2:172.21.15.114:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7eff500bd770 con 0x55a4fc7160a0 2024-04-23T10:27:04.725 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:27:04.719+0000 7eff42ff5640 1 -- 172.21.15.114:0/801836491 <== mon.0 v2:172.21.15.114:3300/0 5 ==== fsmap(e 21) ==== 8338+0+0 (secure 0 0 0) 0x7eff54019030 con 0x55a4fc7160a0 2024-04-23T10:27:04.725 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:27:04.719+0000 7eff681df740 10 client.15594 fetch_fsmap finished waiting for FSMap version 21 2024-04-23T10:27:04.725 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:27:04.719+0000 7eff681df740 10 client.15594 resolve_mds: validated gid 24545 aka daemon mds.b 2024-04-23T10:27:04.725 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:27:04.719+0000 7eff681df740 20 client.15594 populate_metadata read hostname 'smithi114' 2024-04-23T10:27:04.725 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:27:04.719+0000 7eff681df740 1 --2- 172.21.15.114:0/801836491 >> [v2:172.21.15.119:6838/937297340,v1:172.21.15.119:6839/937297340] conn(0x55a4fc731120 0x55a4fc733590 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T10:27:04.725 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:27:04.719+0000 7eff681df740 4 client.15594 mds_command: new command op to 24545 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-23T10:27:04.725 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:27:04.719+0000 7eff681df740 1 -- 172.21.15.114:0/801836491 --> [v2:172.21.15.119:6838/937297340,v1:172.21.15.119:6839/937297340] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55a4fc688d90 con 0x55a4fc731120 2024-04-23T10:27:04.725 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:27:04.719+0000 7eff5e7fc640 1 --2- 172.21.15.114:0/801836491 >> [v2:172.21.15.119:6838/937297340,v1:172.21.15.119:6839/937297340] conn(0x55a4fc731120 0x55a4fc733590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T10:27:04.725 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:27:04.719+0000 7eff42ff5640 10 client.15594 ms_handle_connect on v2:172.21.15.119:6838/937297340 2024-04-23T10:27:04.725 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:27:04.719+0000 7eff5e7fc640 1 --2- 172.21.15.114:0/801836491 >> [v2:172.21.15.119:6838/937297340,v1:172.21.15.119:6839/937297340] conn(0x55a4fc731120 0x55a4fc733590 crc :-1 s=READY pgs=42 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=bbab13074a33245c server_cookie=e0ec107bd0afaf0c in_seq=0 out_seq=0 2024-04-23T10:27:04.725 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:27:04.720+0000 7eff42ff5640 1 -- 172.21.15.114:0/801836491 <== mds.0 v2:172.21.15.119:6838/937297340 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x55a4fc688d90 con 0x55a4fc731120 2024-04-23T10:27:04.725 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:27:04.720+0000 7eff42ff5640 10 client.15594 handle_command_reply: tid=0 2024-04-23T10:27:04.725 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:27:04.720+0000 7eff681df740 1 client.15594 shutdown 2024-04-23T10:27:04.726 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:27:04.720+0000 7eff681df740 1 -- 172.21.15.114:0/801836491 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55a4fc7160a0 msgr2=0x55a4fc71dfa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T10:27:04.726 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:27:04.720+0000 7eff681df740 1 --2- 172.21.15.114:0/801836491 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55a4fc7160a0 0x55a4fc71dfa0 secure :-1 s=READY pgs=497 cs=0 l=1 rev1=1 crypto rx=0x7eff5400bb70 tx=0x7eff540039c0 comp rx=0 tx=0).stop 2024-04-23T10:27:04.726 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:27:04.720+0000 7eff681df740 1 -- 172.21.15.114:0/801836491 shutdown_connections 2024-04-23T10:27:04.726 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:27:04.720+0000 7eff681df740 1 --2- 172.21.15.114:0/801836491 >> [v2:172.21.15.119:6838/937297340,v1:172.21.15.119:6839/937297340] conn(0x55a4fc731120 0x55a4fc733590 crc :-1 s=READY pgs=42 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:27:04.726 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:27:04.720+0000 7eff681df740 1 --2- 172.21.15.114:0/801836491 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x55a4fc721090 0x55a4fc71ea20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:27:04.726 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:27:04.720+0000 7eff42ff5640 0 client.15594 ms_handle_reset on v2:172.21.15.119:6838/937297340 2024-04-23T10:27:04.726 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:27:04.720+0000 7eff681df740 1 --2- 172.21.15.114:0/801836491 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x55a4fc7169c0 0x55a4fc71e4e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:27:04.726 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:27:04.720+0000 7eff681df740 1 --2- 172.21.15.114:0/801836491 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55a4fc7160a0 0x55a4fc71dfa0 unknown :-1 s=CLOSED pgs=497 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:27:04.726 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:27:04.720+0000 7eff681df740 1 -- 172.21.15.114:0/801836491 >> 172.21.15.114:0/801836491 conn(0x55a4fc7123c0 msgr2=0x55a4fc7181c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T10:27:04.726 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:27:04.720+0000 7eff681df740 1 -- 172.21.15.114:0/801836491 shutdown_connections 2024-04-23T10:27:04.726 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:27:04.720+0000 7eff681df740 1 -- 172.21.15.114:0/801836491 wait complete. 2024-04-23T10:27:04.726 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:27:04.721+0000 7eff681df740 20 client.15594 trim_cache size 0 max 16384 2024-04-23T10:27:04.741 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:27:04.738+0000 7eff681df740 1 Processor -- start 2024-04-23T10:27:04.741 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:27:04.738+0000 7eff681df740 1 -- start start 2024-04-23T10:27:04.741 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:27:04.738+0000 7eff681df740 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55a4fc7160a0 0x55a4fc71ce70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T10:27:04.741 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:27:04.738+0000 7eff681df740 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x55a4fc7169c0 0x55a4fc711d80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T10:27:04.741 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:27:04.738+0000 7eff681df740 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x55a4fc721090 0x55a4fc718800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T10:27:04.741 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:27:04.738+0000 7eff681df740 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x7eff601b04a0 con 0x55a4fc7160a0 2024-04-23T10:27:04.741 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:27:04.738+0000 7eff681df740 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 -- 0x7eff601add70 con 0x55a4fc721090 2024-04-23T10:27:04.741 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:27:04.738+0000 7eff681df740 1 -- --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_getmap magic: 0 -- 0x7eff601abbc0 con 0x55a4fc7169c0 2024-04-23T10:27:04.741 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:27:04.738+0000 7eff5effd640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55a4fc7160a0 0x55a4fc71ce70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T10:27:04.742 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:27:04.738+0000 7eff5effd640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55a4fc7160a0 0x55a4fc71ce70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.114:39050/0 (socket says 172.21.15.114:39050) 2024-04-23T10:27:04.742 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:27:04.738+0000 7eff5effd640 1 -- 172.21.15.114:0/3379573704 learned_addr learned my addr 172.21.15.114:0/3379573704 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-04-23T10:27:04.742 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:27:04.738+0000 7eff5f7fe640 1 --2- 172.21.15.114:0/3379573704 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x55a4fc721090 0x55a4fc718800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T10:27:04.742 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:27:04.738+0000 7eff5e7fc640 1 --2- 172.21.15.114:0/3379573704 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x55a4fc7169c0 0x55a4fc711d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T10:27:04.742 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:27:04.738+0000 7eff5effd640 1 -- 172.21.15.114:0/3379573704 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x55a4fc7169c0 msgr2=0x55a4fc711d80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T10:27:04.742 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:27:04.738+0000 7eff5effd640 1 --2- 172.21.15.114:0/3379573704 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x55a4fc7169c0 0x55a4fc711d80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:27:04.742 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:27:04.738+0000 7eff5effd640 1 -- 172.21.15.114:0/3379573704 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x55a4fc721090 msgr2=0x55a4fc718800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T10:27:04.742 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:27:04.738+0000 7eff5effd640 1 --2- 172.21.15.114:0/3379573704 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x55a4fc721090 0x55a4fc718800 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:27:04.742 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:27:04.738+0000 7eff5effd640 1 -- 172.21.15.114:0/3379573704 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55a4fc7156b0 con 0x55a4fc7160a0 2024-04-23T10:27:04.742 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:27:04.738+0000 7eff5e7fc640 1 --2- 172.21.15.114:0/3379573704 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x55a4fc7169c0 0x55a4fc711d80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-23T10:27:04.742 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:27:04.738+0000 7eff5effd640 1 --2- 172.21.15.114:0/3379573704 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55a4fc7160a0 0x55a4fc71ce70 secure :-1 s=READY pgs=498 cs=0 l=1 rev1=1 crypto rx=0x7eff54001eb0 tx=0x7eff54005780 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a78f3c753b5a11e2 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T10:27:04.742 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:27:04.739+0000 7eff44ff9640 1 -- 172.21.15.114:0/3379573704 <== mon.0 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7eff54027040 con 0x55a4fc7160a0 2024-04-23T10:27:04.742 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:27:04.739+0000 7eff44ff9640 1 -- 172.21.15.114:0/3379573704 <== mon.0 v2:172.21.15.114:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7eff54023430 con 0x55a4fc7160a0 2024-04-23T10:27:04.743 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:27:04.739+0000 7eff44ff9640 1 -- 172.21.15.114:0/3379573704 <== mon.0 v2:172.21.15.114:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7eff5401a650 con 0x55a4fc7160a0 2024-04-23T10:27:04.743 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:27:04.739+0000 7eff681df740 1 -- 172.21.15.114:0/3379573704 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55a4fc7160a0 msgr2=0x55a4fc71ce70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T10:27:04.743 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:27:04.739+0000 7eff681df740 1 --2- 172.21.15.114:0/3379573704 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55a4fc7160a0 0x55a4fc71ce70 secure :-1 s=READY pgs=498 cs=0 l=1 rev1=1 crypto rx=0x7eff54001eb0 tx=0x7eff54005780 comp rx=0 tx=0).stop 2024-04-23T10:27:04.743 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:27:04.739+0000 7eff681df740 1 -- 172.21.15.114:0/3379573704 shutdown_connections 2024-04-23T10:27:04.743 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:27:04.739+0000 7eff681df740 1 --2- 172.21.15.114:0/3379573704 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x55a4fc721090 0x55a4fc718800 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:27:04.743 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:27:04.739+0000 7eff681df740 1 --2- 172.21.15.114:0/3379573704 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x55a4fc7169c0 0x55a4fc711d80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:27:04.743 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:27:04.739+0000 7eff681df740 1 --2- 172.21.15.114:0/3379573704 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55a4fc7160a0 0x55a4fc71ce70 unknown :-1 s=CLOSED pgs=498 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:27:04.743 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:27:04.739+0000 7eff681df740 1 -- 172.21.15.114:0/3379573704 >> 172.21.15.114:0/3379573704 conn(0x55a4fc7123c0 msgr2=0x55a4fc71df60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T10:27:04.743 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:27:04.739+0000 7eff681df740 1 -- 172.21.15.114:0/3379573704 shutdown_connections 2024-04-23T10:27:04.743 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:27:04.739+0000 7eff681df740 1 -- 172.21.15.114:0/3379573704 wait complete. 2024-04-23T10:27:04.743 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:27:04.740+0000 7eff681df740 1 -- start start 2024-04-23T10:27:04.743 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:27:04.740+0000 7eff681df740 1 Processor -- start 2024-04-23T10:27:04.743 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:27:04.740+0000 7eff681df740 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x55a4fc7160a0 0x55a4fc71af70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T10:27:04.743 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:27:04.740+0000 7eff681df740 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55a4fc7169c0 0x55a4fc71b4b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T10:27:04.744 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:27:04.740+0000 7eff681df740 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x55a4fc721090 0x55a4fc71b9f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T10:27:04.744 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:27:04.740+0000 7eff681df740 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x7eff601ac230 con 0x55a4fc7169c0 2024-04-23T10:27:04.744 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:27:04.740+0000 7eff681df740 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 -- 0x7eff601ad6a0 con 0x55a4fc721090 2024-04-23T10:27:04.744 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:27:04.740+0000 7eff681df740 1 -- --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_getmap magic: 0 -- 0x7eff601ac900 con 0x55a4fc7160a0 2024-04-23T10:27:04.744 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:27:04.740+0000 7eff681df740 10 client.0 _ll_register_callbacks cb 0x7eff54004230 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-23T10:27:04.744 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:27:04.740+0000 7eff5e7fc640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55a4fc7169c0 0x55a4fc71b4b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T10:27:04.744 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:27:04.740+0000 7eff5e7fc640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55a4fc7169c0 0x55a4fc71b4b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.114:39052/0 (socket says 172.21.15.114:39052) 2024-04-23T10:27:04.744 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:27:04.740+0000 7eff5e7fc640 1 -- 172.21.15.114:0/3109425922 learned_addr learned my addr 172.21.15.114:0/3109425922 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-04-23T10:27:04.744 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:27:04.740+0000 7eff5f7fe640 1 --2- 172.21.15.114:0/3109425922 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x55a4fc721090 0x55a4fc71b9f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T10:27:04.744 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:27:04.740+0000 7eff5effd640 1 --2- 172.21.15.114:0/3109425922 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x55a4fc7160a0 0x55a4fc71af70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T10:27:04.744 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:27:04.740+0000 7eff5e7fc640 1 -- 172.21.15.114:0/3109425922 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x55a4fc7160a0 msgr2=0x55a4fc71af70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T10:27:04.744 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:27:04.740+0000 7eff5e7fc640 1 --2- 172.21.15.114:0/3109425922 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x55a4fc7160a0 0x55a4fc71af70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:27:04.744 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:27:04.740+0000 7eff5e7fc640 1 -- 172.21.15.114:0/3109425922 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x55a4fc721090 msgr2=0x55a4fc71b9f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T10:27:04.744 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:27:04.740+0000 7eff5e7fc640 1 --2- 172.21.15.114:0/3109425922 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x55a4fc721090 0x55a4fc71b9f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:27:04.744 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:27:04.740+0000 7eff5e7fc640 1 -- 172.21.15.114:0/3109425922 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55a4fc796970 con 0x55a4fc7169c0 2024-04-23T10:27:04.745 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:27:04.740+0000 7eff5effd640 1 --2- 172.21.15.114:0/3109425922 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x55a4fc7160a0 0x55a4fc71af70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-23T10:27:04.745 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:27:04.741+0000 7eff5e7fc640 1 --2- 172.21.15.114:0/3109425922 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55a4fc7169c0 0x55a4fc71b4b0 secure :-1 s=READY pgs=499 cs=0 l=1 rev1=1 crypto rx=0x7eff500eefa0 tx=0x7eff5003b460 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e93b24e92effa7b9 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T10:27:04.745 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:27:04.741+0000 7eff42ff5640 10 client.0 ms_handle_connect on v2:172.21.15.114:3300/0 2024-04-23T10:27:04.745 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:27:04.741+0000 7eff42ff5640 1 -- 172.21.15.114:0/3109425922 <== mon.0 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7eff50049040 con 0x55a4fc7169c0 2024-04-23T10:27:04.745 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:27:04.741+0000 7eff42ff5640 1 -- 172.21.15.114:0/3109425922 <== mon.0 v2:172.21.15.114:3300/0 2 ==== osd_map(90..90 src has 1..90) ==== 8527+0+0 (secure 0 0 0) 0x7eff500b96e0 con 0x55a4fc7169c0 2024-04-23T10:27:04.745 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:27:04.741+0000 7eff681df740 1 -- 172.21.15.114:0/3109425922 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7eff54023430 con 0x55a4fc7169c0 2024-04-23T10:27:04.745 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:27:04.741+0000 7eff42ff5640 1 -- 172.21.15.114:0/3109425922 <== mon.0 v2:172.21.15.114:3300/0 3 ==== mon_get_version_reply(handle=1 version=21) ==== 24+0+0 (secure 0 0 0) 0x7eff500bc9d0 con 0x55a4fc7169c0 2024-04-23T10:27:04.745 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:27:04.741+0000 7eff681df740 10 client.15600 fetch_fsmap learned FSMap version 21 2024-04-23T10:27:04.745 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:27:04.741+0000 7eff681df740 1 -- 172.21.15.114:0/3109425922 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({config=0+,fsmap=21,monmap=4+}) -- 0x55a4fc797ba0 con 0x55a4fc7169c0 2024-04-23T10:27:04.745 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:27:04.741+0000 7eff42ff5640 1 -- 172.21.15.114:0/3109425922 <== mon.0 v2:172.21.15.114:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7eff54023430 con 0x55a4fc7169c0 2024-04-23T10:27:04.745 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:27:04.741+0000 7eff42ff5640 1 -- 172.21.15.114:0/3109425922 <== mon.0 v2:172.21.15.114:3300/0 5 ==== fsmap(e 21) ==== 8338+0+0 (secure 0 0 0) 0x7eff5003b4f0 con 0x55a4fc7169c0 2024-04-23T10:27:04.745 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:27:04.741+0000 7eff681df740 10 client.15600 fetch_fsmap finished waiting for FSMap version 21 2024-04-23T10:27:04.745 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:27:04.741+0000 7eff681df740 10 client.15600 resolve_mds: validated gid 24545 aka daemon mds.b 2024-04-23T10:27:04.745 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:27:04.741+0000 7eff681df740 20 client.15600 populate_metadata read hostname 'smithi114' 2024-04-23T10:27:04.745 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:27:04.741+0000 7eff681df740 1 --2- 172.21.15.114:0/3109425922 >> [v2:172.21.15.119:6838/937297340,v1:172.21.15.119:6839/937297340] conn(0x55a4fc7981f0 0x55a4fc79a5e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T10:27:04.746 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:27:04.741+0000 7eff681df740 4 client.15600 mds_command: new command op to 24545 tid=0 multi_id=0 [{"prefix": "quiesce db", "roots": ["/"], "timeout": 60.0, "expiration": 82.7, "await_for": 42.0}] 2024-04-23T10:27:04.746 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:27:04.741+0000 7eff681df740 1 -- 172.21.15.114:0/3109425922 --> [v2:172.21.15.119:6838/937297340,v1:172.21.15.119:6839/937297340] -- command(tid 0: {"prefix": "quiesce db", "roots": ["/"], "timeout": 60.0, "expiration": 82.7, "await_for": 42.0}) -- 0x55a4fc682c00 con 0x55a4fc7981f0 2024-04-23T10:27:04.746 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:27:04.741+0000 7eff5effd640 1 --2- 172.21.15.114:0/3109425922 >> [v2:172.21.15.119:6838/937297340,v1:172.21.15.119:6839/937297340] conn(0x55a4fc7981f0 0x55a4fc79a5e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T10:27:04.746 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:27:04.742+0000 7eff42ff5640 10 client.15600 ms_handle_connect on v2:172.21.15.119:6838/937297340 2024-04-23T10:27:04.746 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:27:04.742+0000 7eff5effd640 1 --2- 172.21.15.114:0/3109425922 >> [v2:172.21.15.119:6838/937297340,v1:172.21.15.119:6839/937297340] conn(0x55a4fc7981f0 0x55a4fc79a5e0 crc :-1 s=READY pgs=43 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=9e266cf4e0df4b96 server_cookie=1c325e1aecb76361 in_seq=0 out_seq=0 2024-04-23T10:27:04.754 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:27:04.755+0000 7eff42ff5640 1 -- 172.21.15.114:0/3109425922 <== mds.0 v2:172.21.15.119:6838/937297340 1 ==== command_reply(tid 0: 0 ) ==== 8+0+588 (crc 0 0 0) 0x55a4fc682c00 con 0x55a4fc7981f0 2024-04-23T10:27:04.754 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:27:04.755+0000 7eff42ff5640 10 client.15600 handle_command_reply: tid=0 2024-04-23T10:27:04.754 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:27:04.755+0000 7eff681df740 1 client.15600 shutdown 2024-04-23T10:27:04.754 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:27:04.755+0000 7eff681df740 1 -- 172.21.15.114:0/3109425922 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55a4fc7169c0 msgr2=0x55a4fc71b4b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T10:27:04.754 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:27:04.755+0000 7eff681df740 1 --2- 172.21.15.114:0/3109425922 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55a4fc7169c0 0x55a4fc71b4b0 secure :-1 s=READY pgs=499 cs=0 l=1 rev1=1 crypto rx=0x7eff500eefa0 tx=0x7eff5003b460 comp rx=0 tx=0).stop 2024-04-23T10:27:04.755 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:27:04.755+0000 7eff681df740 1 -- 172.21.15.114:0/3109425922 shutdown_connections 2024-04-23T10:27:04.755 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:27:04.755+0000 7eff681df740 1 --2- 172.21.15.114:0/3109425922 >> [v2:172.21.15.119:6838/937297340,v1:172.21.15.119:6839/937297340] conn(0x55a4fc7981f0 0x55a4fc79a5e0 crc :-1 s=READY pgs=43 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:27:04.755 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:27:04.755+0000 7eff681df740 1 --2- 172.21.15.114:0/3109425922 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x55a4fc721090 0x55a4fc71b9f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:27:04.755 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:27:04.755+0000 7eff681df740 1 --2- 172.21.15.114:0/3109425922 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55a4fc7169c0 0x55a4fc71b4b0 unknown :-1 s=CLOSED pgs=499 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:27:04.755 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:27:04.755+0000 7eff42ff5640 0 client.15600 ms_handle_reset on v2:172.21.15.119:6838/937297340 2024-04-23T10:27:04.755 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:27:04.755+0000 7eff681df740 1 --2- 172.21.15.114:0/3109425922 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x55a4fc7160a0 0x55a4fc71af70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:27:04.755 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:27:04.755+0000 7eff681df740 1 -- 172.21.15.114:0/3109425922 >> 172.21.15.114:0/3109425922 conn(0x55a4fc7123c0 msgr2=0x55a4fc71e7d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T10:27:04.755 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:27:04.755+0000 7eff681df740 1 -- 172.21.15.114:0/3109425922 shutdown_connections 2024-04-23T10:27:04.755 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:27:04.755+0000 7eff681df740 1 -- 172.21.15.114:0/3109425922 wait complete. 2024-04-23T10:27:04.755 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:27:04.755+0000 7eff681df740 20 client.15600 trim_cache size 0 max 16384 2024-04-23T10:27:04.755 INFO:teuthology.orchestra.run.smithi114.stdout:{ 2024-04-23T10:27:04.756 INFO:teuthology.orchestra.run.smithi114.stdout: "epoch": 21, 2024-04-23T10:27:04.756 INFO:teuthology.orchestra.run.smithi114.stdout: "leader": 24545, 2024-04-23T10:27:04.756 INFO:teuthology.orchestra.run.smithi114.stdout: "set_version": 2, 2024-04-23T10:27:04.756 INFO:teuthology.orchestra.run.smithi114.stdout: "sets": { 2024-04-23T10:27:04.756 INFO:teuthology.orchestra.run.smithi114.stdout: "7f49b8f3": { 2024-04-23T10:27:04.756 INFO:teuthology.orchestra.run.smithi114.stdout: "version": 2, 2024-04-23T10:27:04.756 INFO:teuthology.orchestra.run.smithi114.stdout: "age_ref": 0.0, 2024-04-23T10:27:04.756 INFO:teuthology.orchestra.run.smithi114.stdout: "state": { 2024-04-23T10:27:04.756 INFO:teuthology.orchestra.run.smithi114.stdout: "name": "QUIESCED", 2024-04-23T10:27:04.756 INFO:teuthology.orchestra.run.smithi114.stdout: "age": 0.0 2024-04-23T10:27:04.756 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-23T10:27:04.756 INFO:teuthology.orchestra.run.smithi114.stdout: "timeout": 60.0, 2024-04-23T10:27:04.756 INFO:teuthology.orchestra.run.smithi114.stdout: "expiration": 82.7, 2024-04-23T10:27:04.756 INFO:teuthology.orchestra.run.smithi114.stdout: "members": { 2024-04-23T10:27:04.756 INFO:teuthology.orchestra.run.smithi114.stdout: "file:/": { 2024-04-23T10:27:04.757 INFO:teuthology.orchestra.run.smithi114.stdout: "excluded": false, 2024-04-23T10:27:04.757 INFO:teuthology.orchestra.run.smithi114.stdout: "state": { 2024-04-23T10:27:04.757 INFO:teuthology.orchestra.run.smithi114.stdout: "name": "QUIESCED", 2024-04-23T10:27:04.757 INFO:teuthology.orchestra.run.smithi114.stdout: "age": 0.0 2024-04-23T10:27:04.757 INFO:teuthology.orchestra.run.smithi114.stdout: } 2024-04-23T10:27:04.757 INFO:teuthology.orchestra.run.smithi114.stdout: } 2024-04-23T10:27:04.757 INFO:teuthology.orchestra.run.smithi114.stdout: } 2024-04-23T10:27:04.757 INFO:teuthology.orchestra.run.smithi114.stdout: } 2024-04-23T10:27:04.757 INFO:teuthology.orchestra.run.smithi114.stdout: } 2024-04-23T10:27:04.757 INFO:teuthology.orchestra.run.smithi114.stdout:} 2024-04-23T10:27:04.758 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:27:04.755+0000 7eff653f9640 1 -- 172.21.15.114:0/2168538502 >> [v2:172.21.15.114:6800/1395693466,v1:172.21.15.114:6801/1395693466] conn(0x7eff480a3e40 msgr2=0x7eff480a6300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T10:27:04.758 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:27:04.755+0000 7eff653f9640 1 --2- 172.21.15.114:0/2168538502 >> [v2:172.21.15.114:6800/1395693466,v1:172.21.15.114:6801/1395693466] conn(0x7eff480a3e40 0x7eff480a6300 secure :-1 s=READY pgs=64 cs=0 l=1 rev1=1 crypto rx=0x7eff58005cc0 tx=0x7eff58008620 comp rx=0 tx=0).stop 2024-04-23T10:27:04.758 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:27:04.755+0000 7eff653f9640 1 -- 172.21.15.114:0/2168538502 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7eff60107260 msgr2=0x7eff60198c30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T10:27:04.758 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:27:04.755+0000 7eff653f9640 1 --2- 172.21.15.114:0/2168538502 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7eff60107260 0x7eff60198c30 secure :-1 s=READY pgs=495 cs=0 l=1 rev1=1 crypto rx=0x7eff50005ab0 tx=0x7eff50003cd0 comp rx=0 tx=0).stop 2024-04-23T10:27:04.758 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:27:04.756+0000 7eff653f9640 1 -- 172.21.15.114:0/2168538502 shutdown_connections 2024-04-23T10:27:04.758 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:27:04.756+0000 7eff653f9640 1 --2- 172.21.15.114:0/2168538502 >> [v2:172.21.15.114:6800/1395693466,v1:172.21.15.114:6801/1395693466] conn(0x7eff480a3e40 0x7eff480a6300 unknown :-1 s=CLOSED pgs=64 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:27:04.758 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:27:04.756+0000 7eff653f9640 1 --2- 172.21.15.114:0/2168538502 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7eff6019cfb0 0x7eff6019d460 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:27:04.758 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:27:04.756+0000 7eff653f9640 1 --2- 172.21.15.114:0/2168538502 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7eff60107260 0x7eff60198c30 unknown :-1 s=CLOSED pgs=495 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:27:04.758 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:27:04.756+0000 7eff653f9640 1 --2- 172.21.15.114:0/2168538502 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7eff60102190 0x7eff601986f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:27:04.759 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:27:04.756+0000 7eff653f9640 1 -- 172.21.15.114:0/2168538502 >> 172.21.15.114:0/2168538502 conn(0x7eff60079310 msgr2=0x7eff601000a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T10:27:04.759 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:27:04.756+0000 7eff653f9640 1 -- 172.21.15.114:0/2168538502 shutdown_connections 2024-04-23T10:27:04.759 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:27:04.756+0000 7eff653f9640 1 -- 172.21.15.114:0/2168538502 wait complete. 2024-04-23T10:27:04.765 INFO:tasks.quiescer.fs.[cephfs]:Successfully quiesced set '7f49b8f3', quiesce took 0.6 seconds. Will release after: 22.7 2024-04-23T10:27:05.328 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 23 10:27:05 smithi204 ceph-mon[32649]: from='client.? 172.21.15.114:0/540177368' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-23T10:27:05.492 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 23 10:27:05 smithi119 ceph-mon[32608]: from='client.? 172.21.15.114:0/540177368' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-23T10:27:05.503 INFO:journalctl@ceph.mon.a.smithi114.stdout:Apr 23 10:27:05 smithi114 ceph-mon[29255]: from='client.? 172.21.15.114:0/540177368' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-23T10:27:06.328 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 23 10:27:06 smithi204 ceph-mon[32649]: pgmap v15: 97 pgs: 97 active+clean; 1.3 MiB data, 377 MiB used, 1.0 TiB / 1.0 TiB avail; 12 KiB/s rd, 24 KiB/s wr, 24 op/s 2024-04-23T10:27:06.492 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 23 10:27:06 smithi119 ceph-mon[32608]: pgmap v15: 97 pgs: 97 active+clean; 1.3 MiB data, 377 MiB used, 1.0 TiB / 1.0 TiB avail; 12 KiB/s rd, 24 KiB/s wr, 24 op/s 2024-04-23T10:27:06.504 INFO:journalctl@ceph.mon.a.smithi114.stdout:Apr 23 10:27:06 smithi114 ceph-mon[29255]: pgmap v15: 97 pgs: 97 active+clean; 1.3 MiB data, 377 MiB used, 1.0 TiB / 1.0 TiB avail; 12 KiB/s rd, 24 KiB/s wr, 24 op/s 2024-04-23T10:27:08.328 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 23 10:27:08 smithi204 ceph-mon[32649]: pgmap v16: 97 pgs: 97 active+clean; 1.3 MiB data, 377 MiB used, 1.0 TiB / 1.0 TiB avail; 12 KiB/s rd, 24 KiB/s wr, 24 op/s 2024-04-23T10:27:08.492 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 23 10:27:08 smithi119 ceph-mon[32608]: pgmap v16: 97 pgs: 97 active+clean; 1.3 MiB data, 377 MiB used, 1.0 TiB / 1.0 TiB avail; 12 KiB/s rd, 24 KiB/s wr, 24 op/s 2024-04-23T10:27:08.503 INFO:journalctl@ceph.mon.a.smithi114.stdout:Apr 23 10:27:08 smithi114 ceph-mon[29255]: pgmap v16: 97 pgs: 97 active+clean; 1.3 MiB data, 377 MiB used, 1.0 TiB / 1.0 TiB avail; 12 KiB/s rd, 24 KiB/s wr, 24 op/s 2024-04-23T10:27:08.831 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:27:08.830+0000 7fa4735e2640 1 -- 172.21.15.114:0/3464787397 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fa46c1010f0 msgr2=0x7fa46c1014d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T10:27:08.831 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:27:08.830+0000 7fa4735e2640 1 --2- 172.21.15.114:0/3464787397 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fa46c1010f0 0x7fa46c1014d0 secure :-1 s=READY pgs=500 cs=0 l=1 rev1=1 crypto rx=0x7fa468009900 tx=0x7fa46802a410 comp rx=0 tx=0).stop 2024-04-23T10:27:08.831 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:27:08.830+0000 7fa4735e2640 1 -- 172.21.15.114:0/3464787397 shutdown_connections 2024-04-23T10:27:08.831 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:27:08.830+0000 7fa4735e2640 1 --2- 172.21.15.114:0/3464787397 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fa46c106170 0x7fa46c10dd40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:27:08.831 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:27:08.830+0000 7fa4735e2640 1 --2- 172.21.15.114:0/3464787397 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fa46c101aa0 0x7fa46c105a90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:27:08.832 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:27:08.830+0000 7fa4735e2640 1 --2- 172.21.15.114:0/3464787397 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fa46c1010f0 0x7fa46c1014d0 unknown :-1 s=CLOSED pgs=500 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:27:08.832 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:27:08.830+0000 7fa4735e2640 1 -- 172.21.15.114:0/3464787397 >> 172.21.15.114:0/3464787397 conn(0x7fa46c077fd0 msgr2=0x7fa46c0783e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T10:27:08.832 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:27:08.830+0000 7fa4735e2640 1 -- 172.21.15.114:0/3464787397 shutdown_connections 2024-04-23T10:27:08.832 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:27:08.830+0000 7fa4735e2640 1 -- 172.21.15.114:0/3464787397 wait complete. 2024-04-23T10:27:08.832 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:27:08.830+0000 7fa4735e2640 1 Processor -- start 2024-04-23T10:27:08.832 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:27:08.830+0000 7fa4735e2640 1 -- start start 2024-04-23T10:27:08.832 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:27:08.831+0000 7fa4735e2640 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fa46c1010f0 0x7fa46c102b30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T10:27:08.832 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:27:08.831+0000 7fa4735e2640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fa46c106170 0x7fa46c103070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T10:27:08.832 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:27:08.831+0000 7fa4735e2640 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fa46c1035b0 0x7fa46c1a6810 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T10:27:08.832 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:27:08.831+0000 7fa4735e2640 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x7fa46c1110b0 con 0x7fa46c106170 2024-04-23T10:27:08.832 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:27:08.831+0000 7fa4735e2640 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 -- 0x7fa46c110f30 con 0x7fa46c1010f0 2024-04-23T10:27:08.832 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:27:08.831+0000 7fa4735e2640 1 -- --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_getmap magic: 0 -- 0x7fa46c111230 con 0x7fa46c1035b0 2024-04-23T10:27:08.832 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:27:08.831+0000 7fa470b7d640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fa46c106170 0x7fa46c103070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T10:27:08.832 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:27:08.831+0000 7fa470b7d640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fa46c106170 0x7fa46c103070 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.114:58552/0 (socket says 172.21.15.114:58552) 2024-04-23T10:27:08.833 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:27:08.831+0000 7fa470b7d640 1 -- 172.21.15.114:0/387963399 learned_addr learned my addr 172.21.15.114:0/387963399 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-04-23T10:27:08.833 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:27:08.831+0000 7fa471b7f640 1 --2- 172.21.15.114:0/387963399 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fa46c1035b0 0x7fa46c1a6810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T10:27:08.833 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:27:08.831+0000 7fa47137e640 1 --2- 172.21.15.114:0/387963399 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fa46c1010f0 0x7fa46c102b30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T10:27:08.833 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:27:08.831+0000 7fa470b7d640 1 -- 172.21.15.114:0/387963399 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fa46c1035b0 msgr2=0x7fa46c1a6810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T10:27:08.833 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:27:08.831+0000 7fa470b7d640 1 --2- 172.21.15.114:0/387963399 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fa46c1035b0 0x7fa46c1a6810 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:27:08.833 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:27:08.831+0000 7fa470b7d640 1 -- 172.21.15.114:0/387963399 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fa46c1010f0 msgr2=0x7fa46c102b30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T10:27:08.833 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:27:08.831+0000 7fa470b7d640 1 --2- 172.21.15.114:0/387963399 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fa46c1010f0 0x7fa46c102b30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:27:08.833 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:27:08.831+0000 7fa470b7d640 1 -- 172.21.15.114:0/387963399 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa46c1a6f10 con 0x7fa46c106170 2024-04-23T10:27:08.833 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:27:08.831+0000 7fa471b7f640 1 --2- 172.21.15.114:0/387963399 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fa46c1035b0 0x7fa46c1a6810 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-23T10:27:08.833 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:27:08.831+0000 7fa470b7d640 1 --2- 172.21.15.114:0/387963399 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fa46c106170 0x7fa46c103070 secure :-1 s=READY pgs=501 cs=0 l=1 rev1=1 crypto rx=0x7fa45c00c970 tx=0x7fa45c00ce30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T10:27:08.833 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:27:08.831+0000 7fa4627fc640 1 -- 172.21.15.114:0/387963399 <== mon.0 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa45c014070 con 0x7fa46c106170 2024-04-23T10:27:08.833 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:27:08.831+0000 7fa4627fc640 1 -- 172.21.15.114:0/387963399 <== mon.0 v2:172.21.15.114:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fa45c004440 con 0x7fa46c106170 2024-04-23T10:27:08.833 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:27:08.831+0000 7fa4735e2640 1 -- 172.21.15.114:0/387963399 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fa46c1a7200 con 0x7fa46c106170 2024-04-23T10:27:08.833 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:27:08.831+0000 7fa4627fc640 1 -- 172.21.15.114:0/387963399 <== mon.0 v2:172.21.15.114:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa45c003e50 con 0x7fa46c106170 2024-04-23T10:27:08.834 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:27:08.831+0000 7fa4735e2640 1 -- 172.21.15.114:0/387963399 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fa46c1a7750 con 0x7fa46c106170 2024-04-23T10:27:08.840 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:27:08.834+0000 7fa4627fc640 1 -- 172.21.15.114:0/387963399 <== mon.0 v2:172.21.15.114:3300/0 4 ==== mgrmap(e 25) ==== 138880+0+0 (secure 0 0 0) 0x7fa45c017070 con 0x7fa46c106170 2024-04-23T10:27:08.841 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:27:08.834+0000 7fa4763d5740 1 Processor -- start 2024-04-23T10:27:08.841 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:27:08.834+0000 7fa4763d5740 1 -- start start 2024-04-23T10:27:08.841 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:27:08.834+0000 7fa4763d5740 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x562dbfd1b620 0x562dbfd1ba00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T10:27:08.841 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:27:08.834+0000 7fa4763d5740 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x562dbfd1bf40 0x562dbfd262c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T10:27:08.841 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:27:08.834+0000 7fa4763d5740 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x562dbfd26800 0x562dbfd28bf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T10:27:08.841 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:27:08.834+0000 7fa4763d5740 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x562dbfc80970 con 0x562dbfd1b620 2024-04-23T10:27:08.841 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:27:08.834+0000 7fa4627fc640 1 --2- 172.21.15.114:0/387963399 >> [v2:172.21.15.114:6800/1395693466,v1:172.21.15.114:6801/1395693466] conn(0x7fa4580a40b0 0x7fa4580a6570 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T10:27:08.841 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:27:08.834+0000 7fa4763d5740 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 -- 0x562dbfc1d4a0 con 0x562dbfd1bf40 2024-04-23T10:27:08.841 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:27:08.834+0000 7fa4763d5740 1 -- --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_getmap magic: 0 -- 0x562dbfc7c930 con 0x562dbfd26800 2024-04-23T10:27:08.841 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:27:08.835+0000 7fa471b7f640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x562dbfd1b620 0x562dbfd1ba00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T10:27:08.841 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:27:08.835+0000 7fa471b7f640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x562dbfd1b620 0x562dbfd1ba00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.114:58568/0 (socket says 172.21.15.114:58568) 2024-04-23T10:27:08.841 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:27:08.835+0000 7fa471b7f640 1 -- 172.21.15.114:0/3582606545 learned_addr learned my addr 172.21.15.114:0/3582606545 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-04-23T10:27:08.841 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:27:08.835+0000 7fa471b7f640 1 --2- 172.21.15.114:0/387963399 >> [v2:172.21.15.114:6800/1395693466,v1:172.21.15.114:6801/1395693466] conn(0x7fa4580a40b0 0x7fa4580a6570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T10:27:08.841 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:27:08.835+0000 7fa4627fc640 1 -- 172.21.15.114:0/387963399 <== mon.0 v2:172.21.15.114:3300/0 5 ==== osd_map(90..90 src has 1..90) ==== 8527+0+0 (secure 0 0 0) 0x7fa45c0d0670 con 0x7fa46c106170 2024-04-23T10:27:08.842 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:27:08.835+0000 7fa470b7d640 1 --2- 172.21.15.114:0/3582606545 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x562dbfd26800 0x562dbfd28bf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T10:27:08.842 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:27:08.835+0000 7fa47137e640 1 --2- 172.21.15.114:0/3582606545 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x562dbfd1bf40 0x562dbfd262c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T10:27:08.842 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:27:08.835+0000 7fa471b7f640 1 -- 172.21.15.114:0/3582606545 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x562dbfd26800 msgr2=0x562dbfd28bf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T10:27:08.842 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:27:08.835+0000 7fa471b7f640 1 --2- 172.21.15.114:0/3582606545 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x562dbfd26800 0x562dbfd28bf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:27:08.842 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:27:08.835+0000 7fa471b7f640 1 -- 172.21.15.114:0/3582606545 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x562dbfd1bf40 msgr2=0x562dbfd262c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T10:27:08.842 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:27:08.835+0000 7fa471b7f640 1 --2- 172.21.15.114:0/3582606545 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x562dbfd1bf40 0x562dbfd262c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:27:08.842 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:27:08.835+0000 7fa471b7f640 1 -- 172.21.15.114:0/3582606545 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x562dbf993210 con 0x562dbfd1b620 2024-04-23T10:27:08.842 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:27:08.835+0000 7fa471b7f640 1 --2- 172.21.15.114:0/3582606545 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x562dbfd1b620 0x562dbfd1ba00 secure :-1 s=READY pgs=502 cs=0 l=1 rev1=1 crypto rx=0x7fa464012cf0 tx=0x7fa464010740 comp rx=0 tx=0).ready entity=mon.0 client_cookie=803c3caf647966aa server_cookie=0 in_seq=0 out_seq=0 2024-04-23T10:27:08.842 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:27:08.835+0000 7fa471b7f640 1 --2- 172.21.15.114:0/387963399 >> [v2:172.21.15.114:6800/1395693466,v1:172.21.15.114:6801/1395693466] conn(0x7fa4580a40b0 0x7fa4580a6570 secure :-1 s=READY pgs=65 cs=0 l=1 rev1=1 crypto rx=0x7fa464010c40 tx=0x7fa464017000 comp rx=0 tx=0).ready entity=mgr.15291 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T10:27:08.842 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:27:08.835+0000 7fa4577fe640 1 -- 172.21.15.114:0/3582606545 <== mon.0 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa464022070 con 0x562dbfd1b620 2024-04-23T10:27:08.842 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:27:08.835+0000 7fa4577fe640 1 -- 172.21.15.114:0/3582606545 <== mon.0 v2:172.21.15.114:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fa464009cb0 con 0x562dbfd1b620 2024-04-23T10:27:08.842 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:27:08.835+0000 7fa4577fe640 1 -- 172.21.15.114:0/3582606545 <== mon.0 v2:172.21.15.114:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa46400cdd0 con 0x562dbfd1b620 2024-04-23T10:27:08.842 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:27:08.836+0000 7fa4763d5740 1 -- 172.21.15.114:0/3582606545 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x562dbfd1b620 msgr2=0x562dbfd1ba00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T10:27:08.843 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:27:08.836+0000 7fa4763d5740 1 --2- 172.21.15.114:0/3582606545 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x562dbfd1b620 0x562dbfd1ba00 secure :-1 s=READY pgs=502 cs=0 l=1 rev1=1 crypto rx=0x7fa464012cf0 tx=0x7fa464010740 comp rx=0 tx=0).stop 2024-04-23T10:27:08.843 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:27:08.836+0000 7fa4763d5740 1 -- 172.21.15.114:0/3582606545 shutdown_connections 2024-04-23T10:27:08.843 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:27:08.836+0000 7fa4763d5740 1 --2- 172.21.15.114:0/3582606545 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x562dbfd26800 0x562dbfd28bf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:27:08.843 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:27:08.836+0000 7fa4763d5740 1 --2- 172.21.15.114:0/3582606545 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x562dbfd1bf40 0x562dbfd262c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:27:08.843 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:27:08.836+0000 7fa4763d5740 1 --2- 172.21.15.114:0/3582606545 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x562dbfd1b620 0x562dbfd1ba00 unknown :-1 s=CLOSED pgs=502 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:27:08.843 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:27:08.836+0000 7fa4763d5740 1 -- 172.21.15.114:0/3582606545 >> 172.21.15.114:0/3582606545 conn(0x562dbfd17b30 msgr2=0x562dbfd17f10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T10:27:08.843 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:27:08.836+0000 7fa4763d5740 1 -- 172.21.15.114:0/3582606545 shutdown_connections 2024-04-23T10:27:08.843 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:27:08.836+0000 7fa4763d5740 1 -- 172.21.15.114:0/3582606545 wait complete. 2024-04-23T10:27:08.843 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:27:08.836+0000 7fa4763d5740 1 -- start start 2024-04-23T10:27:08.843 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:27:08.837+0000 7fa4763d5740 1 Processor -- start 2024-04-23T10:27:08.843 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:27:08.837+0000 7fa4763d5740 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x562dbfd1b620 0x562dbfd261b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T10:27:08.843 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:27:08.837+0000 7fa4763d5740 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x562dbfd1bf40 0x562dbfd234a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T10:27:08.843 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:27:08.837+0000 7fa4763d5740 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x562dbfd26800 0x562dbfd23a40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T10:27:08.844 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:27:08.837+0000 7fa4763d5740 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x562dbfd293d0 con 0x562dbfd1b620 2024-04-23T10:27:08.844 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:27:08.837+0000 7fa4763d5740 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 -- 0x562dbfc1b0b0 con 0x562dbfd26800 2024-04-23T10:27:08.844 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:27:08.837+0000 7fa4763d5740 1 -- --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_getmap magic: 0 -- 0x562dbfd29550 con 0x562dbfd1bf40 2024-04-23T10:27:08.844 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:27:08.837+0000 7fa4763d5740 10 client.0 _ll_register_callbacks cb 0x562dbfc08d30 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-23T10:27:08.844 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:27:08.837+0000 7fa47137e640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x562dbfd1b620 0x562dbfd261b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T10:27:08.844 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:27:08.837+0000 7fa47137e640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x562dbfd1b620 0x562dbfd261b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.114:58576/0 (socket says 172.21.15.114:58576) 2024-04-23T10:27:08.844 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:27:08.837+0000 7fa47137e640 1 -- 172.21.15.114:0/3003642374 learned_addr learned my addr 172.21.15.114:0/3003642374 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-04-23T10:27:08.844 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:27:08.837+0000 7fa471b7f640 1 --2- 172.21.15.114:0/3003642374 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x562dbfd26800 0x562dbfd23a40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T10:27:08.844 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:27:08.837+0000 7fa470b7d640 1 --2- 172.21.15.114:0/3003642374 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x562dbfd1bf40 0x562dbfd234a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T10:27:08.844 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:27:08.837+0000 7fa47137e640 1 -- 172.21.15.114:0/3003642374 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x562dbfd1bf40 msgr2=0x562dbfd234a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T10:27:08.844 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:27:08.837+0000 7fa47137e640 1 --2- 172.21.15.114:0/3003642374 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x562dbfd1bf40 0x562dbfd234a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:27:08.844 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:27:08.837+0000 7fa47137e640 1 -- 172.21.15.114:0/3003642374 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x562dbfd26800 msgr2=0x562dbfd23a40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T10:27:08.844 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:27:08.837+0000 7fa47137e640 1 --2- 172.21.15.114:0/3003642374 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x562dbfd26800 0x562dbfd23a40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:27:08.845 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:27:08.837+0000 7fa47137e640 1 -- 172.21.15.114:0/3003642374 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({osdmap=0}) -- 0x562dbfa5f620 con 0x562dbfd1b620 2024-04-23T10:27:08.845 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:27:08.838+0000 7fa47137e640 1 --2- 172.21.15.114:0/3003642374 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x562dbfd1b620 0x562dbfd261b0 secure :-1 s=READY pgs=503 cs=0 l=1 rev1=1 crypto rx=0x7fa46802ac00 tx=0x7fa468031750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=bc3831b2a9afc500 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T10:27:08.845 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:27:08.838+0000 7fa454ff9640 10 client.0 ms_handle_connect on v2:172.21.15.114:3300/0 2024-04-23T10:27:08.845 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:27:08.838+0000 7fa454ff9640 1 -- 172.21.15.114:0/3003642374 <== mon.0 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa46803f070 con 0x562dbfd1b620 2024-04-23T10:27:08.845 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:27:08.838+0000 7fa454ff9640 1 -- 172.21.15.114:0/3003642374 <== mon.0 v2:172.21.15.114:3300/0 2 ==== osd_map(90..90 src has 1..90) ==== 8527+0+0 (secure 0 0 0) 0x7fa468004450 con 0x562dbfd1b620 2024-04-23T10:27:08.845 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:27:08.838+0000 7fa4763d5740 1 -- 172.21.15.114:0/3003642374 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fa464009cb0 con 0x562dbfd1b620 2024-04-23T10:27:08.845 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:27:08.838+0000 7fa454ff9640 1 -- 172.21.15.114:0/3003642374 <== mon.0 v2:172.21.15.114:3300/0 3 ==== mon_get_version_reply(handle=1 version=21) ==== 24+0+0 (secure 0 0 0) 0x7fa468035070 con 0x562dbfd1b620 2024-04-23T10:27:08.845 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:27:08.838+0000 7fa4763d5740 10 client.15612 fetch_fsmap learned FSMap version 21 2024-04-23T10:27:08.845 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:27:08.838+0000 7fa4763d5740 1 -- 172.21.15.114:0/3003642374 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({config=0+,fsmap=21,monmap=4+}) -- 0x562dbfd35340 con 0x562dbfd1b620 2024-04-23T10:27:08.845 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:27:08.838+0000 7fa454ff9640 1 -- 172.21.15.114:0/3003642374 <== mon.0 v2:172.21.15.114:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fa464009cb0 con 0x562dbfd1b620 2024-04-23T10:27:08.845 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:27:08.838+0000 7fa454ff9640 1 -- 172.21.15.114:0/3003642374 <== mon.0 v2:172.21.15.114:3300/0 5 ==== fsmap(e 21) ==== 8338+0+0 (secure 0 0 0) 0x7fa468004ad0 con 0x562dbfd1b620 2024-04-23T10:27:08.845 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:27:08.838+0000 7fa4763d5740 10 client.15612 fetch_fsmap finished waiting for FSMap version 21 2024-04-23T10:27:08.845 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:27:08.838+0000 7fa4763d5740 10 client.15612 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-04-23T10:27:08.845 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:27:08.838+0000 7fa4763d5740 20 client.15612 populate_metadata read hostname 'smithi114' 2024-04-23T10:27:08.846 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:27:08.838+0000 7fa4763d5740 1 --2- 172.21.15.114:0/3003642374 >> [v2:172.21.15.119:6838/937297340,v1:172.21.15.119:6839/937297340] conn(0x562dbfd36800 0x562dbfd38bf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T10:27:08.846 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:27:08.838+0000 7fa4763d5740 4 client.15612 mds_command: new command op to 24545 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-23T10:27:08.846 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:27:08.838+0000 7fa4763d5740 1 -- 172.21.15.114:0/3003642374 --> [v2:172.21.15.119:6838/937297340,v1:172.21.15.119:6839/937297340] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x562dbfc8e390 con 0x562dbfd36800 2024-04-23T10:27:08.846 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:27:08.839+0000 7fa470b7d640 1 --2- 172.21.15.114:0/3003642374 >> [v2:172.21.15.119:6838/937297340,v1:172.21.15.119:6839/937297340] conn(0x562dbfd36800 0x562dbfd38bf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T10:27:08.846 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:27:08.839+0000 7fa470b7d640 1 --2- 172.21.15.114:0/3003642374 >> [v2:172.21.15.119:6838/937297340,v1:172.21.15.119:6839/937297340] conn(0x562dbfd36800 0x562dbfd38bf0 crc :-1 s=READY pgs=44 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=e51873a33c41933f server_cookie=a8e2fbf4f88dee3f in_seq=0 out_seq=0 2024-04-23T10:27:08.846 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:27:08.839+0000 7fa454ff9640 10 client.15612 ms_handle_connect on v2:172.21.15.119:6838/937297340 2024-04-23T10:27:08.846 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:27:08.840+0000 7fa454ff9640 1 -- 172.21.15.114:0/3003642374 <== mds.0 v2:172.21.15.119:6838/937297340 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x562dbfc8e390 con 0x562dbfd36800 2024-04-23T10:27:08.846 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:27:08.840+0000 7fa454ff9640 10 client.15612 handle_command_reply: tid=0 2024-04-23T10:27:08.846 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:27:08.840+0000 7fa4763d5740 1 client.15612 shutdown 2024-04-23T10:27:08.846 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:27:08.840+0000 7fa4763d5740 1 -- 172.21.15.114:0/3003642374 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x562dbfd1b620 msgr2=0x562dbfd261b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T10:27:08.846 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:27:08.840+0000 7fa4763d5740 1 --2- 172.21.15.114:0/3003642374 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x562dbfd1b620 0x562dbfd261b0 secure :-1 s=READY pgs=503 cs=0 l=1 rev1=1 crypto rx=0x7fa46802ac00 tx=0x7fa468031750 comp rx=0 tx=0).stop 2024-04-23T10:27:08.846 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:27:08.840+0000 7fa4763d5740 1 -- 172.21.15.114:0/3003642374 shutdown_connections 2024-04-23T10:27:08.846 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:27:08.840+0000 7fa4763d5740 1 --2- 172.21.15.114:0/3003642374 >> [v2:172.21.15.119:6838/937297340,v1:172.21.15.119:6839/937297340] conn(0x562dbfd36800 0x562dbfd38bf0 crc :-1 s=READY pgs=44 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:27:08.847 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:27:08.840+0000 7fa4763d5740 1 --2- 172.21.15.114:0/3003642374 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x562dbfd26800 0x562dbfd23a40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:27:08.847 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:27:08.840+0000 7fa4763d5740 1 --2- 172.21.15.114:0/3003642374 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x562dbfd1bf40 0x562dbfd234a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:27:08.847 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:27:08.840+0000 7fa4763d5740 1 --2- 172.21.15.114:0/3003642374 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x562dbfd1b620 0x562dbfd261b0 unknown :-1 s=CLOSED pgs=503 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:27:08.847 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:27:08.840+0000 7fa454ff9640 0 client.15612 ms_handle_reset on v2:172.21.15.119:6838/937297340 2024-04-23T10:27:08.847 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:27:08.840+0000 7fa4763d5740 1 -- 172.21.15.114:0/3003642374 >> 172.21.15.114:0/3003642374 conn(0x562dbfd17b30 msgr2=0x562dbfd1d860 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T10:27:08.847 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:27:08.840+0000 7fa4763d5740 1 -- 172.21.15.114:0/3003642374 shutdown_connections 2024-04-23T10:27:08.847 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:27:08.840+0000 7fa4763d5740 1 -- 172.21.15.114:0/3003642374 wait complete. 2024-04-23T10:27:08.847 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:27:08.840+0000 7fa4763d5740 20 client.15612 trim_cache size 0 max 16384 2024-04-23T10:27:08.858 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:27:08.855+0000 7fa4763d5740 1 Processor -- start 2024-04-23T10:27:08.858 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:27:08.855+0000 7fa4763d5740 1 -- start start 2024-04-23T10:27:08.858 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:27:08.855+0000 7fa4763d5740 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x562dbfd1b620 0x562dbfd32290 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T10:27:08.858 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:27:08.855+0000 7fa4763d5740 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x562dbfd1bf40 0x562dbfd16b30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T10:27:08.859 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:27:08.855+0000 7fa4763d5740 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x562dbfd26800 0x562dbfd25ce0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T10:27:08.859 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:27:08.855+0000 7fa4763d5740 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x7fa46c1b2610 con 0x562dbfd1bf40 2024-04-23T10:27:08.859 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:27:08.855+0000 7fa4763d5740 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 -- 0x7fa46c1afee0 con 0x562dbfd26800 2024-04-23T10:27:08.859 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:27:08.855+0000 7fa4763d5740 1 -- --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_getmap magic: 0 -- 0x7fa46c1add30 con 0x562dbfd1b620 2024-04-23T10:27:08.859 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:27:08.855+0000 7fa470b7d640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x562dbfd1bf40 0x562dbfd16b30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T10:27:08.859 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:27:08.855+0000 7fa470b7d640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x562dbfd1bf40 0x562dbfd16b30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.114:58586/0 (socket says 172.21.15.114:58586) 2024-04-23T10:27:08.859 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:27:08.855+0000 7fa470b7d640 1 -- 172.21.15.114:0/3519812954 learned_addr learned my addr 172.21.15.114:0/3519812954 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-04-23T10:27:08.859 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:27:08.855+0000 7fa47137e640 1 --2- 172.21.15.114:0/3519812954 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x562dbfd1b620 0x562dbfd32290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T10:27:08.859 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:27:08.855+0000 7fa470b7d640 1 -- 172.21.15.114:0/3519812954 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x562dbfd1b620 msgr2=0x562dbfd32290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T10:27:08.859 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:27:08.855+0000 7fa470b7d640 1 --2- 172.21.15.114:0/3519812954 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x562dbfd1b620 0x562dbfd32290 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:27:08.859 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:27:08.855+0000 7fa470b7d640 1 -- 172.21.15.114:0/3519812954 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x562dbfd26800 msgr2=0x562dbfd25ce0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T10:27:08.859 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:27:08.855+0000 7fa470b7d640 1 --2- 172.21.15.114:0/3519812954 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x562dbfd26800 0x562dbfd25ce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:27:08.859 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:27:08.855+0000 7fa470b7d640 1 -- 172.21.15.114:0/3519812954 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x562dbfd132e0 con 0x562dbfd1bf40 2024-04-23T10:27:08.860 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:27:08.855+0000 7fa47137e640 1 --2- 172.21.15.114:0/3519812954 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x562dbfd1b620 0x562dbfd32290 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-23T10:27:08.860 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:27:08.855+0000 7fa470b7d640 1 --2- 172.21.15.114:0/3519812954 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x562dbfd1bf40 0x562dbfd16b30 secure :-1 s=READY pgs=504 cs=0 l=1 rev1=1 crypto rx=0x7fa45c098820 tx=0x7fa45c0b2040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e3b476a32ae36fd2 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T10:27:08.860 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:27:08.856+0000 7fa456ffd640 1 -- 172.21.15.114:0/3519812954 <== mon.0 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa45c0b0e60 con 0x562dbfd1bf40 2024-04-23T10:27:08.860 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:27:08.856+0000 7fa456ffd640 1 -- 172.21.15.114:0/3519812954 <== mon.0 v2:172.21.15.114:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fa45c0a0770 con 0x562dbfd1bf40 2024-04-23T10:27:08.860 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:27:08.856+0000 7fa456ffd640 1 -- 172.21.15.114:0/3519812954 <== mon.0 v2:172.21.15.114:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa45c0b3120 con 0x562dbfd1bf40 2024-04-23T10:27:08.860 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:27:08.856+0000 7fa4763d5740 1 -- 172.21.15.114:0/3519812954 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x562dbfd1bf40 msgr2=0x562dbfd16b30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T10:27:08.860 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:27:08.856+0000 7fa4763d5740 1 --2- 172.21.15.114:0/3519812954 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x562dbfd1bf40 0x562dbfd16b30 secure :-1 s=READY pgs=504 cs=0 l=1 rev1=1 crypto rx=0x7fa45c098820 tx=0x7fa45c0b2040 comp rx=0 tx=0).stop 2024-04-23T10:27:08.860 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:27:08.856+0000 7fa4763d5740 1 -- 172.21.15.114:0/3519812954 shutdown_connections 2024-04-23T10:27:08.860 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:27:08.856+0000 7fa4763d5740 1 --2- 172.21.15.114:0/3519812954 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x562dbfd26800 0x562dbfd25ce0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:27:08.860 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:27:08.856+0000 7fa4763d5740 1 --2- 172.21.15.114:0/3519812954 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x562dbfd1bf40 0x562dbfd16b30 unknown :-1 s=CLOSED pgs=504 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:27:08.860 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:27:08.856+0000 7fa4763d5740 1 --2- 172.21.15.114:0/3519812954 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x562dbfd1b620 0x562dbfd32290 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:27:08.860 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:27:08.856+0000 7fa4763d5740 1 -- 172.21.15.114:0/3519812954 >> 172.21.15.114:0/3519812954 conn(0x562dbfd17b30 msgr2=0x562dbfd16420 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T10:27:08.860 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:27:08.856+0000 7fa4763d5740 1 -- 172.21.15.114:0/3519812954 shutdown_connections 2024-04-23T10:27:08.860 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:27:08.856+0000 7fa4763d5740 1 -- 172.21.15.114:0/3519812954 wait complete. 2024-04-23T10:27:08.861 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:27:08.857+0000 7fa4763d5740 1 -- start start 2024-04-23T10:27:08.861 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:27:08.857+0000 7fa4763d5740 1 Processor -- start 2024-04-23T10:27:08.861 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:27:08.857+0000 7fa4763d5740 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x562dbfd1b620 0x562dbfd20170 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T10:27:08.861 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:27:08.857+0000 7fa4763d5740 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x562dbfd1bf40 0x562dbfd206b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T10:27:08.861 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:27:08.857+0000 7fa4763d5740 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x562dbfd26800 0x562dbfd20bf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T10:27:08.861 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:27:08.857+0000 7fa4763d5740 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x7fa46c1ae3a0 con 0x562dbfd26800 2024-04-23T10:27:08.861 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:27:08.857+0000 7fa4763d5740 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 -- 0x7fa46c1af810 con 0x562dbfd1b620 2024-04-23T10:27:08.861 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:27:08.857+0000 7fa4763d5740 1 -- --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_getmap magic: 0 -- 0x7fa46c1aea70 con 0x562dbfd1bf40 2024-04-23T10:27:08.861 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:27:08.857+0000 7fa4763d5740 10 client.0 _ll_register_callbacks cb 0x7fa468005d50 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-23T10:27:08.861 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:27:08.857+0000 7fa471b7f640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x562dbfd26800 0x562dbfd20bf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T10:27:08.861 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:27:08.857+0000 7fa471b7f640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x562dbfd26800 0x562dbfd20bf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.114:58596/0 (socket says 172.21.15.114:58596) 2024-04-23T10:27:08.861 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:27:08.857+0000 7fa471b7f640 1 -- 172.21.15.114:0/2327306453 learned_addr learned my addr 172.21.15.114:0/2327306453 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-04-23T10:27:08.861 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:27:08.857+0000 7fa470b7d640 1 --2- 172.21.15.114:0/2327306453 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x562dbfd1bf40 0x562dbfd206b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T10:27:08.861 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:27:08.857+0000 7fa47137e640 1 --2- 172.21.15.114:0/2327306453 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x562dbfd1b620 0x562dbfd20170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T10:27:08.862 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:27:08.857+0000 7fa471b7f640 1 -- 172.21.15.114:0/2327306453 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x562dbfd1bf40 msgr2=0x562dbfd206b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T10:27:08.862 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:27:08.857+0000 7fa471b7f640 1 --2- 172.21.15.114:0/2327306453 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x562dbfd1bf40 0x562dbfd206b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:27:08.862 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:27:08.857+0000 7fa471b7f640 1 -- 172.21.15.114:0/2327306453 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x562dbfd1b620 msgr2=0x562dbfd20170 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T10:27:08.862 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:27:08.857+0000 7fa471b7f640 1 --2- 172.21.15.114:0/2327306453 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x562dbfd1b620 0x562dbfd20170 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:27:08.862 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:27:08.857+0000 7fa471b7f640 1 -- 172.21.15.114:0/2327306453 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({osdmap=0}) -- 0x562dbfd1bd70 con 0x562dbfd26800 2024-04-23T10:27:08.862 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:27:08.857+0000 7fa47137e640 1 --2- 172.21.15.114:0/2327306453 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x562dbfd1b620 0x562dbfd20170 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-23T10:27:08.862 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:27:08.858+0000 7fa471b7f640 1 --2- 172.21.15.114:0/2327306453 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x562dbfd26800 0x562dbfd20bf0 secure :-1 s=READY pgs=505 cs=0 l=1 rev1=1 crypto rx=0x7fa464012a10 tx=0x7fa4640125b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7ad2b86f554c24f6 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T10:27:08.862 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:27:08.858+0000 7fa454ff9640 10 client.0 ms_handle_connect on v2:172.21.15.114:3300/0 2024-04-23T10:27:08.862 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:27:08.858+0000 7fa454ff9640 1 -- 172.21.15.114:0/2327306453 <== mon.0 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa464022040 con 0x562dbfd26800 2024-04-23T10:27:08.862 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:27:08.858+0000 7fa454ff9640 1 -- 172.21.15.114:0/2327306453 <== mon.0 v2:172.21.15.114:3300/0 2 ==== osd_map(90..90 src has 1..90) ==== 8527+0+0 (secure 0 0 0) 0x7fa46400cb40 con 0x562dbfd26800 2024-04-23T10:27:08.862 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:27:08.858+0000 7fa4763d5740 1 -- 172.21.15.114:0/2327306453 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fa45c0a0770 con 0x562dbfd26800 2024-04-23T10:27:08.862 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:27:08.858+0000 7fa454ff9640 1 -- 172.21.15.114:0/2327306453 <== mon.0 v2:172.21.15.114:3300/0 3 ==== mon_get_version_reply(handle=1 version=21) ==== 24+0+0 (secure 0 0 0) 0x7fa464012ea0 con 0x562dbfd26800 2024-04-23T10:27:08.862 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:27:08.858+0000 7fa4763d5740 10 client.15618 fetch_fsmap learned FSMap version 21 2024-04-23T10:27:08.863 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:27:08.858+0000 7fa4763d5740 1 -- 172.21.15.114:0/2327306453 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({config=0+,fsmap=21,monmap=4+}) -- 0x562dbfd9ce90 con 0x562dbfd26800 2024-04-23T10:27:08.863 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:27:08.858+0000 7fa454ff9640 1 -- 172.21.15.114:0/2327306453 <== mon.0 v2:172.21.15.114:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fa45c0a0770 con 0x562dbfd26800 2024-04-23T10:27:08.863 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:27:08.858+0000 7fa454ff9640 1 -- 172.21.15.114:0/2327306453 <== mon.0 v2:172.21.15.114:3300/0 5 ==== fsmap(e 21) ==== 8338+0+0 (secure 0 0 0) 0x7fa464005170 con 0x562dbfd26800 2024-04-23T10:27:08.863 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:27:08.858+0000 7fa4763d5740 10 client.15618 fetch_fsmap finished waiting for FSMap version 21 2024-04-23T10:27:08.863 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:27:08.858+0000 7fa4763d5740 10 client.15618 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-04-23T10:27:08.863 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:27:08.858+0000 7fa4763d5740 20 client.15618 populate_metadata read hostname 'smithi114' 2024-04-23T10:27:08.863 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:27:08.858+0000 7fa4763d5740 1 --2- 172.21.15.114:0/2327306453 >> [v2:172.21.15.119:6838/937297340,v1:172.21.15.119:6839/937297340] conn(0x562dbfd9d350 0x562dbfd9f770 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T10:27:08.863 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:27:08.858+0000 7fa4763d5740 4 client.15618 mds_command: new command op to 24545 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-04-23T10:27:08.863 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:27:08.858+0000 7fa4763d5740 1 -- 172.21.15.114:0/2327306453 --> [v2:172.21.15.119:6838/937297340,v1:172.21.15.119:6839/937297340] -- command(tid 0: {"prefix": "flush journal"}) -- 0x562dbfc88200 con 0x562dbfd9d350 2024-04-23T10:27:08.863 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:27:08.859+0000 7fa47137e640 1 --2- 172.21.15.114:0/2327306453 >> [v2:172.21.15.119:6838/937297340,v1:172.21.15.119:6839/937297340] conn(0x562dbfd9d350 0x562dbfd9f770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T10:27:08.863 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:27:08.859+0000 7fa47137e640 1 --2- 172.21.15.114:0/2327306453 >> [v2:172.21.15.119:6838/937297340,v1:172.21.15.119:6839/937297340] conn(0x562dbfd9d350 0x562dbfd9f770 crc :-1 s=READY pgs=45 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=51253ead83d8823 server_cookie=e84e77994de59da6 in_seq=0 out_seq=0 2024-04-23T10:27:08.863 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:27:08.859+0000 7fa454ff9640 10 client.15618 ms_handle_connect on v2:172.21.15.119:6838/937297340 2024-04-23T10:27:08.866 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:27:08.866+0000 7fa454ff9640 1 -- 172.21.15.114:0/2327306453 <== mds.0 v2:172.21.15.119:6838/937297340 1 ==== command_reply(tid 0: 0 ) ==== 8+0+44 (crc 0 0 0) 0x562dbfc88200 con 0x562dbfd9d350 2024-04-23T10:27:08.866 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:27:08.866+0000 7fa454ff9640 10 client.15618 handle_command_reply: tid=0 2024-04-23T10:27:08.866 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:27:08.866+0000 7fa4763d5740 1 client.15618 shutdown 2024-04-23T10:27:08.866 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:27:08.867+0000 7fa4763d5740 1 -- 172.21.15.114:0/2327306453 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x562dbfd26800 msgr2=0x562dbfd20bf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T10:27:08.866 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:27:08.867+0000 7fa4763d5740 1 --2- 172.21.15.114:0/2327306453 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x562dbfd26800 0x562dbfd20bf0 secure :-1 s=READY pgs=505 cs=0 l=1 rev1=1 crypto rx=0x7fa464012a10 tx=0x7fa4640125b0 comp rx=0 tx=0).stop 2024-04-23T10:27:08.866 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:27:08.867+0000 7fa4763d5740 1 -- 172.21.15.114:0/2327306453 shutdown_connections 2024-04-23T10:27:08.866 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:27:08.867+0000 7fa4763d5740 1 --2- 172.21.15.114:0/2327306453 >> [v2:172.21.15.119:6838/937297340,v1:172.21.15.119:6839/937297340] conn(0x562dbfd9d350 0x562dbfd9f770 crc :-1 s=READY pgs=45 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:27:08.866 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:27:08.867+0000 7fa4763d5740 1 --2- 172.21.15.114:0/2327306453 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x562dbfd26800 0x562dbfd20bf0 unknown :-1 s=CLOSED pgs=505 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:27:08.867 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:27:08.867+0000 7fa454ff9640 0 client.15618 ms_handle_reset on v2:172.21.15.119:6838/937297340 2024-04-23T10:27:08.867 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:27:08.867+0000 7fa4763d5740 1 --2- 172.21.15.114:0/2327306453 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x562dbfd1bf40 0x562dbfd206b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:27:08.867 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:27:08.867+0000 7fa4763d5740 1 --2- 172.21.15.114:0/2327306453 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x562dbfd1b620 0x562dbfd20170 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:27:08.867 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:27:08.867+0000 7fa4763d5740 1 -- 172.21.15.114:0/2327306453 >> 172.21.15.114:0/2327306453 conn(0x562dbfd17b30 msgr2=0x562dbfd1e9b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T10:27:08.867 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:27:08.867+0000 7fa4763d5740 1 -- 172.21.15.114:0/2327306453 shutdown_connections 2024-04-23T10:27:08.867 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:27:08.867+0000 7fa4763d5740 1 -- 172.21.15.114:0/2327306453 wait complete. 2024-04-23T10:27:08.867 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:27:08.867+0000 7fa4763d5740 20 client.15618 trim_cache size 0 max 16384 2024-04-23T10:27:08.867 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stdout:{ 2024-04-23T10:27:08.867 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stdout: "message": "", 2024-04-23T10:27:08.867 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stdout: "return_code": 0 2024-04-23T10:27:08.868 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stdout:} 2024-04-23T10:27:08.868 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:27:08.868+0000 7fa4735e2640 1 -- 172.21.15.114:0/387963399 >> [v2:172.21.15.114:6800/1395693466,v1:172.21.15.114:6801/1395693466] conn(0x7fa4580a40b0 msgr2=0x7fa4580a6570 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T10:27:08.868 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:27:08.868+0000 7fa4735e2640 1 --2- 172.21.15.114:0/387963399 >> [v2:172.21.15.114:6800/1395693466,v1:172.21.15.114:6801/1395693466] conn(0x7fa4580a40b0 0x7fa4580a6570 secure :-1 s=READY pgs=65 cs=0 l=1 rev1=1 crypto rx=0x7fa464010c40 tx=0x7fa464017000 comp rx=0 tx=0).stop 2024-04-23T10:27:08.868 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:27:08.868+0000 7fa4735e2640 1 -- 172.21.15.114:0/387963399 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fa46c106170 msgr2=0x7fa46c103070 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T10:27:08.869 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:27:08.868+0000 7fa4735e2640 1 --2- 172.21.15.114:0/387963399 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fa46c106170 0x7fa46c103070 secure :-1 s=READY pgs=501 cs=0 l=1 rev1=1 crypto rx=0x7fa45c00c970 tx=0x7fa45c00ce30 comp rx=0 tx=0).stop 2024-04-23T10:27:08.869 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:27:08.868+0000 7fa4735e2640 1 -- 172.21.15.114:0/387963399 shutdown_connections 2024-04-23T10:27:08.869 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:27:08.868+0000 7fa4735e2640 1 --2- 172.21.15.114:0/387963399 >> [v2:172.21.15.114:6800/1395693466,v1:172.21.15.114:6801/1395693466] conn(0x7fa4580a40b0 0x7fa4580a6570 unknown :-1 s=CLOSED pgs=65 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:27:08.869 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:27:08.868+0000 7fa4735e2640 1 --2- 172.21.15.114:0/387963399 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fa46c1035b0 0x7fa46c1a6810 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:27:08.869 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:27:08.868+0000 7fa4735e2640 1 --2- 172.21.15.114:0/387963399 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fa46c106170 0x7fa46c103070 unknown :-1 s=CLOSED pgs=501 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:27:08.869 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:27:08.868+0000 7fa4735e2640 1 --2- 172.21.15.114:0/387963399 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fa46c1010f0 0x7fa46c102b30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:27:08.869 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:27:08.868+0000 7fa4735e2640 1 -- 172.21.15.114:0/387963399 >> 172.21.15.114:0/387963399 conn(0x7fa46c077fd0 msgr2=0x7fa46c0698e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T10:27:08.869 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:27:08.868+0000 7fa4735e2640 1 -- 172.21.15.114:0/387963399 shutdown_connections 2024-04-23T10:27:08.869 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:27:08.868+0000 7fa4735e2640 1 -- 172.21.15.114:0/387963399 wait complete. 2024-04-23T10:27:10.328 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 23 10:27:10 smithi204 ceph-mon[32649]: pgmap v17: 97 pgs: 97 active+clean; 1.3 MiB data, 377 MiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 25 KiB/s wr, 26 op/s 2024-04-23T10:27:10.492 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 23 10:27:10 smithi119 ceph-mon[32608]: pgmap v17: 97 pgs: 97 active+clean; 1.3 MiB data, 377 MiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 25 KiB/s wr, 26 op/s 2024-04-23T10:27:10.504 INFO:journalctl@ceph.mon.a.smithi114.stdout:Apr 23 10:27:10 smithi114 ceph-mon[29255]: pgmap v17: 97 pgs: 97 active+clean; 1.3 MiB data, 377 MiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 25 KiB/s wr, 26 op/s 2024-04-23T10:27:12.328 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 23 10:27:12 smithi204 ceph-mon[32649]: pgmap v18: 97 pgs: 97 active+clean; 1.3 MiB data, 377 MiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 24 KiB/s wr, 20 op/s 2024-04-23T10:27:12.492 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 23 10:27:12 smithi119 ceph-mon[32608]: pgmap v18: 97 pgs: 97 active+clean; 1.3 MiB data, 377 MiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 24 KiB/s wr, 20 op/s 2024-04-23T10:27:12.504 INFO:journalctl@ceph.mon.a.smithi114.stdout:Apr 23 10:27:12 smithi114 ceph-mon[29255]: pgmap v18: 97 pgs: 97 active+clean; 1.3 MiB data, 377 MiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 24 KiB/s wr, 20 op/s 2024-04-23T10:27:14.328 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 23 10:27:14 smithi204 ceph-mon[32649]: pgmap v19: 97 pgs: 97 active+clean; 1.3 MiB data, 377 MiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 25 KiB/s wr, 25 op/s 2024-04-23T10:27:14.492 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 23 10:27:14 smithi119 ceph-mon[32608]: pgmap v19: 97 pgs: 97 active+clean; 1.3 MiB data, 377 MiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 25 KiB/s wr, 25 op/s 2024-04-23T10:27:14.504 INFO:journalctl@ceph.mon.a.smithi114.stdout:Apr 23 10:27:14 smithi114 ceph-mon[29255]: pgmap v19: 97 pgs: 97 active+clean; 1.3 MiB data, 377 MiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 25 KiB/s wr, 25 op/s 2024-04-23T10:27:16.492 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 23 10:27:16 smithi119 ceph-mon[32608]: pgmap v20: 97 pgs: 97 active+clean; 1.3 MiB data, 377 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 2.7 KiB/s wr, 7 op/s 2024-04-23T10:27:16.504 INFO:journalctl@ceph.mon.a.smithi114.stdout:Apr 23 10:27:16 smithi114 ceph-mon[29255]: pgmap v20: 97 pgs: 97 active+clean; 1.3 MiB data, 377 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 2.7 KiB/s wr, 7 op/s 2024-04-23T10:27:16.578 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 23 10:27:16 smithi204 ceph-mon[32649]: pgmap v20: 97 pgs: 97 active+clean; 1.3 MiB data, 377 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 2.7 KiB/s wr, 7 op/s 2024-04-23T10:27:18.254 INFO:journalctl@ceph.mon.a.smithi114.stdout:Apr 23 10:27:18 smithi114 ceph-mon[29255]: pgmap v21: 97 pgs: 97 active+clean; 1.3 MiB data, 377 MiB used, 1.0 TiB / 1.0 TiB avail; 4.7 KiB/s rd, 2.8 KiB/s wr, 8 op/s 2024-04-23T10:27:18.492 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 23 10:27:18 smithi119 ceph-mon[32608]: pgmap v21: 97 pgs: 97 active+clean; 1.3 MiB data, 377 MiB used, 1.0 TiB / 1.0 TiB avail; 4.7 KiB/s rd, 2.8 KiB/s wr, 8 op/s 2024-04-23T10:27:18.578 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 23 10:27:18 smithi204 ceph-mon[32649]: pgmap v21: 97 pgs: 97 active+clean; 1.3 MiB data, 377 MiB used, 1.0 TiB / 1.0 TiB avail; 4.7 KiB/s rd, 2.8 KiB/s wr, 8 op/s 2024-04-23T10:27:20.492 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 23 10:27:20 smithi119 ceph-mon[32608]: pgmap v22: 97 pgs: 97 active+clean; 1.3 MiB data, 377 MiB used, 1.0 TiB / 1.0 TiB avail; 5.5 KiB/s rd, 2.8 KiB/s wr, 10 op/s 2024-04-23T10:27:20.504 INFO:journalctl@ceph.mon.a.smithi114.stdout:Apr 23 10:27:20 smithi114 ceph-mon[29255]: pgmap v22: 97 pgs: 97 active+clean; 1.3 MiB data, 377 MiB used, 1.0 TiB / 1.0 TiB avail; 5.5 KiB/s rd, 2.8 KiB/s wr, 10 op/s 2024-04-23T10:27:20.578 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 23 10:27:20 smithi204 ceph-mon[32649]: pgmap v22: 97 pgs: 97 active+clean; 1.3 MiB data, 377 MiB used, 1.0 TiB / 1.0 TiB avail; 5.5 KiB/s rd, 2.8 KiB/s wr, 10 op/s 2024-04-23T10:27:21.998 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:27:21.997+0000 7fb892b3d640 1 -- 172.21.15.114:0/3134973883 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fb88c101a70 msgr2=0x7fb88c105ab0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T10:27:21.998 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:27:21.997+0000 7fb892b3d640 1 --2- 172.21.15.114:0/3134973883 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fb88c101a70 0x7fb88c105ab0 secure :-1 s=READY pgs=506 cs=0 l=1 rev1=1 crypto rx=0x7fb878009900 tx=0x7fb87802a410 comp rx=0 tx=0).stop 2024-04-23T10:27:21.998 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:27:21.997+0000 7fb892b3d640 1 -- 172.21.15.114:0/3134973883 shutdown_connections 2024-04-23T10:27:21.998 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:27:21.997+0000 7fb892b3d640 1 --2- 172.21.15.114:0/3134973883 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fb88c106190 0x7fb88c10dd60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:27:21.998 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:27:21.997+0000 7fb892b3d640 1 --2- 172.21.15.114:0/3134973883 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fb88c101a70 0x7fb88c105ab0 unknown :-1 s=CLOSED pgs=506 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:27:21.999 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:27:21.997+0000 7fb892b3d640 1 --2- 172.21.15.114:0/3134973883 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fb88c1010c0 0x7fb88c1014a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:27:21.999 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:27:21.997+0000 7fb892b3d640 1 -- 172.21.15.114:0/3134973883 >> 172.21.15.114:0/3134973883 conn(0x7fb88c077f60 msgr2=0x7fb88c078370 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T10:27:21.999 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:27:21.997+0000 7fb892b3d640 1 -- 172.21.15.114:0/3134973883 shutdown_connections 2024-04-23T10:27:21.999 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:27:21.997+0000 7fb892b3d640 1 -- 172.21.15.114:0/3134973883 wait complete. 2024-04-23T10:27:21.999 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:27:21.998+0000 7fb892b3d640 1 Processor -- start 2024-04-23T10:27:21.999 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:27:21.998+0000 7fb892b3d640 1 -- start start 2024-04-23T10:27:21.999 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:27:21.998+0000 7fb892b3d640 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fb88c1010c0 0x7fb88c1986c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T10:27:21.999 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:27:21.998+0000 7fb892b3d640 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fb88c106190 0x7fb88c198c00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T10:27:21.999 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:27:21.998+0000 7fb892b3d640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fb88c19cf80 0x7fb88c19d430 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T10:27:21.999 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:27:21.998+0000 7fb892b3d640 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x7fb88c1121c0 con 0x7fb88c19cf80 2024-04-23T10:27:21.999 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:27:21.998+0000 7fb892b3d640 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 -- 0x7fb88c112040 con 0x7fb88c1010c0 2024-04-23T10:27:21.999 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:27:21.998+0000 7fb892b3d640 1 -- --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_getmap magic: 0 -- 0x7fb88c112340 con 0x7fb88c106190 2024-04-23T10:27:21.999 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:27:21.998+0000 7fb8910da640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fb88c19cf80 0x7fb88c19d430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T10:27:21.999 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:27:21.998+0000 7fb8910da640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fb88c19cf80 0x7fb88c19d430 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.114:60704/0 (socket says 172.21.15.114:60704) 2024-04-23T10:27:21.999 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:27:21.998+0000 7fb8910da640 1 -- 172.21.15.114:0/866718964 learned_addr learned my addr 172.21.15.114:0/866718964 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-04-23T10:27:22.000 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:27:21.998+0000 7fb8908d9640 1 --2- 172.21.15.114:0/866718964 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fb88c1010c0 0x7fb88c1986c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T10:27:22.000 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:27:21.998+0000 7fb88bfff640 1 --2- 172.21.15.114:0/866718964 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fb88c106190 0x7fb88c198c00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T10:27:22.000 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:27:21.999+0000 7fb8910da640 1 -- 172.21.15.114:0/866718964 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fb88c106190 msgr2=0x7fb88c198c00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T10:27:22.000 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:27:21.999+0000 7fb8910da640 1 --2- 172.21.15.114:0/866718964 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fb88c106190 0x7fb88c198c00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:27:22.000 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:27:21.999+0000 7fb8910da640 1 -- 172.21.15.114:0/866718964 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fb88c1010c0 msgr2=0x7fb88c1986c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T10:27:22.000 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:27:21.999+0000 7fb8910da640 1 --2- 172.21.15.114:0/866718964 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fb88c1010c0 0x7fb88c1986c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:27:22.000 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:27:21.999+0000 7fb8910da640 1 -- 172.21.15.114:0/866718964 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb88c19da70 con 0x7fb88c19cf80 2024-04-23T10:27:22.000 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:27:21.999+0000 7fb8910da640 1 --2- 172.21.15.114:0/866718964 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fb88c19cf80 0x7fb88c19d430 secure :-1 s=READY pgs=507 cs=0 l=1 rev1=1 crypto rx=0x7fb88400eaa0 tx=0x7fb88400ef60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T10:27:22.000 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:27:21.999+0000 7fb889ffb640 1 -- 172.21.15.114:0/866718964 <== mon.0 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb88400cdb0 con 0x7fb88c19cf80 2024-04-23T10:27:22.000 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:27:21.999+0000 7fb892b3d640 1 -- 172.21.15.114:0/866718964 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fb88c19dd60 con 0x7fb88c19cf80 2024-04-23T10:27:22.000 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:27:21.999+0000 7fb889ffb640 1 -- 172.21.15.114:0/866718964 <== mon.0 v2:172.21.15.114:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fb884005020 con 0x7fb88c19cf80 2024-04-23T10:27:22.000 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:27:21.999+0000 7fb889ffb640 1 -- 172.21.15.114:0/866718964 <== mon.0 v2:172.21.15.114:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb8840116d0 con 0x7fb88c19cf80 2024-04-23T10:27:22.000 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:27:21.999+0000 7fb892b3d640 1 -- 172.21.15.114:0/866718964 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fb88c1a5520 con 0x7fb88c19cf80 2024-04-23T10:27:22.001 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:27:22.001+0000 7fb889ffb640 1 -- 172.21.15.114:0/866718964 <== mon.0 v2:172.21.15.114:3300/0 4 ==== mgrmap(e 25) ==== 138880+0+0 (secure 0 0 0) 0x7fb8840051d0 con 0x7fb88c19cf80 2024-04-23T10:27:22.001 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:27:22.002+0000 7fb889ffb640 1 --2- 172.21.15.114:0/866718964 >> [v2:172.21.15.114:6800/1395693466,v1:172.21.15.114:6801/1395693466] conn(0x7fb87c0a3fe0 0x7fb87c0a64a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T10:27:22.001 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:27:22.002+0000 7fb889ffb640 1 -- 172.21.15.114:0/866718964 <== mon.0 v2:172.21.15.114:3300/0 5 ==== osd_map(90..90 src has 1..90) ==== 8527+0+0 (secure 0 0 0) 0x7fb8840d0b60 con 0x7fb88c19cf80 2024-04-23T10:27:22.001 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:27:22.002+0000 7fb8908d9640 1 --2- 172.21.15.114:0/866718964 >> [v2:172.21.15.114:6800/1395693466,v1:172.21.15.114:6801/1395693466] conn(0x7fb87c0a3fe0 0x7fb87c0a64a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T10:27:22.001 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:27:22.002+0000 7fb8908d9640 1 --2- 172.21.15.114:0/866718964 >> [v2:172.21.15.114:6800/1395693466,v1:172.21.15.114:6801/1395693466] conn(0x7fb87c0a3fe0 0x7fb87c0a64a0 secure :-1 s=READY pgs=66 cs=0 l=1 rev1=1 crypto rx=0x7fb88000bbe0 tx=0x7fb88000b390 comp rx=0 tx=0).ready entity=mgr.15291 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T10:27:22.003 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:27:22.003+0000 7fb89584b740 1 Processor -- start 2024-04-23T10:27:22.003 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:27:22.003+0000 7fb89584b740 1 -- start start 2024-04-23T10:27:22.003 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:27:22.003+0000 7fb89584b740 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x5565c4850620 0x5565c4850a00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T10:27:22.003 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:27:22.003+0000 7fb89584b740 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x5565c4850f40 0x5565c485b2c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T10:27:22.004 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:27:22.003+0000 7fb89584b740 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x5565c485b800 0x5565c485dbf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T10:27:22.004 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:27:22.003+0000 7fb89584b740 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x5565c47b5970 con 0x5565c485b800 2024-04-23T10:27:22.004 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:27:22.003+0000 7fb89584b740 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 -- 0x5565c47524a0 con 0x5565c4850620 2024-04-23T10:27:22.004 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:27:22.003+0000 7fb89584b740 1 -- --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_getmap magic: 0 -- 0x5565c47b1930 con 0x5565c4850f40 2024-04-23T10:27:22.004 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:27:22.004+0000 7fb8910da640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x5565c485b800 0x5565c485dbf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T10:27:22.004 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:27:22.004+0000 7fb8910da640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x5565c485b800 0x5565c485dbf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.114:60708/0 (socket says 172.21.15.114:60708) 2024-04-23T10:27:22.004 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:27:22.004+0000 7fb8910da640 1 -- 172.21.15.114:0/1652335632 learned_addr learned my addr 172.21.15.114:0/1652335632 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-04-23T10:27:22.004 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:27:22.004+0000 7fb88bfff640 1 --2- 172.21.15.114:0/1652335632 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x5565c4850f40 0x5565c485b2c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T10:27:22.004 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:27:22.004+0000 7fb8908d9640 1 --2- 172.21.15.114:0/1652335632 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x5565c4850620 0x5565c4850a00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T10:27:22.004 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:27:22.004+0000 7fb8910da640 1 -- 172.21.15.114:0/1652335632 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x5565c4850f40 msgr2=0x5565c485b2c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T10:27:22.004 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:27:22.004+0000 7fb8910da640 1 --2- 172.21.15.114:0/1652335632 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x5565c4850f40 0x5565c485b2c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:27:22.004 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:27:22.004+0000 7fb8910da640 1 -- 172.21.15.114:0/1652335632 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x5565c4850620 msgr2=0x5565c4850a00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T10:27:22.004 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:27:22.004+0000 7fb8910da640 1 --2- 172.21.15.114:0/1652335632 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x5565c4850620 0x5565c4850a00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:27:22.004 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:27:22.004+0000 7fb8910da640 1 -- 172.21.15.114:0/1652335632 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5565c44c8210 con 0x5565c485b800 2024-04-23T10:27:22.005 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:27:22.004+0000 7fb8908d9640 1 --2- 172.21.15.114:0/1652335632 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x5565c4850620 0x5565c4850a00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-23T10:27:22.005 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:27:22.004+0000 7fb8910da640 1 --2- 172.21.15.114:0/1652335632 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x5565c485b800 0x5565c485dbf0 secure :-1 s=READY pgs=508 cs=0 l=1 rev1=1 crypto rx=0x7fb88409c600 tx=0x7fb8840a0cc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e1123ffb6607a88e server_cookie=0 in_seq=0 out_seq=0 2024-04-23T10:27:22.005 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:27:22.004+0000 7fb876ffd640 1 -- 172.21.15.114:0/1652335632 <== mon.0 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb88401f040 con 0x5565c485b800 2024-04-23T10:27:22.005 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:27:22.004+0000 7fb876ffd640 1 -- 172.21.15.114:0/1652335632 <== mon.0 v2:172.21.15.114:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fb8840054f0 con 0x5565c485b800 2024-04-23T10:27:22.005 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:27:22.004+0000 7fb876ffd640 1 -- 172.21.15.114:0/1652335632 <== mon.0 v2:172.21.15.114:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb88409dde0 con 0x5565c485b800 2024-04-23T10:27:22.008 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:27:22.005+0000 7fb89584b740 1 -- 172.21.15.114:0/1652335632 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x5565c485b800 msgr2=0x5565c485dbf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T10:27:22.008 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:27:22.005+0000 7fb89584b740 1 --2- 172.21.15.114:0/1652335632 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x5565c485b800 0x5565c485dbf0 secure :-1 s=READY pgs=508 cs=0 l=1 rev1=1 crypto rx=0x7fb88409c600 tx=0x7fb8840a0cc0 comp rx=0 tx=0).stop 2024-04-23T10:27:22.008 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:27:22.005+0000 7fb89584b740 1 -- 172.21.15.114:0/1652335632 shutdown_connections 2024-04-23T10:27:22.008 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:27:22.005+0000 7fb89584b740 1 --2- 172.21.15.114:0/1652335632 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x5565c485b800 0x5565c485dbf0 unknown :-1 s=CLOSED pgs=508 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:27:22.008 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:27:22.005+0000 7fb89584b740 1 --2- 172.21.15.114:0/1652335632 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x5565c4850f40 0x5565c485b2c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:27:22.008 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:27:22.005+0000 7fb89584b740 1 --2- 172.21.15.114:0/1652335632 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x5565c4850620 0x5565c4850a00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:27:22.008 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:27:22.005+0000 7fb89584b740 1 -- 172.21.15.114:0/1652335632 >> 172.21.15.114:0/1652335632 conn(0x5565c484cb30 msgr2=0x5565c484cf10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T10:27:22.008 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:27:22.005+0000 7fb89584b740 1 -- 172.21.15.114:0/1652335632 shutdown_connections 2024-04-23T10:27:22.008 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:27:22.005+0000 7fb89584b740 1 -- 172.21.15.114:0/1652335632 wait complete. 2024-04-23T10:27:22.008 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:27:22.006+0000 7fb89584b740 1 -- start start 2024-04-23T10:27:22.008 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:27:22.006+0000 7fb89584b740 1 Processor -- start 2024-04-23T10:27:22.008 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:27:22.006+0000 7fb89584b740 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x5565c4850620 0x5565c485b170 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T10:27:22.009 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:27:22.006+0000 7fb89584b740 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x5565c4850f40 0x5565c4858460 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T10:27:22.009 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:27:22.006+0000 7fb89584b740 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x5565c485b800 0x5565c48589c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T10:27:22.009 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:27:22.006+0000 7fb89584b740 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x5565c485e430 con 0x5565c485b800 2024-04-23T10:27:22.009 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:27:22.006+0000 7fb89584b740 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 -- 0x5565c47500b0 con 0x5565c4850620 2024-04-23T10:27:22.009 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:27:22.006+0000 7fb89584b740 1 -- --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_getmap magic: 0 -- 0x5565c485e5b0 con 0x5565c4850f40 2024-04-23T10:27:22.009 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:27:22.007+0000 7fb8910da640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x5565c485b800 0x5565c48589c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T10:27:22.009 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:27:22.007+0000 7fb89584b740 10 client.0 _ll_register_callbacks cb 0x5565c473dd30 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-23T10:27:22.009 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:27:22.007+0000 7fb8910da640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x5565c485b800 0x5565c48589c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.114:60716/0 (socket says 172.21.15.114:60716) 2024-04-23T10:27:22.009 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:27:22.007+0000 7fb8910da640 1 -- 172.21.15.114:0/1874315575 learned_addr learned my addr 172.21.15.114:0/1874315575 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-04-23T10:27:22.009 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:27:22.007+0000 7fb88bfff640 1 --2- 172.21.15.114:0/1874315575 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x5565c4850f40 0x5565c4858460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T10:27:22.009 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:27:22.007+0000 7fb8910da640 1 -- 172.21.15.114:0/1874315575 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x5565c4850f40 msgr2=0x5565c4858460 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T10:27:22.009 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:27:22.007+0000 7fb8910da640 1 --2- 172.21.15.114:0/1874315575 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x5565c4850f40 0x5565c4858460 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:27:22.009 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:27:22.007+0000 7fb8910da640 1 -- 172.21.15.114:0/1874315575 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x5565c4850620 msgr2=0x5565c485b170 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2024-04-23T10:27:22.009 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:27:22.007+0000 7fb8910da640 1 --2- 172.21.15.114:0/1874315575 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x5565c4850620 0x5565c485b170 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:27:22.009 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:27:22.007+0000 7fb8910da640 1 -- 172.21.15.114:0/1874315575 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5565c4594620 con 0x5565c485b800 2024-04-23T10:27:22.010 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:27:22.007+0000 7fb88bfff640 1 --2- 172.21.15.114:0/1874315575 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x5565c4850f40 0x5565c4858460 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-23T10:27:22.010 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:27:22.007+0000 7fb8910da640 1 --2- 172.21.15.114:0/1874315575 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x5565c485b800 0x5565c48589c0 secure :-1 s=READY pgs=509 cs=0 l=1 rev1=1 crypto rx=0x7fb88400efd0 tx=0x7fb884087280 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e00d78eb28c8ece3 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T10:27:22.010 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:27:22.007+0000 7fb86bfff640 10 client.0 ms_handle_connect on v2:172.21.15.114:3300/0 2024-04-23T10:27:22.010 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:27:22.007+0000 7fb86bfff640 1 -- 172.21.15.114:0/1874315575 <== mon.0 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb8840b9020 con 0x5565c485b800 2024-04-23T10:27:22.010 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:27:22.007+0000 7fb89584b740 1 -- 172.21.15.114:0/1874315575 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fb8840054f0 con 0x5565c485b800 2024-04-23T10:27:22.010 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:27:22.007+0000 7fb86bfff640 1 -- 172.21.15.114:0/1874315575 <== mon.0 v2:172.21.15.114:3300/0 2 ==== osd_map(90..90 src has 1..90) ==== 8527+0+0 (secure 0 0 0) 0x7fb8840ce3b0 con 0x5565c485b800 2024-04-23T10:27:22.010 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:27:22.007+0000 7fb86bfff640 1 -- 172.21.15.114:0/1874315575 <== mon.0 v2:172.21.15.114:3300/0 3 ==== mon_get_version_reply(handle=1 version=21) ==== 24+0+0 (secure 0 0 0) 0x7fb88401f070 con 0x5565c485b800 2024-04-23T10:27:22.010 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:27:22.007+0000 7fb89584b740 10 client.15630 fetch_fsmap learned FSMap version 21 2024-04-23T10:27:22.010 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:27:22.007+0000 7fb89584b740 1 -- 172.21.15.114:0/1874315575 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({config=0+,fsmap=21,monmap=4+}) -- 0x5565c486a2d0 con 0x5565c485b800 2024-04-23T10:27:22.010 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:27:22.008+0000 7fb86bfff640 1 -- 172.21.15.114:0/1874315575 <== mon.0 v2:172.21.15.114:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fb8840054f0 con 0x5565c485b800 2024-04-23T10:27:22.010 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:27:22.008+0000 7fb86bfff640 1 -- 172.21.15.114:0/1874315575 <== mon.0 v2:172.21.15.114:3300/0 5 ==== fsmap(e 21) ==== 8338+0+0 (secure 0 0 0) 0x7fb8840b2030 con 0x5565c485b800 2024-04-23T10:27:22.010 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:27:22.008+0000 7fb89584b740 10 client.15630 fetch_fsmap finished waiting for FSMap version 21 2024-04-23T10:27:22.010 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:27:22.008+0000 7fb89584b740 10 client.15630 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-04-23T10:27:22.010 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:27:22.008+0000 7fb89584b740 20 client.15630 populate_metadata read hostname 'smithi114' 2024-04-23T10:27:22.010 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:27:22.008+0000 7fb89584b740 1 --2- 172.21.15.114:0/1874315575 >> [v2:172.21.15.119:6838/937297340,v1:172.21.15.119:6839/937297340] conn(0x5565c486b620 0x5565c486da10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T10:27:22.010 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:27:22.008+0000 7fb89584b740 4 client.15630 mds_command: new command op to 24545 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-23T10:27:22.011 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:27:22.008+0000 7fb89584b740 1 -- 172.21.15.114:0/1874315575 --> [v2:172.21.15.119:6838/937297340,v1:172.21.15.119:6839/937297340] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x5565c47c3390 con 0x5565c486b620 2024-04-23T10:27:22.011 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:27:22.008+0000 7fb8908d9640 1 --2- 172.21.15.114:0/1874315575 >> [v2:172.21.15.119:6838/937297340,v1:172.21.15.119:6839/937297340] conn(0x5565c486b620 0x5565c486da10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T10:27:22.011 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:27:22.009+0000 7fb86bfff640 10 client.15630 ms_handle_connect on v2:172.21.15.119:6838/937297340 2024-04-23T10:27:22.011 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:27:22.009+0000 7fb8908d9640 1 --2- 172.21.15.114:0/1874315575 >> [v2:172.21.15.119:6838/937297340,v1:172.21.15.119:6839/937297340] conn(0x5565c486b620 0x5565c486da10 crc :-1 s=READY pgs=46 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=c28de9a7db9998f8 server_cookie=ca757699a1a83508 in_seq=0 out_seq=0 2024-04-23T10:27:22.012 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:27:22.010+0000 7fb86bfff640 1 -- 172.21.15.114:0/1874315575 <== mds.0 v2:172.21.15.119:6838/937297340 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x5565c47c3390 con 0x5565c486b620 2024-04-23T10:27:22.012 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:27:22.010+0000 7fb86bfff640 10 client.15630 handle_command_reply: tid=0 2024-04-23T10:27:22.012 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:27:22.010+0000 7fb89584b740 1 client.15630 shutdown 2024-04-23T10:27:22.012 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:27:22.010+0000 7fb89584b740 1 -- 172.21.15.114:0/1874315575 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x5565c485b800 msgr2=0x5565c48589c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T10:27:22.012 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:27:22.010+0000 7fb89584b740 1 --2- 172.21.15.114:0/1874315575 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x5565c485b800 0x5565c48589c0 secure :-1 s=READY pgs=509 cs=0 l=1 rev1=1 crypto rx=0x7fb88400efd0 tx=0x7fb884087280 comp rx=0 tx=0).stop 2024-04-23T10:27:22.012 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:27:22.010+0000 7fb89584b740 1 -- 172.21.15.114:0/1874315575 shutdown_connections 2024-04-23T10:27:22.012 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:27:22.010+0000 7fb89584b740 1 --2- 172.21.15.114:0/1874315575 >> [v2:172.21.15.119:6838/937297340,v1:172.21.15.119:6839/937297340] conn(0x5565c486b620 0x5565c486da10 crc :-1 s=READY pgs=46 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:27:22.012 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:27:22.010+0000 7fb86bfff640 0 client.15630 ms_handle_reset on v2:172.21.15.119:6838/937297340 2024-04-23T10:27:22.012 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:27:22.010+0000 7fb89584b740 1 --2- 172.21.15.114:0/1874315575 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x5565c485b800 0x5565c48589c0 unknown :-1 s=CLOSED pgs=509 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:27:22.012 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:27:22.010+0000 7fb89584b740 1 --2- 172.21.15.114:0/1874315575 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x5565c4850f40 0x5565c4858460 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:27:22.012 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:27:22.010+0000 7fb89584b740 1 --2- 172.21.15.114:0/1874315575 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x5565c4850620 0x5565c485b170 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:27:22.012 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:27:22.010+0000 7fb89584b740 1 -- 172.21.15.114:0/1874315575 >> 172.21.15.114:0/1874315575 conn(0x5565c484cb30 msgr2=0x5565c4852860 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T10:27:22.012 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:27:22.010+0000 7fb89584b740 1 -- 172.21.15.114:0/1874315575 shutdown_connections 2024-04-23T10:27:22.012 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:27:22.010+0000 7fb89584b740 1 -- 172.21.15.114:0/1874315575 wait complete. 2024-04-23T10:27:22.013 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:27:22.010+0000 7fb89584b740 20 client.15630 trim_cache size 0 max 16384 2024-04-23T10:27:22.039 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:27:22.038+0000 7fb89584b740 1 Processor -- start 2024-04-23T10:27:22.040 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:27:22.039+0000 7fb89584b740 1 -- start start 2024-04-23T10:27:22.040 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:27:22.039+0000 7fb89584b740 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x5565c4850620 0x5565c4866f10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T10:27:22.040 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:27:22.039+0000 7fb89584b740 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x5565c4850f40 0x5565c4870a40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T10:27:22.040 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:27:22.039+0000 7fb89584b740 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x5565c485b800 0x5565c4870f80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T10:27:22.040 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:27:22.039+0000 7fb89584b740 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x7fb88c1b2650 con 0x5565c485b800 2024-04-23T10:27:22.040 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:27:22.039+0000 7fb89584b740 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 -- 0x7fb88c1aff20 con 0x5565c4850f40 2024-04-23T10:27:22.040 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:27:22.039+0000 7fb89584b740 1 -- --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_getmap magic: 0 -- 0x7fb88c1add90 con 0x5565c4850620 2024-04-23T10:27:22.040 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:27:22.039+0000 7fb8910da640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x5565c485b800 0x5565c4870f80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T10:27:22.040 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:27:22.039+0000 7fb8910da640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x5565c485b800 0x5565c4870f80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.114:60722/0 (socket says 172.21.15.114:60722) 2024-04-23T10:27:22.040 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:27:22.039+0000 7fb8910da640 1 -- 172.21.15.114:0/1355889870 learned_addr learned my addr 172.21.15.114:0/1355889870 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-04-23T10:27:22.040 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:27:22.039+0000 7fb8908d9640 1 --2- 172.21.15.114:0/1355889870 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x5565c4850620 0x5565c4866f10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T10:27:22.040 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:27:22.039+0000 7fb88bfff640 1 --2- 172.21.15.114:0/1355889870 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x5565c4850f40 0x5565c4870a40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T10:27:22.040 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:27:22.039+0000 7fb8910da640 1 -- 172.21.15.114:0/1355889870 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x5565c4850620 msgr2=0x5565c4866f10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T10:27:22.040 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:27:22.039+0000 7fb8910da640 1 --2- 172.21.15.114:0/1355889870 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x5565c4850620 0x5565c4866f10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:27:22.041 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:27:22.039+0000 7fb8910da640 1 -- 172.21.15.114:0/1355889870 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x5565c4850f40 msgr2=0x5565c4870a40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T10:27:22.041 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:27:22.039+0000 7fb8910da640 1 --2- 172.21.15.114:0/1355889870 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x5565c4850f40 0x5565c4870a40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:27:22.041 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:27:22.039+0000 7fb8910da640 1 -- 172.21.15.114:0/1355889870 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5565c4858d30 con 0x5565c485b800 2024-04-23T10:27:22.041 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:27:22.040+0000 7fb8910da640 1 --2- 172.21.15.114:0/1355889870 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x5565c485b800 0x5565c4870f80 secure :-1 s=READY pgs=510 cs=0 l=1 rev1=1 crypto rx=0x7fb884098a90 tx=0x7fb8840a0cc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=58dae9f6b5026824 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T10:27:22.041 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:27:22.040+0000 7fb8767fc640 1 -- 172.21.15.114:0/1355889870 <== mon.0 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb8840b9070 con 0x5565c485b800 2024-04-23T10:27:22.041 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:27:22.040+0000 7fb8767fc640 1 -- 172.21.15.114:0/1355889870 <== mon.0 v2:172.21.15.114:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fb884098650 con 0x5565c485b800 2024-04-23T10:27:22.041 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:27:22.040+0000 7fb8767fc640 1 -- 172.21.15.114:0/1355889870 <== mon.0 v2:172.21.15.114:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb8840a1e10 con 0x5565c485b800 2024-04-23T10:27:22.041 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:27:22.041+0000 7fb89584b740 1 -- 172.21.15.114:0/1355889870 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x5565c485b800 msgr2=0x5565c4870f80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T10:27:22.041 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:27:22.041+0000 7fb89584b740 1 --2- 172.21.15.114:0/1355889870 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x5565c485b800 0x5565c4870f80 secure :-1 s=READY pgs=510 cs=0 l=1 rev1=1 crypto rx=0x7fb884098a90 tx=0x7fb8840a0cc0 comp rx=0 tx=0).stop 2024-04-23T10:27:22.041 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:27:22.041+0000 7fb89584b740 1 -- 172.21.15.114:0/1355889870 shutdown_connections 2024-04-23T10:27:22.041 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:27:22.041+0000 7fb89584b740 1 --2- 172.21.15.114:0/1355889870 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x5565c485b800 0x5565c4870f80 unknown :-1 s=CLOSED pgs=510 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:27:22.041 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:27:22.041+0000 7fb89584b740 1 --2- 172.21.15.114:0/1355889870 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x5565c4850f40 0x5565c4870a40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:27:22.041 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:27:22.041+0000 7fb89584b740 1 --2- 172.21.15.114:0/1355889870 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x5565c4850620 0x5565c4866f10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:27:22.041 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:27:22.041+0000 7fb89584b740 1 -- 172.21.15.114:0/1355889870 >> 172.21.15.114:0/1355889870 conn(0x5565c484cb30 msgr2=0x5565c4866800 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T10:27:22.044 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:27:22.041+0000 7fb89584b740 1 -- 172.21.15.114:0/1355889870 shutdown_connections 2024-04-23T10:27:22.044 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:27:22.041+0000 7fb89584b740 1 -- 172.21.15.114:0/1355889870 wait complete. 2024-04-23T10:27:22.044 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:27:22.042+0000 7fb89584b740 1 -- start start 2024-04-23T10:27:22.044 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:27:22.042+0000 7fb89584b740 1 Processor -- start 2024-04-23T10:27:22.045 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:27:22.042+0000 7fb89584b740 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x5565c4850620 0x5565c48c9ed0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T10:27:22.045 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:27:22.042+0000 7fb89584b740 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x5565c4850f40 0x5565c48ca410 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T10:27:22.045 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:27:22.042+0000 7fb89584b740 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x5565c485b800 0x5565c4870000 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T10:27:22.045 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:27:22.042+0000 7fb89584b740 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x7fb88c1ae3e0 con 0x5565c485b800 2024-04-23T10:27:22.045 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:27:22.042+0000 7fb89584b740 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 -- 0x7fb88c1af850 con 0x5565c4850f40 2024-04-23T10:27:22.045 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:27:22.042+0000 7fb89584b740 1 -- --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_getmap magic: 0 -- 0x7fb88c1aeab0 con 0x5565c4850620 2024-04-23T10:27:22.045 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:27:22.042+0000 7fb89584b740 10 client.0 _ll_register_callbacks cb 0x7fb8840873a0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-23T10:27:22.045 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:27:22.042+0000 7fb8910da640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x5565c485b800 0x5565c4870000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T10:27:22.045 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:27:22.042+0000 7fb8908d9640 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x5565c4850620 0x5565c48c9ed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T10:27:22.045 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:27:22.043+0000 7fb88bfff640 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x5565c4850f40 0x5565c48ca410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T10:27:22.045 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:27:22.043+0000 7fb8908d9640 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x5565c4850620 0x5565c48c9ed0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.204:3300/0 says I am v2:172.21.15.114:34032/0 (socket says 172.21.15.114:34032) 2024-04-23T10:27:22.045 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:27:22.043+0000 7fb88bfff640 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x5565c4850f40 0x5565c48ca410 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.119:3300/0 says I am v2:172.21.15.114:51982/0 (socket says 172.21.15.114:51982) 2024-04-23T10:27:22.045 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:27:22.043+0000 7fb8908d9640 1 -- 172.21.15.114:0/3600779362 learned_addr learned my addr 172.21.15.114:0/3600779362 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-04-23T10:27:22.045 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:27:22.043+0000 7fb88bfff640 1 -- 172.21.15.114:0/3600779362 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x5565c4850620 msgr2=0x5565c48c9ed0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T10:27:22.046 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:27:22.043+0000 7fb88bfff640 1 --2- 172.21.15.114:0/3600779362 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x5565c4850620 0x5565c48c9ed0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:27:22.046 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:27:22.043+0000 7fb88bfff640 1 -- 172.21.15.114:0/3600779362 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x5565c485b800 msgr2=0x5565c4870000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T10:27:22.046 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:27:22.043+0000 7fb88bfff640 1 --2- 172.21.15.114:0/3600779362 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x5565c485b800 0x5565c4870000 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:27:22.046 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:27:22.043+0000 7fb88bfff640 1 -- 172.21.15.114:0/3600779362 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5565c4850d70 con 0x5565c4850f40 2024-04-23T10:27:22.046 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:27:22.043+0000 7fb8910da640 1 --2- 172.21.15.114:0/3600779362 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x5565c485b800 0x5565c4870000 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-23T10:27:22.046 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:27:22.043+0000 7fb88bfff640 1 --2- 172.21.15.114:0/3600779362 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x5565c4850f40 0x5565c48ca410 secure :-1 s=READY pgs=58 cs=0 l=1 rev1=1 crypto rx=0x7fb878006010 tx=0x7fb8780316d0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=a76234dea373dcb3 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T10:27:22.046 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:27:22.043+0000 7fb86bfff640 10 client.0 ms_handle_connect on v2:172.21.15.119:3300/0 2024-04-23T10:27:22.046 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:27:22.043+0000 7fb86bfff640 1 -- 172.21.15.114:0/3600779362 <== mon.2 v2:172.21.15.119:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb878035040 con 0x5565c4850f40 2024-04-23T10:27:22.046 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:27:22.043+0000 7fb89584b740 1 -- 172.21.15.114:0/3600779362 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fb884098650 con 0x5565c4850f40 2024-04-23T10:27:22.046 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:27:22.043+0000 7fb86bfff640 1 -- 172.21.15.114:0/3600779362 <== mon.2 v2:172.21.15.119:3300/0 2 ==== osd_map(90..90 src has 1..90) ==== 8527+0+0 (secure 0 0 0) 0x7fb878031b10 con 0x5565c4850f40 2024-04-23T10:27:22.046 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:27:22.043+0000 7fb86bfff640 1 -- 172.21.15.114:0/3600779362 <== mon.2 v2:172.21.15.119:3300/0 3 ==== mon_get_version_reply(handle=1 version=21) ==== 24+0+0 (secure 0 0 0) 0x7fb878031d10 con 0x5565c4850f40 2024-04-23T10:27:22.046 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:27:22.045+0000 7fb89584b740 10 client.25172 fetch_fsmap learned FSMap version 21 2024-04-23T10:27:22.046 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:27:22.045+0000 7fb89584b740 1 -- 172.21.15.114:0/3600779362 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({config=0+,fsmap=21,monmap=4+}) -- 0x5565c48d1570 con 0x5565c4850f40 2024-04-23T10:27:22.046 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:27:22.045+0000 7fb86bfff640 1 -- 172.21.15.114:0/3600779362 <== mon.2 v2:172.21.15.119:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fb884098650 con 0x5565c4850f40 2024-04-23T10:27:22.047 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:27:22.045+0000 7fb86bfff640 1 -- 172.21.15.114:0/3600779362 <== mon.2 v2:172.21.15.119:3300/0 5 ==== fsmap(e 21) ==== 8338+0+0 (secure 0 0 0) 0x7fb878004750 con 0x5565c4850f40 2024-04-23T10:27:22.047 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:27:22.045+0000 7fb89584b740 10 client.25172 fetch_fsmap finished waiting for FSMap version 21 2024-04-23T10:27:22.047 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:27:22.045+0000 7fb89584b740 10 client.25172 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-04-23T10:27:22.047 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:27:22.045+0000 7fb89584b740 20 client.25172 populate_metadata read hostname 'smithi114' 2024-04-23T10:27:22.047 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:27:22.045+0000 7fb89584b740 1 --2- 172.21.15.114:0/3600779362 >> [v2:172.21.15.119:6838/937297340,v1:172.21.15.119:6839/937297340] conn(0x5565c48d1cd0 0x5565c48d40f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T10:27:22.047 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:27:22.045+0000 7fb89584b740 4 client.25172 mds_command: new command op to 24545 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-04-23T10:27:22.047 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:27:22.045+0000 7fb89584b740 1 -- 172.21.15.114:0/3600779362 --> [v2:172.21.15.119:6838/937297340,v1:172.21.15.119:6839/937297340] -- command(tid 0: {"prefix": "flush journal"}) -- 0x5565c47bd200 con 0x5565c48d1cd0 2024-04-23T10:27:22.047 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:27:22.045+0000 7fb8908d9640 1 --2- 172.21.15.114:0/3600779362 >> [v2:172.21.15.119:6838/937297340,v1:172.21.15.119:6839/937297340] conn(0x5565c48d1cd0 0x5565c48d40f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T10:27:22.047 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:27:22.046+0000 7fb86bfff640 10 client.25172 ms_handle_connect on v2:172.21.15.119:6838/937297340 2024-04-23T10:27:22.047 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:27:22.046+0000 7fb8908d9640 1 --2- 172.21.15.114:0/3600779362 >> [v2:172.21.15.119:6838/937297340,v1:172.21.15.119:6839/937297340] conn(0x5565c48d1cd0 0x5565c48d40f0 crc :-1 s=READY pgs=47 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=3ef14de42c9d356a server_cookie=9d8c28a24351539f in_seq=0 out_seq=0 2024-04-23T10:27:22.049 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:27:22.050+0000 7fb86bfff640 1 -- 172.21.15.114:0/3600779362 <== mds.0 v2:172.21.15.119:6838/937297340 1 ==== command_reply(tid 0: 0 ) ==== 8+0+44 (crc 0 0 0) 0x5565c47bd200 con 0x5565c48d1cd0 2024-04-23T10:27:22.049 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:27:22.050+0000 7fb86bfff640 10 client.25172 handle_command_reply: tid=0 2024-04-23T10:27:22.049 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:27:22.050+0000 7fb89584b740 1 client.25172 shutdown 2024-04-23T10:27:22.049 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:27:22.050+0000 7fb89584b740 1 -- 172.21.15.114:0/3600779362 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x5565c4850f40 msgr2=0x5565c48ca410 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T10:27:22.049 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:27:22.050+0000 7fb89584b740 1 --2- 172.21.15.114:0/3600779362 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x5565c4850f40 0x5565c48ca410 secure :-1 s=READY pgs=58 cs=0 l=1 rev1=1 crypto rx=0x7fb878006010 tx=0x7fb8780316d0 comp rx=0 tx=0).stop 2024-04-23T10:27:22.049 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:27:22.050+0000 7fb89584b740 1 -- 172.21.15.114:0/3600779362 shutdown_connections 2024-04-23T10:27:22.049 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:27:22.050+0000 7fb89584b740 1 --2- 172.21.15.114:0/3600779362 >> [v2:172.21.15.119:6838/937297340,v1:172.21.15.119:6839/937297340] conn(0x5565c48d1cd0 0x5565c48d40f0 crc :-1 s=READY pgs=47 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:27:22.049 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:27:22.050+0000 7fb89584b740 1 --2- 172.21.15.114:0/3600779362 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x5565c485b800 0x5565c4870000 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:27:22.050 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:27:22.050+0000 7fb89584b740 1 --2- 172.21.15.114:0/3600779362 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x5565c4850f40 0x5565c48ca410 unknown :-1 s=CLOSED pgs=58 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:27:22.050 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:27:22.050+0000 7fb86bfff640 0 client.25172 ms_handle_reset on v2:172.21.15.119:6838/937297340 2024-04-23T10:27:22.050 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:27:22.050+0000 7fb89584b740 1 --2- 172.21.15.114:0/3600779362 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x5565c4850620 0x5565c48c9ed0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:27:22.050 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:27:22.050+0000 7fb89584b740 1 -- 172.21.15.114:0/3600779362 >> 172.21.15.114:0/3600779362 conn(0x5565c484cb30 msgr2=0x5565c48568f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T10:27:22.050 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:27:22.050+0000 7fb89584b740 1 -- 172.21.15.114:0/3600779362 shutdown_connections 2024-04-23T10:27:22.050 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:27:22.050+0000 7fb89584b740 1 -- 172.21.15.114:0/3600779362 wait complete. 2024-04-23T10:27:22.050 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:27:22.050+0000 7fb89584b740 20 client.25172 trim_cache size 0 max 16384 2024-04-23T10:27:22.050 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stdout:{ 2024-04-23T10:27:22.050 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stdout: "message": "", 2024-04-23T10:27:22.050 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stdout: "return_code": 0 2024-04-23T10:27:22.050 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stdout:} 2024-04-23T10:27:22.051 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:27:22.051+0000 7fb892b3d640 1 -- 172.21.15.114:0/866718964 >> [v2:172.21.15.114:6800/1395693466,v1:172.21.15.114:6801/1395693466] conn(0x7fb87c0a3fe0 msgr2=0x7fb87c0a64a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T10:27:22.051 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:27:22.051+0000 7fb892b3d640 1 --2- 172.21.15.114:0/866718964 >> [v2:172.21.15.114:6800/1395693466,v1:172.21.15.114:6801/1395693466] conn(0x7fb87c0a3fe0 0x7fb87c0a64a0 secure :-1 s=READY pgs=66 cs=0 l=1 rev1=1 crypto rx=0x7fb88000bbe0 tx=0x7fb88000b390 comp rx=0 tx=0).stop 2024-04-23T10:27:22.051 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:27:22.051+0000 7fb892b3d640 1 -- 172.21.15.114:0/866718964 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fb88c19cf80 msgr2=0x7fb88c19d430 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T10:27:22.051 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:27:22.051+0000 7fb892b3d640 1 --2- 172.21.15.114:0/866718964 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fb88c19cf80 0x7fb88c19d430 secure :-1 s=READY pgs=507 cs=0 l=1 rev1=1 crypto rx=0x7fb88400eaa0 tx=0x7fb88400ef60 comp rx=0 tx=0).stop 2024-04-23T10:27:22.051 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:27:22.051+0000 7fb892b3d640 1 -- 172.21.15.114:0/866718964 shutdown_connections 2024-04-23T10:27:22.051 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:27:22.051+0000 7fb892b3d640 1 --2- 172.21.15.114:0/866718964 >> [v2:172.21.15.114:6800/1395693466,v1:172.21.15.114:6801/1395693466] conn(0x7fb87c0a3fe0 0x7fb87c0a64a0 unknown :-1 s=CLOSED pgs=66 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:27:22.051 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:27:22.051+0000 7fb892b3d640 1 --2- 172.21.15.114:0/866718964 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fb88c19cf80 0x7fb88c19d430 unknown :-1 s=CLOSED pgs=507 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:27:22.052 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:27:22.051+0000 7fb892b3d640 1 --2- 172.21.15.114:0/866718964 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fb88c106190 0x7fb88c198c00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:27:22.052 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:27:22.051+0000 7fb892b3d640 1 --2- 172.21.15.114:0/866718964 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fb88c1010c0 0x7fb88c1986c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:27:22.052 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:27:22.051+0000 7fb892b3d640 1 -- 172.21.15.114:0/866718964 >> 172.21.15.114:0/866718964 conn(0x7fb88c077f60 msgr2=0x7fb88c0ff090 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T10:27:22.052 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:27:22.051+0000 7fb892b3d640 1 -- 172.21.15.114:0/866718964 shutdown_connections 2024-04-23T10:27:22.052 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:27:22.051+0000 7fb892b3d640 1 -- 172.21.15.114:0/866718964 wait complete. 2024-04-23T10:27:22.492 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 23 10:27:22 smithi119 ceph-mon[32608]: pgmap v23: 97 pgs: 97 active+clean; 1.3 MiB data, 377 MiB used, 1.0 TiB / 1.0 TiB avail; 2.6 KiB/s rd, 1.4 KiB/s wr, 7 op/s 2024-04-23T10:27:22.504 INFO:journalctl@ceph.mon.a.smithi114.stdout:Apr 23 10:27:22 smithi114 ceph-mon[29255]: pgmap v23: 97 pgs: 97 active+clean; 1.3 MiB data, 377 MiB used, 1.0 TiB / 1.0 TiB avail; 2.6 KiB/s rd, 1.4 KiB/s wr, 7 op/s 2024-04-23T10:27:22.578 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 23 10:27:22 smithi204 ceph-mon[32649]: pgmap v23: 97 pgs: 97 active+clean; 1.3 MiB data, 377 MiB used, 1.0 TiB / 1.0 TiB avail; 2.6 KiB/s rd, 1.4 KiB/s wr, 7 op/s 2024-04-23T10:27:24.492 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 23 10:27:24 smithi119 ceph-mon[32608]: pgmap v24: 97 pgs: 97 active+clean; 1.3 MiB data, 377 MiB used, 1.0 TiB / 1.0 TiB avail; 3.8 KiB/s rd, 1.4 KiB/s wr, 9 op/s 2024-04-23T10:27:24.504 INFO:journalctl@ceph.mon.a.smithi114.stdout:Apr 23 10:27:24 smithi114 ceph-mon[29255]: pgmap v24: 97 pgs: 97 active+clean; 1.3 MiB data, 377 MiB used, 1.0 TiB / 1.0 TiB avail; 3.8 KiB/s rd, 1.4 KiB/s wr, 9 op/s 2024-04-23T10:27:24.578 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 23 10:27:24 smithi204 ceph-mon[32649]: pgmap v24: 97 pgs: 97 active+clean; 1.3 MiB data, 377 MiB used, 1.0 TiB / 1.0 TiB avail; 3.8 KiB/s rd, 1.4 KiB/s wr, 9 op/s 2024-04-23T10:27:26.492 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 23 10:27:26 smithi119 ceph-mon[32608]: pgmap v25: 97 pgs: 97 active+clean; 1.3 MiB data, 377 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 85 B/s wr, 5 op/s 2024-04-23T10:27:26.504 INFO:journalctl@ceph.mon.a.smithi114.stdout:Apr 23 10:27:26 smithi114 ceph-mon[29255]: pgmap v25: 97 pgs: 97 active+clean; 1.3 MiB data, 377 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 85 B/s wr, 5 op/s 2024-04-23T10:27:26.577 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 23 10:27:26 smithi204 ceph-mon[32649]: pgmap v25: 97 pgs: 97 active+clean; 1.3 MiB data, 377 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 85 B/s wr, 5 op/s 2024-04-23T10:27:27.467 DEBUG:teuthology.orchestra.run.smithi114:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-23T10:27:27.571 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:27:27.569+0000 7f7292508640 1 -- 172.21.15.114:0/3310037207 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f728c1031e0 msgr2=0x7f728c1035c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T10:27:27.572 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:27:27.569+0000 7f7292508640 1 --2- 172.21.15.114:0/3310037207 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f728c1031e0 0x7f728c1035c0 secure :-1 s=READY pgs=511 cs=0 l=1 rev1=1 crypto rx=0x7f727c009900 tx=0x7f727c02a510 comp rx=0 tx=0).stop 2024-04-23T10:27:27.572 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:27:27.569+0000 7f7292508640 1 -- 172.21.15.114:0/3310037207 shutdown_connections 2024-04-23T10:27:27.572 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:27:27.569+0000 7f7292508640 1 --2- 172.21.15.114:0/3310037207 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f728c10c520 0x7f728c10e9c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:27:27.572 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:27:27.569+0000 7f7292508640 1 --2- 172.21.15.114:0/3310037207 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f728c103b00 0x7f728c10bfe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:27:27.572 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:27:27.569+0000 7f7292508640 1 --2- 172.21.15.114:0/3310037207 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f728c1031e0 0x7f728c1035c0 unknown :-1 s=CLOSED pgs=511 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:27:27.572 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:27:27.569+0000 7f7292508640 1 -- 172.21.15.114:0/3310037207 >> 172.21.15.114:0/3310037207 conn(0x7f728c077f50 msgr2=0x7f728c078360 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T10:27:27.572 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:27:27.570+0000 7f7292508640 1 -- 172.21.15.114:0/3310037207 shutdown_connections 2024-04-23T10:27:27.572 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:27:27.570+0000 7f7292508640 1 -- 172.21.15.114:0/3310037207 wait complete. 2024-04-23T10:27:27.572 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:27:27.570+0000 7f7292508640 1 Processor -- start 2024-04-23T10:27:27.573 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:27:27.570+0000 7f7292508640 1 -- start start 2024-04-23T10:27:27.573 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:27:27.571+0000 7f7292508640 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f728c1031e0 0x7f728c198750 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T10:27:27.573 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:27:27.571+0000 7f7292508640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f728c103b00 0x7f728c198c90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T10:27:27.573 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:27:27.571+0000 7f7292508640 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f728c10c520 0x7f728c19d010 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T10:27:27.573 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:27:27.571+0000 7f7292508640 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x7f728c112190 con 0x7f728c103b00 2024-04-23T10:27:27.573 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:27:27.571+0000 7f7292508640 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 -- 0x7f728c112010 con 0x7f728c1031e0 2024-04-23T10:27:27.573 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:27:27.571+0000 7f7292508640 1 -- --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_getmap magic: 0 -- 0x7f728c112310 con 0x7f728c10c520 2024-04-23T10:27:27.573 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:27:27.571+0000 7f728b7fe640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f728c103b00 0x7f728c198c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T10:27:27.573 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:27:27.571+0000 7f728b7fe640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f728c103b00 0x7f728c198c90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.114:53484/0 (socket says 172.21.15.114:53484) 2024-04-23T10:27:27.573 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:27:27.571+0000 7f728b7fe640 1 -- 172.21.15.114:0/2063998827 learned_addr learned my addr 172.21.15.114:0/2063998827 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-04-23T10:27:27.573 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:27:27.571+0000 7f728bfff640 1 --2- 172.21.15.114:0/2063998827 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f728c1031e0 0x7f728c198750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T10:27:27.573 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:27:27.571+0000 7f7290aa5640 1 --2- 172.21.15.114:0/2063998827 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f728c10c520 0x7f728c19d010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T10:27:27.574 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:27:27.571+0000 7f728b7fe640 1 -- 172.21.15.114:0/2063998827 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f728c10c520 msgr2=0x7f728c19d010 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T10:27:27.574 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:27:27.571+0000 7f728b7fe640 1 --2- 172.21.15.114:0/2063998827 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f728c10c520 0x7f728c19d010 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:27:27.574 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:27:27.571+0000 7f728b7fe640 1 -- 172.21.15.114:0/2063998827 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f728c1031e0 msgr2=0x7f728c198750 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T10:27:27.574 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:27:27.571+0000 7f728b7fe640 1 --2- 172.21.15.114:0/2063998827 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f728c1031e0 0x7f728c198750 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:27:27.574 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:27:27.571+0000 7f728b7fe640 1 -- 172.21.15.114:0/2063998827 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f728c19d740 con 0x7f728c103b00 2024-04-23T10:27:27.574 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:27:27.571+0000 7f728b7fe640 1 --2- 172.21.15.114:0/2063998827 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f728c103b00 0x7f728c198c90 secure :-1 s=READY pgs=512 cs=0 l=1 rev1=1 crypto rx=0x7f728000d950 tx=0x7f728000de10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T10:27:27.574 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:27:27.571+0000 7f72897fa640 1 -- 172.21.15.114:0/2063998827 <== mon.0 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7280017070 con 0x7f728c103b00 2024-04-23T10:27:27.574 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:27:27.571+0000 7f72897fa640 1 -- 172.21.15.114:0/2063998827 <== mon.0 v2:172.21.15.114:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f7280009c30 con 0x7f728c103b00 2024-04-23T10:27:27.574 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:27:27.571+0000 7f72897fa640 1 -- 172.21.15.114:0/2063998827 <== mon.0 v2:172.21.15.114:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f728000f620 con 0x7f728c103b00 2024-04-23T10:27:27.574 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:27:27.571+0000 7f7292508640 1 -- 172.21.15.114:0/2063998827 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f728c19da30 con 0x7f728c103b00 2024-04-23T10:27:27.574 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:27:27.572+0000 7f7292508640 1 -- 172.21.15.114:0/2063998827 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f728c100d60 con 0x7f728c103b00 2024-04-23T10:27:27.574 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:27:27.572+0000 7f7292508640 1 -- 172.21.15.114:0/2063998827 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f728c068d70 con 0x7f728c103b00 2024-04-23T10:27:27.575 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:27:27.575+0000 7f72897fa640 1 -- 172.21.15.114:0/2063998827 <== mon.0 v2:172.21.15.114:3300/0 4 ==== mgrmap(e 25) ==== 138880+0+0 (secure 0 0 0) 0x7f7280010460 con 0x7f728c103b00 2024-04-23T10:27:27.575 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:27:27.575+0000 7f72897fa640 1 --2- 172.21.15.114:0/2063998827 >> [v2:172.21.15.114:6800/1395693466,v1:172.21.15.114:6801/1395693466] conn(0x7f72740a3e40 0x7f72740a6300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T10:27:27.575 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:27:27.575+0000 7f72897fa640 1 -- 172.21.15.114:0/2063998827 <== mon.0 v2:172.21.15.114:3300/0 5 ==== osd_map(90..90 src has 1..90) ==== 8527+0+0 (secure 0 0 0) 0x7f72800d06c0 con 0x7f728c103b00 2024-04-23T10:27:27.576 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:27:27.575+0000 7f728bfff640 1 --2- 172.21.15.114:0/2063998827 >> [v2:172.21.15.114:6800/1395693466,v1:172.21.15.114:6801/1395693466] conn(0x7f72740a3e40 0x7f72740a6300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T10:27:27.576 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:27:27.576+0000 7f728bfff640 1 --2- 172.21.15.114:0/2063998827 >> [v2:172.21.15.114:6800/1395693466,v1:172.21.15.114:6801/1395693466] conn(0x7f72740a3e40 0x7f72740a6300 secure :-1 s=READY pgs=67 cs=0 l=1 rev1=1 crypto rx=0x7f727c031440 tx=0x7f727c005b50 comp rx=0 tx=0).ready entity=mgr.15291 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T10:27:27.581 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:27:27.582+0000 7f72897fa640 1 -- 172.21.15.114:0/2063998827 <== mon.0 v2:172.21.15.114:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f728009cb10 con 0x7f728c103b00 2024-04-23T10:27:27.823 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:27:27.825+0000 7f7292508640 1 -- 172.21.15.114:0/2063998827 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f728c1076d0 con 0x7f728c103b00 2024-04-23T10:27:27.825 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:27:27.826+0000 7f72897fa640 1 -- 172.21.15.114:0/2063998827 <== mon.0 v2:172.21.15.114:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 21 v21) ==== 94+0+11884 (secure 0 0 0) 0x7f7280009de0 con 0x7f728c103b00 2024-04-23T10:27:27.825 INFO:teuthology.orchestra.run.smithi114.stdout: 2024-04-23T10:27:27.825 INFO:teuthology.orchestra.run.smithi114.stdout:{"epoch":21,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14547,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.114:6835/663493580","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6834","nonce":663493580},{"type":"v1","addr":"172.21.15.114:6835","nonce":663493580}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14565,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.114:6837/1439252154","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6836","nonce":1439252154},{"type":"v1","addr":"172.21.15.114:6837","nonce":1439252154}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5},{"gid":14634,"name":"g","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.114:6839/3849796939","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6838","nonce":3849796939},{"type":"v1","addr":"172.21.15.114:6839","nonce":3849796939}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":11},{"gid":14646,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.114:6841/3544523891","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6840","nonce":3544523891},{"type":"v1","addr":"172.21.15.114:6841","nonce":3544523891}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":12},{"gid":24385,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.204:6833/3866561754","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6832","nonce":3866561754},{"type":"v1","addr":"172.21.15.204:6833","nonce":3866561754}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2},{"gid":24427,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.204:6835/2533793172","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6834","nonce":2533793172},{"type":"v1","addr":"172.21.15.204:6835","nonce":2533793172}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":7}],"filesystems":[{"mdsmap":{"epoch":21,"flags":306,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":true},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-04-23T10:24:43.383158+0000","modified":"2024-04-23T10:26:38.945346+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":3,"in":[0,1,2],"up":{"mds_0":24545,"mds_1":24509,"mds_2":24479},"failed":[],"damaged":[],"stopped":[],"info":{"gid_24439":{"gid":24439,"name":"c","rank":2,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.204:6837/513822179","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6836","nonce":513822179},{"type":"v1","addr":"172.21.15.204:6837","nonce":513822179}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24443":{"gid":24443,"name":"k","rank":1,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.119:6833/783244660","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6832","nonce":783244660},{"type":"v1","addr":"172.21.15.119:6833","nonce":783244660}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24463":{"gid":24463,"name":"i","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.204:6839/3222658893","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6838","nonce":3222658893},{"type":"v1","addr":"172.21.15.204:6839","nonce":3222658893}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24479":{"gid":24479,"name":"e","rank":2,"incarnation":19,"state":"up:active","state_seq":9,"addr":"172.21.15.119:6835/2033916709","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6834","nonce":2033916709},{"type":"v1","addr":"172.21.15.119:6835","nonce":2033916709}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24509":{"gid":24509,"name":"h","rank":1,"incarnation":17,"state":"up:active","state_seq":6,"addr":"172.21.15.119:6837/1872363128","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6836","nonce":1872363128},{"type":"v1","addr":"172.21.15.119:6837","nonce":1872363128}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24545":{"gid":24545,"name":"b","rank":0,"incarnation":14,"state":"up:active","state_seq":3,"addr":"172.21.15.119:6839/937297340","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6838","nonce":937297340},{"type":"v1","addr":"172.21.15.119:6839","nonce":937297340}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":24545,"qdb_cluster":[24479,24509,24545]},"id":1}]} 2024-04-23T10:27:27.825 INFO:teuthology.orchestra.run.smithi114.stderr:dumped fsmap epoch 21 2024-04-23T10:27:27.827 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:27:27.828+0000 7f7292508640 1 -- 172.21.15.114:0/2063998827 >> [v2:172.21.15.114:6800/1395693466,v1:172.21.15.114:6801/1395693466] conn(0x7f72740a3e40 msgr2=0x7f72740a6300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T10:27:27.827 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:27:27.828+0000 7f7292508640 1 --2- 172.21.15.114:0/2063998827 >> [v2:172.21.15.114:6800/1395693466,v1:172.21.15.114:6801/1395693466] conn(0x7f72740a3e40 0x7f72740a6300 secure :-1 s=READY pgs=67 cs=0 l=1 rev1=1 crypto rx=0x7f727c031440 tx=0x7f727c005b50 comp rx=0 tx=0).stop 2024-04-23T10:27:27.827 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:27:27.828+0000 7f7292508640 1 -- 172.21.15.114:0/2063998827 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f728c103b00 msgr2=0x7f728c198c90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T10:27:27.828 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:27:27.828+0000 7f7292508640 1 --2- 172.21.15.114:0/2063998827 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f728c103b00 0x7f728c198c90 secure :-1 s=READY pgs=512 cs=0 l=1 rev1=1 crypto rx=0x7f728000d950 tx=0x7f728000de10 comp rx=0 tx=0).stop 2024-04-23T10:27:27.828 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:27:27.828+0000 7f7292508640 1 -- 172.21.15.114:0/2063998827 shutdown_connections 2024-04-23T10:27:27.828 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:27:27.828+0000 7f7292508640 1 --2- 172.21.15.114:0/2063998827 >> [v2:172.21.15.114:6800/1395693466,v1:172.21.15.114:6801/1395693466] conn(0x7f72740a3e40 0x7f72740a6300 unknown :-1 s=CLOSED pgs=67 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:27:27.828 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:27:27.828+0000 7f7292508640 1 --2- 172.21.15.114:0/2063998827 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f728c10c520 0x7f728c19d010 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:27:27.828 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:27:27.828+0000 7f7292508640 1 --2- 172.21.15.114:0/2063998827 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f728c103b00 0x7f728c198c90 unknown :-1 s=CLOSED pgs=512 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:27:27.828 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:27:27.828+0000 7f7292508640 1 --2- 172.21.15.114:0/2063998827 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f728c1031e0 0x7f728c198750 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:27:27.828 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:27:27.828+0000 7f7292508640 1 -- 172.21.15.114:0/2063998827 >> 172.21.15.114:0/2063998827 conn(0x7f728c077f50 msgr2=0x7f728c0fdfe0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T10:27:27.828 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:27:27.828+0000 7f7292508640 1 -- 172.21.15.114:0/2063998827 shutdown_connections 2024-04-23T10:27:27.828 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:27:27.828+0000 7f7292508640 1 -- 172.21.15.114:0/2063998827 wait complete. 2024-04-23T10:27:27.839 DEBUG:tasks.quiescer.fs.[cephfs]:Running ceph command: 'tell mds.24545 quiesce db --set-id 7f49b8f3 --release --await' 2024-04-23T10:27:27.839 DEBUG:teuthology.orchestra.run.smithi114:> timeout 75 sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.24545 quiesce db --set-id 7f49b8f3 --release --await 2024-04-23T10:27:27.952 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:27:27.950+0000 7f02c4a20640 1 -- 172.21.15.114:0/1109112676 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f02c0101090 msgr2=0x7f02c0101470 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T10:27:27.953 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:27:27.950+0000 7f02c4a20640 1 --2- 172.21.15.114:0/1109112676 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f02c0101090 0x7f02c0101470 secure :-1 s=READY pgs=513 cs=0 l=1 rev1=1 crypto rx=0x7f02b0009900 tx=0x7f02b002a410 comp rx=0 tx=0).stop 2024-04-23T10:27:27.953 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:27:27.950+0000 7f02c4a20640 1 -- 172.21.15.114:0/1109112676 shutdown_connections 2024-04-23T10:27:27.953 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:27:27.950+0000 7f02c4a20640 1 --2- 172.21.15.114:0/1109112676 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f02c0106160 0x7f02c010dd30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:27:27.953 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:27:27.950+0000 7f02c4a20640 1 --2- 172.21.15.114:0/1109112676 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f02c0101a40 0x7f02c0105a80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:27:27.953 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:27:27.950+0000 7f02c4a20640 1 --2- 172.21.15.114:0/1109112676 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f02c0101090 0x7f02c0101470 unknown :-1 s=CLOSED pgs=513 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:27:27.953 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:27:27.951+0000 7f02c4a20640 1 -- 172.21.15.114:0/1109112676 >> 172.21.15.114:0/1109112676 conn(0x7f02c0077f50 msgr2=0x7f02c0078360 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T10:27:27.953 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:27:27.951+0000 7f02c4a20640 1 -- 172.21.15.114:0/1109112676 shutdown_connections 2024-04-23T10:27:27.953 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:27:27.951+0000 7f02c4a20640 1 -- 172.21.15.114:0/1109112676 wait complete. 2024-04-23T10:27:27.953 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:27:27.951+0000 7f02c4a20640 1 Processor -- start 2024-04-23T10:27:27.954 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:27:27.951+0000 7f02c4a20640 1 -- start start 2024-04-23T10:27:27.954 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:27:27.952+0000 7f02c4a20640 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f02c0101090 0x7f02c0197600 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T10:27:27.954 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:27:27.952+0000 7f02c4a20640 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f02c0106160 0x7f02c0197b40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T10:27:27.954 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:27:27.952+0000 7f02c4a20640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f02c019bec0 0x7f02c019c370 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T10:27:27.954 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:27:27.952+0000 7f02c4a20640 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x7f02c01110a0 con 0x7f02c019bec0 2024-04-23T10:27:27.954 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:27:27.952+0000 7f02c4a20640 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 -- 0x7f02c0110f20 con 0x7f02c0106160 2024-04-23T10:27:27.954 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:27:27.952+0000 7f02c4a20640 1 -- --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_getmap magic: 0 -- 0x7f02c0111220 con 0x7f02c0101090 2024-04-23T10:27:27.954 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:27:27.952+0000 7f02bed9d640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f02c019bec0 0x7f02c019c370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T10:27:27.954 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:27:27.952+0000 7f02bed9d640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f02c019bec0 0x7f02c019c370 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.114:53500/0 (socket says 172.21.15.114:53500) 2024-04-23T10:27:27.954 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:27:27.952+0000 7f02bed9d640 1 -- 172.21.15.114:0/834007175 learned_addr learned my addr 172.21.15.114:0/834007175 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-04-23T10:27:27.955 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:27:27.952+0000 7f02be59c640 1 --2- 172.21.15.114:0/834007175 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f02c0101090 0x7f02c0197600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T10:27:27.955 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:27:27.952+0000 7f02b7fff640 1 --2- 172.21.15.114:0/834007175 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f02c0106160 0x7f02c0197b40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T10:27:27.955 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:27:27.952+0000 7f02bed9d640 1 -- 172.21.15.114:0/834007175 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f02c0101090 msgr2=0x7f02c0197600 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T10:27:27.955 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:27:27.952+0000 7f02bed9d640 1 --2- 172.21.15.114:0/834007175 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f02c0101090 0x7f02c0197600 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:27:27.955 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:27:27.952+0000 7f02bed9d640 1 -- 172.21.15.114:0/834007175 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f02c0106160 msgr2=0x7f02c0197b40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T10:27:27.955 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:27:27.952+0000 7f02bed9d640 1 --2- 172.21.15.114:0/834007175 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f02c0106160 0x7f02c0197b40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:27:27.955 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:27:27.952+0000 7f02bed9d640 1 -- 172.21.15.114:0/834007175 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f02c019c9b0 con 0x7f02c019bec0 2024-04-23T10:27:27.955 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:27:27.952+0000 7f02b7fff640 1 --2- 172.21.15.114:0/834007175 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f02c0106160 0x7f02c0197b40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-23T10:27:27.955 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:27:27.952+0000 7f02bed9d640 1 --2- 172.21.15.114:0/834007175 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f02c019bec0 0x7f02c019c370 secure :-1 s=READY pgs=514 cs=0 l=1 rev1=1 crypto rx=0x7f02b800eca0 tx=0x7f02b800b690 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T10:27:27.955 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:27:27.952+0000 7f02b77fe640 1 -- 172.21.15.114:0/834007175 <== mon.0 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f02b8004b20 con 0x7f02c019bec0 2024-04-23T10:27:27.955 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:27:27.952+0000 7f02c4a20640 1 -- 172.21.15.114:0/834007175 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f02c019cca0 con 0x7f02c019bec0 2024-04-23T10:27:27.956 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:27:27.952+0000 7f02c4a20640 1 -- 172.21.15.114:0/834007175 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f02c0069400 con 0x7f02c019bec0 2024-04-23T10:27:27.956 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:27:27.953+0000 7f02b77fe640 1 -- 172.21.15.114:0/834007175 <== mon.0 v2:172.21.15.114:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f02b8004cc0 con 0x7f02c019bec0 2024-04-23T10:27:27.956 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:27:27.953+0000 7f02b77fe640 1 -- 172.21.15.114:0/834007175 <== mon.0 v2:172.21.15.114:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f02b8017670 con 0x7f02c019bec0 2024-04-23T10:27:27.964 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:27:27.955+0000 7f02b77fe640 1 -- 172.21.15.114:0/834007175 <== mon.0 v2:172.21.15.114:3300/0 4 ==== mgrmap(e 25) ==== 138880+0+0 (secure 0 0 0) 0x7f02b8005020 con 0x7f02c019bec0 2024-04-23T10:27:27.964 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:27:27.955+0000 7f02c780b740 1 Processor -- start 2024-04-23T10:27:27.964 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:27:27.955+0000 7f02c780b740 1 -- start start 2024-04-23T10:27:27.964 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:27:27.955+0000 7f02c780b740 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x560c2b34d8a0 0x560c2b34dc80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T10:27:27.964 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:27:27.955+0000 7f02c780b740 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x560c2b34e1c0 0x560c2b358350 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T10:27:27.964 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:27:27.955+0000 7f02c780b740 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x560c2b358890 0x560c2b35ac80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T10:27:27.964 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:27:27.955+0000 7f02c780b740 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x560c2b2ae6d0 con 0x560c2b358890 2024-04-23T10:27:27.964 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:27:27.955+0000 7f02c780b740 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 -- 0x560c2b24b1a0 con 0x560c2b34e1c0 2024-04-23T10:27:27.965 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:27:27.955+0000 7f02c780b740 1 -- --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_getmap magic: 0 -- 0x560c2b2aa690 con 0x560c2b34d8a0 2024-04-23T10:27:27.965 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:27:27.956+0000 7f02b77fe640 1 --2- 172.21.15.114:0/834007175 >> [v2:172.21.15.114:6800/1395693466,v1:172.21.15.114:6801/1395693466] conn(0x7f02ac0a3ec0 0x7f02ac0a6380 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T10:27:27.965 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:27:27.956+0000 7f02b7fff640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x560c2b358890 0x560c2b35ac80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T10:27:27.965 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:27:27.956+0000 7f02b7fff640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x560c2b358890 0x560c2b35ac80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.114:53510/0 (socket says 172.21.15.114:53510) 2024-04-23T10:27:27.965 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:27:27.956+0000 7f02b7fff640 1 -- 172.21.15.114:0/3318349069 learned_addr learned my addr 172.21.15.114:0/3318349069 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-04-23T10:27:27.965 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:27:27.956+0000 7f02b7fff640 1 --2- 172.21.15.114:0/3318349069 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x560c2b34d8a0 0x560c2b34dc80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T10:27:27.965 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:27:27.956+0000 7f02bed9d640 1 --2- 172.21.15.114:0/834007175 >> [v2:172.21.15.114:6800/1395693466,v1:172.21.15.114:6801/1395693466] conn(0x7f02ac0a3ec0 0x7f02ac0a6380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T10:27:27.965 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:27:27.956+0000 7f02be59c640 1 --2- 172.21.15.114:0/3318349069 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x560c2b34e1c0 0x560c2b358350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T10:27:27.965 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:27:27.956+0000 7f02b7fff640 1 -- 172.21.15.114:0/3318349069 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x560c2b34d8a0 msgr2=0x560c2b34dc80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T10:27:27.965 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:27:27.956+0000 7f02b7fff640 1 --2- 172.21.15.114:0/3318349069 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x560c2b34d8a0 0x560c2b34dc80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:27:27.965 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:27:27.956+0000 7f02b7fff640 1 -- 172.21.15.114:0/3318349069 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x560c2b34e1c0 msgr2=0x560c2b358350 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T10:27:27.965 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:27:27.956+0000 7f02b7fff640 1 --2- 172.21.15.114:0/3318349069 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x560c2b34e1c0 0x560c2b358350 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:27:27.966 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:27:27.956+0000 7f02b7fff640 1 -- 172.21.15.114:0/3318349069 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x560c2b2555b0 con 0x560c2b358890 2024-04-23T10:27:27.966 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:27:27.956+0000 7f02be59c640 1 --2- 172.21.15.114:0/3318349069 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x560c2b34e1c0 0x560c2b358350 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-23T10:27:27.966 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:27:27.956+0000 7f02b7fff640 1 --2- 172.21.15.114:0/3318349069 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x560c2b358890 0x560c2b35ac80 secure :-1 s=READY pgs=515 cs=0 l=1 rev1=1 crypto rx=0x7f02a8012800 tx=0x7f02a8009700 comp rx=0 tx=0).ready entity=mon.0 client_cookie=fe6303133335b536 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T10:27:27.966 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:27:27.956+0000 7f02b4ff9640 1 -- 172.21.15.114:0/3318349069 <== mon.0 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f02a8012d80 con 0x560c2b358890 2024-04-23T10:27:27.966 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:27:27.956+0000 7f02b4ff9640 1 -- 172.21.15.114:0/3318349069 <== mon.0 v2:172.21.15.114:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f02a8010c80 con 0x560c2b358890 2024-04-23T10:27:27.966 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:27:27.956+0000 7f02b4ff9640 1 -- 172.21.15.114:0/3318349069 <== mon.0 v2:172.21.15.114:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f02a801ebf0 con 0x560c2b358890 2024-04-23T10:27:27.966 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:27:27.956+0000 7f02b77fe640 1 -- 172.21.15.114:0/834007175 <== mon.0 v2:172.21.15.114:3300/0 5 ==== osd_map(90..90 src has 1..90) ==== 8527+0+0 (secure 0 0 0) 0x7f02b80d2360 con 0x7f02c019bec0 2024-04-23T10:27:27.966 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:27:27.957+0000 7f02c780b740 1 -- 172.21.15.114:0/3318349069 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x560c2b358890 msgr2=0x560c2b35ac80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T10:27:27.966 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:27:27.957+0000 7f02bed9d640 1 --2- 172.21.15.114:0/834007175 >> [v2:172.21.15.114:6800/1395693466,v1:172.21.15.114:6801/1395693466] conn(0x7f02ac0a3ec0 0x7f02ac0a6380 secure :-1 s=READY pgs=68 cs=0 l=1 rev1=1 crypto rx=0x7f02b80cf990 tx=0x7f02b8099d10 comp rx=0 tx=0).ready entity=mgr.15291 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T10:27:27.967 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:27:27.957+0000 7f02c780b740 1 --2- 172.21.15.114:0/3318349069 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x560c2b358890 0x560c2b35ac80 secure :-1 s=READY pgs=515 cs=0 l=1 rev1=1 crypto rx=0x7f02a8012800 tx=0x7f02a8009700 comp rx=0 tx=0).stop 2024-04-23T10:27:27.967 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:27:27.957+0000 7f02c780b740 1 -- 172.21.15.114:0/3318349069 shutdown_connections 2024-04-23T10:27:27.967 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:27:27.957+0000 7f02c780b740 1 --2- 172.21.15.114:0/3318349069 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x560c2b358890 0x560c2b35ac80 unknown :-1 s=CLOSED pgs=515 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:27:27.967 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:27:27.957+0000 7f02c780b740 1 --2- 172.21.15.114:0/3318349069 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x560c2b34e1c0 0x560c2b358350 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:27:27.967 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:27:27.957+0000 7f02c780b740 1 --2- 172.21.15.114:0/3318349069 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x560c2b34d8a0 0x560c2b34dc80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:27:27.967 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:27:27.957+0000 7f02c780b740 1 -- 172.21.15.114:0/3318349069 >> 172.21.15.114:0/3318349069 conn(0x560c2b349bc0 msgr2=0x560c2b349fa0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T10:27:27.967 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:27:27.957+0000 7f02c780b740 1 -- 172.21.15.114:0/3318349069 shutdown_connections 2024-04-23T10:27:27.967 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:27:27.957+0000 7f02c780b740 1 -- 172.21.15.114:0/3318349069 wait complete. 2024-04-23T10:27:27.967 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:27:27.957+0000 7f02c780b740 1 -- start start 2024-04-23T10:27:27.967 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:27:27.958+0000 7f02c780b740 1 Processor -- start 2024-04-23T10:27:27.967 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:27:27.958+0000 7f02c780b740 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x560c2b34d8a0 0x560c2b358320 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T10:27:27.968 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:27:27.958+0000 7f02c780b740 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x560c2b34e1c0 0x560c2b355610 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T10:27:27.968 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:27:27.958+0000 7f02c780b740 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x560c2b358890 0x560c2b355d00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T10:27:27.968 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:27:27.958+0000 7f02c780b740 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x560c2b35b720 con 0x560c2b34e1c0 2024-04-23T10:27:27.968 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:27:27.958+0000 7f02c780b740 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 -- 0x560c2b248b20 con 0x560c2b34d8a0 2024-04-23T10:27:27.968 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:27:27.958+0000 7f02c780b740 1 -- --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_getmap magic: 0 -- 0x560c2b35b8a0 con 0x560c2b358890 2024-04-23T10:27:27.968 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:27:27.958+0000 7f02c780b740 10 client.0 _ll_register_callbacks cb 0x560c2b248530 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-23T10:27:27.968 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:27:27.958+0000 7f02be59c640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x560c2b34e1c0 0x560c2b355610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T10:27:27.968 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:27:27.958+0000 7f02b7fff640 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x560c2b34d8a0 0x560c2b358320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T10:27:27.968 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:27:27.958+0000 7f02b7fff640 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x560c2b34d8a0 0x560c2b358320 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.119:3300/0 says I am v2:172.21.15.114:57900/0 (socket says 172.21.15.114:57900) 2024-04-23T10:27:27.968 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:27:27.958+0000 7f02b7fff640 1 -- 172.21.15.114:0/366183728 learned_addr learned my addr 172.21.15.114:0/366183728 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-04-23T10:27:27.968 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:27:27.958+0000 7f02b7fff640 1 --2- 172.21.15.114:0/366183728 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x560c2b358890 0x560c2b355d00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T10:27:27.968 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:27:27.958+0000 7f02be59c640 1 -- 172.21.15.114:0/366183728 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x560c2b358890 msgr2=0x560c2b355d00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T10:27:27.968 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:27:27.958+0000 7f02be59c640 1 --2- 172.21.15.114:0/366183728 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x560c2b358890 0x560c2b355d00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:27:27.969 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:27:27.958+0000 7f02be59c640 1 -- 172.21.15.114:0/366183728 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x560c2b34d8a0 msgr2=0x560c2b358320 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T10:27:27.969 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:27:27.958+0000 7f02be59c640 1 --2- 172.21.15.114:0/366183728 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x560c2b34d8a0 0x560c2b358320 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:27:27.969 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:27:27.958+0000 7f02be59c640 1 -- 172.21.15.114:0/366183728 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({osdmap=0}) -- 0x560c2afc0920 con 0x560c2b34e1c0 2024-04-23T10:27:27.969 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:27:27.958+0000 7f02b7fff640 1 --2- 172.21.15.114:0/366183728 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x560c2b34d8a0 0x560c2b358320 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-23T10:27:27.969 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:27:27.959+0000 7f02be59c640 1 --2- 172.21.15.114:0/366183728 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x560c2b34e1c0 0x560c2b355610 secure :-1 s=READY pgs=516 cs=0 l=1 rev1=1 crypto rx=0x7f02b0000c00 tx=0x7f02b0032dd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c5490861666969e4 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T10:27:27.969 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:27:27.959+0000 7f02a5ffb640 10 client.0 ms_handle_connect on v2:172.21.15.114:3300/0 2024-04-23T10:27:27.969 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:27:27.959+0000 7f02a5ffb640 1 -- 172.21.15.114:0/366183728 <== mon.0 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f02b0035040 con 0x560c2b34e1c0 2024-04-23T10:27:27.969 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:27:27.959+0000 7f02a5ffb640 1 -- 172.21.15.114:0/366183728 <== mon.0 v2:172.21.15.114:3300/0 2 ==== osd_map(90..90 src has 1..90) ==== 8527+0+0 (secure 0 0 0) 0x7f02b0004040 con 0x560c2b34e1c0 2024-04-23T10:27:27.969 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:27:27.959+0000 7f02c780b740 1 -- 172.21.15.114:0/366183728 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f02a8010c80 con 0x560c2b34e1c0 2024-04-23T10:27:27.969 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:27:27.959+0000 7f02a5ffb640 1 -- 172.21.15.114:0/366183728 <== mon.0 v2:172.21.15.114:3300/0 3 ==== mon_get_version_reply(handle=1 version=21) ==== 24+0+0 (secure 0 0 0) 0x7f02b00318a0 con 0x560c2b34e1c0 2024-04-23T10:27:27.969 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:27:27.959+0000 7f02c780b740 10 client.15654 fetch_fsmap learned FSMap version 21 2024-04-23T10:27:27.969 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:27:27.959+0000 7f02c780b740 1 -- 172.21.15.114:0/366183728 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({config=0+,fsmap=21,monmap=4+}) -- 0x560c2b08cd30 con 0x560c2b34e1c0 2024-04-23T10:27:27.970 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:27:27.959+0000 7f02a5ffb640 1 -- 172.21.15.114:0/366183728 <== mon.0 v2:172.21.15.114:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f02a8010c80 con 0x560c2b34e1c0 2024-04-23T10:27:27.970 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:27:27.959+0000 7f02a5ffb640 1 -- 172.21.15.114:0/366183728 <== mon.0 v2:172.21.15.114:3300/0 5 ==== fsmap(e 21) ==== 8338+0+0 (secure 0 0 0) 0x7f02b00046e0 con 0x560c2b34e1c0 2024-04-23T10:27:27.970 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:27:27.959+0000 7f02c780b740 10 client.15654 fetch_fsmap finished waiting for FSMap version 21 2024-04-23T10:27:27.970 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:27:27.959+0000 7f02c780b740 10 client.15654 resolve_mds: validated gid 24545 aka daemon mds.b 2024-04-23T10:27:27.970 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:27:27.959+0000 7f02c780b740 20 client.15654 populate_metadata read hostname 'smithi114' 2024-04-23T10:27:27.970 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:27:27.959+0000 7f02c780b740 1 --2- 172.21.15.114:0/366183728 >> [v2:172.21.15.119:6838/937297340,v1:172.21.15.119:6839/937297340] conn(0x560c2b368a50 0x560c2b36aeb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T10:27:27.970 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:27:27.959+0000 7f02c780b740 4 client.15654 mds_command: new command op to 24545 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-23T10:27:27.970 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:27:27.959+0000 7f02c780b740 1 -- 172.21.15.114:0/366183728 --> [v2:172.21.15.119:6838/937297340,v1:172.21.15.119:6839/937297340] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x560c2b26c500 con 0x560c2b368a50 2024-04-23T10:27:27.970 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:27:27.960+0000 7f02b7fff640 1 --2- 172.21.15.114:0/366183728 >> [v2:172.21.15.119:6838/937297340,v1:172.21.15.119:6839/937297340] conn(0x560c2b368a50 0x560c2b36aeb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T10:27:27.970 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:27:27.960+0000 7f02a5ffb640 10 client.15654 ms_handle_connect on v2:172.21.15.119:6838/937297340 2024-04-23T10:27:27.970 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:27:27.960+0000 7f02b7fff640 1 --2- 172.21.15.114:0/366183728 >> [v2:172.21.15.119:6838/937297340,v1:172.21.15.119:6839/937297340] conn(0x560c2b368a50 0x560c2b36aeb0 crc :-1 s=READY pgs=48 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=43557e4e3659b836 server_cookie=3cd32e5325c0e054 in_seq=0 out_seq=0 2024-04-23T10:27:27.970 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:27:27.961+0000 7f02a5ffb640 1 -- 172.21.15.114:0/366183728 <== mds.0 v2:172.21.15.119:6838/937297340 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x560c2b26c500 con 0x560c2b368a50 2024-04-23T10:27:27.971 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:27:27.961+0000 7f02a5ffb640 10 client.15654 handle_command_reply: tid=0 2024-04-23T10:27:27.971 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:27:27.961+0000 7f02c780b740 1 client.15654 shutdown 2024-04-23T10:27:27.971 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:27:27.961+0000 7f02c780b740 1 -- 172.21.15.114:0/366183728 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x560c2b34e1c0 msgr2=0x560c2b355610 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T10:27:27.971 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:27:27.961+0000 7f02c780b740 1 --2- 172.21.15.114:0/366183728 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x560c2b34e1c0 0x560c2b355610 secure :-1 s=READY pgs=516 cs=0 l=1 rev1=1 crypto rx=0x7f02b0000c00 tx=0x7f02b0032dd0 comp rx=0 tx=0).stop 2024-04-23T10:27:27.971 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:27:27.961+0000 7f02c780b740 1 -- 172.21.15.114:0/366183728 shutdown_connections 2024-04-23T10:27:27.971 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:27:27.961+0000 7f02c780b740 1 --2- 172.21.15.114:0/366183728 >> [v2:172.21.15.119:6838/937297340,v1:172.21.15.119:6839/937297340] conn(0x560c2b368a50 0x560c2b36aeb0 crc :-1 s=READY pgs=48 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:27:27.971 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:27:27.961+0000 7f02a5ffb640 0 client.15654 ms_handle_reset on v2:172.21.15.119:6838/937297340 2024-04-23T10:27:27.971 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:27:27.961+0000 7f02c780b740 1 --2- 172.21.15.114:0/366183728 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x560c2b358890 0x560c2b355d00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:27:27.971 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:27:27.961+0000 7f02c780b740 1 --2- 172.21.15.114:0/366183728 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x560c2b34e1c0 0x560c2b355610 unknown :-1 s=CLOSED pgs=516 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:27:27.971 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:27:27.961+0000 7f02c780b740 1 --2- 172.21.15.114:0/366183728 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x560c2b34d8a0 0x560c2b358320 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:27:27.971 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:27:27.961+0000 7f02c780b740 1 -- 172.21.15.114:0/366183728 >> 172.21.15.114:0/366183728 conn(0x560c2b349bc0 msgr2=0x560c2b3449d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T10:27:27.971 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:27:27.961+0000 7f02c780b740 1 -- 172.21.15.114:0/366183728 shutdown_connections 2024-04-23T10:27:27.972 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:27:27.961+0000 7f02c780b740 1 -- 172.21.15.114:0/366183728 wait complete. 2024-04-23T10:27:27.972 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:27:27.961+0000 7f02c780b740 20 client.15654 trim_cache size 0 max 16384 2024-04-23T10:27:27.982 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:27:27.978+0000 7f02c780b740 1 Processor -- start 2024-04-23T10:27:27.982 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:27:27.978+0000 7f02c780b740 1 -- start start 2024-04-23T10:27:27.982 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:27:27.978+0000 7f02c780b740 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x560c2b34d8a0 0x560c2b364640 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T10:27:27.982 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:27:27.978+0000 7f02c780b740 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x560c2b34e1c0 0x560c2b357e50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T10:27:27.983 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:27:27.978+0000 7f02c780b740 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x560c2b358890 0x560c2b3707c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T10:27:27.983 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:27:27.978+0000 7f02c780b740 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x7f02a0011090 con 0x560c2b34d8a0 2024-04-23T10:27:27.983 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:27:27.978+0000 7f02c780b740 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 -- 0x7f02a000e960 con 0x560c2b358890 2024-04-23T10:27:27.983 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:27:27.978+0000 7f02c780b740 1 -- --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_getmap magic: 0 -- 0x7f02a000c7b0 con 0x560c2b34e1c0 2024-04-23T10:27:27.983 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:27:27.978+0000 7f02b7fff640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x560c2b34d8a0 0x560c2b364640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T10:27:27.983 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:27:27.979+0000 7f02b7fff640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x560c2b34d8a0 0x560c2b364640 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.114:53534/0 (socket says 172.21.15.114:53534) 2024-04-23T10:27:27.983 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:27:27.979+0000 7f02b7fff640 1 -- 172.21.15.114:0/953350837 learned_addr learned my addr 172.21.15.114:0/953350837 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-04-23T10:27:27.983 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:27:27.979+0000 7f02be59c640 1 --2- 172.21.15.114:0/953350837 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x560c2b34e1c0 0x560c2b357e50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T10:27:27.983 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:27:27.979+0000 7f02b7fff640 1 -- 172.21.15.114:0/953350837 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x560c2b34e1c0 msgr2=0x560c2b357e50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T10:27:27.983 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:27:27.979+0000 7f02b7fff640 1 --2- 172.21.15.114:0/953350837 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x560c2b34e1c0 0x560c2b357e50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:27:27.983 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:27:27.979+0000 7f02b7fff640 1 -- 172.21.15.114:0/953350837 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x560c2b358890 msgr2=0x560c2b3707c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T10:27:27.984 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:27:27.979+0000 7f02b7fff640 1 --2- 172.21.15.114:0/953350837 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x560c2b358890 0x560c2b3707c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:27:27.984 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:27:27.979+0000 7f02b7fff640 1 -- 172.21.15.114:0/953350837 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x560c2b3562e0 con 0x560c2b34d8a0 2024-04-23T10:27:27.984 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:27:27.979+0000 7f02be59c640 1 --2- 172.21.15.114:0/953350837 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x560c2b34e1c0 0x560c2b357e50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-23T10:27:27.984 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:27:27.979+0000 7f02b7fff640 1 --2- 172.21.15.114:0/953350837 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x560c2b34d8a0 0x560c2b364640 secure :-1 s=READY pgs=517 cs=0 l=1 rev1=1 crypto rx=0x7f02a8012800 tx=0x7f02a8009be0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=acf91d34f9f74ef0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T10:27:27.984 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:27:27.979+0000 7f02a7fff640 1 -- 172.21.15.114:0/953350837 <== mon.0 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f02a8024040 con 0x560c2b34d8a0 2024-04-23T10:27:27.984 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:27:27.979+0000 7f02a7fff640 1 -- 172.21.15.114:0/953350837 <== mon.0 v2:172.21.15.114:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f02a8034070 con 0x560c2b34d8a0 2024-04-23T10:27:27.984 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:27:27.979+0000 7f02a7fff640 1 -- 172.21.15.114:0/953350837 <== mon.0 v2:172.21.15.114:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f02a801ec10 con 0x560c2b34d8a0 2024-04-23T10:27:27.984 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:27:27.980+0000 7f02c780b740 1 -- 172.21.15.114:0/953350837 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x560c2b34d8a0 msgr2=0x560c2b364640 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T10:27:27.984 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:27:27.980+0000 7f02c780b740 1 --2- 172.21.15.114:0/953350837 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x560c2b34d8a0 0x560c2b364640 secure :-1 s=READY pgs=517 cs=0 l=1 rev1=1 crypto rx=0x7f02a8012800 tx=0x7f02a8009be0 comp rx=0 tx=0).stop 2024-04-23T10:27:27.984 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:27:27.980+0000 7f02c780b740 1 -- 172.21.15.114:0/953350837 shutdown_connections 2024-04-23T10:27:27.984 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:27:27.980+0000 7f02c780b740 1 --2- 172.21.15.114:0/953350837 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x560c2b358890 0x560c2b3707c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:27:27.984 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:27:27.980+0000 7f02c780b740 1 --2- 172.21.15.114:0/953350837 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x560c2b34e1c0 0x560c2b357e50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:27:27.984 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:27:27.980+0000 7f02c780b740 1 --2- 172.21.15.114:0/953350837 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x560c2b34d8a0 0x560c2b364640 unknown :-1 s=CLOSED pgs=517 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:27:27.985 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:27:27.980+0000 7f02c780b740 1 -- 172.21.15.114:0/953350837 >> 172.21.15.114:0/953350837 conn(0x560c2b349bc0 msgr2=0x560c2b3449d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T10:27:27.985 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:27:27.980+0000 7f02c780b740 1 -- 172.21.15.114:0/953350837 shutdown_connections 2024-04-23T10:27:27.985 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:27:27.980+0000 7f02c780b740 1 -- 172.21.15.114:0/953350837 wait complete. 2024-04-23T10:27:27.985 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:27:27.980+0000 7f02c780b740 1 -- start start 2024-04-23T10:27:27.985 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:27:27.980+0000 7f02c780b740 1 Processor -- start 2024-04-23T10:27:27.985 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:27:27.981+0000 7f02c780b740 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x560c2b34d8a0 0x560c2b361050 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T10:27:27.985 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:27:27.981+0000 7f02c780b740 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x560c2b34e1c0 0x560c2b3518a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T10:27:27.985 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:27:27.981+0000 7f02c780b740 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x560c2b358890 0x560c2b351fe0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T10:27:27.985 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:27:27.981+0000 7f02c780b740 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x7f02a000ce20 con 0x560c2b358890 2024-04-23T10:27:27.985 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:27:27.981+0000 7f02c780b740 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 -- 0x7f02a000e290 con 0x560c2b34e1c0 2024-04-23T10:27:27.985 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:27:27.981+0000 7f02c780b740 1 -- --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_getmap magic: 0 -- 0x7f02a000d4f0 con 0x560c2b34d8a0 2024-04-23T10:27:27.985 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:27:27.981+0000 7f02c780b740 10 client.0 _ll_register_callbacks cb 0x7f02b00314a0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-23T10:27:27.985 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:27:27.981+0000 7f02b7fff640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x560c2b358890 0x560c2b351fe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T10:27:27.986 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:27:27.981+0000 7f02b7fff640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x560c2b358890 0x560c2b351fe0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.114:53550/0 (socket says 172.21.15.114:53550) 2024-04-23T10:27:27.986 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:27:27.981+0000 7f02b7fff640 1 -- 172.21.15.114:0/3538053978 learned_addr learned my addr 172.21.15.114:0/3538053978 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-04-23T10:27:27.986 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:27:27.981+0000 7f02b7fff640 1 --2- 172.21.15.114:0/3538053978 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x560c2b34d8a0 0x560c2b361050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T10:27:27.986 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:27:27.981+0000 7f02be59c640 1 --2- 172.21.15.114:0/3538053978 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x560c2b34e1c0 0x560c2b3518a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T10:27:27.986 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:27:27.981+0000 7f02b7fff640 1 -- 172.21.15.114:0/3538053978 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x560c2b34d8a0 msgr2=0x560c2b361050 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T10:27:27.986 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:27:27.981+0000 7f02b7fff640 1 --2- 172.21.15.114:0/3538053978 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x560c2b34d8a0 0x560c2b361050 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:27:27.986 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:27:27.981+0000 7f02b7fff640 1 -- 172.21.15.114:0/3538053978 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x560c2b34e1c0 msgr2=0x560c2b3518a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T10:27:27.986 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:27:27.981+0000 7f02b7fff640 1 --2- 172.21.15.114:0/3538053978 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x560c2b34e1c0 0x560c2b3518a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:27:27.986 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:27:27.981+0000 7f02b7fff640 1 -- 172.21.15.114:0/3538053978 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({osdmap=0}) -- 0x560c2b352810 con 0x560c2b358890 2024-04-23T10:27:27.986 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:27:27.981+0000 7f02b7fff640 1 --2- 172.21.15.114:0/3538053978 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x560c2b358890 0x560c2b351fe0 secure :-1 s=READY pgs=518 cs=0 l=1 rev1=1 crypto rx=0x7f02a801e8f0 tx=0x7f02a8014040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9a2b1d54a07b6341 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T10:27:27.986 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:27:27.981+0000 7f02a5ffb640 10 client.0 ms_handle_connect on v2:172.21.15.114:3300/0 2024-04-23T10:27:27.987 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:27:27.981+0000 7f02a5ffb640 1 -- 172.21.15.114:0/3538053978 <== mon.0 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f02a801e990 con 0x560c2b358890 2024-04-23T10:27:27.987 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:27:27.982+0000 7f02a5ffb640 1 -- 172.21.15.114:0/3538053978 <== mon.0 v2:172.21.15.114:3300/0 2 ==== osd_map(90..90 src has 1..90) ==== 8527+0+0 (secure 0 0 0) 0x7f02a801eb30 con 0x560c2b358890 2024-04-23T10:27:27.987 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:27:27.982+0000 7f02c780b740 1 -- 172.21.15.114:0/3538053978 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f02a8034070 con 0x560c2b358890 2024-04-23T10:27:27.987 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:27:27.982+0000 7f02a5ffb640 1 -- 172.21.15.114:0/3538053978 <== mon.0 v2:172.21.15.114:3300/0 3 ==== mon_get_version_reply(handle=1 version=21) ==== 24+0+0 (secure 0 0 0) 0x7f02a801ed30 con 0x560c2b358890 2024-04-23T10:27:27.987 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:27:27.982+0000 7f02c780b740 10 client.15660 fetch_fsmap learned FSMap version 21 2024-04-23T10:27:27.987 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:27:27.982+0000 7f02c780b740 1 -- 172.21.15.114:0/3538053978 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({config=0+,fsmap=21,monmap=4+}) -- 0x560c2b3d0190 con 0x560c2b358890 2024-04-23T10:27:27.987 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:27:27.982+0000 7f02a5ffb640 1 -- 172.21.15.114:0/3538053978 <== mon.0 v2:172.21.15.114:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f02a80054f0 con 0x560c2b358890 2024-04-23T10:27:27.987 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:27:27.982+0000 7f02a5ffb640 1 -- 172.21.15.114:0/3538053978 <== mon.0 v2:172.21.15.114:3300/0 5 ==== fsmap(e 21) ==== 8338+0+0 (secure 0 0 0) 0x7f02a802a070 con 0x560c2b358890 2024-04-23T10:27:27.987 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:27:27.982+0000 7f02c780b740 10 client.15660 fetch_fsmap finished waiting for FSMap version 21 2024-04-23T10:27:27.987 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:27:27.982+0000 7f02c780b740 10 client.15660 resolve_mds: validated gid 24545 aka daemon mds.b 2024-04-23T10:27:27.987 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:27:27.982+0000 7f02c780b740 20 client.15660 populate_metadata read hostname 'smithi114' 2024-04-23T10:27:27.987 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:27:27.982+0000 7f02c780b740 1 --2- 172.21.15.114:0/3538053978 >> [v2:172.21.15.119:6838/937297340,v1:172.21.15.119:6839/937297340] conn(0x560c2b368a50 0x560c2b3d2710 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T10:27:27.987 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:27:27.982+0000 7f02c780b740 4 client.15660 mds_command: new command op to 24545 tid=0 multi_id=0 [{"prefix": "quiesce db", "set_id": "7f49b8f3", "release": true, "await": true}] 2024-04-23T10:27:27.988 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:27:27.982+0000 7f02c780b740 1 -- 172.21.15.114:0/3538053978 --> [v2:172.21.15.119:6838/937297340,v1:172.21.15.119:6839/937297340] -- command(tid 0: {"prefix": "quiesce db", "set_id": "7f49b8f3", "release": true, "await": true}) -- 0x7f02a8034070 con 0x560c2b368a50 2024-04-23T10:27:27.988 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:27:27.982+0000 7f02be59c640 1 --2- 172.21.15.114:0/3538053978 >> [v2:172.21.15.119:6838/937297340,v1:172.21.15.119:6839/937297340] conn(0x560c2b368a50 0x560c2b3d2710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T10:27:27.988 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:27:27.983+0000 7f02a5ffb640 10 client.15660 ms_handle_connect on v2:172.21.15.119:6838/937297340 2024-04-23T10:27:27.988 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:27:27.983+0000 7f02be59c640 1 --2- 172.21.15.114:0/3538053978 >> [v2:172.21.15.119:6838/937297340,v1:172.21.15.119:6839/937297340] conn(0x560c2b368a50 0x560c2b3d2710 crc :-1 s=READY pgs=49 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=17cbf626bfeb7656 server_cookie=ae80366a0ceab8d3 in_seq=0 out_seq=0 2024-04-23T10:27:27.989 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:27:27.985+0000 7f02a5ffb640 1 -- 172.21.15.114:0/3538053978 <== mds.0 v2:172.21.15.119:6838/937297340 1 ==== command_reply(tid 0: 0 ) ==== 8+0+588 (crc 0 0 0) 0x7f02a8034070 con 0x560c2b368a50 2024-04-23T10:27:27.989 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:27:27.985+0000 7f02a5ffb640 10 client.15660 handle_command_reply: tid=0 2024-04-23T10:27:27.989 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:27:27.985+0000 7f02c780b740 1 client.15660 shutdown 2024-04-23T10:27:27.989 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:27:27.985+0000 7f02c780b740 1 -- 172.21.15.114:0/3538053978 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x560c2b358890 msgr2=0x560c2b351fe0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T10:27:27.989 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:27:27.985+0000 7f02c780b740 1 --2- 172.21.15.114:0/3538053978 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x560c2b358890 0x560c2b351fe0 secure :-1 s=READY pgs=518 cs=0 l=1 rev1=1 crypto rx=0x7f02a801e8f0 tx=0x7f02a8014040 comp rx=0 tx=0).stop 2024-04-23T10:27:27.989 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:27:27.985+0000 7f02c780b740 1 -- 172.21.15.114:0/3538053978 shutdown_connections 2024-04-23T10:27:27.990 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:27:27.985+0000 7f02c780b740 1 --2- 172.21.15.114:0/3538053978 >> [v2:172.21.15.119:6838/937297340,v1:172.21.15.119:6839/937297340] conn(0x560c2b368a50 0x560c2b3d2710 crc :-1 s=READY pgs=49 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:27:27.990 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:27:27.985+0000 7f02c780b740 1 --2- 172.21.15.114:0/3538053978 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x560c2b358890 0x560c2b351fe0 unknown :-1 s=CLOSED pgs=518 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:27:27.990 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:27:27.985+0000 7f02c780b740 1 --2- 172.21.15.114:0/3538053978 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x560c2b34e1c0 0x560c2b3518a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:27:27.990 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:27:27.985+0000 7f02a5ffb640 0 client.15660 ms_handle_reset on v2:172.21.15.119:6838/937297340 2024-04-23T10:27:27.990 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:27:27.985+0000 7f02c780b740 1 --2- 172.21.15.114:0/3538053978 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x560c2b34d8a0 0x560c2b361050 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:27:27.990 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:27:27.985+0000 7f02c780b740 1 -- 172.21.15.114:0/3538053978 >> 172.21.15.114:0/3538053978 conn(0x560c2b349bc0 msgr2=0x560c2b34f120 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T10:27:27.990 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:27:27.985+0000 7f02c780b740 1 -- 172.21.15.114:0/3538053978 shutdown_connections 2024-04-23T10:27:27.990 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:27:27.985+0000 7f02c780b740 1 -- 172.21.15.114:0/3538053978 wait complete. 2024-04-23T10:27:27.990 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:27:27.985+0000 7f02c780b740 20 client.15660 trim_cache size 0 max 16384 2024-04-23T10:27:27.991 INFO:teuthology.orchestra.run.smithi114.stdout:{ 2024-04-23T10:27:27.991 INFO:teuthology.orchestra.run.smithi114.stdout: "epoch": 21, 2024-04-23T10:27:27.991 INFO:teuthology.orchestra.run.smithi114.stdout: "leader": 24545, 2024-04-23T10:27:27.991 INFO:teuthology.orchestra.run.smithi114.stdout: "set_version": 4, 2024-04-23T10:27:27.991 INFO:teuthology.orchestra.run.smithi114.stdout: "sets": { 2024-04-23T10:27:27.991 INFO:teuthology.orchestra.run.smithi114.stdout: "7f49b8f3": { 2024-04-23T10:27:27.991 INFO:teuthology.orchestra.run.smithi114.stdout: "version": 4, 2024-04-23T10:27:27.991 INFO:teuthology.orchestra.run.smithi114.stdout: "age_ref": 0.0, 2024-04-23T10:27:27.991 INFO:teuthology.orchestra.run.smithi114.stdout: "state": { 2024-04-23T10:27:27.991 INFO:teuthology.orchestra.run.smithi114.stdout: "name": "RELEASED", 2024-04-23T10:27:27.991 INFO:teuthology.orchestra.run.smithi114.stdout: "age": 0.0 2024-04-23T10:27:27.991 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-23T10:27:27.992 INFO:teuthology.orchestra.run.smithi114.stdout: "timeout": 60.0, 2024-04-23T10:27:27.992 INFO:teuthology.orchestra.run.smithi114.stdout: "expiration": 82.7, 2024-04-23T10:27:27.992 INFO:teuthology.orchestra.run.smithi114.stdout: "members": { 2024-04-23T10:27:27.992 INFO:teuthology.orchestra.run.smithi114.stdout: "file:/": { 2024-04-23T10:27:27.992 INFO:teuthology.orchestra.run.smithi114.stdout: "excluded": false, 2024-04-23T10:27:27.992 INFO:teuthology.orchestra.run.smithi114.stdout: "state": { 2024-04-23T10:27:27.992 INFO:teuthology.orchestra.run.smithi114.stdout: "name": "RELEASED", 2024-04-23T10:27:27.992 INFO:teuthology.orchestra.run.smithi114.stdout: "age": 0.0 2024-04-23T10:27:27.992 INFO:teuthology.orchestra.run.smithi114.stdout: } 2024-04-23T10:27:27.992 INFO:teuthology.orchestra.run.smithi114.stdout: } 2024-04-23T10:27:27.992 INFO:teuthology.orchestra.run.smithi114.stdout: } 2024-04-23T10:27:27.992 INFO:teuthology.orchestra.run.smithi114.stdout: } 2024-04-23T10:27:27.992 INFO:teuthology.orchestra.run.smithi114.stdout: } 2024-04-23T10:27:27.992 INFO:teuthology.orchestra.run.smithi114.stdout:} 2024-04-23T10:27:27.993 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:27:27.986+0000 7f02c4a20640 1 -- 172.21.15.114:0/834007175 >> [v2:172.21.15.114:6800/1395693466,v1:172.21.15.114:6801/1395693466] conn(0x7f02ac0a3ec0 msgr2=0x7f02ac0a6380 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T10:27:27.993 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:27:27.986+0000 7f02c4a20640 1 --2- 172.21.15.114:0/834007175 >> [v2:172.21.15.114:6800/1395693466,v1:172.21.15.114:6801/1395693466] conn(0x7f02ac0a3ec0 0x7f02ac0a6380 secure :-1 s=READY pgs=68 cs=0 l=1 rev1=1 crypto rx=0x7f02b80cf990 tx=0x7f02b8099d10 comp rx=0 tx=0).stop 2024-04-23T10:27:27.993 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:27:27.986+0000 7f02c4a20640 1 -- 172.21.15.114:0/834007175 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f02c019bec0 msgr2=0x7f02c019c370 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T10:27:27.994 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:27:27.986+0000 7f02c4a20640 1 --2- 172.21.15.114:0/834007175 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f02c019bec0 0x7f02c019c370 secure :-1 s=READY pgs=514 cs=0 l=1 rev1=1 crypto rx=0x7f02b800eca0 tx=0x7f02b800b690 comp rx=0 tx=0).stop 2024-04-23T10:27:27.994 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:27:27.986+0000 7f02c4a20640 1 -- 172.21.15.114:0/834007175 shutdown_connections 2024-04-23T10:27:27.994 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:27:27.986+0000 7f02c4a20640 1 --2- 172.21.15.114:0/834007175 >> [v2:172.21.15.114:6800/1395693466,v1:172.21.15.114:6801/1395693466] conn(0x7f02ac0a3ec0 0x7f02ac0a6380 unknown :-1 s=CLOSED pgs=68 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:27:27.994 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:27:27.986+0000 7f02c4a20640 1 --2- 172.21.15.114:0/834007175 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f02c019bec0 0x7f02c019c370 unknown :-1 s=CLOSED pgs=514 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:27:27.994 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:27:27.986+0000 7f02c4a20640 1 --2- 172.21.15.114:0/834007175 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f02c0106160 0x7f02c0197b40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:27:27.994 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:27:27.986+0000 7f02c4a20640 1 --2- 172.21.15.114:0/834007175 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f02c0101090 0x7f02c0197600 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:27:27.994 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:27:27.986+0000 7f02c4a20640 1 -- 172.21.15.114:0/834007175 >> 172.21.15.114:0/834007175 conn(0x7f02c0077f50 msgr2=0x7f02c00ff000 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T10:27:27.994 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:27:27.986+0000 7f02c4a20640 1 -- 172.21.15.114:0/834007175 shutdown_connections 2024-04-23T10:27:27.994 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:27:27.986+0000 7f02c4a20640 1 -- 172.21.15.114:0/834007175 wait complete. 2024-04-23T10:27:27.999 INFO:tasks.quiescer.fs.[cephfs]:Successfully released set '7f49b8f3', seconds elapsed: 23.8 2024-04-23T10:27:27.999 DEBUG:tasks.quiescer.fs.[cephfs]:Going to quiesce for duration: 22.7 2024-04-23T10:27:28.000 DEBUG:teuthology.orchestra.run.smithi114:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-23T10:27:28.094 INFO:journalctl@ceph.mon.a.smithi114.stdout:Apr 23 10:27:28 smithi114 ceph-mon[29255]: pgmap v26: 97 pgs: 97 active+clean; 1.3 MiB data, 377 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 85 B/s wr, 5 op/s 2024-04-23T10:27:28.094 INFO:journalctl@ceph.mon.a.smithi114.stdout:Apr 23 10:27:28 smithi114 ceph-mon[29255]: from='client.? 172.21.15.114:0/2063998827' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-23T10:27:28.122 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:27:28.123+0000 7f9a75b36640 1 -- 172.21.15.114:0/1619977994 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f9a701031e0 msgr2=0x7f9a701035c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T10:27:28.123 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:27:28.123+0000 7f9a75b36640 1 --2- 172.21.15.114:0/1619977994 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f9a701031e0 0x7f9a701035c0 secure :-1 s=READY pgs=519 cs=0 l=1 rev1=1 crypto rx=0x7f9a60009900 tx=0x7f9a6002a510 comp rx=0 tx=0).stop 2024-04-23T10:27:28.123 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:27:28.123+0000 7f9a75b36640 1 -- 172.21.15.114:0/1619977994 shutdown_connections 2024-04-23T10:27:28.123 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:27:28.123+0000 7f9a75b36640 1 --2- 172.21.15.114:0/1619977994 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f9a7010c520 0x7f9a7010e9c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:27:28.123 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:27:28.123+0000 7f9a75b36640 1 --2- 172.21.15.114:0/1619977994 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f9a70103b00 0x7f9a7010bfe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:27:28.123 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:27:28.123+0000 7f9a75b36640 1 --2- 172.21.15.114:0/1619977994 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f9a701031e0 0x7f9a701035c0 unknown :-1 s=CLOSED pgs=519 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:27:28.123 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:27:28.123+0000 7f9a75b36640 1 -- 172.21.15.114:0/1619977994 >> 172.21.15.114:0/1619977994 conn(0x7f9a70077f50 msgr2=0x7f9a70078360 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T10:27:28.123 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:27:28.123+0000 7f9a75b36640 1 -- 172.21.15.114:0/1619977994 shutdown_connections 2024-04-23T10:27:28.123 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:27:28.123+0000 7f9a75b36640 1 -- 172.21.15.114:0/1619977994 wait complete. 2024-04-23T10:27:28.126 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:27:28.124+0000 7f9a75b36640 1 Processor -- start 2024-04-23T10:27:28.126 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:27:28.124+0000 7f9a75b36640 1 -- start start 2024-04-23T10:27:28.126 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:27:28.124+0000 7f9a75b36640 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f9a701031e0 0x7f9a7019e450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T10:27:28.126 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:27:28.124+0000 7f9a75b36640 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f9a70103b00 0x7f9a7019e990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T10:27:28.126 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:27:28.124+0000 7f9a75b36640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f9a7010c520 0x7f9a70198600 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T10:27:28.126 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:27:28.125+0000 7f9a75b36640 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x7f9a70112200 con 0x7f9a7010c520 2024-04-23T10:27:28.126 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:27:28.125+0000 7f9a75b36640 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 -- 0x7f9a70112080 con 0x7f9a701031e0 2024-04-23T10:27:28.126 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:27:28.125+0000 7f9a75b36640 1 -- --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_getmap magic: 0 -- 0x7f9a70112380 con 0x7f9a70103b00 2024-04-23T10:27:28.126 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:27:28.125+0000 7f9a6ffff640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f9a7010c520 0x7f9a70198600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T10:27:28.127 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:27:28.125+0000 7f9a6ffff640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f9a7010c520 0x7f9a70198600 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.114:53564/0 (socket says 172.21.15.114:53564) 2024-04-23T10:27:28.127 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:27:28.125+0000 7f9a6ffff640 1 -- 172.21.15.114:0/3947971820 learned_addr learned my addr 172.21.15.114:0/3947971820 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-04-23T10:27:28.127 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:27:28.125+0000 7f9a6f7fe640 1 --2- 172.21.15.114:0/3947971820 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f9a701031e0 0x7f9a7019e450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T10:27:28.127 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:27:28.125+0000 7f9a6effd640 1 --2- 172.21.15.114:0/3947971820 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f9a70103b00 0x7f9a7019e990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T10:27:28.127 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:27:28.125+0000 7f9a6ffff640 1 -- 172.21.15.114:0/3947971820 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f9a70103b00 msgr2=0x7f9a7019e990 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T10:27:28.127 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:27:28.125+0000 7f9a6ffff640 1 --2- 172.21.15.114:0/3947971820 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f9a70103b00 0x7f9a7019e990 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:27:28.127 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:27:28.125+0000 7f9a6ffff640 1 -- 172.21.15.114:0/3947971820 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f9a701031e0 msgr2=0x7f9a7019e450 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T10:27:28.127 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:27:28.125+0000 7f9a6ffff640 1 --2- 172.21.15.114:0/3947971820 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f9a701031e0 0x7f9a7019e450 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:27:28.127 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:27:28.125+0000 7f9a6ffff640 1 -- 172.21.15.114:0/3947971820 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f9a70198d80 con 0x7f9a7010c520 2024-04-23T10:27:28.127 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:27:28.125+0000 7f9a6f7fe640 1 --2- 172.21.15.114:0/3947971820 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f9a701031e0 0x7f9a7019e450 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-23T10:27:28.127 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:27:28.125+0000 7f9a6ffff640 1 --2- 172.21.15.114:0/3947971820 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f9a7010c520 0x7f9a70198600 secure :-1 s=READY pgs=520 cs=0 l=1 rev1=1 crypto rx=0x7f9a6800eaa0 tx=0x7f9a6800ef60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T10:27:28.127 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:27:28.126+0000 7f9a6cff9640 1 -- 172.21.15.114:0/3947971820 <== mon.0 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9a6800cdb0 con 0x7f9a7010c520 2024-04-23T10:27:28.127 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:27:28.126+0000 7f9a75b36640 1 -- 172.21.15.114:0/3947971820 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f9a70199070 con 0x7f9a7010c520 2024-04-23T10:27:28.128 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:27:28.126+0000 7f9a6cff9640 1 -- 172.21.15.114:0/3947971820 <== mon.0 v2:172.21.15.114:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f9a68005020 con 0x7f9a7010c520 2024-04-23T10:27:28.128 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:27:28.126+0000 7f9a6cff9640 1 -- 172.21.15.114:0/3947971820 <== mon.0 v2:172.21.15.114:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9a680116d0 con 0x7f9a7010c520 2024-04-23T10:27:28.128 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:27:28.126+0000 7f9a75b36640 1 -- 172.21.15.114:0/3947971820 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f9a701a5310 con 0x7f9a7010c520 2024-04-23T10:27:28.128 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:27:28.127+0000 7f9a75b36640 1 -- 172.21.15.114:0/3947971820 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f9a70068d70 con 0x7f9a7010c520 2024-04-23T10:27:28.128 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:27:28.129+0000 7f9a6cff9640 1 -- 172.21.15.114:0/3947971820 <== mon.0 v2:172.21.15.114:3300/0 4 ==== mgrmap(e 25) ==== 138880+0+0 (secure 0 0 0) 0x7f9a680051d0 con 0x7f9a7010c520 2024-04-23T10:27:28.129 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:27:28.130+0000 7f9a6cff9640 1 --2- 172.21.15.114:0/3947971820 >> [v2:172.21.15.114:6800/1395693466,v1:172.21.15.114:6801/1395693466] conn(0x7f9a580a3ec0 0x7f9a580a6380 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T10:27:28.129 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:27:28.130+0000 7f9a6f7fe640 1 --2- 172.21.15.114:0/3947971820 >> [v2:172.21.15.114:6800/1395693466,v1:172.21.15.114:6801/1395693466] conn(0x7f9a580a3ec0 0x7f9a580a6380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T10:27:28.129 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:27:28.130+0000 7f9a6cff9640 1 -- 172.21.15.114:0/3947971820 <== mon.0 v2:172.21.15.114:3300/0 5 ==== osd_map(90..90 src has 1..90) ==== 8527+0+0 (secure 0 0 0) 0x7f9a680d0ad0 con 0x7f9a7010c520 2024-04-23T10:27:28.130 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:27:28.131+0000 7f9a6f7fe640 1 --2- 172.21.15.114:0/3947971820 >> [v2:172.21.15.114:6800/1395693466,v1:172.21.15.114:6801/1395693466] conn(0x7f9a580a3ec0 0x7f9a580a6380 secure :-1 s=READY pgs=69 cs=0 l=1 rev1=1 crypto rx=0x7f9a60005ab0 tx=0x7f9a60005a20 comp rx=0 tx=0).ready entity=mgr.15291 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T10:27:28.137 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:27:28.138+0000 7f9a6cff9640 1 -- 172.21.15.114:0/3947971820 <== mon.0 v2:172.21.15.114:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f9a68016030 con 0x7f9a7010c520 2024-04-23T10:27:28.380 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 23 10:27:28 smithi204 ceph-mon[32649]: pgmap v26: 97 pgs: 97 active+clean; 1.3 MiB data, 377 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 85 B/s wr, 5 op/s 2024-04-23T10:27:28.380 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 23 10:27:28 smithi204 ceph-mon[32649]: from='client.? 172.21.15.114:0/2063998827' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-23T10:27:28.424 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:27:28.425+0000 7f9a75b36640 1 -- 172.21.15.114:0/3947971820 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f9a70199b30 con 0x7f9a7010c520 2024-04-23T10:27:28.425 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:27:28.426+0000 7f9a6cff9640 1 -- 172.21.15.114:0/3947971820 <== mon.0 v2:172.21.15.114:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 21 v21) ==== 94+0+11884 (secure 0 0 0) 0x7f9a6809c8d0 con 0x7f9a7010c520 2024-04-23T10:27:28.426 INFO:teuthology.orchestra.run.smithi114.stdout: 2024-04-23T10:27:28.426 INFO:teuthology.orchestra.run.smithi114.stdout:{"epoch":21,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14547,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.114:6835/663493580","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6834","nonce":663493580},{"type":"v1","addr":"172.21.15.114:6835","nonce":663493580}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14565,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.114:6837/1439252154","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6836","nonce":1439252154},{"type":"v1","addr":"172.21.15.114:6837","nonce":1439252154}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5},{"gid":14634,"name":"g","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.114:6839/3849796939","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6838","nonce":3849796939},{"type":"v1","addr":"172.21.15.114:6839","nonce":3849796939}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":11},{"gid":14646,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.114:6841/3544523891","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6840","nonce":3544523891},{"type":"v1","addr":"172.21.15.114:6841","nonce":3544523891}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":12},{"gid":24385,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.204:6833/3866561754","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6832","nonce":3866561754},{"type":"v1","addr":"172.21.15.204:6833","nonce":3866561754}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2},{"gid":24427,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.204:6835/2533793172","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6834","nonce":2533793172},{"type":"v1","addr":"172.21.15.204:6835","nonce":2533793172}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":7}],"filesystems":[{"mdsmap":{"epoch":21,"flags":306,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":true},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-04-23T10:24:43.383158+0000","modified":"2024-04-23T10:26:38.945346+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":3,"in":[0,1,2],"up":{"mds_0":24545,"mds_1":24509,"mds_2":24479},"failed":[],"damaged":[],"stopped":[],"info":{"gid_24439":{"gid":24439,"name":"c","rank":2,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.204:6837/513822179","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6836","nonce":513822179},{"type":"v1","addr":"172.21.15.204:6837","nonce":513822179}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24443":{"gid":24443,"name":"k","rank":1,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.119:6833/783244660","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6832","nonce":783244660},{"type":"v1","addr":"172.21.15.119:6833","nonce":783244660}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24463":{"gid":24463,"name":"i","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.204:6839/3222658893","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6838","nonce":3222658893},{"type":"v1","addr":"172.21.15.204:6839","nonce":3222658893}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24479":{"gid":24479,"name":"e","rank":2,"incarnation":19,"state":"up:active","state_seq":9,"addr":"172.21.15.119:6835/2033916709","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6834","nonce":2033916709},{"type":"v1","addr":"172.21.15.119:6835","nonce":2033916709}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24509":{"gid":24509,"name":"h","rank":1,"incarnation":17,"state":"up:active","state_seq":6,"addr":"172.21.15.119:6837/1872363128","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6836","nonce":1872363128},{"type":"v1","addr":"172.21.15.119:6837","nonce":1872363128}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24545":{"gid":24545,"name":"b","rank":0,"incarnation":14,"state":"up:active","state_seq":3,"addr":"172.21.15.119:6839/937297340","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6838","nonce":937297340},{"type":"v1","addr":"172.21.15.119:6839","nonce":937297340}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":24545,"qdb_cluster":[24479,24509,24545]},"id":1}]} 2024-04-23T10:27:28.426 INFO:teuthology.orchestra.run.smithi114.stderr:dumped fsmap epoch 21 2024-04-23T10:27:28.429 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:27:28.429+0000 7f9a75b36640 1 -- 172.21.15.114:0/3947971820 >> [v2:172.21.15.114:6800/1395693466,v1:172.21.15.114:6801/1395693466] conn(0x7f9a580a3ec0 msgr2=0x7f9a580a6380 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T10:27:28.429 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:27:28.429+0000 7f9a75b36640 1 --2- 172.21.15.114:0/3947971820 >> [v2:172.21.15.114:6800/1395693466,v1:172.21.15.114:6801/1395693466] conn(0x7f9a580a3ec0 0x7f9a580a6380 secure :-1 s=READY pgs=69 cs=0 l=1 rev1=1 crypto rx=0x7f9a60005ab0 tx=0x7f9a60005a20 comp rx=0 tx=0).stop 2024-04-23T10:27:28.429 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:27:28.429+0000 7f9a75b36640 1 -- 172.21.15.114:0/3947971820 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f9a7010c520 msgr2=0x7f9a70198600 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T10:27:28.429 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:27:28.429+0000 7f9a75b36640 1 --2- 172.21.15.114:0/3947971820 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f9a7010c520 0x7f9a70198600 secure :-1 s=READY pgs=520 cs=0 l=1 rev1=1 crypto rx=0x7f9a6800eaa0 tx=0x7f9a6800ef60 comp rx=0 tx=0).stop 2024-04-23T10:27:28.429 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:27:28.429+0000 7f9a75b36640 1 -- 172.21.15.114:0/3947971820 shutdown_connections 2024-04-23T10:27:28.429 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:27:28.429+0000 7f9a75b36640 1 --2- 172.21.15.114:0/3947971820 >> [v2:172.21.15.114:6800/1395693466,v1:172.21.15.114:6801/1395693466] conn(0x7f9a580a3ec0 0x7f9a580a6380 unknown :-1 s=CLOSED pgs=69 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:27:28.429 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:27:28.429+0000 7f9a75b36640 1 --2- 172.21.15.114:0/3947971820 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f9a7010c520 0x7f9a70198600 unknown :-1 s=CLOSED pgs=520 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:27:28.429 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:27:28.429+0000 7f9a75b36640 1 --2- 172.21.15.114:0/3947971820 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f9a70103b00 0x7f9a7019e990 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:27:28.429 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:27:28.429+0000 7f9a75b36640 1 --2- 172.21.15.114:0/3947971820 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f9a701031e0 0x7f9a7019e450 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:27:28.429 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:27:28.429+0000 7f9a75b36640 1 -- 172.21.15.114:0/3947971820 >> 172.21.15.114:0/3947971820 conn(0x7f9a70077f50 msgr2=0x7f9a700ff330 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T10:27:28.430 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:27:28.429+0000 7f9a75b36640 1 -- 172.21.15.114:0/3947971820 shutdown_connections 2024-04-23T10:27:28.430 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:27:28.429+0000 7f9a75b36640 1 -- 172.21.15.114:0/3947971820 wait complete. 2024-04-23T10:27:28.439 DEBUG:tasks.quiescer.fs.[cephfs]:Running ceph command: 'tell mds.24545 quiesce db / --timeout 60 --expiration 82.7 --await-for 42.0' 2024-04-23T10:27:28.440 DEBUG:teuthology.orchestra.run.smithi114:> timeout 75 sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.24545 quiesce db / --timeout 60 --expiration 82.7 --await-for 42.0 2024-04-23T10:27:28.492 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 23 10:27:28 smithi119 ceph-mon[32608]: pgmap v26: 97 pgs: 97 active+clean; 1.3 MiB data, 377 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 85 B/s wr, 5 op/s 2024-04-23T10:27:28.492 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 23 10:27:28 smithi119 ceph-mon[32608]: from='client.? 172.21.15.114:0/2063998827' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-23T10:27:28.552 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:27:28.550+0000 7f299d930640 1 -- 172.21.15.114:0/555314161 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f2998102b60 msgr2=0x7f2998106ba0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T10:27:28.553 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:27:28.550+0000 7f299d930640 1 --2- 172.21.15.114:0/555314161 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f2998102b60 0x7f2998106ba0 secure :-1 s=READY pgs=521 cs=0 l=1 rev1=1 crypto rx=0x7f298c009900 tx=0x7f298c02a410 comp rx=0 tx=0).stop 2024-04-23T10:27:28.553 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:27:28.551+0000 7f299d930640 1 -- 172.21.15.114:0/555314161 shutdown_connections 2024-04-23T10:27:28.553 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:27:28.551+0000 7f299d930640 1 --2- 172.21.15.114:0/555314161 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f2998107280 0x7f299810ee50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:27:28.553 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:27:28.551+0000 7f299d930640 1 --2- 172.21.15.114:0/555314161 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f2998102b60 0x7f2998106ba0 unknown :-1 s=CLOSED pgs=521 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:27:28.553 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:27:28.551+0000 7f299d930640 1 --2- 172.21.15.114:0/555314161 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f29981021b0 0x7f2998102590 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:27:28.553 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:27:28.551+0000 7f299d930640 1 -- 172.21.15.114:0/555314161 >> 172.21.15.114:0/555314161 conn(0x7f29980fd9f0 msgr2=0x7f29980ffe10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T10:27:28.553 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:27:28.551+0000 7f299d930640 1 -- 172.21.15.114:0/555314161 shutdown_connections 2024-04-23T10:27:28.553 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:27:28.551+0000 7f299d930640 1 -- 172.21.15.114:0/555314161 wait complete. 2024-04-23T10:27:28.553 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:27:28.551+0000 7f299d930640 1 Processor -- start 2024-04-23T10:27:28.554 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:27:28.551+0000 7f299d930640 1 -- start start 2024-04-23T10:27:28.554 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:27:28.552+0000 7f299d930640 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f29981021b0 0x7f299819e3e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T10:27:28.554 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:27:28.552+0000 7f299d930640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f2998107280 0x7f299819e920 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T10:27:28.554 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:27:28.552+0000 7f299d930640 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f29981985e0 0x7f2998198a90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T10:27:28.554 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:27:28.552+0000 7f299d930640 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x7f29981121c0 con 0x7f2998107280 2024-04-23T10:27:28.554 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:27:28.552+0000 7f299d930640 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 -- 0x7f2998112040 con 0x7f29981021b0 2024-04-23T10:27:28.554 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:27:28.552+0000 7f299d930640 1 -- --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_getmap magic: 0 -- 0x7f2998112340 con 0x7f29981985e0 2024-04-23T10:27:28.554 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:27:28.552+0000 7f29967fc640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f2998107280 0x7f299819e920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T10:27:28.554 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:27:28.552+0000 7f29967fc640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f2998107280 0x7f299819e920 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.114:53580/0 (socket says 172.21.15.114:53580) 2024-04-23T10:27:28.555 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:27:28.552+0000 7f2996ffd640 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f29981021b0 0x7f299819e3e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T10:27:28.555 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:27:28.552+0000 7f29967fc640 1 -- 172.21.15.114:0/446225035 learned_addr learned my addr 172.21.15.114:0/446225035 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-04-23T10:27:28.555 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:27:28.552+0000 7f29977fe640 1 --2- 172.21.15.114:0/446225035 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f29981985e0 0x7f2998198a90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T10:27:28.555 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:27:28.552+0000 7f2996ffd640 1 -- 172.21.15.114:0/446225035 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f29981985e0 msgr2=0x7f2998198a90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T10:27:28.555 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:27:28.552+0000 7f2996ffd640 1 --2- 172.21.15.114:0/446225035 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f29981985e0 0x7f2998198a90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:27:28.555 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:27:28.552+0000 7f2996ffd640 1 -- 172.21.15.114:0/446225035 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f2998107280 msgr2=0x7f299819e920 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T10:27:28.555 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:27:28.552+0000 7f2996ffd640 1 --2- 172.21.15.114:0/446225035 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f2998107280 0x7f299819e920 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:27:28.555 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:27:28.552+0000 7f2996ffd640 1 -- 172.21.15.114:0/446225035 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f2998199100 con 0x7f29981021b0 2024-04-23T10:27:28.555 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:27:28.552+0000 7f29967fc640 1 --2- 172.21.15.114:0/446225035 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f2998107280 0x7f299819e920 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-23T10:27:28.555 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:27:28.552+0000 7f2996ffd640 1 --2- 172.21.15.114:0/446225035 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f29981021b0 0x7f299819e3e0 secure :-1 s=READY pgs=59 cs=0 l=1 rev1=1 crypto rx=0x7f298800e9f0 tx=0x7f298800eeb0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T10:27:28.556 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:27:28.553+0000 7f299c92e640 1 -- 172.21.15.114:0/446225035 <== mon.2 v2:172.21.15.119:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f298800cd80 con 0x7f29981021b0 2024-04-23T10:27:28.556 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:27:28.553+0000 7f299d930640 1 -- 172.21.15.114:0/446225035 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f29981993f0 con 0x7f29981021b0 2024-04-23T10:27:28.556 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:27:28.553+0000 7f299c92e640 1 -- 172.21.15.114:0/446225035 <== mon.2 v2:172.21.15.119:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f29880035f0 con 0x7f29981021b0 2024-04-23T10:27:28.556 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:27:28.553+0000 7f299d930640 1 -- 172.21.15.114:0/446225035 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f29981a5640 con 0x7f29981021b0 2024-04-23T10:27:28.556 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:27:28.553+0000 7f299c92e640 1 -- 172.21.15.114:0/446225035 <== mon.2 v2:172.21.15.119:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2988010690 con 0x7f29981021b0 2024-04-23T10:27:28.563 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:27:28.555+0000 7f29a064a740 1 Processor -- start 2024-04-23T10:27:28.564 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:27:28.555+0000 7f29a064a740 1 -- start start 2024-04-23T10:27:28.564 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:27:28.556+0000 7f29a064a740 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x55f6587d30a0 0x55f6587d3480 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T10:27:28.564 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:27:28.556+0000 7f29a064a740 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x55f6587d39c0 0x55f6587ddb50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T10:27:28.564 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:27:28.556+0000 7f29a064a740 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55f6587de090 0x55f6587e0480 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T10:27:28.564 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:27:28.556+0000 7f29a064a740 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x55f658738370 con 0x55f6587de090 2024-04-23T10:27:28.564 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:27:28.556+0000 7f29a064a740 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 -- 0x55f6586d52e0 con 0x55f6587d30a0 2024-04-23T10:27:28.564 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:27:28.556+0000 7f29a064a740 1 -- --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_getmap magic: 0 -- 0x55f658734330 con 0x55f6587d39c0 2024-04-23T10:27:28.564 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:27:28.556+0000 7f29967fc640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55f6587de090 0x55f6587e0480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T10:27:28.564 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:27:28.556+0000 7f29977fe640 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x55f6587d30a0 0x55f6587d3480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T10:27:28.564 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:27:28.556+0000 7f29977fe640 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x55f6587d30a0 0x55f6587d3480 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.119:3300/0 says I am v2:172.21.15.114:57970/0 (socket says 172.21.15.114:57970) 2024-04-23T10:27:28.565 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:27:28.556+0000 7f29967fc640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55f6587de090 0x55f6587e0480 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.114:53590/0 (socket says 172.21.15.114:53590) 2024-04-23T10:27:28.565 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:27:28.556+0000 7f29977fe640 1 -- 172.21.15.114:0/1938322455 learned_addr learned my addr 172.21.15.114:0/1938322455 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-04-23T10:27:28.565 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:27:28.556+0000 7f299c92e640 1 -- 172.21.15.114:0/446225035 <== mon.2 v2:172.21.15.119:3300/0 4 ==== mgrmap(e 25) ==== 138880+0+0 (secure 0 0 0) 0x7f29880037a0 con 0x7f29981021b0 2024-04-23T10:27:28.565 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:27:28.556+0000 7f29967fc640 1 -- 172.21.15.114:0/1938322455 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x55f6587d39c0 msgr2=0x55f6587ddb50 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-04-23T10:27:28.565 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:27:28.556+0000 7f29967fc640 1 --2- 172.21.15.114:0/1938322455 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x55f6587d39c0 0x55f6587ddb50 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:27:28.565 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:27:28.556+0000 7f29967fc640 1 -- 172.21.15.114:0/1938322455 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x55f6587d30a0 msgr2=0x55f6587d3480 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T10:27:28.565 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:27:28.556+0000 7f29967fc640 1 --2- 172.21.15.114:0/1938322455 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x55f6587d30a0 0x55f6587d3480 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:27:28.565 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:27:28.556+0000 7f29967fc640 1 -- 172.21.15.114:0/1938322455 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55f658516d80 con 0x55f6587de090 2024-04-23T10:27:28.565 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:27:28.556+0000 7f29967fc640 1 --2- 172.21.15.114:0/1938322455 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55f6587de090 0x55f6587e0480 secure :-1 s=READY pgs=522 cs=0 l=1 rev1=1 crypto rx=0x7f298c004770 tx=0x7f298c005e40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=87600349c371e626 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T10:27:28.565 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:27:28.557+0000 7f297dffb640 1 -- 172.21.15.114:0/1938322455 <== mon.0 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f298c035040 con 0x55f6587de090 2024-04-23T10:27:28.566 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:27:28.557+0000 7f299c92e640 1 --2- 172.21.15.114:0/446225035 >> [v2:172.21.15.114:6800/1395693466,v1:172.21.15.114:6801/1395693466] conn(0x7f29800a3e40 0x7f29800a6300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T10:27:28.566 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:27:28.557+0000 7f297dffb640 1 -- 172.21.15.114:0/1938322455 <== mon.0 v2:172.21.15.114:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f298c031860 con 0x55f6587de090 2024-04-23T10:27:28.566 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:27:28.557+0000 7f29977fe640 1 --2- 172.21.15.114:0/446225035 >> [v2:172.21.15.114:6800/1395693466,v1:172.21.15.114:6801/1395693466] conn(0x7f29800a3e40 0x7f29800a6300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T10:27:28.566 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:27:28.557+0000 7f299c92e640 1 -- 172.21.15.114:0/446225035 <== mon.2 v2:172.21.15.119:3300/0 5 ==== osd_map(90..90 src has 1..90) ==== 8527+0+0 (secure 0 0 0) 0x7f29880d0cc0 con 0x7f29981021b0 2024-04-23T10:27:28.566 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:27:28.557+0000 7f297dffb640 1 -- 172.21.15.114:0/1938322455 <== mon.0 v2:172.21.15.114:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f298c004a00 con 0x55f6587de090 2024-04-23T10:27:28.566 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:27:28.558+0000 7f29977fe640 1 --2- 172.21.15.114:0/446225035 >> [v2:172.21.15.114:6800/1395693466,v1:172.21.15.114:6801/1395693466] conn(0x7f29800a3e40 0x7f29800a6300 secure :-1 s=READY pgs=70 cs=0 l=1 rev1=1 crypto rx=0x7f2990005f40 tx=0x7f2990009600 comp rx=0 tx=0).ready entity=mgr.15291 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T10:27:28.566 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:27:28.558+0000 7f29a064a740 1 -- 172.21.15.114:0/1938322455 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55f6587de090 msgr2=0x55f6587e0480 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T10:27:28.566 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:27:28.558+0000 7f29a064a740 1 --2- 172.21.15.114:0/1938322455 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55f6587de090 0x55f6587e0480 secure :-1 s=READY pgs=522 cs=0 l=1 rev1=1 crypto rx=0x7f298c004770 tx=0x7f298c005e40 comp rx=0 tx=0).stop 2024-04-23T10:27:28.566 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:27:28.558+0000 7f29a064a740 1 -- 172.21.15.114:0/1938322455 shutdown_connections 2024-04-23T10:27:28.566 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:27:28.558+0000 7f29a064a740 1 --2- 172.21.15.114:0/1938322455 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55f6587de090 0x55f6587e0480 unknown :-1 s=CLOSED pgs=522 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:27:28.567 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:27:28.558+0000 7f29a064a740 1 --2- 172.21.15.114:0/1938322455 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x55f6587d39c0 0x55f6587ddb50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:27:28.567 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:27:28.558+0000 7f29a064a740 1 --2- 172.21.15.114:0/1938322455 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x55f6587d30a0 0x55f6587d3480 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:27:28.567 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:27:28.558+0000 7f29a064a740 1 -- 172.21.15.114:0/1938322455 >> 172.21.15.114:0/1938322455 conn(0x55f6587cf3c0 msgr2=0x55f6587cf7a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T10:27:28.567 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:27:28.558+0000 7f29a064a740 1 -- 172.21.15.114:0/1938322455 shutdown_connections 2024-04-23T10:27:28.567 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:27:28.558+0000 7f29a064a740 1 -- 172.21.15.114:0/1938322455 wait complete. 2024-04-23T10:27:28.567 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:27:28.558+0000 7f29a064a740 1 -- start start 2024-04-23T10:27:28.567 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:27:28.559+0000 7f29a064a740 1 Processor -- start 2024-04-23T10:27:28.567 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:27:28.559+0000 7f29a064a740 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x55f6587d30a0 0x55f6587db030 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T10:27:28.567 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:27:28.559+0000 7f29a064a740 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x55f6587d39c0 0x55f6587db570 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T10:27:28.567 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:27:28.559+0000 7f29a064a740 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55f6587de090 0x55f6587dbab0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T10:27:28.567 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:27:28.559+0000 7f29a064a740 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x55f6587e52d0 con 0x55f6587de090 2024-04-23T10:27:28.567 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:27:28.559+0000 7f29977fe640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55f6587de090 0x55f6587dbab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T10:27:28.568 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:27:28.559+0000 7f29967fc640 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x55f6587d39c0 0x55f6587db570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T10:27:28.568 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:27:28.559+0000 7f29977fe640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55f6587de090 0x55f6587dbab0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.114:53606/0 (socket says 172.21.15.114:53606) 2024-04-23T10:27:28.568 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:27:28.559+0000 7f29977fe640 1 -- 172.21.15.114:0/3997168935 learned_addr learned my addr 172.21.15.114:0/3997168935 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-04-23T10:27:28.568 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:27:28.559+0000 7f29a064a740 1 -- 172.21.15.114:0/3997168935 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 -- 0x55f6586d2fb0 con 0x55f6587d39c0 2024-04-23T10:27:28.568 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:27:28.559+0000 7f2996ffd640 1 --2- 172.21.15.114:0/3997168935 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x55f6587d30a0 0x55f6587db030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T10:27:28.568 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:27:28.559+0000 7f29a064a740 1 -- 172.21.15.114:0/3997168935 --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_getmap magic: 0 -- 0x55f6587e5450 con 0x55f6587d30a0 2024-04-23T10:27:28.568 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:27:28.559+0000 7f29a064a740 10 client.0 _ll_register_callbacks cb 0x55f6586d29a0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-23T10:27:28.568 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:27:28.560+0000 7f29967fc640 1 -- 172.21.15.114:0/3997168935 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x55f6587d30a0 msgr2=0x55f6587db030 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T10:27:28.568 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:27:28.560+0000 7f29967fc640 1 --2- 172.21.15.114:0/3997168935 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x55f6587d30a0 0x55f6587db030 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:27:28.568 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:27:28.560+0000 7f29967fc640 1 -- 172.21.15.114:0/3997168935 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55f6587de090 msgr2=0x55f6587dbab0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T10:27:28.568 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:27:28.560+0000 7f29967fc640 1 --2- 172.21.15.114:0/3997168935 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55f6587de090 0x55f6587dbab0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:27:28.569 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:27:28.560+0000 7f29967fc640 1 -- 172.21.15.114:0/3997168935 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55f658577510 con 0x55f6587d39c0 2024-04-23T10:27:28.569 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:27:28.560+0000 7f29977fe640 1 --2- 172.21.15.114:0/3997168935 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55f6587de090 0x55f6587dbab0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-23T10:27:28.569 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:27:28.560+0000 7f29967fc640 1 --2- 172.21.15.114:0/3997168935 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x55f6587d39c0 0x55f6587db570 secure :-1 s=READY pgs=60 cs=0 l=1 rev1=1 crypto rx=0x55f6587d0550 tx=0x7f298c004270 comp rx=0 tx=0).ready entity=mon.2 client_cookie=9f28b9f0d8ee8177 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T10:27:28.569 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:27:28.560+0000 7f2976ffd640 10 client.0 ms_handle_connect on v2:172.21.15.119:3300/0 2024-04-23T10:27:28.569 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:27:28.560+0000 7f2976ffd640 1 -- 172.21.15.114:0/3997168935 <== mon.2 v2:172.21.15.119:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f298c04e020 con 0x55f6587d39c0 2024-04-23T10:27:28.569 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:27:28.560+0000 7f2976ffd640 1 -- 172.21.15.114:0/3997168935 <== mon.2 v2:172.21.15.119:3300/0 2 ==== osd_map(90..90 src has 1..90) ==== 8527+0+0 (secure 0 0 0) 0x7f298c0035f0 con 0x55f6587d39c0 2024-04-23T10:27:28.569 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:27:28.560+0000 7f29a064a740 1 -- 172.21.15.114:0/3997168935 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f298c031860 con 0x55f6587d39c0 2024-04-23T10:27:28.569 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:27:28.560+0000 7f2976ffd640 1 -- 172.21.15.114:0/3997168935 <== mon.2 v2:172.21.15.119:3300/0 3 ==== mon_get_version_reply(handle=1 version=21) ==== 24+0+0 (secure 0 0 0) 0x7f298c059050 con 0x55f6587d39c0 2024-04-23T10:27:28.569 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:27:28.560+0000 7f29a064a740 10 client.25211 fetch_fsmap learned FSMap version 21 2024-04-23T10:27:28.569 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:27:28.560+0000 7f29a064a740 1 -- 172.21.15.114:0/3997168935 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({config=0+,fsmap=21,monmap=4+}) -- 0x55f65840a670 con 0x55f6587d39c0 2024-04-23T10:27:28.569 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:27:28.561+0000 7f2976ffd640 1 -- 172.21.15.114:0/3997168935 <== mon.2 v2:172.21.15.119:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f298c031860 con 0x55f6587d39c0 2024-04-23T10:27:28.570 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:27:28.561+0000 7f2976ffd640 1 -- 172.21.15.114:0/3997168935 <== mon.2 v2:172.21.15.119:3300/0 5 ==== fsmap(e 21) ==== 8338+0+0 (secure 0 0 0) 0x7f298c0467b0 con 0x55f6587d39c0 2024-04-23T10:27:28.570 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:27:28.561+0000 7f29a064a740 10 client.25211 fetch_fsmap finished waiting for FSMap version 21 2024-04-23T10:27:28.570 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:27:28.561+0000 7f29a064a740 10 client.25211 resolve_mds: validated gid 24545 aka daemon mds.b 2024-04-23T10:27:28.570 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:27:28.561+0000 7f29a064a740 20 client.25211 populate_metadata read hostname 'smithi114' 2024-04-23T10:27:28.570 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:27:28.561+0000 7f29a064a740 1 --2- 172.21.15.114:0/3997168935 >> [v2:172.21.15.119:6838/937297340,v1:172.21.15.119:6839/937297340] conn(0x55f6587f2740 0x55f6587f4b30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T10:27:28.570 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:27:28.561+0000 7f29a064a740 4 client.25211 mds_command: new command op to 24545 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-23T10:27:28.570 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:27:28.561+0000 7f29a064a740 1 -- 172.21.15.114:0/3997168935 --> [v2:172.21.15.119:6838/937297340,v1:172.21.15.119:6839/937297340] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55f658745d90 con 0x55f6587f2740 2024-04-23T10:27:28.570 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:27:28.561+0000 7f2996ffd640 1 --2- 172.21.15.114:0/3997168935 >> [v2:172.21.15.119:6838/937297340,v1:172.21.15.119:6839/937297340] conn(0x55f6587f2740 0x55f6587f4b30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T10:27:28.570 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:27:28.562+0000 7f2976ffd640 10 client.25211 ms_handle_connect on v2:172.21.15.119:6838/937297340 2024-04-23T10:27:28.570 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:27:28.562+0000 7f2996ffd640 1 --2- 172.21.15.114:0/3997168935 >> [v2:172.21.15.119:6838/937297340,v1:172.21.15.119:6839/937297340] conn(0x55f6587f2740 0x55f6587f4b30 crc :-1 s=READY pgs=50 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=29906c27eb18697f server_cookie=3bbc9b01c7d79527 in_seq=0 out_seq=0 2024-04-23T10:27:28.571 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:27:28.562+0000 7f2976ffd640 1 -- 172.21.15.114:0/3997168935 <== mds.0 v2:172.21.15.119:6838/937297340 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x55f658745d90 con 0x55f6587f2740 2024-04-23T10:27:28.571 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:27:28.562+0000 7f2976ffd640 10 client.25211 handle_command_reply: tid=0 2024-04-23T10:27:28.571 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:27:28.562+0000 7f29a064a740 1 client.25211 shutdown 2024-04-23T10:27:28.571 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:27:28.563+0000 7f29a064a740 1 -- 172.21.15.114:0/3997168935 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x55f6587d39c0 msgr2=0x55f6587db570 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T10:27:28.571 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:27:28.563+0000 7f29a064a740 1 --2- 172.21.15.114:0/3997168935 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x55f6587d39c0 0x55f6587db570 secure :-1 s=READY pgs=60 cs=0 l=1 rev1=1 crypto rx=0x55f6587d0550 tx=0x7f298c004270 comp rx=0 tx=0).stop 2024-04-23T10:27:28.571 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:27:28.563+0000 7f29a064a740 1 -- 172.21.15.114:0/3997168935 shutdown_connections 2024-04-23T10:27:28.571 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:27:28.563+0000 7f29a064a740 1 --2- 172.21.15.114:0/3997168935 >> [v2:172.21.15.119:6838/937297340,v1:172.21.15.119:6839/937297340] conn(0x55f6587f2740 0x55f6587f4b30 crc :-1 s=READY pgs=50 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:27:28.571 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:27:28.563+0000 7f2976ffd640 0 client.25211 ms_handle_reset on v2:172.21.15.119:6838/937297340 2024-04-23T10:27:28.571 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:27:28.563+0000 7f29a064a740 1 --2- 172.21.15.114:0/3997168935 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55f6587de090 0x55f6587dbab0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:27:28.571 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:27:28.563+0000 7f29a064a740 1 --2- 172.21.15.114:0/3997168935 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x55f6587d39c0 0x55f6587db570 unknown :-1 s=CLOSED pgs=60 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:27:28.572 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:27:28.563+0000 7f29a064a740 1 --2- 172.21.15.114:0/3997168935 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x55f6587d30a0 0x55f6587db030 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:27:28.572 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:27:28.563+0000 7f29a064a740 1 -- 172.21.15.114:0/3997168935 >> 172.21.15.114:0/3997168935 conn(0x55f6587cf3c0 msgr2=0x55f6587d51c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T10:27:28.572 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:27:28.563+0000 7f29a064a740 1 -- 172.21.15.114:0/3997168935 shutdown_connections 2024-04-23T10:27:28.572 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:27:28.563+0000 7f29a064a740 1 -- 172.21.15.114:0/3997168935 wait complete. 2024-04-23T10:27:28.572 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:27:28.563+0000 7f29a064a740 20 client.25211 trim_cache size 0 max 16384 2024-04-23T10:27:28.586 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:27:28.580+0000 7f29a064a740 1 Processor -- start 2024-04-23T10:27:28.586 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:27:28.580+0000 7f29a064a740 1 -- start start 2024-04-23T10:27:28.586 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:27:28.580+0000 7f29a064a740 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55f6587d30a0 0x55f6587d5910 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T10:27:28.586 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:27:28.580+0000 7f29a064a740 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x55f6587d39c0 0x55f6587ce480 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T10:27:28.586 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:27:28.580+0000 7f29a064a740 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x55f6587de090 0x55f6587fa1c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T10:27:28.586 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:27:28.580+0000 7f29a064a740 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x7f29981ae390 con 0x55f6587d30a0 2024-04-23T10:27:28.586 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:27:28.580+0000 7f29a064a740 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 -- 0x7f29981abc60 con 0x55f6587d39c0 2024-04-23T10:27:28.587 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:27:28.580+0000 7f29a064a740 1 -- --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_getmap magic: 0 -- 0x7f29981a9ad0 con 0x55f6587de090 2024-04-23T10:27:28.587 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:27:28.580+0000 7f2996ffd640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55f6587d30a0 0x55f6587d5910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T10:27:28.587 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:27:28.580+0000 7f2996ffd640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55f6587d30a0 0x55f6587d5910 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.114:53618/0 (socket says 172.21.15.114:53618) 2024-04-23T10:27:28.587 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:27:28.580+0000 7f2996ffd640 1 -- 172.21.15.114:0/3385477381 learned_addr learned my addr 172.21.15.114:0/3385477381 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-04-23T10:27:28.587 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:27:28.581+0000 7f29967fc640 1 --2- 172.21.15.114:0/3385477381 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x55f6587d39c0 0x55f6587ce480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T10:27:28.587 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:27:28.581+0000 7f29977fe640 1 --2- 172.21.15.114:0/3385477381 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x55f6587de090 0x55f6587fa1c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T10:27:28.587 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:27:28.581+0000 7f2996ffd640 1 -- 172.21.15.114:0/3385477381 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x55f6587de090 msgr2=0x55f6587fa1c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T10:27:28.587 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:27:28.581+0000 7f2996ffd640 1 --2- 172.21.15.114:0/3385477381 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x55f6587de090 0x55f6587fa1c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:27:28.587 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:27:28.581+0000 7f2996ffd640 1 -- 172.21.15.114:0/3385477381 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x55f6587d39c0 msgr2=0x55f6587ce480 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T10:27:28.587 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:27:28.581+0000 7f2996ffd640 1 --2- 172.21.15.114:0/3385477381 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x55f6587d39c0 0x55f6587ce480 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:27:28.587 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:27:28.581+0000 7f2996ffd640 1 -- 172.21.15.114:0/3385477381 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55f6587d0580 con 0x55f6587d30a0 2024-04-23T10:27:28.587 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:27:28.581+0000 7f2996ffd640 1 --2- 172.21.15.114:0/3385477381 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55f6587d30a0 0x55f6587d5910 secure :-1 s=READY pgs=523 cs=0 l=1 rev1=1 crypto rx=0x7f2988098e70 tx=0x7f2988087040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2ff3b80793a395f server_cookie=0 in_seq=0 out_seq=0 2024-04-23T10:27:28.588 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:27:28.581+0000 7f297d7fa640 1 -- 172.21.15.114:0/3385477381 <== mon.0 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f29880b5020 con 0x55f6587d30a0 2024-04-23T10:27:28.588 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:27:28.581+0000 7f297d7fa640 1 -- 172.21.15.114:0/3385477381 <== mon.0 v2:172.21.15.114:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f29880a5680 con 0x55f6587d30a0 2024-04-23T10:27:28.588 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:27:28.581+0000 7f297d7fa640 1 -- 172.21.15.114:0/3385477381 <== mon.0 v2:172.21.15.114:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f29880bbbd0 con 0x55f6587d30a0 2024-04-23T10:27:28.588 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:27:28.582+0000 7f29a064a740 1 -- 172.21.15.114:0/3385477381 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55f6587d30a0 msgr2=0x55f6587d5910 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T10:27:28.588 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:27:28.582+0000 7f29a064a740 1 --2- 172.21.15.114:0/3385477381 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55f6587d30a0 0x55f6587d5910 secure :-1 s=READY pgs=523 cs=0 l=1 rev1=1 crypto rx=0x7f2988098e70 tx=0x7f2988087040 comp rx=0 tx=0).stop 2024-04-23T10:27:28.588 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:27:28.582+0000 7f29a064a740 1 -- 172.21.15.114:0/3385477381 shutdown_connections 2024-04-23T10:27:28.588 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:27:28.582+0000 7f29a064a740 1 --2- 172.21.15.114:0/3385477381 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x55f6587de090 0x55f6587fa1c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:27:28.588 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:27:28.582+0000 7f29a064a740 1 --2- 172.21.15.114:0/3385477381 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x55f6587d39c0 0x55f6587ce480 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:27:28.588 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:27:28.582+0000 7f29a064a740 1 --2- 172.21.15.114:0/3385477381 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55f6587d30a0 0x55f6587d5910 unknown :-1 s=CLOSED pgs=523 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:27:28.588 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:27:28.582+0000 7f29a064a740 1 -- 172.21.15.114:0/3385477381 >> 172.21.15.114:0/3385477381 conn(0x55f6587cf3c0 msgr2=0x55f6587cdd70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T10:27:28.588 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:27:28.582+0000 7f29a064a740 1 -- 172.21.15.114:0/3385477381 shutdown_connections 2024-04-23T10:27:28.588 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:27:28.582+0000 7f29a064a740 1 -- 172.21.15.114:0/3385477381 wait complete. 2024-04-23T10:27:28.588 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:27:28.582+0000 7f29a064a740 1 -- start start 2024-04-23T10:27:28.589 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:27:28.583+0000 7f29a064a740 1 Processor -- start 2024-04-23T10:27:28.589 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:27:28.583+0000 7f29a064a740 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x55f6587d30a0 0x55f6587ea6f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T10:27:28.589 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:27:28.583+0000 7f29a064a740 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x55f6587d39c0 0x55f6587eac30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T10:27:28.589 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:27:28.583+0000 7f29a064a740 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55f6587de090 0x55f6587e1f00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T10:27:28.589 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:27:28.583+0000 7f29a064a740 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x7f29981aa120 con 0x55f6587de090 2024-04-23T10:27:28.589 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:27:28.583+0000 7f29a064a740 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 -- 0x7f29981ab590 con 0x55f6587d30a0 2024-04-23T10:27:28.589 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:27:28.583+0000 7f29a064a740 1 -- --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_getmap magic: 0 -- 0x7f29981aa7f0 con 0x55f6587d39c0 2024-04-23T10:27:28.589 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:27:28.583+0000 7f2996ffd640 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x55f6587d30a0 0x55f6587ea6f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T10:27:28.589 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:27:28.583+0000 7f29a064a740 10 client.0 _ll_register_callbacks cb 0x7f298c005d90 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-23T10:27:28.589 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:27:28.583+0000 7f29977fe640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55f6587de090 0x55f6587e1f00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T10:27:28.589 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:27:28.583+0000 7f2996ffd640 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x55f6587d30a0 0x55f6587ea6f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.119:3300/0 says I am v2:172.21.15.114:57994/0 (socket says 172.21.15.114:57994) 2024-04-23T10:27:28.590 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:27:28.583+0000 7f2996ffd640 1 -- 172.21.15.114:0/402091397 learned_addr learned my addr 172.21.15.114:0/402091397 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-04-23T10:27:28.590 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:27:28.583+0000 7f29967fc640 1 --2- 172.21.15.114:0/402091397 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x55f6587d39c0 0x55f6587eac30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T10:27:28.590 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:27:28.584+0000 7f2996ffd640 1 -- 172.21.15.114:0/402091397 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x55f6587d39c0 msgr2=0x55f6587eac30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T10:27:28.590 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:27:28.584+0000 7f2996ffd640 1 --2- 172.21.15.114:0/402091397 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x55f6587d39c0 0x55f6587eac30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:27:28.590 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:27:28.584+0000 7f2996ffd640 1 -- 172.21.15.114:0/402091397 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55f6587de090 msgr2=0x55f6587e1f00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T10:27:28.590 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:27:28.584+0000 7f2996ffd640 1 --2- 172.21.15.114:0/402091397 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55f6587de090 0x55f6587e1f00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:27:28.590 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:27:28.584+0000 7f2996ffd640 1 -- 172.21.15.114:0/402091397 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55f6587e2440 con 0x55f6587d30a0 2024-04-23T10:27:28.590 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:27:28.584+0000 7f29977fe640 1 --2- 172.21.15.114:0/402091397 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55f6587de090 0x55f6587e1f00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-23T10:27:28.590 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:27:28.584+0000 7f29967fc640 1 --2- 172.21.15.114:0/402091397 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x55f6587d39c0 0x55f6587eac30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-23T10:27:28.590 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:27:28.584+0000 7f2996ffd640 1 --2- 172.21.15.114:0/402091397 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x55f6587d30a0 0x55f6587ea6f0 secure :-1 s=READY pgs=61 cs=0 l=1 rev1=1 crypto rx=0x7f298809c710 tx=0x7f29880b6090 comp rx=0 tx=0).ready entity=mon.2 client_cookie=4c26bd4dd612f75 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T10:27:28.590 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:27:28.584+0000 7f2976ffd640 10 client.0 ms_handle_connect on v2:172.21.15.119:3300/0 2024-04-23T10:27:28.590 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:27:28.584+0000 7f2976ffd640 1 -- 172.21.15.114:0/402091397 <== mon.2 v2:172.21.15.119:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f29880b5070 con 0x55f6587d30a0 2024-04-23T10:27:28.591 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:27:28.584+0000 7f29a064a740 1 -- 172.21.15.114:0/402091397 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f29880a5680 con 0x55f6587d30a0 2024-04-23T10:27:28.591 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:27:28.584+0000 7f2976ffd640 1 -- 172.21.15.114:0/402091397 <== mon.2 v2:172.21.15.119:3300/0 2 ==== osd_map(90..90 src has 1..90) ==== 8527+0+0 (secure 0 0 0) 0x7f2988009310 con 0x55f6587d30a0 2024-04-23T10:27:28.591 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:27:28.584+0000 7f2976ffd640 1 -- 172.21.15.114:0/402091397 <== mon.2 v2:172.21.15.119:3300/0 3 ==== mon_get_version_reply(handle=1 version=21) ==== 24+0+0 (secure 0 0 0) 0x7f2988099080 con 0x55f6587d30a0 2024-04-23T10:27:28.591 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:27:28.584+0000 7f29a064a740 10 client.25217 fetch_fsmap learned FSMap version 21 2024-04-23T10:27:28.591 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:27:28.584+0000 7f29a064a740 1 -- 172.21.15.114:0/402091397 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({config=0+,fsmap=21,monmap=4+}) -- 0x55f65885da20 con 0x55f6587d30a0 2024-04-23T10:27:28.591 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:27:28.584+0000 7f2976ffd640 1 -- 172.21.15.114:0/402091397 <== mon.2 v2:172.21.15.119:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f29880a5680 con 0x55f6587d30a0 2024-04-23T10:27:28.591 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:27:28.585+0000 7f2976ffd640 1 -- 172.21.15.114:0/402091397 <== mon.2 v2:172.21.15.119:3300/0 5 ==== fsmap(e 21) ==== 8338+0+0 (secure 0 0 0) 0x7f29880cebd0 con 0x55f6587d30a0 2024-04-23T10:27:28.591 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:27:28.585+0000 7f29a064a740 10 client.25217 fetch_fsmap finished waiting for FSMap version 21 2024-04-23T10:27:28.591 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:27:28.585+0000 7f29a064a740 10 client.25217 resolve_mds: validated gid 24545 aka daemon mds.b 2024-04-23T10:27:28.591 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:27:28.585+0000 7f29a064a740 20 client.25217 populate_metadata read hostname 'smithi114' 2024-04-23T10:27:28.591 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:27:28.585+0000 7f29a064a740 1 --2- 172.21.15.114:0/402091397 >> [v2:172.21.15.119:6838/937297340,v1:172.21.15.119:6839/937297340] conn(0x55f6587f2740 0x55f658860100 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T10:27:28.591 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:27:28.585+0000 7f29a064a740 4 client.25217 mds_command: new command op to 24545 tid=0 multi_id=0 [{"prefix": "quiesce db", "roots": ["/"], "timeout": 60.0, "expiration": 82.7, "await_for": 42.0}] 2024-04-23T10:27:28.592 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:27:28.585+0000 7f29a064a740 1 -- 172.21.15.114:0/402091397 --> [v2:172.21.15.119:6838/937297340,v1:172.21.15.119:6839/937297340] -- command(tid 0: {"prefix": "quiesce db", "roots": ["/"], "timeout": 60.0, "expiration": 82.7, "await_for": 42.0}) -- 0x55f6587cf210 con 0x55f6587f2740 2024-04-23T10:27:28.592 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:27:28.585+0000 7f29967fc640 1 --2- 172.21.15.114:0/402091397 >> [v2:172.21.15.119:6838/937297340,v1:172.21.15.119:6839/937297340] conn(0x55f6587f2740 0x55f658860100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T10:27:28.592 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:27:28.585+0000 7f2976ffd640 10 client.25217 ms_handle_connect on v2:172.21.15.119:6838/937297340 2024-04-23T10:27:28.592 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:27:28.585+0000 7f29967fc640 1 --2- 172.21.15.114:0/402091397 >> [v2:172.21.15.119:6838/937297340,v1:172.21.15.119:6839/937297340] conn(0x55f6587f2740 0x55f658860100 crc :-1 s=READY pgs=51 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=b3244513d7d7f8ba server_cookie=374d961c303eb5c4 in_seq=0 out_seq=0 2024-04-23T10:27:28.594 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:27:28.593+0000 7f2976ffd640 1 -- 172.21.15.114:0/402091397 <== mds.0 v2:172.21.15.119:6838/937297340 1 ==== command_reply(tid 0: 0 ) ==== 8+0+588 (crc 0 0 0) 0x55f6587cf210 con 0x55f6587f2740 2024-04-23T10:27:28.594 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:27:28.593+0000 7f2976ffd640 10 client.25217 handle_command_reply: tid=0 2024-04-23T10:27:28.594 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:27:28.593+0000 7f29a064a740 1 client.25217 shutdown 2024-04-23T10:27:28.594 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:27:28.593+0000 7f29a064a740 1 -- 172.21.15.114:0/402091397 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x55f6587d30a0 msgr2=0x55f6587ea6f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T10:27:28.594 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:27:28.593+0000 7f29a064a740 1 --2- 172.21.15.114:0/402091397 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x55f6587d30a0 0x55f6587ea6f0 secure :-1 s=READY pgs=61 cs=0 l=1 rev1=1 crypto rx=0x7f298809c710 tx=0x7f29880b6090 comp rx=0 tx=0).stop 2024-04-23T10:27:28.594 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:27:28.593+0000 7f29a064a740 1 -- 172.21.15.114:0/402091397 shutdown_connections 2024-04-23T10:27:28.594 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:27:28.593+0000 7f29a064a740 1 --2- 172.21.15.114:0/402091397 >> [v2:172.21.15.119:6838/937297340,v1:172.21.15.119:6839/937297340] conn(0x55f6587f2740 0x55f658860100 crc :-1 s=READY pgs=51 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:27:28.594 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:27:28.593+0000 7f29a064a740 1 --2- 172.21.15.114:0/402091397 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55f6587de090 0x55f6587e1f00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:27:28.594 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:27:28.593+0000 7f2976ffd640 0 client.25217 ms_handle_reset on v2:172.21.15.119:6838/937297340 2024-04-23T10:27:28.594 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:27:28.593+0000 7f29a064a740 1 --2- 172.21.15.114:0/402091397 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x55f6587d39c0 0x55f6587eac30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:27:28.595 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:27:28.593+0000 7f29a064a740 1 --2- 172.21.15.114:0/402091397 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x55f6587d30a0 0x55f6587ea6f0 unknown :-1 s=CLOSED pgs=61 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:27:28.595 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:27:28.593+0000 7f29a064a740 1 -- 172.21.15.114:0/402091397 >> 172.21.15.114:0/402091397 conn(0x55f6587cf3c0 msgr2=0x55f6587db860 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T10:27:28.595 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:27:28.593+0000 7f29a064a740 1 -- 172.21.15.114:0/402091397 shutdown_connections 2024-04-23T10:27:28.595 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:27:28.593+0000 7f29a064a740 1 -- 172.21.15.114:0/402091397 wait complete. 2024-04-23T10:27:28.595 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:27:28.593+0000 7f29a064a740 20 client.25217 trim_cache size 0 max 16384 2024-04-23T10:27:28.595 INFO:teuthology.orchestra.run.smithi114.stdout:{ 2024-04-23T10:27:28.595 INFO:teuthology.orchestra.run.smithi114.stdout: "epoch": 21, 2024-04-23T10:27:28.595 INFO:teuthology.orchestra.run.smithi114.stdout: "leader": 24545, 2024-04-23T10:27:28.595 INFO:teuthology.orchestra.run.smithi114.stdout: "set_version": 6, 2024-04-23T10:27:28.596 INFO:teuthology.orchestra.run.smithi114.stdout: "sets": { 2024-04-23T10:27:28.596 INFO:teuthology.orchestra.run.smithi114.stdout: "750460ef": { 2024-04-23T10:27:28.596 INFO:teuthology.orchestra.run.smithi114.stdout: "version": 6, 2024-04-23T10:27:28.596 INFO:teuthology.orchestra.run.smithi114.stdout: "age_ref": 0.0, 2024-04-23T10:27:28.596 INFO:teuthology.orchestra.run.smithi114.stdout: "state": { 2024-04-23T10:27:28.596 INFO:teuthology.orchestra.run.smithi114.stdout: "name": "QUIESCED", 2024-04-23T10:27:28.596 INFO:teuthology.orchestra.run.smithi114.stdout: "age": 0.0 2024-04-23T10:27:28.596 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-23T10:27:28.596 INFO:teuthology.orchestra.run.smithi114.stdout: "timeout": 60.0, 2024-04-23T10:27:28.596 INFO:teuthology.orchestra.run.smithi114.stdout: "expiration": 82.7, 2024-04-23T10:27:28.596 INFO:teuthology.orchestra.run.smithi114.stdout: "members": { 2024-04-23T10:27:28.596 INFO:teuthology.orchestra.run.smithi114.stdout: "file:/": { 2024-04-23T10:27:28.596 INFO:teuthology.orchestra.run.smithi114.stdout: "excluded": false, 2024-04-23T10:27:28.597 INFO:teuthology.orchestra.run.smithi114.stdout: "state": { 2024-04-23T10:27:28.597 INFO:teuthology.orchestra.run.smithi114.stdout: "name": "QUIESCED", 2024-04-23T10:27:28.597 INFO:teuthology.orchestra.run.smithi114.stdout: "age": 0.0 2024-04-23T10:27:28.597 INFO:teuthology.orchestra.run.smithi114.stdout: } 2024-04-23T10:27:28.597 INFO:teuthology.orchestra.run.smithi114.stdout: } 2024-04-23T10:27:28.597 INFO:teuthology.orchestra.run.smithi114.stdout: } 2024-04-23T10:27:28.597 INFO:teuthology.orchestra.run.smithi114.stdout: } 2024-04-23T10:27:28.597 INFO:teuthology.orchestra.run.smithi114.stdout: } 2024-04-23T10:27:28.597 INFO:teuthology.orchestra.run.smithi114.stdout:} 2024-04-23T10:27:28.598 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:27:28.594+0000 7f299d930640 1 -- 172.21.15.114:0/446225035 >> [v2:172.21.15.114:6800/1395693466,v1:172.21.15.114:6801/1395693466] conn(0x7f29800a3e40 msgr2=0x7f29800a6300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T10:27:28.598 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:27:28.594+0000 7f299d930640 1 --2- 172.21.15.114:0/446225035 >> [v2:172.21.15.114:6800/1395693466,v1:172.21.15.114:6801/1395693466] conn(0x7f29800a3e40 0x7f29800a6300 secure :-1 s=READY pgs=70 cs=0 l=1 rev1=1 crypto rx=0x7f2990005f40 tx=0x7f2990009600 comp rx=0 tx=0).stop 2024-04-23T10:27:28.598 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:27:28.594+0000 7f299d930640 1 -- 172.21.15.114:0/446225035 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f29981021b0 msgr2=0x7f299819e3e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T10:27:28.598 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:27:28.594+0000 7f299d930640 1 --2- 172.21.15.114:0/446225035 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f29981021b0 0x7f299819e3e0 secure :-1 s=READY pgs=59 cs=0 l=1 rev1=1 crypto rx=0x7f298800e9f0 tx=0x7f298800eeb0 comp rx=0 tx=0).stop 2024-04-23T10:27:28.598 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:27:28.594+0000 7f299d930640 1 -- 172.21.15.114:0/446225035 shutdown_connections 2024-04-23T10:27:28.599 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:27:28.594+0000 7f299d930640 1 --2- 172.21.15.114:0/446225035 >> [v2:172.21.15.114:6800/1395693466,v1:172.21.15.114:6801/1395693466] conn(0x7f29800a3e40 0x7f29800a6300 unknown :-1 s=CLOSED pgs=70 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:27:28.599 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:27:28.594+0000 7f299d930640 1 --2- 172.21.15.114:0/446225035 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f29981985e0 0x7f2998198a90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:27:28.599 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:27:28.594+0000 7f299d930640 1 --2- 172.21.15.114:0/446225035 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f2998107280 0x7f299819e920 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:27:28.599 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:27:28.594+0000 7f299d930640 1 --2- 172.21.15.114:0/446225035 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f29981021b0 0x7f299819e3e0 unknown :-1 s=CLOSED pgs=59 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:27:28.599 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:27:28.594+0000 7f299d930640 1 -- 172.21.15.114:0/446225035 >> 172.21.15.114:0/446225035 conn(0x7f29980fd9f0 msgr2=0x7f29980fee30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T10:27:28.599 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:27:28.594+0000 7f299d930640 1 -- 172.21.15.114:0/446225035 shutdown_connections 2024-04-23T10:27:28.599 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:27:28.594+0000 7f299d930640 1 -- 172.21.15.114:0/446225035 wait complete. 2024-04-23T10:27:28.604 INFO:tasks.quiescer.fs.[cephfs]:Successfully quiesced set '750460ef', quiesce took 0.6 seconds. Will release after: 22.7 2024-04-23T10:27:29.487 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 23 10:27:29 smithi119 ceph-mon[32608]: from='client.? 172.21.15.114:0/3947971820' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-23T10:27:29.504 INFO:journalctl@ceph.mon.a.smithi114.stdout:Apr 23 10:27:29 smithi114 ceph-mon[29255]: from='client.? 172.21.15.114:0/3947971820' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-23T10:27:29.578 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 23 10:27:29 smithi204 ceph-mon[32649]: from='client.? 172.21.15.114:0/3947971820' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-23T10:27:30.492 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 23 10:27:30 smithi119 ceph-mon[32608]: pgmap v27: 97 pgs: 97 active+clean; 1.3 MiB data, 377 MiB used, 1.0 TiB / 1.0 TiB avail; 3.4 KiB/s rd, 255 B/s wr, 7 op/s 2024-04-23T10:27:30.504 INFO:journalctl@ceph.mon.a.smithi114.stdout:Apr 23 10:27:30 smithi114 ceph-mon[29255]: pgmap v27: 97 pgs: 97 active+clean; 1.3 MiB data, 377 MiB used, 1.0 TiB / 1.0 TiB avail; 3.4 KiB/s rd, 255 B/s wr, 7 op/s 2024-04-23T10:27:30.577 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 23 10:27:30 smithi204 ceph-mon[32649]: pgmap v27: 97 pgs: 97 active+clean; 1.3 MiB data, 377 MiB used, 1.0 TiB / 1.0 TiB avail; 3.4 KiB/s rd, 255 B/s wr, 7 op/s 2024-04-23T10:27:32.492 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 23 10:27:32 smithi119 ceph-mon[32608]: pgmap v28: 97 pgs: 97 active+clean; 1.3 MiB data, 377 MiB used, 1.0 TiB / 1.0 TiB avail; 2.6 KiB/s rd, 255 B/s wr, 5 op/s 2024-04-23T10:27:32.503 INFO:journalctl@ceph.mon.a.smithi114.stdout:Apr 23 10:27:32 smithi114 ceph-mon[29255]: pgmap v28: 97 pgs: 97 active+clean; 1.3 MiB data, 377 MiB used, 1.0 TiB / 1.0 TiB avail; 2.6 KiB/s rd, 255 B/s wr, 5 op/s 2024-04-23T10:27:32.577 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 23 10:27:32 smithi204 ceph-mon[32649]: pgmap v28: 97 pgs: 97 active+clean; 1.3 MiB data, 377 MiB used, 1.0 TiB / 1.0 TiB avail; 2.6 KiB/s rd, 255 B/s wr, 5 op/s 2024-04-23T10:27:34.492 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 23 10:27:34 smithi119 ceph-mon[32608]: pgmap v29: 97 pgs: 97 active+clean; 1.3 MiB data, 377 MiB used, 1.0 TiB / 1.0 TiB avail; 3.8 KiB/s rd, 255 B/s wr, 7 op/s 2024-04-23T10:27:34.504 INFO:journalctl@ceph.mon.a.smithi114.stdout:Apr 23 10:27:34 smithi114 ceph-mon[29255]: pgmap v29: 97 pgs: 97 active+clean; 1.3 MiB data, 377 MiB used, 1.0 TiB / 1.0 TiB avail; 3.8 KiB/s rd, 255 B/s wr, 7 op/s 2024-04-23T10:27:34.577 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 23 10:27:34 smithi204 ceph-mon[32649]: pgmap v29: 97 pgs: 97 active+clean; 1.3 MiB data, 377 MiB used, 1.0 TiB / 1.0 TiB avail; 3.8 KiB/s rd, 255 B/s wr, 7 op/s 2024-04-23T10:27:35.141 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:27:35.140+0000 7fae1bbe2640 1 -- 172.21.15.114:0/1490186265 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fae14102b60 msgr2=0x7fae14106ba0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T10:27:35.141 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:27:35.140+0000 7fae1bbe2640 1 --2- 172.21.15.114:0/1490186265 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fae14102b60 0x7fae14106ba0 secure :-1 s=READY pgs=524 cs=0 l=1 rev1=1 crypto rx=0x7fae04009900 tx=0x7fae0402a410 comp rx=0 tx=0).stop 2024-04-23T10:27:35.141 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:27:35.140+0000 7fae1bbe2640 1 -- 172.21.15.114:0/1490186265 shutdown_connections 2024-04-23T10:27:35.141 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:27:35.140+0000 7fae1bbe2640 1 --2- 172.21.15.114:0/1490186265 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fae14107280 0x7fae1410ee50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:27:35.141 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:27:35.140+0000 7fae1bbe2640 1 --2- 172.21.15.114:0/1490186265 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fae14102b60 0x7fae14106ba0 unknown :-1 s=CLOSED pgs=524 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:27:35.141 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:27:35.140+0000 7fae1bbe2640 1 --2- 172.21.15.114:0/1490186265 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fae141021b0 0x7fae14102590 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:27:35.141 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:27:35.140+0000 7fae1bbe2640 1 -- 172.21.15.114:0/1490186265 >> 172.21.15.114:0/1490186265 conn(0x7fae140fda10 msgr2=0x7fae140ffe30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T10:27:35.141 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:27:35.140+0000 7fae1bbe2640 1 -- 172.21.15.114:0/1490186265 shutdown_connections 2024-04-23T10:27:35.141 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:27:35.140+0000 7fae1bbe2640 1 -- 172.21.15.114:0/1490186265 wait complete. 2024-04-23T10:27:35.142 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:27:35.140+0000 7fae1bbe2640 1 Processor -- start 2024-04-23T10:27:35.142 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:27:35.140+0000 7fae1bbe2640 1 -- start start 2024-04-23T10:27:35.142 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:27:35.141+0000 7fae1bbe2640 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fae141021b0 0x7fae141986f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T10:27:35.142 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:27:35.141+0000 7fae1bbe2640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fae14107280 0x7fae14198c30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T10:27:35.142 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:27:35.141+0000 7fae1bbe2640 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fae1419cf60 0x7fae1419d410 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T10:27:35.142 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:27:35.141+0000 7fae1bbe2640 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x7fae14112210 con 0x7fae14107280 2024-04-23T10:27:35.142 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:27:35.141+0000 7fae1bbe2640 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 -- 0x7fae14112090 con 0x7fae1419cf60 2024-04-23T10:27:35.142 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:27:35.141+0000 7fae1bbe2640 1 -- --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_getmap magic: 0 -- 0x7fae14112390 con 0x7fae141021b0 2024-04-23T10:27:35.142 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:27:35.141+0000 7fae1917d640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fae14107280 0x7fae14198c30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T10:27:35.142 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:27:35.141+0000 7fae1997e640 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fae141021b0 0x7fae141986f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T10:27:35.142 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:27:35.141+0000 7fae1917d640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fae14107280 0x7fae14198c30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.114:53640/0 (socket says 172.21.15.114:53640) 2024-04-23T10:27:35.142 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:27:35.141+0000 7fae1917d640 1 -- 172.21.15.114:0/3354065237 learned_addr learned my addr 172.21.15.114:0/3354065237 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-04-23T10:27:35.142 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:27:35.141+0000 7fae1997e640 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fae141021b0 0x7fae141986f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.204:3300/0 says I am v2:172.21.15.114:40474/0 (socket says 172.21.15.114:40474) 2024-04-23T10:27:35.143 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:27:35.141+0000 7fae1a17f640 1 --2- 172.21.15.114:0/3354065237 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fae1419cf60 0x7fae1419d410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T10:27:35.143 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:27:35.141+0000 7fae1917d640 1 -- 172.21.15.114:0/3354065237 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fae141021b0 msgr2=0x7fae141986f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T10:27:35.143 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:27:35.141+0000 7fae1917d640 1 --2- 172.21.15.114:0/3354065237 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fae141021b0 0x7fae141986f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:27:35.143 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:27:35.141+0000 7fae1917d640 1 -- 172.21.15.114:0/3354065237 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fae1419cf60 msgr2=0x7fae1419d410 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T10:27:35.143 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:27:35.141+0000 7fae1917d640 1 --2- 172.21.15.114:0/3354065237 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fae1419cf60 0x7fae1419d410 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:27:35.143 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:27:35.141+0000 7fae1917d640 1 -- 172.21.15.114:0/3354065237 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fae1419da50 con 0x7fae14107280 2024-04-23T10:27:35.143 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:27:35.141+0000 7fae1997e640 1 --2- 172.21.15.114:0/3354065237 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fae141021b0 0x7fae141986f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-23T10:27:35.143 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:27:35.141+0000 7fae1917d640 1 --2- 172.21.15.114:0/3354065237 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fae14107280 0x7fae14198c30 secure :-1 s=READY pgs=525 cs=0 l=1 rev1=1 crypto rx=0x7fae04005ab0 tx=0x7fae04003b90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T10:27:35.143 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:27:35.142+0000 7fae0affd640 1 -- 172.21.15.114:0/3354065237 <== mon.0 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fae0403f070 con 0x7fae14107280 2024-04-23T10:27:35.143 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:27:35.142+0000 7fae0affd640 1 -- 172.21.15.114:0/3354065237 <== mon.0 v2:172.21.15.114:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fae04031b50 con 0x7fae14107280 2024-04-23T10:27:35.143 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:27:35.142+0000 7fae1bbe2640 1 -- 172.21.15.114:0/3354065237 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fae1419dce0 con 0x7fae14107280 2024-04-23T10:27:35.143 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:27:35.142+0000 7fae0affd640 1 -- 172.21.15.114:0/3354065237 <== mon.0 v2:172.21.15.114:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fae04035030 con 0x7fae14107280 2024-04-23T10:27:35.144 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:27:35.142+0000 7fae1bbe2640 1 -- 172.21.15.114:0/3354065237 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fae14074470 con 0x7fae14107280 2024-04-23T10:27:35.144 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:27:35.145+0000 7fae0affd640 1 -- 172.21.15.114:0/3354065237 <== mon.0 v2:172.21.15.114:3300/0 4 ==== mgrmap(e 25) ==== 138880+0+0 (secure 0 0 0) 0x7fae04042070 con 0x7fae14107280 2024-04-23T10:27:35.144 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:27:35.145+0000 7fae0affd640 1 --2- 172.21.15.114:0/3354065237 >> [v2:172.21.15.114:6800/1395693466,v1:172.21.15.114:6801/1395693466] conn(0x7fae000a3f60 0x7fae000a6420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T10:27:35.144 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:27:35.146+0000 7fae0affd640 1 -- 172.21.15.114:0/3354065237 <== mon.0 v2:172.21.15.114:3300/0 5 ==== osd_map(90..90 src has 1..90) ==== 8527+0+0 (secure 0 0 0) 0x7fae040f0f40 con 0x7fae14107280 2024-04-23T10:27:35.145 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:27:35.146+0000 7fae1997e640 1 --2- 172.21.15.114:0/3354065237 >> [v2:172.21.15.114:6800/1395693466,v1:172.21.15.114:6801/1395693466] conn(0x7fae000a3f60 0x7fae000a6420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T10:27:35.145 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:27:35.146+0000 7fae1997e640 1 --2- 172.21.15.114:0/3354065237 >> [v2:172.21.15.114:6800/1395693466,v1:172.21.15.114:6801/1395693466] conn(0x7fae000a3f60 0x7fae000a6420 secure :-1 s=READY pgs=71 cs=0 l=1 rev1=1 crypto rx=0x7fae1000bfd0 tx=0x7fae1000b390 comp rx=0 tx=0).ready entity=mgr.15291 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T10:27:35.145 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:27:35.146+0000 7fae1e9d2740 1 Processor -- start 2024-04-23T10:27:35.145 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:27:35.147+0000 7fae1e9d2740 1 -- start start 2024-04-23T10:27:35.147 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:27:35.147+0000 7fae1e9d2740 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55e31125b620 0x55e31125ba00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T10:27:35.147 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:27:35.147+0000 7fae1e9d2740 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x55e31125bf40 0x55e3112662c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T10:27:35.147 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:27:35.147+0000 7fae1e9d2740 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x55e311266800 0x55e311268bf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T10:27:35.147 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:27:35.147+0000 7fae1e9d2740 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x55e3111c0970 con 0x55e31125b620 2024-04-23T10:27:35.147 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:27:35.147+0000 7fae1e9d2740 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 -- 0x55e31115d4a0 con 0x55e31125bf40 2024-04-23T10:27:35.148 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:27:35.147+0000 7fae1e9d2740 1 -- --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_getmap magic: 0 -- 0x55e3111bc930 con 0x55e311266800 2024-04-23T10:27:35.148 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:27:35.147+0000 7fae1997e640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55e31125b620 0x55e31125ba00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T10:27:35.148 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:27:35.147+0000 7fae1997e640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55e31125b620 0x55e31125ba00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.114:53646/0 (socket says 172.21.15.114:53646) 2024-04-23T10:27:35.148 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:27:35.147+0000 7fae1997e640 1 -- 172.21.15.114:0/2482896380 learned_addr learned my addr 172.21.15.114:0/2482896380 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-04-23T10:27:35.148 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:27:35.147+0000 7fae1917d640 1 --2- 172.21.15.114:0/2482896380 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x55e31125bf40 0x55e3112662c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T10:27:35.148 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:27:35.147+0000 7fae1a17f640 1 --2- 172.21.15.114:0/2482896380 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x55e311266800 0x55e311268bf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T10:27:35.148 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:27:35.147+0000 7fae1997e640 1 -- 172.21.15.114:0/2482896380 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x55e311266800 msgr2=0x55e311268bf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T10:27:35.148 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:27:35.148+0000 7fae1997e640 1 --2- 172.21.15.114:0/2482896380 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x55e311266800 0x55e311268bf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:27:35.148 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:27:35.148+0000 7fae1997e640 1 -- 172.21.15.114:0/2482896380 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x55e31125bf40 msgr2=0x55e3112662c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T10:27:35.148 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:27:35.148+0000 7fae1997e640 1 --2- 172.21.15.114:0/2482896380 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x55e31125bf40 0x55e3112662c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:27:35.148 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:27:35.148+0000 7fae1997e640 1 -- 172.21.15.114:0/2482896380 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55e310ed3210 con 0x55e31125b620 2024-04-23T10:27:35.148 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:27:35.148+0000 7fae1997e640 1 --2- 172.21.15.114:0/2482896380 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55e31125b620 0x55e31125ba00 secure :-1 s=READY pgs=526 cs=0 l=1 rev1=1 crypto rx=0x7fae10011800 tx=0x7fae10013000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1bace3cb788b595d server_cookie=0 in_seq=0 out_seq=0 2024-04-23T10:27:35.148 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:27:35.148+0000 7fadfffff640 1 -- 172.21.15.114:0/2482896380 <== mon.0 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fae1001cac0 con 0x55e31125b620 2024-04-23T10:27:35.149 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:27:35.148+0000 7fadfffff640 1 -- 172.21.15.114:0/2482896380 <== mon.0 v2:172.21.15.114:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fae10011e40 con 0x55e31125b620 2024-04-23T10:27:35.149 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:27:35.148+0000 7fadfffff640 1 -- 172.21.15.114:0/2482896380 <== mon.0 v2:172.21.15.114:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fae1001d3b0 con 0x55e31125b620 2024-04-23T10:27:35.154 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:27:35.149+0000 7fae1e9d2740 1 -- 172.21.15.114:0/2482896380 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55e31125b620 msgr2=0x55e31125ba00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T10:27:35.154 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:27:35.149+0000 7fae1e9d2740 1 --2- 172.21.15.114:0/2482896380 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55e31125b620 0x55e31125ba00 secure :-1 s=READY pgs=526 cs=0 l=1 rev1=1 crypto rx=0x7fae10011800 tx=0x7fae10013000 comp rx=0 tx=0).stop 2024-04-23T10:27:35.154 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:27:35.149+0000 7fae1e9d2740 1 -- 172.21.15.114:0/2482896380 shutdown_connections 2024-04-23T10:27:35.154 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:27:35.149+0000 7fae1e9d2740 1 --2- 172.21.15.114:0/2482896380 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x55e311266800 0x55e311268bf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:27:35.154 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:27:35.149+0000 7fae1e9d2740 1 --2- 172.21.15.114:0/2482896380 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x55e31125bf40 0x55e3112662c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:27:35.154 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:27:35.149+0000 7fae1e9d2740 1 --2- 172.21.15.114:0/2482896380 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55e31125b620 0x55e31125ba00 unknown :-1 s=CLOSED pgs=526 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:27:35.154 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:27:35.149+0000 7fae1e9d2740 1 -- 172.21.15.114:0/2482896380 >> 172.21.15.114:0/2482896380 conn(0x55e311257b30 msgr2=0x55e311257f10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T10:27:35.154 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:27:35.149+0000 7fae1e9d2740 1 -- 172.21.15.114:0/2482896380 shutdown_connections 2024-04-23T10:27:35.154 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:27:35.149+0000 7fae1e9d2740 1 -- 172.21.15.114:0/2482896380 wait complete. 2024-04-23T10:27:35.154 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:27:35.150+0000 7fae1e9d2740 1 -- start start 2024-04-23T10:27:35.154 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:27:35.150+0000 7fae1e9d2740 1 Processor -- start 2024-04-23T10:27:35.154 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:27:35.150+0000 7fae1e9d2740 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x55e31125b620 0x55e3112661a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T10:27:35.154 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:27:35.150+0000 7fae1e9d2740 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x55e31125bf40 0x55e311263490 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T10:27:35.155 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:27:35.150+0000 7fae1e9d2740 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55e311266800 0x55e311263a30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T10:27:35.155 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:27:35.150+0000 7fae1e9d2740 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x55e311269430 con 0x55e311266800 2024-04-23T10:27:35.155 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:27:35.150+0000 7fae1e9d2740 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 -- 0x55e31115b0b0 con 0x55e31125b620 2024-04-23T10:27:35.155 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:27:35.150+0000 7fae1e9d2740 1 -- --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_getmap magic: 0 -- 0x55e3112695b0 con 0x55e31125bf40 2024-04-23T10:27:35.155 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:27:35.150+0000 7fae1a17f640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55e311266800 0x55e311263a30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T10:27:35.155 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:27:35.150+0000 7fae1a17f640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55e311266800 0x55e311263a30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.114:53650/0 (socket says 172.21.15.114:53650) 2024-04-23T10:27:35.155 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:27:35.150+0000 7fae1a17f640 1 -- 172.21.15.114:0/208513149 learned_addr learned my addr 172.21.15.114:0/208513149 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-04-23T10:27:35.155 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:27:35.150+0000 7fae1e9d2740 10 client.0 _ll_register_callbacks cb 0x55e311148d30 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-23T10:27:35.155 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:27:35.150+0000 7fae1997e640 1 --2- 172.21.15.114:0/208513149 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x55e31125b620 0x55e3112661a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T10:27:35.155 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:27:35.150+0000 7fae1917d640 1 --2- 172.21.15.114:0/208513149 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x55e31125bf40 0x55e311263490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T10:27:35.155 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:27:35.151+0000 7fae1a17f640 1 -- 172.21.15.114:0/208513149 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x55e31125bf40 msgr2=0x55e311263490 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T10:27:35.155 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:27:35.151+0000 7fae1a17f640 1 --2- 172.21.15.114:0/208513149 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x55e31125bf40 0x55e311263490 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:27:35.155 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:27:35.151+0000 7fae1a17f640 1 -- 172.21.15.114:0/208513149 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x55e31125b620 msgr2=0x55e3112661a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T10:27:35.155 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:27:35.151+0000 7fae1a17f640 1 --2- 172.21.15.114:0/208513149 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x55e31125b620 0x55e3112661a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:27:35.155 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:27:35.151+0000 7fae1a17f640 1 -- 172.21.15.114:0/208513149 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55e310f9f620 con 0x55e311266800 2024-04-23T10:27:35.156 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:27:35.151+0000 7fae1a17f640 1 --2- 172.21.15.114:0/208513149 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55e311266800 0x55e311263a30 secure :-1 s=READY pgs=527 cs=0 l=1 rev1=1 crypto rx=0x7fae0c011cf0 tx=0x7fae0c00f790 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ecdc5c0d3ad107da server_cookie=0 in_seq=0 out_seq=0 2024-04-23T10:27:35.156 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:27:35.151+0000 7fadfd7fa640 10 client.0 ms_handle_connect on v2:172.21.15.114:3300/0 2024-04-23T10:27:35.156 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:27:35.151+0000 7fadfd7fa640 1 -- 172.21.15.114:0/208513149 <== mon.0 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fae0c00fe60 con 0x55e311266800 2024-04-23T10:27:35.156 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:27:35.151+0000 7fadfd7fa640 1 -- 172.21.15.114:0/208513149 <== mon.0 v2:172.21.15.114:3300/0 2 ==== osd_map(90..90 src has 1..90) ==== 8527+0+0 (secure 0 0 0) 0x7fae0c003970 con 0x55e311266800 2024-04-23T10:27:35.156 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:27:35.151+0000 7fae1e9d2740 1 -- 172.21.15.114:0/208513149 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x55e3111ce390 con 0x55e311266800 2024-04-23T10:27:35.156 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:27:35.151+0000 7fadfd7fa640 1 -- 172.21.15.114:0/208513149 <== mon.0 v2:172.21.15.114:3300/0 3 ==== mon_get_version_reply(handle=1 version=21) ==== 24+0+0 (secure 0 0 0) 0x7fae0c003b70 con 0x55e311266800 2024-04-23T10:27:35.156 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:27:35.151+0000 7fae1e9d2740 10 client.15696 fetch_fsmap learned FSMap version 21 2024-04-23T10:27:35.156 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:27:35.151+0000 7fae1e9d2740 1 -- 172.21.15.114:0/208513149 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({config=0+,fsmap=21,monmap=4+}) -- 0x55e3112751d0 con 0x55e311266800 2024-04-23T10:27:35.156 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:27:35.151+0000 7fadfd7fa640 1 -- 172.21.15.114:0/208513149 <== mon.0 v2:172.21.15.114:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x55e3111ce390 con 0x55e311266800 2024-04-23T10:27:35.156 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:27:35.152+0000 7fadfd7fa640 1 -- 172.21.15.114:0/208513149 <== mon.0 v2:172.21.15.114:3300/0 5 ==== fsmap(e 21) ==== 8338+0+0 (secure 0 0 0) 0x7fae0c01b5e0 con 0x55e311266800 2024-04-23T10:27:35.156 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:27:35.152+0000 7fae1e9d2740 10 client.15696 fetch_fsmap finished waiting for FSMap version 21 2024-04-23T10:27:35.156 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:27:35.152+0000 7fae1e9d2740 10 client.15696 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-04-23T10:27:35.156 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:27:35.152+0000 7fae1e9d2740 20 client.15696 populate_metadata read hostname 'smithi114' 2024-04-23T10:27:35.156 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:27:35.152+0000 7fae1e9d2740 1 --2- 172.21.15.114:0/208513149 >> [v2:172.21.15.119:6838/937297340,v1:172.21.15.119:6839/937297340] conn(0x55e3112766e0 0x55e311278ad0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T10:27:35.156 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:27:35.152+0000 7fae1e9d2740 4 client.15696 mds_command: new command op to 24545 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-23T10:27:35.156 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:27:35.152+0000 7fae1e9d2740 1 -- 172.21.15.114:0/208513149 --> [v2:172.21.15.119:6838/937297340,v1:172.21.15.119:6839/937297340] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55e3111c8200 con 0x55e3112766e0 2024-04-23T10:27:35.157 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:27:35.152+0000 7fae1997e640 1 --2- 172.21.15.114:0/208513149 >> [v2:172.21.15.119:6838/937297340,v1:172.21.15.119:6839/937297340] conn(0x55e3112766e0 0x55e311278ad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T10:27:35.157 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:27:35.153+0000 7fadfd7fa640 10 client.15696 ms_handle_connect on v2:172.21.15.119:6838/937297340 2024-04-23T10:27:35.157 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:27:35.153+0000 7fae1997e640 1 --2- 172.21.15.114:0/208513149 >> [v2:172.21.15.119:6838/937297340,v1:172.21.15.119:6839/937297340] conn(0x55e3112766e0 0x55e311278ad0 crc :-1 s=READY pgs=52 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=631de69273a34bc3 server_cookie=bad18431b7389946 in_seq=0 out_seq=0 2024-04-23T10:27:35.157 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:27:35.153+0000 7fadfd7fa640 1 -- 172.21.15.114:0/208513149 <== mds.0 v2:172.21.15.119:6838/937297340 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x55e3111c8200 con 0x55e3112766e0 2024-04-23T10:27:35.157 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:27:35.153+0000 7fadfd7fa640 10 client.15696 handle_command_reply: tid=0 2024-04-23T10:27:35.157 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:27:35.153+0000 7fae1e9d2740 1 client.15696 shutdown 2024-04-23T10:27:35.157 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:27:35.153+0000 7fae1e9d2740 1 -- 172.21.15.114:0/208513149 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55e311266800 msgr2=0x55e311263a30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T10:27:35.157 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:27:35.153+0000 7fae1e9d2740 1 --2- 172.21.15.114:0/208513149 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55e311266800 0x55e311263a30 secure :-1 s=READY pgs=527 cs=0 l=1 rev1=1 crypto rx=0x7fae0c011cf0 tx=0x7fae0c00f790 comp rx=0 tx=0).stop 2024-04-23T10:27:35.157 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:27:35.153+0000 7fae1e9d2740 1 -- 172.21.15.114:0/208513149 shutdown_connections 2024-04-23T10:27:35.157 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:27:35.153+0000 7fae1e9d2740 1 --2- 172.21.15.114:0/208513149 >> [v2:172.21.15.119:6838/937297340,v1:172.21.15.119:6839/937297340] conn(0x55e3112766e0 0x55e311278ad0 crc :-1 s=READY pgs=52 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:27:35.157 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:27:35.154+0000 7fadfd7fa640 0 client.15696 ms_handle_reset on v2:172.21.15.119:6838/937297340 2024-04-23T10:27:35.157 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:27:35.153+0000 7fae1e9d2740 1 --2- 172.21.15.114:0/208513149 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55e311266800 0x55e311263a30 unknown :-1 s=CLOSED pgs=527 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:27:35.157 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:27:35.154+0000 7fae1e9d2740 1 --2- 172.21.15.114:0/208513149 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x55e31125bf40 0x55e311263490 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:27:35.157 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:27:35.154+0000 7fae1e9d2740 1 --2- 172.21.15.114:0/208513149 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x55e31125b620 0x55e3112661a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:27:35.157 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:27:35.154+0000 7fae1e9d2740 1 -- 172.21.15.114:0/208513149 >> 172.21.15.114:0/208513149 conn(0x55e311257b30 msgr2=0x55e31125d8b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T10:27:35.158 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:27:35.154+0000 7fae1e9d2740 1 -- 172.21.15.114:0/208513149 shutdown_connections 2024-04-23T10:27:35.158 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:27:35.154+0000 7fae1e9d2740 1 -- 172.21.15.114:0/208513149 wait complete. 2024-04-23T10:27:35.158 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:27:35.154+0000 7fae1e9d2740 20 client.15696 trim_cache size 0 max 16384 2024-04-23T10:27:35.170 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:27:35.170+0000 7fae1e9d2740 1 Processor -- start 2024-04-23T10:27:35.170 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:27:35.170+0000 7fae1e9d2740 1 -- start start 2024-04-23T10:27:35.170 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:27:35.170+0000 7fae1e9d2740 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x55e31125b620 0x55e3112722f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T10:27:35.170 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:27:35.170+0000 7fae1e9d2740 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55e31125bf40 0x55e31125e850 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T10:27:35.171 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:27:35.170+0000 7fae1e9d2740 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x55e311266800 0x55e311265cd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T10:27:35.171 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:27:35.170+0000 7fae1e9d2740 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x7fae141b2620 con 0x55e31125bf40 2024-04-23T10:27:35.171 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:27:35.170+0000 7fae1e9d2740 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 -- 0x7fae141afef0 con 0x55e31125b620 2024-04-23T10:27:35.171 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:27:35.170+0000 7fae1e9d2740 1 -- --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_getmap magic: 0 -- 0x7fae141add40 con 0x55e311266800 2024-04-23T10:27:35.171 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:27:35.170+0000 7fae1917d640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55e31125bf40 0x55e31125e850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T10:27:35.171 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:27:35.170+0000 7fae1917d640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55e31125bf40 0x55e31125e850 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.114:53660/0 (socket says 172.21.15.114:53660) 2024-04-23T10:27:35.171 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:27:35.170+0000 7fae1917d640 1 -- 172.21.15.114:0/3432568010 learned_addr learned my addr 172.21.15.114:0/3432568010 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-04-23T10:27:35.171 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:27:35.171+0000 7fae1997e640 1 --2- 172.21.15.114:0/3432568010 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x55e31125b620 0x55e3112722f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T10:27:35.171 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:27:35.171+0000 7fae1917d640 1 -- 172.21.15.114:0/3432568010 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x55e311266800 msgr2=0x55e311265cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T10:27:35.171 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:27:35.171+0000 7fae1917d640 1 --2- 172.21.15.114:0/3432568010 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x55e311266800 0x55e311265cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:27:35.171 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:27:35.171+0000 7fae1917d640 1 -- 172.21.15.114:0/3432568010 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x55e31125b620 msgr2=0x55e3112722f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T10:27:35.171 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:27:35.171+0000 7fae1917d640 1 --2- 172.21.15.114:0/3432568010 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x55e31125b620 0x55e3112722f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:27:35.171 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:27:35.171+0000 7fae1917d640 1 -- 172.21.15.114:0/3432568010 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55e3112619e0 con 0x55e31125bf40 2024-04-23T10:27:35.171 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:27:35.171+0000 7fae1917d640 1 --2- 172.21.15.114:0/3432568010 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55e31125bf40 0x55e31125e850 secure :-1 s=READY pgs=528 cs=0 l=1 rev1=1 crypto rx=0x7fae040bcda0 tx=0x7fae04031f50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ddc56b3d20f4ec40 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T10:27:35.171 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:27:35.171+0000 7fadff7fe640 1 -- 172.21.15.114:0/3432568010 <== mon.0 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fae04049040 con 0x55e31125bf40 2024-04-23T10:27:35.172 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:27:35.171+0000 7fadff7fe640 1 -- 172.21.15.114:0/3432568010 <== mon.0 v2:172.21.15.114:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fae040c1110 con 0x55e31125bf40 2024-04-23T10:27:35.172 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:27:35.171+0000 7fadff7fe640 1 -- 172.21.15.114:0/3432568010 <== mon.0 v2:172.21.15.114:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fae040bd070 con 0x55e31125bf40 2024-04-23T10:27:35.175 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:27:35.172+0000 7fae1e9d2740 1 -- 172.21.15.114:0/3432568010 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55e31125bf40 msgr2=0x55e31125e850 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T10:27:35.175 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:27:35.172+0000 7fae1e9d2740 1 --2- 172.21.15.114:0/3432568010 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55e31125bf40 0x55e31125e850 secure :-1 s=READY pgs=528 cs=0 l=1 rev1=1 crypto rx=0x7fae040bcda0 tx=0x7fae04031f50 comp rx=0 tx=0).stop 2024-04-23T10:27:35.175 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:27:35.172+0000 7fae1e9d2740 1 -- 172.21.15.114:0/3432568010 shutdown_connections 2024-04-23T10:27:35.175 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:27:35.172+0000 7fae1e9d2740 1 --2- 172.21.15.114:0/3432568010 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x55e311266800 0x55e311265cd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:27:35.175 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:27:35.172+0000 7fae1e9d2740 1 --2- 172.21.15.114:0/3432568010 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55e31125bf40 0x55e31125e850 unknown :-1 s=CLOSED pgs=528 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:27:35.175 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:27:35.172+0000 7fae1e9d2740 1 --2- 172.21.15.114:0/3432568010 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x55e31125b620 0x55e3112722f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:27:35.175 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:27:35.172+0000 7fae1e9d2740 1 -- 172.21.15.114:0/3432568010 >> 172.21.15.114:0/3432568010 conn(0x55e311257b30 msgr2=0x55e31125e140 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T10:27:35.175 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:27:35.172+0000 7fae1e9d2740 1 -- 172.21.15.114:0/3432568010 shutdown_connections 2024-04-23T10:27:35.176 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:27:35.172+0000 7fae1e9d2740 1 -- 172.21.15.114:0/3432568010 wait complete. 2024-04-23T10:27:35.176 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:27:35.172+0000 7fae1e9d2740 1 -- start start 2024-04-23T10:27:35.176 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:27:35.173+0000 7fae1e9d2740 1 Processor -- start 2024-04-23T10:27:35.176 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:27:35.173+0000 7fae1e9d2740 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55e31125b620 0x55e31126a280 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T10:27:35.176 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:27:35.173+0000 7fae1e9d2740 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x55e31125bf40 0x55e31126a7c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T10:27:35.176 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:27:35.173+0000 7fae1e9d2740 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x55e311266800 0x55e31126ad00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T10:27:35.176 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:27:35.173+0000 7fae1e9d2740 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x7fae141ae3b0 con 0x55e31125b620 2024-04-23T10:27:35.176 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:27:35.173+0000 7fae1e9d2740 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 -- 0x7fae141af820 con 0x55e311266800 2024-04-23T10:27:35.176 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:27:35.173+0000 7fae1e9d2740 1 -- --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_getmap magic: 0 -- 0x7fae141aea80 con 0x55e31125bf40 2024-04-23T10:27:35.176 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:27:35.173+0000 7fae1997e640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55e31125b620 0x55e31126a280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T10:27:35.176 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:27:35.173+0000 7fae1e9d2740 10 client.0 _ll_register_callbacks cb 0x7fae0c011900 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-23T10:27:35.176 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:27:35.173+0000 7fae1997e640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55e31125b620 0x55e31126a280 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.114:53670/0 (socket says 172.21.15.114:53670) 2024-04-23T10:27:35.176 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:27:35.173+0000 7fae1997e640 1 -- 172.21.15.114:0/279157301 learned_addr learned my addr 172.21.15.114:0/279157301 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-04-23T10:27:35.176 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:27:35.173+0000 7fae1917d640 1 --2- 172.21.15.114:0/279157301 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x55e31125bf40 0x55e31126a7c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T10:27:35.176 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:27:35.173+0000 7fae1a17f640 1 --2- 172.21.15.114:0/279157301 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x55e311266800 0x55e31126ad00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T10:27:35.176 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:27:35.173+0000 7fae1997e640 1 -- 172.21.15.114:0/279157301 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x55e31125bf40 msgr2=0x55e31126a7c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T10:27:35.177 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:27:35.173+0000 7fae1997e640 1 --2- 172.21.15.114:0/279157301 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x55e31125bf40 0x55e31126a7c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:27:35.177 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:27:35.173+0000 7fae1997e640 1 -- 172.21.15.114:0/279157301 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x55e311266800 msgr2=0x55e31126ad00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T10:27:35.177 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:27:35.173+0000 7fae1997e640 1 --2- 172.21.15.114:0/279157301 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x55e311266800 0x55e31126ad00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:27:35.177 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:27:35.174+0000 7fae1997e640 1 -- 172.21.15.114:0/279157301 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55e3112dbc60 con 0x55e31125b620 2024-04-23T10:27:35.177 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:27:35.174+0000 7fae1a17f640 1 --2- 172.21.15.114:0/279157301 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x55e311266800 0x55e31126ad00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-23T10:27:35.177 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:27:35.174+0000 7fae1997e640 1 --2- 172.21.15.114:0/279157301 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55e31125b620 0x55e31126a280 secure :-1 s=READY pgs=529 cs=0 l=1 rev1=1 crypto rx=0x7fae1000bd80 tx=0x7fae1001cb20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=aee8dfa439c45202 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T10:27:35.177 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:27:35.174+0000 7fadfd7fa640 10 client.0 ms_handle_connect on v2:172.21.15.114:3300/0 2024-04-23T10:27:35.177 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:27:35.174+0000 7fadfd7fa640 1 -- 172.21.15.114:0/279157301 <== mon.0 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fae10005240 con 0x55e31125b620 2024-04-23T10:27:35.177 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:27:35.174+0000 7fadfd7fa640 1 -- 172.21.15.114:0/279157301 <== mon.0 v2:172.21.15.114:3300/0 2 ==== osd_map(90..90 src has 1..90) ==== 8527+0+0 (secure 0 0 0) 0x7fae100053e0 con 0x55e31125b620 2024-04-23T10:27:35.177 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:27:35.174+0000 7fae1e9d2740 1 -- 172.21.15.114:0/279157301 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fae040c1110 con 0x55e31125b620 2024-04-23T10:27:35.177 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:27:35.174+0000 7fadfd7fa640 1 -- 172.21.15.114:0/279157301 <== mon.0 v2:172.21.15.114:3300/0 3 ==== mon_get_version_reply(handle=1 version=21) ==== 24+0+0 (secure 0 0 0) 0x7fae100114b0 con 0x55e31125b620 2024-04-23T10:27:35.177 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:27:35.174+0000 7fae1e9d2740 10 client.15702 fetch_fsmap learned FSMap version 21 2024-04-23T10:27:35.177 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:27:35.174+0000 7fae1e9d2740 1 -- 172.21.15.114:0/279157301 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({config=0+,fsmap=21,monmap=4+}) -- 0x55e3112dcf30 con 0x55e31125b620 2024-04-23T10:27:35.177 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:27:35.174+0000 7fadfd7fa640 1 -- 172.21.15.114:0/279157301 <== mon.0 v2:172.21.15.114:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fae040c1110 con 0x55e31125b620 2024-04-23T10:27:35.177 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:27:35.175+0000 7fadfd7fa640 1 -- 172.21.15.114:0/279157301 <== mon.0 v2:172.21.15.114:3300/0 5 ==== fsmap(e 21) ==== 8338+0+0 (secure 0 0 0) 0x7fae10004660 con 0x55e31125b620 2024-04-23T10:27:35.178 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:27:35.175+0000 7fae1e9d2740 10 client.15702 fetch_fsmap finished waiting for FSMap version 21 2024-04-23T10:27:35.178 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:27:35.175+0000 7fae1e9d2740 10 client.15702 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-04-23T10:27:35.178 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:27:35.175+0000 7fae1e9d2740 20 client.15702 populate_metadata read hostname 'smithi114' 2024-04-23T10:27:35.178 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:27:35.175+0000 7fae1e9d2740 1 --2- 172.21.15.114:0/279157301 >> [v2:172.21.15.119:6838/937297340,v1:172.21.15.119:6839/937297340] conn(0x55e3112766e0 0x55e3112df370 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T10:27:35.178 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:27:35.175+0000 7fae1e9d2740 4 client.15702 mds_command: new command op to 24545 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-04-23T10:27:35.178 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:27:35.175+0000 7fae1e9d2740 1 -- 172.21.15.114:0/279157301 --> [v2:172.21.15.119:6838/937297340,v1:172.21.15.119:6839/937297340] -- command(tid 0: {"prefix": "flush journal"}) -- 0x55e3111445f0 con 0x55e3112766e0 2024-04-23T10:27:35.178 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:27:35.175+0000 7fae1917d640 1 --2- 172.21.15.114:0/279157301 >> [v2:172.21.15.119:6838/937297340,v1:172.21.15.119:6839/937297340] conn(0x55e3112766e0 0x55e3112df370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T10:27:35.178 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:27:35.175+0000 7fadfd7fa640 10 client.15702 ms_handle_connect on v2:172.21.15.119:6838/937297340 2024-04-23T10:27:35.178 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:27:35.175+0000 7fae1917d640 1 --2- 172.21.15.114:0/279157301 >> [v2:172.21.15.119:6838/937297340,v1:172.21.15.119:6839/937297340] conn(0x55e3112766e0 0x55e3112df370 crc :-1 s=READY pgs=53 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=99b07cb5bcd797cd server_cookie=703090bd481c8a9a in_seq=0 out_seq=0 2024-04-23T10:27:35.179 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:27:35.179+0000 7fadfd7fa640 1 -- 172.21.15.114:0/279157301 <== mds.0 v2:172.21.15.119:6838/937297340 1 ==== command_reply(tid 0: 0 ) ==== 8+0+44 (crc 0 0 0) 0x55e3111445f0 con 0x55e3112766e0 2024-04-23T10:27:35.179 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:27:35.179+0000 7fadfd7fa640 10 client.15702 handle_command_reply: tid=0 2024-04-23T10:27:35.179 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:27:35.179+0000 7fae1e9d2740 1 client.15702 shutdown 2024-04-23T10:27:35.180 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:27:35.179+0000 7fae1e9d2740 1 -- 172.21.15.114:0/279157301 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55e31125b620 msgr2=0x55e31126a280 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T10:27:35.180 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:27:35.179+0000 7fae1e9d2740 1 --2- 172.21.15.114:0/279157301 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55e31125b620 0x55e31126a280 secure :-1 s=READY pgs=529 cs=0 l=1 rev1=1 crypto rx=0x7fae1000bd80 tx=0x7fae1001cb20 comp rx=0 tx=0).stop 2024-04-23T10:27:35.180 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:27:35.179+0000 7fae1e9d2740 1 -- 172.21.15.114:0/279157301 shutdown_connections 2024-04-23T10:27:35.180 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:27:35.180+0000 7fae1e9d2740 1 --2- 172.21.15.114:0/279157301 >> [v2:172.21.15.119:6838/937297340,v1:172.21.15.119:6839/937297340] conn(0x55e3112766e0 0x55e3112df370 crc :-1 s=READY pgs=53 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:27:35.180 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:27:35.180+0000 7fadfd7fa640 0 client.15702 ms_handle_reset on v2:172.21.15.119:6838/937297340 2024-04-23T10:27:35.180 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:27:35.180+0000 7fae1e9d2740 1 --2- 172.21.15.114:0/279157301 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x55e311266800 0x55e31126ad00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:27:35.180 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:27:35.180+0000 7fae1e9d2740 1 --2- 172.21.15.114:0/279157301 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x55e31125bf40 0x55e31126a7c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:27:35.180 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:27:35.180+0000 7fae1e9d2740 1 --2- 172.21.15.114:0/279157301 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55e31125b620 0x55e31126a280 unknown :-1 s=CLOSED pgs=529 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:27:35.180 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:27:35.180+0000 7fae1e9d2740 1 -- 172.21.15.114:0/279157301 >> 172.21.15.114:0/279157301 conn(0x55e311257b30 msgr2=0x55e31125d860 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T10:27:35.180 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:27:35.180+0000 7fae1e9d2740 1 -- 172.21.15.114:0/279157301 shutdown_connections 2024-04-23T10:27:35.180 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:27:35.180+0000 7fae1e9d2740 1 -- 172.21.15.114:0/279157301 wait complete. 2024-04-23T10:27:35.180 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:27:35.180+0000 7fae1e9d2740 20 client.15702 trim_cache size 0 max 16384 2024-04-23T10:27:35.181 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stdout:{ 2024-04-23T10:27:35.181 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stdout: "message": "", 2024-04-23T10:27:35.181 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stdout: "return_code": 0 2024-04-23T10:27:35.181 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stdout:} 2024-04-23T10:27:35.181 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:27:35.180+0000 7fae1bbe2640 1 -- 172.21.15.114:0/3354065237 >> [v2:172.21.15.114:6800/1395693466,v1:172.21.15.114:6801/1395693466] conn(0x7fae000a3f60 msgr2=0x7fae000a6420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T10:27:35.182 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:27:35.180+0000 7fae1bbe2640 1 --2- 172.21.15.114:0/3354065237 >> [v2:172.21.15.114:6800/1395693466,v1:172.21.15.114:6801/1395693466] conn(0x7fae000a3f60 0x7fae000a6420 secure :-1 s=READY pgs=71 cs=0 l=1 rev1=1 crypto rx=0x7fae1000bfd0 tx=0x7fae1000b390 comp rx=0 tx=0).stop 2024-04-23T10:27:35.182 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:27:35.180+0000 7fae1bbe2640 1 -- 172.21.15.114:0/3354065237 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fae14107280 msgr2=0x7fae14198c30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T10:27:35.182 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:27:35.180+0000 7fae1bbe2640 1 --2- 172.21.15.114:0/3354065237 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fae14107280 0x7fae14198c30 secure :-1 s=READY pgs=525 cs=0 l=1 rev1=1 crypto rx=0x7fae04005ab0 tx=0x7fae04003b90 comp rx=0 tx=0).stop 2024-04-23T10:27:35.182 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:27:35.180+0000 7fae1bbe2640 1 -- 172.21.15.114:0/3354065237 shutdown_connections 2024-04-23T10:27:35.182 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:27:35.180+0000 7fae1bbe2640 1 --2- 172.21.15.114:0/3354065237 >> [v2:172.21.15.114:6800/1395693466,v1:172.21.15.114:6801/1395693466] conn(0x7fae000a3f60 0x7fae000a6420 unknown :-1 s=CLOSED pgs=71 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:27:35.182 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:27:35.180+0000 7fae1bbe2640 1 --2- 172.21.15.114:0/3354065237 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fae1419cf60 0x7fae1419d410 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:27:35.182 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:27:35.180+0000 7fae1bbe2640 1 --2- 172.21.15.114:0/3354065237 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fae14107280 0x7fae14198c30 unknown :-1 s=CLOSED pgs=525 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:27:35.182 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:27:35.180+0000 7fae1bbe2640 1 --2- 172.21.15.114:0/3354065237 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fae141021b0 0x7fae141986f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:27:35.182 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:27:35.180+0000 7fae1bbe2640 1 -- 172.21.15.114:0/3354065237 >> 172.21.15.114:0/3354065237 conn(0x7fae140fda10 msgr2=0x7fae140ffd70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T10:27:35.182 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:27:35.180+0000 7fae1bbe2640 1 -- 172.21.15.114:0/3354065237 shutdown_connections 2024-04-23T10:27:35.182 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:27:35.180+0000 7fae1bbe2640 1 -- 172.21.15.114:0/3354065237 wait complete. 2024-04-23T10:27:36.493 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 23 10:27:36 smithi119 ceph-mon[32608]: pgmap v30: 97 pgs: 97 active+clean; 1.3 MiB data, 377 MiB used, 1.0 TiB / 1.0 TiB avail; 2.6 KiB/s rd, 255 B/s wr, 5 op/s 2024-04-23T10:27:36.504 INFO:journalctl@ceph.mon.a.smithi114.stdout:Apr 23 10:27:36 smithi114 ceph-mon[29255]: pgmap v30: 97 pgs: 97 active+clean; 1.3 MiB data, 377 MiB used, 1.0 TiB / 1.0 TiB avail; 2.6 KiB/s rd, 255 B/s wr, 5 op/s 2024-04-23T10:27:36.578 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 23 10:27:36 smithi204 ceph-mon[32649]: pgmap v30: 97 pgs: 97 active+clean; 1.3 MiB data, 377 MiB used, 1.0 TiB / 1.0 TiB avail; 2.6 KiB/s rd, 255 B/s wr, 5 op/s 2024-04-23T10:27:38.492 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 23 10:27:38 smithi119 ceph-mon[32608]: pgmap v31: 97 pgs: 97 active+clean; 1.3 MiB data, 377 MiB used, 1.0 TiB / 1.0 TiB avail; 3.0 KiB/s rd, 255 B/s wr, 5 op/s 2024-04-23T10:27:38.504 INFO:journalctl@ceph.mon.a.smithi114.stdout:Apr 23 10:27:38 smithi114 ceph-mon[29255]: pgmap v31: 97 pgs: 97 active+clean; 1.3 MiB data, 377 MiB used, 1.0 TiB / 1.0 TiB avail; 3.0 KiB/s rd, 255 B/s wr, 5 op/s 2024-04-23T10:27:38.578 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 23 10:27:38 smithi204 ceph-mon[32649]: pgmap v31: 97 pgs: 97 active+clean; 1.3 MiB data, 377 MiB used, 1.0 TiB / 1.0 TiB avail; 3.0 KiB/s rd, 255 B/s wr, 5 op/s 2024-04-23T10:27:40.492 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 23 10:27:40 smithi119 ceph-mon[32608]: pgmap v32: 97 pgs: 97 active+clean; 1.3 MiB data, 377 MiB used, 1.0 TiB / 1.0 TiB avail; 3.5 KiB/s rd, 511 B/s wr, 7 op/s 2024-04-23T10:27:40.504 INFO:journalctl@ceph.mon.a.smithi114.stdout:Apr 23 10:27:40 smithi114 ceph-mon[29255]: pgmap v32: 97 pgs: 97 active+clean; 1.3 MiB data, 377 MiB used, 1.0 TiB / 1.0 TiB avail; 3.5 KiB/s rd, 511 B/s wr, 7 op/s 2024-04-23T10:27:40.578 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 23 10:27:40 smithi204 ceph-mon[32649]: pgmap v32: 97 pgs: 97 active+clean; 1.3 MiB data, 377 MiB used, 1.0 TiB / 1.0 TiB avail; 3.5 KiB/s rd, 511 B/s wr, 7 op/s 2024-04-23T10:27:41.492 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 23 10:27:41 smithi119 ceph-mon[32608]: from='mgr.15291 172.21.15.114:0/1095789187' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.b", "id": [8, 11]}]: dispatch 2024-04-23T10:27:41.492 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 23 10:27:41 smithi119 ceph-mon[32608]: from='mgr.15291 172.21.15.114:0/1095789187' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.10", "id": [2, 1]}]: dispatch 2024-04-23T10:27:41.504 INFO:journalctl@ceph.mon.a.smithi114.stdout:Apr 23 10:27:41 smithi114 ceph-mon[29255]: from='mgr.15291 172.21.15.114:0/1095789187' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.b", "id": [8, 11]}]: dispatch 2024-04-23T10:27:41.504 INFO:journalctl@ceph.mon.a.smithi114.stdout:Apr 23 10:27:41 smithi114 ceph-mon[29255]: from='mgr.15291 172.21.15.114:0/1095789187' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.10", "id": [2, 1]}]: dispatch 2024-04-23T10:27:41.577 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 23 10:27:41 smithi204 ceph-mon[32649]: from='mgr.15291 172.21.15.114:0/1095789187' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.b", "id": [8, 11]}]: dispatch 2024-04-23T10:27:41.578 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 23 10:27:41 smithi204 ceph-mon[32649]: from='mgr.15291 172.21.15.114:0/1095789187' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.10", "id": [2, 1]}]: dispatch 2024-04-23T10:27:41.617 INFO:tasks.workunit.client.0.smithi114.stderr:Note: switching to '0a6c3ed699031b80a2b419e7e795368719871394'. 2024-04-23T10:27:41.617 INFO:tasks.workunit.client.0.smithi114.stderr: 2024-04-23T10:27:41.617 INFO:tasks.workunit.client.0.smithi114.stderr:You are in 'detached HEAD' state. You can look around, make experimental 2024-04-23T10:27:41.617 INFO:tasks.workunit.client.0.smithi114.stderr:changes and commit them, and you can discard any commits you make in this 2024-04-23T10:27:41.617 INFO:tasks.workunit.client.0.smithi114.stderr:state without impacting any branches by switching back to a branch. 2024-04-23T10:27:41.617 INFO:tasks.workunit.client.0.smithi114.stderr: 2024-04-23T10:27:41.617 INFO:tasks.workunit.client.0.smithi114.stderr:If you want to create a new branch to retain commits you create, you may 2024-04-23T10:27:41.617 INFO:tasks.workunit.client.0.smithi114.stderr:do so (now or later) by using -c with the switch command. Example: 2024-04-23T10:27:41.617 INFO:tasks.workunit.client.0.smithi114.stderr: 2024-04-23T10:27:41.617 INFO:tasks.workunit.client.0.smithi114.stderr: git switch -c 2024-04-23T10:27:41.617 INFO:tasks.workunit.client.0.smithi114.stderr: 2024-04-23T10:27:41.617 INFO:tasks.workunit.client.0.smithi114.stderr:Or undo this operation with: 2024-04-23T10:27:41.617 INFO:tasks.workunit.client.0.smithi114.stderr: 2024-04-23T10:27:41.618 INFO:tasks.workunit.client.0.smithi114.stderr: git switch - 2024-04-23T10:27:41.618 INFO:tasks.workunit.client.0.smithi114.stderr: 2024-04-23T10:27:41.618 INFO:tasks.workunit.client.0.smithi114.stderr:Turn off this advice by setting config variable advice.detachedHead to false 2024-04-23T10:27:41.618 INFO:tasks.workunit.client.0.smithi114.stderr: 2024-04-23T10:27:41.618 INFO:tasks.workunit.client.0.smithi114.stderr:HEAD is now at 0a6c3ed6990 qa/suites/fs: add quiescer to the fs suite 2024-04-23T10:27:41.619 DEBUG:teuthology.orchestra.run.smithi114:> cd -- /home/ubuntu/cephtest/clone.client.0/qa/workunits && if test -e Makefile ; then make ; fi && find -executable -type f -printf '%P\0' >/home/ubuntu/cephtest/workunits.list.client.0 2024-04-23T10:27:41.658 INFO:tasks.workunit.client.0.smithi114.stdout:for d in direct_io fs ; do ( cd $d ; make all ) ; done 2024-04-23T10:27:41.660 INFO:tasks.workunit.client.0.smithi114.stdout:make[1]: Entering directory '/home/ubuntu/cephtest/clone.client.0/qa/workunits/direct_io' 2024-04-23T10:27:41.660 INFO:tasks.workunit.client.0.smithi114.stdout:cc -Wall -Wextra -D_GNU_SOURCE direct_io_test.c -o direct_io_test 2024-04-23T10:27:42.287 INFO:tasks.workunit.client.0.smithi114.stdout:cc -Wall -Wextra -D_GNU_SOURCE test_sync_io.c -o test_sync_io 2024-04-23T10:27:42.343 INFO:tasks.workunit.client.0.smithi114.stdout:cc -Wall -Wextra -D_GNU_SOURCE test_short_dio_read.c -o test_short_dio_read 2024-04-23T10:27:42.383 INFO:tasks.workunit.client.0.smithi114.stdout:make[1]: Leaving directory '/home/ubuntu/cephtest/clone.client.0/qa/workunits/direct_io' 2024-04-23T10:27:42.384 INFO:tasks.workunit.client.0.smithi114.stdout:make[1]: Entering directory '/home/ubuntu/cephtest/clone.client.0/qa/workunits/fs' 2024-04-23T10:27:42.384 INFO:tasks.workunit.client.0.smithi114.stdout:cc -Wall -Wextra -D_GNU_SOURCE test_o_trunc.c -o test_o_trunc 2024-04-23T10:27:42.422 INFO:tasks.workunit.client.0.smithi114.stdout:make[1]: Leaving directory '/home/ubuntu/cephtest/clone.client.0/qa/workunits/fs' 2024-04-23T10:27:42.425 DEBUG:teuthology.orchestra.run.smithi114:> set -ex 2024-04-23T10:27:42.425 DEBUG:teuthology.orchestra.run.smithi114:> dd if=/home/ubuntu/cephtest/workunits.list.client.0 of=/dev/stdout 2024-04-23T10:27:42.482 INFO:tasks.workunit:Running workunits matching suites/dbench.sh on client.0... 2024-04-23T10:27:42.483 INFO:tasks.workunit:Running workunit suites/dbench.sh... 2024-04-23T10:27:42.484 DEBUG:teuthology.orchestra.run.smithi114:workunit test suites/dbench.sh> mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=0a6c3ed699031b80a2b419e7e795368719871394 TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 CEPH_MNT=/home/ubuntu/cephtest/mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clone.client.0/qa/workunits/suites/dbench.sh 2024-04-23T10:27:42.492 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 23 10:27:42 smithi119 ceph-mon[32608]: pgmap v33: 97 pgs: 97 active+clean; 1.3 MiB data, 377 MiB used, 1.0 TiB / 1.0 TiB avail; 2.6 KiB/s rd, 255 B/s wr, 5 op/s 2024-04-23T10:27:42.492 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 23 10:27:42 smithi119 ceph-mon[32608]: from='mgr.15291 172.21.15.114:0/1095789187' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.b", "id": [8, 11]}]': finished 2024-04-23T10:27:42.492 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 23 10:27:42 smithi119 ceph-mon[32608]: from='mgr.15291 172.21.15.114:0/1095789187' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.10", "id": [2, 1]}]': finished 2024-04-23T10:27:42.492 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 23 10:27:42 smithi119 ceph-mon[32608]: osdmap e91: 12 total, 12 up, 12 in 2024-04-23T10:27:42.503 INFO:journalctl@ceph.mon.a.smithi114.stdout:Apr 23 10:27:42 smithi114 ceph-mon[29255]: pgmap v33: 97 pgs: 97 active+clean; 1.3 MiB data, 377 MiB used, 1.0 TiB / 1.0 TiB avail; 2.6 KiB/s rd, 255 B/s wr, 5 op/s 2024-04-23T10:27:42.504 INFO:journalctl@ceph.mon.a.smithi114.stdout:Apr 23 10:27:42 smithi114 ceph-mon[29255]: from='mgr.15291 172.21.15.114:0/1095789187' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.b", "id": [8, 11]}]': finished 2024-04-23T10:27:42.504 INFO:journalctl@ceph.mon.a.smithi114.stdout:Apr 23 10:27:42 smithi114 ceph-mon[29255]: from='mgr.15291 172.21.15.114:0/1095789187' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.10", "id": [2, 1]}]': finished 2024-04-23T10:27:42.504 INFO:journalctl@ceph.mon.a.smithi114.stdout:Apr 23 10:27:42 smithi114 ceph-mon[29255]: osdmap e91: 12 total, 12 up, 12 in 2024-04-23T10:27:42.578 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 23 10:27:42 smithi204 ceph-mon[32649]: pgmap v33: 97 pgs: 97 active+clean; 1.3 MiB data, 377 MiB used, 1.0 TiB / 1.0 TiB avail; 2.6 KiB/s rd, 255 B/s wr, 5 op/s 2024-04-23T10:27:42.578 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 23 10:27:42 smithi204 ceph-mon[32649]: from='mgr.15291 172.21.15.114:0/1095789187' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.b", "id": [8, 11]}]': finished 2024-04-23T10:27:42.578 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 23 10:27:42 smithi204 ceph-mon[32649]: from='mgr.15291 172.21.15.114:0/1095789187' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.10", "id": [2, 1]}]': finished 2024-04-23T10:27:42.578 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 23 10:27:42 smithi204 ceph-mon[32649]: osdmap e91: 12 total, 12 up, 12 in 2024-04-23T10:27:43.492 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 23 10:27:43 smithi119 ceph-mon[32608]: osdmap e92: 12 total, 12 up, 12 in 2024-04-23T10:27:43.504 INFO:journalctl@ceph.mon.a.smithi114.stdout:Apr 23 10:27:43 smithi114 ceph-mon[29255]: osdmap e92: 12 total, 12 up, 12 in 2024-04-23T10:27:43.577 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 23 10:27:43 smithi204 ceph-mon[32649]: osdmap e92: 12 total, 12 up, 12 in 2024-04-23T10:27:44.492 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 23 10:27:44 smithi119 ceph-mon[32608]: pgmap v36: 97 pgs: 2 peering, 95 active+clean; 1.3 MiB data, 377 MiB used, 1.0 TiB / 1.0 TiB avail; 3.9 KiB/s rd, 383 B/s wr, 8 op/s 2024-04-23T10:27:44.504 INFO:journalctl@ceph.mon.a.smithi114.stdout:Apr 23 10:27:44 smithi114 ceph-mon[29255]: pgmap v36: 97 pgs: 2 peering, 95 active+clean; 1.3 MiB data, 377 MiB used, 1.0 TiB / 1.0 TiB avail; 3.9 KiB/s rd, 383 B/s wr, 8 op/s 2024-04-23T10:27:44.577 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 23 10:27:44 smithi204 ceph-mon[32649]: pgmap v36: 97 pgs: 2 peering, 95 active+clean; 1.3 MiB data, 377 MiB used, 1.0 TiB / 1.0 TiB avail; 3.9 KiB/s rd, 383 B/s wr, 8 op/s 2024-04-23T10:27:45.492 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 23 10:27:45 smithi119 ceph-mon[32608]: from='mgr.15291 172.21.15.114:0/1095789187' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-23T10:27:45.504 INFO:journalctl@ceph.mon.a.smithi114.stdout:Apr 23 10:27:45 smithi114 ceph-mon[29255]: from='mgr.15291 172.21.15.114:0/1095789187' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-23T10:27:45.577 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 23 10:27:45 smithi204 ceph-mon[32649]: from='mgr.15291 172.21.15.114:0/1095789187' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-23T10:27:46.405 INFO:journalctl@ceph.mon.a.smithi114.stdout:Apr 23 10:27:46 smithi114 ceph-mon[29255]: pgmap v37: 97 pgs: 2 peering, 95 active+clean; 1.3 MiB data, 377 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 383 B/s wr, 7 op/s 2024-04-23T10:27:46.405 INFO:journalctl@ceph.mon.a.smithi114.stdout:Apr 23 10:27:46 smithi114 ceph-mon[29255]: from='mgr.15291 172.21.15.114:0/1095789187' entity='mgr.x' 2024-04-23T10:27:46.405 INFO:journalctl@ceph.mon.a.smithi114.stdout:Apr 23 10:27:46 smithi114 ceph-mon[29255]: from='mgr.15291 172.21.15.114:0/1095789187' entity='mgr.x' 2024-04-23T10:27:46.492 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 23 10:27:46 smithi119 ceph-mon[32608]: pgmap v37: 97 pgs: 2 peering, 95 active+clean; 1.3 MiB data, 377 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 383 B/s wr, 7 op/s 2024-04-23T10:27:46.492 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 23 10:27:46 smithi119 ceph-mon[32608]: from='mgr.15291 172.21.15.114:0/1095789187' entity='mgr.x' 2024-04-23T10:27:46.492 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 23 10:27:46 smithi119 ceph-mon[32608]: from='mgr.15291 172.21.15.114:0/1095789187' entity='mgr.x' 2024-04-23T10:27:46.577 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 23 10:27:46 smithi204 ceph-mon[32649]: pgmap v37: 97 pgs: 2 peering, 95 active+clean; 1.3 MiB data, 377 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 383 B/s wr, 7 op/s 2024-04-23T10:27:46.578 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 23 10:27:46 smithi204 ceph-mon[32649]: from='mgr.15291 172.21.15.114:0/1095789187' entity='mgr.x' 2024-04-23T10:27:46.578 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 23 10:27:46 smithi204 ceph-mon[32649]: from='mgr.15291 172.21.15.114:0/1095789187' entity='mgr.x' 2024-04-23T10:27:47.492 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 23 10:27:47 smithi119 ceph-mon[32608]: from='mgr.15291 172.21.15.114:0/1095789187' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-23T10:27:47.492 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 23 10:27:47 smithi119 ceph-mon[32608]: from='mgr.15291 172.21.15.114:0/1095789187' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-23T10:27:47.492 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 23 10:27:47 smithi119 ceph-mon[32608]: from='mgr.15291 172.21.15.114:0/1095789187' entity='mgr.x' 2024-04-23T10:27:47.504 INFO:journalctl@ceph.mon.a.smithi114.stdout:Apr 23 10:27:47 smithi114 ceph-mon[29255]: from='mgr.15291 172.21.15.114:0/1095789187' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-23T10:27:47.504 INFO:journalctl@ceph.mon.a.smithi114.stdout:Apr 23 10:27:47 smithi114 ceph-mon[29255]: from='mgr.15291 172.21.15.114:0/1095789187' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-23T10:27:47.504 INFO:journalctl@ceph.mon.a.smithi114.stdout:Apr 23 10:27:47 smithi114 ceph-mon[29255]: from='mgr.15291 172.21.15.114:0/1095789187' entity='mgr.x' 2024-04-23T10:27:47.578 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 23 10:27:47 smithi204 ceph-mon[32649]: from='mgr.15291 172.21.15.114:0/1095789187' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-23T10:27:47.578 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 23 10:27:47 smithi204 ceph-mon[32649]: from='mgr.15291 172.21.15.114:0/1095789187' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-23T10:27:47.578 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 23 10:27:47 smithi204 ceph-mon[32649]: from='mgr.15291 172.21.15.114:0/1095789187' entity='mgr.x' 2024-04-23T10:27:48.273 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:27:48.274+0000 7ff7d4dba640 1 -- 172.21.15.114:0/375471742 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7ff7d0103200 msgr2=0x7ff7d01035e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T10:27:48.273 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:27:48.274+0000 7ff7d4dba640 1 --2- 172.21.15.114:0/375471742 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7ff7d0103200 0x7ff7d01035e0 secure :-1 s=READY pgs=530 cs=0 l=1 rev1=1 crypto rx=0x7ff7c4009900 tx=0x7ff7c402a480 comp rx=0 tx=0).stop 2024-04-23T10:27:48.273 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:27:48.274+0000 7ff7d4dba640 1 -- 172.21.15.114:0/375471742 shutdown_connections 2024-04-23T10:27:48.273 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:27:48.274+0000 7ff7d4dba640 1 --2- 172.21.15.114:0/375471742 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7ff7d010c540 0x7ff7d010e9b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:27:48.274 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:27:48.274+0000 7ff7d4dba640 1 --2- 172.21.15.114:0/375471742 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7ff7d0103b20 0x7ff7d010c000 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:27:48.274 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:27:48.274+0000 7ff7d4dba640 1 --2- 172.21.15.114:0/375471742 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7ff7d0103200 0x7ff7d01035e0 unknown :-1 s=CLOSED pgs=530 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:27:48.274 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:27:48.274+0000 7ff7d4dba640 1 -- 172.21.15.114:0/375471742 >> 172.21.15.114:0/375471742 conn(0x7ff7d0077fd0 msgr2=0x7ff7d00783e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T10:27:48.274 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:27:48.274+0000 7ff7d4dba640 1 -- 172.21.15.114:0/375471742 shutdown_connections 2024-04-23T10:27:48.274 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:27:48.274+0000 7ff7d4dba640 1 -- 172.21.15.114:0/375471742 wait complete. 2024-04-23T10:27:48.276 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:27:48.275+0000 7ff7d4dba640 1 Processor -- start 2024-04-23T10:27:48.276 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:27:48.275+0000 7ff7d4dba640 1 -- start start 2024-04-23T10:27:48.276 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:27:48.275+0000 7ff7d4dba640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7ff7d0103200 0x7ff7d0198720 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T10:27:48.276 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:27:48.275+0000 7ff7d4dba640 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7ff7d0103b20 0x7ff7d0198c60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T10:27:48.276 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:27:48.275+0000 7ff7d4dba640 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7ff7d010c540 0x7ff7d019cf90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T10:27:48.276 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:27:48.275+0000 7ff7d4dba640 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x7ff7d01121a0 con 0x7ff7d0103200 2024-04-23T10:27:48.276 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:27:48.275+0000 7ff7d4dba640 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 -- 0x7ff7d0112020 con 0x7ff7d0103b20 2024-04-23T10:27:48.276 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:27:48.275+0000 7ff7d4dba640 1 -- --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_getmap magic: 0 -- 0x7ff7d0112320 con 0x7ff7d010c540 2024-04-23T10:27:48.276 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:27:48.275+0000 7ff7ce59c640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7ff7d0103200 0x7ff7d0198720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T10:27:48.277 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:27:48.275+0000 7ff7ce59c640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7ff7d0103200 0x7ff7d0198720 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.114:37074/0 (socket says 172.21.15.114:37074) 2024-04-23T10:27:48.277 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:27:48.275+0000 7ff7ce59c640 1 -- 172.21.15.114:0/3937271828 learned_addr learned my addr 172.21.15.114:0/3937271828 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-04-23T10:27:48.277 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:27:48.275+0000 7ff7cdd9b640 1 --2- 172.21.15.114:0/3937271828 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7ff7d0103b20 0x7ff7d0198c60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T10:27:48.277 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:27:48.275+0000 7ff7ced9d640 1 --2- 172.21.15.114:0/3937271828 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7ff7d010c540 0x7ff7d019cf90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T10:27:48.277 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:27:48.275+0000 7ff7ce59c640 1 -- 172.21.15.114:0/3937271828 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7ff7d010c540 msgr2=0x7ff7d019cf90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T10:27:48.277 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:27:48.275+0000 7ff7ce59c640 1 --2- 172.21.15.114:0/3937271828 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7ff7d010c540 0x7ff7d019cf90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:27:48.277 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:27:48.275+0000 7ff7ce59c640 1 -- 172.21.15.114:0/3937271828 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7ff7d0103b20 msgr2=0x7ff7d0198c60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T10:27:48.277 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:27:48.275+0000 7ff7ce59c640 1 --2- 172.21.15.114:0/3937271828 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7ff7d0103b20 0x7ff7d0198c60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:27:48.277 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:27:48.276+0000 7ff7ce59c640 1 -- 172.21.15.114:0/3937271828 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff7d019d6c0 con 0x7ff7d0103200 2024-04-23T10:27:48.277 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:27:48.276+0000 7ff7ced9d640 1 --2- 172.21.15.114:0/3937271828 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7ff7d010c540 0x7ff7d019cf90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-23T10:27:48.277 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:27:48.276+0000 7ff7cdd9b640 1 --2- 172.21.15.114:0/3937271828 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7ff7d0103b20 0x7ff7d0198c60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-23T10:27:48.277 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:27:48.276+0000 7ff7ce59c640 1 --2- 172.21.15.114:0/3937271828 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7ff7d0103200 0x7ff7d0198720 secure :-1 s=READY pgs=531 cs=0 l=1 rev1=1 crypto rx=0x7ff7c4005940 tx=0x7ff7c40042f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T10:27:48.277 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:27:48.276+0000 7ff7bf7fe640 1 -- 172.21.15.114:0/3937271828 <== mon.0 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff7c403f030 con 0x7ff7d0103200 2024-04-23T10:27:48.277 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:27:48.276+0000 7ff7bf7fe640 1 -- 172.21.15.114:0/3937271828 <== mon.0 v2:172.21.15.114:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7ff7c4003cb0 con 0x7ff7d0103200 2024-04-23T10:27:48.277 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:27:48.276+0000 7ff7bf7fe640 1 -- 172.21.15.114:0/3937271828 <== mon.0 v2:172.21.15.114:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff7c4035070 con 0x7ff7d0103200 2024-04-23T10:27:48.278 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:27:48.276+0000 7ff7d4dba640 1 -- 172.21.15.114:0/3937271828 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7ff7d019d950 con 0x7ff7d0103200 2024-04-23T10:27:48.278 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:27:48.276+0000 7ff7d4dba640 1 -- 172.21.15.114:0/3937271828 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7ff7d01a5240 con 0x7ff7d0103200 2024-04-23T10:27:48.283 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:27:48.278+0000 7ff7bf7fe640 1 -- 172.21.15.114:0/3937271828 <== mon.0 v2:172.21.15.114:3300/0 4 ==== mgrmap(e 25) ==== 138880+0+0 (secure 0 0 0) 0x7ff7c4042030 con 0x7ff7d0103200 2024-04-23T10:27:48.283 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:27:48.278+0000 7ff7d7ba8740 1 Processor -- start 2024-04-23T10:27:48.283 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:27:48.279+0000 7ff7d7ba8740 1 -- start start 2024-04-23T10:27:48.283 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:27:48.279+0000 7ff7d7ba8740 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x56483509c620 0x56483509ca00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T10:27:48.284 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:27:48.279+0000 7ff7bf7fe640 1 --2- 172.21.15.114:0/3937271828 >> [v2:172.21.15.114:6800/1395693466,v1:172.21.15.114:6801/1395693466] conn(0x7ff7b80a3ee0 0x7ff7b80a63a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T10:27:48.284 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:27:48.279+0000 7ff7d7ba8740 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x56483509cf40 0x5648350a72c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T10:27:48.284 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:27:48.279+0000 7ff7d7ba8740 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x5648350a7800 0x5648350a9bf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T10:27:48.284 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:27:48.279+0000 7ff7d7ba8740 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x564835001970 con 0x5648350a7800 2024-04-23T10:27:48.284 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:27:48.279+0000 7ff7d7ba8740 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 -- 0x564834f9e4a0 con 0x56483509c620 2024-04-23T10:27:48.284 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:27:48.279+0000 7ff7d7ba8740 1 -- --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_getmap magic: 0 -- 0x564834ffd930 con 0x56483509cf40 2024-04-23T10:27:48.284 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:27:48.279+0000 7ff7ce59c640 1 --2- 172.21.15.114:0/3937271828 >> [v2:172.21.15.114:6800/1395693466,v1:172.21.15.114:6801/1395693466] conn(0x7ff7b80a3ee0 0x7ff7b80a63a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T10:27:48.284 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:27:48.279+0000 7ff7ced9d640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x5648350a7800 0x5648350a9bf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T10:27:48.284 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:27:48.279+0000 7ff7cdd9b640 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x56483509cf40 0x5648350a72c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T10:27:48.284 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:27:48.279+0000 7ff7ced9d640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x5648350a7800 0x5648350a9bf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.114:37086/0 (socket says 172.21.15.114:37086) 2024-04-23T10:27:48.284 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:27:48.279+0000 7ff7cdd9b640 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x56483509cf40 0x5648350a72c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.204:3300/0 says I am v2:172.21.15.114:34452/0 (socket says 172.21.15.114:34452) 2024-04-23T10:27:48.284 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:27:48.279+0000 7ff7ced9d640 1 -- 172.21.15.114:0/3185887942 learned_addr learned my addr 172.21.15.114:0/3185887942 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-04-23T10:27:48.284 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:27:48.279+0000 7ff7bf7fe640 1 -- 172.21.15.114:0/3937271828 <== mon.0 v2:172.21.15.114:3300/0 5 ==== osd_map(92..92 src has 1..92) ==== 8585+0+0 (secure 0 0 0) 0x7ff7c40f1320 con 0x7ff7d0103200 2024-04-23T10:27:48.284 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:27:48.279+0000 7ff7ced9d640 1 --2- 172.21.15.114:0/3185887942 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x56483509c620 0x56483509ca00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T10:27:48.284 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:27:48.280+0000 7ff7ced9d640 1 -- 172.21.15.114:0/3185887942 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x56483509cf40 msgr2=0x5648350a72c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T10:27:48.285 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:27:48.280+0000 7ff7ced9d640 1 --2- 172.21.15.114:0/3185887942 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x56483509cf40 0x5648350a72c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:27:48.285 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:27:48.280+0000 7ff7ced9d640 1 -- 172.21.15.114:0/3185887942 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x56483509c620 msgr2=0x56483509ca00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T10:27:48.285 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:27:48.280+0000 7ff7ced9d640 1 --2- 172.21.15.114:0/3185887942 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x56483509c620 0x56483509ca00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:27:48.285 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:27:48.280+0000 7ff7ced9d640 1 -- 172.21.15.114:0/3185887942 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x564834d14210 con 0x5648350a7800 2024-04-23T10:27:48.285 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:27:48.280+0000 7ff7ce59c640 1 --2- 172.21.15.114:0/3937271828 >> [v2:172.21.15.114:6800/1395693466,v1:172.21.15.114:6801/1395693466] conn(0x7ff7b80a3ee0 0x7ff7b80a63a0 secure :-1 s=READY pgs=72 cs=0 l=1 rev1=1 crypto rx=0x7ff7c40ee910 tx=0x7ff7c4032f60 comp rx=0 tx=0).ready entity=mgr.15291 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T10:27:48.285 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:27:48.280+0000 7ff7ced9d640 1 --2- 172.21.15.114:0/3185887942 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x5648350a7800 0x5648350a9bf0 secure :-1 s=READY pgs=532 cs=0 l=1 rev1=1 crypto rx=0x7ff7c8014ac0 tx=0x7ff7c80096c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4e70a8efd304fc6f server_cookie=0 in_seq=0 out_seq=0 2024-04-23T10:27:48.285 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:27:48.280+0000 7ff7bcff9640 1 -- 172.21.15.114:0/3185887942 <== mon.0 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff7c80142d0 con 0x5648350a7800 2024-04-23T10:27:48.285 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:27:48.280+0000 7ff7bcff9640 1 -- 172.21.15.114:0/3185887942 <== mon.0 v2:172.21.15.114:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7ff7c800fd30 con 0x5648350a7800 2024-04-23T10:27:48.285 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:27:48.280+0000 7ff7bcff9640 1 -- 172.21.15.114:0/3185887942 <== mon.0 v2:172.21.15.114:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff7c8012b30 con 0x5648350a7800 2024-04-23T10:27:48.285 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:27:48.281+0000 7ff7d7ba8740 1 -- 172.21.15.114:0/3185887942 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x5648350a7800 msgr2=0x5648350a9bf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T10:27:48.285 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:27:48.281+0000 7ff7d7ba8740 1 --2- 172.21.15.114:0/3185887942 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x5648350a7800 0x5648350a9bf0 secure :-1 s=READY pgs=532 cs=0 l=1 rev1=1 crypto rx=0x7ff7c8014ac0 tx=0x7ff7c80096c0 comp rx=0 tx=0).stop 2024-04-23T10:27:48.285 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:27:48.281+0000 7ff7d7ba8740 1 -- 172.21.15.114:0/3185887942 shutdown_connections 2024-04-23T10:27:48.285 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:27:48.281+0000 7ff7d7ba8740 1 --2- 172.21.15.114:0/3185887942 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x5648350a7800 0x5648350a9bf0 unknown :-1 s=CLOSED pgs=532 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:27:48.285 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:27:48.281+0000 7ff7d7ba8740 1 --2- 172.21.15.114:0/3185887942 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x56483509cf40 0x5648350a72c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:27:48.286 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:27:48.281+0000 7ff7d7ba8740 1 --2- 172.21.15.114:0/3185887942 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x56483509c620 0x56483509ca00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:27:48.286 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:27:48.281+0000 7ff7d7ba8740 1 -- 172.21.15.114:0/3185887942 >> 172.21.15.114:0/3185887942 conn(0x564835098b30 msgr2=0x564835098f10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T10:27:48.286 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:27:48.281+0000 7ff7d7ba8740 1 -- 172.21.15.114:0/3185887942 shutdown_connections 2024-04-23T10:27:48.286 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:27:48.281+0000 7ff7d7ba8740 1 -- 172.21.15.114:0/3185887942 wait complete. 2024-04-23T10:27:48.286 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:27:48.281+0000 7ff7d7ba8740 1 -- start start 2024-04-23T10:27:48.286 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:27:48.281+0000 7ff7d7ba8740 1 Processor -- start 2024-04-23T10:27:48.286 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:27:48.281+0000 7ff7d7ba8740 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x56483509c620 0x5648350a7130 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T10:27:48.286 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:27:48.281+0000 7ff7d7ba8740 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x56483509cf40 0x5648350a4420 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T10:27:48.286 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:27:48.281+0000 7ff7d7ba8740 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x5648350a7800 0x5648350a4960 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T10:27:48.286 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:27:48.281+0000 7ff7d7ba8740 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x5648350aa430 con 0x56483509c620 2024-04-23T10:27:48.286 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:27:48.281+0000 7ff7d7ba8740 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 -- 0x564834f9c0b0 con 0x56483509cf40 2024-04-23T10:27:48.286 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:27:48.282+0000 7ff7d7ba8740 1 -- --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_getmap magic: 0 -- 0x5648350aa5b0 con 0x5648350a7800 2024-04-23T10:27:48.286 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:27:48.282+0000 7ff7d7ba8740 10 client.0 _ll_register_callbacks cb 0x564834f89d30 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-23T10:27:48.287 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:27:48.282+0000 7ff7ced9d640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x56483509c620 0x5648350a7130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T10:27:48.287 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:27:48.282+0000 7ff7ced9d640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x56483509c620 0x5648350a7130 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.114:37094/0 (socket says 172.21.15.114:37094) 2024-04-23T10:27:48.287 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:27:48.282+0000 7ff7ced9d640 1 -- 172.21.15.114:0/1198513842 learned_addr learned my addr 172.21.15.114:0/1198513842 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-04-23T10:27:48.287 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:27:48.282+0000 7ff7ced9d640 1 --2- 172.21.15.114:0/1198513842 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x5648350a7800 0x5648350a4960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T10:27:48.287 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:27:48.282+0000 7ff7cdd9b640 1 --2- 172.21.15.114:0/1198513842 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x56483509cf40 0x5648350a4420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T10:27:48.287 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:27:48.282+0000 7ff7ced9d640 1 -- 172.21.15.114:0/1198513842 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x5648350a7800 msgr2=0x5648350a4960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T10:27:48.287 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:27:48.282+0000 7ff7ced9d640 1 --2- 172.21.15.114:0/1198513842 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x5648350a7800 0x5648350a4960 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:27:48.287 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:27:48.282+0000 7ff7ced9d640 1 -- 172.21.15.114:0/1198513842 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x56483509cf40 msgr2=0x5648350a4420 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T10:27:48.287 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:27:48.282+0000 7ff7ced9d640 1 --2- 172.21.15.114:0/1198513842 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x56483509cf40 0x5648350a4420 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:27:48.287 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:27:48.282+0000 7ff7ced9d640 1 -- 172.21.15.114:0/1198513842 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({osdmap=0}) -- 0x564834de0620 con 0x56483509c620 2024-04-23T10:27:48.287 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:27:48.282+0000 7ff7ced9d640 1 --2- 172.21.15.114:0/1198513842 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x56483509c620 0x5648350a7130 secure :-1 s=READY pgs=533 cs=0 l=1 rev1=1 crypto rx=0x7ff7c8006010 tx=0x7ff7c80021d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b6bae959f7734aa4 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T10:27:48.287 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:27:48.282+0000 7ff7b5ffb640 10 client.0 ms_handle_connect on v2:172.21.15.114:3300/0 2024-04-23T10:27:48.287 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:27:48.282+0000 7ff7b5ffb640 1 -- 172.21.15.114:0/1198513842 <== mon.0 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff7c80142d0 con 0x56483509c620 2024-04-23T10:27:48.288 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:27:48.282+0000 7ff7b5ffb640 1 -- 172.21.15.114:0/1198513842 <== mon.0 v2:172.21.15.114:3300/0 2 ==== osd_map(92..92 src has 1..92) ==== 8585+0+0 (secure 0 0 0) 0x7ff7c800f880 con 0x56483509c620 2024-04-23T10:27:48.288 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:27:48.282+0000 7ff7d7ba8740 1 -- 172.21.15.114:0/1198513842 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7ff7c800fd30 con 0x56483509c620 2024-04-23T10:27:48.288 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:27:48.282+0000 7ff7b5ffb640 1 -- 172.21.15.114:0/1198513842 <== mon.0 v2:172.21.15.114:3300/0 3 ==== mon_get_version_reply(handle=1 version=21) ==== 24+0+0 (secure 0 0 0) 0x7ff7c8014750 con 0x56483509c620 2024-04-23T10:27:48.288 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:27:48.282+0000 7ff7d7ba8740 10 client.15714 fetch_fsmap learned FSMap version 21 2024-04-23T10:27:48.288 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:27:48.282+0000 7ff7d7ba8740 1 -- 172.21.15.114:0/1198513842 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({config=0+,fsmap=21,monmap=4+}) -- 0x5648350b6380 con 0x56483509c620 2024-04-23T10:27:48.288 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:27:48.282+0000 7ff7b5ffb640 1 -- 172.21.15.114:0/1198513842 <== mon.0 v2:172.21.15.114:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7ff7c800fd30 con 0x56483509c620 2024-04-23T10:27:48.288 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:27:48.283+0000 7ff7b5ffb640 1 -- 172.21.15.114:0/1198513842 <== mon.0 v2:172.21.15.114:3300/0 5 ==== fsmap(e 21) ==== 8338+0+0 (secure 0 0 0) 0x7ff7c802a070 con 0x56483509c620 2024-04-23T10:27:48.288 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:27:48.283+0000 7ff7d7ba8740 10 client.15714 fetch_fsmap finished waiting for FSMap version 21 2024-04-23T10:27:48.288 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:27:48.283+0000 7ff7d7ba8740 10 client.15714 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-04-23T10:27:48.288 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:27:48.283+0000 7ff7d7ba8740 20 client.15714 populate_metadata read hostname 'smithi114' 2024-04-23T10:27:48.288 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:27:48.283+0000 7ff7d7ba8740 1 --2- 172.21.15.114:0/1198513842 >> [v2:172.21.15.119:6838/937297340,v1:172.21.15.119:6839/937297340] conn(0x5648350b76d0 0x5648350b9ac0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T10:27:48.288 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:27:48.283+0000 7ff7d7ba8740 4 client.15714 mds_command: new command op to 24545 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-23T10:27:48.288 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:27:48.283+0000 7ff7d7ba8740 1 -- 172.21.15.114:0/1198513842 --> [v2:172.21.15.119:6838/937297340,v1:172.21.15.119:6839/937297340] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x56483500f390 con 0x5648350b76d0 2024-04-23T10:27:48.288 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:27:48.283+0000 7ff7cdd9b640 1 --2- 172.21.15.114:0/1198513842 >> [v2:172.21.15.119:6838/937297340,v1:172.21.15.119:6839/937297340] conn(0x5648350b76d0 0x5648350b9ac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T10:27:48.289 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:27:48.283+0000 7ff7b5ffb640 10 client.15714 ms_handle_connect on v2:172.21.15.119:6838/937297340 2024-04-23T10:27:48.289 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:27:48.283+0000 7ff7cdd9b640 1 --2- 172.21.15.114:0/1198513842 >> [v2:172.21.15.119:6838/937297340,v1:172.21.15.119:6839/937297340] conn(0x5648350b76d0 0x5648350b9ac0 crc :-1 s=READY pgs=54 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=4ced8fbc4d3a577f server_cookie=c8711c38d2266fec in_seq=0 out_seq=0 2024-04-23T10:27:48.289 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:27:48.284+0000 7ff7b5ffb640 1 -- 172.21.15.114:0/1198513842 <== mds.0 v2:172.21.15.119:6838/937297340 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x56483500f390 con 0x5648350b76d0 2024-04-23T10:27:48.289 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:27:48.284+0000 7ff7b5ffb640 10 client.15714 handle_command_reply: tid=0 2024-04-23T10:27:48.289 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:27:48.284+0000 7ff7d7ba8740 1 client.15714 shutdown 2024-04-23T10:27:48.289 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:27:48.284+0000 7ff7d7ba8740 1 -- 172.21.15.114:0/1198513842 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x56483509c620 msgr2=0x5648350a7130 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T10:27:48.289 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:27:48.284+0000 7ff7d7ba8740 1 --2- 172.21.15.114:0/1198513842 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x56483509c620 0x5648350a7130 secure :-1 s=READY pgs=533 cs=0 l=1 rev1=1 crypto rx=0x7ff7c8006010 tx=0x7ff7c80021d0 comp rx=0 tx=0).stop 2024-04-23T10:27:48.289 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:27:48.284+0000 7ff7d7ba8740 1 -- 172.21.15.114:0/1198513842 shutdown_connections 2024-04-23T10:27:48.289 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:27:48.284+0000 7ff7d7ba8740 1 --2- 172.21.15.114:0/1198513842 >> [v2:172.21.15.119:6838/937297340,v1:172.21.15.119:6839/937297340] conn(0x5648350b76d0 0x5648350b9ac0 crc :-1 s=READY pgs=54 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:27:48.289 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:27:48.284+0000 7ff7d7ba8740 1 --2- 172.21.15.114:0/1198513842 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x5648350a7800 0x5648350a4960 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:27:48.289 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:27:48.284+0000 7ff7d7ba8740 1 --2- 172.21.15.114:0/1198513842 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x56483509cf40 0x5648350a4420 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:27:48.289 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:27:48.284+0000 7ff7d7ba8740 1 --2- 172.21.15.114:0/1198513842 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x56483509c620 0x5648350a7130 unknown :-1 s=CLOSED pgs=533 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:27:48.289 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:27:48.284+0000 7ff7b5ffb640 0 client.15714 ms_handle_reset on v2:172.21.15.119:6838/937297340 2024-04-23T10:27:48.289 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:27:48.284+0000 7ff7d7ba8740 1 -- 172.21.15.114:0/1198513842 >> 172.21.15.114:0/1198513842 conn(0x564835098b30 msgr2=0x56483509e860 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T10:27:48.289 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:27:48.284+0000 7ff7d7ba8740 1 -- 172.21.15.114:0/1198513842 shutdown_connections 2024-04-23T10:27:48.289 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:27:48.284+0000 7ff7d7ba8740 1 -- 172.21.15.114:0/1198513842 wait complete. 2024-04-23T10:27:48.290 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:27:48.284+0000 7ff7d7ba8740 20 client.15714 trim_cache size 0 max 16384 2024-04-23T10:27:48.303 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:27:48.299+0000 7ff7d7ba8740 1 Processor -- start 2024-04-23T10:27:48.303 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:27:48.300+0000 7ff7d7ba8740 1 -- start start 2024-04-23T10:27:48.303 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:27:48.300+0000 7ff7d7ba8740 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x56483509c620 0x5648350a6c60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T10:27:48.303 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:27:48.300+0000 7ff7d7ba8740 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x56483509cf40 0x5648350bf1e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T10:27:48.303 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:27:48.300+0000 7ff7d7ba8740 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x5648350a7800 0x5648350bf720 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T10:27:48.303 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:27:48.300+0000 7ff7d7ba8740 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x7ff7d01ae3b0 con 0x56483509cf40 2024-04-23T10:27:48.303 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:27:48.300+0000 7ff7d7ba8740 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 -- 0x7ff7d01abc80 con 0x5648350a7800 2024-04-23T10:27:48.303 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:27:48.300+0000 7ff7d7ba8740 1 -- --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_getmap magic: 0 -- 0x7ff7d01a9ad0 con 0x56483509c620 2024-04-23T10:27:48.303 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:27:48.300+0000 7ff7cdd9b640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x56483509cf40 0x5648350bf1e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T10:27:48.303 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:27:48.300+0000 7ff7cdd9b640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x56483509cf40 0x5648350bf1e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.114:37096/0 (socket says 172.21.15.114:37096) 2024-04-23T10:27:48.303 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:27:48.300+0000 7ff7cdd9b640 1 -- 172.21.15.114:0/2637982881 learned_addr learned my addr 172.21.15.114:0/2637982881 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-04-23T10:27:48.303 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:27:48.300+0000 7ff7ced9d640 1 --2- 172.21.15.114:0/2637982881 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x56483509c620 0x5648350a6c60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T10:27:48.303 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:27:48.300+0000 7ff7ced9d640 1 --2- 172.21.15.114:0/2637982881 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x5648350a7800 0x5648350bf720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T10:27:48.303 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:27:48.300+0000 7ff7cdd9b640 1 -- 172.21.15.114:0/2637982881 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x56483509c620 msgr2=0x5648350a6c60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T10:27:48.304 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:27:48.300+0000 7ff7cdd9b640 1 --2- 172.21.15.114:0/2637982881 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x56483509c620 0x5648350a6c60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:27:48.304 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:27:48.300+0000 7ff7cdd9b640 1 -- 172.21.15.114:0/2637982881 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x5648350a7800 msgr2=0x5648350bf720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T10:27:48.304 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:27:48.300+0000 7ff7cdd9b640 1 --2- 172.21.15.114:0/2637982881 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x5648350a7800 0x5648350bf720 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:27:48.304 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:27:48.300+0000 7ff7cdd9b640 1 -- 172.21.15.114:0/2637982881 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x56483509d9e0 con 0x56483509cf40 2024-04-23T10:27:48.304 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:27:48.300+0000 7ff7ced9d640 1 --2- 172.21.15.114:0/2637982881 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x5648350a7800 0x5648350bf720 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-23T10:27:48.304 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:27:48.300+0000 7ff7cdd9b640 1 --2- 172.21.15.114:0/2637982881 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x56483509cf40 0x5648350bf1e0 secure :-1 s=READY pgs=534 cs=0 l=1 rev1=1 crypto rx=0x7ff7c0005fd0 tx=0x7ff7c00118f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=eb4f9bde41c96bad server_cookie=0 in_seq=0 out_seq=0 2024-04-23T10:27:48.304 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:27:48.300+0000 7ff7b7fff640 1 -- 172.21.15.114:0/2637982881 <== mon.0 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff7c0005220 con 0x56483509cf40 2024-04-23T10:27:48.304 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:27:48.300+0000 7ff7b7fff640 1 -- 172.21.15.114:0/2637982881 <== mon.0 v2:172.21.15.114:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7ff7c0004c50 con 0x56483509cf40 2024-04-23T10:27:48.304 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:27:48.300+0000 7ff7b7fff640 1 -- 172.21.15.114:0/2637982881 <== mon.0 v2:172.21.15.114:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff7c00206d0 con 0x56483509cf40 2024-04-23T10:27:48.304 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:27:48.301+0000 7ff7d7ba8740 1 -- 172.21.15.114:0/2637982881 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x56483509cf40 msgr2=0x5648350bf1e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T10:27:48.304 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:27:48.301+0000 7ff7d7ba8740 1 --2- 172.21.15.114:0/2637982881 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x56483509cf40 0x5648350bf1e0 secure :-1 s=READY pgs=534 cs=0 l=1 rev1=1 crypto rx=0x7ff7c0005fd0 tx=0x7ff7c00118f0 comp rx=0 tx=0).stop 2024-04-23T10:27:48.304 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:27:48.301+0000 7ff7d7ba8740 1 -- 172.21.15.114:0/2637982881 shutdown_connections 2024-04-23T10:27:48.304 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:27:48.301+0000 7ff7d7ba8740 1 --2- 172.21.15.114:0/2637982881 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x5648350a7800 0x5648350bf720 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:27:48.304 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:27:48.301+0000 7ff7d7ba8740 1 --2- 172.21.15.114:0/2637982881 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x56483509cf40 0x5648350bf1e0 unknown :-1 s=CLOSED pgs=534 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:27:48.304 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:27:48.301+0000 7ff7d7ba8740 1 --2- 172.21.15.114:0/2637982881 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x56483509c620 0x5648350a6c60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:27:48.305 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:27:48.301+0000 7ff7d7ba8740 1 -- 172.21.15.114:0/2637982881 >> 172.21.15.114:0/2637982881 conn(0x564835098b30 msgr2=0x564835097cb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T10:27:48.305 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:27:48.301+0000 7ff7d7ba8740 1 -- 172.21.15.114:0/2637982881 shutdown_connections 2024-04-23T10:27:48.305 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:27:48.301+0000 7ff7d7ba8740 1 -- 172.21.15.114:0/2637982881 wait complete. 2024-04-23T10:27:48.305 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:27:48.301+0000 7ff7d7ba8740 1 -- start start 2024-04-23T10:27:48.305 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:27:48.301+0000 7ff7d7ba8740 1 Processor -- start 2024-04-23T10:27:48.305 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:27:48.302+0000 7ff7d7ba8740 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x56483509c620 0x5648350afce0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T10:27:48.305 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:27:48.302+0000 7ff7d7ba8740 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x56483509cf40 0x5648350ad070 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T10:27:48.305 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:27:48.302+0000 7ff7d7ba8740 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x5648350a7800 0x5648350ad680 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T10:27:48.305 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:27:48.302+0000 7ff7d7ba8740 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x7ff7d01aa140 con 0x56483509cf40 2024-04-23T10:27:48.305 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:27:48.302+0000 7ff7d7ba8740 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 -- 0x7ff7d01ab5b0 con 0x56483509c620 2024-04-23T10:27:48.305 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:27:48.302+0000 7ff7d7ba8740 1 -- --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_getmap magic: 0 -- 0x7ff7d01aa810 con 0x5648350a7800 2024-04-23T10:27:48.305 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:27:48.302+0000 7ff7d7ba8740 10 client.0 _ll_register_callbacks cb 0x7ff7c8014990 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-23T10:27:48.305 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:27:48.302+0000 7ff7cdd9b640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x56483509cf40 0x5648350ad070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T10:27:48.305 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:27:48.302+0000 7ff7cdd9b640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x56483509cf40 0x5648350ad070 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.114:37100/0 (socket says 172.21.15.114:37100) 2024-04-23T10:27:48.305 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:27:48.302+0000 7ff7cdd9b640 1 -- 172.21.15.114:0/2597070238 learned_addr learned my addr 172.21.15.114:0/2597070238 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-04-23T10:27:48.306 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:27:48.302+0000 7ff7ced9d640 1 --2- 172.21.15.114:0/2597070238 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x56483509c620 0x5648350afce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T10:27:48.306 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:27:48.302+0000 7ff7cdd9b640 1 -- 172.21.15.114:0/2597070238 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x5648350a7800 msgr2=0x5648350ad680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T10:27:48.306 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:27:48.302+0000 7ff7cdd9b640 1 --2- 172.21.15.114:0/2597070238 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x5648350a7800 0x5648350ad680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:27:48.306 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:27:48.302+0000 7ff7cdd9b640 1 -- 172.21.15.114:0/2597070238 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x56483509c620 msgr2=0x5648350afce0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T10:27:48.306 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:27:48.302+0000 7ff7cdd9b640 1 --2- 172.21.15.114:0/2597070238 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x56483509c620 0x5648350afce0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:27:48.306 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:27:48.302+0000 7ff7cdd9b640 1 -- 172.21.15.114:0/2597070238 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5648350adeb0 con 0x56483509cf40 2024-04-23T10:27:48.306 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:27:48.302+0000 7ff7cdd9b640 1 --2- 172.21.15.114:0/2597070238 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x56483509cf40 0x5648350ad070 secure :-1 s=READY pgs=535 cs=0 l=1 rev1=1 crypto rx=0x7ff7c0004840 tx=0x7ff7c0004870 comp rx=0 tx=0).ready entity=mon.0 client_cookie=84b7b6be5fa06099 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T10:27:48.306 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:27:48.302+0000 7ff7b5ffb640 10 client.0 ms_handle_connect on v2:172.21.15.114:3300/0 2024-04-23T10:27:48.306 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:27:48.302+0000 7ff7b5ffb640 1 -- 172.21.15.114:0/2597070238 <== mon.0 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff7c0011240 con 0x56483509cf40 2024-04-23T10:27:48.306 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:27:48.302+0000 7ff7b5ffb640 1 -- 172.21.15.114:0/2597070238 <== mon.0 v2:172.21.15.114:3300/0 2 ==== osd_map(92..92 src has 1..92) ==== 8585+0+0 (secure 0 0 0) 0x7ff7c00113e0 con 0x56483509cf40 2024-04-23T10:27:48.306 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:27:48.302+0000 7ff7d7ba8740 1 -- 172.21.15.114:0/2597070238 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7ff7c0004c50 con 0x56483509cf40 2024-04-23T10:27:48.306 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:27:48.302+0000 7ff7b5ffb640 1 -- 172.21.15.114:0/2597070238 <== mon.0 v2:172.21.15.114:3300/0 3 ==== mon_get_version_reply(handle=1 version=21) ==== 24+0+0 (secure 0 0 0) 0x7ff7c00115e0 con 0x56483509cf40 2024-04-23T10:27:48.306 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:27:48.302+0000 7ff7d7ba8740 10 client.15720 fetch_fsmap learned FSMap version 21 2024-04-23T10:27:48.306 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:27:48.302+0000 7ff7d7ba8740 1 -- 172.21.15.114:0/2597070238 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({config=0+,fsmap=21,monmap=4+}) -- 0x56483511dba0 con 0x56483509cf40 2024-04-23T10:27:48.306 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:27:48.303+0000 7ff7b5ffb640 1 -- 172.21.15.114:0/2597070238 <== mon.0 v2:172.21.15.114:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7ff7c0004c50 con 0x56483509cf40 2024-04-23T10:27:48.307 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:27:48.303+0000 7ff7b5ffb640 1 -- 172.21.15.114:0/2597070238 <== mon.0 v2:172.21.15.114:3300/0 5 ==== fsmap(e 21) ==== 8338+0+0 (secure 0 0 0) 0x7ff7c0010bf0 con 0x56483509cf40 2024-04-23T10:27:48.307 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:27:48.303+0000 7ff7d7ba8740 10 client.15720 fetch_fsmap finished waiting for FSMap version 21 2024-04-23T10:27:48.307 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:27:48.303+0000 7ff7d7ba8740 10 client.15720 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-04-23T10:27:48.307 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:27:48.303+0000 7ff7d7ba8740 20 client.15720 populate_metadata read hostname 'smithi114' 2024-04-23T10:27:48.307 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:27:48.303+0000 7ff7d7ba8740 1 --2- 172.21.15.114:0/2597070238 >> [v2:172.21.15.119:6838/937297340,v1:172.21.15.119:6839/937297340] conn(0x56483511e290 0x5648351206b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T10:27:48.307 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:27:48.303+0000 7ff7d7ba8740 4 client.15720 mds_command: new command op to 24545 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-04-23T10:27:48.307 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:27:48.303+0000 7ff7d7ba8740 1 -- 172.21.15.114:0/2597070238 --> [v2:172.21.15.119:6838/937297340,v1:172.21.15.119:6839/937297340] -- command(tid 0: {"prefix": "flush journal"}) -- 0x564835009200 con 0x56483511e290 2024-04-23T10:27:48.307 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:27:48.303+0000 7ff7ced9d640 1 --2- 172.21.15.114:0/2597070238 >> [v2:172.21.15.119:6838/937297340,v1:172.21.15.119:6839/937297340] conn(0x56483511e290 0x5648351206b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T10:27:48.307 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:27:48.303+0000 7ff7b5ffb640 10 client.15720 ms_handle_connect on v2:172.21.15.119:6838/937297340 2024-04-23T10:27:48.307 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:27:48.303+0000 7ff7ced9d640 1 --2- 172.21.15.114:0/2597070238 >> [v2:172.21.15.119:6838/937297340,v1:172.21.15.119:6839/937297340] conn(0x56483511e290 0x5648351206b0 crc :-1 s=READY pgs=55 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=d4f7098cf94f54b2 server_cookie=be4537ad6b82b103 in_seq=0 out_seq=0 2024-04-23T10:27:48.309 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:27:48.310+0000 7ff7b5ffb640 1 -- 172.21.15.114:0/2597070238 <== mds.0 v2:172.21.15.119:6838/937297340 1 ==== command_reply(tid 0: 0 ) ==== 8+0+44 (crc 0 0 0) 0x564835009200 con 0x56483511e290 2024-04-23T10:27:48.309 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:27:48.310+0000 7ff7b5ffb640 10 client.15720 handle_command_reply: tid=0 2024-04-23T10:27:48.310 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:27:48.310+0000 7ff7d7ba8740 1 client.15720 shutdown 2024-04-23T10:27:48.310 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:27:48.310+0000 7ff7d7ba8740 1 -- 172.21.15.114:0/2597070238 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x56483509cf40 msgr2=0x5648350ad070 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T10:27:48.310 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:27:48.310+0000 7ff7d7ba8740 1 --2- 172.21.15.114:0/2597070238 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x56483509cf40 0x5648350ad070 secure :-1 s=READY pgs=535 cs=0 l=1 rev1=1 crypto rx=0x7ff7c0004840 tx=0x7ff7c0004870 comp rx=0 tx=0).stop 2024-04-23T10:27:48.310 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:27:48.310+0000 7ff7d7ba8740 1 -- 172.21.15.114:0/2597070238 shutdown_connections 2024-04-23T10:27:48.310 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:27:48.310+0000 7ff7d7ba8740 1 --2- 172.21.15.114:0/2597070238 >> [v2:172.21.15.119:6838/937297340,v1:172.21.15.119:6839/937297340] conn(0x56483511e290 0x5648351206b0 crc :-1 s=READY pgs=55 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:27:48.310 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:27:48.310+0000 7ff7d7ba8740 1 --2- 172.21.15.114:0/2597070238 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x5648350a7800 0x5648350ad680 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:27:48.310 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:27:48.310+0000 7ff7d7ba8740 1 --2- 172.21.15.114:0/2597070238 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x56483509cf40 0x5648350ad070 unknown :-1 s=CLOSED pgs=535 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:27:48.310 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:27:48.310+0000 7ff7b5ffb640 0 client.15720 ms_handle_reset on v2:172.21.15.119:6838/937297340 2024-04-23T10:27:48.310 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:27:48.310+0000 7ff7d7ba8740 1 --2- 172.21.15.114:0/2597070238 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x56483509c620 0x5648350afce0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:27:48.310 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:27:48.310+0000 7ff7d7ba8740 1 -- 172.21.15.114:0/2597070238 >> 172.21.15.114:0/2597070238 conn(0x564835098b30 msgr2=0x5648350a3c80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T10:27:48.310 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:27:48.310+0000 7ff7d7ba8740 1 -- 172.21.15.114:0/2597070238 shutdown_connections 2024-04-23T10:27:48.310 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:27:48.310+0000 7ff7d7ba8740 1 -- 172.21.15.114:0/2597070238 wait complete. 2024-04-23T10:27:48.310 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:27:48.310+0000 7ff7d7ba8740 20 client.15720 trim_cache size 0 max 16384 2024-04-23T10:27:48.311 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stdout:{ 2024-04-23T10:27:48.311 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stdout: "message": "", 2024-04-23T10:27:48.311 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stdout: "return_code": 0 2024-04-23T10:27:48.311 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stdout:} 2024-04-23T10:27:48.312 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:27:48.311+0000 7ff7d4dba640 1 -- 172.21.15.114:0/3937271828 >> [v2:172.21.15.114:6800/1395693466,v1:172.21.15.114:6801/1395693466] conn(0x7ff7b80a3ee0 msgr2=0x7ff7b80a63a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T10:27:48.312 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:27:48.311+0000 7ff7d4dba640 1 --2- 172.21.15.114:0/3937271828 >> [v2:172.21.15.114:6800/1395693466,v1:172.21.15.114:6801/1395693466] conn(0x7ff7b80a3ee0 0x7ff7b80a63a0 secure :-1 s=READY pgs=72 cs=0 l=1 rev1=1 crypto rx=0x7ff7c40ee910 tx=0x7ff7c4032f60 comp rx=0 tx=0).stop 2024-04-23T10:27:48.312 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:27:48.311+0000 7ff7d4dba640 1 -- 172.21.15.114:0/3937271828 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7ff7d0103200 msgr2=0x7ff7d0198720 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T10:27:48.312 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:27:48.311+0000 7ff7d4dba640 1 --2- 172.21.15.114:0/3937271828 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7ff7d0103200 0x7ff7d0198720 secure :-1 s=READY pgs=531 cs=0 l=1 rev1=1 crypto rx=0x7ff7c4005940 tx=0x7ff7c40042f0 comp rx=0 tx=0).stop 2024-04-23T10:27:48.312 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:27:48.311+0000 7ff7d4dba640 1 -- 172.21.15.114:0/3937271828 shutdown_connections 2024-04-23T10:27:48.312 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:27:48.311+0000 7ff7d4dba640 1 --2- 172.21.15.114:0/3937271828 >> [v2:172.21.15.114:6800/1395693466,v1:172.21.15.114:6801/1395693466] conn(0x7ff7b80a3ee0 0x7ff7b80a63a0 unknown :-1 s=CLOSED pgs=72 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:27:48.312 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:27:48.311+0000 7ff7d4dba640 1 --2- 172.21.15.114:0/3937271828 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7ff7d010c540 0x7ff7d019cf90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:27:48.312 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:27:48.311+0000 7ff7d4dba640 1 --2- 172.21.15.114:0/3937271828 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7ff7d0103b20 0x7ff7d0198c60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:27:48.312 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:27:48.311+0000 7ff7d4dba640 1 --2- 172.21.15.114:0/3937271828 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7ff7d0103200 0x7ff7d0198720 unknown :-1 s=CLOSED pgs=531 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:27:48.312 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:27:48.311+0000 7ff7d4dba640 1 -- 172.21.15.114:0/3937271828 >> 172.21.15.114:0/3937271828 conn(0x7ff7d0077fd0 msgr2=0x7ff7d00fdfc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T10:27:48.312 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:27:48.311+0000 7ff7d4dba640 1 -- 172.21.15.114:0/3937271828 shutdown_connections 2024-04-23T10:27:48.312 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:27:48.311+0000 7ff7d4dba640 1 -- 172.21.15.114:0/3937271828 wait complete. 2024-04-23T10:27:48.492 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 23 10:27:48 smithi119 ceph-mon[32608]: pgmap v38: 97 pgs: 97 active+clean; 1.3 MiB data, 377 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 127 B/s wr, 4 op/s; 13 B/s, 0 keys/s, 1 objects/s recovering 2024-04-23T10:27:48.504 INFO:journalctl@ceph.mon.a.smithi114.stdout:Apr 23 10:27:48 smithi114 ceph-mon[29255]: pgmap v38: 97 pgs: 97 active+clean; 1.3 MiB data, 377 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 127 B/s wr, 4 op/s; 13 B/s, 0 keys/s, 1 objects/s recovering 2024-04-23T10:27:48.578 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 23 10:27:48 smithi204 ceph-mon[32649]: pgmap v38: 97 pgs: 97 active+clean; 1.3 MiB data, 377 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 127 B/s wr, 4 op/s; 13 B/s, 0 keys/s, 1 objects/s recovering 2024-04-23T10:27:50.492 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 23 10:27:50 smithi119 ceph-mon[32608]: pgmap v39: 97 pgs: 97 active+clean; 1.4 MiB data, 377 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 1023 B/s wr, 7 op/s; 13 B/s, 0 keys/s, 1 objects/s recovering 2024-04-23T10:27:50.504 INFO:journalctl@ceph.mon.a.smithi114.stdout:Apr 23 10:27:50 smithi114 ceph-mon[29255]: pgmap v39: 97 pgs: 97 active+clean; 1.4 MiB data, 377 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 1023 B/s wr, 7 op/s; 13 B/s, 0 keys/s, 1 objects/s recovering 2024-04-23T10:27:50.578 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 23 10:27:50 smithi204 ceph-mon[32649]: pgmap v39: 97 pgs: 97 active+clean; 1.4 MiB data, 377 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 1023 B/s wr, 7 op/s; 13 B/s, 0 keys/s, 1 objects/s recovering 2024-04-23T10:27:51.306 DEBUG:teuthology.orchestra.run.smithi114:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-23T10:27:51.418 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:27:51.417+0000 7f5d1f479640 1 -- 172.21.15.114:0/3048709467 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f5d18106170 msgr2=0x7f5d1810dd40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T10:27:51.418 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:27:51.417+0000 7f5d1f479640 1 --2- 172.21.15.114:0/3048709467 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f5d18106170 0x7f5d1810dd40 secure :-1 s=READY pgs=536 cs=0 l=1 rev1=1 crypto rx=0x7f5d10009ca0 tx=0x7f5d1002a8c0 comp rx=0 tx=0).stop 2024-04-23T10:27:51.418 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:27:51.417+0000 7f5d1f479640 1 -- 172.21.15.114:0/3048709467 shutdown_connections 2024-04-23T10:27:51.419 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:27:51.417+0000 7f5d1f479640 1 --2- 172.21.15.114:0/3048709467 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f5d18106170 0x7f5d1810dd40 unknown :-1 s=CLOSED pgs=536 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:27:51.419 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:27:51.417+0000 7f5d1f479640 1 --2- 172.21.15.114:0/3048709467 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f5d18101a50 0x7f5d18105a90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:27:51.419 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:27:51.417+0000 7f5d1f479640 1 --2- 172.21.15.114:0/3048709467 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f5d181010a0 0x7f5d18101480 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:27:51.419 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:27:51.417+0000 7f5d1f479640 1 -- 172.21.15.114:0/3048709467 >> 172.21.15.114:0/3048709467 conn(0x7f5d18077f50 msgr2=0x7f5d18078360 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T10:27:51.419 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:27:51.417+0000 7f5d1f479640 1 -- 172.21.15.114:0/3048709467 shutdown_connections 2024-04-23T10:27:51.419 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:27:51.417+0000 7f5d1f479640 1 -- 172.21.15.114:0/3048709467 wait complete. 2024-04-23T10:27:51.419 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:27:51.417+0000 7f5d1f479640 1 Processor -- start 2024-04-23T10:27:51.419 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:27:51.417+0000 7f5d1f479640 1 -- start start 2024-04-23T10:27:51.419 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:27:51.418+0000 7f5d1f479640 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f5d181010a0 0x7f5d18199540 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T10:27:51.419 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:27:51.418+0000 7f5d1f479640 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f5d18106170 0x7f5d18199a80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T10:27:51.419 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:27:51.418+0000 7f5d1f479640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f5d1819a010 0x7f5d181a45c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T10:27:51.419 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:27:51.418+0000 7f5d1f479640 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x7f5d181110e0 con 0x7f5d1819a010 2024-04-23T10:27:51.419 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:27:51.418+0000 7f5d1f479640 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 -- 0x7f5d18110f60 con 0x7f5d181010a0 2024-04-23T10:27:51.419 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:27:51.418+0000 7f5d1f479640 1 -- --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_getmap magic: 0 -- 0x7f5d18111260 con 0x7f5d18106170 2024-04-23T10:27:51.419 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:27:51.418+0000 7f5d1da16640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f5d1819a010 0x7f5d181a45c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T10:27:51.420 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:27:51.418+0000 7f5d1da16640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f5d1819a010 0x7f5d181a45c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.114:37124/0 (socket says 172.21.15.114:37124) 2024-04-23T10:27:51.420 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:27:51.418+0000 7f5d1da16640 1 -- 172.21.15.114:0/1245947554 learned_addr learned my addr 172.21.15.114:0/1245947554 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-04-23T10:27:51.420 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:27:51.418+0000 7f5d1d215640 1 --2- 172.21.15.114:0/1245947554 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f5d181010a0 0x7f5d18199540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T10:27:51.420 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:27:51.418+0000 7f5d1ca14640 1 --2- 172.21.15.114:0/1245947554 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f5d18106170 0x7f5d18199a80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T10:27:51.420 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:27:51.418+0000 7f5d1da16640 1 -- 172.21.15.114:0/1245947554 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f5d18106170 msgr2=0x7f5d18199a80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T10:27:51.420 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:27:51.418+0000 7f5d1da16640 1 --2- 172.21.15.114:0/1245947554 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f5d18106170 0x7f5d18199a80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:27:51.420 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:27:51.418+0000 7f5d1da16640 1 -- 172.21.15.114:0/1245947554 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f5d181010a0 msgr2=0x7f5d18199540 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T10:27:51.420 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:27:51.418+0000 7f5d1da16640 1 --2- 172.21.15.114:0/1245947554 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f5d181010a0 0x7f5d18199540 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:27:51.420 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:27:51.418+0000 7f5d1da16640 1 -- 172.21.15.114:0/1245947554 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f5d181a4dc0 con 0x7f5d1819a010 2024-04-23T10:27:51.420 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:27:51.418+0000 7f5d1da16640 1 --2- 172.21.15.114:0/1245947554 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f5d1819a010 0x7f5d181a45c0 secure :-1 s=READY pgs=537 cs=0 l=1 rev1=1 crypto rx=0x7f5d10005cc0 tx=0x7f5d1002f380 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T10:27:51.420 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:27:51.418+0000 7f5d0a7fc640 1 -- 172.21.15.114:0/1245947554 <== mon.0 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5d10003bf0 con 0x7f5d1819a010 2024-04-23T10:27:51.420 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:27:51.418+0000 7f5d1f479640 1 -- 172.21.15.114:0/1245947554 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f5d181a5050 con 0x7f5d1819a010 2024-04-23T10:27:51.420 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:27:51.418+0000 7f5d0a7fc640 1 -- 172.21.15.114:0/1245947554 <== mon.0 v2:172.21.15.114:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f5d10003d90 con 0x7f5d1819a010 2024-04-23T10:27:51.420 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:27:51.418+0000 7f5d0a7fc640 1 -- 172.21.15.114:0/1245947554 <== mon.0 v2:172.21.15.114:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5d1002fbc0 con 0x7f5d1819a010 2024-04-23T10:27:51.420 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:27:51.418+0000 7f5d1f479640 1 -- 172.21.15.114:0/1245947554 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f5d181a55f0 con 0x7f5d1819a010 2024-04-23T10:27:51.421 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:27:51.419+0000 7f5d1f479640 1 -- 172.21.15.114:0/1245947554 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f5d18068d70 con 0x7f5d1819a010 2024-04-23T10:27:51.421 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:27:51.421+0000 7f5d0a7fc640 1 -- 172.21.15.114:0/1245947554 <== mon.0 v2:172.21.15.114:3300/0 4 ==== mgrmap(e 25) ==== 138880+0+0 (secure 0 0 0) 0x7f5d1002fd60 con 0x7f5d1819a010 2024-04-23T10:27:51.421 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:27:51.422+0000 7f5d0a7fc640 1 --2- 172.21.15.114:0/1245947554 >> [v2:172.21.15.114:6800/1395693466,v1:172.21.15.114:6801/1395693466] conn(0x7f5d040a3e40 0x7f5d040a6300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T10:27:51.422 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:27:51.422+0000 7f5d1d215640 1 --2- 172.21.15.114:0/1245947554 >> [v2:172.21.15.114:6800/1395693466,v1:172.21.15.114:6801/1395693466] conn(0x7f5d040a3e40 0x7f5d040a6300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T10:27:51.422 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:27:51.422+0000 7f5d0a7fc640 1 -- 172.21.15.114:0/1245947554 <== mon.0 v2:172.21.15.114:3300/0 5 ==== osd_map(92..92 src has 1..92) ==== 8585+0+0 (secure 0 0 0) 0x7f5d100f2100 con 0x7f5d1819a010 2024-04-23T10:27:51.422 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:27:51.422+0000 7f5d1d215640 1 --2- 172.21.15.114:0/1245947554 >> [v2:172.21.15.114:6800/1395693466,v1:172.21.15.114:6801/1395693466] conn(0x7f5d040a3e40 0x7f5d040a6300 secure :-1 s=READY pgs=73 cs=0 l=1 rev1=1 crypto rx=0x7f5d14005fd0 tx=0x7f5d1400b390 comp rx=0 tx=0).ready entity=mgr.15291 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T10:27:51.424 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:27:51.425+0000 7f5d0a7fc640 1 -- 172.21.15.114:0/1245947554 <== mon.0 v2:172.21.15.114:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f5d100be6c0 con 0x7f5d1819a010 2024-04-23T10:27:51.638 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:27:51.639+0000 7f5d1f479640 1 -- 172.21.15.114:0/1245947554 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f5d18109400 con 0x7f5d1819a010 2024-04-23T10:27:51.639 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:27:51.640+0000 7f5d0a7fc640 1 -- 172.21.15.114:0/1245947554 <== mon.0 v2:172.21.15.114:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 21 v21) ==== 94+0+11884 (secure 0 0 0) 0x7f5d100098b0 con 0x7f5d1819a010 2024-04-23T10:27:51.639 INFO:teuthology.orchestra.run.smithi114.stdout: 2024-04-23T10:27:51.639 INFO:teuthology.orchestra.run.smithi114.stdout:{"epoch":21,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14547,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.114:6835/663493580","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6834","nonce":663493580},{"type":"v1","addr":"172.21.15.114:6835","nonce":663493580}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14565,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.114:6837/1439252154","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6836","nonce":1439252154},{"type":"v1","addr":"172.21.15.114:6837","nonce":1439252154}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5},{"gid":14634,"name":"g","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.114:6839/3849796939","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6838","nonce":3849796939},{"type":"v1","addr":"172.21.15.114:6839","nonce":3849796939}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":11},{"gid":14646,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.114:6841/3544523891","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6840","nonce":3544523891},{"type":"v1","addr":"172.21.15.114:6841","nonce":3544523891}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":12},{"gid":24385,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.204:6833/3866561754","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6832","nonce":3866561754},{"type":"v1","addr":"172.21.15.204:6833","nonce":3866561754}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2},{"gid":24427,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.204:6835/2533793172","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6834","nonce":2533793172},{"type":"v1","addr":"172.21.15.204:6835","nonce":2533793172}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":7}],"filesystems":[{"mdsmap":{"epoch":21,"flags":306,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":true},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-04-23T10:24:43.383158+0000","modified":"2024-04-23T10:26:38.945346+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":3,"in":[0,1,2],"up":{"mds_0":24545,"mds_1":24509,"mds_2":24479},"failed":[],"damaged":[],"stopped":[],"info":{"gid_24439":{"gid":24439,"name":"c","rank":2,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.204:6837/513822179","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6836","nonce":513822179},{"type":"v1","addr":"172.21.15.204:6837","nonce":513822179}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24443":{"gid":24443,"name":"k","rank":1,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.119:6833/783244660","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6832","nonce":783244660},{"type":"v1","addr":"172.21.15.119:6833","nonce":783244660}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24463":{"gid":24463,"name":"i","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.204:6839/3222658893","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6838","nonce":3222658893},{"type":"v1","addr":"172.21.15.204:6839","nonce":3222658893}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24479":{"gid":24479,"name":"e","rank":2,"incarnation":19,"state":"up:active","state_seq":9,"addr":"172.21.15.119:6835/2033916709","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6834","nonce":2033916709},{"type":"v1","addr":"172.21.15.119:6835","nonce":2033916709}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24509":{"gid":24509,"name":"h","rank":1,"incarnation":17,"state":"up:active","state_seq":6,"addr":"172.21.15.119:6837/1872363128","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6836","nonce":1872363128},{"type":"v1","addr":"172.21.15.119:6837","nonce":1872363128}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24545":{"gid":24545,"name":"b","rank":0,"incarnation":14,"state":"up:active","state_seq":3,"addr":"172.21.15.119:6839/937297340","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6838","nonce":937297340},{"type":"v1","addr":"172.21.15.119:6839","nonce":937297340}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":24545,"qdb_cluster":[24479,24509,24545]},"id":1}]} 2024-04-23T10:27:51.640 INFO:teuthology.orchestra.run.smithi114.stderr:dumped fsmap epoch 21 2024-04-23T10:27:51.641 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:27:51.642+0000 7f5d1f479640 1 -- 172.21.15.114:0/1245947554 >> [v2:172.21.15.114:6800/1395693466,v1:172.21.15.114:6801/1395693466] conn(0x7f5d040a3e40 msgr2=0x7f5d040a6300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T10:27:51.641 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:27:51.642+0000 7f5d1f479640 1 --2- 172.21.15.114:0/1245947554 >> [v2:172.21.15.114:6800/1395693466,v1:172.21.15.114:6801/1395693466] conn(0x7f5d040a3e40 0x7f5d040a6300 secure :-1 s=READY pgs=73 cs=0 l=1 rev1=1 crypto rx=0x7f5d14005fd0 tx=0x7f5d1400b390 comp rx=0 tx=0).stop 2024-04-23T10:27:51.641 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:27:51.642+0000 7f5d1f479640 1 -- 172.21.15.114:0/1245947554 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f5d1819a010 msgr2=0x7f5d181a45c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T10:27:51.642 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:27:51.642+0000 7f5d1f479640 1 --2- 172.21.15.114:0/1245947554 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f5d1819a010 0x7f5d181a45c0 secure :-1 s=READY pgs=537 cs=0 l=1 rev1=1 crypto rx=0x7f5d10005cc0 tx=0x7f5d1002f380 comp rx=0 tx=0).stop 2024-04-23T10:27:51.642 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:27:51.642+0000 7f5d1f479640 1 -- 172.21.15.114:0/1245947554 shutdown_connections 2024-04-23T10:27:51.642 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:27:51.642+0000 7f5d1f479640 1 --2- 172.21.15.114:0/1245947554 >> [v2:172.21.15.114:6800/1395693466,v1:172.21.15.114:6801/1395693466] conn(0x7f5d040a3e40 0x7f5d040a6300 unknown :-1 s=CLOSED pgs=73 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:27:51.642 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:27:51.642+0000 7f5d1f479640 1 --2- 172.21.15.114:0/1245947554 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f5d1819a010 0x7f5d181a45c0 unknown :-1 s=CLOSED pgs=537 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:27:51.642 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:27:51.642+0000 7f5d1f479640 1 --2- 172.21.15.114:0/1245947554 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f5d18106170 0x7f5d18199a80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:27:51.642 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:27:51.642+0000 7f5d1f479640 1 --2- 172.21.15.114:0/1245947554 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f5d181010a0 0x7f5d18199540 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:27:51.642 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:27:51.642+0000 7f5d1f479640 1 -- 172.21.15.114:0/1245947554 >> 172.21.15.114:0/1245947554 conn(0x7f5d18077f50 msgr2=0x7f5d180fef10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T10:27:51.642 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:27:51.642+0000 7f5d1f479640 1 -- 172.21.15.114:0/1245947554 shutdown_connections 2024-04-23T10:27:51.642 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:27:51.642+0000 7f5d1f479640 1 -- 172.21.15.114:0/1245947554 wait complete. 2024-04-23T10:27:51.652 DEBUG:tasks.quiescer.fs.[cephfs]:Running ceph command: 'tell mds.24545 quiesce db --set-id 750460ef --release --await' 2024-04-23T10:27:51.652 DEBUG:teuthology.orchestra.run.smithi114:> timeout 75 sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.24545 quiesce db --set-id 750460ef --release --await 2024-04-23T10:27:51.757 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:27:51.758+0000 7f3134b07640 1 -- 172.21.15.114:0/4085031565 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f3130107030 msgr2=0x7f3130107410 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T10:27:51.757 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:27:51.758+0000 7f3134b07640 1 --2- 172.21.15.114:0/4085031565 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f3130107030 0x7f3130107410 secure :-1 s=READY pgs=538 cs=0 l=1 rev1=1 crypto rx=0x7f3120009930 tx=0x7f312002a420 comp rx=0 tx=0).stop 2024-04-23T10:27:51.757 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:27:51.758+0000 7f3134b07640 1 -- 172.21.15.114:0/4085031565 shutdown_connections 2024-04-23T10:27:51.757 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:27:51.758+0000 7f3134b07640 1 --2- 172.21.15.114:0/4085031565 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f3130101b40 0x7f313010dc80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:27:51.758 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:27:51.758+0000 7f3134b07640 1 --2- 172.21.15.114:0/4085031565 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f31301011a0 0x7f3130101600 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:27:51.758 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:27:51.758+0000 7f3134b07640 1 --2- 172.21.15.114:0/4085031565 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f3130107030 0x7f3130107410 unknown :-1 s=CLOSED pgs=538 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:27:51.758 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:27:51.758+0000 7f3134b07640 1 -- 172.21.15.114:0/4085031565 >> 172.21.15.114:0/4085031565 conn(0x7f3130076c50 msgr2=0x7f3130077060 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T10:27:51.758 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:27:51.758+0000 7f3134b07640 1 -- 172.21.15.114:0/4085031565 shutdown_connections 2024-04-23T10:27:51.758 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:27:51.758+0000 7f3134b07640 1 -- 172.21.15.114:0/4085031565 wait complete. 2024-04-23T10:27:51.760 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:27:51.759+0000 7f3134b07640 1 Processor -- start 2024-04-23T10:27:51.760 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:27:51.759+0000 7f3134b07640 1 -- start start 2024-04-23T10:27:51.760 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:27:51.759+0000 7f3134b07640 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f31301011a0 0x7f313019e420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T10:27:51.760 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:27:51.759+0000 7f3134b07640 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f3130101b40 0x7f313019e960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T10:27:51.760 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:27:51.759+0000 7f3134b07640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f3130107030 0x7f31301985d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T10:27:51.760 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:27:51.759+0000 7f3134b07640 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x7f31301110e0 con 0x7f3130107030 2024-04-23T10:27:51.760 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:27:51.759+0000 7f3134b07640 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 -- 0x7f3130110f60 con 0x7f31301011a0 2024-04-23T10:27:51.760 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:27:51.759+0000 7f3134b07640 1 -- --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_getmap magic: 0 -- 0x7f3130111260 con 0x7f3130101b40 2024-04-23T10:27:51.760 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:27:51.759+0000 7f312ed9d640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f3130107030 0x7f31301985d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T10:27:51.761 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:27:51.760+0000 7f312ed9d640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f3130107030 0x7f31301985d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.114:37138/0 (socket says 172.21.15.114:37138) 2024-04-23T10:27:51.761 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:27:51.760+0000 7f312ed9d640 1 -- 172.21.15.114:0/3777468127 learned_addr learned my addr 172.21.15.114:0/3777468127 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-04-23T10:27:51.761 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:27:51.760+0000 7f312e59c640 1 --2- 172.21.15.114:0/3777468127 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f31301011a0 0x7f313019e420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T10:27:51.761 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:27:51.760+0000 7f312dd9b640 1 --2- 172.21.15.114:0/3777468127 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f3130101b40 0x7f313019e960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T10:27:51.761 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:27:51.760+0000 7f312ed9d640 1 -- 172.21.15.114:0/3777468127 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f3130101b40 msgr2=0x7f313019e960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T10:27:51.761 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:27:51.760+0000 7f312ed9d640 1 --2- 172.21.15.114:0/3777468127 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f3130101b40 0x7f313019e960 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:27:51.761 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:27:51.760+0000 7f312ed9d640 1 -- 172.21.15.114:0/3777468127 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f31301011a0 msgr2=0x7f313019e420 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T10:27:51.761 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:27:51.760+0000 7f312ed9d640 1 --2- 172.21.15.114:0/3777468127 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f31301011a0 0x7f313019e420 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:27:51.761 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:27:51.760+0000 7f312ed9d640 1 -- 172.21.15.114:0/3777468127 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f3130198d50 con 0x7f3130107030 2024-04-23T10:27:51.761 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:27:51.760+0000 7f312ed9d640 1 --2- 172.21.15.114:0/3777468127 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f3130107030 0x7f31301985d0 secure :-1 s=READY pgs=539 cs=0 l=1 rev1=1 crypto rx=0x7f312800eaa0 tx=0x7f312800ef60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T10:27:51.761 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:27:51.760+0000 7f311f7fe640 1 -- 172.21.15.114:0/3777468127 <== mon.0 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f312800cdb0 con 0x7f3130107030 2024-04-23T10:27:51.761 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:27:51.760+0000 7f311f7fe640 1 -- 172.21.15.114:0/3777468127 <== mon.0 v2:172.21.15.114:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f3128005020 con 0x7f3130107030 2024-04-23T10:27:51.761 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:27:51.760+0000 7f3134b07640 1 -- 172.21.15.114:0/3777468127 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f3130199040 con 0x7f3130107030 2024-04-23T10:27:51.761 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:27:51.760+0000 7f311f7fe640 1 -- 172.21.15.114:0/3777468127 <== mon.0 v2:172.21.15.114:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f31280116d0 con 0x7f3130107030 2024-04-23T10:27:51.761 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:27:51.760+0000 7f3134b07640 1 -- 172.21.15.114:0/3777468127 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f31301a52e0 con 0x7f3130107030 2024-04-23T10:27:51.768 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:27:51.762+0000 7f311f7fe640 1 -- 172.21.15.114:0/3777468127 <== mon.0 v2:172.21.15.114:3300/0 4 ==== mgrmap(e 25) ==== 138880+0+0 (secure 0 0 0) 0x7f31280051d0 con 0x7f3130107030 2024-04-23T10:27:51.768 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:27:51.763+0000 7f3137819740 1 Processor -- start 2024-04-23T10:27:51.768 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:27:51.763+0000 7f3137819740 1 -- start start 2024-04-23T10:27:51.768 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:27:51.763+0000 7f311f7fe640 1 --2- 172.21.15.114:0/3777468127 >> [v2:172.21.15.114:6800/1395693466,v1:172.21.15.114:6801/1395693466] conn(0x7f31180a3ec0 0x7f31180a6380 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T10:27:51.768 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:27:51.763+0000 7f3137819740 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x5603e4f1b370 0x5603e4f1b750 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T10:27:51.768 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:27:51.763+0000 7f311f7fe640 1 -- 172.21.15.114:0/3777468127 <== mon.0 v2:172.21.15.114:3300/0 5 ==== osd_map(92..92 src has 1..92) ==== 8585+0+0 (secure 0 0 0) 0x7f31280d0bd0 con 0x7f3130107030 2024-04-23T10:27:51.768 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:27:51.763+0000 7f312dd9b640 1 --2- 172.21.15.114:0/3777468127 >> [v2:172.21.15.114:6800/1395693466,v1:172.21.15.114:6801/1395693466] conn(0x7f31180a3ec0 0x7f31180a6380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T10:27:51.768 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:27:51.763+0000 7f3137819740 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x5603e4f1bc90 0x5603e4f2a350 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T10:27:51.768 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:27:51.763+0000 7f3137819740 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x5603e4f2a890 0x5603e4f2cc80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T10:27:51.768 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:27:51.763+0000 7f3137819740 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x5603e4e806d0 con 0x5603e4f1bc90 2024-04-23T10:27:51.768 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:27:51.763+0000 7f3137819740 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 -- 0x5603e4e1d1a0 con 0x5603e4f1b370 2024-04-23T10:27:51.769 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:27:51.763+0000 7f3137819740 1 -- --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_getmap magic: 0 -- 0x5603e4e7c690 con 0x5603e4f2a890 2024-04-23T10:27:51.769 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:27:51.763+0000 7f312dd9b640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x5603e4f1bc90 0x5603e4f2a350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T10:27:51.769 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:27:51.763+0000 7f312e59c640 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x5603e4f1b370 0x5603e4f1b750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T10:27:51.769 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:27:51.763+0000 7f312e59c640 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x5603e4f1b370 0x5603e4f1b750 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.119:3300/0 says I am v2:172.21.15.114:47436/0 (socket says 172.21.15.114:47436) 2024-04-23T10:27:51.769 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:27:51.763+0000 7f312e59c640 1 -- 172.21.15.114:0/2985104172 learned_addr learned my addr 172.21.15.114:0/2985104172 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-04-23T10:27:51.769 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:27:51.763+0000 7f312dd9b640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x5603e4f1bc90 0x5603e4f2a350 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.114:37146/0 (socket says 172.21.15.114:37146) 2024-04-23T10:27:51.769 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:27:51.764+0000 7f312ed9d640 1 --2- 172.21.15.114:0/2985104172 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x5603e4f2a890 0x5603e4f2cc80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T10:27:51.769 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:27:51.764+0000 7f312e59c640 1 -- 172.21.15.114:0/2985104172 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x5603e4f2a890 msgr2=0x5603e4f2cc80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T10:27:51.769 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:27:51.764+0000 7f312e59c640 1 --2- 172.21.15.114:0/2985104172 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x5603e4f2a890 0x5603e4f2cc80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:27:51.769 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:27:51.764+0000 7f312e59c640 1 -- 172.21.15.114:0/2985104172 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x5603e4f1bc90 msgr2=0x5603e4f2a350 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T10:27:51.769 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:27:51.764+0000 7f312e59c640 1 --2- 172.21.15.114:0/2985104172 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x5603e4f1bc90 0x5603e4f2a350 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:27:51.769 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:27:51.764+0000 7f312e59c640 1 -- 172.21.15.114:0/2985104172 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5603e4e275b0 con 0x5603e4f1b370 2024-04-23T10:27:51.769 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:27:51.764+0000 7f312dd9b640 1 --2- 172.21.15.114:0/3777468127 >> [v2:172.21.15.114:6800/1395693466,v1:172.21.15.114:6801/1395693466] conn(0x7f31180a3ec0 0x7f31180a6380 secure :-1 s=READY pgs=74 cs=0 l=1 rev1=1 crypto rx=0x7f3130100bf0 tx=0x7f3124013830 comp rx=0 tx=0).ready entity=mgr.15291 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T10:27:51.769 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:27:51.764+0000 7f312e59c640 1 --2- 172.21.15.114:0/2985104172 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x5603e4f1b370 0x5603e4f1b750 secure :-1 s=READY pgs=62 cs=0 l=1 rev1=1 crypto rx=0x7f312002a940 tx=0x7f3120031bc0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=e65685f45b9252ef server_cookie=0 in_seq=0 out_seq=0 2024-04-23T10:27:51.769 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:27:51.764+0000 7f311cff9640 1 -- 172.21.15.114:0/2985104172 <== mon.2 v2:172.21.15.119:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3120040070 con 0x5603e4f1b370 2024-04-23T10:27:51.770 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:27:51.764+0000 7f311cff9640 1 -- 172.21.15.114:0/2985104172 <== mon.2 v2:172.21.15.119:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f3120004030 con 0x5603e4f1b370 2024-04-23T10:27:51.770 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:27:51.764+0000 7f311cff9640 1 -- 172.21.15.114:0/2985104172 <== mon.2 v2:172.21.15.119:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3120035040 con 0x5603e4f1b370 2024-04-23T10:27:51.770 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:27:51.765+0000 7f3137819740 1 -- 172.21.15.114:0/2985104172 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x5603e4f1b370 msgr2=0x5603e4f1b750 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T10:27:51.770 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:27:51.765+0000 7f3137819740 1 --2- 172.21.15.114:0/2985104172 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x5603e4f1b370 0x5603e4f1b750 secure :-1 s=READY pgs=62 cs=0 l=1 rev1=1 crypto rx=0x7f312002a940 tx=0x7f3120031bc0 comp rx=0 tx=0).stop 2024-04-23T10:27:51.770 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:27:51.765+0000 7f3137819740 1 -- 172.21.15.114:0/2985104172 shutdown_connections 2024-04-23T10:27:51.770 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:27:51.765+0000 7f3137819740 1 --2- 172.21.15.114:0/2985104172 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x5603e4f2a890 0x5603e4f2cc80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:27:51.770 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:27:51.765+0000 7f3137819740 1 --2- 172.21.15.114:0/2985104172 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x5603e4f1bc90 0x5603e4f2a350 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:27:51.770 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:27:51.765+0000 7f3137819740 1 --2- 172.21.15.114:0/2985104172 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x5603e4f1b370 0x5603e4f1b750 unknown :-1 s=CLOSED pgs=62 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:27:51.770 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:27:51.765+0000 7f3137819740 1 -- 172.21.15.114:0/2985104172 >> 172.21.15.114:0/2985104172 conn(0x5603e4f177b0 msgr2=0x5603e4f17b90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T10:27:51.770 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:27:51.765+0000 7f3137819740 1 -- 172.21.15.114:0/2985104172 shutdown_connections 2024-04-23T10:27:51.770 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:27:51.765+0000 7f3137819740 1 -- 172.21.15.114:0/2985104172 wait complete. 2024-04-23T10:27:51.770 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:27:51.765+0000 7f3137819740 1 -- start start 2024-04-23T10:27:51.770 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:27:51.765+0000 7f3137819740 1 Processor -- start 2024-04-23T10:27:51.770 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:27:51.765+0000 7f3137819740 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x5603e4f1b370 0x5603e4f2a200 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T10:27:51.770 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:27:51.765+0000 7f3137819740 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x5603e4f1bc90 0x5603e4f274f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T10:27:51.771 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:27:51.765+0000 7f3137819740 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x5603e4f2a890 0x5603e4f27a50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T10:27:51.771 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:27:51.765+0000 7f3137819740 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x5603e4f2d710 con 0x5603e4f2a890 2024-04-23T10:27:51.771 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:27:51.765+0000 7f3137819740 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 -- 0x5603e4e1ab20 con 0x5603e4f1b370 2024-04-23T10:27:51.771 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:27:51.765+0000 7f3137819740 1 -- --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_getmap magic: 0 -- 0x5603e4f2d890 con 0x5603e4f1bc90 2024-04-23T10:27:51.771 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:27:51.765+0000 7f3137819740 10 client.0 _ll_register_callbacks cb 0x5603e4e1a530 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-23T10:27:51.771 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:27:51.766+0000 7f312ed9d640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x5603e4f2a890 0x5603e4f27a50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T10:27:51.771 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:27:51.766+0000 7f312ed9d640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x5603e4f2a890 0x5603e4f27a50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.114:37158/0 (socket says 172.21.15.114:37158) 2024-04-23T10:27:51.771 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:27:51.766+0000 7f312ed9d640 1 -- 172.21.15.114:0/1390230655 learned_addr learned my addr 172.21.15.114:0/1390230655 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-04-23T10:27:51.771 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:27:51.766+0000 7f312dd9b640 1 --2- 172.21.15.114:0/1390230655 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x5603e4f1bc90 0x5603e4f274f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T10:27:51.771 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:27:51.766+0000 7f312e59c640 1 --2- 172.21.15.114:0/1390230655 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x5603e4f1b370 0x5603e4f2a200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T10:27:51.771 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:27:51.766+0000 7f312ed9d640 1 -- 172.21.15.114:0/1390230655 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x5603e4f1bc90 msgr2=0x5603e4f274f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T10:27:51.771 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:27:51.766+0000 7f312ed9d640 1 --2- 172.21.15.114:0/1390230655 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x5603e4f1bc90 0x5603e4f274f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:27:51.771 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:27:51.766+0000 7f312ed9d640 1 -- 172.21.15.114:0/1390230655 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x5603e4f1b370 msgr2=0x5603e4f2a200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T10:27:51.771 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:27:51.766+0000 7f312ed9d640 1 --2- 172.21.15.114:0/1390230655 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x5603e4f1b370 0x5603e4f2a200 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:27:51.771 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:27:51.766+0000 7f312ed9d640 1 -- 172.21.15.114:0/1390230655 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5603e4b92920 con 0x5603e4f2a890 2024-04-23T10:27:51.772 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:27:51.766+0000 7f312ed9d640 1 --2- 172.21.15.114:0/1390230655 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x5603e4f2a890 0x5603e4f27a50 secure :-1 s=READY pgs=540 cs=0 l=1 rev1=1 crypto rx=0x7f312800efd0 tx=0x7f31280a0d50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=347cbccd80ac750a server_cookie=0 in_seq=0 out_seq=0 2024-04-23T10:27:51.772 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:27:51.766+0000 7f3115ffb640 10 client.0 ms_handle_connect on v2:172.21.15.114:3300/0 2024-04-23T10:27:51.772 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:27:51.766+0000 7f3115ffb640 1 -- 172.21.15.114:0/1390230655 <== mon.0 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f312801f040 con 0x5603e4f2a890 2024-04-23T10:27:51.772 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:27:51.766+0000 7f3115ffb640 1 -- 172.21.15.114:0/1390230655 <== mon.0 v2:172.21.15.114:3300/0 2 ==== osd_map(92..92 src has 1..92) ==== 8585+0+0 (secure 0 0 0) 0x7f31280ce650 con 0x5603e4f2a890 2024-04-23T10:27:51.772 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:27:51.766+0000 7f3137819740 1 -- 172.21.15.114:0/1390230655 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f3120004030 con 0x5603e4f2a890 2024-04-23T10:27:51.772 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:27:51.766+0000 7f3115ffb640 1 -- 172.21.15.114:0/1390230655 <== mon.0 v2:172.21.15.114:3300/0 3 ==== mon_get_version_reply(handle=1 version=21) ==== 24+0+0 (secure 0 0 0) 0x7f31280870b0 con 0x5603e4f2a890 2024-04-23T10:27:51.772 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:27:51.766+0000 7f3137819740 10 client.15738 fetch_fsmap learned FSMap version 21 2024-04-23T10:27:51.772 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:27:51.766+0000 7f3137819740 1 -- 172.21.15.114:0/1390230655 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({config=0+,fsmap=21,monmap=4+}) -- 0x5603e4c5ed30 con 0x5603e4f2a890 2024-04-23T10:27:51.772 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:27:51.766+0000 7f3115ffb640 1 -- 172.21.15.114:0/1390230655 <== mon.0 v2:172.21.15.114:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f3120004030 con 0x5603e4f2a890 2024-04-23T10:27:51.772 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:27:51.766+0000 7f3115ffb640 1 -- 172.21.15.114:0/1390230655 <== mon.0 v2:172.21.15.114:3300/0 5 ==== fsmap(e 21) ==== 8338+0+0 (secure 0 0 0) 0x7f312800f040 con 0x5603e4f2a890 2024-04-23T10:27:51.772 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:27:51.767+0000 7f3137819740 10 client.15738 fetch_fsmap finished waiting for FSMap version 21 2024-04-23T10:27:51.772 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:27:51.767+0000 7f3137819740 10 client.15738 resolve_mds: validated gid 24545 aka daemon mds.b 2024-04-23T10:27:51.772 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:27:51.767+0000 7f3137819740 20 client.15738 populate_metadata read hostname 'smithi114' 2024-04-23T10:27:51.772 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:27:51.767+0000 7f3137819740 1 --2- 172.21.15.114:0/1390230655 >> [v2:172.21.15.119:6838/937297340,v1:172.21.15.119:6839/937297340] conn(0x5603e4f3aa90 0x5603e4f3ced0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T10:27:51.772 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:27:51.767+0000 7f3137819740 4 client.15738 mds_command: new command op to 24545 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-23T10:27:51.772 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:27:51.767+0000 7f3137819740 1 -- 172.21.15.114:0/1390230655 --> [v2:172.21.15.119:6838/937297340,v1:172.21.15.119:6839/937297340] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x5603e4e3e500 con 0x5603e4f3aa90 2024-04-23T10:27:51.772 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:27:51.767+0000 7f312e59c640 1 --2- 172.21.15.114:0/1390230655 >> [v2:172.21.15.119:6838/937297340,v1:172.21.15.119:6839/937297340] conn(0x5603e4f3aa90 0x5603e4f3ced0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T10:27:51.773 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:27:51.767+0000 7f3115ffb640 10 client.15738 ms_handle_connect on v2:172.21.15.119:6838/937297340 2024-04-23T10:27:51.773 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:27:51.767+0000 7f312e59c640 1 --2- 172.21.15.114:0/1390230655 >> [v2:172.21.15.119:6838/937297340,v1:172.21.15.119:6839/937297340] conn(0x5603e4f3aa90 0x5603e4f3ced0 crc :-1 s=READY pgs=56 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=5a1c7dd5a0a244e4 server_cookie=79bee91d627b4027 in_seq=0 out_seq=0 2024-04-23T10:27:51.773 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:27:51.768+0000 7f3115ffb640 1 -- 172.21.15.114:0/1390230655 <== mds.0 v2:172.21.15.119:6838/937297340 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x5603e4e3e500 con 0x5603e4f3aa90 2024-04-23T10:27:51.773 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:27:51.768+0000 7f3115ffb640 10 client.15738 handle_command_reply: tid=0 2024-04-23T10:27:51.773 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:27:51.768+0000 7f3137819740 1 client.15738 shutdown 2024-04-23T10:27:51.773 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:27:51.768+0000 7f3137819740 1 -- 172.21.15.114:0/1390230655 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x5603e4f2a890 msgr2=0x5603e4f27a50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T10:27:51.773 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:27:51.768+0000 7f3137819740 1 --2- 172.21.15.114:0/1390230655 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x5603e4f2a890 0x5603e4f27a50 secure :-1 s=READY pgs=540 cs=0 l=1 rev1=1 crypto rx=0x7f312800efd0 tx=0x7f31280a0d50 comp rx=0 tx=0).stop 2024-04-23T10:27:51.773 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:27:51.768+0000 7f3137819740 1 -- 172.21.15.114:0/1390230655 shutdown_connections 2024-04-23T10:27:51.773 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:27:51.768+0000 7f3137819740 1 --2- 172.21.15.114:0/1390230655 >> [v2:172.21.15.119:6838/937297340,v1:172.21.15.119:6839/937297340] conn(0x5603e4f3aa90 0x5603e4f3ced0 crc :-1 s=READY pgs=56 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:27:51.773 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:27:51.768+0000 7f3137819740 1 --2- 172.21.15.114:0/1390230655 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x5603e4f2a890 0x5603e4f27a50 unknown :-1 s=CLOSED pgs=540 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:27:51.773 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:27:51.768+0000 7f3137819740 1 --2- 172.21.15.114:0/1390230655 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x5603e4f1bc90 0x5603e4f274f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:27:51.773 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:27:51.768+0000 7f3137819740 1 --2- 172.21.15.114:0/1390230655 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x5603e4f1b370 0x5603e4f2a200 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:27:51.773 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:27:51.768+0000 7f3115ffb640 0 client.15738 ms_handle_reset on v2:172.21.15.119:6838/937297340 2024-04-23T10:27:51.773 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:27:51.768+0000 7f3137819740 1 -- 172.21.15.114:0/1390230655 >> 172.21.15.114:0/1390230655 conn(0x5603e4f177b0 msgr2=0x5603e4f1d560 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T10:27:51.773 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:27:51.768+0000 7f3137819740 1 -- 172.21.15.114:0/1390230655 shutdown_connections 2024-04-23T10:27:51.774 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:27:51.768+0000 7f3137819740 1 -- 172.21.15.114:0/1390230655 wait complete. 2024-04-23T10:27:51.774 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:27:51.768+0000 7f3137819740 20 client.15738 trim_cache size 0 max 16384 2024-04-23T10:27:51.788 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:27:51.785+0000 7f3137819740 1 Processor -- start 2024-04-23T10:27:51.788 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:27:51.785+0000 7f3137819740 1 -- start start 2024-04-23T10:27:51.788 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:27:51.785+0000 7f3137819740 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x5603e4f1b370 0x5603e4f36640 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T10:27:51.788 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:27:51.785+0000 7f3137819740 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x5603e4f1bc90 0x5603e4f29d30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T10:27:51.788 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:27:51.785+0000 7f3137819740 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x5603e4f2a890 0x5603e4f427a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T10:27:51.788 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:27:51.785+0000 7f3137819740 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x7f31301ae400 con 0x5603e4f2a890 2024-04-23T10:27:51.788 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:27:51.785+0000 7f3137819740 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 -- 0x7f31301abcd0 con 0x5603e4f1bc90 2024-04-23T10:27:51.788 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:27:51.785+0000 7f3137819740 1 -- --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_getmap magic: 0 -- 0x7f31301a9b40 con 0x5603e4f1b370 2024-04-23T10:27:51.789 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:27:51.785+0000 7f312ed9d640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x5603e4f2a890 0x5603e4f427a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T10:27:51.789 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:27:51.785+0000 7f312ed9d640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x5603e4f2a890 0x5603e4f427a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.114:37172/0 (socket says 172.21.15.114:37172) 2024-04-23T10:27:51.789 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:27:51.785+0000 7f312ed9d640 1 -- 172.21.15.114:0/3714869934 learned_addr learned my addr 172.21.15.114:0/3714869934 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-04-23T10:27:51.789 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:27:51.785+0000 7f312dd9b640 1 --2- 172.21.15.114:0/3714869934 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x5603e4f1bc90 0x5603e4f29d30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T10:27:51.789 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:27:51.785+0000 7f312ed9d640 1 -- 172.21.15.114:0/3714869934 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x5603e4f1b370 msgr2=0x5603e4f36640 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T10:27:51.789 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:27:51.785+0000 7f312e59c640 1 --2- 172.21.15.114:0/3714869934 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x5603e4f1b370 0x5603e4f36640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T10:27:51.789 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:27:51.785+0000 7f312ed9d640 1 --2- 172.21.15.114:0/3714869934 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x5603e4f1b370 0x5603e4f36640 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:27:51.789 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:27:51.785+0000 7f312ed9d640 1 -- 172.21.15.114:0/3714869934 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x5603e4f1bc90 msgr2=0x5603e4f29d30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T10:27:51.789 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:27:51.786+0000 7f312ed9d640 1 --2- 172.21.15.114:0/3714869934 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x5603e4f1bc90 0x5603e4f29d30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:27:51.789 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:27:51.786+0000 7f312ed9d640 1 -- 172.21.15.114:0/3714869934 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5603e4f1bac0 con 0x5603e4f2a890 2024-04-23T10:27:51.789 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:27:51.786+0000 7f312dd9b640 1 --2- 172.21.15.114:0/3714869934 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x5603e4f1bc90 0x5603e4f29d30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-23T10:27:51.789 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:27:51.786+0000 7f312ed9d640 1 --2- 172.21.15.114:0/3714869934 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x5603e4f2a890 0x5603e4f427a0 secure :-1 s=READY pgs=541 cs=0 l=1 rev1=1 crypto rx=0x7f31280ce270 tx=0x7f312809c600 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c8907329e4b3265c server_cookie=0 in_seq=0 out_seq=0 2024-04-23T10:27:51.789 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:27:51.786+0000 7f3117fff640 1 -- 172.21.15.114:0/3714869934 <== mon.0 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f312809bc00 con 0x5603e4f2a890 2024-04-23T10:27:51.789 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:27:51.786+0000 7f3117fff640 1 -- 172.21.15.114:0/3714869934 <== mon.0 v2:172.21.15.114:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f31280ce0a0 con 0x5603e4f2a890 2024-04-23T10:27:51.789 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:27:51.786+0000 7f3117fff640 1 -- 172.21.15.114:0/3714869934 <== mon.0 v2:172.21.15.114:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f31280bbdc0 con 0x5603e4f2a890 2024-04-23T10:27:51.789 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:27:51.786+0000 7f3137819740 1 -- 172.21.15.114:0/3714869934 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x5603e4f2a890 msgr2=0x5603e4f427a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T10:27:51.790 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:27:51.786+0000 7f3137819740 1 --2- 172.21.15.114:0/3714869934 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x5603e4f2a890 0x5603e4f427a0 secure :-1 s=READY pgs=541 cs=0 l=1 rev1=1 crypto rx=0x7f31280ce270 tx=0x7f312809c600 comp rx=0 tx=0).stop 2024-04-23T10:27:51.790 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:27:51.786+0000 7f3137819740 1 -- 172.21.15.114:0/3714869934 shutdown_connections 2024-04-23T10:27:51.790 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:27:51.786+0000 7f3137819740 1 --2- 172.21.15.114:0/3714869934 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x5603e4f2a890 0x5603e4f427a0 unknown :-1 s=CLOSED pgs=541 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:27:51.790 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:27:51.786+0000 7f3137819740 1 --2- 172.21.15.114:0/3714869934 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x5603e4f1bc90 0x5603e4f29d30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:27:51.790 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:27:51.786+0000 7f3137819740 1 --2- 172.21.15.114:0/3714869934 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x5603e4f1b370 0x5603e4f36640 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:27:51.790 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:27:51.786+0000 7f3137819740 1 -- 172.21.15.114:0/3714869934 >> 172.21.15.114:0/3714869934 conn(0x5603e4f177b0 msgr2=0x5603e4f25dc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T10:27:51.790 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:27:51.786+0000 7f3137819740 1 -- 172.21.15.114:0/3714869934 shutdown_connections 2024-04-23T10:27:51.790 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:27:51.786+0000 7f3137819740 1 -- 172.21.15.114:0/3714869934 wait complete. 2024-04-23T10:27:51.790 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:27:51.787+0000 7f3137819740 1 -- start start 2024-04-23T10:27:51.790 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:27:51.787+0000 7f3137819740 1 Processor -- start 2024-04-23T10:27:51.790 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:27:51.787+0000 7f3137819740 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x5603e4f1b370 0x5603e4f32d40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T10:27:51.790 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:27:51.787+0000 7f3137819740 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x5603e4f1bc90 0x5603e4f24770 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T10:27:51.790 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:27:51.787+0000 7f3137819740 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x5603e4f2a890 0x5603e4f24cb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T10:27:51.790 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:27:51.787+0000 7f3137819740 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x7f31301aa190 con 0x5603e4f1b370 2024-04-23T10:27:51.790 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:27:51.787+0000 7f3137819740 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 -- 0x7f31301ab600 con 0x5603e4f2a890 2024-04-23T10:27:51.791 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:27:51.787+0000 7f3137819740 1 -- --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_getmap magic: 0 -- 0x7f31301aa860 con 0x5603e4f1bc90 2024-04-23T10:27:51.791 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:27:51.787+0000 7f3137819740 10 client.0 _ll_register_callbacks cb 0x7f312809c990 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-23T10:27:51.791 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:27:51.787+0000 7f312e59c640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x5603e4f1b370 0x5603e4f32d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T10:27:51.791 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:27:51.787+0000 7f312e59c640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x5603e4f1b370 0x5603e4f32d40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.114:37184/0 (socket says 172.21.15.114:37184) 2024-04-23T10:27:51.791 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:27:51.787+0000 7f312e59c640 1 -- 172.21.15.114:0/795192866 learned_addr learned my addr 172.21.15.114:0/795192866 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-04-23T10:27:51.791 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:27:51.787+0000 7f312ed9d640 1 --2- 172.21.15.114:0/795192866 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x5603e4f2a890 0x5603e4f24cb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T10:27:51.791 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:27:51.787+0000 7f312dd9b640 1 --2- 172.21.15.114:0/795192866 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x5603e4f1bc90 0x5603e4f24770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T10:27:51.791 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:27:51.787+0000 7f312e59c640 1 -- 172.21.15.114:0/795192866 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x5603e4f1bc90 msgr2=0x5603e4f24770 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T10:27:51.791 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:27:51.787+0000 7f312e59c640 1 --2- 172.21.15.114:0/795192866 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x5603e4f1bc90 0x5603e4f24770 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:27:51.791 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:27:51.787+0000 7f312e59c640 1 -- 172.21.15.114:0/795192866 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x5603e4f2a890 msgr2=0x5603e4f24cb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T10:27:51.791 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:27:51.787+0000 7f312e59c640 1 --2- 172.21.15.114:0/795192866 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x5603e4f2a890 0x5603e4f24cb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:27:51.791 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:27:51.787+0000 7f312e59c640 1 -- 172.21.15.114:0/795192866 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5603e4f251f0 con 0x5603e4f1b370 2024-04-23T10:27:51.791 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:27:51.787+0000 7f312dd9b640 1 --2- 172.21.15.114:0/795192866 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x5603e4f1bc90 0x5603e4f24770 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-23T10:27:51.791 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:27:51.787+0000 7f312e59c640 1 --2- 172.21.15.114:0/795192866 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x5603e4f1b370 0x5603e4f32d40 secure :-1 s=READY pgs=542 cs=0 l=1 rev1=1 crypto rx=0x7f3120009930 tx=0x7f31200058a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=10374fa1fdbd3d1d server_cookie=0 in_seq=0 out_seq=0 2024-04-23T10:27:51.792 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:27:51.787+0000 7f3115ffb640 10 client.0 ms_handle_connect on v2:172.21.15.114:3300/0 2024-04-23T10:27:51.792 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:27:51.788+0000 7f3115ffb640 1 -- 172.21.15.114:0/795192866 <== mon.0 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3120054020 con 0x5603e4f1b370 2024-04-23T10:27:51.792 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:27:51.788+0000 7f3115ffb640 1 -- 172.21.15.114:0/795192866 <== mon.0 v2:172.21.15.114:3300/0 2 ==== osd_map(92..92 src has 1..92) ==== 8585+0+0 (secure 0 0 0) 0x7f31200314a0 con 0x5603e4f1b370 2024-04-23T10:27:51.792 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:27:51.788+0000 7f3137819740 1 -- 172.21.15.114:0/795192866 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f31280ce0a0 con 0x5603e4f1b370 2024-04-23T10:27:51.792 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:27:51.788+0000 7f3115ffb640 1 -- 172.21.15.114:0/795192866 <== mon.0 v2:172.21.15.114:3300/0 3 ==== mon_get_version_reply(handle=1 version=21) ==== 24+0+0 (secure 0 0 0) 0x7f312005f050 con 0x5603e4f1b370 2024-04-23T10:27:51.792 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:27:51.788+0000 7f3137819740 10 client.15744 fetch_fsmap learned FSMap version 21 2024-04-23T10:27:51.792 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:27:51.788+0000 7f3137819740 1 -- 172.21.15.114:0/795192866 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({config=0+,fsmap=21,monmap=4+}) -- 0x5603e4f2bda0 con 0x5603e4f1b370 2024-04-23T10:27:51.792 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:27:51.788+0000 7f3115ffb640 1 -- 172.21.15.114:0/795192866 <== mon.0 v2:172.21.15.114:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f31280ce0a0 con 0x5603e4f1b370 2024-04-23T10:27:51.792 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:27:51.788+0000 7f3115ffb640 1 -- 172.21.15.114:0/795192866 <== mon.0 v2:172.21.15.114:3300/0 5 ==== fsmap(e 21) ==== 8338+0+0 (secure 0 0 0) 0x7f3120005250 con 0x5603e4f1b370 2024-04-23T10:27:51.792 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:27:51.788+0000 7f3137819740 10 client.15744 fetch_fsmap finished waiting for FSMap version 21 2024-04-23T10:27:51.792 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:27:51.788+0000 7f3137819740 10 client.15744 resolve_mds: validated gid 24545 aka daemon mds.b 2024-04-23T10:27:51.792 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:27:51.788+0000 7f3137819740 20 client.15744 populate_metadata read hostname 'smithi114' 2024-04-23T10:27:51.792 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:27:51.788+0000 7f3137819740 1 --2- 172.21.15.114:0/795192866 >> [v2:172.21.15.119:6838/937297340,v1:172.21.15.119:6839/937297340] conn(0x5603e4fa10c0 0x5603e4fa34b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T10:27:51.792 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:27:51.788+0000 7f3137819740 4 client.15744 mds_command: new command op to 24545 tid=0 multi_id=0 [{"prefix": "quiesce db", "set_id": "750460ef", "release": true, "await": true}] 2024-04-23T10:27:51.792 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:27:51.788+0000 7f3137819740 1 -- 172.21.15.114:0/795192866 --> [v2:172.21.15.119:6838/937297340,v1:172.21.15.119:6839/937297340] -- command(tid 0: {"prefix": "quiesce db", "set_id": "750460ef", "release": true, "await": true}) -- 0x5603e4e8e0f0 con 0x5603e4fa10c0 2024-04-23T10:27:51.792 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:27:51.788+0000 7f312dd9b640 1 --2- 172.21.15.114:0/795192866 >> [v2:172.21.15.119:6838/937297340,v1:172.21.15.119:6839/937297340] conn(0x5603e4fa10c0 0x5603e4fa34b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T10:27:51.793 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:27:51.789+0000 7f3115ffb640 10 client.15744 ms_handle_connect on v2:172.21.15.119:6838/937297340 2024-04-23T10:27:51.793 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:27:51.789+0000 7f312dd9b640 1 --2- 172.21.15.114:0/795192866 >> [v2:172.21.15.119:6838/937297340,v1:172.21.15.119:6839/937297340] conn(0x5603e4fa10c0 0x5603e4fa34b0 crc :-1 s=READY pgs=57 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=663b0cac448f5ed3 server_cookie=7952fe4d8a20ef98 in_seq=0 out_seq=0 2024-04-23T10:27:51.794 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:27:51.792+0000 7f3115ffb640 1 -- 172.21.15.114:0/795192866 <== mds.0 v2:172.21.15.119:6838/937297340 1 ==== command_reply(tid 0: 0 ) ==== 8+0+588 (crc 0 0 0) 0x5603e4e8e0f0 con 0x5603e4fa10c0 2024-04-23T10:27:51.794 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:27:51.792+0000 7f3115ffb640 10 client.15744 handle_command_reply: tid=0 2024-04-23T10:27:51.794 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:27:51.792+0000 7f3137819740 1 client.15744 shutdown 2024-04-23T10:27:51.794 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:27:51.792+0000 7f3137819740 1 -- 172.21.15.114:0/795192866 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x5603e4f1b370 msgr2=0x5603e4f32d40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T10:27:51.794 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:27:51.792+0000 7f3137819740 1 --2- 172.21.15.114:0/795192866 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x5603e4f1b370 0x5603e4f32d40 secure :-1 s=READY pgs=542 cs=0 l=1 rev1=1 crypto rx=0x7f3120009930 tx=0x7f31200058a0 comp rx=0 tx=0).stop 2024-04-23T10:27:51.794 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:27:51.792+0000 7f3137819740 1 -- 172.21.15.114:0/795192866 shutdown_connections 2024-04-23T10:27:51.794 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:27:51.792+0000 7f3137819740 1 --2- 172.21.15.114:0/795192866 >> [v2:172.21.15.119:6838/937297340,v1:172.21.15.119:6839/937297340] conn(0x5603e4fa10c0 0x5603e4fa34b0 crc :-1 s=READY pgs=57 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:27:51.795 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:27:51.792+0000 7f3137819740 1 --2- 172.21.15.114:0/795192866 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x5603e4f2a890 0x5603e4f24cb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:27:51.795 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:27:51.792+0000 7f3115ffb640 0 client.15744 ms_handle_reset on v2:172.21.15.119:6838/937297340 2024-04-23T10:27:51.795 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:27:51.792+0000 7f3137819740 1 --2- 172.21.15.114:0/795192866 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x5603e4f1bc90 0x5603e4f24770 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:27:51.795 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:27:51.792+0000 7f3137819740 1 --2- 172.21.15.114:0/795192866 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x5603e4f1b370 0x5603e4f32d40 unknown :-1 s=CLOSED pgs=542 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:27:51.795 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:27:51.792+0000 7f3137819740 1 -- 172.21.15.114:0/795192866 >> 172.21.15.114:0/795192866 conn(0x5603e4f177b0 msgr2=0x5603e4f16140 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T10:27:51.795 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:27:51.793+0000 7f3137819740 1 -- 172.21.15.114:0/795192866 shutdown_connections 2024-04-23T10:27:51.795 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:27:51.793+0000 7f3137819740 1 -- 172.21.15.114:0/795192866 wait complete. 2024-04-23T10:27:51.795 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:27:51.793+0000 7f3137819740 20 client.15744 trim_cache size 0 max 16384 2024-04-23T10:27:51.795 INFO:teuthology.orchestra.run.smithi114.stdout:{ 2024-04-23T10:27:51.795 INFO:teuthology.orchestra.run.smithi114.stdout: "epoch": 21, 2024-04-23T10:27:51.795 INFO:teuthology.orchestra.run.smithi114.stdout: "leader": 24545, 2024-04-23T10:27:51.795 INFO:teuthology.orchestra.run.smithi114.stdout: "set_version": 8, 2024-04-23T10:27:51.796 INFO:teuthology.orchestra.run.smithi114.stdout: "sets": { 2024-04-23T10:27:51.796 INFO:teuthology.orchestra.run.smithi114.stdout: "750460ef": { 2024-04-23T10:27:51.796 INFO:teuthology.orchestra.run.smithi114.stdout: "version": 8, 2024-04-23T10:27:51.796 INFO:teuthology.orchestra.run.smithi114.stdout: "age_ref": 0.0, 2024-04-23T10:27:51.796 INFO:teuthology.orchestra.run.smithi114.stdout: "state": { 2024-04-23T10:27:51.796 INFO:teuthology.orchestra.run.smithi114.stdout: "name": "RELEASED", 2024-04-23T10:27:51.796 INFO:teuthology.orchestra.run.smithi114.stdout: "age": 0.0 2024-04-23T10:27:51.796 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-23T10:27:51.796 INFO:teuthology.orchestra.run.smithi114.stdout: "timeout": 60.0, 2024-04-23T10:27:51.796 INFO:teuthology.orchestra.run.smithi114.stdout: "expiration": 82.7, 2024-04-23T10:27:51.796 INFO:teuthology.orchestra.run.smithi114.stdout: "members": { 2024-04-23T10:27:51.796 INFO:teuthology.orchestra.run.smithi114.stdout: "file:/": { 2024-04-23T10:27:51.796 INFO:teuthology.orchestra.run.smithi114.stdout: "excluded": false, 2024-04-23T10:27:51.796 INFO:teuthology.orchestra.run.smithi114.stdout: "state": { 2024-04-23T10:27:51.796 INFO:teuthology.orchestra.run.smithi114.stdout: "name": "RELEASED", 2024-04-23T10:27:51.797 INFO:teuthology.orchestra.run.smithi114.stdout: "age": 0.0 2024-04-23T10:27:51.797 INFO:teuthology.orchestra.run.smithi114.stdout: } 2024-04-23T10:27:51.797 INFO:teuthology.orchestra.run.smithi114.stdout: } 2024-04-23T10:27:51.797 INFO:teuthology.orchestra.run.smithi114.stdout: } 2024-04-23T10:27:51.797 INFO:teuthology.orchestra.run.smithi114.stdout: } 2024-04-23T10:27:51.797 INFO:teuthology.orchestra.run.smithi114.stdout: } 2024-04-23T10:27:51.797 INFO:teuthology.orchestra.run.smithi114.stdout:} 2024-04-23T10:27:51.798 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:27:51.793+0000 7f3134b07640 1 -- 172.21.15.114:0/3777468127 >> [v2:172.21.15.114:6800/1395693466,v1:172.21.15.114:6801/1395693466] conn(0x7f31180a3ec0 msgr2=0x7f31180a6380 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T10:27:51.798 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:27:51.793+0000 7f3134b07640 1 --2- 172.21.15.114:0/3777468127 >> [v2:172.21.15.114:6800/1395693466,v1:172.21.15.114:6801/1395693466] conn(0x7f31180a3ec0 0x7f31180a6380 secure :-1 s=READY pgs=74 cs=0 l=1 rev1=1 crypto rx=0x7f3130100bf0 tx=0x7f3124013830 comp rx=0 tx=0).stop 2024-04-23T10:27:51.798 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:27:51.793+0000 7f3134b07640 1 -- 172.21.15.114:0/3777468127 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f3130107030 msgr2=0x7f31301985d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T10:27:51.798 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:27:51.793+0000 7f3134b07640 1 --2- 172.21.15.114:0/3777468127 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f3130107030 0x7f31301985d0 secure :-1 s=READY pgs=539 cs=0 l=1 rev1=1 crypto rx=0x7f312800eaa0 tx=0x7f312800ef60 comp rx=0 tx=0).stop 2024-04-23T10:27:51.798 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:27:51.793+0000 7f3134b07640 1 -- 172.21.15.114:0/3777468127 shutdown_connections 2024-04-23T10:27:51.798 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:27:51.793+0000 7f3134b07640 1 --2- 172.21.15.114:0/3777468127 >> [v2:172.21.15.114:6800/1395693466,v1:172.21.15.114:6801/1395693466] conn(0x7f31180a3ec0 0x7f31180a6380 unknown :-1 s=CLOSED pgs=74 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:27:51.798 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:27:51.793+0000 7f3134b07640 1 --2- 172.21.15.114:0/3777468127 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f3130107030 0x7f31301985d0 unknown :-1 s=CLOSED pgs=539 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:27:51.798 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:27:51.793+0000 7f3134b07640 1 --2- 172.21.15.114:0/3777468127 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f3130101b40 0x7f313019e960 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:27:51.798 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:27:51.793+0000 7f3134b07640 1 --2- 172.21.15.114:0/3777468127 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f31301011a0 0x7f313019e420 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:27:51.798 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:27:51.793+0000 7f3134b07640 1 -- 172.21.15.114:0/3777468127 >> 172.21.15.114:0/3777468127 conn(0x7f3130076c50 msgr2=0x7f313006a720 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T10:27:51.798 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:27:51.793+0000 7f3134b07640 1 -- 172.21.15.114:0/3777468127 shutdown_connections 2024-04-23T10:27:51.798 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:27:51.793+0000 7f3134b07640 1 -- 172.21.15.114:0/3777468127 wait complete. 2024-04-23T10:27:51.799 INFO:tasks.workunit.client.0.smithi114.stderr:+ dbench 1 2024-04-23T10:27:51.803 INFO:tasks.quiescer.fs.[cephfs]:Successfully released set '750460ef', seconds elapsed: 23.8 2024-04-23T10:27:51.805 INFO:tasks.workunit.client.0.smithi114.stdout:dbench version 4.00 - Copyright Andrew Tridgell 1999-2004 2024-04-23T10:27:51.805 INFO:tasks.workunit.client.0.smithi114.stdout: 2024-04-23T10:27:51.806 INFO:tasks.workunit.client.0.smithi114.stdout:Running for 600 seconds with load '/usr/share/dbench/client.txt' and minimum warmup 120 secs 2024-04-23T10:27:51.806 INFO:tasks.workunit.client.0.smithi114.stdout:0 of 1 processes prepared for launch 0 sec 2024-04-23T10:27:51.906 INFO:tasks.workunit.client.0.smithi114.stdout:1 of 1 processes prepared for launch 0 sec 2024-04-23T10:27:51.906 INFO:tasks.workunit.client.0.smithi114.stdout:releasing clients 2024-04-23T10:27:52.492 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 23 10:27:52 smithi119 ceph-mon[32608]: pgmap v40: 97 pgs: 97 active+clean; 1.4 MiB data, 377 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 891 B/s wr, 6 op/s; 12 B/s, 0 keys/s, 1 objects/s recovering 2024-04-23T10:27:52.492 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 23 10:27:52 smithi119 ceph-mon[32608]: from='client.? 172.21.15.114:0/1245947554' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-23T10:27:52.504 INFO:journalctl@ceph.mon.a.smithi114.stdout:Apr 23 10:27:52 smithi114 ceph-mon[29255]: pgmap v40: 97 pgs: 97 active+clean; 1.4 MiB data, 377 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 891 B/s wr, 6 op/s; 12 B/s, 0 keys/s, 1 objects/s recovering 2024-04-23T10:27:52.504 INFO:journalctl@ceph.mon.a.smithi114.stdout:Apr 23 10:27:52 smithi114 ceph-mon[29255]: from='client.? 172.21.15.114:0/1245947554' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-23T10:27:52.577 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 23 10:27:52 smithi204 ceph-mon[32649]: pgmap v40: 97 pgs: 97 active+clean; 1.4 MiB data, 377 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 891 B/s wr, 6 op/s; 12 B/s, 0 keys/s, 1 objects/s recovering 2024-04-23T10:27:52.578 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 23 10:27:52 smithi204 ceph-mon[32649]: from='client.? 172.21.15.114:0/1245947554' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-23T10:27:52.906 INFO:tasks.workunit.client.0.smithi114.stdout: 1 3343 35.38 MB/sec warmup 1 sec latency 7.620 ms 2024-04-23T10:27:53.907 INFO:tasks.workunit.client.0.smithi114.stdout: 1 6056 23.92 MB/sec warmup 2 sec latency 20.339 ms 2024-04-23T10:27:54.492 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 23 10:27:54 smithi119 ceph-mon[32608]: pgmap v41: 97 pgs: 97 active+clean; 1.4 MiB data, 377 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 1.2 KiB/s wr, 6 op/s; 10 B/s, 0 keys/s, 1 objects/s recovering 2024-04-23T10:27:54.504 INFO:journalctl@ceph.mon.a.smithi114.stdout:Apr 23 10:27:54 smithi114 ceph-mon[29255]: pgmap v41: 97 pgs: 97 active+clean; 1.4 MiB data, 377 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 1.2 KiB/s wr, 6 op/s; 10 B/s, 0 keys/s, 1 objects/s recovering 2024-04-23T10:27:54.577 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 23 10:27:54 smithi204 ceph-mon[32649]: pgmap v41: 97 pgs: 97 active+clean; 1.4 MiB data, 377 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 1.2 KiB/s wr, 6 op/s; 10 B/s, 0 keys/s, 1 objects/s recovering 2024-04-23T10:27:54.906 INFO:tasks.workunit.client.0.smithi114.stdout: 1 8877 21.51 MB/sec warmup 3 sec latency 12.364 ms 2024-04-23T10:27:55.907 INFO:tasks.workunit.client.0.smithi114.stdout: 1 11592 19.78 MB/sec warmup 4 sec latency 20.367 ms 2024-04-23T10:27:56.492 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 23 10:27:56 smithi119 ceph-mon[32608]: pgmap v42: 97 pgs: 97 active+clean; 1.4 MiB data, 377 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 1023 B/s wr, 5 op/s; 9 B/s, 0 keys/s, 0 objects/s recovering 2024-04-23T10:27:56.504 INFO:journalctl@ceph.mon.a.smithi114.stdout:Apr 23 10:27:56 smithi114 ceph-mon[29255]: pgmap v42: 97 pgs: 97 active+clean; 1.4 MiB data, 377 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 1023 B/s wr, 5 op/s; 9 B/s, 0 keys/s, 0 objects/s recovering 2024-04-23T10:27:56.578 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 23 10:27:56 smithi204 ceph-mon[32649]: pgmap v42: 97 pgs: 97 active+clean; 1.4 MiB data, 377 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 1023 B/s wr, 5 op/s; 9 B/s, 0 keys/s, 0 objects/s recovering 2024-04-23T10:27:56.907 INFO:tasks.workunit.client.0.smithi114.stdout: 1 14667 19.49 MB/sec warmup 5 sec latency 9.629 ms 2024-04-23T10:27:57.907 INFO:tasks.workunit.client.0.smithi114.stdout: 1 17980 19.37 MB/sec warmup 6 sec latency 12.686 ms 2024-04-23T10:27:58.492 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 23 10:27:58 smithi119 ceph-mon[32608]: pgmap v43: 97 pgs: 97 active+clean; 7.4 MiB data, 414 MiB used, 1.0 TiB / 1.0 TiB avail; 264 KiB/s rd, 618 KiB/s wr, 81 op/s; 9 B/s, 0 keys/s, 0 objects/s recovering 2024-04-23T10:27:58.504 INFO:journalctl@ceph.mon.a.smithi114.stdout:Apr 23 10:27:58 smithi114 ceph-mon[29255]: pgmap v43: 97 pgs: 97 active+clean; 7.4 MiB data, 414 MiB used, 1.0 TiB / 1.0 TiB avail; 264 KiB/s rd, 618 KiB/s wr, 81 op/s; 9 B/s, 0 keys/s, 0 objects/s recovering 2024-04-23T10:27:58.578 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 23 10:27:58 smithi204 ceph-mon[32649]: pgmap v43: 97 pgs: 97 active+clean; 7.4 MiB data, 414 MiB used, 1.0 TiB / 1.0 TiB avail; 264 KiB/s rd, 618 KiB/s wr, 81 op/s; 9 B/s, 0 keys/s, 0 objects/s recovering 2024-04-23T10:27:58.907 INFO:tasks.workunit.client.0.smithi114.stdout: 1 20769 18.45 MB/sec warmup 7 sec latency 23.382 ms 2024-04-23T10:27:59.907 INFO:tasks.workunit.client.0.smithi114.stdout: 1 23150 17.64 MB/sec warmup 8 sec latency 27.451 ms 2024-04-23T10:28:00.492 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 23 10:28:00 smithi119 ceph-mon[32608]: pgmap v44: 97 pgs: 97 active+clean; 7.8 MiB data, 414 MiB used, 1.0 TiB / 1.0 TiB avail; 264 KiB/s rd, 660 KiB/s wr, 82 op/s 2024-04-23T10:28:00.504 INFO:journalctl@ceph.mon.a.smithi114.stdout:Apr 23 10:28:00 smithi114 ceph-mon[29255]: pgmap v44: 97 pgs: 97 active+clean; 7.8 MiB data, 414 MiB used, 1.0 TiB / 1.0 TiB avail; 264 KiB/s rd, 660 KiB/s wr, 82 op/s 2024-04-23T10:28:00.578 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 23 10:28:00 smithi204 ceph-mon[32649]: pgmap v44: 97 pgs: 97 active+clean; 7.8 MiB data, 414 MiB used, 1.0 TiB / 1.0 TiB avail; 264 KiB/s rd, 660 KiB/s wr, 82 op/s 2024-04-23T10:28:00.907 INFO:tasks.workunit.client.0.smithi114.stdout: 1 25665 17.25 MB/sec warmup 9 sec latency 56.015 ms 2024-04-23T10:28:01.401 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:28:01.399+0000 7f1fcf8ef640 1 -- 172.21.15.114:0/4063230208 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f1fc8103b00 msgr2=0x7f1fc810bfe0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T10:28:01.401 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:28:01.399+0000 7f1fcf8ef640 1 --2- 172.21.15.114:0/4063230208 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f1fc8103b00 0x7f1fc810bfe0 secure :-1 s=READY pgs=63 cs=0 l=1 rev1=1 crypto rx=0x7f1fb8009860 tx=0x7f1fb802a460 comp rx=0 tx=0).stop 2024-04-23T10:28:01.401 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:28:01.399+0000 7f1fcf8ef640 1 -- 172.21.15.114:0/4063230208 shutdown_connections 2024-04-23T10:28:01.401 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:28:01.399+0000 7f1fcf8ef640 1 --2- 172.21.15.114:0/4063230208 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f1fc810c520 0x7f1fc810e9c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:28:01.402 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:28:01.399+0000 7f1fcf8ef640 1 --2- 172.21.15.114:0/4063230208 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f1fc8103b00 0x7f1fc810bfe0 unknown :-1 s=CLOSED pgs=63 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:28:01.402 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:28:01.399+0000 7f1fcf8ef640 1 --2- 172.21.15.114:0/4063230208 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f1fc81031e0 0x7f1fc81035c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:28:01.402 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:28:01.399+0000 7f1fcf8ef640 1 -- 172.21.15.114:0/4063230208 >> 172.21.15.114:0/4063230208 conn(0x7f1fc8077fd0 msgr2=0x7f1fc80783e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T10:28:01.402 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:28:01.400+0000 7f1fcf8ef640 1 -- 172.21.15.114:0/4063230208 shutdown_connections 2024-04-23T10:28:01.402 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:28:01.400+0000 7f1fcf8ef640 1 -- 172.21.15.114:0/4063230208 wait complete. 2024-04-23T10:28:01.402 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:28:01.400+0000 7f1fcf8ef640 1 Processor -- start 2024-04-23T10:28:01.402 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:28:01.400+0000 7f1fcf8ef640 1 -- start start 2024-04-23T10:28:01.402 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:28:01.400+0000 7f1fcf8ef640 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f1fc81031e0 0x7f1fc8198760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T10:28:01.402 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:28:01.400+0000 7f1fcf8ef640 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f1fc8103b00 0x7f1fc8198ca0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T10:28:01.402 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:28:01.400+0000 7f1fcf8ef640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f1fc810c520 0x7f1fc819d020 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T10:28:01.403 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:28:01.400+0000 7f1fcf8ef640 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x7f1fc81121e0 con 0x7f1fc810c520 2024-04-23T10:28:01.403 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:28:01.400+0000 7f1fcf8ef640 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 -- 0x7f1fc8112060 con 0x7f1fc8103b00 2024-04-23T10:28:01.403 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:28:01.400+0000 7f1fcf8ef640 1 -- --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_getmap magic: 0 -- 0x7f1fc8112360 con 0x7f1fc81031e0 2024-04-23T10:28:01.403 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:28:01.400+0000 7f1fcde8c640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f1fc810c520 0x7f1fc819d020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T10:28:01.403 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:28:01.400+0000 7f1fcde8c640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f1fc810c520 0x7f1fc819d020 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.114:51806/0 (socket says 172.21.15.114:51806) 2024-04-23T10:28:01.403 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:28:01.400+0000 7f1fcde8c640 1 -- 172.21.15.114:0/336668180 learned_addr learned my addr 172.21.15.114:0/336668180 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-04-23T10:28:01.403 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:28:01.401+0000 7f1fcce8a640 1 --2- 172.21.15.114:0/336668180 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f1fc8103b00 0x7f1fc8198ca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T10:28:01.403 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:28:01.401+0000 7f1fcd68b640 1 --2- 172.21.15.114:0/336668180 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f1fc81031e0 0x7f1fc8198760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T10:28:01.403 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:28:01.401+0000 7f1fcde8c640 1 -- 172.21.15.114:0/336668180 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f1fc81031e0 msgr2=0x7f1fc8198760 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T10:28:01.403 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:28:01.401+0000 7f1fcde8c640 1 --2- 172.21.15.114:0/336668180 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f1fc81031e0 0x7f1fc8198760 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:28:01.404 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:28:01.401+0000 7f1fcde8c640 1 -- 172.21.15.114:0/336668180 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f1fc8103b00 msgr2=0x7f1fc8198ca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T10:28:01.404 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:28:01.401+0000 7f1fcde8c640 1 --2- 172.21.15.114:0/336668180 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f1fc8103b00 0x7f1fc8198ca0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:28:01.404 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:28:01.401+0000 7f1fcde8c640 1 -- 172.21.15.114:0/336668180 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f1fc819d750 con 0x7f1fc810c520 2024-04-23T10:28:01.404 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:28:01.401+0000 7f1fcde8c640 1 --2- 172.21.15.114:0/336668180 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f1fc810c520 0x7f1fc819d020 secure :-1 s=READY pgs=543 cs=0 l=1 rev1=1 crypto rx=0x7f1fc000e970 tx=0x7f1fc000ee30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T10:28:01.404 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:28:01.401+0000 7f1fbe7fc640 1 -- 172.21.15.114:0/336668180 <== mon.0 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1fc000ccc0 con 0x7f1fc810c520 2024-04-23T10:28:01.404 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:28:01.401+0000 7f1fcf8ef640 1 -- 172.21.15.114:0/336668180 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f1fc819da40 con 0x7f1fc810c520 2024-04-23T10:28:01.404 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:28:01.401+0000 7f1fbe7fc640 1 -- 172.21.15.114:0/336668180 <== mon.0 v2:172.21.15.114:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f1fc00054d0 con 0x7f1fc810c520 2024-04-23T10:28:01.404 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:28:01.401+0000 7f1fbe7fc640 1 -- 172.21.15.114:0/336668180 <== mon.0 v2:172.21.15.114:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1fc0010430 con 0x7f1fc810c520 2024-04-23T10:28:01.404 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:28:01.401+0000 7f1fcf8ef640 1 -- 172.21.15.114:0/336668180 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f1fc81a52d0 con 0x7f1fc810c520 2024-04-23T10:28:01.413 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:28:01.403+0000 7f1fbe7fc640 1 -- 172.21.15.114:0/336668180 <== mon.0 v2:172.21.15.114:3300/0 4 ==== mgrmap(e 25) ==== 138880+0+0 (secure 0 0 0) 0x7f1fc0005020 con 0x7f1fc810c520 2024-04-23T10:28:01.413 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:28:01.404+0000 7f1fd2606740 1 Processor -- start 2024-04-23T10:28:01.413 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:28:01.404+0000 7f1fd2606740 1 -- start start 2024-04-23T10:28:01.413 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:28:01.404+0000 7f1fbe7fc640 1 --2- 172.21.15.114:0/336668180 >> [v2:172.21.15.114:6800/1395693466,v1:172.21.15.114:6801/1395693466] conn(0x7f1fb40a3ee0 0x7f1fb40a63a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T10:28:01.413 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:28:01.404+0000 7f1fcce8a640 1 --2- 172.21.15.114:0/336668180 >> [v2:172.21.15.114:6800/1395693466,v1:172.21.15.114:6801/1395693466] conn(0x7f1fb40a3ee0 0x7f1fb40a63a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T10:28:01.413 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:28:01.404+0000 7f1fd2606740 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x5588bb480620 0x5588bb480a00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T10:28:01.413 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:28:01.404+0000 7f1fd2606740 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x5588bb480f40 0x5588bb48b2c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T10:28:01.413 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:28:01.404+0000 7f1fd2606740 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x5588bb48b800 0x5588bb48dbf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T10:28:01.414 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:28:01.404+0000 7f1fd2606740 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x5588bb3e5970 con 0x5588bb480620 2024-04-23T10:28:01.414 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:28:01.404+0000 7f1fd2606740 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 -- 0x5588bb3824a0 con 0x5588bb480f40 2024-04-23T10:28:01.414 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:28:01.404+0000 7f1fd2606740 1 -- --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_getmap magic: 0 -- 0x5588bb3e1930 con 0x5588bb48b800 2024-04-23T10:28:01.414 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:28:01.404+0000 7f1fbe7fc640 1 -- 172.21.15.114:0/336668180 <== mon.0 v2:172.21.15.114:3300/0 5 ==== osd_map(92..92 src has 1..92) ==== 8585+0+0 (secure 0 0 0) 0x7f1fc00d0f30 con 0x7f1fc810c520 2024-04-23T10:28:01.414 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:28:01.404+0000 7f1fcd68b640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x5588bb480620 0x5588bb480a00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T10:28:01.414 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:28:01.405+0000 7f1fcd68b640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x5588bb480620 0x5588bb480a00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.114:51808/0 (socket says 172.21.15.114:51808) 2024-04-23T10:28:01.414 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:28:01.405+0000 7f1fcd68b640 1 -- 172.21.15.114:0/686608778 learned_addr learned my addr 172.21.15.114:0/686608778 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-04-23T10:28:01.414 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:28:01.405+0000 7f1fcce8a640 1 --2- 172.21.15.114:0/686608778 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x5588bb480f40 0x5588bb48b2c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T10:28:01.414 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:28:01.405+0000 7f1fcde8c640 1 --2- 172.21.15.114:0/686608778 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x5588bb48b800 0x5588bb48dbf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T10:28:01.415 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:28:01.405+0000 7f1fcd68b640 1 -- 172.21.15.114:0/686608778 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x5588bb48b800 msgr2=0x5588bb48dbf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T10:28:01.415 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:28:01.405+0000 7f1fcd68b640 1 --2- 172.21.15.114:0/686608778 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x5588bb48b800 0x5588bb48dbf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:28:01.415 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:28:01.405+0000 7f1fcd68b640 1 -- 172.21.15.114:0/686608778 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x5588bb480f40 msgr2=0x5588bb48b2c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T10:28:01.415 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:28:01.405+0000 7f1fcd68b640 1 --2- 172.21.15.114:0/686608778 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x5588bb480f40 0x5588bb48b2c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:28:01.415 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:28:01.405+0000 7f1fcd68b640 1 -- 172.21.15.114:0/686608778 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5588bb0f8210 con 0x5588bb480620 2024-04-23T10:28:01.415 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:28:01.405+0000 7f1fcce8a640 1 --2- 172.21.15.114:0/336668180 >> [v2:172.21.15.114:6800/1395693466,v1:172.21.15.114:6801/1395693466] conn(0x7f1fb40a3ee0 0x7f1fb40a63a0 secure :-1 s=READY pgs=75 cs=0 l=1 rev1=1 crypto rx=0x7f1fb802aa50 tx=0x7f1fb8030770 comp rx=0 tx=0).ready entity=mgr.15291 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T10:28:01.415 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:28:01.405+0000 7f1fcd68b640 1 --2- 172.21.15.114:0/686608778 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x5588bb480620 0x5588bb480a00 secure :-1 s=READY pgs=544 cs=0 l=1 rev1=1 crypto rx=0x7f1fc40114f0 tx=0x7f1fc40119b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=36adfd4056e3f188 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T10:28:01.415 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:28:01.405+0000 7f1fb37fe640 1 -- 172.21.15.114:0/686608778 <== mon.0 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1fc401e070 con 0x5588bb480620 2024-04-23T10:28:01.415 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:28:01.405+0000 7f1fb37fe640 1 -- 172.21.15.114:0/686608778 <== mon.0 v2:172.21.15.114:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f1fc400fa50 con 0x5588bb480620 2024-04-23T10:28:01.415 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:28:01.405+0000 7f1fb37fe640 1 -- 172.21.15.114:0/686608778 <== mon.0 v2:172.21.15.114:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1fc4019a40 con 0x5588bb480620 2024-04-23T10:28:01.415 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:28:01.406+0000 7f1fd2606740 1 -- 172.21.15.114:0/686608778 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x5588bb480620 msgr2=0x5588bb480a00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T10:28:01.416 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:28:01.406+0000 7f1fd2606740 1 --2- 172.21.15.114:0/686608778 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x5588bb480620 0x5588bb480a00 secure :-1 s=READY pgs=544 cs=0 l=1 rev1=1 crypto rx=0x7f1fc40114f0 tx=0x7f1fc40119b0 comp rx=0 tx=0).stop 2024-04-23T10:28:01.416 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:28:01.406+0000 7f1fd2606740 1 -- 172.21.15.114:0/686608778 shutdown_connections 2024-04-23T10:28:01.416 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:28:01.406+0000 7f1fd2606740 1 --2- 172.21.15.114:0/686608778 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x5588bb48b800 0x5588bb48dbf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:28:01.416 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:28:01.406+0000 7f1fd2606740 1 --2- 172.21.15.114:0/686608778 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x5588bb480f40 0x5588bb48b2c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:28:01.416 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:28:01.406+0000 7f1fd2606740 1 --2- 172.21.15.114:0/686608778 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x5588bb480620 0x5588bb480a00 unknown :-1 s=CLOSED pgs=544 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:28:01.416 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:28:01.406+0000 7f1fd2606740 1 -- 172.21.15.114:0/686608778 >> 172.21.15.114:0/686608778 conn(0x5588bb47cb30 msgr2=0x5588bb47cf10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T10:28:01.416 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:28:01.406+0000 7f1fd2606740 1 -- 172.21.15.114:0/686608778 shutdown_connections 2024-04-23T10:28:01.416 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:28:01.406+0000 7f1fd2606740 1 -- 172.21.15.114:0/686608778 wait complete. 2024-04-23T10:28:01.416 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:28:01.406+0000 7f1fd2606740 1 -- start start 2024-04-23T10:28:01.416 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:28:01.406+0000 7f1fd2606740 1 Processor -- start 2024-04-23T10:28:01.416 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:28:01.407+0000 7f1fd2606740 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x5588bb480620 0x5588bb48b130 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T10:28:01.417 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:28:01.407+0000 7f1fd2606740 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x5588bb480f40 0x5588bb488420 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T10:28:01.417 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:28:01.407+0000 7f1fd2606740 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x5588bb48b800 0x5588bb488960 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T10:28:01.417 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:28:01.407+0000 7f1fd2606740 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x5588bb48e430 con 0x5588bb48b800 2024-04-23T10:28:01.417 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:28:01.407+0000 7f1fd2606740 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 -- 0x5588bb3800b0 con 0x5588bb480f40 2024-04-23T10:28:01.417 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:28:01.407+0000 7f1fd2606740 1 -- --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_getmap magic: 0 -- 0x5588bb48e5b0 con 0x5588bb480620 2024-04-23T10:28:01.417 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:28:01.407+0000 7f1fd2606740 10 client.0 _ll_register_callbacks cb 0x5588bb36dd30 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-23T10:28:01.417 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:28:01.407+0000 7f1fcde8c640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x5588bb48b800 0x5588bb488960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T10:28:01.417 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:28:01.407+0000 7f1fcce8a640 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x5588bb480f40 0x5588bb488420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T10:28:01.417 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:28:01.407+0000 7f1fcde8c640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x5588bb48b800 0x5588bb488960 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.114:51816/0 (socket says 172.21.15.114:51816) 2024-04-23T10:28:01.417 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:28:01.407+0000 7f1fcde8c640 1 -- 172.21.15.114:0/2802977605 learned_addr learned my addr 172.21.15.114:0/2802977605 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-04-23T10:28:01.417 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:28:01.407+0000 7f1fcd68b640 1 --2- 172.21.15.114:0/2802977605 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x5588bb480620 0x5588bb48b130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T10:28:01.418 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:28:01.407+0000 7f1fcde8c640 1 -- 172.21.15.114:0/2802977605 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x5588bb480620 msgr2=0x5588bb48b130 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T10:28:01.418 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:28:01.407+0000 7f1fcde8c640 1 --2- 172.21.15.114:0/2802977605 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x5588bb480620 0x5588bb48b130 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:28:01.418 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:28:01.407+0000 7f1fcde8c640 1 -- 172.21.15.114:0/2802977605 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x5588bb480f40 msgr2=0x5588bb488420 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T10:28:01.418 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:28:01.407+0000 7f1fcde8c640 1 --2- 172.21.15.114:0/2802977605 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x5588bb480f40 0x5588bb488420 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:28:01.418 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:28:01.407+0000 7f1fcde8c640 1 -- 172.21.15.114:0/2802977605 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5588bb1c4620 con 0x5588bb48b800 2024-04-23T10:28:01.418 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:28:01.407+0000 7f1fcd68b640 1 --2- 172.21.15.114:0/2802977605 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x5588bb480620 0x5588bb48b130 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-23T10:28:01.418 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:28:01.407+0000 7f1fcce8a640 1 --2- 172.21.15.114:0/2802977605 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x5588bb480f40 0x5588bb488420 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-23T10:28:01.418 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:28:01.407+0000 7f1fcde8c640 1 --2- 172.21.15.114:0/2802977605 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x5588bb48b800 0x5588bb488960 secure :-1 s=READY pgs=545 cs=0 l=1 rev1=1 crypto rx=0x7f1fc000c6a0 tx=0x7f1fc009c680 comp rx=0 tx=0).ready entity=mon.0 client_cookie=66468c5a2a38512b server_cookie=0 in_seq=0 out_seq=0 2024-04-23T10:28:01.418 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:28:01.407+0000 7f1fb0ff9640 10 client.0 ms_handle_connect on v2:172.21.15.114:3300/0 2024-04-23T10:28:01.418 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:28:01.407+0000 7f1fb0ff9640 1 -- 172.21.15.114:0/2802977605 <== mon.0 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1fc00ce980 con 0x5588bb48b800 2024-04-23T10:28:01.418 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:28:01.407+0000 7f1fb0ff9640 1 -- 172.21.15.114:0/2802977605 <== mon.0 v2:172.21.15.114:3300/0 2 ==== osd_map(92..92 src has 1..92) ==== 8585+0+0 (secure 0 0 0) 0x7f1fc009c070 con 0x5588bb48b800 2024-04-23T10:28:01.419 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:28:01.407+0000 7f1fd2606740 1 -- 172.21.15.114:0/2802977605 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f1fc400fa50 con 0x5588bb48b800 2024-04-23T10:28:01.419 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:28:01.407+0000 7f1fb0ff9640 1 -- 172.21.15.114:0/2802977605 <== mon.0 v2:172.21.15.114:3300/0 3 ==== mon_get_version_reply(handle=1 version=21) ==== 24+0+0 (secure 0 0 0) 0x7f1fc009d320 con 0x5588bb48b800 2024-04-23T10:28:01.419 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:28:01.407+0000 7f1fd2606740 10 client.15756 fetch_fsmap learned FSMap version 21 2024-04-23T10:28:01.419 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:28:01.408+0000 7f1fd2606740 1 -- 172.21.15.114:0/2802977605 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({config=0+,fsmap=21,monmap=4+}) -- 0x5588bb49a340 con 0x5588bb48b800 2024-04-23T10:28:01.419 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:28:01.408+0000 7f1fb0ff9640 1 -- 172.21.15.114:0/2802977605 <== mon.0 v2:172.21.15.114:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f1fc400fa50 con 0x5588bb48b800 2024-04-23T10:28:01.419 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:28:01.408+0000 7f1fb0ff9640 1 -- 172.21.15.114:0/2802977605 <== mon.0 v2:172.21.15.114:3300/0 5 ==== fsmap(e 21) ==== 8338+0+0 (secure 0 0 0) 0x7f1fc009bae0 con 0x5588bb48b800 2024-04-23T10:28:01.419 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:28:01.408+0000 7f1fd2606740 10 client.15756 fetch_fsmap finished waiting for FSMap version 21 2024-04-23T10:28:01.419 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:28:01.408+0000 7f1fd2606740 10 client.15756 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-04-23T10:28:01.419 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:28:01.408+0000 7f1fd2606740 20 client.15756 populate_metadata read hostname 'smithi114' 2024-04-23T10:28:01.419 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:28:01.408+0000 7f1fd2606740 1 --2- 172.21.15.114:0/2802977605 >> [v2:172.21.15.119:6838/937297340,v1:172.21.15.119:6839/937297340] conn(0x5588bb49b800 0x5588bb49dc10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T10:28:01.419 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:28:01.408+0000 7f1fd2606740 4 client.15756 mds_command: new command op to 24545 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-23T10:28:01.419 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:28:01.408+0000 7f1fd2606740 1 -- 172.21.15.114:0/2802977605 --> [v2:172.21.15.119:6838/937297340,v1:172.21.15.119:6839/937297340] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x5588bb3f3390 con 0x5588bb49b800 2024-04-23T10:28:01.420 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:28:01.408+0000 7f1fcd68b640 1 --2- 172.21.15.114:0/2802977605 >> [v2:172.21.15.119:6838/937297340,v1:172.21.15.119:6839/937297340] conn(0x5588bb49b800 0x5588bb49dc10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T10:28:01.420 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:28:01.408+0000 7f1fcd68b640 1 --2- 172.21.15.114:0/2802977605 >> [v2:172.21.15.119:6838/937297340,v1:172.21.15.119:6839/937297340] conn(0x5588bb49b800 0x5588bb49dc10 crc :-1 s=READY pgs=58 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=a10f994809f4d936 server_cookie=c113a9035f2eba67 in_seq=0 out_seq=0 2024-04-23T10:28:01.420 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:28:01.408+0000 7f1fb0ff9640 10 client.15756 ms_handle_connect on v2:172.21.15.119:6838/937297340 2024-04-23T10:28:01.420 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:28:01.409+0000 7f1fb0ff9640 1 -- 172.21.15.114:0/2802977605 <== mds.0 v2:172.21.15.119:6838/937297340 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x5588bb3f3390 con 0x5588bb49b800 2024-04-23T10:28:01.420 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:28:01.409+0000 7f1fb0ff9640 10 client.15756 handle_command_reply: tid=0 2024-04-23T10:28:01.420 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:28:01.409+0000 7f1fd2606740 1 client.15756 shutdown 2024-04-23T10:28:01.420 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:28:01.409+0000 7f1fd2606740 1 -- 172.21.15.114:0/2802977605 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x5588bb48b800 msgr2=0x5588bb488960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T10:28:01.420 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:28:01.409+0000 7f1fd2606740 1 --2- 172.21.15.114:0/2802977605 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x5588bb48b800 0x5588bb488960 secure :-1 s=READY pgs=545 cs=0 l=1 rev1=1 crypto rx=0x7f1fc000c6a0 tx=0x7f1fc009c680 comp rx=0 tx=0).stop 2024-04-23T10:28:01.420 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:28:01.409+0000 7f1fd2606740 1 -- 172.21.15.114:0/2802977605 shutdown_connections 2024-04-23T10:28:01.420 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:28:01.409+0000 7f1fd2606740 1 --2- 172.21.15.114:0/2802977605 >> [v2:172.21.15.119:6838/937297340,v1:172.21.15.119:6839/937297340] conn(0x5588bb49b800 0x5588bb49dc10 crc :-1 s=READY pgs=58 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:28:01.420 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:28:01.409+0000 7f1fd2606740 1 --2- 172.21.15.114:0/2802977605 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x5588bb48b800 0x5588bb488960 unknown :-1 s=CLOSED pgs=545 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:28:01.420 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:28:01.409+0000 7f1fd2606740 1 --2- 172.21.15.114:0/2802977605 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x5588bb480f40 0x5588bb488420 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:28:01.421 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:28:01.409+0000 7f1fb0ff9640 0 client.15756 ms_handle_reset on v2:172.21.15.119:6838/937297340 2024-04-23T10:28:01.421 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:28:01.409+0000 7f1fd2606740 1 --2- 172.21.15.114:0/2802977605 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x5588bb480620 0x5588bb48b130 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:28:01.421 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:28:01.409+0000 7f1fd2606740 1 -- 172.21.15.114:0/2802977605 >> 172.21.15.114:0/2802977605 conn(0x5588bb47cb30 msgr2=0x5588bb4828b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T10:28:01.421 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:28:01.409+0000 7f1fd2606740 1 -- 172.21.15.114:0/2802977605 shutdown_connections 2024-04-23T10:28:01.421 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:28:01.409+0000 7f1fd2606740 1 -- 172.21.15.114:0/2802977605 wait complete. 2024-04-23T10:28:01.421 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:28:01.409+0000 7f1fd2606740 20 client.15756 trim_cache size 0 max 16384 2024-04-23T10:28:01.429 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:28:01.425+0000 7f1fd2606740 1 Processor -- start 2024-04-23T10:28:01.429 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:28:01.425+0000 7f1fd2606740 1 -- start start 2024-04-23T10:28:01.430 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:28:01.425+0000 7f1fd2606740 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x5588bb480620 0x5588bb4972f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T10:28:01.430 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:28:01.425+0000 7f1fd2606740 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x5588bb480f40 0x5588bb477c50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T10:28:01.430 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:28:01.425+0000 7f1fd2606740 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x5588bb48b800 0x5588bb48ac60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T10:28:01.430 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:28:01.425+0000 7f1fd2606740 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x7f1fc81b0650 con 0x5588bb480620 2024-04-23T10:28:01.430 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:28:01.425+0000 7f1fd2606740 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 -- 0x7f1fc81adf20 con 0x5588bb48b800 2024-04-23T10:28:01.430 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:28:01.425+0000 7f1fd2606740 1 -- --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_getmap magic: 0 -- 0x7f1fc81abd70 con 0x5588bb480f40 2024-04-23T10:28:01.430 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:28:01.425+0000 7f1fcce8a640 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x5588bb480f40 0x5588bb477c50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T10:28:01.430 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:28:01.425+0000 7f1fcd68b640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x5588bb480620 0x5588bb4972f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T10:28:01.430 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:28:01.425+0000 7f1fcce8a640 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x5588bb480f40 0x5588bb477c50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.204:3300/0 says I am v2:172.21.15.114:60512/0 (socket says 172.21.15.114:60512) 2024-04-23T10:28:01.430 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:28:01.425+0000 7f1fcce8a640 1 -- 172.21.15.114:0/2141916833 learned_addr learned my addr 172.21.15.114:0/2141916833 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-04-23T10:28:01.430 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:28:01.425+0000 7f1fcde8c640 1 --2- 172.21.15.114:0/2141916833 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x5588bb48b800 0x5588bb48ac60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T10:28:01.430 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:28:01.425+0000 7f1fcd68b640 1 -- 172.21.15.114:0/2141916833 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x5588bb480f40 msgr2=0x5588bb477c50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T10:28:01.431 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:28:01.425+0000 7f1fcd68b640 1 --2- 172.21.15.114:0/2141916833 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x5588bb480f40 0x5588bb477c50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:28:01.431 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:28:01.425+0000 7f1fcd68b640 1 -- 172.21.15.114:0/2141916833 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x5588bb48b800 msgr2=0x5588bb48ac60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T10:28:01.431 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:28:01.425+0000 7f1fcd68b640 1 --2- 172.21.15.114:0/2141916833 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x5588bb48b800 0x5588bb48ac60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:28:01.431 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:28:01.425+0000 7f1fcd68b640 1 -- 172.21.15.114:0/2141916833 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5588bb488cc0 con 0x5588bb480620 2024-04-23T10:28:01.431 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:28:01.425+0000 7f1fcce8a640 1 --2- 172.21.15.114:0/2141916833 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x5588bb480f40 0x5588bb477c50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-23T10:28:01.431 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:28:01.425+0000 7f1fcd68b640 1 --2- 172.21.15.114:0/2141916833 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x5588bb480620 0x5588bb4972f0 secure :-1 s=READY pgs=546 cs=0 l=1 rev1=1 crypto rx=0x7f1fc40114c0 tx=0x7f1fc4019530 comp rx=0 tx=0).ready entity=mon.0 client_cookie=67c684eb15685f9f server_cookie=0 in_seq=0 out_seq=0 2024-04-23T10:28:01.431 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:28:01.426+0000 7f1fb2ffd640 1 -- 172.21.15.114:0/2141916833 <== mon.0 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1fc401e040 con 0x5588bb480620 2024-04-23T10:28:01.431 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:28:01.426+0000 7f1fb2ffd640 1 -- 172.21.15.114:0/2141916833 <== mon.0 v2:172.21.15.114:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f1fc4029e50 con 0x5588bb480620 2024-04-23T10:28:01.431 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:28:01.426+0000 7f1fb2ffd640 1 -- 172.21.15.114:0/2141916833 <== mon.0 v2:172.21.15.114:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1fc4019670 con 0x5588bb480620 2024-04-23T10:28:01.431 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:28:01.426+0000 7f1fd2606740 1 -- 172.21.15.114:0/2141916833 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x5588bb480620 msgr2=0x5588bb4972f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T10:28:01.431 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:28:01.426+0000 7f1fd2606740 1 --2- 172.21.15.114:0/2141916833 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x5588bb480620 0x5588bb4972f0 secure :-1 s=READY pgs=546 cs=0 l=1 rev1=1 crypto rx=0x7f1fc40114c0 tx=0x7f1fc4019530 comp rx=0 tx=0).stop 2024-04-23T10:28:01.432 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:28:01.426+0000 7f1fd2606740 1 -- 172.21.15.114:0/2141916833 shutdown_connections 2024-04-23T10:28:01.432 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:28:01.426+0000 7f1fd2606740 1 --2- 172.21.15.114:0/2141916833 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x5588bb48b800 0x5588bb48ac60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:28:01.432 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:28:01.426+0000 7f1fd2606740 1 --2- 172.21.15.114:0/2141916833 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x5588bb480f40 0x5588bb477c50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:28:01.432 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:28:01.426+0000 7f1fd2606740 1 --2- 172.21.15.114:0/2141916833 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x5588bb480620 0x5588bb4972f0 unknown :-1 s=CLOSED pgs=546 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:28:01.432 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:28:01.426+0000 7f1fd2606740 1 -- 172.21.15.114:0/2141916833 >> 172.21.15.114:0/2141916833 conn(0x5588bb47cb30 msgr2=0x5588bb483af0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T10:28:01.432 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:28:01.426+0000 7f1fd2606740 1 -- 172.21.15.114:0/2141916833 shutdown_connections 2024-04-23T10:28:01.432 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:28:01.426+0000 7f1fd2606740 1 -- 172.21.15.114:0/2141916833 wait complete. 2024-04-23T10:28:01.432 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:28:01.427+0000 7f1fd2606740 1 -- start start 2024-04-23T10:28:01.432 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:28:01.427+0000 7f1fd2606740 1 Processor -- start 2024-04-23T10:28:01.432 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:28:01.427+0000 7f1fd2606740 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x5588bb480620 0x5588bb48e9b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T10:28:01.432 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:28:01.427+0000 7f1fd2606740 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x5588bb480f40 0x5588bb48eef0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T10:28:01.432 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:28:01.427+0000 7f1fd2606740 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x5588bb48b800 0x5588bb4ffdd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T10:28:01.432 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:28:01.427+0000 7f1fd2606740 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x7f1fc81ac3e0 con 0x5588bb48b800 2024-04-23T10:28:01.433 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:28:01.427+0000 7f1fd2606740 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 -- 0x7f1fc81ad850 con 0x5588bb480620 2024-04-23T10:28:01.433 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:28:01.427+0000 7f1fd2606740 1 -- --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_getmap magic: 0 -- 0x7f1fc81acab0 con 0x5588bb480f40 2024-04-23T10:28:01.433 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:28:01.427+0000 7f1fd2606740 10 client.0 _ll_register_callbacks cb 0x7f1fc009c760 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-23T10:28:01.433 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:28:01.427+0000 7f1fcde8c640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x5588bb48b800 0x5588bb4ffdd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T10:28:01.433 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:28:01.427+0000 7f1fcd68b640 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x5588bb480620 0x5588bb48e9b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T10:28:01.433 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:28:01.427+0000 7f1fcce8a640 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x5588bb480f40 0x5588bb48eef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T10:28:01.433 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:28:01.427+0000 7f1fcd68b640 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x5588bb480620 0x5588bb48e9b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.119:3300/0 says I am v2:172.21.15.114:54722/0 (socket says 172.21.15.114:54722) 2024-04-23T10:28:01.433 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:28:01.427+0000 7f1fcd68b640 1 -- 172.21.15.114:0/863871770 learned_addr learned my addr 172.21.15.114:0/863871770 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-04-23T10:28:01.433 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:28:01.427+0000 7f1fcde8c640 1 --2- 172.21.15.114:0/863871770 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x5588bb48b800 0x5588bb4ffdd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.114:51832/0 (socket says 172.21.15.114:51832) 2024-04-23T10:28:01.433 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:28:01.428+0000 7f1fcd68b640 1 -- 172.21.15.114:0/863871770 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x5588bb480f40 msgr2=0x5588bb48eef0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T10:28:01.433 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:28:01.428+0000 7f1fcd68b640 1 --2- 172.21.15.114:0/863871770 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x5588bb480f40 0x5588bb48eef0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:28:01.434 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:28:01.428+0000 7f1fcd68b640 1 -- 172.21.15.114:0/863871770 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x5588bb48b800 msgr2=0x5588bb4ffdd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T10:28:01.434 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:28:01.428+0000 7f1fcd68b640 1 --2- 172.21.15.114:0/863871770 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x5588bb48b800 0x5588bb4ffdd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:28:01.434 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:28:01.428+0000 7f1fcd68b640 1 -- 172.21.15.114:0/863871770 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5588bb500310 con 0x5588bb480620 2024-04-23T10:28:01.434 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:28:01.428+0000 7f1fcd68b640 1 --2- 172.21.15.114:0/863871770 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x5588bb480620 0x5588bb48e9b0 secure :-1 s=READY pgs=64 cs=0 l=1 rev1=1 crypto rx=0x7f1fc4005cf0 tx=0x7f1fc4005d20 comp rx=0 tx=0).ready entity=mon.2 client_cookie=8e941449fd32192f server_cookie=0 in_seq=0 out_seq=0 2024-04-23T10:28:01.434 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:28:01.428+0000 7f1fb0ff9640 10 client.0 ms_handle_connect on v2:172.21.15.119:3300/0 2024-04-23T10:28:01.434 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:28:01.428+0000 7f1fb0ff9640 1 -- 172.21.15.114:0/863871770 <== mon.2 v2:172.21.15.119:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1fc401e070 con 0x5588bb480620 2024-04-23T10:28:01.434 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:28:01.428+0000 7f1fb0ff9640 1 -- 172.21.15.114:0/863871770 <== mon.2 v2:172.21.15.119:3300/0 2 ==== osd_map(92..92 src has 1..92) ==== 8585+0+0 (secure 0 0 0) 0x7f1fc4003970 con 0x5588bb480620 2024-04-23T10:28:01.434 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:28:01.428+0000 7f1fd2606740 1 -- 172.21.15.114:0/863871770 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f1fc4029e50 con 0x5588bb480620 2024-04-23T10:28:01.434 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:28:01.428+0000 7f1fb0ff9640 1 -- 172.21.15.114:0/863871770 <== mon.2 v2:172.21.15.119:3300/0 3 ==== mon_get_version_reply(handle=1 version=21) ==== 24+0+0 (secure 0 0 0) 0x7f1fc403f050 con 0x5588bb480620 2024-04-23T10:28:01.434 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:28:01.428+0000 7f1fd2606740 10 client.25286 fetch_fsmap learned FSMap version 21 2024-04-23T10:28:01.434 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:28:01.428+0000 7f1fd2606740 1 -- 172.21.15.114:0/863871770 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({config=0+,fsmap=21,monmap=4+}) -- 0x5588bb501590 con 0x5588bb480620 2024-04-23T10:28:01.434 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:28:01.428+0000 7f1fb0ff9640 1 -- 172.21.15.114:0/863871770 <== mon.2 v2:172.21.15.119:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f1fc4029e50 con 0x5588bb480620 2024-04-23T10:28:01.435 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:28:01.428+0000 7f1fb0ff9640 1 -- 172.21.15.114:0/863871770 <== mon.2 v2:172.21.15.119:3300/0 5 ==== fsmap(e 21) ==== 8338+0+0 (secure 0 0 0) 0x7f1fc4019a50 con 0x5588bb480620 2024-04-23T10:28:01.435 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:28:01.428+0000 7f1fd2606740 10 client.25286 fetch_fsmap finished waiting for FSMap version 21 2024-04-23T10:28:01.435 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:28:01.429+0000 7f1fd2606740 10 client.25286 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-04-23T10:28:01.435 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:28:01.429+0000 7f1fd2606740 20 client.25286 populate_metadata read hostname 'smithi114' 2024-04-23T10:28:01.435 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:28:01.429+0000 7f1fd2606740 1 --2- 172.21.15.114:0/863871770 >> [v2:172.21.15.119:6838/937297340,v1:172.21.15.119:6839/937297340] conn(0x5588bb501aa0 0x5588bb503e90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T10:28:01.435 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:28:01.429+0000 7f1fd2606740 4 client.25286 mds_command: new command op to 24545 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-04-23T10:28:01.435 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:28:01.429+0000 7f1fd2606740 1 -- 172.21.15.114:0/863871770 --> [v2:172.21.15.119:6838/937297340,v1:172.21.15.119:6839/937297340] -- command(tid 0: {"prefix": "flush journal"}) -- 0x5588bb3ed200 con 0x5588bb501aa0 2024-04-23T10:28:01.435 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:28:01.429+0000 7f1fcce8a640 1 --2- 172.21.15.114:0/863871770 >> [v2:172.21.15.119:6838/937297340,v1:172.21.15.119:6839/937297340] conn(0x5588bb501aa0 0x5588bb503e90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T10:28:01.435 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:28:01.429+0000 7f1fb0ff9640 10 client.25286 ms_handle_connect on v2:172.21.15.119:6838/937297340 2024-04-23T10:28:01.435 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:28:01.429+0000 7f1fcce8a640 1 --2- 172.21.15.114:0/863871770 >> [v2:172.21.15.119:6838/937297340,v1:172.21.15.119:6839/937297340] conn(0x5588bb501aa0 0x5588bb503e90 crc :-1 s=READY pgs=59 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=dae12d707f5c6d5 server_cookie=85d7755ff15a6295 in_seq=0 out_seq=0 2024-04-23T10:28:01.534 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:28:01.534+0000 7f1fb0ff9640 1 -- 172.21.15.114:0/863871770 <== mds.0 v2:172.21.15.119:6838/937297340 1 ==== command_reply(tid 0: 0 ) ==== 8+0+44 (crc 0 0 0) 0x5588bb3ed200 con 0x5588bb501aa0 2024-04-23T10:28:01.534 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:28:01.534+0000 7f1fb0ff9640 10 client.25286 handle_command_reply: tid=0 2024-04-23T10:28:01.534 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:28:01.534+0000 7f1fd2606740 1 client.25286 shutdown 2024-04-23T10:28:01.534 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:28:01.534+0000 7f1fd2606740 1 -- 172.21.15.114:0/863871770 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x5588bb480620 msgr2=0x5588bb48e9b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T10:28:01.534 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:28:01.534+0000 7f1fd2606740 1 --2- 172.21.15.114:0/863871770 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x5588bb480620 0x5588bb48e9b0 secure :-1 s=READY pgs=64 cs=0 l=1 rev1=1 crypto rx=0x7f1fc4005cf0 tx=0x7f1fc4005d20 comp rx=0 tx=0).stop 2024-04-23T10:28:01.534 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:28:01.534+0000 7f1fd2606740 1 -- 172.21.15.114:0/863871770 shutdown_connections 2024-04-23T10:28:01.534 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:28:01.534+0000 7f1fd2606740 1 --2- 172.21.15.114:0/863871770 >> [v2:172.21.15.119:6838/937297340,v1:172.21.15.119:6839/937297340] conn(0x5588bb501aa0 0x5588bb503e90 crc :-1 s=READY pgs=59 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:28:01.534 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:28:01.534+0000 7f1fd2606740 1 --2- 172.21.15.114:0/863871770 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x5588bb48b800 0x5588bb4ffdd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:28:01.534 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:28:01.534+0000 7f1fb0ff9640 0 client.25286 ms_handle_reset on v2:172.21.15.119:6838/937297340 2024-04-23T10:28:01.534 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:28:01.534+0000 7f1fd2606740 1 --2- 172.21.15.114:0/863871770 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x5588bb480f40 0x5588bb48eef0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:28:01.535 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:28:01.534+0000 7f1fd2606740 1 --2- 172.21.15.114:0/863871770 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x5588bb480620 0x5588bb48e9b0 unknown :-1 s=CLOSED pgs=64 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:28:01.535 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:28:01.534+0000 7f1fd2606740 1 -- 172.21.15.114:0/863871770 >> 172.21.15.114:0/863871770 conn(0x5588bb47cb30 msgr2=0x5588bb488510 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T10:28:01.535 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:28:01.534+0000 7f1fd2606740 1 -- 172.21.15.114:0/863871770 shutdown_connections 2024-04-23T10:28:01.535 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:28:01.534+0000 7f1fd2606740 1 -- 172.21.15.114:0/863871770 wait complete. 2024-04-23T10:28:01.535 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:28:01.534+0000 7f1fd2606740 20 client.25286 trim_cache size 0 max 16384 2024-04-23T10:28:01.535 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stdout:{ 2024-04-23T10:28:01.535 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stdout: "message": "", 2024-04-23T10:28:01.535 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stdout: "return_code": 0 2024-04-23T10:28:01.536 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stdout:} 2024-04-23T10:28:01.536 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:28:01.534+0000 7f1fcf8ef640 1 -- 172.21.15.114:0/336668180 >> [v2:172.21.15.114:6800/1395693466,v1:172.21.15.114:6801/1395693466] conn(0x7f1fb40a3ee0 msgr2=0x7f1fb40a63a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T10:28:01.537 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:28:01.534+0000 7f1fcf8ef640 1 --2- 172.21.15.114:0/336668180 >> [v2:172.21.15.114:6800/1395693466,v1:172.21.15.114:6801/1395693466] conn(0x7f1fb40a3ee0 0x7f1fb40a63a0 secure :-1 s=READY pgs=75 cs=0 l=1 rev1=1 crypto rx=0x7f1fb802aa50 tx=0x7f1fb8030770 comp rx=0 tx=0).stop 2024-04-23T10:28:01.537 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:28:01.534+0000 7f1fcf8ef640 1 -- 172.21.15.114:0/336668180 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f1fc810c520 msgr2=0x7f1fc819d020 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T10:28:01.537 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:28:01.534+0000 7f1fcf8ef640 1 --2- 172.21.15.114:0/336668180 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f1fc810c520 0x7f1fc819d020 secure :-1 s=READY pgs=543 cs=0 l=1 rev1=1 crypto rx=0x7f1fc000e970 tx=0x7f1fc000ee30 comp rx=0 tx=0).stop 2024-04-23T10:28:01.537 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:28:01.535+0000 7f1fcf8ef640 1 -- 172.21.15.114:0/336668180 shutdown_connections 2024-04-23T10:28:01.537 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:28:01.535+0000 7f1fcf8ef640 1 --2- 172.21.15.114:0/336668180 >> [v2:172.21.15.114:6800/1395693466,v1:172.21.15.114:6801/1395693466] conn(0x7f1fb40a3ee0 0x7f1fb40a63a0 unknown :-1 s=CLOSED pgs=75 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:28:01.537 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:28:01.535+0000 7f1fcf8ef640 1 --2- 172.21.15.114:0/336668180 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f1fc810c520 0x7f1fc819d020 unknown :-1 s=CLOSED pgs=543 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:28:01.537 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:28:01.535+0000 7f1fcf8ef640 1 --2- 172.21.15.114:0/336668180 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f1fc8103b00 0x7f1fc8198ca0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:28:01.537 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:28:01.535+0000 7f1fcf8ef640 1 --2- 172.21.15.114:0/336668180 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f1fc81031e0 0x7f1fc8198760 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:28:01.537 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:28:01.535+0000 7f1fcf8ef640 1 -- 172.21.15.114:0/336668180 >> 172.21.15.114:0/336668180 conn(0x7f1fc8077fd0 msgr2=0x7f1fc80ff110 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T10:28:01.537 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:28:01.535+0000 7f1fcf8ef640 1 -- 172.21.15.114:0/336668180 shutdown_connections 2024-04-23T10:28:01.538 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:28:01.535+0000 7f1fcf8ef640 1 -- 172.21.15.114:0/336668180 wait complete. 2024-04-23T10:28:01.907 INFO:tasks.workunit.client.0.smithi114.stdout: 1 27853 16.74 MB/sec warmup 10 sec latency 55.226 ms 2024-04-23T10:28:02.492 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 23 10:28:02 smithi119 ceph-mon[32608]: pgmap v45: 97 pgs: 97 active+clean; 7.8 MiB data, 414 MiB used, 1.0 TiB / 1.0 TiB avail; 263 KiB/s rd, 660 KiB/s wr, 80 op/s 2024-04-23T10:28:02.504 INFO:journalctl@ceph.mon.a.smithi114.stdout:Apr 23 10:28:02 smithi114 ceph-mon[29255]: pgmap v45: 97 pgs: 97 active+clean; 7.8 MiB data, 414 MiB used, 1.0 TiB / 1.0 TiB avail; 263 KiB/s rd, 660 KiB/s wr, 80 op/s 2024-04-23T10:28:02.578 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 23 10:28:02 smithi204 ceph-mon[32649]: pgmap v45: 97 pgs: 97 active+clean; 7.8 MiB data, 414 MiB used, 1.0 TiB / 1.0 TiB avail; 263 KiB/s rd, 660 KiB/s wr, 80 op/s 2024-04-23T10:28:02.907 INFO:tasks.workunit.client.0.smithi114.stdout: 1 30509 16.33 MB/sec warmup 11 sec latency 21.004 ms 2024-04-23T10:28:03.907 INFO:tasks.workunit.client.0.smithi114.stdout: 1 33171 16.27 MB/sec warmup 12 sec latency 15.447 ms 2024-04-23T10:28:04.492 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 23 10:28:04 smithi119 ceph-mon[32608]: pgmap v46: 97 pgs: 97 active+clean; 15 MiB data, 443 MiB used, 1.0 TiB / 1.0 TiB avail; 647 KiB/s rd, 1.4 MiB/s wr, 193 op/s 2024-04-23T10:28:04.504 INFO:journalctl@ceph.mon.a.smithi114.stdout:Apr 23 10:28:04 smithi114 ceph-mon[29255]: pgmap v46: 97 pgs: 97 active+clean; 15 MiB data, 443 MiB used, 1.0 TiB / 1.0 TiB avail; 647 KiB/s rd, 1.4 MiB/s wr, 193 op/s 2024-04-23T10:28:04.578 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 23 10:28:04 smithi204 ceph-mon[32649]: pgmap v46: 97 pgs: 97 active+clean; 15 MiB data, 443 MiB used, 1.0 TiB / 1.0 TiB avail; 647 KiB/s rd, 1.4 MiB/s wr, 193 op/s 2024-04-23T10:28:04.907 INFO:tasks.workunit.client.0.smithi114.stdout: 1 35491 16.19 MB/sec warmup 13 sec latency 346.200 ms 2024-04-23T10:28:05.492 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 23 10:28:05 smithi119 ceph-mon[32608]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=e=up:active} 3 up:standby-replay 6 up:standby 2024-04-23T10:28:05.504 INFO:journalctl@ceph.mon.a.smithi114.stdout:Apr 23 10:28:05 smithi114 ceph-mon[29255]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=e=up:active} 3 up:standby-replay 6 up:standby 2024-04-23T10:28:05.577 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 23 10:28:05 smithi204 ceph-mon[32649]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=e=up:active} 3 up:standby-replay 6 up:standby 2024-04-23T10:28:05.907 INFO:tasks.workunit.client.0.smithi114.stdout: 1 35491 15.04 MB/sec warmup 14 sec latency 1346.298 ms 2024-04-23T10:28:06.492 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 23 10:28:06 smithi119 ceph-mon[32608]: pgmap v47: 97 pgs: 97 active+clean; 15 MiB data, 443 MiB used, 1.0 TiB / 1.0 TiB avail; 645 KiB/s rd, 1.4 MiB/s wr, 190 op/s 2024-04-23T10:28:06.504 INFO:journalctl@ceph.mon.a.smithi114.stdout:Apr 23 10:28:06 smithi114 ceph-mon[29255]: pgmap v47: 97 pgs: 97 active+clean; 15 MiB data, 443 MiB used, 1.0 TiB / 1.0 TiB avail; 645 KiB/s rd, 1.4 MiB/s wr, 190 op/s 2024-04-23T10:28:06.577 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 23 10:28:06 smithi204 ceph-mon[32649]: pgmap v47: 97 pgs: 97 active+clean; 15 MiB data, 443 MiB used, 1.0 TiB / 1.0 TiB avail; 645 KiB/s rd, 1.4 MiB/s wr, 190 op/s 2024-04-23T10:28:06.907 INFO:tasks.workunit.client.0.smithi114.stdout: 1 35491 14.03 MB/sec warmup 15 sec latency 2346.400 ms 2024-04-23T10:28:07.504 INFO:journalctl@ceph.mon.a.smithi114.stdout:Apr 23 10:28:07 smithi114 ceph-mon[29255]: pgmap v48: 97 pgs: 97 active+clean; 15 MiB data, 442 MiB used, 1.0 TiB / 1.0 TiB avail; 888 KiB/s rd, 1.8 MiB/s wr, 341 op/s 2024-04-23T10:28:07.578 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 23 10:28:07 smithi204 ceph-mon[32649]: pgmap v48: 97 pgs: 97 active+clean; 15 MiB data, 442 MiB used, 1.0 TiB / 1.0 TiB avail; 888 KiB/s rd, 1.8 MiB/s wr, 341 op/s 2024-04-23T10:28:07.741 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 23 10:28:07 smithi119 ceph-mon[32608]: pgmap v48: 97 pgs: 97 active+clean; 15 MiB data, 442 MiB used, 1.0 TiB / 1.0 TiB avail; 888 KiB/s rd, 1.8 MiB/s wr, 341 op/s 2024-04-23T10:28:07.907 INFO:tasks.workunit.client.0.smithi114.stdout: 1 35491 13.16 MB/sec warmup 16 sec latency 3346.436 ms 2024-04-23T10:28:08.907 INFO:tasks.workunit.client.0.smithi114.stdout: 1 35516 12.39 MB/sec warmup 17 sec latency 3878.512 ms 2024-04-23T10:28:09.741 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 23 10:28:09 smithi119 ceph-mon[32608]: pgmap v49: 97 pgs: 97 active+clean; 15 MiB data, 444 MiB used, 1.0 TiB / 1.0 TiB avail; 675 KiB/s rd, 1.3 MiB/s wr, 281 op/s 2024-04-23T10:28:09.754 INFO:journalctl@ceph.mon.a.smithi114.stdout:Apr 23 10:28:09 smithi114 ceph-mon[29255]: pgmap v49: 97 pgs: 97 active+clean; 15 MiB data, 444 MiB used, 1.0 TiB / 1.0 TiB avail; 675 KiB/s rd, 1.3 MiB/s wr, 281 op/s 2024-04-23T10:28:09.828 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 23 10:28:09 smithi204 ceph-mon[32649]: pgmap v49: 97 pgs: 97 active+clean; 15 MiB data, 444 MiB used, 1.0 TiB / 1.0 TiB avail; 675 KiB/s rd, 1.3 MiB/s wr, 281 op/s 2024-04-23T10:28:09.907 INFO:tasks.workunit.client.0.smithi114.stdout: 1 36334 11.82 MB/sec warmup 18 sec latency 698.005 ms 2024-04-23T10:28:10.907 INFO:tasks.workunit.client.0.smithi114.stdout: 1 39191 12.11 MB/sec warmup 19 sec latency 75.856 ms 2024-04-23T10:28:11.741 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 23 10:28:11 smithi119 ceph-mon[32608]: pgmap v50: 97 pgs: 97 active+clean; 15 MiB data, 444 MiB used, 1.0 TiB / 1.0 TiB avail; 674 KiB/s rd, 1.3 MiB/s wr, 279 op/s 2024-04-23T10:28:11.754 INFO:journalctl@ceph.mon.a.smithi114.stdout:Apr 23 10:28:11 smithi114 ceph-mon[29255]: pgmap v50: 97 pgs: 97 active+clean; 15 MiB data, 444 MiB used, 1.0 TiB / 1.0 TiB avail; 674 KiB/s rd, 1.3 MiB/s wr, 279 op/s 2024-04-23T10:28:11.827 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 23 10:28:11 smithi204 ceph-mon[32649]: pgmap v50: 97 pgs: 97 active+clean; 15 MiB data, 444 MiB used, 1.0 TiB / 1.0 TiB avail; 674 KiB/s rd, 1.3 MiB/s wr, 279 op/s 2024-04-23T10:28:11.907 INFO:tasks.workunit.client.0.smithi114.stdout: 1 41024 11.84 MB/sec warmup 20 sec latency 12.470 ms 2024-04-23T10:28:12.907 INFO:tasks.workunit.client.0.smithi114.stdout: 1 42834 11.91 MB/sec warmup 21 sec latency 178.497 ms 2024-04-23T10:28:13.741 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 23 10:28:13 smithi119 ceph-mon[32608]: pgmap v51: 97 pgs: 97 active+clean; 18 MiB data, 455 MiB used, 1.0 TiB / 1.0 TiB avail; 793 KiB/s rd, 1.5 MiB/s wr, 321 op/s 2024-04-23T10:28:13.742 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 23 10:28:13 smithi119 ceph-mon[32608]: mgrmap e26: x(active, since 92s), standbys: z, y 2024-04-23T10:28:13.753 INFO:journalctl@ceph.mon.a.smithi114.stdout:Apr 23 10:28:13 smithi114 ceph-mon[29255]: pgmap v51: 97 pgs: 97 active+clean; 18 MiB data, 455 MiB used, 1.0 TiB / 1.0 TiB avail; 793 KiB/s rd, 1.5 MiB/s wr, 321 op/s 2024-04-23T10:28:13.754 INFO:journalctl@ceph.mon.a.smithi114.stdout:Apr 23 10:28:13 smithi114 ceph-mon[29255]: mgrmap e26: x(active, since 92s), standbys: z, y 2024-04-23T10:28:13.827 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 23 10:28:13 smithi204 ceph-mon[32649]: pgmap v51: 97 pgs: 97 active+clean; 18 MiB data, 455 MiB used, 1.0 TiB / 1.0 TiB avail; 793 KiB/s rd, 1.5 MiB/s wr, 321 op/s 2024-04-23T10:28:13.827 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 23 10:28:13 smithi204 ceph-mon[32649]: mgrmap e26: x(active, since 92s), standbys: z, y 2024-04-23T10:28:13.907 INFO:tasks.workunit.client.0.smithi114.stdout: 1 44206 11.62 MB/sec warmup 22 sec latency 54.547 ms 2024-04-23T10:28:14.624 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:28:14.623+0000 7fdda6a79640 1 -- 172.21.15.114:0/2758498154 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fdda0101120 msgr2=0x7fdda0101500 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T10:28:14.624 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:28:14.623+0000 7fdda6a79640 1 --2- 172.21.15.114:0/2758498154 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fdda0101120 0x7fdda0101500 secure :-1 s=READY pgs=547 cs=0 l=1 rev1=1 crypto rx=0x7fdd94009900 tx=0x7fdd9402a410 comp rx=0 tx=0).stop 2024-04-23T10:28:14.625 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:28:14.623+0000 7fdda6a79640 1 -- 172.21.15.114:0/2758498154 shutdown_connections 2024-04-23T10:28:14.625 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:28:14.623+0000 7fdda6a79640 1 --2- 172.21.15.114:0/2758498154 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fdda01061a0 0x7fdda010dd70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:28:14.625 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:28:14.623+0000 7fdda6a79640 1 --2- 172.21.15.114:0/2758498154 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fdda0101ad0 0x7fdda0105ac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:28:14.625 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:28:14.623+0000 7fdda6a79640 1 --2- 172.21.15.114:0/2758498154 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fdda0101120 0x7fdda0101500 unknown :-1 s=CLOSED pgs=547 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:28:14.625 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:28:14.623+0000 7fdda6a79640 1 -- 172.21.15.114:0/2758498154 >> 172.21.15.114:0/2758498154 conn(0x7fdda0077fa0 msgr2=0x7fdda00783b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T10:28:14.625 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:28:14.623+0000 7fdda6a79640 1 -- 172.21.15.114:0/2758498154 shutdown_connections 2024-04-23T10:28:14.625 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:28:14.623+0000 7fdda6a79640 1 -- 172.21.15.114:0/2758498154 wait complete. 2024-04-23T10:28:14.625 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:28:14.624+0000 7fdda6a79640 1 Processor -- start 2024-04-23T10:28:14.625 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:28:14.624+0000 7fdda6a79640 1 -- start start 2024-04-23T10:28:14.625 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:28:14.624+0000 7fdda6a79640 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fdda0101120 0x7fdda01976a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T10:28:14.625 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:28:14.624+0000 7fdda6a79640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fdda01061a0 0x7fdda0197be0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T10:28:14.625 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:28:14.624+0000 7fdda6a79640 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fdda019bf10 0x7fdda019c3c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T10:28:14.625 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:28:14.624+0000 7fdda6a79640 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x7fdda0111160 con 0x7fdda01061a0 2024-04-23T10:28:14.626 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:28:14.624+0000 7fdda6a79640 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 -- 0x7fdda0110fe0 con 0x7fdda0101120 2024-04-23T10:28:14.626 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:28:14.624+0000 7fdda6a79640 1 -- --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_getmap magic: 0 -- 0x7fdda01112e0 con 0x7fdda019bf10 2024-04-23T10:28:14.626 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:28:14.624+0000 7fdd9ffff640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fdda01061a0 0x7fdda0197be0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T10:28:14.626 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:28:14.624+0000 7fdda4815640 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fdda0101120 0x7fdda01976a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T10:28:14.626 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:28:14.624+0000 7fdda4815640 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fdda0101120 0x7fdda01976a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.119:3300/0 says I am v2:172.21.15.114:55716/0 (socket says 172.21.15.114:55716) 2024-04-23T10:28:14.626 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:28:14.624+0000 7fdda4815640 1 -- 172.21.15.114:0/2492709733 learned_addr learned my addr 172.21.15.114:0/2492709733 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-04-23T10:28:14.626 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:28:14.624+0000 7fdd9ffff640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fdda01061a0 0x7fdda0197be0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.114:38764/0 (socket says 172.21.15.114:38764) 2024-04-23T10:28:14.626 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:28:14.624+0000 7fdda5016640 1 --2- 172.21.15.114:0/2492709733 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fdda019bf10 0x7fdda019c3c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T10:28:14.626 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:28:14.624+0000 7fdd9ffff640 1 -- 172.21.15.114:0/2492709733 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fdda019bf10 msgr2=0x7fdda019c3c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T10:28:14.626 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:28:14.625+0000 7fdd9ffff640 1 --2- 172.21.15.114:0/2492709733 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fdda019bf10 0x7fdda019c3c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:28:14.626 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:28:14.625+0000 7fdd9ffff640 1 -- 172.21.15.114:0/2492709733 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fdda0101120 msgr2=0x7fdda01976a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T10:28:14.626 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:28:14.625+0000 7fdd9ffff640 1 --2- 172.21.15.114:0/2492709733 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fdda0101120 0x7fdda01976a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:28:14.626 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:28:14.625+0000 7fdd9ffff640 1 -- 172.21.15.114:0/2492709733 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fdda019ca00 con 0x7fdda01061a0 2024-04-23T10:28:14.627 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:28:14.625+0000 7fdd9ffff640 1 --2- 172.21.15.114:0/2492709733 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fdda01061a0 0x7fdda0197be0 secure :-1 s=READY pgs=548 cs=0 l=1 rev1=1 crypto rx=0x7fdd8c00b3c0 tx=0x7fdd8c00b7c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T10:28:14.627 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:28:14.625+0000 7fdd9dffb640 1 -- 172.21.15.114:0/2492709733 <== mon.0 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdd8c015020 con 0x7fdda01061a0 2024-04-23T10:28:14.627 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:28:14.625+0000 7fdd9dffb640 1 -- 172.21.15.114:0/2492709733 <== mon.0 v2:172.21.15.114:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fdd8c011ce0 con 0x7fdda01061a0 2024-04-23T10:28:14.627 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:28:14.625+0000 7fdda6a79640 1 -- 172.21.15.114:0/2492709733 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fdda019ccf0 con 0x7fdda01061a0 2024-04-23T10:28:14.627 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:28:14.625+0000 7fdd9dffb640 1 -- 172.21.15.114:0/2492709733 <== mon.0 v2:172.21.15.114:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdd8c00f2e0 con 0x7fdda01061a0 2024-04-23T10:28:14.627 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:28:14.625+0000 7fdda6a79640 1 -- 172.21.15.114:0/2492709733 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fdda01a44e0 con 0x7fdda01061a0 2024-04-23T10:28:14.634 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:28:14.627+0000 7fdd9dffb640 1 -- 172.21.15.114:0/2492709733 <== mon.0 v2:172.21.15.114:3300/0 4 ==== mgrmap(e 26) ==== 138778+0+0 (secure 0 0 0) 0x7fdd8c018020 con 0x7fdda01061a0 2024-04-23T10:28:14.634 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:28:14.628+0000 7fdda978e740 1 Processor -- start 2024-04-23T10:28:14.634 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:28:14.628+0000 7fdda978e740 1 -- start start 2024-04-23T10:28:14.634 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:28:14.628+0000 7fdd9dffb640 1 --2- 172.21.15.114:0/2492709733 >> [v2:172.21.15.114:6800/1395693466,v1:172.21.15.114:6801/1395693466] conn(0x7fdd900a3dc0 0x7fdd900a6280 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T10:28:14.634 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:28:14.628+0000 7fdda978e740 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x561d679d1620 0x561d679d1a00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T10:28:14.634 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:28:14.628+0000 7fdda978e740 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x561d679d1f40 0x561d679dc2c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T10:28:14.634 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:28:14.628+0000 7fdda978e740 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x561d679dc800 0x561d679debf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T10:28:14.634 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:28:14.628+0000 7fdda5016640 1 --2- 172.21.15.114:0/2492709733 >> [v2:172.21.15.114:6800/1395693466,v1:172.21.15.114:6801/1395693466] conn(0x7fdd900a3dc0 0x7fdd900a6280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T10:28:14.634 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:28:14.628+0000 7fdda978e740 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x561d67936970 con 0x561d679d1f40 2024-04-23T10:28:14.634 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:28:14.628+0000 7fdda978e740 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 -- 0x561d678d34a0 con 0x561d679dc800 2024-04-23T10:28:14.634 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:28:14.628+0000 7fdda978e740 1 -- --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_getmap magic: 0 -- 0x561d67932930 con 0x561d679d1620 2024-04-23T10:28:14.635 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:28:14.628+0000 7fdd9dffb640 1 -- 172.21.15.114:0/2492709733 <== mon.0 v2:172.21.15.114:3300/0 5 ==== osd_map(92..92 src has 1..92) ==== 8585+0+0 (secure 0 0 0) 0x7fdd8c0cffa0 con 0x7fdda01061a0 2024-04-23T10:28:14.635 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:28:14.628+0000 7fdd9ffff640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x561d679d1f40 0x561d679dc2c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T10:28:14.635 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:28:14.628+0000 7fdd9ffff640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x561d679d1f40 0x561d679dc2c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.114:38778/0 (socket says 172.21.15.114:38778) 2024-04-23T10:28:14.635 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:28:14.628+0000 7fdd9ffff640 1 -- 172.21.15.114:0/1932947483 learned_addr learned my addr 172.21.15.114:0/1932947483 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-04-23T10:28:14.635 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:28:14.628+0000 7fdda4815640 1 --2- 172.21.15.114:0/1932947483 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x561d679d1620 0x561d679d1a00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T10:28:14.635 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:28:14.628+0000 7fdda5016640 1 --2- 172.21.15.114:0/1932947483 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x561d679dc800 0x561d679debf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T10:28:14.635 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:28:14.628+0000 7fdda5016640 1 --2- 172.21.15.114:0/2492709733 >> [v2:172.21.15.114:6800/1395693466,v1:172.21.15.114:6801/1395693466] conn(0x7fdd900a3dc0 0x7fdd900a6280 secure :-1 s=READY pgs=76 cs=0 l=1 rev1=1 crypto rx=0x7fdd98005fd0 tx=0x7fdd98011810 comp rx=0 tx=0).ready entity=mgr.15291 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T10:28:14.635 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:28:14.628+0000 7fdd9ffff640 1 -- 172.21.15.114:0/1932947483 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x561d679d1620 msgr2=0x561d679d1a00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T10:28:14.635 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:28:14.628+0000 7fdd9ffff640 1 --2- 172.21.15.114:0/1932947483 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x561d679d1620 0x561d679d1a00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:28:14.635 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:28:14.628+0000 7fdd9ffff640 1 -- 172.21.15.114:0/1932947483 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x561d679dc800 msgr2=0x561d679debf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T10:28:14.635 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:28:14.628+0000 7fdd9ffff640 1 --2- 172.21.15.114:0/1932947483 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x561d679dc800 0x561d679debf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:28:14.635 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:28:14.628+0000 7fdd9ffff640 1 -- 172.21.15.114:0/1932947483 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x561d67649210 con 0x561d679d1f40 2024-04-23T10:28:14.635 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:28:14.629+0000 7fdd9ffff640 1 --2- 172.21.15.114:0/1932947483 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x561d679d1f40 0x561d679dc2c0 secure :-1 s=READY pgs=549 cs=0 l=1 rev1=1 crypto rx=0x7fdd8c0a46f0 tx=0x7fdd8c09b910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2ec3b56d2a69ebc6 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T10:28:14.636 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:28:14.629+0000 7fdd8affd640 1 -- 172.21.15.114:0/1932947483 <== mon.0 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdd8c0bc040 con 0x561d679d1f40 2024-04-23T10:28:14.636 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:28:14.629+0000 7fdd8affd640 1 -- 172.21.15.114:0/1932947483 <== mon.0 v2:172.21.15.114:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fdd8c0a0360 con 0x561d679d1f40 2024-04-23T10:28:14.636 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:28:14.629+0000 7fdd8affd640 1 -- 172.21.15.114:0/1932947483 <== mon.0 v2:172.21.15.114:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdd8c098620 con 0x561d679d1f40 2024-04-23T10:28:14.636 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:28:14.629+0000 7fdda978e740 1 -- 172.21.15.114:0/1932947483 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x561d679d1f40 msgr2=0x561d679dc2c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T10:28:14.636 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:28:14.629+0000 7fdda978e740 1 --2- 172.21.15.114:0/1932947483 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x561d679d1f40 0x561d679dc2c0 secure :-1 s=READY pgs=549 cs=0 l=1 rev1=1 crypto rx=0x7fdd8c0a46f0 tx=0x7fdd8c09b910 comp rx=0 tx=0).stop 2024-04-23T10:28:14.636 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:28:14.629+0000 7fdda978e740 1 -- 172.21.15.114:0/1932947483 shutdown_connections 2024-04-23T10:28:14.636 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:28:14.629+0000 7fdda978e740 1 --2- 172.21.15.114:0/1932947483 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x561d679dc800 0x561d679debf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:28:14.636 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:28:14.629+0000 7fdda978e740 1 --2- 172.21.15.114:0/1932947483 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x561d679d1f40 0x561d679dc2c0 unknown :-1 s=CLOSED pgs=549 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:28:14.636 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:28:14.629+0000 7fdda978e740 1 --2- 172.21.15.114:0/1932947483 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x561d679d1620 0x561d679d1a00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:28:14.636 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:28:14.629+0000 7fdda978e740 1 -- 172.21.15.114:0/1932947483 >> 172.21.15.114:0/1932947483 conn(0x561d679cdb30 msgr2=0x561d679cdf10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T10:28:14.636 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:28:14.630+0000 7fdda978e740 1 -- 172.21.15.114:0/1932947483 shutdown_connections 2024-04-23T10:28:14.636 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:28:14.630+0000 7fdda978e740 1 -- 172.21.15.114:0/1932947483 wait complete. 2024-04-23T10:28:14.636 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:28:14.630+0000 7fdda978e740 1 -- start start 2024-04-23T10:28:14.636 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:28:14.630+0000 7fdda978e740 1 Processor -- start 2024-04-23T10:28:14.637 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:28:14.630+0000 7fdda978e740 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x561d679d1620 0x561d679dc200 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T10:28:14.637 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:28:14.630+0000 7fdda978e740 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x561d679d1f40 0x561d679d94f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T10:28:14.637 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:28:14.630+0000 7fdda978e740 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x561d679dc800 0x561d679d9b00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T10:28:14.637 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:28:14.630+0000 7fdda978e740 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x561d679df3d0 con 0x561d679d1f40 2024-04-23T10:28:14.637 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:28:14.630+0000 7fdda978e740 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 -- 0x561d678d10b0 con 0x561d679d1620 2024-04-23T10:28:14.637 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:28:14.630+0000 7fdda978e740 1 -- --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_getmap magic: 0 -- 0x561d679df550 con 0x561d679dc800 2024-04-23T10:28:14.637 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:28:14.630+0000 7fdda978e740 10 client.0 _ll_register_callbacks cb 0x561d678bed30 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-23T10:28:14.637 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:28:14.630+0000 7fdd9ffff640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x561d679d1f40 0x561d679d94f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T10:28:14.637 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:28:14.630+0000 7fdda4815640 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x561d679d1620 0x561d679dc200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T10:28:14.637 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:28:14.630+0000 7fdd9ffff640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x561d679d1f40 0x561d679d94f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.114:38786/0 (socket says 172.21.15.114:38786) 2024-04-23T10:28:14.637 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:28:14.630+0000 7fdd9ffff640 1 -- 172.21.15.114:0/2982975601 learned_addr learned my addr 172.21.15.114:0/2982975601 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-04-23T10:28:14.637 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:28:14.630+0000 7fdda5016640 1 --2- 172.21.15.114:0/2982975601 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x561d679dc800 0x561d679d9b00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T10:28:14.637 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:28:14.631+0000 7fdd9ffff640 1 -- 172.21.15.114:0/2982975601 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x561d679dc800 msgr2=0x561d679d9b00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T10:28:14.637 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:28:14.631+0000 7fdd9ffff640 1 --2- 172.21.15.114:0/2982975601 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x561d679dc800 0x561d679d9b00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:28:14.638 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:28:14.631+0000 7fdd9ffff640 1 -- 172.21.15.114:0/2982975601 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x561d679d1620 msgr2=0x561d679dc200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T10:28:14.638 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:28:14.631+0000 7fdd9ffff640 1 --2- 172.21.15.114:0/2982975601 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x561d679d1620 0x561d679dc200 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:28:14.638 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:28:14.631+0000 7fdd9ffff640 1 -- 172.21.15.114:0/2982975601 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({osdmap=0}) -- 0x561d67715620 con 0x561d679d1f40 2024-04-23T10:28:14.638 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:28:14.631+0000 7fdda4815640 1 --2- 172.21.15.114:0/2982975601 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x561d679d1620 0x561d679dc200 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-23T10:28:14.638 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:28:14.631+0000 7fdd9ffff640 1 --2- 172.21.15.114:0/2982975601 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x561d679d1f40 0x561d679d94f0 secure :-1 s=READY pgs=550 cs=0 l=1 rev1=1 crypto rx=0x7fdd8c0104e0 tx=0x7fdd8c0cd700 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c436bab4b95faf6e server_cookie=0 in_seq=0 out_seq=0 2024-04-23T10:28:14.638 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:28:14.631+0000 7fdd7ffff640 10 client.0 ms_handle_connect on v2:172.21.15.114:3300/0 2024-04-23T10:28:14.638 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:28:14.631+0000 7fdd7ffff640 1 -- 172.21.15.114:0/2982975601 <== mon.0 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdd8c0bc070 con 0x561d679d1f40 2024-04-23T10:28:14.638 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:28:14.631+0000 7fdd7ffff640 1 -- 172.21.15.114:0/2982975601 <== mon.0 v2:172.21.15.114:3300/0 2 ==== osd_map(92..92 src has 1..92) ==== 8585+0+0 (secure 0 0 0) 0x7fdd8c0cd940 con 0x561d679d1f40 2024-04-23T10:28:14.638 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:28:14.631+0000 7fdda978e740 1 -- 172.21.15.114:0/2982975601 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fdd8c0a0360 con 0x561d679d1f40 2024-04-23T10:28:14.638 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:28:14.631+0000 7fdd7ffff640 1 -- 172.21.15.114:0/2982975601 <== mon.0 v2:172.21.15.114:3300/0 3 ==== mon_get_version_reply(handle=1 version=22) ==== 24+0+0 (secure 0 0 0) 0x7fdd8c081050 con 0x561d679d1f40 2024-04-23T10:28:14.638 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:28:14.631+0000 7fdda978e740 10 client.15774 fetch_fsmap learned FSMap version 22 2024-04-23T10:28:14.638 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:28:14.631+0000 7fdda978e740 1 -- 172.21.15.114:0/2982975601 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({config=0+,fsmap=22,monmap=4+}) -- 0x561d679df3d0 con 0x561d679d1f40 2024-04-23T10:28:14.638 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:28:14.631+0000 7fdd7ffff640 1 -- 172.21.15.114:0/2982975601 <== mon.0 v2:172.21.15.114:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fdd8c0a0360 con 0x561d679d1f40 2024-04-23T10:28:14.638 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:28:14.632+0000 7fdd7ffff640 1 -- 172.21.15.114:0/2982975601 <== mon.0 v2:172.21.15.114:3300/0 5 ==== fsmap(e 22) ==== 8346+0+0 (secure 0 0 0) 0x7fdd8c09b400 con 0x561d679d1f40 2024-04-23T10:28:14.639 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:28:14.632+0000 7fdda978e740 10 client.15774 fetch_fsmap finished waiting for FSMap version 22 2024-04-23T10:28:14.639 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:28:14.632+0000 7fdda978e740 10 client.15774 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-04-23T10:28:14.639 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:28:14.632+0000 7fdda978e740 20 client.15774 populate_metadata read hostname 'smithi114' 2024-04-23T10:28:14.639 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:28:14.632+0000 7fdda978e740 1 --2- 172.21.15.114:0/2982975601 >> [v2:172.21.15.119:6838/937297340,v1:172.21.15.119:6839/937297340] conn(0x561d679ec500 0x561d679ee8f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T10:28:14.639 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:28:14.632+0000 7fdda978e740 4 client.15774 mds_command: new command op to 24545 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-23T10:28:14.639 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:28:14.632+0000 7fdda978e740 1 -- 172.21.15.114:0/2982975601 --> [v2:172.21.15.119:6838/937297340,v1:172.21.15.119:6839/937297340] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x561d67944390 con 0x561d679ec500 2024-04-23T10:28:14.639 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:28:14.632+0000 7fdda4815640 1 --2- 172.21.15.114:0/2982975601 >> [v2:172.21.15.119:6838/937297340,v1:172.21.15.119:6839/937297340] conn(0x561d679ec500 0x561d679ee8f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T10:28:14.639 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:28:14.632+0000 7fdda4815640 1 --2- 172.21.15.114:0/2982975601 >> [v2:172.21.15.119:6838/937297340,v1:172.21.15.119:6839/937297340] conn(0x561d679ec500 0x561d679ee8f0 crc :-1 s=READY pgs=60 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=af761264a38d7fb8 server_cookie=69adafbded24bb91 in_seq=0 out_seq=0 2024-04-23T10:28:14.639 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:28:14.632+0000 7fdd7ffff640 10 client.15774 ms_handle_connect on v2:172.21.15.119:6838/937297340 2024-04-23T10:28:14.639 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:28:14.633+0000 7fdd7ffff640 1 -- 172.21.15.114:0/2982975601 <== mds.0 v2:172.21.15.119:6838/937297340 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x561d67944390 con 0x561d679ec500 2024-04-23T10:28:14.639 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:28:14.633+0000 7fdd7ffff640 10 client.15774 handle_command_reply: tid=0 2024-04-23T10:28:14.639 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:28:14.633+0000 7fdda978e740 1 client.15774 shutdown 2024-04-23T10:28:14.639 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:28:14.633+0000 7fdda978e740 1 -- 172.21.15.114:0/2982975601 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x561d679d1f40 msgr2=0x561d679d94f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T10:28:14.639 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:28:14.633+0000 7fdda978e740 1 --2- 172.21.15.114:0/2982975601 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x561d679d1f40 0x561d679d94f0 secure :-1 s=READY pgs=550 cs=0 l=1 rev1=1 crypto rx=0x7fdd8c0104e0 tx=0x7fdd8c0cd700 comp rx=0 tx=0).stop 2024-04-23T10:28:14.639 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:28:14.633+0000 7fdda978e740 1 -- 172.21.15.114:0/2982975601 shutdown_connections 2024-04-23T10:28:14.640 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:28:14.633+0000 7fdda978e740 1 --2- 172.21.15.114:0/2982975601 >> [v2:172.21.15.119:6838/937297340,v1:172.21.15.119:6839/937297340] conn(0x561d679ec500 0x561d679ee8f0 crc :-1 s=READY pgs=60 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:28:14.640 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:28:14.633+0000 7fdda978e740 1 --2- 172.21.15.114:0/2982975601 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x561d679dc800 0x561d679d9b00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:28:14.640 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:28:14.633+0000 7fdda978e740 1 --2- 172.21.15.114:0/2982975601 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x561d679d1f40 0x561d679d94f0 unknown :-1 s=CLOSED pgs=550 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:28:14.640 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:28:14.633+0000 7fdda978e740 1 --2- 172.21.15.114:0/2982975601 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x561d679d1620 0x561d679dc200 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:28:14.640 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:28:14.633+0000 7fdda978e740 1 -- 172.21.15.114:0/2982975601 >> 172.21.15.114:0/2982975601 conn(0x561d679cdb30 msgr2=0x561d679cccb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T10:28:14.640 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:28:14.633+0000 7fdd7ffff640 0 client.15774 ms_handle_reset on v2:172.21.15.119:6838/937297340 2024-04-23T10:28:14.640 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:28:14.634+0000 7fdda978e740 1 -- 172.21.15.114:0/2982975601 shutdown_connections 2024-04-23T10:28:14.640 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:28:14.634+0000 7fdda978e740 1 -- 172.21.15.114:0/2982975601 wait complete. 2024-04-23T10:28:14.640 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:28:14.634+0000 7fdda978e740 20 client.15774 trim_cache size 0 max 16384 2024-04-23T10:28:14.652 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:28:14.649+0000 7fdda978e740 1 Processor -- start 2024-04-23T10:28:14.652 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:28:14.649+0000 7fdda978e740 1 -- start start 2024-04-23T10:28:14.652 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:28:14.649+0000 7fdda978e740 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x561d679d1620 0x561d679e8290 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T10:28:14.652 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:28:14.649+0000 7fdda978e740 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x561d679d1f40 0x561d679ccb30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T10:28:14.652 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:28:14.649+0000 7fdda978e740 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x561d679dc800 0x561d679f4060 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T10:28:14.652 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:28:14.649+0000 7fdda978e740 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x7fdda01af500 con 0x561d679d1620 2024-04-23T10:28:14.652 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:28:14.649+0000 7fdda978e740 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 -- 0x7fdda01acdd0 con 0x561d679dc800 2024-04-23T10:28:14.653 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:28:14.649+0000 7fdda978e740 1 -- --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_getmap magic: 0 -- 0x7fdda01aac20 con 0x561d679d1f40 2024-04-23T10:28:14.653 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:28:14.649+0000 7fdda4815640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x561d679d1620 0x561d679e8290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T10:28:14.653 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:28:14.649+0000 7fdda4815640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x561d679d1620 0x561d679e8290 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.114:38802/0 (socket says 172.21.15.114:38802) 2024-04-23T10:28:14.653 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:28:14.649+0000 7fdda4815640 1 -- 172.21.15.114:0/1202367391 learned_addr learned my addr 172.21.15.114:0/1202367391 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-04-23T10:28:14.653 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:28:14.649+0000 7fdda5016640 1 --2- 172.21.15.114:0/1202367391 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x561d679dc800 0x561d679f4060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T10:28:14.653 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:28:14.649+0000 7fdd9ffff640 1 --2- 172.21.15.114:0/1202367391 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x561d679d1f40 0x561d679ccb30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T10:28:14.653 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:28:14.649+0000 7fdda4815640 1 -- 172.21.15.114:0/1202367391 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x561d679d1f40 msgr2=0x561d679ccb30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T10:28:14.653 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:28:14.649+0000 7fdda4815640 1 --2- 172.21.15.114:0/1202367391 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x561d679d1f40 0x561d679ccb30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:28:14.653 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:28:14.649+0000 7fdda4815640 1 -- 172.21.15.114:0/1202367391 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x561d679dc800 msgr2=0x561d679f4060 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T10:28:14.653 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:28:14.649+0000 7fdda4815640 1 --2- 172.21.15.114:0/1202367391 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x561d679dc800 0x561d679f4060 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:28:14.653 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:28:14.649+0000 7fdda4815640 1 -- 172.21.15.114:0/1202367391 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x561d679d1d70 con 0x561d679d1620 2024-04-23T10:28:14.653 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:28:14.649+0000 7fdda5016640 1 --2- 172.21.15.114:0/1202367391 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x561d679dc800 0x561d679f4060 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-23T10:28:14.653 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:28:14.649+0000 7fdda4815640 1 --2- 172.21.15.114:0/1202367391 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x561d679d1620 0x561d679e8290 secure :-1 s=READY pgs=551 cs=0 l=1 rev1=1 crypto rx=0x7fdd94006010 tx=0x7fdd9402a410 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e2c99814371bec8b server_cookie=0 in_seq=0 out_seq=0 2024-04-23T10:28:14.653 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:28:14.650+0000 7fdd8a7fc640 1 -- 172.21.15.114:0/1202367391 <== mon.0 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdd94035040 con 0x561d679d1620 2024-04-23T10:28:14.653 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:28:14.650+0000 7fdd8a7fc640 1 -- 172.21.15.114:0/1202367391 <== mon.0 v2:172.21.15.114:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fdd94031940 con 0x561d679d1620 2024-04-23T10:28:14.654 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:28:14.650+0000 7fdd8a7fc640 1 -- 172.21.15.114:0/1202367391 <== mon.0 v2:172.21.15.114:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdd940053a0 con 0x561d679d1620 2024-04-23T10:28:14.654 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:28:14.650+0000 7fdda978e740 1 -- 172.21.15.114:0/1202367391 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x561d679d1620 msgr2=0x561d679e8290 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T10:28:14.654 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:28:14.650+0000 7fdda978e740 1 --2- 172.21.15.114:0/1202367391 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x561d679d1620 0x561d679e8290 secure :-1 s=READY pgs=551 cs=0 l=1 rev1=1 crypto rx=0x7fdd94006010 tx=0x7fdd9402a410 comp rx=0 tx=0).stop 2024-04-23T10:28:14.654 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:28:14.650+0000 7fdda978e740 1 -- 172.21.15.114:0/1202367391 shutdown_connections 2024-04-23T10:28:14.654 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:28:14.650+0000 7fdda978e740 1 --2- 172.21.15.114:0/1202367391 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x561d679dc800 0x561d679f4060 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:28:14.654 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:28:14.650+0000 7fdda978e740 1 --2- 172.21.15.114:0/1202367391 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x561d679d1f40 0x561d679ccb30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:28:14.654 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:28:14.650+0000 7fdda978e740 1 --2- 172.21.15.114:0/1202367391 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x561d679d1620 0x561d679e8290 unknown :-1 s=CLOSED pgs=551 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:28:14.654 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:28:14.650+0000 7fdda978e740 1 -- 172.21.15.114:0/1202367391 >> 172.21.15.114:0/1202367391 conn(0x561d679cdb30 msgr2=0x561d679cc420 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T10:28:14.654 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:28:14.650+0000 7fdda978e740 1 -- 172.21.15.114:0/1202367391 shutdown_connections 2024-04-23T10:28:14.654 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:28:14.650+0000 7fdda978e740 1 -- 172.21.15.114:0/1202367391 wait complete. 2024-04-23T10:28:14.654 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:28:14.651+0000 7fdda978e740 1 -- start start 2024-04-23T10:28:14.654 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:28:14.651+0000 7fdda978e740 1 Processor -- start 2024-04-23T10:28:14.654 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:28:14.651+0000 7fdda978e740 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x561d679d1620 0x561d679e23a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T10:28:14.654 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:28:14.651+0000 7fdda978e740 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x561d679d1f40 0x561d679e28e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T10:28:14.655 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:28:14.651+0000 7fdda978e740 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x561d679dc800 0x561d679e2e20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T10:28:14.655 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:28:14.651+0000 7fdda978e740 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x7fdda01ab290 con 0x561d679d1f40 2024-04-23T10:28:14.655 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:28:14.651+0000 7fdda978e740 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 -- 0x7fdda01ac700 con 0x561d679d1620 2024-04-23T10:28:14.655 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:28:14.651+0000 7fdda978e740 1 -- --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_getmap magic: 0 -- 0x7fdda01ab960 con 0x561d679dc800 2024-04-23T10:28:14.655 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:28:14.651+0000 7fdda978e740 10 client.0 _ll_register_callbacks cb 0x7fdd8c09be10 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-23T10:28:14.655 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:28:14.651+0000 7fdd9ffff640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x561d679d1f40 0x561d679e28e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T10:28:14.655 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:28:14.651+0000 7fdd9ffff640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x561d679d1f40 0x561d679e28e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.114:38806/0 (socket says 172.21.15.114:38806) 2024-04-23T10:28:14.655 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:28:14.651+0000 7fdd9ffff640 1 -- 172.21.15.114:0/3703937415 learned_addr learned my addr 172.21.15.114:0/3703937415 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-04-23T10:28:14.655 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:28:14.651+0000 7fdda4815640 1 --2- 172.21.15.114:0/3703937415 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x561d679d1620 0x561d679e23a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T10:28:14.655 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:28:14.651+0000 7fdda5016640 1 --2- 172.21.15.114:0/3703937415 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x561d679dc800 0x561d679e2e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T10:28:14.655 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:28:14.652+0000 7fdd9ffff640 1 -- 172.21.15.114:0/3703937415 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x561d679dc800 msgr2=0x561d679e2e20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T10:28:14.655 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:28:14.652+0000 7fdd9ffff640 1 --2- 172.21.15.114:0/3703937415 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x561d679dc800 0x561d679e2e20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:28:14.655 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:28:14.652+0000 7fdd9ffff640 1 -- 172.21.15.114:0/3703937415 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x561d679d1620 msgr2=0x561d679e23a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T10:28:14.655 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:28:14.652+0000 7fdd9ffff640 1 --2- 172.21.15.114:0/3703937415 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x561d679d1620 0x561d679e23a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:28:14.656 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:28:14.652+0000 7fdd9ffff640 1 -- 172.21.15.114:0/3703937415 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({osdmap=0}) -- 0x561d67a51ae0 con 0x561d679d1f40 2024-04-23T10:28:14.656 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:28:14.652+0000 7fdd9ffff640 1 --2- 172.21.15.114:0/3703937415 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x561d679d1f40 0x561d679e28e0 secure :-1 s=READY pgs=552 cs=0 l=1 rev1=1 crypto rx=0x7fdd8c09b7a0 tx=0x7fdd8c0b5470 comp rx=0 tx=0).ready entity=mon.0 client_cookie=fb1352adf6e8c3f9 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T10:28:14.656 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:28:14.652+0000 7fdd7ffff640 10 client.0 ms_handle_connect on v2:172.21.15.114:3300/0 2024-04-23T10:28:14.656 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:28:14.652+0000 7fdd7ffff640 1 -- 172.21.15.114:0/3703937415 <== mon.0 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdd8c0bc040 con 0x561d679d1f40 2024-04-23T10:28:14.656 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:28:14.652+0000 7fdd7ffff640 1 -- 172.21.15.114:0/3703937415 <== mon.0 v2:172.21.15.114:3300/0 2 ==== osd_map(92..92 src has 1..92) ==== 8585+0+0 (secure 0 0 0) 0x7fdd8c098620 con 0x561d679d1f40 2024-04-23T10:28:14.656 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:28:14.652+0000 7fdda978e740 1 -- 172.21.15.114:0/3703937415 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fdd94031940 con 0x561d679d1f40 2024-04-23T10:28:14.656 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:28:14.652+0000 7fdd7ffff640 1 -- 172.21.15.114:0/3703937415 <== mon.0 v2:172.21.15.114:3300/0 3 ==== mon_get_version_reply(handle=1 version=22) ==== 24+0+0 (secure 0 0 0) 0x7fdd8c09c6b0 con 0x561d679d1f40 2024-04-23T10:28:14.656 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:28:14.652+0000 7fdda978e740 10 client.15780 fetch_fsmap learned FSMap version 22 2024-04-23T10:28:14.656 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:28:14.652+0000 7fdda978e740 1 -- 172.21.15.114:0/3703937415 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({config=0+,fsmap=22,monmap=4+}) -- 0x561d67a52db0 con 0x561d679d1f40 2024-04-23T10:28:14.656 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:28:14.652+0000 7fdd7ffff640 1 -- 172.21.15.114:0/3703937415 <== mon.0 v2:172.21.15.114:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fdd8c0ab080 con 0x561d679d1f40 2024-04-23T10:28:14.656 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:28:14.652+0000 7fdd7ffff640 1 -- 172.21.15.114:0/3703937415 <== mon.0 v2:172.21.15.114:3300/0 5 ==== fsmap(e 22) ==== 8346+0+0 (secure 0 0 0) 0x7fdd8c09b400 con 0x561d679d1f40 2024-04-23T10:28:14.656 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:28:14.652+0000 7fdda978e740 10 client.15780 fetch_fsmap finished waiting for FSMap version 22 2024-04-23T10:28:14.656 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:28:14.652+0000 7fdda978e740 10 client.15780 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-04-23T10:28:14.656 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:28:14.652+0000 7fdda978e740 20 client.15780 populate_metadata read hostname 'smithi114' 2024-04-23T10:28:14.656 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:28:14.652+0000 7fdda978e740 1 --2- 172.21.15.114:0/3703937415 >> [v2:172.21.15.119:6838/937297340,v1:172.21.15.119:6839/937297340] conn(0x561d67a53320 0x561d67a55740 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T10:28:14.657 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:28:14.652+0000 7fdda978e740 4 client.15780 mds_command: new command op to 24545 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-04-23T10:28:14.657 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:28:14.652+0000 7fdda978e740 1 -- 172.21.15.114:0/3703937415 --> [v2:172.21.15.119:6838/937297340,v1:172.21.15.119:6839/937297340] -- command(tid 0: {"prefix": "flush journal"}) -- 0x561d6793e200 con 0x561d67a53320 2024-04-23T10:28:14.657 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:28:14.652+0000 7fdda4815640 1 --2- 172.21.15.114:0/3703937415 >> [v2:172.21.15.119:6838/937297340,v1:172.21.15.119:6839/937297340] conn(0x561d67a53320 0x561d67a55740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T10:28:14.657 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:28:14.653+0000 7fdd7ffff640 10 client.15780 ms_handle_connect on v2:172.21.15.119:6838/937297340 2024-04-23T10:28:14.657 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:28:14.653+0000 7fdda4815640 1 --2- 172.21.15.114:0/3703937415 >> [v2:172.21.15.119:6838/937297340,v1:172.21.15.119:6839/937297340] conn(0x561d67a53320 0x561d67a55740 crc :-1 s=READY pgs=61 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=a545c9417c2b6827 server_cookie=68c4b8fff38a2db6 in_seq=0 out_seq=0 2024-04-23T10:28:14.690 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:28:14.691+0000 7fdd7ffff640 1 -- 172.21.15.114:0/3703937415 <== mds.0 v2:172.21.15.119:6838/937297340 1 ==== command_reply(tid 0: 0 ) ==== 8+0+44 (crc 0 0 0) 0x561d6793e200 con 0x561d67a53320 2024-04-23T10:28:14.690 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:28:14.691+0000 7fdd7ffff640 10 client.15780 handle_command_reply: tid=0 2024-04-23T10:28:14.691 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:28:14.691+0000 7fdda978e740 1 client.15780 shutdown 2024-04-23T10:28:14.691 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:28:14.691+0000 7fdda978e740 1 -- 172.21.15.114:0/3703937415 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x561d679d1f40 msgr2=0x561d679e28e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T10:28:14.691 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:28:14.691+0000 7fdda978e740 1 --2- 172.21.15.114:0/3703937415 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x561d679d1f40 0x561d679e28e0 secure :-1 s=READY pgs=552 cs=0 l=1 rev1=1 crypto rx=0x7fdd8c09b7a0 tx=0x7fdd8c0b5470 comp rx=0 tx=0).stop 2024-04-23T10:28:14.691 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:28:14.691+0000 7fdda978e740 1 -- 172.21.15.114:0/3703937415 shutdown_connections 2024-04-23T10:28:14.691 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:28:14.691+0000 7fdda978e740 1 --2- 172.21.15.114:0/3703937415 >> [v2:172.21.15.119:6838/937297340,v1:172.21.15.119:6839/937297340] conn(0x561d67a53320 0x561d67a55740 crc :-1 s=READY pgs=61 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:28:14.691 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:28:14.691+0000 7fdda978e740 1 --2- 172.21.15.114:0/3703937415 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x561d679dc800 0x561d679e2e20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:28:14.691 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:28:14.691+0000 7fdda978e740 1 --2- 172.21.15.114:0/3703937415 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x561d679d1f40 0x561d679e28e0 unknown :-1 s=CLOSED pgs=552 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:28:14.691 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:28:14.691+0000 7fdd7ffff640 0 client.15780 ms_handle_reset on v2:172.21.15.119:6838/937297340 2024-04-23T10:28:14.691 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:28:14.691+0000 7fdda978e740 1 --2- 172.21.15.114:0/3703937415 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x561d679d1620 0x561d679e23a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:28:14.691 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:28:14.691+0000 7fdda978e740 1 -- 172.21.15.114:0/3703937415 >> 172.21.15.114:0/3703937415 conn(0x561d679cdb30 msgr2=0x561d679cccb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T10:28:14.691 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:28:14.691+0000 7fdda978e740 1 -- 172.21.15.114:0/3703937415 shutdown_connections 2024-04-23T10:28:14.691 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:28:14.692+0000 7fdda978e740 1 -- 172.21.15.114:0/3703937415 wait complete. 2024-04-23T10:28:14.691 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:28:14.692+0000 7fdda978e740 20 client.15780 trim_cache size 0 max 16384 2024-04-23T10:28:14.692 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stdout:{ 2024-04-23T10:28:14.692 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stdout: "message": "", 2024-04-23T10:28:14.692 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stdout: "return_code": 0 2024-04-23T10:28:14.692 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stdout:} 2024-04-23T10:28:14.693 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:28:14.692+0000 7fdda6a79640 1 -- 172.21.15.114:0/2492709733 >> [v2:172.21.15.114:6800/1395693466,v1:172.21.15.114:6801/1395693466] conn(0x7fdd900a3dc0 msgr2=0x7fdd900a6280 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T10:28:14.693 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:28:14.692+0000 7fdda6a79640 1 --2- 172.21.15.114:0/2492709733 >> [v2:172.21.15.114:6800/1395693466,v1:172.21.15.114:6801/1395693466] conn(0x7fdd900a3dc0 0x7fdd900a6280 secure :-1 s=READY pgs=76 cs=0 l=1 rev1=1 crypto rx=0x7fdd98005fd0 tx=0x7fdd98011810 comp rx=0 tx=0).stop 2024-04-23T10:28:14.693 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:28:14.692+0000 7fdda6a79640 1 -- 172.21.15.114:0/2492709733 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fdda01061a0 msgr2=0x7fdda0197be0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T10:28:14.693 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:28:14.692+0000 7fdda6a79640 1 --2- 172.21.15.114:0/2492709733 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fdda01061a0 0x7fdda0197be0 secure :-1 s=READY pgs=548 cs=0 l=1 rev1=1 crypto rx=0x7fdd8c00b3c0 tx=0x7fdd8c00b7c0 comp rx=0 tx=0).stop 2024-04-23T10:28:14.693 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:28:14.692+0000 7fdda6a79640 1 -- 172.21.15.114:0/2492709733 shutdown_connections 2024-04-23T10:28:14.693 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:28:14.692+0000 7fdda6a79640 1 --2- 172.21.15.114:0/2492709733 >> [v2:172.21.15.114:6800/1395693466,v1:172.21.15.114:6801/1395693466] conn(0x7fdd900a3dc0 0x7fdd900a6280 unknown :-1 s=CLOSED pgs=76 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:28:14.693 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:28:14.692+0000 7fdda6a79640 1 --2- 172.21.15.114:0/2492709733 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fdda019bf10 0x7fdda019c3c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:28:14.693 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:28:14.692+0000 7fdda6a79640 1 --2- 172.21.15.114:0/2492709733 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fdda01061a0 0x7fdda0197be0 unknown :-1 s=CLOSED pgs=548 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:28:14.693 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:28:14.692+0000 7fdda6a79640 1 --2- 172.21.15.114:0/2492709733 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fdda0101120 0x7fdda01976a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:28:14.693 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:28:14.692+0000 7fdda6a79640 1 -- 172.21.15.114:0/2492709733 >> 172.21.15.114:0/2492709733 conn(0x7fdda0077fa0 msgr2=0x7fdda00feff0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T10:28:14.693 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:28:14.692+0000 7fdda6a79640 1 -- 172.21.15.114:0/2492709733 shutdown_connections 2024-04-23T10:28:14.693 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:28:14.692+0000 7fdda6a79640 1 -- 172.21.15.114:0/2492709733 wait complete. 2024-04-23T10:28:14.907 INFO:tasks.workunit.client.0.smithi114.stdout: 1 46562 11.72 MB/sec warmup 23 sec latency 32.753 ms 2024-04-23T10:28:15.741 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 23 10:28:15 smithi119 ceph-mon[32608]: pgmap v52: 97 pgs: 97 active+clean; 18 MiB data, 455 MiB used, 1.0 TiB / 1.0 TiB avail; 409 KiB/s rd, 759 KiB/s wr, 209 op/s 2024-04-23T10:28:15.754 INFO:journalctl@ceph.mon.a.smithi114.stdout:Apr 23 10:28:15 smithi114 ceph-mon[29255]: pgmap v52: 97 pgs: 97 active+clean; 18 MiB data, 455 MiB used, 1.0 TiB / 1.0 TiB avail; 409 KiB/s rd, 759 KiB/s wr, 209 op/s 2024-04-23T10:28:15.827 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 23 10:28:15 smithi204 ceph-mon[32649]: pgmap v52: 97 pgs: 97 active+clean; 18 MiB data, 455 MiB used, 1.0 TiB / 1.0 TiB avail; 409 KiB/s rd, 759 KiB/s wr, 209 op/s 2024-04-23T10:28:15.907 INFO:tasks.workunit.client.0.smithi114.stdout: 1 48923 11.69 MB/sec warmup 24 sec latency 14.592 ms 2024-04-23T10:28:16.908 INFO:tasks.workunit.client.0.smithi114.stdout: 1 50601 11.58 MB/sec warmup 25 sec latency 14.219 ms 2024-04-23T10:28:17.741 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 23 10:28:17 smithi119 ceph-mon[32608]: pgmap v53: 97 pgs: 97 active+clean; 31 MiB data, 523 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 MiB/s rd, 5.8 MiB/s wr, 425 op/s 2024-04-23T10:28:17.753 INFO:journalctl@ceph.mon.a.smithi114.stdout:Apr 23 10:28:17 smithi114 ceph-mon[29255]: pgmap v53: 97 pgs: 97 active+clean; 31 MiB data, 523 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 MiB/s rd, 5.8 MiB/s wr, 425 op/s 2024-04-23T10:28:17.827 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 23 10:28:17 smithi204 ceph-mon[32649]: pgmap v53: 97 pgs: 97 active+clean; 31 MiB data, 523 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 MiB/s rd, 5.8 MiB/s wr, 425 op/s 2024-04-23T10:28:17.907 INFO:tasks.workunit.client.0.smithi114.stdout: 1 53479 11.83 MB/sec warmup 26 sec latency 11.641 ms 2024-04-23T10:28:18.908 INFO:tasks.workunit.client.0.smithi114.stdout: 1 55328 11.61 MB/sec warmup 27 sec latency 14.581 ms 2024-04-23T10:28:19.741 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 23 10:28:19 smithi119 ceph-mon[32608]: pgmap v54: 97 pgs: 97 active+clean; 31 MiB data, 523 MiB used, 1.0 TiB / 1.0 TiB avail; 975 KiB/s rd, 5.3 MiB/s wr, 274 op/s 2024-04-23T10:28:19.753 INFO:journalctl@ceph.mon.a.smithi114.stdout:Apr 23 10:28:19 smithi114 ceph-mon[29255]: pgmap v54: 97 pgs: 97 active+clean; 31 MiB data, 523 MiB used, 1.0 TiB / 1.0 TiB avail; 975 KiB/s rd, 5.3 MiB/s wr, 274 op/s 2024-04-23T10:28:19.827 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 23 10:28:19 smithi204 ceph-mon[32649]: pgmap v54: 97 pgs: 97 active+clean; 31 MiB data, 523 MiB used, 1.0 TiB / 1.0 TiB avail; 975 KiB/s rd, 5.3 MiB/s wr, 274 op/s 2024-04-23T10:28:19.908 INFO:tasks.workunit.client.0.smithi114.stdout: 1 57327 11.68 MB/sec warmup 28 sec latency 63.788 ms 2024-04-23T10:28:20.908 INFO:tasks.workunit.client.0.smithi114.stdout: 1 60141 11.77 MB/sec warmup 29 sec latency 13.391 ms 2024-04-23T10:28:21.741 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 23 10:28:21 smithi119 ceph-mon[32608]: pgmap v55: 97 pgs: 97 active+clean; 31 MiB data, 523 MiB used, 1.0 TiB / 1.0 TiB avail; 928 KiB/s rd, 5.3 MiB/s wr, 259 op/s 2024-04-23T10:28:21.753 INFO:journalctl@ceph.mon.a.smithi114.stdout:Apr 23 10:28:21 smithi114 ceph-mon[29255]: pgmap v55: 97 pgs: 97 active+clean; 31 MiB data, 523 MiB used, 1.0 TiB / 1.0 TiB avail; 928 KiB/s rd, 5.3 MiB/s wr, 259 op/s 2024-04-23T10:28:21.827 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 23 10:28:21 smithi204 ceph-mon[32649]: pgmap v55: 97 pgs: 97 active+clean; 31 MiB data, 523 MiB used, 1.0 TiB / 1.0 TiB avail; 928 KiB/s rd, 5.3 MiB/s wr, 259 op/s 2024-04-23T10:28:21.908 INFO:tasks.workunit.client.0.smithi114.stdout: 1 61714 11.70 MB/sec warmup 30 sec latency 13.560 ms 2024-04-23T10:28:22.908 INFO:tasks.workunit.client.0.smithi114.stdout: 1 64246 11.78 MB/sec warmup 31 sec latency 12.035 ms 2024-04-23T10:28:23.741 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 23 10:28:23 smithi119 ceph-mon[32608]: pgmap v56: 97 pgs: 97 active+clean; 35 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 2.2 MiB/s rd, 5.9 MiB/s wr, 366 op/s 2024-04-23T10:28:23.754 INFO:journalctl@ceph.mon.a.smithi114.stdout:Apr 23 10:28:23 smithi114 ceph-mon[29255]: pgmap v56: 97 pgs: 97 active+clean; 35 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 2.2 MiB/s rd, 5.9 MiB/s wr, 366 op/s 2024-04-23T10:28:23.827 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 23 10:28:23 smithi204 ceph-mon[32649]: pgmap v56: 97 pgs: 97 active+clean; 35 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 2.2 MiB/s rd, 5.9 MiB/s wr, 366 op/s 2024-04-23T10:28:23.908 INFO:tasks.workunit.client.0.smithi114.stdout: 1 66429 11.66 MB/sec warmup 32 sec latency 15.067 ms 2024-04-23T10:28:24.908 INFO:tasks.workunit.client.0.smithi114.stdout: 1 68201 11.66 MB/sec warmup 33 sec latency 16.171 ms 2024-04-23T10:28:25.741 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 23 10:28:25 smithi119 ceph-mon[32608]: pgmap v57: 97 pgs: 97 active+clean; 35 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 2.0 MiB/s rd, 5.6 MiB/s wr, 324 op/s 2024-04-23T10:28:25.754 INFO:journalctl@ceph.mon.a.smithi114.stdout:Apr 23 10:28:25 smithi114 ceph-mon[29255]: pgmap v57: 97 pgs: 97 active+clean; 35 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 2.0 MiB/s rd, 5.6 MiB/s wr, 324 op/s 2024-04-23T10:28:25.827 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 23 10:28:25 smithi204 ceph-mon[32649]: pgmap v57: 97 pgs: 97 active+clean; 35 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 2.0 MiB/s rd, 5.6 MiB/s wr, 324 op/s 2024-04-23T10:28:25.908 INFO:tasks.workunit.client.0.smithi114.stdout: 1 71101 11.83 MB/sec warmup 34 sec latency 26.368 ms 2024-04-23T10:28:26.908 INFO:tasks.workunit.client.0.smithi114.stdout: 1 72622 11.68 MB/sec warmup 35 sec latency 15.679 ms 2024-04-23T10:28:27.741 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 23 10:28:27 smithi119 ceph-mon[32608]: pgmap v58: 97 pgs: 97 active+clean; 41 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 MiB/s rd, 6.3 MiB/s wr, 428 op/s 2024-04-23T10:28:27.753 INFO:journalctl@ceph.mon.a.smithi114.stdout:Apr 23 10:28:27 smithi114 ceph-mon[29255]: pgmap v58: 97 pgs: 97 active+clean; 41 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 MiB/s rd, 6.3 MiB/s wr, 428 op/s 2024-04-23T10:28:27.785 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:28:27.786+0000 7f8d93536640 1 -- 172.21.15.114:0/4216727952 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f8d8c101aa0 msgr2=0x7f8d8c105a90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T10:28:27.785 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:28:27.786+0000 7f8d93536640 1 --2- 172.21.15.114:0/4216727952 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f8d8c101aa0 0x7f8d8c105a90 secure :-1 s=READY pgs=553 cs=0 l=1 rev1=1 crypto rx=0x7f8d7c009900 tx=0x7f8d7c02a410 comp rx=0 tx=0).stop 2024-04-23T10:28:27.785 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:28:27.786+0000 7f8d93536640 1 -- 172.21.15.114:0/4216727952 shutdown_connections 2024-04-23T10:28:27.786 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:28:27.786+0000 7f8d93536640 1 --2- 172.21.15.114:0/4216727952 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f8d8c106170 0x7f8d8c10dd40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:28:27.786 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:28:27.786+0000 7f8d93536640 1 --2- 172.21.15.114:0/4216727952 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f8d8c101aa0 0x7f8d8c105a90 unknown :-1 s=CLOSED pgs=553 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:28:27.786 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:28:27.786+0000 7f8d93536640 1 --2- 172.21.15.114:0/4216727952 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f8d8c1010f0 0x7f8d8c1014d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:28:27.786 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:28:27.786+0000 7f8d93536640 1 -- 172.21.15.114:0/4216727952 >> 172.21.15.114:0/4216727952 conn(0x7f8d8c077fd0 msgr2=0x7f8d8c0783e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T10:28:27.786 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:28:27.786+0000 7f8d93536640 1 -- 172.21.15.114:0/4216727952 shutdown_connections 2024-04-23T10:28:27.786 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:28:27.786+0000 7f8d93536640 1 -- 172.21.15.114:0/4216727952 wait complete. 2024-04-23T10:28:27.788 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:28:27.787+0000 7f8d93536640 1 Processor -- start 2024-04-23T10:28:27.788 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:28:27.787+0000 7f8d93536640 1 -- start start 2024-04-23T10:28:27.788 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:28:27.787+0000 7f8d93536640 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f8d8c1010f0 0x7f8d8c197610 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T10:28:27.788 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:28:27.787+0000 7f8d93536640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f8d8c106170 0x7f8d8c197b50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T10:28:27.788 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:28:27.787+0000 7f8d93536640 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f8d8c19be80 0x7f8d8c19c330 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T10:28:27.788 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:28:27.787+0000 7f8d93536640 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x7f8d8c111090 con 0x7f8d8c106170 2024-04-23T10:28:27.788 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:28:27.787+0000 7f8d93536640 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 -- 0x7f8d8c110f10 con 0x7f8d8c19be80 2024-04-23T10:28:27.788 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:28:27.787+0000 7f8d93536640 1 -- --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_getmap magic: 0 -- 0x7f8d8c111210 con 0x7f8d8c1010f0 2024-04-23T10:28:27.788 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:28:27.787+0000 7f8d912d2640 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f8d8c1010f0 0x7f8d8c197610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T10:28:27.788 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:28:27.787+0000 7f8d90ad1640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f8d8c106170 0x7f8d8c197b50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T10:28:27.788 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:28:27.787+0000 7f8d912d2640 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f8d8c1010f0 0x7f8d8c197610 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.204:3300/0 says I am v2:172.21.15.114:58244/0 (socket says 172.21.15.114:58244) 2024-04-23T10:28:27.788 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:28:27.787+0000 7f8d90ad1640 1 --2- 172.21.15.114:0/3104580118 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f8d8c106170 0x7f8d8c197b50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.114:40020/0 (socket says 172.21.15.114:40020) 2024-04-23T10:28:27.788 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:28:27.787+0000 7f8d912d2640 1 -- 172.21.15.114:0/3104580118 learned_addr learned my addr 172.21.15.114:0/3104580118 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-04-23T10:28:27.788 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:28:27.787+0000 7f8d91ad3640 1 --2- 172.21.15.114:0/3104580118 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f8d8c19be80 0x7f8d8c19c330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T10:28:27.788 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:28:27.788+0000 7f8d91ad3640 1 -- 172.21.15.114:0/3104580118 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f8d8c1010f0 msgr2=0x7f8d8c197610 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T10:28:27.789 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:28:27.788+0000 7f8d91ad3640 1 --2- 172.21.15.114:0/3104580118 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f8d8c1010f0 0x7f8d8c197610 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:28:27.789 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:28:27.788+0000 7f8d91ad3640 1 -- 172.21.15.114:0/3104580118 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f8d8c106170 msgr2=0x7f8d8c197b50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T10:28:27.789 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:28:27.788+0000 7f8d91ad3640 1 --2- 172.21.15.114:0/3104580118 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f8d8c106170 0x7f8d8c197b50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:28:27.789 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:28:27.788+0000 7f8d91ad3640 1 -- 172.21.15.114:0/3104580118 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f8d8c19c970 con 0x7f8d8c19be80 2024-04-23T10:28:27.789 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:28:27.788+0000 7f8d91ad3640 1 --2- 172.21.15.114:0/3104580118 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f8d8c19be80 0x7f8d8c19c330 secure :-1 s=READY pgs=65 cs=0 l=1 rev1=1 crypto rx=0x7f8d84009fd0 tx=0x7f8d8400ef10 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T10:28:27.789 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:28:27.788+0000 7f8d827fc640 1 -- 172.21.15.114:0/3104580118 <== mon.2 v2:172.21.15.119:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8d8400ce20 con 0x7f8d8c19be80 2024-04-23T10:28:27.789 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:28:27.788+0000 7f8d93536640 1 -- 172.21.15.114:0/3104580118 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f8d8c19cc60 con 0x7f8d8c19be80 2024-04-23T10:28:27.789 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:28:27.788+0000 7f8d827fc640 1 -- 172.21.15.114:0/3104580118 <== mon.2 v2:172.21.15.119:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f8d84005020 con 0x7f8d8c19be80 2024-04-23T10:28:27.789 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:28:27.788+0000 7f8d827fc640 1 -- 172.21.15.114:0/3104580118 <== mon.2 v2:172.21.15.119:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8d84010690 con 0x7f8d8c19be80 2024-04-23T10:28:27.789 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:28:27.788+0000 7f8d93536640 1 -- 172.21.15.114:0/3104580118 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f8d8c1a43d0 con 0x7f8d8c19be80 2024-04-23T10:28:27.790 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:28:27.791+0000 7f8d96255740 1 Processor -- start 2024-04-23T10:28:27.793 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:28:27.792+0000 7f8d827fc640 1 -- 172.21.15.114:0/3104580118 <== mon.2 v2:172.21.15.119:3300/0 4 ==== mgrmap(e 26) ==== 138778+0+0 (secure 0 0 0) 0x7f8d840054d0 con 0x7f8d8c19be80 2024-04-23T10:28:27.793 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:28:27.792+0000 7f8d96255740 1 -- start start 2024-04-23T10:28:27.793 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:28:27.792+0000 7f8d96255740 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x5634b6bf9620 0x5634b6bf9a00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T10:28:27.793 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:28:27.792+0000 7f8d96255740 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x5634b6bf9f40 0x5634b6c042c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T10:28:27.793 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:28:27.792+0000 7f8d96255740 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x5634b6c04800 0x5634b6c06bf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T10:28:27.793 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:28:27.792+0000 7f8d96255740 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x5634b6b5e970 con 0x5634b6c04800 2024-04-23T10:28:27.793 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:28:27.792+0000 7f8d96255740 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 -- 0x5634b6afb4a0 con 0x5634b6bf9620 2024-04-23T10:28:27.793 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:28:27.792+0000 7f8d96255740 1 -- --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_getmap magic: 0 -- 0x5634b6b5a930 con 0x5634b6bf9f40 2024-04-23T10:28:27.793 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:28:27.792+0000 7f8d90ad1640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x5634b6c04800 0x5634b6c06bf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T10:28:27.793 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:28:27.792+0000 7f8d90ad1640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x5634b6c04800 0x5634b6c06bf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.114:40028/0 (socket says 172.21.15.114:40028) 2024-04-23T10:28:27.793 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:28:27.792+0000 7f8d912d2640 1 --2- 172.21.15.114:0/3889706819 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x5634b6bf9f40 0x5634b6c042c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T10:28:27.793 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:28:27.792+0000 7f8d90ad1640 1 -- 172.21.15.114:0/3889706819 learned_addr learned my addr 172.21.15.114:0/3889706819 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-04-23T10:28:27.794 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:28:27.792+0000 7f8d90ad1640 1 --2- 172.21.15.114:0/3889706819 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x5634b6bf9620 0x5634b6bf9a00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T10:28:27.794 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:28:27.792+0000 7f8d827fc640 1 --2- 172.21.15.114:0/3104580118 >> [v2:172.21.15.114:6800/1395693466,v1:172.21.15.114:6801/1395693466] conn(0x7f8d780a3dc0 0x7f8d780a6280 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T10:28:27.794 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:28:27.793+0000 7f8d912d2640 1 -- 172.21.15.114:0/3889706819 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x5634b6bf9620 msgr2=0x5634b6bf9a00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T10:28:27.794 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:28:27.793+0000 7f8d912d2640 1 --2- 172.21.15.114:0/3889706819 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x5634b6bf9620 0x5634b6bf9a00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:28:27.794 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:28:27.793+0000 7f8d91ad3640 1 --2- 172.21.15.114:0/3104580118 >> [v2:172.21.15.114:6800/1395693466,v1:172.21.15.114:6801/1395693466] conn(0x7f8d780a3dc0 0x7f8d780a6280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T10:28:27.794 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:28:27.793+0000 7f8d912d2640 1 -- 172.21.15.114:0/3889706819 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x5634b6c04800 msgr2=0x5634b6c06bf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T10:28:27.794 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:28:27.793+0000 7f8d912d2640 1 --2- 172.21.15.114:0/3889706819 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x5634b6c04800 0x5634b6c06bf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:28:27.794 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:28:27.793+0000 7f8d912d2640 1 -- 172.21.15.114:0/3889706819 --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5634b6871210 con 0x5634b6bf9f40 2024-04-23T10:28:27.794 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:28:27.793+0000 7f8d90ad1640 1 --2- 172.21.15.114:0/3889706819 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x5634b6bf9620 0x5634b6bf9a00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-23T10:28:27.794 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:28:27.793+0000 7f8d827fc640 1 -- 172.21.15.114:0/3104580118 <== mon.2 v2:172.21.15.119:3300/0 5 ==== osd_map(92..92 src has 1..92) ==== 8585+0+0 (secure 0 0 0) 0x7f8d840d0ea0 con 0x7f8d8c19be80 2024-04-23T10:28:27.794 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:28:27.793+0000 7f8d912d2640 1 --2- 172.21.15.114:0/3889706819 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x5634b6bf9f40 0x5634b6c042c0 secure :-1 s=READY pgs=63 cs=0 l=1 rev1=1 crypto rx=0x7f8d88009cf0 tx=0x7f8d88011810 comp rx=0 tx=0).ready entity=mon.1 client_cookie=9c3fa4d33bc04b74 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T10:28:27.794 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:28:27.793+0000 7f8d777fe640 1 -- 172.21.15.114:0/3889706819 <== mon.1 v2:172.21.15.204:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8d88005220 con 0x5634b6bf9f40 2024-04-23T10:28:27.794 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:28:27.793+0000 7f8d777fe640 1 -- 172.21.15.114:0/3889706819 <== mon.1 v2:172.21.15.204:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f8d88004c50 con 0x5634b6bf9f40 2024-04-23T10:28:27.794 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:28:27.793+0000 7f8d91ad3640 1 --2- 172.21.15.114:0/3104580118 >> [v2:172.21.15.114:6800/1395693466,v1:172.21.15.114:6801/1395693466] conn(0x7f8d780a3dc0 0x7f8d780a6280 secure :-1 s=READY pgs=77 cs=0 l=1 rev1=1 crypto rx=0x7f8d840ce660 tx=0x7f8d8406e040 comp rx=0 tx=0).ready entity=mgr.15291 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T10:28:27.794 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:28:27.793+0000 7f8d777fe640 1 -- 172.21.15.114:0/3889706819 <== mon.1 v2:172.21.15.204:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8d88019690 con 0x5634b6bf9f40 2024-04-23T10:28:27.795 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:28:27.794+0000 7f8d96255740 1 -- 172.21.15.114:0/3889706819 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x5634b6bf9f40 msgr2=0x5634b6c042c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T10:28:27.795 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:28:27.794+0000 7f8d96255740 1 --2- 172.21.15.114:0/3889706819 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x5634b6bf9f40 0x5634b6c042c0 secure :-1 s=READY pgs=63 cs=0 l=1 rev1=1 crypto rx=0x7f8d88009cf0 tx=0x7f8d88011810 comp rx=0 tx=0).stop 2024-04-23T10:28:27.795 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:28:27.794+0000 7f8d96255740 1 -- 172.21.15.114:0/3889706819 shutdown_connections 2024-04-23T10:28:27.795 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:28:27.794+0000 7f8d96255740 1 --2- 172.21.15.114:0/3889706819 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x5634b6c04800 0x5634b6c06bf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:28:27.795 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:28:27.794+0000 7f8d96255740 1 --2- 172.21.15.114:0/3889706819 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x5634b6bf9f40 0x5634b6c042c0 unknown :-1 s=CLOSED pgs=63 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:28:27.795 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:28:27.794+0000 7f8d96255740 1 --2- 172.21.15.114:0/3889706819 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x5634b6bf9620 0x5634b6bf9a00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:28:27.795 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:28:27.794+0000 7f8d96255740 1 -- 172.21.15.114:0/3889706819 >> 172.21.15.114:0/3889706819 conn(0x5634b6bf5b30 msgr2=0x5634b6bf5f10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T10:28:27.795 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:28:27.794+0000 7f8d96255740 1 -- 172.21.15.114:0/3889706819 shutdown_connections 2024-04-23T10:28:27.795 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:28:27.794+0000 7f8d96255740 1 -- 172.21.15.114:0/3889706819 wait complete. 2024-04-23T10:28:27.795 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:28:27.794+0000 7f8d96255740 1 -- start start 2024-04-23T10:28:27.795 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:28:27.794+0000 7f8d96255740 1 Processor -- start 2024-04-23T10:28:27.799 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:28:27.795+0000 7f8d96255740 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x5634b6bf9620 0x5634b6c040c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T10:28:27.799 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:28:27.795+0000 7f8d96255740 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x5634b6bf9f40 0x5634b6c013b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T10:28:27.799 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:28:27.795+0000 7f8d96255740 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x5634b6c04800 0x5634b6c018f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T10:28:27.799 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:28:27.795+0000 7f8d96255740 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x5634b6c07430 con 0x5634b6c04800 2024-04-23T10:28:27.799 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:28:27.795+0000 7f8d96255740 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 -- 0x5634b6af90b0 con 0x5634b6bf9f40 2024-04-23T10:28:27.799 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:28:27.795+0000 7f8d96255740 1 -- --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_getmap magic: 0 -- 0x5634b6c075b0 con 0x5634b6bf9620 2024-04-23T10:28:27.799 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:28:27.795+0000 7f8d96255740 10 client.0 _ll_register_callbacks cb 0x5634b6ae6d30 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-23T10:28:27.799 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:28:27.795+0000 7f8d90ad1640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x5634b6c04800 0x5634b6c018f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T10:28:27.799 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:28:27.795+0000 7f8d90ad1640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x5634b6c04800 0x5634b6c018f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.114:40034/0 (socket says 172.21.15.114:40034) 2024-04-23T10:28:27.799 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:28:27.795+0000 7f8d90ad1640 1 -- 172.21.15.114:0/316644167 learned_addr learned my addr 172.21.15.114:0/316644167 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-04-23T10:28:27.799 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:28:27.795+0000 7f8d912d2640 1 --2- 172.21.15.114:0/316644167 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x5634b6bf9f40 0x5634b6c013b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T10:28:27.799 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:28:27.795+0000 7f8d90ad1640 1 --2- 172.21.15.114:0/316644167 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x5634b6bf9620 0x5634b6c040c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T10:28:27.799 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:28:27.795+0000 7f8d90ad1640 1 -- 172.21.15.114:0/316644167 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x5634b6bf9620 msgr2=0x5634b6c040c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T10:28:27.799 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:28:27.795+0000 7f8d90ad1640 1 --2- 172.21.15.114:0/316644167 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x5634b6bf9620 0x5634b6c040c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:28:27.800 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:28:27.795+0000 7f8d90ad1640 1 -- 172.21.15.114:0/316644167 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x5634b6bf9f40 msgr2=0x5634b6c013b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T10:28:27.800 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:28:27.795+0000 7f8d90ad1640 1 --2- 172.21.15.114:0/316644167 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x5634b6bf9f40 0x5634b6c013b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:28:27.800 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:28:27.795+0000 7f8d90ad1640 1 -- 172.21.15.114:0/316644167 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5634b693d620 con 0x5634b6c04800 2024-04-23T10:28:27.800 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:28:27.795+0000 7f8d90ad1640 1 --2- 172.21.15.114:0/316644167 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x5634b6c04800 0x5634b6c018f0 secure :-1 s=READY pgs=554 cs=0 l=1 rev1=1 crypto rx=0x7f8d7c006010 tx=0x7f8d7c02a410 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a86d35b69210a435 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T10:28:27.800 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:28:27.795+0000 7f8d74ff9640 10 client.0 ms_handle_connect on v2:172.21.15.114:3300/0 2024-04-23T10:28:27.800 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:28:27.795+0000 7f8d74ff9640 1 -- 172.21.15.114:0/316644167 <== mon.0 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8d7c035070 con 0x5634b6c04800 2024-04-23T10:28:27.800 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:28:27.795+0000 7f8d74ff9640 1 -- 172.21.15.114:0/316644167 <== mon.0 v2:172.21.15.114:3300/0 2 ==== osd_map(92..92 src has 1..92) ==== 8585+0+0 (secure 0 0 0) 0x7f8d7c004250 con 0x5634b6c04800 2024-04-23T10:28:27.800 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:28:27.795+0000 7f8d96255740 1 -- 172.21.15.114:0/316644167 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f8d88004c50 con 0x5634b6c04800 2024-04-23T10:28:27.800 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:28:27.795+0000 7f8d74ff9640 1 -- 172.21.15.114:0/316644167 <== mon.0 v2:172.21.15.114:3300/0 3 ==== mon_get_version_reply(handle=1 version=22) ==== 24+0+0 (secure 0 0 0) 0x7f8d7c005850 con 0x5634b6c04800 2024-04-23T10:28:27.800 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:28:27.795+0000 7f8d96255740 10 client.15792 fetch_fsmap learned FSMap version 22 2024-04-23T10:28:27.800 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:28:27.795+0000 7f8d96255740 1 -- 172.21.15.114:0/316644167 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({config=0+,fsmap=22,monmap=4+}) -- 0x5634b6c132c0 con 0x5634b6c04800 2024-04-23T10:28:27.800 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:28:27.796+0000 7f8d74ff9640 1 -- 172.21.15.114:0/316644167 <== mon.0 v2:172.21.15.114:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f8d88004c50 con 0x5634b6c04800 2024-04-23T10:28:27.800 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:28:27.796+0000 7f8d74ff9640 1 -- 172.21.15.114:0/316644167 <== mon.0 v2:172.21.15.114:3300/0 5 ==== fsmap(e 22) ==== 8346+0+0 (secure 0 0 0) 0x7f8d7c03d550 con 0x5634b6c04800 2024-04-23T10:28:27.800 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:28:27.796+0000 7f8d96255740 10 client.15792 fetch_fsmap finished waiting for FSMap version 22 2024-04-23T10:28:27.800 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:28:27.796+0000 7f8d96255740 10 client.15792 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-04-23T10:28:27.800 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:28:27.796+0000 7f8d96255740 20 client.15792 populate_metadata read hostname 'smithi114' 2024-04-23T10:28:27.801 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:28:27.796+0000 7f8d96255740 1 --2- 172.21.15.114:0/316644167 >> [v2:172.21.15.119:6838/937297340,v1:172.21.15.119:6839/937297340] conn(0x5634b6c14770 0x5634b6c16bf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T10:28:27.801 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:28:27.796+0000 7f8d96255740 4 client.15792 mds_command: new command op to 24545 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-23T10:28:27.801 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:28:27.796+0000 7f8d96255740 1 -- 172.21.15.114:0/316644167 --> [v2:172.21.15.119:6838/937297340,v1:172.21.15.119:6839/937297340] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x5634b6b6c390 con 0x5634b6c14770 2024-04-23T10:28:27.801 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:28:27.796+0000 7f8d912d2640 1 --2- 172.21.15.114:0/316644167 >> [v2:172.21.15.119:6838/937297340,v1:172.21.15.119:6839/937297340] conn(0x5634b6c14770 0x5634b6c16bf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T10:28:27.801 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:28:27.796+0000 7f8d912d2640 1 --2- 172.21.15.114:0/316644167 >> [v2:172.21.15.119:6838/937297340,v1:172.21.15.119:6839/937297340] conn(0x5634b6c14770 0x5634b6c16bf0 crc :-1 s=READY pgs=62 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=475828113bfd97b1 server_cookie=7b01e29805810e5b in_seq=0 out_seq=0 2024-04-23T10:28:27.801 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:28:27.796+0000 7f8d74ff9640 10 client.15792 ms_handle_connect on v2:172.21.15.119:6838/937297340 2024-04-23T10:28:27.801 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:28:27.797+0000 7f8d74ff9640 1 -- 172.21.15.114:0/316644167 <== mds.0 v2:172.21.15.119:6838/937297340 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x5634b6b6c390 con 0x5634b6c14770 2024-04-23T10:28:27.801 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:28:27.797+0000 7f8d74ff9640 10 client.15792 handle_command_reply: tid=0 2024-04-23T10:28:27.801 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:28:27.797+0000 7f8d96255740 1 client.15792 shutdown 2024-04-23T10:28:27.801 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:28:27.797+0000 7f8d96255740 1 -- 172.21.15.114:0/316644167 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x5634b6c04800 msgr2=0x5634b6c018f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T10:28:27.801 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:28:27.797+0000 7f8d96255740 1 --2- 172.21.15.114:0/316644167 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x5634b6c04800 0x5634b6c018f0 secure :-1 s=READY pgs=554 cs=0 l=1 rev1=1 crypto rx=0x7f8d7c006010 tx=0x7f8d7c02a410 comp rx=0 tx=0).stop 2024-04-23T10:28:27.801 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:28:27.797+0000 7f8d96255740 1 -- 172.21.15.114:0/316644167 shutdown_connections 2024-04-23T10:28:27.801 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:28:27.797+0000 7f8d96255740 1 --2- 172.21.15.114:0/316644167 >> [v2:172.21.15.119:6838/937297340,v1:172.21.15.119:6839/937297340] conn(0x5634b6c14770 0x5634b6c16bf0 crc :-1 s=READY pgs=62 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:28:27.801 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:28:27.797+0000 7f8d96255740 1 --2- 172.21.15.114:0/316644167 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x5634b6c04800 0x5634b6c018f0 unknown :-1 s=CLOSED pgs=554 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:28:27.801 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:28:27.797+0000 7f8d96255740 1 --2- 172.21.15.114:0/316644167 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x5634b6bf9f40 0x5634b6c013b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:28:27.801 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:28:27.797+0000 7f8d96255740 1 --2- 172.21.15.114:0/316644167 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x5634b6bf9620 0x5634b6c040c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:28:27.802 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:28:27.797+0000 7f8d96255740 1 -- 172.21.15.114:0/316644167 >> 172.21.15.114:0/316644167 conn(0x5634b6bf5b30 msgr2=0x5634b6bfb860 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T10:28:27.802 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:28:27.797+0000 7f8d74ff9640 0 client.15792 ms_handle_reset on v2:172.21.15.119:6838/937297340 2024-04-23T10:28:27.802 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:28:27.798+0000 7f8d96255740 1 -- 172.21.15.114:0/316644167 shutdown_connections 2024-04-23T10:28:27.802 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:28:27.798+0000 7f8d96255740 1 -- 172.21.15.114:0/316644167 wait complete. 2024-04-23T10:28:27.802 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:28:27.798+0000 7f8d96255740 20 client.15792 trim_cache size 0 max 16384 2024-04-23T10:28:27.814 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:28:27.814+0000 7f8d96255740 1 Processor -- start 2024-04-23T10:28:27.814 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:28:27.814+0000 7f8d96255740 1 -- start start 2024-04-23T10:28:27.814 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:28:27.814+0000 7f8d96255740 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x5634b6bf9620 0x5634b6c102f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T10:28:27.814 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:28:27.814+0000 7f8d96255740 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x5634b6bf9f40 0x5634b6c00510 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T10:28:27.814 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:28:27.814+0000 7f8d96255740 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x5634b6c04800 0x5634b6bf4b30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T10:28:27.814 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:28:27.814+0000 7f8d96255740 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x7f8d8c1af440 con 0x5634b6bf9620 2024-04-23T10:28:27.814 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:28:27.814+0000 7f8d96255740 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 -- 0x7f8d8c1acd10 con 0x5634b6c04800 2024-04-23T10:28:27.814 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:28:27.814+0000 7f8d96255740 1 -- --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_getmap magic: 0 -- 0x7f8d8c1aab60 con 0x5634b6bf9f40 2024-04-23T10:28:27.814 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:28:27.814+0000 7f8d90ad1640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x5634b6bf9620 0x5634b6c102f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T10:28:27.814 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:28:27.814+0000 7f8d90ad1640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x5634b6bf9620 0x5634b6c102f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.114:40044/0 (socket says 172.21.15.114:40044) 2024-04-23T10:28:27.814 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:28:27.814+0000 7f8d90ad1640 1 -- 172.21.15.114:0/1107144571 learned_addr learned my addr 172.21.15.114:0/1107144571 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-04-23T10:28:27.814 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:28:27.814+0000 7f8d912d2640 1 --2- 172.21.15.114:0/1107144571 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x5634b6bf9f40 0x5634b6c00510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T10:28:27.815 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:28:27.814+0000 7f8d90ad1640 1 --2- 172.21.15.114:0/1107144571 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x5634b6c04800 0x5634b6bf4b30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T10:28:27.815 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:28:27.814+0000 7f8d912d2640 1 -- 172.21.15.114:0/1107144571 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x5634b6c04800 msgr2=0x5634b6bf4b30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T10:28:27.815 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:28:27.814+0000 7f8d912d2640 1 --2- 172.21.15.114:0/1107144571 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x5634b6c04800 0x5634b6bf4b30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:28:27.815 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:28:27.814+0000 7f8d912d2640 1 -- 172.21.15.114:0/1107144571 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x5634b6bf9620 msgr2=0x5634b6c102f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T10:28:27.815 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:28:27.814+0000 7f8d912d2640 1 --2- 172.21.15.114:0/1107144571 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x5634b6bf9620 0x5634b6c102f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:28:27.815 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:28:27.814+0000 7f8d912d2640 1 -- 172.21.15.114:0/1107144571 --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5634b6c16f50 con 0x5634b6bf9f40 2024-04-23T10:28:27.815 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:28:27.815+0000 7f8d912d2640 1 --2- 172.21.15.114:0/1107144571 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x5634b6bf9f40 0x5634b6c00510 secure :-1 s=READY pgs=64 cs=0 l=1 rev1=1 crypto rx=0x7f8d8801b040 tx=0x7f8d88005f90 comp rx=0 tx=0).ready entity=mon.1 client_cookie=a31097aaa06acb3d server_cookie=0 in_seq=0 out_seq=0 2024-04-23T10:28:27.815 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:28:27.815+0000 7f8d76ffd640 1 -- 172.21.15.114:0/1107144571 <== mon.1 v2:172.21.15.204:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8d8802d020 con 0x5634b6bf9f40 2024-04-23T10:28:27.815 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:28:27.815+0000 7f8d76ffd640 1 -- 172.21.15.114:0/1107144571 <== mon.1 v2:172.21.15.204:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f8d8802d670 con 0x5634b6bf9f40 2024-04-23T10:28:27.815 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:28:27.815+0000 7f8d76ffd640 1 -- 172.21.15.114:0/1107144571 <== mon.1 v2:172.21.15.204:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8d8802c880 con 0x5634b6bf9f40 2024-04-23T10:28:27.818 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:28:27.816+0000 7f8d96255740 1 -- 172.21.15.114:0/1107144571 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x5634b6bf9f40 msgr2=0x5634b6c00510 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T10:28:27.818 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:28:27.816+0000 7f8d96255740 1 --2- 172.21.15.114:0/1107144571 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x5634b6bf9f40 0x5634b6c00510 secure :-1 s=READY pgs=64 cs=0 l=1 rev1=1 crypto rx=0x7f8d8801b040 tx=0x7f8d88005f90 comp rx=0 tx=0).stop 2024-04-23T10:28:27.818 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:28:27.816+0000 7f8d96255740 1 -- 172.21.15.114:0/1107144571 shutdown_connections 2024-04-23T10:28:27.819 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:28:27.816+0000 7f8d96255740 1 --2- 172.21.15.114:0/1107144571 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x5634b6c04800 0x5634b6bf4b30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:28:27.819 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:28:27.816+0000 7f8d96255740 1 --2- 172.21.15.114:0/1107144571 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x5634b6bf9f40 0x5634b6c00510 unknown :-1 s=CLOSED pgs=64 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:28:27.819 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:28:27.816+0000 7f8d96255740 1 --2- 172.21.15.114:0/1107144571 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x5634b6bf9620 0x5634b6c102f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:28:27.819 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:28:27.816+0000 7f8d96255740 1 -- 172.21.15.114:0/1107144571 >> 172.21.15.114:0/1107144571 conn(0x5634b6bf5b30 msgr2=0x5634b6bf4420 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T10:28:27.819 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:28:27.816+0000 7f8d96255740 1 -- 172.21.15.114:0/1107144571 shutdown_connections 2024-04-23T10:28:27.819 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:28:27.816+0000 7f8d96255740 1 -- 172.21.15.114:0/1107144571 wait complete. 2024-04-23T10:28:27.819 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:28:27.816+0000 7f8d96255740 1 -- start start 2024-04-23T10:28:27.819 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:28:27.816+0000 7f8d96255740 1 Processor -- start 2024-04-23T10:28:27.819 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:28:27.816+0000 7f8d96255740 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x5634b6bf9620 0x5634b6bfda10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T10:28:27.819 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:28:27.816+0000 7f8d96255740 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x5634b6bf9f40 0x5634b6bfdf50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T10:28:27.819 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:28:27.816+0000 7f8d96255740 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x5634b6c04800 0x5634b6c7ad50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T10:28:27.819 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:28:27.816+0000 7f8d96255740 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x7f8d8c1ab1d0 con 0x5634b6bf9620 2024-04-23T10:28:27.819 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:28:27.816+0000 7f8d96255740 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 -- 0x7f8d8c1ac640 con 0x5634b6bf9f40 2024-04-23T10:28:27.819 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:28:27.816+0000 7f8d96255740 1 -- --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_getmap magic: 0 -- 0x7f8d8c1ab8a0 con 0x5634b6c04800 2024-04-23T10:28:27.819 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:28:27.817+0000 7f8d96255740 10 client.0 _ll_register_callbacks cb 0x7f8d7c0315d0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-23T10:28:27.819 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:28:27.817+0000 7f8d90ad1640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x5634b6bf9620 0x5634b6bfda10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T10:28:27.819 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:28:27.817+0000 7f8d90ad1640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x5634b6bf9620 0x5634b6bfda10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.114:40046/0 (socket says 172.21.15.114:40046) 2024-04-23T10:28:27.820 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:28:27.817+0000 7f8d912d2640 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x5634b6bf9f40 0x5634b6bfdf50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T10:28:27.820 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:28:27.817+0000 7f8d90ad1640 1 -- 172.21.15.114:0/1617709985 learned_addr learned my addr 172.21.15.114:0/1617709985 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-04-23T10:28:27.820 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:28:27.817+0000 7f8d90ad1640 1 --2- 172.21.15.114:0/1617709985 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x5634b6c04800 0x5634b6c7ad50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T10:28:27.820 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:28:27.817+0000 7f8d90ad1640 1 -- 172.21.15.114:0/1617709985 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x5634b6c04800 msgr2=0x5634b6c7ad50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T10:28:27.820 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:28:27.817+0000 7f8d90ad1640 1 --2- 172.21.15.114:0/1617709985 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x5634b6c04800 0x5634b6c7ad50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:28:27.820 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:28:27.817+0000 7f8d90ad1640 1 -- 172.21.15.114:0/1617709985 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x5634b6bf9f40 msgr2=0x5634b6bfdf50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T10:28:27.820 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:28:27.817+0000 7f8d90ad1640 1 --2- 172.21.15.114:0/1617709985 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x5634b6bf9f40 0x5634b6bfdf50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:28:27.820 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:28:27.817+0000 7f8d90ad1640 1 -- 172.21.15.114:0/1617709985 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5634b6c7b290 con 0x5634b6bf9620 2024-04-23T10:28:27.820 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:28:27.817+0000 7f8d912d2640 1 --2- 172.21.15.114:0/1617709985 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x5634b6bf9f40 0x5634b6bfdf50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-23T10:28:27.820 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:28:27.817+0000 7f8d90ad1640 1 --2- 172.21.15.114:0/1617709985 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x5634b6bf9620 0x5634b6bfda10 secure :-1 s=READY pgs=555 cs=0 l=1 rev1=1 crypto rx=0x7f8d7c03db10 tx=0x7f8d7c000bc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=815470fc183bed0c server_cookie=0 in_seq=0 out_seq=0 2024-04-23T10:28:27.820 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:28:27.817+0000 7f8d74ff9640 10 client.0 ms_handle_connect on v2:172.21.15.114:3300/0 2024-04-23T10:28:27.820 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:28:27.817+0000 7f8d74ff9640 1 -- 172.21.15.114:0/1617709985 <== mon.0 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8d7c058040 con 0x5634b6bf9620 2024-04-23T10:28:27.820 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:28:27.817+0000 7f8d74ff9640 1 -- 172.21.15.114:0/1617709985 <== mon.0 v2:172.21.15.114:3300/0 2 ==== osd_map(92..92 src has 1..92) ==== 8585+0+0 (secure 0 0 0) 0x7f8d7c031ac0 con 0x5634b6bf9620 2024-04-23T10:28:27.820 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:28:27.817+0000 7f8d96255740 1 -- 172.21.15.114:0/1617709985 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f8d8802d670 con 0x5634b6bf9620 2024-04-23T10:28:27.820 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:28:27.817+0000 7f8d74ff9640 1 -- 172.21.15.114:0/1617709985 <== mon.0 v2:172.21.15.114:3300/0 3 ==== mon_get_version_reply(handle=1 version=22) ==== 24+0+0 (secure 0 0 0) 0x7f8d7c031cc0 con 0x5634b6bf9620 2024-04-23T10:28:27.820 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:28:27.817+0000 7f8d96255740 10 client.15798 fetch_fsmap learned FSMap version 22 2024-04-23T10:28:27.821 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:28:27.817+0000 7f8d96255740 1 -- 172.21.15.114:0/1617709985 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({config=0+,fsmap=22,monmap=4+}) -- 0x5634b6c7c390 con 0x5634b6bf9620 2024-04-23T10:28:27.821 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:28:27.818+0000 7f8d74ff9640 1 -- 172.21.15.114:0/1617709985 <== mon.0 v2:172.21.15.114:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f8d8802d670 con 0x5634b6bf9620 2024-04-23T10:28:27.821 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:28:27.818+0000 7f8d74ff9640 1 -- 172.21.15.114:0/1617709985 <== mon.0 v2:172.21.15.114:3300/0 5 ==== fsmap(e 22) ==== 8346+0+0 (secure 0 0 0) 0x7f8d7c005070 con 0x5634b6bf9620 2024-04-23T10:28:27.821 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:28:27.818+0000 7f8d96255740 10 client.15798 fetch_fsmap finished waiting for FSMap version 22 2024-04-23T10:28:27.821 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:28:27.818+0000 7f8d96255740 10 client.15798 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-04-23T10:28:27.821 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:28:27.818+0000 7f8d96255740 20 client.15798 populate_metadata read hostname 'smithi114' 2024-04-23T10:28:27.821 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:28:27.818+0000 7f8d96255740 1 --2- 172.21.15.114:0/1617709985 >> [v2:172.21.15.119:6838/937297340,v1:172.21.15.119:6839/937297340] conn(0x5634b6c7cac0 0x5634b6c7eee0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T10:28:27.821 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:28:27.818+0000 7f8d96255740 4 client.15798 mds_command: new command op to 24545 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-04-23T10:28:27.821 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:28:27.818+0000 7f8d96255740 1 -- 172.21.15.114:0/1617709985 --> [v2:172.21.15.119:6838/937297340,v1:172.21.15.119:6839/937297340] -- command(tid 0: {"prefix": "flush journal"}) -- 0x5634b6b66200 con 0x5634b6c7cac0 2024-04-23T10:28:27.821 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:28:27.818+0000 7f8d912d2640 1 --2- 172.21.15.114:0/1617709985 >> [v2:172.21.15.119:6838/937297340,v1:172.21.15.119:6839/937297340] conn(0x5634b6c7cac0 0x5634b6c7eee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T10:28:27.821 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:28:27.818+0000 7f8d912d2640 1 --2- 172.21.15.114:0/1617709985 >> [v2:172.21.15.119:6838/937297340,v1:172.21.15.119:6839/937297340] conn(0x5634b6c7cac0 0x5634b6c7eee0 crc :-1 s=READY pgs=63 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=8dec75fc06bc6bf7 server_cookie=923c4fb3a23a45ee in_seq=0 out_seq=0 2024-04-23T10:28:27.821 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:28:27.818+0000 7f8d74ff9640 10 client.15798 ms_handle_connect on v2:172.21.15.119:6838/937297340 2024-04-23T10:28:27.827 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 23 10:28:27 smithi204 ceph-mon[32649]: pgmap v58: 97 pgs: 97 active+clean; 41 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 MiB/s rd, 6.3 MiB/s wr, 428 op/s 2024-04-23T10:28:27.880 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:28:27.880+0000 7f8d74ff9640 1 -- 172.21.15.114:0/1617709985 <== mds.0 v2:172.21.15.119:6838/937297340 1 ==== command_reply(tid 0: 0 ) ==== 8+0+44 (crc 0 0 0) 0x5634b6b66200 con 0x5634b6c7cac0 2024-04-23T10:28:27.880 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:28:27.880+0000 7f8d74ff9640 10 client.15798 handle_command_reply: tid=0 2024-04-23T10:28:27.880 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:28:27.880+0000 7f8d96255740 1 client.15798 shutdown 2024-04-23T10:28:27.880 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:28:27.880+0000 7f8d96255740 1 -- 172.21.15.114:0/1617709985 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x5634b6bf9620 msgr2=0x5634b6bfda10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T10:28:27.880 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:28:27.880+0000 7f8d96255740 1 --2- 172.21.15.114:0/1617709985 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x5634b6bf9620 0x5634b6bfda10 secure :-1 s=READY pgs=555 cs=0 l=1 rev1=1 crypto rx=0x7f8d7c03db10 tx=0x7f8d7c000bc0 comp rx=0 tx=0).stop 2024-04-23T10:28:27.880 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:28:27.880+0000 7f8d96255740 1 -- 172.21.15.114:0/1617709985 shutdown_connections 2024-04-23T10:28:27.880 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:28:27.880+0000 7f8d96255740 1 --2- 172.21.15.114:0/1617709985 >> [v2:172.21.15.119:6838/937297340,v1:172.21.15.119:6839/937297340] conn(0x5634b6c7cac0 0x5634b6c7eee0 crc :-1 s=READY pgs=63 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:28:27.880 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:28:27.880+0000 7f8d96255740 1 --2- 172.21.15.114:0/1617709985 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x5634b6c04800 0x5634b6c7ad50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:28:27.880 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:28:27.880+0000 7f8d74ff9640 0 client.15798 ms_handle_reset on v2:172.21.15.119:6838/937297340 2024-04-23T10:28:27.880 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:28:27.880+0000 7f8d96255740 1 --2- 172.21.15.114:0/1617709985 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x5634b6bf9f40 0x5634b6bfdf50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:28:27.880 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:28:27.880+0000 7f8d96255740 1 --2- 172.21.15.114:0/1617709985 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x5634b6bf9620 0x5634b6bfda10 unknown :-1 s=CLOSED pgs=555 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:28:27.880 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:28:27.881+0000 7f8d96255740 1 -- 172.21.15.114:0/1617709985 >> 172.21.15.114:0/1617709985 conn(0x5634b6bf5b30 msgr2=0x5634b6bf4420 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T10:28:27.880 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:28:27.881+0000 7f8d96255740 1 -- 172.21.15.114:0/1617709985 shutdown_connections 2024-04-23T10:28:27.881 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:28:27.881+0000 7f8d96255740 1 -- 172.21.15.114:0/1617709985 wait complete. 2024-04-23T10:28:27.881 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:28:27.881+0000 7f8d96255740 20 client.15798 trim_cache size 0 max 16384 2024-04-23T10:28:27.881 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stdout:{ 2024-04-23T10:28:27.881 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stdout: "message": "", 2024-04-23T10:28:27.881 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stdout: "return_code": 0 2024-04-23T10:28:27.881 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stdout:} 2024-04-23T10:28:27.882 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:28:27.881+0000 7f8d93536640 1 -- 172.21.15.114:0/3104580118 >> [v2:172.21.15.114:6800/1395693466,v1:172.21.15.114:6801/1395693466] conn(0x7f8d780a3dc0 msgr2=0x7f8d780a6280 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T10:28:27.882 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:28:27.881+0000 7f8d93536640 1 --2- 172.21.15.114:0/3104580118 >> [v2:172.21.15.114:6800/1395693466,v1:172.21.15.114:6801/1395693466] conn(0x7f8d780a3dc0 0x7f8d780a6280 secure :-1 s=READY pgs=77 cs=0 l=1 rev1=1 crypto rx=0x7f8d840ce660 tx=0x7f8d8406e040 comp rx=0 tx=0).stop 2024-04-23T10:28:27.882 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:28:27.881+0000 7f8d93536640 1 -- 172.21.15.114:0/3104580118 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f8d8c19be80 msgr2=0x7f8d8c19c330 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T10:28:27.882 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:28:27.881+0000 7f8d93536640 1 --2- 172.21.15.114:0/3104580118 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f8d8c19be80 0x7f8d8c19c330 secure :-1 s=READY pgs=65 cs=0 l=1 rev1=1 crypto rx=0x7f8d84009fd0 tx=0x7f8d8400ef10 comp rx=0 tx=0).stop 2024-04-23T10:28:27.882 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:28:27.882+0000 7f8d93536640 1 -- 172.21.15.114:0/3104580118 shutdown_connections 2024-04-23T10:28:27.882 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:28:27.882+0000 7f8d93536640 1 --2- 172.21.15.114:0/3104580118 >> [v2:172.21.15.114:6800/1395693466,v1:172.21.15.114:6801/1395693466] conn(0x7f8d780a3dc0 0x7f8d780a6280 unknown :-1 s=CLOSED pgs=77 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:28:27.882 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:28:27.882+0000 7f8d93536640 1 --2- 172.21.15.114:0/3104580118 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f8d8c19be80 0x7f8d8c19c330 unknown :-1 s=CLOSED pgs=65 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:28:27.882 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:28:27.882+0000 7f8d93536640 1 --2- 172.21.15.114:0/3104580118 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f8d8c106170 0x7f8d8c197b50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:28:27.882 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:28:27.882+0000 7f8d93536640 1 --2- 172.21.15.114:0/3104580118 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f8d8c1010f0 0x7f8d8c197610 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:28:27.882 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:28:27.882+0000 7f8d93536640 1 -- 172.21.15.114:0/3104580118 >> 172.21.15.114:0/3104580118 conn(0x7f8d8c077fd0 msgr2=0x7f8d8c0fefb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T10:28:27.882 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:28:27.882+0000 7f8d93536640 1 -- 172.21.15.114:0/3104580118 shutdown_connections 2024-04-23T10:28:27.882 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:28:27.882+0000 7f8d93536640 1 -- 172.21.15.114:0/3104580118 wait complete. 2024-04-23T10:28:27.908 INFO:tasks.workunit.client.0.smithi114.stdout: 1 74975 11.74 MB/sec warmup 36 sec latency 42.660 ms 2024-04-23T10:28:28.908 INFO:tasks.workunit.client.0.smithi114.stdout: 1 77304 11.72 MB/sec warmup 37 sec latency 52.607 ms 2024-04-23T10:28:29.741 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 23 10:28:29 smithi119 ceph-mon[32608]: pgmap v59: 97 pgs: 97 active+clean; 41 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 MiB/s rd, 1.3 MiB/s wr, 211 op/s 2024-04-23T10:28:29.754 INFO:journalctl@ceph.mon.a.smithi114.stdout:Apr 23 10:28:29 smithi114 ceph-mon[29255]: pgmap v59: 97 pgs: 97 active+clean; 41 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 MiB/s rd, 1.3 MiB/s wr, 211 op/s 2024-04-23T10:28:29.827 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 23 10:28:29 smithi204 ceph-mon[32649]: pgmap v59: 97 pgs: 97 active+clean; 41 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 MiB/s rd, 1.3 MiB/s wr, 211 op/s 2024-04-23T10:28:29.908 INFO:tasks.workunit.client.0.smithi114.stdout: 1 78960 11.64 MB/sec warmup 38 sec latency 17.145 ms 2024-04-23T10:28:30.908 INFO:tasks.workunit.client.0.smithi114.stdout: 1 81871 11.81 MB/sec warmup 39 sec latency 11.711 ms 2024-04-23T10:28:31.742 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 23 10:28:31 smithi119 ceph-mon[32608]: pgmap v60: 97 pgs: 97 active+clean; 41 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 MiB/s rd, 1.3 MiB/s wr, 211 op/s 2024-04-23T10:28:31.753 INFO:journalctl@ceph.mon.a.smithi114.stdout:Apr 23 10:28:31 smithi114 ceph-mon[29255]: pgmap v60: 97 pgs: 97 active+clean; 41 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 MiB/s rd, 1.3 MiB/s wr, 211 op/s 2024-04-23T10:28:31.827 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 23 10:28:31 smithi204 ceph-mon[32649]: pgmap v60: 97 pgs: 97 active+clean; 41 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 MiB/s rd, 1.3 MiB/s wr, 211 op/s 2024-04-23T10:28:31.908 INFO:tasks.workunit.client.0.smithi114.stdout: 1 83481 11.66 MB/sec warmup 40 sec latency 13.221 ms 2024-04-23T10:28:32.908 INFO:tasks.workunit.client.0.smithi114.stdout: 1 85680 11.71 MB/sec warmup 41 sec latency 11.964 ms 2024-04-23T10:28:33.741 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 23 10:28:33 smithi119 ceph-mon[32608]: pgmap v61: 97 pgs: 97 active+clean; 39 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 3.8 MiB/s rd, 1.9 MiB/s wr, 372 op/s 2024-04-23T10:28:33.754 INFO:journalctl@ceph.mon.a.smithi114.stdout:Apr 23 10:28:33 smithi114 ceph-mon[29255]: pgmap v61: 97 pgs: 97 active+clean; 39 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 3.8 MiB/s rd, 1.9 MiB/s wr, 372 op/s 2024-04-23T10:28:33.827 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 23 10:28:33 smithi204 ceph-mon[32649]: pgmap v61: 97 pgs: 97 active+clean; 39 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 3.8 MiB/s rd, 1.9 MiB/s wr, 372 op/s 2024-04-23T10:28:33.908 INFO:tasks.workunit.client.0.smithi114.stdout: 1 88372 11.74 MB/sec warmup 42 sec latency 15.576 ms 2024-04-23T10:28:34.909 INFO:tasks.workunit.client.0.smithi114.stdout: 1 89862 11.65 MB/sec warmup 43 sec latency 12.368 ms 2024-04-23T10:28:35.742 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 23 10:28:35 smithi119 ceph-mon[32608]: pgmap v62: 97 pgs: 97 active+clean; 39 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 2.6 MiB/s rd, 1.3 MiB/s wr, 265 op/s 2024-04-23T10:28:35.754 INFO:journalctl@ceph.mon.a.smithi114.stdout:Apr 23 10:28:35 smithi114 ceph-mon[29255]: pgmap v62: 97 pgs: 97 active+clean; 39 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 2.6 MiB/s rd, 1.3 MiB/s wr, 265 op/s 2024-04-23T10:28:35.827 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 23 10:28:35 smithi204 ceph-mon[32649]: pgmap v62: 97 pgs: 97 active+clean; 39 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 2.6 MiB/s rd, 1.3 MiB/s wr, 265 op/s 2024-04-23T10:28:35.908 INFO:tasks.workunit.client.0.smithi114.stdout: 1 92578 11.77 MB/sec warmup 44 sec latency 13.746 ms 2024-04-23T10:28:36.908 INFO:tasks.workunit.client.0.smithi114.stdout: 1 94708 11.67 MB/sec warmup 45 sec latency 14.743 ms 2024-04-23T10:28:37.741 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 23 10:28:37 smithi119 ceph-mon[32608]: pgmap v63: 97 pgs: 97 active+clean; 45 MiB data, 630 MiB used, 1.0 TiB / 1.0 TiB avail; 3.8 MiB/s rd, 1.9 MiB/s wr, 369 op/s 2024-04-23T10:28:37.753 INFO:journalctl@ceph.mon.a.smithi114.stdout:Apr 23 10:28:37 smithi114 ceph-mon[29255]: pgmap v63: 97 pgs: 97 active+clean; 45 MiB data, 630 MiB used, 1.0 TiB / 1.0 TiB avail; 3.8 MiB/s rd, 1.9 MiB/s wr, 369 op/s 2024-04-23T10:28:37.827 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 23 10:28:37 smithi204 ceph-mon[32649]: pgmap v63: 97 pgs: 97 active+clean; 45 MiB data, 630 MiB used, 1.0 TiB / 1.0 TiB avail; 3.8 MiB/s rd, 1.9 MiB/s wr, 369 op/s 2024-04-23T10:28:37.908 INFO:tasks.workunit.client.0.smithi114.stdout: 1 96538 11.69 MB/sec warmup 46 sec latency 11.145 ms 2024-04-23T10:28:38.909 INFO:tasks.workunit.client.0.smithi114.stdout: 1 99508 11.81 MB/sec warmup 47 sec latency 13.563 ms 2024-04-23T10:28:39.741 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 23 10:28:39 smithi119 ceph-mon[32608]: pgmap v64: 97 pgs: 97 active+clean; 45 MiB data, 630 MiB used, 1.0 TiB / 1.0 TiB avail; 2.6 MiB/s rd, 1.3 MiB/s wr, 264 op/s 2024-04-23T10:28:39.754 INFO:journalctl@ceph.mon.a.smithi114.stdout:Apr 23 10:28:39 smithi114 ceph-mon[29255]: pgmap v64: 97 pgs: 97 active+clean; 45 MiB data, 630 MiB used, 1.0 TiB / 1.0 TiB avail; 2.6 MiB/s rd, 1.3 MiB/s wr, 264 op/s 2024-04-23T10:28:39.828 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 23 10:28:39 smithi204 ceph-mon[32649]: pgmap v64: 97 pgs: 97 active+clean; 45 MiB data, 630 MiB used, 1.0 TiB / 1.0 TiB avail; 2.6 MiB/s rd, 1.3 MiB/s wr, 264 op/s 2024-04-23T10:28:39.909 INFO:tasks.workunit.client.0.smithi114.stdout: 1 101010 11.70 MB/sec warmup 48 sec latency 12.716 ms 2024-04-23T10:28:40.741 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 23 10:28:40 smithi119 ceph-mon[32608]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=e=up:active} 3 up:standby-replay 6 up:standby 2024-04-23T10:28:40.753 INFO:journalctl@ceph.mon.a.smithi114.stdout:Apr 23 10:28:40 smithi114 ceph-mon[29255]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=e=up:active} 3 up:standby-replay 6 up:standby 2024-04-23T10:28:40.827 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 23 10:28:40 smithi204 ceph-mon[32649]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=e=up:active} 3 up:standby-replay 6 up:standby 2024-04-23T10:28:40.908 INFO:tasks.workunit.client.0.smithi114.stdout: 1 103350 11.75 MB/sec warmup 49 sec latency 12.094 ms 2024-04-23T10:28:40.972 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:28:40.971+0000 7f25d31c6640 1 -- 172.21.15.114:0/3430052476 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f25cc1010f0 msgr2=0x7f25cc1014d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T10:28:40.973 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:28:40.971+0000 7f25d31c6640 1 --2- 172.21.15.114:0/3430052476 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f25cc1010f0 0x7f25cc1014d0 secure :-1 s=READY pgs=556 cs=0 l=1 rev1=1 crypto rx=0x7f25c0009900 tx=0x7f25c002a410 comp rx=0 tx=0).stop 2024-04-23T10:28:40.973 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:28:40.972+0000 7f25d31c6640 1 -- 172.21.15.114:0/3430052476 shutdown_connections 2024-04-23T10:28:40.973 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:28:40.972+0000 7f25d31c6640 1 --2- 172.21.15.114:0/3430052476 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f25cc106170 0x7f25cc10dd40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:28:40.973 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:28:40.972+0000 7f25d31c6640 1 --2- 172.21.15.114:0/3430052476 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f25cc101aa0 0x7f25cc105a90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:28:40.973 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:28:40.972+0000 7f25d31c6640 1 --2- 172.21.15.114:0/3430052476 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f25cc1010f0 0x7f25cc1014d0 unknown :-1 s=CLOSED pgs=556 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:28:40.973 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:28:40.972+0000 7f25d31c6640 1 -- 172.21.15.114:0/3430052476 >> 172.21.15.114:0/3430052476 conn(0x7f25cc077fd0 msgr2=0x7f25cc0783e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T10:28:40.973 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:28:40.972+0000 7f25d31c6640 1 -- 172.21.15.114:0/3430052476 shutdown_connections 2024-04-23T10:28:40.973 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:28:40.972+0000 7f25d31c6640 1 -- 172.21.15.114:0/3430052476 wait complete. 2024-04-23T10:28:40.973 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:28:40.972+0000 7f25d31c6640 1 Processor -- start 2024-04-23T10:28:40.973 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:28:40.972+0000 7f25d31c6640 1 -- start start 2024-04-23T10:28:40.973 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:28:40.972+0000 7f25d31c6640 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f25cc1010f0 0x7f25cc102b30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T10:28:40.973 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:28:40.972+0000 7f25d31c6640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f25cc106170 0x7f25cc103070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T10:28:40.973 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:28:40.972+0000 7f25d31c6640 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f25cc1035b0 0x7f25cc1a6860 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T10:28:40.973 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:28:40.972+0000 7f25d31c6640 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x7f25cc1110b0 con 0x7f25cc106170 2024-04-23T10:28:40.973 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:28:40.972+0000 7f25d31c6640 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 -- 0x7f25cc110f30 con 0x7f25cc1010f0 2024-04-23T10:28:40.974 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:28:40.972+0000 7f25d31c6640 1 -- --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_getmap magic: 0 -- 0x7f25cc111230 con 0x7f25cc1035b0 2024-04-23T10:28:40.974 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:28:40.972+0000 7f25cbfff640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f25cc106170 0x7f25cc103070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T10:28:40.974 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:28:40.972+0000 7f25d0f62640 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f25cc1010f0 0x7f25cc102b30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T10:28:40.974 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:28:40.972+0000 7f25d0f62640 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f25cc1010f0 0x7f25cc102b30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.119:3300/0 says I am v2:172.21.15.114:44166/0 (socket says 172.21.15.114:44166) 2024-04-23T10:28:40.974 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:28:40.972+0000 7f25cbfff640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f25cc106170 0x7f25cc103070 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.114:44242/0 (socket says 172.21.15.114:44242) 2024-04-23T10:28:40.974 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:28:40.973+0000 7f25d0f62640 1 -- 172.21.15.114:0/2162592299 learned_addr learned my addr 172.21.15.114:0/2162592299 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-04-23T10:28:40.974 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:28:40.973+0000 7f25d1763640 1 --2- 172.21.15.114:0/2162592299 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f25cc1035b0 0x7f25cc1a6860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T10:28:40.974 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:28:40.973+0000 7f25cbfff640 1 -- 172.21.15.114:0/2162592299 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f25cc1035b0 msgr2=0x7f25cc1a6860 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T10:28:40.974 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:28:40.973+0000 7f25cbfff640 1 --2- 172.21.15.114:0/2162592299 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f25cc1035b0 0x7f25cc1a6860 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:28:40.974 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:28:40.973+0000 7f25cbfff640 1 -- 172.21.15.114:0/2162592299 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f25cc1010f0 msgr2=0x7f25cc102b30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T10:28:40.974 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:28:40.973+0000 7f25cbfff640 1 --2- 172.21.15.114:0/2162592299 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f25cc1010f0 0x7f25cc102b30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:28:40.974 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:28:40.973+0000 7f25cbfff640 1 -- 172.21.15.114:0/2162592299 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f25cc1a6f60 con 0x7f25cc106170 2024-04-23T10:28:40.974 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:28:40.973+0000 7f25d1763640 1 --2- 172.21.15.114:0/2162592299 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f25cc1035b0 0x7f25cc1a6860 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-23T10:28:40.975 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:28:40.973+0000 7f25d0f62640 1 --2- 172.21.15.114:0/2162592299 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f25cc1010f0 0x7f25cc102b30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-23T10:28:40.975 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:28:40.973+0000 7f25cbfff640 1 --2- 172.21.15.114:0/2162592299 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f25cc106170 0x7f25cc103070 secure :-1 s=READY pgs=557 cs=0 l=1 rev1=1 crypto rx=0x7f25b800e9c0 tx=0x7f25b800ee80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T10:28:40.975 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:28:40.973+0000 7f25c9ffb640 1 -- 172.21.15.114:0/2162592299 <== mon.0 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f25b800ccc0 con 0x7f25cc106170 2024-04-23T10:28:40.975 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:28:40.973+0000 7f25c9ffb640 1 -- 172.21.15.114:0/2162592299 <== mon.0 v2:172.21.15.114:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f25b80108e0 con 0x7f25cc106170 2024-04-23T10:28:40.975 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:28:40.973+0000 7f25d31c6640 1 -- 172.21.15.114:0/2162592299 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f25cc1a7250 con 0x7f25cc106170 2024-04-23T10:28:40.975 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:28:40.973+0000 7f25c9ffb640 1 -- 172.21.15.114:0/2162592299 <== mon.0 v2:172.21.15.114:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f25b80187c0 con 0x7f25cc106170 2024-04-23T10:28:40.975 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:28:40.973+0000 7f25d31c6640 1 -- 172.21.15.114:0/2162592299 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f25cc1a77a0 con 0x7f25cc106170 2024-04-23T10:28:40.982 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:28:40.975+0000 7f25c9ffb640 1 -- 172.21.15.114:0/2162592299 <== mon.0 v2:172.21.15.114:3300/0 4 ==== mgrmap(e 26) ==== 138778+0+0 (secure 0 0 0) 0x7f25b8010430 con 0x7f25cc106170 2024-04-23T10:28:40.982 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:28:40.976+0000 7f25d5fb8740 1 Processor -- start 2024-04-23T10:28:40.982 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:28:40.976+0000 7f25d5fb8740 1 -- start start 2024-04-23T10:28:40.982 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:28:40.976+0000 7f25c9ffb640 1 --2- 172.21.15.114:0/2162592299 >> [v2:172.21.15.114:6800/1395693466,v1:172.21.15.114:6801/1395693466] conn(0x7f25bc0a3df0 0x7f25bc0a62b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T10:28:40.982 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:28:40.976+0000 7f25d5fb8740 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x5594bcda1620 0x5594bcda1a00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T10:28:40.982 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:28:40.976+0000 7f25d5fb8740 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x5594bcda1f40 0x5594bcdac2c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T10:28:40.982 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:28:40.976+0000 7f25d5fb8740 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x5594bcdac800 0x5594bcdaebf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T10:28:40.982 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:28:40.976+0000 7f25d5fb8740 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x5594bcd06970 con 0x5594bcdac800 2024-04-23T10:28:40.982 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:28:40.976+0000 7f25d5fb8740 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 -- 0x5594bcca34a0 con 0x5594bcda1f40 2024-04-23T10:28:40.982 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:28:40.976+0000 7f25d5fb8740 1 -- --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_getmap magic: 0 -- 0x5594bcd02930 con 0x5594bcda1620 2024-04-23T10:28:40.982 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:28:40.976+0000 7f25d1763640 1 --2- 172.21.15.114:0/2162592299 >> [v2:172.21.15.114:6800/1395693466,v1:172.21.15.114:6801/1395693466] conn(0x7f25bc0a3df0 0x7f25bc0a62b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T10:28:40.983 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:28:40.976+0000 7f25d0f62640 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x5594bcda1620 0x5594bcda1a00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T10:28:40.983 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:28:40.976+0000 7f25cbfff640 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x5594bcda1f40 0x5594bcdac2c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T10:28:40.983 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:28:40.976+0000 7f25d0f62640 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x5594bcda1620 0x5594bcda1a00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.204:3300/0 says I am v2:172.21.15.114:55458/0 (socket says 172.21.15.114:55458) 2024-04-23T10:28:40.983 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:28:40.976+0000 7f25d0f62640 1 -- 172.21.15.114:0/4155909742 learned_addr learned my addr 172.21.15.114:0/4155909742 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-04-23T10:28:40.983 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:28:40.976+0000 7f25d1763640 1 --2- 172.21.15.114:0/4155909742 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x5594bcdac800 0x5594bcdaebf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T10:28:40.983 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:28:40.977+0000 7f25c9ffb640 1 -- 172.21.15.114:0/2162592299 <== mon.0 v2:172.21.15.114:3300/0 5 ==== osd_map(92..92 src has 1..92) ==== 8585+0+0 (secure 0 0 0) 0x7f25b80d0c20 con 0x7f25cc106170 2024-04-23T10:28:40.983 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:28:40.977+0000 7f25cbfff640 1 -- 172.21.15.114:0/4155909742 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x5594bcda1620 msgr2=0x5594bcda1a00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T10:28:40.983 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:28:40.977+0000 7f25cbfff640 1 --2- 172.21.15.114:0/4155909742 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x5594bcda1620 0x5594bcda1a00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:28:40.983 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:28:40.977+0000 7f25cbfff640 1 -- 172.21.15.114:0/4155909742 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x5594bcdac800 msgr2=0x5594bcdaebf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T10:28:40.983 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:28:40.977+0000 7f25cbfff640 1 --2- 172.21.15.114:0/4155909742 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x5594bcdac800 0x5594bcdaebf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:28:40.983 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:28:40.977+0000 7f25cbfff640 1 -- 172.21.15.114:0/4155909742 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5594bca19210 con 0x5594bcda1f40 2024-04-23T10:28:40.983 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:28:40.977+0000 7f25d0f62640 1 --2- 172.21.15.114:0/4155909742 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x5594bcda1620 0x5594bcda1a00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-23T10:28:40.983 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:28:40.977+0000 7f25d1763640 1 --2- 172.21.15.114:0/2162592299 >> [v2:172.21.15.114:6800/1395693466,v1:172.21.15.114:6801/1395693466] conn(0x7f25bc0a3df0 0x7f25bc0a62b0 secure :-1 s=READY pgs=78 cs=0 l=1 rev1=1 crypto rx=0x7f25c4009370 tx=0x7f25c4009e20 comp rx=0 tx=0).ready entity=mgr.15291 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T10:28:40.983 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:28:40.977+0000 7f25cbfff640 1 --2- 172.21.15.114:0/4155909742 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x5594bcda1f40 0x5594bcdac2c0 secure :-1 s=READY pgs=66 cs=0 l=1 rev1=1 crypto rx=0x7f25b80ce2d0 tx=0x7f25b801f000 comp rx=0 tx=0).ready entity=mon.2 client_cookie=de6390d5bbe8d477 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T10:28:40.984 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:28:40.977+0000 7f25b6ffd640 1 -- 172.21.15.114:0/4155909742 <== mon.2 v2:172.21.15.119:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f25b8026040 con 0x5594bcda1f40 2024-04-23T10:28:40.984 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:28:40.977+0000 7f25b6ffd640 1 -- 172.21.15.114:0/4155909742 <== mon.2 v2:172.21.15.119:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f25b8010720 con 0x5594bcda1f40 2024-04-23T10:28:40.984 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:28:40.977+0000 7f25b6ffd640 1 -- 172.21.15.114:0/4155909742 <== mon.2 v2:172.21.15.119:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f25b809e080 con 0x5594bcda1f40 2024-04-23T10:28:40.984 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:28:40.978+0000 7f25d5fb8740 1 -- 172.21.15.114:0/4155909742 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x5594bcda1f40 msgr2=0x5594bcdac2c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T10:28:40.984 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:28:40.978+0000 7f25d5fb8740 1 --2- 172.21.15.114:0/4155909742 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x5594bcda1f40 0x5594bcdac2c0 secure :-1 s=READY pgs=66 cs=0 l=1 rev1=1 crypto rx=0x7f25b80ce2d0 tx=0x7f25b801f000 comp rx=0 tx=0).stop 2024-04-23T10:28:40.984 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:28:40.978+0000 7f25d5fb8740 1 -- 172.21.15.114:0/4155909742 shutdown_connections 2024-04-23T10:28:40.984 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:28:40.978+0000 7f25d5fb8740 1 --2- 172.21.15.114:0/4155909742 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x5594bcdac800 0x5594bcdaebf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:28:40.984 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:28:40.978+0000 7f25d5fb8740 1 --2- 172.21.15.114:0/4155909742 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x5594bcda1f40 0x5594bcdac2c0 unknown :-1 s=CLOSED pgs=66 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:28:40.984 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:28:40.978+0000 7f25d5fb8740 1 --2- 172.21.15.114:0/4155909742 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x5594bcda1620 0x5594bcda1a00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:28:40.984 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:28:40.978+0000 7f25d5fb8740 1 -- 172.21.15.114:0/4155909742 >> 172.21.15.114:0/4155909742 conn(0x5594bcd9db30 msgr2=0x5594bcd9df10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T10:28:40.984 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:28:40.978+0000 7f25d5fb8740 1 -- 172.21.15.114:0/4155909742 shutdown_connections 2024-04-23T10:28:40.984 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:28:40.978+0000 7f25d5fb8740 1 -- 172.21.15.114:0/4155909742 wait complete. 2024-04-23T10:28:40.984 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:28:40.978+0000 7f25d5fb8740 1 -- start start 2024-04-23T10:28:40.984 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:28:40.978+0000 7f25d5fb8740 1 Processor -- start 2024-04-23T10:28:40.984 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:28:40.979+0000 7f25d5fb8740 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x5594bcda1620 0x5594bcdac220 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T10:28:40.985 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:28:40.979+0000 7f25d5fb8740 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x5594bcda1f40 0x5594bcda9510 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T10:28:40.985 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:28:40.979+0000 7f25d5fb8740 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x5594bcdac800 0x5594bcda9b20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T10:28:40.985 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:28:40.979+0000 7f25d5fb8740 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x5594bcdaf370 con 0x5594bcda1620 2024-04-23T10:28:40.985 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:28:40.979+0000 7f25d5fb8740 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 -- 0x5594bcca10b0 con 0x5594bcda1f40 2024-04-23T10:28:40.985 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:28:40.979+0000 7f25d5fb8740 1 -- --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_getmap magic: 0 -- 0x5594bcdaf4f0 con 0x5594bcdac800 2024-04-23T10:28:40.985 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:28:40.979+0000 7f25d5fb8740 10 client.0 _ll_register_callbacks cb 0x5594bcc8ed30 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-23T10:28:40.985 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:28:40.979+0000 7f25d0f62640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x5594bcda1620 0x5594bcdac220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T10:28:40.985 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:28:40.979+0000 7f25d0f62640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x5594bcda1620 0x5594bcdac220 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.114:44266/0 (socket says 172.21.15.114:44266) 2024-04-23T10:28:40.985 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:28:40.979+0000 7f25d0f62640 1 -- 172.21.15.114:0/1688897291 learned_addr learned my addr 172.21.15.114:0/1688897291 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-04-23T10:28:40.985 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:28:40.979+0000 7f25cbfff640 1 --2- 172.21.15.114:0/1688897291 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x5594bcda1f40 0x5594bcda9510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T10:28:40.985 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:28:40.979+0000 7f25d1763640 1 --2- 172.21.15.114:0/1688897291 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x5594bcdac800 0x5594bcda9b20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T10:28:40.985 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:28:40.979+0000 7f25cbfff640 1 -- 172.21.15.114:0/1688897291 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x5594bcdac800 msgr2=0x5594bcda9b20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T10:28:40.985 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:28:40.979+0000 7f25cbfff640 1 --2- 172.21.15.114:0/1688897291 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x5594bcdac800 0x5594bcda9b20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:28:40.985 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:28:40.979+0000 7f25cbfff640 1 -- 172.21.15.114:0/1688897291 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x5594bcda1620 msgr2=0x5594bcdac220 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T10:28:40.986 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:28:40.979+0000 7f25cbfff640 1 --2- 172.21.15.114:0/1688897291 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x5594bcda1620 0x5594bcdac220 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:28:40.986 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:28:40.979+0000 7f25cbfff640 1 -- 172.21.15.114:0/1688897291 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5594bcae5620 con 0x5594bcda1f40 2024-04-23T10:28:40.986 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:28:40.979+0000 7f25d0f62640 1 --2- 172.21.15.114:0/1688897291 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x5594bcda1620 0x5594bcdac220 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-23T10:28:40.986 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:28:40.979+0000 7f25cbfff640 1 --2- 172.21.15.114:0/1688897291 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x5594bcda1f40 0x5594bcda9510 secure :-1 s=READY pgs=67 cs=0 l=1 rev1=1 crypto rx=0x5594bcda2530 tx=0x7f25b80ce270 comp rx=0 tx=0).ready entity=mon.2 client_cookie=a3c14116ba2e092a server_cookie=0 in_seq=0 out_seq=0 2024-04-23T10:28:40.986 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:28:40.979+0000 7f25abfff640 10 client.0 ms_handle_connect on v2:172.21.15.119:3300/0 2024-04-23T10:28:40.986 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:28:40.979+0000 7f25abfff640 1 -- 172.21.15.114:0/1688897291 <== mon.2 v2:172.21.15.119:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f25b80b5020 con 0x5594bcda1f40 2024-04-23T10:28:40.986 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:28:40.979+0000 7f25d5fb8740 1 -- 172.21.15.114:0/1688897291 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x5594bcd14390 con 0x5594bcda1f40 2024-04-23T10:28:40.986 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:28:40.979+0000 7f25abfff640 1 -- 172.21.15.114:0/1688897291 <== mon.2 v2:172.21.15.119:3300/0 2 ==== osd_map(92..92 src has 1..92) ==== 8585+0+0 (secure 0 0 0) 0x7f25b80bd0b0 con 0x5594bcda1f40 2024-04-23T10:28:40.986 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:28:40.979+0000 7f25abfff640 1 -- 172.21.15.114:0/1688897291 <== mon.2 v2:172.21.15.119:3300/0 3 ==== mon_get_version_reply(handle=1 version=23) ==== 24+0+0 (secure 0 0 0) 0x7f25b809cb20 con 0x5594bcda1f40 2024-04-23T10:28:40.986 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:28:40.980+0000 7f25d5fb8740 10 client.25334 fetch_fsmap learned FSMap version 23 2024-04-23T10:28:40.986 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:28:40.980+0000 7f25d5fb8740 1 -- 172.21.15.114:0/1688897291 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({config=0+,fsmap=23,monmap=4+}) -- 0x5594bcdbb360 con 0x5594bcda1f40 2024-04-23T10:28:40.986 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:28:40.980+0000 7f25abfff640 1 -- 172.21.15.114:0/1688897291 <== mon.2 v2:172.21.15.119:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x5594bcd14390 con 0x5594bcda1f40 2024-04-23T10:28:40.986 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:28:40.980+0000 7f25abfff640 1 -- 172.21.15.114:0/1688897291 <== mon.2 v2:172.21.15.119:3300/0 5 ==== fsmap(e 23) ==== 8350+0+0 (secure 0 0 0) 0x7f25b80bbce0 con 0x5594bcda1f40 2024-04-23T10:28:40.986 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:28:40.980+0000 7f25d5fb8740 10 client.25334 fetch_fsmap finished waiting for FSMap version 23 2024-04-23T10:28:40.986 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:28:40.980+0000 7f25d5fb8740 10 client.25334 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-04-23T10:28:40.987 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:28:40.980+0000 7f25d5fb8740 20 client.25334 populate_metadata read hostname 'smithi114' 2024-04-23T10:28:40.987 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:28:40.980+0000 7f25d5fb8740 1 --2- 172.21.15.114:0/1688897291 >> [v2:172.21.15.119:6838/937297340,v1:172.21.15.119:6839/937297340] conn(0x5594bcdbc6b0 0x5594bcdbeaa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T10:28:40.987 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:28:40.980+0000 7f25d5fb8740 4 client.25334 mds_command: new command op to 24545 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-23T10:28:40.987 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:28:40.980+0000 7f25d5fb8740 1 -- 172.21.15.114:0/1688897291 --> [v2:172.21.15.119:6838/937297340,v1:172.21.15.119:6839/937297340] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x5594bcd0e200 con 0x5594bcdbc6b0 2024-04-23T10:28:40.987 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:28:40.980+0000 7f25d0f62640 1 --2- 172.21.15.114:0/1688897291 >> [v2:172.21.15.119:6838/937297340,v1:172.21.15.119:6839/937297340] conn(0x5594bcdbc6b0 0x5594bcdbeaa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T10:28:40.987 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:28:40.980+0000 7f25abfff640 10 client.25334 ms_handle_connect on v2:172.21.15.119:6838/937297340 2024-04-23T10:28:40.987 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:28:40.980+0000 7f25d0f62640 1 --2- 172.21.15.114:0/1688897291 >> [v2:172.21.15.119:6838/937297340,v1:172.21.15.119:6839/937297340] conn(0x5594bcdbc6b0 0x5594bcdbeaa0 crc :-1 s=READY pgs=64 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=b0385251426a4202 server_cookie=cb8c22fe8c6dba58 in_seq=0 out_seq=0 2024-04-23T10:28:40.987 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:28:40.981+0000 7f25abfff640 1 -- 172.21.15.114:0/1688897291 <== mds.0 v2:172.21.15.119:6838/937297340 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x5594bcd0e200 con 0x5594bcdbc6b0 2024-04-23T10:28:40.987 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:28:40.981+0000 7f25abfff640 10 client.25334 handle_command_reply: tid=0 2024-04-23T10:28:40.987 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:28:40.981+0000 7f25d5fb8740 1 client.25334 shutdown 2024-04-23T10:28:40.987 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:28:40.981+0000 7f25d5fb8740 1 -- 172.21.15.114:0/1688897291 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x5594bcda1f40 msgr2=0x5594bcda9510 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T10:28:40.987 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:28:40.981+0000 7f25d5fb8740 1 --2- 172.21.15.114:0/1688897291 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x5594bcda1f40 0x5594bcda9510 secure :-1 s=READY pgs=67 cs=0 l=1 rev1=1 crypto rx=0x5594bcda2530 tx=0x7f25b80ce270 comp rx=0 tx=0).stop 2024-04-23T10:28:40.987 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:28:40.981+0000 7f25d5fb8740 1 -- 172.21.15.114:0/1688897291 shutdown_connections 2024-04-23T10:28:40.987 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:28:40.981+0000 7f25d5fb8740 1 --2- 172.21.15.114:0/1688897291 >> [v2:172.21.15.119:6838/937297340,v1:172.21.15.119:6839/937297340] conn(0x5594bcdbc6b0 0x5594bcdbeaa0 crc :-1 s=READY pgs=64 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:28:40.987 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:28:40.981+0000 7f25d5fb8740 1 --2- 172.21.15.114:0/1688897291 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x5594bcdac800 0x5594bcda9b20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:28:40.988 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:28:40.981+0000 7f25d5fb8740 1 --2- 172.21.15.114:0/1688897291 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x5594bcda1f40 0x5594bcda9510 unknown :-1 s=CLOSED pgs=67 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:28:40.988 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:28:40.981+0000 7f25d5fb8740 1 --2- 172.21.15.114:0/1688897291 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x5594bcda1620 0x5594bcdac220 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:28:40.988 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:28:40.981+0000 7f25d5fb8740 1 -- 172.21.15.114:0/1688897291 >> 172.21.15.114:0/1688897291 conn(0x5594bcd9db30 msgr2=0x5594bcda38b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T10:28:40.988 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:28:40.981+0000 7f25abfff640 0 client.25334 ms_handle_reset on v2:172.21.15.119:6838/937297340 2024-04-23T10:28:40.988 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:28:40.981+0000 7f25d5fb8740 1 -- 172.21.15.114:0/1688897291 shutdown_connections 2024-04-23T10:28:40.988 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:28:40.981+0000 7f25d5fb8740 1 -- 172.21.15.114:0/1688897291 wait complete. 2024-04-23T10:28:40.988 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:28:40.981+0000 7f25d5fb8740 20 client.25334 trim_cache size 0 max 16384 2024-04-23T10:28:40.999 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:28:40.996+0000 7f25d5fb8740 1 Processor -- start 2024-04-23T10:28:40.999 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:28:40.996+0000 7f25d5fb8740 1 -- start start 2024-04-23T10:28:40.999 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:28:40.996+0000 7f25d5fb8740 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x5594bcda1620 0x5594bcd9d430 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T10:28:40.999 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:28:40.996+0000 7f25d5fb8740 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x5594bcda1f40 0x5594bcdabd50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T10:28:40.999 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:28:40.996+0000 7f25d5fb8740 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x5594bcdac800 0x5594bcdc4100 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T10:28:41.000 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:28:40.996+0000 7f25d5fb8740 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x7f25cc1b26d0 con 0x5594bcda1620 2024-04-23T10:28:41.000 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:28:40.996+0000 7f25d5fb8740 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 -- 0x7f25cc1aff70 con 0x5594bcdac800 2024-04-23T10:28:41.000 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:28:40.996+0000 7f25d5fb8740 1 -- --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_getmap magic: 0 -- 0x7f25cc1addc0 con 0x5594bcda1f40 2024-04-23T10:28:41.000 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:28:40.996+0000 7f25d0f62640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x5594bcda1620 0x5594bcd9d430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T10:28:41.000 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:28:40.996+0000 7f25d0f62640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x5594bcda1620 0x5594bcd9d430 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.114:44268/0 (socket says 172.21.15.114:44268) 2024-04-23T10:28:41.000 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:28:40.996+0000 7f25d0f62640 1 -- 172.21.15.114:0/2548508748 learned_addr learned my addr 172.21.15.114:0/2548508748 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-04-23T10:28:41.000 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:28:40.996+0000 7f25d1763640 1 --2- 172.21.15.114:0/2548508748 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x5594bcdac800 0x5594bcdc4100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T10:28:41.000 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:28:40.996+0000 7f25d0f62640 1 -- 172.21.15.114:0/2548508748 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x5594bcda1f40 msgr2=0x5594bcdabd50 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2024-04-23T10:28:41.000 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:28:40.996+0000 7f25d0f62640 1 --2- 172.21.15.114:0/2548508748 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x5594bcda1f40 0x5594bcdabd50 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:28:41.000 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:28:40.996+0000 7f25d0f62640 1 -- 172.21.15.114:0/2548508748 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x5594bcdac800 msgr2=0x5594bcdc4100 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T10:28:41.000 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:28:40.996+0000 7f25d0f62640 1 --2- 172.21.15.114:0/2548508748 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x5594bcdac800 0x5594bcdc4100 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:28:41.000 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:28:40.996+0000 7f25d0f62640 1 -- 172.21.15.114:0/2548508748 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5594bcd99500 con 0x5594bcda1620 2024-04-23T10:28:41.001 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:28:40.996+0000 7f25d1763640 1 --2- 172.21.15.114:0/2548508748 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x5594bcdac800 0x5594bcdc4100 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-23T10:28:41.001 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:28:40.996+0000 7f25d0f62640 1 --2- 172.21.15.114:0/2548508748 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x5594bcda1620 0x5594bcd9d430 secure :-1 s=READY pgs=558 cs=0 l=1 rev1=1 crypto rx=0x7f25c0009a20 tx=0x7f25c0009260 comp rx=0 tx=0).ready entity=mon.0 client_cookie=50b3e3a3617dd124 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T10:28:41.001 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:28:40.997+0000 7f25b67fc640 1 -- 172.21.15.114:0/2548508748 <== mon.0 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f25c0035040 con 0x5594bcda1620 2024-04-23T10:28:41.001 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:28:40.997+0000 7f25b67fc640 1 -- 172.21.15.114:0/2548508748 <== mon.0 v2:172.21.15.114:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f25c0005040 con 0x5594bcda1620 2024-04-23T10:28:41.001 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:28:40.997+0000 7f25b67fc640 1 -- 172.21.15.114:0/2548508748 <== mon.0 v2:172.21.15.114:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f25c0004af0 con 0x5594bcda1620 2024-04-23T10:28:41.001 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:28:40.997+0000 7f25d5fb8740 1 -- 172.21.15.114:0/2548508748 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x5594bcda1620 msgr2=0x5594bcd9d430 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T10:28:41.001 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:28:40.997+0000 7f25d5fb8740 1 --2- 172.21.15.114:0/2548508748 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x5594bcda1620 0x5594bcd9d430 secure :-1 s=READY pgs=558 cs=0 l=1 rev1=1 crypto rx=0x7f25c0009a20 tx=0x7f25c0009260 comp rx=0 tx=0).stop 2024-04-23T10:28:41.001 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:28:40.997+0000 7f25d5fb8740 1 -- 172.21.15.114:0/2548508748 shutdown_connections 2024-04-23T10:28:41.001 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:28:40.997+0000 7f25d5fb8740 1 --2- 172.21.15.114:0/2548508748 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x5594bcdac800 0x5594bcdc4100 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:28:41.001 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:28:40.997+0000 7f25d5fb8740 1 --2- 172.21.15.114:0/2548508748 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x5594bcda1f40 0x5594bcdabd50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:28:41.001 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:28:40.997+0000 7f25d5fb8740 1 --2- 172.21.15.114:0/2548508748 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x5594bcda1620 0x5594bcd9d430 unknown :-1 s=CLOSED pgs=558 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:28:41.001 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:28:40.997+0000 7f25d5fb8740 1 -- 172.21.15.114:0/2548508748 >> 172.21.15.114:0/2548508748 conn(0x5594bcd9db30 msgr2=0x5594bcda4af0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T10:28:41.001 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:28:40.997+0000 7f25d5fb8740 1 -- 172.21.15.114:0/2548508748 shutdown_connections 2024-04-23T10:28:41.002 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:28:40.997+0000 7f25d5fb8740 1 -- 172.21.15.114:0/2548508748 wait complete. 2024-04-23T10:28:41.002 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:28:40.998+0000 7f25d5fb8740 1 -- start start 2024-04-23T10:28:41.002 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:28:40.998+0000 7f25d5fb8740 1 Processor -- start 2024-04-23T10:28:41.002 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:28:40.998+0000 7f25d5fb8740 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x5594bcda1620 0x5594bcda61b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T10:28:41.002 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:28:40.998+0000 7f25d5fb8740 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x5594bcda1f40 0x5594bcda66f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T10:28:41.002 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:28:40.998+0000 7f25d5fb8740 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x5594bcdac800 0x5594bcda6c30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T10:28:41.002 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:28:40.998+0000 7f25d5fb8740 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x7f25cc1ae430 con 0x5594bcda1620 2024-04-23T10:28:41.002 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:28:40.998+0000 7f25d5fb8740 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 -- 0x7f25cc1af8a0 con 0x5594bcda1f40 2024-04-23T10:28:41.002 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:28:40.998+0000 7f25d5fb8740 1 -- --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_getmap magic: 0 -- 0x7f25cc1aeb00 con 0x5594bcdac800 2024-04-23T10:28:41.002 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:28:40.998+0000 7f25d5fb8740 10 client.0 _ll_register_callbacks cb 0x7f25b809c660 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-23T10:28:41.002 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:28:40.998+0000 7f25d0f62640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x5594bcda1620 0x5594bcda61b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T10:28:41.002 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:28:40.998+0000 7f25d0f62640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x5594bcda1620 0x5594bcda61b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.114:44276/0 (socket says 172.21.15.114:44276) 2024-04-23T10:28:41.002 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:28:40.998+0000 7f25cbfff640 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x5594bcda1f40 0x5594bcda66f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T10:28:41.003 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:28:40.998+0000 7f25d0f62640 1 -- 172.21.15.114:0/2824630153 learned_addr learned my addr 172.21.15.114:0/2824630153 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-04-23T10:28:41.003 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:28:40.998+0000 7f25d1763640 1 --2- 172.21.15.114:0/2824630153 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x5594bcdac800 0x5594bcda6c30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T10:28:41.003 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:28:40.998+0000 7f25d0f62640 1 -- 172.21.15.114:0/2824630153 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x5594bcdac800 msgr2=0x5594bcda6c30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T10:28:41.003 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:28:40.999+0000 7f25d0f62640 1 --2- 172.21.15.114:0/2824630153 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x5594bcdac800 0x5594bcda6c30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:28:41.003 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:28:40.999+0000 7f25d0f62640 1 -- 172.21.15.114:0/2824630153 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x5594bcda1f40 msgr2=0x5594bcda66f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T10:28:41.003 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:28:40.999+0000 7f25d0f62640 1 --2- 172.21.15.114:0/2824630153 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x5594bcda1f40 0x5594bcda66f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:28:41.003 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:28:40.999+0000 7f25d0f62640 1 -- 172.21.15.114:0/2824630153 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5594bce219a0 con 0x5594bcda1620 2024-04-23T10:28:41.003 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:28:40.999+0000 7f25cbfff640 1 --2- 172.21.15.114:0/2824630153 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x5594bcda1f40 0x5594bcda66f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-23T10:28:41.003 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:28:40.999+0000 7f25d0f62640 1 --2- 172.21.15.114:0/2824630153 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x5594bcda1620 0x5594bcda61b0 secure :-1 s=READY pgs=559 cs=0 l=1 rev1=1 crypto rx=0x7f25c0032fd0 tx=0x7f25c00051f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3df17475f664cef2 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T10:28:41.003 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:28:40.999+0000 7f25abfff640 10 client.0 ms_handle_connect on v2:172.21.15.114:3300/0 2024-04-23T10:28:41.003 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:28:40.999+0000 7f25abfff640 1 -- 172.21.15.114:0/2824630153 <== mon.0 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f25c0056070 con 0x5594bcda1620 2024-04-23T10:28:41.003 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:28:40.999+0000 7f25abfff640 1 -- 172.21.15.114:0/2824630153 <== mon.0 v2:172.21.15.114:3300/0 2 ==== osd_map(92..92 src has 1..92) ==== 8585+0+0 (secure 0 0 0) 0x7f25c0048630 con 0x5594bcda1620 2024-04-23T10:28:41.003 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:28:40.999+0000 7f25d5fb8740 1 -- 172.21.15.114:0/2824630153 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f25c0005040 con 0x5594bcda1620 2024-04-23T10:28:41.003 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:28:40.999+0000 7f25abfff640 1 -- 172.21.15.114:0/2824630153 <== mon.0 v2:172.21.15.114:3300/0 3 ==== mon_get_version_reply(handle=1 version=23) ==== 24+0+0 (secure 0 0 0) 0x7f25c0062050 con 0x5594bcda1620 2024-04-23T10:28:41.004 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:28:40.999+0000 7f25d5fb8740 10 client.15816 fetch_fsmap learned FSMap version 23 2024-04-23T10:28:41.004 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:28:40.999+0000 7f25d5fb8740 1 -- 172.21.15.114:0/2824630153 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({config=0+,fsmap=23,monmap=4+}) -- 0x5594bce22c90 con 0x5594bcda1620 2024-04-23T10:28:41.004 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:28:40.999+0000 7f25abfff640 1 -- 172.21.15.114:0/2824630153 <== mon.0 v2:172.21.15.114:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f25c0005040 con 0x5594bcda1620 2024-04-23T10:28:41.004 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:28:40.999+0000 7f25abfff640 1 -- 172.21.15.114:0/2824630153 <== mon.0 v2:172.21.15.114:3300/0 5 ==== fsmap(e 23) ==== 8350+0+0 (secure 0 0 0) 0x7f25c0003710 con 0x5594bcda1620 2024-04-23T10:28:41.004 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:28:40.999+0000 7f25d5fb8740 10 client.15816 fetch_fsmap finished waiting for FSMap version 23 2024-04-23T10:28:41.004 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:28:40.999+0000 7f25d5fb8740 10 client.15816 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-04-23T10:28:41.004 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:28:40.999+0000 7f25d5fb8740 20 client.15816 populate_metadata read hostname 'smithi114' 2024-04-23T10:28:41.004 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:28:40.999+0000 7f25d5fb8740 1 --2- 172.21.15.114:0/2824630153 >> [v2:172.21.15.119:6838/937297340,v1:172.21.15.119:6839/937297340] conn(0x5594bce233b0 0x5594bce257d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T10:28:41.004 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:28:40.999+0000 7f25d5fb8740 4 client.15816 mds_command: new command op to 24545 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-04-23T10:28:41.004 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:28:40.999+0000 7f25d5fb8740 1 -- 172.21.15.114:0/2824630153 --> [v2:172.21.15.119:6838/937297340,v1:172.21.15.119:6839/937297340] -- command(tid 0: {"prefix": "flush journal"}) -- 0x5594bcc8a5f0 con 0x5594bce233b0 2024-04-23T10:28:41.004 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:28:41.000+0000 7f25cbfff640 1 --2- 172.21.15.114:0/2824630153 >> [v2:172.21.15.119:6838/937297340,v1:172.21.15.119:6839/937297340] conn(0x5594bce233b0 0x5594bce257d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T10:28:41.004 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:28:41.000+0000 7f25abfff640 10 client.15816 ms_handle_connect on v2:172.21.15.119:6838/937297340 2024-04-23T10:28:41.004 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:28:41.000+0000 7f25cbfff640 1 --2- 172.21.15.114:0/2824630153 >> [v2:172.21.15.119:6838/937297340,v1:172.21.15.119:6839/937297340] conn(0x5594bce233b0 0x5594bce257d0 crc :-1 s=READY pgs=65 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=90e578e14f32b5db server_cookie=984e4a5396ac9e68 in_seq=0 out_seq=0 2024-04-23T10:28:41.081 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:28:41.082+0000 7f25abfff640 1 -- 172.21.15.114:0/2824630153 <== mds.0 v2:172.21.15.119:6838/937297340 1 ==== command_reply(tid 0: 0 ) ==== 8+0+44 (crc 0 0 0) 0x5594bcc8a5f0 con 0x5594bce233b0 2024-04-23T10:28:41.082 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:28:41.082+0000 7f25abfff640 10 client.15816 handle_command_reply: tid=0 2024-04-23T10:28:41.082 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:28:41.082+0000 7f25d5fb8740 1 client.15816 shutdown 2024-04-23T10:28:41.082 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:28:41.082+0000 7f25d5fb8740 1 -- 172.21.15.114:0/2824630153 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x5594bcda1620 msgr2=0x5594bcda61b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T10:28:41.082 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:28:41.082+0000 7f25d5fb8740 1 --2- 172.21.15.114:0/2824630153 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x5594bcda1620 0x5594bcda61b0 secure :-1 s=READY pgs=559 cs=0 l=1 rev1=1 crypto rx=0x7f25c0032fd0 tx=0x7f25c00051f0 comp rx=0 tx=0).stop 2024-04-23T10:28:41.082 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:28:41.082+0000 7f25d5fb8740 1 -- 172.21.15.114:0/2824630153 shutdown_connections 2024-04-23T10:28:41.082 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:28:41.082+0000 7f25d5fb8740 1 --2- 172.21.15.114:0/2824630153 >> [v2:172.21.15.119:6838/937297340,v1:172.21.15.119:6839/937297340] conn(0x5594bce233b0 0x5594bce257d0 crc :-1 s=READY pgs=65 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:28:41.082 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:28:41.082+0000 7f25d5fb8740 1 --2- 172.21.15.114:0/2824630153 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x5594bcdac800 0x5594bcda6c30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:28:41.082 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:28:41.082+0000 7f25d5fb8740 1 --2- 172.21.15.114:0/2824630153 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x5594bcda1f40 0x5594bcda66f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:28:41.082 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:28:41.082+0000 7f25abfff640 0 client.15816 ms_handle_reset on v2:172.21.15.119:6838/937297340 2024-04-23T10:28:41.082 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:28:41.082+0000 7f25d5fb8740 1 --2- 172.21.15.114:0/2824630153 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x5594bcda1620 0x5594bcda61b0 unknown :-1 s=CLOSED pgs=559 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:28:41.082 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:28:41.082+0000 7f25d5fb8740 1 -- 172.21.15.114:0/2824630153 >> 172.21.15.114:0/2824630153 conn(0x5594bcd9db30 msgr2=0x5594bcda4120 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T10:28:41.082 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:28:41.082+0000 7f25d5fb8740 1 -- 172.21.15.114:0/2824630153 shutdown_connections 2024-04-23T10:28:41.082 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:28:41.082+0000 7f25d5fb8740 1 -- 172.21.15.114:0/2824630153 wait complete. 2024-04-23T10:28:41.082 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:28:41.083+0000 7f25d5fb8740 20 client.15816 trim_cache size 0 max 16384 2024-04-23T10:28:41.083 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stdout:{ 2024-04-23T10:28:41.083 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stdout: "message": "", 2024-04-23T10:28:41.083 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stdout: "return_code": 0 2024-04-23T10:28:41.083 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stdout:} 2024-04-23T10:28:41.084 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:28:41.083+0000 7f25d31c6640 1 -- 172.21.15.114:0/2162592299 >> [v2:172.21.15.114:6800/1395693466,v1:172.21.15.114:6801/1395693466] conn(0x7f25bc0a3df0 msgr2=0x7f25bc0a62b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T10:28:41.084 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:28:41.083+0000 7f25d31c6640 1 --2- 172.21.15.114:0/2162592299 >> [v2:172.21.15.114:6800/1395693466,v1:172.21.15.114:6801/1395693466] conn(0x7f25bc0a3df0 0x7f25bc0a62b0 secure :-1 s=READY pgs=78 cs=0 l=1 rev1=1 crypto rx=0x7f25c4009370 tx=0x7f25c4009e20 comp rx=0 tx=0).stop 2024-04-23T10:28:41.084 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:28:41.083+0000 7f25d31c6640 1 -- 172.21.15.114:0/2162592299 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f25cc106170 msgr2=0x7f25cc103070 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T10:28:41.084 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:28:41.083+0000 7f25d31c6640 1 --2- 172.21.15.114:0/2162592299 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f25cc106170 0x7f25cc103070 secure :-1 s=READY pgs=557 cs=0 l=1 rev1=1 crypto rx=0x7f25b800e9c0 tx=0x7f25b800ee80 comp rx=0 tx=0).stop 2024-04-23T10:28:41.084 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:28:41.083+0000 7f25d31c6640 1 -- 172.21.15.114:0/2162592299 shutdown_connections 2024-04-23T10:28:41.084 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:28:41.083+0000 7f25d31c6640 1 --2- 172.21.15.114:0/2162592299 >> [v2:172.21.15.114:6800/1395693466,v1:172.21.15.114:6801/1395693466] conn(0x7f25bc0a3df0 0x7f25bc0a62b0 unknown :-1 s=CLOSED pgs=78 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:28:41.084 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:28:41.083+0000 7f25d31c6640 1 --2- 172.21.15.114:0/2162592299 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f25cc1035b0 0x7f25cc1a6860 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:28:41.084 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:28:41.083+0000 7f25d31c6640 1 --2- 172.21.15.114:0/2162592299 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f25cc106170 0x7f25cc103070 unknown :-1 s=CLOSED pgs=557 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:28:41.084 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:28:41.083+0000 7f25d31c6640 1 --2- 172.21.15.114:0/2162592299 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f25cc1010f0 0x7f25cc102b30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:28:41.084 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:28:41.083+0000 7f25d31c6640 1 -- 172.21.15.114:0/2162592299 >> 172.21.15.114:0/2162592299 conn(0x7f25cc077fd0 msgr2=0x7f25cc104dd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T10:28:41.084 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:28:41.083+0000 7f25d31c6640 1 -- 172.21.15.114:0/2162592299 shutdown_connections 2024-04-23T10:28:41.084 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:28:41.083+0000 7f25d31c6640 1 -- 172.21.15.114:0/2162592299 wait complete. 2024-04-23T10:28:41.742 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 23 10:28:41 smithi119 ceph-mon[32608]: pgmap v65: 97 pgs: 97 active+clean; 45 MiB data, 630 MiB used, 1.0 TiB / 1.0 TiB avail; 2.6 MiB/s rd, 1.3 MiB/s wr, 264 op/s 2024-04-23T10:28:41.754 INFO:journalctl@ceph.mon.a.smithi114.stdout:Apr 23 10:28:41 smithi114 ceph-mon[29255]: pgmap v65: 97 pgs: 97 active+clean; 45 MiB data, 630 MiB used, 1.0 TiB / 1.0 TiB avail; 2.6 MiB/s rd, 1.3 MiB/s wr, 264 op/s 2024-04-23T10:28:41.827 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 23 10:28:41 smithi204 ceph-mon[32649]: pgmap v65: 97 pgs: 97 active+clean; 45 MiB data, 630 MiB used, 1.0 TiB / 1.0 TiB avail; 2.6 MiB/s rd, 1.3 MiB/s wr, 264 op/s 2024-04-23T10:28:41.908 INFO:tasks.workunit.client.0.smithi114.stdout: 1 105366 11.65 MB/sec warmup 50 sec latency 55.221 ms 2024-04-23T10:28:42.909 INFO:tasks.workunit.client.0.smithi114.stdout: 1 107117 11.66 MB/sec warmup 51 sec latency 16.089 ms 2024-04-23T10:28:43.742 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 23 10:28:43 smithi119 ceph-mon[32608]: pgmap v66: 97 pgs: 97 active+clean; 47 MiB data, 645 MiB used, 1.0 TiB / 1.0 TiB avail; 3.8 MiB/s rd, 1.9 MiB/s wr, 421 op/s 2024-04-23T10:28:43.754 INFO:journalctl@ceph.mon.a.smithi114.stdout:Apr 23 10:28:43 smithi114 ceph-mon[29255]: pgmap v66: 97 pgs: 97 active+clean; 47 MiB data, 645 MiB used, 1.0 TiB / 1.0 TiB avail; 3.8 MiB/s rd, 1.9 MiB/s wr, 421 op/s 2024-04-23T10:28:43.828 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 23 10:28:43 smithi204 ceph-mon[32649]: pgmap v66: 97 pgs: 97 active+clean; 47 MiB data, 645 MiB used, 1.0 TiB / 1.0 TiB avail; 3.8 MiB/s rd, 1.9 MiB/s wr, 421 op/s 2024-04-23T10:28:43.909 INFO:tasks.workunit.client.0.smithi114.stdout: 1 110086 11.78 MB/sec warmup 52 sec latency 13.374 ms 2024-04-23T10:28:44.909 INFO:tasks.workunit.client.0.smithi114.stdout: 1 111594 11.68 MB/sec warmup 53 sec latency 14.926 ms 2024-04-23T10:28:45.741 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 23 10:28:45 smithi119 ceph-mon[32608]: pgmap v67: 97 pgs: 97 active+clean; 47 MiB data, 645 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 MiB/s rd, 1.3 MiB/s wr, 260 op/s 2024-04-23T10:28:45.754 INFO:journalctl@ceph.mon.a.smithi114.stdout:Apr 23 10:28:45 smithi114 ceph-mon[29255]: pgmap v67: 97 pgs: 97 active+clean; 47 MiB data, 645 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 MiB/s rd, 1.3 MiB/s wr, 260 op/s 2024-04-23T10:28:45.827 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 23 10:28:45 smithi204 ceph-mon[32649]: pgmap v67: 97 pgs: 97 active+clean; 47 MiB data, 645 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 MiB/s rd, 1.3 MiB/s wr, 260 op/s 2024-04-23T10:28:45.909 INFO:tasks.workunit.client.0.smithi114.stdout: 1 113936 11.72 MB/sec warmup 54 sec latency 76.084 ms 2024-04-23T10:28:46.741 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 23 10:28:46 smithi119 ceph-mon[32608]: from='mgr.15291 172.21.15.114:0/1095789187' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-23T10:28:46.754 INFO:journalctl@ceph.mon.a.smithi114.stdout:Apr 23 10:28:46 smithi114 ceph-mon[29255]: from='mgr.15291 172.21.15.114:0/1095789187' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-23T10:28:46.827 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 23 10:28:46 smithi204 ceph-mon[32649]: from='mgr.15291 172.21.15.114:0/1095789187' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-23T10:28:46.909 INFO:tasks.workunit.client.0.smithi114.stdout: 1 116034 11.65 MB/sec warmup 55 sec latency 17.382 ms 2024-04-23T10:28:47.909 INFO:tasks.workunit.client.0.smithi114.stdout: 1 117701 11.64 MB/sec warmup 56 sec latency 15.988 ms 2024-04-23T10:28:48.242 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 23 10:28:47 smithi119 ceph-mon[32608]: pgmap v68: 97 pgs: 97 active+clean; 52 MiB data, 675 MiB used, 1.0 TiB / 1.0 TiB avail; 3.8 MiB/s rd, 1.9 MiB/s wr, 370 op/s 2024-04-23T10:28:48.242 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 23 10:28:47 smithi119 ceph-mon[32608]: from='mgr.15291 172.21.15.114:0/1095789187' entity='mgr.x' 2024-04-23T10:28:48.242 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 23 10:28:47 smithi119 ceph-mon[32608]: from='mgr.15291 172.21.15.114:0/1095789187' entity='mgr.x' 2024-04-23T10:28:48.242 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 23 10:28:47 smithi119 ceph-mon[32608]: from='mgr.15291 172.21.15.114:0/1095789187' entity='mgr.x' 2024-04-23T10:28:48.242 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 23 10:28:47 smithi119 ceph-mon[32608]: from='mgr.15291 172.21.15.114:0/1095789187' entity='mgr.x' 2024-04-23T10:28:48.242 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 23 10:28:47 smithi119 ceph-mon[32608]: from='mgr.15291 172.21.15.114:0/1095789187' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-23T10:28:48.242 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 23 10:28:47 smithi119 ceph-mon[32608]: from='mgr.15291 172.21.15.114:0/1095789187' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-23T10:28:48.242 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 23 10:28:47 smithi119 ceph-mon[32608]: from='mgr.15291 172.21.15.114:0/1095789187' entity='mgr.x' 2024-04-23T10:28:48.254 INFO:journalctl@ceph.mon.a.smithi114.stdout:Apr 23 10:28:47 smithi114 ceph-mon[29255]: pgmap v68: 97 pgs: 97 active+clean; 52 MiB data, 675 MiB used, 1.0 TiB / 1.0 TiB avail; 3.8 MiB/s rd, 1.9 MiB/s wr, 370 op/s 2024-04-23T10:28:48.254 INFO:journalctl@ceph.mon.a.smithi114.stdout:Apr 23 10:28:47 smithi114 ceph-mon[29255]: from='mgr.15291 172.21.15.114:0/1095789187' entity='mgr.x' 2024-04-23T10:28:48.254 INFO:journalctl@ceph.mon.a.smithi114.stdout:Apr 23 10:28:47 smithi114 ceph-mon[29255]: from='mgr.15291 172.21.15.114:0/1095789187' entity='mgr.x' 2024-04-23T10:28:48.254 INFO:journalctl@ceph.mon.a.smithi114.stdout:Apr 23 10:28:47 smithi114 ceph-mon[29255]: from='mgr.15291 172.21.15.114:0/1095789187' entity='mgr.x' 2024-04-23T10:28:48.254 INFO:journalctl@ceph.mon.a.smithi114.stdout:Apr 23 10:28:47 smithi114 ceph-mon[29255]: from='mgr.15291 172.21.15.114:0/1095789187' entity='mgr.x' 2024-04-23T10:28:48.254 INFO:journalctl@ceph.mon.a.smithi114.stdout:Apr 23 10:28:47 smithi114 ceph-mon[29255]: from='mgr.15291 172.21.15.114:0/1095789187' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-23T10:28:48.254 INFO:journalctl@ceph.mon.a.smithi114.stdout:Apr 23 10:28:47 smithi114 ceph-mon[29255]: from='mgr.15291 172.21.15.114:0/1095789187' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-23T10:28:48.254 INFO:journalctl@ceph.mon.a.smithi114.stdout:Apr 23 10:28:47 smithi114 ceph-mon[29255]: from='mgr.15291 172.21.15.114:0/1095789187' entity='mgr.x' 2024-04-23T10:28:48.328 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 23 10:28:47 smithi204 ceph-mon[32649]: pgmap v68: 97 pgs: 97 active+clean; 52 MiB data, 675 MiB used, 1.0 TiB / 1.0 TiB avail; 3.8 MiB/s rd, 1.9 MiB/s wr, 370 op/s 2024-04-23T10:28:48.328 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 23 10:28:47 smithi204 ceph-mon[32649]: from='mgr.15291 172.21.15.114:0/1095789187' entity='mgr.x' 2024-04-23T10:28:48.328 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 23 10:28:47 smithi204 ceph-mon[32649]: from='mgr.15291 172.21.15.114:0/1095789187' entity='mgr.x' 2024-04-23T10:28:48.328 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 23 10:28:47 smithi204 ceph-mon[32649]: from='mgr.15291 172.21.15.114:0/1095789187' entity='mgr.x' 2024-04-23T10:28:48.328 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 23 10:28:47 smithi204 ceph-mon[32649]: from='mgr.15291 172.21.15.114:0/1095789187' entity='mgr.x' 2024-04-23T10:28:48.328 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 23 10:28:47 smithi204 ceph-mon[32649]: from='mgr.15291 172.21.15.114:0/1095789187' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-23T10:28:48.328 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 23 10:28:47 smithi204 ceph-mon[32649]: from='mgr.15291 172.21.15.114:0/1095789187' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-23T10:28:48.328 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 23 10:28:47 smithi204 ceph-mon[32649]: from='mgr.15291 172.21.15.114:0/1095789187' entity='mgr.x' 2024-04-23T10:28:48.909 INFO:tasks.workunit.client.0.smithi114.stdout: 1 120279 11.67 MB/sec warmup 57 sec latency 43.200 ms 2024-04-23T10:28:49.909 INFO:tasks.workunit.client.0.smithi114.stdout: 1 121759 11.61 MB/sec warmup 58 sec latency 16.873 ms 2024-04-23T10:28:50.241 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 23 10:28:49 smithi119 ceph-mon[32608]: pgmap v69: 97 pgs: 97 active+clean; 52 MiB data, 675 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 MiB/s rd, 1.3 MiB/s wr, 266 op/s 2024-04-23T10:28:50.254 INFO:journalctl@ceph.mon.a.smithi114.stdout:Apr 23 10:28:49 smithi114 ceph-mon[29255]: pgmap v69: 97 pgs: 97 active+clean; 52 MiB data, 675 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 MiB/s rd, 1.3 MiB/s wr, 266 op/s 2024-04-23T10:28:50.327 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 23 10:28:49 smithi204 ceph-mon[32649]: pgmap v69: 97 pgs: 97 active+clean; 52 MiB data, 675 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 MiB/s rd, 1.3 MiB/s wr, 266 op/s 2024-04-23T10:28:50.909 INFO:tasks.workunit.client.0.smithi114.stdout: 1 124457 11.69 MB/sec warmup 59 sec latency 12.949 ms 2024-04-23T10:28:51.909 INFO:tasks.workunit.client.0.smithi114.stdout: 1 126483 11.62 MB/sec warmup 60 sec latency 22.791 ms 2024-04-23T10:28:52.241 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 23 10:28:51 smithi119 ceph-mon[32608]: pgmap v70: 97 pgs: 97 active+clean; 52 MiB data, 675 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 MiB/s rd, 1.3 MiB/s wr, 266 op/s 2024-04-23T10:28:52.254 INFO:journalctl@ceph.mon.a.smithi114.stdout:Apr 23 10:28:51 smithi114 ceph-mon[29255]: pgmap v70: 97 pgs: 97 active+clean; 52 MiB data, 675 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 MiB/s rd, 1.3 MiB/s wr, 266 op/s 2024-04-23T10:28:52.327 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 23 10:28:51 smithi204 ceph-mon[32649]: pgmap v70: 97 pgs: 97 active+clean; 52 MiB data, 675 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 MiB/s rd, 1.3 MiB/s wr, 266 op/s 2024-04-23T10:28:52.909 INFO:tasks.workunit.client.0.smithi114.stdout: 1 128405 11.63 MB/sec warmup 61 sec latency 11.047 ms 2024-04-23T10:28:53.909 INFO:tasks.workunit.client.0.smithi114.stdout: 1 131077 11.67 MB/sec warmup 62 sec latency 55.615 ms 2024-04-23T10:28:54.172 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:28:54.172+0000 7fe9ec043640 1 -- 172.21.15.114:0/645081936 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fe9e4101310 msgr2=0x7fe9e41016f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T10:28:54.172 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:28:54.172+0000 7fe9ec043640 1 --2- 172.21.15.114:0/645081936 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fe9e4101310 0x7fe9e41016f0 secure :-1 s=READY pgs=560 cs=0 l=1 rev1=1 crypto rx=0x7fe9e0009900 tx=0x7fe9e002a410 comp rx=0 tx=0).stop 2024-04-23T10:28:54.172 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:28:54.172+0000 7fe9ec043640 1 -- 172.21.15.114:0/645081936 shutdown_connections 2024-04-23T10:28:54.173 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:28:54.172+0000 7fe9ec043640 1 --2- 172.21.15.114:0/645081936 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fe9e4106190 0x7fe9e410dd80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:28:54.173 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:28:54.172+0000 7fe9ec043640 1 --2- 172.21.15.114:0/645081936 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fe9e4101cc0 0x7fe9e4105ab0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:28:54.173 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:28:54.172+0000 7fe9ec043640 1 --2- 172.21.15.114:0/645081936 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fe9e4101310 0x7fe9e41016f0 unknown :-1 s=CLOSED pgs=560 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:28:54.173 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:28:54.172+0000 7fe9ec043640 1 -- 172.21.15.114:0/645081936 >> 172.21.15.114:0/645081936 conn(0x7fe9e4077cc0 msgr2=0x7fe9e40780d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T10:28:54.173 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:28:54.172+0000 7fe9ec043640 1 -- 172.21.15.114:0/645081936 shutdown_connections 2024-04-23T10:28:54.173 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:28:54.172+0000 7fe9ec043640 1 -- 172.21.15.114:0/645081936 wait complete. 2024-04-23T10:28:54.175 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:28:54.173+0000 7fe9ec043640 1 Processor -- start 2024-04-23T10:28:54.175 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:28:54.173+0000 7fe9ec043640 1 -- start start 2024-04-23T10:28:54.175 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:28:54.173+0000 7fe9ec043640 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fe9e4101310 0x7fe9e41986f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T10:28:54.175 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:28:54.173+0000 7fe9ec043640 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fe9e4106190 0x7fe9e4198c30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T10:28:54.175 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:28:54.173+0000 7fe9ec043640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fe9e419cfb0 0x7fe9e419d460 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T10:28:54.175 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:28:54.173+0000 7fe9ec043640 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x7fe9e4111110 con 0x7fe9e419cfb0 2024-04-23T10:28:54.175 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:28:54.173+0000 7fe9ec043640 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 -- 0x7fe9e4110f90 con 0x7fe9e4106190 2024-04-23T10:28:54.176 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:28:54.173+0000 7fe9ec043640 1 -- --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_getmap magic: 0 -- 0x7fe9e4111290 con 0x7fe9e4101310 2024-04-23T10:28:54.176 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:28:54.173+0000 7fe9ea5e0640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fe9e419cfb0 0x7fe9e419d460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T10:28:54.176 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:28:54.173+0000 7fe9e95de640 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fe9e4106190 0x7fe9e4198c30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T10:28:54.176 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:28:54.173+0000 7fe9ea5e0640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fe9e419cfb0 0x7fe9e419d460 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.114:47638/0 (socket says 172.21.15.114:47638) 2024-04-23T10:28:54.176 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:28:54.173+0000 7fe9ea5e0640 1 -- 172.21.15.114:0/1154106212 learned_addr learned my addr 172.21.15.114:0/1154106212 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-04-23T10:28:54.176 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:28:54.173+0000 7fe9e9ddf640 1 --2- 172.21.15.114:0/1154106212 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fe9e4101310 0x7fe9e41986f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T10:28:54.176 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:28:54.174+0000 7fe9e95de640 1 -- 172.21.15.114:0/1154106212 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fe9e4101310 msgr2=0x7fe9e41986f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T10:28:54.176 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:28:54.174+0000 7fe9e95de640 1 --2- 172.21.15.114:0/1154106212 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fe9e4101310 0x7fe9e41986f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:28:54.176 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:28:54.174+0000 7fe9e95de640 1 -- 172.21.15.114:0/1154106212 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fe9e419cfb0 msgr2=0x7fe9e419d460 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T10:28:54.176 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:28:54.174+0000 7fe9e95de640 1 --2- 172.21.15.114:0/1154106212 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fe9e419cfb0 0x7fe9e419d460 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:28:54.176 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:28:54.174+0000 7fe9e95de640 1 -- 172.21.15.114:0/1154106212 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe9e419daa0 con 0x7fe9e4106190 2024-04-23T10:28:54.176 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:28:54.174+0000 7fe9ea5e0640 1 --2- 172.21.15.114:0/1154106212 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fe9e419cfb0 0x7fe9e419d460 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-23T10:28:54.176 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:28:54.174+0000 7fe9e95de640 1 --2- 172.21.15.114:0/1154106212 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fe9e4106190 0x7fe9e4198c30 secure :-1 s=READY pgs=68 cs=0 l=1 rev1=1 crypto rx=0x7fe9d400e9c0 tx=0x7fe9d400ee80 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T10:28:54.176 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:28:54.174+0000 7fe9daffd640 1 -- 172.21.15.114:0/1154106212 <== mon.2 v2:172.21.15.119:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe9d400ccc0 con 0x7fe9e4106190 2024-04-23T10:28:54.177 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:28:54.174+0000 7fe9ec043640 1 -- 172.21.15.114:0/1154106212 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fe9e419dd90 con 0x7fe9e4106190 2024-04-23T10:28:54.177 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:28:54.174+0000 7fe9daffd640 1 -- 172.21.15.114:0/1154106212 <== mon.2 v2:172.21.15.119:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fe9d4005020 con 0x7fe9e4106190 2024-04-23T10:28:54.177 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:28:54.174+0000 7fe9daffd640 1 -- 172.21.15.114:0/1154106212 <== mon.2 v2:172.21.15.119:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe9d401c070 con 0x7fe9e4106190 2024-04-23T10:28:54.177 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:28:54.174+0000 7fe9ec043640 1 -- 172.21.15.114:0/1154106212 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fe9e41a5530 con 0x7fe9e4106190 2024-04-23T10:28:54.183 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:28:54.176+0000 7fe9daffd640 1 -- 172.21.15.114:0/1154106212 <== mon.2 v2:172.21.15.119:3300/0 4 ==== mgrmap(e 26) ==== 138778+0+0 (secure 0 0 0) 0x7fe9d40051d0 con 0x7fe9e4106190 2024-04-23T10:28:54.183 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:28:54.176+0000 7fe9eee39740 1 Processor -- start 2024-04-23T10:28:54.183 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:28:54.177+0000 7fe9eee39740 1 -- start start 2024-04-23T10:28:54.183 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:28:54.177+0000 7fe9eee39740 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x55e173b68620 0x55e173b68a00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T10:28:54.183 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:28:54.177+0000 7fe9eee39740 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x55e173b68f40 0x55e173b732c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T10:28:54.183 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:28:54.177+0000 7fe9eee39740 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55e173b73800 0x55e173b75bf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T10:28:54.183 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:28:54.177+0000 7fe9eee39740 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x55e173acd970 con 0x55e173b73800 2024-04-23T10:28:54.183 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:28:54.177+0000 7fe9eee39740 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 -- 0x55e173a6a4a0 con 0x55e173b68620 2024-04-23T10:28:54.183 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:28:54.177+0000 7fe9eee39740 1 -- --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_getmap magic: 0 -- 0x55e173ac9930 con 0x55e173b68f40 2024-04-23T10:28:54.184 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:28:54.177+0000 7fe9daffd640 1 --2- 172.21.15.114:0/1154106212 >> [v2:172.21.15.114:6800/1395693466,v1:172.21.15.114:6801/1395693466] conn(0x7fe9d00a3dc0 0x7fe9d00a6280 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T10:28:54.184 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:28:54.177+0000 7fe9ea5e0640 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x55e173b68620 0x55e173b68a00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T10:28:54.184 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:28:54.177+0000 7fe9ea5e0640 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x55e173b68620 0x55e173b68a00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.119:3300/0 says I am v2:172.21.15.114:35436/0 (socket says 172.21.15.114:35436) 2024-04-23T10:28:54.184 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:28:54.177+0000 7fe9ea5e0640 1 -- 172.21.15.114:0/434154168 learned_addr learned my addr 172.21.15.114:0/434154168 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-04-23T10:28:54.184 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:28:54.177+0000 7fe9e95de640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55e173b73800 0x55e173b75bf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T10:28:54.184 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:28:54.177+0000 7fe9e9ddf640 1 --2- 172.21.15.114:0/434154168 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x55e173b68f40 0x55e173b732c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T10:28:54.184 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:28:54.177+0000 7fe9ea5e0640 1 --2- 172.21.15.114:0/1154106212 >> [v2:172.21.15.114:6800/1395693466,v1:172.21.15.114:6801/1395693466] conn(0x7fe9d00a3dc0 0x7fe9d00a6280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T10:28:54.184 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:28:54.177+0000 7fe9daffd640 1 -- 172.21.15.114:0/1154106212 <== mon.2 v2:172.21.15.119:3300/0 5 ==== osd_map(92..92 src has 1..92) ==== 8585+0+0 (secure 0 0 0) 0x7fe9d40d0750 con 0x7fe9e4106190 2024-04-23T10:28:54.184 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:28:54.177+0000 7fe9ea5e0640 1 -- 172.21.15.114:0/434154168 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x55e173b68f40 msgr2=0x55e173b732c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T10:28:54.184 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:28:54.177+0000 7fe9ea5e0640 1 --2- 172.21.15.114:0/434154168 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x55e173b68f40 0x55e173b732c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:28:54.184 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:28:54.177+0000 7fe9ea5e0640 1 -- 172.21.15.114:0/434154168 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55e173b73800 msgr2=0x55e173b75bf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T10:28:54.184 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:28:54.177+0000 7fe9ea5e0640 1 --2- 172.21.15.114:0/434154168 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55e173b73800 0x55e173b75bf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:28:54.184 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:28:54.177+0000 7fe9ea5e0640 1 -- 172.21.15.114:0/434154168 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55e1737e0210 con 0x55e173b68620 2024-04-23T10:28:54.184 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:28:54.178+0000 7fe9ea5e0640 1 --2- 172.21.15.114:0/434154168 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x55e173b68620 0x55e173b68a00 secure :-1 s=READY pgs=69 cs=0 l=1 rev1=1 crypto rx=0x7fe9dc00feb0 tx=0x7fe9dc013830 comp rx=0 tx=0).ready entity=mon.2 client_cookie=863fa3095ed510d9 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T10:28:54.184 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:28:54.178+0000 7fe9ea5e0640 1 --2- 172.21.15.114:0/1154106212 >> [v2:172.21.15.114:6800/1395693466,v1:172.21.15.114:6801/1395693466] conn(0x7fe9d00a3dc0 0x7fe9d00a6280 secure :-1 s=READY pgs=79 cs=0 l=1 rev1=1 crypto rx=0x7fe9dc01e000 tx=0x7fe9dc020000 comp rx=0 tx=0).ready entity=mgr.15291 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T10:28:54.185 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:28:54.178+0000 7fe9cffff640 1 -- 172.21.15.114:0/434154168 <== mon.2 v2:172.21.15.119:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe9dc00ee50 con 0x55e173b68620 2024-04-23T10:28:54.185 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:28:54.178+0000 7fe9cffff640 1 -- 172.21.15.114:0/434154168 <== mon.2 v2:172.21.15.119:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fe9dc023070 con 0x55e173b68620 2024-04-23T10:28:54.185 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:28:54.178+0000 7fe9cffff640 1 -- 172.21.15.114:0/434154168 <== mon.2 v2:172.21.15.119:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe9dc010430 con 0x55e173b68620 2024-04-23T10:28:54.185 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:28:54.178+0000 7fe9eee39740 1 -- 172.21.15.114:0/434154168 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x55e173b68620 msgr2=0x55e173b68a00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T10:28:54.185 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:28:54.178+0000 7fe9eee39740 1 --2- 172.21.15.114:0/434154168 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x55e173b68620 0x55e173b68a00 secure :-1 s=READY pgs=69 cs=0 l=1 rev1=1 crypto rx=0x7fe9dc00feb0 tx=0x7fe9dc013830 comp rx=0 tx=0).stop 2024-04-23T10:28:54.185 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:28:54.179+0000 7fe9eee39740 1 -- 172.21.15.114:0/434154168 shutdown_connections 2024-04-23T10:28:54.185 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:28:54.179+0000 7fe9eee39740 1 --2- 172.21.15.114:0/434154168 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55e173b73800 0x55e173b75bf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:28:54.185 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:28:54.179+0000 7fe9eee39740 1 --2- 172.21.15.114:0/434154168 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x55e173b68f40 0x55e173b732c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:28:54.185 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:28:54.179+0000 7fe9eee39740 1 --2- 172.21.15.114:0/434154168 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x55e173b68620 0x55e173b68a00 unknown :-1 s=CLOSED pgs=69 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:28:54.185 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:28:54.179+0000 7fe9eee39740 1 -- 172.21.15.114:0/434154168 >> 172.21.15.114:0/434154168 conn(0x55e173b64b30 msgr2=0x55e173b64f10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T10:28:54.185 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:28:54.179+0000 7fe9eee39740 1 -- 172.21.15.114:0/434154168 shutdown_connections 2024-04-23T10:28:54.185 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:28:54.179+0000 7fe9eee39740 1 -- 172.21.15.114:0/434154168 wait complete. 2024-04-23T10:28:54.185 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:28:54.179+0000 7fe9eee39740 1 -- start start 2024-04-23T10:28:54.185 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:28:54.179+0000 7fe9eee39740 1 Processor -- start 2024-04-23T10:28:54.185 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:28:54.179+0000 7fe9eee39740 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55e173b68620 0x55e173b73130 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T10:28:54.186 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:28:54.179+0000 7fe9eee39740 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x55e173b68f40 0x55e173b70420 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T10:28:54.186 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:28:54.179+0000 7fe9eee39740 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x55e173b73800 0x55e173b70960 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T10:28:54.186 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:28:54.179+0000 7fe9eee39740 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x55e173b763d0 con 0x55e173b68620 2024-04-23T10:28:54.186 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:28:54.179+0000 7fe9eee39740 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 -- 0x55e173a680b0 con 0x55e173b68f40 2024-04-23T10:28:54.186 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:28:54.179+0000 7fe9eee39740 1 -- --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_getmap magic: 0 -- 0x55e173b76550 con 0x55e173b73800 2024-04-23T10:28:54.186 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:28:54.179+0000 7fe9eee39740 10 client.0 _ll_register_callbacks cb 0x55e173a55d30 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-23T10:28:54.186 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:28:54.179+0000 7fe9e9ddf640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55e173b68620 0x55e173b73130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T10:28:54.186 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:28:54.180+0000 7fe9e9ddf640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55e173b68620 0x55e173b73130 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.114:47662/0 (socket says 172.21.15.114:47662) 2024-04-23T10:28:54.186 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:28:54.180+0000 7fe9e9ddf640 1 -- 172.21.15.114:0/463706101 learned_addr learned my addr 172.21.15.114:0/463706101 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-04-23T10:28:54.186 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:28:54.180+0000 7fe9e95de640 1 --2- 172.21.15.114:0/463706101 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x55e173b68f40 0x55e173b70420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T10:28:54.186 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:28:54.180+0000 7fe9ea5e0640 1 --2- 172.21.15.114:0/463706101 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x55e173b73800 0x55e173b70960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T10:28:54.186 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:28:54.180+0000 7fe9e9ddf640 1 -- 172.21.15.114:0/463706101 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x55e173b73800 msgr2=0x55e173b70960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T10:28:54.187 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:28:54.180+0000 7fe9e9ddf640 1 --2- 172.21.15.114:0/463706101 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x55e173b73800 0x55e173b70960 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:28:54.187 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:28:54.180+0000 7fe9e9ddf640 1 -- 172.21.15.114:0/463706101 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x55e173b68f40 msgr2=0x55e173b70420 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T10:28:54.187 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:28:54.180+0000 7fe9e9ddf640 1 --2- 172.21.15.114:0/463706101 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x55e173b68f40 0x55e173b70420 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:28:54.187 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:28:54.180+0000 7fe9e9ddf640 1 -- 172.21.15.114:0/463706101 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55e1738ac620 con 0x55e173b68620 2024-04-23T10:28:54.187 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:28:54.180+0000 7fe9e9ddf640 1 --2- 172.21.15.114:0/463706101 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55e173b68620 0x55e173b73130 secure :-1 s=READY pgs=561 cs=0 l=1 rev1=1 crypto rx=0x7fe9e0009900 tx=0x7fe9e00314a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=393c71f4f9ca0dd4 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T10:28:54.187 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:28:54.180+0000 7fe9cd7fa640 10 client.0 ms_handle_connect on v2:172.21.15.114:3300/0 2024-04-23T10:28:54.187 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:28:54.180+0000 7fe9cd7fa640 1 -- 172.21.15.114:0/463706101 <== mon.0 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe9e0035040 con 0x55e173b68620 2024-04-23T10:28:54.187 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:28:54.180+0000 7fe9cd7fa640 1 -- 172.21.15.114:0/463706101 <== mon.0 v2:172.21.15.114:3300/0 2 ==== osd_map(92..92 src has 1..92) ==== 8585+0+0 (secure 0 0 0) 0x7fe9e0031810 con 0x55e173b68620 2024-04-23T10:28:54.187 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:28:54.180+0000 7fe9eee39740 1 -- 172.21.15.114:0/463706101 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fe9dc023070 con 0x55e173b68620 2024-04-23T10:28:54.187 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:28:54.180+0000 7fe9cd7fa640 1 -- 172.21.15.114:0/463706101 <== mon.0 v2:172.21.15.114:3300/0 3 ==== mon_get_version_reply(handle=1 version=23) ==== 24+0+0 (secure 0 0 0) 0x7fe9e0004040 con 0x55e173b68620 2024-04-23T10:28:54.187 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:28:54.180+0000 7fe9eee39740 10 client.15828 fetch_fsmap learned FSMap version 23 2024-04-23T10:28:54.187 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:28:54.180+0000 7fe9eee39740 1 -- 172.21.15.114:0/463706101 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({config=0+,fsmap=23,monmap=4+}) -- 0x55e173b822e0 con 0x55e173b68620 2024-04-23T10:28:54.187 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:28:54.180+0000 7fe9cd7fa640 1 -- 172.21.15.114:0/463706101 <== mon.0 v2:172.21.15.114:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fe9dc023070 con 0x55e173b68620 2024-04-23T10:28:54.187 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:28:54.181+0000 7fe9cd7fa640 1 -- 172.21.15.114:0/463706101 <== mon.0 v2:172.21.15.114:3300/0 5 ==== fsmap(e 23) ==== 8350+0+0 (secure 0 0 0) 0x7fe9e00047e0 con 0x55e173b68620 2024-04-23T10:28:54.187 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:28:54.181+0000 7fe9eee39740 10 client.15828 fetch_fsmap finished waiting for FSMap version 23 2024-04-23T10:28:54.187 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:28:54.181+0000 7fe9eee39740 10 client.15828 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-04-23T10:28:54.188 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:28:54.181+0000 7fe9eee39740 20 client.15828 populate_metadata read hostname 'smithi114' 2024-04-23T10:28:54.188 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:28:54.181+0000 7fe9eee39740 1 --2- 172.21.15.114:0/463706101 >> [v2:172.21.15.119:6838/937297340,v1:172.21.15.119:6839/937297340] conn(0x55e173b837a0 0x55e173b85b90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T10:28:54.188 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:28:54.181+0000 7fe9eee39740 4 client.15828 mds_command: new command op to 24545 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-23T10:28:54.188 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:28:54.181+0000 7fe9eee39740 1 -- 172.21.15.114:0/463706101 --> [v2:172.21.15.119:6838/937297340,v1:172.21.15.119:6839/937297340] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55e173adb390 con 0x55e173b837a0 2024-04-23T10:28:54.188 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:28:54.181+0000 7fe9e95de640 1 --2- 172.21.15.114:0/463706101 >> [v2:172.21.15.119:6838/937297340,v1:172.21.15.119:6839/937297340] conn(0x55e173b837a0 0x55e173b85b90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T10:28:54.188 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:28:54.181+0000 7fe9cd7fa640 10 client.15828 ms_handle_connect on v2:172.21.15.119:6838/937297340 2024-04-23T10:28:54.188 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:28:54.181+0000 7fe9e95de640 1 --2- 172.21.15.114:0/463706101 >> [v2:172.21.15.119:6838/937297340,v1:172.21.15.119:6839/937297340] conn(0x55e173b837a0 0x55e173b85b90 crc :-1 s=READY pgs=66 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=1cf0864f7241e0e server_cookie=2b95bd19b90d65e0 in_seq=0 out_seq=0 2024-04-23T10:28:54.188 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:28:54.182+0000 7fe9cd7fa640 1 -- 172.21.15.114:0/463706101 <== mds.0 v2:172.21.15.119:6838/937297340 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x55e173adb390 con 0x55e173b837a0 2024-04-23T10:28:54.188 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:28:54.182+0000 7fe9cd7fa640 10 client.15828 handle_command_reply: tid=0 2024-04-23T10:28:54.188 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:28:54.182+0000 7fe9eee39740 1 client.15828 shutdown 2024-04-23T10:28:54.188 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:28:54.182+0000 7fe9eee39740 1 -- 172.21.15.114:0/463706101 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55e173b68620 msgr2=0x55e173b73130 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T10:28:54.188 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:28:54.182+0000 7fe9eee39740 1 --2- 172.21.15.114:0/463706101 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55e173b68620 0x55e173b73130 secure :-1 s=READY pgs=561 cs=0 l=1 rev1=1 crypto rx=0x7fe9e0009900 tx=0x7fe9e00314a0 comp rx=0 tx=0).stop 2024-04-23T10:28:54.188 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:28:54.183+0000 7fe9eee39740 1 -- 172.21.15.114:0/463706101 shutdown_connections 2024-04-23T10:28:54.188 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:28:54.183+0000 7fe9eee39740 1 --2- 172.21.15.114:0/463706101 >> [v2:172.21.15.119:6838/937297340,v1:172.21.15.119:6839/937297340] conn(0x55e173b837a0 0x55e173b85b90 crc :-1 s=READY pgs=66 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:28:54.188 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:28:54.183+0000 7fe9eee39740 1 --2- 172.21.15.114:0/463706101 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x55e173b73800 0x55e173b70960 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:28:54.189 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:28:54.183+0000 7fe9eee39740 1 --2- 172.21.15.114:0/463706101 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x55e173b68f40 0x55e173b70420 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:28:54.189 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:28:54.183+0000 7fe9cd7fa640 0 client.15828 ms_handle_reset on v2:172.21.15.119:6838/937297340 2024-04-23T10:28:54.189 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:28:54.183+0000 7fe9eee39740 1 --2- 172.21.15.114:0/463706101 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55e173b68620 0x55e173b73130 unknown :-1 s=CLOSED pgs=561 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:28:54.189 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:28:54.183+0000 7fe9eee39740 1 -- 172.21.15.114:0/463706101 >> 172.21.15.114:0/463706101 conn(0x55e173b64b30 msgr2=0x55e173b6a8b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T10:28:54.189 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:28:54.183+0000 7fe9eee39740 1 -- 172.21.15.114:0/463706101 shutdown_connections 2024-04-23T10:28:54.189 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:28:54.183+0000 7fe9eee39740 1 -- 172.21.15.114:0/463706101 wait complete. 2024-04-23T10:28:54.189 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:28:54.183+0000 7fe9eee39740 20 client.15828 trim_cache size 0 max 16384 2024-04-23T10:28:54.202 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:28:54.198+0000 7fe9eee39740 1 Processor -- start 2024-04-23T10:28:54.202 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:28:54.198+0000 7fe9eee39740 1 -- start start 2024-04-23T10:28:54.202 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:28:54.198+0000 7fe9eee39740 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55e173b68620 0x55e173b7f290 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T10:28:54.202 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:28:54.198+0000 7fe9eee39740 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x55e173b68f40 0x55e173b63420 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T10:28:54.202 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:28:54.198+0000 7fe9eee39740 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x55e173b73800 0x55e173b72c60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T10:28:54.202 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:28:54.198+0000 7fe9eee39740 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x7fe9e41b04f0 con 0x55e173b68620 2024-04-23T10:28:54.203 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:28:54.198+0000 7fe9eee39740 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 -- 0x7fe9e41add90 con 0x55e173b68f40 2024-04-23T10:28:54.203 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:28:54.198+0000 7fe9eee39740 1 -- --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_getmap magic: 0 -- 0x7fe9e41abbe0 con 0x55e173b73800 2024-04-23T10:28:54.203 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:28:54.198+0000 7fe9e9ddf640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55e173b68620 0x55e173b7f290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T10:28:54.203 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:28:54.198+0000 7fe9e9ddf640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55e173b68620 0x55e173b7f290 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.114:47674/0 (socket says 172.21.15.114:47674) 2024-04-23T10:28:54.203 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:28:54.198+0000 7fe9e9ddf640 1 -- 172.21.15.114:0/2713687885 learned_addr learned my addr 172.21.15.114:0/2713687885 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-04-23T10:28:54.203 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:28:54.198+0000 7fe9ea5e0640 1 --2- 172.21.15.114:0/2713687885 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x55e173b73800 0x55e173b72c60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T10:28:54.203 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:28:54.198+0000 7fe9e95de640 1 --2- 172.21.15.114:0/2713687885 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x55e173b68f40 0x55e173b63420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T10:28:54.203 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:28:54.198+0000 7fe9e9ddf640 1 -- 172.21.15.114:0/2713687885 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x55e173b73800 msgr2=0x55e173b72c60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T10:28:54.203 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:28:54.198+0000 7fe9e9ddf640 1 --2- 172.21.15.114:0/2713687885 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x55e173b73800 0x55e173b72c60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:28:54.203 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:28:54.198+0000 7fe9e9ddf640 1 -- 172.21.15.114:0/2713687885 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x55e173b68f40 msgr2=0x55e173b63420 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T10:28:54.203 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:28:54.198+0000 7fe9e9ddf640 1 --2- 172.21.15.114:0/2713687885 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x55e173b68f40 0x55e173b63420 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:28:54.203 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:28:54.198+0000 7fe9e9ddf640 1 -- 172.21.15.114:0/2713687885 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55e173b642d0 con 0x55e173b68620 2024-04-23T10:28:54.203 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:28:54.198+0000 7fe9e9ddf640 1 --2- 172.21.15.114:0/2713687885 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55e173b68620 0x55e173b7f290 secure :-1 s=READY pgs=562 cs=0 l=1 rev1=1 crypto rx=0x7fe9e0032d70 tx=0x7fe9e002a410 comp rx=0 tx=0).ready entity=mon.0 client_cookie=65e3bf3dfeb3b2ae server_cookie=0 in_seq=0 out_seq=0 2024-04-23T10:28:54.203 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:28:54.199+0000 7fe9cf7fe640 1 -- 172.21.15.114:0/2713687885 <== mon.0 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe9e0035030 con 0x55e173b68620 2024-04-23T10:28:54.203 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:28:54.199+0000 7fe9cf7fe640 1 -- 172.21.15.114:0/2713687885 <== mon.0 v2:172.21.15.114:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fe9e00043e0 con 0x55e173b68620 2024-04-23T10:28:54.204 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:28:54.199+0000 7fe9cf7fe640 1 -- 172.21.15.114:0/2713687885 <== mon.0 v2:172.21.15.114:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe9e0052070 con 0x55e173b68620 2024-04-23T10:28:54.204 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:28:54.199+0000 7fe9eee39740 1 -- 172.21.15.114:0/2713687885 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55e173b68620 msgr2=0x55e173b7f290 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T10:28:54.204 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:28:54.199+0000 7fe9eee39740 1 --2- 172.21.15.114:0/2713687885 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55e173b68620 0x55e173b7f290 secure :-1 s=READY pgs=562 cs=0 l=1 rev1=1 crypto rx=0x7fe9e0032d70 tx=0x7fe9e002a410 comp rx=0 tx=0).stop 2024-04-23T10:28:54.204 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:28:54.199+0000 7fe9eee39740 1 -- 172.21.15.114:0/2713687885 shutdown_connections 2024-04-23T10:28:54.204 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:28:54.199+0000 7fe9eee39740 1 --2- 172.21.15.114:0/2713687885 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x55e173b73800 0x55e173b72c60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:28:54.204 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:28:54.199+0000 7fe9eee39740 1 --2- 172.21.15.114:0/2713687885 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x55e173b68f40 0x55e173b63420 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:28:54.204 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:28:54.199+0000 7fe9eee39740 1 --2- 172.21.15.114:0/2713687885 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55e173b68620 0x55e173b7f290 unknown :-1 s=CLOSED pgs=562 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:28:54.204 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:28:54.199+0000 7fe9eee39740 1 -- 172.21.15.114:0/2713687885 >> 172.21.15.114:0/2713687885 conn(0x55e173b64b30 msgr2=0x55e173b6b9d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T10:28:54.204 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:28:54.199+0000 7fe9eee39740 1 -- 172.21.15.114:0/2713687885 shutdown_connections 2024-04-23T10:28:54.204 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:28:54.199+0000 7fe9eee39740 1 -- 172.21.15.114:0/2713687885 wait complete. 2024-04-23T10:28:54.204 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:28:54.200+0000 7fe9eee39740 1 -- start start 2024-04-23T10:28:54.204 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:28:54.200+0000 7fe9eee39740 1 Processor -- start 2024-04-23T10:28:54.204 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:28:54.200+0000 7fe9eee39740 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55e173b68620 0x55e173b7bc80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T10:28:54.204 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:28:54.200+0000 7fe9eee39740 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x55e173b68f40 0x55e173b6ce80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T10:28:54.204 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:28:54.200+0000 7fe9eee39740 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x55e173b73800 0x55e173b6d420 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T10:28:54.204 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:28:54.200+0000 7fe9eee39740 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x7fe9e41ac250 con 0x55e173b68620 2024-04-23T10:28:54.205 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:28:54.200+0000 7fe9eee39740 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 -- 0x7fe9e41ad6c0 con 0x55e173b73800 2024-04-23T10:28:54.205 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:28:54.200+0000 7fe9eee39740 1 -- --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_getmap magic: 0 -- 0x7fe9e41ac920 con 0x55e173b68f40 2024-04-23T10:28:54.205 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:28:54.200+0000 7fe9eee39740 10 client.0 _ll_register_callbacks cb 0x7fe9e0032b20 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-23T10:28:54.205 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:28:54.200+0000 7fe9e9ddf640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55e173b68620 0x55e173b7bc80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T10:28:54.205 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:28:54.200+0000 7fe9e9ddf640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55e173b68620 0x55e173b7bc80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.114:47688/0 (socket says 172.21.15.114:47688) 2024-04-23T10:28:54.205 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:28:54.200+0000 7fe9e9ddf640 1 -- 172.21.15.114:0/891994507 learned_addr learned my addr 172.21.15.114:0/891994507 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-04-23T10:28:54.205 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:28:54.200+0000 7fe9e95de640 1 --2- 172.21.15.114:0/891994507 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x55e173b68f40 0x55e173b6ce80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T10:28:54.205 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:28:54.200+0000 7fe9ea5e0640 1 --2- 172.21.15.114:0/891994507 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x55e173b73800 0x55e173b6d420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T10:28:54.205 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:28:54.201+0000 7fe9e9ddf640 1 -- 172.21.15.114:0/891994507 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x55e173b68f40 msgr2=0x55e173b6ce80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T10:28:54.205 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:28:54.201+0000 7fe9e9ddf640 1 --2- 172.21.15.114:0/891994507 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x55e173b68f40 0x55e173b6ce80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:28:54.205 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:28:54.201+0000 7fe9e9ddf640 1 -- 172.21.15.114:0/891994507 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x55e173b73800 msgr2=0x55e173b6d420 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T10:28:54.205 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:28:54.201+0000 7fe9e9ddf640 1 --2- 172.21.15.114:0/891994507 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x55e173b73800 0x55e173b6d420 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:28:54.205 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:28:54.201+0000 7fe9e9ddf640 1 -- 172.21.15.114:0/891994507 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55e173b6dc50 con 0x55e173b68620 2024-04-23T10:28:54.205 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:28:54.201+0000 7fe9e9ddf640 1 --2- 172.21.15.114:0/891994507 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55e173b68620 0x55e173b7bc80 secure :-1 s=READY pgs=563 cs=0 l=1 rev1=1 crypto rx=0x7fe9e0031440 tx=0x7fe9e0032bd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5e8edc6470148a24 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T10:28:54.205 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:28:54.201+0000 7fe9cd7fa640 10 client.0 ms_handle_connect on v2:172.21.15.114:3300/0 2024-04-23T10:28:54.206 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:28:54.201+0000 7fe9cd7fa640 1 -- 172.21.15.114:0/891994507 <== mon.0 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe9e0035070 con 0x55e173b68620 2024-04-23T10:28:54.206 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:28:54.201+0000 7fe9cd7fa640 1 -- 172.21.15.114:0/891994507 <== mon.0 v2:172.21.15.114:3300/0 2 ==== osd_map(92..92 src has 1..92) ==== 8585+0+0 (secure 0 0 0) 0x7fe9e0003850 con 0x55e173b68620 2024-04-23T10:28:54.206 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:28:54.201+0000 7fe9eee39740 1 -- 172.21.15.114:0/891994507 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fe9e00043e0 con 0x55e173b68620 2024-04-23T10:28:54.206 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:28:54.201+0000 7fe9cd7fa640 1 -- 172.21.15.114:0/891994507 <== mon.0 v2:172.21.15.114:3300/0 3 ==== mon_get_version_reply(handle=1 version=23) ==== 24+0+0 (secure 0 0 0) 0x7fe9e0066050 con 0x55e173b68620 2024-04-23T10:28:54.206 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:28:54.201+0000 7fe9eee39740 10 client.15834 fetch_fsmap learned FSMap version 23 2024-04-23T10:28:54.206 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:28:54.201+0000 7fe9eee39740 1 -- 172.21.15.114:0/891994507 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({config=0+,fsmap=23,monmap=4+}) -- 0x55e173be9d70 con 0x55e173b68620 2024-04-23T10:28:54.206 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:28:54.201+0000 7fe9cd7fa640 1 -- 172.21.15.114:0/891994507 <== mon.0 v2:172.21.15.114:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fe9e00043e0 con 0x55e173b68620 2024-04-23T10:28:54.206 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:28:54.201+0000 7fe9cd7fa640 1 -- 172.21.15.114:0/891994507 <== mon.0 v2:172.21.15.114:3300/0 5 ==== fsmap(e 23) ==== 8350+0+0 (secure 0 0 0) 0x7fe9e005f060 con 0x55e173b68620 2024-04-23T10:28:54.206 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:28:54.201+0000 7fe9eee39740 10 client.15834 fetch_fsmap finished waiting for FSMap version 23 2024-04-23T10:28:54.206 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:28:54.201+0000 7fe9eee39740 10 client.15834 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-04-23T10:28:54.206 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:28:54.201+0000 7fe9eee39740 20 client.15834 populate_metadata read hostname 'smithi114' 2024-04-23T10:28:54.206 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:28:54.201+0000 7fe9eee39740 1 --2- 172.21.15.114:0/891994507 >> [v2:172.21.15.119:6838/937297340,v1:172.21.15.119:6839/937297340] conn(0x55e173bea400 0x55e173bec820 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T10:28:54.206 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:28:54.201+0000 7fe9eee39740 4 client.15834 mds_command: new command op to 24545 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-04-23T10:28:54.206 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:28:54.201+0000 7fe9eee39740 1 -- 172.21.15.114:0/891994507 --> [v2:172.21.15.119:6838/937297340,v1:172.21.15.119:6839/937297340] -- command(tid 0: {"prefix": "flush journal"}) -- 0x55e173ad5200 con 0x55e173bea400 2024-04-23T10:28:54.207 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:28:54.202+0000 7fe9e95de640 1 --2- 172.21.15.114:0/891994507 >> [v2:172.21.15.119:6838/937297340,v1:172.21.15.119:6839/937297340] conn(0x55e173bea400 0x55e173bec820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T10:28:54.207 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:28:54.202+0000 7fe9cd7fa640 10 client.15834 ms_handle_connect on v2:172.21.15.119:6838/937297340 2024-04-23T10:28:54.207 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:28:54.202+0000 7fe9e95de640 1 --2- 172.21.15.114:0/891994507 >> [v2:172.21.15.119:6838/937297340,v1:172.21.15.119:6839/937297340] conn(0x55e173bea400 0x55e173bec820 crc :-1 s=READY pgs=67 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=22d346c6ae4aafe6 server_cookie=e7cdaaaf3fcea37e in_seq=0 out_seq=0 2024-04-23T10:28:54.241 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 23 10:28:53 smithi119 ceph-mon[32608]: pgmap v71: 97 pgs: 97 active+clean; 56 MiB data, 689 MiB used, 1.0 TiB / 1.0 TiB avail; 3.8 MiB/s rd, 2.6 MiB/s wr, 390 op/s 2024-04-23T10:28:54.254 INFO:journalctl@ceph.mon.a.smithi114.stdout:Apr 23 10:28:53 smithi114 ceph-mon[29255]: pgmap v71: 97 pgs: 97 active+clean; 56 MiB data, 689 MiB used, 1.0 TiB / 1.0 TiB avail; 3.8 MiB/s rd, 2.6 MiB/s wr, 390 op/s 2024-04-23T10:28:54.261 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:28:54.261+0000 7fe9cd7fa640 1 -- 172.21.15.114:0/891994507 <== mds.0 v2:172.21.15.119:6838/937297340 1 ==== command_reply(tid 0: 0 ) ==== 8+0+44 (crc 0 0 0) 0x55e173ad5200 con 0x55e173bea400 2024-04-23T10:28:54.262 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:28:54.261+0000 7fe9cd7fa640 10 client.15834 handle_command_reply: tid=0 2024-04-23T10:28:54.262 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:28:54.261+0000 7fe9eee39740 1 client.15834 shutdown 2024-04-23T10:28:54.262 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:28:54.261+0000 7fe9eee39740 1 -- 172.21.15.114:0/891994507 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55e173b68620 msgr2=0x55e173b7bc80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T10:28:54.262 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:28:54.261+0000 7fe9eee39740 1 --2- 172.21.15.114:0/891994507 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55e173b68620 0x55e173b7bc80 secure :-1 s=READY pgs=563 cs=0 l=1 rev1=1 crypto rx=0x7fe9e0031440 tx=0x7fe9e0032bd0 comp rx=0 tx=0).stop 2024-04-23T10:28:54.262 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:28:54.261+0000 7fe9eee39740 1 -- 172.21.15.114:0/891994507 shutdown_connections 2024-04-23T10:28:54.262 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:28:54.261+0000 7fe9eee39740 1 --2- 172.21.15.114:0/891994507 >> [v2:172.21.15.119:6838/937297340,v1:172.21.15.119:6839/937297340] conn(0x55e173bea400 0x55e173bec820 crc :-1 s=READY pgs=67 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:28:54.262 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:28:54.261+0000 7fe9eee39740 1 --2- 172.21.15.114:0/891994507 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x55e173b73800 0x55e173b6d420 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:28:54.262 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:28:54.261+0000 7fe9eee39740 1 --2- 172.21.15.114:0/891994507 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x55e173b68f40 0x55e173b6ce80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:28:54.262 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:28:54.261+0000 7fe9cd7fa640 0 client.15834 ms_handle_reset on v2:172.21.15.119:6838/937297340 2024-04-23T10:28:54.262 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:28:54.261+0000 7fe9eee39740 1 --2- 172.21.15.114:0/891994507 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55e173b68620 0x55e173b7bc80 unknown :-1 s=CLOSED pgs=563 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:28:54.262 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:28:54.261+0000 7fe9eee39740 1 -- 172.21.15.114:0/891994507 >> 172.21.15.114:0/891994507 conn(0x55e173b64b30 msgr2=0x55e173b6a020 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T10:28:54.262 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:28:54.261+0000 7fe9eee39740 1 -- 172.21.15.114:0/891994507 shutdown_connections 2024-04-23T10:28:54.262 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:28:54.261+0000 7fe9eee39740 1 -- 172.21.15.114:0/891994507 wait complete. 2024-04-23T10:28:54.263 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:28:54.262+0000 7fe9eee39740 20 client.15834 trim_cache size 0 max 16384 2024-04-23T10:28:54.263 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stdout:{ 2024-04-23T10:28:54.263 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stdout: "message": "", 2024-04-23T10:28:54.263 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stdout: "return_code": 0 2024-04-23T10:28:54.263 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stdout:} 2024-04-23T10:28:54.264 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:28:54.262+0000 7fe9ec043640 1 -- 172.21.15.114:0/1154106212 >> [v2:172.21.15.114:6800/1395693466,v1:172.21.15.114:6801/1395693466] conn(0x7fe9d00a3dc0 msgr2=0x7fe9d00a6280 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T10:28:54.264 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:28:54.262+0000 7fe9ec043640 1 --2- 172.21.15.114:0/1154106212 >> [v2:172.21.15.114:6800/1395693466,v1:172.21.15.114:6801/1395693466] conn(0x7fe9d00a3dc0 0x7fe9d00a6280 secure :-1 s=READY pgs=79 cs=0 l=1 rev1=1 crypto rx=0x7fe9dc01e000 tx=0x7fe9dc020000 comp rx=0 tx=0).stop 2024-04-23T10:28:54.264 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:28:54.262+0000 7fe9ec043640 1 -- 172.21.15.114:0/1154106212 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fe9e4106190 msgr2=0x7fe9e4198c30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T10:28:54.264 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:28:54.262+0000 7fe9ec043640 1 --2- 172.21.15.114:0/1154106212 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fe9e4106190 0x7fe9e4198c30 secure :-1 s=READY pgs=68 cs=0 l=1 rev1=1 crypto rx=0x7fe9d400e9c0 tx=0x7fe9d400ee80 comp rx=0 tx=0).stop 2024-04-23T10:28:54.264 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:28:54.262+0000 7fe9ec043640 1 -- 172.21.15.114:0/1154106212 shutdown_connections 2024-04-23T10:28:54.264 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:28:54.262+0000 7fe9ec043640 1 --2- 172.21.15.114:0/1154106212 >> [v2:172.21.15.114:6800/1395693466,v1:172.21.15.114:6801/1395693466] conn(0x7fe9d00a3dc0 0x7fe9d00a6280 unknown :-1 s=CLOSED pgs=79 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:28:54.264 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:28:54.262+0000 7fe9ec043640 1 --2- 172.21.15.114:0/1154106212 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fe9e419cfb0 0x7fe9e419d460 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:28:54.264 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:28:54.262+0000 7fe9ec043640 1 --2- 172.21.15.114:0/1154106212 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fe9e4106190 0x7fe9e4198c30 unknown :-1 s=CLOSED pgs=68 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:28:54.264 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:28:54.262+0000 7fe9ec043640 1 --2- 172.21.15.114:0/1154106212 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fe9e4101310 0x7fe9e41986f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:28:54.265 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:28:54.262+0000 7fe9ec043640 1 -- 172.21.15.114:0/1154106212 >> 172.21.15.114:0/1154106212 conn(0x7fe9e4077cc0 msgr2=0x7fe9e406aa60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T10:28:54.265 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:28:54.262+0000 7fe9ec043640 1 -- 172.21.15.114:0/1154106212 shutdown_connections 2024-04-23T10:28:54.265 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:28:54.262+0000 7fe9ec043640 1 -- 172.21.15.114:0/1154106212 wait complete. 2024-04-23T10:28:54.327 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 23 10:28:53 smithi204 ceph-mon[32649]: pgmap v71: 97 pgs: 97 active+clean; 56 MiB data, 689 MiB used, 1.0 TiB / 1.0 TiB avail; 3.8 MiB/s rd, 2.6 MiB/s wr, 390 op/s 2024-04-23T10:28:54.909 INFO:tasks.workunit.client.0.smithi114.stdout: 1 132255 11.58 MB/sec warmup 63 sec latency 48.188 ms 2024-04-23T10:28:55.909 INFO:tasks.workunit.client.0.smithi114.stdout: 1 135082 11.68 MB/sec warmup 64 sec latency 13.907 ms 2024-04-23T10:28:56.241 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 23 10:28:55 smithi119 ceph-mon[32608]: pgmap v72: 97 pgs: 97 active+clean; 56 MiB data, 689 MiB used, 1.0 TiB / 1.0 TiB avail; 2.6 MiB/s rd, 1.9 MiB/s wr, 234 op/s 2024-04-23T10:28:56.253 INFO:journalctl@ceph.mon.a.smithi114.stdout:Apr 23 10:28:55 smithi114 ceph-mon[29255]: pgmap v72: 97 pgs: 97 active+clean; 56 MiB data, 689 MiB used, 1.0 TiB / 1.0 TiB avail; 2.6 MiB/s rd, 1.9 MiB/s wr, 234 op/s 2024-04-23T10:28:56.327 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 23 10:28:55 smithi204 ceph-mon[32649]: pgmap v72: 97 pgs: 97 active+clean; 56 MiB data, 689 MiB used, 1.0 TiB / 1.0 TiB avail; 2.6 MiB/s rd, 1.9 MiB/s wr, 234 op/s 2024-04-23T10:28:56.909 INFO:tasks.workunit.client.0.smithi114.stdout: 1 136799 11.59 MB/sec warmup 65 sec latency 17.914 ms 2024-04-23T10:28:57.909 INFO:tasks.workunit.client.0.smithi114.stdout: 1 138859 11.62 MB/sec warmup 66 sec latency 19.322 ms 2024-04-23T10:28:58.241 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 23 10:28:57 smithi119 ceph-mon[32608]: pgmap v73: 97 pgs: 97 active+clean; 52 MiB data, 730 MiB used, 1.0 TiB / 1.0 TiB avail; 3.8 MiB/s rd, 3.3 MiB/s wr, 409 op/s 2024-04-23T10:28:58.253 INFO:journalctl@ceph.mon.a.smithi114.stdout:Apr 23 10:28:57 smithi114 ceph-mon[29255]: pgmap v73: 97 pgs: 97 active+clean; 52 MiB data, 730 MiB used, 1.0 TiB / 1.0 TiB avail; 3.8 MiB/s rd, 3.3 MiB/s wr, 409 op/s 2024-04-23T10:28:58.327 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 23 10:28:57 smithi204 ceph-mon[32649]: pgmap v73: 97 pgs: 97 active+clean; 52 MiB data, 730 MiB used, 1.0 TiB / 1.0 TiB avail; 3.8 MiB/s rd, 3.3 MiB/s wr, 409 op/s 2024-04-23T10:28:58.909 INFO:tasks.workunit.client.0.smithi114.stdout: 1 141115 11.61 MB/sec warmup 67 sec latency 101.363 ms 2024-04-23T10:28:59.909 INFO:tasks.workunit.client.0.smithi114.stdout: 1 142737 11.57 MB/sec warmup 68 sec latency 32.732 ms 2024-04-23T10:29:00.242 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 23 10:28:59 smithi119 ceph-mon[32608]: pgmap v74: 97 pgs: 97 active+clean; 52 MiB data, 730 MiB used, 1.0 TiB / 1.0 TiB avail; 2.6 MiB/s rd, 2.7 MiB/s wr, 299 op/s 2024-04-23T10:29:00.253 INFO:journalctl@ceph.mon.a.smithi114.stdout:Apr 23 10:28:59 smithi114 ceph-mon[29255]: pgmap v74: 97 pgs: 97 active+clean; 52 MiB data, 730 MiB used, 1.0 TiB / 1.0 TiB avail; 2.6 MiB/s rd, 2.7 MiB/s wr, 299 op/s 2024-04-23T10:29:00.327 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 23 10:28:59 smithi204 ceph-mon[32649]: pgmap v74: 97 pgs: 97 active+clean; 52 MiB data, 730 MiB used, 1.0 TiB / 1.0 TiB avail; 2.6 MiB/s rd, 2.7 MiB/s wr, 299 op/s 2024-04-23T10:29:00.909 INFO:tasks.workunit.client.0.smithi114.stdout: 1 145679 11.66 MB/sec warmup 69 sec latency 11.816 ms 2024-04-23T10:29:01.910 INFO:tasks.workunit.client.0.smithi114.stdout: 1 146519 11.53 MB/sec warmup 70 sec latency 22.796 ms 2024-04-23T10:29:02.241 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 23 10:29:01 smithi119 ceph-mon[32608]: pgmap v75: 97 pgs: 97 active+clean; 52 MiB data, 730 MiB used, 1.0 TiB / 1.0 TiB avail; 2.6 MiB/s rd, 2.7 MiB/s wr, 299 op/s 2024-04-23T10:29:02.254 INFO:journalctl@ceph.mon.a.smithi114.stdout:Apr 23 10:29:01 smithi114 ceph-mon[29255]: pgmap v75: 97 pgs: 97 active+clean; 52 MiB data, 730 MiB used, 1.0 TiB / 1.0 TiB avail; 2.6 MiB/s rd, 2.7 MiB/s wr, 299 op/s 2024-04-23T10:29:02.327 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 23 10:29:01 smithi204 ceph-mon[32649]: pgmap v75: 97 pgs: 97 active+clean; 52 MiB data, 730 MiB used, 1.0 TiB / 1.0 TiB avail; 2.6 MiB/s rd, 2.7 MiB/s wr, 299 op/s 2024-04-23T10:29:02.909 INFO:tasks.workunit.client.0.smithi114.stdout: 1 149256 11.60 MB/sec warmup 71 sec latency 20.932 ms 2024-04-23T10:29:03.910 INFO:tasks.workunit.client.0.smithi114.stdout: 1 150704 11.52 MB/sec warmup 72 sec latency 36.416 ms 2024-04-23T10:29:04.241 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 23 10:29:03 smithi119 ceph-mon[32608]: pgmap v76: 97 pgs: 97 active+clean; 57 MiB data, 749 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 MiB/s rd, 4.1 MiB/s wr, 446 op/s 2024-04-23T10:29:04.254 INFO:journalctl@ceph.mon.a.smithi114.stdout:Apr 23 10:29:03 smithi114 ceph-mon[29255]: pgmap v76: 97 pgs: 97 active+clean; 57 MiB data, 749 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 MiB/s rd, 4.1 MiB/s wr, 446 op/s 2024-04-23T10:29:04.327 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 23 10:29:03 smithi204 ceph-mon[32649]: pgmap v76: 97 pgs: 97 active+clean; 57 MiB data, 749 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 MiB/s rd, 4.1 MiB/s wr, 446 op/s 2024-04-23T10:29:04.910 INFO:tasks.workunit.client.0.smithi114.stdout: 1 152996 11.55 MB/sec warmup 73 sec latency 38.837 ms 2024-04-23T10:29:05.910 INFO:tasks.workunit.client.0.smithi114.stdout: 1 155546 11.57 MB/sec warmup 74 sec latency 15.673 ms 2024-04-23T10:29:06.241 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 23 10:29:05 smithi119 ceph-mon[32608]: pgmap v77: 97 pgs: 97 active+clean; 57 MiB data, 749 MiB used, 1.0 TiB / 1.0 TiB avail; 2.8 MiB/s rd, 2.8 MiB/s wr, 321 op/s 2024-04-23T10:29:06.254 INFO:journalctl@ceph.mon.a.smithi114.stdout:Apr 23 10:29:05 smithi114 ceph-mon[29255]: pgmap v77: 97 pgs: 97 active+clean; 57 MiB data, 749 MiB used, 1.0 TiB / 1.0 TiB avail; 2.8 MiB/s rd, 2.8 MiB/s wr, 321 op/s 2024-04-23T10:29:06.328 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 23 10:29:05 smithi204 ceph-mon[32649]: pgmap v77: 97 pgs: 97 active+clean; 57 MiB data, 749 MiB used, 1.0 TiB / 1.0 TiB avail; 2.8 MiB/s rd, 2.8 MiB/s wr, 321 op/s 2024-04-23T10:29:06.910 INFO:tasks.workunit.client.0.smithi114.stdout: 1 157219 11.53 MB/sec warmup 75 sec latency 16.095 ms 2024-04-23T10:29:07.354 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:29:07.353+0000 7f823c39b640 1 -- 172.21.15.114:0/325540923 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f8234106170 msgr2=0x7f823410dd40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T10:29:07.355 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:29:07.353+0000 7f823c39b640 1 --2- 172.21.15.114:0/325540923 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f8234106170 0x7f823410dd40 secure :-1 s=READY pgs=564 cs=0 l=1 rev1=1 crypto rx=0x7f822c009ca0 tx=0x7f822c02a8c0 comp rx=0 tx=0).stop 2024-04-23T10:29:07.355 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:29:07.353+0000 7f823c39b640 1 -- 172.21.15.114:0/325540923 shutdown_connections 2024-04-23T10:29:07.355 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:29:07.353+0000 7f823c39b640 1 --2- 172.21.15.114:0/325540923 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f8234106170 0x7f823410dd40 unknown :-1 s=CLOSED pgs=564 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:29:07.355 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:29:07.353+0000 7f823c39b640 1 --2- 172.21.15.114:0/325540923 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f8234101aa0 0x7f8234105a90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:29:07.355 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:29:07.353+0000 7f823c39b640 1 --2- 172.21.15.114:0/325540923 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f82341010f0 0x7f82341014d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:29:07.355 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:29:07.353+0000 7f823c39b640 1 -- 172.21.15.114:0/325540923 >> 172.21.15.114:0/325540923 conn(0x7f8234077fd0 msgr2=0x7f82340783e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T10:29:07.355 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:29:07.353+0000 7f823c39b640 1 -- 172.21.15.114:0/325540923 shutdown_connections 2024-04-23T10:29:07.355 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:29:07.353+0000 7f823c39b640 1 -- 172.21.15.114:0/325540923 wait complete. 2024-04-23T10:29:07.355 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:29:07.354+0000 7f823c39b640 1 Processor -- start 2024-04-23T10:29:07.355 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:29:07.354+0000 7f823c39b640 1 -- start start 2024-04-23T10:29:07.355 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:29:07.354+0000 7f823c39b640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f82341010f0 0x7f82341975a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T10:29:07.355 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:29:07.354+0000 7f823c39b640 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f8234106170 0x7f8234197ae0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T10:29:07.356 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:29:07.354+0000 7f823c39b640 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f823419be10 0x7f823419c2c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T10:29:07.356 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:29:07.354+0000 7f823c39b640 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x7f82341110e0 con 0x7f82341010f0 2024-04-23T10:29:07.356 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:29:07.354+0000 7f823c39b640 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 -- 0x7f8234110f60 con 0x7f823419be10 2024-04-23T10:29:07.356 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:29:07.354+0000 7f823c39b640 1 -- --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_getmap magic: 0 -- 0x7f8234111260 con 0x7f8234106170 2024-04-23T10:29:07.356 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:29:07.354+0000 7f823a137640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f82341010f0 0x7f82341975a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T10:29:07.356 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:29:07.354+0000 7f823a137640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f82341010f0 0x7f82341975a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.114:42804/0 (socket says 172.21.15.114:42804) 2024-04-23T10:29:07.356 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:29:07.354+0000 7f823a137640 1 -- 172.21.15.114:0/2647969526 learned_addr learned my addr 172.21.15.114:0/2647969526 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-04-23T10:29:07.356 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:29:07.354+0000 7f823a938640 1 --2- 172.21.15.114:0/2647969526 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f823419be10 0x7f823419c2c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T10:29:07.356 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:29:07.354+0000 7f8239936640 1 --2- 172.21.15.114:0/2647969526 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f8234106170 0x7f8234197ae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T10:29:07.356 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:29:07.355+0000 7f823a137640 1 -- 172.21.15.114:0/2647969526 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f8234106170 msgr2=0x7f8234197ae0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T10:29:07.356 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:29:07.355+0000 7f823a137640 1 --2- 172.21.15.114:0/2647969526 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f8234106170 0x7f8234197ae0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:29:07.356 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:29:07.355+0000 7f823a137640 1 -- 172.21.15.114:0/2647969526 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f823419be10 msgr2=0x7f823419c2c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T10:29:07.356 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:29:07.355+0000 7f823a137640 1 --2- 172.21.15.114:0/2647969526 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f823419be10 0x7f823419c2c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:29:07.356 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:29:07.355+0000 7f823a137640 1 -- 172.21.15.114:0/2647969526 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f823419c900 con 0x7f82341010f0 2024-04-23T10:29:07.356 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:29:07.355+0000 7f823a137640 1 --2- 172.21.15.114:0/2647969526 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f82341010f0 0x7f82341975a0 secure :-1 s=READY pgs=565 cs=0 l=1 rev1=1 crypto rx=0x7f823000ea60 tx=0x7f823000ef20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T10:29:07.357 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:29:07.355+0000 7f822b7fe640 1 -- 172.21.15.114:0/2647969526 <== mon.0 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f823000cde0 con 0x7f82341010f0 2024-04-23T10:29:07.357 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:29:07.355+0000 7f822b7fe640 1 -- 172.21.15.114:0/2647969526 <== mon.0 v2:172.21.15.114:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f82300035f0 con 0x7f82341010f0 2024-04-23T10:29:07.357 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:29:07.355+0000 7f823c39b640 1 -- 172.21.15.114:0/2647969526 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f823419cbf0 con 0x7f82341010f0 2024-04-23T10:29:07.357 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:29:07.355+0000 7f822b7fe640 1 -- 172.21.15.114:0/2647969526 <== mon.0 v2:172.21.15.114:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8230010690 con 0x7f82341010f0 2024-04-23T10:29:07.357 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:29:07.355+0000 7f823c39b640 1 -- 172.21.15.114:0/2647969526 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f82341a43e0 con 0x7f82341010f0 2024-04-23T10:29:07.363 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:29:07.357+0000 7f822b7fe640 1 -- 172.21.15.114:0/2647969526 <== mon.0 v2:172.21.15.114:3300/0 4 ==== mgrmap(e 26) ==== 138778+0+0 (secure 0 0 0) 0x7f82300037a0 con 0x7f82341010f0 2024-04-23T10:29:07.363 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:29:07.358+0000 7f823f191740 1 Processor -- start 2024-04-23T10:29:07.363 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:29:07.358+0000 7f823f191740 1 -- start start 2024-04-23T10:29:07.363 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:29:07.358+0000 7f823f191740 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55d6211edee0 0x55d6211ec730 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T10:29:07.363 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:29:07.358+0000 7f823f191740 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x55d6211ecd00 0x55d6211ed180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T10:29:07.364 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:29:07.358+0000 7f823f191740 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x55d6211f9950 0x55d6211fbd40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T10:29:07.364 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:29:07.358+0000 7f823f191740 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x55d62114f970 con 0x55d6211edee0 2024-04-23T10:29:07.364 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:29:07.358+0000 7f823f191740 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 -- 0x55d6210ec4a0 con 0x55d6211f9950 2024-04-23T10:29:07.364 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:29:07.358+0000 7f823f191740 1 -- --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_getmap magic: 0 -- 0x55d62114b930 con 0x55d6211ecd00 2024-04-23T10:29:07.364 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:29:07.358+0000 7f822b7fe640 1 --2- 172.21.15.114:0/2647969526 >> [v2:172.21.15.114:6800/1395693466,v1:172.21.15.114:6801/1395693466] conn(0x7f82200a3dc0 0x7f82200a6280 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T10:29:07.364 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:29:07.358+0000 7f823a137640 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x55d6211ecd00 0x55d6211ed180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T10:29:07.364 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:29:07.358+0000 7f823a938640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55d6211edee0 0x55d6211ec730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T10:29:07.364 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:29:07.358+0000 7f8239936640 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x55d6211f9950 0x55d6211fbd40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T10:29:07.364 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:29:07.358+0000 7f8239936640 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x55d6211f9950 0x55d6211fbd40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.119:3300/0 says I am v2:172.21.15.114:44026/0 (socket says 172.21.15.114:44026) 2024-04-23T10:29:07.364 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:29:07.358+0000 7f823a938640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55d6211edee0 0x55d6211ec730 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.114:42810/0 (socket says 172.21.15.114:42810) 2024-04-23T10:29:07.364 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:29:07.358+0000 7f8239936640 1 -- 172.21.15.114:0/714856751 learned_addr learned my addr 172.21.15.114:0/714856751 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-04-23T10:29:07.364 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:29:07.358+0000 7f823a938640 1 --2- 172.21.15.114:0/2647969526 >> [v2:172.21.15.114:6800/1395693466,v1:172.21.15.114:6801/1395693466] conn(0x7f82200a3dc0 0x7f82200a6280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T10:29:07.364 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:29:07.359+0000 7f822b7fe640 1 -- 172.21.15.114:0/2647969526 <== mon.0 v2:172.21.15.114:3300/0 5 ==== osd_map(92..92 src has 1..92) ==== 8585+0+0 (secure 0 0 0) 0x7f82300d0ae0 con 0x7f82341010f0 2024-04-23T10:29:07.364 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:29:07.359+0000 7f8239936640 1 -- 172.21.15.114:0/714856751 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x55d6211ecd00 msgr2=0x55d6211ed180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T10:29:07.364 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:29:07.359+0000 7f8239936640 1 --2- 172.21.15.114:0/714856751 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x55d6211ecd00 0x55d6211ed180 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:29:07.365 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:29:07.359+0000 7f8239936640 1 -- 172.21.15.114:0/714856751 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55d6211edee0 msgr2=0x55d6211ec730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T10:29:07.365 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:29:07.359+0000 7f8239936640 1 --2- 172.21.15.114:0/714856751 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55d6211edee0 0x55d6211ec730 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:29:07.365 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:29:07.359+0000 7f8239936640 1 -- 172.21.15.114:0/714856751 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55d620e62210 con 0x55d6211f9950 2024-04-23T10:29:07.365 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:29:07.359+0000 7f823a938640 1 --2- 172.21.15.114:0/2647969526 >> [v2:172.21.15.114:6800/1395693466,v1:172.21.15.114:6801/1395693466] conn(0x7f82200a3dc0 0x7f82200a6280 secure :-1 s=READY pgs=80 cs=0 l=1 rev1=1 crypto rx=0x7f822c009a60 tx=0x7f822c0095c0 comp rx=0 tx=0).ready entity=mgr.15291 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T10:29:07.365 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:29:07.359+0000 7f8239936640 1 --2- 172.21.15.114:0/714856751 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x55d6211f9950 0x55d6211fbd40 secure :-1 s=READY pgs=70 cs=0 l=1 rev1=1 crypto rx=0x7f8224009c00 tx=0x7f8224011810 comp rx=0 tx=0).ready entity=mon.2 client_cookie=186a1aed5a74b28 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T10:29:07.365 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:29:07.359+0000 7f8228bfe640 1 -- 172.21.15.114:0/714856751 <== mon.2 v2:172.21.15.119:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8224004230 con 0x55d6211f9950 2024-04-23T10:29:07.365 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:29:07.359+0000 7f8228bfe640 1 -- 172.21.15.114:0/714856751 <== mon.2 v2:172.21.15.119:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f82240043d0 con 0x55d6211f9950 2024-04-23T10:29:07.365 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:29:07.359+0000 7f8228bfe640 1 -- 172.21.15.114:0/714856751 <== mon.2 v2:172.21.15.119:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8224019640 con 0x55d6211f9950 2024-04-23T10:29:07.365 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:29:07.360+0000 7f823f191740 1 -- 172.21.15.114:0/714856751 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x55d6211f9950 msgr2=0x55d6211fbd40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T10:29:07.365 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:29:07.360+0000 7f823f191740 1 --2- 172.21.15.114:0/714856751 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x55d6211f9950 0x55d6211fbd40 secure :-1 s=READY pgs=70 cs=0 l=1 rev1=1 crypto rx=0x7f8224009c00 tx=0x7f8224011810 comp rx=0 tx=0).stop 2024-04-23T10:29:07.365 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:29:07.360+0000 7f823f191740 1 -- 172.21.15.114:0/714856751 shutdown_connections 2024-04-23T10:29:07.365 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:29:07.360+0000 7f823f191740 1 --2- 172.21.15.114:0/714856751 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x55d6211f9950 0x55d6211fbd40 unknown :-1 s=CLOSED pgs=70 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:29:07.365 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:29:07.360+0000 7f823f191740 1 --2- 172.21.15.114:0/714856751 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x55d6211ecd00 0x55d6211ed180 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:29:07.365 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:29:07.360+0000 7f823f191740 1 --2- 172.21.15.114:0/714856751 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55d6211edee0 0x55d6211ec730 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:29:07.365 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:29:07.360+0000 7f823f191740 1 -- 172.21.15.114:0/714856751 >> 172.21.15.114:0/714856751 conn(0x55d6211e8d40 msgr2=0x55d6211e9120 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T10:29:07.366 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:29:07.360+0000 7f823f191740 1 -- 172.21.15.114:0/714856751 shutdown_connections 2024-04-23T10:29:07.366 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:29:07.360+0000 7f823f191740 1 -- 172.21.15.114:0/714856751 wait complete. 2024-04-23T10:29:07.366 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:29:07.360+0000 7f823f191740 1 -- start start 2024-04-23T10:29:07.366 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:29:07.360+0000 7f823f191740 1 Processor -- start 2024-04-23T10:29:07.366 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:29:07.361+0000 7f823f191740 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x55d6211ecd00 0x55d6211f9220 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T10:29:07.366 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:29:07.361+0000 7f823f191740 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x55d6211edee0 0x55d6211f6560 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T10:29:07.366 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:29:07.361+0000 7f823f191740 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55d6211f9950 0x55d6211f6ac0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T10:29:07.366 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:29:07.361+0000 7f823f191740 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x55d6211fc7e0 con 0x55d6211f9950 2024-04-23T10:29:07.366 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:29:07.361+0000 7f823f191740 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 -- 0x55d6210ea0b0 con 0x55d6211ecd00 2024-04-23T10:29:07.366 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:29:07.361+0000 7f823f191740 1 -- --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_getmap magic: 0 -- 0x55d6211fc960 con 0x55d6211edee0 2024-04-23T10:29:07.366 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:29:07.361+0000 7f823f191740 10 client.0 _ll_register_callbacks cb 0x55d6210d7d30 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-23T10:29:07.366 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:29:07.361+0000 7f823a938640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55d6211f9950 0x55d6211f6ac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T10:29:07.366 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:29:07.361+0000 7f823a137640 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x55d6211ecd00 0x55d6211f9220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T10:29:07.366 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:29:07.361+0000 7f823a938640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55d6211f9950 0x55d6211f6ac0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.114:42818/0 (socket says 172.21.15.114:42818) 2024-04-23T10:29:07.366 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:29:07.361+0000 7f823a938640 1 -- 172.21.15.114:0/325212774 learned_addr learned my addr 172.21.15.114:0/325212774 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-04-23T10:29:07.367 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:29:07.361+0000 7f8239936640 1 --2- 172.21.15.114:0/325212774 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x55d6211edee0 0x55d6211f6560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T10:29:07.367 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:29:07.361+0000 7f823a938640 1 -- 172.21.15.114:0/325212774 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x55d6211edee0 msgr2=0x55d6211f6560 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T10:29:07.367 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:29:07.361+0000 7f823a938640 1 --2- 172.21.15.114:0/325212774 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x55d6211edee0 0x55d6211f6560 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:29:07.367 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:29:07.361+0000 7f823a938640 1 -- 172.21.15.114:0/325212774 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x55d6211ecd00 msgr2=0x55d6211f9220 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T10:29:07.367 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:29:07.361+0000 7f823a938640 1 --2- 172.21.15.114:0/325212774 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x55d6211ecd00 0x55d6211f9220 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:29:07.367 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:29:07.361+0000 7f823a938640 1 -- 172.21.15.114:0/325212774 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55d620f2e620 con 0x55d6211f9950 2024-04-23T10:29:07.367 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:29:07.361+0000 7f823a137640 1 --2- 172.21.15.114:0/325212774 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x55d6211ecd00 0x55d6211f9220 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-23T10:29:07.367 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:29:07.361+0000 7f823a938640 1 --2- 172.21.15.114:0/325212774 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55d6211f9950 0x55d6211f6ac0 secure :-1 s=READY pgs=566 cs=0 l=1 rev1=1 crypto rx=0x7f822c035000 tx=0x7f822c02a8c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b8221e03af3b8339 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T10:29:07.367 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:29:07.361+0000 7f821dffb640 10 client.0 ms_handle_connect on v2:172.21.15.114:3300/0 2024-04-23T10:29:07.367 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:29:07.361+0000 7f821dffb640 1 -- 172.21.15.114:0/325212774 <== mon.0 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f822c051070 con 0x55d6211f9950 2024-04-23T10:29:07.367 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:29:07.361+0000 7f821dffb640 1 -- 172.21.15.114:0/325212774 <== mon.0 v2:172.21.15.114:3300/0 2 ==== osd_map(92..92 src has 1..92) ==== 8585+0+0 (secure 0 0 0) 0x7f822c004240 con 0x55d6211f9950 2024-04-23T10:29:07.367 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:29:07.361+0000 7f823f191740 1 -- 172.21.15.114:0/325212774 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f82240043d0 con 0x55d6211f9950 2024-04-23T10:29:07.367 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:29:07.362+0000 7f821dffb640 1 -- 172.21.15.114:0/325212774 <== mon.0 v2:172.21.15.114:3300/0 3 ==== mon_get_version_reply(handle=1 version=23) ==== 24+0+0 (secure 0 0 0) 0x7f822c004440 con 0x55d6211f9950 2024-04-23T10:29:07.367 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:29:07.362+0000 7f823f191740 10 client.15846 fetch_fsmap learned FSMap version 23 2024-04-23T10:29:07.367 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:29:07.362+0000 7f823f191740 1 -- 172.21.15.114:0/325212774 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({config=0+,fsmap=23,monmap=4+}) -- 0x55d621208790 con 0x55d6211f9950 2024-04-23T10:29:07.368 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:29:07.362+0000 7f821dffb640 1 -- 172.21.15.114:0/325212774 <== mon.0 v2:172.21.15.114:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f82240043d0 con 0x55d6211f9950 2024-04-23T10:29:07.368 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:29:07.362+0000 7f821dffb640 1 -- 172.21.15.114:0/325212774 <== mon.0 v2:172.21.15.114:3300/0 5 ==== fsmap(e 23) ==== 8350+0+0 (secure 0 0 0) 0x7f822c0305c0 con 0x55d6211f9950 2024-04-23T10:29:07.368 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:29:07.362+0000 7f823f191740 10 client.15846 fetch_fsmap finished waiting for FSMap version 23 2024-04-23T10:29:07.368 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:29:07.362+0000 7f823f191740 10 client.15846 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-04-23T10:29:07.368 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:29:07.362+0000 7f823f191740 20 client.15846 populate_metadata read hostname 'smithi114' 2024-04-23T10:29:07.368 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:29:07.362+0000 7f823f191740 1 --2- 172.21.15.114:0/325212774 >> [v2:172.21.15.119:6838/937297340,v1:172.21.15.119:6839/937297340] conn(0x55d621209cf0 0x55d62120c170 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T10:29:07.368 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:29:07.362+0000 7f823f191740 4 client.15846 mds_command: new command op to 24545 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-23T10:29:07.368 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:29:07.362+0000 7f823f191740 1 -- 172.21.15.114:0/325212774 --> [v2:172.21.15.119:6838/937297340,v1:172.21.15.119:6839/937297340] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55d62115d390 con 0x55d621209cf0 2024-04-23T10:29:07.368 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:29:07.362+0000 7f823a137640 1 --2- 172.21.15.114:0/325212774 >> [v2:172.21.15.119:6838/937297340,v1:172.21.15.119:6839/937297340] conn(0x55d621209cf0 0x55d62120c170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T10:29:07.368 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:29:07.363+0000 7f823a137640 1 --2- 172.21.15.114:0/325212774 >> [v2:172.21.15.119:6838/937297340,v1:172.21.15.119:6839/937297340] conn(0x55d621209cf0 0x55d62120c170 crc :-1 s=READY pgs=68 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=16667d94ebe3e4b3 server_cookie=d2355a121086a750 in_seq=0 out_seq=0 2024-04-23T10:29:07.368 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:29:07.363+0000 7f821dffb640 10 client.15846 ms_handle_connect on v2:172.21.15.119:6838/937297340 2024-04-23T10:29:07.368 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:29:07.364+0000 7f821dffb640 1 -- 172.21.15.114:0/325212774 <== mds.0 v2:172.21.15.119:6838/937297340 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x55d62115d390 con 0x55d621209cf0 2024-04-23T10:29:07.368 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:29:07.364+0000 7f821dffb640 10 client.15846 handle_command_reply: tid=0 2024-04-23T10:29:07.368 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:29:07.364+0000 7f823f191740 1 client.15846 shutdown 2024-04-23T10:29:07.368 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:29:07.364+0000 7f823f191740 1 -- 172.21.15.114:0/325212774 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55d6211f9950 msgr2=0x55d6211f6ac0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T10:29:07.368 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:29:07.364+0000 7f823f191740 1 --2- 172.21.15.114:0/325212774 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55d6211f9950 0x55d6211f6ac0 secure :-1 s=READY pgs=566 cs=0 l=1 rev1=1 crypto rx=0x7f822c035000 tx=0x7f822c02a8c0 comp rx=0 tx=0).stop 2024-04-23T10:29:07.369 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:29:07.364+0000 7f823f191740 1 -- 172.21.15.114:0/325212774 shutdown_connections 2024-04-23T10:29:07.369 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:29:07.364+0000 7f823f191740 1 --2- 172.21.15.114:0/325212774 >> [v2:172.21.15.119:6838/937297340,v1:172.21.15.119:6839/937297340] conn(0x55d621209cf0 0x55d62120c170 crc :-1 s=READY pgs=68 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:29:07.369 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:29:07.364+0000 7f823f191740 1 --2- 172.21.15.114:0/325212774 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55d6211f9950 0x55d6211f6ac0 unknown :-1 s=CLOSED pgs=566 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:29:07.369 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:29:07.364+0000 7f821dffb640 0 client.15846 ms_handle_reset on v2:172.21.15.119:6838/937297340 2024-04-23T10:29:07.369 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:29:07.364+0000 7f823f191740 1 --2- 172.21.15.114:0/325212774 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x55d6211edee0 0x55d6211f6560 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:29:07.369 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:29:07.364+0000 7f823f191740 1 --2- 172.21.15.114:0/325212774 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x55d6211ecd00 0x55d6211f9220 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:29:07.369 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:29:07.364+0000 7f823f191740 1 -- 172.21.15.114:0/325212774 >> 172.21.15.114:0/325212774 conn(0x55d6211e8d40 msgr2=0x55d6211eebb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T10:29:07.369 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:29:07.364+0000 7f823f191740 1 -- 172.21.15.114:0/325212774 shutdown_connections 2024-04-23T10:29:07.369 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:29:07.364+0000 7f823f191740 1 -- 172.21.15.114:0/325212774 wait complete. 2024-04-23T10:29:07.369 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:29:07.364+0000 7f823f191740 20 client.15846 trim_cache size 0 max 16384 2024-04-23T10:29:07.383 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:29:07.379+0000 7f823f191740 1 Processor -- start 2024-04-23T10:29:07.383 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:29:07.379+0000 7f823f191740 1 -- start start 2024-04-23T10:29:07.383 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:29:07.380+0000 7f823f191740 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x55d6211ecd00 0x55d6212056e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T10:29:07.383 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:29:07.380+0000 7f823f191740 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x55d6211edee0 0x55d6211e5420 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T10:29:07.383 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:29:07.380+0000 7f823f191740 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55d6211f9950 0x55d6211f8d50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T10:29:07.383 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:29:07.380+0000 7f823f191740 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x7f82341ae2d0 con 0x55d6211f9950 2024-04-23T10:29:07.383 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:29:07.380+0000 7f823f191740 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 -- 0x7f82341abb70 con 0x55d6211ecd00 2024-04-23T10:29:07.384 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:29:07.380+0000 7f823f191740 1 -- --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_getmap magic: 0 -- 0x7f82341a99c0 con 0x55d6211edee0 2024-04-23T10:29:07.384 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:29:07.380+0000 7f823a938640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55d6211f9950 0x55d6211f8d50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T10:29:07.384 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:29:07.380+0000 7f823a137640 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x55d6211ecd00 0x55d6212056e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T10:29:07.384 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:29:07.380+0000 7f823a137640 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x55d6211ecd00 0x55d6212056e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.119:3300/0 says I am v2:172.21.15.114:44038/0 (socket says 172.21.15.114:44038) 2024-04-23T10:29:07.384 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:29:07.380+0000 7f823a137640 1 -- 172.21.15.114:0/3245474757 learned_addr learned my addr 172.21.15.114:0/3245474757 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-04-23T10:29:07.384 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:29:07.380+0000 7f8239936640 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x55d6211edee0 0x55d6211e5420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T10:29:07.384 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:29:07.380+0000 7f823a938640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55d6211f9950 0x55d6211f8d50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.114:42832/0 (socket says 172.21.15.114:42832) 2024-04-23T10:29:07.384 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:29:07.380+0000 7f823a938640 1 -- 172.21.15.114:0/3245474757 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x55d6211edee0 msgr2=0x55d6211e5420 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T10:29:07.384 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:29:07.380+0000 7f823a938640 1 --2- 172.21.15.114:0/3245474757 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x55d6211edee0 0x55d6211e5420 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:29:07.384 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:29:07.380+0000 7f823a938640 1 -- 172.21.15.114:0/3245474757 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x55d6211ecd00 msgr2=0x55d6212056e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T10:29:07.384 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:29:07.380+0000 7f823a938640 1 --2- 172.21.15.114:0/3245474757 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x55d6211ecd00 0x55d6212056e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:29:07.384 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:29:07.380+0000 7f823a938640 1 -- 172.21.15.114:0/3245474757 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55d6211ed880 con 0x55d6211f9950 2024-04-23T10:29:07.384 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:29:07.380+0000 7f823a938640 1 --2- 172.21.15.114:0/3245474757 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55d6211f9950 0x55d6211f8d50 secure :-1 s=READY pgs=567 cs=0 l=1 rev1=1 crypto rx=0x7f822c009ca0 tx=0x7f822c036000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e4bb991e1d98596e server_cookie=0 in_seq=0 out_seq=0 2024-04-23T10:29:07.384 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:29:07.380+0000 7f821ffff640 1 -- 172.21.15.114:0/3245474757 <== mon.0 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f822c063030 con 0x55d6211f9950 2024-04-23T10:29:07.384 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:29:07.380+0000 7f821ffff640 1 -- 172.21.15.114:0/3245474757 <== mon.0 v2:172.21.15.114:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f822c036070 con 0x55d6211f9950 2024-04-23T10:29:07.385 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:29:07.380+0000 7f821ffff640 1 -- 172.21.15.114:0/3245474757 <== mon.0 v2:172.21.15.114:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f822c051070 con 0x55d6211f9950 2024-04-23T10:29:07.385 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:29:07.381+0000 7f823f191740 1 -- 172.21.15.114:0/3245474757 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55d6211f9950 msgr2=0x55d6211f8d50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T10:29:07.385 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:29:07.381+0000 7f823f191740 1 --2- 172.21.15.114:0/3245474757 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55d6211f9950 0x55d6211f8d50 secure :-1 s=READY pgs=567 cs=0 l=1 rev1=1 crypto rx=0x7f822c009ca0 tx=0x7f822c036000 comp rx=0 tx=0).stop 2024-04-23T10:29:07.385 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:29:07.381+0000 7f823f191740 1 -- 172.21.15.114:0/3245474757 shutdown_connections 2024-04-23T10:29:07.385 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:29:07.381+0000 7f823f191740 1 --2- 172.21.15.114:0/3245474757 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55d6211f9950 0x55d6211f8d50 unknown :-1 s=CLOSED pgs=567 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:29:07.385 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:29:07.381+0000 7f823f191740 1 --2- 172.21.15.114:0/3245474757 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x55d6211edee0 0x55d6211e5420 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:29:07.385 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:29:07.381+0000 7f823f191740 1 --2- 172.21.15.114:0/3245474757 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x55d6211ecd00 0x55d6212056e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:29:07.385 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:29:07.381+0000 7f823f191740 1 -- 172.21.15.114:0/3245474757 >> 172.21.15.114:0/3245474757 conn(0x55d6211e8d40 msgr2=0x55d6211f4e30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T10:29:07.385 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:29:07.381+0000 7f823f191740 1 -- 172.21.15.114:0/3245474757 shutdown_connections 2024-04-23T10:29:07.385 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:29:07.381+0000 7f823f191740 1 -- 172.21.15.114:0/3245474757 wait complete. 2024-04-23T10:29:07.385 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:29:07.381+0000 7f823f191740 1 -- start start 2024-04-23T10:29:07.385 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:29:07.382+0000 7f823f191740 1 Processor -- start 2024-04-23T10:29:07.385 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:29:07.382+0000 7f823f191740 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x55d6211ecd00 0x55d621211040 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T10:29:07.385 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:29:07.382+0000 7f823f191740 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55d6211edee0 0x55d621211580 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T10:29:07.385 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:29:07.382+0000 7f823f191740 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x55d6211f9950 0x55d621211ac0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T10:29:07.385 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:29:07.382+0000 7f823f191740 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x7f82341aa030 con 0x55d6211edee0 2024-04-23T10:29:07.385 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:29:07.382+0000 7f823f191740 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 -- 0x7f82341ab4a0 con 0x55d6211f9950 2024-04-23T10:29:07.386 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:29:07.382+0000 7f823f191740 1 -- --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_getmap magic: 0 -- 0x7f82341aa700 con 0x55d6211ecd00 2024-04-23T10:29:07.386 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:29:07.382+0000 7f823f191740 10 client.0 _ll_register_callbacks cb 0x7f822c031810 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-23T10:29:07.386 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:29:07.382+0000 7f8239936640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55d6211edee0 0x55d621211580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T10:29:07.386 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:29:07.382+0000 7f8239936640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55d6211edee0 0x55d621211580 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.114:42846/0 (socket says 172.21.15.114:42846) 2024-04-23T10:29:07.386 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:29:07.382+0000 7f823a137640 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x55d6211ecd00 0x55d621211040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T10:29:07.386 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:29:07.382+0000 7f8239936640 1 -- 172.21.15.114:0/749787510 learned_addr learned my addr 172.21.15.114:0/749787510 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-04-23T10:29:07.386 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:29:07.382+0000 7f823a938640 1 --2- 172.21.15.114:0/749787510 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x55d6211f9950 0x55d621211ac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T10:29:07.386 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:29:07.382+0000 7f823a137640 1 -- 172.21.15.114:0/749787510 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x55d6211f9950 msgr2=0x55d621211ac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T10:29:07.386 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:29:07.382+0000 7f823a137640 1 --2- 172.21.15.114:0/749787510 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x55d6211f9950 0x55d621211ac0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:29:07.386 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:29:07.382+0000 7f823a137640 1 -- 172.21.15.114:0/749787510 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55d6211edee0 msgr2=0x55d621211580 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T10:29:07.386 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:29:07.382+0000 7f823a137640 1 --2- 172.21.15.114:0/749787510 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55d6211edee0 0x55d621211580 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:29:07.386 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:29:07.383+0000 7f823a137640 1 -- 172.21.15.114:0/749787510 --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55d62126f360 con 0x55d6211ecd00 2024-04-23T10:29:07.386 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:29:07.383+0000 7f8239936640 1 --2- 172.21.15.114:0/749787510 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55d6211edee0 0x55d621211580 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-23T10:29:07.386 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:29:07.383+0000 7f823a137640 1 --2- 172.21.15.114:0/749787510 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x55d6211ecd00 0x55d621211040 secure :-1 s=READY pgs=65 cs=0 l=1 rev1=1 crypto rx=0x7f8230003e40 tx=0x7f82300d2ff0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=eb013bd18ffb4240 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T10:29:07.387 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:29:07.383+0000 7f821dffb640 10 client.0 ms_handle_connect on v2:172.21.15.204:3300/0 2024-04-23T10:29:07.387 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:29:07.383+0000 7f821dffb640 1 -- 172.21.15.114:0/749787510 <== mon.1 v2:172.21.15.204:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f82300a56b0 con 0x55d6211ecd00 2024-04-23T10:29:07.387 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:29:07.383+0000 7f823f191740 1 -- 172.21.15.114:0/749787510 --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f822c036070 con 0x55d6211ecd00 2024-04-23T10:29:07.387 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:29:07.383+0000 7f821dffb640 1 -- 172.21.15.114:0/749787510 <== mon.1 v2:172.21.15.204:3300/0 2 ==== osd_map(92..92 src has 1..92) ==== 8585+0+0 (secure 0 0 0) 0x7f8230004000 con 0x55d6211ecd00 2024-04-23T10:29:07.387 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:29:07.383+0000 7f821dffb640 1 -- 172.21.15.114:0/749787510 <== mon.1 v2:172.21.15.204:3300/0 3 ==== mon_get_version_reply(handle=1 version=23) ==== 24+0+0 (secure 0 0 0) 0x7f82300ce470 con 0x55d6211ecd00 2024-04-23T10:29:07.387 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:29:07.383+0000 7f823f191740 10 client.25288 fetch_fsmap learned FSMap version 23 2024-04-23T10:29:07.387 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:29:07.383+0000 7f823f191740 1 -- 172.21.15.114:0/749787510 --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_subscribe({config=0+,fsmap=23,monmap=4+}) -- 0x55d621270650 con 0x55d6211ecd00 2024-04-23T10:29:07.387 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:29:07.383+0000 7f821dffb640 1 -- 172.21.15.114:0/749787510 <== mon.1 v2:172.21.15.204:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f822c036070 con 0x55d6211ecd00 2024-04-23T10:29:07.387 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:29:07.383+0000 7f821dffb640 1 -- 172.21.15.114:0/749787510 <== mon.1 v2:172.21.15.204:3300/0 5 ==== fsmap(e 23) ==== 8350+0+0 (secure 0 0 0) 0x7f8230098790 con 0x55d6211ecd00 2024-04-23T10:29:07.387 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:29:07.383+0000 7f823f191740 10 client.25288 fetch_fsmap finished waiting for FSMap version 23 2024-04-23T10:29:07.387 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:29:07.383+0000 7f823f191740 10 client.25288 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-04-23T10:29:07.387 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:29:07.384+0000 7f823f191740 20 client.25288 populate_metadata read hostname 'smithi114' 2024-04-23T10:29:07.387 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:29:07.384+0000 7f823f191740 1 --2- 172.21.15.114:0/749787510 >> [v2:172.21.15.119:6838/937297340,v1:172.21.15.119:6839/937297340] conn(0x55d621270de0 0x55d621273200 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T10:29:07.387 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:29:07.384+0000 7f823f191740 4 client.25288 mds_command: new command op to 24545 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-04-23T10:29:07.387 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:29:07.384+0000 7f823f191740 1 -- 172.21.15.114:0/749787510 --> [v2:172.21.15.119:6838/937297340,v1:172.21.15.119:6839/937297340] -- command(tid 0: {"prefix": "flush journal"}) -- 0x55d621157200 con 0x55d621270de0 2024-04-23T10:29:07.387 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:29:07.384+0000 7f8239936640 1 --2- 172.21.15.114:0/749787510 >> [v2:172.21.15.119:6838/937297340,v1:172.21.15.119:6839/937297340] conn(0x55d621270de0 0x55d621273200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T10:29:07.388 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:29:07.384+0000 7f821dffb640 10 client.25288 ms_handle_connect on v2:172.21.15.119:6838/937297340 2024-04-23T10:29:07.388 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:29:07.384+0000 7f8239936640 1 --2- 172.21.15.114:0/749787510 >> [v2:172.21.15.119:6838/937297340,v1:172.21.15.119:6839/937297340] conn(0x55d621270de0 0x55d621273200 crc :-1 s=READY pgs=69 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=f7bd680508fe70b3 server_cookie=6d077930d72ed33f in_seq=0 out_seq=0 2024-04-23T10:29:07.454 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:29:07.454+0000 7f821dffb640 1 -- 172.21.15.114:0/749787510 <== mds.0 v2:172.21.15.119:6838/937297340 1 ==== command_reply(tid 0: 0 ) ==== 8+0+44 (crc 0 0 0) 0x55d621157200 con 0x55d621270de0 2024-04-23T10:29:07.454 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:29:07.454+0000 7f821dffb640 10 client.25288 handle_command_reply: tid=0 2024-04-23T10:29:07.454 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:29:07.455+0000 7f823f191740 1 client.25288 shutdown 2024-04-23T10:29:07.454 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:29:07.455+0000 7f823f191740 1 -- 172.21.15.114:0/749787510 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x55d6211ecd00 msgr2=0x55d621211040 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T10:29:07.454 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:29:07.455+0000 7f823f191740 1 --2- 172.21.15.114:0/749787510 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x55d6211ecd00 0x55d621211040 secure :-1 s=READY pgs=65 cs=0 l=1 rev1=1 crypto rx=0x7f8230003e40 tx=0x7f82300d2ff0 comp rx=0 tx=0).stop 2024-04-23T10:29:07.455 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:29:07.455+0000 7f823f191740 1 -- 172.21.15.114:0/749787510 shutdown_connections 2024-04-23T10:29:07.455 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:29:07.455+0000 7f823f191740 1 --2- 172.21.15.114:0/749787510 >> [v2:172.21.15.119:6838/937297340,v1:172.21.15.119:6839/937297340] conn(0x55d621270de0 0x55d621273200 crc :-1 s=READY pgs=69 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:29:07.455 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:29:07.455+0000 7f823f191740 1 --2- 172.21.15.114:0/749787510 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x55d6211f9950 0x55d621211ac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:29:07.455 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:29:07.455+0000 7f823f191740 1 --2- 172.21.15.114:0/749787510 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55d6211edee0 0x55d621211580 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:29:07.455 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:29:07.455+0000 7f821dffb640 0 client.25288 ms_handle_reset on v2:172.21.15.119:6838/937297340 2024-04-23T10:29:07.455 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:29:07.455+0000 7f823f191740 1 --2- 172.21.15.114:0/749787510 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x55d6211ecd00 0x55d621211040 unknown :-1 s=CLOSED pgs=65 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:29:07.455 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:29:07.455+0000 7f823f191740 1 -- 172.21.15.114:0/749787510 >> 172.21.15.114:0/749787510 conn(0x55d6211e8d40 msgr2=0x55d6211e73c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T10:29:07.455 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:29:07.455+0000 7f823f191740 1 -- 172.21.15.114:0/749787510 shutdown_connections 2024-04-23T10:29:07.455 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:29:07.455+0000 7f823f191740 1 -- 172.21.15.114:0/749787510 wait complete. 2024-04-23T10:29:07.455 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:29:07.455+0000 7f823f191740 20 client.25288 trim_cache size 0 max 16384 2024-04-23T10:29:07.455 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stdout:{ 2024-04-23T10:29:07.455 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stdout: "message": "", 2024-04-23T10:29:07.456 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stdout: "return_code": 0 2024-04-23T10:29:07.456 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stdout:} 2024-04-23T10:29:07.456 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:29:07.456+0000 7f823c39b640 1 -- 172.21.15.114:0/2647969526 >> [v2:172.21.15.114:6800/1395693466,v1:172.21.15.114:6801/1395693466] conn(0x7f82200a3dc0 msgr2=0x7f82200a6280 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T10:29:07.457 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:29:07.456+0000 7f823c39b640 1 --2- 172.21.15.114:0/2647969526 >> [v2:172.21.15.114:6800/1395693466,v1:172.21.15.114:6801/1395693466] conn(0x7f82200a3dc0 0x7f82200a6280 secure :-1 s=READY pgs=80 cs=0 l=1 rev1=1 crypto rx=0x7f822c009a60 tx=0x7f822c0095c0 comp rx=0 tx=0).stop 2024-04-23T10:29:07.457 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:29:07.456+0000 7f823c39b640 1 -- 172.21.15.114:0/2647969526 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f82341010f0 msgr2=0x7f82341975a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T10:29:07.457 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:29:07.456+0000 7f823c39b640 1 --2- 172.21.15.114:0/2647969526 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f82341010f0 0x7f82341975a0 secure :-1 s=READY pgs=565 cs=0 l=1 rev1=1 crypto rx=0x7f823000ea60 tx=0x7f823000ef20 comp rx=0 tx=0).stop 2024-04-23T10:29:07.457 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:29:07.456+0000 7f823c39b640 1 -- 172.21.15.114:0/2647969526 shutdown_connections 2024-04-23T10:29:07.457 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:29:07.456+0000 7f823c39b640 1 --2- 172.21.15.114:0/2647969526 >> [v2:172.21.15.114:6800/1395693466,v1:172.21.15.114:6801/1395693466] conn(0x7f82200a3dc0 0x7f82200a6280 unknown :-1 s=CLOSED pgs=80 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:29:07.457 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:29:07.456+0000 7f823c39b640 1 --2- 172.21.15.114:0/2647969526 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f823419be10 0x7f823419c2c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:29:07.457 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:29:07.456+0000 7f823c39b640 1 --2- 172.21.15.114:0/2647969526 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f8234106170 0x7f8234197ae0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:29:07.457 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:29:07.456+0000 7f823c39b640 1 --2- 172.21.15.114:0/2647969526 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f82341010f0 0x7f82341975a0 unknown :-1 s=CLOSED pgs=565 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:29:07.457 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:29:07.456+0000 7f823c39b640 1 -- 172.21.15.114:0/2647969526 >> 172.21.15.114:0/2647969526 conn(0x7f8234077fd0 msgr2=0x7f82340fef50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T10:29:07.457 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:29:07.456+0000 7f823c39b640 1 -- 172.21.15.114:0/2647969526 shutdown_connections 2024-04-23T10:29:07.457 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:29:07.456+0000 7f823c39b640 1 -- 172.21.15.114:0/2647969526 wait complete. 2024-04-23T10:29:07.910 INFO:tasks.workunit.client.0.smithi114.stdout: 1 159755 11.58 MB/sec warmup 76 sec latency 71.685 ms 2024-04-23T10:29:08.242 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 23 10:29:07 smithi119 ceph-mon[32608]: pgmap v78: 97 pgs: 97 active+clean; 62 MiB data, 797 MiB used, 1.0 TiB / 1.0 TiB avail; 4.1 MiB/s rd, 4.2 MiB/s wr, 468 op/s 2024-04-23T10:29:08.253 INFO:journalctl@ceph.mon.a.smithi114.stdout:Apr 23 10:29:07 smithi114 ceph-mon[29255]: pgmap v78: 97 pgs: 97 active+clean; 62 MiB data, 797 MiB used, 1.0 TiB / 1.0 TiB avail; 4.1 MiB/s rd, 4.2 MiB/s wr, 468 op/s 2024-04-23T10:29:08.328 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 23 10:29:07 smithi204 ceph-mon[32649]: pgmap v78: 97 pgs: 97 active+clean; 62 MiB data, 797 MiB used, 1.0 TiB / 1.0 TiB avail; 4.1 MiB/s rd, 4.2 MiB/s wr, 468 op/s 2024-04-23T10:29:08.910 INFO:tasks.workunit.client.0.smithi114.stdout: 1 160877 11.48 MB/sec warmup 77 sec latency 46.018 ms 2024-04-23T10:29:09.910 INFO:tasks.workunit.client.0.smithi114.stdout: 1 163462 11.54 MB/sec warmup 78 sec latency 38.191 ms 2024-04-23T10:29:10.242 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 23 10:29:09 smithi119 ceph-mon[32608]: pgmap v79: 97 pgs: 97 active+clean; 62 MiB data, 797 MiB used, 1.0 TiB / 1.0 TiB avail; 2.8 MiB/s rd, 2.8 MiB/s wr, 293 op/s 2024-04-23T10:29:10.254 INFO:journalctl@ceph.mon.a.smithi114.stdout:Apr 23 10:29:09 smithi114 ceph-mon[29255]: pgmap v79: 97 pgs: 97 active+clean; 62 MiB data, 797 MiB used, 1.0 TiB / 1.0 TiB avail; 2.8 MiB/s rd, 2.8 MiB/s wr, 293 op/s 2024-04-23T10:29:10.328 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 23 10:29:09 smithi204 ceph-mon[32649]: pgmap v79: 97 pgs: 97 active+clean; 62 MiB data, 797 MiB used, 1.0 TiB / 1.0 TiB avail; 2.8 MiB/s rd, 2.8 MiB/s wr, 293 op/s 2024-04-23T10:29:10.910 INFO:tasks.workunit.client.0.smithi114.stdout: 1 165383 11.47 MB/sec warmup 79 sec latency 15.557 ms 2024-04-23T10:29:11.910 INFO:tasks.workunit.client.0.smithi114.stdout: 1 167303 11.50 MB/sec warmup 80 sec latency 13.093 ms 2024-04-23T10:29:12.242 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 23 10:29:11 smithi119 ceph-mon[32608]: pgmap v80: 97 pgs: 97 active+clean; 62 MiB data, 797 MiB used, 1.0 TiB / 1.0 TiB avail; 2.8 MiB/s rd, 2.8 MiB/s wr, 293 op/s 2024-04-23T10:29:12.254 INFO:journalctl@ceph.mon.a.smithi114.stdout:Apr 23 10:29:11 smithi114 ceph-mon[29255]: pgmap v80: 97 pgs: 97 active+clean; 62 MiB data, 797 MiB used, 1.0 TiB / 1.0 TiB avail; 2.8 MiB/s rd, 2.8 MiB/s wr, 293 op/s 2024-04-23T10:29:12.328 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 23 10:29:11 smithi204 ceph-mon[32649]: pgmap v80: 97 pgs: 97 active+clean; 62 MiB data, 797 MiB used, 1.0 TiB / 1.0 TiB avail; 2.8 MiB/s rd, 2.8 MiB/s wr, 293 op/s 2024-04-23T10:29:12.910 INFO:tasks.workunit.client.0.smithi114.stdout: 1 170094 11.53 MB/sec warmup 81 sec latency 14.930 ms 2024-04-23T10:29:13.910 INFO:tasks.workunit.client.0.smithi114.stdout: 1 171216 11.46 MB/sec warmup 82 sec latency 48.955 ms 2024-04-23T10:29:14.242 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 23 10:29:13 smithi119 ceph-mon[32608]: pgmap v81: 97 pgs: 97 active+clean; 58 MiB data, 855 MiB used, 1.0 TiB / 1.0 TiB avail; 4.1 MiB/s rd, 4.4 MiB/s wr, 471 op/s 2024-04-23T10:29:14.254 INFO:journalctl@ceph.mon.a.smithi114.stdout:Apr 23 10:29:13 smithi114 ceph-mon[29255]: pgmap v81: 97 pgs: 97 active+clean; 58 MiB data, 855 MiB used, 1.0 TiB / 1.0 TiB avail; 4.1 MiB/s rd, 4.4 MiB/s wr, 471 op/s 2024-04-23T10:29:14.328 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 23 10:29:13 smithi204 ceph-mon[32649]: pgmap v81: 97 pgs: 97 active+clean; 58 MiB data, 855 MiB used, 1.0 TiB / 1.0 TiB avail; 4.1 MiB/s rd, 4.4 MiB/s wr, 471 op/s 2024-04-23T10:29:14.910 INFO:tasks.workunit.client.0.smithi114.stdout: 1 174043 11.54 MB/sec warmup 83 sec latency 26.154 ms 2024-04-23T10:29:15.910 INFO:tasks.workunit.client.0.smithi114.stdout: 1 175517 11.47 MB/sec warmup 84 sec latency 15.346 ms 2024-04-23T10:29:16.118 INFO:tasks.quiescer.fs.[cephfs]:Total duration (42.9) is longer than `split_if_longer` (35.0), will split into two consecutive quiesces 2024-04-23T10:29:16.118 DEBUG:tasks.quiescer.fs.[cephfs]:Going to quiesce for duration: 21.45 2024-04-23T10:29:16.119 DEBUG:teuthology.orchestra.run.smithi114:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-23T10:29:16.223 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:29:16.224+0000 7fb170eef640 1 -- 172.21.15.114:0/2222135471 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fb16c106170 msgr2=0x7fb16c10dd40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T10:29:16.224 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:29:16.224+0000 7fb170eef640 1 --2- 172.21.15.114:0/2222135471 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fb16c106170 0x7fb16c10dd40 secure :-1 s=READY pgs=568 cs=0 l=1 rev1=1 crypto rx=0x7fb15c009900 tx=0x7fb15c02a3f0 comp rx=0 tx=0).stop 2024-04-23T10:29:16.224 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:29:16.224+0000 7fb170eef640 1 -- 172.21.15.114:0/2222135471 shutdown_connections 2024-04-23T10:29:16.224 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:29:16.224+0000 7fb170eef640 1 --2- 172.21.15.114:0/2222135471 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fb16c106170 0x7fb16c10dd40 unknown :-1 s=CLOSED pgs=568 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:29:16.224 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:29:16.224+0000 7fb170eef640 1 --2- 172.21.15.114:0/2222135471 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fb16c101a50 0x7fb16c105a90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:29:16.224 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:29:16.224+0000 7fb170eef640 1 --2- 172.21.15.114:0/2222135471 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fb16c1010a0 0x7fb16c101480 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:29:16.224 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:29:16.224+0000 7fb170eef640 1 -- 172.21.15.114:0/2222135471 >> 172.21.15.114:0/2222135471 conn(0x7fb16c077f50 msgr2=0x7fb16c078360 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T10:29:16.224 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:29:16.224+0000 7fb170eef640 1 -- 172.21.15.114:0/2222135471 shutdown_connections 2024-04-23T10:29:16.224 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:29:16.225+0000 7fb170eef640 1 -- 172.21.15.114:0/2222135471 wait complete. 2024-04-23T10:29:16.226 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:29:16.225+0000 7fb170eef640 1 Processor -- start 2024-04-23T10:29:16.226 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:29:16.225+0000 7fb170eef640 1 -- start start 2024-04-23T10:29:16.227 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:29:16.225+0000 7fb170eef640 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fb16c1010a0 0x7fb16c199540 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T10:29:16.227 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:29:16.225+0000 7fb170eef640 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fb16c106170 0x7fb16c199a80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T10:29:16.227 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:29:16.225+0000 7fb170eef640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fb16c19a010 0x7fb16c1a45c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T10:29:16.227 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:29:16.225+0000 7fb170eef640 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x7fb16c1110e0 con 0x7fb16c19a010 2024-04-23T10:29:16.227 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:29:16.225+0000 7fb170eef640 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 -- 0x7fb16c110f60 con 0x7fb16c1010a0 2024-04-23T10:29:16.227 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:29:16.225+0000 7fb170eef640 1 -- --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_getmap magic: 0 -- 0x7fb16c111260 con 0x7fb16c106170 2024-04-23T10:29:16.227 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:29:16.226+0000 7fb16ad9d640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fb16c19a010 0x7fb16c1a45c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T10:29:16.227 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:29:16.226+0000 7fb16a59c640 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fb16c1010a0 0x7fb16c199540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T10:29:16.227 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:29:16.226+0000 7fb16ad9d640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fb16c19a010 0x7fb16c1a45c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.114:54414/0 (socket says 172.21.15.114:54414) 2024-04-23T10:29:16.227 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:29:16.226+0000 7fb16ad9d640 1 -- 172.21.15.114:0/508373081 learned_addr learned my addr 172.21.15.114:0/508373081 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-04-23T10:29:16.227 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:29:16.226+0000 7fb169d9b640 1 --2- 172.21.15.114:0/508373081 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fb16c106170 0x7fb16c199a80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T10:29:16.227 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:29:16.226+0000 7fb16ad9d640 1 -- 172.21.15.114:0/508373081 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fb16c106170 msgr2=0x7fb16c199a80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T10:29:16.227 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:29:16.226+0000 7fb16ad9d640 1 --2- 172.21.15.114:0/508373081 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fb16c106170 0x7fb16c199a80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:29:16.227 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:29:16.226+0000 7fb16ad9d640 1 -- 172.21.15.114:0/508373081 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fb16c1010a0 msgr2=0x7fb16c199540 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T10:29:16.228 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:29:16.226+0000 7fb16ad9d640 1 --2- 172.21.15.114:0/508373081 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fb16c1010a0 0x7fb16c199540 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:29:16.228 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:29:16.226+0000 7fb16ad9d640 1 -- 172.21.15.114:0/508373081 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb16c1a4dc0 con 0x7fb16c19a010 2024-04-23T10:29:16.228 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:29:16.226+0000 7fb16a59c640 1 --2- 172.21.15.114:0/508373081 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fb16c1010a0 0x7fb16c199540 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-23T10:29:16.228 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:29:16.226+0000 7fb16ad9d640 1 --2- 172.21.15.114:0/508373081 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fb16c19a010 0x7fb16c1a45c0 secure :-1 s=READY pgs=569 cs=0 l=1 rev1=1 crypto rx=0x7fb15c004770 tx=0x7fb15c004040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T10:29:16.228 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:29:16.226+0000 7fb15b7fe640 1 -- 172.21.15.114:0/508373081 <== mon.0 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb15c03f030 con 0x7fb16c19a010 2024-04-23T10:29:16.228 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:29:16.226+0000 7fb15b7fe640 1 -- 172.21.15.114:0/508373081 <== mon.0 v2:172.21.15.114:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fb15c004440 con 0x7fb16c19a010 2024-04-23T10:29:16.228 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:29:16.226+0000 7fb170eef640 1 -- 172.21.15.114:0/508373081 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fb16c1a5050 con 0x7fb16c19a010 2024-04-23T10:29:16.228 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:29:16.226+0000 7fb15b7fe640 1 -- 172.21.15.114:0/508373081 <== mon.0 v2:172.21.15.114:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb15c035070 con 0x7fb16c19a010 2024-04-23T10:29:16.228 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:29:16.226+0000 7fb170eef640 1 -- 172.21.15.114:0/508373081 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fb16c1a5540 con 0x7fb16c19a010 2024-04-23T10:29:16.228 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:29:16.227+0000 7fb170eef640 1 -- 172.21.15.114:0/508373081 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fb16c068d70 con 0x7fb16c19a010 2024-04-23T10:29:16.229 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:29:16.228+0000 7fb15b7fe640 1 -- 172.21.15.114:0/508373081 <== mon.0 v2:172.21.15.114:3300/0 4 ==== mgrmap(e 26) ==== 138778+0+0 (secure 0 0 0) 0x7fb15c042030 con 0x7fb16c19a010 2024-04-23T10:29:16.229 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:29:16.229+0000 7fb15b7fe640 1 --2- 172.21.15.114:0/508373081 >> [v2:172.21.15.114:6800/1395693466,v1:172.21.15.114:6801/1395693466] conn(0x7fb1540a3d20 0x7fb1540a61e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T10:29:16.229 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:29:16.229+0000 7fb16a59c640 1 --2- 172.21.15.114:0/508373081 >> [v2:172.21.15.114:6800/1395693466,v1:172.21.15.114:6801/1395693466] conn(0x7fb1540a3d20 0x7fb1540a61e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T10:29:16.229 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:29:16.229+0000 7fb15b7fe640 1 -- 172.21.15.114:0/508373081 <== mon.0 v2:172.21.15.114:3300/0 5 ==== osd_map(92..92 src has 1..92) ==== 8585+0+0 (secure 0 0 0) 0x7fb15c0f0b20 con 0x7fb16c19a010 2024-04-23T10:29:16.229 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:29:16.229+0000 7fb16a59c640 1 --2- 172.21.15.114:0/508373081 >> [v2:172.21.15.114:6800/1395693466,v1:172.21.15.114:6801/1395693466] conn(0x7fb1540a3d20 0x7fb1540a61e0 secure :-1 s=READY pgs=81 cs=0 l=1 rev1=1 crypto rx=0x7fb164009fc0 tx=0x7fb164005f90 comp rx=0 tx=0).ready entity=mgr.15291 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T10:29:16.232 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:29:16.233+0000 7fb15b7fe640 1 -- 172.21.15.114:0/508373081 <== mon.0 v2:172.21.15.114:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7fb15c0bcb30 con 0x7fb16c19a010 2024-04-23T10:29:16.241 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 23 10:29:15 smithi119 ceph-mon[32608]: pgmap v82: 97 pgs: 97 active+clean; 58 MiB data, 855 MiB used, 1.0 TiB / 1.0 TiB avail; 2.6 MiB/s rd, 3.0 MiB/s wr, 324 op/s 2024-04-23T10:29:16.253 INFO:journalctl@ceph.mon.a.smithi114.stdout:Apr 23 10:29:15 smithi114 ceph-mon[29255]: pgmap v82: 97 pgs: 97 active+clean; 58 MiB data, 855 MiB used, 1.0 TiB / 1.0 TiB avail; 2.6 MiB/s rd, 3.0 MiB/s wr, 324 op/s 2024-04-23T10:29:16.327 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 23 10:29:15 smithi204 ceph-mon[32649]: pgmap v82: 97 pgs: 97 active+clean; 58 MiB data, 855 MiB used, 1.0 TiB / 1.0 TiB avail; 2.6 MiB/s rd, 3.0 MiB/s wr, 324 op/s 2024-04-23T10:29:16.462 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:29:16.464+0000 7fb170eef640 1 -- 172.21.15.114:0/508373081 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fb16c109400 con 0x7fb16c19a010 2024-04-23T10:29:16.463 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:29:16.464+0000 7fb15b7fe640 1 -- 172.21.15.114:0/508373081 <== mon.0 v2:172.21.15.114:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 23 v23) ==== 94+0+11888 (secure 0 0 0) 0x7fb15c0c0e30 con 0x7fb16c19a010 2024-04-23T10:29:16.463 INFO:teuthology.orchestra.run.smithi114.stdout: 2024-04-23T10:29:16.463 INFO:teuthology.orchestra.run.smithi114.stdout:{"epoch":23,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14547,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.114:6835/663493580","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6834","nonce":663493580},{"type":"v1","addr":"172.21.15.114:6835","nonce":663493580}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14565,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.114:6837/1439252154","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6836","nonce":1439252154},{"type":"v1","addr":"172.21.15.114:6837","nonce":1439252154}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5},{"gid":14634,"name":"g","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.114:6839/3849796939","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6838","nonce":3849796939},{"type":"v1","addr":"172.21.15.114:6839","nonce":3849796939}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":11},{"gid":14646,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.114:6841/3544523891","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6840","nonce":3544523891},{"type":"v1","addr":"172.21.15.114:6841","nonce":3544523891}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":12},{"gid":24385,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.204:6833/3866561754","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6832","nonce":3866561754},{"type":"v1","addr":"172.21.15.204:6833","nonce":3866561754}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2},{"gid":24427,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.204:6835/2533793172","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6834","nonce":2533793172},{"type":"v1","addr":"172.21.15.204:6835","nonce":2533793172}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":7}],"filesystems":[{"mdsmap":{"epoch":23,"flags":306,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":true},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-04-23T10:24:43.383158+0000","modified":"2024-04-23T10:28:39.149342+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":3,"in":[0,1,2],"up":{"mds_0":24545,"mds_1":24509,"mds_2":24479},"failed":[],"damaged":[],"stopped":[],"info":{"gid_24439":{"gid":24439,"name":"c","rank":2,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.204:6837/513822179","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6836","nonce":513822179},{"type":"v1","addr":"172.21.15.204:6837","nonce":513822179}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24443":{"gid":24443,"name":"k","rank":1,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.119:6833/783244660","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6832","nonce":783244660},{"type":"v1","addr":"172.21.15.119:6833","nonce":783244660}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24463":{"gid":24463,"name":"i","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.204:6839/3222658893","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6838","nonce":3222658893},{"type":"v1","addr":"172.21.15.204:6839","nonce":3222658893}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24479":{"gid":24479,"name":"e","rank":2,"incarnation":19,"state":"up:active","state_seq":9,"addr":"172.21.15.119:6835/2033916709","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6834","nonce":2033916709},{"type":"v1","addr":"172.21.15.119:6835","nonce":2033916709}]},"join_fscid":-1,"export_targets":[0],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24509":{"gid":24509,"name":"h","rank":1,"incarnation":17,"state":"up:active","state_seq":6,"addr":"172.21.15.119:6837/1872363128","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6836","nonce":1872363128},{"type":"v1","addr":"172.21.15.119:6837","nonce":1872363128}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24545":{"gid":24545,"name":"b","rank":0,"incarnation":14,"state":"up:active","state_seq":3,"addr":"172.21.15.119:6839/937297340","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6838","nonce":937297340},{"type":"v1","addr":"172.21.15.119:6839","nonce":937297340}]},"join_fscid":-1,"export_targets":[1,2],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":24545,"qdb_cluster":[24509,24545,24479]},"id":1}]} 2024-04-23T10:29:16.464 INFO:teuthology.orchestra.run.smithi114.stderr:dumped fsmap epoch 23 2024-04-23T10:29:16.465 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:29:16.466+0000 7fb170eef640 1 -- 172.21.15.114:0/508373081 >> [v2:172.21.15.114:6800/1395693466,v1:172.21.15.114:6801/1395693466] conn(0x7fb1540a3d20 msgr2=0x7fb1540a61e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T10:29:16.465 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:29:16.466+0000 7fb170eef640 1 --2- 172.21.15.114:0/508373081 >> [v2:172.21.15.114:6800/1395693466,v1:172.21.15.114:6801/1395693466] conn(0x7fb1540a3d20 0x7fb1540a61e0 secure :-1 s=READY pgs=81 cs=0 l=1 rev1=1 crypto rx=0x7fb164009fc0 tx=0x7fb164005f90 comp rx=0 tx=0).stop 2024-04-23T10:29:16.465 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:29:16.466+0000 7fb170eef640 1 -- 172.21.15.114:0/508373081 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fb16c19a010 msgr2=0x7fb16c1a45c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T10:29:16.465 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:29:16.466+0000 7fb170eef640 1 --2- 172.21.15.114:0/508373081 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fb16c19a010 0x7fb16c1a45c0 secure :-1 s=READY pgs=569 cs=0 l=1 rev1=1 crypto rx=0x7fb15c004770 tx=0x7fb15c004040 comp rx=0 tx=0).stop 2024-04-23T10:29:16.465 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:29:16.466+0000 7fb170eef640 1 -- 172.21.15.114:0/508373081 shutdown_connections 2024-04-23T10:29:16.466 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:29:16.466+0000 7fb170eef640 1 --2- 172.21.15.114:0/508373081 >> [v2:172.21.15.114:6800/1395693466,v1:172.21.15.114:6801/1395693466] conn(0x7fb1540a3d20 0x7fb1540a61e0 unknown :-1 s=CLOSED pgs=81 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:29:16.466 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:29:16.466+0000 7fb170eef640 1 --2- 172.21.15.114:0/508373081 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fb16c19a010 0x7fb16c1a45c0 unknown :-1 s=CLOSED pgs=569 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:29:16.466 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:29:16.466+0000 7fb170eef640 1 --2- 172.21.15.114:0/508373081 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fb16c106170 0x7fb16c199a80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:29:16.466 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:29:16.466+0000 7fb170eef640 1 --2- 172.21.15.114:0/508373081 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fb16c1010a0 0x7fb16c199540 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:29:16.466 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:29:16.466+0000 7fb170eef640 1 -- 172.21.15.114:0/508373081 >> 172.21.15.114:0/508373081 conn(0x7fb16c077f50 msgr2=0x7fb16c0fef10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T10:29:16.466 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:29:16.466+0000 7fb170eef640 1 -- 172.21.15.114:0/508373081 shutdown_connections 2024-04-23T10:29:16.466 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:29:16.466+0000 7fb170eef640 1 -- 172.21.15.114:0/508373081 wait complete. 2024-04-23T10:29:16.476 DEBUG:tasks.quiescer.fs.[cephfs]:Running ceph command: 'tell mds.24545 quiesce db / --timeout 60 --expiration 81.45 --await-for 42.0' 2024-04-23T10:29:16.476 DEBUG:teuthology.orchestra.run.smithi114:> timeout 75 sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.24545 quiesce db / --timeout 60 --expiration 81.45 --await-for 42.0 2024-04-23T10:29:16.593 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:29:16.592+0000 7fe377e10640 1 -- 172.21.15.114:0/2361293489 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fe370107270 msgr2=0x7fe37010ee40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T10:29:16.593 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:29:16.592+0000 7fe377e10640 1 --2- 172.21.15.114:0/2361293489 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fe370107270 0x7fe37010ee40 secure :-1 s=READY pgs=570 cs=0 l=1 rev1=1 crypto rx=0x7fe368009ca0 tx=0x7fe36802a8c0 comp rx=0 tx=0).stop 2024-04-23T10:29:16.593 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:29:16.592+0000 7fe377e10640 1 -- 172.21.15.114:0/2361293489 shutdown_connections 2024-04-23T10:29:16.593 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:29:16.592+0000 7fe377e10640 1 --2- 172.21.15.114:0/2361293489 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fe370107270 0x7fe37010ee40 unknown :-1 s=CLOSED pgs=570 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:29:16.593 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:29:16.592+0000 7fe377e10640 1 --2- 172.21.15.114:0/2361293489 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fe370102b50 0x7fe370106b90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:29:16.593 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:29:16.592+0000 7fe377e10640 1 --2- 172.21.15.114:0/2361293489 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fe3701021a0 0x7fe370102580 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:29:16.593 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:29:16.592+0000 7fe377e10640 1 -- 172.21.15.114:0/2361293489 >> 172.21.15.114:0/2361293489 conn(0x7fe370079350 msgr2=0x7fe370079760 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T10:29:16.593 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:29:16.592+0000 7fe377e10640 1 -- 172.21.15.114:0/2361293489 shutdown_connections 2024-04-23T10:29:16.593 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:29:16.592+0000 7fe377e10640 1 -- 172.21.15.114:0/2361293489 wait complete. 2024-04-23T10:29:16.593 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:29:16.592+0000 7fe377e10640 1 Processor -- start 2024-04-23T10:29:16.593 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:29:16.592+0000 7fe377e10640 1 -- start start 2024-04-23T10:29:16.594 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:29:16.593+0000 7fe377e10640 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fe3701021a0 0x7fe370198740 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T10:29:16.594 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:29:16.593+0000 7fe377e10640 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fe370107270 0x7fe370198c80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T10:29:16.594 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:29:16.593+0000 7fe377e10640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fe37019d000 0x7fe37019d4b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T10:29:16.594 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:29:16.593+0000 7fe377e10640 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x7fe3701121e0 con 0x7fe37019d000 2024-04-23T10:29:16.594 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:29:16.593+0000 7fe377e10640 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 -- 0x7fe370112060 con 0x7fe370107270 2024-04-23T10:29:16.594 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:29:16.593+0000 7fe377e10640 1 -- --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_getmap magic: 0 -- 0x7fe370112360 con 0x7fe3701021a0 2024-04-23T10:29:16.594 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:29:16.593+0000 7fe3763ad640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fe37019d000 0x7fe37019d4b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T10:29:16.594 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:29:16.593+0000 7fe3763ad640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fe37019d000 0x7fe37019d4b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.114:54434/0 (socket says 172.21.15.114:54434) 2024-04-23T10:29:16.594 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:29:16.593+0000 7fe3763ad640 1 -- 172.21.15.114:0/1130005939 learned_addr learned my addr 172.21.15.114:0/1130005939 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-04-23T10:29:16.594 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:29:16.593+0000 7fe3753ab640 1 --2- 172.21.15.114:0/1130005939 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fe370107270 0x7fe370198c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T10:29:16.594 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:29:16.593+0000 7fe375bac640 1 --2- 172.21.15.114:0/1130005939 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fe3701021a0 0x7fe370198740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T10:29:16.594 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:29:16.593+0000 7fe3763ad640 1 -- 172.21.15.114:0/1130005939 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fe3701021a0 msgr2=0x7fe370198740 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T10:29:16.594 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:29:16.593+0000 7fe3763ad640 1 --2- 172.21.15.114:0/1130005939 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fe3701021a0 0x7fe370198740 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:29:16.594 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:29:16.593+0000 7fe3763ad640 1 -- 172.21.15.114:0/1130005939 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fe370107270 msgr2=0x7fe370198c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T10:29:16.595 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:29:16.593+0000 7fe3763ad640 1 --2- 172.21.15.114:0/1130005939 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fe370107270 0x7fe370198c80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:29:16.595 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:29:16.593+0000 7fe3763ad640 1 -- 172.21.15.114:0/1130005939 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe37019daf0 con 0x7fe37019d000 2024-04-23T10:29:16.595 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:29:16.593+0000 7fe3763ad640 1 --2- 172.21.15.114:0/1130005939 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fe37019d000 0x7fe37019d4b0 secure :-1 s=READY pgs=571 cs=0 l=1 rev1=1 crypto rx=0x7fe368005cc0 tx=0x7fe368031700 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T10:29:16.595 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:29:16.593+0000 7fe366ffd640 1 -- 172.21.15.114:0/1130005939 <== mon.0 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe368003b70 con 0x7fe37019d000 2024-04-23T10:29:16.595 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:29:16.593+0000 7fe366ffd640 1 -- 172.21.15.114:0/1130005939 <== mon.0 v2:172.21.15.114:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fe3680318f0 con 0x7fe37019d000 2024-04-23T10:29:16.595 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:29:16.593+0000 7fe377e10640 1 -- 172.21.15.114:0/1130005939 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fe37019dd80 con 0x7fe37019d000 2024-04-23T10:29:16.595 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:29:16.593+0000 7fe366ffd640 1 -- 172.21.15.114:0/1130005939 <== mon.0 v2:172.21.15.114:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe36802f950 con 0x7fe37019d000 2024-04-23T10:29:16.595 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:29:16.593+0000 7fe377e10640 1 -- 172.21.15.114:0/1130005939 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fe3701a55a0 con 0x7fe37019d000 2024-04-23T10:29:16.601 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:29:16.595+0000 7fe366ffd640 1 -- 172.21.15.114:0/1130005939 <== mon.0 v2:172.21.15.114:3300/0 4 ==== mgrmap(e 26) ==== 138778+0+0 (secure 0 0 0) 0x7fe36802fb50 con 0x7fe37019d000 2024-04-23T10:29:16.602 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:29:16.596+0000 7fe37ac03740 1 Processor -- start 2024-04-23T10:29:16.602 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:29:16.596+0000 7fe366ffd640 1 --2- 172.21.15.114:0/1130005939 >> [v2:172.21.15.114:6800/1395693466,v1:172.21.15.114:6801/1395693466] conn(0x7fe35c0a3d20 0x7fe35c0a61e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T10:29:16.602 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:29:16.596+0000 7fe3753ab640 1 --2- 172.21.15.114:0/1130005939 >> [v2:172.21.15.114:6800/1395693466,v1:172.21.15.114:6801/1395693466] conn(0x7fe35c0a3d20 0x7fe35c0a61e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T10:29:16.602 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:29:16.596+0000 7fe37ac03740 1 -- start start 2024-04-23T10:29:16.602 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:29:16.596+0000 7fe366ffd640 1 -- 172.21.15.114:0/1130005939 <== mon.0 v2:172.21.15.114:3300/0 5 ==== osd_map(92..92 src has 1..92) ==== 8585+0+0 (secure 0 0 0) 0x7fe3680f1a50 con 0x7fe37019d000 2024-04-23T10:29:16.602 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:29:16.596+0000 7fe37ac03740 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x55d8026eb0a0 0x55d8026eb480 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T10:29:16.602 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:29:16.597+0000 7fe37ac03740 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55d8026eb9c0 0x55d8026f5b50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T10:29:16.602 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:29:16.597+0000 7fe37ac03740 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x55d8026f6090 0x55d8026f8480 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T10:29:16.602 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:29:16.597+0000 7fe37ac03740 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x55d802650370 con 0x55d8026eb9c0 2024-04-23T10:29:16.602 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:29:16.597+0000 7fe37ac03740 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 -- 0x55d8025ed2e0 con 0x55d8026eb0a0 2024-04-23T10:29:16.602 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:29:16.597+0000 7fe37ac03740 1 -- --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_getmap magic: 0 -- 0x55d80264c330 con 0x55d8026f6090 2024-04-23T10:29:16.602 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:29:16.597+0000 7fe375bac640 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x55d8026eb0a0 0x55d8026eb480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T10:29:16.603 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:29:16.597+0000 7fe375bac640 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x55d8026eb0a0 0x55d8026eb480 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.119:3300/0 says I am v2:172.21.15.114:48636/0 (socket says 172.21.15.114:48636) 2024-04-23T10:29:16.603 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:29:16.597+0000 7fe375bac640 1 -- 172.21.15.114:0/3852021660 learned_addr learned my addr 172.21.15.114:0/3852021660 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-04-23T10:29:16.603 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:29:16.597+0000 7fe3753ab640 1 --2- 172.21.15.114:0/3852021660 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55d8026eb9c0 0x55d8026f5b50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T10:29:16.603 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:29:16.597+0000 7fe3763ad640 1 --2- 172.21.15.114:0/3852021660 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x55d8026f6090 0x55d8026f8480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T10:29:16.603 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:29:16.597+0000 7fe3753ab640 1 --2- 172.21.15.114:0/1130005939 >> [v2:172.21.15.114:6800/1395693466,v1:172.21.15.114:6801/1395693466] conn(0x7fe35c0a3d20 0x7fe35c0a61e0 secure :-1 s=READY pgs=82 cs=0 l=1 rev1=1 crypto rx=0x7fe370101a80 tx=0x7fe36c009f70 comp rx=0 tx=0).ready entity=mgr.15291 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T10:29:16.603 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:29:16.597+0000 7fe375bac640 1 -- 172.21.15.114:0/3852021660 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x55d8026f6090 msgr2=0x55d8026f8480 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T10:29:16.603 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:29:16.597+0000 7fe375bac640 1 --2- 172.21.15.114:0/3852021660 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x55d8026f6090 0x55d8026f8480 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:29:16.603 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:29:16.597+0000 7fe375bac640 1 -- 172.21.15.114:0/3852021660 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55d8026eb9c0 msgr2=0x55d8026f5b50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T10:29:16.603 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:29:16.597+0000 7fe375bac640 1 --2- 172.21.15.114:0/3852021660 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55d8026eb9c0 0x55d8026f5b50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:29:16.603 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:29:16.597+0000 7fe375bac640 1 -- 172.21.15.114:0/3852021660 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55d80242ed80 con 0x55d8026eb0a0 2024-04-23T10:29:16.603 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:29:16.597+0000 7fe3753ab640 1 --2- 172.21.15.114:0/3852021660 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55d8026eb9c0 0x55d8026f5b50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-23T10:29:16.603 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:29:16.597+0000 7fe375bac640 1 --2- 172.21.15.114:0/3852021660 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x55d8026eb0a0 0x55d8026eb480 secure :-1 s=READY pgs=71 cs=0 l=1 rev1=1 crypto rx=0x7fe360009cf0 tx=0x7fe360011810 comp rx=0 tx=0).ready entity=mon.2 client_cookie=ef8073d9f66dd2d2 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T10:29:16.603 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:29:16.597+0000 7fe35bfff640 1 -- 172.21.15.114:0/3852021660 <== mon.2 v2:172.21.15.119:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe360004230 con 0x55d8026eb0a0 2024-04-23T10:29:16.603 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:29:16.597+0000 7fe35bfff640 1 -- 172.21.15.114:0/3852021660 <== mon.2 v2:172.21.15.119:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fe3600043d0 con 0x55d8026eb0a0 2024-04-23T10:29:16.604 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:29:16.597+0000 7fe35bfff640 1 -- 172.21.15.114:0/3852021660 <== mon.2 v2:172.21.15.119:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe360019880 con 0x55d8026eb0a0 2024-04-23T10:29:16.604 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:29:16.598+0000 7fe37ac03740 1 -- 172.21.15.114:0/3852021660 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x55d8026eb0a0 msgr2=0x55d8026eb480 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T10:29:16.604 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:29:16.598+0000 7fe37ac03740 1 --2- 172.21.15.114:0/3852021660 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x55d8026eb0a0 0x55d8026eb480 secure :-1 s=READY pgs=71 cs=0 l=1 rev1=1 crypto rx=0x7fe360009cf0 tx=0x7fe360011810 comp rx=0 tx=0).stop 2024-04-23T10:29:16.604 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:29:16.598+0000 7fe37ac03740 1 -- 172.21.15.114:0/3852021660 shutdown_connections 2024-04-23T10:29:16.604 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:29:16.598+0000 7fe37ac03740 1 --2- 172.21.15.114:0/3852021660 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x55d8026f6090 0x55d8026f8480 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:29:16.604 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:29:16.598+0000 7fe37ac03740 1 --2- 172.21.15.114:0/3852021660 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55d8026eb9c0 0x55d8026f5b50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:29:16.604 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:29:16.598+0000 7fe37ac03740 1 --2- 172.21.15.114:0/3852021660 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x55d8026eb0a0 0x55d8026eb480 unknown :-1 s=CLOSED pgs=71 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:29:16.604 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:29:16.598+0000 7fe37ac03740 1 -- 172.21.15.114:0/3852021660 >> 172.21.15.114:0/3852021660 conn(0x55d8026e73c0 msgr2=0x55d8026e77a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T10:29:16.604 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:29:16.598+0000 7fe37ac03740 1 -- 172.21.15.114:0/3852021660 shutdown_connections 2024-04-23T10:29:16.604 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:29:16.598+0000 7fe37ac03740 1 -- 172.21.15.114:0/3852021660 wait complete. 2024-04-23T10:29:16.604 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:29:16.599+0000 7fe37ac03740 1 -- start start 2024-04-23T10:29:16.604 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:29:16.599+0000 7fe37ac03740 1 Processor -- start 2024-04-23T10:29:16.604 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:29:16.599+0000 7fe37ac03740 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55d8026eb0a0 0x55d8026f2fa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T10:29:16.605 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:29:16.599+0000 7fe37ac03740 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x55d8026eb9c0 0x55d8026f34e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T10:29:16.605 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:29:16.599+0000 7fe37ac03740 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x55d8026f6090 0x55d8026f3a20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T10:29:16.605 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:29:16.599+0000 7fe37ac03740 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x55d8026f8e80 con 0x55d8026eb0a0 2024-04-23T10:29:16.605 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:29:16.599+0000 7fe37ac03740 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 -- 0x55d8025eafb0 con 0x55d8026eb9c0 2024-04-23T10:29:16.605 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:29:16.599+0000 7fe37ac03740 1 -- --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_getmap magic: 0 -- 0x55d8026f9000 con 0x55d8026f6090 2024-04-23T10:29:16.605 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:29:16.599+0000 7fe37ac03740 10 client.0 _ll_register_callbacks cb 0x55d8025ea9a0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-23T10:29:16.605 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:29:16.599+0000 7fe375bac640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55d8026eb0a0 0x55d8026f2fa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T10:29:16.605 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:29:16.599+0000 7fe375bac640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55d8026eb0a0 0x55d8026f2fa0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.114:54446/0 (socket says 172.21.15.114:54446) 2024-04-23T10:29:16.605 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:29:16.599+0000 7fe375bac640 1 -- 172.21.15.114:0/4140503576 learned_addr learned my addr 172.21.15.114:0/4140503576 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-04-23T10:29:16.605 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:29:16.599+0000 7fe3753ab640 1 --2- 172.21.15.114:0/4140503576 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x55d8026eb9c0 0x55d8026f34e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T10:29:16.605 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:29:16.599+0000 7fe3763ad640 1 --2- 172.21.15.114:0/4140503576 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x55d8026f6090 0x55d8026f3a20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T10:29:16.605 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:29:16.599+0000 7fe375bac640 1 -- 172.21.15.114:0/4140503576 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x55d8026f6090 msgr2=0x55d8026f3a20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T10:29:16.605 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:29:16.599+0000 7fe375bac640 1 --2- 172.21.15.114:0/4140503576 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x55d8026f6090 0x55d8026f3a20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:29:16.605 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:29:16.599+0000 7fe375bac640 1 -- 172.21.15.114:0/4140503576 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x55d8026eb9c0 msgr2=0x55d8026f34e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T10:29:16.605 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:29:16.599+0000 7fe375bac640 1 --2- 172.21.15.114:0/4140503576 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x55d8026eb9c0 0x55d8026f34e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:29:16.606 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:29:16.600+0000 7fe375bac640 1 -- 172.21.15.114:0/4140503576 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55d80248f510 con 0x55d8026eb0a0 2024-04-23T10:29:16.606 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:29:16.600+0000 7fe375bac640 1 --2- 172.21.15.114:0/4140503576 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55d8026eb0a0 0x55d8026f2fa0 secure :-1 s=READY pgs=572 cs=0 l=1 rev1=1 crypto rx=0x7fe3600038a0 tx=0x7fe3600038d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=840a2f99e14fa1da server_cookie=0 in_seq=0 out_seq=0 2024-04-23T10:29:16.606 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:29:16.600+0000 7fe3597fa640 10 client.0 ms_handle_connect on v2:172.21.15.114:3300/0 2024-04-23T10:29:16.606 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:29:16.600+0000 7fe3597fa640 1 -- 172.21.15.114:0/4140503576 <== mon.0 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe360027070 con 0x55d8026eb0a0 2024-04-23T10:29:16.606 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:29:16.600+0000 7fe3597fa640 1 -- 172.21.15.114:0/4140503576 <== mon.0 v2:172.21.15.114:3300/0 2 ==== osd_map(92..92 src has 1..92) ==== 8585+0+0 (secure 0 0 0) 0x7fe360005240 con 0x55d8026eb0a0 2024-04-23T10:29:16.606 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:29:16.600+0000 7fe37ac03740 1 -- 172.21.15.114:0/4140503576 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fe3600043d0 con 0x55d8026eb0a0 2024-04-23T10:29:16.606 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:29:16.600+0000 7fe3597fa640 1 -- 172.21.15.114:0/4140503576 <== mon.0 v2:172.21.15.114:3300/0 3 ==== mon_get_version_reply(handle=1 version=23) ==== 24+0+0 (secure 0 0 0) 0x7fe360033050 con 0x55d8026eb0a0 2024-04-23T10:29:16.606 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:29:16.600+0000 7fe37ac03740 10 client.15870 fetch_fsmap learned FSMap version 23 2024-04-23T10:29:16.606 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:29:16.600+0000 7fe37ac03740 1 -- 172.21.15.114:0/4140503576 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({config=0+,fsmap=23,monmap=4+}) -- 0x55d802322670 con 0x55d8026eb0a0 2024-04-23T10:29:16.606 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:29:16.600+0000 7fe3597fa640 1 -- 172.21.15.114:0/4140503576 <== mon.0 v2:172.21.15.114:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fe3600043d0 con 0x55d8026eb0a0 2024-04-23T10:29:16.606 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:29:16.600+0000 7fe3597fa640 1 -- 172.21.15.114:0/4140503576 <== mon.0 v2:172.21.15.114:3300/0 5 ==== fsmap(e 23) ==== 8350+0+0 (secure 0 0 0) 0x7fe360004b70 con 0x55d8026eb0a0 2024-04-23T10:29:16.606 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:29:16.600+0000 7fe37ac03740 10 client.15870 fetch_fsmap finished waiting for FSMap version 23 2024-04-23T10:29:16.606 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:29:16.600+0000 7fe37ac03740 10 client.15870 resolve_mds: validated gid 24545 aka daemon mds.b 2024-04-23T10:29:16.606 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:29:16.600+0000 7fe37ac03740 20 client.15870 populate_metadata read hostname 'smithi114' 2024-04-23T10:29:16.607 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:29:16.600+0000 7fe37ac03740 1 --2- 172.21.15.114:0/4140503576 >> [v2:172.21.15.119:6838/937297340,v1:172.21.15.119:6839/937297340] conn(0x55d802706330 0x55d802708720 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T10:29:16.607 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:29:16.600+0000 7fe37ac03740 4 client.15870 mds_command: new command op to 24545 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-23T10:29:16.607 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:29:16.600+0000 7fe37ac03740 1 -- 172.21.15.114:0/4140503576 --> [v2:172.21.15.119:6838/937297340,v1:172.21.15.119:6839/937297340] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55d80265dd90 con 0x55d802706330 2024-04-23T10:29:16.607 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:29:16.601+0000 7fe3753ab640 1 --2- 172.21.15.114:0/4140503576 >> [v2:172.21.15.119:6838/937297340,v1:172.21.15.119:6839/937297340] conn(0x55d802706330 0x55d802708720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T10:29:16.607 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:29:16.601+0000 7fe3753ab640 1 --2- 172.21.15.114:0/4140503576 >> [v2:172.21.15.119:6838/937297340,v1:172.21.15.119:6839/937297340] conn(0x55d802706330 0x55d802708720 crc :-1 s=READY pgs=70 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=e6f85a0a59e0e3f1 server_cookie=f6f6e5713acb36c5 in_seq=0 out_seq=0 2024-04-23T10:29:16.607 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:29:16.601+0000 7fe3597fa640 10 client.15870 ms_handle_connect on v2:172.21.15.119:6838/937297340 2024-04-23T10:29:16.607 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:29:16.602+0000 7fe3597fa640 1 -- 172.21.15.114:0/4140503576 <== mds.0 v2:172.21.15.119:6838/937297340 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x55d80265dd90 con 0x55d802706330 2024-04-23T10:29:16.607 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:29:16.602+0000 7fe3597fa640 10 client.15870 handle_command_reply: tid=0 2024-04-23T10:29:16.607 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:29:16.602+0000 7fe37ac03740 1 client.15870 shutdown 2024-04-23T10:29:16.607 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:29:16.602+0000 7fe37ac03740 1 -- 172.21.15.114:0/4140503576 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55d8026eb0a0 msgr2=0x55d8026f2fa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T10:29:16.607 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:29:16.602+0000 7fe37ac03740 1 --2- 172.21.15.114:0/4140503576 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55d8026eb0a0 0x55d8026f2fa0 secure :-1 s=READY pgs=572 cs=0 l=1 rev1=1 crypto rx=0x7fe3600038a0 tx=0x7fe3600038d0 comp rx=0 tx=0).stop 2024-04-23T10:29:16.607 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:29:16.602+0000 7fe37ac03740 1 -- 172.21.15.114:0/4140503576 shutdown_connections 2024-04-23T10:29:16.607 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:29:16.602+0000 7fe37ac03740 1 --2- 172.21.15.114:0/4140503576 >> [v2:172.21.15.119:6838/937297340,v1:172.21.15.119:6839/937297340] conn(0x55d802706330 0x55d802708720 crc :-1 s=READY pgs=70 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:29:16.607 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:29:16.602+0000 7fe37ac03740 1 --2- 172.21.15.114:0/4140503576 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x55d8026f6090 0x55d8026f3a20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:29:16.607 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:29:16.602+0000 7fe37ac03740 1 --2- 172.21.15.114:0/4140503576 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x55d8026eb9c0 0x55d8026f34e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:29:16.608 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:29:16.602+0000 7fe37ac03740 1 --2- 172.21.15.114:0/4140503576 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55d8026eb0a0 0x55d8026f2fa0 unknown :-1 s=CLOSED pgs=572 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:29:16.608 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:29:16.602+0000 7fe3597fa640 0 client.15870 ms_handle_reset on v2:172.21.15.119:6838/937297340 2024-04-23T10:29:16.608 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:29:16.602+0000 7fe37ac03740 1 -- 172.21.15.114:0/4140503576 >> 172.21.15.114:0/4140503576 conn(0x55d8026e73c0 msgr2=0x55d8026e6600 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T10:29:16.608 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:29:16.602+0000 7fe37ac03740 1 -- 172.21.15.114:0/4140503576 shutdown_connections 2024-04-23T10:29:16.608 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:29:16.602+0000 7fe37ac03740 1 -- 172.21.15.114:0/4140503576 wait complete. 2024-04-23T10:29:16.608 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:29:16.602+0000 7fe37ac03740 20 client.15870 trim_cache size 0 max 16384 2024-04-23T10:29:16.622 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:29:16.619+0000 7fe37ac03740 1 Processor -- start 2024-04-23T10:29:16.623 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:29:16.619+0000 7fe37ac03740 1 -- start start 2024-04-23T10:29:16.623 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:29:16.619+0000 7fe37ac03740 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x55d8026eb0a0 0x55d8026e6d80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T10:29:16.623 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:29:16.619+0000 7fe37ac03740 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55d8026eb9c0 0x55d8026ece70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T10:29:16.623 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:29:16.619+0000 7fe37ac03740 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x55d8026f6090 0x55d80270dff0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T10:29:16.623 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:29:16.619+0000 7fe37ac03740 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x7fe3701b05c0 con 0x55d8026eb9c0 2024-04-23T10:29:16.623 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:29:16.619+0000 7fe37ac03740 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 -- 0x7fe3701ade60 con 0x55d8026f6090 2024-04-23T10:29:16.623 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:29:16.619+0000 7fe37ac03740 1 -- --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_getmap magic: 0 -- 0x7fe3701abcb0 con 0x55d8026eb0a0 2024-04-23T10:29:16.623 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:29:16.619+0000 7fe3753ab640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55d8026eb9c0 0x55d8026ece70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T10:29:16.623 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:29:16.619+0000 7fe3753ab640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55d8026eb9c0 0x55d8026ece70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.114:54450/0 (socket says 172.21.15.114:54450) 2024-04-23T10:29:16.623 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:29:16.619+0000 7fe3753ab640 1 -- 172.21.15.114:0/700834048 learned_addr learned my addr 172.21.15.114:0/700834048 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-04-23T10:29:16.623 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:29:16.619+0000 7fe375bac640 1 --2- 172.21.15.114:0/700834048 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x55d8026eb0a0 0x55d8026e6d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T10:29:16.623 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:29:16.620+0000 7fe3763ad640 1 --2- 172.21.15.114:0/700834048 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x55d8026f6090 0x55d80270dff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T10:29:16.623 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:29:16.620+0000 7fe3753ab640 1 -- 172.21.15.114:0/700834048 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x55d8026eb0a0 msgr2=0x55d8026e6d80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T10:29:16.623 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:29:16.620+0000 7fe3753ab640 1 --2- 172.21.15.114:0/700834048 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x55d8026eb0a0 0x55d8026e6d80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:29:16.624 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:29:16.620+0000 7fe3753ab640 1 -- 172.21.15.114:0/700834048 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x55d8026f6090 msgr2=0x55d80270dff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T10:29:16.624 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:29:16.620+0000 7fe3753ab640 1 --2- 172.21.15.114:0/700834048 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x55d8026f6090 0x55d80270dff0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:29:16.624 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:29:16.620+0000 7fe3753ab640 1 -- 172.21.15.114:0/700834048 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55d8026eb7f0 con 0x55d8026eb9c0 2024-04-23T10:29:16.624 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:29:16.620+0000 7fe375bac640 1 --2- 172.21.15.114:0/700834048 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x55d8026eb0a0 0x55d8026e6d80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-23T10:29:16.624 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:29:16.620+0000 7fe3753ab640 1 --2- 172.21.15.114:0/700834048 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55d8026eb9c0 0x55d8026ece70 secure :-1 s=READY pgs=573 cs=0 l=1 rev1=1 crypto rx=0x7fe36c01e710 tx=0x7fe36c01c000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=efa741345cba7ea3 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T10:29:16.624 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:29:16.620+0000 7fe35b7fe640 1 -- 172.21.15.114:0/700834048 <== mon.0 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe36c017040 con 0x55d8026eb9c0 2024-04-23T10:29:16.624 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:29:16.620+0000 7fe35b7fe640 1 -- 172.21.15.114:0/700834048 <== mon.0 v2:172.21.15.114:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fe36c02d070 con 0x55d8026eb9c0 2024-04-23T10:29:16.624 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:29:16.620+0000 7fe35b7fe640 1 -- 172.21.15.114:0/700834048 <== mon.0 v2:172.21.15.114:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe36c01d6a0 con 0x55d8026eb9c0 2024-04-23T10:29:16.624 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:29:16.621+0000 7fe37ac03740 1 -- 172.21.15.114:0/700834048 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55d8026eb9c0 msgr2=0x55d8026ece70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T10:29:16.624 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:29:16.621+0000 7fe37ac03740 1 --2- 172.21.15.114:0/700834048 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55d8026eb9c0 0x55d8026ece70 secure :-1 s=READY pgs=573 cs=0 l=1 rev1=1 crypto rx=0x7fe36c01e710 tx=0x7fe36c01c000 comp rx=0 tx=0).stop 2024-04-23T10:29:16.624 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:29:16.621+0000 7fe37ac03740 1 -- 172.21.15.114:0/700834048 shutdown_connections 2024-04-23T10:29:16.624 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:29:16.621+0000 7fe37ac03740 1 --2- 172.21.15.114:0/700834048 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x55d8026f6090 0x55d80270dff0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:29:16.624 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:29:16.621+0000 7fe37ac03740 1 --2- 172.21.15.114:0/700834048 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55d8026eb9c0 0x55d8026ece70 unknown :-1 s=CLOSED pgs=573 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:29:16.624 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:29:16.621+0000 7fe37ac03740 1 --2- 172.21.15.114:0/700834048 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x55d8026eb0a0 0x55d8026e6d80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:29:16.624 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:29:16.621+0000 7fe37ac03740 1 -- 172.21.15.114:0/700834048 >> 172.21.15.114:0/700834048 conn(0x55d8026e73c0 msgr2=0x55d8026ec760 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T10:29:16.625 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:29:16.621+0000 7fe37ac03740 1 -- 172.21.15.114:0/700834048 shutdown_connections 2024-04-23T10:29:16.625 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:29:16.621+0000 7fe37ac03740 1 -- 172.21.15.114:0/700834048 wait complete. 2024-04-23T10:29:16.625 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:29:16.621+0000 7fe37ac03740 1 -- start start 2024-04-23T10:29:16.625 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:29:16.621+0000 7fe37ac03740 1 Processor -- start 2024-04-23T10:29:16.625 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:29:16.622+0000 7fe37ac03740 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x55d8026eb0a0 0x55d8026f0310 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T10:29:16.625 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:29:16.622+0000 7fe37ac03740 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55d8026eb9c0 0x55d8026f0850 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T10:29:16.625 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:29:16.622+0000 7fe37ac03740 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x55d8026f6090 0x55d80276fc70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T10:29:16.625 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:29:16.622+0000 7fe37ac03740 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x7fe3701ac320 con 0x55d8026eb9c0 2024-04-23T10:29:16.625 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:29:16.622+0000 7fe37ac03740 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 -- 0x7fe3701ad790 con 0x55d8026f6090 2024-04-23T10:29:16.625 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:29:16.622+0000 7fe37ac03740 1 -- --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_getmap magic: 0 -- 0x7fe3701ac9f0 con 0x55d8026eb0a0 2024-04-23T10:29:16.625 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:29:16.622+0000 7fe37ac03740 10 client.0 _ll_register_callbacks cb 0x7fe360009b70 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-23T10:29:16.625 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:29:16.622+0000 7fe3753ab640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55d8026eb9c0 0x55d8026f0850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T10:29:16.625 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:29:16.622+0000 7fe3753ab640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55d8026eb9c0 0x55d8026f0850 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.114:54456/0 (socket says 172.21.15.114:54456) 2024-04-23T10:29:16.625 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:29:16.622+0000 7fe3753ab640 1 -- 172.21.15.114:0/714319291 learned_addr learned my addr 172.21.15.114:0/714319291 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-04-23T10:29:16.626 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:29:16.622+0000 7fe375bac640 1 --2- 172.21.15.114:0/714319291 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x55d8026eb0a0 0x55d8026f0310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T10:29:16.626 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:29:16.622+0000 7fe3763ad640 1 --2- 172.21.15.114:0/714319291 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x55d8026f6090 0x55d80276fc70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T10:29:16.626 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:29:16.622+0000 7fe3753ab640 1 -- 172.21.15.114:0/714319291 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x55d8026eb0a0 msgr2=0x55d8026f0310 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T10:29:16.626 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:29:16.622+0000 7fe3753ab640 1 --2- 172.21.15.114:0/714319291 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x55d8026eb0a0 0x55d8026f0310 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:29:16.626 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:29:16.622+0000 7fe3753ab640 1 -- 172.21.15.114:0/714319291 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x55d8026f6090 msgr2=0x55d80276fc70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T10:29:16.626 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:29:16.622+0000 7fe3753ab640 1 --2- 172.21.15.114:0/714319291 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x55d8026f6090 0x55d80276fc70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:29:16.626 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:29:16.622+0000 7fe3753ab640 1 -- 172.21.15.114:0/714319291 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55d8027701b0 con 0x55d8026eb9c0 2024-04-23T10:29:16.626 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:29:16.622+0000 7fe3753ab640 1 --2- 172.21.15.114:0/714319291 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55d8026eb9c0 0x55d8026f0850 secure :-1 s=READY pgs=574 cs=0 l=1 rev1=1 crypto rx=0x7fe36c02c000 tx=0x7fe36c01e570 comp rx=0 tx=0).ready entity=mon.0 client_cookie=bc0bc505e13bd3d0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T10:29:16.626 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:29:16.622+0000 7fe3597fa640 10 client.0 ms_handle_connect on v2:172.21.15.114:3300/0 2024-04-23T10:29:16.626 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:29:16.622+0000 7fe3597fa640 1 -- 172.21.15.114:0/714319291 <== mon.0 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe36c035020 con 0x55d8026eb9c0 2024-04-23T10:29:16.626 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:29:16.622+0000 7fe3597fa640 1 -- 172.21.15.114:0/714319291 <== mon.0 v2:172.21.15.114:3300/0 2 ==== osd_map(92..92 src has 1..92) ==== 8585+0+0 (secure 0 0 0) 0x7fe36c0285c0 con 0x55d8026eb9c0 2024-04-23T10:29:16.626 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:29:16.622+0000 7fe37ac03740 1 -- 172.21.15.114:0/714319291 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fe36c02d070 con 0x55d8026eb9c0 2024-04-23T10:29:16.626 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:29:16.622+0000 7fe3597fa640 1 -- 172.21.15.114:0/714319291 <== mon.0 v2:172.21.15.114:3300/0 3 ==== mon_get_version_reply(handle=1 version=23) ==== 24+0+0 (secure 0 0 0) 0x7fe36c017070 con 0x55d8026eb9c0 2024-04-23T10:29:16.626 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:29:16.622+0000 7fe37ac03740 10 client.15876 fetch_fsmap learned FSMap version 23 2024-04-23T10:29:16.626 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:29:16.622+0000 7fe37ac03740 1 -- 172.21.15.114:0/714319291 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({config=0+,fsmap=23,monmap=4+}) -- 0x55d802771400 con 0x55d8026eb9c0 2024-04-23T10:29:16.627 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:29:16.623+0000 7fe3597fa640 1 -- 172.21.15.114:0/714319291 <== mon.0 v2:172.21.15.114:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fe36c02d070 con 0x55d8026eb9c0 2024-04-23T10:29:16.627 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:29:16.623+0000 7fe3597fa640 1 -- 172.21.15.114:0/714319291 <== mon.0 v2:172.21.15.114:3300/0 5 ==== fsmap(e 23) ==== 8350+0+0 (secure 0 0 0) 0x7fe36c004970 con 0x55d8026eb9c0 2024-04-23T10:29:16.627 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:29:16.623+0000 7fe37ac03740 10 client.15876 fetch_fsmap finished waiting for FSMap version 23 2024-04-23T10:29:16.627 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:29:16.623+0000 7fe37ac03740 10 client.15876 resolve_mds: validated gid 24545 aka daemon mds.b 2024-04-23T10:29:16.627 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:29:16.623+0000 7fe37ac03740 20 client.15876 populate_metadata read hostname 'smithi114' 2024-04-23T10:29:16.627 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:29:16.623+0000 7fe37ac03740 1 --2- 172.21.15.114:0/714319291 >> [v2:172.21.15.119:6838/937297340,v1:172.21.15.119:6839/937297340] conn(0x55d802771b80 0x55d802773fa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T10:29:16.627 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:29:16.623+0000 7fe37ac03740 4 client.15876 mds_command: new command op to 24545 tid=0 multi_id=0 [{"prefix": "quiesce db", "roots": ["/"], "timeout": 60.0, "expiration": 81.45, "await_for": 42.0}] 2024-04-23T10:29:16.627 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:29:16.623+0000 7fe37ac03740 1 -- 172.21.15.114:0/714319291 --> [v2:172.21.15.119:6838/937297340,v1:172.21.15.119:6839/937297340] -- command(tid 0: {"prefix": "quiesce db", "roots": ["/"], "timeout": 60.0, "expiration": 81.45, "await_for": 42.0}) -- 0x55d802657c00 con 0x55d802771b80 2024-04-23T10:29:16.627 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:29:16.623+0000 7fe375bac640 1 --2- 172.21.15.114:0/714319291 >> [v2:172.21.15.119:6838/937297340,v1:172.21.15.119:6839/937297340] conn(0x55d802771b80 0x55d802773fa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T10:29:16.627 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:29:16.623+0000 7fe3597fa640 10 client.15876 ms_handle_connect on v2:172.21.15.119:6838/937297340 2024-04-23T10:29:16.627 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:29:16.623+0000 7fe375bac640 1 --2- 172.21.15.114:0/714319291 >> [v2:172.21.15.119:6838/937297340,v1:172.21.15.119:6839/937297340] conn(0x55d802771b80 0x55d802773fa0 crc :-1 s=READY pgs=71 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=7720fd28d116eb4b server_cookie=5c21047186d2c268 in_seq=0 out_seq=0 2024-04-23T10:29:16.714 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:29:16.714+0000 7fe3597fa640 1 -- 172.21.15.114:0/714319291 <== mds.0 v2:172.21.15.119:6838/937297340 1 ==== command_reply(tid 0: 0 ) ==== 8+0+590 (crc 0 0 0) 0x55d802657c00 con 0x55d802771b80 2024-04-23T10:29:16.714 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:29:16.714+0000 7fe3597fa640 10 client.15876 handle_command_reply: tid=0 2024-04-23T10:29:16.714 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:29:16.714+0000 7fe37ac03740 1 client.15876 shutdown 2024-04-23T10:29:16.714 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:29:16.714+0000 7fe37ac03740 1 -- 172.21.15.114:0/714319291 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55d8026eb9c0 msgr2=0x55d8026f0850 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T10:29:16.714 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:29:16.714+0000 7fe37ac03740 1 --2- 172.21.15.114:0/714319291 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55d8026eb9c0 0x55d8026f0850 secure :-1 s=READY pgs=574 cs=0 l=1 rev1=1 crypto rx=0x7fe36c02c000 tx=0x7fe36c01e570 comp rx=0 tx=0).stop 2024-04-23T10:29:16.714 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:29:16.715+0000 7fe37ac03740 1 -- 172.21.15.114:0/714319291 shutdown_connections 2024-04-23T10:29:16.714 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:29:16.715+0000 7fe37ac03740 1 --2- 172.21.15.114:0/714319291 >> [v2:172.21.15.119:6838/937297340,v1:172.21.15.119:6839/937297340] conn(0x55d802771b80 0x55d802773fa0 crc :-1 s=READY pgs=71 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:29:16.714 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:29:16.715+0000 7fe37ac03740 1 --2- 172.21.15.114:0/714319291 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x55d8026f6090 0x55d80276fc70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:29:16.714 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:29:16.715+0000 7fe3597fa640 0 client.15876 ms_handle_reset on v2:172.21.15.119:6838/937297340 2024-04-23T10:29:16.714 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:29:16.715+0000 7fe37ac03740 1 --2- 172.21.15.114:0/714319291 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55d8026eb9c0 0x55d8026f0850 unknown :-1 s=CLOSED pgs=574 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:29:16.714 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:29:16.715+0000 7fe37ac03740 1 --2- 172.21.15.114:0/714319291 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x55d8026eb0a0 0x55d8026f0310 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:29:16.715 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:29:16.715+0000 7fe37ac03740 1 -- 172.21.15.114:0/714319291 >> 172.21.15.114:0/714319291 conn(0x55d8026e73c0 msgr2=0x55d8026f37d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T10:29:16.715 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:29:16.715+0000 7fe37ac03740 1 -- 172.21.15.114:0/714319291 shutdown_connections 2024-04-23T10:29:16.715 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:29:16.715+0000 7fe37ac03740 1 -- 172.21.15.114:0/714319291 wait complete. 2024-04-23T10:29:16.715 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:29:16.715+0000 7fe37ac03740 20 client.15876 trim_cache size 0 max 16384 2024-04-23T10:29:16.715 INFO:teuthology.orchestra.run.smithi114.stdout:{ 2024-04-23T10:29:16.715 INFO:teuthology.orchestra.run.smithi114.stdout: "epoch": 23, 2024-04-23T10:29:16.715 INFO:teuthology.orchestra.run.smithi114.stdout: "leader": 24545, 2024-04-23T10:29:16.715 INFO:teuthology.orchestra.run.smithi114.stdout: "set_version": 10, 2024-04-23T10:29:16.715 INFO:teuthology.orchestra.run.smithi114.stdout: "sets": { 2024-04-23T10:29:16.715 INFO:teuthology.orchestra.run.smithi114.stdout: "fec54601": { 2024-04-23T10:29:16.716 INFO:teuthology.orchestra.run.smithi114.stdout: "version": 10, 2024-04-23T10:29:16.716 INFO:teuthology.orchestra.run.smithi114.stdout: "age_ref": 0.0, 2024-04-23T10:29:16.716 INFO:teuthology.orchestra.run.smithi114.stdout: "state": { 2024-04-23T10:29:16.716 INFO:teuthology.orchestra.run.smithi114.stdout: "name": "QUIESCED", 2024-04-23T10:29:16.716 INFO:teuthology.orchestra.run.smithi114.stdout: "age": 0.0 2024-04-23T10:29:16.716 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-23T10:29:16.716 INFO:teuthology.orchestra.run.smithi114.stdout: "timeout": 60.0, 2024-04-23T10:29:16.716 INFO:teuthology.orchestra.run.smithi114.stdout: "expiration": 81.5, 2024-04-23T10:29:16.716 INFO:teuthology.orchestra.run.smithi114.stdout: "members": { 2024-04-23T10:29:16.716 INFO:teuthology.orchestra.run.smithi114.stdout: "file:/": { 2024-04-23T10:29:16.716 INFO:teuthology.orchestra.run.smithi114.stdout: "excluded": false, 2024-04-23T10:29:16.716 INFO:teuthology.orchestra.run.smithi114.stdout: "state": { 2024-04-23T10:29:16.716 INFO:teuthology.orchestra.run.smithi114.stdout: "name": "QUIESCED", 2024-04-23T10:29:16.716 INFO:teuthology.orchestra.run.smithi114.stdout: "age": 0.0 2024-04-23T10:29:16.716 INFO:teuthology.orchestra.run.smithi114.stdout: } 2024-04-23T10:29:16.717 INFO:teuthology.orchestra.run.smithi114.stdout: } 2024-04-23T10:29:16.717 INFO:teuthology.orchestra.run.smithi114.stdout: } 2024-04-23T10:29:16.717 INFO:teuthology.orchestra.run.smithi114.stdout: } 2024-04-23T10:29:16.717 INFO:teuthology.orchestra.run.smithi114.stdout: } 2024-04-23T10:29:16.717 INFO:teuthology.orchestra.run.smithi114.stdout:} 2024-04-23T10:29:16.718 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:29:16.715+0000 7fe377e10640 1 -- 172.21.15.114:0/1130005939 >> [v2:172.21.15.114:6800/1395693466,v1:172.21.15.114:6801/1395693466] conn(0x7fe35c0a3d20 msgr2=0x7fe35c0a61e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T10:29:16.718 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:29:16.715+0000 7fe377e10640 1 --2- 172.21.15.114:0/1130005939 >> [v2:172.21.15.114:6800/1395693466,v1:172.21.15.114:6801/1395693466] conn(0x7fe35c0a3d20 0x7fe35c0a61e0 secure :-1 s=READY pgs=82 cs=0 l=1 rev1=1 crypto rx=0x7fe370101a80 tx=0x7fe36c009f70 comp rx=0 tx=0).stop 2024-04-23T10:29:16.718 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:29:16.715+0000 7fe377e10640 1 -- 172.21.15.114:0/1130005939 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fe37019d000 msgr2=0x7fe37019d4b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T10:29:16.718 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:29:16.715+0000 7fe377e10640 1 --2- 172.21.15.114:0/1130005939 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fe37019d000 0x7fe37019d4b0 secure :-1 s=READY pgs=571 cs=0 l=1 rev1=1 crypto rx=0x7fe368005cc0 tx=0x7fe368031700 comp rx=0 tx=0).stop 2024-04-23T10:29:16.718 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:29:16.715+0000 7fe377e10640 1 -- 172.21.15.114:0/1130005939 shutdown_connections 2024-04-23T10:29:16.718 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:29:16.715+0000 7fe377e10640 1 --2- 172.21.15.114:0/1130005939 >> [v2:172.21.15.114:6800/1395693466,v1:172.21.15.114:6801/1395693466] conn(0x7fe35c0a3d20 0x7fe35c0a61e0 unknown :-1 s=CLOSED pgs=82 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:29:16.718 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:29:16.715+0000 7fe377e10640 1 --2- 172.21.15.114:0/1130005939 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fe37019d000 0x7fe37019d4b0 unknown :-1 s=CLOSED pgs=571 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:29:16.718 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:29:16.715+0000 7fe377e10640 1 --2- 172.21.15.114:0/1130005939 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fe370107270 0x7fe370198c80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:29:16.718 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:29:16.715+0000 7fe377e10640 1 --2- 172.21.15.114:0/1130005939 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fe3701021a0 0x7fe370198740 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:29:16.718 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:29:16.715+0000 7fe377e10640 1 -- 172.21.15.114:0/1130005939 >> 172.21.15.114:0/1130005939 conn(0x7fe370079350 msgr2=0x7fe370100040 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T10:29:16.718 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:29:16.715+0000 7fe377e10640 1 -- 172.21.15.114:0/1130005939 shutdown_connections 2024-04-23T10:29:16.718 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:29:16.715+0000 7fe377e10640 1 -- 172.21.15.114:0/1130005939 wait complete. 2024-04-23T10:29:16.725 INFO:tasks.quiescer.fs.[cephfs]:Successfully quiesced set 'fec54601', quiesce took 0.6 seconds. Will release after: 21.45 2024-04-23T10:29:16.910 INFO:tasks.workunit.client.0.smithi114.stdout: 1 177615 11.49 MB/sec warmup 85 sec latency 209.996 ms 2024-04-23T10:29:17.241 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 23 10:29:16 smithi119 ceph-mon[32608]: from='client.? 172.21.15.114:0/508373081' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-23T10:29:17.254 INFO:journalctl@ceph.mon.a.smithi114.stdout:Apr 23 10:29:16 smithi114 ceph-mon[29255]: from='client.? 172.21.15.114:0/508373081' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-23T10:29:17.327 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 23 10:29:16 smithi204 ceph-mon[32649]: from='client.? 172.21.15.114:0/508373081' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-23T10:29:17.910 INFO:tasks.workunit.client.0.smithi114.stdout: 1 177615 11.36 MB/sec warmup 86 sec latency 1210.053 ms 2024-04-23T10:29:18.242 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 23 10:29:17 smithi119 ceph-mon[32608]: pgmap v83: 97 pgs: 97 active+clean; 66 MiB data, 877 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 MiB/s rd, 4.6 MiB/s wr, 455 op/s 2024-04-23T10:29:18.254 INFO:journalctl@ceph.mon.a.smithi114.stdout:Apr 23 10:29:17 smithi114 ceph-mon[29255]: pgmap v83: 97 pgs: 97 active+clean; 66 MiB data, 877 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 MiB/s rd, 4.6 MiB/s wr, 455 op/s 2024-04-23T10:29:18.327 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 23 10:29:17 smithi204 ceph-mon[32649]: pgmap v83: 97 pgs: 97 active+clean; 66 MiB data, 877 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 MiB/s rd, 4.6 MiB/s wr, 455 op/s 2024-04-23T10:29:18.910 INFO:tasks.workunit.client.0.smithi114.stdout: 1 177615 11.23 MB/sec warmup 87 sec latency 2210.091 ms 2024-04-23T10:29:19.910 INFO:tasks.workunit.client.0.smithi114.stdout: 1 177615 11.10 MB/sec warmup 88 sec latency 3210.188 ms 2024-04-23T10:29:20.242 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 23 10:29:19 smithi119 ceph-mon[32608]: pgmap v84: 97 pgs: 97 active+clean; 66 MiB data, 877 MiB used, 1.0 TiB / 1.0 TiB avail; 2.7 MiB/s rd, 3.2 MiB/s wr, 309 op/s 2024-04-23T10:29:20.254 INFO:journalctl@ceph.mon.a.smithi114.stdout:Apr 23 10:29:19 smithi114 ceph-mon[29255]: pgmap v84: 97 pgs: 97 active+clean; 66 MiB data, 877 MiB used, 1.0 TiB / 1.0 TiB avail; 2.7 MiB/s rd, 3.2 MiB/s wr, 309 op/s 2024-04-23T10:29:20.328 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 23 10:29:19 smithi204 ceph-mon[32649]: pgmap v84: 97 pgs: 97 active+clean; 66 MiB data, 877 MiB used, 1.0 TiB / 1.0 TiB avail; 2.7 MiB/s rd, 3.2 MiB/s wr, 309 op/s 2024-04-23T10:29:20.551 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:29:20.551+0000 7fc4f1059640 1 -- 172.21.15.114:0/2391077718 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fc4ec101aa0 msgr2=0x7fc4ec105a90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T10:29:20.551 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:29:20.551+0000 7fc4f1059640 1 --2- 172.21.15.114:0/2391077718 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fc4ec101aa0 0x7fc4ec105a90 secure :-1 s=READY pgs=575 cs=0 l=1 rev1=1 crypto rx=0x7fc4e0009900 tx=0x7fc4e002a410 comp rx=0 tx=0).stop 2024-04-23T10:29:20.551 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:29:20.551+0000 7fc4f1059640 1 -- 172.21.15.114:0/2391077718 shutdown_connections 2024-04-23T10:29:20.551 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:29:20.551+0000 7fc4f1059640 1 --2- 172.21.15.114:0/2391077718 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fc4ec106170 0x7fc4ec10dd40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:29:20.551 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:29:20.551+0000 7fc4f1059640 1 --2- 172.21.15.114:0/2391077718 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fc4ec101aa0 0x7fc4ec105a90 unknown :-1 s=CLOSED pgs=575 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:29:20.551 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:29:20.551+0000 7fc4f1059640 1 --2- 172.21.15.114:0/2391077718 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fc4ec1010f0 0x7fc4ec1014d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:29:20.551 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:29:20.551+0000 7fc4f1059640 1 -- 172.21.15.114:0/2391077718 >> 172.21.15.114:0/2391077718 conn(0x7fc4ec077fd0 msgr2=0x7fc4ec0783e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T10:29:20.551 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:29:20.551+0000 7fc4f1059640 1 -- 172.21.15.114:0/2391077718 shutdown_connections 2024-04-23T10:29:20.551 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:29:20.551+0000 7fc4f1059640 1 -- 172.21.15.114:0/2391077718 wait complete. 2024-04-23T10:29:20.552 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:29:20.552+0000 7fc4f1059640 1 Processor -- start 2024-04-23T10:29:20.553 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:29:20.552+0000 7fc4f1059640 1 -- start start 2024-04-23T10:29:20.554 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:29:20.553+0000 7fc4f1059640 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fc4ec1010f0 0x7fc4ec1975b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T10:29:20.554 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:29:20.553+0000 7fc4f1059640 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fc4ec106170 0x7fc4ec197af0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T10:29:20.554 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:29:20.553+0000 7fc4f1059640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fc4ec19be20 0x7fc4ec19c2d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T10:29:20.554 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:29:20.553+0000 7fc4f1059640 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x7fc4ec111090 con 0x7fc4ec19be20 2024-04-23T10:29:20.554 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:29:20.553+0000 7fc4f1059640 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 -- 0x7fc4ec110f10 con 0x7fc4ec1010f0 2024-04-23T10:29:20.554 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:29:20.553+0000 7fc4f1059640 1 -- --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_getmap magic: 0 -- 0x7fc4ec111210 con 0x7fc4ec106170 2024-04-23T10:29:20.554 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:29:20.553+0000 7fc4eb59e640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fc4ec19be20 0x7fc4ec19c2d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T10:29:20.554 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:29:20.553+0000 7fc4eb59e640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fc4ec19be20 0x7fc4ec19c2d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.114:54480/0 (socket says 172.21.15.114:54480) 2024-04-23T10:29:20.554 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:29:20.553+0000 7fc4eb59e640 1 -- 172.21.15.114:0/3860987664 learned_addr learned my addr 172.21.15.114:0/3860987664 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-04-23T10:29:20.554 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:29:20.553+0000 7fc4ead9d640 1 --2- 172.21.15.114:0/3860987664 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fc4ec1010f0 0x7fc4ec1975b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T10:29:20.554 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:29:20.553+0000 7fc4ea59c640 1 --2- 172.21.15.114:0/3860987664 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fc4ec106170 0x7fc4ec197af0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T10:29:20.554 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:29:20.553+0000 7fc4eb59e640 1 -- 172.21.15.114:0/3860987664 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fc4ec106170 msgr2=0x7fc4ec197af0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T10:29:20.554 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:29:20.553+0000 7fc4eb59e640 1 --2- 172.21.15.114:0/3860987664 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fc4ec106170 0x7fc4ec197af0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:29:20.555 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:29:20.553+0000 7fc4eb59e640 1 -- 172.21.15.114:0/3860987664 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fc4ec1010f0 msgr2=0x7fc4ec1975b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T10:29:20.555 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:29:20.553+0000 7fc4eb59e640 1 --2- 172.21.15.114:0/3860987664 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fc4ec1010f0 0x7fc4ec1975b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:29:20.555 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:29:20.553+0000 7fc4eb59e640 1 -- 172.21.15.114:0/3860987664 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc4ec19c910 con 0x7fc4ec19be20 2024-04-23T10:29:20.555 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:29:20.553+0000 7fc4eb59e640 1 --2- 172.21.15.114:0/3860987664 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fc4ec19be20 0x7fc4ec19c2d0 secure :-1 s=READY pgs=576 cs=0 l=1 rev1=1 crypto rx=0x7fc4e4007900 tx=0x7fc4e4007dc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T10:29:20.555 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:29:20.553+0000 7fc4d3fff640 1 -- 172.21.15.114:0/3860987664 <== mon.0 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc4e400e6f0 con 0x7fc4ec19be20 2024-04-23T10:29:20.555 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:29:20.554+0000 7fc4d3fff640 1 -- 172.21.15.114:0/3860987664 <== mon.0 v2:172.21.15.114:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fc4e400ed40 con 0x7fc4ec19be20 2024-04-23T10:29:20.555 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:29:20.554+0000 7fc4f1059640 1 -- 172.21.15.114:0/3860987664 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fc4ec19cc00 con 0x7fc4ec19be20 2024-04-23T10:29:20.555 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:29:20.554+0000 7fc4d3fff640 1 -- 172.21.15.114:0/3860987664 <== mon.0 v2:172.21.15.114:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc4e400ac40 con 0x7fc4ec19be20 2024-04-23T10:29:20.555 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:29:20.554+0000 7fc4f1059640 1 -- 172.21.15.114:0/3860987664 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fc4ec0698a0 con 0x7fc4ec19be20 2024-04-23T10:29:20.561 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:29:20.556+0000 7fc4d3fff640 1 -- 172.21.15.114:0/3860987664 <== mon.0 v2:172.21.15.114:3300/0 4 ==== mgrmap(e 26) ==== 138778+0+0 (secure 0 0 0) 0x7fc4e400e890 con 0x7fc4ec19be20 2024-04-23T10:29:20.562 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:29:20.556+0000 7fc4f3d6f740 1 Processor -- start 2024-04-23T10:29:20.562 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:29:20.556+0000 7fc4f3d6f740 1 -- start start 2024-04-23T10:29:20.562 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:29:20.556+0000 7fc4f3d6f740 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x5587111df500 0x5587111df8e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T10:29:20.562 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:29:20.557+0000 7fc4f3d6f740 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x5587111dfe20 0x5587111ee6d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T10:29:20.562 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:29:20.557+0000 7fc4f3d6f740 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x5587111eec10 0x5587111f1000 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T10:29:20.562 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:29:20.557+0000 7fc4f3d6f740 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x558711144970 con 0x5587111df500 2024-04-23T10:29:20.562 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:29:20.557+0000 7fc4f3d6f740 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 -- 0x5587110e14a0 con 0x5587111dfe20 2024-04-23T10:29:20.562 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:29:20.557+0000 7fc4f3d6f740 1 -- --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_getmap magic: 0 -- 0x558711140930 con 0x5587111eec10 2024-04-23T10:29:20.562 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:29:20.557+0000 7fc4d3fff640 1 --2- 172.21.15.114:0/3860987664 >> [v2:172.21.15.114:6800/1395693466,v1:172.21.15.114:6801/1395693466] conn(0x7fc4d40a3dc0 0x7fc4d40a6280 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T10:29:20.562 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:29:20.557+0000 7fc4ea59c640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x5587111df500 0x5587111df8e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T10:29:20.562 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:29:20.557+0000 7fc4ea59c640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x5587111df500 0x5587111df8e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.114:54494/0 (socket says 172.21.15.114:54494) 2024-04-23T10:29:20.563 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:29:20.557+0000 7fc4ea59c640 1 -- 172.21.15.114:0/319916249 learned_addr learned my addr 172.21.15.114:0/319916249 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-04-23T10:29:20.563 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:29:20.557+0000 7fc4eb59e640 1 --2- 172.21.15.114:0/3860987664 >> [v2:172.21.15.114:6800/1395693466,v1:172.21.15.114:6801/1395693466] conn(0x7fc4d40a3dc0 0x7fc4d40a6280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T10:29:20.563 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:29:20.557+0000 7fc4ead9d640 1 --2- 172.21.15.114:0/319916249 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x5587111dfe20 0x5587111ee6d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T10:29:20.563 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:29:20.557+0000 7fc4d3fff640 1 -- 172.21.15.114:0/3860987664 <== mon.0 v2:172.21.15.114:3300/0 5 ==== osd_map(92..92 src has 1..92) ==== 8585+0+0 (secure 0 0 0) 0x7fc4e40d2270 con 0x7fc4ec19be20 2024-04-23T10:29:20.563 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:29:20.557+0000 7fc4ea59c640 1 -- 172.21.15.114:0/319916249 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x5587111eec10 msgr2=0x5587111f1000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T10:29:20.563 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:29:20.557+0000 7fc4ea59c640 1 --2- 172.21.15.114:0/319916249 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x5587111eec10 0x5587111f1000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:29:20.563 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:29:20.557+0000 7fc4ea59c640 1 -- 172.21.15.114:0/319916249 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x5587111dfe20 msgr2=0x5587111ee6d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T10:29:20.563 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:29:20.557+0000 7fc4ea59c640 1 --2- 172.21.15.114:0/319916249 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x5587111dfe20 0x5587111ee6d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:29:20.563 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:29:20.557+0000 7fc4ea59c640 1 -- 172.21.15.114:0/319916249 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x558710e57210 con 0x5587111df500 2024-04-23T10:29:20.563 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:29:20.557+0000 7fc4ea59c640 1 --2- 172.21.15.114:0/319916249 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x5587111df500 0x5587111df8e0 secure :-1 s=READY pgs=577 cs=0 l=1 rev1=1 crypto rx=0x7fc4ec06b730 tx=0x7fc4e0005e10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e6dbe200403912f0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T10:29:20.563 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:29:20.557+0000 7fc4eb59e640 1 --2- 172.21.15.114:0/3860987664 >> [v2:172.21.15.114:6800/1395693466,v1:172.21.15.114:6801/1395693466] conn(0x7fc4d40a3dc0 0x7fc4d40a6280 secure :-1 s=READY pgs=83 cs=0 l=1 rev1=1 crypto rx=0x7fc4e4070c30 tx=0x7fc4e4070c60 comp rx=0 tx=0).ready entity=mgr.15291 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T10:29:20.563 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:29:20.557+0000 7fc4d17fa640 1 -- 172.21.15.114:0/319916249 <== mon.0 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc4e0035050 con 0x5587111df500 2024-04-23T10:29:20.563 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:29:20.557+0000 7fc4d17fa640 1 -- 172.21.15.114:0/319916249 <== mon.0 v2:172.21.15.114:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fc4e00316a0 con 0x5587111df500 2024-04-23T10:29:20.563 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:29:20.557+0000 7fc4d17fa640 1 -- 172.21.15.114:0/319916249 <== mon.0 v2:172.21.15.114:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc4e00046c0 con 0x5587111df500 2024-04-23T10:29:20.563 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:29:20.558+0000 7fc4f3d6f740 1 -- 172.21.15.114:0/319916249 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x5587111df500 msgr2=0x5587111df8e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T10:29:20.564 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:29:20.558+0000 7fc4f3d6f740 1 --2- 172.21.15.114:0/319916249 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x5587111df500 0x5587111df8e0 secure :-1 s=READY pgs=577 cs=0 l=1 rev1=1 crypto rx=0x7fc4ec06b730 tx=0x7fc4e0005e10 comp rx=0 tx=0).stop 2024-04-23T10:29:20.564 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:29:20.558+0000 7fc4f3d6f740 1 -- 172.21.15.114:0/319916249 shutdown_connections 2024-04-23T10:29:20.564 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:29:20.558+0000 7fc4f3d6f740 1 --2- 172.21.15.114:0/319916249 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x5587111eec10 0x5587111f1000 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:29:20.564 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:29:20.558+0000 7fc4f3d6f740 1 --2- 172.21.15.114:0/319916249 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x5587111dfe20 0x5587111ee6d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:29:20.564 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:29:20.558+0000 7fc4f3d6f740 1 --2- 172.21.15.114:0/319916249 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x5587111df500 0x5587111df8e0 unknown :-1 s=CLOSED pgs=577 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:29:20.564 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:29:20.558+0000 7fc4f3d6f740 1 -- 172.21.15.114:0/319916249 >> 172.21.15.114:0/319916249 conn(0x5587111dbb30 msgr2=0x5587111dbf10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T10:29:20.564 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:29:20.558+0000 7fc4f3d6f740 1 -- 172.21.15.114:0/319916249 shutdown_connections 2024-04-23T10:29:20.564 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:29:20.558+0000 7fc4f3d6f740 1 -- 172.21.15.114:0/319916249 wait complete. 2024-04-23T10:29:20.564 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:29:20.559+0000 7fc4f3d6f740 1 -- start start 2024-04-23T10:29:20.564 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:29:20.559+0000 7fc4f3d6f740 1 Processor -- start 2024-04-23T10:29:20.564 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:29:20.559+0000 7fc4f3d6f740 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x5587111df500 0x5587111ee540 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T10:29:20.564 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:29:20.559+0000 7fc4f3d6f740 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x5587111dfe20 0x5587111eb830 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T10:29:20.564 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:29:20.559+0000 7fc4f3d6f740 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x5587111eec10 0x5587111ebd70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T10:29:20.564 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:29:20.559+0000 7fc4f3d6f740 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x5587111f17e0 con 0x5587111eec10 2024-04-23T10:29:20.564 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:29:20.559+0000 7fc4f3d6f740 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 -- 0x5587110df0b0 con 0x5587111df500 2024-04-23T10:29:20.565 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:29:20.559+0000 7fc4f3d6f740 1 -- --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_getmap magic: 0 -- 0x5587111f1960 con 0x5587111dfe20 2024-04-23T10:29:20.565 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:29:20.559+0000 7fc4f3d6f740 10 client.0 _ll_register_callbacks cb 0x5587110ccd30 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-23T10:29:20.565 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:29:20.559+0000 7fc4ea59c640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x5587111eec10 0x5587111ebd70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T10:29:20.565 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:29:20.559+0000 7fc4ea59c640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x5587111eec10 0x5587111ebd70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.114:54500/0 (socket says 172.21.15.114:54500) 2024-04-23T10:29:20.565 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:29:20.559+0000 7fc4ea59c640 1 -- 172.21.15.114:0/1241679964 learned_addr learned my addr 172.21.15.114:0/1241679964 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-04-23T10:29:20.565 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:29:20.559+0000 7fc4ea59c640 1 --2- 172.21.15.114:0/1241679964 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x5587111df500 0x5587111ee540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T10:29:20.565 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:29:20.559+0000 7fc4ead9d640 1 --2- 172.21.15.114:0/1241679964 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x5587111dfe20 0x5587111eb830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T10:29:20.565 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:29:20.559+0000 7fc4ea59c640 1 -- 172.21.15.114:0/1241679964 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x5587111dfe20 msgr2=0x5587111eb830 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T10:29:20.565 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:29:20.559+0000 7fc4ea59c640 1 --2- 172.21.15.114:0/1241679964 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x5587111dfe20 0x5587111eb830 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:29:20.565 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:29:20.559+0000 7fc4ea59c640 1 -- 172.21.15.114:0/1241679964 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x5587111df500 msgr2=0x5587111ee540 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T10:29:20.565 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:29:20.559+0000 7fc4ea59c640 1 --2- 172.21.15.114:0/1241679964 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x5587111df500 0x5587111ee540 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:29:20.565 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:29:20.559+0000 7fc4ea59c640 1 -- 172.21.15.114:0/1241679964 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({osdmap=0}) -- 0x558710f23620 con 0x5587111eec10 2024-04-23T10:29:20.565 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:29:20.560+0000 7fc4ea59c640 1 --2- 172.21.15.114:0/1241679964 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x5587111eec10 0x5587111ebd70 secure :-1 s=READY pgs=578 cs=0 l=1 rev1=1 crypto rx=0x7fc4e0032fd0 tx=0x7fc4e0005340 comp rx=0 tx=0).ready entity=mon.0 client_cookie=cb160073769a9cb1 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T10:29:20.566 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:29:20.560+0000 7fc4ceff5640 10 client.0 ms_handle_connect on v2:172.21.15.114:3300/0 2024-04-23T10:29:20.566 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:29:20.560+0000 7fc4ceff5640 1 -- 172.21.15.114:0/1241679964 <== mon.0 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc4e0035040 con 0x5587111eec10 2024-04-23T10:29:20.566 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:29:20.560+0000 7fc4ceff5640 1 -- 172.21.15.114:0/1241679964 <== mon.0 v2:172.21.15.114:3300/0 2 ==== osd_map(92..92 src has 1..92) ==== 8585+0+0 (secure 0 0 0) 0x7fc4e0004260 con 0x5587111eec10 2024-04-23T10:29:20.566 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:29:20.560+0000 7fc4f3d6f740 1 -- 172.21.15.114:0/1241679964 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fc4e00316a0 con 0x5587111eec10 2024-04-23T10:29:20.566 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:29:20.560+0000 7fc4ceff5640 1 -- 172.21.15.114:0/1241679964 <== mon.0 v2:172.21.15.114:3300/0 3 ==== mon_get_version_reply(handle=1 version=23) ==== 24+0+0 (secure 0 0 0) 0x7fc4e0059050 con 0x5587111eec10 2024-04-23T10:29:20.566 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:29:20.560+0000 7fc4f3d6f740 10 client.15888 fetch_fsmap learned FSMap version 23 2024-04-23T10:29:20.566 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:29:20.560+0000 7fc4f3d6f740 1 -- 172.21.15.114:0/1241679964 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({config=0+,fsmap=23,monmap=4+}) -- 0x5587111fd770 con 0x5587111eec10 2024-04-23T10:29:20.566 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:29:20.560+0000 7fc4ceff5640 1 -- 172.21.15.114:0/1241679964 <== mon.0 v2:172.21.15.114:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fc4e00316a0 con 0x5587111eec10 2024-04-23T10:29:20.566 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:29:20.560+0000 7fc4ceff5640 1 -- 172.21.15.114:0/1241679964 <== mon.0 v2:172.21.15.114:3300/0 5 ==== fsmap(e 23) ==== 8350+0+0 (secure 0 0 0) 0x7fc4e00039c0 con 0x5587111eec10 2024-04-23T10:29:20.566 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:29:20.560+0000 7fc4f3d6f740 10 client.15888 fetch_fsmap finished waiting for FSMap version 23 2024-04-23T10:29:20.566 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:29:20.560+0000 7fc4f3d6f740 10 client.15888 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-04-23T10:29:20.566 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:29:20.560+0000 7fc4f3d6f740 20 client.15888 populate_metadata read hostname 'smithi114' 2024-04-23T10:29:20.566 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:29:20.560+0000 7fc4f3d6f740 1 --2- 172.21.15.114:0/1241679964 >> [v2:172.21.15.119:6838/937297340,v1:172.21.15.119:6839/937297340] conn(0x5587111feac0 0x558711200eb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T10:29:20.567 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:29:20.560+0000 7fc4f3d6f740 4 client.15888 mds_command: new command op to 24545 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-23T10:29:20.567 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:29:20.560+0000 7fc4f3d6f740 1 -- 172.21.15.114:0/1241679964 --> [v2:172.21.15.119:6838/937297340,v1:172.21.15.119:6839/937297340] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x558711152390 con 0x5587111feac0 2024-04-23T10:29:20.567 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:29:20.561+0000 7fc4ead9d640 1 --2- 172.21.15.114:0/1241679964 >> [v2:172.21.15.119:6838/937297340,v1:172.21.15.119:6839/937297340] conn(0x5587111feac0 0x558711200eb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T10:29:20.567 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:29:20.561+0000 7fc4ceff5640 10 client.15888 ms_handle_connect on v2:172.21.15.119:6838/937297340 2024-04-23T10:29:20.567 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:29:20.561+0000 7fc4ead9d640 1 --2- 172.21.15.114:0/1241679964 >> [v2:172.21.15.119:6838/937297340,v1:172.21.15.119:6839/937297340] conn(0x5587111feac0 0x558711200eb0 crc :-1 s=READY pgs=72 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=f4f3ba7b86e4e798 server_cookie=cb27d6d51c767972 in_seq=0 out_seq=0 2024-04-23T10:29:20.567 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:29:20.562+0000 7fc4ceff5640 1 -- 172.21.15.114:0/1241679964 <== mds.0 v2:172.21.15.119:6838/937297340 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x558711152390 con 0x5587111feac0 2024-04-23T10:29:20.567 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:29:20.562+0000 7fc4ceff5640 10 client.15888 handle_command_reply: tid=0 2024-04-23T10:29:20.567 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:29:20.562+0000 7fc4f3d6f740 1 client.15888 shutdown 2024-04-23T10:29:20.567 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:29:20.562+0000 7fc4f3d6f740 1 -- 172.21.15.114:0/1241679964 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x5587111eec10 msgr2=0x5587111ebd70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T10:29:20.567 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:29:20.562+0000 7fc4f3d6f740 1 --2- 172.21.15.114:0/1241679964 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x5587111eec10 0x5587111ebd70 secure :-1 s=READY pgs=578 cs=0 l=1 rev1=1 crypto rx=0x7fc4e0032fd0 tx=0x7fc4e0005340 comp rx=0 tx=0).stop 2024-04-23T10:29:20.567 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:29:20.562+0000 7fc4f3d6f740 1 -- 172.21.15.114:0/1241679964 shutdown_connections 2024-04-23T10:29:20.567 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:29:20.562+0000 7fc4f3d6f740 1 --2- 172.21.15.114:0/1241679964 >> [v2:172.21.15.119:6838/937297340,v1:172.21.15.119:6839/937297340] conn(0x5587111feac0 0x558711200eb0 crc :-1 s=READY pgs=72 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:29:20.567 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:29:20.562+0000 7fc4f3d6f740 1 --2- 172.21.15.114:0/1241679964 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x5587111eec10 0x5587111ebd70 unknown :-1 s=CLOSED pgs=578 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:29:20.567 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:29:20.562+0000 7fc4ceff5640 0 client.15888 ms_handle_reset on v2:172.21.15.119:6838/937297340 2024-04-23T10:29:20.567 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:29:20.562+0000 7fc4f3d6f740 1 --2- 172.21.15.114:0/1241679964 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x5587111dfe20 0x5587111eb830 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:29:20.568 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:29:20.562+0000 7fc4f3d6f740 1 --2- 172.21.15.114:0/1241679964 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x5587111df500 0x5587111ee540 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:29:20.568 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:29:20.562+0000 7fc4f3d6f740 1 -- 172.21.15.114:0/1241679964 >> 172.21.15.114:0/1241679964 conn(0x5587111dbb30 msgr2=0x5587111e1940 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T10:29:20.568 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:29:20.562+0000 7fc4f3d6f740 1 -- 172.21.15.114:0/1241679964 shutdown_connections 2024-04-23T10:29:20.568 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:29:20.562+0000 7fc4f3d6f740 1 -- 172.21.15.114:0/1241679964 wait complete. 2024-04-23T10:29:20.568 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:29:20.562+0000 7fc4f3d6f740 20 client.15888 trim_cache size 0 max 16384 2024-04-23T10:29:20.580 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:29:20.577+0000 7fc4f3d6f740 1 Processor -- start 2024-04-23T10:29:20.580 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:29:20.577+0000 7fc4f3d6f740 1 -- start start 2024-04-23T10:29:20.580 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:29:20.577+0000 7fc4f3d6f740 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x5587111dbb30 0x5587111fa6e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T10:29:20.580 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:29:20.577+0000 7fc4f3d6f740 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x5587111df500 0x5587111ea810 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T10:29:20.580 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:29:20.577+0000 7fc4f3d6f740 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x5587111dfe20 0x5587111ee070 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T10:29:20.581 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:29:20.577+0000 7fc4f3d6f740 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x7fc4ec1ae1c0 con 0x5587111dfe20 2024-04-23T10:29:20.581 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:29:20.577+0000 7fc4f3d6f740 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 -- 0x7fc4ec1aba60 con 0x5587111df500 2024-04-23T10:29:20.581 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:29:20.577+0000 7fc4f3d6f740 1 -- --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_getmap magic: 0 -- 0x7fc4ec1a98b0 con 0x5587111dbb30 2024-04-23T10:29:20.581 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:29:20.577+0000 7fc4ea59c640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x5587111dfe20 0x5587111ee070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T10:29:20.581 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:29:20.577+0000 7fc4ea59c640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x5587111dfe20 0x5587111ee070 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.114:54514/0 (socket says 172.21.15.114:54514) 2024-04-23T10:29:20.581 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:29:20.577+0000 7fc4ea59c640 1 -- 172.21.15.114:0/3583625284 learned_addr learned my addr 172.21.15.114:0/3583625284 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-04-23T10:29:20.581 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:29:20.577+0000 7fc4ea59c640 1 -- 172.21.15.114:0/3583625284 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x5587111dbb30 msgr2=0x5587111fa6e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T10:29:20.581 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:29:20.577+0000 7fc4ea59c640 1 --2- 172.21.15.114:0/3583625284 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x5587111dbb30 0x5587111fa6e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:29:20.581 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:29:20.577+0000 7fc4ea59c640 1 -- 172.21.15.114:0/3583625284 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x5587111df500 msgr2=0x5587111ea810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T10:29:20.581 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:29:20.577+0000 7fc4ead9d640 1 --2- 172.21.15.114:0/3583625284 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x5587111df500 0x5587111ea810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T10:29:20.581 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:29:20.577+0000 7fc4ea59c640 1 --2- 172.21.15.114:0/3583625284 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x5587111df500 0x5587111ea810 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:29:20.581 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:29:20.577+0000 7fc4ea59c640 1 -- 172.21.15.114:0/3583625284 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5587111dfc50 con 0x5587111dfe20 2024-04-23T10:29:20.581 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:29:20.577+0000 7fc4ea59c640 1 --2- 172.21.15.114:0/3583625284 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x5587111dfe20 0x5587111ee070 secure :-1 s=READY pgs=579 cs=0 l=1 rev1=1 crypto rx=0x7fc4e0005270 tx=0x7fc4e00314a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f6816190f65735e7 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T10:29:20.581 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:29:20.578+0000 7fc4d0ff9640 1 -- 172.21.15.114:0/3583625284 <== mon.0 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc4e0035070 con 0x5587111dfe20 2024-04-23T10:29:20.582 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:29:20.578+0000 7fc4d0ff9640 1 -- 172.21.15.114:0/3583625284 <== mon.0 v2:172.21.15.114:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fc4e0032df0 con 0x5587111dfe20 2024-04-23T10:29:20.582 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:29:20.578+0000 7fc4d0ff9640 1 -- 172.21.15.114:0/3583625284 <== mon.0 v2:172.21.15.114:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc4e0003850 con 0x5587111dfe20 2024-04-23T10:29:20.582 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:29:20.578+0000 7fc4f3d6f740 1 -- 172.21.15.114:0/3583625284 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x5587111dfe20 msgr2=0x5587111ee070 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T10:29:20.582 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:29:20.578+0000 7fc4f3d6f740 1 --2- 172.21.15.114:0/3583625284 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x5587111dfe20 0x5587111ee070 secure :-1 s=READY pgs=579 cs=0 l=1 rev1=1 crypto rx=0x7fc4e0005270 tx=0x7fc4e00314a0 comp rx=0 tx=0).stop 2024-04-23T10:29:20.582 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:29:20.578+0000 7fc4f3d6f740 1 -- 172.21.15.114:0/3583625284 shutdown_connections 2024-04-23T10:29:20.582 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:29:20.578+0000 7fc4f3d6f740 1 --2- 172.21.15.114:0/3583625284 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x5587111dfe20 0x5587111ee070 unknown :-1 s=CLOSED pgs=579 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:29:20.582 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:29:20.578+0000 7fc4f3d6f740 1 --2- 172.21.15.114:0/3583625284 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x5587111df500 0x5587111ea810 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:29:20.582 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:29:20.578+0000 7fc4f3d6f740 1 --2- 172.21.15.114:0/3583625284 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x5587111dbb30 0x5587111fa6e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:29:20.582 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:29:20.578+0000 7fc4f3d6f740 1 -- 172.21.15.114:0/3583625284 >> 172.21.15.114:0/3583625284 conn(0x5587111ebed0 msgr2=0x5587111ea100 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T10:29:20.582 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:29:20.578+0000 7fc4f3d6f740 1 -- 172.21.15.114:0/3583625284 shutdown_connections 2024-04-23T10:29:20.582 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:29:20.578+0000 7fc4f3d6f740 1 -- 172.21.15.114:0/3583625284 wait complete. 2024-04-23T10:29:20.582 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:29:20.579+0000 7fc4f3d6f740 1 -- start start 2024-04-23T10:29:20.582 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:29:20.579+0000 7fc4f3d6f740 1 Processor -- start 2024-04-23T10:29:20.582 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:29:20.579+0000 7fc4f3d6f740 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x5587111dbb30 0x5587111f6f70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T10:29:20.582 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:29:20.579+0000 7fc4f3d6f740 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x5587111df500 0x5587111f22e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T10:29:20.583 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:29:20.579+0000 7fc4f3d6f740 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x5587111dfe20 0x5587111f2820 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T10:29:20.583 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:29:20.579+0000 7fc4f3d6f740 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x7fc4ec1a9f20 con 0x5587111df500 2024-04-23T10:29:20.583 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:29:20.579+0000 7fc4f3d6f740 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 -- 0x7fc4ec1ab390 con 0x5587111dfe20 2024-04-23T10:29:20.583 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:29:20.579+0000 7fc4f3d6f740 1 -- --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_getmap magic: 0 -- 0x7fc4ec1aa5f0 con 0x5587111dbb30 2024-04-23T10:29:20.583 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:29:20.579+0000 7fc4f3d6f740 10 client.0 _ll_register_callbacks cb 0x7fc4e0031920 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-23T10:29:20.583 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:29:20.579+0000 7fc4ead9d640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x5587111df500 0x5587111f22e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T10:29:20.583 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:29:20.579+0000 7fc4ead9d640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x5587111df500 0x5587111f22e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.114:54518/0 (socket says 172.21.15.114:54518) 2024-04-23T10:29:20.583 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:29:20.579+0000 7fc4ead9d640 1 -- 172.21.15.114:0/115642431 learned_addr learned my addr 172.21.15.114:0/115642431 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-04-23T10:29:20.583 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:29:20.579+0000 7fc4ea59c640 1 --2- 172.21.15.114:0/115642431 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x5587111dfe20 0x5587111f2820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T10:29:20.583 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:29:20.579+0000 7fc4ea59c640 1 --2- 172.21.15.114:0/115642431 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x5587111dbb30 0x5587111f6f70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T10:29:20.583 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:29:20.579+0000 7fc4ead9d640 1 -- 172.21.15.114:0/115642431 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x5587111dbb30 msgr2=0x5587111f6f70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T10:29:20.583 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:29:20.579+0000 7fc4ead9d640 1 --2- 172.21.15.114:0/115642431 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x5587111dbb30 0x5587111f6f70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:29:20.583 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:29:20.579+0000 7fc4ead9d640 1 -- 172.21.15.114:0/115642431 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x5587111dfe20 msgr2=0x5587111f2820 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T10:29:20.583 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:29:20.579+0000 7fc4ead9d640 1 --2- 172.21.15.114:0/115642431 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x5587111dfe20 0x5587111f2820 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:29:20.583 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:29:20.579+0000 7fc4ead9d640 1 -- 172.21.15.114:0/115642431 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5587111f3020 con 0x5587111df500 2024-04-23T10:29:20.584 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:29:20.579+0000 7fc4ead9d640 1 --2- 172.21.15.114:0/115642431 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x5587111df500 0x5587111f22e0 secure :-1 s=READY pgs=580 cs=0 l=1 rev1=1 crypto rx=0x7fc4dc0095d0 tx=0x7fc4dc012c40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5be3fdac3ab7fe38 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T10:29:20.584 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:29:20.579+0000 7fc4ceff5640 10 client.0 ms_handle_connect on v2:172.21.15.114:3300/0 2024-04-23T10:29:20.584 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:29:20.580+0000 7fc4ceff5640 1 -- 172.21.15.114:0/115642431 <== mon.0 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc4dc004d20 con 0x5587111df500 2024-04-23T10:29:20.584 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:29:20.580+0000 7fc4f3d6f740 1 -- 172.21.15.114:0/115642431 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fc4e0032df0 con 0x5587111df500 2024-04-23T10:29:20.584 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:29:20.580+0000 7fc4ceff5640 1 -- 172.21.15.114:0/115642431 <== mon.0 v2:172.21.15.114:3300/0 2 ==== osd_map(92..92 src has 1..92) ==== 8585+0+0 (secure 0 0 0) 0x7fc4dc011aa0 con 0x5587111df500 2024-04-23T10:29:20.584 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:29:20.580+0000 7fc4ceff5640 1 -- 172.21.15.114:0/115642431 <== mon.0 v2:172.21.15.114:3300/0 3 ==== mon_get_version_reply(handle=1 version=23) ==== 24+0+0 (secure 0 0 0) 0x7fc4dc02d050 con 0x5587111df500 2024-04-23T10:29:20.584 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:29:20.580+0000 7fc4f3d6f740 10 client.15894 fetch_fsmap learned FSMap version 23 2024-04-23T10:29:20.584 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:29:20.580+0000 7fc4f3d6f740 1 -- 172.21.15.114:0/115642431 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({config=0+,fsmap=23,monmap=4+}) -- 0x558711264e80 con 0x5587111df500 2024-04-23T10:29:20.584 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:29:20.580+0000 7fc4ceff5640 1 -- 172.21.15.114:0/115642431 <== mon.0 v2:172.21.15.114:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fc4e0032df0 con 0x5587111df500 2024-04-23T10:29:20.584 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:29:20.580+0000 7fc4ceff5640 1 -- 172.21.15.114:0/115642431 <== mon.0 v2:172.21.15.114:3300/0 5 ==== fsmap(e 23) ==== 8350+0+0 (secure 0 0 0) 0x7fc4dc005020 con 0x5587111df500 2024-04-23T10:29:20.584 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:29:20.580+0000 7fc4f3d6f740 10 client.15894 fetch_fsmap finished waiting for FSMap version 23 2024-04-23T10:29:20.584 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:29:20.580+0000 7fc4f3d6f740 10 client.15894 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-04-23T10:29:20.584 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:29:20.580+0000 7fc4f3d6f740 20 client.15894 populate_metadata read hostname 'smithi114' 2024-04-23T10:29:20.584 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:29:20.580+0000 7fc4f3d6f740 1 --2- 172.21.15.114:0/115642431 >> [v2:172.21.15.119:6838/937297340,v1:172.21.15.119:6839/937297340] conn(0x5587111eec10 0x558711267720 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T10:29:20.584 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:29:20.580+0000 7fc4f3d6f740 4 client.15894 mds_command: new command op to 24545 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-04-23T10:29:20.584 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:29:20.580+0000 7fc4f3d6f740 1 -- 172.21.15.114:0/115642431 --> [v2:172.21.15.119:6838/937297340,v1:172.21.15.119:6839/937297340] -- command(tid 0: {"prefix": "flush journal"}) -- 0x55871114c200 con 0x5587111eec10 2024-04-23T10:29:20.585 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:29:20.580+0000 7fc4ea59c640 1 --2- 172.21.15.114:0/115642431 >> [v2:172.21.15.119:6838/937297340,v1:172.21.15.119:6839/937297340] conn(0x5587111eec10 0x558711267720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T10:29:20.585 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:29:20.581+0000 7fc4ea59c640 1 --2- 172.21.15.114:0/115642431 >> [v2:172.21.15.119:6838/937297340,v1:172.21.15.119:6839/937297340] conn(0x5587111eec10 0x558711267720 crc :-1 s=READY pgs=73 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=8ac2db09526b457a server_cookie=a876fd3495f9c8ab in_seq=0 out_seq=0 2024-04-23T10:29:20.585 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:29:20.581+0000 7fc4ceff5640 10 client.15894 ms_handle_connect on v2:172.21.15.119:6838/937297340 2024-04-23T10:29:20.638 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:29:20.639+0000 7fc4ceff5640 1 -- 172.21.15.114:0/115642431 <== mds.0 v2:172.21.15.119:6838/937297340 1 ==== command_reply(tid 0: 0 ) ==== 8+0+44 (crc 0 0 0) 0x55871114c200 con 0x5587111eec10 2024-04-23T10:29:20.638 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:29:20.639+0000 7fc4ceff5640 10 client.15894 handle_command_reply: tid=0 2024-04-23T10:29:20.638 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:29:20.639+0000 7fc4f3d6f740 1 client.15894 shutdown 2024-04-23T10:29:20.639 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:29:20.639+0000 7fc4f3d6f740 1 -- 172.21.15.114:0/115642431 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x5587111df500 msgr2=0x5587111f22e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T10:29:20.639 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:29:20.639+0000 7fc4f3d6f740 1 --2- 172.21.15.114:0/115642431 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x5587111df500 0x5587111f22e0 secure :-1 s=READY pgs=580 cs=0 l=1 rev1=1 crypto rx=0x7fc4dc0095d0 tx=0x7fc4dc012c40 comp rx=0 tx=0).stop 2024-04-23T10:29:20.639 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:29:20.639+0000 7fc4f3d6f740 1 -- 172.21.15.114:0/115642431 shutdown_connections 2024-04-23T10:29:20.639 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:29:20.639+0000 7fc4f3d6f740 1 --2- 172.21.15.114:0/115642431 >> [v2:172.21.15.119:6838/937297340,v1:172.21.15.119:6839/937297340] conn(0x5587111eec10 0x558711267720 crc :-1 s=READY pgs=73 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:29:20.639 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:29:20.639+0000 7fc4f3d6f740 1 --2- 172.21.15.114:0/115642431 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x5587111dfe20 0x5587111f2820 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:29:20.639 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:29:20.639+0000 7fc4ceff5640 0 client.15894 ms_handle_reset on v2:172.21.15.119:6838/937297340 2024-04-23T10:29:20.639 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:29:20.639+0000 7fc4f3d6f740 1 --2- 172.21.15.114:0/115642431 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x5587111df500 0x5587111f22e0 unknown :-1 s=CLOSED pgs=580 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:29:20.639 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:29:20.639+0000 7fc4f3d6f740 1 --2- 172.21.15.114:0/115642431 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x5587111dbb30 0x5587111f6f70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:29:20.639 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:29:20.639+0000 7fc4f3d6f740 1 -- 172.21.15.114:0/115642431 >> 172.21.15.114:0/115642431 conn(0x5587111ebed0 msgr2=0x5587111da420 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T10:29:20.639 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:29:20.639+0000 7fc4f3d6f740 1 -- 172.21.15.114:0/115642431 shutdown_connections 2024-04-23T10:29:20.639 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:29:20.639+0000 7fc4f3d6f740 1 -- 172.21.15.114:0/115642431 wait complete. 2024-04-23T10:29:20.639 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:29:20.639+0000 7fc4f3d6f740 20 client.15894 trim_cache size 0 max 16384 2024-04-23T10:29:20.640 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stdout:{ 2024-04-23T10:29:20.640 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stdout: "message": "", 2024-04-23T10:29:20.640 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stdout: "return_code": 0 2024-04-23T10:29:20.640 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stdout:} 2024-04-23T10:29:20.641 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:29:20.640+0000 7fc4f1059640 1 -- 172.21.15.114:0/3860987664 >> [v2:172.21.15.114:6800/1395693466,v1:172.21.15.114:6801/1395693466] conn(0x7fc4d40a3dc0 msgr2=0x7fc4d40a6280 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T10:29:20.641 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:29:20.640+0000 7fc4f1059640 1 --2- 172.21.15.114:0/3860987664 >> [v2:172.21.15.114:6800/1395693466,v1:172.21.15.114:6801/1395693466] conn(0x7fc4d40a3dc0 0x7fc4d40a6280 secure :-1 s=READY pgs=83 cs=0 l=1 rev1=1 crypto rx=0x7fc4e4070c30 tx=0x7fc4e4070c60 comp rx=0 tx=0).stop 2024-04-23T10:29:20.641 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:29:20.640+0000 7fc4f1059640 1 -- 172.21.15.114:0/3860987664 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fc4ec19be20 msgr2=0x7fc4ec19c2d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T10:29:20.641 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:29:20.640+0000 7fc4f1059640 1 --2- 172.21.15.114:0/3860987664 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fc4ec19be20 0x7fc4ec19c2d0 secure :-1 s=READY pgs=576 cs=0 l=1 rev1=1 crypto rx=0x7fc4e4007900 tx=0x7fc4e4007dc0 comp rx=0 tx=0).stop 2024-04-23T10:29:20.641 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:29:20.640+0000 7fc4f1059640 1 -- 172.21.15.114:0/3860987664 shutdown_connections 2024-04-23T10:29:20.641 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:29:20.640+0000 7fc4f1059640 1 --2- 172.21.15.114:0/3860987664 >> [v2:172.21.15.114:6800/1395693466,v1:172.21.15.114:6801/1395693466] conn(0x7fc4d40a3dc0 0x7fc4d40a6280 unknown :-1 s=CLOSED pgs=83 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:29:20.641 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:29:20.640+0000 7fc4f1059640 1 --2- 172.21.15.114:0/3860987664 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fc4ec19be20 0x7fc4ec19c2d0 unknown :-1 s=CLOSED pgs=576 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:29:20.641 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:29:20.640+0000 7fc4f1059640 1 --2- 172.21.15.114:0/3860987664 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fc4ec106170 0x7fc4ec197af0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:29:20.641 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:29:20.640+0000 7fc4f1059640 1 --2- 172.21.15.114:0/3860987664 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fc4ec1010f0 0x7fc4ec1975b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:29:20.641 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:29:20.640+0000 7fc4f1059640 1 -- 172.21.15.114:0/3860987664 >> 172.21.15.114:0/3860987664 conn(0x7fc4ec077fd0 msgr2=0x7fc4ec0ff030 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T10:29:20.641 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:29:20.640+0000 7fc4f1059640 1 -- 172.21.15.114:0/3860987664 shutdown_connections 2024-04-23T10:29:20.641 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:29:20.640+0000 7fc4f1059640 1 -- 172.21.15.114:0/3860987664 wait complete. 2024-04-23T10:29:20.910 INFO:tasks.workunit.client.0.smithi114.stdout: 1 177615 10.98 MB/sec warmup 89 sec latency 4210.255 ms 2024-04-23T10:29:21.911 INFO:tasks.workunit.client.0.smithi114.stdout: 1 177615 10.85 MB/sec warmup 90 sec latency 5210.340 ms 2024-04-23T10:29:22.241 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 23 10:29:21 smithi119 ceph-mon[32608]: pgmap v85: 97 pgs: 97 active+clean; 66 MiB data, 877 MiB used, 1.0 TiB / 1.0 TiB avail; 2.7 MiB/s rd, 3.2 MiB/s wr, 309 op/s 2024-04-23T10:29:22.254 INFO:journalctl@ceph.mon.a.smithi114.stdout:Apr 23 10:29:21 smithi114 ceph-mon[29255]: pgmap v85: 97 pgs: 97 active+clean; 66 MiB data, 877 MiB used, 1.0 TiB / 1.0 TiB avail; 2.7 MiB/s rd, 3.2 MiB/s wr, 309 op/s 2024-04-23T10:29:22.328 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 23 10:29:21 smithi204 ceph-mon[32649]: pgmap v85: 97 pgs: 97 active+clean; 66 MiB data, 877 MiB used, 1.0 TiB / 1.0 TiB avail; 2.7 MiB/s rd, 3.2 MiB/s wr, 309 op/s 2024-04-23T10:29:22.911 INFO:tasks.workunit.client.0.smithi114.stdout: 1 177615 10.73 MB/sec warmup 91 sec latency 6210.461 ms 2024-04-23T10:29:23.911 INFO:tasks.workunit.client.0.smithi114.stdout: 1 177615 10.62 MB/sec warmup 92 sec latency 7210.511 ms 2024-04-23T10:29:24.242 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 23 10:29:23 smithi119 ceph-mon[32608]: pgmap v86: 97 pgs: 97 active+clean; 61 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.8 MiB/s rd, 3.9 MiB/s wr, 379 op/s 2024-04-23T10:29:24.254 INFO:journalctl@ceph.mon.a.smithi114.stdout:Apr 23 10:29:23 smithi114 ceph-mon[29255]: pgmap v86: 97 pgs: 97 active+clean; 61 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.8 MiB/s rd, 3.9 MiB/s wr, 379 op/s 2024-04-23T10:29:24.328 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 23 10:29:23 smithi204 ceph-mon[32649]: pgmap v86: 97 pgs: 97 active+clean; 61 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.8 MiB/s rd, 3.9 MiB/s wr, 379 op/s 2024-04-23T10:29:24.911 INFO:tasks.workunit.client.0.smithi114.stdout: 1 177615 10.50 MB/sec warmup 93 sec latency 8210.600 ms 2024-04-23T10:29:25.911 INFO:tasks.workunit.client.0.smithi114.stdout: 1 177615 10.39 MB/sec warmup 94 sec latency 9210.727 ms 2024-04-23T10:29:26.242 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 23 10:29:25 smithi119 ceph-mon[32608]: pgmap v87: 97 pgs: 97 active+clean; 61 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 1.5 MiB/s rd, 2.4 MiB/s wr, 201 op/s 2024-04-23T10:29:26.254 INFO:journalctl@ceph.mon.a.smithi114.stdout:Apr 23 10:29:25 smithi114 ceph-mon[29255]: pgmap v87: 97 pgs: 97 active+clean; 61 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 1.5 MiB/s rd, 2.4 MiB/s wr, 201 op/s 2024-04-23T10:29:26.328 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 23 10:29:25 smithi204 ceph-mon[32649]: pgmap v87: 97 pgs: 97 active+clean; 61 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 1.5 MiB/s rd, 2.4 MiB/s wr, 201 op/s 2024-04-23T10:29:26.911 INFO:tasks.workunit.client.0.smithi114.stdout: 1 177615 10.28 MB/sec warmup 95 sec latency 10210.833 ms 2024-04-23T10:29:27.911 INFO:tasks.workunit.client.0.smithi114.stdout: 1 177615 10.17 MB/sec warmup 96 sec latency 11210.870 ms 2024-04-23T10:29:28.242 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 23 10:29:27 smithi119 ceph-mon[32608]: pgmap v88: 97 pgs: 97 active+clean; 61 MiB data, 886 MiB used, 1.0 TiB / 1.0 TiB avail; 1.5 MiB/s rd, 3.2 MiB/s wr, 233 op/s 2024-04-23T10:29:28.254 INFO:journalctl@ceph.mon.a.smithi114.stdout:Apr 23 10:29:27 smithi114 ceph-mon[29255]: pgmap v88: 97 pgs: 97 active+clean; 61 MiB data, 886 MiB used, 1.0 TiB / 1.0 TiB avail; 1.5 MiB/s rd, 3.2 MiB/s wr, 233 op/s 2024-04-23T10:29:28.327 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 23 10:29:27 smithi204 ceph-mon[32649]: pgmap v88: 97 pgs: 97 active+clean; 61 MiB data, 886 MiB used, 1.0 TiB / 1.0 TiB avail; 1.5 MiB/s rd, 3.2 MiB/s wr, 233 op/s 2024-04-23T10:29:28.911 INFO:tasks.workunit.client.0.smithi114.stdout: 1 177615 10.07 MB/sec warmup 97 sec latency 12210.937 ms 2024-04-23T10:29:29.911 INFO:tasks.workunit.client.0.smithi114.stdout: 1 177615 9.97 MB/sec warmup 98 sec latency 13211.007 ms 2024-04-23T10:29:30.242 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 23 10:29:29 smithi119 ceph-mon[32608]: pgmap v89: 97 pgs: 97 active+clean; 61 MiB data, 886 MiB used, 1.0 TiB / 1.0 TiB avail; 166 KiB/s rd, 1.6 MiB/s wr, 102 op/s 2024-04-23T10:29:30.254 INFO:journalctl@ceph.mon.a.smithi114.stdout:Apr 23 10:29:29 smithi114 ceph-mon[29255]: pgmap v89: 97 pgs: 97 active+clean; 61 MiB data, 886 MiB used, 1.0 TiB / 1.0 TiB avail; 166 KiB/s rd, 1.6 MiB/s wr, 102 op/s 2024-04-23T10:29:30.327 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 23 10:29:29 smithi204 ceph-mon[32649]: pgmap v89: 97 pgs: 97 active+clean; 61 MiB data, 886 MiB used, 1.0 TiB / 1.0 TiB avail; 166 KiB/s rd, 1.6 MiB/s wr, 102 op/s 2024-04-23T10:29:30.911 INFO:tasks.workunit.client.0.smithi114.stdout: 1 177615 9.87 MB/sec warmup 99 sec latency 14211.075 ms 2024-04-23T10:29:31.911 INFO:tasks.workunit.client.0.smithi114.stdout: 1 177615 9.77 MB/sec warmup 100 sec latency 15211.171 ms 2024-04-23T10:29:32.242 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 23 10:29:31 smithi119 ceph-mon[32608]: pgmap v90: 97 pgs: 97 active+clean; 61 MiB data, 886 MiB used, 1.0 TiB / 1.0 TiB avail; 166 KiB/s rd, 1.6 MiB/s wr, 102 op/s 2024-04-23T10:29:32.254 INFO:journalctl@ceph.mon.a.smithi114.stdout:Apr 23 10:29:31 smithi114 ceph-mon[29255]: pgmap v90: 97 pgs: 97 active+clean; 61 MiB data, 886 MiB used, 1.0 TiB / 1.0 TiB avail; 166 KiB/s rd, 1.6 MiB/s wr, 102 op/s 2024-04-23T10:29:32.327 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 23 10:29:31 smithi204 ceph-mon[32649]: pgmap v90: 97 pgs: 97 active+clean; 61 MiB data, 886 MiB used, 1.0 TiB / 1.0 TiB avail; 166 KiB/s rd, 1.6 MiB/s wr, 102 op/s 2024-04-23T10:29:32.911 INFO:tasks.workunit.client.0.smithi114.stdout: 1 177615 9.67 MB/sec warmup 101 sec latency 16211.290 ms 2024-04-23T10:29:33.732 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:29:33.731+0000 7f281ab76640 1 -- 172.21.15.114:0/2105264437 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f2814103b40 msgr2=0x7f281410c020 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T10:29:33.732 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:29:33.731+0000 7f281ab76640 1 --2- 172.21.15.114:0/2105264437 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f2814103b40 0x7f281410c020 secure :-1 s=READY pgs=581 cs=0 l=1 rev1=1 crypto rx=0x7f2804009ca0 tx=0x7f280402a950 comp rx=0 tx=0).stop 2024-04-23T10:29:33.732 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:29:33.731+0000 7f281ab76640 1 -- 172.21.15.114:0/2105264437 shutdown_connections 2024-04-23T10:29:33.732 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:29:33.731+0000 7f281ab76640 1 --2- 172.21.15.114:0/2105264437 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f281410c560 0x7f281410e9b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:29:33.732 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:29:33.731+0000 7f281ab76640 1 --2- 172.21.15.114:0/2105264437 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f2814103b40 0x7f281410c020 unknown :-1 s=CLOSED pgs=581 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:29:33.733 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:29:33.731+0000 7f281ab76640 1 --2- 172.21.15.114:0/2105264437 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f2814103220 0x7f2814103600 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:29:33.733 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:29:33.731+0000 7f281ab76640 1 -- 172.21.15.114:0/2105264437 >> 172.21.15.114:0/2105264437 conn(0x7f2814077fd0 msgr2=0x7f28140783e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T10:29:33.733 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:29:33.731+0000 7f281ab76640 1 -- 172.21.15.114:0/2105264437 shutdown_connections 2024-04-23T10:29:33.733 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:29:33.731+0000 7f281ab76640 1 -- 172.21.15.114:0/2105264437 wait complete. 2024-04-23T10:29:33.733 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:29:33.731+0000 7f281ab76640 1 Processor -- start 2024-04-23T10:29:33.733 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:29:33.731+0000 7f281ab76640 1 -- start start 2024-04-23T10:29:33.733 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:29:33.731+0000 7f281ab76640 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f2814103220 0x7f28141986f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T10:29:33.733 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:29:33.731+0000 7f281ab76640 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f2814103b40 0x7f2814198c30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T10:29:33.733 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:29:33.731+0000 7f281ab76640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f281410c560 0x7f281419cfb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T10:29:33.733 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:29:33.731+0000 7f281ab76640 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x7f28141121b0 con 0x7f281410c560 2024-04-23T10:29:33.733 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:29:33.731+0000 7f281ab76640 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 -- 0x7f2814112030 con 0x7f2814103220 2024-04-23T10:29:33.733 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:29:33.731+0000 7f281ab76640 1 -- --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_getmap magic: 0 -- 0x7f2814112330 con 0x7f2814103b40 2024-04-23T10:29:33.734 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:29:33.732+0000 7f2819113640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f281410c560 0x7f281419cfb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T10:29:33.734 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:29:33.732+0000 7f2819113640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f281410c560 0x7f281419cfb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.114:54132/0 (socket says 172.21.15.114:54132) 2024-04-23T10:29:33.734 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:29:33.732+0000 7f2819113640 1 -- 172.21.15.114:0/3680614296 learned_addr learned my addr 172.21.15.114:0/3680614296 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-04-23T10:29:33.734 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:29:33.732+0000 7f2813fff640 1 --2- 172.21.15.114:0/3680614296 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f2814103b40 0x7f2814198c30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T10:29:33.734 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:29:33.732+0000 7f2818912640 1 --2- 172.21.15.114:0/3680614296 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f2814103220 0x7f28141986f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T10:29:33.734 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:29:33.732+0000 7f2819113640 1 -- 172.21.15.114:0/3680614296 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f2814103b40 msgr2=0x7f2814198c30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T10:29:33.734 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:29:33.732+0000 7f2819113640 1 --2- 172.21.15.114:0/3680614296 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f2814103b40 0x7f2814198c30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:29:33.734 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:29:33.732+0000 7f2819113640 1 -- 172.21.15.114:0/3680614296 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f2814103220 msgr2=0x7f28141986f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T10:29:33.734 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:29:33.732+0000 7f2819113640 1 --2- 172.21.15.114:0/3680614296 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f2814103220 0x7f28141986f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:29:33.734 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:29:33.732+0000 7f2819113640 1 -- 172.21.15.114:0/3680614296 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f281419d6e0 con 0x7f281410c560 2024-04-23T10:29:33.734 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:29:33.732+0000 7f2819113640 1 --2- 172.21.15.114:0/3680614296 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f281410c560 0x7f281419cfb0 secure :-1 s=READY pgs=582 cs=0 l=1 rev1=1 crypto rx=0x7f27fc00d750 tx=0x7f27fc00dc10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T10:29:33.734 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:29:33.732+0000 7f2811ffb640 1 -- 172.21.15.114:0/3680614296 <== mon.0 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f27fc017070 con 0x7f281410c560 2024-04-23T10:29:33.734 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:29:33.732+0000 7f2811ffb640 1 -- 172.21.15.114:0/3680614296 <== mon.0 v2:172.21.15.114:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f27fc0054d0 con 0x7f281410c560 2024-04-23T10:29:33.735 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:29:33.732+0000 7f281ab76640 1 -- 172.21.15.114:0/3680614296 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f281419d9d0 con 0x7f281410c560 2024-04-23T10:29:33.735 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:29:33.732+0000 7f2811ffb640 1 -- 172.21.15.114:0/3680614296 <== mon.0 v2:172.21.15.114:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f27fc004c60 con 0x7f281410c560 2024-04-23T10:29:33.735 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:29:33.732+0000 7f281ab76640 1 -- 172.21.15.114:0/3680614296 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f28141a5260 con 0x7f281410c560 2024-04-23T10:29:33.742 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:29:33.734+0000 7f2811ffb640 1 -- 172.21.15.114:0/3680614296 <== mon.0 v2:172.21.15.114:3300/0 4 ==== mgrmap(e 26) ==== 138778+0+0 (secure 0 0 0) 0x7f27fc009bd0 con 0x7f281410c560 2024-04-23T10:29:33.742 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:29:33.735+0000 7f281d96d740 1 Processor -- start 2024-04-23T10:29:33.742 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:29:33.735+0000 7f2811ffb640 1 --2- 172.21.15.114:0/3680614296 >> [v2:172.21.15.114:6800/1395693466,v1:172.21.15.114:6801/1395693466] conn(0x7f28000a3dc0 0x7f28000a6280 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T10:29:33.742 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:29:33.735+0000 7f281d96d740 1 -- start start 2024-04-23T10:29:33.742 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:29:33.735+0000 7f2813fff640 1 --2- 172.21.15.114:0/3680614296 >> [v2:172.21.15.114:6800/1395693466,v1:172.21.15.114:6801/1395693466] conn(0x7f28000a3dc0 0x7f28000a6280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T10:29:33.742 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:29:33.735+0000 7f2811ffb640 1 -- 172.21.15.114:0/3680614296 <== mon.0 v2:172.21.15.114:3300/0 5 ==== osd_map(92..92 src has 1..92) ==== 8585+0+0 (secure 0 0 0) 0x7f27fc0d0220 con 0x7f281410c560 2024-04-23T10:29:33.742 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:29:33.735+0000 7f281d96d740 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x557c39bff620 0x557c39bffa00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T10:29:33.743 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:29:33.735+0000 7f281d96d740 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x557c39bfff40 0x557c39c0a2c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T10:29:33.743 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:29:33.735+0000 7f281d96d740 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x557c39c0a800 0x557c39c0cbf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T10:29:33.743 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:29:33.735+0000 7f281d96d740 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x557c39b64970 con 0x557c39bfff40 2024-04-23T10:29:33.743 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:29:33.735+0000 7f281d96d740 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 -- 0x557c39b014a0 con 0x557c39bff620 2024-04-23T10:29:33.743 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:29:33.735+0000 7f281d96d740 1 -- --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_getmap magic: 0 -- 0x557c39b60930 con 0x557c39c0a800 2024-04-23T10:29:33.743 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:29:33.735+0000 7f2813fff640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x557c39bfff40 0x557c39c0a2c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T10:29:33.743 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:29:33.735+0000 7f2813fff640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x557c39bfff40 0x557c39c0a2c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.114:54138/0 (socket says 172.21.15.114:54138) 2024-04-23T10:29:33.743 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:29:33.735+0000 7f2813fff640 1 -- 172.21.15.114:0/3688804493 learned_addr learned my addr 172.21.15.114:0/3688804493 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-04-23T10:29:33.743 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:29:33.735+0000 7f2819113640 1 --2- 172.21.15.114:0/3688804493 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x557c39c0a800 0x557c39c0cbf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T10:29:33.743 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:29:33.735+0000 7f2818912640 1 --2- 172.21.15.114:0/3688804493 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x557c39bff620 0x557c39bffa00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T10:29:33.743 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:29:33.736+0000 7f2813fff640 1 --2- 172.21.15.114:0/3680614296 >> [v2:172.21.15.114:6800/1395693466,v1:172.21.15.114:6801/1395693466] conn(0x7f28000a3dc0 0x7f28000a6280 secure :-1 s=READY pgs=84 cs=0 l=1 rev1=1 crypto rx=0x7f2804009ca0 tx=0x7f28040093f0 comp rx=0 tx=0).ready entity=mgr.15291 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T10:29:33.743 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:29:33.736+0000 7f2813fff640 1 -- 172.21.15.114:0/3688804493 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x557c39c0a800 msgr2=0x557c39c0cbf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T10:29:33.743 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:29:33.736+0000 7f2813fff640 1 --2- 172.21.15.114:0/3688804493 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x557c39c0a800 0x557c39c0cbf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:29:33.743 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:29:33.736+0000 7f2813fff640 1 -- 172.21.15.114:0/3688804493 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x557c39bff620 msgr2=0x557c39bffa00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T10:29:33.743 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:29:33.736+0000 7f2813fff640 1 --2- 172.21.15.114:0/3688804493 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x557c39bff620 0x557c39bffa00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:29:33.744 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:29:33.736+0000 7f2813fff640 1 -- 172.21.15.114:0/3688804493 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x557c39877210 con 0x557c39bfff40 2024-04-23T10:29:33.744 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:29:33.736+0000 7f2819113640 1 --2- 172.21.15.114:0/3688804493 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x557c39c0a800 0x557c39c0cbf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-23T10:29:33.744 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:29:33.736+0000 7f2813fff640 1 --2- 172.21.15.114:0/3688804493 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x557c39bfff40 0x557c39c0a2c0 secure :-1 s=READY pgs=583 cs=0 l=1 rev1=1 crypto rx=0x7f2804035000 tx=0x7f2804049000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=dde45d7e7fde3629 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T10:29:33.744 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:29:33.736+0000 7f280affd640 1 -- 172.21.15.114:0/3688804493 <== mon.0 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2804034040 con 0x557c39bfff40 2024-04-23T10:29:33.744 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:29:33.736+0000 7f280affd640 1 -- 172.21.15.114:0/3688804493 <== mon.0 v2:172.21.15.114:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f280403d910 con 0x557c39bfff40 2024-04-23T10:29:33.744 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:29:33.736+0000 7f280affd640 1 -- 172.21.15.114:0/3688804493 <== mon.0 v2:172.21.15.114:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f280404a7d0 con 0x557c39bfff40 2024-04-23T10:29:33.744 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:29:33.737+0000 7f281d96d740 1 -- 172.21.15.114:0/3688804493 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x557c39bfff40 msgr2=0x557c39c0a2c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T10:29:33.744 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:29:33.737+0000 7f281d96d740 1 --2- 172.21.15.114:0/3688804493 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x557c39bfff40 0x557c39c0a2c0 secure :-1 s=READY pgs=583 cs=0 l=1 rev1=1 crypto rx=0x7f2804035000 tx=0x7f2804049000 comp rx=0 tx=0).stop 2024-04-23T10:29:33.744 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:29:33.737+0000 7f281d96d740 1 -- 172.21.15.114:0/3688804493 shutdown_connections 2024-04-23T10:29:33.744 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:29:33.737+0000 7f281d96d740 1 --2- 172.21.15.114:0/3688804493 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x557c39c0a800 0x557c39c0cbf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:29:33.744 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:29:33.737+0000 7f281d96d740 1 --2- 172.21.15.114:0/3688804493 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x557c39bfff40 0x557c39c0a2c0 unknown :-1 s=CLOSED pgs=583 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:29:33.744 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:29:33.737+0000 7f281d96d740 1 --2- 172.21.15.114:0/3688804493 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x557c39bff620 0x557c39bffa00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:29:33.744 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:29:33.737+0000 7f281d96d740 1 -- 172.21.15.114:0/3688804493 >> 172.21.15.114:0/3688804493 conn(0x557c39bfbb30 msgr2=0x557c39bfbf10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T10:29:33.744 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:29:33.737+0000 7f281d96d740 1 -- 172.21.15.114:0/3688804493 shutdown_connections 2024-04-23T10:29:33.745 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:29:33.737+0000 7f281d96d740 1 -- 172.21.15.114:0/3688804493 wait complete. 2024-04-23T10:29:33.745 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:29:33.737+0000 7f281d96d740 1 -- start start 2024-04-23T10:29:33.745 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:29:33.737+0000 7f281d96d740 1 Processor -- start 2024-04-23T10:29:33.745 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:29:33.737+0000 7f281d96d740 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x557c39bff620 0x557c39c0a0b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T10:29:33.745 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:29:33.737+0000 7f281d96d740 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x557c39bfff40 0x557c39c073a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T10:29:33.745 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:29:33.737+0000 7f281d96d740 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x557c39c0a800 0x557c39c078e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T10:29:33.745 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:29:33.737+0000 7f281d96d740 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x557c39c0d3d0 con 0x557c39bff620 2024-04-23T10:29:33.745 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:29:33.737+0000 7f281d96d740 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 -- 0x557c39aff0b0 con 0x557c39bfff40 2024-04-23T10:29:33.745 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:29:33.737+0000 7f281d96d740 1 -- --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_getmap magic: 0 -- 0x557c39c0d550 con 0x557c39c0a800 2024-04-23T10:29:33.745 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:29:33.738+0000 7f281d96d740 10 client.0 _ll_register_callbacks cb 0x557c39aecd30 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-23T10:29:33.745 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:29:33.738+0000 7f2818912640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x557c39bff620 0x557c39c0a0b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T10:29:33.745 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:29:33.738+0000 7f2818912640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x557c39bff620 0x557c39c0a0b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.114:54142/0 (socket says 172.21.15.114:54142) 2024-04-23T10:29:33.745 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:29:33.738+0000 7f2818912640 1 -- 172.21.15.114:0/2487889741 learned_addr learned my addr 172.21.15.114:0/2487889741 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-04-23T10:29:33.745 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:29:33.738+0000 7f2819113640 1 --2- 172.21.15.114:0/2487889741 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x557c39c0a800 0x557c39c078e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T10:29:33.745 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:29:33.738+0000 7f2813fff640 1 --2- 172.21.15.114:0/2487889741 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x557c39bfff40 0x557c39c073a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T10:29:33.746 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:29:33.738+0000 7f2818912640 1 -- 172.21.15.114:0/2487889741 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x557c39c0a800 msgr2=0x557c39c078e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T10:29:33.746 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:29:33.738+0000 7f2818912640 1 --2- 172.21.15.114:0/2487889741 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x557c39c0a800 0x557c39c078e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:29:33.746 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:29:33.738+0000 7f2818912640 1 -- 172.21.15.114:0/2487889741 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x557c39bfff40 msgr2=0x557c39c073a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T10:29:33.746 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:29:33.738+0000 7f2818912640 1 --2- 172.21.15.114:0/2487889741 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x557c39bfff40 0x557c39c073a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:29:33.746 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:29:33.738+0000 7f2818912640 1 -- 172.21.15.114:0/2487889741 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({osdmap=0}) -- 0x557c39943620 con 0x557c39bff620 2024-04-23T10:29:33.746 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:29:33.738+0000 7f2818912640 1 --2- 172.21.15.114:0/2487889741 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x557c39bff620 0x557c39c0a0b0 secure :-1 s=READY pgs=584 cs=0 l=1 rev1=1 crypto rx=0x7f280c00bf60 tx=0x7f280c00bf90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d730cfecae85cebb server_cookie=0 in_seq=0 out_seq=0 2024-04-23T10:29:33.746 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:29:33.738+0000 7f27f3fff640 10 client.0 ms_handle_connect on v2:172.21.15.114:3300/0 2024-04-23T10:29:33.746 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:29:33.738+0000 7f27f3fff640 1 -- 172.21.15.114:0/2487889741 <== mon.0 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f280c01f060 con 0x557c39bff620 2024-04-23T10:29:33.746 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:29:33.738+0000 7f27f3fff640 1 -- 172.21.15.114:0/2487889741 <== mon.0 v2:172.21.15.114:3300/0 2 ==== osd_map(92..92 src has 1..92) ==== 8585+0+0 (secure 0 0 0) 0x7f280c0108a0 con 0x557c39bff620 2024-04-23T10:29:33.746 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:29:33.738+0000 7f281d96d740 1 -- 172.21.15.114:0/2487889741 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f280403d910 con 0x557c39bff620 2024-04-23T10:29:33.746 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:29:33.738+0000 7f27f3fff640 1 -- 172.21.15.114:0/2487889741 <== mon.0 v2:172.21.15.114:3300/0 3 ==== mon_get_version_reply(handle=1 version=23) ==== 24+0+0 (secure 0 0 0) 0x7f280c010aa0 con 0x557c39bff620 2024-04-23T10:29:33.746 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:29:33.738+0000 7f281d96d740 10 client.15906 fetch_fsmap learned FSMap version 23 2024-04-23T10:29:33.746 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:29:33.738+0000 7f281d96d740 1 -- 172.21.15.114:0/2487889741 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({config=0+,fsmap=23,monmap=4+}) -- 0x557c39c192a0 con 0x557c39bff620 2024-04-23T10:29:33.746 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:29:33.739+0000 7f27f3fff640 1 -- 172.21.15.114:0/2487889741 <== mon.0 v2:172.21.15.114:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f280403d910 con 0x557c39bff620 2024-04-23T10:29:33.746 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:29:33.739+0000 7f27f3fff640 1 -- 172.21.15.114:0/2487889741 <== mon.0 v2:172.21.15.114:3300/0 5 ==== fsmap(e 23) ==== 8350+0+0 (secure 0 0 0) 0x7f280c019030 con 0x557c39bff620 2024-04-23T10:29:33.747 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:29:33.739+0000 7f281d96d740 10 client.15906 fetch_fsmap finished waiting for FSMap version 23 2024-04-23T10:29:33.747 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:29:33.739+0000 7f281d96d740 10 client.15906 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-04-23T10:29:33.747 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:29:33.739+0000 7f281d96d740 20 client.15906 populate_metadata read hostname 'smithi114' 2024-04-23T10:29:33.747 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:29:33.739+0000 7f281d96d740 1 --2- 172.21.15.114:0/2487889741 >> [v2:172.21.15.119:6838/937297340,v1:172.21.15.119:6839/937297340] conn(0x557c39c1a760 0x557c39c1cbb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T10:29:33.747 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:29:33.739+0000 7f281d96d740 4 client.15906 mds_command: new command op to 24545 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-23T10:29:33.747 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:29:33.739+0000 7f281d96d740 1 -- 172.21.15.114:0/2487889741 --> [v2:172.21.15.119:6838/937297340,v1:172.21.15.119:6839/937297340] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x557c39b72390 con 0x557c39c1a760 2024-04-23T10:29:33.747 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:29:33.739+0000 7f2813fff640 1 --2- 172.21.15.114:0/2487889741 >> [v2:172.21.15.119:6838/937297340,v1:172.21.15.119:6839/937297340] conn(0x557c39c1a760 0x557c39c1cbb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T10:29:33.747 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:29:33.739+0000 7f27f3fff640 10 client.15906 ms_handle_connect on v2:172.21.15.119:6838/937297340 2024-04-23T10:29:33.747 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:29:33.739+0000 7f2813fff640 1 --2- 172.21.15.114:0/2487889741 >> [v2:172.21.15.119:6838/937297340,v1:172.21.15.119:6839/937297340] conn(0x557c39c1a760 0x557c39c1cbb0 crc :-1 s=READY pgs=74 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=9611ccb77ce97905 server_cookie=71ad1667ee816ee9 in_seq=0 out_seq=0 2024-04-23T10:29:33.747 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:29:33.740+0000 7f27f3fff640 1 -- 172.21.15.114:0/2487889741 <== mds.0 v2:172.21.15.119:6838/937297340 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x557c39b72390 con 0x557c39c1a760 2024-04-23T10:29:33.747 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:29:33.740+0000 7f27f3fff640 10 client.15906 handle_command_reply: tid=0 2024-04-23T10:29:33.747 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:29:33.740+0000 7f281d96d740 1 client.15906 shutdown 2024-04-23T10:29:33.747 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:29:33.740+0000 7f281d96d740 1 -- 172.21.15.114:0/2487889741 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x557c39bff620 msgr2=0x557c39c0a0b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T10:29:33.747 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:29:33.740+0000 7f281d96d740 1 --2- 172.21.15.114:0/2487889741 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x557c39bff620 0x557c39c0a0b0 secure :-1 s=READY pgs=584 cs=0 l=1 rev1=1 crypto rx=0x7f280c00bf60 tx=0x7f280c00bf90 comp rx=0 tx=0).stop 2024-04-23T10:29:33.748 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:29:33.740+0000 7f281d96d740 1 -- 172.21.15.114:0/2487889741 shutdown_connections 2024-04-23T10:29:33.748 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:29:33.740+0000 7f281d96d740 1 --2- 172.21.15.114:0/2487889741 >> [v2:172.21.15.119:6838/937297340,v1:172.21.15.119:6839/937297340] conn(0x557c39c1a760 0x557c39c1cbb0 crc :-1 s=READY pgs=74 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:29:33.748 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:29:33.740+0000 7f281d96d740 1 --2- 172.21.15.114:0/2487889741 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x557c39c0a800 0x557c39c078e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:29:33.748 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:29:33.740+0000 7f281d96d740 1 --2- 172.21.15.114:0/2487889741 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x557c39bfff40 0x557c39c073a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:29:33.748 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:29:33.740+0000 7f27f3fff640 0 client.15906 ms_handle_reset on v2:172.21.15.119:6838/937297340 2024-04-23T10:29:33.748 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:29:33.740+0000 7f281d96d740 1 --2- 172.21.15.114:0/2487889741 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x557c39bff620 0x557c39c0a0b0 unknown :-1 s=CLOSED pgs=584 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:29:33.748 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:29:33.740+0000 7f281d96d740 1 -- 172.21.15.114:0/2487889741 >> 172.21.15.114:0/2487889741 conn(0x557c39bfbb30 msgr2=0x557c39c018b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T10:29:33.748 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:29:33.740+0000 7f281d96d740 1 -- 172.21.15.114:0/2487889741 shutdown_connections 2024-04-23T10:29:33.748 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:29:33.740+0000 7f281d96d740 1 -- 172.21.15.114:0/2487889741 wait complete. 2024-04-23T10:29:33.748 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:29:33.741+0000 7f281d96d740 20 client.15906 trim_cache size 0 max 16384 2024-04-23T10:29:33.759 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:29:33.755+0000 7f281d96d740 1 Processor -- start 2024-04-23T10:29:33.759 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:29:33.755+0000 7f281d96d740 1 -- start start 2024-04-23T10:29:33.759 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:29:33.755+0000 7f281d96d740 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x557c39bff620 0x557c39c16290 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T10:29:33.760 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:29:33.755+0000 7f281d96d740 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x557c39bfff40 0x557c39c02af0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T10:29:33.760 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:29:33.755+0000 7f281d96d740 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x557c39c0a800 0x557c39c09be0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T10:29:33.760 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:29:33.755+0000 7f281d96d740 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x7f28141b05b0 con 0x557c39bfff40 2024-04-23T10:29:33.760 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:29:33.755+0000 7f281d96d740 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 -- 0x7f28141ade50 con 0x557c39c0a800 2024-04-23T10:29:33.760 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:29:33.755+0000 7f281d96d740 1 -- --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_getmap magic: 0 -- 0x7f28141abca0 con 0x557c39bff620 2024-04-23T10:29:33.760 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:29:33.755+0000 7f2813fff640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x557c39bfff40 0x557c39c02af0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T10:29:33.760 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:29:33.755+0000 7f2813fff640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x557c39bfff40 0x557c39c02af0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.114:54146/0 (socket says 172.21.15.114:54146) 2024-04-23T10:29:33.760 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:29:33.755+0000 7f2813fff640 1 -- 172.21.15.114:0/1243084072 learned_addr learned my addr 172.21.15.114:0/1243084072 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-04-23T10:29:33.760 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:29:33.755+0000 7f2819113640 1 --2- 172.21.15.114:0/1243084072 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x557c39c0a800 0x557c39c09be0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T10:29:33.760 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:29:33.755+0000 7f2818912640 1 --2- 172.21.15.114:0/1243084072 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x557c39bff620 0x557c39c16290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T10:29:33.760 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:29:33.756+0000 7f2813fff640 1 -- 172.21.15.114:0/1243084072 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x557c39bff620 msgr2=0x557c39c16290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T10:29:33.760 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:29:33.756+0000 7f2813fff640 1 --2- 172.21.15.114:0/1243084072 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x557c39bff620 0x557c39c16290 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:29:33.760 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:29:33.756+0000 7f2813fff640 1 -- 172.21.15.114:0/1243084072 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x557c39c0a800 msgr2=0x557c39c09be0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T10:29:33.760 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:29:33.756+0000 7f2813fff640 1 --2- 172.21.15.114:0/1243084072 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x557c39c0a800 0x557c39c09be0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:29:33.760 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:29:33.756+0000 7f2813fff640 1 -- 172.21.15.114:0/1243084072 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x557c39c062e0 con 0x557c39bfff40 2024-04-23T10:29:33.761 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:29:33.756+0000 7f2819113640 1 --2- 172.21.15.114:0/1243084072 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x557c39c0a800 0x557c39c09be0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-23T10:29:33.761 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:29:33.756+0000 7f2813fff640 1 --2- 172.21.15.114:0/1243084072 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x557c39bfff40 0x557c39c02af0 secure :-1 s=READY pgs=585 cs=0 l=1 rev1=1 crypto rx=0x7f280402f8e0 tx=0x7f280403c050 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e10331a0ad5532d6 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T10:29:33.761 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:29:33.756+0000 7f280a7fc640 1 -- 172.21.15.114:0/1243084072 <== mon.0 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f280405d070 con 0x557c39bfff40 2024-04-23T10:29:33.761 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:29:33.756+0000 7f280a7fc640 1 -- 172.21.15.114:0/1243084072 <== mon.0 v2:172.21.15.114:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f2804057030 con 0x557c39bfff40 2024-04-23T10:29:33.761 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:29:33.756+0000 7f280a7fc640 1 -- 172.21.15.114:0/1243084072 <== mon.0 v2:172.21.15.114:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2804034040 con 0x557c39bfff40 2024-04-23T10:29:33.761 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:29:33.756+0000 7f281d96d740 1 -- 172.21.15.114:0/1243084072 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x557c39bfff40 msgr2=0x557c39c02af0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T10:29:33.761 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:29:33.756+0000 7f281d96d740 1 --2- 172.21.15.114:0/1243084072 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x557c39bfff40 0x557c39c02af0 secure :-1 s=READY pgs=585 cs=0 l=1 rev1=1 crypto rx=0x7f280402f8e0 tx=0x7f280403c050 comp rx=0 tx=0).stop 2024-04-23T10:29:33.761 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:29:33.756+0000 7f281d96d740 1 -- 172.21.15.114:0/1243084072 shutdown_connections 2024-04-23T10:29:33.761 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:29:33.756+0000 7f281d96d740 1 --2- 172.21.15.114:0/1243084072 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x557c39c0a800 0x557c39c09be0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:29:33.761 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:29:33.756+0000 7f281d96d740 1 --2- 172.21.15.114:0/1243084072 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x557c39bfff40 0x557c39c02af0 unknown :-1 s=CLOSED pgs=585 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:29:33.761 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:29:33.756+0000 7f281d96d740 1 --2- 172.21.15.114:0/1243084072 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x557c39bff620 0x557c39c16290 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:29:33.761 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:29:33.756+0000 7f281d96d740 1 -- 172.21.15.114:0/1243084072 >> 172.21.15.114:0/1243084072 conn(0x557c39bfbb30 msgr2=0x557c39c06c30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T10:29:33.761 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:29:33.757+0000 7f281d96d740 1 -- 172.21.15.114:0/1243084072 shutdown_connections 2024-04-23T10:29:33.761 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:29:33.757+0000 7f281d96d740 1 -- 172.21.15.114:0/1243084072 wait complete. 2024-04-23T10:29:33.762 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:29:33.757+0000 7f281d96d740 1 -- start start 2024-04-23T10:29:33.762 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:29:33.757+0000 7f281d96d740 1 Processor -- start 2024-04-23T10:29:33.762 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:29:33.757+0000 7f281d96d740 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x557c39bff620 0x557c39c12a40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T10:29:33.762 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:29:33.757+0000 7f281d96d740 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x557c39bfff40 0x557c39c0dde0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T10:29:33.762 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:29:33.757+0000 7f281d96d740 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x557c39c0a800 0x557c39c0e320 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T10:29:33.762 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:29:33.757+0000 7f281d96d740 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x7f28141ac310 con 0x557c39bfff40 2024-04-23T10:29:33.762 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:29:33.757+0000 7f281d96d740 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 -- 0x7f28141ad780 con 0x557c39bff620 2024-04-23T10:29:33.762 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:29:33.757+0000 7f281d96d740 1 -- --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_getmap magic: 0 -- 0x7f28141ac9e0 con 0x557c39c0a800 2024-04-23T10:29:33.762 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:29:33.757+0000 7f281d96d740 10 client.0 _ll_register_callbacks cb 0x7f280c00bb70 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-23T10:29:33.762 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:29:33.757+0000 7f2813fff640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x557c39bfff40 0x557c39c0dde0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T10:29:33.762 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:29:33.757+0000 7f2813fff640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x557c39bfff40 0x557c39c0dde0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.114:54154/0 (socket says 172.21.15.114:54154) 2024-04-23T10:29:33.762 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:29:33.757+0000 7f2813fff640 1 -- 172.21.15.114:0/2536477547 learned_addr learned my addr 172.21.15.114:0/2536477547 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-04-23T10:29:33.762 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:29:33.757+0000 7f2818912640 1 --2- 172.21.15.114:0/2536477547 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x557c39bff620 0x557c39c12a40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T10:29:33.763 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:29:33.757+0000 7f2819113640 1 --2- 172.21.15.114:0/2536477547 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x557c39c0a800 0x557c39c0e320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T10:29:33.763 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:29:33.758+0000 7f2813fff640 1 -- 172.21.15.114:0/2536477547 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x557c39c0a800 msgr2=0x557c39c0e320 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T10:29:33.763 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:29:33.758+0000 7f2813fff640 1 --2- 172.21.15.114:0/2536477547 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x557c39c0a800 0x557c39c0e320 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:29:33.763 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:29:33.758+0000 7f2813fff640 1 -- 172.21.15.114:0/2536477547 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x557c39bff620 msgr2=0x557c39c12a40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T10:29:33.763 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:29:33.758+0000 7f2813fff640 1 --2- 172.21.15.114:0/2536477547 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x557c39bff620 0x557c39c12a40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:29:33.763 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:29:33.758+0000 7f2813fff640 1 -- 172.21.15.114:0/2536477547 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({osdmap=0}) -- 0x557c39c0e9e0 con 0x557c39bfff40 2024-04-23T10:29:33.763 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:29:33.758+0000 7f2819113640 1 --2- 172.21.15.114:0/2536477547 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x557c39c0a800 0x557c39c0e320 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-23T10:29:33.763 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:29:33.758+0000 7f2813fff640 1 --2- 172.21.15.114:0/2536477547 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x557c39bfff40 0x557c39c0dde0 secure :-1 s=READY pgs=586 cs=0 l=1 rev1=1 crypto rx=0x7f2804009bf0 tx=0x7f2804005f60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=85eee54776643218 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T10:29:33.763 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:29:33.758+0000 7f27f3fff640 10 client.0 ms_handle_connect on v2:172.21.15.114:3300/0 2024-04-23T10:29:33.763 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:29:33.758+0000 7f27f3fff640 1 -- 172.21.15.114:0/2536477547 <== mon.0 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f280405d030 con 0x557c39bfff40 2024-04-23T10:29:33.763 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:29:33.758+0000 7f27f3fff640 1 -- 172.21.15.114:0/2536477547 <== mon.0 v2:172.21.15.114:3300/0 2 ==== osd_map(92..92 src has 1..92) ==== 8585+0+0 (secure 0 0 0) 0x7f280403d930 con 0x557c39bfff40 2024-04-23T10:29:33.763 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:29:33.758+0000 7f281d96d740 1 -- 172.21.15.114:0/2536477547 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f2804057030 con 0x557c39bfff40 2024-04-23T10:29:33.763 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:29:33.758+0000 7f27f3fff640 1 -- 172.21.15.114:0/2536477547 <== mon.0 v2:172.21.15.114:3300/0 3 ==== mon_get_version_reply(handle=1 version=23) ==== 24+0+0 (secure 0 0 0) 0x7f2804034040 con 0x557c39bfff40 2024-04-23T10:29:33.763 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:29:33.758+0000 7f281d96d740 10 client.15912 fetch_fsmap learned FSMap version 23 2024-04-23T10:29:33.763 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:29:33.758+0000 7f281d96d740 1 -- 172.21.15.114:0/2536477547 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({config=0+,fsmap=23,monmap=4+}) -- 0x557c39c80c10 con 0x557c39bfff40 2024-04-23T10:29:33.764 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:29:33.758+0000 7f27f3fff640 1 -- 172.21.15.114:0/2536477547 <== mon.0 v2:172.21.15.114:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f2804057030 con 0x557c39bfff40 2024-04-23T10:29:33.764 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:29:33.758+0000 7f27f3fff640 1 -- 172.21.15.114:0/2536477547 <== mon.0 v2:172.21.15.114:3300/0 5 ==== fsmap(e 23) ==== 8350+0+0 (secure 0 0 0) 0x7f28040644c0 con 0x557c39bfff40 2024-04-23T10:29:33.764 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:29:33.758+0000 7f281d96d740 10 client.15912 fetch_fsmap finished waiting for FSMap version 23 2024-04-23T10:29:33.764 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:29:33.758+0000 7f281d96d740 10 client.15912 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-04-23T10:29:33.764 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:29:33.758+0000 7f281d96d740 20 client.15912 populate_metadata read hostname 'smithi114' 2024-04-23T10:29:33.764 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:29:33.758+0000 7f281d96d740 1 --2- 172.21.15.114:0/2536477547 >> [v2:172.21.15.119:6838/937297340,v1:172.21.15.119:6839/937297340] conn(0x557c39c80ee0 0x557c39c83300 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T10:29:33.764 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:29:33.758+0000 7f281d96d740 4 client.15912 mds_command: new command op to 24545 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-04-23T10:29:33.764 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:29:33.758+0000 7f281d96d740 1 -- 172.21.15.114:0/2536477547 --> [v2:172.21.15.119:6838/937297340,v1:172.21.15.119:6839/937297340] -- command(tid 0: {"prefix": "flush journal"}) -- 0x557c39b6c200 con 0x557c39c80ee0 2024-04-23T10:29:33.764 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:29:33.759+0000 7f2818912640 1 --2- 172.21.15.114:0/2536477547 >> [v2:172.21.15.119:6838/937297340,v1:172.21.15.119:6839/937297340] conn(0x557c39c80ee0 0x557c39c83300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T10:29:33.764 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:29:33.759+0000 7f2818912640 1 --2- 172.21.15.114:0/2536477547 >> [v2:172.21.15.119:6838/937297340,v1:172.21.15.119:6839/937297340] conn(0x557c39c80ee0 0x557c39c83300 crc :-1 s=READY pgs=75 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=db4daded24784098 server_cookie=f2cdfe07188cce37 in_seq=0 out_seq=0 2024-04-23T10:29:33.764 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:29:33.759+0000 7f27f3fff640 10 client.15912 ms_handle_connect on v2:172.21.15.119:6838/937297340 2024-04-23T10:29:33.788 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:29:33.789+0000 7f27f3fff640 1 -- 172.21.15.114:0/2536477547 <== mds.0 v2:172.21.15.119:6838/937297340 1 ==== command_reply(tid 0: 0 ) ==== 8+0+44 (crc 0 0 0) 0x557c39b6c200 con 0x557c39c80ee0 2024-04-23T10:29:33.788 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:29:33.789+0000 7f27f3fff640 10 client.15912 handle_command_reply: tid=0 2024-04-23T10:29:33.788 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:29:33.789+0000 7f281d96d740 1 client.15912 shutdown 2024-04-23T10:29:33.788 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:29:33.789+0000 7f281d96d740 1 -- 172.21.15.114:0/2536477547 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x557c39bfff40 msgr2=0x557c39c0dde0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T10:29:33.788 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:29:33.789+0000 7f281d96d740 1 --2- 172.21.15.114:0/2536477547 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x557c39bfff40 0x557c39c0dde0 secure :-1 s=READY pgs=586 cs=0 l=1 rev1=1 crypto rx=0x7f2804009bf0 tx=0x7f2804005f60 comp rx=0 tx=0).stop 2024-04-23T10:29:33.788 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:29:33.789+0000 7f281d96d740 1 -- 172.21.15.114:0/2536477547 shutdown_connections 2024-04-23T10:29:33.789 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:29:33.789+0000 7f281d96d740 1 --2- 172.21.15.114:0/2536477547 >> [v2:172.21.15.119:6838/937297340,v1:172.21.15.119:6839/937297340] conn(0x557c39c80ee0 0x557c39c83300 crc :-1 s=READY pgs=75 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:29:33.789 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:29:33.789+0000 7f281d96d740 1 --2- 172.21.15.114:0/2536477547 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x557c39c0a800 0x557c39c0e320 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:29:33.789 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:29:33.789+0000 7f281d96d740 1 --2- 172.21.15.114:0/2536477547 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x557c39bfff40 0x557c39c0dde0 unknown :-1 s=CLOSED pgs=586 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:29:33.789 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:29:33.789+0000 7f281d96d740 1 --2- 172.21.15.114:0/2536477547 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x557c39bff620 0x557c39c12a40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:29:33.789 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:29:33.789+0000 7f27f3fff640 0 client.15912 ms_handle_reset on v2:172.21.15.119:6838/937297340 2024-04-23T10:29:33.789 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:29:33.789+0000 7f281d96d740 1 -- 172.21.15.114:0/2536477547 >> 172.21.15.114:0/2536477547 conn(0x557c39bfbb30 msgr2=0x557c39bfa420 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T10:29:33.789 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:29:33.789+0000 7f281d96d740 1 -- 172.21.15.114:0/2536477547 shutdown_connections 2024-04-23T10:29:33.789 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:29:33.789+0000 7f281d96d740 1 -- 172.21.15.114:0/2536477547 wait complete. 2024-04-23T10:29:33.789 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:29:33.789+0000 7f281d96d740 20 client.15912 trim_cache size 0 max 16384 2024-04-23T10:29:33.789 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stdout:{ 2024-04-23T10:29:33.789 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stdout: "message": "", 2024-04-23T10:29:33.789 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stdout: "return_code": 0 2024-04-23T10:29:33.790 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stdout:} 2024-04-23T10:29:33.790 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:29:33.790+0000 7f281ab76640 1 -- 172.21.15.114:0/3680614296 >> [v2:172.21.15.114:6800/1395693466,v1:172.21.15.114:6801/1395693466] conn(0x7f28000a3dc0 msgr2=0x7f28000a6280 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T10:29:33.791 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:29:33.790+0000 7f281ab76640 1 --2- 172.21.15.114:0/3680614296 >> [v2:172.21.15.114:6800/1395693466,v1:172.21.15.114:6801/1395693466] conn(0x7f28000a3dc0 0x7f28000a6280 secure :-1 s=READY pgs=84 cs=0 l=1 rev1=1 crypto rx=0x7f2804009ca0 tx=0x7f28040093f0 comp rx=0 tx=0).stop 2024-04-23T10:29:33.791 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:29:33.790+0000 7f281ab76640 1 -- 172.21.15.114:0/3680614296 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f281410c560 msgr2=0x7f281419cfb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T10:29:33.791 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:29:33.790+0000 7f281ab76640 1 --2- 172.21.15.114:0/3680614296 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f281410c560 0x7f281419cfb0 secure :-1 s=READY pgs=582 cs=0 l=1 rev1=1 crypto rx=0x7f27fc00d750 tx=0x7f27fc00dc10 comp rx=0 tx=0).stop 2024-04-23T10:29:33.791 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:29:33.790+0000 7f281ab76640 1 -- 172.21.15.114:0/3680614296 shutdown_connections 2024-04-23T10:29:33.791 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:29:33.790+0000 7f281ab76640 1 --2- 172.21.15.114:0/3680614296 >> [v2:172.21.15.114:6800/1395693466,v1:172.21.15.114:6801/1395693466] conn(0x7f28000a3dc0 0x7f28000a6280 unknown :-1 s=CLOSED pgs=84 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:29:33.791 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:29:33.790+0000 7f281ab76640 1 --2- 172.21.15.114:0/3680614296 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f281410c560 0x7f281419cfb0 unknown :-1 s=CLOSED pgs=582 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:29:33.791 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:29:33.790+0000 7f281ab76640 1 --2- 172.21.15.114:0/3680614296 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f2814103b40 0x7f2814198c30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:29:33.791 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:29:33.790+0000 7f281ab76640 1 --2- 172.21.15.114:0/3680614296 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f2814103220 0x7f28141986f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:29:33.791 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:29:33.790+0000 7f281ab76640 1 -- 172.21.15.114:0/3680614296 >> 172.21.15.114:0/3680614296 conn(0x7f2814077fd0 msgr2=0x7f28140fdf20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T10:29:33.791 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:29:33.790+0000 7f281ab76640 1 -- 172.21.15.114:0/3680614296 shutdown_connections 2024-04-23T10:29:33.791 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:29:33.790+0000 7f281ab76640 1 -- 172.21.15.114:0/3680614296 wait complete. 2024-04-23T10:29:33.911 INFO:tasks.workunit.client.0.smithi114.stdout: 1 177615 9.58 MB/sec warmup 102 sec latency 17211.418 ms 2024-04-23T10:29:34.241 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 23 10:29:33 smithi119 ceph-mon[32608]: pgmap v91: 97 pgs: 97 active+clean; 69 MiB data, 910 MiB used, 1.0 TiB / 1.0 TiB avail; 191 KiB/s rd, 2.3 MiB/s wr, 124 op/s 2024-04-23T10:29:34.254 INFO:journalctl@ceph.mon.a.smithi114.stdout:Apr 23 10:29:33 smithi114 ceph-mon[29255]: pgmap v91: 97 pgs: 97 active+clean; 69 MiB data, 910 MiB used, 1.0 TiB / 1.0 TiB avail; 191 KiB/s rd, 2.3 MiB/s wr, 124 op/s 2024-04-23T10:29:34.327 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 23 10:29:33 smithi204 ceph-mon[32649]: pgmap v91: 97 pgs: 97 active+clean; 69 MiB data, 910 MiB used, 1.0 TiB / 1.0 TiB avail; 191 KiB/s rd, 2.3 MiB/s wr, 124 op/s 2024-04-23T10:29:34.912 INFO:tasks.workunit.client.0.smithi114.stdout: 1 177615 9.48 MB/sec warmup 103 sec latency 18211.510 ms 2024-04-23T10:29:35.912 INFO:tasks.workunit.client.0.smithi114.stdout: 1 177615 9.39 MB/sec warmup 104 sec latency 19211.638 ms 2024-04-23T10:29:36.242 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 23 10:29:35 smithi119 ceph-mon[32608]: pgmap v92: 97 pgs: 97 active+clean; 69 MiB data, 910 MiB used, 1.0 TiB / 1.0 TiB avail; 78 KiB/s rd, 1.5 MiB/s wr, 54 op/s 2024-04-23T10:29:36.254 INFO:journalctl@ceph.mon.a.smithi114.stdout:Apr 23 10:29:35 smithi114 ceph-mon[29255]: pgmap v92: 97 pgs: 97 active+clean; 69 MiB data, 910 MiB used, 1.0 TiB / 1.0 TiB avail; 78 KiB/s rd, 1.5 MiB/s wr, 54 op/s 2024-04-23T10:29:36.328 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 23 10:29:35 smithi204 ceph-mon[32649]: pgmap v92: 97 pgs: 97 active+clean; 69 MiB data, 910 MiB used, 1.0 TiB / 1.0 TiB avail; 78 KiB/s rd, 1.5 MiB/s wr, 54 op/s 2024-04-23T10:29:36.912 INFO:tasks.workunit.client.0.smithi114.stdout: 1 177615 9.30 MB/sec warmup 105 sec latency 20211.699 ms 2024-04-23T10:29:37.912 INFO:tasks.workunit.client.0.smithi114.stdout: 1 177615 9.21 MB/sec warmup 106 sec latency 21211.757 ms 2024-04-23T10:29:38.176 DEBUG:teuthology.orchestra.run.smithi114:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-23T10:29:38.201 INFO:journalctl@ceph.mon.a.smithi114.stdout:Apr 23 10:29:37 smithi114 ceph-mon[29255]: pgmap v93: 97 pgs: 97 active+clean; 65 MiB data, 899 MiB used, 1.0 TiB / 1.0 TiB avail; 130 KiB/s rd, 2.1 MiB/s wr, 89 op/s 2024-04-23T10:29:38.242 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 23 10:29:37 smithi119 ceph-mon[32608]: pgmap v93: 97 pgs: 97 active+clean; 65 MiB data, 899 MiB used, 1.0 TiB / 1.0 TiB avail; 130 KiB/s rd, 2.1 MiB/s wr, 89 op/s 2024-04-23T10:29:38.286 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:29:38.286+0000 7fb0ecfcc640 1 -- 172.21.15.114:0/1461625711 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fb0e8106160 msgr2=0x7fb0e810dd30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T10:29:38.286 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:29:38.286+0000 7fb0ecfcc640 1 --2- 172.21.15.114:0/1461625711 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fb0e8106160 0x7fb0e810dd30 secure :-1 s=READY pgs=587 cs=0 l=1 rev1=1 crypto rx=0x7fb0e0009ca0 tx=0x7fb0e002a8c0 comp rx=0 tx=0).stop 2024-04-23T10:29:38.286 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:29:38.286+0000 7fb0ecfcc640 1 -- 172.21.15.114:0/1461625711 shutdown_connections 2024-04-23T10:29:38.286 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:29:38.286+0000 7fb0ecfcc640 1 --2- 172.21.15.114:0/1461625711 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fb0e8106160 0x7fb0e810dd30 unknown :-1 s=CLOSED pgs=587 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:29:38.286 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:29:38.286+0000 7fb0ecfcc640 1 --2- 172.21.15.114:0/1461625711 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fb0e8101a40 0x7fb0e8105a80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:29:38.287 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:29:38.286+0000 7fb0ecfcc640 1 --2- 172.21.15.114:0/1461625711 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fb0e8101090 0x7fb0e8101470 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:29:38.287 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:29:38.286+0000 7fb0ecfcc640 1 -- 172.21.15.114:0/1461625711 >> 172.21.15.114:0/1461625711 conn(0x7fb0e80771d0 msgr2=0x7fb0e80775e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T10:29:38.287 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:29:38.286+0000 7fb0ecfcc640 1 -- 172.21.15.114:0/1461625711 shutdown_connections 2024-04-23T10:29:38.287 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:29:38.286+0000 7fb0ecfcc640 1 -- 172.21.15.114:0/1461625711 wait complete. 2024-04-23T10:29:38.289 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:29:38.287+0000 7fb0ecfcc640 1 Processor -- start 2024-04-23T10:29:38.289 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:29:38.287+0000 7fb0ecfcc640 1 -- start start 2024-04-23T10:29:38.289 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:29:38.287+0000 7fb0ecfcc640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fb0e8101090 0x7fb0e8196560 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T10:29:38.289 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:29:38.287+0000 7fb0ecfcc640 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fb0e8106160 0x7fb0e8196aa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T10:29:38.289 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:29:38.287+0000 7fb0ecfcc640 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fb0e819add0 0x7fb0e819b280 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T10:29:38.289 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:29:38.287+0000 7fb0ecfcc640 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x7fb0e810ffc0 con 0x7fb0e8101090 2024-04-23T10:29:38.289 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:29:38.287+0000 7fb0ecfcc640 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 -- 0x7fb0e810fe40 con 0x7fb0e819add0 2024-04-23T10:29:38.290 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:29:38.287+0000 7fb0ecfcc640 1 -- --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_getmap magic: 0 -- 0x7fb0e8110140 con 0x7fb0e8106160 2024-04-23T10:29:38.290 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:29:38.287+0000 7fb0e659c640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fb0e8101090 0x7fb0e8196560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T10:29:38.290 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:29:38.287+0000 7fb0e659c640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fb0e8101090 0x7fb0e8196560 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.114:48986/0 (socket says 172.21.15.114:48986) 2024-04-23T10:29:38.290 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:29:38.287+0000 7fb0e659c640 1 -- 172.21.15.114:0/1621495308 learned_addr learned my addr 172.21.15.114:0/1621495308 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-04-23T10:29:38.290 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:29:38.287+0000 7fb0e6d9d640 1 --2- 172.21.15.114:0/1621495308 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fb0e819add0 0x7fb0e819b280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T10:29:38.290 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:29:38.287+0000 7fb0e5d9b640 1 --2- 172.21.15.114:0/1621495308 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fb0e8106160 0x7fb0e8196aa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T10:29:38.290 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:29:38.287+0000 7fb0e659c640 1 -- 172.21.15.114:0/1621495308 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fb0e8106160 msgr2=0x7fb0e8196aa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T10:29:38.290 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:29:38.287+0000 7fb0e659c640 1 --2- 172.21.15.114:0/1621495308 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fb0e8106160 0x7fb0e8196aa0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:29:38.290 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:29:38.287+0000 7fb0e659c640 1 -- 172.21.15.114:0/1621495308 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fb0e819add0 msgr2=0x7fb0e819b280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T10:29:38.290 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:29:38.287+0000 7fb0e659c640 1 --2- 172.21.15.114:0/1621495308 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fb0e819add0 0x7fb0e819b280 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:29:38.290 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:29:38.287+0000 7fb0e659c640 1 -- 172.21.15.114:0/1621495308 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb0e819b8c0 con 0x7fb0e8101090 2024-04-23T10:29:38.290 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:29:38.287+0000 7fb0e6d9d640 1 --2- 172.21.15.114:0/1621495308 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fb0e819add0 0x7fb0e819b280 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-23T10:29:38.290 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:29:38.288+0000 7fb0e659c640 1 --2- 172.21.15.114:0/1621495308 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fb0e8101090 0x7fb0e8196560 secure :-1 s=READY pgs=588 cs=0 l=1 rev1=1 crypto rx=0x7fb0dc00d770 tx=0x7fb0dc00dc30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T10:29:38.290 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:29:38.288+0000 7fb0d77fe640 1 -- 172.21.15.114:0/1621495308 <== mon.0 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb0dc017070 con 0x7fb0e8101090 2024-04-23T10:29:38.291 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:29:38.288+0000 7fb0d77fe640 1 -- 172.21.15.114:0/1621495308 <== mon.0 v2:172.21.15.114:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fb0dc0054d0 con 0x7fb0e8101090 2024-04-23T10:29:38.291 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:29:38.288+0000 7fb0ecfcc640 1 -- 172.21.15.114:0/1621495308 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fb0e819bbb0 con 0x7fb0e8101090 2024-04-23T10:29:38.291 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:29:38.288+0000 7fb0d77fe640 1 -- 172.21.15.114:0/1621495308 <== mon.0 v2:172.21.15.114:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb0dc004c30 con 0x7fb0e8101090 2024-04-23T10:29:38.291 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:29:38.288+0000 7fb0ecfcc640 1 -- 172.21.15.114:0/1621495308 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fb0e81a3350 con 0x7fb0e8101090 2024-04-23T10:29:38.291 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:29:38.289+0000 7fb0ecfcc640 1 -- 172.21.15.114:0/1621495308 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fb0e80691c0 con 0x7fb0e8101090 2024-04-23T10:29:38.292 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:29:38.290+0000 7fb0d77fe640 1 -- 172.21.15.114:0/1621495308 <== mon.0 v2:172.21.15.114:3300/0 4 ==== mgrmap(e 26) ==== 138778+0+0 (secure 0 0 0) 0x7fb0dc004dd0 con 0x7fb0e8101090 2024-04-23T10:29:38.292 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:29:38.290+0000 7fb0d77fe640 1 --2- 172.21.15.114:0/1621495308 >> [v2:172.21.15.114:6800/1395693466,v1:172.21.15.114:6801/1395693466] conn(0x7fb0d00a5f30 0x7fb0d00a83f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T10:29:38.292 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:29:38.291+0000 7fb0e5d9b640 1 --2- 172.21.15.114:0/1621495308 >> [v2:172.21.15.114:6800/1395693466,v1:172.21.15.114:6801/1395693466] conn(0x7fb0d00a5f30 0x7fb0d00a83f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T10:29:38.292 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:29:38.291+0000 7fb0d77fe640 1 -- 172.21.15.114:0/1621495308 <== mon.0 v2:172.21.15.114:3300/0 5 ==== osd_map(92..92 src has 1..92) ==== 8585+0+0 (secure 0 0 0) 0x7fb0dc0d0800 con 0x7fb0e8101090 2024-04-23T10:29:38.292 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:29:38.291+0000 7fb0e5d9b640 1 --2- 172.21.15.114:0/1621495308 >> [v2:172.21.15.114:6800/1395693466,v1:172.21.15.114:6801/1395693466] conn(0x7fb0d00a5f30 0x7fb0d00a83f0 secure :-1 s=READY pgs=85 cs=0 l=1 rev1=1 crypto rx=0x7fb0d8005d80 tx=0x7fb0d8005c90 comp rx=0 tx=0).ready entity=mgr.15291 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T10:29:38.294 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:29:38.295+0000 7fb0d77fe640 1 -- 172.21.15.114:0/1621495308 <== mon.0 v2:172.21.15.114:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7fb0dc09ce60 con 0x7fb0e8101090 2024-04-23T10:29:38.327 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 23 10:29:37 smithi204 ceph-mon[32649]: pgmap v93: 97 pgs: 97 active+clean; 65 MiB data, 899 MiB used, 1.0 TiB / 1.0 TiB avail; 130 KiB/s rd, 2.1 MiB/s wr, 89 op/s 2024-04-23T10:29:38.510 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:29:38.510+0000 7fb0ecfcc640 1 -- 172.21.15.114:0/1621495308 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fb0e8197860 con 0x7fb0e8101090 2024-04-23T10:29:38.511 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:29:38.511+0000 7fb0d77fe640 1 -- 172.21.15.114:0/1621495308 <== mon.0 v2:172.21.15.114:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 23 v23) ==== 94+0+11888 (secure 0 0 0) 0x7fb0dc09c760 con 0x7fb0e8101090 2024-04-23T10:29:38.511 INFO:teuthology.orchestra.run.smithi114.stdout: 2024-04-23T10:29:38.511 INFO:teuthology.orchestra.run.smithi114.stdout:{"epoch":23,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14547,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.114:6835/663493580","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6834","nonce":663493580},{"type":"v1","addr":"172.21.15.114:6835","nonce":663493580}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14565,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.114:6837/1439252154","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6836","nonce":1439252154},{"type":"v1","addr":"172.21.15.114:6837","nonce":1439252154}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5},{"gid":14634,"name":"g","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.114:6839/3849796939","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6838","nonce":3849796939},{"type":"v1","addr":"172.21.15.114:6839","nonce":3849796939}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":11},{"gid":14646,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.114:6841/3544523891","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6840","nonce":3544523891},{"type":"v1","addr":"172.21.15.114:6841","nonce":3544523891}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":12},{"gid":24385,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.204:6833/3866561754","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6832","nonce":3866561754},{"type":"v1","addr":"172.21.15.204:6833","nonce":3866561754}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2},{"gid":24427,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.204:6835/2533793172","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6834","nonce":2533793172},{"type":"v1","addr":"172.21.15.204:6835","nonce":2533793172}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":7}],"filesystems":[{"mdsmap":{"epoch":23,"flags":306,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":true},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-04-23T10:24:43.383158+0000","modified":"2024-04-23T10:28:39.149342+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":3,"in":[0,1,2],"up":{"mds_0":24545,"mds_1":24509,"mds_2":24479},"failed":[],"damaged":[],"stopped":[],"info":{"gid_24439":{"gid":24439,"name":"c","rank":2,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.204:6837/513822179","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6836","nonce":513822179},{"type":"v1","addr":"172.21.15.204:6837","nonce":513822179}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24443":{"gid":24443,"name":"k","rank":1,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.119:6833/783244660","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6832","nonce":783244660},{"type":"v1","addr":"172.21.15.119:6833","nonce":783244660}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24463":{"gid":24463,"name":"i","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.204:6839/3222658893","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6838","nonce":3222658893},{"type":"v1","addr":"172.21.15.204:6839","nonce":3222658893}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24479":{"gid":24479,"name":"e","rank":2,"incarnation":19,"state":"up:active","state_seq":9,"addr":"172.21.15.119:6835/2033916709","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6834","nonce":2033916709},{"type":"v1","addr":"172.21.15.119:6835","nonce":2033916709}]},"join_fscid":-1,"export_targets":[0],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24509":{"gid":24509,"name":"h","rank":1,"incarnation":17,"state":"up:active","state_seq":6,"addr":"172.21.15.119:6837/1872363128","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6836","nonce":1872363128},{"type":"v1","addr":"172.21.15.119:6837","nonce":1872363128}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24545":{"gid":24545,"name":"b","rank":0,"incarnation":14,"state":"up:active","state_seq":3,"addr":"172.21.15.119:6839/937297340","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6838","nonce":937297340},{"type":"v1","addr":"172.21.15.119:6839","nonce":937297340}]},"join_fscid":-1,"export_targets":[1,2],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":24545,"qdb_cluster":[24509,24545,24479]},"id":1}]} 2024-04-23T10:29:38.511 INFO:teuthology.orchestra.run.smithi114.stderr:dumped fsmap epoch 23 2024-04-23T10:29:38.513 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:29:38.513+0000 7fb0ecfcc640 1 -- 172.21.15.114:0/1621495308 >> [v2:172.21.15.114:6800/1395693466,v1:172.21.15.114:6801/1395693466] conn(0x7fb0d00a5f30 msgr2=0x7fb0d00a83f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T10:29:38.513 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:29:38.513+0000 7fb0ecfcc640 1 --2- 172.21.15.114:0/1621495308 >> [v2:172.21.15.114:6800/1395693466,v1:172.21.15.114:6801/1395693466] conn(0x7fb0d00a5f30 0x7fb0d00a83f0 secure :-1 s=READY pgs=85 cs=0 l=1 rev1=1 crypto rx=0x7fb0d8005d80 tx=0x7fb0d8005c90 comp rx=0 tx=0).stop 2024-04-23T10:29:38.513 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:29:38.513+0000 7fb0ecfcc640 1 -- 172.21.15.114:0/1621495308 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fb0e8101090 msgr2=0x7fb0e8196560 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T10:29:38.513 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:29:38.513+0000 7fb0ecfcc640 1 --2- 172.21.15.114:0/1621495308 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fb0e8101090 0x7fb0e8196560 secure :-1 s=READY pgs=588 cs=0 l=1 rev1=1 crypto rx=0x7fb0dc00d770 tx=0x7fb0dc00dc30 comp rx=0 tx=0).stop 2024-04-23T10:29:38.513 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:29:38.513+0000 7fb0ecfcc640 1 -- 172.21.15.114:0/1621495308 shutdown_connections 2024-04-23T10:29:38.513 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:29:38.513+0000 7fb0ecfcc640 1 --2- 172.21.15.114:0/1621495308 >> [v2:172.21.15.114:6800/1395693466,v1:172.21.15.114:6801/1395693466] conn(0x7fb0d00a5f30 0x7fb0d00a83f0 unknown :-1 s=CLOSED pgs=85 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:29:38.514 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:29:38.513+0000 7fb0ecfcc640 1 --2- 172.21.15.114:0/1621495308 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fb0e819add0 0x7fb0e819b280 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:29:38.514 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:29:38.513+0000 7fb0ecfcc640 1 --2- 172.21.15.114:0/1621495308 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fb0e8106160 0x7fb0e8196aa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:29:38.514 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:29:38.513+0000 7fb0ecfcc640 1 --2- 172.21.15.114:0/1621495308 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fb0e8101090 0x7fb0e8196560 unknown :-1 s=CLOSED pgs=588 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:29:38.514 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:29:38.513+0000 7fb0ecfcc640 1 -- 172.21.15.114:0/1621495308 >> 172.21.15.114:0/1621495308 conn(0x7fb0e80771d0 msgr2=0x7fb0e80fedc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T10:29:38.514 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:29:38.513+0000 7fb0ecfcc640 1 -- 172.21.15.114:0/1621495308 shutdown_connections 2024-04-23T10:29:38.514 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:29:38.513+0000 7fb0ecfcc640 1 -- 172.21.15.114:0/1621495308 wait complete. 2024-04-23T10:29:38.524 DEBUG:tasks.quiescer.fs.[cephfs]:Running ceph command: 'tell mds.24545 quiesce db --set-id fec54601 --release --await' 2024-04-23T10:29:38.524 DEBUG:teuthology.orchestra.run.smithi114:> timeout 75 sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.24545 quiesce db --set-id fec54601 --release --await 2024-04-23T10:29:38.671 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:29:38.669+0000 7f827c68a640 1 -- 172.21.15.114:0/1073399450 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f8274106170 msgr2=0x7f827410dd40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T10:29:38.671 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:29:38.669+0000 7f827c68a640 1 --2- 172.21.15.114:0/1073399450 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f8274106170 0x7f827410dd40 secure :-1 s=READY pgs=589 cs=0 l=1 rev1=1 crypto rx=0x7f826c009ca0 tx=0x7f826c02a8c0 comp rx=0 tx=0).stop 2024-04-23T10:29:38.671 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:29:38.669+0000 7f827c68a640 1 -- 172.21.15.114:0/1073399450 shutdown_connections 2024-04-23T10:29:38.672 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:29:38.669+0000 7f827c68a640 1 --2- 172.21.15.114:0/1073399450 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f8274106170 0x7f827410dd40 unknown :-1 s=CLOSED pgs=589 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:29:38.672 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:29:38.669+0000 7f827c68a640 1 --2- 172.21.15.114:0/1073399450 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f8274101a50 0x7f8274105a90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:29:38.672 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:29:38.669+0000 7f827c68a640 1 --2- 172.21.15.114:0/1073399450 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f82741010a0 0x7f8274101480 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:29:38.672 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:29:38.669+0000 7f827c68a640 1 -- 172.21.15.114:0/1073399450 >> 172.21.15.114:0/1073399450 conn(0x7f8274077eb0 msgr2=0x7f82740782c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T10:29:38.672 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:29:38.669+0000 7f827c68a640 1 -- 172.21.15.114:0/1073399450 shutdown_connections 2024-04-23T10:29:38.672 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:29:38.669+0000 7f827c68a640 1 -- 172.21.15.114:0/1073399450 wait complete. 2024-04-23T10:29:38.672 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:29:38.670+0000 7f827c68a640 1 Processor -- start 2024-04-23T10:29:38.672 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:29:38.670+0000 7f827c68a640 1 -- start start 2024-04-23T10:29:38.672 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:29:38.670+0000 7f827c68a640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f82741010a0 0x7f8274102b30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T10:29:38.672 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:29:38.670+0000 7f827c68a640 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f8274106170 0x7f8274103070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T10:29:38.672 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:29:38.670+0000 7f827c68a640 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f82741035b0 0x7f82741a68c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T10:29:38.672 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:29:38.670+0000 7f827c68a640 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x7f82741110c0 con 0x7f82741010a0 2024-04-23T10:29:38.672 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:29:38.670+0000 7f827c68a640 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 -- 0x7f8274110f40 con 0x7f82741035b0 2024-04-23T10:29:38.673 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:29:38.670+0000 7f827c68a640 1 -- --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_getmap magic: 0 -- 0x7f8274111240 con 0x7f8274106170 2024-04-23T10:29:38.673 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:29:38.670+0000 7f827a426640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f82741010a0 0x7f8274102b30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T10:29:38.673 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:29:38.670+0000 7f827a426640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f82741010a0 0x7f8274102b30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.114:49008/0 (socket says 172.21.15.114:49008) 2024-04-23T10:29:38.673 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:29:38.670+0000 7f827a426640 1 -- 172.21.15.114:0/2087320188 learned_addr learned my addr 172.21.15.114:0/2087320188 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-04-23T10:29:38.673 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:29:38.670+0000 7f827ac27640 1 --2- 172.21.15.114:0/2087320188 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f82741035b0 0x7f82741a68c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T10:29:38.673 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:29:38.670+0000 7f827a426640 1 -- 172.21.15.114:0/2087320188 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f8274106170 msgr2=0x7f8274103070 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T10:29:38.673 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:29:38.670+0000 7f8279c25640 1 --2- 172.21.15.114:0/2087320188 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f8274106170 0x7f8274103070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T10:29:38.673 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:29:38.670+0000 7f827a426640 1 --2- 172.21.15.114:0/2087320188 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f8274106170 0x7f8274103070 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:29:38.673 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:29:38.670+0000 7f827a426640 1 -- 172.21.15.114:0/2087320188 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f82741035b0 msgr2=0x7f82741a68c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T10:29:38.673 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:29:38.670+0000 7f827a426640 1 --2- 172.21.15.114:0/2087320188 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f82741035b0 0x7f82741a68c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:29:38.673 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:29:38.670+0000 7f827a426640 1 -- 172.21.15.114:0/2087320188 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f82741a6fc0 con 0x7f82741010a0 2024-04-23T10:29:38.673 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:29:38.670+0000 7f827ac27640 1 --2- 172.21.15.114:0/2087320188 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f82741035b0 0x7f82741a68c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-23T10:29:38.674 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:29:38.670+0000 7f8279c25640 1 --2- 172.21.15.114:0/2087320188 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f8274106170 0x7f8274103070 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-23T10:29:38.674 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:29:38.670+0000 7f827a426640 1 --2- 172.21.15.114:0/2087320188 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f82741010a0 0x7f8274102b30 secure :-1 s=READY pgs=590 cs=0 l=1 rev1=1 crypto rx=0x7f827000d950 tx=0x7f827000de10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T10:29:38.674 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:29:38.670+0000 7f826b7fe640 1 -- 172.21.15.114:0/2087320188 <== mon.0 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8270017070 con 0x7f82741010a0 2024-04-23T10:29:38.674 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:29:38.670+0000 7f827c68a640 1 -- 172.21.15.114:0/2087320188 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f82741a72b0 con 0x7f82741010a0 2024-04-23T10:29:38.674 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:29:38.670+0000 7f826b7fe640 1 -- 172.21.15.114:0/2087320188 <== mon.0 v2:172.21.15.114:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f8270009c30 con 0x7f82741010a0 2024-04-23T10:29:38.674 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:29:38.670+0000 7f826b7fe640 1 -- 172.21.15.114:0/2087320188 <== mon.0 v2:172.21.15.114:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f827000f620 con 0x7f82741010a0 2024-04-23T10:29:38.674 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:29:38.670+0000 7f827c68a640 1 -- 172.21.15.114:0/2087320188 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f82741a7800 con 0x7f82741010a0 2024-04-23T10:29:38.681 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:29:38.673+0000 7f826b7fe640 1 -- 172.21.15.114:0/2087320188 <== mon.0 v2:172.21.15.114:3300/0 4 ==== mgrmap(e 26) ==== 138778+0+0 (secure 0 0 0) 0x7f8270010460 con 0x7f82741010a0 2024-04-23T10:29:38.681 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:29:38.673+0000 7f827f3a4740 1 Processor -- start 2024-04-23T10:29:38.681 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:29:38.674+0000 7f827f3a4740 1 -- start start 2024-04-23T10:29:38.681 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:29:38.674+0000 7f827f3a4740 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x557f01455490 0x557f01455870 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T10:29:38.681 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:29:38.674+0000 7f827f3a4740 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x557f01455db0 0x557f0145ff40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T10:29:38.681 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:29:38.674+0000 7f827f3a4740 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x557f01460480 0x557f01462870 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T10:29:38.682 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:29:38.674+0000 7f827f3a4740 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x557f013ba6d0 con 0x557f01460480 2024-04-23T10:29:38.682 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:29:38.674+0000 7f827f3a4740 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 -- 0x557f013571a0 con 0x557f01455db0 2024-04-23T10:29:38.682 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:29:38.674+0000 7f827f3a4740 1 -- --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_getmap magic: 0 -- 0x557f013b6690 con 0x557f01455490 2024-04-23T10:29:38.682 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:29:38.674+0000 7f8279c25640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x557f01460480 0x557f01462870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T10:29:38.682 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:29:38.674+0000 7f8279c25640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x557f01460480 0x557f01462870 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.114:49010/0 (socket says 172.21.15.114:49010) 2024-04-23T10:29:38.682 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:29:38.674+0000 7f8279c25640 1 -- 172.21.15.114:0/2433752285 learned_addr learned my addr 172.21.15.114:0/2433752285 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-04-23T10:29:38.682 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:29:38.674+0000 7f826b7fe640 1 --2- 172.21.15.114:0/2087320188 >> [v2:172.21.15.114:6800/1395693466,v1:172.21.15.114:6801/1395693466] conn(0x7f82600a3d50 0x7f82600a6210 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T10:29:38.682 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:29:38.674+0000 7f827ac27640 1 --2- 172.21.15.114:0/2433752285 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x557f01455490 0x557f01455870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T10:29:38.682 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:29:38.674+0000 7f827a426640 1 --2- 172.21.15.114:0/2433752285 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x557f01455db0 0x557f0145ff40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T10:29:38.682 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:29:38.674+0000 7f827ac27640 1 --2- 172.21.15.114:0/2087320188 >> [v2:172.21.15.114:6800/1395693466,v1:172.21.15.114:6801/1395693466] conn(0x7f82600a3d50 0x7f82600a6210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T10:29:38.682 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:29:38.674+0000 7f826b7fe640 1 -- 172.21.15.114:0/2087320188 <== mon.0 v2:172.21.15.114:3300/0 5 ==== osd_map(92..92 src has 1..92) ==== 8585+0+0 (secure 0 0 0) 0x7f82700d0500 con 0x7f82741010a0 2024-04-23T10:29:38.683 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:29:38.674+0000 7f8279c25640 1 -- 172.21.15.114:0/2433752285 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x557f01455490 msgr2=0x557f01455870 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T10:29:38.683 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:29:38.674+0000 7f8279c25640 1 --2- 172.21.15.114:0/2433752285 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x557f01455490 0x557f01455870 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:29:38.683 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:29:38.674+0000 7f8279c25640 1 -- 172.21.15.114:0/2433752285 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x557f01455db0 msgr2=0x557f0145ff40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T10:29:38.683 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:29:38.674+0000 7f8279c25640 1 --2- 172.21.15.114:0/2433752285 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x557f01455db0 0x557f0145ff40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:29:38.683 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:29:38.674+0000 7f8279c25640 1 -- 172.21.15.114:0/2433752285 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x557f013615b0 con 0x557f01460480 2024-04-23T10:29:38.683 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:29:38.674+0000 7f827a426640 1 --2- 172.21.15.114:0/2433752285 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x557f01455db0 0x557f0145ff40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-23T10:29:38.683 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:29:38.674+0000 7f827ac27640 1 --2- 172.21.15.114:0/2433752285 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x557f01455490 0x557f01455870 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-23T10:29:38.683 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:29:38.674+0000 7f8279c25640 1 --2- 172.21.15.114:0/2433752285 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x557f01460480 0x557f01462870 secure :-1 s=READY pgs=591 cs=0 l=1 rev1=1 crypto rx=0x7f826400ba70 tx=0x7f826400bf30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8d5344aff63a4762 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T10:29:38.683 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:29:38.675+0000 7f8268ff9640 1 -- 172.21.15.114:0/2433752285 <== mon.0 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8264017420 con 0x557f01460480 2024-04-23T10:29:38.683 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:29:38.675+0000 7f827ac27640 1 --2- 172.21.15.114:0/2087320188 >> [v2:172.21.15.114:6800/1395693466,v1:172.21.15.114:6801/1395693466] conn(0x7f82600a3d50 0x7f82600a6210 secure :-1 s=READY pgs=86 cs=0 l=1 rev1=1 crypto rx=0x7f826c005e30 tx=0x7f826c0095c0 comp rx=0 tx=0).ready entity=mgr.15291 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T10:29:38.683 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:29:38.675+0000 7f8268ff9640 1 -- 172.21.15.114:0/2433752285 <== mon.0 v2:172.21.15.114:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f8264017a70 con 0x557f01460480 2024-04-23T10:29:38.683 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:29:38.675+0000 7f8268ff9640 1 -- 172.21.15.114:0/2433752285 <== mon.0 v2:172.21.15.114:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f826400fa20 con 0x557f01460480 2024-04-23T10:29:38.684 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:29:38.675+0000 7f827f3a4740 1 -- 172.21.15.114:0/2433752285 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x557f01460480 msgr2=0x557f01462870 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T10:29:38.684 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:29:38.675+0000 7f827f3a4740 1 --2- 172.21.15.114:0/2433752285 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x557f01460480 0x557f01462870 secure :-1 s=READY pgs=591 cs=0 l=1 rev1=1 crypto rx=0x7f826400ba70 tx=0x7f826400bf30 comp rx=0 tx=0).stop 2024-04-23T10:29:38.684 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:29:38.675+0000 7f827f3a4740 1 -- 172.21.15.114:0/2433752285 shutdown_connections 2024-04-23T10:29:38.684 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:29:38.675+0000 7f827f3a4740 1 --2- 172.21.15.114:0/2433752285 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x557f01460480 0x557f01462870 unknown :-1 s=CLOSED pgs=591 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:29:38.684 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:29:38.675+0000 7f827f3a4740 1 --2- 172.21.15.114:0/2433752285 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x557f01455db0 0x557f0145ff40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:29:38.684 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:29:38.675+0000 7f827f3a4740 1 --2- 172.21.15.114:0/2433752285 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x557f01455490 0x557f01455870 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:29:38.684 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:29:38.675+0000 7f827f3a4740 1 -- 172.21.15.114:0/2433752285 >> 172.21.15.114:0/2433752285 conn(0x557f014517b0 msgr2=0x557f01451b90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T10:29:38.684 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:29:38.675+0000 7f827f3a4740 1 -- 172.21.15.114:0/2433752285 shutdown_connections 2024-04-23T10:29:38.684 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:29:38.675+0000 7f827f3a4740 1 -- 172.21.15.114:0/2433752285 wait complete. 2024-04-23T10:29:38.684 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:29:38.676+0000 7f827f3a4740 1 -- start start 2024-04-23T10:29:38.684 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:29:38.676+0000 7f827f3a4740 1 Processor -- start 2024-04-23T10:29:38.684 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:29:38.676+0000 7f827f3a4740 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x557f01455490 0x557f0145fe10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T10:29:38.685 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:29:38.676+0000 7f827f3a4740 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x557f01455db0 0x557f0145d100 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T10:29:38.685 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:29:38.676+0000 7f827f3a4740 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x557f01460480 0x557f0145d680 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T10:29:38.685 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:29:38.676+0000 7f827f3a4740 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x557f01463310 con 0x557f01460480 2024-04-23T10:29:38.685 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:29:38.676+0000 7f827f3a4740 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 -- 0x557f01354b20 con 0x557f01455490 2024-04-23T10:29:38.685 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:29:38.676+0000 7f827f3a4740 1 -- --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_getmap magic: 0 -- 0x557f01463490 con 0x557f01455db0 2024-04-23T10:29:38.685 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:29:38.676+0000 7f827f3a4740 10 client.0 _ll_register_callbacks cb 0x557f01354530 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-23T10:29:38.685 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:29:38.676+0000 7f827ac27640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x557f01460480 0x557f0145d680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T10:29:38.685 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:29:38.676+0000 7f827ac27640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x557f01460480 0x557f0145d680 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.114:49014/0 (socket says 172.21.15.114:49014) 2024-04-23T10:29:38.685 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:29:38.676+0000 7f827ac27640 1 -- 172.21.15.114:0/2445264026 learned_addr learned my addr 172.21.15.114:0/2445264026 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-04-23T10:29:38.685 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:29:38.676+0000 7f827a426640 1 --2- 172.21.15.114:0/2445264026 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x557f01455490 0x557f0145fe10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T10:29:38.685 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:29:38.676+0000 7f8279c25640 1 --2- 172.21.15.114:0/2445264026 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x557f01455db0 0x557f0145d100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T10:29:38.685 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:29:38.676+0000 7f827ac27640 1 -- 172.21.15.114:0/2445264026 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x557f01455db0 msgr2=0x557f0145d100 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T10:29:38.686 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:29:38.676+0000 7f827ac27640 1 --2- 172.21.15.114:0/2445264026 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x557f01455db0 0x557f0145d100 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:29:38.686 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:29:38.676+0000 7f827ac27640 1 -- 172.21.15.114:0/2445264026 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x557f01455490 msgr2=0x557f0145fe10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T10:29:38.686 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:29:38.676+0000 7f827ac27640 1 --2- 172.21.15.114:0/2445264026 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x557f01455490 0x557f0145fe10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:29:38.686 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:29:38.676+0000 7f827ac27640 1 -- 172.21.15.114:0/2445264026 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({osdmap=0}) -- 0x557f010cc920 con 0x557f01460480 2024-04-23T10:29:38.686 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:29:38.677+0000 7f827ac27640 1 --2- 172.21.15.114:0/2445264026 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x557f01460480 0x557f0145d680 secure :-1 s=READY pgs=592 cs=0 l=1 rev1=1 crypto rx=0x7f826c009dc0 tx=0x7f826c02a8c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2145a43d412d560f server_cookie=0 in_seq=0 out_seq=0 2024-04-23T10:29:38.686 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:29:38.677+0000 7f825dffb640 10 client.0 ms_handle_connect on v2:172.21.15.114:3300/0 2024-04-23T10:29:38.686 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:29:38.677+0000 7f825dffb640 1 -- 172.21.15.114:0/2445264026 <== mon.0 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f826c031920 con 0x557f01460480 2024-04-23T10:29:38.686 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:29:38.677+0000 7f825dffb640 1 -- 172.21.15.114:0/2445264026 <== mon.0 v2:172.21.15.114:3300/0 2 ==== osd_map(92..92 src has 1..92) ==== 8585+0+0 (secure 0 0 0) 0x7f826c004450 con 0x557f01460480 2024-04-23T10:29:38.686 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:29:38.677+0000 7f827f3a4740 1 -- 172.21.15.114:0/2445264026 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f8264017a70 con 0x557f01460480 2024-04-23T10:29:38.686 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:29:38.677+0000 7f825dffb640 1 -- 172.21.15.114:0/2445264026 <== mon.0 v2:172.21.15.114:3300/0 3 ==== mon_get_version_reply(handle=1 version=23) ==== 24+0+0 (secure 0 0 0) 0x7f826c004650 con 0x557f01460480 2024-04-23T10:29:38.686 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:29:38.677+0000 7f827f3a4740 10 client.15930 fetch_fsmap learned FSMap version 23 2024-04-23T10:29:38.686 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:29:38.677+0000 7f827f3a4740 1 -- 172.21.15.114:0/2445264026 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({config=0+,fsmap=23,monmap=4+}) -- 0x557f01198d30 con 0x557f01460480 2024-04-23T10:29:38.686 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:29:38.677+0000 7f825dffb640 1 -- 172.21.15.114:0/2445264026 <== mon.0 v2:172.21.15.114:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f8264017a70 con 0x557f01460480 2024-04-23T10:29:38.687 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:29:38.677+0000 7f825dffb640 1 -- 172.21.15.114:0/2445264026 <== mon.0 v2:172.21.15.114:3300/0 5 ==== fsmap(e 23) ==== 8350+0+0 (secure 0 0 0) 0x7f826c030650 con 0x557f01460480 2024-04-23T10:29:38.687 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:29:38.677+0000 7f827f3a4740 10 client.15930 fetch_fsmap finished waiting for FSMap version 23 2024-04-23T10:29:38.687 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:29:38.677+0000 7f827f3a4740 10 client.15930 resolve_mds: validated gid 24545 aka daemon mds.b 2024-04-23T10:29:38.687 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:29:38.677+0000 7f827f3a4740 20 client.15930 populate_metadata read hostname 'smithi114' 2024-04-23T10:29:38.687 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:29:38.677+0000 7f827f3a4740 1 --2- 172.21.15.114:0/2445264026 >> [v2:172.21.15.119:6838/937297340,v1:172.21.15.119:6839/937297340] conn(0x557f01470530 0x557f014729d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T10:29:38.687 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:29:38.677+0000 7f827f3a4740 4 client.15930 mds_command: new command op to 24545 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-23T10:29:38.687 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:29:38.677+0000 7f827f3a4740 1 -- 172.21.15.114:0/2445264026 --> [v2:172.21.15.119:6838/937297340,v1:172.21.15.119:6839/937297340] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x557f01378500 con 0x557f01470530 2024-04-23T10:29:38.687 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:29:38.678+0000 7f827a426640 1 --2- 172.21.15.114:0/2445264026 >> [v2:172.21.15.119:6838/937297340,v1:172.21.15.119:6839/937297340] conn(0x557f01470530 0x557f014729d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T10:29:38.687 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:29:38.678+0000 7f827a426640 1 --2- 172.21.15.114:0/2445264026 >> [v2:172.21.15.119:6838/937297340,v1:172.21.15.119:6839/937297340] conn(0x557f01470530 0x557f014729d0 crc :-1 s=READY pgs=76 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=4ab86b7af85055cc server_cookie=39b327835b0274ff in_seq=0 out_seq=0 2024-04-23T10:29:38.687 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:29:38.678+0000 7f825dffb640 10 client.15930 ms_handle_connect on v2:172.21.15.119:6838/937297340 2024-04-23T10:29:38.687 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:29:38.679+0000 7f825dffb640 1 -- 172.21.15.114:0/2445264026 <== mds.0 v2:172.21.15.119:6838/937297340 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x557f01378500 con 0x557f01470530 2024-04-23T10:29:38.687 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:29:38.679+0000 7f825dffb640 10 client.15930 handle_command_reply: tid=0 2024-04-23T10:29:38.688 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:29:38.679+0000 7f827f3a4740 1 client.15930 shutdown 2024-04-23T10:29:38.688 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:29:38.679+0000 7f827f3a4740 1 -- 172.21.15.114:0/2445264026 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x557f01460480 msgr2=0x557f0145d680 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T10:29:38.688 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:29:38.679+0000 7f827f3a4740 1 --2- 172.21.15.114:0/2445264026 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x557f01460480 0x557f0145d680 secure :-1 s=READY pgs=592 cs=0 l=1 rev1=1 crypto rx=0x7f826c009dc0 tx=0x7f826c02a8c0 comp rx=0 tx=0).stop 2024-04-23T10:29:38.688 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:29:38.679+0000 7f827f3a4740 1 -- 172.21.15.114:0/2445264026 shutdown_connections 2024-04-23T10:29:38.688 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:29:38.679+0000 7f827f3a4740 1 --2- 172.21.15.114:0/2445264026 >> [v2:172.21.15.119:6838/937297340,v1:172.21.15.119:6839/937297340] conn(0x557f01470530 0x557f014729d0 crc :-1 s=READY pgs=76 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:29:38.688 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:29:38.679+0000 7f827f3a4740 1 --2- 172.21.15.114:0/2445264026 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x557f01460480 0x557f0145d680 unknown :-1 s=CLOSED pgs=592 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:29:38.688 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:29:38.679+0000 7f827f3a4740 1 --2- 172.21.15.114:0/2445264026 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x557f01455db0 0x557f0145d100 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:29:38.688 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:29:38.679+0000 7f827f3a4740 1 --2- 172.21.15.114:0/2445264026 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x557f01455490 0x557f0145fe10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:29:38.688 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:29:38.679+0000 7f827f3a4740 1 -- 172.21.15.114:0/2445264026 >> 172.21.15.114:0/2445264026 conn(0x557f014517b0 msgr2=0x557f014575b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T10:29:38.688 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:29:38.679+0000 7f825dffb640 0 client.15930 ms_handle_reset on v2:172.21.15.119:6838/937297340 2024-04-23T10:29:38.688 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:29:38.679+0000 7f827f3a4740 1 -- 172.21.15.114:0/2445264026 shutdown_connections 2024-04-23T10:29:38.688 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:29:38.679+0000 7f827f3a4740 1 -- 172.21.15.114:0/2445264026 wait complete. 2024-04-23T10:29:38.688 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:29:38.679+0000 7f827f3a4740 20 client.15930 trim_cache size 0 max 16384 2024-04-23T10:29:38.701 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:29:38.696+0000 7f827f3a4740 1 Processor -- start 2024-04-23T10:29:38.701 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:29:38.696+0000 7f827f3a4740 1 -- start start 2024-04-23T10:29:38.701 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:29:38.696+0000 7f827f3a4740 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x557f01455490 0x557f01451150 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T10:29:38.702 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:29:38.696+0000 7f827f3a4740 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x557f01455db0 0x557f0145f940 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T10:29:38.702 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:29:38.696+0000 7f827f3a4740 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x557f01460480 0x557f014782a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T10:29:38.702 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:29:38.696+0000 7f827f3a4740 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x7f82741b27f0 con 0x557f01460480 2024-04-23T10:29:38.702 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:29:38.696+0000 7f827f3a4740 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 -- 0x7f82741b0090 con 0x557f01455db0 2024-04-23T10:29:38.702 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:29:38.696+0000 7f827f3a4740 1 -- --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_getmap magic: 0 -- 0x7f82741adee0 con 0x557f01455490 2024-04-23T10:29:38.702 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:29:38.696+0000 7f827ac27640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x557f01460480 0x557f014782a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T10:29:38.702 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:29:38.696+0000 7f827ac27640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x557f01460480 0x557f014782a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.114:49022/0 (socket says 172.21.15.114:49022) 2024-04-23T10:29:38.702 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:29:38.696+0000 7f827ac27640 1 -- 172.21.15.114:0/1372627732 learned_addr learned my addr 172.21.15.114:0/1372627732 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-04-23T10:29:38.702 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:29:38.696+0000 7f827ac27640 1 -- 172.21.15.114:0/1372627732 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x557f01455490 msgr2=0x557f01451150 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T10:29:38.702 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:29:38.696+0000 7f827a426640 1 --2- 172.21.15.114:0/1372627732 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x557f01455490 0x557f01451150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T10:29:38.702 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:29:38.696+0000 7f8279c25640 1 --2- 172.21.15.114:0/1372627732 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x557f01455db0 0x557f0145f940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T10:29:38.702 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:29:38.696+0000 7f827ac27640 1 --2- 172.21.15.114:0/1372627732 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x557f01455490 0x557f01451150 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:29:38.703 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:29:38.696+0000 7f827ac27640 1 -- 172.21.15.114:0/1372627732 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x557f01455db0 msgr2=0x557f0145f940 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T10:29:38.703 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:29:38.696+0000 7f827ac27640 1 --2- 172.21.15.114:0/1372627732 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x557f01455db0 0x557f0145f940 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:29:38.703 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:29:38.696+0000 7f827ac27640 1 -- 172.21.15.114:0/1372627732 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x557f0145d7a0 con 0x557f01460480 2024-04-23T10:29:38.703 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:29:38.696+0000 7f827a426640 1 --2- 172.21.15.114:0/1372627732 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x557f01455490 0x557f01451150 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-23T10:29:38.703 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:29:38.696+0000 7f827ac27640 1 --2- 172.21.15.114:0/1372627732 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x557f01460480 0x557f014782a0 secure :-1 s=READY pgs=593 cs=0 l=1 rev1=1 crypto rx=0x7f826c030b10 tx=0x7f826c02b040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f5022078693ed9c9 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T10:29:38.703 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:29:38.696+0000 7f825ffff640 1 -- 172.21.15.114:0/1372627732 <== mon.0 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f826c0047f0 con 0x557f01460480 2024-04-23T10:29:38.703 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:29:38.696+0000 7f825ffff640 1 -- 172.21.15.114:0/1372627732 <== mon.0 v2:172.21.15.114:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f826c04c070 con 0x557f01460480 2024-04-23T10:29:38.703 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:29:38.696+0000 7f825ffff640 1 -- 172.21.15.114:0/1372627732 <== mon.0 v2:172.21.15.114:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f826c02fb10 con 0x557f01460480 2024-04-23T10:29:38.703 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:29:38.697+0000 7f827f3a4740 1 -- 172.21.15.114:0/1372627732 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x557f01460480 msgr2=0x557f014782a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T10:29:38.703 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:29:38.697+0000 7f827f3a4740 1 --2- 172.21.15.114:0/1372627732 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x557f01460480 0x557f014782a0 secure :-1 s=READY pgs=593 cs=0 l=1 rev1=1 crypto rx=0x7f826c030b10 tx=0x7f826c02b040 comp rx=0 tx=0).stop 2024-04-23T10:29:38.703 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:29:38.697+0000 7f827f3a4740 1 -- 172.21.15.114:0/1372627732 shutdown_connections 2024-04-23T10:29:38.703 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:29:38.697+0000 7f827f3a4740 1 --2- 172.21.15.114:0/1372627732 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x557f01460480 0x557f014782a0 unknown :-1 s=CLOSED pgs=593 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:29:38.704 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:29:38.697+0000 7f827f3a4740 1 --2- 172.21.15.114:0/1372627732 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x557f01455db0 0x557f0145f940 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:29:38.704 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:29:38.697+0000 7f827f3a4740 1 --2- 172.21.15.114:0/1372627732 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x557f01455490 0x557f01451150 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:29:38.704 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:29:38.697+0000 7f827f3a4740 1 -- 172.21.15.114:0/1372627732 >> 172.21.15.114:0/1372627732 conn(0x557f014517b0 msgr2=0x557f014583e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T10:29:38.704 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:29:38.697+0000 7f827f3a4740 1 -- 172.21.15.114:0/1372627732 shutdown_connections 2024-04-23T10:29:38.704 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:29:38.697+0000 7f827f3a4740 1 -- 172.21.15.114:0/1372627732 wait complete. 2024-04-23T10:29:38.704 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:29:38.697+0000 7f827f3a4740 1 -- start start 2024-04-23T10:29:38.704 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:29:38.697+0000 7f827f3a4740 1 Processor -- start 2024-04-23T10:29:38.704 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:29:38.698+0000 7f827f3a4740 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x557f01455490 0x557f01468a30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T10:29:38.704 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:29:38.698+0000 7f827f3a4740 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x557f01455db0 0x557f01460e00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T10:29:38.704 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:29:38.698+0000 7f827f3a4740 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x557f01460480 0x557f01461340 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T10:29:38.704 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:29:38.698+0000 7f827f3a4740 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x7f82741ae550 con 0x557f01460480 2024-04-23T10:29:38.704 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:29:38.698+0000 7f827f3a4740 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 -- 0x7f82741af9c0 con 0x557f01455490 2024-04-23T10:29:38.704 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:29:38.698+0000 7f827f3a4740 1 -- --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_getmap magic: 0 -- 0x7f82741aec20 con 0x557f01455db0 2024-04-23T10:29:38.705 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:29:38.698+0000 7f827f3a4740 10 client.0 _ll_register_callbacks cb 0x7f826c031450 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-23T10:29:38.705 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:29:38.698+0000 7f827ac27640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x557f01460480 0x557f01461340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T10:29:38.705 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:29:38.698+0000 7f827ac27640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x557f01460480 0x557f01461340 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.114:49030/0 (socket says 172.21.15.114:49030) 2024-04-23T10:29:38.705 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:29:38.698+0000 7f827ac27640 1 -- 172.21.15.114:0/1849780824 learned_addr learned my addr 172.21.15.114:0/1849780824 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-04-23T10:29:38.705 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:29:38.698+0000 7f8279c25640 1 --2- 172.21.15.114:0/1849780824 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x557f01455db0 0x557f01460e00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T10:29:38.705 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:29:38.698+0000 7f827ac27640 1 -- 172.21.15.114:0/1849780824 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x557f01455db0 msgr2=0x557f01460e00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T10:29:38.705 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:29:38.698+0000 7f827ac27640 1 --2- 172.21.15.114:0/1849780824 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x557f01455db0 0x557f01460e00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:29:38.705 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:29:38.698+0000 7f827ac27640 1 -- 172.21.15.114:0/1849780824 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x557f01455490 msgr2=0x557f01468a30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T10:29:38.705 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:29:38.698+0000 7f827ac27640 1 --2- 172.21.15.114:0/1849780824 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x557f01455490 0x557f01468a30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:29:38.705 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:29:38.698+0000 7f827ac27640 1 -- 172.21.15.114:0/1849780824 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({osdmap=0}) -- 0x557f01461b70 con 0x557f01460480 2024-04-23T10:29:38.705 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:29:38.698+0000 7f8279c25640 1 --2- 172.21.15.114:0/1849780824 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x557f01455db0 0x557f01460e00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-23T10:29:38.705 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:29:38.698+0000 7f827ac27640 1 --2- 172.21.15.114:0/1849780824 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x557f01460480 0x557f01461340 secure :-1 s=READY pgs=594 cs=0 l=1 rev1=1 crypto rx=0x7f826c009ca0 tx=0x7f826c030eb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b9ee44662be3050a server_cookie=0 in_seq=0 out_seq=0 2024-04-23T10:29:38.706 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:29:38.698+0000 7f825dffb640 10 client.0 ms_handle_connect on v2:172.21.15.114:3300/0 2024-04-23T10:29:38.706 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:29:38.698+0000 7f825dffb640 1 -- 172.21.15.114:0/1849780824 <== mon.0 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f826c066070 con 0x557f01460480 2024-04-23T10:29:38.706 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:29:38.698+0000 7f827f3a4740 1 -- 172.21.15.114:0/1849780824 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f826c04c070 con 0x557f01460480 2024-04-23T10:29:38.706 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:29:38.698+0000 7f825dffb640 1 -- 172.21.15.114:0/1849780824 <== mon.0 v2:172.21.15.114:3300/0 2 ==== osd_map(92..92 src has 1..92) ==== 8585+0+0 (secure 0 0 0) 0x7f826c02f860 con 0x557f01460480 2024-04-23T10:29:38.706 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:29:38.698+0000 7f825dffb640 1 -- 172.21.15.114:0/1849780824 <== mon.0 v2:172.21.15.114:3300/0 3 ==== mon_get_version_reply(handle=1 version=23) ==== 24+0+0 (secure 0 0 0) 0x7f826c072050 con 0x557f01460480 2024-04-23T10:29:38.706 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:29:38.698+0000 7f827f3a4740 10 client.15936 fetch_fsmap learned FSMap version 23 2024-04-23T10:29:38.706 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:29:38.698+0000 7f827f3a4740 1 -- 172.21.15.114:0/1849780824 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({config=0+,fsmap=23,monmap=4+}) -- 0x557f014d6b10 con 0x557f01460480 2024-04-23T10:29:38.706 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:29:38.699+0000 7f825dffb640 1 -- 172.21.15.114:0/1849780824 <== mon.0 v2:172.21.15.114:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f826c04c070 con 0x557f01460480 2024-04-23T10:29:38.706 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:29:38.699+0000 7f825dffb640 1 -- 172.21.15.114:0/1849780824 <== mon.0 v2:172.21.15.114:3300/0 5 ==== fsmap(e 23) ==== 8350+0+0 (secure 0 0 0) 0x7f826c051af0 con 0x557f01460480 2024-04-23T10:29:38.706 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:29:38.699+0000 7f827f3a4740 10 client.15936 fetch_fsmap finished waiting for FSMap version 23 2024-04-23T10:29:38.706 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:29:38.699+0000 7f827f3a4740 10 client.15936 resolve_mds: validated gid 24545 aka daemon mds.b 2024-04-23T10:29:38.706 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:29:38.699+0000 7f827f3a4740 20 client.15936 populate_metadata read hostname 'smithi114' 2024-04-23T10:29:38.707 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:29:38.699+0000 7f827f3a4740 1 --2- 172.21.15.114:0/1849780824 >> [v2:172.21.15.119:6838/937297340,v1:172.21.15.119:6839/937297340] conn(0x557f014d70d0 0x557f014d94f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T10:29:38.707 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:29:38.699+0000 7f827f3a4740 4 client.15936 mds_command: new command op to 24545 tid=0 multi_id=0 [{"prefix": "quiesce db", "set_id": "fec54601", "release": true, "await": true}] 2024-04-23T10:29:38.707 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:29:38.699+0000 7f827f3a4740 1 -- 172.21.15.114:0/1849780824 --> [v2:172.21.15.119:6838/937297340,v1:172.21.15.119:6839/937297340] -- command(tid 0: {"prefix": "quiesce db", "set_id": "fec54601", "release": true, "await": true}) -- 0x557f013c80f0 con 0x557f014d70d0 2024-04-23T10:29:38.707 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:29:38.699+0000 7f827a426640 1 --2- 172.21.15.114:0/1849780824 >> [v2:172.21.15.119:6838/937297340,v1:172.21.15.119:6839/937297340] conn(0x557f014d70d0 0x557f014d94f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T10:29:38.707 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:29:38.699+0000 7f825dffb640 10 client.15936 ms_handle_connect on v2:172.21.15.119:6838/937297340 2024-04-23T10:29:38.707 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:29:38.699+0000 7f827a426640 1 --2- 172.21.15.114:0/1849780824 >> [v2:172.21.15.119:6838/937297340,v1:172.21.15.119:6839/937297340] conn(0x557f014d70d0 0x557f014d94f0 crc :-1 s=READY pgs=77 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=6d5ba2200074f5ab server_cookie=22ba2f004fdf3ceb in_seq=0 out_seq=0 2024-04-23T10:29:38.709 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:29:38.710+0000 7f825dffb640 1 -- 172.21.15.114:0/1849780824 <== mds.0 v2:172.21.15.119:6838/937297340 1 ==== command_reply(tid 0: 0 ) ==== 8+0+590 (crc 0 0 0) 0x557f013c80f0 con 0x557f014d70d0 2024-04-23T10:29:38.710 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:29:38.710+0000 7f825dffb640 10 client.15936 handle_command_reply: tid=0 2024-04-23T10:29:38.710 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:29:38.710+0000 7f827f3a4740 1 client.15936 shutdown 2024-04-23T10:29:38.710 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:29:38.710+0000 7f827f3a4740 1 -- 172.21.15.114:0/1849780824 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x557f01460480 msgr2=0x557f01461340 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T10:29:38.710 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:29:38.710+0000 7f827f3a4740 1 --2- 172.21.15.114:0/1849780824 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x557f01460480 0x557f01461340 secure :-1 s=READY pgs=594 cs=0 l=1 rev1=1 crypto rx=0x7f826c009ca0 tx=0x7f826c030eb0 comp rx=0 tx=0).stop 2024-04-23T10:29:38.710 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:29:38.710+0000 7f827f3a4740 1 -- 172.21.15.114:0/1849780824 shutdown_connections 2024-04-23T10:29:38.710 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:29:38.710+0000 7f827f3a4740 1 --2- 172.21.15.114:0/1849780824 >> [v2:172.21.15.119:6838/937297340,v1:172.21.15.119:6839/937297340] conn(0x557f014d70d0 0x557f014d94f0 crc :-1 s=READY pgs=77 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:29:38.710 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:29:38.710+0000 7f827f3a4740 1 --2- 172.21.15.114:0/1849780824 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x557f01460480 0x557f01461340 unknown :-1 s=CLOSED pgs=594 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:29:38.710 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:29:38.710+0000 7f825dffb640 0 client.15936 ms_handle_reset on v2:172.21.15.119:6838/937297340 2024-04-23T10:29:38.710 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:29:38.710+0000 7f827f3a4740 1 --2- 172.21.15.114:0/1849780824 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x557f01455db0 0x557f01460e00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:29:38.710 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:29:38.710+0000 7f827f3a4740 1 --2- 172.21.15.114:0/1849780824 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x557f01455490 0x557f01468a30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:29:38.710 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:29:38.710+0000 7f827f3a4740 1 -- 172.21.15.114:0/1849780824 >> 172.21.15.114:0/1849780824 conn(0x557f014517b0 msgr2=0x557f01456be0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T10:29:38.711 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:29:38.710+0000 7f827f3a4740 1 -- 172.21.15.114:0/1849780824 shutdown_connections 2024-04-23T10:29:38.711 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:29:38.710+0000 7f827f3a4740 1 -- 172.21.15.114:0/1849780824 wait complete. 2024-04-23T10:29:38.711 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:29:38.710+0000 7f827f3a4740 20 client.15936 trim_cache size 0 max 16384 2024-04-23T10:29:38.711 INFO:teuthology.orchestra.run.smithi114.stdout:{ 2024-04-23T10:29:38.711 INFO:teuthology.orchestra.run.smithi114.stdout: "epoch": 23, 2024-04-23T10:29:38.711 INFO:teuthology.orchestra.run.smithi114.stdout: "leader": 24545, 2024-04-23T10:29:38.711 INFO:teuthology.orchestra.run.smithi114.stdout: "set_version": 12, 2024-04-23T10:29:38.711 INFO:teuthology.orchestra.run.smithi114.stdout: "sets": { 2024-04-23T10:29:38.711 INFO:teuthology.orchestra.run.smithi114.stdout: "fec54601": { 2024-04-23T10:29:38.711 INFO:teuthology.orchestra.run.smithi114.stdout: "version": 12, 2024-04-23T10:29:38.712 INFO:teuthology.orchestra.run.smithi114.stdout: "age_ref": 0.0, 2024-04-23T10:29:38.712 INFO:teuthology.orchestra.run.smithi114.stdout: "state": { 2024-04-23T10:29:38.712 INFO:teuthology.orchestra.run.smithi114.stdout: "name": "RELEASED", 2024-04-23T10:29:38.712 INFO:teuthology.orchestra.run.smithi114.stdout: "age": 0.0 2024-04-23T10:29:38.712 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-23T10:29:38.712 INFO:teuthology.orchestra.run.smithi114.stdout: "timeout": 60.0, 2024-04-23T10:29:38.712 INFO:teuthology.orchestra.run.smithi114.stdout: "expiration": 81.5, 2024-04-23T10:29:38.712 INFO:teuthology.orchestra.run.smithi114.stdout: "members": { 2024-04-23T10:29:38.712 INFO:teuthology.orchestra.run.smithi114.stdout: "file:/": { 2024-04-23T10:29:38.712 INFO:teuthology.orchestra.run.smithi114.stdout: "excluded": false, 2024-04-23T10:29:38.712 INFO:teuthology.orchestra.run.smithi114.stdout: "state": { 2024-04-23T10:29:38.712 INFO:teuthology.orchestra.run.smithi114.stdout: "name": "RELEASED", 2024-04-23T10:29:38.712 INFO:teuthology.orchestra.run.smithi114.stdout: "age": 0.0 2024-04-23T10:29:38.713 INFO:teuthology.orchestra.run.smithi114.stdout: } 2024-04-23T10:29:38.713 INFO:teuthology.orchestra.run.smithi114.stdout: } 2024-04-23T10:29:38.713 INFO:teuthology.orchestra.run.smithi114.stdout: } 2024-04-23T10:29:38.713 INFO:teuthology.orchestra.run.smithi114.stdout: } 2024-04-23T10:29:38.713 INFO:teuthology.orchestra.run.smithi114.stdout: } 2024-04-23T10:29:38.713 INFO:teuthology.orchestra.run.smithi114.stdout:} 2024-04-23T10:29:38.714 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:29:38.711+0000 7f827c68a640 1 -- 172.21.15.114:0/2087320188 >> [v2:172.21.15.114:6800/1395693466,v1:172.21.15.114:6801/1395693466] conn(0x7f82600a3d50 msgr2=0x7f82600a6210 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T10:29:38.714 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:29:38.711+0000 7f827c68a640 1 --2- 172.21.15.114:0/2087320188 >> [v2:172.21.15.114:6800/1395693466,v1:172.21.15.114:6801/1395693466] conn(0x7f82600a3d50 0x7f82600a6210 secure :-1 s=READY pgs=86 cs=0 l=1 rev1=1 crypto rx=0x7f826c005e30 tx=0x7f826c0095c0 comp rx=0 tx=0).stop 2024-04-23T10:29:38.714 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:29:38.711+0000 7f827c68a640 1 -- 172.21.15.114:0/2087320188 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f82741010a0 msgr2=0x7f8274102b30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T10:29:38.714 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:29:38.711+0000 7f827c68a640 1 --2- 172.21.15.114:0/2087320188 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f82741010a0 0x7f8274102b30 secure :-1 s=READY pgs=590 cs=0 l=1 rev1=1 crypto rx=0x7f827000d950 tx=0x7f827000de10 comp rx=0 tx=0).stop 2024-04-23T10:29:38.714 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:29:38.711+0000 7f827c68a640 1 -- 172.21.15.114:0/2087320188 shutdown_connections 2024-04-23T10:29:38.714 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:29:38.711+0000 7f827c68a640 1 --2- 172.21.15.114:0/2087320188 >> [v2:172.21.15.114:6800/1395693466,v1:172.21.15.114:6801/1395693466] conn(0x7f82600a3d50 0x7f82600a6210 unknown :-1 s=CLOSED pgs=86 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:29:38.714 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:29:38.711+0000 7f827c68a640 1 --2- 172.21.15.114:0/2087320188 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f82741035b0 0x7f82741a68c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:29:38.714 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:29:38.711+0000 7f827c68a640 1 --2- 172.21.15.114:0/2087320188 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f8274106170 0x7f8274103070 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:29:38.714 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:29:38.711+0000 7f827c68a640 1 --2- 172.21.15.114:0/2087320188 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f82741010a0 0x7f8274102b30 unknown :-1 s=CLOSED pgs=590 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:29:38.715 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:29:38.711+0000 7f827c68a640 1 -- 172.21.15.114:0/2087320188 >> 172.21.15.114:0/2087320188 conn(0x7f8274077eb0 msgr2=0x7f827406a830 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T10:29:38.715 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:29:38.711+0000 7f827c68a640 1 -- 172.21.15.114:0/2087320188 shutdown_connections 2024-04-23T10:29:38.715 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:29:38.711+0000 7f827c68a640 1 -- 172.21.15.114:0/2087320188 wait complete. 2024-04-23T10:29:38.720 INFO:tasks.quiescer.fs.[cephfs]:Successfully released set 'fec54601', seconds elapsed: 22.6 2024-04-23T10:29:38.720 DEBUG:tasks.quiescer.fs.[cephfs]:Going to quiesce for duration: 21.45 2024-04-23T10:29:38.721 DEBUG:teuthology.orchestra.run.smithi114:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-23T10:29:38.826 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:29:38.824+0000 7f7609dad640 1 -- 172.21.15.114:0/719042343 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f7604103b00 msgr2=0x7f760410bfe0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T10:29:38.826 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:29:38.824+0000 7f7609dad640 1 --2- 172.21.15.114:0/719042343 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f7604103b00 0x7f760410bfe0 secure :-1 s=READY pgs=595 cs=0 l=1 rev1=1 crypto rx=0x7f75f8009900 tx=0x7f75f802a510 comp rx=0 tx=0).stop 2024-04-23T10:29:38.827 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:29:38.824+0000 7f7609dad640 1 -- 172.21.15.114:0/719042343 shutdown_connections 2024-04-23T10:29:38.827 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:29:38.825+0000 7f7609dad640 1 --2- 172.21.15.114:0/719042343 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f760410c520 0x7f760410e9c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:29:38.827 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:29:38.825+0000 7f7609dad640 1 --2- 172.21.15.114:0/719042343 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f7604103b00 0x7f760410bfe0 unknown :-1 s=CLOSED pgs=595 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:29:38.827 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:29:38.825+0000 7f7609dad640 1 --2- 172.21.15.114:0/719042343 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f76041031e0 0x7f76041035c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:29:38.827 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:29:38.825+0000 7f7609dad640 1 -- 172.21.15.114:0/719042343 >> 172.21.15.114:0/719042343 conn(0x7f7604077f50 msgr2=0x7f7604078360 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T10:29:38.827 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:29:38.825+0000 7f7609dad640 1 -- 172.21.15.114:0/719042343 shutdown_connections 2024-04-23T10:29:38.827 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:29:38.825+0000 7f7609dad640 1 -- 172.21.15.114:0/719042343 wait complete. 2024-04-23T10:29:38.827 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:29:38.825+0000 7f7609dad640 1 Processor -- start 2024-04-23T10:29:38.827 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:29:38.825+0000 7f7609dad640 1 -- start start 2024-04-23T10:29:38.827 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:29:38.826+0000 7f7609dad640 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f76041031e0 0x7f7604198750 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T10:29:38.828 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:29:38.826+0000 7f7609dad640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f7604103b00 0x7f7604198c90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T10:29:38.828 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:29:38.826+0000 7f7609dad640 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f760410c520 0x7f760419d010 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T10:29:38.828 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:29:38.826+0000 7f7609dad640 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x7f7604112190 con 0x7f7604103b00 2024-04-23T10:29:38.828 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:29:38.826+0000 7f7609dad640 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 -- 0x7f7604112010 con 0x7f76041031e0 2024-04-23T10:29:38.828 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:29:38.826+0000 7f7609dad640 1 -- --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_getmap magic: 0 -- 0x7f7604112310 con 0x7f760410c520 2024-04-23T10:29:38.828 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:29:38.826+0000 7f7602ffd640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f7604103b00 0x7f7604198c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T10:29:38.828 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:29:38.826+0000 7f7602ffd640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f7604103b00 0x7f7604198c90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.114:49042/0 (socket says 172.21.15.114:49042) 2024-04-23T10:29:38.828 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:29:38.826+0000 7f7602ffd640 1 -- 172.21.15.114:0/2866484889 learned_addr learned my addr 172.21.15.114:0/2866484889 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-04-23T10:29:38.828 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:29:38.826+0000 7f76037fe640 1 --2- 172.21.15.114:0/2866484889 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f76041031e0 0x7f7604198750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T10:29:38.828 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:29:38.826+0000 7f7603fff640 1 --2- 172.21.15.114:0/2866484889 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f760410c520 0x7f760419d010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T10:29:38.828 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:29:38.826+0000 7f7602ffd640 1 -- 172.21.15.114:0/2866484889 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f760410c520 msgr2=0x7f760419d010 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T10:29:38.828 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:29:38.826+0000 7f7602ffd640 1 --2- 172.21.15.114:0/2866484889 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f760410c520 0x7f760419d010 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:29:38.829 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:29:38.826+0000 7f7602ffd640 1 -- 172.21.15.114:0/2866484889 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f76041031e0 msgr2=0x7f7604198750 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T10:29:38.829 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:29:38.826+0000 7f7602ffd640 1 --2- 172.21.15.114:0/2866484889 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f76041031e0 0x7f7604198750 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:29:38.829 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:29:38.826+0000 7f7602ffd640 1 -- 172.21.15.114:0/2866484889 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f760419d740 con 0x7f7604103b00 2024-04-23T10:29:38.829 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:29:38.826+0000 7f7603fff640 1 --2- 172.21.15.114:0/2866484889 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f760410c520 0x7f760419d010 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-23T10:29:38.829 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:29:38.826+0000 7f7602ffd640 1 --2- 172.21.15.114:0/2866484889 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f7604103b00 0x7f7604198c90 secure :-1 s=READY pgs=596 cs=0 l=1 rev1=1 crypto rx=0x7f75f8005940 tx=0x7f75f8004270 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T10:29:38.829 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:29:38.826+0000 7f7600ff9640 1 -- 172.21.15.114:0/2866484889 <== mon.0 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f75f803f030 con 0x7f7604103b00 2024-04-23T10:29:38.829 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:29:38.826+0000 7f7600ff9640 1 -- 172.21.15.114:0/2866484889 <== mon.0 v2:172.21.15.114:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f75f8003aa0 con 0x7f7604103b00 2024-04-23T10:29:38.829 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:29:38.826+0000 7f7609dad640 1 -- 172.21.15.114:0/2866484889 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f760419d9d0 con 0x7f7604103b00 2024-04-23T10:29:38.829 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:29:38.826+0000 7f7600ff9640 1 -- 172.21.15.114:0/2866484889 <== mon.0 v2:172.21.15.114:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f75f8035070 con 0x7f7604103b00 2024-04-23T10:29:38.829 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:29:38.826+0000 7f7609dad640 1 -- 172.21.15.114:0/2866484889 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f76041a52c0 con 0x7f7604103b00 2024-04-23T10:29:38.829 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:29:38.828+0000 7f7609dad640 1 -- 172.21.15.114:0/2866484889 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f7604068d70 con 0x7f7604103b00 2024-04-23T10:29:38.830 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:29:38.829+0000 7f7600ff9640 1 -- 172.21.15.114:0/2866484889 <== mon.0 v2:172.21.15.114:3300/0 4 ==== mgrmap(e 26) ==== 138778+0+0 (secure 0 0 0) 0x7f75f8042030 con 0x7f7604103b00 2024-04-23T10:29:38.830 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:29:38.829+0000 7f7600ff9640 1 --2- 172.21.15.114:0/2866484889 >> [v2:172.21.15.114:6800/1395693466,v1:172.21.15.114:6801/1395693466] conn(0x7f75ec0a3d20 0x7f75ec0a61e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T10:29:38.830 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:29:38.829+0000 7f76037fe640 1 --2- 172.21.15.114:0/2866484889 >> [v2:172.21.15.114:6800/1395693466,v1:172.21.15.114:6801/1395693466] conn(0x7f75ec0a3d20 0x7f75ec0a61e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T10:29:38.830 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:29:38.829+0000 7f7600ff9640 1 -- 172.21.15.114:0/2866484889 <== mon.0 v2:172.21.15.114:3300/0 5 ==== osd_map(92..92 src has 1..92) ==== 8585+0+0 (secure 0 0 0) 0x7f75f80f10c0 con 0x7f7604103b00 2024-04-23T10:29:38.831 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:29:38.830+0000 7f76037fe640 1 --2- 172.21.15.114:0/2866484889 >> [v2:172.21.15.114:6800/1395693466,v1:172.21.15.114:6801/1395693466] conn(0x7f75ec0a3d20 0x7f75ec0a61e0 secure :-1 s=READY pgs=87 cs=0 l=1 rev1=1 crypto rx=0x7f75f4009bd0 tx=0x7f75f4005f90 comp rx=0 tx=0).ready entity=mgr.15291 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T10:29:38.831 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:29:38.833+0000 7f7600ff9640 1 -- 172.21.15.114:0/2866484889 <== mon.0 v2:172.21.15.114:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f75f80bd720 con 0x7f7604103b00 2024-04-23T10:29:38.912 INFO:tasks.workunit.client.0.smithi114.stdout: 1 177755 9.13 MB/sec warmup 107 sec latency 22019.147 ms 2024-04-23T10:29:39.078 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:29:39.079+0000 7f7609dad640 1 -- 172.21.15.114:0/2866484889 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f76041076d0 con 0x7f7604103b00 2024-04-23T10:29:39.079 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:29:39.080+0000 7f7600ff9640 1 -- 172.21.15.114:0/2866484889 <== mon.0 v2:172.21.15.114:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 23 v23) ==== 94+0+11888 (secure 0 0 0) 0x7f75f80bd020 con 0x7f7604103b00 2024-04-23T10:29:39.079 INFO:teuthology.orchestra.run.smithi114.stdout: 2024-04-23T10:29:39.079 INFO:teuthology.orchestra.run.smithi114.stdout:{"epoch":23,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14547,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.114:6835/663493580","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6834","nonce":663493580},{"type":"v1","addr":"172.21.15.114:6835","nonce":663493580}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14565,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.114:6837/1439252154","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6836","nonce":1439252154},{"type":"v1","addr":"172.21.15.114:6837","nonce":1439252154}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5},{"gid":14634,"name":"g","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.114:6839/3849796939","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6838","nonce":3849796939},{"type":"v1","addr":"172.21.15.114:6839","nonce":3849796939}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":11},{"gid":14646,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.114:6841/3544523891","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6840","nonce":3544523891},{"type":"v1","addr":"172.21.15.114:6841","nonce":3544523891}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":12},{"gid":24385,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.204:6833/3866561754","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6832","nonce":3866561754},{"type":"v1","addr":"172.21.15.204:6833","nonce":3866561754}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2},{"gid":24427,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.204:6835/2533793172","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6834","nonce":2533793172},{"type":"v1","addr":"172.21.15.204:6835","nonce":2533793172}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":7}],"filesystems":[{"mdsmap":{"epoch":23,"flags":306,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":true},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-04-23T10:24:43.383158+0000","modified":"2024-04-23T10:28:39.149342+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":3,"in":[0,1,2],"up":{"mds_0":24545,"mds_1":24509,"mds_2":24479},"failed":[],"damaged":[],"stopped":[],"info":{"gid_24439":{"gid":24439,"name":"c","rank":2,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.204:6837/513822179","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6836","nonce":513822179},{"type":"v1","addr":"172.21.15.204:6837","nonce":513822179}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24443":{"gid":24443,"name":"k","rank":1,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.119:6833/783244660","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6832","nonce":783244660},{"type":"v1","addr":"172.21.15.119:6833","nonce":783244660}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24463":{"gid":24463,"name":"i","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.204:6839/3222658893","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6838","nonce":3222658893},{"type":"v1","addr":"172.21.15.204:6839","nonce":3222658893}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24479":{"gid":24479,"name":"e","rank":2,"incarnation":19,"state":"up:active","state_seq":9,"addr":"172.21.15.119:6835/2033916709","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6834","nonce":2033916709},{"type":"v1","addr":"172.21.15.119:6835","nonce":2033916709}]},"join_fscid":-1,"export_targets":[0],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24509":{"gid":24509,"name":"h","rank":1,"incarnation":17,"state":"up:active","state_seq":6,"addr":"172.21.15.119:6837/1872363128","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6836","nonce":1872363128},{"type":"v1","addr":"172.21.15.119:6837","nonce":1872363128}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24545":{"gid":24545,"name":"b","rank":0,"incarnation":14,"state":"up:active","state_seq":3,"addr":"172.21.15.119:6839/937297340","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6838","nonce":937297340},{"type":"v1","addr":"172.21.15.119:6839","nonce":937297340}]},"join_fscid":-1,"export_targets":[1,2],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":24545,"qdb_cluster":[24509,24545,24479]},"id":1}]} 2024-04-23T10:29:39.079 INFO:teuthology.orchestra.run.smithi114.stderr:dumped fsmap epoch 23 2024-04-23T10:29:39.081 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:29:39.082+0000 7f7609dad640 1 -- 172.21.15.114:0/2866484889 >> [v2:172.21.15.114:6800/1395693466,v1:172.21.15.114:6801/1395693466] conn(0x7f75ec0a3d20 msgr2=0x7f75ec0a61e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T10:29:39.081 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:29:39.082+0000 7f7609dad640 1 --2- 172.21.15.114:0/2866484889 >> [v2:172.21.15.114:6800/1395693466,v1:172.21.15.114:6801/1395693466] conn(0x7f75ec0a3d20 0x7f75ec0a61e0 secure :-1 s=READY pgs=87 cs=0 l=1 rev1=1 crypto rx=0x7f75f4009bd0 tx=0x7f75f4005f90 comp rx=0 tx=0).stop 2024-04-23T10:29:39.081 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:29:39.082+0000 7f7609dad640 1 -- 172.21.15.114:0/2866484889 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f7604103b00 msgr2=0x7f7604198c90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T10:29:39.081 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:29:39.082+0000 7f7609dad640 1 --2- 172.21.15.114:0/2866484889 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f7604103b00 0x7f7604198c90 secure :-1 s=READY pgs=596 cs=0 l=1 rev1=1 crypto rx=0x7f75f8005940 tx=0x7f75f8004270 comp rx=0 tx=0).stop 2024-04-23T10:29:39.082 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:29:39.082+0000 7f7609dad640 1 -- 172.21.15.114:0/2866484889 shutdown_connections 2024-04-23T10:29:39.082 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:29:39.082+0000 7f7609dad640 1 --2- 172.21.15.114:0/2866484889 >> [v2:172.21.15.114:6800/1395693466,v1:172.21.15.114:6801/1395693466] conn(0x7f75ec0a3d20 0x7f75ec0a61e0 unknown :-1 s=CLOSED pgs=87 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:29:39.082 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:29:39.082+0000 7f7609dad640 1 --2- 172.21.15.114:0/2866484889 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f760410c520 0x7f760419d010 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:29:39.082 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:29:39.082+0000 7f7609dad640 1 --2- 172.21.15.114:0/2866484889 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f7604103b00 0x7f7604198c90 unknown :-1 s=CLOSED pgs=596 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:29:39.082 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:29:39.082+0000 7f7609dad640 1 --2- 172.21.15.114:0/2866484889 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f76041031e0 0x7f7604198750 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:29:39.082 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:29:39.082+0000 7f7609dad640 1 -- 172.21.15.114:0/2866484889 >> 172.21.15.114:0/2866484889 conn(0x7f7604077f50 msgr2=0x7f76040fdfc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T10:29:39.082 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:29:39.082+0000 7f7609dad640 1 -- 172.21.15.114:0/2866484889 shutdown_connections 2024-04-23T10:29:39.082 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:29:39.082+0000 7f7609dad640 1 -- 172.21.15.114:0/2866484889 wait complete. 2024-04-23T10:29:39.092 DEBUG:tasks.quiescer.fs.[cephfs]:Running ceph command: 'tell mds.24545 quiesce db / --timeout 60 --expiration 81.45 --await-for 42.0' 2024-04-23T10:29:39.092 DEBUG:teuthology.orchestra.run.smithi114:> timeout 75 sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.24545 quiesce db / --timeout 60 --expiration 81.45 --await-for 42.0 2024-04-23T10:29:39.214 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:29:39.213+0000 7f3dc6ba2640 1 -- 172.21.15.114:0/1321844292 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f3dc0103ad0 msgr2=0x7f3dc010bfb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T10:29:39.214 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:29:39.213+0000 7f3dc6ba2640 1 --2- 172.21.15.114:0/1321844292 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f3dc0103ad0 0x7f3dc010bfb0 secure :-1 s=READY pgs=597 cs=0 l=1 rev1=1 crypto rx=0x7f3dac009900 tx=0x7f3dac02a480 comp rx=0 tx=0).stop 2024-04-23T10:29:39.214 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:29:39.213+0000 7f3dc6ba2640 1 -- 172.21.15.114:0/1321844292 shutdown_connections 2024-04-23T10:29:39.214 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:29:39.213+0000 7f3dc6ba2640 1 --2- 172.21.15.114:0/1321844292 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f3dc010c580 0x7f3dc010e9b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:29:39.214 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:29:39.213+0000 7f3dc6ba2640 1 --2- 172.21.15.114:0/1321844292 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f3dc0103ad0 0x7f3dc010bfb0 unknown :-1 s=CLOSED pgs=597 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:29:39.214 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:29:39.213+0000 7f3dc6ba2640 1 --2- 172.21.15.114:0/1321844292 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f3dc01031b0 0x7f3dc0103590 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:29:39.215 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:29:39.213+0000 7f3dc6ba2640 1 -- 172.21.15.114:0/1321844292 >> 172.21.15.114:0/1321844292 conn(0x7f3dc00771d0 msgr2=0x7f3dc00775e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T10:29:39.215 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:29:39.213+0000 7f3dc6ba2640 1 -- 172.21.15.114:0/1321844292 shutdown_connections 2024-04-23T10:29:39.215 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:29:39.213+0000 7f3dc6ba2640 1 -- 172.21.15.114:0/1321844292 wait complete. 2024-04-23T10:29:39.215 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:29:39.214+0000 7f3dc6ba2640 1 Processor -- start 2024-04-23T10:29:39.215 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:29:39.214+0000 7f3dc6ba2640 1 -- start start 2024-04-23T10:29:39.215 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:29:39.214+0000 7f3dc6ba2640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f3dc01031b0 0x7f3dc019e420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T10:29:39.215 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:29:39.214+0000 7f3dc6ba2640 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f3dc0103ad0 0x7f3dc019e960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T10:29:39.215 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:29:39.214+0000 7f3dc6ba2640 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f3dc010c580 0x7f3dc01985d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T10:29:39.215 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:29:39.214+0000 7f3dc6ba2640 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x7f3dc01121c0 con 0x7f3dc01031b0 2024-04-23T10:29:39.215 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:29:39.214+0000 7f3dc6ba2640 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 -- 0x7f3dc0112040 con 0x7f3dc0103ad0 2024-04-23T10:29:39.215 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:29:39.214+0000 7f3dc6ba2640 1 -- --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_getmap magic: 0 -- 0x7f3dc0112340 con 0x7f3dc010c580 2024-04-23T10:29:39.215 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:29:39.214+0000 7f3dc493e640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f3dc01031b0 0x7f3dc019e420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T10:29:39.215 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:29:39.214+0000 7f3dc493e640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f3dc01031b0 0x7f3dc019e420 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.114:49070/0 (socket says 172.21.15.114:49070) 2024-04-23T10:29:39.215 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:29:39.214+0000 7f3dc493e640 1 -- 172.21.15.114:0/1271895219 learned_addr learned my addr 172.21.15.114:0/1271895219 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-04-23T10:29:39.216 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:29:39.214+0000 7f3dbffff640 1 --2- 172.21.15.114:0/1271895219 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f3dc0103ad0 0x7f3dc019e960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T10:29:39.216 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:29:39.214+0000 7f3dc513f640 1 --2- 172.21.15.114:0/1271895219 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f3dc010c580 0x7f3dc01985d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T10:29:39.216 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:29:39.214+0000 7f3dc493e640 1 -- 172.21.15.114:0/1271895219 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f3dc010c580 msgr2=0x7f3dc01985d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T10:29:39.216 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:29:39.214+0000 7f3dc493e640 1 --2- 172.21.15.114:0/1271895219 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f3dc010c580 0x7f3dc01985d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:29:39.216 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:29:39.214+0000 7f3dc493e640 1 -- 172.21.15.114:0/1271895219 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f3dc0103ad0 msgr2=0x7f3dc019e960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T10:29:39.216 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:29:39.214+0000 7f3dc493e640 1 --2- 172.21.15.114:0/1271895219 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f3dc0103ad0 0x7f3dc019e960 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:29:39.216 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:29:39.214+0000 7f3dc493e640 1 -- 172.21.15.114:0/1271895219 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f3dc0198d50 con 0x7f3dc01031b0 2024-04-23T10:29:39.216 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:29:39.214+0000 7f3dc493e640 1 --2- 172.21.15.114:0/1271895219 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f3dc01031b0 0x7f3dc019e420 secure :-1 s=READY pgs=598 cs=0 l=1 rev1=1 crypto rx=0x7f3db400e9f0 tx=0x7f3db400edf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T10:29:39.216 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:29:39.215+0000 7f3dbdffb640 1 -- 172.21.15.114:0/1271895219 <== mon.0 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3db400cd80 con 0x7f3dc01031b0 2024-04-23T10:29:39.216 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:29:39.215+0000 7f3dc6ba2640 1 -- 172.21.15.114:0/1271895219 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f3dc0199040 con 0x7f3dc01031b0 2024-04-23T10:29:39.216 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:29:39.215+0000 7f3dbdffb640 1 -- 172.21.15.114:0/1271895219 <== mon.0 v2:172.21.15.114:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f3db40054d0 con 0x7f3dc01031b0 2024-04-23T10:29:39.216 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:29:39.215+0000 7f3dbdffb640 1 -- 172.21.15.114:0/1271895219 <== mon.0 v2:172.21.15.114:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3db4010690 con 0x7f3dc01031b0 2024-04-23T10:29:39.216 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:29:39.215+0000 7f3dc6ba2640 1 -- 172.21.15.114:0/1271895219 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f3dc01a52e0 con 0x7f3dc01031b0 2024-04-23T10:29:39.223 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:29:39.217+0000 7f3dbdffb640 1 -- 172.21.15.114:0/1271895219 <== mon.0 v2:172.21.15.114:3300/0 4 ==== mgrmap(e 26) ==== 138778+0+0 (secure 0 0 0) 0x7f3db4004e90 con 0x7f3dc01031b0 2024-04-23T10:29:39.223 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:29:39.217+0000 7f3dc9998740 1 Processor -- start 2024-04-23T10:29:39.223 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:29:39.217+0000 7f3dc9998740 1 -- start start 2024-04-23T10:29:39.223 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:29:39.218+0000 7f3dc9998740 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x556fa9ea80a0 0x556fa9ea8480 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T10:29:39.223 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:29:39.218+0000 7f3dc9998740 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x556fa9ea89c0 0x556fa9eb2b50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T10:29:39.223 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:29:39.218+0000 7f3dc9998740 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x556fa9eb3090 0x556fa9eb5480 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T10:29:39.223 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:29:39.218+0000 7f3dc9998740 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x556fa9e0d370 con 0x556fa9ea80a0 2024-04-23T10:29:39.223 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:29:39.218+0000 7f3dc9998740 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 -- 0x556fa9daa2e0 con 0x556fa9ea89c0 2024-04-23T10:29:39.223 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:29:39.218+0000 7f3dc9998740 1 -- --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_getmap magic: 0 -- 0x556fa9e09330 con 0x556fa9eb3090 2024-04-23T10:29:39.223 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:29:39.218+0000 7f3dc513f640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x556fa9ea80a0 0x556fa9ea8480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T10:29:39.223 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:29:39.218+0000 7f3dc513f640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x556fa9ea80a0 0x556fa9ea8480 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.114:49074/0 (socket says 172.21.15.114:49074) 2024-04-23T10:29:39.224 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:29:39.218+0000 7f3dc513f640 1 -- 172.21.15.114:0/828570980 learned_addr learned my addr 172.21.15.114:0/828570980 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-04-23T10:29:39.224 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:29:39.218+0000 7f3dc493e640 1 --2- 172.21.15.114:0/828570980 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x556fa9ea89c0 0x556fa9eb2b50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T10:29:39.224 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:29:39.218+0000 7f3dbffff640 1 --2- 172.21.15.114:0/828570980 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x556fa9eb3090 0x556fa9eb5480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T10:29:39.224 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:29:39.218+0000 7f3dc513f640 1 -- 172.21.15.114:0/828570980 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x556fa9eb3090 msgr2=0x556fa9eb5480 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T10:29:39.224 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:29:39.218+0000 7f3dc513f640 1 --2- 172.21.15.114:0/828570980 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x556fa9eb3090 0x556fa9eb5480 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:29:39.224 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:29:39.218+0000 7f3dc513f640 1 -- 172.21.15.114:0/828570980 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x556fa9ea89c0 msgr2=0x556fa9eb2b50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T10:29:39.224 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:29:39.218+0000 7f3dc513f640 1 --2- 172.21.15.114:0/828570980 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x556fa9ea89c0 0x556fa9eb2b50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:29:39.224 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:29:39.218+0000 7f3dc513f640 1 -- 172.21.15.114:0/828570980 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x556fa9bebd80 con 0x556fa9ea80a0 2024-04-23T10:29:39.224 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:29:39.218+0000 7f3dc493e640 1 --2- 172.21.15.114:0/828570980 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x556fa9ea89c0 0x556fa9eb2b50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-23T10:29:39.224 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:29:39.218+0000 7f3dbdffb640 1 --2- 172.21.15.114:0/1271895219 >> [v2:172.21.15.114:6800/1395693466,v1:172.21.15.114:6801/1395693466] conn(0x7f3db00a3d20 0x7f3db00a61e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T10:29:39.224 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:29:39.218+0000 7f3dc513f640 1 --2- 172.21.15.114:0/828570980 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x556fa9ea80a0 0x556fa9ea8480 secure :-1 s=READY pgs=599 cs=0 l=1 rev1=1 crypto rx=0x7f3db8009f30 tx=0x7f3db80118d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e6bd55c795a3a0fc server_cookie=0 in_seq=0 out_seq=0 2024-04-23T10:29:39.224 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:29:39.218+0000 7f3dc493e640 1 --2- 172.21.15.114:0/1271895219 >> [v2:172.21.15.114:6800/1395693466,v1:172.21.15.114:6801/1395693466] conn(0x7f3db00a3d20 0x7f3db00a61e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T10:29:39.224 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:29:39.219+0000 7f3daaffd640 1 -- 172.21.15.114:0/828570980 <== mon.0 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3db8004d20 con 0x556fa9ea80a0 2024-04-23T10:29:39.224 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:29:39.219+0000 7f3daaffd640 1 -- 172.21.15.114:0/828570980 <== mon.0 v2:172.21.15.114:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f3db8004050 con 0x556fa9ea80a0 2024-04-23T10:29:39.225 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:29:39.219+0000 7f3dbdffb640 1 -- 172.21.15.114:0/1271895219 <== mon.0 v2:172.21.15.114:3300/0 5 ==== osd_map(92..92 src has 1..92) ==== 8585+0+0 (secure 0 0 0) 0x7f3db40d0c40 con 0x7f3dc01031b0 2024-04-23T10:29:39.225 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:29:39.219+0000 7f3daaffd640 1 -- 172.21.15.114:0/828570980 <== mon.0 v2:172.21.15.114:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3db8005260 con 0x556fa9ea80a0 2024-04-23T10:29:39.225 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:29:39.219+0000 7f3dc9998740 1 -- 172.21.15.114:0/828570980 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x556fa9ea80a0 msgr2=0x556fa9ea8480 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T10:29:39.225 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:29:39.219+0000 7f3dc9998740 1 --2- 172.21.15.114:0/828570980 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x556fa9ea80a0 0x556fa9ea8480 secure :-1 s=READY pgs=599 cs=0 l=1 rev1=1 crypto rx=0x7f3db8009f30 tx=0x7f3db80118d0 comp rx=0 tx=0).stop 2024-04-23T10:29:39.225 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:29:39.219+0000 7f3dc493e640 1 --2- 172.21.15.114:0/1271895219 >> [v2:172.21.15.114:6800/1395693466,v1:172.21.15.114:6801/1395693466] conn(0x7f3db00a3d20 0x7f3db00a61e0 secure :-1 s=READY pgs=88 cs=0 l=1 rev1=1 crypto rx=0x7f3db40ce400 tx=0x7f3db406fe10 comp rx=0 tx=0).ready entity=mgr.15291 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T10:29:39.225 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:29:39.219+0000 7f3dc9998740 1 -- 172.21.15.114:0/828570980 shutdown_connections 2024-04-23T10:29:39.225 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:29:39.219+0000 7f3dc9998740 1 --2- 172.21.15.114:0/828570980 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x556fa9eb3090 0x556fa9eb5480 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:29:39.225 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:29:39.219+0000 7f3dc9998740 1 --2- 172.21.15.114:0/828570980 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x556fa9ea89c0 0x556fa9eb2b50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:29:39.225 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:29:39.219+0000 7f3dc9998740 1 --2- 172.21.15.114:0/828570980 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x556fa9ea80a0 0x556fa9ea8480 unknown :-1 s=CLOSED pgs=599 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:29:39.225 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:29:39.219+0000 7f3dc9998740 1 -- 172.21.15.114:0/828570980 >> 172.21.15.114:0/828570980 conn(0x556fa9ea43c0 msgr2=0x556fa9ea47a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T10:29:39.225 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:29:39.220+0000 7f3dc9998740 1 -- 172.21.15.114:0/828570980 shutdown_connections 2024-04-23T10:29:39.225 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:29:39.220+0000 7f3dc9998740 1 -- 172.21.15.114:0/828570980 wait complete. 2024-04-23T10:29:39.225 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:29:39.220+0000 7f3dc9998740 1 -- start start 2024-04-23T10:29:39.225 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:29:39.220+0000 7f3dc9998740 1 Processor -- start 2024-04-23T10:29:39.225 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:29:39.220+0000 7f3dc9998740 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x556fa9ea80a0 0x556fa9eb0000 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T10:29:39.226 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:29:39.220+0000 7f3dc9998740 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x556fa9ea89c0 0x556fa9eb0540 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T10:29:39.226 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:29:39.220+0000 7f3dc9998740 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x556fa9eb3090 0x556fa9eb0a80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T10:29:39.226 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:29:39.220+0000 7f3dc9998740 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x556fa9eb5e80 con 0x556fa9ea89c0 2024-04-23T10:29:39.226 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:29:39.220+0000 7f3dc9998740 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 -- 0x556fa9da7fb0 con 0x556fa9ea80a0 2024-04-23T10:29:39.226 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:29:39.220+0000 7f3dc9998740 1 -- --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_getmap magic: 0 -- 0x556fa9eb6000 con 0x556fa9eb3090 2024-04-23T10:29:39.226 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:29:39.220+0000 7f3dc9998740 10 client.0 _ll_register_callbacks cb 0x556fa9da79a0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-23T10:29:39.226 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:29:39.220+0000 7f3dbffff640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x556fa9ea89c0 0x556fa9eb0540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T10:29:39.226 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:29:39.220+0000 7f3dc513f640 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x556fa9ea80a0 0x556fa9eb0000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T10:29:39.226 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:29:39.221+0000 7f3dc513f640 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x556fa9ea80a0 0x556fa9eb0000 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.119:3300/0 says I am v2:172.21.15.114:60896/0 (socket says 172.21.15.114:60896) 2024-04-23T10:29:39.226 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:29:39.221+0000 7f3dc513f640 1 -- 172.21.15.114:0/1558815263 learned_addr learned my addr 172.21.15.114:0/1558815263 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-04-23T10:29:39.226 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:29:39.221+0000 7f3dc513f640 1 --2- 172.21.15.114:0/1558815263 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x556fa9eb3090 0x556fa9eb0a80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T10:29:39.226 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:29:39.221+0000 7f3dbffff640 1 -- 172.21.15.114:0/1558815263 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x556fa9eb3090 msgr2=0x556fa9eb0a80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T10:29:39.226 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:29:39.221+0000 7f3dbffff640 1 --2- 172.21.15.114:0/1558815263 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x556fa9eb3090 0x556fa9eb0a80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:29:39.226 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:29:39.221+0000 7f3dbffff640 1 -- 172.21.15.114:0/1558815263 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x556fa9ea80a0 msgr2=0x556fa9eb0000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T10:29:39.227 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:29:39.221+0000 7f3dbffff640 1 --2- 172.21.15.114:0/1558815263 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x556fa9ea80a0 0x556fa9eb0000 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:29:39.227 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:29:39.221+0000 7f3dbffff640 1 -- 172.21.15.114:0/1558815263 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({osdmap=0}) -- 0x556fa9c4c510 con 0x556fa9ea89c0 2024-04-23T10:29:39.227 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:29:39.221+0000 7f3dbffff640 1 --2- 172.21.15.114:0/1558815263 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x556fa9ea89c0 0x556fa9eb0540 secure :-1 s=READY pgs=600 cs=0 l=1 rev1=1 crypto rx=0x7f3dac009230 tx=0x7f3dac032dc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d7e30351925517fe server_cookie=0 in_seq=0 out_seq=0 2024-04-23T10:29:39.227 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:29:39.221+0000 7f3d9ffff640 10 client.0 ms_handle_connect on v2:172.21.15.114:3300/0 2024-04-23T10:29:39.227 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:29:39.221+0000 7f3d9ffff640 1 -- 172.21.15.114:0/1558815263 <== mon.0 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3dac035040 con 0x556fa9ea89c0 2024-04-23T10:29:39.227 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:29:39.221+0000 7f3d9ffff640 1 -- 172.21.15.114:0/1558815263 <== mon.0 v2:172.21.15.114:3300/0 2 ==== osd_map(92..92 src has 1..92) ==== 8585+0+0 (secure 0 0 0) 0x7f3dac0316a0 con 0x556fa9ea89c0 2024-04-23T10:29:39.227 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:29:39.221+0000 7f3dc9998740 1 -- 172.21.15.114:0/1558815263 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f3db8004050 con 0x556fa9ea89c0 2024-04-23T10:29:39.227 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:29:39.221+0000 7f3d9ffff640 1 -- 172.21.15.114:0/1558815263 <== mon.0 v2:172.21.15.114:3300/0 3 ==== mon_get_version_reply(handle=1 version=23) ==== 24+0+0 (secure 0 0 0) 0x7f3dac032e30 con 0x556fa9ea89c0 2024-04-23T10:29:39.227 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:29:39.221+0000 7f3dc9998740 10 client.15954 fetch_fsmap learned FSMap version 23 2024-04-23T10:29:39.227 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:29:39.221+0000 7f3dc9998740 1 -- 172.21.15.114:0/1558815263 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({config=0+,fsmap=23,monmap=4+}) -- 0x556fa9adf670 con 0x556fa9ea89c0 2024-04-23T10:29:39.227 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:29:39.222+0000 7f3d9ffff640 1 -- 172.21.15.114:0/1558815263 <== mon.0 v2:172.21.15.114:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f3db8004050 con 0x556fa9ea89c0 2024-04-23T10:29:39.227 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:29:39.222+0000 7f3d9ffff640 1 -- 172.21.15.114:0/1558815263 <== mon.0 v2:172.21.15.114:3300/0 5 ==== fsmap(e 23) ==== 8350+0+0 (secure 0 0 0) 0x7f3dac004750 con 0x556fa9ea89c0 2024-04-23T10:29:39.227 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:29:39.222+0000 7f3dc9998740 10 client.15954 fetch_fsmap finished waiting for FSMap version 23 2024-04-23T10:29:39.227 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:29:39.222+0000 7f3dc9998740 10 client.15954 resolve_mds: validated gid 24545 aka daemon mds.b 2024-04-23T10:29:39.228 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:29:39.222+0000 7f3dc9998740 20 client.15954 populate_metadata read hostname 'smithi114' 2024-04-23T10:29:39.228 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:29:39.222+0000 7f3dc9998740 1 --2- 172.21.15.114:0/1558815263 >> [v2:172.21.15.119:6838/937297340,v1:172.21.15.119:6839/937297340] conn(0x556fa9ec33e0 0x556fa9ec57d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T10:29:39.228 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:29:39.222+0000 7f3dc9998740 4 client.15954 mds_command: new command op to 24545 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-23T10:29:39.228 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:29:39.222+0000 7f3dc9998740 1 -- 172.21.15.114:0/1558815263 --> [v2:172.21.15.119:6838/937297340,v1:172.21.15.119:6839/937297340] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x556fa9e1ad90 con 0x556fa9ec33e0 2024-04-23T10:29:39.228 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:29:39.222+0000 7f3dc513f640 1 --2- 172.21.15.114:0/1558815263 >> [v2:172.21.15.119:6838/937297340,v1:172.21.15.119:6839/937297340] conn(0x556fa9ec33e0 0x556fa9ec57d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T10:29:39.228 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:29:39.222+0000 7f3dc513f640 1 --2- 172.21.15.114:0/1558815263 >> [v2:172.21.15.119:6838/937297340,v1:172.21.15.119:6839/937297340] conn(0x556fa9ec33e0 0x556fa9ec57d0 crc :-1 s=READY pgs=78 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=52a2adeea208adcd server_cookie=cfd6fb8396dc6431 in_seq=0 out_seq=0 2024-04-23T10:29:39.228 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:29:39.222+0000 7f3d9ffff640 10 client.15954 ms_handle_connect on v2:172.21.15.119:6838/937297340 2024-04-23T10:29:39.228 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:29:39.223+0000 7f3d9ffff640 1 -- 172.21.15.114:0/1558815263 <== mds.0 v2:172.21.15.119:6838/937297340 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x556fa9e1ad90 con 0x556fa9ec33e0 2024-04-23T10:29:39.228 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:29:39.223+0000 7f3d9ffff640 10 client.15954 handle_command_reply: tid=0 2024-04-23T10:29:39.228 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:29:39.223+0000 7f3dc9998740 1 client.15954 shutdown 2024-04-23T10:29:39.228 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:29:39.223+0000 7f3dc9998740 1 -- 172.21.15.114:0/1558815263 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x556fa9ea89c0 msgr2=0x556fa9eb0540 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T10:29:39.228 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:29:39.223+0000 7f3dc9998740 1 --2- 172.21.15.114:0/1558815263 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x556fa9ea89c0 0x556fa9eb0540 secure :-1 s=READY pgs=600 cs=0 l=1 rev1=1 crypto rx=0x7f3dac009230 tx=0x7f3dac032dc0 comp rx=0 tx=0).stop 2024-04-23T10:29:39.228 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:29:39.223+0000 7f3dc9998740 1 -- 172.21.15.114:0/1558815263 shutdown_connections 2024-04-23T10:29:39.228 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:29:39.223+0000 7f3dc9998740 1 --2- 172.21.15.114:0/1558815263 >> [v2:172.21.15.119:6838/937297340,v1:172.21.15.119:6839/937297340] conn(0x556fa9ec33e0 0x556fa9ec57d0 crc :-1 s=READY pgs=78 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:29:39.228 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:29:39.223+0000 7f3dc9998740 1 --2- 172.21.15.114:0/1558815263 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x556fa9eb3090 0x556fa9eb0a80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:29:39.229 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:29:39.223+0000 7f3d9ffff640 0 client.15954 ms_handle_reset on v2:172.21.15.119:6838/937297340 2024-04-23T10:29:39.229 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:29:39.223+0000 7f3dc9998740 1 --2- 172.21.15.114:0/1558815263 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x556fa9ea89c0 0x556fa9eb0540 unknown :-1 s=CLOSED pgs=600 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:29:39.229 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:29:39.223+0000 7f3dc9998740 1 --2- 172.21.15.114:0/1558815263 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x556fa9ea80a0 0x556fa9eb0000 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:29:39.229 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:29:39.223+0000 7f3dc9998740 1 -- 172.21.15.114:0/1558815263 >> 172.21.15.114:0/1558815263 conn(0x556fa9ea43c0 msgr2=0x556fa9ea3600 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T10:29:39.229 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:29:39.224+0000 7f3dc9998740 1 -- 172.21.15.114:0/1558815263 shutdown_connections 2024-04-23T10:29:39.229 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:29:39.224+0000 7f3dc9998740 1 -- 172.21.15.114:0/1558815263 wait complete. 2024-04-23T10:29:39.229 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:29:39.224+0000 7f3dc9998740 20 client.15954 trim_cache size 0 max 16384 2024-04-23T10:29:39.245 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:29:39.241+0000 7f3dc9998740 1 Processor -- start 2024-04-23T10:29:39.245 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:29:39.241+0000 7f3dc9998740 1 -- start start 2024-04-23T10:29:39.246 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:29:39.242+0000 7f3dc9998740 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x556fa9ea80a0 0x556fa9ebed40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T10:29:39.246 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:29:39.242+0000 7f3dc9998740 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x556fa9ea89c0 0x556fa9ea3d80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T10:29:39.246 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:29:39.242+0000 7f3dc9998740 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x556fa9eb3090 0x556fa9ecb0e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T10:29:39.246 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:29:39.242+0000 7f3dc9998740 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x7f3dc01b06c0 con 0x556fa9ea89c0 2024-04-23T10:29:39.246 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:29:39.242+0000 7f3dc9998740 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 -- 0x7f3dc01adf60 con 0x556fa9ea80a0 2024-04-23T10:29:39.246 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:29:39.242+0000 7f3dc9998740 1 -- --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_getmap magic: 0 -- 0x7f3dc01abdb0 con 0x556fa9eb3090 2024-04-23T10:29:39.246 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:29:39.242+0000 7f3dbffff640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x556fa9ea89c0 0x556fa9ea3d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T10:29:39.246 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:29:39.242+0000 7f3dbffff640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x556fa9ea89c0 0x556fa9ea3d80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.114:49086/0 (socket says 172.21.15.114:49086) 2024-04-23T10:29:39.246 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:29:39.242+0000 7f3dbffff640 1 -- 172.21.15.114:0/3286594475 learned_addr learned my addr 172.21.15.114:0/3286594475 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-04-23T10:29:39.246 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:29:39.242+0000 7f3dc513f640 1 --2- 172.21.15.114:0/3286594475 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x556fa9ea80a0 0x556fa9ebed40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T10:29:39.246 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:29:39.242+0000 7f3dc513f640 1 --2- 172.21.15.114:0/3286594475 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x556fa9eb3090 0x556fa9ecb0e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T10:29:39.246 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:29:39.242+0000 7f3dbffff640 1 -- 172.21.15.114:0/3286594475 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x556fa9eb3090 msgr2=0x556fa9ecb0e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T10:29:39.246 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:29:39.242+0000 7f3dbffff640 1 --2- 172.21.15.114:0/3286594475 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x556fa9eb3090 0x556fa9ecb0e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:29:39.246 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:29:39.242+0000 7f3dbffff640 1 -- 172.21.15.114:0/3286594475 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x556fa9ea80a0 msgr2=0x556fa9ebed40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T10:29:39.247 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:29:39.242+0000 7f3dbffff640 1 --2- 172.21.15.114:0/3286594475 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x556fa9ea80a0 0x556fa9ebed40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:29:39.247 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:29:39.242+0000 7f3dbffff640 1 -- 172.21.15.114:0/3286594475 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x556fa9e9e820 con 0x556fa9ea89c0 2024-04-23T10:29:39.247 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:29:39.242+0000 7f3dbffff640 1 --2- 172.21.15.114:0/3286594475 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x556fa9ea89c0 0x556fa9ea3d80 secure :-1 s=READY pgs=601 cs=0 l=1 rev1=1 crypto rx=0x7f3dac031440 tx=0x7f3dac02a480 comp rx=0 tx=0).ready entity=mon.0 client_cookie=be30f91909a2de27 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T10:29:39.247 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:29:39.242+0000 7f3daa7fc640 1 -- 172.21.15.114:0/3286594475 <== mon.0 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3dac035030 con 0x556fa9ea89c0 2024-04-23T10:29:39.247 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:29:39.242+0000 7f3daa7fc640 1 -- 172.21.15.114:0/3286594475 <== mon.0 v2:172.21.15.114:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f3dac004d90 con 0x556fa9ea89c0 2024-04-23T10:29:39.247 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:29:39.242+0000 7f3daa7fc640 1 -- 172.21.15.114:0/3286594475 <== mon.0 v2:172.21.15.114:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3dac0044e0 con 0x556fa9ea89c0 2024-04-23T10:29:39.247 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:29:39.243+0000 7f3dc9998740 1 -- 172.21.15.114:0/3286594475 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x556fa9ea89c0 msgr2=0x556fa9ea3d80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T10:29:39.247 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:29:39.243+0000 7f3dc9998740 1 --2- 172.21.15.114:0/3286594475 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x556fa9ea89c0 0x556fa9ea3d80 secure :-1 s=READY pgs=601 cs=0 l=1 rev1=1 crypto rx=0x7f3dac031440 tx=0x7f3dac02a480 comp rx=0 tx=0).stop 2024-04-23T10:29:39.247 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:29:39.243+0000 7f3dc9998740 1 -- 172.21.15.114:0/3286594475 shutdown_connections 2024-04-23T10:29:39.247 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:29:39.243+0000 7f3dc9998740 1 --2- 172.21.15.114:0/3286594475 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x556fa9eb3090 0x556fa9ecb0e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:29:39.247 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:29:39.243+0000 7f3dc9998740 1 --2- 172.21.15.114:0/3286594475 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x556fa9ea89c0 0x556fa9ea3d80 unknown :-1 s=CLOSED pgs=601 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:29:39.247 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:29:39.243+0000 7f3dc9998740 1 --2- 172.21.15.114:0/3286594475 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x556fa9ea80a0 0x556fa9ebed40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:29:39.247 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:29:39.243+0000 7f3dc9998740 1 -- 172.21.15.114:0/3286594475 >> 172.21.15.114:0/3286594475 conn(0x556fa9ea43c0 msgr2=0x556fa9eaa080 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T10:29:39.247 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:29:39.243+0000 7f3dc9998740 1 -- 172.21.15.114:0/3286594475 shutdown_connections 2024-04-23T10:29:39.248 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:29:39.243+0000 7f3dc9998740 1 -- 172.21.15.114:0/3286594475 wait complete. 2024-04-23T10:29:39.248 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:29:39.243+0000 7f3dc9998740 1 -- start start 2024-04-23T10:29:39.248 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:29:39.244+0000 7f3dc9998740 1 Processor -- start 2024-04-23T10:29:39.248 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:29:39.244+0000 7f3dc9998740 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x556fa9ea80a0 0x556fa9eacc90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T10:29:39.248 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:29:39.244+0000 7f3dc9998740 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x556fa9ea89c0 0x556fa9ead1d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T10:29:39.248 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:29:39.244+0000 7f3dc9998740 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x556fa9eb3090 0x556fa9ead710 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T10:29:39.248 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:29:39.244+0000 7f3dc9998740 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x7f3dc01ac420 con 0x556fa9ea89c0 2024-04-23T10:29:39.248 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:29:39.244+0000 7f3dc9998740 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 -- 0x7f3dc01ad890 con 0x556fa9eb3090 2024-04-23T10:29:39.248 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:29:39.244+0000 7f3dc9998740 1 -- --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_getmap magic: 0 -- 0x7f3dc01acaf0 con 0x556fa9ea80a0 2024-04-23T10:29:39.248 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:29:39.244+0000 7f3dc9998740 10 client.0 _ll_register_callbacks cb 0x7f3dac0095f0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-23T10:29:39.248 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:29:39.244+0000 7f3dbffff640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x556fa9ea89c0 0x556fa9ead1d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T10:29:39.248 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:29:39.244+0000 7f3dbffff640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x556fa9ea89c0 0x556fa9ead1d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.114:49096/0 (socket says 172.21.15.114:49096) 2024-04-23T10:29:39.248 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:29:39.244+0000 7f3dc513f640 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x556fa9ea80a0 0x556fa9eacc90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T10:29:39.248 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:29:39.244+0000 7f3dbffff640 1 -- 172.21.15.114:0/1736841201 learned_addr learned my addr 172.21.15.114:0/1736841201 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-04-23T10:29:39.248 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:29:39.244+0000 7f3dc513f640 1 --2- 172.21.15.114:0/1736841201 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x556fa9eb3090 0x556fa9ead710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T10:29:39.249 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:29:39.244+0000 7f3dbffff640 1 -- 172.21.15.114:0/1736841201 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x556fa9ea80a0 msgr2=0x556fa9eacc90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T10:29:39.249 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:29:39.244+0000 7f3dbffff640 1 --2- 172.21.15.114:0/1736841201 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x556fa9ea80a0 0x556fa9eacc90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:29:39.249 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:29:39.244+0000 7f3dbffff640 1 -- 172.21.15.114:0/1736841201 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x556fa9eb3090 msgr2=0x556fa9ead710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T10:29:39.249 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:29:39.244+0000 7f3dbffff640 1 --2- 172.21.15.114:0/1736841201 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x556fa9eb3090 0x556fa9ead710 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:29:39.249 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:29:39.244+0000 7f3dbffff640 1 -- 172.21.15.114:0/1736841201 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({osdmap=0}) -- 0x556fa9f28980 con 0x556fa9ea89c0 2024-04-23T10:29:39.249 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:29:39.245+0000 7f3dbffff640 1 --2- 172.21.15.114:0/1736841201 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x556fa9ea89c0 0x556fa9ead1d0 secure :-1 s=READY pgs=602 cs=0 l=1 rev1=1 crypto rx=0x7f3dac004d10 tx=0x7f3dac032da0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4bba3eae31ea7ca server_cookie=0 in_seq=0 out_seq=0 2024-04-23T10:29:39.249 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:29:39.245+0000 7f3d9ffff640 10 client.0 ms_handle_connect on v2:172.21.15.114:3300/0 2024-04-23T10:29:39.249 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:29:39.245+0000 7f3d9ffff640 1 -- 172.21.15.114:0/1736841201 <== mon.0 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3dac05a060 con 0x556fa9ea89c0 2024-04-23T10:29:39.249 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:29:39.245+0000 7f3d9ffff640 1 -- 172.21.15.114:0/1736841201 <== mon.0 v2:172.21.15.114:3300/0 2 ==== osd_map(92..92 src has 1..92) ==== 8585+0+0 (secure 0 0 0) 0x7f3dac032480 con 0x556fa9ea89c0 2024-04-23T10:29:39.249 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:29:39.245+0000 7f3dc9998740 1 -- 172.21.15.114:0/1736841201 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f3dac004d90 con 0x556fa9ea89c0 2024-04-23T10:29:39.249 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:29:39.245+0000 7f3d9ffff640 1 -- 172.21.15.114:0/1736841201 <== mon.0 v2:172.21.15.114:3300/0 3 ==== mon_get_version_reply(handle=1 version=23) ==== 24+0+0 (secure 0 0 0) 0x7f3dac035070 con 0x556fa9ea89c0 2024-04-23T10:29:39.249 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:29:39.245+0000 7f3dc9998740 10 client.15960 fetch_fsmap learned FSMap version 23 2024-04-23T10:29:39.249 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:29:39.245+0000 7f3dc9998740 1 -- 172.21.15.114:0/1736841201 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({config=0+,fsmap=23,monmap=4+}) -- 0x556fa9f29c40 con 0x556fa9ea89c0 2024-04-23T10:29:39.249 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:29:39.245+0000 7f3d9ffff640 1 -- 172.21.15.114:0/1736841201 <== mon.0 v2:172.21.15.114:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f3dac004d90 con 0x556fa9ea89c0 2024-04-23T10:29:39.249 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:29:39.245+0000 7f3d9ffff640 1 -- 172.21.15.114:0/1736841201 <== mon.0 v2:172.21.15.114:3300/0 5 ==== fsmap(e 23) ==== 8350+0+0 (secure 0 0 0) 0x7f3dac0318c0 con 0x556fa9ea89c0 2024-04-23T10:29:39.250 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:29:39.245+0000 7f3dc9998740 10 client.15960 fetch_fsmap finished waiting for FSMap version 23 2024-04-23T10:29:39.250 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:29:39.245+0000 7f3dc9998740 10 client.15960 resolve_mds: validated gid 24545 aka daemon mds.b 2024-04-23T10:29:39.250 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:29:39.245+0000 7f3dc9998740 20 client.15960 populate_metadata read hostname 'smithi114' 2024-04-23T10:29:39.250 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:29:39.245+0000 7f3dc9998740 1 --2- 172.21.15.114:0/1736841201 >> [v2:172.21.15.119:6838/937297340,v1:172.21.15.119:6839/937297340] conn(0x556fa9f2a2f0 0x556fa9f2c710 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T10:29:39.250 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:29:39.245+0000 7f3dc9998740 4 client.15960 mds_command: new command op to 24545 tid=0 multi_id=0 [{"prefix": "quiesce db", "roots": ["/"], "timeout": 60.0, "expiration": 81.45, "await_for": 42.0}] 2024-04-23T10:29:39.250 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:29:39.245+0000 7f3dc9998740 1 -- 172.21.15.114:0/1736841201 --> [v2:172.21.15.119:6838/937297340,v1:172.21.15.119:6839/937297340] -- command(tid 0: {"prefix": "quiesce db", "roots": ["/"], "timeout": 60.0, "expiration": 81.45, "await_for": 42.0}) -- 0x556fa9e14c00 con 0x556fa9f2a2f0 2024-04-23T10:29:39.250 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:29:39.246+0000 7f3dc513f640 1 --2- 172.21.15.114:0/1736841201 >> [v2:172.21.15.119:6838/937297340,v1:172.21.15.119:6839/937297340] conn(0x556fa9f2a2f0 0x556fa9f2c710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T10:29:39.250 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:29:39.246+0000 7f3d9ffff640 10 client.15960 ms_handle_connect on v2:172.21.15.119:6838/937297340 2024-04-23T10:29:39.250 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:29:39.246+0000 7f3dc513f640 1 --2- 172.21.15.114:0/1736841201 >> [v2:172.21.15.119:6838/937297340,v1:172.21.15.119:6839/937297340] conn(0x556fa9f2a2f0 0x556fa9f2c710 crc :-1 s=READY pgs=79 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=3c313fe019ed1970 server_cookie=39d9e75f3a637ebf in_seq=0 out_seq=0 2024-04-23T10:29:39.250 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 23 10:29:38 smithi119 ceph-mon[32608]: from='client.? 172.21.15.114:0/1621495308' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-23T10:29:39.253 INFO:journalctl@ceph.mon.a.smithi114.stdout:Apr 23 10:29:38 smithi114 ceph-mon[29255]: from='client.? 172.21.15.114:0/1621495308' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-23T10:29:39.308 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:29:39.308+0000 7f3d9ffff640 1 -- 172.21.15.114:0/1736841201 <== mds.0 v2:172.21.15.119:6838/937297340 1 ==== command_reply(tid 0: 0 ) ==== 8+0+590 (crc 0 0 0) 0x556fa9e14c00 con 0x556fa9f2a2f0 2024-04-23T10:29:39.308 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:29:39.308+0000 7f3d9ffff640 10 client.15960 handle_command_reply: tid=0 2024-04-23T10:29:39.308 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:29:39.308+0000 7f3dc9998740 1 client.15960 shutdown 2024-04-23T10:29:39.308 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:29:39.308+0000 7f3dc9998740 1 -- 172.21.15.114:0/1736841201 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x556fa9ea89c0 msgr2=0x556fa9ead1d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T10:29:39.308 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:29:39.308+0000 7f3dc9998740 1 --2- 172.21.15.114:0/1736841201 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x556fa9ea89c0 0x556fa9ead1d0 secure :-1 s=READY pgs=602 cs=0 l=1 rev1=1 crypto rx=0x7f3dac004d10 tx=0x7f3dac032da0 comp rx=0 tx=0).stop 2024-04-23T10:29:39.308 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:29:39.308+0000 7f3dc9998740 1 -- 172.21.15.114:0/1736841201 shutdown_connections 2024-04-23T10:29:39.308 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:29:39.308+0000 7f3dc9998740 1 --2- 172.21.15.114:0/1736841201 >> [v2:172.21.15.119:6838/937297340,v1:172.21.15.119:6839/937297340] conn(0x556fa9f2a2f0 0x556fa9f2c710 crc :-1 s=READY pgs=79 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:29:39.308 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:29:39.308+0000 7f3dc9998740 1 --2- 172.21.15.114:0/1736841201 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x556fa9eb3090 0x556fa9ead710 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:29:39.308 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:29:39.308+0000 7f3dc9998740 1 --2- 172.21.15.114:0/1736841201 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x556fa9ea89c0 0x556fa9ead1d0 unknown :-1 s=CLOSED pgs=602 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:29:39.308 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:29:39.308+0000 7f3d9ffff640 0 client.15960 ms_handle_reset on v2:172.21.15.119:6838/937297340 2024-04-23T10:29:39.308 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:29:39.308+0000 7f3dc9998740 1 --2- 172.21.15.114:0/1736841201 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x556fa9ea80a0 0x556fa9eacc90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:29:39.308 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:29:39.308+0000 7f3dc9998740 1 -- 172.21.15.114:0/1736841201 >> 172.21.15.114:0/1736841201 conn(0x556fa9ea43c0 msgr2=0x556fa9eb1830 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T10:29:39.308 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:29:39.308+0000 7f3dc9998740 1 -- 172.21.15.114:0/1736841201 shutdown_connections 2024-04-23T10:29:39.309 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:29:39.308+0000 7f3dc9998740 1 -- 172.21.15.114:0/1736841201 wait complete. 2024-04-23T10:29:39.309 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:29:39.308+0000 7f3dc9998740 20 client.15960 trim_cache size 0 max 16384 2024-04-23T10:29:39.309 INFO:teuthology.orchestra.run.smithi114.stdout:{ 2024-04-23T10:29:39.309 INFO:teuthology.orchestra.run.smithi114.stdout: "epoch": 23, 2024-04-23T10:29:39.309 INFO:teuthology.orchestra.run.smithi114.stdout: "leader": 24545, 2024-04-23T10:29:39.309 INFO:teuthology.orchestra.run.smithi114.stdout: "set_version": 14, 2024-04-23T10:29:39.309 INFO:teuthology.orchestra.run.smithi114.stdout: "sets": { 2024-04-23T10:29:39.309 INFO:teuthology.orchestra.run.smithi114.stdout: "4e8b9ad0": { 2024-04-23T10:29:39.309 INFO:teuthology.orchestra.run.smithi114.stdout: "version": 14, 2024-04-23T10:29:39.309 INFO:teuthology.orchestra.run.smithi114.stdout: "age_ref": 0.0, 2024-04-23T10:29:39.309 INFO:teuthology.orchestra.run.smithi114.stdout: "state": { 2024-04-23T10:29:39.309 INFO:teuthology.orchestra.run.smithi114.stdout: "name": "QUIESCED", 2024-04-23T10:29:39.310 INFO:teuthology.orchestra.run.smithi114.stdout: "age": 0.0 2024-04-23T10:29:39.310 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-23T10:29:39.310 INFO:teuthology.orchestra.run.smithi114.stdout: "timeout": 60.0, 2024-04-23T10:29:39.310 INFO:teuthology.orchestra.run.smithi114.stdout: "expiration": 81.5, 2024-04-23T10:29:39.310 INFO:teuthology.orchestra.run.smithi114.stdout: "members": { 2024-04-23T10:29:39.310 INFO:teuthology.orchestra.run.smithi114.stdout: "file:/": { 2024-04-23T10:29:39.310 INFO:teuthology.orchestra.run.smithi114.stdout: "excluded": false, 2024-04-23T10:29:39.310 INFO:teuthology.orchestra.run.smithi114.stdout: "state": { 2024-04-23T10:29:39.310 INFO:teuthology.orchestra.run.smithi114.stdout: "name": "QUIESCED", 2024-04-23T10:29:39.310 INFO:teuthology.orchestra.run.smithi114.stdout: "age": 0.0 2024-04-23T10:29:39.310 INFO:teuthology.orchestra.run.smithi114.stdout: } 2024-04-23T10:29:39.310 INFO:teuthology.orchestra.run.smithi114.stdout: } 2024-04-23T10:29:39.310 INFO:teuthology.orchestra.run.smithi114.stdout: } 2024-04-23T10:29:39.310 INFO:teuthology.orchestra.run.smithi114.stdout: } 2024-04-23T10:29:39.310 INFO:teuthology.orchestra.run.smithi114.stdout: } 2024-04-23T10:29:39.310 INFO:teuthology.orchestra.run.smithi114.stdout:} 2024-04-23T10:29:39.311 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:29:39.309+0000 7f3dc6ba2640 1 -- 172.21.15.114:0/1271895219 >> [v2:172.21.15.114:6800/1395693466,v1:172.21.15.114:6801/1395693466] conn(0x7f3db00a3d20 msgr2=0x7f3db00a61e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T10:29:39.311 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:29:39.309+0000 7f3dc6ba2640 1 --2- 172.21.15.114:0/1271895219 >> [v2:172.21.15.114:6800/1395693466,v1:172.21.15.114:6801/1395693466] conn(0x7f3db00a3d20 0x7f3db00a61e0 secure :-1 s=READY pgs=88 cs=0 l=1 rev1=1 crypto rx=0x7f3db40ce400 tx=0x7f3db406fe10 comp rx=0 tx=0).stop 2024-04-23T10:29:39.311 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:29:39.309+0000 7f3dc6ba2640 1 -- 172.21.15.114:0/1271895219 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f3dc01031b0 msgr2=0x7f3dc019e420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T10:29:39.311 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:29:39.309+0000 7f3dc6ba2640 1 --2- 172.21.15.114:0/1271895219 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f3dc01031b0 0x7f3dc019e420 secure :-1 s=READY pgs=598 cs=0 l=1 rev1=1 crypto rx=0x7f3db400e9f0 tx=0x7f3db400edf0 comp rx=0 tx=0).stop 2024-04-23T10:29:39.311 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:29:39.309+0000 7f3dc6ba2640 1 -- 172.21.15.114:0/1271895219 shutdown_connections 2024-04-23T10:29:39.311 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:29:39.309+0000 7f3dc6ba2640 1 --2- 172.21.15.114:0/1271895219 >> [v2:172.21.15.114:6800/1395693466,v1:172.21.15.114:6801/1395693466] conn(0x7f3db00a3d20 0x7f3db00a61e0 unknown :-1 s=CLOSED pgs=88 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:29:39.312 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:29:39.309+0000 7f3dc6ba2640 1 --2- 172.21.15.114:0/1271895219 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f3dc010c580 0x7f3dc01985d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:29:39.312 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:29:39.309+0000 7f3dc6ba2640 1 --2- 172.21.15.114:0/1271895219 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f3dc0103ad0 0x7f3dc019e960 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:29:39.312 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:29:39.309+0000 7f3dc6ba2640 1 --2- 172.21.15.114:0/1271895219 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f3dc01031b0 0x7f3dc019e420 unknown :-1 s=CLOSED pgs=598 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:29:39.312 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:29:39.309+0000 7f3dc6ba2640 1 -- 172.21.15.114:0/1271895219 >> 172.21.15.114:0/1271895219 conn(0x7f3dc00771d0 msgr2=0x7f3dc00fde10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T10:29:39.312 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:29:39.309+0000 7f3dc6ba2640 1 -- 172.21.15.114:0/1271895219 shutdown_connections 2024-04-23T10:29:39.312 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:29:39.309+0000 7f3dc6ba2640 1 -- 172.21.15.114:0/1271895219 wait complete. 2024-04-23T10:29:39.319 INFO:tasks.quiescer.fs.[cephfs]:Successfully quiesced set '4e8b9ad0', quiesce took 0.6 seconds. Will release after: 21.45 2024-04-23T10:29:39.327 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 23 10:29:38 smithi204 ceph-mon[32649]: from='client.? 172.21.15.114:0/1621495308' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-23T10:29:39.912 INFO:tasks.workunit.client.0.smithi114.stdout: 1 178221 9.05 MB/sec warmup 108 sec latency 605.653 ms 2024-04-23T10:29:40.242 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 23 10:29:39 smithi119 ceph-mon[32608]: pgmap v94: 97 pgs: 97 active+clean; 65 MiB data, 899 MiB used, 1.0 TiB / 1.0 TiB avail; 77 KiB/s rd, 1.2 MiB/s wr, 57 op/s 2024-04-23T10:29:40.242 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 23 10:29:39 smithi119 ceph-mon[32608]: from='client.? 172.21.15.114:0/2866484889' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-23T10:29:40.254 INFO:journalctl@ceph.mon.a.smithi114.stdout:Apr 23 10:29:39 smithi114 ceph-mon[29255]: pgmap v94: 97 pgs: 97 active+clean; 65 MiB data, 899 MiB used, 1.0 TiB / 1.0 TiB avail; 77 KiB/s rd, 1.2 MiB/s wr, 57 op/s 2024-04-23T10:29:40.254 INFO:journalctl@ceph.mon.a.smithi114.stdout:Apr 23 10:29:39 smithi114 ceph-mon[29255]: from='client.? 172.21.15.114:0/2866484889' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-23T10:29:40.327 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 23 10:29:39 smithi204 ceph-mon[32649]: pgmap v94: 97 pgs: 97 active+clean; 65 MiB data, 899 MiB used, 1.0 TiB / 1.0 TiB avail; 77 KiB/s rd, 1.2 MiB/s wr, 57 op/s 2024-04-23T10:29:40.327 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 23 10:29:39 smithi204 ceph-mon[32649]: from='client.? 172.21.15.114:0/2866484889' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-23T10:29:40.912 INFO:tasks.workunit.client.0.smithi114.stdout: 1 178221 8.97 MB/sec warmup 109 sec latency 1605.773 ms 2024-04-23T10:29:41.912 INFO:tasks.workunit.client.0.smithi114.stdout: 1 178221 8.89 MB/sec warmup 110 sec latency 2605.872 ms 2024-04-23T10:29:42.242 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 23 10:29:41 smithi119 ceph-mon[32608]: pgmap v95: 97 pgs: 97 active+clean; 65 MiB data, 899 MiB used, 1.0 TiB / 1.0 TiB avail; 77 KiB/s rd, 1.2 MiB/s wr, 57 op/s 2024-04-23T10:29:42.254 INFO:journalctl@ceph.mon.a.smithi114.stdout:Apr 23 10:29:41 smithi114 ceph-mon[29255]: pgmap v95: 97 pgs: 97 active+clean; 65 MiB data, 899 MiB used, 1.0 TiB / 1.0 TiB avail; 77 KiB/s rd, 1.2 MiB/s wr, 57 op/s 2024-04-23T10:29:42.327 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 23 10:29:41 smithi204 ceph-mon[32649]: pgmap v95: 97 pgs: 97 active+clean; 65 MiB data, 899 MiB used, 1.0 TiB / 1.0 TiB avail; 77 KiB/s rd, 1.2 MiB/s wr, 57 op/s 2024-04-23T10:29:42.912 INFO:tasks.workunit.client.0.smithi114.stdout: 1 178221 8.81 MB/sec warmup 111 sec latency 3606.011 ms 2024-04-23T10:29:43.913 INFO:tasks.workunit.client.0.smithi114.stdout: 1 178221 8.73 MB/sec warmup 112 sec latency 4606.164 ms 2024-04-23T10:29:44.241 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 23 10:29:43 smithi119 ceph-mon[32608]: pgmap v96: 97 pgs: 97 active+clean; 67 MiB data, 896 MiB used, 1.0 TiB / 1.0 TiB avail; 186 KiB/s rd, 2.0 MiB/s wr, 98 op/s 2024-04-23T10:29:44.254 INFO:journalctl@ceph.mon.a.smithi114.stdout:Apr 23 10:29:43 smithi114 ceph-mon[29255]: pgmap v96: 97 pgs: 97 active+clean; 67 MiB data, 896 MiB used, 1.0 TiB / 1.0 TiB avail; 186 KiB/s rd, 2.0 MiB/s wr, 98 op/s 2024-04-23T10:29:44.328 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 23 10:29:43 smithi204 ceph-mon[32649]: pgmap v96: 97 pgs: 97 active+clean; 67 MiB data, 896 MiB used, 1.0 TiB / 1.0 TiB avail; 186 KiB/s rd, 2.0 MiB/s wr, 98 op/s 2024-04-23T10:29:44.913 INFO:tasks.workunit.client.0.smithi114.stdout: 1 178221 8.65 MB/sec warmup 113 sec latency 5606.284 ms 2024-04-23T10:29:45.913 INFO:tasks.workunit.client.0.smithi114.stdout: 1 178221 8.58 MB/sec warmup 114 sec latency 6606.413 ms 2024-04-23T10:29:46.241 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 23 10:29:45 smithi119 ceph-mon[32608]: pgmap v97: 97 pgs: 97 active+clean; 67 MiB data, 896 MiB used, 1.0 TiB / 1.0 TiB avail; 161 KiB/s rd, 1.3 MiB/s wr, 75 op/s 2024-04-23T10:29:46.254 INFO:journalctl@ceph.mon.a.smithi114.stdout:Apr 23 10:29:45 smithi114 ceph-mon[29255]: pgmap v97: 97 pgs: 97 active+clean; 67 MiB data, 896 MiB used, 1.0 TiB / 1.0 TiB avail; 161 KiB/s rd, 1.3 MiB/s wr, 75 op/s 2024-04-23T10:29:46.328 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 23 10:29:45 smithi204 ceph-mon[32649]: pgmap v97: 97 pgs: 97 active+clean; 67 MiB data, 896 MiB used, 1.0 TiB / 1.0 TiB avail; 161 KiB/s rd, 1.3 MiB/s wr, 75 op/s 2024-04-23T10:29:46.886 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:29:46.885+0000 7f2213975640 1 -- 172.21.15.114:0/2935552764 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f220c100e10 msgr2=0x7f220c105ba0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T10:29:46.887 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:29:46.885+0000 7f2213975640 1 --2- 172.21.15.114:0/2935552764 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f220c100e10 0x7f220c105ba0 secure :-1 s=READY pgs=603 cs=0 l=1 rev1=1 crypto rx=0x7f2204009ca0 tx=0x7f220402a8c0 comp rx=0 tx=0).stop 2024-04-23T10:29:46.887 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:29:46.885+0000 7f2213975640 1 -- 172.21.15.114:0/2935552764 shutdown_connections 2024-04-23T10:29:46.887 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:29:46.885+0000 7f2213975640 1 --2- 172.21.15.114:0/2935552764 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f220c100e10 0x7f220c105ba0 unknown :-1 s=CLOSED pgs=603 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:29:46.887 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:29:46.885+0000 7f2213975640 1 --2- 172.21.15.114:0/2935552764 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f220c100450 0x7f220c1008d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:29:46.887 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:29:46.885+0000 7f2213975640 1 --2- 172.21.15.114:0/2935552764 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f220c106f40 0x7f220c107320 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:29:46.887 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:29:46.885+0000 7f2213975640 1 -- 172.21.15.114:0/2935552764 >> 172.21.15.114:0/2935552764 conn(0x7f220c077390 msgr2=0x7f220c0fe760 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T10:29:46.887 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:29:46.885+0000 7f2213975640 1 -- 172.21.15.114:0/2935552764 shutdown_connections 2024-04-23T10:29:46.887 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:29:46.885+0000 7f2213975640 1 -- 172.21.15.114:0/2935552764 wait complete. 2024-04-23T10:29:46.887 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:29:46.886+0000 7f2213975640 1 Processor -- start 2024-04-23T10:29:46.887 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:29:46.886+0000 7f2213975640 1 -- start start 2024-04-23T10:29:46.887 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:29:46.886+0000 7f2213975640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f220c100450 0x7f220c1a4840 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T10:29:46.887 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:29:46.886+0000 7f2213975640 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f220c100e10 0x7f220c1a4d80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T10:29:46.887 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:29:46.886+0000 7f2213975640 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f220c106f40 0x7f220c19e9f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T10:29:46.888 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:29:46.886+0000 7f2213975640 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x7f220c118830 con 0x7f220c100450 2024-04-23T10:29:46.888 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:29:46.886+0000 7f2213975640 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 -- 0x7f220c1186b0 con 0x7f220c100e10 2024-04-23T10:29:46.888 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:29:46.886+0000 7f2213975640 1 -- --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_getmap magic: 0 -- 0x7f220c1189b0 con 0x7f220c106f40 2024-04-23T10:29:46.888 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:29:46.887+0000 7f2211711640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f220c100450 0x7f220c1a4840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T10:29:46.888 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:29:46.887+0000 7f2210f10640 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f220c100e10 0x7f220c1a4d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T10:29:46.888 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:29:46.887+0000 7f2211711640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f220c100450 0x7f220c1a4840 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.114:35786/0 (socket says 172.21.15.114:35786) 2024-04-23T10:29:46.888 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:29:46.887+0000 7f2211711640 1 -- 172.21.15.114:0/3856718228 learned_addr learned my addr 172.21.15.114:0/3856718228 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-04-23T10:29:46.888 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:29:46.887+0000 7f2211f12640 1 --2- 172.21.15.114:0/3856718228 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f220c106f40 0x7f220c19e9f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T10:29:46.888 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:29:46.887+0000 7f2211711640 1 -- 172.21.15.114:0/3856718228 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f220c106f40 msgr2=0x7f220c19e9f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T10:29:46.888 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:29:46.887+0000 7f2211711640 1 --2- 172.21.15.114:0/3856718228 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f220c106f40 0x7f220c19e9f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:29:46.888 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:29:46.887+0000 7f2211711640 1 -- 172.21.15.114:0/3856718228 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f220c100e10 msgr2=0x7f220c1a4d80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T10:29:46.888 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:29:46.887+0000 7f2211711640 1 --2- 172.21.15.114:0/3856718228 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f220c100e10 0x7f220c1a4d80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:29:46.888 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:29:46.887+0000 7f2211711640 1 -- 172.21.15.114:0/3856718228 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f220c19f170 con 0x7f220c100450 2024-04-23T10:29:46.889 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:29:46.887+0000 7f2211711640 1 --2- 172.21.15.114:0/3856718228 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f220c100450 0x7f220c1a4840 secure :-1 s=READY pgs=604 cs=0 l=1 rev1=1 crypto rx=0x7f220800d950 tx=0x7f220800de10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T10:29:46.889 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:29:46.887+0000 7f22027fc640 1 -- 172.21.15.114:0/3856718228 <== mon.0 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2208017020 con 0x7f220c100450 2024-04-23T10:29:46.889 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:29:46.887+0000 7f22027fc640 1 -- 172.21.15.114:0/3856718228 <== mon.0 v2:172.21.15.114:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f2208009c30 con 0x7f220c100450 2024-04-23T10:29:46.889 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:29:46.887+0000 7f2213975640 1 -- 172.21.15.114:0/3856718228 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f220c19f460 con 0x7f220c100450 2024-04-23T10:29:46.889 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:29:46.887+0000 7f22027fc640 1 -- 172.21.15.114:0/3856718228 <== mon.0 v2:172.21.15.114:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f220800f6b0 con 0x7f220c100450 2024-04-23T10:29:46.889 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:29:46.887+0000 7f2213975640 1 -- 172.21.15.114:0/3856718228 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f220c1ab700 con 0x7f220c100450 2024-04-23T10:29:46.896 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:29:46.890+0000 7f22027fc640 1 -- 172.21.15.114:0/3856718228 <== mon.0 v2:172.21.15.114:3300/0 4 ==== mgrmap(e 26) ==== 138778+0+0 (secure 0 0 0) 0x7f220801a020 con 0x7f220c100450 2024-04-23T10:29:46.896 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:29:46.890+0000 7f2216685740 1 Processor -- start 2024-04-23T10:29:46.897 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:29:46.890+0000 7f2216685740 1 -- start start 2024-04-23T10:29:46.897 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:29:46.890+0000 7f22027fc640 1 --2- 172.21.15.114:0/3856718228 >> [v2:172.21.15.114:6800/1395693466,v1:172.21.15.114:6801/1395693466] conn(0x7f21f80a3dc0 0x7f21f80a6280 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T10:29:46.897 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:29:46.890+0000 7f2216685740 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x5582fe2bf620 0x5582fe2bfa00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T10:29:46.897 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:29:46.890+0000 7f2216685740 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x5582fe2bff40 0x5582fe2ca2c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T10:29:46.897 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:29:46.890+0000 7f2216685740 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x5582fe2ca800 0x5582fe2ccbf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T10:29:46.897 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:29:46.890+0000 7f2216685740 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x5582fe224970 con 0x5582fe2ca800 2024-04-23T10:29:46.897 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:29:46.891+0000 7f2216685740 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 -- 0x5582fe1c14a0 con 0x5582fe2bff40 2024-04-23T10:29:46.897 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:29:46.891+0000 7f2216685740 1 -- --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_getmap magic: 0 -- 0x5582fe220930 con 0x5582fe2bf620 2024-04-23T10:29:46.897 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:29:46.891+0000 7f2211f12640 1 --2- 172.21.15.114:0/3856718228 >> [v2:172.21.15.114:6800/1395693466,v1:172.21.15.114:6801/1395693466] conn(0x7f21f80a3dc0 0x7f21f80a6280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T10:29:46.897 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:29:46.891+0000 7f22027fc640 1 -- 172.21.15.114:0/3856718228 <== mon.0 v2:172.21.15.114:3300/0 5 ==== osd_map(92..92 src has 1..92) ==== 8585+0+0 (secure 0 0 0) 0x7f22080d0390 con 0x7f220c100450 2024-04-23T10:29:46.897 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:29:46.891+0000 7f2211f12640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x5582fe2ca800 0x5582fe2ccbf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T10:29:46.897 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:29:46.891+0000 7f2211711640 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x5582fe2bf620 0x5582fe2bfa00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T10:29:46.897 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:29:46.891+0000 7f2211f12640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x5582fe2ca800 0x5582fe2ccbf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.114:35788/0 (socket says 172.21.15.114:35788) 2024-04-23T10:29:46.897 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:29:46.891+0000 7f2211f12640 1 -- 172.21.15.114:0/779650923 learned_addr learned my addr 172.21.15.114:0/779650923 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-04-23T10:29:46.898 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:29:46.891+0000 7f2210f10640 1 --2- 172.21.15.114:0/779650923 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x5582fe2bff40 0x5582fe2ca2c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T10:29:46.898 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:29:46.891+0000 7f2211711640 1 -- 172.21.15.114:0/779650923 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x5582fe2bff40 msgr2=0x5582fe2ca2c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T10:29:46.898 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:29:46.891+0000 7f2211711640 1 --2- 172.21.15.114:0/779650923 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x5582fe2bff40 0x5582fe2ca2c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:29:46.898 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:29:46.891+0000 7f2211711640 1 -- 172.21.15.114:0/779650923 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x5582fe2ca800 msgr2=0x5582fe2ccbf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T10:29:46.898 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:29:46.891+0000 7f2211711640 1 --2- 172.21.15.114:0/779650923 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x5582fe2ca800 0x5582fe2ccbf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:29:46.898 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:29:46.891+0000 7f2211711640 1 -- 172.21.15.114:0/779650923 --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5582fdf37210 con 0x5582fe2bf620 2024-04-23T10:29:46.898 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:29:46.891+0000 7f2211f12640 1 --2- 172.21.15.114:0/779650923 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x5582fe2ca800 0x5582fe2ccbf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-23T10:29:46.898 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:29:46.891+0000 7f2211f12640 1 --2- 172.21.15.114:0/3856718228 >> [v2:172.21.15.114:6800/1395693466,v1:172.21.15.114:6801/1395693466] conn(0x7f21f80a3dc0 0x7f21f80a6280 secure :-1 s=READY pgs=89 cs=0 l=1 rev1=1 crypto rx=0x7f220402f500 tx=0x7f220402f490 comp rx=0 tx=0).ready entity=mgr.15291 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T10:29:46.898 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:29:46.891+0000 7f2211711640 1 --2- 172.21.15.114:0/779650923 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x5582fe2bf620 0x5582fe2bfa00 secure :-1 s=READY pgs=66 cs=0 l=1 rev1=1 crypto rx=0x7f22080cdaa0 tx=0x7f2208009de0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=dda684bd4812cb20 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T10:29:46.898 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:29:46.891+0000 7f21f77fe640 1 -- 172.21.15.114:0/779650923 <== mon.1 v2:172.21.15.204:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2208096e60 con 0x5582fe2bf620 2024-04-23T10:29:46.898 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:29:46.891+0000 7f21f77fe640 1 -- 172.21.15.114:0/779650923 <== mon.1 v2:172.21.15.204:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f22080a06a0 con 0x5582fe2bf620 2024-04-23T10:29:46.898 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:29:46.892+0000 7f21f77fe640 1 -- 172.21.15.114:0/779650923 <== mon.1 v2:172.21.15.204:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f220809a680 con 0x5582fe2bf620 2024-04-23T10:29:46.899 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:29:46.892+0000 7f2216685740 1 -- 172.21.15.114:0/779650923 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x5582fe2bf620 msgr2=0x5582fe2bfa00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T10:29:46.899 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:29:46.892+0000 7f2216685740 1 --2- 172.21.15.114:0/779650923 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x5582fe2bf620 0x5582fe2bfa00 secure :-1 s=READY pgs=66 cs=0 l=1 rev1=1 crypto rx=0x7f22080cdaa0 tx=0x7f2208009de0 comp rx=0 tx=0).stop 2024-04-23T10:29:46.899 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:29:46.892+0000 7f2216685740 1 -- 172.21.15.114:0/779650923 shutdown_connections 2024-04-23T10:29:46.899 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:29:46.892+0000 7f2216685740 1 --2- 172.21.15.114:0/779650923 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x5582fe2ca800 0x5582fe2ccbf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:29:46.899 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:29:46.892+0000 7f2216685740 1 --2- 172.21.15.114:0/779650923 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x5582fe2bff40 0x5582fe2ca2c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:29:46.899 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:29:46.892+0000 7f2216685740 1 --2- 172.21.15.114:0/779650923 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x5582fe2bf620 0x5582fe2bfa00 unknown :-1 s=CLOSED pgs=66 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:29:46.899 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:29:46.892+0000 7f2216685740 1 -- 172.21.15.114:0/779650923 >> 172.21.15.114:0/779650923 conn(0x5582fe2bbb30 msgr2=0x5582fe2bbf10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T10:29:46.899 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:29:46.892+0000 7f2216685740 1 -- 172.21.15.114:0/779650923 shutdown_connections 2024-04-23T10:29:46.899 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:29:46.892+0000 7f2216685740 1 -- 172.21.15.114:0/779650923 wait complete. 2024-04-23T10:29:46.899 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:29:46.893+0000 7f2216685740 1 -- start start 2024-04-23T10:29:46.899 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:29:46.893+0000 7f2216685740 1 Processor -- start 2024-04-23T10:29:46.899 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:29:46.893+0000 7f2216685740 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x5582fe2bf620 0x5582fe2ca0d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T10:29:46.899 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:29:46.893+0000 7f2216685740 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x5582fe2bff40 0x5582fe2c73c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T10:29:46.899 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:29:46.893+0000 7f2216685740 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x5582fe2ca800 0x5582fe2c7900 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T10:29:46.900 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:29:46.893+0000 7f2216685740 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x5582fe2cd430 con 0x5582fe2ca800 2024-04-23T10:29:46.900 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:29:46.893+0000 7f2216685740 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 -- 0x5582fe1bf0b0 con 0x5582fe2bff40 2024-04-23T10:29:46.900 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:29:46.893+0000 7f2216685740 1 -- --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_getmap magic: 0 -- 0x5582fe2cd5b0 con 0x5582fe2bf620 2024-04-23T10:29:46.900 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:29:46.893+0000 7f2216685740 10 client.0 _ll_register_callbacks cb 0x5582fe1acd30 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-23T10:29:46.900 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:29:46.893+0000 7f2211f12640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x5582fe2ca800 0x5582fe2c7900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T10:29:46.900 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:29:46.893+0000 7f2211f12640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x5582fe2ca800 0x5582fe2c7900 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.114:35794/0 (socket says 172.21.15.114:35794) 2024-04-23T10:29:46.900 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:29:46.893+0000 7f2211f12640 1 -- 172.21.15.114:0/702277384 learned_addr learned my addr 172.21.15.114:0/702277384 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-04-23T10:29:46.900 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:29:46.893+0000 7f2211711640 1 --2- 172.21.15.114:0/702277384 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x5582fe2bf620 0x5582fe2ca0d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T10:29:46.900 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:29:46.893+0000 7f2210f10640 1 --2- 172.21.15.114:0/702277384 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x5582fe2bff40 0x5582fe2c73c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T10:29:46.900 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:29:46.894+0000 7f2211f12640 1 -- 172.21.15.114:0/702277384 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x5582fe2bf620 msgr2=0x5582fe2ca0d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T10:29:46.900 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:29:46.894+0000 7f2211f12640 1 --2- 172.21.15.114:0/702277384 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x5582fe2bf620 0x5582fe2ca0d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:29:46.900 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:29:46.894+0000 7f2211f12640 1 -- 172.21.15.114:0/702277384 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x5582fe2bff40 msgr2=0x5582fe2c73c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T10:29:46.900 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:29:46.894+0000 7f2211f12640 1 --2- 172.21.15.114:0/702277384 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x5582fe2bff40 0x5582fe2c73c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:29:46.900 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:29:46.894+0000 7f2211f12640 1 -- 172.21.15.114:0/702277384 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5582fe003620 con 0x5582fe2ca800 2024-04-23T10:29:46.901 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:29:46.894+0000 7f2210f10640 1 --2- 172.21.15.114:0/702277384 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x5582fe2bff40 0x5582fe2c73c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-23T10:29:46.901 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:29:46.894+0000 7f2211f12640 1 --2- 172.21.15.114:0/702277384 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x5582fe2ca800 0x5582fe2c7900 secure :-1 s=READY pgs=605 cs=0 l=1 rev1=1 crypto rx=0x7f2204031a00 tx=0x7f2204049000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=cacac643e2157c29 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T10:29:46.901 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:29:46.894+0000 7f21f4ff9640 10 client.0 ms_handle_connect on v2:172.21.15.114:3300/0 2024-04-23T10:29:46.901 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:29:46.894+0000 7f21f4ff9640 1 -- 172.21.15.114:0/702277384 <== mon.0 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2204051070 con 0x5582fe2ca800 2024-04-23T10:29:46.901 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:29:46.894+0000 7f21f4ff9640 1 -- 172.21.15.114:0/702277384 <== mon.0 v2:172.21.15.114:3300/0 2 ==== osd_map(92..92 src has 1..92) ==== 8585+0+0 (secure 0 0 0) 0x7f2204004240 con 0x5582fe2ca800 2024-04-23T10:29:46.901 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:29:46.894+0000 7f2216685740 1 -- 172.21.15.114:0/702277384 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f22080a06a0 con 0x5582fe2ca800 2024-04-23T10:29:46.901 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:29:46.894+0000 7f21f4ff9640 1 -- 172.21.15.114:0/702277384 <== mon.0 v2:172.21.15.114:3300/0 3 ==== mon_get_version_reply(handle=1 version=23) ==== 24+0+0 (secure 0 0 0) 0x7f220405d050 con 0x5582fe2ca800 2024-04-23T10:29:46.901 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:29:46.894+0000 7f2216685740 10 client.15972 fetch_fsmap learned FSMap version 23 2024-04-23T10:29:46.901 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:29:46.894+0000 7f2216685740 1 -- 172.21.15.114:0/702277384 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({config=0+,fsmap=23,monmap=4+}) -- 0x5582fe2d92d0 con 0x5582fe2ca800 2024-04-23T10:29:46.901 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:29:46.894+0000 7f21f4ff9640 1 -- 172.21.15.114:0/702277384 <== mon.0 v2:172.21.15.114:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f22080a06a0 con 0x5582fe2ca800 2024-04-23T10:29:46.901 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:29:46.895+0000 7f21f4ff9640 1 -- 172.21.15.114:0/702277384 <== mon.0 v2:172.21.15.114:3300/0 5 ==== fsmap(e 23) ==== 8350+0+0 (secure 0 0 0) 0x7f220402fbd0 con 0x5582fe2ca800 2024-04-23T10:29:46.901 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:29:46.895+0000 7f2216685740 10 client.15972 fetch_fsmap finished waiting for FSMap version 23 2024-04-23T10:29:46.901 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:29:46.895+0000 7f2216685740 10 client.15972 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-04-23T10:29:46.901 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:29:46.895+0000 7f2216685740 20 client.15972 populate_metadata read hostname 'smithi114' 2024-04-23T10:29:46.902 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:29:46.895+0000 7f2216685740 1 --2- 172.21.15.114:0/702277384 >> [v2:172.21.15.119:6838/937297340,v1:172.21.15.119:6839/937297340] conn(0x5582fe2da8b0 0x5582fe2dcd10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T10:29:46.902 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:29:46.895+0000 7f2216685740 4 client.15972 mds_command: new command op to 24545 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-23T10:29:46.902 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:29:46.895+0000 7f2216685740 1 -- 172.21.15.114:0/702277384 --> [v2:172.21.15.119:6838/937297340,v1:172.21.15.119:6839/937297340] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x5582fe232390 con 0x5582fe2da8b0 2024-04-23T10:29:46.902 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:29:46.895+0000 7f2211711640 1 --2- 172.21.15.114:0/702277384 >> [v2:172.21.15.119:6838/937297340,v1:172.21.15.119:6839/937297340] conn(0x5582fe2da8b0 0x5582fe2dcd10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T10:29:46.902 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:29:46.895+0000 7f21f4ff9640 10 client.15972 ms_handle_connect on v2:172.21.15.119:6838/937297340 2024-04-23T10:29:46.902 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:29:46.895+0000 7f2211711640 1 --2- 172.21.15.114:0/702277384 >> [v2:172.21.15.119:6838/937297340,v1:172.21.15.119:6839/937297340] conn(0x5582fe2da8b0 0x5582fe2dcd10 crc :-1 s=READY pgs=80 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=81dda9cef7fa11f1 server_cookie=72dbe7f93b4778af in_seq=0 out_seq=0 2024-04-23T10:29:46.902 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:29:46.896+0000 7f21f4ff9640 1 -- 172.21.15.114:0/702277384 <== mds.0 v2:172.21.15.119:6838/937297340 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x5582fe232390 con 0x5582fe2da8b0 2024-04-23T10:29:46.902 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:29:46.896+0000 7f21f4ff9640 10 client.15972 handle_command_reply: tid=0 2024-04-23T10:29:46.902 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:29:46.896+0000 7f2216685740 1 client.15972 shutdown 2024-04-23T10:29:46.902 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:29:46.896+0000 7f2216685740 1 -- 172.21.15.114:0/702277384 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x5582fe2ca800 msgr2=0x5582fe2c7900 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T10:29:46.902 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:29:46.896+0000 7f2216685740 1 --2- 172.21.15.114:0/702277384 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x5582fe2ca800 0x5582fe2c7900 secure :-1 s=READY pgs=605 cs=0 l=1 rev1=1 crypto rx=0x7f2204031a00 tx=0x7f2204049000 comp rx=0 tx=0).stop 2024-04-23T10:29:46.902 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:29:46.896+0000 7f2216685740 1 -- 172.21.15.114:0/702277384 shutdown_connections 2024-04-23T10:29:46.902 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:29:46.896+0000 7f2216685740 1 --2- 172.21.15.114:0/702277384 >> [v2:172.21.15.119:6838/937297340,v1:172.21.15.119:6839/937297340] conn(0x5582fe2da8b0 0x5582fe2dcd10 crc :-1 s=READY pgs=80 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:29:46.902 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:29:46.896+0000 7f2216685740 1 --2- 172.21.15.114:0/702277384 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x5582fe2ca800 0x5582fe2c7900 unknown :-1 s=CLOSED pgs=605 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:29:46.903 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:29:46.896+0000 7f21f4ff9640 0 client.15972 ms_handle_reset on v2:172.21.15.119:6838/937297340 2024-04-23T10:29:46.903 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:29:46.896+0000 7f2216685740 1 --2- 172.21.15.114:0/702277384 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x5582fe2bff40 0x5582fe2c73c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:29:46.903 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:29:46.896+0000 7f2216685740 1 --2- 172.21.15.114:0/702277384 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x5582fe2bf620 0x5582fe2ca0d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:29:46.903 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:29:46.896+0000 7f2216685740 1 -- 172.21.15.114:0/702277384 >> 172.21.15.114:0/702277384 conn(0x5582fe2bbb30 msgr2=0x5582fe2c18b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T10:29:46.903 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:29:46.896+0000 7f2216685740 1 -- 172.21.15.114:0/702277384 shutdown_connections 2024-04-23T10:29:46.903 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:29:46.896+0000 7f2216685740 1 -- 172.21.15.114:0/702277384 wait complete. 2024-04-23T10:29:46.903 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:29:46.896+0000 7f2216685740 20 client.15972 trim_cache size 0 max 16384 2024-04-23T10:29:46.914 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:29:46.911+0000 7f2216685740 1 Processor -- start 2024-04-23T10:29:46.914 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:29:46.911+0000 7f2216685740 1 -- start start 2024-04-23T10:29:46.914 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:29:46.911+0000 7f2216685740 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x5582fe2bf620 0x5582fe2d62f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T10:29:46.914 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:29:46.911+0000 7f2216685740 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x5582fe2bff40 0x5582fe2e25e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T10:29:46.914 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:29:46.911+0000 7f2216685740 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x5582fe2ca800 0x5582fe2e2b20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T10:29:46.914 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:29:46.911+0000 7f2216685740 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x7f220c1b6bb0 con 0x5582fe2bf620 2024-04-23T10:29:46.914 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:29:46.911+0000 7f2216685740 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 -- 0x7f220c1b4450 con 0x5582fe2ca800 2024-04-23T10:29:46.915 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:29:46.911+0000 7f2216685740 1 -- --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_getmap magic: 0 -- 0x7f220c1b22a0 con 0x5582fe2bff40 2024-04-23T10:29:46.915 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:29:46.911+0000 7f2211711640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x5582fe2bf620 0x5582fe2d62f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T10:29:46.915 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:29:46.911+0000 7f2211711640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x5582fe2bf620 0x5582fe2d62f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.114:35806/0 (socket says 172.21.15.114:35806) 2024-04-23T10:29:46.915 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:29:46.911+0000 7f2211711640 1 -- 172.21.15.114:0/3738316153 learned_addr learned my addr 172.21.15.114:0/3738316153 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-04-23T10:29:46.915 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:29:46.912+0000 7f2210f10640 1 --2- 172.21.15.114:0/3738316153 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x5582fe2bff40 0x5582fe2e25e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T10:29:46.915 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:29:46.912+0000 7f2211f12640 1 --2- 172.21.15.114:0/3738316153 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x5582fe2ca800 0x5582fe2e2b20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T10:29:46.915 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:29:46.912+0000 7f2211711640 1 -- 172.21.15.114:0/3738316153 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x5582fe2bff40 msgr2=0x5582fe2e25e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T10:29:46.915 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:29:46.912+0000 7f2211711640 1 --2- 172.21.15.114:0/3738316153 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x5582fe2bff40 0x5582fe2e25e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:29:46.915 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:29:46.912+0000 7f2211711640 1 -- 172.21.15.114:0/3738316153 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x5582fe2ca800 msgr2=0x5582fe2e2b20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T10:29:46.915 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:29:46.912+0000 7f2211711640 1 --2- 172.21.15.114:0/3738316153 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x5582fe2ca800 0x5582fe2e2b20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:29:46.915 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:29:46.912+0000 7f2211711640 1 -- 172.21.15.114:0/3738316153 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5582fe2c6370 con 0x5582fe2bf620 2024-04-23T10:29:46.915 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:29:46.912+0000 7f2211f12640 1 --2- 172.21.15.114:0/3738316153 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x5582fe2ca800 0x5582fe2e2b20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-23T10:29:46.915 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:29:46.912+0000 7f2211711640 1 --2- 172.21.15.114:0/3738316153 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x5582fe2bf620 0x5582fe2d62f0 secure :-1 s=READY pgs=606 cs=0 l=1 rev1=1 crypto rx=0x7f22080cdb50 tx=0x7f2208088040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=31ea2f2943d494ff server_cookie=0 in_seq=0 out_seq=0 2024-04-23T10:29:46.915 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:29:46.912+0000 7f21f6ffd640 1 -- 172.21.15.114:0/3738316153 <== mon.0 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2208096e60 con 0x5582fe2bf620 2024-04-23T10:29:46.916 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:29:46.912+0000 7f21f6ffd640 1 -- 172.21.15.114:0/3738316153 <== mon.0 v2:172.21.15.114:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f22080b5070 con 0x5582fe2bf620 2024-04-23T10:29:46.916 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:29:46.912+0000 7f21f6ffd640 1 -- 172.21.15.114:0/3738316153 <== mon.0 v2:172.21.15.114:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2208098960 con 0x5582fe2bf620 2024-04-23T10:29:46.916 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:29:46.913+0000 7f2216685740 1 -- 172.21.15.114:0/3738316153 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x5582fe2bf620 msgr2=0x5582fe2d62f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T10:29:46.916 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:29:46.913+0000 7f2216685740 1 --2- 172.21.15.114:0/3738316153 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x5582fe2bf620 0x5582fe2d62f0 secure :-1 s=READY pgs=606 cs=0 l=1 rev1=1 crypto rx=0x7f22080cdb50 tx=0x7f2208088040 comp rx=0 tx=0).stop 2024-04-23T10:29:46.916 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:29:46.913+0000 7f2216685740 1 -- 172.21.15.114:0/3738316153 shutdown_connections 2024-04-23T10:29:46.916 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:29:46.913+0000 7f2216685740 1 --2- 172.21.15.114:0/3738316153 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x5582fe2ca800 0x5582fe2e2b20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:29:46.916 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:29:46.913+0000 7f2216685740 1 --2- 172.21.15.114:0/3738316153 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x5582fe2bff40 0x5582fe2e25e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:29:46.916 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:29:46.913+0000 7f2216685740 1 --2- 172.21.15.114:0/3738316153 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x5582fe2bf620 0x5582fe2d62f0 unknown :-1 s=CLOSED pgs=606 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:29:46.916 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:29:46.913+0000 7f2216685740 1 -- 172.21.15.114:0/3738316153 >> 172.21.15.114:0/3738316153 conn(0x5582fe2bbb30 msgr2=0x5582fe2bacb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T10:29:46.916 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:29:46.913+0000 7f2216685740 1 -- 172.21.15.114:0/3738316153 shutdown_connections 2024-04-23T10:29:46.916 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:29:46.913+0000 7f2216685740 1 -- 172.21.15.114:0/3738316153 wait complete. 2024-04-23T10:29:46.916 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:29:46.913+0000 7f2216685740 1 -- start start 2024-04-23T10:29:46.916 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:29:46.913+0000 7f2216685740 1 Processor -- start 2024-04-23T10:29:46.916 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:29:46.913+0000 7f2216685740 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x5582fe2bf620 0x5582fe2d05d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T10:29:46.916 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:29:46.913+0000 7f2216685740 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x5582fe2bff40 0x5582fe2d0b10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T10:29:46.917 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:29:46.913+0000 7f2216685740 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x5582fe2ca800 0x5582fe2d1090 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T10:29:46.917 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:29:46.913+0000 7f2216685740 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x7f220c1b2910 con 0x5582fe2bf620 2024-04-23T10:29:46.917 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:29:46.913+0000 7f2216685740 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 -- 0x7f220c1b3d80 con 0x5582fe2bff40 2024-04-23T10:29:46.917 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:29:46.913+0000 7f2216685740 1 -- --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_getmap magic: 0 -- 0x7f220c1b2fe0 con 0x5582fe2ca800 2024-04-23T10:29:46.917 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:29:46.913+0000 7f2216685740 10 client.0 _ll_register_callbacks cb 0x5582fe1acd30 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-23T10:29:46.917 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:29:46.914+0000 7f2211711640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x5582fe2bf620 0x5582fe2d05d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T10:29:46.917 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:29:46.914+0000 7f2211711640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x5582fe2bf620 0x5582fe2d05d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.114:35818/0 (socket says 172.21.15.114:35818) 2024-04-23T10:29:46.917 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:29:46.914+0000 7f2211711640 1 -- 172.21.15.114:0/4123989586 learned_addr learned my addr 172.21.15.114:0/4123989586 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-04-23T10:29:46.917 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:29:46.914+0000 7f2211f12640 1 --2- 172.21.15.114:0/4123989586 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x5582fe2ca800 0x5582fe2d1090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T10:29:46.917 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:29:46.914+0000 7f2210f10640 1 --2- 172.21.15.114:0/4123989586 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x5582fe2bff40 0x5582fe2d0b10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T10:29:46.917 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:29:46.914+0000 7f2211711640 1 -- 172.21.15.114:0/4123989586 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x5582fe2ca800 msgr2=0x5582fe2d1090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T10:29:46.917 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:29:46.914+0000 7f2211711640 1 --2- 172.21.15.114:0/4123989586 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x5582fe2ca800 0x5582fe2d1090 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:29:46.917 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:29:46.914+0000 7f2211711640 1 -- 172.21.15.114:0/4123989586 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x5582fe2bff40 msgr2=0x5582fe2d0b10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T10:29:46.917 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:29:46.914+0000 7f2211711640 1 --2- 172.21.15.114:0/4123989586 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x5582fe2bff40 0x5582fe2d0b10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:29:46.917 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:29:46.914+0000 7f2211711640 1 -- 172.21.15.114:0/4123989586 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5582fe340150 con 0x5582fe2bf620 2024-04-23T10:29:46.918 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:29:46.914+0000 7f2211711640 1 --2- 172.21.15.114:0/4123989586 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x5582fe2bf620 0x5582fe2d05d0 secure :-1 s=READY pgs=607 cs=0 l=1 rev1=1 crypto rx=0x7f2208098750 tx=0x7f2208010d40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f70bf0c905ae44a5 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T10:29:46.918 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:29:46.914+0000 7f21f4ff9640 10 client.0 ms_handle_connect on v2:172.21.15.114:3300/0 2024-04-23T10:29:46.918 INFO:tasks.workunit.client.0.smithi114.stdout: 1 178221 8.50 MB/sec warmup 115 sec latency 7606.450 ms 2024-04-23T10:29:46.919 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:29:46.914+0000 7f21f4ff9640 1 -- 172.21.15.114:0/4123989586 <== mon.0 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f22080af020 con 0x5582fe2bf620 2024-04-23T10:29:46.919 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:29:46.914+0000 7f2216685740 1 -- 172.21.15.114:0/4123989586 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f22080b5070 con 0x5582fe2bf620 2024-04-23T10:29:46.919 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:29:46.914+0000 7f21f4ff9640 1 -- 172.21.15.114:0/4123989586 <== mon.0 v2:172.21.15.114:3300/0 2 ==== osd_map(92..92 src has 1..92) ==== 8585+0+0 (secure 0 0 0) 0x7f22080cdd20 con 0x5582fe2bf620 2024-04-23T10:29:46.920 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:29:46.914+0000 7f21f4ff9640 1 -- 172.21.15.114:0/4123989586 <== mon.0 v2:172.21.15.114:3300/0 3 ==== mon_get_version_reply(handle=1 version=23) ==== 24+0+0 (secure 0 0 0) 0x7f2208096e60 con 0x5582fe2bf620 2024-04-23T10:29:46.920 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:29:46.914+0000 7f2216685740 10 client.15978 fetch_fsmap learned FSMap version 23 2024-04-23T10:29:46.920 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:29:46.914+0000 7f2216685740 1 -- 172.21.15.114:0/4123989586 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({config=0+,fsmap=23,monmap=4+}) -- 0x5582fe3413d0 con 0x5582fe2bf620 2024-04-23T10:29:46.920 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:29:46.914+0000 7f21f4ff9640 1 -- 172.21.15.114:0/4123989586 <== mon.0 v2:172.21.15.114:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f22080b5070 con 0x5582fe2bf620 2024-04-23T10:29:46.920 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:29:46.915+0000 7f21f4ff9640 1 -- 172.21.15.114:0/4123989586 <== mon.0 v2:172.21.15.114:3300/0 5 ==== fsmap(e 23) ==== 8350+0+0 (secure 0 0 0) 0x7f22080bb020 con 0x5582fe2bf620 2024-04-23T10:29:46.920 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:29:46.915+0000 7f2216685740 10 client.15978 fetch_fsmap finished waiting for FSMap version 23 2024-04-23T10:29:46.920 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:29:46.915+0000 7f2216685740 10 client.15978 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-04-23T10:29:46.920 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:29:46.915+0000 7f2216685740 20 client.15978 populate_metadata read hostname 'smithi114' 2024-04-23T10:29:46.920 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:29:46.915+0000 7f2216685740 1 --2- 172.21.15.114:0/4123989586 >> [v2:172.21.15.119:6838/937297340,v1:172.21.15.119:6839/937297340] conn(0x5582fe341600 0x5582fe3439f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T10:29:46.920 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:29:46.915+0000 7f2216685740 4 client.15978 mds_command: new command op to 24545 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-04-23T10:29:46.920 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:29:46.915+0000 7f2216685740 1 -- 172.21.15.114:0/4123989586 --> [v2:172.21.15.119:6838/937297340,v1:172.21.15.119:6839/937297340] -- command(tid 0: {"prefix": "flush journal"}) -- 0x5582fe22c200 con 0x5582fe341600 2024-04-23T10:29:46.920 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:29:46.915+0000 7f2210f10640 1 --2- 172.21.15.114:0/4123989586 >> [v2:172.21.15.119:6838/937297340,v1:172.21.15.119:6839/937297340] conn(0x5582fe341600 0x5582fe3439f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T10:29:46.920 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:29:46.915+0000 7f2210f10640 1 --2- 172.21.15.114:0/4123989586 >> [v2:172.21.15.119:6838/937297340,v1:172.21.15.119:6839/937297340] conn(0x5582fe341600 0x5582fe3439f0 crc :-1 s=READY pgs=81 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=644e42b48ac354cf server_cookie=c832b41350d12455 in_seq=0 out_seq=0 2024-04-23T10:29:46.920 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:29:46.915+0000 7f21f4ff9640 10 client.15978 ms_handle_connect on v2:172.21.15.119:6838/937297340 2024-04-23T10:29:46.956 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:29:46.956+0000 7f21f4ff9640 1 -- 172.21.15.114:0/4123989586 <== mds.0 v2:172.21.15.119:6838/937297340 1 ==== command_reply(tid 0: 0 ) ==== 8+0+44 (crc 0 0 0) 0x5582fe22c200 con 0x5582fe341600 2024-04-23T10:29:46.956 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:29:46.956+0000 7f21f4ff9640 10 client.15978 handle_command_reply: tid=0 2024-04-23T10:29:46.956 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:29:46.956+0000 7f2216685740 1 client.15978 shutdown 2024-04-23T10:29:46.956 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:29:46.956+0000 7f2216685740 1 -- 172.21.15.114:0/4123989586 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x5582fe2bf620 msgr2=0x5582fe2d05d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T10:29:46.956 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:29:46.956+0000 7f2216685740 1 --2- 172.21.15.114:0/4123989586 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x5582fe2bf620 0x5582fe2d05d0 secure :-1 s=READY pgs=607 cs=0 l=1 rev1=1 crypto rx=0x7f2208098750 tx=0x7f2208010d40 comp rx=0 tx=0).stop 2024-04-23T10:29:46.956 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:29:46.956+0000 7f2216685740 1 -- 172.21.15.114:0/4123989586 shutdown_connections 2024-04-23T10:29:46.956 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:29:46.956+0000 7f2216685740 1 --2- 172.21.15.114:0/4123989586 >> [v2:172.21.15.119:6838/937297340,v1:172.21.15.119:6839/937297340] conn(0x5582fe341600 0x5582fe3439f0 crc :-1 s=READY pgs=81 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:29:46.956 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:29:46.956+0000 7f2216685740 1 --2- 172.21.15.114:0/4123989586 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x5582fe2ca800 0x5582fe2d1090 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:29:46.956 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:29:46.956+0000 7f21f4ff9640 0 client.15978 ms_handle_reset on v2:172.21.15.119:6838/937297340 2024-04-23T10:29:46.956 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:29:46.956+0000 7f2216685740 1 --2- 172.21.15.114:0/4123989586 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x5582fe2bff40 0x5582fe2d0b10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:29:46.956 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:29:46.956+0000 7f2216685740 1 --2- 172.21.15.114:0/4123989586 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x5582fe2bf620 0x5582fe2d05d0 unknown :-1 s=CLOSED pgs=607 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:29:46.956 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:29:46.956+0000 7f2216685740 1 -- 172.21.15.114:0/4123989586 >> 172.21.15.114:0/4123989586 conn(0x5582fe2bbb30 msgr2=0x5582fe2d52e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T10:29:46.956 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:29:46.956+0000 7f2216685740 1 -- 172.21.15.114:0/4123989586 shutdown_connections 2024-04-23T10:29:46.956 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:29:46.956+0000 7f2216685740 1 -- 172.21.15.114:0/4123989586 wait complete. 2024-04-23T10:29:46.957 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:29:46.956+0000 7f2216685740 20 client.15978 trim_cache size 0 max 16384 2024-04-23T10:29:46.957 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stdout:{ 2024-04-23T10:29:46.957 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stdout: "message": "", 2024-04-23T10:29:46.957 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stdout: "return_code": 0 2024-04-23T10:29:46.957 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stdout:} 2024-04-23T10:29:46.958 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:29:46.957+0000 7f2213975640 1 -- 172.21.15.114:0/3856718228 >> [v2:172.21.15.114:6800/1395693466,v1:172.21.15.114:6801/1395693466] conn(0x7f21f80a3dc0 msgr2=0x7f21f80a6280 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T10:29:46.958 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:29:46.957+0000 7f2213975640 1 --2- 172.21.15.114:0/3856718228 >> [v2:172.21.15.114:6800/1395693466,v1:172.21.15.114:6801/1395693466] conn(0x7f21f80a3dc0 0x7f21f80a6280 secure :-1 s=READY pgs=89 cs=0 l=1 rev1=1 crypto rx=0x7f220402f500 tx=0x7f220402f490 comp rx=0 tx=0).stop 2024-04-23T10:29:46.958 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:29:46.957+0000 7f2213975640 1 -- 172.21.15.114:0/3856718228 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f220c100450 msgr2=0x7f220c1a4840 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T10:29:46.958 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:29:46.957+0000 7f2213975640 1 --2- 172.21.15.114:0/3856718228 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f220c100450 0x7f220c1a4840 secure :-1 s=READY pgs=604 cs=0 l=1 rev1=1 crypto rx=0x7f220800d950 tx=0x7f220800de10 comp rx=0 tx=0).stop 2024-04-23T10:29:46.958 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:29:46.957+0000 7f2213975640 1 -- 172.21.15.114:0/3856718228 shutdown_connections 2024-04-23T10:29:46.958 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:29:46.957+0000 7f2213975640 1 --2- 172.21.15.114:0/3856718228 >> [v2:172.21.15.114:6800/1395693466,v1:172.21.15.114:6801/1395693466] conn(0x7f21f80a3dc0 0x7f21f80a6280 unknown :-1 s=CLOSED pgs=89 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:29:46.958 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:29:46.957+0000 7f2213975640 1 --2- 172.21.15.114:0/3856718228 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f220c106f40 0x7f220c19e9f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:29:46.958 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:29:46.957+0000 7f2213975640 1 --2- 172.21.15.114:0/3856718228 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f220c100e10 0x7f220c1a4d80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:29:46.958 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:29:46.957+0000 7f2213975640 1 --2- 172.21.15.114:0/3856718228 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f220c100450 0x7f220c1a4840 unknown :-1 s=CLOSED pgs=604 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:29:46.958 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:29:46.957+0000 7f2213975640 1 -- 172.21.15.114:0/3856718228 >> 172.21.15.114:0/3856718228 conn(0x7f220c077390 msgr2=0x7f220c103b70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T10:29:46.959 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:29:46.957+0000 7f2213975640 1 -- 172.21.15.114:0/3856718228 shutdown_connections 2024-04-23T10:29:46.959 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:29:46.957+0000 7f2213975640 1 -- 172.21.15.114:0/3856718228 wait complete. 2024-04-23T10:29:47.913 INFO:tasks.workunit.client.0.smithi114.stdout: 1 178221 8.43 MB/sec warmup 116 sec latency 8606.501 ms 2024-04-23T10:29:48.241 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 23 10:29:47 smithi119 ceph-mon[32608]: pgmap v98: 97 pgs: 97 active+clean; 66 MiB data, 893 MiB used, 1.0 TiB / 1.0 TiB avail; 203 KiB/s rd, 2.0 MiB/s wr, 107 op/s 2024-04-23T10:29:48.242 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 23 10:29:47 smithi119 ceph-mon[32608]: from='mgr.15291 172.21.15.114:0/1095789187' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-23T10:29:48.254 INFO:journalctl@ceph.mon.a.smithi114.stdout:Apr 23 10:29:47 smithi114 ceph-mon[29255]: pgmap v98: 97 pgs: 97 active+clean; 66 MiB data, 893 MiB used, 1.0 TiB / 1.0 TiB avail; 203 KiB/s rd, 2.0 MiB/s wr, 107 op/s 2024-04-23T10:29:48.254 INFO:journalctl@ceph.mon.a.smithi114.stdout:Apr 23 10:29:47 smithi114 ceph-mon[29255]: from='mgr.15291 172.21.15.114:0/1095789187' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-23T10:29:48.328 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 23 10:29:47 smithi204 ceph-mon[32649]: pgmap v98: 97 pgs: 97 active+clean; 66 MiB data, 893 MiB used, 1.0 TiB / 1.0 TiB avail; 203 KiB/s rd, 2.0 MiB/s wr, 107 op/s 2024-04-23T10:29:48.328 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 23 10:29:47 smithi204 ceph-mon[32649]: from='mgr.15291 172.21.15.114:0/1095789187' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-23T10:29:48.913 INFO:tasks.workunit.client.0.smithi114.stdout: 1 178221 8.36 MB/sec warmup 117 sec latency 9606.599 ms 2024-04-23T10:29:49.242 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 23 10:29:48 smithi119 ceph-mon[32608]: from='mgr.15291 172.21.15.114:0/1095789187' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-23T10:29:49.242 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 23 10:29:48 smithi119 ceph-mon[32608]: from='mgr.15291 172.21.15.114:0/1095789187' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-23T10:29:49.242 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 23 10:29:48 smithi119 ceph-mon[32608]: from='mgr.15291 172.21.15.114:0/1095789187' entity='mgr.x' 2024-04-23T10:29:49.254 INFO:journalctl@ceph.mon.a.smithi114.stdout:Apr 23 10:29:48 smithi114 ceph-mon[29255]: from='mgr.15291 172.21.15.114:0/1095789187' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-23T10:29:49.254 INFO:journalctl@ceph.mon.a.smithi114.stdout:Apr 23 10:29:48 smithi114 ceph-mon[29255]: from='mgr.15291 172.21.15.114:0/1095789187' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-23T10:29:49.254 INFO:journalctl@ceph.mon.a.smithi114.stdout:Apr 23 10:29:48 smithi114 ceph-mon[29255]: from='mgr.15291 172.21.15.114:0/1095789187' entity='mgr.x' 2024-04-23T10:29:49.328 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 23 10:29:48 smithi204 ceph-mon[32649]: from='mgr.15291 172.21.15.114:0/1095789187' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-23T10:29:49.328 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 23 10:29:48 smithi204 ceph-mon[32649]: from='mgr.15291 172.21.15.114:0/1095789187' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-23T10:29:49.328 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 23 10:29:48 smithi204 ceph-mon[32649]: from='mgr.15291 172.21.15.114:0/1095789187' entity='mgr.x' 2024-04-23T10:29:49.913 INFO:tasks.workunit.client.0.smithi114.stdout: 1 178221 8.29 MB/sec warmup 118 sec latency 10606.702 ms 2024-04-23T10:29:50.241 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 23 10:29:49 smithi119 ceph-mon[32608]: pgmap v99: 97 pgs: 97 active+clean; 66 MiB data, 893 MiB used, 1.0 TiB / 1.0 TiB avail; 151 KiB/s rd, 1.5 MiB/s wr, 72 op/s 2024-04-23T10:29:50.254 INFO:journalctl@ceph.mon.a.smithi114.stdout:Apr 23 10:29:49 smithi114 ceph-mon[29255]: pgmap v99: 97 pgs: 97 active+clean; 66 MiB data, 893 MiB used, 1.0 TiB / 1.0 TiB avail; 151 KiB/s rd, 1.5 MiB/s wr, 72 op/s 2024-04-23T10:29:50.327 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 23 10:29:49 smithi204 ceph-mon[32649]: pgmap v99: 97 pgs: 97 active+clean; 66 MiB data, 893 MiB used, 1.0 TiB / 1.0 TiB avail; 151 KiB/s rd, 1.5 MiB/s wr, 72 op/s 2024-04-23T10:29:50.913 INFO:tasks.workunit.client.0.smithi114.stdout: 1 178221 8.22 MB/sec warmup 119 sec latency 11606.801 ms 2024-04-23T10:29:52.242 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 23 10:29:51 smithi119 ceph-mon[32608]: pgmap v100: 97 pgs: 97 active+clean; 66 MiB data, 893 MiB used, 1.0 TiB / 1.0 TiB avail; 151 KiB/s rd, 1.5 MiB/s wr, 72 op/s 2024-04-23T10:29:52.254 INFO:journalctl@ceph.mon.a.smithi114.stdout:Apr 23 10:29:51 smithi114 ceph-mon[29255]: pgmap v100: 97 pgs: 97 active+clean; 66 MiB data, 893 MiB used, 1.0 TiB / 1.0 TiB avail; 151 KiB/s rd, 1.5 MiB/s wr, 72 op/s 2024-04-23T10:29:52.328 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 23 10:29:51 smithi204 ceph-mon[32649]: pgmap v100: 97 pgs: 97 active+clean; 66 MiB data, 893 MiB used, 1.0 TiB / 1.0 TiB avail; 151 KiB/s rd, 1.5 MiB/s wr, 72 op/s 2024-04-23T10:29:52.913 INFO:tasks.workunit.client.0.smithi114.stdout: 1 178221 0.00 MB/sec execute 1 sec latency 13606.979 ms 2024-04-23T10:29:53.914 INFO:tasks.workunit.client.0.smithi114.stdout: 1 178221 0.00 MB/sec execute 2 sec latency 14607.086 ms 2024-04-23T10:29:54.242 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 23 10:29:53 smithi119 ceph-mon[32608]: pgmap v101: 97 pgs: 97 active+clean; 56 MiB data, 862 MiB used, 1.0 TiB / 1.0 TiB avail; 199 KiB/s rd, 1.8 MiB/s wr, 105 op/s 2024-04-23T10:29:54.254 INFO:journalctl@ceph.mon.a.smithi114.stdout:Apr 23 10:29:53 smithi114 ceph-mon[29255]: pgmap v101: 97 pgs: 97 active+clean; 56 MiB data, 862 MiB used, 1.0 TiB / 1.0 TiB avail; 199 KiB/s rd, 1.8 MiB/s wr, 105 op/s 2024-04-23T10:29:54.327 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 23 10:29:53 smithi204 ceph-mon[32649]: pgmap v101: 97 pgs: 97 active+clean; 56 MiB data, 862 MiB used, 1.0 TiB / 1.0 TiB avail; 199 KiB/s rd, 1.8 MiB/s wr, 105 op/s 2024-04-23T10:29:54.914 INFO:tasks.workunit.client.0.smithi114.stdout: 1 178221 0.00 MB/sec execute 3 sec latency 15607.212 ms 2024-04-23T10:29:55.914 INFO:tasks.workunit.client.0.smithi114.stdout: 1 178221 0.00 MB/sec execute 4 sec latency 16607.322 ms 2024-04-23T10:29:56.242 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 23 10:29:55 smithi119 ceph-mon[32608]: pgmap v102: 97 pgs: 97 active+clean; 56 MiB data, 862 MiB used, 1.0 TiB / 1.0 TiB avail; 90 KiB/s rd, 1.0 MiB/s wr, 64 op/s 2024-04-23T10:29:56.254 INFO:journalctl@ceph.mon.a.smithi114.stdout:Apr 23 10:29:55 smithi114 ceph-mon[29255]: pgmap v102: 97 pgs: 97 active+clean; 56 MiB data, 862 MiB used, 1.0 TiB / 1.0 TiB avail; 90 KiB/s rd, 1.0 MiB/s wr, 64 op/s 2024-04-23T10:29:56.327 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 23 10:29:55 smithi204 ceph-mon[32649]: pgmap v102: 97 pgs: 97 active+clean; 56 MiB data, 862 MiB used, 1.0 TiB / 1.0 TiB avail; 90 KiB/s rd, 1.0 MiB/s wr, 64 op/s 2024-04-23T10:29:56.914 INFO:tasks.workunit.client.0.smithi114.stdout: 1 178221 0.00 MB/sec execute 5 sec latency 17607.395 ms 2024-04-23T10:29:57.914 INFO:tasks.workunit.client.0.smithi114.stdout: 1 178221 0.00 MB/sec execute 6 sec latency 18607.496 ms 2024-04-23T10:29:58.242 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 23 10:29:57 smithi119 ceph-mon[32608]: pgmap v103: 97 pgs: 97 active+clean; 52 MiB data, 851 MiB used, 1.0 TiB / 1.0 TiB avail; 110 KiB/s rd, 1.1 MiB/s wr, 74 op/s 2024-04-23T10:29:58.253 INFO:journalctl@ceph.mon.a.smithi114.stdout:Apr 23 10:29:57 smithi114 ceph-mon[29255]: pgmap v103: 97 pgs: 97 active+clean; 52 MiB data, 851 MiB used, 1.0 TiB / 1.0 TiB avail; 110 KiB/s rd, 1.1 MiB/s wr, 74 op/s 2024-04-23T10:29:58.327 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 23 10:29:57 smithi204 ceph-mon[32649]: pgmap v103: 97 pgs: 97 active+clean; 52 MiB data, 851 MiB used, 1.0 TiB / 1.0 TiB avail; 110 KiB/s rd, 1.1 MiB/s wr, 74 op/s 2024-04-23T10:29:58.914 INFO:tasks.workunit.client.0.smithi114.stdout: 1 178221 0.00 MB/sec execute 7 sec latency 19607.582 ms 2024-04-23T10:29:59.914 INFO:tasks.workunit.client.0.smithi114.stdout: 1 178221 0.00 MB/sec execute 8 sec latency 20607.679 ms 2024-04-23T10:30:00.056 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:30:00.056+0000 7f106cb08640 1 -- 172.21.15.114:0/2690683568 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f1068102b80 msgr2=0x7f1068106bc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T10:30:00.056 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:30:00.056+0000 7f106cb08640 1 --2- 172.21.15.114:0/2690683568 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f1068102b80 0x7f1068106bc0 secure :-1 s=READY pgs=608 cs=0 l=1 rev1=1 crypto rx=0x7f105c009ca0 tx=0x7f105c02a8c0 comp rx=0 tx=0).stop 2024-04-23T10:30:00.056 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:30:00.056+0000 7f106cb08640 1 -- 172.21.15.114:0/2690683568 shutdown_connections 2024-04-23T10:30:00.056 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:30:00.056+0000 7f106cb08640 1 --2- 172.21.15.114:0/2690683568 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f10681072a0 0x7f106810ee70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:30:00.056 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:30:00.056+0000 7f106cb08640 1 --2- 172.21.15.114:0/2690683568 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f1068102b80 0x7f1068106bc0 unknown :-1 s=CLOSED pgs=608 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:30:00.056 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:30:00.056+0000 7f106cb08640 1 --2- 172.21.15.114:0/2690683568 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f10681021d0 0x7f10681025b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:30:00.056 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:30:00.056+0000 7f106cb08640 1 -- 172.21.15.114:0/2690683568 >> 172.21.15.114:0/2690683568 conn(0x7f10680793e0 msgr2=0x7f10680797f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T10:30:00.056 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:30:00.057+0000 7f106cb08640 1 -- 172.21.15.114:0/2690683568 shutdown_connections 2024-04-23T10:30:00.056 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:30:00.057+0000 7f106cb08640 1 -- 172.21.15.114:0/2690683568 wait complete. 2024-04-23T10:30:00.059 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:30:00.057+0000 7f106cb08640 1 Processor -- start 2024-04-23T10:30:00.059 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:30:00.057+0000 7f106cb08640 1 -- start start 2024-04-23T10:30:00.059 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:30:00.058+0000 7f106cb08640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f10681021d0 0x7f10681986e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T10:30:00.059 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:30:00.058+0000 7f106cb08640 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f10681072a0 0x7f1068198c20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T10:30:00.059 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:30:00.058+0000 7f106cb08640 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f106819cfa0 0x7f106819d450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T10:30:00.059 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:30:00.058+0000 7f106cb08640 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x7f10681121e0 con 0x7f10681021d0 2024-04-23T10:30:00.059 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:30:00.058+0000 7f106cb08640 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 -- 0x7f1068112060 con 0x7f106819cfa0 2024-04-23T10:30:00.059 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:30:00.058+0000 7f106cb08640 1 -- --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_getmap magic: 0 -- 0x7f1068112360 con 0x7f10681072a0 2024-04-23T10:30:00.060 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:30:00.058+0000 7f106659c640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f10681021d0 0x7f10681986e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T10:30:00.060 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:30:00.058+0000 7f106659c640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f10681021d0 0x7f10681986e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.114:48548/0 (socket says 172.21.15.114:48548) 2024-04-23T10:30:00.060 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:30:00.058+0000 7f106659c640 1 -- 172.21.15.114:0/2172959367 learned_addr learned my addr 172.21.15.114:0/2172959367 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-04-23T10:30:00.060 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:30:00.058+0000 7f1065d9b640 1 --2- 172.21.15.114:0/2172959367 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f10681072a0 0x7f1068198c20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T10:30:00.060 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:30:00.058+0000 7f1066d9d640 1 --2- 172.21.15.114:0/2172959367 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f106819cfa0 0x7f106819d450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T10:30:00.060 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:30:00.058+0000 7f106659c640 1 -- 172.21.15.114:0/2172959367 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f10681072a0 msgr2=0x7f1068198c20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T10:30:00.060 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:30:00.058+0000 7f106659c640 1 --2- 172.21.15.114:0/2172959367 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f10681072a0 0x7f1068198c20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:30:00.060 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:30:00.058+0000 7f106659c640 1 -- 172.21.15.114:0/2172959367 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f106819cfa0 msgr2=0x7f106819d450 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T10:30:00.060 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:30:00.058+0000 7f106659c640 1 --2- 172.21.15.114:0/2172959367 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f106819cfa0 0x7f106819d450 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:30:00.060 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:30:00.058+0000 7f106659c640 1 -- 172.21.15.114:0/2172959367 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f106819da90 con 0x7f10681021d0 2024-04-23T10:30:00.060 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:30:00.058+0000 7f1066d9d640 1 --2- 172.21.15.114:0/2172959367 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f106819cfa0 0x7f106819d450 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-23T10:30:00.060 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:30:00.058+0000 7f106659c640 1 --2- 172.21.15.114:0/2172959367 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f10681021d0 0x7f10681986e0 secure :-1 s=READY pgs=609 cs=0 l=1 rev1=1 crypto rx=0x7f106000d770 tx=0x7f106000dc30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T10:30:00.060 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:30:00.058+0000 7f10577fe640 1 -- 172.21.15.114:0/2172959367 <== mon.0 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1060017070 con 0x7f10681021d0 2024-04-23T10:30:00.060 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:30:00.058+0000 7f106cb08640 1 -- 172.21.15.114:0/2172959367 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f106819dd80 con 0x7f10681021d0 2024-04-23T10:30:00.060 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:30:00.058+0000 7f10577fe640 1 -- 172.21.15.114:0/2172959367 <== mon.0 v2:172.21.15.114:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f10600054d0 con 0x7f10681021d0 2024-04-23T10:30:00.061 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:30:00.059+0000 7f10577fe640 1 -- 172.21.15.114:0/2172959367 <== mon.0 v2:172.21.15.114:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1060004c30 con 0x7f10681021d0 2024-04-23T10:30:00.061 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:30:00.059+0000 7f106cb08640 1 -- 172.21.15.114:0/2172959367 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f10681a5520 con 0x7f10681021d0 2024-04-23T10:30:00.067 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:30:00.061+0000 7f10577fe640 1 -- 172.21.15.114:0/2172959367 <== mon.0 v2:172.21.15.114:3300/0 4 ==== mgrmap(e 26) ==== 138778+0+0 (secure 0 0 0) 0x7f1060004e10 con 0x7f10681021d0 2024-04-23T10:30:00.067 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:30:00.061+0000 7f106f81d740 1 Processor -- start 2024-04-23T10:30:00.067 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:30:00.061+0000 7f106f81d740 1 -- start start 2024-04-23T10:30:00.067 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:30:00.061+0000 7f106f81d740 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55fb69934540 0x55fb69934920 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T10:30:00.067 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:30:00.061+0000 7f106f81d740 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x55fb69934e60 0x55fb699436d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T10:30:00.067 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:30:00.061+0000 7f106f81d740 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x55fb69943c10 0x55fb69946000 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T10:30:00.067 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:30:00.061+0000 7f106f81d740 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x55fb69899970 con 0x55fb69934540 2024-04-23T10:30:00.067 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:30:00.061+0000 7f106f81d740 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 -- 0x55fb698364a0 con 0x55fb69943c10 2024-04-23T10:30:00.068 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:30:00.061+0000 7f106f81d740 1 -- --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_getmap magic: 0 -- 0x55fb69895930 con 0x55fb69934e60 2024-04-23T10:30:00.068 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:30:00.062+0000 7f1066d9d640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55fb69934540 0x55fb69934920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T10:30:00.068 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:30:00.062+0000 7f1066d9d640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55fb69934540 0x55fb69934920 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.114:48550/0 (socket says 172.21.15.114:48550) 2024-04-23T10:30:00.068 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:30:00.062+0000 7f1066d9d640 1 -- 172.21.15.114:0/1292621140 learned_addr learned my addr 172.21.15.114:0/1292621140 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-04-23T10:30:00.068 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:30:00.062+0000 7f10577fe640 1 --2- 172.21.15.114:0/2172959367 >> [v2:172.21.15.114:6800/1395693466,v1:172.21.15.114:6801/1395693466] conn(0x7f10500a3dc0 0x7f10500a6280 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T10:30:00.068 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:30:00.062+0000 7f1065d9b640 1 --2- 172.21.15.114:0/1292621140 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x55fb69943c10 0x55fb69946000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T10:30:00.068 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:30:00.062+0000 7f106659c640 1 --2- 172.21.15.114:0/1292621140 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x55fb69934e60 0x55fb699436d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T10:30:00.068 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:30:00.062+0000 7f1066d9d640 1 --2- 172.21.15.114:0/2172959367 >> [v2:172.21.15.114:6800/1395693466,v1:172.21.15.114:6801/1395693466] conn(0x7f10500a3dc0 0x7f10500a6280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T10:30:00.068 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:30:00.062+0000 7f1066d9d640 1 -- 172.21.15.114:0/1292621140 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x55fb69934e60 msgr2=0x55fb699436d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T10:30:00.068 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:30:00.062+0000 7f1066d9d640 1 --2- 172.21.15.114:0/1292621140 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x55fb69934e60 0x55fb699436d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:30:00.068 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:30:00.062+0000 7f1066d9d640 1 -- 172.21.15.114:0/1292621140 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x55fb69943c10 msgr2=0x55fb69946000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T10:30:00.068 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:30:00.062+0000 7f1066d9d640 1 --2- 172.21.15.114:0/1292621140 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x55fb69943c10 0x55fb69946000 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:30:00.068 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:30:00.062+0000 7f10577fe640 1 -- 172.21.15.114:0/2172959367 <== mon.0 v2:172.21.15.114:3300/0 5 ==== osd_map(92..92 src has 1..92) ==== 8585+0+0 (secure 0 0 0) 0x7f10600d03c0 con 0x7f10681021d0 2024-04-23T10:30:00.068 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:30:00.062+0000 7f1066d9d640 1 -- 172.21.15.114:0/1292621140 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55fb695ac210 con 0x55fb69934540 2024-04-23T10:30:00.068 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:30:00.062+0000 7f106659c640 1 --2- 172.21.15.114:0/1292621140 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x55fb69934e60 0x55fb699436d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-23T10:30:00.068 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:30:00.062+0000 7f1066d9d640 1 --2- 172.21.15.114:0/1292621140 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55fb69934540 0x55fb69934920 secure :-1 s=READY pgs=610 cs=0 l=1 rev1=1 crypto rx=0x7f1058015870 tx=0x7f1058015d30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3cf7d79df2faef82 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T10:30:00.069 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:30:00.062+0000 7f1054ff9640 1 -- 172.21.15.114:0/1292621140 <== mon.0 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1058022070 con 0x55fb69934540 2024-04-23T10:30:00.069 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:30:00.062+0000 7f1054ff9640 1 -- 172.21.15.114:0/1292621140 <== mon.0 v2:172.21.15.114:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f1058009cf0 con 0x55fb69934540 2024-04-23T10:30:00.069 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:30:00.062+0000 7f1066d9d640 1 --2- 172.21.15.114:0/2172959367 >> [v2:172.21.15.114:6800/1395693466,v1:172.21.15.114:6801/1395693466] conn(0x7f10500a3dc0 0x7f10500a6280 secure :-1 s=READY pgs=90 cs=0 l=1 rev1=1 crypto rx=0x7f105801f000 tx=0x7f105801b000 comp rx=0 tx=0).ready entity=mgr.15291 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T10:30:00.069 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:30:00.062+0000 7f1054ff9640 1 -- 172.21.15.114:0/1292621140 <== mon.0 v2:172.21.15.114:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f105800fc50 con 0x55fb69934540 2024-04-23T10:30:00.069 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:30:00.063+0000 7f106f81d740 1 -- 172.21.15.114:0/1292621140 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55fb69934540 msgr2=0x55fb69934920 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T10:30:00.069 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:30:00.063+0000 7f106f81d740 1 --2- 172.21.15.114:0/1292621140 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55fb69934540 0x55fb69934920 secure :-1 s=READY pgs=610 cs=0 l=1 rev1=1 crypto rx=0x7f1058015870 tx=0x7f1058015d30 comp rx=0 tx=0).stop 2024-04-23T10:30:00.069 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:30:00.063+0000 7f106f81d740 1 -- 172.21.15.114:0/1292621140 shutdown_connections 2024-04-23T10:30:00.069 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:30:00.063+0000 7f106f81d740 1 --2- 172.21.15.114:0/1292621140 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x55fb69943c10 0x55fb69946000 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:30:00.069 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:30:00.063+0000 7f106f81d740 1 --2- 172.21.15.114:0/1292621140 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x55fb69934e60 0x55fb699436d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:30:00.069 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:30:00.063+0000 7f106f81d740 1 --2- 172.21.15.114:0/1292621140 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55fb69934540 0x55fb69934920 unknown :-1 s=CLOSED pgs=610 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:30:00.069 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:30:00.063+0000 7f106f81d740 1 -- 172.21.15.114:0/1292621140 >> 172.21.15.114:0/1292621140 conn(0x55fb69930b30 msgr2=0x55fb69930f10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T10:30:00.069 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:30:00.063+0000 7f106f81d740 1 -- 172.21.15.114:0/1292621140 shutdown_connections 2024-04-23T10:30:00.069 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:30:00.063+0000 7f106f81d740 1 -- 172.21.15.114:0/1292621140 wait complete. 2024-04-23T10:30:00.069 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:30:00.064+0000 7f106f81d740 1 -- start start 2024-04-23T10:30:00.069 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:30:00.064+0000 7f106f81d740 1 Processor -- start 2024-04-23T10:30:00.070 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:30:00.064+0000 7f106f81d740 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x55fb69934540 0x55fb69943510 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T10:30:00.070 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:30:00.064+0000 7f106f81d740 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x55fb69934e60 0x55fb69940800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T10:30:00.070 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:30:00.064+0000 7f106f81d740 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55fb69943c10 0x55fb69940d40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T10:30:00.070 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:30:00.064+0000 7f106f81d740 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x55fb69946840 con 0x55fb69943c10 2024-04-23T10:30:00.070 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:30:00.064+0000 7f106f81d740 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 -- 0x55fb698340b0 con 0x55fb69934e60 2024-04-23T10:30:00.070 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:30:00.064+0000 7f106f81d740 1 -- --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_getmap magic: 0 -- 0x55fb699469c0 con 0x55fb69934540 2024-04-23T10:30:00.070 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:30:00.064+0000 7f106f81d740 10 client.0 _ll_register_callbacks cb 0x55fb69821d30 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-23T10:30:00.070 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:30:00.064+0000 7f1066d9d640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55fb69943c10 0x55fb69940d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T10:30:00.070 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:30:00.064+0000 7f1066d9d640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55fb69943c10 0x55fb69940d40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.114:48556/0 (socket says 172.21.15.114:48556) 2024-04-23T10:30:00.070 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:30:00.064+0000 7f1066d9d640 1 -- 172.21.15.114:0/1530800126 learned_addr learned my addr 172.21.15.114:0/1530800126 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-04-23T10:30:00.070 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:30:00.064+0000 7f1065d9b640 1 --2- 172.21.15.114:0/1530800126 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x55fb69934e60 0x55fb69940800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T10:30:00.070 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:30:00.064+0000 7f106659c640 1 --2- 172.21.15.114:0/1530800126 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x55fb69934540 0x55fb69943510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T10:30:00.070 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:30:00.064+0000 7f1066d9d640 1 -- 172.21.15.114:0/1530800126 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x55fb69934540 msgr2=0x55fb69943510 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T10:30:00.070 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:30:00.064+0000 7f1066d9d640 1 --2- 172.21.15.114:0/1530800126 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x55fb69934540 0x55fb69943510 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:30:00.070 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:30:00.064+0000 7f1066d9d640 1 -- 172.21.15.114:0/1530800126 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x55fb69934e60 msgr2=0x55fb69940800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T10:30:00.071 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:30:00.064+0000 7f1066d9d640 1 --2- 172.21.15.114:0/1530800126 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x55fb69934e60 0x55fb69940800 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:30:00.071 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:30:00.064+0000 7f1066d9d640 1 -- 172.21.15.114:0/1530800126 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55fb69678620 con 0x55fb69943c10 2024-04-23T10:30:00.071 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:30:00.065+0000 7f1066d9d640 1 --2- 172.21.15.114:0/1530800126 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55fb69943c10 0x55fb69940d40 secure :-1 s=READY pgs=611 cs=0 l=1 rev1=1 crypto rx=0x7f105800f990 tx=0x7f105800fc60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2e515535021d7321 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T10:30:00.071 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:30:00.065+0000 7f104dffb640 10 client.0 ms_handle_connect on v2:172.21.15.114:3300/0 2024-04-23T10:30:00.071 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:30:00.065+0000 7f104dffb640 1 -- 172.21.15.114:0/1530800126 <== mon.0 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1058022040 con 0x55fb69943c10 2024-04-23T10:30:00.071 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:30:00.065+0000 7f104dffb640 1 -- 172.21.15.114:0/1530800126 <== mon.0 v2:172.21.15.114:3300/0 2 ==== osd_map(92..92 src has 1..92) ==== 8585+0+0 (secure 0 0 0) 0x7f1058015050 con 0x55fb69943c10 2024-04-23T10:30:00.071 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:30:00.065+0000 7f106f81d740 1 -- 172.21.15.114:0/1530800126 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f1058009cf0 con 0x55fb69943c10 2024-04-23T10:30:00.071 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:30:00.065+0000 7f104dffb640 1 -- 172.21.15.114:0/1530800126 <== mon.0 v2:172.21.15.114:3300/0 3 ==== mon_get_version_reply(handle=1 version=23) ==== 24+0+0 (secure 0 0 0) 0x7f1058009510 con 0x55fb69943c10 2024-04-23T10:30:00.071 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:30:00.065+0000 7f106f81d740 10 client.15990 fetch_fsmap learned FSMap version 23 2024-04-23T10:30:00.071 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:30:00.065+0000 7f106f81d740 1 -- 172.21.15.114:0/1530800126 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({config=0+,fsmap=23,monmap=4+}) -- 0x55fb69952710 con 0x55fb69943c10 2024-04-23T10:30:00.071 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:30:00.065+0000 7f104dffb640 1 -- 172.21.15.114:0/1530800126 <== mon.0 v2:172.21.15.114:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f1058009cf0 con 0x55fb69943c10 2024-04-23T10:30:00.071 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:30:00.065+0000 7f104dffb640 1 -- 172.21.15.114:0/1530800126 <== mon.0 v2:172.21.15.114:3300/0 5 ==== fsmap(e 23) ==== 8350+0+0 (secure 0 0 0) 0x7f1058036030 con 0x55fb69943c10 2024-04-23T10:30:00.071 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:30:00.065+0000 7f106f81d740 10 client.15990 fetch_fsmap finished waiting for FSMap version 23 2024-04-23T10:30:00.071 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:30:00.065+0000 7f106f81d740 10 client.15990 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-04-23T10:30:00.072 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:30:00.065+0000 7f106f81d740 20 client.15990 populate_metadata read hostname 'smithi114' 2024-04-23T10:30:00.072 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:30:00.065+0000 7f106f81d740 1 --2- 172.21.15.114:0/1530800126 >> [v2:172.21.15.119:6838/937297340,v1:172.21.15.119:6839/937297340] conn(0x55fb69953a60 0x55fb69955e50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T10:30:00.072 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:30:00.065+0000 7f106f81d740 4 client.15990 mds_command: new command op to 24545 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-23T10:30:00.072 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:30:00.065+0000 7f106f81d740 1 -- 172.21.15.114:0/1530800126 --> [v2:172.21.15.119:6838/937297340,v1:172.21.15.119:6839/937297340] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55fb698a7390 con 0x55fb69953a60 2024-04-23T10:30:00.072 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:30:00.066+0000 7f106659c640 1 --2- 172.21.15.114:0/1530800126 >> [v2:172.21.15.119:6838/937297340,v1:172.21.15.119:6839/937297340] conn(0x55fb69953a60 0x55fb69955e50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T10:30:00.072 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:30:00.066+0000 7f106659c640 1 --2- 172.21.15.114:0/1530800126 >> [v2:172.21.15.119:6838/937297340,v1:172.21.15.119:6839/937297340] conn(0x55fb69953a60 0x55fb69955e50 crc :-1 s=READY pgs=82 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=d630b47d5673d0f9 server_cookie=9868163b51794793 in_seq=0 out_seq=0 2024-04-23T10:30:00.072 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:30:00.066+0000 7f104dffb640 10 client.15990 ms_handle_connect on v2:172.21.15.119:6838/937297340 2024-04-23T10:30:00.072 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:30:00.067+0000 7f104dffb640 1 -- 172.21.15.114:0/1530800126 <== mds.0 v2:172.21.15.119:6838/937297340 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x55fb698a7390 con 0x55fb69953a60 2024-04-23T10:30:00.072 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:30:00.067+0000 7f104dffb640 10 client.15990 handle_command_reply: tid=0 2024-04-23T10:30:00.072 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:30:00.067+0000 7f106f81d740 1 client.15990 shutdown 2024-04-23T10:30:00.072 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:30:00.067+0000 7f106f81d740 1 -- 172.21.15.114:0/1530800126 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55fb69943c10 msgr2=0x55fb69940d40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T10:30:00.072 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:30:00.067+0000 7f106f81d740 1 --2- 172.21.15.114:0/1530800126 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55fb69943c10 0x55fb69940d40 secure :-1 s=READY pgs=611 cs=0 l=1 rev1=1 crypto rx=0x7f105800f990 tx=0x7f105800fc60 comp rx=0 tx=0).stop 2024-04-23T10:30:00.072 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:30:00.067+0000 7f106f81d740 1 -- 172.21.15.114:0/1530800126 shutdown_connections 2024-04-23T10:30:00.072 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:30:00.067+0000 7f106f81d740 1 --2- 172.21.15.114:0/1530800126 >> [v2:172.21.15.119:6838/937297340,v1:172.21.15.119:6839/937297340] conn(0x55fb69953a60 0x55fb69955e50 crc :-1 s=READY pgs=82 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:30:00.072 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:30:00.067+0000 7f106f81d740 1 --2- 172.21.15.114:0/1530800126 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55fb69943c10 0x55fb69940d40 unknown :-1 s=CLOSED pgs=611 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:30:00.072 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:30:00.067+0000 7f106f81d740 1 --2- 172.21.15.114:0/1530800126 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x55fb69934e60 0x55fb69940800 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:30:00.073 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:30:00.067+0000 7f104dffb640 0 client.15990 ms_handle_reset on v2:172.21.15.119:6838/937297340 2024-04-23T10:30:00.073 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:30:00.067+0000 7f106f81d740 1 --2- 172.21.15.114:0/1530800126 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x55fb69934540 0x55fb69943510 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:30:00.073 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:30:00.067+0000 7f106f81d740 1 -- 172.21.15.114:0/1530800126 >> 172.21.15.114:0/1530800126 conn(0x55fb69930b30 msgr2=0x55fb699369a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T10:30:00.073 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:30:00.067+0000 7f106f81d740 1 -- 172.21.15.114:0/1530800126 shutdown_connections 2024-04-23T10:30:00.073 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:30:00.067+0000 7f106f81d740 1 -- 172.21.15.114:0/1530800126 wait complete. 2024-04-23T10:30:00.073 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:30:00.067+0000 7f106f81d740 20 client.15990 trim_cache size 0 max 16384 2024-04-23T10:30:00.082 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:30:00.082+0000 7f106f81d740 1 Processor -- start 2024-04-23T10:30:00.082 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:30:00.082+0000 7f106f81d740 1 -- start start 2024-04-23T10:30:00.082 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:30:00.082+0000 7f106f81d740 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55fb69934540 0x55fb6994f700 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T10:30:00.082 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:30:00.082+0000 7f106f81d740 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x55fb69934e60 0x55fb69943040 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T10:30:00.082 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:30:00.082+0000 7f106f81d740 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x55fb69943c10 0x55fb6995b790 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T10:30:00.082 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:30:00.082+0000 7f106f81d740 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x7f10681ae2a0 con 0x55fb69934540 2024-04-23T10:30:00.082 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:30:00.082+0000 7f106f81d740 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 -- 0x7f10681abb40 con 0x55fb69934e60 2024-04-23T10:30:00.082 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:30:00.082+0000 7f106f81d740 1 -- --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_getmap magic: 0 -- 0x7f10681a9990 con 0x55fb69943c10 2024-04-23T10:30:00.082 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:30:00.082+0000 7f106659c640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55fb69934540 0x55fb6994f700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T10:30:00.083 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:30:00.082+0000 7f106659c640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55fb69934540 0x55fb6994f700 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.114:48564/0 (socket says 172.21.15.114:48564) 2024-04-23T10:30:00.083 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:30:00.082+0000 7f106659c640 1 -- 172.21.15.114:0/4205440525 learned_addr learned my addr 172.21.15.114:0/4205440525 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-04-23T10:30:00.083 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:30:00.083+0000 7f106659c640 1 -- 172.21.15.114:0/4205440525 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x55fb69943c10 msgr2=0x55fb6995b790 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T10:30:00.083 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:30:00.083+0000 7f106659c640 1 --2- 172.21.15.114:0/4205440525 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x55fb69943c10 0x55fb6995b790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:30:00.083 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:30:00.083+0000 7f106659c640 1 -- 172.21.15.114:0/4205440525 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x55fb69934e60 msgr2=0x55fb69943040 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T10:30:00.083 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:30:00.083+0000 7f106659c640 1 --2- 172.21.15.114:0/4205440525 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x55fb69934e60 0x55fb69943040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:30:00.083 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:30:00.083+0000 7f106659c640 1 -- 172.21.15.114:0/4205440525 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55fb699561c0 con 0x55fb69934540 2024-04-23T10:30:00.083 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:30:00.083+0000 7f106659c640 1 --2- 172.21.15.114:0/4205440525 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55fb69934540 0x55fb6994f700 secure :-1 s=READY pgs=612 cs=0 l=1 rev1=1 crypto rx=0x7f106009c2b0 tx=0x7f106006ee10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9bd02d1a11440488 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T10:30:00.083 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:30:00.083+0000 7f104ffff640 1 -- 172.21.15.114:0/4205440525 <== mon.0 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1060020040 con 0x55fb69934540 2024-04-23T10:30:00.083 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:30:00.083+0000 7f104ffff640 1 -- 172.21.15.114:0/4205440525 <== mon.0 v2:172.21.15.114:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f10600a4f90 con 0x55fb69934540 2024-04-23T10:30:00.083 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:30:00.083+0000 7f104ffff640 1 -- 172.21.15.114:0/4205440525 <== mon.0 v2:172.21.15.114:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1060098990 con 0x55fb69934540 2024-04-23T10:30:00.086 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:30:00.083+0000 7f106f81d740 1 -- 172.21.15.114:0/4205440525 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55fb69934540 msgr2=0x55fb6994f700 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T10:30:00.087 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:30:00.083+0000 7f106f81d740 1 --2- 172.21.15.114:0/4205440525 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55fb69934540 0x55fb6994f700 secure :-1 s=READY pgs=612 cs=0 l=1 rev1=1 crypto rx=0x7f106009c2b0 tx=0x7f106006ee10 comp rx=0 tx=0).stop 2024-04-23T10:30:00.087 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:30:00.084+0000 7f106f81d740 1 -- 172.21.15.114:0/4205440525 shutdown_connections 2024-04-23T10:30:00.087 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:30:00.084+0000 7f106f81d740 1 --2- 172.21.15.114:0/4205440525 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x55fb69943c10 0x55fb6995b790 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:30:00.087 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:30:00.084+0000 7f106f81d740 1 --2- 172.21.15.114:0/4205440525 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x55fb69934e60 0x55fb69943040 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:30:00.087 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:30:00.084+0000 7f106f81d740 1 --2- 172.21.15.114:0/4205440525 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55fb69934540 0x55fb6994f700 unknown :-1 s=CLOSED pgs=612 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:30:00.087 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:30:00.084+0000 7f106f81d740 1 -- 172.21.15.114:0/4205440525 >> 172.21.15.114:0/4205440525 conn(0x55fb69930b30 msgr2=0x55fb6992fcb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T10:30:00.087 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:30:00.084+0000 7f106f81d740 1 -- 172.21.15.114:0/4205440525 shutdown_connections 2024-04-23T10:30:00.087 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:30:00.084+0000 7f106f81d740 1 -- 172.21.15.114:0/4205440525 wait complete. 2024-04-23T10:30:00.087 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:30:00.084+0000 7f106f81d740 1 -- start start 2024-04-23T10:30:00.087 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:30:00.084+0000 7f106f81d740 1 Processor -- start 2024-04-23T10:30:00.087 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:30:00.084+0000 7f106f81d740 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x55fb69934540 0x55fb6994bd60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T10:30:00.087 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:30:00.084+0000 7f106f81d740 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x55fb69934e60 0x55fb699471f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T10:30:00.087 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:30:00.084+0000 7f106f81d740 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55fb69943c10 0x55fb69947730 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T10:30:00.087 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:30:00.084+0000 7f106f81d740 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x7f10681aa000 con 0x55fb69943c10 2024-04-23T10:30:00.088 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:30:00.084+0000 7f106f81d740 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 -- 0x7f10681ab470 con 0x55fb69934e60 2024-04-23T10:30:00.088 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:30:00.084+0000 7f106f81d740 1 -- --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_getmap magic: 0 -- 0x7f10681aa6d0 con 0x55fb69934540 2024-04-23T10:30:00.088 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:30:00.084+0000 7f106f81d740 10 client.0 _ll_register_callbacks cb 0x7f1058015640 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-23T10:30:00.088 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:30:00.084+0000 7f1066d9d640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55fb69943c10 0x55fb69947730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T10:30:00.088 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:30:00.084+0000 7f1066d9d640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55fb69943c10 0x55fb69947730 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.114:48576/0 (socket says 172.21.15.114:48576) 2024-04-23T10:30:00.088 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:30:00.084+0000 7f1066d9d640 1 -- 172.21.15.114:0/3286927427 learned_addr learned my addr 172.21.15.114:0/3286927427 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-04-23T10:30:00.088 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:30:00.084+0000 7f106659c640 1 --2- 172.21.15.114:0/3286927427 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x55fb69934540 0x55fb6994bd60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T10:30:00.088 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:30:00.084+0000 7f1065d9b640 1 --2- 172.21.15.114:0/3286927427 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x55fb69934e60 0x55fb699471f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T10:30:00.088 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:30:00.085+0000 7f1066d9d640 1 -- 172.21.15.114:0/3286927427 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x55fb69934540 msgr2=0x55fb6994bd60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T10:30:00.088 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:30:00.085+0000 7f1066d9d640 1 --2- 172.21.15.114:0/3286927427 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x55fb69934540 0x55fb6994bd60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:30:00.088 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:30:00.085+0000 7f1066d9d640 1 -- 172.21.15.114:0/3286927427 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x55fb69934e60 msgr2=0x55fb699471f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T10:30:00.088 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:30:00.085+0000 7f1066d9d640 1 --2- 172.21.15.114:0/3286927427 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x55fb69934e60 0x55fb699471f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:30:00.088 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:30:00.085+0000 7f1066d9d640 1 -- 172.21.15.114:0/3286927427 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55fb69947cf0 con 0x55fb69943c10 2024-04-23T10:30:00.088 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:30:00.085+0000 7f1066d9d640 1 --2- 172.21.15.114:0/3286927427 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55fb69943c10 0x55fb69947730 secure :-1 s=READY pgs=613 cs=0 l=1 rev1=1 crypto rx=0x7f1058030440 tx=0x7f1058009e60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e382d3c7b63409ac server_cookie=0 in_seq=0 out_seq=0 2024-04-23T10:30:00.088 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:30:00.085+0000 7f104dffb640 10 client.0 ms_handle_connect on v2:172.21.15.114:3300/0 2024-04-23T10:30:00.088 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:30:00.085+0000 7f104dffb640 1 -- 172.21.15.114:0/3286927427 <== mon.0 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1058022040 con 0x55fb69943c10 2024-04-23T10:30:00.089 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:30:00.085+0000 7f104dffb640 1 -- 172.21.15.114:0/3286927427 <== mon.0 v2:172.21.15.114:3300/0 2 ==== osd_map(92..92 src has 1..92) ==== 8585+0+0 (secure 0 0 0) 0x7f1058030d50 con 0x55fb69943c10 2024-04-23T10:30:00.089 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:30:00.085+0000 7f106f81d740 1 -- 172.21.15.114:0/3286927427 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f10600a4f90 con 0x55fb69943c10 2024-04-23T10:30:00.089 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:30:00.085+0000 7f104dffb640 1 -- 172.21.15.114:0/3286927427 <== mon.0 v2:172.21.15.114:3300/0 3 ==== mon_get_version_reply(handle=1 version=23) ==== 24+0+0 (secure 0 0 0) 0x7f105800fc40 con 0x55fb69943c10 2024-04-23T10:30:00.089 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:30:00.085+0000 7f106f81d740 10 client.15996 fetch_fsmap learned FSMap version 23 2024-04-23T10:30:00.089 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:30:00.085+0000 7f106f81d740 1 -- 172.21.15.114:0/3286927427 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({config=0+,fsmap=23,monmap=4+}) -- 0x55fb699b9dc0 con 0x55fb69943c10 2024-04-23T10:30:00.089 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:30:00.085+0000 7f104dffb640 1 -- 172.21.15.114:0/3286927427 <== mon.0 v2:172.21.15.114:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f10600a4f90 con 0x55fb69943c10 2024-04-23T10:30:00.089 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:30:00.085+0000 7f104dffb640 1 -- 172.21.15.114:0/3286927427 <== mon.0 v2:172.21.15.114:3300/0 5 ==== fsmap(e 23) ==== 8350+0+0 (secure 0 0 0) 0x7f105802e070 con 0x55fb69943c10 2024-04-23T10:30:00.089 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:30:00.085+0000 7f106f81d740 10 client.15996 fetch_fsmap finished waiting for FSMap version 23 2024-04-23T10:30:00.089 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:30:00.085+0000 7f106f81d740 10 client.15996 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-04-23T10:30:00.089 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:30:00.085+0000 7f106f81d740 20 client.15996 populate_metadata read hostname 'smithi114' 2024-04-23T10:30:00.089 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:30:00.085+0000 7f106f81d740 1 --2- 172.21.15.114:0/3286927427 >> [v2:172.21.15.119:6838/937297340,v1:172.21.15.119:6839/937297340] conn(0x55fb699ba510 0x55fb699bc930 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T10:30:00.089 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:30:00.085+0000 7f106f81d740 4 client.15996 mds_command: new command op to 24545 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-04-23T10:30:00.089 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:30:00.085+0000 7f106f81d740 1 -- 172.21.15.114:0/3286927427 --> [v2:172.21.15.119:6838/937297340,v1:172.21.15.119:6839/937297340] -- command(tid 0: {"prefix": "flush journal"}) -- 0x55fb698a1200 con 0x55fb699ba510 2024-04-23T10:30:00.089 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:30:00.086+0000 7f106659c640 1 --2- 172.21.15.114:0/3286927427 >> [v2:172.21.15.119:6838/937297340,v1:172.21.15.119:6839/937297340] conn(0x55fb699ba510 0x55fb699bc930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T10:30:00.089 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:30:00.086+0000 7f104dffb640 10 client.15996 ms_handle_connect on v2:172.21.15.119:6838/937297340 2024-04-23T10:30:00.090 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:30:00.086+0000 7f106659c640 1 --2- 172.21.15.114:0/3286927427 >> [v2:172.21.15.119:6838/937297340,v1:172.21.15.119:6839/937297340] conn(0x55fb699ba510 0x55fb699bc930 crc :-1 s=READY pgs=83 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=b9e0ad5d33abc162 server_cookie=d95ef2ae6ba4e166 in_seq=0 out_seq=0 2024-04-23T10:30:00.107 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:30:00.107+0000 7f104dffb640 1 -- 172.21.15.114:0/3286927427 <== mds.0 v2:172.21.15.119:6838/937297340 1 ==== command_reply(tid 0: 0 ) ==== 8+0+44 (crc 0 0 0) 0x55fb698a1200 con 0x55fb699ba510 2024-04-23T10:30:00.107 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:30:00.107+0000 7f104dffb640 10 client.15996 handle_command_reply: tid=0 2024-04-23T10:30:00.107 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:30:00.107+0000 7f106f81d740 1 client.15996 shutdown 2024-04-23T10:30:00.107 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:30:00.107+0000 7f106f81d740 1 -- 172.21.15.114:0/3286927427 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55fb69943c10 msgr2=0x55fb69947730 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T10:30:00.107 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:30:00.107+0000 7f106f81d740 1 --2- 172.21.15.114:0/3286927427 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55fb69943c10 0x55fb69947730 secure :-1 s=READY pgs=613 cs=0 l=1 rev1=1 crypto rx=0x7f1058030440 tx=0x7f1058009e60 comp rx=0 tx=0).stop 2024-04-23T10:30:00.107 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:30:00.107+0000 7f106f81d740 1 -- 172.21.15.114:0/3286927427 shutdown_connections 2024-04-23T10:30:00.107 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:30:00.107+0000 7f106f81d740 1 --2- 172.21.15.114:0/3286927427 >> [v2:172.21.15.119:6838/937297340,v1:172.21.15.119:6839/937297340] conn(0x55fb699ba510 0x55fb699bc930 crc :-1 s=READY pgs=83 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:30:00.107 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:30:00.107+0000 7f106f81d740 1 --2- 172.21.15.114:0/3286927427 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55fb69943c10 0x55fb69947730 unknown :-1 s=CLOSED pgs=613 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:30:00.107 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:30:00.107+0000 7f106f81d740 1 --2- 172.21.15.114:0/3286927427 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x55fb69934e60 0x55fb699471f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:30:00.107 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:30:00.107+0000 7f104dffb640 0 client.15996 ms_handle_reset on v2:172.21.15.119:6838/937297340 2024-04-23T10:30:00.107 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:30:00.107+0000 7f106f81d740 1 --2- 172.21.15.114:0/3286927427 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x55fb69934540 0x55fb6994bd60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:30:00.107 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:30:00.107+0000 7f106f81d740 1 -- 172.21.15.114:0/3286927427 >> 172.21.15.114:0/3286927427 conn(0x55fb69930b30 msgr2=0x55fb69940060 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T10:30:00.107 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:30:00.108+0000 7f106f81d740 1 -- 172.21.15.114:0/3286927427 shutdown_connections 2024-04-23T10:30:00.107 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:30:00.108+0000 7f106f81d740 1 -- 172.21.15.114:0/3286927427 wait complete. 2024-04-23T10:30:00.108 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:30:00.108+0000 7f106f81d740 20 client.15996 trim_cache size 0 max 16384 2024-04-23T10:30:00.108 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stdout:{ 2024-04-23T10:30:00.108 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stdout: "message": "", 2024-04-23T10:30:00.108 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stdout: "return_code": 0 2024-04-23T10:30:00.108 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stdout:} 2024-04-23T10:30:00.109 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:30:00.108+0000 7f106cb08640 1 -- 172.21.15.114:0/2172959367 >> [v2:172.21.15.114:6800/1395693466,v1:172.21.15.114:6801/1395693466] conn(0x7f10500a3dc0 msgr2=0x7f10500a6280 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T10:30:00.109 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:30:00.108+0000 7f106cb08640 1 --2- 172.21.15.114:0/2172959367 >> [v2:172.21.15.114:6800/1395693466,v1:172.21.15.114:6801/1395693466] conn(0x7f10500a3dc0 0x7f10500a6280 secure :-1 s=READY pgs=90 cs=0 l=1 rev1=1 crypto rx=0x7f105801f000 tx=0x7f105801b000 comp rx=0 tx=0).stop 2024-04-23T10:30:00.109 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:30:00.108+0000 7f106cb08640 1 -- 172.21.15.114:0/2172959367 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f10681021d0 msgr2=0x7f10681986e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T10:30:00.109 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:30:00.108+0000 7f106cb08640 1 --2- 172.21.15.114:0/2172959367 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f10681021d0 0x7f10681986e0 secure :-1 s=READY pgs=609 cs=0 l=1 rev1=1 crypto rx=0x7f106000d770 tx=0x7f106000dc30 comp rx=0 tx=0).stop 2024-04-23T10:30:00.109 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:30:00.108+0000 7f106cb08640 1 -- 172.21.15.114:0/2172959367 shutdown_connections 2024-04-23T10:30:00.109 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:30:00.108+0000 7f106cb08640 1 --2- 172.21.15.114:0/2172959367 >> [v2:172.21.15.114:6800/1395693466,v1:172.21.15.114:6801/1395693466] conn(0x7f10500a3dc0 0x7f10500a6280 unknown :-1 s=CLOSED pgs=90 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:30:00.109 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:30:00.108+0000 7f106cb08640 1 --2- 172.21.15.114:0/2172959367 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f106819cfa0 0x7f106819d450 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:30:00.109 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:30:00.108+0000 7f106cb08640 1 --2- 172.21.15.114:0/2172959367 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f10681072a0 0x7f1068198c20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:30:00.109 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:30:00.108+0000 7f106cb08640 1 --2- 172.21.15.114:0/2172959367 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f10681021d0 0x7f10681986e0 unknown :-1 s=CLOSED pgs=609 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:30:00.109 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:30:00.108+0000 7f106cb08640 1 -- 172.21.15.114:0/2172959367 >> 172.21.15.114:0/2172959367 conn(0x7f10680793e0 msgr2=0x7f10680ffe00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T10:30:00.109 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:30:00.108+0000 7f106cb08640 1 -- 172.21.15.114:0/2172959367 shutdown_connections 2024-04-23T10:30:00.109 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:30:00.108+0000 7f106cb08640 1 -- 172.21.15.114:0/2172959367 wait complete. 2024-04-23T10:30:00.241 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 23 10:29:59 smithi119 ceph-mon[32608]: pgmap v104: 97 pgs: 97 active+clean; 52 MiB data, 851 MiB used, 1.0 TiB / 1.0 TiB avail; 67 KiB/s rd, 343 KiB/s wr, 42 op/s 2024-04-23T10:30:00.254 INFO:journalctl@ceph.mon.a.smithi114.stdout:Apr 23 10:29:59 smithi114 ceph-mon[29255]: pgmap v104: 97 pgs: 97 active+clean; 52 MiB data, 851 MiB used, 1.0 TiB / 1.0 TiB avail; 67 KiB/s rd, 343 KiB/s wr, 42 op/s 2024-04-23T10:30:00.327 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 23 10:29:59 smithi204 ceph-mon[32649]: pgmap v104: 97 pgs: 97 active+clean; 52 MiB data, 851 MiB used, 1.0 TiB / 1.0 TiB avail; 67 KiB/s rd, 343 KiB/s wr, 42 op/s 2024-04-23T10:30:00.770 DEBUG:teuthology.orchestra.run.smithi114:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-23T10:30:00.877 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:30:00.876+0000 7f37ef74f640 1 -- 172.21.15.114:0/2478662785 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f37e8101ac0 msgr2=0x7f37e8105ab0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T10:30:00.877 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:30:00.876+0000 7f37ef74f640 1 --2- 172.21.15.114:0/2478662785 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f37e8101ac0 0x7f37e8105ab0 secure :-1 s=READY pgs=614 cs=0 l=1 rev1=1 crypto rx=0x7f37d8009900 tx=0x7f37d802a410 comp rx=0 tx=0).stop 2024-04-23T10:30:00.877 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:30:00.876+0000 7f37ef74f640 1 -- 172.21.15.114:0/2478662785 shutdown_connections 2024-04-23T10:30:00.877 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:30:00.876+0000 7f37ef74f640 1 --2- 172.21.15.114:0/2478662785 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f37e8106190 0x7f37e810dd60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:30:00.877 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:30:00.876+0000 7f37ef74f640 1 --2- 172.21.15.114:0/2478662785 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f37e8101ac0 0x7f37e8105ab0 unknown :-1 s=CLOSED pgs=614 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:30:00.878 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:30:00.876+0000 7f37ef74f640 1 --2- 172.21.15.114:0/2478662785 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f37e8101110 0x7f37e81014f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:30:00.878 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:30:00.876+0000 7f37ef74f640 1 -- 172.21.15.114:0/2478662785 >> 172.21.15.114:0/2478662785 conn(0x7f37e8077fb0 msgr2=0x7f37e80783c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T10:30:00.878 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:30:00.876+0000 7f37ef74f640 1 -- 172.21.15.114:0/2478662785 shutdown_connections 2024-04-23T10:30:00.878 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:30:00.876+0000 7f37ef74f640 1 -- 172.21.15.114:0/2478662785 wait complete. 2024-04-23T10:30:00.878 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:30:00.877+0000 7f37ef74f640 1 Processor -- start 2024-04-23T10:30:00.878 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:30:00.877+0000 7f37ef74f640 1 -- start start 2024-04-23T10:30:00.878 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:30:00.877+0000 7f37ef74f640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f37e8101110 0x7f37e819e450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T10:30:00.878 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:30:00.877+0000 7f37ef74f640 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f37e8106190 0x7f37e819e990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T10:30:00.878 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:30:00.877+0000 7f37ef74f640 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f37e8198600 0x7f37e8198ab0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T10:30:00.878 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:30:00.877+0000 7f37ef74f640 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x7f37e81110b0 con 0x7f37e8101110 2024-04-23T10:30:00.878 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:30:00.877+0000 7f37ef74f640 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 -- 0x7f37e8110f30 con 0x7f37e8106190 2024-04-23T10:30:00.878 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:30:00.877+0000 7f37ef74f640 1 -- --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_getmap magic: 0 -- 0x7f37e8111230 con 0x7f37e8198600 2024-04-23T10:30:00.878 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:30:00.877+0000 7f37ed4eb640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f37e8101110 0x7f37e819e450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T10:30:00.878 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:30:00.877+0000 7f37ed4eb640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f37e8101110 0x7f37e819e450 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.114:48584/0 (socket says 172.21.15.114:48584) 2024-04-23T10:30:00.879 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:30:00.877+0000 7f37ed4eb640 1 -- 172.21.15.114:0/3734756329 learned_addr learned my addr 172.21.15.114:0/3734756329 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-04-23T10:30:00.879 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:30:00.877+0000 7f37eccea640 1 --2- 172.21.15.114:0/3734756329 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f37e8106190 0x7f37e819e990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T10:30:00.879 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:30:00.877+0000 7f37edcec640 1 --2- 172.21.15.114:0/3734756329 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f37e8198600 0x7f37e8198ab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T10:30:00.879 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:30:00.878+0000 7f37ed4eb640 1 -- 172.21.15.114:0/3734756329 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f37e8198600 msgr2=0x7f37e8198ab0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T10:30:00.879 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:30:00.878+0000 7f37ed4eb640 1 --2- 172.21.15.114:0/3734756329 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f37e8198600 0x7f37e8198ab0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:30:00.879 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:30:00.878+0000 7f37ed4eb640 1 -- 172.21.15.114:0/3734756329 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f37e8106190 msgr2=0x7f37e819e990 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T10:30:00.879 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:30:00.878+0000 7f37ed4eb640 1 --2- 172.21.15.114:0/3734756329 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f37e8106190 0x7f37e819e990 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:30:00.879 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:30:00.878+0000 7f37ed4eb640 1 -- 172.21.15.114:0/3734756329 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f37e8199120 con 0x7f37e8101110 2024-04-23T10:30:00.879 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:30:00.878+0000 7f37eccea640 1 --2- 172.21.15.114:0/3734756329 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f37e8106190 0x7f37e819e990 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-23T10:30:00.879 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:30:00.878+0000 7f37ed4eb640 1 --2- 172.21.15.114:0/3734756329 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f37e8101110 0x7f37e819e450 secure :-1 s=READY pgs=615 cs=0 l=1 rev1=1 crypto rx=0x7f37e400c890 tx=0x7f37e400cd50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T10:30:00.879 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:30:00.878+0000 7f37de7fc640 1 -- 172.21.15.114:0/3734756329 <== mon.0 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f37e4004220 con 0x7f37e8101110 2024-04-23T10:30:00.879 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:30:00.878+0000 7f37de7fc640 1 -- 172.21.15.114:0/3734756329 <== mon.0 v2:172.21.15.114:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f37e40043c0 con 0x7f37e8101110 2024-04-23T10:30:00.879 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:30:00.878+0000 7f37ef74f640 1 -- 172.21.15.114:0/3734756329 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f37e8199410 con 0x7f37e8101110 2024-04-23T10:30:00.879 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:30:00.878+0000 7f37de7fc640 1 -- 172.21.15.114:0/3734756329 <== mon.0 v2:172.21.15.114:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f37e4004a00 con 0x7f37e8101110 2024-04-23T10:30:00.879 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:30:00.878+0000 7f37ef74f640 1 -- 172.21.15.114:0/3734756329 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f37e81a5660 con 0x7f37e8101110 2024-04-23T10:30:00.880 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:30:00.879+0000 7f37ef74f640 1 -- 172.21.15.114:0/3734756329 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f37e80691c0 con 0x7f37e8101110 2024-04-23T10:30:00.880 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:30:00.880+0000 7f37de7fc640 1 -- 172.21.15.114:0/3734756329 <== mon.0 v2:172.21.15.114:3300/0 4 ==== mgrmap(e 26) ==== 138778+0+0 (secure 0 0 0) 0x7f37e40079b0 con 0x7f37e8101110 2024-04-23T10:30:00.880 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:30:00.880+0000 7f37de7fc640 1 --2- 172.21.15.114:0/3734756329 >> [v2:172.21.15.114:6800/1395693466,v1:172.21.15.114:6801/1395693466] conn(0x7f37d40a3d20 0x7f37d40a61e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T10:30:00.880 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:30:00.881+0000 7f37eccea640 1 --2- 172.21.15.114:0/3734756329 >> [v2:172.21.15.114:6800/1395693466,v1:172.21.15.114:6801/1395693466] conn(0x7f37d40a3d20 0x7f37d40a61e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T10:30:00.881 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:30:00.881+0000 7f37de7fc640 1 -- 172.21.15.114:0/3734756329 <== mon.0 v2:172.21.15.114:3300/0 5 ==== osd_map(92..92 src has 1..92) ==== 8585+0+0 (secure 0 0 0) 0x7f37e40d0a50 con 0x7f37e8101110 2024-04-23T10:30:00.881 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:30:00.881+0000 7f37eccea640 1 --2- 172.21.15.114:0/3734756329 >> [v2:172.21.15.114:6800/1395693466,v1:172.21.15.114:6801/1395693466] conn(0x7f37d40a3d20 0x7f37d40a61e0 secure :-1 s=READY pgs=91 cs=0 l=1 rev1=1 crypto rx=0x7f37d8000c00 tx=0x7f37d8009610 comp rx=0 tx=0).ready entity=mgr.15291 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T10:30:00.883 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:30:00.884+0000 7f37de7fc640 1 -- 172.21.15.114:0/3734756329 <== mon.0 v2:172.21.15.114:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f37e4014030 con 0x7f37e8101110 2024-04-23T10:30:00.914 INFO:tasks.workunit.client.0.smithi114.stdout: 1 178221 0.00 MB/sec execute 9 sec latency 21607.748 ms 2024-04-23T10:30:01.115 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:30:01.116+0000 7f37ef74f640 1 -- 172.21.15.114:0/3734756329 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f37e8199b30 con 0x7f37e8101110 2024-04-23T10:30:01.116 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:30:01.117+0000 7f37de7fc640 1 -- 172.21.15.114:0/3734756329 <== mon.0 v2:172.21.15.114:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 23 v23) ==== 94+0+11888 (secure 0 0 0) 0x7f37e409d0b0 con 0x7f37e8101110 2024-04-23T10:30:01.116 INFO:teuthology.orchestra.run.smithi114.stdout: 2024-04-23T10:30:01.116 INFO:teuthology.orchestra.run.smithi114.stdout:{"epoch":23,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14547,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.114:6835/663493580","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6834","nonce":663493580},{"type":"v1","addr":"172.21.15.114:6835","nonce":663493580}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14565,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.114:6837/1439252154","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6836","nonce":1439252154},{"type":"v1","addr":"172.21.15.114:6837","nonce":1439252154}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5},{"gid":14634,"name":"g","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.114:6839/3849796939","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6838","nonce":3849796939},{"type":"v1","addr":"172.21.15.114:6839","nonce":3849796939}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":11},{"gid":14646,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.114:6841/3544523891","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6840","nonce":3544523891},{"type":"v1","addr":"172.21.15.114:6841","nonce":3544523891}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":12},{"gid":24385,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.204:6833/3866561754","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6832","nonce":3866561754},{"type":"v1","addr":"172.21.15.204:6833","nonce":3866561754}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2},{"gid":24427,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.204:6835/2533793172","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6834","nonce":2533793172},{"type":"v1","addr":"172.21.15.204:6835","nonce":2533793172}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":7}],"filesystems":[{"mdsmap":{"epoch":23,"flags":306,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":true},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-04-23T10:24:43.383158+0000","modified":"2024-04-23T10:28:39.149342+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":3,"in":[0,1,2],"up":{"mds_0":24545,"mds_1":24509,"mds_2":24479},"failed":[],"damaged":[],"stopped":[],"info":{"gid_24439":{"gid":24439,"name":"c","rank":2,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.204:6837/513822179","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6836","nonce":513822179},{"type":"v1","addr":"172.21.15.204:6837","nonce":513822179}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24443":{"gid":24443,"name":"k","rank":1,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.119:6833/783244660","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6832","nonce":783244660},{"type":"v1","addr":"172.21.15.119:6833","nonce":783244660}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24463":{"gid":24463,"name":"i","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.204:6839/3222658893","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6838","nonce":3222658893},{"type":"v1","addr":"172.21.15.204:6839","nonce":3222658893}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24479":{"gid":24479,"name":"e","rank":2,"incarnation":19,"state":"up:active","state_seq":9,"addr":"172.21.15.119:6835/2033916709","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6834","nonce":2033916709},{"type":"v1","addr":"172.21.15.119:6835","nonce":2033916709}]},"join_fscid":-1,"export_targets":[0],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24509":{"gid":24509,"name":"h","rank":1,"incarnation":17,"state":"up:active","state_seq":6,"addr":"172.21.15.119:6837/1872363128","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6836","nonce":1872363128},{"type":"v1","addr":"172.21.15.119:6837","nonce":1872363128}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24545":{"gid":24545,"name":"b","rank":0,"incarnation":14,"state":"up:active","state_seq":3,"addr":"172.21.15.119:6839/937297340","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6838","nonce":937297340},{"type":"v1","addr":"172.21.15.119:6839","nonce":937297340}]},"join_fscid":-1,"export_targets":[1,2],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":24545,"qdb_cluster":[24509,24545,24479]},"id":1}]} 2024-04-23T10:30:01.117 INFO:teuthology.orchestra.run.smithi114.stderr:dumped fsmap epoch 23 2024-04-23T10:30:01.118 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:30:01.119+0000 7f37ef74f640 1 -- 172.21.15.114:0/3734756329 >> [v2:172.21.15.114:6800/1395693466,v1:172.21.15.114:6801/1395693466] conn(0x7f37d40a3d20 msgr2=0x7f37d40a61e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T10:30:01.118 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:30:01.119+0000 7f37ef74f640 1 --2- 172.21.15.114:0/3734756329 >> [v2:172.21.15.114:6800/1395693466,v1:172.21.15.114:6801/1395693466] conn(0x7f37d40a3d20 0x7f37d40a61e0 secure :-1 s=READY pgs=91 cs=0 l=1 rev1=1 crypto rx=0x7f37d8000c00 tx=0x7f37d8009610 comp rx=0 tx=0).stop 2024-04-23T10:30:01.119 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:30:01.119+0000 7f37ef74f640 1 -- 172.21.15.114:0/3734756329 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f37e8101110 msgr2=0x7f37e819e450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T10:30:01.119 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:30:01.119+0000 7f37ef74f640 1 --2- 172.21.15.114:0/3734756329 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f37e8101110 0x7f37e819e450 secure :-1 s=READY pgs=615 cs=0 l=1 rev1=1 crypto rx=0x7f37e400c890 tx=0x7f37e400cd50 comp rx=0 tx=0).stop 2024-04-23T10:30:01.119 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:30:01.119+0000 7f37ef74f640 1 -- 172.21.15.114:0/3734756329 shutdown_connections 2024-04-23T10:30:01.119 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:30:01.119+0000 7f37ef74f640 1 --2- 172.21.15.114:0/3734756329 >> [v2:172.21.15.114:6800/1395693466,v1:172.21.15.114:6801/1395693466] conn(0x7f37d40a3d20 0x7f37d40a61e0 unknown :-1 s=CLOSED pgs=91 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:30:01.119 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:30:01.119+0000 7f37ef74f640 1 --2- 172.21.15.114:0/3734756329 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f37e8198600 0x7f37e8198ab0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:30:01.119 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:30:01.119+0000 7f37ef74f640 1 --2- 172.21.15.114:0/3734756329 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f37e8106190 0x7f37e819e990 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:30:01.119 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:30:01.119+0000 7f37ef74f640 1 --2- 172.21.15.114:0/3734756329 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f37e8101110 0x7f37e819e450 unknown :-1 s=CLOSED pgs=615 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:30:01.119 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:30:01.119+0000 7f37ef74f640 1 -- 172.21.15.114:0/3734756329 >> 172.21.15.114:0/3734756329 conn(0x7f37e8077fb0 msgr2=0x7f37e8104f90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T10:30:01.119 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:30:01.119+0000 7f37ef74f640 1 -- 172.21.15.114:0/3734756329 shutdown_connections 2024-04-23T10:30:01.119 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:30:01.119+0000 7f37ef74f640 1 -- 172.21.15.114:0/3734756329 wait complete. 2024-04-23T10:30:01.128 INFO:journalctl@ceph.mon.a.smithi114.stdout:Apr 23 10:30:00 smithi114 ceph-mon[29255]: overall HEALTH_OK 2024-04-23T10:30:01.129 DEBUG:tasks.quiescer.fs.[cephfs]:Running ceph command: 'tell mds.24545 quiesce db --set-id 4e8b9ad0 --release --await' 2024-04-23T10:30:01.129 DEBUG:teuthology.orchestra.run.smithi114:> timeout 75 sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.24545 quiesce db --set-id 4e8b9ad0 --release --await 2024-04-23T10:30:01.234 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:30:01.235+0000 7f84de719640 1 -- 172.21.15.114:0/3479490704 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f84d8101a50 msgr2=0x7f84d8105a90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T10:30:01.234 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:30:01.235+0000 7f84de719640 1 --2- 172.21.15.114:0/3479490704 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f84d8101a50 0x7f84d8105a90 secure :-1 s=READY pgs=616 cs=0 l=1 rev1=1 crypto rx=0x7f84cc009900 tx=0x7f84cc02a410 comp rx=0 tx=0).stop 2024-04-23T10:30:01.234 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:30:01.235+0000 7f84de719640 1 -- 172.21.15.114:0/3479490704 shutdown_connections 2024-04-23T10:30:01.235 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:30:01.235+0000 7f84de719640 1 --2- 172.21.15.114:0/3479490704 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f84d8106170 0x7f84d810dd40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:30:01.235 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:30:01.235+0000 7f84de719640 1 --2- 172.21.15.114:0/3479490704 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f84d8101a50 0x7f84d8105a90 unknown :-1 s=CLOSED pgs=616 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:30:01.235 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:30:01.235+0000 7f84de719640 1 --2- 172.21.15.114:0/3479490704 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f84d81010a0 0x7f84d8101480 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:30:01.235 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:30:01.235+0000 7f84de719640 1 -- 172.21.15.114:0/3479490704 >> 172.21.15.114:0/3479490704 conn(0x7f84d8077eb0 msgr2=0x7f84d80782c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T10:30:01.235 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:30:01.235+0000 7f84de719640 1 -- 172.21.15.114:0/3479490704 shutdown_connections 2024-04-23T10:30:01.235 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:30:01.235+0000 7f84de719640 1 -- 172.21.15.114:0/3479490704 wait complete. 2024-04-23T10:30:01.235 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:30:01.236+0000 7f84de719640 1 Processor -- start 2024-04-23T10:30:01.235 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:30:01.236+0000 7f84de719640 1 -- start start 2024-04-23T10:30:01.237 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:30:01.236+0000 7f84de719640 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f84d81010a0 0x7f84d81986e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T10:30:01.237 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:30:01.236+0000 7f84de719640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f84d8106170 0x7f84d8198c20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T10:30:01.237 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:30:01.236+0000 7f84de719640 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f84d819cfa0 0x7f84d819d450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T10:30:01.237 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:30:01.236+0000 7f84de719640 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x7f84d8111090 con 0x7f84d8106170 2024-04-23T10:30:01.237 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:30:01.236+0000 7f84de719640 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 -- 0x7f84d8110f10 con 0x7f84d819cfa0 2024-04-23T10:30:01.237 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:30:01.236+0000 7f84de719640 1 -- --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_getmap magic: 0 -- 0x7f84d8111210 con 0x7f84d81010a0 2024-04-23T10:30:01.237 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:30:01.236+0000 7f84d77fe640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f84d8106170 0x7f84d8198c20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T10:30:01.237 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:30:01.236+0000 7f84d77fe640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f84d8106170 0x7f84d8198c20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.114:48602/0 (socket says 172.21.15.114:48602) 2024-04-23T10:30:01.237 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:30:01.236+0000 7f84d77fe640 1 -- 172.21.15.114:0/992652486 learned_addr learned my addr 172.21.15.114:0/992652486 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-04-23T10:30:01.237 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:30:01.236+0000 7f84dccb6640 1 --2- 172.21.15.114:0/992652486 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f84d819cfa0 0x7f84d819d450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T10:30:01.238 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:30:01.236+0000 7f84d7fff640 1 --2- 172.21.15.114:0/992652486 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f84d81010a0 0x7f84d81986e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T10:30:01.238 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:30:01.236+0000 7f84d77fe640 1 -- 172.21.15.114:0/992652486 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f84d81010a0 msgr2=0x7f84d81986e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T10:30:01.238 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:30:01.236+0000 7f84d77fe640 1 --2- 172.21.15.114:0/992652486 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f84d81010a0 0x7f84d81986e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:30:01.238 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:30:01.236+0000 7f84d77fe640 1 -- 172.21.15.114:0/992652486 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f84d819cfa0 msgr2=0x7f84d819d450 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T10:30:01.238 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:30:01.236+0000 7f84d77fe640 1 --2- 172.21.15.114:0/992652486 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f84d819cfa0 0x7f84d819d450 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:30:01.238 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:30:01.236+0000 7f84d77fe640 1 -- 172.21.15.114:0/992652486 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f84d819da90 con 0x7f84d8106170 2024-04-23T10:30:01.238 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:30:01.237+0000 7f84d77fe640 1 --2- 172.21.15.114:0/992652486 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f84d8106170 0x7f84d8198c20 secure :-1 s=READY pgs=617 cs=0 l=1 rev1=1 crypto rx=0x7f84cc032fd0 tx=0x7f84cc004270 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T10:30:01.238 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:30:01.237+0000 7f84d57fa640 1 -- 172.21.15.114:0/992652486 <== mon.0 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f84cc03f030 con 0x7f84d8106170 2024-04-23T10:30:01.238 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:30:01.237+0000 7f84de719640 1 -- 172.21.15.114:0/992652486 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f84d819dd20 con 0x7f84d8106170 2024-04-23T10:30:01.238 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:30:01.237+0000 7f84d57fa640 1 -- 172.21.15.114:0/992652486 <== mon.0 v2:172.21.15.114:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f84cc004490 con 0x7f84d8106170 2024-04-23T10:30:01.238 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:30:01.237+0000 7f84d57fa640 1 -- 172.21.15.114:0/992652486 <== mon.0 v2:172.21.15.114:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f84cc035070 con 0x7f84d8106170 2024-04-23T10:30:01.238 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:30:01.237+0000 7f84de719640 1 -- 172.21.15.114:0/992652486 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f84d81a54f0 con 0x7f84d8106170 2024-04-23T10:30:01.244 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 23 10:30:00 smithi119 ceph-mon[32608]: overall HEALTH_OK 2024-04-23T10:30:01.245 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:30:01.239+0000 7f84d57fa640 1 -- 172.21.15.114:0/992652486 <== mon.0 v2:172.21.15.114:3300/0 4 ==== mgrmap(e 26) ==== 138778+0+0 (secure 0 0 0) 0x7f84cc042030 con 0x7f84d8106170 2024-04-23T10:30:01.245 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:30:01.239+0000 7f84e142b740 1 Processor -- start 2024-04-23T10:30:01.245 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:30:01.239+0000 7f84e142b740 1 -- start start 2024-04-23T10:30:01.245 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:30:01.239+0000 7f84e142b740 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x5574697e3490 0x5574697e3870 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T10:30:01.245 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:30:01.239+0000 7f84e142b740 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x5574697e3db0 0x5574697edf40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T10:30:01.246 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:30:01.239+0000 7f84e142b740 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x5574697ee480 0x5574697f0870 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T10:30:01.246 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:30:01.239+0000 7f84e142b740 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x5574697486d0 con 0x5574697e3490 2024-04-23T10:30:01.246 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:30:01.239+0000 7f84e142b740 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 -- 0x5574696e51a0 con 0x5574697e3db0 2024-04-23T10:30:01.246 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:30:01.239+0000 7f84e142b740 1 -- --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_getmap magic: 0 -- 0x557469744690 con 0x5574697ee480 2024-04-23T10:30:01.246 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:30:01.239+0000 7f84dccb6640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x5574697e3490 0x5574697e3870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T10:30:01.246 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:30:01.239+0000 7f84dccb6640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x5574697e3490 0x5574697e3870 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.114:48604/0 (socket says 172.21.15.114:48604) 2024-04-23T10:30:01.246 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:30:01.239+0000 7f84dccb6640 1 -- 172.21.15.114:0/3755718391 learned_addr learned my addr 172.21.15.114:0/3755718391 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-04-23T10:30:01.246 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:30:01.240+0000 7f84d77fe640 1 --2- 172.21.15.114:0/3755718391 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x5574697ee480 0x5574697f0870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T10:30:01.246 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:30:01.240+0000 7f84d7fff640 1 --2- 172.21.15.114:0/3755718391 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x5574697e3db0 0x5574697edf40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T10:30:01.246 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:30:01.240+0000 7f84d57fa640 1 --2- 172.21.15.114:0/992652486 >> [v2:172.21.15.114:6800/1395693466,v1:172.21.15.114:6801/1395693466] conn(0x7f84c00a3d20 0x7f84c00a61e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T10:30:01.246 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:30:01.240+0000 7f84dccb6640 1 -- 172.21.15.114:0/3755718391 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x5574697ee480 msgr2=0x5574697f0870 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T10:30:01.246 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:30:01.240+0000 7f84dccb6640 1 --2- 172.21.15.114:0/3755718391 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x5574697ee480 0x5574697f0870 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:30:01.246 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:30:01.240+0000 7f84dccb6640 1 -- 172.21.15.114:0/3755718391 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x5574697e3db0 msgr2=0x5574697edf40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T10:30:01.246 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:30:01.240+0000 7f84dccb6640 1 --2- 172.21.15.114:0/3755718391 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x5574697e3db0 0x5574697edf40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:30:01.246 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:30:01.240+0000 7f84dccb6640 1 -- 172.21.15.114:0/3755718391 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5574696ef5b0 con 0x5574697e3490 2024-04-23T10:30:01.247 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:30:01.240+0000 7f84dccb6640 1 --2- 172.21.15.114:0/992652486 >> [v2:172.21.15.114:6800/1395693466,v1:172.21.15.114:6801/1395693466] conn(0x7f84c00a3d20 0x7f84c00a61e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T10:30:01.247 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:30:01.240+0000 7f84d57fa640 1 -- 172.21.15.114:0/992652486 <== mon.0 v2:172.21.15.114:3300/0 5 ==== osd_map(92..92 src has 1..92) ==== 8585+0+0 (secure 0 0 0) 0x7f84cc0f0cf0 con 0x7f84d8106170 2024-04-23T10:30:01.247 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:30:01.240+0000 7f84dccb6640 1 --2- 172.21.15.114:0/3755718391 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x5574697e3490 0x5574697e3870 secure :-1 s=READY pgs=618 cs=0 l=1 rev1=1 crypto rx=0x7f84d0010440 tx=0x7f84d0010900 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a4201a775ed540f0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T10:30:01.247 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:30:01.240+0000 7f84be7fc640 1 -- 172.21.15.114:0/3755718391 <== mon.0 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f84d001d400 con 0x5574697e3490 2024-04-23T10:30:01.247 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:30:01.240+0000 7f84be7fc640 1 -- 172.21.15.114:0/3755718391 <== mon.0 v2:172.21.15.114:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f84d00134c0 con 0x5574697e3490 2024-04-23T10:30:01.247 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:30:01.240+0000 7f84be7fc640 1 -- 172.21.15.114:0/3755718391 <== mon.0 v2:172.21.15.114:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f84d00117e0 con 0x5574697e3490 2024-04-23T10:30:01.247 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:30:01.241+0000 7f84e142b740 1 -- 172.21.15.114:0/3755718391 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x5574697e3490 msgr2=0x5574697e3870 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T10:30:01.247 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:30:01.241+0000 7f84dccb6640 1 --2- 172.21.15.114:0/992652486 >> [v2:172.21.15.114:6800/1395693466,v1:172.21.15.114:6801/1395693466] conn(0x7f84c00a3d20 0x7f84c00a61e0 secure :-1 s=READY pgs=92 cs=0 l=1 rev1=1 crypto rx=0x7f84d0021000 tx=0x7f84d001e000 comp rx=0 tx=0).ready entity=mgr.15291 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T10:30:01.247 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:30:01.241+0000 7f84e142b740 1 --2- 172.21.15.114:0/3755718391 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x5574697e3490 0x5574697e3870 secure :-1 s=READY pgs=618 cs=0 l=1 rev1=1 crypto rx=0x7f84d0010440 tx=0x7f84d0010900 comp rx=0 tx=0).stop 2024-04-23T10:30:01.247 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:30:01.241+0000 7f84e142b740 1 -- 172.21.15.114:0/3755718391 shutdown_connections 2024-04-23T10:30:01.247 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:30:01.241+0000 7f84e142b740 1 --2- 172.21.15.114:0/3755718391 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x5574697ee480 0x5574697f0870 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:30:01.247 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:30:01.241+0000 7f84e142b740 1 --2- 172.21.15.114:0/3755718391 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x5574697e3db0 0x5574697edf40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:30:01.247 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:30:01.241+0000 7f84e142b740 1 --2- 172.21.15.114:0/3755718391 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x5574697e3490 0x5574697e3870 unknown :-1 s=CLOSED pgs=618 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:30:01.247 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:30:01.241+0000 7f84e142b740 1 -- 172.21.15.114:0/3755718391 >> 172.21.15.114:0/3755718391 conn(0x5574697df7b0 msgr2=0x5574697dfb90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T10:30:01.247 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:30:01.241+0000 7f84e142b740 1 -- 172.21.15.114:0/3755718391 shutdown_connections 2024-04-23T10:30:01.248 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:30:01.241+0000 7f84e142b740 1 -- 172.21.15.114:0/3755718391 wait complete. 2024-04-23T10:30:01.248 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:30:01.241+0000 7f84e142b740 1 -- start start 2024-04-23T10:30:01.248 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:30:01.241+0000 7f84e142b740 1 Processor -- start 2024-04-23T10:30:01.248 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:30:01.242+0000 7f84e142b740 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x5574697e3490 0x5574697edf00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T10:30:01.248 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:30:01.242+0000 7f84e142b740 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x5574697e3db0 0x5574697eb1f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T10:30:01.248 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:30:01.242+0000 7f84e142b740 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x5574697ee480 0x5574697eb930 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T10:30:01.248 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:30:01.242+0000 7f84e142b740 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x5574697f1310 con 0x5574697e3490 2024-04-23T10:30:01.248 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:30:01.242+0000 7f84e142b740 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 -- 0x5574696e2b20 con 0x5574697ee480 2024-04-23T10:30:01.248 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:30:01.242+0000 7f84e142b740 1 -- --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_getmap magic: 0 -- 0x5574697f1490 con 0x5574697e3db0 2024-04-23T10:30:01.248 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:30:01.242+0000 7f84e142b740 10 client.0 _ll_register_callbacks cb 0x5574696e2530 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-23T10:30:01.248 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:30:01.242+0000 7f84d7fff640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x5574697e3490 0x5574697edf00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T10:30:01.248 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:30:01.242+0000 7f84d7fff640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x5574697e3490 0x5574697edf00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.114:48606/0 (socket says 172.21.15.114:48606) 2024-04-23T10:30:01.248 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:30:01.242+0000 7f84d7fff640 1 -- 172.21.15.114:0/1371277440 learned_addr learned my addr 172.21.15.114:0/1371277440 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-04-23T10:30:01.248 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:30:01.242+0000 7f84dccb6640 1 --2- 172.21.15.114:0/1371277440 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x5574697ee480 0x5574697eb930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T10:30:01.248 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:30:01.242+0000 7f84d77fe640 1 --2- 172.21.15.114:0/1371277440 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x5574697e3db0 0x5574697eb1f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T10:30:01.249 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:30:01.242+0000 7f84d7fff640 1 -- 172.21.15.114:0/1371277440 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x5574697e3db0 msgr2=0x5574697eb1f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T10:30:01.249 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:30:01.242+0000 7f84d7fff640 1 --2- 172.21.15.114:0/1371277440 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x5574697e3db0 0x5574697eb1f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:30:01.249 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:30:01.242+0000 7f84d7fff640 1 -- 172.21.15.114:0/1371277440 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x5574697ee480 msgr2=0x5574697eb930 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T10:30:01.249 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:30:01.242+0000 7f84d7fff640 1 --2- 172.21.15.114:0/1371277440 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x5574697ee480 0x5574697eb930 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:30:01.249 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:30:01.242+0000 7f84d7fff640 1 -- 172.21.15.114:0/1371277440 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55746945a920 con 0x5574697e3490 2024-04-23T10:30:01.249 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:30:01.242+0000 7f84d7fff640 1 --2- 172.21.15.114:0/1371277440 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x5574697e3490 0x5574697edf00 secure :-1 s=READY pgs=619 cs=0 l=1 rev1=1 crypto rx=0x7f84c8010810 tx=0x7f84c8010cd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6714b40de420e807 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T10:30:01.249 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:30:01.242+0000 7f84b77fe640 10 client.0 ms_handle_connect on v2:172.21.15.114:3300/0 2024-04-23T10:30:01.249 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:30:01.242+0000 7f84b77fe640 1 -- 172.21.15.114:0/1371277440 <== mon.0 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f84c800cab0 con 0x5574697e3490 2024-04-23T10:30:01.249 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:30:01.242+0000 7f84b77fe640 1 -- 172.21.15.114:0/1371277440 <== mon.0 v2:172.21.15.114:3300/0 2 ==== osd_map(92..92 src has 1..92) ==== 8585+0+0 (secure 0 0 0) 0x7f84c800cc50 con 0x5574697e3490 2024-04-23T10:30:01.249 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:30:01.242+0000 7f84e142b740 1 -- 172.21.15.114:0/1371277440 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f84d00134c0 con 0x5574697e3490 2024-04-23T10:30:01.249 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:30:01.243+0000 7f84b77fe640 1 -- 172.21.15.114:0/1371277440 <== mon.0 v2:172.21.15.114:3300/0 3 ==== mon_get_version_reply(handle=1 version=23) ==== 24+0+0 (secure 0 0 0) 0x7f84c800ce70 con 0x5574697e3490 2024-04-23T10:30:01.249 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:30:01.243+0000 7f84e142b740 10 client.16014 fetch_fsmap learned FSMap version 23 2024-04-23T10:30:01.249 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:30:01.243+0000 7f84e142b740 1 -- 172.21.15.114:0/1371277440 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({config=0+,fsmap=23,monmap=4+}) -- 0x557469526d30 con 0x5574697e3490 2024-04-23T10:30:01.249 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:30:01.243+0000 7f84b77fe640 1 -- 172.21.15.114:0/1371277440 <== mon.0 v2:172.21.15.114:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f84d00134c0 con 0x5574697e3490 2024-04-23T10:30:01.249 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:30:01.243+0000 7f84b77fe640 1 -- 172.21.15.114:0/1371277440 <== mon.0 v2:172.21.15.114:3300/0 5 ==== fsmap(e 23) ==== 8350+0+0 (secure 0 0 0) 0x7f84c8020030 con 0x5574697e3490 2024-04-23T10:30:01.249 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:30:01.243+0000 7f84e142b740 10 client.16014 fetch_fsmap finished waiting for FSMap version 23 2024-04-23T10:30:01.250 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:30:01.243+0000 7f84e142b740 10 client.16014 resolve_mds: validated gid 24545 aka daemon mds.b 2024-04-23T10:30:01.250 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:30:01.243+0000 7f84e142b740 20 client.16014 populate_metadata read hostname 'smithi114' 2024-04-23T10:30:01.250 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:30:01.243+0000 7f84e142b740 1 --2- 172.21.15.114:0/1371277440 >> [v2:172.21.15.119:6838/937297340,v1:172.21.15.119:6839/937297340] conn(0x5574697fe740 0x557469800bc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T10:30:01.250 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:30:01.243+0000 7f84e142b740 4 client.16014 mds_command: new command op to 24545 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-23T10:30:01.250 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:30:01.243+0000 7f84e142b740 1 -- 172.21.15.114:0/1371277440 --> [v2:172.21.15.119:6838/937297340,v1:172.21.15.119:6839/937297340] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x557469706500 con 0x5574697fe740 2024-04-23T10:30:01.250 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:30:01.243+0000 7f84d77fe640 1 --2- 172.21.15.114:0/1371277440 >> [v2:172.21.15.119:6838/937297340,v1:172.21.15.119:6839/937297340] conn(0x5574697fe740 0x557469800bc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T10:30:01.250 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:30:01.243+0000 7f84b77fe640 10 client.16014 ms_handle_connect on v2:172.21.15.119:6838/937297340 2024-04-23T10:30:01.250 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:30:01.243+0000 7f84d77fe640 1 --2- 172.21.15.114:0/1371277440 >> [v2:172.21.15.119:6838/937297340,v1:172.21.15.119:6839/937297340] conn(0x5574697fe740 0x557469800bc0 crc :-1 s=READY pgs=84 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=b2c05c0f1ab90de1 server_cookie=bf1bf5708cb096e3 in_seq=0 out_seq=0 2024-04-23T10:30:01.250 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:30:01.244+0000 7f84b77fe640 1 -- 172.21.15.114:0/1371277440 <== mds.0 v2:172.21.15.119:6838/937297340 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x557469706500 con 0x5574697fe740 2024-04-23T10:30:01.250 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:30:01.244+0000 7f84b77fe640 10 client.16014 handle_command_reply: tid=0 2024-04-23T10:30:01.250 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:30:01.244+0000 7f84e142b740 1 client.16014 shutdown 2024-04-23T10:30:01.250 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:30:01.244+0000 7f84e142b740 1 -- 172.21.15.114:0/1371277440 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x5574697e3490 msgr2=0x5574697edf00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T10:30:01.250 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:30:01.244+0000 7f84e142b740 1 --2- 172.21.15.114:0/1371277440 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x5574697e3490 0x5574697edf00 secure :-1 s=READY pgs=619 cs=0 l=1 rev1=1 crypto rx=0x7f84c8010810 tx=0x7f84c8010cd0 comp rx=0 tx=0).stop 2024-04-23T10:30:01.250 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:30:01.244+0000 7f84e142b740 1 -- 172.21.15.114:0/1371277440 shutdown_connections 2024-04-23T10:30:01.251 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:30:01.244+0000 7f84e142b740 1 --2- 172.21.15.114:0/1371277440 >> [v2:172.21.15.119:6838/937297340,v1:172.21.15.119:6839/937297340] conn(0x5574697fe740 0x557469800bc0 crc :-1 s=READY pgs=84 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:30:01.251 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:30:01.244+0000 7f84e142b740 1 --2- 172.21.15.114:0/1371277440 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x5574697ee480 0x5574697eb930 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:30:01.251 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:30:01.244+0000 7f84e142b740 1 --2- 172.21.15.114:0/1371277440 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x5574697e3db0 0x5574697eb1f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:30:01.251 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:30:01.244+0000 7f84e142b740 1 --2- 172.21.15.114:0/1371277440 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x5574697e3490 0x5574697edf00 unknown :-1 s=CLOSED pgs=619 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:30:01.251 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:30:01.244+0000 7f84e142b740 1 -- 172.21.15.114:0/1371277440 >> 172.21.15.114:0/1371277440 conn(0x5574697df7b0 msgr2=0x5574697de9d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T10:30:01.251 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:30:01.244+0000 7f84b77fe640 0 client.16014 ms_handle_reset on v2:172.21.15.119:6838/937297340 2024-04-23T10:30:01.251 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:30:01.245+0000 7f84e142b740 1 -- 172.21.15.114:0/1371277440 shutdown_connections 2024-04-23T10:30:01.251 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:30:01.245+0000 7f84e142b740 1 -- 172.21.15.114:0/1371277440 wait complete. 2024-04-23T10:30:01.251 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:30:01.245+0000 7f84e142b740 20 client.16014 trim_cache size 0 max 16384 2024-04-23T10:30:01.264 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:30:01.261+0000 7f84e142b740 1 Processor -- start 2024-04-23T10:30:01.264 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:30:01.261+0000 7f84e142b740 1 -- start start 2024-04-23T10:30:01.265 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:30:01.262+0000 7f84e142b740 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x5574697e3490 0x5574697e5bf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T10:30:01.265 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:30:01.262+0000 7f84e142b740 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x5574697e3db0 0x5574697eda30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T10:30:01.265 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:30:01.262+0000 7f84e142b740 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x5574697ee480 0x557469806490 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T10:30:01.265 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:30:01.262+0000 7f84e142b740 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x7f84d81b0550 con 0x5574697e3490 2024-04-23T10:30:01.265 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:30:01.262+0000 7f84e142b740 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 -- 0x7f84d81addf0 con 0x5574697e3db0 2024-04-23T10:30:01.265 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:30:01.262+0000 7f84e142b740 1 -- --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_getmap magic: 0 -- 0x7f84d81abc40 con 0x5574697ee480 2024-04-23T10:30:01.265 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:30:01.262+0000 7f84d7fff640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x5574697e3490 0x5574697e5bf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T10:30:01.265 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:30:01.262+0000 7f84d7fff640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x5574697e3490 0x5574697e5bf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.114:48610/0 (socket says 172.21.15.114:48610) 2024-04-23T10:30:01.265 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:30:01.262+0000 7f84d7fff640 1 -- 172.21.15.114:0/897512595 learned_addr learned my addr 172.21.15.114:0/897512595 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-04-23T10:30:01.265 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:30:01.262+0000 7f84d7fff640 1 -- 172.21.15.114:0/897512595 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x5574697ee480 msgr2=0x557469806490 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T10:30:01.265 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:30:01.262+0000 7f84d7fff640 1 --2- 172.21.15.114:0/897512595 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x5574697ee480 0x557469806490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:30:01.265 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:30:01.262+0000 7f84d7fff640 1 -- 172.21.15.114:0/897512595 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x5574697e3db0 msgr2=0x5574697eda30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T10:30:01.265 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:30:01.262+0000 7f84d7fff640 1 --2- 172.21.15.114:0/897512595 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x5574697e3db0 0x5574697eda30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:30:01.265 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:30:01.262+0000 7f84d7fff640 1 -- 172.21.15.114:0/897512595 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5574697e2aa0 con 0x5574697e3490 2024-04-23T10:30:01.266 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:30:01.262+0000 7f84d7fff640 1 --2- 172.21.15.114:0/897512595 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x5574697e3490 0x5574697e5bf0 secure :-1 s=READY pgs=620 cs=0 l=1 rev1=1 crypto rx=0x7f84c8000c00 tx=0x7f84c801a4b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a49c18eb75e26c9d server_cookie=0 in_seq=0 out_seq=0 2024-04-23T10:30:01.266 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:30:01.262+0000 7f84bdffb640 1 -- 172.21.15.114:0/897512595 <== mon.0 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f84c80052e0 con 0x5574697e3490 2024-04-23T10:30:01.266 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:30:01.262+0000 7f84bdffb640 1 -- 172.21.15.114:0/897512595 <== mon.0 v2:172.21.15.114:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f84c800cbb0 con 0x5574697e3490 2024-04-23T10:30:01.266 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:30:01.262+0000 7f84bdffb640 1 -- 172.21.15.114:0/897512595 <== mon.0 v2:172.21.15.114:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f84c8003b80 con 0x5574697e3490 2024-04-23T10:30:01.266 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:30:01.263+0000 7f84e142b740 1 -- 172.21.15.114:0/897512595 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x5574697e3490 msgr2=0x5574697e5bf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T10:30:01.266 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:30:01.263+0000 7f84e142b740 1 --2- 172.21.15.114:0/897512595 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x5574697e3490 0x5574697e5bf0 secure :-1 s=READY pgs=620 cs=0 l=1 rev1=1 crypto rx=0x7f84c8000c00 tx=0x7f84c801a4b0 comp rx=0 tx=0).stop 2024-04-23T10:30:01.266 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:30:01.263+0000 7f84e142b740 1 -- 172.21.15.114:0/897512595 shutdown_connections 2024-04-23T10:30:01.266 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:30:01.263+0000 7f84e142b740 1 --2- 172.21.15.114:0/897512595 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x5574697ee480 0x557469806490 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:30:01.266 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:30:01.263+0000 7f84e142b740 1 --2- 172.21.15.114:0/897512595 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x5574697e3db0 0x5574697eda30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:30:01.266 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:30:01.263+0000 7f84e142b740 1 --2- 172.21.15.114:0/897512595 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x5574697e3490 0x5574697e5bf0 unknown :-1 s=CLOSED pgs=620 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:30:01.266 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:30:01.263+0000 7f84e142b740 1 -- 172.21.15.114:0/897512595 >> 172.21.15.114:0/897512595 conn(0x5574697df7b0 msgr2=0x5574697f91c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T10:30:01.266 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:30:01.263+0000 7f84e142b740 1 -- 172.21.15.114:0/897512595 shutdown_connections 2024-04-23T10:30:01.266 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:30:01.263+0000 7f84e142b740 1 -- 172.21.15.114:0/897512595 wait complete. 2024-04-23T10:30:01.266 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:30:01.263+0000 7f84e142b740 1 -- start start 2024-04-23T10:30:01.266 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:30:01.263+0000 7f84e142b740 1 Processor -- start 2024-04-23T10:30:01.267 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:30:01.264+0000 7f84e142b740 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x5574697e3490 0x5574697f66e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T10:30:01.267 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:30:01.264+0000 7f84e142b740 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x5574697e3db0 0x5574697f6c20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T10:30:01.267 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:30:01.264+0000 7f84e142b740 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x5574697ee480 0x5574697e7460 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T10:30:01.267 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:30:01.264+0000 7f84e142b740 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x7f84d81ac2b0 con 0x5574697e3db0 2024-04-23T10:30:01.267 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:30:01.264+0000 7f84e142b740 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 -- 0x7f84d81ad720 con 0x5574697ee480 2024-04-23T10:30:01.267 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:30:01.264+0000 7f84e142b740 1 -- --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_getmap magic: 0 -- 0x7f84d81ac980 con 0x5574697e3490 2024-04-23T10:30:01.267 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:30:01.264+0000 7f84e142b740 10 client.0 _ll_register_callbacks cb 0x7f84c80103d0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-23T10:30:01.267 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:30:01.264+0000 7f84d77fe640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x5574697e3db0 0x5574697f6c20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T10:30:01.267 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:30:01.264+0000 7f84d77fe640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x5574697e3db0 0x5574697f6c20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.114:48622/0 (socket says 172.21.15.114:48622) 2024-04-23T10:30:01.267 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:30:01.264+0000 7f84d77fe640 1 -- 172.21.15.114:0/397274896 learned_addr learned my addr 172.21.15.114:0/397274896 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-04-23T10:30:01.267 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:30:01.264+0000 7f84dccb6640 1 --2- 172.21.15.114:0/397274896 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x5574697ee480 0x5574697e7460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T10:30:01.267 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:30:01.264+0000 7f84d7fff640 1 --2- 172.21.15.114:0/397274896 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x5574697e3490 0x5574697f66e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T10:30:01.267 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:30:01.264+0000 7f84d77fe640 1 -- 172.21.15.114:0/397274896 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x5574697e3490 msgr2=0x5574697f66e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T10:30:01.267 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:30:01.264+0000 7f84d77fe640 1 --2- 172.21.15.114:0/397274896 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x5574697e3490 0x5574697f66e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:30:01.267 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:30:01.264+0000 7f84d77fe640 1 -- 172.21.15.114:0/397274896 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x5574697ee480 msgr2=0x5574697e7460 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T10:30:01.268 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:30:01.264+0000 7f84d77fe640 1 --2- 172.21.15.114:0/397274896 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x5574697ee480 0x5574697e7460 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:30:01.268 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:30:01.264+0000 7f84d77fe640 1 -- 172.21.15.114:0/397274896 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5574697e7aa0 con 0x5574697e3db0 2024-04-23T10:30:01.268 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:30:01.264+0000 7f84d77fe640 1 --2- 172.21.15.114:0/397274896 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x5574697e3db0 0x5574697f6c20 secure :-1 s=READY pgs=621 cs=0 l=1 rev1=1 crypto rx=0x7f84cc0b9620 tx=0x7f84cc02a410 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5845c384b738f497 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T10:30:01.268 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:30:01.264+0000 7f84b77fe640 10 client.0 ms_handle_connect on v2:172.21.15.114:3300/0 2024-04-23T10:30:01.268 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:30:01.264+0000 7f84b77fe640 1 -- 172.21.15.114:0/397274896 <== mon.0 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f84cc049040 con 0x5574697e3db0 2024-04-23T10:30:01.268 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:30:01.264+0000 7f84b77fe640 1 -- 172.21.15.114:0/397274896 <== mon.0 v2:172.21.15.114:3300/0 2 ==== osd_map(92..92 src has 1..92) ==== 8585+0+0 (secure 0 0 0) 0x7f84cc0ee680 con 0x5574697e3db0 2024-04-23T10:30:01.268 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:30:01.264+0000 7f84e142b740 1 -- 172.21.15.114:0/397274896 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f84c800cbb0 con 0x5574697e3db0 2024-04-23T10:30:01.268 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:30:01.264+0000 7f84b77fe640 1 -- 172.21.15.114:0/397274896 <== mon.0 v2:172.21.15.114:3300/0 3 ==== mon_get_version_reply(handle=1 version=23) ==== 24+0+0 (secure 0 0 0) 0x7f84cc0bd350 con 0x5574697e3db0 2024-04-23T10:30:01.268 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:30:01.264+0000 7f84e142b740 10 client.16020 fetch_fsmap learned FSMap version 23 2024-04-23T10:30:01.268 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:30:01.264+0000 7f84e142b740 1 -- 172.21.15.114:0/397274896 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({config=0+,fsmap=23,monmap=4+}) -- 0x5574697e8e20 con 0x5574697e3db0 2024-04-23T10:30:01.268 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:30:01.265+0000 7f84b77fe640 1 -- 172.21.15.114:0/397274896 <== mon.0 v2:172.21.15.114:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f84c800cbb0 con 0x5574697e3db0 2024-04-23T10:30:01.268 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:30:01.265+0000 7f84b77fe640 1 -- 172.21.15.114:0/397274896 <== mon.0 v2:172.21.15.114:3300/0 5 ==== fsmap(e 23) ==== 8350+0+0 (secure 0 0 0) 0x7f84cc0b92c0 con 0x5574697e3db0 2024-04-23T10:30:01.268 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:30:01.265+0000 7f84e142b740 10 client.16020 fetch_fsmap finished waiting for FSMap version 23 2024-04-23T10:30:01.268 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:30:01.265+0000 7f84e142b740 10 client.16020 resolve_mds: validated gid 24545 aka daemon mds.b 2024-04-23T10:30:01.269 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:30:01.265+0000 7f84e142b740 20 client.16020 populate_metadata read hostname 'smithi114' 2024-04-23T10:30:01.269 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:30:01.265+0000 7f84e142b740 1 --2- 172.21.15.114:0/397274896 >> [v2:172.21.15.119:6838/937297340,v1:172.21.15.119:6839/937297340] conn(0x557469865ef0 0x557469868310 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T10:30:01.269 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:30:01.265+0000 7f84e142b740 4 client.16020 mds_command: new command op to 24545 tid=0 multi_id=0 [{"prefix": "quiesce db", "set_id": "4e8b9ad0", "release": true, "await": true}] 2024-04-23T10:30:01.269 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:30:01.265+0000 7f84e142b740 1 -- 172.21.15.114:0/397274896 --> [v2:172.21.15.119:6838/937297340,v1:172.21.15.119:6839/937297340] -- command(tid 0: {"prefix": "quiesce db", "set_id": "4e8b9ad0", "release": true, "await": true}) -- 0x5574697560f0 con 0x557469865ef0 2024-04-23T10:30:01.269 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:30:01.265+0000 7f84d7fff640 1 --2- 172.21.15.114:0/397274896 >> [v2:172.21.15.119:6838/937297340,v1:172.21.15.119:6839/937297340] conn(0x557469865ef0 0x557469868310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T10:30:01.269 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:30:01.265+0000 7f84b77fe640 10 client.16020 ms_handle_connect on v2:172.21.15.119:6838/937297340 2024-04-23T10:30:01.269 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:30:01.265+0000 7f84d7fff640 1 --2- 172.21.15.114:0/397274896 >> [v2:172.21.15.119:6838/937297340,v1:172.21.15.119:6839/937297340] conn(0x557469865ef0 0x557469868310 crc :-1 s=READY pgs=85 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=28a23b1dc325dc01 server_cookie=6ea7e04ce44ff702 in_seq=0 out_seq=0 2024-04-23T10:30:01.271 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:30:01.272+0000 7f84b77fe640 1 -- 172.21.15.114:0/397274896 <== mds.0 v2:172.21.15.119:6838/937297340 1 ==== command_reply(tid 0: 0 ) ==== 8+0+590 (crc 0 0 0) 0x5574697560f0 con 0x557469865ef0 2024-04-23T10:30:01.271 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:30:01.272+0000 7f84b77fe640 10 client.16020 handle_command_reply: tid=0 2024-04-23T10:30:01.271 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:30:01.272+0000 7f84e142b740 1 client.16020 shutdown 2024-04-23T10:30:01.271 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:30:01.272+0000 7f84e142b740 1 -- 172.21.15.114:0/397274896 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x5574697e3db0 msgr2=0x5574697f6c20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T10:30:01.271 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:30:01.272+0000 7f84e142b740 1 --2- 172.21.15.114:0/397274896 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x5574697e3db0 0x5574697f6c20 secure :-1 s=READY pgs=621 cs=0 l=1 rev1=1 crypto rx=0x7f84cc0b9620 tx=0x7f84cc02a410 comp rx=0 tx=0).stop 2024-04-23T10:30:01.271 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:30:01.272+0000 7f84e142b740 1 -- 172.21.15.114:0/397274896 shutdown_connections 2024-04-23T10:30:01.271 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:30:01.272+0000 7f84e142b740 1 --2- 172.21.15.114:0/397274896 >> [v2:172.21.15.119:6838/937297340,v1:172.21.15.119:6839/937297340] conn(0x557469865ef0 0x557469868310 crc :-1 s=READY pgs=85 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:30:01.272 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:30:01.272+0000 7f84e142b740 1 --2- 172.21.15.114:0/397274896 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x5574697ee480 0x5574697e7460 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:30:01.272 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:30:01.272+0000 7f84e142b740 1 --2- 172.21.15.114:0/397274896 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x5574697e3db0 0x5574697f6c20 unknown :-1 s=CLOSED pgs=621 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:30:01.272 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:30:01.272+0000 7f84b77fe640 0 client.16020 ms_handle_reset on v2:172.21.15.119:6838/937297340 2024-04-23T10:30:01.272 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:30:01.272+0000 7f84e142b740 1 --2- 172.21.15.114:0/397274896 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x5574697e3490 0x5574697f66e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:30:01.272 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:30:01.272+0000 7f84e142b740 1 -- 172.21.15.114:0/397274896 >> 172.21.15.114:0/397274896 conn(0x5574697df7b0 msgr2=0x5574697e5470 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T10:30:01.272 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:30:01.272+0000 7f84e142b740 1 -- 172.21.15.114:0/397274896 shutdown_connections 2024-04-23T10:30:01.272 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:30:01.272+0000 7f84e142b740 1 -- 172.21.15.114:0/397274896 wait complete. 2024-04-23T10:30:01.272 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:30:01.272+0000 7f84e142b740 20 client.16020 trim_cache size 0 max 16384 2024-04-23T10:30:01.272 INFO:teuthology.orchestra.run.smithi114.stdout:{ 2024-04-23T10:30:01.272 INFO:teuthology.orchestra.run.smithi114.stdout: "epoch": 23, 2024-04-23T10:30:01.272 INFO:teuthology.orchestra.run.smithi114.stdout: "leader": 24545, 2024-04-23T10:30:01.273 INFO:teuthology.orchestra.run.smithi114.stdout: "set_version": 16, 2024-04-23T10:30:01.273 INFO:teuthology.orchestra.run.smithi114.stdout: "sets": { 2024-04-23T10:30:01.273 INFO:teuthology.orchestra.run.smithi114.stdout: "4e8b9ad0": { 2024-04-23T10:30:01.273 INFO:teuthology.orchestra.run.smithi114.stdout: "version": 16, 2024-04-23T10:30:01.273 INFO:teuthology.orchestra.run.smithi114.stdout: "age_ref": 0.0, 2024-04-23T10:30:01.273 INFO:teuthology.orchestra.run.smithi114.stdout: "state": { 2024-04-23T10:30:01.273 INFO:teuthology.orchestra.run.smithi114.stdout: "name": "RELEASED", 2024-04-23T10:30:01.273 INFO:teuthology.orchestra.run.smithi114.stdout: "age": 0.0 2024-04-23T10:30:01.273 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-23T10:30:01.273 INFO:teuthology.orchestra.run.smithi114.stdout: "timeout": 60.0, 2024-04-23T10:30:01.273 INFO:teuthology.orchestra.run.smithi114.stdout: "expiration": 81.5, 2024-04-23T10:30:01.273 INFO:teuthology.orchestra.run.smithi114.stdout: "members": { 2024-04-23T10:30:01.273 INFO:teuthology.orchestra.run.smithi114.stdout: "file:/": { 2024-04-23T10:30:01.273 INFO:teuthology.orchestra.run.smithi114.stdout: "excluded": false, 2024-04-23T10:30:01.273 INFO:teuthology.orchestra.run.smithi114.stdout: "state": { 2024-04-23T10:30:01.273 INFO:teuthology.orchestra.run.smithi114.stdout: "name": "RELEASED", 2024-04-23T10:30:01.274 INFO:teuthology.orchestra.run.smithi114.stdout: "age": 0.0 2024-04-23T10:30:01.274 INFO:teuthology.orchestra.run.smithi114.stdout: } 2024-04-23T10:30:01.274 INFO:teuthology.orchestra.run.smithi114.stdout: } 2024-04-23T10:30:01.274 INFO:teuthology.orchestra.run.smithi114.stdout: } 2024-04-23T10:30:01.274 INFO:teuthology.orchestra.run.smithi114.stdout: } 2024-04-23T10:30:01.274 INFO:teuthology.orchestra.run.smithi114.stdout: } 2024-04-23T10:30:01.274 INFO:teuthology.orchestra.run.smithi114.stdout:} 2024-04-23T10:30:01.275 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:30:01.273+0000 7f84de719640 1 -- 172.21.15.114:0/992652486 >> [v2:172.21.15.114:6800/1395693466,v1:172.21.15.114:6801/1395693466] conn(0x7f84c00a3d20 msgr2=0x7f84c00a61e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T10:30:01.275 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:30:01.273+0000 7f84de719640 1 --2- 172.21.15.114:0/992652486 >> [v2:172.21.15.114:6800/1395693466,v1:172.21.15.114:6801/1395693466] conn(0x7f84c00a3d20 0x7f84c00a61e0 secure :-1 s=READY pgs=92 cs=0 l=1 rev1=1 crypto rx=0x7f84d0021000 tx=0x7f84d001e000 comp rx=0 tx=0).stop 2024-04-23T10:30:01.275 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:30:01.273+0000 7f84de719640 1 -- 172.21.15.114:0/992652486 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f84d8106170 msgr2=0x7f84d8198c20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T10:30:01.275 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:30:01.273+0000 7f84de719640 1 --2- 172.21.15.114:0/992652486 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f84d8106170 0x7f84d8198c20 secure :-1 s=READY pgs=617 cs=0 l=1 rev1=1 crypto rx=0x7f84cc032fd0 tx=0x7f84cc004270 comp rx=0 tx=0).stop 2024-04-23T10:30:01.275 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:30:01.273+0000 7f84de719640 1 -- 172.21.15.114:0/992652486 shutdown_connections 2024-04-23T10:30:01.275 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:30:01.273+0000 7f84de719640 1 --2- 172.21.15.114:0/992652486 >> [v2:172.21.15.114:6800/1395693466,v1:172.21.15.114:6801/1395693466] conn(0x7f84c00a3d20 0x7f84c00a61e0 unknown :-1 s=CLOSED pgs=92 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:30:01.275 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:30:01.273+0000 7f84de719640 1 --2- 172.21.15.114:0/992652486 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f84d819cfa0 0x7f84d819d450 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:30:01.275 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:30:01.273+0000 7f84de719640 1 --2- 172.21.15.114:0/992652486 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f84d8106170 0x7f84d8198c20 unknown :-1 s=CLOSED pgs=617 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:30:01.275 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:30:01.273+0000 7f84de719640 1 --2- 172.21.15.114:0/992652486 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f84d81010a0 0x7f84d81986e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:30:01.275 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:30:01.273+0000 7f84de719640 1 -- 172.21.15.114:0/992652486 >> 172.21.15.114:0/992652486 conn(0x7f84d8077eb0 msgr2=0x7f84d8100590 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T10:30:01.275 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:30:01.273+0000 7f84de719640 1 -- 172.21.15.114:0/992652486 shutdown_connections 2024-04-23T10:30:01.275 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:30:01.273+0000 7f84de719640 1 -- 172.21.15.114:0/992652486 wait complete. 2024-04-23T10:30:01.282 INFO:tasks.quiescer.fs.[cephfs]:Successfully released set '4e8b9ad0', seconds elapsed: 22.6 2024-04-23T10:30:01.327 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 23 10:30:00 smithi204 ceph-mon[32649]: overall HEALTH_OK 2024-04-23T10:30:01.914 INFO:tasks.workunit.client.0.smithi114.stdout: 1 178936 0.45 MB/sec execute 10 sec latency 21963.411 ms 2024-04-23T10:30:02.241 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 23 10:30:01 smithi119 ceph-mon[32608]: pgmap v105: 97 pgs: 97 active+clean; 52 MiB data, 851 MiB used, 1.0 TiB / 1.0 TiB avail; 67 KiB/s rd, 343 KiB/s wr, 42 op/s 2024-04-23T10:30:02.242 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 23 10:30:01 smithi119 ceph-mon[32608]: from='client.? 172.21.15.114:0/3734756329' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-23T10:30:02.253 INFO:journalctl@ceph.mon.a.smithi114.stdout:Apr 23 10:30:01 smithi114 ceph-mon[29255]: pgmap v105: 97 pgs: 97 active+clean; 52 MiB data, 851 MiB used, 1.0 TiB / 1.0 TiB avail; 67 KiB/s rd, 343 KiB/s wr, 42 op/s 2024-04-23T10:30:02.254 INFO:journalctl@ceph.mon.a.smithi114.stdout:Apr 23 10:30:01 smithi114 ceph-mon[29255]: from='client.? 172.21.15.114:0/3734756329' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-23T10:30:02.327 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 23 10:30:01 smithi204 ceph-mon[32649]: pgmap v105: 97 pgs: 97 active+clean; 52 MiB data, 851 MiB used, 1.0 TiB / 1.0 TiB avail; 67 KiB/s rd, 343 KiB/s wr, 42 op/s 2024-04-23T10:30:02.327 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 23 10:30:01 smithi204 ceph-mon[32649]: from='client.? 172.21.15.114:0/3734756329' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-23T10:30:02.914 INFO:tasks.workunit.client.0.smithi114.stdout: 1 180908 1.53 MB/sec execute 11 sec latency 273.685 ms 2024-04-23T10:30:03.914 INFO:tasks.workunit.client.0.smithi114.stdout: 1 182092 1.71 MB/sec execute 12 sec latency 13.667 ms 2024-04-23T10:30:04.241 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 23 10:30:03 smithi119 ceph-mon[32608]: pgmap v106: 97 pgs: 97 active+clean; 52 MiB data, 852 MiB used, 1.0 TiB / 1.0 TiB avail; 86 KiB/s rd, 388 KiB/s wr, 66 op/s 2024-04-23T10:30:04.254 INFO:journalctl@ceph.mon.a.smithi114.stdout:Apr 23 10:30:03 smithi114 ceph-mon[29255]: pgmap v106: 97 pgs: 97 active+clean; 52 MiB data, 852 MiB used, 1.0 TiB / 1.0 TiB avail; 86 KiB/s rd, 388 KiB/s wr, 66 op/s 2024-04-23T10:30:04.327 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 23 10:30:03 smithi204 ceph-mon[32649]: pgmap v106: 97 pgs: 97 active+clean; 52 MiB data, 852 MiB used, 1.0 TiB / 1.0 TiB avail; 86 KiB/s rd, 388 KiB/s wr, 66 op/s 2024-04-23T10:30:04.914 INFO:tasks.workunit.client.0.smithi114.stdout: 1 183569 2.12 MB/sec execute 13 sec latency 364.400 ms 2024-04-23T10:30:05.914 INFO:tasks.workunit.client.0.smithi114.stdout: 1 184869 2.69 MB/sec execute 14 sec latency 625.985 ms 2024-04-23T10:30:06.241 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 23 10:30:05 smithi119 ceph-mon[32608]: pgmap v107: 97 pgs: 97 active+clean; 52 MiB data, 852 MiB used, 1.0 TiB / 1.0 TiB avail; 39 KiB/s rd, 64 KiB/s wr, 33 op/s 2024-04-23T10:30:06.253 INFO:journalctl@ceph.mon.a.smithi114.stdout:Apr 23 10:30:05 smithi114 ceph-mon[29255]: pgmap v107: 97 pgs: 97 active+clean; 52 MiB data, 852 MiB used, 1.0 TiB / 1.0 TiB avail; 39 KiB/s rd, 64 KiB/s wr, 33 op/s 2024-04-23T10:30:06.327 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 23 10:30:05 smithi204 ceph-mon[32649]: pgmap v107: 97 pgs: 97 active+clean; 52 MiB data, 852 MiB used, 1.0 TiB / 1.0 TiB avail; 39 KiB/s rd, 64 KiB/s wr, 33 op/s 2024-04-23T10:30:06.914 INFO:tasks.workunit.client.0.smithi114.stdout: 1 186853 2.98 MB/sec execute 15 sec latency 15.908 ms 2024-04-23T10:30:07.914 INFO:tasks.workunit.client.0.smithi114.stdout: 1 188455 3.55 MB/sec execute 16 sec latency 54.562 ms 2024-04-23T10:30:08.241 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 23 10:30:07 smithi119 ceph-mon[32608]: pgmap v108: 97 pgs: 97 active+clean; 58 MiB data, 924 MiB used, 1.0 TiB / 1.0 TiB avail; 78 MiB/s rd, 988 KiB/s wr, 197 op/s 2024-04-23T10:30:08.253 INFO:journalctl@ceph.mon.a.smithi114.stdout:Apr 23 10:30:07 smithi114 ceph-mon[29255]: pgmap v108: 97 pgs: 97 active+clean; 58 MiB data, 924 MiB used, 1.0 TiB / 1.0 TiB avail; 78 MiB/s rd, 988 KiB/s wr, 197 op/s 2024-04-23T10:30:08.327 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 23 10:30:07 smithi204 ceph-mon[32649]: pgmap v108: 97 pgs: 97 active+clean; 58 MiB data, 924 MiB used, 1.0 TiB / 1.0 TiB avail; 78 MiB/s rd, 988 KiB/s wr, 197 op/s 2024-04-23T10:30:08.914 INFO:tasks.workunit.client.0.smithi114.stdout: 1 190629 3.82 MB/sec execute 17 sec latency 16.114 ms 2024-04-23T10:30:09.914 INFO:tasks.workunit.client.0.smithi114.stdout: 1 192223 4.23 MB/sec execute 18 sec latency 23.737 ms 2024-04-23T10:30:10.254 INFO:journalctl@ceph.mon.a.smithi114.stdout:Apr 23 10:30:09 smithi114 ceph-mon[29255]: pgmap v109: 97 pgs: 97 active+clean; 58 MiB data, 924 MiB used, 1.0 TiB / 1.0 TiB avail; 78 MiB/s rd, 969 KiB/s wr, 187 op/s 2024-04-23T10:30:10.327 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 23 10:30:09 smithi204 ceph-mon[32649]: pgmap v109: 97 pgs: 97 active+clean; 58 MiB data, 924 MiB used, 1.0 TiB / 1.0 TiB avail; 78 MiB/s rd, 969 KiB/s wr, 187 op/s 2024-04-23T10:30:10.491 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 23 10:30:09 smithi119 ceph-mon[32608]: pgmap v109: 97 pgs: 97 active+clean; 58 MiB data, 924 MiB used, 1.0 TiB / 1.0 TiB avail; 78 MiB/s rd, 969 KiB/s wr, 187 op/s 2024-04-23T10:30:10.915 INFO:tasks.workunit.client.0.smithi114.stdout: 1 194754 4.69 MB/sec execute 19 sec latency 9.613 ms 2024-04-23T10:30:11.915 INFO:tasks.workunit.client.0.smithi114.stdout: 1 196065 4.79 MB/sec execute 20 sec latency 14.801 ms 2024-04-23T10:30:12.254 INFO:journalctl@ceph.mon.a.smithi114.stdout:Apr 23 10:30:11 smithi114 ceph-mon[29255]: pgmap v110: 97 pgs: 97 active+clean; 58 MiB data, 924 MiB used, 1.0 TiB / 1.0 TiB avail; 78 MiB/s rd, 969 KiB/s wr, 187 op/s 2024-04-23T10:30:12.327 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 23 10:30:11 smithi204 ceph-mon[32649]: pgmap v110: 97 pgs: 97 active+clean; 58 MiB data, 924 MiB used, 1.0 TiB / 1.0 TiB avail; 78 MiB/s rd, 969 KiB/s wr, 187 op/s 2024-04-23T10:30:12.491 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 23 10:30:11 smithi119 ceph-mon[32608]: pgmap v110: 97 pgs: 97 active+clean; 58 MiB data, 924 MiB used, 1.0 TiB / 1.0 TiB avail; 78 MiB/s rd, 969 KiB/s wr, 187 op/s 2024-04-23T10:30:12.915 INFO:tasks.workunit.client.0.smithi114.stdout: 1 198296 5.16 MB/sec execute 21 sec latency 155.141 ms 2024-04-23T10:30:13.200 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:30:13.198+0000 7fced9ccf640 1 -- 172.21.15.114:0/2745616402 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fced41010f0 msgr2=0x7fced41014d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T10:30:13.200 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:30:13.198+0000 7fced9ccf640 1 --2- 172.21.15.114:0/2745616402 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fced41010f0 0x7fced41014d0 secure :-1 s=READY pgs=622 cs=0 l=1 rev1=1 crypto rx=0x7fcec4009900 tx=0x7fcec402a410 comp rx=0 tx=0).stop 2024-04-23T10:30:13.200 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:30:13.198+0000 7fced9ccf640 1 -- 172.21.15.114:0/2745616402 shutdown_connections 2024-04-23T10:30:13.200 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:30:13.198+0000 7fced9ccf640 1 --2- 172.21.15.114:0/2745616402 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fced4106170 0x7fced410dd40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:30:13.200 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:30:13.198+0000 7fced9ccf640 1 --2- 172.21.15.114:0/2745616402 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fced4101aa0 0x7fced4105a90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:30:13.200 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:30:13.198+0000 7fced9ccf640 1 --2- 172.21.15.114:0/2745616402 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fced41010f0 0x7fced41014d0 unknown :-1 s=CLOSED pgs=622 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:30:13.200 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:30:13.198+0000 7fced9ccf640 1 -- 172.21.15.114:0/2745616402 >> 172.21.15.114:0/2745616402 conn(0x7fced4077fd0 msgr2=0x7fced40783e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T10:30:13.200 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:30:13.198+0000 7fced9ccf640 1 -- 172.21.15.114:0/2745616402 shutdown_connections 2024-04-23T10:30:13.200 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:30:13.198+0000 7fced9ccf640 1 -- 172.21.15.114:0/2745616402 wait complete. 2024-04-23T10:30:13.200 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:30:13.199+0000 7fced9ccf640 1 Processor -- start 2024-04-23T10:30:13.200 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:30:13.199+0000 7fced9ccf640 1 -- start start 2024-04-23T10:30:13.200 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:30:13.199+0000 7fced9ccf640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fced41010f0 0x7fced4198780 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T10:30:13.201 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:30:13.199+0000 7fced9ccf640 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fced4106170 0x7fced4198cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T10:30:13.201 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:30:13.199+0000 7fced9ccf640 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fced419cff0 0x7fced419d4a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T10:30:13.201 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:30:13.199+0000 7fced9ccf640 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x7fced41110b0 con 0x7fced41010f0 2024-04-23T10:30:13.201 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:30:13.199+0000 7fced9ccf640 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 -- 0x7fced4110f30 con 0x7fced4106170 2024-04-23T10:30:13.201 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:30:13.199+0000 7fced9ccf640 1 -- --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_getmap magic: 0 -- 0x7fced4111230 con 0x7fced419cff0 2024-04-23T10:30:13.201 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:30:13.199+0000 7fced37fe640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fced41010f0 0x7fced4198780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T10:30:13.201 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:30:13.199+0000 7fced37fe640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fced41010f0 0x7fced4198780 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.114:39748/0 (socket says 172.21.15.114:39748) 2024-04-23T10:30:13.201 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:30:13.199+0000 7fced37fe640 1 -- 172.21.15.114:0/1028007654 learned_addr learned my addr 172.21.15.114:0/1028007654 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-04-23T10:30:13.201 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:30:13.199+0000 7fced3fff640 1 --2- 172.21.15.114:0/1028007654 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fced419cff0 0x7fced419d4a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T10:30:13.201 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:30:13.199+0000 7fced2ffd640 1 --2- 172.21.15.114:0/1028007654 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fced4106170 0x7fced4198cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T10:30:13.201 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:30:13.199+0000 7fced37fe640 1 -- 172.21.15.114:0/1028007654 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fced419cff0 msgr2=0x7fced419d4a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T10:30:13.201 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:30:13.199+0000 7fced37fe640 1 --2- 172.21.15.114:0/1028007654 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fced419cff0 0x7fced419d4a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:30:13.202 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:30:13.199+0000 7fced37fe640 1 -- 172.21.15.114:0/1028007654 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fced4106170 msgr2=0x7fced4198cc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T10:30:13.202 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:30:13.199+0000 7fced37fe640 1 --2- 172.21.15.114:0/1028007654 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fced4106170 0x7fced4198cc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:30:13.202 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:30:13.199+0000 7fced37fe640 1 -- 172.21.15.114:0/1028007654 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fced419dae0 con 0x7fced41010f0 2024-04-23T10:30:13.202 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:30:13.200+0000 7fced37fe640 1 --2- 172.21.15.114:0/1028007654 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fced41010f0 0x7fced4198780 secure :-1 s=READY pgs=623 cs=0 l=1 rev1=1 crypto rx=0x7fcec4005940 tx=0x7fcec40042f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T10:30:13.202 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:30:13.200+0000 7fced0ff9640 1 -- 172.21.15.114:0/1028007654 <== mon.0 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcec403f030 con 0x7fced41010f0 2024-04-23T10:30:13.202 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:30:13.200+0000 7fced0ff9640 1 -- 172.21.15.114:0/1028007654 <== mon.0 v2:172.21.15.114:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fcec4003cb0 con 0x7fced41010f0 2024-04-23T10:30:13.202 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:30:13.200+0000 7fced9ccf640 1 -- 172.21.15.114:0/1028007654 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fced419dd70 con 0x7fced41010f0 2024-04-23T10:30:13.202 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:30:13.200+0000 7fced0ff9640 1 -- 172.21.15.114:0/1028007654 <== mon.0 v2:172.21.15.114:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcec4035070 con 0x7fced41010f0 2024-04-23T10:30:13.202 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:30:13.200+0000 7fced9ccf640 1 -- 172.21.15.114:0/1028007654 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fced41a5530 con 0x7fced41010f0 2024-04-23T10:30:13.209 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:30:13.202+0000 7fced0ff9640 1 -- 172.21.15.114:0/1028007654 <== mon.0 v2:172.21.15.114:3300/0 4 ==== mgrmap(e 26) ==== 138778+0+0 (secure 0 0 0) 0x7fcec4042030 con 0x7fced41010f0 2024-04-23T10:30:13.209 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:30:13.202+0000 7fcedc9e9740 1 Processor -- start 2024-04-23T10:30:13.210 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:30:13.202+0000 7fced0ff9640 1 --2- 172.21.15.114:0/1028007654 >> [v2:172.21.15.114:6800/1395693466,v1:172.21.15.114:6801/1395693466] conn(0x7fcebc0a3dc0 0x7fcebc0a6280 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T10:30:13.210 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:30:13.202+0000 7fcedc9e9740 1 -- start start 2024-04-23T10:30:13.210 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:30:13.203+0000 7fced3fff640 1 --2- 172.21.15.114:0/1028007654 >> [v2:172.21.15.114:6800/1395693466,v1:172.21.15.114:6801/1395693466] conn(0x7fcebc0a3dc0 0x7fcebc0a6280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T10:30:13.210 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:30:13.203+0000 7fced0ff9640 1 -- 172.21.15.114:0/1028007654 <== mon.0 v2:172.21.15.114:3300/0 5 ==== osd_map(92..92 src has 1..92) ==== 8585+0+0 (secure 0 0 0) 0x7fcec40f1010 con 0x7fced41010f0 2024-04-23T10:30:13.210 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:30:13.203+0000 7fcedc9e9740 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x564be4111540 0x564be4111920 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T10:30:13.210 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:30:13.203+0000 7fcedc9e9740 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x564be4111e60 0x564be41206d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T10:30:13.210 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:30:13.203+0000 7fcedc9e9740 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x564be4120c10 0x564be4123000 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T10:30:13.210 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:30:13.203+0000 7fcedc9e9740 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x564be4076970 con 0x564be4120c10 2024-04-23T10:30:13.210 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:30:13.203+0000 7fcedc9e9740 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 -- 0x564be40134a0 con 0x564be4111e60 2024-04-23T10:30:13.210 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:30:13.203+0000 7fcedc9e9740 1 -- --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_getmap magic: 0 -- 0x564be4072930 con 0x564be4111540 2024-04-23T10:30:13.210 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:30:13.203+0000 7fced37fe640 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x564be4111540 0x564be4111920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T10:30:13.210 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:30:13.203+0000 7fced37fe640 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x564be4111540 0x564be4111920 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.204:3300/0 says I am v2:172.21.15.114:44964/0 (socket says 172.21.15.114:44964) 2024-04-23T10:30:13.211 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:30:13.203+0000 7fced37fe640 1 -- 172.21.15.114:0/3877783890 learned_addr learned my addr 172.21.15.114:0/3877783890 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-04-23T10:30:13.211 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:30:13.203+0000 7fced2ffd640 1 --2- 172.21.15.114:0/3877783890 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x564be4111e60 0x564be41206d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T10:30:13.211 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:30:13.203+0000 7fced3fff640 1 --2- 172.21.15.114:0/3877783890 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x564be4120c10 0x564be4123000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T10:30:13.211 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:30:13.203+0000 7fced3fff640 1 --2- 172.21.15.114:0/1028007654 >> [v2:172.21.15.114:6800/1395693466,v1:172.21.15.114:6801/1395693466] conn(0x7fcebc0a3dc0 0x7fcebc0a6280 secure :-1 s=READY pgs=93 cs=0 l=1 rev1=1 crypto rx=0x7fcecc005fd0 tx=0x7fcecc010740 comp rx=0 tx=0).ready entity=mgr.15291 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T10:30:13.211 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:30:13.203+0000 7fced37fe640 1 -- 172.21.15.114:0/3877783890 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x564be4111e60 msgr2=0x564be41206d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T10:30:13.211 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:30:13.203+0000 7fced37fe640 1 --2- 172.21.15.114:0/3877783890 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x564be4111e60 0x564be41206d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:30:13.211 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:30:13.203+0000 7fced37fe640 1 -- 172.21.15.114:0/3877783890 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x564be4120c10 msgr2=0x564be4123000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T10:30:13.211 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:30:13.203+0000 7fced37fe640 1 --2- 172.21.15.114:0/3877783890 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x564be4120c10 0x564be4123000 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:30:13.211 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:30:13.203+0000 7fced37fe640 1 -- 172.21.15.114:0/3877783890 --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x564be3d89210 con 0x564be4111540 2024-04-23T10:30:13.211 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:30:13.203+0000 7fced37fe640 1 --2- 172.21.15.114:0/3877783890 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x564be4111540 0x564be4111920 secure :-1 s=READY pgs=67 cs=0 l=1 rev1=1 crypto rx=0x7fcec40bcfb0 tx=0x7fcec403b6b0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=2d993a42971ddfb server_cookie=0 in_seq=0 out_seq=0 2024-04-23T10:30:13.211 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:30:13.203+0000 7fceb9ffb640 1 -- 172.21.15.114:0/3877783890 <== mon.1 v2:172.21.15.204:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcec4049040 con 0x564be4111540 2024-04-23T10:30:13.211 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:30:13.204+0000 7fceb9ffb640 1 -- 172.21.15.114:0/3877783890 <== mon.1 v2:172.21.15.204:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fcec40bd020 con 0x564be4111540 2024-04-23T10:30:13.212 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:30:13.204+0000 7fceb9ffb640 1 -- 172.21.15.114:0/3877783890 <== mon.1 v2:172.21.15.204:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcec403bbf0 con 0x564be4111540 2024-04-23T10:30:13.212 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:30:13.204+0000 7fcedc9e9740 1 -- 172.21.15.114:0/3877783890 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x564be4111540 msgr2=0x564be4111920 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T10:30:13.212 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:30:13.204+0000 7fcedc9e9740 1 --2- 172.21.15.114:0/3877783890 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x564be4111540 0x564be4111920 secure :-1 s=READY pgs=67 cs=0 l=1 rev1=1 crypto rx=0x7fcec40bcfb0 tx=0x7fcec403b6b0 comp rx=0 tx=0).stop 2024-04-23T10:30:13.212 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:30:13.204+0000 7fcedc9e9740 1 -- 172.21.15.114:0/3877783890 shutdown_connections 2024-04-23T10:30:13.212 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:30:13.204+0000 7fcedc9e9740 1 --2- 172.21.15.114:0/3877783890 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x564be4120c10 0x564be4123000 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:30:13.212 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:30:13.204+0000 7fcedc9e9740 1 --2- 172.21.15.114:0/3877783890 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x564be4111e60 0x564be41206d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:30:13.212 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:30:13.204+0000 7fcedc9e9740 1 --2- 172.21.15.114:0/3877783890 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x564be4111540 0x564be4111920 unknown :-1 s=CLOSED pgs=67 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:30:13.212 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:30:13.204+0000 7fcedc9e9740 1 -- 172.21.15.114:0/3877783890 >> 172.21.15.114:0/3877783890 conn(0x564be410db30 msgr2=0x564be410df10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T10:30:13.212 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:30:13.204+0000 7fcedc9e9740 1 -- 172.21.15.114:0/3877783890 shutdown_connections 2024-04-23T10:30:13.212 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:30:13.204+0000 7fcedc9e9740 1 -- 172.21.15.114:0/3877783890 wait complete. 2024-04-23T10:30:13.212 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:30:13.205+0000 7fcedc9e9740 1 -- start start 2024-04-23T10:30:13.212 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:30:13.205+0000 7fcedc9e9740 1 Processor -- start 2024-04-23T10:30:13.212 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:30:13.205+0000 7fcedc9e9740 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x564be4111540 0x564be41205c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T10:30:13.213 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:30:13.205+0000 7fcedc9e9740 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x564be4111e60 0x564be411d8b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T10:30:13.213 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:30:13.205+0000 7fcedc9e9740 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x564be4120c10 0x564be411de50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T10:30:13.213 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:30:13.205+0000 7fcedc9e9740 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x564be4123600 con 0x564be4111540 2024-04-23T10:30:13.213 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:30:13.205+0000 7fcedc9e9740 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 -- 0x564be40110b0 con 0x564be4120c10 2024-04-23T10:30:13.213 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:30:13.205+0000 7fcedc9e9740 1 -- --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_getmap magic: 0 -- 0x564be4123780 con 0x564be4111e60 2024-04-23T10:30:13.213 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:30:13.205+0000 7fcedc9e9740 10 client.0 _ll_register_callbacks cb 0x564be3ffed30 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-23T10:30:13.213 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:30:13.205+0000 7fced37fe640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x564be4111540 0x564be41205c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T10:30:13.213 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:30:13.205+0000 7fced37fe640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x564be4111540 0x564be41205c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.114:39770/0 (socket says 172.21.15.114:39770) 2024-04-23T10:30:13.213 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:30:13.205+0000 7fced37fe640 1 -- 172.21.15.114:0/3404834013 learned_addr learned my addr 172.21.15.114:0/3404834013 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-04-23T10:30:13.213 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:30:13.205+0000 7fced2ffd640 1 --2- 172.21.15.114:0/3404834013 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x564be4111e60 0x564be411d8b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T10:30:13.213 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:30:13.205+0000 7fced3fff640 1 --2- 172.21.15.114:0/3404834013 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x564be4120c10 0x564be411de50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T10:30:13.213 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:30:13.205+0000 7fced37fe640 1 -- 172.21.15.114:0/3404834013 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x564be4111e60 msgr2=0x564be411d8b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T10:30:13.214 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:30:13.205+0000 7fced37fe640 1 --2- 172.21.15.114:0/3404834013 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x564be4111e60 0x564be411d8b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:30:13.214 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:30:13.205+0000 7fced37fe640 1 -- 172.21.15.114:0/3404834013 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x564be4120c10 msgr2=0x564be411de50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T10:30:13.214 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:30:13.205+0000 7fced37fe640 1 --2- 172.21.15.114:0/3404834013 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x564be4120c10 0x564be411de50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:30:13.214 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:30:13.205+0000 7fced37fe640 1 -- 172.21.15.114:0/3404834013 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({osdmap=0}) -- 0x564be3e55620 con 0x564be4111540 2024-04-23T10:30:13.214 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:30:13.205+0000 7fced37fe640 1 --2- 172.21.15.114:0/3404834013 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x564be4111540 0x564be41205c0 secure :-1 s=READY pgs=624 cs=0 l=1 rev1=1 crypto rx=0x7fcec40ee7d0 tx=0x7fcec4005bd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f6302cda50e49553 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T10:30:13.214 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:30:13.205+0000 7fceb2ffd640 10 client.0 ms_handle_connect on v2:172.21.15.114:3300/0 2024-04-23T10:30:13.214 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:30:13.206+0000 7fceb2ffd640 1 -- 172.21.15.114:0/3404834013 <== mon.0 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcec40a0060 con 0x564be4111540 2024-04-23T10:30:13.214 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:30:13.206+0000 7fceb2ffd640 1 -- 172.21.15.114:0/3404834013 <== mon.0 v2:172.21.15.114:3300/0 2 ==== osd_map(92..92 src has 1..92) ==== 8585+0+0 (secure 0 0 0) 0x7fcec403bbf0 con 0x564be4111540 2024-04-23T10:30:13.214 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:30:13.206+0000 7fcedc9e9740 1 -- 172.21.15.114:0/3404834013 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fcec40bd020 con 0x564be4111540 2024-04-23T10:30:13.214 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:30:13.206+0000 7fceb2ffd640 1 -- 172.21.15.114:0/3404834013 <== mon.0 v2:172.21.15.114:3300/0 3 ==== mon_get_version_reply(handle=1 version=23) ==== 24+0+0 (secure 0 0 0) 0x7fcec40db070 con 0x564be4111540 2024-04-23T10:30:13.214 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:30:13.206+0000 7fcedc9e9740 10 client.16032 fetch_fsmap learned FSMap version 23 2024-04-23T10:30:13.214 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:30:13.206+0000 7fcedc9e9740 1 -- 172.21.15.114:0/3404834013 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({config=0+,fsmap=23,monmap=4+}) -- 0x564be4123600 con 0x564be4111540 2024-04-23T10:30:13.215 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:30:13.206+0000 7fceb2ffd640 1 -- 172.21.15.114:0/3404834013 <== mon.0 v2:172.21.15.114:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fcec40bd020 con 0x564be4111540 2024-04-23T10:30:13.215 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:30:13.206+0000 7fceb2ffd640 1 -- 172.21.15.114:0/3404834013 <== mon.0 v2:172.21.15.114:3300/0 5 ==== fsmap(e 23) ==== 8350+0+0 (secure 0 0 0) 0x7fcec40bd280 con 0x564be4111540 2024-04-23T10:30:13.215 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:30:13.206+0000 7fcedc9e9740 10 client.16032 fetch_fsmap finished waiting for FSMap version 23 2024-04-23T10:30:13.215 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:30:13.206+0000 7fcedc9e9740 10 client.16032 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-04-23T10:30:13.215 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:30:13.206+0000 7fcedc9e9740 20 client.16032 populate_metadata read hostname 'smithi114' 2024-04-23T10:30:13.215 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:30:13.206+0000 7fcedc9e9740 1 --2- 172.21.15.114:0/3404834013 >> [v2:172.21.15.119:6838/937297340,v1:172.21.15.119:6839/937297340] conn(0x564be4130660 0x564be4132a50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T10:30:13.215 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:30:13.206+0000 7fcedc9e9740 4 client.16032 mds_command: new command op to 24545 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-23T10:30:13.215 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:30:13.206+0000 7fcedc9e9740 1 -- 172.21.15.114:0/3404834013 --> [v2:172.21.15.119:6838/937297340,v1:172.21.15.119:6839/937297340] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x564be4084390 con 0x564be4130660 2024-04-23T10:30:13.215 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:30:13.206+0000 7fced2ffd640 1 --2- 172.21.15.114:0/3404834013 >> [v2:172.21.15.119:6838/937297340,v1:172.21.15.119:6839/937297340] conn(0x564be4130660 0x564be4132a50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T10:30:13.215 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:30:13.207+0000 7fceb2ffd640 10 client.16032 ms_handle_connect on v2:172.21.15.119:6838/937297340 2024-04-23T10:30:13.215 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:30:13.207+0000 7fced2ffd640 1 --2- 172.21.15.114:0/3404834013 >> [v2:172.21.15.119:6838/937297340,v1:172.21.15.119:6839/937297340] conn(0x564be4130660 0x564be4132a50 crc :-1 s=READY pgs=86 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=2e86573ab178d96e server_cookie=7ac466ee7d73929d in_seq=0 out_seq=0 2024-04-23T10:30:13.215 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:30:13.208+0000 7fceb2ffd640 1 -- 172.21.15.114:0/3404834013 <== mds.0 v2:172.21.15.119:6838/937297340 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x564be4084390 con 0x564be4130660 2024-04-23T10:30:13.216 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:30:13.208+0000 7fceb2ffd640 10 client.16032 handle_command_reply: tid=0 2024-04-23T10:30:13.216 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:30:13.208+0000 7fcedc9e9740 1 client.16032 shutdown 2024-04-23T10:30:13.216 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:30:13.208+0000 7fcedc9e9740 1 -- 172.21.15.114:0/3404834013 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x564be4111540 msgr2=0x564be41205c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T10:30:13.216 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:30:13.208+0000 7fcedc9e9740 1 --2- 172.21.15.114:0/3404834013 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x564be4111540 0x564be41205c0 secure :-1 s=READY pgs=624 cs=0 l=1 rev1=1 crypto rx=0x7fcec40ee7d0 tx=0x7fcec4005bd0 comp rx=0 tx=0).stop 2024-04-23T10:30:13.216 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:30:13.208+0000 7fcedc9e9740 1 -- 172.21.15.114:0/3404834013 shutdown_connections 2024-04-23T10:30:13.216 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:30:13.208+0000 7fcedc9e9740 1 --2- 172.21.15.114:0/3404834013 >> [v2:172.21.15.119:6838/937297340,v1:172.21.15.119:6839/937297340] conn(0x564be4130660 0x564be4132a50 crc :-1 s=READY pgs=86 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:30:13.216 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:30:13.208+0000 7fcedc9e9740 1 --2- 172.21.15.114:0/3404834013 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x564be4120c10 0x564be411de50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:30:13.216 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:30:13.208+0000 7fcedc9e9740 1 --2- 172.21.15.114:0/3404834013 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x564be4111e60 0x564be411d8b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:30:13.216 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:30:13.208+0000 7fcedc9e9740 1 --2- 172.21.15.114:0/3404834013 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x564be4111540 0x564be41205c0 unknown :-1 s=CLOSED pgs=624 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:30:13.216 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:30:13.208+0000 7fcedc9e9740 1 -- 172.21.15.114:0/3404834013 >> 172.21.15.114:0/3404834013 conn(0x564be410db30 msgr2=0x564be41139a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T10:30:13.216 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:30:13.208+0000 7fceb2ffd640 0 client.16032 ms_handle_reset on v2:172.21.15.119:6838/937297340 2024-04-23T10:30:13.216 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:30:13.208+0000 7fcedc9e9740 1 -- 172.21.15.114:0/3404834013 shutdown_connections 2024-04-23T10:30:13.216 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:30:13.208+0000 7fcedc9e9740 1 -- 172.21.15.114:0/3404834013 wait complete. 2024-04-23T10:30:13.217 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:30:13.208+0000 7fcedc9e9740 20 client.16032 trim_cache size 0 max 16384 2024-04-23T10:30:13.225 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:30:13.224+0000 7fcedc9e9740 1 Processor -- start 2024-04-23T10:30:13.225 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:30:13.224+0000 7fcedc9e9740 1 -- start start 2024-04-23T10:30:13.225 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:30:13.224+0000 7fcedc9e9740 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x564be4111540 0x564be412c410 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T10:30:13.225 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:30:13.224+0000 7fcedc9e9740 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x564be4111e60 0x564be41200f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T10:30:13.225 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:30:13.224+0000 7fcedc9e9740 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x564be4120c10 0x564be41381e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T10:30:13.225 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:30:13.224+0000 7fcedc9e9740 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x7fced41ae300 con 0x564be4111e60 2024-04-23T10:30:13.226 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:30:13.224+0000 7fcedc9e9740 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 -- 0x7fced41abba0 con 0x564be4111540 2024-04-23T10:30:13.226 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:30:13.224+0000 7fcedc9e9740 1 -- --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_getmap magic: 0 -- 0x7fced41a99f0 con 0x564be4120c10 2024-04-23T10:30:13.226 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:30:13.224+0000 7fced2ffd640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x564be4111e60 0x564be41200f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T10:30:13.226 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:30:13.225+0000 7fced2ffd640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x564be4111e60 0x564be41200f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.114:39774/0 (socket says 172.21.15.114:39774) 2024-04-23T10:30:13.226 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:30:13.225+0000 7fced2ffd640 1 -- 172.21.15.114:0/2939111357 learned_addr learned my addr 172.21.15.114:0/2939111357 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-04-23T10:30:13.226 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:30:13.225+0000 7fced37fe640 1 --2- 172.21.15.114:0/2939111357 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x564be4111540 0x564be412c410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T10:30:13.226 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:30:13.225+0000 7fced3fff640 1 --2- 172.21.15.114:0/2939111357 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x564be4120c10 0x564be41381e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T10:30:13.226 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:30:13.225+0000 7fced2ffd640 1 -- 172.21.15.114:0/2939111357 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x564be4120c10 msgr2=0x564be41381e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T10:30:13.226 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:30:13.225+0000 7fced2ffd640 1 --2- 172.21.15.114:0/2939111357 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x564be4120c10 0x564be41381e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:30:13.226 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:30:13.225+0000 7fced2ffd640 1 -- 172.21.15.114:0/2939111357 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x564be4111540 msgr2=0x564be412c410 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T10:30:13.226 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:30:13.225+0000 7fced2ffd640 1 --2- 172.21.15.114:0/2939111357 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x564be4111540 0x564be412c410 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:30:13.226 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:30:13.225+0000 7fced2ffd640 1 -- 172.21.15.114:0/2939111357 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x564be3ffeaf0 con 0x564be4111e60 2024-04-23T10:30:13.227 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:30:13.225+0000 7fced2ffd640 1 --2- 172.21.15.114:0/2939111357 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x564be4111e60 0x564be41200f0 secure :-1 s=READY pgs=625 cs=0 l=1 rev1=1 crypto rx=0x7fcec8005d00 tx=0x7fcec800b410 comp rx=0 tx=0).ready entity=mon.0 client_cookie=16d1b261e6e60f63 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T10:30:13.227 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:30:13.225+0000 7fceb97fa640 1 -- 172.21.15.114:0/2939111357 <== mon.0 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcec8005220 con 0x564be4111e60 2024-04-23T10:30:13.227 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:30:13.225+0000 7fceb97fa640 1 -- 172.21.15.114:0/2939111357 <== mon.0 v2:172.21.15.114:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fcec8024030 con 0x564be4111e60 2024-04-23T10:30:13.227 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:30:13.225+0000 7fceb97fa640 1 -- 172.21.15.114:0/2939111357 <== mon.0 v2:172.21.15.114:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcec8020450 con 0x564be4111e60 2024-04-23T10:30:13.227 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:30:13.226+0000 7fcedc9e9740 1 -- 172.21.15.114:0/2939111357 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x564be4111e60 msgr2=0x564be41200f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T10:30:13.227 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:30:13.226+0000 7fcedc9e9740 1 --2- 172.21.15.114:0/2939111357 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x564be4111e60 0x564be41200f0 secure :-1 s=READY pgs=625 cs=0 l=1 rev1=1 crypto rx=0x7fcec8005d00 tx=0x7fcec800b410 comp rx=0 tx=0).stop 2024-04-23T10:30:13.227 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:30:13.226+0000 7fcedc9e9740 1 -- 172.21.15.114:0/2939111357 shutdown_connections 2024-04-23T10:30:13.227 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:30:13.226+0000 7fcedc9e9740 1 --2- 172.21.15.114:0/2939111357 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x564be4120c10 0x564be41381e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:30:13.227 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:30:13.226+0000 7fcedc9e9740 1 --2- 172.21.15.114:0/2939111357 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x564be4111e60 0x564be41200f0 unknown :-1 s=CLOSED pgs=625 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:30:13.227 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:30:13.226+0000 7fcedc9e9740 1 --2- 172.21.15.114:0/2939111357 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x564be4111540 0x564be412c410 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:30:13.227 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:30:13.226+0000 7fcedc9e9740 1 -- 172.21.15.114:0/2939111357 >> 172.21.15.114:0/2939111357 conn(0x564be410db30 msgr2=0x564be410df10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T10:30:13.227 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:30:13.226+0000 7fcedc9e9740 1 -- 172.21.15.114:0/2939111357 shutdown_connections 2024-04-23T10:30:13.228 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:30:13.226+0000 7fcedc9e9740 1 -- 172.21.15.114:0/2939111357 wait complete. 2024-04-23T10:30:13.228 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:30:13.226+0000 7fcedc9e9740 1 -- start start 2024-04-23T10:30:13.228 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:30:13.226+0000 7fcedc9e9740 1 Processor -- start 2024-04-23T10:30:13.231 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:30:13.227+0000 7fcedc9e9740 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x564be4111540 0x564be4128b90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T10:30:13.231 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:30:13.227+0000 7fcedc9e9740 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x564be4111e60 0x564be411a290 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T10:30:13.231 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:30:13.227+0000 7fcedc9e9740 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x564be4120c10 0x564be411a7d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T10:30:13.231 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:30:13.227+0000 7fcedc9e9740 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x7fced41aa060 con 0x564be4111e60 2024-04-23T10:30:13.231 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:30:13.227+0000 7fcedc9e9740 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 -- 0x7fced41ab4d0 con 0x564be4111540 2024-04-23T10:30:13.231 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:30:13.227+0000 7fcedc9e9740 1 -- --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_getmap magic: 0 -- 0x7fced41aa730 con 0x564be4120c10 2024-04-23T10:30:13.231 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:30:13.227+0000 7fcedc9e9740 10 client.0 _ll_register_callbacks cb 0x7fcec40ee9a0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-23T10:30:13.232 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:30:13.227+0000 7fced37fe640 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x564be4111540 0x564be4128b90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T10:30:13.232 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:30:13.227+0000 7fced2ffd640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x564be4111e60 0x564be411a290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T10:30:13.232 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:30:13.227+0000 7fced37fe640 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x564be4111540 0x564be4128b90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.119:3300/0 says I am v2:172.21.15.114:40540/0 (socket says 172.21.15.114:40540) 2024-04-23T10:30:13.232 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:30:13.227+0000 7fced37fe640 1 -- 172.21.15.114:0/3839592263 learned_addr learned my addr 172.21.15.114:0/3839592263 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-04-23T10:30:13.232 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:30:13.227+0000 7fced3fff640 1 --2- 172.21.15.114:0/3839592263 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x564be4120c10 0x564be411a7d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T10:30:13.232 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:30:13.227+0000 7fced2ffd640 1 -- 172.21.15.114:0/3839592263 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x564be4120c10 msgr2=0x564be411a7d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T10:30:13.232 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:30:13.227+0000 7fced2ffd640 1 --2- 172.21.15.114:0/3839592263 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x564be4120c10 0x564be411a7d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:30:13.232 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:30:13.227+0000 7fced2ffd640 1 -- 172.21.15.114:0/3839592263 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x564be4111540 msgr2=0x564be4128b90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T10:30:13.232 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:30:13.227+0000 7fced2ffd640 1 --2- 172.21.15.114:0/3839592263 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x564be4111540 0x564be4128b90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:30:13.232 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:30:13.227+0000 7fced2ffd640 1 -- 172.21.15.114:0/3839592263 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({osdmap=0}) -- 0x564be411ae40 con 0x564be4111e60 2024-04-23T10:30:13.232 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:30:13.227+0000 7fced37fe640 1 --2- 172.21.15.114:0/3839592263 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x564be4111540 0x564be4128b90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-23T10:30:13.232 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:30:13.227+0000 7fced2ffd640 1 --2- 172.21.15.114:0/3839592263 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x564be4111e60 0x564be411a290 secure :-1 s=READY pgs=626 cs=0 l=1 rev1=1 crypto rx=0x564be411ee30 tx=0x7fcec800bbc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5f0955cbbf9b990d server_cookie=0 in_seq=0 out_seq=0 2024-04-23T10:30:13.232 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:30:13.227+0000 7fceb2ffd640 10 client.0 ms_handle_connect on v2:172.21.15.114:3300/0 2024-04-23T10:30:13.233 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:30:13.227+0000 7fceb2ffd640 1 -- 172.21.15.114:0/3839592263 <== mon.0 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcec80053a0 con 0x564be4111e60 2024-04-23T10:30:13.233 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:30:13.227+0000 7fceb2ffd640 1 -- 172.21.15.114:0/3839592263 <== mon.0 v2:172.21.15.114:3300/0 2 ==== osd_map(92..92 src has 1..92) ==== 8585+0+0 (secure 0 0 0) 0x7fcec8004730 con 0x564be4111e60 2024-04-23T10:30:13.233 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:30:13.227+0000 7fcedc9e9740 1 -- 172.21.15.114:0/3839592263 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fcec8024030 con 0x564be4111e60 2024-04-23T10:30:13.233 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:30:13.228+0000 7fceb2ffd640 1 -- 172.21.15.114:0/3839592263 <== mon.0 v2:172.21.15.114:3300/0 3 ==== mon_get_version_reply(handle=1 version=23) ==== 24+0+0 (secure 0 0 0) 0x7fcec8037050 con 0x564be4111e60 2024-04-23T10:30:13.233 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:30:13.228+0000 7fcedc9e9740 10 client.16038 fetch_fsmap learned FSMap version 23 2024-04-23T10:30:13.233 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:30:13.228+0000 7fcedc9e9740 1 -- 172.21.15.114:0/3839592263 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({config=0+,fsmap=23,monmap=4+}) -- 0x564be41968f0 con 0x564be4111e60 2024-04-23T10:30:13.233 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:30:13.228+0000 7fceb2ffd640 1 -- 172.21.15.114:0/3839592263 <== mon.0 v2:172.21.15.114:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fcec8024030 con 0x564be4111e60 2024-04-23T10:30:13.233 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:30:13.228+0000 7fceb2ffd640 1 -- 172.21.15.114:0/3839592263 <== mon.0 v2:172.21.15.114:3300/0 5 ==== fsmap(e 23) ==== 8350+0+0 (secure 0 0 0) 0x7fcec80334b0 con 0x564be4111e60 2024-04-23T10:30:13.233 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:30:13.228+0000 7fcedc9e9740 10 client.16038 fetch_fsmap finished waiting for FSMap version 23 2024-04-23T10:30:13.233 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:30:13.228+0000 7fcedc9e9740 10 client.16038 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-04-23T10:30:13.233 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:30:13.228+0000 7fcedc9e9740 20 client.16038 populate_metadata read hostname 'smithi114' 2024-04-23T10:30:13.233 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:30:13.228+0000 7fcedc9e9740 1 --2- 172.21.15.114:0/3839592263 >> [v2:172.21.15.119:6838/937297340,v1:172.21.15.119:6839/937297340] conn(0x564be4196fb0 0x564be41993d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T10:30:13.234 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:30:13.228+0000 7fcedc9e9740 4 client.16038 mds_command: new command op to 24545 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-04-23T10:30:13.234 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:30:13.228+0000 7fcedc9e9740 1 -- 172.21.15.114:0/3839592263 --> [v2:172.21.15.119:6838/937297340,v1:172.21.15.119:6839/937297340] -- command(tid 0: {"prefix": "flush journal"}) -- 0x564be407e200 con 0x564be4196fb0 2024-04-23T10:30:13.234 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:30:13.228+0000 7fced37fe640 1 --2- 172.21.15.114:0/3839592263 >> [v2:172.21.15.119:6838/937297340,v1:172.21.15.119:6839/937297340] conn(0x564be4196fb0 0x564be41993d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T10:30:13.234 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:30:13.229+0000 7fceb2ffd640 10 client.16038 ms_handle_connect on v2:172.21.15.119:6838/937297340 2024-04-23T10:30:13.234 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:30:13.229+0000 7fced37fe640 1 --2- 172.21.15.114:0/3839592263 >> [v2:172.21.15.119:6838/937297340,v1:172.21.15.119:6839/937297340] conn(0x564be4196fb0 0x564be41993d0 crc :-1 s=READY pgs=87 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=6b4f7230ac6a1a6 server_cookie=354c12d6d5a2311e in_seq=0 out_seq=0 2024-04-23T10:30:13.285 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:30:13.285+0000 7fceb2ffd640 1 -- 172.21.15.114:0/3839592263 <== mds.0 v2:172.21.15.119:6838/937297340 1 ==== command_reply(tid 0: 0 ) ==== 8+0+44 (crc 0 0 0) 0x564be407e200 con 0x564be4196fb0 2024-04-23T10:30:13.285 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:30:13.285+0000 7fceb2ffd640 10 client.16038 handle_command_reply: tid=0 2024-04-23T10:30:13.285 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:30:13.285+0000 7fcedc9e9740 1 client.16038 shutdown 2024-04-23T10:30:13.285 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:30:13.286+0000 7fcedc9e9740 1 -- 172.21.15.114:0/3839592263 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x564be4111e60 msgr2=0x564be411a290 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T10:30:13.286 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:30:13.286+0000 7fcedc9e9740 1 --2- 172.21.15.114:0/3839592263 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x564be4111e60 0x564be411a290 secure :-1 s=READY pgs=626 cs=0 l=1 rev1=1 crypto rx=0x564be411ee30 tx=0x7fcec800bbc0 comp rx=0 tx=0).stop 2024-04-23T10:30:13.286 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:30:13.286+0000 7fcedc9e9740 1 -- 172.21.15.114:0/3839592263 shutdown_connections 2024-04-23T10:30:13.286 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:30:13.286+0000 7fcedc9e9740 1 --2- 172.21.15.114:0/3839592263 >> [v2:172.21.15.119:6838/937297340,v1:172.21.15.119:6839/937297340] conn(0x564be4196fb0 0x564be41993d0 crc :-1 s=READY pgs=87 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:30:13.286 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:30:13.286+0000 7fcedc9e9740 1 --2- 172.21.15.114:0/3839592263 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x564be4120c10 0x564be411a7d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:30:13.286 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:30:13.286+0000 7fceb2ffd640 0 client.16038 ms_handle_reset on v2:172.21.15.119:6838/937297340 2024-04-23T10:30:13.286 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:30:13.286+0000 7fcedc9e9740 1 --2- 172.21.15.114:0/3839592263 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x564be4111e60 0x564be411a290 unknown :-1 s=CLOSED pgs=626 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:30:13.286 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:30:13.286+0000 7fcedc9e9740 1 --2- 172.21.15.114:0/3839592263 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x564be4111540 0x564be4128b90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:30:13.286 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:30:13.286+0000 7fcedc9e9740 1 -- 172.21.15.114:0/3839592263 >> 172.21.15.114:0/3839592263 conn(0x564be410db30 msgr2=0x564be410c420 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T10:30:13.286 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:30:13.286+0000 7fcedc9e9740 1 -- 172.21.15.114:0/3839592263 shutdown_connections 2024-04-23T10:30:13.286 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:30:13.286+0000 7fcedc9e9740 1 -- 172.21.15.114:0/3839592263 wait complete. 2024-04-23T10:30:13.286 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:30:13.286+0000 7fcedc9e9740 20 client.16038 trim_cache size 0 max 16384 2024-04-23T10:30:13.287 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stdout:{ 2024-04-23T10:30:13.287 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stdout: "message": "", 2024-04-23T10:30:13.287 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stdout: "return_code": 0 2024-04-23T10:30:13.287 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stdout:} 2024-04-23T10:30:13.287 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:30:13.286+0000 7fced9ccf640 1 -- 172.21.15.114:0/1028007654 >> [v2:172.21.15.114:6800/1395693466,v1:172.21.15.114:6801/1395693466] conn(0x7fcebc0a3dc0 msgr2=0x7fcebc0a6280 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T10:30:13.288 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:30:13.286+0000 7fced9ccf640 1 --2- 172.21.15.114:0/1028007654 >> [v2:172.21.15.114:6800/1395693466,v1:172.21.15.114:6801/1395693466] conn(0x7fcebc0a3dc0 0x7fcebc0a6280 secure :-1 s=READY pgs=93 cs=0 l=1 rev1=1 crypto rx=0x7fcecc005fd0 tx=0x7fcecc010740 comp rx=0 tx=0).stop 2024-04-23T10:30:13.288 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:30:13.286+0000 7fced9ccf640 1 -- 172.21.15.114:0/1028007654 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fced41010f0 msgr2=0x7fced4198780 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T10:30:13.288 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:30:13.286+0000 7fced9ccf640 1 --2- 172.21.15.114:0/1028007654 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fced41010f0 0x7fced4198780 secure :-1 s=READY pgs=623 cs=0 l=1 rev1=1 crypto rx=0x7fcec4005940 tx=0x7fcec40042f0 comp rx=0 tx=0).stop 2024-04-23T10:30:13.288 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:30:13.286+0000 7fced9ccf640 1 -- 172.21.15.114:0/1028007654 shutdown_connections 2024-04-23T10:30:13.288 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:30:13.286+0000 7fced9ccf640 1 --2- 172.21.15.114:0/1028007654 >> [v2:172.21.15.114:6800/1395693466,v1:172.21.15.114:6801/1395693466] conn(0x7fcebc0a3dc0 0x7fcebc0a6280 unknown :-1 s=CLOSED pgs=93 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:30:13.288 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:30:13.286+0000 7fced9ccf640 1 --2- 172.21.15.114:0/1028007654 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fced419cff0 0x7fced419d4a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:30:13.288 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:30:13.286+0000 7fced9ccf640 1 --2- 172.21.15.114:0/1028007654 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fced4106170 0x7fced4198cc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:30:13.288 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:30:13.286+0000 7fced9ccf640 1 --2- 172.21.15.114:0/1028007654 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fced41010f0 0x7fced4198780 unknown :-1 s=CLOSED pgs=623 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:30:13.288 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:30:13.286+0000 7fced9ccf640 1 -- 172.21.15.114:0/1028007654 >> 172.21.15.114:0/1028007654 conn(0x7fced4077fd0 msgr2=0x7fced4069a00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T10:30:13.288 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:30:13.287+0000 7fced9ccf640 1 -- 172.21.15.114:0/1028007654 shutdown_connections 2024-04-23T10:30:13.288 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:30:13.287+0000 7fced9ccf640 1 -- 172.21.15.114:0/1028007654 wait complete. 2024-04-23T10:30:13.915 INFO:tasks.workunit.client.0.smithi114.stdout: 1 199461 5.21 MB/sec execute 22 sec latency 40.340 ms 2024-04-23T10:30:14.254 INFO:journalctl@ceph.mon.a.smithi114.stdout:Apr 23 10:30:13 smithi114 ceph-mon[29255]: pgmap v111: 97 pgs: 97 active+clean; 65 MiB data, 976 MiB used, 1.0 TiB / 1.0 TiB avail; 86 MiB/s rd, 2.2 MiB/s wr, 357 op/s 2024-04-23T10:30:14.328 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 23 10:30:13 smithi204 ceph-mon[32649]: pgmap v111: 97 pgs: 97 active+clean; 65 MiB data, 976 MiB used, 1.0 TiB / 1.0 TiB avail; 86 MiB/s rd, 2.2 MiB/s wr, 357 op/s 2024-04-23T10:30:14.492 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 23 10:30:14 smithi119 ceph-mon[32608]: pgmap v111: 97 pgs: 97 active+clean; 65 MiB data, 976 MiB used, 1.0 TiB / 1.0 TiB avail; 86 MiB/s rd, 2.2 MiB/s wr, 357 op/s 2024-04-23T10:30:14.915 INFO:tasks.workunit.client.0.smithi114.stdout: 1 202148 5.72 MB/sec execute 23 sec latency 8.867 ms 2024-04-23T10:30:15.915 INFO:tasks.workunit.client.0.smithi114.stdout: 1 203369 5.64 MB/sec execute 24 sec latency 33.785 ms 2024-04-23T10:30:16.254 INFO:journalctl@ceph.mon.a.smithi114.stdout:Apr 23 10:30:16 smithi114 ceph-mon[29255]: pgmap v112: 97 pgs: 97 active+clean; 65 MiB data, 976 MiB used, 1.0 TiB / 1.0 TiB avail; 86 MiB/s rd, 2.2 MiB/s wr, 333 op/s 2024-04-23T10:30:16.327 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 23 10:30:16 smithi204 ceph-mon[32649]: pgmap v112: 97 pgs: 97 active+clean; 65 MiB data, 976 MiB used, 1.0 TiB / 1.0 TiB avail; 86 MiB/s rd, 2.2 MiB/s wr, 333 op/s 2024-04-23T10:30:16.491 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 23 10:30:16 smithi119 ceph-mon[32608]: pgmap v112: 97 pgs: 97 active+clean; 65 MiB data, 976 MiB used, 1.0 TiB / 1.0 TiB avail; 86 MiB/s rd, 2.2 MiB/s wr, 333 op/s 2024-04-23T10:30:16.915 INFO:tasks.workunit.client.0.smithi114.stdout: 1 205616 5.95 MB/sec execute 25 sec latency 167.972 ms 2024-04-23T10:30:17.915 INFO:tasks.workunit.client.0.smithi114.stdout: 1 206917 5.94 MB/sec execute 26 sec latency 180.658 ms 2024-04-23T10:30:18.254 INFO:journalctl@ceph.mon.a.smithi114.stdout:Apr 23 10:30:18 smithi114 ceph-mon[29255]: pgmap v113: 97 pgs: 97 active+clean; 67 MiB data, 1.0 GiB used, 1.0 TiB / 1.0 TiB avail; 101 MiB/s rd, 3.4 MiB/s wr, 534 op/s 2024-04-23T10:30:18.327 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 23 10:30:18 smithi204 ceph-mon[32649]: pgmap v113: 97 pgs: 97 active+clean; 67 MiB data, 1.0 GiB used, 1.0 TiB / 1.0 TiB avail; 101 MiB/s rd, 3.4 MiB/s wr, 534 op/s 2024-04-23T10:30:18.491 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 23 10:30:18 smithi119 ceph-mon[32608]: pgmap v113: 97 pgs: 97 active+clean; 67 MiB data, 1.0 GiB used, 1.0 TiB / 1.0 TiB avail; 101 MiB/s rd, 3.4 MiB/s wr, 534 op/s 2024-04-23T10:30:18.915 INFO:tasks.workunit.client.0.smithi114.stdout: 1 209420 6.30 MB/sec execute 27 sec latency 26.003 ms 2024-04-23T10:30:19.915 INFO:tasks.workunit.client.0.smithi114.stdout: 1 210688 6.30 MB/sec execute 28 sec latency 17.852 ms 2024-04-23T10:30:20.254 INFO:journalctl@ceph.mon.a.smithi114.stdout:Apr 23 10:30:20 smithi114 ceph-mon[29255]: pgmap v114: 97 pgs: 97 active+clean; 67 MiB data, 1.0 GiB used, 1.0 TiB / 1.0 TiB avail; 23 MiB/s rd, 2.5 MiB/s wr, 370 op/s 2024-04-23T10:30:20.327 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 23 10:30:20 smithi204 ceph-mon[32649]: pgmap v114: 97 pgs: 97 active+clean; 67 MiB data, 1.0 GiB used, 1.0 TiB / 1.0 TiB avail; 23 MiB/s rd, 2.5 MiB/s wr, 370 op/s 2024-04-23T10:30:20.491 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 23 10:30:20 smithi119 ceph-mon[32608]: pgmap v114: 97 pgs: 97 active+clean; 67 MiB data, 1.0 GiB used, 1.0 TiB / 1.0 TiB avail; 23 MiB/s rd, 2.5 MiB/s wr, 370 op/s 2024-04-23T10:30:20.915 INFO:tasks.workunit.client.0.smithi114.stdout: 1 213058 6.56 MB/sec execute 29 sec latency 62.839 ms 2024-04-23T10:30:21.915 INFO:tasks.workunit.client.0.smithi114.stdout: 1 214460 6.53 MB/sec execute 30 sec latency 14.117 ms 2024-04-23T10:30:22.327 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 23 10:30:22 smithi204 ceph-mon[32649]: pgmap v115: 97 pgs: 97 active+clean; 67 MiB data, 1.0 GiB used, 1.0 TiB / 1.0 TiB avail; 23 MiB/s rd, 2.5 MiB/s wr, 370 op/s 2024-04-23T10:30:22.491 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 23 10:30:22 smithi119 ceph-mon[32608]: pgmap v115: 97 pgs: 97 active+clean; 67 MiB data, 1.0 GiB used, 1.0 TiB / 1.0 TiB avail; 23 MiB/s rd, 2.5 MiB/s wr, 370 op/s 2024-04-23T10:30:22.503 INFO:journalctl@ceph.mon.a.smithi114.stdout:Apr 23 10:30:22 smithi114 ceph-mon[29255]: pgmap v115: 97 pgs: 97 active+clean; 67 MiB data, 1.0 GiB used, 1.0 TiB / 1.0 TiB avail; 23 MiB/s rd, 2.5 MiB/s wr, 370 op/s 2024-04-23T10:30:22.915 INFO:tasks.workunit.client.0.smithi114.stdout: 1 216731 6.76 MB/sec execute 31 sec latency 11.850 ms 2024-04-23T10:30:23.915 INFO:tasks.workunit.client.0.smithi114.stdout: 1 217749 6.66 MB/sec execute 32 sec latency 288.511 ms 2024-04-23T10:30:24.327 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 23 10:30:24 smithi204 ceph-mon[32649]: pgmap v116: 97 pgs: 97 active+clean; 73 MiB data, 1015 MiB used, 1.0 TiB / 1.0 TiB avail; 44 MiB/s rd, 3.6 MiB/s wr, 535 op/s 2024-04-23T10:30:24.492 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 23 10:30:24 smithi119 ceph-mon[32608]: pgmap v116: 97 pgs: 97 active+clean; 73 MiB data, 1015 MiB used, 1.0 TiB / 1.0 TiB avail; 44 MiB/s rd, 3.6 MiB/s wr, 535 op/s 2024-04-23T10:30:24.504 INFO:journalctl@ceph.mon.a.smithi114.stdout:Apr 23 10:30:24 smithi114 ceph-mon[29255]: pgmap v116: 97 pgs: 97 active+clean; 73 MiB data, 1015 MiB used, 1.0 TiB / 1.0 TiB avail; 44 MiB/s rd, 3.6 MiB/s wr, 535 op/s 2024-04-23T10:30:24.915 INFO:tasks.workunit.client.0.smithi114.stdout: 1 219413 6.74 MB/sec execute 33 sec latency 309.987 ms 2024-04-23T10:30:25.915 INFO:tasks.workunit.client.0.smithi114.stdout: 1 220788 6.75 MB/sec execute 34 sec latency 13.611 ms 2024-04-23T10:30:26.328 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 23 10:30:26 smithi204 ceph-mon[32649]: pgmap v117: 97 pgs: 97 active+clean; 73 MiB data, 1015 MiB used, 1.0 TiB / 1.0 TiB avail; 36 MiB/s rd, 2.3 MiB/s wr, 366 op/s 2024-04-23T10:30:26.392 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:30:26.393+0000 7f1f6d1cf640 1 -- 172.21.15.114:0/4080733171 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f1f68101860 msgr2=0x7f1f681058a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T10:30:26.392 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:30:26.393+0000 7f1f6d1cf640 1 --2- 172.21.15.114:0/4080733171 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f1f68101860 0x7f1f681058a0 secure :-1 s=READY pgs=627 cs=0 l=1 rev1=1 crypto rx=0x7f1f5c009900 tx=0x7f1f5c02a410 comp rx=0 tx=0).stop 2024-04-23T10:30:26.392 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:30:26.393+0000 7f1f6d1cf640 1 -- 172.21.15.114:0/4080733171 shutdown_connections 2024-04-23T10:30:26.393 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:30:26.393+0000 7f1f6d1cf640 1 --2- 172.21.15.114:0/4080733171 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f1f68105f80 0x7f1f6810db50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:30:26.393 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:30:26.393+0000 7f1f6d1cf640 1 --2- 172.21.15.114:0/4080733171 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f1f68101860 0x7f1f681058a0 unknown :-1 s=CLOSED pgs=627 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:30:26.393 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:30:26.393+0000 7f1f6d1cf640 1 --2- 172.21.15.114:0/4080733171 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f1f68100eb0 0x7f1f68101290 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:30:26.393 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:30:26.393+0000 7f1f6d1cf640 1 -- 172.21.15.114:0/4080733171 >> 172.21.15.114:0/4080733171 conn(0x7f1f680fc770 msgr2=0x7f1f680feb90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T10:30:26.393 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:30:26.393+0000 7f1f6d1cf640 1 -- 172.21.15.114:0/4080733171 shutdown_connections 2024-04-23T10:30:26.393 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:30:26.393+0000 7f1f6d1cf640 1 -- 172.21.15.114:0/4080733171 wait complete. 2024-04-23T10:30:26.395 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:30:26.394+0000 7f1f6d1cf640 1 Processor -- start 2024-04-23T10:30:26.395 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:30:26.394+0000 7f1f6d1cf640 1 -- start start 2024-04-23T10:30:26.395 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:30:26.394+0000 7f1f6d1cf640 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f1f68100eb0 0x7f1f6819a600 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T10:30:26.395 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:30:26.394+0000 7f1f6d1cf640 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f1f68105f80 0x7f1f6819ab40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T10:30:26.395 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:30:26.394+0000 7f1f6d1cf640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f1f6819eec0 0x7f1f6819f370 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T10:30:26.395 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:30:26.394+0000 7f1f6d1cf640 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x7f1f68110f70 con 0x7f1f6819eec0 2024-04-23T10:30:26.396 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:30:26.394+0000 7f1f6d1cf640 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 -- 0x7f1f68110df0 con 0x7f1f68105f80 2024-04-23T10:30:26.396 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:30:26.394+0000 7f1f6d1cf640 1 -- --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_getmap magic: 0 -- 0x7f1f681110f0 con 0x7f1f68100eb0 2024-04-23T10:30:26.396 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:30:26.395+0000 7f1f6c9ce640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f1f6819eec0 0x7f1f6819f370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T10:30:26.396 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:30:26.395+0000 7f1f6c9ce640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f1f6819eec0 0x7f1f6819f370 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.114:44620/0 (socket says 172.21.15.114:44620) 2024-04-23T10:30:26.396 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:30:26.395+0000 7f1f6c9ce640 1 -- 172.21.15.114:0/1300506096 learned_addr learned my addr 172.21.15.114:0/1300506096 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-04-23T10:30:26.396 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:30:26.395+0000 7f1f677fe640 1 --2- 172.21.15.114:0/1300506096 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f1f68105f80 0x7f1f6819ab40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T10:30:26.396 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:30:26.395+0000 7f1f67fff640 1 --2- 172.21.15.114:0/1300506096 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f1f68100eb0 0x7f1f6819a600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T10:30:26.396 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:30:26.395+0000 7f1f6c9ce640 1 -- 172.21.15.114:0/1300506096 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f1f68100eb0 msgr2=0x7f1f6819a600 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T10:30:26.396 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:30:26.395+0000 7f1f6c9ce640 1 --2- 172.21.15.114:0/1300506096 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f1f68100eb0 0x7f1f6819a600 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:30:26.396 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:30:26.395+0000 7f1f6c9ce640 1 -- 172.21.15.114:0/1300506096 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f1f68105f80 msgr2=0x7f1f6819ab40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T10:30:26.396 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:30:26.395+0000 7f1f6c9ce640 1 --2- 172.21.15.114:0/1300506096 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f1f68105f80 0x7f1f6819ab40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:30:26.396 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:30:26.395+0000 7f1f6c9ce640 1 -- 172.21.15.114:0/1300506096 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f1f6819f9b0 con 0x7f1f6819eec0 2024-04-23T10:30:26.396 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:30:26.395+0000 7f1f677fe640 1 --2- 172.21.15.114:0/1300506096 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f1f68105f80 0x7f1f6819ab40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-23T10:30:26.397 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:30:26.395+0000 7f1f67fff640 1 --2- 172.21.15.114:0/1300506096 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f1f68100eb0 0x7f1f6819a600 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-23T10:30:26.397 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:30:26.395+0000 7f1f6c9ce640 1 --2- 172.21.15.114:0/1300506096 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f1f6819eec0 0x7f1f6819f370 secure :-1 s=READY pgs=628 cs=0 l=1 rev1=1 crypto rx=0x7f1f6000b970 tx=0x7f1f6000be30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T10:30:26.397 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:30:26.395+0000 7f1f657fa640 1 -- 172.21.15.114:0/1300506096 <== mon.0 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1f60019070 con 0x7f1f6819eec0 2024-04-23T10:30:26.397 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:30:26.395+0000 7f1f6d1cf640 1 -- 172.21.15.114:0/1300506096 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f1f6819fca0 con 0x7f1f6819eec0 2024-04-23T10:30:26.397 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:30:26.395+0000 7f1f6d1cf640 1 -- 172.21.15.114:0/1300506096 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f1f681a74e0 con 0x7f1f6819eec0 2024-04-23T10:30:26.397 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:30:26.396+0000 7f1f657fa640 1 -- 172.21.15.114:0/1300506096 <== mon.0 v2:172.21.15.114:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f1f60014440 con 0x7f1f6819eec0 2024-04-23T10:30:26.397 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:30:26.396+0000 7f1f657fa640 1 -- 172.21.15.114:0/1300506096 <== mon.0 v2:172.21.15.114:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1f600075d0 con 0x7f1f6819eec0 2024-04-23T10:30:26.403 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:30:26.398+0000 7f1f657fa640 1 -- 172.21.15.114:0/1300506096 <== mon.0 v2:172.21.15.114:3300/0 4 ==== mgrmap(e 26) ==== 138778+0+0 (secure 0 0 0) 0x7f1f6001c070 con 0x7f1f6819eec0 2024-04-23T10:30:26.403 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:30:26.398+0000 7f1f71228740 1 Processor -- start 2024-04-23T10:30:26.403 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:30:26.398+0000 7f1f71228740 1 -- start start 2024-04-23T10:30:26.403 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:30:26.398+0000 7f1f71228740 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55d5df1747e0 0x55d5df174bc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T10:30:26.403 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:30:26.398+0000 7f1f71228740 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x55d5df175100 0x55d5df17f480 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T10:30:26.403 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:30:26.398+0000 7f1f71228740 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x55d5df17f9c0 0x55d5df181db0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T10:30:26.403 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:30:26.398+0000 7f1f71228740 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x55d5df0d9970 con 0x55d5df1747e0 2024-04-23T10:30:26.403 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:30:26.398+0000 7f1f71228740 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 -- 0x55d5df0764a0 con 0x55d5df17f9c0 2024-04-23T10:30:26.403 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:30:26.398+0000 7f1f71228740 1 -- --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_getmap magic: 0 -- 0x55d5df0d5930 con 0x55d5df175100 2024-04-23T10:30:26.403 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:30:26.399+0000 7f1f677fe640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55d5df1747e0 0x55d5df174bc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T10:30:26.403 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:30:26.399+0000 7f1f657fa640 1 --2- 172.21.15.114:0/1300506096 >> [v2:172.21.15.114:6800/1395693466,v1:172.21.15.114:6801/1395693466] conn(0x7f1f500a3d10 0x7f1f500a61d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T10:30:26.404 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:30:26.399+0000 7f1f677fe640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55d5df1747e0 0x55d5df174bc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.114:44628/0 (socket says 172.21.15.114:44628) 2024-04-23T10:30:26.404 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:30:26.399+0000 7f1f677fe640 1 -- 172.21.15.114:0/3650608530 learned_addr learned my addr 172.21.15.114:0/3650608530 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-04-23T10:30:26.404 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:30:26.399+0000 7f1f67fff640 1 --2- 172.21.15.114:0/3650608530 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x55d5df175100 0x55d5df17f480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T10:30:26.404 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:30:26.399+0000 7f1f677fe640 1 --2- 172.21.15.114:0/3650608530 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x55d5df17f9c0 0x55d5df181db0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T10:30:26.404 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:30:26.399+0000 7f1f6c9ce640 1 --2- 172.21.15.114:0/1300506096 >> [v2:172.21.15.114:6800/1395693466,v1:172.21.15.114:6801/1395693466] conn(0x7f1f500a3d10 0x7f1f500a61d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T10:30:26.404 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:30:26.399+0000 7f1f67fff640 1 -- 172.21.15.114:0/3650608530 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x55d5df17f9c0 msgr2=0x55d5df181db0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T10:30:26.404 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:30:26.399+0000 7f1f67fff640 1 --2- 172.21.15.114:0/3650608530 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x55d5df17f9c0 0x55d5df181db0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:30:26.404 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:30:26.399+0000 7f1f67fff640 1 -- 172.21.15.114:0/3650608530 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55d5df1747e0 msgr2=0x55d5df174bc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T10:30:26.404 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:30:26.399+0000 7f1f67fff640 1 --2- 172.21.15.114:0/3650608530 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55d5df1747e0 0x55d5df174bc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:30:26.404 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:30:26.399+0000 7f1f67fff640 1 -- 172.21.15.114:0/3650608530 --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55d5dedec210 con 0x55d5df175100 2024-04-23T10:30:26.404 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:30:26.399+0000 7f1f657fa640 1 -- 172.21.15.114:0/1300506096 <== mon.0 v2:172.21.15.114:3300/0 5 ==== osd_map(92..92 src has 1..92) ==== 8585+0+0 (secure 0 0 0) 0x7f1f600cfc00 con 0x7f1f6819eec0 2024-04-23T10:30:26.404 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:30:26.399+0000 7f1f677fe640 1 --2- 172.21.15.114:0/3650608530 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55d5df1747e0 0x55d5df174bc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-23T10:30:26.404 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:30:26.399+0000 7f1f67fff640 1 --2- 172.21.15.114:0/3650608530 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x55d5df175100 0x55d5df17f480 secure :-1 s=READY pgs=68 cs=0 l=1 rev1=1 crypto rx=0x7f1f58009c40 tx=0x7f1f58010740 comp rx=0 tx=0).ready entity=mon.1 client_cookie=3d7729ecbee727db server_cookie=0 in_seq=0 out_seq=0 2024-04-23T10:30:26.404 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:30:26.399+0000 7f1f4e7fc640 1 -- 172.21.15.114:0/3650608530 <== mon.1 v2:172.21.15.204:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1f58004030 con 0x55d5df175100 2024-04-23T10:30:26.405 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:30:26.399+0000 7f1f4e7fc640 1 -- 172.21.15.114:0/3650608530 <== mon.1 v2:172.21.15.204:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f1f580035f0 con 0x55d5df175100 2024-04-23T10:30:26.405 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:30:26.399+0000 7f1f6c9ce640 1 --2- 172.21.15.114:0/1300506096 >> [v2:172.21.15.114:6800/1395693466,v1:172.21.15.114:6801/1395693466] conn(0x7f1f500a3d10 0x7f1f500a61d0 secure :-1 s=READY pgs=94 cs=0 l=1 rev1=1 crypto rx=0x7f1f60000f80 tx=0x7f1f60011000 comp rx=0 tx=0).ready entity=mgr.15291 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T10:30:26.405 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:30:26.400+0000 7f1f4e7fc640 1 -- 172.21.15.114:0/3650608530 <== mon.1 v2:172.21.15.204:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1f580126b0 con 0x55d5df175100 2024-04-23T10:30:26.405 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:30:26.400+0000 7f1f71228740 1 -- 172.21.15.114:0/3650608530 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x55d5df175100 msgr2=0x55d5df17f480 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T10:30:26.405 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:30:26.400+0000 7f1f71228740 1 --2- 172.21.15.114:0/3650608530 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x55d5df175100 0x55d5df17f480 secure :-1 s=READY pgs=68 cs=0 l=1 rev1=1 crypto rx=0x7f1f58009c40 tx=0x7f1f58010740 comp rx=0 tx=0).stop 2024-04-23T10:30:26.405 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:30:26.400+0000 7f1f71228740 1 -- 172.21.15.114:0/3650608530 shutdown_connections 2024-04-23T10:30:26.405 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:30:26.400+0000 7f1f71228740 1 --2- 172.21.15.114:0/3650608530 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x55d5df17f9c0 0x55d5df181db0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:30:26.405 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:30:26.400+0000 7f1f71228740 1 --2- 172.21.15.114:0/3650608530 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x55d5df175100 0x55d5df17f480 unknown :-1 s=CLOSED pgs=68 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:30:26.405 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:30:26.400+0000 7f1f71228740 1 --2- 172.21.15.114:0/3650608530 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55d5df1747e0 0x55d5df174bc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:30:26.405 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:30:26.400+0000 7f1f71228740 1 -- 172.21.15.114:0/3650608530 >> 172.21.15.114:0/3650608530 conn(0x55d5df170cf0 msgr2=0x55d5df1710d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T10:30:26.405 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:30:26.400+0000 7f1f71228740 1 -- 172.21.15.114:0/3650608530 shutdown_connections 2024-04-23T10:30:26.405 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:30:26.400+0000 7f1f71228740 1 -- 172.21.15.114:0/3650608530 wait complete. 2024-04-23T10:30:26.405 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:30:26.401+0000 7f1f71228740 1 -- start start 2024-04-23T10:30:26.405 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:30:26.401+0000 7f1f71228740 1 Processor -- start 2024-04-23T10:30:26.405 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:30:26.401+0000 7f1f71228740 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55d5df1747e0 0x55d5df17f280 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T10:30:26.406 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:30:26.401+0000 7f1f71228740 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x55d5df175100 0x55d5df17c570 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T10:30:26.406 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:30:26.401+0000 7f1f71228740 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x55d5df17f9c0 0x55d5df17cab0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T10:30:26.406 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:30:26.401+0000 7f1f71228740 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x55d5df1825f0 con 0x55d5df1747e0 2024-04-23T10:30:26.406 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:30:26.401+0000 7f1f71228740 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 -- 0x55d5df0740b0 con 0x55d5df17f9c0 2024-04-23T10:30:26.406 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:30:26.401+0000 7f1f71228740 1 -- --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_getmap magic: 0 -- 0x55d5df182770 con 0x55d5df175100 2024-04-23T10:30:26.406 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:30:26.401+0000 7f1f71228740 10 client.0 _ll_register_callbacks cb 0x55d5df061d30 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-23T10:30:26.406 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:30:26.402+0000 7f1f677fe640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55d5df1747e0 0x55d5df17f280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T10:30:26.406 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:30:26.402+0000 7f1f67fff640 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x55d5df175100 0x55d5df17c570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T10:30:26.406 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:30:26.402+0000 7f1f677fe640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55d5df1747e0 0x55d5df17f280 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.114:44640/0 (socket says 172.21.15.114:44640) 2024-04-23T10:30:26.406 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:30:26.402+0000 7f1f67fff640 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x55d5df175100 0x55d5df17c570 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.204:3300/0 says I am v2:172.21.15.114:36962/0 (socket says 172.21.15.114:36962) 2024-04-23T10:30:26.406 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:30:26.402+0000 7f1f677fe640 1 -- 172.21.15.114:0/1065506955 learned_addr learned my addr 172.21.15.114:0/1065506955 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-04-23T10:30:26.406 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:30:26.402+0000 7f1f677fe640 1 --2- 172.21.15.114:0/1065506955 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x55d5df17f9c0 0x55d5df17cab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T10:30:26.406 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:30:26.402+0000 7f1f677fe640 1 -- 172.21.15.114:0/1065506955 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x55d5df175100 msgr2=0x55d5df17c570 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T10:30:26.407 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:30:26.402+0000 7f1f677fe640 1 --2- 172.21.15.114:0/1065506955 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x55d5df175100 0x55d5df17c570 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:30:26.407 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:30:26.402+0000 7f1f677fe640 1 -- 172.21.15.114:0/1065506955 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x55d5df17f9c0 msgr2=0x55d5df17cab0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T10:30:26.407 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:30:26.402+0000 7f1f677fe640 1 --2- 172.21.15.114:0/1065506955 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x55d5df17f9c0 0x55d5df17cab0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:30:26.407 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:30:26.402+0000 7f1f677fe640 1 -- 172.21.15.114:0/1065506955 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55d5deeb8620 con 0x55d5df1747e0 2024-04-23T10:30:26.407 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:30:26.402+0000 7f1f677fe640 1 --2- 172.21.15.114:0/1065506955 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55d5df1747e0 0x55d5df17f280 secure :-1 s=READY pgs=629 cs=0 l=1 rev1=1 crypto rx=0x7f1f5c009850 tx=0x7f1f5c000bc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7bcb0e2230ef84b3 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T10:30:26.407 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:30:26.402+0000 7f1f477fe640 10 client.0 ms_handle_connect on v2:172.21.15.114:3300/0 2024-04-23T10:30:26.407 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:30:26.402+0000 7f1f477fe640 1 -- 172.21.15.114:0/1065506955 <== mon.0 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1f5c035070 con 0x55d5df1747e0 2024-04-23T10:30:26.407 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:30:26.402+0000 7f1f477fe640 1 -- 172.21.15.114:0/1065506955 <== mon.0 v2:172.21.15.114:3300/0 2 ==== osd_map(92..92 src has 1..92) ==== 8585+0+0 (secure 0 0 0) 0x7f1f5c004460 con 0x55d5df1747e0 2024-04-23T10:30:26.407 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:30:26.402+0000 7f1f71228740 1 -- 172.21.15.114:0/1065506955 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f1f580035f0 con 0x55d5df1747e0 2024-04-23T10:30:26.407 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:30:26.403+0000 7f1f477fe640 1 -- 172.21.15.114:0/1065506955 <== mon.0 v2:172.21.15.114:3300/0 3 ==== mon_get_version_reply(handle=1 version=23) ==== 24+0+0 (secure 0 0 0) 0x7f1f5c046040 con 0x55d5df1747e0 2024-04-23T10:30:26.407 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:30:26.403+0000 7f1f71228740 10 client.16050 fetch_fsmap learned FSMap version 23 2024-04-23T10:30:26.407 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:30:26.403+0000 7f1f71228740 1 -- 172.21.15.114:0/1065506955 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({config=0+,fsmap=23,monmap=4+}) -- 0x55d5df18e520 con 0x55d5df1747e0 2024-04-23T10:30:26.407 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:30:26.403+0000 7f1f477fe640 1 -- 172.21.15.114:0/1065506955 <== mon.0 v2:172.21.15.114:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f1f580035f0 con 0x55d5df1747e0 2024-04-23T10:30:26.407 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:30:26.403+0000 7f1f477fe640 1 -- 172.21.15.114:0/1065506955 <== mon.0 v2:172.21.15.114:3300/0 5 ==== fsmap(e 23) ==== 8350+0+0 (secure 0 0 0) 0x7f1f5c031660 con 0x55d5df1747e0 2024-04-23T10:30:26.407 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:30:26.403+0000 7f1f71228740 10 client.16050 fetch_fsmap finished waiting for FSMap version 23 2024-04-23T10:30:26.408 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:30:26.403+0000 7f1f71228740 10 client.16050 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-04-23T10:30:26.408 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:30:26.403+0000 7f1f71228740 20 client.16050 populate_metadata read hostname 'smithi114' 2024-04-23T10:30:26.408 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:30:26.403+0000 7f1f71228740 1 --2- 172.21.15.114:0/1065506955 >> [v2:172.21.15.119:6838/937297340,v1:172.21.15.119:6839/937297340] conn(0x55d5df18fa60 0x55d5df191f00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T10:30:26.408 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:30:26.403+0000 7f1f71228740 4 client.16050 mds_command: new command op to 24545 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-23T10:30:26.408 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:30:26.403+0000 7f1f71228740 1 -- 172.21.15.114:0/1065506955 --> [v2:172.21.15.119:6838/937297340,v1:172.21.15.119:6839/937297340] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55d5df0e7390 con 0x55d5df18fa60 2024-04-23T10:30:26.408 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:30:26.403+0000 7f1f67fff640 1 --2- 172.21.15.114:0/1065506955 >> [v2:172.21.15.119:6838/937297340,v1:172.21.15.119:6839/937297340] conn(0x55d5df18fa60 0x55d5df191f00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T10:30:26.408 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:30:26.404+0000 7f1f477fe640 10 client.16050 ms_handle_connect on v2:172.21.15.119:6838/937297340 2024-04-23T10:30:26.408 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:30:26.404+0000 7f1f67fff640 1 --2- 172.21.15.114:0/1065506955 >> [v2:172.21.15.119:6838/937297340,v1:172.21.15.119:6839/937297340] conn(0x55d5df18fa60 0x55d5df191f00 crc :-1 s=READY pgs=88 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=fd6c9c07d54250b server_cookie=ead2a1eea9c39d72 in_seq=0 out_seq=0 2024-04-23T10:30:26.409 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:30:26.404+0000 7f1f477fe640 1 -- 172.21.15.114:0/1065506955 <== mds.0 v2:172.21.15.119:6838/937297340 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x55d5df0e7390 con 0x55d5df18fa60 2024-04-23T10:30:26.409 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:30:26.404+0000 7f1f477fe640 10 client.16050 handle_command_reply: tid=0 2024-04-23T10:30:26.409 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:30:26.404+0000 7f1f71228740 1 client.16050 shutdown 2024-04-23T10:30:26.409 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:30:26.405+0000 7f1f71228740 1 -- 172.21.15.114:0/1065506955 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55d5df1747e0 msgr2=0x55d5df17f280 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T10:30:26.409 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:30:26.405+0000 7f1f71228740 1 --2- 172.21.15.114:0/1065506955 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55d5df1747e0 0x55d5df17f280 secure :-1 s=READY pgs=629 cs=0 l=1 rev1=1 crypto rx=0x7f1f5c009850 tx=0x7f1f5c000bc0 comp rx=0 tx=0).stop 2024-04-23T10:30:26.410 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:30:26.405+0000 7f1f71228740 1 -- 172.21.15.114:0/1065506955 shutdown_connections 2024-04-23T10:30:26.410 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:30:26.405+0000 7f1f71228740 1 --2- 172.21.15.114:0/1065506955 >> [v2:172.21.15.119:6838/937297340,v1:172.21.15.119:6839/937297340] conn(0x55d5df18fa60 0x55d5df191f00 crc :-1 s=READY pgs=88 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:30:26.410 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:30:26.405+0000 7f1f71228740 1 --2- 172.21.15.114:0/1065506955 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x55d5df17f9c0 0x55d5df17cab0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:30:26.410 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:30:26.405+0000 7f1f71228740 1 --2- 172.21.15.114:0/1065506955 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x55d5df175100 0x55d5df17c570 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:30:26.410 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:30:26.405+0000 7f1f477fe640 0 client.16050 ms_handle_reset on v2:172.21.15.119:6838/937297340 2024-04-23T10:30:26.410 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:30:26.405+0000 7f1f71228740 1 --2- 172.21.15.114:0/1065506955 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55d5df1747e0 0x55d5df17f280 unknown :-1 s=CLOSED pgs=629 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:30:26.410 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:30:26.405+0000 7f1f71228740 1 -- 172.21.15.114:0/1065506955 >> 172.21.15.114:0/1065506955 conn(0x55d5df170cf0 msgr2=0x55d5df176a70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T10:30:26.410 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:30:26.405+0000 7f1f71228740 1 -- 172.21.15.114:0/1065506955 shutdown_connections 2024-04-23T10:30:26.410 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:30:26.405+0000 7f1f71228740 1 -- 172.21.15.114:0/1065506955 wait complete. 2024-04-23T10:30:26.410 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:30:26.405+0000 7f1f71228740 20 client.16050 trim_cache size 0 max 16384 2024-04-23T10:30:26.425 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:30:26.421+0000 7f1f71228740 1 Processor -- start 2024-04-23T10:30:26.425 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:30:26.421+0000 7f1f71228740 1 -- start start 2024-04-23T10:30:26.426 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:30:26.421+0000 7f1f71228740 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55d5df1747e0 0x55d5df18b510 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T10:30:26.426 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:30:26.421+0000 7f1f71228740 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x55d5df175100 0x55d5df177cb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T10:30:26.426 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:30:26.421+0000 7f1f71228740 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x55d5df17f9c0 0x55d5df16fc60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T10:30:26.426 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:30:26.421+0000 7f1f71228740 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x7f1f540111f0 con 0x55d5df1747e0 2024-04-23T10:30:26.426 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:30:26.421+0000 7f1f71228740 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 -- 0x7f1f5400ea90 con 0x55d5df175100 2024-04-23T10:30:26.426 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:30:26.421+0000 7f1f71228740 1 -- --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_getmap magic: 0 -- 0x7f1f5400c8e0 con 0x55d5df17f9c0 2024-04-23T10:30:26.426 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:30:26.422+0000 7f1f677fe640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55d5df1747e0 0x55d5df18b510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T10:30:26.426 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:30:26.422+0000 7f1f67fff640 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x55d5df175100 0x55d5df177cb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T10:30:26.426 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:30:26.422+0000 7f1f67fff640 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x55d5df175100 0x55d5df177cb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.119:3300/0 says I am v2:172.21.15.114:50336/0 (socket says 172.21.15.114:50336) 2024-04-23T10:30:26.426 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:30:26.422+0000 7f1f677fe640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55d5df1747e0 0x55d5df18b510 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.114:44652/0 (socket says 172.21.15.114:44652) 2024-04-23T10:30:26.426 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:30:26.422+0000 7f1f67fff640 1 -- 172.21.15.114:0/977855098 learned_addr learned my addr 172.21.15.114:0/977855098 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-04-23T10:30:26.427 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:30:26.422+0000 7f1f677fe640 1 --2- 172.21.15.114:0/977855098 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x55d5df17f9c0 0x55d5df16fc60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T10:30:26.427 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:30:26.422+0000 7f1f67fff640 1 -- 172.21.15.114:0/977855098 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x55d5df17f9c0 msgr2=0x55d5df16fc60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T10:30:26.427 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:30:26.422+0000 7f1f67fff640 1 --2- 172.21.15.114:0/977855098 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x55d5df17f9c0 0x55d5df16fc60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:30:26.427 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:30:26.422+0000 7f1f67fff640 1 -- 172.21.15.114:0/977855098 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55d5df1747e0 msgr2=0x55d5df18b510 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T10:30:26.427 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:30:26.422+0000 7f1f67fff640 1 --2- 172.21.15.114:0/977855098 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55d5df1747e0 0x55d5df18b510 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:30:26.427 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:30:26.422+0000 7f1f67fff640 1 -- 172.21.15.114:0/977855098 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55d5df174f30 con 0x55d5df175100 2024-04-23T10:30:26.427 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:30:26.422+0000 7f1f67fff640 1 --2- 172.21.15.114:0/977855098 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x55d5df175100 0x55d5df177cb0 secure :-1 s=READY pgs=72 cs=0 l=1 rev1=1 crypto rx=0x7f1f58009c40 tx=0x7f1f58010fb0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=1c2e661ee3abc085 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T10:30:26.427 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:30:26.422+0000 7f1f4dffb640 1 -- 172.21.15.114:0/977855098 <== mon.2 v2:172.21.15.119:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1f58005240 con 0x55d5df175100 2024-04-23T10:30:26.427 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:30:26.422+0000 7f1f4dffb640 1 -- 172.21.15.114:0/977855098 <== mon.2 v2:172.21.15.119:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f1f580053e0 con 0x55d5df175100 2024-04-23T10:30:26.427 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:30:26.422+0000 7f1f4dffb640 1 -- 172.21.15.114:0/977855098 <== mon.2 v2:172.21.15.119:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1f5802a380 con 0x55d5df175100 2024-04-23T10:30:26.427 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:30:26.423+0000 7f1f71228740 1 -- 172.21.15.114:0/977855098 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x55d5df175100 msgr2=0x55d5df177cb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T10:30:26.427 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:30:26.423+0000 7f1f71228740 1 --2- 172.21.15.114:0/977855098 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x55d5df175100 0x55d5df177cb0 secure :-1 s=READY pgs=72 cs=0 l=1 rev1=1 crypto rx=0x7f1f58009c40 tx=0x7f1f58010fb0 comp rx=0 tx=0).stop 2024-04-23T10:30:26.427 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:30:26.423+0000 7f1f71228740 1 -- 172.21.15.114:0/977855098 shutdown_connections 2024-04-23T10:30:26.428 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:30:26.423+0000 7f1f71228740 1 --2- 172.21.15.114:0/977855098 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x55d5df17f9c0 0x55d5df16fc60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:30:26.428 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:30:26.423+0000 7f1f71228740 1 --2- 172.21.15.114:0/977855098 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x55d5df175100 0x55d5df177cb0 unknown :-1 s=CLOSED pgs=72 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:30:26.428 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:30:26.423+0000 7f1f71228740 1 --2- 172.21.15.114:0/977855098 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55d5df1747e0 0x55d5df18b510 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:30:26.428 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:30:26.423+0000 7f1f71228740 1 -- 172.21.15.114:0/977855098 >> 172.21.15.114:0/977855098 conn(0x55d5df170cf0 msgr2=0x55d5df16f550 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T10:30:26.428 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:30:26.423+0000 7f1f71228740 1 -- 172.21.15.114:0/977855098 shutdown_connections 2024-04-23T10:30:26.428 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:30:26.423+0000 7f1f71228740 1 -- 172.21.15.114:0/977855098 wait complete. 2024-04-23T10:30:26.428 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:30:26.424+0000 7f1f71228740 1 -- start start 2024-04-23T10:30:26.428 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:30:26.424+0000 7f1f71228740 1 Processor -- start 2024-04-23T10:30:26.428 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:30:26.424+0000 7f1f71228740 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55d5df1747e0 0x55d5df187730 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T10:30:26.428 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:30:26.424+0000 7f1f71228740 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x55d5df175100 0x55d5df187c70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T10:30:26.428 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:30:26.424+0000 7f1f71228740 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x55d5df17f9c0 0x55d5df182bc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T10:30:26.428 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:30:26.424+0000 7f1f71228740 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x7f1f5400cf50 con 0x55d5df1747e0 2024-04-23T10:30:26.428 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:30:26.424+0000 7f1f71228740 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 -- 0x7f1f5400e3c0 con 0x55d5df17f9c0 2024-04-23T10:30:26.428 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:30:26.424+0000 7f1f71228740 1 -- --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_getmap magic: 0 -- 0x7f1f5400d620 con 0x55d5df175100 2024-04-23T10:30:26.429 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:30:26.424+0000 7f1f71228740 10 client.0 _ll_register_callbacks cb 0x7f1f5c004a10 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-23T10:30:26.429 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:30:26.424+0000 7f1f677fe640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55d5df1747e0 0x55d5df187730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T10:30:26.429 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:30:26.424+0000 7f1f677fe640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55d5df1747e0 0x55d5df187730 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.114:44666/0 (socket says 172.21.15.114:44666) 2024-04-23T10:30:26.429 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:30:26.424+0000 7f1f677fe640 1 -- 172.21.15.114:0/3926228411 learned_addr learned my addr 172.21.15.114:0/3926228411 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-04-23T10:30:26.429 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:30:26.424+0000 7f1f67fff640 1 --2- 172.21.15.114:0/3926228411 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x55d5df175100 0x55d5df187c70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T10:30:26.429 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:30:26.424+0000 7f1f677fe640 1 --2- 172.21.15.114:0/3926228411 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x55d5df17f9c0 0x55d5df182bc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T10:30:26.429 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:30:26.425+0000 7f1f677fe640 1 -- 172.21.15.114:0/3926228411 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x55d5df175100 msgr2=0x55d5df187c70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T10:30:26.429 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:30:26.425+0000 7f1f677fe640 1 --2- 172.21.15.114:0/3926228411 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x55d5df175100 0x55d5df187c70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:30:26.429 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:30:26.425+0000 7f1f677fe640 1 -- 172.21.15.114:0/3926228411 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x55d5df17f9c0 msgr2=0x55d5df182bc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T10:30:26.429 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:30:26.425+0000 7f1f677fe640 1 --2- 172.21.15.114:0/3926228411 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x55d5df17f9c0 0x55d5df182bc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:30:26.429 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:30:26.425+0000 7f1f677fe640 1 -- 172.21.15.114:0/3926228411 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55d5df183330 con 0x55d5df1747e0 2024-04-23T10:30:26.429 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:30:26.425+0000 7f1f677fe640 1 --2- 172.21.15.114:0/3926228411 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55d5df1747e0 0x55d5df187730 secure :-1 s=READY pgs=630 cs=0 l=1 rev1=1 crypto rx=0x7f1f5c031b80 tx=0x7f1f5c005fd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=56598168c3198286 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T10:30:26.429 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:30:26.425+0000 7f1f477fe640 10 client.0 ms_handle_connect on v2:172.21.15.114:3300/0 2024-04-23T10:30:26.429 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:30:26.425+0000 7f1f477fe640 1 -- 172.21.15.114:0/3926228411 <== mon.0 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1f5c035070 con 0x55d5df1747e0 2024-04-23T10:30:26.430 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:30:26.425+0000 7f1f477fe640 1 -- 172.21.15.114:0/3926228411 <== mon.0 v2:172.21.15.114:3300/0 2 ==== osd_map(92..92 src has 1..92) ==== 8585+0+0 (secure 0 0 0) 0x7f1f5c0314a0 con 0x55d5df1747e0 2024-04-23T10:30:26.430 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:30:26.425+0000 7f1f71228740 1 -- 172.21.15.114:0/3926228411 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f1f580053e0 con 0x55d5df1747e0 2024-04-23T10:30:26.430 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:30:26.425+0000 7f1f477fe640 1 -- 172.21.15.114:0/3926228411 <== mon.0 v2:172.21.15.114:3300/0 3 ==== mon_get_version_reply(handle=1 version=23) ==== 24+0+0 (secure 0 0 0) 0x7f1f5c064050 con 0x55d5df1747e0 2024-04-23T10:30:26.430 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:30:26.425+0000 7f1f71228740 10 client.16056 fetch_fsmap learned FSMap version 23 2024-04-23T10:30:26.430 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:30:26.425+0000 7f1f71228740 1 -- 172.21.15.114:0/3926228411 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({config=0+,fsmap=23,monmap=4+}) -- 0x55d5df1fe2c0 con 0x55d5df1747e0 2024-04-23T10:30:26.430 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:30:26.426+0000 7f1f477fe640 1 -- 172.21.15.114:0/3926228411 <== mon.0 v2:172.21.15.114:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f1f580053e0 con 0x55d5df1747e0 2024-04-23T10:30:26.430 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:30:26.426+0000 7f1f477fe640 1 -- 172.21.15.114:0/3926228411 <== mon.0 v2:172.21.15.114:3300/0 5 ==== fsmap(e 23) ==== 8350+0+0 (secure 0 0 0) 0x7f1f5c03d480 con 0x55d5df1747e0 2024-04-23T10:30:26.430 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:30:26.426+0000 7f1f71228740 10 client.16056 fetch_fsmap finished waiting for FSMap version 23 2024-04-23T10:30:26.430 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:30:26.426+0000 7f1f71228740 10 client.16056 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-04-23T10:30:26.430 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:30:26.426+0000 7f1f71228740 20 client.16056 populate_metadata read hostname 'smithi114' 2024-04-23T10:30:26.430 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:30:26.426+0000 7f1f71228740 1 --2- 172.21.15.114:0/3926228411 >> [v2:172.21.15.119:6838/937297340,v1:172.21.15.119:6839/937297340] conn(0x55d5df1fea10 0x55d5df200e30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T10:30:26.430 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:30:26.426+0000 7f1f71228740 4 client.16056 mds_command: new command op to 24545 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-04-23T10:30:26.430 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:30:26.426+0000 7f1f71228740 1 -- 172.21.15.114:0/3926228411 --> [v2:172.21.15.119:6838/937297340,v1:172.21.15.119:6839/937297340] -- command(tid 0: {"prefix": "flush journal"}) -- 0x55d5df0e1200 con 0x55d5df1fea10 2024-04-23T10:30:26.431 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:30:26.426+0000 7f1f67fff640 1 --2- 172.21.15.114:0/3926228411 >> [v2:172.21.15.119:6838/937297340,v1:172.21.15.119:6839/937297340] conn(0x55d5df1fea10 0x55d5df200e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T10:30:26.431 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:30:26.426+0000 7f1f477fe640 10 client.16056 ms_handle_connect on v2:172.21.15.119:6838/937297340 2024-04-23T10:30:26.431 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:30:26.426+0000 7f1f67fff640 1 --2- 172.21.15.114:0/3926228411 >> [v2:172.21.15.119:6838/937297340,v1:172.21.15.119:6839/937297340] conn(0x55d5df1fea10 0x55d5df200e30 crc :-1 s=READY pgs=89 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=19ef889392215164 server_cookie=836278895e29108 in_seq=0 out_seq=0 2024-04-23T10:30:26.477 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:30:26.477+0000 7f1f477fe640 1 -- 172.21.15.114:0/3926228411 <== mds.0 v2:172.21.15.119:6838/937297340 1 ==== command_reply(tid 0: 0 ) ==== 8+0+44 (crc 0 0 0) 0x55d5df0e1200 con 0x55d5df1fea10 2024-04-23T10:30:26.477 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:30:26.477+0000 7f1f477fe640 10 client.16056 handle_command_reply: tid=0 2024-04-23T10:30:26.477 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:30:26.477+0000 7f1f71228740 1 client.16056 shutdown 2024-04-23T10:30:26.477 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:30:26.478+0000 7f1f71228740 1 -- 172.21.15.114:0/3926228411 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55d5df1747e0 msgr2=0x55d5df187730 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T10:30:26.477 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:30:26.478+0000 7f1f71228740 1 --2- 172.21.15.114:0/3926228411 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55d5df1747e0 0x55d5df187730 secure :-1 s=READY pgs=630 cs=0 l=1 rev1=1 crypto rx=0x7f1f5c031b80 tx=0x7f1f5c005fd0 comp rx=0 tx=0).stop 2024-04-23T10:30:26.477 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:30:26.478+0000 7f1f71228740 1 -- 172.21.15.114:0/3926228411 shutdown_connections 2024-04-23T10:30:26.477 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:30:26.478+0000 7f1f71228740 1 --2- 172.21.15.114:0/3926228411 >> [v2:172.21.15.119:6838/937297340,v1:172.21.15.119:6839/937297340] conn(0x55d5df1fea10 0x55d5df200e30 crc :-1 s=READY pgs=89 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:30:26.477 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:30:26.478+0000 7f1f71228740 1 --2- 172.21.15.114:0/3926228411 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x55d5df17f9c0 0x55d5df182bc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:30:26.478 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:30:26.478+0000 7f1f71228740 1 --2- 172.21.15.114:0/3926228411 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x55d5df175100 0x55d5df187c70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:30:26.478 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:30:26.478+0000 7f1f477fe640 0 client.16056 ms_handle_reset on v2:172.21.15.119:6838/937297340 2024-04-23T10:30:26.478 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:30:26.478+0000 7f1f71228740 1 --2- 172.21.15.114:0/3926228411 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55d5df1747e0 0x55d5df187730 unknown :-1 s=CLOSED pgs=630 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:30:26.478 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:30:26.478+0000 7f1f71228740 1 -- 172.21.15.114:0/3926228411 >> 172.21.15.114:0/3926228411 conn(0x55d5df170cf0 msgr2=0x55d5df176a20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T10:30:26.478 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:30:26.478+0000 7f1f71228740 1 -- 172.21.15.114:0/3926228411 shutdown_connections 2024-04-23T10:30:26.478 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:30:26.478+0000 7f1f71228740 1 -- 172.21.15.114:0/3926228411 wait complete. 2024-04-23T10:30:26.478 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:30:26.478+0000 7f1f71228740 20 client.16056 trim_cache size 0 max 16384 2024-04-23T10:30:26.478 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stdout:{ 2024-04-23T10:30:26.478 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stdout: "message": "", 2024-04-23T10:30:26.478 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stdout: "return_code": 0 2024-04-23T10:30:26.478 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stdout:} 2024-04-23T10:30:26.479 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:30:26.478+0000 7f1f6d1cf640 1 -- 172.21.15.114:0/1300506096 >> [v2:172.21.15.114:6800/1395693466,v1:172.21.15.114:6801/1395693466] conn(0x7f1f500a3d10 msgr2=0x7f1f500a61d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T10:30:26.479 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:30:26.478+0000 7f1f6d1cf640 1 --2- 172.21.15.114:0/1300506096 >> [v2:172.21.15.114:6800/1395693466,v1:172.21.15.114:6801/1395693466] conn(0x7f1f500a3d10 0x7f1f500a61d0 secure :-1 s=READY pgs=94 cs=0 l=1 rev1=1 crypto rx=0x7f1f60000f80 tx=0x7f1f60011000 comp rx=0 tx=0).stop 2024-04-23T10:30:26.479 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:30:26.478+0000 7f1f6d1cf640 1 -- 172.21.15.114:0/1300506096 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f1f6819eec0 msgr2=0x7f1f6819f370 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T10:30:26.479 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:30:26.478+0000 7f1f6d1cf640 1 --2- 172.21.15.114:0/1300506096 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f1f6819eec0 0x7f1f6819f370 secure :-1 s=READY pgs=628 cs=0 l=1 rev1=1 crypto rx=0x7f1f6000b970 tx=0x7f1f6000be30 comp rx=0 tx=0).stop 2024-04-23T10:30:26.479 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:30:26.479+0000 7f1f6d1cf640 1 -- 172.21.15.114:0/1300506096 shutdown_connections 2024-04-23T10:30:26.480 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:30:26.479+0000 7f1f6d1cf640 1 --2- 172.21.15.114:0/1300506096 >> [v2:172.21.15.114:6800/1395693466,v1:172.21.15.114:6801/1395693466] conn(0x7f1f500a3d10 0x7f1f500a61d0 unknown :-1 s=CLOSED pgs=94 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:30:26.480 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:30:26.479+0000 7f1f6d1cf640 1 --2- 172.21.15.114:0/1300506096 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f1f6819eec0 0x7f1f6819f370 unknown :-1 s=CLOSED pgs=628 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:30:26.480 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:30:26.479+0000 7f1f6d1cf640 1 --2- 172.21.15.114:0/1300506096 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f1f68105f80 0x7f1f6819ab40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:30:26.480 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:30:26.479+0000 7f1f6d1cf640 1 --2- 172.21.15.114:0/1300506096 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f1f68100eb0 0x7f1f6819a600 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:30:26.480 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:30:26.479+0000 7f1f6d1cf640 1 -- 172.21.15.114:0/1300506096 >> 172.21.15.114:0/1300506096 conn(0x7f1f680fc770 msgr2=0x7f1f6810c6b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T10:30:26.480 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:30:26.479+0000 7f1f6d1cf640 1 -- 172.21.15.114:0/1300506096 shutdown_connections 2024-04-23T10:30:26.480 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:30:26.479+0000 7f1f6d1cf640 1 -- 172.21.15.114:0/1300506096 wait complete. 2024-04-23T10:30:26.492 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 23 10:30:26 smithi119 ceph-mon[32608]: pgmap v117: 97 pgs: 97 active+clean; 73 MiB data, 1015 MiB used, 1.0 TiB / 1.0 TiB avail; 36 MiB/s rd, 2.3 MiB/s wr, 366 op/s 2024-04-23T10:30:26.503 INFO:journalctl@ceph.mon.a.smithi114.stdout:Apr 23 10:30:26 smithi114 ceph-mon[29255]: pgmap v117: 97 pgs: 97 active+clean; 73 MiB data, 1015 MiB used, 1.0 TiB / 1.0 TiB avail; 36 MiB/s rd, 2.3 MiB/s wr, 366 op/s 2024-04-23T10:30:26.915 INFO:tasks.workunit.client.0.smithi114.stdout: 1 222383 6.74 MB/sec execute 35 sec latency 268.961 ms 2024-04-23T10:30:27.916 INFO:tasks.workunit.client.0.smithi114.stdout: 1 223953 6.89 MB/sec execute 36 sec latency 30.577 ms 2024-04-23T10:30:28.327 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 23 10:30:28 smithi204 ceph-mon[32649]: pgmap v118: 97 pgs: 97 active+clean; 78 MiB data, 1.0 GiB used, 1.0 TiB / 1.0 TiB avail; 88 MiB/s rd, 3.4 MiB/s wr, 526 op/s 2024-04-23T10:30:28.492 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 23 10:30:28 smithi119 ceph-mon[32608]: pgmap v118: 97 pgs: 97 active+clean; 78 MiB data, 1.0 GiB used, 1.0 TiB / 1.0 TiB avail; 88 MiB/s rd, 3.4 MiB/s wr, 526 op/s 2024-04-23T10:30:28.504 INFO:journalctl@ceph.mon.a.smithi114.stdout:Apr 23 10:30:28 smithi114 ceph-mon[29255]: pgmap v118: 97 pgs: 97 active+clean; 78 MiB data, 1.0 GiB used, 1.0 TiB / 1.0 TiB avail; 88 MiB/s rd, 3.4 MiB/s wr, 526 op/s 2024-04-23T10:30:28.915 INFO:tasks.workunit.client.0.smithi114.stdout: 1 225848 6.89 MB/sec execute 37 sec latency 43.891 ms 2024-04-23T10:30:29.915 INFO:tasks.workunit.client.0.smithi114.stdout: 1 227490 7.03 MB/sec execute 38 sec latency 9.882 ms 2024-04-23T10:30:30.293 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 23 10:30:30 smithi119 ceph-mon[32608]: pgmap v119: 97 pgs: 97 active+clean; 78 MiB data, 1.0 GiB used, 1.0 TiB / 1.0 TiB avail; 74 MiB/s rd, 2.2 MiB/s wr, 324 op/s 2024-04-23T10:30:30.293 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 23 10:30:30 smithi119 ceph-mon[32608]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=e=up:active} 3 up:standby-replay 6 up:standby 2024-04-23T10:30:30.327 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 23 10:30:30 smithi204 ceph-mon[32649]: pgmap v119: 97 pgs: 97 active+clean; 78 MiB data, 1.0 GiB used, 1.0 TiB / 1.0 TiB avail; 74 MiB/s rd, 2.2 MiB/s wr, 324 op/s 2024-04-23T10:30:30.328 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 23 10:30:30 smithi204 ceph-mon[32649]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=e=up:active} 3 up:standby-replay 6 up:standby 2024-04-23T10:30:30.505 INFO:journalctl@ceph.mon.a.smithi114.stdout:Apr 23 10:30:30 smithi114 ceph-mon[29255]: pgmap v119: 97 pgs: 97 active+clean; 78 MiB data, 1.0 GiB used, 1.0 TiB / 1.0 TiB avail; 74 MiB/s rd, 2.2 MiB/s wr, 324 op/s 2024-04-23T10:30:30.505 INFO:journalctl@ceph.mon.a.smithi114.stdout:Apr 23 10:30:30 smithi114 ceph-mon[29255]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=e=up:active} 3 up:standby-replay 6 up:standby 2024-04-23T10:30:30.916 INFO:tasks.workunit.client.0.smithi114.stdout: 1 229480 7.03 MB/sec execute 39 sec latency 20.019 ms 2024-04-23T10:30:31.915 INFO:tasks.workunit.client.0.smithi114.stdout: 1 231054 7.16 MB/sec execute 40 sec latency 15.334 ms 2024-04-23T10:30:32.327 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 23 10:30:32 smithi204 ceph-mon[32649]: pgmap v120: 97 pgs: 97 active+clean; 78 MiB data, 1.0 GiB used, 1.0 TiB / 1.0 TiB avail; 74 MiB/s rd, 2.2 MiB/s wr, 324 op/s 2024-04-23T10:30:32.491 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 23 10:30:32 smithi119 ceph-mon[32608]: pgmap v120: 97 pgs: 97 active+clean; 78 MiB data, 1.0 GiB used, 1.0 TiB / 1.0 TiB avail; 74 MiB/s rd, 2.2 MiB/s wr, 324 op/s 2024-04-23T10:30:32.503 INFO:journalctl@ceph.mon.a.smithi114.stdout:Apr 23 10:30:32 smithi114 ceph-mon[29255]: pgmap v120: 97 pgs: 97 active+clean; 78 MiB data, 1.0 GiB used, 1.0 TiB / 1.0 TiB avail; 74 MiB/s rd, 2.2 MiB/s wr, 324 op/s 2024-04-23T10:30:32.916 INFO:tasks.workunit.client.0.smithi114.stdout: 1 233012 7.15 MB/sec execute 41 sec latency 13.902 ms 2024-04-23T10:30:33.916 INFO:tasks.workunit.client.0.smithi114.stdout: 1 234605 7.27 MB/sec execute 42 sec latency 18.921 ms 2024-04-23T10:30:34.327 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 23 10:30:34 smithi204 ceph-mon[32649]: pgmap v121: 97 pgs: 97 active+clean; 76 MiB data, 1023 MiB used, 1.0 TiB / 1.0 TiB avail; 90 MiB/s rd, 3.3 MiB/s wr, 516 op/s 2024-04-23T10:30:34.492 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 23 10:30:34 smithi119 ceph-mon[32608]: pgmap v121: 97 pgs: 97 active+clean; 76 MiB data, 1023 MiB used, 1.0 TiB / 1.0 TiB avail; 90 MiB/s rd, 3.3 MiB/s wr, 516 op/s 2024-04-23T10:30:34.504 INFO:journalctl@ceph.mon.a.smithi114.stdout:Apr 23 10:30:34 smithi114 ceph-mon[29255]: pgmap v121: 97 pgs: 97 active+clean; 76 MiB data, 1023 MiB used, 1.0 TiB / 1.0 TiB avail; 90 MiB/s rd, 3.3 MiB/s wr, 516 op/s 2024-04-23T10:30:34.916 INFO:tasks.workunit.client.0.smithi114.stdout: 1 236537 7.26 MB/sec execute 43 sec latency 26.164 ms 2024-04-23T10:30:35.327 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 23 10:30:35 smithi204 ceph-mon[32649]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=e=up:active} 3 up:standby-replay 6 up:standby 2024-04-23T10:30:35.491 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 23 10:30:35 smithi119 ceph-mon[32608]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=e=up:active} 3 up:standby-replay 6 up:standby 2024-04-23T10:30:35.503 INFO:journalctl@ceph.mon.a.smithi114.stdout:Apr 23 10:30:35 smithi114 ceph-mon[29255]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=e=up:active} 3 up:standby-replay 6 up:standby 2024-04-23T10:30:35.916 INFO:tasks.workunit.client.0.smithi114.stdout: 1 238116 7.37 MB/sec execute 44 sec latency 31.263 ms 2024-04-23T10:30:36.327 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 23 10:30:36 smithi204 ceph-mon[32649]: pgmap v122: 97 pgs: 97 active+clean; 76 MiB data, 1023 MiB used, 1.0 TiB / 1.0 TiB avail; 69 MiB/s rd, 2.1 MiB/s wr, 351 op/s 2024-04-23T10:30:36.492 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 23 10:30:36 smithi119 ceph-mon[32608]: pgmap v122: 97 pgs: 97 active+clean; 76 MiB data, 1023 MiB used, 1.0 TiB / 1.0 TiB avail; 69 MiB/s rd, 2.1 MiB/s wr, 351 op/s 2024-04-23T10:30:36.503 INFO:journalctl@ceph.mon.a.smithi114.stdout:Apr 23 10:30:36 smithi114 ceph-mon[29255]: pgmap v122: 97 pgs: 97 active+clean; 76 MiB data, 1023 MiB used, 1.0 TiB / 1.0 TiB avail; 69 MiB/s rd, 2.1 MiB/s wr, 351 op/s 2024-04-23T10:30:36.916 INFO:tasks.workunit.client.0.smithi114.stdout: 1 240125 7.37 MB/sec execute 45 sec latency 28.767 ms 2024-04-23T10:30:37.916 INFO:tasks.workunit.client.0.smithi114.stdout: 1 241559 7.46 MB/sec execute 46 sec latency 225.483 ms 2024-04-23T10:30:38.327 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 23 10:30:38 smithi204 ceph-mon[32649]: pgmap v123: 97 pgs: 97 active+clean; 87 MiB data, 1.0 GiB used, 1.0 TiB / 1.0 TiB avail; 76 MiB/s rd, 3.7 MiB/s wr, 497 op/s 2024-04-23T10:30:38.491 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 23 10:30:38 smithi119 ceph-mon[32608]: pgmap v123: 97 pgs: 97 active+clean; 87 MiB data, 1.0 GiB used, 1.0 TiB / 1.0 TiB avail; 76 MiB/s rd, 3.7 MiB/s wr, 497 op/s 2024-04-23T10:30:38.504 INFO:journalctl@ceph.mon.a.smithi114.stdout:Apr 23 10:30:38 smithi114 ceph-mon[29255]: pgmap v123: 97 pgs: 97 active+clean; 87 MiB data, 1.0 GiB used, 1.0 TiB / 1.0 TiB avail; 76 MiB/s rd, 3.7 MiB/s wr, 497 op/s 2024-04-23T10:30:38.916 INFO:tasks.workunit.client.0.smithi114.stdout: 1 243100 7.43 MB/sec execute 47 sec latency 16.119 ms 2024-04-23T10:30:39.568 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:30:39.567+0000 7fe60c491640 1 -- 172.21.15.114:0/3034940508 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fe6041021d0 msgr2=0x7fe6041025b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T10:30:39.568 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:30:39.567+0000 7fe60c491640 1 --2- 172.21.15.114:0/3034940508 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fe6041021d0 0x7fe6041025b0 secure :-1 s=READY pgs=631 cs=0 l=1 rev1=1 crypto rx=0x7fe600009900 tx=0x7fe60002a410 comp rx=0 tx=0).stop 2024-04-23T10:30:39.568 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:30:39.567+0000 7fe60c491640 1 -- 172.21.15.114:0/3034940508 shutdown_connections 2024-04-23T10:30:39.569 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:30:39.567+0000 7fe60c491640 1 --2- 172.21.15.114:0/3034940508 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fe6041072a0 0x7fe60410ee70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:30:39.569 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:30:39.567+0000 7fe60c491640 1 --2- 172.21.15.114:0/3034940508 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fe604102b80 0x7fe604106bc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:30:39.569 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:30:39.567+0000 7fe60c491640 1 --2- 172.21.15.114:0/3034940508 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fe6041021d0 0x7fe6041025b0 unknown :-1 s=CLOSED pgs=631 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:30:39.569 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:30:39.567+0000 7fe60c491640 1 -- 172.21.15.114:0/3034940508 >> 172.21.15.114:0/3034940508 conn(0x7fe6040793e0 msgr2=0x7fe6040797f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T10:30:39.569 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:30:39.567+0000 7fe60c491640 1 -- 172.21.15.114:0/3034940508 shutdown_connections 2024-04-23T10:30:39.569 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:30:39.567+0000 7fe60c491640 1 -- 172.21.15.114:0/3034940508 wait complete. 2024-04-23T10:30:39.569 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:30:39.568+0000 7fe60c491640 1 Processor -- start 2024-04-23T10:30:39.569 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:30:39.568+0000 7fe60c491640 1 -- start start 2024-04-23T10:30:39.569 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:30:39.568+0000 7fe60c491640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fe6041021d0 0x7fe6041986a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T10:30:39.569 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:30:39.568+0000 7fe60c491640 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fe6041072a0 0x7fe604198be0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T10:30:39.569 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:30:39.568+0000 7fe60c491640 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fe60419cf60 0x7fe60419d410 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T10:30:39.569 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:30:39.568+0000 7fe60c491640 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x7fe6041121e0 con 0x7fe6041021d0 2024-04-23T10:30:39.569 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:30:39.568+0000 7fe60c491640 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 -- 0x7fe604112060 con 0x7fe6041072a0 2024-04-23T10:30:39.569 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:30:39.568+0000 7fe60c491640 1 -- --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_getmap magic: 0 -- 0x7fe604112360 con 0x7fe60419cf60 2024-04-23T10:30:39.569 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:30:39.568+0000 7fe60a22d640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fe6041021d0 0x7fe6041986a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T10:30:39.570 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:30:39.568+0000 7fe60a22d640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fe6041021d0 0x7fe6041986a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.114:47190/0 (socket says 172.21.15.114:47190) 2024-04-23T10:30:39.570 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:30:39.568+0000 7fe609a2c640 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fe6041072a0 0x7fe604198be0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T10:30:39.570 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:30:39.568+0000 7fe60a22d640 1 -- 172.21.15.114:0/168447071 learned_addr learned my addr 172.21.15.114:0/168447071 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-04-23T10:30:39.570 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:30:39.568+0000 7fe60aa2e640 1 --2- 172.21.15.114:0/168447071 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fe60419cf60 0x7fe60419d410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T10:30:39.570 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:30:39.568+0000 7fe60a22d640 1 -- 172.21.15.114:0/168447071 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fe60419cf60 msgr2=0x7fe60419d410 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T10:30:39.570 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:30:39.568+0000 7fe60a22d640 1 --2- 172.21.15.114:0/168447071 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fe60419cf60 0x7fe60419d410 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:30:39.570 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:30:39.568+0000 7fe60a22d640 1 -- 172.21.15.114:0/168447071 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fe6041072a0 msgr2=0x7fe604198be0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T10:30:39.570 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:30:39.568+0000 7fe60a22d640 1 --2- 172.21.15.114:0/168447071 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fe6041072a0 0x7fe604198be0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:30:39.570 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:30:39.568+0000 7fe60a22d640 1 -- 172.21.15.114:0/168447071 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe60419da50 con 0x7fe6041021d0 2024-04-23T10:30:39.570 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:30:39.568+0000 7fe609a2c640 1 --2- 172.21.15.114:0/168447071 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fe6041072a0 0x7fe604198be0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-23T10:30:39.570 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:30:39.569+0000 7fe60a22d640 1 --2- 172.21.15.114:0/168447071 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fe6041021d0 0x7fe6041986a0 secure :-1 s=READY pgs=632 cs=0 l=1 rev1=1 crypto rx=0x7fe600005940 tx=0x7fe6000040c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T10:30:39.570 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:30:39.569+0000 7fe5fb7fe640 1 -- 172.21.15.114:0/168447071 <== mon.0 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe60003f030 con 0x7fe6041021d0 2024-04-23T10:30:39.570 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:30:39.569+0000 7fe5fb7fe640 1 -- 172.21.15.114:0/168447071 <== mon.0 v2:172.21.15.114:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fe6000044c0 con 0x7fe6041021d0 2024-04-23T10:30:39.570 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:30:39.569+0000 7fe5fb7fe640 1 -- 172.21.15.114:0/168447071 <== mon.0 v2:172.21.15.114:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe600035070 con 0x7fe6041021d0 2024-04-23T10:30:39.571 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:30:39.569+0000 7fe60c491640 1 -- 172.21.15.114:0/168447071 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fe60419dce0 con 0x7fe6041021d0 2024-04-23T10:30:39.571 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:30:39.569+0000 7fe60c491640 1 -- 172.21.15.114:0/168447071 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fe6041a54a0 con 0x7fe6041021d0 2024-04-23T10:30:39.577 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:30:39.571+0000 7fe5fb7fe640 1 -- 172.21.15.114:0/168447071 <== mon.0 v2:172.21.15.114:3300/0 4 ==== mgrmap(e 26) ==== 138778+0+0 (secure 0 0 0) 0x7fe600042030 con 0x7fe6041021d0 2024-04-23T10:30:39.577 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:30:39.571+0000 7fe60f1ae740 1 Processor -- start 2024-04-23T10:30:39.578 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:30:39.572+0000 7fe60f1ae740 1 -- start start 2024-04-23T10:30:39.578 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:30:39.572+0000 7fe5fb7fe640 1 --2- 172.21.15.114:0/168447071 >> [v2:172.21.15.114:6800/1395693466,v1:172.21.15.114:6801/1395693466] conn(0x7fe5f00a3dc0 0x7fe5f00a6280 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T10:30:39.578 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:30:39.572+0000 7fe60f1ae740 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x559b311b5500 0x559b311b58e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T10:30:39.578 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:30:39.572+0000 7fe60f1ae740 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x559b311b5e20 0x559b311c46d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T10:30:39.578 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:30:39.572+0000 7fe60f1ae740 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x559b311c4c10 0x559b311c7000 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T10:30:39.578 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:30:39.572+0000 7fe60f1ae740 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x559b3111a970 con 0x559b311b5500 2024-04-23T10:30:39.578 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:30:39.572+0000 7fe60f1ae740 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 -- 0x559b310b74a0 con 0x559b311b5e20 2024-04-23T10:30:39.578 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:30:39.572+0000 7fe60f1ae740 1 -- --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_getmap magic: 0 -- 0x559b31116930 con 0x559b311c4c10 2024-04-23T10:30:39.578 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:30:39.572+0000 7fe60aa2e640 1 --2- 172.21.15.114:0/168447071 >> [v2:172.21.15.114:6800/1395693466,v1:172.21.15.114:6801/1395693466] conn(0x7fe5f00a3dc0 0x7fe5f00a6280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T10:30:39.578 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:30:39.572+0000 7fe60a22d640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x559b311b5500 0x559b311b58e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T10:30:39.578 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:30:39.572+0000 7fe609a2c640 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x559b311b5e20 0x559b311c46d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T10:30:39.578 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:30:39.572+0000 7fe609a2c640 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x559b311b5e20 0x559b311c46d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.119:3300/0 says I am v2:172.21.15.114:59194/0 (socket says 172.21.15.114:59194) 2024-04-23T10:30:39.578 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:30:39.572+0000 7fe60a22d640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x559b311b5500 0x559b311b58e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.114:47202/0 (socket says 172.21.15.114:47202) 2024-04-23T10:30:39.579 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:30:39.572+0000 7fe609a2c640 1 -- 172.21.15.114:0/1499629769 learned_addr learned my addr 172.21.15.114:0/1499629769 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-04-23T10:30:39.579 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:30:39.572+0000 7fe5fb7fe640 1 -- 172.21.15.114:0/168447071 <== mon.0 v2:172.21.15.114:3300/0 5 ==== osd_map(92..92 src has 1..92) ==== 8585+0+0 (secure 0 0 0) 0x7fe6000f0b80 con 0x7fe6041021d0 2024-04-23T10:30:39.579 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:30:39.572+0000 7fe60aa2e640 1 --2- 172.21.15.114:0/1499629769 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x559b311c4c10 0x559b311c7000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T10:30:39.579 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:30:39.572+0000 7fe609a2c640 1 -- 172.21.15.114:0/1499629769 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x559b311c4c10 msgr2=0x559b311c7000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T10:30:39.579 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:30:39.572+0000 7fe609a2c640 1 --2- 172.21.15.114:0/1499629769 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x559b311c4c10 0x559b311c7000 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:30:39.579 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:30:39.572+0000 7fe609a2c640 1 -- 172.21.15.114:0/1499629769 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x559b311b5500 msgr2=0x559b311b58e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T10:30:39.579 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:30:39.572+0000 7fe609a2c640 1 --2- 172.21.15.114:0/1499629769 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x559b311b5500 0x559b311b58e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:30:39.579 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:30:39.572+0000 7fe609a2c640 1 -- 172.21.15.114:0/1499629769 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x559b30e2d210 con 0x559b311b5e20 2024-04-23T10:30:39.579 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:30:39.573+0000 7fe60aa2e640 1 --2- 172.21.15.114:0/168447071 >> [v2:172.21.15.114:6800/1395693466,v1:172.21.15.114:6801/1395693466] conn(0x7fe5f00a3dc0 0x7fe5f00a6280 secure :-1 s=READY pgs=95 cs=0 l=1 rev1=1 crypto rx=0x7fe5fc005fd0 tx=0x7fe5fc013ae0 comp rx=0 tx=0).ready entity=mgr.15291 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T10:30:39.579 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:30:39.573+0000 7fe609a2c640 1 --2- 172.21.15.114:0/1499629769 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x559b311b5e20 0x559b311c46d0 secure :-1 s=READY pgs=73 cs=0 l=1 rev1=1 crypto rx=0x7fe5f40114c0 tx=0x7fe5f4011980 comp rx=0 tx=0).ready entity=mon.2 client_cookie=21aff6e9c83781d9 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T10:30:39.579 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:30:39.573+0000 7fe5f8bfe640 1 -- 172.21.15.114:0/1499629769 <== mon.2 v2:172.21.15.119:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe5f4005020 con 0x559b311b5e20 2024-04-23T10:30:39.579 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:30:39.573+0000 7fe5f8bfe640 1 -- 172.21.15.114:0/1499629769 <== mon.2 v2:172.21.15.119:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fe5f40047a0 con 0x559b311b5e20 2024-04-23T10:30:39.579 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:30:39.573+0000 7fe5f8bfe640 1 -- 172.21.15.114:0/1499629769 <== mon.2 v2:172.21.15.119:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe5f400f8e0 con 0x559b311b5e20 2024-04-23T10:30:39.579 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:30:39.573+0000 7fe60f1ae740 1 -- 172.21.15.114:0/1499629769 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x559b311b5e20 msgr2=0x559b311c46d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T10:30:39.580 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:30:39.573+0000 7fe60f1ae740 1 --2- 172.21.15.114:0/1499629769 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x559b311b5e20 0x559b311c46d0 secure :-1 s=READY pgs=73 cs=0 l=1 rev1=1 crypto rx=0x7fe5f40114c0 tx=0x7fe5f4011980 comp rx=0 tx=0).stop 2024-04-23T10:30:39.580 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:30:39.574+0000 7fe60f1ae740 1 -- 172.21.15.114:0/1499629769 shutdown_connections 2024-04-23T10:30:39.580 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:30:39.574+0000 7fe60f1ae740 1 --2- 172.21.15.114:0/1499629769 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x559b311c4c10 0x559b311c7000 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:30:39.580 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:30:39.574+0000 7fe60f1ae740 1 --2- 172.21.15.114:0/1499629769 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x559b311b5e20 0x559b311c46d0 unknown :-1 s=CLOSED pgs=73 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:30:39.580 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:30:39.574+0000 7fe60f1ae740 1 --2- 172.21.15.114:0/1499629769 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x559b311b5500 0x559b311b58e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:30:39.580 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:30:39.574+0000 7fe60f1ae740 1 -- 172.21.15.114:0/1499629769 >> 172.21.15.114:0/1499629769 conn(0x559b311b1b30 msgr2=0x559b311b1f10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T10:30:39.580 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:30:39.574+0000 7fe60f1ae740 1 -- 172.21.15.114:0/1499629769 shutdown_connections 2024-04-23T10:30:39.580 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:30:39.574+0000 7fe60f1ae740 1 -- 172.21.15.114:0/1499629769 wait complete. 2024-04-23T10:30:39.580 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:30:39.574+0000 7fe60f1ae740 1 -- start start 2024-04-23T10:30:39.580 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:30:39.574+0000 7fe60f1ae740 1 Processor -- start 2024-04-23T10:30:39.580 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:30:39.574+0000 7fe60f1ae740 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x559b311b5500 0x559b311c45a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T10:30:39.580 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:30:39.574+0000 7fe60f1ae740 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x559b311b5e20 0x559b311c1890 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T10:30:39.580 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:30:39.574+0000 7fe60f1ae740 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x559b311c4c10 0x559b311c1e10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T10:30:39.580 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:30:39.574+0000 7fe60f1ae740 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x559b311c7840 con 0x559b311b5e20 2024-04-23T10:30:39.581 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:30:39.574+0000 7fe60f1ae740 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 -- 0x559b310b50b0 con 0x559b311c4c10 2024-04-23T10:30:39.581 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:30:39.574+0000 7fe60f1ae740 1 -- --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_getmap magic: 0 -- 0x559b311c79c0 con 0x559b311b5500 2024-04-23T10:30:39.581 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:30:39.574+0000 7fe60f1ae740 10 client.0 _ll_register_callbacks cb 0x559b310a2d30 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-23T10:30:39.581 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:30:39.574+0000 7fe609a2c640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x559b311b5e20 0x559b311c1890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T10:30:39.581 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:30:39.574+0000 7fe609a2c640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x559b311b5e20 0x559b311c1890 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.114:47218/0 (socket says 172.21.15.114:47218) 2024-04-23T10:30:39.581 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:30:39.574+0000 7fe609a2c640 1 -- 172.21.15.114:0/1034328 learned_addr learned my addr 172.21.15.114:0/1034328 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-04-23T10:30:39.581 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:30:39.574+0000 7fe60a22d640 1 --2- 172.21.15.114:0/1034328 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x559b311b5500 0x559b311c45a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T10:30:39.581 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:30:39.574+0000 7fe60aa2e640 1 --2- 172.21.15.114:0/1034328 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x559b311c4c10 0x559b311c1e10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T10:30:39.581 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:30:39.575+0000 7fe609a2c640 1 -- 172.21.15.114:0/1034328 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x559b311b5500 msgr2=0x559b311c45a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T10:30:39.581 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:30:39.575+0000 7fe609a2c640 1 --2- 172.21.15.114:0/1034328 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x559b311b5500 0x559b311c45a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:30:39.581 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:30:39.575+0000 7fe609a2c640 1 -- 172.21.15.114:0/1034328 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x559b311c4c10 msgr2=0x559b311c1e10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T10:30:39.581 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:30:39.575+0000 7fe609a2c640 1 --2- 172.21.15.114:0/1034328 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x559b311c4c10 0x559b311c1e10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:30:39.581 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:30:39.575+0000 7fe609a2c640 1 -- 172.21.15.114:0/1034328 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({osdmap=0}) -- 0x559b30ef9620 con 0x559b311b5e20 2024-04-23T10:30:39.581 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:30:39.575+0000 7fe609a2c640 1 --2- 172.21.15.114:0/1034328 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x559b311b5e20 0x559b311c1890 secure :-1 s=READY pgs=633 cs=0 l=1 rev1=1 crypto rx=0x7fe5f4009350 tx=0x7fe5f40051d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=209be40a32c1ada3 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T10:30:39.582 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:30:39.575+0000 7fe5edffb640 10 client.0 ms_handle_connect on v2:172.21.15.114:3300/0 2024-04-23T10:30:39.582 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:30:39.575+0000 7fe5edffb640 1 -- 172.21.15.114:0/1034328 <== mon.0 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe5f4027030 con 0x559b311b5e20 2024-04-23T10:30:39.582 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:30:39.575+0000 7fe5edffb640 1 -- 172.21.15.114:0/1034328 <== mon.0 v2:172.21.15.114:3300/0 2 ==== osd_map(92..92 src has 1..92) ==== 8585+0+0 (secure 0 0 0) 0x7fe5f4016040 con 0x559b311b5e20 2024-04-23T10:30:39.582 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:30:39.575+0000 7fe60f1ae740 1 -- 172.21.15.114:0/1034328 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fe5f40047a0 con 0x559b311b5e20 2024-04-23T10:30:39.582 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:30:39.575+0000 7fe5edffb640 1 -- 172.21.15.114:0/1034328 <== mon.0 v2:172.21.15.114:3300/0 3 ==== mon_get_version_reply(handle=1 version=25) ==== 24+0+0 (secure 0 0 0) 0x7fe5f4031050 con 0x559b311b5e20 2024-04-23T10:30:39.582 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:30:39.575+0000 7fe60f1ae740 10 client.16068 fetch_fsmap learned FSMap version 25 2024-04-23T10:30:39.582 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:30:39.575+0000 7fe60f1ae740 1 -- 172.21.15.114:0/1034328 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({config=0+,fsmap=25,monmap=4+}) -- 0x559b311d3840 con 0x559b311b5e20 2024-04-23T10:30:39.582 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:30:39.575+0000 7fe5edffb640 1 -- 172.21.15.114:0/1034328 <== mon.0 v2:172.21.15.114:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fe5f40047a0 con 0x559b311b5e20 2024-04-23T10:30:39.582 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:30:39.575+0000 7fe5edffb640 1 -- 172.21.15.114:0/1034328 <== mon.0 v2:172.21.15.114:3300/0 5 ==== fsmap(e 25) ==== 8342+0+0 (secure 0 0 0) 0x7fe5f4020030 con 0x559b311b5e20 2024-04-23T10:30:39.582 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:30:39.575+0000 7fe60f1ae740 10 client.16068 fetch_fsmap finished waiting for FSMap version 25 2024-04-23T10:30:39.582 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:30:39.575+0000 7fe60f1ae740 10 client.16068 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-04-23T10:30:39.582 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:30:39.575+0000 7fe60f1ae740 20 client.16068 populate_metadata read hostname 'smithi114' 2024-04-23T10:30:39.582 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:30:39.575+0000 7fe60f1ae740 1 --2- 172.21.15.114:0/1034328 >> [v2:172.21.15.119:6838/937297340,v1:172.21.15.119:6839/937297340] conn(0x559b311d4d00 0x559b311d70f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T10:30:39.582 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:30:39.575+0000 7fe60f1ae740 4 client.16068 mds_command: new command op to 24545 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-23T10:30:39.582 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:30:39.575+0000 7fe60f1ae740 1 -- 172.21.15.114:0/1034328 --> [v2:172.21.15.119:6838/937297340,v1:172.21.15.119:6839/937297340] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x559b31128390 con 0x559b311d4d00 2024-04-23T10:30:39.583 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:30:39.576+0000 7fe60a22d640 1 --2- 172.21.15.114:0/1034328 >> [v2:172.21.15.119:6838/937297340,v1:172.21.15.119:6839/937297340] conn(0x559b311d4d00 0x559b311d70f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T10:30:39.583 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:30:39.576+0000 7fe60a22d640 1 --2- 172.21.15.114:0/1034328 >> [v2:172.21.15.119:6838/937297340,v1:172.21.15.119:6839/937297340] conn(0x559b311d4d00 0x559b311d70f0 crc :-1 s=READY pgs=90 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=9f677e558992f5ea server_cookie=819af546c2504a5d in_seq=0 out_seq=0 2024-04-23T10:30:39.583 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:30:39.576+0000 7fe5edffb640 10 client.16068 ms_handle_connect on v2:172.21.15.119:6838/937297340 2024-04-23T10:30:39.583 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:30:39.577+0000 7fe5edffb640 1 -- 172.21.15.114:0/1034328 <== mds.0 v2:172.21.15.119:6838/937297340 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x559b31128390 con 0x559b311d4d00 2024-04-23T10:30:39.583 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:30:39.577+0000 7fe5edffb640 10 client.16068 handle_command_reply: tid=0 2024-04-23T10:30:39.583 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:30:39.577+0000 7fe60f1ae740 1 client.16068 shutdown 2024-04-23T10:30:39.583 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:30:39.577+0000 7fe60f1ae740 1 -- 172.21.15.114:0/1034328 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x559b311b5e20 msgr2=0x559b311c1890 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T10:30:39.583 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:30:39.577+0000 7fe60f1ae740 1 --2- 172.21.15.114:0/1034328 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x559b311b5e20 0x559b311c1890 secure :-1 s=READY pgs=633 cs=0 l=1 rev1=1 crypto rx=0x7fe5f4009350 tx=0x7fe5f40051d0 comp rx=0 tx=0).stop 2024-04-23T10:30:39.583 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:30:39.577+0000 7fe60f1ae740 1 -- 172.21.15.114:0/1034328 shutdown_connections 2024-04-23T10:30:39.583 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:30:39.577+0000 7fe60f1ae740 1 --2- 172.21.15.114:0/1034328 >> [v2:172.21.15.119:6838/937297340,v1:172.21.15.119:6839/937297340] conn(0x559b311d4d00 0x559b311d70f0 crc :-1 s=READY pgs=90 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:30:39.583 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:30:39.577+0000 7fe60f1ae740 1 --2- 172.21.15.114:0/1034328 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x559b311c4c10 0x559b311c1e10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:30:39.583 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:30:39.577+0000 7fe60f1ae740 1 --2- 172.21.15.114:0/1034328 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x559b311b5e20 0x559b311c1890 unknown :-1 s=CLOSED pgs=633 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:30:39.583 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:30:39.577+0000 7fe60f1ae740 1 --2- 172.21.15.114:0/1034328 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x559b311b5500 0x559b311c45a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:30:39.583 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:30:39.577+0000 7fe5edffb640 0 client.16068 ms_handle_reset on v2:172.21.15.119:6838/937297340 2024-04-23T10:30:39.584 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:30:39.577+0000 7fe60f1ae740 1 -- 172.21.15.114:0/1034328 >> 172.21.15.114:0/1034328 conn(0x559b311b1b30 msgr2=0x559b311b7940 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T10:30:39.584 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:30:39.577+0000 7fe60f1ae740 1 -- 172.21.15.114:0/1034328 shutdown_connections 2024-04-23T10:30:39.584 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:30:39.577+0000 7fe60f1ae740 1 -- 172.21.15.114:0/1034328 wait complete. 2024-04-23T10:30:39.584 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:30:39.577+0000 7fe60f1ae740 20 client.16068 trim_cache size 0 max 16384 2024-04-23T10:30:39.595 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:30:39.592+0000 7fe60f1ae740 1 Processor -- start 2024-04-23T10:30:39.595 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:30:39.592+0000 7fe60f1ae740 1 -- start start 2024-04-23T10:30:39.595 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:30:39.592+0000 7fe60f1ae740 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x559b311b5500 0x559b311b0b30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T10:30:39.595 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:30:39.592+0000 7fe60f1ae740 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x559b311b5e20 0x559b311c40d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T10:30:39.595 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:30:39.592+0000 7fe60f1ae740 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x559b311c4c10 0x559b311dc8a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T10:30:39.595 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:30:39.592+0000 7fe60f1ae740 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x7fe6041ae2e0 con 0x559b311c4c10 2024-04-23T10:30:39.595 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:30:39.592+0000 7fe60f1ae740 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 -- 0x7fe6041abb80 con 0x559b311b5e20 2024-04-23T10:30:39.595 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:30:39.592+0000 7fe60f1ae740 1 -- --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_getmap magic: 0 -- 0x7fe6041a99d0 con 0x559b311b5500 2024-04-23T10:30:39.596 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:30:39.592+0000 7fe60aa2e640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x559b311c4c10 0x559b311dc8a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T10:30:39.596 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:30:39.592+0000 7fe60aa2e640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x559b311c4c10 0x559b311dc8a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.114:47230/0 (socket says 172.21.15.114:47230) 2024-04-23T10:30:39.596 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:30:39.592+0000 7fe60aa2e640 1 -- 172.21.15.114:0/1973164146 learned_addr learned my addr 172.21.15.114:0/1973164146 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-04-23T10:30:39.596 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:30:39.592+0000 7fe609a2c640 1 --2- 172.21.15.114:0/1973164146 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x559b311b5e20 0x559b311c40d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T10:30:39.596 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:30:39.592+0000 7fe60a22d640 1 --2- 172.21.15.114:0/1973164146 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x559b311b5500 0x559b311b0b30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T10:30:39.596 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:30:39.592+0000 7fe60aa2e640 1 -- 172.21.15.114:0/1973164146 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x559b311b5500 msgr2=0x559b311b0b30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T10:30:39.596 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:30:39.592+0000 7fe60aa2e640 1 --2- 172.21.15.114:0/1973164146 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x559b311b5500 0x559b311b0b30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:30:39.596 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:30:39.592+0000 7fe60aa2e640 1 -- 172.21.15.114:0/1973164146 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x559b311b5e20 msgr2=0x559b311c40d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T10:30:39.596 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:30:39.592+0000 7fe60aa2e640 1 --2- 172.21.15.114:0/1973164146 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x559b311b5e20 0x559b311c40d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:30:39.596 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:30:39.592+0000 7fe60aa2e640 1 -- 172.21.15.114:0/1973164146 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x559b311c7000 con 0x559b311c4c10 2024-04-23T10:30:39.596 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:30:39.592+0000 7fe60a22d640 1 --2- 172.21.15.114:0/1973164146 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x559b311b5500 0x559b311b0b30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-23T10:30:39.596 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:30:39.593+0000 7fe60aa2e640 1 --2- 172.21.15.114:0/1973164146 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x559b311c4c10 0x559b311dc8a0 secure :-1 s=READY pgs=634 cs=0 l=1 rev1=1 crypto rx=0x7fe5fc012400 tx=0x7fe5fc009500 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e66d96b52fcfedd1 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T10:30:39.596 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:30:39.593+0000 7fe5effff640 1 -- 172.21.15.114:0/1973164146 <== mon.0 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe5fc004050 con 0x559b311c4c10 2024-04-23T10:30:39.596 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:30:39.593+0000 7fe5effff640 1 -- 172.21.15.114:0/1973164146 <== mon.0 v2:172.21.15.114:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fe5fc025070 con 0x559b311c4c10 2024-04-23T10:30:39.597 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:30:39.593+0000 7fe5effff640 1 -- 172.21.15.114:0/1973164146 <== mon.0 v2:172.21.15.114:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe5fc010a00 con 0x559b311c4c10 2024-04-23T10:30:39.597 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:30:39.593+0000 7fe60f1ae740 1 -- 172.21.15.114:0/1973164146 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x559b311c4c10 msgr2=0x559b311dc8a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T10:30:39.597 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:30:39.593+0000 7fe60f1ae740 1 --2- 172.21.15.114:0/1973164146 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x559b311c4c10 0x559b311dc8a0 secure :-1 s=READY pgs=634 cs=0 l=1 rev1=1 crypto rx=0x7fe5fc012400 tx=0x7fe5fc009500 comp rx=0 tx=0).stop 2024-04-23T10:30:39.597 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:30:39.593+0000 7fe60f1ae740 1 -- 172.21.15.114:0/1973164146 shutdown_connections 2024-04-23T10:30:39.597 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:30:39.593+0000 7fe60f1ae740 1 --2- 172.21.15.114:0/1973164146 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x559b311c4c10 0x559b311dc8a0 unknown :-1 s=CLOSED pgs=634 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:30:39.597 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:30:39.593+0000 7fe60f1ae740 1 --2- 172.21.15.114:0/1973164146 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x559b311b5e20 0x559b311c40d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:30:39.597 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:30:39.593+0000 7fe60f1ae740 1 --2- 172.21.15.114:0/1973164146 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x559b311b5500 0x559b311b0b30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:30:39.597 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:30:39.593+0000 7fe60f1ae740 1 -- 172.21.15.114:0/1973164146 >> 172.21.15.114:0/1973164146 conn(0x559b311b1b30 msgr2=0x559b311b0420 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T10:30:39.597 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:30:39.593+0000 7fe60f1ae740 1 -- 172.21.15.114:0/1973164146 shutdown_connections 2024-04-23T10:30:39.597 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:30:39.593+0000 7fe60f1ae740 1 -- 172.21.15.114:0/1973164146 wait complete. 2024-04-23T10:30:39.597 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:30:39.594+0000 7fe60f1ae740 1 -- start start 2024-04-23T10:30:39.597 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:30:39.594+0000 7fe60f1ae740 1 Processor -- start 2024-04-23T10:30:39.597 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:30:39.594+0000 7fe60f1ae740 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x559b311b5500 0x559b311c8940 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T10:30:39.597 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:30:39.594+0000 7fe60f1ae740 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x559b311b5e20 0x559b311c8e80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T10:30:39.597 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:30:39.594+0000 7fe60f1ae740 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x559b311c4c10 0x559b311c93c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T10:30:39.598 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:30:39.594+0000 7fe60f1ae740 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x7fe6041aa040 con 0x559b311c4c10 2024-04-23T10:30:39.598 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:30:39.594+0000 7fe60f1ae740 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 -- 0x7fe6041ab4b0 con 0x559b311b5e20 2024-04-23T10:30:39.598 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:30:39.594+0000 7fe60f1ae740 1 -- --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_getmap magic: 0 -- 0x7fe6041aa710 con 0x559b311b5500 2024-04-23T10:30:39.598 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:30:39.594+0000 7fe60f1ae740 10 client.0 _ll_register_callbacks cb 0x7fe5f4004d10 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-23T10:30:39.598 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:30:39.594+0000 7fe60aa2e640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x559b311c4c10 0x559b311c93c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T10:30:39.598 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:30:39.594+0000 7fe60aa2e640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x559b311c4c10 0x559b311c93c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.114:47234/0 (socket says 172.21.15.114:47234) 2024-04-23T10:30:39.598 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:30:39.594+0000 7fe60a22d640 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x559b311b5500 0x559b311c8940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T10:30:39.598 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:30:39.594+0000 7fe60aa2e640 1 -- 172.21.15.114:0/1655295310 learned_addr learned my addr 172.21.15.114:0/1655295310 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-04-23T10:30:39.598 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:30:39.594+0000 7fe609a2c640 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x559b311b5e20 0x559b311c8e80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T10:30:39.598 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:30:39.594+0000 7fe60aa2e640 1 -- 172.21.15.114:0/1655295310 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x559b311b5500 msgr2=0x559b311c8940 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T10:30:39.598 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:30:39.594+0000 7fe60aa2e640 1 --2- 172.21.15.114:0/1655295310 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x559b311b5500 0x559b311c8940 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:30:39.598 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:30:39.594+0000 7fe60aa2e640 1 -- 172.21.15.114:0/1655295310 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x559b311b5e20 msgr2=0x559b311c8e80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T10:30:39.598 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:30:39.594+0000 7fe60aa2e640 1 --2- 172.21.15.114:0/1655295310 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x559b311b5e20 0x559b311c8e80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:30:39.599 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:30:39.594+0000 7fe60aa2e640 1 -- 172.21.15.114:0/1655295310 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({osdmap=0}) -- 0x559b3123a3b0 con 0x559b311c4c10 2024-04-23T10:30:39.599 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:30:39.595+0000 7fe60aa2e640 1 --2- 172.21.15.114:0/1655295310 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x559b311c4c10 0x559b311c93c0 secure :-1 s=READY pgs=635 cs=0 l=1 rev1=1 crypto rx=0x7fe5fc00f3c0 tx=0x7fe5fc010540 comp rx=0 tx=0).ready entity=mon.0 client_cookie=534a8edb3aa8f9fc server_cookie=0 in_seq=0 out_seq=0 2024-04-23T10:30:39.599 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:30:39.595+0000 7fe5edffb640 10 client.0 ms_handle_connect on v2:172.21.15.114:3300/0 2024-04-23T10:30:39.599 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:30:39.595+0000 7fe5edffb640 1 -- 172.21.15.114:0/1655295310 <== mon.0 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe5fc010960 con 0x559b311c4c10 2024-04-23T10:30:39.599 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:30:39.595+0000 7fe5edffb640 1 -- 172.21.15.114:0/1655295310 <== mon.0 v2:172.21.15.114:3300/0 2 ==== osd_map(92..92 src has 1..92) ==== 8585+0+0 (secure 0 0 0) 0x7fe5fc010b00 con 0x559b311c4c10 2024-04-23T10:30:39.599 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:30:39.595+0000 7fe60f1ae740 1 -- 172.21.15.114:0/1655295310 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fe5fc025070 con 0x559b311c4c10 2024-04-23T10:30:39.599 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:30:39.595+0000 7fe5edffb640 1 -- 172.21.15.114:0/1655295310 <== mon.0 v2:172.21.15.114:3300/0 3 ==== mon_get_version_reply(handle=1 version=25) ==== 24+0+0 (secure 0 0 0) 0x7fe5fc012120 con 0x559b311c4c10 2024-04-23T10:30:39.599 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:30:39.595+0000 7fe60f1ae740 10 client.16074 fetch_fsmap learned FSMap version 25 2024-04-23T10:30:39.599 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:30:39.595+0000 7fe60f1ae740 1 -- 172.21.15.114:0/1655295310 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({config=0+,fsmap=25,monmap=4+}) -- 0x559b3123b630 con 0x559b311c4c10 2024-04-23T10:30:39.599 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:30:39.595+0000 7fe5edffb640 1 -- 172.21.15.114:0/1655295310 <== mon.0 v2:172.21.15.114:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fe5fc025070 con 0x559b311c4c10 2024-04-23T10:30:39.599 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:30:39.595+0000 7fe5edffb640 1 -- 172.21.15.114:0/1655295310 <== mon.0 v2:172.21.15.114:3300/0 5 ==== fsmap(e 25) ==== 8342+0+0 (secure 0 0 0) 0x7fe5fc005040 con 0x559b311c4c10 2024-04-23T10:30:39.599 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:30:39.595+0000 7fe60f1ae740 10 client.16074 fetch_fsmap finished waiting for FSMap version 25 2024-04-23T10:30:39.599 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:30:39.595+0000 7fe60f1ae740 10 client.16074 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-04-23T10:30:39.599 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:30:39.595+0000 7fe60f1ae740 20 client.16074 populate_metadata read hostname 'smithi114' 2024-04-23T10:30:39.599 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:30:39.595+0000 7fe60f1ae740 1 --2- 172.21.15.114:0/1655295310 >> [v2:172.21.15.119:6838/937297340,v1:172.21.15.119:6839/937297340] conn(0x559b3123bd50 0x559b3123e170 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T10:30:39.600 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:30:39.595+0000 7fe60f1ae740 4 client.16074 mds_command: new command op to 24545 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-04-23T10:30:39.600 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:30:39.595+0000 7fe60f1ae740 1 -- 172.21.15.114:0/1655295310 --> [v2:172.21.15.119:6838/937297340,v1:172.21.15.119:6839/937297340] -- command(tid 0: {"prefix": "flush journal"}) -- 0x559b31122200 con 0x559b3123bd50 2024-04-23T10:30:39.600 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:30:39.595+0000 7fe60a22d640 1 --2- 172.21.15.114:0/1655295310 >> [v2:172.21.15.119:6838/937297340,v1:172.21.15.119:6839/937297340] conn(0x559b3123bd50 0x559b3123e170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T10:30:39.600 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:30:39.596+0000 7fe5edffb640 10 client.16074 ms_handle_connect on v2:172.21.15.119:6838/937297340 2024-04-23T10:30:39.600 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:30:39.596+0000 7fe60a22d640 1 --2- 172.21.15.114:0/1655295310 >> [v2:172.21.15.119:6838/937297340,v1:172.21.15.119:6839/937297340] conn(0x559b3123bd50 0x559b3123e170 crc :-1 s=READY pgs=91 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=b0eee05361da5551 server_cookie=b2fbd63b9b6f247b in_seq=0 out_seq=0 2024-04-23T10:30:39.655 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:30:39.656+0000 7fe5edffb640 1 -- 172.21.15.114:0/1655295310 <== mds.0 v2:172.21.15.119:6838/937297340 1 ==== command_reply(tid 0: 0 ) ==== 8+0+44 (crc 0 0 0) 0x559b31122200 con 0x559b3123bd50 2024-04-23T10:30:39.656 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:30:39.656+0000 7fe5edffb640 10 client.16074 handle_command_reply: tid=0 2024-04-23T10:30:39.656 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:30:39.656+0000 7fe60f1ae740 1 client.16074 shutdown 2024-04-23T10:30:39.656 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:30:39.656+0000 7fe60f1ae740 1 -- 172.21.15.114:0/1655295310 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x559b311c4c10 msgr2=0x559b311c93c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T10:30:39.656 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:30:39.656+0000 7fe60f1ae740 1 --2- 172.21.15.114:0/1655295310 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x559b311c4c10 0x559b311c93c0 secure :-1 s=READY pgs=635 cs=0 l=1 rev1=1 crypto rx=0x7fe5fc00f3c0 tx=0x7fe5fc010540 comp rx=0 tx=0).stop 2024-04-23T10:30:39.656 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:30:39.656+0000 7fe60f1ae740 1 -- 172.21.15.114:0/1655295310 shutdown_connections 2024-04-23T10:30:39.656 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:30:39.656+0000 7fe60f1ae740 1 --2- 172.21.15.114:0/1655295310 >> [v2:172.21.15.119:6838/937297340,v1:172.21.15.119:6839/937297340] conn(0x559b3123bd50 0x559b3123e170 crc :-1 s=READY pgs=91 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:30:39.656 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:30:39.656+0000 7fe60f1ae740 1 --2- 172.21.15.114:0/1655295310 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x559b311c4c10 0x559b311c93c0 unknown :-1 s=CLOSED pgs=635 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:30:39.656 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:30:39.656+0000 7fe5edffb640 0 client.16074 ms_handle_reset on v2:172.21.15.119:6838/937297340 2024-04-23T10:30:39.656 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:30:39.656+0000 7fe60f1ae740 1 --2- 172.21.15.114:0/1655295310 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x559b311b5e20 0x559b311c8e80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:30:39.656 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:30:39.656+0000 7fe60f1ae740 1 --2- 172.21.15.114:0/1655295310 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x559b311b5500 0x559b311c8940 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:30:39.656 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:30:39.656+0000 7fe60f1ae740 1 -- 172.21.15.114:0/1655295310 >> 172.21.15.114:0/1655295310 conn(0x559b311b1b30 msgr2=0x559b311b0cb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T10:30:39.656 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:30:39.656+0000 7fe60f1ae740 1 -- 172.21.15.114:0/1655295310 shutdown_connections 2024-04-23T10:30:39.656 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:30:39.656+0000 7fe60f1ae740 1 -- 172.21.15.114:0/1655295310 wait complete. 2024-04-23T10:30:39.656 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:30:39.657+0000 7fe60f1ae740 20 client.16074 trim_cache size 0 max 16384 2024-04-23T10:30:39.657 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stdout:{ 2024-04-23T10:30:39.657 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stdout: "message": "", 2024-04-23T10:30:39.657 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stdout: "return_code": 0 2024-04-23T10:30:39.657 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stdout:} 2024-04-23T10:30:39.658 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:30:39.657+0000 7fe60c491640 1 -- 172.21.15.114:0/168447071 >> [v2:172.21.15.114:6800/1395693466,v1:172.21.15.114:6801/1395693466] conn(0x7fe5f00a3dc0 msgr2=0x7fe5f00a6280 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T10:30:39.658 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:30:39.657+0000 7fe60c491640 1 --2- 172.21.15.114:0/168447071 >> [v2:172.21.15.114:6800/1395693466,v1:172.21.15.114:6801/1395693466] conn(0x7fe5f00a3dc0 0x7fe5f00a6280 secure :-1 s=READY pgs=95 cs=0 l=1 rev1=1 crypto rx=0x7fe5fc005fd0 tx=0x7fe5fc013ae0 comp rx=0 tx=0).stop 2024-04-23T10:30:39.658 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:30:39.657+0000 7fe60c491640 1 -- 172.21.15.114:0/168447071 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fe6041021d0 msgr2=0x7fe6041986a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T10:30:39.658 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:30:39.657+0000 7fe60c491640 1 --2- 172.21.15.114:0/168447071 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fe6041021d0 0x7fe6041986a0 secure :-1 s=READY pgs=632 cs=0 l=1 rev1=1 crypto rx=0x7fe600005940 tx=0x7fe6000040c0 comp rx=0 tx=0).stop 2024-04-23T10:30:39.658 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:30:39.657+0000 7fe60c491640 1 -- 172.21.15.114:0/168447071 shutdown_connections 2024-04-23T10:30:39.658 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:30:39.657+0000 7fe60c491640 1 --2- 172.21.15.114:0/168447071 >> [v2:172.21.15.114:6800/1395693466,v1:172.21.15.114:6801/1395693466] conn(0x7fe5f00a3dc0 0x7fe5f00a6280 unknown :-1 s=CLOSED pgs=95 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:30:39.658 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:30:39.657+0000 7fe60c491640 1 --2- 172.21.15.114:0/168447071 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fe60419cf60 0x7fe60419d410 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:30:39.658 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:30:39.657+0000 7fe60c491640 1 --2- 172.21.15.114:0/168447071 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fe6041072a0 0x7fe604198be0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:30:39.658 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:30:39.657+0000 7fe60c491640 1 --2- 172.21.15.114:0/168447071 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fe6041021d0 0x7fe6041986a0 unknown :-1 s=CLOSED pgs=632 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:30:39.658 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:30:39.657+0000 7fe60c491640 1 -- 172.21.15.114:0/168447071 >> 172.21.15.114:0/168447071 conn(0x7fe6040793e0 msgr2=0x7fe6040ffda0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T10:30:39.659 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:30:39.657+0000 7fe60c491640 1 -- 172.21.15.114:0/168447071 shutdown_connections 2024-04-23T10:30:39.659 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:30:39.657+0000 7fe60c491640 1 -- 172.21.15.114:0/168447071 wait complete. 2024-04-23T10:30:39.916 INFO:tasks.workunit.client.0.smithi114.stdout: 1 245019 7.55 MB/sec execute 48 sec latency 129.191 ms 2024-04-23T10:30:40.327 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 23 10:30:40 smithi204 ceph-mon[32649]: pgmap v124: 97 pgs: 97 active+clean; 87 MiB data, 1.0 GiB used, 1.0 TiB / 1.0 TiB avail; 24 MiB/s rd, 2.6 MiB/s wr, 337 op/s 2024-04-23T10:30:40.492 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 23 10:30:40 smithi119 ceph-mon[32608]: pgmap v124: 97 pgs: 97 active+clean; 87 MiB data, 1.0 GiB used, 1.0 TiB / 1.0 TiB avail; 24 MiB/s rd, 2.6 MiB/s wr, 337 op/s 2024-04-23T10:30:40.503 INFO:journalctl@ceph.mon.a.smithi114.stdout:Apr 23 10:30:40 smithi114 ceph-mon[29255]: pgmap v124: 97 pgs: 97 active+clean; 87 MiB data, 1.0 GiB used, 1.0 TiB / 1.0 TiB avail; 24 MiB/s rd, 2.6 MiB/s wr, 337 op/s 2024-04-23T10:30:40.916 INFO:tasks.workunit.client.0.smithi114.stdout: 1 246292 7.51 MB/sec execute 49 sec latency 40.848 ms 2024-04-23T10:30:41.916 INFO:tasks.workunit.client.0.smithi114.stdout: 1 248546 7.63 MB/sec execute 50 sec latency 75.241 ms 2024-04-23T10:30:42.327 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 23 10:30:42 smithi204 ceph-mon[32649]: pgmap v125: 97 pgs: 97 active+clean; 87 MiB data, 1.0 GiB used, 1.0 TiB / 1.0 TiB avail; 24 MiB/s rd, 2.6 MiB/s wr, 337 op/s 2024-04-23T10:30:42.491 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 23 10:30:42 smithi119 ceph-mon[32608]: pgmap v125: 97 pgs: 97 active+clean; 87 MiB data, 1.0 GiB used, 1.0 TiB / 1.0 TiB avail; 24 MiB/s rd, 2.6 MiB/s wr, 337 op/s 2024-04-23T10:30:42.503 INFO:journalctl@ceph.mon.a.smithi114.stdout:Apr 23 10:30:42 smithi114 ceph-mon[29255]: pgmap v125: 97 pgs: 97 active+clean; 87 MiB data, 1.0 GiB used, 1.0 TiB / 1.0 TiB avail; 24 MiB/s rd, 2.6 MiB/s wr, 337 op/s 2024-04-23T10:30:42.916 INFO:tasks.workunit.client.0.smithi114.stdout: 1 249988 7.59 MB/sec execute 51 sec latency 12.996 ms 2024-04-23T10:30:43.916 INFO:tasks.workunit.client.0.smithi114.stdout: 1 252247 7.71 MB/sec execute 52 sec latency 9.459 ms 2024-04-23T10:30:44.327 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 23 10:30:44 smithi204 ceph-mon[32649]: pgmap v126: 97 pgs: 97 active+clean; 87 MiB data, 1.0 GiB used, 1.0 TiB / 1.0 TiB avail; 45 MiB/s rd, 4.4 MiB/s wr, 548 op/s 2024-04-23T10:30:44.491 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 23 10:30:44 smithi119 ceph-mon[32608]: pgmap v126: 97 pgs: 97 active+clean; 87 MiB data, 1.0 GiB used, 1.0 TiB / 1.0 TiB avail; 45 MiB/s rd, 4.4 MiB/s wr, 548 op/s 2024-04-23T10:30:44.503 INFO:journalctl@ceph.mon.a.smithi114.stdout:Apr 23 10:30:44 smithi114 ceph-mon[29255]: pgmap v126: 97 pgs: 97 active+clean; 87 MiB data, 1.0 GiB used, 1.0 TiB / 1.0 TiB avail; 45 MiB/s rd, 4.4 MiB/s wr, 548 op/s 2024-04-23T10:30:44.916 INFO:tasks.workunit.client.0.smithi114.stdout: 1 254021 7.67 MB/sec execute 53 sec latency 14.715 ms 2024-04-23T10:30:45.916 INFO:tasks.workunit.client.0.smithi114.stdout: 1 255831 7.78 MB/sec execute 54 sec latency 11.256 ms 2024-04-23T10:30:46.328 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 23 10:30:46 smithi204 ceph-mon[32649]: pgmap v127: 97 pgs: 97 active+clean; 87 MiB data, 1.0 GiB used, 1.0 TiB / 1.0 TiB avail; 28 MiB/s rd, 3.3 MiB/s wr, 356 op/s 2024-04-23T10:30:46.328 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 23 10:30:46 smithi204 ceph-mon[32649]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=e=up:active} 3 up:standby-replay 6 up:standby 2024-04-23T10:30:46.491 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 23 10:30:46 smithi119 ceph-mon[32608]: pgmap v127: 97 pgs: 97 active+clean; 87 MiB data, 1.0 GiB used, 1.0 TiB / 1.0 TiB avail; 28 MiB/s rd, 3.3 MiB/s wr, 356 op/s 2024-04-23T10:30:46.491 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 23 10:30:46 smithi119 ceph-mon[32608]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=e=up:active} 3 up:standby-replay 6 up:standby 2024-04-23T10:30:46.504 INFO:journalctl@ceph.mon.a.smithi114.stdout:Apr 23 10:30:46 smithi114 ceph-mon[29255]: pgmap v127: 97 pgs: 97 active+clean; 87 MiB data, 1.0 GiB used, 1.0 TiB / 1.0 TiB avail; 28 MiB/s rd, 3.3 MiB/s wr, 356 op/s 2024-04-23T10:30:46.504 INFO:journalctl@ceph.mon.a.smithi114.stdout:Apr 23 10:30:46 smithi114 ceph-mon[29255]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=e=up:active} 3 up:standby-replay 6 up:standby 2024-04-23T10:30:46.916 INFO:tasks.workunit.client.0.smithi114.stdout: 1 257684 7.74 MB/sec execute 55 sec latency 39.637 ms 2024-04-23T10:30:47.916 INFO:tasks.workunit.client.0.smithi114.stdout: 1 259415 7.84 MB/sec execute 56 sec latency 18.992 ms 2024-04-23T10:30:48.305 INFO:journalctl@ceph.mon.a.smithi114.stdout:Apr 23 10:30:48 smithi114 ceph-mon[29255]: pgmap v128: 97 pgs: 97 active+clean; 90 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 38 MiB/s rd, 5.0 MiB/s wr, 533 op/s 2024-04-23T10:30:48.327 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 23 10:30:48 smithi204 ceph-mon[32649]: pgmap v128: 97 pgs: 97 active+clean; 90 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 38 MiB/s rd, 5.0 MiB/s wr, 533 op/s 2024-04-23T10:30:48.491 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 23 10:30:48 smithi119 ceph-mon[32608]: pgmap v128: 97 pgs: 97 active+clean; 90 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 38 MiB/s rd, 5.0 MiB/s wr, 533 op/s 2024-04-23T10:30:48.916 INFO:tasks.workunit.client.0.smithi114.stdout: 1 261410 7.83 MB/sec execute 57 sec latency 52.471 ms 2024-04-23T10:30:49.328 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 23 10:30:49 smithi204 ceph-mon[32649]: from='mgr.15291 172.21.15.114:0/1095789187' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-23T10:30:49.492 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 23 10:30:49 smithi119 ceph-mon[32608]: from='mgr.15291 172.21.15.114:0/1095789187' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-23T10:30:49.503 INFO:journalctl@ceph.mon.a.smithi114.stdout:Apr 23 10:30:49 smithi114 ceph-mon[29255]: from='mgr.15291 172.21.15.114:0/1095789187' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-23T10:30:49.916 INFO:tasks.workunit.client.0.smithi114.stdout: 1 263050 7.90 MB/sec execute 58 sec latency 16.887 ms 2024-04-23T10:30:50.328 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 23 10:30:50 smithi204 ceph-mon[32649]: pgmap v129: 97 pgs: 97 active+clean; 90 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 31 MiB/s rd, 3.5 MiB/s wr, 387 op/s 2024-04-23T10:30:50.328 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 23 10:30:50 smithi204 ceph-mon[32649]: from='mgr.15291 172.21.15.114:0/1095789187' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-23T10:30:50.328 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 23 10:30:50 smithi204 ceph-mon[32649]: from='mgr.15291 172.21.15.114:0/1095789187' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-23T10:30:50.328 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 23 10:30:50 smithi204 ceph-mon[32649]: from='mgr.15291 172.21.15.114:0/1095789187' entity='mgr.x' 2024-04-23T10:30:50.491 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 23 10:30:50 smithi119 ceph-mon[32608]: pgmap v129: 97 pgs: 97 active+clean; 90 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 31 MiB/s rd, 3.5 MiB/s wr, 387 op/s 2024-04-23T10:30:50.492 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 23 10:30:50 smithi119 ceph-mon[32608]: from='mgr.15291 172.21.15.114:0/1095789187' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-23T10:30:50.492 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 23 10:30:50 smithi119 ceph-mon[32608]: from='mgr.15291 172.21.15.114:0/1095789187' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-23T10:30:50.492 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 23 10:30:50 smithi119 ceph-mon[32608]: from='mgr.15291 172.21.15.114:0/1095789187' entity='mgr.x' 2024-04-23T10:30:50.504 INFO:journalctl@ceph.mon.a.smithi114.stdout:Apr 23 10:30:50 smithi114 ceph-mon[29255]: pgmap v129: 97 pgs: 97 active+clean; 90 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 31 MiB/s rd, 3.5 MiB/s wr, 387 op/s 2024-04-23T10:30:50.504 INFO:journalctl@ceph.mon.a.smithi114.stdout:Apr 23 10:30:50 smithi114 ceph-mon[29255]: from='mgr.15291 172.21.15.114:0/1095789187' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-23T10:30:50.504 INFO:journalctl@ceph.mon.a.smithi114.stdout:Apr 23 10:30:50 smithi114 ceph-mon[29255]: from='mgr.15291 172.21.15.114:0/1095789187' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-23T10:30:50.504 INFO:journalctl@ceph.mon.a.smithi114.stdout:Apr 23 10:30:50 smithi114 ceph-mon[29255]: from='mgr.15291 172.21.15.114:0/1095789187' entity='mgr.x' 2024-04-23T10:30:50.917 INFO:tasks.workunit.client.0.smithi114.stdout: 1 265167 7.92 MB/sec execute 59 sec latency 14.270 ms 2024-04-23T10:30:51.917 INFO:tasks.workunit.client.0.smithi114.stdout: 1 266453 7.96 MB/sec execute 60 sec latency 94.017 ms 2024-04-23T10:30:52.328 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 23 10:30:52 smithi204 ceph-mon[32649]: pgmap v130: 97 pgs: 97 active+clean; 90 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 31 MiB/s rd, 3.5 MiB/s wr, 387 op/s 2024-04-23T10:30:52.492 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 23 10:30:52 smithi119 ceph-mon[32608]: pgmap v130: 97 pgs: 97 active+clean; 90 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 31 MiB/s rd, 3.5 MiB/s wr, 387 op/s 2024-04-23T10:30:52.504 INFO:journalctl@ceph.mon.a.smithi114.stdout:Apr 23 10:30:52 smithi114 ceph-mon[29255]: pgmap v130: 97 pgs: 97 active+clean; 90 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 31 MiB/s rd, 3.5 MiB/s wr, 387 op/s 2024-04-23T10:30:52.753 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:30:52.751+0000 7f5045631640 1 -- 172.21.15.114:0/3435279358 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f5040106f30 msgr2=0x7f5040107310 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T10:30:52.753 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:30:52.751+0000 7f5045631640 1 --2- 172.21.15.114:0/3435279358 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f5040106f30 0x7f5040107310 secure :-1 s=READY pgs=636 cs=0 l=1 rev1=1 crypto rx=0x7f5030009930 tx=0x7f503002a420 comp rx=0 tx=0).stop 2024-04-23T10:30:52.754 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:30:52.751+0000 7f5045631640 1 -- 172.21.15.114:0/3435279358 shutdown_connections 2024-04-23T10:30:52.754 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:30:52.751+0000 7f5045631640 1 --2- 172.21.15.114:0/3435279358 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f5040100e10 0x7f504010ede0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:30:52.754 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:30:52.751+0000 7f5045631640 1 --2- 172.21.15.114:0/3435279358 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f5040100450 0x7f50401008d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:30:52.754 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:30:52.751+0000 7f5045631640 1 --2- 172.21.15.114:0/3435279358 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f5040106f30 0x7f5040107310 unknown :-1 s=CLOSED pgs=636 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:30:52.754 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:30:52.751+0000 7f5045631640 1 -- 172.21.15.114:0/3435279358 >> 172.21.15.114:0/3435279358 conn(0x7f50400773e0 msgr2=0x7f50400fe760 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T10:30:52.754 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:30:52.751+0000 7f5045631640 1 -- 172.21.15.114:0/3435279358 shutdown_connections 2024-04-23T10:30:52.754 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:30:52.751+0000 7f5045631640 1 -- 172.21.15.114:0/3435279358 wait complete. 2024-04-23T10:30:52.754 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:30:52.751+0000 7f5045631640 1 Processor -- start 2024-04-23T10:30:52.754 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:30:52.752+0000 7f5045631640 1 -- start start 2024-04-23T10:30:52.755 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:30:52.752+0000 7f5045631640 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f5040100450 0x7f50401987b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T10:30:52.755 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:30:52.752+0000 7f5045631640 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f5040100e10 0x7f5040198cf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T10:30:52.755 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:30:52.752+0000 7f5045631640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f5040106f30 0x7f504019d020 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T10:30:52.755 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:30:52.752+0000 7f5045631640 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x7f5040112290 con 0x7f5040106f30 2024-04-23T10:30:52.755 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:30:52.752+0000 7f5045631640 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 -- 0x7f5040112110 con 0x7f5040100450 2024-04-23T10:30:52.755 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:30:52.752+0000 7f5045631640 1 -- --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_getmap magic: 0 -- 0x7f5040112410 con 0x7f5040100e10 2024-04-23T10:30:52.755 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:30:52.752+0000 7f503f7fe640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f5040106f30 0x7f504019d020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T10:30:52.755 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:30:52.752+0000 7f503f7fe640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f5040106f30 0x7f504019d020 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.114:45118/0 (socket says 172.21.15.114:45118) 2024-04-23T10:30:52.755 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:30:52.752+0000 7f503f7fe640 1 -- 172.21.15.114:0/3383325432 learned_addr learned my addr 172.21.15.114:0/3383325432 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-04-23T10:30:52.755 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:30:52.752+0000 7f503effd640 1 --2- 172.21.15.114:0/3383325432 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f5040100450 0x7f50401987b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T10:30:52.756 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:30:52.752+0000 7f503e7fc640 1 --2- 172.21.15.114:0/3383325432 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f5040100e10 0x7f5040198cf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T10:30:52.756 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:30:52.752+0000 7f503f7fe640 1 -- 172.21.15.114:0/3383325432 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f5040100e10 msgr2=0x7f5040198cf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T10:30:52.756 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:30:52.752+0000 7f503f7fe640 1 --2- 172.21.15.114:0/3383325432 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f5040100e10 0x7f5040198cf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:30:52.756 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:30:52.752+0000 7f503f7fe640 1 -- 172.21.15.114:0/3383325432 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f5040100450 msgr2=0x7f50401987b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T10:30:52.756 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:30:52.752+0000 7f503f7fe640 1 --2- 172.21.15.114:0/3383325432 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f5040100450 0x7f50401987b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:30:52.756 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:30:52.752+0000 7f503f7fe640 1 -- 172.21.15.114:0/3383325432 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f504019d750 con 0x7f5040106f30 2024-04-23T10:30:52.756 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:30:52.752+0000 7f503f7fe640 1 --2- 172.21.15.114:0/3383325432 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f5040106f30 0x7f504019d020 secure :-1 s=READY pgs=637 cs=0 l=1 rev1=1 crypto rx=0x7f503800efd0 tx=0x7f503800c6a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T10:30:52.756 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:30:52.752+0000 7f5027fff640 1 -- 172.21.15.114:0/3383325432 <== mon.0 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5038003b70 con 0x7f5040106f30 2024-04-23T10:30:52.756 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:30:52.753+0000 7f5045631640 1 -- 172.21.15.114:0/3383325432 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f504019da40 con 0x7f5040106f30 2024-04-23T10:30:52.756 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:30:52.753+0000 7f5027fff640 1 -- 172.21.15.114:0/3383325432 <== mon.0 v2:172.21.15.114:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f5038003d10 con 0x7f5040106f30 2024-04-23T10:30:52.756 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:30:52.753+0000 7f5027fff640 1 -- 172.21.15.114:0/3383325432 <== mon.0 v2:172.21.15.114:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f50380108a0 con 0x7f5040106f30 2024-04-23T10:30:52.757 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:30:52.753+0000 7f5045631640 1 -- 172.21.15.114:0/3383325432 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f50401a52d0 con 0x7f5040106f30 2024-04-23T10:30:52.765 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:30:52.755+0000 7f5048427740 1 Processor -- start 2024-04-23T10:30:52.765 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:30:52.755+0000 7f5027fff640 1 -- 172.21.15.114:0/3383325432 <== mon.0 v2:172.21.15.114:3300/0 4 ==== mgrmap(e 26) ==== 138778+0+0 (secure 0 0 0) 0x7f5038004020 con 0x7f5040106f30 2024-04-23T10:30:52.765 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:30:52.756+0000 7f5048427740 1 -- start start 2024-04-23T10:30:52.765 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:30:52.756+0000 7f5048427740 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x5648c1df9520 0x5648c1df9900 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T10:30:52.765 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:30:52.756+0000 7f5048427740 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x5648c1df9e40 0x5648c1e086d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T10:30:52.765 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:30:52.756+0000 7f5048427740 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x5648c1e08c10 0x5648c1e0b000 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T10:30:52.765 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:30:52.756+0000 7f5048427740 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x5648c1d5e970 con 0x5648c1df9520 2024-04-23T10:30:52.765 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:30:52.756+0000 7f5048427740 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 -- 0x5648c1cfb4a0 con 0x5648c1df9e40 2024-04-23T10:30:52.765 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:30:52.756+0000 7f5048427740 1 -- --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_getmap magic: 0 -- 0x5648c1d5a930 con 0x5648c1e08c10 2024-04-23T10:30:52.765 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:30:52.756+0000 7f503e7fc640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x5648c1df9520 0x5648c1df9900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T10:30:52.766 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:30:52.756+0000 7f503e7fc640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x5648c1df9520 0x5648c1df9900 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.114:45128/0 (socket says 172.21.15.114:45128) 2024-04-23T10:30:52.766 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:30:52.756+0000 7f503e7fc640 1 -- 172.21.15.114:0/3249014124 learned_addr learned my addr 172.21.15.114:0/3249014124 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-04-23T10:30:52.766 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:30:52.756+0000 7f503effd640 1 --2- 172.21.15.114:0/3249014124 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x5648c1df9e40 0x5648c1e086d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T10:30:52.766 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:30:52.756+0000 7f5027fff640 1 --2- 172.21.15.114:0/3383325432 >> [v2:172.21.15.114:6800/1395693466,v1:172.21.15.114:6801/1395693466] conn(0x7f50280a3dc0 0x7f50280a6280 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T10:30:52.766 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:30:52.756+0000 7f503e7fc640 1 -- 172.21.15.114:0/3249014124 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x5648c1e08c10 msgr2=0x5648c1e0b000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T10:30:52.766 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:30:52.756+0000 7f503e7fc640 1 --2- 172.21.15.114:0/3249014124 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x5648c1e08c10 0x5648c1e0b000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:30:52.766 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:30:52.756+0000 7f503e7fc640 1 -- 172.21.15.114:0/3249014124 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x5648c1df9e40 msgr2=0x5648c1e086d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T10:30:52.766 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:30:52.756+0000 7f503e7fc640 1 --2- 172.21.15.114:0/3249014124 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x5648c1df9e40 0x5648c1e086d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:30:52.766 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:30:52.756+0000 7f503e7fc640 1 -- 172.21.15.114:0/3249014124 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5648c1a71210 con 0x5648c1df9520 2024-04-23T10:30:52.767 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:30:52.756+0000 7f503f7fe640 1 --2- 172.21.15.114:0/3383325432 >> [v2:172.21.15.114:6800/1395693466,v1:172.21.15.114:6801/1395693466] conn(0x7f50280a3dc0 0x7f50280a6280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T10:30:52.767 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:30:52.756+0000 7f503e7fc640 1 --2- 172.21.15.114:0/3249014124 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x5648c1df9520 0x5648c1df9900 secure :-1 s=READY pgs=638 cs=0 l=1 rev1=1 crypto rx=0x7f5034009d40 tx=0x7f5034013830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c8a10e1b68c984bc server_cookie=0 in_seq=0 out_seq=0 2024-04-23T10:30:52.767 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:30:52.756+0000 7f5027fff640 1 -- 172.21.15.114:0/3383325432 <== mon.0 v2:172.21.15.114:3300/0 5 ==== osd_map(92..92 src has 1..92) ==== 8585+0+0 (secure 0 0 0) 0x7f50380d0c30 con 0x7f5040106f30 2024-04-23T10:30:52.767 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:30:52.756+0000 7f50257fa640 1 -- 172.21.15.114:0/3249014124 <== mon.0 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5034004050 con 0x5648c1df9520 2024-04-23T10:30:52.767 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:30:52.757+0000 7f50257fa640 1 -- 172.21.15.114:0/3249014124 <== mon.0 v2:172.21.15.114:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f5034017030 con 0x5648c1df9520 2024-04-23T10:30:52.767 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:30:52.757+0000 7f50257fa640 1 -- 172.21.15.114:0/3249014124 <== mon.0 v2:172.21.15.114:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5034010430 con 0x5648c1df9520 2024-04-23T10:30:52.767 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:30:52.757+0000 7f5048427740 1 -- 172.21.15.114:0/3249014124 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x5648c1df9520 msgr2=0x5648c1df9900 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T10:30:52.767 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:30:52.757+0000 7f5048427740 1 --2- 172.21.15.114:0/3249014124 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x5648c1df9520 0x5648c1df9900 secure :-1 s=READY pgs=638 cs=0 l=1 rev1=1 crypto rx=0x7f5034009d40 tx=0x7f5034013830 comp rx=0 tx=0).stop 2024-04-23T10:30:52.767 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:30:52.757+0000 7f503f7fe640 1 --2- 172.21.15.114:0/3383325432 >> [v2:172.21.15.114:6800/1395693466,v1:172.21.15.114:6801/1395693466] conn(0x7f50280a3dc0 0x7f50280a6280 secure :-1 s=READY pgs=96 cs=0 l=1 rev1=1 crypto rx=0x7f503809cb20 tx=0x7f503807d040 comp rx=0 tx=0).ready entity=mgr.15291 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T10:30:52.767 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:30:52.757+0000 7f5048427740 1 -- 172.21.15.114:0/3249014124 shutdown_connections 2024-04-23T10:30:52.767 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:30:52.757+0000 7f5048427740 1 --2- 172.21.15.114:0/3249014124 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x5648c1e08c10 0x5648c1e0b000 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:30:52.768 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:30:52.757+0000 7f5048427740 1 --2- 172.21.15.114:0/3249014124 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x5648c1df9e40 0x5648c1e086d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:30:52.768 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:30:52.757+0000 7f5048427740 1 --2- 172.21.15.114:0/3249014124 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x5648c1df9520 0x5648c1df9900 unknown :-1 s=CLOSED pgs=638 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:30:52.768 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:30:52.757+0000 7f5048427740 1 -- 172.21.15.114:0/3249014124 >> 172.21.15.114:0/3249014124 conn(0x5648c1df5b30 msgr2=0x5648c1df5f10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T10:30:52.768 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:30:52.757+0000 7f5048427740 1 -- 172.21.15.114:0/3249014124 shutdown_connections 2024-04-23T10:30:52.768 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:30:52.757+0000 7f5048427740 1 -- 172.21.15.114:0/3249014124 wait complete. 2024-04-23T10:30:52.768 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:30:52.758+0000 7f5048427740 1 -- start start 2024-04-23T10:30:52.768 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:30:52.758+0000 7f5048427740 1 Processor -- start 2024-04-23T10:30:52.768 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:30:52.758+0000 7f5048427740 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x5648c1df9520 0x5648c1e084d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T10:30:52.768 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:30:52.758+0000 7f5048427740 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x5648c1df9e40 0x5648c1e057c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T10:30:52.768 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:30:52.758+0000 7f5048427740 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x5648c1e08c10 0x5648c1e05d00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T10:30:52.768 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:30:52.758+0000 7f5048427740 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x5648c1e0b840 con 0x5648c1df9e40 2024-04-23T10:30:52.769 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:30:52.758+0000 7f5048427740 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 -- 0x5648c1cf90b0 con 0x5648c1df9520 2024-04-23T10:30:52.769 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:30:52.758+0000 7f5048427740 1 -- --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_getmap magic: 0 -- 0x5648c1e0b9c0 con 0x5648c1e08c10 2024-04-23T10:30:52.769 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:30:52.758+0000 7f5048427740 10 client.0 _ll_register_callbacks cb 0x5648c1ce6d30 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-23T10:30:52.769 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:30:52.758+0000 7f503effd640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x5648c1df9e40 0x5648c1e057c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T10:30:52.769 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:30:52.758+0000 7f503effd640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x5648c1df9e40 0x5648c1e057c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.114:45132/0 (socket says 172.21.15.114:45132) 2024-04-23T10:30:52.769 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:30:52.758+0000 7f503e7fc640 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x5648c1df9520 0x5648c1e084d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T10:30:52.769 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:30:52.758+0000 7f503effd640 1 -- 172.21.15.114:0/1486050918 learned_addr learned my addr 172.21.15.114:0/1486050918 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-04-23T10:30:52.769 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:30:52.758+0000 7f503e7fc640 1 --2- 172.21.15.114:0/1486050918 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x5648c1e08c10 0x5648c1e05d00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T10:30:52.769 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:30:52.758+0000 7f503effd640 1 -- 172.21.15.114:0/1486050918 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x5648c1e08c10 msgr2=0x5648c1e05d00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T10:30:52.769 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:30:52.758+0000 7f503effd640 1 --2- 172.21.15.114:0/1486050918 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x5648c1e08c10 0x5648c1e05d00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:30:52.769 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:30:52.758+0000 7f503effd640 1 -- 172.21.15.114:0/1486050918 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x5648c1df9520 msgr2=0x5648c1e084d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T10:30:52.769 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:30:52.758+0000 7f503effd640 1 --2- 172.21.15.114:0/1486050918 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x5648c1df9520 0x5648c1e084d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:30:52.770 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:30:52.758+0000 7f503effd640 1 -- 172.21.15.114:0/1486050918 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5648c1b3d620 con 0x5648c1df9e40 2024-04-23T10:30:52.770 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:30:52.758+0000 7f503e7fc640 1 --2- 172.21.15.114:0/1486050918 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x5648c1df9520 0x5648c1e084d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-23T10:30:52.770 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:30:52.758+0000 7f503effd640 1 --2- 172.21.15.114:0/1486050918 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x5648c1df9e40 0x5648c1e057c0 secure :-1 s=READY pgs=639 cs=0 l=1 rev1=1 crypto rx=0x7f503002a940 tx=0x7f5030031e80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1b7114e9cc642771 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T10:30:52.770 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:30:52.758+0000 7f5022ff5640 10 client.0 ms_handle_connect on v2:172.21.15.114:3300/0 2024-04-23T10:30:52.770 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:30:52.759+0000 7f5022ff5640 1 -- 172.21.15.114:0/1486050918 <== mon.0 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5030035040 con 0x5648c1df9e40 2024-04-23T10:30:52.770 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:30:52.759+0000 7f5022ff5640 1 -- 172.21.15.114:0/1486050918 <== mon.0 v2:172.21.15.114:3300/0 2 ==== osd_map(92..92 src has 1..92) ==== 8585+0+0 (secure 0 0 0) 0x7f50300317e0 con 0x5648c1df9e40 2024-04-23T10:30:52.770 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:30:52.759+0000 7f5048427740 1 -- 172.21.15.114:0/1486050918 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f5034017030 con 0x5648c1df9e40 2024-04-23T10:30:52.770 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:30:52.759+0000 7f5022ff5640 1 -- 172.21.15.114:0/1486050918 <== mon.0 v2:172.21.15.114:3300/0 3 ==== mon_get_version_reply(handle=1 version=26) ==== 24+0+0 (secure 0 0 0) 0x7f5030004040 con 0x5648c1df9e40 2024-04-23T10:30:52.770 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:30:52.759+0000 7f5048427740 10 client.16086 fetch_fsmap learned FSMap version 26 2024-04-23T10:30:52.770 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:30:52.759+0000 7f5048427740 1 -- 172.21.15.114:0/1486050918 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({config=0+,fsmap=26,monmap=4+}) -- 0x5648c1e17750 con 0x5648c1df9e40 2024-04-23T10:30:52.771 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:30:52.759+0000 7f5022ff5640 1 -- 172.21.15.114:0/1486050918 <== mon.0 v2:172.21.15.114:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f5034017030 con 0x5648c1df9e40 2024-04-23T10:30:52.771 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:30:52.759+0000 7f5022ff5640 1 -- 172.21.15.114:0/1486050918 <== mon.0 v2:172.21.15.114:3300/0 5 ==== fsmap(e 26) ==== 8338+0+0 (secure 0 0 0) 0x7f50300048c0 con 0x5648c1df9e40 2024-04-23T10:30:52.771 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:30:52.759+0000 7f5048427740 10 client.16086 fetch_fsmap finished waiting for FSMap version 26 2024-04-23T10:30:52.771 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:30:52.759+0000 7f5048427740 10 client.16086 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-04-23T10:30:52.771 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:30:52.759+0000 7f5048427740 20 client.16086 populate_metadata read hostname 'smithi114' 2024-04-23T10:30:52.771 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:30:52.759+0000 7f5048427740 1 --2- 172.21.15.114:0/1486050918 >> [v2:172.21.15.119:6838/937297340,v1:172.21.15.119:6839/937297340] conn(0x5648c1e18c10 0x5648c1e1b000 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T10:30:52.771 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:30:52.760+0000 7f5048427740 4 client.16086 mds_command: new command op to 24545 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-23T10:30:52.771 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:30:52.760+0000 7f5048427740 1 -- 172.21.15.114:0/1486050918 --> [v2:172.21.15.119:6838/937297340,v1:172.21.15.119:6839/937297340] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x5648c1d6c390 con 0x5648c1e18c10 2024-04-23T10:30:52.771 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:30:52.760+0000 7f503e7fc640 1 --2- 172.21.15.114:0/1486050918 >> [v2:172.21.15.119:6838/937297340,v1:172.21.15.119:6839/937297340] conn(0x5648c1e18c10 0x5648c1e1b000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T10:30:52.771 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:30:52.760+0000 7f503e7fc640 1 --2- 172.21.15.114:0/1486050918 >> [v2:172.21.15.119:6838/937297340,v1:172.21.15.119:6839/937297340] conn(0x5648c1e18c10 0x5648c1e1b000 crc :-1 s=READY pgs=92 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=328baff2976aaabe server_cookie=a98cff8e298a4427 in_seq=0 out_seq=0 2024-04-23T10:30:52.771 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:30:52.760+0000 7f5022ff5640 10 client.16086 ms_handle_connect on v2:172.21.15.119:6838/937297340 2024-04-23T10:30:52.771 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:30:52.761+0000 7f5022ff5640 1 -- 172.21.15.114:0/1486050918 <== mds.0 v2:172.21.15.119:6838/937297340 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x5648c1d6c390 con 0x5648c1e18c10 2024-04-23T10:30:52.772 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:30:52.761+0000 7f5022ff5640 10 client.16086 handle_command_reply: tid=0 2024-04-23T10:30:52.772 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:30:52.761+0000 7f5048427740 1 client.16086 shutdown 2024-04-23T10:30:52.772 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:30:52.761+0000 7f5048427740 1 -- 172.21.15.114:0/1486050918 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x5648c1df9e40 msgr2=0x5648c1e057c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T10:30:52.772 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:30:52.761+0000 7f5048427740 1 --2- 172.21.15.114:0/1486050918 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x5648c1df9e40 0x5648c1e057c0 secure :-1 s=READY pgs=639 cs=0 l=1 rev1=1 crypto rx=0x7f503002a940 tx=0x7f5030031e80 comp rx=0 tx=0).stop 2024-04-23T10:30:52.772 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:30:52.761+0000 7f5048427740 1 -- 172.21.15.114:0/1486050918 shutdown_connections 2024-04-23T10:30:52.772 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:30:52.761+0000 7f5048427740 1 --2- 172.21.15.114:0/1486050918 >> [v2:172.21.15.119:6838/937297340,v1:172.21.15.119:6839/937297340] conn(0x5648c1e18c10 0x5648c1e1b000 crc :-1 s=READY pgs=92 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:30:52.772 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:30:52.761+0000 7f5048427740 1 --2- 172.21.15.114:0/1486050918 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x5648c1e08c10 0x5648c1e05d00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:30:52.772 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:30:52.761+0000 7f5048427740 1 --2- 172.21.15.114:0/1486050918 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x5648c1df9e40 0x5648c1e057c0 unknown :-1 s=CLOSED pgs=639 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:30:52.772 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:30:52.761+0000 7f5022ff5640 0 client.16086 ms_handle_reset on v2:172.21.15.119:6838/937297340 2024-04-23T10:30:52.772 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:30:52.761+0000 7f5048427740 1 --2- 172.21.15.114:0/1486050918 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x5648c1df9520 0x5648c1e084d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:30:52.772 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:30:52.761+0000 7f5048427740 1 -- 172.21.15.114:0/1486050918 >> 172.21.15.114:0/1486050918 conn(0x5648c1df5b30 msgr2=0x5648c1dfb9a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T10:30:52.773 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:30:52.761+0000 7f5048427740 1 -- 172.21.15.114:0/1486050918 shutdown_connections 2024-04-23T10:30:52.773 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:30:52.761+0000 7f5048427740 1 -- 172.21.15.114:0/1486050918 wait complete. 2024-04-23T10:30:52.773 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:30:52.761+0000 7f5048427740 20 client.16086 trim_cache size 0 max 16384 2024-04-23T10:30:52.783 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:30:52.778+0000 7f5048427740 1 Processor -- start 2024-04-23T10:30:52.783 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:30:52.778+0000 7f5048427740 1 -- start start 2024-04-23T10:30:52.783 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:30:52.778+0000 7f5048427740 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x5648c1df9520 0x5648c1e14740 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T10:30:52.783 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:30:52.778+0000 7f5048427740 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x5648c1df9e40 0x5648c1e00d80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T10:30:52.783 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:30:52.778+0000 7f5048427740 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x5648c1e08c10 0x5648c1e08000 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T10:30:52.783 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:30:52.778+0000 7f5048427740 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x7f50401ae240 con 0x5648c1df9520 2024-04-23T10:30:52.783 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:30:52.778+0000 7f5048427740 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 -- 0x7f50401abb10 con 0x5648c1df9e40 2024-04-23T10:30:52.784 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:30:52.778+0000 7f5048427740 1 -- --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_getmap magic: 0 -- 0x7f50401a9960 con 0x5648c1e08c10 2024-04-23T10:30:52.784 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:30:52.778+0000 7f503e7fc640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x5648c1df9520 0x5648c1e14740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T10:30:52.784 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:30:52.778+0000 7f503e7fc640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x5648c1df9520 0x5648c1e14740 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.114:45138/0 (socket says 172.21.15.114:45138) 2024-04-23T10:30:52.784 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:30:52.778+0000 7f503e7fc640 1 -- 172.21.15.114:0/916543749 learned_addr learned my addr 172.21.15.114:0/916543749 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-04-23T10:30:52.784 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:30:52.778+0000 7f503effd640 1 --2- 172.21.15.114:0/916543749 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x5648c1df9e40 0x5648c1e00d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T10:30:52.784 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:30:52.778+0000 7f503e7fc640 1 --2- 172.21.15.114:0/916543749 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x5648c1e08c10 0x5648c1e08000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T10:30:52.784 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:30:52.779+0000 7f503e7fc640 1 -- 172.21.15.114:0/916543749 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x5648c1e08c10 msgr2=0x5648c1e08000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T10:30:52.784 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:30:52.779+0000 7f503e7fc640 1 --2- 172.21.15.114:0/916543749 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x5648c1e08c10 0x5648c1e08000 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:30:52.784 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:30:52.779+0000 7f503e7fc640 1 -- 172.21.15.114:0/916543749 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x5648c1df9e40 msgr2=0x5648c1e00d80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T10:30:52.784 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:30:52.779+0000 7f503e7fc640 1 --2- 172.21.15.114:0/916543749 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x5648c1df9e40 0x5648c1e00d80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:30:52.784 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:30:52.779+0000 7f503e7fc640 1 -- 172.21.15.114:0/916543749 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5648c1e1b370 con 0x5648c1df9520 2024-04-23T10:30:52.785 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:30:52.779+0000 7f503effd640 1 --2- 172.21.15.114:0/916543749 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x5648c1df9e40 0x5648c1e00d80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-23T10:30:52.785 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:30:52.779+0000 7f503e7fc640 1 --2- 172.21.15.114:0/916543749 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x5648c1df9520 0x5648c1e14740 secure :-1 s=READY pgs=640 cs=0 l=1 rev1=1 crypto rx=0x7f5034009250 tx=0x7f5034005d30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9f3946abd5bae096 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T10:30:52.785 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:30:52.779+0000 7f5024ff9640 1 -- 172.21.15.114:0/916543749 <== mon.0 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5034013180 con 0x5648c1df9520 2024-04-23T10:30:52.785 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:30:52.779+0000 7f5024ff9640 1 -- 172.21.15.114:0/916543749 <== mon.0 v2:172.21.15.114:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f5034017070 con 0x5648c1df9520 2024-04-23T10:30:52.785 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:30:52.779+0000 7f5024ff9640 1 -- 172.21.15.114:0/916543749 <== mon.0 v2:172.21.15.114:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5034003f00 con 0x5648c1df9520 2024-04-23T10:30:52.785 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:30:52.780+0000 7f5048427740 1 -- 172.21.15.114:0/916543749 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x5648c1df9520 msgr2=0x5648c1e14740 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T10:30:52.785 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:30:52.780+0000 7f5048427740 1 --2- 172.21.15.114:0/916543749 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x5648c1df9520 0x5648c1e14740 secure :-1 s=READY pgs=640 cs=0 l=1 rev1=1 crypto rx=0x7f5034009250 tx=0x7f5034005d30 comp rx=0 tx=0).stop 2024-04-23T10:30:52.785 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:30:52.780+0000 7f5048427740 1 -- 172.21.15.114:0/916543749 shutdown_connections 2024-04-23T10:30:52.785 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:30:52.780+0000 7f5048427740 1 --2- 172.21.15.114:0/916543749 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x5648c1e08c10 0x5648c1e08000 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:30:52.785 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:30:52.780+0000 7f5048427740 1 --2- 172.21.15.114:0/916543749 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x5648c1df9e40 0x5648c1e00d80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:30:52.785 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:30:52.780+0000 7f5048427740 1 --2- 172.21.15.114:0/916543749 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x5648c1df9520 0x5648c1e14740 unknown :-1 s=CLOSED pgs=640 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:30:52.785 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:30:52.780+0000 7f5048427740 1 -- 172.21.15.114:0/916543749 >> 172.21.15.114:0/916543749 conn(0x5648c1df5b30 msgr2=0x5648c1e00670 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T10:30:52.786 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:30:52.780+0000 7f5048427740 1 -- 172.21.15.114:0/916543749 shutdown_connections 2024-04-23T10:30:52.786 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:30:52.780+0000 7f5048427740 1 -- 172.21.15.114:0/916543749 wait complete. 2024-04-23T10:30:52.786 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:30:52.780+0000 7f5048427740 1 -- start start 2024-04-23T10:30:52.786 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:30:52.780+0000 7f5048427740 1 Processor -- start 2024-04-23T10:30:52.786 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:30:52.780+0000 7f5048427740 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x5648c1df9520 0x5648c1e10fb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T10:30:52.786 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:30:52.780+0000 7f5048427740 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x5648c1df9e40 0x5648c1e02150 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T10:30:52.786 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:30:52.780+0000 7f5048427740 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x5648c1e08c10 0x5648c1e02690 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T10:30:52.786 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:30:52.780+0000 7f5048427740 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x7f50401a9fd0 con 0x5648c1df9520 2024-04-23T10:30:52.786 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:30:52.780+0000 7f5048427740 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 -- 0x7f50401ab440 con 0x5648c1e08c10 2024-04-23T10:30:52.786 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:30:52.780+0000 7f5048427740 1 -- --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_getmap magic: 0 -- 0x7f50401aa6a0 con 0x5648c1df9e40 2024-04-23T10:30:52.786 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:30:52.781+0000 7f5048427740 10 client.0 _ll_register_callbacks cb 0x7f5030031b10 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-23T10:30:52.787 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:30:52.781+0000 7f503e7fc640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x5648c1df9520 0x5648c1e10fb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T10:30:52.787 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:30:52.781+0000 7f503e7fc640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x5648c1df9520 0x5648c1e10fb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.114:45146/0 (socket says 172.21.15.114:45146) 2024-04-23T10:30:52.787 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:30:52.781+0000 7f503e7fc640 1 -- 172.21.15.114:0/2556609688 learned_addr learned my addr 172.21.15.114:0/2556609688 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-04-23T10:30:52.787 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:30:52.781+0000 7f503effd640 1 --2- 172.21.15.114:0/2556609688 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x5648c1df9e40 0x5648c1e02150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T10:30:52.787 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:30:52.781+0000 7f503e7fc640 1 --2- 172.21.15.114:0/2556609688 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x5648c1e08c10 0x5648c1e02690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T10:30:52.787 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:30:52.781+0000 7f503e7fc640 1 -- 172.21.15.114:0/2556609688 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x5648c1df9e40 msgr2=0x5648c1e02150 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T10:30:52.787 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:30:52.781+0000 7f503e7fc640 1 --2- 172.21.15.114:0/2556609688 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x5648c1df9e40 0x5648c1e02150 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:30:52.787 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:30:52.781+0000 7f503e7fc640 1 -- 172.21.15.114:0/2556609688 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x5648c1e08c10 msgr2=0x5648c1e02690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T10:30:52.787 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:30:52.781+0000 7f503e7fc640 1 --2- 172.21.15.114:0/2556609688 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x5648c1e08c10 0x5648c1e02690 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:30:52.787 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:30:52.781+0000 7f503e7fc640 1 -- 172.21.15.114:0/2556609688 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5648c1e02e60 con 0x5648c1df9520 2024-04-23T10:30:52.788 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:30:52.781+0000 7f503effd640 1 --2- 172.21.15.114:0/2556609688 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x5648c1df9e40 0x5648c1e02150 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-23T10:30:52.788 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:30:52.781+0000 7f503e7fc640 1 --2- 172.21.15.114:0/2556609688 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x5648c1df9520 0x5648c1e10fb0 secure :-1 s=READY pgs=641 cs=0 l=1 rev1=1 crypto rx=0x7f5034006010 tx=0x7f5034005970 comp rx=0 tx=0).ready entity=mon.0 client_cookie=27eaba67273d4304 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T10:30:52.788 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:30:52.781+0000 7f5022ff5640 10 client.0 ms_handle_connect on v2:172.21.15.114:3300/0 2024-04-23T10:30:52.788 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:30:52.781+0000 7f5022ff5640 1 -- 172.21.15.114:0/2556609688 <== mon.0 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5034013200 con 0x5648c1df9520 2024-04-23T10:30:52.788 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:30:52.781+0000 7f5022ff5640 1 -- 172.21.15.114:0/2556609688 <== mon.0 v2:172.21.15.114:3300/0 2 ==== osd_map(92..92 src has 1..92) ==== 8585+0+0 (secure 0 0 0) 0x7f5034003d80 con 0x5648c1df9520 2024-04-23T10:30:52.788 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:30:52.781+0000 7f5048427740 1 -- 172.21.15.114:0/2556609688 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f5034017070 con 0x5648c1df9520 2024-04-23T10:30:52.788 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:30:52.781+0000 7f5022ff5640 1 -- 172.21.15.114:0/2556609688 <== mon.0 v2:172.21.15.114:3300/0 3 ==== mon_get_version_reply(handle=1 version=26) ==== 24+0+0 (secure 0 0 0) 0x7f5034003f80 con 0x5648c1df9520 2024-04-23T10:30:52.788 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:30:52.781+0000 7f5048427740 10 client.16092 fetch_fsmap learned FSMap version 26 2024-04-23T10:30:52.788 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:30:52.781+0000 7f5048427740 1 -- 172.21.15.114:0/2556609688 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({config=0+,fsmap=26,monmap=4+}) -- 0x5648c1e7f170 con 0x5648c1df9520 2024-04-23T10:30:52.788 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:30:52.782+0000 7f5022ff5640 1 -- 172.21.15.114:0/2556609688 <== mon.0 v2:172.21.15.114:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f5034017070 con 0x5648c1df9520 2024-04-23T10:30:52.788 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:30:52.782+0000 7f5022ff5640 1 -- 172.21.15.114:0/2556609688 <== mon.0 v2:172.21.15.114:3300/0 5 ==== fsmap(e 26) ==== 8338+0+0 (secure 0 0 0) 0x7f5034029030 con 0x5648c1df9520 2024-04-23T10:30:52.789 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:30:52.782+0000 7f5048427740 10 client.16092 fetch_fsmap finished waiting for FSMap version 26 2024-04-23T10:30:52.789 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:30:52.782+0000 7f5048427740 10 client.16092 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-04-23T10:30:52.789 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:30:52.782+0000 7f5048427740 20 client.16092 populate_metadata read hostname 'smithi114' 2024-04-23T10:30:52.789 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:30:52.782+0000 7f5048427740 1 --2- 172.21.15.114:0/2556609688 >> [v2:172.21.15.119:6838/937297340,v1:172.21.15.119:6839/937297340] conn(0x5648c1e7f640 0x5648c1e81a30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T10:30:52.789 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:30:52.782+0000 7f5048427740 4 client.16092 mds_command: new command op to 24545 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-04-23T10:30:52.789 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:30:52.782+0000 7f5048427740 1 -- 172.21.15.114:0/2556609688 --> [v2:172.21.15.119:6838/937297340,v1:172.21.15.119:6839/937297340] -- command(tid 0: {"prefix": "flush journal"}) -- 0x5648c1e06090 con 0x5648c1e7f640 2024-04-23T10:30:52.789 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:30:52.782+0000 7f503effd640 1 --2- 172.21.15.114:0/2556609688 >> [v2:172.21.15.119:6838/937297340,v1:172.21.15.119:6839/937297340] conn(0x5648c1e7f640 0x5648c1e81a30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T10:30:52.789 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:30:52.782+0000 7f5022ff5640 10 client.16092 ms_handle_connect on v2:172.21.15.119:6838/937297340 2024-04-23T10:30:52.789 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:30:52.782+0000 7f503effd640 1 --2- 172.21.15.114:0/2556609688 >> [v2:172.21.15.119:6838/937297340,v1:172.21.15.119:6839/937297340] conn(0x5648c1e7f640 0x5648c1e81a30 crc :-1 s=READY pgs=93 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=25ab23bb76c1f0a3 server_cookie=e5b5c6f36b5ddaa6 in_seq=0 out_seq=0 2024-04-23T10:30:52.836 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:30:52.836+0000 7f5022ff5640 1 -- 172.21.15.114:0/2556609688 <== mds.0 v2:172.21.15.119:6838/937297340 1 ==== command_reply(tid 0: 0 ) ==== 8+0+44 (crc 0 0 0) 0x5648c1e06090 con 0x5648c1e7f640 2024-04-23T10:30:52.836 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:30:52.836+0000 7f5022ff5640 10 client.16092 handle_command_reply: tid=0 2024-04-23T10:30:52.836 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:30:52.836+0000 7f5048427740 1 client.16092 shutdown 2024-04-23T10:30:52.836 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:30:52.837+0000 7f5048427740 1 -- 172.21.15.114:0/2556609688 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x5648c1df9520 msgr2=0x5648c1e10fb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T10:30:52.837 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:30:52.837+0000 7f5048427740 1 --2- 172.21.15.114:0/2556609688 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x5648c1df9520 0x5648c1e10fb0 secure :-1 s=READY pgs=641 cs=0 l=1 rev1=1 crypto rx=0x7f5034006010 tx=0x7f5034005970 comp rx=0 tx=0).stop 2024-04-23T10:30:52.837 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:30:52.837+0000 7f5048427740 1 -- 172.21.15.114:0/2556609688 shutdown_connections 2024-04-23T10:30:52.837 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:30:52.837+0000 7f5048427740 1 --2- 172.21.15.114:0/2556609688 >> [v2:172.21.15.119:6838/937297340,v1:172.21.15.119:6839/937297340] conn(0x5648c1e7f640 0x5648c1e81a30 crc :-1 s=READY pgs=93 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:30:52.837 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:30:52.837+0000 7f5048427740 1 --2- 172.21.15.114:0/2556609688 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x5648c1e08c10 0x5648c1e02690 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:30:52.837 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:30:52.837+0000 7f5048427740 1 --2- 172.21.15.114:0/2556609688 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x5648c1df9e40 0x5648c1e02150 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:30:52.837 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:30:52.837+0000 7f5022ff5640 0 client.16092 ms_handle_reset on v2:172.21.15.119:6838/937297340 2024-04-23T10:30:52.837 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:30:52.837+0000 7f5048427740 1 --2- 172.21.15.114:0/2556609688 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x5648c1df9520 0x5648c1e10fb0 unknown :-1 s=CLOSED pgs=641 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:30:52.837 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:30:52.837+0000 7f5048427740 1 -- 172.21.15.114:0/2556609688 >> 172.21.15.114:0/2556609688 conn(0x5648c1df5b30 msgr2=0x5648c1df4420 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T10:30:52.837 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:30:52.837+0000 7f5048427740 1 -- 172.21.15.114:0/2556609688 shutdown_connections 2024-04-23T10:30:52.837 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:30:52.837+0000 7f5048427740 1 -- 172.21.15.114:0/2556609688 wait complete. 2024-04-23T10:30:52.837 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:30:52.837+0000 7f5048427740 20 client.16092 trim_cache size 0 max 16384 2024-04-23T10:30:52.838 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stdout:{ 2024-04-23T10:30:52.838 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stdout: "message": "", 2024-04-23T10:30:52.838 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stdout: "return_code": 0 2024-04-23T10:30:52.838 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stdout:} 2024-04-23T10:30:52.839 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:30:52.837+0000 7f5045631640 1 -- 172.21.15.114:0/3383325432 >> [v2:172.21.15.114:6800/1395693466,v1:172.21.15.114:6801/1395693466] conn(0x7f50280a3dc0 msgr2=0x7f50280a6280 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T10:30:52.839 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:30:52.838+0000 7f5045631640 1 --2- 172.21.15.114:0/3383325432 >> [v2:172.21.15.114:6800/1395693466,v1:172.21.15.114:6801/1395693466] conn(0x7f50280a3dc0 0x7f50280a6280 secure :-1 s=READY pgs=96 cs=0 l=1 rev1=1 crypto rx=0x7f503809cb20 tx=0x7f503807d040 comp rx=0 tx=0).stop 2024-04-23T10:30:52.839 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:30:52.838+0000 7f5045631640 1 -- 172.21.15.114:0/3383325432 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f5040106f30 msgr2=0x7f504019d020 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T10:30:52.839 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:30:52.838+0000 7f5045631640 1 --2- 172.21.15.114:0/3383325432 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f5040106f30 0x7f504019d020 secure :-1 s=READY pgs=637 cs=0 l=1 rev1=1 crypto rx=0x7f503800efd0 tx=0x7f503800c6a0 comp rx=0 tx=0).stop 2024-04-23T10:30:52.839 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:30:52.838+0000 7f5045631640 1 -- 172.21.15.114:0/3383325432 shutdown_connections 2024-04-23T10:30:52.839 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:30:52.838+0000 7f5045631640 1 --2- 172.21.15.114:0/3383325432 >> [v2:172.21.15.114:6800/1395693466,v1:172.21.15.114:6801/1395693466] conn(0x7f50280a3dc0 0x7f50280a6280 unknown :-1 s=CLOSED pgs=96 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:30:52.839 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:30:52.838+0000 7f5045631640 1 --2- 172.21.15.114:0/3383325432 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f5040106f30 0x7f504019d020 unknown :-1 s=CLOSED pgs=637 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:30:52.839 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:30:52.838+0000 7f5045631640 1 --2- 172.21.15.114:0/3383325432 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f5040100e10 0x7f5040198cf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:30:52.839 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:30:52.838+0000 7f5045631640 1 --2- 172.21.15.114:0/3383325432 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f5040100450 0x7f50401987b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:30:52.840 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:30:52.838+0000 7f5045631640 1 -- 172.21.15.114:0/3383325432 >> 172.21.15.114:0/3383325432 conn(0x7f50400773e0 msgr2=0x7f50400fe760 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T10:30:52.840 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:30:52.838+0000 7f5045631640 1 -- 172.21.15.114:0/3383325432 shutdown_connections 2024-04-23T10:30:52.840 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:30:52.838+0000 7f5045631640 1 -- 172.21.15.114:0/3383325432 wait complete. 2024-04-23T10:30:52.917 INFO:tasks.workunit.client.0.smithi114.stdout: 1 268151 7.92 MB/sec execute 61 sec latency 74.993 ms 2024-04-23T10:30:53.917 INFO:tasks.workunit.client.0.smithi114.stdout: 1 269831 8.00 MB/sec execute 62 sec latency 193.147 ms 2024-04-23T10:30:54.066 INFO:journalctl@ceph.mds.i.smithi204.stdout:Apr 23 10:30:53 smithi204 ceph-aa240572-0159-11ef-bc93-c7b262605968-mds-i[63304]: ignoring --setuser ceph since I am not root 2024-04-23T10:30:54.067 INFO:journalctl@ceph.mds.i.smithi204.stdout:Apr 23 10:30:53 smithi204 ceph-aa240572-0159-11ef-bc93-c7b262605968-mds-i[63304]: ignoring --setgroup ceph since I am not root 2024-04-23T10:30:54.067 INFO:journalctl@ceph.mds.i.smithi204.stdout:Apr 23 10:30:53 smithi204 ceph-mds[63308]: -- 172.21.15.204:0/3941114799 <== mon.1 v2:172.21.15.204:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 194+0+0 (secure 0 0 0) 0x563925297a40 con 0x56392526b000 2024-04-23T10:30:54.067 INFO:journalctl@ceph.mds.i.smithi204.stdout:Apr 23 10:30:53 smithi204 ceph-aa240572-0159-11ef-bc93-c7b262605968-mds-i[63304]: starting mds.i at 2024-04-23T10:30:54.328 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 23 10:30:54 smithi204 ceph-mon[32649]: pgmap v131: 97 pgs: 97 active+clean; 96 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 43 MiB/s rd, 5.3 MiB/s wr, 562 op/s 2024-04-23T10:30:54.328 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 23 10:30:54 smithi204 ceph-mon[32649]: daemon mds.i restarted 2024-04-23T10:30:54.492 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 23 10:30:54 smithi119 ceph-mon[32608]: pgmap v131: 97 pgs: 97 active+clean; 96 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 43 MiB/s rd, 5.3 MiB/s wr, 562 op/s 2024-04-23T10:30:54.492 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 23 10:30:54 smithi119 ceph-mon[32608]: daemon mds.i restarted 2024-04-23T10:30:54.504 INFO:journalctl@ceph.mon.a.smithi114.stdout:Apr 23 10:30:54 smithi114 ceph-mon[29255]: pgmap v131: 97 pgs: 97 active+clean; 96 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 43 MiB/s rd, 5.3 MiB/s wr, 562 op/s 2024-04-23T10:30:54.504 INFO:journalctl@ceph.mon.a.smithi114.stdout:Apr 23 10:30:54 smithi114 ceph-mon[29255]: daemon mds.i restarted 2024-04-23T10:30:54.917 INFO:tasks.workunit.client.0.smithi114.stdout: 1 270928 7.93 MB/sec execute 63 sec latency 91.445 ms 2024-04-23T10:30:55.328 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 23 10:30:55 smithi204 ceph-mon[32649]: osdmap e93: 12 total, 12 up, 12 in 2024-04-23T10:30:55.328 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 23 10:30:55 smithi204 ceph-mon[32649]: mds.? [v2:172.21.15.204:6838/417363607,v1:172.21.15.204:6839/417363607] up:boot 2024-04-23T10:30:55.328 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 23 10:30:55 smithi204 ceph-mon[32649]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=e=up:active} 2 up:standby-replay 7 up:standby 2024-04-23T10:30:55.328 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 23 10:30:55 smithi204 ceph-mon[32649]: from='mgr.15291 172.21.15.114:0/1095789187' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "i"}]: dispatch 2024-04-23T10:30:55.328 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 23 10:30:55 smithi204 ceph-mon[32649]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=e=up:active} 3 up:standby-replay 6 up:standby 2024-04-23T10:30:55.491 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 23 10:30:55 smithi119 ceph-mon[32608]: osdmap e93: 12 total, 12 up, 12 in 2024-04-23T10:30:55.492 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 23 10:30:55 smithi119 ceph-mon[32608]: mds.? [v2:172.21.15.204:6838/417363607,v1:172.21.15.204:6839/417363607] up:boot 2024-04-23T10:30:55.492 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 23 10:30:55 smithi119 ceph-mon[32608]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=e=up:active} 2 up:standby-replay 7 up:standby 2024-04-23T10:30:55.492 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 23 10:30:55 smithi119 ceph-mon[32608]: from='mgr.15291 172.21.15.114:0/1095789187' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "i"}]: dispatch 2024-04-23T10:30:55.492 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 23 10:30:55 smithi119 ceph-mon[32608]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=e=up:active} 3 up:standby-replay 6 up:standby 2024-04-23T10:30:55.504 INFO:journalctl@ceph.mon.a.smithi114.stdout:Apr 23 10:30:55 smithi114 ceph-mon[29255]: osdmap e93: 12 total, 12 up, 12 in 2024-04-23T10:30:55.504 INFO:journalctl@ceph.mon.a.smithi114.stdout:Apr 23 10:30:55 smithi114 ceph-mon[29255]: mds.? [v2:172.21.15.204:6838/417363607,v1:172.21.15.204:6839/417363607] up:boot 2024-04-23T10:30:55.504 INFO:journalctl@ceph.mon.a.smithi114.stdout:Apr 23 10:30:55 smithi114 ceph-mon[29255]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=e=up:active} 2 up:standby-replay 7 up:standby 2024-04-23T10:30:55.504 INFO:journalctl@ceph.mon.a.smithi114.stdout:Apr 23 10:30:55 smithi114 ceph-mon[29255]: from='mgr.15291 172.21.15.114:0/1095789187' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "i"}]: dispatch 2024-04-23T10:30:55.504 INFO:journalctl@ceph.mon.a.smithi114.stdout:Apr 23 10:30:55 smithi114 ceph-mon[29255]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=e=up:active} 3 up:standby-replay 6 up:standby 2024-04-23T10:30:55.917 INFO:tasks.workunit.client.0.smithi114.stdout: 1 273031 8.00 MB/sec execute 64 sec latency 248.992 ms 2024-04-23T10:30:56.328 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 23 10:30:56 smithi204 ceph-mon[32649]: pgmap v133: 97 pgs: 97 active+clean; 96 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 26 MiB/s rd, 4.3 MiB/s wr, 422 op/s 2024-04-23T10:30:56.492 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 23 10:30:56 smithi119 ceph-mon[32608]: pgmap v133: 97 pgs: 97 active+clean; 96 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 26 MiB/s rd, 4.3 MiB/s wr, 422 op/s 2024-04-23T10:30:56.504 INFO:journalctl@ceph.mon.a.smithi114.stdout:Apr 23 10:30:56 smithi114 ceph-mon[29255]: pgmap v133: 97 pgs: 97 active+clean; 96 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 26 MiB/s rd, 4.3 MiB/s wr, 422 op/s 2024-04-23T10:30:56.917 INFO:tasks.workunit.client.0.smithi114.stdout: 1 273371 7.93 MB/sec execute 65 sec latency 671.029 ms 2024-04-23T10:30:57.917 INFO:tasks.workunit.client.0.smithi114.stdout: 1 274533 7.89 MB/sec execute 66 sec latency 75.203 ms 2024-04-23T10:30:58.328 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 23 10:30:58 smithi204 ceph-mon[32649]: pgmap v134: 97 pgs: 97 active+clean; 85 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 86 MiB/s rd, 3.9 MiB/s wr, 461 op/s 2024-04-23T10:30:58.491 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 23 10:30:58 smithi119 ceph-mon[32608]: pgmap v134: 97 pgs: 97 active+clean; 85 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 86 MiB/s rd, 3.9 MiB/s wr, 461 op/s 2024-04-23T10:30:58.504 INFO:journalctl@ceph.mon.a.smithi114.stdout:Apr 23 10:30:58 smithi114 ceph-mon[29255]: pgmap v134: 97 pgs: 97 active+clean; 85 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 86 MiB/s rd, 3.9 MiB/s wr, 461 op/s 2024-04-23T10:30:58.917 INFO:tasks.workunit.client.0.smithi114.stdout: 1 276929 7.98 MB/sec execute 67 sec latency 11.359 ms 2024-04-23T10:30:59.917 INFO:tasks.workunit.client.0.smithi114.stdout: 1 278181 7.94 MB/sec execute 68 sec latency 21.418 ms 2024-04-23T10:31:00.328 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 23 10:31:00 smithi204 ceph-mon[32649]: pgmap v135: 97 pgs: 97 active+clean; 85 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 86 MiB/s rd, 3.9 MiB/s wr, 461 op/s 2024-04-23T10:31:00.492 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 23 10:31:00 smithi119 ceph-mon[32608]: pgmap v135: 97 pgs: 97 active+clean; 85 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 86 MiB/s rd, 3.9 MiB/s wr, 461 op/s 2024-04-23T10:31:00.504 INFO:journalctl@ceph.mon.a.smithi114.stdout:Apr 23 10:31:00 smithi114 ceph-mon[29255]: pgmap v135: 97 pgs: 97 active+clean; 85 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 86 MiB/s rd, 3.9 MiB/s wr, 461 op/s 2024-04-23T10:31:00.917 INFO:tasks.workunit.client.0.smithi114.stdout: 1 278397 7.83 MB/sec execute 69 sec latency 867.856 ms 2024-04-23T10:31:01.917 INFO:tasks.workunit.client.0.smithi114.stdout: 1 278397 7.72 MB/sec execute 70 sec latency 1867.930 ms 2024-04-23T10:31:02.491 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 23 10:31:02 smithi119 ceph-mon[32608]: pgmap v136: 97 pgs: 97 active+clean; 85 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 86 MiB/s rd, 3.9 MiB/s wr, 461 op/s 2024-04-23T10:31:02.504 INFO:journalctl@ceph.mon.a.smithi114.stdout:Apr 23 10:31:02 smithi114 ceph-mon[29255]: pgmap v136: 97 pgs: 97 active+clean; 85 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 86 MiB/s rd, 3.9 MiB/s wr, 461 op/s 2024-04-23T10:31:02.578 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 23 10:31:02 smithi204 ceph-mon[32649]: pgmap v136: 97 pgs: 97 active+clean; 85 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 86 MiB/s rd, 3.9 MiB/s wr, 461 op/s 2024-04-23T10:31:02.917 INFO:tasks.workunit.client.0.smithi114.stdout: 1 279349 7.66 MB/sec execute 71 sec latency 2397.212 ms 2024-04-23T10:31:03.917 INFO:tasks.workunit.client.0.smithi114.stdout: 1 280673 7.69 MB/sec execute 72 sec latency 209.485 ms 2024-04-23T10:31:04.492 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 23 10:31:04 smithi119 ceph-mon[32608]: pgmap v137: 97 pgs: 97 active+clean; 82 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 103 MiB/s rd, 2.6 MiB/s wr, 401 op/s 2024-04-23T10:31:04.504 INFO:journalctl@ceph.mon.a.smithi114.stdout:Apr 23 10:31:04 smithi114 ceph-mon[29255]: pgmap v137: 97 pgs: 97 active+clean; 82 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 103 MiB/s rd, 2.6 MiB/s wr, 401 op/s 2024-04-23T10:31:04.577 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 23 10:31:04 smithi204 ceph-mon[32649]: pgmap v137: 97 pgs: 97 active+clean; 82 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 103 MiB/s rd, 2.6 MiB/s wr, 401 op/s 2024-04-23T10:31:04.917 INFO:tasks.workunit.client.0.smithi114.stdout: 1 282637 7.68 MB/sec execute 73 sec latency 14.217 ms 2024-04-23T10:31:05.917 INFO:tasks.workunit.client.0.smithi114.stdout: 1 284336 7.74 MB/sec execute 74 sec latency 11.760 ms 2024-04-23T10:31:05.929 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:31:05.928+0000 7f18ca605640 1 -- 172.21.15.114:0/4262499412 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f18c410c520 msgr2=0x7f18c410e9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T10:31:05.929 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:31:05.928+0000 7f18ca605640 1 --2- 172.21.15.114:0/4262499412 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f18c410c520 0x7f18c410e9c0 secure :-1 s=READY pgs=642 cs=0 l=1 rev1=1 crypto rx=0x7f18bc009ca0 tx=0x7f18bc02a9e0 comp rx=0 tx=0).stop 2024-04-23T10:31:05.929 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:31:05.928+0000 7f18ca605640 1 -- 172.21.15.114:0/4262499412 shutdown_connections 2024-04-23T10:31:05.929 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:31:05.928+0000 7f18ca605640 1 --2- 172.21.15.114:0/4262499412 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f18c410c520 0x7f18c410e9c0 unknown :-1 s=CLOSED pgs=642 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:31:05.929 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:31:05.928+0000 7f18ca605640 1 --2- 172.21.15.114:0/4262499412 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f18c4103b00 0x7f18c410bfe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:31:05.929 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:31:05.928+0000 7f18ca605640 1 --2- 172.21.15.114:0/4262499412 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f18c41031e0 0x7f18c41035c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:31:05.929 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:31:05.928+0000 7f18ca605640 1 -- 172.21.15.114:0/4262499412 >> 172.21.15.114:0/4262499412 conn(0x7f18c4077fd0 msgr2=0x7f18c40783e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T10:31:05.929 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:31:05.928+0000 7f18ca605640 1 -- 172.21.15.114:0/4262499412 shutdown_connections 2024-04-23T10:31:05.929 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:31:05.928+0000 7f18ca605640 1 -- 172.21.15.114:0/4262499412 wait complete. 2024-04-23T10:31:05.929 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:31:05.929+0000 7f18ca605640 1 Processor -- start 2024-04-23T10:31:05.929 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:31:05.929+0000 7f18ca605640 1 -- start start 2024-04-23T10:31:05.931 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:31:05.929+0000 7f18ca605640 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f18c41031e0 0x7f18c41985d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T10:31:05.931 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:31:05.929+0000 7f18ca605640 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f18c4103b00 0x7f18c4198b10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T10:31:05.931 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:31:05.929+0000 7f18ca605640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f18c410c520 0x7f18c419ce90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T10:31:05.931 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:31:05.929+0000 7f18ca605640 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x7f18c4112190 con 0x7f18c410c520 2024-04-23T10:31:05.931 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:31:05.929+0000 7f18ca605640 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 -- 0x7f18c4112010 con 0x7f18c41031e0 2024-04-23T10:31:05.931 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:31:05.929+0000 7f18ca605640 1 -- --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_getmap magic: 0 -- 0x7f18c4112310 con 0x7f18c4103b00 2024-04-23T10:31:05.931 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:31:05.929+0000 7f18c8ba2640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f18c410c520 0x7f18c419ce90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T10:31:05.932 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:31:05.929+0000 7f18c8ba2640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f18c410c520 0x7f18c419ce90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.114:57146/0 (socket says 172.21.15.114:57146) 2024-04-23T10:31:05.932 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:31:05.929+0000 7f18c8ba2640 1 -- 172.21.15.114:0/2729800531 learned_addr learned my addr 172.21.15.114:0/2729800531 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-04-23T10:31:05.932 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:31:05.929+0000 7f18c3fff640 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f18c41031e0 0x7f18c41985d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T10:31:05.932 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:31:05.929+0000 7f18c37fe640 1 --2- 172.21.15.114:0/2729800531 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f18c4103b00 0x7f18c4198b10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T10:31:05.932 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:31:05.930+0000 7f18c8ba2640 1 -- 172.21.15.114:0/2729800531 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f18c4103b00 msgr2=0x7f18c4198b10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T10:31:05.932 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:31:05.930+0000 7f18c8ba2640 1 --2- 172.21.15.114:0/2729800531 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f18c4103b00 0x7f18c4198b10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:31:05.932 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:31:05.930+0000 7f18c8ba2640 1 -- 172.21.15.114:0/2729800531 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f18c41031e0 msgr2=0x7f18c41985d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T10:31:05.932 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:31:05.930+0000 7f18c8ba2640 1 --2- 172.21.15.114:0/2729800531 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f18c41031e0 0x7f18c41985d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:31:05.932 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:31:05.930+0000 7f18c8ba2640 1 -- 172.21.15.114:0/2729800531 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f18c419d5c0 con 0x7f18c410c520 2024-04-23T10:31:05.932 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:31:05.930+0000 7f18c8ba2640 1 --2- 172.21.15.114:0/2729800531 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f18c410c520 0x7f18c419ce90 secure :-1 s=READY pgs=643 cs=0 l=1 rev1=1 crypto rx=0x7f18bc006010 tx=0x7f18bc02fbe0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T10:31:05.932 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:31:05.930+0000 7f18c17fa640 1 -- 172.21.15.114:0/2729800531 <== mon.0 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f18bc033040 con 0x7f18c410c520 2024-04-23T10:31:05.932 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:31:05.930+0000 7f18c17fa640 1 -- 172.21.15.114:0/2729800531 <== mon.0 v2:172.21.15.114:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f18bc02e7a0 con 0x7f18c410c520 2024-04-23T10:31:05.932 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:31:05.930+0000 7f18ca605640 1 -- 172.21.15.114:0/2729800531 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f18c419d850 con 0x7f18c410c520 2024-04-23T10:31:05.932 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:31:05.930+0000 7f18c17fa640 1 -- 172.21.15.114:0/2729800531 <== mon.0 v2:172.21.15.114:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f18bc043630 con 0x7f18c410c520 2024-04-23T10:31:05.932 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:31:05.930+0000 7f18ca605640 1 -- 172.21.15.114:0/2729800531 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f18c41a5140 con 0x7f18c410c520 2024-04-23T10:31:05.939 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:31:05.932+0000 7f18c17fa640 1 -- 172.21.15.114:0/2729800531 <== mon.0 v2:172.21.15.114:3300/0 4 ==== mgrmap(e 26) ==== 138778+0+0 (secure 0 0 0) 0x7f18bc036070 con 0x7f18c410c520 2024-04-23T10:31:05.939 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:31:05.933+0000 7f18cd3f8740 1 Processor -- start 2024-04-23T10:31:05.939 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:31:05.933+0000 7f18cd3f8740 1 -- start start 2024-04-23T10:31:05.939 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:31:05.933+0000 7f18c17fa640 1 --2- 172.21.15.114:0/2729800531 >> [v2:172.21.15.114:6800/1395693466,v1:172.21.15.114:6801/1395693466] conn(0x7f18ac0a3dc0 0x7f18ac0a6280 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T10:31:05.939 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:31:05.933+0000 7f18cd3f8740 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55da05674620 0x55da05674a00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T10:31:05.939 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:31:05.933+0000 7f18cd3f8740 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x55da05674f40 0x55da0567f2c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T10:31:05.939 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:31:05.933+0000 7f18cd3f8740 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x55da0567f800 0x55da05681bf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T10:31:05.939 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:31:05.933+0000 7f18cd3f8740 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x55da055d9970 con 0x55da05674620 2024-04-23T10:31:05.940 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:31:05.933+0000 7f18cd3f8740 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 -- 0x55da055764a0 con 0x55da05674f40 2024-04-23T10:31:05.940 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:31:05.933+0000 7f18cd3f8740 1 -- --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_getmap magic: 0 -- 0x55da055d5930 con 0x55da0567f800 2024-04-23T10:31:05.940 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:31:05.933+0000 7f18c37fe640 1 --2- 172.21.15.114:0/2729800531 >> [v2:172.21.15.114:6800/1395693466,v1:172.21.15.114:6801/1395693466] conn(0x7f18ac0a3dc0 0x7f18ac0a6280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T10:31:05.940 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:31:05.933+0000 7f18c17fa640 1 -- 172.21.15.114:0/2729800531 <== mon.0 v2:172.21.15.114:3300/0 5 ==== osd_map(93..93 src has 1..93) ==== 8585+0+0 (secure 0 0 0) 0x7f18bc0f1500 con 0x7f18c410c520 2024-04-23T10:31:05.940 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:31:05.933+0000 7f18c3fff640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55da05674620 0x55da05674a00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T10:31:05.940 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:31:05.933+0000 7f18c37fe640 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x55da05674f40 0x55da0567f2c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T10:31:05.940 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:31:05.933+0000 7f18c3fff640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55da05674620 0x55da05674a00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.114:57160/0 (socket says 172.21.15.114:57160) 2024-04-23T10:31:05.940 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:31:05.933+0000 7f18c37fe640 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x55da05674f40 0x55da0567f2c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.119:3300/0 says I am v2:172.21.15.114:59298/0 (socket says 172.21.15.114:59298) 2024-04-23T10:31:05.940 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:31:05.933+0000 7f18c3fff640 1 -- 172.21.15.114:0/1672949176 learned_addr learned my addr 172.21.15.114:0/1672949176 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-04-23T10:31:05.940 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:31:05.933+0000 7f18c8ba2640 1 --2- 172.21.15.114:0/1672949176 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x55da0567f800 0x55da05681bf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T10:31:05.940 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:31:05.933+0000 7f18c3fff640 1 -- 172.21.15.114:0/1672949176 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x55da0567f800 msgr2=0x55da05681bf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T10:31:05.940 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:31:05.933+0000 7f18c3fff640 1 --2- 172.21.15.114:0/1672949176 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x55da0567f800 0x55da05681bf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:31:05.940 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:31:05.933+0000 7f18c3fff640 1 -- 172.21.15.114:0/1672949176 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x55da05674f40 msgr2=0x55da0567f2c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T10:31:05.940 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:31:05.933+0000 7f18c3fff640 1 --2- 172.21.15.114:0/1672949176 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x55da05674f40 0x55da0567f2c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:31:05.940 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:31:05.933+0000 7f18c3fff640 1 -- 172.21.15.114:0/1672949176 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55da052ec210 con 0x55da05674620 2024-04-23T10:31:05.941 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:31:05.933+0000 7f18c37fe640 1 --2- 172.21.15.114:0/1672949176 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x55da05674f40 0x55da0567f2c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-23T10:31:05.941 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:31:05.933+0000 7f18c37fe640 1 --2- 172.21.15.114:0/2729800531 >> [v2:172.21.15.114:6800/1395693466,v1:172.21.15.114:6801/1395693466] conn(0x7f18ac0a3dc0 0x7f18ac0a6280 secure :-1 s=READY pgs=98 cs=0 l=1 rev1=1 crypto rx=0x7f18b800afd0 tx=0x7f18b800ab20 comp rx=0 tx=0).ready entity=mgr.15291 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T10:31:05.941 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:31:05.934+0000 7f18c3fff640 1 --2- 172.21.15.114:0/1672949176 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55da05674620 0x55da05674a00 secure :-1 s=READY pgs=644 cs=0 l=1 rev1=1 crypto rx=0x7f18b4009cf0 tx=0x7f18b4011810 comp rx=0 tx=0).ready entity=mon.0 client_cookie=805594e363ef81cc server_cookie=0 in_seq=0 out_seq=0 2024-04-23T10:31:05.941 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:31:05.934+0000 7f18aa7fc640 1 -- 172.21.15.114:0/1672949176 <== mon.0 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f18b4009de0 con 0x55da05674620 2024-04-23T10:31:05.941 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:31:05.934+0000 7f18aa7fc640 1 -- 172.21.15.114:0/1672949176 <== mon.0 v2:172.21.15.114:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f18b4005020 con 0x55da05674620 2024-04-23T10:31:05.941 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:31:05.934+0000 7f18aa7fc640 1 -- 172.21.15.114:0/1672949176 <== mon.0 v2:172.21.15.114:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f18b4004b00 con 0x55da05674620 2024-04-23T10:31:05.941 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:31:05.934+0000 7f18cd3f8740 1 -- 172.21.15.114:0/1672949176 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55da05674620 msgr2=0x55da05674a00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T10:31:05.941 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:31:05.934+0000 7f18cd3f8740 1 --2- 172.21.15.114:0/1672949176 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55da05674620 0x55da05674a00 secure :-1 s=READY pgs=644 cs=0 l=1 rev1=1 crypto rx=0x7f18b4009cf0 tx=0x7f18b4011810 comp rx=0 tx=0).stop 2024-04-23T10:31:05.941 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:31:05.934+0000 7f18cd3f8740 1 -- 172.21.15.114:0/1672949176 shutdown_connections 2024-04-23T10:31:05.941 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:31:05.934+0000 7f18cd3f8740 1 --2- 172.21.15.114:0/1672949176 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x55da0567f800 0x55da05681bf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:31:05.941 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:31:05.934+0000 7f18cd3f8740 1 --2- 172.21.15.114:0/1672949176 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x55da05674f40 0x55da0567f2c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:31:05.941 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:31:05.934+0000 7f18cd3f8740 1 --2- 172.21.15.114:0/1672949176 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55da05674620 0x55da05674a00 unknown :-1 s=CLOSED pgs=644 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:31:05.941 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:31:05.934+0000 7f18cd3f8740 1 -- 172.21.15.114:0/1672949176 >> 172.21.15.114:0/1672949176 conn(0x55da05670b30 msgr2=0x55da05670f10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T10:31:05.941 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:31:05.934+0000 7f18cd3f8740 1 -- 172.21.15.114:0/1672949176 shutdown_connections 2024-04-23T10:31:05.941 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:31:05.934+0000 7f18cd3f8740 1 -- 172.21.15.114:0/1672949176 wait complete. 2024-04-23T10:31:05.942 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:31:05.935+0000 7f18cd3f8740 1 -- start start 2024-04-23T10:31:05.942 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:31:05.935+0000 7f18cd3f8740 1 Processor -- start 2024-04-23T10:31:05.942 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:31:05.935+0000 7f18cd3f8740 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x55da05674620 0x55da0567f110 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T10:31:05.942 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:31:05.935+0000 7f18cd3f8740 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x55da05674f40 0x55da0567c400 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T10:31:05.942 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:31:05.935+0000 7f18cd3f8740 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55da0567f800 0x55da0567c940 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T10:31:05.942 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:31:05.935+0000 7f18cd3f8740 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x55da05682430 con 0x55da0567f800 2024-04-23T10:31:05.942 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:31:05.935+0000 7f18cd3f8740 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 -- 0x55da055740b0 con 0x55da05674f40 2024-04-23T10:31:05.942 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:31:05.935+0000 7f18cd3f8740 1 -- --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_getmap magic: 0 -- 0x55da056825b0 con 0x55da05674620 2024-04-23T10:31:05.942 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:31:05.935+0000 7f18cd3f8740 10 client.0 _ll_register_callbacks cb 0x55da05561d30 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-23T10:31:05.942 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:31:05.935+0000 7f18c8ba2640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55da0567f800 0x55da0567c940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T10:31:05.942 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:31:05.935+0000 7f18c37fe640 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x55da05674f40 0x55da0567c400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T10:31:05.942 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:31:05.935+0000 7f18c8ba2640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55da0567f800 0x55da0567c940 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.114:57162/0 (socket says 172.21.15.114:57162) 2024-04-23T10:31:05.942 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:31:05.935+0000 7f18c3fff640 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x55da05674620 0x55da0567f110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T10:31:05.942 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:31:05.935+0000 7f18c8ba2640 1 -- 172.21.15.114:0/3990364127 learned_addr learned my addr 172.21.15.114:0/3990364127 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-04-23T10:31:05.942 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:31:05.936+0000 7f18c8ba2640 1 -- 172.21.15.114:0/3990364127 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x55da05674620 msgr2=0x55da0567f110 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T10:31:05.943 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:31:05.936+0000 7f18c8ba2640 1 --2- 172.21.15.114:0/3990364127 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x55da05674620 0x55da0567f110 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:31:05.943 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:31:05.936+0000 7f18c8ba2640 1 -- 172.21.15.114:0/3990364127 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x55da05674f40 msgr2=0x55da0567c400 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T10:31:05.943 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:31:05.936+0000 7f18c8ba2640 1 --2- 172.21.15.114:0/3990364127 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x55da05674f40 0x55da0567c400 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:31:05.943 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:31:05.936+0000 7f18c8ba2640 1 -- 172.21.15.114:0/3990364127 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55da053b8620 con 0x55da0567f800 2024-04-23T10:31:05.943 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:31:05.936+0000 7f18c3fff640 1 --2- 172.21.15.114:0/3990364127 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x55da05674620 0x55da0567f110 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-23T10:31:05.943 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:31:05.936+0000 7f18c37fe640 1 --2- 172.21.15.114:0/3990364127 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x55da05674f40 0x55da0567c400 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-23T10:31:05.943 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:31:05.936+0000 7f18c8ba2640 1 --2- 172.21.15.114:0/3990364127 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55da0567f800 0x55da0567c940 secure :-1 s=READY pgs=645 cs=0 l=1 rev1=1 crypto rx=0x7f18bc0ef210 tx=0x7f18bc0bcfa0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2021ef08a7211637 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T10:31:05.943 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:31:05.936+0000 7f18a37fe640 10 client.0 ms_handle_connect on v2:172.21.15.114:3300/0 2024-04-23T10:31:05.943 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:31:05.936+0000 7f18a37fe640 1 -- 172.21.15.114:0/3990364127 <== mon.0 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f18bc0452b0 con 0x55da0567f800 2024-04-23T10:31:05.943 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:31:05.936+0000 7f18a37fe640 1 -- 172.21.15.114:0/3990364127 <== mon.0 v2:172.21.15.114:3300/0 2 ==== osd_map(93..93 src has 1..93) ==== 8585+0+0 (secure 0 0 0) 0x7f18bc0b9ad0 con 0x55da0567f800 2024-04-23T10:31:05.943 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:31:05.936+0000 7f18cd3f8740 1 -- 172.21.15.114:0/3990364127 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f18b4005020 con 0x55da0567f800 2024-04-23T10:31:05.943 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:31:05.936+0000 7f18a37fe640 1 -- 172.21.15.114:0/3990364127 <== mon.0 v2:172.21.15.114:3300/0 3 ==== mon_get_version_reply(handle=1 version=28) ==== 24+0+0 (secure 0 0 0) 0x7f18bc0450b0 con 0x55da0567f800 2024-04-23T10:31:05.943 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:31:05.936+0000 7f18cd3f8740 10 client.16110 fetch_fsmap learned FSMap version 28 2024-04-23T10:31:05.943 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:31:05.936+0000 7f18cd3f8740 1 -- 172.21.15.114:0/3990364127 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({config=0+,fsmap=28,monmap=4+}) -- 0x55da05682430 con 0x55da0567f800 2024-04-23T10:31:05.944 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:31:05.936+0000 7f18a37fe640 1 -- 172.21.15.114:0/3990364127 <== mon.0 v2:172.21.15.114:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f18b4005020 con 0x55da0567f800 2024-04-23T10:31:05.944 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:31:05.936+0000 7f18a37fe640 1 -- 172.21.15.114:0/3990364127 <== mon.0 v2:172.21.15.114:3300/0 5 ==== fsmap(e 28) ==== 8338+0+0 (secure 0 0 0) 0x7f18bc0820a0 con 0x55da0567f800 2024-04-23T10:31:05.944 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:31:05.936+0000 7f18cd3f8740 10 client.16110 fetch_fsmap finished waiting for FSMap version 28 2024-04-23T10:31:05.944 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:31:05.936+0000 7f18cd3f8740 10 client.16110 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-04-23T10:31:05.944 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:31:05.936+0000 7f18cd3f8740 20 client.16110 populate_metadata read hostname 'smithi114' 2024-04-23T10:31:05.944 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:31:05.936+0000 7f18cd3f8740 1 --2- 172.21.15.114:0/3990364127 >> [v2:172.21.15.119:6838/937297340,v1:172.21.15.119:6839/937297340] conn(0x55da0568f760 0x55da05691b90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T10:31:05.944 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:31:05.936+0000 7f18cd3f8740 4 client.16110 mds_command: new command op to 24545 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-23T10:31:05.944 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:31:05.936+0000 7f18cd3f8740 1 -- 172.21.15.114:0/3990364127 --> [v2:172.21.15.119:6838/937297340,v1:172.21.15.119:6839/937297340] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55da055e7390 con 0x55da0568f760 2024-04-23T10:31:05.944 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:31:05.937+0000 7f18c3fff640 1 --2- 172.21.15.114:0/3990364127 >> [v2:172.21.15.119:6838/937297340,v1:172.21.15.119:6839/937297340] conn(0x55da0568f760 0x55da05691b90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T10:31:05.944 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:31:05.937+0000 7f18c3fff640 1 --2- 172.21.15.114:0/3990364127 >> [v2:172.21.15.119:6838/937297340,v1:172.21.15.119:6839/937297340] conn(0x55da0568f760 0x55da05691b90 crc :-1 s=READY pgs=94 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=73293500600a5204 server_cookie=e255ea5a63c58650 in_seq=0 out_seq=0 2024-04-23T10:31:05.944 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:31:05.937+0000 7f18a37fe640 10 client.16110 ms_handle_connect on v2:172.21.15.119:6838/937297340 2024-04-23T10:31:05.944 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:31:05.938+0000 7f18a37fe640 1 -- 172.21.15.114:0/3990364127 <== mds.0 v2:172.21.15.119:6838/937297340 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x55da055e7390 con 0x55da0568f760 2024-04-23T10:31:05.944 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:31:05.938+0000 7f18a37fe640 10 client.16110 handle_command_reply: tid=0 2024-04-23T10:31:05.944 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:31:05.938+0000 7f18cd3f8740 1 client.16110 shutdown 2024-04-23T10:31:05.944 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:31:05.938+0000 7f18cd3f8740 1 -- 172.21.15.114:0/3990364127 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55da0567f800 msgr2=0x55da0567c940 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T10:31:05.945 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:31:05.938+0000 7f18cd3f8740 1 --2- 172.21.15.114:0/3990364127 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55da0567f800 0x55da0567c940 secure :-1 s=READY pgs=645 cs=0 l=1 rev1=1 crypto rx=0x7f18bc0ef210 tx=0x7f18bc0bcfa0 comp rx=0 tx=0).stop 2024-04-23T10:31:05.945 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:31:05.938+0000 7f18cd3f8740 1 -- 172.21.15.114:0/3990364127 shutdown_connections 2024-04-23T10:31:05.945 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:31:05.938+0000 7f18cd3f8740 1 --2- 172.21.15.114:0/3990364127 >> [v2:172.21.15.119:6838/937297340,v1:172.21.15.119:6839/937297340] conn(0x55da0568f760 0x55da05691b90 crc :-1 s=READY pgs=94 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:31:05.945 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:31:05.938+0000 7f18cd3f8740 1 --2- 172.21.15.114:0/3990364127 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55da0567f800 0x55da0567c940 unknown :-1 s=CLOSED pgs=645 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:31:05.945 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:31:05.938+0000 7f18cd3f8740 1 --2- 172.21.15.114:0/3990364127 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x55da05674f40 0x55da0567c400 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:31:05.945 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:31:05.938+0000 7f18a37fe640 0 client.16110 ms_handle_reset on v2:172.21.15.119:6838/937297340 2024-04-23T10:31:05.945 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:31:05.938+0000 7f18cd3f8740 1 --2- 172.21.15.114:0/3990364127 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x55da05674620 0x55da0567f110 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:31:05.945 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:31:05.938+0000 7f18cd3f8740 1 -- 172.21.15.114:0/3990364127 >> 172.21.15.114:0/3990364127 conn(0x55da05670b30 msgr2=0x55da056768b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T10:31:05.945 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:31:05.938+0000 7f18cd3f8740 1 -- 172.21.15.114:0/3990364127 shutdown_connections 2024-04-23T10:31:05.945 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:31:05.938+0000 7f18cd3f8740 1 -- 172.21.15.114:0/3990364127 wait complete. 2024-04-23T10:31:05.945 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:31:05.938+0000 7f18cd3f8740 20 client.16110 trim_cache size 0 max 16384 2024-04-23T10:31:05.957 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:31:05.953+0000 7f18cd3f8740 1 Processor -- start 2024-04-23T10:31:05.958 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:31:05.953+0000 7f18cd3f8740 1 -- start start 2024-04-23T10:31:05.958 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:31:05.953+0000 7f18cd3f8740 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x55da05670b30 0x55da0568b2f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T10:31:05.958 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:31:05.953+0000 7f18cd3f8740 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55da05674620 0x55da05697470 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T10:31:05.958 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:31:05.953+0000 7f18cd3f8740 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x55da05674f40 0x55da056979b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T10:31:05.958 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:31:05.953+0000 7f18cd3f8740 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x7f18c41b0530 con 0x55da05674620 2024-04-23T10:31:05.958 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:31:05.953+0000 7f18cd3f8740 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 -- 0x7f18c41ade00 con 0x55da05670b30 2024-04-23T10:31:05.958 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:31:05.953+0000 7f18cd3f8740 1 -- --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_getmap magic: 0 -- 0x7f18c41abc50 con 0x55da05674f40 2024-04-23T10:31:05.958 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:31:05.954+0000 7f18c37fe640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55da05674620 0x55da05697470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T10:31:05.958 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:31:05.954+0000 7f18c37fe640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55da05674620 0x55da05697470 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.114:57174/0 (socket says 172.21.15.114:57174) 2024-04-23T10:31:05.958 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:31:05.954+0000 7f18c37fe640 1 -- 172.21.15.114:0/1744819539 learned_addr learned my addr 172.21.15.114:0/1744819539 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-04-23T10:31:05.958 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:31:05.954+0000 7f18c3fff640 1 --2- 172.21.15.114:0/1744819539 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x55da05670b30 0x55da0568b2f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T10:31:05.958 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:31:05.954+0000 7f18c8ba2640 1 --2- 172.21.15.114:0/1744819539 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x55da05674f40 0x55da056979b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T10:31:05.958 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:31:05.954+0000 7f18c37fe640 1 -- 172.21.15.114:0/1744819539 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x55da05674f40 msgr2=0x55da056979b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T10:31:05.958 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:31:05.954+0000 7f18c37fe640 1 --2- 172.21.15.114:0/1744819539 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x55da05674f40 0x55da056979b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:31:05.959 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:31:05.954+0000 7f18c37fe640 1 -- 172.21.15.114:0/1744819539 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x55da05670b30 msgr2=0x55da0568b2f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T10:31:05.959 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:31:05.954+0000 7f18c37fe640 1 --2- 172.21.15.114:0/1744819539 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x55da05670b30 0x55da0568b2f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:31:05.959 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:31:05.954+0000 7f18c37fe640 1 -- 172.21.15.114:0/1744819539 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55da05674d30 con 0x55da05674620 2024-04-23T10:31:05.959 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:31:05.954+0000 7f18c37fe640 1 --2- 172.21.15.114:0/1744819539 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55da05674620 0x55da05697470 secure :-1 s=READY pgs=646 cs=0 l=1 rev1=1 crypto rx=0x7f18c410e0e0 tx=0x7f18b801f000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c610f94c707d9145 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T10:31:05.959 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:31:05.954+0000 7f18a9ffb640 1 -- 172.21.15.114:0/1744819539 <== mon.0 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f18b8011e40 con 0x55da05674620 2024-04-23T10:31:05.959 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:31:05.954+0000 7f18a9ffb640 1 -- 172.21.15.114:0/1744819539 <== mon.0 v2:172.21.15.114:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f18b8012e00 con 0x55da05674620 2024-04-23T10:31:05.959 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:31:05.954+0000 7f18a9ffb640 1 -- 172.21.15.114:0/1744819539 <== mon.0 v2:172.21.15.114:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f18b802a970 con 0x55da05674620 2024-04-23T10:31:05.959 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:31:05.955+0000 7f18cd3f8740 1 -- 172.21.15.114:0/1744819539 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55da05674620 msgr2=0x55da05697470 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T10:31:05.959 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:31:05.955+0000 7f18cd3f8740 1 --2- 172.21.15.114:0/1744819539 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55da05674620 0x55da05697470 secure :-1 s=READY pgs=646 cs=0 l=1 rev1=1 crypto rx=0x7f18c410e0e0 tx=0x7f18b801f000 comp rx=0 tx=0).stop 2024-04-23T10:31:05.959 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:31:05.955+0000 7f18cd3f8740 1 -- 172.21.15.114:0/1744819539 shutdown_connections 2024-04-23T10:31:05.959 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:31:05.955+0000 7f18cd3f8740 1 --2- 172.21.15.114:0/1744819539 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x55da05674f40 0x55da056979b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:31:05.959 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:31:05.955+0000 7f18cd3f8740 1 --2- 172.21.15.114:0/1744819539 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55da05674620 0x55da05697470 unknown :-1 s=CLOSED pgs=646 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:31:05.959 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:31:05.955+0000 7f18cd3f8740 1 --2- 172.21.15.114:0/1744819539 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x55da05670b30 0x55da0568b2f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:31:05.959 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:31:05.955+0000 7f18cd3f8740 1 -- 172.21.15.114:0/1744819539 >> 172.21.15.114:0/1744819539 conn(0x55da0567caa0 msgr2=0x55da0566f420 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T10:31:05.959 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:31:05.955+0000 7f18cd3f8740 1 -- 172.21.15.114:0/1744819539 shutdown_connections 2024-04-23T10:31:05.960 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:31:05.955+0000 7f18cd3f8740 1 -- 172.21.15.114:0/1744819539 wait complete. 2024-04-23T10:31:05.960 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:31:05.955+0000 7f18cd3f8740 1 -- start start 2024-04-23T10:31:05.960 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:31:05.955+0000 7f18cd3f8740 1 Processor -- start 2024-04-23T10:31:05.960 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:31:05.955+0000 7f18cd3f8740 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55da05670b30 0x55da05685170 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T10:31:05.960 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:31:05.955+0000 7f18cd3f8740 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x55da05674620 0x55da056856b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T10:31:05.960 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:31:05.955+0000 7f18cd3f8740 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x55da05674f40 0x55da05685bf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T10:31:05.960 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:31:05.956+0000 7f18cd3f8740 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x7f18c41ac2c0 con 0x55da05670b30 2024-04-23T10:31:05.960 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:31:05.956+0000 7f18cd3f8740 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 -- 0x7f18c41ad730 con 0x55da05674f40 2024-04-23T10:31:05.960 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:31:05.956+0000 7f18cd3f8740 1 -- --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_getmap magic: 0 -- 0x7f18c41ac990 con 0x55da05674620 2024-04-23T10:31:05.960 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:31:05.956+0000 7f18cd3f8740 10 client.0 _ll_register_callbacks cb 0x7f18bc0b97d0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-23T10:31:05.960 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:31:05.956+0000 7f18c3fff640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55da05670b30 0x55da05685170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T10:31:05.960 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:31:05.956+0000 7f18c3fff640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55da05670b30 0x55da05685170 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.114:57190/0 (socket says 172.21.15.114:57190) 2024-04-23T10:31:05.960 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:31:05.956+0000 7f18c3fff640 1 -- 172.21.15.114:0/3541301670 learned_addr learned my addr 172.21.15.114:0/3541301670 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-04-23T10:31:05.960 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:31:05.956+0000 7f18c8ba2640 1 --2- 172.21.15.114:0/3541301670 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x55da05674f40 0x55da05685bf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T10:31:05.961 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:31:05.956+0000 7f18c37fe640 1 --2- 172.21.15.114:0/3541301670 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x55da05674620 0x55da056856b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T10:31:05.961 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:31:05.956+0000 7f18c3fff640 1 -- 172.21.15.114:0/3541301670 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x55da05674620 msgr2=0x55da056856b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T10:31:05.961 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:31:05.956+0000 7f18c3fff640 1 --2- 172.21.15.114:0/3541301670 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x55da05674620 0x55da056856b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:31:05.961 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:31:05.956+0000 7f18c3fff640 1 -- 172.21.15.114:0/3541301670 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x55da05674f40 msgr2=0x55da05685bf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T10:31:05.961 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:31:05.956+0000 7f18c3fff640 1 --2- 172.21.15.114:0/3541301670 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x55da05674f40 0x55da05685bf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:31:05.961 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:31:05.956+0000 7f18c3fff640 1 -- 172.21.15.114:0/3541301670 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55da056f4cf0 con 0x55da05670b30 2024-04-23T10:31:05.961 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:31:05.956+0000 7f18c8ba2640 1 --2- 172.21.15.114:0/3541301670 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x55da05674f40 0x55da05685bf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-23T10:31:05.961 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:31:05.956+0000 7f18c3fff640 1 --2- 172.21.15.114:0/3541301670 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55da05670b30 0x55da05685170 secure :-1 s=READY pgs=647 cs=0 l=1 rev1=1 crypto rx=0x7f18b4005580 tx=0x7f18b4005f90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d697e720c3321f26 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T10:31:05.961 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:31:05.956+0000 7f18a37fe640 10 client.0 ms_handle_connect on v2:172.21.15.114:3300/0 2024-04-23T10:31:05.961 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:31:05.956+0000 7f18a37fe640 1 -- 172.21.15.114:0/3541301670 <== mon.0 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f18b402ae00 con 0x55da05670b30 2024-04-23T10:31:05.961 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:31:05.956+0000 7f18cd3f8740 1 -- 172.21.15.114:0/3541301670 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f18b8012e00 con 0x55da05670b30 2024-04-23T10:31:05.961 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:31:05.956+0000 7f18a37fe640 1 -- 172.21.15.114:0/3541301670 <== mon.0 v2:172.21.15.114:3300/0 2 ==== osd_map(93..93 src has 1..93) ==== 8585+0+0 (secure 0 0 0) 0x7f18b402a4d0 con 0x55da05670b30 2024-04-23T10:31:05.961 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:31:05.956+0000 7f18a37fe640 1 -- 172.21.15.114:0/3541301670 <== mon.0 v2:172.21.15.114:3300/0 3 ==== mon_get_version_reply(handle=1 version=28) ==== 24+0+0 (secure 0 0 0) 0x7f18b402a6f0 con 0x55da05670b30 2024-04-23T10:31:05.961 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:31:05.956+0000 7f18cd3f8740 10 client.16116 fetch_fsmap learned FSMap version 28 2024-04-23T10:31:05.961 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:31:05.956+0000 7f18cd3f8740 1 -- 172.21.15.114:0/3541301670 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({config=0+,fsmap=28,monmap=4+}) -- 0x55da056f5f20 con 0x55da05670b30 2024-04-23T10:31:05.961 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:31:05.957+0000 7f18a37fe640 1 -- 172.21.15.114:0/3541301670 <== mon.0 v2:172.21.15.114:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f18b8012e00 con 0x55da05670b30 2024-04-23T10:31:05.962 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:31:05.957+0000 7f18a37fe640 1 -- 172.21.15.114:0/3541301670 <== mon.0 v2:172.21.15.114:3300/0 5 ==== fsmap(e 28) ==== 8338+0+0 (secure 0 0 0) 0x7f18b4004e30 con 0x55da05670b30 2024-04-23T10:31:05.962 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:31:05.957+0000 7f18cd3f8740 10 client.16116 fetch_fsmap finished waiting for FSMap version 28 2024-04-23T10:31:05.962 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:31:05.957+0000 7f18cd3f8740 10 client.16116 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-04-23T10:31:05.962 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:31:05.957+0000 7f18cd3f8740 20 client.16116 populate_metadata read hostname 'smithi114' 2024-04-23T10:31:05.962 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:31:05.957+0000 7f18cd3f8740 1 --2- 172.21.15.114:0/3541301670 >> [v2:172.21.15.119:6838/937297340,v1:172.21.15.119:6839/937297340] conn(0x55da0567f800 0x55da056f8520 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T10:31:05.962 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:31:05.957+0000 7f18cd3f8740 4 client.16116 mds_command: new command op to 24545 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-04-23T10:31:05.962 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:31:05.957+0000 7f18cd3f8740 1 -- 172.21.15.114:0/3541301670 --> [v2:172.21.15.119:6838/937297340,v1:172.21.15.119:6839/937297340] -- command(tid 0: {"prefix": "flush journal"}) -- 0x55da055e1200 con 0x55da0567f800 2024-04-23T10:31:05.962 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:31:05.957+0000 7f18c37fe640 1 --2- 172.21.15.114:0/3541301670 >> [v2:172.21.15.119:6838/937297340,v1:172.21.15.119:6839/937297340] conn(0x55da0567f800 0x55da056f8520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T10:31:05.962 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:31:05.957+0000 7f18a37fe640 10 client.16116 ms_handle_connect on v2:172.21.15.119:6838/937297340 2024-04-23T10:31:05.962 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:31:05.957+0000 7f18c37fe640 1 --2- 172.21.15.114:0/3541301670 >> [v2:172.21.15.119:6838/937297340,v1:172.21.15.119:6839/937297340] conn(0x55da0567f800 0x55da056f8520 crc :-1 s=READY pgs=95 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=4af88db25c5ca099 server_cookie=19958d33ec7cc8ed in_seq=0 out_seq=0 2024-04-23T10:31:06.024 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:31:06.023+0000 7f18a37fe640 1 -- 172.21.15.114:0/3541301670 <== mds.0 v2:172.21.15.119:6838/937297340 1 ==== command_reply(tid 0: 0 ) ==== 8+0+44 (crc 0 0 0) 0x55da055e1200 con 0x55da0567f800 2024-04-23T10:31:06.024 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:31:06.023+0000 7f18a37fe640 10 client.16116 handle_command_reply: tid=0 2024-04-23T10:31:06.024 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:31:06.023+0000 7f18cd3f8740 1 client.16116 shutdown 2024-04-23T10:31:06.024 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:31:06.024+0000 7f18cd3f8740 1 -- 172.21.15.114:0/3541301670 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55da05670b30 msgr2=0x55da05685170 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T10:31:06.024 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:31:06.024+0000 7f18cd3f8740 1 --2- 172.21.15.114:0/3541301670 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55da05670b30 0x55da05685170 secure :-1 s=READY pgs=647 cs=0 l=1 rev1=1 crypto rx=0x7f18b4005580 tx=0x7f18b4005f90 comp rx=0 tx=0).stop 2024-04-23T10:31:06.024 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:31:06.024+0000 7f18cd3f8740 1 -- 172.21.15.114:0/3541301670 shutdown_connections 2024-04-23T10:31:06.024 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:31:06.024+0000 7f18cd3f8740 1 --2- 172.21.15.114:0/3541301670 >> [v2:172.21.15.119:6838/937297340,v1:172.21.15.119:6839/937297340] conn(0x55da0567f800 0x55da056f8520 crc :-1 s=READY pgs=95 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:31:06.024 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:31:06.024+0000 7f18cd3f8740 1 --2- 172.21.15.114:0/3541301670 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x55da05674f40 0x55da05685bf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:31:06.024 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:31:06.024+0000 7f18cd3f8740 1 --2- 172.21.15.114:0/3541301670 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x55da05674620 0x55da056856b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:31:06.024 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:31:06.024+0000 7f18cd3f8740 1 --2- 172.21.15.114:0/3541301670 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55da05670b30 0x55da05685170 unknown :-1 s=CLOSED pgs=647 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:31:06.025 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:31:06.024+0000 7f18a37fe640 0 client.16116 ms_handle_reset on v2:172.21.15.119:6838/937297340 2024-04-23T10:31:06.025 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:31:06.024+0000 7f18cd3f8740 1 -- 172.21.15.114:0/3541301670 >> 172.21.15.114:0/3541301670 conn(0x55da0567caa0 msgr2=0x55da0566fcb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T10:31:06.025 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:31:06.024+0000 7f18cd3f8740 1 -- 172.21.15.114:0/3541301670 shutdown_connections 2024-04-23T10:31:06.025 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:31:06.024+0000 7f18cd3f8740 1 -- 172.21.15.114:0/3541301670 wait complete. 2024-04-23T10:31:06.025 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:31:06.024+0000 7f18cd3f8740 20 client.16116 trim_cache size 0 max 16384 2024-04-23T10:31:06.025 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stdout:{ 2024-04-23T10:31:06.025 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stdout: "message": "", 2024-04-23T10:31:06.025 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stdout: "return_code": 0 2024-04-23T10:31:06.025 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stdout:} 2024-04-23T10:31:06.026 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:31:06.024+0000 7f18ca605640 1 -- 172.21.15.114:0/2729800531 >> [v2:172.21.15.114:6800/1395693466,v1:172.21.15.114:6801/1395693466] conn(0x7f18ac0a3dc0 msgr2=0x7f18ac0a6280 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T10:31:06.026 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:31:06.024+0000 7f18ca605640 1 --2- 172.21.15.114:0/2729800531 >> [v2:172.21.15.114:6800/1395693466,v1:172.21.15.114:6801/1395693466] conn(0x7f18ac0a3dc0 0x7f18ac0a6280 secure :-1 s=READY pgs=98 cs=0 l=1 rev1=1 crypto rx=0x7f18b800afd0 tx=0x7f18b800ab20 comp rx=0 tx=0).stop 2024-04-23T10:31:06.026 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:31:06.024+0000 7f18ca605640 1 -- 172.21.15.114:0/2729800531 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f18c410c520 msgr2=0x7f18c419ce90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T10:31:06.026 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:31:06.024+0000 7f18ca605640 1 --2- 172.21.15.114:0/2729800531 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f18c410c520 0x7f18c419ce90 secure :-1 s=READY pgs=643 cs=0 l=1 rev1=1 crypto rx=0x7f18bc006010 tx=0x7f18bc02fbe0 comp rx=0 tx=0).stop 2024-04-23T10:31:06.026 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:31:06.025+0000 7f18ca605640 1 -- 172.21.15.114:0/2729800531 shutdown_connections 2024-04-23T10:31:06.026 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:31:06.025+0000 7f18ca605640 1 --2- 172.21.15.114:0/2729800531 >> [v2:172.21.15.114:6800/1395693466,v1:172.21.15.114:6801/1395693466] conn(0x7f18ac0a3dc0 0x7f18ac0a6280 unknown :-1 s=CLOSED pgs=98 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:31:06.027 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:31:06.025+0000 7f18ca605640 1 --2- 172.21.15.114:0/2729800531 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f18c410c520 0x7f18c419ce90 unknown :-1 s=CLOSED pgs=643 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:31:06.027 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:31:06.025+0000 7f18ca605640 1 --2- 172.21.15.114:0/2729800531 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f18c4103b00 0x7f18c4198b10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:31:06.027 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:31:06.025+0000 7f18ca605640 1 --2- 172.21.15.114:0/2729800531 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f18c41031e0 0x7f18c41985d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:31:06.027 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:31:06.025+0000 7f18ca605640 1 -- 172.21.15.114:0/2729800531 >> 172.21.15.114:0/2729800531 conn(0x7f18c4077fd0 msgr2=0x7f18c40fde40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T10:31:06.027 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:31:06.025+0000 7f18ca605640 1 -- 172.21.15.114:0/2729800531 shutdown_connections 2024-04-23T10:31:06.027 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:31:06.025+0000 7f18ca605640 1 -- 172.21.15.114:0/2729800531 wait complete. 2024-04-23T10:31:06.491 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 23 10:31:06 smithi119 ceph-mon[32608]: pgmap v138: 97 pgs: 97 active+clean; 82 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 100 MiB/s rd, 2.5 MiB/s wr, 390 op/s 2024-04-23T10:31:06.503 INFO:journalctl@ceph.mon.a.smithi114.stdout:Apr 23 10:31:06 smithi114 ceph-mon[29255]: pgmap v138: 97 pgs: 97 active+clean; 82 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 100 MiB/s rd, 2.5 MiB/s wr, 390 op/s 2024-04-23T10:31:06.577 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 23 10:31:06 smithi204 ceph-mon[32649]: pgmap v138: 97 pgs: 97 active+clean; 82 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 100 MiB/s rd, 2.5 MiB/s wr, 390 op/s 2024-04-23T10:31:06.917 INFO:tasks.workunit.client.0.smithi114.stdout: 1 286215 7.74 MB/sec execute 75 sec latency 45.759 ms 2024-04-23T10:31:07.917 INFO:tasks.workunit.client.0.smithi114.stdout: 1 287863 7.79 MB/sec execute 76 sec latency 17.017 ms 2024-04-23T10:31:08.491 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 23 10:31:08 smithi119 ceph-mon[32608]: pgmap v139: 97 pgs: 97 active+clean; 83 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 103 MiB/s rd, 3.8 MiB/s wr, 521 op/s 2024-04-23T10:31:08.504 INFO:journalctl@ceph.mon.a.smithi114.stdout:Apr 23 10:31:08 smithi114 ceph-mon[29255]: pgmap v139: 97 pgs: 97 active+clean; 83 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 103 MiB/s rd, 3.8 MiB/s wr, 521 op/s 2024-04-23T10:31:08.577 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 23 10:31:08 smithi204 ceph-mon[32649]: pgmap v139: 97 pgs: 97 active+clean; 83 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 103 MiB/s rd, 3.8 MiB/s wr, 521 op/s 2024-04-23T10:31:08.917 INFO:tasks.workunit.client.0.smithi114.stdout: 1 289936 7.79 MB/sec execute 77 sec latency 38.358 ms 2024-04-23T10:31:09.918 INFO:tasks.workunit.client.0.smithi114.stdout: 1 291467 7.84 MB/sec execute 78 sec latency 20.972 ms 2024-04-23T10:31:10.491 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 23 10:31:10 smithi119 ceph-mon[32608]: pgmap v140: 97 pgs: 97 active+clean; 83 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 43 MiB/s rd, 2.4 MiB/s wr, 311 op/s 2024-04-23T10:31:10.503 INFO:journalctl@ceph.mon.a.smithi114.stdout:Apr 23 10:31:10 smithi114 ceph-mon[29255]: pgmap v140: 97 pgs: 97 active+clean; 83 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 43 MiB/s rd, 2.4 MiB/s wr, 311 op/s 2024-04-23T10:31:10.577 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 23 10:31:10 smithi204 ceph-mon[32649]: pgmap v140: 97 pgs: 97 active+clean; 83 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 43 MiB/s rd, 2.4 MiB/s wr, 311 op/s 2024-04-23T10:31:10.917 INFO:tasks.workunit.client.0.smithi114.stdout: 1 293798 7.90 MB/sec execute 79 sec latency 24.528 ms 2024-04-23T10:31:11.917 INFO:tasks.workunit.client.0.smithi114.stdout: 1 295042 7.88 MB/sec execute 80 sec latency 40.519 ms 2024-04-23T10:31:12.492 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 23 10:31:12 smithi119 ceph-mon[32608]: pgmap v141: 97 pgs: 97 active+clean; 83 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 43 MiB/s rd, 2.4 MiB/s wr, 311 op/s 2024-04-23T10:31:12.504 INFO:journalctl@ceph.mon.a.smithi114.stdout:Apr 23 10:31:12 smithi114 ceph-mon[29255]: pgmap v141: 97 pgs: 97 active+clean; 83 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 43 MiB/s rd, 2.4 MiB/s wr, 311 op/s 2024-04-23T10:31:12.577 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 23 10:31:12 smithi204 ceph-mon[32649]: pgmap v141: 97 pgs: 97 active+clean; 83 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 43 MiB/s rd, 2.4 MiB/s wr, 311 op/s 2024-04-23T10:31:12.917 INFO:tasks.workunit.client.0.smithi114.stdout: 1 297453 7.94 MB/sec execute 81 sec latency 13.721 ms 2024-04-23T10:31:13.917 INFO:tasks.workunit.client.0.smithi114.stdout: 1 298758 7.93 MB/sec execute 82 sec latency 70.280 ms 2024-04-23T10:31:14.491 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 23 10:31:14 smithi119 ceph-mon[32608]: pgmap v142: 97 pgs: 97 active+clean; 89 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 53 MiB/s rd, 4.2 MiB/s wr, 492 op/s 2024-04-23T10:31:14.503 INFO:journalctl@ceph.mon.a.smithi114.stdout:Apr 23 10:31:14 smithi114 ceph-mon[29255]: pgmap v142: 97 pgs: 97 active+clean; 89 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 53 MiB/s rd, 4.2 MiB/s wr, 492 op/s 2024-04-23T10:31:14.578 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 23 10:31:14 smithi204 ceph-mon[32649]: pgmap v142: 97 pgs: 97 active+clean; 89 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 53 MiB/s rd, 4.2 MiB/s wr, 492 op/s 2024-04-23T10:31:14.918 INFO:tasks.workunit.client.0.smithi114.stdout: 1 300630 7.94 MB/sec execute 83 sec latency 125.438 ms 2024-04-23T10:31:15.918 INFO:tasks.workunit.client.0.smithi114.stdout: 1 302201 7.96 MB/sec execute 84 sec latency 133.741 ms 2024-04-23T10:31:16.491 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 23 10:31:16 smithi119 ceph-mon[32608]: pgmap v143: 97 pgs: 97 active+clean; 89 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 27 MiB/s rd, 3.4 MiB/s wr, 367 op/s 2024-04-23T10:31:16.504 INFO:journalctl@ceph.mon.a.smithi114.stdout:Apr 23 10:31:16 smithi114 ceph-mon[29255]: pgmap v143: 97 pgs: 97 active+clean; 89 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 27 MiB/s rd, 3.4 MiB/s wr, 367 op/s 2024-04-23T10:31:16.577 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 23 10:31:16 smithi204 ceph-mon[32649]: pgmap v143: 97 pgs: 97 active+clean; 89 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 27 MiB/s rd, 3.4 MiB/s wr, 367 op/s 2024-04-23T10:31:16.918 INFO:tasks.workunit.client.0.smithi114.stdout: 1 304534 8.01 MB/sec execute 85 sec latency 31.805 ms 2024-04-23T10:31:17.918 INFO:tasks.workunit.client.0.smithi114.stdout: 1 305661 8.00 MB/sec execute 86 sec latency 142.096 ms 2024-04-23T10:31:18.493 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 23 10:31:18 smithi119 ceph-mon[32608]: pgmap v144: 97 pgs: 97 active+clean; 94 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 40 MiB/s rd, 5.3 MiB/s wr, 535 op/s 2024-04-23T10:31:18.504 INFO:journalctl@ceph.mon.a.smithi114.stdout:Apr 23 10:31:18 smithi114 ceph-mon[29255]: pgmap v144: 97 pgs: 97 active+clean; 94 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 40 MiB/s rd, 5.3 MiB/s wr, 535 op/s 2024-04-23T10:31:18.577 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 23 10:31:18 smithi204 ceph-mon[32649]: pgmap v144: 97 pgs: 97 active+clean; 94 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 40 MiB/s rd, 5.3 MiB/s wr, 535 op/s 2024-04-23T10:31:18.918 INFO:tasks.workunit.client.0.smithi114.stdout: 1 305661 7.91 MB/sec execute 87 sec latency 1142.177 ms 2024-04-23T10:31:19.123 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:31:19.122+0000 7f69fffe2640 1 -- 172.21.15.114:0/4075097243 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f69f81010e0 msgr2=0x7f69f81014c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T10:31:19.123 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:31:19.122+0000 7f69fffe2640 1 --2- 172.21.15.114:0/4075097243 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f69f81010e0 0x7f69f81014c0 secure :-1 s=READY pgs=648 cs=0 l=1 rev1=1 crypto rx=0x7f69f4009ca0 tx=0x7f69f402a8c0 comp rx=0 tx=0).stop 2024-04-23T10:31:19.124 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:31:19.122+0000 7f69fffe2640 1 -- 172.21.15.114:0/4075097243 shutdown_connections 2024-04-23T10:31:19.124 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:31:19.122+0000 7f69fffe2640 1 --2- 172.21.15.114:0/4075097243 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f69f81061b0 0x7f69f810dd80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:31:19.124 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:31:19.122+0000 7f69fffe2640 1 --2- 172.21.15.114:0/4075097243 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f69f8101a90 0x7f69f8105ad0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:31:19.124 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:31:19.122+0000 7f69fffe2640 1 --2- 172.21.15.114:0/4075097243 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f69f81010e0 0x7f69f81014c0 unknown :-1 s=CLOSED pgs=648 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:31:19.124 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:31:19.122+0000 7f69fffe2640 1 -- 172.21.15.114:0/4075097243 >> 172.21.15.114:0/4075097243 conn(0x7f69f80771d0 msgr2=0x7f69f80775e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T10:31:19.124 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:31:19.122+0000 7f69fffe2640 1 -- 172.21.15.114:0/4075097243 shutdown_connections 2024-04-23T10:31:19.124 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:31:19.122+0000 7f69fffe2640 1 -- 172.21.15.114:0/4075097243 wait complete. 2024-04-23T10:31:19.124 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:31:19.123+0000 7f69fffe2640 1 Processor -- start 2024-04-23T10:31:19.124 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:31:19.123+0000 7f69fffe2640 1 -- start start 2024-04-23T10:31:19.124 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:31:19.123+0000 7f69fffe2640 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f69f81010e0 0x7f69f81986d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T10:31:19.124 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:31:19.123+0000 7f69fffe2640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f69f81061b0 0x7f69f8198c10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T10:31:19.124 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:31:19.123+0000 7f69fffe2640 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f69f819cf40 0x7f69f819d3f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T10:31:19.124 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:31:19.123+0000 7f69fffe2640 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x7f69f8112230 con 0x7f69f81061b0 2024-04-23T10:31:19.124 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:31:19.123+0000 7f69fffe2640 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 -- 0x7f69f81120b0 con 0x7f69f819cf40 2024-04-23T10:31:19.125 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:31:19.123+0000 7f69fffe2640 1 -- --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_getmap magic: 0 -- 0x7f69f81123b0 con 0x7f69f81010e0 2024-04-23T10:31:19.125 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:31:19.123+0000 7f69fd57d640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f69f81061b0 0x7f69f8198c10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T10:31:19.125 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:31:19.123+0000 7f69fd57d640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f69f81061b0 0x7f69f8198c10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.114:36944/0 (socket says 172.21.15.114:36944) 2024-04-23T10:31:19.125 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:31:19.123+0000 7f69fd57d640 1 -- 172.21.15.114:0/446651473 learned_addr learned my addr 172.21.15.114:0/446651473 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-04-23T10:31:19.125 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:31:19.123+0000 7f69fdd7e640 1 --2- 172.21.15.114:0/446651473 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f69f81010e0 0x7f69f81986d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T10:31:19.125 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:31:19.123+0000 7f69fe57f640 1 --2- 172.21.15.114:0/446651473 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f69f819cf40 0x7f69f819d3f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T10:31:19.125 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:31:19.123+0000 7f69fd57d640 1 -- 172.21.15.114:0/446651473 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f69f81010e0 msgr2=0x7f69f81986d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T10:31:19.125 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:31:19.123+0000 7f69fd57d640 1 --2- 172.21.15.114:0/446651473 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f69f81010e0 0x7f69f81986d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:31:19.125 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:31:19.123+0000 7f69fd57d640 1 -- 172.21.15.114:0/446651473 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f69f819cf40 msgr2=0x7f69f819d3f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T10:31:19.125 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:31:19.124+0000 7f69fd57d640 1 --2- 172.21.15.114:0/446651473 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f69f819cf40 0x7f69f819d3f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:31:19.125 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:31:19.124+0000 7f69fd57d640 1 -- 172.21.15.114:0/446651473 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f69f819da30 con 0x7f69f81061b0 2024-04-23T10:31:19.125 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:31:19.124+0000 7f69fdd7e640 1 --2- 172.21.15.114:0/446651473 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f69f81010e0 0x7f69f81986d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-23T10:31:19.125 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:31:19.124+0000 7f69fe57f640 1 --2- 172.21.15.114:0/446651473 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f69f819cf40 0x7f69f819d3f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-23T10:31:19.125 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:31:19.124+0000 7f69fd57d640 1 --2- 172.21.15.114:0/446651473 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f69f81061b0 0x7f69f8198c10 secure :-1 s=READY pgs=649 cs=0 l=1 rev1=1 crypto rx=0x7f69e800c970 tx=0x7f69e800ce30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T10:31:19.126 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:31:19.124+0000 7f69eeffd640 1 -- 172.21.15.114:0/446651473 <== mon.0 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f69e8014070 con 0x7f69f81061b0 2024-04-23T10:31:19.126 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:31:19.124+0000 7f69eeffd640 1 -- 172.21.15.114:0/446651473 <== mon.0 v2:172.21.15.114:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f69e8004440 con 0x7f69f81061b0 2024-04-23T10:31:19.126 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:31:19.124+0000 7f69fffe2640 1 -- 172.21.15.114:0/446651473 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f69f819dd20 con 0x7f69f81061b0 2024-04-23T10:31:19.126 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:31:19.124+0000 7f69eeffd640 1 -- 172.21.15.114:0/446651473 <== mon.0 v2:172.21.15.114:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f69e8003e50 con 0x7f69f81061b0 2024-04-23T10:31:19.126 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:31:19.124+0000 7f69fffe2640 1 -- 172.21.15.114:0/446651473 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f69f81a5560 con 0x7f69f81061b0 2024-04-23T10:31:19.132 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:31:19.126+0000 7f69eeffd640 1 -- 172.21.15.114:0/446651473 <== mon.0 v2:172.21.15.114:3300/0 4 ==== mgrmap(e 26) ==== 138778+0+0 (secure 0 0 0) 0x7f69e8017070 con 0x7f69f81061b0 2024-04-23T10:31:19.132 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:31:19.127+0000 7f6a02dd1740 1 Processor -- start 2024-04-23T10:31:19.132 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:31:19.127+0000 7f6a02dd1740 1 -- start start 2024-04-23T10:31:19.133 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:31:19.127+0000 7f69eeffd640 1 --2- 172.21.15.114:0/446651473 >> [v2:172.21.15.114:6800/1395693466,v1:172.21.15.114:6801/1395693466] conn(0x7f69e40a3fe0 0x7f69e40a64a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T10:31:19.133 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:31:19.127+0000 7f6a02dd1740 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x55989b6ad620 0x55989b6ada00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T10:31:19.133 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:31:19.127+0000 7f6a02dd1740 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55989b6adf40 0x55989b6b82c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T10:31:19.133 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:31:19.127+0000 7f6a02dd1740 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x55989b6b8800 0x55989b6babf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T10:31:19.133 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:31:19.127+0000 7f6a02dd1740 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x55989b612970 con 0x55989b6adf40 2024-04-23T10:31:19.133 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:31:19.127+0000 7f6a02dd1740 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 -- 0x55989b5af4a0 con 0x55989b6ad620 2024-04-23T10:31:19.133 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:31:19.127+0000 7f6a02dd1740 1 -- --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_getmap magic: 0 -- 0x55989b60e930 con 0x55989b6b8800 2024-04-23T10:31:19.133 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:31:19.127+0000 7f69fe57f640 1 --2- 172.21.15.114:0/446651473 >> [v2:172.21.15.114:6800/1395693466,v1:172.21.15.114:6801/1395693466] conn(0x7f69e40a3fe0 0x7f69e40a64a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T10:31:19.133 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:31:19.127+0000 7f69fd57d640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55989b6adf40 0x55989b6b82c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T10:31:19.133 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:31:19.127+0000 7f69eeffd640 1 -- 172.21.15.114:0/446651473 <== mon.0 v2:172.21.15.114:3300/0 5 ==== osd_map(93..93 src has 1..93) ==== 8585+0+0 (secure 0 0 0) 0x7f69e80d03d0 con 0x7f69f81061b0 2024-04-23T10:31:19.133 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:31:19.127+0000 7f69fd57d640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55989b6adf40 0x55989b6b82c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.114:36948/0 (socket says 172.21.15.114:36948) 2024-04-23T10:31:19.133 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:31:19.127+0000 7f69fd57d640 1 -- 172.21.15.114:0/2899245882 learned_addr learned my addr 172.21.15.114:0/2899245882 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-04-23T10:31:19.133 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:31:19.127+0000 7f69fdd7e640 1 --2- 172.21.15.114:0/2899245882 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x55989b6ad620 0x55989b6ada00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T10:31:19.133 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:31:19.127+0000 7f69fe57f640 1 --2- 172.21.15.114:0/2899245882 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x55989b6b8800 0x55989b6babf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T10:31:19.134 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:31:19.128+0000 7f69fd57d640 1 -- 172.21.15.114:0/2899245882 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x55989b6b8800 msgr2=0x55989b6babf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T10:31:19.134 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:31:19.128+0000 7f69fd57d640 1 --2- 172.21.15.114:0/2899245882 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x55989b6b8800 0x55989b6babf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:31:19.134 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:31:19.128+0000 7f69fd57d640 1 -- 172.21.15.114:0/2899245882 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x55989b6ad620 msgr2=0x55989b6ada00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T10:31:19.134 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:31:19.128+0000 7f69fd57d640 1 --2- 172.21.15.114:0/2899245882 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x55989b6ad620 0x55989b6ada00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:31:19.134 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:31:19.128+0000 7f69fd57d640 1 -- 172.21.15.114:0/2899245882 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55989b325210 con 0x55989b6adf40 2024-04-23T10:31:19.134 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:31:19.128+0000 7f69fe57f640 1 --2- 172.21.15.114:0/2899245882 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x55989b6b8800 0x55989b6babf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-23T10:31:19.134 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:31:19.128+0000 7f69fe57f640 1 --2- 172.21.15.114:0/446651473 >> [v2:172.21.15.114:6800/1395693466,v1:172.21.15.114:6801/1395693466] conn(0x7f69e40a3fe0 0x7f69e40a64a0 secure :-1 s=READY pgs=99 cs=0 l=1 rev1=1 crypto rx=0x55989b6aaf50 tx=0x7f69f0005d10 comp rx=0 tx=0).ready entity=mgr.15291 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T10:31:19.134 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:31:19.128+0000 7f69fd57d640 1 --2- 172.21.15.114:0/2899245882 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55989b6adf40 0x55989b6b82c0 secure :-1 s=READY pgs=650 cs=0 l=1 rev1=1 crypto rx=0x7f69e800c940 tx=0x7f69e809bfa0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d97f1887e84e1ee6 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T10:31:19.134 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:31:19.128+0000 7f69e3fff640 1 -- 172.21.15.114:0/2899245882 <== mon.0 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f69e809ca30 con 0x55989b6adf40 2024-04-23T10:31:19.134 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:31:19.128+0000 7f69e3fff640 1 -- 172.21.15.114:0/2899245882 <== mon.0 v2:172.21.15.114:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f69e809c330 con 0x55989b6adf40 2024-04-23T10:31:19.134 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:31:19.128+0000 7f69e3fff640 1 -- 172.21.15.114:0/2899245882 <== mon.0 v2:172.21.15.114:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f69e80989a0 con 0x55989b6adf40 2024-04-23T10:31:19.134 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:31:19.129+0000 7f6a02dd1740 1 -- 172.21.15.114:0/2899245882 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55989b6adf40 msgr2=0x55989b6b82c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T10:31:19.134 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:31:19.129+0000 7f6a02dd1740 1 --2- 172.21.15.114:0/2899245882 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55989b6adf40 0x55989b6b82c0 secure :-1 s=READY pgs=650 cs=0 l=1 rev1=1 crypto rx=0x7f69e800c940 tx=0x7f69e809bfa0 comp rx=0 tx=0).stop 2024-04-23T10:31:19.134 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:31:19.129+0000 7f6a02dd1740 1 -- 172.21.15.114:0/2899245882 shutdown_connections 2024-04-23T10:31:19.135 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:31:19.129+0000 7f6a02dd1740 1 --2- 172.21.15.114:0/2899245882 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x55989b6b8800 0x55989b6babf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:31:19.135 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:31:19.129+0000 7f6a02dd1740 1 --2- 172.21.15.114:0/2899245882 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55989b6adf40 0x55989b6b82c0 unknown :-1 s=CLOSED pgs=650 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:31:19.135 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:31:19.129+0000 7f6a02dd1740 1 --2- 172.21.15.114:0/2899245882 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x55989b6ad620 0x55989b6ada00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:31:19.135 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:31:19.129+0000 7f6a02dd1740 1 -- 172.21.15.114:0/2899245882 >> 172.21.15.114:0/2899245882 conn(0x55989b6a9b30 msgr2=0x55989b6a9f10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T10:31:19.135 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:31:19.129+0000 7f6a02dd1740 1 -- 172.21.15.114:0/2899245882 shutdown_connections 2024-04-23T10:31:19.135 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:31:19.129+0000 7f6a02dd1740 1 -- 172.21.15.114:0/2899245882 wait complete. 2024-04-23T10:31:19.135 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:31:19.129+0000 7f6a02dd1740 1 -- start start 2024-04-23T10:31:19.135 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:31:19.129+0000 7f6a02dd1740 1 Processor -- start 2024-04-23T10:31:19.135 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:31:19.130+0000 7f6a02dd1740 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55989b6ad620 0x55989b6b8210 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T10:31:19.135 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:31:19.130+0000 7f6a02dd1740 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x55989b6adf40 0x55989b6b5500 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T10:31:19.135 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:31:19.130+0000 7f6a02dd1740 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x55989b6b8800 0x55989b6b5b10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T10:31:19.135 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:31:19.130+0000 7f6a02dd1740 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x55989b6bb430 con 0x55989b6ad620 2024-04-23T10:31:19.135 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:31:19.130+0000 7f6a02dd1740 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 -- 0x55989b5ad0b0 con 0x55989b6adf40 2024-04-23T10:31:19.135 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:31:19.130+0000 7f6a02dd1740 1 -- --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_getmap magic: 0 -- 0x55989b6bb5b0 con 0x55989b6b8800 2024-04-23T10:31:19.135 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:31:19.130+0000 7f6a02dd1740 10 client.0 _ll_register_callbacks cb 0x55989b59ad30 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-23T10:31:19.136 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:31:19.130+0000 7f69fdd7e640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55989b6ad620 0x55989b6b8210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T10:31:19.136 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:31:19.130+0000 7f69fdd7e640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55989b6ad620 0x55989b6b8210 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.114:36954/0 (socket says 172.21.15.114:36954) 2024-04-23T10:31:19.136 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:31:19.130+0000 7f69fdd7e640 1 -- 172.21.15.114:0/1078028275 learned_addr learned my addr 172.21.15.114:0/1078028275 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-04-23T10:31:19.136 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:31:19.130+0000 7f69fd57d640 1 --2- 172.21.15.114:0/1078028275 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x55989b6adf40 0x55989b6b5500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T10:31:19.136 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:31:19.130+0000 7f69fe57f640 1 --2- 172.21.15.114:0/1078028275 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x55989b6b8800 0x55989b6b5b10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T10:31:19.136 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:31:19.130+0000 7f69fdd7e640 1 -- 172.21.15.114:0/1078028275 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x55989b6b8800 msgr2=0x55989b6b5b10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T10:31:19.136 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:31:19.130+0000 7f69fdd7e640 1 --2- 172.21.15.114:0/1078028275 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x55989b6b8800 0x55989b6b5b10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:31:19.136 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:31:19.130+0000 7f69fdd7e640 1 -- 172.21.15.114:0/1078028275 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x55989b6adf40 msgr2=0x55989b6b5500 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T10:31:19.136 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:31:19.130+0000 7f69fdd7e640 1 --2- 172.21.15.114:0/1078028275 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x55989b6adf40 0x55989b6b5500 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:31:19.136 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:31:19.130+0000 7f69fdd7e640 1 -- 172.21.15.114:0/1078028275 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55989b3f1620 con 0x55989b6ad620 2024-04-23T10:31:19.136 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:31:19.130+0000 7f69fe57f640 1 --2- 172.21.15.114:0/1078028275 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x55989b6b8800 0x55989b6b5b10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-23T10:31:19.136 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:31:19.130+0000 7f69fdd7e640 1 --2- 172.21.15.114:0/1078028275 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55989b6ad620 0x55989b6b8210 secure :-1 s=READY pgs=651 cs=0 l=1 rev1=1 crypto rx=0x7f69f4035000 tx=0x7f69f4031680 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5a64e176408a3313 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T10:31:19.136 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:31:19.130+0000 7f69e17fa640 10 client.0 ms_handle_connect on v2:172.21.15.114:3300/0 2024-04-23T10:31:19.136 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:31:19.130+0000 7f69e17fa640 1 -- 172.21.15.114:0/1078028275 <== mon.0 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f69f4003970 con 0x55989b6ad620 2024-04-23T10:31:19.136 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:31:19.131+0000 7f6a02dd1740 1 -- 172.21.15.114:0/1078028275 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f69e809c330 con 0x55989b6ad620 2024-04-23T10:31:19.137 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:31:19.131+0000 7f69e17fa640 1 -- 172.21.15.114:0/1078028275 <== mon.0 v2:172.21.15.114:3300/0 2 ==== osd_map(93..93 src has 1..93) ==== 8585+0+0 (secure 0 0 0) 0x7f69f4003b10 con 0x55989b6ad620 2024-04-23T10:31:19.137 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:31:19.131+0000 7f69e17fa640 1 -- 172.21.15.114:0/1078028275 <== mon.0 v2:172.21.15.114:3300/0 3 ==== mon_get_version_reply(handle=1 version=28) ==== 24+0+0 (secure 0 0 0) 0x7f69f4003d10 con 0x55989b6ad620 2024-04-23T10:31:19.137 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:31:19.131+0000 7f6a02dd1740 10 client.16128 fetch_fsmap learned FSMap version 28 2024-04-23T10:31:19.137 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:31:19.131+0000 7f6a02dd1740 1 -- 172.21.15.114:0/1078028275 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({config=0+,fsmap=28,monmap=4+}) -- 0x55989b6c7420 con 0x55989b6ad620 2024-04-23T10:31:19.137 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:31:19.131+0000 7f69e17fa640 1 -- 172.21.15.114:0/1078028275 <== mon.0 v2:172.21.15.114:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f69e809c330 con 0x55989b6ad620 2024-04-23T10:31:19.137 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:31:19.131+0000 7f69e17fa640 1 -- 172.21.15.114:0/1078028275 <== mon.0 v2:172.21.15.114:3300/0 5 ==== fsmap(e 28) ==== 8338+0+0 (secure 0 0 0) 0x7f69f40045f0 con 0x55989b6ad620 2024-04-23T10:31:19.137 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:31:19.131+0000 7f6a02dd1740 10 client.16128 fetch_fsmap finished waiting for FSMap version 28 2024-04-23T10:31:19.137 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:31:19.131+0000 7f6a02dd1740 10 client.16128 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-04-23T10:31:19.137 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:31:19.131+0000 7f6a02dd1740 20 client.16128 populate_metadata read hostname 'smithi114' 2024-04-23T10:31:19.137 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:31:19.131+0000 7f6a02dd1740 1 --2- 172.21.15.114:0/1078028275 >> [v2:172.21.15.119:6838/937297340,v1:172.21.15.119:6839/937297340] conn(0x55989b6c88e0 0x55989b6cacf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T10:31:19.137 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:31:19.131+0000 7f6a02dd1740 4 client.16128 mds_command: new command op to 24545 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-23T10:31:19.137 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:31:19.131+0000 7f6a02dd1740 1 -- 172.21.15.114:0/1078028275 --> [v2:172.21.15.119:6838/937297340,v1:172.21.15.119:6839/937297340] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55989b620390 con 0x55989b6c88e0 2024-04-23T10:31:19.137 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:31:19.131+0000 7f69fd57d640 1 --2- 172.21.15.114:0/1078028275 >> [v2:172.21.15.119:6838/937297340,v1:172.21.15.119:6839/937297340] conn(0x55989b6c88e0 0x55989b6cacf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T10:31:19.137 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:31:19.132+0000 7f69e17fa640 10 client.16128 ms_handle_connect on v2:172.21.15.119:6838/937297340 2024-04-23T10:31:19.138 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:31:19.132+0000 7f69fd57d640 1 --2- 172.21.15.114:0/1078028275 >> [v2:172.21.15.119:6838/937297340,v1:172.21.15.119:6839/937297340] conn(0x55989b6c88e0 0x55989b6cacf0 crc :-1 s=READY pgs=96 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=6bbee3bed3ba48fa server_cookie=5e8057c17e02c9f7 in_seq=0 out_seq=0 2024-04-23T10:31:19.138 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:31:19.132+0000 7f69e17fa640 1 -- 172.21.15.114:0/1078028275 <== mds.0 v2:172.21.15.119:6838/937297340 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x55989b620390 con 0x55989b6c88e0 2024-04-23T10:31:19.138 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:31:19.132+0000 7f69e17fa640 10 client.16128 handle_command_reply: tid=0 2024-04-23T10:31:19.138 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:31:19.133+0000 7f6a02dd1740 1 client.16128 shutdown 2024-04-23T10:31:19.138 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:31:19.133+0000 7f6a02dd1740 1 -- 172.21.15.114:0/1078028275 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55989b6ad620 msgr2=0x55989b6b8210 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T10:31:19.138 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:31:19.133+0000 7f6a02dd1740 1 --2- 172.21.15.114:0/1078028275 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55989b6ad620 0x55989b6b8210 secure :-1 s=READY pgs=651 cs=0 l=1 rev1=1 crypto rx=0x7f69f4035000 tx=0x7f69f4031680 comp rx=0 tx=0).stop 2024-04-23T10:31:19.138 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:31:19.133+0000 7f6a02dd1740 1 -- 172.21.15.114:0/1078028275 shutdown_connections 2024-04-23T10:31:19.138 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:31:19.133+0000 7f6a02dd1740 1 --2- 172.21.15.114:0/1078028275 >> [v2:172.21.15.119:6838/937297340,v1:172.21.15.119:6839/937297340] conn(0x55989b6c88e0 0x55989b6cacf0 crc :-1 s=READY pgs=96 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:31:19.138 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:31:19.133+0000 7f6a02dd1740 1 --2- 172.21.15.114:0/1078028275 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x55989b6b8800 0x55989b6b5b10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:31:19.138 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:31:19.133+0000 7f6a02dd1740 1 --2- 172.21.15.114:0/1078028275 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x55989b6adf40 0x55989b6b5500 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:31:19.138 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:31:19.133+0000 7f69e17fa640 0 client.16128 ms_handle_reset on v2:172.21.15.119:6838/937297340 2024-04-23T10:31:19.138 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:31:19.133+0000 7f6a02dd1740 1 --2- 172.21.15.114:0/1078028275 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55989b6ad620 0x55989b6b8210 unknown :-1 s=CLOSED pgs=651 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:31:19.138 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:31:19.133+0000 7f6a02dd1740 1 -- 172.21.15.114:0/1078028275 >> 172.21.15.114:0/1078028275 conn(0x55989b6a9b30 msgr2=0x55989b6af8b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T10:31:19.139 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:31:19.133+0000 7f6a02dd1740 1 -- 172.21.15.114:0/1078028275 shutdown_connections 2024-04-23T10:31:19.139 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:31:19.133+0000 7f6a02dd1740 1 -- 172.21.15.114:0/1078028275 wait complete. 2024-04-23T10:31:19.139 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:31:19.133+0000 7f6a02dd1740 20 client.16128 trim_cache size 0 max 16384 2024-04-23T10:31:19.151 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:31:19.148+0000 7f6a02dd1740 1 Processor -- start 2024-04-23T10:31:19.151 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:31:19.148+0000 7f6a02dd1740 1 -- start start 2024-04-23T10:31:19.151 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:31:19.148+0000 7f6a02dd1740 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x55989b6ad620 0x55989b6c42f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T10:31:19.151 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:31:19.148+0000 7f6a02dd1740 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55989b6adf40 0x55989b6a8b30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T10:31:19.152 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:31:19.148+0000 7f6a02dd1740 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x55989b6b8800 0x55989b6d0600 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T10:31:19.152 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:31:19.148+0000 7f6a02dd1740 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x7f69f81b0520 con 0x55989b6adf40 2024-04-23T10:31:19.152 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:31:19.148+0000 7f6a02dd1740 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 -- 0x7f69f81addf0 con 0x55989b6b8800 2024-04-23T10:31:19.152 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:31:19.148+0000 7f6a02dd1740 1 -- --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_getmap magic: 0 -- 0x7f69f81abc40 con 0x55989b6ad620 2024-04-23T10:31:19.152 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:31:19.148+0000 7f69fd57d640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55989b6adf40 0x55989b6a8b30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T10:31:19.152 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:31:19.148+0000 7f69fd57d640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55989b6adf40 0x55989b6a8b30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.114:36970/0 (socket says 172.21.15.114:36970) 2024-04-23T10:31:19.152 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:31:19.148+0000 7f69fd57d640 1 -- 172.21.15.114:0/1621505508 learned_addr learned my addr 172.21.15.114:0/1621505508 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-04-23T10:31:19.152 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:31:19.148+0000 7f69fdd7e640 1 --2- 172.21.15.114:0/1621505508 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x55989b6ad620 0x55989b6c42f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T10:31:19.152 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:31:19.148+0000 7f69fd57d640 1 -- 172.21.15.114:0/1621505508 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x55989b6ad620 msgr2=0x55989b6c42f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T10:31:19.152 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:31:19.148+0000 7f69fd57d640 1 --2- 172.21.15.114:0/1621505508 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x55989b6ad620 0x55989b6c42f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:31:19.152 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:31:19.148+0000 7f69fd57d640 1 -- 172.21.15.114:0/1621505508 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x55989b6b8800 msgr2=0x55989b6d0600 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T10:31:19.152 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:31:19.148+0000 7f69fd57d640 1 --2- 172.21.15.114:0/1621505508 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x55989b6b8800 0x55989b6d0600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:31:19.152 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:31:19.148+0000 7f69fd57d640 1 -- 172.21.15.114:0/1621505508 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55989b6baf60 con 0x55989b6adf40 2024-04-23T10:31:19.152 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:31:19.148+0000 7f69fd57d640 1 --2- 172.21.15.114:0/1621505508 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55989b6adf40 0x55989b6a8b30 secure :-1 s=READY pgs=652 cs=0 l=1 rev1=1 crypto rx=0x7f69e8098790 tx=0x7f69e8000bc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=23a4546f64b5dcf1 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T10:31:19.153 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:31:19.149+0000 7f69e37fe640 1 -- 172.21.15.114:0/1621505508 <== mon.0 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f69e80b4070 con 0x55989b6adf40 2024-04-23T10:31:19.153 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:31:19.149+0000 7f69e37fe640 1 -- 172.21.15.114:0/1621505508 <== mon.0 v2:172.21.15.114:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f69e809c330 con 0x55989b6adf40 2024-04-23T10:31:19.153 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:31:19.149+0000 7f69e37fe640 1 -- 172.21.15.114:0/1621505508 <== mon.0 v2:172.21.15.114:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f69e809a710 con 0x55989b6adf40 2024-04-23T10:31:19.153 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:31:19.149+0000 7f6a02dd1740 1 -- 172.21.15.114:0/1621505508 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55989b6adf40 msgr2=0x55989b6a8b30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T10:31:19.153 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:31:19.149+0000 7f6a02dd1740 1 --2- 172.21.15.114:0/1621505508 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55989b6adf40 0x55989b6a8b30 secure :-1 s=READY pgs=652 cs=0 l=1 rev1=1 crypto rx=0x7f69e8098790 tx=0x7f69e8000bc0 comp rx=0 tx=0).stop 2024-04-23T10:31:19.153 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:31:19.149+0000 7f6a02dd1740 1 -- 172.21.15.114:0/1621505508 shutdown_connections 2024-04-23T10:31:19.153 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:31:19.149+0000 7f6a02dd1740 1 --2- 172.21.15.114:0/1621505508 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x55989b6b8800 0x55989b6d0600 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:31:19.153 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:31:19.149+0000 7f6a02dd1740 1 --2- 172.21.15.114:0/1621505508 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55989b6adf40 0x55989b6a8b30 unknown :-1 s=CLOSED pgs=652 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:31:19.153 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:31:19.149+0000 7f6a02dd1740 1 --2- 172.21.15.114:0/1621505508 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x55989b6ad620 0x55989b6c42f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:31:19.153 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:31:19.149+0000 7f6a02dd1740 1 -- 172.21.15.114:0/1621505508 >> 172.21.15.114:0/1621505508 conn(0x55989b6a9b30 msgr2=0x55989b6a8420 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T10:31:19.153 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:31:19.149+0000 7f6a02dd1740 1 -- 172.21.15.114:0/1621505508 shutdown_connections 2024-04-23T10:31:19.153 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:31:19.149+0000 7f6a02dd1740 1 -- 172.21.15.114:0/1621505508 wait complete. 2024-04-23T10:31:19.153 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:31:19.149+0000 7f6a02dd1740 1 -- start start 2024-04-23T10:31:19.153 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:31:19.150+0000 7f6a02dd1740 1 Processor -- start 2024-04-23T10:31:19.153 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:31:19.150+0000 7f6a02dd1740 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x55989b6ad620 0x55989b6c0c90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T10:31:19.154 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:31:19.150+0000 7f6a02dd1740 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55989b6adf40 0x55989b6be020 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T10:31:19.154 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:31:19.150+0000 7f6a02dd1740 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x55989b6b8800 0x55989b6be630 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T10:31:19.154 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:31:19.150+0000 7f6a02dd1740 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x7f69f81ac2b0 con 0x55989b6adf40 2024-04-23T10:31:19.154 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:31:19.150+0000 7f6a02dd1740 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 -- 0x7f69f81ad720 con 0x55989b6ad620 2024-04-23T10:31:19.154 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:31:19.150+0000 7f6a02dd1740 1 -- --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_getmap magic: 0 -- 0x7f69f81ac980 con 0x55989b6b8800 2024-04-23T10:31:19.154 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:31:19.150+0000 7f6a02dd1740 10 client.0 _ll_register_callbacks cb 0x55989b59ad30 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-23T10:31:19.154 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:31:19.150+0000 7f69fd57d640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55989b6adf40 0x55989b6be020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T10:31:19.154 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:31:19.150+0000 7f69fd57d640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55989b6adf40 0x55989b6be020 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.114:36976/0 (socket says 172.21.15.114:36976) 2024-04-23T10:31:19.154 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:31:19.150+0000 7f69fd57d640 1 -- 172.21.15.114:0/3273030893 learned_addr learned my addr 172.21.15.114:0/3273030893 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-04-23T10:31:19.154 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:31:19.150+0000 7f69fe57f640 1 --2- 172.21.15.114:0/3273030893 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x55989b6b8800 0x55989b6be630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T10:31:19.154 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:31:19.150+0000 7f69fdd7e640 1 --2- 172.21.15.114:0/3273030893 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x55989b6ad620 0x55989b6c0c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T10:31:19.154 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:31:19.150+0000 7f69fd57d640 1 -- 172.21.15.114:0/3273030893 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x55989b6b8800 msgr2=0x55989b6be630 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T10:31:19.154 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:31:19.150+0000 7f69fd57d640 1 --2- 172.21.15.114:0/3273030893 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x55989b6b8800 0x55989b6be630 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:31:19.154 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:31:19.150+0000 7f69fd57d640 1 -- 172.21.15.114:0/3273030893 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x55989b6ad620 msgr2=0x55989b6c0c90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T10:31:19.155 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:31:19.150+0000 7f69fd57d640 1 --2- 172.21.15.114:0/3273030893 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x55989b6ad620 0x55989b6c0c90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:31:19.155 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:31:19.150+0000 7f69fd57d640 1 -- 172.21.15.114:0/3273030893 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55989b6bee30 con 0x55989b6adf40 2024-04-23T10:31:19.155 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:31:19.151+0000 7f69fd57d640 1 --2- 172.21.15.114:0/3273030893 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55989b6adf40 0x55989b6be020 secure :-1 s=READY pgs=653 cs=0 l=1 rev1=1 crypto rx=0x7f69e800c940 tx=0x7f69e80035f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4ab7329557d872f4 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T10:31:19.155 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:31:19.151+0000 7f69e17fa640 10 client.0 ms_handle_connect on v2:172.21.15.114:3300/0 2024-04-23T10:31:19.155 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:31:19.151+0000 7f69e17fa640 1 -- 172.21.15.114:0/3273030893 <== mon.0 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f69e80b4070 con 0x55989b6adf40 2024-04-23T10:31:19.155 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:31:19.151+0000 7f69e17fa640 1 -- 172.21.15.114:0/3273030893 <== mon.0 v2:172.21.15.114:3300/0 2 ==== osd_map(93..93 src has 1..93) ==== 8585+0+0 (secure 0 0 0) 0x7f69e80989a0 con 0x55989b6adf40 2024-04-23T10:31:19.155 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:31:19.151+0000 7f6a02dd1740 1 -- 172.21.15.114:0/3273030893 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f69e809c330 con 0x55989b6adf40 2024-04-23T10:31:19.155 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:31:19.151+0000 7f69e17fa640 1 -- 172.21.15.114:0/3273030893 <== mon.0 v2:172.21.15.114:3300/0 3 ==== mon_get_version_reply(handle=1 version=28) ==== 24+0+0 (secure 0 0 0) 0x7f69e8051350 con 0x55989b6adf40 2024-04-23T10:31:19.155 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:31:19.151+0000 7f6a02dd1740 10 client.16134 fetch_fsmap learned FSMap version 28 2024-04-23T10:31:19.155 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:31:19.151+0000 7f6a02dd1740 1 -- 172.21.15.114:0/3273030893 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({config=0+,fsmap=28,monmap=4+}) -- 0x55989b72ef60 con 0x55989b6adf40 2024-04-23T10:31:19.155 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:31:19.151+0000 7f69e17fa640 1 -- 172.21.15.114:0/3273030893 <== mon.0 v2:172.21.15.114:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f69e809c330 con 0x55989b6adf40 2024-04-23T10:31:19.155 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:31:19.151+0000 7f69e17fa640 1 -- 172.21.15.114:0/3273030893 <== mon.0 v2:172.21.15.114:3300/0 5 ==== fsmap(e 28) ==== 8338+0+0 (secure 0 0 0) 0x7f69e8098080 con 0x55989b6adf40 2024-04-23T10:31:19.155 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:31:19.151+0000 7f6a02dd1740 10 client.16134 fetch_fsmap finished waiting for FSMap version 28 2024-04-23T10:31:19.155 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:31:19.151+0000 7f6a02dd1740 10 client.16134 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-04-23T10:31:19.155 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:31:19.151+0000 7f6a02dd1740 20 client.16134 populate_metadata read hostname 'smithi114' 2024-04-23T10:31:19.156 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:31:19.151+0000 7f6a02dd1740 1 --2- 172.21.15.114:0/3273030893 >> [v2:172.21.15.119:6838/937297340,v1:172.21.15.119:6839/937297340] conn(0x55989b72f180 0x55989b731570 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T10:31:19.156 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:31:19.151+0000 7f6a02dd1740 4 client.16134 mds_command: new command op to 24545 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-04-23T10:31:19.156 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:31:19.151+0000 7f6a02dd1740 1 -- 172.21.15.114:0/3273030893 --> [v2:172.21.15.119:6838/937297340,v1:172.21.15.119:6839/937297340] -- command(tid 0: {"prefix": "flush journal"}) -- 0x55989b61a200 con 0x55989b72f180 2024-04-23T10:31:19.156 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:31:19.152+0000 7f69fdd7e640 1 --2- 172.21.15.114:0/3273030893 >> [v2:172.21.15.119:6838/937297340,v1:172.21.15.119:6839/937297340] conn(0x55989b72f180 0x55989b731570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T10:31:19.156 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:31:19.152+0000 7f69e17fa640 10 client.16134 ms_handle_connect on v2:172.21.15.119:6838/937297340 2024-04-23T10:31:19.156 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:31:19.152+0000 7f69fdd7e640 1 --2- 172.21.15.114:0/3273030893 >> [v2:172.21.15.119:6838/937297340,v1:172.21.15.119:6839/937297340] conn(0x55989b72f180 0x55989b731570 crc :-1 s=READY pgs=97 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=870f48a3dc99833 server_cookie=e9ea1b9d69d1561a in_seq=0 out_seq=0 2024-04-23T10:31:19.224 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:31:19.224+0000 7f69e17fa640 1 -- 172.21.15.114:0/3273030893 <== mds.0 v2:172.21.15.119:6838/937297340 1 ==== command_reply(tid 0: 0 ) ==== 8+0+44 (crc 0 0 0) 0x55989b61a200 con 0x55989b72f180 2024-04-23T10:31:19.224 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:31:19.224+0000 7f69e17fa640 10 client.16134 handle_command_reply: tid=0 2024-04-23T10:31:19.224 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:31:19.224+0000 7f6a02dd1740 1 client.16134 shutdown 2024-04-23T10:31:19.224 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:31:19.224+0000 7f6a02dd1740 1 -- 172.21.15.114:0/3273030893 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55989b6adf40 msgr2=0x55989b6be020 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T10:31:19.224 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:31:19.224+0000 7f6a02dd1740 1 --2- 172.21.15.114:0/3273030893 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55989b6adf40 0x55989b6be020 secure :-1 s=READY pgs=653 cs=0 l=1 rev1=1 crypto rx=0x7f69e800c940 tx=0x7f69e80035f0 comp rx=0 tx=0).stop 2024-04-23T10:31:19.224 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:31:19.225+0000 7f6a02dd1740 1 -- 172.21.15.114:0/3273030893 shutdown_connections 2024-04-23T10:31:19.224 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:31:19.225+0000 7f6a02dd1740 1 --2- 172.21.15.114:0/3273030893 >> [v2:172.21.15.119:6838/937297340,v1:172.21.15.119:6839/937297340] conn(0x55989b72f180 0x55989b731570 crc :-1 s=READY pgs=97 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:31:19.224 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:31:19.225+0000 7f6a02dd1740 1 --2- 172.21.15.114:0/3273030893 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x55989b6b8800 0x55989b6be630 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:31:19.224 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:31:19.225+0000 7f69e17fa640 0 client.16134 ms_handle_reset on v2:172.21.15.119:6838/937297340 2024-04-23T10:31:19.224 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:31:19.225+0000 7f6a02dd1740 1 --2- 172.21.15.114:0/3273030893 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55989b6adf40 0x55989b6be020 unknown :-1 s=CLOSED pgs=653 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:31:19.224 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:31:19.225+0000 7f6a02dd1740 1 --2- 172.21.15.114:0/3273030893 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x55989b6ad620 0x55989b6c0c90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:31:19.225 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:31:19.225+0000 7f6a02dd1740 1 -- 172.21.15.114:0/3273030893 >> 172.21.15.114:0/3273030893 conn(0x55989b6a9b30 msgr2=0x55989b6a8cb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T10:31:19.225 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:31:19.225+0000 7f6a02dd1740 1 -- 172.21.15.114:0/3273030893 shutdown_connections 2024-04-23T10:31:19.225 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:31:19.225+0000 7f6a02dd1740 1 -- 172.21.15.114:0/3273030893 wait complete. 2024-04-23T10:31:19.225 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:31:19.225+0000 7f6a02dd1740 20 client.16134 trim_cache size 0 max 16384 2024-04-23T10:31:19.225 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stdout:{ 2024-04-23T10:31:19.225 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stdout: "message": "", 2024-04-23T10:31:19.225 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stdout: "return_code": 0 2024-04-23T10:31:19.225 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stdout:} 2024-04-23T10:31:19.226 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:31:19.225+0000 7f69fffe2640 1 -- 172.21.15.114:0/446651473 >> [v2:172.21.15.114:6800/1395693466,v1:172.21.15.114:6801/1395693466] conn(0x7f69e40a3fe0 msgr2=0x7f69e40a64a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T10:31:19.226 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:31:19.225+0000 7f69fffe2640 1 --2- 172.21.15.114:0/446651473 >> [v2:172.21.15.114:6800/1395693466,v1:172.21.15.114:6801/1395693466] conn(0x7f69e40a3fe0 0x7f69e40a64a0 secure :-1 s=READY pgs=99 cs=0 l=1 rev1=1 crypto rx=0x55989b6aaf50 tx=0x7f69f0005d10 comp rx=0 tx=0).stop 2024-04-23T10:31:19.226 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:31:19.225+0000 7f69fffe2640 1 -- 172.21.15.114:0/446651473 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f69f81061b0 msgr2=0x7f69f8198c10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T10:31:19.226 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:31:19.225+0000 7f69fffe2640 1 --2- 172.21.15.114:0/446651473 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f69f81061b0 0x7f69f8198c10 secure :-1 s=READY pgs=649 cs=0 l=1 rev1=1 crypto rx=0x7f69e800c970 tx=0x7f69e800ce30 comp rx=0 tx=0).stop 2024-04-23T10:31:19.226 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:31:19.225+0000 7f69fffe2640 1 -- 172.21.15.114:0/446651473 shutdown_connections 2024-04-23T10:31:19.226 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:31:19.225+0000 7f69fffe2640 1 --2- 172.21.15.114:0/446651473 >> [v2:172.21.15.114:6800/1395693466,v1:172.21.15.114:6801/1395693466] conn(0x7f69e40a3fe0 0x7f69e40a64a0 unknown :-1 s=CLOSED pgs=99 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:31:19.226 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:31:19.225+0000 7f69fffe2640 1 --2- 172.21.15.114:0/446651473 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f69f819cf40 0x7f69f819d3f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:31:19.227 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:31:19.225+0000 7f69fffe2640 1 --2- 172.21.15.114:0/446651473 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f69f81061b0 0x7f69f8198c10 unknown :-1 s=CLOSED pgs=649 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:31:19.227 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:31:19.225+0000 7f69fffe2640 1 --2- 172.21.15.114:0/446651473 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f69f81010e0 0x7f69f81986d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:31:19.227 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:31:19.225+0000 7f69fffe2640 1 -- 172.21.15.114:0/446651473 >> 172.21.15.114:0/446651473 conn(0x7f69f80771d0 msgr2=0x7f69f80fec90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T10:31:19.227 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:31:19.225+0000 7f69fffe2640 1 -- 172.21.15.114:0/446651473 shutdown_connections 2024-04-23T10:31:19.227 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:31:19.225+0000 7f69fffe2640 1 -- 172.21.15.114:0/446651473 wait complete. 2024-04-23T10:31:19.918 INFO:tasks.workunit.client.0.smithi114.stdout: 1 305661 7.82 MB/sec execute 88 sec latency 2142.322 ms 2024-04-23T10:31:20.492 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 23 10:31:20 smithi119 ceph-mon[32608]: pgmap v145: 97 pgs: 97 active+clean; 94 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 23 MiB/s rd, 3.6 MiB/s wr, 348 op/s 2024-04-23T10:31:20.504 INFO:journalctl@ceph.mon.a.smithi114.stdout:Apr 23 10:31:20 smithi114 ceph-mon[29255]: pgmap v145: 97 pgs: 97 active+clean; 94 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 23 MiB/s rd, 3.6 MiB/s wr, 348 op/s 2024-04-23T10:31:20.578 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 23 10:31:20 smithi204 ceph-mon[32649]: pgmap v145: 97 pgs: 97 active+clean; 94 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 23 MiB/s rd, 3.6 MiB/s wr, 348 op/s 2024-04-23T10:31:20.918 INFO:tasks.workunit.client.0.smithi114.stdout: 1 305661 7.73 MB/sec execute 89 sec latency 3142.419 ms 2024-04-23T10:31:20.955 DEBUG:tasks.quiescer.fs.[cephfs]:Going to quiesce for duration: 22.6 2024-04-23T10:31:20.955 DEBUG:teuthology.orchestra.run.smithi114:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-23T10:31:21.068 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:31:21.067+0000 7f62b3330640 1 -- 172.21.15.114:0/3687389700 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f62ac101090 msgr2=0x7f62ac101470 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T10:31:21.068 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:31:21.067+0000 7f62b3330640 1 --2- 172.21.15.114:0/3687389700 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f62ac101090 0x7f62ac101470 secure :-1 s=READY pgs=654 cs=0 l=1 rev1=1 crypto rx=0x7f62a8009900 tx=0x7f62a802a410 comp rx=0 tx=0).stop 2024-04-23T10:31:21.068 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:31:21.067+0000 7f62b3330640 1 -- 172.21.15.114:0/3687389700 shutdown_connections 2024-04-23T10:31:21.068 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:31:21.067+0000 7f62b3330640 1 --2- 172.21.15.114:0/3687389700 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f62ac106160 0x7f62ac10dd30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:31:21.068 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:31:21.067+0000 7f62b3330640 1 --2- 172.21.15.114:0/3687389700 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f62ac101a40 0x7f62ac105a80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:31:21.068 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:31:21.067+0000 7f62b3330640 1 --2- 172.21.15.114:0/3687389700 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f62ac101090 0x7f62ac101470 unknown :-1 s=CLOSED pgs=654 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:31:21.068 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:31:21.067+0000 7f62b3330640 1 -- 172.21.15.114:0/3687389700 >> 172.21.15.114:0/3687389700 conn(0x7f62ac0771d0 msgr2=0x7f62ac0775e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T10:31:21.068 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:31:21.067+0000 7f62b3330640 1 -- 172.21.15.114:0/3687389700 shutdown_connections 2024-04-23T10:31:21.068 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:31:21.067+0000 7f62b3330640 1 -- 172.21.15.114:0/3687389700 wait complete. 2024-04-23T10:31:21.069 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:31:21.067+0000 7f62b3330640 1 Processor -- start 2024-04-23T10:31:21.069 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:31:21.067+0000 7f62b3330640 1 -- start start 2024-04-23T10:31:21.069 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:31:21.067+0000 7f62b3330640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f62ac101090 0x7f62ac199590 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T10:31:21.069 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:31:21.067+0000 7f62b3330640 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f62ac106160 0x7f62ac199ad0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T10:31:21.069 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:31:21.067+0000 7f62b3330640 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f62ac19a060 0x7f62ac1a4610 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T10:31:21.069 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:31:21.067+0000 7f62b3330640 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x7f62ac1110d0 con 0x7f62ac101090 2024-04-23T10:31:21.069 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:31:21.068+0000 7f62b3330640 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 -- 0x7f62ac110f50 con 0x7f62ac19a060 2024-04-23T10:31:21.069 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:31:21.068+0000 7f62b3330640 1 -- --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_getmap magic: 0 -- 0x7f62ac111250 con 0x7f62ac106160 2024-04-23T10:31:21.069 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:31:21.068+0000 7f62b10cc640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f62ac101090 0x7f62ac199590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T10:31:21.069 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:31:21.068+0000 7f62b10cc640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f62ac101090 0x7f62ac199590 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.114:37004/0 (socket says 172.21.15.114:37004) 2024-04-23T10:31:21.069 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:31:21.068+0000 7f62b10cc640 1 -- 172.21.15.114:0/81410384 learned_addr learned my addr 172.21.15.114:0/81410384 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-04-23T10:31:21.069 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:31:21.068+0000 7f62b08cb640 1 --2- 172.21.15.114:0/81410384 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f62ac106160 0x7f62ac199ad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T10:31:21.069 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:31:21.068+0000 7f62b10cc640 1 -- 172.21.15.114:0/81410384 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f62ac106160 msgr2=0x7f62ac199ad0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T10:31:21.070 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:31:21.068+0000 7f62b18cd640 1 --2- 172.21.15.114:0/81410384 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f62ac19a060 0x7f62ac1a4610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T10:31:21.070 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:31:21.068+0000 7f62b10cc640 1 --2- 172.21.15.114:0/81410384 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f62ac106160 0x7f62ac199ad0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:31:21.070 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:31:21.068+0000 7f62b10cc640 1 -- 172.21.15.114:0/81410384 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f62ac19a060 msgr2=0x7f62ac1a4610 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T10:31:21.070 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:31:21.068+0000 7f62b10cc640 1 --2- 172.21.15.114:0/81410384 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f62ac19a060 0x7f62ac1a4610 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:31:21.070 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:31:21.068+0000 7f62b10cc640 1 -- 172.21.15.114:0/81410384 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f62ac1a4de0 con 0x7f62ac101090 2024-04-23T10:31:21.070 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:31:21.068+0000 7f62b08cb640 1 --2- 172.21.15.114:0/81410384 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f62ac106160 0x7f62ac199ad0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-23T10:31:21.070 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:31:21.068+0000 7f62b10cc640 1 --2- 172.21.15.114:0/81410384 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f62ac101090 0x7f62ac199590 secure :-1 s=READY pgs=655 cs=0 l=1 rev1=1 crypto rx=0x7f62a8005940 tx=0x7f62a80042f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T10:31:21.070 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:31:21.068+0000 7f62a27fc640 1 -- 172.21.15.114:0/81410384 <== mon.0 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f62a803f030 con 0x7f62ac101090 2024-04-23T10:31:21.070 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:31:21.068+0000 7f62a27fc640 1 -- 172.21.15.114:0/81410384 <== mon.0 v2:172.21.15.114:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f62a8003cb0 con 0x7f62ac101090 2024-04-23T10:31:21.070 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:31:21.068+0000 7f62b3330640 1 -- 172.21.15.114:0/81410384 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f62ac1a5070 con 0x7f62ac101090 2024-04-23T10:31:21.070 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:31:21.068+0000 7f62a27fc640 1 -- 172.21.15.114:0/81410384 <== mon.0 v2:172.21.15.114:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f62a8035070 con 0x7f62ac101090 2024-04-23T10:31:21.070 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:31:21.068+0000 7f62b3330640 1 -- 172.21.15.114:0/81410384 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f62ac1a5560 con 0x7f62ac101090 2024-04-23T10:31:21.070 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:31:21.069+0000 7f62b3330640 1 -- 172.21.15.114:0/81410384 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f62ac102560 con 0x7f62ac101090 2024-04-23T10:31:21.071 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:31:21.070+0000 7f62a27fc640 1 -- 172.21.15.114:0/81410384 <== mon.0 v2:172.21.15.114:3300/0 4 ==== mgrmap(e 26) ==== 138778+0+0 (secure 0 0 0) 0x7f62a8042030 con 0x7f62ac101090 2024-04-23T10:31:21.071 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:31:21.071+0000 7f62a27fc640 1 --2- 172.21.15.114:0/81410384 >> [v2:172.21.15.114:6800/1395693466,v1:172.21.15.114:6801/1395693466] conn(0x7f62980a3d20 0x7f62980a61e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T10:31:21.071 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:31:21.071+0000 7f62a27fc640 1 -- 172.21.15.114:0/81410384 <== mon.0 v2:172.21.15.114:3300/0 5 ==== osd_map(93..93 src has 1..93) ==== 8585+0+0 (secure 0 0 0) 0x7f62a80f1450 con 0x7f62ac101090 2024-04-23T10:31:21.071 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:31:21.071+0000 7f62b08cb640 1 --2- 172.21.15.114:0/81410384 >> [v2:172.21.15.114:6800/1395693466,v1:172.21.15.114:6801/1395693466] conn(0x7f62980a3d20 0x7f62980a61e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T10:31:21.071 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:31:21.071+0000 7f62b08cb640 1 --2- 172.21.15.114:0/81410384 >> [v2:172.21.15.114:6800/1395693466,v1:172.21.15.114:6801/1395693466] conn(0x7f62980a3d20 0x7f62980a61e0 secure :-1 s=READY pgs=100 cs=0 l=1 rev1=1 crypto rx=0x7f629c0058c0 tx=0x7f629c005850 comp rx=0 tx=0).ready entity=mgr.15291 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T10:31:21.073 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:31:21.075+0000 7f62a27fc640 1 -- 172.21.15.114:0/81410384 <== mon.0 v2:172.21.15.114:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f62a80bdab0 con 0x7f62ac101090 2024-04-23T10:31:21.290 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:31:21.291+0000 7f62b3330640 1 -- 172.21.15.114:0/81410384 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f62ac109390 con 0x7f62ac101090 2024-04-23T10:31:21.291 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:31:21.292+0000 7f62a27fc640 1 -- 172.21.15.114:0/81410384 <== mon.0 v2:172.21.15.114:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 28 v28) ==== 94+0+11881 (secure 0 0 0) 0x7f62a80bd3b0 con 0x7f62ac101090 2024-04-23T10:31:21.291 INFO:teuthology.orchestra.run.smithi114.stdout: 2024-04-23T10:31:21.291 INFO:teuthology.orchestra.run.smithi114.stdout:{"epoch":28,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14547,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.114:6835/663493580","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6834","nonce":663493580},{"type":"v1","addr":"172.21.15.114:6835","nonce":663493580}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14565,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.114:6837/1439252154","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6836","nonce":1439252154},{"type":"v1","addr":"172.21.15.114:6837","nonce":1439252154}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5},{"gid":14634,"name":"g","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.114:6839/3849796939","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6838","nonce":3849796939},{"type":"v1","addr":"172.21.15.114:6839","nonce":3849796939}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":11},{"gid":14646,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.114:6841/3544523891","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6840","nonce":3544523891},{"type":"v1","addr":"172.21.15.114:6841","nonce":3544523891}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":12},{"gid":24385,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.204:6833/3866561754","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6832","nonce":3866561754},{"type":"v1","addr":"172.21.15.204:6833","nonce":3866561754}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2},{"gid":24427,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.204:6835/2533793172","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6834","nonce":2533793172},{"type":"v1","addr":"172.21.15.204:6835","nonce":2533793172}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":7}],"filesystems":[{"mdsmap":{"epoch":28,"flags":306,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":true},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-04-23T10:24:43.383158+0000","modified":"2024-04-23T10:30:54.064807+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":3,"in":[0,1,2],"up":{"mds_0":24545,"mds_1":24509,"mds_2":24479},"failed":[],"damaged":[],"stopped":[],"info":{"gid_24439":{"gid":24439,"name":"c","rank":2,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.204:6837/513822179","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6836","nonce":513822179},{"type":"v1","addr":"172.21.15.204:6837","nonce":513822179}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24443":{"gid":24443,"name":"k","rank":1,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.119:6833/783244660","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6832","nonce":783244660},{"type":"v1","addr":"172.21.15.119:6833","nonce":783244660}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24479":{"gid":24479,"name":"e","rank":2,"incarnation":19,"state":"up:active","state_seq":9,"addr":"172.21.15.119:6835/2033916709","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6834","nonce":2033916709},{"type":"v1","addr":"172.21.15.119:6835","nonce":2033916709}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24509":{"gid":24509,"name":"h","rank":1,"incarnation":17,"state":"up:active","state_seq":6,"addr":"172.21.15.119:6837/1872363128","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6836","nonce":1872363128},{"type":"v1","addr":"172.21.15.119:6837","nonce":1872363128}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24545":{"gid":24545,"name":"b","rank":0,"incarnation":14,"state":"up:active","state_seq":3,"addr":"172.21.15.119:6839/937297340","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6838","nonce":937297340},{"type":"v1","addr":"172.21.15.119:6839","nonce":937297340}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_25477":{"gid":25477,"name":"i","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.204:6839/417363607","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6838","nonce":417363607},{"type":"v1","addr":"172.21.15.204:6839","nonce":417363607}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":24545,"qdb_cluster":[24545,24479,24509]},"id":1}]} 2024-04-23T10:31:21.291 INFO:teuthology.orchestra.run.smithi114.stderr:dumped fsmap epoch 28 2024-04-23T10:31:21.293 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:31:21.294+0000 7f62b3330640 1 -- 172.21.15.114:0/81410384 >> [v2:172.21.15.114:6800/1395693466,v1:172.21.15.114:6801/1395693466] conn(0x7f62980a3d20 msgr2=0x7f62980a61e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T10:31:21.293 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:31:21.294+0000 7f62b3330640 1 --2- 172.21.15.114:0/81410384 >> [v2:172.21.15.114:6800/1395693466,v1:172.21.15.114:6801/1395693466] conn(0x7f62980a3d20 0x7f62980a61e0 secure :-1 s=READY pgs=100 cs=0 l=1 rev1=1 crypto rx=0x7f629c0058c0 tx=0x7f629c005850 comp rx=0 tx=0).stop 2024-04-23T10:31:21.294 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:31:21.294+0000 7f62b3330640 1 -- 172.21.15.114:0/81410384 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f62ac101090 msgr2=0x7f62ac199590 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T10:31:21.294 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:31:21.294+0000 7f62b3330640 1 --2- 172.21.15.114:0/81410384 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f62ac101090 0x7f62ac199590 secure :-1 s=READY pgs=655 cs=0 l=1 rev1=1 crypto rx=0x7f62a8005940 tx=0x7f62a80042f0 comp rx=0 tx=0).stop 2024-04-23T10:31:21.294 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:31:21.294+0000 7f62b3330640 1 -- 172.21.15.114:0/81410384 shutdown_connections 2024-04-23T10:31:21.294 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:31:21.294+0000 7f62b3330640 1 --2- 172.21.15.114:0/81410384 >> [v2:172.21.15.114:6800/1395693466,v1:172.21.15.114:6801/1395693466] conn(0x7f62980a3d20 0x7f62980a61e0 unknown :-1 s=CLOSED pgs=100 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:31:21.294 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:31:21.294+0000 7f62b3330640 1 --2- 172.21.15.114:0/81410384 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f62ac19a060 0x7f62ac1a4610 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:31:21.294 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:31:21.294+0000 7f62b3330640 1 --2- 172.21.15.114:0/81410384 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f62ac106160 0x7f62ac199ad0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:31:21.294 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:31:21.294+0000 7f62b3330640 1 --2- 172.21.15.114:0/81410384 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f62ac101090 0x7f62ac199590 unknown :-1 s=CLOSED pgs=655 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:31:21.294 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:31:21.294+0000 7f62b3330640 1 -- 172.21.15.114:0/81410384 >> 172.21.15.114:0/81410384 conn(0x7f62ac0771d0 msgr2=0x7f62ac0fee20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T10:31:21.294 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:31:21.294+0000 7f62b3330640 1 -- 172.21.15.114:0/81410384 shutdown_connections 2024-04-23T10:31:21.294 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:31:21.294+0000 7f62b3330640 1 -- 172.21.15.114:0/81410384 wait complete. 2024-04-23T10:31:21.304 DEBUG:tasks.quiescer.fs.[cephfs]:Running ceph command: 'tell mds.24545 quiesce db / --timeout 60 --expiration 82.6 --await-for 42.0' 2024-04-23T10:31:21.304 DEBUG:teuthology.orchestra.run.smithi114:> timeout 75 sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.24545 quiesce db / --timeout 60 --expiration 82.6 --await-for 42.0 2024-04-23T10:31:21.413 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:31:21.412+0000 7f27d7d07640 1 -- 172.21.15.114:0/4182196238 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f27d0102b40 msgr2=0x7f27d0106b80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T10:31:21.413 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:31:21.412+0000 7f27d7d07640 1 --2- 172.21.15.114:0/4182196238 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f27d0102b40 0x7f27d0106b80 secure :-1 s=READY pgs=656 cs=0 l=1 rev1=1 crypto rx=0x7f27c0009900 tx=0x7f27c002a410 comp rx=0 tx=0).stop 2024-04-23T10:31:21.413 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:31:21.412+0000 7f27d7d07640 1 -- 172.21.15.114:0/4182196238 shutdown_connections 2024-04-23T10:31:21.413 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:31:21.412+0000 7f27d7d07640 1 --2- 172.21.15.114:0/4182196238 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f27d0107260 0x7f27d010ee30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:31:21.414 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:31:21.412+0000 7f27d7d07640 1 --2- 172.21.15.114:0/4182196238 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f27d0102b40 0x7f27d0106b80 unknown :-1 s=CLOSED pgs=656 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:31:21.414 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:31:21.412+0000 7f27d7d07640 1 --2- 172.21.15.114:0/4182196238 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f27d0102190 0x7f27d0102570 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:31:21.414 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:31:21.412+0000 7f27d7d07640 1 -- 172.21.15.114:0/4182196238 >> 172.21.15.114:0/4182196238 conn(0x7f27d0079310 msgr2=0x7f27d0079720 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T10:31:21.414 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:31:21.412+0000 7f27d7d07640 1 -- 172.21.15.114:0/4182196238 shutdown_connections 2024-04-23T10:31:21.414 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:31:21.412+0000 7f27d7d07640 1 -- 172.21.15.114:0/4182196238 wait complete. 2024-04-23T10:31:21.414 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:31:21.413+0000 7f27d7d07640 1 Processor -- start 2024-04-23T10:31:21.414 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:31:21.413+0000 7f27d7d07640 1 -- start start 2024-04-23T10:31:21.414 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:31:21.413+0000 7f27d7d07640 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f27d0102190 0x7f27d0074250 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T10:31:21.414 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:31:21.413+0000 7f27d7d07640 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f27d0107260 0x7f27d0074790 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T10:31:21.414 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:31:21.413+0000 7f27d7d07640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f27d0074cd0 0x7f27d01a69b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T10:31:21.414 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:31:21.413+0000 7f27d7d07640 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x7f27d01121a0 con 0x7f27d0074cd0 2024-04-23T10:31:21.414 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:31:21.413+0000 7f27d7d07640 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 -- 0x7f27d0112020 con 0x7f27d0102190 2024-04-23T10:31:21.414 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:31:21.413+0000 7f27d7d07640 1 -- --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_getmap magic: 0 -- 0x7f27d0112320 con 0x7f27d0107260 2024-04-23T10:31:21.414 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:31:21.413+0000 7f27d62a4640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f27d0074cd0 0x7f27d01a69b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T10:31:21.415 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:31:21.413+0000 7f27d62a4640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f27d0074cd0 0x7f27d01a69b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.114:37030/0 (socket says 172.21.15.114:37030) 2024-04-23T10:31:21.415 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:31:21.413+0000 7f27d62a4640 1 -- 172.21.15.114:0/3404163860 learned_addr learned my addr 172.21.15.114:0/3404163860 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-04-23T10:31:21.415 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:31:21.413+0000 7f27d52a2640 1 --2- 172.21.15.114:0/3404163860 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f27d0107260 0x7f27d0074790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T10:31:21.415 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:31:21.413+0000 7f27d5aa3640 1 --2- 172.21.15.114:0/3404163860 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f27d0102190 0x7f27d0074250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T10:31:21.415 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:31:21.413+0000 7f27d62a4640 1 -- 172.21.15.114:0/3404163860 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f27d0107260 msgr2=0x7f27d0074790 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T10:31:21.415 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:31:21.413+0000 7f27d62a4640 1 --2- 172.21.15.114:0/3404163860 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f27d0107260 0x7f27d0074790 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:31:21.415 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:31:21.413+0000 7f27d62a4640 1 -- 172.21.15.114:0/3404163860 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f27d0102190 msgr2=0x7f27d0074250 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T10:31:21.415 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:31:21.413+0000 7f27d62a4640 1 --2- 172.21.15.114:0/3404163860 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f27d0102190 0x7f27d0074250 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:31:21.415 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:31:21.413+0000 7f27d62a4640 1 -- 172.21.15.114:0/3404163860 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f27d01a7050 con 0x7f27d0074cd0 2024-04-23T10:31:21.415 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:31:21.413+0000 7f27d52a2640 1 --2- 172.21.15.114:0/3404163860 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f27d0107260 0x7f27d0074790 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-23T10:31:21.415 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:31:21.413+0000 7f27d62a4640 1 --2- 172.21.15.114:0/3404163860 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f27d0074cd0 0x7f27d01a69b0 secure :-1 s=READY pgs=657 cs=0 l=1 rev1=1 crypto rx=0x7f27c800cad0 tx=0x7f27c800cf90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T10:31:21.415 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:31:21.414+0000 7f27c6ffd640 1 -- 172.21.15.114:0/3404163860 <== mon.0 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f27c800ae20 con 0x7f27d0074cd0 2024-04-23T10:31:21.415 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:31:21.414+0000 7f27c6ffd640 1 -- 172.21.15.114:0/3404163860 <== mon.0 v2:172.21.15.114:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f27c8005020 con 0x7f27d0074cd0 2024-04-23T10:31:21.415 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:31:21.414+0000 7f27d7d07640 1 -- 172.21.15.114:0/3404163860 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f27d01a7340 con 0x7f27d0074cd0 2024-04-23T10:31:21.416 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:31:21.414+0000 7f27c6ffd640 1 -- 172.21.15.114:0/3404163860 <== mon.0 v2:172.21.15.114:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f27c800f6b0 con 0x7f27d0074cd0 2024-04-23T10:31:21.416 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:31:21.414+0000 7f27d7d07640 1 -- 172.21.15.114:0/3404163860 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f27d01a7860 con 0x7f27d0074cd0 2024-04-23T10:31:21.422 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:31:21.416+0000 7f27c6ffd640 1 -- 172.21.15.114:0/3404163860 <== mon.0 v2:172.21.15.114:3300/0 4 ==== mgrmap(e 26) ==== 138778+0+0 (secure 0 0 0) 0x7f27c80051d0 con 0x7f27d0074cd0 2024-04-23T10:31:21.423 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:31:21.416+0000 7f27daa1a740 1 Processor -- start 2024-04-23T10:31:21.423 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:31:21.417+0000 7f27daa1a740 1 -- start start 2024-04-23T10:31:21.423 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:31:21.417+0000 7f27daa1a740 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x556ffeb6f0a0 0x556ffeb6f480 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T10:31:21.423 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:31:21.417+0000 7f27daa1a740 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x556ffeb6f9c0 0x556ffeb79b50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T10:31:21.423 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:31:21.417+0000 7f27daa1a740 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x556ffeb7a090 0x556ffeb7c480 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T10:31:21.423 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:31:21.417+0000 7f27daa1a740 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x556ffead4370 con 0x556ffeb6f9c0 2024-04-23T10:31:21.423 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:31:21.417+0000 7f27daa1a740 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 -- 0x556ffea712e0 con 0x556ffeb7a090 2024-04-23T10:31:21.423 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:31:21.417+0000 7f27daa1a740 1 -- --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_getmap magic: 0 -- 0x556ffead0330 con 0x556ffeb6f0a0 2024-04-23T10:31:21.423 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:31:21.417+0000 7f27c6ffd640 1 --2- 172.21.15.114:0/3404163860 >> [v2:172.21.15.114:6800/1395693466,v1:172.21.15.114:6801/1395693466] conn(0x7f27bc0a3d20 0x7f27bc0a61e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T10:31:21.423 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:31:21.417+0000 7f27d5aa3640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x556ffeb6f9c0 0x556ffeb79b50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T10:31:21.423 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:31:21.417+0000 7f27d62a4640 1 --2- 172.21.15.114:0/3404163860 >> [v2:172.21.15.114:6800/1395693466,v1:172.21.15.114:6801/1395693466] conn(0x7f27bc0a3d20 0x7f27bc0a61e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T10:31:21.423 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:31:21.417+0000 7f27d5aa3640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x556ffeb6f9c0 0x556ffeb79b50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.114:37044/0 (socket says 172.21.15.114:37044) 2024-04-23T10:31:21.423 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:31:21.417+0000 7f27d52a2640 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x556ffeb6f0a0 0x556ffeb6f480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T10:31:21.423 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:31:21.417+0000 7f27d5aa3640 1 -- 172.21.15.114:0/1733784606 learned_addr learned my addr 172.21.15.114:0/1733784606 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-04-23T10:31:21.424 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:31:21.417+0000 7f27d52a2640 1 --2- 172.21.15.114:0/1733784606 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x556ffeb7a090 0x556ffeb7c480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T10:31:21.424 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:31:21.417+0000 7f27c6ffd640 1 -- 172.21.15.114:0/3404163860 <== mon.0 v2:172.21.15.114:3300/0 5 ==== osd_map(93..93 src has 1..93) ==== 8585+0+0 (secure 0 0 0) 0x7f27c80d08d0 con 0x7f27d0074cd0 2024-04-23T10:31:21.424 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:31:21.417+0000 7f27d5aa3640 1 -- 172.21.15.114:0/1733784606 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x556ffeb6f0a0 msgr2=0x556ffeb6f480 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T10:31:21.424 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:31:21.417+0000 7f27d5aa3640 1 --2- 172.21.15.114:0/1733784606 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x556ffeb6f0a0 0x556ffeb6f480 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:31:21.424 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:31:21.417+0000 7f27d5aa3640 1 -- 172.21.15.114:0/1733784606 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x556ffeb7a090 msgr2=0x556ffeb7c480 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T10:31:21.424 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:31:21.417+0000 7f27d5aa3640 1 --2- 172.21.15.114:0/1733784606 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x556ffeb7a090 0x556ffeb7c480 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:31:21.424 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:31:21.417+0000 7f27d5aa3640 1 -- 172.21.15.114:0/1733784606 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x556ffe8b2d80 con 0x556ffeb6f9c0 2024-04-23T10:31:21.424 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:31:21.417+0000 7f27d5aa3640 1 --2- 172.21.15.114:0/1733784606 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x556ffeb6f9c0 0x556ffeb79b50 secure :-1 s=READY pgs=658 cs=0 l=1 rev1=1 crypto rx=0x7f27cc00dcb0 tx=0x7f27cc00f730 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a9cc943044fa213c server_cookie=0 in_seq=0 out_seq=0 2024-04-23T10:31:21.424 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:31:21.417+0000 7f27d62a4640 1 --2- 172.21.15.114:0/3404163860 >> [v2:172.21.15.114:6800/1395693466,v1:172.21.15.114:6801/1395693466] conn(0x7f27bc0a3d20 0x7f27bc0a61e0 secure :-1 s=READY pgs=101 cs=0 l=1 rev1=1 crypto rx=0x7f27c8098c90 tx=0x7f27c80983b0 comp rx=0 tx=0).ready entity=mgr.15291 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T10:31:21.424 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:31:21.418+0000 7f27bbfff640 1 -- 172.21.15.114:0/1733784606 <== mon.0 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f27cc011440 con 0x556ffeb6f9c0 2024-04-23T10:31:21.424 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:31:21.418+0000 7f27bbfff640 1 -- 172.21.15.114:0/1733784606 <== mon.0 v2:172.21.15.114:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f27cc011a90 con 0x556ffeb6f9c0 2024-04-23T10:31:21.424 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:31:21.418+0000 7f27bbfff640 1 -- 172.21.15.114:0/1733784606 <== mon.0 v2:172.21.15.114:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f27cc010990 con 0x556ffeb6f9c0 2024-04-23T10:31:21.424 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:31:21.418+0000 7f27daa1a740 1 -- 172.21.15.114:0/1733784606 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x556ffeb6f9c0 msgr2=0x556ffeb79b50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T10:31:21.424 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:31:21.418+0000 7f27daa1a740 1 --2- 172.21.15.114:0/1733784606 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x556ffeb6f9c0 0x556ffeb79b50 secure :-1 s=READY pgs=658 cs=0 l=1 rev1=1 crypto rx=0x7f27cc00dcb0 tx=0x7f27cc00f730 comp rx=0 tx=0).stop 2024-04-23T10:31:21.424 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:31:21.418+0000 7f27daa1a740 1 -- 172.21.15.114:0/1733784606 shutdown_connections 2024-04-23T10:31:21.425 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:31:21.418+0000 7f27daa1a740 1 --2- 172.21.15.114:0/1733784606 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x556ffeb7a090 0x556ffeb7c480 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:31:21.425 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:31:21.418+0000 7f27daa1a740 1 --2- 172.21.15.114:0/1733784606 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x556ffeb6f9c0 0x556ffeb79b50 unknown :-1 s=CLOSED pgs=658 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:31:21.425 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:31:21.418+0000 7f27daa1a740 1 --2- 172.21.15.114:0/1733784606 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x556ffeb6f0a0 0x556ffeb6f480 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:31:21.425 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:31:21.418+0000 7f27daa1a740 1 -- 172.21.15.114:0/1733784606 >> 172.21.15.114:0/1733784606 conn(0x556ffeb6b3c0 msgr2=0x556ffeb6b7a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T10:31:21.425 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:31:21.418+0000 7f27daa1a740 1 -- 172.21.15.114:0/1733784606 shutdown_connections 2024-04-23T10:31:21.425 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:31:21.418+0000 7f27daa1a740 1 -- 172.21.15.114:0/1733784606 wait complete. 2024-04-23T10:31:21.425 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:31:21.419+0000 7f27daa1a740 1 -- start start 2024-04-23T10:31:21.425 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:31:21.419+0000 7f27daa1a740 1 Processor -- start 2024-04-23T10:31:21.425 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:31:21.419+0000 7f27daa1a740 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x556ffeb6f0a0 0x556ffeb77010 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T10:31:21.425 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:31:21.419+0000 7f27daa1a740 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x556ffeb6f9c0 0x556ffeb77550 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T10:31:21.425 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:31:21.419+0000 7f27daa1a740 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x556ffeb7a090 0x556ffeb77a90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T10:31:21.425 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:31:21.419+0000 7f27daa1a740 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x556ffeb7ce80 con 0x556ffeb6f9c0 2024-04-23T10:31:21.425 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:31:21.419+0000 7f27daa1a740 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 -- 0x556ffea6efb0 con 0x556ffeb6f0a0 2024-04-23T10:31:21.426 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:31:21.419+0000 7f27daa1a740 1 -- --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_getmap magic: 0 -- 0x556ffeb7d000 con 0x556ffeb7a090 2024-04-23T10:31:21.426 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:31:21.419+0000 7f27daa1a740 10 client.0 _ll_register_callbacks cb 0x556ffea6e9a0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-23T10:31:21.426 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:31:21.419+0000 7f27d5aa3640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x556ffeb6f9c0 0x556ffeb77550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T10:31:21.426 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:31:21.419+0000 7f27d5aa3640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x556ffeb6f9c0 0x556ffeb77550 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.114:37050/0 (socket says 172.21.15.114:37050) 2024-04-23T10:31:21.426 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:31:21.419+0000 7f27d5aa3640 1 -- 172.21.15.114:0/2791270913 learned_addr learned my addr 172.21.15.114:0/2791270913 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-04-23T10:31:21.426 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:31:21.419+0000 7f27d52a2640 1 --2- 172.21.15.114:0/2791270913 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x556ffeb7a090 0x556ffeb77a90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T10:31:21.426 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:31:21.419+0000 7f27d52a2640 1 --2- 172.21.15.114:0/2791270913 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x556ffeb6f0a0 0x556ffeb77010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T10:31:21.426 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:31:21.419+0000 7f27d5aa3640 1 -- 172.21.15.114:0/2791270913 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x556ffeb7a090 msgr2=0x556ffeb77a90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T10:31:21.426 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:31:21.419+0000 7f27d5aa3640 1 --2- 172.21.15.114:0/2791270913 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x556ffeb7a090 0x556ffeb77a90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:31:21.426 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:31:21.419+0000 7f27d5aa3640 1 -- 172.21.15.114:0/2791270913 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x556ffeb6f0a0 msgr2=0x556ffeb77010 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T10:31:21.426 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:31:21.420+0000 7f27d5aa3640 1 --2- 172.21.15.114:0/2791270913 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x556ffeb6f0a0 0x556ffeb77010 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:31:21.426 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:31:21.420+0000 7f27d5aa3640 1 -- 172.21.15.114:0/2791270913 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({osdmap=0}) -- 0x556ffe913510 con 0x556ffeb6f9c0 2024-04-23T10:31:21.426 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:31:21.420+0000 7f27d5aa3640 1 --2- 172.21.15.114:0/2791270913 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x556ffeb6f9c0 0x556ffeb77550 secure :-1 s=READY pgs=659 cs=0 l=1 rev1=1 crypto rx=0x7f27cc0105b0 tx=0x7f27cc0103d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2c6de8db15ee112d server_cookie=0 in_seq=0 out_seq=0 2024-04-23T10:31:21.426 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:31:21.420+0000 7f27b97fa640 10 client.0 ms_handle_connect on v2:172.21.15.114:3300/0 2024-04-23T10:31:21.427 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:31:21.420+0000 7f27b97fa640 1 -- 172.21.15.114:0/2791270913 <== mon.0 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f27cc02b060 con 0x556ffeb6f9c0 2024-04-23T10:31:21.427 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:31:21.420+0000 7f27b97fa640 1 -- 172.21.15.114:0/2791270913 <== mon.0 v2:172.21.15.114:3300/0 2 ==== osd_map(93..93 src has 1..93) ==== 8585+0+0 (secure 0 0 0) 0x7f27cc011c40 con 0x556ffeb6f9c0 2024-04-23T10:31:21.427 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:31:21.420+0000 7f27daa1a740 1 -- 172.21.15.114:0/2791270913 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f27cc011a90 con 0x556ffeb6f9c0 2024-04-23T10:31:21.427 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:31:21.420+0000 7f27b97fa640 1 -- 172.21.15.114:0/2791270913 <== mon.0 v2:172.21.15.114:3300/0 3 ==== mon_get_version_reply(handle=1 version=28) ==== 24+0+0 (secure 0 0 0) 0x7f27cc00d880 con 0x556ffeb6f9c0 2024-04-23T10:31:21.427 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:31:21.420+0000 7f27daa1a740 10 client.16152 fetch_fsmap learned FSMap version 28 2024-04-23T10:31:21.427 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:31:21.420+0000 7f27daa1a740 1 -- 172.21.15.114:0/2791270913 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({config=0+,fsmap=28,monmap=4+}) -- 0x556ffe7a6670 con 0x556ffeb6f9c0 2024-04-23T10:31:21.427 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:31:21.420+0000 7f27b97fa640 1 -- 172.21.15.114:0/2791270913 <== mon.0 v2:172.21.15.114:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f27cc011a90 con 0x556ffeb6f9c0 2024-04-23T10:31:21.427 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:31:21.420+0000 7f27b97fa640 1 -- 172.21.15.114:0/2791270913 <== mon.0 v2:172.21.15.114:3300/0 5 ==== fsmap(e 28) ==== 8338+0+0 (secure 0 0 0) 0x7f27cc0107a0 con 0x556ffeb6f9c0 2024-04-23T10:31:21.427 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:31:21.420+0000 7f27daa1a740 10 client.16152 fetch_fsmap finished waiting for FSMap version 28 2024-04-23T10:31:21.427 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:31:21.420+0000 7f27daa1a740 10 client.16152 resolve_mds: validated gid 24545 aka daemon mds.b 2024-04-23T10:31:21.427 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:31:21.420+0000 7f27daa1a740 20 client.16152 populate_metadata read hostname 'smithi114' 2024-04-23T10:31:21.427 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:31:21.420+0000 7f27daa1a740 1 --2- 172.21.15.114:0/2791270913 >> [v2:172.21.15.119:6838/937297340,v1:172.21.15.119:6839/937297340] conn(0x556ffeb8a2e0 0x556ffeb8c6d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T10:31:21.427 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:31:21.420+0000 7f27daa1a740 4 client.16152 mds_command: new command op to 24545 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-23T10:31:21.427 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:31:21.420+0000 7f27daa1a740 1 -- 172.21.15.114:0/2791270913 --> [v2:172.21.15.119:6838/937297340,v1:172.21.15.119:6839/937297340] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x556ffeae1d90 con 0x556ffeb8a2e0 2024-04-23T10:31:21.427 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:31:21.421+0000 7f27d52a2640 1 --2- 172.21.15.114:0/2791270913 >> [v2:172.21.15.119:6838/937297340,v1:172.21.15.119:6839/937297340] conn(0x556ffeb8a2e0 0x556ffeb8c6d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T10:31:21.428 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:31:21.421+0000 7f27b97fa640 10 client.16152 ms_handle_connect on v2:172.21.15.119:6838/937297340 2024-04-23T10:31:21.428 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:31:21.421+0000 7f27d52a2640 1 --2- 172.21.15.114:0/2791270913 >> [v2:172.21.15.119:6838/937297340,v1:172.21.15.119:6839/937297340] conn(0x556ffeb8a2e0 0x556ffeb8c6d0 crc :-1 s=READY pgs=98 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=c9c1eab98d2e2590 server_cookie=e4472e72695b7f3e in_seq=0 out_seq=0 2024-04-23T10:31:21.428 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:31:21.422+0000 7f27b97fa640 1 -- 172.21.15.114:0/2791270913 <== mds.0 v2:172.21.15.119:6838/937297340 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x556ffeae1d90 con 0x556ffeb8a2e0 2024-04-23T10:31:21.428 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:31:21.422+0000 7f27b97fa640 10 client.16152 handle_command_reply: tid=0 2024-04-23T10:31:21.428 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:31:21.422+0000 7f27daa1a740 1 client.16152 shutdown 2024-04-23T10:31:21.428 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:31:21.422+0000 7f27daa1a740 1 -- 172.21.15.114:0/2791270913 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x556ffeb6f9c0 msgr2=0x556ffeb77550 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T10:31:21.428 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:31:21.422+0000 7f27daa1a740 1 --2- 172.21.15.114:0/2791270913 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x556ffeb6f9c0 0x556ffeb77550 secure :-1 s=READY pgs=659 cs=0 l=1 rev1=1 crypto rx=0x7f27cc0105b0 tx=0x7f27cc0103d0 comp rx=0 tx=0).stop 2024-04-23T10:31:21.428 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:31:21.422+0000 7f27daa1a740 1 -- 172.21.15.114:0/2791270913 shutdown_connections 2024-04-23T10:31:21.428 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:31:21.422+0000 7f27daa1a740 1 --2- 172.21.15.114:0/2791270913 >> [v2:172.21.15.119:6838/937297340,v1:172.21.15.119:6839/937297340] conn(0x556ffeb8a2e0 0x556ffeb8c6d0 crc :-1 s=READY pgs=98 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:31:21.428 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:31:21.422+0000 7f27daa1a740 1 --2- 172.21.15.114:0/2791270913 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x556ffeb7a090 0x556ffeb77a90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:31:21.428 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:31:21.422+0000 7f27b97fa640 0 client.16152 ms_handle_reset on v2:172.21.15.119:6838/937297340 2024-04-23T10:31:21.428 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:31:21.422+0000 7f27daa1a740 1 --2- 172.21.15.114:0/2791270913 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x556ffeb6f9c0 0x556ffeb77550 unknown :-1 s=CLOSED pgs=659 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:31:21.428 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:31:21.422+0000 7f27daa1a740 1 --2- 172.21.15.114:0/2791270913 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x556ffeb6f0a0 0x556ffeb77010 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:31:21.429 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:31:21.422+0000 7f27daa1a740 1 -- 172.21.15.114:0/2791270913 >> 172.21.15.114:0/2791270913 conn(0x556ffeb6b3c0 msgr2=0x556ffeb711c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T10:31:21.429 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:31:21.422+0000 7f27daa1a740 1 -- 172.21.15.114:0/2791270913 shutdown_connections 2024-04-23T10:31:21.429 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:31:21.422+0000 7f27daa1a740 1 -- 172.21.15.114:0/2791270913 wait complete. 2024-04-23T10:31:21.429 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:31:21.422+0000 7f27daa1a740 20 client.16152 trim_cache size 0 max 16384 2024-04-23T10:31:21.443 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:31:21.439+0000 7f27daa1a740 1 Processor -- start 2024-04-23T10:31:21.443 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:31:21.439+0000 7f27daa1a740 1 -- start start 2024-04-23T10:31:21.443 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:31:21.439+0000 7f27daa1a740 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x556ffeb6f0a0 0x556ffeb71910 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T10:31:21.443 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:31:21.439+0000 7f27daa1a740 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x556ffeb6f9c0 0x556ffeb8f6f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T10:31:21.443 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:31:21.439+0000 7f27daa1a740 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x556ffeb7a090 0x556ffeb8fc30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T10:31:21.443 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:31:21.439+0000 7f27daa1a740 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x7f27d01b28f0 con 0x556ffeb7a090 2024-04-23T10:31:21.443 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:31:21.439+0000 7f27daa1a740 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 -- 0x7f27d01b01c0 con 0x556ffeb6f0a0 2024-04-23T10:31:21.443 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:31:21.439+0000 7f27daa1a740 1 -- --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_getmap magic: 0 -- 0x7f27d01ae010 con 0x556ffeb6f9c0 2024-04-23T10:31:21.443 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:31:21.439+0000 7f27d52a2640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x556ffeb7a090 0x556ffeb8fc30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T10:31:21.443 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:31:21.440+0000 7f27d52a2640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x556ffeb7a090 0x556ffeb8fc30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.114:37060/0 (socket says 172.21.15.114:37060) 2024-04-23T10:31:21.443 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:31:21.440+0000 7f27d52a2640 1 -- 172.21.15.114:0/1528169654 learned_addr learned my addr 172.21.15.114:0/1528169654 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-04-23T10:31:21.443 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:31:21.440+0000 7f27d52a2640 1 --2- 172.21.15.114:0/1528169654 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x556ffeb6f0a0 0x556ffeb71910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T10:31:21.443 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:31:21.440+0000 7f27d5aa3640 1 --2- 172.21.15.114:0/1528169654 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x556ffeb6f9c0 0x556ffeb8f6f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T10:31:21.443 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:31:21.440+0000 7f27d52a2640 1 -- 172.21.15.114:0/1528169654 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x556ffeb6f9c0 msgr2=0x556ffeb8f6f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T10:31:21.444 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:31:21.440+0000 7f27d52a2640 1 --2- 172.21.15.114:0/1528169654 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x556ffeb6f9c0 0x556ffeb8f6f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:31:21.444 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:31:21.440+0000 7f27d52a2640 1 -- 172.21.15.114:0/1528169654 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x556ffeb6f0a0 msgr2=0x556ffeb71910 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T10:31:21.444 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:31:21.440+0000 7f27d52a2640 1 --2- 172.21.15.114:0/1528169654 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x556ffeb6f0a0 0x556ffeb71910 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:31:21.444 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:31:21.440+0000 7f27d52a2640 1 -- 172.21.15.114:0/1528169654 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x556ffeb6f7f0 con 0x556ffeb7a090 2024-04-23T10:31:21.444 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:31:21.440+0000 7f27d52a2640 1 --2- 172.21.15.114:0/1528169654 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x556ffeb7a090 0x556ffeb8fc30 secure :-1 s=READY pgs=660 cs=0 l=1 rev1=1 crypto rx=0x7f27c0000c00 tx=0x7f27c0031400 comp rx=0 tx=0).ready entity=mon.0 client_cookie=409d71e33ecce77 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T10:31:21.444 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:31:21.440+0000 7f27bb7fe640 1 -- 172.21.15.114:0/1528169654 <== mon.0 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f27c0035040 con 0x556ffeb7a090 2024-04-23T10:31:21.444 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:31:21.440+0000 7f27bb7fe640 1 -- 172.21.15.114:0/1528169654 <== mon.0 v2:172.21.15.114:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f27c004e070 con 0x556ffeb7a090 2024-04-23T10:31:21.444 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:31:21.440+0000 7f27bb7fe640 1 -- 172.21.15.114:0/1528169654 <== mon.0 v2:172.21.15.114:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f27c00496a0 con 0x556ffeb7a090 2024-04-23T10:31:21.444 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:31:21.441+0000 7f27daa1a740 1 -- 172.21.15.114:0/1528169654 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x556ffeb7a090 msgr2=0x556ffeb8fc30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T10:31:21.444 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:31:21.441+0000 7f27daa1a740 1 --2- 172.21.15.114:0/1528169654 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x556ffeb7a090 0x556ffeb8fc30 secure :-1 s=READY pgs=660 cs=0 l=1 rev1=1 crypto rx=0x7f27c0000c00 tx=0x7f27c0031400 comp rx=0 tx=0).stop 2024-04-23T10:31:21.444 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:31:21.441+0000 7f27daa1a740 1 -- 172.21.15.114:0/1528169654 shutdown_connections 2024-04-23T10:31:21.444 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:31:21.441+0000 7f27daa1a740 1 --2- 172.21.15.114:0/1528169654 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x556ffeb7a090 0x556ffeb8fc30 unknown :-1 s=CLOSED pgs=660 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:31:21.444 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:31:21.441+0000 7f27daa1a740 1 --2- 172.21.15.114:0/1528169654 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x556ffeb6f9c0 0x556ffeb8f6f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:31:21.444 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:31:21.441+0000 7f27daa1a740 1 --2- 172.21.15.114:0/1528169654 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x556ffeb6f0a0 0x556ffeb71910 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:31:21.444 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:31:21.441+0000 7f27daa1a740 1 -- 172.21.15.114:0/1528169654 >> 172.21.15.114:0/1528169654 conn(0x556ffeb6b3c0 msgr2=0x556ffeb790d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T10:31:21.445 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:31:21.441+0000 7f27daa1a740 1 -- 172.21.15.114:0/1528169654 shutdown_connections 2024-04-23T10:31:21.445 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:31:21.441+0000 7f27daa1a740 1 -- 172.21.15.114:0/1528169654 wait complete. 2024-04-23T10:31:21.445 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:31:21.441+0000 7f27daa1a740 1 -- start start 2024-04-23T10:31:21.445 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:31:21.441+0000 7f27daa1a740 1 Processor -- start 2024-04-23T10:31:21.445 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:31:21.441+0000 7f27daa1a740 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x556ffeb6f0a0 0x556ffeb8d9b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T10:31:21.445 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:31:21.441+0000 7f27daa1a740 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x556ffeb6f9c0 0x556ffeb8def0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T10:31:21.445 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:31:21.441+0000 7f27daa1a740 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x556ffeb7a090 0x556ffebe9490 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T10:31:21.445 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:31:21.441+0000 7f27daa1a740 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x7f27d01ae680 con 0x556ffeb7a090 2024-04-23T10:31:21.445 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:31:21.441+0000 7f27daa1a740 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 -- 0x7f27d01afaf0 con 0x556ffeb6f0a0 2024-04-23T10:31:21.445 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:31:21.441+0000 7f27daa1a740 1 -- --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_getmap magic: 0 -- 0x7f27d01aed50 con 0x556ffeb6f9c0 2024-04-23T10:31:21.445 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:31:21.441+0000 7f27daa1a740 10 client.0 _ll_register_callbacks cb 0x7f27c0009780 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-23T10:31:21.445 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:31:21.442+0000 7f27d52a2640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x556ffeb7a090 0x556ffebe9490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T10:31:21.445 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:31:21.442+0000 7f27d52a2640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x556ffeb7a090 0x556ffebe9490 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.114:37072/0 (socket says 172.21.15.114:37072) 2024-04-23T10:31:21.445 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:31:21.442+0000 7f27d52a2640 1 -- 172.21.15.114:0/1140352738 learned_addr learned my addr 172.21.15.114:0/1140352738 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-04-23T10:31:21.446 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:31:21.442+0000 7f27d5aa3640 1 --2- 172.21.15.114:0/1140352738 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x556ffeb6f9c0 0x556ffeb8def0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T10:31:21.446 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:31:21.442+0000 7f27d52a2640 1 --2- 172.21.15.114:0/1140352738 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x556ffeb6f0a0 0x556ffeb8d9b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T10:31:21.446 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:31:21.442+0000 7f27d52a2640 1 -- 172.21.15.114:0/1140352738 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x556ffeb6f9c0 msgr2=0x556ffeb8def0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T10:31:21.446 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:31:21.442+0000 7f27d52a2640 1 --2- 172.21.15.114:0/1140352738 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x556ffeb6f9c0 0x556ffeb8def0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:31:21.446 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:31:21.442+0000 7f27d52a2640 1 -- 172.21.15.114:0/1140352738 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x556ffeb6f0a0 msgr2=0x556ffeb8d9b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T10:31:21.446 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:31:21.442+0000 7f27d52a2640 1 --2- 172.21.15.114:0/1140352738 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x556ffeb6f0a0 0x556ffeb8d9b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:31:21.446 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:31:21.442+0000 7f27d52a2640 1 -- 172.21.15.114:0/1140352738 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({osdmap=0}) -- 0x556ffebe9ad0 con 0x556ffeb7a090 2024-04-23T10:31:21.446 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:31:21.442+0000 7f27d52a2640 1 --2- 172.21.15.114:0/1140352738 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x556ffeb7a090 0x556ffebe9490 secure :-1 s=READY pgs=661 cs=0 l=1 rev1=1 crypto rx=0x7f27c0001eb0 tx=0x7f27c0049360 comp rx=0 tx=0).ready entity=mon.0 client_cookie=aa79673a66ed8277 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T10:31:21.446 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:31:21.442+0000 7f27b97fa640 10 client.0 ms_handle_connect on v2:172.21.15.114:3300/0 2024-04-23T10:31:21.446 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:31:21.442+0000 7f27b97fa640 1 -- 172.21.15.114:0/1140352738 <== mon.0 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f27c0058070 con 0x556ffeb7a090 2024-04-23T10:31:21.446 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:31:21.442+0000 7f27b97fa640 1 -- 172.21.15.114:0/1140352738 <== mon.0 v2:172.21.15.114:3300/0 2 ==== osd_map(93..93 src has 1..93) ==== 8585+0+0 (secure 0 0 0) 0x7f27c0049640 con 0x556ffeb7a090 2024-04-23T10:31:21.446 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:31:21.442+0000 7f27daa1a740 1 -- 172.21.15.114:0/1140352738 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f27c004e070 con 0x556ffeb7a090 2024-04-23T10:31:21.446 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:31:21.442+0000 7f27b97fa640 1 -- 172.21.15.114:0/1140352738 <== mon.0 v2:172.21.15.114:3300/0 3 ==== mon_get_version_reply(handle=1 version=28) ==== 24+0+0 (secure 0 0 0) 0x7f27c0035030 con 0x556ffeb7a090 2024-04-23T10:31:21.447 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:31:21.442+0000 7f27daa1a740 10 client.16158 fetch_fsmap learned FSMap version 28 2024-04-23T10:31:21.447 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:31:21.442+0000 7f27daa1a740 1 -- 172.21.15.114:0/1140352738 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({config=0+,fsmap=28,monmap=4+}) -- 0x556ffebf0e20 con 0x556ffeb7a090 2024-04-23T10:31:21.447 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:31:21.442+0000 7f27b97fa640 1 -- 172.21.15.114:0/1140352738 <== mon.0 v2:172.21.15.114:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f27c004e070 con 0x556ffeb7a090 2024-04-23T10:31:21.447 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:31:21.443+0000 7f27b97fa640 1 -- 172.21.15.114:0/1140352738 <== mon.0 v2:172.21.15.114:3300/0 5 ==== fsmap(e 28) ==== 8338+0+0 (secure 0 0 0) 0x7f27c003a040 con 0x556ffeb7a090 2024-04-23T10:31:21.447 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:31:21.443+0000 7f27daa1a740 10 client.16158 fetch_fsmap finished waiting for FSMap version 28 2024-04-23T10:31:21.447 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:31:21.443+0000 7f27daa1a740 10 client.16158 resolve_mds: validated gid 24545 aka daemon mds.b 2024-04-23T10:31:21.447 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:31:21.443+0000 7f27daa1a740 20 client.16158 populate_metadata read hostname 'smithi114' 2024-04-23T10:31:21.447 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:31:21.443+0000 7f27daa1a740 1 --2- 172.21.15.114:0/1140352738 >> [v2:172.21.15.119:6838/937297340,v1:172.21.15.119:6839/937297340] conn(0x556ffebf14e0 0x556ffebf3900 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T10:31:21.447 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:31:21.443+0000 7f27daa1a740 4 client.16158 mds_command: new command op to 24545 tid=0 multi_id=0 [{"prefix": "quiesce db", "roots": ["/"], "timeout": 60.0, "expiration": 82.6, "await_for": 42.0}] 2024-04-23T10:31:21.447 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:31:21.443+0000 7f27daa1a740 1 -- 172.21.15.114:0/1140352738 --> [v2:172.21.15.119:6838/937297340,v1:172.21.15.119:6839/937297340] -- command(tid 0: {"prefix": "quiesce db", "roots": ["/"], "timeout": 60.0, "expiration": 82.6, "await_for": 42.0}) -- 0x556ffeadbc00 con 0x556ffebf14e0 2024-04-23T10:31:21.447 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:31:21.443+0000 7f27d5aa3640 1 --2- 172.21.15.114:0/1140352738 >> [v2:172.21.15.119:6838/937297340,v1:172.21.15.119:6839/937297340] conn(0x556ffebf14e0 0x556ffebf3900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T10:31:21.447 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:31:21.443+0000 7f27b97fa640 10 client.16158 ms_handle_connect on v2:172.21.15.119:6838/937297340 2024-04-23T10:31:21.447 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:31:21.443+0000 7f27d5aa3640 1 --2- 172.21.15.114:0/1140352738 >> [v2:172.21.15.119:6838/937297340,v1:172.21.15.119:6839/937297340] conn(0x556ffebf14e0 0x556ffebf3900 crc :-1 s=READY pgs=99 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=c5321ed02644b2e3 server_cookie=aadb005f7481b68d in_seq=0 out_seq=0 2024-04-23T10:31:21.918 INFO:tasks.workunit.client.0.smithi114.stdout: 1 305661 7.64 MB/sec execute 90 sec latency 4142.522 ms 2024-04-23T10:31:22.492 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 23 10:31:22 smithi119 ceph-mon[32608]: pgmap v146: 97 pgs: 97 active+clean; 94 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 23 MiB/s rd, 3.6 MiB/s wr, 348 op/s 2024-04-23T10:31:22.492 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 23 10:31:22 smithi119 ceph-mon[32608]: from='client.? 172.21.15.114:0/81410384' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-23T10:31:22.504 INFO:journalctl@ceph.mon.a.smithi114.stdout:Apr 23 10:31:22 smithi114 ceph-mon[29255]: pgmap v146: 97 pgs: 97 active+clean; 94 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 23 MiB/s rd, 3.6 MiB/s wr, 348 op/s 2024-04-23T10:31:22.504 INFO:journalctl@ceph.mon.a.smithi114.stdout:Apr 23 10:31:22 smithi114 ceph-mon[29255]: from='client.? 172.21.15.114:0/81410384' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-23T10:31:22.577 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 23 10:31:22 smithi204 ceph-mon[32649]: pgmap v146: 97 pgs: 97 active+clean; 94 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 23 MiB/s rd, 3.6 MiB/s wr, 348 op/s 2024-04-23T10:31:22.578 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 23 10:31:22 smithi204 ceph-mon[32649]: from='client.? 172.21.15.114:0/81410384' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-23T10:31:22.918 INFO:tasks.workunit.client.0.smithi114.stdout: 1 305661 7.56 MB/sec execute 91 sec latency 5142.634 ms 2024-04-23T10:31:23.918 INFO:tasks.workunit.client.0.smithi114.stdout: 1 305661 7.48 MB/sec execute 92 sec latency 6142.724 ms 2024-04-23T10:31:24.492 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 23 10:31:24 smithi119 ceph-mon[32608]: pgmap v147: 97 pgs: 97 active+clean; 87 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 24 MiB/s rd, 4.5 MiB/s wr, 451 op/s 2024-04-23T10:31:24.504 INFO:journalctl@ceph.mon.a.smithi114.stdout:Apr 23 10:31:24 smithi114 ceph-mon[29255]: pgmap v147: 97 pgs: 97 active+clean; 87 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 24 MiB/s rd, 4.5 MiB/s wr, 451 op/s 2024-04-23T10:31:24.577 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 23 10:31:24 smithi204 ceph-mon[32649]: pgmap v147: 97 pgs: 97 active+clean; 87 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 24 MiB/s rd, 4.5 MiB/s wr, 451 op/s 2024-04-23T10:31:24.918 INFO:tasks.workunit.client.0.smithi114.stdout: 1 305661 7.40 MB/sec execute 93 sec latency 7142.848 ms 2024-04-23T10:31:25.919 INFO:tasks.workunit.client.0.smithi114.stdout: 1 305661 7.32 MB/sec execute 94 sec latency 8142.935 ms 2024-04-23T10:31:26.492 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 23 10:31:26 smithi119 ceph-mon[32608]: pgmap v148: 97 pgs: 97 active+clean; 87 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 15 MiB/s rd, 2.7 MiB/s wr, 270 op/s 2024-04-23T10:31:26.504 INFO:journalctl@ceph.mon.a.smithi114.stdout:Apr 23 10:31:26 smithi114 ceph-mon[29255]: pgmap v148: 97 pgs: 97 active+clean; 87 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 15 MiB/s rd, 2.7 MiB/s wr, 270 op/s 2024-04-23T10:31:26.578 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 23 10:31:26 smithi204 ceph-mon[32649]: pgmap v148: 97 pgs: 97 active+clean; 87 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 15 MiB/s rd, 2.7 MiB/s wr, 270 op/s 2024-04-23T10:31:26.919 INFO:tasks.workunit.client.0.smithi114.stdout: 1 305661 7.24 MB/sec execute 95 sec latency 9143.040 ms 2024-04-23T10:31:27.919 INFO:tasks.workunit.client.0.smithi114.stdout: 1 305661 7.16 MB/sec execute 96 sec latency 10143.177 ms 2024-04-23T10:31:28.492 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 23 10:31:28 smithi119 ceph-mon[32608]: pgmap v149: 97 pgs: 97 active+clean; 86 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 15 MiB/s rd, 3.4 MiB/s wr, 300 op/s 2024-04-23T10:31:28.504 INFO:journalctl@ceph.mon.a.smithi114.stdout:Apr 23 10:31:28 smithi114 ceph-mon[29255]: pgmap v149: 97 pgs: 97 active+clean; 86 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 15 MiB/s rd, 3.4 MiB/s wr, 300 op/s 2024-04-23T10:31:28.577 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 23 10:31:28 smithi204 ceph-mon[32649]: pgmap v149: 97 pgs: 97 active+clean; 86 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 15 MiB/s rd, 3.4 MiB/s wr, 300 op/s 2024-04-23T10:31:28.919 INFO:tasks.workunit.client.0.smithi114.stdout: 1 305661 7.09 MB/sec execute 97 sec latency 11143.241 ms 2024-04-23T10:31:29.919 INFO:tasks.workunit.client.0.smithi114.stdout: 1 305661 7.02 MB/sec execute 98 sec latency 12143.309 ms 2024-04-23T10:31:30.380 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 23 10:31:30 smithi204 ceph-mon[32649]: pgmap v150: 97 pgs: 97 active+clean; 86 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 MiB/s rd, 1.6 MiB/s wr, 133 op/s 2024-04-23T10:31:30.491 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 23 10:31:30 smithi119 ceph-mon[32608]: pgmap v150: 97 pgs: 97 active+clean; 86 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 MiB/s rd, 1.6 MiB/s wr, 133 op/s 2024-04-23T10:31:30.504 INFO:journalctl@ceph.mon.a.smithi114.stdout:Apr 23 10:31:30 smithi114 ceph-mon[29255]: pgmap v150: 97 pgs: 97 active+clean; 86 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 MiB/s rd, 1.6 MiB/s wr, 133 op/s 2024-04-23T10:31:30.919 INFO:tasks.workunit.client.0.smithi114.stdout: 1 305661 6.95 MB/sec execute 99 sec latency 13143.399 ms 2024-04-23T10:31:31.919 INFO:tasks.workunit.client.0.smithi114.stdout: 1 305661 6.88 MB/sec execute 100 sec latency 14143.460 ms 2024-04-23T10:31:32.318 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:31:32.317+0000 7f4d8d907640 1 -- 172.21.15.114:0/3293136528 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f4d881061b0 msgr2=0x7f4d8810dd80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T10:31:32.318 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:31:32.317+0000 7f4d8d907640 1 --2- 172.21.15.114:0/3293136528 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f4d881061b0 0x7f4d8810dd80 secure :-1 s=READY pgs=662 cs=0 l=1 rev1=1 crypto rx=0x7f4d80009ca0 tx=0x7f4d8002a8c0 comp rx=0 tx=0).stop 2024-04-23T10:31:32.318 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:31:32.317+0000 7f4d8d907640 1 -- 172.21.15.114:0/3293136528 shutdown_connections 2024-04-23T10:31:32.318 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:31:32.317+0000 7f4d8d907640 1 --2- 172.21.15.114:0/3293136528 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f4d881061b0 0x7f4d8810dd80 unknown :-1 s=CLOSED pgs=662 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:31:32.318 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:31:32.317+0000 7f4d8d907640 1 --2- 172.21.15.114:0/3293136528 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f4d88101a90 0x7f4d88105ad0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:31:32.318 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:31:32.317+0000 7f4d8d907640 1 --2- 172.21.15.114:0/3293136528 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f4d881010e0 0x7f4d881014c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:31:32.319 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:31:32.317+0000 7f4d8d907640 1 -- 172.21.15.114:0/3293136528 >> 172.21.15.114:0/3293136528 conn(0x7f4d880771d0 msgr2=0x7f4d880775e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T10:31:32.319 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:31:32.317+0000 7f4d8d907640 1 -- 172.21.15.114:0/3293136528 shutdown_connections 2024-04-23T10:31:32.319 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:31:32.317+0000 7f4d8d907640 1 -- 172.21.15.114:0/3293136528 wait complete. 2024-04-23T10:31:32.319 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:31:32.318+0000 7f4d8d907640 1 Processor -- start 2024-04-23T10:31:32.319 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:31:32.318+0000 7f4d8d907640 1 -- start start 2024-04-23T10:31:32.319 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:31:32.318+0000 7f4d8d907640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f4d881010e0 0x7f4d881975a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T10:31:32.319 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:31:32.318+0000 7f4d8d907640 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f4d881061b0 0x7f4d88197ae0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T10:31:32.319 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:31:32.318+0000 7f4d8d907640 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f4d8819be10 0x7f4d8819c2c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T10:31:32.319 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:31:32.318+0000 7f4d8d907640 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x7f4d88111120 con 0x7f4d881010e0 2024-04-23T10:31:32.319 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:31:32.318+0000 7f4d8d907640 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 -- 0x7f4d88110fa0 con 0x7f4d8819be10 2024-04-23T10:31:32.319 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:31:32.318+0000 7f4d8d907640 1 -- --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_getmap magic: 0 -- 0x7f4d881112a0 con 0x7f4d881061b0 2024-04-23T10:31:32.319 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:31:32.318+0000 7f4d86ffd640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f4d881010e0 0x7f4d881975a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T10:31:32.319 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:31:32.318+0000 7f4d86ffd640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f4d881010e0 0x7f4d881975a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.114:52222/0 (socket says 172.21.15.114:52222) 2024-04-23T10:31:32.319 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:31:32.318+0000 7f4d86ffd640 1 -- 172.21.15.114:0/2157914180 learned_addr learned my addr 172.21.15.114:0/2157914180 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-04-23T10:31:32.319 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:31:32.318+0000 7f4d877fe640 1 --2- 172.21.15.114:0/2157914180 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f4d8819be10 0x7f4d8819c2c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T10:31:32.319 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:31:32.318+0000 7f4d7ffff640 1 --2- 172.21.15.114:0/2157914180 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f4d881061b0 0x7f4d88197ae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T10:31:32.320 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:31:32.318+0000 7f4d86ffd640 1 -- 172.21.15.114:0/2157914180 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f4d881061b0 msgr2=0x7f4d88197ae0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T10:31:32.320 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:31:32.318+0000 7f4d86ffd640 1 --2- 172.21.15.114:0/2157914180 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f4d881061b0 0x7f4d88197ae0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:31:32.320 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:31:32.318+0000 7f4d86ffd640 1 -- 172.21.15.114:0/2157914180 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f4d8819be10 msgr2=0x7f4d8819c2c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T10:31:32.320 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:31:32.318+0000 7f4d86ffd640 1 --2- 172.21.15.114:0/2157914180 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f4d8819be10 0x7f4d8819c2c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:31:32.320 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:31:32.318+0000 7f4d86ffd640 1 -- 172.21.15.114:0/2157914180 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f4d8819c900 con 0x7f4d881010e0 2024-04-23T10:31:32.320 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:31:32.318+0000 7f4d86ffd640 1 --2- 172.21.15.114:0/2157914180 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f4d881010e0 0x7f4d881975a0 secure :-1 s=READY pgs=663 cs=0 l=1 rev1=1 crypto rx=0x7f4d7800c970 tx=0x7f4d7800ce30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T10:31:32.320 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:31:32.318+0000 7f4d8c905640 1 -- 172.21.15.114:0/2157914180 <== mon.0 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4d78007bb0 con 0x7f4d881010e0 2024-04-23T10:31:32.320 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:31:32.319+0000 7f4d8c905640 1 -- 172.21.15.114:0/2157914180 <== mon.0 v2:172.21.15.114:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f4d78007d50 con 0x7f4d881010e0 2024-04-23T10:31:32.320 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:31:32.319+0000 7f4d8c905640 1 -- 172.21.15.114:0/2157914180 <== mon.0 v2:172.21.15.114:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4d7800f450 con 0x7f4d881010e0 2024-04-23T10:31:32.320 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:31:32.319+0000 7f4d8d907640 1 -- 172.21.15.114:0/2157914180 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f4d8819cbf0 con 0x7f4d881010e0 2024-04-23T10:31:32.320 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:31:32.319+0000 7f4d8d907640 1 -- 172.21.15.114:0/2157914180 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f4d881a43e0 con 0x7f4d881010e0 2024-04-23T10:31:32.326 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:31:32.321+0000 7f4d8c905640 1 -- 172.21.15.114:0/2157914180 <== mon.0 v2:172.21.15.114:3300/0 4 ==== mgrmap(e 26) ==== 138778+0+0 (secure 0 0 0) 0x7f4d78004f20 con 0x7f4d881010e0 2024-04-23T10:31:32.326 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:31:32.321+0000 7f4d90617740 1 Processor -- start 2024-04-23T10:31:32.327 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:31:32.321+0000 7f4d90617740 1 -- start start 2024-04-23T10:31:32.327 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:31:32.322+0000 7f4d90617740 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x5606f46fc4c0 0x5606f46fc8a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T10:31:32.327 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:31:32.322+0000 7f4d90617740 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x5606f46fcde0 0x5606f47094d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T10:31:32.327 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:31:32.322+0000 7f4d90617740 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x5606f4709a10 0x5606f470be00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T10:31:32.327 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:31:32.322+0000 7f4d90617740 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x5606f4661970 con 0x5606f46fcde0 2024-04-23T10:31:32.327 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:31:32.322+0000 7f4d90617740 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 -- 0x5606f45fe4a0 con 0x5606f4709a10 2024-04-23T10:31:32.327 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:31:32.322+0000 7f4d90617740 1 -- --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_getmap magic: 0 -- 0x5606f465d930 con 0x5606f46fc4c0 2024-04-23T10:31:32.327 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:31:32.322+0000 7f4d86ffd640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x5606f46fcde0 0x5606f47094d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T10:31:32.327 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:31:32.322+0000 7f4d86ffd640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x5606f46fcde0 0x5606f47094d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.114:52228/0 (socket says 172.21.15.114:52228) 2024-04-23T10:31:32.327 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:31:32.322+0000 7f4d86ffd640 1 -- 172.21.15.114:0/1094002580 learned_addr learned my addr 172.21.15.114:0/1094002580 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-04-23T10:31:32.327 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:31:32.322+0000 7f4d877fe640 1 --2- 172.21.15.114:0/1094002580 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x5606f46fc4c0 0x5606f46fc8a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T10:31:32.327 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:31:32.322+0000 7f4d8c905640 1 --2- 172.21.15.114:0/2157914180 >> [v2:172.21.15.114:6800/1395693466,v1:172.21.15.114:6801/1395693466] conn(0x7f4d740a3e40 0x7f4d740a6300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T10:31:32.327 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:31:32.322+0000 7f4d7ffff640 1 --2- 172.21.15.114:0/1094002580 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x5606f4709a10 0x5606f470be00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T10:31:32.327 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:31:32.322+0000 7f4d877fe640 1 --2- 172.21.15.114:0/2157914180 >> [v2:172.21.15.114:6800/1395693466,v1:172.21.15.114:6801/1395693466] conn(0x7f4d740a3e40 0x7f4d740a6300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T10:31:32.327 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:31:32.322+0000 7f4d86ffd640 1 -- 172.21.15.114:0/1094002580 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x5606f46fc4c0 msgr2=0x5606f46fc8a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T10:31:32.328 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:31:32.322+0000 7f4d86ffd640 1 --2- 172.21.15.114:0/1094002580 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x5606f46fc4c0 0x5606f46fc8a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:31:32.328 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:31:32.322+0000 7f4d86ffd640 1 -- 172.21.15.114:0/1094002580 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x5606f4709a10 msgr2=0x5606f470be00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T10:31:32.328 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:31:32.322+0000 7f4d86ffd640 1 --2- 172.21.15.114:0/1094002580 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x5606f4709a10 0x5606f470be00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:31:32.328 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:31:32.322+0000 7f4d86ffd640 1 -- 172.21.15.114:0/1094002580 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5606f4374210 con 0x5606f46fcde0 2024-04-23T10:31:32.328 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:31:32.322+0000 7f4d8c905640 1 -- 172.21.15.114:0/2157914180 <== mon.0 v2:172.21.15.114:3300/0 5 ==== osd_map(93..93 src has 1..93) ==== 8585+0+0 (secure 0 0 0) 0x7f4d780d05e0 con 0x7f4d881010e0 2024-04-23T10:31:32.328 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:31:32.322+0000 7f4d86ffd640 1 --2- 172.21.15.114:0/1094002580 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x5606f46fcde0 0x5606f47094d0 secure :-1 s=READY pgs=664 cs=0 l=1 rev1=1 crypto rx=0x7f4d78098f10 tx=0x7f4d7809c1b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=83b0d9171037d846 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T10:31:32.328 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:31:32.323+0000 7f4d877fe640 1 --2- 172.21.15.114:0/2157914180 >> [v2:172.21.15.114:6800/1395693466,v1:172.21.15.114:6801/1395693466] conn(0x7f4d740a3e40 0x7f4d740a6300 secure :-1 s=READY pgs=102 cs=0 l=1 rev1=1 crypto rx=0x7f4d8002f500 tx=0x7f4d8002a8c0 comp rx=0 tx=0).ready entity=mgr.15291 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T10:31:32.328 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:31:32.323+0000 7f4d7dffb640 1 -- 172.21.15.114:0/1094002580 <== mon.0 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4d7801d030 con 0x5606f46fcde0 2024-04-23T10:31:32.328 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:31:32.323+0000 7f4d7dffb640 1 -- 172.21.15.114:0/1094002580 <== mon.0 v2:172.21.15.114:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f4d780055f0 con 0x5606f46fcde0 2024-04-23T10:31:32.328 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:31:32.323+0000 7f4d7dffb640 1 -- 172.21.15.114:0/1094002580 <== mon.0 v2:172.21.15.114:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4d780cdf70 con 0x5606f46fcde0 2024-04-23T10:31:32.328 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:31:32.323+0000 7f4d90617740 1 -- 172.21.15.114:0/1094002580 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x5606f46fcde0 msgr2=0x5606f47094d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T10:31:32.328 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:31:32.323+0000 7f4d90617740 1 --2- 172.21.15.114:0/1094002580 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x5606f46fcde0 0x5606f47094d0 secure :-1 s=READY pgs=664 cs=0 l=1 rev1=1 crypto rx=0x7f4d78098f10 tx=0x7f4d7809c1b0 comp rx=0 tx=0).stop 2024-04-23T10:31:32.328 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:31:32.323+0000 7f4d90617740 1 -- 172.21.15.114:0/1094002580 shutdown_connections 2024-04-23T10:31:32.328 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:31:32.323+0000 7f4d90617740 1 --2- 172.21.15.114:0/1094002580 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x5606f4709a10 0x5606f470be00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:31:32.328 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:31:32.323+0000 7f4d90617740 1 --2- 172.21.15.114:0/1094002580 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x5606f46fcde0 0x5606f47094d0 unknown :-1 s=CLOSED pgs=664 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:31:32.329 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:31:32.323+0000 7f4d90617740 1 --2- 172.21.15.114:0/1094002580 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x5606f46fc4c0 0x5606f46fc8a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:31:32.329 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:31:32.323+0000 7f4d90617740 1 -- 172.21.15.114:0/1094002580 >> 172.21.15.114:0/1094002580 conn(0x5606f46f8b30 msgr2=0x5606f46f8f10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T10:31:32.329 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:31:32.323+0000 7f4d90617740 1 -- 172.21.15.114:0/1094002580 shutdown_connections 2024-04-23T10:31:32.329 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:31:32.323+0000 7f4d90617740 1 -- 172.21.15.114:0/1094002580 wait complete. 2024-04-23T10:31:32.329 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:31:32.324+0000 7f4d90617740 1 -- start start 2024-04-23T10:31:32.329 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:31:32.324+0000 7f4d90617740 1 Processor -- start 2024-04-23T10:31:32.329 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:31:32.324+0000 7f4d90617740 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x5606f46fc4c0 0x5606f4709360 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T10:31:32.329 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:31:32.324+0000 7f4d90617740 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x5606f46fcde0 0x5606f4706650 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T10:31:32.329 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:31:32.324+0000 7f4d90617740 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x5606f4709a10 0x5606f4706bb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T10:31:32.329 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:31:32.324+0000 7f4d90617740 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x5606f470c5e0 con 0x5606f46fcde0 2024-04-23T10:31:32.329 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:31:32.324+0000 7f4d90617740 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 -- 0x5606f45fc0b0 con 0x5606f46fc4c0 2024-04-23T10:31:32.329 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:31:32.324+0000 7f4d90617740 1 -- --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_getmap magic: 0 -- 0x5606f470c760 con 0x5606f4709a10 2024-04-23T10:31:32.329 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:31:32.324+0000 7f4d90617740 10 client.0 _ll_register_callbacks cb 0x5606f45e9d30 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-23T10:31:32.329 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:31:32.324+0000 7f4d7ffff640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x5606f46fcde0 0x5606f4706650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T10:31:32.329 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:31:32.324+0000 7f4d7ffff640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x5606f46fcde0 0x5606f4706650 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.114:52242/0 (socket says 172.21.15.114:52242) 2024-04-23T10:31:32.330 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:31:32.324+0000 7f4d7ffff640 1 -- 172.21.15.114:0/1298260094 learned_addr learned my addr 172.21.15.114:0/1298260094 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-04-23T10:31:32.330 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:31:32.324+0000 7f4d877fe640 1 --2- 172.21.15.114:0/1298260094 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x5606f4709a10 0x5606f4706bb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T10:31:32.330 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:31:32.324+0000 7f4d86ffd640 1 --2- 172.21.15.114:0/1298260094 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x5606f46fc4c0 0x5606f4709360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T10:31:32.330 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:31:32.324+0000 7f4d7ffff640 1 -- 172.21.15.114:0/1298260094 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x5606f4709a10 msgr2=0x5606f4706bb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T10:31:32.330 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:31:32.324+0000 7f4d7ffff640 1 --2- 172.21.15.114:0/1298260094 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x5606f4709a10 0x5606f4706bb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:31:32.330 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:31:32.324+0000 7f4d7ffff640 1 -- 172.21.15.114:0/1298260094 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x5606f46fc4c0 msgr2=0x5606f4709360 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T10:31:32.330 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:31:32.325+0000 7f4d7ffff640 1 --2- 172.21.15.114:0/1298260094 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x5606f46fc4c0 0x5606f4709360 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:31:32.330 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:31:32.325+0000 7f4d7ffff640 1 -- 172.21.15.114:0/1298260094 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5606f4440620 con 0x5606f46fcde0 2024-04-23T10:31:32.330 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:31:32.325+0000 7f4d86ffd640 1 --2- 172.21.15.114:0/1298260094 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x5606f46fc4c0 0x5606f4709360 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-23T10:31:32.330 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:31:32.325+0000 7f4d7ffff640 1 --2- 172.21.15.114:0/1298260094 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x5606f46fcde0 0x5606f4706650 secure :-1 s=READY pgs=665 cs=0 l=1 rev1=1 crypto rx=0x7f4d70013460 tx=0x7f4d70013920 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1ac0637f2e875e73 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T10:31:32.330 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:31:32.325+0000 7f4d66ffd640 10 client.0 ms_handle_connect on v2:172.21.15.114:3300/0 2024-04-23T10:31:32.330 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:31:32.325+0000 7f4d66ffd640 1 -- 172.21.15.114:0/1298260094 <== mon.0 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4d70004020 con 0x5606f46fcde0 2024-04-23T10:31:32.330 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:31:32.325+0000 7f4d66ffd640 1 -- 172.21.15.114:0/1298260094 <== mon.0 v2:172.21.15.114:3300/0 2 ==== osd_map(93..93 src has 1..93) ==== 8585+0+0 (secure 0 0 0) 0x7f4d700041c0 con 0x5606f46fcde0 2024-04-23T10:31:32.330 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:31:32.325+0000 7f4d90617740 1 -- 172.21.15.114:0/1298260094 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x5606f466f390 con 0x5606f46fcde0 2024-04-23T10:31:32.331 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:31:32.325+0000 7f4d66ffd640 1 -- 172.21.15.114:0/1298260094 <== mon.0 v2:172.21.15.114:3300/0 3 ==== mon_get_version_reply(handle=1 version=28) ==== 24+0+0 (secure 0 0 0) 0x7f4d700043c0 con 0x5606f46fcde0 2024-04-23T10:31:32.331 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:31:32.325+0000 7f4d90617740 10 client.16170 fetch_fsmap learned FSMap version 28 2024-04-23T10:31:32.331 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:31:32.325+0000 7f4d90617740 1 -- 172.21.15.114:0/1298260094 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({config=0+,fsmap=28,monmap=4+}) -- 0x5606f47185b0 con 0x5606f46fcde0 2024-04-23T10:31:32.331 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:31:32.325+0000 7f4d66ffd640 1 -- 172.21.15.114:0/1298260094 <== mon.0 v2:172.21.15.114:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x5606f466f390 con 0x5606f46fcde0 2024-04-23T10:31:32.331 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:31:32.325+0000 7f4d66ffd640 1 -- 172.21.15.114:0/1298260094 <== mon.0 v2:172.21.15.114:3300/0 5 ==== fsmap(e 28) ==== 8338+0+0 (secure 0 0 0) 0x7f4d7000f040 con 0x5606f46fcde0 2024-04-23T10:31:32.331 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:31:32.325+0000 7f4d90617740 10 client.16170 fetch_fsmap finished waiting for FSMap version 28 2024-04-23T10:31:32.331 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:31:32.325+0000 7f4d90617740 10 client.16170 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-04-23T10:31:32.331 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:31:32.325+0000 7f4d90617740 20 client.16170 populate_metadata read hostname 'smithi114' 2024-04-23T10:31:32.331 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:31:32.325+0000 7f4d90617740 1 --2- 172.21.15.114:0/1298260094 >> [v2:172.21.15.119:6838/937297340,v1:172.21.15.119:6839/937297340] conn(0x5606f4719b30 0x5606f471bf20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T10:31:32.331 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:31:32.325+0000 7f4d90617740 4 client.16170 mds_command: new command op to 24545 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-23T10:31:32.331 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:31:32.325+0000 7f4d90617740 1 -- 172.21.15.114:0/1298260094 --> [v2:172.21.15.119:6838/937297340,v1:172.21.15.119:6839/937297340] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x5606f4669200 con 0x5606f4719b30 2024-04-23T10:31:32.331 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:31:32.326+0000 7f4d86ffd640 1 --2- 172.21.15.114:0/1298260094 >> [v2:172.21.15.119:6838/937297340,v1:172.21.15.119:6839/937297340] conn(0x5606f4719b30 0x5606f471bf20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T10:31:32.331 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:31:32.326+0000 7f4d86ffd640 1 --2- 172.21.15.114:0/1298260094 >> [v2:172.21.15.119:6838/937297340,v1:172.21.15.119:6839/937297340] conn(0x5606f4719b30 0x5606f471bf20 crc :-1 s=READY pgs=100 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=3838f0f4e3c3c2ca server_cookie=4c036ea6ba9e8432 in_seq=0 out_seq=0 2024-04-23T10:31:32.331 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:31:32.326+0000 7f4d66ffd640 10 client.16170 ms_handle_connect on v2:172.21.15.119:6838/937297340 2024-04-23T10:31:32.331 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:31:32.327+0000 7f4d66ffd640 1 -- 172.21.15.114:0/1298260094 <== mds.0 v2:172.21.15.119:6838/937297340 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x5606f4669200 con 0x5606f4719b30 2024-04-23T10:31:32.331 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:31:32.327+0000 7f4d66ffd640 10 client.16170 handle_command_reply: tid=0 2024-04-23T10:31:32.332 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:31:32.327+0000 7f4d90617740 1 client.16170 shutdown 2024-04-23T10:31:32.332 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:31:32.327+0000 7f4d90617740 1 -- 172.21.15.114:0/1298260094 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x5606f46fcde0 msgr2=0x5606f4706650 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T10:31:32.332 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:31:32.327+0000 7f4d90617740 1 --2- 172.21.15.114:0/1298260094 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x5606f46fcde0 0x5606f4706650 secure :-1 s=READY pgs=665 cs=0 l=1 rev1=1 crypto rx=0x7f4d70013460 tx=0x7f4d70013920 comp rx=0 tx=0).stop 2024-04-23T10:31:32.332 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:31:32.327+0000 7f4d90617740 1 -- 172.21.15.114:0/1298260094 shutdown_connections 2024-04-23T10:31:32.332 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:31:32.327+0000 7f4d90617740 1 --2- 172.21.15.114:0/1298260094 >> [v2:172.21.15.119:6838/937297340,v1:172.21.15.119:6839/937297340] conn(0x5606f4719b30 0x5606f471bf20 crc :-1 s=READY pgs=100 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:31:32.332 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:31:32.327+0000 7f4d90617740 1 --2- 172.21.15.114:0/1298260094 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x5606f4709a10 0x5606f4706bb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:31:32.332 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:31:32.327+0000 7f4d90617740 1 --2- 172.21.15.114:0/1298260094 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x5606f46fcde0 0x5606f4706650 unknown :-1 s=CLOSED pgs=665 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:31:32.332 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:31:32.327+0000 7f4d66ffd640 0 client.16170 ms_handle_reset on v2:172.21.15.119:6838/937297340 2024-04-23T10:31:32.332 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:31:32.327+0000 7f4d90617740 1 --2- 172.21.15.114:0/1298260094 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x5606f46fc4c0 0x5606f4709360 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:31:32.332 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:31:32.327+0000 7f4d90617740 1 -- 172.21.15.114:0/1298260094 >> 172.21.15.114:0/1298260094 conn(0x5606f46f8b30 msgr2=0x5606f46fe9a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T10:31:32.332 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:31:32.327+0000 7f4d90617740 1 -- 172.21.15.114:0/1298260094 shutdown_connections 2024-04-23T10:31:32.332 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:31:32.327+0000 7f4d90617740 1 -- 172.21.15.114:0/1298260094 wait complete. 2024-04-23T10:31:32.332 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:31:32.327+0000 7f4d90617740 20 client.16170 trim_cache size 0 max 16384 2024-04-23T10:31:32.345 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:31:32.342+0000 7f4d90617740 1 Processor -- start 2024-04-23T10:31:32.346 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:31:32.342+0000 7f4d90617740 1 -- start start 2024-04-23T10:31:32.346 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:31:32.342+0000 7f4d90617740 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x5606f46fc4c0 0x5606f47154a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T10:31:32.346 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:31:32.342+0000 7f4d90617740 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x5606f46fcde0 0x5606f4708e90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T10:31:32.346 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:31:32.342+0000 7f4d90617740 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x5606f4709a10 0x5606f4721830 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T10:31:32.346 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:31:32.342+0000 7f4d90617740 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x7f4d881ae3a0 con 0x5606f46fcde0 2024-04-23T10:31:32.346 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:31:32.342+0000 7f4d90617740 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 -- 0x7f4d881abc70 con 0x5606f4709a10 2024-04-23T10:31:32.346 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:31:32.342+0000 7f4d90617740 1 -- --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_getmap magic: 0 -- 0x7f4d881a9ac0 con 0x5606f46fc4c0 2024-04-23T10:31:32.346 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:31:32.342+0000 7f4d7ffff640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x5606f46fcde0 0x5606f4708e90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T10:31:32.346 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:31:32.342+0000 7f4d7ffff640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x5606f46fcde0 0x5606f4708e90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.114:52252/0 (socket says 172.21.15.114:52252) 2024-04-23T10:31:32.346 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:31:32.343+0000 7f4d7ffff640 1 -- 172.21.15.114:0/2857413551 learned_addr learned my addr 172.21.15.114:0/2857413551 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-04-23T10:31:32.346 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:31:32.343+0000 7f4d86ffd640 1 --2- 172.21.15.114:0/2857413551 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x5606f46fc4c0 0x5606f47154a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T10:31:32.346 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:31:32.343+0000 7f4d7ffff640 1 -- 172.21.15.114:0/2857413551 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x5606f46fc4c0 msgr2=0x5606f47154a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T10:31:32.346 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:31:32.343+0000 7f4d7ffff640 1 --2- 172.21.15.114:0/2857413551 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x5606f46fc4c0 0x5606f47154a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:31:32.346 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:31:32.343+0000 7f4d7ffff640 1 -- 172.21.15.114:0/2857413551 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x5606f4709a10 msgr2=0x5606f4721830 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T10:31:32.347 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:31:32.343+0000 7f4d877fe640 1 --2- 172.21.15.114:0/2857413551 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x5606f4709a10 0x5606f4721830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T10:31:32.347 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:31:32.343+0000 7f4d7ffff640 1 --2- 172.21.15.114:0/2857413551 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x5606f4709a10 0x5606f4721830 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:31:32.347 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:31:32.343+0000 7f4d7ffff640 1 -- 172.21.15.114:0/2857413551 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5606f4706f20 con 0x5606f46fcde0 2024-04-23T10:31:32.347 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:31:32.343+0000 7f4d877fe640 1 --2- 172.21.15.114:0/2857413551 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x5606f4709a10 0x5606f4721830 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-23T10:31:32.347 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:31:32.343+0000 7f4d7ffff640 1 --2- 172.21.15.114:0/2857413551 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x5606f46fcde0 0x5606f4708e90 secure :-1 s=READY pgs=666 cs=0 l=1 rev1=1 crypto rx=0x7f4d70003b60 tx=0x7f4d70005cd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=646d1163d76ea084 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T10:31:32.347 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:31:32.343+0000 7f4d7d7fa640 1 -- 172.21.15.114:0/2857413551 <== mon.0 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4d70003d50 con 0x5606f46fcde0 2024-04-23T10:31:32.347 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:31:32.343+0000 7f4d7d7fa640 1 -- 172.21.15.114:0/2857413551 <== mon.0 v2:172.21.15.114:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f4d70005020 con 0x5606f46fcde0 2024-04-23T10:31:32.347 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:31:32.343+0000 7f4d7d7fa640 1 -- 172.21.15.114:0/2857413551 <== mon.0 v2:172.21.15.114:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4d70011c30 con 0x5606f46fcde0 2024-04-23T10:31:32.347 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:31:32.344+0000 7f4d90617740 1 -- 172.21.15.114:0/2857413551 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x5606f46fcde0 msgr2=0x5606f4708e90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T10:31:32.347 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:31:32.344+0000 7f4d90617740 1 --2- 172.21.15.114:0/2857413551 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x5606f46fcde0 0x5606f4708e90 secure :-1 s=READY pgs=666 cs=0 l=1 rev1=1 crypto rx=0x7f4d70003b60 tx=0x7f4d70005cd0 comp rx=0 tx=0).stop 2024-04-23T10:31:32.347 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:31:32.344+0000 7f4d90617740 1 -- 172.21.15.114:0/2857413551 shutdown_connections 2024-04-23T10:31:32.347 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:31:32.344+0000 7f4d90617740 1 --2- 172.21.15.114:0/2857413551 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x5606f4709a10 0x5606f4721830 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:31:32.347 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:31:32.344+0000 7f4d90617740 1 --2- 172.21.15.114:0/2857413551 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x5606f46fcde0 0x5606f4708e90 unknown :-1 s=CLOSED pgs=666 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:31:32.347 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:31:32.344+0000 7f4d90617740 1 --2- 172.21.15.114:0/2857413551 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x5606f46fc4c0 0x5606f47154a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:31:32.347 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:31:32.344+0000 7f4d90617740 1 -- 172.21.15.114:0/2857413551 >> 172.21.15.114:0/2857413551 conn(0x5606f46f8b30 msgr2=0x5606f46f7cb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T10:31:32.348 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:31:32.344+0000 7f4d90617740 1 -- 172.21.15.114:0/2857413551 shutdown_connections 2024-04-23T10:31:32.348 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:31:32.344+0000 7f4d90617740 1 -- 172.21.15.114:0/2857413551 wait complete. 2024-04-23T10:31:32.348 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:31:32.344+0000 7f4d90617740 1 -- start start 2024-04-23T10:31:32.348 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:31:32.344+0000 7f4d90617740 1 Processor -- start 2024-04-23T10:31:32.348 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:31:32.344+0000 7f4d90617740 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x5606f46fc4c0 0x5606f4700010 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T10:31:32.348 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:31:32.344+0000 7f4d90617740 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x5606f46fcde0 0x5606f4700550 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T10:31:32.348 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:31:32.344+0000 7f4d90617740 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x5606f4709a10 0x5606f4700a90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T10:31:32.348 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:31:32.344+0000 7f4d90617740 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x7f4d881aa130 con 0x5606f46fcde0 2024-04-23T10:31:32.348 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:31:32.344+0000 7f4d90617740 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 -- 0x7f4d881ab5a0 con 0x5606f4709a10 2024-04-23T10:31:32.348 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:31:32.344+0000 7f4d90617740 1 -- --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_getmap magic: 0 -- 0x7f4d881aa800 con 0x5606f46fc4c0 2024-04-23T10:31:32.348 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:31:32.344+0000 7f4d90617740 10 client.0 _ll_register_callbacks cb 0x7f4d70009c40 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-23T10:31:32.348 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:31:32.345+0000 7f4d7ffff640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x5606f46fcde0 0x5606f4700550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T10:31:32.348 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:31:32.345+0000 7f4d7ffff640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x5606f46fcde0 0x5606f4700550 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.114:52268/0 (socket says 172.21.15.114:52268) 2024-04-23T10:31:32.348 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:31:32.345+0000 7f4d7ffff640 1 -- 172.21.15.114:0/2774024442 learned_addr learned my addr 172.21.15.114:0/2774024442 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-04-23T10:31:32.348 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:31:32.345+0000 7f4d86ffd640 1 --2- 172.21.15.114:0/2774024442 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x5606f46fc4c0 0x5606f4700010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T10:31:32.348 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:31:32.345+0000 7f4d877fe640 1 --2- 172.21.15.114:0/2774024442 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x5606f4709a10 0x5606f4700a90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T10:31:32.349 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:31:32.345+0000 7f4d7ffff640 1 -- 172.21.15.114:0/2774024442 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x5606f46fc4c0 msgr2=0x5606f4700010 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T10:31:32.349 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:31:32.345+0000 7f4d7ffff640 1 --2- 172.21.15.114:0/2774024442 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x5606f46fc4c0 0x5606f4700010 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:31:32.349 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:31:32.345+0000 7f4d7ffff640 1 -- 172.21.15.114:0/2774024442 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x5606f4709a10 msgr2=0x5606f4700a90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T10:31:32.349 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:31:32.345+0000 7f4d7ffff640 1 --2- 172.21.15.114:0/2774024442 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x5606f4709a10 0x5606f4700a90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:31:32.349 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:31:32.345+0000 7f4d7ffff640 1 -- 172.21.15.114:0/2774024442 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5606f47810c0 con 0x5606f46fcde0 2024-04-23T10:31:32.349 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:31:32.345+0000 7f4d7ffff640 1 --2- 172.21.15.114:0/2774024442 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x5606f46fcde0 0x5606f4700550 secure :-1 s=READY pgs=667 cs=0 l=1 rev1=1 crypto rx=0x7f4d70013460 tx=0x7f4d70003b90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9649e8f4c3951b86 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T10:31:32.349 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:31:32.345+0000 7f4d66ffd640 10 client.0 ms_handle_connect on v2:172.21.15.114:3300/0 2024-04-23T10:31:32.349 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:31:32.345+0000 7f4d66ffd640 1 -- 172.21.15.114:0/2774024442 <== mon.0 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4d7003b060 con 0x5606f46fcde0 2024-04-23T10:31:32.349 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:31:32.345+0000 7f4d66ffd640 1 -- 172.21.15.114:0/2774024442 <== mon.0 v2:172.21.15.114:3300/0 2 ==== osd_map(93..93 src has 1..93) ==== 8585+0+0 (secure 0 0 0) 0x7f4d700119b0 con 0x5606f46fcde0 2024-04-23T10:31:32.349 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:31:32.345+0000 7f4d90617740 1 -- 172.21.15.114:0/2774024442 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f4d70005020 con 0x5606f46fcde0 2024-04-23T10:31:32.349 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:31:32.345+0000 7f4d66ffd640 1 -- 172.21.15.114:0/2774024442 <== mon.0 v2:172.21.15.114:3300/0 3 ==== mon_get_version_reply(handle=1 version=28) ==== 24+0+0 (secure 0 0 0) 0x7f4d70032040 con 0x5606f46fcde0 2024-04-23T10:31:32.349 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:31:32.345+0000 7f4d90617740 10 client.16176 fetch_fsmap learned FSMap version 28 2024-04-23T10:31:32.349 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:31:32.345+0000 7f4d90617740 1 -- 172.21.15.114:0/2774024442 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({config=0+,fsmap=28,monmap=4+}) -- 0x5606f47822a0 con 0x5606f46fcde0 2024-04-23T10:31:32.349 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:31:32.346+0000 7f4d66ffd640 1 -- 172.21.15.114:0/2774024442 <== mon.0 v2:172.21.15.114:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f4d70005020 con 0x5606f46fcde0 2024-04-23T10:31:32.349 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:31:32.346+0000 7f4d66ffd640 1 -- 172.21.15.114:0/2774024442 <== mon.0 v2:172.21.15.114:3300/0 5 ==== fsmap(e 28) ==== 8338+0+0 (secure 0 0 0) 0x7f4d70035070 con 0x5606f46fcde0 2024-04-23T10:31:32.350 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:31:32.346+0000 7f4d90617740 10 client.16176 fetch_fsmap finished waiting for FSMap version 28 2024-04-23T10:31:32.350 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:31:32.346+0000 7f4d90617740 10 client.16176 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-04-23T10:31:32.350 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:31:32.346+0000 7f4d90617740 20 client.16176 populate_metadata read hostname 'smithi114' 2024-04-23T10:31:32.350 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:31:32.346+0000 7f4d90617740 1 --2- 172.21.15.114:0/2774024442 >> [v2:172.21.15.119:6838/937297340,v1:172.21.15.119:6839/937297340] conn(0x5606f4782850 0x5606f4784c70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T10:31:32.350 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:31:32.346+0000 7f4d90617740 4 client.16176 mds_command: new command op to 24545 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-04-23T10:31:32.350 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:31:32.346+0000 7f4d90617740 1 -- 172.21.15.114:0/2774024442 --> [v2:172.21.15.119:6838/937297340,v1:172.21.15.119:6839/937297340] -- command(tid 0: {"prefix": "flush journal"}) -- 0x5606f45e55f0 con 0x5606f4782850 2024-04-23T10:31:32.350 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:31:32.346+0000 7f4d86ffd640 1 --2- 172.21.15.114:0/2774024442 >> [v2:172.21.15.119:6838/937297340,v1:172.21.15.119:6839/937297340] conn(0x5606f4782850 0x5606f4784c70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T10:31:32.350 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:31:32.346+0000 7f4d86ffd640 1 --2- 172.21.15.114:0/2774024442 >> [v2:172.21.15.119:6838/937297340,v1:172.21.15.119:6839/937297340] conn(0x5606f4782850 0x5606f4784c70 crc :-1 s=READY pgs=101 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=43c133a77348f104 server_cookie=c125295ad8ea9a44 in_seq=0 out_seq=0 2024-04-23T10:31:32.350 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:31:32.346+0000 7f4d66ffd640 10 client.16176 ms_handle_connect on v2:172.21.15.119:6838/937297340 2024-04-23T10:31:32.372 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:31:32.373+0000 7f4d66ffd640 1 -- 172.21.15.114:0/2774024442 <== mds.0 v2:172.21.15.119:6838/937297340 1 ==== command_reply(tid 0: 0 ) ==== 8+0+44 (crc 0 0 0) 0x5606f45e55f0 con 0x5606f4782850 2024-04-23T10:31:32.373 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:31:32.373+0000 7f4d66ffd640 10 client.16176 handle_command_reply: tid=0 2024-04-23T10:31:32.373 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:31:32.373+0000 7f4d90617740 1 client.16176 shutdown 2024-04-23T10:31:32.373 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:31:32.373+0000 7f4d90617740 1 -- 172.21.15.114:0/2774024442 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x5606f46fcde0 msgr2=0x5606f4700550 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T10:31:32.373 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:31:32.373+0000 7f4d90617740 1 --2- 172.21.15.114:0/2774024442 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x5606f46fcde0 0x5606f4700550 secure :-1 s=READY pgs=667 cs=0 l=1 rev1=1 crypto rx=0x7f4d70013460 tx=0x7f4d70003b90 comp rx=0 tx=0).stop 2024-04-23T10:31:32.373 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:31:32.373+0000 7f4d90617740 1 -- 172.21.15.114:0/2774024442 shutdown_connections 2024-04-23T10:31:32.373 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:31:32.373+0000 7f4d90617740 1 --2- 172.21.15.114:0/2774024442 >> [v2:172.21.15.119:6838/937297340,v1:172.21.15.119:6839/937297340] conn(0x5606f4782850 0x5606f4784c70 crc :-1 s=READY pgs=101 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:31:32.373 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:31:32.373+0000 7f4d90617740 1 --2- 172.21.15.114:0/2774024442 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x5606f4709a10 0x5606f4700a90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:31:32.373 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:31:32.373+0000 7f4d90617740 1 --2- 172.21.15.114:0/2774024442 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x5606f46fcde0 0x5606f4700550 unknown :-1 s=CLOSED pgs=667 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:31:32.373 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:31:32.373+0000 7f4d90617740 1 --2- 172.21.15.114:0/2774024442 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x5606f46fc4c0 0x5606f4700010 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:31:32.373 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:31:32.373+0000 7f4d66ffd640 0 client.16176 ms_handle_reset on v2:172.21.15.119:6838/937297340 2024-04-23T10:31:32.373 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:31:32.373+0000 7f4d90617740 1 -- 172.21.15.114:0/2774024442 >> 172.21.15.114:0/2774024442 conn(0x5606f46f8b30 msgr2=0x5606f4714490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T10:31:32.373 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:31:32.373+0000 7f4d90617740 1 -- 172.21.15.114:0/2774024442 shutdown_connections 2024-04-23T10:31:32.373 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:31:32.373+0000 7f4d90617740 1 -- 172.21.15.114:0/2774024442 wait complete. 2024-04-23T10:31:32.373 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:31:32.373+0000 7f4d90617740 20 client.16176 trim_cache size 0 max 16384 2024-04-23T10:31:32.374 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stdout:{ 2024-04-23T10:31:32.374 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stdout: "message": "", 2024-04-23T10:31:32.374 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stdout: "return_code": 0 2024-04-23T10:31:32.374 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stdout:} 2024-04-23T10:31:32.375 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:31:32.374+0000 7f4d8d907640 1 -- 172.21.15.114:0/2157914180 >> [v2:172.21.15.114:6800/1395693466,v1:172.21.15.114:6801/1395693466] conn(0x7f4d740a3e40 msgr2=0x7f4d740a6300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T10:31:32.375 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:31:32.374+0000 7f4d8d907640 1 --2- 172.21.15.114:0/2157914180 >> [v2:172.21.15.114:6800/1395693466,v1:172.21.15.114:6801/1395693466] conn(0x7f4d740a3e40 0x7f4d740a6300 secure :-1 s=READY pgs=102 cs=0 l=1 rev1=1 crypto rx=0x7f4d8002f500 tx=0x7f4d8002a8c0 comp rx=0 tx=0).stop 2024-04-23T10:31:32.375 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:31:32.374+0000 7f4d8d907640 1 -- 172.21.15.114:0/2157914180 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f4d881010e0 msgr2=0x7f4d881975a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T10:31:32.375 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:31:32.374+0000 7f4d8d907640 1 --2- 172.21.15.114:0/2157914180 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f4d881010e0 0x7f4d881975a0 secure :-1 s=READY pgs=663 cs=0 l=1 rev1=1 crypto rx=0x7f4d7800c970 tx=0x7f4d7800ce30 comp rx=0 tx=0).stop 2024-04-23T10:31:32.375 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:31:32.374+0000 7f4d8d907640 1 -- 172.21.15.114:0/2157914180 shutdown_connections 2024-04-23T10:31:32.375 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:31:32.374+0000 7f4d8d907640 1 --2- 172.21.15.114:0/2157914180 >> [v2:172.21.15.114:6800/1395693466,v1:172.21.15.114:6801/1395693466] conn(0x7f4d740a3e40 0x7f4d740a6300 unknown :-1 s=CLOSED pgs=102 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:31:32.375 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:31:32.374+0000 7f4d8d907640 1 --2- 172.21.15.114:0/2157914180 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f4d8819be10 0x7f4d8819c2c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:31:32.375 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:31:32.374+0000 7f4d8d907640 1 --2- 172.21.15.114:0/2157914180 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f4d881061b0 0x7f4d88197ae0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:31:32.375 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:31:32.374+0000 7f4d8d907640 1 --2- 172.21.15.114:0/2157914180 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f4d881010e0 0x7f4d881975a0 unknown :-1 s=CLOSED pgs=663 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:31:32.375 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:31:32.374+0000 7f4d8d907640 1 -- 172.21.15.114:0/2157914180 >> 172.21.15.114:0/2157914180 conn(0x7f4d880771d0 msgr2=0x7f4d880febf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T10:31:32.375 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:31:32.374+0000 7f4d8d907640 1 -- 172.21.15.114:0/2157914180 shutdown_connections 2024-04-23T10:31:32.375 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:31:32.374+0000 7f4d8d907640 1 -- 172.21.15.114:0/2157914180 wait complete. 2024-04-23T10:31:32.491 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 23 10:31:32 smithi119 ceph-mon[32608]: pgmap v151: 97 pgs: 97 active+clean; 86 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 MiB/s rd, 1.6 MiB/s wr, 133 op/s 2024-04-23T10:31:32.504 INFO:journalctl@ceph.mon.a.smithi114.stdout:Apr 23 10:31:32 smithi114 ceph-mon[29255]: pgmap v151: 97 pgs: 97 active+clean; 86 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 MiB/s rd, 1.6 MiB/s wr, 133 op/s 2024-04-23T10:31:32.577 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 23 10:31:32 smithi204 ceph-mon[32649]: pgmap v151: 97 pgs: 97 active+clean; 86 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 MiB/s rd, 1.6 MiB/s wr, 133 op/s 2024-04-23T10:31:32.919 INFO:tasks.workunit.client.0.smithi114.stdout: 1 305661 6.81 MB/sec execute 101 sec latency 15143.619 ms 2024-04-23T10:31:33.078 INFO:journalctl@ceph.mds.i.smithi204.stdout:Apr 23 10:31:32 smithi204 ceph-aa240572-0159-11ef-bc93-c7b262605968-mds-i[63304]: ignoring --setuser ceph since I am not root 2024-04-23T10:31:33.078 INFO:journalctl@ceph.mds.i.smithi204.stdout:Apr 23 10:31:32 smithi204 ceph-aa240572-0159-11ef-bc93-c7b262605968-mds-i[63304]: ignoring --setgroup ceph since I am not root 2024-04-23T10:31:33.078 INFO:journalctl@ceph.mds.i.smithi204.stdout:Apr 23 10:31:32 smithi204 ceph-mds[63308]: -- 172.21.15.204:0/1664246162 <== mon.1 v2:172.21.15.204:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 194+0+0 (secure 0 0 0) 0x55e27cb45a40 con 0x55e27cb18c00 2024-04-23T10:31:33.078 INFO:journalctl@ceph.mds.i.smithi204.stdout:Apr 23 10:31:32 smithi204 ceph-aa240572-0159-11ef-bc93-c7b262605968-mds-i[63304]: starting mds.i at 2024-04-23T10:31:33.492 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 23 10:31:33 smithi119 ceph-mon[32608]: daemon mds.i restarted 2024-04-23T10:31:33.504 INFO:journalctl@ceph.mon.a.smithi114.stdout:Apr 23 10:31:33 smithi114 ceph-mon[29255]: daemon mds.i restarted 2024-04-23T10:31:33.577 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 23 10:31:33 smithi204 ceph-mon[32649]: daemon mds.i restarted 2024-04-23T10:31:33.919 INFO:tasks.workunit.client.0.smithi114.stdout: 1 305661 6.74 MB/sec execute 102 sec latency 16143.714 ms 2024-04-23T10:31:34.492 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 23 10:31:34 smithi119 ceph-mon[32608]: pgmap v152: 97 pgs: 97 active+clean; 85 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 MiB/s rd, 2.1 MiB/s wr, 156 op/s 2024-04-23T10:31:34.492 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 23 10:31:34 smithi119 ceph-mon[32608]: osdmap e94: 12 total, 12 up, 12 in 2024-04-23T10:31:34.492 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 23 10:31:34 smithi119 ceph-mon[32608]: mds.? [v2:172.21.15.204:6838/2396114218,v1:172.21.15.204:6839/2396114218] up:boot 2024-04-23T10:31:34.492 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 23 10:31:34 smithi119 ceph-mon[32608]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=e=up:active} 2 up:standby-replay 7 up:standby 2024-04-23T10:31:34.492 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 23 10:31:34 smithi119 ceph-mon[32608]: from='mgr.15291 172.21.15.114:0/1095789187' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "i"}]: dispatch 2024-04-23T10:31:34.492 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 23 10:31:34 smithi119 ceph-mon[32608]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=e=up:active} 3 up:standby-replay 6 up:standby 2024-04-23T10:31:34.504 INFO:journalctl@ceph.mon.a.smithi114.stdout:Apr 23 10:31:34 smithi114 ceph-mon[29255]: pgmap v152: 97 pgs: 97 active+clean; 85 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 MiB/s rd, 2.1 MiB/s wr, 156 op/s 2024-04-23T10:31:34.504 INFO:journalctl@ceph.mon.a.smithi114.stdout:Apr 23 10:31:34 smithi114 ceph-mon[29255]: osdmap e94: 12 total, 12 up, 12 in 2024-04-23T10:31:34.504 INFO:journalctl@ceph.mon.a.smithi114.stdout:Apr 23 10:31:34 smithi114 ceph-mon[29255]: mds.? [v2:172.21.15.204:6838/2396114218,v1:172.21.15.204:6839/2396114218] up:boot 2024-04-23T10:31:34.504 INFO:journalctl@ceph.mon.a.smithi114.stdout:Apr 23 10:31:34 smithi114 ceph-mon[29255]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=e=up:active} 2 up:standby-replay 7 up:standby 2024-04-23T10:31:34.504 INFO:journalctl@ceph.mon.a.smithi114.stdout:Apr 23 10:31:34 smithi114 ceph-mon[29255]: from='mgr.15291 172.21.15.114:0/1095789187' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "i"}]: dispatch 2024-04-23T10:31:34.504 INFO:journalctl@ceph.mon.a.smithi114.stdout:Apr 23 10:31:34 smithi114 ceph-mon[29255]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=e=up:active} 3 up:standby-replay 6 up:standby 2024-04-23T10:31:34.577 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 23 10:31:34 smithi204 ceph-mon[32649]: pgmap v152: 97 pgs: 97 active+clean; 85 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 MiB/s rd, 2.1 MiB/s wr, 156 op/s 2024-04-23T10:31:34.578 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 23 10:31:34 smithi204 ceph-mon[32649]: osdmap e94: 12 total, 12 up, 12 in 2024-04-23T10:31:34.578 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 23 10:31:34 smithi204 ceph-mon[32649]: mds.? [v2:172.21.15.204:6838/2396114218,v1:172.21.15.204:6839/2396114218] up:boot 2024-04-23T10:31:34.578 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 23 10:31:34 smithi204 ceph-mon[32649]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=e=up:active} 2 up:standby-replay 7 up:standby 2024-04-23T10:31:34.578 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 23 10:31:34 smithi204 ceph-mon[32649]: from='mgr.15291 172.21.15.114:0/1095789187' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "i"}]: dispatch 2024-04-23T10:31:34.578 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 23 10:31:34 smithi204 ceph-mon[32649]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=e=up:active} 3 up:standby-replay 6 up:standby 2024-04-23T10:31:34.919 INFO:tasks.workunit.client.0.smithi114.stdout: 1 305661 6.68 MB/sec execute 103 sec latency 17143.832 ms 2024-04-23T10:31:35.920 INFO:tasks.workunit.client.0.smithi114.stdout: 1 305661 6.61 MB/sec execute 104 sec latency 18143.943 ms 2024-04-23T10:31:36.492 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 23 10:31:36 smithi119 ceph-mon[32608]: pgmap v154: 97 pgs: 97 active+clean; 85 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 118 KiB/s rd, 1.5 MiB/s wr, 64 op/s 2024-04-23T10:31:36.504 INFO:journalctl@ceph.mon.a.smithi114.stdout:Apr 23 10:31:36 smithi114 ceph-mon[29255]: pgmap v154: 97 pgs: 97 active+clean; 85 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 118 KiB/s rd, 1.5 MiB/s wr, 64 op/s 2024-04-23T10:31:36.577 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 23 10:31:36 smithi204 ceph-mon[32649]: pgmap v154: 97 pgs: 97 active+clean; 85 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 118 KiB/s rd, 1.5 MiB/s wr, 64 op/s 2024-04-23T10:31:36.920 INFO:tasks.workunit.client.0.smithi114.stdout: 1 305661 6.55 MB/sec execute 105 sec latency 19144.075 ms 2024-04-23T10:31:37.920 INFO:tasks.workunit.client.0.smithi114.stdout: 1 305661 6.49 MB/sec execute 106 sec latency 20144.208 ms 2024-04-23T10:31:38.492 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 23 10:31:38 smithi119 ceph-mon[32608]: pgmap v155: 97 pgs: 97 active+clean; 81 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 102 KiB/s rd, 981 KiB/s wr, 56 op/s 2024-04-23T10:31:38.504 INFO:journalctl@ceph.mon.a.smithi114.stdout:Apr 23 10:31:38 smithi114 ceph-mon[29255]: pgmap v155: 97 pgs: 97 active+clean; 81 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 102 KiB/s rd, 981 KiB/s wr, 56 op/s 2024-04-23T10:31:38.577 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 23 10:31:38 smithi204 ceph-mon[32649]: pgmap v155: 97 pgs: 97 active+clean; 81 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 102 KiB/s rd, 981 KiB/s wr, 56 op/s 2024-04-23T10:31:38.920 INFO:tasks.workunit.client.0.smithi114.stdout: 1 305661 6.43 MB/sec execute 107 sec latency 21144.269 ms 2024-04-23T10:31:39.920 INFO:tasks.workunit.client.0.smithi114.stdout: 1 305661 6.37 MB/sec execute 108 sec latency 22144.313 ms 2024-04-23T10:31:40.492 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 23 10:31:40 smithi119 ceph-mon[32608]: pgmap v156: 97 pgs: 97 active+clean; 81 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 102 KiB/s rd, 981 KiB/s wr, 56 op/s 2024-04-23T10:31:40.504 INFO:journalctl@ceph.mon.a.smithi114.stdout:Apr 23 10:31:40 smithi114 ceph-mon[29255]: pgmap v156: 97 pgs: 97 active+clean; 81 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 102 KiB/s rd, 981 KiB/s wr, 56 op/s 2024-04-23T10:31:40.577 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 23 10:31:40 smithi204 ceph-mon[32649]: pgmap v156: 97 pgs: 97 active+clean; 81 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 102 KiB/s rd, 981 KiB/s wr, 56 op/s 2024-04-23T10:31:40.920 INFO:tasks.workunit.client.0.smithi114.stdout: 1 305661 6.31 MB/sec execute 109 sec latency 23144.406 ms 2024-04-23T10:31:41.920 INFO:tasks.workunit.client.0.smithi114.stdout: 1 305661 6.25 MB/sec execute 110 sec latency 24144.504 ms 2024-04-23T10:31:42.491 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 23 10:31:42 smithi119 ceph-mon[32608]: pgmap v157: 97 pgs: 97 active+clean; 81 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 102 KiB/s rd, 981 KiB/s wr, 56 op/s 2024-04-23T10:31:42.504 INFO:journalctl@ceph.mon.a.smithi114.stdout:Apr 23 10:31:42 smithi114 ceph-mon[29255]: pgmap v157: 97 pgs: 97 active+clean; 81 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 102 KiB/s rd, 981 KiB/s wr, 56 op/s 2024-04-23T10:31:42.577 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 23 10:31:42 smithi204 ceph-mon[32649]: pgmap v157: 97 pgs: 97 active+clean; 81 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 102 KiB/s rd, 981 KiB/s wr, 56 op/s 2024-04-23T10:31:42.920 INFO:tasks.workunit.client.0.smithi114.stdout: 1 305661 6.20 MB/sec execute 111 sec latency 25144.638 ms 2024-04-23T10:31:43.920 INFO:tasks.workunit.client.0.smithi114.stdout: 1 305661 6.14 MB/sec execute 112 sec latency 26144.732 ms 2024-04-23T10:31:44.492 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 23 10:31:44 smithi119 ceph-mon[32608]: pgmap v158: 97 pgs: 97 active+clean; 80 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 88 KiB/s rd, 1.2 MiB/s wr, 58 op/s 2024-04-23T10:31:44.504 INFO:journalctl@ceph.mon.a.smithi114.stdout:Apr 23 10:31:44 smithi114 ceph-mon[29255]: pgmap v158: 97 pgs: 97 active+clean; 80 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 88 KiB/s rd, 1.2 MiB/s wr, 58 op/s 2024-04-23T10:31:44.577 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 23 10:31:44 smithi204 ceph-mon[32649]: pgmap v158: 97 pgs: 97 active+clean; 80 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 88 KiB/s rd, 1.2 MiB/s wr, 58 op/s 2024-04-23T10:31:44.921 INFO:tasks.workunit.client.0.smithi114.stdout: 1 305661 6.09 MB/sec execute 113 sec latency 27144.864 ms 2024-04-23T10:31:45.470 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:31:45.470+0000 7f6df1343640 1 -- 172.21.15.114:0/4160302134 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f6dec06b120 msgr2=0x7f6dec10edf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T10:31:45.470 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:31:45.470+0000 7f6df1343640 1 --2- 172.21.15.114:0/4160302134 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f6dec06b120 0x7f6dec10edf0 secure :-1 s=READY pgs=668 cs=0 l=1 rev1=1 crypto rx=0x7f6de4009900 tx=0x7f6de402a480 comp rx=0 tx=0).stop 2024-04-23T10:31:45.470 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:31:45.470+0000 7f6df1343640 1 -- 172.21.15.114:0/4160302134 shutdown_connections 2024-04-23T10:31:45.470 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:31:45.470+0000 7f6df1343640 1 --2- 172.21.15.114:0/4160302134 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f6dec06b120 0x7f6dec10edf0 unknown :-1 s=CLOSED pgs=668 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:31:45.470 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:31:45.470+0000 7f6df1343640 1 --2- 172.21.15.114:0/4160302134 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f6dec06a780 0x7f6dec06abe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:31:45.470 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:31:45.470+0000 7f6df1343640 1 --2- 172.21.15.114:0/4160302134 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f6dec1080e0 0x7f6dec1084c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:31:45.470 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:31:45.470+0000 7f6df1343640 1 -- 172.21.15.114:0/4160302134 >> 172.21.15.114:0/4160302134 conn(0x7f6dec076c50 msgr2=0x7f6dec077060 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T10:31:45.470 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:31:45.470+0000 7f6df1343640 1 -- 172.21.15.114:0/4160302134 shutdown_connections 2024-04-23T10:31:45.470 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:31:45.471+0000 7f6df1343640 1 -- 172.21.15.114:0/4160302134 wait complete. 2024-04-23T10:31:45.470 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:31:45.471+0000 7f6df1343640 1 Processor -- start 2024-04-23T10:31:45.472 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:31:45.471+0000 7f6df1343640 1 -- start start 2024-04-23T10:31:45.472 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:31:45.472+0000 7f6df1343640 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f6dec06a780 0x7f6dec101270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T10:31:45.473 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:31:45.472+0000 7f6df1343640 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f6dec06b120 0x7f6dec1017b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T10:31:45.473 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:31:45.472+0000 7f6df1343640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f6dec1080e0 0x7f6dec101cf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T10:31:45.473 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:31:45.472+0000 7f6df1343640 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x7f6dec1121b0 con 0x7f6dec1080e0 2024-04-23T10:31:45.473 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:31:45.472+0000 7f6df1343640 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 -- 0x7f6dec112030 con 0x7f6dec06a780 2024-04-23T10:31:45.473 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:31:45.472+0000 7f6df1343640 1 -- --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_getmap magic: 0 -- 0x7f6dec112330 con 0x7f6dec06b120 2024-04-23T10:31:45.473 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:31:45.472+0000 7f6deb7fe640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f6dec1080e0 0x7f6dec101cf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T10:31:45.473 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:31:45.472+0000 7f6deb7fe640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f6dec1080e0 0x7f6dec101cf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.114:35982/0 (socket says 172.21.15.114:35982) 2024-04-23T10:31:45.473 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:31:45.472+0000 7f6deb7fe640 1 -- 172.21.15.114:0/2363342369 learned_addr learned my addr 172.21.15.114:0/2363342369 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-04-23T10:31:45.473 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:31:45.472+0000 7f6dea7fc640 1 --2- 172.21.15.114:0/2363342369 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f6dec06b120 0x7f6dec1017b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T10:31:45.473 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:31:45.472+0000 7f6deaffd640 1 --2- 172.21.15.114:0/2363342369 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f6dec06a780 0x7f6dec101270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T10:31:45.473 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:31:45.472+0000 7f6deb7fe640 1 -- 172.21.15.114:0/2363342369 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f6dec06b120 msgr2=0x7f6dec1017b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T10:31:45.473 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:31:45.472+0000 7f6deb7fe640 1 --2- 172.21.15.114:0/2363342369 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f6dec06b120 0x7f6dec1017b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:31:45.473 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:31:45.472+0000 7f6deb7fe640 1 -- 172.21.15.114:0/2363342369 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f6dec06a780 msgr2=0x7f6dec101270 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T10:31:45.473 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:31:45.472+0000 7f6deb7fe640 1 --2- 172.21.15.114:0/2363342369 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f6dec06a780 0x7f6dec101270 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:31:45.474 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:31:45.472+0000 7f6deb7fe640 1 -- 172.21.15.114:0/2363342369 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f6dec1a6ad0 con 0x7f6dec1080e0 2024-04-23T10:31:45.474 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:31:45.472+0000 7f6deb7fe640 1 --2- 172.21.15.114:0/2363342369 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f6dec1080e0 0x7f6dec101cf0 secure :-1 s=READY pgs=669 cs=0 l=1 rev1=1 crypto rx=0x7f6de4004770 tx=0x7f6de4004270 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T10:31:45.474 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:31:45.472+0000 7f6dd3fff640 1 -- 172.21.15.114:0/2363342369 <== mon.0 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6de403f030 con 0x7f6dec1080e0 2024-04-23T10:31:45.474 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:31:45.472+0000 7f6dd3fff640 1 -- 172.21.15.114:0/2363342369 <== mon.0 v2:172.21.15.114:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f6de4003aa0 con 0x7f6dec1080e0 2024-04-23T10:31:45.474 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:31:45.472+0000 7f6df1343640 1 -- 172.21.15.114:0/2363342369 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f6dec1a6d60 con 0x7f6dec1080e0 2024-04-23T10:31:45.474 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:31:45.472+0000 7f6dd3fff640 1 -- 172.21.15.114:0/2363342369 <== mon.0 v2:172.21.15.114:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6de4035070 con 0x7f6dec1080e0 2024-04-23T10:31:45.474 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:31:45.472+0000 7f6df1343640 1 -- 172.21.15.114:0/2363342369 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f6dec1a7250 con 0x7f6dec1080e0 2024-04-23T10:31:45.481 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:31:45.475+0000 7f6dd3fff640 1 -- 172.21.15.114:0/2363342369 <== mon.0 v2:172.21.15.114:3300/0 4 ==== mgrmap(e 26) ==== 138778+0+0 (secure 0 0 0) 0x7f6de4042030 con 0x7f6dec1080e0 2024-04-23T10:31:45.481 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:31:45.475+0000 7f6df4057740 1 Processor -- start 2024-04-23T10:31:45.481 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:31:45.475+0000 7f6df4057740 1 -- start start 2024-04-23T10:31:45.481 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:31:45.475+0000 7f6df4057740 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x55cf3a245620 0x55cf3a245a00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T10:31:45.481 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:31:45.475+0000 7f6df4057740 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x55cf3a245f40 0x55cf3a2502c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T10:31:45.481 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:31:45.475+0000 7f6df4057740 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55cf3a250800 0x55cf3a252bf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T10:31:45.482 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:31:45.475+0000 7f6df4057740 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x55cf3a1aa970 con 0x55cf3a250800 2024-04-23T10:31:45.482 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:31:45.475+0000 7f6df4057740 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 -- 0x55cf3a1474a0 con 0x55cf3a245f40 2024-04-23T10:31:45.482 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:31:45.475+0000 7f6df4057740 1 -- --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_getmap magic: 0 -- 0x55cf3a1a6930 con 0x55cf3a245620 2024-04-23T10:31:45.482 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:31:45.475+0000 7f6dea7fc640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55cf3a250800 0x55cf3a252bf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T10:31:45.482 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:31:45.475+0000 7f6dd3fff640 1 --2- 172.21.15.114:0/2363342369 >> [v2:172.21.15.114:6800/1395693466,v1:172.21.15.114:6801/1395693466] conn(0x7f6dd40a3dc0 0x7f6dd40a6280 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T10:31:45.482 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:31:45.475+0000 7f6dea7fc640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55cf3a250800 0x55cf3a252bf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.114:35990/0 (socket says 172.21.15.114:35990) 2024-04-23T10:31:45.482 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:31:45.475+0000 7f6dea7fc640 1 -- 172.21.15.114:0/3406953783 learned_addr learned my addr 172.21.15.114:0/3406953783 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-04-23T10:31:45.482 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:31:45.475+0000 7f6deaffd640 1 --2- 172.21.15.114:0/3406953783 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x55cf3a245f40 0x55cf3a2502c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T10:31:45.482 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:31:45.475+0000 7f6dea7fc640 1 --2- 172.21.15.114:0/3406953783 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x55cf3a245620 0x55cf3a245a00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T10:31:45.482 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:31:45.476+0000 7f6deb7fe640 1 --2- 172.21.15.114:0/2363342369 >> [v2:172.21.15.114:6800/1395693466,v1:172.21.15.114:6801/1395693466] conn(0x7f6dd40a3dc0 0x7f6dd40a6280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T10:31:45.482 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:31:45.476+0000 7f6dd3fff640 1 -- 172.21.15.114:0/2363342369 <== mon.0 v2:172.21.15.114:3300/0 5 ==== osd_map(94..94 src has 1..94) ==== 8585+0+0 (secure 0 0 0) 0x7f6de40f0eb0 con 0x7f6dec1080e0 2024-04-23T10:31:45.482 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:31:45.476+0000 7f6dea7fc640 1 -- 172.21.15.114:0/3406953783 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x55cf3a245620 msgr2=0x55cf3a245a00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T10:31:45.482 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:31:45.476+0000 7f6dea7fc640 1 --2- 172.21.15.114:0/3406953783 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x55cf3a245620 0x55cf3a245a00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:31:45.482 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:31:45.476+0000 7f6dea7fc640 1 -- 172.21.15.114:0/3406953783 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x55cf3a245f40 msgr2=0x55cf3a2502c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T10:31:45.483 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:31:45.476+0000 7f6dea7fc640 1 --2- 172.21.15.114:0/3406953783 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x55cf3a245f40 0x55cf3a2502c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:31:45.483 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:31:45.476+0000 7f6dea7fc640 1 -- 172.21.15.114:0/3406953783 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55cf39ebd210 con 0x55cf3a250800 2024-04-23T10:31:45.483 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:31:45.476+0000 7f6deaffd640 1 --2- 172.21.15.114:0/3406953783 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x55cf3a245f40 0x55cf3a2502c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-23T10:31:45.483 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:31:45.476+0000 7f6dea7fc640 1 --2- 172.21.15.114:0/3406953783 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55cf3a250800 0x55cf3a252bf0 secure :-1 s=READY pgs=670 cs=0 l=1 rev1=1 crypto rx=0x7f6de00127a0 tx=0x7f6de000b7c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5b93f54e2f4d78ba server_cookie=0 in_seq=0 out_seq=0 2024-04-23T10:31:45.483 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:31:45.476+0000 7f6deb7fe640 1 --2- 172.21.15.114:0/2363342369 >> [v2:172.21.15.114:6800/1395693466,v1:172.21.15.114:6801/1395693466] conn(0x7f6dd40a3dc0 0x7f6dd40a6280 secure :-1 s=READY pgs=104 cs=0 l=1 rev1=1 crypto rx=0x7f6de40dbfd0 tx=0x7f6de40b2350 comp rx=0 tx=0).ready entity=mgr.15291 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T10:31:45.483 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:31:45.476+0000 7f6dd17fa640 1 -- 172.21.15.114:0/3406953783 <== mon.0 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6de0012d20 con 0x55cf3a250800 2024-04-23T10:31:45.483 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:31:45.476+0000 7f6dd17fa640 1 -- 172.21.15.114:0/3406953783 <== mon.0 v2:172.21.15.114:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f6de0005020 con 0x55cf3a250800 2024-04-23T10:31:45.483 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:31:45.476+0000 7f6dd17fa640 1 -- 172.21.15.114:0/3406953783 <== mon.0 v2:172.21.15.114:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6de00049c0 con 0x55cf3a250800 2024-04-23T10:31:45.483 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:31:45.477+0000 7f6df4057740 1 -- 172.21.15.114:0/3406953783 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55cf3a250800 msgr2=0x55cf3a252bf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T10:31:45.483 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:31:45.477+0000 7f6df4057740 1 --2- 172.21.15.114:0/3406953783 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55cf3a250800 0x55cf3a252bf0 secure :-1 s=READY pgs=670 cs=0 l=1 rev1=1 crypto rx=0x7f6de00127a0 tx=0x7f6de000b7c0 comp rx=0 tx=0).stop 2024-04-23T10:31:45.483 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:31:45.477+0000 7f6df4057740 1 -- 172.21.15.114:0/3406953783 shutdown_connections 2024-04-23T10:31:45.483 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:31:45.477+0000 7f6df4057740 1 --2- 172.21.15.114:0/3406953783 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55cf3a250800 0x55cf3a252bf0 unknown :-1 s=CLOSED pgs=670 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:31:45.483 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:31:45.477+0000 7f6df4057740 1 --2- 172.21.15.114:0/3406953783 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x55cf3a245f40 0x55cf3a2502c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:31:45.483 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:31:45.477+0000 7f6df4057740 1 --2- 172.21.15.114:0/3406953783 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x55cf3a245620 0x55cf3a245a00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:31:45.483 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:31:45.477+0000 7f6df4057740 1 -- 172.21.15.114:0/3406953783 >> 172.21.15.114:0/3406953783 conn(0x55cf3a241b30 msgr2=0x55cf3a241f10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T10:31:45.484 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:31:45.477+0000 7f6df4057740 1 -- 172.21.15.114:0/3406953783 shutdown_connections 2024-04-23T10:31:45.484 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:31:45.477+0000 7f6df4057740 1 -- 172.21.15.114:0/3406953783 wait complete. 2024-04-23T10:31:45.484 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:31:45.477+0000 7f6df4057740 1 -- start start 2024-04-23T10:31:45.484 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:31:45.477+0000 7f6df4057740 1 Processor -- start 2024-04-23T10:31:45.484 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:31:45.477+0000 7f6df4057740 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55cf3a245620 0x55cf3a250150 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T10:31:45.484 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:31:45.478+0000 7f6df4057740 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x55cf3a245f40 0x55cf3a24d440 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T10:31:45.484 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:31:45.478+0000 7f6df4057740 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x55cf3a250800 0x55cf3a24d980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T10:31:45.484 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:31:45.478+0000 7f6df4057740 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x55cf3a253430 con 0x55cf3a245620 2024-04-23T10:31:45.484 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:31:45.478+0000 7f6df4057740 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 -- 0x55cf3a1450b0 con 0x55cf3a250800 2024-04-23T10:31:45.484 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:31:45.478+0000 7f6df4057740 1 -- --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_getmap magic: 0 -- 0x55cf3a2535b0 con 0x55cf3a245f40 2024-04-23T10:31:45.484 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:31:45.478+0000 7f6df4057740 10 client.0 _ll_register_callbacks cb 0x55cf3a132d30 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-23T10:31:45.484 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:31:45.478+0000 7f6dea7fc640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55cf3a245620 0x55cf3a250150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T10:31:45.484 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:31:45.478+0000 7f6dea7fc640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55cf3a245620 0x55cf3a250150 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.114:36004/0 (socket says 172.21.15.114:36004) 2024-04-23T10:31:45.484 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:31:45.478+0000 7f6dea7fc640 1 -- 172.21.15.114:0/3426618912 learned_addr learned my addr 172.21.15.114:0/3426618912 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-04-23T10:31:45.484 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:31:45.478+0000 7f6deaffd640 1 --2- 172.21.15.114:0/3426618912 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x55cf3a245f40 0x55cf3a24d440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T10:31:45.485 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:31:45.478+0000 7f6dea7fc640 1 --2- 172.21.15.114:0/3426618912 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x55cf3a250800 0x55cf3a24d980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T10:31:45.485 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:31:45.478+0000 7f6dea7fc640 1 -- 172.21.15.114:0/3426618912 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x55cf3a245f40 msgr2=0x55cf3a24d440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T10:31:45.485 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:31:45.478+0000 7f6dea7fc640 1 --2- 172.21.15.114:0/3426618912 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x55cf3a245f40 0x55cf3a24d440 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:31:45.485 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:31:45.478+0000 7f6dea7fc640 1 -- 172.21.15.114:0/3426618912 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x55cf3a250800 msgr2=0x55cf3a24d980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T10:31:45.485 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:31:45.478+0000 7f6dea7fc640 1 --2- 172.21.15.114:0/3426618912 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x55cf3a250800 0x55cf3a24d980 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:31:45.485 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:31:45.478+0000 7f6dea7fc640 1 -- 172.21.15.114:0/3426618912 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55cf39f89620 con 0x55cf3a245620 2024-04-23T10:31:45.485 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:31:45.478+0000 7f6dea7fc640 1 --2- 172.21.15.114:0/3426618912 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55cf3a245620 0x55cf3a250150 secure :-1 s=READY pgs=671 cs=0 l=1 rev1=1 crypto rx=0x7f6de0015040 tx=0x7f6de0012f60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2c6e7902dfdc34be server_cookie=0 in_seq=0 out_seq=0 2024-04-23T10:31:45.485 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:31:45.478+0000 7f6dceff5640 10 client.0 ms_handle_connect on v2:172.21.15.114:3300/0 2024-04-23T10:31:45.485 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:31:45.478+0000 7f6dceff5640 1 -- 172.21.15.114:0/3426618912 <== mon.0 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6de002c060 con 0x55cf3a245620 2024-04-23T10:31:45.485 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:31:45.478+0000 7f6dceff5640 1 -- 172.21.15.114:0/3426618912 <== mon.0 v2:172.21.15.114:3300/0 2 ==== osd_map(94..94 src has 1..94) ==== 8585+0+0 (secure 0 0 0) 0x7f6de0010970 con 0x55cf3a245620 2024-04-23T10:31:45.485 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:31:45.478+0000 7f6df4057740 1 -- 172.21.15.114:0/3426618912 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f6de0005020 con 0x55cf3a245620 2024-04-23T10:31:45.485 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:31:45.479+0000 7f6dceff5640 1 -- 172.21.15.114:0/3426618912 <== mon.0 v2:172.21.15.114:3300/0 3 ==== mon_get_version_reply(handle=1 version=30) ==== 24+0+0 (secure 0 0 0) 0x7f6de0010b70 con 0x55cf3a245620 2024-04-23T10:31:45.485 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:31:45.479+0000 7f6df4057740 10 client.16194 fetch_fsmap learned FSMap version 30 2024-04-23T10:31:45.485 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:31:45.479+0000 7f6df4057740 1 -- 172.21.15.114:0/3426618912 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({config=0+,fsmap=30,monmap=4+}) -- 0x55cf3a25f3b0 con 0x55cf3a245620 2024-04-23T10:31:45.485 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:31:45.479+0000 7f6dceff5640 1 -- 172.21.15.114:0/3426618912 <== mon.0 v2:172.21.15.114:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f6de0005020 con 0x55cf3a245620 2024-04-23T10:31:45.486 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:31:45.479+0000 7f6dceff5640 1 -- 172.21.15.114:0/3426618912 <== mon.0 v2:172.21.15.114:3300/0 5 ==== fsmap(e 30) ==== 8338+0+0 (secure 0 0 0) 0x7f6de001f420 con 0x55cf3a245620 2024-04-23T10:31:45.486 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:31:45.479+0000 7f6df4057740 10 client.16194 fetch_fsmap finished waiting for FSMap version 30 2024-04-23T10:31:45.486 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:31:45.479+0000 7f6df4057740 10 client.16194 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-04-23T10:31:45.486 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:31:45.479+0000 7f6df4057740 20 client.16194 populate_metadata read hostname 'smithi114' 2024-04-23T10:31:45.486 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:31:45.479+0000 7f6df4057740 1 --2- 172.21.15.114:0/3426618912 >> [v2:172.21.15.119:6838/937297340,v1:172.21.15.119:6839/937297340] conn(0x55cf3a2607c0 0x55cf3a262bb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T10:31:45.486 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:31:45.479+0000 7f6df4057740 4 client.16194 mds_command: new command op to 24545 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-23T10:31:45.486 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:31:45.479+0000 7f6df4057740 1 -- 172.21.15.114:0/3426618912 --> [v2:172.21.15.119:6838/937297340,v1:172.21.15.119:6839/937297340] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55cf3a1b8390 con 0x55cf3a2607c0 2024-04-23T10:31:45.486 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:31:45.479+0000 7f6deaffd640 1 --2- 172.21.15.114:0/3426618912 >> [v2:172.21.15.119:6838/937297340,v1:172.21.15.119:6839/937297340] conn(0x55cf3a2607c0 0x55cf3a262bb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T10:31:45.486 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:31:45.480+0000 7f6deaffd640 1 --2- 172.21.15.114:0/3426618912 >> [v2:172.21.15.119:6838/937297340,v1:172.21.15.119:6839/937297340] conn(0x55cf3a2607c0 0x55cf3a262bb0 crc :-1 s=READY pgs=102 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=700bd659721d7d8c server_cookie=e581c2e7be428af8 in_seq=0 out_seq=0 2024-04-23T10:31:45.486 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:31:45.480+0000 7f6dceff5640 10 client.16194 ms_handle_connect on v2:172.21.15.119:6838/937297340 2024-04-23T10:31:45.486 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:31:45.480+0000 7f6dceff5640 1 -- 172.21.15.114:0/3426618912 <== mds.0 v2:172.21.15.119:6838/937297340 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x55cf3a1b8390 con 0x55cf3a2607c0 2024-04-23T10:31:45.486 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:31:45.480+0000 7f6dceff5640 10 client.16194 handle_command_reply: tid=0 2024-04-23T10:31:45.486 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:31:45.480+0000 7f6df4057740 1 client.16194 shutdown 2024-04-23T10:31:45.486 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:31:45.480+0000 7f6df4057740 1 -- 172.21.15.114:0/3426618912 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55cf3a245620 msgr2=0x55cf3a250150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T10:31:45.486 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:31:45.480+0000 7f6df4057740 1 --2- 172.21.15.114:0/3426618912 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55cf3a245620 0x55cf3a250150 secure :-1 s=READY pgs=671 cs=0 l=1 rev1=1 crypto rx=0x7f6de0015040 tx=0x7f6de0012f60 comp rx=0 tx=0).stop 2024-04-23T10:31:45.487 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:31:45.480+0000 7f6df4057740 1 -- 172.21.15.114:0/3426618912 shutdown_connections 2024-04-23T10:31:45.487 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:31:45.480+0000 7f6df4057740 1 --2- 172.21.15.114:0/3426618912 >> [v2:172.21.15.119:6838/937297340,v1:172.21.15.119:6839/937297340] conn(0x55cf3a2607c0 0x55cf3a262bb0 crc :-1 s=READY pgs=102 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:31:45.487 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:31:45.480+0000 7f6df4057740 1 --2- 172.21.15.114:0/3426618912 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x55cf3a250800 0x55cf3a24d980 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:31:45.487 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:31:45.480+0000 7f6df4057740 1 --2- 172.21.15.114:0/3426618912 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x55cf3a245f40 0x55cf3a24d440 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:31:45.487 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:31:45.480+0000 7f6df4057740 1 --2- 172.21.15.114:0/3426618912 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55cf3a245620 0x55cf3a250150 unknown :-1 s=CLOSED pgs=671 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:31:45.487 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:31:45.480+0000 7f6dceff5640 0 client.16194 ms_handle_reset on v2:172.21.15.119:6838/937297340 2024-04-23T10:31:45.487 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:31:45.481+0000 7f6df4057740 1 -- 172.21.15.114:0/3426618912 >> 172.21.15.114:0/3426618912 conn(0x55cf3a241b30 msgr2=0x55cf3a247860 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T10:31:45.487 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:31:45.481+0000 7f6df4057740 1 -- 172.21.15.114:0/3426618912 shutdown_connections 2024-04-23T10:31:45.487 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:31:45.481+0000 7f6df4057740 1 -- 172.21.15.114:0/3426618912 wait complete. 2024-04-23T10:31:45.487 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:31:45.481+0000 7f6df4057740 20 client.16194 trim_cache size 0 max 16384 2024-04-23T10:31:45.499 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:31:45.495+0000 7f6df4057740 1 Processor -- start 2024-04-23T10:31:45.499 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:31:45.495+0000 7f6df4057740 1 -- start start 2024-04-23T10:31:45.499 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:31:45.496+0000 7f6df4057740 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55cf3a245620 0x55cf3a25c320 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T10:31:45.500 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:31:45.496+0000 7f6df4057740 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x55cf3a245f40 0x55cf3a2682d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T10:31:45.500 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:31:45.496+0000 7f6df4057740 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x55cf3a250800 0x55cf3a268810 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T10:31:45.500 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:31:45.496+0000 7f6df4057740 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x7f6dec1b2710 con 0x55cf3a245620 2024-04-23T10:31:45.500 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:31:45.496+0000 7f6df4057740 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 -- 0x7f6dec1affe0 con 0x55cf3a245f40 2024-04-23T10:31:45.500 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:31:45.496+0000 7f6df4057740 1 -- --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_getmap magic: 0 -- 0x7f6dec1ade30 con 0x55cf3a250800 2024-04-23T10:31:45.500 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:31:45.496+0000 7f6dea7fc640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55cf3a245620 0x55cf3a25c320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T10:31:45.500 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:31:45.496+0000 7f6dea7fc640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55cf3a245620 0x55cf3a25c320 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.114:36012/0 (socket says 172.21.15.114:36012) 2024-04-23T10:31:45.500 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:31:45.496+0000 7f6dea7fc640 1 -- 172.21.15.114:0/1663308219 learned_addr learned my addr 172.21.15.114:0/1663308219 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-04-23T10:31:45.500 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:31:45.496+0000 7f6dea7fc640 1 --2- 172.21.15.114:0/1663308219 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x55cf3a250800 0x55cf3a268810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T10:31:45.500 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:31:45.496+0000 7f6deaffd640 1 --2- 172.21.15.114:0/1663308219 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x55cf3a245f40 0x55cf3a2682d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T10:31:45.500 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:31:45.496+0000 7f6dea7fc640 1 -- 172.21.15.114:0/1663308219 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x55cf3a250800 msgr2=0x55cf3a268810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T10:31:45.500 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:31:45.496+0000 7f6dea7fc640 1 --2- 172.21.15.114:0/1663308219 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x55cf3a250800 0x55cf3a268810 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:31:45.500 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:31:45.496+0000 7f6dea7fc640 1 -- 172.21.15.114:0/1663308219 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x55cf3a245f40 msgr2=0x55cf3a2682d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T10:31:45.500 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:31:45.496+0000 7f6dea7fc640 1 --2- 172.21.15.114:0/1663308219 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x55cf3a245f40 0x55cf3a2682d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:31:45.501 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:31:45.496+0000 7f6dea7fc640 1 -- 172.21.15.114:0/1663308219 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55cf3a253310 con 0x55cf3a245620 2024-04-23T10:31:45.501 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:31:45.496+0000 7f6deaffd640 1 --2- 172.21.15.114:0/1663308219 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x55cf3a245f40 0x55cf3a2682d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-23T10:31:45.501 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:31:45.496+0000 7f6dea7fc640 1 --2- 172.21.15.114:0/1663308219 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55cf3a245620 0x55cf3a25c320 secure :-1 s=READY pgs=672 cs=0 l=1 rev1=1 crypto rx=0x55cf3a25c8a0 tx=0x7f6de0012720 comp rx=0 tx=0).ready entity=mon.0 client_cookie=992e7eb39248e30e server_cookie=0 in_seq=0 out_seq=0 2024-04-23T10:31:45.501 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:31:45.496+0000 7f6dd0ff9640 1 -- 172.21.15.114:0/1663308219 <== mon.0 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6de002c070 con 0x55cf3a245620 2024-04-23T10:31:45.501 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:31:45.496+0000 7f6dd0ff9640 1 -- 172.21.15.114:0/1663308219 <== mon.0 v2:172.21.15.114:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f6de0016040 con 0x55cf3a245620 2024-04-23T10:31:45.501 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:31:45.496+0000 7f6dd0ff9640 1 -- 172.21.15.114:0/1663308219 <== mon.0 v2:172.21.15.114:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6de00122f0 con 0x55cf3a245620 2024-04-23T10:31:45.501 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:31:45.497+0000 7f6df4057740 1 -- 172.21.15.114:0/1663308219 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55cf3a245620 msgr2=0x55cf3a25c320 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T10:31:45.501 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:31:45.497+0000 7f6df4057740 1 --2- 172.21.15.114:0/1663308219 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55cf3a245620 0x55cf3a25c320 secure :-1 s=READY pgs=672 cs=0 l=1 rev1=1 crypto rx=0x55cf3a25c8a0 tx=0x7f6de0012720 comp rx=0 tx=0).stop 2024-04-23T10:31:45.501 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:31:45.497+0000 7f6df4057740 1 -- 172.21.15.114:0/1663308219 shutdown_connections 2024-04-23T10:31:45.501 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:31:45.497+0000 7f6df4057740 1 --2- 172.21.15.114:0/1663308219 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x55cf3a250800 0x55cf3a268810 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:31:45.501 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:31:45.497+0000 7f6df4057740 1 --2- 172.21.15.114:0/1663308219 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x55cf3a245f40 0x55cf3a2682d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:31:45.501 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:31:45.497+0000 7f6df4057740 1 --2- 172.21.15.114:0/1663308219 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55cf3a245620 0x55cf3a25c320 unknown :-1 s=CLOSED pgs=672 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:31:45.501 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:31:45.497+0000 7f6df4057740 1 -- 172.21.15.114:0/1663308219 >> 172.21.15.114:0/1663308219 conn(0x55cf3a241b30 msgr2=0x55cf3a247860 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T10:31:45.501 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:31:45.497+0000 7f6df4057740 1 -- 172.21.15.114:0/1663308219 shutdown_connections 2024-04-23T10:31:45.502 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:31:45.497+0000 7f6df4057740 1 -- 172.21.15.114:0/1663308219 wait complete. 2024-04-23T10:31:45.502 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:31:45.497+0000 7f6df4057740 1 -- start start 2024-04-23T10:31:45.502 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:31:45.497+0000 7f6df4057740 1 Processor -- start 2024-04-23T10:31:45.502 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:31:45.497+0000 7f6df4057740 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x55cf3a245620 0x55cf3a258910 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T10:31:45.502 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:31:45.497+0000 7f6df4057740 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x55cf3a245f40 0x55cf3a253db0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T10:31:45.502 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:31:45.497+0000 7f6df4057740 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55cf3a250800 0x55cf3a2542f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T10:31:45.502 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:31:45.497+0000 7f6df4057740 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x7f6dec1ae4a0 con 0x55cf3a250800 2024-04-23T10:31:45.502 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:31:45.497+0000 7f6df4057740 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 -- 0x7f6dec1af910 con 0x55cf3a245f40 2024-04-23T10:31:45.502 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:31:45.497+0000 7f6df4057740 1 -- --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_getmap magic: 0 -- 0x7f6dec1aeb70 con 0x55cf3a245620 2024-04-23T10:31:45.502 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:31:45.498+0000 7f6df4057740 10 client.0 _ll_register_callbacks cb 0x7f6de0012670 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-23T10:31:45.502 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:31:45.498+0000 7f6dea7fc640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55cf3a250800 0x55cf3a2542f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T10:31:45.502 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:31:45.498+0000 7f6dea7fc640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55cf3a250800 0x55cf3a2542f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.114:36026/0 (socket says 172.21.15.114:36026) 2024-04-23T10:31:45.502 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:31:45.498+0000 7f6dea7fc640 1 -- 172.21.15.114:0/1017720387 learned_addr learned my addr 172.21.15.114:0/1017720387 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-04-23T10:31:45.502 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:31:45.498+0000 7f6dea7fc640 1 --2- 172.21.15.114:0/1017720387 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x55cf3a245620 0x55cf3a258910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T10:31:45.502 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:31:45.498+0000 7f6deaffd640 1 --2- 172.21.15.114:0/1017720387 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x55cf3a245f40 0x55cf3a253db0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T10:31:45.503 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:31:45.498+0000 7f6dea7fc640 1 -- 172.21.15.114:0/1017720387 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x55cf3a245620 msgr2=0x55cf3a258910 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T10:31:45.503 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:31:45.498+0000 7f6dea7fc640 1 --2- 172.21.15.114:0/1017720387 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x55cf3a245620 0x55cf3a258910 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:31:45.503 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:31:45.498+0000 7f6dea7fc640 1 -- 172.21.15.114:0/1017720387 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x55cf3a245f40 msgr2=0x55cf3a253db0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T10:31:45.503 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:31:45.498+0000 7f6dea7fc640 1 --2- 172.21.15.114:0/1017720387 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x55cf3a245f40 0x55cf3a253db0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:31:45.503 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:31:45.498+0000 7f6dea7fc640 1 -- 172.21.15.114:0/1017720387 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55cf3a254830 con 0x55cf3a250800 2024-04-23T10:31:45.503 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:31:45.498+0000 7f6dea7fc640 1 --2- 172.21.15.114:0/1017720387 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55cf3a250800 0x55cf3a2542f0 secure :-1 s=READY pgs=673 cs=0 l=1 rev1=1 crypto rx=0x7f6de000bef0 tx=0x7f6de0005fb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b450bbbfef6c4ba6 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T10:31:45.503 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:31:45.498+0000 7f6dceff5640 10 client.0 ms_handle_connect on v2:172.21.15.114:3300/0 2024-04-23T10:31:45.503 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:31:45.498+0000 7f6dceff5640 1 -- 172.21.15.114:0/1017720387 <== mon.0 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6de002c070 con 0x55cf3a250800 2024-04-23T10:31:45.503 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:31:45.498+0000 7f6dceff5640 1 -- 172.21.15.114:0/1017720387 <== mon.0 v2:172.21.15.114:3300/0 2 ==== osd_map(94..94 src has 1..94) ==== 8585+0+0 (secure 0 0 0) 0x7f6de001f420 con 0x55cf3a250800 2024-04-23T10:31:45.503 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:31:45.498+0000 7f6df4057740 1 -- 172.21.15.114:0/1017720387 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f6de0016040 con 0x55cf3a250800 2024-04-23T10:31:45.503 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:31:45.498+0000 7f6dceff5640 1 -- 172.21.15.114:0/1017720387 <== mon.0 v2:172.21.15.114:3300/0 3 ==== mon_get_version_reply(handle=1 version=30) ==== 24+0+0 (secure 0 0 0) 0x7f6de001f620 con 0x55cf3a250800 2024-04-23T10:31:45.503 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:31:45.498+0000 7f6df4057740 10 client.16200 fetch_fsmap learned FSMap version 30 2024-04-23T10:31:45.503 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:31:45.498+0000 7f6df4057740 1 -- 172.21.15.114:0/1017720387 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({config=0+,fsmap=30,monmap=4+}) -- 0x55cf3a2c6770 con 0x55cf3a250800 2024-04-23T10:31:45.503 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:31:45.499+0000 7f6dceff5640 1 -- 172.21.15.114:0/1017720387 <== mon.0 v2:172.21.15.114:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f6de0016040 con 0x55cf3a250800 2024-04-23T10:31:45.504 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:31:45.499+0000 7f6dceff5640 1 -- 172.21.15.114:0/1017720387 <== mon.0 v2:172.21.15.114:3300/0 5 ==== fsmap(e 30) ==== 8338+0+0 (secure 0 0 0) 0x7f6de0036020 con 0x55cf3a250800 2024-04-23T10:31:45.504 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:31:45.499+0000 7f6df4057740 10 client.16200 fetch_fsmap finished waiting for FSMap version 30 2024-04-23T10:31:45.504 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:31:45.499+0000 7f6df4057740 10 client.16200 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-04-23T10:31:45.504 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:31:45.499+0000 7f6df4057740 20 client.16200 populate_metadata read hostname 'smithi114' 2024-04-23T10:31:45.504 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:31:45.499+0000 7f6df4057740 1 --2- 172.21.15.114:0/1017720387 >> [v2:172.21.15.119:6838/937297340,v1:172.21.15.119:6839/937297340] conn(0x55cf3a2c6d80 0x55cf3a2c9170 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T10:31:45.504 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:31:45.499+0000 7f6df4057740 4 client.16200 mds_command: new command op to 24545 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-04-23T10:31:45.504 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:31:45.499+0000 7f6df4057740 1 -- 172.21.15.114:0/1017720387 --> [v2:172.21.15.119:6838/937297340,v1:172.21.15.119:6839/937297340] -- command(tid 0: {"prefix": "flush journal"}) -- 0x55cf3a1b2200 con 0x55cf3a2c6d80 2024-04-23T10:31:45.504 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:31:45.499+0000 7f6deaffd640 1 --2- 172.21.15.114:0/1017720387 >> [v2:172.21.15.119:6838/937297340,v1:172.21.15.119:6839/937297340] conn(0x55cf3a2c6d80 0x55cf3a2c9170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T10:31:45.504 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:31:45.499+0000 7f6deaffd640 1 --2- 172.21.15.114:0/1017720387 >> [v2:172.21.15.119:6838/937297340,v1:172.21.15.119:6839/937297340] conn(0x55cf3a2c6d80 0x55cf3a2c9170 crc :-1 s=READY pgs=103 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=4c7a66d33e010709 server_cookie=5ecb1e8557568aec in_seq=0 out_seq=0 2024-04-23T10:31:45.504 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:31:45.499+0000 7f6dceff5640 10 client.16200 ms_handle_connect on v2:172.21.15.119:6838/937297340 2024-04-23T10:31:45.524 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:31:45.524+0000 7f6dceff5640 1 -- 172.21.15.114:0/1017720387 <== mds.0 v2:172.21.15.119:6838/937297340 1 ==== command_reply(tid 0: 0 ) ==== 8+0+44 (crc 0 0 0) 0x55cf3a1b2200 con 0x55cf3a2c6d80 2024-04-23T10:31:45.524 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:31:45.524+0000 7f6dceff5640 10 client.16200 handle_command_reply: tid=0 2024-04-23T10:31:45.524 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:31:45.524+0000 7f6df4057740 1 client.16200 shutdown 2024-04-23T10:31:45.524 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:31:45.524+0000 7f6df4057740 1 -- 172.21.15.114:0/1017720387 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55cf3a250800 msgr2=0x55cf3a2542f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T10:31:45.524 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:31:45.524+0000 7f6df4057740 1 --2- 172.21.15.114:0/1017720387 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55cf3a250800 0x55cf3a2542f0 secure :-1 s=READY pgs=673 cs=0 l=1 rev1=1 crypto rx=0x7f6de000bef0 tx=0x7f6de0005fb0 comp rx=0 tx=0).stop 2024-04-23T10:31:45.524 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:31:45.524+0000 7f6df4057740 1 -- 172.21.15.114:0/1017720387 shutdown_connections 2024-04-23T10:31:45.524 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:31:45.524+0000 7f6df4057740 1 --2- 172.21.15.114:0/1017720387 >> [v2:172.21.15.119:6838/937297340,v1:172.21.15.119:6839/937297340] conn(0x55cf3a2c6d80 0x55cf3a2c9170 crc :-1 s=READY pgs=103 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:31:45.524 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:31:45.524+0000 7f6df4057740 1 --2- 172.21.15.114:0/1017720387 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55cf3a250800 0x55cf3a2542f0 unknown :-1 s=CLOSED pgs=673 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:31:45.525 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:31:45.524+0000 7f6df4057740 1 --2- 172.21.15.114:0/1017720387 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x55cf3a245f40 0x55cf3a253db0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:31:45.525 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:31:45.524+0000 7f6df4057740 1 --2- 172.21.15.114:0/1017720387 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x55cf3a245620 0x55cf3a258910 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:31:45.525 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:31:45.524+0000 7f6df4057740 1 -- 172.21.15.114:0/1017720387 >> 172.21.15.114:0/1017720387 conn(0x55cf3a241b30 msgr2=0x55cf3a2481b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T10:31:45.525 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:31:45.524+0000 7f6dceff5640 0 client.16200 ms_handle_reset on v2:172.21.15.119:6838/937297340 2024-04-23T10:31:45.525 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:31:45.524+0000 7f6df4057740 1 -- 172.21.15.114:0/1017720387 shutdown_connections 2024-04-23T10:31:45.525 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:31:45.524+0000 7f6df4057740 1 -- 172.21.15.114:0/1017720387 wait complete. 2024-04-23T10:31:45.525 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:31:45.524+0000 7f6df4057740 20 client.16200 trim_cache size 0 max 16384 2024-04-23T10:31:45.525 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stdout:{ 2024-04-23T10:31:45.525 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stdout: "message": "", 2024-04-23T10:31:45.525 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stdout: "return_code": 0 2024-04-23T10:31:45.525 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stdout:} 2024-04-23T10:31:45.526 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:31:45.525+0000 7f6df1343640 1 -- 172.21.15.114:0/2363342369 >> [v2:172.21.15.114:6800/1395693466,v1:172.21.15.114:6801/1395693466] conn(0x7f6dd40a3dc0 msgr2=0x7f6dd40a6280 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T10:31:45.526 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:31:45.525+0000 7f6df1343640 1 --2- 172.21.15.114:0/2363342369 >> [v2:172.21.15.114:6800/1395693466,v1:172.21.15.114:6801/1395693466] conn(0x7f6dd40a3dc0 0x7f6dd40a6280 secure :-1 s=READY pgs=104 cs=0 l=1 rev1=1 crypto rx=0x7f6de40dbfd0 tx=0x7f6de40b2350 comp rx=0 tx=0).stop 2024-04-23T10:31:45.526 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:31:45.525+0000 7f6df1343640 1 -- 172.21.15.114:0/2363342369 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f6dec1080e0 msgr2=0x7f6dec101cf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T10:31:45.526 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:31:45.525+0000 7f6df1343640 1 --2- 172.21.15.114:0/2363342369 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f6dec1080e0 0x7f6dec101cf0 secure :-1 s=READY pgs=669 cs=0 l=1 rev1=1 crypto rx=0x7f6de4004770 tx=0x7f6de4004270 comp rx=0 tx=0).stop 2024-04-23T10:31:45.526 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:31:45.525+0000 7f6df1343640 1 -- 172.21.15.114:0/2363342369 shutdown_connections 2024-04-23T10:31:45.527 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:31:45.525+0000 7f6df1343640 1 --2- 172.21.15.114:0/2363342369 >> [v2:172.21.15.114:6800/1395693466,v1:172.21.15.114:6801/1395693466] conn(0x7f6dd40a3dc0 0x7f6dd40a6280 unknown :-1 s=CLOSED pgs=104 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:31:45.527 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:31:45.525+0000 7f6df1343640 1 --2- 172.21.15.114:0/2363342369 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f6dec1080e0 0x7f6dec101cf0 unknown :-1 s=CLOSED pgs=669 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:31:45.527 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:31:45.525+0000 7f6df1343640 1 --2- 172.21.15.114:0/2363342369 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f6dec06b120 0x7f6dec1017b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:31:45.527 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:31:45.525+0000 7f6df1343640 1 --2- 172.21.15.114:0/2363342369 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f6dec06a780 0x7f6dec101270 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:31:45.527 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:31:45.525+0000 7f6df1343640 1 -- 172.21.15.114:0/2363342369 >> 172.21.15.114:0/2363342369 conn(0x7f6dec076c50 msgr2=0x7f6dec0ffe60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T10:31:45.527 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:31:45.525+0000 7f6df1343640 1 -- 172.21.15.114:0/2363342369 shutdown_connections 2024-04-23T10:31:45.527 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:31:45.525+0000 7f6df1343640 1 -- 172.21.15.114:0/2363342369 wait complete. 2024-04-23T10:31:45.921 INFO:tasks.workunit.client.0.smithi114.stdout: 1 305661 6.03 MB/sec execute 114 sec latency 28144.965 ms 2024-04-23T10:31:46.492 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 23 10:31:46 smithi119 ceph-mon[32608]: pgmap v159: 97 pgs: 97 active+clean; 80 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 78 KiB/s rd, 1.1 MiB/s wr, 52 op/s 2024-04-23T10:31:46.504 INFO:journalctl@ceph.mon.a.smithi114.stdout:Apr 23 10:31:46 smithi114 ceph-mon[29255]: pgmap v159: 97 pgs: 97 active+clean; 80 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 78 KiB/s rd, 1.1 MiB/s wr, 52 op/s 2024-04-23T10:31:46.577 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 23 10:31:46 smithi204 ceph-mon[32649]: pgmap v159: 97 pgs: 97 active+clean; 80 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 78 KiB/s rd, 1.1 MiB/s wr, 52 op/s 2024-04-23T10:31:46.921 INFO:tasks.workunit.client.0.smithi114.stdout: 1 305661 5.98 MB/sec execute 115 sec latency 29145.062 ms 2024-04-23T10:31:47.921 INFO:tasks.workunit.client.0.smithi114.stdout: 1 305661 5.93 MB/sec execute 116 sec latency 30145.109 ms 2024-04-23T10:31:48.492 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 23 10:31:48 smithi119 ceph-mon[32608]: pgmap v160: 97 pgs: 97 active+clean; 79 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 119 KiB/s rd, 1.5 MiB/s wr, 79 op/s 2024-04-23T10:31:48.504 INFO:journalctl@ceph.mon.a.smithi114.stdout:Apr 23 10:31:48 smithi114 ceph-mon[29255]: pgmap v160: 97 pgs: 97 active+clean; 79 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 119 KiB/s rd, 1.5 MiB/s wr, 79 op/s 2024-04-23T10:31:48.578 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 23 10:31:48 smithi204 ceph-mon[32649]: pgmap v160: 97 pgs: 97 active+clean; 79 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 119 KiB/s rd, 1.5 MiB/s wr, 79 op/s 2024-04-23T10:31:48.921 INFO:tasks.workunit.client.0.smithi114.stdout: 1 305661 5.88 MB/sec execute 117 sec latency 31145.172 ms 2024-04-23T10:31:49.921 INFO:tasks.workunit.client.0.smithi114.stdout: 1 305661 5.83 MB/sec execute 118 sec latency 32145.258 ms 2024-04-23T10:31:50.492 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 23 10:31:50 smithi119 ceph-mon[32608]: pgmap v161: 97 pgs: 97 active+clean; 79 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 83 KiB/s rd, 1.2 MiB/s wr, 55 op/s 2024-04-23T10:31:50.492 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 23 10:31:50 smithi119 ceph-mon[32608]: from='mgr.15291 172.21.15.114:0/1095789187' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-23T10:31:50.492 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 23 10:31:50 smithi119 ceph-mon[32608]: from='mgr.15291 172.21.15.114:0/1095789187' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-23T10:31:50.492 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 23 10:31:50 smithi119 ceph-mon[32608]: from='mgr.15291 172.21.15.114:0/1095789187' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-23T10:31:50.492 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 23 10:31:50 smithi119 ceph-mon[32608]: from='mgr.15291 172.21.15.114:0/1095789187' entity='mgr.x' 2024-04-23T10:31:50.504 INFO:journalctl@ceph.mon.a.smithi114.stdout:Apr 23 10:31:50 smithi114 ceph-mon[29255]: pgmap v161: 97 pgs: 97 active+clean; 79 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 83 KiB/s rd, 1.2 MiB/s wr, 55 op/s 2024-04-23T10:31:50.504 INFO:journalctl@ceph.mon.a.smithi114.stdout:Apr 23 10:31:50 smithi114 ceph-mon[29255]: from='mgr.15291 172.21.15.114:0/1095789187' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-23T10:31:50.504 INFO:journalctl@ceph.mon.a.smithi114.stdout:Apr 23 10:31:50 smithi114 ceph-mon[29255]: from='mgr.15291 172.21.15.114:0/1095789187' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-23T10:31:50.504 INFO:journalctl@ceph.mon.a.smithi114.stdout:Apr 23 10:31:50 smithi114 ceph-mon[29255]: from='mgr.15291 172.21.15.114:0/1095789187' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-23T10:31:50.504 INFO:journalctl@ceph.mon.a.smithi114.stdout:Apr 23 10:31:50 smithi114 ceph-mon[29255]: from='mgr.15291 172.21.15.114:0/1095789187' entity='mgr.x' 2024-04-23T10:31:50.577 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 23 10:31:50 smithi204 ceph-mon[32649]: pgmap v161: 97 pgs: 97 active+clean; 79 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 83 KiB/s rd, 1.2 MiB/s wr, 55 op/s 2024-04-23T10:31:50.577 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 23 10:31:50 smithi204 ceph-mon[32649]: from='mgr.15291 172.21.15.114:0/1095789187' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-23T10:31:50.578 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 23 10:31:50 smithi204 ceph-mon[32649]: from='mgr.15291 172.21.15.114:0/1095789187' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-23T10:31:50.578 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 23 10:31:50 smithi204 ceph-mon[32649]: from='mgr.15291 172.21.15.114:0/1095789187' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-23T10:31:50.578 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 23 10:31:50 smithi204 ceph-mon[32649]: from='mgr.15291 172.21.15.114:0/1095789187' entity='mgr.x' 2024-04-23T10:31:50.921 INFO:tasks.workunit.client.0.smithi114.stdout: 1 305661 5.78 MB/sec execute 119 sec latency 33145.379 ms 2024-04-23T10:31:51.921 INFO:tasks.workunit.client.0.smithi114.stdout: 1 305661 5.73 MB/sec execute 120 sec latency 34145.461 ms 2024-04-23T10:31:52.492 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 23 10:31:52 smithi119 ceph-mon[32608]: pgmap v162: 97 pgs: 97 active+clean; 79 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 83 KiB/s rd, 1.2 MiB/s wr, 55 op/s 2024-04-23T10:31:52.504 INFO:journalctl@ceph.mon.a.smithi114.stdout:Apr 23 10:31:52 smithi114 ceph-mon[29255]: pgmap v162: 97 pgs: 97 active+clean; 79 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 83 KiB/s rd, 1.2 MiB/s wr, 55 op/s 2024-04-23T10:31:52.578 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 23 10:31:52 smithi204 ceph-mon[32649]: pgmap v162: 97 pgs: 97 active+clean; 79 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 83 KiB/s rd, 1.2 MiB/s wr, 55 op/s 2024-04-23T10:31:52.921 INFO:tasks.workunit.client.0.smithi114.stdout: 1 305661 5.68 MB/sec execute 121 sec latency 35145.539 ms 2024-04-23T10:31:53.921 INFO:tasks.workunit.client.0.smithi114.stdout: 1 305661 5.64 MB/sec execute 122 sec latency 36145.647 ms 2024-04-23T10:31:54.492 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 23 10:31:54 smithi119 ceph-mon[32608]: pgmap v163: 97 pgs: 97 active+clean; 78 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 111 KiB/s rd, 1.9 MiB/s wr, 79 op/s 2024-04-23T10:31:54.504 INFO:journalctl@ceph.mon.a.smithi114.stdout:Apr 23 10:31:54 smithi114 ceph-mon[29255]: pgmap v163: 97 pgs: 97 active+clean; 78 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 111 KiB/s rd, 1.9 MiB/s wr, 79 op/s 2024-04-23T10:31:54.577 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 23 10:31:54 smithi204 ceph-mon[32649]: pgmap v163: 97 pgs: 97 active+clean; 78 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 111 KiB/s rd, 1.9 MiB/s wr, 79 op/s 2024-04-23T10:31:54.921 INFO:tasks.workunit.client.0.smithi114.stdout: 1 305661 5.59 MB/sec execute 123 sec latency 37145.756 ms 2024-04-23T10:31:55.921 INFO:tasks.workunit.client.0.smithi114.stdout: 1 305661 5.55 MB/sec execute 124 sec latency 38145.843 ms 2024-04-23T10:31:56.492 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 23 10:31:56 smithi119 ceph-mon[32608]: pgmap v164: 97 pgs: 97 active+clean; 78 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 74 KiB/s rd, 1.2 MiB/s wr, 54 op/s 2024-04-23T10:31:56.505 INFO:journalctl@ceph.mon.a.smithi114.stdout:Apr 23 10:31:56 smithi114 ceph-mon[29255]: pgmap v164: 97 pgs: 97 active+clean; 78 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 74 KiB/s rd, 1.2 MiB/s wr, 54 op/s 2024-04-23T10:31:56.578 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 23 10:31:56 smithi204 ceph-mon[32649]: pgmap v164: 97 pgs: 97 active+clean; 78 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 74 KiB/s rd, 1.2 MiB/s wr, 54 op/s 2024-04-23T10:31:56.922 INFO:tasks.workunit.client.0.smithi114.stdout: 1 305661 5.50 MB/sec execute 125 sec latency 39145.947 ms 2024-04-23T10:31:57.922 INFO:tasks.workunit.client.0.smithi114.stdout: 1 305661 5.46 MB/sec execute 126 sec latency 40146.056 ms 2024-04-23T10:31:58.492 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 23 10:31:58 smithi119 ceph-mon[32608]: pgmap v165: 97 pgs: 97 active+clean; 78 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 97 KiB/s rd, 1.3 MiB/s wr, 62 op/s 2024-04-23T10:31:58.504 INFO:journalctl@ceph.mon.a.smithi114.stdout:Apr 23 10:31:58 smithi114 ceph-mon[29255]: pgmap v165: 97 pgs: 97 active+clean; 78 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 97 KiB/s rd, 1.3 MiB/s wr, 62 op/s 2024-04-23T10:31:58.577 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 23 10:31:58 smithi204 ceph-mon[32649]: pgmap v165: 97 pgs: 97 active+clean; 78 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 97 KiB/s rd, 1.3 MiB/s wr, 62 op/s 2024-04-23T10:31:58.622 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:31:58.620+0000 7f78daf08640 1 -- 172.21.15.114:0/208864231 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f78d41063a0 msgr2=0x7f78d410df70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T10:31:58.622 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:31:58.620+0000 7f78daf08640 1 --2- 172.21.15.114:0/208864231 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f78d41063a0 0x7f78d410df70 secure :-1 s=READY pgs=674 cs=0 l=1 rev1=1 crypto rx=0x7f78cc009ca0 tx=0x7f78cc02a8c0 comp rx=0 tx=0).stop 2024-04-23T10:31:58.622 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:31:58.620+0000 7f78daf08640 1 -- 172.21.15.114:0/208864231 shutdown_connections 2024-04-23T10:31:58.622 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:31:58.620+0000 7f78daf08640 1 --2- 172.21.15.114:0/208864231 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f78d41063a0 0x7f78d410df70 unknown :-1 s=CLOSED pgs=674 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:31:58.622 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:31:58.620+0000 7f78daf08640 1 --2- 172.21.15.114:0/208864231 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f78d41021a0 0x7f78d4105cc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:31:58.622 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:31:58.620+0000 7f78daf08640 1 --2- 172.21.15.114:0/208864231 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f78d41017f0 0x7f78d4101bd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:31:58.622 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:31:58.620+0000 7f78daf08640 1 -- 172.21.15.114:0/208864231 >> 172.21.15.114:0/208864231 conn(0x7f78d4077390 msgr2=0x7f78d40fe760 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T10:31:58.622 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:31:58.620+0000 7f78daf08640 1 -- 172.21.15.114:0/208864231 shutdown_connections 2024-04-23T10:31:58.622 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:31:58.620+0000 7f78daf08640 1 -- 172.21.15.114:0/208864231 wait complete. 2024-04-23T10:31:58.622 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:31:58.621+0000 7f78daf08640 1 Processor -- start 2024-04-23T10:31:58.622 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:31:58.621+0000 7f78daf08640 1 -- start start 2024-04-23T10:31:58.622 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:31:58.621+0000 7f78daf08640 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f78d41017f0 0x7f78d419e610 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T10:31:58.623 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:31:58.621+0000 7f78daf08640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f78d41063a0 0x7f78d419eb50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T10:31:58.623 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:31:58.621+0000 7f78daf08640 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f78d41986d0 0x7f78d4198b50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T10:31:58.623 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:31:58.621+0000 7f78daf08640 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x7f78d4111310 con 0x7f78d41063a0 2024-04-23T10:31:58.623 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:31:58.621+0000 7f78daf08640 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 -- 0x7f78d4111190 con 0x7f78d41017f0 2024-04-23T10:31:58.623 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:31:58.621+0000 7f78daf08640 1 -- --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_getmap magic: 0 -- 0x7f78d4111490 con 0x7f78d41986d0 2024-04-23T10:31:58.623 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:31:58.621+0000 7f78d3fff640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f78d41063a0 0x7f78d419eb50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T10:31:58.623 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:31:58.621+0000 7f78d3fff640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f78d41063a0 0x7f78d419eb50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.114:51582/0 (socket says 172.21.15.114:51582) 2024-04-23T10:31:58.623 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:31:58.621+0000 7f78d3fff640 1 -- 172.21.15.114:0/7987839 learned_addr learned my addr 172.21.15.114:0/7987839 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-04-23T10:31:58.623 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:31:58.621+0000 7f78d94a5640 1 --2- 172.21.15.114:0/7987839 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f78d41986d0 0x7f78d4198b50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T10:31:58.623 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:31:58.622+0000 7f78d8ca4640 1 --2- 172.21.15.114:0/7987839 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f78d41017f0 0x7f78d419e610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T10:31:58.623 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:31:58.622+0000 7f78d3fff640 1 -- 172.21.15.114:0/7987839 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f78d41986d0 msgr2=0x7f78d4198b50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T10:31:58.623 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:31:58.622+0000 7f78d3fff640 1 --2- 172.21.15.114:0/7987839 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f78d41986d0 0x7f78d4198b50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:31:58.623 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:31:58.622+0000 7f78d3fff640 1 -- 172.21.15.114:0/7987839 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f78d41017f0 msgr2=0x7f78d419e610 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T10:31:58.624 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:31:58.622+0000 7f78d3fff640 1 --2- 172.21.15.114:0/7987839 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f78d41017f0 0x7f78d419e610 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:31:58.624 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:31:58.622+0000 7f78d3fff640 1 -- 172.21.15.114:0/7987839 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f78d41992d0 con 0x7f78d41063a0 2024-04-23T10:31:58.624 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:31:58.622+0000 7f78d3fff640 1 --2- 172.21.15.114:0/7987839 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f78d41063a0 0x7f78d419eb50 secure :-1 s=READY pgs=675 cs=0 l=1 rev1=1 crypto rx=0x7f78c000d770 tx=0x7f78c000dc30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T10:31:58.624 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:31:58.622+0000 7f78d1ffb640 1 -- 172.21.15.114:0/7987839 <== mon.0 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f78c0017070 con 0x7f78d41063a0 2024-04-23T10:31:58.624 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:31:58.622+0000 7f78d1ffb640 1 -- 172.21.15.114:0/7987839 <== mon.0 v2:172.21.15.114:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f78c0009c30 con 0x7f78d41063a0 2024-04-23T10:31:58.624 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:31:58.622+0000 7f78daf08640 1 -- 172.21.15.114:0/7987839 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f78d41a53b0 con 0x7f78d41063a0 2024-04-23T10:31:58.624 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:31:58.622+0000 7f78d1ffb640 1 -- 172.21.15.114:0/7987839 <== mon.0 v2:172.21.15.114:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f78c000f620 con 0x7f78d41063a0 2024-04-23T10:31:58.624 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:31:58.622+0000 7f78daf08640 1 -- 172.21.15.114:0/7987839 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f78d41a57b0 con 0x7f78d41063a0 2024-04-23T10:31:58.631 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:31:58.625+0000 7f78d1ffb640 1 -- 172.21.15.114:0/7987839 <== mon.0 v2:172.21.15.114:3300/0 4 ==== mgrmap(e 26) ==== 138778+0+0 (secure 0 0 0) 0x7f78c001a070 con 0x7f78d41063a0 2024-04-23T10:31:58.631 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:31:58.625+0000 7f78ddc29740 1 Processor -- start 2024-04-23T10:31:58.631 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:31:58.625+0000 7f78ddc29740 1 -- start start 2024-04-23T10:31:58.631 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:31:58.625+0000 7f78ddc29740 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x55d94d20f620 0x55d94d20fa00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T10:31:58.631 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:31:58.625+0000 7f78ddc29740 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55d94d20ff40 0x55d94d21a2c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T10:31:58.631 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:31:58.625+0000 7f78ddc29740 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x55d94d21a800 0x55d94d21cbf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T10:31:58.631 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:31:58.625+0000 7f78ddc29740 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x55d94d174970 con 0x55d94d20ff40 2024-04-23T10:31:58.631 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:31:58.625+0000 7f78ddc29740 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 -- 0x55d94d1114a0 con 0x55d94d20f620 2024-04-23T10:31:58.631 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:31:58.625+0000 7f78ddc29740 1 -- --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_getmap magic: 0 -- 0x55d94d170930 con 0x55d94d21a800 2024-04-23T10:31:58.631 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:31:58.625+0000 7f78d1ffb640 1 --2- 172.21.15.114:0/7987839 >> [v2:172.21.15.114:6800/1395693466,v1:172.21.15.114:6801/1395693466] conn(0x7f78c40a3dc0 0x7f78c40a6280 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T10:31:58.632 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:31:58.625+0000 7f78d8ca4640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55d94d20ff40 0x55d94d21a2c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T10:31:58.632 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:31:58.625+0000 7f78d8ca4640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55d94d20ff40 0x55d94d21a2c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.114:51594/0 (socket says 172.21.15.114:51594) 2024-04-23T10:31:58.632 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:31:58.625+0000 7f78d8ca4640 1 -- 172.21.15.114:0/1694088776 learned_addr learned my addr 172.21.15.114:0/1694088776 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-04-23T10:31:58.632 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:31:58.625+0000 7f78d94a5640 1 --2- 172.21.15.114:0/1694088776 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x55d94d20f620 0x55d94d20fa00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T10:31:58.632 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:31:58.625+0000 7f78d3fff640 1 --2- 172.21.15.114:0/1694088776 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x55d94d21a800 0x55d94d21cbf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T10:31:58.632 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:31:58.625+0000 7f78d94a5640 1 --2- 172.21.15.114:0/7987839 >> [v2:172.21.15.114:6800/1395693466,v1:172.21.15.114:6801/1395693466] conn(0x7f78c40a3dc0 0x7f78c40a6280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T10:31:58.632 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:31:58.626+0000 7f78d8ca4640 1 -- 172.21.15.114:0/1694088776 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x55d94d21a800 msgr2=0x55d94d21cbf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T10:31:58.632 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:31:58.626+0000 7f78d8ca4640 1 --2- 172.21.15.114:0/1694088776 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x55d94d21a800 0x55d94d21cbf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:31:58.632 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:31:58.626+0000 7f78d8ca4640 1 -- 172.21.15.114:0/1694088776 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x55d94d20f620 msgr2=0x55d94d20fa00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T10:31:58.632 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:31:58.626+0000 7f78d8ca4640 1 --2- 172.21.15.114:0/1694088776 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x55d94d20f620 0x55d94d20fa00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:31:58.632 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:31:58.626+0000 7f78d1ffb640 1 -- 172.21.15.114:0/7987839 <== mon.0 v2:172.21.15.114:3300/0 5 ==== osd_map(94..94 src has 1..94) ==== 8585+0+0 (secure 0 0 0) 0x7f78c00d0240 con 0x7f78d41063a0 2024-04-23T10:31:58.632 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:31:58.626+0000 7f78d8ca4640 1 -- 172.21.15.114:0/1694088776 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55d94ce87210 con 0x55d94d20ff40 2024-04-23T10:31:58.632 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:31:58.626+0000 7f78d8ca4640 1 --2- 172.21.15.114:0/1694088776 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55d94d20ff40 0x55d94d21a2c0 secure :-1 s=READY pgs=676 cs=0 l=1 rev1=1 crypto rx=0x7f78c8009d20 tx=0x7f78c8011810 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8cf6300ab9c2a55 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T10:31:58.632 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:31:58.626+0000 7f78beffd640 1 -- 172.21.15.114:0/1694088776 <== mon.0 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f78c8023070 con 0x55d94d20ff40 2024-04-23T10:31:58.633 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:31:58.626+0000 7f78d94a5640 1 --2- 172.21.15.114:0/7987839 >> [v2:172.21.15.114:6800/1395693466,v1:172.21.15.114:6801/1395693466] conn(0x7f78c40a3dc0 0x7f78c40a6280 secure :-1 s=READY pgs=105 cs=0 l=1 rev1=1 crypto rx=0x7f78cc02f500 tx=0x7f78cc0095c0 comp rx=0 tx=0).ready entity=mgr.15291 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T10:31:58.633 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:31:58.626+0000 7f78beffd640 1 -- 172.21.15.114:0/1694088776 <== mon.0 v2:172.21.15.114:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f78c8016b00 con 0x55d94d20ff40 2024-04-23T10:31:58.633 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:31:58.626+0000 7f78beffd640 1 -- 172.21.15.114:0/1694088776 <== mon.0 v2:172.21.15.114:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f78c801eb10 con 0x55d94d20ff40 2024-04-23T10:31:58.633 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:31:58.627+0000 7f78ddc29740 1 -- 172.21.15.114:0/1694088776 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55d94d20ff40 msgr2=0x55d94d21a2c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T10:31:58.633 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:31:58.627+0000 7f78ddc29740 1 --2- 172.21.15.114:0/1694088776 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55d94d20ff40 0x55d94d21a2c0 secure :-1 s=READY pgs=676 cs=0 l=1 rev1=1 crypto rx=0x7f78c8009d20 tx=0x7f78c8011810 comp rx=0 tx=0).stop 2024-04-23T10:31:58.633 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:31:58.627+0000 7f78ddc29740 1 -- 172.21.15.114:0/1694088776 shutdown_connections 2024-04-23T10:31:58.633 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:31:58.627+0000 7f78ddc29740 1 --2- 172.21.15.114:0/1694088776 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x55d94d21a800 0x55d94d21cbf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:31:58.633 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:31:58.627+0000 7f78ddc29740 1 --2- 172.21.15.114:0/1694088776 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55d94d20ff40 0x55d94d21a2c0 unknown :-1 s=CLOSED pgs=676 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:31:58.633 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:31:58.627+0000 7f78ddc29740 1 --2- 172.21.15.114:0/1694088776 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x55d94d20f620 0x55d94d20fa00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:31:58.633 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:31:58.627+0000 7f78ddc29740 1 -- 172.21.15.114:0/1694088776 >> 172.21.15.114:0/1694088776 conn(0x55d94d20bb30 msgr2=0x55d94d20bf10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T10:31:58.633 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:31:58.627+0000 7f78ddc29740 1 -- 172.21.15.114:0/1694088776 shutdown_connections 2024-04-23T10:31:58.633 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:31:58.627+0000 7f78ddc29740 1 -- 172.21.15.114:0/1694088776 wait complete. 2024-04-23T10:31:58.633 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:31:58.627+0000 7f78ddc29740 1 -- start start 2024-04-23T10:31:58.633 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:31:58.627+0000 7f78ddc29740 1 Processor -- start 2024-04-23T10:31:58.633 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:31:58.628+0000 7f78ddc29740 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x55d94d20f620 0x55d94d21a100 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T10:31:58.634 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:31:58.628+0000 7f78ddc29740 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55d94d20ff40 0x55d94d2173f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T10:31:58.634 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:31:58.628+0000 7f78ddc29740 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x55d94d21a800 0x55d94d217930 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T10:31:58.634 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:31:58.628+0000 7f78ddc29740 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x55d94d21d430 con 0x55d94d20ff40 2024-04-23T10:31:58.634 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:31:58.628+0000 7f78ddc29740 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 -- 0x55d94d10f0b0 con 0x55d94d21a800 2024-04-23T10:31:58.634 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:31:58.628+0000 7f78ddc29740 1 -- --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_getmap magic: 0 -- 0x55d94d21d5b0 con 0x55d94d20f620 2024-04-23T10:31:58.634 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:31:58.628+0000 7f78ddc29740 10 client.0 _ll_register_callbacks cb 0x55d94d0fcd30 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-23T10:31:58.634 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:31:58.628+0000 7f78d3fff640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55d94d20ff40 0x55d94d2173f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T10:31:58.634 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:31:58.628+0000 7f78d3fff640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55d94d20ff40 0x55d94d2173f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.114:51602/0 (socket says 172.21.15.114:51602) 2024-04-23T10:31:58.634 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:31:58.628+0000 7f78d3fff640 1 -- 172.21.15.114:0/3470379913 learned_addr learned my addr 172.21.15.114:0/3470379913 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-04-23T10:31:58.634 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:31:58.628+0000 7f78d94a5640 1 --2- 172.21.15.114:0/3470379913 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x55d94d21a800 0x55d94d217930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T10:31:58.634 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:31:58.628+0000 7f78d8ca4640 1 --2- 172.21.15.114:0/3470379913 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x55d94d20f620 0x55d94d21a100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T10:31:58.634 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:31:58.628+0000 7f78d3fff640 1 -- 172.21.15.114:0/3470379913 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x55d94d20f620 msgr2=0x55d94d21a100 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T10:31:58.634 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:31:58.628+0000 7f78d3fff640 1 --2- 172.21.15.114:0/3470379913 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x55d94d20f620 0x55d94d21a100 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:31:58.635 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:31:58.628+0000 7f78d3fff640 1 -- 172.21.15.114:0/3470379913 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x55d94d21a800 msgr2=0x55d94d217930 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T10:31:58.635 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:31:58.628+0000 7f78d3fff640 1 --2- 172.21.15.114:0/3470379913 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x55d94d21a800 0x55d94d217930 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:31:58.635 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:31:58.628+0000 7f78d3fff640 1 -- 172.21.15.114:0/3470379913 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55d94cf53620 con 0x55d94d20ff40 2024-04-23T10:31:58.635 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:31:58.628+0000 7f78d94a5640 1 --2- 172.21.15.114:0/3470379913 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x55d94d21a800 0x55d94d217930 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-23T10:31:58.635 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:31:58.628+0000 7f78d8ca4640 1 --2- 172.21.15.114:0/3470379913 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x55d94d20f620 0x55d94d21a100 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-23T10:31:58.635 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:31:58.628+0000 7f78d3fff640 1 --2- 172.21.15.114:0/3470379913 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55d94d20ff40 0x55d94d2173f0 secure :-1 s=READY pgs=677 cs=0 l=1 rev1=1 crypto rx=0x7f78c00986b0 tx=0x7f78c0066da0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3f5e57e698416213 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T10:31:58.635 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:31:58.628+0000 7f78b3fff640 10 client.0 ms_handle_connect on v2:172.21.15.114:3300/0 2024-04-23T10:31:58.635 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:31:58.628+0000 7f78b3fff640 1 -- 172.21.15.114:0/3470379913 <== mon.0 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f78c0021050 con 0x55d94d20ff40 2024-04-23T10:31:58.635 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:31:58.628+0000 7f78b3fff640 1 -- 172.21.15.114:0/3470379913 <== mon.0 v2:172.21.15.114:3300/0 2 ==== osd_map(94..94 src has 1..94) ==== 8585+0+0 (secure 0 0 0) 0x7f78c00988c0 con 0x55d94d20ff40 2024-04-23T10:31:58.635 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:31:58.628+0000 7f78ddc29740 1 -- 172.21.15.114:0/3470379913 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f78c8016b00 con 0x55d94d20ff40 2024-04-23T10:31:58.635 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:31:58.629+0000 7f78b3fff640 1 -- 172.21.15.114:0/3470379913 <== mon.0 v2:172.21.15.114:3300/0 3 ==== mon_get_version_reply(handle=1 version=30) ==== 24+0+0 (secure 0 0 0) 0x7f78c0009de0 con 0x55d94d20ff40 2024-04-23T10:31:58.635 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:31:58.629+0000 7f78ddc29740 10 client.16212 fetch_fsmap learned FSMap version 30 2024-04-23T10:31:58.635 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:31:58.629+0000 7f78ddc29740 1 -- 172.21.15.114:0/3470379913 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({config=0+,fsmap=30,monmap=4+}) -- 0x55d94d229300 con 0x55d94d20ff40 2024-04-23T10:31:58.635 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:31:58.629+0000 7f78b3fff640 1 -- 172.21.15.114:0/3470379913 <== mon.0 v2:172.21.15.114:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f78c8016b00 con 0x55d94d20ff40 2024-04-23T10:31:58.636 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:31:58.629+0000 7f78b3fff640 1 -- 172.21.15.114:0/3470379913 <== mon.0 v2:172.21.15.114:3300/0 5 ==== fsmap(e 30) ==== 8338+0+0 (secure 0 0 0) 0x7f78c00a0160 con 0x55d94d20ff40 2024-04-23T10:31:58.636 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:31:58.629+0000 7f78ddc29740 10 client.16212 fetch_fsmap finished waiting for FSMap version 30 2024-04-23T10:31:58.636 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:31:58.629+0000 7f78ddc29740 10 client.16212 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-04-23T10:31:58.636 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:31:58.629+0000 7f78ddc29740 20 client.16212 populate_metadata read hostname 'smithi114' 2024-04-23T10:31:58.636 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:31:58.629+0000 7f78ddc29740 1 --2- 172.21.15.114:0/3470379913 >> [v2:172.21.15.119:6838/937297340,v1:172.21.15.119:6839/937297340] conn(0x55d94d22a7c0 0x55d94d22cbb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T10:31:58.636 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:31:58.629+0000 7f78ddc29740 4 client.16212 mds_command: new command op to 24545 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-23T10:31:58.636 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:31:58.629+0000 7f78ddc29740 1 -- 172.21.15.114:0/3470379913 --> [v2:172.21.15.119:6838/937297340,v1:172.21.15.119:6839/937297340] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55d94d182390 con 0x55d94d22a7c0 2024-04-23T10:31:58.636 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:31:58.629+0000 7f78d8ca4640 1 --2- 172.21.15.114:0/3470379913 >> [v2:172.21.15.119:6838/937297340,v1:172.21.15.119:6839/937297340] conn(0x55d94d22a7c0 0x55d94d22cbb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T10:31:58.636 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:31:58.630+0000 7f78b3fff640 10 client.16212 ms_handle_connect on v2:172.21.15.119:6838/937297340 2024-04-23T10:31:58.636 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:31:58.630+0000 7f78d8ca4640 1 --2- 172.21.15.114:0/3470379913 >> [v2:172.21.15.119:6838/937297340,v1:172.21.15.119:6839/937297340] conn(0x55d94d22a7c0 0x55d94d22cbb0 crc :-1 s=READY pgs=104 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=baaec6796618584d server_cookie=3d2cc827195ed3f8 in_seq=0 out_seq=0 2024-04-23T10:31:58.636 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:31:58.630+0000 7f78b3fff640 1 -- 172.21.15.114:0/3470379913 <== mds.0 v2:172.21.15.119:6838/937297340 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x55d94d182390 con 0x55d94d22a7c0 2024-04-23T10:31:58.636 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:31:58.630+0000 7f78b3fff640 10 client.16212 handle_command_reply: tid=0 2024-04-23T10:31:58.636 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:31:58.630+0000 7f78ddc29740 1 client.16212 shutdown 2024-04-23T10:31:58.637 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:31:58.630+0000 7f78ddc29740 1 -- 172.21.15.114:0/3470379913 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55d94d20ff40 msgr2=0x55d94d2173f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T10:31:58.637 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:31:58.630+0000 7f78ddc29740 1 --2- 172.21.15.114:0/3470379913 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55d94d20ff40 0x55d94d2173f0 secure :-1 s=READY pgs=677 cs=0 l=1 rev1=1 crypto rx=0x7f78c00986b0 tx=0x7f78c0066da0 comp rx=0 tx=0).stop 2024-04-23T10:31:58.637 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:31:58.630+0000 7f78ddc29740 1 -- 172.21.15.114:0/3470379913 shutdown_connections 2024-04-23T10:31:58.637 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:31:58.630+0000 7f78ddc29740 1 --2- 172.21.15.114:0/3470379913 >> [v2:172.21.15.119:6838/937297340,v1:172.21.15.119:6839/937297340] conn(0x55d94d22a7c0 0x55d94d22cbb0 crc :-1 s=READY pgs=104 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:31:58.637 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:31:58.630+0000 7f78ddc29740 1 --2- 172.21.15.114:0/3470379913 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x55d94d21a800 0x55d94d217930 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:31:58.637 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:31:58.630+0000 7f78b3fff640 0 client.16212 ms_handle_reset on v2:172.21.15.119:6838/937297340 2024-04-23T10:31:58.637 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:31:58.630+0000 7f78ddc29740 1 --2- 172.21.15.114:0/3470379913 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55d94d20ff40 0x55d94d2173f0 unknown :-1 s=CLOSED pgs=677 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:31:58.637 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:31:58.630+0000 7f78ddc29740 1 --2- 172.21.15.114:0/3470379913 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x55d94d20f620 0x55d94d21a100 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:31:58.637 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:31:58.630+0000 7f78ddc29740 1 -- 172.21.15.114:0/3470379913 >> 172.21.15.114:0/3470379913 conn(0x55d94d20bb30 msgr2=0x55d94d2118b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T10:31:58.637 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:31:58.631+0000 7f78ddc29740 1 -- 172.21.15.114:0/3470379913 shutdown_connections 2024-04-23T10:31:58.637 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:31:58.631+0000 7f78ddc29740 1 -- 172.21.15.114:0/3470379913 wait complete. 2024-04-23T10:31:58.637 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:31:58.631+0000 7f78ddc29740 20 client.16212 trim_cache size 0 max 16384 2024-04-23T10:31:58.645 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:31:58.645+0000 7f78ddc29740 1 Processor -- start 2024-04-23T10:31:58.645 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:31:58.645+0000 7f78ddc29740 1 -- start start 2024-04-23T10:31:58.645 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:31:58.646+0000 7f78ddc29740 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x55d94d20f620 0x55d94d2262f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T10:31:58.645 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:31:58.646+0000 7f78ddc29740 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x55d94d20ff40 0x55d94d232470 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T10:31:58.645 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:31:58.646+0000 7f78ddc29740 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55d94d21a800 0x55d94d2329b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T10:31:58.646 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:31:58.646+0000 7f78ddc29740 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x7f78d41ae400 con 0x55d94d21a800 2024-04-23T10:31:58.646 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:31:58.646+0000 7f78ddc29740 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 -- 0x7f78d41abcd0 con 0x55d94d20f620 2024-04-23T10:31:58.646 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:31:58.646+0000 7f78ddc29740 1 -- --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_getmap magic: 0 -- 0x7f78d41a9b40 con 0x55d94d20ff40 2024-04-23T10:31:58.646 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:31:58.646+0000 7f78d94a5640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55d94d21a800 0x55d94d2329b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T10:31:58.646 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:31:58.646+0000 7f78d94a5640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55d94d21a800 0x55d94d2329b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.114:51616/0 (socket says 172.21.15.114:51616) 2024-04-23T10:31:58.646 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:31:58.646+0000 7f78d94a5640 1 -- 172.21.15.114:0/4273230123 learned_addr learned my addr 172.21.15.114:0/4273230123 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-04-23T10:31:58.646 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:31:58.646+0000 7f78d94a5640 1 -- 172.21.15.114:0/4273230123 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x55d94d20ff40 msgr2=0x55d94d232470 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T10:31:58.646 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:31:58.646+0000 7f78d94a5640 1 --2- 172.21.15.114:0/4273230123 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x55d94d20ff40 0x55d94d232470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:31:58.646 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:31:58.646+0000 7f78d94a5640 1 -- 172.21.15.114:0/4273230123 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x55d94d20f620 msgr2=0x55d94d2262f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T10:31:58.646 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:31:58.646+0000 7f78d94a5640 1 --2- 172.21.15.114:0/4273230123 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x55d94d20f620 0x55d94d2262f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:31:58.646 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:31:58.646+0000 7f78d94a5640 1 -- 172.21.15.114:0/4273230123 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55d94d226e20 con 0x55d94d21a800 2024-04-23T10:31:58.646 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:31:58.646+0000 7f78d94a5640 1 --2- 172.21.15.114:0/4273230123 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55d94d21a800 0x55d94d2329b0 secure :-1 s=READY pgs=678 cs=0 l=1 rev1=1 crypto rx=0x7f78cc009ca0 tx=0x7f78cc044000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1a55330a35bf7d33 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T10:31:58.646 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:31:58.646+0000 7f78be7fc640 1 -- 172.21.15.114:0/4273230123 <== mon.0 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f78cc055070 con 0x55d94d21a800 2024-04-23T10:31:58.646 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:31:58.646+0000 7f78be7fc640 1 -- 172.21.15.114:0/4273230123 <== mon.0 v2:172.21.15.114:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f78cc036070 con 0x55d94d21a800 2024-04-23T10:31:58.647 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:31:58.646+0000 7f78be7fc640 1 -- 172.21.15.114:0/4273230123 <== mon.0 v2:172.21.15.114:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f78cc004680 con 0x55d94d21a800 2024-04-23T10:31:58.650 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:31:58.647+0000 7f78ddc29740 1 -- 172.21.15.114:0/4273230123 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55d94d21a800 msgr2=0x55d94d2329b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T10:31:58.650 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:31:58.647+0000 7f78ddc29740 1 --2- 172.21.15.114:0/4273230123 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55d94d21a800 0x55d94d2329b0 secure :-1 s=READY pgs=678 cs=0 l=1 rev1=1 crypto rx=0x7f78cc009ca0 tx=0x7f78cc044000 comp rx=0 tx=0).stop 2024-04-23T10:31:58.650 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:31:58.647+0000 7f78ddc29740 1 -- 172.21.15.114:0/4273230123 shutdown_connections 2024-04-23T10:31:58.650 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:31:58.647+0000 7f78ddc29740 1 --2- 172.21.15.114:0/4273230123 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55d94d21a800 0x55d94d2329b0 unknown :-1 s=CLOSED pgs=678 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:31:58.651 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:31:58.647+0000 7f78ddc29740 1 --2- 172.21.15.114:0/4273230123 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x55d94d20ff40 0x55d94d232470 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:31:58.651 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:31:58.647+0000 7f78ddc29740 1 --2- 172.21.15.114:0/4273230123 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x55d94d20f620 0x55d94d2262f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:31:58.651 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:31:58.647+0000 7f78ddc29740 1 -- 172.21.15.114:0/4273230123 >> 172.21.15.114:0/4273230123 conn(0x55d94d20bb30 msgr2=0x55d94d20a420 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T10:31:58.651 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:31:58.647+0000 7f78ddc29740 1 -- 172.21.15.114:0/4273230123 shutdown_connections 2024-04-23T10:31:58.651 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:31:58.647+0000 7f78ddc29740 1 -- 172.21.15.114:0/4273230123 wait complete. 2024-04-23T10:31:58.651 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:31:58.647+0000 7f78ddc29740 1 -- start start 2024-04-23T10:31:58.651 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:31:58.647+0000 7f78ddc29740 1 Processor -- start 2024-04-23T10:31:58.651 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:31:58.648+0000 7f78ddc29740 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x55d94d20f620 0x55d94d222ac0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T10:31:58.651 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:31:58.648+0000 7f78ddc29740 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55d94d20ff40 0x55d94d21fea0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T10:31:58.651 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:31:58.648+0000 7f78ddc29740 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x55d94d21a800 0x55d94d2203e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T10:31:58.651 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:31:58.648+0000 7f78ddc29740 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x7f78d41aa190 con 0x55d94d20ff40 2024-04-23T10:31:58.651 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:31:58.648+0000 7f78ddc29740 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 -- 0x7f78d41ab600 con 0x55d94d21a800 2024-04-23T10:31:58.651 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:31:58.648+0000 7f78ddc29740 1 -- --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_getmap magic: 0 -- 0x7f78d41aa860 con 0x55d94d20f620 2024-04-23T10:31:58.651 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:31:58.648+0000 7f78ddc29740 10 client.0 _ll_register_callbacks cb 0x7f78c0098510 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-23T10:31:58.652 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:31:58.648+0000 7f78d3fff640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55d94d20ff40 0x55d94d21fea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T10:31:58.652 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:31:58.648+0000 7f78d3fff640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55d94d20ff40 0x55d94d21fea0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.114:51632/0 (socket says 172.21.15.114:51632) 2024-04-23T10:31:58.652 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:31:58.648+0000 7f78d3fff640 1 -- 172.21.15.114:0/1269759685 learned_addr learned my addr 172.21.15.114:0/1269759685 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-04-23T10:31:58.652 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:31:58.648+0000 7f78d8ca4640 1 --2- 172.21.15.114:0/1269759685 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x55d94d20f620 0x55d94d222ac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T10:31:58.652 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:31:58.648+0000 7f78d94a5640 1 --2- 172.21.15.114:0/1269759685 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x55d94d21a800 0x55d94d2203e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T10:31:58.652 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:31:58.648+0000 7f78d3fff640 1 -- 172.21.15.114:0/1269759685 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x55d94d20f620 msgr2=0x55d94d222ac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T10:31:58.652 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:31:58.648+0000 7f78d3fff640 1 --2- 172.21.15.114:0/1269759685 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x55d94d20f620 0x55d94d222ac0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:31:58.652 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:31:58.648+0000 7f78d3fff640 1 -- 172.21.15.114:0/1269759685 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x55d94d21a800 msgr2=0x55d94d2203e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T10:31:58.652 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:31:58.648+0000 7f78d3fff640 1 --2- 172.21.15.114:0/1269759685 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x55d94d21a800 0x55d94d2203e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:31:58.652 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:31:58.648+0000 7f78d3fff640 1 -- 172.21.15.114:0/1269759685 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55d94d220ad0 con 0x55d94d20ff40 2024-04-23T10:31:58.652 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:31:58.648+0000 7f78d8ca4640 1 --2- 172.21.15.114:0/1269759685 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x55d94d20f620 0x55d94d222ac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-23T10:31:58.652 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:31:58.648+0000 7f78d94a5640 1 --2- 172.21.15.114:0/1269759685 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x55d94d21a800 0x55d94d2203e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-23T10:31:58.652 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:31:58.648+0000 7f78d3fff640 1 --2- 172.21.15.114:0/1269759685 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55d94d20ff40 0x55d94d21fea0 secure :-1 s=READY pgs=679 cs=0 l=1 rev1=1 crypto rx=0x7f78c00cda00 tx=0x7f78c00cde00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ec37b6984a8e545f server_cookie=0 in_seq=0 out_seq=0 2024-04-23T10:31:58.652 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:31:58.648+0000 7f78b3fff640 10 client.0 ms_handle_connect on v2:172.21.15.114:3300/0 2024-04-23T10:31:58.653 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:31:58.648+0000 7f78b3fff640 1 -- 172.21.15.114:0/1269759685 <== mon.0 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f78c006c020 con 0x55d94d20ff40 2024-04-23T10:31:58.653 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:31:58.649+0000 7f78b3fff640 1 -- 172.21.15.114:0/1269759685 <== mon.0 v2:172.21.15.114:3300/0 2 ==== osd_map(94..94 src has 1..94) ==== 8585+0+0 (secure 0 0 0) 0x7f78c009b8a0 con 0x55d94d20ff40 2024-04-23T10:31:58.653 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:31:58.649+0000 7f78ddc29740 1 -- 172.21.15.114:0/1269759685 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f78cc036070 con 0x55d94d20ff40 2024-04-23T10:31:58.653 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:31:58.649+0000 7f78b3fff640 1 -- 172.21.15.114:0/1269759685 <== mon.0 v2:172.21.15.114:3300/0 3 ==== mon_get_version_reply(handle=1 version=30) ==== 24+0+0 (secure 0 0 0) 0x7f78c0021070 con 0x55d94d20ff40 2024-04-23T10:31:58.653 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:31:58.649+0000 7f78ddc29740 10 client.16218 fetch_fsmap learned FSMap version 30 2024-04-23T10:31:58.653 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:31:58.649+0000 7f78ddc29740 1 -- 172.21.15.114:0/1269759685 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({config=0+,fsmap=30,monmap=4+}) -- 0x55d94d290c00 con 0x55d94d20ff40 2024-04-23T10:31:58.653 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:31:58.649+0000 7f78b3fff640 1 -- 172.21.15.114:0/1269759685 <== mon.0 v2:172.21.15.114:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f78cc036070 con 0x55d94d20ff40 2024-04-23T10:31:58.653 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:31:58.649+0000 7f78b3fff640 1 -- 172.21.15.114:0/1269759685 <== mon.0 v2:172.21.15.114:3300/0 5 ==== fsmap(e 30) ==== 8338+0+0 (secure 0 0 0) 0x7f78c009b110 con 0x55d94d20ff40 2024-04-23T10:31:58.653 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:31:58.649+0000 7f78ddc29740 10 client.16218 fetch_fsmap finished waiting for FSMap version 30 2024-04-23T10:31:58.653 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:31:58.649+0000 7f78ddc29740 10 client.16218 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-04-23T10:31:58.653 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:31:58.649+0000 7f78ddc29740 20 client.16218 populate_metadata read hostname 'smithi114' 2024-04-23T10:31:58.653 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:31:58.649+0000 7f78ddc29740 1 --2- 172.21.15.114:0/1269759685 >> [v2:172.21.15.119:6838/937297340,v1:172.21.15.119:6839/937297340] conn(0x55d94d291160 0x55d94d293580 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T10:31:58.653 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:31:58.649+0000 7f78ddc29740 4 client.16218 mds_command: new command op to 24545 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-04-23T10:31:58.653 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:31:58.649+0000 7f78ddc29740 1 -- 172.21.15.114:0/1269759685 --> [v2:172.21.15.119:6838/937297340,v1:172.21.15.119:6839/937297340] -- command(tid 0: {"prefix": "flush journal"}) -- 0x55d94d217cc0 con 0x55d94d291160 2024-04-23T10:31:58.653 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:31:58.649+0000 7f78d8ca4640 1 --2- 172.21.15.114:0/1269759685 >> [v2:172.21.15.119:6838/937297340,v1:172.21.15.119:6839/937297340] conn(0x55d94d291160 0x55d94d293580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T10:31:58.653 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:31:58.650+0000 7f78b3fff640 10 client.16218 ms_handle_connect on v2:172.21.15.119:6838/937297340 2024-04-23T10:31:58.654 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:31:58.650+0000 7f78d8ca4640 1 --2- 172.21.15.114:0/1269759685 >> [v2:172.21.15.119:6838/937297340,v1:172.21.15.119:6839/937297340] conn(0x55d94d291160 0x55d94d293580 crc :-1 s=READY pgs=105 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=f41f5e329e48dea4 server_cookie=a49dbf104e6120ab in_seq=0 out_seq=0 2024-04-23T10:31:58.669 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:31:58.669+0000 7f78b3fff640 1 -- 172.21.15.114:0/1269759685 <== mds.0 v2:172.21.15.119:6838/937297340 1 ==== command_reply(tid 0: 0 ) ==== 8+0+44 (crc 0 0 0) 0x55d94d217cc0 con 0x55d94d291160 2024-04-23T10:31:58.669 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:31:58.669+0000 7f78b3fff640 10 client.16218 handle_command_reply: tid=0 2024-04-23T10:31:58.669 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:31:58.669+0000 7f78ddc29740 1 client.16218 shutdown 2024-04-23T10:31:58.669 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:31:58.669+0000 7f78ddc29740 1 -- 172.21.15.114:0/1269759685 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55d94d20ff40 msgr2=0x55d94d21fea0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T10:31:58.669 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:31:58.669+0000 7f78ddc29740 1 --2- 172.21.15.114:0/1269759685 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55d94d20ff40 0x55d94d21fea0 secure :-1 s=READY pgs=679 cs=0 l=1 rev1=1 crypto rx=0x7f78c00cda00 tx=0x7f78c00cde00 comp rx=0 tx=0).stop 2024-04-23T10:31:58.670 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:31:58.669+0000 7f78ddc29740 1 -- 172.21.15.114:0/1269759685 shutdown_connections 2024-04-23T10:31:58.670 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:31:58.669+0000 7f78ddc29740 1 --2- 172.21.15.114:0/1269759685 >> [v2:172.21.15.119:6838/937297340,v1:172.21.15.119:6839/937297340] conn(0x55d94d291160 0x55d94d293580 crc :-1 s=READY pgs=105 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:31:58.670 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:31:58.669+0000 7f78ddc29740 1 --2- 172.21.15.114:0/1269759685 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x55d94d21a800 0x55d94d2203e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:31:58.670 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:31:58.669+0000 7f78ddc29740 1 --2- 172.21.15.114:0/1269759685 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55d94d20ff40 0x55d94d21fea0 unknown :-1 s=CLOSED pgs=679 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:31:58.670 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:31:58.669+0000 7f78ddc29740 1 --2- 172.21.15.114:0/1269759685 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x55d94d20f620 0x55d94d222ac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:31:58.670 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:31:58.669+0000 7f78b3fff640 0 client.16218 ms_handle_reset on v2:172.21.15.119:6838/937297340 2024-04-23T10:31:58.670 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:31:58.669+0000 7f78ddc29740 1 -- 172.21.15.114:0/1269759685 >> 172.21.15.114:0/1269759685 conn(0x55d94d20bb30 msgr2=0x55d94d20acb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T10:31:58.670 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:31:58.670+0000 7f78ddc29740 1 -- 172.21.15.114:0/1269759685 shutdown_connections 2024-04-23T10:31:58.671 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:31:58.670+0000 7f78ddc29740 1 -- 172.21.15.114:0/1269759685 wait complete. 2024-04-23T10:31:58.671 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:31:58.670+0000 7f78ddc29740 20 client.16218 trim_cache size 0 max 16384 2024-04-23T10:31:58.671 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stdout:{ 2024-04-23T10:31:58.671 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stdout: "message": "", 2024-04-23T10:31:58.671 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stdout: "return_code": 0 2024-04-23T10:31:58.671 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stdout:} 2024-04-23T10:31:58.672 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:31:58.670+0000 7f78daf08640 1 -- 172.21.15.114:0/7987839 >> [v2:172.21.15.114:6800/1395693466,v1:172.21.15.114:6801/1395693466] conn(0x7f78c40a3dc0 msgr2=0x7f78c40a6280 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T10:31:58.672 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:31:58.670+0000 7f78daf08640 1 --2- 172.21.15.114:0/7987839 >> [v2:172.21.15.114:6800/1395693466,v1:172.21.15.114:6801/1395693466] conn(0x7f78c40a3dc0 0x7f78c40a6280 secure :-1 s=READY pgs=105 cs=0 l=1 rev1=1 crypto rx=0x7f78cc02f500 tx=0x7f78cc0095c0 comp rx=0 tx=0).stop 2024-04-23T10:31:58.672 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:31:58.670+0000 7f78daf08640 1 -- 172.21.15.114:0/7987839 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f78d41063a0 msgr2=0x7f78d419eb50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T10:31:58.673 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:31:58.670+0000 7f78daf08640 1 --2- 172.21.15.114:0/7987839 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f78d41063a0 0x7f78d419eb50 secure :-1 s=READY pgs=675 cs=0 l=1 rev1=1 crypto rx=0x7f78c000d770 tx=0x7f78c000dc30 comp rx=0 tx=0).stop 2024-04-23T10:31:58.673 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:31:58.670+0000 7f78daf08640 1 -- 172.21.15.114:0/7987839 shutdown_connections 2024-04-23T10:31:58.673 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:31:58.670+0000 7f78daf08640 1 --2- 172.21.15.114:0/7987839 >> [v2:172.21.15.114:6800/1395693466,v1:172.21.15.114:6801/1395693466] conn(0x7f78c40a3dc0 0x7f78c40a6280 unknown :-1 s=CLOSED pgs=105 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:31:58.673 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:31:58.670+0000 7f78daf08640 1 --2- 172.21.15.114:0/7987839 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f78d41986d0 0x7f78d4198b50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:31:58.673 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:31:58.670+0000 7f78daf08640 1 --2- 172.21.15.114:0/7987839 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f78d41063a0 0x7f78d419eb50 unknown :-1 s=CLOSED pgs=675 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:31:58.673 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:31:58.670+0000 7f78daf08640 1 --2- 172.21.15.114:0/7987839 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f78d41017f0 0x7f78d419e610 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:31:58.673 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:31:58.670+0000 7f78daf08640 1 -- 172.21.15.114:0/7987839 >> 172.21.15.114:0/7987839 conn(0x7f78d4077390 msgr2=0x7f78d4100730 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T10:31:58.673 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:31:58.670+0000 7f78daf08640 1 -- 172.21.15.114:0/7987839 shutdown_connections 2024-04-23T10:31:58.673 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:31:58.670+0000 7f78daf08640 1 -- 172.21.15.114:0/7987839 wait complete. 2024-04-23T10:31:58.922 INFO:tasks.workunit.client.0.smithi114.stdout: 1 305661 5.42 MB/sec execute 127 sec latency 41146.103 ms 2024-04-23T10:31:59.922 INFO:tasks.workunit.client.0.smithi114.stdout: 1 305661 5.37 MB/sec execute 128 sec latency 42146.143 ms 2024-04-23T10:32:00.491 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 23 10:32:00 smithi119 ceph-mon[32608]: pgmap v166: 97 pgs: 97 active+clean; 78 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 52 KiB/s rd, 750 KiB/s wr, 32 op/s 2024-04-23T10:32:00.504 INFO:journalctl@ceph.mon.a.smithi114.stdout:Apr 23 10:32:00 smithi114 ceph-mon[29255]: pgmap v166: 97 pgs: 97 active+clean; 78 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 52 KiB/s rd, 750 KiB/s wr, 32 op/s 2024-04-23T10:32:00.577 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 23 10:32:00 smithi204 ceph-mon[32649]: pgmap v166: 97 pgs: 97 active+clean; 78 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 52 KiB/s rd, 750 KiB/s wr, 32 op/s 2024-04-23T10:32:00.922 INFO:tasks.workunit.client.0.smithi114.stdout: 1 305661 5.33 MB/sec execute 129 sec latency 43146.226 ms 2024-04-23T10:32:01.922 INFO:tasks.workunit.client.0.smithi114.stdout: 1 305661 5.29 MB/sec execute 130 sec latency 44146.315 ms 2024-04-23T10:32:02.492 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 23 10:32:02 smithi119 ceph-mon[32608]: pgmap v167: 97 pgs: 97 active+clean; 78 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 52 KiB/s rd, 750 KiB/s wr, 32 op/s 2024-04-23T10:32:02.504 INFO:journalctl@ceph.mon.a.smithi114.stdout:Apr 23 10:32:02 smithi114 ceph-mon[29255]: pgmap v167: 97 pgs: 97 active+clean; 78 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 52 KiB/s rd, 750 KiB/s wr, 32 op/s 2024-04-23T10:32:02.577 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 23 10:32:02 smithi204 ceph-mon[32649]: pgmap v167: 97 pgs: 97 active+clean; 78 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 52 KiB/s rd, 750 KiB/s wr, 32 op/s 2024-04-23T10:32:02.922 INFO:tasks.workunit.client.0.smithi114.stdout: 1 305661 5.25 MB/sec execute 131 sec latency 45146.418 ms 2024-04-23T10:32:03.443 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:32:03.444+0000 7f27b97fa640 1 -- 172.21.15.114:0/1140352738 <== mds.0 v2:172.21.15.119:6838/937297340 1 ==== command_reply(tid 0: -115 ) ==== 8+0+594 (crc 0 0 0) 0x556ffeadbc00 con 0x556ffebf14e0 2024-04-23T10:32:03.444 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:32:03.444+0000 7f27b97fa640 10 client.16158 handle_command_reply: tid=0 2024-04-23T10:32:03.444 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:32:03.444+0000 7f27daa1a740 1 client.16158 shutdown 2024-04-23T10:32:03.444 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:32:03.444+0000 7f27daa1a740 1 -- 172.21.15.114:0/1140352738 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x556ffeb7a090 msgr2=0x556ffebe9490 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T10:32:03.444 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:32:03.444+0000 7f27daa1a740 1 --2- 172.21.15.114:0/1140352738 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x556ffeb7a090 0x556ffebe9490 secure :-1 s=READY pgs=661 cs=0 l=1 rev1=1 crypto rx=0x7f27c0001eb0 tx=0x7f27c0049360 comp rx=0 tx=0).stop 2024-04-23T10:32:03.444 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:32:03.444+0000 7f27daa1a740 1 -- 172.21.15.114:0/1140352738 shutdown_connections 2024-04-23T10:32:03.444 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:32:03.444+0000 7f27daa1a740 1 --2- 172.21.15.114:0/1140352738 >> [v2:172.21.15.119:6838/937297340,v1:172.21.15.119:6839/937297340] conn(0x556ffebf14e0 0x556ffebf3900 crc :-1 s=READY pgs=99 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:32:03.444 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:32:03.444+0000 7f27daa1a740 1 --2- 172.21.15.114:0/1140352738 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x556ffeb7a090 0x556ffebe9490 unknown :-1 s=CLOSED pgs=661 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:32:03.444 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:32:03.444+0000 7f27daa1a740 1 --2- 172.21.15.114:0/1140352738 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x556ffeb6f9c0 0x556ffeb8def0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:32:03.444 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:32:03.444+0000 7f27daa1a740 1 --2- 172.21.15.114:0/1140352738 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x556ffeb6f0a0 0x556ffeb8d9b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:32:03.444 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:32:03.444+0000 7f27b97fa640 0 client.16158 ms_handle_reset on v2:172.21.15.119:6838/937297340 2024-04-23T10:32:03.444 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:32:03.444+0000 7f27daa1a740 1 -- 172.21.15.114:0/1140352738 >> 172.21.15.114:0/1140352738 conn(0x556ffeb6b3c0 msgr2=0x556ffeb85250 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T10:32:03.445 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:32:03.444+0000 7f27daa1a740 1 -- 172.21.15.114:0/1140352738 shutdown_connections 2024-04-23T10:32:03.445 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:32:03.444+0000 7f27daa1a740 1 -- 172.21.15.114:0/1140352738 wait complete. 2024-04-23T10:32:03.445 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:32:03.444+0000 7f27daa1a740 20 client.16158 trim_cache size 0 max 16384 2024-04-23T10:32:03.445 INFO:teuthology.orchestra.run.smithi114.stdout:{ 2024-04-23T10:32:03.445 INFO:teuthology.orchestra.run.smithi114.stdout: "epoch": 30, 2024-04-23T10:32:03.445 INFO:teuthology.orchestra.run.smithi114.stdout: "leader": 24545, 2024-04-23T10:32:03.445 INFO:teuthology.orchestra.run.smithi114.stdout: "set_version": 17, 2024-04-23T10:32:03.445 INFO:teuthology.orchestra.run.smithi114.stdout: "sets": { 2024-04-23T10:32:03.445 INFO:teuthology.orchestra.run.smithi114.stdout: "11138c86": { 2024-04-23T10:32:03.445 INFO:teuthology.orchestra.run.smithi114.stdout: "version": 17, 2024-04-23T10:32:03.446 INFO:teuthology.orchestra.run.smithi114.stdout: "age_ref": 0.0, 2024-04-23T10:32:03.446 INFO:teuthology.orchestra.run.smithi114.stdout: "state": { 2024-04-23T10:32:03.446 INFO:teuthology.orchestra.run.smithi114.stdout: "name": "QUIESCING", 2024-04-23T10:32:03.446 INFO:teuthology.orchestra.run.smithi114.stdout: "age": 42.0 2024-04-23T10:32:03.446 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-23T10:32:03.446 INFO:teuthology.orchestra.run.smithi114.stdout: "timeout": 60.0, 2024-04-23T10:32:03.446 INFO:teuthology.orchestra.run.smithi114.stdout: "expiration": 82.6, 2024-04-23T10:32:03.446 INFO:teuthology.orchestra.run.smithi114.stdout: "members": { 2024-04-23T10:32:03.446 INFO:teuthology.orchestra.run.smithi114.stdout: "file:/": { 2024-04-23T10:32:03.446 INFO:teuthology.orchestra.run.smithi114.stdout: "excluded": false, 2024-04-23T10:32:03.446 INFO:teuthology.orchestra.run.smithi114.stdout: "state": { 2024-04-23T10:32:03.446 INFO:teuthology.orchestra.run.smithi114.stdout: "name": "QUIESCING", 2024-04-23T10:32:03.446 INFO:teuthology.orchestra.run.smithi114.stdout: "age": 42.0 2024-04-23T10:32:03.447 INFO:teuthology.orchestra.run.smithi114.stdout: } 2024-04-23T10:32:03.447 INFO:teuthology.orchestra.run.smithi114.stdout: } 2024-04-23T10:32:03.447 INFO:teuthology.orchestra.run.smithi114.stdout: } 2024-04-23T10:32:03.447 INFO:teuthology.orchestra.run.smithi114.stdout: } 2024-04-23T10:32:03.447 INFO:teuthology.orchestra.run.smithi114.stdout: } 2024-04-23T10:32:03.447 INFO:teuthology.orchestra.run.smithi114.stdout:} 2024-04-23T10:32:03.447 DEBUG:teuthology.orchestra.run:got remote process result: 115 2024-04-23T10:32:03.448 INFO:teuthology.orchestra.run.smithi114.stderr:Error EINPROGRESS: 2024-04-23T10:32:03.448 WARNING:tasks.quiescer.fs.[cephfs]:Set '11138c86' hasn't quiesced after 42.5 seconds (timeout: 60). Dumping ops with locks from all ranks. 2024-04-23T10:32:03.449 DEBUG:teuthology.orchestra.run.smithi114:> timeout 75 sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell 'mds.*' ops --flags=locks 2024-04-23T10:32:03.601 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:32:03.602+0000 7f0fe540b640 1 -- 172.21.15.114:0/2866990892 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f0fe0101090 msgr2=0x7f0fe0101470 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T10:32:03.601 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:32:03.602+0000 7f0fe540b640 1 --2- 172.21.15.114:0/2866990892 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f0fe0101090 0x7f0fe0101470 secure :-1 s=READY pgs=680 cs=0 l=1 rev1=1 crypto rx=0x7f0fd0009ca0 tx=0x7f0fd002a8c0 comp rx=0 tx=0).stop 2024-04-23T10:32:03.602 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:32:03.602+0000 7f0fe540b640 1 -- 172.21.15.114:0/2866990892 shutdown_connections 2024-04-23T10:32:03.602 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:32:03.602+0000 7f0fe540b640 1 --2- 172.21.15.114:0/2866990892 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f0fe0106160 0x7f0fe010dd30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:32:03.602 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:32:03.602+0000 7f0fe540b640 1 --2- 172.21.15.114:0/2866990892 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f0fe0101a40 0x7f0fe0105a80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:32:03.602 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:32:03.602+0000 7f0fe540b640 1 --2- 172.21.15.114:0/2866990892 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f0fe0101090 0x7f0fe0101470 unknown :-1 s=CLOSED pgs=680 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:32:03.602 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:32:03.602+0000 7f0fe540b640 1 -- 172.21.15.114:0/2866990892 >> 172.21.15.114:0/2866990892 conn(0x7f0fe0077f50 msgr2=0x7f0fe0078360 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T10:32:03.602 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:32:03.602+0000 7f0fe540b640 1 -- 172.21.15.114:0/2866990892 shutdown_connections 2024-04-23T10:32:03.602 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:32:03.602+0000 7f0fe540b640 1 -- 172.21.15.114:0/2866990892 wait complete. 2024-04-23T10:32:03.604 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:32:03.603+0000 7f0fe540b640 1 Processor -- start 2024-04-23T10:32:03.604 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:32:03.603+0000 7f0fe540b640 1 -- start start 2024-04-23T10:32:03.604 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:32:03.603+0000 7f0fe540b640 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f0fe0101090 0x7f0fe019e3f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T10:32:03.604 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:32:03.603+0000 7f0fe540b640 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f0fe0106160 0x7f0fe019e930 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T10:32:03.605 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:32:03.603+0000 7f0fe540b640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f0fe01985a0 0x7f0fe0198a50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T10:32:03.605 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:32:03.603+0000 7f0fe540b640 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x7f0fe01121e0 con 0x7f0fe01985a0 2024-04-23T10:32:03.605 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:32:03.603+0000 7f0fe540b640 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 -- 0x7f0fe0112060 con 0x7f0fe0106160 2024-04-23T10:32:03.605 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:32:03.604+0000 7f0fe540b640 1 -- --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_getmap magic: 0 -- 0x7f0fe0112360 con 0x7f0fe0101090 2024-04-23T10:32:03.605 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:32:03.604+0000 7f0fdf7fe640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f0fe01985a0 0x7f0fe0198a50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T10:32:03.605 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:32:03.604+0000 7f0fdf7fe640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f0fe01985a0 0x7f0fe0198a50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.114:51654/0 (socket says 172.21.15.114:51654) 2024-04-23T10:32:03.605 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:32:03.604+0000 7f0fdf7fe640 1 -- 172.21.15.114:0/2509710607 learned_addr learned my addr 172.21.15.114:0/2509710607 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-04-23T10:32:03.605 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:32:03.604+0000 7f0fdeffd640 1 --2- 172.21.15.114:0/2509710607 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f0fe0101090 0x7f0fe019e3f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T10:32:03.605 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:32:03.604+0000 7f0fde7fc640 1 --2- 172.21.15.114:0/2509710607 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f0fe0106160 0x7f0fe019e930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T10:32:03.605 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:32:03.604+0000 7f0fdf7fe640 1 -- 172.21.15.114:0/2509710607 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f0fe0101090 msgr2=0x7f0fe019e3f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T10:32:03.605 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:32:03.604+0000 7f0fdf7fe640 1 --2- 172.21.15.114:0/2509710607 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f0fe0101090 0x7f0fe019e3f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:32:03.605 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:32:03.604+0000 7f0fdf7fe640 1 -- 172.21.15.114:0/2509710607 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f0fe0106160 msgr2=0x7f0fe019e930 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T10:32:03.605 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:32:03.604+0000 7f0fdf7fe640 1 --2- 172.21.15.114:0/2509710607 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f0fe0106160 0x7f0fe019e930 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:32:03.605 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:32:03.604+0000 7f0fdf7fe640 1 -- 172.21.15.114:0/2509710607 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f0fe01990c0 con 0x7f0fe01985a0 2024-04-23T10:32:03.605 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:32:03.604+0000 7f0fde7fc640 1 --2- 172.21.15.114:0/2509710607 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f0fe0106160 0x7f0fe019e930 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-23T10:32:03.606 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:32:03.604+0000 7f0fdf7fe640 1 --2- 172.21.15.114:0/2509710607 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f0fe01985a0 0x7f0fe0198a50 secure :-1 s=READY pgs=681 cs=0 l=1 rev1=1 crypto rx=0x7f0fd800d750 tx=0x7f0fd800dc10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T10:32:03.606 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:32:03.604+0000 7f0fc7fff640 1 -- 172.21.15.114:0/2509710607 <== mon.0 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0fd8017070 con 0x7f0fe01985a0 2024-04-23T10:32:03.606 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:32:03.604+0000 7f0fe540b640 1 -- 172.21.15.114:0/2509710607 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f0fe01993b0 con 0x7f0fe01985a0 2024-04-23T10:32:03.606 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:32:03.604+0000 7f0fc7fff640 1 -- 172.21.15.114:0/2509710607 <== mon.0 v2:172.21.15.114:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f0fd80054d0 con 0x7f0fe01985a0 2024-04-23T10:32:03.606 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:32:03.604+0000 7f0fc7fff640 1 -- 172.21.15.114:0/2509710607 <== mon.0 v2:172.21.15.114:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0fd8004c60 con 0x7f0fe01985a0 2024-04-23T10:32:03.606 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:32:03.604+0000 7f0fe540b640 1 -- 172.21.15.114:0/2509710607 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f0fe01a5600 con 0x7f0fe01985a0 2024-04-23T10:32:03.606 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:32:03.605+0000 7f0fe540b640 1 -- 172.21.15.114:0/2509710607 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f0fe0068d70 con 0x7f0fe01985a0 2024-04-23T10:32:03.607 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:32:03.606+0000 7f0fc7fff640 1 -- 172.21.15.114:0/2509710607 <== mon.0 v2:172.21.15.114:3300/0 4 ==== mgrmap(e 26) ==== 138778+0+0 (secure 0 0 0) 0x7f0fd8009bd0 con 0x7f0fe01985a0 2024-04-23T10:32:03.607 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:32:03.607+0000 7f0fc7fff640 1 --2- 172.21.15.114:0/2509710607 >> [v2:172.21.15.114:6800/1395693466,v1:172.21.15.114:6801/1395693466] conn(0x7f0fc80a3d20 0x7f0fc80a61e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T10:32:03.607 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:32:03.607+0000 7f0fdeffd640 1 --2- 172.21.15.114:0/2509710607 >> [v2:172.21.15.114:6800/1395693466,v1:172.21.15.114:6801/1395693466] conn(0x7f0fc80a3d20 0x7f0fc80a61e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T10:32:03.607 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:32:03.607+0000 7f0fc7fff640 1 -- 172.21.15.114:0/2509710607 <== mon.0 v2:172.21.15.114:3300/0 5 ==== osd_map(94..94 src has 1..94) ==== 8585+0+0 (secure 0 0 0) 0x7f0fd80d05c0 con 0x7f0fe01985a0 2024-04-23T10:32:03.607 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:32:03.607+0000 7f0fc7fff640 1 -- 172.21.15.114:0/2509710607 <== mon.0 v2:172.21.15.114:3300/0 6 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 30 v30) ==== 94+0+11884 (secure 0 0 0) 0x7f0fd80d0af0 con 0x7f0fe01985a0 2024-04-23T10:32:03.607 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:32:03.607+0000 7f0fdeffd640 1 --2- 172.21.15.114:0/2509710607 >> [v2:172.21.15.114:6800/1395693466,v1:172.21.15.114:6801/1395693466] conn(0x7f0fc80a3d20 0x7f0fc80a61e0 secure :-1 s=READY pgs=106 cs=0 l=1 rev1=1 crypto rx=0x7f0fd0000f80 tx=0x7f0fd0005ef0 comp rx=0 tx=0).ready entity=mgr.15291 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T10:32:03.609 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:32:03.609+0000 7f0fe81f4740 1 Processor -- start 2024-04-23T10:32:03.609 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:32:03.609+0000 7f0fe81f4740 1 -- start start 2024-04-23T10:32:03.609 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:32:03.609+0000 7f0fe81f4740 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55c8d0ee0ba0 0x55c8d0ee0f80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T10:32:03.609 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:32:03.609+0000 7f0fe81f4740 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x55c8d0ee14c0 0x55c8d0eefb60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T10:32:03.610 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:32:03.609+0000 7f0fe81f4740 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x55c8d0ef00a0 0x55c8d0ef2490 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T10:32:03.610 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:32:03.609+0000 7f0fe81f4740 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x55c8d0ea9540 con 0x55c8d0ee0ba0 2024-04-23T10:32:03.610 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:32:03.609+0000 7f0fe81f4740 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 -- 0x55c8d0e46ec0 con 0x55c8d0ee14c0 2024-04-23T10:32:03.610 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:32:03.609+0000 7f0fe81f4740 1 -- --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_getmap magic: 0 -- 0x55c8d0de3a90 con 0x55c8d0ef00a0 2024-04-23T10:32:03.610 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:32:03.610+0000 7f0fdeffd640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55c8d0ee0ba0 0x55c8d0ee0f80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T10:32:03.610 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:32:03.610+0000 7f0fdeffd640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55c8d0ee0ba0 0x55c8d0ee0f80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.114:51666/0 (socket says 172.21.15.114:51666) 2024-04-23T10:32:03.610 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:32:03.610+0000 7f0fdeffd640 1 -- 172.21.15.114:0/2349391170 learned_addr learned my addr 172.21.15.114:0/2349391170 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-04-23T10:32:03.610 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:32:03.610+0000 7f0fdf7fe640 1 --2- 172.21.15.114:0/2349391170 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x55c8d0ef00a0 0x55c8d0ef2490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T10:32:03.610 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:32:03.610+0000 7f0fde7fc640 1 --2- 172.21.15.114:0/2349391170 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x55c8d0ee14c0 0x55c8d0eefb60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T10:32:03.610 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:32:03.610+0000 7f0fdeffd640 1 -- 172.21.15.114:0/2349391170 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x55c8d0ef00a0 msgr2=0x55c8d0ef2490 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T10:32:03.610 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:32:03.610+0000 7f0fdeffd640 1 --2- 172.21.15.114:0/2349391170 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x55c8d0ef00a0 0x55c8d0ef2490 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:32:03.610 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:32:03.610+0000 7f0fdeffd640 1 -- 172.21.15.114:0/2349391170 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x55c8d0ee14c0 msgr2=0x55c8d0eefb60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T10:32:03.610 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:32:03.610+0000 7f0fdeffd640 1 --2- 172.21.15.114:0/2349391170 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x55c8d0ee14c0 0x55c8d0eefb60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:32:03.611 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:32:03.610+0000 7f0fdeffd640 1 -- 172.21.15.114:0/2349391170 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55c8d0dffec0 con 0x55c8d0ee0ba0 2024-04-23T10:32:03.611 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:32:03.610+0000 7f0fdeffd640 1 --2- 172.21.15.114:0/2349391170 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55c8d0ee0ba0 0x55c8d0ee0f80 secure :-1 s=READY pgs=682 cs=0 l=1 rev1=1 crypto rx=0x7f0fd0031d80 tx=0x7f0fd003e000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=df7ce98d85a5335 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T10:32:03.611 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:32:03.610+0000 7f0fc57fa640 1 -- 172.21.15.114:0/2349391170 <== mon.0 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0fd0003b70 con 0x55c8d0ee0ba0 2024-04-23T10:32:03.611 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:32:03.610+0000 7f0fc57fa640 1 -- 172.21.15.114:0/2349391170 <== mon.0 v2:172.21.15.114:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f0fd0036040 con 0x55c8d0ee0ba0 2024-04-23T10:32:03.611 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:32:03.610+0000 7f0fc57fa640 1 -- 172.21.15.114:0/2349391170 <== mon.0 v2:172.21.15.114:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0fd002f8c0 con 0x55c8d0ee0ba0 2024-04-23T10:32:03.615 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:32:03.611+0000 7f0fe81f4740 1 -- 172.21.15.114:0/2349391170 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55c8d0ee0ba0 msgr2=0x55c8d0ee0f80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T10:32:03.615 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:32:03.611+0000 7f0fe81f4740 1 --2- 172.21.15.114:0/2349391170 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55c8d0ee0ba0 0x55c8d0ee0f80 secure :-1 s=READY pgs=682 cs=0 l=1 rev1=1 crypto rx=0x7f0fd0031d80 tx=0x7f0fd003e000 comp rx=0 tx=0).stop 2024-04-23T10:32:03.615 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:32:03.611+0000 7f0fe81f4740 1 -- 172.21.15.114:0/2349391170 shutdown_connections 2024-04-23T10:32:03.615 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:32:03.611+0000 7f0fe81f4740 1 --2- 172.21.15.114:0/2349391170 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x55c8d0ef00a0 0x55c8d0ef2490 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:32:03.615 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:32:03.611+0000 7f0fe81f4740 1 --2- 172.21.15.114:0/2349391170 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x55c8d0ee14c0 0x55c8d0eefb60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:32:03.615 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:32:03.611+0000 7f0fe81f4740 1 --2- 172.21.15.114:0/2349391170 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55c8d0ee0ba0 0x55c8d0ee0f80 unknown :-1 s=CLOSED pgs=682 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:32:03.616 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:32:03.611+0000 7f0fe81f4740 1 -- 172.21.15.114:0/2349391170 >> 172.21.15.114:0/2349391170 conn(0x55c8d0edcfc0 msgr2=0x55c8d0edd3a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T10:32:03.616 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:32:03.611+0000 7f0fe81f4740 1 -- 172.21.15.114:0/2349391170 shutdown_connections 2024-04-23T10:32:03.616 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:32:03.611+0000 7f0fe81f4740 1 -- 172.21.15.114:0/2349391170 wait complete. 2024-04-23T10:32:03.616 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:32:03.611+0000 7f0fe81f4740 1 -- start start 2024-04-23T10:32:03.616 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:32:03.611+0000 7f0fe81f4740 1 Processor -- start 2024-04-23T10:32:03.616 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:32:03.612+0000 7f0fe81f4740 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x55c8d0ee0ba0 0x55c8d0eed0d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T10:32:03.616 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:32:03.612+0000 7f0fe81f4740 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55c8d0ee14c0 0x55c8d0eed610 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T10:32:03.616 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:32:03.612+0000 7f0fe81f4740 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x55c8d0ef00a0 0x55c8d0eedb50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T10:32:03.616 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:32:03.612+0000 7f0fe81f4740 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x55c8d0de1740 con 0x55c8d0ee14c0 2024-04-23T10:32:03.616 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:32:03.612+0000 7f0fe81f4740 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 -- 0x55c8d0e42e80 con 0x55c8d0ee0ba0 2024-04-23T10:32:03.616 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:32:03.612+0000 7f0fe81f4740 1 -- --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_getmap magic: 0 -- 0x55c8d0ef2e00 con 0x55c8d0ef00a0 2024-04-23T10:32:03.616 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:32:03.612+0000 7f0fe81f4740 10 client.0 _ll_register_callbacks cb 0x55c8d0e53c70 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-23T10:32:03.616 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:32:03.612+0000 7f0fde7fc640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55c8d0ee14c0 0x55c8d0eed610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T10:32:03.616 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:32:03.612+0000 7f0fde7fc640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55c8d0ee14c0 0x55c8d0eed610 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.114:51682/0 (socket says 172.21.15.114:51682) 2024-04-23T10:32:03.616 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:32:03.612+0000 7f0fde7fc640 1 -- 172.21.15.114:0/2243778097 learned_addr learned my addr 172.21.15.114:0/2243778097 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-04-23T10:32:03.617 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:32:03.612+0000 7f0fdeffd640 1 --2- 172.21.15.114:0/2243778097 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x55c8d0ee0ba0 0x55c8d0eed0d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T10:32:03.617 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:32:03.612+0000 7f0fdf7fe640 1 --2- 172.21.15.114:0/2243778097 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x55c8d0ef00a0 0x55c8d0eedb50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T10:32:03.617 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:32:03.612+0000 7f0fde7fc640 1 -- 172.21.15.114:0/2243778097 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x55c8d0ef00a0 msgr2=0x55c8d0eedb50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T10:32:03.617 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:32:03.612+0000 7f0fde7fc640 1 --2- 172.21.15.114:0/2243778097 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x55c8d0ef00a0 0x55c8d0eedb50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:32:03.617 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:32:03.612+0000 7f0fde7fc640 1 -- 172.21.15.114:0/2243778097 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x55c8d0ee0ba0 msgr2=0x55c8d0eed0d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T10:32:03.617 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:32:03.612+0000 7f0fde7fc640 1 --2- 172.21.15.114:0/2243778097 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x55c8d0ee0ba0 0x55c8d0eed0d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:32:03.617 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:32:03.612+0000 7f0fde7fc640 1 -- 172.21.15.114:0/2243778097 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55c8d0c237d0 con 0x55c8d0ee14c0 2024-04-23T10:32:03.617 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:32:03.612+0000 7f0fde7fc640 1 --2- 172.21.15.114:0/2243778097 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55c8d0ee14c0 0x55c8d0eed610 secure :-1 s=READY pgs=683 cs=0 l=1 rev1=1 crypto rx=0x7f0fe006b6e0 tx=0x7f0fd40117b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a4ee366d12b838e0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T10:32:03.617 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:32:03.612+0000 7f0fc2ff5640 10 client.0 ms_handle_connect on v2:172.21.15.114:3300/0 2024-04-23T10:32:03.617 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:32:03.612+0000 7f0fc2ff5640 1 -- 172.21.15.114:0/2243778097 <== mon.0 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0fd4004220 con 0x55c8d0ee14c0 2024-04-23T10:32:03.617 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:32:03.612+0000 7f0fc2ff5640 1 -- 172.21.15.114:0/2243778097 <== mon.0 v2:172.21.15.114:3300/0 2 ==== osd_map(94..94 src has 1..94) ==== 8585+0+0 (secure 0 0 0) 0x7f0fd40043c0 con 0x55c8d0ee14c0 2024-04-23T10:32:03.617 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:32:03.612+0000 7f0fe81f4740 1 -- 172.21.15.114:0/2243778097 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f0fd0036040 con 0x55c8d0ee14c0 2024-04-23T10:32:03.617 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:32:03.612+0000 7f0fc2ff5640 1 -- 172.21.15.114:0/2243778097 <== mon.0 v2:172.21.15.114:3300/0 3 ==== mon_get_version_reply(handle=1 version=30) ==== 24+0+0 (secure 0 0 0) 0x7f0fd4012660 con 0x55c8d0ee14c0 2024-04-23T10:32:03.617 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:32:03.612+0000 7f0fe81f4740 10 client.16230 fetch_fsmap learned FSMap version 30 2024-04-23T10:32:03.617 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:32:03.613+0000 7f0fe81f4740 1 -- 172.21.15.114:0/2243778097 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({config=0+,fsmap=30,monmap=4+}) -- 0x55c8d0c83f60 con 0x55c8d0ee14c0 2024-04-23T10:32:03.618 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:32:03.613+0000 7f0fc2ff5640 1 -- 172.21.15.114:0/2243778097 <== mon.0 v2:172.21.15.114:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f0fd0036040 con 0x55c8d0ee14c0 2024-04-23T10:32:03.618 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:32:03.613+0000 7f0fc2ff5640 1 -- 172.21.15.114:0/2243778097 <== mon.0 v2:172.21.15.114:3300/0 5 ==== fsmap(e 30) ==== 8338+0+0 (secure 0 0 0) 0x7f0fd4020030 con 0x55c8d0ee14c0 2024-04-23T10:32:03.618 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:32:03.613+0000 7f0fe81f4740 10 client.16230 fetch_fsmap finished waiting for FSMap version 30 2024-04-23T10:32:03.618 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:32:03.613+0000 7f0fe81f4740 10 client.16230 resolve_mds: resolved name 'j' to daemon mds.j 2024-04-23T10:32:03.618 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:32:03.613+0000 7f0fe81f4740 20 client.16230 populate_metadata read hostname 'smithi114' 2024-04-23T10:32:03.618 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:32:03.613+0000 7f0fe81f4740 1 --2- 172.21.15.114:0/2243778097 >> [v2:172.21.15.114:6834/663493580,v1:172.21.15.114:6835/663493580] conn(0x55c8d0f01520 0x55c8d0f02990 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T10:32:03.618 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:32:03.613+0000 7f0fe81f4740 4 client.16230 mds_command: new command op to 14547 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-23T10:32:03.618 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:32:03.613+0000 7f0fe81f4740 1 -- 172.21.15.114:0/2243778097 --> [v2:172.21.15.114:6834/663493580,v1:172.21.15.114:6835/663493580] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55c8d0df5530 con 0x55c8d0f01520 2024-04-23T10:32:03.618 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:32:03.613+0000 7f0fdeffd640 1 --2- 172.21.15.114:0/2243778097 >> [v2:172.21.15.114:6834/663493580,v1:172.21.15.114:6835/663493580] conn(0x55c8d0f01520 0x55c8d0f02990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T10:32:03.618 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:32:03.613+0000 7f0fdeffd640 1 --2- 172.21.15.114:0/2243778097 >> [v2:172.21.15.114:6834/663493580,v1:172.21.15.114:6835/663493580] conn(0x55c8d0f01520 0x55c8d0f02990 crc :-1 s=READY pgs=13 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.? client_cookie=543013986fee1089 server_cookie=a3ca4c795d9ebb2c in_seq=0 out_seq=0 2024-04-23T10:32:03.618 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:32:03.613+0000 7f0fc2ff5640 10 client.16230 ms_handle_connect on v2:172.21.15.114:6834/663493580 2024-04-23T10:32:03.618 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:32:03.614+0000 7f0fc2ff5640 1 -- 172.21.15.114:0/2243778097 <== mds.? v2:172.21.15.114:6834/663493580 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23115 (crc 0 0 0) 0x55c8d0df5530 con 0x55c8d0f01520 2024-04-23T10:32:03.618 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:32:03.614+0000 7f0fc2ff5640 10 client.16230 handle_command_reply: tid=0 2024-04-23T10:32:03.618 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:32:03.614+0000 7f0fe81f4740 1 client.16230 shutdown 2024-04-23T10:32:03.619 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:32:03.614+0000 7f0fe81f4740 1 -- 172.21.15.114:0/2243778097 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55c8d0ee14c0 msgr2=0x55c8d0eed610 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T10:32:03.619 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:32:03.614+0000 7f0fe81f4740 1 --2- 172.21.15.114:0/2243778097 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55c8d0ee14c0 0x55c8d0eed610 secure :-1 s=READY pgs=683 cs=0 l=1 rev1=1 crypto rx=0x7f0fe006b6e0 tx=0x7f0fd40117b0 comp rx=0 tx=0).stop 2024-04-23T10:32:03.619 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:32:03.614+0000 7f0fe81f4740 1 -- 172.21.15.114:0/2243778097 shutdown_connections 2024-04-23T10:32:03.619 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:32:03.614+0000 7f0fe81f4740 1 --2- 172.21.15.114:0/2243778097 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x55c8d0ef00a0 0x55c8d0eedb50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:32:03.619 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:32:03.614+0000 7f0fe81f4740 1 --2- 172.21.15.114:0/2243778097 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55c8d0ee14c0 0x55c8d0eed610 unknown :-1 s=CLOSED pgs=683 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:32:03.619 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:32:03.614+0000 7f0fe81f4740 1 --2- 172.21.15.114:0/2243778097 >> [v2:172.21.15.114:6834/663493580,v1:172.21.15.114:6835/663493580] conn(0x55c8d0f01520 0x55c8d0f02990 crc :-1 s=READY pgs=13 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:32:03.619 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:32:03.614+0000 7f0fe81f4740 1 --2- 172.21.15.114:0/2243778097 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x55c8d0ee0ba0 0x55c8d0eed0d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:32:03.619 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:32:03.614+0000 7f0fe81f4740 1 -- 172.21.15.114:0/2243778097 >> 172.21.15.114:0/2243778097 conn(0x55c8d0edcfc0 msgr2=0x55c8d0ed8d00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T10:32:03.619 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:32:03.614+0000 7f0fc2ff5640 0 client.16230 ms_handle_reset on v2:172.21.15.114:6834/663493580 2024-04-23T10:32:03.619 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:32:03.614+0000 7f0fe81f4740 1 -- 172.21.15.114:0/2243778097 shutdown_connections 2024-04-23T10:32:03.619 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:32:03.614+0000 7f0fe81f4740 1 -- 172.21.15.114:0/2243778097 wait complete. 2024-04-23T10:32:03.619 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:32:03.614+0000 7f0fe81f4740 20 client.16230 trim_cache size 0 max 16384 2024-04-23T10:32:03.640 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:32:03.630+0000 7f0fe81f4740 1 Processor -- start 2024-04-23T10:32:03.640 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:32:03.630+0000 7f0fe81f4740 1 -- start start 2024-04-23T10:32:03.640 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:32:03.630+0000 7f0fe81f4740 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x55c8d0ee0ba0 0x55c8d0eef910 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T10:32:03.640 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:32:03.630+0000 7f0fe81f4740 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55c8d0ee14c0 0x55c8d0eed970 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T10:32:03.640 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:32:03.630+0000 7f0fe81f4740 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x55c8d0ef00a0 0x55c8d0f0ac70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T10:32:03.641 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:32:03.630+0000 7f0fe81f4740 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x55c8d0f00cd0 con 0x55c8d0ee14c0 2024-04-23T10:32:03.641 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:32:03.630+0000 7f0fe81f4740 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 -- 0x55c8d0f013a0 con 0x55c8d0ef00a0 2024-04-23T10:32:03.641 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:32:03.630+0000 7f0fe81f4740 1 -- --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_getmap magic: 0 -- 0x55c8d0f01a70 con 0x55c8d0ee0ba0 2024-04-23T10:32:03.641 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:32:03.630+0000 7f0fde7fc640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55c8d0ee14c0 0x55c8d0eed970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T10:32:03.641 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:32:03.630+0000 7f0fde7fc640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55c8d0ee14c0 0x55c8d0eed970 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.114:51690/0 (socket says 172.21.15.114:51690) 2024-04-23T10:32:03.641 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:32:03.630+0000 7f0fde7fc640 1 -- 172.21.15.114:0/2924422396 learned_addr learned my addr 172.21.15.114:0/2924422396 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-04-23T10:32:03.641 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:32:03.630+0000 7f0fdeffd640 1 --2- 172.21.15.114:0/2924422396 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x55c8d0ee0ba0 0x55c8d0eef910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T10:32:03.641 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:32:03.630+0000 7f0fde7fc640 1 -- 172.21.15.114:0/2924422396 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x55c8d0ee0ba0 msgr2=0x55c8d0eef910 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T10:32:03.641 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:32:03.630+0000 7f0fdf7fe640 1 --2- 172.21.15.114:0/2924422396 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x55c8d0ef00a0 0x55c8d0f0ac70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T10:32:03.641 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:32:03.630+0000 7f0fde7fc640 1 --2- 172.21.15.114:0/2924422396 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x55c8d0ee0ba0 0x55c8d0eef910 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:32:03.641 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:32:03.631+0000 7f0fde7fc640 1 -- 172.21.15.114:0/2924422396 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x55c8d0ef00a0 msgr2=0x55c8d0f0ac70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T10:32:03.641 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:32:03.631+0000 7f0fde7fc640 1 --2- 172.21.15.114:0/2924422396 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x55c8d0ef00a0 0x55c8d0f0ac70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:32:03.641 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:32:03.631+0000 7f0fde7fc640 1 -- 172.21.15.114:0/2924422396 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55c8d0b190f0 con 0x55c8d0ee14c0 2024-04-23T10:32:03.641 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:32:03.631+0000 7f0fdeffd640 1 --2- 172.21.15.114:0/2924422396 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x55c8d0ee0ba0 0x55c8d0eef910 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-23T10:32:03.642 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:32:03.631+0000 7f0fdf7fe640 1 --2- 172.21.15.114:0/2924422396 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x55c8d0ef00a0 0x55c8d0f0ac70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-23T10:32:03.642 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:32:03.631+0000 7f0fde7fc640 1 --2- 172.21.15.114:0/2924422396 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55c8d0ee14c0 0x55c8d0eed970 secure :-1 s=READY pgs=684 cs=0 l=1 rev1=1 crypto rx=0x7f0fd4001eb0 tx=0x7f0fd40058f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7b39b33f25691580 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T10:32:03.642 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:32:03.631+0000 7f0fc4ff9640 1 -- 172.21.15.114:0/2924422396 <== mon.0 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0fd402f070 con 0x55c8d0ee14c0 2024-04-23T10:32:03.642 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:32:03.631+0000 7f0fc4ff9640 1 -- 172.21.15.114:0/2924422396 <== mon.0 v2:172.21.15.114:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f0fd4011cb0 con 0x55c8d0ee14c0 2024-04-23T10:32:03.642 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:32:03.631+0000 7f0fc4ff9640 1 -- 172.21.15.114:0/2924422396 <== mon.0 v2:172.21.15.114:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0fd4003fe0 con 0x55c8d0ee14c0 2024-04-23T10:32:03.642 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:32:03.631+0000 7f0fe81f4740 1 -- 172.21.15.114:0/2924422396 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55c8d0ee14c0 msgr2=0x55c8d0eed970 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T10:32:03.642 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:32:03.631+0000 7f0fe81f4740 1 --2- 172.21.15.114:0/2924422396 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55c8d0ee14c0 0x55c8d0eed970 secure :-1 s=READY pgs=684 cs=0 l=1 rev1=1 crypto rx=0x7f0fd4001eb0 tx=0x7f0fd40058f0 comp rx=0 tx=0).stop 2024-04-23T10:32:03.642 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:32:03.631+0000 7f0fe81f4740 1 -- 172.21.15.114:0/2924422396 shutdown_connections 2024-04-23T10:32:03.642 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:32:03.631+0000 7f0fe81f4740 1 --2- 172.21.15.114:0/2924422396 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x55c8d0ef00a0 0x55c8d0f0ac70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:32:03.642 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:32:03.631+0000 7f0fe81f4740 1 --2- 172.21.15.114:0/2924422396 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55c8d0ee14c0 0x55c8d0eed970 unknown :-1 s=CLOSED pgs=684 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:32:03.642 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:32:03.631+0000 7f0fe81f4740 1 --2- 172.21.15.114:0/2924422396 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x55c8d0ee0ba0 0x55c8d0eef910 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:32:03.642 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:32:03.631+0000 7f0fe81f4740 1 -- 172.21.15.114:0/2924422396 >> 172.21.15.114:0/2924422396 conn(0x55c8d0edcfc0 msgr2=0x55c8d0ee8460 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T10:32:03.642 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:32:03.631+0000 7f0fe81f4740 1 -- 172.21.15.114:0/2924422396 shutdown_connections 2024-04-23T10:32:03.642 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:32:03.632+0000 7f0fe81f4740 1 -- 172.21.15.114:0/2924422396 wait complete. 2024-04-23T10:32:03.643 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:32:03.632+0000 7f0fe81f4740 1 -- start start 2024-04-23T10:32:03.643 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:32:03.632+0000 7f0fe81f4740 1 Processor -- start 2024-04-23T10:32:03.643 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:32:03.632+0000 7f0fe81f4740 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55c8d0ee0ba0 0x55c8d0eea4e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T10:32:03.643 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:32:03.632+0000 7f0fe81f4740 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x55c8d0ee14c0 0x55c8d0eeaa20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T10:32:03.643 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:32:03.632+0000 7f0fe81f4740 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x55c8d0ef00a0 0x55c8d0f092a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T10:32:03.643 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:32:03.632+0000 7f0fe81f4740 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x55c8d0f02810 con 0x55c8d0ee0ba0 2024-04-23T10:32:03.643 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:32:03.632+0000 7f0fe81f4740 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 -- 0x55c8d0f02140 con 0x55c8d0ef00a0 2024-04-23T10:32:03.643 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:32:03.632+0000 7f0fe81f4740 1 -- --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_getmap magic: 0 -- 0x55c8d0f02ee0 con 0x55c8d0ee14c0 2024-04-23T10:32:03.643 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:32:03.632+0000 7f0fe81f4740 10 client.0 _ll_register_callbacks cb 0x7f0fd4009800 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-23T10:32:03.643 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:32:03.632+0000 7f0fdeffd640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55c8d0ee0ba0 0x55c8d0eea4e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T10:32:03.643 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:32:03.632+0000 7f0fdeffd640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55c8d0ee0ba0 0x55c8d0eea4e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.114:51706/0 (socket says 172.21.15.114:51706) 2024-04-23T10:32:03.643 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:32:03.632+0000 7f0fdeffd640 1 -- 172.21.15.114:0/2642090197 learned_addr learned my addr 172.21.15.114:0/2642090197 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-04-23T10:32:03.643 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:32:03.632+0000 7f0fde7fc640 1 --2- 172.21.15.114:0/2642090197 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x55c8d0ee14c0 0x55c8d0eeaa20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T10:32:03.643 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:32:03.632+0000 7f0fdeffd640 1 -- 172.21.15.114:0/2642090197 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x55c8d0ee14c0 msgr2=0x55c8d0eeaa20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T10:32:03.644 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:32:03.632+0000 7f0fdf7fe640 1 --2- 172.21.15.114:0/2642090197 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x55c8d0ef00a0 0x55c8d0f092a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T10:32:03.644 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:32:03.632+0000 7f0fdeffd640 1 --2- 172.21.15.114:0/2642090197 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x55c8d0ee14c0 0x55c8d0eeaa20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:32:03.644 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:32:03.632+0000 7f0fdeffd640 1 -- 172.21.15.114:0/2642090197 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x55c8d0ef00a0 msgr2=0x55c8d0f092a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T10:32:03.644 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:32:03.632+0000 7f0fdeffd640 1 --2- 172.21.15.114:0/2642090197 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x55c8d0ef00a0 0x55c8d0f092a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:32:03.644 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:32:03.632+0000 7f0fdeffd640 1 -- 172.21.15.114:0/2642090197 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55c8d0f09810 con 0x55c8d0ee0ba0 2024-04-23T10:32:03.644 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:32:03.633+0000 7f0fdeffd640 1 --2- 172.21.15.114:0/2642090197 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55c8d0ee0ba0 0x55c8d0eea4e0 secure :-1 s=READY pgs=685 cs=0 l=1 rev1=1 crypto rx=0x7f0fd0050000 tx=0x7f0fd0009970 comp rx=0 tx=0).ready entity=mon.0 client_cookie=41eee21dccc3bf41 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T10:32:03.644 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:32:03.633+0000 7f0fc2ff5640 10 client.0 ms_handle_connect on v2:172.21.15.114:3300/0 2024-04-23T10:32:03.644 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:32:03.633+0000 7f0fc2ff5640 1 -- 172.21.15.114:0/2642090197 <== mon.0 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0fd0003e00 con 0x55c8d0ee0ba0 2024-04-23T10:32:03.644 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:32:03.633+0000 7f0fc2ff5640 1 -- 172.21.15.114:0/2642090197 <== mon.0 v2:172.21.15.114:3300/0 2 ==== osd_map(94..94 src has 1..94) ==== 8585+0+0 (secure 0 0 0) 0x7f0fd0004040 con 0x55c8d0ee0ba0 2024-04-23T10:32:03.644 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:32:03.633+0000 7f0fe81f4740 1 -- 172.21.15.114:0/2642090197 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f0fd4011cb0 con 0x55c8d0ee0ba0 2024-04-23T10:32:03.644 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:32:03.633+0000 7f0fc2ff5640 1 -- 172.21.15.114:0/2642090197 <== mon.0 v2:172.21.15.114:3300/0 3 ==== mon_get_version_reply(handle=1 version=30) ==== 24+0+0 (secure 0 0 0) 0x7f0fd0004240 con 0x55c8d0ee0ba0 2024-04-23T10:32:03.644 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:32:03.633+0000 7f0fe81f4740 10 client.16236 fetch_fsmap learned FSMap version 30 2024-04-23T10:32:03.644 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:32:03.633+0000 7f0fe81f4740 1 -- 172.21.15.114:0/2642090197 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({config=0+,fsmap=30,monmap=4+}) -- 0x55c8d0f67bb0 con 0x55c8d0ee0ba0 2024-04-23T10:32:03.644 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:32:03.633+0000 7f0fc2ff5640 1 -- 172.21.15.114:0/2642090197 <== mon.0 v2:172.21.15.114:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f0fd4011cb0 con 0x55c8d0ee0ba0 2024-04-23T10:32:03.645 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:32:03.633+0000 7f0fc2ff5640 1 -- 172.21.15.114:0/2642090197 <== mon.0 v2:172.21.15.114:3300/0 5 ==== fsmap(e 30) ==== 8338+0+0 (secure 0 0 0) 0x7f0fd0066030 con 0x55c8d0ee0ba0 2024-04-23T10:32:03.645 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:32:03.633+0000 7f0fe81f4740 10 client.16236 fetch_fsmap finished waiting for FSMap version 30 2024-04-23T10:32:03.645 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:32:03.633+0000 7f0fe81f4740 10 client.16236 resolve_mds: resolved name 'j' to daemon mds.j 2024-04-23T10:32:03.645 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:32:03.633+0000 7f0fe81f4740 20 client.16236 populate_metadata read hostname 'smithi114' 2024-04-23T10:32:03.645 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:32:03.633+0000 7f0fe81f4740 1 --2- 172.21.15.114:0/2642090197 >> [v2:172.21.15.114:6834/663493580,v1:172.21.15.114:6835/663493580] conn(0x55c8d0f6c390 0x55c8d0f6b0a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T10:32:03.645 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:32:03.633+0000 7f0fe81f4740 4 client.16236 mds_command: new command op to 14547 tid=0 multi_id=0 [{"prefix": "ops", "flags": ["locks"]}] 2024-04-23T10:32:03.645 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:32:03.633+0000 7f0fe81f4740 1 -- 172.21.15.114:0/2642090197 --> [v2:172.21.15.114:6834/663493580,v1:172.21.15.114:6835/663493580] -- command(tid 0: {"prefix": "ops", "flags": ["locks"]}) -- 0x55c8d0e548e0 con 0x55c8d0f6c390 2024-04-23T10:32:03.645 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:32:03.633+0000 7f0fde7fc640 1 --2- 172.21.15.114:0/2642090197 >> [v2:172.21.15.114:6834/663493580,v1:172.21.15.114:6835/663493580] conn(0x55c8d0f6c390 0x55c8d0f6b0a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T10:32:03.645 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:32:03.634+0000 7f0fc2ff5640 10 client.16236 ms_handle_connect on v2:172.21.15.114:6834/663493580 2024-04-23T10:32:03.645 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:32:03.634+0000 7f0fde7fc640 1 --2- 172.21.15.114:0/2642090197 >> [v2:172.21.15.114:6834/663493580,v1:172.21.15.114:6835/663493580] conn(0x55c8d0f6c390 0x55c8d0f6b0a0 crc :-1 s=READY pgs=14 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.? client_cookie=b9a36443458c519f server_cookie=8afc93b5ec9b2eb8 in_seq=0 out_seq=0 2024-04-23T10:32:03.645 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:32:03.634+0000 7f0fc2ff5640 1 -- 172.21.15.114:0/2642090197 <== mds.? v2:172.21.15.114:6834/663493580 1 ==== command_reply(tid 0: -38 ) ==== 8+0+0 (crc 0 0 0) 0x55c8d0e548e0 con 0x55c8d0f6c390 2024-04-23T10:32:03.645 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:32:03.634+0000 7f0fc2ff5640 10 client.16236 handle_command_reply: tid=0 2024-04-23T10:32:03.645 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:32:03.634+0000 7f0fe81f4740 1 client.16236 shutdown 2024-04-23T10:32:03.645 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:32:03.634+0000 7f0fe81f4740 1 -- 172.21.15.114:0/2642090197 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55c8d0ee0ba0 msgr2=0x55c8d0eea4e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T10:32:03.646 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:32:03.634+0000 7f0fe81f4740 1 --2- 172.21.15.114:0/2642090197 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55c8d0ee0ba0 0x55c8d0eea4e0 secure :-1 s=READY pgs=685 cs=0 l=1 rev1=1 crypto rx=0x7f0fd0050000 tx=0x7f0fd0009970 comp rx=0 tx=0).stop 2024-04-23T10:32:03.646 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:32:03.634+0000 7f0fe81f4740 1 -- 172.21.15.114:0/2642090197 shutdown_connections 2024-04-23T10:32:03.646 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:32:03.634+0000 7f0fe81f4740 1 --2- 172.21.15.114:0/2642090197 >> [v2:172.21.15.114:6834/663493580,v1:172.21.15.114:6835/663493580] conn(0x55c8d0f6c390 0x55c8d0f6b0a0 crc :-1 s=READY pgs=14 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:32:03.646 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:32:03.634+0000 7f0fe81f4740 1 --2- 172.21.15.114:0/2642090197 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x55c8d0ef00a0 0x55c8d0f092a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:32:03.646 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:32:03.634+0000 7f0fe81f4740 1 --2- 172.21.15.114:0/2642090197 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x55c8d0ee14c0 0x55c8d0eeaa20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:32:03.646 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:32:03.634+0000 7f0fe81f4740 1 --2- 172.21.15.114:0/2642090197 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55c8d0ee0ba0 0x55c8d0eea4e0 unknown :-1 s=CLOSED pgs=685 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:32:03.646 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:32:03.634+0000 7f0fc2ff5640 0 client.16236 ms_handle_reset on v2:172.21.15.114:6834/663493580 2024-04-23T10:32:03.646 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:32:03.634+0000 7f0fe81f4740 1 -- 172.21.15.114:0/2642090197 >> 172.21.15.114:0/2642090197 conn(0x55c8d0edcfc0 msgr2=0x55c8d0eeb900 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T10:32:03.646 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:32:03.634+0000 7f0fe81f4740 1 -- 172.21.15.114:0/2642090197 shutdown_connections 2024-04-23T10:32:03.646 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:32:03.634+0000 7f0fe81f4740 1 -- 172.21.15.114:0/2642090197 wait complete. 2024-04-23T10:32:03.646 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:32:03.634+0000 7f0fe81f4740 20 client.16236 trim_cache size 0 max 16384 2024-04-23T10:32:03.646 INFO:teuthology.orchestra.run.smithi114.stderr:Error ENOSYS: 2024-04-23T10:32:03.646 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:32:03.635+0000 7f0fe81f4740 1 Processor -- start 2024-04-23T10:32:03.646 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:32:03.635+0000 7f0fe81f4740 1 -- start start 2024-04-23T10:32:03.647 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:32:03.635+0000 7f0fe81f4740 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55c8d0ee0ba0 0x55c8d0f6b0a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T10:32:03.647 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:32:03.635+0000 7f0fe81f4740 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x55c8d0ee14c0 0x55c8d0f6a9d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T10:32:03.647 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:32:03.635+0000 7f0fe81f4740 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x55c8d0ef00a0 0x55c8d0f6a300 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T10:32:03.647 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:32:03.635+0000 7f0fe81f4740 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x55c8d0f693e0 con 0x55c8d0ee0ba0 2024-04-23T10:32:03.647 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:32:03.635+0000 7f0fe81f4740 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 -- 0x55c8d0f69ab0 con 0x55c8d0ee14c0 2024-04-23T10:32:03.647 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:32:03.635+0000 7f0fe81f4740 1 -- --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_getmap magic: 0 -- 0x55c8d0f6a180 con 0x55c8d0ef00a0 2024-04-23T10:32:03.647 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:32:03.635+0000 7f0fdeffd640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55c8d0ee0ba0 0x55c8d0f6b0a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T10:32:03.647 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:32:03.635+0000 7f0fdeffd640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55c8d0ee0ba0 0x55c8d0f6b0a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.114:51718/0 (socket says 172.21.15.114:51718) 2024-04-23T10:32:03.647 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:32:03.635+0000 7f0fdeffd640 1 -- 172.21.15.114:0/4282188277 learned_addr learned my addr 172.21.15.114:0/4282188277 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-04-23T10:32:03.647 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:32:03.635+0000 7f0fde7fc640 1 --2- 172.21.15.114:0/4282188277 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x55c8d0ee14c0 0x55c8d0f6a9d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T10:32:03.647 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:32:03.635+0000 7f0fdeffd640 1 -- 172.21.15.114:0/4282188277 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x55c8d0ef00a0 msgr2=0x55c8d0f6a300 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T10:32:03.647 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:32:03.635+0000 7f0fdeffd640 1 --2- 172.21.15.114:0/4282188277 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x55c8d0ef00a0 0x55c8d0f6a300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:32:03.647 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:32:03.635+0000 7f0fdeffd640 1 -- 172.21.15.114:0/4282188277 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x55c8d0ee14c0 msgr2=0x55c8d0f6a9d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T10:32:03.647 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:32:03.635+0000 7f0fdeffd640 1 --2- 172.21.15.114:0/4282188277 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x55c8d0ee14c0 0x55c8d0f6a9d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:32:03.647 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:32:03.635+0000 7f0fdeffd640 1 -- 172.21.15.114:0/4282188277 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55c8d0ed9160 con 0x55c8d0ee0ba0 2024-04-23T10:32:03.648 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:32:03.635+0000 7f0fde7fc640 1 --2- 172.21.15.114:0/4282188277 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x55c8d0ee14c0 0x55c8d0f6a9d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-23T10:32:03.648 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:32:03.635+0000 7f0fdeffd640 1 --2- 172.21.15.114:0/4282188277 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55c8d0ee0ba0 0x55c8d0f6b0a0 secure :-1 s=READY pgs=686 cs=0 l=1 rev1=1 crypto rx=0x7f0fd002ade0 tx=0x7f0fd005f8a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=58e77131a1d4b03f server_cookie=0 in_seq=0 out_seq=0 2024-04-23T10:32:03.648 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:32:03.635+0000 7f0fc57fa640 1 -- 172.21.15.114:0/4282188277 <== mon.0 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0fd00043e0 con 0x55c8d0ee0ba0 2024-04-23T10:32:03.648 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:32:03.635+0000 7f0fc57fa640 1 -- 172.21.15.114:0/4282188277 <== mon.0 v2:172.21.15.114:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f0fd0036040 con 0x55c8d0ee0ba0 2024-04-23T10:32:03.648 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:32:03.635+0000 7f0fc57fa640 1 -- 172.21.15.114:0/4282188277 <== mon.0 v2:172.21.15.114:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0fd002f980 con 0x55c8d0ee0ba0 2024-04-23T10:32:03.648 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:32:03.636+0000 7f0fe81f4740 1 -- 172.21.15.114:0/4282188277 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55c8d0ee0ba0 msgr2=0x55c8d0f6b0a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T10:32:03.648 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:32:03.636+0000 7f0fe81f4740 1 --2- 172.21.15.114:0/4282188277 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55c8d0ee0ba0 0x55c8d0f6b0a0 secure :-1 s=READY pgs=686 cs=0 l=1 rev1=1 crypto rx=0x7f0fd002ade0 tx=0x7f0fd005f8a0 comp rx=0 tx=0).stop 2024-04-23T10:32:03.648 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:32:03.636+0000 7f0fe81f4740 1 -- 172.21.15.114:0/4282188277 shutdown_connections 2024-04-23T10:32:03.648 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:32:03.636+0000 7f0fe81f4740 1 --2- 172.21.15.114:0/4282188277 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x55c8d0ef00a0 0x55c8d0f6a300 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:32:03.648 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:32:03.636+0000 7f0fe81f4740 1 --2- 172.21.15.114:0/4282188277 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x55c8d0ee14c0 0x55c8d0f6a9d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:32:03.648 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:32:03.636+0000 7f0fe81f4740 1 --2- 172.21.15.114:0/4282188277 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55c8d0ee0ba0 0x55c8d0f6b0a0 unknown :-1 s=CLOSED pgs=686 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:32:03.648 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:32:03.636+0000 7f0fe81f4740 1 -- 172.21.15.114:0/4282188277 >> 172.21.15.114:0/4282188277 conn(0x55c8d0edcfc0 msgr2=0x55c8d0ed80d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T10:32:03.648 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:32:03.636+0000 7f0fe81f4740 1 -- 172.21.15.114:0/4282188277 shutdown_connections 2024-04-23T10:32:03.648 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:32:03.636+0000 7f0fe81f4740 1 -- 172.21.15.114:0/4282188277 wait complete. 2024-04-23T10:32:03.648 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:32:03.636+0000 7f0fe81f4740 1 -- start start 2024-04-23T10:32:03.649 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:32:03.636+0000 7f0fe81f4740 1 Processor -- start 2024-04-23T10:32:03.649 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:32:03.636+0000 7f0fe81f4740 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x55c8d0ee0ba0 0x55c8d0f6b0a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T10:32:03.649 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:32:03.636+0000 7f0fe81f4740 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55c8d0ee14c0 0x55c8d0f69560 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T10:32:03.649 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:32:03.636+0000 7f0fe81f4740 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x55c8d0ef00a0 0x55c8d0eefb10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T10:32:03.649 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:32:03.637+0000 7f0fe81f4740 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x55c8d0f6af20 con 0x55c8d0ee14c0 2024-04-23T10:32:03.649 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:32:03.637+0000 7f0fe81f4740 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 -- 0x55c8d0f6a850 con 0x55c8d0ef00a0 2024-04-23T10:32:03.649 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:32:03.637+0000 7f0fe81f4740 1 -- --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_getmap magic: 0 -- 0x55c8d0f6b5f0 con 0x55c8d0ee0ba0 2024-04-23T10:32:03.649 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:32:03.637+0000 7f0fe81f4740 10 client.0 _ll_register_callbacks cb 0x7f0fd80046c0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-23T10:32:03.649 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:32:03.637+0000 7f0fde7fc640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55c8d0ee14c0 0x55c8d0f69560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T10:32:03.649 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:32:03.637+0000 7f0fde7fc640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55c8d0ee14c0 0x55c8d0f69560 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.114:51726/0 (socket says 172.21.15.114:51726) 2024-04-23T10:32:03.649 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:32:03.637+0000 7f0fde7fc640 1 -- 172.21.15.114:0/3597282773 learned_addr learned my addr 172.21.15.114:0/3597282773 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-04-23T10:32:03.649 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:32:03.637+0000 7f0fdeffd640 1 --2- 172.21.15.114:0/3597282773 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x55c8d0ee0ba0 0x55c8d0f6b0a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T10:32:03.649 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:32:03.637+0000 7f0fdf7fe640 1 --2- 172.21.15.114:0/3597282773 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x55c8d0ef00a0 0x55c8d0eefb10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T10:32:03.649 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:32:03.637+0000 7f0fde7fc640 1 -- 172.21.15.114:0/3597282773 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x55c8d0ee0ba0 msgr2=0x55c8d0f6b0a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T10:32:03.650 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:32:03.637+0000 7f0fde7fc640 1 --2- 172.21.15.114:0/3597282773 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x55c8d0ee0ba0 0x55c8d0f6b0a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:32:03.650 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:32:03.637+0000 7f0fde7fc640 1 -- 172.21.15.114:0/3597282773 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x55c8d0ef00a0 msgr2=0x55c8d0eefb10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T10:32:03.650 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:32:03.637+0000 7f0fde7fc640 1 --2- 172.21.15.114:0/3597282773 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x55c8d0ef00a0 0x55c8d0eefb10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:32:03.650 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:32:03.637+0000 7f0fde7fc640 1 -- 172.21.15.114:0/3597282773 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55c8d0f09240 con 0x55c8d0ee14c0 2024-04-23T10:32:03.650 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:32:03.637+0000 7f0fdf7fe640 1 --2- 172.21.15.114:0/3597282773 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x55c8d0ef00a0 0x55c8d0eefb10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-23T10:32:03.650 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:32:03.637+0000 7f0fde7fc640 1 --2- 172.21.15.114:0/3597282773 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55c8d0ee14c0 0x55c8d0f69560 secure :-1 s=READY pgs=687 cs=0 l=1 rev1=1 crypto rx=0x7f0fd4014010 tx=0x7f0fd4005ae0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=bf5de9bb5fc3542f server_cookie=0 in_seq=0 out_seq=0 2024-04-23T10:32:03.650 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:32:03.637+0000 7f0fc2ff5640 10 client.0 ms_handle_connect on v2:172.21.15.114:3300/0 2024-04-23T10:32:03.650 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:32:03.637+0000 7f0fc2ff5640 1 -- 172.21.15.114:0/3597282773 <== mon.0 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0fd402f040 con 0x55c8d0ee14c0 2024-04-23T10:32:03.650 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:32:03.637+0000 7f0fc2ff5640 1 -- 172.21.15.114:0/3597282773 <== mon.0 v2:172.21.15.114:3300/0 2 ==== osd_map(94..94 src has 1..94) ==== 8585+0+0 (secure 0 0 0) 0x7f0fd4012800 con 0x55c8d0ee14c0 2024-04-23T10:32:03.650 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:32:03.637+0000 7f0fe81f4740 1 -- 172.21.15.114:0/3597282773 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f0fd0036040 con 0x55c8d0ee14c0 2024-04-23T10:32:03.650 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:32:03.637+0000 7f0fc2ff5640 1 -- 172.21.15.114:0/3597282773 <== mon.0 v2:172.21.15.114:3300/0 3 ==== mon_get_version_reply(handle=1 version=30) ==== 24+0+0 (secure 0 0 0) 0x7f0fd4046050 con 0x55c8d0ee14c0 2024-04-23T10:32:03.650 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:32:03.637+0000 7f0fe81f4740 10 client.16242 fetch_fsmap learned FSMap version 30 2024-04-23T10:32:03.650 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:32:03.637+0000 7f0fe81f4740 1 -- 172.21.15.114:0/3597282773 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({config=0+,fsmap=30,monmap=4+}) -- 0x55c8d0eea430 con 0x55c8d0ee14c0 2024-04-23T10:32:03.650 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:32:03.638+0000 7f0fc2ff5640 1 -- 172.21.15.114:0/3597282773 <== mon.0 v2:172.21.15.114:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f0fd0036040 con 0x55c8d0ee14c0 2024-04-23T10:32:03.651 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:32:03.638+0000 7f0fc2ff5640 1 -- 172.21.15.114:0/3597282773 <== mon.0 v2:172.21.15.114:3300/0 5 ==== fsmap(e 30) ==== 8338+0+0 (secure 0 0 0) 0x7f0fd400f040 con 0x55c8d0ee14c0 2024-04-23T10:32:03.651 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:32:03.638+0000 7f0fe81f4740 10 client.16242 fetch_fsmap finished waiting for FSMap version 30 2024-04-23T10:32:03.651 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:32:03.638+0000 7f0fe81f4740 10 client.16242 resolve_mds: resolved name 'd' to daemon mds.d 2024-04-23T10:32:03.651 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:32:03.638+0000 7f0fe81f4740 20 client.16242 populate_metadata read hostname 'smithi114' 2024-04-23T10:32:03.651 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:32:03.638+0000 7f0fe81f4740 1 --2- 172.21.15.114:0/3597282773 >> [v2:172.21.15.114:6836/1439252154,v1:172.21.15.114:6837/1439252154] conn(0x55c8d0f62810 0x55c8d0f61520 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T10:32:03.651 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:32:03.638+0000 7f0fe81f4740 4 client.16242 mds_command: new command op to 14565 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-23T10:32:03.651 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:32:03.638+0000 7f0fe81f4740 1 -- 172.21.15.114:0/3597282773 --> [v2:172.21.15.114:6836/1439252154,v1:172.21.15.114:6837/1439252154] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55c8d0e4e750 con 0x55c8d0f62810 2024-04-23T10:32:03.651 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:32:03.638+0000 7f0fdeffd640 1 --2- 172.21.15.114:0/3597282773 >> [v2:172.21.15.114:6836/1439252154,v1:172.21.15.114:6837/1439252154] conn(0x55c8d0f62810 0x55c8d0f61520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T10:32:03.651 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:32:03.638+0000 7f0fc2ff5640 10 client.16242 ms_handle_connect on v2:172.21.15.114:6836/1439252154 2024-04-23T10:32:03.651 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:32:03.638+0000 7f0fdeffd640 1 --2- 172.21.15.114:0/3597282773 >> [v2:172.21.15.114:6836/1439252154,v1:172.21.15.114:6837/1439252154] conn(0x55c8d0f62810 0x55c8d0f61520 crc :-1 s=READY pgs=13 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.? client_cookie=35e61c3054f1b21b server_cookie=7d6fb5e06c5e3d4e in_seq=0 out_seq=0 2024-04-23T10:32:03.651 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:32:03.639+0000 7f0fc2ff5640 1 -- 172.21.15.114:0/3597282773 <== mds.? v2:172.21.15.114:6836/1439252154 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23115 (crc 0 0 0) 0x55c8d0e4e750 con 0x55c8d0f62810 2024-04-23T10:32:03.651 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:32:03.639+0000 7f0fc2ff5640 10 client.16242 handle_command_reply: tid=0 2024-04-23T10:32:03.651 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:32:03.639+0000 7f0fe81f4740 1 client.16242 shutdown 2024-04-23T10:32:03.651 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:32:03.639+0000 7f0fe81f4740 1 -- 172.21.15.114:0/3597282773 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55c8d0ee14c0 msgr2=0x55c8d0f69560 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T10:32:03.652 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:32:03.639+0000 7f0fe81f4740 1 --2- 172.21.15.114:0/3597282773 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55c8d0ee14c0 0x55c8d0f69560 secure :-1 s=READY pgs=687 cs=0 l=1 rev1=1 crypto rx=0x7f0fd4014010 tx=0x7f0fd4005ae0 comp rx=0 tx=0).stop 2024-04-23T10:32:03.652 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:32:03.639+0000 7f0fe81f4740 1 -- 172.21.15.114:0/3597282773 shutdown_connections 2024-04-23T10:32:03.652 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:32:03.639+0000 7f0fe81f4740 1 --2- 172.21.15.114:0/3597282773 >> [v2:172.21.15.114:6836/1439252154,v1:172.21.15.114:6837/1439252154] conn(0x55c8d0f62810 0x55c8d0f61520 crc :-1 s=READY pgs=13 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:32:03.652 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:32:03.639+0000 7f0fe81f4740 1 --2- 172.21.15.114:0/3597282773 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x55c8d0ef00a0 0x55c8d0eefb10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:32:03.652 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:32:03.639+0000 7f0fc2ff5640 0 client.16242 ms_handle_reset on v2:172.21.15.114:6836/1439252154 2024-04-23T10:32:03.652 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:32:03.639+0000 7f0fe81f4740 1 --2- 172.21.15.114:0/3597282773 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55c8d0ee14c0 0x55c8d0f69560 unknown :-1 s=CLOSED pgs=687 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:32:03.652 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:32:03.639+0000 7f0fe81f4740 1 --2- 172.21.15.114:0/3597282773 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x55c8d0ee0ba0 0x55c8d0f6b0a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:32:03.652 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:32:03.639+0000 7f0fe81f4740 1 -- 172.21.15.114:0/3597282773 >> 172.21.15.114:0/3597282773 conn(0x55c8d0edcfc0 msgr2=0x55c8d0ee8460 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T10:32:03.652 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:32:03.639+0000 7f0fe81f4740 1 -- 172.21.15.114:0/3597282773 shutdown_connections 2024-04-23T10:32:03.652 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:32:03.639+0000 7f0fe81f4740 1 -- 172.21.15.114:0/3597282773 wait complete. 2024-04-23T10:32:03.652 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:32:03.639+0000 7f0fe81f4740 20 client.16242 trim_cache size 0 max 16384 2024-04-23T10:32:03.665 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:32:03.655+0000 7f0fe81f4740 1 Processor -- start 2024-04-23T10:32:03.665 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:32:03.655+0000 7f0fe81f4740 1 -- start start 2024-04-23T10:32:03.665 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:32:03.655+0000 7f0fe81f4740 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x55c8d0ee0ba0 0x55c8d0f69c30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T10:32:03.665 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:32:03.655+0000 7f0fe81f4740 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x55c8d0ee14c0 0x55c8d0f61520 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T10:32:03.665 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:32:03.655+0000 7f0fe81f4740 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55c8d0ef00a0 0x55c8d0f69560 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T10:32:03.665 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:32:03.655+0000 7f0fe81f4740 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x55c8d0ee3250 con 0x55c8d0ef00a0 2024-04-23T10:32:03.665 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:32:03.655+0000 7f0fe81f4740 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 -- 0x55c8d0f0fb40 con 0x55c8d0ee0ba0 2024-04-23T10:32:03.665 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:32:03.655+0000 7f0fe81f4740 1 -- --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_getmap magic: 0 -- 0x55c8d0f10210 con 0x55c8d0ee14c0 2024-04-23T10:32:03.665 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:32:03.655+0000 7f0fdf7fe640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55c8d0ef00a0 0x55c8d0f69560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T10:32:03.666 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:32:03.655+0000 7f0fdf7fe640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55c8d0ef00a0 0x55c8d0f69560 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.114:51742/0 (socket says 172.21.15.114:51742) 2024-04-23T10:32:03.666 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:32:03.655+0000 7f0fdf7fe640 1 -- 172.21.15.114:0/446884760 learned_addr learned my addr 172.21.15.114:0/446884760 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-04-23T10:32:03.666 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:32:03.655+0000 7f0fde7fc640 1 --2- 172.21.15.114:0/446884760 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x55c8d0ee14c0 0x55c8d0f61520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T10:32:03.666 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:32:03.655+0000 7f0fdeffd640 1 --2- 172.21.15.114:0/446884760 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x55c8d0ee0ba0 0x55c8d0f69c30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T10:32:03.666 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:32:03.655+0000 7f0fdf7fe640 1 -- 172.21.15.114:0/446884760 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x55c8d0ee14c0 msgr2=0x55c8d0f61520 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T10:32:03.666 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:32:03.655+0000 7f0fdf7fe640 1 --2- 172.21.15.114:0/446884760 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x55c8d0ee14c0 0x55c8d0f61520 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:32:03.666 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:32:03.655+0000 7f0fdf7fe640 1 -- 172.21.15.114:0/446884760 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x55c8d0ee0ba0 msgr2=0x55c8d0f69c30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T10:32:03.666 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:32:03.656+0000 7f0fdf7fe640 1 --2- 172.21.15.114:0/446884760 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x55c8d0ee0ba0 0x55c8d0f69c30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:32:03.666 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:32:03.656+0000 7f0fdf7fe640 1 -- 172.21.15.114:0/446884760 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55c8d0f03350 con 0x55c8d0ef00a0 2024-04-23T10:32:03.666 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:32:03.656+0000 7f0fdeffd640 1 --2- 172.21.15.114:0/446884760 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x55c8d0ee0ba0 0x55c8d0f69c30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-23T10:32:03.666 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:32:03.656+0000 7f0fdf7fe640 1 --2- 172.21.15.114:0/446884760 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55c8d0ef00a0 0x55c8d0f69560 secure :-1 s=READY pgs=688 cs=0 l=1 rev1=1 crypto rx=0x7f0fd8098f80 tx=0x7f0fd80981d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=fbed055f1c4740a8 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T10:32:03.666 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:32:03.656+0000 7f0fc4ff9640 1 -- 172.21.15.114:0/446884760 <== mon.0 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0fd8020040 con 0x55c8d0ef00a0 2024-04-23T10:32:03.666 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:32:03.656+0000 7f0fc4ff9640 1 -- 172.21.15.114:0/446884760 <== mon.0 v2:172.21.15.114:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f0fd809c5b0 con 0x55c8d0ef00a0 2024-04-23T10:32:03.667 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:32:03.656+0000 7f0fc4ff9640 1 -- 172.21.15.114:0/446884760 <== mon.0 v2:172.21.15.114:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0fd80d00e0 con 0x55c8d0ef00a0 2024-04-23T10:32:03.667 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:32:03.656+0000 7f0fe81f4740 1 -- 172.21.15.114:0/446884760 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55c8d0ef00a0 msgr2=0x55c8d0f69560 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T10:32:03.667 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:32:03.656+0000 7f0fe81f4740 1 --2- 172.21.15.114:0/446884760 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55c8d0ef00a0 0x55c8d0f69560 secure :-1 s=READY pgs=688 cs=0 l=1 rev1=1 crypto rx=0x7f0fd8098f80 tx=0x7f0fd80981d0 comp rx=0 tx=0).stop 2024-04-23T10:32:03.667 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:32:03.656+0000 7f0fe81f4740 1 -- 172.21.15.114:0/446884760 shutdown_connections 2024-04-23T10:32:03.667 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:32:03.656+0000 7f0fe81f4740 1 --2- 172.21.15.114:0/446884760 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55c8d0ef00a0 0x55c8d0f69560 unknown :-1 s=CLOSED pgs=688 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:32:03.667 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:32:03.656+0000 7f0fe81f4740 1 --2- 172.21.15.114:0/446884760 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x55c8d0ee14c0 0x55c8d0f61520 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:32:03.667 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:32:03.656+0000 7f0fe81f4740 1 --2- 172.21.15.114:0/446884760 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x55c8d0ee0ba0 0x55c8d0f69c30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:32:03.667 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:32:03.656+0000 7f0fe81f4740 1 -- 172.21.15.114:0/446884760 >> 172.21.15.114:0/446884760 conn(0x55c8d0edcfc0 msgr2=0x55c8d0edc0a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T10:32:03.667 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:32:03.656+0000 7f0fe81f4740 1 -- 172.21.15.114:0/446884760 shutdown_connections 2024-04-23T10:32:03.667 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:32:03.656+0000 7f0fe81f4740 1 -- 172.21.15.114:0/446884760 wait complete. 2024-04-23T10:32:03.667 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:32:03.657+0000 7f0fe81f4740 1 -- start start 2024-04-23T10:32:03.667 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:32:03.657+0000 7f0fe81f4740 1 Processor -- start 2024-04-23T10:32:03.667 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:32:03.657+0000 7f0fe81f4740 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x55c8d0ee0ba0 0x55c8d0f69c30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T10:32:03.667 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:32:03.657+0000 7f0fe81f4740 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x55c8d0ee14c0 0x55c8d0f61520 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T10:32:03.667 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:32:03.657+0000 7f0fe81f4740 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55c8d0ef00a0 0x55c8d0f69560 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T10:32:03.668 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:32:03.657+0000 7f0fe81f4740 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x55c8d0f613a0 con 0x55c8d0ef00a0 2024-04-23T10:32:03.668 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:32:03.657+0000 7f0fe81f4740 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 -- 0x55c8d0f108e0 con 0x55c8d0ee0ba0 2024-04-23T10:32:03.668 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:32:03.657+0000 7f0fe81f4740 1 -- --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_getmap magic: 0 -- 0x55c8d0f61a70 con 0x55c8d0ee14c0 2024-04-23T10:32:03.668 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:32:03.657+0000 7f0fe81f4740 10 client.0 _ll_register_callbacks cb 0x7f0fd402d070 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-23T10:32:03.668 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:32:03.657+0000 7f0fdf7fe640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55c8d0ef00a0 0x55c8d0f69560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T10:32:03.668 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:32:03.657+0000 7f0fdf7fe640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55c8d0ef00a0 0x55c8d0f69560 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.114:51752/0 (socket says 172.21.15.114:51752) 2024-04-23T10:32:03.668 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:32:03.657+0000 7f0fdf7fe640 1 -- 172.21.15.114:0/3286118491 learned_addr learned my addr 172.21.15.114:0/3286118491 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-04-23T10:32:03.668 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:32:03.657+0000 7f0fdeffd640 1 --2- 172.21.15.114:0/3286118491 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x55c8d0ee0ba0 0x55c8d0f69c30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T10:32:03.668 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:32:03.657+0000 7f0fde7fc640 1 --2- 172.21.15.114:0/3286118491 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x55c8d0ee14c0 0x55c8d0f61520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T10:32:03.668 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:32:03.657+0000 7f0fdf7fe640 1 -- 172.21.15.114:0/3286118491 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x55c8d0ee14c0 msgr2=0x55c8d0f61520 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T10:32:03.668 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:32:03.657+0000 7f0fdf7fe640 1 --2- 172.21.15.114:0/3286118491 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x55c8d0ee14c0 0x55c8d0f61520 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:32:03.668 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:32:03.657+0000 7f0fdf7fe640 1 -- 172.21.15.114:0/3286118491 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x55c8d0ee0ba0 msgr2=0x55c8d0f69c30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T10:32:03.668 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:32:03.657+0000 7f0fdf7fe640 1 --2- 172.21.15.114:0/3286118491 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x55c8d0ee0ba0 0x55c8d0f69c30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:32:03.668 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:32:03.657+0000 7f0fdf7fe640 1 -- 172.21.15.114:0/3286118491 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55c8d0f0ffb0 con 0x55c8d0ef00a0 2024-04-23T10:32:03.669 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:32:03.657+0000 7f0fdeffd640 1 --2- 172.21.15.114:0/3286118491 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x55c8d0ee0ba0 0x55c8d0f69c30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-23T10:32:03.669 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:32:03.658+0000 7f0fdf7fe640 1 --2- 172.21.15.114:0/3286118491 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55c8d0ef00a0 0x55c8d0f69560 secure :-1 s=READY pgs=689 cs=0 l=1 rev1=1 crypto rx=0x7f0fd80a4c60 tx=0x7f0fd800ded0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=76baf089fb21e7ec server_cookie=0 in_seq=0 out_seq=0 2024-04-23T10:32:03.669 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:32:03.658+0000 7f0fc2ff5640 10 client.0 ms_handle_connect on v2:172.21.15.114:3300/0 2024-04-23T10:32:03.669 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:32:03.658+0000 7f0fc2ff5640 1 -- 172.21.15.114:0/3286118491 <== mon.0 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0fd80ba060 con 0x55c8d0ef00a0 2024-04-23T10:32:03.669 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:32:03.658+0000 7f0fc2ff5640 1 -- 172.21.15.114:0/3286118491 <== mon.0 v2:172.21.15.114:3300/0 2 ==== osd_map(94..94 src has 1..94) ==== 8585+0+0 (secure 0 0 0) 0x7f0fd809ba00 con 0x55c8d0ef00a0 2024-04-23T10:32:03.669 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:32:03.658+0000 7f0fe81f4740 1 -- 172.21.15.114:0/3286118491 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f0fd809c5b0 con 0x55c8d0ef00a0 2024-04-23T10:32:03.669 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:32:03.658+0000 7f0fc2ff5640 1 -- 172.21.15.114:0/3286118491 <== mon.0 v2:172.21.15.114:3300/0 3 ==== mon_get_version_reply(handle=1 version=30) ==== 24+0+0 (secure 0 0 0) 0x7f0fd8020070 con 0x55c8d0ef00a0 2024-04-23T10:32:03.669 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:32:03.658+0000 7f0fe81f4740 10 client.16248 fetch_fsmap learned FSMap version 30 2024-04-23T10:32:03.669 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:32:03.658+0000 7f0fe81f4740 1 -- 172.21.15.114:0/3286118491 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({config=0+,fsmap=30,monmap=4+}) -- 0x55c8d0f0ae20 con 0x55c8d0ef00a0 2024-04-23T10:32:03.669 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:32:03.658+0000 7f0fc2ff5640 1 -- 172.21.15.114:0/3286118491 <== mon.0 v2:172.21.15.114:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f0fd809c5b0 con 0x55c8d0ef00a0 2024-04-23T10:32:03.669 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:32:03.658+0000 7f0fc2ff5640 1 -- 172.21.15.114:0/3286118491 <== mon.0 v2:172.21.15.114:3300/0 5 ==== fsmap(e 30) ==== 8338+0+0 (secure 0 0 0) 0x7f0fd80a1af0 con 0x55c8d0ef00a0 2024-04-23T10:32:03.669 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:32:03.658+0000 7f0fe81f4740 10 client.16248 fetch_fsmap finished waiting for FSMap version 30 2024-04-23T10:32:03.669 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:32:03.658+0000 7f0fe81f4740 10 client.16248 resolve_mds: resolved name 'd' to daemon mds.d 2024-04-23T10:32:03.669 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:32:03.658+0000 7f0fe81f4740 20 client.16248 populate_metadata read hostname 'smithi114' 2024-04-23T10:32:03.670 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:32:03.658+0000 7f0fe81f4740 1 --2- 172.21.15.114:0/3286118491 >> [v2:172.21.15.114:6836/1439252154,v1:172.21.15.114:6837/1439252154] conn(0x55c8d0f7bc70 0x55c8d0f680c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T10:32:03.670 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:32:03.658+0000 7f0fe81f4740 4 client.16248 mds_command: new command op to 14565 tid=0 multi_id=0 [{"prefix": "ops", "flags": ["locks"]}] 2024-04-23T10:32:03.670 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:32:03.658+0000 7f0fe81f4740 1 -- 172.21.15.114:0/3286118491 --> [v2:172.21.15.114:6836/1439252154,v1:172.21.15.114:6837/1439252154] -- command(tid 0: {"prefix": "ops", "flags": ["locks"]}) -- 0x55c8d0dcab40 con 0x55c8d0f7bc70 2024-04-23T10:32:03.670 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:32:03.658+0000 7f0fdeffd640 1 --2- 172.21.15.114:0/3286118491 >> [v2:172.21.15.114:6836/1439252154,v1:172.21.15.114:6837/1439252154] conn(0x55c8d0f7bc70 0x55c8d0f680c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T10:32:03.670 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:32:03.659+0000 7f0fdeffd640 1 --2- 172.21.15.114:0/3286118491 >> [v2:172.21.15.114:6836/1439252154,v1:172.21.15.114:6837/1439252154] conn(0x55c8d0f7bc70 0x55c8d0f680c0 crc :-1 s=READY pgs=14 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.? client_cookie=2ebe66bef19e2375 server_cookie=e5f278e78bf9276c in_seq=0 out_seq=0 2024-04-23T10:32:03.670 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:32:03.659+0000 7f0fc2ff5640 10 client.16248 ms_handle_connect on v2:172.21.15.114:6836/1439252154 2024-04-23T10:32:03.670 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:32:03.659+0000 7f0fc2ff5640 1 -- 172.21.15.114:0/3286118491 <== mds.? v2:172.21.15.114:6836/1439252154 1 ==== command_reply(tid 0: -38 ) ==== 8+0+0 (crc 0 0 0) 0x55c8d0dcab40 con 0x55c8d0f7bc70 2024-04-23T10:32:03.670 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:32:03.659+0000 7f0fc2ff5640 10 client.16248 handle_command_reply: tid=0 2024-04-23T10:32:03.670 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:32:03.659+0000 7f0fe81f4740 1 client.16248 shutdown 2024-04-23T10:32:03.670 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:32:03.659+0000 7f0fe81f4740 1 -- 172.21.15.114:0/3286118491 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55c8d0ef00a0 msgr2=0x55c8d0f69560 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T10:32:03.670 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:32:03.659+0000 7f0fe81f4740 1 --2- 172.21.15.114:0/3286118491 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55c8d0ef00a0 0x55c8d0f69560 secure :-1 s=READY pgs=689 cs=0 l=1 rev1=1 crypto rx=0x7f0fd80a4c60 tx=0x7f0fd800ded0 comp rx=0 tx=0).stop 2024-04-23T10:32:03.670 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:32:03.659+0000 7f0fe81f4740 1 -- 172.21.15.114:0/3286118491 shutdown_connections 2024-04-23T10:32:03.670 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:32:03.659+0000 7f0fe81f4740 1 --2- 172.21.15.114:0/3286118491 >> [v2:172.21.15.114:6836/1439252154,v1:172.21.15.114:6837/1439252154] conn(0x55c8d0f7bc70 0x55c8d0f680c0 crc :-1 s=READY pgs=14 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:32:03.670 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:32:03.659+0000 7f0fe81f4740 1 --2- 172.21.15.114:0/3286118491 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55c8d0ef00a0 0x55c8d0f69560 unknown :-1 s=CLOSED pgs=689 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:32:03.671 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:32:03.659+0000 7f0fe81f4740 1 --2- 172.21.15.114:0/3286118491 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x55c8d0ee14c0 0x55c8d0f61520 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:32:03.671 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:32:03.659+0000 7f0fe81f4740 1 --2- 172.21.15.114:0/3286118491 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x55c8d0ee0ba0 0x55c8d0f69c30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:32:03.671 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:32:03.659+0000 7f0fc2ff5640 0 client.16248 ms_handle_reset on v2:172.21.15.114:6836/1439252154 2024-04-23T10:32:03.671 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:32:03.659+0000 7f0fe81f4740 1 -- 172.21.15.114:0/3286118491 >> 172.21.15.114:0/3286118491 conn(0x55c8d0edcfc0 msgr2=0x55c8d0ee8460 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T10:32:03.671 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:32:03.659+0000 7f0fe81f4740 1 -- 172.21.15.114:0/3286118491 shutdown_connections 2024-04-23T10:32:03.671 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:32:03.659+0000 7f0fe81f4740 1 -- 172.21.15.114:0/3286118491 wait complete. 2024-04-23T10:32:03.671 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:32:03.659+0000 7f0fe81f4740 20 client.16248 trim_cache size 0 max 16384 2024-04-23T10:32:03.671 INFO:teuthology.orchestra.run.smithi114.stderr:Error ENOSYS: 2024-04-23T10:32:03.671 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:32:03.659+0000 7f0fe81f4740 1 Processor -- start 2024-04-23T10:32:03.671 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:32:03.659+0000 7f0fe81f4740 1 -- start start 2024-04-23T10:32:03.671 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:32:03.660+0000 7f0fe81f4740 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x55c8d0ee0ba0 0x55c8d0f680c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T10:32:03.671 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:32:03.660+0000 7f0fe81f4740 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55c8d0ee14c0 0x55c8d0f69c30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T10:32:03.671 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:32:03.660+0000 7f0fe81f4740 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x55c8d0ef00a0 0x55c8d0f69560 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T10:32:03.671 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:32:03.660+0000 7f0fe81f4740 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x55c8d0f6ad20 con 0x55c8d0ee14c0 2024-04-23T10:32:03.671 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:32:03.660+0000 7f0fe81f4740 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 -- 0x55c8d0f6b3f0 con 0x55c8d0ee0ba0 2024-04-23T10:32:03.672 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:32:03.660+0000 7f0fe81f4740 1 -- --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_getmap magic: 0 -- 0x55c8d0ee27e0 con 0x55c8d0ef00a0 2024-04-23T10:32:03.672 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:32:03.660+0000 7f0fde7fc640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55c8d0ee14c0 0x55c8d0f69c30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T10:32:03.672 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:32:03.660+0000 7f0fde7fc640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55c8d0ee14c0 0x55c8d0f69c30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.114:51760/0 (socket says 172.21.15.114:51760) 2024-04-23T10:32:03.672 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:32:03.660+0000 7f0fde7fc640 1 -- 172.21.15.114:0/2401464013 learned_addr learned my addr 172.21.15.114:0/2401464013 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-04-23T10:32:03.672 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:32:03.660+0000 7f0fde7fc640 1 -- 172.21.15.114:0/2401464013 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x55c8d0ef00a0 msgr2=0x55c8d0f69560 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T10:32:03.672 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:32:03.660+0000 7f0fdf7fe640 1 --2- 172.21.15.114:0/2401464013 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x55c8d0ef00a0 0x55c8d0f69560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T10:32:03.672 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:32:03.660+0000 7f0fde7fc640 1 --2- 172.21.15.114:0/2401464013 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x55c8d0ef00a0 0x55c8d0f69560 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:32:03.672 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:32:03.660+0000 7f0fde7fc640 1 -- 172.21.15.114:0/2401464013 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x55c8d0ee0ba0 msgr2=0x55c8d0f680c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T10:32:03.672 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:32:03.660+0000 7f0fde7fc640 1 --2- 172.21.15.114:0/2401464013 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x55c8d0ee0ba0 0x55c8d0f680c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:32:03.672 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:32:03.660+0000 7f0fde7fc640 1 -- 172.21.15.114:0/2401464013 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55c8d0ee8be0 con 0x55c8d0ee14c0 2024-04-23T10:32:03.672 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:32:03.660+0000 7f0fde7fc640 1 --2- 172.21.15.114:0/2401464013 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55c8d0ee14c0 0x55c8d0f69c30 secure :-1 s=READY pgs=690 cs=0 l=1 rev1=1 crypto rx=0x55c8d0ee36c0 tx=0x7f0fd4005ed0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8b43819a4cd79ee8 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T10:32:03.672 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:32:03.660+0000 7f0fc57fa640 1 -- 172.21.15.114:0/2401464013 <== mon.0 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0fd402f070 con 0x55c8d0ee14c0 2024-04-23T10:32:03.672 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:32:03.660+0000 7f0fc57fa640 1 -- 172.21.15.114:0/2401464013 <== mon.0 v2:172.21.15.114:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f0fd4011cb0 con 0x55c8d0ee14c0 2024-04-23T10:32:03.673 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:32:03.660+0000 7f0fc57fa640 1 -- 172.21.15.114:0/2401464013 <== mon.0 v2:172.21.15.114:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0fd4005040 con 0x55c8d0ee14c0 2024-04-23T10:32:03.673 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:32:03.661+0000 7f0fe81f4740 1 -- 172.21.15.114:0/2401464013 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55c8d0ee14c0 msgr2=0x55c8d0f69c30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T10:32:03.673 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:32:03.661+0000 7f0fe81f4740 1 --2- 172.21.15.114:0/2401464013 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55c8d0ee14c0 0x55c8d0f69c30 secure :-1 s=READY pgs=690 cs=0 l=1 rev1=1 crypto rx=0x55c8d0ee36c0 tx=0x7f0fd4005ed0 comp rx=0 tx=0).stop 2024-04-23T10:32:03.673 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:32:03.661+0000 7f0fe81f4740 1 -- 172.21.15.114:0/2401464013 shutdown_connections 2024-04-23T10:32:03.673 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:32:03.661+0000 7f0fe81f4740 1 --2- 172.21.15.114:0/2401464013 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x55c8d0ef00a0 0x55c8d0f69560 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:32:03.673 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:32:03.661+0000 7f0fe81f4740 1 --2- 172.21.15.114:0/2401464013 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55c8d0ee14c0 0x55c8d0f69c30 unknown :-1 s=CLOSED pgs=690 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:32:03.673 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:32:03.661+0000 7f0fe81f4740 1 --2- 172.21.15.114:0/2401464013 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x55c8d0ee0ba0 0x55c8d0f680c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:32:03.673 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:32:03.661+0000 7f0fe81f4740 1 -- 172.21.15.114:0/2401464013 >> 172.21.15.114:0/2401464013 conn(0x55c8d0edcfc0 msgr2=0x55c8d0ed80d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T10:32:03.673 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:32:03.661+0000 7f0fe81f4740 1 -- 172.21.15.114:0/2401464013 shutdown_connections 2024-04-23T10:32:03.673 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:32:03.661+0000 7f0fe81f4740 1 -- 172.21.15.114:0/2401464013 wait complete. 2024-04-23T10:32:03.673 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:32:03.661+0000 7f0fe81f4740 1 -- start start 2024-04-23T10:32:03.673 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:32:03.661+0000 7f0fe81f4740 1 Processor -- start 2024-04-23T10:32:03.673 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:32:03.661+0000 7f0fe81f4740 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x55c8d0ee0ba0 0x55c8d0f6be40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T10:32:03.673 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:32:03.661+0000 7f0fe81f4740 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55c8d0ee14c0 0x55c8d0f680c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T10:32:03.673 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:32:03.661+0000 7f0fe81f4740 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x55c8d0ef00a0 0x55c8d0f69c30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T10:32:03.674 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:32:03.661+0000 7f0fe81f4740 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x55c8d0f67f40 con 0x55c8d0ee14c0 2024-04-23T10:32:03.674 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:32:03.661+0000 7f0fe81f4740 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 -- 0x55c8d0ee2eb0 con 0x55c8d0ef00a0 2024-04-23T10:32:03.674 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:32:03.661+0000 7f0fe81f4740 1 -- --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_getmap magic: 0 -- 0x55c8d0f68610 con 0x55c8d0ee0ba0 2024-04-23T10:32:03.674 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:32:03.661+0000 7f0fe81f4740 10 client.0 _ll_register_callbacks cb 0x7f0fd8013030 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-23T10:32:03.674 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:32:03.661+0000 7f0fde7fc640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55c8d0ee14c0 0x55c8d0f680c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T10:32:03.674 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:32:03.661+0000 7f0fde7fc640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55c8d0ee14c0 0x55c8d0f680c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.114:51770/0 (socket says 172.21.15.114:51770) 2024-04-23T10:32:03.674 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:32:03.661+0000 7f0fde7fc640 1 -- 172.21.15.114:0/1197318304 learned_addr learned my addr 172.21.15.114:0/1197318304 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-04-23T10:32:03.674 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:32:03.661+0000 7f0fdeffd640 1 --2- 172.21.15.114:0/1197318304 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x55c8d0ee0ba0 0x55c8d0f6be40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T10:32:03.674 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:32:03.661+0000 7f0fdf7fe640 1 --2- 172.21.15.114:0/1197318304 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x55c8d0ef00a0 0x55c8d0f69c30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T10:32:03.674 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:32:03.662+0000 7f0fde7fc640 1 -- 172.21.15.114:0/1197318304 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x55c8d0ee0ba0 msgr2=0x55c8d0f6be40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T10:32:03.674 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:32:03.662+0000 7f0fde7fc640 1 --2- 172.21.15.114:0/1197318304 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x55c8d0ee0ba0 0x55c8d0f6be40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:32:03.674 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:32:03.662+0000 7f0fde7fc640 1 -- 172.21.15.114:0/1197318304 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x55c8d0ef00a0 msgr2=0x55c8d0f69c30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T10:32:03.674 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:32:03.662+0000 7f0fde7fc640 1 --2- 172.21.15.114:0/1197318304 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x55c8d0ef00a0 0x55c8d0f69c30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:32:03.674 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:32:03.662+0000 7f0fde7fc640 1 -- 172.21.15.114:0/1197318304 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55c8d0f61ee0 con 0x55c8d0ee14c0 2024-04-23T10:32:03.675 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:32:03.662+0000 7f0fdf7fe640 1 --2- 172.21.15.114:0/1197318304 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x55c8d0ef00a0 0x55c8d0f69c30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-23T10:32:03.675 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:32:03.662+0000 7f0fde7fc640 1 --2- 172.21.15.114:0/1197318304 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55c8d0ee14c0 0x55c8d0f680c0 secure :-1 s=READY pgs=691 cs=0 l=1 rev1=1 crypto rx=0x55c8d0ee36c0 tx=0x7f0fd4009d60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=eb4c9b9237c471b2 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T10:32:03.675 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:32:03.662+0000 7f0fc2ff5640 10 client.0 ms_handle_connect on v2:172.21.15.114:3300/0 2024-04-23T10:32:03.675 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:32:03.662+0000 7f0fc2ff5640 1 -- 172.21.15.114:0/1197318304 <== mon.0 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0fd402f040 con 0x55c8d0ee14c0 2024-04-23T10:32:03.675 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:32:03.662+0000 7f0fc2ff5640 1 -- 172.21.15.114:0/1197318304 <== mon.0 v2:172.21.15.114:3300/0 2 ==== osd_map(94..94 src has 1..94) ==== 8585+0+0 (secure 0 0 0) 0x7f0fd40037f0 con 0x55c8d0ee14c0 2024-04-23T10:32:03.675 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:32:03.662+0000 7f0fe81f4740 1 -- 172.21.15.114:0/1197318304 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f0fd4011cb0 con 0x55c8d0ee14c0 2024-04-23T10:32:03.675 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:32:03.662+0000 7f0fc2ff5640 1 -- 172.21.15.114:0/1197318304 <== mon.0 v2:172.21.15.114:3300/0 3 ==== mon_get_version_reply(handle=1 version=30) ==== 24+0+0 (secure 0 0 0) 0x7f0fd4003e90 con 0x55c8d0ee14c0 2024-04-23T10:32:03.675 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:32:03.662+0000 7f0fe81f4740 10 client.16254 fetch_fsmap learned FSMap version 30 2024-04-23T10:32:03.675 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:32:03.662+0000 7f0fe81f4740 1 -- 172.21.15.114:0/1197318304 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({config=0+,fsmap=30,monmap=4+}) -- 0x55c8d0f61ee0 con 0x55c8d0ee14c0 2024-04-23T10:32:03.675 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:32:03.662+0000 7f0fc2ff5640 1 -- 172.21.15.114:0/1197318304 <== mon.0 v2:172.21.15.114:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f0fd4011cb0 con 0x55c8d0ee14c0 2024-04-23T10:32:03.675 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:32:03.662+0000 7f0fc2ff5640 1 -- 172.21.15.114:0/1197318304 <== mon.0 v2:172.21.15.114:3300/0 5 ==== fsmap(e 30) ==== 8338+0+0 (secure 0 0 0) 0x7f0fd4050030 con 0x55c8d0ee14c0 2024-04-23T10:32:03.675 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:32:03.662+0000 7f0fe81f4740 10 client.16254 fetch_fsmap finished waiting for FSMap version 30 2024-04-23T10:32:03.675 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:32:03.662+0000 7f0fe81f4740 10 client.16254 resolve_mds: resolved name 'g' to daemon mds.g 2024-04-23T10:32:03.675 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:32:03.662+0000 7f0fe81f4740 20 client.16254 populate_metadata read hostname 'smithi114' 2024-04-23T10:32:03.675 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:32:03.662+0000 7f0fe81f4740 1 --2- 172.21.15.114:0/1197318304 >> [v2:172.21.15.114:6838/3849796939,v1:172.21.15.114:6839/3849796939] conn(0x55c8d0edf7e0 0x55c8d0f66d90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T10:32:03.676 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:32:03.662+0000 7f0fe81f4740 4 client.16254 mds_command: new command op to 14634 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-23T10:32:03.676 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:32:03.662+0000 7f0fe81f4740 1 -- 172.21.15.114:0/1197318304 --> [v2:172.21.15.114:6838/3849796939,v1:172.21.15.114:6839/3849796939] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55c8d0e178d0 con 0x55c8d0edf7e0 2024-04-23T10:32:03.676 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:32:03.663+0000 7f0fdeffd640 1 --2- 172.21.15.114:0/1197318304 >> [v2:172.21.15.114:6838/3849796939,v1:172.21.15.114:6839/3849796939] conn(0x55c8d0edf7e0 0x55c8d0f66d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T10:32:03.676 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:32:03.663+0000 7f0fdeffd640 1 --2- 172.21.15.114:0/1197318304 >> [v2:172.21.15.114:6838/3849796939,v1:172.21.15.114:6839/3849796939] conn(0x55c8d0edf7e0 0x55c8d0f66d90 crc :-1 s=READY pgs=13 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.? client_cookie=9afbed47374097b0 server_cookie=d95f612aa44e86e4 in_seq=0 out_seq=0 2024-04-23T10:32:03.676 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:32:03.663+0000 7f0fc2ff5640 10 client.16254 ms_handle_connect on v2:172.21.15.114:6838/3849796939 2024-04-23T10:32:03.676 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:32:03.663+0000 7f0fc2ff5640 1 -- 172.21.15.114:0/1197318304 <== mds.? v2:172.21.15.114:6838/3849796939 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23115 (crc 0 0 0) 0x55c8d0e178d0 con 0x55c8d0edf7e0 2024-04-23T10:32:03.676 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:32:03.663+0000 7f0fc2ff5640 10 client.16254 handle_command_reply: tid=0 2024-04-23T10:32:03.676 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:32:03.663+0000 7f0fe81f4740 1 client.16254 shutdown 2024-04-23T10:32:03.676 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:32:03.664+0000 7f0fe81f4740 1 -- 172.21.15.114:0/1197318304 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55c8d0ee14c0 msgr2=0x55c8d0f680c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T10:32:03.676 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:32:03.664+0000 7f0fe81f4740 1 --2- 172.21.15.114:0/1197318304 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55c8d0ee14c0 0x55c8d0f680c0 secure :-1 s=READY pgs=691 cs=0 l=1 rev1=1 crypto rx=0x55c8d0ee36c0 tx=0x7f0fd4009d60 comp rx=0 tx=0).stop 2024-04-23T10:32:03.676 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:32:03.664+0000 7f0fe81f4740 1 -- 172.21.15.114:0/1197318304 shutdown_connections 2024-04-23T10:32:03.676 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:32:03.664+0000 7f0fe81f4740 1 --2- 172.21.15.114:0/1197318304 >> [v2:172.21.15.114:6838/3849796939,v1:172.21.15.114:6839/3849796939] conn(0x55c8d0edf7e0 0x55c8d0f66d90 crc :-1 s=READY pgs=13 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:32:03.676 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:32:03.664+0000 7f0fe81f4740 1 --2- 172.21.15.114:0/1197318304 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x55c8d0ef00a0 0x55c8d0f69c30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:32:03.676 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:32:03.664+0000 7f0fc2ff5640 0 client.16254 ms_handle_reset on v2:172.21.15.114:6838/3849796939 2024-04-23T10:32:03.676 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:32:03.664+0000 7f0fe81f4740 1 --2- 172.21.15.114:0/1197318304 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55c8d0ee14c0 0x55c8d0f680c0 unknown :-1 s=CLOSED pgs=691 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:32:03.677 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:32:03.664+0000 7f0fe81f4740 1 --2- 172.21.15.114:0/1197318304 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x55c8d0ee0ba0 0x55c8d0f6be40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:32:03.677 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:32:03.664+0000 7f0fe81f4740 1 -- 172.21.15.114:0/1197318304 >> 172.21.15.114:0/1197318304 conn(0x55c8d0edcfc0 msgr2=0x55c8d0edc0a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T10:32:03.677 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:32:03.664+0000 7f0fe81f4740 1 -- 172.21.15.114:0/1197318304 shutdown_connections 2024-04-23T10:32:03.677 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:32:03.664+0000 7f0fe81f4740 1 -- 172.21.15.114:0/1197318304 wait complete. 2024-04-23T10:32:03.677 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:32:03.664+0000 7f0fe81f4740 20 client.16254 trim_cache size 0 max 16384 2024-04-23T10:32:03.690 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:32:03.679+0000 7f0fe81f4740 1 Processor -- start 2024-04-23T10:32:03.690 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:32:03.679+0000 7f0fe81f4740 1 -- start start 2024-04-23T10:32:03.690 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:32:03.679+0000 7f0fe81f4740 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x55c8d0ee0ba0 0x55c8d0f66d90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T10:32:03.690 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:32:03.679+0000 7f0fe81f4740 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55c8d0ee14c0 0x55c8d0f6be40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T10:32:03.690 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:32:03.679+0000 7f0fe81f4740 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x55c8d0ef00a0 0x55c8d0f69c30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T10:32:03.690 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:32:03.679+0000 7f0fe81f4740 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x55c8d0f6aff0 con 0x55c8d0ee14c0 2024-04-23T10:32:03.690 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:32:03.679+0000 7f0fe81f4740 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 -- 0x55c8d0eeda60 con 0x55c8d0ef00a0 2024-04-23T10:32:03.691 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:32:03.679+0000 7f0fe81f4740 1 -- --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_getmap magic: 0 -- 0x55c8d0f66c10 con 0x55c8d0ee0ba0 2024-04-23T10:32:03.691 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:32:03.679+0000 7f0fde7fc640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55c8d0ee14c0 0x55c8d0f6be40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T10:32:03.691 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:32:03.679+0000 7f0fde7fc640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55c8d0ee14c0 0x55c8d0f6be40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.114:51778/0 (socket says 172.21.15.114:51778) 2024-04-23T10:32:03.691 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:32:03.679+0000 7f0fde7fc640 1 -- 172.21.15.114:0/232627067 learned_addr learned my addr 172.21.15.114:0/232627067 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-04-23T10:32:03.691 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:32:03.679+0000 7f0fdf7fe640 1 --2- 172.21.15.114:0/232627067 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x55c8d0ef00a0 0x55c8d0f69c30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T10:32:03.691 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:32:03.679+0000 7f0fdeffd640 1 --2- 172.21.15.114:0/232627067 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x55c8d0ee0ba0 0x55c8d0f66d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T10:32:03.691 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:32:03.679+0000 7f0fde7fc640 1 -- 172.21.15.114:0/232627067 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x55c8d0ee0ba0 msgr2=0x55c8d0f66d90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T10:32:03.691 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:32:03.679+0000 7f0fde7fc640 1 --2- 172.21.15.114:0/232627067 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x55c8d0ee0ba0 0x55c8d0f66d90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:32:03.691 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:32:03.679+0000 7f0fde7fc640 1 -- 172.21.15.114:0/232627067 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x55c8d0ef00a0 msgr2=0x55c8d0f69c30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T10:32:03.691 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:32:03.679+0000 7f0fde7fc640 1 --2- 172.21.15.114:0/232627067 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x55c8d0ef00a0 0x55c8d0f69c30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:32:03.691 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:32:03.679+0000 7f0fde7fc640 1 -- 172.21.15.114:0/232627067 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55c8d0f09070 con 0x55c8d0ee14c0 2024-04-23T10:32:03.691 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:32:03.679+0000 7f0fdf7fe640 1 --2- 172.21.15.114:0/232627067 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x55c8d0ef00a0 0x55c8d0f69c30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-23T10:32:03.691 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:32:03.679+0000 7f0fdeffd640 1 --2- 172.21.15.114:0/232627067 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x55c8d0ee0ba0 0x55c8d0f66d90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-23T10:32:03.692 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:32:03.679+0000 7f0fde7fc640 1 --2- 172.21.15.114:0/232627067 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55c8d0ee14c0 0x55c8d0f6be40 secure :-1 s=READY pgs=692 cs=0 l=1 rev1=1 crypto rx=0x7f0fd4012e10 tx=0x7f0fd4012ef0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3b6aed2964458abe server_cookie=0 in_seq=0 out_seq=0 2024-04-23T10:32:03.692 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:32:03.679+0000 7f0fc4ff9640 1 -- 172.21.15.114:0/232627067 <== mon.0 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0fd402f040 con 0x55c8d0ee14c0 2024-04-23T10:32:03.692 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:32:03.679+0000 7f0fc4ff9640 1 -- 172.21.15.114:0/232627067 <== mon.0 v2:172.21.15.114:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f0fd4011cb0 con 0x55c8d0ee14c0 2024-04-23T10:32:03.692 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:32:03.680+0000 7f0fc4ff9640 1 -- 172.21.15.114:0/232627067 <== mon.0 v2:172.21.15.114:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0fd4003850 con 0x55c8d0ee14c0 2024-04-23T10:32:03.692 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:32:03.680+0000 7f0fe81f4740 1 -- 172.21.15.114:0/232627067 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55c8d0ee14c0 msgr2=0x55c8d0f6be40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T10:32:03.692 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:32:03.680+0000 7f0fe81f4740 1 --2- 172.21.15.114:0/232627067 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55c8d0ee14c0 0x55c8d0f6be40 secure :-1 s=READY pgs=692 cs=0 l=1 rev1=1 crypto rx=0x7f0fd4012e10 tx=0x7f0fd4012ef0 comp rx=0 tx=0).stop 2024-04-23T10:32:03.692 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:32:03.680+0000 7f0fe81f4740 1 -- 172.21.15.114:0/232627067 shutdown_connections 2024-04-23T10:32:03.692 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:32:03.680+0000 7f0fe81f4740 1 --2- 172.21.15.114:0/232627067 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x55c8d0ef00a0 0x55c8d0f69c30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:32:03.692 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:32:03.680+0000 7f0fe81f4740 1 --2- 172.21.15.114:0/232627067 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55c8d0ee14c0 0x55c8d0f6be40 unknown :-1 s=CLOSED pgs=692 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:32:03.692 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:32:03.680+0000 7f0fe81f4740 1 --2- 172.21.15.114:0/232627067 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x55c8d0ee0ba0 0x55c8d0f66d90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:32:03.692 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:32:03.680+0000 7f0fe81f4740 1 -- 172.21.15.114:0/232627067 >> 172.21.15.114:0/232627067 conn(0x55c8d0edcfc0 msgr2=0x55c8d0ed80d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T10:32:03.692 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:32:03.680+0000 7f0fe81f4740 1 -- 172.21.15.114:0/232627067 shutdown_connections 2024-04-23T10:32:03.692 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:32:03.680+0000 7f0fe81f4740 1 -- 172.21.15.114:0/232627067 wait complete. 2024-04-23T10:32:03.692 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:32:03.680+0000 7f0fe81f4740 1 -- start start 2024-04-23T10:32:03.693 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:32:03.680+0000 7f0fe81f4740 1 Processor -- start 2024-04-23T10:32:03.693 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:32:03.681+0000 7f0fe81f4740 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55c8d0ee0ba0 0x55c8d0f66d90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T10:32:03.693 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:32:03.681+0000 7f0fe81f4740 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x55c8d0ee14c0 0x55c8d0f6be40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T10:32:03.693 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:32:03.681+0000 7f0fe81f4740 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x55c8d0ef00a0 0x55c8d0f69c30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T10:32:03.693 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:32:03.681+0000 7f0fe81f4740 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x55c8d0f09ad0 con 0x55c8d0ee0ba0 2024-04-23T10:32:03.693 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:32:03.681+0000 7f0fe81f4740 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 -- 0x55c8d0f672e0 con 0x55c8d0ee14c0 2024-04-23T10:32:03.693 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:32:03.681+0000 7f0fe81f4740 1 -- --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_getmap magic: 0 -- 0x55c8d0f0a180 con 0x55c8d0ef00a0 2024-04-23T10:32:03.693 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:32:03.681+0000 7f0fe81f4740 10 client.0 _ll_register_callbacks cb 0x7f0fd4009300 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-23T10:32:03.693 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:32:03.681+0000 7f0fdeffd640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55c8d0ee0ba0 0x55c8d0f66d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T10:32:03.693 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:32:03.681+0000 7f0fdeffd640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55c8d0ee0ba0 0x55c8d0f66d90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.114:51794/0 (socket says 172.21.15.114:51794) 2024-04-23T10:32:03.693 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:32:03.681+0000 7f0fdeffd640 1 -- 172.21.15.114:0/860303294 learned_addr learned my addr 172.21.15.114:0/860303294 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-04-23T10:32:03.693 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:32:03.681+0000 7f0fdf7fe640 1 --2- 172.21.15.114:0/860303294 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x55c8d0ef00a0 0x55c8d0f69c30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T10:32:03.693 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:32:03.681+0000 7f0fdeffd640 1 -- 172.21.15.114:0/860303294 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x55c8d0ef00a0 msgr2=0x55c8d0f69c30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T10:32:03.693 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:32:03.681+0000 7f0fdeffd640 1 --2- 172.21.15.114:0/860303294 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x55c8d0ef00a0 0x55c8d0f69c30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:32:03.694 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:32:03.681+0000 7f0fde7fc640 1 --2- 172.21.15.114:0/860303294 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x55c8d0ee14c0 0x55c8d0f6be40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T10:32:03.694 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:32:03.681+0000 7f0fdeffd640 1 -- 172.21.15.114:0/860303294 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x55c8d0ee14c0 msgr2=0x55c8d0f6be40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T10:32:03.694 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:32:03.681+0000 7f0fdeffd640 1 --2- 172.21.15.114:0/860303294 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x55c8d0ee14c0 0x55c8d0f6be40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:32:03.694 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:32:03.681+0000 7f0fdeffd640 1 -- 172.21.15.114:0/860303294 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55c8d0f0ac50 con 0x55c8d0ee0ba0 2024-04-23T10:32:03.694 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:32:03.681+0000 7f0fde7fc640 1 --2- 172.21.15.114:0/860303294 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x55c8d0ee14c0 0x55c8d0f6be40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-23T10:32:03.694 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:32:03.681+0000 7f0fdeffd640 1 --2- 172.21.15.114:0/860303294 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55c8d0ee0ba0 0x55c8d0f66d90 secure :-1 s=READY pgs=693 cs=0 l=1 rev1=1 crypto rx=0x7f0fd0009ca0 tx=0x7f0fd002f4c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c09aa9a2d1868686 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T10:32:03.694 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:32:03.681+0000 7f0fc2ff5640 10 client.0 ms_handle_connect on v2:172.21.15.114:3300/0 2024-04-23T10:32:03.694 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:32:03.681+0000 7f0fc2ff5640 1 -- 172.21.15.114:0/860303294 <== mon.0 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0fd0031820 con 0x55c8d0ee0ba0 2024-04-23T10:32:03.694 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:32:03.681+0000 7f0fc2ff5640 1 -- 172.21.15.114:0/860303294 <== mon.0 v2:172.21.15.114:3300/0 2 ==== osd_map(94..94 src has 1..94) ==== 8585+0+0 (secure 0 0 0) 0x7f0fd005f800 con 0x55c8d0ee0ba0 2024-04-23T10:32:03.694 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:32:03.681+0000 7f0fe81f4740 1 -- 172.21.15.114:0/860303294 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f0fd4011cb0 con 0x55c8d0ee0ba0 2024-04-23T10:32:03.694 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:32:03.681+0000 7f0fc2ff5640 1 -- 172.21.15.114:0/860303294 <== mon.0 v2:172.21.15.114:3300/0 3 ==== mon_get_version_reply(handle=1 version=30) ==== 24+0+0 (secure 0 0 0) 0x7f0fd005f2a0 con 0x55c8d0ee0ba0 2024-04-23T10:32:03.694 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:32:03.681+0000 7f0fe81f4740 10 client.16260 fetch_fsmap learned FSMap version 30 2024-04-23T10:32:03.694 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:32:03.681+0000 7f0fe81f4740 1 -- 172.21.15.114:0/860303294 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({config=0+,fsmap=30,monmap=4+}) -- 0x55c8d0eefa00 con 0x55c8d0ee0ba0 2024-04-23T10:32:03.695 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:32:03.682+0000 7f0fc2ff5640 1 -- 172.21.15.114:0/860303294 <== mon.0 v2:172.21.15.114:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f0fd4011cb0 con 0x55c8d0ee0ba0 2024-04-23T10:32:03.695 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:32:03.682+0000 7f0fc2ff5640 1 -- 172.21.15.114:0/860303294 <== mon.0 v2:172.21.15.114:3300/0 5 ==== fsmap(e 30) ==== 8338+0+0 (secure 0 0 0) 0x7f0fd006f040 con 0x55c8d0ee0ba0 2024-04-23T10:32:03.695 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:32:03.682+0000 7f0fe81f4740 10 client.16260 fetch_fsmap finished waiting for FSMap version 30 2024-04-23T10:32:03.695 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:32:03.682+0000 7f0fe81f4740 10 client.16260 resolve_mds: resolved name 'g' to daemon mds.g 2024-04-23T10:32:03.695 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:32:03.682+0000 7f0fe81f4740 20 client.16260 populate_metadata read hostname 'smithi114' 2024-04-23T10:32:03.695 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:32:03.682+0000 7f0fe81f4740 1 --2- 172.21.15.114:0/860303294 >> [v2:172.21.15.114:6838/3849796939,v1:172.21.15.114:6839/3849796939] conn(0x55c8d0f605c0 0x55c8d0f681b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T10:32:03.695 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:32:03.682+0000 7f0fe81f4740 4 client.16260 mds_command: new command op to 14634 tid=0 multi_id=0 [{"prefix": "ops", "flags": ["locks"]}] 2024-04-23T10:32:03.695 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:32:03.682+0000 7f0fe81f4740 1 -- 172.21.15.114:0/860303294 --> [v2:172.21.15.114:6838/3849796939,v1:172.21.15.114:6839/3849796939] -- command(tid 0: {"prefix": "ops", "flags": ["locks"]}) -- 0x55c8d0f76010 con 0x55c8d0f605c0 2024-04-23T10:32:03.695 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:32:03.682+0000 7f0fde7fc640 1 --2- 172.21.15.114:0/860303294 >> [v2:172.21.15.114:6838/3849796939,v1:172.21.15.114:6839/3849796939] conn(0x55c8d0f605c0 0x55c8d0f681b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T10:32:03.695 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:32:03.682+0000 7f0fde7fc640 1 --2- 172.21.15.114:0/860303294 >> [v2:172.21.15.114:6838/3849796939,v1:172.21.15.114:6839/3849796939] conn(0x55c8d0f605c0 0x55c8d0f681b0 crc :-1 s=READY pgs=14 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.? client_cookie=9b428bfd6ee19a94 server_cookie=74b5167766d96500 in_seq=0 out_seq=0 2024-04-23T10:32:03.695 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:32:03.682+0000 7f0fc2ff5640 10 client.16260 ms_handle_connect on v2:172.21.15.114:6838/3849796939 2024-04-23T10:32:03.695 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:32:03.682+0000 7f0fc2ff5640 1 -- 172.21.15.114:0/860303294 <== mds.? v2:172.21.15.114:6838/3849796939 1 ==== command_reply(tid 0: -38 ) ==== 8+0+0 (crc 0 0 0) 0x55c8d0f76010 con 0x55c8d0f605c0 2024-04-23T10:32:03.695 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:32:03.682+0000 7f0fc2ff5640 10 client.16260 handle_command_reply: tid=0 2024-04-23T10:32:03.695 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:32:03.682+0000 7f0fe81f4740 1 client.16260 shutdown 2024-04-23T10:32:03.695 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:32:03.682+0000 7f0fe81f4740 1 -- 172.21.15.114:0/860303294 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55c8d0ee0ba0 msgr2=0x55c8d0f66d90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T10:32:03.696 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:32:03.682+0000 7f0fe81f4740 1 --2- 172.21.15.114:0/860303294 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55c8d0ee0ba0 0x55c8d0f66d90 secure :-1 s=READY pgs=693 cs=0 l=1 rev1=1 crypto rx=0x7f0fd0009ca0 tx=0x7f0fd002f4c0 comp rx=0 tx=0).stop 2024-04-23T10:32:03.696 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:32:03.682+0000 7f0fe81f4740 1 -- 172.21.15.114:0/860303294 shutdown_connections 2024-04-23T10:32:03.696 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:32:03.682+0000 7f0fe81f4740 1 --2- 172.21.15.114:0/860303294 >> [v2:172.21.15.114:6838/3849796939,v1:172.21.15.114:6839/3849796939] conn(0x55c8d0f605c0 0x55c8d0f681b0 crc :-1 s=READY pgs=14 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:32:03.696 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:32:03.682+0000 7f0fe81f4740 1 --2- 172.21.15.114:0/860303294 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x55c8d0ef00a0 0x55c8d0f69c30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:32:03.696 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:32:03.682+0000 7f0fc2ff5640 0 client.16260 ms_handle_reset on v2:172.21.15.114:6838/3849796939 2024-04-23T10:32:03.696 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:32:03.682+0000 7f0fe81f4740 1 --2- 172.21.15.114:0/860303294 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x55c8d0ee14c0 0x55c8d0f6be40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:32:03.696 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:32:03.682+0000 7f0fe81f4740 1 --2- 172.21.15.114:0/860303294 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55c8d0ee0ba0 0x55c8d0f66d90 unknown :-1 s=CLOSED pgs=693 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:32:03.696 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:32:03.683+0000 7f0fe81f4740 1 -- 172.21.15.114:0/860303294 >> 172.21.15.114:0/860303294 conn(0x55c8d0edcfc0 msgr2=0x55c8d0ed80d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T10:32:03.696 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:32:03.683+0000 7f0fe81f4740 1 -- 172.21.15.114:0/860303294 shutdown_connections 2024-04-23T10:32:03.696 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:32:03.683+0000 7f0fe81f4740 1 -- 172.21.15.114:0/860303294 wait complete. 2024-04-23T10:32:03.696 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:32:03.683+0000 7f0fe81f4740 20 client.16260 trim_cache size 0 max 16384 2024-04-23T10:32:03.696 INFO:teuthology.orchestra.run.smithi114.stderr:Error ENOSYS: 2024-04-23T10:32:03.696 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:32:03.683+0000 7f0fe81f4740 1 Processor -- start 2024-04-23T10:32:03.696 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:32:03.683+0000 7f0fe81f4740 1 -- start start 2024-04-23T10:32:03.697 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:32:03.683+0000 7f0fe81f4740 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55c8d0ee0ba0 0x55c8d0f681b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T10:32:03.697 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:32:03.683+0000 7f0fe81f4740 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x55c8d0ee14c0 0x55c8d0f6be40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T10:32:03.697 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:32:03.683+0000 7f0fe81f4740 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x55c8d0ef00a0 0x55c8d0f75940 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T10:32:03.697 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:32:03.683+0000 7f0fe81f4740 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x55c8d0eea830 con 0x55c8d0ee0ba0 2024-04-23T10:32:03.697 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:32:03.683+0000 7f0fe81f4740 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 -- 0x55c8d0f757c0 con 0x55c8d0ee14c0 2024-04-23T10:32:03.697 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:32:03.683+0000 7f0fe81f4740 1 -- --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_getmap magic: 0 -- 0x55c8d0f75e90 con 0x55c8d0ef00a0 2024-04-23T10:32:03.697 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:32:03.683+0000 7f0fdeffd640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55c8d0ee0ba0 0x55c8d0f681b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T10:32:03.697 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:32:03.683+0000 7f0fdeffd640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55c8d0ee0ba0 0x55c8d0f681b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.114:51800/0 (socket says 172.21.15.114:51800) 2024-04-23T10:32:03.697 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:32:03.683+0000 7f0fdeffd640 1 -- 172.21.15.114:0/1868937387 learned_addr learned my addr 172.21.15.114:0/1868937387 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-04-23T10:32:03.697 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:32:03.683+0000 7f0fdf7fe640 1 --2- 172.21.15.114:0/1868937387 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x55c8d0ef00a0 0x55c8d0f75940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T10:32:03.697 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:32:03.683+0000 7f0fde7fc640 1 --2- 172.21.15.114:0/1868937387 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x55c8d0ee14c0 0x55c8d0f6be40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T10:32:03.697 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:32:03.684+0000 7f0fdeffd640 1 -- 172.21.15.114:0/1868937387 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x55c8d0ef00a0 msgr2=0x55c8d0f75940 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T10:32:03.697 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:32:03.684+0000 7f0fdeffd640 1 --2- 172.21.15.114:0/1868937387 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x55c8d0ef00a0 0x55c8d0f75940 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:32:03.697 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:32:03.684+0000 7f0fdeffd640 1 -- 172.21.15.114:0/1868937387 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x55c8d0ee14c0 msgr2=0x55c8d0f6be40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T10:32:03.697 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:32:03.684+0000 7f0fdeffd640 1 --2- 172.21.15.114:0/1868937387 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x55c8d0ee14c0 0x55c8d0f6be40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:32:03.698 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:32:03.684+0000 7f0fdeffd640 1 -- 172.21.15.114:0/1868937387 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55c8d0f679e0 con 0x55c8d0ee0ba0 2024-04-23T10:32:03.698 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:32:03.684+0000 7f0fdf7fe640 1 --2- 172.21.15.114:0/1868937387 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x55c8d0ef00a0 0x55c8d0f75940 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-23T10:32:03.698 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:32:03.684+0000 7f0fdeffd640 1 --2- 172.21.15.114:0/1868937387 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55c8d0ee0ba0 0x55c8d0f681b0 secure :-1 s=READY pgs=694 cs=0 l=1 rev1=1 crypto rx=0x7f0fd002a900 tx=0x7f0fd00093e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=540145686ee3e679 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T10:32:03.698 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:32:03.684+0000 7f0fc57fa640 1 -- 172.21.15.114:0/1868937387 <== mon.0 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0fd005f760 con 0x55c8d0ee0ba0 2024-04-23T10:32:03.698 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:32:03.684+0000 7f0fc57fa640 1 -- 172.21.15.114:0/1868937387 <== mon.0 v2:172.21.15.114:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f0fd0036040 con 0x55c8d0ee0ba0 2024-04-23T10:32:03.698 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:32:03.684+0000 7f0fc57fa640 1 -- 172.21.15.114:0/1868937387 <== mon.0 v2:172.21.15.114:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0fd00043e0 con 0x55c8d0ee0ba0 2024-04-23T10:32:03.698 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:32:03.684+0000 7f0fe81f4740 1 -- 172.21.15.114:0/1868937387 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55c8d0ee0ba0 msgr2=0x55c8d0f681b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T10:32:03.698 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:32:03.684+0000 7f0fe81f4740 1 --2- 172.21.15.114:0/1868937387 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55c8d0ee0ba0 0x55c8d0f681b0 secure :-1 s=READY pgs=694 cs=0 l=1 rev1=1 crypto rx=0x7f0fd002a900 tx=0x7f0fd00093e0 comp rx=0 tx=0).stop 2024-04-23T10:32:03.698 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:32:03.684+0000 7f0fe81f4740 1 -- 172.21.15.114:0/1868937387 shutdown_connections 2024-04-23T10:32:03.698 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:32:03.684+0000 7f0fe81f4740 1 --2- 172.21.15.114:0/1868937387 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x55c8d0ef00a0 0x55c8d0f75940 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:32:03.698 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:32:03.684+0000 7f0fe81f4740 1 --2- 172.21.15.114:0/1868937387 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x55c8d0ee14c0 0x55c8d0f6be40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:32:03.698 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:32:03.684+0000 7f0fe81f4740 1 --2- 172.21.15.114:0/1868937387 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55c8d0ee0ba0 0x55c8d0f681b0 unknown :-1 s=CLOSED pgs=694 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:32:03.698 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:32:03.684+0000 7f0fe81f4740 1 -- 172.21.15.114:0/1868937387 >> 172.21.15.114:0/1868937387 conn(0x55c8d0edcfc0 msgr2=0x55c8d0ee8460 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T10:32:03.698 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:32:03.684+0000 7f0fe81f4740 1 -- 172.21.15.114:0/1868937387 shutdown_connections 2024-04-23T10:32:03.699 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:32:03.684+0000 7f0fe81f4740 1 -- 172.21.15.114:0/1868937387 wait complete. 2024-04-23T10:32:03.699 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:32:03.685+0000 7f0fe81f4740 1 -- start start 2024-04-23T10:32:03.699 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:32:03.685+0000 7f0fe81f4740 1 Processor -- start 2024-04-23T10:32:03.699 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:32:03.685+0000 7f0fe81f4740 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55c8d0ee0ba0 0x55c8d0f681b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T10:32:03.699 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:32:03.685+0000 7f0fe81f4740 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x55c8d0ee14c0 0x55c8d0f75940 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T10:32:03.699 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:32:03.685+0000 7f0fe81f4740 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x55c8d0ef00a0 0x55c8d0eedbe0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T10:32:03.699 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:32:03.685+0000 7f0fe81f4740 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x55c8d0f68700 con 0x55c8d0ee0ba0 2024-04-23T10:32:03.699 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:32:03.685+0000 7f0fe81f4740 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 -- 0x55c8d0f68030 con 0x55c8d0ef00a0 2024-04-23T10:32:03.699 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:32:03.685+0000 7f0fe81f4740 1 -- --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_getmap magic: 0 -- 0x55c8d0eecd60 con 0x55c8d0ee14c0 2024-04-23T10:32:03.699 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:32:03.685+0000 7f0fe81f4740 10 client.0 _ll_register_callbacks cb 0x7f0fd0009560 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-23T10:32:03.699 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:32:03.685+0000 7f0fdeffd640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55c8d0ee0ba0 0x55c8d0f681b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T10:32:03.699 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:32:03.685+0000 7f0fdeffd640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55c8d0ee0ba0 0x55c8d0f681b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.114:51816/0 (socket says 172.21.15.114:51816) 2024-04-23T10:32:03.699 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:32:03.685+0000 7f0fdeffd640 1 -- 172.21.15.114:0/2479956990 learned_addr learned my addr 172.21.15.114:0/2479956990 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-04-23T10:32:03.699 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:32:03.685+0000 7f0fde7fc640 1 --2- 172.21.15.114:0/2479956990 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x55c8d0ee14c0 0x55c8d0f75940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T10:32:03.700 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:32:03.685+0000 7f0fdf7fe640 1 --2- 172.21.15.114:0/2479956990 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x55c8d0ef00a0 0x55c8d0eedbe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T10:32:03.700 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:32:03.685+0000 7f0fdeffd640 1 -- 172.21.15.114:0/2479956990 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x55c8d0ee14c0 msgr2=0x55c8d0f75940 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T10:32:03.700 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:32:03.685+0000 7f0fdeffd640 1 --2- 172.21.15.114:0/2479956990 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x55c8d0ee14c0 0x55c8d0f75940 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:32:03.700 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:32:03.685+0000 7f0fdeffd640 1 -- 172.21.15.114:0/2479956990 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x55c8d0ef00a0 msgr2=0x55c8d0eedbe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T10:32:03.700 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:32:03.685+0000 7f0fdeffd640 1 --2- 172.21.15.114:0/2479956990 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x55c8d0ef00a0 0x55c8d0eedbe0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:32:03.700 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:32:03.685+0000 7f0fdeffd640 1 -- 172.21.15.114:0/2479956990 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55c8d0f09510 con 0x55c8d0ee0ba0 2024-04-23T10:32:03.700 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:32:03.685+0000 7f0fde7fc640 1 --2- 172.21.15.114:0/2479956990 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x55c8d0ee14c0 0x55c8d0f75940 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-23T10:32:03.700 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:32:03.685+0000 7f0fdeffd640 1 --2- 172.21.15.114:0/2479956990 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55c8d0ee0ba0 0x55c8d0f681b0 secure :-1 s=READY pgs=695 cs=0 l=1 rev1=1 crypto rx=0x7f0fd004b040 tx=0x7f0fd00318b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4efeb305ab3ce7b7 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T10:32:03.700 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:32:03.685+0000 7f0fc2ff5640 10 client.0 ms_handle_connect on v2:172.21.15.114:3300/0 2024-04-23T10:32:03.700 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:32:03.685+0000 7f0fc2ff5640 1 -- 172.21.15.114:0/2479956990 <== mon.0 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0fd0077040 con 0x55c8d0ee0ba0 2024-04-23T10:32:03.700 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:32:03.686+0000 7f0fc2ff5640 1 -- 172.21.15.114:0/2479956990 <== mon.0 v2:172.21.15.114:3300/0 2 ==== osd_map(94..94 src has 1..94) ==== 8585+0+0 (secure 0 0 0) 0x7f0fd005f860 con 0x55c8d0ee0ba0 2024-04-23T10:32:03.700 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:32:03.686+0000 7f0fe81f4740 1 -- 172.21.15.114:0/2479956990 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f0fd0036040 con 0x55c8d0ee0ba0 2024-04-23T10:32:03.700 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:32:03.686+0000 7f0fc2ff5640 1 -- 172.21.15.114:0/2479956990 <== mon.0 v2:172.21.15.114:3300/0 3 ==== mon_get_version_reply(handle=1 version=30) ==== 24+0+0 (secure 0 0 0) 0x7f0fd0003c00 con 0x55c8d0ee0ba0 2024-04-23T10:32:03.700 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:32:03.686+0000 7f0fe81f4740 10 client.16266 fetch_fsmap learned FSMap version 30 2024-04-23T10:32:03.701 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:32:03.686+0000 7f0fe81f4740 1 -- 172.21.15.114:0/2479956990 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({config=0+,fsmap=30,monmap=4+}) -- 0x55c8d0f09510 con 0x55c8d0ee0ba0 2024-04-23T10:32:03.701 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:32:03.686+0000 7f0fc2ff5640 1 -- 172.21.15.114:0/2479956990 <== mon.0 v2:172.21.15.114:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f0fd0036040 con 0x55c8d0ee0ba0 2024-04-23T10:32:03.701 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:32:03.686+0000 7f0fc2ff5640 1 -- 172.21.15.114:0/2479956990 <== mon.0 v2:172.21.15.114:3300/0 5 ==== fsmap(e 30) ==== 8338+0+0 (secure 0 0 0) 0x7f0fd006f040 con 0x55c8d0ee0ba0 2024-04-23T10:32:03.701 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:32:03.686+0000 7f0fe81f4740 10 client.16266 fetch_fsmap finished waiting for FSMap version 30 2024-04-23T10:32:03.701 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:32:03.686+0000 7f0fe81f4740 10 client.16266 resolve_mds: resolved name 'a' to daemon mds.a 2024-04-23T10:32:03.701 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:32:03.686+0000 7f0fe81f4740 20 client.16266 populate_metadata read hostname 'smithi114' 2024-04-23T10:32:03.701 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:32:03.686+0000 7f0fe81f4740 1 --2- 172.21.15.114:0/2479956990 >> [v2:172.21.15.114:6840/3544523891,v1:172.21.15.114:6841/3544523891] conn(0x55c8d0eeac10 0x55c8d0f6b880 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T10:32:03.701 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:32:03.686+0000 7f0fe81f4740 4 client.16266 mds_command: new command op to 14646 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-23T10:32:03.701 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:32:03.686+0000 7f0fe81f4740 1 -- 172.21.15.114:0/2479956990 --> [v2:172.21.15.114:6840/3544523891,v1:172.21.15.114:6841/3544523891] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55c8d0eed8b0 con 0x55c8d0eeac10 2024-04-23T10:32:03.701 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:32:03.686+0000 7f0fde7fc640 1 --2- 172.21.15.114:0/2479956990 >> [v2:172.21.15.114:6840/3544523891,v1:172.21.15.114:6841/3544523891] conn(0x55c8d0eeac10 0x55c8d0f6b880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T10:32:03.701 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:32:03.686+0000 7f0fc2ff5640 10 client.16266 ms_handle_connect on v2:172.21.15.114:6840/3544523891 2024-04-23T10:32:03.701 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:32:03.686+0000 7f0fde7fc640 1 --2- 172.21.15.114:0/2479956990 >> [v2:172.21.15.114:6840/3544523891,v1:172.21.15.114:6841/3544523891] conn(0x55c8d0eeac10 0x55c8d0f6b880 crc :-1 s=READY pgs=13 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.? client_cookie=fb810c9fbcc75a5e server_cookie=e157f68fa69c6d43 in_seq=0 out_seq=0 2024-04-23T10:32:03.701 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:32:03.687+0000 7f0fc2ff5640 1 -- 172.21.15.114:0/2479956990 <== mds.? v2:172.21.15.114:6840/3544523891 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23115 (crc 0 0 0) 0x55c8d0eed8b0 con 0x55c8d0eeac10 2024-04-23T10:32:03.701 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:32:03.687+0000 7f0fc2ff5640 10 client.16266 handle_command_reply: tid=0 2024-04-23T10:32:03.702 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:32:03.687+0000 7f0fe81f4740 1 client.16266 shutdown 2024-04-23T10:32:03.702 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:32:03.687+0000 7f0fe81f4740 1 -- 172.21.15.114:0/2479956990 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55c8d0ee0ba0 msgr2=0x55c8d0f681b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T10:32:03.702 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:32:03.687+0000 7f0fe81f4740 1 --2- 172.21.15.114:0/2479956990 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55c8d0ee0ba0 0x55c8d0f681b0 secure :-1 s=READY pgs=695 cs=0 l=1 rev1=1 crypto rx=0x7f0fd004b040 tx=0x7f0fd00318b0 comp rx=0 tx=0).stop 2024-04-23T10:32:03.702 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:32:03.687+0000 7f0fe81f4740 1 -- 172.21.15.114:0/2479956990 shutdown_connections 2024-04-23T10:32:03.702 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:32:03.687+0000 7f0fe81f4740 1 --2- 172.21.15.114:0/2479956990 >> [v2:172.21.15.114:6840/3544523891,v1:172.21.15.114:6841/3544523891] conn(0x55c8d0eeac10 0x55c8d0f6b880 crc :-1 s=READY pgs=13 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:32:03.702 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:32:03.687+0000 7f0fe81f4740 1 --2- 172.21.15.114:0/2479956990 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x55c8d0ef00a0 0x55c8d0eedbe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:32:03.702 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:32:03.687+0000 7f0fe81f4740 1 --2- 172.21.15.114:0/2479956990 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x55c8d0ee14c0 0x55c8d0f75940 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:32:03.702 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:32:03.687+0000 7f0fc2ff5640 0 client.16266 ms_handle_reset on v2:172.21.15.114:6840/3544523891 2024-04-23T10:32:03.702 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:32:03.687+0000 7f0fe81f4740 1 --2- 172.21.15.114:0/2479956990 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55c8d0ee0ba0 0x55c8d0f681b0 unknown :-1 s=CLOSED pgs=695 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:32:03.702 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:32:03.687+0000 7f0fe81f4740 1 -- 172.21.15.114:0/2479956990 >> 172.21.15.114:0/2479956990 conn(0x55c8d0edcfc0 msgr2=0x55c8d0ed80d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T10:32:03.702 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:32:03.687+0000 7f0fe81f4740 1 -- 172.21.15.114:0/2479956990 shutdown_connections 2024-04-23T10:32:03.702 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:32:03.687+0000 7f0fe81f4740 1 -- 172.21.15.114:0/2479956990 wait complete. 2024-04-23T10:32:03.702 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:32:03.687+0000 7f0fe81f4740 20 client.16266 trim_cache size 0 max 16384 2024-04-23T10:32:03.714 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:32:03.702+0000 7f0fe81f4740 1 Processor -- start 2024-04-23T10:32:03.715 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:32:03.702+0000 7f0fe81f4740 1 -- start start 2024-04-23T10:32:03.715 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:32:03.703+0000 7f0fe81f4740 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55c8d0ee0ba0 0x55c8d0eedbe0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T10:32:03.715 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:32:03.703+0000 7f0fe81f4740 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x55c8d0ee14c0 0x55c8d0f6be40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T10:32:03.715 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:32:03.703+0000 7f0fe81f4740 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x55c8d0ef00a0 0x55c8d0f02cc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T10:32:03.715 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:32:03.703+0000 7f0fe81f4740 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x55c8d0f619b0 con 0x55c8d0ee0ba0 2024-04-23T10:32:03.715 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:32:03.703+0000 7f0fe81f4740 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 -- 0x55c8d0efbb00 con 0x55c8d0ef00a0 2024-04-23T10:32:03.715 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:32:03.703+0000 7f0fe81f4740 1 -- --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_getmap magic: 0 -- 0x55c8d0efc180 con 0x55c8d0ee14c0 2024-04-23T10:32:03.715 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:32:03.703+0000 7f0fdeffd640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55c8d0ee0ba0 0x55c8d0eedbe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T10:32:03.715 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:32:03.703+0000 7f0fdeffd640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55c8d0ee0ba0 0x55c8d0eedbe0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.114:51824/0 (socket says 172.21.15.114:51824) 2024-04-23T10:32:03.715 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:32:03.703+0000 7f0fdeffd640 1 -- 172.21.15.114:0/315929758 learned_addr learned my addr 172.21.15.114:0/315929758 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-04-23T10:32:03.715 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:32:03.703+0000 7f0fde7fc640 1 --2- 172.21.15.114:0/315929758 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x55c8d0ee14c0 0x55c8d0f6be40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T10:32:03.715 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:32:03.703+0000 7f0fdeffd640 1 -- 172.21.15.114:0/315929758 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x55c8d0ee14c0 msgr2=0x55c8d0f6be40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T10:32:03.715 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:32:03.703+0000 7f0fdeffd640 1 --2- 172.21.15.114:0/315929758 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x55c8d0ee14c0 0x55c8d0f6be40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:32:03.716 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:32:03.703+0000 7f0fdeffd640 1 -- 172.21.15.114:0/315929758 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x55c8d0ef00a0 msgr2=0x55c8d0f02cc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T10:32:03.716 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:32:03.703+0000 7f0fdeffd640 1 --2- 172.21.15.114:0/315929758 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x55c8d0ef00a0 0x55c8d0f02cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:32:03.716 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:32:03.703+0000 7f0fdeffd640 1 -- 172.21.15.114:0/315929758 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55c8d0f679e0 con 0x55c8d0ee0ba0 2024-04-23T10:32:03.716 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:32:03.703+0000 7f0fdeffd640 1 --2- 172.21.15.114:0/315929758 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55c8d0ee0ba0 0x55c8d0eedbe0 secure :-1 s=READY pgs=696 cs=0 l=1 rev1=1 crypto rx=0x7f0fd005f220 tx=0x7f0fd002f4c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b556df0bdf4264ab server_cookie=0 in_seq=0 out_seq=0 2024-04-23T10:32:03.716 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:32:03.703+0000 7f0fc4ff9640 1 -- 172.21.15.114:0/315929758 <== mon.0 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0fd0077040 con 0x55c8d0ee0ba0 2024-04-23T10:32:03.716 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:32:03.703+0000 7f0fc4ff9640 1 -- 172.21.15.114:0/315929758 <== mon.0 v2:172.21.15.114:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f0fd0036070 con 0x55c8d0ee0ba0 2024-04-23T10:32:03.716 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:32:03.703+0000 7f0fc4ff9640 1 -- 172.21.15.114:0/315929758 <== mon.0 v2:172.21.15.114:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0fd005f2c0 con 0x55c8d0ee0ba0 2024-04-23T10:32:03.716 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:32:03.704+0000 7f0fe81f4740 1 -- 172.21.15.114:0/315929758 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55c8d0ee0ba0 msgr2=0x55c8d0eedbe0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T10:32:03.716 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:32:03.704+0000 7f0fe81f4740 1 --2- 172.21.15.114:0/315929758 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55c8d0ee0ba0 0x55c8d0eedbe0 secure :-1 s=READY pgs=696 cs=0 l=1 rev1=1 crypto rx=0x7f0fd005f220 tx=0x7f0fd002f4c0 comp rx=0 tx=0).stop 2024-04-23T10:32:03.716 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:32:03.704+0000 7f0fe81f4740 1 -- 172.21.15.114:0/315929758 shutdown_connections 2024-04-23T10:32:03.716 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:32:03.704+0000 7f0fe81f4740 1 --2- 172.21.15.114:0/315929758 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x55c8d0ef00a0 0x55c8d0f02cc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:32:03.716 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:32:03.704+0000 7f0fe81f4740 1 --2- 172.21.15.114:0/315929758 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x55c8d0ee14c0 0x55c8d0f6be40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:32:03.716 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:32:03.704+0000 7f0fe81f4740 1 --2- 172.21.15.114:0/315929758 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55c8d0ee0ba0 0x55c8d0eedbe0 unknown :-1 s=CLOSED pgs=696 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:32:03.716 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:32:03.704+0000 7f0fe81f4740 1 -- 172.21.15.114:0/315929758 >> 172.21.15.114:0/315929758 conn(0x55c8d0edcfc0 msgr2=0x55c8d0ee8460 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T10:32:03.716 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:32:03.704+0000 7f0fe81f4740 1 -- 172.21.15.114:0/315929758 shutdown_connections 2024-04-23T10:32:03.717 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:32:03.704+0000 7f0fe81f4740 1 -- 172.21.15.114:0/315929758 wait complete. 2024-04-23T10:32:03.717 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:32:03.704+0000 7f0fe81f4740 1 -- start start 2024-04-23T10:32:03.717 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:32:03.704+0000 7f0fe81f4740 1 Processor -- start 2024-04-23T10:32:03.717 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:32:03.704+0000 7f0fe81f4740 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x55c8d0ee0ba0 0x55c8d0eedbe0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T10:32:03.717 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:32:03.704+0000 7f0fe81f4740 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x55c8d0ee14c0 0x55c8d0f02cc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T10:32:03.717 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:32:03.704+0000 7f0fe81f4740 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55c8d0ef00a0 0x55c8d0f6be40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T10:32:03.717 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:32:03.704+0000 7f0fe81f4740 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x55c8d0f69d10 con 0x55c8d0ef00a0 2024-04-23T10:32:03.717 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:32:03.704+0000 7f0fe81f4740 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 -- 0x55c8d0ee00c0 con 0x55c8d0ee14c0 2024-04-23T10:32:03.717 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:32:03.704+0000 7f0fe81f4740 1 -- --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_getmap magic: 0 -- 0x55c8d0f6a620 con 0x55c8d0ee0ba0 2024-04-23T10:32:03.717 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:32:03.704+0000 7f0fe81f4740 10 client.0 _ll_register_callbacks cb 0x7f0fd002af40 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-23T10:32:03.717 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:32:03.705+0000 7f0fdf7fe640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55c8d0ef00a0 0x55c8d0f6be40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T10:32:03.717 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:32:03.705+0000 7f0fdf7fe640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55c8d0ef00a0 0x55c8d0f6be40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.114:51834/0 (socket says 172.21.15.114:51834) 2024-04-23T10:32:03.717 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:32:03.705+0000 7f0fdf7fe640 1 -- 172.21.15.114:0/2685447735 learned_addr learned my addr 172.21.15.114:0/2685447735 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-04-23T10:32:03.717 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:32:03.705+0000 7f0fdeffd640 1 --2- 172.21.15.114:0/2685447735 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x55c8d0ee0ba0 0x55c8d0eedbe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T10:32:03.717 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:32:03.705+0000 7f0fde7fc640 1 --2- 172.21.15.114:0/2685447735 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x55c8d0ee14c0 0x55c8d0f02cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T10:32:03.718 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:32:03.705+0000 7f0fdf7fe640 1 -- 172.21.15.114:0/2685447735 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x55c8d0ee0ba0 msgr2=0x55c8d0eedbe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T10:32:03.718 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:32:03.705+0000 7f0fdf7fe640 1 --2- 172.21.15.114:0/2685447735 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x55c8d0ee0ba0 0x55c8d0eedbe0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:32:03.718 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:32:03.705+0000 7f0fdf7fe640 1 -- 172.21.15.114:0/2685447735 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x55c8d0ee14c0 msgr2=0x55c8d0f02cc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T10:32:03.718 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:32:03.705+0000 7f0fdf7fe640 1 --2- 172.21.15.114:0/2685447735 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x55c8d0ee14c0 0x55c8d0f02cc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:32:03.718 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:32:03.705+0000 7f0fdf7fe640 1 -- 172.21.15.114:0/2685447735 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55c8d0f67d80 con 0x55c8d0ef00a0 2024-04-23T10:32:03.718 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:32:03.705+0000 7f0fdf7fe640 1 --2- 172.21.15.114:0/2685447735 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55c8d0ef00a0 0x55c8d0f6be40 secure :-1 s=READY pgs=697 cs=0 l=1 rev1=1 crypto rx=0x7f0fd809c2b0 tx=0x7f0fd8000bc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7e1b6c9c7c834456 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T10:32:03.718 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:32:03.705+0000 7f0fc2ff5640 10 client.0 ms_handle_connect on v2:172.21.15.114:3300/0 2024-04-23T10:32:03.718 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:32:03.705+0000 7f0fc2ff5640 1 -- 172.21.15.114:0/2685447735 <== mon.0 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0fd8014020 con 0x55c8d0ef00a0 2024-04-23T10:32:03.718 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:32:03.705+0000 7f0fc2ff5640 1 -- 172.21.15.114:0/2685447735 <== mon.0 v2:172.21.15.114:3300/0 2 ==== osd_map(94..94 src has 1..94) ==== 8585+0+0 (secure 0 0 0) 0x7f0fd809ba00 con 0x55c8d0ef00a0 2024-04-23T10:32:03.718 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:32:03.705+0000 7f0fe81f4740 1 -- 172.21.15.114:0/2685447735 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f0fd0036070 con 0x55c8d0ef00a0 2024-04-23T10:32:03.718 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:32:03.705+0000 7f0fc2ff5640 1 -- 172.21.15.114:0/2685447735 <== mon.0 v2:172.21.15.114:3300/0 3 ==== mon_get_version_reply(handle=1 version=30) ==== 24+0+0 (secure 0 0 0) 0x7f0fd8020040 con 0x55c8d0ef00a0 2024-04-23T10:32:03.718 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:32:03.705+0000 7f0fe81f4740 10 client.16272 fetch_fsmap learned FSMap version 30 2024-04-23T10:32:03.718 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:32:03.705+0000 7f0fe81f4740 1 -- 172.21.15.114:0/2685447735 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({config=0+,fsmap=30,monmap=4+}) -- 0x55c8d0eed310 con 0x55c8d0ef00a0 2024-04-23T10:32:03.718 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:32:03.705+0000 7f0fc2ff5640 1 -- 172.21.15.114:0/2685447735 <== mon.0 v2:172.21.15.114:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f0fd0036070 con 0x55c8d0ef00a0 2024-04-23T10:32:03.719 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:32:03.705+0000 7f0fc2ff5640 1 -- 172.21.15.114:0/2685447735 <== mon.0 v2:172.21.15.114:3300/0 5 ==== fsmap(e 30) ==== 8338+0+0 (secure 0 0 0) 0x7f0fd80d0070 con 0x55c8d0ef00a0 2024-04-23T10:32:03.719 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:32:03.706+0000 7f0fe81f4740 10 client.16272 fetch_fsmap finished waiting for FSMap version 30 2024-04-23T10:32:03.719 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:32:03.706+0000 7f0fe81f4740 10 client.16272 resolve_mds: resolved name 'a' to daemon mds.a 2024-04-23T10:32:03.719 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:32:03.706+0000 7f0fe81f4740 20 client.16272 populate_metadata read hostname 'smithi114' 2024-04-23T10:32:03.719 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:32:03.706+0000 7f0fe81f4740 1 --2- 172.21.15.114:0/2685447735 >> [v2:172.21.15.114:6840/3544523891,v1:172.21.15.114:6841/3544523891] conn(0x55c8d0f0a100 0x55c8d0f61b30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T10:32:03.719 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:32:03.706+0000 7f0fe81f4740 4 client.16272 mds_command: new command op to 14646 tid=0 multi_id=0 [{"prefix": "ops", "flags": ["locks"]}] 2024-04-23T10:32:03.719 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:32:03.706+0000 7f0fe81f4740 1 -- 172.21.15.114:0/2685447735 --> [v2:172.21.15.114:6840/3544523891,v1:172.21.15.114:6841/3544523891] -- command(tid 0: {"prefix": "ops", "flags": ["locks"]}) -- 0x55c8d0f67820 con 0x55c8d0f0a100 2024-04-23T10:32:03.719 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:32:03.706+0000 7f0fdeffd640 1 --2- 172.21.15.114:0/2685447735 >> [v2:172.21.15.114:6840/3544523891,v1:172.21.15.114:6841/3544523891] conn(0x55c8d0f0a100 0x55c8d0f61b30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T10:32:03.719 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:32:03.706+0000 7f0fdeffd640 1 --2- 172.21.15.114:0/2685447735 >> [v2:172.21.15.114:6840/3544523891,v1:172.21.15.114:6841/3544523891] conn(0x55c8d0f0a100 0x55c8d0f61b30 crc :-1 s=READY pgs=14 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.? client_cookie=265c7896687089a7 server_cookie=ac04754781503ace in_seq=0 out_seq=0 2024-04-23T10:32:03.719 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:32:03.706+0000 7f0fc2ff5640 10 client.16272 ms_handle_connect on v2:172.21.15.114:6840/3544523891 2024-04-23T10:32:03.719 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:32:03.706+0000 7f0fc2ff5640 1 -- 172.21.15.114:0/2685447735 <== mds.? v2:172.21.15.114:6840/3544523891 1 ==== command_reply(tid 0: -38 ) ==== 8+0+0 (crc 0 0 0) 0x7f0fd0003d00 con 0x55c8d0f0a100 2024-04-23T10:32:03.719 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:32:03.706+0000 7f0fc2ff5640 10 client.16272 handle_command_reply: tid=0 2024-04-23T10:32:03.719 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:32:03.706+0000 7f0fe81f4740 1 client.16272 shutdown 2024-04-23T10:32:03.719 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:32:03.706+0000 7f0fe81f4740 1 -- 172.21.15.114:0/2685447735 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55c8d0ef00a0 msgr2=0x55c8d0f6be40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T10:32:03.719 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:32:03.706+0000 7f0fe81f4740 1 --2- 172.21.15.114:0/2685447735 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55c8d0ef00a0 0x55c8d0f6be40 secure :-1 s=READY pgs=697 cs=0 l=1 rev1=1 crypto rx=0x7f0fd809c2b0 tx=0x7f0fd8000bc0 comp rx=0 tx=0).stop 2024-04-23T10:32:03.720 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:32:03.706+0000 7f0fe81f4740 1 -- 172.21.15.114:0/2685447735 shutdown_connections 2024-04-23T10:32:03.720 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:32:03.706+0000 7f0fe81f4740 1 --2- 172.21.15.114:0/2685447735 >> [v2:172.21.15.114:6840/3544523891,v1:172.21.15.114:6841/3544523891] conn(0x55c8d0f0a100 0x55c8d0f61b30 crc :-1 s=READY pgs=14 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:32:03.720 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:32:03.706+0000 7f0fe81f4740 1 --2- 172.21.15.114:0/2685447735 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55c8d0ef00a0 0x55c8d0f6be40 unknown :-1 s=CLOSED pgs=697 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:32:03.720 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:32:03.706+0000 7f0fe81f4740 1 --2- 172.21.15.114:0/2685447735 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x55c8d0ee14c0 0x55c8d0f02cc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:32:03.720 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:32:03.706+0000 7f0fe81f4740 1 --2- 172.21.15.114:0/2685447735 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x55c8d0ee0ba0 0x55c8d0eedbe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:32:03.720 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:32:03.706+0000 7f0fe81f4740 1 -- 172.21.15.114:0/2685447735 >> 172.21.15.114:0/2685447735 conn(0x55c8d0edcfc0 msgr2=0x55c8d0eec930 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T10:32:03.720 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:32:03.706+0000 7f0fc2ff5640 0 client.16272 ms_handle_reset on v2:172.21.15.114:6840/3544523891 2024-04-23T10:32:03.720 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:32:03.706+0000 7f0fe81f4740 1 -- 172.21.15.114:0/2685447735 shutdown_connections 2024-04-23T10:32:03.720 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:32:03.706+0000 7f0fe81f4740 1 -- 172.21.15.114:0/2685447735 wait complete. 2024-04-23T10:32:03.720 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:32:03.706+0000 7f0fe81f4740 20 client.16272 trim_cache size 0 max 16384 2024-04-23T10:32:03.720 INFO:teuthology.orchestra.run.smithi114.stderr:Error ENOSYS: 2024-04-23T10:32:03.720 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:32:03.707+0000 7f0fe81f4740 1 Processor -- start 2024-04-23T10:32:03.720 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:32:03.707+0000 7f0fe81f4740 1 -- start start 2024-04-23T10:32:03.720 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:32:03.707+0000 7f0fe81f4740 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55c8d0ee0ba0 0x55c8d0f02cc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T10:32:03.721 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:32:03.707+0000 7f0fe81f4740 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x55c8d0ee14c0 0x55c8d0f6be10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T10:32:03.721 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:32:03.707+0000 7f0fe81f4740 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x55c8d0ef00a0 0x55c8d0f66d90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T10:32:03.721 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:32:03.707+0000 7f0fe81f4740 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x55c8d0f6ba50 con 0x55c8d0ee0ba0 2024-04-23T10:32:03.721 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:32:03.707+0000 7f0fe81f4740 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 -- 0x55c8d0f676a0 con 0x55c8d0ee14c0 2024-04-23T10:32:03.721 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:32:03.707+0000 7f0fe81f4740 1 -- --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_getmap magic: 0 -- 0x55c8d0edff50 con 0x55c8d0ef00a0 2024-04-23T10:32:03.721 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:32:03.707+0000 7f0fdeffd640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55c8d0ee0ba0 0x55c8d0f02cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T10:32:03.721 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:32:03.707+0000 7f0fdeffd640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55c8d0ee0ba0 0x55c8d0f02cc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.114:51838/0 (socket says 172.21.15.114:51838) 2024-04-23T10:32:03.721 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:32:03.707+0000 7f0fdeffd640 1 -- 172.21.15.114:0/3144179561 learned_addr learned my addr 172.21.15.114:0/3144179561 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-04-23T10:32:03.721 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:32:03.707+0000 7f0fdf7fe640 1 --2- 172.21.15.114:0/3144179561 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x55c8d0ef00a0 0x55c8d0f66d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T10:32:03.721 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:32:03.707+0000 7f0fde7fc640 1 --2- 172.21.15.114:0/3144179561 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x55c8d0ee14c0 0x55c8d0f6be10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T10:32:03.721 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:32:03.707+0000 7f0fdeffd640 1 -- 172.21.15.114:0/3144179561 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x55c8d0ef00a0 msgr2=0x55c8d0f66d90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T10:32:03.721 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:32:03.707+0000 7f0fdeffd640 1 --2- 172.21.15.114:0/3144179561 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x55c8d0ef00a0 0x55c8d0f66d90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:32:03.721 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:32:03.707+0000 7f0fdeffd640 1 -- 172.21.15.114:0/3144179561 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x55c8d0ee14c0 msgr2=0x55c8d0f6be10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T10:32:03.721 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:32:03.707+0000 7f0fdeffd640 1 --2- 172.21.15.114:0/3144179561 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x55c8d0ee14c0 0x55c8d0f6be10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:32:03.721 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:32:03.707+0000 7f0fdeffd640 1 -- 172.21.15.114:0/3144179561 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55c8d0f6bbd0 con 0x55c8d0ee0ba0 2024-04-23T10:32:03.722 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:32:03.707+0000 7f0fdeffd640 1 --2- 172.21.15.114:0/3144179561 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55c8d0ee0ba0 0x55c8d0f02cc0 secure :-1 s=READY pgs=698 cs=0 l=1 rev1=1 crypto rx=0x7f0fd005f220 tx=0x7f0fd003e040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6e8d3f423c34c8b8 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T10:32:03.722 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:32:03.707+0000 7f0fc57fa640 1 -- 172.21.15.114:0/3144179561 <== mon.0 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0fd0077040 con 0x55c8d0ee0ba0 2024-04-23T10:32:03.722 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:32:03.708+0000 7f0fc57fa640 1 -- 172.21.15.114:0/3144179561 <== mon.0 v2:172.21.15.114:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f0fd0036040 con 0x55c8d0ee0ba0 2024-04-23T10:32:03.722 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:32:03.708+0000 7f0fc57fa640 1 -- 172.21.15.114:0/3144179561 <== mon.0 v2:172.21.15.114:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0fd00043e0 con 0x55c8d0ee0ba0 2024-04-23T10:32:03.722 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:32:03.708+0000 7f0fe81f4740 1 -- 172.21.15.114:0/3144179561 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55c8d0ee0ba0 msgr2=0x55c8d0f02cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T10:32:03.722 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:32:03.708+0000 7f0fe81f4740 1 --2- 172.21.15.114:0/3144179561 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55c8d0ee0ba0 0x55c8d0f02cc0 secure :-1 s=READY pgs=698 cs=0 l=1 rev1=1 crypto rx=0x7f0fd005f220 tx=0x7f0fd003e040 comp rx=0 tx=0).stop 2024-04-23T10:32:03.722 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:32:03.708+0000 7f0fe81f4740 1 -- 172.21.15.114:0/3144179561 shutdown_connections 2024-04-23T10:32:03.722 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:32:03.708+0000 7f0fe81f4740 1 --2- 172.21.15.114:0/3144179561 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x55c8d0ef00a0 0x55c8d0f66d90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:32:03.722 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:32:03.708+0000 7f0fe81f4740 1 --2- 172.21.15.114:0/3144179561 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x55c8d0ee14c0 0x55c8d0f6be10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:32:03.722 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:32:03.708+0000 7f0fe81f4740 1 --2- 172.21.15.114:0/3144179561 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55c8d0ee0ba0 0x55c8d0f02cc0 unknown :-1 s=CLOSED pgs=698 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:32:03.722 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:32:03.708+0000 7f0fe81f4740 1 -- 172.21.15.114:0/3144179561 >> 172.21.15.114:0/3144179561 conn(0x55c8d0edcfc0 msgr2=0x55c8d0ee8460 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T10:32:03.722 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:32:03.708+0000 7f0fe81f4740 1 -- 172.21.15.114:0/3144179561 shutdown_connections 2024-04-23T10:32:03.722 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:32:03.708+0000 7f0fe81f4740 1 -- 172.21.15.114:0/3144179561 wait complete. 2024-04-23T10:32:03.722 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:32:03.708+0000 7f0fe81f4740 1 -- start start 2024-04-23T10:32:03.723 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:32:03.708+0000 7f0fe81f4740 1 Processor -- start 2024-04-23T10:32:03.723 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:32:03.709+0000 7f0fe81f4740 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x55c8d0ee0ba0 0x55c8d0f02cc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T10:32:03.723 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:32:03.709+0000 7f0fe81f4740 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55c8d0ee14c0 0x55c8d0f69db0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T10:32:03.723 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:32:03.709+0000 7f0fe81f4740 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x55c8d0ef00a0 0x55c8d0eea600 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T10:32:03.723 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:32:03.709+0000 7f0fe81f4740 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x55c8d0ed84b0 con 0x55c8d0ee14c0 2024-04-23T10:32:03.723 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:32:03.709+0000 7f0fe81f4740 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 -- 0x55c8d0ee7810 con 0x55c8d0ee0ba0 2024-04-23T10:32:03.723 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:32:03.709+0000 7f0fe81f4740 1 -- --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_getmap magic: 0 -- 0x55c8d0edc300 con 0x55c8d0ef00a0 2024-04-23T10:32:03.723 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:32:03.709+0000 7f0fe81f4740 10 client.0 _ll_register_callbacks cb 0x7f0fd0009b20 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-23T10:32:03.723 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:32:03.709+0000 7f0fde7fc640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55c8d0ee14c0 0x55c8d0f69db0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T10:32:03.723 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:32:03.709+0000 7f0fde7fc640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55c8d0ee14c0 0x55c8d0f69db0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.114:51852/0 (socket says 172.21.15.114:51852) 2024-04-23T10:32:03.723 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:32:03.709+0000 7f0fde7fc640 1 -- 172.21.15.114:0/1677909263 learned_addr learned my addr 172.21.15.114:0/1677909263 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-04-23T10:32:03.723 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:32:03.709+0000 7f0fdf7fe640 1 --2- 172.21.15.114:0/1677909263 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x55c8d0ef00a0 0x55c8d0eea600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T10:32:03.723 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:32:03.709+0000 7f0fdeffd640 1 --2- 172.21.15.114:0/1677909263 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x55c8d0ee0ba0 0x55c8d0f02cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T10:32:03.723 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:32:03.709+0000 7f0fde7fc640 1 -- 172.21.15.114:0/1677909263 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x55c8d0ef00a0 msgr2=0x55c8d0eea600 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T10:32:03.724 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:32:03.709+0000 7f0fde7fc640 1 --2- 172.21.15.114:0/1677909263 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x55c8d0ef00a0 0x55c8d0eea600 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:32:03.724 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:32:03.709+0000 7f0fde7fc640 1 -- 172.21.15.114:0/1677909263 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x55c8d0ee0ba0 msgr2=0x55c8d0f02cc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T10:32:03.724 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:32:03.709+0000 7f0fde7fc640 1 --2- 172.21.15.114:0/1677909263 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x55c8d0ee0ba0 0x55c8d0f02cc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:32:03.724 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:32:03.709+0000 7f0fde7fc640 1 -- 172.21.15.114:0/1677909263 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55c8d0f67380 con 0x55c8d0ee14c0 2024-04-23T10:32:03.724 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:32:03.709+0000 7f0fdf7fe640 1 --2- 172.21.15.114:0/1677909263 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x55c8d0ef00a0 0x55c8d0eea600 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-23T10:32:03.724 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:32:03.709+0000 7f0fde7fc640 1 --2- 172.21.15.114:0/1677909263 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55c8d0ee14c0 0x55c8d0f69db0 secure :-1 s=READY pgs=699 cs=0 l=1 rev1=1 crypto rx=0x7f0fd40117f0 tx=0x7f0fd4009940 comp rx=0 tx=0).ready entity=mon.0 client_cookie=be8982f871c7d5f7 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T10:32:03.724 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:32:03.709+0000 7f0fc2ff5640 10 client.0 ms_handle_connect on v2:172.21.15.114:3300/0 2024-04-23T10:32:03.724 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:32:03.709+0000 7f0fc2ff5640 1 -- 172.21.15.114:0/1677909263 <== mon.0 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0fd402f040 con 0x55c8d0ee14c0 2024-04-23T10:32:03.724 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:32:03.709+0000 7f0fc2ff5640 1 -- 172.21.15.114:0/1677909263 <== mon.0 v2:172.21.15.114:3300/0 2 ==== osd_map(94..94 src has 1..94) ==== 8585+0+0 (secure 0 0 0) 0x7f0fd40042c0 con 0x55c8d0ee14c0 2024-04-23T10:32:03.724 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:32:03.709+0000 7f0fe81f4740 1 -- 172.21.15.114:0/1677909263 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f0fd0036040 con 0x55c8d0ee14c0 2024-04-23T10:32:03.724 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:32:03.709+0000 7f0fc2ff5640 1 -- 172.21.15.114:0/1677909263 <== mon.0 v2:172.21.15.114:3300/0 3 ==== mon_get_version_reply(handle=1 version=30) ==== 24+0+0 (secure 0 0 0) 0x7f0fd4011cb0 con 0x55c8d0ee14c0 2024-04-23T10:32:03.724 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:32:03.709+0000 7f0fe81f4740 10 client.16278 fetch_fsmap learned FSMap version 30 2024-04-23T10:32:03.724 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:32:03.709+0000 7f0fe81f4740 1 -- 172.21.15.114:0/1677909263 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({config=0+,fsmap=30,monmap=4+}) -- 0x55c8d0ee01f0 con 0x55c8d0ee14c0 2024-04-23T10:32:03.724 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:32:03.710+0000 7f0fc2ff5640 1 -- 172.21.15.114:0/1677909263 <== mon.0 v2:172.21.15.114:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f0fd0036040 con 0x55c8d0ee14c0 2024-04-23T10:32:03.724 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:32:03.710+0000 7f0fc2ff5640 1 -- 172.21.15.114:0/1677909263 <== mon.0 v2:172.21.15.114:3300/0 5 ==== fsmap(e 30) ==== 8338+0+0 (secure 0 0 0) 0x7f0fd40364e0 con 0x55c8d0ee14c0 2024-04-23T10:32:03.724 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:32:03.710+0000 7f0fe81f4740 10 client.16278 fetch_fsmap finished waiting for FSMap version 30 2024-04-23T10:32:03.725 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:32:03.710+0000 7f0fe81f4740 10 client.16278 resolve_mds: resolved name 'l' to daemon mds.l 2024-04-23T10:32:03.725 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:32:03.710+0000 7f0fe81f4740 20 client.16278 populate_metadata read hostname 'smithi114' 2024-04-23T10:32:03.725 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:32:03.710+0000 7f0fe81f4740 1 --2- 172.21.15.114:0/1677909263 >> [v2:172.21.15.204:6832/3866561754,v1:172.21.15.204:6833/3866561754] conn(0x55c8d0f7c050 0x55c8d0f689a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T10:32:03.725 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:32:03.710+0000 7f0fe81f4740 4 client.16278 mds_command: new command op to 24385 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-23T10:32:03.725 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:32:03.710+0000 7f0fe81f4740 1 -- 172.21.15.114:0/1677909263 --> [v2:172.21.15.204:6832/3866561754,v1:172.21.15.204:6833/3866561754] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55c8d0ee76f0 con 0x55c8d0f7c050 2024-04-23T10:32:03.725 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:32:03.710+0000 7f0fdeffd640 1 --2- 172.21.15.114:0/1677909263 >> [v2:172.21.15.204:6832/3866561754,v1:172.21.15.204:6833/3866561754] conn(0x55c8d0f7c050 0x55c8d0f689a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T10:32:03.725 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:32:03.711+0000 7f0fc2ff5640 10 client.16278 ms_handle_connect on v2:172.21.15.204:6832/3866561754 2024-04-23T10:32:03.725 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:32:03.711+0000 7f0fdeffd640 1 --2- 172.21.15.114:0/1677909263 >> [v2:172.21.15.204:6832/3866561754,v1:172.21.15.204:6833/3866561754] conn(0x55c8d0f7c050 0x55c8d0f689a0 crc :-1 s=READY pgs=13 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.? client_cookie=59bcb9bb839a0558 server_cookie=6b4bad1a3d760dd1 in_seq=0 out_seq=0 2024-04-23T10:32:03.725 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:32:03.711+0000 7f0fc2ff5640 1 -- 172.21.15.114:0/1677909263 <== mds.? v2:172.21.15.204:6832/3866561754 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23115 (crc 0 0 0) 0x55c8d0ee76f0 con 0x55c8d0f7c050 2024-04-23T10:32:03.725 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:32:03.711+0000 7f0fc2ff5640 10 client.16278 handle_command_reply: tid=0 2024-04-23T10:32:03.725 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:32:03.711+0000 7f0fe81f4740 1 client.16278 shutdown 2024-04-23T10:32:03.725 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:32:03.711+0000 7f0fe81f4740 1 -- 172.21.15.114:0/1677909263 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55c8d0ee14c0 msgr2=0x55c8d0f69db0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T10:32:03.725 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:32:03.711+0000 7f0fe81f4740 1 --2- 172.21.15.114:0/1677909263 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55c8d0ee14c0 0x55c8d0f69db0 secure :-1 s=READY pgs=699 cs=0 l=1 rev1=1 crypto rx=0x7f0fd40117f0 tx=0x7f0fd4009940 comp rx=0 tx=0).stop 2024-04-23T10:32:03.725 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:32:03.711+0000 7f0fe81f4740 1 -- 172.21.15.114:0/1677909263 shutdown_connections 2024-04-23T10:32:03.725 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:32:03.711+0000 7f0fe81f4740 1 --2- 172.21.15.114:0/1677909263 >> [v2:172.21.15.204:6832/3866561754,v1:172.21.15.204:6833/3866561754] conn(0x55c8d0f7c050 0x55c8d0f689a0 crc :-1 s=READY pgs=13 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:32:03.726 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:32:03.711+0000 7f0fe81f4740 1 --2- 172.21.15.114:0/1677909263 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x55c8d0ef00a0 0x55c8d0eea600 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:32:03.726 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:32:03.711+0000 7f0fc2ff5640 0 client.16278 ms_handle_reset on v2:172.21.15.204:6832/3866561754 2024-04-23T10:32:03.726 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:32:03.711+0000 7f0fe81f4740 1 --2- 172.21.15.114:0/1677909263 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55c8d0ee14c0 0x55c8d0f69db0 unknown :-1 s=CLOSED pgs=699 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:32:03.726 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:32:03.711+0000 7f0fe81f4740 1 --2- 172.21.15.114:0/1677909263 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x55c8d0ee0ba0 0x55c8d0f02cc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:32:03.726 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:32:03.712+0000 7f0fe81f4740 1 -- 172.21.15.114:0/1677909263 >> 172.21.15.114:0/1677909263 conn(0x55c8d0edcfc0 msgr2=0x55c8d0f61b30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T10:32:03.726 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:32:03.712+0000 7f0fe81f4740 1 -- 172.21.15.114:0/1677909263 shutdown_connections 2024-04-23T10:32:03.726 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:32:03.712+0000 7f0fe81f4740 1 -- 172.21.15.114:0/1677909263 wait complete. 2024-04-23T10:32:03.726 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:32:03.712+0000 7f0fe81f4740 20 client.16278 trim_cache size 0 max 16384 2024-04-23T10:32:03.737 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:32:03.726+0000 7f0fe81f4740 1 Processor -- start 2024-04-23T10:32:03.737 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:32:03.727+0000 7f0fe81f4740 1 -- start start 2024-04-23T10:32:03.738 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:32:03.727+0000 7f0fe81f4740 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x55c8d0ee0ba0 0x55c8d0eea600 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T10:32:03.738 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:32:03.727+0000 7f0fe81f4740 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55c8d0ee14c0 0x55c8d0f689a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T10:32:03.738 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:32:03.727+0000 7f0fe81f4740 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x55c8d0ef00a0 0x55c8d0f02cc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T10:32:03.738 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:32:03.727+0000 7f0fe81f4740 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x55c8d0ed80d0 con 0x55c8d0ee14c0 2024-04-23T10:32:03.738 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:32:03.727+0000 7f0fe81f4740 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 -- 0x55c8d0ee7570 con 0x55c8d0ee0ba0 2024-04-23T10:32:03.738 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:32:03.727+0000 7f0fe81f4740 1 -- --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_getmap magic: 0 -- 0x55c8d0f6b2a0 con 0x55c8d0ef00a0 2024-04-23T10:32:03.738 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:32:03.727+0000 7f0fde7fc640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55c8d0ee14c0 0x55c8d0f689a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T10:32:03.738 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:32:03.727+0000 7f0fde7fc640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55c8d0ee14c0 0x55c8d0f689a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.114:51854/0 (socket says 172.21.15.114:51854) 2024-04-23T10:32:03.738 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:32:03.727+0000 7f0fde7fc640 1 -- 172.21.15.114:0/2134402229 learned_addr learned my addr 172.21.15.114:0/2134402229 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-04-23T10:32:03.738 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:32:03.727+0000 7f0fde7fc640 1 -- 172.21.15.114:0/2134402229 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x55c8d0ef00a0 msgr2=0x55c8d0f02cc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T10:32:03.738 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:32:03.727+0000 7f0fde7fc640 1 --2- 172.21.15.114:0/2134402229 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x55c8d0ef00a0 0x55c8d0f02cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:32:03.738 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:32:03.727+0000 7f0fdeffd640 1 --2- 172.21.15.114:0/2134402229 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x55c8d0ee0ba0 0x55c8d0eea600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T10:32:03.738 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:32:03.727+0000 7f0fde7fc640 1 -- 172.21.15.114:0/2134402229 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x55c8d0ee0ba0 msgr2=0x55c8d0eea600 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T10:32:03.739 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:32:03.727+0000 7f0fde7fc640 1 --2- 172.21.15.114:0/2134402229 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x55c8d0ee0ba0 0x55c8d0eea600 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:32:03.739 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:32:03.727+0000 7f0fde7fc640 1 -- 172.21.15.114:0/2134402229 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55c8d0ed9330 con 0x55c8d0ee14c0 2024-04-23T10:32:03.739 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:32:03.727+0000 7f0fdeffd640 1 --2- 172.21.15.114:0/2134402229 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x55c8d0ee0ba0 0x55c8d0eea600 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-23T10:32:03.739 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:32:03.727+0000 7f0fde7fc640 1 --2- 172.21.15.114:0/2134402229 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55c8d0ee14c0 0x55c8d0f689a0 secure :-1 s=READY pgs=700 cs=0 l=1 rev1=1 crypto rx=0x7f0fd4009880 tx=0x7f0fd40095f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4e9ca306e0b50c42 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T10:32:03.739 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:32:03.727+0000 7f0fc4ff9640 1 -- 172.21.15.114:0/2134402229 <== mon.0 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0fd402f070 con 0x55c8d0ee14c0 2024-04-23T10:32:03.739 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:32:03.727+0000 7f0fc4ff9640 1 -- 172.21.15.114:0/2134402229 <== mon.0 v2:172.21.15.114:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f0fd4003aa0 con 0x55c8d0ee14c0 2024-04-23T10:32:03.739 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:32:03.727+0000 7f0fc4ff9640 1 -- 172.21.15.114:0/2134402229 <== mon.0 v2:172.21.15.114:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0fd4012a60 con 0x55c8d0ee14c0 2024-04-23T10:32:03.739 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:32:03.728+0000 7f0fe81f4740 1 -- 172.21.15.114:0/2134402229 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55c8d0ee14c0 msgr2=0x55c8d0f689a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T10:32:03.739 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:32:03.728+0000 7f0fe81f4740 1 --2- 172.21.15.114:0/2134402229 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55c8d0ee14c0 0x55c8d0f689a0 secure :-1 s=READY pgs=700 cs=0 l=1 rev1=1 crypto rx=0x7f0fd4009880 tx=0x7f0fd40095f0 comp rx=0 tx=0).stop 2024-04-23T10:32:03.739 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:32:03.728+0000 7f0fe81f4740 1 -- 172.21.15.114:0/2134402229 shutdown_connections 2024-04-23T10:32:03.739 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:32:03.728+0000 7f0fe81f4740 1 --2- 172.21.15.114:0/2134402229 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x55c8d0ef00a0 0x55c8d0f02cc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:32:03.739 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:32:03.728+0000 7f0fe81f4740 1 --2- 172.21.15.114:0/2134402229 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55c8d0ee14c0 0x55c8d0f689a0 unknown :-1 s=CLOSED pgs=700 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:32:03.739 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:32:03.728+0000 7f0fe81f4740 1 --2- 172.21.15.114:0/2134402229 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x55c8d0ee0ba0 0x55c8d0eea600 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:32:03.739 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:32:03.728+0000 7f0fe81f4740 1 -- 172.21.15.114:0/2134402229 >> 172.21.15.114:0/2134402229 conn(0x55c8d0edcfc0 msgr2=0x55c8d0f67f50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T10:32:03.740 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:32:03.728+0000 7f0fe81f4740 1 -- 172.21.15.114:0/2134402229 shutdown_connections 2024-04-23T10:32:03.740 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:32:03.728+0000 7f0fe81f4740 1 -- 172.21.15.114:0/2134402229 wait complete. 2024-04-23T10:32:03.740 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:32:03.728+0000 7f0fe81f4740 1 -- start start 2024-04-23T10:32:03.740 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:32:03.728+0000 7f0fe81f4740 1 Processor -- start 2024-04-23T10:32:03.740 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:32:03.728+0000 7f0fe81f4740 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x55c8d0ee0ba0 0x55c8d0f02cc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T10:32:03.740 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:32:03.728+0000 7f0fe81f4740 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55c8d0ee14c0 0x55c8d0f689a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T10:32:03.740 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:32:03.728+0000 7f0fe81f4740 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x55c8d0ef00a0 0x55c8d0ee8460 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T10:32:03.740 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:32:03.728+0000 7f0fe81f4740 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x55c8d0f68820 con 0x55c8d0ee14c0 2024-04-23T10:32:03.740 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:32:03.728+0000 7f0fe81f4740 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 -- 0x55c8d0f0a500 con 0x55c8d0ee0ba0 2024-04-23T10:32:03.740 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:32:03.728+0000 7f0fe81f4740 1 -- --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_getmap magic: 0 -- 0x55c8d0f68ef0 con 0x55c8d0ef00a0 2024-04-23T10:32:03.740 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:32:03.729+0000 7f0fe81f4740 10 client.0 _ll_register_callbacks cb 0x7f0fd4005ed0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-23T10:32:03.740 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:32:03.729+0000 7f0fde7fc640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55c8d0ee14c0 0x55c8d0f689a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T10:32:03.740 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:32:03.729+0000 7f0fde7fc640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55c8d0ee14c0 0x55c8d0f689a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.114:51860/0 (socket says 172.21.15.114:51860) 2024-04-23T10:32:03.740 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:32:03.729+0000 7f0fde7fc640 1 -- 172.21.15.114:0/3029585804 learned_addr learned my addr 172.21.15.114:0/3029585804 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-04-23T10:32:03.740 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:32:03.729+0000 7f0fdf7fe640 1 --2- 172.21.15.114:0/3029585804 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x55c8d0ef00a0 0x55c8d0ee8460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T10:32:03.741 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:32:03.729+0000 7f0fde7fc640 1 -- 172.21.15.114:0/3029585804 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x55c8d0ef00a0 msgr2=0x55c8d0ee8460 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T10:32:03.741 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:32:03.729+0000 7f0fdeffd640 1 --2- 172.21.15.114:0/3029585804 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x55c8d0ee0ba0 0x55c8d0f02cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T10:32:03.741 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:32:03.729+0000 7f0fde7fc640 1 --2- 172.21.15.114:0/3029585804 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x55c8d0ef00a0 0x55c8d0ee8460 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:32:03.741 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:32:03.729+0000 7f0fde7fc640 1 -- 172.21.15.114:0/3029585804 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x55c8d0ee0ba0 msgr2=0x55c8d0f02cc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T10:32:03.741 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:32:03.729+0000 7f0fde7fc640 1 --2- 172.21.15.114:0/3029585804 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x55c8d0ee0ba0 0x55c8d0f02cc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:32:03.741 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:32:03.729+0000 7f0fde7fc640 1 -- 172.21.15.114:0/3029585804 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55c8d0eee1e0 con 0x55c8d0ee14c0 2024-04-23T10:32:03.741 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:32:03.729+0000 7f0fdf7fe640 1 --2- 172.21.15.114:0/3029585804 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x55c8d0ef00a0 0x55c8d0ee8460 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-23T10:32:03.741 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:32:03.729+0000 7f0fdeffd640 1 --2- 172.21.15.114:0/3029585804 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x55c8d0ee0ba0 0x55c8d0f02cc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-23T10:32:03.741 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:32:03.729+0000 7f0fde7fc640 1 --2- 172.21.15.114:0/3029585804 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55c8d0ee14c0 0x55c8d0f689a0 secure :-1 s=READY pgs=701 cs=0 l=1 rev1=1 crypto rx=0x7f0fd40368b0 tx=0x7f0fd40364e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=eca182600c6a949c server_cookie=0 in_seq=0 out_seq=0 2024-04-23T10:32:03.741 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:32:03.729+0000 7f0fc2ff5640 10 client.0 ms_handle_connect on v2:172.21.15.114:3300/0 2024-04-23T10:32:03.741 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:32:03.729+0000 7f0fc2ff5640 1 -- 172.21.15.114:0/3029585804 <== mon.0 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0fd402f030 con 0x55c8d0ee14c0 2024-04-23T10:32:03.741 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:32:03.729+0000 7f0fc2ff5640 1 -- 172.21.15.114:0/3029585804 <== mon.0 v2:172.21.15.114:3300/0 2 ==== osd_map(94..94 src has 1..94) ==== 8585+0+0 (secure 0 0 0) 0x7f0fd405c5f0 con 0x55c8d0ee14c0 2024-04-23T10:32:03.741 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:32:03.729+0000 7f0fe81f4740 1 -- 172.21.15.114:0/3029585804 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f0fd4003aa0 con 0x55c8d0ee14c0 2024-04-23T10:32:03.742 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:32:03.729+0000 7f0fc2ff5640 1 -- 172.21.15.114:0/3029585804 <== mon.0 v2:172.21.15.114:3300/0 3 ==== mon_get_version_reply(handle=1 version=30) ==== 24+0+0 (secure 0 0 0) 0x7f0fd405c7f0 con 0x55c8d0ee14c0 2024-04-23T10:32:03.742 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:32:03.729+0000 7f0fe81f4740 10 client.16284 fetch_fsmap learned FSMap version 30 2024-04-23T10:32:03.742 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:32:03.729+0000 7f0fe81f4740 1 -- 172.21.15.114:0/3029585804 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({config=0+,fsmap=30,monmap=4+}) -- 0x55c8d0eee1e0 con 0x55c8d0ee14c0 2024-04-23T10:32:03.742 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:32:03.729+0000 7f0fc2ff5640 1 -- 172.21.15.114:0/3029585804 <== mon.0 v2:172.21.15.114:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f0fd4003aa0 con 0x55c8d0ee14c0 2024-04-23T10:32:03.742 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:32:03.730+0000 7f0fc2ff5640 1 -- 172.21.15.114:0/3029585804 <== mon.0 v2:172.21.15.114:3300/0 5 ==== fsmap(e 30) ==== 8338+0+0 (secure 0 0 0) 0x7f0fd4012a70 con 0x55c8d0ee14c0 2024-04-23T10:32:03.742 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:32:03.730+0000 7f0fe81f4740 10 client.16284 fetch_fsmap finished waiting for FSMap version 30 2024-04-23T10:32:03.742 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:32:03.730+0000 7f0fe81f4740 10 client.16284 resolve_mds: resolved name 'l' to daemon mds.l 2024-04-23T10:32:03.742 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:32:03.730+0000 7f0fe81f4740 20 client.16284 populate_metadata read hostname 'smithi114' 2024-04-23T10:32:03.742 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:32:03.730+0000 7f0fe81f4740 1 --2- 172.21.15.114:0/3029585804 >> [v2:172.21.15.204:6832/3866561754,v1:172.21.15.204:6833/3866561754] conn(0x55c8d0f6f9f0 0x55c8d0eed8f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T10:32:03.742 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:32:03.730+0000 7f0fe81f4740 4 client.16284 mds_command: new command op to 24385 tid=0 multi_id=0 [{"prefix": "ops", "flags": ["locks"]}] 2024-04-23T10:32:03.742 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:32:03.730+0000 7f0fe81f4740 1 -- 172.21.15.114:0/3029585804 --> [v2:172.21.15.204:6832/3866561754,v1:172.21.15.204:6833/3866561754] -- command(tid 0: {"prefix": "ops", "flags": ["locks"]}) -- 0x55c8d0eee1e0 con 0x55c8d0f6f9f0 2024-04-23T10:32:03.742 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:32:03.730+0000 7f0fdeffd640 1 --2- 172.21.15.114:0/3029585804 >> [v2:172.21.15.204:6832/3866561754,v1:172.21.15.204:6833/3866561754] conn(0x55c8d0f6f9f0 0x55c8d0eed8f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T10:32:03.742 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:32:03.730+0000 7f0fdeffd640 1 --2- 172.21.15.114:0/3029585804 >> [v2:172.21.15.204:6832/3866561754,v1:172.21.15.204:6833/3866561754] conn(0x55c8d0f6f9f0 0x55c8d0eed8f0 crc :-1 s=READY pgs=14 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.? client_cookie=b80993ea1978499b server_cookie=c2cba8b58636fd44 in_seq=0 out_seq=0 2024-04-23T10:32:03.742 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:32:03.730+0000 7f0fc2ff5640 10 client.16284 ms_handle_connect on v2:172.21.15.204:6832/3866561754 2024-04-23T10:32:03.743 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:32:03.731+0000 7f0fc2ff5640 1 -- 172.21.15.114:0/3029585804 <== mds.? v2:172.21.15.204:6832/3866561754 1 ==== command_reply(tid 0: -38 ) ==== 8+0+0 (crc 0 0 0) 0x55c8d0eee1e0 con 0x55c8d0f6f9f0 2024-04-23T10:32:03.743 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:32:03.731+0000 7f0fc2ff5640 10 client.16284 handle_command_reply: tid=0 2024-04-23T10:32:03.743 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:32:03.731+0000 7f0fe81f4740 1 client.16284 shutdown 2024-04-23T10:32:03.743 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:32:03.731+0000 7f0fe81f4740 1 -- 172.21.15.114:0/3029585804 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55c8d0ee14c0 msgr2=0x55c8d0f689a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T10:32:03.743 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:32:03.731+0000 7f0fe81f4740 1 --2- 172.21.15.114:0/3029585804 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55c8d0ee14c0 0x55c8d0f689a0 secure :-1 s=READY pgs=701 cs=0 l=1 rev1=1 crypto rx=0x7f0fd40368b0 tx=0x7f0fd40364e0 comp rx=0 tx=0).stop 2024-04-23T10:32:03.743 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:32:03.731+0000 7f0fe81f4740 1 -- 172.21.15.114:0/3029585804 shutdown_connections 2024-04-23T10:32:03.743 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:32:03.731+0000 7f0fe81f4740 1 --2- 172.21.15.114:0/3029585804 >> [v2:172.21.15.204:6832/3866561754,v1:172.21.15.204:6833/3866561754] conn(0x55c8d0f6f9f0 0x55c8d0eed8f0 crc :-1 s=READY pgs=14 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:32:03.743 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:32:03.731+0000 7f0fe81f4740 1 --2- 172.21.15.114:0/3029585804 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x55c8d0ef00a0 0x55c8d0ee8460 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:32:03.743 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:32:03.731+0000 7f0fc2ff5640 0 client.16284 ms_handle_reset on v2:172.21.15.204:6832/3866561754 2024-04-23T10:32:03.743 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:32:03.731+0000 7f0fe81f4740 1 --2- 172.21.15.114:0/3029585804 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55c8d0ee14c0 0x55c8d0f689a0 unknown :-1 s=CLOSED pgs=701 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:32:03.743 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:32:03.731+0000 7f0fe81f4740 1 --2- 172.21.15.114:0/3029585804 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x55c8d0ee0ba0 0x55c8d0f02cc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:32:03.743 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:32:03.731+0000 7f0fe81f4740 1 -- 172.21.15.114:0/3029585804 >> 172.21.15.114:0/3029585804 conn(0x55c8d0edcfc0 msgr2=0x55c8d0f69bb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T10:32:03.743 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:32:03.731+0000 7f0fe81f4740 1 -- 172.21.15.114:0/3029585804 shutdown_connections 2024-04-23T10:32:03.743 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:32:03.731+0000 7f0fe81f4740 1 -- 172.21.15.114:0/3029585804 wait complete. 2024-04-23T10:32:03.743 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:32:03.731+0000 7f0fe81f4740 20 client.16284 trim_cache size 0 max 16384 2024-04-23T10:32:03.744 INFO:teuthology.orchestra.run.smithi114.stderr:Error ENOSYS: 2024-04-23T10:32:03.744 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:32:03.732+0000 7f0fe81f4740 1 Processor -- start 2024-04-23T10:32:03.744 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:32:03.732+0000 7f0fe81f4740 1 -- start start 2024-04-23T10:32:03.744 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:32:03.732+0000 7f0fe81f4740 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x55c8d0ee0ba0 0x55c8d0f02cc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T10:32:03.744 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:32:03.732+0000 7f0fe81f4740 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x55c8d0ee14c0 0x55c8d0eed8f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T10:32:03.744 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:32:03.732+0000 7f0fe81f4740 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55c8d0ef00a0 0x55c8d0eecee0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T10:32:03.744 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:32:03.732+0000 7f0fe81f4740 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x55c8d0f61f90 con 0x55c8d0ef00a0 2024-04-23T10:32:03.744 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:32:03.732+0000 7f0fe81f4740 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 -- 0x55c8d0f6a8d0 con 0x55c8d0ee14c0 2024-04-23T10:32:03.744 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:32:03.732+0000 7f0fe81f4740 1 -- --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_getmap magic: 0 -- 0x55c8d0f0f7a0 con 0x55c8d0ee0ba0 2024-04-23T10:32:03.744 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:32:03.732+0000 7f0fdf7fe640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55c8d0ef00a0 0x55c8d0eecee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T10:32:03.744 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:32:03.732+0000 7f0fdf7fe640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55c8d0ef00a0 0x55c8d0eecee0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.114:51864/0 (socket says 172.21.15.114:51864) 2024-04-23T10:32:03.744 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:32:03.732+0000 7f0fdf7fe640 1 -- 172.21.15.114:0/605811190 learned_addr learned my addr 172.21.15.114:0/605811190 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-04-23T10:32:03.744 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:32:03.732+0000 7f0fdeffd640 1 --2- 172.21.15.114:0/605811190 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x55c8d0ee0ba0 0x55c8d0f02cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T10:32:03.744 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:32:03.732+0000 7f0fde7fc640 1 --2- 172.21.15.114:0/605811190 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x55c8d0ee14c0 0x55c8d0eed8f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T10:32:03.745 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:32:03.732+0000 7f0fdf7fe640 1 -- 172.21.15.114:0/605811190 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x55c8d0ee0ba0 msgr2=0x55c8d0f02cc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T10:32:03.745 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:32:03.732+0000 7f0fdf7fe640 1 --2- 172.21.15.114:0/605811190 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x55c8d0ee0ba0 0x55c8d0f02cc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:32:03.745 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:32:03.732+0000 7f0fdf7fe640 1 -- 172.21.15.114:0/605811190 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x55c8d0ee14c0 msgr2=0x55c8d0eed8f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T10:32:03.745 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:32:03.732+0000 7f0fdf7fe640 1 --2- 172.21.15.114:0/605811190 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x55c8d0ee14c0 0x55c8d0eed8f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:32:03.745 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:32:03.732+0000 7f0fdf7fe640 1 -- 172.21.15.114:0/605811190 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55c8d0eee1e0 con 0x55c8d0ef00a0 2024-04-23T10:32:03.745 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:32:03.732+0000 7f0fdf7fe640 1 --2- 172.21.15.114:0/605811190 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55c8d0ef00a0 0x55c8d0eecee0 secure :-1 s=READY pgs=702 cs=0 l=1 rev1=1 crypto rx=0x7f0fd809a860 tx=0x7f0fd8000bc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=90ad3e2af8e65960 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T10:32:03.745 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:32:03.732+0000 7f0fc57fa640 1 -- 172.21.15.114:0/605811190 <== mon.0 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0fd8014040 con 0x55c8d0ef00a0 2024-04-23T10:32:03.745 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:32:03.732+0000 7f0fc57fa640 1 -- 172.21.15.114:0/605811190 <== mon.0 v2:172.21.15.114:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f0fd80a0960 con 0x55c8d0ef00a0 2024-04-23T10:32:03.745 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:32:03.732+0000 7f0fc57fa640 1 -- 172.21.15.114:0/605811190 <== mon.0 v2:172.21.15.114:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0fd8020040 con 0x55c8d0ef00a0 2024-04-23T10:32:03.745 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:32:03.733+0000 7f0fe81f4740 1 -- 172.21.15.114:0/605811190 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55c8d0ef00a0 msgr2=0x55c8d0eecee0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T10:32:03.745 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:32:03.733+0000 7f0fe81f4740 1 --2- 172.21.15.114:0/605811190 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55c8d0ef00a0 0x55c8d0eecee0 secure :-1 s=READY pgs=702 cs=0 l=1 rev1=1 crypto rx=0x7f0fd809a860 tx=0x7f0fd8000bc0 comp rx=0 tx=0).stop 2024-04-23T10:32:03.745 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:32:03.733+0000 7f0fe81f4740 1 -- 172.21.15.114:0/605811190 shutdown_connections 2024-04-23T10:32:03.745 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:32:03.733+0000 7f0fe81f4740 1 --2- 172.21.15.114:0/605811190 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55c8d0ef00a0 0x55c8d0eecee0 unknown :-1 s=CLOSED pgs=702 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:32:03.745 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:32:03.733+0000 7f0fe81f4740 1 --2- 172.21.15.114:0/605811190 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x55c8d0ee14c0 0x55c8d0eed8f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:32:03.745 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:32:03.733+0000 7f0fe81f4740 1 --2- 172.21.15.114:0/605811190 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x55c8d0ee0ba0 0x55c8d0f02cc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:32:03.746 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:32:03.733+0000 7f0fe81f4740 1 -- 172.21.15.114:0/605811190 >> 172.21.15.114:0/605811190 conn(0x55c8d0edcfc0 msgr2=0x55c8d0f67f50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T10:32:03.746 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:32:03.733+0000 7f0fe81f4740 1 -- 172.21.15.114:0/605811190 shutdown_connections 2024-04-23T10:32:03.746 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:32:03.733+0000 7f0fe81f4740 1 -- 172.21.15.114:0/605811190 wait complete. 2024-04-23T10:32:03.746 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:32:03.733+0000 7f0fe81f4740 1 -- start start 2024-04-23T10:32:03.746 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:32:03.733+0000 7f0fe81f4740 1 Processor -- start 2024-04-23T10:32:03.746 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:32:03.734+0000 7f0fe81f4740 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x55c8d0ee0ba0 0x55c8d0eecee0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T10:32:03.746 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:32:03.734+0000 7f0fe81f4740 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x55c8d0ee14c0 0x55c8d0eed8f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T10:32:03.746 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:32:03.734+0000 7f0fe81f4740 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55c8d0ef00a0 0x55c8d0f0f920 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T10:32:03.746 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:32:03.734+0000 7f0fe81f4740 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x55c8d0eede40 con 0x55c8d0ef00a0 2024-04-23T10:32:03.746 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:32:03.734+0000 7f0fe81f4740 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 -- 0x55c8d0eed770 con 0x55c8d0ee0ba0 2024-04-23T10:32:03.746 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:32:03.734+0000 7f0fe81f4740 1 -- --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_getmap magic: 0 -- 0x55c8d0f7c5d0 con 0x55c8d0ee14c0 2024-04-23T10:32:03.746 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:32:03.734+0000 7f0fe81f4740 10 client.0 _ll_register_callbacks cb 0x7f0fd4009f60 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-23T10:32:03.746 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:32:03.734+0000 7f0fdf7fe640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55c8d0ef00a0 0x55c8d0f0f920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T10:32:03.746 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:32:03.734+0000 7f0fdf7fe640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55c8d0ef00a0 0x55c8d0f0f920 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.114:51880/0 (socket says 172.21.15.114:51880) 2024-04-23T10:32:03.747 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:32:03.734+0000 7f0fdf7fe640 1 -- 172.21.15.114:0/1928646148 learned_addr learned my addr 172.21.15.114:0/1928646148 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-04-23T10:32:03.747 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:32:03.734+0000 7f0fde7fc640 1 --2- 172.21.15.114:0/1928646148 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x55c8d0ee14c0 0x55c8d0eed8f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T10:32:03.747 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:32:03.734+0000 7f0fdeffd640 1 --2- 172.21.15.114:0/1928646148 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x55c8d0ee0ba0 0x55c8d0eecee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T10:32:03.747 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:32:03.734+0000 7f0fdf7fe640 1 -- 172.21.15.114:0/1928646148 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x55c8d0ee14c0 msgr2=0x55c8d0eed8f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T10:32:03.747 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:32:03.734+0000 7f0fdf7fe640 1 --2- 172.21.15.114:0/1928646148 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x55c8d0ee14c0 0x55c8d0eed8f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:32:03.747 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:32:03.734+0000 7f0fdf7fe640 1 -- 172.21.15.114:0/1928646148 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x55c8d0ee0ba0 msgr2=0x55c8d0eecee0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T10:32:03.747 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:32:03.734+0000 7f0fdf7fe640 1 --2- 172.21.15.114:0/1928646148 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x55c8d0ee0ba0 0x55c8d0eecee0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:32:03.747 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:32:03.734+0000 7f0fdf7fe640 1 -- 172.21.15.114:0/1928646148 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55c8d0f09070 con 0x55c8d0ef00a0 2024-04-23T10:32:03.747 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:32:03.734+0000 7f0fdeffd640 1 --2- 172.21.15.114:0/1928646148 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x55c8d0ee0ba0 0x55c8d0eecee0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-23T10:32:03.747 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:32:03.734+0000 7f0fdf7fe640 1 --2- 172.21.15.114:0/1928646148 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55c8d0ef00a0 0x55c8d0f0f920 secure :-1 s=READY pgs=703 cs=0 l=1 rev1=1 crypto rx=0x7f0fd80d00b0 tx=0x7f0fd809aac0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b910d2511a7f096f server_cookie=0 in_seq=0 out_seq=0 2024-04-23T10:32:03.747 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:32:03.734+0000 7f0fc2ff5640 10 client.0 ms_handle_connect on v2:172.21.15.114:3300/0 2024-04-23T10:32:03.747 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:32:03.734+0000 7f0fc2ff5640 1 -- 172.21.15.114:0/1928646148 <== mon.0 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0fd8014070 con 0x55c8d0ef00a0 2024-04-23T10:32:03.747 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:32:03.734+0000 7f0fc2ff5640 1 -- 172.21.15.114:0/1928646148 <== mon.0 v2:172.21.15.114:3300/0 2 ==== osd_map(94..94 src has 1..94) ==== 8585+0+0 (secure 0 0 0) 0x7f0fd8098620 con 0x55c8d0ef00a0 2024-04-23T10:32:03.748 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:32:03.734+0000 7f0fe81f4740 1 -- 172.21.15.114:0/1928646148 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f0fd80a0960 con 0x55c8d0ef00a0 2024-04-23T10:32:03.748 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:32:03.734+0000 7f0fc2ff5640 1 -- 172.21.15.114:0/1928646148 <== mon.0 v2:172.21.15.114:3300/0 3 ==== mon_get_version_reply(handle=1 version=30) ==== 24+0+0 (secure 0 0 0) 0x7f0fd80ad050 con 0x55c8d0ef00a0 2024-04-23T10:32:03.748 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:32:03.734+0000 7f0fe81f4740 10 client.16290 fetch_fsmap learned FSMap version 30 2024-04-23T10:32:03.748 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:32:03.734+0000 7f0fe81f4740 1 -- 172.21.15.114:0/1928646148 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({config=0+,fsmap=30,monmap=4+}) -- 0x55c8d0f09070 con 0x55c8d0ef00a0 2024-04-23T10:32:03.748 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:32:03.735+0000 7f0fc2ff5640 1 -- 172.21.15.114:0/1928646148 <== mon.0 v2:172.21.15.114:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f0fd80a0960 con 0x55c8d0ef00a0 2024-04-23T10:32:03.748 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:32:03.735+0000 7f0fc2ff5640 1 -- 172.21.15.114:0/1928646148 <== mon.0 v2:172.21.15.114:3300/0 5 ==== fsmap(e 30) ==== 8338+0+0 (secure 0 0 0) 0x7f0fd809b470 con 0x55c8d0ef00a0 2024-04-23T10:32:03.748 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:32:03.735+0000 7f0fe81f4740 10 client.16290 fetch_fsmap finished waiting for FSMap version 30 2024-04-23T10:32:03.748 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:32:03.735+0000 7f0fe81f4740 10 client.16290 resolve_mds: resolved name 'f' to daemon mds.f 2024-04-23T10:32:03.748 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:32:03.735+0000 7f0fe81f4740 20 client.16290 populate_metadata read hostname 'smithi114' 2024-04-23T10:32:03.748 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:32:03.735+0000 7f0fe81f4740 1 --2- 172.21.15.114:0/1928646148 >> [v2:172.21.15.204:6834/2533793172,v1:172.21.15.204:6835/2533793172] conn(0x55c8d0f6faa0 0x55c8d0f6fe80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T10:32:03.748 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:32:03.735+0000 7f0fe81f4740 4 client.16290 mds_command: new command op to 24427 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-23T10:32:03.748 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:32:03.735+0000 7f0fe81f4740 1 -- 172.21.15.114:0/1928646148 --> [v2:172.21.15.204:6834/2533793172,v1:172.21.15.204:6835/2533793172] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55c8d0ed8730 con 0x55c8d0f6faa0 2024-04-23T10:32:03.748 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:32:03.735+0000 7f0fdeffd640 1 --2- 172.21.15.114:0/1928646148 >> [v2:172.21.15.204:6834/2533793172,v1:172.21.15.204:6835/2533793172] conn(0x55c8d0f6faa0 0x55c8d0f6fe80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T10:32:03.748 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:32:03.735+0000 7f0fdeffd640 1 --2- 172.21.15.114:0/1928646148 >> [v2:172.21.15.204:6834/2533793172,v1:172.21.15.204:6835/2533793172] conn(0x55c8d0f6faa0 0x55c8d0f6fe80 crc :-1 s=READY pgs=12 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.? client_cookie=7489e305bf097185 server_cookie=3fb940833790b52d in_seq=0 out_seq=0 2024-04-23T10:32:03.749 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:32:03.735+0000 7f0fc2ff5640 10 client.16290 ms_handle_connect on v2:172.21.15.204:6834/2533793172 2024-04-23T10:32:03.749 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:32:03.736+0000 7f0fc2ff5640 1 -- 172.21.15.114:0/1928646148 <== mds.? v2:172.21.15.204:6834/2533793172 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23115 (crc 0 0 0) 0x55c8d0ed8730 con 0x55c8d0f6faa0 2024-04-23T10:32:03.749 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:32:03.736+0000 7f0fc2ff5640 10 client.16290 handle_command_reply: tid=0 2024-04-23T10:32:03.749 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:32:03.736+0000 7f0fe81f4740 1 client.16290 shutdown 2024-04-23T10:32:03.749 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:32:03.736+0000 7f0fe81f4740 1 -- 172.21.15.114:0/1928646148 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55c8d0ef00a0 msgr2=0x55c8d0f0f920 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T10:32:03.749 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:32:03.736+0000 7f0fe81f4740 1 --2- 172.21.15.114:0/1928646148 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55c8d0ef00a0 0x55c8d0f0f920 secure :-1 s=READY pgs=703 cs=0 l=1 rev1=1 crypto rx=0x7f0fd80d00b0 tx=0x7f0fd809aac0 comp rx=0 tx=0).stop 2024-04-23T10:32:03.749 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:32:03.737+0000 7f0fe81f4740 1 -- 172.21.15.114:0/1928646148 shutdown_connections 2024-04-23T10:32:03.749 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:32:03.737+0000 7f0fe81f4740 1 --2- 172.21.15.114:0/1928646148 >> [v2:172.21.15.204:6834/2533793172,v1:172.21.15.204:6835/2533793172] conn(0x55c8d0f6faa0 0x55c8d0f6fe80 crc :-1 s=READY pgs=12 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:32:03.749 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:32:03.737+0000 7f0fe81f4740 1 --2- 172.21.15.114:0/1928646148 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55c8d0ef00a0 0x55c8d0f0f920 unknown :-1 s=CLOSED pgs=703 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:32:03.749 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:32:03.737+0000 7f0fe81f4740 1 --2- 172.21.15.114:0/1928646148 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x55c8d0ee14c0 0x55c8d0eed8f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:32:03.749 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:32:03.737+0000 7f0fc2ff5640 0 client.16290 ms_handle_reset on v2:172.21.15.204:6834/2533793172 2024-04-23T10:32:03.749 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:32:03.737+0000 7f0fe81f4740 1 --2- 172.21.15.114:0/1928646148 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x55c8d0ee0ba0 0x55c8d0eecee0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:32:03.749 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:32:03.737+0000 7f0fe81f4740 1 -- 172.21.15.114:0/1928646148 >> 172.21.15.114:0/1928646148 conn(0x55c8d0edcfc0 msgr2=0x55c8d0ee8460 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T10:32:03.750 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:32:03.737+0000 7f0fe81f4740 1 -- 172.21.15.114:0/1928646148 shutdown_connections 2024-04-23T10:32:03.750 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:32:03.737+0000 7f0fe81f4740 1 -- 172.21.15.114:0/1928646148 wait complete. 2024-04-23T10:32:03.750 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:32:03.737+0000 7f0fe81f4740 20 client.16290 trim_cache size 0 max 16384 2024-04-23T10:32:03.762 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:32:03.751+0000 7f0fe81f4740 1 Processor -- start 2024-04-23T10:32:03.762 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:32:03.752+0000 7f0fe81f4740 1 -- start start 2024-04-23T10:32:03.762 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:32:03.752+0000 7f0fe81f4740 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x55c8d0ee0ba0 0x55c8d0f02cc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T10:32:03.762 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:32:03.752+0000 7f0fe81f4740 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x55c8d0ee14c0 0x55c8d0f6bdb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T10:32:03.762 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:32:03.752+0000 7f0fe81f4740 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55c8d0ef00a0 0x55c8d0eea600 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T10:32:03.762 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:32:03.752+0000 7f0fe81f4740 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x55c8d0efbf60 con 0x55c8d0ef00a0 2024-04-23T10:32:03.762 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:32:03.752+0000 7f0fe81f4740 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 -- 0x55c8d0f6ad50 con 0x55c8d0ee0ba0 2024-04-23T10:32:03.762 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:32:03.752+0000 7f0fe81f4740 1 -- --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_getmap magic: 0 -- 0x55c8d0ed85b0 con 0x55c8d0ee14c0 2024-04-23T10:32:03.762 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:32:03.752+0000 7f0fdf7fe640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55c8d0ef00a0 0x55c8d0eea600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T10:32:03.763 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:32:03.752+0000 7f0fdf7fe640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55c8d0ef00a0 0x55c8d0eea600 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.114:51894/0 (socket says 172.21.15.114:51894) 2024-04-23T10:32:03.763 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:32:03.752+0000 7f0fdf7fe640 1 -- 172.21.15.114:0/3850295962 learned_addr learned my addr 172.21.15.114:0/3850295962 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-04-23T10:32:03.763 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:32:03.752+0000 7f0fdf7fe640 1 -- 172.21.15.114:0/3850295962 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x55c8d0ee14c0 msgr2=0x55c8d0f6bdb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T10:32:03.763 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:32:03.752+0000 7f0fdf7fe640 1 --2- 172.21.15.114:0/3850295962 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x55c8d0ee14c0 0x55c8d0f6bdb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:32:03.763 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:32:03.752+0000 7f0fdf7fe640 1 -- 172.21.15.114:0/3850295962 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x55c8d0ee0ba0 msgr2=0x55c8d0f02cc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T10:32:03.763 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:32:03.752+0000 7f0fdf7fe640 1 --2- 172.21.15.114:0/3850295962 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x55c8d0ee0ba0 0x55c8d0f02cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:32:03.763 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:32:03.752+0000 7f0fdf7fe640 1 -- 172.21.15.114:0/3850295962 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55c8d0f6b810 con 0x55c8d0ef00a0 2024-04-23T10:32:03.763 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:32:03.752+0000 7f0fdf7fe640 1 --2- 172.21.15.114:0/3850295962 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55c8d0ef00a0 0x55c8d0eea600 secure :-1 s=READY pgs=704 cs=0 l=1 rev1=1 crypto rx=0x7f0fd8098a10 tx=0x7f0fd8009f00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a7c1212dd1d098ea server_cookie=0 in_seq=0 out_seq=0 2024-04-23T10:32:03.763 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:32:03.752+0000 7f0fc4ff9640 1 -- 172.21.15.114:0/3850295962 <== mon.0 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0fd80ad070 con 0x55c8d0ef00a0 2024-04-23T10:32:03.763 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:32:03.752+0000 7f0fc4ff9640 1 -- 172.21.15.114:0/3850295962 <== mon.0 v2:172.21.15.114:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f0fd809bd10 con 0x55c8d0ef00a0 2024-04-23T10:32:03.763 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:32:03.752+0000 7f0fc4ff9640 1 -- 172.21.15.114:0/3850295962 <== mon.0 v2:172.21.15.114:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0fd8014050 con 0x55c8d0ef00a0 2024-04-23T10:32:03.763 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:32:03.753+0000 7f0fe81f4740 1 -- 172.21.15.114:0/3850295962 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55c8d0ef00a0 msgr2=0x55c8d0eea600 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T10:32:03.763 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:32:03.753+0000 7f0fe81f4740 1 --2- 172.21.15.114:0/3850295962 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55c8d0ef00a0 0x55c8d0eea600 secure :-1 s=READY pgs=704 cs=0 l=1 rev1=1 crypto rx=0x7f0fd8098a10 tx=0x7f0fd8009f00 comp rx=0 tx=0).stop 2024-04-23T10:32:03.763 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:32:03.753+0000 7f0fe81f4740 1 -- 172.21.15.114:0/3850295962 shutdown_connections 2024-04-23T10:32:03.764 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:32:03.753+0000 7f0fe81f4740 1 --2- 172.21.15.114:0/3850295962 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55c8d0ef00a0 0x55c8d0eea600 unknown :-1 s=CLOSED pgs=704 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:32:03.764 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:32:03.753+0000 7f0fe81f4740 1 --2- 172.21.15.114:0/3850295962 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x55c8d0ee14c0 0x55c8d0f6bdb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:32:03.764 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:32:03.753+0000 7f0fe81f4740 1 --2- 172.21.15.114:0/3850295962 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x55c8d0ee0ba0 0x55c8d0f02cc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:32:03.764 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:32:03.753+0000 7f0fe81f4740 1 -- 172.21.15.114:0/3850295962 >> 172.21.15.114:0/3850295962 conn(0x55c8d0edcfc0 msgr2=0x55c8d0ee8460 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T10:32:03.764 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:32:03.753+0000 7f0fe81f4740 1 -- 172.21.15.114:0/3850295962 shutdown_connections 2024-04-23T10:32:03.764 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:32:03.753+0000 7f0fe81f4740 1 -- 172.21.15.114:0/3850295962 wait complete. 2024-04-23T10:32:03.764 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:32:03.753+0000 7f0fe81f4740 1 -- start start 2024-04-23T10:32:03.764 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:32:03.753+0000 7f0fe81f4740 1 Processor -- start 2024-04-23T10:32:03.764 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:32:03.753+0000 7f0fe81f4740 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x55c8d0ee0ba0 0x55c8d0f02cc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T10:32:03.764 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:32:03.753+0000 7f0fe81f4740 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55c8d0ee14c0 0x55c8d0f6bdb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T10:32:03.764 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:32:03.753+0000 7f0fe81f4740 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x55c8d0ef00a0 0x55c8d0eea600 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T10:32:03.764 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:32:03.753+0000 7f0fe81f4740 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x55c8d0f66ce0 con 0x55c8d0ee14c0 2024-04-23T10:32:03.764 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:32:03.753+0000 7f0fe81f4740 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 -- 0x55c8d0f682a0 con 0x55c8d0ee0ba0 2024-04-23T10:32:03.764 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:32:03.753+0000 7f0fe81f4740 1 -- --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_getmap magic: 0 -- 0x55c8d0f67390 con 0x55c8d0ef00a0 2024-04-23T10:32:03.764 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:32:03.753+0000 7f0fe81f4740 10 client.0 _ll_register_callbacks cb 0x7f0fd4005ed0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-23T10:32:03.765 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:32:03.754+0000 7f0fde7fc640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55c8d0ee14c0 0x55c8d0f6bdb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T10:32:03.765 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:32:03.754+0000 7f0fde7fc640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55c8d0ee14c0 0x55c8d0f6bdb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.114:51906/0 (socket says 172.21.15.114:51906) 2024-04-23T10:32:03.765 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:32:03.754+0000 7f0fde7fc640 1 -- 172.21.15.114:0/3487530968 learned_addr learned my addr 172.21.15.114:0/3487530968 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-04-23T10:32:03.765 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:32:03.754+0000 7f0fdeffd640 1 --2- 172.21.15.114:0/3487530968 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x55c8d0ee0ba0 0x55c8d0f02cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T10:32:03.765 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:32:03.754+0000 7f0fde7fc640 1 -- 172.21.15.114:0/3487530968 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x55c8d0ef00a0 msgr2=0x55c8d0eea600 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T10:32:03.765 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:32:03.754+0000 7f0fde7fc640 1 --2- 172.21.15.114:0/3487530968 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x55c8d0ef00a0 0x55c8d0eea600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:32:03.765 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:32:03.754+0000 7f0fde7fc640 1 -- 172.21.15.114:0/3487530968 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x55c8d0ee0ba0 msgr2=0x55c8d0f02cc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T10:32:03.765 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:32:03.754+0000 7f0fde7fc640 1 --2- 172.21.15.114:0/3487530968 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x55c8d0ee0ba0 0x55c8d0f02cc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:32:03.765 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:32:03.754+0000 7f0fde7fc640 1 -- 172.21.15.114:0/3487530968 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55c8d0f679e0 con 0x55c8d0ee14c0 2024-04-23T10:32:03.765 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:32:03.754+0000 7f0fdeffd640 1 --2- 172.21.15.114:0/3487530968 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x55c8d0ee0ba0 0x55c8d0f02cc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-23T10:32:03.765 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:32:03.754+0000 7f0fde7fc640 1 --2- 172.21.15.114:0/3487530968 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55c8d0ee14c0 0x55c8d0f6bdb0 secure :-1 s=READY pgs=705 cs=0 l=1 rev1=1 crypto rx=0x7f0fd405e010 tx=0x7f0fd405c820 comp rx=0 tx=0).ready entity=mon.0 client_cookie=bb4308ea6ef3608f server_cookie=0 in_seq=0 out_seq=0 2024-04-23T10:32:03.765 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:32:03.754+0000 7f0fc2ff5640 10 client.0 ms_handle_connect on v2:172.21.15.114:3300/0 2024-04-23T10:32:03.766 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:32:03.754+0000 7f0fc2ff5640 1 -- 172.21.15.114:0/3487530968 <== mon.0 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0fd402f040 con 0x55c8d0ee14c0 2024-04-23T10:32:03.766 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:32:03.754+0000 7f0fc2ff5640 1 -- 172.21.15.114:0/3487530968 <== mon.0 v2:172.21.15.114:3300/0 2 ==== osd_map(94..94 src has 1..94) ==== 8585+0+0 (secure 0 0 0) 0x7f0fd405c0d0 con 0x55c8d0ee14c0 2024-04-23T10:32:03.766 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:32:03.754+0000 7f0fe81f4740 1 -- 172.21.15.114:0/3487530968 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x55c8d0f679e0 con 0x55c8d0ee14c0 2024-04-23T10:32:03.766 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:32:03.754+0000 7f0fc2ff5640 1 -- 172.21.15.114:0/3487530968 <== mon.0 v2:172.21.15.114:3300/0 3 ==== mon_get_version_reply(handle=1 version=30) ==== 24+0+0 (secure 0 0 0) 0x7f0fd405c2d0 con 0x55c8d0ee14c0 2024-04-23T10:32:03.766 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:32:03.754+0000 7f0fe81f4740 10 client.16296 fetch_fsmap learned FSMap version 30 2024-04-23T10:32:03.766 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:32:03.754+0000 7f0fe81f4740 1 -- 172.21.15.114:0/3487530968 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({config=0+,fsmap=30,monmap=4+}) -- 0x55c8d0f67f50 con 0x55c8d0ee14c0 2024-04-23T10:32:03.766 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:32:03.754+0000 7f0fc2ff5640 1 -- 172.21.15.114:0/3487530968 <== mon.0 v2:172.21.15.114:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x55c8d0f679e0 con 0x55c8d0ee14c0 2024-04-23T10:32:03.766 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:32:03.754+0000 7f0fc2ff5640 1 -- 172.21.15.114:0/3487530968 <== mon.0 v2:172.21.15.114:3300/0 5 ==== fsmap(e 30) ==== 8338+0+0 (secure 0 0 0) 0x7f0fd4005040 con 0x55c8d0ee14c0 2024-04-23T10:32:03.766 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:32:03.754+0000 7f0fe81f4740 10 client.16296 fetch_fsmap finished waiting for FSMap version 30 2024-04-23T10:32:03.766 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:32:03.754+0000 7f0fe81f4740 10 client.16296 resolve_mds: resolved name 'f' to daemon mds.f 2024-04-23T10:32:03.766 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:32:03.754+0000 7f0fe81f4740 20 client.16296 populate_metadata read hostname 'smithi114' 2024-04-23T10:32:03.766 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:32:03.754+0000 7f0fe81f4740 1 --2- 172.21.15.114:0/3487530968 >> [v2:172.21.15.204:6834/2533793172,v1:172.21.15.204:6835/2533793172] conn(0x55c8d0efb270 0x55c8d0f6ed10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T10:32:03.766 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:32:03.754+0000 7f0fe81f4740 4 client.16296 mds_command: new command op to 24427 tid=0 multi_id=0 [{"prefix": "ops", "flags": ["locks"]}] 2024-04-23T10:32:03.766 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:32:03.755+0000 7f0fe81f4740 1 -- 172.21.15.114:0/3487530968 --> [v2:172.21.15.204:6834/2533793172,v1:172.21.15.204:6835/2533793172] -- command(tid 0: {"prefix": "ops", "flags": ["locks"]}) -- 0x55c8d0f67f50 con 0x55c8d0efb270 2024-04-23T10:32:03.766 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:32:03.755+0000 7f0fdeffd640 1 --2- 172.21.15.114:0/3487530968 >> [v2:172.21.15.204:6834/2533793172,v1:172.21.15.204:6835/2533793172] conn(0x55c8d0efb270 0x55c8d0f6ed10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T10:32:03.767 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:32:03.755+0000 7f0fc2ff5640 10 client.16296 ms_handle_connect on v2:172.21.15.204:6834/2533793172 2024-04-23T10:32:03.767 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:32:03.755+0000 7f0fdeffd640 1 --2- 172.21.15.114:0/3487530968 >> [v2:172.21.15.204:6834/2533793172,v1:172.21.15.204:6835/2533793172] conn(0x55c8d0efb270 0x55c8d0f6ed10 crc :-1 s=READY pgs=13 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.? client_cookie=408d4cfaf73db8b8 server_cookie=b6812550576fc32a in_seq=0 out_seq=0 2024-04-23T10:32:03.767 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:32:03.755+0000 7f0fc2ff5640 1 -- 172.21.15.114:0/3487530968 <== mds.? v2:172.21.15.204:6834/2533793172 1 ==== command_reply(tid 0: -38 ) ==== 8+0+0 (crc 0 0 0) 0x55c8d0f67f50 con 0x55c8d0efb270 2024-04-23T10:32:03.767 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:32:03.755+0000 7f0fc2ff5640 10 client.16296 handle_command_reply: tid=0 2024-04-23T10:32:03.767 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:32:03.755+0000 7f0fe81f4740 1 client.16296 shutdown 2024-04-23T10:32:03.767 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:32:03.755+0000 7f0fe81f4740 1 -- 172.21.15.114:0/3487530968 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55c8d0ee14c0 msgr2=0x55c8d0f6bdb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T10:32:03.767 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:32:03.755+0000 7f0fe81f4740 1 --2- 172.21.15.114:0/3487530968 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55c8d0ee14c0 0x55c8d0f6bdb0 secure :-1 s=READY pgs=705 cs=0 l=1 rev1=1 crypto rx=0x7f0fd405e010 tx=0x7f0fd405c820 comp rx=0 tx=0).stop 2024-04-23T10:32:03.767 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:32:03.755+0000 7f0fe81f4740 1 -- 172.21.15.114:0/3487530968 shutdown_connections 2024-04-23T10:32:03.767 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:32:03.755+0000 7f0fe81f4740 1 --2- 172.21.15.114:0/3487530968 >> [v2:172.21.15.204:6834/2533793172,v1:172.21.15.204:6835/2533793172] conn(0x55c8d0efb270 0x55c8d0f6ed10 crc :-1 s=READY pgs=13 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:32:03.767 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:32:03.755+0000 7f0fe81f4740 1 --2- 172.21.15.114:0/3487530968 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x55c8d0ef00a0 0x55c8d0eea600 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:32:03.767 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:32:03.755+0000 7f0fe81f4740 1 --2- 172.21.15.114:0/3487530968 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55c8d0ee14c0 0x55c8d0f6bdb0 unknown :-1 s=CLOSED pgs=705 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:32:03.767 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:32:03.755+0000 7f0fc2ff5640 0 client.16296 ms_handle_reset on v2:172.21.15.204:6834/2533793172 2024-04-23T10:32:03.767 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:32:03.755+0000 7f0fe81f4740 1 --2- 172.21.15.114:0/3487530968 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x55c8d0ee0ba0 0x55c8d0f02cc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:32:03.767 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:32:03.755+0000 7f0fe81f4740 1 -- 172.21.15.114:0/3487530968 >> 172.21.15.114:0/3487530968 conn(0x55c8d0edcfc0 msgr2=0x55c8d0f0f660 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T10:32:03.767 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:32:03.756+0000 7f0fe81f4740 1 -- 172.21.15.114:0/3487530968 shutdown_connections 2024-04-23T10:32:03.767 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:32:03.756+0000 7f0fe81f4740 1 -- 172.21.15.114:0/3487530968 wait complete. 2024-04-23T10:32:03.768 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:32:03.756+0000 7f0fe81f4740 20 client.16296 trim_cache size 0 max 16384 2024-04-23T10:32:03.768 INFO:teuthology.orchestra.run.smithi114.stderr:Error ENOSYS: 2024-04-23T10:32:03.768 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:32:03.756+0000 7f0fe81f4740 1 Processor -- start 2024-04-23T10:32:03.768 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:32:03.756+0000 7f0fe81f4740 1 -- start start 2024-04-23T10:32:03.768 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:32:03.756+0000 7f0fe81f4740 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x55c8d0ee0ba0 0x55c8d0f0aff0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T10:32:03.768 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:32:03.756+0000 7f0fe81f4740 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x55c8d0ee14c0 0x55c8d0f6bdb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T10:32:03.768 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:32:03.756+0000 7f0fe81f4740 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55c8d0ef00a0 0x55c8d0eea600 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T10:32:03.768 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:32:03.756+0000 7f0fe81f4740 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x55c8d0f6af90 con 0x55c8d0ef00a0 2024-04-23T10:32:03.768 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:32:03.756+0000 7f0fe81f4740 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 -- 0x55c8d0ed8170 con 0x55c8d0ee0ba0 2024-04-23T10:32:03.768 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:32:03.756+0000 7f0fe81f4740 1 -- --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_getmap magic: 0 -- 0x55c8d0f68560 con 0x55c8d0ee14c0 2024-04-23T10:32:03.768 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:32:03.756+0000 7f0fdf7fe640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55c8d0ef00a0 0x55c8d0eea600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T10:32:03.768 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:32:03.756+0000 7f0fdf7fe640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55c8d0ef00a0 0x55c8d0eea600 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.114:51916/0 (socket says 172.21.15.114:51916) 2024-04-23T10:32:03.768 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:32:03.756+0000 7f0fdf7fe640 1 -- 172.21.15.114:0/921565928 learned_addr learned my addr 172.21.15.114:0/921565928 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-04-23T10:32:03.768 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:32:03.756+0000 7f0fde7fc640 1 --2- 172.21.15.114:0/921565928 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x55c8d0ee14c0 0x55c8d0f6bdb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T10:32:03.768 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:32:03.756+0000 7f0fdeffd640 1 --2- 172.21.15.114:0/921565928 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x55c8d0ee0ba0 0x55c8d0f0aff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T10:32:03.769 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:32:03.756+0000 7f0fdf7fe640 1 -- 172.21.15.114:0/921565928 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x55c8d0ee14c0 msgr2=0x55c8d0f6bdb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T10:32:03.769 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:32:03.756+0000 7f0fdf7fe640 1 --2- 172.21.15.114:0/921565928 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x55c8d0ee14c0 0x55c8d0f6bdb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:32:03.769 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:32:03.756+0000 7f0fdf7fe640 1 -- 172.21.15.114:0/921565928 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x55c8d0ee0ba0 msgr2=0x55c8d0f0aff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T10:32:03.769 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:32:03.756+0000 7f0fdf7fe640 1 --2- 172.21.15.114:0/921565928 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x55c8d0ee0ba0 0x55c8d0f0aff0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:32:03.769 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:32:03.756+0000 7f0fdf7fe640 1 -- 172.21.15.114:0/921565928 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55c8d0f67f50 con 0x55c8d0ef00a0 2024-04-23T10:32:03.769 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:32:03.756+0000 7f0fde7fc640 1 --2- 172.21.15.114:0/921565928 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x55c8d0ee14c0 0x55c8d0f6bdb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-23T10:32:03.769 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:32:03.756+0000 7f0fdeffd640 1 --2- 172.21.15.114:0/921565928 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x55c8d0ee0ba0 0x55c8d0f0aff0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-23T10:32:03.769 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:32:03.757+0000 7f0fdf7fe640 1 --2- 172.21.15.114:0/921565928 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55c8d0ef00a0 0x55c8d0eea600 secure :-1 s=READY pgs=706 cs=0 l=1 rev1=1 crypto rx=0x7f0fd809c1a0 tx=0x7f0fd8000bc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5b0a50b49849574e server_cookie=0 in_seq=0 out_seq=0 2024-04-23T10:32:03.769 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:32:03.757+0000 7f0fc57fa640 1 -- 172.21.15.114:0/921565928 <== mon.0 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0fd80ad070 con 0x55c8d0ef00a0 2024-04-23T10:32:03.769 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:32:03.757+0000 7f0fc57fa640 1 -- 172.21.15.114:0/921565928 <== mon.0 v2:172.21.15.114:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f0fd80a0960 con 0x55c8d0ef00a0 2024-04-23T10:32:03.769 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:32:03.757+0000 7f0fc57fa640 1 -- 172.21.15.114:0/921565928 <== mon.0 v2:172.21.15.114:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0fd8014070 con 0x55c8d0ef00a0 2024-04-23T10:32:03.769 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:32:03.757+0000 7f0fe81f4740 1 -- 172.21.15.114:0/921565928 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55c8d0ef00a0 msgr2=0x55c8d0eea600 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T10:32:03.769 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:32:03.757+0000 7f0fe81f4740 1 --2- 172.21.15.114:0/921565928 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55c8d0ef00a0 0x55c8d0eea600 secure :-1 s=READY pgs=706 cs=0 l=1 rev1=1 crypto rx=0x7f0fd809c1a0 tx=0x7f0fd8000bc0 comp rx=0 tx=0).stop 2024-04-23T10:32:03.769 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:32:03.757+0000 7f0fe81f4740 1 -- 172.21.15.114:0/921565928 shutdown_connections 2024-04-23T10:32:03.769 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:32:03.757+0000 7f0fe81f4740 1 --2- 172.21.15.114:0/921565928 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55c8d0ef00a0 0x55c8d0eea600 unknown :-1 s=CLOSED pgs=706 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:32:03.770 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:32:03.757+0000 7f0fe81f4740 1 --2- 172.21.15.114:0/921565928 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x55c8d0ee14c0 0x55c8d0f6bdb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:32:03.770 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:32:03.757+0000 7f0fe81f4740 1 --2- 172.21.15.114:0/921565928 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x55c8d0ee0ba0 0x55c8d0f0aff0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:32:03.770 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:32:03.757+0000 7f0fe81f4740 1 -- 172.21.15.114:0/921565928 >> 172.21.15.114:0/921565928 conn(0x55c8d0edcfc0 msgr2=0x55c8d0ee8460 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T10:32:03.770 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:32:03.757+0000 7f0fe81f4740 1 -- 172.21.15.114:0/921565928 shutdown_connections 2024-04-23T10:32:03.770 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:32:03.757+0000 7f0fe81f4740 1 -- 172.21.15.114:0/921565928 wait complete. 2024-04-23T10:32:03.770 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:32:03.757+0000 7f0fe81f4740 1 -- start start 2024-04-23T10:32:03.770 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:32:03.758+0000 7f0fe81f4740 1 Processor -- start 2024-04-23T10:32:03.770 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:32:03.758+0000 7f0fe81f4740 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x55c8d0ee0ba0 0x55c8d0eea600 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T10:32:03.770 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:32:03.758+0000 7f0fe81f4740 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x55c8d0ee14c0 0x55c8d0f6ed10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T10:32:03.770 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:32:03.758+0000 7f0fe81f4740 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55c8d0ef00a0 0x55c8d0f686e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T10:32:03.770 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:32:03.758+0000 7f0fe81f4740 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x55c8d0f6eb90 con 0x55c8d0ef00a0 2024-04-23T10:32:03.770 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:32:03.758+0000 7f0fe81f4740 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 -- 0x55c8d0f68c30 con 0x55c8d0ee0ba0 2024-04-23T10:32:03.770 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:32:03.758+0000 7f0fe81f4740 1 -- --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_getmap magic: 0 -- 0x55c8d0f6f260 con 0x55c8d0ee14c0 2024-04-23T10:32:03.770 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:32:03.758+0000 7f0fe81f4740 10 client.0 _ll_register_callbacks cb 0x7f0fd4011f40 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-23T10:32:03.770 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:32:03.758+0000 7f0fdf7fe640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55c8d0ef00a0 0x55c8d0f686e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T10:32:03.771 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:32:03.758+0000 7f0fdf7fe640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55c8d0ef00a0 0x55c8d0f686e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.114:51920/0 (socket says 172.21.15.114:51920) 2024-04-23T10:32:03.771 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:32:03.758+0000 7f0fdf7fe640 1 -- 172.21.15.114:0/1371697131 learned_addr learned my addr 172.21.15.114:0/1371697131 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-04-23T10:32:03.771 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:32:03.758+0000 7f0fde7fc640 1 --2- 172.21.15.114:0/1371697131 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x55c8d0ee14c0 0x55c8d0f6ed10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T10:32:03.771 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:32:03.758+0000 7f0fdf7fe640 1 -- 172.21.15.114:0/1371697131 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x55c8d0ee14c0 msgr2=0x55c8d0f6ed10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T10:32:03.771 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:32:03.758+0000 7f0fdf7fe640 1 --2- 172.21.15.114:0/1371697131 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x55c8d0ee14c0 0x55c8d0f6ed10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:32:03.771 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:32:03.758+0000 7f0fdf7fe640 1 -- 172.21.15.114:0/1371697131 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x55c8d0ee0ba0 msgr2=0x55c8d0eea600 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T10:32:03.771 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:32:03.758+0000 7f0fdf7fe640 1 --2- 172.21.15.114:0/1371697131 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x55c8d0ee0ba0 0x55c8d0eea600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:32:03.771 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:32:03.758+0000 7f0fdf7fe640 1 -- 172.21.15.114:0/1371697131 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55c8d0f09070 con 0x55c8d0ef00a0 2024-04-23T10:32:03.771 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:32:03.758+0000 7f0fdf7fe640 1 --2- 172.21.15.114:0/1371697131 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55c8d0ef00a0 0x55c8d0f686e0 secure :-1 s=READY pgs=707 cs=0 l=1 rev1=1 crypto rx=0x7f0fd809c2b0 tx=0x7f0fd809aac0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=dc935a5629d5d340 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T10:32:03.771 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:32:03.758+0000 7f0fc2ff5640 10 client.0 ms_handle_connect on v2:172.21.15.114:3300/0 2024-04-23T10:32:03.771 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:32:03.758+0000 7f0fc2ff5640 1 -- 172.21.15.114:0/1371697131 <== mon.0 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0fd80ad030 con 0x55c8d0ef00a0 2024-04-23T10:32:03.771 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:32:03.758+0000 7f0fe81f4740 1 -- 172.21.15.114:0/1371697131 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f0fd80a0960 con 0x55c8d0ef00a0 2024-04-23T10:32:03.771 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:32:03.758+0000 7f0fc2ff5640 1 -- 172.21.15.114:0/1371697131 <== mon.0 v2:172.21.15.114:3300/0 2 ==== osd_map(94..94 src has 1..94) ==== 8585+0+0 (secure 0 0 0) 0x7f0fd80bb460 con 0x55c8d0ef00a0 2024-04-23T10:32:03.771 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:32:03.758+0000 7f0fc2ff5640 1 -- 172.21.15.114:0/1371697131 <== mon.0 v2:172.21.15.114:3300/0 3 ==== mon_get_version_reply(handle=1 version=30) ==== 24+0+0 (secure 0 0 0) 0x7f0fd808e050 con 0x55c8d0ef00a0 2024-04-23T10:32:03.771 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:32:03.759+0000 7f0fe81f4740 10 client.16302 fetch_fsmap learned FSMap version 30 2024-04-23T10:32:03.772 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:32:03.759+0000 7f0fe81f4740 1 -- 172.21.15.114:0/1371697131 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({config=0+,fsmap=30,monmap=4+}) -- 0x55c8d0f679e0 con 0x55c8d0ef00a0 2024-04-23T10:32:03.772 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:32:03.759+0000 7f0fc2ff5640 1 -- 172.21.15.114:0/1371697131 <== mon.0 v2:172.21.15.114:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f0fd80a0960 con 0x55c8d0ef00a0 2024-04-23T10:32:03.772 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:32:03.759+0000 7f0fc2ff5640 1 -- 172.21.15.114:0/1371697131 <== mon.0 v2:172.21.15.114:3300/0 5 ==== fsmap(e 30) ==== 8338+0+0 (secure 0 0 0) 0x7f0fd8004e00 con 0x55c8d0ef00a0 2024-04-23T10:32:03.772 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:32:03.759+0000 7f0fe81f4740 10 client.16302 fetch_fsmap finished waiting for FSMap version 30 2024-04-23T10:32:03.772 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:32:03.759+0000 7f0fe81f4740 10 client.16302 resolve_mds: resolved name 'c' to daemon mds.c 2024-04-23T10:32:03.772 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:32:03.759+0000 7f0fe81f4740 20 client.16302 populate_metadata read hostname 'smithi114' 2024-04-23T10:32:03.772 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:32:03.759+0000 7f0fe81f4740 1 --2- 172.21.15.114:0/1371697131 >> [v2:172.21.15.204:6836/513822179,v1:172.21.15.204:6837/513822179] conn(0x55c8d0f7c7f0 0x55c8d0f7cbd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T10:32:03.772 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:32:03.759+0000 7f0fe81f4740 4 client.16302 mds_command: new command op to 24439 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-23T10:32:03.772 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:32:03.759+0000 7f0fe81f4740 1 -- 172.21.15.114:0/1371697131 --> [v2:172.21.15.204:6836/513822179,v1:172.21.15.204:6837/513822179] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55c8d0f679e0 con 0x55c8d0f7c7f0 2024-04-23T10:32:03.772 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:32:03.759+0000 7f0fdeffd640 1 --2- 172.21.15.114:0/1371697131 >> [v2:172.21.15.204:6836/513822179,v1:172.21.15.204:6837/513822179] conn(0x55c8d0f7c7f0 0x55c8d0f7cbd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T10:32:03.772 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:32:03.759+0000 7f0fc2ff5640 10 client.16302 ms_handle_connect on v2:172.21.15.204:6836/513822179 2024-04-23T10:32:03.772 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:32:03.759+0000 7f0fdeffd640 1 --2- 172.21.15.114:0/1371697131 >> [v2:172.21.15.204:6836/513822179,v1:172.21.15.204:6837/513822179] conn(0x55c8d0f7c7f0 0x55c8d0f7cbd0 crc :-1 s=READY pgs=21 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.24439 client_cookie=a8a4955ff80607ed server_cookie=d08d139779d196b4 in_seq=0 out_seq=0 2024-04-23T10:32:03.772 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:32:03.760+0000 7f0fc2ff5640 1 -- 172.21.15.114:0/1371697131 <== mds.24439 v2:172.21.15.204:6836/513822179 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x55c8d0f679e0 con 0x55c8d0f7c7f0 2024-04-23T10:32:03.772 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:32:03.760+0000 7f0fc2ff5640 10 client.16302 handle_command_reply: tid=0 2024-04-23T10:32:03.773 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:32:03.760+0000 7f0fe81f4740 1 client.16302 shutdown 2024-04-23T10:32:03.773 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:32:03.760+0000 7f0fe81f4740 1 -- 172.21.15.114:0/1371697131 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55c8d0ef00a0 msgr2=0x55c8d0f686e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T10:32:03.773 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:32:03.760+0000 7f0fe81f4740 1 --2- 172.21.15.114:0/1371697131 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55c8d0ef00a0 0x55c8d0f686e0 secure :-1 s=READY pgs=707 cs=0 l=1 rev1=1 crypto rx=0x7f0fd809c2b0 tx=0x7f0fd809aac0 comp rx=0 tx=0).stop 2024-04-23T10:32:03.773 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:32:03.760+0000 7f0fe81f4740 1 -- 172.21.15.114:0/1371697131 shutdown_connections 2024-04-23T10:32:03.773 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:32:03.760+0000 7f0fe81f4740 1 --2- 172.21.15.114:0/1371697131 >> [v2:172.21.15.204:6836/513822179,v1:172.21.15.204:6837/513822179] conn(0x55c8d0f7c7f0 0x55c8d0f7cbd0 crc :-1 s=READY pgs=21 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:32:03.773 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:32:03.760+0000 7f0fe81f4740 1 --2- 172.21.15.114:0/1371697131 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55c8d0ef00a0 0x55c8d0f686e0 unknown :-1 s=CLOSED pgs=707 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:32:03.773 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:32:03.760+0000 7f0fc2ff5640 0 client.16302 ms_handle_reset on v2:172.21.15.204:6836/513822179 2024-04-23T10:32:03.773 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:32:03.760+0000 7f0fe81f4740 1 --2- 172.21.15.114:0/1371697131 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x55c8d0ee14c0 0x55c8d0f6ed10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:32:03.773 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:32:03.760+0000 7f0fe81f4740 1 --2- 172.21.15.114:0/1371697131 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x55c8d0ee0ba0 0x55c8d0eea600 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:32:03.773 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:32:03.760+0000 7f0fe81f4740 1 -- 172.21.15.114:0/1371697131 >> 172.21.15.114:0/1371697131 conn(0x55c8d0edcfc0 msgr2=0x55c8d0f61b30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T10:32:03.773 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:32:03.760+0000 7f0fe81f4740 1 -- 172.21.15.114:0/1371697131 shutdown_connections 2024-04-23T10:32:03.773 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:32:03.760+0000 7f0fe81f4740 1 -- 172.21.15.114:0/1371697131 wait complete. 2024-04-23T10:32:03.773 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:32:03.761+0000 7f0fe81f4740 20 client.16302 trim_cache size 0 max 16384 2024-04-23T10:32:03.781 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:32:03.776+0000 7f0fe81f4740 1 Processor -- start 2024-04-23T10:32:03.781 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:32:03.776+0000 7f0fe81f4740 1 -- start start 2024-04-23T10:32:03.781 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:32:03.776+0000 7f0fe81f4740 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x55c8d0ee0ba0 0x55c8d0f7cbd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T10:32:03.781 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:32:03.776+0000 7f0fe81f4740 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55c8d0ee14c0 0x55c8d0f0aff0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T10:32:03.781 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:32:03.776+0000 7f0fe81f4740 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x55c8d0ef00a0 0x55c8d0f6bdb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T10:32:03.781 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:32:03.776+0000 7f0fe81f4740 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x55c8d0f7ba20 con 0x55c8d0ee14c0 2024-04-23T10:32:03.781 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:32:03.776+0000 7f0fe81f4740 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 -- 0x55c8d0f68180 con 0x55c8d0ee0ba0 2024-04-23T10:32:03.781 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:32:03.776+0000 7f0fe81f4740 1 -- --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_getmap magic: 0 -- 0x55c8d0f02e00 con 0x55c8d0ef00a0 2024-04-23T10:32:03.781 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:32:03.776+0000 7f0fde7fc640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55c8d0ee14c0 0x55c8d0f0aff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T10:32:03.781 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:32:03.776+0000 7f0fde7fc640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55c8d0ee14c0 0x55c8d0f0aff0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.114:51934/0 (socket says 172.21.15.114:51934) 2024-04-23T10:32:03.781 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:32:03.776+0000 7f0fde7fc640 1 -- 172.21.15.114:0/2864787093 learned_addr learned my addr 172.21.15.114:0/2864787093 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-04-23T10:32:03.782 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:32:03.776+0000 7f0fdf7fe640 1 --2- 172.21.15.114:0/2864787093 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x55c8d0ef00a0 0x55c8d0f6bdb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T10:32:03.782 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:32:03.776+0000 7f0fde7fc640 1 -- 172.21.15.114:0/2864787093 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x55c8d0ef00a0 msgr2=0x55c8d0f6bdb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T10:32:03.782 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:32:03.776+0000 7f0fde7fc640 1 --2- 172.21.15.114:0/2864787093 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x55c8d0ef00a0 0x55c8d0f6bdb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:32:03.782 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:32:03.776+0000 7f0fde7fc640 1 -- 172.21.15.114:0/2864787093 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x55c8d0ee0ba0 msgr2=0x55c8d0f7cbd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T10:32:03.782 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:32:03.776+0000 7f0fdeffd640 1 --2- 172.21.15.114:0/2864787093 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x55c8d0ee0ba0 0x55c8d0f7cbd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T10:32:03.782 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:32:03.776+0000 7f0fde7fc640 1 --2- 172.21.15.114:0/2864787093 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x55c8d0ee0ba0 0x55c8d0f7cbd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:32:03.782 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:32:03.776+0000 7f0fde7fc640 1 -- 172.21.15.114:0/2864787093 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55c8d0f679e0 con 0x55c8d0ee14c0 2024-04-23T10:32:03.782 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:32:03.776+0000 7f0fdf7fe640 1 --2- 172.21.15.114:0/2864787093 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x55c8d0ef00a0 0x55c8d0f6bdb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-23T10:32:03.782 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:32:03.776+0000 7f0fdeffd640 1 --2- 172.21.15.114:0/2864787093 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x55c8d0ee0ba0 0x55c8d0f7cbd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-23T10:32:03.782 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:32:03.776+0000 7f0fde7fc640 1 --2- 172.21.15.114:0/2864787093 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55c8d0ee14c0 0x55c8d0f0aff0 secure :-1 s=READY pgs=708 cs=0 l=1 rev1=1 crypto rx=0x55c8d0eedf60 tx=0x7f0fd4009f40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e49fb032cd5682be server_cookie=0 in_seq=0 out_seq=0 2024-04-23T10:32:03.782 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:32:03.776+0000 7f0fc4ff9640 1 -- 172.21.15.114:0/2864787093 <== mon.0 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0fd402f040 con 0x55c8d0ee14c0 2024-04-23T10:32:03.782 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:32:03.776+0000 7f0fc4ff9640 1 -- 172.21.15.114:0/2864787093 <== mon.0 v2:172.21.15.114:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f0fd4036790 con 0x55c8d0ee14c0 2024-04-23T10:32:03.782 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:32:03.776+0000 7f0fc4ff9640 1 -- 172.21.15.114:0/2864787093 <== mon.0 v2:172.21.15.114:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0fd4064070 con 0x55c8d0ee14c0 2024-04-23T10:32:03.782 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:32:03.777+0000 7f0fe81f4740 1 -- 172.21.15.114:0/2864787093 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55c8d0ee14c0 msgr2=0x55c8d0f0aff0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T10:32:03.783 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:32:03.777+0000 7f0fe81f4740 1 --2- 172.21.15.114:0/2864787093 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55c8d0ee14c0 0x55c8d0f0aff0 secure :-1 s=READY pgs=708 cs=0 l=1 rev1=1 crypto rx=0x55c8d0eedf60 tx=0x7f0fd4009f40 comp rx=0 tx=0).stop 2024-04-23T10:32:03.783 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:32:03.777+0000 7f0fe81f4740 1 -- 172.21.15.114:0/2864787093 shutdown_connections 2024-04-23T10:32:03.783 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:32:03.777+0000 7f0fe81f4740 1 --2- 172.21.15.114:0/2864787093 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x55c8d0ef00a0 0x55c8d0f6bdb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:32:03.783 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:32:03.777+0000 7f0fe81f4740 1 --2- 172.21.15.114:0/2864787093 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55c8d0ee14c0 0x55c8d0f0aff0 unknown :-1 s=CLOSED pgs=708 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:32:03.783 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:32:03.777+0000 7f0fe81f4740 1 --2- 172.21.15.114:0/2864787093 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x55c8d0ee0ba0 0x55c8d0f7cbd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:32:03.783 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:32:03.777+0000 7f0fe81f4740 1 -- 172.21.15.114:0/2864787093 >> 172.21.15.114:0/2864787093 conn(0x55c8d0edcfc0 msgr2=0x55c8d0ee8460 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T10:32:03.783 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:32:03.777+0000 7f0fe81f4740 1 -- 172.21.15.114:0/2864787093 shutdown_connections 2024-04-23T10:32:03.783 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:32:03.777+0000 7f0fe81f4740 1 -- 172.21.15.114:0/2864787093 wait complete. 2024-04-23T10:32:03.783 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:32:03.777+0000 7f0fe81f4740 1 -- start start 2024-04-23T10:32:03.783 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:32:03.777+0000 7f0fe81f4740 1 Processor -- start 2024-04-23T10:32:03.783 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:32:03.777+0000 7f0fe81f4740 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55c8d0ee0ba0 0x55c8d0f7cbd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T10:32:03.783 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:32:03.778+0000 7f0fe81f4740 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x55c8d0ee14c0 0x55c8d0f6bdb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T10:32:03.783 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:32:03.778+0000 7f0fe81f4740 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x55c8d0ef00a0 0x55c8d0ed8280 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T10:32:03.783 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:32:03.778+0000 7f0fe81f4740 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x55c8d0ee7860 con 0x55c8d0ee0ba0 2024-04-23T10:32:03.783 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:32:03.778+0000 7f0fe81f4740 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 -- 0x55c8d0f6ea30 con 0x55c8d0ee14c0 2024-04-23T10:32:03.784 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:32:03.778+0000 7f0fe81f4740 1 -- --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_getmap magic: 0 -- 0x55c8d0f0fa90 con 0x55c8d0ef00a0 2024-04-23T10:32:03.784 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:32:03.778+0000 7f0fe81f4740 10 client.0 _ll_register_callbacks cb 0x7f0fd40099b0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-23T10:32:03.784 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:32:03.778+0000 7f0fdeffd640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55c8d0ee0ba0 0x55c8d0f7cbd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T10:32:03.784 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:32:03.778+0000 7f0fdeffd640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55c8d0ee0ba0 0x55c8d0f7cbd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.114:51942/0 (socket says 172.21.15.114:51942) 2024-04-23T10:32:03.784 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:32:03.778+0000 7f0fdeffd640 1 -- 172.21.15.114:0/2490268543 learned_addr learned my addr 172.21.15.114:0/2490268543 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-04-23T10:32:03.784 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:32:03.778+0000 7f0fde7fc640 1 --2- 172.21.15.114:0/2490268543 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x55c8d0ee14c0 0x55c8d0f6bdb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T10:32:03.784 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:32:03.778+0000 7f0fdeffd640 1 -- 172.21.15.114:0/2490268543 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x55c8d0ef00a0 msgr2=0x55c8d0ed8280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T10:32:03.784 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:32:03.778+0000 7f0fdeffd640 1 --2- 172.21.15.114:0/2490268543 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x55c8d0ef00a0 0x55c8d0ed8280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:32:03.784 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:32:03.778+0000 7f0fdeffd640 1 -- 172.21.15.114:0/2490268543 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x55c8d0ee14c0 msgr2=0x55c8d0f6bdb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T10:32:03.784 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:32:03.778+0000 7f0fdeffd640 1 --2- 172.21.15.114:0/2490268543 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x55c8d0ee14c0 0x55c8d0f6bdb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:32:03.784 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:32:03.778+0000 7f0fdeffd640 1 -- 172.21.15.114:0/2490268543 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55c8d0f67f50 con 0x55c8d0ee0ba0 2024-04-23T10:32:03.784 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:32:03.778+0000 7f0fdeffd640 1 --2- 172.21.15.114:0/2490268543 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55c8d0ee0ba0 0x55c8d0f7cbd0 secure :-1 s=READY pgs=709 cs=0 l=1 rev1=1 crypto rx=0x7f0fd005f220 tx=0x7f0fd005f040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3d10e6c34cf980f3 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T10:32:03.784 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:32:03.778+0000 7f0fc2ff5640 10 client.0 ms_handle_connect on v2:172.21.15.114:3300/0 2024-04-23T10:32:03.784 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:32:03.778+0000 7f0fc2ff5640 1 -- 172.21.15.114:0/2490268543 <== mon.0 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0fd0077040 con 0x55c8d0ee0ba0 2024-04-23T10:32:03.785 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:32:03.778+0000 7f0fc2ff5640 1 -- 172.21.15.114:0/2490268543 <== mon.0 v2:172.21.15.114:3300/0 2 ==== osd_map(94..94 src has 1..94) ==== 8585+0+0 (secure 0 0 0) 0x7f0fd0031bb0 con 0x55c8d0ee0ba0 2024-04-23T10:32:03.785 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:32:03.778+0000 7f0fe81f4740 1 -- 172.21.15.114:0/2490268543 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f0fd4036790 con 0x55c8d0ee0ba0 2024-04-23T10:32:03.785 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:32:03.778+0000 7f0fc2ff5640 1 -- 172.21.15.114:0/2490268543 <== mon.0 v2:172.21.15.114:3300/0 3 ==== mon_get_version_reply(handle=1 version=30) ==== 24+0+0 (secure 0 0 0) 0x7f0fd0003c00 con 0x55c8d0ee0ba0 2024-04-23T10:32:03.785 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:32:03.778+0000 7f0fe81f4740 10 client.16308 fetch_fsmap learned FSMap version 30 2024-04-23T10:32:03.785 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:32:03.778+0000 7f0fe81f4740 1 -- 172.21.15.114:0/2490268543 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({config=0+,fsmap=30,monmap=4+}) -- 0x55c8d0f68e90 con 0x55c8d0ee0ba0 2024-04-23T10:32:03.785 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:32:03.778+0000 7f0fc2ff5640 1 -- 172.21.15.114:0/2490268543 <== mon.0 v2:172.21.15.114:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f0fd4036790 con 0x55c8d0ee0ba0 2024-04-23T10:32:03.785 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:32:03.778+0000 7f0fc2ff5640 1 -- 172.21.15.114:0/2490268543 <== mon.0 v2:172.21.15.114:3300/0 5 ==== fsmap(e 30) ==== 8338+0+0 (secure 0 0 0) 0x7f0fd006f040 con 0x55c8d0ee0ba0 2024-04-23T10:32:03.785 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:32:03.779+0000 7f0fe81f4740 10 client.16308 fetch_fsmap finished waiting for FSMap version 30 2024-04-23T10:32:03.785 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:32:03.779+0000 7f0fe81f4740 10 client.16308 resolve_mds: resolved name 'c' to daemon mds.c 2024-04-23T10:32:03.785 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:32:03.779+0000 7f0fe81f4740 20 client.16308 populate_metadata read hostname 'smithi114' 2024-04-23T10:32:03.785 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:32:03.779+0000 7f0fe81f4740 1 --2- 172.21.15.114:0/2490268543 >> [v2:172.21.15.204:6836/513822179,v1:172.21.15.204:6837/513822179] conn(0x55c8d0f7c7f0 0x55c8d0f6ebb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T10:32:03.785 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:32:03.779+0000 7f0fe81f4740 4 client.16308 mds_command: new command op to 24439 tid=0 multi_id=0 [{"prefix": "ops", "flags": ["locks"]}] 2024-04-23T10:32:03.785 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:32:03.779+0000 7f0fe81f4740 1 -- 172.21.15.114:0/2490268543 --> [v2:172.21.15.204:6836/513822179,v1:172.21.15.204:6837/513822179] -- command(tid 0: {"prefix": "ops", "flags": ["locks"]}) -- 0x55c8d0f619b0 con 0x55c8d0f7c7f0 2024-04-23T10:32:03.785 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:32:03.779+0000 7f0fde7fc640 1 --2- 172.21.15.114:0/2490268543 >> [v2:172.21.15.204:6836/513822179,v1:172.21.15.204:6837/513822179] conn(0x55c8d0f7c7f0 0x55c8d0f6ebb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T10:32:03.786 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:32:03.779+0000 7f0fde7fc640 1 --2- 172.21.15.114:0/2490268543 >> [v2:172.21.15.204:6836/513822179,v1:172.21.15.204:6837/513822179] conn(0x55c8d0f7c7f0 0x55c8d0f6ebb0 crc :-1 s=READY pgs=22 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.24439 client_cookie=7a89d4a7f3ab3680 server_cookie=633757acffde8d06 in_seq=0 out_seq=0 2024-04-23T10:32:03.786 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:32:03.779+0000 7f0fc2ff5640 10 client.16308 ms_handle_connect on v2:172.21.15.204:6836/513822179 2024-04-23T10:32:03.786 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:32:03.779+0000 7f0fc2ff5640 1 -- 172.21.15.114:0/2490268543 <== mds.24439 v2:172.21.15.204:6836/513822179 1 ==== command_reply(tid 0: 0 ) ==== 8+0+36 (crc 0 0 0) 0x7f0fd4003a10 con 0x55c8d0f7c7f0 2024-04-23T10:32:03.786 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:32:03.779+0000 7f0fc2ff5640 10 client.16308 handle_command_reply: tid=0 2024-04-23T10:32:03.786 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:32:03.779+0000 7f0fe81f4740 1 client.16308 shutdown 2024-04-23T10:32:03.786 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:32:03.779+0000 7f0fe81f4740 1 -- 172.21.15.114:0/2490268543 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55c8d0ee0ba0 msgr2=0x55c8d0f7cbd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T10:32:03.786 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:32:03.779+0000 7f0fe81f4740 1 --2- 172.21.15.114:0/2490268543 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55c8d0ee0ba0 0x55c8d0f7cbd0 secure :-1 s=READY pgs=709 cs=0 l=1 rev1=1 crypto rx=0x7f0fd005f220 tx=0x7f0fd005f040 comp rx=0 tx=0).stop 2024-04-23T10:32:03.786 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:32:03.780+0000 7f0fe81f4740 1 -- 172.21.15.114:0/2490268543 shutdown_connections 2024-04-23T10:32:03.786 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:32:03.780+0000 7f0fe81f4740 1 --2- 172.21.15.114:0/2490268543 >> [v2:172.21.15.204:6836/513822179,v1:172.21.15.204:6837/513822179] conn(0x55c8d0f7c7f0 0x55c8d0f6ebb0 crc :-1 s=READY pgs=22 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:32:03.786 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:32:03.780+0000 7f0fe81f4740 1 --2- 172.21.15.114:0/2490268543 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x55c8d0ef00a0 0x55c8d0ed8280 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:32:03.786 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:32:03.780+0000 7f0fe81f4740 1 --2- 172.21.15.114:0/2490268543 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x55c8d0ee14c0 0x55c8d0f6bdb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:32:03.786 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:32:03.780+0000 7f0fc2ff5640 0 client.16308 ms_handle_reset on v2:172.21.15.204:6836/513822179 2024-04-23T10:32:03.786 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:32:03.780+0000 7f0fe81f4740 1 --2- 172.21.15.114:0/2490268543 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55c8d0ee0ba0 0x55c8d0f7cbd0 unknown :-1 s=CLOSED pgs=709 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:32:03.787 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:32:03.780+0000 7f0fe81f4740 1 -- 172.21.15.114:0/2490268543 >> 172.21.15.114:0/2490268543 conn(0x55c8d0edcfc0 msgr2=0x55c8d0eed7c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T10:32:03.787 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:32:03.780+0000 7f0fe81f4740 1 -- 172.21.15.114:0/2490268543 shutdown_connections 2024-04-23T10:32:03.787 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:32:03.780+0000 7f0fe81f4740 1 -- 172.21.15.114:0/2490268543 wait complete. 2024-04-23T10:32:03.787 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:32:03.780+0000 7f0fe81f4740 20 client.16308 trim_cache size 0 max 16384 2024-04-23T10:32:03.787 INFO:teuthology.orchestra.run.smithi114.stdout:mds.c: { 2024-04-23T10:32:03.787 INFO:teuthology.orchestra.run.smithi114.stdout: "ops": [], 2024-04-23T10:32:03.787 INFO:teuthology.orchestra.run.smithi114.stdout: "num_ops": 0 2024-04-23T10:32:03.787 INFO:teuthology.orchestra.run.smithi114.stdout:} 2024-04-23T10:32:03.793 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:32:03.780+0000 7f0fe81f4740 1 Processor -- start 2024-04-23T10:32:03.793 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:32:03.780+0000 7f0fe81f4740 1 -- start start 2024-04-23T10:32:03.793 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:32:03.780+0000 7f0fe81f4740 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x55c8d0ee0ba0 0x55c8d0f7cbd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T10:32:03.793 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:32:03.780+0000 7f0fe81f4740 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55c8d0ee14c0 0x55c8d0f6bdb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T10:32:03.793 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:32:03.780+0000 7f0fe81f4740 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x55c8d0ef00a0 0x55c8d0f6ebb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T10:32:03.793 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:32:03.780+0000 7f0fe81f4740 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x55c8d0edc490 con 0x55c8d0ee14c0 2024-04-23T10:32:03.794 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:32:03.780+0000 7f0fe81f4740 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 -- 0x55c8d0f60070 con 0x55c8d0ee0ba0 2024-04-23T10:32:03.794 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:32:03.780+0000 7f0fe81f4740 1 -- --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_getmap magic: 0 -- 0x55c8d0f61b70 con 0x55c8d0ef00a0 2024-04-23T10:32:03.794 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:32:03.780+0000 7f0fde7fc640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55c8d0ee14c0 0x55c8d0f6bdb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T10:32:03.794 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:32:03.780+0000 7f0fde7fc640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55c8d0ee14c0 0x55c8d0f6bdb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.114:51944/0 (socket says 172.21.15.114:51944) 2024-04-23T10:32:03.794 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:32:03.780+0000 7f0fde7fc640 1 -- 172.21.15.114:0/2640583993 learned_addr learned my addr 172.21.15.114:0/2640583993 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-04-23T10:32:03.794 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:32:03.780+0000 7f0fdeffd640 1 --2- 172.21.15.114:0/2640583993 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x55c8d0ee0ba0 0x55c8d0f7cbd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T10:32:03.794 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:32:03.780+0000 7f0fdf7fe640 1 --2- 172.21.15.114:0/2640583993 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x55c8d0ef00a0 0x55c8d0f6ebb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T10:32:03.794 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:32:03.780+0000 7f0fde7fc640 1 -- 172.21.15.114:0/2640583993 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x55c8d0ef00a0 msgr2=0x55c8d0f6ebb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T10:32:03.794 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:32:03.780+0000 7f0fde7fc640 1 --2- 172.21.15.114:0/2640583993 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x55c8d0ef00a0 0x55c8d0f6ebb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:32:03.794 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:32:03.780+0000 7f0fde7fc640 1 -- 172.21.15.114:0/2640583993 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x55c8d0ee0ba0 msgr2=0x55c8d0f7cbd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T10:32:03.794 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:32:03.781+0000 7f0fde7fc640 1 --2- 172.21.15.114:0/2640583993 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x55c8d0ee0ba0 0x55c8d0f7cbd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:32:03.794 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:32:03.781+0000 7f0fde7fc640 1 -- 172.21.15.114:0/2640583993 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55c8d0f020b0 con 0x55c8d0ee14c0 2024-04-23T10:32:03.794 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:32:03.781+0000 7f0fdeffd640 1 --2- 172.21.15.114:0/2640583993 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x55c8d0ee0ba0 0x55c8d0f7cbd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-23T10:32:03.794 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:32:03.781+0000 7f0fde7fc640 1 --2- 172.21.15.114:0/2640583993 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55c8d0ee14c0 0x55c8d0f6bdb0 secure :-1 s=READY pgs=710 cs=0 l=1 rev1=1 crypto rx=0x7f0fd4009880 tx=0x7f0fd4005fd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=179b0ca0aa486368 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T10:32:03.795 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:32:03.781+0000 7f0fc57fa640 1 -- 172.21.15.114:0/2640583993 <== mon.0 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0fd4064040 con 0x55c8d0ee14c0 2024-04-23T10:32:03.795 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:32:03.781+0000 7f0fc57fa640 1 -- 172.21.15.114:0/2640583993 <== mon.0 v2:172.21.15.114:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f0fd4003a10 con 0x55c8d0ee14c0 2024-04-23T10:32:03.795 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:32:03.781+0000 7f0fc57fa640 1 -- 172.21.15.114:0/2640583993 <== mon.0 v2:172.21.15.114:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0fd402f040 con 0x55c8d0ee14c0 2024-04-23T10:32:03.795 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:32:03.781+0000 7f0fe81f4740 1 -- 172.21.15.114:0/2640583993 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55c8d0ee14c0 msgr2=0x55c8d0f6bdb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T10:32:03.795 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:32:03.781+0000 7f0fe81f4740 1 --2- 172.21.15.114:0/2640583993 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55c8d0ee14c0 0x55c8d0f6bdb0 secure :-1 s=READY pgs=710 cs=0 l=1 rev1=1 crypto rx=0x7f0fd4009880 tx=0x7f0fd4005fd0 comp rx=0 tx=0).stop 2024-04-23T10:32:03.795 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:32:03.781+0000 7f0fe81f4740 1 -- 172.21.15.114:0/2640583993 shutdown_connections 2024-04-23T10:32:03.795 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:32:03.781+0000 7f0fe81f4740 1 --2- 172.21.15.114:0/2640583993 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x55c8d0ef00a0 0x55c8d0f6ebb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:32:03.795 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:32:03.781+0000 7f0fe81f4740 1 --2- 172.21.15.114:0/2640583993 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55c8d0ee14c0 0x55c8d0f6bdb0 unknown :-1 s=CLOSED pgs=710 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:32:03.795 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:32:03.781+0000 7f0fe81f4740 1 --2- 172.21.15.114:0/2640583993 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x55c8d0ee0ba0 0x55c8d0f7cbd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:32:03.795 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:32:03.781+0000 7f0fe81f4740 1 -- 172.21.15.114:0/2640583993 >> 172.21.15.114:0/2640583993 conn(0x55c8d0edcfc0 msgr2=0x55c8d0ee8460 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T10:32:03.795 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:32:03.781+0000 7f0fe81f4740 1 -- 172.21.15.114:0/2640583993 shutdown_connections 2024-04-23T10:32:03.795 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:32:03.781+0000 7f0fe81f4740 1 -- 172.21.15.114:0/2640583993 wait complete. 2024-04-23T10:32:03.795 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:32:03.782+0000 7f0fe81f4740 1 -- start start 2024-04-23T10:32:03.795 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:32:03.782+0000 7f0fe81f4740 1 Processor -- start 2024-04-23T10:32:03.795 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:32:03.782+0000 7f0fe81f4740 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55c8d0ee0ba0 0x55c8d0f6bdb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T10:32:03.796 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:32:03.782+0000 7f0fe81f4740 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x55c8d0ee14c0 0x55c8d0f6ebb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T10:32:03.796 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:32:03.782+0000 7f0fe81f4740 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x55c8d0ef00a0 0x55c8d0f0aff0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T10:32:03.796 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:32:03.782+0000 7f0fe81f4740 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x55c8d0f6f100 con 0x55c8d0ee0ba0 2024-04-23T10:32:03.796 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:32:03.782+0000 7f0fe81f4740 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 -- 0x55c8d0f6ea30 con 0x55c8d0ef00a0 2024-04-23T10:32:03.796 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:32:03.782+0000 7f0fe81f4740 1 -- --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_getmap magic: 0 -- 0x55c8d0f08840 con 0x55c8d0ee14c0 2024-04-23T10:32:03.796 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:32:03.782+0000 7f0fe81f4740 10 client.0 _ll_register_callbacks cb 0x7f0fd0005cd0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-23T10:32:03.796 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:32:03.782+0000 7f0fdeffd640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55c8d0ee0ba0 0x55c8d0f6bdb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T10:32:03.796 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:32:03.782+0000 7f0fdeffd640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55c8d0ee0ba0 0x55c8d0f6bdb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.114:51954/0 (socket says 172.21.15.114:51954) 2024-04-23T10:32:03.796 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:32:03.782+0000 7f0fdeffd640 1 -- 172.21.15.114:0/3702534993 learned_addr learned my addr 172.21.15.114:0/3702534993 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-04-23T10:32:03.796 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:32:03.782+0000 7f0fde7fc640 1 --2- 172.21.15.114:0/3702534993 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x55c8d0ee14c0 0x55c8d0f6ebb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T10:32:03.796 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:32:03.782+0000 7f0fdf7fe640 1 --2- 172.21.15.114:0/3702534993 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x55c8d0ef00a0 0x55c8d0f0aff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T10:32:03.796 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:32:03.782+0000 7f0fdeffd640 1 -- 172.21.15.114:0/3702534993 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x55c8d0ee14c0 msgr2=0x55c8d0f6ebb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T10:32:03.796 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:32:03.782+0000 7f0fdeffd640 1 --2- 172.21.15.114:0/3702534993 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x55c8d0ee14c0 0x55c8d0f6ebb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:32:03.796 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:32:03.782+0000 7f0fdeffd640 1 -- 172.21.15.114:0/3702534993 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x55c8d0ef00a0 msgr2=0x55c8d0f0aff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T10:32:03.796 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:32:03.782+0000 7f0fdeffd640 1 --2- 172.21.15.114:0/3702534993 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x55c8d0ef00a0 0x55c8d0f0aff0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:32:03.797 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:32:03.782+0000 7f0fdeffd640 1 -- 172.21.15.114:0/3702534993 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55c8d0eefa40 con 0x55c8d0ee0ba0 2024-04-23T10:32:03.797 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:32:03.782+0000 7f0fdf7fe640 1 --2- 172.21.15.114:0/3702534993 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x55c8d0ef00a0 0x55c8d0f0aff0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-23T10:32:03.797 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:32:03.782+0000 7f0fdeffd640 1 --2- 172.21.15.114:0/3702534993 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55c8d0ee0ba0 0x55c8d0f6bdb0 secure :-1 s=READY pgs=711 cs=0 l=1 rev1=1 crypto rx=0x7f0fd005f220 tx=0x7f0fd005f130 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4db19627552cf436 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T10:32:03.797 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:32:03.782+0000 7f0fc2ff5640 10 client.0 ms_handle_connect on v2:172.21.15.114:3300/0 2024-04-23T10:32:03.797 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:32:03.782+0000 7f0fc2ff5640 1 -- 172.21.15.114:0/3702534993 <== mon.0 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0fd0077040 con 0x55c8d0ee0ba0 2024-04-23T10:32:03.797 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:32:03.782+0000 7f0fc2ff5640 1 -- 172.21.15.114:0/3702534993 <== mon.0 v2:172.21.15.114:3300/0 2 ==== osd_map(94..94 src has 1..94) ==== 8585+0+0 (secure 0 0 0) 0x7f0fd00318b0 con 0x55c8d0ee0ba0 2024-04-23T10:32:03.797 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:32:03.782+0000 7f0fe81f4740 1 -- 172.21.15.114:0/3702534993 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f0fd4003a10 con 0x55c8d0ee0ba0 2024-04-23T10:32:03.797 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:32:03.783+0000 7f0fc2ff5640 1 -- 172.21.15.114:0/3702534993 <== mon.0 v2:172.21.15.114:3300/0 3 ==== mon_get_version_reply(handle=1 version=30) ==== 24+0+0 (secure 0 0 0) 0x7f0fd005f250 con 0x55c8d0ee0ba0 2024-04-23T10:32:03.797 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:32:03.783+0000 7f0fe81f4740 10 client.16314 fetch_fsmap learned FSMap version 30 2024-04-23T10:32:03.797 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:32:03.783+0000 7f0fe81f4740 1 -- 172.21.15.114:0/3702534993 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({config=0+,fsmap=30,monmap=4+}) -- 0x55c8d0eefa40 con 0x55c8d0ee0ba0 2024-04-23T10:32:03.797 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:32:03.783+0000 7f0fc2ff5640 1 -- 172.21.15.114:0/3702534993 <== mon.0 v2:172.21.15.114:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f0fd4003a10 con 0x55c8d0ee0ba0 2024-04-23T10:32:03.797 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:32:03.783+0000 7f0fc2ff5640 1 -- 172.21.15.114:0/3702534993 <== mon.0 v2:172.21.15.114:3300/0 5 ==== fsmap(e 30) ==== 8338+0+0 (secure 0 0 0) 0x7f0fd006f040 con 0x55c8d0ee0ba0 2024-04-23T10:32:03.797 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:32:03.783+0000 7f0fe81f4740 10 client.16314 fetch_fsmap finished waiting for FSMap version 30 2024-04-23T10:32:03.797 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:32:03.783+0000 7f0fe81f4740 10 client.16314 resolve_mds: resolved name 'k' to daemon mds.k 2024-04-23T10:32:03.798 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:32:03.783+0000 7f0fe81f4740 20 client.16314 populate_metadata read hostname 'smithi114' 2024-04-23T10:32:03.798 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:32:03.783+0000 7f0fe81f4740 1 --2- 172.21.15.114:0/3702534993 >> [v2:172.21.15.119:6832/783244660,v1:172.21.15.119:6833/783244660] conn(0x55c8d0f609a0 0x55c8d0f0f660 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T10:32:03.798 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:32:03.783+0000 7f0fe81f4740 4 client.16314 mds_command: new command op to 24443 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-23T10:32:03.798 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:32:03.783+0000 7f0fe81f4740 1 -- 172.21.15.114:0/3702534993 --> [v2:172.21.15.119:6832/783244660,v1:172.21.15.119:6833/783244660] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55c8d0eefa40 con 0x55c8d0f609a0 2024-04-23T10:32:03.798 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:32:03.783+0000 7f0fde7fc640 1 --2- 172.21.15.114:0/3702534993 >> [v2:172.21.15.119:6832/783244660,v1:172.21.15.119:6833/783244660] conn(0x55c8d0f609a0 0x55c8d0f0f660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T10:32:03.798 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:32:03.784+0000 7f0fc2ff5640 10 client.16314 ms_handle_connect on v2:172.21.15.119:6832/783244660 2024-04-23T10:32:03.798 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:32:03.784+0000 7f0fde7fc640 1 --2- 172.21.15.114:0/3702534993 >> [v2:172.21.15.119:6832/783244660,v1:172.21.15.119:6833/783244660] conn(0x55c8d0f609a0 0x55c8d0f0f660 crc :-1 s=READY pgs=18 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.24443 client_cookie=f4d7938596d1c25f server_cookie=a913732f53931334 in_seq=0 out_seq=0 2024-04-23T10:32:03.798 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:32:03.784+0000 7f0fc2ff5640 1 -- 172.21.15.114:0/3702534993 <== mds.24443 v2:172.21.15.119:6832/783244660 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x55c8d0eefa40 con 0x55c8d0f609a0 2024-04-23T10:32:03.798 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:32:03.784+0000 7f0fc2ff5640 10 client.16314 handle_command_reply: tid=0 2024-04-23T10:32:03.798 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:32:03.784+0000 7f0fe81f4740 1 client.16314 shutdown 2024-04-23T10:32:03.798 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:32:03.784+0000 7f0fe81f4740 1 -- 172.21.15.114:0/3702534993 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55c8d0ee0ba0 msgr2=0x55c8d0f6bdb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T10:32:03.798 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:32:03.784+0000 7f0fe81f4740 1 --2- 172.21.15.114:0/3702534993 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55c8d0ee0ba0 0x55c8d0f6bdb0 secure :-1 s=READY pgs=711 cs=0 l=1 rev1=1 crypto rx=0x7f0fd005f220 tx=0x7f0fd005f130 comp rx=0 tx=0).stop 2024-04-23T10:32:03.798 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:32:03.784+0000 7f0fe81f4740 1 -- 172.21.15.114:0/3702534993 shutdown_connections 2024-04-23T10:32:03.798 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:32:03.784+0000 7f0fe81f4740 1 --2- 172.21.15.114:0/3702534993 >> [v2:172.21.15.119:6832/783244660,v1:172.21.15.119:6833/783244660] conn(0x55c8d0f609a0 0x55c8d0f0f660 crc :-1 s=READY pgs=18 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:32:03.799 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:32:03.784+0000 7f0fe81f4740 1 --2- 172.21.15.114:0/3702534993 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x55c8d0ef00a0 0x55c8d0f0aff0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:32:03.799 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:32:03.784+0000 7f0fe81f4740 1 --2- 172.21.15.114:0/3702534993 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x55c8d0ee14c0 0x55c8d0f6ebb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:32:03.799 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:32:03.784+0000 7f0fc2ff5640 0 client.16314 ms_handle_reset on v2:172.21.15.119:6832/783244660 2024-04-23T10:32:03.799 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:32:03.784+0000 7f0fe81f4740 1 --2- 172.21.15.114:0/3702534993 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55c8d0ee0ba0 0x55c8d0f6bdb0 unknown :-1 s=CLOSED pgs=711 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:32:03.799 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:32:03.784+0000 7f0fe81f4740 1 -- 172.21.15.114:0/3702534993 >> 172.21.15.114:0/3702534993 conn(0x55c8d0edcfc0 msgr2=0x55c8d0f69710 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T10:32:03.799 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:32:03.784+0000 7f0fe81f4740 1 -- 172.21.15.114:0/3702534993 shutdown_connections 2024-04-23T10:32:03.799 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:32:03.785+0000 7f0fe81f4740 1 -- 172.21.15.114:0/3702534993 wait complete. 2024-04-23T10:32:03.799 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:32:03.785+0000 7f0fe81f4740 20 client.16314 trim_cache size 0 max 16384 2024-04-23T10:32:03.806 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:32:03.799+0000 7f0fe81f4740 1 Processor -- start 2024-04-23T10:32:03.806 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:32:03.799+0000 7f0fe81f4740 1 -- start start 2024-04-23T10:32:03.806 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:32:03.800+0000 7f0fe81f4740 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55c8d0ee0ba0 0x55c8d0f7cbd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T10:32:03.806 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:32:03.800+0000 7f0fe81f4740 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x55c8d0ee14c0 0x55c8d0f6bdb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T10:32:03.806 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:32:03.800+0000 7f0fe81f4740 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x55c8d0ef00a0 0x55c8d0f0aff0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T10:32:03.806 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:32:03.800+0000 7f0fe81f4740 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x55c8d0f68e90 con 0x55c8d0ee0ba0 2024-04-23T10:32:03.806 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:32:03.800+0000 7f0fe81f4740 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 -- 0x55c8d0efc160 con 0x55c8d0ef00a0 2024-04-23T10:32:03.806 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:32:03.800+0000 7f0fe81f4740 1 -- --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_getmap magic: 0 -- 0x55c8d0f6af10 con 0x55c8d0ee14c0 2024-04-23T10:32:03.806 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:32:03.800+0000 7f0fdeffd640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55c8d0ee0ba0 0x55c8d0f7cbd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T10:32:03.806 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:32:03.800+0000 7f0fdeffd640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55c8d0ee0ba0 0x55c8d0f7cbd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.114:51960/0 (socket says 172.21.15.114:51960) 2024-04-23T10:32:03.806 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:32:03.800+0000 7f0fdeffd640 1 -- 172.21.15.114:0/2222458412 learned_addr learned my addr 172.21.15.114:0/2222458412 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-04-23T10:32:03.807 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:32:03.800+0000 7f0fdf7fe640 1 --2- 172.21.15.114:0/2222458412 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x55c8d0ef00a0 0x55c8d0f0aff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T10:32:03.807 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:32:03.800+0000 7f0fdeffd640 1 -- 172.21.15.114:0/2222458412 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x55c8d0ee14c0 msgr2=0x55c8d0f6bdb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T10:32:03.807 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:32:03.800+0000 7f0fde7fc640 1 --2- 172.21.15.114:0/2222458412 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x55c8d0ee14c0 0x55c8d0f6bdb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T10:32:03.807 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:32:03.800+0000 7f0fdeffd640 1 --2- 172.21.15.114:0/2222458412 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x55c8d0ee14c0 0x55c8d0f6bdb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:32:03.807 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:32:03.800+0000 7f0fdeffd640 1 -- 172.21.15.114:0/2222458412 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x55c8d0ef00a0 msgr2=0x55c8d0f0aff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T10:32:03.807 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:32:03.800+0000 7f0fdeffd640 1 --2- 172.21.15.114:0/2222458412 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x55c8d0ef00a0 0x55c8d0f0aff0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:32:03.807 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:32:03.800+0000 7f0fdeffd640 1 -- 172.21.15.114:0/2222458412 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55c8d0f679e0 con 0x55c8d0ee0ba0 2024-04-23T10:32:03.807 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:32:03.800+0000 7f0fdeffd640 1 --2- 172.21.15.114:0/2222458412 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55c8d0ee0ba0 0x55c8d0f7cbd0 secure :-1 s=READY pgs=712 cs=0 l=1 rev1=1 crypto rx=0x7f0fd002a900 tx=0x7f0fd002af40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4c98a961c2cf2fd9 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T10:32:03.807 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:32:03.800+0000 7f0fc4ff9640 1 -- 172.21.15.114:0/2222458412 <== mon.0 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0fd0077040 con 0x55c8d0ee0ba0 2024-04-23T10:32:03.807 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:32:03.800+0000 7f0fc4ff9640 1 -- 172.21.15.114:0/2222458412 <== mon.0 v2:172.21.15.114:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f0fd0036040 con 0x55c8d0ee0ba0 2024-04-23T10:32:03.807 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:32:03.800+0000 7f0fc4ff9640 1 -- 172.21.15.114:0/2222458412 <== mon.0 v2:172.21.15.114:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0fd0004640 con 0x55c8d0ee0ba0 2024-04-23T10:32:03.807 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:32:03.801+0000 7f0fe81f4740 1 -- 172.21.15.114:0/2222458412 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55c8d0ee0ba0 msgr2=0x55c8d0f7cbd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T10:32:03.807 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:32:03.801+0000 7f0fe81f4740 1 --2- 172.21.15.114:0/2222458412 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55c8d0ee0ba0 0x55c8d0f7cbd0 secure :-1 s=READY pgs=712 cs=0 l=1 rev1=1 crypto rx=0x7f0fd002a900 tx=0x7f0fd002af40 comp rx=0 tx=0).stop 2024-04-23T10:32:03.807 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:32:03.801+0000 7f0fe81f4740 1 -- 172.21.15.114:0/2222458412 shutdown_connections 2024-04-23T10:32:03.808 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:32:03.801+0000 7f0fe81f4740 1 --2- 172.21.15.114:0/2222458412 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x55c8d0ef00a0 0x55c8d0f0aff0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:32:03.808 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:32:03.801+0000 7f0fe81f4740 1 --2- 172.21.15.114:0/2222458412 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x55c8d0ee14c0 0x55c8d0f6bdb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:32:03.808 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:32:03.801+0000 7f0fe81f4740 1 --2- 172.21.15.114:0/2222458412 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55c8d0ee0ba0 0x55c8d0f7cbd0 unknown :-1 s=CLOSED pgs=712 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:32:03.808 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:32:03.801+0000 7f0fe81f4740 1 -- 172.21.15.114:0/2222458412 >> 172.21.15.114:0/2222458412 conn(0x55c8d0edcfc0 msgr2=0x55c8d0ee8460 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T10:32:03.808 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:32:03.801+0000 7f0fe81f4740 1 -- 172.21.15.114:0/2222458412 shutdown_connections 2024-04-23T10:32:03.808 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:32:03.801+0000 7f0fe81f4740 1 -- 172.21.15.114:0/2222458412 wait complete. 2024-04-23T10:32:03.808 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:32:03.801+0000 7f0fe81f4740 1 -- start start 2024-04-23T10:32:03.808 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:32:03.801+0000 7f0fe81f4740 1 Processor -- start 2024-04-23T10:32:03.808 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:32:03.801+0000 7f0fe81f4740 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x55c8d0ee0ba0 0x55c8d0f0aff0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T10:32:03.808 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:32:03.801+0000 7f0fe81f4740 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x55c8d0ee14c0 0x55c8d0ee7430 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T10:32:03.808 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:32:03.801+0000 7f0fe81f4740 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55c8d0ef00a0 0x55c8d0edfe70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T10:32:03.808 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:32:03.801+0000 7f0fe81f4740 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x55c8d0eed940 con 0x55c8d0ef00a0 2024-04-23T10:32:03.808 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:32:03.801+0000 7f0fe81f4740 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 -- 0x55c8d0f61cf0 con 0x55c8d0ee14c0 2024-04-23T10:32:03.808 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:32:03.801+0000 7f0fe81f4740 1 -- --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_getmap magic: 0 -- 0x55c8d0f67170 con 0x55c8d0ee0ba0 2024-04-23T10:32:03.808 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:32:03.801+0000 7f0fe81f4740 10 client.0 _ll_register_callbacks cb 0x7f0fd007b070 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-23T10:32:03.809 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:32:03.802+0000 7f0fdf7fe640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55c8d0ef00a0 0x55c8d0edfe70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T10:32:03.809 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:32:03.802+0000 7f0fdf7fe640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55c8d0ef00a0 0x55c8d0edfe70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.114:51976/0 (socket says 172.21.15.114:51976) 2024-04-23T10:32:03.809 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:32:03.802+0000 7f0fdf7fe640 1 -- 172.21.15.114:0/4049826438 learned_addr learned my addr 172.21.15.114:0/4049826438 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-04-23T10:32:03.809 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:32:03.802+0000 7f0fdeffd640 1 --2- 172.21.15.114:0/4049826438 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x55c8d0ee0ba0 0x55c8d0f0aff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T10:32:03.809 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:32:03.802+0000 7f0fde7fc640 1 --2- 172.21.15.114:0/4049826438 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x55c8d0ee14c0 0x55c8d0ee7430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T10:32:03.809 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:32:03.802+0000 7f0fdf7fe640 1 -- 172.21.15.114:0/4049826438 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x55c8d0ee0ba0 msgr2=0x55c8d0f0aff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T10:32:03.809 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:32:03.802+0000 7f0fdf7fe640 1 --2- 172.21.15.114:0/4049826438 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x55c8d0ee0ba0 0x55c8d0f0aff0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:32:03.809 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:32:03.802+0000 7f0fdf7fe640 1 -- 172.21.15.114:0/4049826438 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x55c8d0ee14c0 msgr2=0x55c8d0ee7430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T10:32:03.809 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:32:03.802+0000 7f0fdf7fe640 1 --2- 172.21.15.114:0/4049826438 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x55c8d0ee14c0 0x55c8d0ee7430 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:32:03.809 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:32:03.802+0000 7f0fdf7fe640 1 -- 172.21.15.114:0/4049826438 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55c8d0efb6c0 con 0x55c8d0ef00a0 2024-04-23T10:32:03.809 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:32:03.802+0000 7f0fde7fc640 1 --2- 172.21.15.114:0/4049826438 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x55c8d0ee14c0 0x55c8d0ee7430 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-23T10:32:03.809 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:32:03.802+0000 7f0fdf7fe640 1 --2- 172.21.15.114:0/4049826438 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55c8d0ef00a0 0x55c8d0edfe70 secure :-1 s=READY pgs=713 cs=0 l=1 rev1=1 crypto rx=0x7f0fd8098a10 tx=0x7f0fd8009f00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=de818e251d33bc11 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T10:32:03.809 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:32:03.802+0000 7f0fc2ff5640 10 client.0 ms_handle_connect on v2:172.21.15.114:3300/0 2024-04-23T10:32:03.809 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:32:03.802+0000 7f0fc2ff5640 1 -- 172.21.15.114:0/4049826438 <== mon.0 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0fd80ad070 con 0x55c8d0ef00a0 2024-04-23T10:32:03.810 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:32:03.802+0000 7f0fc2ff5640 1 -- 172.21.15.114:0/4049826438 <== mon.0 v2:172.21.15.114:3300/0 2 ==== osd_map(94..94 src has 1..94) ==== 8585+0+0 (secure 0 0 0) 0x7f0fd8098510 con 0x55c8d0ef00a0 2024-04-23T10:32:03.810 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:32:03.802+0000 7f0fe81f4740 1 -- 172.21.15.114:0/4049826438 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f0fd0036040 con 0x55c8d0ef00a0 2024-04-23T10:32:03.810 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:32:03.802+0000 7f0fc2ff5640 1 -- 172.21.15.114:0/4049826438 <== mon.0 v2:172.21.15.114:3300/0 3 ==== mon_get_version_reply(handle=1 version=30) ==== 24+0+0 (secure 0 0 0) 0x7f0fd808e040 con 0x55c8d0ef00a0 2024-04-23T10:32:03.810 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:32:03.802+0000 7f0fe81f4740 10 client.16320 fetch_fsmap learned FSMap version 30 2024-04-23T10:32:03.810 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:32:03.802+0000 7f0fe81f4740 1 -- 172.21.15.114:0/4049826438 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({config=0+,fsmap=30,monmap=4+}) -- 0x55c8d0efb6c0 con 0x55c8d0ef00a0 2024-04-23T10:32:03.810 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:32:03.802+0000 7f0fc2ff5640 1 -- 172.21.15.114:0/4049826438 <== mon.0 v2:172.21.15.114:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f0fd0036040 con 0x55c8d0ef00a0 2024-04-23T10:32:03.810 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:32:03.802+0000 7f0fc2ff5640 1 -- 172.21.15.114:0/4049826438 <== mon.0 v2:172.21.15.114:3300/0 5 ==== fsmap(e 30) ==== 8338+0+0 (secure 0 0 0) 0x7f0fd8004e00 con 0x55c8d0ef00a0 2024-04-23T10:32:03.810 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:32:03.802+0000 7f0fe81f4740 10 client.16320 fetch_fsmap finished waiting for FSMap version 30 2024-04-23T10:32:03.810 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:32:03.803+0000 7f0fe81f4740 10 client.16320 resolve_mds: resolved name 'k' to daemon mds.k 2024-04-23T10:32:03.810 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:32:03.803+0000 7f0fe81f4740 20 client.16320 populate_metadata read hostname 'smithi114' 2024-04-23T10:32:03.810 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:32:03.803+0000 7f0fe81f4740 1 --2- 172.21.15.114:0/4049826438 >> [v2:172.21.15.119:6832/783244660,v1:172.21.15.119:6833/783244660] conn(0x55c8d0f68700 0x55c8d0f6ed90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T10:32:03.810 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:32:03.803+0000 7f0fe81f4740 4 client.16320 mds_command: new command op to 24443 tid=0 multi_id=0 [{"prefix": "ops", "flags": ["locks"]}] 2024-04-23T10:32:03.810 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:32:03.803+0000 7f0fe81f4740 1 -- 172.21.15.114:0/4049826438 --> [v2:172.21.15.119:6832/783244660,v1:172.21.15.119:6833/783244660] -- command(tid 0: {"prefix": "ops", "flags": ["locks"]}) -- 0x55c8d0edc2c0 con 0x55c8d0f68700 2024-04-23T10:32:03.810 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:32:03.803+0000 7f0fdeffd640 1 --2- 172.21.15.114:0/4049826438 >> [v2:172.21.15.119:6832/783244660,v1:172.21.15.119:6833/783244660] conn(0x55c8d0f68700 0x55c8d0f6ed90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T10:32:03.810 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:32:03.803+0000 7f0fc2ff5640 10 client.16320 ms_handle_connect on v2:172.21.15.119:6832/783244660 2024-04-23T10:32:03.811 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:32:03.803+0000 7f0fdeffd640 1 --2- 172.21.15.114:0/4049826438 >> [v2:172.21.15.119:6832/783244660,v1:172.21.15.119:6833/783244660] conn(0x55c8d0f68700 0x55c8d0f6ed90 crc :-1 s=READY pgs=19 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.24443 client_cookie=29d8fdd6229f137 server_cookie=9c191ebcfa00054b in_seq=0 out_seq=0 2024-04-23T10:32:03.811 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:32:03.803+0000 7f0fc2ff5640 1 -- 172.21.15.114:0/4049826438 <== mds.24443 v2:172.21.15.119:6832/783244660 1 ==== command_reply(tid 0: 0 ) ==== 8+0+36 (crc 0 0 0) 0x55c8d0edc2c0 con 0x55c8d0f68700 2024-04-23T10:32:03.811 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:32:03.803+0000 7f0fc2ff5640 10 client.16320 handle_command_reply: tid=0 2024-04-23T10:32:03.811 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:32:03.803+0000 7f0fe81f4740 1 client.16320 shutdown 2024-04-23T10:32:03.811 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:32:03.803+0000 7f0fe81f4740 1 -- 172.21.15.114:0/4049826438 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55c8d0ef00a0 msgr2=0x55c8d0edfe70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T10:32:03.811 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:32:03.803+0000 7f0fe81f4740 1 --2- 172.21.15.114:0/4049826438 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55c8d0ef00a0 0x55c8d0edfe70 secure :-1 s=READY pgs=713 cs=0 l=1 rev1=1 crypto rx=0x7f0fd8098a10 tx=0x7f0fd8009f00 comp rx=0 tx=0).stop 2024-04-23T10:32:03.811 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:32:03.803+0000 7f0fe81f4740 1 -- 172.21.15.114:0/4049826438 shutdown_connections 2024-04-23T10:32:03.811 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:32:03.803+0000 7f0fe81f4740 1 --2- 172.21.15.114:0/4049826438 >> [v2:172.21.15.119:6832/783244660,v1:172.21.15.119:6833/783244660] conn(0x55c8d0f68700 0x55c8d0f6ed90 crc :-1 s=READY pgs=19 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:32:03.811 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:32:03.803+0000 7f0fe81f4740 1 --2- 172.21.15.114:0/4049826438 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55c8d0ef00a0 0x55c8d0edfe70 unknown :-1 s=CLOSED pgs=713 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:32:03.811 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:32:03.803+0000 7f0fe81f4740 1 --2- 172.21.15.114:0/4049826438 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x55c8d0ee14c0 0x55c8d0ee7430 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:32:03.811 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:32:03.803+0000 7f0fc2ff5640 0 client.16320 ms_handle_reset on v2:172.21.15.119:6832/783244660 2024-04-23T10:32:03.811 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:32:03.803+0000 7f0fe81f4740 1 --2- 172.21.15.114:0/4049826438 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x55c8d0ee0ba0 0x55c8d0f0aff0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:32:03.811 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:32:03.803+0000 7f0fe81f4740 1 -- 172.21.15.114:0/4049826438 >> 172.21.15.114:0/4049826438 conn(0x55c8d0edcfc0 msgr2=0x55c8d0f0f660 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T10:32:03.812 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:32:03.804+0000 7f0fe81f4740 1 -- 172.21.15.114:0/4049826438 shutdown_connections 2024-04-23T10:32:03.812 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:32:03.804+0000 7f0fe81f4740 1 -- 172.21.15.114:0/4049826438 wait complete. 2024-04-23T10:32:03.812 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:32:03.804+0000 7f0fe81f4740 20 client.16320 trim_cache size 0 max 16384 2024-04-23T10:32:03.812 INFO:teuthology.orchestra.run.smithi114.stdout:mds.k: { 2024-04-23T10:32:03.812 INFO:teuthology.orchestra.run.smithi114.stdout: "ops": [], 2024-04-23T10:32:03.812 INFO:teuthology.orchestra.run.smithi114.stdout: "num_ops": 0 2024-04-23T10:32:03.812 INFO:teuthology.orchestra.run.smithi114.stdout:} 2024-04-23T10:32:03.818 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:32:03.804+0000 7f0fe81f4740 1 Processor -- start 2024-04-23T10:32:03.818 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:32:03.804+0000 7f0fe81f4740 1 -- start start 2024-04-23T10:32:03.818 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:32:03.804+0000 7f0fe81f4740 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x55c8d0ee0ba0 0x55c8d0f7cbd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T10:32:03.818 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:32:03.804+0000 7f0fe81f4740 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x55c8d0ee14c0 0x55c8d0f68700 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T10:32:03.818 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:32:03.804+0000 7f0fe81f4740 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55c8d0ef00a0 0x55c8d0edfe70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T10:32:03.819 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:32:03.804+0000 7f0fe81f4740 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x55c8d0f6b090 con 0x55c8d0ef00a0 2024-04-23T10:32:03.819 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:32:03.804+0000 7f0fe81f4740 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 -- 0x55c8d0f61e30 con 0x55c8d0ee14c0 2024-04-23T10:32:03.819 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:32:03.804+0000 7f0fe81f4740 1 -- --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_getmap magic: 0 -- 0x55c8d0eedae0 con 0x55c8d0ee0ba0 2024-04-23T10:32:03.819 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:32:03.804+0000 7f0fdf7fe640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55c8d0ef00a0 0x55c8d0edfe70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T10:32:03.819 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:32:03.804+0000 7f0fdf7fe640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55c8d0ef00a0 0x55c8d0edfe70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.114:51984/0 (socket says 172.21.15.114:51984) 2024-04-23T10:32:03.819 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:32:03.804+0000 7f0fdf7fe640 1 -- 172.21.15.114:0/235982082 learned_addr learned my addr 172.21.15.114:0/235982082 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-04-23T10:32:03.819 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:32:03.804+0000 7f0fde7fc640 1 --2- 172.21.15.114:0/235982082 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x55c8d0ee14c0 0x55c8d0f68700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T10:32:03.819 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:32:03.804+0000 7f0fdeffd640 1 --2- 172.21.15.114:0/235982082 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x55c8d0ee0ba0 0x55c8d0f7cbd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T10:32:03.819 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:32:03.804+0000 7f0fdf7fe640 1 -- 172.21.15.114:0/235982082 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x55c8d0ee0ba0 msgr2=0x55c8d0f7cbd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T10:32:03.819 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:32:03.804+0000 7f0fdf7fe640 1 --2- 172.21.15.114:0/235982082 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x55c8d0ee0ba0 0x55c8d0f7cbd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:32:03.819 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:32:03.804+0000 7f0fdf7fe640 1 -- 172.21.15.114:0/235982082 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x55c8d0ee14c0 msgr2=0x55c8d0f68700 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T10:32:03.819 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:32:03.804+0000 7f0fdf7fe640 1 --2- 172.21.15.114:0/235982082 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x55c8d0ee14c0 0x55c8d0f68700 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:32:03.819 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:32:03.804+0000 7f0fdf7fe640 1 -- 172.21.15.114:0/235982082 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55c8d0f679e0 con 0x55c8d0ef00a0 2024-04-23T10:32:03.820 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:32:03.805+0000 7f0fdf7fe640 1 --2- 172.21.15.114:0/235982082 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55c8d0ef00a0 0x55c8d0edfe70 secure :-1 s=READY pgs=714 cs=0 l=1 rev1=1 crypto rx=0x7f0fd809c1a0 tx=0x7f0fd8098250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=53fe83ae04f5770d server_cookie=0 in_seq=0 out_seq=0 2024-04-23T10:32:03.820 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:32:03.805+0000 7f0fc57fa640 1 -- 172.21.15.114:0/235982082 <== mon.0 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0fd80ad050 con 0x55c8d0ef00a0 2024-04-23T10:32:03.820 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:32:03.805+0000 7f0fc57fa640 1 -- 172.21.15.114:0/235982082 <== mon.0 v2:172.21.15.114:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f0fd800b0b0 con 0x55c8d0ef00a0 2024-04-23T10:32:03.820 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:32:03.805+0000 7f0fc57fa640 1 -- 172.21.15.114:0/235982082 <== mon.0 v2:172.21.15.114:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0fd8014070 con 0x55c8d0ef00a0 2024-04-23T10:32:03.820 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:32:03.805+0000 7f0fe81f4740 1 -- 172.21.15.114:0/235982082 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55c8d0ef00a0 msgr2=0x55c8d0edfe70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T10:32:03.820 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:32:03.805+0000 7f0fe81f4740 1 --2- 172.21.15.114:0/235982082 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55c8d0ef00a0 0x55c8d0edfe70 secure :-1 s=READY pgs=714 cs=0 l=1 rev1=1 crypto rx=0x7f0fd809c1a0 tx=0x7f0fd8098250 comp rx=0 tx=0).stop 2024-04-23T10:32:03.820 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:32:03.805+0000 7f0fe81f4740 1 -- 172.21.15.114:0/235982082 shutdown_connections 2024-04-23T10:32:03.820 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:32:03.805+0000 7f0fe81f4740 1 --2- 172.21.15.114:0/235982082 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55c8d0ef00a0 0x55c8d0edfe70 unknown :-1 s=CLOSED pgs=714 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:32:03.820 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:32:03.805+0000 7f0fe81f4740 1 --2- 172.21.15.114:0/235982082 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x55c8d0ee14c0 0x55c8d0f68700 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:32:03.820 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:32:03.805+0000 7f0fe81f4740 1 --2- 172.21.15.114:0/235982082 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x55c8d0ee0ba0 0x55c8d0f7cbd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:32:03.820 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:32:03.805+0000 7f0fe81f4740 1 -- 172.21.15.114:0/235982082 >> 172.21.15.114:0/235982082 conn(0x55c8d0edcfc0 msgr2=0x55c8d0f6a5d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T10:32:03.820 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:32:03.805+0000 7f0fe81f4740 1 -- 172.21.15.114:0/235982082 shutdown_connections 2024-04-23T10:32:03.820 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:32:03.805+0000 7f0fe81f4740 1 -- 172.21.15.114:0/235982082 wait complete. 2024-04-23T10:32:03.820 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:32:03.805+0000 7f0fe81f4740 1 -- start start 2024-04-23T10:32:03.821 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:32:03.806+0000 7f0fe81f4740 1 Processor -- start 2024-04-23T10:32:03.821 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:32:03.806+0000 7f0fe81f4740 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x55c8d0ee0ba0 0x55c8d0f7cbd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T10:32:03.821 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:32:03.806+0000 7f0fe81f4740 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55c8d0ee14c0 0x55c8d0f68700 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T10:32:03.821 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:32:03.806+0000 7f0fe81f4740 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x55c8d0ef00a0 0x55c8d0edfe70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T10:32:03.821 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:32:03.806+0000 7f0fe81f4740 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x55c8d0f6ec10 con 0x55c8d0ee14c0 2024-04-23T10:32:03.821 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:32:03.806+0000 7f0fe81f4740 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 -- 0x55c8d0ed8300 con 0x55c8d0ef00a0 2024-04-23T10:32:03.821 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:32:03.806+0000 7f0fe81f4740 1 -- --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_getmap magic: 0 -- 0x55c8d0f60d80 con 0x55c8d0ee0ba0 2024-04-23T10:32:03.821 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:32:03.806+0000 7f0fe81f4740 10 client.0 _ll_register_callbacks cb 0x7f0fd808c070 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-23T10:32:03.821 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:32:03.806+0000 7f0fde7fc640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55c8d0ee14c0 0x55c8d0f68700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T10:32:03.821 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:32:03.806+0000 7f0fde7fc640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55c8d0ee14c0 0x55c8d0f68700 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.114:51988/0 (socket says 172.21.15.114:51988) 2024-04-23T10:32:03.821 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:32:03.806+0000 7f0fdeffd640 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x55c8d0ee0ba0 0x55c8d0f7cbd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T10:32:03.821 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:32:03.806+0000 7f0fde7fc640 1 -- 172.21.15.114:0/4111150061 learned_addr learned my addr 172.21.15.114:0/4111150061 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-04-23T10:32:03.821 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:32:03.806+0000 7f0fdf7fe640 1 --2- 172.21.15.114:0/4111150061 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x55c8d0ef00a0 0x55c8d0edfe70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T10:32:03.821 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:32:03.806+0000 7f0fde7fc640 1 -- 172.21.15.114:0/4111150061 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x55c8d0ee0ba0 msgr2=0x55c8d0f7cbd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T10:32:03.821 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:32:03.806+0000 7f0fde7fc640 1 --2- 172.21.15.114:0/4111150061 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x55c8d0ee0ba0 0x55c8d0f7cbd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:32:03.822 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:32:03.806+0000 7f0fde7fc640 1 -- 172.21.15.114:0/4111150061 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x55c8d0ef00a0 msgr2=0x55c8d0edfe70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T10:32:03.822 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:32:03.806+0000 7f0fde7fc640 1 --2- 172.21.15.114:0/4111150061 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x55c8d0ef00a0 0x55c8d0edfe70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:32:03.822 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:32:03.806+0000 7f0fde7fc640 1 -- 172.21.15.114:0/4111150061 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55c8d0f02f80 con 0x55c8d0ee14c0 2024-04-23T10:32:03.822 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:32:03.806+0000 7f0fdf7fe640 1 --2- 172.21.15.114:0/4111150061 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x55c8d0ef00a0 0x55c8d0edfe70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-23T10:32:03.822 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:32:03.806+0000 7f0fde7fc640 1 --2- 172.21.15.114:0/4111150061 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55c8d0ee14c0 0x55c8d0f68700 secure :-1 s=READY pgs=715 cs=0 l=1 rev1=1 crypto rx=0x7f0fd4014040 tx=0x7f0fd40058f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=671e1f3a38b3326 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T10:32:03.822 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:32:03.806+0000 7f0fc2ff5640 10 client.0 ms_handle_connect on v2:172.21.15.114:3300/0 2024-04-23T10:32:03.822 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:32:03.806+0000 7f0fc2ff5640 1 -- 172.21.15.114:0/4111150061 <== mon.0 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0fd4064040 con 0x55c8d0ee14c0 2024-04-23T10:32:03.822 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:32:03.806+0000 7f0fc2ff5640 1 -- 172.21.15.114:0/4111150061 <== mon.0 v2:172.21.15.114:3300/0 2 ==== osd_map(94..94 src has 1..94) ==== 8585+0+0 (secure 0 0 0) 0x7f0fd4000bc0 con 0x55c8d0ee14c0 2024-04-23T10:32:03.822 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:32:03.806+0000 7f0fe81f4740 1 -- 172.21.15.114:0/4111150061 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f0fd800b0b0 con 0x55c8d0ee14c0 2024-04-23T10:32:03.822 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:32:03.807+0000 7f0fc2ff5640 1 -- 172.21.15.114:0/4111150061 <== mon.0 v2:172.21.15.114:3300/0 3 ==== mon_get_version_reply(handle=1 version=30) ==== 24+0+0 (secure 0 0 0) 0x7f0fd402f040 con 0x55c8d0ee14c0 2024-04-23T10:32:03.822 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:32:03.807+0000 7f0fe81f4740 10 client.16326 fetch_fsmap learned FSMap version 30 2024-04-23T10:32:03.822 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:32:03.807+0000 7f0fe81f4740 1 -- 172.21.15.114:0/4111150061 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({config=0+,fsmap=30,monmap=4+}) -- 0x55c8d0f02f80 con 0x55c8d0ee14c0 2024-04-23T10:32:03.822 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:32:03.807+0000 7f0fc2ff5640 1 -- 172.21.15.114:0/4111150061 <== mon.0 v2:172.21.15.114:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f0fd800b0b0 con 0x55c8d0ee14c0 2024-04-23T10:32:03.823 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:32:03.807+0000 7f0fc2ff5640 1 -- 172.21.15.114:0/4111150061 <== mon.0 v2:172.21.15.114:3300/0 5 ==== fsmap(e 30) ==== 8338+0+0 (secure 0 0 0) 0x7f0fd4036c50 con 0x55c8d0ee14c0 2024-04-23T10:32:03.823 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:32:03.807+0000 7f0fe81f4740 10 client.16326 fetch_fsmap finished waiting for FSMap version 30 2024-04-23T10:32:03.823 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:32:03.807+0000 7f0fe81f4740 10 client.16326 resolve_mds: resolved name 'e' to daemon mds.e 2024-04-23T10:32:03.823 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:32:03.807+0000 7f0fe81f4740 20 client.16326 populate_metadata read hostname 'smithi114' 2024-04-23T10:32:03.823 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:32:03.807+0000 7f0fe81f4740 1 --2- 172.21.15.114:0/4111150061 >> [v2:172.21.15.119:6834/2033916709,v1:172.21.15.119:6835/2033916709] conn(0x55c8d0ee9ba0 0x55c8d0f71db0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T10:32:03.823 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:32:03.807+0000 7f0fe81f4740 4 client.16326 mds_command: new command op to 24479 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-23T10:32:03.823 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:32:03.807+0000 7f0fe81f4740 1 -- 172.21.15.114:0/4111150061 --> [v2:172.21.15.119:6834/2033916709,v1:172.21.15.119:6835/2033916709] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55c8d0f69fc0 con 0x55c8d0ee9ba0 2024-04-23T10:32:03.823 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:32:03.807+0000 7f0fdeffd640 1 --2- 172.21.15.114:0/4111150061 >> [v2:172.21.15.119:6834/2033916709,v1:172.21.15.119:6835/2033916709] conn(0x55c8d0ee9ba0 0x55c8d0f71db0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T10:32:03.823 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:32:03.807+0000 7f0fc2ff5640 10 client.16326 ms_handle_connect on v2:172.21.15.119:6834/2033916709 2024-04-23T10:32:03.823 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:32:03.807+0000 7f0fdeffd640 1 --2- 172.21.15.114:0/4111150061 >> [v2:172.21.15.119:6834/2033916709,v1:172.21.15.119:6835/2033916709] conn(0x55c8d0ee9ba0 0x55c8d0f71db0 crc :-1 s=READY pgs=35 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.2 client_cookie=9e82b6c4b5521215 server_cookie=352962d10b883d3e in_seq=0 out_seq=0 2024-04-23T10:32:03.823 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:32:03.808+0000 7f0fc2ff5640 1 -- 172.21.15.114:0/4111150061 <== mds.2 v2:172.21.15.119:6834/2033916709 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x55c8d0f69fc0 con 0x55c8d0ee9ba0 2024-04-23T10:32:03.823 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:32:03.808+0000 7f0fc2ff5640 10 client.16326 handle_command_reply: tid=0 2024-04-23T10:32:03.823 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:32:03.808+0000 7f0fe81f4740 1 client.16326 shutdown 2024-04-23T10:32:03.823 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:32:03.808+0000 7f0fe81f4740 1 -- 172.21.15.114:0/4111150061 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55c8d0ee14c0 msgr2=0x55c8d0f68700 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T10:32:03.824 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:32:03.808+0000 7f0fe81f4740 1 --2- 172.21.15.114:0/4111150061 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55c8d0ee14c0 0x55c8d0f68700 secure :-1 s=READY pgs=715 cs=0 l=1 rev1=1 crypto rx=0x7f0fd4014040 tx=0x7f0fd40058f0 comp rx=0 tx=0).stop 2024-04-23T10:32:03.824 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:32:03.808+0000 7f0fe81f4740 1 -- 172.21.15.114:0/4111150061 shutdown_connections 2024-04-23T10:32:03.824 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:32:03.808+0000 7f0fe81f4740 1 --2- 172.21.15.114:0/4111150061 >> [v2:172.21.15.119:6834/2033916709,v1:172.21.15.119:6835/2033916709] conn(0x55c8d0ee9ba0 0x55c8d0f71db0 crc :-1 s=READY pgs=35 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:32:03.824 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:32:03.808+0000 7f0fe81f4740 1 --2- 172.21.15.114:0/4111150061 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x55c8d0ef00a0 0x55c8d0edfe70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:32:03.824 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:32:03.808+0000 7f0fc2ff5640 0 client.16326 ms_handle_reset on v2:172.21.15.119:6834/2033916709 2024-04-23T10:32:03.824 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:32:03.808+0000 7f0fe81f4740 1 --2- 172.21.15.114:0/4111150061 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55c8d0ee14c0 0x55c8d0f68700 unknown :-1 s=CLOSED pgs=715 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:32:03.824 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:32:03.808+0000 7f0fe81f4740 1 --2- 172.21.15.114:0/4111150061 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x55c8d0ee0ba0 0x55c8d0f7cbd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:32:03.824 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:32:03.808+0000 7f0fe81f4740 1 -- 172.21.15.114:0/4111150061 >> 172.21.15.114:0/4111150061 conn(0x55c8d0edcfc0 msgr2=0x55c8d0f6a5d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T10:32:03.824 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:32:03.808+0000 7f0fe81f4740 1 -- 172.21.15.114:0/4111150061 shutdown_connections 2024-04-23T10:32:03.824 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:32:03.808+0000 7f0fe81f4740 1 -- 172.21.15.114:0/4111150061 wait complete. 2024-04-23T10:32:03.824 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:32:03.808+0000 7f0fe81f4740 20 client.16326 trim_cache size 0 max 16384 2024-04-23T10:32:03.829 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:32:03.823+0000 7f0fe81f4740 1 Processor -- start 2024-04-23T10:32:03.829 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:32:03.823+0000 7f0fe81f4740 1 -- start start 2024-04-23T10:32:03.830 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:32:03.824+0000 7f0fe81f4740 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x55c8d0ee0ba0 0x55c8d0f7cbd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T10:32:03.830 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:32:03.824+0000 7f0fe81f4740 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55c8d0ee14c0 0x55c8d0ee75b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T10:32:03.830 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:32:03.824+0000 7f0fe81f4740 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x55c8d0ef00a0 0x55c8d0f0aff0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T10:32:03.830 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:32:03.824+0000 7f0fe81f4740 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x55c8d0eefdd0 con 0x55c8d0ee14c0 2024-04-23T10:32:03.830 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:32:03.824+0000 7f0fe81f4740 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 -- 0x55c8d0efd390 con 0x55c8d0ee0ba0 2024-04-23T10:32:03.830 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:32:03.824+0000 7f0fe81f4740 1 -- --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_getmap magic: 0 -- 0x55c8d0edc6b0 con 0x55c8d0ef00a0 2024-04-23T10:32:03.830 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:32:03.824+0000 7f0fde7fc640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55c8d0ee14c0 0x55c8d0ee75b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T10:32:03.830 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:32:03.824+0000 7f0fde7fc640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55c8d0ee14c0 0x55c8d0ee75b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.114:51994/0 (socket says 172.21.15.114:51994) 2024-04-23T10:32:03.830 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:32:03.824+0000 7f0fde7fc640 1 -- 172.21.15.114:0/2362136826 learned_addr learned my addr 172.21.15.114:0/2362136826 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-04-23T10:32:03.830 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:32:03.824+0000 7f0fdf7fe640 1 --2- 172.21.15.114:0/2362136826 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x55c8d0ef00a0 0x55c8d0f0aff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T10:32:03.830 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:32:03.824+0000 7f0fde7fc640 1 -- 172.21.15.114:0/2362136826 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x55c8d0ef00a0 msgr2=0x55c8d0f0aff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T10:32:03.830 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:32:03.824+0000 7f0fde7fc640 1 --2- 172.21.15.114:0/2362136826 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x55c8d0ef00a0 0x55c8d0f0aff0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:32:03.830 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:32:03.824+0000 7f0fdeffd640 1 --2- 172.21.15.114:0/2362136826 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x55c8d0ee0ba0 0x55c8d0f7cbd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T10:32:03.830 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:32:03.824+0000 7f0fde7fc640 1 -- 172.21.15.114:0/2362136826 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x55c8d0ee0ba0 msgr2=0x55c8d0f7cbd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T10:32:03.831 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:32:03.824+0000 7f0fde7fc640 1 --2- 172.21.15.114:0/2362136826 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x55c8d0ee0ba0 0x55c8d0f7cbd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:32:03.831 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:32:03.824+0000 7f0fde7fc640 1 -- 172.21.15.114:0/2362136826 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55c8d0f09070 con 0x55c8d0ee14c0 2024-04-23T10:32:03.831 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:32:03.824+0000 7f0fdf7fe640 1 --2- 172.21.15.114:0/2362136826 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x55c8d0ef00a0 0x55c8d0f0aff0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-23T10:32:03.831 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:32:03.824+0000 7f0fdeffd640 1 --2- 172.21.15.114:0/2362136826 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x55c8d0ee0ba0 0x55c8d0f7cbd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-23T10:32:03.831 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:32:03.824+0000 7f0fde7fc640 1 --2- 172.21.15.114:0/2362136826 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55c8d0ee14c0 0x55c8d0ee75b0 secure :-1 s=READY pgs=716 cs=0 l=1 rev1=1 crypto rx=0x7f0fd4009880 tx=0x7f0fd4012e90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=839067f8ecf698 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T10:32:03.831 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:32:03.824+0000 7f0fc4ff9640 1 -- 172.21.15.114:0/2362136826 <== mon.0 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0fd4064070 con 0x55c8d0ee14c0 2024-04-23T10:32:03.831 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:32:03.824+0000 7f0fc4ff9640 1 -- 172.21.15.114:0/2362136826 <== mon.0 v2:172.21.15.114:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f0fd4000bc0 con 0x55c8d0ee14c0 2024-04-23T10:32:03.831 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:32:03.824+0000 7f0fc4ff9640 1 -- 172.21.15.114:0/2362136826 <== mon.0 v2:172.21.15.114:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0fd402f040 con 0x55c8d0ee14c0 2024-04-23T10:32:03.831 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:32:03.825+0000 7f0fe81f4740 1 -- 172.21.15.114:0/2362136826 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55c8d0ee14c0 msgr2=0x55c8d0ee75b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T10:32:03.831 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:32:03.825+0000 7f0fe81f4740 1 --2- 172.21.15.114:0/2362136826 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55c8d0ee14c0 0x55c8d0ee75b0 secure :-1 s=READY pgs=716 cs=0 l=1 rev1=1 crypto rx=0x7f0fd4009880 tx=0x7f0fd4012e90 comp rx=0 tx=0).stop 2024-04-23T10:32:03.831 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:32:03.825+0000 7f0fe81f4740 1 -- 172.21.15.114:0/2362136826 shutdown_connections 2024-04-23T10:32:03.831 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:32:03.825+0000 7f0fe81f4740 1 --2- 172.21.15.114:0/2362136826 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x55c8d0ef00a0 0x55c8d0f0aff0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:32:03.831 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:32:03.825+0000 7f0fe81f4740 1 --2- 172.21.15.114:0/2362136826 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55c8d0ee14c0 0x55c8d0ee75b0 unknown :-1 s=CLOSED pgs=716 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:32:03.831 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:32:03.825+0000 7f0fe81f4740 1 --2- 172.21.15.114:0/2362136826 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x55c8d0ee0ba0 0x55c8d0f7cbd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:32:03.831 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:32:03.825+0000 7f0fe81f4740 1 -- 172.21.15.114:0/2362136826 >> 172.21.15.114:0/2362136826 conn(0x55c8d0edcfc0 msgr2=0x55c8d0ee9ba0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T10:32:03.832 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:32:03.825+0000 7f0fe81f4740 1 -- 172.21.15.114:0/2362136826 shutdown_connections 2024-04-23T10:32:03.832 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:32:03.825+0000 7f0fe81f4740 1 -- 172.21.15.114:0/2362136826 wait complete. 2024-04-23T10:32:03.832 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:32:03.825+0000 7f0fe81f4740 1 -- start start 2024-04-23T10:32:03.832 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:32:03.825+0000 7f0fe81f4740 1 Processor -- start 2024-04-23T10:32:03.832 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:32:03.825+0000 7f0fe81f4740 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x55c8d0ee0ba0 0x55c8d0f7cbd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T10:32:03.832 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:32:03.825+0000 7f0fe81f4740 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55c8d0ee14c0 0x55c8d0f0aff0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T10:32:03.832 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:32:03.825+0000 7f0fe81f4740 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x55c8d0ef00a0 0x55c8d0ee75b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T10:32:03.832 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:32:03.825+0000 7f0fe81f4740 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x55c8d0eea600 con 0x55c8d0ee14c0 2024-04-23T10:32:03.832 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:32:03.825+0000 7f0fe81f4740 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 -- 0x55c8d0ee7430 con 0x55c8d0ee0ba0 2024-04-23T10:32:03.832 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:32:03.825+0000 7f0fe81f4740 1 -- --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_getmap magic: 0 -- 0x55c8d0ed83d0 con 0x55c8d0ef00a0 2024-04-23T10:32:03.832 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:32:03.825+0000 7f0fe81f4740 10 client.0 _ll_register_callbacks cb 0x7f0fd4009f40 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-23T10:32:03.832 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:32:03.825+0000 7f0fde7fc640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55c8d0ee14c0 0x55c8d0f0aff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T10:32:03.832 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:32:03.825+0000 7f0fde7fc640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55c8d0ee14c0 0x55c8d0f0aff0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.114:52004/0 (socket says 172.21.15.114:52004) 2024-04-23T10:32:03.832 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:32:03.825+0000 7f0fde7fc640 1 -- 172.21.15.114:0/4176536170 learned_addr learned my addr 172.21.15.114:0/4176536170 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-04-23T10:32:03.833 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:32:03.826+0000 7f0fdf7fe640 1 --2- 172.21.15.114:0/4176536170 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x55c8d0ef00a0 0x55c8d0ee75b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T10:32:03.833 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:32:03.826+0000 7f0fdeffd640 1 --2- 172.21.15.114:0/4176536170 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x55c8d0ee0ba0 0x55c8d0f7cbd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T10:32:03.833 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:32:03.826+0000 7f0fde7fc640 1 -- 172.21.15.114:0/4176536170 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x55c8d0ef00a0 msgr2=0x55c8d0ee75b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T10:32:03.833 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:32:03.826+0000 7f0fde7fc640 1 --2- 172.21.15.114:0/4176536170 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x55c8d0ef00a0 0x55c8d0ee75b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:32:03.833 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:32:03.826+0000 7f0fde7fc640 1 -- 172.21.15.114:0/4176536170 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x55c8d0ee0ba0 msgr2=0x55c8d0f7cbd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T10:32:03.833 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:32:03.826+0000 7f0fde7fc640 1 --2- 172.21.15.114:0/4176536170 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x55c8d0ee0ba0 0x55c8d0f7cbd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:32:03.833 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:32:03.826+0000 7f0fde7fc640 1 -- 172.21.15.114:0/4176536170 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55c8d0f679e0 con 0x55c8d0ee14c0 2024-04-23T10:32:03.833 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:32:03.826+0000 7f0fdeffd640 1 --2- 172.21.15.114:0/4176536170 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x55c8d0ee0ba0 0x55c8d0f7cbd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-23T10:32:03.833 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:32:03.826+0000 7f0fde7fc640 1 --2- 172.21.15.114:0/4176536170 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55c8d0ee14c0 0x55c8d0f0aff0 secure :-1 s=READY pgs=717 cs=0 l=1 rev1=1 crypto rx=0x7f0fd4012850 tx=0x7f0fd4005690 comp rx=0 tx=0).ready entity=mon.0 client_cookie=724e2fdf0d1ab0f4 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T10:32:03.833 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:32:03.826+0000 7f0fc2ff5640 10 client.0 ms_handle_connect on v2:172.21.15.114:3300/0 2024-04-23T10:32:03.833 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:32:03.826+0000 7f0fc2ff5640 1 -- 172.21.15.114:0/4176536170 <== mon.0 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0fd4064070 con 0x55c8d0ee14c0 2024-04-23T10:32:03.833 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:32:03.826+0000 7f0fc2ff5640 1 -- 172.21.15.114:0/4176536170 <== mon.0 v2:172.21.15.114:3300/0 2 ==== osd_map(94..94 src has 1..94) ==== 8585+0+0 (secure 0 0 0) 0x7f0fd405c290 con 0x55c8d0ee14c0 2024-04-23T10:32:03.833 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:32:03.826+0000 7f0fe81f4740 1 -- 172.21.15.114:0/4176536170 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f0fd4000bc0 con 0x55c8d0ee14c0 2024-04-23T10:32:03.833 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:32:03.826+0000 7f0fc2ff5640 1 -- 172.21.15.114:0/4176536170 <== mon.0 v2:172.21.15.114:3300/0 3 ==== mon_get_version_reply(handle=1 version=30) ==== 24+0+0 (secure 0 0 0) 0x7f0fd402f040 con 0x55c8d0ee14c0 2024-04-23T10:32:03.833 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:32:03.826+0000 7f0fe81f4740 10 client.16332 fetch_fsmap learned FSMap version 30 2024-04-23T10:32:03.834 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:32:03.826+0000 7f0fe81f4740 1 -- 172.21.15.114:0/4176536170 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({config=0+,fsmap=30,monmap=4+}) -- 0x55c8d0f679e0 con 0x55c8d0ee14c0 2024-04-23T10:32:03.834 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:32:03.826+0000 7f0fc2ff5640 1 -- 172.21.15.114:0/4176536170 <== mon.0 v2:172.21.15.114:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f0fd4000bc0 con 0x55c8d0ee14c0 2024-04-23T10:32:03.834 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:32:03.826+0000 7f0fc2ff5640 1 -- 172.21.15.114:0/4176536170 <== mon.0 v2:172.21.15.114:3300/0 5 ==== fsmap(e 30) ==== 8338+0+0 (secure 0 0 0) 0x7f0fd4004000 con 0x55c8d0ee14c0 2024-04-23T10:32:03.834 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:32:03.826+0000 7f0fe81f4740 10 client.16332 fetch_fsmap finished waiting for FSMap version 30 2024-04-23T10:32:03.834 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:32:03.826+0000 7f0fe81f4740 10 client.16332 resolve_mds: resolved name 'e' to daemon mds.e 2024-04-23T10:32:03.834 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:32:03.826+0000 7f0fe81f4740 20 client.16332 populate_metadata read hostname 'smithi114' 2024-04-23T10:32:03.834 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:32:03.826+0000 7f0fe81f4740 1 --2- 172.21.15.114:0/4176536170 >> [v2:172.21.15.119:6834/2033916709,v1:172.21.15.119:6835/2033916709] conn(0x55c8d0f13870 0x55c8d0f13c50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T10:32:03.834 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:32:03.826+0000 7f0fe81f4740 4 client.16332 mds_command: new command op to 24479 tid=0 multi_id=0 [{"prefix": "ops", "flags": ["locks"]}] 2024-04-23T10:32:03.834 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:32:03.826+0000 7f0fe81f4740 1 -- 172.21.15.114:0/4176536170 --> [v2:172.21.15.119:6834/2033916709,v1:172.21.15.119:6835/2033916709] -- command(tid 0: {"prefix": "ops", "flags": ["locks"]}) -- 0x55c8d0edfe70 con 0x55c8d0f13870 2024-04-23T10:32:03.834 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:32:03.827+0000 7f0fdeffd640 1 --2- 172.21.15.114:0/4176536170 >> [v2:172.21.15.119:6834/2033916709,v1:172.21.15.119:6835/2033916709] conn(0x55c8d0f13870 0x55c8d0f13c50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T10:32:03.834 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:32:03.827+0000 7f0fdeffd640 1 --2- 172.21.15.114:0/4176536170 >> [v2:172.21.15.119:6834/2033916709,v1:172.21.15.119:6835/2033916709] conn(0x55c8d0f13870 0x55c8d0f13c50 crc :-1 s=READY pgs=36 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.2 client_cookie=41dda7b3f6e49e8b server_cookie=eac32f0dbd834558 in_seq=0 out_seq=0 2024-04-23T10:32:03.834 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:32:03.827+0000 7f0fc2ff5640 10 client.16332 ms_handle_connect on v2:172.21.15.119:6834/2033916709 2024-04-23T10:32:03.836 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:32:03.831+0000 7f0fc2ff5640 1 -- 172.21.15.114:0/4176536170 <== mds.2 v2:172.21.15.119:6834/2033916709 1 ==== command_reply(tid 0: 0 ) ==== 8+0+209639 (crc 0 0 0) 0x55c8d0edfe70 con 0x55c8d0f13870 2024-04-23T10:32:03.836 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:32:03.831+0000 7f0fc2ff5640 10 client.16332 handle_command_reply: tid=0 2024-04-23T10:32:03.836 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:32:03.831+0000 7f0fe81f4740 1 client.16332 shutdown 2024-04-23T10:32:03.836 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:32:03.831+0000 7f0fe81f4740 1 -- 172.21.15.114:0/4176536170 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55c8d0ee14c0 msgr2=0x55c8d0f0aff0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T10:32:03.836 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:32:03.831+0000 7f0fe81f4740 1 --2- 172.21.15.114:0/4176536170 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55c8d0ee14c0 0x55c8d0f0aff0 secure :-1 s=READY pgs=717 cs=0 l=1 rev1=1 crypto rx=0x7f0fd4012850 tx=0x7f0fd4005690 comp rx=0 tx=0).stop 2024-04-23T10:32:03.836 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:32:03.831+0000 7f0fe81f4740 1 -- 172.21.15.114:0/4176536170 shutdown_connections 2024-04-23T10:32:03.836 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:32:03.831+0000 7f0fe81f4740 1 --2- 172.21.15.114:0/4176536170 >> [v2:172.21.15.119:6834/2033916709,v1:172.21.15.119:6835/2033916709] conn(0x55c8d0f13870 0x55c8d0f13c50 crc :-1 s=READY pgs=36 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:32:03.836 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:32:03.831+0000 7f0fe81f4740 1 --2- 172.21.15.114:0/4176536170 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x55c8d0ef00a0 0x55c8d0ee75b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:32:03.836 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:32:03.831+0000 7f0fc2ff5640 0 client.16332 ms_handle_reset on v2:172.21.15.119:6834/2033916709 2024-04-23T10:32:03.836 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:32:03.831+0000 7f0fe81f4740 1 --2- 172.21.15.114:0/4176536170 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55c8d0ee14c0 0x55c8d0f0aff0 unknown :-1 s=CLOSED pgs=717 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:32:03.836 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:32:03.832+0000 7f0fe81f4740 1 --2- 172.21.15.114:0/4176536170 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x55c8d0ee0ba0 0x55c8d0f7cbd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:32:03.837 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:32:03.832+0000 7f0fe81f4740 1 -- 172.21.15.114:0/4176536170 >> 172.21.15.114:0/4176536170 conn(0x55c8d0edcfc0 msgr2=0x55c8d0f68050 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T10:32:03.837 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:32:03.832+0000 7f0fe81f4740 1 -- 172.21.15.114:0/4176536170 shutdown_connections 2024-04-23T10:32:03.837 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:32:03.832+0000 7f0fe81f4740 1 -- 172.21.15.114:0/4176536170 wait complete. 2024-04-23T10:32:03.837 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:32:03.832+0000 7f0fe81f4740 20 client.16332 trim_cache size 0 max 16384 2024-04-23T10:32:03.841 INFO:teuthology.orchestra.run.smithi114.stdout:mds.e: { 2024-04-23T10:32:03.841 INFO:teuthology.orchestra.run.smithi114.stdout: "ops": [ 2024-04-23T10:32:03.841 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-04-23T10:32:03.841 INFO:teuthology.orchestra.run.smithi114.stdout: "description": "peer_request:mds.0:190", 2024-04-23T10:32:03.841 INFO:teuthology.orchestra.run.smithi114.stdout: "initiated_at": "2024-04-23T10:31:21.483611+0000", 2024-04-23T10:32:03.841 INFO:teuthology.orchestra.run.smithi114.stdout: "age": 42.345133371000003, 2024-04-23T10:32:03.841 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 42.345151907999998, 2024-04-23T10:32:03.841 INFO:teuthology.orchestra.run.smithi114.stdout: "continuous": false, 2024-04-23T10:32:03.841 INFO:teuthology.orchestra.run.smithi114.stdout: "type_data": { 2024-04-23T10:32:03.841 INFO:teuthology.orchestra.run.smithi114.stdout: "result": -2147483648, 2024-04-23T10:32:03.841 INFO:teuthology.orchestra.run.smithi114.stdout: "flag_point": "dispatched", 2024-04-23T10:32:03.841 INFO:teuthology.orchestra.run.smithi114.stdout: "reqid": { 2024-04-23T10:32:03.841 INFO:teuthology.orchestra.run.smithi114.stdout: "entity": { 2024-04-23T10:32:03.842 INFO:teuthology.orchestra.run.smithi114.stdout: "type": "mds", 2024-04-23T10:32:03.842 INFO:teuthology.orchestra.run.smithi114.stdout: "num": 0 2024-04-23T10:32:03.842 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-23T10:32:03.842 INFO:teuthology.orchestra.run.smithi114.stdout: "tid": 190 2024-04-23T10:32:03.842 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-23T10:32:03.842 INFO:teuthology.orchestra.run.smithi114.stdout: "op_type": "peer_request", 2024-04-23T10:32:03.842 INFO:teuthology.orchestra.run.smithi114.stdout: "leader_info": { 2024-04-23T10:32:03.842 INFO:teuthology.orchestra.run.smithi114.stdout: "leader": "0" 2024-04-23T10:32:03.842 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-23T10:32:03.842 INFO:teuthology.orchestra.run.smithi114.stdout: "events": [ 2024-04-23T10:32:03.842 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-04-23T10:32:03.842 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-04-23T10:31:21.483611+0000", 2024-04-23T10:32:03.842 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "initiated" 2024-04-23T10:32:03.842 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-23T10:32:03.842 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-04-23T10:32:03.843 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-04-23T10:31:21.483611+0000", 2024-04-23T10:32:03.843 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "header_read" 2024-04-23T10:32:03.843 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-23T10:32:03.843 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-04-23T10:32:03.843 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-04-23T10:31:21.483612+0000", 2024-04-23T10:32:03.843 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "throttled" 2024-04-23T10:32:03.843 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-23T10:32:03.843 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-04-23T10:32:03.843 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-04-23T10:31:21.483618+0000", 2024-04-23T10:32:03.843 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "all_read" 2024-04-23T10:32:03.843 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-23T10:32:03.843 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-04-23T10:32:03.843 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-04-23T10:31:21.484647+0000", 2024-04-23T10:32:03.843 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "dispatched" 2024-04-23T10:32:03.844 INFO:teuthology.orchestra.run.smithi114.stdout: } 2024-04-23T10:32:03.844 INFO:teuthology.orchestra.run.smithi114.stdout: ], 2024-04-23T10:32:03.844 INFO:teuthology.orchestra.run.smithi114.stdout: "locks": [] 2024-04-23T10:32:03.844 INFO:teuthology.orchestra.run.smithi114.stdout: } 2024-04-23T10:32:03.844 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-23T10:32:03.844 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-04-23T10:32:03.844 INFO:teuthology.orchestra.run.smithi114.stdout: "description": "internal op quiesce_path:mds.2:106 fp=#0x1", 2024-04-23T10:32:03.844 INFO:teuthology.orchestra.run.smithi114.stdout: "initiated_at": "2024-04-23T10:31:21.445229+0000", 2024-04-23T10:32:03.844 INFO:teuthology.orchestra.run.smithi114.stdout: "age": 42.383516229999998, 2024-04-23T10:32:03.844 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 42.383564112000002, 2024-04-23T10:32:03.844 INFO:teuthology.orchestra.run.smithi114.stdout: "continuous": true, 2024-04-23T10:32:03.844 INFO:teuthology.orchestra.run.smithi114.stdout: "type_data": { 2024-04-23T10:32:03.844 INFO:teuthology.orchestra.run.smithi114.stdout: "result": 0, 2024-04-23T10:32:03.844 INFO:teuthology.orchestra.run.smithi114.stdout: "flag_point": "quiesce complete", 2024-04-23T10:32:03.844 INFO:teuthology.orchestra.run.smithi114.stdout: "reqid": { 2024-04-23T10:32:03.845 INFO:teuthology.orchestra.run.smithi114.stdout: "entity": { 2024-04-23T10:32:03.845 INFO:teuthology.orchestra.run.smithi114.stdout: "type": "mds", 2024-04-23T10:32:03.845 INFO:teuthology.orchestra.run.smithi114.stdout: "num": 2 2024-04-23T10:32:03.845 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-23T10:32:03.845 INFO:teuthology.orchestra.run.smithi114.stdout: "tid": 106 2024-04-23T10:32:03.845 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-23T10:32:03.845 INFO:teuthology.orchestra.run.smithi114.stdout: "op_type": "internal_op", 2024-04-23T10:32:03.845 INFO:teuthology.orchestra.run.smithi114.stdout: "internal_op": 5384, 2024-04-23T10:32:03.845 INFO:teuthology.orchestra.run.smithi114.stdout: "op_name": "quiesce_path", 2024-04-23T10:32:03.845 INFO:teuthology.orchestra.run.smithi114.stdout: "events": [ 2024-04-23T10:32:03.845 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-04-23T10:32:03.845 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-04-23T10:31:21.445229+0000", 2024-04-23T10:32:03.845 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "initiated" 2024-04-23T10:32:03.845 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-23T10:32:03.845 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-04-23T10:32:03.846 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-04-23T10:31:21.445229+0000", 2024-04-23T10:32:03.846 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "header_read" 2024-04-23T10:32:03.846 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-23T10:32:03.846 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-04-23T10:32:03.846 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-04-23T10:31:21.445229+0000", 2024-04-23T10:32:03.846 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "throttled" 2024-04-23T10:32:03.846 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-23T10:32:03.846 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-04-23T10:32:03.846 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-04-23T10:31:21.445229+0000", 2024-04-23T10:32:03.846 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "all_read" 2024-04-23T10:32:03.846 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-23T10:32:03.846 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-04-23T10:32:03.846 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-04-23T10:31:21.445229+0000", 2024-04-23T10:32:03.846 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "dispatched" 2024-04-23T10:32:03.847 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-23T10:32:03.847 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-04-23T10:32:03.847 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-04-23T10:31:21.509505+0000", 2024-04-23T10:32:03.847 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "quiesce complete" 2024-04-23T10:32:03.847 INFO:teuthology.orchestra.run.smithi114.stdout: } 2024-04-23T10:32:03.847 INFO:teuthology.orchestra.run.smithi114.stdout: ], 2024-04-23T10:32:03.847 INFO:teuthology.orchestra.run.smithi114.stdout: "locks": [] 2024-04-23T10:32:03.847 INFO:teuthology.orchestra.run.smithi114.stdout: } 2024-04-23T10:32:03.847 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-23T10:32:03.847 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-04-23T10:32:03.847 INFO:teuthology.orchestra.run.smithi114.stdout: "description": "peer_request:mds.0:191", 2024-04-23T10:32:03.847 INFO:teuthology.orchestra.run.smithi114.stdout: "initiated_at": "2024-04-23T10:31:21.483666+0000", 2024-04-23T10:32:03.847 INFO:teuthology.orchestra.run.smithi114.stdout: "age": 42.345078950000001, 2024-04-23T10:32:03.847 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 42.345151399000002, 2024-04-23T10:32:03.848 INFO:teuthology.orchestra.run.smithi114.stdout: "continuous": false, 2024-04-23T10:32:03.848 INFO:teuthology.orchestra.run.smithi114.stdout: "type_data": { 2024-04-23T10:32:03.848 INFO:teuthology.orchestra.run.smithi114.stdout: "result": -2147483648, 2024-04-23T10:32:03.848 INFO:teuthology.orchestra.run.smithi114.stdout: "flag_point": "dispatched", 2024-04-23T10:32:03.848 INFO:teuthology.orchestra.run.smithi114.stdout: "reqid": { 2024-04-23T10:32:03.848 INFO:teuthology.orchestra.run.smithi114.stdout: "entity": { 2024-04-23T10:32:03.848 INFO:teuthology.orchestra.run.smithi114.stdout: "type": "mds", 2024-04-23T10:32:03.848 INFO:teuthology.orchestra.run.smithi114.stdout: "num": 0 2024-04-23T10:32:03.848 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-23T10:32:03.848 INFO:teuthology.orchestra.run.smithi114.stdout: "tid": 191 2024-04-23T10:32:03.848 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-23T10:32:03.848 INFO:teuthology.orchestra.run.smithi114.stdout: "op_type": "peer_request", 2024-04-23T10:32:03.848 INFO:teuthology.orchestra.run.smithi114.stdout: "leader_info": { 2024-04-23T10:32:03.849 INFO:teuthology.orchestra.run.smithi114.stdout: "leader": "0" 2024-04-23T10:32:03.849 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-23T10:32:03.849 INFO:teuthology.orchestra.run.smithi114.stdout: "events": [ 2024-04-23T10:32:03.849 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-04-23T10:32:03.849 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-04-23T10:31:21.483666+0000", 2024-04-23T10:32:03.849 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "initiated" 2024-04-23T10:32:03.849 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-23T10:32:03.849 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-04-23T10:32:03.849 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-04-23T10:31:21.483666+0000", 2024-04-23T10:32:03.849 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "header_read" 2024-04-23T10:32:03.849 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-23T10:32:03.849 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-04-23T10:32:03.849 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-04-23T10:31:21.483666+0000", 2024-04-23T10:32:03.849 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "throttled" 2024-04-23T10:32:03.849 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-23T10:32:03.850 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-04-23T10:32:03.850 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-04-23T10:31:21.483671+0000", 2024-04-23T10:32:03.850 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "all_read" 2024-04-23T10:32:03.850 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-23T10:32:03.850 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-04-23T10:32:03.850 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-04-23T10:31:21.485256+0000", 2024-04-23T10:32:03.850 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "dispatched" 2024-04-23T10:32:03.850 INFO:teuthology.orchestra.run.smithi114.stdout: } 2024-04-23T10:32:03.850 INFO:teuthology.orchestra.run.smithi114.stdout: ], 2024-04-23T10:32:03.850 INFO:teuthology.orchestra.run.smithi114.stdout: "locks": [] 2024-04-23T10:32:03.850 INFO:teuthology.orchestra.run.smithi114.stdout: } 2024-04-23T10:32:03.850 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-23T10:32:03.850 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-04-23T10:32:03.850 INFO:teuthology.orchestra.run.smithi114.stdout: "description": "internal op quiesce_inode:mds.2:107 fp=#0x1", 2024-04-23T10:32:03.851 INFO:teuthology.orchestra.run.smithi114.stdout: "initiated_at": "2024-04-23T10:31:21.445284+0000", 2024-04-23T10:32:03.851 INFO:teuthology.orchestra.run.smithi114.stdout: "age": 42.383460773000003, 2024-04-23T10:32:03.851 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 42.383555897999997, 2024-04-23T10:32:03.851 INFO:teuthology.orchestra.run.smithi114.stdout: "continuous": true, 2024-04-23T10:32:03.851 INFO:teuthology.orchestra.run.smithi114.stdout: "type_data": { 2024-04-23T10:32:03.851 INFO:teuthology.orchestra.run.smithi114.stdout: "result": -2147483648, 2024-04-23T10:32:03.851 INFO:teuthology.orchestra.run.smithi114.stdout: "flag_point": "quiesce complete for non-auth inode", 2024-04-23T10:32:03.851 INFO:teuthology.orchestra.run.smithi114.stdout: "reqid": { 2024-04-23T10:32:03.851 INFO:teuthology.orchestra.run.smithi114.stdout: "entity": { 2024-04-23T10:32:03.851 INFO:teuthology.orchestra.run.smithi114.stdout: "type": "mds", 2024-04-23T10:32:03.851 INFO:teuthology.orchestra.run.smithi114.stdout: "num": 2 2024-04-23T10:32:03.851 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-23T10:32:03.851 INFO:teuthology.orchestra.run.smithi114.stdout: "tid": 107 2024-04-23T10:32:03.852 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-23T10:32:03.852 INFO:teuthology.orchestra.run.smithi114.stdout: "op_type": "internal_op", 2024-04-23T10:32:03.852 INFO:teuthology.orchestra.run.smithi114.stdout: "internal_op": 5385, 2024-04-23T10:32:03.852 INFO:teuthology.orchestra.run.smithi114.stdout: "op_name": "quiesce_inode", 2024-04-23T10:32:03.852 INFO:teuthology.orchestra.run.smithi114.stdout: "events": [ 2024-04-23T10:32:03.852 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-04-23T10:32:03.852 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-04-23T10:31:21.445284+0000", 2024-04-23T10:32:03.852 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "initiated" 2024-04-23T10:32:03.852 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-23T10:32:03.852 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-04-23T10:32:03.852 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-04-23T10:31:21.445284+0000", 2024-04-23T10:32:03.852 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "header_read" 2024-04-23T10:32:03.852 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-23T10:32:03.852 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-04-23T10:32:03.852 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-04-23T10:31:21.445284+0000", 2024-04-23T10:32:03.853 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "throttled" 2024-04-23T10:32:03.853 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-23T10:32:03.853 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-04-23T10:32:03.853 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-04-23T10:31:21.445284+0000", 2024-04-23T10:32:03.853 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "all_read" 2024-04-23T10:32:03.853 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-23T10:32:03.853 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-04-23T10:32:03.853 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-04-23T10:31:21.445284+0000", 2024-04-23T10:32:03.853 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "dispatched" 2024-04-23T10:32:03.853 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-23T10:32:03.853 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-04-23T10:32:03.853 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-04-23T10:31:21.445398+0000", 2024-04-23T10:32:03.853 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "requesting remote authpins" 2024-04-23T10:32:03.853 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-23T10:32:03.853 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-04-23T10:32:03.854 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-04-23T10:31:21.446094+0000", 2024-04-23T10:32:03.854 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "acquired locks" 2024-04-23T10:32:03.854 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-23T10:32:03.854 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-04-23T10:32:03.854 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-04-23T10:31:21.509469+0000", 2024-04-23T10:32:03.854 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "quiesce complete for non-auth inode" 2024-04-23T10:32:03.854 INFO:teuthology.orchestra.run.smithi114.stdout: } 2024-04-23T10:32:03.854 INFO:teuthology.orchestra.run.smithi114.stdout: ], 2024-04-23T10:32:03.854 INFO:teuthology.orchestra.run.smithi114.stdout: "locks": [ 2024-04-23T10:32:03.854 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-04-23T10:32:03.854 INFO:teuthology.orchestra.run.smithi114.stdout: "object": { 2024-04-23T10:32:03.854 INFO:teuthology.orchestra.run.smithi114.stdout: "is_auth": false, 2024-04-23T10:32:03.854 INFO:teuthology.orchestra.run.smithi114.stdout: "auth_state": { 2024-04-23T10:32:03.854 INFO:teuthology.orchestra.run.smithi114.stdout: "replicas": {} 2024-04-23T10:32:03.855 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-23T10:32:03.855 INFO:teuthology.orchestra.run.smithi114.stdout: "replica_state": { 2024-04-23T10:32:03.855 INFO:teuthology.orchestra.run.smithi114.stdout: "authority": [ 2024-04-23T10:32:03.855 INFO:teuthology.orchestra.run.smithi114.stdout: 0, 2024-04-23T10:32:03.855 INFO:teuthology.orchestra.run.smithi114.stdout: -2 2024-04-23T10:32:03.855 INFO:teuthology.orchestra.run.smithi114.stdout: ], 2024-04-23T10:32:03.855 INFO:teuthology.orchestra.run.smithi114.stdout: "replica_nonce": 1 2024-04-23T10:32:03.855 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-23T10:32:03.855 INFO:teuthology.orchestra.run.smithi114.stdout: "auth_pins": 0, 2024-04-23T10:32:03.855 INFO:teuthology.orchestra.run.smithi114.stdout: "is_frozen": false, 2024-04-23T10:32:03.855 INFO:teuthology.orchestra.run.smithi114.stdout: "is_freezing": false, 2024-04-23T10:32:03.855 INFO:teuthology.orchestra.run.smithi114.stdout: "pins": { 2024-04-23T10:32:03.855 INFO:teuthology.orchestra.run.smithi114.stdout: "request": 1, 2024-04-23T10:32:03.855 INFO:teuthology.orchestra.run.smithi114.stdout: "lock": 1, 2024-04-23T10:32:03.855 INFO:teuthology.orchestra.run.smithi114.stdout: "dirfrag": 1, 2024-04-23T10:32:03.856 INFO:teuthology.orchestra.run.smithi114.stdout: "caps": 0, 2024-04-23T10:32:03.856 INFO:teuthology.orchestra.run.smithi114.stdout: "dirwaiter": 0, 2024-04-23T10:32:03.856 INFO:teuthology.orchestra.run.smithi114.stdout: "waiter": 0, 2024-04-23T10:32:03.856 INFO:teuthology.orchestra.run.smithi114.stdout: "discoverbase": 0 2024-04-23T10:32:03.856 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-23T10:32:03.856 INFO:teuthology.orchestra.run.smithi114.stdout: "nref": 3 2024-04-23T10:32:03.856 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-23T10:32:03.856 INFO:teuthology.orchestra.run.smithi114.stdout: "object_string": "[inode 0x1 [...7,head] / rep@0.1 v187 snaprealm=0x55c68b7b0480 f(v0 m2024-04-23T10:24:48.676117+0000 1=0+1) n(v27 rc2024-04-23T10:31:17.764716+0000 b21075308 rs5 88=67+21)/n(v0 rc2024-04-23T10:24:44.319412+0000 1=0+1) (inest mix) (iquiesce lock x=1 by request(mds.2:107 nref=3)) | request=1 lock=1 dirfrag=1 caps=0 dirwaiter=0 waiter=0 discoverbase=0 0x55c68ba05080]", 2024-04-23T10:32:03.856 INFO:teuthology.orchestra.run.smithi114.stdout: "lock": { 2024-04-23T10:32:03.856 INFO:teuthology.orchestra.run.smithi114.stdout: "gather_set": [], 2024-04-23T10:32:03.856 INFO:teuthology.orchestra.run.smithi114.stdout: "state": "lock", 2024-04-23T10:32:03.856 INFO:teuthology.orchestra.run.smithi114.stdout: "type": "iquiesce", 2024-04-23T10:32:03.856 INFO:teuthology.orchestra.run.smithi114.stdout: "is_leased": false, 2024-04-23T10:32:03.856 INFO:teuthology.orchestra.run.smithi114.stdout: "num_rdlocks": 0, 2024-04-23T10:32:03.856 INFO:teuthology.orchestra.run.smithi114.stdout: "num_wrlocks": 0, 2024-04-23T10:32:03.856 INFO:teuthology.orchestra.run.smithi114.stdout: "num_xlocks": 1, 2024-04-23T10:32:03.857 INFO:teuthology.orchestra.run.smithi114.stdout: "xlock_by": { 2024-04-23T10:32:03.857 INFO:teuthology.orchestra.run.smithi114.stdout: "reqid": { 2024-04-23T10:32:03.857 INFO:teuthology.orchestra.run.smithi114.stdout: "entity": { 2024-04-23T10:32:03.857 INFO:teuthology.orchestra.run.smithi114.stdout: "type": "mds", 2024-04-23T10:32:03.857 INFO:teuthology.orchestra.run.smithi114.stdout: "num": 2 2024-04-23T10:32:03.857 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-23T10:32:03.857 INFO:teuthology.orchestra.run.smithi114.stdout: "tid": 107 2024-04-23T10:32:03.857 INFO:teuthology.orchestra.run.smithi114.stdout: } 2024-04-23T10:32:03.857 INFO:teuthology.orchestra.run.smithi114.stdout: } 2024-04-23T10:32:03.857 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-23T10:32:03.857 INFO:teuthology.orchestra.run.smithi114.stdout: "flags": 4, 2024-04-23T10:32:03.857 INFO:teuthology.orchestra.run.smithi114.stdout: "wrlock_target": -1 2024-04-23T10:32:03.857 INFO:teuthology.orchestra.run.smithi114.stdout: } 2024-04-23T10:32:03.857 INFO:teuthology.orchestra.run.smithi114.stdout: ] 2024-04-23T10:32:03.858 INFO:teuthology.orchestra.run.smithi114.stdout: } 2024-04-23T10:32:03.858 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-23T10:32:03.858 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-04-23T10:32:03.858 INFO:teuthology.orchestra.run.smithi114.stdout: "description": "peer_request:mds.0:192", 2024-04-23T10:32:03.858 INFO:teuthology.orchestra.run.smithi114.stdout: "initiated_at": "2024-04-23T10:31:21.483707+0000", 2024-04-23T10:32:03.858 INFO:teuthology.orchestra.run.smithi114.stdout: "age": 42.345037998999999, 2024-04-23T10:32:03.858 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 42.345206638000001, 2024-04-23T10:32:03.858 INFO:teuthology.orchestra.run.smithi114.stdout: "continuous": false, 2024-04-23T10:32:03.858 INFO:teuthology.orchestra.run.smithi114.stdout: "type_data": { 2024-04-23T10:32:03.858 INFO:teuthology.orchestra.run.smithi114.stdout: "result": -2147483648, 2024-04-23T10:32:03.858 INFO:teuthology.orchestra.run.smithi114.stdout: "flag_point": "dispatched", 2024-04-23T10:32:03.858 INFO:teuthology.orchestra.run.smithi114.stdout: "reqid": { 2024-04-23T10:32:03.858 INFO:teuthology.orchestra.run.smithi114.stdout: "entity": { 2024-04-23T10:32:03.858 INFO:teuthology.orchestra.run.smithi114.stdout: "type": "mds", 2024-04-23T10:32:03.858 INFO:teuthology.orchestra.run.smithi114.stdout: "num": 0 2024-04-23T10:32:03.859 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-23T10:32:03.859 INFO:teuthology.orchestra.run.smithi114.stdout: "tid": 192 2024-04-23T10:32:03.859 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-23T10:32:03.859 INFO:teuthology.orchestra.run.smithi114.stdout: "op_type": "peer_request", 2024-04-23T10:32:03.859 INFO:teuthology.orchestra.run.smithi114.stdout: "leader_info": { 2024-04-23T10:32:03.859 INFO:teuthology.orchestra.run.smithi114.stdout: "leader": "0" 2024-04-23T10:32:03.859 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-23T10:32:03.859 INFO:teuthology.orchestra.run.smithi114.stdout: "events": [ 2024-04-23T10:32:03.859 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-04-23T10:32:03.859 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-04-23T10:31:21.483707+0000", 2024-04-23T10:32:03.859 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "initiated" 2024-04-23T10:32:03.859 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-23T10:32:03.859 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-04-23T10:32:03.859 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-04-23T10:31:21.483707+0000", 2024-04-23T10:32:03.859 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "header_read" 2024-04-23T10:32:03.859 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-23T10:32:03.860 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-04-23T10:32:03.860 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-04-23T10:31:21.483707+0000", 2024-04-23T10:32:03.860 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "throttled" 2024-04-23T10:32:03.860 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-23T10:32:03.860 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-04-23T10:32:03.860 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-04-23T10:31:21.483712+0000", 2024-04-23T10:32:03.860 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "all_read" 2024-04-23T10:32:03.860 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-23T10:32:03.860 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-04-23T10:32:03.860 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-04-23T10:31:21.485846+0000", 2024-04-23T10:32:03.860 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "dispatched" 2024-04-23T10:32:03.860 INFO:teuthology.orchestra.run.smithi114.stdout: } 2024-04-23T10:32:03.860 INFO:teuthology.orchestra.run.smithi114.stdout: ], 2024-04-23T10:32:03.860 INFO:teuthology.orchestra.run.smithi114.stdout: "locks": [] 2024-04-23T10:32:03.860 INFO:teuthology.orchestra.run.smithi114.stdout: } 2024-04-23T10:32:03.861 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-23T10:32:03.861 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-04-23T10:32:03.861 INFO:teuthology.orchestra.run.smithi114.stdout: "description": "internal op quiesce_inode:mds.2:108 fp=#0x10000000000", 2024-04-23T10:32:03.861 INFO:teuthology.orchestra.run.smithi114.stdout: "initiated_at": "2024-04-23T10:31:21.446108+0000", 2024-04-23T10:32:03.861 INFO:teuthology.orchestra.run.smithi114.stdout: "age": 42.382636726000001, 2024-04-23T10:32:03.861 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 42.382827669000001, 2024-04-23T10:32:03.861 INFO:teuthology.orchestra.run.smithi114.stdout: "continuous": true, 2024-04-23T10:32:03.861 INFO:teuthology.orchestra.run.smithi114.stdout: "type_data": { 2024-04-23T10:32:03.861 INFO:teuthology.orchestra.run.smithi114.stdout: "result": -2147483648, 2024-04-23T10:32:03.861 INFO:teuthology.orchestra.run.smithi114.stdout: "flag_point": "quiesce complete for non-auth inode", 2024-04-23T10:32:03.861 INFO:teuthology.orchestra.run.smithi114.stdout: "reqid": { 2024-04-23T10:32:03.861 INFO:teuthology.orchestra.run.smithi114.stdout: "entity": { 2024-04-23T10:32:03.861 INFO:teuthology.orchestra.run.smithi114.stdout: "type": "mds", 2024-04-23T10:32:03.861 INFO:teuthology.orchestra.run.smithi114.stdout: "num": 2 2024-04-23T10:32:03.861 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-23T10:32:03.861 INFO:teuthology.orchestra.run.smithi114.stdout: "tid": 108 2024-04-23T10:32:03.862 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-23T10:32:03.862 INFO:teuthology.orchestra.run.smithi114.stdout: "op_type": "internal_op", 2024-04-23T10:32:03.862 INFO:teuthology.orchestra.run.smithi114.stdout: "internal_op": 5385, 2024-04-23T10:32:03.862 INFO:teuthology.orchestra.run.smithi114.stdout: "op_name": "quiesce_inode", 2024-04-23T10:32:03.862 INFO:teuthology.orchestra.run.smithi114.stdout: "events": [ 2024-04-23T10:32:03.862 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-04-23T10:32:03.862 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-04-23T10:31:21.446108+0000", 2024-04-23T10:32:03.862 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "initiated" 2024-04-23T10:32:03.862 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-23T10:32:03.862 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-04-23T10:32:03.862 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-04-23T10:31:21.446108+0000", 2024-04-23T10:32:03.862 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "header_read" 2024-04-23T10:32:03.862 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-23T10:32:03.862 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-04-23T10:32:03.863 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-04-23T10:31:21.446108+0000", 2024-04-23T10:32:03.863 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "throttled" 2024-04-23T10:32:03.863 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-23T10:32:03.863 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-04-23T10:32:03.863 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-04-23T10:31:21.446108+0000", 2024-04-23T10:32:03.863 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "all_read" 2024-04-23T10:32:03.863 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-23T10:32:03.863 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-04-23T10:32:03.863 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-04-23T10:31:21.446108+0000", 2024-04-23T10:32:03.863 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "dispatched" 2024-04-23T10:32:03.863 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-23T10:32:03.863 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-04-23T10:32:03.863 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-04-23T10:31:21.446194+0000", 2024-04-23T10:32:03.863 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "requesting remote authpins" 2024-04-23T10:32:03.863 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-23T10:32:03.864 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-04-23T10:32:03.864 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-04-23T10:31:21.447335+0000", 2024-04-23T10:32:03.864 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "acquired locks" 2024-04-23T10:32:03.864 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-23T10:32:03.864 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-04-23T10:32:03.864 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-04-23T10:31:21.509440+0000", 2024-04-23T10:32:03.864 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "quiesce complete for non-auth inode" 2024-04-23T10:32:03.864 INFO:teuthology.orchestra.run.smithi114.stdout: } 2024-04-23T10:32:03.864 INFO:teuthology.orchestra.run.smithi114.stdout: ], 2024-04-23T10:32:03.864 INFO:teuthology.orchestra.run.smithi114.stdout: "locks": [ 2024-04-23T10:32:03.864 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-04-23T10:32:03.864 INFO:teuthology.orchestra.run.smithi114.stdout: "object": { 2024-04-23T10:32:03.864 INFO:teuthology.orchestra.run.smithi114.stdout: "is_auth": false, 2024-04-23T10:32:03.864 INFO:teuthology.orchestra.run.smithi114.stdout: "auth_state": { 2024-04-23T10:32:03.864 INFO:teuthology.orchestra.run.smithi114.stdout: "replicas": {} 2024-04-23T10:32:03.864 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-23T10:32:03.865 INFO:teuthology.orchestra.run.smithi114.stdout: "replica_state": { 2024-04-23T10:32:03.865 INFO:teuthology.orchestra.run.smithi114.stdout: "authority": [ 2024-04-23T10:32:03.865 INFO:teuthology.orchestra.run.smithi114.stdout: 0, 2024-04-23T10:32:03.865 INFO:teuthology.orchestra.run.smithi114.stdout: -2 2024-04-23T10:32:03.865 INFO:teuthology.orchestra.run.smithi114.stdout: ], 2024-04-23T10:32:03.865 INFO:teuthology.orchestra.run.smithi114.stdout: "replica_nonce": 1 2024-04-23T10:32:03.865 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-23T10:32:03.865 INFO:teuthology.orchestra.run.smithi114.stdout: "auth_pins": 0, 2024-04-23T10:32:03.865 INFO:teuthology.orchestra.run.smithi114.stdout: "is_frozen": false, 2024-04-23T10:32:03.865 INFO:teuthology.orchestra.run.smithi114.stdout: "is_freezing": false, 2024-04-23T10:32:03.865 INFO:teuthology.orchestra.run.smithi114.stdout: "pins": { 2024-04-23T10:32:03.865 INFO:teuthology.orchestra.run.smithi114.stdout: "request": 1, 2024-04-23T10:32:03.865 INFO:teuthology.orchestra.run.smithi114.stdout: "lock": 1, 2024-04-23T10:32:03.865 INFO:teuthology.orchestra.run.smithi114.stdout: "dirfrag": 1, 2024-04-23T10:32:03.865 INFO:teuthology.orchestra.run.smithi114.stdout: "dirwaiter": 0, 2024-04-23T10:32:03.865 INFO:teuthology.orchestra.run.smithi114.stdout: "discoverbase": 0 2024-04-23T10:32:03.866 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-23T10:32:03.866 INFO:teuthology.orchestra.run.smithi114.stdout: "nref": 3 2024-04-23T10:32:03.866 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-23T10:32:03.866 INFO:teuthology.orchestra.run.smithi114.stdout: "object_string": "[inode 0x10000000000 [...7,head] /volumes/ rep@0.1 v404 f(v0 m2024-04-23T10:24:49.736808+0000 3=2+1) n(v29 rc2024-04-23T10:31:17.764716+0000 b21075308 87=67+20)/n(v0 1=0+1) (inest mix) (iquiesce lock x=1 by request(mds.2:108 nref=3)) | request=1 lock=1 dirfrag=1 dirwaiter=0 discoverbase=0 0x55c68ba05600]", 2024-04-23T10:32:03.866 INFO:teuthology.orchestra.run.smithi114.stdout: "lock": { 2024-04-23T10:32:03.866 INFO:teuthology.orchestra.run.smithi114.stdout: "gather_set": [], 2024-04-23T10:32:03.866 INFO:teuthology.orchestra.run.smithi114.stdout: "state": "lock", 2024-04-23T10:32:03.866 INFO:teuthology.orchestra.run.smithi114.stdout: "type": "iquiesce", 2024-04-23T10:32:03.866 INFO:teuthology.orchestra.run.smithi114.stdout: "is_leased": false, 2024-04-23T10:32:03.866 INFO:teuthology.orchestra.run.smithi114.stdout: "num_rdlocks": 0, 2024-04-23T10:32:03.866 INFO:teuthology.orchestra.run.smithi114.stdout: "num_wrlocks": 0, 2024-04-23T10:32:03.866 INFO:teuthology.orchestra.run.smithi114.stdout: "num_xlocks": 1, 2024-04-23T10:32:03.866 INFO:teuthology.orchestra.run.smithi114.stdout: "xlock_by": { 2024-04-23T10:32:03.867 INFO:teuthology.orchestra.run.smithi114.stdout: "reqid": { 2024-04-23T10:32:03.867 INFO:teuthology.orchestra.run.smithi114.stdout: "entity": { 2024-04-23T10:32:03.867 INFO:teuthology.orchestra.run.smithi114.stdout: "type": "mds", 2024-04-23T10:32:03.867 INFO:teuthology.orchestra.run.smithi114.stdout: "num": 2 2024-04-23T10:32:03.867 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-23T10:32:03.867 INFO:teuthology.orchestra.run.smithi114.stdout: "tid": 108 2024-04-23T10:32:03.867 INFO:teuthology.orchestra.run.smithi114.stdout: } 2024-04-23T10:32:03.867 INFO:teuthology.orchestra.run.smithi114.stdout: } 2024-04-23T10:32:03.867 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-23T10:32:03.867 INFO:teuthology.orchestra.run.smithi114.stdout: "flags": 4, 2024-04-23T10:32:03.867 INFO:teuthology.orchestra.run.smithi114.stdout: "wrlock_target": -1 2024-04-23T10:32:03.867 INFO:teuthology.orchestra.run.smithi114.stdout: } 2024-04-23T10:32:03.867 INFO:teuthology.orchestra.run.smithi114.stdout: ] 2024-04-23T10:32:03.867 INFO:teuthology.orchestra.run.smithi114.stdout: } 2024-04-23T10:32:03.867 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-23T10:32:03.867 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-04-23T10:32:03.868 INFO:teuthology.orchestra.run.smithi114.stdout: "description": "peer_request:mds.0:193", 2024-04-23T10:32:03.868 INFO:teuthology.orchestra.run.smithi114.stdout: "initiated_at": "2024-04-23T10:31:21.483754+0000", 2024-04-23T10:32:03.868 INFO:teuthology.orchestra.run.smithi114.stdout: "age": 42.344990783, 2024-04-23T10:32:03.868 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 42.345240236999999, 2024-04-23T10:32:03.868 INFO:teuthology.orchestra.run.smithi114.stdout: "continuous": false, 2024-04-23T10:32:03.868 INFO:teuthology.orchestra.run.smithi114.stdout: "type_data": { 2024-04-23T10:32:03.868 INFO:teuthology.orchestra.run.smithi114.stdout: "result": -2147483648, 2024-04-23T10:32:03.868 INFO:teuthology.orchestra.run.smithi114.stdout: "flag_point": "dispatched", 2024-04-23T10:32:03.868 INFO:teuthology.orchestra.run.smithi114.stdout: "reqid": { 2024-04-23T10:32:03.868 INFO:teuthology.orchestra.run.smithi114.stdout: "entity": { 2024-04-23T10:32:03.868 INFO:teuthology.orchestra.run.smithi114.stdout: "type": "mds", 2024-04-23T10:32:03.868 INFO:teuthology.orchestra.run.smithi114.stdout: "num": 0 2024-04-23T10:32:03.868 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-23T10:32:03.868 INFO:teuthology.orchestra.run.smithi114.stdout: "tid": 193 2024-04-23T10:32:03.868 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-23T10:32:03.868 INFO:teuthology.orchestra.run.smithi114.stdout: "op_type": "peer_request", 2024-04-23T10:32:03.868 INFO:teuthology.orchestra.run.smithi114.stdout: "leader_info": { 2024-04-23T10:32:03.869 INFO:teuthology.orchestra.run.smithi114.stdout: "leader": "0" 2024-04-23T10:32:03.869 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-23T10:32:03.869 INFO:teuthology.orchestra.run.smithi114.stdout: "events": [ 2024-04-23T10:32:03.869 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-04-23T10:32:03.869 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-04-23T10:31:21.483754+0000", 2024-04-23T10:32:03.869 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "initiated" 2024-04-23T10:32:03.869 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-23T10:32:03.869 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-04-23T10:32:03.869 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-04-23T10:31:21.483754+0000", 2024-04-23T10:32:03.869 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "header_read" 2024-04-23T10:32:03.869 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-23T10:32:03.869 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-04-23T10:32:03.869 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-04-23T10:31:21.483754+0000", 2024-04-23T10:32:03.869 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "throttled" 2024-04-23T10:32:03.869 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-23T10:32:03.869 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-04-23T10:32:03.870 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-04-23T10:31:21.483759+0000", 2024-04-23T10:32:03.870 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "all_read" 2024-04-23T10:32:03.870 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-23T10:32:03.870 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-04-23T10:32:03.870 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-04-23T10:31:21.486610+0000", 2024-04-23T10:32:03.870 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "dispatched" 2024-04-23T10:32:03.870 INFO:teuthology.orchestra.run.smithi114.stdout: } 2024-04-23T10:32:03.870 INFO:teuthology.orchestra.run.smithi114.stdout: ], 2024-04-23T10:32:03.870 INFO:teuthology.orchestra.run.smithi114.stdout: "locks": [] 2024-04-23T10:32:03.870 INFO:teuthology.orchestra.run.smithi114.stdout: } 2024-04-23T10:32:03.870 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-23T10:32:03.870 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-04-23T10:32:03.870 INFO:teuthology.orchestra.run.smithi114.stdout: "description": "internal op quiesce_inode:mds.2:109 fp=#0x10000000001", 2024-04-23T10:32:03.870 INFO:teuthology.orchestra.run.smithi114.stdout: "initiated_at": "2024-04-23T10:31:21.447349+0000", 2024-04-23T10:32:03.870 INFO:teuthology.orchestra.run.smithi114.stdout: "age": 42.381396215999999, 2024-04-23T10:32:03.871 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 42.381670167999999, 2024-04-23T10:32:03.871 INFO:teuthology.orchestra.run.smithi114.stdout: "continuous": true, 2024-04-23T10:32:03.871 INFO:teuthology.orchestra.run.smithi114.stdout: "type_data": { 2024-04-23T10:32:03.871 INFO:teuthology.orchestra.run.smithi114.stdout: "result": -2147483648, 2024-04-23T10:32:03.871 INFO:teuthology.orchestra.run.smithi114.stdout: "flag_point": "quiesce complete for non-auth inode", 2024-04-23T10:32:03.871 INFO:teuthology.orchestra.run.smithi114.stdout: "reqid": { 2024-04-23T10:32:03.871 INFO:teuthology.orchestra.run.smithi114.stdout: "entity": { 2024-04-23T10:32:03.871 INFO:teuthology.orchestra.run.smithi114.stdout: "type": "mds", 2024-04-23T10:32:03.871 INFO:teuthology.orchestra.run.smithi114.stdout: "num": 2 2024-04-23T10:32:03.871 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-23T10:32:03.871 INFO:teuthology.orchestra.run.smithi114.stdout: "tid": 109 2024-04-23T10:32:03.871 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-23T10:32:03.871 INFO:teuthology.orchestra.run.smithi114.stdout: "op_type": "internal_op", 2024-04-23T10:32:03.871 INFO:teuthology.orchestra.run.smithi114.stdout: "internal_op": 5385, 2024-04-23T10:32:03.871 INFO:teuthology.orchestra.run.smithi114.stdout: "op_name": "quiesce_inode", 2024-04-23T10:32:03.872 INFO:teuthology.orchestra.run.smithi114.stdout: "events": [ 2024-04-23T10:32:03.872 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-04-23T10:32:03.872 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-04-23T10:31:21.447349+0000", 2024-04-23T10:32:03.872 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "initiated" 2024-04-23T10:32:03.872 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-23T10:32:03.872 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-04-23T10:32:03.872 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-04-23T10:31:21.447349+0000", 2024-04-23T10:32:03.872 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "header_read" 2024-04-23T10:32:03.872 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-23T10:32:03.872 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-04-23T10:32:03.872 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-04-23T10:31:21.447349+0000", 2024-04-23T10:32:03.872 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "throttled" 2024-04-23T10:32:03.872 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-23T10:32:03.873 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-04-23T10:32:03.873 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-04-23T10:31:21.447349+0000", 2024-04-23T10:32:03.873 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "all_read" 2024-04-23T10:32:03.873 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-23T10:32:03.873 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-04-23T10:32:03.873 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-04-23T10:31:21.447349+0000", 2024-04-23T10:32:03.873 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "dispatched" 2024-04-23T10:32:03.873 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-23T10:32:03.873 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-04-23T10:32:03.873 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-04-23T10:31:21.447418+0000", 2024-04-23T10:32:03.873 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "requesting remote authpins" 2024-04-23T10:32:03.873 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-23T10:32:03.873 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-04-23T10:32:03.874 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-04-23T10:31:21.448374+0000", 2024-04-23T10:32:03.874 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "acquired locks" 2024-04-23T10:32:03.874 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-23T10:32:03.874 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-04-23T10:32:03.874 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-04-23T10:31:21.509412+0000", 2024-04-23T10:32:03.874 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "quiesce complete for non-auth inode" 2024-04-23T10:32:03.874 INFO:teuthology.orchestra.run.smithi114.stdout: } 2024-04-23T10:32:03.874 INFO:teuthology.orchestra.run.smithi114.stdout: ], 2024-04-23T10:32:03.874 INFO:teuthology.orchestra.run.smithi114.stdout: "locks": [ 2024-04-23T10:32:03.874 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-04-23T10:32:03.874 INFO:teuthology.orchestra.run.smithi114.stdout: "object": { 2024-04-23T10:32:03.874 INFO:teuthology.orchestra.run.smithi114.stdout: "is_auth": false, 2024-04-23T10:32:03.874 INFO:teuthology.orchestra.run.smithi114.stdout: "auth_state": { 2024-04-23T10:32:03.874 INFO:teuthology.orchestra.run.smithi114.stdout: "replicas": {} 2024-04-23T10:32:03.875 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-23T10:32:03.875 INFO:teuthology.orchestra.run.smithi114.stdout: "replica_state": { 2024-04-23T10:32:03.875 INFO:teuthology.orchestra.run.smithi114.stdout: "authority": [ 2024-04-23T10:32:03.875 INFO:teuthology.orchestra.run.smithi114.stdout: 0, 2024-04-23T10:32:03.875 INFO:teuthology.orchestra.run.smithi114.stdout: -2 2024-04-23T10:32:03.875 INFO:teuthology.orchestra.run.smithi114.stdout: ], 2024-04-23T10:32:03.875 INFO:teuthology.orchestra.run.smithi114.stdout: "replica_nonce": 1 2024-04-23T10:32:03.875 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-23T10:32:03.875 INFO:teuthology.orchestra.run.smithi114.stdout: "auth_pins": 0, 2024-04-23T10:32:03.875 INFO:teuthology.orchestra.run.smithi114.stdout: "is_frozen": false, 2024-04-23T10:32:03.875 INFO:teuthology.orchestra.run.smithi114.stdout: "is_freezing": false, 2024-04-23T10:32:03.875 INFO:teuthology.orchestra.run.smithi114.stdout: "pins": { 2024-04-23T10:32:03.875 INFO:teuthology.orchestra.run.smithi114.stdout: "request": 1, 2024-04-23T10:32:03.875 INFO:teuthology.orchestra.run.smithi114.stdout: "lock": 1, 2024-04-23T10:32:03.875 INFO:teuthology.orchestra.run.smithi114.stdout: "dirfrag": 1, 2024-04-23T10:32:03.876 INFO:teuthology.orchestra.run.smithi114.stdout: "discoverbase": 0 2024-04-23T10:32:03.876 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-23T10:32:03.876 INFO:teuthology.orchestra.run.smithi114.stdout: "nref": 3 2024-04-23T10:32:03.876 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-23T10:32:03.876 INFO:teuthology.orchestra.run.smithi114.stdout: "object_string": "[inode 0x10000000001 [...7,head] /volumes/qa/ rep@0.1 v416 f(v0 m2024-04-23T10:24:49.714717+0000 2=0+2) n(v30 rc2024-04-23T10:31:17.764716+0000 b21075308 84=65+19)/n(v0 rc2024-04-23T10:24:49.023727+0000 3=0+3) (inest mix) (iquiesce lock x=1 by request(mds.2:109 nref=3)) | request=1 lock=1 dirfrag=1 discoverbase=0 0x55c68ba05b80]", 2024-04-23T10:32:03.876 INFO:teuthology.orchestra.run.smithi114.stdout: "lock": { 2024-04-23T10:32:03.876 INFO:teuthology.orchestra.run.smithi114.stdout: "gather_set": [], 2024-04-23T10:32:03.876 INFO:teuthology.orchestra.run.smithi114.stdout: "state": "lock", 2024-04-23T10:32:03.876 INFO:teuthology.orchestra.run.smithi114.stdout: "type": "iquiesce", 2024-04-23T10:32:03.876 INFO:teuthology.orchestra.run.smithi114.stdout: "is_leased": false, 2024-04-23T10:32:03.876 INFO:teuthology.orchestra.run.smithi114.stdout: "num_rdlocks": 0, 2024-04-23T10:32:03.876 INFO:teuthology.orchestra.run.smithi114.stdout: "num_wrlocks": 0, 2024-04-23T10:32:03.876 INFO:teuthology.orchestra.run.smithi114.stdout: "num_xlocks": 1, 2024-04-23T10:32:03.876 INFO:teuthology.orchestra.run.smithi114.stdout: "xlock_by": { 2024-04-23T10:32:03.876 INFO:teuthology.orchestra.run.smithi114.stdout: "reqid": { 2024-04-23T10:32:03.877 INFO:teuthology.orchestra.run.smithi114.stdout: "entity": { 2024-04-23T10:32:03.877 INFO:teuthology.orchestra.run.smithi114.stdout: "type": "mds", 2024-04-23T10:32:03.877 INFO:teuthology.orchestra.run.smithi114.stdout: "num": 2 2024-04-23T10:32:03.877 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-23T10:32:03.877 INFO:teuthology.orchestra.run.smithi114.stdout: "tid": 109 2024-04-23T10:32:03.877 INFO:teuthology.orchestra.run.smithi114.stdout: } 2024-04-23T10:32:03.877 INFO:teuthology.orchestra.run.smithi114.stdout: } 2024-04-23T10:32:03.877 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-23T10:32:03.877 INFO:teuthology.orchestra.run.smithi114.stdout: "flags": 4, 2024-04-23T10:32:03.877 INFO:teuthology.orchestra.run.smithi114.stdout: "wrlock_target": -1 2024-04-23T10:32:03.877 INFO:teuthology.orchestra.run.smithi114.stdout: } 2024-04-23T10:32:03.877 INFO:teuthology.orchestra.run.smithi114.stdout: ] 2024-04-23T10:32:03.877 INFO:teuthology.orchestra.run.smithi114.stdout: } 2024-04-23T10:32:03.877 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-23T10:32:03.878 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-04-23T10:32:03.878 INFO:teuthology.orchestra.run.smithi114.stdout: "description": "peer_request:mds.0:194", 2024-04-23T10:32:03.878 INFO:teuthology.orchestra.run.smithi114.stdout: "initiated_at": "2024-04-23T10:31:21.483806+0000", 2024-04-23T10:32:03.878 INFO:teuthology.orchestra.run.smithi114.stdout: "age": 42.344938784, 2024-04-23T10:32:03.878 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 42.345270112999998, 2024-04-23T10:32:03.878 INFO:teuthology.orchestra.run.smithi114.stdout: "continuous": false, 2024-04-23T10:32:03.878 INFO:teuthology.orchestra.run.smithi114.stdout: "type_data": { 2024-04-23T10:32:03.878 INFO:teuthology.orchestra.run.smithi114.stdout: "result": -2147483648, 2024-04-23T10:32:03.878 INFO:teuthology.orchestra.run.smithi114.stdout: "flag_point": "dispatched", 2024-04-23T10:32:03.878 INFO:teuthology.orchestra.run.smithi114.stdout: "reqid": { 2024-04-23T10:32:03.878 INFO:teuthology.orchestra.run.smithi114.stdout: "entity": { 2024-04-23T10:32:03.878 INFO:teuthology.orchestra.run.smithi114.stdout: "type": "mds", 2024-04-23T10:32:03.878 INFO:teuthology.orchestra.run.smithi114.stdout: "num": 0 2024-04-23T10:32:03.879 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-23T10:32:03.879 INFO:teuthology.orchestra.run.smithi114.stdout: "tid": 194 2024-04-23T10:32:03.879 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-23T10:32:03.879 INFO:teuthology.orchestra.run.smithi114.stdout: "op_type": "peer_request", 2024-04-23T10:32:03.879 INFO:teuthology.orchestra.run.smithi114.stdout: "leader_info": { 2024-04-23T10:32:03.879 INFO:teuthology.orchestra.run.smithi114.stdout: "leader": "0" 2024-04-23T10:32:03.879 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-23T10:32:03.879 INFO:teuthology.orchestra.run.smithi114.stdout: "events": [ 2024-04-23T10:32:03.879 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-04-23T10:32:03.879 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-04-23T10:31:21.483806+0000", 2024-04-23T10:32:03.879 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "initiated" 2024-04-23T10:32:03.879 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-23T10:32:03.879 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-04-23T10:32:03.879 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-04-23T10:31:21.483806+0000", 2024-04-23T10:32:03.879 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "header_read" 2024-04-23T10:32:03.880 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-23T10:32:03.880 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-04-23T10:32:03.880 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-04-23T10:31:21.483806+0000", 2024-04-23T10:32:03.880 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "throttled" 2024-04-23T10:32:03.880 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-23T10:32:03.880 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-04-23T10:32:03.880 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-04-23T10:31:21.483811+0000", 2024-04-23T10:32:03.880 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "all_read" 2024-04-23T10:32:03.880 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-23T10:32:03.880 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-04-23T10:32:03.880 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-04-23T10:31:21.487168+0000", 2024-04-23T10:32:03.880 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "dispatched" 2024-04-23T10:32:03.880 INFO:teuthology.orchestra.run.smithi114.stdout: } 2024-04-23T10:32:03.880 INFO:teuthology.orchestra.run.smithi114.stdout: ], 2024-04-23T10:32:03.880 INFO:teuthology.orchestra.run.smithi114.stdout: "locks": [] 2024-04-23T10:32:03.881 INFO:teuthology.orchestra.run.smithi114.stdout: } 2024-04-23T10:32:03.881 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-23T10:32:03.881 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-04-23T10:32:03.881 INFO:teuthology.orchestra.run.smithi114.stdout: "description": "internal op quiesce_inode:mds.2:110 fp=#0x100000001fa", 2024-04-23T10:32:03.881 INFO:teuthology.orchestra.run.smithi114.stdout: "initiated_at": "2024-04-23T10:31:21.448394+0000", 2024-04-23T10:32:03.881 INFO:teuthology.orchestra.run.smithi114.stdout: "age": 42.380351050000002, 2024-04-23T10:32:03.881 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 42.380704240999997, 2024-04-23T10:32:03.881 INFO:teuthology.orchestra.run.smithi114.stdout: "continuous": true, 2024-04-23T10:32:03.881 INFO:teuthology.orchestra.run.smithi114.stdout: "type_data": { 2024-04-23T10:32:03.881 INFO:teuthology.orchestra.run.smithi114.stdout: "result": -2147483648, 2024-04-23T10:32:03.881 INFO:teuthology.orchestra.run.smithi114.stdout: "flag_point": "quiesce complete for non-auth inode", 2024-04-23T10:32:03.881 INFO:teuthology.orchestra.run.smithi114.stdout: "reqid": { 2024-04-23T10:32:03.881 INFO:teuthology.orchestra.run.smithi114.stdout: "entity": { 2024-04-23T10:32:03.881 INFO:teuthology.orchestra.run.smithi114.stdout: "type": "mds", 2024-04-23T10:32:03.881 INFO:teuthology.orchestra.run.smithi114.stdout: "num": 2 2024-04-23T10:32:03.882 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-23T10:32:03.882 INFO:teuthology.orchestra.run.smithi114.stdout: "tid": 110 2024-04-23T10:32:03.882 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-23T10:32:03.882 INFO:teuthology.orchestra.run.smithi114.stdout: "op_type": "internal_op", 2024-04-23T10:32:03.882 INFO:teuthology.orchestra.run.smithi114.stdout: "internal_op": 5385, 2024-04-23T10:32:03.882 INFO:teuthology.orchestra.run.smithi114.stdout: "op_name": "quiesce_inode", 2024-04-23T10:32:03.882 INFO:teuthology.orchestra.run.smithi114.stdout: "events": [ 2024-04-23T10:32:03.882 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-04-23T10:32:03.882 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-04-23T10:31:21.448394+0000", 2024-04-23T10:32:03.882 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "initiated" 2024-04-23T10:32:03.882 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-23T10:32:03.882 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-04-23T10:32:03.882 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-04-23T10:31:21.448394+0000", 2024-04-23T10:32:03.882 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "header_read" 2024-04-23T10:32:03.882 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-23T10:32:03.883 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-04-23T10:32:03.883 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-04-23T10:31:21.448394+0000", 2024-04-23T10:32:03.883 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "throttled" 2024-04-23T10:32:03.883 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-23T10:32:03.883 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-04-23T10:32:03.883 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-04-23T10:31:21.448394+0000", 2024-04-23T10:32:03.883 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "all_read" 2024-04-23T10:32:03.883 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-23T10:32:03.883 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-04-23T10:32:03.883 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-04-23T10:31:21.448394+0000", 2024-04-23T10:32:03.883 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "dispatched" 2024-04-23T10:32:03.883 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-23T10:32:03.883 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-04-23T10:32:03.883 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-04-23T10:31:21.448493+0000", 2024-04-23T10:32:03.884 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "requesting remote authpins" 2024-04-23T10:32:03.884 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-23T10:32:03.884 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-04-23T10:32:03.884 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-04-23T10:31:21.449533+0000", 2024-04-23T10:32:03.884 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "acquired locks" 2024-04-23T10:32:03.884 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-23T10:32:03.884 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-04-23T10:32:03.884 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-04-23T10:31:21.509389+0000", 2024-04-23T10:32:03.884 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "quiesce complete for non-auth inode" 2024-04-23T10:32:03.884 INFO:teuthology.orchestra.run.smithi114.stdout: } 2024-04-23T10:32:03.884 INFO:teuthology.orchestra.run.smithi114.stdout: ], 2024-04-23T10:32:03.884 INFO:teuthology.orchestra.run.smithi114.stdout: "locks": [ 2024-04-23T10:32:03.884 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-04-23T10:32:03.884 INFO:teuthology.orchestra.run.smithi114.stdout: "object": { 2024-04-23T10:32:03.884 INFO:teuthology.orchestra.run.smithi114.stdout: "is_auth": false, 2024-04-23T10:32:03.885 INFO:teuthology.orchestra.run.smithi114.stdout: "auth_state": { 2024-04-23T10:32:03.885 INFO:teuthology.orchestra.run.smithi114.stdout: "replicas": {} 2024-04-23T10:32:03.885 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-23T10:32:03.885 INFO:teuthology.orchestra.run.smithi114.stdout: "replica_state": { 2024-04-23T10:32:03.885 INFO:teuthology.orchestra.run.smithi114.stdout: "authority": [ 2024-04-23T10:32:03.885 INFO:teuthology.orchestra.run.smithi114.stdout: 0, 2024-04-23T10:32:03.885 INFO:teuthology.orchestra.run.smithi114.stdout: -2 2024-04-23T10:32:03.885 INFO:teuthology.orchestra.run.smithi114.stdout: ], 2024-04-23T10:32:03.885 INFO:teuthology.orchestra.run.smithi114.stdout: "replica_nonce": 1 2024-04-23T10:32:03.885 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-23T10:32:03.885 INFO:teuthology.orchestra.run.smithi114.stdout: "auth_pins": 0, 2024-04-23T10:32:03.885 INFO:teuthology.orchestra.run.smithi114.stdout: "is_frozen": false, 2024-04-23T10:32:03.885 INFO:teuthology.orchestra.run.smithi114.stdout: "is_freezing": false, 2024-04-23T10:32:03.885 INFO:teuthology.orchestra.run.smithi114.stdout: "pins": { 2024-04-23T10:32:03.886 INFO:teuthology.orchestra.run.smithi114.stdout: "request": 1, 2024-04-23T10:32:03.886 INFO:teuthology.orchestra.run.smithi114.stdout: "lock": 1, 2024-04-23T10:32:03.886 INFO:teuthology.orchestra.run.smithi114.stdout: "dirfrag": 1, 2024-04-23T10:32:03.886 INFO:teuthology.orchestra.run.smithi114.stdout: "caps": 1, 2024-04-23T10:32:03.886 INFO:teuthology.orchestra.run.smithi114.stdout: "dirwaiter": 0, 2024-04-23T10:32:03.886 INFO:teuthology.orchestra.run.smithi114.stdout: "discoverbase": 0 2024-04-23T10:32:03.886 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-23T10:32:03.886 INFO:teuthology.orchestra.run.smithi114.stdout: "nref": 4 2024-04-23T10:32:03.886 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-23T10:32:03.886 INFO:teuthology.orchestra.run.smithi114.stdout: "object_string": "[inode 0x100000001fa [...7,head] /volumes/qa/sv_1/ rep@0.1 v434 snaprealm=0x55c68b7b06c0 f(v0 m2024-04-23T10:24:49.735055+0000 2=1+1) n(v29 rc2024-04-23T10:31:17.764716+0000 b21075191 80=64+16)/n(v0 rc2024-04-23T10:24:49.717706+0000 2=0+2) (inest mix) (iquiesce lock x=1 by request(mds.2:110 nref=3)) caps={15312=pAsLsXs/-@2} | request=1 lock=1 dirfrag=1 caps=1 dirwaiter=0 discoverbase=0 0x55c68ba07700]", 2024-04-23T10:32:03.886 INFO:teuthology.orchestra.run.smithi114.stdout: "lock": { 2024-04-23T10:32:03.886 INFO:teuthology.orchestra.run.smithi114.stdout: "gather_set": [], 2024-04-23T10:32:03.886 INFO:teuthology.orchestra.run.smithi114.stdout: "state": "lock", 2024-04-23T10:32:03.886 INFO:teuthology.orchestra.run.smithi114.stdout: "type": "iquiesce", 2024-04-23T10:32:03.887 INFO:teuthology.orchestra.run.smithi114.stdout: "is_leased": false, 2024-04-23T10:32:03.887 INFO:teuthology.orchestra.run.smithi114.stdout: "num_rdlocks": 0, 2024-04-23T10:32:03.887 INFO:teuthology.orchestra.run.smithi114.stdout: "num_wrlocks": 0, 2024-04-23T10:32:03.887 INFO:teuthology.orchestra.run.smithi114.stdout: "num_xlocks": 1, 2024-04-23T10:32:03.887 INFO:teuthology.orchestra.run.smithi114.stdout: "xlock_by": { 2024-04-23T10:32:03.887 INFO:teuthology.orchestra.run.smithi114.stdout: "reqid": { 2024-04-23T10:32:03.887 INFO:teuthology.orchestra.run.smithi114.stdout: "entity": { 2024-04-23T10:32:03.887 INFO:teuthology.orchestra.run.smithi114.stdout: "type": "mds", 2024-04-23T10:32:03.887 INFO:teuthology.orchestra.run.smithi114.stdout: "num": 2 2024-04-23T10:32:03.887 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-23T10:32:03.887 INFO:teuthology.orchestra.run.smithi114.stdout: "tid": 110 2024-04-23T10:32:03.887 INFO:teuthology.orchestra.run.smithi114.stdout: } 2024-04-23T10:32:03.887 INFO:teuthology.orchestra.run.smithi114.stdout: } 2024-04-23T10:32:03.887 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-23T10:32:03.888 INFO:teuthology.orchestra.run.smithi114.stdout: "flags": 4, 2024-04-23T10:32:03.888 INFO:teuthology.orchestra.run.smithi114.stdout: "wrlock_target": -1 2024-04-23T10:32:03.888 INFO:teuthology.orchestra.run.smithi114.stdout: } 2024-04-23T10:32:03.888 INFO:teuthology.orchestra.run.smithi114.stdout: ] 2024-04-23T10:32:03.888 INFO:teuthology.orchestra.run.smithi114.stdout: } 2024-04-23T10:32:03.888 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-23T10:32:03.888 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-04-23T10:32:03.888 INFO:teuthology.orchestra.run.smithi114.stdout: "description": "peer_request:mds.0:195", 2024-04-23T10:32:03.888 INFO:teuthology.orchestra.run.smithi114.stdout: "initiated_at": "2024-04-23T10:31:21.483857+0000", 2024-04-23T10:32:03.888 INFO:teuthology.orchestra.run.smithi114.stdout: "age": 42.344887266999997, 2024-04-23T10:32:03.888 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 42.345311197000001, 2024-04-23T10:32:03.888 INFO:teuthology.orchestra.run.smithi114.stdout: "continuous": false, 2024-04-23T10:32:03.888 INFO:teuthology.orchestra.run.smithi114.stdout: "type_data": { 2024-04-23T10:32:03.888 INFO:teuthology.orchestra.run.smithi114.stdout: "result": -2147483648, 2024-04-23T10:32:03.889 INFO:teuthology.orchestra.run.smithi114.stdout: "flag_point": "dispatched", 2024-04-23T10:32:03.889 INFO:teuthology.orchestra.run.smithi114.stdout: "reqid": { 2024-04-23T10:32:03.889 INFO:teuthology.orchestra.run.smithi114.stdout: "entity": { 2024-04-23T10:32:03.889 INFO:teuthology.orchestra.run.smithi114.stdout: "type": "mds", 2024-04-23T10:32:03.889 INFO:teuthology.orchestra.run.smithi114.stdout: "num": 0 2024-04-23T10:32:03.889 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-23T10:32:03.889 INFO:teuthology.orchestra.run.smithi114.stdout: "tid": 195 2024-04-23T10:32:03.889 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-23T10:32:03.889 INFO:teuthology.orchestra.run.smithi114.stdout: "op_type": "peer_request", 2024-04-23T10:32:03.889 INFO:teuthology.orchestra.run.smithi114.stdout: "leader_info": { 2024-04-23T10:32:03.889 INFO:teuthology.orchestra.run.smithi114.stdout: "leader": "0" 2024-04-23T10:32:03.889 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-23T10:32:03.889 INFO:teuthology.orchestra.run.smithi114.stdout: "events": [ 2024-04-23T10:32:03.889 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-04-23T10:32:03.890 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-04-23T10:31:21.483857+0000", 2024-04-23T10:32:03.890 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "initiated" 2024-04-23T10:32:03.890 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-23T10:32:03.890 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-04-23T10:32:03.890 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-04-23T10:31:21.483857+0000", 2024-04-23T10:32:03.890 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "header_read" 2024-04-23T10:32:03.890 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-23T10:32:03.890 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-04-23T10:32:03.890 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-04-23T10:31:21.483858+0000", 2024-04-23T10:32:03.890 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "throttled" 2024-04-23T10:32:03.890 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-23T10:32:03.890 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-04-23T10:32:03.890 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-04-23T10:31:21.483862+0000", 2024-04-23T10:32:03.890 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "all_read" 2024-04-23T10:32:03.891 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-23T10:32:03.891 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-04-23T10:32:03.891 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-04-23T10:31:21.487796+0000", 2024-04-23T10:32:03.891 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "dispatched" 2024-04-23T10:32:03.891 INFO:teuthology.orchestra.run.smithi114.stdout: } 2024-04-23T10:32:03.891 INFO:teuthology.orchestra.run.smithi114.stdout: ], 2024-04-23T10:32:03.891 INFO:teuthology.orchestra.run.smithi114.stdout: "locks": [] 2024-04-23T10:32:03.891 INFO:teuthology.orchestra.run.smithi114.stdout: } 2024-04-23T10:32:03.891 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-23T10:32:03.891 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-04-23T10:32:03.891 INFO:teuthology.orchestra.run.smithi114.stdout: "description": "internal op quiesce_inode:mds.2:111 fp=#0x100000001fb", 2024-04-23T10:32:03.891 INFO:teuthology.orchestra.run.smithi114.stdout: "initiated_at": "2024-04-23T10:31:21.449549+0000", 2024-04-23T10:32:03.891 INFO:teuthology.orchestra.run.smithi114.stdout: "age": 42.379195478, 2024-04-23T10:32:03.891 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 42.379643676999997, 2024-04-23T10:32:03.891 INFO:teuthology.orchestra.run.smithi114.stdout: "continuous": true, 2024-04-23T10:32:03.891 INFO:teuthology.orchestra.run.smithi114.stdout: "type_data": { 2024-04-23T10:32:03.892 INFO:teuthology.orchestra.run.smithi114.stdout: "result": -2147483648, 2024-04-23T10:32:03.892 INFO:teuthology.orchestra.run.smithi114.stdout: "flag_point": "quiesce complete for non-auth inode", 2024-04-23T10:32:03.892 INFO:teuthology.orchestra.run.smithi114.stdout: "reqid": { 2024-04-23T10:32:03.892 INFO:teuthology.orchestra.run.smithi114.stdout: "entity": { 2024-04-23T10:32:03.892 INFO:teuthology.orchestra.run.smithi114.stdout: "type": "mds", 2024-04-23T10:32:03.892 INFO:teuthology.orchestra.run.smithi114.stdout: "num": 2 2024-04-23T10:32:03.892 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-23T10:32:03.892 INFO:teuthology.orchestra.run.smithi114.stdout: "tid": 111 2024-04-23T10:32:03.892 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-23T10:32:03.892 INFO:teuthology.orchestra.run.smithi114.stdout: "op_type": "internal_op", 2024-04-23T10:32:03.892 INFO:teuthology.orchestra.run.smithi114.stdout: "internal_op": 5385, 2024-04-23T10:32:03.892 INFO:teuthology.orchestra.run.smithi114.stdout: "op_name": "quiesce_inode", 2024-04-23T10:32:03.892 INFO:teuthology.orchestra.run.smithi114.stdout: "events": [ 2024-04-23T10:32:03.892 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-04-23T10:32:03.892 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-04-23T10:31:21.449549+0000", 2024-04-23T10:32:03.893 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "initiated" 2024-04-23T10:32:03.893 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-23T10:32:03.893 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-04-23T10:32:03.893 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-04-23T10:31:21.449549+0000", 2024-04-23T10:32:03.893 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "header_read" 2024-04-23T10:32:03.893 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-23T10:32:03.893 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-04-23T10:32:03.893 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-04-23T10:31:21.449549+0000", 2024-04-23T10:32:03.893 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "throttled" 2024-04-23T10:32:03.893 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-23T10:32:03.893 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-04-23T10:32:03.893 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-04-23T10:31:21.449549+0000", 2024-04-23T10:32:03.893 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "all_read" 2024-04-23T10:32:03.893 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-23T10:32:03.893 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-04-23T10:32:03.893 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-04-23T10:31:21.449549+0000", 2024-04-23T10:32:03.894 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "dispatched" 2024-04-23T10:32:03.894 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-23T10:32:03.894 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-04-23T10:32:03.894 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-04-23T10:31:21.449635+0000", 2024-04-23T10:32:03.894 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "requesting remote authpins" 2024-04-23T10:32:03.894 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-23T10:32:03.894 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-04-23T10:32:03.894 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-04-23T10:31:21.450800+0000", 2024-04-23T10:32:03.894 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "acquired locks" 2024-04-23T10:32:03.894 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-23T10:32:03.894 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-04-23T10:32:03.894 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-04-23T10:31:21.509364+0000", 2024-04-23T10:32:03.894 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "quiesce complete for non-auth inode" 2024-04-23T10:32:03.894 INFO:teuthology.orchestra.run.smithi114.stdout: } 2024-04-23T10:32:03.895 INFO:teuthology.orchestra.run.smithi114.stdout: ], 2024-04-23T10:32:03.895 INFO:teuthology.orchestra.run.smithi114.stdout: "locks": [ 2024-04-23T10:32:03.895 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-04-23T10:32:03.895 INFO:teuthology.orchestra.run.smithi114.stdout: "object": { 2024-04-23T10:32:03.895 INFO:teuthology.orchestra.run.smithi114.stdout: "is_auth": false, 2024-04-23T10:32:03.895 INFO:teuthology.orchestra.run.smithi114.stdout: "auth_state": { 2024-04-23T10:32:03.895 INFO:teuthology.orchestra.run.smithi114.stdout: "replicas": {} 2024-04-23T10:32:03.895 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-23T10:32:03.895 INFO:teuthology.orchestra.run.smithi114.stdout: "replica_state": { 2024-04-23T10:32:03.895 INFO:teuthology.orchestra.run.smithi114.stdout: "authority": [ 2024-04-23T10:32:03.895 INFO:teuthology.orchestra.run.smithi114.stdout: 0, 2024-04-23T10:32:03.895 INFO:teuthology.orchestra.run.smithi114.stdout: -2 2024-04-23T10:32:03.895 INFO:teuthology.orchestra.run.smithi114.stdout: ], 2024-04-23T10:32:03.895 INFO:teuthology.orchestra.run.smithi114.stdout: "replica_nonce": 1 2024-04-23T10:32:03.896 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-23T10:32:03.896 INFO:teuthology.orchestra.run.smithi114.stdout: "auth_pins": 0, 2024-04-23T10:32:03.896 INFO:teuthology.orchestra.run.smithi114.stdout: "is_frozen": false, 2024-04-23T10:32:03.896 INFO:teuthology.orchestra.run.smithi114.stdout: "is_freezing": false, 2024-04-23T10:32:03.896 INFO:teuthology.orchestra.run.smithi114.stdout: "pins": { 2024-04-23T10:32:03.896 INFO:teuthology.orchestra.run.smithi114.stdout: "request": 1, 2024-04-23T10:32:03.896 INFO:teuthology.orchestra.run.smithi114.stdout: "lock": 1, 2024-04-23T10:32:03.896 INFO:teuthology.orchestra.run.smithi114.stdout: "dirfrag": 1, 2024-04-23T10:32:03.896 INFO:teuthology.orchestra.run.smithi114.stdout: "dirwaiter": 0, 2024-04-23T10:32:03.896 INFO:teuthology.orchestra.run.smithi114.stdout: "discoverbase": 0 2024-04-23T10:32:03.896 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-23T10:32:03.896 INFO:teuthology.orchestra.run.smithi114.stdout: "nref": 3 2024-04-23T10:32:03.896 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-23T10:32:03.896 INFO:teuthology.orchestra.run.smithi114.stdout: "object_string": "[inode 0x100000001fb [...7,head] /volumes/qa/sv_1/b206794e-7622-4ab8-8943-f3ae4e65b7a3/ rep@0.1 v426 f(v0 m2024-04-23T10:26:48.475842+0000 1=0+1) n(v25 rc2024-04-23T10:31:17.764716+0000 b21075074 78=63+15)/n(v0 rc2024-04-23T10:26:48.480842+0000 2=0+2) old_inodes=1 (inest mix) (iquiesce lock x=1 by request(mds.2:111 nref=3)) | request=1 lock=1 dirfrag=1 dirwaiter=0 discoverbase=0 0x55c68b9d1600]", 2024-04-23T10:32:03.896 INFO:teuthology.orchestra.run.smithi114.stdout: "lock": { 2024-04-23T10:32:03.896 INFO:teuthology.orchestra.run.smithi114.stdout: "gather_set": [], 2024-04-23T10:32:03.896 INFO:teuthology.orchestra.run.smithi114.stdout: "state": "lock", 2024-04-23T10:32:03.897 INFO:teuthology.orchestra.run.smithi114.stdout: "type": "iquiesce", 2024-04-23T10:32:03.897 INFO:teuthology.orchestra.run.smithi114.stdout: "is_leased": false, 2024-04-23T10:32:03.897 INFO:teuthology.orchestra.run.smithi114.stdout: "num_rdlocks": 0, 2024-04-23T10:32:03.897 INFO:teuthology.orchestra.run.smithi114.stdout: "num_wrlocks": 0, 2024-04-23T10:32:03.897 INFO:teuthology.orchestra.run.smithi114.stdout: "num_xlocks": 1, 2024-04-23T10:32:03.897 INFO:teuthology.orchestra.run.smithi114.stdout: "xlock_by": { 2024-04-23T10:32:03.897 INFO:teuthology.orchestra.run.smithi114.stdout: "reqid": { 2024-04-23T10:32:03.897 INFO:teuthology.orchestra.run.smithi114.stdout: "entity": { 2024-04-23T10:32:03.897 INFO:teuthology.orchestra.run.smithi114.stdout: "type": "mds", 2024-04-23T10:32:03.897 INFO:teuthology.orchestra.run.smithi114.stdout: "num": 2 2024-04-23T10:32:03.897 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-23T10:32:03.897 INFO:teuthology.orchestra.run.smithi114.stdout: "tid": 111 2024-04-23T10:32:03.897 INFO:teuthology.orchestra.run.smithi114.stdout: } 2024-04-23T10:32:03.897 INFO:teuthology.orchestra.run.smithi114.stdout: } 2024-04-23T10:32:03.897 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-23T10:32:03.898 INFO:teuthology.orchestra.run.smithi114.stdout: "flags": 4, 2024-04-23T10:32:03.898 INFO:teuthology.orchestra.run.smithi114.stdout: "wrlock_target": -1 2024-04-23T10:32:03.898 INFO:teuthology.orchestra.run.smithi114.stdout: } 2024-04-23T10:32:03.898 INFO:teuthology.orchestra.run.smithi114.stdout: ] 2024-04-23T10:32:03.898 INFO:teuthology.orchestra.run.smithi114.stdout: } 2024-04-23T10:32:03.898 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-23T10:32:03.898 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-04-23T10:32:03.898 INFO:teuthology.orchestra.run.smithi114.stdout: "description": "peer_request:mds.0:196", 2024-04-23T10:32:03.898 INFO:teuthology.orchestra.run.smithi114.stdout: "initiated_at": "2024-04-23T10:31:21.483910+0000", 2024-04-23T10:32:03.898 INFO:teuthology.orchestra.run.smithi114.stdout: "age": 42.344834759000001, 2024-04-23T10:32:03.898 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 42.345370498999998, 2024-04-23T10:32:03.898 INFO:teuthology.orchestra.run.smithi114.stdout: "continuous": false, 2024-04-23T10:32:03.898 INFO:teuthology.orchestra.run.smithi114.stdout: "type_data": { 2024-04-23T10:32:03.898 INFO:teuthology.orchestra.run.smithi114.stdout: "result": -2147483648, 2024-04-23T10:32:03.898 INFO:teuthology.orchestra.run.smithi114.stdout: "flag_point": "dispatched", 2024-04-23T10:32:03.899 INFO:teuthology.orchestra.run.smithi114.stdout: "reqid": { 2024-04-23T10:32:03.899 INFO:teuthology.orchestra.run.smithi114.stdout: "entity": { 2024-04-23T10:32:03.899 INFO:teuthology.orchestra.run.smithi114.stdout: "type": "mds", 2024-04-23T10:32:03.899 INFO:teuthology.orchestra.run.smithi114.stdout: "num": 0 2024-04-23T10:32:03.899 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-23T10:32:03.899 INFO:teuthology.orchestra.run.smithi114.stdout: "tid": 196 2024-04-23T10:32:03.899 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-23T10:32:03.899 INFO:teuthology.orchestra.run.smithi114.stdout: "op_type": "peer_request", 2024-04-23T10:32:03.899 INFO:teuthology.orchestra.run.smithi114.stdout: "leader_info": { 2024-04-23T10:32:03.899 INFO:teuthology.orchestra.run.smithi114.stdout: "leader": "0" 2024-04-23T10:32:03.899 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-23T10:32:03.899 INFO:teuthology.orchestra.run.smithi114.stdout: "events": [ 2024-04-23T10:32:03.899 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-04-23T10:32:03.899 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-04-23T10:31:21.483910+0000", 2024-04-23T10:32:03.899 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "initiated" 2024-04-23T10:32:03.900 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-23T10:32:03.900 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-04-23T10:32:03.900 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-04-23T10:31:21.483910+0000", 2024-04-23T10:32:03.900 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "header_read" 2024-04-23T10:32:03.900 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-23T10:32:03.900 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-04-23T10:32:03.900 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-04-23T10:31:21.483911+0000", 2024-04-23T10:32:03.900 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "throttled" 2024-04-23T10:32:03.900 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-23T10:32:03.900 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-04-23T10:32:03.900 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-04-23T10:31:21.483915+0000", 2024-04-23T10:32:03.900 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "all_read" 2024-04-23T10:32:03.900 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-23T10:32:03.900 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-04-23T10:32:03.900 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-04-23T10:31:21.488559+0000", 2024-04-23T10:32:03.901 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "dispatched" 2024-04-23T10:32:03.901 INFO:teuthology.orchestra.run.smithi114.stdout: } 2024-04-23T10:32:03.901 INFO:teuthology.orchestra.run.smithi114.stdout: ], 2024-04-23T10:32:03.901 INFO:teuthology.orchestra.run.smithi114.stdout: "locks": [] 2024-04-23T10:32:03.901 INFO:teuthology.orchestra.run.smithi114.stdout: } 2024-04-23T10:32:03.901 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-23T10:32:03.901 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-04-23T10:32:03.901 INFO:teuthology.orchestra.run.smithi114.stdout: "description": "internal op quiesce_inode:mds.2:112 fp=#0x10000000005", 2024-04-23T10:32:03.901 INFO:teuthology.orchestra.run.smithi114.stdout: "initiated_at": "2024-04-23T10:31:21.450819+0000", 2024-04-23T10:32:03.901 INFO:teuthology.orchestra.run.smithi114.stdout: "age": 42.377925865000002, 2024-04-23T10:32:03.901 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 42.378484041, 2024-04-23T10:32:03.901 INFO:teuthology.orchestra.run.smithi114.stdout: "continuous": true, 2024-04-23T10:32:03.901 INFO:teuthology.orchestra.run.smithi114.stdout: "type_data": { 2024-04-23T10:32:03.901 INFO:teuthology.orchestra.run.smithi114.stdout: "result": -2147483648, 2024-04-23T10:32:03.901 INFO:teuthology.orchestra.run.smithi114.stdout: "flag_point": "quiesce complete for non-auth inode", 2024-04-23T10:32:03.902 INFO:teuthology.orchestra.run.smithi114.stdout: "reqid": { 2024-04-23T10:32:03.902 INFO:teuthology.orchestra.run.smithi114.stdout: "entity": { 2024-04-23T10:32:03.902 INFO:teuthology.orchestra.run.smithi114.stdout: "type": "mds", 2024-04-23T10:32:03.902 INFO:teuthology.orchestra.run.smithi114.stdout: "num": 2 2024-04-23T10:32:03.902 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-23T10:32:03.902 INFO:teuthology.orchestra.run.smithi114.stdout: "tid": 112 2024-04-23T10:32:03.902 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-23T10:32:03.902 INFO:teuthology.orchestra.run.smithi114.stdout: "op_type": "internal_op", 2024-04-23T10:32:03.902 INFO:teuthology.orchestra.run.smithi114.stdout: "internal_op": 5385, 2024-04-23T10:32:03.902 INFO:teuthology.orchestra.run.smithi114.stdout: "op_name": "quiesce_inode", 2024-04-23T10:32:03.902 INFO:teuthology.orchestra.run.smithi114.stdout: "events": [ 2024-04-23T10:32:03.902 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-04-23T10:32:03.902 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-04-23T10:31:21.450819+0000", 2024-04-23T10:32:03.902 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "initiated" 2024-04-23T10:32:03.903 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-23T10:32:03.903 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-04-23T10:32:03.903 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-04-23T10:31:21.450819+0000", 2024-04-23T10:32:03.903 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "header_read" 2024-04-23T10:32:03.903 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-23T10:32:03.903 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-04-23T10:32:03.903 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-04-23T10:31:21.450819+0000", 2024-04-23T10:32:03.903 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "throttled" 2024-04-23T10:32:03.903 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-23T10:32:03.903 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-04-23T10:32:03.903 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-04-23T10:31:21.450819+0000", 2024-04-23T10:32:03.903 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "all_read" 2024-04-23T10:32:03.903 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-23T10:32:03.903 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-04-23T10:32:03.903 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-04-23T10:31:21.450819+0000", 2024-04-23T10:32:03.903 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "dispatched" 2024-04-23T10:32:03.904 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-23T10:32:03.904 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-04-23T10:32:03.904 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-04-23T10:31:21.450899+0000", 2024-04-23T10:32:03.904 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "requesting remote authpins" 2024-04-23T10:32:03.904 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-23T10:32:03.904 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-04-23T10:32:03.904 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-04-23T10:31:21.451997+0000", 2024-04-23T10:32:03.904 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "acquired locks" 2024-04-23T10:32:03.904 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-23T10:32:03.904 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-04-23T10:32:03.904 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-04-23T10:31:21.509342+0000", 2024-04-23T10:32:03.904 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "quiesce complete for non-auth inode" 2024-04-23T10:32:03.904 INFO:teuthology.orchestra.run.smithi114.stdout: } 2024-04-23T10:32:03.904 INFO:teuthology.orchestra.run.smithi114.stdout: ], 2024-04-23T10:32:03.904 INFO:teuthology.orchestra.run.smithi114.stdout: "locks": [ 2024-04-23T10:32:03.904 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-04-23T10:32:03.905 INFO:teuthology.orchestra.run.smithi114.stdout: "object": { 2024-04-23T10:32:03.905 INFO:teuthology.orchestra.run.smithi114.stdout: "is_auth": false, 2024-04-23T10:32:03.905 INFO:teuthology.orchestra.run.smithi114.stdout: "auth_state": { 2024-04-23T10:32:03.905 INFO:teuthology.orchestra.run.smithi114.stdout: "replicas": {} 2024-04-23T10:32:03.905 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-23T10:32:03.905 INFO:teuthology.orchestra.run.smithi114.stdout: "replica_state": { 2024-04-23T10:32:03.905 INFO:teuthology.orchestra.run.smithi114.stdout: "authority": [ 2024-04-23T10:32:03.905 INFO:teuthology.orchestra.run.smithi114.stdout: 0, 2024-04-23T10:32:03.905 INFO:teuthology.orchestra.run.smithi114.stdout: -2 2024-04-23T10:32:03.905 INFO:teuthology.orchestra.run.smithi114.stdout: ], 2024-04-23T10:32:03.905 INFO:teuthology.orchestra.run.smithi114.stdout: "replica_nonce": 1 2024-04-23T10:32:03.905 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-23T10:32:03.905 INFO:teuthology.orchestra.run.smithi114.stdout: "auth_pins": 0, 2024-04-23T10:32:03.905 INFO:teuthology.orchestra.run.smithi114.stdout: "is_frozen": false, 2024-04-23T10:32:03.905 INFO:teuthology.orchestra.run.smithi114.stdout: "is_freezing": false, 2024-04-23T10:32:03.905 INFO:teuthology.orchestra.run.smithi114.stdout: "pins": { 2024-04-23T10:32:03.906 INFO:teuthology.orchestra.run.smithi114.stdout: "request": 1, 2024-04-23T10:32:03.906 INFO:teuthology.orchestra.run.smithi114.stdout: "lock": 1, 2024-04-23T10:32:03.906 INFO:teuthology.orchestra.run.smithi114.stdout: "dirfrag": 1 2024-04-23T10:32:03.906 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-23T10:32:03.906 INFO:teuthology.orchestra.run.smithi114.stdout: "nref": 3 2024-04-23T10:32:03.906 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-23T10:32:03.906 INFO:teuthology.orchestra.run.smithi114.stdout: "object_string": "[inode 0x10000000005 [...7,head] /volumes/qa/sv_1/b206794e-7622-4ab8-8943-f3ae4e65b7a3/client.0/ rep@0.1 v421 f(v0 m2024-04-23T10:27:42.521612+0000 1=0+1) n(v32 rc2024-04-23T10:31:17.764716+0000 b21075074 77=63+14)/n(v0 rc2024-04-23T10:27:42.521612+0000 2=0+2) old_inodes=1 (inest mix) (iquiesce lock x=1 by request(mds.2:112 nref=3)) | request=1 lock=1 dirfrag=1 0x55c68b9d1b80]", 2024-04-23T10:32:03.906 INFO:teuthology.orchestra.run.smithi114.stdout: "lock": { 2024-04-23T10:32:03.906 INFO:teuthology.orchestra.run.smithi114.stdout: "gather_set": [], 2024-04-23T10:32:03.906 INFO:teuthology.orchestra.run.smithi114.stdout: "state": "lock", 2024-04-23T10:32:03.906 INFO:teuthology.orchestra.run.smithi114.stdout: "type": "iquiesce", 2024-04-23T10:32:03.906 INFO:teuthology.orchestra.run.smithi114.stdout: "is_leased": false, 2024-04-23T10:32:03.906 INFO:teuthology.orchestra.run.smithi114.stdout: "num_rdlocks": 0, 2024-04-23T10:32:03.906 INFO:teuthology.orchestra.run.smithi114.stdout: "num_wrlocks": 0, 2024-04-23T10:32:03.906 INFO:teuthology.orchestra.run.smithi114.stdout: "num_xlocks": 1, 2024-04-23T10:32:03.907 INFO:teuthology.orchestra.run.smithi114.stdout: "xlock_by": { 2024-04-23T10:32:03.907 INFO:teuthology.orchestra.run.smithi114.stdout: "reqid": { 2024-04-23T10:32:03.907 INFO:teuthology.orchestra.run.smithi114.stdout: "entity": { 2024-04-23T10:32:03.907 INFO:teuthology.orchestra.run.smithi114.stdout: "type": "mds", 2024-04-23T10:32:03.907 INFO:teuthology.orchestra.run.smithi114.stdout: "num": 2 2024-04-23T10:32:03.907 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-23T10:32:03.907 INFO:teuthology.orchestra.run.smithi114.stdout: "tid": 112 2024-04-23T10:32:03.907 INFO:teuthology.orchestra.run.smithi114.stdout: } 2024-04-23T10:32:03.907 INFO:teuthology.orchestra.run.smithi114.stdout: } 2024-04-23T10:32:03.907 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-23T10:32:03.907 INFO:teuthology.orchestra.run.smithi114.stdout: "flags": 4, 2024-04-23T10:32:03.907 INFO:teuthology.orchestra.run.smithi114.stdout: "wrlock_target": -1 2024-04-23T10:32:03.907 INFO:teuthology.orchestra.run.smithi114.stdout: } 2024-04-23T10:32:03.907 INFO:teuthology.orchestra.run.smithi114.stdout: ] 2024-04-23T10:32:03.908 INFO:teuthology.orchestra.run.smithi114.stdout: } 2024-04-23T10:32:03.908 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-23T10:32:03.908 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-04-23T10:32:03.908 INFO:teuthology.orchestra.run.smithi114.stdout: "description": "peer_request:mds.0:197", 2024-04-23T10:32:03.908 INFO:teuthology.orchestra.run.smithi114.stdout: "initiated_at": "2024-04-23T10:31:21.483962+0000", 2024-04-23T10:32:03.908 INFO:teuthology.orchestra.run.smithi114.stdout: "age": 42.344782944000002, 2024-04-23T10:32:03.908 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 42.345402452000002, 2024-04-23T10:32:03.908 INFO:teuthology.orchestra.run.smithi114.stdout: "continuous": false, 2024-04-23T10:32:03.908 INFO:teuthology.orchestra.run.smithi114.stdout: "type_data": { 2024-04-23T10:32:03.908 INFO:teuthology.orchestra.run.smithi114.stdout: "result": -2147483648, 2024-04-23T10:32:03.908 INFO:teuthology.orchestra.run.smithi114.stdout: "flag_point": "dispatched", 2024-04-23T10:32:03.908 INFO:teuthology.orchestra.run.smithi114.stdout: "reqid": { 2024-04-23T10:32:03.908 INFO:teuthology.orchestra.run.smithi114.stdout: "entity": { 2024-04-23T10:32:03.908 INFO:teuthology.orchestra.run.smithi114.stdout: "type": "mds", 2024-04-23T10:32:03.908 INFO:teuthology.orchestra.run.smithi114.stdout: "num": 0 2024-04-23T10:32:03.909 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-23T10:32:03.909 INFO:teuthology.orchestra.run.smithi114.stdout: "tid": 197 2024-04-23T10:32:03.909 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-23T10:32:03.909 INFO:teuthology.orchestra.run.smithi114.stdout: "op_type": "peer_request", 2024-04-23T10:32:03.909 INFO:teuthology.orchestra.run.smithi114.stdout: "leader_info": { 2024-04-23T10:32:03.909 INFO:teuthology.orchestra.run.smithi114.stdout: "leader": "0" 2024-04-23T10:32:03.909 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-23T10:32:03.909 INFO:teuthology.orchestra.run.smithi114.stdout: "events": [ 2024-04-23T10:32:03.909 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-04-23T10:32:03.909 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-04-23T10:31:21.483962+0000", 2024-04-23T10:32:03.909 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "initiated" 2024-04-23T10:32:03.909 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-23T10:32:03.909 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-04-23T10:32:03.909 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-04-23T10:31:21.483962+0000", 2024-04-23T10:32:03.909 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "header_read" 2024-04-23T10:32:03.910 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-23T10:32:03.910 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-04-23T10:32:03.910 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-04-23T10:31:21.483962+0000", 2024-04-23T10:32:03.910 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "throttled" 2024-04-23T10:32:03.910 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-23T10:32:03.910 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-04-23T10:32:03.910 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-04-23T10:31:21.483966+0000", 2024-04-23T10:32:03.910 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "all_read" 2024-04-23T10:32:03.910 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-23T10:32:03.910 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-04-23T10:32:03.910 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-04-23T10:31:21.488644+0000", 2024-04-23T10:32:03.910 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "dispatched" 2024-04-23T10:32:03.910 INFO:teuthology.orchestra.run.smithi114.stdout: } 2024-04-23T10:32:03.911 INFO:teuthology.orchestra.run.smithi114.stdout: ], 2024-04-23T10:32:03.911 INFO:teuthology.orchestra.run.smithi114.stdout: "locks": [] 2024-04-23T10:32:03.911 INFO:teuthology.orchestra.run.smithi114.stdout: } 2024-04-23T10:32:03.911 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-23T10:32:03.911 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-04-23T10:32:03.911 INFO:teuthology.orchestra.run.smithi114.stdout: "description": "internal op quiesce_inode:mds.2:113 fp=#0x10000000006", 2024-04-23T10:32:03.911 INFO:teuthology.orchestra.run.smithi114.stdout: "initiated_at": "2024-04-23T10:31:21.452009+0000", 2024-04-23T10:32:03.911 INFO:teuthology.orchestra.run.smithi114.stdout: "age": 42.376735879000002, 2024-04-23T10:32:03.911 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 42.377379026, 2024-04-23T10:32:03.911 INFO:teuthology.orchestra.run.smithi114.stdout: "continuous": true, 2024-04-23T10:32:03.911 INFO:teuthology.orchestra.run.smithi114.stdout: "type_data": { 2024-04-23T10:32:03.911 INFO:teuthology.orchestra.run.smithi114.stdout: "result": -2147483648, 2024-04-23T10:32:03.911 INFO:teuthology.orchestra.run.smithi114.stdout: "flag_point": "quiesce complete for non-auth inode", 2024-04-23T10:32:03.911 INFO:teuthology.orchestra.run.smithi114.stdout: "reqid": { 2024-04-23T10:32:03.911 INFO:teuthology.orchestra.run.smithi114.stdout: "entity": { 2024-04-23T10:32:03.912 INFO:teuthology.orchestra.run.smithi114.stdout: "type": "mds", 2024-04-23T10:32:03.912 INFO:teuthology.orchestra.run.smithi114.stdout: "num": 2 2024-04-23T10:32:03.912 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-23T10:32:03.912 INFO:teuthology.orchestra.run.smithi114.stdout: "tid": 113 2024-04-23T10:32:03.912 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-23T10:32:03.912 INFO:teuthology.orchestra.run.smithi114.stdout: "op_type": "internal_op", 2024-04-23T10:32:03.912 INFO:teuthology.orchestra.run.smithi114.stdout: "internal_op": 5385, 2024-04-23T10:32:03.912 INFO:teuthology.orchestra.run.smithi114.stdout: "op_name": "quiesce_inode", 2024-04-23T10:32:03.912 INFO:teuthology.orchestra.run.smithi114.stdout: "events": [ 2024-04-23T10:32:03.912 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-04-23T10:32:03.912 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-04-23T10:31:21.452009+0000", 2024-04-23T10:32:03.912 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "initiated" 2024-04-23T10:32:03.912 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-23T10:32:03.912 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-04-23T10:32:03.913 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-04-23T10:31:21.452009+0000", 2024-04-23T10:32:03.913 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "header_read" 2024-04-23T10:32:03.913 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-23T10:32:03.913 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-04-23T10:32:03.913 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-04-23T10:31:21.452009+0000", 2024-04-23T10:32:03.913 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "throttled" 2024-04-23T10:32:03.913 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-23T10:32:03.913 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-04-23T10:32:03.913 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-04-23T10:31:21.452009+0000", 2024-04-23T10:32:03.913 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "all_read" 2024-04-23T10:32:03.913 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-23T10:32:03.913 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-04-23T10:32:03.913 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-04-23T10:31:21.452009+0000", 2024-04-23T10:32:03.913 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "dispatched" 2024-04-23T10:32:03.914 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-23T10:32:03.914 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-04-23T10:32:03.914 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-04-23T10:31:21.452087+0000", 2024-04-23T10:32:03.914 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "requesting remote authpins" 2024-04-23T10:32:03.914 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-23T10:32:03.914 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-04-23T10:32:03.914 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-04-23T10:31:21.453420+0000", 2024-04-23T10:32:03.914 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "acquired locks" 2024-04-23T10:32:03.914 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-23T10:32:03.914 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-04-23T10:32:03.914 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-04-23T10:31:21.509319+0000", 2024-04-23T10:32:03.914 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "quiesce complete for non-auth inode" 2024-04-23T10:32:03.914 INFO:teuthology.orchestra.run.smithi114.stdout: } 2024-04-23T10:32:03.914 INFO:teuthology.orchestra.run.smithi114.stdout: ], 2024-04-23T10:32:03.915 INFO:teuthology.orchestra.run.smithi114.stdout: "locks": [ 2024-04-23T10:32:03.915 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-04-23T10:32:03.915 INFO:teuthology.orchestra.run.smithi114.stdout: "object": { 2024-04-23T10:32:03.915 INFO:teuthology.orchestra.run.smithi114.stdout: "is_auth": false, 2024-04-23T10:32:03.915 INFO:teuthology.orchestra.run.smithi114.stdout: "auth_state": { 2024-04-23T10:32:03.915 INFO:teuthology.orchestra.run.smithi114.stdout: "replicas": {} 2024-04-23T10:32:03.915 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-23T10:32:03.915 INFO:teuthology.orchestra.run.smithi114.stdout: "replica_state": { 2024-04-23T10:32:03.915 INFO:teuthology.orchestra.run.smithi114.stdout: "authority": [ 2024-04-23T10:32:03.915 INFO:teuthology.orchestra.run.smithi114.stdout: 0, 2024-04-23T10:32:03.915 INFO:teuthology.orchestra.run.smithi114.stdout: -2 2024-04-23T10:32:03.915 INFO:teuthology.orchestra.run.smithi114.stdout: ], 2024-04-23T10:32:03.915 INFO:teuthology.orchestra.run.smithi114.stdout: "replica_nonce": 1 2024-04-23T10:32:03.915 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-23T10:32:03.916 INFO:teuthology.orchestra.run.smithi114.stdout: "auth_pins": 0, 2024-04-23T10:32:03.916 INFO:teuthology.orchestra.run.smithi114.stdout: "is_frozen": false, 2024-04-23T10:32:03.916 INFO:teuthology.orchestra.run.smithi114.stdout: "is_freezing": false, 2024-04-23T10:32:03.916 INFO:teuthology.orchestra.run.smithi114.stdout: "pins": { 2024-04-23T10:32:03.916 INFO:teuthology.orchestra.run.smithi114.stdout: "request": 1, 2024-04-23T10:32:03.916 INFO:teuthology.orchestra.run.smithi114.stdout: "lock": 1, 2024-04-23T10:32:03.916 INFO:teuthology.orchestra.run.smithi114.stdout: "dirfrag": 1 2024-04-23T10:32:03.916 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-23T10:32:03.916 INFO:teuthology.orchestra.run.smithi114.stdout: "nref": 3 2024-04-23T10:32:03.916 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-23T10:32:03.916 INFO:teuthology.orchestra.run.smithi114.stdout: "object_string": "[inode 0x10000000006 [...7,head] /volumes/qa/sv_1/b206794e-7622-4ab8-8943-f3ae4e65b7a3/client.0/tmp/ rep@0.1 v408 f(v0 m2024-04-23T10:27:51.919399+0000 1=0+1) n(v30 rc2024-04-23T10:31:17.764716+0000 b21075074 76=63+13)/n(v0 rc2024-04-23T10:27:51.919399+0000 2=0+2) (inest mix) (iquiesce lock x=1 by request(mds.2:113 nref=3)) | request=1 lock=1 dirfrag=1 0x55c68b9d2100]", 2024-04-23T10:32:03.916 INFO:teuthology.orchestra.run.smithi114.stdout: "lock": { 2024-04-23T10:32:03.916 INFO:teuthology.orchestra.run.smithi114.stdout: "gather_set": [], 2024-04-23T10:32:03.916 INFO:teuthology.orchestra.run.smithi114.stdout: "state": "lock", 2024-04-23T10:32:03.917 INFO:teuthology.orchestra.run.smithi114.stdout: "type": "iquiesce", 2024-04-23T10:32:03.917 INFO:teuthology.orchestra.run.smithi114.stdout: "is_leased": false, 2024-04-23T10:32:03.917 INFO:teuthology.orchestra.run.smithi114.stdout: "num_rdlocks": 0, 2024-04-23T10:32:03.917 INFO:teuthology.orchestra.run.smithi114.stdout: "num_wrlocks": 0, 2024-04-23T10:32:03.917 INFO:teuthology.orchestra.run.smithi114.stdout: "num_xlocks": 1, 2024-04-23T10:32:03.917 INFO:teuthology.orchestra.run.smithi114.stdout: "xlock_by": { 2024-04-23T10:32:03.917 INFO:teuthology.orchestra.run.smithi114.stdout: "reqid": { 2024-04-23T10:32:03.917 INFO:teuthology.orchestra.run.smithi114.stdout: "entity": { 2024-04-23T10:32:03.917 INFO:teuthology.orchestra.run.smithi114.stdout: "type": "mds", 2024-04-23T10:32:03.917 INFO:teuthology.orchestra.run.smithi114.stdout: "num": 2 2024-04-23T10:32:03.917 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-23T10:32:03.917 INFO:teuthology.orchestra.run.smithi114.stdout: "tid": 113 2024-04-23T10:32:03.917 INFO:teuthology.orchestra.run.smithi114.stdout: } 2024-04-23T10:32:03.917 INFO:teuthology.orchestra.run.smithi114.stdout: } 2024-04-23T10:32:03.918 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-23T10:32:03.918 INFO:teuthology.orchestra.run.smithi114.stdout: "flags": 4, 2024-04-23T10:32:03.918 INFO:teuthology.orchestra.run.smithi114.stdout: "wrlock_target": -1 2024-04-23T10:32:03.918 INFO:teuthology.orchestra.run.smithi114.stdout: } 2024-04-23T10:32:03.918 INFO:teuthology.orchestra.run.smithi114.stdout: ] 2024-04-23T10:32:03.918 INFO:teuthology.orchestra.run.smithi114.stdout: } 2024-04-23T10:32:03.918 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-23T10:32:03.918 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-04-23T10:32:03.918 INFO:teuthology.orchestra.run.smithi114.stdout: "description": "peer_request:mds.0:198", 2024-04-23T10:32:03.918 INFO:teuthology.orchestra.run.smithi114.stdout: "initiated_at": "2024-04-23T10:31:21.484024+0000", 2024-04-23T10:32:03.918 INFO:teuthology.orchestra.run.smithi114.stdout: "age": 42.344720899000002, 2024-04-23T10:32:03.918 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 42.345440551999999, 2024-04-23T10:32:03.918 INFO:teuthology.orchestra.run.smithi114.stdout: "continuous": false, 2024-04-23T10:32:03.918 INFO:teuthology.orchestra.run.smithi114.stdout: "type_data": { 2024-04-23T10:32:03.919 INFO:teuthology.orchestra.run.smithi114.stdout: "result": -2147483648, 2024-04-23T10:32:03.919 INFO:teuthology.orchestra.run.smithi114.stdout: "flag_point": "dispatched", 2024-04-23T10:32:03.919 INFO:teuthology.orchestra.run.smithi114.stdout: "reqid": { 2024-04-23T10:32:03.919 INFO:teuthology.orchestra.run.smithi114.stdout: "entity": { 2024-04-23T10:32:03.919 INFO:teuthology.orchestra.run.smithi114.stdout: "type": "mds", 2024-04-23T10:32:03.919 INFO:teuthology.orchestra.run.smithi114.stdout: "num": 0 2024-04-23T10:32:03.919 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-23T10:32:03.919 INFO:teuthology.orchestra.run.smithi114.stdout: "tid": 198 2024-04-23T10:32:03.919 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-23T10:32:03.919 INFO:teuthology.orchestra.run.smithi114.stdout: "op_type": "peer_request", 2024-04-23T10:32:03.919 INFO:teuthology.orchestra.run.smithi114.stdout: "leader_info": { 2024-04-23T10:32:03.919 INFO:teuthology.orchestra.run.smithi114.stdout: "leader": "0" 2024-04-23T10:32:03.919 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-23T10:32:03.919 INFO:teuthology.orchestra.run.smithi114.stdout: "events": [ 2024-04-23T10:32:03.920 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-04-23T10:32:03.920 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-04-23T10:31:21.484024+0000", 2024-04-23T10:32:03.920 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "initiated" 2024-04-23T10:32:03.920 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-23T10:32:03.920 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-04-23T10:32:03.920 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-04-23T10:31:21.484024+0000", 2024-04-23T10:32:03.920 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "header_read" 2024-04-23T10:32:03.920 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-23T10:32:03.920 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-04-23T10:32:03.920 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-04-23T10:31:21.484025+0000", 2024-04-23T10:32:03.920 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "throttled" 2024-04-23T10:32:03.920 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-23T10:32:03.920 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-04-23T10:32:03.920 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-04-23T10:31:21.484032+0000", 2024-04-23T10:32:03.920 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "all_read" 2024-04-23T10:32:03.921 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-23T10:32:03.921 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-04-23T10:32:03.921 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-04-23T10:31:21.488737+0000", 2024-04-23T10:32:03.921 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "dispatched" 2024-04-23T10:32:03.921 INFO:teuthology.orchestra.run.smithi114.stdout: } 2024-04-23T10:32:03.921 INFO:teuthology.orchestra.run.smithi114.stdout: ], 2024-04-23T10:32:03.921 INFO:teuthology.orchestra.run.smithi114.stdout: "locks": [] 2024-04-23T10:32:03.921 INFO:teuthology.orchestra.run.smithi114.stdout: } 2024-04-23T10:32:03.921 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-23T10:32:03.921 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-04-23T10:32:03.921 INFO:teuthology.orchestra.run.smithi114.stdout: "description": "internal op quiesce_inode:mds.2:114 fp=#0x10000000007", 2024-04-23T10:32:03.921 INFO:teuthology.orchestra.run.smithi114.stdout: "initiated_at": "2024-04-23T10:31:21.453434+0000", 2024-04-23T10:32:03.921 INFO:teuthology.orchestra.run.smithi114.stdout: "age": 42.375310267000003, 2024-04-23T10:32:03.921 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 42.376054576000001, 2024-04-23T10:32:03.921 INFO:teuthology.orchestra.run.smithi114.stdout: "continuous": true, 2024-04-23T10:32:03.922 INFO:teuthology.orchestra.run.smithi114.stdout: "type_data": { 2024-04-23T10:32:03.922 INFO:teuthology.orchestra.run.smithi114.stdout: "result": -2147483648, 2024-04-23T10:32:03.922 INFO:teuthology.orchestra.run.smithi114.stdout: "flag_point": "quiesce complete for non-auth inode", 2024-04-23T10:32:03.922 INFO:teuthology.orchestra.run.smithi114.stdout: "reqid": { 2024-04-23T10:32:03.922 INFO:teuthology.orchestra.run.smithi114.stdout: "entity": { 2024-04-23T10:32:03.922 INFO:teuthology.orchestra.run.smithi114.stdout: "type": "mds", 2024-04-23T10:32:03.922 INFO:teuthology.orchestra.run.smithi114.stdout: "num": 2 2024-04-23T10:32:03.922 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-23T10:32:03.922 INFO:teuthology.orchestra.run.smithi114.stdout: "tid": 114 2024-04-23T10:32:03.922 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-23T10:32:03.922 INFO:teuthology.orchestra.run.smithi114.stdout: "op_type": "internal_op", 2024-04-23T10:32:03.922 INFO:teuthology.orchestra.run.smithi114.stdout: "internal_op": 5385, 2024-04-23T10:32:03.922 INFO:teuthology.orchestra.run.smithi114.stdout: "op_name": "quiesce_inode", 2024-04-23T10:32:03.922 INFO:teuthology.orchestra.run.smithi114.stdout: "events": [ 2024-04-23T10:32:03.923 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-04-23T10:32:03.923 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-04-23T10:31:21.453434+0000", 2024-04-23T10:32:03.923 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "initiated" 2024-04-23T10:32:03.923 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-23T10:32:03.923 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-04-23T10:32:03.923 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-04-23T10:31:21.453434+0000", 2024-04-23T10:32:03.923 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "header_read" 2024-04-23T10:32:03.923 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-23T10:32:03.923 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-04-23T10:32:03.923 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-04-23T10:31:21.453434+0000", 2024-04-23T10:32:03.923 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "throttled" 2024-04-23T10:32:03.923 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-23T10:32:03.923 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-04-23T10:32:03.923 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-04-23T10:31:21.453434+0000", 2024-04-23T10:32:03.923 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "all_read" 2024-04-23T10:32:03.924 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-23T10:32:03.924 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-04-23T10:32:03.924 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-04-23T10:31:21.453434+0000", 2024-04-23T10:32:03.924 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "dispatched" 2024-04-23T10:32:03.924 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-23T10:32:03.924 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-04-23T10:32:03.924 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-04-23T10:31:21.453528+0000", 2024-04-23T10:32:03.924 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "requesting remote authpins" 2024-04-23T10:32:03.924 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-23T10:32:03.924 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-04-23T10:32:03.924 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-04-23T10:31:21.455137+0000", 2024-04-23T10:32:03.924 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "acquired locks" 2024-04-23T10:32:03.924 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-23T10:32:03.924 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-04-23T10:32:03.924 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-04-23T10:31:21.509290+0000", 2024-04-23T10:32:03.924 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "quiesce complete for non-auth inode" 2024-04-23T10:32:03.925 INFO:teuthology.orchestra.run.smithi114.stdout: } 2024-04-23T10:32:03.925 INFO:teuthology.orchestra.run.smithi114.stdout: ], 2024-04-23T10:32:03.925 INFO:teuthology.orchestra.run.smithi114.stdout: "locks": [ 2024-04-23T10:32:03.925 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-04-23T10:32:03.925 INFO:teuthology.orchestra.run.smithi114.stdout: "object": { 2024-04-23T10:32:03.925 INFO:teuthology.orchestra.run.smithi114.stdout: "is_auth": false, 2024-04-23T10:32:03.925 INFO:teuthology.orchestra.run.smithi114.stdout: "auth_state": { 2024-04-23T10:32:03.925 INFO:teuthology.orchestra.run.smithi114.stdout: "replicas": {} 2024-04-23T10:32:03.925 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-23T10:32:03.925 INFO:teuthology.orchestra.run.smithi114.stdout: "replica_state": { 2024-04-23T10:32:03.925 INFO:teuthology.orchestra.run.smithi114.stdout: "authority": [ 2024-04-23T10:32:03.925 INFO:teuthology.orchestra.run.smithi114.stdout: 0, 2024-04-23T10:32:03.925 INFO:teuthology.orchestra.run.smithi114.stdout: -2 2024-04-23T10:32:03.925 INFO:teuthology.orchestra.run.smithi114.stdout: ], 2024-04-23T10:32:03.926 INFO:teuthology.orchestra.run.smithi114.stdout: "replica_nonce": 1 2024-04-23T10:32:03.926 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-23T10:32:03.926 INFO:teuthology.orchestra.run.smithi114.stdout: "auth_pins": 0, 2024-04-23T10:32:03.926 INFO:teuthology.orchestra.run.smithi114.stdout: "is_frozen": false, 2024-04-23T10:32:03.926 INFO:teuthology.orchestra.run.smithi114.stdout: "is_freezing": false, 2024-04-23T10:32:03.926 INFO:teuthology.orchestra.run.smithi114.stdout: "pins": { 2024-04-23T10:32:03.926 INFO:teuthology.orchestra.run.smithi114.stdout: "request": 1, 2024-04-23T10:32:03.926 INFO:teuthology.orchestra.run.smithi114.stdout: "lock": 1, 2024-04-23T10:32:03.926 INFO:teuthology.orchestra.run.smithi114.stdout: "dirfrag": 1 2024-04-23T10:32:03.926 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-23T10:32:03.926 INFO:teuthology.orchestra.run.smithi114.stdout: "nref": 3 2024-04-23T10:32:03.926 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-23T10:32:03.926 INFO:teuthology.orchestra.run.smithi114.stdout: "object_string": "[inode 0x10000000007 [...7,head] /volumes/qa/sv_1/b206794e-7622-4ab8-8943-f3ae4e65b7a3/client.0/tmp/clients/ rep@0.1 v412 f(v0 m2024-04-23T10:27:51.919399+0000 1=0+1) n(v30 rc2024-04-23T10:31:17.764716+0000 b21075074 75=63+12)/n(v0 rc2024-04-23T10:27:51.919399+0000 2=0+2) (inest mix) (iquiesce lock x=1 by request(mds.2:114 nref=3)) | request=1 lock=1 dirfrag=1 0x55c68b9d2680]", 2024-04-23T10:32:03.926 INFO:teuthology.orchestra.run.smithi114.stdout: "lock": { 2024-04-23T10:32:03.926 INFO:teuthology.orchestra.run.smithi114.stdout: "gather_set": [], 2024-04-23T10:32:03.927 INFO:teuthology.orchestra.run.smithi114.stdout: "state": "lock", 2024-04-23T10:32:03.927 INFO:teuthology.orchestra.run.smithi114.stdout: "type": "iquiesce", 2024-04-23T10:32:03.927 INFO:teuthology.orchestra.run.smithi114.stdout: "is_leased": false, 2024-04-23T10:32:03.927 INFO:teuthology.orchestra.run.smithi114.stdout: "num_rdlocks": 0, 2024-04-23T10:32:03.927 INFO:teuthology.orchestra.run.smithi114.stdout: "num_wrlocks": 0, 2024-04-23T10:32:03.927 INFO:teuthology.orchestra.run.smithi114.stdout: "num_xlocks": 1, 2024-04-23T10:32:03.927 INFO:teuthology.orchestra.run.smithi114.stdout: "xlock_by": { 2024-04-23T10:32:03.927 INFO:teuthology.orchestra.run.smithi114.stdout: "reqid": { 2024-04-23T10:32:03.927 INFO:teuthology.orchestra.run.smithi114.stdout: "entity": { 2024-04-23T10:32:03.927 INFO:teuthology.orchestra.run.smithi114.stdout: "type": "mds", 2024-04-23T10:32:03.927 INFO:teuthology.orchestra.run.smithi114.stdout: "num": 2 2024-04-23T10:32:03.927 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-23T10:32:03.927 INFO:teuthology.orchestra.run.smithi114.stdout: "tid": 114 2024-04-23T10:32:03.927 INFO:teuthology.orchestra.run.smithi114.stdout: } 2024-04-23T10:32:03.927 INFO:teuthology.orchestra.run.smithi114.stdout: } 2024-04-23T10:32:03.928 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-23T10:32:03.928 INFO:teuthology.orchestra.run.smithi114.stdout: "flags": 4, 2024-04-23T10:32:03.928 INFO:teuthology.orchestra.run.smithi114.stdout: "wrlock_target": -1 2024-04-23T10:32:03.928 INFO:teuthology.orchestra.run.smithi114.stdout: } 2024-04-23T10:32:03.928 INFO:teuthology.orchestra.run.smithi114.stdout: ] 2024-04-23T10:32:03.928 INFO:teuthology.orchestra.run.smithi114.stdout: } 2024-04-23T10:32:03.928 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-23T10:32:03.928 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-04-23T10:32:03.928 INFO:teuthology.orchestra.run.smithi114.stdout: "description": "peer_request:mds.0:199", 2024-04-23T10:32:03.928 INFO:teuthology.orchestra.run.smithi114.stdout: "initiated_at": "2024-04-23T10:31:21.484068+0000", 2024-04-23T10:32:03.928 INFO:teuthology.orchestra.run.smithi114.stdout: "age": 42.344677247999996, 2024-04-23T10:32:03.928 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 42.345484798000001, 2024-04-23T10:32:03.928 INFO:teuthology.orchestra.run.smithi114.stdout: "continuous": false, 2024-04-23T10:32:03.928 INFO:teuthology.orchestra.run.smithi114.stdout: "type_data": { 2024-04-23T10:32:03.928 INFO:teuthology.orchestra.run.smithi114.stdout: "result": -2147483648, 2024-04-23T10:32:03.929 INFO:teuthology.orchestra.run.smithi114.stdout: "flag_point": "dispatched", 2024-04-23T10:32:03.929 INFO:teuthology.orchestra.run.smithi114.stdout: "reqid": { 2024-04-23T10:32:03.929 INFO:teuthology.orchestra.run.smithi114.stdout: "entity": { 2024-04-23T10:32:03.929 INFO:teuthology.orchestra.run.smithi114.stdout: "type": "mds", 2024-04-23T10:32:03.929 INFO:teuthology.orchestra.run.smithi114.stdout: "num": 0 2024-04-23T10:32:03.929 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-23T10:32:03.929 INFO:teuthology.orchestra.run.smithi114.stdout: "tid": 199 2024-04-23T10:32:03.929 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-23T10:32:03.929 INFO:teuthology.orchestra.run.smithi114.stdout: "op_type": "peer_request", 2024-04-23T10:32:03.929 INFO:teuthology.orchestra.run.smithi114.stdout: "leader_info": { 2024-04-23T10:32:03.929 INFO:teuthology.orchestra.run.smithi114.stdout: "leader": "0" 2024-04-23T10:32:03.929 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-23T10:32:03.929 INFO:teuthology.orchestra.run.smithi114.stdout: "events": [ 2024-04-23T10:32:03.929 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-04-23T10:32:03.929 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-04-23T10:31:21.484068+0000", 2024-04-23T10:32:03.929 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "initiated" 2024-04-23T10:32:03.930 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-23T10:32:03.930 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-04-23T10:32:03.930 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-04-23T10:31:21.484068+0000", 2024-04-23T10:32:03.930 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "header_read" 2024-04-23T10:32:03.930 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-23T10:32:03.930 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-04-23T10:32:03.930 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-04-23T10:31:21.484068+0000", 2024-04-23T10:32:03.930 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "throttled" 2024-04-23T10:32:03.930 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-23T10:32:03.930 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-04-23T10:32:03.930 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-04-23T10:31:21.484073+0000", 2024-04-23T10:32:03.930 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "all_read" 2024-04-23T10:32:03.930 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-23T10:32:03.930 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-04-23T10:32:03.930 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-04-23T10:31:21.488833+0000", 2024-04-23T10:32:03.931 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "dispatched" 2024-04-23T10:32:03.931 INFO:teuthology.orchestra.run.smithi114.stdout: } 2024-04-23T10:32:03.931 INFO:teuthology.orchestra.run.smithi114.stdout: ], 2024-04-23T10:32:03.931 INFO:teuthology.orchestra.run.smithi114.stdout: "locks": [] 2024-04-23T10:32:03.931 INFO:teuthology.orchestra.run.smithi114.stdout: } 2024-04-23T10:32:03.931 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-23T10:32:03.931 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-04-23T10:32:03.931 INFO:teuthology.orchestra.run.smithi114.stdout: "description": "internal op quiesce_inode:mds.2:115 fp=#0x10000000008", 2024-04-23T10:32:03.931 INFO:teuthology.orchestra.run.smithi114.stdout: "initiated_at": "2024-04-23T10:31:21.455152+0000", 2024-04-23T10:32:03.931 INFO:teuthology.orchestra.run.smithi114.stdout: "age": 42.373593233999998, 2024-04-23T10:32:03.931 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 42.374422787, 2024-04-23T10:32:03.931 INFO:teuthology.orchestra.run.smithi114.stdout: "continuous": true, 2024-04-23T10:32:03.931 INFO:teuthology.orchestra.run.smithi114.stdout: "type_data": { 2024-04-23T10:32:03.931 INFO:teuthology.orchestra.run.smithi114.stdout: "result": -2147483648, 2024-04-23T10:32:03.931 INFO:teuthology.orchestra.run.smithi114.stdout: "flag_point": "quiesce complete for non-auth inode", 2024-04-23T10:32:03.932 INFO:teuthology.orchestra.run.smithi114.stdout: "reqid": { 2024-04-23T10:32:03.932 INFO:teuthology.orchestra.run.smithi114.stdout: "entity": { 2024-04-23T10:32:03.932 INFO:teuthology.orchestra.run.smithi114.stdout: "type": "mds", 2024-04-23T10:32:03.932 INFO:teuthology.orchestra.run.smithi114.stdout: "num": 2 2024-04-23T10:32:03.932 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-23T10:32:03.932 INFO:teuthology.orchestra.run.smithi114.stdout: "tid": 115 2024-04-23T10:32:03.932 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-23T10:32:03.932 INFO:teuthology.orchestra.run.smithi114.stdout: "op_type": "internal_op", 2024-04-23T10:32:03.932 INFO:teuthology.orchestra.run.smithi114.stdout: "internal_op": 5385, 2024-04-23T10:32:03.932 INFO:teuthology.orchestra.run.smithi114.stdout: "op_name": "quiesce_inode", 2024-04-23T10:32:03.932 INFO:teuthology.orchestra.run.smithi114.stdout: "events": [ 2024-04-23T10:32:03.932 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-04-23T10:32:03.932 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-04-23T10:31:21.455152+0000", 2024-04-23T10:32:03.932 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "initiated" 2024-04-23T10:32:03.932 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-23T10:32:03.933 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-04-23T10:32:03.933 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-04-23T10:31:21.455152+0000", 2024-04-23T10:32:03.933 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "header_read" 2024-04-23T10:32:03.933 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-23T10:32:03.933 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-04-23T10:32:03.933 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-04-23T10:31:21.455152+0000", 2024-04-23T10:32:03.933 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "throttled" 2024-04-23T10:32:03.933 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-23T10:32:03.933 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-04-23T10:32:03.933 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-04-23T10:31:21.455152+0000", 2024-04-23T10:32:03.933 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "all_read" 2024-04-23T10:32:03.933 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-23T10:32:03.933 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-04-23T10:32:03.933 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-04-23T10:31:21.455152+0000", 2024-04-23T10:32:03.933 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "dispatched" 2024-04-23T10:32:03.934 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-23T10:32:03.934 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-04-23T10:32:03.934 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-04-23T10:31:21.455229+0000", 2024-04-23T10:32:03.934 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "requesting remote authpins" 2024-04-23T10:32:03.934 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-23T10:32:03.934 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-04-23T10:32:03.934 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-04-23T10:31:21.456936+0000", 2024-04-23T10:32:03.934 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "acquired locks" 2024-04-23T10:32:03.934 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-23T10:32:03.934 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-04-23T10:32:03.934 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-04-23T10:31:21.509277+0000", 2024-04-23T10:32:03.934 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "quiesce complete for non-auth inode" 2024-04-23T10:32:03.934 INFO:teuthology.orchestra.run.smithi114.stdout: } 2024-04-23T10:32:03.934 INFO:teuthology.orchestra.run.smithi114.stdout: ], 2024-04-23T10:32:03.934 INFO:teuthology.orchestra.run.smithi114.stdout: "locks": [ 2024-04-23T10:32:03.934 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-04-23T10:32:03.935 INFO:teuthology.orchestra.run.smithi114.stdout: "object": { 2024-04-23T10:32:03.935 INFO:teuthology.orchestra.run.smithi114.stdout: "is_auth": false, 2024-04-23T10:32:03.935 INFO:teuthology.orchestra.run.smithi114.stdout: "auth_state": { 2024-04-23T10:32:03.935 INFO:teuthology.orchestra.run.smithi114.stdout: "replicas": {} 2024-04-23T10:32:03.935 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-23T10:32:03.935 INFO:teuthology.orchestra.run.smithi114.stdout: "replica_state": { 2024-04-23T10:32:03.935 INFO:teuthology.orchestra.run.smithi114.stdout: "authority": [ 2024-04-23T10:32:03.935 INFO:teuthology.orchestra.run.smithi114.stdout: 0, 2024-04-23T10:32:03.935 INFO:teuthology.orchestra.run.smithi114.stdout: -2 2024-04-23T10:32:03.935 INFO:teuthology.orchestra.run.smithi114.stdout: ], 2024-04-23T10:32:03.935 INFO:teuthology.orchestra.run.smithi114.stdout: "replica_nonce": 1 2024-04-23T10:32:03.935 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-23T10:32:03.935 INFO:teuthology.orchestra.run.smithi114.stdout: "auth_pins": 0, 2024-04-23T10:32:03.935 INFO:teuthology.orchestra.run.smithi114.stdout: "is_frozen": false, 2024-04-23T10:32:03.935 INFO:teuthology.orchestra.run.smithi114.stdout: "is_freezing": false, 2024-04-23T10:32:03.935 INFO:teuthology.orchestra.run.smithi114.stdout: "pins": { 2024-04-23T10:32:03.936 INFO:teuthology.orchestra.run.smithi114.stdout: "request": 1, 2024-04-23T10:32:03.936 INFO:teuthology.orchestra.run.smithi114.stdout: "lock": 1, 2024-04-23T10:32:03.936 INFO:teuthology.orchestra.run.smithi114.stdout: "dirfrag": 1, 2024-04-23T10:32:03.936 INFO:teuthology.orchestra.run.smithi114.stdout: "dirwaiter": 0, 2024-04-23T10:32:03.936 INFO:teuthology.orchestra.run.smithi114.stdout: "discoverbase": 0 2024-04-23T10:32:03.936 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-23T10:32:03.936 INFO:teuthology.orchestra.run.smithi114.stdout: "nref": 3 2024-04-23T10:32:03.936 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-23T10:32:03.936 INFO:teuthology.orchestra.run.smithi114.stdout: "object_string": "[inode 0x10000000008 [...7,head] /volumes/qa/sv_1/b206794e-7622-4ab8-8943-f3ae4e65b7a3/client.0/tmp/clients/client0/ rep@0.1 v456 f(v0 m2024-04-23T10:27:51.950398+0000 6=5+1) n(v30 rc2024-04-23T10:31:17.764716+0000 b21075074 74=63+11)/n(v0 rc2024-04-23T10:27:51.920398+0000 2=0+2) (inest mix) (iquiesce lock x=1 by request(mds.2:115 nref=3)) | request=1 lock=1 dirfrag=1 dirwaiter=0 discoverbase=0 0x55c68b9d2c00]", 2024-04-23T10:32:03.936 INFO:teuthology.orchestra.run.smithi114.stdout: "lock": { 2024-04-23T10:32:03.936 INFO:teuthology.orchestra.run.smithi114.stdout: "gather_set": [], 2024-04-23T10:32:03.936 INFO:teuthology.orchestra.run.smithi114.stdout: "state": "lock", 2024-04-23T10:32:03.936 INFO:teuthology.orchestra.run.smithi114.stdout: "type": "iquiesce", 2024-04-23T10:32:03.936 INFO:teuthology.orchestra.run.smithi114.stdout: "is_leased": false, 2024-04-23T10:32:03.936 INFO:teuthology.orchestra.run.smithi114.stdout: "num_rdlocks": 0, 2024-04-23T10:32:03.937 INFO:teuthology.orchestra.run.smithi114.stdout: "num_wrlocks": 0, 2024-04-23T10:32:03.937 INFO:teuthology.orchestra.run.smithi114.stdout: "num_xlocks": 1, 2024-04-23T10:32:03.937 INFO:teuthology.orchestra.run.smithi114.stdout: "xlock_by": { 2024-04-23T10:32:03.937 INFO:teuthology.orchestra.run.smithi114.stdout: "reqid": { 2024-04-23T10:32:03.937 INFO:teuthology.orchestra.run.smithi114.stdout: "entity": { 2024-04-23T10:32:03.937 INFO:teuthology.orchestra.run.smithi114.stdout: "type": "mds", 2024-04-23T10:32:03.937 INFO:teuthology.orchestra.run.smithi114.stdout: "num": 2 2024-04-23T10:32:03.937 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-23T10:32:03.937 INFO:teuthology.orchestra.run.smithi114.stdout: "tid": 115 2024-04-23T10:32:03.937 INFO:teuthology.orchestra.run.smithi114.stdout: } 2024-04-23T10:32:03.937 INFO:teuthology.orchestra.run.smithi114.stdout: } 2024-04-23T10:32:03.937 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-23T10:32:03.937 INFO:teuthology.orchestra.run.smithi114.stdout: "flags": 4, 2024-04-23T10:32:03.937 INFO:teuthology.orchestra.run.smithi114.stdout: "wrlock_target": -1 2024-04-23T10:32:03.937 INFO:teuthology.orchestra.run.smithi114.stdout: } 2024-04-23T10:32:03.938 INFO:teuthology.orchestra.run.smithi114.stdout: ] 2024-04-23T10:32:03.938 INFO:teuthology.orchestra.run.smithi114.stdout: } 2024-04-23T10:32:03.938 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-23T10:32:03.938 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-04-23T10:32:03.938 INFO:teuthology.orchestra.run.smithi114.stdout: "description": "peer_request:mds.0:200", 2024-04-23T10:32:03.938 INFO:teuthology.orchestra.run.smithi114.stdout: "initiated_at": "2024-04-23T10:31:21.484125+0000", 2024-04-23T10:32:03.938 INFO:teuthology.orchestra.run.smithi114.stdout: "age": 42.344619600000001, 2024-04-23T10:32:03.938 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 42.345512378999999, 2024-04-23T10:32:03.938 INFO:teuthology.orchestra.run.smithi114.stdout: "continuous": false, 2024-04-23T10:32:03.938 INFO:teuthology.orchestra.run.smithi114.stdout: "type_data": { 2024-04-23T10:32:03.938 INFO:teuthology.orchestra.run.smithi114.stdout: "result": -2147483648, 2024-04-23T10:32:03.938 INFO:teuthology.orchestra.run.smithi114.stdout: "flag_point": "dispatched", 2024-04-23T10:32:03.938 INFO:teuthology.orchestra.run.smithi114.stdout: "reqid": { 2024-04-23T10:32:03.938 INFO:teuthology.orchestra.run.smithi114.stdout: "entity": { 2024-04-23T10:32:03.939 INFO:teuthology.orchestra.run.smithi114.stdout: "type": "mds", 2024-04-23T10:32:03.939 INFO:teuthology.orchestra.run.smithi114.stdout: "num": 0 2024-04-23T10:32:03.939 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-23T10:32:03.939 INFO:teuthology.orchestra.run.smithi114.stdout: "tid": 200 2024-04-23T10:32:03.939 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-23T10:32:03.939 INFO:teuthology.orchestra.run.smithi114.stdout: "op_type": "peer_request", 2024-04-23T10:32:03.939 INFO:teuthology.orchestra.run.smithi114.stdout: "leader_info": { 2024-04-23T10:32:03.939 INFO:teuthology.orchestra.run.smithi114.stdout: "leader": "0" 2024-04-23T10:32:03.939 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-23T10:32:03.939 INFO:teuthology.orchestra.run.smithi114.stdout: "events": [ 2024-04-23T10:32:03.939 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-04-23T10:32:03.939 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-04-23T10:31:21.484125+0000", 2024-04-23T10:32:03.939 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "initiated" 2024-04-23T10:32:03.939 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-23T10:32:03.939 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-04-23T10:32:03.940 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-04-23T10:31:21.484125+0000", 2024-04-23T10:32:03.940 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "header_read" 2024-04-23T10:32:03.940 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-23T10:32:03.940 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-04-23T10:32:03.940 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-04-23T10:31:21.484126+0000", 2024-04-23T10:32:03.940 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "throttled" 2024-04-23T10:32:03.940 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-23T10:32:03.940 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-04-23T10:32:03.940 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-04-23T10:31:21.484131+0000", 2024-04-23T10:32:03.940 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "all_read" 2024-04-23T10:32:03.940 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-23T10:32:03.940 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-04-23T10:32:03.940 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-04-23T10:31:21.488931+0000", 2024-04-23T10:32:03.940 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "dispatched" 2024-04-23T10:32:03.940 INFO:teuthology.orchestra.run.smithi114.stdout: } 2024-04-23T10:32:03.941 INFO:teuthology.orchestra.run.smithi114.stdout: ], 2024-04-23T10:32:03.941 INFO:teuthology.orchestra.run.smithi114.stdout: "locks": [] 2024-04-23T10:32:03.941 INFO:teuthology.orchestra.run.smithi114.stdout: } 2024-04-23T10:32:03.941 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-23T10:32:03.941 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-04-23T10:32:03.941 INFO:teuthology.orchestra.run.smithi114.stdout: "description": "internal op quiesce_inode:mds.2:116 fp=#0x10000000009", 2024-04-23T10:32:03.941 INFO:teuthology.orchestra.run.smithi114.stdout: "initiated_at": "2024-04-23T10:31:21.456950+0000", 2024-04-23T10:32:03.941 INFO:teuthology.orchestra.run.smithi114.stdout: "age": 42.371794465999997, 2024-04-23T10:32:03.941 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 42.372710093000002, 2024-04-23T10:32:03.941 INFO:teuthology.orchestra.run.smithi114.stdout: "continuous": true, 2024-04-23T10:32:03.941 INFO:teuthology.orchestra.run.smithi114.stdout: "type_data": { 2024-04-23T10:32:03.941 INFO:teuthology.orchestra.run.smithi114.stdout: "result": -2147483648, 2024-04-23T10:32:03.941 INFO:teuthology.orchestra.run.smithi114.stdout: "flag_point": "quiesce complete for non-auth inode", 2024-04-23T10:32:03.941 INFO:teuthology.orchestra.run.smithi114.stdout: "reqid": { 2024-04-23T10:32:03.942 INFO:teuthology.orchestra.run.smithi114.stdout: "entity": { 2024-04-23T10:32:03.942 INFO:teuthology.orchestra.run.smithi114.stdout: "type": "mds", 2024-04-23T10:32:03.942 INFO:teuthology.orchestra.run.smithi114.stdout: "num": 2 2024-04-23T10:32:03.942 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-23T10:32:03.942 INFO:teuthology.orchestra.run.smithi114.stdout: "tid": 116 2024-04-23T10:32:03.942 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-23T10:32:03.942 INFO:teuthology.orchestra.run.smithi114.stdout: "op_type": "internal_op", 2024-04-23T10:32:03.942 INFO:teuthology.orchestra.run.smithi114.stdout: "internal_op": 5385, 2024-04-23T10:32:03.942 INFO:teuthology.orchestra.run.smithi114.stdout: "op_name": "quiesce_inode", 2024-04-23T10:32:03.942 INFO:teuthology.orchestra.run.smithi114.stdout: "events": [ 2024-04-23T10:32:03.942 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-04-23T10:32:03.942 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-04-23T10:31:21.456950+0000", 2024-04-23T10:32:03.942 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "initiated" 2024-04-23T10:32:03.942 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-23T10:32:03.942 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-04-23T10:32:03.943 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-04-23T10:31:21.456950+0000", 2024-04-23T10:32:03.943 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "header_read" 2024-04-23T10:32:03.943 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-23T10:32:03.943 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-04-23T10:32:03.943 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-04-23T10:31:21.456950+0000", 2024-04-23T10:32:03.943 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "throttled" 2024-04-23T10:32:03.943 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-23T10:32:03.943 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-04-23T10:32:03.943 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-04-23T10:31:21.456950+0000", 2024-04-23T10:32:03.943 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "all_read" 2024-04-23T10:32:03.943 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-23T10:32:03.943 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-04-23T10:32:03.943 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-04-23T10:31:21.456950+0000", 2024-04-23T10:32:03.943 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "dispatched" 2024-04-23T10:32:03.943 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-23T10:32:03.944 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-04-23T10:32:03.944 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-04-23T10:31:21.457023+0000", 2024-04-23T10:32:03.944 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "requesting remote authpins" 2024-04-23T10:32:03.944 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-23T10:32:03.944 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-04-23T10:32:03.944 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-04-23T10:31:21.463073+0000", 2024-04-23T10:32:03.944 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "acquired locks" 2024-04-23T10:32:03.944 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-23T10:32:03.944 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-04-23T10:32:03.944 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-04-23T10:31:21.509262+0000", 2024-04-23T10:32:03.944 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "quiesce complete for non-auth inode" 2024-04-23T10:32:03.944 INFO:teuthology.orchestra.run.smithi114.stdout: } 2024-04-23T10:32:03.944 INFO:teuthology.orchestra.run.smithi114.stdout: ], 2024-04-23T10:32:03.944 INFO:teuthology.orchestra.run.smithi114.stdout: "locks": [ 2024-04-23T10:32:03.944 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-04-23T10:32:03.945 INFO:teuthology.orchestra.run.smithi114.stdout: "object": { 2024-04-23T10:32:03.945 INFO:teuthology.orchestra.run.smithi114.stdout: "is_auth": false, 2024-04-23T10:32:03.945 INFO:teuthology.orchestra.run.smithi114.stdout: "auth_state": { 2024-04-23T10:32:03.945 INFO:teuthology.orchestra.run.smithi114.stdout: "replicas": {} 2024-04-23T10:32:03.945 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-23T10:32:03.945 INFO:teuthology.orchestra.run.smithi114.stdout: "replica_state": { 2024-04-23T10:32:03.945 INFO:teuthology.orchestra.run.smithi114.stdout: "authority": [ 2024-04-23T10:32:03.945 INFO:teuthology.orchestra.run.smithi114.stdout: 0, 2024-04-23T10:32:03.945 INFO:teuthology.orchestra.run.smithi114.stdout: -2 2024-04-23T10:32:03.945 INFO:teuthology.orchestra.run.smithi114.stdout: ], 2024-04-23T10:32:03.945 INFO:teuthology.orchestra.run.smithi114.stdout: "replica_nonce": 1 2024-04-23T10:32:03.945 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-23T10:32:03.945 INFO:teuthology.orchestra.run.smithi114.stdout: "auth_pins": 0, 2024-04-23T10:32:03.945 INFO:teuthology.orchestra.run.smithi114.stdout: "is_frozen": false, 2024-04-23T10:32:03.945 INFO:teuthology.orchestra.run.smithi114.stdout: "is_freezing": false, 2024-04-23T10:32:03.946 INFO:teuthology.orchestra.run.smithi114.stdout: "pins": { 2024-04-23T10:32:03.946 INFO:teuthology.orchestra.run.smithi114.stdout: "request": 1, 2024-04-23T10:32:03.946 INFO:teuthology.orchestra.run.smithi114.stdout: "lock": 1, 2024-04-23T10:32:03.946 INFO:teuthology.orchestra.run.smithi114.stdout: "dirfrag": 1, 2024-04-23T10:32:03.946 INFO:teuthology.orchestra.run.smithi114.stdout: "discoverbase": 0 2024-04-23T10:32:03.946 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-23T10:32:03.946 INFO:teuthology.orchestra.run.smithi114.stdout: "nref": 3 2024-04-23T10:32:03.946 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-23T10:32:03.946 INFO:teuthology.orchestra.run.smithi114.stdout: "object_string": "[inode 0x10000000009 [...7,head] /volumes/qa/sv_1/b206794e-7622-4ab8-8943-f3ae4e65b7a3/client.0/tmp/clients/client0/~dmtmp/ rep@0.1 v15176 f(v0 m2024-04-23T10:27:52.186392+0000 9=0+9) n(v31 rc2024-04-23T10:31:17.764716+0000 b15832194 68=58+10)/n(v0 rc2024-04-23T10:27:51.953398+0000 2=0+2) (inest mix) (iquiesce lock x=1 by request(mds.2:116 nref=3)) | request=1 lock=1 dirfrag=1 discoverbase=0 0x55c68b9d3180]", 2024-04-23T10:32:03.946 INFO:teuthology.orchestra.run.smithi114.stdout: "lock": { 2024-04-23T10:32:03.946 INFO:teuthology.orchestra.run.smithi114.stdout: "gather_set": [], 2024-04-23T10:32:03.946 INFO:teuthology.orchestra.run.smithi114.stdout: "state": "lock", 2024-04-23T10:32:03.946 INFO:teuthology.orchestra.run.smithi114.stdout: "type": "iquiesce", 2024-04-23T10:32:03.946 INFO:teuthology.orchestra.run.smithi114.stdout: "is_leased": false, 2024-04-23T10:32:03.946 INFO:teuthology.orchestra.run.smithi114.stdout: "num_rdlocks": 0, 2024-04-23T10:32:03.947 INFO:teuthology.orchestra.run.smithi114.stdout: "num_wrlocks": 0, 2024-04-23T10:32:03.947 INFO:teuthology.orchestra.run.smithi114.stdout: "num_xlocks": 1, 2024-04-23T10:32:03.947 INFO:teuthology.orchestra.run.smithi114.stdout: "xlock_by": { 2024-04-23T10:32:03.947 INFO:teuthology.orchestra.run.smithi114.stdout: "reqid": { 2024-04-23T10:32:03.947 INFO:teuthology.orchestra.run.smithi114.stdout: "entity": { 2024-04-23T10:32:03.947 INFO:teuthology.orchestra.run.smithi114.stdout: "type": "mds", 2024-04-23T10:32:03.947 INFO:teuthology.orchestra.run.smithi114.stdout: "num": 2 2024-04-23T10:32:03.947 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-23T10:32:03.947 INFO:teuthology.orchestra.run.smithi114.stdout: "tid": 116 2024-04-23T10:32:03.947 INFO:teuthology.orchestra.run.smithi114.stdout: } 2024-04-23T10:32:03.947 INFO:teuthology.orchestra.run.smithi114.stdout: } 2024-04-23T10:32:03.947 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-23T10:32:03.947 INFO:teuthology.orchestra.run.smithi114.stdout: "flags": 4, 2024-04-23T10:32:03.947 INFO:teuthology.orchestra.run.smithi114.stdout: "wrlock_target": -1 2024-04-23T10:32:03.948 INFO:teuthology.orchestra.run.smithi114.stdout: } 2024-04-23T10:32:03.948 INFO:teuthology.orchestra.run.smithi114.stdout: ] 2024-04-23T10:32:03.948 INFO:teuthology.orchestra.run.smithi114.stdout: } 2024-04-23T10:32:03.948 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-23T10:32:03.948 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-04-23T10:32:03.948 INFO:teuthology.orchestra.run.smithi114.stdout: "description": "internal op quiesce_inode:mds.2:137 fp=#0x10000000241", 2024-04-23T10:32:03.948 INFO:teuthology.orchestra.run.smithi114.stdout: "initiated_at": "2024-04-23T10:31:21.489235+0000", 2024-04-23T10:32:03.948 INFO:teuthology.orchestra.run.smithi114.stdout: "age": 42.339509425000003, 2024-04-23T10:32:03.948 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 42.340487074999999, 2024-04-23T10:32:03.948 INFO:teuthology.orchestra.run.smithi114.stdout: "continuous": true, 2024-04-23T10:32:03.948 INFO:teuthology.orchestra.run.smithi114.stdout: "type_data": { 2024-04-23T10:32:03.948 INFO:teuthology.orchestra.run.smithi114.stdout: "result": -2147483648, 2024-04-23T10:32:03.948 INFO:teuthology.orchestra.run.smithi114.stdout: "flag_point": "quiesce complete for non-auth inode", 2024-04-23T10:32:03.948 INFO:teuthology.orchestra.run.smithi114.stdout: "reqid": { 2024-04-23T10:32:03.948 INFO:teuthology.orchestra.run.smithi114.stdout: "entity": { 2024-04-23T10:32:03.949 INFO:teuthology.orchestra.run.smithi114.stdout: "type": "mds", 2024-04-23T10:32:03.949 INFO:teuthology.orchestra.run.smithi114.stdout: "num": 2 2024-04-23T10:32:03.949 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-23T10:32:03.949 INFO:teuthology.orchestra.run.smithi114.stdout: "tid": 137 2024-04-23T10:32:03.949 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-23T10:32:03.949 INFO:teuthology.orchestra.run.smithi114.stdout: "op_type": "internal_op", 2024-04-23T10:32:03.949 INFO:teuthology.orchestra.run.smithi114.stdout: "internal_op": 5385, 2024-04-23T10:32:03.949 INFO:teuthology.orchestra.run.smithi114.stdout: "op_name": "quiesce_inode", 2024-04-23T10:32:03.949 INFO:teuthology.orchestra.run.smithi114.stdout: "events": [ 2024-04-23T10:32:03.949 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-04-23T10:32:03.949 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-04-23T10:31:21.489235+0000", 2024-04-23T10:32:03.949 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "initiated" 2024-04-23T10:32:03.949 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-23T10:32:03.949 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-04-23T10:32:03.949 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-04-23T10:31:21.489235+0000", 2024-04-23T10:32:03.949 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "header_read" 2024-04-23T10:32:03.950 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-23T10:32:03.950 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-04-23T10:32:03.950 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-04-23T10:31:21.489235+0000", 2024-04-23T10:32:03.950 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "throttled" 2024-04-23T10:32:03.950 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-23T10:32:03.950 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-04-23T10:32:03.950 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-04-23T10:31:21.489235+0000", 2024-04-23T10:32:03.950 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "all_read" 2024-04-23T10:32:03.950 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-23T10:32:03.950 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-04-23T10:32:03.950 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-04-23T10:31:21.489235+0000", 2024-04-23T10:32:03.950 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "dispatched" 2024-04-23T10:32:03.950 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-23T10:32:03.950 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-04-23T10:32:03.951 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-04-23T10:31:21.489345+0000", 2024-04-23T10:32:03.951 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "requesting remote authpins" 2024-04-23T10:32:03.951 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-23T10:32:03.951 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-04-23T10:32:03.951 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-04-23T10:31:21.508959+0000", 2024-04-23T10:32:03.951 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "acquired locks" 2024-04-23T10:32:03.951 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-23T10:32:03.951 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-04-23T10:32:03.951 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-04-23T10:31:21.508974+0000", 2024-04-23T10:32:03.951 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "quiesce complete for non-auth inode" 2024-04-23T10:32:03.951 INFO:teuthology.orchestra.run.smithi114.stdout: } 2024-04-23T10:32:03.951 INFO:teuthology.orchestra.run.smithi114.stdout: ], 2024-04-23T10:32:03.951 INFO:teuthology.orchestra.run.smithi114.stdout: "locks": [ 2024-04-23T10:32:03.951 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-04-23T10:32:03.952 INFO:teuthology.orchestra.run.smithi114.stdout: "object": { 2024-04-23T10:32:03.952 INFO:teuthology.orchestra.run.smithi114.stdout: "is_auth": false, 2024-04-23T10:32:03.952 INFO:teuthology.orchestra.run.smithi114.stdout: "auth_state": { 2024-04-23T10:32:03.952 INFO:teuthology.orchestra.run.smithi114.stdout: "replicas": {} 2024-04-23T10:32:03.952 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-23T10:32:03.952 INFO:teuthology.orchestra.run.smithi114.stdout: "replica_state": { 2024-04-23T10:32:03.952 INFO:teuthology.orchestra.run.smithi114.stdout: "authority": [ 2024-04-23T10:32:03.952 INFO:teuthology.orchestra.run.smithi114.stdout: 0, 2024-04-23T10:32:03.952 INFO:teuthology.orchestra.run.smithi114.stdout: -2 2024-04-23T10:32:03.952 INFO:teuthology.orchestra.run.smithi114.stdout: ], 2024-04-23T10:32:03.952 INFO:teuthology.orchestra.run.smithi114.stdout: "replica_nonce": 1 2024-04-23T10:32:03.952 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-23T10:32:03.952 INFO:teuthology.orchestra.run.smithi114.stdout: "auth_pins": 0, 2024-04-23T10:32:03.952 INFO:teuthology.orchestra.run.smithi114.stdout: "is_frozen": false, 2024-04-23T10:32:03.953 INFO:teuthology.orchestra.run.smithi114.stdout: "is_freezing": false, 2024-04-23T10:32:03.953 INFO:teuthology.orchestra.run.smithi114.stdout: "pins": { 2024-04-23T10:32:03.953 INFO:teuthology.orchestra.run.smithi114.stdout: "ptrwaiter": 0, 2024-04-23T10:32:03.953 INFO:teuthology.orchestra.run.smithi114.stdout: "request": 1, 2024-04-23T10:32:03.953 INFO:teuthology.orchestra.run.smithi114.stdout: "lock": 1, 2024-04-23T10:32:03.953 INFO:teuthology.orchestra.run.smithi114.stdout: "importingcaps": 0, 2024-04-23T10:32:03.953 INFO:teuthology.orchestra.run.smithi114.stdout: "caps": 0, 2024-04-23T10:32:03.953 INFO:teuthology.orchestra.run.smithi114.stdout: "exportingcaps": 0, 2024-04-23T10:32:03.953 INFO:teuthology.orchestra.run.smithi114.stdout: "replicated": 0, 2024-04-23T10:32:03.953 INFO:teuthology.orchestra.run.smithi114.stdout: "dirty": 0, 2024-04-23T10:32:03.953 INFO:teuthology.orchestra.run.smithi114.stdout: "authpin": 0, 2024-04-23T10:32:03.953 INFO:teuthology.orchestra.run.smithi114.stdout: "tempexporting": 0 2024-04-23T10:32:03.953 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-23T10:32:03.954 INFO:teuthology.orchestra.run.smithi114.stdout: "nref": 2 2024-04-23T10:32:03.954 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-23T10:32:03.954 INFO:teuthology.orchestra.run.smithi114.stdout: "object_string": "[inode 0x10000000241 [7,head] /volumes/qa/sv_1/b206794e-7622-4ab8-8943-f3ae4e65b7a3/client.0/tmp/clients/client0/~dmtmp/SEED/SMALL.FIL rep@0.1 v646 s=327680 n(v0 rc2024-04-23T10:29:16.539476+0000 b327680 1=1+0)/n(v0 rc2024-04-23T10:28:22.485704+0000 b327680 1=1+0) (iquiesce lock x=1 by request(mds.2:137 nref=3)) | ptrwaiter=0 request=1 lock=1 importingcaps=0 caps=0 exportingcaps=0 replicated=0 dirty=0 authpin=0 tempexporting=0 0x55c68bc80c00]", 2024-04-23T10:32:03.954 INFO:teuthology.orchestra.run.smithi114.stdout: "lock": { 2024-04-23T10:32:03.954 INFO:teuthology.orchestra.run.smithi114.stdout: "gather_set": [], 2024-04-23T10:32:03.954 INFO:teuthology.orchestra.run.smithi114.stdout: "state": "lock", 2024-04-23T10:32:03.954 INFO:teuthology.orchestra.run.smithi114.stdout: "type": "iquiesce", 2024-04-23T10:32:03.954 INFO:teuthology.orchestra.run.smithi114.stdout: "is_leased": false, 2024-04-23T10:32:03.954 INFO:teuthology.orchestra.run.smithi114.stdout: "num_rdlocks": 0, 2024-04-23T10:32:03.954 INFO:teuthology.orchestra.run.smithi114.stdout: "num_wrlocks": 0, 2024-04-23T10:32:03.954 INFO:teuthology.orchestra.run.smithi114.stdout: "num_xlocks": 1, 2024-04-23T10:32:03.954 INFO:teuthology.orchestra.run.smithi114.stdout: "xlock_by": { 2024-04-23T10:32:03.954 INFO:teuthology.orchestra.run.smithi114.stdout: "reqid": { 2024-04-23T10:32:03.955 INFO:teuthology.orchestra.run.smithi114.stdout: "entity": { 2024-04-23T10:32:03.955 INFO:teuthology.orchestra.run.smithi114.stdout: "type": "mds", 2024-04-23T10:32:03.955 INFO:teuthology.orchestra.run.smithi114.stdout: "num": 2 2024-04-23T10:32:03.955 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-23T10:32:03.955 INFO:teuthology.orchestra.run.smithi114.stdout: "tid": 137 2024-04-23T10:32:03.955 INFO:teuthology.orchestra.run.smithi114.stdout: } 2024-04-23T10:32:03.955 INFO:teuthology.orchestra.run.smithi114.stdout: } 2024-04-23T10:32:03.955 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-23T10:32:03.955 INFO:teuthology.orchestra.run.smithi114.stdout: "flags": 4, 2024-04-23T10:32:03.955 INFO:teuthology.orchestra.run.smithi114.stdout: "wrlock_target": -1 2024-04-23T10:32:03.955 INFO:teuthology.orchestra.run.smithi114.stdout: } 2024-04-23T10:32:03.955 INFO:teuthology.orchestra.run.smithi114.stdout: ] 2024-04-23T10:32:03.955 INFO:teuthology.orchestra.run.smithi114.stdout: } 2024-04-23T10:32:03.955 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-23T10:32:03.956 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-04-23T10:32:03.956 INFO:teuthology.orchestra.run.smithi114.stdout: "description": "internal op quiesce_inode:mds.2:117 fp=#0x10000000215", 2024-04-23T10:32:03.956 INFO:teuthology.orchestra.run.smithi114.stdout: "initiated_at": "2024-04-23T10:31:21.463094+0000", 2024-04-23T10:32:03.956 INFO:teuthology.orchestra.run.smithi114.stdout: "age": 42.365650399000003, 2024-04-23T10:32:03.956 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 42.366730732999997, 2024-04-23T10:32:03.956 INFO:teuthology.orchestra.run.smithi114.stdout: "continuous": true, 2024-04-23T10:32:03.956 INFO:teuthology.orchestra.run.smithi114.stdout: "type_data": { 2024-04-23T10:32:03.956 INFO:teuthology.orchestra.run.smithi114.stdout: "result": -2147483648, 2024-04-23T10:32:03.956 INFO:teuthology.orchestra.run.smithi114.stdout: "flag_point": "quiesce complete for non-auth inode", 2024-04-23T10:32:03.956 INFO:teuthology.orchestra.run.smithi114.stdout: "reqid": { 2024-04-23T10:32:03.956 INFO:teuthology.orchestra.run.smithi114.stdout: "entity": { 2024-04-23T10:32:03.956 INFO:teuthology.orchestra.run.smithi114.stdout: "type": "mds", 2024-04-23T10:32:03.956 INFO:teuthology.orchestra.run.smithi114.stdout: "num": 2 2024-04-23T10:32:03.956 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-23T10:32:03.956 INFO:teuthology.orchestra.run.smithi114.stdout: "tid": 117 2024-04-23T10:32:03.956 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-23T10:32:03.957 INFO:teuthology.orchestra.run.smithi114.stdout: "op_type": "internal_op", 2024-04-23T10:32:03.957 INFO:teuthology.orchestra.run.smithi114.stdout: "internal_op": 5385, 2024-04-23T10:32:03.957 INFO:teuthology.orchestra.run.smithi114.stdout: "op_name": "quiesce_inode", 2024-04-23T10:32:03.957 INFO:teuthology.orchestra.run.smithi114.stdout: "events": [ 2024-04-23T10:32:03.957 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-04-23T10:32:03.957 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-04-23T10:31:21.463094+0000", 2024-04-23T10:32:03.957 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "initiated" 2024-04-23T10:32:03.957 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-23T10:32:03.957 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-04-23T10:32:03.957 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-04-23T10:31:21.463094+0000", 2024-04-23T10:32:03.957 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "header_read" 2024-04-23T10:32:03.957 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-23T10:32:03.957 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-04-23T10:32:03.957 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-04-23T10:31:21.463094+0000", 2024-04-23T10:32:03.957 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "throttled" 2024-04-23T10:32:03.958 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-23T10:32:03.958 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-04-23T10:32:03.958 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-04-23T10:31:21.463094+0000", 2024-04-23T10:32:03.958 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "all_read" 2024-04-23T10:32:03.958 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-23T10:32:03.958 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-04-23T10:32:03.958 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-04-23T10:31:21.463094+0000", 2024-04-23T10:32:03.958 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "dispatched" 2024-04-23T10:32:03.958 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-23T10:32:03.958 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-04-23T10:32:03.958 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-04-23T10:31:21.463297+0000", 2024-04-23T10:32:03.958 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "requesting remote authpins" 2024-04-23T10:32:03.958 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-23T10:32:03.958 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-04-23T10:32:03.958 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-04-23T10:31:21.477857+0000", 2024-04-23T10:32:03.959 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "acquired locks" 2024-04-23T10:32:03.959 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-23T10:32:03.959 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-04-23T10:32:03.959 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-04-23T10:31:21.506084+0000", 2024-04-23T10:32:03.959 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "quiesce complete for non-auth inode" 2024-04-23T10:32:03.959 INFO:teuthology.orchestra.run.smithi114.stdout: } 2024-04-23T10:32:03.959 INFO:teuthology.orchestra.run.smithi114.stdout: ], 2024-04-23T10:32:03.959 INFO:teuthology.orchestra.run.smithi114.stdout: "locks": [ 2024-04-23T10:32:03.959 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-04-23T10:32:03.959 INFO:teuthology.orchestra.run.smithi114.stdout: "object": { 2024-04-23T10:32:03.959 INFO:teuthology.orchestra.run.smithi114.stdout: "is_auth": false, 2024-04-23T10:32:03.959 INFO:teuthology.orchestra.run.smithi114.stdout: "auth_state": { 2024-04-23T10:32:03.959 INFO:teuthology.orchestra.run.smithi114.stdout: "replicas": {} 2024-04-23T10:32:03.959 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-23T10:32:03.959 INFO:teuthology.orchestra.run.smithi114.stdout: "replica_state": { 2024-04-23T10:32:03.959 INFO:teuthology.orchestra.run.smithi114.stdout: "authority": [ 2024-04-23T10:32:03.960 INFO:teuthology.orchestra.run.smithi114.stdout: 0, 2024-04-23T10:32:03.960 INFO:teuthology.orchestra.run.smithi114.stdout: -2 2024-04-23T10:32:03.960 INFO:teuthology.orchestra.run.smithi114.stdout: ], 2024-04-23T10:32:03.960 INFO:teuthology.orchestra.run.smithi114.stdout: "replica_nonce": 1 2024-04-23T10:32:03.960 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-23T10:32:03.960 INFO:teuthology.orchestra.run.smithi114.stdout: "auth_pins": 0, 2024-04-23T10:32:03.960 INFO:teuthology.orchestra.run.smithi114.stdout: "is_frozen": false, 2024-04-23T10:32:03.960 INFO:teuthology.orchestra.run.smithi114.stdout: "is_freezing": false, 2024-04-23T10:32:03.960 INFO:teuthology.orchestra.run.smithi114.stdout: "pins": { 2024-04-23T10:32:03.960 INFO:teuthology.orchestra.run.smithi114.stdout: "dirtyscattered": 0, 2024-04-23T10:32:03.960 INFO:teuthology.orchestra.run.smithi114.stdout: "request": 1, 2024-04-23T10:32:03.960 INFO:teuthology.orchestra.run.smithi114.stdout: "lock": 1, 2024-04-23T10:32:03.960 INFO:teuthology.orchestra.run.smithi114.stdout: "importing": 0, 2024-04-23T10:32:03.960 INFO:teuthology.orchestra.run.smithi114.stdout: "dirfrag": 1, 2024-04-23T10:32:03.960 INFO:teuthology.orchestra.run.smithi114.stdout: "caps": 1, 2024-04-23T10:32:03.961 INFO:teuthology.orchestra.run.smithi114.stdout: "dirwaiter": 0, 2024-04-23T10:32:03.961 INFO:teuthology.orchestra.run.smithi114.stdout: "waiter": 0, 2024-04-23T10:32:03.961 INFO:teuthology.orchestra.run.smithi114.stdout: "discoverbase": 0 2024-04-23T10:32:03.961 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-23T10:32:03.961 INFO:teuthology.orchestra.run.smithi114.stdout: "nref": 4 2024-04-23T10:32:03.961 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-23T10:32:03.961 INFO:teuthology.orchestra.run.smithi114.stdout: "object_string": "[inode 0x10000000215 [...7,head] /volumes/qa/sv_1/b206794e-7622-4ab8-8943-f3ae4e65b7a3/client.0/tmp/clients/client0/~dmtmp/COREL/ rep@0.1 v45589 f(v1232 m2024-04-23T10:31:17.745716+0000 11=11+0) n(v19 rc2024-04-23T10:31:17.745716+0000 b2861056 12=11+1)/n(v0 rc2024-04-23T10:27:52.004397+0000 2=1+1) (inest mix) (iquiesce lock x=1 by request(mds.2:117 nref=3)) caps={15312=pAsLsXsFs/pAsLsXsFsxcral@5564} | dirtyscattered=0 request=1 lock=1 importing=0 dirfrag=1 caps=1 dirwaiter=0 waiter=0 discoverbase=0 0x55c68bbfa580]", 2024-04-23T10:32:03.961 INFO:teuthology.orchestra.run.smithi114.stdout: "lock": { 2024-04-23T10:32:03.961 INFO:teuthology.orchestra.run.smithi114.stdout: "gather_set": [], 2024-04-23T10:32:03.961 INFO:teuthology.orchestra.run.smithi114.stdout: "state": "lock", 2024-04-23T10:32:03.961 INFO:teuthology.orchestra.run.smithi114.stdout: "type": "iquiesce", 2024-04-23T10:32:03.961 INFO:teuthology.orchestra.run.smithi114.stdout: "is_leased": false, 2024-04-23T10:32:03.961 INFO:teuthology.orchestra.run.smithi114.stdout: "num_rdlocks": 0, 2024-04-23T10:32:03.961 INFO:teuthology.orchestra.run.smithi114.stdout: "num_wrlocks": 0, 2024-04-23T10:32:03.961 INFO:teuthology.orchestra.run.smithi114.stdout: "num_xlocks": 1, 2024-04-23T10:32:03.961 INFO:teuthology.orchestra.run.smithi114.stdout: "xlock_by": { 2024-04-23T10:32:03.962 INFO:teuthology.orchestra.run.smithi114.stdout: "reqid": { 2024-04-23T10:32:03.962 INFO:teuthology.orchestra.run.smithi114.stdout: "entity": { 2024-04-23T10:32:03.962 INFO:teuthology.orchestra.run.smithi114.stdout: "type": "mds", 2024-04-23T10:32:03.962 INFO:teuthology.orchestra.run.smithi114.stdout: "num": 2 2024-04-23T10:32:03.962 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-23T10:32:03.962 INFO:teuthology.orchestra.run.smithi114.stdout: "tid": 117 2024-04-23T10:32:03.962 INFO:teuthology.orchestra.run.smithi114.stdout: } 2024-04-23T10:32:03.962 INFO:teuthology.orchestra.run.smithi114.stdout: } 2024-04-23T10:32:03.962 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-23T10:32:03.962 INFO:teuthology.orchestra.run.smithi114.stdout: "flags": 4, 2024-04-23T10:32:03.962 INFO:teuthology.orchestra.run.smithi114.stdout: "wrlock_target": -1 2024-04-23T10:32:03.962 INFO:teuthology.orchestra.run.smithi114.stdout: } 2024-04-23T10:32:03.962 INFO:teuthology.orchestra.run.smithi114.stdout: ] 2024-04-23T10:32:03.962 INFO:teuthology.orchestra.run.smithi114.stdout: } 2024-04-23T10:32:03.963 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-23T10:32:03.963 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-04-23T10:32:03.963 INFO:teuthology.orchestra.run.smithi114.stdout: "description": "internal op quiesce_inode:mds.2:138 fp=#0x10000000242", 2024-04-23T10:32:03.963 INFO:teuthology.orchestra.run.smithi114.stdout: "initiated_at": "2024-04-23T10:31:21.489251+0000", 2024-04-23T10:32:03.963 INFO:teuthology.orchestra.run.smithi114.stdout: "age": 42.339494180999999, 2024-04-23T10:32:03.963 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 42.340645232999996, 2024-04-23T10:32:03.963 INFO:teuthology.orchestra.run.smithi114.stdout: "continuous": true, 2024-04-23T10:32:03.963 INFO:teuthology.orchestra.run.smithi114.stdout: "type_data": { 2024-04-23T10:32:03.963 INFO:teuthology.orchestra.run.smithi114.stdout: "result": -2147483648, 2024-04-23T10:32:03.963 INFO:teuthology.orchestra.run.smithi114.stdout: "flag_point": "quiesce complete for non-auth inode", 2024-04-23T10:32:03.963 INFO:teuthology.orchestra.run.smithi114.stdout: "reqid": { 2024-04-23T10:32:03.963 INFO:teuthology.orchestra.run.smithi114.stdout: "entity": { 2024-04-23T10:32:03.963 INFO:teuthology.orchestra.run.smithi114.stdout: "type": "mds", 2024-04-23T10:32:03.963 INFO:teuthology.orchestra.run.smithi114.stdout: "num": 2 2024-04-23T10:32:03.963 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-23T10:32:03.963 INFO:teuthology.orchestra.run.smithi114.stdout: "tid": 138 2024-04-23T10:32:03.964 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-23T10:32:03.964 INFO:teuthology.orchestra.run.smithi114.stdout: "op_type": "internal_op", 2024-04-23T10:32:03.964 INFO:teuthology.orchestra.run.smithi114.stdout: "internal_op": 5385, 2024-04-23T10:32:03.964 INFO:teuthology.orchestra.run.smithi114.stdout: "op_name": "quiesce_inode", 2024-04-23T10:32:03.964 INFO:teuthology.orchestra.run.smithi114.stdout: "events": [ 2024-04-23T10:32:03.964 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-04-23T10:32:03.964 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-04-23T10:31:21.489251+0000", 2024-04-23T10:32:03.964 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "initiated" 2024-04-23T10:32:03.964 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-23T10:32:03.964 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-04-23T10:32:03.964 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-04-23T10:31:21.489251+0000", 2024-04-23T10:32:03.964 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "header_read" 2024-04-23T10:32:03.964 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-23T10:32:03.964 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-04-23T10:32:03.965 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-04-23T10:31:21.489251+0000", 2024-04-23T10:32:03.965 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "throttled" 2024-04-23T10:32:03.965 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-23T10:32:03.965 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-04-23T10:32:03.965 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-04-23T10:31:21.489251+0000", 2024-04-23T10:32:03.965 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "all_read" 2024-04-23T10:32:03.965 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-23T10:32:03.965 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-04-23T10:32:03.965 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-04-23T10:31:21.489251+0000", 2024-04-23T10:32:03.965 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "dispatched" 2024-04-23T10:32:03.965 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-23T10:32:03.965 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-04-23T10:32:03.965 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-04-23T10:31:21.489406+0000", 2024-04-23T10:32:03.965 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "requesting remote authpins" 2024-04-23T10:32:03.965 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-23T10:32:03.966 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-04-23T10:32:03.966 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-04-23T10:31:21.509077+0000", 2024-04-23T10:32:03.966 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "acquired locks" 2024-04-23T10:32:03.966 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-23T10:32:03.966 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-04-23T10:32:03.966 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-04-23T10:31:21.509083+0000", 2024-04-23T10:32:03.966 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "quiesce complete for non-auth inode" 2024-04-23T10:32:03.966 INFO:teuthology.orchestra.run.smithi114.stdout: } 2024-04-23T10:32:03.966 INFO:teuthology.orchestra.run.smithi114.stdout: ], 2024-04-23T10:32:03.966 INFO:teuthology.orchestra.run.smithi114.stdout: "locks": [ 2024-04-23T10:32:03.966 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-04-23T10:32:03.966 INFO:teuthology.orchestra.run.smithi114.stdout: "object": { 2024-04-23T10:32:03.966 INFO:teuthology.orchestra.run.smithi114.stdout: "is_auth": false, 2024-04-23T10:32:03.966 INFO:teuthology.orchestra.run.smithi114.stdout: "auth_state": { 2024-04-23T10:32:03.966 INFO:teuthology.orchestra.run.smithi114.stdout: "replicas": {} 2024-04-23T10:32:03.966 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-23T10:32:03.967 INFO:teuthology.orchestra.run.smithi114.stdout: "replica_state": { 2024-04-23T10:32:03.967 INFO:teuthology.orchestra.run.smithi114.stdout: "authority": [ 2024-04-23T10:32:03.967 INFO:teuthology.orchestra.run.smithi114.stdout: 0, 2024-04-23T10:32:03.967 INFO:teuthology.orchestra.run.smithi114.stdout: -2 2024-04-23T10:32:03.967 INFO:teuthology.orchestra.run.smithi114.stdout: ], 2024-04-23T10:32:03.967 INFO:teuthology.orchestra.run.smithi114.stdout: "replica_nonce": 1 2024-04-23T10:32:03.967 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-23T10:32:03.967 INFO:teuthology.orchestra.run.smithi114.stdout: "auth_pins": 0, 2024-04-23T10:32:03.967 INFO:teuthology.orchestra.run.smithi114.stdout: "is_frozen": false, 2024-04-23T10:32:03.967 INFO:teuthology.orchestra.run.smithi114.stdout: "is_freezing": false, 2024-04-23T10:32:03.967 INFO:teuthology.orchestra.run.smithi114.stdout: "pins": { 2024-04-23T10:32:03.967 INFO:teuthology.orchestra.run.smithi114.stdout: "ptrwaiter": 0, 2024-04-23T10:32:03.967 INFO:teuthology.orchestra.run.smithi114.stdout: "request": 1, 2024-04-23T10:32:03.967 INFO:teuthology.orchestra.run.smithi114.stdout: "lock": 1, 2024-04-23T10:32:03.967 INFO:teuthology.orchestra.run.smithi114.stdout: "importingcaps": 0, 2024-04-23T10:32:03.968 INFO:teuthology.orchestra.run.smithi114.stdout: "caps": 0, 2024-04-23T10:32:03.968 INFO:teuthology.orchestra.run.smithi114.stdout: "exportingcaps": 0, 2024-04-23T10:32:03.968 INFO:teuthology.orchestra.run.smithi114.stdout: "replicated": 0, 2024-04-23T10:32:03.968 INFO:teuthology.orchestra.run.smithi114.stdout: "dirty": 0, 2024-04-23T10:32:03.968 INFO:teuthology.orchestra.run.smithi114.stdout: "authpin": 0, 2024-04-23T10:32:03.968 INFO:teuthology.orchestra.run.smithi114.stdout: "tempexporting": 0 2024-04-23T10:32:03.968 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-23T10:32:03.968 INFO:teuthology.orchestra.run.smithi114.stdout: "nref": 2 2024-04-23T10:32:03.968 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-23T10:32:03.968 INFO:teuthology.orchestra.run.smithi114.stdout: "object_string": "[inode 0x10000000242 [7,head] /volumes/qa/sv_1/b206794e-7622-4ab8-8943-f3ae4e65b7a3/client.0/tmp/clients/client0/~dmtmp/SEED/MEDIUM.FIL rep@0.1 v650 s=1048576 n(v0 rc2024-04-23T10:29:16.539476+0000 b1048576 1=1+0)/n(v0 rc2024-04-23T10:28:22.485704+0000 b1048576 1=1+0) (iquiesce lock x=1 by request(mds.2:138 nref=3)) | ptrwaiter=0 request=1 lock=1 importingcaps=0 caps=0 exportingcaps=0 replicated=0 dirty=0 authpin=0 tempexporting=0 0x55c68bc81180]", 2024-04-23T10:32:03.968 INFO:teuthology.orchestra.run.smithi114.stdout: "lock": { 2024-04-23T10:32:03.968 INFO:teuthology.orchestra.run.smithi114.stdout: "gather_set": [], 2024-04-23T10:32:03.968 INFO:teuthology.orchestra.run.smithi114.stdout: "state": "lock", 2024-04-23T10:32:03.968 INFO:teuthology.orchestra.run.smithi114.stdout: "type": "iquiesce", 2024-04-23T10:32:03.968 INFO:teuthology.orchestra.run.smithi114.stdout: "is_leased": false, 2024-04-23T10:32:03.969 INFO:teuthology.orchestra.run.smithi114.stdout: "num_rdlocks": 0, 2024-04-23T10:32:03.969 INFO:teuthology.orchestra.run.smithi114.stdout: "num_wrlocks": 0, 2024-04-23T10:32:03.969 INFO:teuthology.orchestra.run.smithi114.stdout: "num_xlocks": 1, 2024-04-23T10:32:03.969 INFO:teuthology.orchestra.run.smithi114.stdout: "xlock_by": { 2024-04-23T10:32:03.969 INFO:teuthology.orchestra.run.smithi114.stdout: "reqid": { 2024-04-23T10:32:03.969 INFO:teuthology.orchestra.run.smithi114.stdout: "entity": { 2024-04-23T10:32:03.969 INFO:teuthology.orchestra.run.smithi114.stdout: "type": "mds", 2024-04-23T10:32:03.969 INFO:teuthology.orchestra.run.smithi114.stdout: "num": 2 2024-04-23T10:32:03.969 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-23T10:32:03.969 INFO:teuthology.orchestra.run.smithi114.stdout: "tid": 138 2024-04-23T10:32:03.969 INFO:teuthology.orchestra.run.smithi114.stdout: } 2024-04-23T10:32:03.969 INFO:teuthology.orchestra.run.smithi114.stdout: } 2024-04-23T10:32:03.969 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-23T10:32:03.969 INFO:teuthology.orchestra.run.smithi114.stdout: "flags": 4, 2024-04-23T10:32:03.969 INFO:teuthology.orchestra.run.smithi114.stdout: "wrlock_target": -1 2024-04-23T10:32:03.970 INFO:teuthology.orchestra.run.smithi114.stdout: } 2024-04-23T10:32:03.970 INFO:teuthology.orchestra.run.smithi114.stdout: ] 2024-04-23T10:32:03.970 INFO:teuthology.orchestra.run.smithi114.stdout: } 2024-04-23T10:32:03.970 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-23T10:32:03.970 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-04-23T10:32:03.970 INFO:teuthology.orchestra.run.smithi114.stdout: "description": "internal op quiesce_inode:mds.2:118 fp=#0x1000000020c", 2024-04-23T10:32:03.970 INFO:teuthology.orchestra.run.smithi114.stdout: "initiated_at": "2024-04-23T10:31:21.463112+0000", 2024-04-23T10:32:03.970 INFO:teuthology.orchestra.run.smithi114.stdout: "age": 42.365632798, 2024-04-23T10:32:03.970 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 42.366852076999997, 2024-04-23T10:32:03.970 INFO:teuthology.orchestra.run.smithi114.stdout: "continuous": true, 2024-04-23T10:32:03.970 INFO:teuthology.orchestra.run.smithi114.stdout: "type_data": { 2024-04-23T10:32:03.970 INFO:teuthology.orchestra.run.smithi114.stdout: "result": -2147483648, 2024-04-23T10:32:03.970 INFO:teuthology.orchestra.run.smithi114.stdout: "flag_point": "quiesce complete for non-auth inode", 2024-04-23T10:32:03.970 INFO:teuthology.orchestra.run.smithi114.stdout: "reqid": { 2024-04-23T10:32:03.971 INFO:teuthology.orchestra.run.smithi114.stdout: "entity": { 2024-04-23T10:32:03.971 INFO:teuthology.orchestra.run.smithi114.stdout: "type": "mds", 2024-04-23T10:32:03.971 INFO:teuthology.orchestra.run.smithi114.stdout: "num": 2 2024-04-23T10:32:03.971 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-23T10:32:03.971 INFO:teuthology.orchestra.run.smithi114.stdout: "tid": 118 2024-04-23T10:32:03.971 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-23T10:32:03.971 INFO:teuthology.orchestra.run.smithi114.stdout: "op_type": "internal_op", 2024-04-23T10:32:03.971 INFO:teuthology.orchestra.run.smithi114.stdout: "internal_op": 5385, 2024-04-23T10:32:03.971 INFO:teuthology.orchestra.run.smithi114.stdout: "op_name": "quiesce_inode", 2024-04-23T10:32:03.971 INFO:teuthology.orchestra.run.smithi114.stdout: "events": [ 2024-04-23T10:32:03.971 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-04-23T10:32:03.971 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-04-23T10:31:21.463112+0000", 2024-04-23T10:32:03.971 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "initiated" 2024-04-23T10:32:03.971 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-23T10:32:03.972 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-04-23T10:32:03.972 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-04-23T10:31:21.463112+0000", 2024-04-23T10:32:03.972 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "header_read" 2024-04-23T10:32:03.972 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-23T10:32:03.972 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-04-23T10:32:03.972 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-04-23T10:31:21.463112+0000", 2024-04-23T10:32:03.972 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "throttled" 2024-04-23T10:32:03.972 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-23T10:32:03.972 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-04-23T10:32:03.972 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-04-23T10:31:21.463112+0000", 2024-04-23T10:32:03.972 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "all_read" 2024-04-23T10:32:03.972 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-23T10:32:03.972 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-04-23T10:32:03.973 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-04-23T10:31:21.463112+0000", 2024-04-23T10:32:03.973 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "dispatched" 2024-04-23T10:32:03.973 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-23T10:32:03.973 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-04-23T10:32:03.973 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-04-23T10:31:21.463360+0000", 2024-04-23T10:32:03.973 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "requesting remote authpins" 2024-04-23T10:32:03.973 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-23T10:32:03.973 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-04-23T10:32:03.973 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-04-23T10:31:21.481931+0000", 2024-04-23T10:32:03.973 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "acquired locks" 2024-04-23T10:32:03.973 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-23T10:32:03.973 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-04-23T10:32:03.973 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-04-23T10:31:21.481945+0000", 2024-04-23T10:32:03.973 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "quiesce complete for non-auth inode" 2024-04-23T10:32:03.974 INFO:teuthology.orchestra.run.smithi114.stdout: } 2024-04-23T10:32:03.974 INFO:teuthology.orchestra.run.smithi114.stdout: ], 2024-04-23T10:32:03.974 INFO:teuthology.orchestra.run.smithi114.stdout: "locks": [ 2024-04-23T10:32:03.974 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-04-23T10:32:03.974 INFO:teuthology.orchestra.run.smithi114.stdout: "object": { 2024-04-23T10:32:03.974 INFO:teuthology.orchestra.run.smithi114.stdout: "is_auth": false, 2024-04-23T10:32:03.974 INFO:teuthology.orchestra.run.smithi114.stdout: "auth_state": { 2024-04-23T10:32:03.974 INFO:teuthology.orchestra.run.smithi114.stdout: "replicas": {} 2024-04-23T10:32:03.974 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-23T10:32:03.974 INFO:teuthology.orchestra.run.smithi114.stdout: "replica_state": { 2024-04-23T10:32:03.974 INFO:teuthology.orchestra.run.smithi114.stdout: "authority": [ 2024-04-23T10:32:03.974 INFO:teuthology.orchestra.run.smithi114.stdout: 0, 2024-04-23T10:32:03.974 INFO:teuthology.orchestra.run.smithi114.stdout: -2 2024-04-23T10:32:03.974 INFO:teuthology.orchestra.run.smithi114.stdout: ], 2024-04-23T10:32:03.975 INFO:teuthology.orchestra.run.smithi114.stdout: "replica_nonce": 1 2024-04-23T10:32:03.975 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-23T10:32:03.975 INFO:teuthology.orchestra.run.smithi114.stdout: "auth_pins": 0, 2024-04-23T10:32:03.975 INFO:teuthology.orchestra.run.smithi114.stdout: "is_frozen": false, 2024-04-23T10:32:03.975 INFO:teuthology.orchestra.run.smithi114.stdout: "is_freezing": false, 2024-04-23T10:32:03.975 INFO:teuthology.orchestra.run.smithi114.stdout: "pins": { 2024-04-23T10:32:03.975 INFO:teuthology.orchestra.run.smithi114.stdout: "request": 1, 2024-04-23T10:32:03.975 INFO:teuthology.orchestra.run.smithi114.stdout: "lock": 1, 2024-04-23T10:32:03.975 INFO:teuthology.orchestra.run.smithi114.stdout: "dirwaiter": 0, 2024-04-23T10:32:03.975 INFO:teuthology.orchestra.run.smithi114.stdout: "discoverbase": 0 2024-04-23T10:32:03.975 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-23T10:32:03.975 INFO:teuthology.orchestra.run.smithi114.stdout: "nref": 2 2024-04-23T10:32:03.975 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-23T10:32:03.975 INFO:teuthology.orchestra.run.smithi114.stdout: "object_string": "[inode 0x1000000020c [...7,head] /volumes/qa/sv_1/b206794e-7622-4ab8-8943-f3ae4e65b7a3/client.0/tmp/clients/client0/~dmtmp/EXCEL/ rep@0.1 v45489 f(v0 m2024-04-23T10:31:17.692717+0000 2=2+0) n(v0 rc2024-04-23T10:27:51.972397+0000 3=2+1)/n(v0 rc2024-04-23T10:27:51.967397+0000 2=1+1) (inest mix) (iquiesce lock x=1 by request(mds.2:118 nref=3)) | request=1 lock=1 dirwaiter=0 discoverbase=0 0x55c68b9d0000]", 2024-04-23T10:32:03.976 INFO:teuthology.orchestra.run.smithi114.stdout: "lock": { 2024-04-23T10:32:03.976 INFO:teuthology.orchestra.run.smithi114.stdout: "gather_set": [], 2024-04-23T10:32:03.976 INFO:teuthology.orchestra.run.smithi114.stdout: "state": "lock", 2024-04-23T10:32:03.976 INFO:teuthology.orchestra.run.smithi114.stdout: "type": "iquiesce", 2024-04-23T10:32:03.976 INFO:teuthology.orchestra.run.smithi114.stdout: "is_leased": false, 2024-04-23T10:32:03.976 INFO:teuthology.orchestra.run.smithi114.stdout: "num_rdlocks": 0, 2024-04-23T10:32:03.976 INFO:teuthology.orchestra.run.smithi114.stdout: "num_wrlocks": 0, 2024-04-23T10:32:03.976 INFO:teuthology.orchestra.run.smithi114.stdout: "num_xlocks": 1, 2024-04-23T10:32:03.976 INFO:teuthology.orchestra.run.smithi114.stdout: "xlock_by": { 2024-04-23T10:32:03.976 INFO:teuthology.orchestra.run.smithi114.stdout: "reqid": { 2024-04-23T10:32:03.976 INFO:teuthology.orchestra.run.smithi114.stdout: "entity": { 2024-04-23T10:32:03.976 INFO:teuthology.orchestra.run.smithi114.stdout: "type": "mds", 2024-04-23T10:32:03.976 INFO:teuthology.orchestra.run.smithi114.stdout: "num": 2 2024-04-23T10:32:03.976 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-23T10:32:03.977 INFO:teuthology.orchestra.run.smithi114.stdout: "tid": 118 2024-04-23T10:32:03.977 INFO:teuthology.orchestra.run.smithi114.stdout: } 2024-04-23T10:32:03.977 INFO:teuthology.orchestra.run.smithi114.stdout: } 2024-04-23T10:32:03.977 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-23T10:32:03.977 INFO:teuthology.orchestra.run.smithi114.stdout: "flags": 4, 2024-04-23T10:32:03.977 INFO:teuthology.orchestra.run.smithi114.stdout: "wrlock_target": -1 2024-04-23T10:32:03.977 INFO:teuthology.orchestra.run.smithi114.stdout: } 2024-04-23T10:32:03.977 INFO:teuthology.orchestra.run.smithi114.stdout: ] 2024-04-23T10:32:03.977 INFO:teuthology.orchestra.run.smithi114.stdout: } 2024-04-23T10:32:03.977 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-23T10:32:03.977 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-04-23T10:32:03.977 INFO:teuthology.orchestra.run.smithi114.stdout: "description": "internal op quiesce_inode:mds.2:139 fp=#0x10000000243", 2024-04-23T10:32:03.977 INFO:teuthology.orchestra.run.smithi114.stdout: "initiated_at": "2024-04-23T10:31:21.489269+0000", 2024-04-23T10:32:03.977 INFO:teuthology.orchestra.run.smithi114.stdout: "age": 42.339475419999999, 2024-04-23T10:32:03.978 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 42.340755917000003, 2024-04-23T10:32:03.978 INFO:teuthology.orchestra.run.smithi114.stdout: "continuous": true, 2024-04-23T10:32:03.978 INFO:teuthology.orchestra.run.smithi114.stdout: "type_data": { 2024-04-23T10:32:03.978 INFO:teuthology.orchestra.run.smithi114.stdout: "result": -2147483648, 2024-04-23T10:32:03.978 INFO:teuthology.orchestra.run.smithi114.stdout: "flag_point": "quiesce complete for non-auth inode", 2024-04-23T10:32:03.978 INFO:teuthology.orchestra.run.smithi114.stdout: "reqid": { 2024-04-23T10:32:03.978 INFO:teuthology.orchestra.run.smithi114.stdout: "entity": { 2024-04-23T10:32:03.978 INFO:teuthology.orchestra.run.smithi114.stdout: "type": "mds", 2024-04-23T10:32:03.978 INFO:teuthology.orchestra.run.smithi114.stdout: "num": 2 2024-04-23T10:32:03.978 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-23T10:32:03.978 INFO:teuthology.orchestra.run.smithi114.stdout: "tid": 139 2024-04-23T10:32:03.978 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-23T10:32:03.978 INFO:teuthology.orchestra.run.smithi114.stdout: "op_type": "internal_op", 2024-04-23T10:32:03.978 INFO:teuthology.orchestra.run.smithi114.stdout: "internal_op": 5385, 2024-04-23T10:32:03.979 INFO:teuthology.orchestra.run.smithi114.stdout: "op_name": "quiesce_inode", 2024-04-23T10:32:03.979 INFO:teuthology.orchestra.run.smithi114.stdout: "events": [ 2024-04-23T10:32:03.979 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-04-23T10:32:03.979 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-04-23T10:31:21.489269+0000", 2024-04-23T10:32:03.979 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "initiated" 2024-04-23T10:32:03.979 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-23T10:32:03.979 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-04-23T10:32:03.979 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-04-23T10:31:21.489269+0000", 2024-04-23T10:32:03.979 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "header_read" 2024-04-23T10:32:03.979 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-23T10:32:03.979 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-04-23T10:32:03.979 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-04-23T10:31:21.489269+0000", 2024-04-23T10:32:03.979 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "throttled" 2024-04-23T10:32:03.979 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-23T10:32:03.980 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-04-23T10:32:03.980 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-04-23T10:31:21.489269+0000", 2024-04-23T10:32:03.980 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "all_read" 2024-04-23T10:32:03.980 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-23T10:32:03.980 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-04-23T10:32:03.980 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-04-23T10:31:21.489269+0000", 2024-04-23T10:32:03.980 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "dispatched" 2024-04-23T10:32:03.980 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-23T10:32:03.980 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-04-23T10:32:03.980 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-04-23T10:31:21.489494+0000", 2024-04-23T10:32:03.980 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "requesting remote authpins" 2024-04-23T10:32:03.980 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-23T10:32:03.980 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-04-23T10:32:03.980 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-04-23T10:31:21.509204+0000", 2024-04-23T10:32:03.981 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "acquired locks" 2024-04-23T10:32:03.981 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-23T10:32:03.981 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-04-23T10:32:03.981 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-04-23T10:31:21.509216+0000", 2024-04-23T10:32:03.981 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "quiesce complete for non-auth inode" 2024-04-23T10:32:03.981 INFO:teuthology.orchestra.run.smithi114.stdout: } 2024-04-23T10:32:03.981 INFO:teuthology.orchestra.run.smithi114.stdout: ], 2024-04-23T10:32:03.981 INFO:teuthology.orchestra.run.smithi114.stdout: "locks": [ 2024-04-23T10:32:03.981 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-04-23T10:32:03.981 INFO:teuthology.orchestra.run.smithi114.stdout: "object": { 2024-04-23T10:32:03.981 INFO:teuthology.orchestra.run.smithi114.stdout: "is_auth": false, 2024-04-23T10:32:03.981 INFO:teuthology.orchestra.run.smithi114.stdout: "auth_state": { 2024-04-23T10:32:03.981 INFO:teuthology.orchestra.run.smithi114.stdout: "replicas": {} 2024-04-23T10:32:03.981 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-23T10:32:03.981 INFO:teuthology.orchestra.run.smithi114.stdout: "replica_state": { 2024-04-23T10:32:03.982 INFO:teuthology.orchestra.run.smithi114.stdout: "authority": [ 2024-04-23T10:32:03.982 INFO:teuthology.orchestra.run.smithi114.stdout: 0, 2024-04-23T10:32:03.982 INFO:teuthology.orchestra.run.smithi114.stdout: -2 2024-04-23T10:32:03.982 INFO:teuthology.orchestra.run.smithi114.stdout: ], 2024-04-23T10:32:03.982 INFO:teuthology.orchestra.run.smithi114.stdout: "replica_nonce": 1 2024-04-23T10:32:03.982 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-23T10:32:03.982 INFO:teuthology.orchestra.run.smithi114.stdout: "auth_pins": 0, 2024-04-23T10:32:03.982 INFO:teuthology.orchestra.run.smithi114.stdout: "is_frozen": false, 2024-04-23T10:32:03.982 INFO:teuthology.orchestra.run.smithi114.stdout: "is_freezing": false, 2024-04-23T10:32:03.982 INFO:teuthology.orchestra.run.smithi114.stdout: "pins": { 2024-04-23T10:32:03.982 INFO:teuthology.orchestra.run.smithi114.stdout: "ptrwaiter": 0, 2024-04-23T10:32:03.982 INFO:teuthology.orchestra.run.smithi114.stdout: "request": 1, 2024-04-23T10:32:03.982 INFO:teuthology.orchestra.run.smithi114.stdout: "lock": 1, 2024-04-23T10:32:03.982 INFO:teuthology.orchestra.run.smithi114.stdout: "importingcaps": 0, 2024-04-23T10:32:03.983 INFO:teuthology.orchestra.run.smithi114.stdout: "caps": 0, 2024-04-23T10:32:03.983 INFO:teuthology.orchestra.run.smithi114.stdout: "exportingcaps": 0, 2024-04-23T10:32:03.983 INFO:teuthology.orchestra.run.smithi114.stdout: "replicated": 0, 2024-04-23T10:32:03.983 INFO:teuthology.orchestra.run.smithi114.stdout: "dirty": 0, 2024-04-23T10:32:03.983 INFO:teuthology.orchestra.run.smithi114.stdout: "authpin": 0, 2024-04-23T10:32:03.983 INFO:teuthology.orchestra.run.smithi114.stdout: "tempexporting": 0 2024-04-23T10:32:03.983 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-23T10:32:03.983 INFO:teuthology.orchestra.run.smithi114.stdout: "nref": 2 2024-04-23T10:32:03.983 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-23T10:32:03.983 INFO:teuthology.orchestra.run.smithi114.stdout: "object_string": "[inode 0x10000000243 [7,head] /volumes/qa/sv_1/b206794e-7622-4ab8-8943-f3ae4e65b7a3/client.0/tmp/clients/client0/~dmtmp/SEED/LARGE.FIL rep@0.1 v648 s=1867776 n(v0 rc2024-04-23T10:29:16.539476+0000 b1867776 1=1+0)/n(v0 rc2024-04-23T10:28:22.485704+0000 b1867776 1=1+0) (iquiesce lock x=1 by request(mds.2:139 nref=3)) | ptrwaiter=0 request=1 lock=1 importingcaps=0 caps=0 exportingcaps=0 replicated=0 dirty=0 authpin=0 tempexporting=0 0x55c68bc81700]", 2024-04-23T10:32:03.983 INFO:teuthology.orchestra.run.smithi114.stdout: "lock": { 2024-04-23T10:32:03.983 INFO:teuthology.orchestra.run.smithi114.stdout: "gather_set": [], 2024-04-23T10:32:03.983 INFO:teuthology.orchestra.run.smithi114.stdout: "state": "lock", 2024-04-23T10:32:03.983 INFO:teuthology.orchestra.run.smithi114.stdout: "type": "iquiesce", 2024-04-23T10:32:03.984 INFO:teuthology.orchestra.run.smithi114.stdout: "is_leased": false, 2024-04-23T10:32:03.984 INFO:teuthology.orchestra.run.smithi114.stdout: "num_rdlocks": 0, 2024-04-23T10:32:03.984 INFO:teuthology.orchestra.run.smithi114.stdout: "num_wrlocks": 0, 2024-04-23T10:32:03.984 INFO:teuthology.orchestra.run.smithi114.stdout: "num_xlocks": 1, 2024-04-23T10:32:03.984 INFO:teuthology.orchestra.run.smithi114.stdout: "xlock_by": { 2024-04-23T10:32:03.984 INFO:teuthology.orchestra.run.smithi114.stdout: "reqid": { 2024-04-23T10:32:03.984 INFO:teuthology.orchestra.run.smithi114.stdout: "entity": { 2024-04-23T10:32:03.984 INFO:teuthology.orchestra.run.smithi114.stdout: "type": "mds", 2024-04-23T10:32:03.984 INFO:teuthology.orchestra.run.smithi114.stdout: "num": 2 2024-04-23T10:32:03.984 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-23T10:32:03.984 INFO:teuthology.orchestra.run.smithi114.stdout: "tid": 139 2024-04-23T10:32:03.984 INFO:teuthology.orchestra.run.smithi114.stdout: } 2024-04-23T10:32:03.984 INFO:teuthology.orchestra.run.smithi114.stdout: } 2024-04-23T10:32:03.984 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-23T10:32:03.984 INFO:teuthology.orchestra.run.smithi114.stdout: "flags": 4, 2024-04-23T10:32:03.985 INFO:teuthology.orchestra.run.smithi114.stdout: "wrlock_target": -1 2024-04-23T10:32:03.985 INFO:teuthology.orchestra.run.smithi114.stdout: } 2024-04-23T10:32:03.985 INFO:teuthology.orchestra.run.smithi114.stdout: ] 2024-04-23T10:32:03.985 INFO:teuthology.orchestra.run.smithi114.stdout: } 2024-04-23T10:32:03.985 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-23T10:32:03.985 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-04-23T10:32:03.985 INFO:teuthology.orchestra.run.smithi114.stdout: "description": "internal op quiesce_inode:mds.2:119 fp=#0x10000000208", 2024-04-23T10:32:03.985 INFO:teuthology.orchestra.run.smithi114.stdout: "initiated_at": "2024-04-23T10:31:21.463127+0000", 2024-04-23T10:32:03.985 INFO:teuthology.orchestra.run.smithi114.stdout: "age": 42.365618232000003, 2024-04-23T10:32:03.985 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 42.366964119999999, 2024-04-23T10:32:03.985 INFO:teuthology.orchestra.run.smithi114.stdout: "continuous": true, 2024-04-23T10:32:03.985 INFO:teuthology.orchestra.run.smithi114.stdout: "type_data": { 2024-04-23T10:32:03.985 INFO:teuthology.orchestra.run.smithi114.stdout: "result": -2147483648, 2024-04-23T10:32:03.986 INFO:teuthology.orchestra.run.smithi114.stdout: "flag_point": "quiesce complete for non-auth inode", 2024-04-23T10:32:03.986 INFO:teuthology.orchestra.run.smithi114.stdout: "reqid": { 2024-04-23T10:32:03.986 INFO:teuthology.orchestra.run.smithi114.stdout: "entity": { 2024-04-23T10:32:03.986 INFO:teuthology.orchestra.run.smithi114.stdout: "type": "mds", 2024-04-23T10:32:03.986 INFO:teuthology.orchestra.run.smithi114.stdout: "num": 2 2024-04-23T10:32:03.986 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-23T10:32:03.986 INFO:teuthology.orchestra.run.smithi114.stdout: "tid": 119 2024-04-23T10:32:03.986 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-23T10:32:03.986 INFO:teuthology.orchestra.run.smithi114.stdout: "op_type": "internal_op", 2024-04-23T10:32:03.986 INFO:teuthology.orchestra.run.smithi114.stdout: "internal_op": 5385, 2024-04-23T10:32:03.986 INFO:teuthology.orchestra.run.smithi114.stdout: "op_name": "quiesce_inode", 2024-04-23T10:32:03.986 INFO:teuthology.orchestra.run.smithi114.stdout: "events": [ 2024-04-23T10:32:03.986 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-04-23T10:32:03.986 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-04-23T10:31:21.463127+0000", 2024-04-23T10:32:03.987 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "initiated" 2024-04-23T10:32:03.987 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-23T10:32:03.987 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-04-23T10:32:03.987 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-04-23T10:31:21.463127+0000", 2024-04-23T10:32:03.987 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "header_read" 2024-04-23T10:32:03.987 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-23T10:32:03.987 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-04-23T10:32:03.987 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-04-23T10:31:21.463127+0000", 2024-04-23T10:32:03.987 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "throttled" 2024-04-23T10:32:03.987 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-23T10:32:03.987 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-04-23T10:32:03.987 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-04-23T10:31:21.463127+0000", 2024-04-23T10:32:03.987 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "all_read" 2024-04-23T10:32:03.987 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-23T10:32:03.987 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-04-23T10:32:03.988 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-04-23T10:31:21.463127+0000", 2024-04-23T10:32:03.988 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "dispatched" 2024-04-23T10:32:03.988 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-23T10:32:03.988 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-04-23T10:32:03.988 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-04-23T10:31:21.463424+0000", 2024-04-23T10:32:03.988 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "requesting remote authpins" 2024-04-23T10:32:03.988 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-23T10:32:03.988 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-04-23T10:32:03.988 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-04-23T10:31:21.482852+0000", 2024-04-23T10:32:03.988 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "acquired locks" 2024-04-23T10:32:03.988 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-23T10:32:03.988 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-04-23T10:32:03.988 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-04-23T10:31:21.482861+0000", 2024-04-23T10:32:03.988 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "quiesce complete for non-auth inode" 2024-04-23T10:32:03.988 INFO:teuthology.orchestra.run.smithi114.stdout: } 2024-04-23T10:32:03.989 INFO:teuthology.orchestra.run.smithi114.stdout: ], 2024-04-23T10:32:03.989 INFO:teuthology.orchestra.run.smithi114.stdout: "locks": [ 2024-04-23T10:32:03.989 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-04-23T10:32:03.989 INFO:teuthology.orchestra.run.smithi114.stdout: "object": { 2024-04-23T10:32:03.989 INFO:teuthology.orchestra.run.smithi114.stdout: "is_auth": false, 2024-04-23T10:32:03.989 INFO:teuthology.orchestra.run.smithi114.stdout: "auth_state": { 2024-04-23T10:32:03.989 INFO:teuthology.orchestra.run.smithi114.stdout: "replicas": {} 2024-04-23T10:32:03.989 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-23T10:32:03.989 INFO:teuthology.orchestra.run.smithi114.stdout: "replica_state": { 2024-04-23T10:32:03.989 INFO:teuthology.orchestra.run.smithi114.stdout: "authority": [ 2024-04-23T10:32:03.989 INFO:teuthology.orchestra.run.smithi114.stdout: 0, 2024-04-23T10:32:03.989 INFO:teuthology.orchestra.run.smithi114.stdout: -2 2024-04-23T10:32:03.989 INFO:teuthology.orchestra.run.smithi114.stdout: ], 2024-04-23T10:32:03.989 INFO:teuthology.orchestra.run.smithi114.stdout: "replica_nonce": 1 2024-04-23T10:32:03.989 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-23T10:32:03.989 INFO:teuthology.orchestra.run.smithi114.stdout: "auth_pins": 0, 2024-04-23T10:32:03.990 INFO:teuthology.orchestra.run.smithi114.stdout: "is_frozen": false, 2024-04-23T10:32:03.990 INFO:teuthology.orchestra.run.smithi114.stdout: "is_freezing": false, 2024-04-23T10:32:03.990 INFO:teuthology.orchestra.run.smithi114.stdout: "pins": { 2024-04-23T10:32:03.990 INFO:teuthology.orchestra.run.smithi114.stdout: "request": 1, 2024-04-23T10:32:03.990 INFO:teuthology.orchestra.run.smithi114.stdout: "lock": 1, 2024-04-23T10:32:03.990 INFO:teuthology.orchestra.run.smithi114.stdout: "dirfrag": 0, 2024-04-23T10:32:03.990 INFO:teuthology.orchestra.run.smithi114.stdout: "caps": 1, 2024-04-23T10:32:03.990 INFO:teuthology.orchestra.run.smithi114.stdout: "dirwaiter": 0, 2024-04-23T10:32:03.990 INFO:teuthology.orchestra.run.smithi114.stdout: "discoverbase": 0 2024-04-23T10:32:03.990 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-23T10:32:03.990 INFO:teuthology.orchestra.run.smithi114.stdout: "nref": 3 2024-04-23T10:32:03.990 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-23T10:32:03.990 INFO:teuthology.orchestra.run.smithi114.stdout: "object_string": "[inode 0x10000000208 [...7,head] /volumes/qa/sv_1/b206794e-7622-4ab8-8943-f3ae4e65b7a3/client.0/tmp/clients/client0/~dmtmp/WORD/ rep@0.1 v45514 f(v0 m2024-04-23T10:31:17.675718+0000 3=3+0) n(v0 rc2024-04-23T10:28:39.366321+0000 b780288 4=3+1) old_inodes=1 (inest mix) (iquiesce lock x=1 by request(mds.2:119 nref=3)) caps={15312=pAsLsXsFs/pAsLsXsFsxcral@1378} | request=1 lock=1 dirfrag=0 caps=1 dirwaiter=0 discoverbase=0 0x55c68c112100]", 2024-04-23T10:32:03.990 INFO:teuthology.orchestra.run.smithi114.stdout: "lock": { 2024-04-23T10:32:03.991 INFO:teuthology.orchestra.run.smithi114.stdout: "gather_set": [], 2024-04-23T10:32:03.991 INFO:teuthology.orchestra.run.smithi114.stdout: "state": "lock", 2024-04-23T10:32:03.991 INFO:teuthology.orchestra.run.smithi114.stdout: "type": "iquiesce", 2024-04-23T10:32:03.991 INFO:teuthology.orchestra.run.smithi114.stdout: "is_leased": false, 2024-04-23T10:32:03.991 INFO:teuthology.orchestra.run.smithi114.stdout: "num_rdlocks": 0, 2024-04-23T10:32:03.991 INFO:teuthology.orchestra.run.smithi114.stdout: "num_wrlocks": 0, 2024-04-23T10:32:03.991 INFO:teuthology.orchestra.run.smithi114.stdout: "num_xlocks": 1, 2024-04-23T10:32:03.991 INFO:teuthology.orchestra.run.smithi114.stdout: "xlock_by": { 2024-04-23T10:32:03.991 INFO:teuthology.orchestra.run.smithi114.stdout: "reqid": { 2024-04-23T10:32:03.991 INFO:teuthology.orchestra.run.smithi114.stdout: "entity": { 2024-04-23T10:32:03.991 INFO:teuthology.orchestra.run.smithi114.stdout: "type": "mds", 2024-04-23T10:32:03.991 INFO:teuthology.orchestra.run.smithi114.stdout: "num": 2 2024-04-23T10:32:03.991 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-23T10:32:03.991 INFO:teuthology.orchestra.run.smithi114.stdout: "tid": 119 2024-04-23T10:32:03.992 INFO:teuthology.orchestra.run.smithi114.stdout: } 2024-04-23T10:32:03.992 INFO:teuthology.orchestra.run.smithi114.stdout: } 2024-04-23T10:32:03.992 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-23T10:32:03.992 INFO:teuthology.orchestra.run.smithi114.stdout: "flags": 4, 2024-04-23T10:32:03.992 INFO:teuthology.orchestra.run.smithi114.stdout: "wrlock_target": -1 2024-04-23T10:32:03.992 INFO:teuthology.orchestra.run.smithi114.stdout: } 2024-04-23T10:32:03.992 INFO:teuthology.orchestra.run.smithi114.stdout: ] 2024-04-23T10:32:03.992 INFO:teuthology.orchestra.run.smithi114.stdout: } 2024-04-23T10:32:03.992 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-23T10:32:03.992 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-04-23T10:32:03.992 INFO:teuthology.orchestra.run.smithi114.stdout: "description": "internal op quiesce_inode:mds.2:140 fp=#0x10000000226", 2024-04-23T10:32:03.992 INFO:teuthology.orchestra.run.smithi114.stdout: "initiated_at": "2024-04-23T10:31:21.494635+0000", 2024-04-23T10:32:03.992 INFO:teuthology.orchestra.run.smithi114.stdout: "age": 42.334109484000003, 2024-04-23T10:32:03.992 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 42.335519542999997, 2024-04-23T10:32:03.992 INFO:teuthology.orchestra.run.smithi114.stdout: "continuous": true, 2024-04-23T10:32:03.992 INFO:teuthology.orchestra.run.smithi114.stdout: "type_data": { 2024-04-23T10:32:03.993 INFO:teuthology.orchestra.run.smithi114.stdout: "result": -2147483648, 2024-04-23T10:32:03.993 INFO:teuthology.orchestra.run.smithi114.stdout: "flag_point": "quiesce complete", 2024-04-23T10:32:03.993 INFO:teuthology.orchestra.run.smithi114.stdout: "reqid": { 2024-04-23T10:32:03.993 INFO:teuthology.orchestra.run.smithi114.stdout: "entity": { 2024-04-23T10:32:03.993 INFO:teuthology.orchestra.run.smithi114.stdout: "type": "mds", 2024-04-23T10:32:03.993 INFO:teuthology.orchestra.run.smithi114.stdout: "num": 2 2024-04-23T10:32:03.993 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-23T10:32:03.993 INFO:teuthology.orchestra.run.smithi114.stdout: "tid": 140 2024-04-23T10:32:03.993 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-23T10:32:03.993 INFO:teuthology.orchestra.run.smithi114.stdout: "op_type": "internal_op", 2024-04-23T10:32:03.993 INFO:teuthology.orchestra.run.smithi114.stdout: "internal_op": 5385, 2024-04-23T10:32:03.993 INFO:teuthology.orchestra.run.smithi114.stdout: "op_name": "quiesce_inode", 2024-04-23T10:32:03.993 INFO:teuthology.orchestra.run.smithi114.stdout: "events": [ 2024-04-23T10:32:03.993 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-04-23T10:32:03.993 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-04-23T10:31:21.494635+0000", 2024-04-23T10:32:03.994 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "initiated" 2024-04-23T10:32:03.994 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-23T10:32:03.994 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-04-23T10:32:03.994 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-04-23T10:31:21.494635+0000", 2024-04-23T10:32:03.994 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "header_read" 2024-04-23T10:32:03.994 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-23T10:32:03.994 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-04-23T10:32:03.994 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-04-23T10:31:21.494635+0000", 2024-04-23T10:32:03.994 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "throttled" 2024-04-23T10:32:03.994 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-23T10:32:03.994 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-04-23T10:32:03.994 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-04-23T10:31:21.494635+0000", 2024-04-23T10:32:03.994 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "all_read" 2024-04-23T10:32:03.994 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-23T10:32:03.994 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-04-23T10:32:03.995 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-04-23T10:31:21.494635+0000", 2024-04-23T10:32:03.995 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "dispatched" 2024-04-23T10:32:03.995 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-23T10:32:03.995 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-04-23T10:32:03.995 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-04-23T10:31:21.494909+0000", 2024-04-23T10:32:03.995 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "acquired locks" 2024-04-23T10:32:03.995 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-23T10:32:03.995 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-04-23T10:32:03.995 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-04-23T10:31:21.495384+0000", 2024-04-23T10:32:03.995 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "quiesce complete" 2024-04-23T10:32:03.995 INFO:teuthology.orchestra.run.smithi114.stdout: } 2024-04-23T10:32:03.995 INFO:teuthology.orchestra.run.smithi114.stdout: ], 2024-04-23T10:32:03.995 INFO:teuthology.orchestra.run.smithi114.stdout: "locks": [ 2024-04-23T10:32:03.995 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-04-23T10:32:03.995 INFO:teuthology.orchestra.run.smithi114.stdout: "object": { 2024-04-23T10:32:03.996 INFO:teuthology.orchestra.run.smithi114.stdout: "is_auth": true, 2024-04-23T10:32:03.996 INFO:teuthology.orchestra.run.smithi114.stdout: "auth_state": { 2024-04-23T10:32:03.996 INFO:teuthology.orchestra.run.smithi114.stdout: "replicas": {} 2024-04-23T10:32:03.996 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-23T10:32:03.996 INFO:teuthology.orchestra.run.smithi114.stdout: "replica_state": { 2024-04-23T10:32:03.996 INFO:teuthology.orchestra.run.smithi114.stdout: "authority": [ 2024-04-23T10:32:03.996 INFO:teuthology.orchestra.run.smithi114.stdout: 2, 2024-04-23T10:32:03.996 INFO:teuthology.orchestra.run.smithi114.stdout: -2 2024-04-23T10:32:03.996 INFO:teuthology.orchestra.run.smithi114.stdout: ], 2024-04-23T10:32:03.996 INFO:teuthology.orchestra.run.smithi114.stdout: "replica_nonce": 0 2024-04-23T10:32:03.996 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-23T10:32:03.996 INFO:teuthology.orchestra.run.smithi114.stdout: "auth_pins": 1, 2024-04-23T10:32:03.996 INFO:teuthology.orchestra.run.smithi114.stdout: "is_frozen": false, 2024-04-23T10:32:03.996 INFO:teuthology.orchestra.run.smithi114.stdout: "is_freezing": false, 2024-04-23T10:32:03.996 INFO:teuthology.orchestra.run.smithi114.stdout: "pins": { 2024-04-23T10:32:03.997 INFO:teuthology.orchestra.run.smithi114.stdout: "ptrwaiter": 0, 2024-04-23T10:32:03.997 INFO:teuthology.orchestra.run.smithi114.stdout: "request": 1, 2024-04-23T10:32:03.997 INFO:teuthology.orchestra.run.smithi114.stdout: "lock": 1, 2024-04-23T10:32:03.997 INFO:teuthology.orchestra.run.smithi114.stdout: "importingcaps": 0, 2024-04-23T10:32:03.997 INFO:teuthology.orchestra.run.smithi114.stdout: "caps": 0, 2024-04-23T10:32:03.997 INFO:teuthology.orchestra.run.smithi114.stdout: "needsnapflush": 0, 2024-04-23T10:32:03.997 INFO:teuthology.orchestra.run.smithi114.stdout: "dirtyrstat": 0, 2024-04-23T10:32:03.997 INFO:teuthology.orchestra.run.smithi114.stdout: "replicated": 0, 2024-04-23T10:32:03.997 INFO:teuthology.orchestra.run.smithi114.stdout: "dirty": 1, 2024-04-23T10:32:03.997 INFO:teuthology.orchestra.run.smithi114.stdout: "waiter": 0, 2024-04-23T10:32:03.997 INFO:teuthology.orchestra.run.smithi114.stdout: "authpin": 1 2024-04-23T10:32:03.997 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-23T10:32:03.997 INFO:teuthology.orchestra.run.smithi114.stdout: "nref": 4 2024-04-23T10:32:03.997 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-23T10:32:03.997 INFO:teuthology.orchestra.run.smithi114.stdout: "object_string": "[inode 0x10000000226 [7,head] /volumes/qa/sv_1/b206794e-7622-4ab8-8943-f3ae4e65b7a3/client.0/tmp/clients/client0/~dmtmp/PM/T1.XLS auth v16150 ap=1 s=260096 n(v0 rc2024-04-23T10:27:52.072395+0000 b260096 1=1+0) (iquiesce lock x=1 by request(mds.2:140 nref=3)) | ptrwaiter=0 request=1 lock=1 importingcaps=0 caps=0 needsnapflush=0 dirtyrstat=0 replicated=0 dirty=1 waiter=0 authpin=1 0x55c68bc4e680]", 2024-04-23T10:32:03.998 INFO:teuthology.orchestra.run.smithi114.stdout: "lock": { 2024-04-23T10:32:03.998 INFO:teuthology.orchestra.run.smithi114.stdout: "gather_set": [], 2024-04-23T10:32:03.998 INFO:teuthology.orchestra.run.smithi114.stdout: "state": "lock", 2024-04-23T10:32:03.998 INFO:teuthology.orchestra.run.smithi114.stdout: "type": "iquiesce", 2024-04-23T10:32:03.998 INFO:teuthology.orchestra.run.smithi114.stdout: "is_leased": false, 2024-04-23T10:32:03.998 INFO:teuthology.orchestra.run.smithi114.stdout: "num_rdlocks": 0, 2024-04-23T10:32:03.998 INFO:teuthology.orchestra.run.smithi114.stdout: "num_wrlocks": 0, 2024-04-23T10:32:03.998 INFO:teuthology.orchestra.run.smithi114.stdout: "num_xlocks": 1, 2024-04-23T10:32:03.998 INFO:teuthology.orchestra.run.smithi114.stdout: "xlock_by": { 2024-04-23T10:32:03.998 INFO:teuthology.orchestra.run.smithi114.stdout: "reqid": { 2024-04-23T10:32:03.998 INFO:teuthology.orchestra.run.smithi114.stdout: "entity": { 2024-04-23T10:32:03.998 INFO:teuthology.orchestra.run.smithi114.stdout: "type": "mds", 2024-04-23T10:32:03.998 INFO:teuthology.orchestra.run.smithi114.stdout: "num": 2 2024-04-23T10:32:03.998 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-23T10:32:03.998 INFO:teuthology.orchestra.run.smithi114.stdout: "tid": 140 2024-04-23T10:32:03.999 INFO:teuthology.orchestra.run.smithi114.stdout: } 2024-04-23T10:32:03.999 INFO:teuthology.orchestra.run.smithi114.stdout: } 2024-04-23T10:32:03.999 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-23T10:32:03.999 INFO:teuthology.orchestra.run.smithi114.stdout: "flags": 4, 2024-04-23T10:32:03.999 INFO:teuthology.orchestra.run.smithi114.stdout: "wrlock_target": -1 2024-04-23T10:32:03.999 INFO:teuthology.orchestra.run.smithi114.stdout: } 2024-04-23T10:32:03.999 INFO:teuthology.orchestra.run.smithi114.stdout: ] 2024-04-23T10:32:03.999 INFO:teuthology.orchestra.run.smithi114.stdout: } 2024-04-23T10:32:03.999 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-23T10:32:03.999 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-04-23T10:32:03.999 INFO:teuthology.orchestra.run.smithi114.stdout: "description": "internal op quiesce_inode:mds.2:120 fp=#0x10000000240", 2024-04-23T10:32:03.999 INFO:teuthology.orchestra.run.smithi114.stdout: "initiated_at": "2024-04-23T10:31:21.463144+0000", 2024-04-23T10:32:03.999 INFO:teuthology.orchestra.run.smithi114.stdout: "age": 42.365600370000003, 2024-04-23T10:32:03.999 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 42.367073902000001, 2024-04-23T10:32:03.999 INFO:teuthology.orchestra.run.smithi114.stdout: "continuous": true, 2024-04-23T10:32:04.000 INFO:teuthology.orchestra.run.smithi114.stdout: "type_data": { 2024-04-23T10:32:04.000 INFO:teuthology.orchestra.run.smithi114.stdout: "result": -2147483648, 2024-04-23T10:32:04.000 INFO:teuthology.orchestra.run.smithi114.stdout: "flag_point": "quiesce complete for non-auth inode", 2024-04-23T10:32:04.000 INFO:teuthology.orchestra.run.smithi114.stdout: "reqid": { 2024-04-23T10:32:04.000 INFO:teuthology.orchestra.run.smithi114.stdout: "entity": { 2024-04-23T10:32:04.000 INFO:teuthology.orchestra.run.smithi114.stdout: "type": "mds", 2024-04-23T10:32:04.000 INFO:teuthology.orchestra.run.smithi114.stdout: "num": 2 2024-04-23T10:32:04.000 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-23T10:32:04.000 INFO:teuthology.orchestra.run.smithi114.stdout: "tid": 120 2024-04-23T10:32:04.000 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-23T10:32:04.000 INFO:teuthology.orchestra.run.smithi114.stdout: "op_type": "internal_op", 2024-04-23T10:32:04.000 INFO:teuthology.orchestra.run.smithi114.stdout: "internal_op": 5385, 2024-04-23T10:32:04.000 INFO:teuthology.orchestra.run.smithi114.stdout: "op_name": "quiesce_inode", 2024-04-23T10:32:04.000 INFO:teuthology.orchestra.run.smithi114.stdout: "events": [ 2024-04-23T10:32:04.000 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-04-23T10:32:04.001 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-04-23T10:31:21.463144+0000", 2024-04-23T10:32:04.001 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "initiated" 2024-04-23T10:32:04.001 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-23T10:32:04.001 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-04-23T10:32:04.001 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-04-23T10:31:21.463144+0000", 2024-04-23T10:32:04.001 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "header_read" 2024-04-23T10:32:04.001 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-23T10:32:04.001 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-04-23T10:32:04.001 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-04-23T10:31:21.463144+0000", 2024-04-23T10:32:04.001 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "throttled" 2024-04-23T10:32:04.001 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-23T10:32:04.001 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-04-23T10:32:04.001 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-04-23T10:31:21.463144+0000", 2024-04-23T10:32:04.001 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "all_read" 2024-04-23T10:32:04.001 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-23T10:32:04.002 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-04-23T10:32:04.002 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-04-23T10:31:21.463144+0000", 2024-04-23T10:32:04.002 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "dispatched" 2024-04-23T10:32:04.002 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-23T10:32:04.002 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-04-23T10:32:04.002 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-04-23T10:31:21.463539+0000", 2024-04-23T10:32:04.002 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "requesting remote authpins" 2024-04-23T10:32:04.002 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-23T10:32:04.002 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-04-23T10:32:04.002 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-04-23T10:31:21.489203+0000", 2024-04-23T10:32:04.002 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "acquired locks" 2024-04-23T10:32:04.002 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-23T10:32:04.002 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-04-23T10:32:04.002 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-04-23T10:31:21.509244+0000", 2024-04-23T10:32:04.003 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "quiesce complete for non-auth inode" 2024-04-23T10:32:04.003 INFO:teuthology.orchestra.run.smithi114.stdout: } 2024-04-23T10:32:04.003 INFO:teuthology.orchestra.run.smithi114.stdout: ], 2024-04-23T10:32:04.003 INFO:teuthology.orchestra.run.smithi114.stdout: "locks": [ 2024-04-23T10:32:04.003 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-04-23T10:32:04.003 INFO:teuthology.orchestra.run.smithi114.stdout: "object": { 2024-04-23T10:32:04.003 INFO:teuthology.orchestra.run.smithi114.stdout: "is_auth": false, 2024-04-23T10:32:04.003 INFO:teuthology.orchestra.run.smithi114.stdout: "auth_state": { 2024-04-23T10:32:04.003 INFO:teuthology.orchestra.run.smithi114.stdout: "replicas": {} 2024-04-23T10:32:04.003 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-23T10:32:04.003 INFO:teuthology.orchestra.run.smithi114.stdout: "replica_state": { 2024-04-23T10:32:04.003 INFO:teuthology.orchestra.run.smithi114.stdout: "authority": [ 2024-04-23T10:32:04.003 INFO:teuthology.orchestra.run.smithi114.stdout: 0, 2024-04-23T10:32:04.003 INFO:teuthology.orchestra.run.smithi114.stdout: -2 2024-04-23T10:32:04.003 INFO:teuthology.orchestra.run.smithi114.stdout: ], 2024-04-23T10:32:04.004 INFO:teuthology.orchestra.run.smithi114.stdout: "replica_nonce": 1 2024-04-23T10:32:04.004 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-23T10:32:04.004 INFO:teuthology.orchestra.run.smithi114.stdout: "auth_pins": 0, 2024-04-23T10:32:04.004 INFO:teuthology.orchestra.run.smithi114.stdout: "is_frozen": false, 2024-04-23T10:32:04.004 INFO:teuthology.orchestra.run.smithi114.stdout: "is_freezing": false, 2024-04-23T10:32:04.004 INFO:teuthology.orchestra.run.smithi114.stdout: "pins": { 2024-04-23T10:32:04.004 INFO:teuthology.orchestra.run.smithi114.stdout: "dirtyscattered": 0, 2024-04-23T10:32:04.004 INFO:teuthology.orchestra.run.smithi114.stdout: "request": 1, 2024-04-23T10:32:04.004 INFO:teuthology.orchestra.run.smithi114.stdout: "lock": 1, 2024-04-23T10:32:04.004 INFO:teuthology.orchestra.run.smithi114.stdout: "importing": 0, 2024-04-23T10:32:04.004 INFO:teuthology.orchestra.run.smithi114.stdout: "dirfrag": 1, 2024-04-23T10:32:04.004 INFO:teuthology.orchestra.run.smithi114.stdout: "caps": 1, 2024-04-23T10:32:04.004 INFO:teuthology.orchestra.run.smithi114.stdout: "dirwaiter": 0, 2024-04-23T10:32:04.004 INFO:teuthology.orchestra.run.smithi114.stdout: "waiter": 0, 2024-04-23T10:32:04.004 INFO:teuthology.orchestra.run.smithi114.stdout: "discoverbase": 0 2024-04-23T10:32:04.005 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-23T10:32:04.005 INFO:teuthology.orchestra.run.smithi114.stdout: "nref": 4 2024-04-23T10:32:04.005 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-23T10:32:04.005 INFO:teuthology.orchestra.run.smithi114.stdout: "object_string": "[inode 0x10000000240 [...7,head] /volumes/qa/sv_1/b206794e-7622-4ab8-8943-f3ae4e65b7a3/client.0/tmp/clients/client0/~dmtmp/SEED/ rep@0.1 v45361 f(v0 m2024-04-23T10:27:52.196392+0000 3=3+0) n(v2 rc2024-04-23T10:28:22.485704+0000 b3244032 4=3+1)/n(v0 rc2024-04-23T10:27:52.187392+0000 2=1+1) (inest mix) (iquiesce lock x=1 by request(mds.2:120 nref=3)) caps={15312=pAsLsXsFs/pAsLsXsFsxcral@15} | dirtyscattered=0 request=1 lock=1 importing=0 dirfrag=1 caps=1 dirwaiter=0 waiter=0 discoverbase=0 0x55c68bbfbb80]", 2024-04-23T10:32:04.005 INFO:teuthology.orchestra.run.smithi114.stdout: "lock": { 2024-04-23T10:32:04.005 INFO:teuthology.orchestra.run.smithi114.stdout: "gather_set": [], 2024-04-23T10:32:04.005 INFO:teuthology.orchestra.run.smithi114.stdout: "state": "lock", 2024-04-23T10:32:04.005 INFO:teuthology.orchestra.run.smithi114.stdout: "type": "iquiesce", 2024-04-23T10:32:04.005 INFO:teuthology.orchestra.run.smithi114.stdout: "is_leased": false, 2024-04-23T10:32:04.005 INFO:teuthology.orchestra.run.smithi114.stdout: "num_rdlocks": 0, 2024-04-23T10:32:04.005 INFO:teuthology.orchestra.run.smithi114.stdout: "num_wrlocks": 0, 2024-04-23T10:32:04.005 INFO:teuthology.orchestra.run.smithi114.stdout: "num_xlocks": 1, 2024-04-23T10:32:04.005 INFO:teuthology.orchestra.run.smithi114.stdout: "xlock_by": { 2024-04-23T10:32:04.005 INFO:teuthology.orchestra.run.smithi114.stdout: "reqid": { 2024-04-23T10:32:04.005 INFO:teuthology.orchestra.run.smithi114.stdout: "entity": { 2024-04-23T10:32:04.006 INFO:teuthology.orchestra.run.smithi114.stdout: "type": "mds", 2024-04-23T10:32:04.006 INFO:teuthology.orchestra.run.smithi114.stdout: "num": 2 2024-04-23T10:32:04.006 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-23T10:32:04.006 INFO:teuthology.orchestra.run.smithi114.stdout: "tid": 120 2024-04-23T10:32:04.006 INFO:teuthology.orchestra.run.smithi114.stdout: } 2024-04-23T10:32:04.006 INFO:teuthology.orchestra.run.smithi114.stdout: } 2024-04-23T10:32:04.006 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-23T10:32:04.006 INFO:teuthology.orchestra.run.smithi114.stdout: "flags": 4, 2024-04-23T10:32:04.006 INFO:teuthology.orchestra.run.smithi114.stdout: "wrlock_target": -1 2024-04-23T10:32:04.006 INFO:teuthology.orchestra.run.smithi114.stdout: } 2024-04-23T10:32:04.006 INFO:teuthology.orchestra.run.smithi114.stdout: ] 2024-04-23T10:32:04.006 INFO:teuthology.orchestra.run.smithi114.stdout: } 2024-04-23T10:32:04.006 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-23T10:32:04.006 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-04-23T10:32:04.007 INFO:teuthology.orchestra.run.smithi114.stdout: "description": "internal op quiesce_inode:mds.2:141 fp=#0x10000000227", 2024-04-23T10:32:04.007 INFO:teuthology.orchestra.run.smithi114.stdout: "initiated_at": "2024-04-23T10:31:21.494648+0000", 2024-04-23T10:32:04.007 INFO:teuthology.orchestra.run.smithi114.stdout: "age": 42.334096457999998, 2024-04-23T10:32:04.007 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 42.335637937999998, 2024-04-23T10:32:04.007 INFO:teuthology.orchestra.run.smithi114.stdout: "continuous": true, 2024-04-23T10:32:04.007 INFO:teuthology.orchestra.run.smithi114.stdout: "type_data": { 2024-04-23T10:32:04.007 INFO:teuthology.orchestra.run.smithi114.stdout: "result": -2147483648, 2024-04-23T10:32:04.007 INFO:teuthology.orchestra.run.smithi114.stdout: "flag_point": "quiesce complete", 2024-04-23T10:32:04.007 INFO:teuthology.orchestra.run.smithi114.stdout: "reqid": { 2024-04-23T10:32:04.007 INFO:teuthology.orchestra.run.smithi114.stdout: "entity": { 2024-04-23T10:32:04.007 INFO:teuthology.orchestra.run.smithi114.stdout: "type": "mds", 2024-04-23T10:32:04.007 INFO:teuthology.orchestra.run.smithi114.stdout: "num": 2 2024-04-23T10:32:04.007 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-23T10:32:04.007 INFO:teuthology.orchestra.run.smithi114.stdout: "tid": 141 2024-04-23T10:32:04.007 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-23T10:32:04.007 INFO:teuthology.orchestra.run.smithi114.stdout: "op_type": "internal_op", 2024-04-23T10:32:04.008 INFO:teuthology.orchestra.run.smithi114.stdout: "internal_op": 5385, 2024-04-23T10:32:04.008 INFO:teuthology.orchestra.run.smithi114.stdout: "op_name": "quiesce_inode", 2024-04-23T10:32:04.008 INFO:teuthology.orchestra.run.smithi114.stdout: "events": [ 2024-04-23T10:32:04.008 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-04-23T10:32:04.008 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-04-23T10:31:21.494648+0000", 2024-04-23T10:32:04.008 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "initiated" 2024-04-23T10:32:04.008 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-23T10:32:04.008 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-04-23T10:32:04.008 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-04-23T10:31:21.494648+0000", 2024-04-23T10:32:04.008 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "header_read" 2024-04-23T10:32:04.008 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-23T10:32:04.008 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-04-23T10:32:04.008 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-04-23T10:31:21.494648+0000", 2024-04-23T10:32:04.008 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "throttled" 2024-04-23T10:32:04.009 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-23T10:32:04.009 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-04-23T10:32:04.009 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-04-23T10:31:21.494648+0000", 2024-04-23T10:32:04.009 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "all_read" 2024-04-23T10:32:04.009 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-23T10:32:04.009 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-04-23T10:32:04.009 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-04-23T10:31:21.494648+0000", 2024-04-23T10:32:04.009 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "dispatched" 2024-04-23T10:32:04.009 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-23T10:32:04.009 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-04-23T10:32:04.009 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-04-23T10:31:21.495697+0000", 2024-04-23T10:32:04.009 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "failed to xlock, waiting" 2024-04-23T10:32:04.009 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-23T10:32:04.009 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-04-23T10:32:04.009 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-04-23T10:31:21.496927+0000", 2024-04-23T10:32:04.010 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "acquired locks" 2024-04-23T10:32:04.010 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-23T10:32:04.010 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-04-23T10:32:04.010 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-04-23T10:31:21.497138+0000", 2024-04-23T10:32:04.010 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "quiesce complete" 2024-04-23T10:32:04.010 INFO:teuthology.orchestra.run.smithi114.stdout: } 2024-04-23T10:32:04.010 INFO:teuthology.orchestra.run.smithi114.stdout: ], 2024-04-23T10:32:04.010 INFO:teuthology.orchestra.run.smithi114.stdout: "locks": [ 2024-04-23T10:32:04.010 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-04-23T10:32:04.010 INFO:teuthology.orchestra.run.smithi114.stdout: "object": { 2024-04-23T10:32:04.010 INFO:teuthology.orchestra.run.smithi114.stdout: "is_auth": true, 2024-04-23T10:32:04.010 INFO:teuthology.orchestra.run.smithi114.stdout: "auth_state": { 2024-04-23T10:32:04.010 INFO:teuthology.orchestra.run.smithi114.stdout: "replicas": {} 2024-04-23T10:32:04.010 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-23T10:32:04.011 INFO:teuthology.orchestra.run.smithi114.stdout: "replica_state": { 2024-04-23T10:32:04.011 INFO:teuthology.orchestra.run.smithi114.stdout: "authority": [ 2024-04-23T10:32:04.011 INFO:teuthology.orchestra.run.smithi114.stdout: 2, 2024-04-23T10:32:04.011 INFO:teuthology.orchestra.run.smithi114.stdout: -2 2024-04-23T10:32:04.011 INFO:teuthology.orchestra.run.smithi114.stdout: ], 2024-04-23T10:32:04.011 INFO:teuthology.orchestra.run.smithi114.stdout: "replica_nonce": 0 2024-04-23T10:32:04.011 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-23T10:32:04.011 INFO:teuthology.orchestra.run.smithi114.stdout: "auth_pins": 1, 2024-04-23T10:32:04.011 INFO:teuthology.orchestra.run.smithi114.stdout: "is_frozen": false, 2024-04-23T10:32:04.011 INFO:teuthology.orchestra.run.smithi114.stdout: "is_freezing": false, 2024-04-23T10:32:04.011 INFO:teuthology.orchestra.run.smithi114.stdout: "pins": { 2024-04-23T10:32:04.011 INFO:teuthology.orchestra.run.smithi114.stdout: "ptrwaiter": 0, 2024-04-23T10:32:04.011 INFO:teuthology.orchestra.run.smithi114.stdout: "request": 1, 2024-04-23T10:32:04.011 INFO:teuthology.orchestra.run.smithi114.stdout: "lock": 1, 2024-04-23T10:32:04.011 INFO:teuthology.orchestra.run.smithi114.stdout: "importingcaps": 0, 2024-04-23T10:32:04.012 INFO:teuthology.orchestra.run.smithi114.stdout: "caps": 0, 2024-04-23T10:32:04.012 INFO:teuthology.orchestra.run.smithi114.stdout: "replicated": 0, 2024-04-23T10:32:04.012 INFO:teuthology.orchestra.run.smithi114.stdout: "dirty": 1, 2024-04-23T10:32:04.012 INFO:teuthology.orchestra.run.smithi114.stdout: "waiter": 0, 2024-04-23T10:32:04.012 INFO:teuthology.orchestra.run.smithi114.stdout: "authpin": 1 2024-04-23T10:32:04.012 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-23T10:32:04.012 INFO:teuthology.orchestra.run.smithi114.stdout: "nref": 4 2024-04-23T10:32:04.012 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-23T10:32:04.012 INFO:teuthology.orchestra.run.smithi114.stdout: "object_string": "[inode 0x10000000227 [7,head] /volumes/qa/sv_1/b206794e-7622-4ab8-8943-f3ae4e65b7a3/client.0/tmp/clients/client0/~dmtmp/PM/EVENTS.DOC auth v16153 ap=1 s=260096 n(v0 rc2024-04-23T10:27:52.076395+0000 b260096 1=1+0) (iquiesce lock x=1 by request(mds.2:141 nref=3)) | ptrwaiter=0 request=1 lock=1 importingcaps=0 caps=0 replicated=0 dirty=1 waiter=0 authpin=1 0x55c68bc4ec00]", 2024-04-23T10:32:04.012 INFO:teuthology.orchestra.run.smithi114.stdout: "lock": { 2024-04-23T10:32:04.012 INFO:teuthology.orchestra.run.smithi114.stdout: "gather_set": [], 2024-04-23T10:32:04.012 INFO:teuthology.orchestra.run.smithi114.stdout: "state": "lock", 2024-04-23T10:32:04.012 INFO:teuthology.orchestra.run.smithi114.stdout: "type": "iquiesce", 2024-04-23T10:32:04.012 INFO:teuthology.orchestra.run.smithi114.stdout: "is_leased": false, 2024-04-23T10:32:04.012 INFO:teuthology.orchestra.run.smithi114.stdout: "num_rdlocks": 0, 2024-04-23T10:32:04.013 INFO:teuthology.orchestra.run.smithi114.stdout: "num_wrlocks": 0, 2024-04-23T10:32:04.013 INFO:teuthology.orchestra.run.smithi114.stdout: "num_xlocks": 1, 2024-04-23T10:32:04.013 INFO:teuthology.orchestra.run.smithi114.stdout: "xlock_by": { 2024-04-23T10:32:04.013 INFO:teuthology.orchestra.run.smithi114.stdout: "reqid": { 2024-04-23T10:32:04.013 INFO:teuthology.orchestra.run.smithi114.stdout: "entity": { 2024-04-23T10:32:04.013 INFO:teuthology.orchestra.run.smithi114.stdout: "type": "mds", 2024-04-23T10:32:04.013 INFO:teuthology.orchestra.run.smithi114.stdout: "num": 2 2024-04-23T10:32:04.013 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-23T10:32:04.013 INFO:teuthology.orchestra.run.smithi114.stdout: "tid": 141 2024-04-23T10:32:04.013 INFO:teuthology.orchestra.run.smithi114.stdout: } 2024-04-23T10:32:04.013 INFO:teuthology.orchestra.run.smithi114.stdout: } 2024-04-23T10:32:04.013 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-23T10:32:04.013 INFO:teuthology.orchestra.run.smithi114.stdout: "flags": 4, 2024-04-23T10:32:04.013 INFO:teuthology.orchestra.run.smithi114.stdout: "wrlock_target": -1 2024-04-23T10:32:04.013 INFO:teuthology.orchestra.run.smithi114.stdout: } 2024-04-23T10:32:04.014 INFO:teuthology.orchestra.run.smithi114.stdout: ] 2024-04-23T10:32:04.014 INFO:teuthology.orchestra.run.smithi114.stdout: } 2024-04-23T10:32:04.014 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-23T10:32:04.014 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-04-23T10:32:04.014 INFO:teuthology.orchestra.run.smithi114.stdout: "description": "internal op quiesce_inode:mds.2:121 fp=#0x10000000221", 2024-04-23T10:32:04.014 INFO:teuthology.orchestra.run.smithi114.stdout: "initiated_at": "2024-04-23T10:31:21.463157+0000", 2024-04-23T10:32:04.014 INFO:teuthology.orchestra.run.smithi114.stdout: "age": 42.365587916999999, 2024-04-23T10:32:04.014 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 42.367192140999997, 2024-04-23T10:32:04.014 INFO:teuthology.orchestra.run.smithi114.stdout: "continuous": true, 2024-04-23T10:32:04.014 INFO:teuthology.orchestra.run.smithi114.stdout: "type_data": { 2024-04-23T10:32:04.014 INFO:teuthology.orchestra.run.smithi114.stdout: "result": -2147483648, 2024-04-23T10:32:04.014 INFO:teuthology.orchestra.run.smithi114.stdout: "flag_point": "quiesce complete for non-auth inode", 2024-04-23T10:32:04.014 INFO:teuthology.orchestra.run.smithi114.stdout: "reqid": { 2024-04-23T10:32:04.014 INFO:teuthology.orchestra.run.smithi114.stdout: "entity": { 2024-04-23T10:32:04.015 INFO:teuthology.orchestra.run.smithi114.stdout: "type": "mds", 2024-04-23T10:32:04.015 INFO:teuthology.orchestra.run.smithi114.stdout: "num": 2 2024-04-23T10:32:04.015 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-23T10:32:04.015 INFO:teuthology.orchestra.run.smithi114.stdout: "tid": 121 2024-04-23T10:32:04.015 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-23T10:32:04.015 INFO:teuthology.orchestra.run.smithi114.stdout: "op_type": "internal_op", 2024-04-23T10:32:04.015 INFO:teuthology.orchestra.run.smithi114.stdout: "internal_op": 5385, 2024-04-23T10:32:04.015 INFO:teuthology.orchestra.run.smithi114.stdout: "op_name": "quiesce_inode", 2024-04-23T10:32:04.015 INFO:teuthology.orchestra.run.smithi114.stdout: "events": [ 2024-04-23T10:32:04.015 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-04-23T10:32:04.015 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-04-23T10:31:21.463157+0000", 2024-04-23T10:32:04.015 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "initiated" 2024-04-23T10:32:04.016 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-23T10:32:04.016 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-04-23T10:32:04.016 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-04-23T10:31:21.463157+0000", 2024-04-23T10:32:04.016 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "header_read" 2024-04-23T10:32:04.016 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-23T10:32:04.016 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-04-23T10:32:04.016 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-04-23T10:31:21.463157+0000", 2024-04-23T10:32:04.016 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "throttled" 2024-04-23T10:32:04.016 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-23T10:32:04.016 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-04-23T10:32:04.016 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-04-23T10:31:21.463157+0000", 2024-04-23T10:32:04.016 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "all_read" 2024-04-23T10:32:04.016 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-23T10:32:04.016 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-04-23T10:32:04.017 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-04-23T10:31:21.463157+0000", 2024-04-23T10:32:04.017 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "dispatched" 2024-04-23T10:32:04.017 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-23T10:32:04.017 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-04-23T10:32:04.017 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-04-23T10:31:21.463611+0000", 2024-04-23T10:32:04.017 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "requesting remote authpins" 2024-04-23T10:32:04.017 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-23T10:32:04.017 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-04-23T10:32:04.017 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-04-23T10:31:21.490252+0000", 2024-04-23T10:32:04.017 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "acquired locks" 2024-04-23T10:32:04.017 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-23T10:32:04.017 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-04-23T10:32:04.017 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-04-23T10:31:21.490264+0000", 2024-04-23T10:32:04.017 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "quiesce complete for non-auth inode" 2024-04-23T10:32:04.018 INFO:teuthology.orchestra.run.smithi114.stdout: } 2024-04-23T10:32:04.018 INFO:teuthology.orchestra.run.smithi114.stdout: ], 2024-04-23T10:32:04.018 INFO:teuthology.orchestra.run.smithi114.stdout: "locks": [ 2024-04-23T10:32:04.018 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-04-23T10:32:04.018 INFO:teuthology.orchestra.run.smithi114.stdout: "object": { 2024-04-23T10:32:04.018 INFO:teuthology.orchestra.run.smithi114.stdout: "is_auth": false, 2024-04-23T10:32:04.018 INFO:teuthology.orchestra.run.smithi114.stdout: "auth_state": { 2024-04-23T10:32:04.018 INFO:teuthology.orchestra.run.smithi114.stdout: "replicas": {} 2024-04-23T10:32:04.018 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-23T10:32:04.018 INFO:teuthology.orchestra.run.smithi114.stdout: "replica_state": { 2024-04-23T10:32:04.018 INFO:teuthology.orchestra.run.smithi114.stdout: "authority": [ 2024-04-23T10:32:04.018 INFO:teuthology.orchestra.run.smithi114.stdout: 0, 2024-04-23T10:32:04.018 INFO:teuthology.orchestra.run.smithi114.stdout: -2 2024-04-23T10:32:04.018 INFO:teuthology.orchestra.run.smithi114.stdout: ], 2024-04-23T10:32:04.019 INFO:teuthology.orchestra.run.smithi114.stdout: "replica_nonce": 1 2024-04-23T10:32:04.019 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-23T10:32:04.019 INFO:teuthology.orchestra.run.smithi114.stdout: "auth_pins": 0, 2024-04-23T10:32:04.019 INFO:teuthology.orchestra.run.smithi114.stdout: "is_frozen": false, 2024-04-23T10:32:04.019 INFO:teuthology.orchestra.run.smithi114.stdout: "is_freezing": false, 2024-04-23T10:32:04.019 INFO:teuthology.orchestra.run.smithi114.stdout: "pins": { 2024-04-23T10:32:04.019 INFO:teuthology.orchestra.run.smithi114.stdout: "request": 1, 2024-04-23T10:32:04.019 INFO:teuthology.orchestra.run.smithi114.stdout: "lock": 1, 2024-04-23T10:32:04.019 INFO:teuthology.orchestra.run.smithi114.stdout: "dirwaiter": 0, 2024-04-23T10:32:04.019 INFO:teuthology.orchestra.run.smithi114.stdout: "discoverbase": 0 2024-04-23T10:32:04.019 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-23T10:32:04.019 INFO:teuthology.orchestra.run.smithi114.stdout: "nref": 2 2024-04-23T10:32:04.019 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-23T10:32:04.019 INFO:teuthology.orchestra.run.smithi114.stdout: "object_string": "[inode 0x10000000221 [...7,head] /volumes/qa/sv_1/b206794e-7622-4ab8-8943-f3ae4e65b7a3/client.0/tmp/clients/client0/~dmtmp/WORDPRO/ rep@0.1 v45516 f(v0 m2024-04-23T10:31:17.760716+0000 3=3+0) n(v0 rc2024-04-23T10:28:42.855241+0000 b780288 4=3+1) old_inodes=1 (inest mix) (iquiesce lock x=1 by request(mds.2:121 nref=3)) | request=1 lock=1 dirwaiter=0 discoverbase=0 0x55c68c0d6c00]", 2024-04-23T10:32:04.020 INFO:teuthology.orchestra.run.smithi114.stdout: "lock": { 2024-04-23T10:32:04.020 INFO:teuthology.orchestra.run.smithi114.stdout: "gather_set": [], 2024-04-23T10:32:04.020 INFO:teuthology.orchestra.run.smithi114.stdout: "state": "lock", 2024-04-23T10:32:04.020 INFO:teuthology.orchestra.run.smithi114.stdout: "type": "iquiesce", 2024-04-23T10:32:04.020 INFO:teuthology.orchestra.run.smithi114.stdout: "is_leased": false, 2024-04-23T10:32:04.020 INFO:teuthology.orchestra.run.smithi114.stdout: "num_rdlocks": 0, 2024-04-23T10:32:04.020 INFO:teuthology.orchestra.run.smithi114.stdout: "num_wrlocks": 0, 2024-04-23T10:32:04.020 INFO:teuthology.orchestra.run.smithi114.stdout: "num_xlocks": 1, 2024-04-23T10:32:04.020 INFO:teuthology.orchestra.run.smithi114.stdout: "xlock_by": { 2024-04-23T10:32:04.020 INFO:teuthology.orchestra.run.smithi114.stdout: "reqid": { 2024-04-23T10:32:04.020 INFO:teuthology.orchestra.run.smithi114.stdout: "entity": { 2024-04-23T10:32:04.020 INFO:teuthology.orchestra.run.smithi114.stdout: "type": "mds", 2024-04-23T10:32:04.020 INFO:teuthology.orchestra.run.smithi114.stdout: "num": 2 2024-04-23T10:32:04.020 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-23T10:32:04.020 INFO:teuthology.orchestra.run.smithi114.stdout: "tid": 121 2024-04-23T10:32:04.021 INFO:teuthology.orchestra.run.smithi114.stdout: } 2024-04-23T10:32:04.021 INFO:teuthology.orchestra.run.smithi114.stdout: } 2024-04-23T10:32:04.021 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-23T10:32:04.021 INFO:teuthology.orchestra.run.smithi114.stdout: "flags": 4, 2024-04-23T10:32:04.021 INFO:teuthology.orchestra.run.smithi114.stdout: "wrlock_target": -1 2024-04-23T10:32:04.021 INFO:teuthology.orchestra.run.smithi114.stdout: } 2024-04-23T10:32:04.021 INFO:teuthology.orchestra.run.smithi114.stdout: ] 2024-04-23T10:32:04.021 INFO:teuthology.orchestra.run.smithi114.stdout: } 2024-04-23T10:32:04.021 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-23T10:32:04.021 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-04-23T10:32:04.021 INFO:teuthology.orchestra.run.smithi114.stdout: "description": "internal op quiesce_inode:mds.2:142 fp=#0x10000000228", 2024-04-23T10:32:04.021 INFO:teuthology.orchestra.run.smithi114.stdout: "initiated_at": "2024-04-23T10:31:21.494670+0000", 2024-04-23T10:32:04.021 INFO:teuthology.orchestra.run.smithi114.stdout: "age": 42.334074563999998, 2024-04-23T10:32:04.022 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 42.335740854999997, 2024-04-23T10:32:04.022 INFO:teuthology.orchestra.run.smithi114.stdout: "continuous": true, 2024-04-23T10:32:04.022 INFO:teuthology.orchestra.run.smithi114.stdout: "type_data": { 2024-04-23T10:32:04.022 INFO:teuthology.orchestra.run.smithi114.stdout: "result": -2147483648, 2024-04-23T10:32:04.022 INFO:teuthology.orchestra.run.smithi114.stdout: "flag_point": "quiesce complete", 2024-04-23T10:32:04.022 INFO:teuthology.orchestra.run.smithi114.stdout: "reqid": { 2024-04-23T10:32:04.022 INFO:teuthology.orchestra.run.smithi114.stdout: "entity": { 2024-04-23T10:32:04.022 INFO:teuthology.orchestra.run.smithi114.stdout: "type": "mds", 2024-04-23T10:32:04.022 INFO:teuthology.orchestra.run.smithi114.stdout: "num": 2 2024-04-23T10:32:04.022 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-23T10:32:04.022 INFO:teuthology.orchestra.run.smithi114.stdout: "tid": 142 2024-04-23T10:32:04.022 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-23T10:32:04.022 INFO:teuthology.orchestra.run.smithi114.stdout: "op_type": "internal_op", 2024-04-23T10:32:04.022 INFO:teuthology.orchestra.run.smithi114.stdout: "internal_op": 5385, 2024-04-23T10:32:04.023 INFO:teuthology.orchestra.run.smithi114.stdout: "op_name": "quiesce_inode", 2024-04-23T10:32:04.023 INFO:teuthology.orchestra.run.smithi114.stdout: "events": [ 2024-04-23T10:32:04.023 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-04-23T10:32:04.023 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-04-23T10:31:21.494670+0000", 2024-04-23T10:32:04.023 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "initiated" 2024-04-23T10:32:04.023 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-23T10:32:04.023 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-04-23T10:32:04.023 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-04-23T10:31:21.494670+0000", 2024-04-23T10:32:04.023 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "header_read" 2024-04-23T10:32:04.023 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-23T10:32:04.023 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-04-23T10:32:04.023 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-04-23T10:31:21.494670+0000", 2024-04-23T10:32:04.023 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "throttled" 2024-04-23T10:32:04.023 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-23T10:32:04.023 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-04-23T10:32:04.024 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-04-23T10:31:21.494670+0000", 2024-04-23T10:32:04.024 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "all_read" 2024-04-23T10:32:04.024 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-23T10:32:04.024 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-04-23T10:32:04.024 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-04-23T10:31:21.494670+0000", 2024-04-23T10:32:04.024 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "dispatched" 2024-04-23T10:32:04.024 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-23T10:32:04.024 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-04-23T10:32:04.024 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-04-23T10:31:21.495979+0000", 2024-04-23T10:32:04.024 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "failed to xlock, waiting" 2024-04-23T10:32:04.024 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-23T10:32:04.024 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-04-23T10:32:04.024 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-04-23T10:31:21.497990+0000", 2024-04-23T10:32:04.024 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "acquired locks" 2024-04-23T10:32:04.024 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-23T10:32:04.025 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-04-23T10:32:04.025 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-04-23T10:31:21.498222+0000", 2024-04-23T10:32:04.025 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "quiesce complete" 2024-04-23T10:32:04.025 INFO:teuthology.orchestra.run.smithi114.stdout: } 2024-04-23T10:32:04.025 INFO:teuthology.orchestra.run.smithi114.stdout: ], 2024-04-23T10:32:04.025 INFO:teuthology.orchestra.run.smithi114.stdout: "locks": [ 2024-04-23T10:32:04.025 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-04-23T10:32:04.025 INFO:teuthology.orchestra.run.smithi114.stdout: "object": { 2024-04-23T10:32:04.025 INFO:teuthology.orchestra.run.smithi114.stdout: "is_auth": true, 2024-04-23T10:32:04.025 INFO:teuthology.orchestra.run.smithi114.stdout: "auth_state": { 2024-04-23T10:32:04.025 INFO:teuthology.orchestra.run.smithi114.stdout: "replicas": {} 2024-04-23T10:32:04.025 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-23T10:32:04.025 INFO:teuthology.orchestra.run.smithi114.stdout: "replica_state": { 2024-04-23T10:32:04.025 INFO:teuthology.orchestra.run.smithi114.stdout: "authority": [ 2024-04-23T10:32:04.025 INFO:teuthology.orchestra.run.smithi114.stdout: 2, 2024-04-23T10:32:04.026 INFO:teuthology.orchestra.run.smithi114.stdout: -2 2024-04-23T10:32:04.026 INFO:teuthology.orchestra.run.smithi114.stdout: ], 2024-04-23T10:32:04.026 INFO:teuthology.orchestra.run.smithi114.stdout: "replica_nonce": 0 2024-04-23T10:32:04.026 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-23T10:32:04.026 INFO:teuthology.orchestra.run.smithi114.stdout: "auth_pins": 1, 2024-04-23T10:32:04.026 INFO:teuthology.orchestra.run.smithi114.stdout: "is_frozen": false, 2024-04-23T10:32:04.026 INFO:teuthology.orchestra.run.smithi114.stdout: "is_freezing": false, 2024-04-23T10:32:04.026 INFO:teuthology.orchestra.run.smithi114.stdout: "pins": { 2024-04-23T10:32:04.026 INFO:teuthology.orchestra.run.smithi114.stdout: "ptrwaiter": 0, 2024-04-23T10:32:04.026 INFO:teuthology.orchestra.run.smithi114.stdout: "request": 1, 2024-04-23T10:32:04.026 INFO:teuthology.orchestra.run.smithi114.stdout: "lock": 1, 2024-04-23T10:32:04.026 INFO:teuthology.orchestra.run.smithi114.stdout: "importingcaps": 0, 2024-04-23T10:32:04.026 INFO:teuthology.orchestra.run.smithi114.stdout: "caps": 0, 2024-04-23T10:32:04.026 INFO:teuthology.orchestra.run.smithi114.stdout: "replicated": 0, 2024-04-23T10:32:04.026 INFO:teuthology.orchestra.run.smithi114.stdout: "dirty": 1, 2024-04-23T10:32:04.027 INFO:teuthology.orchestra.run.smithi114.stdout: "waiter": 0, 2024-04-23T10:32:04.027 INFO:teuthology.orchestra.run.smithi114.stdout: "authpin": 1 2024-04-23T10:32:04.027 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-23T10:32:04.027 INFO:teuthology.orchestra.run.smithi114.stdout: "nref": 4 2024-04-23T10:32:04.027 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-23T10:32:04.027 INFO:teuthology.orchestra.run.smithi114.stdout: "object_string": "[inode 0x10000000228 [7,head] /volumes/qa/sv_1/b206794e-7622-4ab8-8943-f3ae4e65b7a3/client.0/tmp/clients/client0/~dmtmp/PM/MOVED.DOC auth v16155 ap=1 s=260096 n(v0 rc2024-04-23T10:27:52.081395+0000 b260096 1=1+0) (iquiesce lock x=1 by request(mds.2:142 nref=3)) | ptrwaiter=0 request=1 lock=1 importingcaps=0 caps=0 replicated=0 dirty=1 waiter=0 authpin=1 0x55c68bc4f180]", 2024-04-23T10:32:04.027 INFO:teuthology.orchestra.run.smithi114.stdout: "lock": { 2024-04-23T10:32:04.027 INFO:teuthology.orchestra.run.smithi114.stdout: "gather_set": [], 2024-04-23T10:32:04.027 INFO:teuthology.orchestra.run.smithi114.stdout: "state": "lock", 2024-04-23T10:32:04.027 INFO:teuthology.orchestra.run.smithi114.stdout: "type": "iquiesce", 2024-04-23T10:32:04.027 INFO:teuthology.orchestra.run.smithi114.stdout: "is_leased": false, 2024-04-23T10:32:04.027 INFO:teuthology.orchestra.run.smithi114.stdout: "num_rdlocks": 0, 2024-04-23T10:32:04.027 INFO:teuthology.orchestra.run.smithi114.stdout: "num_wrlocks": 0, 2024-04-23T10:32:04.027 INFO:teuthology.orchestra.run.smithi114.stdout: "num_xlocks": 1, 2024-04-23T10:32:04.027 INFO:teuthology.orchestra.run.smithi114.stdout: "xlock_by": { 2024-04-23T10:32:04.027 INFO:teuthology.orchestra.run.smithi114.stdout: "reqid": { 2024-04-23T10:32:04.028 INFO:teuthology.orchestra.run.smithi114.stdout: "entity": { 2024-04-23T10:32:04.028 INFO:teuthology.orchestra.run.smithi114.stdout: "type": "mds", 2024-04-23T10:32:04.028 INFO:teuthology.orchestra.run.smithi114.stdout: "num": 2 2024-04-23T10:32:04.028 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-23T10:32:04.028 INFO:teuthology.orchestra.run.smithi114.stdout: "tid": 142 2024-04-23T10:32:04.028 INFO:teuthology.orchestra.run.smithi114.stdout: } 2024-04-23T10:32:04.028 INFO:teuthology.orchestra.run.smithi114.stdout: } 2024-04-23T10:32:04.028 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-23T10:32:04.028 INFO:teuthology.orchestra.run.smithi114.stdout: "flags": 4, 2024-04-23T10:32:04.028 INFO:teuthology.orchestra.run.smithi114.stdout: "wrlock_target": -1 2024-04-23T10:32:04.028 INFO:teuthology.orchestra.run.smithi114.stdout: } 2024-04-23T10:32:04.028 INFO:teuthology.orchestra.run.smithi114.stdout: ] 2024-04-23T10:32:04.028 INFO:teuthology.orchestra.run.smithi114.stdout: } 2024-04-23T10:32:04.028 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-23T10:32:04.028 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-04-23T10:32:04.029 INFO:teuthology.orchestra.run.smithi114.stdout: "description": "internal op quiesce_inode:mds.2:122 fp=#0x10000000229", 2024-04-23T10:32:04.029 INFO:teuthology.orchestra.run.smithi114.stdout: "initiated_at": "2024-04-23T10:31:21.463170+0000", 2024-04-23T10:32:04.029 INFO:teuthology.orchestra.run.smithi114.stdout: "age": 42.365574952999999, 2024-04-23T10:32:04.029 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 42.367315818999998, 2024-04-23T10:32:04.029 INFO:teuthology.orchestra.run.smithi114.stdout: "continuous": true, 2024-04-23T10:32:04.029 INFO:teuthology.orchestra.run.smithi114.stdout: "type_data": { 2024-04-23T10:32:04.029 INFO:teuthology.orchestra.run.smithi114.stdout: "result": -2147483648, 2024-04-23T10:32:04.029 INFO:teuthology.orchestra.run.smithi114.stdout: "flag_point": "quiesce complete for non-auth inode", 2024-04-23T10:32:04.029 INFO:teuthology.orchestra.run.smithi114.stdout: "reqid": { 2024-04-23T10:32:04.029 INFO:teuthology.orchestra.run.smithi114.stdout: "entity": { 2024-04-23T10:32:04.029 INFO:teuthology.orchestra.run.smithi114.stdout: "type": "mds", 2024-04-23T10:32:04.029 INFO:teuthology.orchestra.run.smithi114.stdout: "num": 2 2024-04-23T10:32:04.029 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-23T10:32:04.029 INFO:teuthology.orchestra.run.smithi114.stdout: "tid": 122 2024-04-23T10:32:04.029 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-23T10:32:04.029 INFO:teuthology.orchestra.run.smithi114.stdout: "op_type": "internal_op", 2024-04-23T10:32:04.030 INFO:teuthology.orchestra.run.smithi114.stdout: "internal_op": 5385, 2024-04-23T10:32:04.030 INFO:teuthology.orchestra.run.smithi114.stdout: "op_name": "quiesce_inode", 2024-04-23T10:32:04.030 INFO:teuthology.orchestra.run.smithi114.stdout: "events": [ 2024-04-23T10:32:04.030 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-04-23T10:32:04.030 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-04-23T10:31:21.463170+0000", 2024-04-23T10:32:04.030 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "initiated" 2024-04-23T10:32:04.030 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-23T10:32:04.030 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-04-23T10:32:04.030 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-04-23T10:31:21.463170+0000", 2024-04-23T10:32:04.030 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "header_read" 2024-04-23T10:32:04.030 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-23T10:32:04.030 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-04-23T10:32:04.030 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-04-23T10:31:21.463170+0000", 2024-04-23T10:32:04.030 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "throttled" 2024-04-23T10:32:04.031 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-23T10:32:04.031 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-04-23T10:32:04.031 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-04-23T10:31:21.463170+0000", 2024-04-23T10:32:04.031 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "all_read" 2024-04-23T10:32:04.031 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-23T10:32:04.031 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-04-23T10:32:04.031 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-04-23T10:31:21.463170+0000", 2024-04-23T10:32:04.031 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "dispatched" 2024-04-23T10:32:04.031 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-23T10:32:04.031 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-04-23T10:32:04.031 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-04-23T10:31:21.463682+0000", 2024-04-23T10:32:04.031 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "requesting remote authpins" 2024-04-23T10:32:04.031 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-23T10:32:04.031 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-04-23T10:32:04.032 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-04-23T10:31:21.493342+0000", 2024-04-23T10:32:04.032 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "acquired locks" 2024-04-23T10:32:04.032 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-23T10:32:04.032 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-04-23T10:32:04.032 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-04-23T10:31:21.493351+0000", 2024-04-23T10:32:04.032 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "quiesce complete for non-auth inode" 2024-04-23T10:32:04.032 INFO:teuthology.orchestra.run.smithi114.stdout: } 2024-04-23T10:32:04.032 INFO:teuthology.orchestra.run.smithi114.stdout: ], 2024-04-23T10:32:04.032 INFO:teuthology.orchestra.run.smithi114.stdout: "locks": [ 2024-04-23T10:32:04.032 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-04-23T10:32:04.032 INFO:teuthology.orchestra.run.smithi114.stdout: "object": { 2024-04-23T10:32:04.032 INFO:teuthology.orchestra.run.smithi114.stdout: "is_auth": false, 2024-04-23T10:32:04.032 INFO:teuthology.orchestra.run.smithi114.stdout: "auth_state": { 2024-04-23T10:32:04.033 INFO:teuthology.orchestra.run.smithi114.stdout: "replicas": {} 2024-04-23T10:32:04.033 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-23T10:32:04.033 INFO:teuthology.orchestra.run.smithi114.stdout: "replica_state": { 2024-04-23T10:32:04.033 INFO:teuthology.orchestra.run.smithi114.stdout: "authority": [ 2024-04-23T10:32:04.033 INFO:teuthology.orchestra.run.smithi114.stdout: 0, 2024-04-23T10:32:04.033 INFO:teuthology.orchestra.run.smithi114.stdout: -2 2024-04-23T10:32:04.033 INFO:teuthology.orchestra.run.smithi114.stdout: ], 2024-04-23T10:32:04.033 INFO:teuthology.orchestra.run.smithi114.stdout: "replica_nonce": 1 2024-04-23T10:32:04.033 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-23T10:32:04.033 INFO:teuthology.orchestra.run.smithi114.stdout: "auth_pins": 0, 2024-04-23T10:32:04.033 INFO:teuthology.orchestra.run.smithi114.stdout: "is_frozen": false, 2024-04-23T10:32:04.033 INFO:teuthology.orchestra.run.smithi114.stdout: "is_freezing": false, 2024-04-23T10:32:04.033 INFO:teuthology.orchestra.run.smithi114.stdout: "pins": { 2024-04-23T10:32:04.033 INFO:teuthology.orchestra.run.smithi114.stdout: "request": 1, 2024-04-23T10:32:04.033 INFO:teuthology.orchestra.run.smithi114.stdout: "lock": 1, 2024-04-23T10:32:04.034 INFO:teuthology.orchestra.run.smithi114.stdout: "dirfrag": 0, 2024-04-23T10:32:04.034 INFO:teuthology.orchestra.run.smithi114.stdout: "dirwaiter": 0, 2024-04-23T10:32:04.034 INFO:teuthology.orchestra.run.smithi114.stdout: "discoverbase": 0 2024-04-23T10:32:04.034 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-23T10:32:04.034 INFO:teuthology.orchestra.run.smithi114.stdout: "nref": 2 2024-04-23T10:32:04.034 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-23T10:32:04.034 INFO:teuthology.orchestra.run.smithi114.stdout: "object_string": "[inode 0x10000000229 [...7,head] /volumes/qa/sv_1/b206794e-7622-4ab8-8943-f3ae4e65b7a3/client.0/tmp/clients/client0/~dmtmp/PARADOX/ rep@0.1 v45567 f(v1976 m2024-04-23T10:31:16.541744+0000 24=24+0) n(v19 rc2024-04-23T10:31:16.541744+0000 b5201920 25=24+1)/n(v0 rc2024-04-23T10:27:52.084395+0000 2=1+1) (inest mix) (iquiesce lock x=1 by request(mds.2:122 nref=3)) | request=1 lock=1 dirfrag=0 dirwaiter=0 discoverbase=0 0x55c68bbfb600]", 2024-04-23T10:32:04.034 INFO:teuthology.orchestra.run.smithi114.stdout: "lock": { 2024-04-23T10:32:04.034 INFO:teuthology.orchestra.run.smithi114.stdout: "gather_set": [], 2024-04-23T10:32:04.034 INFO:teuthology.orchestra.run.smithi114.stdout: "state": "lock", 2024-04-23T10:32:04.034 INFO:teuthology.orchestra.run.smithi114.stdout: "type": "iquiesce", 2024-04-23T10:32:04.034 INFO:teuthology.orchestra.run.smithi114.stdout: "is_leased": false, 2024-04-23T10:32:04.034 INFO:teuthology.orchestra.run.smithi114.stdout: "num_rdlocks": 0, 2024-04-23T10:32:04.034 INFO:teuthology.orchestra.run.smithi114.stdout: "num_wrlocks": 0, 2024-04-23T10:32:04.035 INFO:teuthology.orchestra.run.smithi114.stdout: "num_xlocks": 1, 2024-04-23T10:32:04.035 INFO:teuthology.orchestra.run.smithi114.stdout: "xlock_by": { 2024-04-23T10:32:04.035 INFO:teuthology.orchestra.run.smithi114.stdout: "reqid": { 2024-04-23T10:32:04.035 INFO:teuthology.orchestra.run.smithi114.stdout: "entity": { 2024-04-23T10:32:04.035 INFO:teuthology.orchestra.run.smithi114.stdout: "type": "mds", 2024-04-23T10:32:04.035 INFO:teuthology.orchestra.run.smithi114.stdout: "num": 2 2024-04-23T10:32:04.035 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-23T10:32:04.035 INFO:teuthology.orchestra.run.smithi114.stdout: "tid": 122 2024-04-23T10:32:04.035 INFO:teuthology.orchestra.run.smithi114.stdout: } 2024-04-23T10:32:04.035 INFO:teuthology.orchestra.run.smithi114.stdout: } 2024-04-23T10:32:04.035 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-23T10:32:04.035 INFO:teuthology.orchestra.run.smithi114.stdout: "flags": 4, 2024-04-23T10:32:04.035 INFO:teuthology.orchestra.run.smithi114.stdout: "wrlock_target": -1 2024-04-23T10:32:04.035 INFO:teuthology.orchestra.run.smithi114.stdout: } 2024-04-23T10:32:04.035 INFO:teuthology.orchestra.run.smithi114.stdout: ] 2024-04-23T10:32:04.035 INFO:teuthology.orchestra.run.smithi114.stdout: } 2024-04-23T10:32:04.036 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-23T10:32:04.036 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-04-23T10:32:04.036 INFO:teuthology.orchestra.run.smithi114.stdout: "description": "internal op quiesce_inode:mds.2:143 fp=#0x10000000210", 2024-04-23T10:32:04.036 INFO:teuthology.orchestra.run.smithi114.stdout: "initiated_at": "2024-04-23T10:31:21.498646+0000", 2024-04-23T10:32:04.036 INFO:teuthology.orchestra.run.smithi114.stdout: "age": 42.330098722999999, 2024-04-23T10:32:04.036 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 42.331903072000003, 2024-04-23T10:32:04.036 INFO:teuthology.orchestra.run.smithi114.stdout: "continuous": true, 2024-04-23T10:32:04.036 INFO:teuthology.orchestra.run.smithi114.stdout: "type_data": { 2024-04-23T10:32:04.036 INFO:teuthology.orchestra.run.smithi114.stdout: "result": -2147483648, 2024-04-23T10:32:04.036 INFO:teuthology.orchestra.run.smithi114.stdout: "flag_point": "quiesce complete", 2024-04-23T10:32:04.036 INFO:teuthology.orchestra.run.smithi114.stdout: "reqid": { 2024-04-23T10:32:04.036 INFO:teuthology.orchestra.run.smithi114.stdout: "entity": { 2024-04-23T10:32:04.036 INFO:teuthology.orchestra.run.smithi114.stdout: "type": "mds", 2024-04-23T10:32:04.036 INFO:teuthology.orchestra.run.smithi114.stdout: "num": 2 2024-04-23T10:32:04.036 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-23T10:32:04.037 INFO:teuthology.orchestra.run.smithi114.stdout: "tid": 143 2024-04-23T10:32:04.037 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-23T10:32:04.037 INFO:teuthology.orchestra.run.smithi114.stdout: "op_type": "internal_op", 2024-04-23T10:32:04.037 INFO:teuthology.orchestra.run.smithi114.stdout: "internal_op": 5385, 2024-04-23T10:32:04.037 INFO:teuthology.orchestra.run.smithi114.stdout: "op_name": "quiesce_inode", 2024-04-23T10:32:04.037 INFO:teuthology.orchestra.run.smithi114.stdout: "events": [ 2024-04-23T10:32:04.037 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-04-23T10:32:04.037 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-04-23T10:31:21.498646+0000", 2024-04-23T10:32:04.037 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "initiated" 2024-04-23T10:32:04.037 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-23T10:32:04.037 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-04-23T10:32:04.037 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-04-23T10:31:21.498646+0000", 2024-04-23T10:32:04.037 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "header_read" 2024-04-23T10:32:04.037 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-23T10:32:04.037 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-04-23T10:32:04.037 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-04-23T10:31:21.498646+0000", 2024-04-23T10:32:04.037 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "throttled" 2024-04-23T10:32:04.038 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-23T10:32:04.038 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-04-23T10:32:04.038 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-04-23T10:31:21.498646+0000", 2024-04-23T10:32:04.038 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "all_read" 2024-04-23T10:32:04.038 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-23T10:32:04.038 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-04-23T10:32:04.038 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-04-23T10:31:21.498646+0000", 2024-04-23T10:32:04.038 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "dispatched" 2024-04-23T10:32:04.038 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-23T10:32:04.038 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-04-23T10:32:04.038 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-04-23T10:31:21.498985+0000", 2024-04-23T10:32:04.038 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "failed to xlock, waiting" 2024-04-23T10:32:04.038 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-23T10:32:04.038 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-04-23T10:32:04.038 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-04-23T10:31:21.500907+0000", 2024-04-23T10:32:04.039 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "failed to xlock, waiting" 2024-04-23T10:32:04.039 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-23T10:32:04.039 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-04-23T10:32:04.039 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-04-23T10:31:21.506656+0000", 2024-04-23T10:32:04.039 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "acquired locks" 2024-04-23T10:32:04.039 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-23T10:32:04.039 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-04-23T10:32:04.039 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-04-23T10:31:21.506825+0000", 2024-04-23T10:32:04.039 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "quiesce complete" 2024-04-23T10:32:04.039 INFO:teuthology.orchestra.run.smithi114.stdout: } 2024-04-23T10:32:04.039 INFO:teuthology.orchestra.run.smithi114.stdout: ], 2024-04-23T10:32:04.039 INFO:teuthology.orchestra.run.smithi114.stdout: "locks": [ 2024-04-23T10:32:04.039 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-04-23T10:32:04.039 INFO:teuthology.orchestra.run.smithi114.stdout: "object": { 2024-04-23T10:32:04.039 INFO:teuthology.orchestra.run.smithi114.stdout: "is_auth": true, 2024-04-23T10:32:04.039 INFO:teuthology.orchestra.run.smithi114.stdout: "auth_state": { 2024-04-23T10:32:04.039 INFO:teuthology.orchestra.run.smithi114.stdout: "replicas": {} 2024-04-23T10:32:04.040 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-23T10:32:04.040 INFO:teuthology.orchestra.run.smithi114.stdout: "replica_state": { 2024-04-23T10:32:04.040 INFO:teuthology.orchestra.run.smithi114.stdout: "authority": [ 2024-04-23T10:32:04.040 INFO:teuthology.orchestra.run.smithi114.stdout: 2, 2024-04-23T10:32:04.040 INFO:teuthology.orchestra.run.smithi114.stdout: -2 2024-04-23T10:32:04.040 INFO:teuthology.orchestra.run.smithi114.stdout: ], 2024-04-23T10:32:04.040 INFO:teuthology.orchestra.run.smithi114.stdout: "replica_nonce": 0 2024-04-23T10:32:04.040 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-23T10:32:04.040 INFO:teuthology.orchestra.run.smithi114.stdout: "auth_pins": 1, 2024-04-23T10:32:04.040 INFO:teuthology.orchestra.run.smithi114.stdout: "is_frozen": false, 2024-04-23T10:32:04.040 INFO:teuthology.orchestra.run.smithi114.stdout: "is_freezing": false, 2024-04-23T10:32:04.040 INFO:teuthology.orchestra.run.smithi114.stdout: "pins": { 2024-04-23T10:32:04.040 INFO:teuthology.orchestra.run.smithi114.stdout: "ptrwaiter": 0, 2024-04-23T10:32:04.040 INFO:teuthology.orchestra.run.smithi114.stdout: "request": 1, 2024-04-23T10:32:04.041 INFO:teuthology.orchestra.run.smithi114.stdout: "lock": 1, 2024-04-23T10:32:04.041 INFO:teuthology.orchestra.run.smithi114.stdout: "importingcaps": 0, 2024-04-23T10:32:04.041 INFO:teuthology.orchestra.run.smithi114.stdout: "caps": 0, 2024-04-23T10:32:04.041 INFO:teuthology.orchestra.run.smithi114.stdout: "needsnapflush": 0, 2024-04-23T10:32:04.041 INFO:teuthology.orchestra.run.smithi114.stdout: "replicated": 0, 2024-04-23T10:32:04.041 INFO:teuthology.orchestra.run.smithi114.stdout: "dirty": 1, 2024-04-23T10:32:04.041 INFO:teuthology.orchestra.run.smithi114.stdout: "waiter": 0, 2024-04-23T10:32:04.041 INFO:teuthology.orchestra.run.smithi114.stdout: "authpin": 1 2024-04-23T10:32:04.041 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-23T10:32:04.041 INFO:teuthology.orchestra.run.smithi114.stdout: "nref": 4 2024-04-23T10:32:04.041 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-23T10:32:04.041 INFO:teuthology.orchestra.run.smithi114.stdout: "object_string": "[inode 0x10000000210 [7,head] /volumes/qa/sv_1/b206794e-7622-4ab8-8943-f3ae4e65b7a3/client.0/tmp/clients/client0/~dmtmp/PWRPNT/PCBENCHM.PPT auth v12260 ap=1 s=260096 n(v0 rc2024-04-23T10:31:15.901758+0000 b260096 1=1+0) (iquiesce lock x=1 by request(mds.2:143 nref=3)) | ptrwaiter=0 request=1 lock=1 importingcaps=0 caps=0 needsnapflush=0 replicated=0 dirty=1 waiter=0 authpin=1 0x55c68bc7e000]", 2024-04-23T10:32:04.041 INFO:teuthology.orchestra.run.smithi114.stdout: "lock": { 2024-04-23T10:32:04.041 INFO:teuthology.orchestra.run.smithi114.stdout: "gather_set": [], 2024-04-23T10:32:04.041 INFO:teuthology.orchestra.run.smithi114.stdout: "state": "lock", 2024-04-23T10:32:04.042 INFO:teuthology.orchestra.run.smithi114.stdout: "type": "iquiesce", 2024-04-23T10:32:04.042 INFO:teuthology.orchestra.run.smithi114.stdout: "is_leased": false, 2024-04-23T10:32:04.042 INFO:teuthology.orchestra.run.smithi114.stdout: "num_rdlocks": 0, 2024-04-23T10:32:04.042 INFO:teuthology.orchestra.run.smithi114.stdout: "num_wrlocks": 0, 2024-04-23T10:32:04.042 INFO:teuthology.orchestra.run.smithi114.stdout: "num_xlocks": 1, 2024-04-23T10:32:04.042 INFO:teuthology.orchestra.run.smithi114.stdout: "xlock_by": { 2024-04-23T10:32:04.042 INFO:teuthology.orchestra.run.smithi114.stdout: "reqid": { 2024-04-23T10:32:04.042 INFO:teuthology.orchestra.run.smithi114.stdout: "entity": { 2024-04-23T10:32:04.042 INFO:teuthology.orchestra.run.smithi114.stdout: "type": "mds", 2024-04-23T10:32:04.042 INFO:teuthology.orchestra.run.smithi114.stdout: "num": 2 2024-04-23T10:32:04.042 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-23T10:32:04.042 INFO:teuthology.orchestra.run.smithi114.stdout: "tid": 143 2024-04-23T10:32:04.042 INFO:teuthology.orchestra.run.smithi114.stdout: } 2024-04-23T10:32:04.043 INFO:teuthology.orchestra.run.smithi114.stdout: } 2024-04-23T10:32:04.043 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-23T10:32:04.043 INFO:teuthology.orchestra.run.smithi114.stdout: "flags": 4, 2024-04-23T10:32:04.043 INFO:teuthology.orchestra.run.smithi114.stdout: "wrlock_target": -1 2024-04-23T10:32:04.043 INFO:teuthology.orchestra.run.smithi114.stdout: } 2024-04-23T10:32:04.043 INFO:teuthology.orchestra.run.smithi114.stdout: ] 2024-04-23T10:32:04.043 INFO:teuthology.orchestra.run.smithi114.stdout: } 2024-04-23T10:32:04.043 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-23T10:32:04.043 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-04-23T10:32:04.043 INFO:teuthology.orchestra.run.smithi114.stdout: "description": "internal op quiesce_inode:mds.2:123 fp=#0x10000000225", 2024-04-23T10:32:04.043 INFO:teuthology.orchestra.run.smithi114.stdout: "initiated_at": "2024-04-23T10:31:21.463187+0000", 2024-04-23T10:32:04.043 INFO:teuthology.orchestra.run.smithi114.stdout: "age": 42.365558229999998, 2024-04-23T10:32:04.043 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 42.367428386, 2024-04-23T10:32:04.043 INFO:teuthology.orchestra.run.smithi114.stdout: "continuous": true, 2024-04-23T10:32:04.043 INFO:teuthology.orchestra.run.smithi114.stdout: "type_data": { 2024-04-23T10:32:04.044 INFO:teuthology.orchestra.run.smithi114.stdout: "result": -2147483648, 2024-04-23T10:32:04.044 INFO:teuthology.orchestra.run.smithi114.stdout: "flag_point": "quiesce complete for non-auth inode", 2024-04-23T10:32:04.044 INFO:teuthology.orchestra.run.smithi114.stdout: "reqid": { 2024-04-23T10:32:04.044 INFO:teuthology.orchestra.run.smithi114.stdout: "entity": { 2024-04-23T10:32:04.044 INFO:teuthology.orchestra.run.smithi114.stdout: "type": "mds", 2024-04-23T10:32:04.044 INFO:teuthology.orchestra.run.smithi114.stdout: "num": 2 2024-04-23T10:32:04.044 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-23T10:32:04.044 INFO:teuthology.orchestra.run.smithi114.stdout: "tid": 123 2024-04-23T10:32:04.044 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-23T10:32:04.044 INFO:teuthology.orchestra.run.smithi114.stdout: "op_type": "internal_op", 2024-04-23T10:32:04.044 INFO:teuthology.orchestra.run.smithi114.stdout: "internal_op": 5385, 2024-04-23T10:32:04.044 INFO:teuthology.orchestra.run.smithi114.stdout: "op_name": "quiesce_inode", 2024-04-23T10:32:04.044 INFO:teuthology.orchestra.run.smithi114.stdout: "events": [ 2024-04-23T10:32:04.044 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-04-23T10:32:04.044 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-04-23T10:31:21.463187+0000", 2024-04-23T10:32:04.045 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "initiated" 2024-04-23T10:32:04.045 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-23T10:32:04.045 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-04-23T10:32:04.045 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-04-23T10:31:21.463187+0000", 2024-04-23T10:32:04.045 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "header_read" 2024-04-23T10:32:04.045 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-23T10:32:04.045 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-04-23T10:32:04.045 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-04-23T10:31:21.463187+0000", 2024-04-23T10:32:04.045 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "throttled" 2024-04-23T10:32:04.045 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-23T10:32:04.045 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-04-23T10:32:04.045 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-04-23T10:31:21.463187+0000", 2024-04-23T10:32:04.045 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "all_read" 2024-04-23T10:32:04.045 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-23T10:32:04.045 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-04-23T10:32:04.046 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-04-23T10:31:21.463187+0000", 2024-04-23T10:32:04.046 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "dispatched" 2024-04-23T10:32:04.046 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-23T10:32:04.046 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-04-23T10:32:04.046 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-04-23T10:31:21.463762+0000", 2024-04-23T10:32:04.046 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "requesting remote authpins" 2024-04-23T10:32:04.046 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-23T10:32:04.046 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-04-23T10:32:04.046 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-04-23T10:31:21.494626+0000", 2024-04-23T10:32:04.046 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "acquired locks" 2024-04-23T10:32:04.046 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-23T10:32:04.046 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-04-23T10:32:04.046 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-04-23T10:31:21.498260+0000", 2024-04-23T10:32:04.046 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "quiesce complete for non-auth inode" 2024-04-23T10:32:04.047 INFO:teuthology.orchestra.run.smithi114.stdout: } 2024-04-23T10:32:04.047 INFO:teuthology.orchestra.run.smithi114.stdout: ], 2024-04-23T10:32:04.047 INFO:teuthology.orchestra.run.smithi114.stdout: "locks": [ 2024-04-23T10:32:04.047 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-04-23T10:32:04.047 INFO:teuthology.orchestra.run.smithi114.stdout: "object": { 2024-04-23T10:32:04.047 INFO:teuthology.orchestra.run.smithi114.stdout: "is_auth": false, 2024-04-23T10:32:04.047 INFO:teuthology.orchestra.run.smithi114.stdout: "auth_state": { 2024-04-23T10:32:04.047 INFO:teuthology.orchestra.run.smithi114.stdout: "replicas": {} 2024-04-23T10:32:04.047 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-23T10:32:04.047 INFO:teuthology.orchestra.run.smithi114.stdout: "replica_state": { 2024-04-23T10:32:04.047 INFO:teuthology.orchestra.run.smithi114.stdout: "authority": [ 2024-04-23T10:32:04.047 INFO:teuthology.orchestra.run.smithi114.stdout: 0, 2024-04-23T10:32:04.047 INFO:teuthology.orchestra.run.smithi114.stdout: -2 2024-04-23T10:32:04.047 INFO:teuthology.orchestra.run.smithi114.stdout: ], 2024-04-23T10:32:04.048 INFO:teuthology.orchestra.run.smithi114.stdout: "replica_nonce": 1 2024-04-23T10:32:04.048 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-23T10:32:04.048 INFO:teuthology.orchestra.run.smithi114.stdout: "auth_pins": 0, 2024-04-23T10:32:04.048 INFO:teuthology.orchestra.run.smithi114.stdout: "is_frozen": false, 2024-04-23T10:32:04.048 INFO:teuthology.orchestra.run.smithi114.stdout: "is_freezing": false, 2024-04-23T10:32:04.048 INFO:teuthology.orchestra.run.smithi114.stdout: "pins": { 2024-04-23T10:32:04.048 INFO:teuthology.orchestra.run.smithi114.stdout: "dirtyscattered": 0, 2024-04-23T10:32:04.048 INFO:teuthology.orchestra.run.smithi114.stdout: "request": 1, 2024-04-23T10:32:04.048 INFO:teuthology.orchestra.run.smithi114.stdout: "lock": 1, 2024-04-23T10:32:04.048 INFO:teuthology.orchestra.run.smithi114.stdout: "importing": 0, 2024-04-23T10:32:04.048 INFO:teuthology.orchestra.run.smithi114.stdout: "dirfrag": 1, 2024-04-23T10:32:04.048 INFO:teuthology.orchestra.run.smithi114.stdout: "caps": 1, 2024-04-23T10:32:04.048 INFO:teuthology.orchestra.run.smithi114.stdout: "dirwaiter": 0, 2024-04-23T10:32:04.048 INFO:teuthology.orchestra.run.smithi114.stdout: "waiter": 0, 2024-04-23T10:32:04.048 INFO:teuthology.orchestra.run.smithi114.stdout: "discoverbase": 0 2024-04-23T10:32:04.049 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-23T10:32:04.049 INFO:teuthology.orchestra.run.smithi114.stdout: "nref": 4 2024-04-23T10:32:04.049 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-23T10:32:04.049 INFO:teuthology.orchestra.run.smithi114.stdout: "object_string": "[inode 0x10000000225 [...7,head] /volumes/qa/sv_1/b206794e-7622-4ab8-8943-f3ae4e65b7a3/client.0/tmp/clients/client0/~dmtmp/PM/ rep@0.1 v45574 f(v2926 m2024-04-23T10:31:17.764716+0000 3=3+0) n(v17 rc2024-04-23T10:31:17.764716+0000 b780288 4=3+1)/n(v0 rc2024-04-23T10:27:52.071395+0000 2=1+1) (inest mix) (iquiesce lock x=1 by request(mds.2:123 nref=3)) caps={15312=pAsLsXsFs/pAsLsXsFsxcral@20013} | dirtyscattered=0 request=1 lock=1 importing=0 dirfrag=1 caps=1 dirwaiter=0 waiter=0 discoverbase=0 0x55c68bbfb080]", 2024-04-23T10:32:04.049 INFO:teuthology.orchestra.run.smithi114.stdout: "lock": { 2024-04-23T10:32:04.049 INFO:teuthology.orchestra.run.smithi114.stdout: "gather_set": [], 2024-04-23T10:32:04.049 INFO:teuthology.orchestra.run.smithi114.stdout: "state": "lock", 2024-04-23T10:32:04.049 INFO:teuthology.orchestra.run.smithi114.stdout: "type": "iquiesce", 2024-04-23T10:32:04.049 INFO:teuthology.orchestra.run.smithi114.stdout: "is_leased": false, 2024-04-23T10:32:04.049 INFO:teuthology.orchestra.run.smithi114.stdout: "num_rdlocks": 0, 2024-04-23T10:32:04.049 INFO:teuthology.orchestra.run.smithi114.stdout: "num_wrlocks": 0, 2024-04-23T10:32:04.049 INFO:teuthology.orchestra.run.smithi114.stdout: "num_xlocks": 1, 2024-04-23T10:32:04.049 INFO:teuthology.orchestra.run.smithi114.stdout: "xlock_by": { 2024-04-23T10:32:04.049 INFO:teuthology.orchestra.run.smithi114.stdout: "reqid": { 2024-04-23T10:32:04.050 INFO:teuthology.orchestra.run.smithi114.stdout: "entity": { 2024-04-23T10:32:04.050 INFO:teuthology.orchestra.run.smithi114.stdout: "type": "mds", 2024-04-23T10:32:04.050 INFO:teuthology.orchestra.run.smithi114.stdout: "num": 2 2024-04-23T10:32:04.050 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-23T10:32:04.050 INFO:teuthology.orchestra.run.smithi114.stdout: "tid": 123 2024-04-23T10:32:04.050 INFO:teuthology.orchestra.run.smithi114.stdout: } 2024-04-23T10:32:04.050 INFO:teuthology.orchestra.run.smithi114.stdout: } 2024-04-23T10:32:04.050 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-23T10:32:04.050 INFO:teuthology.orchestra.run.smithi114.stdout: "flags": 4, 2024-04-23T10:32:04.050 INFO:teuthology.orchestra.run.smithi114.stdout: "wrlock_target": -1 2024-04-23T10:32:04.050 INFO:teuthology.orchestra.run.smithi114.stdout: } 2024-04-23T10:32:04.050 INFO:teuthology.orchestra.run.smithi114.stdout: ] 2024-04-23T10:32:04.050 INFO:teuthology.orchestra.run.smithi114.stdout: } 2024-04-23T10:32:04.051 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-23T10:32:04.051 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-04-23T10:32:04.051 INFO:teuthology.orchestra.run.smithi114.stdout: "description": "internal op quiesce_inode:mds.2:144 fp=#0x10000000211", 2024-04-23T10:32:04.051 INFO:teuthology.orchestra.run.smithi114.stdout: "initiated_at": "2024-04-23T10:31:21.498664+0000", 2024-04-23T10:32:04.051 INFO:teuthology.orchestra.run.smithi114.stdout: "age": 42.330080453999997, 2024-04-23T10:32:04.051 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 42.332019819999999, 2024-04-23T10:32:04.051 INFO:teuthology.orchestra.run.smithi114.stdout: "continuous": true, 2024-04-23T10:32:04.051 INFO:teuthology.orchestra.run.smithi114.stdout: "type_data": { 2024-04-23T10:32:04.051 INFO:teuthology.orchestra.run.smithi114.stdout: "result": -2147483648, 2024-04-23T10:32:04.051 INFO:teuthology.orchestra.run.smithi114.stdout: "flag_point": "quiesce complete", 2024-04-23T10:32:04.051 INFO:teuthology.orchestra.run.smithi114.stdout: "reqid": { 2024-04-23T10:32:04.051 INFO:teuthology.orchestra.run.smithi114.stdout: "entity": { 2024-04-23T10:32:04.051 INFO:teuthology.orchestra.run.smithi114.stdout: "type": "mds", 2024-04-23T10:32:04.051 INFO:teuthology.orchestra.run.smithi114.stdout: "num": 2 2024-04-23T10:32:04.051 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-23T10:32:04.052 INFO:teuthology.orchestra.run.smithi114.stdout: "tid": 144 2024-04-23T10:32:04.052 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-23T10:32:04.052 INFO:teuthology.orchestra.run.smithi114.stdout: "op_type": "internal_op", 2024-04-23T10:32:04.052 INFO:teuthology.orchestra.run.smithi114.stdout: "internal_op": 5385, 2024-04-23T10:32:04.052 INFO:teuthology.orchestra.run.smithi114.stdout: "op_name": "quiesce_inode", 2024-04-23T10:32:04.052 INFO:teuthology.orchestra.run.smithi114.stdout: "events": [ 2024-04-23T10:32:04.052 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-04-23T10:32:04.052 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-04-23T10:31:21.498664+0000", 2024-04-23T10:32:04.052 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "initiated" 2024-04-23T10:32:04.052 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-23T10:32:04.052 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-04-23T10:32:04.052 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-04-23T10:31:21.498664+0000", 2024-04-23T10:32:04.052 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "header_read" 2024-04-23T10:32:04.052 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-23T10:32:04.053 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-04-23T10:32:04.053 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-04-23T10:31:21.498664+0000", 2024-04-23T10:32:04.053 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "throttled" 2024-04-23T10:32:04.053 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-23T10:32:04.053 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-04-23T10:32:04.053 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-04-23T10:31:21.498664+0000", 2024-04-23T10:32:04.053 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "all_read" 2024-04-23T10:32:04.053 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-23T10:32:04.053 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-04-23T10:32:04.053 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-04-23T10:31:21.498664+0000", 2024-04-23T10:32:04.053 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "dispatched" 2024-04-23T10:32:04.053 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-23T10:32:04.053 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-04-23T10:32:04.053 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-04-23T10:31:21.499221+0000", 2024-04-23T10:32:04.054 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "failed to xlock, waiting" 2024-04-23T10:32:04.054 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-23T10:32:04.054 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-04-23T10:32:04.054 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-04-23T10:31:21.502033+0000", 2024-04-23T10:32:04.054 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "failed to xlock, waiting" 2024-04-23T10:32:04.054 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-23T10:32:04.054 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-04-23T10:32:04.054 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-04-23T10:31:21.507099+0000", 2024-04-23T10:32:04.054 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "acquired locks" 2024-04-23T10:32:04.054 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-23T10:32:04.054 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-04-23T10:32:04.054 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-04-23T10:31:21.507266+0000", 2024-04-23T10:32:04.054 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "quiesce complete" 2024-04-23T10:32:04.054 INFO:teuthology.orchestra.run.smithi114.stdout: } 2024-04-23T10:32:04.054 INFO:teuthology.orchestra.run.smithi114.stdout: ], 2024-04-23T10:32:04.055 INFO:teuthology.orchestra.run.smithi114.stdout: "locks": [ 2024-04-23T10:32:04.055 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-04-23T10:32:04.055 INFO:teuthology.orchestra.run.smithi114.stdout: "object": { 2024-04-23T10:32:04.055 INFO:teuthology.orchestra.run.smithi114.stdout: "is_auth": true, 2024-04-23T10:32:04.055 INFO:teuthology.orchestra.run.smithi114.stdout: "auth_state": { 2024-04-23T10:32:04.055 INFO:teuthology.orchestra.run.smithi114.stdout: "replicas": {} 2024-04-23T10:32:04.055 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-23T10:32:04.055 INFO:teuthology.orchestra.run.smithi114.stdout: "replica_state": { 2024-04-23T10:32:04.055 INFO:teuthology.orchestra.run.smithi114.stdout: "authority": [ 2024-04-23T10:32:04.055 INFO:teuthology.orchestra.run.smithi114.stdout: 2, 2024-04-23T10:32:04.055 INFO:teuthology.orchestra.run.smithi114.stdout: -2 2024-04-23T10:32:04.055 INFO:teuthology.orchestra.run.smithi114.stdout: ], 2024-04-23T10:32:04.055 INFO:teuthology.orchestra.run.smithi114.stdout: "replica_nonce": 0 2024-04-23T10:32:04.055 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-23T10:32:04.056 INFO:teuthology.orchestra.run.smithi114.stdout: "auth_pins": 1, 2024-04-23T10:32:04.056 INFO:teuthology.orchestra.run.smithi114.stdout: "is_frozen": false, 2024-04-23T10:32:04.056 INFO:teuthology.orchestra.run.smithi114.stdout: "is_freezing": false, 2024-04-23T10:32:04.056 INFO:teuthology.orchestra.run.smithi114.stdout: "pins": { 2024-04-23T10:32:04.056 INFO:teuthology.orchestra.run.smithi114.stdout: "ptrwaiter": 0, 2024-04-23T10:32:04.056 INFO:teuthology.orchestra.run.smithi114.stdout: "request": 1, 2024-04-23T10:32:04.056 INFO:teuthology.orchestra.run.smithi114.stdout: "lock": 1, 2024-04-23T10:32:04.056 INFO:teuthology.orchestra.run.smithi114.stdout: "importingcaps": 0, 2024-04-23T10:32:04.056 INFO:teuthology.orchestra.run.smithi114.stdout: "caps": 0, 2024-04-23T10:32:04.056 INFO:teuthology.orchestra.run.smithi114.stdout: "replicated": 0, 2024-04-23T10:32:04.056 INFO:teuthology.orchestra.run.smithi114.stdout: "dirty": 1, 2024-04-23T10:32:04.056 INFO:teuthology.orchestra.run.smithi114.stdout: "waiter": 0, 2024-04-23T10:32:04.056 INFO:teuthology.orchestra.run.smithi114.stdout: "authpin": 1 2024-04-23T10:32:04.056 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-23T10:32:04.056 INFO:teuthology.orchestra.run.smithi114.stdout: "nref": 4 2024-04-23T10:32:04.057 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-23T10:32:04.057 INFO:teuthology.orchestra.run.smithi114.stdout: "object_string": "[inode 0x10000000211 [4,head] /volumes/qa/sv_1/b206794e-7622-4ab8-8943-f3ae4e65b7a3/client.0/tmp/clients/client0/~dmtmp/PWRPNT/ZD16.BMP auth v12262 ap=1 s=260096 n(v0 rc2024-04-23T10:27:51.983397+0000 b260096 1=1+0) (iquiesce lock x=1 by request(mds.2:144 nref=3)) | ptrwaiter=0 request=1 lock=1 importingcaps=0 caps=0 replicated=0 dirty=1 waiter=0 authpin=1 0x55c68bc7e580]", 2024-04-23T10:32:04.057 INFO:teuthology.orchestra.run.smithi114.stdout: "lock": { 2024-04-23T10:32:04.057 INFO:teuthology.orchestra.run.smithi114.stdout: "gather_set": [], 2024-04-23T10:32:04.057 INFO:teuthology.orchestra.run.smithi114.stdout: "state": "lock", 2024-04-23T10:32:04.057 INFO:teuthology.orchestra.run.smithi114.stdout: "type": "iquiesce", 2024-04-23T10:32:04.057 INFO:teuthology.orchestra.run.smithi114.stdout: "is_leased": false, 2024-04-23T10:32:04.057 INFO:teuthology.orchestra.run.smithi114.stdout: "num_rdlocks": 0, 2024-04-23T10:32:04.057 INFO:teuthology.orchestra.run.smithi114.stdout: "num_wrlocks": 0, 2024-04-23T10:32:04.057 INFO:teuthology.orchestra.run.smithi114.stdout: "num_xlocks": 1, 2024-04-23T10:32:04.057 INFO:teuthology.orchestra.run.smithi114.stdout: "xlock_by": { 2024-04-23T10:32:04.057 INFO:teuthology.orchestra.run.smithi114.stdout: "reqid": { 2024-04-23T10:32:04.057 INFO:teuthology.orchestra.run.smithi114.stdout: "entity": { 2024-04-23T10:32:04.057 INFO:teuthology.orchestra.run.smithi114.stdout: "type": "mds", 2024-04-23T10:32:04.057 INFO:teuthology.orchestra.run.smithi114.stdout: "num": 2 2024-04-23T10:32:04.058 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-23T10:32:04.058 INFO:teuthology.orchestra.run.smithi114.stdout: "tid": 144 2024-04-23T10:32:04.058 INFO:teuthology.orchestra.run.smithi114.stdout: } 2024-04-23T10:32:04.058 INFO:teuthology.orchestra.run.smithi114.stdout: } 2024-04-23T10:32:04.058 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-23T10:32:04.058 INFO:teuthology.orchestra.run.smithi114.stdout: "flags": 4, 2024-04-23T10:32:04.058 INFO:teuthology.orchestra.run.smithi114.stdout: "wrlock_target": -1 2024-04-23T10:32:04.058 INFO:teuthology.orchestra.run.smithi114.stdout: } 2024-04-23T10:32:04.058 INFO:teuthology.orchestra.run.smithi114.stdout: ] 2024-04-23T10:32:04.058 INFO:teuthology.orchestra.run.smithi114.stdout: } 2024-04-23T10:32:04.058 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-23T10:32:04.058 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-04-23T10:32:04.058 INFO:teuthology.orchestra.run.smithi114.stdout: "description": "internal op quiesce_inode:mds.2:124 fp=#0x1000000020f", 2024-04-23T10:32:04.058 INFO:teuthology.orchestra.run.smithi114.stdout: "initiated_at": "2024-04-23T10:31:21.463203+0000", 2024-04-23T10:32:04.059 INFO:teuthology.orchestra.run.smithi114.stdout: "age": 42.365541929000003, 2024-04-23T10:32:04.059 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 42.367729769, 2024-04-23T10:32:04.059 INFO:teuthology.orchestra.run.smithi114.stdout: "continuous": true, 2024-04-23T10:32:04.059 INFO:teuthology.orchestra.run.smithi114.stdout: "type_data": { 2024-04-23T10:32:04.059 INFO:teuthology.orchestra.run.smithi114.stdout: "result": -2147483648, 2024-04-23T10:32:04.059 INFO:teuthology.orchestra.run.smithi114.stdout: "flag_point": "quiesce complete for non-auth inode", 2024-04-23T10:32:04.059 INFO:teuthology.orchestra.run.smithi114.stdout: "reqid": { 2024-04-23T10:32:04.059 INFO:teuthology.orchestra.run.smithi114.stdout: "entity": { 2024-04-23T10:32:04.059 INFO:teuthology.orchestra.run.smithi114.stdout: "type": "mds", 2024-04-23T10:32:04.059 INFO:teuthology.orchestra.run.smithi114.stdout: "num": 2 2024-04-23T10:32:04.059 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-23T10:32:04.059 INFO:teuthology.orchestra.run.smithi114.stdout: "tid": 124 2024-04-23T10:32:04.061 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-23T10:32:04.061 INFO:teuthology.orchestra.run.smithi114.stdout: "op_type": "internal_op", 2024-04-23T10:32:04.061 INFO:teuthology.orchestra.run.smithi114.stdout: "internal_op": 5385, 2024-04-23T10:32:04.061 INFO:teuthology.orchestra.run.smithi114.stdout: "op_name": "quiesce_inode", 2024-04-23T10:32:04.061 INFO:teuthology.orchestra.run.smithi114.stdout: "events": [ 2024-04-23T10:32:04.061 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-04-23T10:32:04.061 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-04-23T10:31:21.463203+0000", 2024-04-23T10:32:04.061 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "initiated" 2024-04-23T10:32:04.061 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-23T10:32:04.061 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-04-23T10:32:04.061 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-04-23T10:31:21.463203+0000", 2024-04-23T10:32:04.061 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "header_read" 2024-04-23T10:32:04.062 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-23T10:32:04.062 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-04-23T10:32:04.062 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-04-23T10:31:21.463203+0000", 2024-04-23T10:32:04.062 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "throttled" 2024-04-23T10:32:04.062 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-23T10:32:04.062 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-04-23T10:32:04.062 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-04-23T10:31:21.463203+0000", 2024-04-23T10:32:04.062 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "all_read" 2024-04-23T10:32:04.062 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-23T10:32:04.062 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-04-23T10:32:04.062 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-04-23T10:31:21.463203+0000", 2024-04-23T10:32:04.062 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "dispatched" 2024-04-23T10:32:04.062 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-23T10:32:04.062 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-04-23T10:32:04.063 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-04-23T10:31:21.463869+0000", 2024-04-23T10:32:04.063 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "requesting remote authpins" 2024-04-23T10:32:04.063 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-23T10:32:04.063 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-04-23T10:32:04.063 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-04-23T10:31:21.498627+0000", 2024-04-23T10:32:04.063 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "acquired locks" 2024-04-23T10:32:04.063 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-23T10:32:04.063 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-04-23T10:32:04.063 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-04-23T10:31:21.508694+0000", 2024-04-23T10:32:04.063 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "quiesce complete for non-auth inode" 2024-04-23T10:32:04.063 INFO:teuthology.orchestra.run.smithi114.stdout: } 2024-04-23T10:32:04.063 INFO:teuthology.orchestra.run.smithi114.stdout: ], 2024-04-23T10:32:04.063 INFO:teuthology.orchestra.run.smithi114.stdout: "locks": [ 2024-04-23T10:32:04.063 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-04-23T10:32:04.063 INFO:teuthology.orchestra.run.smithi114.stdout: "object": { 2024-04-23T10:32:04.063 INFO:teuthology.orchestra.run.smithi114.stdout: "is_auth": false, 2024-04-23T10:32:04.064 INFO:teuthology.orchestra.run.smithi114.stdout: "auth_state": { 2024-04-23T10:32:04.064 INFO:teuthology.orchestra.run.smithi114.stdout: "replicas": {} 2024-04-23T10:32:04.064 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-23T10:32:04.064 INFO:teuthology.orchestra.run.smithi114.stdout: "replica_state": { 2024-04-23T10:32:04.064 INFO:teuthology.orchestra.run.smithi114.stdout: "authority": [ 2024-04-23T10:32:04.064 INFO:teuthology.orchestra.run.smithi114.stdout: 0, 2024-04-23T10:32:04.064 INFO:teuthology.orchestra.run.smithi114.stdout: -2 2024-04-23T10:32:04.064 INFO:teuthology.orchestra.run.smithi114.stdout: ], 2024-04-23T10:32:04.064 INFO:teuthology.orchestra.run.smithi114.stdout: "replica_nonce": 1 2024-04-23T10:32:04.064 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-23T10:32:04.064 INFO:teuthology.orchestra.run.smithi114.stdout: "auth_pins": 0, 2024-04-23T10:32:04.064 INFO:teuthology.orchestra.run.smithi114.stdout: "is_frozen": false, 2024-04-23T10:32:04.064 INFO:teuthology.orchestra.run.smithi114.stdout: "is_freezing": false, 2024-04-23T10:32:04.064 INFO:teuthology.orchestra.run.smithi114.stdout: "pins": { 2024-04-23T10:32:04.064 INFO:teuthology.orchestra.run.smithi114.stdout: "dirtyscattered": 0, 2024-04-23T10:32:04.065 INFO:teuthology.orchestra.run.smithi114.stdout: "request": 1, 2024-04-23T10:32:04.065 INFO:teuthology.orchestra.run.smithi114.stdout: "lock": 1, 2024-04-23T10:32:04.065 INFO:teuthology.orchestra.run.smithi114.stdout: "importing": 0, 2024-04-23T10:32:04.065 INFO:teuthology.orchestra.run.smithi114.stdout: "dirfrag": 1, 2024-04-23T10:32:04.065 INFO:teuthology.orchestra.run.smithi114.stdout: "caps": 1, 2024-04-23T10:32:04.065 INFO:teuthology.orchestra.run.smithi114.stdout: "dirwaiter": 0, 2024-04-23T10:32:04.065 INFO:teuthology.orchestra.run.smithi114.stdout: "waiter": 0, 2024-04-23T10:32:04.065 INFO:teuthology.orchestra.run.smithi114.stdout: "discoverbase": 0 2024-04-23T10:32:04.065 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-23T10:32:04.065 INFO:teuthology.orchestra.run.smithi114.stdout: "nref": 4 2024-04-23T10:32:04.065 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-23T10:32:04.065 INFO:teuthology.orchestra.run.smithi114.stdout: "object_string": "[inode 0x1000000020f [...7,head] /volumes/qa/sv_1/b206794e-7622-4ab8-8943-f3ae4e65b7a3/client.0/tmp/clients/client0/~dmtmp/PWRPNT/ rep@0.1 v45572 f(v1293 m2024-04-23T10:31:17.718717+0000 5=5+0) n(v18 rc2024-04-23T10:31:17.718717+0000 b1300480 6=5+1)/n(v0 rc2024-04-23T10:27:51.976397+0000 2=1+1) (inest mix) (iquiesce lock x=1 by request(mds.2:124 nref=3)) caps={15312=pAsLsXsFs/pAsLsXsFsxcral@6268} | dirtyscattered=0 request=1 lock=1 importing=0 dirfrag=1 caps=1 dirwaiter=0 waiter=0 discoverbase=0 0x55c68bbfa000]", 2024-04-23T10:32:04.065 INFO:teuthology.orchestra.run.smithi114.stdout: "lock": { 2024-04-23T10:32:04.065 INFO:teuthology.orchestra.run.smithi114.stdout: "gather_set": [], 2024-04-23T10:32:04.065 INFO:teuthology.orchestra.run.smithi114.stdout: "state": "lock", 2024-04-23T10:32:04.065 INFO:teuthology.orchestra.run.smithi114.stdout: "type": "iquiesce", 2024-04-23T10:32:04.065 INFO:teuthology.orchestra.run.smithi114.stdout: "is_leased": false, 2024-04-23T10:32:04.066 INFO:teuthology.orchestra.run.smithi114.stdout: "num_rdlocks": 0, 2024-04-23T10:32:04.066 INFO:teuthology.orchestra.run.smithi114.stdout: "num_wrlocks": 0, 2024-04-23T10:32:04.066 INFO:teuthology.orchestra.run.smithi114.stdout: "num_xlocks": 1, 2024-04-23T10:32:04.066 INFO:teuthology.orchestra.run.smithi114.stdout: "xlock_by": { 2024-04-23T10:32:04.066 INFO:teuthology.orchestra.run.smithi114.stdout: "reqid": { 2024-04-23T10:32:04.066 INFO:teuthology.orchestra.run.smithi114.stdout: "entity": { 2024-04-23T10:32:04.066 INFO:teuthology.orchestra.run.smithi114.stdout: "type": "mds", 2024-04-23T10:32:04.066 INFO:teuthology.orchestra.run.smithi114.stdout: "num": 2 2024-04-23T10:32:04.066 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-23T10:32:04.066 INFO:teuthology.orchestra.run.smithi114.stdout: "tid": 124 2024-04-23T10:32:04.066 INFO:teuthology.orchestra.run.smithi114.stdout: } 2024-04-23T10:32:04.066 INFO:teuthology.orchestra.run.smithi114.stdout: } 2024-04-23T10:32:04.066 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-23T10:32:04.066 INFO:teuthology.orchestra.run.smithi114.stdout: "flags": 4, 2024-04-23T10:32:04.066 INFO:teuthology.orchestra.run.smithi114.stdout: "wrlock_target": -1 2024-04-23T10:32:04.067 INFO:teuthology.orchestra.run.smithi114.stdout: } 2024-04-23T10:32:04.067 INFO:teuthology.orchestra.run.smithi114.stdout: ] 2024-04-23T10:32:04.067 INFO:teuthology.orchestra.run.smithi114.stdout: } 2024-04-23T10:32:04.067 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-23T10:32:04.067 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-04-23T10:32:04.067 INFO:teuthology.orchestra.run.smithi114.stdout: "description": "internal op quiesce_inode:mds.2:145 fp=#0x10000000214", 2024-04-23T10:32:04.067 INFO:teuthology.orchestra.run.smithi114.stdout: "initiated_at": "2024-04-23T10:31:21.498682+0000", 2024-04-23T10:32:04.067 INFO:teuthology.orchestra.run.smithi114.stdout: "age": 42.330062400000003, 2024-04-23T10:32:04.067 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 42.332320840000001, 2024-04-23T10:32:04.067 INFO:teuthology.orchestra.run.smithi114.stdout: "continuous": true, 2024-04-23T10:32:04.067 INFO:teuthology.orchestra.run.smithi114.stdout: "type_data": { 2024-04-23T10:32:04.067 INFO:teuthology.orchestra.run.smithi114.stdout: "result": -2147483648, 2024-04-23T10:32:04.067 INFO:teuthology.orchestra.run.smithi114.stdout: "flag_point": "quiesce complete", 2024-04-23T10:32:04.067 INFO:teuthology.orchestra.run.smithi114.stdout: "reqid": { 2024-04-23T10:32:04.067 INFO:teuthology.orchestra.run.smithi114.stdout: "entity": { 2024-04-23T10:32:04.067 INFO:teuthology.orchestra.run.smithi114.stdout: "type": "mds", 2024-04-23T10:32:04.067 INFO:teuthology.orchestra.run.smithi114.stdout: "num": 2 2024-04-23T10:32:04.068 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-23T10:32:04.068 INFO:teuthology.orchestra.run.smithi114.stdout: "tid": 145 2024-04-23T10:32:04.068 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-23T10:32:04.068 INFO:teuthology.orchestra.run.smithi114.stdout: "op_type": "internal_op", 2024-04-23T10:32:04.068 INFO:teuthology.orchestra.run.smithi114.stdout: "internal_op": 5385, 2024-04-23T10:32:04.068 INFO:teuthology.orchestra.run.smithi114.stdout: "op_name": "quiesce_inode", 2024-04-23T10:32:04.068 INFO:teuthology.orchestra.run.smithi114.stdout: "events": [ 2024-04-23T10:32:04.068 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-04-23T10:32:04.068 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-04-23T10:31:21.498682+0000", 2024-04-23T10:32:04.068 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "initiated" 2024-04-23T10:32:04.068 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-23T10:32:04.068 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-04-23T10:32:04.068 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-04-23T10:31:21.498682+0000", 2024-04-23T10:32:04.068 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "header_read" 2024-04-23T10:32:04.068 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-23T10:32:04.068 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-04-23T10:32:04.069 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-04-23T10:31:21.498682+0000", 2024-04-23T10:32:04.069 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "throttled" 2024-04-23T10:32:04.069 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-23T10:32:04.069 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-04-23T10:32:04.069 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-04-23T10:31:21.498682+0000", 2024-04-23T10:32:04.069 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "all_read" 2024-04-23T10:32:04.069 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-23T10:32:04.069 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-04-23T10:32:04.069 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-04-23T10:31:21.498682+0000", 2024-04-23T10:32:04.069 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "dispatched" 2024-04-23T10:32:04.069 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-23T10:32:04.069 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-04-23T10:32:04.069 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-04-23T10:31:21.499441+0000", 2024-04-23T10:32:04.069 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "failed to xlock, waiting" 2024-04-23T10:32:04.069 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-23T10:32:04.069 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-04-23T10:32:04.070 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-04-23T10:31:21.502467+0000", 2024-04-23T10:32:04.070 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "acquired locks" 2024-04-23T10:32:04.070 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-23T10:32:04.070 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-04-23T10:32:04.070 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-04-23T10:31:21.502711+0000", 2024-04-23T10:32:04.070 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "quiesce complete" 2024-04-23T10:32:04.070 INFO:teuthology.orchestra.run.smithi114.stdout: } 2024-04-23T10:32:04.070 INFO:teuthology.orchestra.run.smithi114.stdout: ], 2024-04-23T10:32:04.070 INFO:teuthology.orchestra.run.smithi114.stdout: "locks": [ 2024-04-23T10:32:04.070 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-04-23T10:32:04.070 INFO:teuthology.orchestra.run.smithi114.stdout: "object": { 2024-04-23T10:32:04.070 INFO:teuthology.orchestra.run.smithi114.stdout: "is_auth": true, 2024-04-23T10:32:04.070 INFO:teuthology.orchestra.run.smithi114.stdout: "auth_state": { 2024-04-23T10:32:04.070 INFO:teuthology.orchestra.run.smithi114.stdout: "replicas": {} 2024-04-23T10:32:04.070 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-23T10:32:04.070 INFO:teuthology.orchestra.run.smithi114.stdout: "replica_state": { 2024-04-23T10:32:04.071 INFO:teuthology.orchestra.run.smithi114.stdout: "authority": [ 2024-04-23T10:32:04.071 INFO:teuthology.orchestra.run.smithi114.stdout: 2, 2024-04-23T10:32:04.071 INFO:teuthology.orchestra.run.smithi114.stdout: -2 2024-04-23T10:32:04.071 INFO:teuthology.orchestra.run.smithi114.stdout: ], 2024-04-23T10:32:04.071 INFO:teuthology.orchestra.run.smithi114.stdout: "replica_nonce": 0 2024-04-23T10:32:04.071 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-23T10:32:04.071 INFO:teuthology.orchestra.run.smithi114.stdout: "auth_pins": 1, 2024-04-23T10:32:04.071 INFO:teuthology.orchestra.run.smithi114.stdout: "is_frozen": false, 2024-04-23T10:32:04.071 INFO:teuthology.orchestra.run.smithi114.stdout: "is_freezing": false, 2024-04-23T10:32:04.071 INFO:teuthology.orchestra.run.smithi114.stdout: "pins": { 2024-04-23T10:32:04.071 INFO:teuthology.orchestra.run.smithi114.stdout: "ptrwaiter": 0, 2024-04-23T10:32:04.071 INFO:teuthology.orchestra.run.smithi114.stdout: "request": 1, 2024-04-23T10:32:04.071 INFO:teuthology.orchestra.run.smithi114.stdout: "lock": 1, 2024-04-23T10:32:04.071 INFO:teuthology.orchestra.run.smithi114.stdout: "importingcaps": 0, 2024-04-23T10:32:04.071 INFO:teuthology.orchestra.run.smithi114.stdout: "caps": 0, 2024-04-23T10:32:04.072 INFO:teuthology.orchestra.run.smithi114.stdout: "needsnapflush": 0, 2024-04-23T10:32:04.072 INFO:teuthology.orchestra.run.smithi114.stdout: "replicated": 0, 2024-04-23T10:32:04.072 INFO:teuthology.orchestra.run.smithi114.stdout: "dirty": 1, 2024-04-23T10:32:04.072 INFO:teuthology.orchestra.run.smithi114.stdout: "waiter": 0, 2024-04-23T10:32:04.072 INFO:teuthology.orchestra.run.smithi114.stdout: "authpin": 1 2024-04-23T10:32:04.072 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-23T10:32:04.072 INFO:teuthology.orchestra.run.smithi114.stdout: "nref": 4 2024-04-23T10:32:04.072 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-23T10:32:04.072 INFO:teuthology.orchestra.run.smithi114.stdout: "object_string": "[inode 0x10000000214 [7,head] /volumes/qa/sv_1/b206794e-7622-4ab8-8943-f3ae4e65b7a3/client.0/tmp/clients/client0/~dmtmp/PWRPNT/TRIDOTS.POT auth v12245 ap=1 s=260096 n(v0 rc2024-04-23T10:27:52.001397+0000 b260096 1=1+0) (iquiesce lock x=1 by request(mds.2:145 nref=3)) | ptrwaiter=0 request=1 lock=1 importingcaps=0 caps=0 needsnapflush=0 replicated=0 dirty=1 waiter=0 authpin=1 0x55c68bc7f080]", 2024-04-23T10:32:04.072 INFO:teuthology.orchestra.run.smithi114.stdout: "lock": { 2024-04-23T10:32:04.072 INFO:teuthology.orchestra.run.smithi114.stdout: "gather_set": [], 2024-04-23T10:32:04.072 INFO:teuthology.orchestra.run.smithi114.stdout: "state": "lock", 2024-04-23T10:32:04.072 INFO:teuthology.orchestra.run.smithi114.stdout: "type": "iquiesce", 2024-04-23T10:32:04.072 INFO:teuthology.orchestra.run.smithi114.stdout: "is_leased": false, 2024-04-23T10:32:04.072 INFO:teuthology.orchestra.run.smithi114.stdout: "num_rdlocks": 0, 2024-04-23T10:32:04.073 INFO:teuthology.orchestra.run.smithi114.stdout: "num_wrlocks": 0, 2024-04-23T10:32:04.073 INFO:teuthology.orchestra.run.smithi114.stdout: "num_xlocks": 1, 2024-04-23T10:32:04.073 INFO:teuthology.orchestra.run.smithi114.stdout: "xlock_by": { 2024-04-23T10:32:04.073 INFO:teuthology.orchestra.run.smithi114.stdout: "reqid": { 2024-04-23T10:32:04.073 INFO:teuthology.orchestra.run.smithi114.stdout: "entity": { 2024-04-23T10:32:04.073 INFO:teuthology.orchestra.run.smithi114.stdout: "type": "mds", 2024-04-23T10:32:04.073 INFO:teuthology.orchestra.run.smithi114.stdout: "num": 2 2024-04-23T10:32:04.073 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-23T10:32:04.073 INFO:teuthology.orchestra.run.smithi114.stdout: "tid": 145 2024-04-23T10:32:04.073 INFO:teuthology.orchestra.run.smithi114.stdout: } 2024-04-23T10:32:04.073 INFO:teuthology.orchestra.run.smithi114.stdout: } 2024-04-23T10:32:04.073 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-23T10:32:04.073 INFO:teuthology.orchestra.run.smithi114.stdout: "flags": 4, 2024-04-23T10:32:04.073 INFO:teuthology.orchestra.run.smithi114.stdout: "wrlock_target": -1 2024-04-23T10:32:04.074 INFO:teuthology.orchestra.run.smithi114.stdout: } 2024-04-23T10:32:04.074 INFO:teuthology.orchestra.run.smithi114.stdout: ] 2024-04-23T10:32:04.074 INFO:teuthology.orchestra.run.smithi114.stdout: } 2024-04-23T10:32:04.074 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-23T10:32:04.074 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-04-23T10:32:04.074 INFO:teuthology.orchestra.run.smithi114.stdout: "description": "internal op quiesce_inode:mds.2:125 fp=#0x1000000023e", 2024-04-23T10:32:04.074 INFO:teuthology.orchestra.run.smithi114.stdout: "initiated_at": "2024-04-23T10:31:21.463216+0000", 2024-04-23T10:32:04.074 INFO:teuthology.orchestra.run.smithi114.stdout: "age": 42.365528842000003, 2024-04-23T10:32:04.074 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 42.367851899999998, 2024-04-23T10:32:04.074 INFO:teuthology.orchestra.run.smithi114.stdout: "continuous": true, 2024-04-23T10:32:04.074 INFO:teuthology.orchestra.run.smithi114.stdout: "type_data": { 2024-04-23T10:32:04.074 INFO:teuthology.orchestra.run.smithi114.stdout: "result": -2147483648, 2024-04-23T10:32:04.074 INFO:teuthology.orchestra.run.smithi114.stdout: "flag_point": "quiesce complete for non-auth inode", 2024-04-23T10:32:04.074 INFO:teuthology.orchestra.run.smithi114.stdout: "reqid": { 2024-04-23T10:32:04.074 INFO:teuthology.orchestra.run.smithi114.stdout: "entity": { 2024-04-23T10:32:04.075 INFO:teuthology.orchestra.run.smithi114.stdout: "type": "mds", 2024-04-23T10:32:04.075 INFO:teuthology.orchestra.run.smithi114.stdout: "num": 2 2024-04-23T10:32:04.075 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-23T10:32:04.075 INFO:teuthology.orchestra.run.smithi114.stdout: "tid": 125 2024-04-23T10:32:04.075 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-23T10:32:04.075 INFO:teuthology.orchestra.run.smithi114.stdout: "op_type": "internal_op", 2024-04-23T10:32:04.075 INFO:teuthology.orchestra.run.smithi114.stdout: "internal_op": 5385, 2024-04-23T10:32:04.075 INFO:teuthology.orchestra.run.smithi114.stdout: "op_name": "quiesce_inode", 2024-04-23T10:32:04.075 INFO:teuthology.orchestra.run.smithi114.stdout: "events": [ 2024-04-23T10:32:04.075 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-04-23T10:32:04.075 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-04-23T10:31:21.463216+0000", 2024-04-23T10:32:04.075 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "initiated" 2024-04-23T10:32:04.075 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-23T10:32:04.075 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-04-23T10:32:04.076 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-04-23T10:31:21.463216+0000", 2024-04-23T10:32:04.076 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "header_read" 2024-04-23T10:32:04.076 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-23T10:32:04.076 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-04-23T10:32:04.076 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-04-23T10:31:21.463216+0000", 2024-04-23T10:32:04.076 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "throttled" 2024-04-23T10:32:04.076 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-23T10:32:04.076 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-04-23T10:32:04.076 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-04-23T10:31:21.463216+0000", 2024-04-23T10:32:04.076 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "all_read" 2024-04-23T10:32:04.076 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-23T10:32:04.076 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-04-23T10:32:04.076 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-04-23T10:31:21.463216+0000", 2024-04-23T10:32:04.076 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "dispatched" 2024-04-23T10:32:04.076 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-23T10:32:04.077 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-04-23T10:32:04.077 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-04-23T10:31:21.463936+0000", 2024-04-23T10:32:04.077 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "requesting remote authpins" 2024-04-23T10:32:04.077 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-23T10:32:04.077 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-04-23T10:32:04.077 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-04-23T10:31:21.501193+0000", 2024-04-23T10:32:04.077 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "acquired locks" 2024-04-23T10:32:04.077 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-23T10:32:04.077 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-04-23T10:32:04.077 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-04-23T10:31:21.501205+0000", 2024-04-23T10:32:04.077 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "quiesce complete for non-auth inode" 2024-04-23T10:32:04.077 INFO:teuthology.orchestra.run.smithi114.stdout: } 2024-04-23T10:32:04.077 INFO:teuthology.orchestra.run.smithi114.stdout: ], 2024-04-23T10:32:04.077 INFO:teuthology.orchestra.run.smithi114.stdout: "locks": [ 2024-04-23T10:32:04.078 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-04-23T10:32:04.078 INFO:teuthology.orchestra.run.smithi114.stdout: "object": { 2024-04-23T10:32:04.078 INFO:teuthology.orchestra.run.smithi114.stdout: "is_auth": false, 2024-04-23T10:32:04.078 INFO:teuthology.orchestra.run.smithi114.stdout: "auth_state": { 2024-04-23T10:32:04.078 INFO:teuthology.orchestra.run.smithi114.stdout: "replicas": {} 2024-04-23T10:32:04.078 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-23T10:32:04.078 INFO:teuthology.orchestra.run.smithi114.stdout: "replica_state": { 2024-04-23T10:32:04.078 INFO:teuthology.orchestra.run.smithi114.stdout: "authority": [ 2024-04-23T10:32:04.078 INFO:teuthology.orchestra.run.smithi114.stdout: 0, 2024-04-23T10:32:04.078 INFO:teuthology.orchestra.run.smithi114.stdout: -2 2024-04-23T10:32:04.078 INFO:teuthology.orchestra.run.smithi114.stdout: ], 2024-04-23T10:32:04.078 INFO:teuthology.orchestra.run.smithi114.stdout: "replica_nonce": 1 2024-04-23T10:32:04.078 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-23T10:32:04.078 INFO:teuthology.orchestra.run.smithi114.stdout: "auth_pins": 0, 2024-04-23T10:32:04.079 INFO:teuthology.orchestra.run.smithi114.stdout: "is_frozen": false, 2024-04-23T10:32:04.079 INFO:teuthology.orchestra.run.smithi114.stdout: "is_freezing": false, 2024-04-23T10:32:04.079 INFO:teuthology.orchestra.run.smithi114.stdout: "pins": { 2024-04-23T10:32:04.079 INFO:teuthology.orchestra.run.smithi114.stdout: "request": 1, 2024-04-23T10:32:04.079 INFO:teuthology.orchestra.run.smithi114.stdout: "lock": 1, 2024-04-23T10:32:04.079 INFO:teuthology.orchestra.run.smithi114.stdout: "dirwaiter": 0, 2024-04-23T10:32:04.079 INFO:teuthology.orchestra.run.smithi114.stdout: "discoverbase": 0 2024-04-23T10:32:04.079 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-23T10:32:04.079 INFO:teuthology.orchestra.run.smithi114.stdout: "nref": 2 2024-04-23T10:32:04.079 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-23T10:32:04.079 INFO:teuthology.orchestra.run.smithi114.stdout: "object_string": "[inode 0x1000000023e [...7,head] /volumes/qa/sv_1/b206794e-7622-4ab8-8943-f3ae4e65b7a3/client.0/tmp/clients/client0/~dmtmp/ACCESS/ rep@0.1 v45357 f(v0 m2024-04-23T10:31:17.039732+0000 4=4+0) n(v0 rc2024-04-23T10:28:42.888241+0000 b260096 2=1+1) old_inodes=1 (inest mix) (iquiesce lock x=1 by request(mds.2:125 nref=3)) | request=1 lock=1 dirwaiter=0 discoverbase=0 0x55c68c138580]", 2024-04-23T10:32:04.079 INFO:teuthology.orchestra.run.smithi114.stdout: "lock": { 2024-04-23T10:32:04.079 INFO:teuthology.orchestra.run.smithi114.stdout: "gather_set": [], 2024-04-23T10:32:04.079 INFO:teuthology.orchestra.run.smithi114.stdout: "state": "lock", 2024-04-23T10:32:04.079 INFO:teuthology.orchestra.run.smithi114.stdout: "type": "iquiesce", 2024-04-23T10:32:04.080 INFO:teuthology.orchestra.run.smithi114.stdout: "is_leased": false, 2024-04-23T10:32:04.080 INFO:teuthology.orchestra.run.smithi114.stdout: "num_rdlocks": 0, 2024-04-23T10:32:04.080 INFO:teuthology.orchestra.run.smithi114.stdout: "num_wrlocks": 0, 2024-04-23T10:32:04.080 INFO:teuthology.orchestra.run.smithi114.stdout: "num_xlocks": 1, 2024-04-23T10:32:04.080 INFO:teuthology.orchestra.run.smithi114.stdout: "xlock_by": { 2024-04-23T10:32:04.080 INFO:teuthology.orchestra.run.smithi114.stdout: "reqid": { 2024-04-23T10:32:04.080 INFO:teuthology.orchestra.run.smithi114.stdout: "entity": { 2024-04-23T10:32:04.080 INFO:teuthology.orchestra.run.smithi114.stdout: "type": "mds", 2024-04-23T10:32:04.080 INFO:teuthology.orchestra.run.smithi114.stdout: "num": 2 2024-04-23T10:32:04.080 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-23T10:32:04.080 INFO:teuthology.orchestra.run.smithi114.stdout: "tid": 125 2024-04-23T10:32:04.080 INFO:teuthology.orchestra.run.smithi114.stdout: } 2024-04-23T10:32:04.080 INFO:teuthology.orchestra.run.smithi114.stdout: } 2024-04-23T10:32:04.080 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-23T10:32:04.081 INFO:teuthology.orchestra.run.smithi114.stdout: "flags": 4, 2024-04-23T10:32:04.081 INFO:teuthology.orchestra.run.smithi114.stdout: "wrlock_target": -1 2024-04-23T10:32:04.081 INFO:teuthology.orchestra.run.smithi114.stdout: } 2024-04-23T10:32:04.081 INFO:teuthology.orchestra.run.smithi114.stdout: ] 2024-04-23T10:32:04.081 INFO:teuthology.orchestra.run.smithi114.stdout: } 2024-04-23T10:32:04.081 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-23T10:32:04.081 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-04-23T10:32:04.081 INFO:teuthology.orchestra.run.smithi114.stdout: "description": "internal op quiesce_inode:mds.2:146 fp=#0x10000000213", 2024-04-23T10:32:04.081 INFO:teuthology.orchestra.run.smithi114.stdout: "initiated_at": "2024-04-23T10:31:21.498698+0000", 2024-04-23T10:32:04.081 INFO:teuthology.orchestra.run.smithi114.stdout: "age": 42.330046457000002, 2024-04-23T10:32:04.081 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 42.332431438, 2024-04-23T10:32:04.081 INFO:teuthology.orchestra.run.smithi114.stdout: "continuous": true, 2024-04-23T10:32:04.081 INFO:teuthology.orchestra.run.smithi114.stdout: "type_data": { 2024-04-23T10:32:04.081 INFO:teuthology.orchestra.run.smithi114.stdout: "result": -2147483648, 2024-04-23T10:32:04.082 INFO:teuthology.orchestra.run.smithi114.stdout: "flag_point": "quiesce complete", 2024-04-23T10:32:04.082 INFO:teuthology.orchestra.run.smithi114.stdout: "reqid": { 2024-04-23T10:32:04.082 INFO:teuthology.orchestra.run.smithi114.stdout: "entity": { 2024-04-23T10:32:04.082 INFO:teuthology.orchestra.run.smithi114.stdout: "type": "mds", 2024-04-23T10:32:04.082 INFO:teuthology.orchestra.run.smithi114.stdout: "num": 2 2024-04-23T10:32:04.082 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-23T10:32:04.082 INFO:teuthology.orchestra.run.smithi114.stdout: "tid": 146 2024-04-23T10:32:04.082 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-23T10:32:04.082 INFO:teuthology.orchestra.run.smithi114.stdout: "op_type": "internal_op", 2024-04-23T10:32:04.082 INFO:teuthology.orchestra.run.smithi114.stdout: "internal_op": 5385, 2024-04-23T10:32:04.082 INFO:teuthology.orchestra.run.smithi114.stdout: "op_name": "quiesce_inode", 2024-04-23T10:32:04.082 INFO:teuthology.orchestra.run.smithi114.stdout: "events": [ 2024-04-23T10:32:04.082 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-04-23T10:32:04.083 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-04-23T10:31:21.498698+0000", 2024-04-23T10:32:04.083 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "initiated" 2024-04-23T10:32:04.083 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-23T10:32:04.083 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-04-23T10:32:04.083 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-04-23T10:31:21.498698+0000", 2024-04-23T10:32:04.083 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "header_read" 2024-04-23T10:32:04.083 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-23T10:32:04.083 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-04-23T10:32:04.083 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-04-23T10:31:21.498698+0000", 2024-04-23T10:32:04.083 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "throttled" 2024-04-23T10:32:04.083 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-23T10:32:04.083 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-04-23T10:32:04.083 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-04-23T10:31:21.498698+0000", 2024-04-23T10:32:04.083 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "all_read" 2024-04-23T10:32:04.083 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-23T10:32:04.084 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-04-23T10:32:04.084 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-04-23T10:31:21.498698+0000", 2024-04-23T10:32:04.084 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "dispatched" 2024-04-23T10:32:04.084 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-23T10:32:04.084 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-04-23T10:32:04.084 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-04-23T10:31:21.499644+0000", 2024-04-23T10:32:04.084 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "failed to xlock, waiting" 2024-04-23T10:32:04.084 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-23T10:32:04.084 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-04-23T10:32:04.084 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-04-23T10:31:21.503691+0000", 2024-04-23T10:32:04.084 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "failed to xlock, waiting" 2024-04-23T10:32:04.084 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-23T10:32:04.084 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-04-23T10:32:04.084 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-04-23T10:31:21.507743+0000", 2024-04-23T10:32:04.084 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "acquired locks" 2024-04-23T10:32:04.085 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-23T10:32:04.085 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-04-23T10:32:04.085 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-04-23T10:31:21.508049+0000", 2024-04-23T10:32:04.085 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "quiesce complete" 2024-04-23T10:32:04.085 INFO:teuthology.orchestra.run.smithi114.stdout: } 2024-04-23T10:32:04.085 INFO:teuthology.orchestra.run.smithi114.stdout: ], 2024-04-23T10:32:04.085 INFO:teuthology.orchestra.run.smithi114.stdout: "locks": [ 2024-04-23T10:32:04.085 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-04-23T10:32:04.085 INFO:teuthology.orchestra.run.smithi114.stdout: "object": { 2024-04-23T10:32:04.085 INFO:teuthology.orchestra.run.smithi114.stdout: "is_auth": true, 2024-04-23T10:32:04.085 INFO:teuthology.orchestra.run.smithi114.stdout: "auth_state": { 2024-04-23T10:32:04.085 INFO:teuthology.orchestra.run.smithi114.stdout: "replicas": {} 2024-04-23T10:32:04.085 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-23T10:32:04.085 INFO:teuthology.orchestra.run.smithi114.stdout: "replica_state": { 2024-04-23T10:32:04.086 INFO:teuthology.orchestra.run.smithi114.stdout: "authority": [ 2024-04-23T10:32:04.086 INFO:teuthology.orchestra.run.smithi114.stdout: 2, 2024-04-23T10:32:04.086 INFO:teuthology.orchestra.run.smithi114.stdout: -2 2024-04-23T10:32:04.086 INFO:teuthology.orchestra.run.smithi114.stdout: ], 2024-04-23T10:32:04.086 INFO:teuthology.orchestra.run.smithi114.stdout: "replica_nonce": 0 2024-04-23T10:32:04.086 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-23T10:32:04.086 INFO:teuthology.orchestra.run.smithi114.stdout: "auth_pins": 1, 2024-04-23T10:32:04.086 INFO:teuthology.orchestra.run.smithi114.stdout: "is_frozen": false, 2024-04-23T10:32:04.086 INFO:teuthology.orchestra.run.smithi114.stdout: "is_freezing": false, 2024-04-23T10:32:04.086 INFO:teuthology.orchestra.run.smithi114.stdout: "pins": { 2024-04-23T10:32:04.086 INFO:teuthology.orchestra.run.smithi114.stdout: "ptrwaiter": 0, 2024-04-23T10:32:04.086 INFO:teuthology.orchestra.run.smithi114.stdout: "request": 1, 2024-04-23T10:32:04.086 INFO:teuthology.orchestra.run.smithi114.stdout: "lock": 1, 2024-04-23T10:32:04.086 INFO:teuthology.orchestra.run.smithi114.stdout: "importingcaps": 0, 2024-04-23T10:32:04.086 INFO:teuthology.orchestra.run.smithi114.stdout: "caps": 0, 2024-04-23T10:32:04.087 INFO:teuthology.orchestra.run.smithi114.stdout: "needsnapflush": 0, 2024-04-23T10:32:04.087 INFO:teuthology.orchestra.run.smithi114.stdout: "replicated": 0, 2024-04-23T10:32:04.087 INFO:teuthology.orchestra.run.smithi114.stdout: "dirty": 1, 2024-04-23T10:32:04.087 INFO:teuthology.orchestra.run.smithi114.stdout: "waiter": 0, 2024-04-23T10:32:04.087 INFO:teuthology.orchestra.run.smithi114.stdout: "authpin": 1 2024-04-23T10:32:04.087 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-23T10:32:04.087 INFO:teuthology.orchestra.run.smithi114.stdout: "nref": 4 2024-04-23T10:32:04.087 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-23T10:32:04.087 INFO:teuthology.orchestra.run.smithi114.stdout: "object_string": "[inode 0x10000000213 [7,head] /volumes/qa/sv_1/b206794e-7622-4ab8-8943-f3ae4e65b7a3/client.0/tmp/clients/client0/~dmtmp/PWRPNT/TIPS.PPT auth v12268 ap=1 s=260096 n(v0 rc2024-04-23T10:31:16.010756+0000 b260096 1=1+0) (iquiesce lock x=1 by request(mds.2:146 nref=3)) | ptrwaiter=0 request=1 lock=1 importingcaps=0 caps=0 needsnapflush=0 replicated=0 dirty=1 waiter=0 authpin=1 0x55c68bc7f600]", 2024-04-23T10:32:04.087 INFO:teuthology.orchestra.run.smithi114.stdout: "lock": { 2024-04-23T10:32:04.087 INFO:teuthology.orchestra.run.smithi114.stdout: "gather_set": [], 2024-04-23T10:32:04.087 INFO:teuthology.orchestra.run.smithi114.stdout: "state": "lock", 2024-04-23T10:32:04.087 INFO:teuthology.orchestra.run.smithi114.stdout: "type": "iquiesce", 2024-04-23T10:32:04.087 INFO:teuthology.orchestra.run.smithi114.stdout: "is_leased": false, 2024-04-23T10:32:04.088 INFO:teuthology.orchestra.run.smithi114.stdout: "num_rdlocks": 0, 2024-04-23T10:32:04.088 INFO:teuthology.orchestra.run.smithi114.stdout: "num_wrlocks": 0, 2024-04-23T10:32:04.088 INFO:teuthology.orchestra.run.smithi114.stdout: "num_xlocks": 1, 2024-04-23T10:32:04.088 INFO:teuthology.orchestra.run.smithi114.stdout: "xlock_by": { 2024-04-23T10:32:04.088 INFO:teuthology.orchestra.run.smithi114.stdout: "reqid": { 2024-04-23T10:32:04.088 INFO:teuthology.orchestra.run.smithi114.stdout: "entity": { 2024-04-23T10:32:04.088 INFO:teuthology.orchestra.run.smithi114.stdout: "type": "mds", 2024-04-23T10:32:04.088 INFO:teuthology.orchestra.run.smithi114.stdout: "num": 2 2024-04-23T10:32:04.088 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-23T10:32:04.088 INFO:teuthology.orchestra.run.smithi114.stdout: "tid": 146 2024-04-23T10:32:04.088 INFO:teuthology.orchestra.run.smithi114.stdout: } 2024-04-23T10:32:04.088 INFO:teuthology.orchestra.run.smithi114.stdout: } 2024-04-23T10:32:04.088 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-23T10:32:04.088 INFO:teuthology.orchestra.run.smithi114.stdout: "flags": 4, 2024-04-23T10:32:04.088 INFO:teuthology.orchestra.run.smithi114.stdout: "wrlock_target": -1 2024-04-23T10:32:04.088 INFO:teuthology.orchestra.run.smithi114.stdout: } 2024-04-23T10:32:04.089 INFO:teuthology.orchestra.run.smithi114.stdout: ] 2024-04-23T10:32:04.089 INFO:teuthology.orchestra.run.smithi114.stdout: } 2024-04-23T10:32:04.089 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-23T10:32:04.089 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-04-23T10:32:04.089 INFO:teuthology.orchestra.run.smithi114.stdout: "description": "internal op quiesce_inode:mds.2:126 fp=#0x1000000021f", 2024-04-23T10:32:04.089 INFO:teuthology.orchestra.run.smithi114.stdout: "initiated_at": "2024-04-23T10:31:21.477876+0000", 2024-04-23T10:32:04.089 INFO:teuthology.orchestra.run.smithi114.stdout: "age": 42.350869230999997, 2024-04-23T10:32:04.089 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 42.353319935999998, 2024-04-23T10:32:04.089 INFO:teuthology.orchestra.run.smithi114.stdout: "continuous": true, 2024-04-23T10:32:04.089 INFO:teuthology.orchestra.run.smithi114.stdout: "type_data": { 2024-04-23T10:32:04.089 INFO:teuthology.orchestra.run.smithi114.stdout: "result": -2147483648, 2024-04-23T10:32:04.089 INFO:teuthology.orchestra.run.smithi114.stdout: "flag_point": "quiesce complete", 2024-04-23T10:32:04.089 INFO:teuthology.orchestra.run.smithi114.stdout: "reqid": { 2024-04-23T10:32:04.089 INFO:teuthology.orchestra.run.smithi114.stdout: "entity": { 2024-04-23T10:32:04.090 INFO:teuthology.orchestra.run.smithi114.stdout: "type": "mds", 2024-04-23T10:32:04.090 INFO:teuthology.orchestra.run.smithi114.stdout: "num": 2 2024-04-23T10:32:04.090 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-23T10:32:04.090 INFO:teuthology.orchestra.run.smithi114.stdout: "tid": 126 2024-04-23T10:32:04.090 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-23T10:32:04.090 INFO:teuthology.orchestra.run.smithi114.stdout: "op_type": "internal_op", 2024-04-23T10:32:04.090 INFO:teuthology.orchestra.run.smithi114.stdout: "internal_op": 5385, 2024-04-23T10:32:04.090 INFO:teuthology.orchestra.run.smithi114.stdout: "op_name": "quiesce_inode", 2024-04-23T10:32:04.090 INFO:teuthology.orchestra.run.smithi114.stdout: "events": [ 2024-04-23T10:32:04.090 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-04-23T10:32:04.090 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-04-23T10:31:21.477876+0000", 2024-04-23T10:32:04.090 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "initiated" 2024-04-23T10:32:04.090 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-23T10:32:04.091 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-04-23T10:32:04.091 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-04-23T10:31:21.477876+0000", 2024-04-23T10:32:04.091 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "header_read" 2024-04-23T10:32:04.091 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-23T10:32:04.091 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-04-23T10:32:04.091 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-04-23T10:31:21.477876+0000", 2024-04-23T10:32:04.091 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "throttled" 2024-04-23T10:32:04.091 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-23T10:32:04.091 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-04-23T10:32:04.091 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-04-23T10:31:21.477876+0000", 2024-04-23T10:32:04.091 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "all_read" 2024-04-23T10:32:04.091 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-23T10:32:04.091 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-04-23T10:32:04.091 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-04-23T10:31:21.477876+0000", 2024-04-23T10:32:04.091 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "dispatched" 2024-04-23T10:32:04.092 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-23T10:32:04.092 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-04-23T10:32:04.092 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-04-23T10:31:21.478331+0000", 2024-04-23T10:32:04.092 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "failed to xlock, waiting" 2024-04-23T10:32:04.092 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-23T10:32:04.092 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-04-23T10:32:04.092 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-04-23T10:31:21.481526+0000", 2024-04-23T10:32:04.092 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "acquired locks" 2024-04-23T10:32:04.092 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-23T10:32:04.092 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-04-23T10:32:04.092 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-04-23T10:31:21.481777+0000", 2024-04-23T10:32:04.092 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "quiesce complete" 2024-04-23T10:32:04.092 INFO:teuthology.orchestra.run.smithi114.stdout: } 2024-04-23T10:32:04.092 INFO:teuthology.orchestra.run.smithi114.stdout: ], 2024-04-23T10:32:04.093 INFO:teuthology.orchestra.run.smithi114.stdout: "locks": [ 2024-04-23T10:32:04.093 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-04-23T10:32:04.093 INFO:teuthology.orchestra.run.smithi114.stdout: "object": { 2024-04-23T10:32:04.093 INFO:teuthology.orchestra.run.smithi114.stdout: "is_auth": true, 2024-04-23T10:32:04.093 INFO:teuthology.orchestra.run.smithi114.stdout: "auth_state": { 2024-04-23T10:32:04.093 INFO:teuthology.orchestra.run.smithi114.stdout: "replicas": { 2024-04-23T10:32:04.093 INFO:teuthology.orchestra.run.smithi114.stdout: "0": 1 2024-04-23T10:32:04.093 INFO:teuthology.orchestra.run.smithi114.stdout: } 2024-04-23T10:32:04.093 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-23T10:32:04.093 INFO:teuthology.orchestra.run.smithi114.stdout: "replica_state": { 2024-04-23T10:32:04.093 INFO:teuthology.orchestra.run.smithi114.stdout: "authority": [ 2024-04-23T10:32:04.093 INFO:teuthology.orchestra.run.smithi114.stdout: 2, 2024-04-23T10:32:04.093 INFO:teuthology.orchestra.run.smithi114.stdout: -2 2024-04-23T10:32:04.093 INFO:teuthology.orchestra.run.smithi114.stdout: ], 2024-04-23T10:32:04.093 INFO:teuthology.orchestra.run.smithi114.stdout: "replica_nonce": 0 2024-04-23T10:32:04.094 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-23T10:32:04.094 INFO:teuthology.orchestra.run.smithi114.stdout: "auth_pins": 2, 2024-04-23T10:32:04.094 INFO:teuthology.orchestra.run.smithi114.stdout: "is_frozen": false, 2024-04-23T10:32:04.094 INFO:teuthology.orchestra.run.smithi114.stdout: "is_freezing": false, 2024-04-23T10:32:04.094 INFO:teuthology.orchestra.run.smithi114.stdout: "pins": { 2024-04-23T10:32:04.094 INFO:teuthology.orchestra.run.smithi114.stdout: "ptrwaiter": 0, 2024-04-23T10:32:04.094 INFO:teuthology.orchestra.run.smithi114.stdout: "request": 1, 2024-04-23T10:32:04.094 INFO:teuthology.orchestra.run.smithi114.stdout: "lock": 1, 2024-04-23T10:32:04.094 INFO:teuthology.orchestra.run.smithi114.stdout: "importingcaps": 0, 2024-04-23T10:32:04.094 INFO:teuthology.orchestra.run.smithi114.stdout: "caps": 0, 2024-04-23T10:32:04.094 INFO:teuthology.orchestra.run.smithi114.stdout: "replicated": 1, 2024-04-23T10:32:04.094 INFO:teuthology.orchestra.run.smithi114.stdout: "dirty": 1, 2024-04-23T10:32:04.094 INFO:teuthology.orchestra.run.smithi114.stdout: "waiter": 0, 2024-04-23T10:32:04.094 INFO:teuthology.orchestra.run.smithi114.stdout: "authpin": 1 2024-04-23T10:32:04.095 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-23T10:32:04.095 INFO:teuthology.orchestra.run.smithi114.stdout: "nref": 5 2024-04-23T10:32:04.095 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-23T10:32:04.095 INFO:teuthology.orchestra.run.smithi114.stdout: "object_string": "[inode 0x1000000021f [7,head] /volumes/qa/sv_1/b206794e-7622-4ab8-8943-f3ae4e65b7a3/client.0/tmp/clients/client0/~dmtmp/COREL/ROSE.CPT auth{0=1} v7256 ap=2 s=260096 n(v0 rc2024-04-23T10:27:52.049396+0000 b260096 1=1+0) (iquiesce lock x=1 by request(mds.2:126 nref=3)) | ptrwaiter=0 request=1 lock=1 importingcaps=0 caps=0 replicated=1 dirty=1 waiter=0 authpin=1 0x55c68bbfc680]", 2024-04-23T10:32:04.095 INFO:teuthology.orchestra.run.smithi114.stdout: "lock": { 2024-04-23T10:32:04.095 INFO:teuthology.orchestra.run.smithi114.stdout: "gather_set": [], 2024-04-23T10:32:04.095 INFO:teuthology.orchestra.run.smithi114.stdout: "state": "lock", 2024-04-23T10:32:04.095 INFO:teuthology.orchestra.run.smithi114.stdout: "type": "iquiesce", 2024-04-23T10:32:04.095 INFO:teuthology.orchestra.run.smithi114.stdout: "is_leased": false, 2024-04-23T10:32:04.095 INFO:teuthology.orchestra.run.smithi114.stdout: "num_rdlocks": 0, 2024-04-23T10:32:04.095 INFO:teuthology.orchestra.run.smithi114.stdout: "num_wrlocks": 0, 2024-04-23T10:32:04.095 INFO:teuthology.orchestra.run.smithi114.stdout: "num_xlocks": 1, 2024-04-23T10:32:04.095 INFO:teuthology.orchestra.run.smithi114.stdout: "xlock_by": { 2024-04-23T10:32:04.095 INFO:teuthology.orchestra.run.smithi114.stdout: "reqid": { 2024-04-23T10:32:04.096 INFO:teuthology.orchestra.run.smithi114.stdout: "entity": { 2024-04-23T10:32:04.096 INFO:teuthology.orchestra.run.smithi114.stdout: "type": "mds", 2024-04-23T10:32:04.096 INFO:teuthology.orchestra.run.smithi114.stdout: "num": 2 2024-04-23T10:32:04.096 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-23T10:32:04.096 INFO:teuthology.orchestra.run.smithi114.stdout: "tid": 126 2024-04-23T10:32:04.096 INFO:teuthology.orchestra.run.smithi114.stdout: } 2024-04-23T10:32:04.096 INFO:teuthology.orchestra.run.smithi114.stdout: } 2024-04-23T10:32:04.096 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-23T10:32:04.096 INFO:teuthology.orchestra.run.smithi114.stdout: "flags": 4, 2024-04-23T10:32:04.096 INFO:teuthology.orchestra.run.smithi114.stdout: "wrlock_target": -1 2024-04-23T10:32:04.096 INFO:teuthology.orchestra.run.smithi114.stdout: } 2024-04-23T10:32:04.096 INFO:teuthology.orchestra.run.smithi114.stdout: ] 2024-04-23T10:32:04.096 INFO:teuthology.orchestra.run.smithi114.stdout: } 2024-04-23T10:32:04.096 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-23T10:32:04.096 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-04-23T10:32:04.097 INFO:teuthology.orchestra.run.smithi114.stdout: "description": "internal op quiesce_inode:mds.2:147 fp=#0x10000000212", 2024-04-23T10:32:04.097 INFO:teuthology.orchestra.run.smithi114.stdout: "initiated_at": "2024-04-23T10:31:21.498718+0000", 2024-04-23T10:32:04.097 INFO:teuthology.orchestra.run.smithi114.stdout: "age": 42.330026959000001, 2024-04-23T10:32:04.097 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 42.332541767999999, 2024-04-23T10:32:04.097 INFO:teuthology.orchestra.run.smithi114.stdout: "continuous": true, 2024-04-23T10:32:04.097 INFO:teuthology.orchestra.run.smithi114.stdout: "type_data": { 2024-04-23T10:32:04.097 INFO:teuthology.orchestra.run.smithi114.stdout: "result": -2147483648, 2024-04-23T10:32:04.097 INFO:teuthology.orchestra.run.smithi114.stdout: "flag_point": "quiesce complete", 2024-04-23T10:32:04.097 INFO:teuthology.orchestra.run.smithi114.stdout: "reqid": { 2024-04-23T10:32:04.097 INFO:teuthology.orchestra.run.smithi114.stdout: "entity": { 2024-04-23T10:32:04.097 INFO:teuthology.orchestra.run.smithi114.stdout: "type": "mds", 2024-04-23T10:32:04.097 INFO:teuthology.orchestra.run.smithi114.stdout: "num": 2 2024-04-23T10:32:04.097 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-23T10:32:04.097 INFO:teuthology.orchestra.run.smithi114.stdout: "tid": 147 2024-04-23T10:32:04.097 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-23T10:32:04.098 INFO:teuthology.orchestra.run.smithi114.stdout: "op_type": "internal_op", 2024-04-23T10:32:04.098 INFO:teuthology.orchestra.run.smithi114.stdout: "internal_op": 5385, 2024-04-23T10:32:04.098 INFO:teuthology.orchestra.run.smithi114.stdout: "op_name": "quiesce_inode", 2024-04-23T10:32:04.098 INFO:teuthology.orchestra.run.smithi114.stdout: "events": [ 2024-04-23T10:32:04.098 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-04-23T10:32:04.098 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-04-23T10:31:21.498718+0000", 2024-04-23T10:32:04.098 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "initiated" 2024-04-23T10:32:04.098 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-23T10:32:04.098 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-04-23T10:32:04.098 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-04-23T10:31:21.498718+0000", 2024-04-23T10:32:04.098 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "header_read" 2024-04-23T10:32:04.098 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-23T10:32:04.098 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-04-23T10:32:04.099 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-04-23T10:31:21.498718+0000", 2024-04-23T10:32:04.099 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "throttled" 2024-04-23T10:32:04.099 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-23T10:32:04.099 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-04-23T10:32:04.099 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-04-23T10:31:21.498718+0000", 2024-04-23T10:32:04.099 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "all_read" 2024-04-23T10:32:04.099 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-23T10:32:04.099 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-04-23T10:32:04.099 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-04-23T10:31:21.498718+0000", 2024-04-23T10:32:04.099 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "dispatched" 2024-04-23T10:32:04.099 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-23T10:32:04.099 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-04-23T10:32:04.099 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-04-23T10:31:21.499826+0000", 2024-04-23T10:32:04.100 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "failed to xlock, waiting" 2024-04-23T10:32:04.100 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-23T10:32:04.100 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-04-23T10:32:04.100 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-04-23T10:31:21.504506+0000", 2024-04-23T10:32:04.100 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "failed to xlock, waiting" 2024-04-23T10:32:04.100 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-23T10:32:04.100 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-04-23T10:32:04.100 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-04-23T10:31:21.508468+0000", 2024-04-23T10:32:04.100 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "acquired locks" 2024-04-23T10:32:04.100 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-23T10:32:04.100 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-04-23T10:32:04.101 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-04-23T10:31:21.508653+0000", 2024-04-23T10:32:04.101 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "quiesce complete" 2024-04-23T10:32:04.101 INFO:teuthology.orchestra.run.smithi114.stdout: } 2024-04-23T10:32:04.101 INFO:teuthology.orchestra.run.smithi114.stdout: ], 2024-04-23T10:32:04.101 INFO:teuthology.orchestra.run.smithi114.stdout: "locks": [ 2024-04-23T10:32:04.101 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-04-23T10:32:04.101 INFO:teuthology.orchestra.run.smithi114.stdout: "object": { 2024-04-23T10:32:04.101 INFO:teuthology.orchestra.run.smithi114.stdout: "is_auth": true, 2024-04-23T10:32:04.101 INFO:teuthology.orchestra.run.smithi114.stdout: "auth_state": { 2024-04-23T10:32:04.101 INFO:teuthology.orchestra.run.smithi114.stdout: "replicas": {} 2024-04-23T10:32:04.101 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-23T10:32:04.101 INFO:teuthology.orchestra.run.smithi114.stdout: "replica_state": { 2024-04-23T10:32:04.101 INFO:teuthology.orchestra.run.smithi114.stdout: "authority": [ 2024-04-23T10:32:04.101 INFO:teuthology.orchestra.run.smithi114.stdout: 2, 2024-04-23T10:32:04.102 INFO:teuthology.orchestra.run.smithi114.stdout: -2 2024-04-23T10:32:04.102 INFO:teuthology.orchestra.run.smithi114.stdout: ], 2024-04-23T10:32:04.102 INFO:teuthology.orchestra.run.smithi114.stdout: "replica_nonce": 0 2024-04-23T10:32:04.102 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-23T10:32:04.102 INFO:teuthology.orchestra.run.smithi114.stdout: "auth_pins": 1, 2024-04-23T10:32:04.102 INFO:teuthology.orchestra.run.smithi114.stdout: "is_frozen": false, 2024-04-23T10:32:04.102 INFO:teuthology.orchestra.run.smithi114.stdout: "is_freezing": false, 2024-04-23T10:32:04.102 INFO:teuthology.orchestra.run.smithi114.stdout: "pins": { 2024-04-23T10:32:04.102 INFO:teuthology.orchestra.run.smithi114.stdout: "ptrwaiter": 0, 2024-04-23T10:32:04.102 INFO:teuthology.orchestra.run.smithi114.stdout: "request": 1, 2024-04-23T10:32:04.102 INFO:teuthology.orchestra.run.smithi114.stdout: "lock": 1, 2024-04-23T10:32:04.102 INFO:teuthology.orchestra.run.smithi114.stdout: "importingcaps": 0, 2024-04-23T10:32:04.102 INFO:teuthology.orchestra.run.smithi114.stdout: "caps": 0, 2024-04-23T10:32:04.102 INFO:teuthology.orchestra.run.smithi114.stdout: "replicated": 0, 2024-04-23T10:32:04.103 INFO:teuthology.orchestra.run.smithi114.stdout: "dirty": 1, 2024-04-23T10:32:04.103 INFO:teuthology.orchestra.run.smithi114.stdout: "waiter": 0, 2024-04-23T10:32:04.103 INFO:teuthology.orchestra.run.smithi114.stdout: "authpin": 1 2024-04-23T10:32:04.103 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-23T10:32:04.103 INFO:teuthology.orchestra.run.smithi114.stdout: "nref": 4 2024-04-23T10:32:04.103 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-23T10:32:04.103 INFO:teuthology.orchestra.run.smithi114.stdout: "object_string": "[inode 0x10000000212 [4,head] /volumes/qa/sv_1/b206794e-7622-4ab8-8943-f3ae4e65b7a3/client.0/tmp/clients/client0/~dmtmp/PWRPNT/PPTOOLS1.PPA auth v12266 ap=1 s=260096 n(v0 rc2024-04-23T10:27:51.988397+0000 b260096 1=1+0) (iquiesce lock x=1 by request(mds.2:147 nref=3)) | ptrwaiter=0 request=1 lock=1 importingcaps=0 caps=0 replicated=0 dirty=1 waiter=0 authpin=1 0x55c68bc7fb80]", 2024-04-23T10:32:04.103 INFO:teuthology.orchestra.run.smithi114.stdout: "lock": { 2024-04-23T10:32:04.103 INFO:teuthology.orchestra.run.smithi114.stdout: "gather_set": [], 2024-04-23T10:32:04.103 INFO:teuthology.orchestra.run.smithi114.stdout: "state": "lock", 2024-04-23T10:32:04.103 INFO:teuthology.orchestra.run.smithi114.stdout: "type": "iquiesce", 2024-04-23T10:32:04.103 INFO:teuthology.orchestra.run.smithi114.stdout: "is_leased": false, 2024-04-23T10:32:04.103 INFO:teuthology.orchestra.run.smithi114.stdout: "num_rdlocks": 0, 2024-04-23T10:32:04.103 INFO:teuthology.orchestra.run.smithi114.stdout: "num_wrlocks": 0, 2024-04-23T10:32:04.103 INFO:teuthology.orchestra.run.smithi114.stdout: "num_xlocks": 1, 2024-04-23T10:32:04.103 INFO:teuthology.orchestra.run.smithi114.stdout: "xlock_by": { 2024-04-23T10:32:04.104 INFO:teuthology.orchestra.run.smithi114.stdout: "reqid": { 2024-04-23T10:32:04.104 INFO:teuthology.orchestra.run.smithi114.stdout: "entity": { 2024-04-23T10:32:04.104 INFO:teuthology.orchestra.run.smithi114.stdout: "type": "mds", 2024-04-23T10:32:04.104 INFO:teuthology.orchestra.run.smithi114.stdout: "num": 2 2024-04-23T10:32:04.104 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-23T10:32:04.104 INFO:teuthology.orchestra.run.smithi114.stdout: "tid": 147 2024-04-23T10:32:04.104 INFO:teuthology.orchestra.run.smithi114.stdout: } 2024-04-23T10:32:04.104 INFO:teuthology.orchestra.run.smithi114.stdout: } 2024-04-23T10:32:04.104 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-23T10:32:04.104 INFO:teuthology.orchestra.run.smithi114.stdout: "flags": 4, 2024-04-23T10:32:04.104 INFO:teuthology.orchestra.run.smithi114.stdout: "wrlock_target": -1 2024-04-23T10:32:04.104 INFO:teuthology.orchestra.run.smithi114.stdout: } 2024-04-23T10:32:04.104 INFO:teuthology.orchestra.run.smithi114.stdout: ] 2024-04-23T10:32:04.105 INFO:teuthology.orchestra.run.smithi114.stdout: } 2024-04-23T10:32:04.105 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-23T10:32:04.105 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-04-23T10:32:04.105 INFO:teuthology.orchestra.run.smithi114.stdout: "description": "internal op quiesce_inode:mds.2:127 fp=#0x1000000021b", 2024-04-23T10:32:04.105 INFO:teuthology.orchestra.run.smithi114.stdout: "initiated_at": "2024-04-23T10:31:21.477896+0000", 2024-04-23T10:32:04.105 INFO:teuthology.orchestra.run.smithi114.stdout: "age": 42.350849224000001, 2024-04-23T10:32:04.105 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 42.353428264999998, 2024-04-23T10:32:04.105 INFO:teuthology.orchestra.run.smithi114.stdout: "continuous": true, 2024-04-23T10:32:04.105 INFO:teuthology.orchestra.run.smithi114.stdout: "type_data": { 2024-04-23T10:32:04.105 INFO:teuthology.orchestra.run.smithi114.stdout: "result": -2147483648, 2024-04-23T10:32:04.105 INFO:teuthology.orchestra.run.smithi114.stdout: "flag_point": "quiesce complete", 2024-04-23T10:32:04.105 INFO:teuthology.orchestra.run.smithi114.stdout: "reqid": { 2024-04-23T10:32:04.105 INFO:teuthology.orchestra.run.smithi114.stdout: "entity": { 2024-04-23T10:32:04.105 INFO:teuthology.orchestra.run.smithi114.stdout: "type": "mds", 2024-04-23T10:32:04.105 INFO:teuthology.orchestra.run.smithi114.stdout: "num": 2 2024-04-23T10:32:04.106 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-23T10:32:04.106 INFO:teuthology.orchestra.run.smithi114.stdout: "tid": 127 2024-04-23T10:32:04.106 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-23T10:32:04.106 INFO:teuthology.orchestra.run.smithi114.stdout: "op_type": "internal_op", 2024-04-23T10:32:04.106 INFO:teuthology.orchestra.run.smithi114.stdout: "internal_op": 5385, 2024-04-23T10:32:04.106 INFO:teuthology.orchestra.run.smithi114.stdout: "op_name": "quiesce_inode", 2024-04-23T10:32:04.106 INFO:teuthology.orchestra.run.smithi114.stdout: "events": [ 2024-04-23T10:32:04.106 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-04-23T10:32:04.106 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-04-23T10:31:21.477896+0000", 2024-04-23T10:32:04.106 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "initiated" 2024-04-23T10:32:04.106 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-23T10:32:04.106 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-04-23T10:32:04.106 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-04-23T10:31:21.477896+0000", 2024-04-23T10:32:04.106 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "header_read" 2024-04-23T10:32:04.106 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-23T10:32:04.107 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-04-23T10:32:04.107 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-04-23T10:31:21.477896+0000", 2024-04-23T10:32:04.107 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "throttled" 2024-04-23T10:32:04.107 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-23T10:32:04.107 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-04-23T10:32:04.107 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-04-23T10:31:21.477896+0000", 2024-04-23T10:32:04.107 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "all_read" 2024-04-23T10:32:04.107 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-23T10:32:04.107 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-04-23T10:32:04.107 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-04-23T10:31:21.477896+0000", 2024-04-23T10:32:04.107 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "dispatched" 2024-04-23T10:32:04.107 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-23T10:32:04.107 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-04-23T10:32:04.107 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-04-23T10:31:21.478669+0000", 2024-04-23T10:32:04.108 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "failed to xlock, waiting" 2024-04-23T10:32:04.108 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-23T10:32:04.108 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-04-23T10:32:04.108 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-04-23T10:31:21.482509+0000", 2024-04-23T10:32:04.108 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "acquired locks" 2024-04-23T10:32:04.108 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-23T10:32:04.108 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-04-23T10:32:04.108 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-04-23T10:31:21.482729+0000", 2024-04-23T10:32:04.108 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "quiesce complete" 2024-04-23T10:32:04.108 INFO:teuthology.orchestra.run.smithi114.stdout: } 2024-04-23T10:32:04.108 INFO:teuthology.orchestra.run.smithi114.stdout: ], 2024-04-23T10:32:04.108 INFO:teuthology.orchestra.run.smithi114.stdout: "locks": [ 2024-04-23T10:32:04.108 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-04-23T10:32:04.108 INFO:teuthology.orchestra.run.smithi114.stdout: "object": { 2024-04-23T10:32:04.108 INFO:teuthology.orchestra.run.smithi114.stdout: "is_auth": true, 2024-04-23T10:32:04.109 INFO:teuthology.orchestra.run.smithi114.stdout: "auth_state": { 2024-04-23T10:32:04.109 INFO:teuthology.orchestra.run.smithi114.stdout: "replicas": { 2024-04-23T10:32:04.109 INFO:teuthology.orchestra.run.smithi114.stdout: "0": 1 2024-04-23T10:32:04.109 INFO:teuthology.orchestra.run.smithi114.stdout: } 2024-04-23T10:32:04.109 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-23T10:32:04.109 INFO:teuthology.orchestra.run.smithi114.stdout: "replica_state": { 2024-04-23T10:32:04.109 INFO:teuthology.orchestra.run.smithi114.stdout: "authority": [ 2024-04-23T10:32:04.109 INFO:teuthology.orchestra.run.smithi114.stdout: 2, 2024-04-23T10:32:04.109 INFO:teuthology.orchestra.run.smithi114.stdout: -2 2024-04-23T10:32:04.109 INFO:teuthology.orchestra.run.smithi114.stdout: ], 2024-04-23T10:32:04.109 INFO:teuthology.orchestra.run.smithi114.stdout: "replica_nonce": 0 2024-04-23T10:32:04.109 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-23T10:32:04.109 INFO:teuthology.orchestra.run.smithi114.stdout: "auth_pins": 2, 2024-04-23T10:32:04.109 INFO:teuthology.orchestra.run.smithi114.stdout: "is_frozen": false, 2024-04-23T10:32:04.109 INFO:teuthology.orchestra.run.smithi114.stdout: "is_freezing": false, 2024-04-23T10:32:04.110 INFO:teuthology.orchestra.run.smithi114.stdout: "pins": { 2024-04-23T10:32:04.110 INFO:teuthology.orchestra.run.smithi114.stdout: "ptrwaiter": 0, 2024-04-23T10:32:04.110 INFO:teuthology.orchestra.run.smithi114.stdout: "request": 1, 2024-04-23T10:32:04.110 INFO:teuthology.orchestra.run.smithi114.stdout: "lock": 1, 2024-04-23T10:32:04.110 INFO:teuthology.orchestra.run.smithi114.stdout: "importingcaps": 0, 2024-04-23T10:32:04.110 INFO:teuthology.orchestra.run.smithi114.stdout: "caps": 0, 2024-04-23T10:32:04.110 INFO:teuthology.orchestra.run.smithi114.stdout: "replicated": 1, 2024-04-23T10:32:04.110 INFO:teuthology.orchestra.run.smithi114.stdout: "dirty": 1, 2024-04-23T10:32:04.110 INFO:teuthology.orchestra.run.smithi114.stdout: "waiter": 0, 2024-04-23T10:32:04.110 INFO:teuthology.orchestra.run.smithi114.stdout: "authpin": 1 2024-04-23T10:32:04.110 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-23T10:32:04.110 INFO:teuthology.orchestra.run.smithi114.stdout: "nref": 5 2024-04-23T10:32:04.110 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-23T10:32:04.110 INFO:teuthology.orchestra.run.smithi114.stdout: "object_string": "[inode 0x1000000021b [7,head] /volumes/qa/sv_1/b206794e-7622-4ab8-8943-f3ae4e65b7a3/client.0/tmp/clients/client0/~dmtmp/COREL/TEAPOT2.CPT auth{0=1} v7248 ap=2 s=260096 n(v0 rc2024-04-23T10:27:52.028396+0000 b260096 1=1+0) (iquiesce lock x=1 by request(mds.2:127 nref=3)) | ptrwaiter=0 request=1 lock=1 importingcaps=0 caps=0 replicated=1 dirty=1 waiter=0 authpin=1 0x55c68bbfcc00]", 2024-04-23T10:32:04.111 INFO:teuthology.orchestra.run.smithi114.stdout: "lock": { 2024-04-23T10:32:04.111 INFO:teuthology.orchestra.run.smithi114.stdout: "gather_set": [], 2024-04-23T10:32:04.111 INFO:teuthology.orchestra.run.smithi114.stdout: "state": "lock", 2024-04-23T10:32:04.111 INFO:teuthology.orchestra.run.smithi114.stdout: "type": "iquiesce", 2024-04-23T10:32:04.111 INFO:teuthology.orchestra.run.smithi114.stdout: "is_leased": false, 2024-04-23T10:32:04.111 INFO:teuthology.orchestra.run.smithi114.stdout: "num_rdlocks": 0, 2024-04-23T10:32:04.111 INFO:teuthology.orchestra.run.smithi114.stdout: "num_wrlocks": 0, 2024-04-23T10:32:04.111 INFO:teuthology.orchestra.run.smithi114.stdout: "num_xlocks": 1, 2024-04-23T10:32:04.111 INFO:teuthology.orchestra.run.smithi114.stdout: "xlock_by": { 2024-04-23T10:32:04.111 INFO:teuthology.orchestra.run.smithi114.stdout: "reqid": { 2024-04-23T10:32:04.111 INFO:teuthology.orchestra.run.smithi114.stdout: "entity": { 2024-04-23T10:32:04.111 INFO:teuthology.orchestra.run.smithi114.stdout: "type": "mds", 2024-04-23T10:32:04.111 INFO:teuthology.orchestra.run.smithi114.stdout: "num": 2 2024-04-23T10:32:04.111 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-23T10:32:04.112 INFO:teuthology.orchestra.run.smithi114.stdout: "tid": 127 2024-04-23T10:32:04.112 INFO:teuthology.orchestra.run.smithi114.stdout: } 2024-04-23T10:32:04.112 INFO:teuthology.orchestra.run.smithi114.stdout: } 2024-04-23T10:32:04.112 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-23T10:32:04.112 INFO:teuthology.orchestra.run.smithi114.stdout: "flags": 4, 2024-04-23T10:32:04.112 INFO:teuthology.orchestra.run.smithi114.stdout: "wrlock_target": -1 2024-04-23T10:32:04.112 INFO:teuthology.orchestra.run.smithi114.stdout: } 2024-04-23T10:32:04.112 INFO:teuthology.orchestra.run.smithi114.stdout: ] 2024-04-23T10:32:04.112 INFO:teuthology.orchestra.run.smithi114.stdout: } 2024-04-23T10:32:04.112 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-23T10:32:04.112 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-04-23T10:32:04.112 INFO:teuthology.orchestra.run.smithi114.stdout: "description": "internal op quiesce_inode:mds.2:128 fp=#0x1000000021a", 2024-04-23T10:32:04.112 INFO:teuthology.orchestra.run.smithi114.stdout: "initiated_at": "2024-04-23T10:31:21.477913+0000", 2024-04-23T10:32:04.113 INFO:teuthology.orchestra.run.smithi114.stdout: "age": 42.350832021000002, 2024-04-23T10:32:04.113 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 42.353474875000003, 2024-04-23T10:32:04.113 INFO:teuthology.orchestra.run.smithi114.stdout: "continuous": true, 2024-04-23T10:32:04.113 INFO:teuthology.orchestra.run.smithi114.stdout: "type_data": { 2024-04-23T10:32:04.113 INFO:teuthology.orchestra.run.smithi114.stdout: "result": -2147483648, 2024-04-23T10:32:04.113 INFO:teuthology.orchestra.run.smithi114.stdout: "flag_point": "quiesce complete", 2024-04-23T10:32:04.113 INFO:teuthology.orchestra.run.smithi114.stdout: "reqid": { 2024-04-23T10:32:04.113 INFO:teuthology.orchestra.run.smithi114.stdout: "entity": { 2024-04-23T10:32:04.113 INFO:teuthology.orchestra.run.smithi114.stdout: "type": "mds", 2024-04-23T10:32:04.113 INFO:teuthology.orchestra.run.smithi114.stdout: "num": 2 2024-04-23T10:32:04.113 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-23T10:32:04.113 INFO:teuthology.orchestra.run.smithi114.stdout: "tid": 128 2024-04-23T10:32:04.113 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-23T10:32:04.113 INFO:teuthology.orchestra.run.smithi114.stdout: "op_type": "internal_op", 2024-04-23T10:32:04.113 INFO:teuthology.orchestra.run.smithi114.stdout: "internal_op": 5385, 2024-04-23T10:32:04.114 INFO:teuthology.orchestra.run.smithi114.stdout: "op_name": "quiesce_inode", 2024-04-23T10:32:04.114 INFO:teuthology.orchestra.run.smithi114.stdout: "events": [ 2024-04-23T10:32:04.114 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-04-23T10:32:04.114 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-04-23T10:31:21.477913+0000", 2024-04-23T10:32:04.114 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "initiated" 2024-04-23T10:32:04.114 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-23T10:32:04.114 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-04-23T10:32:04.114 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-04-23T10:31:21.477913+0000", 2024-04-23T10:32:04.114 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "header_read" 2024-04-23T10:32:04.114 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-23T10:32:04.114 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-04-23T10:32:04.114 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-04-23T10:31:21.477913+0000", 2024-04-23T10:32:04.114 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "throttled" 2024-04-23T10:32:04.114 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-23T10:32:04.115 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-04-23T10:32:04.115 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-04-23T10:31:21.477913+0000", 2024-04-23T10:32:04.115 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "all_read" 2024-04-23T10:32:04.115 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-23T10:32:04.115 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-04-23T10:32:04.115 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-04-23T10:31:21.477913+0000", 2024-04-23T10:32:04.115 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "dispatched" 2024-04-23T10:32:04.115 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-23T10:32:04.115 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-04-23T10:32:04.115 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-04-23T10:31:21.479061+0000", 2024-04-23T10:32:04.115 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "failed to xlock, waiting" 2024-04-23T10:32:04.115 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-23T10:32:04.115 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-04-23T10:32:04.115 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-04-23T10:31:21.483283+0000", 2024-04-23T10:32:04.116 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "acquired locks" 2024-04-23T10:32:04.116 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-23T10:32:04.116 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-04-23T10:32:04.116 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-04-23T10:31:21.483442+0000", 2024-04-23T10:32:04.116 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "quiesce complete" 2024-04-23T10:32:04.116 INFO:teuthology.orchestra.run.smithi114.stdout: } 2024-04-23T10:32:04.116 INFO:teuthology.orchestra.run.smithi114.stdout: ], 2024-04-23T10:32:04.116 INFO:teuthology.orchestra.run.smithi114.stdout: "locks": [ 2024-04-23T10:32:04.116 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-04-23T10:32:04.116 INFO:teuthology.orchestra.run.smithi114.stdout: "object": { 2024-04-23T10:32:04.116 INFO:teuthology.orchestra.run.smithi114.stdout: "is_auth": true, 2024-04-23T10:32:04.116 INFO:teuthology.orchestra.run.smithi114.stdout: "auth_state": { 2024-04-23T10:32:04.116 INFO:teuthology.orchestra.run.smithi114.stdout: "replicas": { 2024-04-23T10:32:04.116 INFO:teuthology.orchestra.run.smithi114.stdout: "0": 1 2024-04-23T10:32:04.116 INFO:teuthology.orchestra.run.smithi114.stdout: } 2024-04-23T10:32:04.117 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-23T10:32:04.117 INFO:teuthology.orchestra.run.smithi114.stdout: "replica_state": { 2024-04-23T10:32:04.117 INFO:teuthology.orchestra.run.smithi114.stdout: "authority": [ 2024-04-23T10:32:04.117 INFO:teuthology.orchestra.run.smithi114.stdout: 2, 2024-04-23T10:32:04.117 INFO:teuthology.orchestra.run.smithi114.stdout: -2 2024-04-23T10:32:04.117 INFO:teuthology.orchestra.run.smithi114.stdout: ], 2024-04-23T10:32:04.117 INFO:teuthology.orchestra.run.smithi114.stdout: "replica_nonce": 0 2024-04-23T10:32:04.117 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-23T10:32:04.117 INFO:teuthology.orchestra.run.smithi114.stdout: "auth_pins": 2, 2024-04-23T10:32:04.117 INFO:teuthology.orchestra.run.smithi114.stdout: "is_frozen": false, 2024-04-23T10:32:04.117 INFO:teuthology.orchestra.run.smithi114.stdout: "is_freezing": false, 2024-04-23T10:32:04.117 INFO:teuthology.orchestra.run.smithi114.stdout: "pins": { 2024-04-23T10:32:04.117 INFO:teuthology.orchestra.run.smithi114.stdout: "ptrwaiter": 0, 2024-04-23T10:32:04.117 INFO:teuthology.orchestra.run.smithi114.stdout: "request": 1, 2024-04-23T10:32:04.117 INFO:teuthology.orchestra.run.smithi114.stdout: "lock": 1, 2024-04-23T10:32:04.118 INFO:teuthology.orchestra.run.smithi114.stdout: "importingcaps": 0, 2024-04-23T10:32:04.118 INFO:teuthology.orchestra.run.smithi114.stdout: "caps": 0, 2024-04-23T10:32:04.118 INFO:teuthology.orchestra.run.smithi114.stdout: "replicated": 1, 2024-04-23T10:32:04.118 INFO:teuthology.orchestra.run.smithi114.stdout: "dirty": 1, 2024-04-23T10:32:04.118 INFO:teuthology.orchestra.run.smithi114.stdout: "waiter": 0, 2024-04-23T10:32:04.118 INFO:teuthology.orchestra.run.smithi114.stdout: "authpin": 1 2024-04-23T10:32:04.118 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-23T10:32:04.118 INFO:teuthology.orchestra.run.smithi114.stdout: "nref": 5 2024-04-23T10:32:04.118 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-23T10:32:04.118 INFO:teuthology.orchestra.run.smithi114.stdout: "object_string": "[inode 0x1000000021a [7,head] /volumes/qa/sv_1/b206794e-7622-4ab8-8943-f3ae4e65b7a3/client.0/tmp/clients/client0/~dmtmp/COREL/MAILBOX2.CPT auth{0=1} v7246 ap=2 s=260096 n(v0 rc2024-04-23T10:27:52.023396+0000 b260096 1=1+0) (iquiesce lock x=1 by request(mds.2:128 nref=3)) | ptrwaiter=0 request=1 lock=1 importingcaps=0 caps=0 replicated=1 dirty=1 waiter=0 authpin=1 0x55c68bbfd180]", 2024-04-23T10:32:04.118 INFO:teuthology.orchestra.run.smithi114.stdout: "lock": { 2024-04-23T10:32:04.118 INFO:teuthology.orchestra.run.smithi114.stdout: "gather_set": [], 2024-04-23T10:32:04.118 INFO:teuthology.orchestra.run.smithi114.stdout: "state": "lock", 2024-04-23T10:32:04.118 INFO:teuthology.orchestra.run.smithi114.stdout: "type": "iquiesce", 2024-04-23T10:32:04.119 INFO:teuthology.orchestra.run.smithi114.stdout: "is_leased": false, 2024-04-23T10:32:04.119 INFO:teuthology.orchestra.run.smithi114.stdout: "num_rdlocks": 0, 2024-04-23T10:32:04.119 INFO:teuthology.orchestra.run.smithi114.stdout: "num_wrlocks": 0, 2024-04-23T10:32:04.119 INFO:teuthology.orchestra.run.smithi114.stdout: "num_xlocks": 1, 2024-04-23T10:32:04.119 INFO:teuthology.orchestra.run.smithi114.stdout: "xlock_by": { 2024-04-23T10:32:04.119 INFO:teuthology.orchestra.run.smithi114.stdout: "reqid": { 2024-04-23T10:32:04.119 INFO:teuthology.orchestra.run.smithi114.stdout: "entity": { 2024-04-23T10:32:04.119 INFO:teuthology.orchestra.run.smithi114.stdout: "type": "mds", 2024-04-23T10:32:04.119 INFO:teuthology.orchestra.run.smithi114.stdout: "num": 2 2024-04-23T10:32:04.119 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-23T10:32:04.119 INFO:teuthology.orchestra.run.smithi114.stdout: "tid": 128 2024-04-23T10:32:04.119 INFO:teuthology.orchestra.run.smithi114.stdout: } 2024-04-23T10:32:04.119 INFO:teuthology.orchestra.run.smithi114.stdout: } 2024-04-23T10:32:04.119 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-23T10:32:04.119 INFO:teuthology.orchestra.run.smithi114.stdout: "flags": 4, 2024-04-23T10:32:04.120 INFO:teuthology.orchestra.run.smithi114.stdout: "wrlock_target": -1 2024-04-23T10:32:04.120 INFO:teuthology.orchestra.run.smithi114.stdout: } 2024-04-23T10:32:04.120 INFO:teuthology.orchestra.run.smithi114.stdout: ] 2024-04-23T10:32:04.120 INFO:teuthology.orchestra.run.smithi114.stdout: } 2024-04-23T10:32:04.120 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-23T10:32:04.120 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-04-23T10:32:04.120 INFO:teuthology.orchestra.run.smithi114.stdout: "description": "internal op quiesce_inode:mds.2:129 fp=#0x10000000220", 2024-04-23T10:32:04.120 INFO:teuthology.orchestra.run.smithi114.stdout: "initiated_at": "2024-04-23T10:31:21.477929+0000", 2024-04-23T10:32:04.120 INFO:teuthology.orchestra.run.smithi114.stdout: "age": 42.350815931, 2024-04-23T10:32:04.120 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 42.353530345999999, 2024-04-23T10:32:04.120 INFO:teuthology.orchestra.run.smithi114.stdout: "continuous": true, 2024-04-23T10:32:04.120 INFO:teuthology.orchestra.run.smithi114.stdout: "type_data": { 2024-04-23T10:32:04.120 INFO:teuthology.orchestra.run.smithi114.stdout: "result": -2147483648, 2024-04-23T10:32:04.120 INFO:teuthology.orchestra.run.smithi114.stdout: "flag_point": "quiesce complete", 2024-04-23T10:32:04.120 INFO:teuthology.orchestra.run.smithi114.stdout: "reqid": { 2024-04-23T10:32:04.121 INFO:teuthology.orchestra.run.smithi114.stdout: "entity": { 2024-04-23T10:32:04.121 INFO:teuthology.orchestra.run.smithi114.stdout: "type": "mds", 2024-04-23T10:32:04.121 INFO:teuthology.orchestra.run.smithi114.stdout: "num": 2 2024-04-23T10:32:04.121 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-23T10:32:04.121 INFO:teuthology.orchestra.run.smithi114.stdout: "tid": 129 2024-04-23T10:32:04.121 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-23T10:32:04.121 INFO:teuthology.orchestra.run.smithi114.stdout: "op_type": "internal_op", 2024-04-23T10:32:04.121 INFO:teuthology.orchestra.run.smithi114.stdout: "internal_op": 5385, 2024-04-23T10:32:04.121 INFO:teuthology.orchestra.run.smithi114.stdout: "op_name": "quiesce_inode", 2024-04-23T10:32:04.121 INFO:teuthology.orchestra.run.smithi114.stdout: "events": [ 2024-04-23T10:32:04.121 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-04-23T10:32:04.121 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-04-23T10:31:21.477929+0000", 2024-04-23T10:32:04.121 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "initiated" 2024-04-23T10:32:04.121 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-23T10:32:04.121 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-04-23T10:32:04.122 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-04-23T10:31:21.477929+0000", 2024-04-23T10:32:04.122 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "header_read" 2024-04-23T10:32:04.122 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-23T10:32:04.122 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-04-23T10:32:04.122 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-04-23T10:31:21.477929+0000", 2024-04-23T10:32:04.122 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "throttled" 2024-04-23T10:32:04.122 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-23T10:32:04.122 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-04-23T10:32:04.122 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-04-23T10:31:21.477929+0000", 2024-04-23T10:32:04.122 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "all_read" 2024-04-23T10:32:04.122 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-23T10:32:04.122 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-04-23T10:32:04.122 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-04-23T10:31:21.477929+0000", 2024-04-23T10:32:04.122 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "dispatched" 2024-04-23T10:32:04.123 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-23T10:32:04.123 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-04-23T10:32:04.123 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-04-23T10:31:21.479343+0000", 2024-04-23T10:32:04.123 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "failed to xlock, waiting" 2024-04-23T10:32:04.123 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-23T10:32:04.123 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-04-23T10:32:04.123 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-04-23T10:31:21.483782+0000", 2024-04-23T10:32:04.123 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "acquired locks" 2024-04-23T10:32:04.123 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-23T10:32:04.123 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-04-23T10:32:04.123 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-04-23T10:31:21.483892+0000", 2024-04-23T10:32:04.123 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "quiesce complete" 2024-04-23T10:32:04.123 INFO:teuthology.orchestra.run.smithi114.stdout: } 2024-04-23T10:32:04.123 INFO:teuthology.orchestra.run.smithi114.stdout: ], 2024-04-23T10:32:04.123 INFO:teuthology.orchestra.run.smithi114.stdout: "locks": [ 2024-04-23T10:32:04.124 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-04-23T10:32:04.124 INFO:teuthology.orchestra.run.smithi114.stdout: "object": { 2024-04-23T10:32:04.124 INFO:teuthology.orchestra.run.smithi114.stdout: "is_auth": true, 2024-04-23T10:32:04.124 INFO:teuthology.orchestra.run.smithi114.stdout: "auth_state": { 2024-04-23T10:32:04.124 INFO:teuthology.orchestra.run.smithi114.stdout: "replicas": { 2024-04-23T10:32:04.124 INFO:teuthology.orchestra.run.smithi114.stdout: "0": 1 2024-04-23T10:32:04.124 INFO:teuthology.orchestra.run.smithi114.stdout: } 2024-04-23T10:32:04.124 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-23T10:32:04.124 INFO:teuthology.orchestra.run.smithi114.stdout: "replica_state": { 2024-04-23T10:32:04.124 INFO:teuthology.orchestra.run.smithi114.stdout: "authority": [ 2024-04-23T10:32:04.124 INFO:teuthology.orchestra.run.smithi114.stdout: 2, 2024-04-23T10:32:04.124 INFO:teuthology.orchestra.run.smithi114.stdout: -2 2024-04-23T10:32:04.124 INFO:teuthology.orchestra.run.smithi114.stdout: ], 2024-04-23T10:32:04.124 INFO:teuthology.orchestra.run.smithi114.stdout: "replica_nonce": 0 2024-04-23T10:32:04.125 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-23T10:32:04.125 INFO:teuthology.orchestra.run.smithi114.stdout: "auth_pins": 2, 2024-04-23T10:32:04.125 INFO:teuthology.orchestra.run.smithi114.stdout: "is_frozen": false, 2024-04-23T10:32:04.125 INFO:teuthology.orchestra.run.smithi114.stdout: "is_freezing": false, 2024-04-23T10:32:04.125 INFO:teuthology.orchestra.run.smithi114.stdout: "pins": { 2024-04-23T10:32:04.125 INFO:teuthology.orchestra.run.smithi114.stdout: "ptrwaiter": 0, 2024-04-23T10:32:04.125 INFO:teuthology.orchestra.run.smithi114.stdout: "request": 1, 2024-04-23T10:32:04.125 INFO:teuthology.orchestra.run.smithi114.stdout: "lock": 1, 2024-04-23T10:32:04.125 INFO:teuthology.orchestra.run.smithi114.stdout: "importingcaps": 0, 2024-04-23T10:32:04.125 INFO:teuthology.orchestra.run.smithi114.stdout: "caps": 0, 2024-04-23T10:32:04.125 INFO:teuthology.orchestra.run.smithi114.stdout: "needsnapflush": 0, 2024-04-23T10:32:04.125 INFO:teuthology.orchestra.run.smithi114.stdout: "replicated": 1, 2024-04-23T10:32:04.125 INFO:teuthology.orchestra.run.smithi114.stdout: "dirty": 1, 2024-04-23T10:32:04.125 INFO:teuthology.orchestra.run.smithi114.stdout: "waiter": 0, 2024-04-23T10:32:04.125 INFO:teuthology.orchestra.run.smithi114.stdout: "authpin": 1 2024-04-23T10:32:04.126 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-23T10:32:04.126 INFO:teuthology.orchestra.run.smithi114.stdout: "nref": 5 2024-04-23T10:32:04.126 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-23T10:32:04.126 INFO:teuthology.orchestra.run.smithi114.stdout: "object_string": "[inode 0x10000000220 [7,head] /volumes/qa/sv_1/b206794e-7622-4ab8-8943-f3ae4e65b7a3/client.0/tmp/clients/client0/~dmtmp/COREL/CORELDRW.CDT auth{0=1} v7242 ap=2 s=260096 n(v0 rc2024-04-23T10:27:52.054395+0000 b260096 1=1+0) (iquiesce lock x=1 by request(mds.2:129 nref=3)) | ptrwaiter=0 request=1 lock=1 importingcaps=0 caps=0 needsnapflush=0 replicated=1 dirty=1 waiter=0 authpin=1 0x55c68bbfd700]", 2024-04-23T10:32:04.126 INFO:teuthology.orchestra.run.smithi114.stdout: "lock": { 2024-04-23T10:32:04.126 INFO:teuthology.orchestra.run.smithi114.stdout: "gather_set": [], 2024-04-23T10:32:04.126 INFO:teuthology.orchestra.run.smithi114.stdout: "state": "lock", 2024-04-23T10:32:04.126 INFO:teuthology.orchestra.run.smithi114.stdout: "type": "iquiesce", 2024-04-23T10:32:04.126 INFO:teuthology.orchestra.run.smithi114.stdout: "is_leased": false, 2024-04-23T10:32:04.126 INFO:teuthology.orchestra.run.smithi114.stdout: "num_rdlocks": 0, 2024-04-23T10:32:04.126 INFO:teuthology.orchestra.run.smithi114.stdout: "num_wrlocks": 0, 2024-04-23T10:32:04.126 INFO:teuthology.orchestra.run.smithi114.stdout: "num_xlocks": 1, 2024-04-23T10:32:04.126 INFO:teuthology.orchestra.run.smithi114.stdout: "xlock_by": { 2024-04-23T10:32:04.126 INFO:teuthology.orchestra.run.smithi114.stdout: "reqid": { 2024-04-23T10:32:04.127 INFO:teuthology.orchestra.run.smithi114.stdout: "entity": { 2024-04-23T10:32:04.127 INFO:teuthology.orchestra.run.smithi114.stdout: "type": "mds", 2024-04-23T10:32:04.127 INFO:teuthology.orchestra.run.smithi114.stdout: "num": 2 2024-04-23T10:32:04.127 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-23T10:32:04.127 INFO:teuthology.orchestra.run.smithi114.stdout: "tid": 129 2024-04-23T10:32:04.127 INFO:teuthology.orchestra.run.smithi114.stdout: } 2024-04-23T10:32:04.127 INFO:teuthology.orchestra.run.smithi114.stdout: } 2024-04-23T10:32:04.127 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-23T10:32:04.127 INFO:teuthology.orchestra.run.smithi114.stdout: "flags": 4, 2024-04-23T10:32:04.127 INFO:teuthology.orchestra.run.smithi114.stdout: "wrlock_target": -1 2024-04-23T10:32:04.127 INFO:teuthology.orchestra.run.smithi114.stdout: } 2024-04-23T10:32:04.127 INFO:teuthology.orchestra.run.smithi114.stdout: ] 2024-04-23T10:32:04.127 INFO:teuthology.orchestra.run.smithi114.stdout: } 2024-04-23T10:32:04.127 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-23T10:32:04.127 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-04-23T10:32:04.128 INFO:teuthology.orchestra.run.smithi114.stdout: "description": "internal op quiesce_inode:mds.2:130 fp=#0x1000000021e", 2024-04-23T10:32:04.128 INFO:teuthology.orchestra.run.smithi114.stdout: "initiated_at": "2024-04-23T10:31:21.477946+0000", 2024-04-23T10:32:04.128 INFO:teuthology.orchestra.run.smithi114.stdout: "age": 42.350799174999999, 2024-04-23T10:32:04.128 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 42.353584402999999, 2024-04-23T10:32:04.128 INFO:teuthology.orchestra.run.smithi114.stdout: "continuous": true, 2024-04-23T10:32:04.128 INFO:teuthology.orchestra.run.smithi114.stdout: "type_data": { 2024-04-23T10:32:04.128 INFO:teuthology.orchestra.run.smithi114.stdout: "result": -2147483648, 2024-04-23T10:32:04.128 INFO:teuthology.orchestra.run.smithi114.stdout: "flag_point": "quiesce complete", 2024-04-23T10:32:04.128 INFO:teuthology.orchestra.run.smithi114.stdout: "reqid": { 2024-04-23T10:32:04.128 INFO:teuthology.orchestra.run.smithi114.stdout: "entity": { 2024-04-23T10:32:04.128 INFO:teuthology.orchestra.run.smithi114.stdout: "type": "mds", 2024-04-23T10:32:04.128 INFO:teuthology.orchestra.run.smithi114.stdout: "num": 2 2024-04-23T10:32:04.128 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-23T10:32:04.128 INFO:teuthology.orchestra.run.smithi114.stdout: "tid": 130 2024-04-23T10:32:04.128 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-23T10:32:04.129 INFO:teuthology.orchestra.run.smithi114.stdout: "op_type": "internal_op", 2024-04-23T10:32:04.129 INFO:teuthology.orchestra.run.smithi114.stdout: "internal_op": 5385, 2024-04-23T10:32:04.129 INFO:teuthology.orchestra.run.smithi114.stdout: "op_name": "quiesce_inode", 2024-04-23T10:32:04.129 INFO:teuthology.orchestra.run.smithi114.stdout: "events": [ 2024-04-23T10:32:04.129 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-04-23T10:32:04.129 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-04-23T10:31:21.477946+0000", 2024-04-23T10:32:04.129 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "initiated" 2024-04-23T10:32:04.129 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-23T10:32:04.129 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-04-23T10:32:04.129 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-04-23T10:31:21.477946+0000", 2024-04-23T10:32:04.129 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "header_read" 2024-04-23T10:32:04.129 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-23T10:32:04.129 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-04-23T10:32:04.129 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-04-23T10:31:21.477946+0000", 2024-04-23T10:32:04.129 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "throttled" 2024-04-23T10:32:04.130 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-23T10:32:04.130 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-04-23T10:32:04.130 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-04-23T10:31:21.477946+0000", 2024-04-23T10:32:04.130 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "all_read" 2024-04-23T10:32:04.130 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-23T10:32:04.130 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-04-23T10:32:04.130 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-04-23T10:31:21.477946+0000", 2024-04-23T10:32:04.130 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "dispatched" 2024-04-23T10:32:04.130 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-23T10:32:04.130 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-04-23T10:32:04.130 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-04-23T10:31:21.479632+0000", 2024-04-23T10:32:04.130 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "failed to xlock, waiting" 2024-04-23T10:32:04.130 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-23T10:32:04.130 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-04-23T10:32:04.130 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-04-23T10:31:21.484483+0000", 2024-04-23T10:32:04.131 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "acquired locks" 2024-04-23T10:32:04.131 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-23T10:32:04.131 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-04-23T10:32:04.131 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-04-23T10:31:21.484617+0000", 2024-04-23T10:32:04.131 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "quiesce complete" 2024-04-23T10:32:04.131 INFO:teuthology.orchestra.run.smithi114.stdout: } 2024-04-23T10:32:04.131 INFO:teuthology.orchestra.run.smithi114.stdout: ], 2024-04-23T10:32:04.131 INFO:teuthology.orchestra.run.smithi114.stdout: "locks": [ 2024-04-23T10:32:04.131 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-04-23T10:32:04.131 INFO:teuthology.orchestra.run.smithi114.stdout: "object": { 2024-04-23T10:32:04.131 INFO:teuthology.orchestra.run.smithi114.stdout: "is_auth": true, 2024-04-23T10:32:04.131 INFO:teuthology.orchestra.run.smithi114.stdout: "auth_state": { 2024-04-23T10:32:04.131 INFO:teuthology.orchestra.run.smithi114.stdout: "replicas": { 2024-04-23T10:32:04.131 INFO:teuthology.orchestra.run.smithi114.stdout: "0": 1 2024-04-23T10:32:04.131 INFO:teuthology.orchestra.run.smithi114.stdout: } 2024-04-23T10:32:04.132 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-23T10:32:04.132 INFO:teuthology.orchestra.run.smithi114.stdout: "replica_state": { 2024-04-23T10:32:04.132 INFO:teuthology.orchestra.run.smithi114.stdout: "authority": [ 2024-04-23T10:32:04.132 INFO:teuthology.orchestra.run.smithi114.stdout: 2, 2024-04-23T10:32:04.132 INFO:teuthology.orchestra.run.smithi114.stdout: -2 2024-04-23T10:32:04.132 INFO:teuthology.orchestra.run.smithi114.stdout: ], 2024-04-23T10:32:04.132 INFO:teuthology.orchestra.run.smithi114.stdout: "replica_nonce": 0 2024-04-23T10:32:04.132 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-23T10:32:04.132 INFO:teuthology.orchestra.run.smithi114.stdout: "auth_pins": 2, 2024-04-23T10:32:04.132 INFO:teuthology.orchestra.run.smithi114.stdout: "is_frozen": false, 2024-04-23T10:32:04.132 INFO:teuthology.orchestra.run.smithi114.stdout: "is_freezing": false, 2024-04-23T10:32:04.132 INFO:teuthology.orchestra.run.smithi114.stdout: "pins": { 2024-04-23T10:32:04.132 INFO:teuthology.orchestra.run.smithi114.stdout: "ptrwaiter": 0, 2024-04-23T10:32:04.132 INFO:teuthology.orchestra.run.smithi114.stdout: "request": 1, 2024-04-23T10:32:04.132 INFO:teuthology.orchestra.run.smithi114.stdout: "lock": 1, 2024-04-23T10:32:04.133 INFO:teuthology.orchestra.run.smithi114.stdout: "importingcaps": 0, 2024-04-23T10:32:04.133 INFO:teuthology.orchestra.run.smithi114.stdout: "caps": 0, 2024-04-23T10:32:04.133 INFO:teuthology.orchestra.run.smithi114.stdout: "needsnapflush": 0, 2024-04-23T10:32:04.133 INFO:teuthology.orchestra.run.smithi114.stdout: "replicated": 1, 2024-04-23T10:32:04.133 INFO:teuthology.orchestra.run.smithi114.stdout: "dirty": 1, 2024-04-23T10:32:04.133 INFO:teuthology.orchestra.run.smithi114.stdout: "waiter": 0, 2024-04-23T10:32:04.133 INFO:teuthology.orchestra.run.smithi114.stdout: "authpin": 1 2024-04-23T10:32:04.133 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-23T10:32:04.133 INFO:teuthology.orchestra.run.smithi114.stdout: "nref": 5 2024-04-23T10:32:04.133 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-23T10:32:04.133 INFO:teuthology.orchestra.run.smithi114.stdout: "object_string": "[inode 0x1000000021e [7,head] /volumes/qa/sv_1/b206794e-7622-4ab8-8943-f3ae4e65b7a3/client.0/tmp/clients/client0/~dmtmp/COREL/GRAPHIC2.CDR auth{0=1} v7254 ap=2 s=260096 n(v0 rc2024-04-23T10:27:52.043396+0000 b260096 1=1+0) (iquiesce lock x=1 by request(mds.2:130 nref=3)) | ptrwaiter=0 request=1 lock=1 importingcaps=0 caps=0 needsnapflush=0 replicated=1 dirty=1 waiter=0 authpin=1 0x55c68bc4c000]", 2024-04-23T10:32:04.133 INFO:teuthology.orchestra.run.smithi114.stdout: "lock": { 2024-04-23T10:32:04.133 INFO:teuthology.orchestra.run.smithi114.stdout: "gather_set": [], 2024-04-23T10:32:04.133 INFO:teuthology.orchestra.run.smithi114.stdout: "state": "lock", 2024-04-23T10:32:04.134 INFO:teuthology.orchestra.run.smithi114.stdout: "type": "iquiesce", 2024-04-23T10:32:04.134 INFO:teuthology.orchestra.run.smithi114.stdout: "is_leased": false, 2024-04-23T10:32:04.134 INFO:teuthology.orchestra.run.smithi114.stdout: "num_rdlocks": 0, 2024-04-23T10:32:04.134 INFO:teuthology.orchestra.run.smithi114.stdout: "num_wrlocks": 0, 2024-04-23T10:32:04.134 INFO:teuthology.orchestra.run.smithi114.stdout: "num_xlocks": 1, 2024-04-23T10:32:04.134 INFO:teuthology.orchestra.run.smithi114.stdout: "xlock_by": { 2024-04-23T10:32:04.134 INFO:teuthology.orchestra.run.smithi114.stdout: "reqid": { 2024-04-23T10:32:04.134 INFO:teuthology.orchestra.run.smithi114.stdout: "entity": { 2024-04-23T10:32:04.134 INFO:teuthology.orchestra.run.smithi114.stdout: "type": "mds", 2024-04-23T10:32:04.134 INFO:teuthology.orchestra.run.smithi114.stdout: "num": 2 2024-04-23T10:32:04.134 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-23T10:32:04.134 INFO:teuthology.orchestra.run.smithi114.stdout: "tid": 130 2024-04-23T10:32:04.134 INFO:teuthology.orchestra.run.smithi114.stdout: } 2024-04-23T10:32:04.134 INFO:teuthology.orchestra.run.smithi114.stdout: } 2024-04-23T10:32:04.135 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-23T10:32:04.135 INFO:teuthology.orchestra.run.smithi114.stdout: "flags": 4, 2024-04-23T10:32:04.135 INFO:teuthology.orchestra.run.smithi114.stdout: "wrlock_target": -1 2024-04-23T10:32:04.135 INFO:teuthology.orchestra.run.smithi114.stdout: } 2024-04-23T10:32:04.135 INFO:teuthology.orchestra.run.smithi114.stdout: ] 2024-04-23T10:32:04.135 INFO:teuthology.orchestra.run.smithi114.stdout: } 2024-04-23T10:32:04.135 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-23T10:32:04.135 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-04-23T10:32:04.135 INFO:teuthology.orchestra.run.smithi114.stdout: "description": "internal op quiesce_inode:mds.2:131 fp=#0x10000000218", 2024-04-23T10:32:04.135 INFO:teuthology.orchestra.run.smithi114.stdout: "initiated_at": "2024-04-23T10:31:21.477962+0000", 2024-04-23T10:32:04.135 INFO:teuthology.orchestra.run.smithi114.stdout: "age": 42.350782817000002, 2024-04-23T10:32:04.135 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 42.353634391, 2024-04-23T10:32:04.135 INFO:teuthology.orchestra.run.smithi114.stdout: "continuous": true, 2024-04-23T10:32:04.135 INFO:teuthology.orchestra.run.smithi114.stdout: "type_data": { 2024-04-23T10:32:04.136 INFO:teuthology.orchestra.run.smithi114.stdout: "result": -2147483648, 2024-04-23T10:32:04.136 INFO:teuthology.orchestra.run.smithi114.stdout: "flag_point": "quiesce complete", 2024-04-23T10:32:04.136 INFO:teuthology.orchestra.run.smithi114.stdout: "reqid": { 2024-04-23T10:32:04.136 INFO:teuthology.orchestra.run.smithi114.stdout: "entity": { 2024-04-23T10:32:04.136 INFO:teuthology.orchestra.run.smithi114.stdout: "type": "mds", 2024-04-23T10:32:04.136 INFO:teuthology.orchestra.run.smithi114.stdout: "num": 2 2024-04-23T10:32:04.136 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-23T10:32:04.136 INFO:teuthology.orchestra.run.smithi114.stdout: "tid": 131 2024-04-23T10:32:04.136 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-23T10:32:04.136 INFO:teuthology.orchestra.run.smithi114.stdout: "op_type": "internal_op", 2024-04-23T10:32:04.136 INFO:teuthology.orchestra.run.smithi114.stdout: "internal_op": 5385, 2024-04-23T10:32:04.136 INFO:teuthology.orchestra.run.smithi114.stdout: "op_name": "quiesce_inode", 2024-04-23T10:32:04.136 INFO:teuthology.orchestra.run.smithi114.stdout: "events": [ 2024-04-23T10:32:04.136 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-04-23T10:32:04.136 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-04-23T10:31:21.477962+0000", 2024-04-23T10:32:04.137 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "initiated" 2024-04-23T10:32:04.137 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-23T10:32:04.137 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-04-23T10:32:04.137 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-04-23T10:31:21.477962+0000", 2024-04-23T10:32:04.137 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "header_read" 2024-04-23T10:32:04.137 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-23T10:32:04.137 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-04-23T10:32:04.137 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-04-23T10:31:21.477962+0000", 2024-04-23T10:32:04.137 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "throttled" 2024-04-23T10:32:04.137 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-23T10:32:04.137 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-04-23T10:32:04.137 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-04-23T10:31:21.477962+0000", 2024-04-23T10:32:04.137 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "all_read" 2024-04-23T10:32:04.137 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-23T10:32:04.137 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-04-23T10:32:04.138 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-04-23T10:31:21.477962+0000", 2024-04-23T10:32:04.138 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "dispatched" 2024-04-23T10:32:04.138 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-23T10:32:04.138 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-04-23T10:32:04.138 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-04-23T10:31:21.479962+0000", 2024-04-23T10:32:04.138 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "failed to xlock, waiting" 2024-04-23T10:32:04.138 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-23T10:32:04.138 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-04-23T10:32:04.138 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-04-23T10:31:21.485114+0000", 2024-04-23T10:32:04.138 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "acquired locks" 2024-04-23T10:32:04.138 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-23T10:32:04.138 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-04-23T10:32:04.138 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-04-23T10:31:21.485237+0000", 2024-04-23T10:32:04.138 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "quiesce complete" 2024-04-23T10:32:04.139 INFO:teuthology.orchestra.run.smithi114.stdout: } 2024-04-23T10:32:04.139 INFO:teuthology.orchestra.run.smithi114.stdout: ], 2024-04-23T10:32:04.139 INFO:teuthology.orchestra.run.smithi114.stdout: "locks": [ 2024-04-23T10:32:04.139 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-04-23T10:32:04.139 INFO:teuthology.orchestra.run.smithi114.stdout: "object": { 2024-04-23T10:32:04.139 INFO:teuthology.orchestra.run.smithi114.stdout: "is_auth": true, 2024-04-23T10:32:04.139 INFO:teuthology.orchestra.run.smithi114.stdout: "auth_state": { 2024-04-23T10:32:04.139 INFO:teuthology.orchestra.run.smithi114.stdout: "replicas": { 2024-04-23T10:32:04.139 INFO:teuthology.orchestra.run.smithi114.stdout: "0": 1 2024-04-23T10:32:04.139 INFO:teuthology.orchestra.run.smithi114.stdout: } 2024-04-23T10:32:04.139 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-23T10:32:04.139 INFO:teuthology.orchestra.run.smithi114.stdout: "replica_state": { 2024-04-23T10:32:04.139 INFO:teuthology.orchestra.run.smithi114.stdout: "authority": [ 2024-04-23T10:32:04.139 INFO:teuthology.orchestra.run.smithi114.stdout: 2, 2024-04-23T10:32:04.139 INFO:teuthology.orchestra.run.smithi114.stdout: -2 2024-04-23T10:32:04.140 INFO:teuthology.orchestra.run.smithi114.stdout: ], 2024-04-23T10:32:04.140 INFO:teuthology.orchestra.run.smithi114.stdout: "replica_nonce": 0 2024-04-23T10:32:04.140 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-23T10:32:04.140 INFO:teuthology.orchestra.run.smithi114.stdout: "auth_pins": 2, 2024-04-23T10:32:04.140 INFO:teuthology.orchestra.run.smithi114.stdout: "is_frozen": false, 2024-04-23T10:32:04.140 INFO:teuthology.orchestra.run.smithi114.stdout: "is_freezing": false, 2024-04-23T10:32:04.140 INFO:teuthology.orchestra.run.smithi114.stdout: "pins": { 2024-04-23T10:32:04.140 INFO:teuthology.orchestra.run.smithi114.stdout: "ptrwaiter": 0, 2024-04-23T10:32:04.140 INFO:teuthology.orchestra.run.smithi114.stdout: "request": 1, 2024-04-23T10:32:04.140 INFO:teuthology.orchestra.run.smithi114.stdout: "lock": 1, 2024-04-23T10:32:04.140 INFO:teuthology.orchestra.run.smithi114.stdout: "importingcaps": 0, 2024-04-23T10:32:04.140 INFO:teuthology.orchestra.run.smithi114.stdout: "caps": 0, 2024-04-23T10:32:04.140 INFO:teuthology.orchestra.run.smithi114.stdout: "needsnapflush": 0, 2024-04-23T10:32:04.140 INFO:teuthology.orchestra.run.smithi114.stdout: "replicated": 1, 2024-04-23T10:32:04.141 INFO:teuthology.orchestra.run.smithi114.stdout: "dirty": 1, 2024-04-23T10:32:04.141 INFO:teuthology.orchestra.run.smithi114.stdout: "waiter": 0, 2024-04-23T10:32:04.141 INFO:teuthology.orchestra.run.smithi114.stdout: "authpin": 1 2024-04-23T10:32:04.141 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-23T10:32:04.141 INFO:teuthology.orchestra.run.smithi114.stdout: "nref": 5 2024-04-23T10:32:04.141 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-23T10:32:04.141 INFO:teuthology.orchestra.run.smithi114.stdout: "object_string": "[inode 0x10000000218 [7,head] /volumes/qa/sv_1/b206794e-7622-4ab8-8943-f3ae4e65b7a3/client.0/tmp/clients/client0/~dmtmp/COREL/ARTISTIC.ACL auth{0=1} v7240 ap=2 s=260096 n(v0 rc2024-04-23T10:27:52.014396+0000 b260096 1=1+0) (iquiesce lock x=1 by request(mds.2:131 nref=3)) | ptrwaiter=0 request=1 lock=1 importingcaps=0 caps=0 needsnapflush=0 replicated=1 dirty=1 waiter=0 authpin=1 0x55c68bc4c580]", 2024-04-23T10:32:04.141 INFO:teuthology.orchestra.run.smithi114.stdout: "lock": { 2024-04-23T10:32:04.141 INFO:teuthology.orchestra.run.smithi114.stdout: "gather_set": [], 2024-04-23T10:32:04.141 INFO:teuthology.orchestra.run.smithi114.stdout: "state": "lock", 2024-04-23T10:32:04.141 INFO:teuthology.orchestra.run.smithi114.stdout: "type": "iquiesce", 2024-04-23T10:32:04.141 INFO:teuthology.orchestra.run.smithi114.stdout: "is_leased": false, 2024-04-23T10:32:04.141 INFO:teuthology.orchestra.run.smithi114.stdout: "num_rdlocks": 0, 2024-04-23T10:32:04.141 INFO:teuthology.orchestra.run.smithi114.stdout: "num_wrlocks": 0, 2024-04-23T10:32:04.141 INFO:teuthology.orchestra.run.smithi114.stdout: "num_xlocks": 1, 2024-04-23T10:32:04.141 INFO:teuthology.orchestra.run.smithi114.stdout: "xlock_by": { 2024-04-23T10:32:04.142 INFO:teuthology.orchestra.run.smithi114.stdout: "reqid": { 2024-04-23T10:32:04.142 INFO:teuthology.orchestra.run.smithi114.stdout: "entity": { 2024-04-23T10:32:04.142 INFO:teuthology.orchestra.run.smithi114.stdout: "type": "mds", 2024-04-23T10:32:04.142 INFO:teuthology.orchestra.run.smithi114.stdout: "num": 2 2024-04-23T10:32:04.142 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-23T10:32:04.142 INFO:teuthology.orchestra.run.smithi114.stdout: "tid": 131 2024-04-23T10:32:04.142 INFO:teuthology.orchestra.run.smithi114.stdout: } 2024-04-23T10:32:04.142 INFO:teuthology.orchestra.run.smithi114.stdout: } 2024-04-23T10:32:04.142 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-23T10:32:04.142 INFO:teuthology.orchestra.run.smithi114.stdout: "flags": 4, 2024-04-23T10:32:04.142 INFO:teuthology.orchestra.run.smithi114.stdout: "wrlock_target": -1 2024-04-23T10:32:04.142 INFO:teuthology.orchestra.run.smithi114.stdout: } 2024-04-23T10:32:04.142 INFO:teuthology.orchestra.run.smithi114.stdout: ] 2024-04-23T10:32:04.142 INFO:teuthology.orchestra.run.smithi114.stdout: } 2024-04-23T10:32:04.143 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-23T10:32:04.143 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-04-23T10:32:04.143 INFO:teuthology.orchestra.run.smithi114.stdout: "description": "internal op quiesce_inode:mds.2:132 fp=#0x1000000021c", 2024-04-23T10:32:04.143 INFO:teuthology.orchestra.run.smithi114.stdout: "initiated_at": "2024-04-23T10:31:21.477991+0000", 2024-04-23T10:32:04.143 INFO:teuthology.orchestra.run.smithi114.stdout: "age": 42.350753672000003, 2024-04-23T10:32:04.143 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 42.353669670000002, 2024-04-23T10:32:04.143 INFO:teuthology.orchestra.run.smithi114.stdout: "continuous": true, 2024-04-23T10:32:04.143 INFO:teuthology.orchestra.run.smithi114.stdout: "type_data": { 2024-04-23T10:32:04.143 INFO:teuthology.orchestra.run.smithi114.stdout: "result": -2147483648, 2024-04-23T10:32:04.143 INFO:teuthology.orchestra.run.smithi114.stdout: "flag_point": "quiesce complete", 2024-04-23T10:32:04.143 INFO:teuthology.orchestra.run.smithi114.stdout: "reqid": { 2024-04-23T10:32:04.143 INFO:teuthology.orchestra.run.smithi114.stdout: "entity": { 2024-04-23T10:32:04.143 INFO:teuthology.orchestra.run.smithi114.stdout: "type": "mds", 2024-04-23T10:32:04.143 INFO:teuthology.orchestra.run.smithi114.stdout: "num": 2 2024-04-23T10:32:04.143 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-23T10:32:04.143 INFO:teuthology.orchestra.run.smithi114.stdout: "tid": 132 2024-04-23T10:32:04.144 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-23T10:32:04.144 INFO:teuthology.orchestra.run.smithi114.stdout: "op_type": "internal_op", 2024-04-23T10:32:04.144 INFO:teuthology.orchestra.run.smithi114.stdout: "internal_op": 5385, 2024-04-23T10:32:04.144 INFO:teuthology.orchestra.run.smithi114.stdout: "op_name": "quiesce_inode", 2024-04-23T10:32:04.144 INFO:teuthology.orchestra.run.smithi114.stdout: "events": [ 2024-04-23T10:32:04.144 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-04-23T10:32:04.144 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-04-23T10:31:21.477991+0000", 2024-04-23T10:32:04.144 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "initiated" 2024-04-23T10:32:04.144 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-23T10:32:04.144 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-04-23T10:32:04.144 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-04-23T10:31:21.477991+0000", 2024-04-23T10:32:04.144 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "header_read" 2024-04-23T10:32:04.144 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-23T10:32:04.144 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-04-23T10:32:04.144 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-04-23T10:31:21.477991+0000", 2024-04-23T10:32:04.145 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "throttled" 2024-04-23T10:32:04.145 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-23T10:32:04.145 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-04-23T10:32:04.145 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-04-23T10:31:21.477991+0000", 2024-04-23T10:32:04.145 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "all_read" 2024-04-23T10:32:04.145 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-23T10:32:04.145 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-04-23T10:32:04.145 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-04-23T10:31:21.477991+0000", 2024-04-23T10:32:04.145 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "dispatched" 2024-04-23T10:32:04.145 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-23T10:32:04.145 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-04-23T10:32:04.145 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-04-23T10:31:21.480251+0000", 2024-04-23T10:32:04.145 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "failed to xlock, waiting" 2024-04-23T10:32:04.145 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-23T10:32:04.145 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-04-23T10:32:04.146 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-04-23T10:31:21.485677+0000", 2024-04-23T10:32:04.146 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "acquired locks" 2024-04-23T10:32:04.146 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-23T10:32:04.146 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-04-23T10:32:04.146 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-04-23T10:31:21.485823+0000", 2024-04-23T10:32:04.146 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "quiesce complete" 2024-04-23T10:32:04.146 INFO:teuthology.orchestra.run.smithi114.stdout: } 2024-04-23T10:32:04.146 INFO:teuthology.orchestra.run.smithi114.stdout: ], 2024-04-23T10:32:04.146 INFO:teuthology.orchestra.run.smithi114.stdout: "locks": [ 2024-04-23T10:32:04.146 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-04-23T10:32:04.146 INFO:teuthology.orchestra.run.smithi114.stdout: "object": { 2024-04-23T10:32:04.146 INFO:teuthology.orchestra.run.smithi114.stdout: "is_auth": true, 2024-04-23T10:32:04.146 INFO:teuthology.orchestra.run.smithi114.stdout: "auth_state": { 2024-04-23T10:32:04.146 INFO:teuthology.orchestra.run.smithi114.stdout: "replicas": { 2024-04-23T10:32:04.147 INFO:teuthology.orchestra.run.smithi114.stdout: "0": 1 2024-04-23T10:32:04.147 INFO:teuthology.orchestra.run.smithi114.stdout: } 2024-04-23T10:32:04.147 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-23T10:32:04.147 INFO:teuthology.orchestra.run.smithi114.stdout: "replica_state": { 2024-04-23T10:32:04.147 INFO:teuthology.orchestra.run.smithi114.stdout: "authority": [ 2024-04-23T10:32:04.147 INFO:teuthology.orchestra.run.smithi114.stdout: 2, 2024-04-23T10:32:04.147 INFO:teuthology.orchestra.run.smithi114.stdout: -2 2024-04-23T10:32:04.147 INFO:teuthology.orchestra.run.smithi114.stdout: ], 2024-04-23T10:32:04.147 INFO:teuthology.orchestra.run.smithi114.stdout: "replica_nonce": 0 2024-04-23T10:32:04.147 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-23T10:32:04.147 INFO:teuthology.orchestra.run.smithi114.stdout: "auth_pins": 2, 2024-04-23T10:32:04.147 INFO:teuthology.orchestra.run.smithi114.stdout: "is_frozen": false, 2024-04-23T10:32:04.147 INFO:teuthology.orchestra.run.smithi114.stdout: "is_freezing": false, 2024-04-23T10:32:04.147 INFO:teuthology.orchestra.run.smithi114.stdout: "pins": { 2024-04-23T10:32:04.147 INFO:teuthology.orchestra.run.smithi114.stdout: "ptrwaiter": 0, 2024-04-23T10:32:04.148 INFO:teuthology.orchestra.run.smithi114.stdout: "request": 1, 2024-04-23T10:32:04.148 INFO:teuthology.orchestra.run.smithi114.stdout: "lock": 1, 2024-04-23T10:32:04.148 INFO:teuthology.orchestra.run.smithi114.stdout: "importingcaps": 0, 2024-04-23T10:32:04.148 INFO:teuthology.orchestra.run.smithi114.stdout: "caps": 0, 2024-04-23T10:32:04.148 INFO:teuthology.orchestra.run.smithi114.stdout: "replicated": 1, 2024-04-23T10:32:04.148 INFO:teuthology.orchestra.run.smithi114.stdout: "dirty": 1, 2024-04-23T10:32:04.148 INFO:teuthology.orchestra.run.smithi114.stdout: "waiter": 0, 2024-04-23T10:32:04.148 INFO:teuthology.orchestra.run.smithi114.stdout: "authpin": 1 2024-04-23T10:32:04.148 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-23T10:32:04.148 INFO:teuthology.orchestra.run.smithi114.stdout: "nref": 5 2024-04-23T10:32:04.148 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-23T10:32:04.148 INFO:teuthology.orchestra.run.smithi114.stdout: "object_string": "[inode 0x1000000021c [7,head] /volumes/qa/sv_1/b206794e-7622-4ab8-8943-f3ae4e65b7a3/client.0/tmp/clients/client0/~dmtmp/COREL/SIGN2.CPT auth{0=1} v7250 ap=2 s=260096 n(v0 rc2024-04-23T10:27:52.033396+0000 b260096 1=1+0) (iquiesce lock x=1 by request(mds.2:132 nref=3)) | ptrwaiter=0 request=1 lock=1 importingcaps=0 caps=0 replicated=1 dirty=1 waiter=0 authpin=1 0x55c68bc4cb00]", 2024-04-23T10:32:04.148 INFO:teuthology.orchestra.run.smithi114.stdout: "lock": { 2024-04-23T10:32:04.148 INFO:teuthology.orchestra.run.smithi114.stdout: "gather_set": [], 2024-04-23T10:32:04.148 INFO:teuthology.orchestra.run.smithi114.stdout: "state": "lock", 2024-04-23T10:32:04.149 INFO:teuthology.orchestra.run.smithi114.stdout: "type": "iquiesce", 2024-04-23T10:32:04.149 INFO:teuthology.orchestra.run.smithi114.stdout: "is_leased": false, 2024-04-23T10:32:04.149 INFO:teuthology.orchestra.run.smithi114.stdout: "num_rdlocks": 0, 2024-04-23T10:32:04.149 INFO:teuthology.orchestra.run.smithi114.stdout: "num_wrlocks": 0, 2024-04-23T10:32:04.149 INFO:teuthology.orchestra.run.smithi114.stdout: "num_xlocks": 1, 2024-04-23T10:32:04.149 INFO:teuthology.orchestra.run.smithi114.stdout: "xlock_by": { 2024-04-23T10:32:04.149 INFO:teuthology.orchestra.run.smithi114.stdout: "reqid": { 2024-04-23T10:32:04.149 INFO:teuthology.orchestra.run.smithi114.stdout: "entity": { 2024-04-23T10:32:04.149 INFO:teuthology.orchestra.run.smithi114.stdout: "type": "mds", 2024-04-23T10:32:04.149 INFO:teuthology.orchestra.run.smithi114.stdout: "num": 2 2024-04-23T10:32:04.149 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-23T10:32:04.149 INFO:teuthology.orchestra.run.smithi114.stdout: "tid": 132 2024-04-23T10:32:04.149 INFO:teuthology.orchestra.run.smithi114.stdout: } 2024-04-23T10:32:04.149 INFO:teuthology.orchestra.run.smithi114.stdout: } 2024-04-23T10:32:04.150 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-23T10:32:04.150 INFO:teuthology.orchestra.run.smithi114.stdout: "flags": 4, 2024-04-23T10:32:04.150 INFO:teuthology.orchestra.run.smithi114.stdout: "wrlock_target": -1 2024-04-23T10:32:04.150 INFO:teuthology.orchestra.run.smithi114.stdout: } 2024-04-23T10:32:04.150 INFO:teuthology.orchestra.run.smithi114.stdout: ] 2024-04-23T10:32:04.150 INFO:teuthology.orchestra.run.smithi114.stdout: } 2024-04-23T10:32:04.150 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-23T10:32:04.150 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-04-23T10:32:04.150 INFO:teuthology.orchestra.run.smithi114.stdout: "description": "internal op quiesce_inode:mds.2:133 fp=#0x10000000217", 2024-04-23T10:32:04.150 INFO:teuthology.orchestra.run.smithi114.stdout: "initiated_at": "2024-04-23T10:31:21.478006+0000", 2024-04-23T10:32:04.150 INFO:teuthology.orchestra.run.smithi114.stdout: "age": 42.350738485000001, 2024-04-23T10:32:04.150 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 42.353734189999997, 2024-04-23T10:32:04.150 INFO:teuthology.orchestra.run.smithi114.stdout: "continuous": true, 2024-04-23T10:32:04.150 INFO:teuthology.orchestra.run.smithi114.stdout: "type_data": { 2024-04-23T10:32:04.151 INFO:teuthology.orchestra.run.smithi114.stdout: "result": -2147483648, 2024-04-23T10:32:04.151 INFO:teuthology.orchestra.run.smithi114.stdout: "flag_point": "quiesce complete", 2024-04-23T10:32:04.151 INFO:teuthology.orchestra.run.smithi114.stdout: "reqid": { 2024-04-23T10:32:04.151 INFO:teuthology.orchestra.run.smithi114.stdout: "entity": { 2024-04-23T10:32:04.151 INFO:teuthology.orchestra.run.smithi114.stdout: "type": "mds", 2024-04-23T10:32:04.151 INFO:teuthology.orchestra.run.smithi114.stdout: "num": 2 2024-04-23T10:32:04.151 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-23T10:32:04.151 INFO:teuthology.orchestra.run.smithi114.stdout: "tid": 133 2024-04-23T10:32:04.151 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-23T10:32:04.151 INFO:teuthology.orchestra.run.smithi114.stdout: "op_type": "internal_op", 2024-04-23T10:32:04.151 INFO:teuthology.orchestra.run.smithi114.stdout: "internal_op": 5385, 2024-04-23T10:32:04.151 INFO:teuthology.orchestra.run.smithi114.stdout: "op_name": "quiesce_inode", 2024-04-23T10:32:04.151 INFO:teuthology.orchestra.run.smithi114.stdout: "events": [ 2024-04-23T10:32:04.151 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-04-23T10:32:04.151 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-04-23T10:31:21.478006+0000", 2024-04-23T10:32:04.152 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "initiated" 2024-04-23T10:32:04.152 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-23T10:32:04.152 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-04-23T10:32:04.152 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-04-23T10:31:21.478006+0000", 2024-04-23T10:32:04.152 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "header_read" 2024-04-23T10:32:04.152 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-23T10:32:04.152 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-04-23T10:32:04.152 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-04-23T10:31:21.478006+0000", 2024-04-23T10:32:04.152 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "throttled" 2024-04-23T10:32:04.152 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-23T10:32:04.152 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-04-23T10:32:04.152 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-04-23T10:31:21.478006+0000", 2024-04-23T10:32:04.152 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "all_read" 2024-04-23T10:32:04.152 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-23T10:32:04.152 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-04-23T10:32:04.153 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-04-23T10:31:21.478006+0000", 2024-04-23T10:32:04.153 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "dispatched" 2024-04-23T10:32:04.153 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-23T10:32:04.153 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-04-23T10:32:04.153 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-04-23T10:31:21.480540+0000", 2024-04-23T10:32:04.153 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "failed to xlock, waiting" 2024-04-23T10:32:04.153 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-23T10:32:04.153 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-04-23T10:32:04.153 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-04-23T10:31:21.486581+0000", 2024-04-23T10:32:04.153 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "failed to xlock, waiting" 2024-04-23T10:32:04.153 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-23T10:32:04.153 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-04-23T10:32:04.153 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-04-23T10:31:21.505374+0000", 2024-04-23T10:32:04.153 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "acquired locks" 2024-04-23T10:32:04.153 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-23T10:32:04.154 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-04-23T10:32:04.154 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-04-23T10:31:21.505510+0000", 2024-04-23T10:32:04.154 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "quiesce complete" 2024-04-23T10:32:04.154 INFO:teuthology.orchestra.run.smithi114.stdout: } 2024-04-23T10:32:04.154 INFO:teuthology.orchestra.run.smithi114.stdout: ], 2024-04-23T10:32:04.154 INFO:teuthology.orchestra.run.smithi114.stdout: "locks": [ 2024-04-23T10:32:04.154 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-04-23T10:32:04.154 INFO:teuthology.orchestra.run.smithi114.stdout: "object": { 2024-04-23T10:32:04.154 INFO:teuthology.orchestra.run.smithi114.stdout: "is_auth": true, 2024-04-23T10:32:04.154 INFO:teuthology.orchestra.run.smithi114.stdout: "auth_state": { 2024-04-23T10:32:04.154 INFO:teuthology.orchestra.run.smithi114.stdout: "replicas": { 2024-04-23T10:32:04.154 INFO:teuthology.orchestra.run.smithi114.stdout: "0": 1 2024-04-23T10:32:04.154 INFO:teuthology.orchestra.run.smithi114.stdout: } 2024-04-23T10:32:04.154 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-23T10:32:04.154 INFO:teuthology.orchestra.run.smithi114.stdout: "replica_state": { 2024-04-23T10:32:04.155 INFO:teuthology.orchestra.run.smithi114.stdout: "authority": [ 2024-04-23T10:32:04.155 INFO:teuthology.orchestra.run.smithi114.stdout: 2, 2024-04-23T10:32:04.155 INFO:teuthology.orchestra.run.smithi114.stdout: -2 2024-04-23T10:32:04.155 INFO:teuthology.orchestra.run.smithi114.stdout: ], 2024-04-23T10:32:04.155 INFO:teuthology.orchestra.run.smithi114.stdout: "replica_nonce": 0 2024-04-23T10:32:04.155 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-23T10:32:04.155 INFO:teuthology.orchestra.run.smithi114.stdout: "auth_pins": 2, 2024-04-23T10:32:04.155 INFO:teuthology.orchestra.run.smithi114.stdout: "is_frozen": false, 2024-04-23T10:32:04.155 INFO:teuthology.orchestra.run.smithi114.stdout: "is_freezing": false, 2024-04-23T10:32:04.155 INFO:teuthology.orchestra.run.smithi114.stdout: "pins": { 2024-04-23T10:32:04.155 INFO:teuthology.orchestra.run.smithi114.stdout: "ptrwaiter": 0, 2024-04-23T10:32:04.155 INFO:teuthology.orchestra.run.smithi114.stdout: "request": 1, 2024-04-23T10:32:04.155 INFO:teuthology.orchestra.run.smithi114.stdout: "lock": 1, 2024-04-23T10:32:04.155 INFO:teuthology.orchestra.run.smithi114.stdout: "importingcaps": 0, 2024-04-23T10:32:04.155 INFO:teuthology.orchestra.run.smithi114.stdout: "caps": 0, 2024-04-23T10:32:04.156 INFO:teuthology.orchestra.run.smithi114.stdout: "needsnapflush": 0, 2024-04-23T10:32:04.156 INFO:teuthology.orchestra.run.smithi114.stdout: "replicated": 1, 2024-04-23T10:32:04.156 INFO:teuthology.orchestra.run.smithi114.stdout: "dirty": 1, 2024-04-23T10:32:04.156 INFO:teuthology.orchestra.run.smithi114.stdout: "waiter": 0, 2024-04-23T10:32:04.156 INFO:teuthology.orchestra.run.smithi114.stdout: "authpin": 1 2024-04-23T10:32:04.156 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-23T10:32:04.156 INFO:teuthology.orchestra.run.smithi114.stdout: "nref": 5 2024-04-23T10:32:04.156 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-23T10:32:04.156 INFO:teuthology.orchestra.run.smithi114.stdout: "object_string": "[inode 0x10000000217 [7,head] /volumes/qa/sv_1/b206794e-7622-4ab8-8943-f3ae4e65b7a3/client.0/tmp/clients/client0/~dmtmp/COREL/CDRBARS.CFG auth{0=1} v7236 ap=2 s=260096 n(v0 rc2024-04-23T10:31:17.211728+0000 b260096 1=1+0) (iquiesce lock x=1 by request(mds.2:133 nref=3)) | ptrwaiter=0 request=1 lock=1 importingcaps=0 caps=0 needsnapflush=0 replicated=1 dirty=1 waiter=0 authpin=1 0x55c68bc4d080]", 2024-04-23T10:32:04.156 INFO:teuthology.orchestra.run.smithi114.stdout: "lock": { 2024-04-23T10:32:04.156 INFO:teuthology.orchestra.run.smithi114.stdout: "gather_set": [], 2024-04-23T10:32:04.156 INFO:teuthology.orchestra.run.smithi114.stdout: "state": "lock", 2024-04-23T10:32:04.156 INFO:teuthology.orchestra.run.smithi114.stdout: "type": "iquiesce", 2024-04-23T10:32:04.156 INFO:teuthology.orchestra.run.smithi114.stdout: "is_leased": false, 2024-04-23T10:32:04.157 INFO:teuthology.orchestra.run.smithi114.stdout: "num_rdlocks": 0, 2024-04-23T10:32:04.157 INFO:teuthology.orchestra.run.smithi114.stdout: "num_wrlocks": 0, 2024-04-23T10:32:04.157 INFO:teuthology.orchestra.run.smithi114.stdout: "num_xlocks": 1, 2024-04-23T10:32:04.157 INFO:teuthology.orchestra.run.smithi114.stdout: "xlock_by": { 2024-04-23T10:32:04.157 INFO:teuthology.orchestra.run.smithi114.stdout: "reqid": { 2024-04-23T10:32:04.157 INFO:teuthology.orchestra.run.smithi114.stdout: "entity": { 2024-04-23T10:32:04.157 INFO:teuthology.orchestra.run.smithi114.stdout: "type": "mds", 2024-04-23T10:32:04.157 INFO:teuthology.orchestra.run.smithi114.stdout: "num": 2 2024-04-23T10:32:04.157 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-23T10:32:04.157 INFO:teuthology.orchestra.run.smithi114.stdout: "tid": 133 2024-04-23T10:32:04.157 INFO:teuthology.orchestra.run.smithi114.stdout: } 2024-04-23T10:32:04.157 INFO:teuthology.orchestra.run.smithi114.stdout: } 2024-04-23T10:32:04.157 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-23T10:32:04.157 INFO:teuthology.orchestra.run.smithi114.stdout: "flags": 4, 2024-04-23T10:32:04.157 INFO:teuthology.orchestra.run.smithi114.stdout: "wrlock_target": -1 2024-04-23T10:32:04.158 INFO:teuthology.orchestra.run.smithi114.stdout: } 2024-04-23T10:32:04.158 INFO:teuthology.orchestra.run.smithi114.stdout: ] 2024-04-23T10:32:04.158 INFO:teuthology.orchestra.run.smithi114.stdout: } 2024-04-23T10:32:04.158 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-23T10:32:04.158 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-04-23T10:32:04.158 INFO:teuthology.orchestra.run.smithi114.stdout: "description": "internal op quiesce_inode:mds.2:134 fp=#0x1000000021d", 2024-04-23T10:32:04.158 INFO:teuthology.orchestra.run.smithi114.stdout: "initiated_at": "2024-04-23T10:31:21.478022+0000", 2024-04-23T10:32:04.158 INFO:teuthology.orchestra.run.smithi114.stdout: "age": 42.350722484000002, 2024-04-23T10:32:04.158 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 42.353781849000001, 2024-04-23T10:32:04.158 INFO:teuthology.orchestra.run.smithi114.stdout: "continuous": true, 2024-04-23T10:32:04.158 INFO:teuthology.orchestra.run.smithi114.stdout: "type_data": { 2024-04-23T10:32:04.158 INFO:teuthology.orchestra.run.smithi114.stdout: "result": -2147483648, 2024-04-23T10:32:04.158 INFO:teuthology.orchestra.run.smithi114.stdout: "flag_point": "quiesce complete", 2024-04-23T10:32:04.158 INFO:teuthology.orchestra.run.smithi114.stdout: "reqid": { 2024-04-23T10:32:04.158 INFO:teuthology.orchestra.run.smithi114.stdout: "entity": { 2024-04-23T10:32:04.159 INFO:teuthology.orchestra.run.smithi114.stdout: "type": "mds", 2024-04-23T10:32:04.159 INFO:teuthology.orchestra.run.smithi114.stdout: "num": 2 2024-04-23T10:32:04.159 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-23T10:32:04.159 INFO:teuthology.orchestra.run.smithi114.stdout: "tid": 134 2024-04-23T10:32:04.159 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-23T10:32:04.159 INFO:teuthology.orchestra.run.smithi114.stdout: "op_type": "internal_op", 2024-04-23T10:32:04.159 INFO:teuthology.orchestra.run.smithi114.stdout: "internal_op": 5385, 2024-04-23T10:32:04.159 INFO:teuthology.orchestra.run.smithi114.stdout: "op_name": "quiesce_inode", 2024-04-23T10:32:04.159 INFO:teuthology.orchestra.run.smithi114.stdout: "events": [ 2024-04-23T10:32:04.159 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-04-23T10:32:04.159 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-04-23T10:31:21.478022+0000", 2024-04-23T10:32:04.159 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "initiated" 2024-04-23T10:32:04.159 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-23T10:32:04.159 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-04-23T10:32:04.159 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-04-23T10:31:21.478022+0000", 2024-04-23T10:32:04.160 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "header_read" 2024-04-23T10:32:04.160 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-23T10:32:04.160 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-04-23T10:32:04.160 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-04-23T10:31:21.478022+0000", 2024-04-23T10:32:04.160 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "throttled" 2024-04-23T10:32:04.160 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-23T10:32:04.160 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-04-23T10:32:04.160 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-04-23T10:31:21.478022+0000", 2024-04-23T10:32:04.160 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "all_read" 2024-04-23T10:32:04.160 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-23T10:32:04.160 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-04-23T10:32:04.160 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-04-23T10:31:21.478022+0000", 2024-04-23T10:32:04.160 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "dispatched" 2024-04-23T10:32:04.160 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-23T10:32:04.161 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-04-23T10:32:04.161 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-04-23T10:31:21.480740+0000", 2024-04-23T10:32:04.161 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "failed to xlock, waiting" 2024-04-23T10:32:04.161 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-23T10:32:04.161 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-04-23T10:32:04.161 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-04-23T10:31:21.487025+0000", 2024-04-23T10:32:04.161 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "acquired locks" 2024-04-23T10:32:04.161 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-23T10:32:04.161 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-04-23T10:32:04.161 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-04-23T10:31:21.487146+0000", 2024-04-23T10:32:04.161 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "quiesce complete" 2024-04-23T10:32:04.161 INFO:teuthology.orchestra.run.smithi114.stdout: } 2024-04-23T10:32:04.161 INFO:teuthology.orchestra.run.smithi114.stdout: ], 2024-04-23T10:32:04.161 INFO:teuthology.orchestra.run.smithi114.stdout: "locks": [ 2024-04-23T10:32:04.161 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-04-23T10:32:04.162 INFO:teuthology.orchestra.run.smithi114.stdout: "object": { 2024-04-23T10:32:04.162 INFO:teuthology.orchestra.run.smithi114.stdout: "is_auth": true, 2024-04-23T10:32:04.162 INFO:teuthology.orchestra.run.smithi114.stdout: "auth_state": { 2024-04-23T10:32:04.162 INFO:teuthology.orchestra.run.smithi114.stdout: "replicas": { 2024-04-23T10:32:04.162 INFO:teuthology.orchestra.run.smithi114.stdout: "0": 1 2024-04-23T10:32:04.162 INFO:teuthology.orchestra.run.smithi114.stdout: } 2024-04-23T10:32:04.162 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-23T10:32:04.162 INFO:teuthology.orchestra.run.smithi114.stdout: "replica_state": { 2024-04-23T10:32:04.162 INFO:teuthology.orchestra.run.smithi114.stdout: "authority": [ 2024-04-23T10:32:04.162 INFO:teuthology.orchestra.run.smithi114.stdout: 2, 2024-04-23T10:32:04.162 INFO:teuthology.orchestra.run.smithi114.stdout: -2 2024-04-23T10:32:04.162 INFO:teuthology.orchestra.run.smithi114.stdout: ], 2024-04-23T10:32:04.162 INFO:teuthology.orchestra.run.smithi114.stdout: "replica_nonce": 0 2024-04-23T10:32:04.162 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-23T10:32:04.162 INFO:teuthology.orchestra.run.smithi114.stdout: "auth_pins": 2, 2024-04-23T10:32:04.163 INFO:teuthology.orchestra.run.smithi114.stdout: "is_frozen": false, 2024-04-23T10:32:04.163 INFO:teuthology.orchestra.run.smithi114.stdout: "is_freezing": false, 2024-04-23T10:32:04.163 INFO:teuthology.orchestra.run.smithi114.stdout: "pins": { 2024-04-23T10:32:04.163 INFO:teuthology.orchestra.run.smithi114.stdout: "ptrwaiter": 0, 2024-04-23T10:32:04.163 INFO:teuthology.orchestra.run.smithi114.stdout: "request": 1, 2024-04-23T10:32:04.163 INFO:teuthology.orchestra.run.smithi114.stdout: "lock": 1, 2024-04-23T10:32:04.163 INFO:teuthology.orchestra.run.smithi114.stdout: "importingcaps": 0, 2024-04-23T10:32:04.163 INFO:teuthology.orchestra.run.smithi114.stdout: "caps": 0, 2024-04-23T10:32:04.163 INFO:teuthology.orchestra.run.smithi114.stdout: "replicated": 1, 2024-04-23T10:32:04.163 INFO:teuthology.orchestra.run.smithi114.stdout: "dirty": 1, 2024-04-23T10:32:04.163 INFO:teuthology.orchestra.run.smithi114.stdout: "waiter": 0, 2024-04-23T10:32:04.163 INFO:teuthology.orchestra.run.smithi114.stdout: "authpin": 1 2024-04-23T10:32:04.163 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-23T10:32:04.163 INFO:teuthology.orchestra.run.smithi114.stdout: "nref": 5 2024-04-23T10:32:04.163 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-23T10:32:04.164 INFO:teuthology.orchestra.run.smithi114.stdout: "object_string": "[inode 0x1000000021d [7,head] /volumes/qa/sv_1/b206794e-7622-4ab8-8943-f3ae4e65b7a3/client.0/tmp/clients/client0/~dmtmp/COREL/TRAIN.CDR auth{0=1} v7252 ap=2 s=260096 n(v0 rc2024-04-23T10:27:52.038396+0000 b260096 1=1+0) (iquiesce lock x=1 by request(mds.2:134 nref=3)) | ptrwaiter=0 request=1 lock=1 importingcaps=0 caps=0 replicated=1 dirty=1 waiter=0 authpin=1 0x55c68bc4d600]", 2024-04-23T10:32:04.164 INFO:teuthology.orchestra.run.smithi114.stdout: "lock": { 2024-04-23T10:32:04.164 INFO:teuthology.orchestra.run.smithi114.stdout: "gather_set": [], 2024-04-23T10:32:04.164 INFO:teuthology.orchestra.run.smithi114.stdout: "state": "lock", 2024-04-23T10:32:04.164 INFO:teuthology.orchestra.run.smithi114.stdout: "type": "iquiesce", 2024-04-23T10:32:04.164 INFO:teuthology.orchestra.run.smithi114.stdout: "is_leased": false, 2024-04-23T10:32:04.164 INFO:teuthology.orchestra.run.smithi114.stdout: "num_rdlocks": 0, 2024-04-23T10:32:04.164 INFO:teuthology.orchestra.run.smithi114.stdout: "num_wrlocks": 0, 2024-04-23T10:32:04.164 INFO:teuthology.orchestra.run.smithi114.stdout: "num_xlocks": 1, 2024-04-23T10:32:04.164 INFO:teuthology.orchestra.run.smithi114.stdout: "xlock_by": { 2024-04-23T10:32:04.164 INFO:teuthology.orchestra.run.smithi114.stdout: "reqid": { 2024-04-23T10:32:04.164 INFO:teuthology.orchestra.run.smithi114.stdout: "entity": { 2024-04-23T10:32:04.164 INFO:teuthology.orchestra.run.smithi114.stdout: "type": "mds", 2024-04-23T10:32:04.164 INFO:teuthology.orchestra.run.smithi114.stdout: "num": 2 2024-04-23T10:32:04.164 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-23T10:32:04.164 INFO:teuthology.orchestra.run.smithi114.stdout: "tid": 134 2024-04-23T10:32:04.165 INFO:teuthology.orchestra.run.smithi114.stdout: } 2024-04-23T10:32:04.165 INFO:teuthology.orchestra.run.smithi114.stdout: } 2024-04-23T10:32:04.165 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-23T10:32:04.165 INFO:teuthology.orchestra.run.smithi114.stdout: "flags": 4, 2024-04-23T10:32:04.165 INFO:teuthology.orchestra.run.smithi114.stdout: "wrlock_target": -1 2024-04-23T10:32:04.165 INFO:teuthology.orchestra.run.smithi114.stdout: } 2024-04-23T10:32:04.165 INFO:teuthology.orchestra.run.smithi114.stdout: ] 2024-04-23T10:32:04.165 INFO:teuthology.orchestra.run.smithi114.stdout: } 2024-04-23T10:32:04.165 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-23T10:32:04.165 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-04-23T10:32:04.165 INFO:teuthology.orchestra.run.smithi114.stdout: "description": "internal op quiesce_inode:mds.2:135 fp=#0x10000000219", 2024-04-23T10:32:04.165 INFO:teuthology.orchestra.run.smithi114.stdout: "initiated_at": "2024-04-23T10:31:21.478038+0000", 2024-04-23T10:32:04.165 INFO:teuthology.orchestra.run.smithi114.stdout: "age": 42.350706348999999, 2024-04-23T10:32:04.165 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 42.353826496000003, 2024-04-23T10:32:04.165 INFO:teuthology.orchestra.run.smithi114.stdout: "continuous": true, 2024-04-23T10:32:04.166 INFO:teuthology.orchestra.run.smithi114.stdout: "type_data": { 2024-04-23T10:32:04.166 INFO:teuthology.orchestra.run.smithi114.stdout: "result": -2147483648, 2024-04-23T10:32:04.166 INFO:teuthology.orchestra.run.smithi114.stdout: "flag_point": "quiesce complete", 2024-04-23T10:32:04.166 INFO:teuthology.orchestra.run.smithi114.stdout: "reqid": { 2024-04-23T10:32:04.166 INFO:teuthology.orchestra.run.smithi114.stdout: "entity": { 2024-04-23T10:32:04.166 INFO:teuthology.orchestra.run.smithi114.stdout: "type": "mds", 2024-04-23T10:32:04.166 INFO:teuthology.orchestra.run.smithi114.stdout: "num": 2 2024-04-23T10:32:04.166 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-23T10:32:04.166 INFO:teuthology.orchestra.run.smithi114.stdout: "tid": 135 2024-04-23T10:32:04.166 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-23T10:32:04.166 INFO:teuthology.orchestra.run.smithi114.stdout: "op_type": "internal_op", 2024-04-23T10:32:04.166 INFO:teuthology.orchestra.run.smithi114.stdout: "internal_op": 5385, 2024-04-23T10:32:04.166 INFO:teuthology.orchestra.run.smithi114.stdout: "op_name": "quiesce_inode", 2024-04-23T10:32:04.166 INFO:teuthology.orchestra.run.smithi114.stdout: "events": [ 2024-04-23T10:32:04.166 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-04-23T10:32:04.167 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-04-23T10:31:21.478038+0000", 2024-04-23T10:32:04.167 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "initiated" 2024-04-23T10:32:04.167 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-23T10:32:04.167 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-04-23T10:32:04.167 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-04-23T10:31:21.478038+0000", 2024-04-23T10:32:04.167 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "header_read" 2024-04-23T10:32:04.167 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-23T10:32:04.167 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-04-23T10:32:04.167 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-04-23T10:31:21.478038+0000", 2024-04-23T10:32:04.167 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "throttled" 2024-04-23T10:32:04.167 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-23T10:32:04.167 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-04-23T10:32:04.167 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-04-23T10:31:21.478038+0000", 2024-04-23T10:32:04.167 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "all_read" 2024-04-23T10:32:04.167 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-23T10:32:04.168 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-04-23T10:32:04.168 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-04-23T10:31:21.478038+0000", 2024-04-23T10:32:04.168 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "dispatched" 2024-04-23T10:32:04.168 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-23T10:32:04.168 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-04-23T10:32:04.168 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-04-23T10:31:21.480933+0000", 2024-04-23T10:32:04.168 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "failed to xlock, waiting" 2024-04-23T10:32:04.168 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-23T10:32:04.168 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-04-23T10:32:04.168 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-04-23T10:31:21.487514+0000", 2024-04-23T10:32:04.168 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "acquired locks" 2024-04-23T10:32:04.168 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-23T10:32:04.168 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-04-23T10:32:04.168 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-04-23T10:31:21.487768+0000", 2024-04-23T10:32:04.168 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "quiesce complete" 2024-04-23T10:32:04.168 INFO:teuthology.orchestra.run.smithi114.stdout: } 2024-04-23T10:32:04.168 INFO:teuthology.orchestra.run.smithi114.stdout: ], 2024-04-23T10:32:04.169 INFO:teuthology.orchestra.run.smithi114.stdout: "locks": [ 2024-04-23T10:32:04.169 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-04-23T10:32:04.169 INFO:teuthology.orchestra.run.smithi114.stdout: "object": { 2024-04-23T10:32:04.169 INFO:teuthology.orchestra.run.smithi114.stdout: "is_auth": true, 2024-04-23T10:32:04.169 INFO:teuthology.orchestra.run.smithi114.stdout: "auth_state": { 2024-04-23T10:32:04.169 INFO:teuthology.orchestra.run.smithi114.stdout: "replicas": { 2024-04-23T10:32:04.169 INFO:teuthology.orchestra.run.smithi114.stdout: "0": 1 2024-04-23T10:32:04.169 INFO:teuthology.orchestra.run.smithi114.stdout: } 2024-04-23T10:32:04.169 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-23T10:32:04.169 INFO:teuthology.orchestra.run.smithi114.stdout: "replica_state": { 2024-04-23T10:32:04.169 INFO:teuthology.orchestra.run.smithi114.stdout: "authority": [ 2024-04-23T10:32:04.169 INFO:teuthology.orchestra.run.smithi114.stdout: 2, 2024-04-23T10:32:04.169 INFO:teuthology.orchestra.run.smithi114.stdout: -2 2024-04-23T10:32:04.169 INFO:teuthology.orchestra.run.smithi114.stdout: ], 2024-04-23T10:32:04.170 INFO:teuthology.orchestra.run.smithi114.stdout: "replica_nonce": 0 2024-04-23T10:32:04.170 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-23T10:32:04.170 INFO:teuthology.orchestra.run.smithi114.stdout: "auth_pins": 2, 2024-04-23T10:32:04.170 INFO:teuthology.orchestra.run.smithi114.stdout: "is_frozen": false, 2024-04-23T10:32:04.170 INFO:teuthology.orchestra.run.smithi114.stdout: "is_freezing": false, 2024-04-23T10:32:04.170 INFO:teuthology.orchestra.run.smithi114.stdout: "pins": { 2024-04-23T10:32:04.170 INFO:teuthology.orchestra.run.smithi114.stdout: "ptrwaiter": 0, 2024-04-23T10:32:04.170 INFO:teuthology.orchestra.run.smithi114.stdout: "request": 1, 2024-04-23T10:32:04.170 INFO:teuthology.orchestra.run.smithi114.stdout: "lock": 1, 2024-04-23T10:32:04.170 INFO:teuthology.orchestra.run.smithi114.stdout: "importingcaps": 0, 2024-04-23T10:32:04.170 INFO:teuthology.orchestra.run.smithi114.stdout: "caps": 0, 2024-04-23T10:32:04.170 INFO:teuthology.orchestra.run.smithi114.stdout: "needsnapflush": 0, 2024-04-23T10:32:04.170 INFO:teuthology.orchestra.run.smithi114.stdout: "replicated": 1, 2024-04-23T10:32:04.170 INFO:teuthology.orchestra.run.smithi114.stdout: "dirty": 1, 2024-04-23T10:32:04.170 INFO:teuthology.orchestra.run.smithi114.stdout: "waiter": 0, 2024-04-23T10:32:04.171 INFO:teuthology.orchestra.run.smithi114.stdout: "authpin": 1 2024-04-23T10:32:04.171 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-23T10:32:04.171 INFO:teuthology.orchestra.run.smithi114.stdout: "nref": 5 2024-04-23T10:32:04.171 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-23T10:32:04.171 INFO:teuthology.orchestra.run.smithi114.stdout: "object_string": "[inode 0x10000000219 [7,head] /volumes/qa/sv_1/b206794e-7622-4ab8-8943-f3ae4e65b7a3/client.0/tmp/clients/client0/~dmtmp/COREL/GRAPHIC1.CDR auth{0=1} v7244 ap=2 s=260096 n(v0 rc2024-04-23T10:27:52.019396+0000 b260096 1=1+0) (iquiesce lock x=1 by request(mds.2:135 nref=3)) | ptrwaiter=0 request=1 lock=1 importingcaps=0 caps=0 needsnapflush=0 replicated=1 dirty=1 waiter=0 authpin=1 0x55c68bc4db80]", 2024-04-23T10:32:04.171 INFO:teuthology.orchestra.run.smithi114.stdout: "lock": { 2024-04-23T10:32:04.171 INFO:teuthology.orchestra.run.smithi114.stdout: "gather_set": [], 2024-04-23T10:32:04.171 INFO:teuthology.orchestra.run.smithi114.stdout: "state": "lock", 2024-04-23T10:32:04.171 INFO:teuthology.orchestra.run.smithi114.stdout: "type": "iquiesce", 2024-04-23T10:32:04.171 INFO:teuthology.orchestra.run.smithi114.stdout: "is_leased": false, 2024-04-23T10:32:04.171 INFO:teuthology.orchestra.run.smithi114.stdout: "num_rdlocks": 0, 2024-04-23T10:32:04.171 INFO:teuthology.orchestra.run.smithi114.stdout: "num_wrlocks": 0, 2024-04-23T10:32:04.171 INFO:teuthology.orchestra.run.smithi114.stdout: "num_xlocks": 1, 2024-04-23T10:32:04.171 INFO:teuthology.orchestra.run.smithi114.stdout: "xlock_by": { 2024-04-23T10:32:04.171 INFO:teuthology.orchestra.run.smithi114.stdout: "reqid": { 2024-04-23T10:32:04.172 INFO:teuthology.orchestra.run.smithi114.stdout: "entity": { 2024-04-23T10:32:04.172 INFO:teuthology.orchestra.run.smithi114.stdout: "type": "mds", 2024-04-23T10:32:04.172 INFO:teuthology.orchestra.run.smithi114.stdout: "num": 2 2024-04-23T10:32:04.172 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-23T10:32:04.172 INFO:teuthology.orchestra.run.smithi114.stdout: "tid": 135 2024-04-23T10:32:04.172 INFO:teuthology.orchestra.run.smithi114.stdout: } 2024-04-23T10:32:04.172 INFO:teuthology.orchestra.run.smithi114.stdout: } 2024-04-23T10:32:04.172 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-23T10:32:04.172 INFO:teuthology.orchestra.run.smithi114.stdout: "flags": 4, 2024-04-23T10:32:04.172 INFO:teuthology.orchestra.run.smithi114.stdout: "wrlock_target": -1 2024-04-23T10:32:04.172 INFO:teuthology.orchestra.run.smithi114.stdout: } 2024-04-23T10:32:04.172 INFO:teuthology.orchestra.run.smithi114.stdout: ] 2024-04-23T10:32:04.172 INFO:teuthology.orchestra.run.smithi114.stdout: } 2024-04-23T10:32:04.172 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-23T10:32:04.172 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-04-23T10:32:04.173 INFO:teuthology.orchestra.run.smithi114.stdout: "description": "internal op quiesce_inode:mds.2:136 fp=#0x10000000216", 2024-04-23T10:32:04.173 INFO:teuthology.orchestra.run.smithi114.stdout: "initiated_at": "2024-04-23T10:31:21.478055+0000", 2024-04-23T10:32:04.173 INFO:teuthology.orchestra.run.smithi114.stdout: "age": 42.350689918999997, 2024-04-23T10:32:04.173 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 42.353873546999999, 2024-04-23T10:32:04.173 INFO:teuthology.orchestra.run.smithi114.stdout: "continuous": true, 2024-04-23T10:32:04.173 INFO:teuthology.orchestra.run.smithi114.stdout: "type_data": { 2024-04-23T10:32:04.173 INFO:teuthology.orchestra.run.smithi114.stdout: "result": -2147483648, 2024-04-23T10:32:04.173 INFO:teuthology.orchestra.run.smithi114.stdout: "flag_point": "quiesce complete", 2024-04-23T10:32:04.173 INFO:teuthology.orchestra.run.smithi114.stdout: "reqid": { 2024-04-23T10:32:04.173 INFO:teuthology.orchestra.run.smithi114.stdout: "entity": { 2024-04-23T10:32:04.173 INFO:teuthology.orchestra.run.smithi114.stdout: "type": "mds", 2024-04-23T10:32:04.173 INFO:teuthology.orchestra.run.smithi114.stdout: "num": 2 2024-04-23T10:32:04.173 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-23T10:32:04.173 INFO:teuthology.orchestra.run.smithi114.stdout: "tid": 136 2024-04-23T10:32:04.173 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-23T10:32:04.174 INFO:teuthology.orchestra.run.smithi114.stdout: "op_type": "internal_op", 2024-04-23T10:32:04.174 INFO:teuthology.orchestra.run.smithi114.stdout: "internal_op": 5385, 2024-04-23T10:32:04.174 INFO:teuthology.orchestra.run.smithi114.stdout: "op_name": "quiesce_inode", 2024-04-23T10:32:04.174 INFO:teuthology.orchestra.run.smithi114.stdout: "events": [ 2024-04-23T10:32:04.174 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-04-23T10:32:04.174 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-04-23T10:31:21.478055+0000", 2024-04-23T10:32:04.174 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "initiated" 2024-04-23T10:32:04.174 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-23T10:32:04.174 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-04-23T10:32:04.174 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-04-23T10:31:21.478055+0000", 2024-04-23T10:32:04.174 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "header_read" 2024-04-23T10:32:04.174 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-23T10:32:04.174 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-04-23T10:32:04.174 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-04-23T10:31:21.478055+0000", 2024-04-23T10:32:04.174 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "throttled" 2024-04-23T10:32:04.174 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-23T10:32:04.175 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-04-23T10:32:04.175 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-04-23T10:31:21.478055+0000", 2024-04-23T10:32:04.175 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "all_read" 2024-04-23T10:32:04.175 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-23T10:32:04.175 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-04-23T10:32:04.175 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-04-23T10:31:21.478055+0000", 2024-04-23T10:32:04.175 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "dispatched" 2024-04-23T10:32:04.175 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-23T10:32:04.175 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-04-23T10:32:04.175 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-04-23T10:31:21.481114+0000", 2024-04-23T10:32:04.175 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "failed to xlock, waiting" 2024-04-23T10:32:04.175 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-23T10:32:04.175 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-04-23T10:32:04.175 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-04-23T10:31:21.488531+0000", 2024-04-23T10:32:04.175 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "failed to xlock, waiting" 2024-04-23T10:32:04.175 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-23T10:32:04.176 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-04-23T10:32:04.176 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-04-23T10:31:21.505820+0000", 2024-04-23T10:32:04.176 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "acquired locks" 2024-04-23T10:32:04.176 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-23T10:32:04.176 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-04-23T10:32:04.176 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-04-23T10:31:21.506034+0000", 2024-04-23T10:32:04.176 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "quiesce complete" 2024-04-23T10:32:04.176 INFO:teuthology.orchestra.run.smithi114.stdout: } 2024-04-23T10:32:04.176 INFO:teuthology.orchestra.run.smithi114.stdout: ], 2024-04-23T10:32:04.176 INFO:teuthology.orchestra.run.smithi114.stdout: "locks": [ 2024-04-23T10:32:04.176 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-04-23T10:32:04.176 INFO:teuthology.orchestra.run.smithi114.stdout: "object": { 2024-04-23T10:32:04.176 INFO:teuthology.orchestra.run.smithi114.stdout: "is_auth": true, 2024-04-23T10:32:04.176 INFO:teuthology.orchestra.run.smithi114.stdout: "auth_state": { 2024-04-23T10:32:04.177 INFO:teuthology.orchestra.run.smithi114.stdout: "replicas": { 2024-04-23T10:32:04.177 INFO:teuthology.orchestra.run.smithi114.stdout: "0": 1 2024-04-23T10:32:04.177 INFO:teuthology.orchestra.run.smithi114.stdout: } 2024-04-23T10:32:04.177 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-23T10:32:04.177 INFO:teuthology.orchestra.run.smithi114.stdout: "replica_state": { 2024-04-23T10:32:04.177 INFO:teuthology.orchestra.run.smithi114.stdout: "authority": [ 2024-04-23T10:32:04.177 INFO:teuthology.orchestra.run.smithi114.stdout: 2, 2024-04-23T10:32:04.177 INFO:teuthology.orchestra.run.smithi114.stdout: -2 2024-04-23T10:32:04.177 INFO:teuthology.orchestra.run.smithi114.stdout: ], 2024-04-23T10:32:04.177 INFO:teuthology.orchestra.run.smithi114.stdout: "replica_nonce": 0 2024-04-23T10:32:04.177 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-23T10:32:04.177 INFO:teuthology.orchestra.run.smithi114.stdout: "auth_pins": 2, 2024-04-23T10:32:04.177 INFO:teuthology.orchestra.run.smithi114.stdout: "is_frozen": false, 2024-04-23T10:32:04.177 INFO:teuthology.orchestra.run.smithi114.stdout: "is_freezing": false, 2024-04-23T10:32:04.178 INFO:teuthology.orchestra.run.smithi114.stdout: "pins": { 2024-04-23T10:32:04.178 INFO:teuthology.orchestra.run.smithi114.stdout: "ptrwaiter": 0, 2024-04-23T10:32:04.178 INFO:teuthology.orchestra.run.smithi114.stdout: "request": 1, 2024-04-23T10:32:04.178 INFO:teuthology.orchestra.run.smithi114.stdout: "lock": 1, 2024-04-23T10:32:04.178 INFO:teuthology.orchestra.run.smithi114.stdout: "importingcaps": 0, 2024-04-23T10:32:04.178 INFO:teuthology.orchestra.run.smithi114.stdout: "caps": 0, 2024-04-23T10:32:04.178 INFO:teuthology.orchestra.run.smithi114.stdout: "needsnapflush": 0, 2024-04-23T10:32:04.178 INFO:teuthology.orchestra.run.smithi114.stdout: "replicated": 1, 2024-04-23T10:32:04.178 INFO:teuthology.orchestra.run.smithi114.stdout: "dirty": 1, 2024-04-23T10:32:04.178 INFO:teuthology.orchestra.run.smithi114.stdout: "waiter": 0, 2024-04-23T10:32:04.178 INFO:teuthology.orchestra.run.smithi114.stdout: "authpin": 1 2024-04-23T10:32:04.178 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-23T10:32:04.178 INFO:teuthology.orchestra.run.smithi114.stdout: "nref": 5 2024-04-23T10:32:04.178 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-23T10:32:04.178 INFO:teuthology.orchestra.run.smithi114.stdout: "object_string": "[inode 0x10000000216 [7,head] /volumes/qa/sv_1/b206794e-7622-4ab8-8943-f3ae4e65b7a3/client.0/tmp/clients/client0/~dmtmp/COREL/CDRROLS.CFG auth{0=1} v7238 ap=2 s=260096 n(v0 rc2024-04-23T10:31:17.221728+0000 b260096 1=1+0) (iquiesce lock x=1 by request(mds.2:136 nref=3)) | ptrwaiter=0 request=1 lock=1 importingcaps=0 caps=0 needsnapflush=0 replicated=1 dirty=1 waiter=0 authpin=1 0x55c68bc4e100]", 2024-04-23T10:32:04.179 INFO:teuthology.orchestra.run.smithi114.stdout: "lock": { 2024-04-23T10:32:04.179 INFO:teuthology.orchestra.run.smithi114.stdout: "gather_set": [], 2024-04-23T10:32:04.179 INFO:teuthology.orchestra.run.smithi114.stdout: "state": "lock", 2024-04-23T10:32:04.179 INFO:teuthology.orchestra.run.smithi114.stdout: "type": "iquiesce", 2024-04-23T10:32:04.179 INFO:teuthology.orchestra.run.smithi114.stdout: "is_leased": false, 2024-04-23T10:32:04.179 INFO:teuthology.orchestra.run.smithi114.stdout: "num_rdlocks": 0, 2024-04-23T10:32:04.179 INFO:teuthology.orchestra.run.smithi114.stdout: "num_wrlocks": 0, 2024-04-23T10:32:04.179 INFO:teuthology.orchestra.run.smithi114.stdout: "num_xlocks": 1, 2024-04-23T10:32:04.179 INFO:teuthology.orchestra.run.smithi114.stdout: "xlock_by": { 2024-04-23T10:32:04.179 INFO:teuthology.orchestra.run.smithi114.stdout: "reqid": { 2024-04-23T10:32:04.179 INFO:teuthology.orchestra.run.smithi114.stdout: "entity": { 2024-04-23T10:32:04.179 INFO:teuthology.orchestra.run.smithi114.stdout: "type": "mds", 2024-04-23T10:32:04.179 INFO:teuthology.orchestra.run.smithi114.stdout: "num": 2 2024-04-23T10:32:04.179 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-23T10:32:04.180 INFO:teuthology.orchestra.run.smithi114.stdout: "tid": 136 2024-04-23T10:32:04.180 INFO:teuthology.orchestra.run.smithi114.stdout: } 2024-04-23T10:32:04.180 INFO:teuthology.orchestra.run.smithi114.stdout: } 2024-04-23T10:32:04.180 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-23T10:32:04.180 INFO:teuthology.orchestra.run.smithi114.stdout: "flags": 4, 2024-04-23T10:32:04.180 INFO:teuthology.orchestra.run.smithi114.stdout: "wrlock_target": -1 2024-04-23T10:32:04.180 INFO:teuthology.orchestra.run.smithi114.stdout: } 2024-04-23T10:32:04.180 INFO:teuthology.orchestra.run.smithi114.stdout: ] 2024-04-23T10:32:04.180 INFO:teuthology.orchestra.run.smithi114.stdout: } 2024-04-23T10:32:04.180 INFO:teuthology.orchestra.run.smithi114.stdout: } 2024-04-23T10:32:04.180 INFO:teuthology.orchestra.run.smithi114.stdout: ], 2024-04-23T10:32:04.180 INFO:teuthology.orchestra.run.smithi114.stdout: "num_ops": 53 2024-04-23T10:32:04.180 INFO:teuthology.orchestra.run.smithi114.stdout:} 2024-04-23T10:32:04.192 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:32:03.833+0000 7f0fe81f4740 1 Processor -- start 2024-04-23T10:32:04.192 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:32:03.833+0000 7f0fe81f4740 1 -- start start 2024-04-23T10:32:04.193 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:32:03.833+0000 7f0fe81f4740 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55c8d0ee0ba0 0x55c8d0f7cbd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T10:32:04.193 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:32:03.833+0000 7f0fe81f4740 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x55c8d0ee14c0 0x55c8d0f61020 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T10:32:04.193 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:32:03.833+0000 7f0fe81f4740 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x55c8d0ef00a0 0x55c8d0f0aff0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T10:32:04.193 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:32:03.833+0000 7f0fe81f4740 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x55c8d0f60160 con 0x55c8d0ee0ba0 2024-04-23T10:32:04.193 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:32:03.833+0000 7f0fe81f4740 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 -- 0x55c8d0f687b0 con 0x55c8d0ee14c0 2024-04-23T10:32:04.193 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:32:03.833+0000 7f0fe81f4740 1 -- --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_getmap magic: 0 -- 0x55c8d0eea7a0 con 0x55c8d0ef00a0 2024-04-23T10:32:04.193 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:32:03.833+0000 7f0fdeffd640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55c8d0ee0ba0 0x55c8d0f7cbd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T10:32:04.193 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:32:03.833+0000 7f0fdeffd640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55c8d0ee0ba0 0x55c8d0f7cbd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.114:52012/0 (socket says 172.21.15.114:52012) 2024-04-23T10:32:04.193 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:32:03.833+0000 7f0fdeffd640 1 -- 172.21.15.114:0/220810321 learned_addr learned my addr 172.21.15.114:0/220810321 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-04-23T10:32:04.193 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:32:03.833+0000 7f0fde7fc640 1 --2- 172.21.15.114:0/220810321 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x55c8d0ee14c0 0x55c8d0f61020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T10:32:04.193 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:32:03.833+0000 7f0fdeffd640 1 -- 172.21.15.114:0/220810321 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x55c8d0ef00a0 msgr2=0x55c8d0f0aff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T10:32:04.193 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:32:03.833+0000 7f0fdf7fe640 1 --2- 172.21.15.114:0/220810321 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x55c8d0ef00a0 0x55c8d0f0aff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T10:32:04.193 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:32:03.833+0000 7f0fdeffd640 1 --2- 172.21.15.114:0/220810321 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x55c8d0ef00a0 0x55c8d0f0aff0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:32:04.193 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:32:03.833+0000 7f0fdeffd640 1 -- 172.21.15.114:0/220810321 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x55c8d0ee14c0 msgr2=0x55c8d0f61020 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T10:32:04.193 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:32:03.833+0000 7f0fdeffd640 1 --2- 172.21.15.114:0/220810321 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x55c8d0ee14c0 0x55c8d0f61020 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:32:04.194 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:32:03.833+0000 7f0fdeffd640 1 -- 172.21.15.114:0/220810321 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55c8d0f679e0 con 0x55c8d0ee0ba0 2024-04-23T10:32:04.194 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:32:03.833+0000 7f0fde7fc640 1 --2- 172.21.15.114:0/220810321 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x55c8d0ee14c0 0x55c8d0f61020 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-23T10:32:04.194 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:32:03.833+0000 7f0fdf7fe640 1 --2- 172.21.15.114:0/220810321 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x55c8d0ef00a0 0x55c8d0f0aff0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-23T10:32:04.194 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:32:03.833+0000 7f0fdeffd640 1 --2- 172.21.15.114:0/220810321 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55c8d0ee0ba0 0x55c8d0f7cbd0 secure :-1 s=READY pgs=718 cs=0 l=1 rev1=1 crypto rx=0x7f0fd0004850 tx=0x7f0fd002af40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4ca898716ebf7a8a server_cookie=0 in_seq=0 out_seq=0 2024-04-23T10:32:04.194 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:32:03.833+0000 7f0fc57fa640 1 -- 172.21.15.114:0/220810321 <== mon.0 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0fd0077040 con 0x55c8d0ee0ba0 2024-04-23T10:32:04.194 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:32:03.833+0000 7f0fc57fa640 1 -- 172.21.15.114:0/220810321 <== mon.0 v2:172.21.15.114:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f0fd0036040 con 0x55c8d0ee0ba0 2024-04-23T10:32:04.194 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:32:03.833+0000 7f0fc57fa640 1 -- 172.21.15.114:0/220810321 <== mon.0 v2:172.21.15.114:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0fd0004640 con 0x55c8d0ee0ba0 2024-04-23T10:32:04.194 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:32:03.834+0000 7f0fe81f4740 1 -- 172.21.15.114:0/220810321 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55c8d0ee0ba0 msgr2=0x55c8d0f7cbd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T10:32:04.194 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:32:03.834+0000 7f0fe81f4740 1 --2- 172.21.15.114:0/220810321 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55c8d0ee0ba0 0x55c8d0f7cbd0 secure :-1 s=READY pgs=718 cs=0 l=1 rev1=1 crypto rx=0x7f0fd0004850 tx=0x7f0fd002af40 comp rx=0 tx=0).stop 2024-04-23T10:32:04.194 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:32:03.834+0000 7f0fe81f4740 1 -- 172.21.15.114:0/220810321 shutdown_connections 2024-04-23T10:32:04.194 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:32:03.834+0000 7f0fe81f4740 1 --2- 172.21.15.114:0/220810321 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x55c8d0ef00a0 0x55c8d0f0aff0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:32:04.194 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:32:03.834+0000 7f0fe81f4740 1 --2- 172.21.15.114:0/220810321 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x55c8d0ee14c0 0x55c8d0f61020 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:32:04.194 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:32:03.834+0000 7f0fe81f4740 1 --2- 172.21.15.114:0/220810321 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55c8d0ee0ba0 0x55c8d0f7cbd0 unknown :-1 s=CLOSED pgs=718 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:32:04.194 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:32:03.834+0000 7f0fe81f4740 1 -- 172.21.15.114:0/220810321 >> 172.21.15.114:0/220810321 conn(0x55c8d0edcfc0 msgr2=0x55c8d0f67f50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T10:32:04.195 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:32:03.834+0000 7f0fe81f4740 1 -- 172.21.15.114:0/220810321 shutdown_connections 2024-04-23T10:32:04.195 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:32:03.834+0000 7f0fe81f4740 1 -- 172.21.15.114:0/220810321 wait complete. 2024-04-23T10:32:04.195 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:32:03.834+0000 7f0fe81f4740 1 -- start start 2024-04-23T10:32:04.195 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:32:03.834+0000 7f0fe81f4740 1 Processor -- start 2024-04-23T10:32:04.195 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:32:03.835+0000 7f0fe81f4740 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x55c8d0ee0ba0 0x55c8d0f7cbd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T10:32:04.195 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:32:03.835+0000 7f0fe81f4740 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55c8d0ee14c0 0x55c8d0f61020 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T10:32:04.195 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:32:03.835+0000 7f0fe81f4740 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x55c8d0ef00a0 0x55c8d0ee96f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T10:32:04.195 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:32:03.835+0000 7f0fe81f4740 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x55c8d0f66ef0 con 0x55c8d0ee14c0 2024-04-23T10:32:04.195 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:32:03.835+0000 7f0fe81f4740 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 -- 0x55c8d0ee00a0 con 0x55c8d0ef00a0 2024-04-23T10:32:04.195 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:32:03.835+0000 7f0fe81f4740 1 -- --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_getmap magic: 0 -- 0x55c8d0f087f0 con 0x55c8d0ee0ba0 2024-04-23T10:32:04.195 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:32:03.835+0000 7f0fe81f4740 10 client.0 _ll_register_callbacks cb 0x7f0fd4009b50 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-23T10:32:04.195 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:32:03.835+0000 7f0fde7fc640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55c8d0ee14c0 0x55c8d0f61020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T10:32:04.195 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:32:03.835+0000 7f0fde7fc640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55c8d0ee14c0 0x55c8d0f61020 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.114:52022/0 (socket says 172.21.15.114:52022) 2024-04-23T10:32:04.196 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:32:03.835+0000 7f0fde7fc640 1 -- 172.21.15.114:0/4123865525 learned_addr learned my addr 172.21.15.114:0/4123865525 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-04-23T10:32:04.196 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:32:03.835+0000 7f0fdf7fe640 1 --2- 172.21.15.114:0/4123865525 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x55c8d0ef00a0 0x55c8d0ee96f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T10:32:04.196 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:32:03.835+0000 7f0fde7fc640 1 -- 172.21.15.114:0/4123865525 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x55c8d0ee0ba0 msgr2=0x55c8d0f7cbd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T10:32:04.196 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:32:03.835+0000 7f0fde7fc640 1 --2- 172.21.15.114:0/4123865525 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x55c8d0ee0ba0 0x55c8d0f7cbd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:32:04.196 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:32:03.835+0000 7f0fde7fc640 1 -- 172.21.15.114:0/4123865525 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x55c8d0ef00a0 msgr2=0x55c8d0ee96f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T10:32:04.196 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:32:03.835+0000 7f0fde7fc640 1 --2- 172.21.15.114:0/4123865525 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x55c8d0ef00a0 0x55c8d0ee96f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:32:04.196 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:32:03.835+0000 7f0fde7fc640 1 -- 172.21.15.114:0/4123865525 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55c8d0f5fed0 con 0x55c8d0ee14c0 2024-04-23T10:32:04.196 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:32:03.835+0000 7f0fdf7fe640 1 --2- 172.21.15.114:0/4123865525 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x55c8d0ef00a0 0x55c8d0ee96f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-23T10:32:04.196 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:32:03.835+0000 7f0fde7fc640 1 --2- 172.21.15.114:0/4123865525 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55c8d0ee14c0 0x55c8d0f61020 secure :-1 s=READY pgs=719 cs=0 l=1 rev1=1 crypto rx=0x55c8d0ede040 tx=0x7f0fd4009c80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4c9300345b9a41d2 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T10:32:04.196 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:32:03.835+0000 7f0fc2ff5640 10 client.0 ms_handle_connect on v2:172.21.15.114:3300/0 2024-04-23T10:32:04.196 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:32:03.835+0000 7f0fc2ff5640 1 -- 172.21.15.114:0/4123865525 <== mon.0 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0fd4064070 con 0x55c8d0ee14c0 2024-04-23T10:32:04.196 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:32:03.835+0000 7f0fe81f4740 1 -- 172.21.15.114:0/4123865525 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f0fd0036040 con 0x55c8d0ee14c0 2024-04-23T10:32:04.196 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:32:03.835+0000 7f0fc2ff5640 1 -- 172.21.15.114:0/4123865525 <== mon.0 v2:172.21.15.114:3300/0 2 ==== osd_map(94..94 src has 1..94) ==== 8585+0+0 (secure 0 0 0) 0x7f0fd405c770 con 0x55c8d0ee14c0 2024-04-23T10:32:04.196 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:32:03.835+0000 7f0fc2ff5640 1 -- 172.21.15.114:0/4123865525 <== mon.0 v2:172.21.15.114:3300/0 3 ==== mon_get_version_reply(handle=1 version=30) ==== 24+0+0 (secure 0 0 0) 0x7f0fd402f070 con 0x55c8d0ee14c0 2024-04-23T10:32:04.197 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:32:03.835+0000 7f0fe81f4740 10 client.16338 fetch_fsmap learned FSMap version 30 2024-04-23T10:32:04.197 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:32:03.835+0000 7f0fe81f4740 1 -- 172.21.15.114:0/4123865525 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({config=0+,fsmap=30,monmap=4+}) -- 0x55c8d0f5fed0 con 0x55c8d0ee14c0 2024-04-23T10:32:04.197 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:32:03.835+0000 7f0fc2ff5640 1 -- 172.21.15.114:0/4123865525 <== mon.0 v2:172.21.15.114:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f0fd0036040 con 0x55c8d0ee14c0 2024-04-23T10:32:04.197 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:32:03.836+0000 7f0fc2ff5640 1 -- 172.21.15.114:0/4123865525 <== mon.0 v2:172.21.15.114:3300/0 5 ==== fsmap(e 30) ==== 8338+0+0 (secure 0 0 0) 0x7f0fd4036460 con 0x55c8d0ee14c0 2024-04-23T10:32:04.197 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:32:03.836+0000 7f0fe81f4740 10 client.16338 fetch_fsmap finished waiting for FSMap version 30 2024-04-23T10:32:04.197 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:32:03.836+0000 7f0fe81f4740 10 client.16338 resolve_mds: resolved name 'h' to daemon mds.h 2024-04-23T10:32:04.197 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:32:03.836+0000 7f0fe81f4740 20 client.16338 populate_metadata read hostname 'smithi114' 2024-04-23T10:32:04.197 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:32:03.836+0000 7f0fe81f4740 1 --2- 172.21.15.114:0/4123865525 >> [v2:172.21.15.119:6836/1872363128,v1:172.21.15.119:6837/1872363128] conn(0x55c8d0f0b640 0x55c8d0f0da30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T10:32:04.197 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:32:03.836+0000 7f0fe81f4740 4 client.16338 mds_command: new command op to 24509 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-23T10:32:04.197 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:32:03.836+0000 7f0fe81f4740 1 -- 172.21.15.114:0/4123865525 --> [v2:172.21.15.119:6836/1872363128,v1:172.21.15.119:6837/1872363128] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55c8d0f09410 con 0x55c8d0f0b640 2024-04-23T10:32:04.197 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:32:03.836+0000 7f0fdeffd640 1 --2- 172.21.15.114:0/4123865525 >> [v2:172.21.15.119:6836/1872363128,v1:172.21.15.119:6837/1872363128] conn(0x55c8d0f0b640 0x55c8d0f0da30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T10:32:04.197 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:32:03.836+0000 7f0fc2ff5640 10 client.16338 ms_handle_connect on v2:172.21.15.119:6836/1872363128 2024-04-23T10:32:04.197 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:32:03.836+0000 7f0fdeffd640 1 --2- 172.21.15.114:0/4123865525 >> [v2:172.21.15.119:6836/1872363128,v1:172.21.15.119:6837/1872363128] conn(0x55c8d0f0b640 0x55c8d0f0da30 crc :-1 s=READY pgs=37 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.1 client_cookie=45d3c6ca8df26442 server_cookie=4db8f0f59ac49dc9 in_seq=0 out_seq=0 2024-04-23T10:32:04.197 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:32:03.837+0000 7f0fc2ff5640 1 -- 172.21.15.114:0/4123865525 <== mds.1 v2:172.21.15.119:6836/1872363128 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x7f0fd0031bb0 con 0x55c8d0f0b640 2024-04-23T10:32:04.197 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:32:03.837+0000 7f0fc2ff5640 10 client.16338 handle_command_reply: tid=0 2024-04-23T10:32:04.198 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:32:03.837+0000 7f0fe81f4740 1 client.16338 shutdown 2024-04-23T10:32:04.198 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:32:03.837+0000 7f0fe81f4740 1 -- 172.21.15.114:0/4123865525 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55c8d0ee14c0 msgr2=0x55c8d0f61020 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T10:32:04.198 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:32:03.837+0000 7f0fe81f4740 1 --2- 172.21.15.114:0/4123865525 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55c8d0ee14c0 0x55c8d0f61020 secure :-1 s=READY pgs=719 cs=0 l=1 rev1=1 crypto rx=0x55c8d0ede040 tx=0x7f0fd4009c80 comp rx=0 tx=0).stop 2024-04-23T10:32:04.198 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:32:03.837+0000 7f0fe81f4740 1 -- 172.21.15.114:0/4123865525 shutdown_connections 2024-04-23T10:32:04.198 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:32:03.837+0000 7f0fe81f4740 1 --2- 172.21.15.114:0/4123865525 >> [v2:172.21.15.119:6836/1872363128,v1:172.21.15.119:6837/1872363128] conn(0x55c8d0f0b640 0x55c8d0f0da30 crc :-1 s=READY pgs=37 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:32:04.198 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:32:03.837+0000 7f0fe81f4740 1 --2- 172.21.15.114:0/4123865525 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x55c8d0ef00a0 0x55c8d0ee96f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:32:04.198 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:32:03.837+0000 7f0fc2ff5640 0 client.16338 ms_handle_reset on v2:172.21.15.119:6836/1872363128 2024-04-23T10:32:04.198 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:32:03.837+0000 7f0fe81f4740 1 --2- 172.21.15.114:0/4123865525 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55c8d0ee14c0 0x55c8d0f61020 unknown :-1 s=CLOSED pgs=719 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:32:04.198 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:32:03.837+0000 7f0fe81f4740 1 --2- 172.21.15.114:0/4123865525 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x55c8d0ee0ba0 0x55c8d0f7cbd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:32:04.198 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:32:03.837+0000 7f0fe81f4740 1 -- 172.21.15.114:0/4123865525 >> 172.21.15.114:0/4123865525 conn(0x55c8d0edcfc0 msgr2=0x55c8d0edc0a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T10:32:04.198 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:32:03.837+0000 7f0fe81f4740 1 -- 172.21.15.114:0/4123865525 shutdown_connections 2024-04-23T10:32:04.198 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:32:03.837+0000 7f0fe81f4740 1 -- 172.21.15.114:0/4123865525 wait complete. 2024-04-23T10:32:04.198 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:32:03.837+0000 7f0fe81f4740 20 client.16338 trim_cache size 0 max 16384 2024-04-23T10:32:04.198 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:32:03.852+0000 7f0fe81f4740 1 Processor -- start 2024-04-23T10:32:04.199 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:32:03.852+0000 7f0fe81f4740 1 -- start start 2024-04-23T10:32:04.199 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:32:03.852+0000 7f0fe81f4740 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x55c8d0ee0ba0 0x55c8d0f02cc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T10:32:04.199 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:32:03.852+0000 7f0fe81f4740 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55c8d0ee14c0 0x55c8d0f61020 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T10:32:04.199 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:32:03.852+0000 7f0fe81f4740 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x55c8d0ef00a0 0x55c8d0f7cbd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T10:32:04.199 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:32:03.852+0000 7f0fe81f4740 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x55c8d0f095d0 con 0x55c8d0ee14c0 2024-04-23T10:32:04.199 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:32:03.852+0000 7f0fe81f4740 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 -- 0x55c8d0f67390 con 0x55c8d0ef00a0 2024-04-23T10:32:04.199 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:32:03.852+0000 7f0fe81f4740 1 -- --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_getmap magic: 0 -- 0x55c8d0f7d470 con 0x55c8d0ee0ba0 2024-04-23T10:32:04.199 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:32:03.852+0000 7f0fde7fc640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55c8d0ee14c0 0x55c8d0f61020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T10:32:04.199 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:32:03.853+0000 7f0fde7fc640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55c8d0ee14c0 0x55c8d0f61020 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.114:52032/0 (socket says 172.21.15.114:52032) 2024-04-23T10:32:04.199 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:32:03.853+0000 7f0fde7fc640 1 -- 172.21.15.114:0/476797864 learned_addr learned my addr 172.21.15.114:0/476797864 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-04-23T10:32:04.199 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:32:03.853+0000 7f0fdf7fe640 1 --2- 172.21.15.114:0/476797864 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x55c8d0ef00a0 0x55c8d0f7cbd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T10:32:04.199 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:32:03.853+0000 7f0fdeffd640 1 --2- 172.21.15.114:0/476797864 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x55c8d0ee0ba0 0x55c8d0f02cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T10:32:04.199 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:32:03.853+0000 7f0fde7fc640 1 -- 172.21.15.114:0/476797864 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x55c8d0ee0ba0 msgr2=0x55c8d0f02cc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T10:32:04.199 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:32:03.853+0000 7f0fde7fc640 1 --2- 172.21.15.114:0/476797864 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x55c8d0ee0ba0 0x55c8d0f02cc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:32:04.200 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:32:03.853+0000 7f0fde7fc640 1 -- 172.21.15.114:0/476797864 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x55c8d0ef00a0 msgr2=0x55c8d0f7cbd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T10:32:04.200 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:32:03.853+0000 7f0fde7fc640 1 --2- 172.21.15.114:0/476797864 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x55c8d0ef00a0 0x55c8d0f7cbd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:32:04.200 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:32:03.853+0000 7f0fde7fc640 1 -- 172.21.15.114:0/476797864 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55c8d0f679e0 con 0x55c8d0ee14c0 2024-04-23T10:32:04.200 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:32:03.853+0000 7f0fde7fc640 1 --2- 172.21.15.114:0/476797864 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55c8d0ee14c0 0x55c8d0f61020 secure :-1 s=READY pgs=720 cs=0 l=1 rev1=1 crypto rx=0x7f0fd405e040 tx=0x7f0fd4005ed0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8911d561bf5aead3 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T10:32:04.200 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:32:03.853+0000 7f0fc4ff9640 1 -- 172.21.15.114:0/476797864 <== mon.0 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0fd4064070 con 0x55c8d0ee14c0 2024-04-23T10:32:04.200 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:32:03.853+0000 7f0fc4ff9640 1 -- 172.21.15.114:0/476797864 <== mon.0 v2:172.21.15.114:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f0fd4004490 con 0x55c8d0ee14c0 2024-04-23T10:32:04.200 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:32:03.853+0000 7f0fc4ff9640 1 -- 172.21.15.114:0/476797864 <== mon.0 v2:172.21.15.114:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0fd402f040 con 0x55c8d0ee14c0 2024-04-23T10:32:04.200 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:32:03.854+0000 7f0fe81f4740 1 -- 172.21.15.114:0/476797864 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55c8d0ee14c0 msgr2=0x55c8d0f61020 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T10:32:04.200 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:32:03.854+0000 7f0fe81f4740 1 --2- 172.21.15.114:0/476797864 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55c8d0ee14c0 0x55c8d0f61020 secure :-1 s=READY pgs=720 cs=0 l=1 rev1=1 crypto rx=0x7f0fd405e040 tx=0x7f0fd4005ed0 comp rx=0 tx=0).stop 2024-04-23T10:32:04.200 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:32:03.854+0000 7f0fe81f4740 1 -- 172.21.15.114:0/476797864 shutdown_connections 2024-04-23T10:32:04.200 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:32:03.854+0000 7f0fe81f4740 1 --2- 172.21.15.114:0/476797864 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x55c8d0ef00a0 0x55c8d0f7cbd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:32:04.200 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:32:03.854+0000 7f0fe81f4740 1 --2- 172.21.15.114:0/476797864 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55c8d0ee14c0 0x55c8d0f61020 unknown :-1 s=CLOSED pgs=720 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:32:04.200 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:32:03.854+0000 7f0fe81f4740 1 --2- 172.21.15.114:0/476797864 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x55c8d0ee0ba0 0x55c8d0f02cc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:32:04.201 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:32:03.854+0000 7f0fe81f4740 1 -- 172.21.15.114:0/476797864 >> 172.21.15.114:0/476797864 conn(0x55c8d0edcfc0 msgr2=0x55c8d0ee8460 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T10:32:04.201 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:32:03.854+0000 7f0fe81f4740 1 -- 172.21.15.114:0/476797864 shutdown_connections 2024-04-23T10:32:04.201 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:32:03.854+0000 7f0fe81f4740 1 -- 172.21.15.114:0/476797864 wait complete. 2024-04-23T10:32:04.201 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:32:03.854+0000 7f0fe81f4740 1 -- start start 2024-04-23T10:32:04.201 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:32:03.854+0000 7f0fe81f4740 1 Processor -- start 2024-04-23T10:32:04.201 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:32:03.854+0000 7f0fe81f4740 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55c8d0ee0ba0 0x55c8d0f02cc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T10:32:04.201 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:32:03.854+0000 7f0fe81f4740 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x55c8d0ee14c0 0x55c8d0f61020 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T10:32:04.201 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:32:03.854+0000 7f0fe81f4740 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x55c8d0ef00a0 0x55c8d0f7cbd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T10:32:04.201 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:32:03.854+0000 7f0fe81f4740 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x55c8d0ee9da0 con 0x55c8d0ee0ba0 2024-04-23T10:32:04.201 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:32:03.854+0000 7f0fe81f4740 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 -- 0x55c8d0efb1f0 con 0x55c8d0ef00a0 2024-04-23T10:32:04.201 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:32:03.854+0000 7f0fe81f4740 1 -- --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_getmap magic: 0 -- 0x55c8d0eeca40 con 0x55c8d0ee14c0 2024-04-23T10:32:04.201 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:32:03.854+0000 7f0fe81f4740 10 client.0 _ll_register_callbacks cb 0x7f0fd0009b20 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-23T10:32:04.201 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:32:03.854+0000 7f0fdeffd640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55c8d0ee0ba0 0x55c8d0f02cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T10:32:04.201 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:32:03.854+0000 7f0fdeffd640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55c8d0ee0ba0 0x55c8d0f02cc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.114:52034/0 (socket says 172.21.15.114:52034) 2024-04-23T10:32:04.202 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:32:03.854+0000 7f0fdeffd640 1 -- 172.21.15.114:0/3971074661 learned_addr learned my addr 172.21.15.114:0/3971074661 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-04-23T10:32:04.202 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:32:03.854+0000 7f0fde7fc640 1 --2- 172.21.15.114:0/3971074661 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x55c8d0ee14c0 0x55c8d0f61020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T10:32:04.202 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:32:03.855+0000 7f0fdf7fe640 1 --2- 172.21.15.114:0/3971074661 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x55c8d0ef00a0 0x55c8d0f7cbd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T10:32:04.202 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:32:03.855+0000 7f0fdeffd640 1 -- 172.21.15.114:0/3971074661 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x55c8d0ee14c0 msgr2=0x55c8d0f61020 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T10:32:04.202 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:32:03.855+0000 7f0fdeffd640 1 --2- 172.21.15.114:0/3971074661 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x55c8d0ee14c0 0x55c8d0f61020 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:32:04.202 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:32:03.855+0000 7f0fdeffd640 1 -- 172.21.15.114:0/3971074661 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x55c8d0ef00a0 msgr2=0x55c8d0f7cbd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T10:32:04.202 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:32:03.855+0000 7f0fdeffd640 1 --2- 172.21.15.114:0/3971074661 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x55c8d0ef00a0 0x55c8d0f7cbd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:32:04.202 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:32:03.855+0000 7f0fdeffd640 1 -- 172.21.15.114:0/3971074661 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55c8d0ee0290 con 0x55c8d0ee0ba0 2024-04-23T10:32:04.202 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:32:03.855+0000 7f0fde7fc640 1 --2- 172.21.15.114:0/3971074661 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x55c8d0ee14c0 0x55c8d0f61020 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-23T10:32:04.202 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:32:03.855+0000 7f0fdeffd640 1 --2- 172.21.15.114:0/3971074661 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55c8d0ee0ba0 0x55c8d0f02cc0 secure :-1 s=READY pgs=721 cs=0 l=1 rev1=1 crypto rx=0x7f0fd002ade0 tx=0x7f0fd0009880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=71722b4ab7295fa5 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T10:32:04.202 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:32:03.855+0000 7f0fc2ff5640 10 client.0 ms_handle_connect on v2:172.21.15.114:3300/0 2024-04-23T10:32:04.202 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:32:03.855+0000 7f0fc2ff5640 1 -- 172.21.15.114:0/3971074661 <== mon.0 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0fd0077040 con 0x55c8d0ee0ba0 2024-04-23T10:32:04.202 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:32:03.855+0000 7f0fc2ff5640 1 -- 172.21.15.114:0/3971074661 <== mon.0 v2:172.21.15.114:3300/0 2 ==== osd_map(94..94 src has 1..94) ==== 8585+0+0 (secure 0 0 0) 0x7f0fd0031bb0 con 0x55c8d0ee0ba0 2024-04-23T10:32:04.203 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:32:03.855+0000 7f0fe81f4740 1 -- 172.21.15.114:0/3971074661 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f0fd4004490 con 0x55c8d0ee0ba0 2024-04-23T10:32:04.203 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:32:03.855+0000 7f0fc2ff5640 1 -- 172.21.15.114:0/3971074661 <== mon.0 v2:172.21.15.114:3300/0 3 ==== mon_get_version_reply(handle=1 version=30) ==== 24+0+0 (secure 0 0 0) 0x7f0fd0077850 con 0x55c8d0ee0ba0 2024-04-23T10:32:04.203 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:32:03.855+0000 7f0fe81f4740 10 client.16344 fetch_fsmap learned FSMap version 30 2024-04-23T10:32:04.203 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:32:03.855+0000 7f0fe81f4740 1 -- 172.21.15.114:0/3971074661 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({config=0+,fsmap=30,monmap=4+}) -- 0x55c8d0f09070 con 0x55c8d0ee0ba0 2024-04-23T10:32:04.203 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:32:03.855+0000 7f0fc2ff5640 1 -- 172.21.15.114:0/3971074661 <== mon.0 v2:172.21.15.114:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f0fd4004490 con 0x55c8d0ee0ba0 2024-04-23T10:32:04.203 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:32:03.855+0000 7f0fc2ff5640 1 -- 172.21.15.114:0/3971074661 <== mon.0 v2:172.21.15.114:3300/0 5 ==== fsmap(e 30) ==== 8338+0+0 (secure 0 0 0) 0x7f0fd402a450 con 0x55c8d0ee0ba0 2024-04-23T10:32:04.203 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:32:03.855+0000 7f0fe81f4740 10 client.16344 fetch_fsmap finished waiting for FSMap version 30 2024-04-23T10:32:04.203 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:32:03.855+0000 7f0fe81f4740 10 client.16344 resolve_mds: resolved name 'h' to daemon mds.h 2024-04-23T10:32:04.203 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:32:03.855+0000 7f0fe81f4740 20 client.16344 populate_metadata read hostname 'smithi114' 2024-04-23T10:32:04.203 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:32:03.855+0000 7f0fe81f4740 1 --2- 172.21.15.114:0/3971074661 >> [v2:172.21.15.119:6836/1872363128,v1:172.21.15.119:6837/1872363128] conn(0x55c8d0ee9aa0 0x55c8d0f6a910 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T10:32:04.203 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:32:03.855+0000 7f0fe81f4740 4 client.16344 mds_command: new command op to 24509 tid=0 multi_id=0 [{"prefix": "ops", "flags": ["locks"]}] 2024-04-23T10:32:04.203 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:32:03.855+0000 7f0fe81f4740 1 -- 172.21.15.114:0/3971074661 --> [v2:172.21.15.119:6836/1872363128,v1:172.21.15.119:6837/1872363128] -- command(tid 0: {"prefix": "ops", "flags": ["locks"]}) -- 0x55c8d0f6a5d0 con 0x55c8d0ee9aa0 2024-04-23T10:32:04.203 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:32:03.856+0000 7f0fde7fc640 1 --2- 172.21.15.114:0/3971074661 >> [v2:172.21.15.119:6836/1872363128,v1:172.21.15.119:6837/1872363128] conn(0x55c8d0ee9aa0 0x55c8d0f6a910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T10:32:04.203 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:32:03.856+0000 7f0fc2ff5640 10 client.16344 ms_handle_connect on v2:172.21.15.119:6836/1872363128 2024-04-23T10:32:04.203 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:32:03.856+0000 7f0fde7fc640 1 --2- 172.21.15.114:0/3971074661 >> [v2:172.21.15.119:6836/1872363128,v1:172.21.15.119:6837/1872363128] conn(0x55c8d0ee9aa0 0x55c8d0f6a910 crc :-1 s=READY pgs=38 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.1 client_cookie=7e42cf3d2f5e2a09 server_cookie=2f5147e0c5479de0 in_seq=0 out_seq=0 2024-04-23T10:32:04.204 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:32:03.859+0000 7f0fc2ff5640 1 -- 172.21.15.114:0/3971074661 <== mds.1 v2:172.21.15.119:6836/1872363128 1 ==== command_reply(tid 0: 0 ) ==== 8+0+136174 (crc 0 0 0) 0x7f0fd40094e0 con 0x55c8d0ee9aa0 2024-04-23T10:32:04.204 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:32:03.859+0000 7f0fc2ff5640 10 client.16344 handle_command_reply: tid=0 2024-04-23T10:32:04.204 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:32:03.859+0000 7f0fe81f4740 1 client.16344 shutdown 2024-04-23T10:32:04.204 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:32:03.859+0000 7f0fe81f4740 1 -- 172.21.15.114:0/3971074661 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55c8d0ee0ba0 msgr2=0x55c8d0f02cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T10:32:04.204 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:32:03.859+0000 7f0fe81f4740 1 --2- 172.21.15.114:0/3971074661 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55c8d0ee0ba0 0x55c8d0f02cc0 secure :-1 s=READY pgs=721 cs=0 l=1 rev1=1 crypto rx=0x7f0fd002ade0 tx=0x7f0fd0009880 comp rx=0 tx=0).stop 2024-04-23T10:32:04.204 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:32:03.859+0000 7f0fe81f4740 1 -- 172.21.15.114:0/3971074661 shutdown_connections 2024-04-23T10:32:04.204 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:32:03.859+0000 7f0fe81f4740 1 --2- 172.21.15.114:0/3971074661 >> [v2:172.21.15.119:6836/1872363128,v1:172.21.15.119:6837/1872363128] conn(0x55c8d0ee9aa0 0x55c8d0f6a910 crc :-1 s=READY pgs=38 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:32:04.204 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:32:03.859+0000 7f0fe81f4740 1 --2- 172.21.15.114:0/3971074661 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x55c8d0ef00a0 0x55c8d0f7cbd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:32:04.204 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:32:03.859+0000 7f0fe81f4740 1 --2- 172.21.15.114:0/3971074661 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x55c8d0ee14c0 0x55c8d0f61020 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:32:04.204 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:32:03.859+0000 7f0fc2ff5640 0 client.16344 ms_handle_reset on v2:172.21.15.119:6836/1872363128 2024-04-23T10:32:04.204 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:32:03.859+0000 7f0fe81f4740 1 --2- 172.21.15.114:0/3971074661 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55c8d0ee0ba0 0x55c8d0f02cc0 unknown :-1 s=CLOSED pgs=721 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:32:04.204 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:32:03.859+0000 7f0fe81f4740 1 -- 172.21.15.114:0/3971074661 >> 172.21.15.114:0/3971074661 conn(0x55c8d0edcfc0 msgr2=0x55c8d0f67f50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T10:32:04.204 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:32:03.859+0000 7f0fe81f4740 1 -- 172.21.15.114:0/3971074661 shutdown_connections 2024-04-23T10:32:04.204 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:32:03.859+0000 7f0fe81f4740 1 -- 172.21.15.114:0/3971074661 wait complete. 2024-04-23T10:32:04.204 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:32:03.859+0000 7f0fe81f4740 20 client.16344 trim_cache size 0 max 16384 2024-04-23T10:32:04.206 INFO:teuthology.orchestra.run.smithi114.stdout:mds.h: { 2024-04-23T10:32:04.206 INFO:teuthology.orchestra.run.smithi114.stdout: "ops": [ 2024-04-23T10:32:04.206 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-04-23T10:32:04.206 INFO:teuthology.orchestra.run.smithi114.stdout: "description": "internal op quiesce_inode:mds.1:131 fp=#0x10000000233", 2024-04-23T10:32:04.206 INFO:teuthology.orchestra.run.smithi114.stdout: "initiated_at": "2024-04-23T10:31:21.494203+0000", 2024-04-23T10:32:04.207 INFO:teuthology.orchestra.run.smithi114.stdout: "age": 42.363435672999998, 2024-04-23T10:32:04.207 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 42.363456708999998, 2024-04-23T10:32:04.207 INFO:teuthology.orchestra.run.smithi114.stdout: "continuous": true, 2024-04-23T10:32:04.207 INFO:teuthology.orchestra.run.smithi114.stdout: "type_data": { 2024-04-23T10:32:04.207 INFO:teuthology.orchestra.run.smithi114.stdout: "result": -2147483648, 2024-04-23T10:32:04.207 INFO:teuthology.orchestra.run.smithi114.stdout: "flag_point": "failed to authpin, dir is being fragmented", 2024-04-23T10:32:04.207 INFO:teuthology.orchestra.run.smithi114.stdout: "reqid": { 2024-04-23T10:32:04.207 INFO:teuthology.orchestra.run.smithi114.stdout: "entity": { 2024-04-23T10:32:04.207 INFO:teuthology.orchestra.run.smithi114.stdout: "type": "mds", 2024-04-23T10:32:04.207 INFO:teuthology.orchestra.run.smithi114.stdout: "num": 1 2024-04-23T10:32:04.207 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-23T10:32:04.207 INFO:teuthology.orchestra.run.smithi114.stdout: "tid": 131 2024-04-23T10:32:04.207 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-23T10:32:04.207 INFO:teuthology.orchestra.run.smithi114.stdout: "op_type": "internal_op", 2024-04-23T10:32:04.208 INFO:teuthology.orchestra.run.smithi114.stdout: "internal_op": 5385, 2024-04-23T10:32:04.208 INFO:teuthology.orchestra.run.smithi114.stdout: "op_name": "quiesce_inode", 2024-04-23T10:32:04.208 INFO:teuthology.orchestra.run.smithi114.stdout: "events": [ 2024-04-23T10:32:04.208 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-04-23T10:32:04.208 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-04-23T10:31:21.494203+0000", 2024-04-23T10:32:04.208 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "initiated" 2024-04-23T10:32:04.208 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-23T10:32:04.208 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-04-23T10:32:04.208 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-04-23T10:31:21.494203+0000", 2024-04-23T10:32:04.208 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "header_read" 2024-04-23T10:32:04.208 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-23T10:32:04.208 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-04-23T10:32:04.208 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-04-23T10:31:21.494203+0000", 2024-04-23T10:32:04.208 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "throttled" 2024-04-23T10:32:04.208 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-23T10:32:04.208 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-04-23T10:32:04.209 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-04-23T10:31:21.494203+0000", 2024-04-23T10:32:04.209 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "all_read" 2024-04-23T10:32:04.209 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-23T10:32:04.209 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-04-23T10:32:04.209 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-04-23T10:31:21.494203+0000", 2024-04-23T10:32:04.209 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "dispatched" 2024-04-23T10:32:04.209 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-23T10:32:04.209 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-04-23T10:32:04.209 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-04-23T10:31:21.495293+0000", 2024-04-23T10:32:04.209 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "failed to authpin, dir is being fragmented" 2024-04-23T10:32:04.209 INFO:teuthology.orchestra.run.smithi114.stdout: } 2024-04-23T10:32:04.209 INFO:teuthology.orchestra.run.smithi114.stdout: ], 2024-04-23T10:32:04.209 INFO:teuthology.orchestra.run.smithi114.stdout: "locks": [] 2024-04-23T10:32:04.209 INFO:teuthology.orchestra.run.smithi114.stdout: } 2024-04-23T10:32:04.210 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-23T10:32:04.210 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-04-23T10:32:04.210 INFO:teuthology.orchestra.run.smithi114.stdout: "description": "client_request(client.15312:74454 unlink #0x10000000229/COURSES.VAL 2024-04-23T10:31:17.777716+0000 caller_uid=1000, caller_gid=1316{6,36,1000,1316,})", 2024-04-23T10:32:04.210 INFO:teuthology.orchestra.run.smithi114.stdout: "initiated_at": "2024-04-23T10:31:17.778339+0000", 2024-04-23T10:32:04.210 INFO:teuthology.orchestra.run.smithi114.stdout: "age": 46.079299241999998, 2024-04-23T10:32:04.210 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 46.079362150999998, 2024-04-23T10:32:04.210 INFO:teuthology.orchestra.run.smithi114.stdout: "continuous": false, 2024-04-23T10:32:04.210 INFO:teuthology.orchestra.run.smithi114.stdout: "type_data": { 2024-04-23T10:32:04.210 INFO:teuthology.orchestra.run.smithi114.stdout: "result": -2147483648, 2024-04-23T10:32:04.210 INFO:teuthology.orchestra.run.smithi114.stdout: "flag_point": "failed to authpin, dir is being fragmented", 2024-04-23T10:32:04.210 INFO:teuthology.orchestra.run.smithi114.stdout: "reqid": { 2024-04-23T10:32:04.210 INFO:teuthology.orchestra.run.smithi114.stdout: "entity": { 2024-04-23T10:32:04.210 INFO:teuthology.orchestra.run.smithi114.stdout: "type": "client", 2024-04-23T10:32:04.210 INFO:teuthology.orchestra.run.smithi114.stdout: "num": 15312 2024-04-23T10:32:04.210 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-23T10:32:04.211 INFO:teuthology.orchestra.run.smithi114.stdout: "tid": 74454 2024-04-23T10:32:04.211 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-23T10:32:04.211 INFO:teuthology.orchestra.run.smithi114.stdout: "op_type": "client_request", 2024-04-23T10:32:04.211 INFO:teuthology.orchestra.run.smithi114.stdout: "events": [ 2024-04-23T10:32:04.211 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-04-23T10:32:04.211 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-04-23T10:31:17.778339+0000", 2024-04-23T10:32:04.211 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "initiated" 2024-04-23T10:32:04.211 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-23T10:32:04.211 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-04-23T10:32:04.211 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-04-23T10:31:17.778339+0000", 2024-04-23T10:32:04.211 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "header_read" 2024-04-23T10:32:04.211 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-23T10:32:04.211 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-04-23T10:32:04.211 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-04-23T10:31:17.778340+0000", 2024-04-23T10:32:04.212 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "throttled" 2024-04-23T10:32:04.212 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-23T10:32:04.212 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-04-23T10:32:04.212 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-04-23T10:31:17.778345+0000", 2024-04-23T10:32:04.212 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "all_read" 2024-04-23T10:32:04.212 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-23T10:32:04.212 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-04-23T10:32:04.212 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-04-23T10:31:17.778370+0000", 2024-04-23T10:32:04.212 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "dispatched" 2024-04-23T10:32:04.212 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-23T10:32:04.212 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-04-23T10:32:04.212 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-04-23T10:31:17.778657+0000", 2024-04-23T10:32:04.212 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "failed to authpin, dir is being fragmented" 2024-04-23T10:32:04.212 INFO:teuthology.orchestra.run.smithi114.stdout: } 2024-04-23T10:32:04.212 INFO:teuthology.orchestra.run.smithi114.stdout: ], 2024-04-23T10:32:04.213 INFO:teuthology.orchestra.run.smithi114.stdout: "locks": [] 2024-04-23T10:32:04.213 INFO:teuthology.orchestra.run.smithi114.stdout: } 2024-04-23T10:32:04.213 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-23T10:32:04.213 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-04-23T10:32:04.213 INFO:teuthology.orchestra.run.smithi114.stdout: "description": "internal op quiesce_inode:mds.1:132 fp=#0x10000000239", 2024-04-23T10:32:04.213 INFO:teuthology.orchestra.run.smithi114.stdout: "initiated_at": "2024-04-23T10:31:21.494238+0000", 2024-04-23T10:32:04.213 INFO:teuthology.orchestra.run.smithi114.stdout: "age": 42.363400931999998, 2024-04-23T10:32:04.213 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 42.363489846999997, 2024-04-23T10:32:04.213 INFO:teuthology.orchestra.run.smithi114.stdout: "continuous": true, 2024-04-23T10:32:04.213 INFO:teuthology.orchestra.run.smithi114.stdout: "type_data": { 2024-04-23T10:32:04.213 INFO:teuthology.orchestra.run.smithi114.stdout: "result": -2147483648, 2024-04-23T10:32:04.213 INFO:teuthology.orchestra.run.smithi114.stdout: "flag_point": "failed to authpin, dir is being fragmented", 2024-04-23T10:32:04.213 INFO:teuthology.orchestra.run.smithi114.stdout: "reqid": { 2024-04-23T10:32:04.213 INFO:teuthology.orchestra.run.smithi114.stdout: "entity": { 2024-04-23T10:32:04.214 INFO:teuthology.orchestra.run.smithi114.stdout: "type": "mds", 2024-04-23T10:32:04.214 INFO:teuthology.orchestra.run.smithi114.stdout: "num": 1 2024-04-23T10:32:04.214 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-23T10:32:04.214 INFO:teuthology.orchestra.run.smithi114.stdout: "tid": 132 2024-04-23T10:32:04.214 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-23T10:32:04.214 INFO:teuthology.orchestra.run.smithi114.stdout: "op_type": "internal_op", 2024-04-23T10:32:04.214 INFO:teuthology.orchestra.run.smithi114.stdout: "internal_op": 5385, 2024-04-23T10:32:04.214 INFO:teuthology.orchestra.run.smithi114.stdout: "op_name": "quiesce_inode", 2024-04-23T10:32:04.214 INFO:teuthology.orchestra.run.smithi114.stdout: "events": [ 2024-04-23T10:32:04.214 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-04-23T10:32:04.214 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-04-23T10:31:21.494238+0000", 2024-04-23T10:32:04.214 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "initiated" 2024-04-23T10:32:04.214 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-23T10:32:04.214 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-04-23T10:32:04.215 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-04-23T10:31:21.494238+0000", 2024-04-23T10:32:04.215 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "header_read" 2024-04-23T10:32:04.215 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-23T10:32:04.215 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-04-23T10:32:04.215 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-04-23T10:31:21.494238+0000", 2024-04-23T10:32:04.215 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "throttled" 2024-04-23T10:32:04.215 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-23T10:32:04.215 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-04-23T10:32:04.215 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-04-23T10:31:21.494238+0000", 2024-04-23T10:32:04.215 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "all_read" 2024-04-23T10:32:04.215 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-23T10:32:04.215 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-04-23T10:32:04.215 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-04-23T10:31:21.494238+0000", 2024-04-23T10:32:04.215 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "dispatched" 2024-04-23T10:32:04.215 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-23T10:32:04.216 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-04-23T10:32:04.216 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-04-23T10:31:21.495383+0000", 2024-04-23T10:32:04.216 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "failed to authpin, dir is being fragmented" 2024-04-23T10:32:04.216 INFO:teuthology.orchestra.run.smithi114.stdout: } 2024-04-23T10:32:04.216 INFO:teuthology.orchestra.run.smithi114.stdout: ], 2024-04-23T10:32:04.216 INFO:teuthology.orchestra.run.smithi114.stdout: "locks": [] 2024-04-23T10:32:04.216 INFO:teuthology.orchestra.run.smithi114.stdout: } 2024-04-23T10:32:04.216 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-23T10:32:04.216 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-04-23T10:32:04.216 INFO:teuthology.orchestra.run.smithi114.stdout: "description": "internal op quiesce_path:mds.1:101 fp=#0x1", 2024-04-23T10:32:04.216 INFO:teuthology.orchestra.run.smithi114.stdout: "initiated_at": "2024-04-23T10:31:21.445231+0000", 2024-04-23T10:32:04.216 INFO:teuthology.orchestra.run.smithi114.stdout: "age": 42.412407909000002, 2024-04-23T10:32:04.216 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 42.412538318999999, 2024-04-23T10:32:04.216 INFO:teuthology.orchestra.run.smithi114.stdout: "continuous": true, 2024-04-23T10:32:04.217 INFO:teuthology.orchestra.run.smithi114.stdout: "type_data": { 2024-04-23T10:32:04.217 INFO:teuthology.orchestra.run.smithi114.stdout: "result": -2147483648, 2024-04-23T10:32:04.217 INFO:teuthology.orchestra.run.smithi114.stdout: "flag_point": "dispatched", 2024-04-23T10:32:04.217 INFO:teuthology.orchestra.run.smithi114.stdout: "reqid": { 2024-04-23T10:32:04.217 INFO:teuthology.orchestra.run.smithi114.stdout: "entity": { 2024-04-23T10:32:04.217 INFO:teuthology.orchestra.run.smithi114.stdout: "type": "mds", 2024-04-23T10:32:04.217 INFO:teuthology.orchestra.run.smithi114.stdout: "num": 1 2024-04-23T10:32:04.217 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-23T10:32:04.217 INFO:teuthology.orchestra.run.smithi114.stdout: "tid": 101 2024-04-23T10:32:04.217 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-23T10:32:04.217 INFO:teuthology.orchestra.run.smithi114.stdout: "op_type": "internal_op", 2024-04-23T10:32:04.217 INFO:teuthology.orchestra.run.smithi114.stdout: "internal_op": 5384, 2024-04-23T10:32:04.217 INFO:teuthology.orchestra.run.smithi114.stdout: "op_name": "quiesce_path", 2024-04-23T10:32:04.217 INFO:teuthology.orchestra.run.smithi114.stdout: "events": [ 2024-04-23T10:32:04.218 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-04-23T10:32:04.218 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-04-23T10:31:21.445231+0000", 2024-04-23T10:32:04.218 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "initiated" 2024-04-23T10:32:04.218 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-23T10:32:04.218 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-04-23T10:32:04.218 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-04-23T10:31:21.445231+0000", 2024-04-23T10:32:04.218 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "header_read" 2024-04-23T10:32:04.218 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-23T10:32:04.218 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-04-23T10:32:04.218 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-04-23T10:31:21.445231+0000", 2024-04-23T10:32:04.218 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "throttled" 2024-04-23T10:32:04.218 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-23T10:32:04.218 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-04-23T10:32:04.218 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-04-23T10:31:21.445231+0000", 2024-04-23T10:32:04.219 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "all_read" 2024-04-23T10:32:04.219 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-23T10:32:04.219 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-04-23T10:32:04.219 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-04-23T10:31:21.445231+0000", 2024-04-23T10:32:04.219 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "dispatched" 2024-04-23T10:32:04.219 INFO:teuthology.orchestra.run.smithi114.stdout: } 2024-04-23T10:32:04.219 INFO:teuthology.orchestra.run.smithi114.stdout: ], 2024-04-23T10:32:04.219 INFO:teuthology.orchestra.run.smithi114.stdout: "locks": [] 2024-04-23T10:32:04.219 INFO:teuthology.orchestra.run.smithi114.stdout: } 2024-04-23T10:32:04.219 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-23T10:32:04.219 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-04-23T10:32:04.219 INFO:teuthology.orchestra.run.smithi114.stdout: "description": "internal op quiesce_inode:mds.1:133 fp=#0x1000000023d", 2024-04-23T10:32:04.219 INFO:teuthology.orchestra.run.smithi114.stdout: "initiated_at": "2024-04-23T10:31:21.494254+0000", 2024-04-23T10:32:04.219 INFO:teuthology.orchestra.run.smithi114.stdout: "age": 42.363384936000003, 2024-04-23T10:32:04.220 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 42.363535954, 2024-04-23T10:32:04.220 INFO:teuthology.orchestra.run.smithi114.stdout: "continuous": true, 2024-04-23T10:32:04.220 INFO:teuthology.orchestra.run.smithi114.stdout: "type_data": { 2024-04-23T10:32:04.220 INFO:teuthology.orchestra.run.smithi114.stdout: "result": -2147483648, 2024-04-23T10:32:04.220 INFO:teuthology.orchestra.run.smithi114.stdout: "flag_point": "failed to authpin, dir is being fragmented", 2024-04-23T10:32:04.220 INFO:teuthology.orchestra.run.smithi114.stdout: "reqid": { 2024-04-23T10:32:04.220 INFO:teuthology.orchestra.run.smithi114.stdout: "entity": { 2024-04-23T10:32:04.220 INFO:teuthology.orchestra.run.smithi114.stdout: "type": "mds", 2024-04-23T10:32:04.220 INFO:teuthology.orchestra.run.smithi114.stdout: "num": 1 2024-04-23T10:32:04.220 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-23T10:32:04.220 INFO:teuthology.orchestra.run.smithi114.stdout: "tid": 133 2024-04-23T10:32:04.220 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-23T10:32:04.220 INFO:teuthology.orchestra.run.smithi114.stdout: "op_type": "internal_op", 2024-04-23T10:32:04.220 INFO:teuthology.orchestra.run.smithi114.stdout: "internal_op": 5385, 2024-04-23T10:32:04.220 INFO:teuthology.orchestra.run.smithi114.stdout: "op_name": "quiesce_inode", 2024-04-23T10:32:04.221 INFO:teuthology.orchestra.run.smithi114.stdout: "events": [ 2024-04-23T10:32:04.221 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-04-23T10:32:04.221 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-04-23T10:31:21.494254+0000", 2024-04-23T10:32:04.221 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "initiated" 2024-04-23T10:32:04.221 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-23T10:32:04.221 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-04-23T10:32:04.221 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-04-23T10:31:21.494254+0000", 2024-04-23T10:32:04.221 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "header_read" 2024-04-23T10:32:04.221 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-23T10:32:04.221 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-04-23T10:32:04.221 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-04-23T10:31:21.494254+0000", 2024-04-23T10:32:04.221 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "throttled" 2024-04-23T10:32:04.221 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-23T10:32:04.221 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-04-23T10:32:04.221 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-04-23T10:31:21.494254+0000", 2024-04-23T10:32:04.222 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "all_read" 2024-04-23T10:32:04.222 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-23T10:32:04.222 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-04-23T10:32:04.222 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-04-23T10:31:21.494254+0000", 2024-04-23T10:32:04.222 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "dispatched" 2024-04-23T10:32:04.222 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-23T10:32:04.222 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-04-23T10:32:04.222 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-04-23T10:31:21.495478+0000", 2024-04-23T10:32:04.222 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "failed to authpin, dir is being fragmented" 2024-04-23T10:32:04.222 INFO:teuthology.orchestra.run.smithi114.stdout: } 2024-04-23T10:32:04.222 INFO:teuthology.orchestra.run.smithi114.stdout: ], 2024-04-23T10:32:04.222 INFO:teuthology.orchestra.run.smithi114.stdout: "locks": [] 2024-04-23T10:32:04.222 INFO:teuthology.orchestra.run.smithi114.stdout: } 2024-04-23T10:32:04.223 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-23T10:32:04.223 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-04-23T10:32:04.223 INFO:teuthology.orchestra.run.smithi114.stdout: "description": "internal op quiesce_inode:mds.1:102 fp=#0x1", 2024-04-23T10:32:04.223 INFO:teuthology.orchestra.run.smithi114.stdout: "initiated_at": "2024-04-23T10:31:21.445286+0000", 2024-04-23T10:32:04.223 INFO:teuthology.orchestra.run.smithi114.stdout: "age": 42.412352829, 2024-04-23T10:32:04.223 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 42.412528283, 2024-04-23T10:32:04.223 INFO:teuthology.orchestra.run.smithi114.stdout: "continuous": true, 2024-04-23T10:32:04.223 INFO:teuthology.orchestra.run.smithi114.stdout: "type_data": { 2024-04-23T10:32:04.223 INFO:teuthology.orchestra.run.smithi114.stdout: "result": -2147483648, 2024-04-23T10:32:04.223 INFO:teuthology.orchestra.run.smithi114.stdout: "flag_point": "acquired locks", 2024-04-23T10:32:04.223 INFO:teuthology.orchestra.run.smithi114.stdout: "reqid": { 2024-04-23T10:32:04.223 INFO:teuthology.orchestra.run.smithi114.stdout: "entity": { 2024-04-23T10:32:04.223 INFO:teuthology.orchestra.run.smithi114.stdout: "type": "mds", 2024-04-23T10:32:04.223 INFO:teuthology.orchestra.run.smithi114.stdout: "num": 1 2024-04-23T10:32:04.224 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-23T10:32:04.224 INFO:teuthology.orchestra.run.smithi114.stdout: "tid": 102 2024-04-23T10:32:04.224 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-23T10:32:04.224 INFO:teuthology.orchestra.run.smithi114.stdout: "op_type": "internal_op", 2024-04-23T10:32:04.224 INFO:teuthology.orchestra.run.smithi114.stdout: "internal_op": 5385, 2024-04-23T10:32:04.224 INFO:teuthology.orchestra.run.smithi114.stdout: "op_name": "quiesce_inode", 2024-04-23T10:32:04.224 INFO:teuthology.orchestra.run.smithi114.stdout: "events": [ 2024-04-23T10:32:04.224 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-04-23T10:32:04.224 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-04-23T10:31:21.445286+0000", 2024-04-23T10:32:04.224 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "initiated" 2024-04-23T10:32:04.224 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-23T10:32:04.224 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-04-23T10:32:04.224 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-04-23T10:31:21.445286+0000", 2024-04-23T10:32:04.224 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "header_read" 2024-04-23T10:32:04.224 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-23T10:32:04.224 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-04-23T10:32:04.225 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-04-23T10:31:21.445286+0000", 2024-04-23T10:32:04.225 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "throttled" 2024-04-23T10:32:04.225 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-23T10:32:04.225 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-04-23T10:32:04.225 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-04-23T10:31:21.445286+0000", 2024-04-23T10:32:04.225 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "all_read" 2024-04-23T10:32:04.225 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-23T10:32:04.225 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-04-23T10:32:04.225 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-04-23T10:31:21.445286+0000", 2024-04-23T10:32:04.225 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "dispatched" 2024-04-23T10:32:04.225 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-23T10:32:04.225 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-04-23T10:32:04.225 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-04-23T10:31:21.445409+0000", 2024-04-23T10:32:04.225 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "requesting remote authpins" 2024-04-23T10:32:04.225 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-23T10:32:04.225 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-04-23T10:32:04.226 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-04-23T10:31:21.446064+0000", 2024-04-23T10:32:04.226 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "acquired locks" 2024-04-23T10:32:04.226 INFO:teuthology.orchestra.run.smithi114.stdout: } 2024-04-23T10:32:04.226 INFO:teuthology.orchestra.run.smithi114.stdout: ], 2024-04-23T10:32:04.226 INFO:teuthology.orchestra.run.smithi114.stdout: "locks": [ 2024-04-23T10:32:04.226 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-04-23T10:32:04.226 INFO:teuthology.orchestra.run.smithi114.stdout: "object": { 2024-04-23T10:32:04.226 INFO:teuthology.orchestra.run.smithi114.stdout: "is_auth": false, 2024-04-23T10:32:04.226 INFO:teuthology.orchestra.run.smithi114.stdout: "auth_state": { 2024-04-23T10:32:04.226 INFO:teuthology.orchestra.run.smithi114.stdout: "replicas": {} 2024-04-23T10:32:04.226 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-23T10:32:04.226 INFO:teuthology.orchestra.run.smithi114.stdout: "replica_state": { 2024-04-23T10:32:04.226 INFO:teuthology.orchestra.run.smithi114.stdout: "authority": [ 2024-04-23T10:32:04.226 INFO:teuthology.orchestra.run.smithi114.stdout: 0, 2024-04-23T10:32:04.227 INFO:teuthology.orchestra.run.smithi114.stdout: -2 2024-04-23T10:32:04.227 INFO:teuthology.orchestra.run.smithi114.stdout: ], 2024-04-23T10:32:04.227 INFO:teuthology.orchestra.run.smithi114.stdout: "replica_nonce": 1 2024-04-23T10:32:04.227 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-23T10:32:04.227 INFO:teuthology.orchestra.run.smithi114.stdout: "auth_pins": 0, 2024-04-23T10:32:04.227 INFO:teuthology.orchestra.run.smithi114.stdout: "is_frozen": false, 2024-04-23T10:32:04.227 INFO:teuthology.orchestra.run.smithi114.stdout: "is_freezing": false, 2024-04-23T10:32:04.227 INFO:teuthology.orchestra.run.smithi114.stdout: "pins": { 2024-04-23T10:32:04.227 INFO:teuthology.orchestra.run.smithi114.stdout: "request": 1, 2024-04-23T10:32:04.227 INFO:teuthology.orchestra.run.smithi114.stdout: "lock": 1, 2024-04-23T10:32:04.227 INFO:teuthology.orchestra.run.smithi114.stdout: "dirfrag": 1, 2024-04-23T10:32:04.227 INFO:teuthology.orchestra.run.smithi114.stdout: "caps": 1, 2024-04-23T10:32:04.227 INFO:teuthology.orchestra.run.smithi114.stdout: "dirwaiter": 0, 2024-04-23T10:32:04.228 INFO:teuthology.orchestra.run.smithi114.stdout: "discoverbase": 0 2024-04-23T10:32:04.228 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-23T10:32:04.228 INFO:teuthology.orchestra.run.smithi114.stdout: "nref": 4 2024-04-23T10:32:04.228 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-23T10:32:04.228 INFO:teuthology.orchestra.run.smithi114.stdout: "object_string": "[inode 0x1 [...7,head] / rep@0.1 v187 snaprealm=0x55e71896c480 f(v0 m2024-04-23T10:24:48.676117+0000 1=0+1) n(v27 rc2024-04-23T10:31:17.764716+0000 b21075308 rs5 88=67+21)/n(v0 rc2024-04-23T10:24:44.319412+0000 1=0+1) (inest mix) (iquiesce lock x=1 by request(mds.1:102 nref=4)) caps={15576=pAsLsXs/-@4} | request=1 lock=1 dirfrag=1 caps=1 dirwaiter=0 discoverbase=0 0x55e718955600]", 2024-04-23T10:32:04.228 INFO:teuthology.orchestra.run.smithi114.stdout: "lock": { 2024-04-23T10:32:04.228 INFO:teuthology.orchestra.run.smithi114.stdout: "gather_set": [], 2024-04-23T10:32:04.228 INFO:teuthology.orchestra.run.smithi114.stdout: "state": "lock", 2024-04-23T10:32:04.228 INFO:teuthology.orchestra.run.smithi114.stdout: "type": "iquiesce", 2024-04-23T10:32:04.228 INFO:teuthology.orchestra.run.smithi114.stdout: "is_leased": false, 2024-04-23T10:32:04.228 INFO:teuthology.orchestra.run.smithi114.stdout: "num_rdlocks": 0, 2024-04-23T10:32:04.228 INFO:teuthology.orchestra.run.smithi114.stdout: "num_wrlocks": 0, 2024-04-23T10:32:04.228 INFO:teuthology.orchestra.run.smithi114.stdout: "num_xlocks": 1, 2024-04-23T10:32:04.228 INFO:teuthology.orchestra.run.smithi114.stdout: "xlock_by": { 2024-04-23T10:32:04.228 INFO:teuthology.orchestra.run.smithi114.stdout: "reqid": { 2024-04-23T10:32:04.229 INFO:teuthology.orchestra.run.smithi114.stdout: "entity": { 2024-04-23T10:32:04.229 INFO:teuthology.orchestra.run.smithi114.stdout: "type": "mds", 2024-04-23T10:32:04.229 INFO:teuthology.orchestra.run.smithi114.stdout: "num": 1 2024-04-23T10:32:04.229 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-23T10:32:04.229 INFO:teuthology.orchestra.run.smithi114.stdout: "tid": 102 2024-04-23T10:32:04.229 INFO:teuthology.orchestra.run.smithi114.stdout: } 2024-04-23T10:32:04.229 INFO:teuthology.orchestra.run.smithi114.stdout: } 2024-04-23T10:32:04.229 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-23T10:32:04.229 INFO:teuthology.orchestra.run.smithi114.stdout: "flags": 4, 2024-04-23T10:32:04.229 INFO:teuthology.orchestra.run.smithi114.stdout: "wrlock_target": -1 2024-04-23T10:32:04.229 INFO:teuthology.orchestra.run.smithi114.stdout: } 2024-04-23T10:32:04.229 INFO:teuthology.orchestra.run.smithi114.stdout: ] 2024-04-23T10:32:04.229 INFO:teuthology.orchestra.run.smithi114.stdout: } 2024-04-23T10:32:04.229 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-23T10:32:04.229 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-04-23T10:32:04.230 INFO:teuthology.orchestra.run.smithi114.stdout: "description": "internal op quiesce_inode:mds.1:134 fp=#0x10000000234", 2024-04-23T10:32:04.230 INFO:teuthology.orchestra.run.smithi114.stdout: "initiated_at": "2024-04-23T10:31:21.494267+0000", 2024-04-23T10:32:04.230 INFO:teuthology.orchestra.run.smithi114.stdout: "age": 42.363371192999999, 2024-04-23T10:32:04.230 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 42.363609945, 2024-04-23T10:32:04.230 INFO:teuthology.orchestra.run.smithi114.stdout: "continuous": true, 2024-04-23T10:32:04.230 INFO:teuthology.orchestra.run.smithi114.stdout: "type_data": { 2024-04-23T10:32:04.230 INFO:teuthology.orchestra.run.smithi114.stdout: "result": -2147483648, 2024-04-23T10:32:04.230 INFO:teuthology.orchestra.run.smithi114.stdout: "flag_point": "failed to authpin, dir is being fragmented", 2024-04-23T10:32:04.230 INFO:teuthology.orchestra.run.smithi114.stdout: "reqid": { 2024-04-23T10:32:04.230 INFO:teuthology.orchestra.run.smithi114.stdout: "entity": { 2024-04-23T10:32:04.230 INFO:teuthology.orchestra.run.smithi114.stdout: "type": "mds", 2024-04-23T10:32:04.230 INFO:teuthology.orchestra.run.smithi114.stdout: "num": 1 2024-04-23T10:32:04.230 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-23T10:32:04.230 INFO:teuthology.orchestra.run.smithi114.stdout: "tid": 134 2024-04-23T10:32:04.231 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-23T10:32:04.231 INFO:teuthology.orchestra.run.smithi114.stdout: "op_type": "internal_op", 2024-04-23T10:32:04.231 INFO:teuthology.orchestra.run.smithi114.stdout: "internal_op": 5385, 2024-04-23T10:32:04.231 INFO:teuthology.orchestra.run.smithi114.stdout: "op_name": "quiesce_inode", 2024-04-23T10:32:04.231 INFO:teuthology.orchestra.run.smithi114.stdout: "events": [ 2024-04-23T10:32:04.231 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-04-23T10:32:04.231 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-04-23T10:31:21.494267+0000", 2024-04-23T10:32:04.231 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "initiated" 2024-04-23T10:32:04.231 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-23T10:32:04.231 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-04-23T10:32:04.231 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-04-23T10:31:21.494267+0000", 2024-04-23T10:32:04.231 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "header_read" 2024-04-23T10:32:04.231 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-23T10:32:04.231 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-04-23T10:32:04.231 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-04-23T10:31:21.494267+0000", 2024-04-23T10:32:04.232 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "throttled" 2024-04-23T10:32:04.232 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-23T10:32:04.232 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-04-23T10:32:04.232 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-04-23T10:31:21.494267+0000", 2024-04-23T10:32:04.232 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "all_read" 2024-04-23T10:32:04.232 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-23T10:32:04.232 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-04-23T10:32:04.232 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-04-23T10:31:21.494267+0000", 2024-04-23T10:32:04.232 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "dispatched" 2024-04-23T10:32:04.232 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-23T10:32:04.232 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-04-23T10:32:04.232 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-04-23T10:31:21.495566+0000", 2024-04-23T10:32:04.232 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "failed to authpin, dir is being fragmented" 2024-04-23T10:32:04.232 INFO:teuthology.orchestra.run.smithi114.stdout: } 2024-04-23T10:32:04.233 INFO:teuthology.orchestra.run.smithi114.stdout: ], 2024-04-23T10:32:04.233 INFO:teuthology.orchestra.run.smithi114.stdout: "locks": [] 2024-04-23T10:32:04.233 INFO:teuthology.orchestra.run.smithi114.stdout: } 2024-04-23T10:32:04.233 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-23T10:32:04.233 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-04-23T10:32:04.233 INFO:teuthology.orchestra.run.smithi114.stdout: "description": "internal op quiesce_inode:mds.1:103 fp=#0x10000000000", 2024-04-23T10:32:04.233 INFO:teuthology.orchestra.run.smithi114.stdout: "initiated_at": "2024-04-23T10:31:21.446077+0000", 2024-04-23T10:32:04.233 INFO:teuthology.orchestra.run.smithi114.stdout: "age": 42.411561650000003, 2024-04-23T10:32:04.233 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 42.411824176000003, 2024-04-23T10:32:04.233 INFO:teuthology.orchestra.run.smithi114.stdout: "continuous": true, 2024-04-23T10:32:04.233 INFO:teuthology.orchestra.run.smithi114.stdout: "type_data": { 2024-04-23T10:32:04.233 INFO:teuthology.orchestra.run.smithi114.stdout: "result": -2147483648, 2024-04-23T10:32:04.233 INFO:teuthology.orchestra.run.smithi114.stdout: "flag_point": "acquired locks", 2024-04-23T10:32:04.233 INFO:teuthology.orchestra.run.smithi114.stdout: "reqid": { 2024-04-23T10:32:04.233 INFO:teuthology.orchestra.run.smithi114.stdout: "entity": { 2024-04-23T10:32:04.234 INFO:teuthology.orchestra.run.smithi114.stdout: "type": "mds", 2024-04-23T10:32:04.234 INFO:teuthology.orchestra.run.smithi114.stdout: "num": 1 2024-04-23T10:32:04.234 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-23T10:32:04.234 INFO:teuthology.orchestra.run.smithi114.stdout: "tid": 103 2024-04-23T10:32:04.234 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-23T10:32:04.234 INFO:teuthology.orchestra.run.smithi114.stdout: "op_type": "internal_op", 2024-04-23T10:32:04.234 INFO:teuthology.orchestra.run.smithi114.stdout: "internal_op": 5385, 2024-04-23T10:32:04.234 INFO:teuthology.orchestra.run.smithi114.stdout: "op_name": "quiesce_inode", 2024-04-23T10:32:04.234 INFO:teuthology.orchestra.run.smithi114.stdout: "events": [ 2024-04-23T10:32:04.234 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-04-23T10:32:04.234 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-04-23T10:31:21.446077+0000", 2024-04-23T10:32:04.234 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "initiated" 2024-04-23T10:32:04.234 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-23T10:32:04.234 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-04-23T10:32:04.235 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-04-23T10:31:21.446077+0000", 2024-04-23T10:32:04.235 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "header_read" 2024-04-23T10:32:04.235 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-23T10:32:04.235 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-04-23T10:32:04.235 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-04-23T10:31:21.446077+0000", 2024-04-23T10:32:04.235 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "throttled" 2024-04-23T10:32:04.235 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-23T10:32:04.235 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-04-23T10:32:04.235 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-04-23T10:31:21.446077+0000", 2024-04-23T10:32:04.235 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "all_read" 2024-04-23T10:32:04.235 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-23T10:32:04.235 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-04-23T10:32:04.235 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-04-23T10:31:21.446077+0000", 2024-04-23T10:32:04.235 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "dispatched" 2024-04-23T10:32:04.235 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-23T10:32:04.236 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-04-23T10:32:04.236 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-04-23T10:31:21.446137+0000", 2024-04-23T10:32:04.236 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "requesting remote authpins" 2024-04-23T10:32:04.236 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-23T10:32:04.236 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-04-23T10:32:04.236 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-04-23T10:31:21.447260+0000", 2024-04-23T10:32:04.236 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "acquired locks" 2024-04-23T10:32:04.236 INFO:teuthology.orchestra.run.smithi114.stdout: } 2024-04-23T10:32:04.236 INFO:teuthology.orchestra.run.smithi114.stdout: ], 2024-04-23T10:32:04.236 INFO:teuthology.orchestra.run.smithi114.stdout: "locks": [ 2024-04-23T10:32:04.236 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-04-23T10:32:04.236 INFO:teuthology.orchestra.run.smithi114.stdout: "object": { 2024-04-23T10:32:04.236 INFO:teuthology.orchestra.run.smithi114.stdout: "is_auth": false, 2024-04-23T10:32:04.236 INFO:teuthology.orchestra.run.smithi114.stdout: "auth_state": { 2024-04-23T10:32:04.236 INFO:teuthology.orchestra.run.smithi114.stdout: "replicas": {} 2024-04-23T10:32:04.236 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-23T10:32:04.237 INFO:teuthology.orchestra.run.smithi114.stdout: "replica_state": { 2024-04-23T10:32:04.237 INFO:teuthology.orchestra.run.smithi114.stdout: "authority": [ 2024-04-23T10:32:04.237 INFO:teuthology.orchestra.run.smithi114.stdout: 0, 2024-04-23T10:32:04.237 INFO:teuthology.orchestra.run.smithi114.stdout: -2 2024-04-23T10:32:04.237 INFO:teuthology.orchestra.run.smithi114.stdout: ], 2024-04-23T10:32:04.237 INFO:teuthology.orchestra.run.smithi114.stdout: "replica_nonce": 1 2024-04-23T10:32:04.237 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-23T10:32:04.237 INFO:teuthology.orchestra.run.smithi114.stdout: "auth_pins": 0, 2024-04-23T10:32:04.237 INFO:teuthology.orchestra.run.smithi114.stdout: "is_frozen": false, 2024-04-23T10:32:04.237 INFO:teuthology.orchestra.run.smithi114.stdout: "is_freezing": false, 2024-04-23T10:32:04.237 INFO:teuthology.orchestra.run.smithi114.stdout: "pins": { 2024-04-23T10:32:04.237 INFO:teuthology.orchestra.run.smithi114.stdout: "request": 1, 2024-04-23T10:32:04.237 INFO:teuthology.orchestra.run.smithi114.stdout: "lock": 1, 2024-04-23T10:32:04.237 INFO:teuthology.orchestra.run.smithi114.stdout: "dirfrag": 1, 2024-04-23T10:32:04.237 INFO:teuthology.orchestra.run.smithi114.stdout: "caps": 0 2024-04-23T10:32:04.238 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-23T10:32:04.238 INFO:teuthology.orchestra.run.smithi114.stdout: "nref": 3 2024-04-23T10:32:04.238 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-23T10:32:04.238 INFO:teuthology.orchestra.run.smithi114.stdout: "object_string": "[inode 0x10000000000 [...7,head] /volumes/ rep@0.1 v404 f(v0 m2024-04-23T10:24:49.736808+0000 3=2+1) n(v29 rc2024-04-23T10:31:17.764716+0000 b21075308 87=67+20)/n(v0 rc2024-04-23T10:24:48.685915+0000 2=0+2) (inest mix) (iquiesce lock x=1 by request(mds.1:103 nref=4)) | request=1 lock=1 dirfrag=1 caps=0 0x55e718955b80]", 2024-04-23T10:32:04.238 INFO:teuthology.orchestra.run.smithi114.stdout: "lock": { 2024-04-23T10:32:04.238 INFO:teuthology.orchestra.run.smithi114.stdout: "gather_set": [], 2024-04-23T10:32:04.238 INFO:teuthology.orchestra.run.smithi114.stdout: "state": "lock", 2024-04-23T10:32:04.238 INFO:teuthology.orchestra.run.smithi114.stdout: "type": "iquiesce", 2024-04-23T10:32:04.238 INFO:teuthology.orchestra.run.smithi114.stdout: "is_leased": false, 2024-04-23T10:32:04.238 INFO:teuthology.orchestra.run.smithi114.stdout: "num_rdlocks": 0, 2024-04-23T10:32:04.238 INFO:teuthology.orchestra.run.smithi114.stdout: "num_wrlocks": 0, 2024-04-23T10:32:04.238 INFO:teuthology.orchestra.run.smithi114.stdout: "num_xlocks": 1, 2024-04-23T10:32:04.238 INFO:teuthology.orchestra.run.smithi114.stdout: "xlock_by": { 2024-04-23T10:32:04.238 INFO:teuthology.orchestra.run.smithi114.stdout: "reqid": { 2024-04-23T10:32:04.238 INFO:teuthology.orchestra.run.smithi114.stdout: "entity": { 2024-04-23T10:32:04.239 INFO:teuthology.orchestra.run.smithi114.stdout: "type": "mds", 2024-04-23T10:32:04.239 INFO:teuthology.orchestra.run.smithi114.stdout: "num": 1 2024-04-23T10:32:04.239 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-23T10:32:04.239 INFO:teuthology.orchestra.run.smithi114.stdout: "tid": 103 2024-04-23T10:32:04.239 INFO:teuthology.orchestra.run.smithi114.stdout: } 2024-04-23T10:32:04.239 INFO:teuthology.orchestra.run.smithi114.stdout: } 2024-04-23T10:32:04.239 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-23T10:32:04.239 INFO:teuthology.orchestra.run.smithi114.stdout: "flags": 4, 2024-04-23T10:32:04.239 INFO:teuthology.orchestra.run.smithi114.stdout: "wrlock_target": -1 2024-04-23T10:32:04.239 INFO:teuthology.orchestra.run.smithi114.stdout: } 2024-04-23T10:32:04.239 INFO:teuthology.orchestra.run.smithi114.stdout: ] 2024-04-23T10:32:04.239 INFO:teuthology.orchestra.run.smithi114.stdout: } 2024-04-23T10:32:04.239 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-23T10:32:04.239 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-04-23T10:32:04.239 INFO:teuthology.orchestra.run.smithi114.stdout: "description": "internal op quiesce_inode:mds.1:135 fp=#0x1000000022a", 2024-04-23T10:32:04.239 INFO:teuthology.orchestra.run.smithi114.stdout: "initiated_at": "2024-04-23T10:31:21.494283+0000", 2024-04-23T10:32:04.239 INFO:teuthology.orchestra.run.smithi114.stdout: "age": 42.363355947999999, 2024-04-23T10:32:04.240 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 42.363674441999997, 2024-04-23T10:32:04.240 INFO:teuthology.orchestra.run.smithi114.stdout: "continuous": true, 2024-04-23T10:32:04.240 INFO:teuthology.orchestra.run.smithi114.stdout: "type_data": { 2024-04-23T10:32:04.240 INFO:teuthology.orchestra.run.smithi114.stdout: "result": -2147483648, 2024-04-23T10:32:04.240 INFO:teuthology.orchestra.run.smithi114.stdout: "flag_point": "failed to authpin, dir is being fragmented", 2024-04-23T10:32:04.240 INFO:teuthology.orchestra.run.smithi114.stdout: "reqid": { 2024-04-23T10:32:04.240 INFO:teuthology.orchestra.run.smithi114.stdout: "entity": { 2024-04-23T10:32:04.240 INFO:teuthology.orchestra.run.smithi114.stdout: "type": "mds", 2024-04-23T10:32:04.240 INFO:teuthology.orchestra.run.smithi114.stdout: "num": 1 2024-04-23T10:32:04.240 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-23T10:32:04.240 INFO:teuthology.orchestra.run.smithi114.stdout: "tid": 135 2024-04-23T10:32:04.240 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-23T10:32:04.240 INFO:teuthology.orchestra.run.smithi114.stdout: "op_type": "internal_op", 2024-04-23T10:32:04.240 INFO:teuthology.orchestra.run.smithi114.stdout: "internal_op": 5385, 2024-04-23T10:32:04.240 INFO:teuthology.orchestra.run.smithi114.stdout: "op_name": "quiesce_inode", 2024-04-23T10:32:04.241 INFO:teuthology.orchestra.run.smithi114.stdout: "events": [ 2024-04-23T10:32:04.241 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-04-23T10:32:04.241 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-04-23T10:31:21.494283+0000", 2024-04-23T10:32:04.241 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "initiated" 2024-04-23T10:32:04.241 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-23T10:32:04.241 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-04-23T10:32:04.241 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-04-23T10:31:21.494283+0000", 2024-04-23T10:32:04.241 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "header_read" 2024-04-23T10:32:04.241 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-23T10:32:04.241 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-04-23T10:32:04.241 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-04-23T10:31:21.494283+0000", 2024-04-23T10:32:04.241 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "throttled" 2024-04-23T10:32:04.241 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-23T10:32:04.241 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-04-23T10:32:04.241 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-04-23T10:31:21.494283+0000", 2024-04-23T10:32:04.241 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "all_read" 2024-04-23T10:32:04.242 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-23T10:32:04.242 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-04-23T10:32:04.242 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-04-23T10:31:21.494283+0000", 2024-04-23T10:32:04.242 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "dispatched" 2024-04-23T10:32:04.242 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-23T10:32:04.242 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-04-23T10:32:04.242 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-04-23T10:31:21.495661+0000", 2024-04-23T10:32:04.242 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "failed to authpin, dir is being fragmented" 2024-04-23T10:32:04.242 INFO:teuthology.orchestra.run.smithi114.stdout: } 2024-04-23T10:32:04.242 INFO:teuthology.orchestra.run.smithi114.stdout: ], 2024-04-23T10:32:04.242 INFO:teuthology.orchestra.run.smithi114.stdout: "locks": [] 2024-04-23T10:32:04.242 INFO:teuthology.orchestra.run.smithi114.stdout: } 2024-04-23T10:32:04.242 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-23T10:32:04.242 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-04-23T10:32:04.242 INFO:teuthology.orchestra.run.smithi114.stdout: "description": "internal op quiesce_inode:mds.1:104 fp=#0x10000000001", 2024-04-23T10:32:04.243 INFO:teuthology.orchestra.run.smithi114.stdout: "initiated_at": "2024-04-23T10:31:21.447285+0000", 2024-04-23T10:32:04.243 INFO:teuthology.orchestra.run.smithi114.stdout: "age": 42.410353696000001, 2024-04-23T10:32:04.243 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 42.410696027, 2024-04-23T10:32:04.243 INFO:teuthology.orchestra.run.smithi114.stdout: "continuous": true, 2024-04-23T10:32:04.243 INFO:teuthology.orchestra.run.smithi114.stdout: "type_data": { 2024-04-23T10:32:04.243 INFO:teuthology.orchestra.run.smithi114.stdout: "result": -2147483648, 2024-04-23T10:32:04.243 INFO:teuthology.orchestra.run.smithi114.stdout: "flag_point": "acquired locks", 2024-04-23T10:32:04.243 INFO:teuthology.orchestra.run.smithi114.stdout: "reqid": { 2024-04-23T10:32:04.243 INFO:teuthology.orchestra.run.smithi114.stdout: "entity": { 2024-04-23T10:32:04.243 INFO:teuthology.orchestra.run.smithi114.stdout: "type": "mds", 2024-04-23T10:32:04.243 INFO:teuthology.orchestra.run.smithi114.stdout: "num": 1 2024-04-23T10:32:04.243 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-23T10:32:04.243 INFO:teuthology.orchestra.run.smithi114.stdout: "tid": 104 2024-04-23T10:32:04.243 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-23T10:32:04.243 INFO:teuthology.orchestra.run.smithi114.stdout: "op_type": "internal_op", 2024-04-23T10:32:04.243 INFO:teuthology.orchestra.run.smithi114.stdout: "internal_op": 5385, 2024-04-23T10:32:04.244 INFO:teuthology.orchestra.run.smithi114.stdout: "op_name": "quiesce_inode", 2024-04-23T10:32:04.244 INFO:teuthology.orchestra.run.smithi114.stdout: "events": [ 2024-04-23T10:32:04.244 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-04-23T10:32:04.244 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-04-23T10:31:21.447285+0000", 2024-04-23T10:32:04.244 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "initiated" 2024-04-23T10:32:04.244 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-23T10:32:04.244 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-04-23T10:32:04.244 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-04-23T10:31:21.447285+0000", 2024-04-23T10:32:04.244 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "header_read" 2024-04-23T10:32:04.244 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-23T10:32:04.244 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-04-23T10:32:04.244 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-04-23T10:31:21.447285+0000", 2024-04-23T10:32:04.244 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "throttled" 2024-04-23T10:32:04.244 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-23T10:32:04.244 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-04-23T10:32:04.244 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-04-23T10:31:21.447285+0000", 2024-04-23T10:32:04.245 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "all_read" 2024-04-23T10:32:04.245 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-23T10:32:04.245 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-04-23T10:32:04.245 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-04-23T10:31:21.447285+0000", 2024-04-23T10:32:04.245 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "dispatched" 2024-04-23T10:32:04.245 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-23T10:32:04.245 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-04-23T10:32:04.245 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-04-23T10:31:21.447364+0000", 2024-04-23T10:32:04.245 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "requesting remote authpins" 2024-04-23T10:32:04.245 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-23T10:32:04.245 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-04-23T10:32:04.245 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-04-23T10:31:21.448241+0000", 2024-04-23T10:32:04.245 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "acquired locks" 2024-04-23T10:32:04.245 INFO:teuthology.orchestra.run.smithi114.stdout: } 2024-04-23T10:32:04.245 INFO:teuthology.orchestra.run.smithi114.stdout: ], 2024-04-23T10:32:04.245 INFO:teuthology.orchestra.run.smithi114.stdout: "locks": [ 2024-04-23T10:32:04.246 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-04-23T10:32:04.246 INFO:teuthology.orchestra.run.smithi114.stdout: "object": { 2024-04-23T10:32:04.246 INFO:teuthology.orchestra.run.smithi114.stdout: "is_auth": false, 2024-04-23T10:32:04.246 INFO:teuthology.orchestra.run.smithi114.stdout: "auth_state": { 2024-04-23T10:32:04.246 INFO:teuthology.orchestra.run.smithi114.stdout: "replicas": {} 2024-04-23T10:32:04.246 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-23T10:32:04.246 INFO:teuthology.orchestra.run.smithi114.stdout: "replica_state": { 2024-04-23T10:32:04.246 INFO:teuthology.orchestra.run.smithi114.stdout: "authority": [ 2024-04-23T10:32:04.246 INFO:teuthology.orchestra.run.smithi114.stdout: 0, 2024-04-23T10:32:04.246 INFO:teuthology.orchestra.run.smithi114.stdout: -2 2024-04-23T10:32:04.246 INFO:teuthology.orchestra.run.smithi114.stdout: ], 2024-04-23T10:32:04.246 INFO:teuthology.orchestra.run.smithi114.stdout: "replica_nonce": 1 2024-04-23T10:32:04.246 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-23T10:32:04.246 INFO:teuthology.orchestra.run.smithi114.stdout: "auth_pins": 0, 2024-04-23T10:32:04.246 INFO:teuthology.orchestra.run.smithi114.stdout: "is_frozen": false, 2024-04-23T10:32:04.247 INFO:teuthology.orchestra.run.smithi114.stdout: "is_freezing": false, 2024-04-23T10:32:04.247 INFO:teuthology.orchestra.run.smithi114.stdout: "pins": { 2024-04-23T10:32:04.247 INFO:teuthology.orchestra.run.smithi114.stdout: "request": 1, 2024-04-23T10:32:04.247 INFO:teuthology.orchestra.run.smithi114.stdout: "lock": 1, 2024-04-23T10:32:04.247 INFO:teuthology.orchestra.run.smithi114.stdout: "dirfrag": 1, 2024-04-23T10:32:04.247 INFO:teuthology.orchestra.run.smithi114.stdout: "discoverbase": 0 2024-04-23T10:32:04.247 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-23T10:32:04.247 INFO:teuthology.orchestra.run.smithi114.stdout: "nref": 3 2024-04-23T10:32:04.247 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-23T10:32:04.247 INFO:teuthology.orchestra.run.smithi114.stdout: "object_string": "[inode 0x10000000001 [...7,head] /volumes/qa/ rep@0.1 v416 f(v0 m2024-04-23T10:24:49.714717+0000 2=0+2) n(v30 rc2024-04-23T10:31:17.764716+0000 b21075308 84=65+19)/n(v0 rc2024-04-23T10:24:49.023727+0000 3=0+3) (inest mix) (iquiesce lock x=1 by request(mds.1:104 nref=4)) | request=1 lock=1 dirfrag=1 discoverbase=0 0x55e718956100]", 2024-04-23T10:32:04.247 INFO:teuthology.orchestra.run.smithi114.stdout: "lock": { 2024-04-23T10:32:04.247 INFO:teuthology.orchestra.run.smithi114.stdout: "gather_set": [], 2024-04-23T10:32:04.247 INFO:teuthology.orchestra.run.smithi114.stdout: "state": "lock", 2024-04-23T10:32:04.247 INFO:teuthology.orchestra.run.smithi114.stdout: "type": "iquiesce", 2024-04-23T10:32:04.247 INFO:teuthology.orchestra.run.smithi114.stdout: "is_leased": false, 2024-04-23T10:32:04.248 INFO:teuthology.orchestra.run.smithi114.stdout: "num_rdlocks": 0, 2024-04-23T10:32:04.248 INFO:teuthology.orchestra.run.smithi114.stdout: "num_wrlocks": 0, 2024-04-23T10:32:04.248 INFO:teuthology.orchestra.run.smithi114.stdout: "num_xlocks": 1, 2024-04-23T10:32:04.248 INFO:teuthology.orchestra.run.smithi114.stdout: "xlock_by": { 2024-04-23T10:32:04.248 INFO:teuthology.orchestra.run.smithi114.stdout: "reqid": { 2024-04-23T10:32:04.248 INFO:teuthology.orchestra.run.smithi114.stdout: "entity": { 2024-04-23T10:32:04.248 INFO:teuthology.orchestra.run.smithi114.stdout: "type": "mds", 2024-04-23T10:32:04.248 INFO:teuthology.orchestra.run.smithi114.stdout: "num": 1 2024-04-23T10:32:04.248 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-23T10:32:04.248 INFO:teuthology.orchestra.run.smithi114.stdout: "tid": 104 2024-04-23T10:32:04.248 INFO:teuthology.orchestra.run.smithi114.stdout: } 2024-04-23T10:32:04.248 INFO:teuthology.orchestra.run.smithi114.stdout: } 2024-04-23T10:32:04.248 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-23T10:32:04.248 INFO:teuthology.orchestra.run.smithi114.stdout: "flags": 4, 2024-04-23T10:32:04.248 INFO:teuthology.orchestra.run.smithi114.stdout: "wrlock_target": -1 2024-04-23T10:32:04.249 INFO:teuthology.orchestra.run.smithi114.stdout: } 2024-04-23T10:32:04.249 INFO:teuthology.orchestra.run.smithi114.stdout: ] 2024-04-23T10:32:04.249 INFO:teuthology.orchestra.run.smithi114.stdout: } 2024-04-23T10:32:04.249 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-23T10:32:04.249 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-04-23T10:32:04.249 INFO:teuthology.orchestra.run.smithi114.stdout: "description": "internal op quiesce_inode:mds.1:136 fp=#0x1000000022d", 2024-04-23T10:32:04.249 INFO:teuthology.orchestra.run.smithi114.stdout: "initiated_at": "2024-04-23T10:31:21.494296+0000", 2024-04-23T10:32:04.249 INFO:teuthology.orchestra.run.smithi114.stdout: "age": 42.363342252999999, 2024-04-23T10:32:04.249 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 42.363740145000001, 2024-04-23T10:32:04.249 INFO:teuthology.orchestra.run.smithi114.stdout: "continuous": true, 2024-04-23T10:32:04.249 INFO:teuthology.orchestra.run.smithi114.stdout: "type_data": { 2024-04-23T10:32:04.249 INFO:teuthology.orchestra.run.smithi114.stdout: "result": -2147483648, 2024-04-23T10:32:04.249 INFO:teuthology.orchestra.run.smithi114.stdout: "flag_point": "failed to authpin, dir is being fragmented", 2024-04-23T10:32:04.249 INFO:teuthology.orchestra.run.smithi114.stdout: "reqid": { 2024-04-23T10:32:04.249 INFO:teuthology.orchestra.run.smithi114.stdout: "entity": { 2024-04-23T10:32:04.250 INFO:teuthology.orchestra.run.smithi114.stdout: "type": "mds", 2024-04-23T10:32:04.250 INFO:teuthology.orchestra.run.smithi114.stdout: "num": 1 2024-04-23T10:32:04.250 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-23T10:32:04.250 INFO:teuthology.orchestra.run.smithi114.stdout: "tid": 136 2024-04-23T10:32:04.250 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-23T10:32:04.250 INFO:teuthology.orchestra.run.smithi114.stdout: "op_type": "internal_op", 2024-04-23T10:32:04.250 INFO:teuthology.orchestra.run.smithi114.stdout: "internal_op": 5385, 2024-04-23T10:32:04.250 INFO:teuthology.orchestra.run.smithi114.stdout: "op_name": "quiesce_inode", 2024-04-23T10:32:04.250 INFO:teuthology.orchestra.run.smithi114.stdout: "events": [ 2024-04-23T10:32:04.250 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-04-23T10:32:04.250 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-04-23T10:31:21.494296+0000", 2024-04-23T10:32:04.250 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "initiated" 2024-04-23T10:32:04.250 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-23T10:32:04.250 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-04-23T10:32:04.251 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-04-23T10:31:21.494296+0000", 2024-04-23T10:32:04.251 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "header_read" 2024-04-23T10:32:04.251 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-23T10:32:04.251 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-04-23T10:32:04.251 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-04-23T10:31:21.494296+0000", 2024-04-23T10:32:04.251 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "throttled" 2024-04-23T10:32:04.251 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-23T10:32:04.251 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-04-23T10:32:04.251 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-04-23T10:31:21.494296+0000", 2024-04-23T10:32:04.251 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "all_read" 2024-04-23T10:32:04.251 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-23T10:32:04.251 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-04-23T10:32:04.251 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-04-23T10:31:21.494296+0000", 2024-04-23T10:32:04.251 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "dispatched" 2024-04-23T10:32:04.252 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-23T10:32:04.252 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-04-23T10:32:04.252 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-04-23T10:31:21.495757+0000", 2024-04-23T10:32:04.252 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "failed to authpin, dir is being fragmented" 2024-04-23T10:32:04.252 INFO:teuthology.orchestra.run.smithi114.stdout: } 2024-04-23T10:32:04.252 INFO:teuthology.orchestra.run.smithi114.stdout: ], 2024-04-23T10:32:04.252 INFO:teuthology.orchestra.run.smithi114.stdout: "locks": [] 2024-04-23T10:32:04.252 INFO:teuthology.orchestra.run.smithi114.stdout: } 2024-04-23T10:32:04.252 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-23T10:32:04.252 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-04-23T10:32:04.252 INFO:teuthology.orchestra.run.smithi114.stdout: "description": "internal op quiesce_inode:mds.1:105 fp=#0x100000001fa", 2024-04-23T10:32:04.252 INFO:teuthology.orchestra.run.smithi114.stdout: "initiated_at": "2024-04-23T10:31:21.448256+0000", 2024-04-23T10:32:04.252 INFO:teuthology.orchestra.run.smithi114.stdout: "age": 42.409382190999999, 2024-04-23T10:32:04.253 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 42.409805163999998, 2024-04-23T10:32:04.253 INFO:teuthology.orchestra.run.smithi114.stdout: "continuous": true, 2024-04-23T10:32:04.253 INFO:teuthology.orchestra.run.smithi114.stdout: "type_data": { 2024-04-23T10:32:04.253 INFO:teuthology.orchestra.run.smithi114.stdout: "result": -2147483648, 2024-04-23T10:32:04.253 INFO:teuthology.orchestra.run.smithi114.stdout: "flag_point": "acquired locks", 2024-04-23T10:32:04.253 INFO:teuthology.orchestra.run.smithi114.stdout: "reqid": { 2024-04-23T10:32:04.253 INFO:teuthology.orchestra.run.smithi114.stdout: "entity": { 2024-04-23T10:32:04.253 INFO:teuthology.orchestra.run.smithi114.stdout: "type": "mds", 2024-04-23T10:32:04.253 INFO:teuthology.orchestra.run.smithi114.stdout: "num": 1 2024-04-23T10:32:04.253 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-23T10:32:04.253 INFO:teuthology.orchestra.run.smithi114.stdout: "tid": 105 2024-04-23T10:32:04.253 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-23T10:32:04.253 INFO:teuthology.orchestra.run.smithi114.stdout: "op_type": "internal_op", 2024-04-23T10:32:04.253 INFO:teuthology.orchestra.run.smithi114.stdout: "internal_op": 5385, 2024-04-23T10:32:04.253 INFO:teuthology.orchestra.run.smithi114.stdout: "op_name": "quiesce_inode", 2024-04-23T10:32:04.254 INFO:teuthology.orchestra.run.smithi114.stdout: "events": [ 2024-04-23T10:32:04.254 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-04-23T10:32:04.254 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-04-23T10:31:21.448256+0000", 2024-04-23T10:32:04.254 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "initiated" 2024-04-23T10:32:04.254 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-23T10:32:04.254 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-04-23T10:32:04.254 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-04-23T10:31:21.448256+0000", 2024-04-23T10:32:04.254 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "header_read" 2024-04-23T10:32:04.254 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-23T10:32:04.254 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-04-23T10:32:04.254 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-04-23T10:31:21.448256+0000", 2024-04-23T10:32:04.254 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "throttled" 2024-04-23T10:32:04.254 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-23T10:32:04.255 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-04-23T10:32:04.255 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-04-23T10:31:21.448256+0000", 2024-04-23T10:32:04.255 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "all_read" 2024-04-23T10:32:04.255 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-23T10:32:04.255 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-04-23T10:32:04.255 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-04-23T10:31:21.448256+0000", 2024-04-23T10:32:04.255 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "dispatched" 2024-04-23T10:32:04.255 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-23T10:32:04.255 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-04-23T10:32:04.255 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-04-23T10:31:21.448346+0000", 2024-04-23T10:32:04.255 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "requesting remote authpins" 2024-04-23T10:32:04.255 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-23T10:32:04.255 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-04-23T10:32:04.255 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-04-23T10:31:21.449369+0000", 2024-04-23T10:32:04.255 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "acquired locks" 2024-04-23T10:32:04.256 INFO:teuthology.orchestra.run.smithi114.stdout: } 2024-04-23T10:32:04.256 INFO:teuthology.orchestra.run.smithi114.stdout: ], 2024-04-23T10:32:04.256 INFO:teuthology.orchestra.run.smithi114.stdout: "locks": [ 2024-04-23T10:32:04.256 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-04-23T10:32:04.256 INFO:teuthology.orchestra.run.smithi114.stdout: "object": { 2024-04-23T10:32:04.256 INFO:teuthology.orchestra.run.smithi114.stdout: "is_auth": false, 2024-04-23T10:32:04.256 INFO:teuthology.orchestra.run.smithi114.stdout: "auth_state": { 2024-04-23T10:32:04.256 INFO:teuthology.orchestra.run.smithi114.stdout: "replicas": {} 2024-04-23T10:32:04.256 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-23T10:32:04.256 INFO:teuthology.orchestra.run.smithi114.stdout: "replica_state": { 2024-04-23T10:32:04.256 INFO:teuthology.orchestra.run.smithi114.stdout: "authority": [ 2024-04-23T10:32:04.256 INFO:teuthology.orchestra.run.smithi114.stdout: 0, 2024-04-23T10:32:04.256 INFO:teuthology.orchestra.run.smithi114.stdout: -2 2024-04-23T10:32:04.256 INFO:teuthology.orchestra.run.smithi114.stdout: ], 2024-04-23T10:32:04.256 INFO:teuthology.orchestra.run.smithi114.stdout: "replica_nonce": 1 2024-04-23T10:32:04.257 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-23T10:32:04.257 INFO:teuthology.orchestra.run.smithi114.stdout: "auth_pins": 0, 2024-04-23T10:32:04.257 INFO:teuthology.orchestra.run.smithi114.stdout: "is_frozen": false, 2024-04-23T10:32:04.257 INFO:teuthology.orchestra.run.smithi114.stdout: "is_freezing": false, 2024-04-23T10:32:04.257 INFO:teuthology.orchestra.run.smithi114.stdout: "pins": { 2024-04-23T10:32:04.257 INFO:teuthology.orchestra.run.smithi114.stdout: "request": 1, 2024-04-23T10:32:04.257 INFO:teuthology.orchestra.run.smithi114.stdout: "lock": 1, 2024-04-23T10:32:04.257 INFO:teuthology.orchestra.run.smithi114.stdout: "dirfrag": 1, 2024-04-23T10:32:04.257 INFO:teuthology.orchestra.run.smithi114.stdout: "dirwaiter": 0, 2024-04-23T10:32:04.257 INFO:teuthology.orchestra.run.smithi114.stdout: "discoverbase": 0 2024-04-23T10:32:04.257 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-23T10:32:04.257 INFO:teuthology.orchestra.run.smithi114.stdout: "nref": 3 2024-04-23T10:32:04.257 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-23T10:32:04.257 INFO:teuthology.orchestra.run.smithi114.stdout: "object_string": "[inode 0x100000001fa [...7,head] /volumes/qa/sv_1/ rep@0.1 v434 snaprealm=0x55e71896c6c0 f(v0 m2024-04-23T10:24:49.735055+0000 2=1+1) n(v29 rc2024-04-23T10:31:17.764716+0000 b21075191 80=64+16)/n(v0 rc2024-04-23T10:24:49.717706+0000 2=0+2) (inest mix) (iquiesce lock x=1 by request(mds.1:105 nref=4)) | request=1 lock=1 dirfrag=1 dirwaiter=0 discoverbase=0 0x55e7186f0000]", 2024-04-23T10:32:04.257 INFO:teuthology.orchestra.run.smithi114.stdout: "lock": { 2024-04-23T10:32:04.258 INFO:teuthology.orchestra.run.smithi114.stdout: "gather_set": [], 2024-04-23T10:32:04.258 INFO:teuthology.orchestra.run.smithi114.stdout: "state": "lock", 2024-04-23T10:32:04.258 INFO:teuthology.orchestra.run.smithi114.stdout: "type": "iquiesce", 2024-04-23T10:32:04.258 INFO:teuthology.orchestra.run.smithi114.stdout: "is_leased": false, 2024-04-23T10:32:04.258 INFO:teuthology.orchestra.run.smithi114.stdout: "num_rdlocks": 0, 2024-04-23T10:32:04.258 INFO:teuthology.orchestra.run.smithi114.stdout: "num_wrlocks": 0, 2024-04-23T10:32:04.258 INFO:teuthology.orchestra.run.smithi114.stdout: "num_xlocks": 1, 2024-04-23T10:32:04.258 INFO:teuthology.orchestra.run.smithi114.stdout: "xlock_by": { 2024-04-23T10:32:04.258 INFO:teuthology.orchestra.run.smithi114.stdout: "reqid": { 2024-04-23T10:32:04.258 INFO:teuthology.orchestra.run.smithi114.stdout: "entity": { 2024-04-23T10:32:04.258 INFO:teuthology.orchestra.run.smithi114.stdout: "type": "mds", 2024-04-23T10:32:04.258 INFO:teuthology.orchestra.run.smithi114.stdout: "num": 1 2024-04-23T10:32:04.258 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-23T10:32:04.258 INFO:teuthology.orchestra.run.smithi114.stdout: "tid": 105 2024-04-23T10:32:04.258 INFO:teuthology.orchestra.run.smithi114.stdout: } 2024-04-23T10:32:04.259 INFO:teuthology.orchestra.run.smithi114.stdout: } 2024-04-23T10:32:04.259 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-23T10:32:04.259 INFO:teuthology.orchestra.run.smithi114.stdout: "flags": 4, 2024-04-23T10:32:04.259 INFO:teuthology.orchestra.run.smithi114.stdout: "wrlock_target": -1 2024-04-23T10:32:04.259 INFO:teuthology.orchestra.run.smithi114.stdout: } 2024-04-23T10:32:04.259 INFO:teuthology.orchestra.run.smithi114.stdout: ] 2024-04-23T10:32:04.259 INFO:teuthology.orchestra.run.smithi114.stdout: } 2024-04-23T10:32:04.259 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-23T10:32:04.259 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-04-23T10:32:04.259 INFO:teuthology.orchestra.run.smithi114.stdout: "description": "internal op quiesce_inode:mds.1:137 fp=#0x10000000238", 2024-04-23T10:32:04.259 INFO:teuthology.orchestra.run.smithi114.stdout: "initiated_at": "2024-04-23T10:31:21.494310+0000", 2024-04-23T10:32:04.259 INFO:teuthology.orchestra.run.smithi114.stdout: "age": 42.363328478, 2024-04-23T10:32:04.259 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 42.363808689999999, 2024-04-23T10:32:04.259 INFO:teuthology.orchestra.run.smithi114.stdout: "continuous": true, 2024-04-23T10:32:04.260 INFO:teuthology.orchestra.run.smithi114.stdout: "type_data": { 2024-04-23T10:32:04.260 INFO:teuthology.orchestra.run.smithi114.stdout: "result": -2147483648, 2024-04-23T10:32:04.260 INFO:teuthology.orchestra.run.smithi114.stdout: "flag_point": "failed to authpin, dir is being fragmented", 2024-04-23T10:32:04.260 INFO:teuthology.orchestra.run.smithi114.stdout: "reqid": { 2024-04-23T10:32:04.260 INFO:teuthology.orchestra.run.smithi114.stdout: "entity": { 2024-04-23T10:32:04.260 INFO:teuthology.orchestra.run.smithi114.stdout: "type": "mds", 2024-04-23T10:32:04.260 INFO:teuthology.orchestra.run.smithi114.stdout: "num": 1 2024-04-23T10:32:04.260 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-23T10:32:04.260 INFO:teuthology.orchestra.run.smithi114.stdout: "tid": 137 2024-04-23T10:32:04.260 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-23T10:32:04.260 INFO:teuthology.orchestra.run.smithi114.stdout: "op_type": "internal_op", 2024-04-23T10:32:04.260 INFO:teuthology.orchestra.run.smithi114.stdout: "internal_op": 5385, 2024-04-23T10:32:04.260 INFO:teuthology.orchestra.run.smithi114.stdout: "op_name": "quiesce_inode", 2024-04-23T10:32:04.260 INFO:teuthology.orchestra.run.smithi114.stdout: "events": [ 2024-04-23T10:32:04.261 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-04-23T10:32:04.261 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-04-23T10:31:21.494310+0000", 2024-04-23T10:32:04.261 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "initiated" 2024-04-23T10:32:04.261 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-23T10:32:04.261 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-04-23T10:32:04.261 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-04-23T10:31:21.494310+0000", 2024-04-23T10:32:04.261 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "header_read" 2024-04-23T10:32:04.261 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-23T10:32:04.261 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-04-23T10:32:04.261 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-04-23T10:31:21.494310+0000", 2024-04-23T10:32:04.261 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "throttled" 2024-04-23T10:32:04.261 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-23T10:32:04.261 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-04-23T10:32:04.261 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-04-23T10:31:21.494310+0000", 2024-04-23T10:32:04.261 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "all_read" 2024-04-23T10:32:04.261 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-23T10:32:04.262 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-04-23T10:32:04.262 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-04-23T10:31:21.494310+0000", 2024-04-23T10:32:04.262 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "dispatched" 2024-04-23T10:32:04.262 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-23T10:32:04.262 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-04-23T10:32:04.262 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-04-23T10:31:21.495867+0000", 2024-04-23T10:32:04.262 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "failed to authpin, dir is being fragmented" 2024-04-23T10:32:04.262 INFO:teuthology.orchestra.run.smithi114.stdout: } 2024-04-23T10:32:04.262 INFO:teuthology.orchestra.run.smithi114.stdout: ], 2024-04-23T10:32:04.262 INFO:teuthology.orchestra.run.smithi114.stdout: "locks": [] 2024-04-23T10:32:04.262 INFO:teuthology.orchestra.run.smithi114.stdout: } 2024-04-23T10:32:04.262 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-23T10:32:04.262 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-04-23T10:32:04.263 INFO:teuthology.orchestra.run.smithi114.stdout: "description": "internal op quiesce_inode:mds.1:106 fp=#0x10000000002", 2024-04-23T10:32:04.263 INFO:teuthology.orchestra.run.smithi114.stdout: "initiated_at": "2024-04-23T10:31:21.448273+0000", 2024-04-23T10:32:04.263 INFO:teuthology.orchestra.run.smithi114.stdout: "age": 42.409365283, 2024-04-23T10:32:04.263 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 42.409890222999998, 2024-04-23T10:32:04.263 INFO:teuthology.orchestra.run.smithi114.stdout: "continuous": true, 2024-04-23T10:32:04.263 INFO:teuthology.orchestra.run.smithi114.stdout: "type_data": { 2024-04-23T10:32:04.263 INFO:teuthology.orchestra.run.smithi114.stdout: "result": -2147483648, 2024-04-23T10:32:04.263 INFO:teuthology.orchestra.run.smithi114.stdout: "flag_point": "quiesce complete for non-auth inode", 2024-04-23T10:32:04.263 INFO:teuthology.orchestra.run.smithi114.stdout: "reqid": { 2024-04-23T10:32:04.263 INFO:teuthology.orchestra.run.smithi114.stdout: "entity": { 2024-04-23T10:32:04.263 INFO:teuthology.orchestra.run.smithi114.stdout: "type": "mds", 2024-04-23T10:32:04.263 INFO:teuthology.orchestra.run.smithi114.stdout: "num": 1 2024-04-23T10:32:04.263 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-23T10:32:04.263 INFO:teuthology.orchestra.run.smithi114.stdout: "tid": 106 2024-04-23T10:32:04.263 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-23T10:32:04.263 INFO:teuthology.orchestra.run.smithi114.stdout: "op_type": "internal_op", 2024-04-23T10:32:04.264 INFO:teuthology.orchestra.run.smithi114.stdout: "internal_op": 5385, 2024-04-23T10:32:04.264 INFO:teuthology.orchestra.run.smithi114.stdout: "op_name": "quiesce_inode", 2024-04-23T10:32:04.264 INFO:teuthology.orchestra.run.smithi114.stdout: "events": [ 2024-04-23T10:32:04.264 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-04-23T10:32:04.264 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-04-23T10:31:21.448273+0000", 2024-04-23T10:32:04.264 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "initiated" 2024-04-23T10:32:04.264 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-23T10:32:04.264 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-04-23T10:32:04.264 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-04-23T10:31:21.448273+0000", 2024-04-23T10:32:04.264 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "header_read" 2024-04-23T10:32:04.264 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-23T10:32:04.264 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-04-23T10:32:04.264 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-04-23T10:31:21.448273+0000", 2024-04-23T10:32:04.264 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "throttled" 2024-04-23T10:32:04.264 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-23T10:32:04.264 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-04-23T10:32:04.265 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-04-23T10:31:21.448273+0000", 2024-04-23T10:32:04.265 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "all_read" 2024-04-23T10:32:04.265 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-23T10:32:04.265 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-04-23T10:32:04.265 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-04-23T10:31:21.448273+0000", 2024-04-23T10:32:04.265 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "dispatched" 2024-04-23T10:32:04.265 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-23T10:32:04.265 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-04-23T10:32:04.265 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-04-23T10:31:21.448407+0000", 2024-04-23T10:32:04.265 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "requesting remote authpins" 2024-04-23T10:32:04.265 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-23T10:32:04.265 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-04-23T10:32:04.265 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-04-23T10:31:21.449578+0000", 2024-04-23T10:32:04.265 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "acquired locks" 2024-04-23T10:32:04.265 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-23T10:32:04.265 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-04-23T10:32:04.265 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-04-23T10:31:21.449586+0000", 2024-04-23T10:32:04.266 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "quiesce complete for non-auth inode" 2024-04-23T10:32:04.266 INFO:teuthology.orchestra.run.smithi114.stdout: } 2024-04-23T10:32:04.266 INFO:teuthology.orchestra.run.smithi114.stdout: ], 2024-04-23T10:32:04.266 INFO:teuthology.orchestra.run.smithi114.stdout: "locks": [ 2024-04-23T10:32:04.266 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-04-23T10:32:04.266 INFO:teuthology.orchestra.run.smithi114.stdout: "object": { 2024-04-23T10:32:04.266 INFO:teuthology.orchestra.run.smithi114.stdout: "is_auth": false, 2024-04-23T10:32:04.266 INFO:teuthology.orchestra.run.smithi114.stdout: "auth_state": { 2024-04-23T10:32:04.266 INFO:teuthology.orchestra.run.smithi114.stdout: "replicas": {} 2024-04-23T10:32:04.266 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-23T10:32:04.266 INFO:teuthology.orchestra.run.smithi114.stdout: "replica_state": { 2024-04-23T10:32:04.266 INFO:teuthology.orchestra.run.smithi114.stdout: "authority": [ 2024-04-23T10:32:04.266 INFO:teuthology.orchestra.run.smithi114.stdout: 0, 2024-04-23T10:32:04.266 INFO:teuthology.orchestra.run.smithi114.stdout: -2 2024-04-23T10:32:04.266 INFO:teuthology.orchestra.run.smithi114.stdout: ], 2024-04-23T10:32:04.267 INFO:teuthology.orchestra.run.smithi114.stdout: "replica_nonce": 1 2024-04-23T10:32:04.267 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-23T10:32:04.267 INFO:teuthology.orchestra.run.smithi114.stdout: "auth_pins": 0, 2024-04-23T10:32:04.267 INFO:teuthology.orchestra.run.smithi114.stdout: "is_frozen": false, 2024-04-23T10:32:04.267 INFO:teuthology.orchestra.run.smithi114.stdout: "is_freezing": false, 2024-04-23T10:32:04.267 INFO:teuthology.orchestra.run.smithi114.stdout: "pins": { 2024-04-23T10:32:04.267 INFO:teuthology.orchestra.run.smithi114.stdout: "request": 1, 2024-04-23T10:32:04.267 INFO:teuthology.orchestra.run.smithi114.stdout: "lock": 1, 2024-04-23T10:32:04.267 INFO:teuthology.orchestra.run.smithi114.stdout: "dirwaiter": 0, 2024-04-23T10:32:04.267 INFO:teuthology.orchestra.run.smithi114.stdout: "discoverbase": 0 2024-04-23T10:32:04.267 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-23T10:32:04.267 INFO:teuthology.orchestra.run.smithi114.stdout: "nref": 2 2024-04-23T10:32:04.267 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-23T10:32:04.267 INFO:teuthology.orchestra.run.smithi114.stdout: "object_string": "[inode 0x10000000002 [...2,head] /volumes/qa/sv_0/ rep@0.1 v34 snaprealm=0x55e71896c240 f(v0 m2024-04-23T10:24:49.040186+0000 2=1+1) n(v0 rc2024-04-23T10:24:49.031387+0000 2=0+2)/n(v0 rc2024-04-23T10:24:49.023727+0000 2=0+2) (inest mix) (iquiesce lock x=1 by request(mds.1:106 nref=3)) | request=1 lock=1 dirwaiter=0 discoverbase=0 0x55e718957700]", 2024-04-23T10:32:04.267 INFO:teuthology.orchestra.run.smithi114.stdout: "lock": { 2024-04-23T10:32:04.267 INFO:teuthology.orchestra.run.smithi114.stdout: "gather_set": [], 2024-04-23T10:32:04.268 INFO:teuthology.orchestra.run.smithi114.stdout: "state": "lock", 2024-04-23T10:32:04.268 INFO:teuthology.orchestra.run.smithi114.stdout: "type": "iquiesce", 2024-04-23T10:32:04.268 INFO:teuthology.orchestra.run.smithi114.stdout: "is_leased": false, 2024-04-23T10:32:04.268 INFO:teuthology.orchestra.run.smithi114.stdout: "num_rdlocks": 0, 2024-04-23T10:32:04.268 INFO:teuthology.orchestra.run.smithi114.stdout: "num_wrlocks": 0, 2024-04-23T10:32:04.268 INFO:teuthology.orchestra.run.smithi114.stdout: "num_xlocks": 1, 2024-04-23T10:32:04.268 INFO:teuthology.orchestra.run.smithi114.stdout: "xlock_by": { 2024-04-23T10:32:04.268 INFO:teuthology.orchestra.run.smithi114.stdout: "reqid": { 2024-04-23T10:32:04.268 INFO:teuthology.orchestra.run.smithi114.stdout: "entity": { 2024-04-23T10:32:04.268 INFO:teuthology.orchestra.run.smithi114.stdout: "type": "mds", 2024-04-23T10:32:04.268 INFO:teuthology.orchestra.run.smithi114.stdout: "num": 1 2024-04-23T10:32:04.268 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-23T10:32:04.268 INFO:teuthology.orchestra.run.smithi114.stdout: "tid": 106 2024-04-23T10:32:04.268 INFO:teuthology.orchestra.run.smithi114.stdout: } 2024-04-23T10:32:04.268 INFO:teuthology.orchestra.run.smithi114.stdout: } 2024-04-23T10:32:04.268 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-23T10:32:04.269 INFO:teuthology.orchestra.run.smithi114.stdout: "flags": 4, 2024-04-23T10:32:04.269 INFO:teuthology.orchestra.run.smithi114.stdout: "wrlock_target": -1 2024-04-23T10:32:04.269 INFO:teuthology.orchestra.run.smithi114.stdout: } 2024-04-23T10:32:04.269 INFO:teuthology.orchestra.run.smithi114.stdout: ] 2024-04-23T10:32:04.269 INFO:teuthology.orchestra.run.smithi114.stdout: } 2024-04-23T10:32:04.269 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-23T10:32:04.269 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-04-23T10:32:04.269 INFO:teuthology.orchestra.run.smithi114.stdout: "description": "internal op quiesce_inode:mds.1:138 fp=#0x10000000235", 2024-04-23T10:32:04.269 INFO:teuthology.orchestra.run.smithi114.stdout: "initiated_at": "2024-04-23T10:31:21.494330+0000", 2024-04-23T10:32:04.269 INFO:teuthology.orchestra.run.smithi114.stdout: "age": 42.363308988, 2024-04-23T10:32:04.269 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 42.363892266999997, 2024-04-23T10:32:04.269 INFO:teuthology.orchestra.run.smithi114.stdout: "continuous": true, 2024-04-23T10:32:04.269 INFO:teuthology.orchestra.run.smithi114.stdout: "type_data": { 2024-04-23T10:32:04.269 INFO:teuthology.orchestra.run.smithi114.stdout: "result": -2147483648, 2024-04-23T10:32:04.269 INFO:teuthology.orchestra.run.smithi114.stdout: "flag_point": "failed to authpin, dir is being fragmented", 2024-04-23T10:32:04.269 INFO:teuthology.orchestra.run.smithi114.stdout: "reqid": { 2024-04-23T10:32:04.270 INFO:teuthology.orchestra.run.smithi114.stdout: "entity": { 2024-04-23T10:32:04.270 INFO:teuthology.orchestra.run.smithi114.stdout: "type": "mds", 2024-04-23T10:32:04.270 INFO:teuthology.orchestra.run.smithi114.stdout: "num": 1 2024-04-23T10:32:04.270 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-23T10:32:04.270 INFO:teuthology.orchestra.run.smithi114.stdout: "tid": 138 2024-04-23T10:32:04.270 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-23T10:32:04.270 INFO:teuthology.orchestra.run.smithi114.stdout: "op_type": "internal_op", 2024-04-23T10:32:04.270 INFO:teuthology.orchestra.run.smithi114.stdout: "internal_op": 5385, 2024-04-23T10:32:04.270 INFO:teuthology.orchestra.run.smithi114.stdout: "op_name": "quiesce_inode", 2024-04-23T10:32:04.270 INFO:teuthology.orchestra.run.smithi114.stdout: "events": [ 2024-04-23T10:32:04.270 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-04-23T10:32:04.270 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-04-23T10:31:21.494330+0000", 2024-04-23T10:32:04.270 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "initiated" 2024-04-23T10:32:04.270 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-23T10:32:04.270 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-04-23T10:32:04.270 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-04-23T10:31:21.494330+0000", 2024-04-23T10:32:04.270 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "header_read" 2024-04-23T10:32:04.271 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-23T10:32:04.271 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-04-23T10:32:04.271 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-04-23T10:31:21.494330+0000", 2024-04-23T10:32:04.271 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "throttled" 2024-04-23T10:32:04.271 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-23T10:32:04.271 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-04-23T10:32:04.271 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-04-23T10:31:21.494330+0000", 2024-04-23T10:32:04.271 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "all_read" 2024-04-23T10:32:04.271 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-23T10:32:04.271 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-04-23T10:32:04.271 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-04-23T10:31:21.494330+0000", 2024-04-23T10:32:04.271 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "dispatched" 2024-04-23T10:32:04.271 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-23T10:32:04.271 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-04-23T10:32:04.271 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-04-23T10:31:21.495977+0000", 2024-04-23T10:32:04.272 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "failed to authpin, dir is being fragmented" 2024-04-23T10:32:04.272 INFO:teuthology.orchestra.run.smithi114.stdout: } 2024-04-23T10:32:04.272 INFO:teuthology.orchestra.run.smithi114.stdout: ], 2024-04-23T10:32:04.272 INFO:teuthology.orchestra.run.smithi114.stdout: "locks": [] 2024-04-23T10:32:04.272 INFO:teuthology.orchestra.run.smithi114.stdout: } 2024-04-23T10:32:04.272 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-23T10:32:04.272 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-04-23T10:32:04.272 INFO:teuthology.orchestra.run.smithi114.stdout: "description": "internal op quiesce_inode:mds.1:107 fp=#0x100000001fb", 2024-04-23T10:32:04.272 INFO:teuthology.orchestra.run.smithi114.stdout: "initiated_at": "2024-04-23T10:31:21.449385+0000", 2024-04-23T10:32:04.272 INFO:teuthology.orchestra.run.smithi114.stdout: "age": 42.408253705999996, 2024-04-23T10:32:04.272 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 42.408860955000002, 2024-04-23T10:32:04.272 INFO:teuthology.orchestra.run.smithi114.stdout: "continuous": true, 2024-04-23T10:32:04.272 INFO:teuthology.orchestra.run.smithi114.stdout: "type_data": { 2024-04-23T10:32:04.272 INFO:teuthology.orchestra.run.smithi114.stdout: "result": -2147483648, 2024-04-23T10:32:04.272 INFO:teuthology.orchestra.run.smithi114.stdout: "flag_point": "acquired locks", 2024-04-23T10:32:04.273 INFO:teuthology.orchestra.run.smithi114.stdout: "reqid": { 2024-04-23T10:32:04.273 INFO:teuthology.orchestra.run.smithi114.stdout: "entity": { 2024-04-23T10:32:04.273 INFO:teuthology.orchestra.run.smithi114.stdout: "type": "mds", 2024-04-23T10:32:04.273 INFO:teuthology.orchestra.run.smithi114.stdout: "num": 1 2024-04-23T10:32:04.273 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-23T10:32:04.273 INFO:teuthology.orchestra.run.smithi114.stdout: "tid": 107 2024-04-23T10:32:04.273 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-23T10:32:04.273 INFO:teuthology.orchestra.run.smithi114.stdout: "op_type": "internal_op", 2024-04-23T10:32:04.273 INFO:teuthology.orchestra.run.smithi114.stdout: "internal_op": 5385, 2024-04-23T10:32:04.273 INFO:teuthology.orchestra.run.smithi114.stdout: "op_name": "quiesce_inode", 2024-04-23T10:32:04.273 INFO:teuthology.orchestra.run.smithi114.stdout: "events": [ 2024-04-23T10:32:04.273 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-04-23T10:32:04.273 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-04-23T10:31:21.449385+0000", 2024-04-23T10:32:04.273 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "initiated" 2024-04-23T10:32:04.273 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-23T10:32:04.273 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-04-23T10:32:04.274 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-04-23T10:31:21.449385+0000", 2024-04-23T10:32:04.274 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "header_read" 2024-04-23T10:32:04.274 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-23T10:32:04.274 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-04-23T10:32:04.274 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-04-23T10:31:21.449385+0000", 2024-04-23T10:32:04.274 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "throttled" 2024-04-23T10:32:04.274 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-23T10:32:04.274 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-04-23T10:32:04.274 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-04-23T10:31:21.449385+0000", 2024-04-23T10:32:04.274 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "all_read" 2024-04-23T10:32:04.274 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-23T10:32:04.274 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-04-23T10:32:04.274 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-04-23T10:31:21.449385+0000", 2024-04-23T10:32:04.274 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "dispatched" 2024-04-23T10:32:04.274 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-23T10:32:04.274 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-04-23T10:32:04.275 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-04-23T10:31:21.449471+0000", 2024-04-23T10:32:04.275 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "requesting remote authpins" 2024-04-23T10:32:04.275 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-23T10:32:04.275 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-04-23T10:32:04.275 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-04-23T10:31:21.450864+0000", 2024-04-23T10:32:04.275 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "acquired locks" 2024-04-23T10:32:04.275 INFO:teuthology.orchestra.run.smithi114.stdout: } 2024-04-23T10:32:04.275 INFO:teuthology.orchestra.run.smithi114.stdout: ], 2024-04-23T10:32:04.275 INFO:teuthology.orchestra.run.smithi114.stdout: "locks": [ 2024-04-23T10:32:04.275 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-04-23T10:32:04.275 INFO:teuthology.orchestra.run.smithi114.stdout: "object": { 2024-04-23T10:32:04.275 INFO:teuthology.orchestra.run.smithi114.stdout: "is_auth": false, 2024-04-23T10:32:04.275 INFO:teuthology.orchestra.run.smithi114.stdout: "auth_state": { 2024-04-23T10:32:04.275 INFO:teuthology.orchestra.run.smithi114.stdout: "replicas": {} 2024-04-23T10:32:04.275 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-23T10:32:04.276 INFO:teuthology.orchestra.run.smithi114.stdout: "replica_state": { 2024-04-23T10:32:04.276 INFO:teuthology.orchestra.run.smithi114.stdout: "authority": [ 2024-04-23T10:32:04.276 INFO:teuthology.orchestra.run.smithi114.stdout: 0, 2024-04-23T10:32:04.276 INFO:teuthology.orchestra.run.smithi114.stdout: -2 2024-04-23T10:32:04.276 INFO:teuthology.orchestra.run.smithi114.stdout: ], 2024-04-23T10:32:04.276 INFO:teuthology.orchestra.run.smithi114.stdout: "replica_nonce": 1 2024-04-23T10:32:04.276 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-23T10:32:04.276 INFO:teuthology.orchestra.run.smithi114.stdout: "auth_pins": 0, 2024-04-23T10:32:04.276 INFO:teuthology.orchestra.run.smithi114.stdout: "is_frozen": false, 2024-04-23T10:32:04.276 INFO:teuthology.orchestra.run.smithi114.stdout: "is_freezing": false, 2024-04-23T10:32:04.276 INFO:teuthology.orchestra.run.smithi114.stdout: "pins": { 2024-04-23T10:32:04.276 INFO:teuthology.orchestra.run.smithi114.stdout: "request": 1, 2024-04-23T10:32:04.276 INFO:teuthology.orchestra.run.smithi114.stdout: "lock": 1, 2024-04-23T10:32:04.276 INFO:teuthology.orchestra.run.smithi114.stdout: "dirfrag": 1, 2024-04-23T10:32:04.276 INFO:teuthology.orchestra.run.smithi114.stdout: "dirwaiter": 0, 2024-04-23T10:32:04.276 INFO:teuthology.orchestra.run.smithi114.stdout: "discoverbase": 0 2024-04-23T10:32:04.277 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-23T10:32:04.277 INFO:teuthology.orchestra.run.smithi114.stdout: "nref": 3 2024-04-23T10:32:04.277 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-23T10:32:04.277 INFO:teuthology.orchestra.run.smithi114.stdout: "object_string": "[inode 0x100000001fb [...7,head] /volumes/qa/sv_1/b206794e-7622-4ab8-8943-f3ae4e65b7a3/ rep@0.1 v426 f(v0 m2024-04-23T10:26:48.475842+0000 1=0+1) n(v25 rc2024-04-23T10:31:17.764716+0000 b21075074 78=63+15)/n(v0 rc2024-04-23T10:26:48.480842+0000 2=0+2) old_inodes=1 (inest mix) (iquiesce lock x=1 by request(mds.1:107 nref=4)) | request=1 lock=1 dirfrag=1 dirwaiter=0 discoverbase=0 0x55e7186f1080]", 2024-04-23T10:32:04.277 INFO:teuthology.orchestra.run.smithi114.stdout: "lock": { 2024-04-23T10:32:04.277 INFO:teuthology.orchestra.run.smithi114.stdout: "gather_set": [], 2024-04-23T10:32:04.277 INFO:teuthology.orchestra.run.smithi114.stdout: "state": "lock", 2024-04-23T10:32:04.277 INFO:teuthology.orchestra.run.smithi114.stdout: "type": "iquiesce", 2024-04-23T10:32:04.277 INFO:teuthology.orchestra.run.smithi114.stdout: "is_leased": false, 2024-04-23T10:32:04.277 INFO:teuthology.orchestra.run.smithi114.stdout: "num_rdlocks": 0, 2024-04-23T10:32:04.277 INFO:teuthology.orchestra.run.smithi114.stdout: "num_wrlocks": 0, 2024-04-23T10:32:04.277 INFO:teuthology.orchestra.run.smithi114.stdout: "num_xlocks": 1, 2024-04-23T10:32:04.277 INFO:teuthology.orchestra.run.smithi114.stdout: "xlock_by": { 2024-04-23T10:32:04.277 INFO:teuthology.orchestra.run.smithi114.stdout: "reqid": { 2024-04-23T10:32:04.277 INFO:teuthology.orchestra.run.smithi114.stdout: "entity": { 2024-04-23T10:32:04.277 INFO:teuthology.orchestra.run.smithi114.stdout: "type": "mds", 2024-04-23T10:32:04.278 INFO:teuthology.orchestra.run.smithi114.stdout: "num": 1 2024-04-23T10:32:04.278 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-23T10:32:04.278 INFO:teuthology.orchestra.run.smithi114.stdout: "tid": 107 2024-04-23T10:32:04.278 INFO:teuthology.orchestra.run.smithi114.stdout: } 2024-04-23T10:32:04.278 INFO:teuthology.orchestra.run.smithi114.stdout: } 2024-04-23T10:32:04.278 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-23T10:32:04.278 INFO:teuthology.orchestra.run.smithi114.stdout: "flags": 4, 2024-04-23T10:32:04.278 INFO:teuthology.orchestra.run.smithi114.stdout: "wrlock_target": -1 2024-04-23T10:32:04.278 INFO:teuthology.orchestra.run.smithi114.stdout: } 2024-04-23T10:32:04.278 INFO:teuthology.orchestra.run.smithi114.stdout: ] 2024-04-23T10:32:04.278 INFO:teuthology.orchestra.run.smithi114.stdout: } 2024-04-23T10:32:04.278 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-23T10:32:04.278 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-04-23T10:32:04.278 INFO:teuthology.orchestra.run.smithi114.stdout: "description": "internal op quiesce_inode:mds.1:139 fp=#0x1000000023a", 2024-04-23T10:32:04.278 INFO:teuthology.orchestra.run.smithi114.stdout: "initiated_at": "2024-04-23T10:31:21.494345+0000", 2024-04-23T10:32:04.279 INFO:teuthology.orchestra.run.smithi114.stdout: "age": 42.363293798000001, 2024-04-23T10:32:04.279 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 42.363957581999998, 2024-04-23T10:32:04.279 INFO:teuthology.orchestra.run.smithi114.stdout: "continuous": true, 2024-04-23T10:32:04.279 INFO:teuthology.orchestra.run.smithi114.stdout: "type_data": { 2024-04-23T10:32:04.279 INFO:teuthology.orchestra.run.smithi114.stdout: "result": -2147483648, 2024-04-23T10:32:04.279 INFO:teuthology.orchestra.run.smithi114.stdout: "flag_point": "failed to authpin, dir is being fragmented", 2024-04-23T10:32:04.279 INFO:teuthology.orchestra.run.smithi114.stdout: "reqid": { 2024-04-23T10:32:04.279 INFO:teuthology.orchestra.run.smithi114.stdout: "entity": { 2024-04-23T10:32:04.279 INFO:teuthology.orchestra.run.smithi114.stdout: "type": "mds", 2024-04-23T10:32:04.279 INFO:teuthology.orchestra.run.smithi114.stdout: "num": 1 2024-04-23T10:32:04.279 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-23T10:32:04.279 INFO:teuthology.orchestra.run.smithi114.stdout: "tid": 139 2024-04-23T10:32:04.279 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-23T10:32:04.279 INFO:teuthology.orchestra.run.smithi114.stdout: "op_type": "internal_op", 2024-04-23T10:32:04.279 INFO:teuthology.orchestra.run.smithi114.stdout: "internal_op": 5385, 2024-04-23T10:32:04.279 INFO:teuthology.orchestra.run.smithi114.stdout: "op_name": "quiesce_inode", 2024-04-23T10:32:04.280 INFO:teuthology.orchestra.run.smithi114.stdout: "events": [ 2024-04-23T10:32:04.280 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-04-23T10:32:04.280 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-04-23T10:31:21.494345+0000", 2024-04-23T10:32:04.280 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "initiated" 2024-04-23T10:32:04.280 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-23T10:32:04.280 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-04-23T10:32:04.280 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-04-23T10:31:21.494345+0000", 2024-04-23T10:32:04.280 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "header_read" 2024-04-23T10:32:04.280 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-23T10:32:04.280 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-04-23T10:32:04.280 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-04-23T10:31:21.494345+0000", 2024-04-23T10:32:04.280 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "throttled" 2024-04-23T10:32:04.280 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-23T10:32:04.280 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-04-23T10:32:04.281 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-04-23T10:31:21.494345+0000", 2024-04-23T10:32:04.281 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "all_read" 2024-04-23T10:32:04.281 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-23T10:32:04.281 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-04-23T10:32:04.281 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-04-23T10:31:21.494345+0000", 2024-04-23T10:32:04.281 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "dispatched" 2024-04-23T10:32:04.281 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-23T10:32:04.281 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-04-23T10:32:04.281 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-04-23T10:31:21.496076+0000", 2024-04-23T10:32:04.281 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "failed to authpin, dir is being fragmented" 2024-04-23T10:32:04.281 INFO:teuthology.orchestra.run.smithi114.stdout: } 2024-04-23T10:32:04.281 INFO:teuthology.orchestra.run.smithi114.stdout: ], 2024-04-23T10:32:04.281 INFO:teuthology.orchestra.run.smithi114.stdout: "locks": [] 2024-04-23T10:32:04.281 INFO:teuthology.orchestra.run.smithi114.stdout: } 2024-04-23T10:32:04.281 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-23T10:32:04.282 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-04-23T10:32:04.282 INFO:teuthology.orchestra.run.smithi114.stdout: "description": "internal op quiesce_inode:mds.1:108 fp=#0x10000000005", 2024-04-23T10:32:04.282 INFO:teuthology.orchestra.run.smithi114.stdout: "initiated_at": "2024-04-23T10:31:21.450873+0000", 2024-04-23T10:32:04.282 INFO:teuthology.orchestra.run.smithi114.stdout: "age": 42.406765493999998, 2024-04-23T10:32:04.282 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 42.407454528000002, 2024-04-23T10:32:04.282 INFO:teuthology.orchestra.run.smithi114.stdout: "continuous": true, 2024-04-23T10:32:04.282 INFO:teuthology.orchestra.run.smithi114.stdout: "type_data": { 2024-04-23T10:32:04.282 INFO:teuthology.orchestra.run.smithi114.stdout: "result": -2147483648, 2024-04-23T10:32:04.282 INFO:teuthology.orchestra.run.smithi114.stdout: "flag_point": "acquired locks", 2024-04-23T10:32:04.282 INFO:teuthology.orchestra.run.smithi114.stdout: "reqid": { 2024-04-23T10:32:04.282 INFO:teuthology.orchestra.run.smithi114.stdout: "entity": { 2024-04-23T10:32:04.282 INFO:teuthology.orchestra.run.smithi114.stdout: "type": "mds", 2024-04-23T10:32:04.282 INFO:teuthology.orchestra.run.smithi114.stdout: "num": 1 2024-04-23T10:32:04.282 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-23T10:32:04.283 INFO:teuthology.orchestra.run.smithi114.stdout: "tid": 108 2024-04-23T10:32:04.283 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-23T10:32:04.283 INFO:teuthology.orchestra.run.smithi114.stdout: "op_type": "internal_op", 2024-04-23T10:32:04.283 INFO:teuthology.orchestra.run.smithi114.stdout: "internal_op": 5385, 2024-04-23T10:32:04.283 INFO:teuthology.orchestra.run.smithi114.stdout: "op_name": "quiesce_inode", 2024-04-23T10:32:04.283 INFO:teuthology.orchestra.run.smithi114.stdout: "events": [ 2024-04-23T10:32:04.283 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-04-23T10:32:04.283 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-04-23T10:31:21.450873+0000", 2024-04-23T10:32:04.283 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "initiated" 2024-04-23T10:32:04.283 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-23T10:32:04.283 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-04-23T10:32:04.283 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-04-23T10:31:21.450873+0000", 2024-04-23T10:32:04.283 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "header_read" 2024-04-23T10:32:04.283 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-23T10:32:04.283 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-04-23T10:32:04.284 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-04-23T10:31:21.450873+0000", 2024-04-23T10:32:04.284 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "throttled" 2024-04-23T10:32:04.284 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-23T10:32:04.284 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-04-23T10:32:04.284 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-04-23T10:31:21.450873+0000", 2024-04-23T10:32:04.284 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "all_read" 2024-04-23T10:32:04.284 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-23T10:32:04.284 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-04-23T10:32:04.284 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-04-23T10:31:21.450873+0000", 2024-04-23T10:32:04.284 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "dispatched" 2024-04-23T10:32:04.284 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-23T10:32:04.284 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-04-23T10:32:04.284 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-04-23T10:31:21.450934+0000", 2024-04-23T10:32:04.284 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "requesting remote authpins" 2024-04-23T10:32:04.285 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-23T10:32:04.285 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-04-23T10:32:04.285 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-04-23T10:31:21.452061+0000", 2024-04-23T10:32:04.285 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "acquired locks" 2024-04-23T10:32:04.285 INFO:teuthology.orchestra.run.smithi114.stdout: } 2024-04-23T10:32:04.285 INFO:teuthology.orchestra.run.smithi114.stdout: ], 2024-04-23T10:32:04.285 INFO:teuthology.orchestra.run.smithi114.stdout: "locks": [ 2024-04-23T10:32:04.285 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-04-23T10:32:04.285 INFO:teuthology.orchestra.run.smithi114.stdout: "object": { 2024-04-23T10:32:04.285 INFO:teuthology.orchestra.run.smithi114.stdout: "is_auth": false, 2024-04-23T10:32:04.285 INFO:teuthology.orchestra.run.smithi114.stdout: "auth_state": { 2024-04-23T10:32:04.285 INFO:teuthology.orchestra.run.smithi114.stdout: "replicas": {} 2024-04-23T10:32:04.285 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-23T10:32:04.285 INFO:teuthology.orchestra.run.smithi114.stdout: "replica_state": { 2024-04-23T10:32:04.285 INFO:teuthology.orchestra.run.smithi114.stdout: "authority": [ 2024-04-23T10:32:04.285 INFO:teuthology.orchestra.run.smithi114.stdout: 0, 2024-04-23T10:32:04.286 INFO:teuthology.orchestra.run.smithi114.stdout: -2 2024-04-23T10:32:04.286 INFO:teuthology.orchestra.run.smithi114.stdout: ], 2024-04-23T10:32:04.286 INFO:teuthology.orchestra.run.smithi114.stdout: "replica_nonce": 1 2024-04-23T10:32:04.286 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-23T10:32:04.286 INFO:teuthology.orchestra.run.smithi114.stdout: "auth_pins": 0, 2024-04-23T10:32:04.286 INFO:teuthology.orchestra.run.smithi114.stdout: "is_frozen": false, 2024-04-23T10:32:04.286 INFO:teuthology.orchestra.run.smithi114.stdout: "is_freezing": false, 2024-04-23T10:32:04.286 INFO:teuthology.orchestra.run.smithi114.stdout: "pins": { 2024-04-23T10:32:04.286 INFO:teuthology.orchestra.run.smithi114.stdout: "request": 1, 2024-04-23T10:32:04.286 INFO:teuthology.orchestra.run.smithi114.stdout: "lock": 1, 2024-04-23T10:32:04.286 INFO:teuthology.orchestra.run.smithi114.stdout: "dirfrag": 1 2024-04-23T10:32:04.286 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-23T10:32:04.286 INFO:teuthology.orchestra.run.smithi114.stdout: "nref": 3 2024-04-23T10:32:04.287 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-23T10:32:04.287 INFO:teuthology.orchestra.run.smithi114.stdout: "object_string": "[inode 0x10000000005 [...7,head] /volumes/qa/sv_1/b206794e-7622-4ab8-8943-f3ae4e65b7a3/client.0/ rep@0.1 v421 f(v0 m2024-04-23T10:27:42.521612+0000 1=0+1) n(v32 rc2024-04-23T10:31:17.764716+0000 b21075074 77=63+14)/n(v0 rc2024-04-23T10:27:42.521612+0000 2=0+2) old_inodes=1 (inest mix) (iquiesce lock x=1 by request(mds.1:108 nref=4)) | request=1 lock=1 dirfrag=1 0x55e7186f0b00]", 2024-04-23T10:32:04.287 INFO:teuthology.orchestra.run.smithi114.stdout: "lock": { 2024-04-23T10:32:04.287 INFO:teuthology.orchestra.run.smithi114.stdout: "gather_set": [], 2024-04-23T10:32:04.287 INFO:teuthology.orchestra.run.smithi114.stdout: "state": "lock", 2024-04-23T10:32:04.287 INFO:teuthology.orchestra.run.smithi114.stdout: "type": "iquiesce", 2024-04-23T10:32:04.287 INFO:teuthology.orchestra.run.smithi114.stdout: "is_leased": false, 2024-04-23T10:32:04.287 INFO:teuthology.orchestra.run.smithi114.stdout: "num_rdlocks": 0, 2024-04-23T10:32:04.287 INFO:teuthology.orchestra.run.smithi114.stdout: "num_wrlocks": 0, 2024-04-23T10:32:04.287 INFO:teuthology.orchestra.run.smithi114.stdout: "num_xlocks": 1, 2024-04-23T10:32:04.287 INFO:teuthology.orchestra.run.smithi114.stdout: "xlock_by": { 2024-04-23T10:32:04.287 INFO:teuthology.orchestra.run.smithi114.stdout: "reqid": { 2024-04-23T10:32:04.287 INFO:teuthology.orchestra.run.smithi114.stdout: "entity": { 2024-04-23T10:32:04.287 INFO:teuthology.orchestra.run.smithi114.stdout: "type": "mds", 2024-04-23T10:32:04.288 INFO:teuthology.orchestra.run.smithi114.stdout: "num": 1 2024-04-23T10:32:04.288 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-23T10:32:04.288 INFO:teuthology.orchestra.run.smithi114.stdout: "tid": 108 2024-04-23T10:32:04.288 INFO:teuthology.orchestra.run.smithi114.stdout: } 2024-04-23T10:32:04.288 INFO:teuthology.orchestra.run.smithi114.stdout: } 2024-04-23T10:32:04.288 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-23T10:32:04.288 INFO:teuthology.orchestra.run.smithi114.stdout: "flags": 4, 2024-04-23T10:32:04.288 INFO:teuthology.orchestra.run.smithi114.stdout: "wrlock_target": -1 2024-04-23T10:32:04.288 INFO:teuthology.orchestra.run.smithi114.stdout: } 2024-04-23T10:32:04.288 INFO:teuthology.orchestra.run.smithi114.stdout: ] 2024-04-23T10:32:04.288 INFO:teuthology.orchestra.run.smithi114.stdout: } 2024-04-23T10:32:04.288 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-23T10:32:04.288 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-04-23T10:32:04.288 INFO:teuthology.orchestra.run.smithi114.stdout: "description": "internal op quiesce_inode:mds.1:140 fp=#0x10000000236", 2024-04-23T10:32:04.289 INFO:teuthology.orchestra.run.smithi114.stdout: "initiated_at": "2024-04-23T10:31:21.494359+0000", 2024-04-23T10:32:04.289 INFO:teuthology.orchestra.run.smithi114.stdout: "age": 42.363279785000003, 2024-04-23T10:32:04.289 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 42.364023766000003, 2024-04-23T10:32:04.289 INFO:teuthology.orchestra.run.smithi114.stdout: "continuous": true, 2024-04-23T10:32:04.289 INFO:teuthology.orchestra.run.smithi114.stdout: "type_data": { 2024-04-23T10:32:04.289 INFO:teuthology.orchestra.run.smithi114.stdout: "result": -2147483648, 2024-04-23T10:32:04.289 INFO:teuthology.orchestra.run.smithi114.stdout: "flag_point": "failed to authpin, dir is being fragmented", 2024-04-23T10:32:04.289 INFO:teuthology.orchestra.run.smithi114.stdout: "reqid": { 2024-04-23T10:32:04.289 INFO:teuthology.orchestra.run.smithi114.stdout: "entity": { 2024-04-23T10:32:04.289 INFO:teuthology.orchestra.run.smithi114.stdout: "type": "mds", 2024-04-23T10:32:04.289 INFO:teuthology.orchestra.run.smithi114.stdout: "num": 1 2024-04-23T10:32:04.289 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-23T10:32:04.289 INFO:teuthology.orchestra.run.smithi114.stdout: "tid": 140 2024-04-23T10:32:04.289 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-23T10:32:04.289 INFO:teuthology.orchestra.run.smithi114.stdout: "op_type": "internal_op", 2024-04-23T10:32:04.289 INFO:teuthology.orchestra.run.smithi114.stdout: "internal_op": 5385, 2024-04-23T10:32:04.290 INFO:teuthology.orchestra.run.smithi114.stdout: "op_name": "quiesce_inode", 2024-04-23T10:32:04.290 INFO:teuthology.orchestra.run.smithi114.stdout: "events": [ 2024-04-23T10:32:04.290 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-04-23T10:32:04.290 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-04-23T10:31:21.494359+0000", 2024-04-23T10:32:04.290 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "initiated" 2024-04-23T10:32:04.290 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-23T10:32:04.290 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-04-23T10:32:04.290 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-04-23T10:31:21.494359+0000", 2024-04-23T10:32:04.290 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "header_read" 2024-04-23T10:32:04.290 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-23T10:32:04.290 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-04-23T10:32:04.290 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-04-23T10:31:21.494359+0000", 2024-04-23T10:32:04.290 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "throttled" 2024-04-23T10:32:04.291 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-23T10:32:04.291 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-04-23T10:32:04.291 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-04-23T10:31:21.494359+0000", 2024-04-23T10:32:04.291 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "all_read" 2024-04-23T10:32:04.291 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-23T10:32:04.291 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-04-23T10:32:04.291 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-04-23T10:31:21.494359+0000", 2024-04-23T10:32:04.291 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "dispatched" 2024-04-23T10:32:04.291 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-23T10:32:04.291 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-04-23T10:32:04.291 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-04-23T10:31:21.496168+0000", 2024-04-23T10:32:04.291 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "failed to authpin, dir is being fragmented" 2024-04-23T10:32:04.291 INFO:teuthology.orchestra.run.smithi114.stdout: } 2024-04-23T10:32:04.291 INFO:teuthology.orchestra.run.smithi114.stdout: ], 2024-04-23T10:32:04.291 INFO:teuthology.orchestra.run.smithi114.stdout: "locks": [] 2024-04-23T10:32:04.292 INFO:teuthology.orchestra.run.smithi114.stdout: } 2024-04-23T10:32:04.292 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-23T10:32:04.292 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-04-23T10:32:04.292 INFO:teuthology.orchestra.run.smithi114.stdout: "description": "internal op quiesce_inode:mds.1:109 fp=#0x10000000006", 2024-04-23T10:32:04.292 INFO:teuthology.orchestra.run.smithi114.stdout: "initiated_at": "2024-04-23T10:31:21.452069+0000", 2024-04-23T10:32:04.292 INFO:teuthology.orchestra.run.smithi114.stdout: "age": 42.405569727, 2024-04-23T10:32:04.292 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 42.406337188000002, 2024-04-23T10:32:04.292 INFO:teuthology.orchestra.run.smithi114.stdout: "continuous": true, 2024-04-23T10:32:04.292 INFO:teuthology.orchestra.run.smithi114.stdout: "type_data": { 2024-04-23T10:32:04.292 INFO:teuthology.orchestra.run.smithi114.stdout: "result": -2147483648, 2024-04-23T10:32:04.292 INFO:teuthology.orchestra.run.smithi114.stdout: "flag_point": "acquired locks", 2024-04-23T10:32:04.292 INFO:teuthology.orchestra.run.smithi114.stdout: "reqid": { 2024-04-23T10:32:04.292 INFO:teuthology.orchestra.run.smithi114.stdout: "entity": { 2024-04-23T10:32:04.292 INFO:teuthology.orchestra.run.smithi114.stdout: "type": "mds", 2024-04-23T10:32:04.293 INFO:teuthology.orchestra.run.smithi114.stdout: "num": 1 2024-04-23T10:32:04.293 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-23T10:32:04.293 INFO:teuthology.orchestra.run.smithi114.stdout: "tid": 109 2024-04-23T10:32:04.293 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-23T10:32:04.293 INFO:teuthology.orchestra.run.smithi114.stdout: "op_type": "internal_op", 2024-04-23T10:32:04.293 INFO:teuthology.orchestra.run.smithi114.stdout: "internal_op": 5385, 2024-04-23T10:32:04.293 INFO:teuthology.orchestra.run.smithi114.stdout: "op_name": "quiesce_inode", 2024-04-23T10:32:04.293 INFO:teuthology.orchestra.run.smithi114.stdout: "events": [ 2024-04-23T10:32:04.293 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-04-23T10:32:04.293 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-04-23T10:31:21.452069+0000", 2024-04-23T10:32:04.293 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "initiated" 2024-04-23T10:32:04.293 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-23T10:32:04.293 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-04-23T10:32:04.293 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-04-23T10:31:21.452069+0000", 2024-04-23T10:32:04.294 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "header_read" 2024-04-23T10:32:04.294 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-23T10:32:04.294 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-04-23T10:32:04.294 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-04-23T10:31:21.452069+0000", 2024-04-23T10:32:04.294 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "throttled" 2024-04-23T10:32:04.294 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-23T10:32:04.294 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-04-23T10:32:04.294 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-04-23T10:31:21.452069+0000", 2024-04-23T10:32:04.294 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "all_read" 2024-04-23T10:32:04.294 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-23T10:32:04.294 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-04-23T10:32:04.294 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-04-23T10:31:21.452069+0000", 2024-04-23T10:32:04.294 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "dispatched" 2024-04-23T10:32:04.294 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-23T10:32:04.294 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-04-23T10:32:04.295 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-04-23T10:31:21.452141+0000", 2024-04-23T10:32:04.295 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "requesting remote authpins" 2024-04-23T10:32:04.295 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-23T10:32:04.295 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-04-23T10:32:04.295 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-04-23T10:31:21.453664+0000", 2024-04-23T10:32:04.295 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "acquired locks" 2024-04-23T10:32:04.295 INFO:teuthology.orchestra.run.smithi114.stdout: } 2024-04-23T10:32:04.295 INFO:teuthology.orchestra.run.smithi114.stdout: ], 2024-04-23T10:32:04.295 INFO:teuthology.orchestra.run.smithi114.stdout: "locks": [ 2024-04-23T10:32:04.295 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-04-23T10:32:04.295 INFO:teuthology.orchestra.run.smithi114.stdout: "object": { 2024-04-23T10:32:04.295 INFO:teuthology.orchestra.run.smithi114.stdout: "is_auth": false, 2024-04-23T10:32:04.295 INFO:teuthology.orchestra.run.smithi114.stdout: "auth_state": { 2024-04-23T10:32:04.295 INFO:teuthology.orchestra.run.smithi114.stdout: "replicas": {} 2024-04-23T10:32:04.295 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-23T10:32:04.296 INFO:teuthology.orchestra.run.smithi114.stdout: "replica_state": { 2024-04-23T10:32:04.296 INFO:teuthology.orchestra.run.smithi114.stdout: "authority": [ 2024-04-23T10:32:04.296 INFO:teuthology.orchestra.run.smithi114.stdout: 0, 2024-04-23T10:32:04.296 INFO:teuthology.orchestra.run.smithi114.stdout: -2 2024-04-23T10:32:04.296 INFO:teuthology.orchestra.run.smithi114.stdout: ], 2024-04-23T10:32:04.296 INFO:teuthology.orchestra.run.smithi114.stdout: "replica_nonce": 1 2024-04-23T10:32:04.296 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-23T10:32:04.296 INFO:teuthology.orchestra.run.smithi114.stdout: "auth_pins": 0, 2024-04-23T10:32:04.296 INFO:teuthology.orchestra.run.smithi114.stdout: "is_frozen": false, 2024-04-23T10:32:04.296 INFO:teuthology.orchestra.run.smithi114.stdout: "is_freezing": false, 2024-04-23T10:32:04.296 INFO:teuthology.orchestra.run.smithi114.stdout: "pins": { 2024-04-23T10:32:04.296 INFO:teuthology.orchestra.run.smithi114.stdout: "request": 1, 2024-04-23T10:32:04.296 INFO:teuthology.orchestra.run.smithi114.stdout: "lock": 1, 2024-04-23T10:32:04.296 INFO:teuthology.orchestra.run.smithi114.stdout: "dirfrag": 1 2024-04-23T10:32:04.296 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-23T10:32:04.296 INFO:teuthology.orchestra.run.smithi114.stdout: "nref": 3 2024-04-23T10:32:04.297 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-23T10:32:04.297 INFO:teuthology.orchestra.run.smithi114.stdout: "object_string": "[inode 0x10000000006 [...7,head] /volumes/qa/sv_1/b206794e-7622-4ab8-8943-f3ae4e65b7a3/client.0/tmp/ rep@0.1 v408 f(v0 m2024-04-23T10:27:51.919399+0000 1=0+1) n(v30 rc2024-04-23T10:31:17.764716+0000 b21075074 76=63+13)/n(v0 rc2024-04-23T10:27:51.919399+0000 2=0+2) (inest mix) (iquiesce lock x=1 by request(mds.1:109 nref=4)) | request=1 lock=1 dirfrag=1 0x55e7186f0580]", 2024-04-23T10:32:04.297 INFO:teuthology.orchestra.run.smithi114.stdout: "lock": { 2024-04-23T10:32:04.297 INFO:teuthology.orchestra.run.smithi114.stdout: "gather_set": [], 2024-04-23T10:32:04.297 INFO:teuthology.orchestra.run.smithi114.stdout: "state": "lock", 2024-04-23T10:32:04.297 INFO:teuthology.orchestra.run.smithi114.stdout: "type": "iquiesce", 2024-04-23T10:32:04.297 INFO:teuthology.orchestra.run.smithi114.stdout: "is_leased": false, 2024-04-23T10:32:04.297 INFO:teuthology.orchestra.run.smithi114.stdout: "num_rdlocks": 0, 2024-04-23T10:32:04.297 INFO:teuthology.orchestra.run.smithi114.stdout: "num_wrlocks": 0, 2024-04-23T10:32:04.297 INFO:teuthology.orchestra.run.smithi114.stdout: "num_xlocks": 1, 2024-04-23T10:32:04.297 INFO:teuthology.orchestra.run.smithi114.stdout: "xlock_by": { 2024-04-23T10:32:04.297 INFO:teuthology.orchestra.run.smithi114.stdout: "reqid": { 2024-04-23T10:32:04.297 INFO:teuthology.orchestra.run.smithi114.stdout: "entity": { 2024-04-23T10:32:04.297 INFO:teuthology.orchestra.run.smithi114.stdout: "type": "mds", 2024-04-23T10:32:04.297 INFO:teuthology.orchestra.run.smithi114.stdout: "num": 1 2024-04-23T10:32:04.297 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-23T10:32:04.298 INFO:teuthology.orchestra.run.smithi114.stdout: "tid": 109 2024-04-23T10:32:04.298 INFO:teuthology.orchestra.run.smithi114.stdout: } 2024-04-23T10:32:04.298 INFO:teuthology.orchestra.run.smithi114.stdout: } 2024-04-23T10:32:04.298 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-23T10:32:04.298 INFO:teuthology.orchestra.run.smithi114.stdout: "flags": 4, 2024-04-23T10:32:04.298 INFO:teuthology.orchestra.run.smithi114.stdout: "wrlock_target": -1 2024-04-23T10:32:04.298 INFO:teuthology.orchestra.run.smithi114.stdout: } 2024-04-23T10:32:04.298 INFO:teuthology.orchestra.run.smithi114.stdout: ] 2024-04-23T10:32:04.298 INFO:teuthology.orchestra.run.smithi114.stdout: } 2024-04-23T10:32:04.298 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-23T10:32:04.298 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-04-23T10:32:04.298 INFO:teuthology.orchestra.run.smithi114.stdout: "description": "internal op quiesce_inode:mds.1:141 fp=#0x10000000232", 2024-04-23T10:32:04.298 INFO:teuthology.orchestra.run.smithi114.stdout: "initiated_at": "2024-04-23T10:31:21.494372+0000", 2024-04-23T10:32:04.298 INFO:teuthology.orchestra.run.smithi114.stdout: "age": 42.363266496999998, 2024-04-23T10:32:04.299 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 42.364115597000001, 2024-04-23T10:32:04.299 INFO:teuthology.orchestra.run.smithi114.stdout: "continuous": true, 2024-04-23T10:32:04.299 INFO:teuthology.orchestra.run.smithi114.stdout: "type_data": { 2024-04-23T10:32:04.299 INFO:teuthology.orchestra.run.smithi114.stdout: "result": -2147483648, 2024-04-23T10:32:04.299 INFO:teuthology.orchestra.run.smithi114.stdout: "flag_point": "failed to authpin, dir is being fragmented", 2024-04-23T10:32:04.299 INFO:teuthology.orchestra.run.smithi114.stdout: "reqid": { 2024-04-23T10:32:04.299 INFO:teuthology.orchestra.run.smithi114.stdout: "entity": { 2024-04-23T10:32:04.299 INFO:teuthology.orchestra.run.smithi114.stdout: "type": "mds", 2024-04-23T10:32:04.299 INFO:teuthology.orchestra.run.smithi114.stdout: "num": 1 2024-04-23T10:32:04.299 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-23T10:32:04.299 INFO:teuthology.orchestra.run.smithi114.stdout: "tid": 141 2024-04-23T10:32:04.299 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-23T10:32:04.299 INFO:teuthology.orchestra.run.smithi114.stdout: "op_type": "internal_op", 2024-04-23T10:32:04.299 INFO:teuthology.orchestra.run.smithi114.stdout: "internal_op": 5385, 2024-04-23T10:32:04.299 INFO:teuthology.orchestra.run.smithi114.stdout: "op_name": "quiesce_inode", 2024-04-23T10:32:04.299 INFO:teuthology.orchestra.run.smithi114.stdout: "events": [ 2024-04-23T10:32:04.300 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-04-23T10:32:04.300 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-04-23T10:31:21.494372+0000", 2024-04-23T10:32:04.300 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "initiated" 2024-04-23T10:32:04.300 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-23T10:32:04.300 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-04-23T10:32:04.300 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-04-23T10:31:21.494372+0000", 2024-04-23T10:32:04.300 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "header_read" 2024-04-23T10:32:04.300 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-23T10:32:04.300 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-04-23T10:32:04.300 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-04-23T10:31:21.494372+0000", 2024-04-23T10:32:04.300 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "throttled" 2024-04-23T10:32:04.300 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-23T10:32:04.300 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-04-23T10:32:04.300 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-04-23T10:31:21.494372+0000", 2024-04-23T10:32:04.300 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "all_read" 2024-04-23T10:32:04.301 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-23T10:32:04.301 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-04-23T10:32:04.301 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-04-23T10:31:21.494372+0000", 2024-04-23T10:32:04.301 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "dispatched" 2024-04-23T10:32:04.301 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-23T10:32:04.301 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-04-23T10:32:04.301 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-04-23T10:31:21.496272+0000", 2024-04-23T10:32:04.301 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "failed to authpin, dir is being fragmented" 2024-04-23T10:32:04.301 INFO:teuthology.orchestra.run.smithi114.stdout: } 2024-04-23T10:32:04.301 INFO:teuthology.orchestra.run.smithi114.stdout: ], 2024-04-23T10:32:04.301 INFO:teuthology.orchestra.run.smithi114.stdout: "locks": [] 2024-04-23T10:32:04.301 INFO:teuthology.orchestra.run.smithi114.stdout: } 2024-04-23T10:32:04.301 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-23T10:32:04.301 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-04-23T10:32:04.301 INFO:teuthology.orchestra.run.smithi114.stdout: "description": "internal op quiesce_inode:mds.1:110 fp=#0x10000000007", 2024-04-23T10:32:04.302 INFO:teuthology.orchestra.run.smithi114.stdout: "initiated_at": "2024-04-23T10:31:21.453671+0000", 2024-04-23T10:32:04.302 INFO:teuthology.orchestra.run.smithi114.stdout: "age": 42.403967201999997, 2024-04-23T10:32:04.302 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 42.404843812999999, 2024-04-23T10:32:04.302 INFO:teuthology.orchestra.run.smithi114.stdout: "continuous": true, 2024-04-23T10:32:04.302 INFO:teuthology.orchestra.run.smithi114.stdout: "type_data": { 2024-04-23T10:32:04.302 INFO:teuthology.orchestra.run.smithi114.stdout: "result": -2147483648, 2024-04-23T10:32:04.302 INFO:teuthology.orchestra.run.smithi114.stdout: "flag_point": "acquired locks", 2024-04-23T10:32:04.302 INFO:teuthology.orchestra.run.smithi114.stdout: "reqid": { 2024-04-23T10:32:04.302 INFO:teuthology.orchestra.run.smithi114.stdout: "entity": { 2024-04-23T10:32:04.302 INFO:teuthology.orchestra.run.smithi114.stdout: "type": "mds", 2024-04-23T10:32:04.302 INFO:teuthology.orchestra.run.smithi114.stdout: "num": 1 2024-04-23T10:32:04.302 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-23T10:32:04.302 INFO:teuthology.orchestra.run.smithi114.stdout: "tid": 110 2024-04-23T10:32:04.302 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-23T10:32:04.303 INFO:teuthology.orchestra.run.smithi114.stdout: "op_type": "internal_op", 2024-04-23T10:32:04.303 INFO:teuthology.orchestra.run.smithi114.stdout: "internal_op": 5385, 2024-04-23T10:32:04.303 INFO:teuthology.orchestra.run.smithi114.stdout: "op_name": "quiesce_inode", 2024-04-23T10:32:04.303 INFO:teuthology.orchestra.run.smithi114.stdout: "events": [ 2024-04-23T10:32:04.303 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-04-23T10:32:04.303 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-04-23T10:31:21.453671+0000", 2024-04-23T10:32:04.303 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "initiated" 2024-04-23T10:32:04.303 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-23T10:32:04.303 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-04-23T10:32:04.303 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-04-23T10:31:21.453671+0000", 2024-04-23T10:32:04.303 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "header_read" 2024-04-23T10:32:04.303 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-23T10:32:04.303 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-04-23T10:32:04.303 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-04-23T10:31:21.453671+0000", 2024-04-23T10:32:04.303 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "throttled" 2024-04-23T10:32:04.303 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-23T10:32:04.303 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-04-23T10:32:04.304 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-04-23T10:31:21.453671+0000", 2024-04-23T10:32:04.304 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "all_read" 2024-04-23T10:32:04.304 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-23T10:32:04.304 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-04-23T10:32:04.304 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-04-23T10:31:21.453671+0000", 2024-04-23T10:32:04.304 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "dispatched" 2024-04-23T10:32:04.304 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-23T10:32:04.304 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-04-23T10:32:04.304 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-04-23T10:31:21.453719+0000", 2024-04-23T10:32:04.304 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "requesting remote authpins" 2024-04-23T10:32:04.304 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-23T10:32:04.304 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-04-23T10:32:04.304 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-04-23T10:31:21.455232+0000", 2024-04-23T10:32:04.304 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "acquired locks" 2024-04-23T10:32:04.305 INFO:teuthology.orchestra.run.smithi114.stdout: } 2024-04-23T10:32:04.305 INFO:teuthology.orchestra.run.smithi114.stdout: ], 2024-04-23T10:32:04.305 INFO:teuthology.orchestra.run.smithi114.stdout: "locks": [ 2024-04-23T10:32:04.305 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-04-23T10:32:04.305 INFO:teuthology.orchestra.run.smithi114.stdout: "object": { 2024-04-23T10:32:04.305 INFO:teuthology.orchestra.run.smithi114.stdout: "is_auth": false, 2024-04-23T10:32:04.305 INFO:teuthology.orchestra.run.smithi114.stdout: "auth_state": { 2024-04-23T10:32:04.305 INFO:teuthology.orchestra.run.smithi114.stdout: "replicas": {} 2024-04-23T10:32:04.305 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-23T10:32:04.305 INFO:teuthology.orchestra.run.smithi114.stdout: "replica_state": { 2024-04-23T10:32:04.305 INFO:teuthology.orchestra.run.smithi114.stdout: "authority": [ 2024-04-23T10:32:04.305 INFO:teuthology.orchestra.run.smithi114.stdout: 0, 2024-04-23T10:32:04.305 INFO:teuthology.orchestra.run.smithi114.stdout: -2 2024-04-23T10:32:04.305 INFO:teuthology.orchestra.run.smithi114.stdout: ], 2024-04-23T10:32:04.305 INFO:teuthology.orchestra.run.smithi114.stdout: "replica_nonce": 1 2024-04-23T10:32:04.306 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-23T10:32:04.306 INFO:teuthology.orchestra.run.smithi114.stdout: "auth_pins": 0, 2024-04-23T10:32:04.306 INFO:teuthology.orchestra.run.smithi114.stdout: "is_frozen": false, 2024-04-23T10:32:04.306 INFO:teuthology.orchestra.run.smithi114.stdout: "is_freezing": false, 2024-04-23T10:32:04.306 INFO:teuthology.orchestra.run.smithi114.stdout: "pins": { 2024-04-23T10:32:04.306 INFO:teuthology.orchestra.run.smithi114.stdout: "request": 1, 2024-04-23T10:32:04.306 INFO:teuthology.orchestra.run.smithi114.stdout: "lock": 1, 2024-04-23T10:32:04.306 INFO:teuthology.orchestra.run.smithi114.stdout: "dirfrag": 1 2024-04-23T10:32:04.306 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-23T10:32:04.306 INFO:teuthology.orchestra.run.smithi114.stdout: "nref": 3 2024-04-23T10:32:04.306 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-23T10:32:04.306 INFO:teuthology.orchestra.run.smithi114.stdout: "object_string": "[inode 0x10000000007 [...7,head] /volumes/qa/sv_1/b206794e-7622-4ab8-8943-f3ae4e65b7a3/client.0/tmp/clients/ rep@0.1 v412 f(v0 m2024-04-23T10:27:51.919399+0000 1=0+1) n(v30 rc2024-04-23T10:31:17.764716+0000 b21075074 75=63+12)/n(v0 rc2024-04-23T10:27:51.919399+0000 2=0+2) (inest mix) (iquiesce lock x=1 by request(mds.1:110 nref=4)) | request=1 lock=1 dirfrag=1 0x55e7186f3180]", 2024-04-23T10:32:04.306 INFO:teuthology.orchestra.run.smithi114.stdout: "lock": { 2024-04-23T10:32:04.306 INFO:teuthology.orchestra.run.smithi114.stdout: "gather_set": [], 2024-04-23T10:32:04.306 INFO:teuthology.orchestra.run.smithi114.stdout: "state": "lock", 2024-04-23T10:32:04.306 INFO:teuthology.orchestra.run.smithi114.stdout: "type": "iquiesce", 2024-04-23T10:32:04.307 INFO:teuthology.orchestra.run.smithi114.stdout: "is_leased": false, 2024-04-23T10:32:04.307 INFO:teuthology.orchestra.run.smithi114.stdout: "num_rdlocks": 0, 2024-04-23T10:32:04.307 INFO:teuthology.orchestra.run.smithi114.stdout: "num_wrlocks": 0, 2024-04-23T10:32:04.307 INFO:teuthology.orchestra.run.smithi114.stdout: "num_xlocks": 1, 2024-04-23T10:32:04.307 INFO:teuthology.orchestra.run.smithi114.stdout: "xlock_by": { 2024-04-23T10:32:04.307 INFO:teuthology.orchestra.run.smithi114.stdout: "reqid": { 2024-04-23T10:32:04.307 INFO:teuthology.orchestra.run.smithi114.stdout: "entity": { 2024-04-23T10:32:04.307 INFO:teuthology.orchestra.run.smithi114.stdout: "type": "mds", 2024-04-23T10:32:04.307 INFO:teuthology.orchestra.run.smithi114.stdout: "num": 1 2024-04-23T10:32:04.307 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-23T10:32:04.307 INFO:teuthology.orchestra.run.smithi114.stdout: "tid": 110 2024-04-23T10:32:04.307 INFO:teuthology.orchestra.run.smithi114.stdout: } 2024-04-23T10:32:04.307 INFO:teuthology.orchestra.run.smithi114.stdout: } 2024-04-23T10:32:04.307 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-23T10:32:04.307 INFO:teuthology.orchestra.run.smithi114.stdout: "flags": 4, 2024-04-23T10:32:04.308 INFO:teuthology.orchestra.run.smithi114.stdout: "wrlock_target": -1 2024-04-23T10:32:04.308 INFO:teuthology.orchestra.run.smithi114.stdout: } 2024-04-23T10:32:04.308 INFO:teuthology.orchestra.run.smithi114.stdout: ] 2024-04-23T10:32:04.308 INFO:teuthology.orchestra.run.smithi114.stdout: } 2024-04-23T10:32:04.308 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-23T10:32:04.308 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-04-23T10:32:04.308 INFO:teuthology.orchestra.run.smithi114.stdout: "description": "internal op quiesce_inode:mds.1:142 fp=#0x200000006f8", 2024-04-23T10:32:04.308 INFO:teuthology.orchestra.run.smithi114.stdout: "initiated_at": "2024-04-23T10:31:21.494389+0000", 2024-04-23T10:32:04.308 INFO:teuthology.orchestra.run.smithi114.stdout: "age": 42.363249875000001, 2024-04-23T10:32:04.308 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 42.364184604000002, 2024-04-23T10:32:04.308 INFO:teuthology.orchestra.run.smithi114.stdout: "continuous": true, 2024-04-23T10:32:04.308 INFO:teuthology.orchestra.run.smithi114.stdout: "type_data": { 2024-04-23T10:32:04.308 INFO:teuthology.orchestra.run.smithi114.stdout: "result": -2147483648, 2024-04-23T10:32:04.308 INFO:teuthology.orchestra.run.smithi114.stdout: "flag_point": "failed to authpin, dir is being fragmented", 2024-04-23T10:32:04.308 INFO:teuthology.orchestra.run.smithi114.stdout: "reqid": { 2024-04-23T10:32:04.309 INFO:teuthology.orchestra.run.smithi114.stdout: "entity": { 2024-04-23T10:32:04.309 INFO:teuthology.orchestra.run.smithi114.stdout: "type": "mds", 2024-04-23T10:32:04.309 INFO:teuthology.orchestra.run.smithi114.stdout: "num": 1 2024-04-23T10:32:04.309 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-23T10:32:04.309 INFO:teuthology.orchestra.run.smithi114.stdout: "tid": 142 2024-04-23T10:32:04.309 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-23T10:32:04.309 INFO:teuthology.orchestra.run.smithi114.stdout: "op_type": "internal_op", 2024-04-23T10:32:04.309 INFO:teuthology.orchestra.run.smithi114.stdout: "internal_op": 5385, 2024-04-23T10:32:04.309 INFO:teuthology.orchestra.run.smithi114.stdout: "op_name": "quiesce_inode", 2024-04-23T10:32:04.309 INFO:teuthology.orchestra.run.smithi114.stdout: "events": [ 2024-04-23T10:32:04.309 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-04-23T10:32:04.309 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-04-23T10:31:21.494389+0000", 2024-04-23T10:32:04.309 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "initiated" 2024-04-23T10:32:04.309 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-23T10:32:04.309 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-04-23T10:32:04.309 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-04-23T10:31:21.494389+0000", 2024-04-23T10:32:04.310 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "header_read" 2024-04-23T10:32:04.310 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-23T10:32:04.310 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-04-23T10:32:04.310 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-04-23T10:31:21.494389+0000", 2024-04-23T10:32:04.310 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "throttled" 2024-04-23T10:32:04.310 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-23T10:32:04.310 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-04-23T10:32:04.310 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-04-23T10:31:21.494389+0000", 2024-04-23T10:32:04.310 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "all_read" 2024-04-23T10:32:04.310 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-23T10:32:04.310 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-04-23T10:32:04.310 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-04-23T10:31:21.494389+0000", 2024-04-23T10:32:04.310 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "dispatched" 2024-04-23T10:32:04.310 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-23T10:32:04.310 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-04-23T10:32:04.311 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-04-23T10:31:21.496355+0000", 2024-04-23T10:32:04.311 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "failed to authpin, dir is being fragmented" 2024-04-23T10:32:04.311 INFO:teuthology.orchestra.run.smithi114.stdout: } 2024-04-23T10:32:04.311 INFO:teuthology.orchestra.run.smithi114.stdout: ], 2024-04-23T10:32:04.311 INFO:teuthology.orchestra.run.smithi114.stdout: "locks": [] 2024-04-23T10:32:04.311 INFO:teuthology.orchestra.run.smithi114.stdout: } 2024-04-23T10:32:04.311 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-23T10:32:04.311 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-04-23T10:32:04.311 INFO:teuthology.orchestra.run.smithi114.stdout: "description": "internal op quiesce_inode:mds.1:111 fp=#0x10000000008", 2024-04-23T10:32:04.311 INFO:teuthology.orchestra.run.smithi114.stdout: "initiated_at": "2024-04-23T10:31:21.455245+0000", 2024-04-23T10:32:04.311 INFO:teuthology.orchestra.run.smithi114.stdout: "age": 42.402393187000001, 2024-04-23T10:32:04.311 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 42.403352562000002, 2024-04-23T10:32:04.311 INFO:teuthology.orchestra.run.smithi114.stdout: "continuous": true, 2024-04-23T10:32:04.311 INFO:teuthology.orchestra.run.smithi114.stdout: "type_data": { 2024-04-23T10:32:04.311 INFO:teuthology.orchestra.run.smithi114.stdout: "result": -2147483648, 2024-04-23T10:32:04.312 INFO:teuthology.orchestra.run.smithi114.stdout: "flag_point": "acquired locks", 2024-04-23T10:32:04.312 INFO:teuthology.orchestra.run.smithi114.stdout: "reqid": { 2024-04-23T10:32:04.312 INFO:teuthology.orchestra.run.smithi114.stdout: "entity": { 2024-04-23T10:32:04.312 INFO:teuthology.orchestra.run.smithi114.stdout: "type": "mds", 2024-04-23T10:32:04.312 INFO:teuthology.orchestra.run.smithi114.stdout: "num": 1 2024-04-23T10:32:04.312 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-23T10:32:04.312 INFO:teuthology.orchestra.run.smithi114.stdout: "tid": 111 2024-04-23T10:32:04.312 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-23T10:32:04.312 INFO:teuthology.orchestra.run.smithi114.stdout: "op_type": "internal_op", 2024-04-23T10:32:04.312 INFO:teuthology.orchestra.run.smithi114.stdout: "internal_op": 5385, 2024-04-23T10:32:04.312 INFO:teuthology.orchestra.run.smithi114.stdout: "op_name": "quiesce_inode", 2024-04-23T10:32:04.312 INFO:teuthology.orchestra.run.smithi114.stdout: "events": [ 2024-04-23T10:32:04.312 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-04-23T10:32:04.312 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-04-23T10:31:21.455245+0000", 2024-04-23T10:32:04.312 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "initiated" 2024-04-23T10:32:04.313 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-23T10:32:04.313 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-04-23T10:32:04.313 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-04-23T10:31:21.455245+0000", 2024-04-23T10:32:04.313 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "header_read" 2024-04-23T10:32:04.313 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-23T10:32:04.313 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-04-23T10:32:04.313 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-04-23T10:31:21.455245+0000", 2024-04-23T10:32:04.313 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "throttled" 2024-04-23T10:32:04.313 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-23T10:32:04.313 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-04-23T10:32:04.313 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-04-23T10:31:21.455245+0000", 2024-04-23T10:32:04.313 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "all_read" 2024-04-23T10:32:04.313 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-23T10:32:04.313 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-04-23T10:32:04.313 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-04-23T10:31:21.455245+0000", 2024-04-23T10:32:04.313 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "dispatched" 2024-04-23T10:32:04.314 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-23T10:32:04.314 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-04-23T10:32:04.314 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-04-23T10:31:21.455296+0000", 2024-04-23T10:32:04.314 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "requesting remote authpins" 2024-04-23T10:32:04.314 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-23T10:32:04.314 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-04-23T10:32:04.314 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-04-23T10:31:21.456965+0000", 2024-04-23T10:32:04.314 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "acquired locks" 2024-04-23T10:32:04.314 INFO:teuthology.orchestra.run.smithi114.stdout: } 2024-04-23T10:32:04.314 INFO:teuthology.orchestra.run.smithi114.stdout: ], 2024-04-23T10:32:04.314 INFO:teuthology.orchestra.run.smithi114.stdout: "locks": [ 2024-04-23T10:32:04.314 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-04-23T10:32:04.314 INFO:teuthology.orchestra.run.smithi114.stdout: "object": { 2024-04-23T10:32:04.314 INFO:teuthology.orchestra.run.smithi114.stdout: "is_auth": false, 2024-04-23T10:32:04.314 INFO:teuthology.orchestra.run.smithi114.stdout: "auth_state": { 2024-04-23T10:32:04.314 INFO:teuthology.orchestra.run.smithi114.stdout: "replicas": {} 2024-04-23T10:32:04.315 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-23T10:32:04.315 INFO:teuthology.orchestra.run.smithi114.stdout: "replica_state": { 2024-04-23T10:32:04.315 INFO:teuthology.orchestra.run.smithi114.stdout: "authority": [ 2024-04-23T10:32:04.315 INFO:teuthology.orchestra.run.smithi114.stdout: 0, 2024-04-23T10:32:04.315 INFO:teuthology.orchestra.run.smithi114.stdout: -2 2024-04-23T10:32:04.315 INFO:teuthology.orchestra.run.smithi114.stdout: ], 2024-04-23T10:32:04.315 INFO:teuthology.orchestra.run.smithi114.stdout: "replica_nonce": 1 2024-04-23T10:32:04.315 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-23T10:32:04.315 INFO:teuthology.orchestra.run.smithi114.stdout: "auth_pins": 0, 2024-04-23T10:32:04.315 INFO:teuthology.orchestra.run.smithi114.stdout: "is_frozen": false, 2024-04-23T10:32:04.315 INFO:teuthology.orchestra.run.smithi114.stdout: "is_freezing": false, 2024-04-23T10:32:04.315 INFO:teuthology.orchestra.run.smithi114.stdout: "pins": { 2024-04-23T10:32:04.315 INFO:teuthology.orchestra.run.smithi114.stdout: "request": 1, 2024-04-23T10:32:04.315 INFO:teuthology.orchestra.run.smithi114.stdout: "lock": 1, 2024-04-23T10:32:04.316 INFO:teuthology.orchestra.run.smithi114.stdout: "dirfrag": 1, 2024-04-23T10:32:04.316 INFO:teuthology.orchestra.run.smithi114.stdout: "dirwaiter": 0, 2024-04-23T10:32:04.316 INFO:teuthology.orchestra.run.smithi114.stdout: "discoverbase": 0 2024-04-23T10:32:04.316 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-23T10:32:04.316 INFO:teuthology.orchestra.run.smithi114.stdout: "nref": 3 2024-04-23T10:32:04.316 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-23T10:32:04.316 INFO:teuthology.orchestra.run.smithi114.stdout: "object_string": "[inode 0x10000000008 [...7,head] /volumes/qa/sv_1/b206794e-7622-4ab8-8943-f3ae4e65b7a3/client.0/tmp/clients/client0/ rep@0.1 v456 f(v0 m2024-04-23T10:27:51.950398+0000 6=5+1) n(v30 rc2024-04-23T10:31:17.764716+0000 b21075074 74=63+11)/n(v0 rc2024-04-23T10:27:51.920398+0000 2=0+2) (inest mix) (iquiesce lock x=1 by request(mds.1:111 nref=4)) | request=1 lock=1 dirfrag=1 dirwaiter=0 discoverbase=0 0x55e7186f3700]", 2024-04-23T10:32:04.316 INFO:teuthology.orchestra.run.smithi114.stdout: "lock": { 2024-04-23T10:32:04.316 INFO:teuthology.orchestra.run.smithi114.stdout: "gather_set": [], 2024-04-23T10:32:04.316 INFO:teuthology.orchestra.run.smithi114.stdout: "state": "lock", 2024-04-23T10:32:04.316 INFO:teuthology.orchestra.run.smithi114.stdout: "type": "iquiesce", 2024-04-23T10:32:04.316 INFO:teuthology.orchestra.run.smithi114.stdout: "is_leased": false, 2024-04-23T10:32:04.316 INFO:teuthology.orchestra.run.smithi114.stdout: "num_rdlocks": 0, 2024-04-23T10:32:04.316 INFO:teuthology.orchestra.run.smithi114.stdout: "num_wrlocks": 0, 2024-04-23T10:32:04.317 INFO:teuthology.orchestra.run.smithi114.stdout: "num_xlocks": 1, 2024-04-23T10:32:04.317 INFO:teuthology.orchestra.run.smithi114.stdout: "xlock_by": { 2024-04-23T10:32:04.317 INFO:teuthology.orchestra.run.smithi114.stdout: "reqid": { 2024-04-23T10:32:04.317 INFO:teuthology.orchestra.run.smithi114.stdout: "entity": { 2024-04-23T10:32:04.317 INFO:teuthology.orchestra.run.smithi114.stdout: "type": "mds", 2024-04-23T10:32:04.317 INFO:teuthology.orchestra.run.smithi114.stdout: "num": 1 2024-04-23T10:32:04.317 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-23T10:32:04.317 INFO:teuthology.orchestra.run.smithi114.stdout: "tid": 111 2024-04-23T10:32:04.317 INFO:teuthology.orchestra.run.smithi114.stdout: } 2024-04-23T10:32:04.317 INFO:teuthology.orchestra.run.smithi114.stdout: } 2024-04-23T10:32:04.317 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-23T10:32:04.317 INFO:teuthology.orchestra.run.smithi114.stdout: "flags": 4, 2024-04-23T10:32:04.317 INFO:teuthology.orchestra.run.smithi114.stdout: "wrlock_target": -1 2024-04-23T10:32:04.317 INFO:teuthology.orchestra.run.smithi114.stdout: } 2024-04-23T10:32:04.318 INFO:teuthology.orchestra.run.smithi114.stdout: ] 2024-04-23T10:32:04.318 INFO:teuthology.orchestra.run.smithi114.stdout: } 2024-04-23T10:32:04.318 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-23T10:32:04.318 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-04-23T10:32:04.318 INFO:teuthology.orchestra.run.smithi114.stdout: "description": "internal op quiesce_inode:mds.1:143 fp=#0x10000000230", 2024-04-23T10:32:04.318 INFO:teuthology.orchestra.run.smithi114.stdout: "initiated_at": "2024-04-23T10:31:21.494413+0000", 2024-04-23T10:32:04.318 INFO:teuthology.orchestra.run.smithi114.stdout: "age": 42.363225432999997, 2024-04-23T10:32:04.318 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 42.364282203999998, 2024-04-23T10:32:04.318 INFO:teuthology.orchestra.run.smithi114.stdout: "continuous": true, 2024-04-23T10:32:04.318 INFO:teuthology.orchestra.run.smithi114.stdout: "type_data": { 2024-04-23T10:32:04.318 INFO:teuthology.orchestra.run.smithi114.stdout: "result": -2147483648, 2024-04-23T10:32:04.318 INFO:teuthology.orchestra.run.smithi114.stdout: "flag_point": "failed to authpin, dir is being fragmented", 2024-04-23T10:32:04.318 INFO:teuthology.orchestra.run.smithi114.stdout: "reqid": { 2024-04-23T10:32:04.318 INFO:teuthology.orchestra.run.smithi114.stdout: "entity": { 2024-04-23T10:32:04.318 INFO:teuthology.orchestra.run.smithi114.stdout: "type": "mds", 2024-04-23T10:32:04.319 INFO:teuthology.orchestra.run.smithi114.stdout: "num": 1 2024-04-23T10:32:04.319 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-23T10:32:04.319 INFO:teuthology.orchestra.run.smithi114.stdout: "tid": 143 2024-04-23T10:32:04.319 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-23T10:32:04.319 INFO:teuthology.orchestra.run.smithi114.stdout: "op_type": "internal_op", 2024-04-23T10:32:04.319 INFO:teuthology.orchestra.run.smithi114.stdout: "internal_op": 5385, 2024-04-23T10:32:04.319 INFO:teuthology.orchestra.run.smithi114.stdout: "op_name": "quiesce_inode", 2024-04-23T10:32:04.319 INFO:teuthology.orchestra.run.smithi114.stdout: "events": [ 2024-04-23T10:32:04.319 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-04-23T10:32:04.319 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-04-23T10:31:21.494413+0000", 2024-04-23T10:32:04.319 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "initiated" 2024-04-23T10:32:04.319 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-23T10:32:04.319 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-04-23T10:32:04.319 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-04-23T10:31:21.494413+0000", 2024-04-23T10:32:04.320 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "header_read" 2024-04-23T10:32:04.320 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-23T10:32:04.320 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-04-23T10:32:04.320 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-04-23T10:31:21.494413+0000", 2024-04-23T10:32:04.320 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "throttled" 2024-04-23T10:32:04.320 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-23T10:32:04.320 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-04-23T10:32:04.320 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-04-23T10:31:21.494413+0000", 2024-04-23T10:32:04.320 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "all_read" 2024-04-23T10:32:04.320 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-23T10:32:04.320 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-04-23T10:32:04.320 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-04-23T10:31:21.494413+0000", 2024-04-23T10:32:04.320 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "dispatched" 2024-04-23T10:32:04.320 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-23T10:32:04.320 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-04-23T10:32:04.321 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-04-23T10:31:21.496473+0000", 2024-04-23T10:32:04.321 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "failed to authpin, dir is being fragmented" 2024-04-23T10:32:04.321 INFO:teuthology.orchestra.run.smithi114.stdout: } 2024-04-23T10:32:04.321 INFO:teuthology.orchestra.run.smithi114.stdout: ], 2024-04-23T10:32:04.321 INFO:teuthology.orchestra.run.smithi114.stdout: "locks": [] 2024-04-23T10:32:04.321 INFO:teuthology.orchestra.run.smithi114.stdout: } 2024-04-23T10:32:04.321 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-23T10:32:04.321 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-04-23T10:32:04.321 INFO:teuthology.orchestra.run.smithi114.stdout: "description": "internal op quiesce_inode:mds.1:112 fp=#0x10000000009", 2024-04-23T10:32:04.321 INFO:teuthology.orchestra.run.smithi114.stdout: "initiated_at": "2024-04-23T10:31:21.456973+0000", 2024-04-23T10:32:04.321 INFO:teuthology.orchestra.run.smithi114.stdout: "age": 42.400665381000003, 2024-04-23T10:32:04.321 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 42.401747045999997, 2024-04-23T10:32:04.321 INFO:teuthology.orchestra.run.smithi114.stdout: "continuous": true, 2024-04-23T10:32:04.321 INFO:teuthology.orchestra.run.smithi114.stdout: "type_data": { 2024-04-23T10:32:04.322 INFO:teuthology.orchestra.run.smithi114.stdout: "result": -2147483648, 2024-04-23T10:32:04.322 INFO:teuthology.orchestra.run.smithi114.stdout: "flag_point": "acquired locks", 2024-04-23T10:32:04.322 INFO:teuthology.orchestra.run.smithi114.stdout: "reqid": { 2024-04-23T10:32:04.322 INFO:teuthology.orchestra.run.smithi114.stdout: "entity": { 2024-04-23T10:32:04.322 INFO:teuthology.orchestra.run.smithi114.stdout: "type": "mds", 2024-04-23T10:32:04.322 INFO:teuthology.orchestra.run.smithi114.stdout: "num": 1 2024-04-23T10:32:04.322 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-23T10:32:04.322 INFO:teuthology.orchestra.run.smithi114.stdout: "tid": 112 2024-04-23T10:32:04.322 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-23T10:32:04.322 INFO:teuthology.orchestra.run.smithi114.stdout: "op_type": "internal_op", 2024-04-23T10:32:04.322 INFO:teuthology.orchestra.run.smithi114.stdout: "internal_op": 5385, 2024-04-23T10:32:04.322 INFO:teuthology.orchestra.run.smithi114.stdout: "op_name": "quiesce_inode", 2024-04-23T10:32:04.322 INFO:teuthology.orchestra.run.smithi114.stdout: "events": [ 2024-04-23T10:32:04.322 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-04-23T10:32:04.323 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-04-23T10:31:21.456973+0000", 2024-04-23T10:32:04.323 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "initiated" 2024-04-23T10:32:04.323 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-23T10:32:04.323 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-04-23T10:32:04.323 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-04-23T10:31:21.456973+0000", 2024-04-23T10:32:04.323 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "header_read" 2024-04-23T10:32:04.323 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-23T10:32:04.323 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-04-23T10:32:04.323 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-04-23T10:31:21.456973+0000", 2024-04-23T10:32:04.323 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "throttled" 2024-04-23T10:32:04.323 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-23T10:32:04.323 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-04-23T10:32:04.323 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-04-23T10:31:21.456973+0000", 2024-04-23T10:32:04.323 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "all_read" 2024-04-23T10:32:04.324 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-23T10:32:04.324 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-04-23T10:32:04.324 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-04-23T10:31:21.456973+0000", 2024-04-23T10:32:04.324 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "dispatched" 2024-04-23T10:32:04.324 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-23T10:32:04.324 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-04-23T10:32:04.324 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-04-23T10:31:21.457057+0000", 2024-04-23T10:32:04.324 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "requesting remote authpins" 2024-04-23T10:32:04.324 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-23T10:32:04.324 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-04-23T10:32:04.324 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-04-23T10:31:21.464343+0000", 2024-04-23T10:32:04.324 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "acquired locks" 2024-04-23T10:32:04.324 INFO:teuthology.orchestra.run.smithi114.stdout: } 2024-04-23T10:32:04.324 INFO:teuthology.orchestra.run.smithi114.stdout: ], 2024-04-23T10:32:04.325 INFO:teuthology.orchestra.run.smithi114.stdout: "locks": [ 2024-04-23T10:32:04.325 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-04-23T10:32:04.325 INFO:teuthology.orchestra.run.smithi114.stdout: "object": { 2024-04-23T10:32:04.325 INFO:teuthology.orchestra.run.smithi114.stdout: "is_auth": false, 2024-04-23T10:32:04.325 INFO:teuthology.orchestra.run.smithi114.stdout: "auth_state": { 2024-04-23T10:32:04.325 INFO:teuthology.orchestra.run.smithi114.stdout: "replicas": {} 2024-04-23T10:32:04.325 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-23T10:32:04.325 INFO:teuthology.orchestra.run.smithi114.stdout: "replica_state": { 2024-04-23T10:32:04.325 INFO:teuthology.orchestra.run.smithi114.stdout: "authority": [ 2024-04-23T10:32:04.325 INFO:teuthology.orchestra.run.smithi114.stdout: 0, 2024-04-23T10:32:04.325 INFO:teuthology.orchestra.run.smithi114.stdout: -2 2024-04-23T10:32:04.325 INFO:teuthology.orchestra.run.smithi114.stdout: ], 2024-04-23T10:32:04.325 INFO:teuthology.orchestra.run.smithi114.stdout: "replica_nonce": 1 2024-04-23T10:32:04.325 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-23T10:32:04.325 INFO:teuthology.orchestra.run.smithi114.stdout: "auth_pins": 0, 2024-04-23T10:32:04.326 INFO:teuthology.orchestra.run.smithi114.stdout: "is_frozen": false, 2024-04-23T10:32:04.326 INFO:teuthology.orchestra.run.smithi114.stdout: "is_freezing": false, 2024-04-23T10:32:04.326 INFO:teuthology.orchestra.run.smithi114.stdout: "pins": { 2024-04-23T10:32:04.326 INFO:teuthology.orchestra.run.smithi114.stdout: "request": 1, 2024-04-23T10:32:04.326 INFO:teuthology.orchestra.run.smithi114.stdout: "lock": 1, 2024-04-23T10:32:04.326 INFO:teuthology.orchestra.run.smithi114.stdout: "dirfrag": 1, 2024-04-23T10:32:04.326 INFO:teuthology.orchestra.run.smithi114.stdout: "discoverbase": 0 2024-04-23T10:32:04.326 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-23T10:32:04.326 INFO:teuthology.orchestra.run.smithi114.stdout: "nref": 3 2024-04-23T10:32:04.326 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-23T10:32:04.326 INFO:teuthology.orchestra.run.smithi114.stdout: "object_string": "[inode 0x10000000009 [...7,head] /volumes/qa/sv_1/b206794e-7622-4ab8-8943-f3ae4e65b7a3/client.0/tmp/clients/client0/~dmtmp/ rep@0.1 v15176 f(v0 m2024-04-23T10:27:52.186392+0000 9=0+9) n(v31 rc2024-04-23T10:31:17.764716+0000 b15832194 68=58+10)/n(v0 rc2024-04-23T10:27:51.953398+0000 2=0+2) (inest mix) (iquiesce lock x=1 by request(mds.1:112 nref=4)) | request=1 lock=1 dirfrag=1 discoverbase=0 0x55e718b90000]", 2024-04-23T10:32:04.326 INFO:teuthology.orchestra.run.smithi114.stdout: "lock": { 2024-04-23T10:32:04.326 INFO:teuthology.orchestra.run.smithi114.stdout: "gather_set": [], 2024-04-23T10:32:04.326 INFO:teuthology.orchestra.run.smithi114.stdout: "state": "lock", 2024-04-23T10:32:04.327 INFO:teuthology.orchestra.run.smithi114.stdout: "type": "iquiesce", 2024-04-23T10:32:04.327 INFO:teuthology.orchestra.run.smithi114.stdout: "is_leased": false, 2024-04-23T10:32:04.327 INFO:teuthology.orchestra.run.smithi114.stdout: "num_rdlocks": 0, 2024-04-23T10:32:04.327 INFO:teuthology.orchestra.run.smithi114.stdout: "num_wrlocks": 0, 2024-04-23T10:32:04.327 INFO:teuthology.orchestra.run.smithi114.stdout: "num_xlocks": 1, 2024-04-23T10:32:04.327 INFO:teuthology.orchestra.run.smithi114.stdout: "xlock_by": { 2024-04-23T10:32:04.327 INFO:teuthology.orchestra.run.smithi114.stdout: "reqid": { 2024-04-23T10:32:04.327 INFO:teuthology.orchestra.run.smithi114.stdout: "entity": { 2024-04-23T10:32:04.327 INFO:teuthology.orchestra.run.smithi114.stdout: "type": "mds", 2024-04-23T10:32:04.327 INFO:teuthology.orchestra.run.smithi114.stdout: "num": 1 2024-04-23T10:32:04.327 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-23T10:32:04.327 INFO:teuthology.orchestra.run.smithi114.stdout: "tid": 112 2024-04-23T10:32:04.327 INFO:teuthology.orchestra.run.smithi114.stdout: } 2024-04-23T10:32:04.327 INFO:teuthology.orchestra.run.smithi114.stdout: } 2024-04-23T10:32:04.328 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-23T10:32:04.328 INFO:teuthology.orchestra.run.smithi114.stdout: "flags": 4, 2024-04-23T10:32:04.328 INFO:teuthology.orchestra.run.smithi114.stdout: "wrlock_target": -1 2024-04-23T10:32:04.328 INFO:teuthology.orchestra.run.smithi114.stdout: } 2024-04-23T10:32:04.328 INFO:teuthology.orchestra.run.smithi114.stdout: ] 2024-04-23T10:32:04.328 INFO:teuthology.orchestra.run.smithi114.stdout: } 2024-04-23T10:32:04.328 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-23T10:32:04.328 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-04-23T10:32:04.328 INFO:teuthology.orchestra.run.smithi114.stdout: "description": "internal op quiesce_inode:mds.1:144 fp=#0x1000000022e", 2024-04-23T10:32:04.328 INFO:teuthology.orchestra.run.smithi114.stdout: "initiated_at": "2024-04-23T10:31:21.494428+0000", 2024-04-23T10:32:04.328 INFO:teuthology.orchestra.run.smithi114.stdout: "age": 42.363210180999999, 2024-04-23T10:32:04.328 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 42.364371300999998, 2024-04-23T10:32:04.329 INFO:teuthology.orchestra.run.smithi114.stdout: "continuous": true, 2024-04-23T10:32:04.329 INFO:teuthology.orchestra.run.smithi114.stdout: "type_data": { 2024-04-23T10:32:04.329 INFO:teuthology.orchestra.run.smithi114.stdout: "result": -2147483648, 2024-04-23T10:32:04.329 INFO:teuthology.orchestra.run.smithi114.stdout: "flag_point": "failed to authpin, dir is being fragmented", 2024-04-23T10:32:04.329 INFO:teuthology.orchestra.run.smithi114.stdout: "reqid": { 2024-04-23T10:32:04.329 INFO:teuthology.orchestra.run.smithi114.stdout: "entity": { 2024-04-23T10:32:04.329 INFO:teuthology.orchestra.run.smithi114.stdout: "type": "mds", 2024-04-23T10:32:04.329 INFO:teuthology.orchestra.run.smithi114.stdout: "num": 1 2024-04-23T10:32:04.329 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-23T10:32:04.329 INFO:teuthology.orchestra.run.smithi114.stdout: "tid": 144 2024-04-23T10:32:04.329 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-23T10:32:04.329 INFO:teuthology.orchestra.run.smithi114.stdout: "op_type": "internal_op", 2024-04-23T10:32:04.329 INFO:teuthology.orchestra.run.smithi114.stdout: "internal_op": 5385, 2024-04-23T10:32:04.329 INFO:teuthology.orchestra.run.smithi114.stdout: "op_name": "quiesce_inode", 2024-04-23T10:32:04.329 INFO:teuthology.orchestra.run.smithi114.stdout: "events": [ 2024-04-23T10:32:04.330 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-04-23T10:32:04.330 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-04-23T10:31:21.494428+0000", 2024-04-23T10:32:04.330 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "initiated" 2024-04-23T10:32:04.330 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-23T10:32:04.330 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-04-23T10:32:04.330 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-04-23T10:31:21.494428+0000", 2024-04-23T10:32:04.330 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "header_read" 2024-04-23T10:32:04.330 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-23T10:32:04.330 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-04-23T10:32:04.330 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-04-23T10:31:21.494428+0000", 2024-04-23T10:32:04.330 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "throttled" 2024-04-23T10:32:04.330 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-23T10:32:04.330 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-04-23T10:32:04.330 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-04-23T10:31:21.494428+0000", 2024-04-23T10:32:04.331 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "all_read" 2024-04-23T10:32:04.331 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-23T10:32:04.331 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-04-23T10:32:04.331 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-04-23T10:31:21.494428+0000", 2024-04-23T10:32:04.331 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "dispatched" 2024-04-23T10:32:04.331 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-23T10:32:04.331 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-04-23T10:32:04.331 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-04-23T10:31:21.496570+0000", 2024-04-23T10:32:04.331 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "failed to authpin, dir is being fragmented" 2024-04-23T10:32:04.331 INFO:teuthology.orchestra.run.smithi114.stdout: } 2024-04-23T10:32:04.331 INFO:teuthology.orchestra.run.smithi114.stdout: ], 2024-04-23T10:32:04.331 INFO:teuthology.orchestra.run.smithi114.stdout: "locks": [] 2024-04-23T10:32:04.331 INFO:teuthology.orchestra.run.smithi114.stdout: } 2024-04-23T10:32:04.331 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-23T10:32:04.331 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-04-23T10:32:04.332 INFO:teuthology.orchestra.run.smithi114.stdout: "description": "internal op quiesce_inode:mds.1:113 fp=#0x10000000215", 2024-04-23T10:32:04.332 INFO:teuthology.orchestra.run.smithi114.stdout: "initiated_at": "2024-04-23T10:31:21.464355+0000", 2024-04-23T10:32:04.332 INFO:teuthology.orchestra.run.smithi114.stdout: "age": 42.393283500999999, 2024-04-23T10:32:04.332 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 42.394488328999998, 2024-04-23T10:32:04.332 INFO:teuthology.orchestra.run.smithi114.stdout: "continuous": true, 2024-04-23T10:32:04.332 INFO:teuthology.orchestra.run.smithi114.stdout: "type_data": { 2024-04-23T10:32:04.332 INFO:teuthology.orchestra.run.smithi114.stdout: "result": -2147483648, 2024-04-23T10:32:04.332 INFO:teuthology.orchestra.run.smithi114.stdout: "flag_point": "quiesce complete for non-auth inode", 2024-04-23T10:32:04.332 INFO:teuthology.orchestra.run.smithi114.stdout: "reqid": { 2024-04-23T10:32:04.332 INFO:teuthology.orchestra.run.smithi114.stdout: "entity": { 2024-04-23T10:32:04.332 INFO:teuthology.orchestra.run.smithi114.stdout: "type": "mds", 2024-04-23T10:32:04.332 INFO:teuthology.orchestra.run.smithi114.stdout: "num": 1 2024-04-23T10:32:04.332 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-23T10:32:04.332 INFO:teuthology.orchestra.run.smithi114.stdout: "tid": 113 2024-04-23T10:32:04.333 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-23T10:32:04.333 INFO:teuthology.orchestra.run.smithi114.stdout: "op_type": "internal_op", 2024-04-23T10:32:04.333 INFO:teuthology.orchestra.run.smithi114.stdout: "internal_op": 5385, 2024-04-23T10:32:04.333 INFO:teuthology.orchestra.run.smithi114.stdout: "op_name": "quiesce_inode", 2024-04-23T10:32:04.333 INFO:teuthology.orchestra.run.smithi114.stdout: "events": [ 2024-04-23T10:32:04.333 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-04-23T10:32:04.333 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-04-23T10:31:21.464355+0000", 2024-04-23T10:32:04.333 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "initiated" 2024-04-23T10:32:04.333 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-23T10:32:04.333 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-04-23T10:32:04.333 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-04-23T10:31:21.464355+0000", 2024-04-23T10:32:04.333 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "header_read" 2024-04-23T10:32:04.333 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-23T10:32:04.333 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-04-23T10:32:04.333 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-04-23T10:31:21.464355+0000", 2024-04-23T10:32:04.333 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "throttled" 2024-04-23T10:32:04.334 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-23T10:32:04.334 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-04-23T10:32:04.334 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-04-23T10:31:21.464355+0000", 2024-04-23T10:32:04.334 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "all_read" 2024-04-23T10:32:04.334 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-23T10:32:04.334 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-04-23T10:32:04.334 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-04-23T10:31:21.464355+0000", 2024-04-23T10:32:04.334 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "dispatched" 2024-04-23T10:32:04.334 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-23T10:32:04.334 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-04-23T10:32:04.334 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-04-23T10:31:21.464598+0000", 2024-04-23T10:32:04.334 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "requesting remote authpins" 2024-04-23T10:32:04.334 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-23T10:32:04.334 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-04-23T10:32:04.334 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-04-23T10:31:21.478568+0000", 2024-04-23T10:32:04.334 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "acquired locks" 2024-04-23T10:32:04.335 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-23T10:32:04.335 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-04-23T10:32:04.335 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-04-23T10:31:21.478582+0000", 2024-04-23T10:32:04.335 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "quiesce complete for non-auth inode" 2024-04-23T10:32:04.335 INFO:teuthology.orchestra.run.smithi114.stdout: } 2024-04-23T10:32:04.335 INFO:teuthology.orchestra.run.smithi114.stdout: ], 2024-04-23T10:32:04.335 INFO:teuthology.orchestra.run.smithi114.stdout: "locks": [ 2024-04-23T10:32:04.335 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-04-23T10:32:04.335 INFO:teuthology.orchestra.run.smithi114.stdout: "object": { 2024-04-23T10:32:04.335 INFO:teuthology.orchestra.run.smithi114.stdout: "is_auth": false, 2024-04-23T10:32:04.335 INFO:teuthology.orchestra.run.smithi114.stdout: "auth_state": { 2024-04-23T10:32:04.335 INFO:teuthology.orchestra.run.smithi114.stdout: "replicas": {} 2024-04-23T10:32:04.335 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-23T10:32:04.335 INFO:teuthology.orchestra.run.smithi114.stdout: "replica_state": { 2024-04-23T10:32:04.335 INFO:teuthology.orchestra.run.smithi114.stdout: "authority": [ 2024-04-23T10:32:04.335 INFO:teuthology.orchestra.run.smithi114.stdout: 0, 2024-04-23T10:32:04.336 INFO:teuthology.orchestra.run.smithi114.stdout: -2 2024-04-23T10:32:04.336 INFO:teuthology.orchestra.run.smithi114.stdout: ], 2024-04-23T10:32:04.336 INFO:teuthology.orchestra.run.smithi114.stdout: "replica_nonce": 1 2024-04-23T10:32:04.336 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-23T10:32:04.336 INFO:teuthology.orchestra.run.smithi114.stdout: "auth_pins": 0, 2024-04-23T10:32:04.336 INFO:teuthology.orchestra.run.smithi114.stdout: "is_frozen": false, 2024-04-23T10:32:04.336 INFO:teuthology.orchestra.run.smithi114.stdout: "is_freezing": false, 2024-04-23T10:32:04.336 INFO:teuthology.orchestra.run.smithi114.stdout: "pins": { 2024-04-23T10:32:04.336 INFO:teuthology.orchestra.run.smithi114.stdout: "request": 1, 2024-04-23T10:32:04.336 INFO:teuthology.orchestra.run.smithi114.stdout: "lock": 1, 2024-04-23T10:32:04.336 INFO:teuthology.orchestra.run.smithi114.stdout: "dirfrag": 0, 2024-04-23T10:32:04.336 INFO:teuthology.orchestra.run.smithi114.stdout: "dirwaiter": 0, 2024-04-23T10:32:04.336 INFO:teuthology.orchestra.run.smithi114.stdout: "discoverbase": 0 2024-04-23T10:32:04.336 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-23T10:32:04.336 INFO:teuthology.orchestra.run.smithi114.stdout: "nref": 2 2024-04-23T10:32:04.336 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-23T10:32:04.336 INFO:teuthology.orchestra.run.smithi114.stdout: "object_string": "[inode 0x10000000215 [...7,head] /volumes/qa/sv_1/b206794e-7622-4ab8-8943-f3ae4e65b7a3/client.0/tmp/clients/client0/~dmtmp/COREL/ rep@0.1 v45589 f(v1232 m2024-04-23T10:31:17.745716+0000 11=11+0) n(v19 rc2024-04-23T10:31:17.745716+0000 b2861056 12=11+1)/n(v0 rc2024-04-23T10:27:52.004397+0000 2=1+1) (inest mix) (iquiesce lock x=1 by request(mds.1:113 nref=3)) | request=1 lock=1 dirfrag=0 dirwaiter=0 discoverbase=0 0x55e718b91600]", 2024-04-23T10:32:04.337 INFO:teuthology.orchestra.run.smithi114.stdout: "lock": { 2024-04-23T10:32:04.337 INFO:teuthology.orchestra.run.smithi114.stdout: "gather_set": [], 2024-04-23T10:32:04.337 INFO:teuthology.orchestra.run.smithi114.stdout: "state": "lock", 2024-04-23T10:32:04.337 INFO:teuthology.orchestra.run.smithi114.stdout: "type": "iquiesce", 2024-04-23T10:32:04.337 INFO:teuthology.orchestra.run.smithi114.stdout: "is_leased": false, 2024-04-23T10:32:04.337 INFO:teuthology.orchestra.run.smithi114.stdout: "num_rdlocks": 0, 2024-04-23T10:32:04.337 INFO:teuthology.orchestra.run.smithi114.stdout: "num_wrlocks": 0, 2024-04-23T10:32:04.337 INFO:teuthology.orchestra.run.smithi114.stdout: "num_xlocks": 1, 2024-04-23T10:32:04.337 INFO:teuthology.orchestra.run.smithi114.stdout: "xlock_by": { 2024-04-23T10:32:04.337 INFO:teuthology.orchestra.run.smithi114.stdout: "reqid": { 2024-04-23T10:32:04.337 INFO:teuthology.orchestra.run.smithi114.stdout: "entity": { 2024-04-23T10:32:04.337 INFO:teuthology.orchestra.run.smithi114.stdout: "type": "mds", 2024-04-23T10:32:04.337 INFO:teuthology.orchestra.run.smithi114.stdout: "num": 1 2024-04-23T10:32:04.337 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-23T10:32:04.337 INFO:teuthology.orchestra.run.smithi114.stdout: "tid": 113 2024-04-23T10:32:04.338 INFO:teuthology.orchestra.run.smithi114.stdout: } 2024-04-23T10:32:04.338 INFO:teuthology.orchestra.run.smithi114.stdout: } 2024-04-23T10:32:04.338 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-23T10:32:04.338 INFO:teuthology.orchestra.run.smithi114.stdout: "flags": 4, 2024-04-23T10:32:04.338 INFO:teuthology.orchestra.run.smithi114.stdout: "wrlock_target": -1 2024-04-23T10:32:04.338 INFO:teuthology.orchestra.run.smithi114.stdout: } 2024-04-23T10:32:04.338 INFO:teuthology.orchestra.run.smithi114.stdout: ] 2024-04-23T10:32:04.338 INFO:teuthology.orchestra.run.smithi114.stdout: } 2024-04-23T10:32:04.338 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-23T10:32:04.338 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-04-23T10:32:04.338 INFO:teuthology.orchestra.run.smithi114.stdout: "description": "internal op quiesce_inode:mds.1:145 fp=#0x200000006f1", 2024-04-23T10:32:04.338 INFO:teuthology.orchestra.run.smithi114.stdout: "initiated_at": "2024-04-23T10:31:21.494440+0000", 2024-04-23T10:32:04.338 INFO:teuthology.orchestra.run.smithi114.stdout: "age": 42.363198320000002, 2024-04-23T10:32:04.338 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 42.364471997999999, 2024-04-23T10:32:04.338 INFO:teuthology.orchestra.run.smithi114.stdout: "continuous": true, 2024-04-23T10:32:04.338 INFO:teuthology.orchestra.run.smithi114.stdout: "type_data": { 2024-04-23T10:32:04.339 INFO:teuthology.orchestra.run.smithi114.stdout: "result": -2147483648, 2024-04-23T10:32:04.339 INFO:teuthology.orchestra.run.smithi114.stdout: "flag_point": "failed to authpin, dir is being fragmented", 2024-04-23T10:32:04.339 INFO:teuthology.orchestra.run.smithi114.stdout: "reqid": { 2024-04-23T10:32:04.339 INFO:teuthology.orchestra.run.smithi114.stdout: "entity": { 2024-04-23T10:32:04.339 INFO:teuthology.orchestra.run.smithi114.stdout: "type": "mds", 2024-04-23T10:32:04.339 INFO:teuthology.orchestra.run.smithi114.stdout: "num": 1 2024-04-23T10:32:04.339 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-23T10:32:04.339 INFO:teuthology.orchestra.run.smithi114.stdout: "tid": 145 2024-04-23T10:32:04.339 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-23T10:32:04.339 INFO:teuthology.orchestra.run.smithi114.stdout: "op_type": "internal_op", 2024-04-23T10:32:04.339 INFO:teuthology.orchestra.run.smithi114.stdout: "internal_op": 5385, 2024-04-23T10:32:04.339 INFO:teuthology.orchestra.run.smithi114.stdout: "op_name": "quiesce_inode", 2024-04-23T10:32:04.339 INFO:teuthology.orchestra.run.smithi114.stdout: "events": [ 2024-04-23T10:32:04.339 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-04-23T10:32:04.340 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-04-23T10:31:21.494440+0000", 2024-04-23T10:32:04.340 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "initiated" 2024-04-23T10:32:04.340 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-23T10:32:04.340 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-04-23T10:32:04.340 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-04-23T10:31:21.494440+0000", 2024-04-23T10:32:04.340 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "header_read" 2024-04-23T10:32:04.340 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-23T10:32:04.340 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-04-23T10:32:04.340 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-04-23T10:31:21.494440+0000", 2024-04-23T10:32:04.340 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "throttled" 2024-04-23T10:32:04.340 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-23T10:32:04.340 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-04-23T10:32:04.340 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-04-23T10:31:21.494440+0000", 2024-04-23T10:32:04.340 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "all_read" 2024-04-23T10:32:04.340 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-23T10:32:04.340 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-04-23T10:32:04.340 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-04-23T10:31:21.494440+0000", 2024-04-23T10:32:04.340 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "dispatched" 2024-04-23T10:32:04.341 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-23T10:32:04.341 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-04-23T10:32:04.341 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-04-23T10:31:21.496627+0000", 2024-04-23T10:32:04.341 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "failed to authpin, dir is being fragmented" 2024-04-23T10:32:04.341 INFO:teuthology.orchestra.run.smithi114.stdout: } 2024-04-23T10:32:04.341 INFO:teuthology.orchestra.run.smithi114.stdout: ], 2024-04-23T10:32:04.341 INFO:teuthology.orchestra.run.smithi114.stdout: "locks": [] 2024-04-23T10:32:04.341 INFO:teuthology.orchestra.run.smithi114.stdout: } 2024-04-23T10:32:04.341 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-23T10:32:04.341 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-04-23T10:32:04.341 INFO:teuthology.orchestra.run.smithi114.stdout: "description": "internal op quiesce_inode:mds.1:114 fp=#0x1000000020c", 2024-04-23T10:32:04.341 INFO:teuthology.orchestra.run.smithi114.stdout: "initiated_at": "2024-04-23T10:31:21.464372+0000", 2024-04-23T10:32:04.341 INFO:teuthology.orchestra.run.smithi114.stdout: "age": 42.393267004999998, 2024-04-23T10:32:04.342 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 42.394574124999998, 2024-04-23T10:32:04.342 INFO:teuthology.orchestra.run.smithi114.stdout: "continuous": true, 2024-04-23T10:32:04.342 INFO:teuthology.orchestra.run.smithi114.stdout: "type_data": { 2024-04-23T10:32:04.342 INFO:teuthology.orchestra.run.smithi114.stdout: "result": -2147483648, 2024-04-23T10:32:04.342 INFO:teuthology.orchestra.run.smithi114.stdout: "flag_point": "quiesce complete for non-auth inode", 2024-04-23T10:32:04.342 INFO:teuthology.orchestra.run.smithi114.stdout: "reqid": { 2024-04-23T10:32:04.342 INFO:teuthology.orchestra.run.smithi114.stdout: "entity": { 2024-04-23T10:32:04.342 INFO:teuthology.orchestra.run.smithi114.stdout: "type": "mds", 2024-04-23T10:32:04.342 INFO:teuthology.orchestra.run.smithi114.stdout: "num": 1 2024-04-23T10:32:04.342 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-23T10:32:04.342 INFO:teuthology.orchestra.run.smithi114.stdout: "tid": 114 2024-04-23T10:32:04.342 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-23T10:32:04.342 INFO:teuthology.orchestra.run.smithi114.stdout: "op_type": "internal_op", 2024-04-23T10:32:04.342 INFO:teuthology.orchestra.run.smithi114.stdout: "internal_op": 5385, 2024-04-23T10:32:04.342 INFO:teuthology.orchestra.run.smithi114.stdout: "op_name": "quiesce_inode", 2024-04-23T10:32:04.343 INFO:teuthology.orchestra.run.smithi114.stdout: "events": [ 2024-04-23T10:32:04.343 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-04-23T10:32:04.343 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-04-23T10:31:21.464372+0000", 2024-04-23T10:32:04.343 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "initiated" 2024-04-23T10:32:04.343 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-23T10:32:04.343 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-04-23T10:32:04.343 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-04-23T10:31:21.464372+0000", 2024-04-23T10:32:04.343 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "header_read" 2024-04-23T10:32:04.343 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-23T10:32:04.343 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-04-23T10:32:04.343 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-04-23T10:31:21.464372+0000", 2024-04-23T10:32:04.343 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "throttled" 2024-04-23T10:32:04.343 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-23T10:32:04.343 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-04-23T10:32:04.343 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-04-23T10:31:21.464372+0000", 2024-04-23T10:32:04.343 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "all_read" 2024-04-23T10:32:04.343 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-23T10:32:04.344 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-04-23T10:32:04.344 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-04-23T10:31:21.464372+0000", 2024-04-23T10:32:04.344 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "dispatched" 2024-04-23T10:32:04.344 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-23T10:32:04.344 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-04-23T10:32:04.344 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-04-23T10:31:21.464671+0000", 2024-04-23T10:32:04.344 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "requesting remote authpins" 2024-04-23T10:32:04.344 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-23T10:32:04.344 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-04-23T10:32:04.344 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-04-23T10:31:21.480456+0000", 2024-04-23T10:32:04.344 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "acquired locks" 2024-04-23T10:32:04.344 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-23T10:32:04.344 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-04-23T10:32:04.344 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-04-23T10:31:21.480474+0000", 2024-04-23T10:32:04.344 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "quiesce complete for non-auth inode" 2024-04-23T10:32:04.344 INFO:teuthology.orchestra.run.smithi114.stdout: } 2024-04-23T10:32:04.345 INFO:teuthology.orchestra.run.smithi114.stdout: ], 2024-04-23T10:32:04.345 INFO:teuthology.orchestra.run.smithi114.stdout: "locks": [ 2024-04-23T10:32:04.345 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-04-23T10:32:04.345 INFO:teuthology.orchestra.run.smithi114.stdout: "object": { 2024-04-23T10:32:04.345 INFO:teuthology.orchestra.run.smithi114.stdout: "is_auth": false, 2024-04-23T10:32:04.345 INFO:teuthology.orchestra.run.smithi114.stdout: "auth_state": { 2024-04-23T10:32:04.345 INFO:teuthology.orchestra.run.smithi114.stdout: "replicas": {} 2024-04-23T10:32:04.345 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-23T10:32:04.345 INFO:teuthology.orchestra.run.smithi114.stdout: "replica_state": { 2024-04-23T10:32:04.345 INFO:teuthology.orchestra.run.smithi114.stdout: "authority": [ 2024-04-23T10:32:04.345 INFO:teuthology.orchestra.run.smithi114.stdout: 0, 2024-04-23T10:32:04.345 INFO:teuthology.orchestra.run.smithi114.stdout: -2 2024-04-23T10:32:04.345 INFO:teuthology.orchestra.run.smithi114.stdout: ], 2024-04-23T10:32:04.345 INFO:teuthology.orchestra.run.smithi114.stdout: "replica_nonce": 1 2024-04-23T10:32:04.345 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-23T10:32:04.345 INFO:teuthology.orchestra.run.smithi114.stdout: "auth_pins": 0, 2024-04-23T10:32:04.346 INFO:teuthology.orchestra.run.smithi114.stdout: "is_frozen": false, 2024-04-23T10:32:04.346 INFO:teuthology.orchestra.run.smithi114.stdout: "is_freezing": false, 2024-04-23T10:32:04.346 INFO:teuthology.orchestra.run.smithi114.stdout: "pins": { 2024-04-23T10:32:04.346 INFO:teuthology.orchestra.run.smithi114.stdout: "request": 1, 2024-04-23T10:32:04.346 INFO:teuthology.orchestra.run.smithi114.stdout: "lock": 1, 2024-04-23T10:32:04.346 INFO:teuthology.orchestra.run.smithi114.stdout: "dirwaiter": 0, 2024-04-23T10:32:04.346 INFO:teuthology.orchestra.run.smithi114.stdout: "discoverbase": 0 2024-04-23T10:32:04.346 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-23T10:32:04.346 INFO:teuthology.orchestra.run.smithi114.stdout: "nref": 2 2024-04-23T10:32:04.346 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-23T10:32:04.346 INFO:teuthology.orchestra.run.smithi114.stdout: "object_string": "[inode 0x1000000020c [...7,head] /volumes/qa/sv_1/b206794e-7622-4ab8-8943-f3ae4e65b7a3/client.0/tmp/clients/client0/~dmtmp/EXCEL/ rep@0.1 v45489 f(v0 m2024-04-23T10:31:17.692717+0000 2=2+0) n(v0 rc2024-04-23T10:27:51.972397+0000 3=2+1)/n(v0 rc2024-04-23T10:27:51.967397+0000 2=1+1) (inest mix) (iquiesce lock x=1 by request(mds.1:114 nref=3)) | request=1 lock=1 dirwaiter=0 discoverbase=0 0x55e718b90b00]", 2024-04-23T10:32:04.346 INFO:teuthology.orchestra.run.smithi114.stdout: "lock": { 2024-04-23T10:32:04.346 INFO:teuthology.orchestra.run.smithi114.stdout: "gather_set": [], 2024-04-23T10:32:04.346 INFO:teuthology.orchestra.run.smithi114.stdout: "state": "lock", 2024-04-23T10:32:04.346 INFO:teuthology.orchestra.run.smithi114.stdout: "type": "iquiesce", 2024-04-23T10:32:04.346 INFO:teuthology.orchestra.run.smithi114.stdout: "is_leased": false, 2024-04-23T10:32:04.347 INFO:teuthology.orchestra.run.smithi114.stdout: "num_rdlocks": 0, 2024-04-23T10:32:04.347 INFO:teuthology.orchestra.run.smithi114.stdout: "num_wrlocks": 0, 2024-04-23T10:32:04.347 INFO:teuthology.orchestra.run.smithi114.stdout: "num_xlocks": 1, 2024-04-23T10:32:04.347 INFO:teuthology.orchestra.run.smithi114.stdout: "xlock_by": { 2024-04-23T10:32:04.347 INFO:teuthology.orchestra.run.smithi114.stdout: "reqid": { 2024-04-23T10:32:04.347 INFO:teuthology.orchestra.run.smithi114.stdout: "entity": { 2024-04-23T10:32:04.347 INFO:teuthology.orchestra.run.smithi114.stdout: "type": "mds", 2024-04-23T10:32:04.347 INFO:teuthology.orchestra.run.smithi114.stdout: "num": 1 2024-04-23T10:32:04.347 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-23T10:32:04.347 INFO:teuthology.orchestra.run.smithi114.stdout: "tid": 114 2024-04-23T10:32:04.347 INFO:teuthology.orchestra.run.smithi114.stdout: } 2024-04-23T10:32:04.347 INFO:teuthology.orchestra.run.smithi114.stdout: } 2024-04-23T10:32:04.347 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-23T10:32:04.347 INFO:teuthology.orchestra.run.smithi114.stdout: "flags": 4, 2024-04-23T10:32:04.347 INFO:teuthology.orchestra.run.smithi114.stdout: "wrlock_target": -1 2024-04-23T10:32:04.347 INFO:teuthology.orchestra.run.smithi114.stdout: } 2024-04-23T10:32:04.348 INFO:teuthology.orchestra.run.smithi114.stdout: ] 2024-04-23T10:32:04.348 INFO:teuthology.orchestra.run.smithi114.stdout: } 2024-04-23T10:32:04.348 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-23T10:32:04.348 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-04-23T10:32:04.348 INFO:teuthology.orchestra.run.smithi114.stdout: "description": "internal op quiesce_inode:mds.1:115 fp=#0x10000000208", 2024-04-23T10:32:04.348 INFO:teuthology.orchestra.run.smithi114.stdout: "initiated_at": "2024-04-23T10:31:21.464386+0000", 2024-04-23T10:32:04.348 INFO:teuthology.orchestra.run.smithi114.stdout: "age": 42.393252167999997, 2024-04-23T10:32:04.348 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 42.394623408000001, 2024-04-23T10:32:04.348 INFO:teuthology.orchestra.run.smithi114.stdout: "continuous": true, 2024-04-23T10:32:04.348 INFO:teuthology.orchestra.run.smithi114.stdout: "type_data": { 2024-04-23T10:32:04.348 INFO:teuthology.orchestra.run.smithi114.stdout: "result": -2147483648, 2024-04-23T10:32:04.348 INFO:teuthology.orchestra.run.smithi114.stdout: "flag_point": "quiesce complete for non-auth inode", 2024-04-23T10:32:04.348 INFO:teuthology.orchestra.run.smithi114.stdout: "reqid": { 2024-04-23T10:32:04.348 INFO:teuthology.orchestra.run.smithi114.stdout: "entity": { 2024-04-23T10:32:04.349 INFO:teuthology.orchestra.run.smithi114.stdout: "type": "mds", 2024-04-23T10:32:04.349 INFO:teuthology.orchestra.run.smithi114.stdout: "num": 1 2024-04-23T10:32:04.349 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-23T10:32:04.349 INFO:teuthology.orchestra.run.smithi114.stdout: "tid": 115 2024-04-23T10:32:04.349 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-23T10:32:04.349 INFO:teuthology.orchestra.run.smithi114.stdout: "op_type": "internal_op", 2024-04-23T10:32:04.349 INFO:teuthology.orchestra.run.smithi114.stdout: "internal_op": 5385, 2024-04-23T10:32:04.349 INFO:teuthology.orchestra.run.smithi114.stdout: "op_name": "quiesce_inode", 2024-04-23T10:32:04.349 INFO:teuthology.orchestra.run.smithi114.stdout: "events": [ 2024-04-23T10:32:04.349 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-04-23T10:32:04.349 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-04-23T10:31:21.464386+0000", 2024-04-23T10:32:04.349 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "initiated" 2024-04-23T10:32:04.349 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-23T10:32:04.349 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-04-23T10:32:04.349 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-04-23T10:31:21.464386+0000", 2024-04-23T10:32:04.349 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "header_read" 2024-04-23T10:32:04.350 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-23T10:32:04.350 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-04-23T10:32:04.350 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-04-23T10:31:21.464386+0000", 2024-04-23T10:32:04.350 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "throttled" 2024-04-23T10:32:04.350 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-23T10:32:04.350 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-04-23T10:32:04.350 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-04-23T10:31:21.464386+0000", 2024-04-23T10:32:04.350 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "all_read" 2024-04-23T10:32:04.350 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-23T10:32:04.350 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-04-23T10:32:04.350 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-04-23T10:31:21.464386+0000", 2024-04-23T10:32:04.350 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "dispatched" 2024-04-23T10:32:04.350 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-23T10:32:04.350 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-04-23T10:32:04.351 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-04-23T10:31:21.464740+0000", 2024-04-23T10:32:04.351 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "requesting remote authpins" 2024-04-23T10:32:04.351 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-23T10:32:04.351 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-04-23T10:32:04.351 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-04-23T10:31:21.484765+0000", 2024-04-23T10:32:04.351 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "acquired locks" 2024-04-23T10:32:04.351 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-23T10:32:04.351 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-04-23T10:32:04.351 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-04-23T10:31:21.484777+0000", 2024-04-23T10:32:04.351 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "quiesce complete for non-auth inode" 2024-04-23T10:32:04.351 INFO:teuthology.orchestra.run.smithi114.stdout: } 2024-04-23T10:32:04.351 INFO:teuthology.orchestra.run.smithi114.stdout: ], 2024-04-23T10:32:04.351 INFO:teuthology.orchestra.run.smithi114.stdout: "locks": [ 2024-04-23T10:32:04.351 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-04-23T10:32:04.351 INFO:teuthology.orchestra.run.smithi114.stdout: "object": { 2024-04-23T10:32:04.352 INFO:teuthology.orchestra.run.smithi114.stdout: "is_auth": false, 2024-04-23T10:32:04.352 INFO:teuthology.orchestra.run.smithi114.stdout: "auth_state": { 2024-04-23T10:32:04.352 INFO:teuthology.orchestra.run.smithi114.stdout: "replicas": {} 2024-04-23T10:32:04.352 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-23T10:32:04.352 INFO:teuthology.orchestra.run.smithi114.stdout: "replica_state": { 2024-04-23T10:32:04.352 INFO:teuthology.orchestra.run.smithi114.stdout: "authority": [ 2024-04-23T10:32:04.352 INFO:teuthology.orchestra.run.smithi114.stdout: 0, 2024-04-23T10:32:04.352 INFO:teuthology.orchestra.run.smithi114.stdout: -2 2024-04-23T10:32:04.352 INFO:teuthology.orchestra.run.smithi114.stdout: ], 2024-04-23T10:32:04.352 INFO:teuthology.orchestra.run.smithi114.stdout: "replica_nonce": 1 2024-04-23T10:32:04.352 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-23T10:32:04.352 INFO:teuthology.orchestra.run.smithi114.stdout: "auth_pins": 0, 2024-04-23T10:32:04.352 INFO:teuthology.orchestra.run.smithi114.stdout: "is_frozen": false, 2024-04-23T10:32:04.352 INFO:teuthology.orchestra.run.smithi114.stdout: "is_freezing": false, 2024-04-23T10:32:04.353 INFO:teuthology.orchestra.run.smithi114.stdout: "pins": { 2024-04-23T10:32:04.353 INFO:teuthology.orchestra.run.smithi114.stdout: "request": 1, 2024-04-23T10:32:04.353 INFO:teuthology.orchestra.run.smithi114.stdout: "lock": 1, 2024-04-23T10:32:04.353 INFO:teuthology.orchestra.run.smithi114.stdout: "dirwaiter": 0, 2024-04-23T10:32:04.353 INFO:teuthology.orchestra.run.smithi114.stdout: "discoverbase": 0 2024-04-23T10:32:04.353 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-23T10:32:04.353 INFO:teuthology.orchestra.run.smithi114.stdout: "nref": 2 2024-04-23T10:32:04.353 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-23T10:32:04.353 INFO:teuthology.orchestra.run.smithi114.stdout: "object_string": "[inode 0x10000000208 [...7,head] /volumes/qa/sv_1/b206794e-7622-4ab8-8943-f3ae4e65b7a3/client.0/tmp/clients/client0/~dmtmp/WORD/ rep@0.1 v45514 f(v0 m2024-04-23T10:31:17.675718+0000 3=3+0) n(v0 rc2024-04-23T10:27:51.958398+0000 3=2+1)/n(v0 rc2024-04-23T10:27:51.954398+0000 2=1+1) (inest mix) (iquiesce lock x=1 by request(mds.1:115 nref=3)) | request=1 lock=1 dirwaiter=0 discoverbase=0 0x55e718b90580]", 2024-04-23T10:32:04.353 INFO:teuthology.orchestra.run.smithi114.stdout: "lock": { 2024-04-23T10:32:04.353 INFO:teuthology.orchestra.run.smithi114.stdout: "gather_set": [], 2024-04-23T10:32:04.353 INFO:teuthology.orchestra.run.smithi114.stdout: "state": "lock", 2024-04-23T10:32:04.353 INFO:teuthology.orchestra.run.smithi114.stdout: "type": "iquiesce", 2024-04-23T10:32:04.353 INFO:teuthology.orchestra.run.smithi114.stdout: "is_leased": false, 2024-04-23T10:32:04.353 INFO:teuthology.orchestra.run.smithi114.stdout: "num_rdlocks": 0, 2024-04-23T10:32:04.353 INFO:teuthology.orchestra.run.smithi114.stdout: "num_wrlocks": 0, 2024-04-23T10:32:04.354 INFO:teuthology.orchestra.run.smithi114.stdout: "num_xlocks": 1, 2024-04-23T10:32:04.354 INFO:teuthology.orchestra.run.smithi114.stdout: "xlock_by": { 2024-04-23T10:32:04.354 INFO:teuthology.orchestra.run.smithi114.stdout: "reqid": { 2024-04-23T10:32:04.354 INFO:teuthology.orchestra.run.smithi114.stdout: "entity": { 2024-04-23T10:32:04.354 INFO:teuthology.orchestra.run.smithi114.stdout: "type": "mds", 2024-04-23T10:32:04.354 INFO:teuthology.orchestra.run.smithi114.stdout: "num": 1 2024-04-23T10:32:04.354 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-23T10:32:04.354 INFO:teuthology.orchestra.run.smithi114.stdout: "tid": 115 2024-04-23T10:32:04.354 INFO:teuthology.orchestra.run.smithi114.stdout: } 2024-04-23T10:32:04.354 INFO:teuthology.orchestra.run.smithi114.stdout: } 2024-04-23T10:32:04.354 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-23T10:32:04.354 INFO:teuthology.orchestra.run.smithi114.stdout: "flags": 4, 2024-04-23T10:32:04.354 INFO:teuthology.orchestra.run.smithi114.stdout: "wrlock_target": -1 2024-04-23T10:32:04.355 INFO:teuthology.orchestra.run.smithi114.stdout: } 2024-04-23T10:32:04.355 INFO:teuthology.orchestra.run.smithi114.stdout: ] 2024-04-23T10:32:04.355 INFO:teuthology.orchestra.run.smithi114.stdout: } 2024-04-23T10:32:04.355 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-23T10:32:04.355 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-04-23T10:32:04.355 INFO:teuthology.orchestra.run.smithi114.stdout: "description": "internal op quiesce_inode:mds.1:116 fp=#0x10000000240", 2024-04-23T10:32:04.355 INFO:teuthology.orchestra.run.smithi114.stdout: "initiated_at": "2024-04-23T10:31:21.464402+0000", 2024-04-23T10:32:04.355 INFO:teuthology.orchestra.run.smithi114.stdout: "age": 42.393236377000001, 2024-04-23T10:32:04.355 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 42.394668606000003, 2024-04-23T10:32:04.355 INFO:teuthology.orchestra.run.smithi114.stdout: "continuous": true, 2024-04-23T10:32:04.356 INFO:teuthology.orchestra.run.smithi114.stdout: "type_data": { 2024-04-23T10:32:04.356 INFO:teuthology.orchestra.run.smithi114.stdout: "result": -2147483648, 2024-04-23T10:32:04.356 INFO:teuthology.orchestra.run.smithi114.stdout: "flag_point": "quiesce complete for non-auth inode", 2024-04-23T10:32:04.356 INFO:teuthology.orchestra.run.smithi114.stdout: "reqid": { 2024-04-23T10:32:04.356 INFO:teuthology.orchestra.run.smithi114.stdout: "entity": { 2024-04-23T10:32:04.356 INFO:teuthology.orchestra.run.smithi114.stdout: "type": "mds", 2024-04-23T10:32:04.356 INFO:teuthology.orchestra.run.smithi114.stdout: "num": 1 2024-04-23T10:32:04.356 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-23T10:32:04.356 INFO:teuthology.orchestra.run.smithi114.stdout: "tid": 116 2024-04-23T10:32:04.357 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-23T10:32:04.357 INFO:teuthology.orchestra.run.smithi114.stdout: "op_type": "internal_op", 2024-04-23T10:32:04.357 INFO:teuthology.orchestra.run.smithi114.stdout: "internal_op": 5385, 2024-04-23T10:32:04.357 INFO:teuthology.orchestra.run.smithi114.stdout: "op_name": "quiesce_inode", 2024-04-23T10:32:04.357 INFO:teuthology.orchestra.run.smithi114.stdout: "events": [ 2024-04-23T10:32:04.357 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-04-23T10:32:04.357 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-04-23T10:31:21.464402+0000", 2024-04-23T10:32:04.357 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "initiated" 2024-04-23T10:32:04.357 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-23T10:32:04.358 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-04-23T10:32:04.358 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-04-23T10:31:21.464402+0000", 2024-04-23T10:32:04.358 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "header_read" 2024-04-23T10:32:04.358 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-23T10:32:04.358 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-04-23T10:32:04.358 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-04-23T10:31:21.464402+0000", 2024-04-23T10:32:04.358 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "throttled" 2024-04-23T10:32:04.358 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-23T10:32:04.358 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-04-23T10:32:04.358 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-04-23T10:31:21.464402+0000", 2024-04-23T10:32:04.358 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "all_read" 2024-04-23T10:32:04.358 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-23T10:32:04.358 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-04-23T10:32:04.359 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-04-23T10:31:21.464402+0000", 2024-04-23T10:32:04.359 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "dispatched" 2024-04-23T10:32:04.359 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-23T10:32:04.359 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-04-23T10:32:04.359 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-04-23T10:31:21.464820+0000", 2024-04-23T10:32:04.359 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "requesting remote authpins" 2024-04-23T10:32:04.359 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-23T10:32:04.359 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-04-23T10:32:04.359 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-04-23T10:31:21.488980+0000", 2024-04-23T10:32:04.359 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "acquired locks" 2024-04-23T10:32:04.359 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-23T10:32:04.359 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-04-23T10:32:04.360 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-04-23T10:31:21.488992+0000", 2024-04-23T10:32:04.360 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "quiesce complete for non-auth inode" 2024-04-23T10:32:04.360 INFO:teuthology.orchestra.run.smithi114.stdout: } 2024-04-23T10:32:04.360 INFO:teuthology.orchestra.run.smithi114.stdout: ], 2024-04-23T10:32:04.360 INFO:teuthology.orchestra.run.smithi114.stdout: "locks": [ 2024-04-23T10:32:04.360 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-04-23T10:32:04.360 INFO:teuthology.orchestra.run.smithi114.stdout: "object": { 2024-04-23T10:32:04.360 INFO:teuthology.orchestra.run.smithi114.stdout: "is_auth": false, 2024-04-23T10:32:04.360 INFO:teuthology.orchestra.run.smithi114.stdout: "auth_state": { 2024-04-23T10:32:04.360 INFO:teuthology.orchestra.run.smithi114.stdout: "replicas": {} 2024-04-23T10:32:04.360 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-23T10:32:04.361 INFO:teuthology.orchestra.run.smithi114.stdout: "replica_state": { 2024-04-23T10:32:04.361 INFO:teuthology.orchestra.run.smithi114.stdout: "authority": [ 2024-04-23T10:32:04.361 INFO:teuthology.orchestra.run.smithi114.stdout: 0, 2024-04-23T10:32:04.361 INFO:teuthology.orchestra.run.smithi114.stdout: -2 2024-04-23T10:32:04.361 INFO:teuthology.orchestra.run.smithi114.stdout: ], 2024-04-23T10:32:04.361 INFO:teuthology.orchestra.run.smithi114.stdout: "replica_nonce": 1 2024-04-23T10:32:04.361 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-23T10:32:04.361 INFO:teuthology.orchestra.run.smithi114.stdout: "auth_pins": 0, 2024-04-23T10:32:04.361 INFO:teuthology.orchestra.run.smithi114.stdout: "is_frozen": false, 2024-04-23T10:32:04.361 INFO:teuthology.orchestra.run.smithi114.stdout: "is_freezing": false, 2024-04-23T10:32:04.361 INFO:teuthology.orchestra.run.smithi114.stdout: "pins": { 2024-04-23T10:32:04.361 INFO:teuthology.orchestra.run.smithi114.stdout: "request": 1, 2024-04-23T10:32:04.362 INFO:teuthology.orchestra.run.smithi114.stdout: "lock": 1, 2024-04-23T10:32:04.362 INFO:teuthology.orchestra.run.smithi114.stdout: "dirfrag": 0, 2024-04-23T10:32:04.362 INFO:teuthology.orchestra.run.smithi114.stdout: "dirwaiter": 0, 2024-04-23T10:32:04.362 INFO:teuthology.orchestra.run.smithi114.stdout: "discoverbase": 0 2024-04-23T10:32:04.362 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-23T10:32:04.362 INFO:teuthology.orchestra.run.smithi114.stdout: "nref": 2 2024-04-23T10:32:04.362 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-23T10:32:04.362 INFO:teuthology.orchestra.run.smithi114.stdout: "object_string": "[inode 0x10000000240 [...7,head] /volumes/qa/sv_1/b206794e-7622-4ab8-8943-f3ae4e65b7a3/client.0/tmp/clients/client0/~dmtmp/SEED/ rep@0.1 v45361 f(v0 m2024-04-23T10:27:52.196392+0000 3=3+0) n(v2 rc2024-04-23T10:28:22.485704+0000 b3244032 4=3+1)/n(v0 rc2024-04-23T10:27:52.187392+0000 2=1+1) (inest mix) (iquiesce lock x=1 by request(mds.1:116 nref=3)) | request=1 lock=1 dirfrag=0 dirwaiter=0 discoverbase=0 0x55e718b92c00]", 2024-04-23T10:32:04.362 INFO:teuthology.orchestra.run.smithi114.stdout: "lock": { 2024-04-23T10:32:04.362 INFO:teuthology.orchestra.run.smithi114.stdout: "gather_set": [], 2024-04-23T10:32:04.362 INFO:teuthology.orchestra.run.smithi114.stdout: "state": "lock", 2024-04-23T10:32:04.363 INFO:teuthology.orchestra.run.smithi114.stdout: "type": "iquiesce", 2024-04-23T10:32:04.363 INFO:teuthology.orchestra.run.smithi114.stdout: "is_leased": false, 2024-04-23T10:32:04.363 INFO:teuthology.orchestra.run.smithi114.stdout: "num_rdlocks": 0, 2024-04-23T10:32:04.363 INFO:teuthology.orchestra.run.smithi114.stdout: "num_wrlocks": 0, 2024-04-23T10:32:04.363 INFO:teuthology.orchestra.run.smithi114.stdout: "num_xlocks": 1, 2024-04-23T10:32:04.363 INFO:teuthology.orchestra.run.smithi114.stdout: "xlock_by": { 2024-04-23T10:32:04.363 INFO:teuthology.orchestra.run.smithi114.stdout: "reqid": { 2024-04-23T10:32:04.363 INFO:teuthology.orchestra.run.smithi114.stdout: "entity": { 2024-04-23T10:32:04.363 INFO:teuthology.orchestra.run.smithi114.stdout: "type": "mds", 2024-04-23T10:32:04.363 INFO:teuthology.orchestra.run.smithi114.stdout: "num": 1 2024-04-23T10:32:04.363 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-23T10:32:04.363 INFO:teuthology.orchestra.run.smithi114.stdout: "tid": 116 2024-04-23T10:32:04.363 INFO:teuthology.orchestra.run.smithi114.stdout: } 2024-04-23T10:32:04.364 INFO:teuthology.orchestra.run.smithi114.stdout: } 2024-04-23T10:32:04.364 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-23T10:32:04.364 INFO:teuthology.orchestra.run.smithi114.stdout: "flags": 4, 2024-04-23T10:32:04.364 INFO:teuthology.orchestra.run.smithi114.stdout: "wrlock_target": -1 2024-04-23T10:32:04.364 INFO:teuthology.orchestra.run.smithi114.stdout: } 2024-04-23T10:32:04.364 INFO:teuthology.orchestra.run.smithi114.stdout: ] 2024-04-23T10:32:04.364 INFO:teuthology.orchestra.run.smithi114.stdout: } 2024-04-23T10:32:04.364 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-23T10:32:04.364 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-04-23T10:32:04.364 INFO:teuthology.orchestra.run.smithi114.stdout: "description": "internal op quiesce_inode:mds.1:117 fp=#0x10000000221", 2024-04-23T10:32:04.364 INFO:teuthology.orchestra.run.smithi114.stdout: "initiated_at": "2024-04-23T10:31:21.464418+0000", 2024-04-23T10:32:04.364 INFO:teuthology.orchestra.run.smithi114.stdout: "age": 42.393221013999998, 2024-04-23T10:32:04.364 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 42.394714657999998, 2024-04-23T10:32:04.364 INFO:teuthology.orchestra.run.smithi114.stdout: "continuous": true, 2024-04-23T10:32:04.364 INFO:teuthology.orchestra.run.smithi114.stdout: "type_data": { 2024-04-23T10:32:04.364 INFO:teuthology.orchestra.run.smithi114.stdout: "result": -2147483648, 2024-04-23T10:32:04.365 INFO:teuthology.orchestra.run.smithi114.stdout: "flag_point": "quiesce complete for non-auth inode", 2024-04-23T10:32:04.365 INFO:teuthology.orchestra.run.smithi114.stdout: "reqid": { 2024-04-23T10:32:04.365 INFO:teuthology.orchestra.run.smithi114.stdout: "entity": { 2024-04-23T10:32:04.365 INFO:teuthology.orchestra.run.smithi114.stdout: "type": "mds", 2024-04-23T10:32:04.365 INFO:teuthology.orchestra.run.smithi114.stdout: "num": 1 2024-04-23T10:32:04.365 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-23T10:32:04.365 INFO:teuthology.orchestra.run.smithi114.stdout: "tid": 117 2024-04-23T10:32:04.365 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-23T10:32:04.365 INFO:teuthology.orchestra.run.smithi114.stdout: "op_type": "internal_op", 2024-04-23T10:32:04.365 INFO:teuthology.orchestra.run.smithi114.stdout: "internal_op": 5385, 2024-04-23T10:32:04.365 INFO:teuthology.orchestra.run.smithi114.stdout: "op_name": "quiesce_inode", 2024-04-23T10:32:04.365 INFO:teuthology.orchestra.run.smithi114.stdout: "events": [ 2024-04-23T10:32:04.365 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-04-23T10:32:04.365 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-04-23T10:31:21.464418+0000", 2024-04-23T10:32:04.365 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "initiated" 2024-04-23T10:32:04.365 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-23T10:32:04.366 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-04-23T10:32:04.366 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-04-23T10:31:21.464418+0000", 2024-04-23T10:32:04.366 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "header_read" 2024-04-23T10:32:04.366 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-23T10:32:04.366 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-04-23T10:32:04.366 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-04-23T10:31:21.464418+0000", 2024-04-23T10:32:04.366 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "throttled" 2024-04-23T10:32:04.366 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-23T10:32:04.366 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-04-23T10:32:04.366 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-04-23T10:31:21.464418+0000", 2024-04-23T10:32:04.366 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "all_read" 2024-04-23T10:32:04.366 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-23T10:32:04.366 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-04-23T10:32:04.366 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-04-23T10:31:21.464418+0000", 2024-04-23T10:32:04.367 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "dispatched" 2024-04-23T10:32:04.367 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-23T10:32:04.367 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-04-23T10:32:04.367 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-04-23T10:31:21.464887+0000", 2024-04-23T10:32:04.367 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "requesting remote authpins" 2024-04-23T10:32:04.367 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-23T10:32:04.367 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-04-23T10:32:04.367 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-04-23T10:31:21.493110+0000", 2024-04-23T10:32:04.367 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "acquired locks" 2024-04-23T10:32:04.367 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-23T10:32:04.367 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-04-23T10:32:04.367 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-04-23T10:31:21.493124+0000", 2024-04-23T10:32:04.367 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "quiesce complete for non-auth inode" 2024-04-23T10:32:04.367 INFO:teuthology.orchestra.run.smithi114.stdout: } 2024-04-23T10:32:04.367 INFO:teuthology.orchestra.run.smithi114.stdout: ], 2024-04-23T10:32:04.367 INFO:teuthology.orchestra.run.smithi114.stdout: "locks": [ 2024-04-23T10:32:04.367 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-04-23T10:32:04.368 INFO:teuthology.orchestra.run.smithi114.stdout: "object": { 2024-04-23T10:32:04.368 INFO:teuthology.orchestra.run.smithi114.stdout: "is_auth": false, 2024-04-23T10:32:04.368 INFO:teuthology.orchestra.run.smithi114.stdout: "auth_state": { 2024-04-23T10:32:04.368 INFO:teuthology.orchestra.run.smithi114.stdout: "replicas": {} 2024-04-23T10:32:04.368 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-23T10:32:04.368 INFO:teuthology.orchestra.run.smithi114.stdout: "replica_state": { 2024-04-23T10:32:04.368 INFO:teuthology.orchestra.run.smithi114.stdout: "authority": [ 2024-04-23T10:32:04.368 INFO:teuthology.orchestra.run.smithi114.stdout: 0, 2024-04-23T10:32:04.368 INFO:teuthology.orchestra.run.smithi114.stdout: -2 2024-04-23T10:32:04.368 INFO:teuthology.orchestra.run.smithi114.stdout: ], 2024-04-23T10:32:04.368 INFO:teuthology.orchestra.run.smithi114.stdout: "replica_nonce": 1 2024-04-23T10:32:04.368 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-23T10:32:04.368 INFO:teuthology.orchestra.run.smithi114.stdout: "auth_pins": 0, 2024-04-23T10:32:04.368 INFO:teuthology.orchestra.run.smithi114.stdout: "is_frozen": false, 2024-04-23T10:32:04.368 INFO:teuthology.orchestra.run.smithi114.stdout: "is_freezing": false, 2024-04-23T10:32:04.368 INFO:teuthology.orchestra.run.smithi114.stdout: "pins": { 2024-04-23T10:32:04.369 INFO:teuthology.orchestra.run.smithi114.stdout: "request": 1, 2024-04-23T10:32:04.369 INFO:teuthology.orchestra.run.smithi114.stdout: "lock": 1, 2024-04-23T10:32:04.369 INFO:teuthology.orchestra.run.smithi114.stdout: "dirwaiter": 0, 2024-04-23T10:32:04.369 INFO:teuthology.orchestra.run.smithi114.stdout: "discoverbase": 0 2024-04-23T10:32:04.369 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-23T10:32:04.369 INFO:teuthology.orchestra.run.smithi114.stdout: "nref": 2 2024-04-23T10:32:04.369 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-23T10:32:04.369 INFO:teuthology.orchestra.run.smithi114.stdout: "object_string": "[inode 0x10000000221 [...7,head] /volumes/qa/sv_1/b206794e-7622-4ab8-8943-f3ae4e65b7a3/client.0/tmp/clients/client0/~dmtmp/WORDPRO/ rep@0.1 v45516 f(v0 m2024-04-23T10:31:17.760716+0000 3=3+0) n(v0 rc2024-04-23T10:27:52.061395+0000 3=2+1)/n(v0 rc2024-04-23T10:27:52.057395+0000 2=1+1) (inest mix) (iquiesce lock x=1 by request(mds.1:117 nref=3)) | request=1 lock=1 dirwaiter=0 discoverbase=0 0x55e718b91b80]", 2024-04-23T10:32:04.369 INFO:teuthology.orchestra.run.smithi114.stdout: "lock": { 2024-04-23T10:32:04.369 INFO:teuthology.orchestra.run.smithi114.stdout: "gather_set": [], 2024-04-23T10:32:04.369 INFO:teuthology.orchestra.run.smithi114.stdout: "state": "lock", 2024-04-23T10:32:04.369 INFO:teuthology.orchestra.run.smithi114.stdout: "type": "iquiesce", 2024-04-23T10:32:04.369 INFO:teuthology.orchestra.run.smithi114.stdout: "is_leased": false, 2024-04-23T10:32:04.369 INFO:teuthology.orchestra.run.smithi114.stdout: "num_rdlocks": 0, 2024-04-23T10:32:04.369 INFO:teuthology.orchestra.run.smithi114.stdout: "num_wrlocks": 0, 2024-04-23T10:32:04.370 INFO:teuthology.orchestra.run.smithi114.stdout: "num_xlocks": 1, 2024-04-23T10:32:04.370 INFO:teuthology.orchestra.run.smithi114.stdout: "xlock_by": { 2024-04-23T10:32:04.370 INFO:teuthology.orchestra.run.smithi114.stdout: "reqid": { 2024-04-23T10:32:04.370 INFO:teuthology.orchestra.run.smithi114.stdout: "entity": { 2024-04-23T10:32:04.370 INFO:teuthology.orchestra.run.smithi114.stdout: "type": "mds", 2024-04-23T10:32:04.370 INFO:teuthology.orchestra.run.smithi114.stdout: "num": 1 2024-04-23T10:32:04.370 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-23T10:32:04.370 INFO:teuthology.orchestra.run.smithi114.stdout: "tid": 117 2024-04-23T10:32:04.370 INFO:teuthology.orchestra.run.smithi114.stdout: } 2024-04-23T10:32:04.370 INFO:teuthology.orchestra.run.smithi114.stdout: } 2024-04-23T10:32:04.370 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-23T10:32:04.370 INFO:teuthology.orchestra.run.smithi114.stdout: "flags": 4, 2024-04-23T10:32:04.370 INFO:teuthology.orchestra.run.smithi114.stdout: "wrlock_target": -1 2024-04-23T10:32:04.370 INFO:teuthology.orchestra.run.smithi114.stdout: } 2024-04-23T10:32:04.370 INFO:teuthology.orchestra.run.smithi114.stdout: ] 2024-04-23T10:32:04.371 INFO:teuthology.orchestra.run.smithi114.stdout: } 2024-04-23T10:32:04.371 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-23T10:32:04.371 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-04-23T10:32:04.371 INFO:teuthology.orchestra.run.smithi114.stdout: "description": "internal op quiesce_inode:mds.1:118 fp=#0x10000000229", 2024-04-23T10:32:04.371 INFO:teuthology.orchestra.run.smithi114.stdout: "initiated_at": "2024-04-23T10:31:21.464436+0000", 2024-04-23T10:32:04.371 INFO:teuthology.orchestra.run.smithi114.stdout: "age": 42.393202465999998, 2024-04-23T10:32:04.371 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 42.394761938000002, 2024-04-23T10:32:04.371 INFO:teuthology.orchestra.run.smithi114.stdout: "continuous": true, 2024-04-23T10:32:04.371 INFO:teuthology.orchestra.run.smithi114.stdout: "type_data": { 2024-04-23T10:32:04.371 INFO:teuthology.orchestra.run.smithi114.stdout: "result": -2147483648, 2024-04-23T10:32:04.371 INFO:teuthology.orchestra.run.smithi114.stdout: "flag_point": "acquired locks", 2024-04-23T10:32:04.371 INFO:teuthology.orchestra.run.smithi114.stdout: "reqid": { 2024-04-23T10:32:04.371 INFO:teuthology.orchestra.run.smithi114.stdout: "entity": { 2024-04-23T10:32:04.371 INFO:teuthology.orchestra.run.smithi114.stdout: "type": "mds", 2024-04-23T10:32:04.371 INFO:teuthology.orchestra.run.smithi114.stdout: "num": 1 2024-04-23T10:32:04.372 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-23T10:32:04.372 INFO:teuthology.orchestra.run.smithi114.stdout: "tid": 118 2024-04-23T10:32:04.372 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-23T10:32:04.372 INFO:teuthology.orchestra.run.smithi114.stdout: "op_type": "internal_op", 2024-04-23T10:32:04.372 INFO:teuthology.orchestra.run.smithi114.stdout: "internal_op": 5385, 2024-04-23T10:32:04.372 INFO:teuthology.orchestra.run.smithi114.stdout: "op_name": "quiesce_inode", 2024-04-23T10:32:04.372 INFO:teuthology.orchestra.run.smithi114.stdout: "events": [ 2024-04-23T10:32:04.372 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-04-23T10:32:04.372 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-04-23T10:31:21.464436+0000", 2024-04-23T10:32:04.372 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "initiated" 2024-04-23T10:32:04.372 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-23T10:32:04.372 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-04-23T10:32:04.372 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-04-23T10:31:21.464436+0000", 2024-04-23T10:32:04.372 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "header_read" 2024-04-23T10:32:04.372 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-23T10:32:04.373 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-04-23T10:32:04.373 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-04-23T10:31:21.464436+0000", 2024-04-23T10:32:04.373 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "throttled" 2024-04-23T10:32:04.373 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-23T10:32:04.373 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-04-23T10:32:04.373 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-04-23T10:31:21.464436+0000", 2024-04-23T10:32:04.373 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "all_read" 2024-04-23T10:32:04.373 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-23T10:32:04.373 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-04-23T10:32:04.373 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-04-23T10:31:21.464436+0000", 2024-04-23T10:32:04.373 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "dispatched" 2024-04-23T10:32:04.373 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-23T10:32:04.373 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-04-23T10:32:04.373 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-04-23T10:31:21.464971+0000", 2024-04-23T10:32:04.373 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "requesting remote authpins" 2024-04-23T10:32:04.374 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-23T10:32:04.374 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-04-23T10:32:04.374 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-04-23T10:31:21.494019+0000", 2024-04-23T10:32:04.374 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "acquired locks" 2024-04-23T10:32:04.374 INFO:teuthology.orchestra.run.smithi114.stdout: } 2024-04-23T10:32:04.374 INFO:teuthology.orchestra.run.smithi114.stdout: ], 2024-04-23T10:32:04.374 INFO:teuthology.orchestra.run.smithi114.stdout: "locks": [ 2024-04-23T10:32:04.374 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-04-23T10:32:04.374 INFO:teuthology.orchestra.run.smithi114.stdout: "object": { 2024-04-23T10:32:04.374 INFO:teuthology.orchestra.run.smithi114.stdout: "is_auth": false, 2024-04-23T10:32:04.374 INFO:teuthology.orchestra.run.smithi114.stdout: "auth_state": { 2024-04-23T10:32:04.374 INFO:teuthology.orchestra.run.smithi114.stdout: "replicas": {} 2024-04-23T10:32:04.374 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-23T10:32:04.374 INFO:teuthology.orchestra.run.smithi114.stdout: "replica_state": { 2024-04-23T10:32:04.374 INFO:teuthology.orchestra.run.smithi114.stdout: "authority": [ 2024-04-23T10:32:04.375 INFO:teuthology.orchestra.run.smithi114.stdout: 0, 2024-04-23T10:32:04.375 INFO:teuthology.orchestra.run.smithi114.stdout: -2 2024-04-23T10:32:04.375 INFO:teuthology.orchestra.run.smithi114.stdout: ], 2024-04-23T10:32:04.375 INFO:teuthology.orchestra.run.smithi114.stdout: "replica_nonce": 1 2024-04-23T10:32:04.375 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-23T10:32:04.375 INFO:teuthology.orchestra.run.smithi114.stdout: "auth_pins": 0, 2024-04-23T10:32:04.375 INFO:teuthology.orchestra.run.smithi114.stdout: "is_frozen": false, 2024-04-23T10:32:04.375 INFO:teuthology.orchestra.run.smithi114.stdout: "is_freezing": false, 2024-04-23T10:32:04.375 INFO:teuthology.orchestra.run.smithi114.stdout: "pins": { 2024-04-23T10:32:04.375 INFO:teuthology.orchestra.run.smithi114.stdout: "dirtyscattered": 0, 2024-04-23T10:32:04.375 INFO:teuthology.orchestra.run.smithi114.stdout: "request": 2, 2024-04-23T10:32:04.375 INFO:teuthology.orchestra.run.smithi114.stdout: "lock": 1, 2024-04-23T10:32:04.375 INFO:teuthology.orchestra.run.smithi114.stdout: "importing": 0, 2024-04-23T10:32:04.375 INFO:teuthology.orchestra.run.smithi114.stdout: "dirfrag": 1, 2024-04-23T10:32:04.375 INFO:teuthology.orchestra.run.smithi114.stdout: "caps": 1, 2024-04-23T10:32:04.375 INFO:teuthology.orchestra.run.smithi114.stdout: "dirwaiter": 0, 2024-04-23T10:32:04.376 INFO:teuthology.orchestra.run.smithi114.stdout: "waiter": 1, 2024-04-23T10:32:04.376 INFO:teuthology.orchestra.run.smithi114.stdout: "discoverbase": 0 2024-04-23T10:32:04.376 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-23T10:32:04.376 INFO:teuthology.orchestra.run.smithi114.stdout: "nref": 6 2024-04-23T10:32:04.376 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-23T10:32:04.376 INFO:teuthology.orchestra.run.smithi114.stdout: "object_string": "[inode 0x10000000229 [...7,head] /volumes/qa/sv_1/b206794e-7622-4ab8-8943-f3ae4e65b7a3/client.0/tmp/clients/client0/~dmtmp/PARADOX/ rep@0.1 v45567 f(v1976 m2024-04-23T10:31:16.541744+0000 24=24+0) n(v19 rc2024-04-23T10:31:16.541744+0000 b5201920 25=24+1)/n(v0 rc2024-04-23T10:27:52.084395+0000 2=1+1) (inest mix) (iquiesce lock x=1 by request(mds.1:118 nref=4)) caps={15312=pAsLsXsFs/pAsLsXsFsxcral@10631} | dirtyscattered=0 request=2 lock=1 importing=0 dirfrag=1 caps=1 dirwaiter=0 waiter=1 discoverbase=0 0x55e718b92680]", 2024-04-23T10:32:04.376 INFO:teuthology.orchestra.run.smithi114.stdout: "lock": { 2024-04-23T10:32:04.376 INFO:teuthology.orchestra.run.smithi114.stdout: "gather_set": [], 2024-04-23T10:32:04.376 INFO:teuthology.orchestra.run.smithi114.stdout: "state": "lock", 2024-04-23T10:32:04.376 INFO:teuthology.orchestra.run.smithi114.stdout: "type": "iquiesce", 2024-04-23T10:32:04.376 INFO:teuthology.orchestra.run.smithi114.stdout: "is_leased": false, 2024-04-23T10:32:04.376 INFO:teuthology.orchestra.run.smithi114.stdout: "num_rdlocks": 0, 2024-04-23T10:32:04.376 INFO:teuthology.orchestra.run.smithi114.stdout: "num_wrlocks": 0, 2024-04-23T10:32:04.376 INFO:teuthology.orchestra.run.smithi114.stdout: "num_xlocks": 1, 2024-04-23T10:32:04.376 INFO:teuthology.orchestra.run.smithi114.stdout: "xlock_by": { 2024-04-23T10:32:04.376 INFO:teuthology.orchestra.run.smithi114.stdout: "reqid": { 2024-04-23T10:32:04.377 INFO:teuthology.orchestra.run.smithi114.stdout: "entity": { 2024-04-23T10:32:04.377 INFO:teuthology.orchestra.run.smithi114.stdout: "type": "mds", 2024-04-23T10:32:04.377 INFO:teuthology.orchestra.run.smithi114.stdout: "num": 1 2024-04-23T10:32:04.377 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-23T10:32:04.377 INFO:teuthology.orchestra.run.smithi114.stdout: "tid": 118 2024-04-23T10:32:04.377 INFO:teuthology.orchestra.run.smithi114.stdout: } 2024-04-23T10:32:04.377 INFO:teuthology.orchestra.run.smithi114.stdout: } 2024-04-23T10:32:04.377 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-23T10:32:04.377 INFO:teuthology.orchestra.run.smithi114.stdout: "flags": 4, 2024-04-23T10:32:04.377 INFO:teuthology.orchestra.run.smithi114.stdout: "wrlock_target": -1 2024-04-23T10:32:04.377 INFO:teuthology.orchestra.run.smithi114.stdout: } 2024-04-23T10:32:04.377 INFO:teuthology.orchestra.run.smithi114.stdout: ] 2024-04-23T10:32:04.377 INFO:teuthology.orchestra.run.smithi114.stdout: } 2024-04-23T10:32:04.377 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-23T10:32:04.377 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-04-23T10:32:04.377 INFO:teuthology.orchestra.run.smithi114.stdout: "description": "internal op quiesce_inode:mds.1:119 fp=#0x10000000225", 2024-04-23T10:32:04.378 INFO:teuthology.orchestra.run.smithi114.stdout: "initiated_at": "2024-04-23T10:31:21.464459+0000", 2024-04-23T10:32:04.378 INFO:teuthology.orchestra.run.smithi114.stdout: "age": 42.393179863, 2024-04-23T10:32:04.378 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 42.394805542999997, 2024-04-23T10:32:04.378 INFO:teuthology.orchestra.run.smithi114.stdout: "continuous": true, 2024-04-23T10:32:04.378 INFO:teuthology.orchestra.run.smithi114.stdout: "type_data": { 2024-04-23T10:32:04.378 INFO:teuthology.orchestra.run.smithi114.stdout: "result": -2147483648, 2024-04-23T10:32:04.378 INFO:teuthology.orchestra.run.smithi114.stdout: "flag_point": "quiesce complete for non-auth inode", 2024-04-23T10:32:04.378 INFO:teuthology.orchestra.run.smithi114.stdout: "reqid": { 2024-04-23T10:32:04.378 INFO:teuthology.orchestra.run.smithi114.stdout: "entity": { 2024-04-23T10:32:04.378 INFO:teuthology.orchestra.run.smithi114.stdout: "type": "mds", 2024-04-23T10:32:04.378 INFO:teuthology.orchestra.run.smithi114.stdout: "num": 1 2024-04-23T10:32:04.378 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-23T10:32:04.378 INFO:teuthology.orchestra.run.smithi114.stdout: "tid": 119 2024-04-23T10:32:04.378 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-23T10:32:04.378 INFO:teuthology.orchestra.run.smithi114.stdout: "op_type": "internal_op", 2024-04-23T10:32:04.378 INFO:teuthology.orchestra.run.smithi114.stdout: "internal_op": 5385, 2024-04-23T10:32:04.379 INFO:teuthology.orchestra.run.smithi114.stdout: "op_name": "quiesce_inode", 2024-04-23T10:32:04.379 INFO:teuthology.orchestra.run.smithi114.stdout: "events": [ 2024-04-23T10:32:04.379 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-04-23T10:32:04.379 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-04-23T10:31:21.464459+0000", 2024-04-23T10:32:04.379 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "initiated" 2024-04-23T10:32:04.379 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-23T10:32:04.379 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-04-23T10:32:04.379 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-04-23T10:31:21.464459+0000", 2024-04-23T10:32:04.379 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "header_read" 2024-04-23T10:32:04.379 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-23T10:32:04.379 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-04-23T10:32:04.379 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-04-23T10:31:21.464459+0000", 2024-04-23T10:32:04.379 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "throttled" 2024-04-23T10:32:04.379 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-23T10:32:04.379 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-04-23T10:32:04.379 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-04-23T10:31:21.464459+0000", 2024-04-23T10:32:04.380 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "all_read" 2024-04-23T10:32:04.380 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-23T10:32:04.380 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-04-23T10:32:04.380 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-04-23T10:31:21.464459+0000", 2024-04-23T10:32:04.380 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "dispatched" 2024-04-23T10:32:04.380 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-23T10:32:04.380 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-04-23T10:32:04.380 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-04-23T10:31:21.465043+0000", 2024-04-23T10:32:04.380 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "requesting remote authpins" 2024-04-23T10:32:04.380 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-23T10:32:04.380 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-04-23T10:32:04.380 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-04-23T10:31:21.496741+0000", 2024-04-23T10:32:04.380 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "acquired locks" 2024-04-23T10:32:04.380 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-23T10:32:04.380 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-04-23T10:32:04.380 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-04-23T10:31:21.496754+0000", 2024-04-23T10:32:04.381 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "quiesce complete for non-auth inode" 2024-04-23T10:32:04.381 INFO:teuthology.orchestra.run.smithi114.stdout: } 2024-04-23T10:32:04.381 INFO:teuthology.orchestra.run.smithi114.stdout: ], 2024-04-23T10:32:04.381 INFO:teuthology.orchestra.run.smithi114.stdout: "locks": [ 2024-04-23T10:32:04.381 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-04-23T10:32:04.381 INFO:teuthology.orchestra.run.smithi114.stdout: "object": { 2024-04-23T10:32:04.381 INFO:teuthology.orchestra.run.smithi114.stdout: "is_auth": false, 2024-04-23T10:32:04.381 INFO:teuthology.orchestra.run.smithi114.stdout: "auth_state": { 2024-04-23T10:32:04.381 INFO:teuthology.orchestra.run.smithi114.stdout: "replicas": {} 2024-04-23T10:32:04.381 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-23T10:32:04.381 INFO:teuthology.orchestra.run.smithi114.stdout: "replica_state": { 2024-04-23T10:32:04.381 INFO:teuthology.orchestra.run.smithi114.stdout: "authority": [ 2024-04-23T10:32:04.381 INFO:teuthology.orchestra.run.smithi114.stdout: 0, 2024-04-23T10:32:04.382 INFO:teuthology.orchestra.run.smithi114.stdout: -2 2024-04-23T10:32:04.382 INFO:teuthology.orchestra.run.smithi114.stdout: ], 2024-04-23T10:32:04.382 INFO:teuthology.orchestra.run.smithi114.stdout: "replica_nonce": 1 2024-04-23T10:32:04.382 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-23T10:32:04.382 INFO:teuthology.orchestra.run.smithi114.stdout: "auth_pins": 0, 2024-04-23T10:32:04.382 INFO:teuthology.orchestra.run.smithi114.stdout: "is_frozen": false, 2024-04-23T10:32:04.382 INFO:teuthology.orchestra.run.smithi114.stdout: "is_freezing": false, 2024-04-23T10:32:04.382 INFO:teuthology.orchestra.run.smithi114.stdout: "pins": { 2024-04-23T10:32:04.382 INFO:teuthology.orchestra.run.smithi114.stdout: "request": 1, 2024-04-23T10:32:04.382 INFO:teuthology.orchestra.run.smithi114.stdout: "lock": 1, 2024-04-23T10:32:04.382 INFO:teuthology.orchestra.run.smithi114.stdout: "dirfrag": 0, 2024-04-23T10:32:04.382 INFO:teuthology.orchestra.run.smithi114.stdout: "dirwaiter": 0, 2024-04-23T10:32:04.382 INFO:teuthology.orchestra.run.smithi114.stdout: "discoverbase": 0 2024-04-23T10:32:04.382 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-23T10:32:04.383 INFO:teuthology.orchestra.run.smithi114.stdout: "nref": 2 2024-04-23T10:32:04.383 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-23T10:32:04.383 INFO:teuthology.orchestra.run.smithi114.stdout: "object_string": "[inode 0x10000000225 [...7,head] /volumes/qa/sv_1/b206794e-7622-4ab8-8943-f3ae4e65b7a3/client.0/tmp/clients/client0/~dmtmp/PM/ rep@0.1 v45574 f(v2926 m2024-04-23T10:31:17.764716+0000 3=3+0) n(v17 rc2024-04-23T10:31:17.764716+0000 b780288 4=3+1)/n(v0 rc2024-04-23T10:27:52.071395+0000 2=1+1) (inest mix) (iquiesce lock x=1 by request(mds.1:119 nref=3)) | request=1 lock=1 dirfrag=0 dirwaiter=0 discoverbase=0 0x55e718b92100]", 2024-04-23T10:32:04.383 INFO:teuthology.orchestra.run.smithi114.stdout: "lock": { 2024-04-23T10:32:04.383 INFO:teuthology.orchestra.run.smithi114.stdout: "gather_set": [], 2024-04-23T10:32:04.383 INFO:teuthology.orchestra.run.smithi114.stdout: "state": "lock", 2024-04-23T10:32:04.383 INFO:teuthology.orchestra.run.smithi114.stdout: "type": "iquiesce", 2024-04-23T10:32:04.383 INFO:teuthology.orchestra.run.smithi114.stdout: "is_leased": false, 2024-04-23T10:32:04.383 INFO:teuthology.orchestra.run.smithi114.stdout: "num_rdlocks": 0, 2024-04-23T10:32:04.383 INFO:teuthology.orchestra.run.smithi114.stdout: "num_wrlocks": 0, 2024-04-23T10:32:04.383 INFO:teuthology.orchestra.run.smithi114.stdout: "num_xlocks": 1, 2024-04-23T10:32:04.383 INFO:teuthology.orchestra.run.smithi114.stdout: "xlock_by": { 2024-04-23T10:32:04.383 INFO:teuthology.orchestra.run.smithi114.stdout: "reqid": { 2024-04-23T10:32:04.383 INFO:teuthology.orchestra.run.smithi114.stdout: "entity": { 2024-04-23T10:32:04.384 INFO:teuthology.orchestra.run.smithi114.stdout: "type": "mds", 2024-04-23T10:32:04.384 INFO:teuthology.orchestra.run.smithi114.stdout: "num": 1 2024-04-23T10:32:04.384 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-23T10:32:04.384 INFO:teuthology.orchestra.run.smithi114.stdout: "tid": 119 2024-04-23T10:32:04.384 INFO:teuthology.orchestra.run.smithi114.stdout: } 2024-04-23T10:32:04.384 INFO:teuthology.orchestra.run.smithi114.stdout: } 2024-04-23T10:32:04.384 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-23T10:32:04.384 INFO:teuthology.orchestra.run.smithi114.stdout: "flags": 4, 2024-04-23T10:32:04.384 INFO:teuthology.orchestra.run.smithi114.stdout: "wrlock_target": -1 2024-04-23T10:32:04.384 INFO:teuthology.orchestra.run.smithi114.stdout: } 2024-04-23T10:32:04.384 INFO:teuthology.orchestra.run.smithi114.stdout: ] 2024-04-23T10:32:04.384 INFO:teuthology.orchestra.run.smithi114.stdout: } 2024-04-23T10:32:04.384 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-23T10:32:04.384 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-04-23T10:32:04.384 INFO:teuthology.orchestra.run.smithi114.stdout: "description": "internal op quiesce_inode:mds.1:120 fp=#0x1000000020f", 2024-04-23T10:32:04.385 INFO:teuthology.orchestra.run.smithi114.stdout: "initiated_at": "2024-04-23T10:31:21.464470+0000", 2024-04-23T10:32:04.385 INFO:teuthology.orchestra.run.smithi114.stdout: "age": 42.393168217000003, 2024-04-23T10:32:04.385 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 42.394857037000001, 2024-04-23T10:32:04.385 INFO:teuthology.orchestra.run.smithi114.stdout: "continuous": true, 2024-04-23T10:32:04.385 INFO:teuthology.orchestra.run.smithi114.stdout: "type_data": { 2024-04-23T10:32:04.385 INFO:teuthology.orchestra.run.smithi114.stdout: "result": -2147483648, 2024-04-23T10:32:04.385 INFO:teuthology.orchestra.run.smithi114.stdout: "flag_point": "quiesce complete for non-auth inode", 2024-04-23T10:32:04.385 INFO:teuthology.orchestra.run.smithi114.stdout: "reqid": { 2024-04-23T10:32:04.385 INFO:teuthology.orchestra.run.smithi114.stdout: "entity": { 2024-04-23T10:32:04.385 INFO:teuthology.orchestra.run.smithi114.stdout: "type": "mds", 2024-04-23T10:32:04.385 INFO:teuthology.orchestra.run.smithi114.stdout: "num": 1 2024-04-23T10:32:04.385 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-23T10:32:04.385 INFO:teuthology.orchestra.run.smithi114.stdout: "tid": 120 2024-04-23T10:32:04.385 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-23T10:32:04.385 INFO:teuthology.orchestra.run.smithi114.stdout: "op_type": "internal_op", 2024-04-23T10:32:04.385 INFO:teuthology.orchestra.run.smithi114.stdout: "internal_op": 5385, 2024-04-23T10:32:04.386 INFO:teuthology.orchestra.run.smithi114.stdout: "op_name": "quiesce_inode", 2024-04-23T10:32:04.386 INFO:teuthology.orchestra.run.smithi114.stdout: "events": [ 2024-04-23T10:32:04.386 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-04-23T10:32:04.386 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-04-23T10:31:21.464470+0000", 2024-04-23T10:32:04.386 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "initiated" 2024-04-23T10:32:04.386 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-23T10:32:04.386 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-04-23T10:32:04.386 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-04-23T10:31:21.464470+0000", 2024-04-23T10:32:04.386 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "header_read" 2024-04-23T10:32:04.386 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-23T10:32:04.386 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-04-23T10:32:04.386 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-04-23T10:31:21.464470+0000", 2024-04-23T10:32:04.386 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "throttled" 2024-04-23T10:32:04.386 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-23T10:32:04.386 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-04-23T10:32:04.387 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-04-23T10:31:21.464470+0000", 2024-04-23T10:32:04.387 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "all_read" 2024-04-23T10:32:04.387 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-23T10:32:04.387 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-04-23T10:32:04.387 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-04-23T10:31:21.464470+0000", 2024-04-23T10:32:04.387 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "dispatched" 2024-04-23T10:32:04.387 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-23T10:32:04.387 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-04-23T10:32:04.387 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-04-23T10:31:21.465112+0000", 2024-04-23T10:32:04.387 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "requesting remote authpins" 2024-04-23T10:32:04.387 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-23T10:32:04.387 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-04-23T10:32:04.387 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-04-23T10:31:21.496858+0000", 2024-04-23T10:32:04.387 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "acquired locks" 2024-04-23T10:32:04.387 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-23T10:32:04.388 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-04-23T10:32:04.388 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-04-23T10:31:21.496882+0000", 2024-04-23T10:32:04.388 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "quiesce complete for non-auth inode" 2024-04-23T10:32:04.388 INFO:teuthology.orchestra.run.smithi114.stdout: } 2024-04-23T10:32:04.388 INFO:teuthology.orchestra.run.smithi114.stdout: ], 2024-04-23T10:32:04.388 INFO:teuthology.orchestra.run.smithi114.stdout: "locks": [ 2024-04-23T10:32:04.388 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-04-23T10:32:04.388 INFO:teuthology.orchestra.run.smithi114.stdout: "object": { 2024-04-23T10:32:04.388 INFO:teuthology.orchestra.run.smithi114.stdout: "is_auth": false, 2024-04-23T10:32:04.388 INFO:teuthology.orchestra.run.smithi114.stdout: "auth_state": { 2024-04-23T10:32:04.388 INFO:teuthology.orchestra.run.smithi114.stdout: "replicas": {} 2024-04-23T10:32:04.388 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-23T10:32:04.388 INFO:teuthology.orchestra.run.smithi114.stdout: "replica_state": { 2024-04-23T10:32:04.388 INFO:teuthology.orchestra.run.smithi114.stdout: "authority": [ 2024-04-23T10:32:04.388 INFO:teuthology.orchestra.run.smithi114.stdout: 0, 2024-04-23T10:32:04.389 INFO:teuthology.orchestra.run.smithi114.stdout: -2 2024-04-23T10:32:04.389 INFO:teuthology.orchestra.run.smithi114.stdout: ], 2024-04-23T10:32:04.389 INFO:teuthology.orchestra.run.smithi114.stdout: "replica_nonce": 1 2024-04-23T10:32:04.389 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-23T10:32:04.389 INFO:teuthology.orchestra.run.smithi114.stdout: "auth_pins": 0, 2024-04-23T10:32:04.389 INFO:teuthology.orchestra.run.smithi114.stdout: "is_frozen": false, 2024-04-23T10:32:04.389 INFO:teuthology.orchestra.run.smithi114.stdout: "is_freezing": false, 2024-04-23T10:32:04.389 INFO:teuthology.orchestra.run.smithi114.stdout: "pins": { 2024-04-23T10:32:04.389 INFO:teuthology.orchestra.run.smithi114.stdout: "request": 1, 2024-04-23T10:32:04.389 INFO:teuthology.orchestra.run.smithi114.stdout: "lock": 1, 2024-04-23T10:32:04.389 INFO:teuthology.orchestra.run.smithi114.stdout: "dirfrag": 0, 2024-04-23T10:32:04.389 INFO:teuthology.orchestra.run.smithi114.stdout: "dirwaiter": 0, 2024-04-23T10:32:04.389 INFO:teuthology.orchestra.run.smithi114.stdout: "discoverbase": 0 2024-04-23T10:32:04.389 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-23T10:32:04.389 INFO:teuthology.orchestra.run.smithi114.stdout: "nref": 2 2024-04-23T10:32:04.390 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-23T10:32:04.390 INFO:teuthology.orchestra.run.smithi114.stdout: "object_string": "[inode 0x1000000020f [...7,head] /volumes/qa/sv_1/b206794e-7622-4ab8-8943-f3ae4e65b7a3/client.0/tmp/clients/client0/~dmtmp/PWRPNT/ rep@0.1 v45572 f(v1293 m2024-04-23T10:31:17.718717+0000 5=5+0) n(v18 rc2024-04-23T10:31:17.718717+0000 b1300480 6=5+1)/n(v0 rc2024-04-23T10:27:51.976397+0000 2=1+1) (inest mix) (iquiesce lock x=1 by request(mds.1:120 nref=3)) | request=1 lock=1 dirfrag=0 dirwaiter=0 discoverbase=0 0x55e718b91080]", 2024-04-23T10:32:04.390 INFO:teuthology.orchestra.run.smithi114.stdout: "lock": { 2024-04-23T10:32:04.390 INFO:teuthology.orchestra.run.smithi114.stdout: "gather_set": [], 2024-04-23T10:32:04.390 INFO:teuthology.orchestra.run.smithi114.stdout: "state": "lock", 2024-04-23T10:32:04.390 INFO:teuthology.orchestra.run.smithi114.stdout: "type": "iquiesce", 2024-04-23T10:32:04.390 INFO:teuthology.orchestra.run.smithi114.stdout: "is_leased": false, 2024-04-23T10:32:04.390 INFO:teuthology.orchestra.run.smithi114.stdout: "num_rdlocks": 0, 2024-04-23T10:32:04.390 INFO:teuthology.orchestra.run.smithi114.stdout: "num_wrlocks": 0, 2024-04-23T10:32:04.390 INFO:teuthology.orchestra.run.smithi114.stdout: "num_xlocks": 1, 2024-04-23T10:32:04.390 INFO:teuthology.orchestra.run.smithi114.stdout: "xlock_by": { 2024-04-23T10:32:04.390 INFO:teuthology.orchestra.run.smithi114.stdout: "reqid": { 2024-04-23T10:32:04.390 INFO:teuthology.orchestra.run.smithi114.stdout: "entity": { 2024-04-23T10:32:04.391 INFO:teuthology.orchestra.run.smithi114.stdout: "type": "mds", 2024-04-23T10:32:04.391 INFO:teuthology.orchestra.run.smithi114.stdout: "num": 1 2024-04-23T10:32:04.391 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-23T10:32:04.391 INFO:teuthology.orchestra.run.smithi114.stdout: "tid": 120 2024-04-23T10:32:04.391 INFO:teuthology.orchestra.run.smithi114.stdout: } 2024-04-23T10:32:04.391 INFO:teuthology.orchestra.run.smithi114.stdout: } 2024-04-23T10:32:04.391 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-23T10:32:04.391 INFO:teuthology.orchestra.run.smithi114.stdout: "flags": 4, 2024-04-23T10:32:04.391 INFO:teuthology.orchestra.run.smithi114.stdout: "wrlock_target": -1 2024-04-23T10:32:04.391 INFO:teuthology.orchestra.run.smithi114.stdout: } 2024-04-23T10:32:04.391 INFO:teuthology.orchestra.run.smithi114.stdout: ] 2024-04-23T10:32:04.391 INFO:teuthology.orchestra.run.smithi114.stdout: } 2024-04-23T10:32:04.391 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-23T10:32:04.391 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-04-23T10:32:04.391 INFO:teuthology.orchestra.run.smithi114.stdout: "description": "internal op quiesce_inode:mds.1:121 fp=#0x1000000023e", 2024-04-23T10:32:04.392 INFO:teuthology.orchestra.run.smithi114.stdout: "initiated_at": "2024-04-23T10:31:21.464493+0000", 2024-04-23T10:32:04.392 INFO:teuthology.orchestra.run.smithi114.stdout: "age": 42.393145695000001, 2024-04-23T10:32:04.392 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 42.394896596000002, 2024-04-23T10:32:04.392 INFO:teuthology.orchestra.run.smithi114.stdout: "continuous": true, 2024-04-23T10:32:04.392 INFO:teuthology.orchestra.run.smithi114.stdout: "type_data": { 2024-04-23T10:32:04.392 INFO:teuthology.orchestra.run.smithi114.stdout: "result": -2147483648, 2024-04-23T10:32:04.392 INFO:teuthology.orchestra.run.smithi114.stdout: "flag_point": "quiesce complete for non-auth inode", 2024-04-23T10:32:04.392 INFO:teuthology.orchestra.run.smithi114.stdout: "reqid": { 2024-04-23T10:32:04.392 INFO:teuthology.orchestra.run.smithi114.stdout: "entity": { 2024-04-23T10:32:04.392 INFO:teuthology.orchestra.run.smithi114.stdout: "type": "mds", 2024-04-23T10:32:04.392 INFO:teuthology.orchestra.run.smithi114.stdout: "num": 1 2024-04-23T10:32:04.392 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-23T10:32:04.392 INFO:teuthology.orchestra.run.smithi114.stdout: "tid": 121 2024-04-23T10:32:04.392 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-23T10:32:04.392 INFO:teuthology.orchestra.run.smithi114.stdout: "op_type": "internal_op", 2024-04-23T10:32:04.393 INFO:teuthology.orchestra.run.smithi114.stdout: "internal_op": 5385, 2024-04-23T10:32:04.393 INFO:teuthology.orchestra.run.smithi114.stdout: "op_name": "quiesce_inode", 2024-04-23T10:32:04.393 INFO:teuthology.orchestra.run.smithi114.stdout: "events": [ 2024-04-23T10:32:04.393 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-04-23T10:32:04.393 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-04-23T10:31:21.464493+0000", 2024-04-23T10:32:04.393 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "initiated" 2024-04-23T10:32:04.393 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-23T10:32:04.393 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-04-23T10:32:04.393 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-04-23T10:31:21.464493+0000", 2024-04-23T10:32:04.393 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "header_read" 2024-04-23T10:32:04.393 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-23T10:32:04.393 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-04-23T10:32:04.393 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-04-23T10:31:21.464493+0000", 2024-04-23T10:32:04.394 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "throttled" 2024-04-23T10:32:04.394 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-23T10:32:04.394 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-04-23T10:32:04.394 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-04-23T10:31:21.464493+0000", 2024-04-23T10:32:04.394 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "all_read" 2024-04-23T10:32:04.394 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-23T10:32:04.394 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-04-23T10:32:04.394 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-04-23T10:31:21.464493+0000", 2024-04-23T10:32:04.394 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "dispatched" 2024-04-23T10:32:04.394 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-23T10:32:04.394 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-04-23T10:32:04.394 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-04-23T10:31:21.465182+0000", 2024-04-23T10:32:04.395 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "requesting remote authpins" 2024-04-23T10:32:04.395 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-23T10:32:04.395 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-04-23T10:32:04.395 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-04-23T10:31:21.498658+0000", 2024-04-23T10:32:04.395 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "acquired locks" 2024-04-23T10:32:04.395 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-23T10:32:04.395 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-04-23T10:32:04.395 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-04-23T10:31:21.498671+0000", 2024-04-23T10:32:04.395 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "quiesce complete for non-auth inode" 2024-04-23T10:32:04.395 INFO:teuthology.orchestra.run.smithi114.stdout: } 2024-04-23T10:32:04.395 INFO:teuthology.orchestra.run.smithi114.stdout: ], 2024-04-23T10:32:04.395 INFO:teuthology.orchestra.run.smithi114.stdout: "locks": [ 2024-04-23T10:32:04.395 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-04-23T10:32:04.395 INFO:teuthology.orchestra.run.smithi114.stdout: "object": { 2024-04-23T10:32:04.395 INFO:teuthology.orchestra.run.smithi114.stdout: "is_auth": false, 2024-04-23T10:32:04.396 INFO:teuthology.orchestra.run.smithi114.stdout: "auth_state": { 2024-04-23T10:32:04.396 INFO:teuthology.orchestra.run.smithi114.stdout: "replicas": {} 2024-04-23T10:32:04.396 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-23T10:32:04.396 INFO:teuthology.orchestra.run.smithi114.stdout: "replica_state": { 2024-04-23T10:32:04.396 INFO:teuthology.orchestra.run.smithi114.stdout: "authority": [ 2024-04-23T10:32:04.396 INFO:teuthology.orchestra.run.smithi114.stdout: 0, 2024-04-23T10:32:04.396 INFO:teuthology.orchestra.run.smithi114.stdout: -2 2024-04-23T10:32:04.396 INFO:teuthology.orchestra.run.smithi114.stdout: ], 2024-04-23T10:32:04.396 INFO:teuthology.orchestra.run.smithi114.stdout: "replica_nonce": 1 2024-04-23T10:32:04.396 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-23T10:32:04.396 INFO:teuthology.orchestra.run.smithi114.stdout: "auth_pins": 0, 2024-04-23T10:32:04.396 INFO:teuthology.orchestra.run.smithi114.stdout: "is_frozen": false, 2024-04-23T10:32:04.396 INFO:teuthology.orchestra.run.smithi114.stdout: "is_freezing": false, 2024-04-23T10:32:04.396 INFO:teuthology.orchestra.run.smithi114.stdout: "pins": { 2024-04-23T10:32:04.397 INFO:teuthology.orchestra.run.smithi114.stdout: "request": 1, 2024-04-23T10:32:04.397 INFO:teuthology.orchestra.run.smithi114.stdout: "lock": 1, 2024-04-23T10:32:04.397 INFO:teuthology.orchestra.run.smithi114.stdout: "dirwaiter": 0, 2024-04-23T10:32:04.397 INFO:teuthology.orchestra.run.smithi114.stdout: "discoverbase": 0 2024-04-23T10:32:04.397 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-23T10:32:04.397 INFO:teuthology.orchestra.run.smithi114.stdout: "nref": 2 2024-04-23T10:32:04.397 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-23T10:32:04.397 INFO:teuthology.orchestra.run.smithi114.stdout: "object_string": "[inode 0x1000000023e [...7,head] /volumes/qa/sv_1/b206794e-7622-4ab8-8943-f3ae4e65b7a3/client.0/tmp/clients/client0/~dmtmp/ACCESS/ rep@0.1 v45357 f(v0 m2024-04-23T10:31:17.039732+0000 4=4+0) n(v0 rc2024-04-23T10:27:52.903376+0000 b260096 3=2+1)/n(v0 rc2024-04-23T10:27:52.181393+0000 2=1+1) (inest mix) (iquiesce lock x=1 by request(mds.1:121 nref=3)) | request=1 lock=1 dirwaiter=0 discoverbase=0 0x55e718b93180]", 2024-04-23T10:32:04.397 INFO:teuthology.orchestra.run.smithi114.stdout: "lock": { 2024-04-23T10:32:04.397 INFO:teuthology.orchestra.run.smithi114.stdout: "gather_set": [], 2024-04-23T10:32:04.397 INFO:teuthology.orchestra.run.smithi114.stdout: "state": "lock", 2024-04-23T10:32:04.397 INFO:teuthology.orchestra.run.smithi114.stdout: "type": "iquiesce", 2024-04-23T10:32:04.397 INFO:teuthology.orchestra.run.smithi114.stdout: "is_leased": false, 2024-04-23T10:32:04.397 INFO:teuthology.orchestra.run.smithi114.stdout: "num_rdlocks": 0, 2024-04-23T10:32:04.397 INFO:teuthology.orchestra.run.smithi114.stdout: "num_wrlocks": 0, 2024-04-23T10:32:04.398 INFO:teuthology.orchestra.run.smithi114.stdout: "num_xlocks": 1, 2024-04-23T10:32:04.398 INFO:teuthology.orchestra.run.smithi114.stdout: "xlock_by": { 2024-04-23T10:32:04.398 INFO:teuthology.orchestra.run.smithi114.stdout: "reqid": { 2024-04-23T10:32:04.398 INFO:teuthology.orchestra.run.smithi114.stdout: "entity": { 2024-04-23T10:32:04.398 INFO:teuthology.orchestra.run.smithi114.stdout: "type": "mds", 2024-04-23T10:32:04.398 INFO:teuthology.orchestra.run.smithi114.stdout: "num": 1 2024-04-23T10:32:04.398 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-23T10:32:04.398 INFO:teuthology.orchestra.run.smithi114.stdout: "tid": 121 2024-04-23T10:32:04.398 INFO:teuthology.orchestra.run.smithi114.stdout: } 2024-04-23T10:32:04.398 INFO:teuthology.orchestra.run.smithi114.stdout: } 2024-04-23T10:32:04.398 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-23T10:32:04.398 INFO:teuthology.orchestra.run.smithi114.stdout: "flags": 4, 2024-04-23T10:32:04.398 INFO:teuthology.orchestra.run.smithi114.stdout: "wrlock_target": -1 2024-04-23T10:32:04.398 INFO:teuthology.orchestra.run.smithi114.stdout: } 2024-04-23T10:32:04.399 INFO:teuthology.orchestra.run.smithi114.stdout: ] 2024-04-23T10:32:04.399 INFO:teuthology.orchestra.run.smithi114.stdout: } 2024-04-23T10:32:04.399 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-23T10:32:04.399 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-04-23T10:32:04.399 INFO:teuthology.orchestra.run.smithi114.stdout: "description": "internal op quiesce_inode:mds.1:122 fp=#0x1000000022b", 2024-04-23T10:32:04.399 INFO:teuthology.orchestra.run.smithi114.stdout: "initiated_at": "2024-04-23T10:31:21.494037+0000", 2024-04-23T10:32:04.399 INFO:teuthology.orchestra.run.smithi114.stdout: "age": 42.363601457999998, 2024-04-23T10:32:04.399 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 42.365420164, 2024-04-23T10:32:04.399 INFO:teuthology.orchestra.run.smithi114.stdout: "continuous": true, 2024-04-23T10:32:04.399 INFO:teuthology.orchestra.run.smithi114.stdout: "type_data": { 2024-04-23T10:32:04.399 INFO:teuthology.orchestra.run.smithi114.stdout: "result": -2147483648, 2024-04-23T10:32:04.399 INFO:teuthology.orchestra.run.smithi114.stdout: "flag_point": "failed to authpin, dir is being fragmented", 2024-04-23T10:32:04.399 INFO:teuthology.orchestra.run.smithi114.stdout: "reqid": { 2024-04-23T10:32:04.399 INFO:teuthology.orchestra.run.smithi114.stdout: "entity": { 2024-04-23T10:32:04.399 INFO:teuthology.orchestra.run.smithi114.stdout: "type": "mds", 2024-04-23T10:32:04.400 INFO:teuthology.orchestra.run.smithi114.stdout: "num": 1 2024-04-23T10:32:04.400 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-23T10:32:04.400 INFO:teuthology.orchestra.run.smithi114.stdout: "tid": 122 2024-04-23T10:32:04.400 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-23T10:32:04.400 INFO:teuthology.orchestra.run.smithi114.stdout: "op_type": "internal_op", 2024-04-23T10:32:04.400 INFO:teuthology.orchestra.run.smithi114.stdout: "internal_op": 5385, 2024-04-23T10:32:04.400 INFO:teuthology.orchestra.run.smithi114.stdout: "op_name": "quiesce_inode", 2024-04-23T10:32:04.400 INFO:teuthology.orchestra.run.smithi114.stdout: "events": [ 2024-04-23T10:32:04.400 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-04-23T10:32:04.400 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-04-23T10:31:21.494037+0000", 2024-04-23T10:32:04.400 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "initiated" 2024-04-23T10:32:04.400 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-23T10:32:04.400 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-04-23T10:32:04.400 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-04-23T10:31:21.494037+0000", 2024-04-23T10:32:04.400 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "header_read" 2024-04-23T10:32:04.401 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-23T10:32:04.401 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-04-23T10:32:04.401 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-04-23T10:31:21.494037+0000", 2024-04-23T10:32:04.401 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "throttled" 2024-04-23T10:32:04.401 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-23T10:32:04.401 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-04-23T10:32:04.401 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-04-23T10:31:21.494037+0000", 2024-04-23T10:32:04.401 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "all_read" 2024-04-23T10:32:04.401 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-23T10:32:04.401 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-04-23T10:32:04.401 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-04-23T10:31:21.494037+0000", 2024-04-23T10:32:04.401 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "dispatched" 2024-04-23T10:32:04.401 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-23T10:32:04.401 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-04-23T10:32:04.401 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-04-23T10:31:21.494549+0000", 2024-04-23T10:32:04.402 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "failed to authpin, dir is being fragmented" 2024-04-23T10:32:04.402 INFO:teuthology.orchestra.run.smithi114.stdout: } 2024-04-23T10:32:04.402 INFO:teuthology.orchestra.run.smithi114.stdout: ], 2024-04-23T10:32:04.402 INFO:teuthology.orchestra.run.smithi114.stdout: "locks": [] 2024-04-23T10:32:04.402 INFO:teuthology.orchestra.run.smithi114.stdout: } 2024-04-23T10:32:04.402 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-23T10:32:04.402 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-04-23T10:32:04.402 INFO:teuthology.orchestra.run.smithi114.stdout: "description": "internal op quiesce_inode:mds.1:123 fp=#0x200000006f9", 2024-04-23T10:32:04.402 INFO:teuthology.orchestra.run.smithi114.stdout: "initiated_at": "2024-04-23T10:31:21.494053+0000", 2024-04-23T10:32:04.402 INFO:teuthology.orchestra.run.smithi114.stdout: "age": 42.363585348999997, 2024-04-23T10:32:04.402 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 42.365431334999997, 2024-04-23T10:32:04.402 INFO:teuthology.orchestra.run.smithi114.stdout: "continuous": true, 2024-04-23T10:32:04.402 INFO:teuthology.orchestra.run.smithi114.stdout: "type_data": { 2024-04-23T10:32:04.402 INFO:teuthology.orchestra.run.smithi114.stdout: "result": -2147483648, 2024-04-23T10:32:04.403 INFO:teuthology.orchestra.run.smithi114.stdout: "flag_point": "failed to authpin, dir is being fragmented", 2024-04-23T10:32:04.403 INFO:teuthology.orchestra.run.smithi114.stdout: "reqid": { 2024-04-23T10:32:04.403 INFO:teuthology.orchestra.run.smithi114.stdout: "entity": { 2024-04-23T10:32:04.403 INFO:teuthology.orchestra.run.smithi114.stdout: "type": "mds", 2024-04-23T10:32:04.403 INFO:teuthology.orchestra.run.smithi114.stdout: "num": 1 2024-04-23T10:32:04.403 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-23T10:32:04.403 INFO:teuthology.orchestra.run.smithi114.stdout: "tid": 123 2024-04-23T10:32:04.403 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-23T10:32:04.403 INFO:teuthology.orchestra.run.smithi114.stdout: "op_type": "internal_op", 2024-04-23T10:32:04.403 INFO:teuthology.orchestra.run.smithi114.stdout: "internal_op": 5385, 2024-04-23T10:32:04.403 INFO:teuthology.orchestra.run.smithi114.stdout: "op_name": "quiesce_inode", 2024-04-23T10:32:04.403 INFO:teuthology.orchestra.run.smithi114.stdout: "events": [ 2024-04-23T10:32:04.403 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-04-23T10:32:04.403 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-04-23T10:31:21.494053+0000", 2024-04-23T10:32:04.403 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "initiated" 2024-04-23T10:32:04.404 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-23T10:32:04.404 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-04-23T10:32:04.404 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-04-23T10:31:21.494053+0000", 2024-04-23T10:32:04.404 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "header_read" 2024-04-23T10:32:04.404 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-23T10:32:04.404 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-04-23T10:32:04.404 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-04-23T10:31:21.494053+0000", 2024-04-23T10:32:04.404 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "throttled" 2024-04-23T10:32:04.404 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-23T10:32:04.404 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-04-23T10:32:04.404 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-04-23T10:31:21.494053+0000", 2024-04-23T10:32:04.404 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "all_read" 2024-04-23T10:32:04.404 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-23T10:32:04.404 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-04-23T10:32:04.404 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-04-23T10:31:21.494053+0000", 2024-04-23T10:32:04.404 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "dispatched" 2024-04-23T10:32:04.405 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-23T10:32:04.405 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-04-23T10:32:04.405 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-04-23T10:31:21.494624+0000", 2024-04-23T10:32:04.405 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "failed to authpin, dir is being fragmented" 2024-04-23T10:32:04.405 INFO:teuthology.orchestra.run.smithi114.stdout: } 2024-04-23T10:32:04.405 INFO:teuthology.orchestra.run.smithi114.stdout: ], 2024-04-23T10:32:04.405 INFO:teuthology.orchestra.run.smithi114.stdout: "locks": [] 2024-04-23T10:32:04.405 INFO:teuthology.orchestra.run.smithi114.stdout: } 2024-04-23T10:32:04.405 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-23T10:32:04.405 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-04-23T10:32:04.405 INFO:teuthology.orchestra.run.smithi114.stdout: "description": "internal op quiesce_inode:mds.1:124 fp=#0x1000000022c", 2024-04-23T10:32:04.405 INFO:teuthology.orchestra.run.smithi114.stdout: "initiated_at": "2024-04-23T10:31:21.494068+0000", 2024-04-23T10:32:04.405 INFO:teuthology.orchestra.run.smithi114.stdout: "age": 42.363570078999999, 2024-04-23T10:32:04.405 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 42.365441021000002, 2024-04-23T10:32:04.405 INFO:teuthology.orchestra.run.smithi114.stdout: "continuous": true, 2024-04-23T10:32:04.405 INFO:teuthology.orchestra.run.smithi114.stdout: "type_data": { 2024-04-23T10:32:04.406 INFO:teuthology.orchestra.run.smithi114.stdout: "result": -2147483648, 2024-04-23T10:32:04.406 INFO:teuthology.orchestra.run.smithi114.stdout: "flag_point": "failed to authpin, dir is being fragmented", 2024-04-23T10:32:04.406 INFO:teuthology.orchestra.run.smithi114.stdout: "reqid": { 2024-04-23T10:32:04.406 INFO:teuthology.orchestra.run.smithi114.stdout: "entity": { 2024-04-23T10:32:04.406 INFO:teuthology.orchestra.run.smithi114.stdout: "type": "mds", 2024-04-23T10:32:04.406 INFO:teuthology.orchestra.run.smithi114.stdout: "num": 1 2024-04-23T10:32:04.406 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-23T10:32:04.406 INFO:teuthology.orchestra.run.smithi114.stdout: "tid": 124 2024-04-23T10:32:04.406 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-23T10:32:04.406 INFO:teuthology.orchestra.run.smithi114.stdout: "op_type": "internal_op", 2024-04-23T10:32:04.406 INFO:teuthology.orchestra.run.smithi114.stdout: "internal_op": 5385, 2024-04-23T10:32:04.406 INFO:teuthology.orchestra.run.smithi114.stdout: "op_name": "quiesce_inode", 2024-04-23T10:32:04.406 INFO:teuthology.orchestra.run.smithi114.stdout: "events": [ 2024-04-23T10:32:04.406 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-04-23T10:32:04.406 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-04-23T10:31:21.494068+0000", 2024-04-23T10:32:04.406 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "initiated" 2024-04-23T10:32:04.407 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-23T10:32:04.407 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-04-23T10:32:04.407 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-04-23T10:31:21.494068+0000", 2024-04-23T10:32:04.407 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "header_read" 2024-04-23T10:32:04.407 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-23T10:32:04.407 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-04-23T10:32:04.407 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-04-23T10:31:21.494068+0000", 2024-04-23T10:32:04.407 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "throttled" 2024-04-23T10:32:04.407 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-23T10:32:04.407 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-04-23T10:32:04.407 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-04-23T10:31:21.494068+0000", 2024-04-23T10:32:04.407 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "all_read" 2024-04-23T10:32:04.407 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-23T10:32:04.407 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-04-23T10:32:04.407 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-04-23T10:31:21.494068+0000", 2024-04-23T10:32:04.408 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "dispatched" 2024-04-23T10:32:04.408 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-23T10:32:04.408 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-04-23T10:32:04.408 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-04-23T10:31:21.494716+0000", 2024-04-23T10:32:04.408 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "failed to authpin, dir is being fragmented" 2024-04-23T10:32:04.408 INFO:teuthology.orchestra.run.smithi114.stdout: } 2024-04-23T10:32:04.408 INFO:teuthology.orchestra.run.smithi114.stdout: ], 2024-04-23T10:32:04.408 INFO:teuthology.orchestra.run.smithi114.stdout: "locks": [] 2024-04-23T10:32:04.408 INFO:teuthology.orchestra.run.smithi114.stdout: } 2024-04-23T10:32:04.408 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-23T10:32:04.408 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-04-23T10:32:04.408 INFO:teuthology.orchestra.run.smithi114.stdout: "description": "internal op quiesce_inode:mds.1:125 fp=#0x10000000231", 2024-04-23T10:32:04.408 INFO:teuthology.orchestra.run.smithi114.stdout: "initiated_at": "2024-04-23T10:31:21.494090+0000", 2024-04-23T10:32:04.408 INFO:teuthology.orchestra.run.smithi114.stdout: "age": 42.363548352000002, 2024-04-23T10:32:04.408 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 42.365445117999997, 2024-04-23T10:32:04.408 INFO:teuthology.orchestra.run.smithi114.stdout: "continuous": true, 2024-04-23T10:32:04.409 INFO:teuthology.orchestra.run.smithi114.stdout: "type_data": { 2024-04-23T10:32:04.409 INFO:teuthology.orchestra.run.smithi114.stdout: "result": -2147483648, 2024-04-23T10:32:04.409 INFO:teuthology.orchestra.run.smithi114.stdout: "flag_point": "failed to authpin, dir is being fragmented", 2024-04-23T10:32:04.409 INFO:teuthology.orchestra.run.smithi114.stdout: "reqid": { 2024-04-23T10:32:04.409 INFO:teuthology.orchestra.run.smithi114.stdout: "entity": { 2024-04-23T10:32:04.409 INFO:teuthology.orchestra.run.smithi114.stdout: "type": "mds", 2024-04-23T10:32:04.409 INFO:teuthology.orchestra.run.smithi114.stdout: "num": 1 2024-04-23T10:32:04.409 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-23T10:32:04.409 INFO:teuthology.orchestra.run.smithi114.stdout: "tid": 125 2024-04-23T10:32:04.409 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-23T10:32:04.409 INFO:teuthology.orchestra.run.smithi114.stdout: "op_type": "internal_op", 2024-04-23T10:32:04.409 INFO:teuthology.orchestra.run.smithi114.stdout: "internal_op": 5385, 2024-04-23T10:32:04.409 INFO:teuthology.orchestra.run.smithi114.stdout: "op_name": "quiesce_inode", 2024-04-23T10:32:04.409 INFO:teuthology.orchestra.run.smithi114.stdout: "events": [ 2024-04-23T10:32:04.410 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-04-23T10:32:04.410 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-04-23T10:31:21.494090+0000", 2024-04-23T10:32:04.410 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "initiated" 2024-04-23T10:32:04.410 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-23T10:32:04.410 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-04-23T10:32:04.410 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-04-23T10:31:21.494090+0000", 2024-04-23T10:32:04.410 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "header_read" 2024-04-23T10:32:04.410 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-23T10:32:04.410 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-04-23T10:32:04.410 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-04-23T10:31:21.494090+0000", 2024-04-23T10:32:04.410 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "throttled" 2024-04-23T10:32:04.410 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-23T10:32:04.410 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-04-23T10:32:04.410 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-04-23T10:31:21.494090+0000", 2024-04-23T10:32:04.410 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "all_read" 2024-04-23T10:32:04.411 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-23T10:32:04.411 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-04-23T10:32:04.411 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-04-23T10:31:21.494090+0000", 2024-04-23T10:32:04.411 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "dispatched" 2024-04-23T10:32:04.411 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-23T10:32:04.411 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-04-23T10:32:04.411 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-04-23T10:31:21.494806+0000", 2024-04-23T10:32:04.411 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "failed to authpin, dir is being fragmented" 2024-04-23T10:32:04.411 INFO:teuthology.orchestra.run.smithi114.stdout: } 2024-04-23T10:32:04.411 INFO:teuthology.orchestra.run.smithi114.stdout: ], 2024-04-23T10:32:04.411 INFO:teuthology.orchestra.run.smithi114.stdout: "locks": [] 2024-04-23T10:32:04.411 INFO:teuthology.orchestra.run.smithi114.stdout: } 2024-04-23T10:32:04.411 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-23T10:32:04.411 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-04-23T10:32:04.412 INFO:teuthology.orchestra.run.smithi114.stdout: "description": "internal op fragmentdir:mds.1:100", 2024-04-23T10:32:04.412 INFO:teuthology.orchestra.run.smithi114.stdout: "initiated_at": "2024-04-23T10:31:16.892500+0000", 2024-04-23T10:32:04.412 INFO:teuthology.orchestra.run.smithi114.stdout: "age": 46.965138816, 2024-04-23T10:32:04.412 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 46.967060001999997, 2024-04-23T10:32:04.412 INFO:teuthology.orchestra.run.smithi114.stdout: "continuous": false, 2024-04-23T10:32:04.412 INFO:teuthology.orchestra.run.smithi114.stdout: "type_data": { 2024-04-23T10:32:04.412 INFO:teuthology.orchestra.run.smithi114.stdout: "result": -2147483648, 2024-04-23T10:32:04.412 INFO:teuthology.orchestra.run.smithi114.stdout: "flag_point": "failed to acquire quiesce lock", 2024-04-23T10:32:04.412 INFO:teuthology.orchestra.run.smithi114.stdout: "reqid": { 2024-04-23T10:32:04.412 INFO:teuthology.orchestra.run.smithi114.stdout: "entity": { 2024-04-23T10:32:04.412 INFO:teuthology.orchestra.run.smithi114.stdout: "type": "mds", 2024-04-23T10:32:04.412 INFO:teuthology.orchestra.run.smithi114.stdout: "num": 1 2024-04-23T10:32:04.412 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-23T10:32:04.412 INFO:teuthology.orchestra.run.smithi114.stdout: "tid": 100 2024-04-23T10:32:04.412 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-23T10:32:04.413 INFO:teuthology.orchestra.run.smithi114.stdout: "op_type": "internal_op", 2024-04-23T10:32:04.413 INFO:teuthology.orchestra.run.smithi114.stdout: "internal_op": 5376, 2024-04-23T10:32:04.413 INFO:teuthology.orchestra.run.smithi114.stdout: "op_name": "fragmentdir", 2024-04-23T10:32:04.413 INFO:teuthology.orchestra.run.smithi114.stdout: "events": [ 2024-04-23T10:32:04.413 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-04-23T10:32:04.413 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-04-23T10:31:16.892500+0000", 2024-04-23T10:32:04.413 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "initiated" 2024-04-23T10:32:04.413 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-23T10:32:04.413 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-04-23T10:32:04.413 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-04-23T10:31:16.892500+0000", 2024-04-23T10:32:04.413 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "header_read" 2024-04-23T10:32:04.413 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-23T10:32:04.413 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-04-23T10:32:04.413 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-04-23T10:31:16.892500+0000", 2024-04-23T10:32:04.414 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "throttled" 2024-04-23T10:32:04.414 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-23T10:32:04.414 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-04-23T10:32:04.414 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-04-23T10:31:16.892500+0000", 2024-04-23T10:32:04.414 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "all_read" 2024-04-23T10:32:04.414 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-23T10:32:04.414 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-04-23T10:32:04.414 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-04-23T10:31:16.892500+0000", 2024-04-23T10:32:04.414 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "dispatched" 2024-04-23T10:32:04.414 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-23T10:32:04.414 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-04-23T10:32:04.414 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-04-23T10:31:22.426076+0000", 2024-04-23T10:32:04.414 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "requesting remote authpins" 2024-04-23T10:32:04.414 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-23T10:32:04.414 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-04-23T10:32:04.415 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-04-23T10:31:22.426703+0000", 2024-04-23T10:32:04.415 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "failed to acquire quiesce lock" 2024-04-23T10:32:04.415 INFO:teuthology.orchestra.run.smithi114.stdout: } 2024-04-23T10:32:04.415 INFO:teuthology.orchestra.run.smithi114.stdout: ], 2024-04-23T10:32:04.415 INFO:teuthology.orchestra.run.smithi114.stdout: "locks": [] 2024-04-23T10:32:04.415 INFO:teuthology.orchestra.run.smithi114.stdout: } 2024-04-23T10:32:04.415 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-23T10:32:04.415 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-04-23T10:32:04.415 INFO:teuthology.orchestra.run.smithi114.stdout: "description": "internal op quiesce_inode:mds.1:126 fp=#0x10000000237", 2024-04-23T10:32:04.415 INFO:teuthology.orchestra.run.smithi114.stdout: "initiated_at": "2024-04-23T10:31:21.494106+0000", 2024-04-23T10:32:04.415 INFO:teuthology.orchestra.run.smithi114.stdout: "age": 42.363532720000002, 2024-04-23T10:32:04.415 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 42.365481694000003, 2024-04-23T10:32:04.415 INFO:teuthology.orchestra.run.smithi114.stdout: "continuous": true, 2024-04-23T10:32:04.415 INFO:teuthology.orchestra.run.smithi114.stdout: "type_data": { 2024-04-23T10:32:04.416 INFO:teuthology.orchestra.run.smithi114.stdout: "result": -2147483648, 2024-04-23T10:32:04.416 INFO:teuthology.orchestra.run.smithi114.stdout: "flag_point": "failed to authpin, dir is being fragmented", 2024-04-23T10:32:04.416 INFO:teuthology.orchestra.run.smithi114.stdout: "reqid": { 2024-04-23T10:32:04.416 INFO:teuthology.orchestra.run.smithi114.stdout: "entity": { 2024-04-23T10:32:04.416 INFO:teuthology.orchestra.run.smithi114.stdout: "type": "mds", 2024-04-23T10:32:04.416 INFO:teuthology.orchestra.run.smithi114.stdout: "num": 1 2024-04-23T10:32:04.416 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-23T10:32:04.416 INFO:teuthology.orchestra.run.smithi114.stdout: "tid": 126 2024-04-23T10:32:04.416 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-23T10:32:04.416 INFO:teuthology.orchestra.run.smithi114.stdout: "op_type": "internal_op", 2024-04-23T10:32:04.416 INFO:teuthology.orchestra.run.smithi114.stdout: "internal_op": 5385, 2024-04-23T10:32:04.416 INFO:teuthology.orchestra.run.smithi114.stdout: "op_name": "quiesce_inode", 2024-04-23T10:32:04.416 INFO:teuthology.orchestra.run.smithi114.stdout: "events": [ 2024-04-23T10:32:04.416 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-04-23T10:32:04.416 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-04-23T10:31:21.494106+0000", 2024-04-23T10:32:04.417 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "initiated" 2024-04-23T10:32:04.417 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-23T10:32:04.417 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-04-23T10:32:04.417 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-04-23T10:31:21.494106+0000", 2024-04-23T10:32:04.417 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "header_read" 2024-04-23T10:32:04.417 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-23T10:32:04.417 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-04-23T10:32:04.417 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-04-23T10:31:21.494106+0000", 2024-04-23T10:32:04.417 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "throttled" 2024-04-23T10:32:04.417 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-23T10:32:04.417 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-04-23T10:32:04.417 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-04-23T10:31:21.494106+0000", 2024-04-23T10:32:04.417 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "all_read" 2024-04-23T10:32:04.417 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-23T10:32:04.418 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-04-23T10:32:04.418 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-04-23T10:31:21.494106+0000", 2024-04-23T10:32:04.418 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "dispatched" 2024-04-23T10:32:04.418 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-23T10:32:04.418 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-04-23T10:32:04.418 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-04-23T10:31:21.494900+0000", 2024-04-23T10:32:04.418 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "failed to authpin, dir is being fragmented" 2024-04-23T10:32:04.418 INFO:teuthology.orchestra.run.smithi114.stdout: } 2024-04-23T10:32:04.418 INFO:teuthology.orchestra.run.smithi114.stdout: ], 2024-04-23T10:32:04.418 INFO:teuthology.orchestra.run.smithi114.stdout: "locks": [] 2024-04-23T10:32:04.418 INFO:teuthology.orchestra.run.smithi114.stdout: } 2024-04-23T10:32:04.418 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-23T10:32:04.418 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-04-23T10:32:04.418 INFO:teuthology.orchestra.run.smithi114.stdout: "description": "internal op quiesce_inode:mds.1:127 fp=#0x200000006f3", 2024-04-23T10:32:04.419 INFO:teuthology.orchestra.run.smithi114.stdout: "initiated_at": "2024-04-23T10:31:21.494119+0000", 2024-04-23T10:32:04.419 INFO:teuthology.orchestra.run.smithi114.stdout: "age": 42.363519453000002, 2024-04-23T10:32:04.419 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 42.365492904, 2024-04-23T10:32:04.419 INFO:teuthology.orchestra.run.smithi114.stdout: "continuous": true, 2024-04-23T10:32:04.419 INFO:teuthology.orchestra.run.smithi114.stdout: "type_data": { 2024-04-23T10:32:04.419 INFO:teuthology.orchestra.run.smithi114.stdout: "result": -2147483648, 2024-04-23T10:32:04.419 INFO:teuthology.orchestra.run.smithi114.stdout: "flag_point": "failed to authpin, dir is being fragmented", 2024-04-23T10:32:04.419 INFO:teuthology.orchestra.run.smithi114.stdout: "reqid": { 2024-04-23T10:32:04.419 INFO:teuthology.orchestra.run.smithi114.stdout: "entity": { 2024-04-23T10:32:04.419 INFO:teuthology.orchestra.run.smithi114.stdout: "type": "mds", 2024-04-23T10:32:04.419 INFO:teuthology.orchestra.run.smithi114.stdout: "num": 1 2024-04-23T10:32:04.419 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-23T10:32:04.419 INFO:teuthology.orchestra.run.smithi114.stdout: "tid": 127 2024-04-23T10:32:04.419 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-23T10:32:04.420 INFO:teuthology.orchestra.run.smithi114.stdout: "op_type": "internal_op", 2024-04-23T10:32:04.420 INFO:teuthology.orchestra.run.smithi114.stdout: "internal_op": 5385, 2024-04-23T10:32:04.420 INFO:teuthology.orchestra.run.smithi114.stdout: "op_name": "quiesce_inode", 2024-04-23T10:32:04.420 INFO:teuthology.orchestra.run.smithi114.stdout: "events": [ 2024-04-23T10:32:04.420 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-04-23T10:32:04.420 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-04-23T10:31:21.494119+0000", 2024-04-23T10:32:04.420 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "initiated" 2024-04-23T10:32:04.420 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-23T10:32:04.420 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-04-23T10:32:04.420 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-04-23T10:31:21.494119+0000", 2024-04-23T10:32:04.420 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "header_read" 2024-04-23T10:32:04.420 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-23T10:32:04.420 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-04-23T10:32:04.420 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-04-23T10:31:21.494119+0000", 2024-04-23T10:32:04.420 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "throttled" 2024-04-23T10:32:04.421 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-23T10:32:04.421 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-04-23T10:32:04.421 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-04-23T10:31:21.494119+0000", 2024-04-23T10:32:04.421 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "all_read" 2024-04-23T10:32:04.421 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-23T10:32:04.421 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-04-23T10:32:04.421 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-04-23T10:31:21.494119+0000", 2024-04-23T10:32:04.421 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "dispatched" 2024-04-23T10:32:04.421 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-23T10:32:04.421 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-04-23T10:32:04.421 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-04-23T10:31:21.494960+0000", 2024-04-23T10:32:04.421 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "failed to authpin, dir is being fragmented" 2024-04-23T10:32:04.421 INFO:teuthology.orchestra.run.smithi114.stdout: } 2024-04-23T10:32:04.421 INFO:teuthology.orchestra.run.smithi114.stdout: ], 2024-04-23T10:32:04.421 INFO:teuthology.orchestra.run.smithi114.stdout: "locks": [] 2024-04-23T10:32:04.422 INFO:teuthology.orchestra.run.smithi114.stdout: } 2024-04-23T10:32:04.422 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-23T10:32:04.422 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-04-23T10:32:04.422 INFO:teuthology.orchestra.run.smithi114.stdout: "description": "internal op quiesce_inode:mds.1:128 fp=#0x1000000022f", 2024-04-23T10:32:04.422 INFO:teuthology.orchestra.run.smithi114.stdout: "initiated_at": "2024-04-23T10:31:21.494154+0000", 2024-04-23T10:32:04.422 INFO:teuthology.orchestra.run.smithi114.stdout: "age": 42.363484606999997, 2024-04-23T10:32:04.422 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 42.365482450000002, 2024-04-23T10:32:04.422 INFO:teuthology.orchestra.run.smithi114.stdout: "continuous": true, 2024-04-23T10:32:04.422 INFO:teuthology.orchestra.run.smithi114.stdout: "type_data": { 2024-04-23T10:32:04.422 INFO:teuthology.orchestra.run.smithi114.stdout: "result": -2147483648, 2024-04-23T10:32:04.422 INFO:teuthology.orchestra.run.smithi114.stdout: "flag_point": "failed to authpin, dir is being fragmented", 2024-04-23T10:32:04.422 INFO:teuthology.orchestra.run.smithi114.stdout: "reqid": { 2024-04-23T10:32:04.422 INFO:teuthology.orchestra.run.smithi114.stdout: "entity": { 2024-04-23T10:32:04.422 INFO:teuthology.orchestra.run.smithi114.stdout: "type": "mds", 2024-04-23T10:32:04.423 INFO:teuthology.orchestra.run.smithi114.stdout: "num": 1 2024-04-23T10:32:04.423 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-23T10:32:04.423 INFO:teuthology.orchestra.run.smithi114.stdout: "tid": 128 2024-04-23T10:32:04.423 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-23T10:32:04.423 INFO:teuthology.orchestra.run.smithi114.stdout: "op_type": "internal_op", 2024-04-23T10:32:04.423 INFO:teuthology.orchestra.run.smithi114.stdout: "internal_op": 5385, 2024-04-23T10:32:04.423 INFO:teuthology.orchestra.run.smithi114.stdout: "op_name": "quiesce_inode", 2024-04-23T10:32:04.423 INFO:teuthology.orchestra.run.smithi114.stdout: "events": [ 2024-04-23T10:32:04.423 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-04-23T10:32:04.423 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-04-23T10:31:21.494154+0000", 2024-04-23T10:32:04.423 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "initiated" 2024-04-23T10:32:04.423 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-23T10:32:04.423 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-04-23T10:32:04.423 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-04-23T10:31:21.494154+0000", 2024-04-23T10:32:04.423 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "header_read" 2024-04-23T10:32:04.424 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-23T10:32:04.424 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-04-23T10:32:04.424 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-04-23T10:31:21.494154+0000", 2024-04-23T10:32:04.424 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "throttled" 2024-04-23T10:32:04.424 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-23T10:32:04.424 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-04-23T10:32:04.424 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-04-23T10:31:21.494154+0000", 2024-04-23T10:32:04.424 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "all_read" 2024-04-23T10:32:04.424 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-23T10:32:04.424 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-04-23T10:32:04.424 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-04-23T10:31:21.494154+0000", 2024-04-23T10:32:04.424 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "dispatched" 2024-04-23T10:32:04.424 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-23T10:32:04.424 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-04-23T10:32:04.425 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-04-23T10:31:21.495048+0000", 2024-04-23T10:32:04.425 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "failed to authpin, dir is being fragmented" 2024-04-23T10:32:04.425 INFO:teuthology.orchestra.run.smithi114.stdout: } 2024-04-23T10:32:04.425 INFO:teuthology.orchestra.run.smithi114.stdout: ], 2024-04-23T10:32:04.425 INFO:teuthology.orchestra.run.smithi114.stdout: "locks": [] 2024-04-23T10:32:04.425 INFO:teuthology.orchestra.run.smithi114.stdout: } 2024-04-23T10:32:04.425 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-23T10:32:04.425 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-04-23T10:32:04.425 INFO:teuthology.orchestra.run.smithi114.stdout: "description": "internal op quiesce_inode:mds.1:129 fp=#0x1000000023b", 2024-04-23T10:32:04.425 INFO:teuthology.orchestra.run.smithi114.stdout: "initiated_at": "2024-04-23T10:31:21.494170+0000", 2024-04-23T10:32:04.425 INFO:teuthology.orchestra.run.smithi114.stdout: "age": 42.363468840000003, 2024-04-23T10:32:04.425 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 42.365558710000002, 2024-04-23T10:32:04.425 INFO:teuthology.orchestra.run.smithi114.stdout: "continuous": true, 2024-04-23T10:32:04.425 INFO:teuthology.orchestra.run.smithi114.stdout: "type_data": { 2024-04-23T10:32:04.425 INFO:teuthology.orchestra.run.smithi114.stdout: "result": -2147483648, 2024-04-23T10:32:04.426 INFO:teuthology.orchestra.run.smithi114.stdout: "flag_point": "failed to authpin, dir is being fragmented", 2024-04-23T10:32:04.426 INFO:teuthology.orchestra.run.smithi114.stdout: "reqid": { 2024-04-23T10:32:04.426 INFO:teuthology.orchestra.run.smithi114.stdout: "entity": { 2024-04-23T10:32:04.426 INFO:teuthology.orchestra.run.smithi114.stdout: "type": "mds", 2024-04-23T10:32:04.426 INFO:teuthology.orchestra.run.smithi114.stdout: "num": 1 2024-04-23T10:32:04.426 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-23T10:32:04.426 INFO:teuthology.orchestra.run.smithi114.stdout: "tid": 129 2024-04-23T10:32:04.426 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-23T10:32:04.426 INFO:teuthology.orchestra.run.smithi114.stdout: "op_type": "internal_op", 2024-04-23T10:32:04.426 INFO:teuthology.orchestra.run.smithi114.stdout: "internal_op": 5385, 2024-04-23T10:32:04.426 INFO:teuthology.orchestra.run.smithi114.stdout: "op_name": "quiesce_inode", 2024-04-23T10:32:04.426 INFO:teuthology.orchestra.run.smithi114.stdout: "events": [ 2024-04-23T10:32:04.426 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-04-23T10:32:04.426 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-04-23T10:31:21.494170+0000", 2024-04-23T10:32:04.427 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "initiated" 2024-04-23T10:32:04.427 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-23T10:32:04.427 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-04-23T10:32:04.427 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-04-23T10:31:21.494170+0000", 2024-04-23T10:32:04.427 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "header_read" 2024-04-23T10:32:04.427 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-23T10:32:04.427 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-04-23T10:32:04.427 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-04-23T10:31:21.494170+0000", 2024-04-23T10:32:04.427 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "throttled" 2024-04-23T10:32:04.427 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-23T10:32:04.427 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-04-23T10:32:04.427 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-04-23T10:31:21.494170+0000", 2024-04-23T10:32:04.427 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "all_read" 2024-04-23T10:32:04.428 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-23T10:32:04.428 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-04-23T10:32:04.428 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-04-23T10:31:21.494170+0000", 2024-04-23T10:32:04.428 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "dispatched" 2024-04-23T10:32:04.428 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-23T10:32:04.428 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-04-23T10:32:04.428 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-04-23T10:31:21.495132+0000", 2024-04-23T10:32:04.428 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "failed to authpin, dir is being fragmented" 2024-04-23T10:32:04.428 INFO:teuthology.orchestra.run.smithi114.stdout: } 2024-04-23T10:32:04.428 INFO:teuthology.orchestra.run.smithi114.stdout: ], 2024-04-23T10:32:04.428 INFO:teuthology.orchestra.run.smithi114.stdout: "locks": [] 2024-04-23T10:32:04.428 INFO:teuthology.orchestra.run.smithi114.stdout: } 2024-04-23T10:32:04.428 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-23T10:32:04.428 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-04-23T10:32:04.429 INFO:teuthology.orchestra.run.smithi114.stdout: "description": "internal op quiesce_inode:mds.1:130 fp=#0x1000000023c", 2024-04-23T10:32:04.429 INFO:teuthology.orchestra.run.smithi114.stdout: "initiated_at": "2024-04-23T10:31:21.494185+0000", 2024-04-23T10:32:04.429 INFO:teuthology.orchestra.run.smithi114.stdout: "age": 42.363453972000002, 2024-04-23T10:32:04.429 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 42.365585023000001, 2024-04-23T10:32:04.429 INFO:teuthology.orchestra.run.smithi114.stdout: "continuous": true, 2024-04-23T10:32:04.429 INFO:teuthology.orchestra.run.smithi114.stdout: "type_data": { 2024-04-23T10:32:04.429 INFO:teuthology.orchestra.run.smithi114.stdout: "result": -2147483648, 2024-04-23T10:32:04.429 INFO:teuthology.orchestra.run.smithi114.stdout: "flag_point": "failed to authpin, dir is being fragmented", 2024-04-23T10:32:04.429 INFO:teuthology.orchestra.run.smithi114.stdout: "reqid": { 2024-04-23T10:32:04.429 INFO:teuthology.orchestra.run.smithi114.stdout: "entity": { 2024-04-23T10:32:04.429 INFO:teuthology.orchestra.run.smithi114.stdout: "type": "mds", 2024-04-23T10:32:04.429 INFO:teuthology.orchestra.run.smithi114.stdout: "num": 1 2024-04-23T10:32:04.429 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-23T10:32:04.429 INFO:teuthology.orchestra.run.smithi114.stdout: "tid": 130 2024-04-23T10:32:04.430 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-23T10:32:04.430 INFO:teuthology.orchestra.run.smithi114.stdout: "op_type": "internal_op", 2024-04-23T10:32:04.430 INFO:teuthology.orchestra.run.smithi114.stdout: "internal_op": 5385, 2024-04-23T10:32:04.430 INFO:teuthology.orchestra.run.smithi114.stdout: "op_name": "quiesce_inode", 2024-04-23T10:32:04.430 INFO:teuthology.orchestra.run.smithi114.stdout: "events": [ 2024-04-23T10:32:04.430 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-04-23T10:32:04.430 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-04-23T10:31:21.494185+0000", 2024-04-23T10:32:04.430 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "initiated" 2024-04-23T10:32:04.430 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-23T10:32:04.430 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-04-23T10:32:04.430 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-04-23T10:31:21.494185+0000", 2024-04-23T10:32:04.430 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "header_read" 2024-04-23T10:32:04.430 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-23T10:32:04.431 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-04-23T10:32:04.431 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-04-23T10:31:21.494185+0000", 2024-04-23T10:32:04.431 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "throttled" 2024-04-23T10:32:04.431 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-23T10:32:04.431 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-04-23T10:32:04.431 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-04-23T10:31:21.494185+0000", 2024-04-23T10:32:04.431 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "all_read" 2024-04-23T10:32:04.431 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-23T10:32:04.431 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-04-23T10:32:04.431 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-04-23T10:31:21.494185+0000", 2024-04-23T10:32:04.431 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "dispatched" 2024-04-23T10:32:04.431 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-23T10:32:04.431 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-04-23T10:32:04.431 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-04-23T10:31:21.495209+0000", 2024-04-23T10:32:04.431 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "failed to authpin, dir is being fragmented" 2024-04-23T10:32:04.431 INFO:teuthology.orchestra.run.smithi114.stdout: } 2024-04-23T10:32:04.432 INFO:teuthology.orchestra.run.smithi114.stdout: ], 2024-04-23T10:32:04.432 INFO:teuthology.orchestra.run.smithi114.stdout: "locks": [] 2024-04-23T10:32:04.432 INFO:teuthology.orchestra.run.smithi114.stdout: } 2024-04-23T10:32:04.432 INFO:teuthology.orchestra.run.smithi114.stdout: } 2024-04-23T10:32:04.432 INFO:teuthology.orchestra.run.smithi114.stdout: ], 2024-04-23T10:32:04.432 INFO:teuthology.orchestra.run.smithi114.stdout: "num_ops": 47 2024-04-23T10:32:04.432 INFO:teuthology.orchestra.run.smithi114.stdout:} 2024-04-23T10:32:04.443 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:32:03.860+0000 7f0fe81f4740 1 Processor -- start 2024-04-23T10:32:04.443 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:32:03.860+0000 7f0fe81f4740 1 -- start start 2024-04-23T10:32:04.444 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:32:03.860+0000 7f0fe81f4740 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x55c8d0ee0ba0 0x55c8d0f02cc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T10:32:04.444 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:32:03.860+0000 7f0fe81f4740 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55c8d0ee14c0 0x55c8d0f08a10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T10:32:04.444 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:32:03.860+0000 7f0fe81f4740 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x55c8d0ef00a0 0x55c8d0f6a910 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T10:32:04.444 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:32:03.860+0000 7f0fe81f4740 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x55c8d0f68800 con 0x55c8d0ee14c0 2024-04-23T10:32:04.444 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:32:03.860+0000 7f0fe81f4740 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 -- 0x55c8d0ee0010 con 0x55c8d0ee0ba0 2024-04-23T10:32:04.444 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:32:03.860+0000 7f0fe81f4740 1 -- --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_getmap magic: 0 -- 0x55c8d0ee76b0 con 0x55c8d0ef00a0 2024-04-23T10:32:04.444 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:32:03.860+0000 7f0fde7fc640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55c8d0ee14c0 0x55c8d0f08a10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T10:32:04.444 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:32:03.860+0000 7f0fde7fc640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55c8d0ee14c0 0x55c8d0f08a10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.114:52044/0 (socket says 172.21.15.114:52044) 2024-04-23T10:32:04.444 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:32:03.860+0000 7f0fde7fc640 1 -- 172.21.15.114:0/2950167532 learned_addr learned my addr 172.21.15.114:0/2950167532 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-04-23T10:32:04.444 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:32:03.860+0000 7f0fdeffd640 1 --2- 172.21.15.114:0/2950167532 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x55c8d0ee0ba0 0x55c8d0f02cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T10:32:04.444 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:32:03.860+0000 7f0fde7fc640 1 -- 172.21.15.114:0/2950167532 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x55c8d0ef00a0 msgr2=0x55c8d0f6a910 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T10:32:04.444 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:32:03.860+0000 7f0fde7fc640 1 --2- 172.21.15.114:0/2950167532 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x55c8d0ef00a0 0x55c8d0f6a910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:32:04.444 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:32:03.860+0000 7f0fde7fc640 1 -- 172.21.15.114:0/2950167532 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x55c8d0ee0ba0 msgr2=0x55c8d0f02cc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T10:32:04.444 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:32:03.860+0000 7f0fde7fc640 1 --2- 172.21.15.114:0/2950167532 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x55c8d0ee0ba0 0x55c8d0f02cc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:32:04.444 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:32:03.860+0000 7f0fde7fc640 1 -- 172.21.15.114:0/2950167532 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55c8d0f09070 con 0x55c8d0ee14c0 2024-04-23T10:32:04.445 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:32:03.860+0000 7f0fde7fc640 1 --2- 172.21.15.114:0/2950167532 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55c8d0ee14c0 0x55c8d0f08a10 secure :-1 s=READY pgs=722 cs=0 l=1 rev1=1 crypto rx=0x7f0fd4014040 tx=0x7f0fd4005af0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7fd3ca4353bacd60 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T10:32:04.445 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:32:03.861+0000 7f0fc57fa640 1 -- 172.21.15.114:0/2950167532 <== mon.0 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0fd4064040 con 0x55c8d0ee14c0 2024-04-23T10:32:04.445 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:32:03.861+0000 7f0fc57fa640 1 -- 172.21.15.114:0/2950167532 <== mon.0 v2:172.21.15.114:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f0fd4012cd0 con 0x55c8d0ee14c0 2024-04-23T10:32:04.445 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:32:03.861+0000 7f0fc57fa640 1 -- 172.21.15.114:0/2950167532 <== mon.0 v2:172.21.15.114:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0fd402f040 con 0x55c8d0ee14c0 2024-04-23T10:32:04.445 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:32:03.861+0000 7f0fe81f4740 1 -- 172.21.15.114:0/2950167532 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55c8d0ee14c0 msgr2=0x55c8d0f08a10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T10:32:04.445 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:32:03.861+0000 7f0fe81f4740 1 --2- 172.21.15.114:0/2950167532 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55c8d0ee14c0 0x55c8d0f08a10 secure :-1 s=READY pgs=722 cs=0 l=1 rev1=1 crypto rx=0x7f0fd4014040 tx=0x7f0fd4005af0 comp rx=0 tx=0).stop 2024-04-23T10:32:04.445 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:32:03.861+0000 7f0fe81f4740 1 -- 172.21.15.114:0/2950167532 shutdown_connections 2024-04-23T10:32:04.445 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:32:03.861+0000 7f0fe81f4740 1 --2- 172.21.15.114:0/2950167532 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x55c8d0ef00a0 0x55c8d0f6a910 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:32:04.445 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:32:03.861+0000 7f0fe81f4740 1 --2- 172.21.15.114:0/2950167532 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55c8d0ee14c0 0x55c8d0f08a10 unknown :-1 s=CLOSED pgs=722 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:32:04.445 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:32:03.861+0000 7f0fe81f4740 1 --2- 172.21.15.114:0/2950167532 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x55c8d0ee0ba0 0x55c8d0f02cc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:32:04.445 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:32:03.861+0000 7f0fe81f4740 1 -- 172.21.15.114:0/2950167532 >> 172.21.15.114:0/2950167532 conn(0x55c8d0edcfc0 msgr2=0x55c8d0ee8460 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T10:32:04.445 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:32:03.861+0000 7f0fe81f4740 1 -- 172.21.15.114:0/2950167532 shutdown_connections 2024-04-23T10:32:04.445 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:32:03.861+0000 7f0fe81f4740 1 -- 172.21.15.114:0/2950167532 wait complete. 2024-04-23T10:32:04.445 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:32:03.861+0000 7f0fe81f4740 1 -- start start 2024-04-23T10:32:04.445 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:32:03.861+0000 7f0fe81f4740 1 Processor -- start 2024-04-23T10:32:04.446 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:32:03.862+0000 7f0fe81f4740 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x55c8d0ee0ba0 0x55c8d0ee9400 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T10:32:04.446 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:32:03.862+0000 7f0fe81f4740 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55c8d0ee14c0 0x55c8d0f08a10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T10:32:04.446 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:32:03.862+0000 7f0fe81f4740 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x55c8d0ef00a0 0x55c8d0f02cc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T10:32:04.446 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:32:03.862+0000 7f0fe81f4740 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x55c8d0f6a790 con 0x55c8d0ee14c0 2024-04-23T10:32:04.446 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:32:03.862+0000 7f0fe81f4740 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 -- 0x55c8d0eefb90 con 0x55c8d0ee0ba0 2024-04-23T10:32:04.446 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:32:03.862+0000 7f0fe81f4740 1 -- --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_getmap magic: 0 -- 0x55c8d0f0f660 con 0x55c8d0ef00a0 2024-04-23T10:32:04.446 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:32:03.862+0000 7f0fe81f4740 10 client.0 _ll_register_callbacks cb 0x7f0fd0005cd0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-23T10:32:04.446 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:32:03.862+0000 7f0fde7fc640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55c8d0ee14c0 0x55c8d0f08a10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T10:32:04.446 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:32:03.862+0000 7f0fde7fc640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55c8d0ee14c0 0x55c8d0f08a10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.114:52056/0 (socket says 172.21.15.114:52056) 2024-04-23T10:32:04.446 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:32:03.862+0000 7f0fde7fc640 1 -- 172.21.15.114:0/4155829205 learned_addr learned my addr 172.21.15.114:0/4155829205 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-04-23T10:32:04.446 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:32:03.862+0000 7f0fdeffd640 1 --2- 172.21.15.114:0/4155829205 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x55c8d0ee0ba0 0x55c8d0ee9400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T10:32:04.446 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:32:03.862+0000 7f0fdf7fe640 1 --2- 172.21.15.114:0/4155829205 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x55c8d0ef00a0 0x55c8d0f02cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T10:32:04.446 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:32:03.862+0000 7f0fde7fc640 1 -- 172.21.15.114:0/4155829205 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x55c8d0ef00a0 msgr2=0x55c8d0f02cc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T10:32:04.446 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:32:03.862+0000 7f0fde7fc640 1 --2- 172.21.15.114:0/4155829205 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x55c8d0ef00a0 0x55c8d0f02cc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:32:04.446 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:32:03.862+0000 7f0fde7fc640 1 -- 172.21.15.114:0/4155829205 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x55c8d0ee0ba0 msgr2=0x55c8d0ee9400 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T10:32:04.447 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:32:03.862+0000 7f0fde7fc640 1 --2- 172.21.15.114:0/4155829205 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x55c8d0ee0ba0 0x55c8d0ee9400 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:32:04.447 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:32:03.862+0000 7f0fde7fc640 1 -- 172.21.15.114:0/4155829205 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55c8d0edde90 con 0x55c8d0ee14c0 2024-04-23T10:32:04.447 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:32:03.862+0000 7f0fdf7fe640 1 --2- 172.21.15.114:0/4155829205 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x55c8d0ef00a0 0x55c8d0f02cc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-23T10:32:04.447 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:32:03.862+0000 7f0fde7fc640 1 --2- 172.21.15.114:0/4155829205 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55c8d0ee14c0 0x55c8d0f08a10 secure :-1 s=READY pgs=723 cs=0 l=1 rev1=1 crypto rx=0x7f0fd4009880 tx=0x7f0fd4009cf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b956cf53ff32b595 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T10:32:04.447 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:32:03.862+0000 7f0fc2ff5640 10 client.0 ms_handle_connect on v2:172.21.15.114:3300/0 2024-04-23T10:32:04.447 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:32:03.862+0000 7f0fc2ff5640 1 -- 172.21.15.114:0/4155829205 <== mon.0 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0fd4064070 con 0x55c8d0ee14c0 2024-04-23T10:32:04.447 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:32:03.862+0000 7f0fc2ff5640 1 -- 172.21.15.114:0/4155829205 <== mon.0 v2:172.21.15.114:3300/0 2 ==== osd_map(94..94 src has 1..94) ==== 8585+0+0 (secure 0 0 0) 0x7f0fd4009420 con 0x55c8d0ee14c0 2024-04-23T10:32:04.447 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:32:03.862+0000 7f0fe81f4740 1 -- 172.21.15.114:0/4155829205 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f0fd4012cd0 con 0x55c8d0ee14c0 2024-04-23T10:32:04.447 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:32:03.862+0000 7f0fc2ff5640 1 -- 172.21.15.114:0/4155829205 <== mon.0 v2:172.21.15.114:3300/0 3 ==== mon_get_version_reply(handle=1 version=30) ==== 24+0+0 (secure 0 0 0) 0x7f0fd402f070 con 0x55c8d0ee14c0 2024-04-23T10:32:04.447 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:32:03.862+0000 7f0fe81f4740 10 client.16350 fetch_fsmap learned FSMap version 30 2024-04-23T10:32:04.447 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:32:03.862+0000 7f0fe81f4740 1 -- 172.21.15.114:0/4155829205 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({config=0+,fsmap=30,monmap=4+}) -- 0x55c8d0edde90 con 0x55c8d0ee14c0 2024-04-23T10:32:04.447 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:32:03.863+0000 7f0fc2ff5640 1 -- 172.21.15.114:0/4155829205 <== mon.0 v2:172.21.15.114:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f0fd4012cd0 con 0x55c8d0ee14c0 2024-04-23T10:32:04.447 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:32:03.863+0000 7f0fc2ff5640 1 -- 172.21.15.114:0/4155829205 <== mon.0 v2:172.21.15.114:3300/0 5 ==== fsmap(e 30) ==== 8338+0+0 (secure 0 0 0) 0x7f0fd4036a70 con 0x55c8d0ee14c0 2024-04-23T10:32:04.447 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:32:03.863+0000 7f0fe81f4740 10 client.16350 fetch_fsmap finished waiting for FSMap version 30 2024-04-23T10:32:04.447 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:32:03.863+0000 7f0fe81f4740 10 client.16350 resolve_mds: resolved name 'b' to daemon mds.b 2024-04-23T10:32:04.448 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:32:03.863+0000 7f0fe81f4740 20 client.16350 populate_metadata read hostname 'smithi114' 2024-04-23T10:32:04.448 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:32:03.863+0000 7f0fe81f4740 1 --2- 172.21.15.114:0/4155829205 >> [v2:172.21.15.119:6838/937297340,v1:172.21.15.119:6839/937297340] conn(0x55c8d0f7ac70 0x55c8d0f7b050 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T10:32:04.448 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:32:03.863+0000 7f0fe81f4740 4 client.16350 mds_command: new command op to 24545 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-23T10:32:04.448 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:32:03.863+0000 7f0fe81f4740 1 -- 172.21.15.114:0/4155829205 --> [v2:172.21.15.119:6838/937297340,v1:172.21.15.119:6839/937297340] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55c8d0edde90 con 0x55c8d0f7ac70 2024-04-23T10:32:04.448 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:32:03.863+0000 7f0fdeffd640 1 --2- 172.21.15.114:0/4155829205 >> [v2:172.21.15.119:6838/937297340,v1:172.21.15.119:6839/937297340] conn(0x55c8d0f7ac70 0x55c8d0f7b050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T10:32:04.448 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:32:03.863+0000 7f0fc2ff5640 10 client.16350 ms_handle_connect on v2:172.21.15.119:6838/937297340 2024-04-23T10:32:04.448 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:32:03.863+0000 7f0fdeffd640 1 --2- 172.21.15.114:0/4155829205 >> [v2:172.21.15.119:6838/937297340,v1:172.21.15.119:6839/937297340] conn(0x55c8d0f7ac70 0x55c8d0f7b050 crc :-1 s=READY pgs=106 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=506baf2459c1a724 server_cookie=50201a7316d1d707 in_seq=0 out_seq=0 2024-04-23T10:32:04.448 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:32:03.864+0000 7f0fc2ff5640 1 -- 172.21.15.114:0/4155829205 <== mds.0 v2:172.21.15.119:6838/937297340 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x55c8d0edde90 con 0x55c8d0f7ac70 2024-04-23T10:32:04.448 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:32:03.864+0000 7f0fc2ff5640 10 client.16350 handle_command_reply: tid=0 2024-04-23T10:32:04.448 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:32:03.864+0000 7f0fe81f4740 1 client.16350 shutdown 2024-04-23T10:32:04.448 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:32:03.864+0000 7f0fe81f4740 1 -- 172.21.15.114:0/4155829205 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55c8d0ee14c0 msgr2=0x55c8d0f08a10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T10:32:04.448 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:32:03.864+0000 7f0fe81f4740 1 --2- 172.21.15.114:0/4155829205 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55c8d0ee14c0 0x55c8d0f08a10 secure :-1 s=READY pgs=723 cs=0 l=1 rev1=1 crypto rx=0x7f0fd4009880 tx=0x7f0fd4009cf0 comp rx=0 tx=0).stop 2024-04-23T10:32:04.448 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:32:03.864+0000 7f0fe81f4740 1 -- 172.21.15.114:0/4155829205 shutdown_connections 2024-04-23T10:32:04.448 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:32:03.864+0000 7f0fe81f4740 1 --2- 172.21.15.114:0/4155829205 >> [v2:172.21.15.119:6838/937297340,v1:172.21.15.119:6839/937297340] conn(0x55c8d0f7ac70 0x55c8d0f7b050 crc :-1 s=READY pgs=106 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:32:04.448 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:32:03.864+0000 7f0fe81f4740 1 --2- 172.21.15.114:0/4155829205 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x55c8d0ef00a0 0x55c8d0f02cc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:32:04.449 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:32:03.864+0000 7f0fe81f4740 1 --2- 172.21.15.114:0/4155829205 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55c8d0ee14c0 0x55c8d0f08a10 unknown :-1 s=CLOSED pgs=723 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:32:04.449 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:32:03.864+0000 7f0fc2ff5640 0 client.16350 ms_handle_reset on v2:172.21.15.119:6838/937297340 2024-04-23T10:32:04.449 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:32:03.864+0000 7f0fe81f4740 1 --2- 172.21.15.114:0/4155829205 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x55c8d0ee0ba0 0x55c8d0ee9400 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:32:04.449 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:32:03.864+0000 7f0fe81f4740 1 -- 172.21.15.114:0/4155829205 >> 172.21.15.114:0/4155829205 conn(0x55c8d0edcfc0 msgr2=0x55c8d0eed330 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T10:32:04.449 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:32:03.864+0000 7f0fe81f4740 1 -- 172.21.15.114:0/4155829205 shutdown_connections 2024-04-23T10:32:04.449 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:32:03.864+0000 7f0fe81f4740 1 -- 172.21.15.114:0/4155829205 wait complete. 2024-04-23T10:32:04.449 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:32:03.864+0000 7f0fe81f4740 20 client.16350 trim_cache size 0 max 16384 2024-04-23T10:32:04.449 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:32:03.879+0000 7f0fe81f4740 1 Processor -- start 2024-04-23T10:32:04.449 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:32:03.879+0000 7f0fe81f4740 1 -- start start 2024-04-23T10:32:04.449 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:32:03.879+0000 7f0fe81f4740 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x55c8d0ee0ba0 0x55c8d0f02cc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T10:32:04.449 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:32:03.879+0000 7f0fe81f4740 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x55c8d0ee14c0 0x55c8d0f08a10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T10:32:04.449 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:32:03.879+0000 7f0fe81f4740 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55c8d0ef00a0 0x55c8d0ee9e80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T10:32:04.449 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:32:03.879+0000 7f0fe81f4740 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x55c8d0f7cce0 con 0x55c8d0ef00a0 2024-04-23T10:32:04.449 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:32:03.879+0000 7f0fe81f4740 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 -- 0x55c8d0f6bf50 con 0x55c8d0ee0ba0 2024-04-23T10:32:04.449 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:32:03.879+0000 7f0fe81f4740 1 -- --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_getmap magic: 0 -- 0x55c8d0f612b0 con 0x55c8d0ee14c0 2024-04-23T10:32:04.450 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:32:03.880+0000 7f0fdf7fe640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55c8d0ef00a0 0x55c8d0ee9e80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T10:32:04.450 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:32:03.880+0000 7f0fdf7fe640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55c8d0ef00a0 0x55c8d0ee9e80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.114:52070/0 (socket says 172.21.15.114:52070) 2024-04-23T10:32:04.450 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:32:03.880+0000 7f0fdf7fe640 1 -- 172.21.15.114:0/305622436 learned_addr learned my addr 172.21.15.114:0/305622436 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-04-23T10:32:04.450 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:32:03.880+0000 7f0fdeffd640 1 --2- 172.21.15.114:0/305622436 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x55c8d0ee0ba0 0x55c8d0f02cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T10:32:04.450 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:32:03.880+0000 7f0fde7fc640 1 --2- 172.21.15.114:0/305622436 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x55c8d0ee14c0 0x55c8d0f08a10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T10:32:04.450 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:32:03.880+0000 7f0fdf7fe640 1 -- 172.21.15.114:0/305622436 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x55c8d0ee14c0 msgr2=0x55c8d0f08a10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T10:32:04.450 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:32:03.880+0000 7f0fdf7fe640 1 --2- 172.21.15.114:0/305622436 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x55c8d0ee14c0 0x55c8d0f08a10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:32:04.450 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:32:03.880+0000 7f0fdf7fe640 1 -- 172.21.15.114:0/305622436 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x55c8d0ee0ba0 msgr2=0x55c8d0f02cc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T10:32:04.450 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:32:03.880+0000 7f0fdf7fe640 1 --2- 172.21.15.114:0/305622436 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x55c8d0ee0ba0 0x55c8d0f02cc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:32:04.450 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:32:03.880+0000 7f0fdf7fe640 1 -- 172.21.15.114:0/305622436 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55c8d0f09070 con 0x55c8d0ef00a0 2024-04-23T10:32:04.450 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:32:03.880+0000 7f0fde7fc640 1 --2- 172.21.15.114:0/305622436 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x55c8d0ee14c0 0x55c8d0f08a10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-23T10:32:04.450 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:32:03.880+0000 7f0fdf7fe640 1 --2- 172.21.15.114:0/305622436 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55c8d0ef00a0 0x55c8d0ee9e80 secure :-1 s=READY pgs=724 cs=0 l=1 rev1=1 crypto rx=0x7f0fd8098a10 tx=0x7f0fd800df60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=744dac6fddb2e7eb server_cookie=0 in_seq=0 out_seq=0 2024-04-23T10:32:04.450 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:32:03.880+0000 7f0fc4ff9640 1 -- 172.21.15.114:0/305622436 <== mon.0 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0fd80ad030 con 0x55c8d0ef00a0 2024-04-23T10:32:04.450 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:32:03.880+0000 7f0fc4ff9640 1 -- 172.21.15.114:0/305622436 <== mon.0 v2:172.21.15.114:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f0fd800b0b0 con 0x55c8d0ef00a0 2024-04-23T10:32:04.451 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:32:03.880+0000 7f0fc4ff9640 1 -- 172.21.15.114:0/305622436 <== mon.0 v2:172.21.15.114:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0fd808e070 con 0x55c8d0ef00a0 2024-04-23T10:32:04.451 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:32:03.881+0000 7f0fe81f4740 1 -- 172.21.15.114:0/305622436 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55c8d0ef00a0 msgr2=0x55c8d0ee9e80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T10:32:04.451 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:32:03.881+0000 7f0fe81f4740 1 --2- 172.21.15.114:0/305622436 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55c8d0ef00a0 0x55c8d0ee9e80 secure :-1 s=READY pgs=724 cs=0 l=1 rev1=1 crypto rx=0x7f0fd8098a10 tx=0x7f0fd800df60 comp rx=0 tx=0).stop 2024-04-23T10:32:04.451 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:32:03.881+0000 7f0fe81f4740 1 -- 172.21.15.114:0/305622436 shutdown_connections 2024-04-23T10:32:04.451 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:32:03.881+0000 7f0fe81f4740 1 --2- 172.21.15.114:0/305622436 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55c8d0ef00a0 0x55c8d0ee9e80 unknown :-1 s=CLOSED pgs=724 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:32:04.451 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:32:03.881+0000 7f0fe81f4740 1 --2- 172.21.15.114:0/305622436 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x55c8d0ee14c0 0x55c8d0f08a10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:32:04.451 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:32:03.881+0000 7f0fe81f4740 1 --2- 172.21.15.114:0/305622436 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x55c8d0ee0ba0 0x55c8d0f02cc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:32:04.451 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:32:03.881+0000 7f0fe81f4740 1 -- 172.21.15.114:0/305622436 >> 172.21.15.114:0/305622436 conn(0x55c8d0edcfc0 msgr2=0x55c8d0eed330 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T10:32:04.451 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:32:03.881+0000 7f0fe81f4740 1 -- 172.21.15.114:0/305622436 shutdown_connections 2024-04-23T10:32:04.451 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:32:03.881+0000 7f0fe81f4740 1 -- 172.21.15.114:0/305622436 wait complete. 2024-04-23T10:32:04.451 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:32:03.881+0000 7f0fe81f4740 1 -- start start 2024-04-23T10:32:04.451 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:32:03.881+0000 7f0fe81f4740 1 Processor -- start 2024-04-23T10:32:04.451 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:32:03.881+0000 7f0fe81f4740 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x55c8d0ee0ba0 0x55c8d0f08a10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T10:32:04.451 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:32:03.881+0000 7f0fe81f4740 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55c8d0ee14c0 0x55c8d0ee9e80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T10:32:04.451 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:32:03.881+0000 7f0fe81f4740 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x55c8d0ef00a0 0x55c8d0f02cc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T10:32:04.452 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:32:03.881+0000 7f0fe81f4740 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x55c8d0ee0280 con 0x55c8d0ee14c0 2024-04-23T10:32:04.452 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:32:03.881+0000 7f0fe81f4740 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 -- 0x55c8d0eea950 con 0x55c8d0ef00a0 2024-04-23T10:32:04.452 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:32:03.881+0000 7f0fe81f4740 1 -- --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_getmap magic: 0 -- 0x55c8d0f6ac90 con 0x55c8d0ee0ba0 2024-04-23T10:32:04.452 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:32:03.881+0000 7f0fe81f4740 10 client.0 _ll_register_callbacks cb 0x7f0fd0009b20 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-23T10:32:04.452 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:32:03.881+0000 7f0fde7fc640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55c8d0ee14c0 0x55c8d0ee9e80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T10:32:04.452 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:32:03.881+0000 7f0fde7fc640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55c8d0ee14c0 0x55c8d0ee9e80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.114:52072/0 (socket says 172.21.15.114:52072) 2024-04-23T10:32:04.452 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:32:03.881+0000 7f0fde7fc640 1 -- 172.21.15.114:0/3677781094 learned_addr learned my addr 172.21.15.114:0/3677781094 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-04-23T10:32:04.452 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:32:03.881+0000 7f0fdeffd640 1 --2- 172.21.15.114:0/3677781094 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x55c8d0ee0ba0 0x55c8d0f08a10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T10:32:04.452 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:32:03.881+0000 7f0fdf7fe640 1 --2- 172.21.15.114:0/3677781094 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x55c8d0ef00a0 0x55c8d0f02cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T10:32:04.452 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:32:03.881+0000 7f0fde7fc640 1 -- 172.21.15.114:0/3677781094 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x55c8d0ee0ba0 msgr2=0x55c8d0f08a10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T10:32:04.452 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:32:03.881+0000 7f0fde7fc640 1 --2- 172.21.15.114:0/3677781094 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x55c8d0ee0ba0 0x55c8d0f08a10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:32:04.452 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:32:03.881+0000 7f0fde7fc640 1 -- 172.21.15.114:0/3677781094 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x55c8d0ef00a0 msgr2=0x55c8d0f02cc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T10:32:04.453 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:32:03.881+0000 7f0fde7fc640 1 --2- 172.21.15.114:0/3677781094 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x55c8d0ef00a0 0x55c8d0f02cc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:32:04.453 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:32:03.881+0000 7f0fde7fc640 1 -- 172.21.15.114:0/3677781094 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55c8d0edde90 con 0x55c8d0ee14c0 2024-04-23T10:32:04.453 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:32:03.881+0000 7f0fdf7fe640 1 --2- 172.21.15.114:0/3677781094 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x55c8d0ef00a0 0x55c8d0f02cc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-23T10:32:04.453 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:32:03.882+0000 7f0fde7fc640 1 --2- 172.21.15.114:0/3677781094 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55c8d0ee14c0 0x55c8d0ee9e80 secure :-1 s=READY pgs=725 cs=0 l=1 rev1=1 crypto rx=0x55c8d0f0aae0 tx=0x7f0fd405c630 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9d07d6f1e7446492 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T10:32:04.453 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:32:03.882+0000 7f0fc2ff5640 10 client.0 ms_handle_connect on v2:172.21.15.114:3300/0 2024-04-23T10:32:04.453 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:32:03.882+0000 7f0fc2ff5640 1 -- 172.21.15.114:0/3677781094 <== mon.0 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0fd402f070 con 0x55c8d0ee14c0 2024-04-23T10:32:04.453 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:32:03.882+0000 7f0fc2ff5640 1 -- 172.21.15.114:0/3677781094 <== mon.0 v2:172.21.15.114:3300/0 2 ==== osd_map(94..94 src has 1..94) ==== 8585+0+0 (secure 0 0 0) 0x7f0fd4005240 con 0x55c8d0ee14c0 2024-04-23T10:32:04.453 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:32:03.882+0000 7f0fe81f4740 1 -- 172.21.15.114:0/3677781094 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f0fd800b0b0 con 0x55c8d0ee14c0 2024-04-23T10:32:04.453 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:32:03.882+0000 7f0fc2ff5640 1 -- 172.21.15.114:0/3677781094 <== mon.0 v2:172.21.15.114:3300/0 3 ==== mon_get_version_reply(handle=1 version=30) ==== 24+0+0 (secure 0 0 0) 0x7f0fd4064040 con 0x55c8d0ee14c0 2024-04-23T10:32:04.453 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:32:03.882+0000 7f0fe81f4740 10 client.16356 fetch_fsmap learned FSMap version 30 2024-04-23T10:32:04.453 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:32:03.882+0000 7f0fe81f4740 1 -- 172.21.15.114:0/3677781094 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({config=0+,fsmap=30,monmap=4+}) -- 0x55c8d0edde90 con 0x55c8d0ee14c0 2024-04-23T10:32:04.453 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:32:03.882+0000 7f0fc2ff5640 1 -- 172.21.15.114:0/3677781094 <== mon.0 v2:172.21.15.114:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f0fd800b0b0 con 0x55c8d0ee14c0 2024-04-23T10:32:04.453 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:32:03.882+0000 7f0fc2ff5640 1 -- 172.21.15.114:0/3677781094 <== mon.0 v2:172.21.15.114:3300/0 5 ==== fsmap(e 30) ==== 8338+0+0 (secure 0 0 0) 0x7f0fd4036670 con 0x55c8d0ee14c0 2024-04-23T10:32:04.453 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:32:03.882+0000 7f0fe81f4740 10 client.16356 fetch_fsmap finished waiting for FSMap version 30 2024-04-23T10:32:04.453 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:32:03.882+0000 7f0fe81f4740 10 client.16356 resolve_mds: resolved name 'b' to daemon mds.b 2024-04-23T10:32:04.453 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:32:03.882+0000 7f0fe81f4740 20 client.16356 populate_metadata read hostname 'smithi114' 2024-04-23T10:32:04.454 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:32:03.882+0000 7f0fe81f4740 1 --2- 172.21.15.114:0/3677781094 >> [v2:172.21.15.119:6838/937297340,v1:172.21.15.119:6839/937297340] conn(0x55c8d0f7ac70 0x55c8d0f099e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T10:32:04.454 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:32:03.882+0000 7f0fe81f4740 4 client.16356 mds_command: new command op to 24545 tid=0 multi_id=0 [{"prefix": "ops", "flags": ["locks"]}] 2024-04-23T10:32:04.454 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:32:03.882+0000 7f0fe81f4740 1 -- 172.21.15.114:0/3677781094 --> [v2:172.21.15.119:6838/937297340,v1:172.21.15.119:6839/937297340] -- command(tid 0: {"prefix": "ops", "flags": ["locks"]}) -- 0x55c8d0f67390 con 0x55c8d0f7ac70 2024-04-23T10:32:04.454 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:32:03.882+0000 7f0fdeffd640 1 --2- 172.21.15.114:0/3677781094 >> [v2:172.21.15.119:6838/937297340,v1:172.21.15.119:6839/937297340] conn(0x55c8d0f7ac70 0x55c8d0f099e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T10:32:04.454 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:32:03.883+0000 7f0fc2ff5640 10 client.16356 ms_handle_connect on v2:172.21.15.119:6838/937297340 2024-04-23T10:32:04.454 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:32:03.883+0000 7f0fdeffd640 1 --2- 172.21.15.114:0/3677781094 >> [v2:172.21.15.119:6838/937297340,v1:172.21.15.119:6839/937297340] conn(0x55c8d0f7ac70 0x55c8d0f099e0 crc :-1 s=READY pgs=107 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=fc1a6896e32b5561 server_cookie=c32b171777490bfa in_seq=0 out_seq=0 2024-04-23T10:32:04.454 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:32:03.889+0000 7f0fc2ff5640 1 -- 172.21.15.114:0/3677781094 <== mds.0 v2:172.21.15.119:6838/937297340 1 ==== command_reply(tid 0: 0 ) ==== 8+0+310926 (crc 0 0 0) 0x7f0fd0036040 con 0x55c8d0f7ac70 2024-04-23T10:32:04.454 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:32:03.889+0000 7f0fc2ff5640 10 client.16356 handle_command_reply: tid=0 2024-04-23T10:32:04.454 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:32:03.889+0000 7f0fe81f4740 1 client.16356 shutdown 2024-04-23T10:32:04.454 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:32:03.889+0000 7f0fe81f4740 1 -- 172.21.15.114:0/3677781094 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55c8d0ee14c0 msgr2=0x55c8d0ee9e80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T10:32:04.454 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:32:03.889+0000 7f0fe81f4740 1 --2- 172.21.15.114:0/3677781094 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55c8d0ee14c0 0x55c8d0ee9e80 secure :-1 s=READY pgs=725 cs=0 l=1 rev1=1 crypto rx=0x55c8d0f0aae0 tx=0x7f0fd405c630 comp rx=0 tx=0).stop 2024-04-23T10:32:04.454 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:32:03.889+0000 7f0fe81f4740 1 -- 172.21.15.114:0/3677781094 shutdown_connections 2024-04-23T10:32:04.454 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:32:03.889+0000 7f0fe81f4740 1 --2- 172.21.15.114:0/3677781094 >> [v2:172.21.15.119:6838/937297340,v1:172.21.15.119:6839/937297340] conn(0x55c8d0f7ac70 0x55c8d0f099e0 crc :-1 s=READY pgs=107 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:32:04.454 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:32:03.889+0000 7f0fe81f4740 1 --2- 172.21.15.114:0/3677781094 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x55c8d0ef00a0 0x55c8d0f02cc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:32:04.454 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:32:03.889+0000 7f0fc2ff5640 0 client.16356 ms_handle_reset on v2:172.21.15.119:6838/937297340 2024-04-23T10:32:04.455 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:32:03.889+0000 7f0fe81f4740 1 --2- 172.21.15.114:0/3677781094 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55c8d0ee14c0 0x55c8d0ee9e80 unknown :-1 s=CLOSED pgs=725 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:32:04.455 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:32:03.889+0000 7f0fe81f4740 1 --2- 172.21.15.114:0/3677781094 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x55c8d0ee0ba0 0x55c8d0f08a10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:32:04.455 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:32:03.889+0000 7f0fe81f4740 1 -- 172.21.15.114:0/3677781094 >> 172.21.15.114:0/3677781094 conn(0x55c8d0edcfc0 msgr2=0x55c8d0efb240 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T10:32:04.455 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:32:03.889+0000 7f0fe81f4740 1 -- 172.21.15.114:0/3677781094 shutdown_connections 2024-04-23T10:32:04.455 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:32:03.889+0000 7f0fe81f4740 1 -- 172.21.15.114:0/3677781094 wait complete. 2024-04-23T10:32:04.455 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:32:03.889+0000 7f0fe81f4740 20 client.16356 trim_cache size 0 max 16384 2024-04-23T10:32:04.459 INFO:teuthology.orchestra.run.smithi114.stdout:mds.b: { 2024-04-23T10:32:04.459 INFO:teuthology.orchestra.run.smithi114.stdout: "ops": [ 2024-04-23T10:32:04.459 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-04-23T10:32:04.459 INFO:teuthology.orchestra.run.smithi114.stdout: "description": "peer_request:mds.1:110", 2024-04-23T10:32:04.459 INFO:teuthology.orchestra.run.smithi114.stdout: "initiated_at": "2024-04-23T10:31:21.453770+0000", 2024-04-23T10:32:04.459 INFO:teuthology.orchestra.run.smithi114.stdout: "age": 42.430619086999997, 2024-04-23T10:32:04.459 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 42.430634918000003, 2024-04-23T10:32:04.459 INFO:teuthology.orchestra.run.smithi114.stdout: "continuous": false, 2024-04-23T10:32:04.459 INFO:teuthology.orchestra.run.smithi114.stdout: "type_data": { 2024-04-23T10:32:04.459 INFO:teuthology.orchestra.run.smithi114.stdout: "result": -2147483648, 2024-04-23T10:32:04.459 INFO:teuthology.orchestra.run.smithi114.stdout: "flag_point": "dispatched", 2024-04-23T10:32:04.459 INFO:teuthology.orchestra.run.smithi114.stdout: "reqid": { 2024-04-23T10:32:04.459 INFO:teuthology.orchestra.run.smithi114.stdout: "entity": { 2024-04-23T10:32:04.460 INFO:teuthology.orchestra.run.smithi114.stdout: "type": "mds", 2024-04-23T10:32:04.460 INFO:teuthology.orchestra.run.smithi114.stdout: "num": 1 2024-04-23T10:32:04.460 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-23T10:32:04.460 INFO:teuthology.orchestra.run.smithi114.stdout: "tid": 110 2024-04-23T10:32:04.460 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-23T10:32:04.460 INFO:teuthology.orchestra.run.smithi114.stdout: "op_type": "peer_request", 2024-04-23T10:32:04.460 INFO:teuthology.orchestra.run.smithi114.stdout: "leader_info": { 2024-04-23T10:32:04.460 INFO:teuthology.orchestra.run.smithi114.stdout: "leader": "1" 2024-04-23T10:32:04.460 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-23T10:32:04.460 INFO:teuthology.orchestra.run.smithi114.stdout: "events": [ 2024-04-23T10:32:04.460 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-04-23T10:32:04.460 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-04-23T10:31:21.453770+0000", 2024-04-23T10:32:04.460 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "initiated" 2024-04-23T10:32:04.460 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-23T10:32:04.461 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-04-23T10:32:04.461 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-04-23T10:31:21.453770+0000", 2024-04-23T10:32:04.461 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "header_read" 2024-04-23T10:32:04.461 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-23T10:32:04.461 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-04-23T10:32:04.461 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-04-23T10:31:21.453771+0000", 2024-04-23T10:32:04.461 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "throttled" 2024-04-23T10:32:04.461 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-23T10:32:04.461 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-04-23T10:32:04.461 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-04-23T10:31:21.453778+0000", 2024-04-23T10:32:04.461 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "all_read" 2024-04-23T10:32:04.461 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-23T10:32:04.461 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-04-23T10:32:04.461 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-04-23T10:31:21.454984+0000", 2024-04-23T10:32:04.461 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "dispatched" 2024-04-23T10:32:04.462 INFO:teuthology.orchestra.run.smithi114.stdout: } 2024-04-23T10:32:04.462 INFO:teuthology.orchestra.run.smithi114.stdout: ], 2024-04-23T10:32:04.462 INFO:teuthology.orchestra.run.smithi114.stdout: "locks": [] 2024-04-23T10:32:04.462 INFO:teuthology.orchestra.run.smithi114.stdout: } 2024-04-23T10:32:04.462 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-23T10:32:04.462 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-04-23T10:32:04.462 INFO:teuthology.orchestra.run.smithi114.stdout: "description": "internal op quiesce_inode:mds.0:186 fp=#0x100000014db", 2024-04-23T10:32:04.462 INFO:teuthology.orchestra.run.smithi114.stdout: "initiated_at": "2024-04-23T10:31:21.471262+0000", 2024-04-23T10:32:04.462 INFO:teuthology.orchestra.run.smithi114.stdout: "age": 42.413126943999998, 2024-04-23T10:32:04.462 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 42.413170811000001, 2024-04-23T10:32:04.462 INFO:teuthology.orchestra.run.smithi114.stdout: "continuous": true, 2024-04-23T10:32:04.462 INFO:teuthology.orchestra.run.smithi114.stdout: "type_data": { 2024-04-23T10:32:04.462 INFO:teuthology.orchestra.run.smithi114.stdout: "result": -2147483648, 2024-04-23T10:32:04.463 INFO:teuthology.orchestra.run.smithi114.stdout: "flag_point": "quiesce complete", 2024-04-23T10:32:04.463 INFO:teuthology.orchestra.run.smithi114.stdout: "reqid": { 2024-04-23T10:32:04.463 INFO:teuthology.orchestra.run.smithi114.stdout: "entity": { 2024-04-23T10:32:04.463 INFO:teuthology.orchestra.run.smithi114.stdout: "type": "mds", 2024-04-23T10:32:04.463 INFO:teuthology.orchestra.run.smithi114.stdout: "num": 0 2024-04-23T10:32:04.463 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-23T10:32:04.463 INFO:teuthology.orchestra.run.smithi114.stdout: "tid": 186 2024-04-23T10:32:04.463 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-23T10:32:04.463 INFO:teuthology.orchestra.run.smithi114.stdout: "op_type": "internal_op", 2024-04-23T10:32:04.463 INFO:teuthology.orchestra.run.smithi114.stdout: "internal_op": 5385, 2024-04-23T10:32:04.463 INFO:teuthology.orchestra.run.smithi114.stdout: "op_name": "quiesce_inode", 2024-04-23T10:32:04.463 INFO:teuthology.orchestra.run.smithi114.stdout: "events": [ 2024-04-23T10:32:04.463 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-04-23T10:32:04.464 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-04-23T10:31:21.471262+0000", 2024-04-23T10:32:04.464 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "initiated" 2024-04-23T10:32:04.464 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-23T10:32:04.464 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-04-23T10:32:04.464 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-04-23T10:31:21.471262+0000", 2024-04-23T10:32:04.464 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "header_read" 2024-04-23T10:32:04.464 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-23T10:32:04.464 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-04-23T10:32:04.464 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-04-23T10:31:21.471262+0000", 2024-04-23T10:32:04.464 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "throttled" 2024-04-23T10:32:04.464 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-23T10:32:04.464 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-04-23T10:32:04.464 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-04-23T10:31:21.471262+0000", 2024-04-23T10:32:04.464 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "all_read" 2024-04-23T10:32:04.465 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-23T10:32:04.465 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-04-23T10:32:04.465 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-04-23T10:31:21.471262+0000", 2024-04-23T10:32:04.465 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "dispatched" 2024-04-23T10:32:04.465 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-23T10:32:04.465 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-04-23T10:32:04.465 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-04-23T10:31:21.471540+0000", 2024-04-23T10:32:04.465 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "failed to xlock, waiting" 2024-04-23T10:32:04.465 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-23T10:32:04.465 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-04-23T10:32:04.465 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-04-23T10:31:21.498297+0000", 2024-04-23T10:32:04.465 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "failed to xlock, waiting" 2024-04-23T10:32:04.465 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-23T10:32:04.465 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-04-23T10:32:04.466 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-04-23T10:31:21.503163+0000", 2024-04-23T10:32:04.466 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "acquired locks" 2024-04-23T10:32:04.466 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-23T10:32:04.466 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-04-23T10:32:04.466 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-04-23T10:31:21.503306+0000", 2024-04-23T10:32:04.466 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "quiesce complete" 2024-04-23T10:32:04.466 INFO:teuthology.orchestra.run.smithi114.stdout: } 2024-04-23T10:32:04.466 INFO:teuthology.orchestra.run.smithi114.stdout: ], 2024-04-23T10:32:04.466 INFO:teuthology.orchestra.run.smithi114.stdout: "locks": [ 2024-04-23T10:32:04.466 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-04-23T10:32:04.466 INFO:teuthology.orchestra.run.smithi114.stdout: "object": { 2024-04-23T10:32:04.466 INFO:teuthology.orchestra.run.smithi114.stdout: "is_auth": true, 2024-04-23T10:32:04.466 INFO:teuthology.orchestra.run.smithi114.stdout: "auth_state": { 2024-04-23T10:32:04.467 INFO:teuthology.orchestra.run.smithi114.stdout: "replicas": {} 2024-04-23T10:32:04.467 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-23T10:32:04.467 INFO:teuthology.orchestra.run.smithi114.stdout: "replica_state": { 2024-04-23T10:32:04.467 INFO:teuthology.orchestra.run.smithi114.stdout: "authority": [ 2024-04-23T10:32:04.467 INFO:teuthology.orchestra.run.smithi114.stdout: 0, 2024-04-23T10:32:04.467 INFO:teuthology.orchestra.run.smithi114.stdout: -2 2024-04-23T10:32:04.467 INFO:teuthology.orchestra.run.smithi114.stdout: ], 2024-04-23T10:32:04.467 INFO:teuthology.orchestra.run.smithi114.stdout: "replica_nonce": 0 2024-04-23T10:32:04.467 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-23T10:32:04.470 INFO:teuthology.orchestra.run.smithi114.stdout: "auth_pins": 1, 2024-04-23T10:32:04.470 INFO:teuthology.orchestra.run.smithi114.stdout: "is_frozen": false, 2024-04-23T10:32:04.470 INFO:teuthology.orchestra.run.smithi114.stdout: "is_freezing": false, 2024-04-23T10:32:04.470 INFO:teuthology.orchestra.run.smithi114.stdout: "pins": { 2024-04-23T10:32:04.470 INFO:teuthology.orchestra.run.smithi114.stdout: "ptrwaiter": 0, 2024-04-23T10:32:04.470 INFO:teuthology.orchestra.run.smithi114.stdout: "request": 1, 2024-04-23T10:32:04.470 INFO:teuthology.orchestra.run.smithi114.stdout: "lock": 1, 2024-04-23T10:32:04.470 INFO:teuthology.orchestra.run.smithi114.stdout: "caps": 0, 2024-04-23T10:32:04.470 INFO:teuthology.orchestra.run.smithi114.stdout: "dirtyparent": 0, 2024-04-23T10:32:04.470 INFO:teuthology.orchestra.run.smithi114.stdout: "dirty": 0, 2024-04-23T10:32:04.470 INFO:teuthology.orchestra.run.smithi114.stdout: "waiter": 0, 2024-04-23T10:32:04.470 INFO:teuthology.orchestra.run.smithi114.stdout: "authpin": 1 2024-04-23T10:32:04.470 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-23T10:32:04.470 INFO:teuthology.orchestra.run.smithi114.stdout: "nref": 3 2024-04-23T10:32:04.470 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-23T10:32:04.471 INFO:teuthology.orchestra.run.smithi114.stdout: "object_string": "[inode 0x100000014db [7,head] /volumes/qa/sv_1/b206794e-7622-4ab8-8943-f3ae4e65b7a3/client.0/tmp/clients/client0/~dmtmp/ACCESS/INV.PRN auth v3326 ap=1 s=62591 n(v0 rc2024-04-23T10:31:17.015733+0000 b62591 1=1+0) (iquiesce lock x=1 by request(mds.0:186 nref=3)) | ptrwaiter=0 request=1 lock=1 caps=0 dirtyparent=0 dirty=0 waiter=0 authpin=1 0x5571cf822100]", 2024-04-23T10:32:04.471 INFO:teuthology.orchestra.run.smithi114.stdout: "lock": { 2024-04-23T10:32:04.471 INFO:teuthology.orchestra.run.smithi114.stdout: "gather_set": [], 2024-04-23T10:32:04.471 INFO:teuthology.orchestra.run.smithi114.stdout: "state": "lock", 2024-04-23T10:32:04.471 INFO:teuthology.orchestra.run.smithi114.stdout: "type": "iquiesce", 2024-04-23T10:32:04.471 INFO:teuthology.orchestra.run.smithi114.stdout: "is_leased": false, 2024-04-23T10:32:04.471 INFO:teuthology.orchestra.run.smithi114.stdout: "num_rdlocks": 0, 2024-04-23T10:32:04.471 INFO:teuthology.orchestra.run.smithi114.stdout: "num_wrlocks": 0, 2024-04-23T10:32:04.471 INFO:teuthology.orchestra.run.smithi114.stdout: "num_xlocks": 1, 2024-04-23T10:32:04.471 INFO:teuthology.orchestra.run.smithi114.stdout: "xlock_by": { 2024-04-23T10:32:04.471 INFO:teuthology.orchestra.run.smithi114.stdout: "reqid": { 2024-04-23T10:32:04.471 INFO:teuthology.orchestra.run.smithi114.stdout: "entity": { 2024-04-23T10:32:04.471 INFO:teuthology.orchestra.run.smithi114.stdout: "type": "mds", 2024-04-23T10:32:04.471 INFO:teuthology.orchestra.run.smithi114.stdout: "num": 0 2024-04-23T10:32:04.471 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-23T10:32:04.472 INFO:teuthology.orchestra.run.smithi114.stdout: "tid": 186 2024-04-23T10:32:04.472 INFO:teuthology.orchestra.run.smithi114.stdout: } 2024-04-23T10:32:04.472 INFO:teuthology.orchestra.run.smithi114.stdout: } 2024-04-23T10:32:04.472 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-23T10:32:04.472 INFO:teuthology.orchestra.run.smithi114.stdout: "flags": 4, 2024-04-23T10:32:04.472 INFO:teuthology.orchestra.run.smithi114.stdout: "wrlock_target": -1 2024-04-23T10:32:04.472 INFO:teuthology.orchestra.run.smithi114.stdout: } 2024-04-23T10:32:04.472 INFO:teuthology.orchestra.run.smithi114.stdout: ] 2024-04-23T10:32:04.472 INFO:teuthology.orchestra.run.smithi114.stdout: } 2024-04-23T10:32:04.472 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-23T10:32:04.472 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-04-23T10:32:04.472 INFO:teuthology.orchestra.run.smithi114.stdout: "description": "peer_request:mds.1:121", 2024-04-23T10:32:04.472 INFO:teuthology.orchestra.run.smithi114.stdout: "initiated_at": "2024-04-23T10:31:21.465203+0000", 2024-04-23T10:32:04.472 INFO:teuthology.orchestra.run.smithi114.stdout: "age": 42.419186437, 2024-04-23T10:32:04.472 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 42.419321269000001, 2024-04-23T10:32:04.473 INFO:teuthology.orchestra.run.smithi114.stdout: "continuous": false, 2024-04-23T10:32:04.473 INFO:teuthology.orchestra.run.smithi114.stdout: "type_data": { 2024-04-23T10:32:04.473 INFO:teuthology.orchestra.run.smithi114.stdout: "result": -2147483648, 2024-04-23T10:32:04.473 INFO:teuthology.orchestra.run.smithi114.stdout: "flag_point": "dispatched", 2024-04-23T10:32:04.473 INFO:teuthology.orchestra.run.smithi114.stdout: "reqid": { 2024-04-23T10:32:04.473 INFO:teuthology.orchestra.run.smithi114.stdout: "entity": { 2024-04-23T10:32:04.473 INFO:teuthology.orchestra.run.smithi114.stdout: "type": "mds", 2024-04-23T10:32:04.473 INFO:teuthology.orchestra.run.smithi114.stdout: "num": 1 2024-04-23T10:32:04.473 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-23T10:32:04.473 INFO:teuthology.orchestra.run.smithi114.stdout: "tid": 121 2024-04-23T10:32:04.473 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-23T10:32:04.473 INFO:teuthology.orchestra.run.smithi114.stdout: "op_type": "peer_request", 2024-04-23T10:32:04.473 INFO:teuthology.orchestra.run.smithi114.stdout: "leader_info": { 2024-04-23T10:32:04.473 INFO:teuthology.orchestra.run.smithi114.stdout: "leader": "1" 2024-04-23T10:32:04.473 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-23T10:32:04.473 INFO:teuthology.orchestra.run.smithi114.stdout: "events": [ 2024-04-23T10:32:04.473 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-04-23T10:32:04.474 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-04-23T10:31:21.465203+0000", 2024-04-23T10:32:04.474 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "initiated" 2024-04-23T10:32:04.474 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-23T10:32:04.474 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-04-23T10:32:04.474 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-04-23T10:31:21.465203+0000", 2024-04-23T10:32:04.474 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "header_read" 2024-04-23T10:32:04.474 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-23T10:32:04.474 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-04-23T10:32:04.474 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-04-23T10:31:21.465203+0000", 2024-04-23T10:32:04.474 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "throttled" 2024-04-23T10:32:04.474 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-23T10:32:04.474 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-04-23T10:32:04.474 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-04-23T10:31:21.465208+0000", 2024-04-23T10:32:04.474 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "all_read" 2024-04-23T10:32:04.474 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-23T10:32:04.475 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-04-23T10:32:04.475 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-04-23T10:31:21.498343+0000", 2024-04-23T10:32:04.475 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "dispatched" 2024-04-23T10:32:04.475 INFO:teuthology.orchestra.run.smithi114.stdout: } 2024-04-23T10:32:04.475 INFO:teuthology.orchestra.run.smithi114.stdout: ], 2024-04-23T10:32:04.475 INFO:teuthology.orchestra.run.smithi114.stdout: "locks": [] 2024-04-23T10:32:04.475 INFO:teuthology.orchestra.run.smithi114.stdout: } 2024-04-23T10:32:04.475 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-23T10:32:04.475 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-04-23T10:32:04.475 INFO:teuthology.orchestra.run.smithi114.stdout: "description": "internal op quiesce_inode:mds.0:159 fp=#0x10000000008", 2024-04-23T10:32:04.475 INFO:teuthology.orchestra.run.smithi114.stdout: "initiated_at": "2024-04-23T10:31:21.455845+0000", 2024-04-23T10:32:04.475 INFO:teuthology.orchestra.run.smithi114.stdout: "age": 42.428544623999997, 2024-04-23T10:32:04.475 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 42.428702510999997, 2024-04-23T10:32:04.475 INFO:teuthology.orchestra.run.smithi114.stdout: "continuous": true, 2024-04-23T10:32:04.475 INFO:teuthology.orchestra.run.smithi114.stdout: "type_data": { 2024-04-23T10:32:04.475 INFO:teuthology.orchestra.run.smithi114.stdout: "result": -2147483648, 2024-04-23T10:32:04.476 INFO:teuthology.orchestra.run.smithi114.stdout: "flag_point": "quiesce complete", 2024-04-23T10:32:04.476 INFO:teuthology.orchestra.run.smithi114.stdout: "reqid": { 2024-04-23T10:32:04.476 INFO:teuthology.orchestra.run.smithi114.stdout: "entity": { 2024-04-23T10:32:04.476 INFO:teuthology.orchestra.run.smithi114.stdout: "type": "mds", 2024-04-23T10:32:04.476 INFO:teuthology.orchestra.run.smithi114.stdout: "num": 0 2024-04-23T10:32:04.476 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-23T10:32:04.476 INFO:teuthology.orchestra.run.smithi114.stdout: "tid": 159 2024-04-23T10:32:04.476 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-23T10:32:04.476 INFO:teuthology.orchestra.run.smithi114.stdout: "op_type": "internal_op", 2024-04-23T10:32:04.476 INFO:teuthology.orchestra.run.smithi114.stdout: "internal_op": 5385, 2024-04-23T10:32:04.476 INFO:teuthology.orchestra.run.smithi114.stdout: "op_name": "quiesce_inode", 2024-04-23T10:32:04.476 INFO:teuthology.orchestra.run.smithi114.stdout: "events": [ 2024-04-23T10:32:04.476 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-04-23T10:32:04.476 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-04-23T10:31:21.455845+0000", 2024-04-23T10:32:04.476 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "initiated" 2024-04-23T10:32:04.476 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-23T10:32:04.477 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-04-23T10:32:04.477 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-04-23T10:31:21.455845+0000", 2024-04-23T10:32:04.477 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "header_read" 2024-04-23T10:32:04.477 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-23T10:32:04.477 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-04-23T10:32:04.477 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-04-23T10:31:21.455845+0000", 2024-04-23T10:32:04.477 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "throttled" 2024-04-23T10:32:04.477 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-23T10:32:04.477 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-04-23T10:32:04.477 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-04-23T10:31:21.455845+0000", 2024-04-23T10:32:04.477 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "all_read" 2024-04-23T10:32:04.477 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-23T10:32:04.477 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-04-23T10:32:04.477 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-04-23T10:31:21.455845+0000", 2024-04-23T10:32:04.477 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "dispatched" 2024-04-23T10:32:04.477 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-23T10:32:04.477 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-04-23T10:32:04.478 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-04-23T10:31:21.456206+0000", 2024-04-23T10:32:04.478 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "failed to xlock, waiting" 2024-04-23T10:32:04.478 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-23T10:32:04.478 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-04-23T10:32:04.478 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-04-23T10:31:21.457273+0000", 2024-04-23T10:32:04.478 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "acquired locks" 2024-04-23T10:32:04.478 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-23T10:32:04.478 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-04-23T10:32:04.478 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-04-23T10:31:21.509852+0000", 2024-04-23T10:32:04.478 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "quiesce complete" 2024-04-23T10:32:04.478 INFO:teuthology.orchestra.run.smithi114.stdout: } 2024-04-23T10:32:04.478 INFO:teuthology.orchestra.run.smithi114.stdout: ], 2024-04-23T10:32:04.478 INFO:teuthology.orchestra.run.smithi114.stdout: "locks": [ 2024-04-23T10:32:04.478 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-04-23T10:32:04.478 INFO:teuthology.orchestra.run.smithi114.stdout: "object": { 2024-04-23T10:32:04.479 INFO:teuthology.orchestra.run.smithi114.stdout: "is_auth": true, 2024-04-23T10:32:04.479 INFO:teuthology.orchestra.run.smithi114.stdout: "auth_state": { 2024-04-23T10:32:04.479 INFO:teuthology.orchestra.run.smithi114.stdout: "replicas": { 2024-04-23T10:32:04.479 INFO:teuthology.orchestra.run.smithi114.stdout: "1": 1, 2024-04-23T10:32:04.479 INFO:teuthology.orchestra.run.smithi114.stdout: "2": 1 2024-04-23T10:32:04.479 INFO:teuthology.orchestra.run.smithi114.stdout: } 2024-04-23T10:32:04.479 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-23T10:32:04.479 INFO:teuthology.orchestra.run.smithi114.stdout: "replica_state": { 2024-04-23T10:32:04.479 INFO:teuthology.orchestra.run.smithi114.stdout: "authority": [ 2024-04-23T10:32:04.479 INFO:teuthology.orchestra.run.smithi114.stdout: 0, 2024-04-23T10:32:04.479 INFO:teuthology.orchestra.run.smithi114.stdout: -2 2024-04-23T10:32:04.479 INFO:teuthology.orchestra.run.smithi114.stdout: ], 2024-04-23T10:32:04.479 INFO:teuthology.orchestra.run.smithi114.stdout: "replica_nonce": 0 2024-04-23T10:32:04.479 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-23T10:32:04.479 INFO:teuthology.orchestra.run.smithi114.stdout: "auth_pins": 3, 2024-04-23T10:32:04.479 INFO:teuthology.orchestra.run.smithi114.stdout: "is_frozen": false, 2024-04-23T10:32:04.480 INFO:teuthology.orchestra.run.smithi114.stdout: "is_freezing": false, 2024-04-23T10:32:04.480 INFO:teuthology.orchestra.run.smithi114.stdout: "pins": { 2024-04-23T10:32:04.480 INFO:teuthology.orchestra.run.smithi114.stdout: "dirtyscattered": 0, 2024-04-23T10:32:04.480 INFO:teuthology.orchestra.run.smithi114.stdout: "request": 1, 2024-04-23T10:32:04.480 INFO:teuthology.orchestra.run.smithi114.stdout: "lock": 1, 2024-04-23T10:32:04.480 INFO:teuthology.orchestra.run.smithi114.stdout: "dirfrag": 1, 2024-04-23T10:32:04.480 INFO:teuthology.orchestra.run.smithi114.stdout: "caps": 1, 2024-04-23T10:32:04.480 INFO:teuthology.orchestra.run.smithi114.stdout: "dirtyrstat": 0, 2024-04-23T10:32:04.480 INFO:teuthology.orchestra.run.smithi114.stdout: "dirtyparent": 0, 2024-04-23T10:32:04.480 INFO:teuthology.orchestra.run.smithi114.stdout: "replicated": 1, 2024-04-23T10:32:04.480 INFO:teuthology.orchestra.run.smithi114.stdout: "dirty": 1, 2024-04-23T10:32:04.480 INFO:teuthology.orchestra.run.smithi114.stdout: "waiter": 0, 2024-04-23T10:32:04.480 INFO:teuthology.orchestra.run.smithi114.stdout: "authpin": 1 2024-04-23T10:32:04.480 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-23T10:32:04.480 INFO:teuthology.orchestra.run.smithi114.stdout: "nref": 7 2024-04-23T10:32:04.480 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-23T10:32:04.481 INFO:teuthology.orchestra.run.smithi114.stdout: "object_string": "[inode 0x10000000008 [...7,head] /volumes/qa/sv_1/b206794e-7622-4ab8-8943-f3ae4e65b7a3/client.0/tmp/clients/client0/ auth{1=1,2=1} v456 ap=3 f(v0 m2024-04-23T10:27:51.950398+0000 6=5+1) n(v30 rc2024-04-23T10:31:17.764716+0000 b21075074 74=63+11) old_inodes=4 (inest mix) (iquiesce lock x=1 by request(mds.0:159 nref=3)) caps={15312=pAsLsXsFs/pAsLsXsFs@29},l=15312 | dirtyscattered=0 request=1 lock=1 dirfrag=1 caps=1 dirtyrstat=0 dirtyparent=0 replicated=1 dirty=1 waiter=0 authpin=1 0x5571cf4d4580]", 2024-04-23T10:32:04.481 INFO:teuthology.orchestra.run.smithi114.stdout: "lock": { 2024-04-23T10:32:04.481 INFO:teuthology.orchestra.run.smithi114.stdout: "gather_set": [], 2024-04-23T10:32:04.481 INFO:teuthology.orchestra.run.smithi114.stdout: "state": "lock", 2024-04-23T10:32:04.481 INFO:teuthology.orchestra.run.smithi114.stdout: "type": "iquiesce", 2024-04-23T10:32:04.481 INFO:teuthology.orchestra.run.smithi114.stdout: "is_leased": false, 2024-04-23T10:32:04.481 INFO:teuthology.orchestra.run.smithi114.stdout: "num_rdlocks": 0, 2024-04-23T10:32:04.481 INFO:teuthology.orchestra.run.smithi114.stdout: "num_wrlocks": 0, 2024-04-23T10:32:04.481 INFO:teuthology.orchestra.run.smithi114.stdout: "num_xlocks": 1, 2024-04-23T10:32:04.481 INFO:teuthology.orchestra.run.smithi114.stdout: "xlock_by": { 2024-04-23T10:32:04.481 INFO:teuthology.orchestra.run.smithi114.stdout: "reqid": { 2024-04-23T10:32:04.481 INFO:teuthology.orchestra.run.smithi114.stdout: "entity": { 2024-04-23T10:32:04.481 INFO:teuthology.orchestra.run.smithi114.stdout: "type": "mds", 2024-04-23T10:32:04.481 INFO:teuthology.orchestra.run.smithi114.stdout: "num": 0 2024-04-23T10:32:04.481 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-23T10:32:04.481 INFO:teuthology.orchestra.run.smithi114.stdout: "tid": 159 2024-04-23T10:32:04.481 INFO:teuthology.orchestra.run.smithi114.stdout: } 2024-04-23T10:32:04.482 INFO:teuthology.orchestra.run.smithi114.stdout: } 2024-04-23T10:32:04.482 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-23T10:32:04.482 INFO:teuthology.orchestra.run.smithi114.stdout: "flags": 4, 2024-04-23T10:32:04.482 INFO:teuthology.orchestra.run.smithi114.stdout: "wrlock_target": -1 2024-04-23T10:32:04.482 INFO:teuthology.orchestra.run.smithi114.stdout: } 2024-04-23T10:32:04.482 INFO:teuthology.orchestra.run.smithi114.stdout: ] 2024-04-23T10:32:04.482 INFO:teuthology.orchestra.run.smithi114.stdout: } 2024-04-23T10:32:04.482 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-23T10:32:04.482 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-04-23T10:32:04.482 INFO:teuthology.orchestra.run.smithi114.stdout: "description": "internal op quiesce_inode:mds.0:187 fp=#0x1000000023f", 2024-04-23T10:32:04.482 INFO:teuthology.orchestra.run.smithi114.stdout: "initiated_at": "2024-04-23T10:31:21.471282+0000", 2024-04-23T10:32:04.482 INFO:teuthology.orchestra.run.smithi114.stdout: "age": 42.413107613000001, 2024-04-23T10:32:04.482 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 42.413338134, 2024-04-23T10:32:04.482 INFO:teuthology.orchestra.run.smithi114.stdout: "continuous": true, 2024-04-23T10:32:04.482 INFO:teuthology.orchestra.run.smithi114.stdout: "type_data": { 2024-04-23T10:32:04.483 INFO:teuthology.orchestra.run.smithi114.stdout: "result": -2147483648, 2024-04-23T10:32:04.483 INFO:teuthology.orchestra.run.smithi114.stdout: "flag_point": "quiesce complete", 2024-04-23T10:32:04.483 INFO:teuthology.orchestra.run.smithi114.stdout: "reqid": { 2024-04-23T10:32:04.483 INFO:teuthology.orchestra.run.smithi114.stdout: "entity": { 2024-04-23T10:32:04.483 INFO:teuthology.orchestra.run.smithi114.stdout: "type": "mds", 2024-04-23T10:32:04.483 INFO:teuthology.orchestra.run.smithi114.stdout: "num": 0 2024-04-23T10:32:04.483 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-23T10:32:04.483 INFO:teuthology.orchestra.run.smithi114.stdout: "tid": 187 2024-04-23T10:32:04.483 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-23T10:32:04.483 INFO:teuthology.orchestra.run.smithi114.stdout: "op_type": "internal_op", 2024-04-23T10:32:04.483 INFO:teuthology.orchestra.run.smithi114.stdout: "internal_op": 5385, 2024-04-23T10:32:04.483 INFO:teuthology.orchestra.run.smithi114.stdout: "op_name": "quiesce_inode", 2024-04-23T10:32:04.483 INFO:teuthology.orchestra.run.smithi114.stdout: "events": [ 2024-04-23T10:32:04.483 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-04-23T10:32:04.483 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-04-23T10:31:21.471282+0000", 2024-04-23T10:32:04.483 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "initiated" 2024-04-23T10:32:04.484 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-23T10:32:04.484 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-04-23T10:32:04.484 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-04-23T10:31:21.471282+0000", 2024-04-23T10:32:04.484 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "header_read" 2024-04-23T10:32:04.484 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-23T10:32:04.484 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-04-23T10:32:04.484 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-04-23T10:31:21.471282+0000", 2024-04-23T10:32:04.484 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "throttled" 2024-04-23T10:32:04.484 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-23T10:32:04.484 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-04-23T10:32:04.484 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-04-23T10:31:21.471282+0000", 2024-04-23T10:32:04.484 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "all_read" 2024-04-23T10:32:04.484 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-23T10:32:04.484 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-04-23T10:32:04.484 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-04-23T10:31:21.471282+0000", 2024-04-23T10:32:04.484 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "dispatched" 2024-04-23T10:32:04.485 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-23T10:32:04.485 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-04-23T10:32:04.485 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-04-23T10:31:21.471808+0000", 2024-04-23T10:32:04.485 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "failed to xlock, waiting" 2024-04-23T10:32:04.485 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-23T10:32:04.485 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-04-23T10:32:04.485 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-04-23T10:31:21.500182+0000", 2024-04-23T10:32:04.485 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "failed to xlock, waiting" 2024-04-23T10:32:04.485 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-23T10:32:04.485 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-04-23T10:32:04.485 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-04-23T10:31:21.503967+0000", 2024-04-23T10:32:04.485 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "acquired locks" 2024-04-23T10:32:04.485 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-23T10:32:04.485 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-04-23T10:32:04.485 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-04-23T10:31:21.504156+0000", 2024-04-23T10:32:04.485 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "quiesce complete" 2024-04-23T10:32:04.486 INFO:teuthology.orchestra.run.smithi114.stdout: } 2024-04-23T10:32:04.486 INFO:teuthology.orchestra.run.smithi114.stdout: ], 2024-04-23T10:32:04.486 INFO:teuthology.orchestra.run.smithi114.stdout: "locks": [ 2024-04-23T10:32:04.486 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-04-23T10:32:04.486 INFO:teuthology.orchestra.run.smithi114.stdout: "object": { 2024-04-23T10:32:04.486 INFO:teuthology.orchestra.run.smithi114.stdout: "is_auth": true, 2024-04-23T10:32:04.486 INFO:teuthology.orchestra.run.smithi114.stdout: "auth_state": { 2024-04-23T10:32:04.486 INFO:teuthology.orchestra.run.smithi114.stdout: "replicas": {} 2024-04-23T10:32:04.486 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-23T10:32:04.486 INFO:teuthology.orchestra.run.smithi114.stdout: "replica_state": { 2024-04-23T10:32:04.486 INFO:teuthology.orchestra.run.smithi114.stdout: "authority": [ 2024-04-23T10:32:04.486 INFO:teuthology.orchestra.run.smithi114.stdout: 0, 2024-04-23T10:32:04.486 INFO:teuthology.orchestra.run.smithi114.stdout: -2 2024-04-23T10:32:04.486 INFO:teuthology.orchestra.run.smithi114.stdout: ], 2024-04-23T10:32:04.486 INFO:teuthology.orchestra.run.smithi114.stdout: "replica_nonce": 0 2024-04-23T10:32:04.487 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-23T10:32:04.487 INFO:teuthology.orchestra.run.smithi114.stdout: "auth_pins": 1, 2024-04-23T10:32:04.487 INFO:teuthology.orchestra.run.smithi114.stdout: "is_frozen": false, 2024-04-23T10:32:04.487 INFO:teuthology.orchestra.run.smithi114.stdout: "is_freezing": false, 2024-04-23T10:32:04.487 INFO:teuthology.orchestra.run.smithi114.stdout: "pins": { 2024-04-23T10:32:04.487 INFO:teuthology.orchestra.run.smithi114.stdout: "ptrwaiter": 0, 2024-04-23T10:32:04.487 INFO:teuthology.orchestra.run.smithi114.stdout: "request": 1, 2024-04-23T10:32:04.487 INFO:teuthology.orchestra.run.smithi114.stdout: "lock": 1, 2024-04-23T10:32:04.487 INFO:teuthology.orchestra.run.smithi114.stdout: "caps": 0, 2024-04-23T10:32:04.487 INFO:teuthology.orchestra.run.smithi114.stdout: "needsnapflush": 0, 2024-04-23T10:32:04.487 INFO:teuthology.orchestra.run.smithi114.stdout: "dirtyparent": 0, 2024-04-23T10:32:04.487 INFO:teuthology.orchestra.run.smithi114.stdout: "dirty": 0, 2024-04-23T10:32:04.487 INFO:teuthology.orchestra.run.smithi114.stdout: "waiter": 0, 2024-04-23T10:32:04.487 INFO:teuthology.orchestra.run.smithi114.stdout: "authpin": 1 2024-04-23T10:32:04.487 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-23T10:32:04.487 INFO:teuthology.orchestra.run.smithi114.stdout: "nref": 3 2024-04-23T10:32:04.488 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-23T10:32:04.488 INFO:teuthology.orchestra.run.smithi114.stdout: "object_string": "[inode 0x1000000023f [7,head] /volumes/qa/sv_1/b206794e-7622-4ab8-8943-f3ae4e65b7a3/client.0/tmp/clients/client0/~dmtmp/ACCESS/FASTENER.MDB auth v3324 ap=1 s=260096 n(v0 rc2024-04-23T10:31:17.038732+0000 b260096 1=1+0) (iquiesce lock x=1 by request(mds.0:187 nref=3)) | ptrwaiter=0 request=1 lock=1 caps=0 needsnapflush=0 dirtyparent=0 dirty=0 waiter=0 authpin=1 0x5571cf56ac00]", 2024-04-23T10:32:04.488 INFO:teuthology.orchestra.run.smithi114.stdout: "lock": { 2024-04-23T10:32:04.488 INFO:teuthology.orchestra.run.smithi114.stdout: "gather_set": [], 2024-04-23T10:32:04.488 INFO:teuthology.orchestra.run.smithi114.stdout: "state": "lock", 2024-04-23T10:32:04.488 INFO:teuthology.orchestra.run.smithi114.stdout: "type": "iquiesce", 2024-04-23T10:32:04.488 INFO:teuthology.orchestra.run.smithi114.stdout: "is_leased": false, 2024-04-23T10:32:04.488 INFO:teuthology.orchestra.run.smithi114.stdout: "num_rdlocks": 0, 2024-04-23T10:32:04.488 INFO:teuthology.orchestra.run.smithi114.stdout: "num_wrlocks": 0, 2024-04-23T10:32:04.488 INFO:teuthology.orchestra.run.smithi114.stdout: "num_xlocks": 1, 2024-04-23T10:32:04.488 INFO:teuthology.orchestra.run.smithi114.stdout: "xlock_by": { 2024-04-23T10:32:04.488 INFO:teuthology.orchestra.run.smithi114.stdout: "reqid": { 2024-04-23T10:32:04.488 INFO:teuthology.orchestra.run.smithi114.stdout: "entity": { 2024-04-23T10:32:04.488 INFO:teuthology.orchestra.run.smithi114.stdout: "type": "mds", 2024-04-23T10:32:04.488 INFO:teuthology.orchestra.run.smithi114.stdout: "num": 0 2024-04-23T10:32:04.488 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-23T10:32:04.489 INFO:teuthology.orchestra.run.smithi114.stdout: "tid": 187 2024-04-23T10:32:04.489 INFO:teuthology.orchestra.run.smithi114.stdout: } 2024-04-23T10:32:04.489 INFO:teuthology.orchestra.run.smithi114.stdout: } 2024-04-23T10:32:04.489 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-23T10:32:04.489 INFO:teuthology.orchestra.run.smithi114.stdout: "flags": 4, 2024-04-23T10:32:04.489 INFO:teuthology.orchestra.run.smithi114.stdout: "wrlock_target": -1 2024-04-23T10:32:04.489 INFO:teuthology.orchestra.run.smithi114.stdout: } 2024-04-23T10:32:04.489 INFO:teuthology.orchestra.run.smithi114.stdout: ] 2024-04-23T10:32:04.489 INFO:teuthology.orchestra.run.smithi114.stdout: } 2024-04-23T10:32:04.489 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-23T10:32:04.489 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-04-23T10:32:04.489 INFO:teuthology.orchestra.run.smithi114.stdout: "description": "peer_request:mds.2:137", 2024-04-23T10:32:04.489 INFO:teuthology.orchestra.run.smithi114.stdout: "initiated_at": "2024-04-23T10:31:21.489495+0000", 2024-04-23T10:32:04.489 INFO:teuthology.orchestra.run.smithi114.stdout: "age": 42.394893967999998, 2024-04-23T10:32:04.489 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 42.395189960000003, 2024-04-23T10:32:04.490 INFO:teuthology.orchestra.run.smithi114.stdout: "continuous": false, 2024-04-23T10:32:04.490 INFO:teuthology.orchestra.run.smithi114.stdout: "type_data": { 2024-04-23T10:32:04.490 INFO:teuthology.orchestra.run.smithi114.stdout: "result": -2147483648, 2024-04-23T10:32:04.490 INFO:teuthology.orchestra.run.smithi114.stdout: "flag_point": "dispatched", 2024-04-23T10:32:04.490 INFO:teuthology.orchestra.run.smithi114.stdout: "reqid": { 2024-04-23T10:32:04.490 INFO:teuthology.orchestra.run.smithi114.stdout: "entity": { 2024-04-23T10:32:04.490 INFO:teuthology.orchestra.run.smithi114.stdout: "type": "mds", 2024-04-23T10:32:04.490 INFO:teuthology.orchestra.run.smithi114.stdout: "num": 2 2024-04-23T10:32:04.490 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-23T10:32:04.490 INFO:teuthology.orchestra.run.smithi114.stdout: "tid": 137 2024-04-23T10:32:04.490 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-23T10:32:04.490 INFO:teuthology.orchestra.run.smithi114.stdout: "op_type": "peer_request", 2024-04-23T10:32:04.490 INFO:teuthology.orchestra.run.smithi114.stdout: "leader_info": { 2024-04-23T10:32:04.490 INFO:teuthology.orchestra.run.smithi114.stdout: "leader": "2" 2024-04-23T10:32:04.491 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-23T10:32:04.491 INFO:teuthology.orchestra.run.smithi114.stdout: "events": [ 2024-04-23T10:32:04.491 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-04-23T10:32:04.491 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-04-23T10:31:21.489495+0000", 2024-04-23T10:32:04.491 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "initiated" 2024-04-23T10:32:04.491 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-23T10:32:04.491 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-04-23T10:32:04.491 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-04-23T10:31:21.489495+0000", 2024-04-23T10:32:04.491 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "header_read" 2024-04-23T10:32:04.491 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-23T10:32:04.491 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-04-23T10:32:04.491 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-04-23T10:31:21.489496+0000", 2024-04-23T10:32:04.491 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "throttled" 2024-04-23T10:32:04.491 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-23T10:32:04.492 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-04-23T10:32:04.492 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-04-23T10:31:21.489503+0000", 2024-04-23T10:32:04.492 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "all_read" 2024-04-23T10:32:04.492 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-23T10:32:04.492 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-04-23T10:32:04.492 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-04-23T10:31:21.508236+0000", 2024-04-23T10:32:04.492 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "dispatched" 2024-04-23T10:32:04.492 INFO:teuthology.orchestra.run.smithi114.stdout: } 2024-04-23T10:32:04.492 INFO:teuthology.orchestra.run.smithi114.stdout: ], 2024-04-23T10:32:04.492 INFO:teuthology.orchestra.run.smithi114.stdout: "locks": [] 2024-04-23T10:32:04.492 INFO:teuthology.orchestra.run.smithi114.stdout: } 2024-04-23T10:32:04.492 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-23T10:32:04.492 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-04-23T10:32:04.492 INFO:teuthology.orchestra.run.smithi114.stdout: "description": "peer_request:mds.2:115", 2024-04-23T10:32:04.493 INFO:teuthology.orchestra.run.smithi114.stdout: "initiated_at": "2024-04-23T10:31:21.455271+0000", 2024-04-23T10:32:04.493 INFO:teuthology.orchestra.run.smithi114.stdout: "age": 42.429118481000003, 2024-04-23T10:32:04.493 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 42.429435904000002, 2024-04-23T10:32:04.493 INFO:teuthology.orchestra.run.smithi114.stdout: "continuous": false, 2024-04-23T10:32:04.493 INFO:teuthology.orchestra.run.smithi114.stdout: "type_data": { 2024-04-23T10:32:04.493 INFO:teuthology.orchestra.run.smithi114.stdout: "result": -2147483648, 2024-04-23T10:32:04.493 INFO:teuthology.orchestra.run.smithi114.stdout: "flag_point": "dispatched", 2024-04-23T10:32:04.493 INFO:teuthology.orchestra.run.smithi114.stdout: "reqid": { 2024-04-23T10:32:04.493 INFO:teuthology.orchestra.run.smithi114.stdout: "entity": { 2024-04-23T10:32:04.493 INFO:teuthology.orchestra.run.smithi114.stdout: "type": "mds", 2024-04-23T10:32:04.493 INFO:teuthology.orchestra.run.smithi114.stdout: "num": 2 2024-04-23T10:32:04.493 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-23T10:32:04.493 INFO:teuthology.orchestra.run.smithi114.stdout: "tid": 115 2024-04-23T10:32:04.493 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-23T10:32:04.493 INFO:teuthology.orchestra.run.smithi114.stdout: "op_type": "peer_request", 2024-04-23T10:32:04.494 INFO:teuthology.orchestra.run.smithi114.stdout: "leader_info": { 2024-04-23T10:32:04.494 INFO:teuthology.orchestra.run.smithi114.stdout: "leader": "2" 2024-04-23T10:32:04.494 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-23T10:32:04.494 INFO:teuthology.orchestra.run.smithi114.stdout: "events": [ 2024-04-23T10:32:04.494 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-04-23T10:32:04.494 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-04-23T10:31:21.455271+0000", 2024-04-23T10:32:04.494 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "initiated" 2024-04-23T10:32:04.494 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-23T10:32:04.494 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-04-23T10:32:04.494 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-04-23T10:31:21.455271+0000", 2024-04-23T10:32:04.494 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "header_read" 2024-04-23T10:32:04.494 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-23T10:32:04.494 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-04-23T10:32:04.494 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-04-23T10:31:21.455271+0000", 2024-04-23T10:32:04.495 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "throttled" 2024-04-23T10:32:04.495 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-23T10:32:04.495 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-04-23T10:32:04.495 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-04-23T10:31:21.455278+0000", 2024-04-23T10:32:04.495 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "all_read" 2024-04-23T10:32:04.495 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-23T10:32:04.495 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-04-23T10:32:04.495 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-04-23T10:31:21.456696+0000", 2024-04-23T10:32:04.495 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "dispatched" 2024-04-23T10:32:04.495 INFO:teuthology.orchestra.run.smithi114.stdout: } 2024-04-23T10:32:04.495 INFO:teuthology.orchestra.run.smithi114.stdout: ], 2024-04-23T10:32:04.495 INFO:teuthology.orchestra.run.smithi114.stdout: "locks": [] 2024-04-23T10:32:04.495 INFO:teuthology.orchestra.run.smithi114.stdout: } 2024-04-23T10:32:04.495 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-23T10:32:04.495 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-04-23T10:32:04.496 INFO:teuthology.orchestra.run.smithi114.stdout: "description": "internal op quiesce_inode:mds.0:188 fp=#0x100000014dd", 2024-04-23T10:32:04.496 INFO:teuthology.orchestra.run.smithi114.stdout: "initiated_at": "2024-04-23T10:31:21.471294+0000", 2024-04-23T10:32:04.496 INFO:teuthology.orchestra.run.smithi114.stdout: "age": 42.413095052999999, 2024-04-23T10:32:04.496 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 42.413434293000002, 2024-04-23T10:32:04.496 INFO:teuthology.orchestra.run.smithi114.stdout: "continuous": true, 2024-04-23T10:32:04.496 INFO:teuthology.orchestra.run.smithi114.stdout: "type_data": { 2024-04-23T10:32:04.496 INFO:teuthology.orchestra.run.smithi114.stdout: "result": -2147483648, 2024-04-23T10:32:04.496 INFO:teuthology.orchestra.run.smithi114.stdout: "flag_point": "quiesce complete", 2024-04-23T10:32:04.496 INFO:teuthology.orchestra.run.smithi114.stdout: "reqid": { 2024-04-23T10:32:04.496 INFO:teuthology.orchestra.run.smithi114.stdout: "entity": { 2024-04-23T10:32:04.496 INFO:teuthology.orchestra.run.smithi114.stdout: "type": "mds", 2024-04-23T10:32:04.496 INFO:teuthology.orchestra.run.smithi114.stdout: "num": 0 2024-04-23T10:32:04.496 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-23T10:32:04.496 INFO:teuthology.orchestra.run.smithi114.stdout: "tid": 188 2024-04-23T10:32:04.496 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-23T10:32:04.497 INFO:teuthology.orchestra.run.smithi114.stdout: "op_type": "internal_op", 2024-04-23T10:32:04.497 INFO:teuthology.orchestra.run.smithi114.stdout: "internal_op": 5385, 2024-04-23T10:32:04.497 INFO:teuthology.orchestra.run.smithi114.stdout: "op_name": "quiesce_inode", 2024-04-23T10:32:04.497 INFO:teuthology.orchestra.run.smithi114.stdout: "events": [ 2024-04-23T10:32:04.497 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-04-23T10:32:04.497 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-04-23T10:31:21.471294+0000", 2024-04-23T10:32:04.497 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "initiated" 2024-04-23T10:32:04.497 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-23T10:32:04.497 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-04-23T10:32:04.497 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-04-23T10:31:21.471294+0000", 2024-04-23T10:32:04.497 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "header_read" 2024-04-23T10:32:04.497 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-23T10:32:04.497 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-04-23T10:32:04.497 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-04-23T10:31:21.471294+0000", 2024-04-23T10:32:04.497 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "throttled" 2024-04-23T10:32:04.498 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-23T10:32:04.498 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-04-23T10:32:04.498 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-04-23T10:31:21.471294+0000", 2024-04-23T10:32:04.498 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "all_read" 2024-04-23T10:32:04.498 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-23T10:32:04.498 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-04-23T10:32:04.498 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-04-23T10:31:21.471294+0000", 2024-04-23T10:32:04.498 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "dispatched" 2024-04-23T10:32:04.498 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-23T10:32:04.498 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-04-23T10:32:04.498 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-04-23T10:31:21.472009+0000", 2024-04-23T10:32:04.498 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "failed to xlock, waiting" 2024-04-23T10:32:04.498 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-23T10:32:04.498 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-04-23T10:32:04.499 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-04-23T10:31:21.502362+0000", 2024-04-23T10:32:04.499 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "failed to xlock, waiting" 2024-04-23T10:32:04.499 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-23T10:32:04.499 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-04-23T10:32:04.499 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-04-23T10:31:21.505667+0000", 2024-04-23T10:32:04.499 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "acquired locks" 2024-04-23T10:32:04.499 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-23T10:32:04.499 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-04-23T10:32:04.499 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-04-23T10:31:21.505865+0000", 2024-04-23T10:32:04.499 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "quiesce complete" 2024-04-23T10:32:04.499 INFO:teuthology.orchestra.run.smithi114.stdout: } 2024-04-23T10:32:04.499 INFO:teuthology.orchestra.run.smithi114.stdout: ], 2024-04-23T10:32:04.499 INFO:teuthology.orchestra.run.smithi114.stdout: "locks": [ 2024-04-23T10:32:04.499 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-04-23T10:32:04.500 INFO:teuthology.orchestra.run.smithi114.stdout: "object": { 2024-04-23T10:32:04.500 INFO:teuthology.orchestra.run.smithi114.stdout: "is_auth": true, 2024-04-23T10:32:04.500 INFO:teuthology.orchestra.run.smithi114.stdout: "auth_state": { 2024-04-23T10:32:04.500 INFO:teuthology.orchestra.run.smithi114.stdout: "replicas": {} 2024-04-23T10:32:04.500 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-23T10:32:04.500 INFO:teuthology.orchestra.run.smithi114.stdout: "replica_state": { 2024-04-23T10:32:04.500 INFO:teuthology.orchestra.run.smithi114.stdout: "authority": [ 2024-04-23T10:32:04.500 INFO:teuthology.orchestra.run.smithi114.stdout: 0, 2024-04-23T10:32:04.500 INFO:teuthology.orchestra.run.smithi114.stdout: -2 2024-04-23T10:32:04.500 INFO:teuthology.orchestra.run.smithi114.stdout: ], 2024-04-23T10:32:04.500 INFO:teuthology.orchestra.run.smithi114.stdout: "replica_nonce": 0 2024-04-23T10:32:04.500 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-23T10:32:04.500 INFO:teuthology.orchestra.run.smithi114.stdout: "auth_pins": 1, 2024-04-23T10:32:04.500 INFO:teuthology.orchestra.run.smithi114.stdout: "is_frozen": false, 2024-04-23T10:32:04.501 INFO:teuthology.orchestra.run.smithi114.stdout: "is_freezing": false, 2024-04-23T10:32:04.501 INFO:teuthology.orchestra.run.smithi114.stdout: "pins": { 2024-04-23T10:32:04.501 INFO:teuthology.orchestra.run.smithi114.stdout: "ptrwaiter": 0, 2024-04-23T10:32:04.501 INFO:teuthology.orchestra.run.smithi114.stdout: "request": 1, 2024-04-23T10:32:04.501 INFO:teuthology.orchestra.run.smithi114.stdout: "lock": 1, 2024-04-23T10:32:04.501 INFO:teuthology.orchestra.run.smithi114.stdout: "caps": 0, 2024-04-23T10:32:04.501 INFO:teuthology.orchestra.run.smithi114.stdout: "dirtyparent": 0, 2024-04-23T10:32:04.501 INFO:teuthology.orchestra.run.smithi114.stdout: "dirty": 0, 2024-04-23T10:32:04.501 INFO:teuthology.orchestra.run.smithi114.stdout: "waiter": 0, 2024-04-23T10:32:04.501 INFO:teuthology.orchestra.run.smithi114.stdout: "authpin": 1 2024-04-23T10:32:04.501 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-23T10:32:04.501 INFO:teuthology.orchestra.run.smithi114.stdout: "nref": 3 2024-04-23T10:32:04.501 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-23T10:32:04.502 INFO:teuthology.orchestra.run.smithi114.stdout: "object_string": "[inode 0x100000014dd [7,head] /volumes/qa/sv_1/b206794e-7622-4ab8-8943-f3ae4e65b7a3/client.0/tmp/clients/client0/~dmtmp/ACCESS/SALES.PRN auth v3330 ap=1 s=10845 n(v0 rc2024-04-23T10:31:17.026733+0000 b10845 1=1+0) (iquiesce lock x=1 by request(mds.0:188 nref=3)) | ptrwaiter=0 request=1 lock=1 caps=0 dirtyparent=0 dirty=0 waiter=0 authpin=1 0x5571cf5cf180]", 2024-04-23T10:32:04.502 INFO:teuthology.orchestra.run.smithi114.stdout: "lock": { 2024-04-23T10:32:04.502 INFO:teuthology.orchestra.run.smithi114.stdout: "gather_set": [], 2024-04-23T10:32:04.502 INFO:teuthology.orchestra.run.smithi114.stdout: "state": "lock", 2024-04-23T10:32:04.502 INFO:teuthology.orchestra.run.smithi114.stdout: "type": "iquiesce", 2024-04-23T10:32:04.502 INFO:teuthology.orchestra.run.smithi114.stdout: "is_leased": false, 2024-04-23T10:32:04.502 INFO:teuthology.orchestra.run.smithi114.stdout: "num_rdlocks": 0, 2024-04-23T10:32:04.502 INFO:teuthology.orchestra.run.smithi114.stdout: "num_wrlocks": 0, 2024-04-23T10:32:04.502 INFO:teuthology.orchestra.run.smithi114.stdout: "num_xlocks": 1, 2024-04-23T10:32:04.502 INFO:teuthology.orchestra.run.smithi114.stdout: "xlock_by": { 2024-04-23T10:32:04.502 INFO:teuthology.orchestra.run.smithi114.stdout: "reqid": { 2024-04-23T10:32:04.502 INFO:teuthology.orchestra.run.smithi114.stdout: "entity": { 2024-04-23T10:32:04.503 INFO:teuthology.orchestra.run.smithi114.stdout: "type": "mds", 2024-04-23T10:32:04.503 INFO:teuthology.orchestra.run.smithi114.stdout: "num": 0 2024-04-23T10:32:04.503 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-23T10:32:04.503 INFO:teuthology.orchestra.run.smithi114.stdout: "tid": 188 2024-04-23T10:32:04.503 INFO:teuthology.orchestra.run.smithi114.stdout: } 2024-04-23T10:32:04.503 INFO:teuthology.orchestra.run.smithi114.stdout: } 2024-04-23T10:32:04.503 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-23T10:32:04.503 INFO:teuthology.orchestra.run.smithi114.stdout: "flags": 4, 2024-04-23T10:32:04.503 INFO:teuthology.orchestra.run.smithi114.stdout: "wrlock_target": -1 2024-04-23T10:32:04.503 INFO:teuthology.orchestra.run.smithi114.stdout: } 2024-04-23T10:32:04.503 INFO:teuthology.orchestra.run.smithi114.stdout: ] 2024-04-23T10:32:04.503 INFO:teuthology.orchestra.run.smithi114.stdout: } 2024-04-23T10:32:04.503 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-23T10:32:04.503 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-04-23T10:32:04.504 INFO:teuthology.orchestra.run.smithi114.stdout: "description": "peer_request:mds.2:138", 2024-04-23T10:32:04.504 INFO:teuthology.orchestra.run.smithi114.stdout: "initiated_at": "2024-04-23T10:31:21.489509+0000", 2024-04-23T10:32:04.504 INFO:teuthology.orchestra.run.smithi114.stdout: "age": 42.394880688000001, 2024-04-23T10:32:04.504 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 42.395288555999997, 2024-04-23T10:32:04.504 INFO:teuthology.orchestra.run.smithi114.stdout: "continuous": false, 2024-04-23T10:32:04.504 INFO:teuthology.orchestra.run.smithi114.stdout: "type_data": { 2024-04-23T10:32:04.504 INFO:teuthology.orchestra.run.smithi114.stdout: "result": -2147483648, 2024-04-23T10:32:04.504 INFO:teuthology.orchestra.run.smithi114.stdout: "flag_point": "dispatched", 2024-04-23T10:32:04.504 INFO:teuthology.orchestra.run.smithi114.stdout: "reqid": { 2024-04-23T10:32:04.504 INFO:teuthology.orchestra.run.smithi114.stdout: "entity": { 2024-04-23T10:32:04.504 INFO:teuthology.orchestra.run.smithi114.stdout: "type": "mds", 2024-04-23T10:32:04.504 INFO:teuthology.orchestra.run.smithi114.stdout: "num": 2 2024-04-23T10:32:04.504 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-23T10:32:04.504 INFO:teuthology.orchestra.run.smithi114.stdout: "tid": 138 2024-04-23T10:32:04.504 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-23T10:32:04.505 INFO:teuthology.orchestra.run.smithi114.stdout: "op_type": "peer_request", 2024-04-23T10:32:04.505 INFO:teuthology.orchestra.run.smithi114.stdout: "leader_info": { 2024-04-23T10:32:04.505 INFO:teuthology.orchestra.run.smithi114.stdout: "leader": "2" 2024-04-23T10:32:04.505 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-23T10:32:04.505 INFO:teuthology.orchestra.run.smithi114.stdout: "events": [ 2024-04-23T10:32:04.505 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-04-23T10:32:04.505 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-04-23T10:31:21.489509+0000", 2024-04-23T10:32:04.505 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "initiated" 2024-04-23T10:32:04.505 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-23T10:32:04.505 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-04-23T10:32:04.505 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-04-23T10:31:21.489509+0000", 2024-04-23T10:32:04.505 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "header_read" 2024-04-23T10:32:04.505 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-23T10:32:04.505 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-04-23T10:32:04.506 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-04-23T10:31:21.489509+0000", 2024-04-23T10:32:04.506 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "throttled" 2024-04-23T10:32:04.506 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-23T10:32:04.506 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-04-23T10:32:04.506 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-04-23T10:31:21.489511+0000", 2024-04-23T10:32:04.506 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "all_read" 2024-04-23T10:32:04.506 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-23T10:32:04.506 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-04-23T10:32:04.506 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-04-23T10:31:21.508324+0000", 2024-04-23T10:32:04.506 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "dispatched" 2024-04-23T10:32:04.506 INFO:teuthology.orchestra.run.smithi114.stdout: } 2024-04-23T10:32:04.506 INFO:teuthology.orchestra.run.smithi114.stdout: ], 2024-04-23T10:32:04.506 INFO:teuthology.orchestra.run.smithi114.stdout: "locks": [] 2024-04-23T10:32:04.506 INFO:teuthology.orchestra.run.smithi114.stdout: } 2024-04-23T10:32:04.506 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-23T10:32:04.507 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-04-23T10:32:04.507 INFO:teuthology.orchestra.run.smithi114.stdout: "description": "peer_request:mds.1:111", 2024-04-23T10:32:04.507 INFO:teuthology.orchestra.run.smithi114.stdout: "initiated_at": "2024-04-23T10:31:21.455336+0000", 2024-04-23T10:32:04.507 INFO:teuthology.orchestra.run.smithi114.stdout: "age": 42.429053641000003, 2024-04-23T10:32:04.507 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 42.429483251999997, 2024-04-23T10:32:04.507 INFO:teuthology.orchestra.run.smithi114.stdout: "continuous": false, 2024-04-23T10:32:04.507 INFO:teuthology.orchestra.run.smithi114.stdout: "type_data": { 2024-04-23T10:32:04.507 INFO:teuthology.orchestra.run.smithi114.stdout: "result": -2147483648, 2024-04-23T10:32:04.507 INFO:teuthology.orchestra.run.smithi114.stdout: "flag_point": "dispatched", 2024-04-23T10:32:04.507 INFO:teuthology.orchestra.run.smithi114.stdout: "reqid": { 2024-04-23T10:32:04.507 INFO:teuthology.orchestra.run.smithi114.stdout: "entity": { 2024-04-23T10:32:04.507 INFO:teuthology.orchestra.run.smithi114.stdout: "type": "mds", 2024-04-23T10:32:04.507 INFO:teuthology.orchestra.run.smithi114.stdout: "num": 1 2024-04-23T10:32:04.507 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-23T10:32:04.507 INFO:teuthology.orchestra.run.smithi114.stdout: "tid": 111 2024-04-23T10:32:04.508 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-23T10:32:04.508 INFO:teuthology.orchestra.run.smithi114.stdout: "op_type": "peer_request", 2024-04-23T10:32:04.508 INFO:teuthology.orchestra.run.smithi114.stdout: "leader_info": { 2024-04-23T10:32:04.508 INFO:teuthology.orchestra.run.smithi114.stdout: "leader": "1" 2024-04-23T10:32:04.508 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-23T10:32:04.508 INFO:teuthology.orchestra.run.smithi114.stdout: "events": [ 2024-04-23T10:32:04.508 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-04-23T10:32:04.508 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-04-23T10:31:21.455336+0000", 2024-04-23T10:32:04.508 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "initiated" 2024-04-23T10:32:04.508 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-23T10:32:04.508 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-04-23T10:32:04.508 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-04-23T10:31:21.455336+0000", 2024-04-23T10:32:04.508 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "header_read" 2024-04-23T10:32:04.508 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-23T10:32:04.508 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-04-23T10:32:04.509 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-04-23T10:31:21.455336+0000", 2024-04-23T10:32:04.509 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "throttled" 2024-04-23T10:32:04.509 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-23T10:32:04.509 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-04-23T10:32:04.509 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-04-23T10:31:21.455341+0000", 2024-04-23T10:32:04.509 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "all_read" 2024-04-23T10:32:04.509 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-23T10:32:04.509 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-04-23T10:32:04.509 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-04-23T10:31:21.456770+0000", 2024-04-23T10:32:04.509 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "dispatched" 2024-04-23T10:32:04.509 INFO:teuthology.orchestra.run.smithi114.stdout: } 2024-04-23T10:32:04.509 INFO:teuthology.orchestra.run.smithi114.stdout: ], 2024-04-23T10:32:04.509 INFO:teuthology.orchestra.run.smithi114.stdout: "locks": [] 2024-04-23T10:32:04.509 INFO:teuthology.orchestra.run.smithi114.stdout: } 2024-04-23T10:32:04.509 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-23T10:32:04.509 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-04-23T10:32:04.510 INFO:teuthology.orchestra.run.smithi114.stdout: "description": "internal op quiesce_inode:mds.0:189 fp=#0x100000014df", 2024-04-23T10:32:04.510 INFO:teuthology.orchestra.run.smithi114.stdout: "initiated_at": "2024-04-23T10:31:21.471306+0000", 2024-04-23T10:32:04.510 INFO:teuthology.orchestra.run.smithi114.stdout: "age": 42.413082793999997, 2024-04-23T10:32:04.510 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 42.413536282000003, 2024-04-23T10:32:04.510 INFO:teuthology.orchestra.run.smithi114.stdout: "continuous": true, 2024-04-23T10:32:04.510 INFO:teuthology.orchestra.run.smithi114.stdout: "type_data": { 2024-04-23T10:32:04.510 INFO:teuthology.orchestra.run.smithi114.stdout: "result": -2147483648, 2024-04-23T10:32:04.510 INFO:teuthology.orchestra.run.smithi114.stdout: "flag_point": "quiesce complete", 2024-04-23T10:32:04.510 INFO:teuthology.orchestra.run.smithi114.stdout: "reqid": { 2024-04-23T10:32:04.510 INFO:teuthology.orchestra.run.smithi114.stdout: "entity": { 2024-04-23T10:32:04.510 INFO:teuthology.orchestra.run.smithi114.stdout: "type": "mds", 2024-04-23T10:32:04.510 INFO:teuthology.orchestra.run.smithi114.stdout: "num": 0 2024-04-23T10:32:04.510 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-23T10:32:04.510 INFO:teuthology.orchestra.run.smithi114.stdout: "tid": 189 2024-04-23T10:32:04.510 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-23T10:32:04.511 INFO:teuthology.orchestra.run.smithi114.stdout: "op_type": "internal_op", 2024-04-23T10:32:04.511 INFO:teuthology.orchestra.run.smithi114.stdout: "internal_op": 5385, 2024-04-23T10:32:04.511 INFO:teuthology.orchestra.run.smithi114.stdout: "op_name": "quiesce_inode", 2024-04-23T10:32:04.511 INFO:teuthology.orchestra.run.smithi114.stdout: "events": [ 2024-04-23T10:32:04.511 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-04-23T10:32:04.511 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-04-23T10:31:21.471306+0000", 2024-04-23T10:32:04.511 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "initiated" 2024-04-23T10:32:04.511 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-23T10:32:04.511 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-04-23T10:32:04.511 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-04-23T10:31:21.471306+0000", 2024-04-23T10:32:04.511 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "header_read" 2024-04-23T10:32:04.511 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-23T10:32:04.511 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-04-23T10:32:04.511 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-04-23T10:31:21.471306+0000", 2024-04-23T10:32:04.511 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "throttled" 2024-04-23T10:32:04.512 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-23T10:32:04.512 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-04-23T10:32:04.512 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-04-23T10:31:21.471306+0000", 2024-04-23T10:32:04.512 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "all_read" 2024-04-23T10:32:04.512 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-23T10:32:04.512 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-04-23T10:32:04.512 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-04-23T10:31:21.471306+0000", 2024-04-23T10:32:04.512 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "dispatched" 2024-04-23T10:32:04.512 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-23T10:32:04.512 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-04-23T10:32:04.512 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-04-23T10:31:21.472207+0000", 2024-04-23T10:32:04.512 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "failed to xlock, waiting" 2024-04-23T10:32:04.512 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-23T10:32:04.512 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-04-23T10:32:04.512 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-04-23T10:31:21.505006+0000", 2024-04-23T10:32:04.513 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "failed to xlock, waiting" 2024-04-23T10:32:04.513 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-23T10:32:04.513 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-04-23T10:32:04.513 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-04-23T10:31:21.507040+0000", 2024-04-23T10:32:04.513 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "acquired locks" 2024-04-23T10:32:04.513 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-23T10:32:04.513 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-04-23T10:32:04.513 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-04-23T10:31:21.507152+0000", 2024-04-23T10:32:04.513 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "quiesce complete" 2024-04-23T10:32:04.513 INFO:teuthology.orchestra.run.smithi114.stdout: } 2024-04-23T10:32:04.513 INFO:teuthology.orchestra.run.smithi114.stdout: ], 2024-04-23T10:32:04.513 INFO:teuthology.orchestra.run.smithi114.stdout: "locks": [ 2024-04-23T10:32:04.513 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-04-23T10:32:04.513 INFO:teuthology.orchestra.run.smithi114.stdout: "object": { 2024-04-23T10:32:04.514 INFO:teuthology.orchestra.run.smithi114.stdout: "is_auth": true, 2024-04-23T10:32:04.514 INFO:teuthology.orchestra.run.smithi114.stdout: "auth_state": { 2024-04-23T10:32:04.514 INFO:teuthology.orchestra.run.smithi114.stdout: "replicas": {} 2024-04-23T10:32:04.514 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-23T10:32:04.514 INFO:teuthology.orchestra.run.smithi114.stdout: "replica_state": { 2024-04-23T10:32:04.514 INFO:teuthology.orchestra.run.smithi114.stdout: "authority": [ 2024-04-23T10:32:04.514 INFO:teuthology.orchestra.run.smithi114.stdout: 0, 2024-04-23T10:32:04.514 INFO:teuthology.orchestra.run.smithi114.stdout: -2 2024-04-23T10:32:04.514 INFO:teuthology.orchestra.run.smithi114.stdout: ], 2024-04-23T10:32:04.514 INFO:teuthology.orchestra.run.smithi114.stdout: "replica_nonce": 0 2024-04-23T10:32:04.514 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-23T10:32:04.514 INFO:teuthology.orchestra.run.smithi114.stdout: "auth_pins": 1, 2024-04-23T10:32:04.514 INFO:teuthology.orchestra.run.smithi114.stdout: "is_frozen": false, 2024-04-23T10:32:04.514 INFO:teuthology.orchestra.run.smithi114.stdout: "is_freezing": false, 2024-04-23T10:32:04.515 INFO:teuthology.orchestra.run.smithi114.stdout: "pins": { 2024-04-23T10:32:04.515 INFO:teuthology.orchestra.run.smithi114.stdout: "ptrwaiter": 0, 2024-04-23T10:32:04.515 INFO:teuthology.orchestra.run.smithi114.stdout: "request": 1, 2024-04-23T10:32:04.515 INFO:teuthology.orchestra.run.smithi114.stdout: "lock": 1, 2024-04-23T10:32:04.515 INFO:teuthology.orchestra.run.smithi114.stdout: "caps": 0, 2024-04-23T10:32:04.515 INFO:teuthology.orchestra.run.smithi114.stdout: "dirtyparent": 0, 2024-04-23T10:32:04.515 INFO:teuthology.orchestra.run.smithi114.stdout: "dirty": 0, 2024-04-23T10:32:04.515 INFO:teuthology.orchestra.run.smithi114.stdout: "waiter": 0, 2024-04-23T10:32:04.515 INFO:teuthology.orchestra.run.smithi114.stdout: "authpin": 1 2024-04-23T10:32:04.515 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-23T10:32:04.515 INFO:teuthology.orchestra.run.smithi114.stdout: "nref": 3 2024-04-23T10:32:04.515 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-23T10:32:04.515 INFO:teuthology.orchestra.run.smithi114.stdout: "object_string": "[inode 0x100000014df [7,head] /volumes/qa/sv_1/b206794e-7622-4ab8-8943-f3ae4e65b7a3/client.0/tmp/clients/client0/~dmtmp/ACCESS/LABELS.PRN auth v3328 ap=1 s=30118 n(v0 rc2024-04-23T10:31:17.038732+0000 b30118 1=1+0) (iquiesce lock x=1 by request(mds.0:189 nref=3)) | ptrwaiter=0 request=1 lock=1 caps=0 dirtyparent=0 dirty=0 waiter=0 authpin=1 0x5571cfaaf180]", 2024-04-23T10:32:04.515 INFO:teuthology.orchestra.run.smithi114.stdout: "lock": { 2024-04-23T10:32:04.515 INFO:teuthology.orchestra.run.smithi114.stdout: "gather_set": [], 2024-04-23T10:32:04.515 INFO:teuthology.orchestra.run.smithi114.stdout: "state": "lock", 2024-04-23T10:32:04.516 INFO:teuthology.orchestra.run.smithi114.stdout: "type": "iquiesce", 2024-04-23T10:32:04.516 INFO:teuthology.orchestra.run.smithi114.stdout: "is_leased": false, 2024-04-23T10:32:04.516 INFO:teuthology.orchestra.run.smithi114.stdout: "num_rdlocks": 0, 2024-04-23T10:32:04.516 INFO:teuthology.orchestra.run.smithi114.stdout: "num_wrlocks": 0, 2024-04-23T10:32:04.516 INFO:teuthology.orchestra.run.smithi114.stdout: "num_xlocks": 1, 2024-04-23T10:32:04.516 INFO:teuthology.orchestra.run.smithi114.stdout: "xlock_by": { 2024-04-23T10:32:04.516 INFO:teuthology.orchestra.run.smithi114.stdout: "reqid": { 2024-04-23T10:32:04.516 INFO:teuthology.orchestra.run.smithi114.stdout: "entity": { 2024-04-23T10:32:04.516 INFO:teuthology.orchestra.run.smithi114.stdout: "type": "mds", 2024-04-23T10:32:04.516 INFO:teuthology.orchestra.run.smithi114.stdout: "num": 0 2024-04-23T10:32:04.516 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-23T10:32:04.516 INFO:teuthology.orchestra.run.smithi114.stdout: "tid": 189 2024-04-23T10:32:04.516 INFO:teuthology.orchestra.run.smithi114.stdout: } 2024-04-23T10:32:04.516 INFO:teuthology.orchestra.run.smithi114.stdout: } 2024-04-23T10:32:04.517 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-23T10:32:04.517 INFO:teuthology.orchestra.run.smithi114.stdout: "flags": 4, 2024-04-23T10:32:04.517 INFO:teuthology.orchestra.run.smithi114.stdout: "wrlock_target": -1 2024-04-23T10:32:04.517 INFO:teuthology.orchestra.run.smithi114.stdout: } 2024-04-23T10:32:04.517 INFO:teuthology.orchestra.run.smithi114.stdout: ] 2024-04-23T10:32:04.517 INFO:teuthology.orchestra.run.smithi114.stdout: } 2024-04-23T10:32:04.517 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-23T10:32:04.517 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-04-23T10:32:04.517 INFO:teuthology.orchestra.run.smithi114.stdout: "description": "peer_request:mds.2:139", 2024-04-23T10:32:04.517 INFO:teuthology.orchestra.run.smithi114.stdout: "initiated_at": "2024-04-23T10:31:21.489583+0000", 2024-04-23T10:32:04.517 INFO:teuthology.orchestra.run.smithi114.stdout: "age": 42.394806473000003, 2024-04-23T10:32:04.517 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 42.395343420000003, 2024-04-23T10:32:04.517 INFO:teuthology.orchestra.run.smithi114.stdout: "continuous": false, 2024-04-23T10:32:04.517 INFO:teuthology.orchestra.run.smithi114.stdout: "type_data": { 2024-04-23T10:32:04.517 INFO:teuthology.orchestra.run.smithi114.stdout: "result": -2147483648, 2024-04-23T10:32:04.518 INFO:teuthology.orchestra.run.smithi114.stdout: "flag_point": "dispatched", 2024-04-23T10:32:04.518 INFO:teuthology.orchestra.run.smithi114.stdout: "reqid": { 2024-04-23T10:32:04.518 INFO:teuthology.orchestra.run.smithi114.stdout: "entity": { 2024-04-23T10:32:04.518 INFO:teuthology.orchestra.run.smithi114.stdout: "type": "mds", 2024-04-23T10:32:04.518 INFO:teuthology.orchestra.run.smithi114.stdout: "num": 2 2024-04-23T10:32:04.518 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-23T10:32:04.518 INFO:teuthology.orchestra.run.smithi114.stdout: "tid": 139 2024-04-23T10:32:04.518 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-23T10:32:04.518 INFO:teuthology.orchestra.run.smithi114.stdout: "op_type": "peer_request", 2024-04-23T10:32:04.518 INFO:teuthology.orchestra.run.smithi114.stdout: "leader_info": { 2024-04-23T10:32:04.518 INFO:teuthology.orchestra.run.smithi114.stdout: "leader": "2" 2024-04-23T10:32:04.518 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-23T10:32:04.518 INFO:teuthology.orchestra.run.smithi114.stdout: "events": [ 2024-04-23T10:32:04.518 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-04-23T10:32:04.518 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-04-23T10:31:21.489583+0000", 2024-04-23T10:32:04.519 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "initiated" 2024-04-23T10:32:04.519 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-23T10:32:04.519 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-04-23T10:32:04.519 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-04-23T10:31:21.489583+0000", 2024-04-23T10:32:04.519 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "header_read" 2024-04-23T10:32:04.519 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-23T10:32:04.519 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-04-23T10:32:04.519 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-04-23T10:31:21.489583+0000", 2024-04-23T10:32:04.519 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "throttled" 2024-04-23T10:32:04.519 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-23T10:32:04.519 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-04-23T10:32:04.519 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-04-23T10:31:21.489589+0000", 2024-04-23T10:32:04.519 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "all_read" 2024-04-23T10:32:04.519 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-23T10:32:04.519 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-04-23T10:32:04.520 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-04-23T10:31:21.508404+0000", 2024-04-23T10:32:04.520 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "dispatched" 2024-04-23T10:32:04.520 INFO:teuthology.orchestra.run.smithi114.stdout: } 2024-04-23T10:32:04.520 INFO:teuthology.orchestra.run.smithi114.stdout: ], 2024-04-23T10:32:04.520 INFO:teuthology.orchestra.run.smithi114.stdout: "locks": [] 2024-04-23T10:32:04.520 INFO:teuthology.orchestra.run.smithi114.stdout: } 2024-04-23T10:32:04.520 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-23T10:32:04.520 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-04-23T10:32:04.520 INFO:teuthology.orchestra.run.smithi114.stdout: "description": "internal op quiesce_inode:mds.0:160 fp=#0x10000000009", 2024-04-23T10:32:04.520 INFO:teuthology.orchestra.run.smithi114.stdout: "initiated_at": "2024-04-23T10:31:21.457565+0000", 2024-04-23T10:32:04.520 INFO:teuthology.orchestra.run.smithi114.stdout: "age": 42.426823845000001, 2024-04-23T10:32:04.520 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 42.427383431999999, 2024-04-23T10:32:04.520 INFO:teuthology.orchestra.run.smithi114.stdout: "continuous": true, 2024-04-23T10:32:04.520 INFO:teuthology.orchestra.run.smithi114.stdout: "type_data": { 2024-04-23T10:32:04.520 INFO:teuthology.orchestra.run.smithi114.stdout: "result": -2147483648, 2024-04-23T10:32:04.520 INFO:teuthology.orchestra.run.smithi114.stdout: "flag_point": "quiesce complete", 2024-04-23T10:32:04.521 INFO:teuthology.orchestra.run.smithi114.stdout: "reqid": { 2024-04-23T10:32:04.521 INFO:teuthology.orchestra.run.smithi114.stdout: "entity": { 2024-04-23T10:32:04.521 INFO:teuthology.orchestra.run.smithi114.stdout: "type": "mds", 2024-04-23T10:32:04.521 INFO:teuthology.orchestra.run.smithi114.stdout: "num": 0 2024-04-23T10:32:04.521 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-23T10:32:04.521 INFO:teuthology.orchestra.run.smithi114.stdout: "tid": 160 2024-04-23T10:32:04.521 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-23T10:32:04.521 INFO:teuthology.orchestra.run.smithi114.stdout: "op_type": "internal_op", 2024-04-23T10:32:04.521 INFO:teuthology.orchestra.run.smithi114.stdout: "internal_op": 5385, 2024-04-23T10:32:04.521 INFO:teuthology.orchestra.run.smithi114.stdout: "op_name": "quiesce_inode", 2024-04-23T10:32:04.521 INFO:teuthology.orchestra.run.smithi114.stdout: "events": [ 2024-04-23T10:32:04.521 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-04-23T10:32:04.521 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-04-23T10:31:21.457565+0000", 2024-04-23T10:32:04.522 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "initiated" 2024-04-23T10:32:04.522 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-23T10:32:04.522 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-04-23T10:32:04.522 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-04-23T10:31:21.457565+0000", 2024-04-23T10:32:04.522 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "header_read" 2024-04-23T10:32:04.522 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-23T10:32:04.522 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-04-23T10:32:04.522 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-04-23T10:31:21.457565+0000", 2024-04-23T10:32:04.522 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "throttled" 2024-04-23T10:32:04.522 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-23T10:32:04.522 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-04-23T10:32:04.522 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-04-23T10:31:21.457565+0000", 2024-04-23T10:32:04.522 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "all_read" 2024-04-23T10:32:04.522 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-23T10:32:04.523 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-04-23T10:32:04.523 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-04-23T10:31:21.457565+0000", 2024-04-23T10:32:04.523 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "dispatched" 2024-04-23T10:32:04.523 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-23T10:32:04.523 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-04-23T10:32:04.523 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-04-23T10:31:21.457783+0000", 2024-04-23T10:32:04.523 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "failed to xlock, waiting" 2024-04-23T10:32:04.523 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-23T10:32:04.523 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-04-23T10:32:04.523 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-04-23T10:31:21.459494+0000", 2024-04-23T10:32:04.523 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "acquired locks" 2024-04-23T10:32:04.523 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-23T10:32:04.523 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-04-23T10:32:04.523 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-04-23T10:31:21.509817+0000", 2024-04-23T10:32:04.523 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "quiesce complete" 2024-04-23T10:32:04.524 INFO:teuthology.orchestra.run.smithi114.stdout: } 2024-04-23T10:32:04.524 INFO:teuthology.orchestra.run.smithi114.stdout: ], 2024-04-23T10:32:04.524 INFO:teuthology.orchestra.run.smithi114.stdout: "locks": [ 2024-04-23T10:32:04.524 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-04-23T10:32:04.524 INFO:teuthology.orchestra.run.smithi114.stdout: "object": { 2024-04-23T10:32:04.524 INFO:teuthology.orchestra.run.smithi114.stdout: "is_auth": true, 2024-04-23T10:32:04.524 INFO:teuthology.orchestra.run.smithi114.stdout: "auth_state": { 2024-04-23T10:32:04.524 INFO:teuthology.orchestra.run.smithi114.stdout: "replicas": { 2024-04-23T10:32:04.524 INFO:teuthology.orchestra.run.smithi114.stdout: "1": 1, 2024-04-23T10:32:04.524 INFO:teuthology.orchestra.run.smithi114.stdout: "2": 1 2024-04-23T10:32:04.524 INFO:teuthology.orchestra.run.smithi114.stdout: } 2024-04-23T10:32:04.524 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-23T10:32:04.524 INFO:teuthology.orchestra.run.smithi114.stdout: "replica_state": { 2024-04-23T10:32:04.524 INFO:teuthology.orchestra.run.smithi114.stdout: "authority": [ 2024-04-23T10:32:04.524 INFO:teuthology.orchestra.run.smithi114.stdout: 0, 2024-04-23T10:32:04.525 INFO:teuthology.orchestra.run.smithi114.stdout: -2 2024-04-23T10:32:04.525 INFO:teuthology.orchestra.run.smithi114.stdout: ], 2024-04-23T10:32:04.525 INFO:teuthology.orchestra.run.smithi114.stdout: "replica_nonce": 0 2024-04-23T10:32:04.525 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-23T10:32:04.525 INFO:teuthology.orchestra.run.smithi114.stdout: "auth_pins": 3, 2024-04-23T10:32:04.525 INFO:teuthology.orchestra.run.smithi114.stdout: "is_frozen": false, 2024-04-23T10:32:04.525 INFO:teuthology.orchestra.run.smithi114.stdout: "is_freezing": false, 2024-04-23T10:32:04.525 INFO:teuthology.orchestra.run.smithi114.stdout: "pins": { 2024-04-23T10:32:04.525 INFO:teuthology.orchestra.run.smithi114.stdout: "dirtyscattered": 0, 2024-04-23T10:32:04.525 INFO:teuthology.orchestra.run.smithi114.stdout: "request": 1, 2024-04-23T10:32:04.525 INFO:teuthology.orchestra.run.smithi114.stdout: "lock": 1, 2024-04-23T10:32:04.525 INFO:teuthology.orchestra.run.smithi114.stdout: "dirfrag": 1, 2024-04-23T10:32:04.525 INFO:teuthology.orchestra.run.smithi114.stdout: "caps": 1, 2024-04-23T10:32:04.525 INFO:teuthology.orchestra.run.smithi114.stdout: "dirtyrstat": 0, 2024-04-23T10:32:04.525 INFO:teuthology.orchestra.run.smithi114.stdout: "dirtyparent": 0, 2024-04-23T10:32:04.526 INFO:teuthology.orchestra.run.smithi114.stdout: "replicated": 1, 2024-04-23T10:32:04.526 INFO:teuthology.orchestra.run.smithi114.stdout: "dirty": 0, 2024-04-23T10:32:04.526 INFO:teuthology.orchestra.run.smithi114.stdout: "waiter": 0, 2024-04-23T10:32:04.526 INFO:teuthology.orchestra.run.smithi114.stdout: "authpin": 1 2024-04-23T10:32:04.526 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-23T10:32:04.526 INFO:teuthology.orchestra.run.smithi114.stdout: "nref": 6 2024-04-23T10:32:04.526 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-23T10:32:04.526 INFO:teuthology.orchestra.run.smithi114.stdout: "object_string": "[inode 0x10000000009 [...7,head] /volumes/qa/sv_1/b206794e-7622-4ab8-8943-f3ae4e65b7a3/client.0/tmp/clients/client0/~dmtmp/ auth{1=1,2=1} v15178 ap=3 f(v0 m2024-04-23T10:27:52.186392+0000 9=0+9) n(v31 rc2024-04-23T10:31:17.764716+0000 b15832194 68=58+10) old_inodes=4 (inest mix) (iquiesce lock x=1 by request(mds.0:160 nref=3)) caps={15312=pAsLsXsFs/pAsLsXsFsxcral@56},l=15312 | dirtyscattered=0 request=1 lock=1 dirfrag=1 caps=1 dirtyrstat=0 dirtyparent=0 replicated=1 dirty=0 waiter=0 authpin=1 0x5571cf341080]", 2024-04-23T10:32:04.526 INFO:teuthology.orchestra.run.smithi114.stdout: "lock": { 2024-04-23T10:32:04.526 INFO:teuthology.orchestra.run.smithi114.stdout: "gather_set": [], 2024-04-23T10:32:04.526 INFO:teuthology.orchestra.run.smithi114.stdout: "state": "lock", 2024-04-23T10:32:04.526 INFO:teuthology.orchestra.run.smithi114.stdout: "type": "iquiesce", 2024-04-23T10:32:04.526 INFO:teuthology.orchestra.run.smithi114.stdout: "is_leased": false, 2024-04-23T10:32:04.526 INFO:teuthology.orchestra.run.smithi114.stdout: "num_rdlocks": 0, 2024-04-23T10:32:04.526 INFO:teuthology.orchestra.run.smithi114.stdout: "num_wrlocks": 0, 2024-04-23T10:32:04.527 INFO:teuthology.orchestra.run.smithi114.stdout: "num_xlocks": 1, 2024-04-23T10:32:04.527 INFO:teuthology.orchestra.run.smithi114.stdout: "xlock_by": { 2024-04-23T10:32:04.527 INFO:teuthology.orchestra.run.smithi114.stdout: "reqid": { 2024-04-23T10:32:04.527 INFO:teuthology.orchestra.run.smithi114.stdout: "entity": { 2024-04-23T10:32:04.527 INFO:teuthology.orchestra.run.smithi114.stdout: "type": "mds", 2024-04-23T10:32:04.527 INFO:teuthology.orchestra.run.smithi114.stdout: "num": 0 2024-04-23T10:32:04.527 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-23T10:32:04.527 INFO:teuthology.orchestra.run.smithi114.stdout: "tid": 160 2024-04-23T10:32:04.527 INFO:teuthology.orchestra.run.smithi114.stdout: } 2024-04-23T10:32:04.527 INFO:teuthology.orchestra.run.smithi114.stdout: } 2024-04-23T10:32:04.527 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-23T10:32:04.527 INFO:teuthology.orchestra.run.smithi114.stdout: "flags": 4, 2024-04-23T10:32:04.527 INFO:teuthology.orchestra.run.smithi114.stdout: "wrlock_target": -1 2024-04-23T10:32:04.527 INFO:teuthology.orchestra.run.smithi114.stdout: } 2024-04-23T10:32:04.528 INFO:teuthology.orchestra.run.smithi114.stdout: ] 2024-04-23T10:32:04.528 INFO:teuthology.orchestra.run.smithi114.stdout: } 2024-04-23T10:32:04.528 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-23T10:32:04.528 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-04-23T10:32:04.528 INFO:teuthology.orchestra.run.smithi114.stdout: "description": "peer_request:mds.2:117", 2024-04-23T10:32:04.528 INFO:teuthology.orchestra.run.smithi114.stdout: "initiated_at": "2024-04-23T10:31:21.463330+0000", 2024-04-23T10:32:04.528 INFO:teuthology.orchestra.run.smithi114.stdout: "age": 42.421059130000003, 2024-04-23T10:32:04.528 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 42.421691584999998, 2024-04-23T10:32:04.528 INFO:teuthology.orchestra.run.smithi114.stdout: "continuous": false, 2024-04-23T10:32:04.528 INFO:teuthology.orchestra.run.smithi114.stdout: "type_data": { 2024-04-23T10:32:04.528 INFO:teuthology.orchestra.run.smithi114.stdout: "result": -2147483648, 2024-04-23T10:32:04.528 INFO:teuthology.orchestra.run.smithi114.stdout: "flag_point": "dispatched", 2024-04-23T10:32:04.528 INFO:teuthology.orchestra.run.smithi114.stdout: "reqid": { 2024-04-23T10:32:04.528 INFO:teuthology.orchestra.run.smithi114.stdout: "entity": { 2024-04-23T10:32:04.528 INFO:teuthology.orchestra.run.smithi114.stdout: "type": "mds", 2024-04-23T10:32:04.529 INFO:teuthology.orchestra.run.smithi114.stdout: "num": 2 2024-04-23T10:32:04.529 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-23T10:32:04.529 INFO:teuthology.orchestra.run.smithi114.stdout: "tid": 117 2024-04-23T10:32:04.529 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-23T10:32:04.529 INFO:teuthology.orchestra.run.smithi114.stdout: "op_type": "peer_request", 2024-04-23T10:32:04.529 INFO:teuthology.orchestra.run.smithi114.stdout: "leader_info": { 2024-04-23T10:32:04.529 INFO:teuthology.orchestra.run.smithi114.stdout: "leader": "2" 2024-04-23T10:32:04.529 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-23T10:32:04.529 INFO:teuthology.orchestra.run.smithi114.stdout: "events": [ 2024-04-23T10:32:04.529 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-04-23T10:32:04.529 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-04-23T10:31:21.463330+0000", 2024-04-23T10:32:04.529 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "initiated" 2024-04-23T10:32:04.529 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-23T10:32:04.529 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-04-23T10:32:04.529 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-04-23T10:31:21.463330+0000", 2024-04-23T10:32:04.529 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "header_read" 2024-04-23T10:32:04.530 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-23T10:32:04.530 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-04-23T10:32:04.530 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-04-23T10:31:21.463331+0000", 2024-04-23T10:32:04.530 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "throttled" 2024-04-23T10:32:04.530 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-23T10:32:04.530 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-04-23T10:32:04.530 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-04-23T10:31:21.463338+0000", 2024-04-23T10:32:04.530 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "all_read" 2024-04-23T10:32:04.530 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-23T10:32:04.530 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-04-23T10:32:04.530 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-04-23T10:31:21.477419+0000", 2024-04-23T10:32:04.530 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "dispatched" 2024-04-23T10:32:04.530 INFO:teuthology.orchestra.run.smithi114.stdout: } 2024-04-23T10:32:04.531 INFO:teuthology.orchestra.run.smithi114.stdout: ], 2024-04-23T10:32:04.531 INFO:teuthology.orchestra.run.smithi114.stdout: "locks": [] 2024-04-23T10:32:04.531 INFO:teuthology.orchestra.run.smithi114.stdout: } 2024-04-23T10:32:04.531 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-23T10:32:04.531 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-04-23T10:32:04.531 INFO:teuthology.orchestra.run.smithi114.stdout: "description": "peer_request:mds.1:100", 2024-04-23T10:32:04.531 INFO:teuthology.orchestra.run.smithi114.stdout: "initiated_at": "2024-04-23T10:31:22.426164+0000", 2024-04-23T10:32:04.531 INFO:teuthology.orchestra.run.smithi114.stdout: "age": 41.458225577, 2024-04-23T10:32:04.531 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 41.458880985999997, 2024-04-23T10:32:04.531 INFO:teuthology.orchestra.run.smithi114.stdout: "continuous": false, 2024-04-23T10:32:04.531 INFO:teuthology.orchestra.run.smithi114.stdout: "type_data": { 2024-04-23T10:32:04.531 INFO:teuthology.orchestra.run.smithi114.stdout: "result": -2147483648, 2024-04-23T10:32:04.531 INFO:teuthology.orchestra.run.smithi114.stdout: "flag_point": "dispatched", 2024-04-23T10:32:04.531 INFO:teuthology.orchestra.run.smithi114.stdout: "reqid": { 2024-04-23T10:32:04.532 INFO:teuthology.orchestra.run.smithi114.stdout: "entity": { 2024-04-23T10:32:04.532 INFO:teuthology.orchestra.run.smithi114.stdout: "type": "mds", 2024-04-23T10:32:04.532 INFO:teuthology.orchestra.run.smithi114.stdout: "num": 1 2024-04-23T10:32:04.532 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-23T10:32:04.532 INFO:teuthology.orchestra.run.smithi114.stdout: "tid": 100 2024-04-23T10:32:04.532 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-23T10:32:04.532 INFO:teuthology.orchestra.run.smithi114.stdout: "op_type": "peer_request", 2024-04-23T10:32:04.532 INFO:teuthology.orchestra.run.smithi114.stdout: "leader_info": { 2024-04-23T10:32:04.532 INFO:teuthology.orchestra.run.smithi114.stdout: "leader": "1" 2024-04-23T10:32:04.532 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-23T10:32:04.532 INFO:teuthology.orchestra.run.smithi114.stdout: "events": [ 2024-04-23T10:32:04.532 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-04-23T10:32:04.532 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-04-23T10:31:22.426164+0000", 2024-04-23T10:32:04.532 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "initiated" 2024-04-23T10:32:04.532 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-23T10:32:04.533 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-04-23T10:32:04.533 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-04-23T10:31:22.426164+0000", 2024-04-23T10:32:04.533 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "header_read" 2024-04-23T10:32:04.533 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-23T10:32:04.533 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-04-23T10:32:04.533 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-04-23T10:31:22.426165+0000", 2024-04-23T10:32:04.533 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "throttled" 2024-04-23T10:32:04.533 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-23T10:32:04.533 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-04-23T10:32:04.533 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-04-23T10:31:22.426175+0000", 2024-04-23T10:32:04.533 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "all_read" 2024-04-23T10:32:04.533 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-23T10:32:04.533 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-04-23T10:32:04.533 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-04-23T10:31:22.426205+0000", 2024-04-23T10:32:04.534 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "dispatched" 2024-04-23T10:32:04.534 INFO:teuthology.orchestra.run.smithi114.stdout: } 2024-04-23T10:32:04.534 INFO:teuthology.orchestra.run.smithi114.stdout: ], 2024-04-23T10:32:04.534 INFO:teuthology.orchestra.run.smithi114.stdout: "locks": [] 2024-04-23T10:32:04.534 INFO:teuthology.orchestra.run.smithi114.stdout: } 2024-04-23T10:32:04.534 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-23T10:32:04.534 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-04-23T10:32:04.534 INFO:teuthology.orchestra.run.smithi114.stdout: "description": "internal op quiesce_inode:mds.0:161 fp=#0x10000000204", 2024-04-23T10:32:04.534 INFO:teuthology.orchestra.run.smithi114.stdout: "initiated_at": "2024-04-23T10:31:21.457577+0000", 2024-04-23T10:32:04.534 INFO:teuthology.orchestra.run.smithi114.stdout: "age": 42.426811915999998, 2024-04-23T10:32:04.534 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 42.427489895999997, 2024-04-23T10:32:04.534 INFO:teuthology.orchestra.run.smithi114.stdout: "continuous": true, 2024-04-23T10:32:04.534 INFO:teuthology.orchestra.run.smithi114.stdout: "type_data": { 2024-04-23T10:32:04.535 INFO:teuthology.orchestra.run.smithi114.stdout: "result": -2147483648, 2024-04-23T10:32:04.535 INFO:teuthology.orchestra.run.smithi114.stdout: "flag_point": "quiesce complete", 2024-04-23T10:32:04.535 INFO:teuthology.orchestra.run.smithi114.stdout: "reqid": { 2024-04-23T10:32:04.535 INFO:teuthology.orchestra.run.smithi114.stdout: "entity": { 2024-04-23T10:32:04.535 INFO:teuthology.orchestra.run.smithi114.stdout: "type": "mds", 2024-04-23T10:32:04.535 INFO:teuthology.orchestra.run.smithi114.stdout: "num": 0 2024-04-23T10:32:04.535 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-23T10:32:04.535 INFO:teuthology.orchestra.run.smithi114.stdout: "tid": 161 2024-04-23T10:32:04.535 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-23T10:32:04.535 INFO:teuthology.orchestra.run.smithi114.stdout: "op_type": "internal_op", 2024-04-23T10:32:04.535 INFO:teuthology.orchestra.run.smithi114.stdout: "internal_op": 5385, 2024-04-23T10:32:04.535 INFO:teuthology.orchestra.run.smithi114.stdout: "op_name": "quiesce_inode", 2024-04-23T10:32:04.535 INFO:teuthology.orchestra.run.smithi114.stdout: "events": [ 2024-04-23T10:32:04.535 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-04-23T10:32:04.536 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-04-23T10:31:21.457577+0000", 2024-04-23T10:32:04.536 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "initiated" 2024-04-23T10:32:04.536 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-23T10:32:04.536 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-04-23T10:32:04.536 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-04-23T10:31:21.457577+0000", 2024-04-23T10:32:04.536 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "header_read" 2024-04-23T10:32:04.536 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-23T10:32:04.536 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-04-23T10:32:04.536 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-04-23T10:31:21.457577+0000", 2024-04-23T10:32:04.536 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "throttled" 2024-04-23T10:32:04.536 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-23T10:32:04.536 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-04-23T10:32:04.536 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-04-23T10:31:21.457577+0000", 2024-04-23T10:32:04.537 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "all_read" 2024-04-23T10:32:04.537 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-23T10:32:04.537 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-04-23T10:32:04.537 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-04-23T10:31:21.457577+0000", 2024-04-23T10:32:04.537 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "dispatched" 2024-04-23T10:32:04.537 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-23T10:32:04.537 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-04-23T10:32:04.537 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-04-23T10:31:21.457900+0000", 2024-04-23T10:32:04.537 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "acquired locks" 2024-04-23T10:32:04.537 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-23T10:32:04.537 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-04-23T10:32:04.537 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-04-23T10:31:21.457989+0000", 2024-04-23T10:32:04.537 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "quiesce complete" 2024-04-23T10:32:04.537 INFO:teuthology.orchestra.run.smithi114.stdout: } 2024-04-23T10:32:04.537 INFO:teuthology.orchestra.run.smithi114.stdout: ], 2024-04-23T10:32:04.537 INFO:teuthology.orchestra.run.smithi114.stdout: "locks": [ 2024-04-23T10:32:04.538 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-04-23T10:32:04.538 INFO:teuthology.orchestra.run.smithi114.stdout: "object": { 2024-04-23T10:32:04.538 INFO:teuthology.orchestra.run.smithi114.stdout: "is_auth": true, 2024-04-23T10:32:04.538 INFO:teuthology.orchestra.run.smithi114.stdout: "auth_state": { 2024-04-23T10:32:04.538 INFO:teuthology.orchestra.run.smithi114.stdout: "replicas": {} 2024-04-23T10:32:04.538 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-23T10:32:04.538 INFO:teuthology.orchestra.run.smithi114.stdout: "replica_state": { 2024-04-23T10:32:04.538 INFO:teuthology.orchestra.run.smithi114.stdout: "authority": [ 2024-04-23T10:32:04.538 INFO:teuthology.orchestra.run.smithi114.stdout: 0, 2024-04-23T10:32:04.538 INFO:teuthology.orchestra.run.smithi114.stdout: -2 2024-04-23T10:32:04.538 INFO:teuthology.orchestra.run.smithi114.stdout: ], 2024-04-23T10:32:04.538 INFO:teuthology.orchestra.run.smithi114.stdout: "replica_nonce": 0 2024-04-23T10:32:04.538 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-23T10:32:04.538 INFO:teuthology.orchestra.run.smithi114.stdout: "auth_pins": 1, 2024-04-23T10:32:04.538 INFO:teuthology.orchestra.run.smithi114.stdout: "is_frozen": false, 2024-04-23T10:32:04.539 INFO:teuthology.orchestra.run.smithi114.stdout: "is_freezing": false, 2024-04-23T10:32:04.539 INFO:teuthology.orchestra.run.smithi114.stdout: "pins": { 2024-04-23T10:32:04.539 INFO:teuthology.orchestra.run.smithi114.stdout: "ptrwaiter": 0, 2024-04-23T10:32:04.539 INFO:teuthology.orchestra.run.smithi114.stdout: "request": 1, 2024-04-23T10:32:04.539 INFO:teuthology.orchestra.run.smithi114.stdout: "lock": 1, 2024-04-23T10:32:04.539 INFO:teuthology.orchestra.run.smithi114.stdout: "caps": 0, 2024-04-23T10:32:04.539 INFO:teuthology.orchestra.run.smithi114.stdout: "dirtyparent": 0, 2024-04-23T10:32:04.539 INFO:teuthology.orchestra.run.smithi114.stdout: "dirty": 0, 2024-04-23T10:32:04.539 INFO:teuthology.orchestra.run.smithi114.stdout: "authpin": 1 2024-04-23T10:32:04.539 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-23T10:32:04.539 INFO:teuthology.orchestra.run.smithi114.stdout: "nref": 3 2024-04-23T10:32:04.539 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-23T10:32:04.539 INFO:teuthology.orchestra.run.smithi114.stdout: "object_string": "[inode 0x10000000204 [4,head] /volumes/qa/sv_1/b206794e-7622-4ab8-8943-f3ae4e65b7a3/client.0/tmp/clients/client0/filler.001 auth v483 ap=1 s=1048576 n(v0 rc2024-04-23T10:27:51.936398+0000 b1048576 1=1+0) (iquiesce lock x=1 by request(mds.0:161 nref=3)) | ptrwaiter=0 request=1 lock=1 caps=0 dirtyparent=0 dirty=0 authpin=1 0x5571cf341b80]", 2024-04-23T10:32:04.539 INFO:teuthology.orchestra.run.smithi114.stdout: "lock": { 2024-04-23T10:32:04.539 INFO:teuthology.orchestra.run.smithi114.stdout: "gather_set": [], 2024-04-23T10:32:04.539 INFO:teuthology.orchestra.run.smithi114.stdout: "state": "lock", 2024-04-23T10:32:04.540 INFO:teuthology.orchestra.run.smithi114.stdout: "type": "iquiesce", 2024-04-23T10:32:04.540 INFO:teuthology.orchestra.run.smithi114.stdout: "is_leased": false, 2024-04-23T10:32:04.540 INFO:teuthology.orchestra.run.smithi114.stdout: "num_rdlocks": 0, 2024-04-23T10:32:04.540 INFO:teuthology.orchestra.run.smithi114.stdout: "num_wrlocks": 0, 2024-04-23T10:32:04.540 INFO:teuthology.orchestra.run.smithi114.stdout: "num_xlocks": 1, 2024-04-23T10:32:04.540 INFO:teuthology.orchestra.run.smithi114.stdout: "xlock_by": { 2024-04-23T10:32:04.540 INFO:teuthology.orchestra.run.smithi114.stdout: "reqid": { 2024-04-23T10:32:04.540 INFO:teuthology.orchestra.run.smithi114.stdout: "entity": { 2024-04-23T10:32:04.540 INFO:teuthology.orchestra.run.smithi114.stdout: "type": "mds", 2024-04-23T10:32:04.540 INFO:teuthology.orchestra.run.smithi114.stdout: "num": 0 2024-04-23T10:32:04.540 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-23T10:32:04.540 INFO:teuthology.orchestra.run.smithi114.stdout: "tid": 161 2024-04-23T10:32:04.540 INFO:teuthology.orchestra.run.smithi114.stdout: } 2024-04-23T10:32:04.540 INFO:teuthology.orchestra.run.smithi114.stdout: } 2024-04-23T10:32:04.540 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-23T10:32:04.541 INFO:teuthology.orchestra.run.smithi114.stdout: "flags": 4, 2024-04-23T10:32:04.541 INFO:teuthology.orchestra.run.smithi114.stdout: "wrlock_target": -1 2024-04-23T10:32:04.541 INFO:teuthology.orchestra.run.smithi114.stdout: } 2024-04-23T10:32:04.541 INFO:teuthology.orchestra.run.smithi114.stdout: ] 2024-04-23T10:32:04.541 INFO:teuthology.orchestra.run.smithi114.stdout: } 2024-04-23T10:32:04.541 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-23T10:32:04.541 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-04-23T10:32:04.541 INFO:teuthology.orchestra.run.smithi114.stdout: "description": "peer_request:mds.1:113", 2024-04-23T10:32:04.541 INFO:teuthology.orchestra.run.smithi114.stdout: "initiated_at": "2024-04-23T10:31:21.464643+0000", 2024-04-23T10:32:04.541 INFO:teuthology.orchestra.run.smithi114.stdout: "age": 42.419745823, 2024-04-23T10:32:04.541 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 42.420481959999996, 2024-04-23T10:32:04.541 INFO:teuthology.orchestra.run.smithi114.stdout: "continuous": false, 2024-04-23T10:32:04.541 INFO:teuthology.orchestra.run.smithi114.stdout: "type_data": { 2024-04-23T10:32:04.541 INFO:teuthology.orchestra.run.smithi114.stdout: "result": -2147483648, 2024-04-23T10:32:04.542 INFO:teuthology.orchestra.run.smithi114.stdout: "flag_point": "dispatched", 2024-04-23T10:32:04.542 INFO:teuthology.orchestra.run.smithi114.stdout: "reqid": { 2024-04-23T10:32:04.542 INFO:teuthology.orchestra.run.smithi114.stdout: "entity": { 2024-04-23T10:32:04.542 INFO:teuthology.orchestra.run.smithi114.stdout: "type": "mds", 2024-04-23T10:32:04.542 INFO:teuthology.orchestra.run.smithi114.stdout: "num": 1 2024-04-23T10:32:04.542 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-23T10:32:04.542 INFO:teuthology.orchestra.run.smithi114.stdout: "tid": 113 2024-04-23T10:32:04.542 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-23T10:32:04.542 INFO:teuthology.orchestra.run.smithi114.stdout: "op_type": "peer_request", 2024-04-23T10:32:04.542 INFO:teuthology.orchestra.run.smithi114.stdout: "leader_info": { 2024-04-23T10:32:04.542 INFO:teuthology.orchestra.run.smithi114.stdout: "leader": "1" 2024-04-23T10:32:04.542 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-23T10:32:04.542 INFO:teuthology.orchestra.run.smithi114.stdout: "events": [ 2024-04-23T10:32:04.542 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-04-23T10:32:04.542 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-04-23T10:31:21.464643+0000", 2024-04-23T10:32:04.543 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "initiated" 2024-04-23T10:32:04.543 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-23T10:32:04.543 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-04-23T10:32:04.543 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-04-23T10:31:21.464643+0000", 2024-04-23T10:32:04.543 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "header_read" 2024-04-23T10:32:04.543 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-23T10:32:04.543 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-04-23T10:32:04.543 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-04-23T10:31:21.464644+0000", 2024-04-23T10:32:04.543 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "throttled" 2024-04-23T10:32:04.543 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-23T10:32:04.543 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-04-23T10:32:04.543 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-04-23T10:31:21.464652+0000", 2024-04-23T10:32:04.543 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "all_read" 2024-04-23T10:32:04.543 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-23T10:32:04.543 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-04-23T10:32:04.544 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-04-23T10:31:21.478154+0000", 2024-04-23T10:32:04.544 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "dispatched" 2024-04-23T10:32:04.544 INFO:teuthology.orchestra.run.smithi114.stdout: } 2024-04-23T10:32:04.544 INFO:teuthology.orchestra.run.smithi114.stdout: ], 2024-04-23T10:32:04.544 INFO:teuthology.orchestra.run.smithi114.stdout: "locks": [] 2024-04-23T10:32:04.544 INFO:teuthology.orchestra.run.smithi114.stdout: } 2024-04-23T10:32:04.544 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-23T10:32:04.544 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-04-23T10:32:04.544 INFO:teuthology.orchestra.run.smithi114.stdout: "description": "internal op quiesce_path:mds.0:149 fp=#0x1", 2024-04-23T10:32:04.544 INFO:teuthology.orchestra.run.smithi114.stdout: "initiated_at": "2024-04-23T10:31:21.445093+0000", 2024-04-23T10:32:04.544 INFO:teuthology.orchestra.run.smithi114.stdout: "age": 42.439296069999997, 2024-04-23T10:32:04.544 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 42.440054617999998, 2024-04-23T10:32:04.544 INFO:teuthology.orchestra.run.smithi114.stdout: "continuous": true, 2024-04-23T10:32:04.544 INFO:teuthology.orchestra.run.smithi114.stdout: "type_data": { 2024-04-23T10:32:04.544 INFO:teuthology.orchestra.run.smithi114.stdout: "result": 0, 2024-04-23T10:32:04.544 INFO:teuthology.orchestra.run.smithi114.stdout: "flag_point": "quiesce complete", 2024-04-23T10:32:04.545 INFO:teuthology.orchestra.run.smithi114.stdout: "reqid": { 2024-04-23T10:32:04.545 INFO:teuthology.orchestra.run.smithi114.stdout: "entity": { 2024-04-23T10:32:04.545 INFO:teuthology.orchestra.run.smithi114.stdout: "type": "mds", 2024-04-23T10:32:04.545 INFO:teuthology.orchestra.run.smithi114.stdout: "num": 0 2024-04-23T10:32:04.545 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-23T10:32:04.545 INFO:teuthology.orchestra.run.smithi114.stdout: "tid": 149 2024-04-23T10:32:04.545 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-23T10:32:04.545 INFO:teuthology.orchestra.run.smithi114.stdout: "op_type": "internal_op", 2024-04-23T10:32:04.545 INFO:teuthology.orchestra.run.smithi114.stdout: "internal_op": 5384, 2024-04-23T10:32:04.545 INFO:teuthology.orchestra.run.smithi114.stdout: "op_name": "quiesce_path", 2024-04-23T10:32:04.545 INFO:teuthology.orchestra.run.smithi114.stdout: "events": [ 2024-04-23T10:32:04.545 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-04-23T10:32:04.545 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-04-23T10:31:21.445093+0000", 2024-04-23T10:32:04.545 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "initiated" 2024-04-23T10:32:04.545 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-23T10:32:04.545 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-04-23T10:32:04.546 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-04-23T10:31:21.445093+0000", 2024-04-23T10:32:04.546 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "header_read" 2024-04-23T10:32:04.546 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-23T10:32:04.546 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-04-23T10:32:04.546 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-04-23T10:31:21.445093+0000", 2024-04-23T10:32:04.546 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "throttled" 2024-04-23T10:32:04.546 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-23T10:32:04.546 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-04-23T10:32:04.546 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-04-23T10:31:21.445093+0000", 2024-04-23T10:32:04.546 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "all_read" 2024-04-23T10:32:04.546 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-23T10:32:04.546 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-04-23T10:32:04.546 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-04-23T10:31:21.445093+0000", 2024-04-23T10:32:04.546 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "dispatched" 2024-04-23T10:32:04.547 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-23T10:32:04.547 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-04-23T10:32:04.547 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-04-23T10:31:21.510121+0000", 2024-04-23T10:32:04.547 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "quiesce complete" 2024-04-23T10:32:04.547 INFO:teuthology.orchestra.run.smithi114.stdout: } 2024-04-23T10:32:04.547 INFO:teuthology.orchestra.run.smithi114.stdout: ], 2024-04-23T10:32:04.547 INFO:teuthology.orchestra.run.smithi114.stdout: "locks": [] 2024-04-23T10:32:04.547 INFO:teuthology.orchestra.run.smithi114.stdout: } 2024-04-23T10:32:04.547 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-23T10:32:04.547 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-04-23T10:32:04.547 INFO:teuthology.orchestra.run.smithi114.stdout: "description": "internal op quiesce_inode:mds.0:162 fp=#0x10000000205", 2024-04-23T10:32:04.547 INFO:teuthology.orchestra.run.smithi114.stdout: "initiated_at": "2024-04-23T10:31:21.457585+0000", 2024-04-23T10:32:04.547 INFO:teuthology.orchestra.run.smithi114.stdout: "age": 42.426803874000001, 2024-04-23T10:32:04.547 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 42.427589140999999, 2024-04-23T10:32:04.547 INFO:teuthology.orchestra.run.smithi114.stdout: "continuous": true, 2024-04-23T10:32:04.547 INFO:teuthology.orchestra.run.smithi114.stdout: "type_data": { 2024-04-23T10:32:04.548 INFO:teuthology.orchestra.run.smithi114.stdout: "result": -2147483648, 2024-04-23T10:32:04.548 INFO:teuthology.orchestra.run.smithi114.stdout: "flag_point": "quiesce complete", 2024-04-23T10:32:04.548 INFO:teuthology.orchestra.run.smithi114.stdout: "reqid": { 2024-04-23T10:32:04.548 INFO:teuthology.orchestra.run.smithi114.stdout: "entity": { 2024-04-23T10:32:04.548 INFO:teuthology.orchestra.run.smithi114.stdout: "type": "mds", 2024-04-23T10:32:04.548 INFO:teuthology.orchestra.run.smithi114.stdout: "num": 0 2024-04-23T10:32:04.548 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-23T10:32:04.548 INFO:teuthology.orchestra.run.smithi114.stdout: "tid": 162 2024-04-23T10:32:04.548 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-23T10:32:04.548 INFO:teuthology.orchestra.run.smithi114.stdout: "op_type": "internal_op", 2024-04-23T10:32:04.548 INFO:teuthology.orchestra.run.smithi114.stdout: "internal_op": 5385, 2024-04-23T10:32:04.548 INFO:teuthology.orchestra.run.smithi114.stdout: "op_name": "quiesce_inode", 2024-04-23T10:32:04.548 INFO:teuthology.orchestra.run.smithi114.stdout: "events": [ 2024-04-23T10:32:04.548 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-04-23T10:32:04.548 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-04-23T10:31:21.457585+0000", 2024-04-23T10:32:04.549 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "initiated" 2024-04-23T10:32:04.549 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-23T10:32:04.549 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-04-23T10:32:04.549 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-04-23T10:31:21.457585+0000", 2024-04-23T10:32:04.549 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "header_read" 2024-04-23T10:32:04.549 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-23T10:32:04.549 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-04-23T10:32:04.549 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-04-23T10:31:21.457585+0000", 2024-04-23T10:32:04.549 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "throttled" 2024-04-23T10:32:04.549 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-23T10:32:04.549 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-04-23T10:32:04.549 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-04-23T10:31:21.457585+0000", 2024-04-23T10:32:04.549 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "all_read" 2024-04-23T10:32:04.549 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-23T10:32:04.549 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-04-23T10:32:04.549 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-04-23T10:31:21.457585+0000", 2024-04-23T10:32:04.550 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "dispatched" 2024-04-23T10:32:04.550 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-23T10:32:04.550 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-04-23T10:32:04.550 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-04-23T10:31:21.458102+0000", 2024-04-23T10:32:04.550 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "acquired locks" 2024-04-23T10:32:04.550 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-23T10:32:04.550 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-04-23T10:32:04.550 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-04-23T10:31:21.458185+0000", 2024-04-23T10:32:04.550 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "quiesce complete" 2024-04-23T10:32:04.550 INFO:teuthology.orchestra.run.smithi114.stdout: } 2024-04-23T10:32:04.550 INFO:teuthology.orchestra.run.smithi114.stdout: ], 2024-04-23T10:32:04.550 INFO:teuthology.orchestra.run.smithi114.stdout: "locks": [ 2024-04-23T10:32:04.550 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-04-23T10:32:04.550 INFO:teuthology.orchestra.run.smithi114.stdout: "object": { 2024-04-23T10:32:04.550 INFO:teuthology.orchestra.run.smithi114.stdout: "is_auth": true, 2024-04-23T10:32:04.551 INFO:teuthology.orchestra.run.smithi114.stdout: "auth_state": { 2024-04-23T10:32:04.551 INFO:teuthology.orchestra.run.smithi114.stdout: "replicas": {} 2024-04-23T10:32:04.551 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-23T10:32:04.551 INFO:teuthology.orchestra.run.smithi114.stdout: "replica_state": { 2024-04-23T10:32:04.551 INFO:teuthology.orchestra.run.smithi114.stdout: "authority": [ 2024-04-23T10:32:04.551 INFO:teuthology.orchestra.run.smithi114.stdout: 0, 2024-04-23T10:32:04.551 INFO:teuthology.orchestra.run.smithi114.stdout: -2 2024-04-23T10:32:04.551 INFO:teuthology.orchestra.run.smithi114.stdout: ], 2024-04-23T10:32:04.551 INFO:teuthology.orchestra.run.smithi114.stdout: "replica_nonce": 0 2024-04-23T10:32:04.551 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-23T10:32:04.551 INFO:teuthology.orchestra.run.smithi114.stdout: "auth_pins": 1, 2024-04-23T10:32:04.551 INFO:teuthology.orchestra.run.smithi114.stdout: "is_frozen": false, 2024-04-23T10:32:04.551 INFO:teuthology.orchestra.run.smithi114.stdout: "is_freezing": false, 2024-04-23T10:32:04.551 INFO:teuthology.orchestra.run.smithi114.stdout: "pins": { 2024-04-23T10:32:04.551 INFO:teuthology.orchestra.run.smithi114.stdout: "ptrwaiter": 0, 2024-04-23T10:32:04.552 INFO:teuthology.orchestra.run.smithi114.stdout: "request": 1, 2024-04-23T10:32:04.552 INFO:teuthology.orchestra.run.smithi114.stdout: "lock": 1, 2024-04-23T10:32:04.552 INFO:teuthology.orchestra.run.smithi114.stdout: "caps": 0, 2024-04-23T10:32:04.552 INFO:teuthology.orchestra.run.smithi114.stdout: "dirtyparent": 0, 2024-04-23T10:32:04.552 INFO:teuthology.orchestra.run.smithi114.stdout: "dirty": 0, 2024-04-23T10:32:04.552 INFO:teuthology.orchestra.run.smithi114.stdout: "authpin": 1 2024-04-23T10:32:04.552 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-23T10:32:04.552 INFO:teuthology.orchestra.run.smithi114.stdout: "nref": 3 2024-04-23T10:32:04.552 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-23T10:32:04.552 INFO:teuthology.orchestra.run.smithi114.stdout: "object_string": "[inode 0x10000000205 [4,head] /volumes/qa/sv_1/b206794e-7622-4ab8-8943-f3ae4e65b7a3/client.0/tmp/clients/client0/filler.002 auth v479 ap=1 s=1048576 n(v0 rc2024-04-23T10:27:51.942398+0000 b1048576 1=1+0) (iquiesce lock x=1 by request(mds.0:162 nref=3)) | ptrwaiter=0 request=1 lock=1 caps=0 dirtyparent=0 dirty=0 authpin=1 0x5571cf342100]", 2024-04-23T10:32:04.552 INFO:teuthology.orchestra.run.smithi114.stdout: "lock": { 2024-04-23T10:32:04.552 INFO:teuthology.orchestra.run.smithi114.stdout: "gather_set": [], 2024-04-23T10:32:04.552 INFO:teuthology.orchestra.run.smithi114.stdout: "state": "lock", 2024-04-23T10:32:04.552 INFO:teuthology.orchestra.run.smithi114.stdout: "type": "iquiesce", 2024-04-23T10:32:04.552 INFO:teuthology.orchestra.run.smithi114.stdout: "is_leased": false, 2024-04-23T10:32:04.553 INFO:teuthology.orchestra.run.smithi114.stdout: "num_rdlocks": 0, 2024-04-23T10:32:04.553 INFO:teuthology.orchestra.run.smithi114.stdout: "num_wrlocks": 0, 2024-04-23T10:32:04.553 INFO:teuthology.orchestra.run.smithi114.stdout: "num_xlocks": 1, 2024-04-23T10:32:04.553 INFO:teuthology.orchestra.run.smithi114.stdout: "xlock_by": { 2024-04-23T10:32:04.553 INFO:teuthology.orchestra.run.smithi114.stdout: "reqid": { 2024-04-23T10:32:04.553 INFO:teuthology.orchestra.run.smithi114.stdout: "entity": { 2024-04-23T10:32:04.553 INFO:teuthology.orchestra.run.smithi114.stdout: "type": "mds", 2024-04-23T10:32:04.553 INFO:teuthology.orchestra.run.smithi114.stdout: "num": 0 2024-04-23T10:32:04.553 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-23T10:32:04.553 INFO:teuthology.orchestra.run.smithi114.stdout: "tid": 162 2024-04-23T10:32:04.553 INFO:teuthology.orchestra.run.smithi114.stdout: } 2024-04-23T10:32:04.553 INFO:teuthology.orchestra.run.smithi114.stdout: } 2024-04-23T10:32:04.553 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-23T10:32:04.553 INFO:teuthology.orchestra.run.smithi114.stdout: "flags": 4, 2024-04-23T10:32:04.553 INFO:teuthology.orchestra.run.smithi114.stdout: "wrlock_target": -1 2024-04-23T10:32:04.554 INFO:teuthology.orchestra.run.smithi114.stdout: } 2024-04-23T10:32:04.554 INFO:teuthology.orchestra.run.smithi114.stdout: ] 2024-04-23T10:32:04.554 INFO:teuthology.orchestra.run.smithi114.stdout: } 2024-04-23T10:32:04.554 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-23T10:32:04.554 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-04-23T10:32:04.554 INFO:teuthology.orchestra.run.smithi114.stdout: "description": "peer_request:mds.2:118", 2024-04-23T10:32:04.554 INFO:teuthology.orchestra.run.smithi114.stdout: "initiated_at": "2024-04-23T10:31:21.463407+0000", 2024-04-23T10:32:04.554 INFO:teuthology.orchestra.run.smithi114.stdout: "age": 42.420982723000002, 2024-04-23T10:32:04.554 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 42.421825890000001, 2024-04-23T10:32:04.554 INFO:teuthology.orchestra.run.smithi114.stdout: "continuous": false, 2024-04-23T10:32:04.554 INFO:teuthology.orchestra.run.smithi114.stdout: "type_data": { 2024-04-23T10:32:04.554 INFO:teuthology.orchestra.run.smithi114.stdout: "result": -2147483648, 2024-04-23T10:32:04.554 INFO:teuthology.orchestra.run.smithi114.stdout: "flag_point": "dispatched", 2024-04-23T10:32:04.554 INFO:teuthology.orchestra.run.smithi114.stdout: "reqid": { 2024-04-23T10:32:04.554 INFO:teuthology.orchestra.run.smithi114.stdout: "entity": { 2024-04-23T10:32:04.555 INFO:teuthology.orchestra.run.smithi114.stdout: "type": "mds", 2024-04-23T10:32:04.555 INFO:teuthology.orchestra.run.smithi114.stdout: "num": 2 2024-04-23T10:32:04.555 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-23T10:32:04.555 INFO:teuthology.orchestra.run.smithi114.stdout: "tid": 118 2024-04-23T10:32:04.555 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-23T10:32:04.555 INFO:teuthology.orchestra.run.smithi114.stdout: "op_type": "peer_request", 2024-04-23T10:32:04.555 INFO:teuthology.orchestra.run.smithi114.stdout: "leader_info": { 2024-04-23T10:32:04.555 INFO:teuthology.orchestra.run.smithi114.stdout: "leader": "2" 2024-04-23T10:32:04.555 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-23T10:32:04.555 INFO:teuthology.orchestra.run.smithi114.stdout: "events": [ 2024-04-23T10:32:04.555 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-04-23T10:32:04.555 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-04-23T10:31:21.463407+0000", 2024-04-23T10:32:04.555 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "initiated" 2024-04-23T10:32:04.555 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-23T10:32:04.555 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-04-23T10:32:04.556 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-04-23T10:31:21.463407+0000", 2024-04-23T10:32:04.556 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "header_read" 2024-04-23T10:32:04.556 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-23T10:32:04.556 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-04-23T10:32:04.556 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-04-23T10:31:21.463407+0000", 2024-04-23T10:32:04.556 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "throttled" 2024-04-23T10:32:04.556 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-23T10:32:04.556 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-04-23T10:32:04.556 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-04-23T10:31:21.463415+0000", 2024-04-23T10:32:04.556 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "all_read" 2024-04-23T10:32:04.556 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-23T10:32:04.556 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-04-23T10:32:04.556 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-04-23T10:31:21.478325+0000", 2024-04-23T10:32:04.556 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "dispatched" 2024-04-23T10:32:04.556 INFO:teuthology.orchestra.run.smithi114.stdout: } 2024-04-23T10:32:04.557 INFO:teuthology.orchestra.run.smithi114.stdout: ], 2024-04-23T10:32:04.557 INFO:teuthology.orchestra.run.smithi114.stdout: "locks": [] 2024-04-23T10:32:04.557 INFO:teuthology.orchestra.run.smithi114.stdout: } 2024-04-23T10:32:04.557 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-23T10:32:04.557 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-04-23T10:32:04.557 INFO:teuthology.orchestra.run.smithi114.stdout: "description": "internal op quiesce_inode:mds.0:150 fp=#0x1", 2024-04-23T10:32:04.557 INFO:teuthology.orchestra.run.smithi114.stdout: "initiated_at": "2024-04-23T10:31:21.445156+0000", 2024-04-23T10:32:04.557 INFO:teuthology.orchestra.run.smithi114.stdout: "age": 42.439233471000001, 2024-04-23T10:32:04.557 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 42.440098505000002, 2024-04-23T10:32:04.557 INFO:teuthology.orchestra.run.smithi114.stdout: "continuous": true, 2024-04-23T10:32:04.557 INFO:teuthology.orchestra.run.smithi114.stdout: "type_data": { 2024-04-23T10:32:04.557 INFO:teuthology.orchestra.run.smithi114.stdout: "result": -2147483648, 2024-04-23T10:32:04.557 INFO:teuthology.orchestra.run.smithi114.stdout: "flag_point": "quiesce complete", 2024-04-23T10:32:04.557 INFO:teuthology.orchestra.run.smithi114.stdout: "reqid": { 2024-04-23T10:32:04.557 INFO:teuthology.orchestra.run.smithi114.stdout: "entity": { 2024-04-23T10:32:04.558 INFO:teuthology.orchestra.run.smithi114.stdout: "type": "mds", 2024-04-23T10:32:04.558 INFO:teuthology.orchestra.run.smithi114.stdout: "num": 0 2024-04-23T10:32:04.558 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-23T10:32:04.558 INFO:teuthology.orchestra.run.smithi114.stdout: "tid": 150 2024-04-23T10:32:04.558 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-23T10:32:04.558 INFO:teuthology.orchestra.run.smithi114.stdout: "op_type": "internal_op", 2024-04-23T10:32:04.558 INFO:teuthology.orchestra.run.smithi114.stdout: "internal_op": 5385, 2024-04-23T10:32:04.558 INFO:teuthology.orchestra.run.smithi114.stdout: "op_name": "quiesce_inode", 2024-04-23T10:32:04.558 INFO:teuthology.orchestra.run.smithi114.stdout: "events": [ 2024-04-23T10:32:04.558 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-04-23T10:32:04.558 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-04-23T10:31:21.445156+0000", 2024-04-23T10:32:04.558 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "initiated" 2024-04-23T10:32:04.558 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-23T10:32:04.558 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-04-23T10:32:04.559 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-04-23T10:31:21.445156+0000", 2024-04-23T10:32:04.559 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "header_read" 2024-04-23T10:32:04.559 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-23T10:32:04.559 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-04-23T10:32:04.559 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-04-23T10:31:21.445156+0000", 2024-04-23T10:32:04.559 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "throttled" 2024-04-23T10:32:04.559 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-23T10:32:04.559 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-04-23T10:32:04.559 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-04-23T10:31:21.445156+0000", 2024-04-23T10:32:04.559 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "all_read" 2024-04-23T10:32:04.559 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-23T10:32:04.559 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-04-23T10:32:04.559 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-04-23T10:31:21.445156+0000", 2024-04-23T10:32:04.559 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "dispatched" 2024-04-23T10:32:04.559 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-23T10:32:04.560 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-04-23T10:32:04.560 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-04-23T10:31:21.445579+0000", 2024-04-23T10:32:04.560 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "failed to xlock, waiting" 2024-04-23T10:32:04.560 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-23T10:32:04.560 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-04-23T10:32:04.560 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-04-23T10:31:21.446412+0000", 2024-04-23T10:32:04.560 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "acquired locks" 2024-04-23T10:32:04.560 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-23T10:32:04.560 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-04-23T10:32:04.560 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-04-23T10:31:21.510071+0000", 2024-04-23T10:32:04.560 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "quiesce complete" 2024-04-23T10:32:04.560 INFO:teuthology.orchestra.run.smithi114.stdout: } 2024-04-23T10:32:04.560 INFO:teuthology.orchestra.run.smithi114.stdout: ], 2024-04-23T10:32:04.560 INFO:teuthology.orchestra.run.smithi114.stdout: "locks": [ 2024-04-23T10:32:04.560 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-04-23T10:32:04.561 INFO:teuthology.orchestra.run.smithi114.stdout: "object": { 2024-04-23T10:32:04.561 INFO:teuthology.orchestra.run.smithi114.stdout: "is_auth": true, 2024-04-23T10:32:04.561 INFO:teuthology.orchestra.run.smithi114.stdout: "auth_state": { 2024-04-23T10:32:04.561 INFO:teuthology.orchestra.run.smithi114.stdout: "replicas": { 2024-04-23T10:32:04.561 INFO:teuthology.orchestra.run.smithi114.stdout: "1": 1, 2024-04-23T10:32:04.561 INFO:teuthology.orchestra.run.smithi114.stdout: "2": 1 2024-04-23T10:32:04.561 INFO:teuthology.orchestra.run.smithi114.stdout: } 2024-04-23T10:32:04.561 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-23T10:32:04.561 INFO:teuthology.orchestra.run.smithi114.stdout: "replica_state": { 2024-04-23T10:32:04.561 INFO:teuthology.orchestra.run.smithi114.stdout: "authority": [ 2024-04-23T10:32:04.561 INFO:teuthology.orchestra.run.smithi114.stdout: 0, 2024-04-23T10:32:04.561 INFO:teuthology.orchestra.run.smithi114.stdout: -2 2024-04-23T10:32:04.561 INFO:teuthology.orchestra.run.smithi114.stdout: ], 2024-04-23T10:32:04.561 INFO:teuthology.orchestra.run.smithi114.stdout: "replica_nonce": 0 2024-04-23T10:32:04.562 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-23T10:32:04.562 INFO:teuthology.orchestra.run.smithi114.stdout: "auth_pins": 3, 2024-04-23T10:32:04.562 INFO:teuthology.orchestra.run.smithi114.stdout: "is_frozen": false, 2024-04-23T10:32:04.562 INFO:teuthology.orchestra.run.smithi114.stdout: "is_freezing": false, 2024-04-23T10:32:04.562 INFO:teuthology.orchestra.run.smithi114.stdout: "pins": { 2024-04-23T10:32:04.562 INFO:teuthology.orchestra.run.smithi114.stdout: "dirtyscattered": 1, 2024-04-23T10:32:04.562 INFO:teuthology.orchestra.run.smithi114.stdout: "request": 1, 2024-04-23T10:32:04.562 INFO:teuthology.orchestra.run.smithi114.stdout: "lock": 1, 2024-04-23T10:32:04.562 INFO:teuthology.orchestra.run.smithi114.stdout: "dirfrag": 1, 2024-04-23T10:32:04.562 INFO:teuthology.orchestra.run.smithi114.stdout: "caps": 1, 2024-04-23T10:32:04.562 INFO:teuthology.orchestra.run.smithi114.stdout: "dirtyparent": 0, 2024-04-23T10:32:04.562 INFO:teuthology.orchestra.run.smithi114.stdout: "replicated": 1, 2024-04-23T10:32:04.562 INFO:teuthology.orchestra.run.smithi114.stdout: "dirty": 1, 2024-04-23T10:32:04.562 INFO:teuthology.orchestra.run.smithi114.stdout: "waiter": 0, 2024-04-23T10:32:04.562 INFO:teuthology.orchestra.run.smithi114.stdout: "authpin": 1 2024-04-23T10:32:04.563 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-23T10:32:04.563 INFO:teuthology.orchestra.run.smithi114.stdout: "nref": 8 2024-04-23T10:32:04.563 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-23T10:32:04.563 INFO:teuthology.orchestra.run.smithi114.stdout: "object_string": "[inode 0x1 [...7,head] / auth{1=1,2=1} v187 ap=3 snaprealm=0x5571ce2bf8c0 f(v0 m2024-04-23T10:24:48.676117+0000 1=0+1) n(v27 rc2024-04-23T10:32:00.511998+0000 b21075308 rs6 88=67+21)/n(v0 rc2024-04-23T10:24:44.319412+0000 1=0+1) old_inodes=5 (inest mix dirty) (iquiesce lock x=1 by request(mds.0:150 nref=3)) caps={15312=pAsLsXs/-@2,15576=pAsLsXsFs/-@31} | dirtyscattered=1 request=1 lock=1 dirfrag=1 caps=1 dirtyparent=0 replicated=1 dirty=1 waiter=0 authpin=1 0x5571cef23b80]", 2024-04-23T10:32:04.563 INFO:teuthology.orchestra.run.smithi114.stdout: "lock": { 2024-04-23T10:32:04.563 INFO:teuthology.orchestra.run.smithi114.stdout: "gather_set": [], 2024-04-23T10:32:04.563 INFO:teuthology.orchestra.run.smithi114.stdout: "state": "lock", 2024-04-23T10:32:04.563 INFO:teuthology.orchestra.run.smithi114.stdout: "type": "iquiesce", 2024-04-23T10:32:04.563 INFO:teuthology.orchestra.run.smithi114.stdout: "is_leased": false, 2024-04-23T10:32:04.563 INFO:teuthology.orchestra.run.smithi114.stdout: "num_rdlocks": 0, 2024-04-23T10:32:04.563 INFO:teuthology.orchestra.run.smithi114.stdout: "num_wrlocks": 0, 2024-04-23T10:32:04.563 INFO:teuthology.orchestra.run.smithi114.stdout: "num_xlocks": 1, 2024-04-23T10:32:04.563 INFO:teuthology.orchestra.run.smithi114.stdout: "xlock_by": { 2024-04-23T10:32:04.563 INFO:teuthology.orchestra.run.smithi114.stdout: "reqid": { 2024-04-23T10:32:04.564 INFO:teuthology.orchestra.run.smithi114.stdout: "entity": { 2024-04-23T10:32:04.564 INFO:teuthology.orchestra.run.smithi114.stdout: "type": "mds", 2024-04-23T10:32:04.564 INFO:teuthology.orchestra.run.smithi114.stdout: "num": 0 2024-04-23T10:32:04.564 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-23T10:32:04.564 INFO:teuthology.orchestra.run.smithi114.stdout: "tid": 150 2024-04-23T10:32:04.564 INFO:teuthology.orchestra.run.smithi114.stdout: } 2024-04-23T10:32:04.564 INFO:teuthology.orchestra.run.smithi114.stdout: } 2024-04-23T10:32:04.564 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-23T10:32:04.564 INFO:teuthology.orchestra.run.smithi114.stdout: "flags": 4, 2024-04-23T10:32:04.564 INFO:teuthology.orchestra.run.smithi114.stdout: "wrlock_target": -1 2024-04-23T10:32:04.564 INFO:teuthology.orchestra.run.smithi114.stdout: } 2024-04-23T10:32:04.564 INFO:teuthology.orchestra.run.smithi114.stdout: ] 2024-04-23T10:32:04.564 INFO:teuthology.orchestra.run.smithi114.stdout: } 2024-04-23T10:32:04.564 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-23T10:32:04.564 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-04-23T10:32:04.565 INFO:teuthology.orchestra.run.smithi114.stdout: "description": "internal op quiesce_inode:mds.0:163 fp=#0x10000000207", 2024-04-23T10:32:04.565 INFO:teuthology.orchestra.run.smithi114.stdout: "initiated_at": "2024-04-23T10:31:21.457592+0000", 2024-04-23T10:32:04.565 INFO:teuthology.orchestra.run.smithi114.stdout: "age": 42.426797319999999, 2024-04-23T10:32:04.565 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 42.427741163999997, 2024-04-23T10:32:04.565 INFO:teuthology.orchestra.run.smithi114.stdout: "continuous": true, 2024-04-23T10:32:04.565 INFO:teuthology.orchestra.run.smithi114.stdout: "type_data": { 2024-04-23T10:32:04.565 INFO:teuthology.orchestra.run.smithi114.stdout: "result": -2147483648, 2024-04-23T10:32:04.565 INFO:teuthology.orchestra.run.smithi114.stdout: "flag_point": "quiesce complete", 2024-04-23T10:32:04.565 INFO:teuthology.orchestra.run.smithi114.stdout: "reqid": { 2024-04-23T10:32:04.565 INFO:teuthology.orchestra.run.smithi114.stdout: "entity": { 2024-04-23T10:32:04.565 INFO:teuthology.orchestra.run.smithi114.stdout: "type": "mds", 2024-04-23T10:32:04.565 INFO:teuthology.orchestra.run.smithi114.stdout: "num": 0 2024-04-23T10:32:04.565 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-23T10:32:04.565 INFO:teuthology.orchestra.run.smithi114.stdout: "tid": 163 2024-04-23T10:32:04.565 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-23T10:32:04.566 INFO:teuthology.orchestra.run.smithi114.stdout: "op_type": "internal_op", 2024-04-23T10:32:04.566 INFO:teuthology.orchestra.run.smithi114.stdout: "internal_op": 5385, 2024-04-23T10:32:04.566 INFO:teuthology.orchestra.run.smithi114.stdout: "op_name": "quiesce_inode", 2024-04-23T10:32:04.566 INFO:teuthology.orchestra.run.smithi114.stdout: "events": [ 2024-04-23T10:32:04.566 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-04-23T10:32:04.566 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-04-23T10:31:21.457592+0000", 2024-04-23T10:32:04.566 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "initiated" 2024-04-23T10:32:04.566 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-23T10:32:04.566 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-04-23T10:32:04.566 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-04-23T10:31:21.457592+0000", 2024-04-23T10:32:04.566 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "header_read" 2024-04-23T10:32:04.566 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-23T10:32:04.566 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-04-23T10:32:04.566 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-04-23T10:31:21.457592+0000", 2024-04-23T10:32:04.567 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "throttled" 2024-04-23T10:32:04.567 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-23T10:32:04.567 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-04-23T10:32:04.567 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-04-23T10:31:21.457592+0000", 2024-04-23T10:32:04.567 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "all_read" 2024-04-23T10:32:04.567 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-23T10:32:04.567 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-04-23T10:32:04.567 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-04-23T10:31:21.457592+0000", 2024-04-23T10:32:04.567 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "dispatched" 2024-04-23T10:32:04.567 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-23T10:32:04.567 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-04-23T10:32:04.567 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-04-23T10:31:21.458283+0000", 2024-04-23T10:32:04.567 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "acquired locks" 2024-04-23T10:32:04.567 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-23T10:32:04.567 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-04-23T10:32:04.568 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-04-23T10:31:21.458355+0000", 2024-04-23T10:32:04.568 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "quiesce complete" 2024-04-23T10:32:04.568 INFO:teuthology.orchestra.run.smithi114.stdout: } 2024-04-23T10:32:04.568 INFO:teuthology.orchestra.run.smithi114.stdout: ], 2024-04-23T10:32:04.568 INFO:teuthology.orchestra.run.smithi114.stdout: "locks": [ 2024-04-23T10:32:04.568 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-04-23T10:32:04.568 INFO:teuthology.orchestra.run.smithi114.stdout: "object": { 2024-04-23T10:32:04.568 INFO:teuthology.orchestra.run.smithi114.stdout: "is_auth": true, 2024-04-23T10:32:04.568 INFO:teuthology.orchestra.run.smithi114.stdout: "auth_state": { 2024-04-23T10:32:04.568 INFO:teuthology.orchestra.run.smithi114.stdout: "replicas": {} 2024-04-23T10:32:04.568 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-23T10:32:04.568 INFO:teuthology.orchestra.run.smithi114.stdout: "replica_state": { 2024-04-23T10:32:04.568 INFO:teuthology.orchestra.run.smithi114.stdout: "authority": [ 2024-04-23T10:32:04.568 INFO:teuthology.orchestra.run.smithi114.stdout: 0, 2024-04-23T10:32:04.569 INFO:teuthology.orchestra.run.smithi114.stdout: -2 2024-04-23T10:32:04.569 INFO:teuthology.orchestra.run.smithi114.stdout: ], 2024-04-23T10:32:04.569 INFO:teuthology.orchestra.run.smithi114.stdout: "replica_nonce": 0 2024-04-23T10:32:04.569 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-23T10:32:04.569 INFO:teuthology.orchestra.run.smithi114.stdout: "auth_pins": 1, 2024-04-23T10:32:04.569 INFO:teuthology.orchestra.run.smithi114.stdout: "is_frozen": false, 2024-04-23T10:32:04.569 INFO:teuthology.orchestra.run.smithi114.stdout: "is_freezing": false, 2024-04-23T10:32:04.569 INFO:teuthology.orchestra.run.smithi114.stdout: "pins": { 2024-04-23T10:32:04.569 INFO:teuthology.orchestra.run.smithi114.stdout: "ptrwaiter": 0, 2024-04-23T10:32:04.569 INFO:teuthology.orchestra.run.smithi114.stdout: "request": 1, 2024-04-23T10:32:04.569 INFO:teuthology.orchestra.run.smithi114.stdout: "lock": 1, 2024-04-23T10:32:04.569 INFO:teuthology.orchestra.run.smithi114.stdout: "caps": 0, 2024-04-23T10:32:04.569 INFO:teuthology.orchestra.run.smithi114.stdout: "dirtyparent": 0, 2024-04-23T10:32:04.569 INFO:teuthology.orchestra.run.smithi114.stdout: "dirty": 0, 2024-04-23T10:32:04.569 INFO:teuthology.orchestra.run.smithi114.stdout: "waiter": 0, 2024-04-23T10:32:04.570 INFO:teuthology.orchestra.run.smithi114.stdout: "authpin": 1 2024-04-23T10:32:04.570 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-23T10:32:04.570 INFO:teuthology.orchestra.run.smithi114.stdout: "nref": 3 2024-04-23T10:32:04.570 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-23T10:32:04.570 INFO:teuthology.orchestra.run.smithi114.stdout: "object_string": "[inode 0x10000000207 [4,head] /volumes/qa/sv_1/b206794e-7622-4ab8-8943-f3ae4e65b7a3/client.0/tmp/clients/client0/filler.004 auth v487 ap=1 s=1048576 n(v0 rc2024-04-23T10:27:51.952398+0000 b1048576 1=1+0) (iquiesce lock x=1 by request(mds.0:163 nref=3)) | ptrwaiter=0 request=1 lock=1 caps=0 dirtyparent=0 dirty=0 waiter=0 authpin=1 0x5571cf342c00]", 2024-04-23T10:32:04.570 INFO:teuthology.orchestra.run.smithi114.stdout: "lock": { 2024-04-23T10:32:04.570 INFO:teuthology.orchestra.run.smithi114.stdout: "gather_set": [], 2024-04-23T10:32:04.570 INFO:teuthology.orchestra.run.smithi114.stdout: "state": "lock", 2024-04-23T10:32:04.570 INFO:teuthology.orchestra.run.smithi114.stdout: "type": "iquiesce", 2024-04-23T10:32:04.570 INFO:teuthology.orchestra.run.smithi114.stdout: "is_leased": false, 2024-04-23T10:32:04.570 INFO:teuthology.orchestra.run.smithi114.stdout: "num_rdlocks": 0, 2024-04-23T10:32:04.570 INFO:teuthology.orchestra.run.smithi114.stdout: "num_wrlocks": 0, 2024-04-23T10:32:04.570 INFO:teuthology.orchestra.run.smithi114.stdout: "num_xlocks": 1, 2024-04-23T10:32:04.570 INFO:teuthology.orchestra.run.smithi114.stdout: "xlock_by": { 2024-04-23T10:32:04.570 INFO:teuthology.orchestra.run.smithi114.stdout: "reqid": { 2024-04-23T10:32:04.571 INFO:teuthology.orchestra.run.smithi114.stdout: "entity": { 2024-04-23T10:32:04.571 INFO:teuthology.orchestra.run.smithi114.stdout: "type": "mds", 2024-04-23T10:32:04.571 INFO:teuthology.orchestra.run.smithi114.stdout: "num": 0 2024-04-23T10:32:04.571 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-23T10:32:04.571 INFO:teuthology.orchestra.run.smithi114.stdout: "tid": 163 2024-04-23T10:32:04.571 INFO:teuthology.orchestra.run.smithi114.stdout: } 2024-04-23T10:32:04.571 INFO:teuthology.orchestra.run.smithi114.stdout: } 2024-04-23T10:32:04.571 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-23T10:32:04.571 INFO:teuthology.orchestra.run.smithi114.stdout: "flags": 4, 2024-04-23T10:32:04.571 INFO:teuthology.orchestra.run.smithi114.stdout: "wrlock_target": -1 2024-04-23T10:32:04.571 INFO:teuthology.orchestra.run.smithi114.stdout: } 2024-04-23T10:32:04.571 INFO:teuthology.orchestra.run.smithi114.stdout: ] 2024-04-23T10:32:04.571 INFO:teuthology.orchestra.run.smithi114.stdout: } 2024-04-23T10:32:04.571 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-23T10:32:04.572 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-04-23T10:32:04.572 INFO:teuthology.orchestra.run.smithi114.stdout: "description": "peer_request:mds.1:114", 2024-04-23T10:32:04.572 INFO:teuthology.orchestra.run.smithi114.stdout: "initiated_at": "2024-04-23T10:31:21.464702+0000", 2024-04-23T10:32:04.572 INFO:teuthology.orchestra.run.smithi114.stdout: "age": 42.419687234000001, 2024-04-23T10:32:04.572 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 42.420722597000001, 2024-04-23T10:32:04.572 INFO:teuthology.orchestra.run.smithi114.stdout: "continuous": false, 2024-04-23T10:32:04.572 INFO:teuthology.orchestra.run.smithi114.stdout: "type_data": { 2024-04-23T10:32:04.572 INFO:teuthology.orchestra.run.smithi114.stdout: "result": -2147483648, 2024-04-23T10:32:04.572 INFO:teuthology.orchestra.run.smithi114.stdout: "flag_point": "dispatched", 2024-04-23T10:32:04.572 INFO:teuthology.orchestra.run.smithi114.stdout: "reqid": { 2024-04-23T10:32:04.572 INFO:teuthology.orchestra.run.smithi114.stdout: "entity": { 2024-04-23T10:32:04.572 INFO:teuthology.orchestra.run.smithi114.stdout: "type": "mds", 2024-04-23T10:32:04.572 INFO:teuthology.orchestra.run.smithi114.stdout: "num": 1 2024-04-23T10:32:04.572 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-23T10:32:04.572 INFO:teuthology.orchestra.run.smithi114.stdout: "tid": 114 2024-04-23T10:32:04.573 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-23T10:32:04.573 INFO:teuthology.orchestra.run.smithi114.stdout: "op_type": "peer_request", 2024-04-23T10:32:04.573 INFO:teuthology.orchestra.run.smithi114.stdout: "leader_info": { 2024-04-23T10:32:04.573 INFO:teuthology.orchestra.run.smithi114.stdout: "leader": "1" 2024-04-23T10:32:04.573 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-23T10:32:04.573 INFO:teuthology.orchestra.run.smithi114.stdout: "events": [ 2024-04-23T10:32:04.573 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-04-23T10:32:04.573 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-04-23T10:31:21.464702+0000", 2024-04-23T10:32:04.573 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "initiated" 2024-04-23T10:32:04.573 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-23T10:32:04.573 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-04-23T10:32:04.573 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-04-23T10:31:21.464702+0000", 2024-04-23T10:32:04.573 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "header_read" 2024-04-23T10:32:04.573 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-23T10:32:04.573 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-04-23T10:32:04.574 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-04-23T10:31:21.464702+0000", 2024-04-23T10:32:04.574 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "throttled" 2024-04-23T10:32:04.574 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-23T10:32:04.574 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-04-23T10:32:04.574 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-04-23T10:31:21.464708+0000", 2024-04-23T10:32:04.574 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "all_read" 2024-04-23T10:32:04.574 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-23T10:32:04.574 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-04-23T10:32:04.574 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-04-23T10:31:21.479803+0000", 2024-04-23T10:32:04.574 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "dispatched" 2024-04-23T10:32:04.574 INFO:teuthology.orchestra.run.smithi114.stdout: } 2024-04-23T10:32:04.574 INFO:teuthology.orchestra.run.smithi114.stdout: ], 2024-04-23T10:32:04.574 INFO:teuthology.orchestra.run.smithi114.stdout: "locks": [] 2024-04-23T10:32:04.574 INFO:teuthology.orchestra.run.smithi114.stdout: } 2024-04-23T10:32:04.575 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-23T10:32:04.575 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-04-23T10:32:04.575 INFO:teuthology.orchestra.run.smithi114.stdout: "description": "peer_request:mds.1:102", 2024-04-23T10:32:04.575 INFO:teuthology.orchestra.run.smithi114.stdout: "initiated_at": "2024-04-23T10:31:21.445466+0000", 2024-04-23T10:32:04.575 INFO:teuthology.orchestra.run.smithi114.stdout: "age": 42.438923181, 2024-04-23T10:32:04.575 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 42.439989287000003, 2024-04-23T10:32:04.575 INFO:teuthology.orchestra.run.smithi114.stdout: "continuous": false, 2024-04-23T10:32:04.575 INFO:teuthology.orchestra.run.smithi114.stdout: "type_data": { 2024-04-23T10:32:04.575 INFO:teuthology.orchestra.run.smithi114.stdout: "result": -2147483648, 2024-04-23T10:32:04.575 INFO:teuthology.orchestra.run.smithi114.stdout: "flag_point": "dispatched", 2024-04-23T10:32:04.575 INFO:teuthology.orchestra.run.smithi114.stdout: "reqid": { 2024-04-23T10:32:04.575 INFO:teuthology.orchestra.run.smithi114.stdout: "entity": { 2024-04-23T10:32:04.575 INFO:teuthology.orchestra.run.smithi114.stdout: "type": "mds", 2024-04-23T10:32:04.575 INFO:teuthology.orchestra.run.smithi114.stdout: "num": 1 2024-04-23T10:32:04.575 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-23T10:32:04.576 INFO:teuthology.orchestra.run.smithi114.stdout: "tid": 102 2024-04-23T10:32:04.576 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-23T10:32:04.576 INFO:teuthology.orchestra.run.smithi114.stdout: "op_type": "peer_request", 2024-04-23T10:32:04.576 INFO:teuthology.orchestra.run.smithi114.stdout: "leader_info": { 2024-04-23T10:32:04.576 INFO:teuthology.orchestra.run.smithi114.stdout: "leader": "1" 2024-04-23T10:32:04.576 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-23T10:32:04.576 INFO:teuthology.orchestra.run.smithi114.stdout: "events": [ 2024-04-23T10:32:04.576 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-04-23T10:32:04.576 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-04-23T10:31:21.445466+0000", 2024-04-23T10:32:04.576 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "initiated" 2024-04-23T10:32:04.576 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-23T10:32:04.576 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-04-23T10:32:04.576 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-04-23T10:31:21.445466+0000", 2024-04-23T10:32:04.576 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "header_read" 2024-04-23T10:32:04.576 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-23T10:32:04.577 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-04-23T10:32:04.577 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-04-23T10:31:21.445467+0000", 2024-04-23T10:32:04.577 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "throttled" 2024-04-23T10:32:04.577 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-23T10:32:04.577 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-04-23T10:32:04.577 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-04-23T10:31:21.445474+0000", 2024-04-23T10:32:04.577 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "all_read" 2024-04-23T10:32:04.577 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-23T10:32:04.577 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-04-23T10:32:04.577 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-04-23T10:31:21.445755+0000", 2024-04-23T10:32:04.577 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "dispatched" 2024-04-23T10:32:04.577 INFO:teuthology.orchestra.run.smithi114.stdout: } 2024-04-23T10:32:04.577 INFO:teuthology.orchestra.run.smithi114.stdout: ], 2024-04-23T10:32:04.577 INFO:teuthology.orchestra.run.smithi114.stdout: "locks": [] 2024-04-23T10:32:04.577 INFO:teuthology.orchestra.run.smithi114.stdout: } 2024-04-23T10:32:04.578 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-23T10:32:04.578 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-04-23T10:32:04.578 INFO:teuthology.orchestra.run.smithi114.stdout: "description": "internal op quiesce_inode:mds.0:164 fp=#0x10000000206", 2024-04-23T10:32:04.578 INFO:teuthology.orchestra.run.smithi114.stdout: "initiated_at": "2024-04-23T10:31:21.457599+0000", 2024-04-23T10:32:04.578 INFO:teuthology.orchestra.run.smithi114.stdout: "age": 42.426790124999997, 2024-04-23T10:32:04.578 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 42.42787938, 2024-04-23T10:32:04.578 INFO:teuthology.orchestra.run.smithi114.stdout: "continuous": true, 2024-04-23T10:32:04.578 INFO:teuthology.orchestra.run.smithi114.stdout: "type_data": { 2024-04-23T10:32:04.578 INFO:teuthology.orchestra.run.smithi114.stdout: "result": -2147483648, 2024-04-23T10:32:04.578 INFO:teuthology.orchestra.run.smithi114.stdout: "flag_point": "quiesce complete", 2024-04-23T10:32:04.578 INFO:teuthology.orchestra.run.smithi114.stdout: "reqid": { 2024-04-23T10:32:04.578 INFO:teuthology.orchestra.run.smithi114.stdout: "entity": { 2024-04-23T10:32:04.578 INFO:teuthology.orchestra.run.smithi114.stdout: "type": "mds", 2024-04-23T10:32:04.578 INFO:teuthology.orchestra.run.smithi114.stdout: "num": 0 2024-04-23T10:32:04.578 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-23T10:32:04.579 INFO:teuthology.orchestra.run.smithi114.stdout: "tid": 164 2024-04-23T10:32:04.579 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-23T10:32:04.579 INFO:teuthology.orchestra.run.smithi114.stdout: "op_type": "internal_op", 2024-04-23T10:32:04.579 INFO:teuthology.orchestra.run.smithi114.stdout: "internal_op": 5385, 2024-04-23T10:32:04.579 INFO:teuthology.orchestra.run.smithi114.stdout: "op_name": "quiesce_inode", 2024-04-23T10:32:04.579 INFO:teuthology.orchestra.run.smithi114.stdout: "events": [ 2024-04-23T10:32:04.579 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-04-23T10:32:04.579 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-04-23T10:31:21.457599+0000", 2024-04-23T10:32:04.579 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "initiated" 2024-04-23T10:32:04.579 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-23T10:32:04.579 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-04-23T10:32:04.579 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-04-23T10:31:21.457599+0000", 2024-04-23T10:32:04.579 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "header_read" 2024-04-23T10:32:04.579 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-23T10:32:04.579 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-04-23T10:32:04.580 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-04-23T10:31:21.457599+0000", 2024-04-23T10:32:04.580 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "throttled" 2024-04-23T10:32:04.580 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-23T10:32:04.580 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-04-23T10:32:04.580 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-04-23T10:31:21.457599+0000", 2024-04-23T10:32:04.580 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "all_read" 2024-04-23T10:32:04.580 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-23T10:32:04.580 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-04-23T10:32:04.580 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-04-23T10:31:21.457599+0000", 2024-04-23T10:32:04.580 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "dispatched" 2024-04-23T10:32:04.580 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-23T10:32:04.580 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-04-23T10:32:04.580 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-04-23T10:31:21.458477+0000", 2024-04-23T10:32:04.580 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "acquired locks" 2024-04-23T10:32:04.580 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-23T10:32:04.581 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-04-23T10:32:04.581 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-04-23T10:31:21.458559+0000", 2024-04-23T10:32:04.581 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "quiesce complete" 2024-04-23T10:32:04.581 INFO:teuthology.orchestra.run.smithi114.stdout: } 2024-04-23T10:32:04.581 INFO:teuthology.orchestra.run.smithi114.stdout: ], 2024-04-23T10:32:04.581 INFO:teuthology.orchestra.run.smithi114.stdout: "locks": [ 2024-04-23T10:32:04.581 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-04-23T10:32:04.581 INFO:teuthology.orchestra.run.smithi114.stdout: "object": { 2024-04-23T10:32:04.581 INFO:teuthology.orchestra.run.smithi114.stdout: "is_auth": true, 2024-04-23T10:32:04.581 INFO:teuthology.orchestra.run.smithi114.stdout: "auth_state": { 2024-04-23T10:32:04.581 INFO:teuthology.orchestra.run.smithi114.stdout: "replicas": {} 2024-04-23T10:32:04.581 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-23T10:32:04.581 INFO:teuthology.orchestra.run.smithi114.stdout: "replica_state": { 2024-04-23T10:32:04.581 INFO:teuthology.orchestra.run.smithi114.stdout: "authority": [ 2024-04-23T10:32:04.581 INFO:teuthology.orchestra.run.smithi114.stdout: 0, 2024-04-23T10:32:04.582 INFO:teuthology.orchestra.run.smithi114.stdout: -2 2024-04-23T10:32:04.582 INFO:teuthology.orchestra.run.smithi114.stdout: ], 2024-04-23T10:32:04.582 INFO:teuthology.orchestra.run.smithi114.stdout: "replica_nonce": 0 2024-04-23T10:32:04.582 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-23T10:32:04.582 INFO:teuthology.orchestra.run.smithi114.stdout: "auth_pins": 1, 2024-04-23T10:32:04.582 INFO:teuthology.orchestra.run.smithi114.stdout: "is_frozen": false, 2024-04-23T10:32:04.582 INFO:teuthology.orchestra.run.smithi114.stdout: "is_freezing": false, 2024-04-23T10:32:04.582 INFO:teuthology.orchestra.run.smithi114.stdout: "pins": { 2024-04-23T10:32:04.582 INFO:teuthology.orchestra.run.smithi114.stdout: "ptrwaiter": 0, 2024-04-23T10:32:04.582 INFO:teuthology.orchestra.run.smithi114.stdout: "request": 1, 2024-04-23T10:32:04.582 INFO:teuthology.orchestra.run.smithi114.stdout: "lock": 1, 2024-04-23T10:32:04.582 INFO:teuthology.orchestra.run.smithi114.stdout: "caps": 0, 2024-04-23T10:32:04.582 INFO:teuthology.orchestra.run.smithi114.stdout: "dirtyparent": 0, 2024-04-23T10:32:04.582 INFO:teuthology.orchestra.run.smithi114.stdout: "dirty": 0, 2024-04-23T10:32:04.582 INFO:teuthology.orchestra.run.smithi114.stdout: "authpin": 1 2024-04-23T10:32:04.583 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-23T10:32:04.583 INFO:teuthology.orchestra.run.smithi114.stdout: "nref": 3 2024-04-23T10:32:04.583 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-23T10:32:04.583 INFO:teuthology.orchestra.run.smithi114.stdout: "object_string": "[inode 0x10000000206 [4,head] /volumes/qa/sv_1/b206794e-7622-4ab8-8943-f3ae4e65b7a3/client.0/tmp/clients/client0/filler.003 auth v469 ap=1 s=1048576 n(v0 rc2024-04-23T10:27:51.947398+0000 b1048576 1=1+0) (iquiesce lock x=1 by request(mds.0:164 nref=3)) | ptrwaiter=0 request=1 lock=1 caps=0 dirtyparent=0 dirty=0 authpin=1 0x5571cf342680]", 2024-04-23T10:32:04.583 INFO:teuthology.orchestra.run.smithi114.stdout: "lock": { 2024-04-23T10:32:04.583 INFO:teuthology.orchestra.run.smithi114.stdout: "gather_set": [], 2024-04-23T10:32:04.583 INFO:teuthology.orchestra.run.smithi114.stdout: "state": "lock", 2024-04-23T10:32:04.583 INFO:teuthology.orchestra.run.smithi114.stdout: "type": "iquiesce", 2024-04-23T10:32:04.583 INFO:teuthology.orchestra.run.smithi114.stdout: "is_leased": false, 2024-04-23T10:32:04.583 INFO:teuthology.orchestra.run.smithi114.stdout: "num_rdlocks": 0, 2024-04-23T10:32:04.583 INFO:teuthology.orchestra.run.smithi114.stdout: "num_wrlocks": 0, 2024-04-23T10:32:04.583 INFO:teuthology.orchestra.run.smithi114.stdout: "num_xlocks": 1, 2024-04-23T10:32:04.583 INFO:teuthology.orchestra.run.smithi114.stdout: "xlock_by": { 2024-04-23T10:32:04.583 INFO:teuthology.orchestra.run.smithi114.stdout: "reqid": { 2024-04-23T10:32:04.583 INFO:teuthology.orchestra.run.smithi114.stdout: "entity": { 2024-04-23T10:32:04.583 INFO:teuthology.orchestra.run.smithi114.stdout: "type": "mds", 2024-04-23T10:32:04.584 INFO:teuthology.orchestra.run.smithi114.stdout: "num": 0 2024-04-23T10:32:04.584 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-23T10:32:04.584 INFO:teuthology.orchestra.run.smithi114.stdout: "tid": 164 2024-04-23T10:32:04.584 INFO:teuthology.orchestra.run.smithi114.stdout: } 2024-04-23T10:32:04.584 INFO:teuthology.orchestra.run.smithi114.stdout: } 2024-04-23T10:32:04.584 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-23T10:32:04.584 INFO:teuthology.orchestra.run.smithi114.stdout: "flags": 4, 2024-04-23T10:32:04.584 INFO:teuthology.orchestra.run.smithi114.stdout: "wrlock_target": -1 2024-04-23T10:32:04.584 INFO:teuthology.orchestra.run.smithi114.stdout: } 2024-04-23T10:32:04.584 INFO:teuthology.orchestra.run.smithi114.stdout: ] 2024-04-23T10:32:04.584 INFO:teuthology.orchestra.run.smithi114.stdout: } 2024-04-23T10:32:04.584 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-23T10:32:04.584 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-04-23T10:32:04.584 INFO:teuthology.orchestra.run.smithi114.stdout: "description": "peer_request:mds.2:119", 2024-04-23T10:32:04.584 INFO:teuthology.orchestra.run.smithi114.stdout: "initiated_at": "2024-04-23T10:31:21.463515+0000", 2024-04-23T10:32:04.584 INFO:teuthology.orchestra.run.smithi114.stdout: "age": 42.420874140999999, 2024-04-23T10:32:04.585 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 42.422023795000001, 2024-04-23T10:32:04.585 INFO:teuthology.orchestra.run.smithi114.stdout: "continuous": false, 2024-04-23T10:32:04.585 INFO:teuthology.orchestra.run.smithi114.stdout: "type_data": { 2024-04-23T10:32:04.585 INFO:teuthology.orchestra.run.smithi114.stdout: "result": -2147483648, 2024-04-23T10:32:04.585 INFO:teuthology.orchestra.run.smithi114.stdout: "flag_point": "dispatched", 2024-04-23T10:32:04.585 INFO:teuthology.orchestra.run.smithi114.stdout: "reqid": { 2024-04-23T10:32:04.585 INFO:teuthology.orchestra.run.smithi114.stdout: "entity": { 2024-04-23T10:32:04.585 INFO:teuthology.orchestra.run.smithi114.stdout: "type": "mds", 2024-04-23T10:32:04.585 INFO:teuthology.orchestra.run.smithi114.stdout: "num": 2 2024-04-23T10:32:04.585 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-23T10:32:04.585 INFO:teuthology.orchestra.run.smithi114.stdout: "tid": 119 2024-04-23T10:32:04.585 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-23T10:32:04.585 INFO:teuthology.orchestra.run.smithi114.stdout: "op_type": "peer_request", 2024-04-23T10:32:04.585 INFO:teuthology.orchestra.run.smithi114.stdout: "leader_info": { 2024-04-23T10:32:04.585 INFO:teuthology.orchestra.run.smithi114.stdout: "leader": "2" 2024-04-23T10:32:04.586 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-23T10:32:04.586 INFO:teuthology.orchestra.run.smithi114.stdout: "events": [ 2024-04-23T10:32:04.586 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-04-23T10:32:04.586 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-04-23T10:31:21.463515+0000", 2024-04-23T10:32:04.586 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "initiated" 2024-04-23T10:32:04.586 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-23T10:32:04.586 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-04-23T10:32:04.586 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-04-23T10:31:21.463515+0000", 2024-04-23T10:32:04.586 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "header_read" 2024-04-23T10:32:04.586 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-23T10:32:04.586 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-04-23T10:32:04.586 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-04-23T10:31:21.463516+0000", 2024-04-23T10:32:04.586 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "throttled" 2024-04-23T10:32:04.586 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-23T10:32:04.587 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-04-23T10:32:04.587 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-04-23T10:31:21.463523+0000", 2024-04-23T10:32:04.587 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "all_read" 2024-04-23T10:32:04.587 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-23T10:32:04.587 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-04-23T10:32:04.587 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-04-23T10:31:21.480061+0000", 2024-04-23T10:32:04.587 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "dispatched" 2024-04-23T10:32:04.587 INFO:teuthology.orchestra.run.smithi114.stdout: } 2024-04-23T10:32:04.587 INFO:teuthology.orchestra.run.smithi114.stdout: ], 2024-04-23T10:32:04.587 INFO:teuthology.orchestra.run.smithi114.stdout: "locks": [] 2024-04-23T10:32:04.587 INFO:teuthology.orchestra.run.smithi114.stdout: } 2024-04-23T10:32:04.587 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-23T10:32:04.587 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-04-23T10:32:04.587 INFO:teuthology.orchestra.run.smithi114.stdout: "description": "peer_request:mds.2:107", 2024-04-23T10:32:04.588 INFO:teuthology.orchestra.run.smithi114.stdout: "initiated_at": "2024-04-23T10:31:21.445440+0000", 2024-04-23T10:32:04.588 INFO:teuthology.orchestra.run.smithi114.stdout: "age": 42.438949735999998, 2024-04-23T10:32:04.588 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 42.440122483000003, 2024-04-23T10:32:04.588 INFO:teuthology.orchestra.run.smithi114.stdout: "continuous": false, 2024-04-23T10:32:04.588 INFO:teuthology.orchestra.run.smithi114.stdout: "type_data": { 2024-04-23T10:32:04.588 INFO:teuthology.orchestra.run.smithi114.stdout: "result": -2147483648, 2024-04-23T10:32:04.588 INFO:teuthology.orchestra.run.smithi114.stdout: "flag_point": "dispatched", 2024-04-23T10:32:04.588 INFO:teuthology.orchestra.run.smithi114.stdout: "reqid": { 2024-04-23T10:32:04.588 INFO:teuthology.orchestra.run.smithi114.stdout: "entity": { 2024-04-23T10:32:04.588 INFO:teuthology.orchestra.run.smithi114.stdout: "type": "mds", 2024-04-23T10:32:04.588 INFO:teuthology.orchestra.run.smithi114.stdout: "num": 2 2024-04-23T10:32:04.588 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-23T10:32:04.588 INFO:teuthology.orchestra.run.smithi114.stdout: "tid": 107 2024-04-23T10:32:04.588 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-23T10:32:04.589 INFO:teuthology.orchestra.run.smithi114.stdout: "op_type": "peer_request", 2024-04-23T10:32:04.589 INFO:teuthology.orchestra.run.smithi114.stdout: "leader_info": { 2024-04-23T10:32:04.589 INFO:teuthology.orchestra.run.smithi114.stdout: "leader": "2" 2024-04-23T10:32:04.589 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-23T10:32:04.589 INFO:teuthology.orchestra.run.smithi114.stdout: "events": [ 2024-04-23T10:32:04.589 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-04-23T10:32:04.589 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-04-23T10:31:21.445440+0000", 2024-04-23T10:32:04.589 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "initiated" 2024-04-23T10:32:04.589 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-23T10:32:04.589 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-04-23T10:32:04.589 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-04-23T10:31:21.445440+0000", 2024-04-23T10:32:04.589 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "header_read" 2024-04-23T10:32:04.589 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-23T10:32:04.589 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-04-23T10:32:04.589 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-04-23T10:31:21.445440+0000", 2024-04-23T10:32:04.590 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "throttled" 2024-04-23T10:32:04.590 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-23T10:32:04.590 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-04-23T10:32:04.590 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-04-23T10:31:21.445459+0000", 2024-04-23T10:32:04.590 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "all_read" 2024-04-23T10:32:04.590 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-23T10:32:04.590 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-04-23T10:32:04.590 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-04-23T10:31:21.445874+0000", 2024-04-23T10:32:04.590 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "dispatched" 2024-04-23T10:32:04.590 INFO:teuthology.orchestra.run.smithi114.stdout: } 2024-04-23T10:32:04.590 INFO:teuthology.orchestra.run.smithi114.stdout: ], 2024-04-23T10:32:04.590 INFO:teuthology.orchestra.run.smithi114.stdout: "locks": [] 2024-04-23T10:32:04.590 INFO:teuthology.orchestra.run.smithi114.stdout: } 2024-04-23T10:32:04.590 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-23T10:32:04.590 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-04-23T10:32:04.591 INFO:teuthology.orchestra.run.smithi114.stdout: "description": "internal op quiesce_inode:mds.0:165 fp=#0x1000000000a", 2024-04-23T10:32:04.591 INFO:teuthology.orchestra.run.smithi114.stdout: "initiated_at": "2024-04-23T10:31:21.457606+0000", 2024-04-23T10:32:04.591 INFO:teuthology.orchestra.run.smithi114.stdout: "age": 42.426783227999998, 2024-04-23T10:32:04.591 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 42.427978015000001, 2024-04-23T10:32:04.591 INFO:teuthology.orchestra.run.smithi114.stdout: "continuous": true, 2024-04-23T10:32:04.591 INFO:teuthology.orchestra.run.smithi114.stdout: "type_data": { 2024-04-23T10:32:04.591 INFO:teuthology.orchestra.run.smithi114.stdout: "result": -2147483648, 2024-04-23T10:32:04.591 INFO:teuthology.orchestra.run.smithi114.stdout: "flag_point": "quiesce complete", 2024-04-23T10:32:04.591 INFO:teuthology.orchestra.run.smithi114.stdout: "reqid": { 2024-04-23T10:32:04.591 INFO:teuthology.orchestra.run.smithi114.stdout: "entity": { 2024-04-23T10:32:04.591 INFO:teuthology.orchestra.run.smithi114.stdout: "type": "mds", 2024-04-23T10:32:04.591 INFO:teuthology.orchestra.run.smithi114.stdout: "num": 0 2024-04-23T10:32:04.591 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-23T10:32:04.591 INFO:teuthology.orchestra.run.smithi114.stdout: "tid": 165 2024-04-23T10:32:04.592 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-23T10:32:04.592 INFO:teuthology.orchestra.run.smithi114.stdout: "op_type": "internal_op", 2024-04-23T10:32:04.592 INFO:teuthology.orchestra.run.smithi114.stdout: "internal_op": 5385, 2024-04-23T10:32:04.592 INFO:teuthology.orchestra.run.smithi114.stdout: "op_name": "quiesce_inode", 2024-04-23T10:32:04.592 INFO:teuthology.orchestra.run.smithi114.stdout: "events": [ 2024-04-23T10:32:04.592 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-04-23T10:32:04.592 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-04-23T10:31:21.457606+0000", 2024-04-23T10:32:04.592 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "initiated" 2024-04-23T10:32:04.592 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-23T10:32:04.592 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-04-23T10:32:04.592 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-04-23T10:31:21.457606+0000", 2024-04-23T10:32:04.592 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "header_read" 2024-04-23T10:32:04.592 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-23T10:32:04.592 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-04-23T10:32:04.592 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-04-23T10:31:21.457606+0000", 2024-04-23T10:32:04.593 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "throttled" 2024-04-23T10:32:04.593 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-23T10:32:04.593 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-04-23T10:32:04.593 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-04-23T10:31:21.457606+0000", 2024-04-23T10:32:04.593 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "all_read" 2024-04-23T10:32:04.593 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-23T10:32:04.593 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-04-23T10:32:04.593 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-04-23T10:31:21.457606+0000", 2024-04-23T10:32:04.593 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "dispatched" 2024-04-23T10:32:04.593 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-23T10:32:04.593 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-04-23T10:32:04.593 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-04-23T10:31:21.458673+0000", 2024-04-23T10:32:04.593 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "acquired locks" 2024-04-23T10:32:04.593 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-23T10:32:04.593 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-04-23T10:32:04.594 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-04-23T10:31:21.458757+0000", 2024-04-23T10:32:04.594 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "quiesce complete" 2024-04-23T10:32:04.594 INFO:teuthology.orchestra.run.smithi114.stdout: } 2024-04-23T10:32:04.594 INFO:teuthology.orchestra.run.smithi114.stdout: ], 2024-04-23T10:32:04.594 INFO:teuthology.orchestra.run.smithi114.stdout: "locks": [ 2024-04-23T10:32:04.594 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-04-23T10:32:04.594 INFO:teuthology.orchestra.run.smithi114.stdout: "object": { 2024-04-23T10:32:04.594 INFO:teuthology.orchestra.run.smithi114.stdout: "is_auth": true, 2024-04-23T10:32:04.594 INFO:teuthology.orchestra.run.smithi114.stdout: "auth_state": { 2024-04-23T10:32:04.594 INFO:teuthology.orchestra.run.smithi114.stdout: "replicas": {} 2024-04-23T10:32:04.594 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-23T10:32:04.594 INFO:teuthology.orchestra.run.smithi114.stdout: "replica_state": { 2024-04-23T10:32:04.594 INFO:teuthology.orchestra.run.smithi114.stdout: "authority": [ 2024-04-23T10:32:04.594 INFO:teuthology.orchestra.run.smithi114.stdout: 0, 2024-04-23T10:32:04.594 INFO:teuthology.orchestra.run.smithi114.stdout: -2 2024-04-23T10:32:04.595 INFO:teuthology.orchestra.run.smithi114.stdout: ], 2024-04-23T10:32:04.595 INFO:teuthology.orchestra.run.smithi114.stdout: "replica_nonce": 0 2024-04-23T10:32:04.595 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-23T10:32:04.595 INFO:teuthology.orchestra.run.smithi114.stdout: "auth_pins": 1, 2024-04-23T10:32:04.595 INFO:teuthology.orchestra.run.smithi114.stdout: "is_frozen": false, 2024-04-23T10:32:04.595 INFO:teuthology.orchestra.run.smithi114.stdout: "is_freezing": false, 2024-04-23T10:32:04.595 INFO:teuthology.orchestra.run.smithi114.stdout: "pins": { 2024-04-23T10:32:04.595 INFO:teuthology.orchestra.run.smithi114.stdout: "ptrwaiter": 0, 2024-04-23T10:32:04.595 INFO:teuthology.orchestra.run.smithi114.stdout: "request": 1, 2024-04-23T10:32:04.595 INFO:teuthology.orchestra.run.smithi114.stdout: "lock": 1, 2024-04-23T10:32:04.595 INFO:teuthology.orchestra.run.smithi114.stdout: "caps": 0, 2024-04-23T10:32:04.595 INFO:teuthology.orchestra.run.smithi114.stdout: "dirtyparent": 0, 2024-04-23T10:32:04.595 INFO:teuthology.orchestra.run.smithi114.stdout: "dirty": 0, 2024-04-23T10:32:04.595 INFO:teuthology.orchestra.run.smithi114.stdout: "authpin": 1 2024-04-23T10:32:04.595 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-23T10:32:04.596 INFO:teuthology.orchestra.run.smithi114.stdout: "nref": 3 2024-04-23T10:32:04.596 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-23T10:32:04.596 INFO:teuthology.orchestra.run.smithi114.stdout: "object_string": "[inode 0x1000000000a [4,head] /volumes/qa/sv_1/b206794e-7622-4ab8-8943-f3ae4e65b7a3/client.0/tmp/clients/client0/filler.000 auth v473 ap=1 s=1048576 n(v0 rc2024-04-23T10:27:51.931398+0000 b1048576 1=1+0) (iquiesce lock x=1 by request(mds.0:165 nref=3)) | ptrwaiter=0 request=1 lock=1 caps=0 dirtyparent=0 dirty=0 authpin=1 0x5571cf341600]", 2024-04-23T10:32:04.596 INFO:teuthology.orchestra.run.smithi114.stdout: "lock": { 2024-04-23T10:32:04.596 INFO:teuthology.orchestra.run.smithi114.stdout: "gather_set": [], 2024-04-23T10:32:04.596 INFO:teuthology.orchestra.run.smithi114.stdout: "state": "lock", 2024-04-23T10:32:04.596 INFO:teuthology.orchestra.run.smithi114.stdout: "type": "iquiesce", 2024-04-23T10:32:04.596 INFO:teuthology.orchestra.run.smithi114.stdout: "is_leased": false, 2024-04-23T10:32:04.596 INFO:teuthology.orchestra.run.smithi114.stdout: "num_rdlocks": 0, 2024-04-23T10:32:04.596 INFO:teuthology.orchestra.run.smithi114.stdout: "num_wrlocks": 0, 2024-04-23T10:32:04.596 INFO:teuthology.orchestra.run.smithi114.stdout: "num_xlocks": 1, 2024-04-23T10:32:04.596 INFO:teuthology.orchestra.run.smithi114.stdout: "xlock_by": { 2024-04-23T10:32:04.596 INFO:teuthology.orchestra.run.smithi114.stdout: "reqid": { 2024-04-23T10:32:04.596 INFO:teuthology.orchestra.run.smithi114.stdout: "entity": { 2024-04-23T10:32:04.596 INFO:teuthology.orchestra.run.smithi114.stdout: "type": "mds", 2024-04-23T10:32:04.597 INFO:teuthology.orchestra.run.smithi114.stdout: "num": 0 2024-04-23T10:32:04.597 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-23T10:32:04.597 INFO:teuthology.orchestra.run.smithi114.stdout: "tid": 165 2024-04-23T10:32:04.597 INFO:teuthology.orchestra.run.smithi114.stdout: } 2024-04-23T10:32:04.597 INFO:teuthology.orchestra.run.smithi114.stdout: } 2024-04-23T10:32:04.597 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-23T10:32:04.597 INFO:teuthology.orchestra.run.smithi114.stdout: "flags": 4, 2024-04-23T10:32:04.597 INFO:teuthology.orchestra.run.smithi114.stdout: "wrlock_target": -1 2024-04-23T10:32:04.597 INFO:teuthology.orchestra.run.smithi114.stdout: } 2024-04-23T10:32:04.597 INFO:teuthology.orchestra.run.smithi114.stdout: ] 2024-04-23T10:32:04.597 INFO:teuthology.orchestra.run.smithi114.stdout: } 2024-04-23T10:32:04.597 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-23T10:32:04.597 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-04-23T10:32:04.597 INFO:teuthology.orchestra.run.smithi114.stdout: "description": "internal op quiesce_inode:mds.0:190 fp=#0x1000000021f", 2024-04-23T10:32:04.597 INFO:teuthology.orchestra.run.smithi114.stdout: "initiated_at": "2024-04-23T10:31:21.483372+0000", 2024-04-23T10:32:04.598 INFO:teuthology.orchestra.run.smithi114.stdout: "age": 42.401016972999997, 2024-04-23T10:32:04.598 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 42.402270540000004, 2024-04-23T10:32:04.598 INFO:teuthology.orchestra.run.smithi114.stdout: "continuous": true, 2024-04-23T10:32:04.598 INFO:teuthology.orchestra.run.smithi114.stdout: "type_data": { 2024-04-23T10:32:04.598 INFO:teuthology.orchestra.run.smithi114.stdout: "result": -2147483648, 2024-04-23T10:32:04.598 INFO:teuthology.orchestra.run.smithi114.stdout: "flag_point": "quiesce complete for non-auth inode", 2024-04-23T10:32:04.598 INFO:teuthology.orchestra.run.smithi114.stdout: "reqid": { 2024-04-23T10:32:04.598 INFO:teuthology.orchestra.run.smithi114.stdout: "entity": { 2024-04-23T10:32:04.598 INFO:teuthology.orchestra.run.smithi114.stdout: "type": "mds", 2024-04-23T10:32:04.598 INFO:teuthology.orchestra.run.smithi114.stdout: "num": 0 2024-04-23T10:32:04.598 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-23T10:32:04.598 INFO:teuthology.orchestra.run.smithi114.stdout: "tid": 190 2024-04-23T10:32:04.598 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-23T10:32:04.598 INFO:teuthology.orchestra.run.smithi114.stdout: "op_type": "internal_op", 2024-04-23T10:32:04.599 INFO:teuthology.orchestra.run.smithi114.stdout: "internal_op": 5385, 2024-04-23T10:32:04.599 INFO:teuthology.orchestra.run.smithi114.stdout: "op_name": "quiesce_inode", 2024-04-23T10:32:04.599 INFO:teuthology.orchestra.run.smithi114.stdout: "events": [ 2024-04-23T10:32:04.599 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-04-23T10:32:04.599 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-04-23T10:31:21.483372+0000", 2024-04-23T10:32:04.599 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "initiated" 2024-04-23T10:32:04.599 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-23T10:32:04.599 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-04-23T10:32:04.599 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-04-23T10:31:21.483372+0000", 2024-04-23T10:32:04.599 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "header_read" 2024-04-23T10:32:04.599 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-23T10:32:04.599 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-04-23T10:32:04.599 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-04-23T10:31:21.483372+0000", 2024-04-23T10:32:04.599 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "throttled" 2024-04-23T10:32:04.599 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-23T10:32:04.599 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-04-23T10:32:04.600 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-04-23T10:31:21.483372+0000", 2024-04-23T10:32:04.600 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "all_read" 2024-04-23T10:32:04.600 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-23T10:32:04.600 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-04-23T10:32:04.600 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-04-23T10:31:21.483372+0000", 2024-04-23T10:32:04.600 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "dispatched" 2024-04-23T10:32:04.600 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-23T10:32:04.600 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-04-23T10:32:04.600 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-04-23T10:31:21.483581+0000", 2024-04-23T10:32:04.600 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "requesting remote authpins" 2024-04-23T10:32:04.600 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-23T10:32:04.600 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-04-23T10:32:04.600 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-04-23T10:31:21.506357+0000", 2024-04-23T10:32:04.601 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "acquired locks" 2024-04-23T10:32:04.601 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-23T10:32:04.601 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-04-23T10:32:04.601 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-04-23T10:31:21.506366+0000", 2024-04-23T10:32:04.601 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "quiesce complete for non-auth inode" 2024-04-23T10:32:04.601 INFO:teuthology.orchestra.run.smithi114.stdout: } 2024-04-23T10:32:04.601 INFO:teuthology.orchestra.run.smithi114.stdout: ], 2024-04-23T10:32:04.601 INFO:teuthology.orchestra.run.smithi114.stdout: "locks": [ 2024-04-23T10:32:04.601 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-04-23T10:32:04.601 INFO:teuthology.orchestra.run.smithi114.stdout: "object": { 2024-04-23T10:32:04.601 INFO:teuthology.orchestra.run.smithi114.stdout: "is_auth": false, 2024-04-23T10:32:04.601 INFO:teuthology.orchestra.run.smithi114.stdout: "auth_state": { 2024-04-23T10:32:04.601 INFO:teuthology.orchestra.run.smithi114.stdout: "replicas": {} 2024-04-23T10:32:04.601 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-23T10:32:04.602 INFO:teuthology.orchestra.run.smithi114.stdout: "replica_state": { 2024-04-23T10:32:04.602 INFO:teuthology.orchestra.run.smithi114.stdout: "authority": [ 2024-04-23T10:32:04.602 INFO:teuthology.orchestra.run.smithi114.stdout: 2, 2024-04-23T10:32:04.602 INFO:teuthology.orchestra.run.smithi114.stdout: -2 2024-04-23T10:32:04.602 INFO:teuthology.orchestra.run.smithi114.stdout: ], 2024-04-23T10:32:04.602 INFO:teuthology.orchestra.run.smithi114.stdout: "replica_nonce": 1 2024-04-23T10:32:04.602 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-23T10:32:04.602 INFO:teuthology.orchestra.run.smithi114.stdout: "auth_pins": 0, 2024-04-23T10:32:04.602 INFO:teuthology.orchestra.run.smithi114.stdout: "is_frozen": false, 2024-04-23T10:32:04.602 INFO:teuthology.orchestra.run.smithi114.stdout: "is_freezing": false, 2024-04-23T10:32:04.602 INFO:teuthology.orchestra.run.smithi114.stdout: "pins": { 2024-04-23T10:32:04.602 INFO:teuthology.orchestra.run.smithi114.stdout: "request": 1, 2024-04-23T10:32:04.602 INFO:teuthology.orchestra.run.smithi114.stdout: "lock": 1, 2024-04-23T10:32:04.602 INFO:teuthology.orchestra.run.smithi114.stdout: "caps": 0, 2024-04-23T10:32:04.603 INFO:teuthology.orchestra.run.smithi114.stdout: "exportingcaps": 0, 2024-04-23T10:32:04.603 INFO:teuthology.orchestra.run.smithi114.stdout: "dirtyparent": 0, 2024-04-23T10:32:04.603 INFO:teuthology.orchestra.run.smithi114.stdout: "dirty": 0, 2024-04-23T10:32:04.603 INFO:teuthology.orchestra.run.smithi114.stdout: "authpin": 0, 2024-04-23T10:32:04.603 INFO:teuthology.orchestra.run.smithi114.stdout: "tempexporting": 0 2024-04-23T10:32:04.603 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-23T10:32:04.603 INFO:teuthology.orchestra.run.smithi114.stdout: "nref": 2 2024-04-23T10:32:04.603 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-23T10:32:04.603 INFO:teuthology.orchestra.run.smithi114.stdout: "object_string": "[inode 0x1000000021f [7,head] /volumes/qa/sv_1/b206794e-7622-4ab8-8943-f3ae4e65b7a3/client.0/tmp/clients/client0/~dmtmp/COREL/ROSE.CPT rep@2.1 v7256 s=260096 n(v0 rc2024-04-23T10:27:52.049396+0000 b260096 1=1+0)/n(v0 1=1+0) (iquiesce lock x=1 by request(mds.0:190 nref=3)) | request=1 lock=1 caps=0 exportingcaps=0 dirtyparent=0 dirty=0 authpin=0 tempexporting=0 0x5571cf563180]", 2024-04-23T10:32:04.603 INFO:teuthology.orchestra.run.smithi114.stdout: "lock": { 2024-04-23T10:32:04.603 INFO:teuthology.orchestra.run.smithi114.stdout: "gather_set": [], 2024-04-23T10:32:04.603 INFO:teuthology.orchestra.run.smithi114.stdout: "state": "lock", 2024-04-23T10:32:04.603 INFO:teuthology.orchestra.run.smithi114.stdout: "type": "iquiesce", 2024-04-23T10:32:04.603 INFO:teuthology.orchestra.run.smithi114.stdout: "is_leased": false, 2024-04-23T10:32:04.604 INFO:teuthology.orchestra.run.smithi114.stdout: "num_rdlocks": 0, 2024-04-23T10:32:04.604 INFO:teuthology.orchestra.run.smithi114.stdout: "num_wrlocks": 0, 2024-04-23T10:32:04.604 INFO:teuthology.orchestra.run.smithi114.stdout: "num_xlocks": 1, 2024-04-23T10:32:04.604 INFO:teuthology.orchestra.run.smithi114.stdout: "xlock_by": { 2024-04-23T10:32:04.604 INFO:teuthology.orchestra.run.smithi114.stdout: "reqid": { 2024-04-23T10:32:04.604 INFO:teuthology.orchestra.run.smithi114.stdout: "entity": { 2024-04-23T10:32:04.604 INFO:teuthology.orchestra.run.smithi114.stdout: "type": "mds", 2024-04-23T10:32:04.604 INFO:teuthology.orchestra.run.smithi114.stdout: "num": 0 2024-04-23T10:32:04.604 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-23T10:32:04.604 INFO:teuthology.orchestra.run.smithi114.stdout: "tid": 190 2024-04-23T10:32:04.604 INFO:teuthology.orchestra.run.smithi114.stdout: } 2024-04-23T10:32:04.604 INFO:teuthology.orchestra.run.smithi114.stdout: } 2024-04-23T10:32:04.604 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-23T10:32:04.604 INFO:teuthology.orchestra.run.smithi114.stdout: "flags": 4, 2024-04-23T10:32:04.604 INFO:teuthology.orchestra.run.smithi114.stdout: "wrlock_target": -1 2024-04-23T10:32:04.605 INFO:teuthology.orchestra.run.smithi114.stdout: } 2024-04-23T10:32:04.605 INFO:teuthology.orchestra.run.smithi114.stdout: ] 2024-04-23T10:32:04.605 INFO:teuthology.orchestra.run.smithi114.stdout: } 2024-04-23T10:32:04.605 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-23T10:32:04.605 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-04-23T10:32:04.605 INFO:teuthology.orchestra.run.smithi114.stdout: "description": "internal op quiesce_inode:mds.0:151 fp=#0x10000000000", 2024-04-23T10:32:04.605 INFO:teuthology.orchestra.run.smithi114.stdout: "initiated_at": "2024-04-23T10:31:21.446733+0000", 2024-04-23T10:32:04.605 INFO:teuthology.orchestra.run.smithi114.stdout: "age": 42.437656287999999, 2024-04-23T10:32:04.605 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 42.438973836999999, 2024-04-23T10:32:04.605 INFO:teuthology.orchestra.run.smithi114.stdout: "continuous": true, 2024-04-23T10:32:04.605 INFO:teuthology.orchestra.run.smithi114.stdout: "type_data": { 2024-04-23T10:32:04.605 INFO:teuthology.orchestra.run.smithi114.stdout: "result": -2147483648, 2024-04-23T10:32:04.605 INFO:teuthology.orchestra.run.smithi114.stdout: "flag_point": "quiesce complete", 2024-04-23T10:32:04.605 INFO:teuthology.orchestra.run.smithi114.stdout: "reqid": { 2024-04-23T10:32:04.606 INFO:teuthology.orchestra.run.smithi114.stdout: "entity": { 2024-04-23T10:32:04.606 INFO:teuthology.orchestra.run.smithi114.stdout: "type": "mds", 2024-04-23T10:32:04.606 INFO:teuthology.orchestra.run.smithi114.stdout: "num": 0 2024-04-23T10:32:04.606 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-23T10:32:04.606 INFO:teuthology.orchestra.run.smithi114.stdout: "tid": 151 2024-04-23T10:32:04.606 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-23T10:32:04.606 INFO:teuthology.orchestra.run.smithi114.stdout: "op_type": "internal_op", 2024-04-23T10:32:04.606 INFO:teuthology.orchestra.run.smithi114.stdout: "internal_op": 5385, 2024-04-23T10:32:04.606 INFO:teuthology.orchestra.run.smithi114.stdout: "op_name": "quiesce_inode", 2024-04-23T10:32:04.606 INFO:teuthology.orchestra.run.smithi114.stdout: "events": [ 2024-04-23T10:32:04.606 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-04-23T10:32:04.606 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-04-23T10:31:21.446733+0000", 2024-04-23T10:32:04.606 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "initiated" 2024-04-23T10:32:04.606 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-23T10:32:04.606 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-04-23T10:32:04.607 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-04-23T10:31:21.446733+0000", 2024-04-23T10:32:04.607 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "header_read" 2024-04-23T10:32:04.607 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-23T10:32:04.607 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-04-23T10:32:04.607 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-04-23T10:31:21.446733+0000", 2024-04-23T10:32:04.607 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "throttled" 2024-04-23T10:32:04.607 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-23T10:32:04.607 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-04-23T10:32:04.607 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-04-23T10:31:21.446733+0000", 2024-04-23T10:32:04.607 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "all_read" 2024-04-23T10:32:04.607 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-23T10:32:04.607 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-04-23T10:32:04.607 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-04-23T10:31:21.446733+0000", 2024-04-23T10:32:04.607 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "dispatched" 2024-04-23T10:32:04.607 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-23T10:32:04.608 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-04-23T10:32:04.608 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-04-23T10:31:21.446918+0000", 2024-04-23T10:32:04.608 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "failed to xlock, waiting" 2024-04-23T10:32:04.608 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-23T10:32:04.608 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-04-23T10:32:04.608 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-04-23T10:31:21.447563+0000", 2024-04-23T10:32:04.608 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "acquired locks" 2024-04-23T10:32:04.608 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-23T10:32:04.608 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-04-23T10:32:04.608 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-04-23T10:31:21.510037+0000", 2024-04-23T10:32:04.608 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "quiesce complete" 2024-04-23T10:32:04.608 INFO:teuthology.orchestra.run.smithi114.stdout: } 2024-04-23T10:32:04.608 INFO:teuthology.orchestra.run.smithi114.stdout: ], 2024-04-23T10:32:04.608 INFO:teuthology.orchestra.run.smithi114.stdout: "locks": [ 2024-04-23T10:32:04.608 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-04-23T10:32:04.609 INFO:teuthology.orchestra.run.smithi114.stdout: "object": { 2024-04-23T10:32:04.609 INFO:teuthology.orchestra.run.smithi114.stdout: "is_auth": true, 2024-04-23T10:32:04.609 INFO:teuthology.orchestra.run.smithi114.stdout: "auth_state": { 2024-04-23T10:32:04.609 INFO:teuthology.orchestra.run.smithi114.stdout: "replicas": { 2024-04-23T10:32:04.609 INFO:teuthology.orchestra.run.smithi114.stdout: "1": 1, 2024-04-23T10:32:04.609 INFO:teuthology.orchestra.run.smithi114.stdout: "2": 1 2024-04-23T10:32:04.609 INFO:teuthology.orchestra.run.smithi114.stdout: } 2024-04-23T10:32:04.609 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-23T10:32:04.609 INFO:teuthology.orchestra.run.smithi114.stdout: "replica_state": { 2024-04-23T10:32:04.609 INFO:teuthology.orchestra.run.smithi114.stdout: "authority": [ 2024-04-23T10:32:04.609 INFO:teuthology.orchestra.run.smithi114.stdout: 0, 2024-04-23T10:32:04.609 INFO:teuthology.orchestra.run.smithi114.stdout: -2 2024-04-23T10:32:04.609 INFO:teuthology.orchestra.run.smithi114.stdout: ], 2024-04-23T10:32:04.609 INFO:teuthology.orchestra.run.smithi114.stdout: "replica_nonce": 0 2024-04-23T10:32:04.609 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-23T10:32:04.609 INFO:teuthology.orchestra.run.smithi114.stdout: "auth_pins": 3, 2024-04-23T10:32:04.610 INFO:teuthology.orchestra.run.smithi114.stdout: "is_frozen": false, 2024-04-23T10:32:04.610 INFO:teuthology.orchestra.run.smithi114.stdout: "is_freezing": false, 2024-04-23T10:32:04.610 INFO:teuthology.orchestra.run.smithi114.stdout: "pins": { 2024-04-23T10:32:04.610 INFO:teuthology.orchestra.run.smithi114.stdout: "dirtyscattered": 1, 2024-04-23T10:32:04.610 INFO:teuthology.orchestra.run.smithi114.stdout: "request": 1, 2024-04-23T10:32:04.610 INFO:teuthology.orchestra.run.smithi114.stdout: "lock": 1, 2024-04-23T10:32:04.610 INFO:teuthology.orchestra.run.smithi114.stdout: "dirfrag": 1, 2024-04-23T10:32:04.610 INFO:teuthology.orchestra.run.smithi114.stdout: "caps": 0, 2024-04-23T10:32:04.610 INFO:teuthology.orchestra.run.smithi114.stdout: "dirtyrstat": 0, 2024-04-23T10:32:04.610 INFO:teuthology.orchestra.run.smithi114.stdout: "dirtyparent": 0, 2024-04-23T10:32:04.610 INFO:teuthology.orchestra.run.smithi114.stdout: "replicated": 1, 2024-04-23T10:32:04.610 INFO:teuthology.orchestra.run.smithi114.stdout: "dirty": 1, 2024-04-23T10:32:04.610 INFO:teuthology.orchestra.run.smithi114.stdout: "waiter": 0, 2024-04-23T10:32:04.610 INFO:teuthology.orchestra.run.smithi114.stdout: "authpin": 1 2024-04-23T10:32:04.611 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-23T10:32:04.611 INFO:teuthology.orchestra.run.smithi114.stdout: "nref": 7 2024-04-23T10:32:04.611 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-23T10:32:04.611 INFO:teuthology.orchestra.run.smithi114.stdout: "object_string": "[inode 0x10000000000 [...7,head] /volumes/ auth{1=1,2=1} v406 ap=3 f(v0 m2024-04-23T10:24:49.736808+0000 3=2+1) n(v29 rc2024-04-23T10:31:17.764716+0000 b21075308 87=67+20) old_inodes=5 (inest mix dirty) (iquiesce lock x=1 by request(mds.0:151 nref=3)) | dirtyscattered=1 request=1 lock=1 dirfrag=1 caps=0 dirtyrstat=0 dirtyparent=0 replicated=1 dirty=1 waiter=0 authpin=1 0x5571cf18f080]", 2024-04-23T10:32:04.611 INFO:teuthology.orchestra.run.smithi114.stdout: "lock": { 2024-04-23T10:32:04.611 INFO:teuthology.orchestra.run.smithi114.stdout: "gather_set": [], 2024-04-23T10:32:04.611 INFO:teuthology.orchestra.run.smithi114.stdout: "state": "lock", 2024-04-23T10:32:04.611 INFO:teuthology.orchestra.run.smithi114.stdout: "type": "iquiesce", 2024-04-23T10:32:04.611 INFO:teuthology.orchestra.run.smithi114.stdout: "is_leased": false, 2024-04-23T10:32:04.611 INFO:teuthology.orchestra.run.smithi114.stdout: "num_rdlocks": 0, 2024-04-23T10:32:04.611 INFO:teuthology.orchestra.run.smithi114.stdout: "num_wrlocks": 0, 2024-04-23T10:32:04.611 INFO:teuthology.orchestra.run.smithi114.stdout: "num_xlocks": 1, 2024-04-23T10:32:04.611 INFO:teuthology.orchestra.run.smithi114.stdout: "xlock_by": { 2024-04-23T10:32:04.611 INFO:teuthology.orchestra.run.smithi114.stdout: "reqid": { 2024-04-23T10:32:04.612 INFO:teuthology.orchestra.run.smithi114.stdout: "entity": { 2024-04-23T10:32:04.612 INFO:teuthology.orchestra.run.smithi114.stdout: "type": "mds", 2024-04-23T10:32:04.612 INFO:teuthology.orchestra.run.smithi114.stdout: "num": 0 2024-04-23T10:32:04.612 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-23T10:32:04.612 INFO:teuthology.orchestra.run.smithi114.stdout: "tid": 151 2024-04-23T10:32:04.612 INFO:teuthology.orchestra.run.smithi114.stdout: } 2024-04-23T10:32:04.612 INFO:teuthology.orchestra.run.smithi114.stdout: } 2024-04-23T10:32:04.612 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-23T10:32:04.612 INFO:teuthology.orchestra.run.smithi114.stdout: "flags": 4, 2024-04-23T10:32:04.612 INFO:teuthology.orchestra.run.smithi114.stdout: "wrlock_target": -1 2024-04-23T10:32:04.612 INFO:teuthology.orchestra.run.smithi114.stdout: } 2024-04-23T10:32:04.612 INFO:teuthology.orchestra.run.smithi114.stdout: ] 2024-04-23T10:32:04.612 INFO:teuthology.orchestra.run.smithi114.stdout: } 2024-04-23T10:32:04.612 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-23T10:32:04.612 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-04-23T10:32:04.613 INFO:teuthology.orchestra.run.smithi114.stdout: "description": "internal op quiesce_inode:mds.0:166 fp=#0x10000000215", 2024-04-23T10:32:04.613 INFO:teuthology.orchestra.run.smithi114.stdout: "initiated_at": "2024-04-23T10:31:21.459820+0000", 2024-04-23T10:32:04.613 INFO:teuthology.orchestra.run.smithi114.stdout: "age": 42.424569529000003, 2024-04-23T10:32:04.613 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 42.425971025000003, 2024-04-23T10:32:04.613 INFO:teuthology.orchestra.run.smithi114.stdout: "continuous": true, 2024-04-23T10:32:04.613 INFO:teuthology.orchestra.run.smithi114.stdout: "type_data": { 2024-04-23T10:32:04.613 INFO:teuthology.orchestra.run.smithi114.stdout: "result": -2147483648, 2024-04-23T10:32:04.613 INFO:teuthology.orchestra.run.smithi114.stdout: "flag_point": "quiesce complete", 2024-04-23T10:32:04.613 INFO:teuthology.orchestra.run.smithi114.stdout: "reqid": { 2024-04-23T10:32:04.613 INFO:teuthology.orchestra.run.smithi114.stdout: "entity": { 2024-04-23T10:32:04.613 INFO:teuthology.orchestra.run.smithi114.stdout: "type": "mds", 2024-04-23T10:32:04.613 INFO:teuthology.orchestra.run.smithi114.stdout: "num": 0 2024-04-23T10:32:04.613 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-23T10:32:04.614 INFO:teuthology.orchestra.run.smithi114.stdout: "tid": 166 2024-04-23T10:32:04.614 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-23T10:32:04.614 INFO:teuthology.orchestra.run.smithi114.stdout: "op_type": "internal_op", 2024-04-23T10:32:04.614 INFO:teuthology.orchestra.run.smithi114.stdout: "internal_op": 5385, 2024-04-23T10:32:04.614 INFO:teuthology.orchestra.run.smithi114.stdout: "op_name": "quiesce_inode", 2024-04-23T10:32:04.614 INFO:teuthology.orchestra.run.smithi114.stdout: "events": [ 2024-04-23T10:32:04.614 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-04-23T10:32:04.614 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-04-23T10:31:21.459820+0000", 2024-04-23T10:32:04.614 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "initiated" 2024-04-23T10:32:04.614 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-23T10:32:04.614 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-04-23T10:32:04.614 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-04-23T10:31:21.459820+0000", 2024-04-23T10:32:04.614 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "header_read" 2024-04-23T10:32:04.614 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-23T10:32:04.614 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-04-23T10:32:04.615 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-04-23T10:31:21.459820+0000", 2024-04-23T10:32:04.615 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "throttled" 2024-04-23T10:32:04.615 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-23T10:32:04.615 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-04-23T10:32:04.615 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-04-23T10:31:21.459820+0000", 2024-04-23T10:32:04.615 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "all_read" 2024-04-23T10:32:04.615 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-23T10:32:04.615 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-04-23T10:32:04.615 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-04-23T10:31:21.459820+0000", 2024-04-23T10:32:04.615 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "dispatched" 2024-04-23T10:32:04.615 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-23T10:32:04.615 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-04-23T10:32:04.615 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-04-23T10:31:21.460243+0000", 2024-04-23T10:32:04.615 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "failed to xlock, waiting" 2024-04-23T10:32:04.616 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-23T10:32:04.616 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-04-23T10:32:04.616 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-04-23T10:31:21.482957+0000", 2024-04-23T10:32:04.616 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "acquired locks" 2024-04-23T10:32:04.616 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-23T10:32:04.616 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-04-23T10:32:04.616 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-04-23T10:31:21.508224+0000", 2024-04-23T10:32:04.616 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "quiesce complete" 2024-04-23T10:32:04.616 INFO:teuthology.orchestra.run.smithi114.stdout: } 2024-04-23T10:32:04.616 INFO:teuthology.orchestra.run.smithi114.stdout: ], 2024-04-23T10:32:04.616 INFO:teuthology.orchestra.run.smithi114.stdout: "locks": [ 2024-04-23T10:32:04.616 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-04-23T10:32:04.616 INFO:teuthology.orchestra.run.smithi114.stdout: "object": { 2024-04-23T10:32:04.616 INFO:teuthology.orchestra.run.smithi114.stdout: "is_auth": true, 2024-04-23T10:32:04.616 INFO:teuthology.orchestra.run.smithi114.stdout: "auth_state": { 2024-04-23T10:32:04.617 INFO:teuthology.orchestra.run.smithi114.stdout: "replicas": { 2024-04-23T10:32:04.617 INFO:teuthology.orchestra.run.smithi114.stdout: "1": 1, 2024-04-23T10:32:04.617 INFO:teuthology.orchestra.run.smithi114.stdout: "2": 1 2024-04-23T10:32:04.617 INFO:teuthology.orchestra.run.smithi114.stdout: } 2024-04-23T10:32:04.617 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-23T10:32:04.617 INFO:teuthology.orchestra.run.smithi114.stdout: "replica_state": { 2024-04-23T10:32:04.617 INFO:teuthology.orchestra.run.smithi114.stdout: "authority": [ 2024-04-23T10:32:04.617 INFO:teuthology.orchestra.run.smithi114.stdout: 0, 2024-04-23T10:32:04.617 INFO:teuthology.orchestra.run.smithi114.stdout: -2 2024-04-23T10:32:04.617 INFO:teuthology.orchestra.run.smithi114.stdout: ], 2024-04-23T10:32:04.617 INFO:teuthology.orchestra.run.smithi114.stdout: "replica_nonce": 0 2024-04-23T10:32:04.617 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-23T10:32:04.617 INFO:teuthology.orchestra.run.smithi114.stdout: "auth_pins": 3, 2024-04-23T10:32:04.617 INFO:teuthology.orchestra.run.smithi114.stdout: "is_frozen": false, 2024-04-23T10:32:04.617 INFO:teuthology.orchestra.run.smithi114.stdout: "is_freezing": false, 2024-04-23T10:32:04.618 INFO:teuthology.orchestra.run.smithi114.stdout: "pins": { 2024-04-23T10:32:04.618 INFO:teuthology.orchestra.run.smithi114.stdout: "dirtyscattered": 0, 2024-04-23T10:32:04.618 INFO:teuthology.orchestra.run.smithi114.stdout: "request": 1, 2024-04-23T10:32:04.618 INFO:teuthology.orchestra.run.smithi114.stdout: "lock": 1, 2024-04-23T10:32:04.618 INFO:teuthology.orchestra.run.smithi114.stdout: "dirfrag": 1, 2024-04-23T10:32:04.618 INFO:teuthology.orchestra.run.smithi114.stdout: "caps": 1, 2024-04-23T10:32:04.618 INFO:teuthology.orchestra.run.smithi114.stdout: "dirtyrstat": 0, 2024-04-23T10:32:04.618 INFO:teuthology.orchestra.run.smithi114.stdout: "dirtyparent": 0, 2024-04-23T10:32:04.618 INFO:teuthology.orchestra.run.smithi114.stdout: "replicated": 1, 2024-04-23T10:32:04.618 INFO:teuthology.orchestra.run.smithi114.stdout: "dirty": 0, 2024-04-23T10:32:04.618 INFO:teuthology.orchestra.run.smithi114.stdout: "waiter": 0, 2024-04-23T10:32:04.618 INFO:teuthology.orchestra.run.smithi114.stdout: "authpin": 1 2024-04-23T10:32:04.618 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-23T10:32:04.618 INFO:teuthology.orchestra.run.smithi114.stdout: "nref": 6 2024-04-23T10:32:04.619 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-23T10:32:04.619 INFO:teuthology.orchestra.run.smithi114.stdout: "object_string": "[inode 0x10000000215 [...7,head] /volumes/qa/sv_1/b206794e-7622-4ab8-8943-f3ae4e65b7a3/client.0/tmp/clients/client0/~dmtmp/COREL/ auth{1=1,2=1} v45589 ap=3 f(v1232 m2024-04-23T10:31:17.745716+0000 11=11+0) n(v19 rc2024-04-23T10:31:17.745716+0000 b2861056 12=11+1) old_inodes=4 (inest mix) (iquiesce lock x=1 by request(mds.0:166 nref=3)) caps={15312=pAsLsXsFs/pAsLsXsFsxcral@3233} mcw={2=AsLsXsFs} | dirtyscattered=0 request=1 lock=1 dirfrag=1 caps=1 dirtyrstat=0 dirtyparent=0 replicated=1 dirty=0 waiter=0 authpin=1 0x5571cf541700]", 2024-04-23T10:32:04.619 INFO:teuthology.orchestra.run.smithi114.stdout: "lock": { 2024-04-23T10:32:04.619 INFO:teuthology.orchestra.run.smithi114.stdout: "gather_set": [], 2024-04-23T10:32:04.619 INFO:teuthology.orchestra.run.smithi114.stdout: "state": "lock", 2024-04-23T10:32:04.619 INFO:teuthology.orchestra.run.smithi114.stdout: "type": "iquiesce", 2024-04-23T10:32:04.619 INFO:teuthology.orchestra.run.smithi114.stdout: "is_leased": false, 2024-04-23T10:32:04.619 INFO:teuthology.orchestra.run.smithi114.stdout: "num_rdlocks": 0, 2024-04-23T10:32:04.619 INFO:teuthology.orchestra.run.smithi114.stdout: "num_wrlocks": 0, 2024-04-23T10:32:04.619 INFO:teuthology.orchestra.run.smithi114.stdout: "num_xlocks": 1, 2024-04-23T10:32:04.619 INFO:teuthology.orchestra.run.smithi114.stdout: "xlock_by": { 2024-04-23T10:32:04.619 INFO:teuthology.orchestra.run.smithi114.stdout: "reqid": { 2024-04-23T10:32:04.619 INFO:teuthology.orchestra.run.smithi114.stdout: "entity": { 2024-04-23T10:32:04.619 INFO:teuthology.orchestra.run.smithi114.stdout: "type": "mds", 2024-04-23T10:32:04.619 INFO:teuthology.orchestra.run.smithi114.stdout: "num": 0 2024-04-23T10:32:04.620 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-23T10:32:04.620 INFO:teuthology.orchestra.run.smithi114.stdout: "tid": 166 2024-04-23T10:32:04.620 INFO:teuthology.orchestra.run.smithi114.stdout: } 2024-04-23T10:32:04.620 INFO:teuthology.orchestra.run.smithi114.stdout: } 2024-04-23T10:32:04.620 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-23T10:32:04.620 INFO:teuthology.orchestra.run.smithi114.stdout: "flags": 4, 2024-04-23T10:32:04.620 INFO:teuthology.orchestra.run.smithi114.stdout: "wrlock_target": -1 2024-04-23T10:32:04.620 INFO:teuthology.orchestra.run.smithi114.stdout: } 2024-04-23T10:32:04.620 INFO:teuthology.orchestra.run.smithi114.stdout: ] 2024-04-23T10:32:04.620 INFO:teuthology.orchestra.run.smithi114.stdout: } 2024-04-23T10:32:04.620 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-23T10:32:04.620 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-04-23T10:32:04.620 INFO:teuthology.orchestra.run.smithi114.stdout: "description": "internal op quiesce_inode:mds.0:191 fp=#0x1000000021b", 2024-04-23T10:32:04.620 INFO:teuthology.orchestra.run.smithi114.stdout: "initiated_at": "2024-04-23T10:31:21.483386+0000", 2024-04-23T10:32:04.620 INFO:teuthology.orchestra.run.smithi114.stdout: "age": 42.401002789000003, 2024-04-23T10:32:04.620 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 42.402473235999999, 2024-04-23T10:32:04.621 INFO:teuthology.orchestra.run.smithi114.stdout: "continuous": true, 2024-04-23T10:32:04.621 INFO:teuthology.orchestra.run.smithi114.stdout: "type_data": { 2024-04-23T10:32:04.621 INFO:teuthology.orchestra.run.smithi114.stdout: "result": -2147483648, 2024-04-23T10:32:04.621 INFO:teuthology.orchestra.run.smithi114.stdout: "flag_point": "quiesce complete for non-auth inode", 2024-04-23T10:32:04.621 INFO:teuthology.orchestra.run.smithi114.stdout: "reqid": { 2024-04-23T10:32:04.621 INFO:teuthology.orchestra.run.smithi114.stdout: "entity": { 2024-04-23T10:32:04.621 INFO:teuthology.orchestra.run.smithi114.stdout: "type": "mds", 2024-04-23T10:32:04.621 INFO:teuthology.orchestra.run.smithi114.stdout: "num": 0 2024-04-23T10:32:04.621 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-23T10:32:04.621 INFO:teuthology.orchestra.run.smithi114.stdout: "tid": 191 2024-04-23T10:32:04.621 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-23T10:32:04.621 INFO:teuthology.orchestra.run.smithi114.stdout: "op_type": "internal_op", 2024-04-23T10:32:04.621 INFO:teuthology.orchestra.run.smithi114.stdout: "internal_op": 5385, 2024-04-23T10:32:04.621 INFO:teuthology.orchestra.run.smithi114.stdout: "op_name": "quiesce_inode", 2024-04-23T10:32:04.622 INFO:teuthology.orchestra.run.smithi114.stdout: "events": [ 2024-04-23T10:32:04.622 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-04-23T10:32:04.622 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-04-23T10:31:21.483386+0000", 2024-04-23T10:32:04.622 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "initiated" 2024-04-23T10:32:04.622 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-23T10:32:04.622 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-04-23T10:32:04.622 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-04-23T10:31:21.483386+0000", 2024-04-23T10:32:04.622 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "header_read" 2024-04-23T10:32:04.622 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-23T10:32:04.622 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-04-23T10:32:04.622 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-04-23T10:31:21.483386+0000", 2024-04-23T10:32:04.622 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "throttled" 2024-04-23T10:32:04.622 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-23T10:32:04.622 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-04-23T10:32:04.623 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-04-23T10:31:21.483386+0000", 2024-04-23T10:32:04.623 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "all_read" 2024-04-23T10:32:04.623 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-23T10:32:04.623 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-04-23T10:32:04.623 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-04-23T10:31:21.483386+0000", 2024-04-23T10:32:04.623 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "dispatched" 2024-04-23T10:32:04.623 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-23T10:32:04.623 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-04-23T10:32:04.623 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-04-23T10:31:21.483634+0000", 2024-04-23T10:32:04.623 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "requesting remote authpins" 2024-04-23T10:32:04.623 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-23T10:32:04.623 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-04-23T10:32:04.623 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-04-23T10:31:21.506519+0000", 2024-04-23T10:32:04.623 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "acquired locks" 2024-04-23T10:32:04.624 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-23T10:32:04.624 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-04-23T10:32:04.624 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-04-23T10:31:21.506527+0000", 2024-04-23T10:32:04.624 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "quiesce complete for non-auth inode" 2024-04-23T10:32:04.624 INFO:teuthology.orchestra.run.smithi114.stdout: } 2024-04-23T10:32:04.624 INFO:teuthology.orchestra.run.smithi114.stdout: ], 2024-04-23T10:32:04.624 INFO:teuthology.orchestra.run.smithi114.stdout: "locks": [ 2024-04-23T10:32:04.624 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-04-23T10:32:04.624 INFO:teuthology.orchestra.run.smithi114.stdout: "object": { 2024-04-23T10:32:04.624 INFO:teuthology.orchestra.run.smithi114.stdout: "is_auth": false, 2024-04-23T10:32:04.624 INFO:teuthology.orchestra.run.smithi114.stdout: "auth_state": { 2024-04-23T10:32:04.624 INFO:teuthology.orchestra.run.smithi114.stdout: "replicas": {} 2024-04-23T10:32:04.624 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-23T10:32:04.624 INFO:teuthology.orchestra.run.smithi114.stdout: "replica_state": { 2024-04-23T10:32:04.624 INFO:teuthology.orchestra.run.smithi114.stdout: "authority": [ 2024-04-23T10:32:04.624 INFO:teuthology.orchestra.run.smithi114.stdout: 2, 2024-04-23T10:32:04.625 INFO:teuthology.orchestra.run.smithi114.stdout: -2 2024-04-23T10:32:04.625 INFO:teuthology.orchestra.run.smithi114.stdout: ], 2024-04-23T10:32:04.625 INFO:teuthology.orchestra.run.smithi114.stdout: "replica_nonce": 1 2024-04-23T10:32:04.625 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-23T10:32:04.625 INFO:teuthology.orchestra.run.smithi114.stdout: "auth_pins": 0, 2024-04-23T10:32:04.625 INFO:teuthology.orchestra.run.smithi114.stdout: "is_frozen": false, 2024-04-23T10:32:04.625 INFO:teuthology.orchestra.run.smithi114.stdout: "is_freezing": false, 2024-04-23T10:32:04.625 INFO:teuthology.orchestra.run.smithi114.stdout: "pins": { 2024-04-23T10:32:04.625 INFO:teuthology.orchestra.run.smithi114.stdout: "request": 1, 2024-04-23T10:32:04.625 INFO:teuthology.orchestra.run.smithi114.stdout: "lock": 1, 2024-04-23T10:32:04.625 INFO:teuthology.orchestra.run.smithi114.stdout: "caps": 0, 2024-04-23T10:32:04.625 INFO:teuthology.orchestra.run.smithi114.stdout: "exportingcaps": 0, 2024-04-23T10:32:04.625 INFO:teuthology.orchestra.run.smithi114.stdout: "dirtyparent": 0, 2024-04-23T10:32:04.625 INFO:teuthology.orchestra.run.smithi114.stdout: "dirty": 0, 2024-04-23T10:32:04.625 INFO:teuthology.orchestra.run.smithi114.stdout: "authpin": 0, 2024-04-23T10:32:04.626 INFO:teuthology.orchestra.run.smithi114.stdout: "tempexporting": 0 2024-04-23T10:32:04.626 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-23T10:32:04.626 INFO:teuthology.orchestra.run.smithi114.stdout: "nref": 2 2024-04-23T10:32:04.626 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-23T10:32:04.626 INFO:teuthology.orchestra.run.smithi114.stdout: "object_string": "[inode 0x1000000021b [7,head] /volumes/qa/sv_1/b206794e-7622-4ab8-8943-f3ae4e65b7a3/client.0/tmp/clients/client0/~dmtmp/COREL/TEAPOT2.CPT rep@2.1 v7248 s=260096 n(v0 rc2024-04-23T10:27:52.028396+0000 b260096 1=1+0)/n(v0 1=1+0) (iquiesce lock x=1 by request(mds.0:191 nref=3)) | request=1 lock=1 caps=0 exportingcaps=0 dirtyparent=0 dirty=0 authpin=0 tempexporting=0 0x5571cf561b80]", 2024-04-23T10:32:04.626 INFO:teuthology.orchestra.run.smithi114.stdout: "lock": { 2024-04-23T10:32:04.626 INFO:teuthology.orchestra.run.smithi114.stdout: "gather_set": [], 2024-04-23T10:32:04.626 INFO:teuthology.orchestra.run.smithi114.stdout: "state": "lock", 2024-04-23T10:32:04.626 INFO:teuthology.orchestra.run.smithi114.stdout: "type": "iquiesce", 2024-04-23T10:32:04.626 INFO:teuthology.orchestra.run.smithi114.stdout: "is_leased": false, 2024-04-23T10:32:04.626 INFO:teuthology.orchestra.run.smithi114.stdout: "num_rdlocks": 0, 2024-04-23T10:32:04.626 INFO:teuthology.orchestra.run.smithi114.stdout: "num_wrlocks": 0, 2024-04-23T10:32:04.626 INFO:teuthology.orchestra.run.smithi114.stdout: "num_xlocks": 1, 2024-04-23T10:32:04.627 INFO:teuthology.orchestra.run.smithi114.stdout: "xlock_by": { 2024-04-23T10:32:04.627 INFO:teuthology.orchestra.run.smithi114.stdout: "reqid": { 2024-04-23T10:32:04.627 INFO:teuthology.orchestra.run.smithi114.stdout: "entity": { 2024-04-23T10:32:04.627 INFO:teuthology.orchestra.run.smithi114.stdout: "type": "mds", 2024-04-23T10:32:04.627 INFO:teuthology.orchestra.run.smithi114.stdout: "num": 0 2024-04-23T10:32:04.627 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-23T10:32:04.627 INFO:teuthology.orchestra.run.smithi114.stdout: "tid": 191 2024-04-23T10:32:04.627 INFO:teuthology.orchestra.run.smithi114.stdout: } 2024-04-23T10:32:04.627 INFO:teuthology.orchestra.run.smithi114.stdout: } 2024-04-23T10:32:04.627 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-23T10:32:04.627 INFO:teuthology.orchestra.run.smithi114.stdout: "flags": 4, 2024-04-23T10:32:04.627 INFO:teuthology.orchestra.run.smithi114.stdout: "wrlock_target": -1 2024-04-23T10:32:04.627 INFO:teuthology.orchestra.run.smithi114.stdout: } 2024-04-23T10:32:04.627 INFO:teuthology.orchestra.run.smithi114.stdout: ] 2024-04-23T10:32:04.628 INFO:teuthology.orchestra.run.smithi114.stdout: } 2024-04-23T10:32:04.628 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-23T10:32:04.628 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-04-23T10:32:04.628 INFO:teuthology.orchestra.run.smithi114.stdout: "description": "peer_request:mds.1:103", 2024-04-23T10:32:04.628 INFO:teuthology.orchestra.run.smithi114.stdout: "initiated_at": "2024-04-23T10:31:21.446187+0000", 2024-04-23T10:32:04.628 INFO:teuthology.orchestra.run.smithi114.stdout: "age": 42.438202668000002, 2024-04-23T10:32:04.628 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 42.439732186999997, 2024-04-23T10:32:04.628 INFO:teuthology.orchestra.run.smithi114.stdout: "continuous": false, 2024-04-23T10:32:04.628 INFO:teuthology.orchestra.run.smithi114.stdout: "type_data": { 2024-04-23T10:32:04.628 INFO:teuthology.orchestra.run.smithi114.stdout: "result": -2147483648, 2024-04-23T10:32:04.628 INFO:teuthology.orchestra.run.smithi114.stdout: "flag_point": "dispatched", 2024-04-23T10:32:04.628 INFO:teuthology.orchestra.run.smithi114.stdout: "reqid": { 2024-04-23T10:32:04.628 INFO:teuthology.orchestra.run.smithi114.stdout: "entity": { 2024-04-23T10:32:04.628 INFO:teuthology.orchestra.run.smithi114.stdout: "type": "mds", 2024-04-23T10:32:04.628 INFO:teuthology.orchestra.run.smithi114.stdout: "num": 1 2024-04-23T10:32:04.628 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-23T10:32:04.629 INFO:teuthology.orchestra.run.smithi114.stdout: "tid": 103 2024-04-23T10:32:04.629 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-23T10:32:04.629 INFO:teuthology.orchestra.run.smithi114.stdout: "op_type": "peer_request", 2024-04-23T10:32:04.629 INFO:teuthology.orchestra.run.smithi114.stdout: "leader_info": { 2024-04-23T10:32:04.629 INFO:teuthology.orchestra.run.smithi114.stdout: "leader": "1" 2024-04-23T10:32:04.629 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-23T10:32:04.629 INFO:teuthology.orchestra.run.smithi114.stdout: "events": [ 2024-04-23T10:32:04.629 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-04-23T10:32:04.629 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-04-23T10:31:21.446187+0000", 2024-04-23T10:32:04.629 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "initiated" 2024-04-23T10:32:04.629 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-23T10:32:04.629 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-04-23T10:32:04.629 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-04-23T10:31:21.446187+0000", 2024-04-23T10:32:04.629 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "header_read" 2024-04-23T10:32:04.630 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-23T10:32:04.630 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-04-23T10:32:04.630 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-04-23T10:31:21.446187+0000", 2024-04-23T10:32:04.630 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "throttled" 2024-04-23T10:32:04.630 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-23T10:32:04.630 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-04-23T10:32:04.630 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-04-23T10:31:21.446194+0000", 2024-04-23T10:32:04.630 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "all_read" 2024-04-23T10:32:04.630 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-23T10:32:04.630 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-04-23T10:32:04.630 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-04-23T10:31:21.447001+0000", 2024-04-23T10:32:04.630 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "dispatched" 2024-04-23T10:32:04.631 INFO:teuthology.orchestra.run.smithi114.stdout: } 2024-04-23T10:32:04.631 INFO:teuthology.orchestra.run.smithi114.stdout: ], 2024-04-23T10:32:04.631 INFO:teuthology.orchestra.run.smithi114.stdout: "locks": [] 2024-04-23T10:32:04.631 INFO:teuthology.orchestra.run.smithi114.stdout: } 2024-04-23T10:32:04.631 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-23T10:32:04.631 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-04-23T10:32:04.631 INFO:teuthology.orchestra.run.smithi114.stdout: "description": "internal op quiesce_inode:mds.0:167 fp=#0x1000000020c", 2024-04-23T10:32:04.631 INFO:teuthology.orchestra.run.smithi114.stdout: "initiated_at": "2024-04-23T10:31:21.459843+0000", 2024-04-23T10:32:04.631 INFO:teuthology.orchestra.run.smithi114.stdout: "age": 42.424546264, 2024-04-23T10:32:04.631 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 42.426096846, 2024-04-23T10:32:04.631 INFO:teuthology.orchestra.run.smithi114.stdout: "continuous": true, 2024-04-23T10:32:04.631 INFO:teuthology.orchestra.run.smithi114.stdout: "type_data": { 2024-04-23T10:32:04.631 INFO:teuthology.orchestra.run.smithi114.stdout: "result": -2147483648, 2024-04-23T10:32:04.632 INFO:teuthology.orchestra.run.smithi114.stdout: "flag_point": "quiesce complete", 2024-04-23T10:32:04.632 INFO:teuthology.orchestra.run.smithi114.stdout: "reqid": { 2024-04-23T10:32:04.632 INFO:teuthology.orchestra.run.smithi114.stdout: "entity": { 2024-04-23T10:32:04.632 INFO:teuthology.orchestra.run.smithi114.stdout: "type": "mds", 2024-04-23T10:32:04.632 INFO:teuthology.orchestra.run.smithi114.stdout: "num": 0 2024-04-23T10:32:04.632 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-23T10:32:04.632 INFO:teuthology.orchestra.run.smithi114.stdout: "tid": 167 2024-04-23T10:32:04.632 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-23T10:32:04.632 INFO:teuthology.orchestra.run.smithi114.stdout: "op_type": "internal_op", 2024-04-23T10:32:04.632 INFO:teuthology.orchestra.run.smithi114.stdout: "internal_op": 5385, 2024-04-23T10:32:04.632 INFO:teuthology.orchestra.run.smithi114.stdout: "op_name": "quiesce_inode", 2024-04-23T10:32:04.632 INFO:teuthology.orchestra.run.smithi114.stdout: "events": [ 2024-04-23T10:32:04.632 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-04-23T10:32:04.632 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-04-23T10:31:21.459843+0000", 2024-04-23T10:32:04.633 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "initiated" 2024-04-23T10:32:04.633 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-23T10:32:04.633 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-04-23T10:32:04.633 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-04-23T10:31:21.459843+0000", 2024-04-23T10:32:04.633 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "header_read" 2024-04-23T10:32:04.633 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-23T10:32:04.633 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-04-23T10:32:04.633 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-04-23T10:31:21.459843+0000", 2024-04-23T10:32:04.633 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "throttled" 2024-04-23T10:32:04.633 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-23T10:32:04.633 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-04-23T10:32:04.633 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-04-23T10:31:21.459843+0000", 2024-04-23T10:32:04.633 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "all_read" 2024-04-23T10:32:04.633 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-23T10:32:04.634 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-04-23T10:32:04.634 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-04-23T10:31:21.459843+0000", 2024-04-23T10:32:04.634 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "dispatched" 2024-04-23T10:32:04.634 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-23T10:32:04.634 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-04-23T10:32:04.634 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-04-23T10:31:21.460564+0000", 2024-04-23T10:32:04.634 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "failed to xlock, waiting" 2024-04-23T10:32:04.634 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-23T10:32:04.634 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-04-23T10:32:04.634 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-04-23T10:31:21.463248+0000", 2024-04-23T10:32:04.634 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "acquired locks" 2024-04-23T10:32:04.634 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-23T10:32:04.634 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-04-23T10:32:04.634 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-04-23T10:31:21.488473+0000", 2024-04-23T10:32:04.635 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "quiesce complete" 2024-04-23T10:32:04.635 INFO:teuthology.orchestra.run.smithi114.stdout: } 2024-04-23T10:32:04.635 INFO:teuthology.orchestra.run.smithi114.stdout: ], 2024-04-23T10:32:04.635 INFO:teuthology.orchestra.run.smithi114.stdout: "locks": [ 2024-04-23T10:32:04.635 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-04-23T10:32:04.635 INFO:teuthology.orchestra.run.smithi114.stdout: "object": { 2024-04-23T10:32:04.635 INFO:teuthology.orchestra.run.smithi114.stdout: "is_auth": true, 2024-04-23T10:32:04.635 INFO:teuthology.orchestra.run.smithi114.stdout: "auth_state": { 2024-04-23T10:32:04.635 INFO:teuthology.orchestra.run.smithi114.stdout: "replicas": { 2024-04-23T10:32:04.635 INFO:teuthology.orchestra.run.smithi114.stdout: "1": 1, 2024-04-23T10:32:04.635 INFO:teuthology.orchestra.run.smithi114.stdout: "2": 1 2024-04-23T10:32:04.635 INFO:teuthology.orchestra.run.smithi114.stdout: } 2024-04-23T10:32:04.635 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-23T10:32:04.635 INFO:teuthology.orchestra.run.smithi114.stdout: "replica_state": { 2024-04-23T10:32:04.635 INFO:teuthology.orchestra.run.smithi114.stdout: "authority": [ 2024-04-23T10:32:04.636 INFO:teuthology.orchestra.run.smithi114.stdout: 0, 2024-04-23T10:32:04.636 INFO:teuthology.orchestra.run.smithi114.stdout: -2 2024-04-23T10:32:04.636 INFO:teuthology.orchestra.run.smithi114.stdout: ], 2024-04-23T10:32:04.636 INFO:teuthology.orchestra.run.smithi114.stdout: "replica_nonce": 0 2024-04-23T10:32:04.636 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-23T10:32:04.636 INFO:teuthology.orchestra.run.smithi114.stdout: "auth_pins": 3, 2024-04-23T10:32:04.636 INFO:teuthology.orchestra.run.smithi114.stdout: "is_frozen": false, 2024-04-23T10:32:04.636 INFO:teuthology.orchestra.run.smithi114.stdout: "is_freezing": false, 2024-04-23T10:32:04.636 INFO:teuthology.orchestra.run.smithi114.stdout: "pins": { 2024-04-23T10:32:04.636 INFO:teuthology.orchestra.run.smithi114.stdout: "request": 1, 2024-04-23T10:32:04.636 INFO:teuthology.orchestra.run.smithi114.stdout: "lock": 1, 2024-04-23T10:32:04.636 INFO:teuthology.orchestra.run.smithi114.stdout: "dirfrag": 1, 2024-04-23T10:32:04.636 INFO:teuthology.orchestra.run.smithi114.stdout: "caps": 1, 2024-04-23T10:32:04.636 INFO:teuthology.orchestra.run.smithi114.stdout: "dirtyrstat": 0, 2024-04-23T10:32:04.637 INFO:teuthology.orchestra.run.smithi114.stdout: "dirtyparent": 0, 2024-04-23T10:32:04.637 INFO:teuthology.orchestra.run.smithi114.stdout: "replicated": 1, 2024-04-23T10:32:04.637 INFO:teuthology.orchestra.run.smithi114.stdout: "dirty": 0, 2024-04-23T10:32:04.637 INFO:teuthology.orchestra.run.smithi114.stdout: "waiter": 0, 2024-04-23T10:32:04.637 INFO:teuthology.orchestra.run.smithi114.stdout: "authpin": 1 2024-04-23T10:32:04.637 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-23T10:32:04.637 INFO:teuthology.orchestra.run.smithi114.stdout: "nref": 6 2024-04-23T10:32:04.637 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-23T10:32:04.637 INFO:teuthology.orchestra.run.smithi114.stdout: "object_string": "[inode 0x1000000020c [...7,head] /volumes/qa/sv_1/b206794e-7622-4ab8-8943-f3ae4e65b7a3/client.0/tmp/clients/client0/~dmtmp/EXCEL/ auth{1=1,2=1} v45586 ap=3 f(v0 m2024-04-23T10:31:17.692717+0000 2=2+0) n(v0 rc2024-04-23T10:31:17.692717+0000 b520192 3=2+1) old_inodes=4 (inest mix) (iquiesce lock x=1 by request(mds.0:167 nref=3)) caps={15312=pAsLsXsFs/pAsLsXsFsxcral@9632},l=15312 | request=1 lock=1 dirfrag=1 caps=1 dirtyrstat=0 dirtyparent=0 replicated=1 dirty=0 waiter=0 authpin=1 0x5571cf53e580]", 2024-04-23T10:32:04.637 INFO:teuthology.orchestra.run.smithi114.stdout: "lock": { 2024-04-23T10:32:04.637 INFO:teuthology.orchestra.run.smithi114.stdout: "gather_set": [], 2024-04-23T10:32:04.637 INFO:teuthology.orchestra.run.smithi114.stdout: "state": "lock", 2024-04-23T10:32:04.637 INFO:teuthology.orchestra.run.smithi114.stdout: "type": "iquiesce", 2024-04-23T10:32:04.637 INFO:teuthology.orchestra.run.smithi114.stdout: "is_leased": false, 2024-04-23T10:32:04.637 INFO:teuthology.orchestra.run.smithi114.stdout: "num_rdlocks": 0, 2024-04-23T10:32:04.638 INFO:teuthology.orchestra.run.smithi114.stdout: "num_wrlocks": 0, 2024-04-23T10:32:04.638 INFO:teuthology.orchestra.run.smithi114.stdout: "num_xlocks": 1, 2024-04-23T10:32:04.638 INFO:teuthology.orchestra.run.smithi114.stdout: "xlock_by": { 2024-04-23T10:32:04.638 INFO:teuthology.orchestra.run.smithi114.stdout: "reqid": { 2024-04-23T10:32:04.638 INFO:teuthology.orchestra.run.smithi114.stdout: "entity": { 2024-04-23T10:32:04.638 INFO:teuthology.orchestra.run.smithi114.stdout: "type": "mds", 2024-04-23T10:32:04.638 INFO:teuthology.orchestra.run.smithi114.stdout: "num": 0 2024-04-23T10:32:04.638 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-23T10:32:04.638 INFO:teuthology.orchestra.run.smithi114.stdout: "tid": 167 2024-04-23T10:32:04.638 INFO:teuthology.orchestra.run.smithi114.stdout: } 2024-04-23T10:32:04.638 INFO:teuthology.orchestra.run.smithi114.stdout: } 2024-04-23T10:32:04.638 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-23T10:32:04.638 INFO:teuthology.orchestra.run.smithi114.stdout: "flags": 4, 2024-04-23T10:32:04.638 INFO:teuthology.orchestra.run.smithi114.stdout: "wrlock_target": -1 2024-04-23T10:32:04.639 INFO:teuthology.orchestra.run.smithi114.stdout: } 2024-04-23T10:32:04.639 INFO:teuthology.orchestra.run.smithi114.stdout: ] 2024-04-23T10:32:04.639 INFO:teuthology.orchestra.run.smithi114.stdout: } 2024-04-23T10:32:04.639 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-23T10:32:04.639 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-04-23T10:32:04.639 INFO:teuthology.orchestra.run.smithi114.stdout: "description": "internal op quiesce_inode:mds.0:192 fp=#0x1000000021a", 2024-04-23T10:32:04.639 INFO:teuthology.orchestra.run.smithi114.stdout: "initiated_at": "2024-04-23T10:31:21.483400+0000", 2024-04-23T10:32:04.639 INFO:teuthology.orchestra.run.smithi114.stdout: "age": 42.400989170999999, 2024-04-23T10:32:04.639 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 42.402606259000002, 2024-04-23T10:32:04.639 INFO:teuthology.orchestra.run.smithi114.stdout: "continuous": true, 2024-04-23T10:32:04.639 INFO:teuthology.orchestra.run.smithi114.stdout: "type_data": { 2024-04-23T10:32:04.639 INFO:teuthology.orchestra.run.smithi114.stdout: "result": -2147483648, 2024-04-23T10:32:04.639 INFO:teuthology.orchestra.run.smithi114.stdout: "flag_point": "quiesce complete for non-auth inode", 2024-04-23T10:32:04.639 INFO:teuthology.orchestra.run.smithi114.stdout: "reqid": { 2024-04-23T10:32:04.640 INFO:teuthology.orchestra.run.smithi114.stdout: "entity": { 2024-04-23T10:32:04.640 INFO:teuthology.orchestra.run.smithi114.stdout: "type": "mds", 2024-04-23T10:32:04.640 INFO:teuthology.orchestra.run.smithi114.stdout: "num": 0 2024-04-23T10:32:04.640 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-23T10:32:04.640 INFO:teuthology.orchestra.run.smithi114.stdout: "tid": 192 2024-04-23T10:32:04.640 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-23T10:32:04.640 INFO:teuthology.orchestra.run.smithi114.stdout: "op_type": "internal_op", 2024-04-23T10:32:04.640 INFO:teuthology.orchestra.run.smithi114.stdout: "internal_op": 5385, 2024-04-23T10:32:04.640 INFO:teuthology.orchestra.run.smithi114.stdout: "op_name": "quiesce_inode", 2024-04-23T10:32:04.640 INFO:teuthology.orchestra.run.smithi114.stdout: "events": [ 2024-04-23T10:32:04.640 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-04-23T10:32:04.640 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-04-23T10:31:21.483400+0000", 2024-04-23T10:32:04.640 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "initiated" 2024-04-23T10:32:04.640 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-23T10:32:04.640 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-04-23T10:32:04.641 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-04-23T10:31:21.483400+0000", 2024-04-23T10:32:04.641 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "header_read" 2024-04-23T10:32:04.641 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-23T10:32:04.641 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-04-23T10:32:04.641 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-04-23T10:31:21.483400+0000", 2024-04-23T10:32:04.641 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "throttled" 2024-04-23T10:32:04.641 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-23T10:32:04.641 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-04-23T10:32:04.641 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-04-23T10:31:21.483400+0000", 2024-04-23T10:32:04.641 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "all_read" 2024-04-23T10:32:04.641 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-23T10:32:04.641 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-04-23T10:32:04.641 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-04-23T10:31:21.483400+0000", 2024-04-23T10:32:04.641 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "dispatched" 2024-04-23T10:32:04.642 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-23T10:32:04.642 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-04-23T10:32:04.642 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-04-23T10:31:21.483686+0000", 2024-04-23T10:32:04.642 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "requesting remote authpins" 2024-04-23T10:32:04.642 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-23T10:32:04.642 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-04-23T10:32:04.642 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-04-23T10:31:21.506653+0000", 2024-04-23T10:32:04.642 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "acquired locks" 2024-04-23T10:32:04.642 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-23T10:32:04.642 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-04-23T10:32:04.642 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-04-23T10:31:21.506681+0000", 2024-04-23T10:32:04.642 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "quiesce complete for non-auth inode" 2024-04-23T10:32:04.642 INFO:teuthology.orchestra.run.smithi114.stdout: } 2024-04-23T10:32:04.642 INFO:teuthology.orchestra.run.smithi114.stdout: ], 2024-04-23T10:32:04.643 INFO:teuthology.orchestra.run.smithi114.stdout: "locks": [ 2024-04-23T10:32:04.643 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-04-23T10:32:04.643 INFO:teuthology.orchestra.run.smithi114.stdout: "object": { 2024-04-23T10:32:04.643 INFO:teuthology.orchestra.run.smithi114.stdout: "is_auth": false, 2024-04-23T10:32:04.643 INFO:teuthology.orchestra.run.smithi114.stdout: "auth_state": { 2024-04-23T10:32:04.643 INFO:teuthology.orchestra.run.smithi114.stdout: "replicas": {} 2024-04-23T10:32:04.643 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-23T10:32:04.643 INFO:teuthology.orchestra.run.smithi114.stdout: "replica_state": { 2024-04-23T10:32:04.643 INFO:teuthology.orchestra.run.smithi114.stdout: "authority": [ 2024-04-23T10:32:04.643 INFO:teuthology.orchestra.run.smithi114.stdout: 2, 2024-04-23T10:32:04.643 INFO:teuthology.orchestra.run.smithi114.stdout: -2 2024-04-23T10:32:04.643 INFO:teuthology.orchestra.run.smithi114.stdout: ], 2024-04-23T10:32:04.643 INFO:teuthology.orchestra.run.smithi114.stdout: "replica_nonce": 1 2024-04-23T10:32:04.643 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-23T10:32:04.643 INFO:teuthology.orchestra.run.smithi114.stdout: "auth_pins": 0, 2024-04-23T10:32:04.644 INFO:teuthology.orchestra.run.smithi114.stdout: "is_frozen": false, 2024-04-23T10:32:04.644 INFO:teuthology.orchestra.run.smithi114.stdout: "is_freezing": false, 2024-04-23T10:32:04.644 INFO:teuthology.orchestra.run.smithi114.stdout: "pins": { 2024-04-23T10:32:04.644 INFO:teuthology.orchestra.run.smithi114.stdout: "request": 1, 2024-04-23T10:32:04.644 INFO:teuthology.orchestra.run.smithi114.stdout: "lock": 1, 2024-04-23T10:32:04.644 INFO:teuthology.orchestra.run.smithi114.stdout: "caps": 0, 2024-04-23T10:32:04.644 INFO:teuthology.orchestra.run.smithi114.stdout: "exportingcaps": 0, 2024-04-23T10:32:04.644 INFO:teuthology.orchestra.run.smithi114.stdout: "dirtyparent": 0, 2024-04-23T10:32:04.644 INFO:teuthology.orchestra.run.smithi114.stdout: "dirty": 0, 2024-04-23T10:32:04.644 INFO:teuthology.orchestra.run.smithi114.stdout: "authpin": 0, 2024-04-23T10:32:04.644 INFO:teuthology.orchestra.run.smithi114.stdout: "tempexporting": 0 2024-04-23T10:32:04.644 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-23T10:32:04.644 INFO:teuthology.orchestra.run.smithi114.stdout: "nref": 2 2024-04-23T10:32:04.644 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-23T10:32:04.644 INFO:teuthology.orchestra.run.smithi114.stdout: "object_string": "[inode 0x1000000021a [7,head] /volumes/qa/sv_1/b206794e-7622-4ab8-8943-f3ae4e65b7a3/client.0/tmp/clients/client0/~dmtmp/COREL/MAILBOX2.CPT rep@2.1 v7246 s=260096 n(v0 rc2024-04-23T10:27:52.023396+0000 b260096 1=1+0)/n(v0 1=1+0) (iquiesce lock x=1 by request(mds.0:192 nref=3)) | request=1 lock=1 caps=0 exportingcaps=0 dirtyparent=0 dirty=0 authpin=0 tempexporting=0 0x5571cf561600]", 2024-04-23T10:32:04.645 INFO:teuthology.orchestra.run.smithi114.stdout: "lock": { 2024-04-23T10:32:04.645 INFO:teuthology.orchestra.run.smithi114.stdout: "gather_set": [], 2024-04-23T10:32:04.645 INFO:teuthology.orchestra.run.smithi114.stdout: "state": "lock", 2024-04-23T10:32:04.645 INFO:teuthology.orchestra.run.smithi114.stdout: "type": "iquiesce", 2024-04-23T10:32:04.645 INFO:teuthology.orchestra.run.smithi114.stdout: "is_leased": false, 2024-04-23T10:32:04.645 INFO:teuthology.orchestra.run.smithi114.stdout: "num_rdlocks": 0, 2024-04-23T10:32:04.645 INFO:teuthology.orchestra.run.smithi114.stdout: "num_wrlocks": 0, 2024-04-23T10:32:04.645 INFO:teuthology.orchestra.run.smithi114.stdout: "num_xlocks": 1, 2024-04-23T10:32:04.645 INFO:teuthology.orchestra.run.smithi114.stdout: "xlock_by": { 2024-04-23T10:32:04.645 INFO:teuthology.orchestra.run.smithi114.stdout: "reqid": { 2024-04-23T10:32:04.645 INFO:teuthology.orchestra.run.smithi114.stdout: "entity": { 2024-04-23T10:32:04.645 INFO:teuthology.orchestra.run.smithi114.stdout: "type": "mds", 2024-04-23T10:32:04.645 INFO:teuthology.orchestra.run.smithi114.stdout: "num": 0 2024-04-23T10:32:04.645 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-23T10:32:04.646 INFO:teuthology.orchestra.run.smithi114.stdout: "tid": 192 2024-04-23T10:32:04.646 INFO:teuthology.orchestra.run.smithi114.stdout: } 2024-04-23T10:32:04.646 INFO:teuthology.orchestra.run.smithi114.stdout: } 2024-04-23T10:32:04.646 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-23T10:32:04.646 INFO:teuthology.orchestra.run.smithi114.stdout: "flags": 4, 2024-04-23T10:32:04.646 INFO:teuthology.orchestra.run.smithi114.stdout: "wrlock_target": -1 2024-04-23T10:32:04.646 INFO:teuthology.orchestra.run.smithi114.stdout: } 2024-04-23T10:32:04.646 INFO:teuthology.orchestra.run.smithi114.stdout: ] 2024-04-23T10:32:04.646 INFO:teuthology.orchestra.run.smithi114.stdout: } 2024-04-23T10:32:04.646 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-23T10:32:04.646 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-04-23T10:32:04.646 INFO:teuthology.orchestra.run.smithi114.stdout: "description": "peer_request:mds.2:108", 2024-04-23T10:32:04.646 INFO:teuthology.orchestra.run.smithi114.stdout: "initiated_at": "2024-04-23T10:31:21.446235+0000", 2024-04-23T10:32:04.646 INFO:teuthology.orchestra.run.smithi114.stdout: "age": 42.438153837999998, 2024-04-23T10:32:04.647 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 42.439831419999997, 2024-04-23T10:32:04.647 INFO:teuthology.orchestra.run.smithi114.stdout: "continuous": false, 2024-04-23T10:32:04.647 INFO:teuthology.orchestra.run.smithi114.stdout: "type_data": { 2024-04-23T10:32:04.647 INFO:teuthology.orchestra.run.smithi114.stdout: "result": -2147483648, 2024-04-23T10:32:04.647 INFO:teuthology.orchestra.run.smithi114.stdout: "flag_point": "dispatched", 2024-04-23T10:32:04.647 INFO:teuthology.orchestra.run.smithi114.stdout: "reqid": { 2024-04-23T10:32:04.647 INFO:teuthology.orchestra.run.smithi114.stdout: "entity": { 2024-04-23T10:32:04.647 INFO:teuthology.orchestra.run.smithi114.stdout: "type": "mds", 2024-04-23T10:32:04.647 INFO:teuthology.orchestra.run.smithi114.stdout: "num": 2 2024-04-23T10:32:04.647 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-23T10:32:04.647 INFO:teuthology.orchestra.run.smithi114.stdout: "tid": 108 2024-04-23T10:32:04.647 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-23T10:32:04.647 INFO:teuthology.orchestra.run.smithi114.stdout: "op_type": "peer_request", 2024-04-23T10:32:04.647 INFO:teuthology.orchestra.run.smithi114.stdout: "leader_info": { 2024-04-23T10:32:04.647 INFO:teuthology.orchestra.run.smithi114.stdout: "leader": "2" 2024-04-23T10:32:04.647 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-23T10:32:04.648 INFO:teuthology.orchestra.run.smithi114.stdout: "events": [ 2024-04-23T10:32:04.648 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-04-23T10:32:04.648 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-04-23T10:31:21.446235+0000", 2024-04-23T10:32:04.648 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "initiated" 2024-04-23T10:32:04.648 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-23T10:32:04.648 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-04-23T10:32:04.648 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-04-23T10:31:21.446235+0000", 2024-04-23T10:32:04.648 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "header_read" 2024-04-23T10:32:04.648 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-23T10:32:04.648 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-04-23T10:32:04.648 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-04-23T10:31:21.446236+0000", 2024-04-23T10:32:04.648 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "throttled" 2024-04-23T10:32:04.648 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-23T10:32:04.648 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-04-23T10:32:04.649 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-04-23T10:31:21.446243+0000", 2024-04-23T10:32:04.649 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "all_read" 2024-04-23T10:32:04.649 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-23T10:32:04.649 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-04-23T10:32:04.649 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-04-23T10:31:21.447104+0000", 2024-04-23T10:32:04.649 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "dispatched" 2024-04-23T10:32:04.649 INFO:teuthology.orchestra.run.smithi114.stdout: } 2024-04-23T10:32:04.649 INFO:teuthology.orchestra.run.smithi114.stdout: ], 2024-04-23T10:32:04.649 INFO:teuthology.orchestra.run.smithi114.stdout: "locks": [] 2024-04-23T10:32:04.649 INFO:teuthology.orchestra.run.smithi114.stdout: } 2024-04-23T10:32:04.649 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-23T10:32:04.649 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-04-23T10:32:04.649 INFO:teuthology.orchestra.run.smithi114.stdout: "description": "internal op quiesce_inode:mds.0:168 fp=#0x10000000208", 2024-04-23T10:32:04.649 INFO:teuthology.orchestra.run.smithi114.stdout: "initiated_at": "2024-04-23T10:31:21.459861+0000", 2024-04-23T10:32:04.649 INFO:teuthology.orchestra.run.smithi114.stdout: "age": 42.424527853999997, 2024-04-23T10:32:04.649 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 42.426226503999999, 2024-04-23T10:32:04.650 INFO:teuthology.orchestra.run.smithi114.stdout: "continuous": true, 2024-04-23T10:32:04.650 INFO:teuthology.orchestra.run.smithi114.stdout: "type_data": { 2024-04-23T10:32:04.650 INFO:teuthology.orchestra.run.smithi114.stdout: "result": -2147483648, 2024-04-23T10:32:04.650 INFO:teuthology.orchestra.run.smithi114.stdout: "flag_point": "quiesce complete", 2024-04-23T10:32:04.650 INFO:teuthology.orchestra.run.smithi114.stdout: "reqid": { 2024-04-23T10:32:04.650 INFO:teuthology.orchestra.run.smithi114.stdout: "entity": { 2024-04-23T10:32:04.650 INFO:teuthology.orchestra.run.smithi114.stdout: "type": "mds", 2024-04-23T10:32:04.650 INFO:teuthology.orchestra.run.smithi114.stdout: "num": 0 2024-04-23T10:32:04.650 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-23T10:32:04.650 INFO:teuthology.orchestra.run.smithi114.stdout: "tid": 168 2024-04-23T10:32:04.650 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-23T10:32:04.650 INFO:teuthology.orchestra.run.smithi114.stdout: "op_type": "internal_op", 2024-04-23T10:32:04.650 INFO:teuthology.orchestra.run.smithi114.stdout: "internal_op": 5385, 2024-04-23T10:32:04.650 INFO:teuthology.orchestra.run.smithi114.stdout: "op_name": "quiesce_inode", 2024-04-23T10:32:04.650 INFO:teuthology.orchestra.run.smithi114.stdout: "events": [ 2024-04-23T10:32:04.651 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-04-23T10:32:04.651 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-04-23T10:31:21.459861+0000", 2024-04-23T10:32:04.651 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "initiated" 2024-04-23T10:32:04.651 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-23T10:32:04.651 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-04-23T10:32:04.651 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-04-23T10:31:21.459861+0000", 2024-04-23T10:32:04.651 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "header_read" 2024-04-23T10:32:04.651 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-23T10:32:04.651 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-04-23T10:32:04.651 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-04-23T10:31:21.459861+0000", 2024-04-23T10:32:04.651 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "throttled" 2024-04-23T10:32:04.651 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-23T10:32:04.651 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-04-23T10:32:04.651 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-04-23T10:31:21.459861+0000", 2024-04-23T10:32:04.651 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "all_read" 2024-04-23T10:32:04.651 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-23T10:32:04.652 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-04-23T10:32:04.652 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-04-23T10:31:21.459861+0000", 2024-04-23T10:32:04.652 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "dispatched" 2024-04-23T10:32:04.652 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-23T10:32:04.652 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-04-23T10:32:04.652 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-04-23T10:31:21.460889+0000", 2024-04-23T10:32:04.652 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "failed to xlock, waiting" 2024-04-23T10:32:04.652 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-23T10:32:04.652 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-04-23T10:32:04.652 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-04-23T10:31:21.468246+0000", 2024-04-23T10:32:04.652 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "acquired locks" 2024-04-23T10:32:04.652 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-23T10:32:04.652 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-04-23T10:32:04.652 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-04-23T10:31:21.497176+0000", 2024-04-23T10:32:04.653 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "quiesce complete" 2024-04-23T10:32:04.653 INFO:teuthology.orchestra.run.smithi114.stdout: } 2024-04-23T10:32:04.653 INFO:teuthology.orchestra.run.smithi114.stdout: ], 2024-04-23T10:32:04.653 INFO:teuthology.orchestra.run.smithi114.stdout: "locks": [ 2024-04-23T10:32:04.653 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-04-23T10:32:04.653 INFO:teuthology.orchestra.run.smithi114.stdout: "object": { 2024-04-23T10:32:04.653 INFO:teuthology.orchestra.run.smithi114.stdout: "is_auth": true, 2024-04-23T10:32:04.653 INFO:teuthology.orchestra.run.smithi114.stdout: "auth_state": { 2024-04-23T10:32:04.653 INFO:teuthology.orchestra.run.smithi114.stdout: "replicas": { 2024-04-23T10:32:04.653 INFO:teuthology.orchestra.run.smithi114.stdout: "1": 1, 2024-04-23T10:32:04.653 INFO:teuthology.orchestra.run.smithi114.stdout: "2": 1 2024-04-23T10:32:04.653 INFO:teuthology.orchestra.run.smithi114.stdout: } 2024-04-23T10:32:04.654 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-23T10:32:04.654 INFO:teuthology.orchestra.run.smithi114.stdout: "replica_state": { 2024-04-23T10:32:04.654 INFO:teuthology.orchestra.run.smithi114.stdout: "authority": [ 2024-04-23T10:32:04.654 INFO:teuthology.orchestra.run.smithi114.stdout: 0, 2024-04-23T10:32:04.654 INFO:teuthology.orchestra.run.smithi114.stdout: -2 2024-04-23T10:32:04.654 INFO:teuthology.orchestra.run.smithi114.stdout: ], 2024-04-23T10:32:04.654 INFO:teuthology.orchestra.run.smithi114.stdout: "replica_nonce": 0 2024-04-23T10:32:04.654 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-23T10:32:04.654 INFO:teuthology.orchestra.run.smithi114.stdout: "auth_pins": 3, 2024-04-23T10:32:04.654 INFO:teuthology.orchestra.run.smithi114.stdout: "is_frozen": false, 2024-04-23T10:32:04.654 INFO:teuthology.orchestra.run.smithi114.stdout: "is_freezing": false, 2024-04-23T10:32:04.654 INFO:teuthology.orchestra.run.smithi114.stdout: "pins": { 2024-04-23T10:32:04.654 INFO:teuthology.orchestra.run.smithi114.stdout: "request": 1, 2024-04-23T10:32:04.654 INFO:teuthology.orchestra.run.smithi114.stdout: "lock": 1, 2024-04-23T10:32:04.655 INFO:teuthology.orchestra.run.smithi114.stdout: "dirfrag": 1, 2024-04-23T10:32:04.655 INFO:teuthology.orchestra.run.smithi114.stdout: "caps": 1, 2024-04-23T10:32:04.655 INFO:teuthology.orchestra.run.smithi114.stdout: "dirtyrstat": 0, 2024-04-23T10:32:04.655 INFO:teuthology.orchestra.run.smithi114.stdout: "dirtyparent": 0, 2024-04-23T10:32:04.655 INFO:teuthology.orchestra.run.smithi114.stdout: "replicated": 1, 2024-04-23T10:32:04.655 INFO:teuthology.orchestra.run.smithi114.stdout: "dirty": 0, 2024-04-23T10:32:04.655 INFO:teuthology.orchestra.run.smithi114.stdout: "waiter": 0, 2024-04-23T10:32:04.655 INFO:teuthology.orchestra.run.smithi114.stdout: "authpin": 1 2024-04-23T10:32:04.655 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-23T10:32:04.655 INFO:teuthology.orchestra.run.smithi114.stdout: "nref": 6 2024-04-23T10:32:04.655 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-23T10:32:04.655 INFO:teuthology.orchestra.run.smithi114.stdout: "object_string": "[inode 0x10000000208 [...7,head] /volumes/qa/sv_1/b206794e-7622-4ab8-8943-f3ae4e65b7a3/client.0/tmp/clients/client0/~dmtmp/WORD/ auth{1=1,2=1} v45541 ap=3 f(v0 m2024-04-23T10:31:17.675718+0000 3=3+0) n(v0 rc2024-04-23T10:31:17.675718+0000 b780288 4=3+1) old_inodes=4 (inest mix) (iquiesce lock x=1 by request(mds.0:168 nref=3)) caps={15312=pAsLsXsFs/pAsLsXsFsxcral@6600} mcw={2=AsLsXsFs} | request=1 lock=1 dirfrag=1 caps=1 dirtyrstat=0 dirtyparent=0 replicated=1 dirty=0 waiter=0 authpin=1 0x5571cf343180]", 2024-04-23T10:32:04.655 INFO:teuthology.orchestra.run.smithi114.stdout: "lock": { 2024-04-23T10:32:04.655 INFO:teuthology.orchestra.run.smithi114.stdout: "gather_set": [], 2024-04-23T10:32:04.656 INFO:teuthology.orchestra.run.smithi114.stdout: "state": "lock", 2024-04-23T10:32:04.656 INFO:teuthology.orchestra.run.smithi114.stdout: "type": "iquiesce", 2024-04-23T10:32:04.656 INFO:teuthology.orchestra.run.smithi114.stdout: "is_leased": false, 2024-04-23T10:32:04.656 INFO:teuthology.orchestra.run.smithi114.stdout: "num_rdlocks": 0, 2024-04-23T10:32:04.656 INFO:teuthology.orchestra.run.smithi114.stdout: "num_wrlocks": 0, 2024-04-23T10:32:04.656 INFO:teuthology.orchestra.run.smithi114.stdout: "num_xlocks": 1, 2024-04-23T10:32:04.656 INFO:teuthology.orchestra.run.smithi114.stdout: "xlock_by": { 2024-04-23T10:32:04.656 INFO:teuthology.orchestra.run.smithi114.stdout: "reqid": { 2024-04-23T10:32:04.656 INFO:teuthology.orchestra.run.smithi114.stdout: "entity": { 2024-04-23T10:32:04.656 INFO:teuthology.orchestra.run.smithi114.stdout: "type": "mds", 2024-04-23T10:32:04.656 INFO:teuthology.orchestra.run.smithi114.stdout: "num": 0 2024-04-23T10:32:04.656 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-23T10:32:04.656 INFO:teuthology.orchestra.run.smithi114.stdout: "tid": 168 2024-04-23T10:32:04.656 INFO:teuthology.orchestra.run.smithi114.stdout: } 2024-04-23T10:32:04.656 INFO:teuthology.orchestra.run.smithi114.stdout: } 2024-04-23T10:32:04.657 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-23T10:32:04.657 INFO:teuthology.orchestra.run.smithi114.stdout: "flags": 4, 2024-04-23T10:32:04.657 INFO:teuthology.orchestra.run.smithi114.stdout: "wrlock_target": -1 2024-04-23T10:32:04.657 INFO:teuthology.orchestra.run.smithi114.stdout: } 2024-04-23T10:32:04.657 INFO:teuthology.orchestra.run.smithi114.stdout: ] 2024-04-23T10:32:04.657 INFO:teuthology.orchestra.run.smithi114.stdout: } 2024-04-23T10:32:04.657 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-23T10:32:04.657 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-04-23T10:32:04.657 INFO:teuthology.orchestra.run.smithi114.stdout: "description": "internal op quiesce_inode:mds.0:193 fp=#0x10000000220", 2024-04-23T10:32:04.657 INFO:teuthology.orchestra.run.smithi114.stdout: "initiated_at": "2024-04-23T10:31:21.483415+0000", 2024-04-23T10:32:04.657 INFO:teuthology.orchestra.run.smithi114.stdout: "age": 42.400974654000002, 2024-04-23T10:32:04.657 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 42.402740053999999, 2024-04-23T10:32:04.657 INFO:teuthology.orchestra.run.smithi114.stdout: "continuous": true, 2024-04-23T10:32:04.657 INFO:teuthology.orchestra.run.smithi114.stdout: "type_data": { 2024-04-23T10:32:04.657 INFO:teuthology.orchestra.run.smithi114.stdout: "result": -2147483648, 2024-04-23T10:32:04.658 INFO:teuthology.orchestra.run.smithi114.stdout: "flag_point": "quiesce complete for non-auth inode", 2024-04-23T10:32:04.658 INFO:teuthology.orchestra.run.smithi114.stdout: "reqid": { 2024-04-23T10:32:04.658 INFO:teuthology.orchestra.run.smithi114.stdout: "entity": { 2024-04-23T10:32:04.658 INFO:teuthology.orchestra.run.smithi114.stdout: "type": "mds", 2024-04-23T10:32:04.658 INFO:teuthology.orchestra.run.smithi114.stdout: "num": 0 2024-04-23T10:32:04.658 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-23T10:32:04.658 INFO:teuthology.orchestra.run.smithi114.stdout: "tid": 193 2024-04-23T10:32:04.658 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-23T10:32:04.658 INFO:teuthology.orchestra.run.smithi114.stdout: "op_type": "internal_op", 2024-04-23T10:32:04.658 INFO:teuthology.orchestra.run.smithi114.stdout: "internal_op": 5385, 2024-04-23T10:32:04.658 INFO:teuthology.orchestra.run.smithi114.stdout: "op_name": "quiesce_inode", 2024-04-23T10:32:04.658 INFO:teuthology.orchestra.run.smithi114.stdout: "events": [ 2024-04-23T10:32:04.658 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-04-23T10:32:04.658 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-04-23T10:31:21.483415+0000", 2024-04-23T10:32:04.659 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "initiated" 2024-04-23T10:32:04.659 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-23T10:32:04.659 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-04-23T10:32:04.659 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-04-23T10:31:21.483415+0000", 2024-04-23T10:32:04.659 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "header_read" 2024-04-23T10:32:04.659 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-23T10:32:04.659 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-04-23T10:32:04.659 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-04-23T10:31:21.483415+0000", 2024-04-23T10:32:04.659 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "throttled" 2024-04-23T10:32:04.659 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-23T10:32:04.659 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-04-23T10:32:04.659 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-04-23T10:31:21.483415+0000", 2024-04-23T10:32:04.659 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "all_read" 2024-04-23T10:32:04.659 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-23T10:32:04.660 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-04-23T10:32:04.660 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-04-23T10:31:21.483415+0000", 2024-04-23T10:32:04.660 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "dispatched" 2024-04-23T10:32:04.660 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-23T10:32:04.660 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-04-23T10:32:04.660 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-04-23T10:31:21.483737+0000", 2024-04-23T10:32:04.660 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "requesting remote authpins" 2024-04-23T10:32:04.660 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-23T10:32:04.660 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-04-23T10:32:04.660 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-04-23T10:31:21.507336+0000", 2024-04-23T10:32:04.660 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "acquired locks" 2024-04-23T10:32:04.660 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-23T10:32:04.660 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-04-23T10:32:04.660 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-04-23T10:31:21.507347+0000", 2024-04-23T10:32:04.660 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "quiesce complete for non-auth inode" 2024-04-23T10:32:04.660 INFO:teuthology.orchestra.run.smithi114.stdout: } 2024-04-23T10:32:04.661 INFO:teuthology.orchestra.run.smithi114.stdout: ], 2024-04-23T10:32:04.661 INFO:teuthology.orchestra.run.smithi114.stdout: "locks": [ 2024-04-23T10:32:04.661 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-04-23T10:32:04.661 INFO:teuthology.orchestra.run.smithi114.stdout: "object": { 2024-04-23T10:32:04.661 INFO:teuthology.orchestra.run.smithi114.stdout: "is_auth": false, 2024-04-23T10:32:04.661 INFO:teuthology.orchestra.run.smithi114.stdout: "auth_state": { 2024-04-23T10:32:04.661 INFO:teuthology.orchestra.run.smithi114.stdout: "replicas": {} 2024-04-23T10:32:04.661 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-23T10:32:04.661 INFO:teuthology.orchestra.run.smithi114.stdout: "replica_state": { 2024-04-23T10:32:04.661 INFO:teuthology.orchestra.run.smithi114.stdout: "authority": [ 2024-04-23T10:32:04.661 INFO:teuthology.orchestra.run.smithi114.stdout: 2, 2024-04-23T10:32:04.661 INFO:teuthology.orchestra.run.smithi114.stdout: -2 2024-04-23T10:32:04.661 INFO:teuthology.orchestra.run.smithi114.stdout: ], 2024-04-23T10:32:04.661 INFO:teuthology.orchestra.run.smithi114.stdout: "replica_nonce": 1 2024-04-23T10:32:04.662 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-23T10:32:04.662 INFO:teuthology.orchestra.run.smithi114.stdout: "auth_pins": 0, 2024-04-23T10:32:04.662 INFO:teuthology.orchestra.run.smithi114.stdout: "is_frozen": false, 2024-04-23T10:32:04.662 INFO:teuthology.orchestra.run.smithi114.stdout: "is_freezing": false, 2024-04-23T10:32:04.662 INFO:teuthology.orchestra.run.smithi114.stdout: "pins": { 2024-04-23T10:32:04.662 INFO:teuthology.orchestra.run.smithi114.stdout: "request": 1, 2024-04-23T10:32:04.662 INFO:teuthology.orchestra.run.smithi114.stdout: "lock": 1, 2024-04-23T10:32:04.662 INFO:teuthology.orchestra.run.smithi114.stdout: "caps": 0, 2024-04-23T10:32:04.662 INFO:teuthology.orchestra.run.smithi114.stdout: "exportingcaps": 0, 2024-04-23T10:32:04.662 INFO:teuthology.orchestra.run.smithi114.stdout: "dirtyparent": 0, 2024-04-23T10:32:04.662 INFO:teuthology.orchestra.run.smithi114.stdout: "dirty": 0, 2024-04-23T10:32:04.662 INFO:teuthology.orchestra.run.smithi114.stdout: "authpin": 0, 2024-04-23T10:32:04.662 INFO:teuthology.orchestra.run.smithi114.stdout: "tempexporting": 0 2024-04-23T10:32:04.662 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-23T10:32:04.663 INFO:teuthology.orchestra.run.smithi114.stdout: "nref": 2 2024-04-23T10:32:04.663 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-23T10:32:04.663 INFO:teuthology.orchestra.run.smithi114.stdout: "object_string": "[inode 0x10000000220 [7,head] /volumes/qa/sv_1/b206794e-7622-4ab8-8943-f3ae4e65b7a3/client.0/tmp/clients/client0/~dmtmp/COREL/CORELDRW.CDT rep@2.1 v7242 s=260096 n(v0 rc2024-04-23T10:27:52.054395+0000 b260096 1=1+0)/n(v0 1=1+0) (iquiesce lock x=1 by request(mds.0:193 nref=3)) | request=1 lock=1 caps=0 exportingcaps=0 dirtyparent=0 dirty=0 authpin=0 tempexporting=0 0x5571cf563700]", 2024-04-23T10:32:04.663 INFO:teuthology.orchestra.run.smithi114.stdout: "lock": { 2024-04-23T10:32:04.663 INFO:teuthology.orchestra.run.smithi114.stdout: "gather_set": [], 2024-04-23T10:32:04.663 INFO:teuthology.orchestra.run.smithi114.stdout: "state": "lock", 2024-04-23T10:32:04.663 INFO:teuthology.orchestra.run.smithi114.stdout: "type": "iquiesce", 2024-04-23T10:32:04.663 INFO:teuthology.orchestra.run.smithi114.stdout: "is_leased": false, 2024-04-23T10:32:04.663 INFO:teuthology.orchestra.run.smithi114.stdout: "num_rdlocks": 0, 2024-04-23T10:32:04.663 INFO:teuthology.orchestra.run.smithi114.stdout: "num_wrlocks": 0, 2024-04-23T10:32:04.663 INFO:teuthology.orchestra.run.smithi114.stdout: "num_xlocks": 1, 2024-04-23T10:32:04.663 INFO:teuthology.orchestra.run.smithi114.stdout: "xlock_by": { 2024-04-23T10:32:04.663 INFO:teuthology.orchestra.run.smithi114.stdout: "reqid": { 2024-04-23T10:32:04.663 INFO:teuthology.orchestra.run.smithi114.stdout: "entity": { 2024-04-23T10:32:04.663 INFO:teuthology.orchestra.run.smithi114.stdout: "type": "mds", 2024-04-23T10:32:04.664 INFO:teuthology.orchestra.run.smithi114.stdout: "num": 0 2024-04-23T10:32:04.664 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-23T10:32:04.664 INFO:teuthology.orchestra.run.smithi114.stdout: "tid": 193 2024-04-23T10:32:04.664 INFO:teuthology.orchestra.run.smithi114.stdout: } 2024-04-23T10:32:04.664 INFO:teuthology.orchestra.run.smithi114.stdout: } 2024-04-23T10:32:04.664 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-23T10:32:04.664 INFO:teuthology.orchestra.run.smithi114.stdout: "flags": 4, 2024-04-23T10:32:04.664 INFO:teuthology.orchestra.run.smithi114.stdout: "wrlock_target": -1 2024-04-23T10:32:04.664 INFO:teuthology.orchestra.run.smithi114.stdout: } 2024-04-23T10:32:04.664 INFO:teuthology.orchestra.run.smithi114.stdout: ] 2024-04-23T10:32:04.664 INFO:teuthology.orchestra.run.smithi114.stdout: } 2024-04-23T10:32:04.664 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-23T10:32:04.664 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-04-23T10:32:04.664 INFO:teuthology.orchestra.run.smithi114.stdout: "description": "internal op quiesce_inode:mds.0:152 fp=#0x10000000001", 2024-04-23T10:32:04.665 INFO:teuthology.orchestra.run.smithi114.stdout: "initiated_at": "2024-04-23T10:31:21.447800+0000", 2024-04-23T10:32:04.665 INFO:teuthology.orchestra.run.smithi114.stdout: "age": 42.436588751000002, 2024-04-23T10:32:04.665 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 42.438416095999997, 2024-04-23T10:32:04.665 INFO:teuthology.orchestra.run.smithi114.stdout: "continuous": true, 2024-04-23T10:32:04.665 INFO:teuthology.orchestra.run.smithi114.stdout: "type_data": { 2024-04-23T10:32:04.665 INFO:teuthology.orchestra.run.smithi114.stdout: "result": -2147483648, 2024-04-23T10:32:04.665 INFO:teuthology.orchestra.run.smithi114.stdout: "flag_point": "quiesce complete", 2024-04-23T10:32:04.665 INFO:teuthology.orchestra.run.smithi114.stdout: "reqid": { 2024-04-23T10:32:04.665 INFO:teuthology.orchestra.run.smithi114.stdout: "entity": { 2024-04-23T10:32:04.665 INFO:teuthology.orchestra.run.smithi114.stdout: "type": "mds", 2024-04-23T10:32:04.665 INFO:teuthology.orchestra.run.smithi114.stdout: "num": 0 2024-04-23T10:32:04.665 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-23T10:32:04.665 INFO:teuthology.orchestra.run.smithi114.stdout: "tid": 152 2024-04-23T10:32:04.665 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-23T10:32:04.665 INFO:teuthology.orchestra.run.smithi114.stdout: "op_type": "internal_op", 2024-04-23T10:32:04.666 INFO:teuthology.orchestra.run.smithi114.stdout: "internal_op": 5385, 2024-04-23T10:32:04.666 INFO:teuthology.orchestra.run.smithi114.stdout: "op_name": "quiesce_inode", 2024-04-23T10:32:04.666 INFO:teuthology.orchestra.run.smithi114.stdout: "events": [ 2024-04-23T10:32:04.666 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-04-23T10:32:04.666 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-04-23T10:31:21.447800+0000", 2024-04-23T10:32:04.666 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "initiated" 2024-04-23T10:32:04.666 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-23T10:32:04.666 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-04-23T10:32:04.666 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-04-23T10:31:21.447800+0000", 2024-04-23T10:32:04.666 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "header_read" 2024-04-23T10:32:04.666 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-23T10:32:04.666 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-04-23T10:32:04.666 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-04-23T10:31:21.447800+0000", 2024-04-23T10:32:04.666 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "throttled" 2024-04-23T10:32:04.666 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-23T10:32:04.666 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-04-23T10:32:04.667 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-04-23T10:31:21.447800+0000", 2024-04-23T10:32:04.667 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "all_read" 2024-04-23T10:32:04.667 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-23T10:32:04.667 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-04-23T10:32:04.667 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-04-23T10:31:21.447800+0000", 2024-04-23T10:32:04.667 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "dispatched" 2024-04-23T10:32:04.667 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-23T10:32:04.667 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-04-23T10:32:04.667 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-04-23T10:31:21.447978+0000", 2024-04-23T10:32:04.667 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "failed to xlock, waiting" 2024-04-23T10:32:04.667 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-23T10:32:04.667 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-04-23T10:32:04.667 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-04-23T10:31:21.448441+0000", 2024-04-23T10:32:04.667 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "acquired locks" 2024-04-23T10:32:04.667 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-23T10:32:04.668 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-04-23T10:32:04.668 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-04-23T10:31:21.510014+0000", 2024-04-23T10:32:04.668 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "quiesce complete" 2024-04-23T10:32:04.668 INFO:teuthology.orchestra.run.smithi114.stdout: } 2024-04-23T10:32:04.668 INFO:teuthology.orchestra.run.smithi114.stdout: ], 2024-04-23T10:32:04.668 INFO:teuthology.orchestra.run.smithi114.stdout: "locks": [ 2024-04-23T10:32:04.668 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-04-23T10:32:04.668 INFO:teuthology.orchestra.run.smithi114.stdout: "object": { 2024-04-23T10:32:04.668 INFO:teuthology.orchestra.run.smithi114.stdout: "is_auth": true, 2024-04-23T10:32:04.668 INFO:teuthology.orchestra.run.smithi114.stdout: "auth_state": { 2024-04-23T10:32:04.668 INFO:teuthology.orchestra.run.smithi114.stdout: "replicas": { 2024-04-23T10:32:04.668 INFO:teuthology.orchestra.run.smithi114.stdout: "1": 1, 2024-04-23T10:32:04.668 INFO:teuthology.orchestra.run.smithi114.stdout: "2": 1 2024-04-23T10:32:04.668 INFO:teuthology.orchestra.run.smithi114.stdout: } 2024-04-23T10:32:04.669 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-23T10:32:04.669 INFO:teuthology.orchestra.run.smithi114.stdout: "replica_state": { 2024-04-23T10:32:04.669 INFO:teuthology.orchestra.run.smithi114.stdout: "authority": [ 2024-04-23T10:32:04.669 INFO:teuthology.orchestra.run.smithi114.stdout: 0, 2024-04-23T10:32:04.669 INFO:teuthology.orchestra.run.smithi114.stdout: -2 2024-04-23T10:32:04.669 INFO:teuthology.orchestra.run.smithi114.stdout: ], 2024-04-23T10:32:04.669 INFO:teuthology.orchestra.run.smithi114.stdout: "replica_nonce": 0 2024-04-23T10:32:04.669 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-23T10:32:04.669 INFO:teuthology.orchestra.run.smithi114.stdout: "auth_pins": 3, 2024-04-23T10:32:04.669 INFO:teuthology.orchestra.run.smithi114.stdout: "is_frozen": false, 2024-04-23T10:32:04.669 INFO:teuthology.orchestra.run.smithi114.stdout: "is_freezing": false, 2024-04-23T10:32:04.669 INFO:teuthology.orchestra.run.smithi114.stdout: "pins": { 2024-04-23T10:32:04.669 INFO:teuthology.orchestra.run.smithi114.stdout: "dirtyscattered": 1, 2024-04-23T10:32:04.670 INFO:teuthology.orchestra.run.smithi114.stdout: "request": 1, 2024-04-23T10:32:04.670 INFO:teuthology.orchestra.run.smithi114.stdout: "lock": 1, 2024-04-23T10:32:04.670 INFO:teuthology.orchestra.run.smithi114.stdout: "dirfrag": 1, 2024-04-23T10:32:04.670 INFO:teuthology.orchestra.run.smithi114.stdout: "caps": 0, 2024-04-23T10:32:04.670 INFO:teuthology.orchestra.run.smithi114.stdout: "dirtyrstat": 0, 2024-04-23T10:32:04.670 INFO:teuthology.orchestra.run.smithi114.stdout: "dirtyparent": 0, 2024-04-23T10:32:04.670 INFO:teuthology.orchestra.run.smithi114.stdout: "replicated": 1, 2024-04-23T10:32:04.670 INFO:teuthology.orchestra.run.smithi114.stdout: "dirty": 1, 2024-04-23T10:32:04.670 INFO:teuthology.orchestra.run.smithi114.stdout: "waiter": 0, 2024-04-23T10:32:04.670 INFO:teuthology.orchestra.run.smithi114.stdout: "authpin": 1 2024-04-23T10:32:04.670 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-23T10:32:04.670 INFO:teuthology.orchestra.run.smithi114.stdout: "nref": 7 2024-04-23T10:32:04.670 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-23T10:32:04.670 INFO:teuthology.orchestra.run.smithi114.stdout: "object_string": "[inode 0x10000000001 [...7,head] /volumes/qa/ auth{1=1,2=1} v418 ap=3 f(v0 m2024-04-23T10:24:49.714717+0000 2=0+2) n(v30 rc2024-04-23T10:31:17.764716+0000 b21075308 84=65+19) old_inodes=5 (inest mix dirty) (iquiesce lock x=1 by request(mds.0:152 nref=3)) | dirtyscattered=1 request=1 lock=1 dirfrag=1 caps=0 dirtyrstat=0 dirtyparent=0 replicated=1 dirty=1 waiter=0 authpin=1 0x5571cf18f600]", 2024-04-23T10:32:04.670 INFO:teuthology.orchestra.run.smithi114.stdout: "lock": { 2024-04-23T10:32:04.671 INFO:teuthology.orchestra.run.smithi114.stdout: "gather_set": [], 2024-04-23T10:32:04.671 INFO:teuthology.orchestra.run.smithi114.stdout: "state": "lock", 2024-04-23T10:32:04.671 INFO:teuthology.orchestra.run.smithi114.stdout: "type": "iquiesce", 2024-04-23T10:32:04.671 INFO:teuthology.orchestra.run.smithi114.stdout: "is_leased": false, 2024-04-23T10:32:04.671 INFO:teuthology.orchestra.run.smithi114.stdout: "num_rdlocks": 0, 2024-04-23T10:32:04.671 INFO:teuthology.orchestra.run.smithi114.stdout: "num_wrlocks": 0, 2024-04-23T10:32:04.671 INFO:teuthology.orchestra.run.smithi114.stdout: "num_xlocks": 1, 2024-04-23T10:32:04.671 INFO:teuthology.orchestra.run.smithi114.stdout: "xlock_by": { 2024-04-23T10:32:04.671 INFO:teuthology.orchestra.run.smithi114.stdout: "reqid": { 2024-04-23T10:32:04.671 INFO:teuthology.orchestra.run.smithi114.stdout: "entity": { 2024-04-23T10:32:04.671 INFO:teuthology.orchestra.run.smithi114.stdout: "type": "mds", 2024-04-23T10:32:04.671 INFO:teuthology.orchestra.run.smithi114.stdout: "num": 0 2024-04-23T10:32:04.671 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-23T10:32:04.671 INFO:teuthology.orchestra.run.smithi114.stdout: "tid": 152 2024-04-23T10:32:04.671 INFO:teuthology.orchestra.run.smithi114.stdout: } 2024-04-23T10:32:04.672 INFO:teuthology.orchestra.run.smithi114.stdout: } 2024-04-23T10:32:04.672 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-23T10:32:04.672 INFO:teuthology.orchestra.run.smithi114.stdout: "flags": 4, 2024-04-23T10:32:04.672 INFO:teuthology.orchestra.run.smithi114.stdout: "wrlock_target": -1 2024-04-23T10:32:04.672 INFO:teuthology.orchestra.run.smithi114.stdout: } 2024-04-23T10:32:04.672 INFO:teuthology.orchestra.run.smithi114.stdout: ] 2024-04-23T10:32:04.672 INFO:teuthology.orchestra.run.smithi114.stdout: } 2024-04-23T10:32:04.672 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-23T10:32:04.672 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-04-23T10:32:04.672 INFO:teuthology.orchestra.run.smithi114.stdout: "description": "internal op quiesce_inode:mds.0:169 fp=#0x10000000240", 2024-04-23T10:32:04.672 INFO:teuthology.orchestra.run.smithi114.stdout: "initiated_at": "2024-04-23T10:31:21.459879+0000", 2024-04-23T10:32:04.672 INFO:teuthology.orchestra.run.smithi114.stdout: "age": 42.424509798000003, 2024-04-23T10:32:04.672 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 42.426401943000002, 2024-04-23T10:32:04.672 INFO:teuthology.orchestra.run.smithi114.stdout: "continuous": true, 2024-04-23T10:32:04.673 INFO:teuthology.orchestra.run.smithi114.stdout: "type_data": { 2024-04-23T10:32:04.673 INFO:teuthology.orchestra.run.smithi114.stdout: "result": -2147483648, 2024-04-23T10:32:04.673 INFO:teuthology.orchestra.run.smithi114.stdout: "flag_point": "quiesce complete", 2024-04-23T10:32:04.673 INFO:teuthology.orchestra.run.smithi114.stdout: "reqid": { 2024-04-23T10:32:04.673 INFO:teuthology.orchestra.run.smithi114.stdout: "entity": { 2024-04-23T10:32:04.673 INFO:teuthology.orchestra.run.smithi114.stdout: "type": "mds", 2024-04-23T10:32:04.673 INFO:teuthology.orchestra.run.smithi114.stdout: "num": 0 2024-04-23T10:32:04.673 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-23T10:32:04.673 INFO:teuthology.orchestra.run.smithi114.stdout: "tid": 169 2024-04-23T10:32:04.673 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-23T10:32:04.673 INFO:teuthology.orchestra.run.smithi114.stdout: "op_type": "internal_op", 2024-04-23T10:32:04.673 INFO:teuthology.orchestra.run.smithi114.stdout: "internal_op": 5385, 2024-04-23T10:32:04.673 INFO:teuthology.orchestra.run.smithi114.stdout: "op_name": "quiesce_inode", 2024-04-23T10:32:04.673 INFO:teuthology.orchestra.run.smithi114.stdout: "events": [ 2024-04-23T10:32:04.673 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-04-23T10:32:04.673 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-04-23T10:31:21.459879+0000", 2024-04-23T10:32:04.674 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "initiated" 2024-04-23T10:32:04.674 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-23T10:32:04.674 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-04-23T10:32:04.674 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-04-23T10:31:21.459879+0000", 2024-04-23T10:32:04.674 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "header_read" 2024-04-23T10:32:04.674 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-23T10:32:04.674 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-04-23T10:32:04.674 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-04-23T10:31:21.459879+0000", 2024-04-23T10:32:04.674 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "throttled" 2024-04-23T10:32:04.674 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-23T10:32:04.674 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-04-23T10:32:04.674 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-04-23T10:31:21.459879+0000", 2024-04-23T10:32:04.674 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "all_read" 2024-04-23T10:32:04.674 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-23T10:32:04.675 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-04-23T10:32:04.675 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-04-23T10:31:21.459879+0000", 2024-04-23T10:32:04.675 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "dispatched" 2024-04-23T10:32:04.675 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-23T10:32:04.675 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-04-23T10:32:04.675 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-04-23T10:31:21.461207+0000", 2024-04-23T10:32:04.675 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "failed to xlock, waiting" 2024-04-23T10:32:04.675 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-23T10:32:04.675 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-04-23T10:32:04.675 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-04-23T10:31:21.469589+0000", 2024-04-23T10:32:04.675 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "acquired locks" 2024-04-23T10:32:04.675 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-23T10:32:04.675 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-04-23T10:32:04.675 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-04-23T10:31:21.509773+0000", 2024-04-23T10:32:04.675 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "quiesce complete" 2024-04-23T10:32:04.676 INFO:teuthology.orchestra.run.smithi114.stdout: } 2024-04-23T10:32:04.676 INFO:teuthology.orchestra.run.smithi114.stdout: ], 2024-04-23T10:32:04.676 INFO:teuthology.orchestra.run.smithi114.stdout: "locks": [ 2024-04-23T10:32:04.676 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-04-23T10:32:04.676 INFO:teuthology.orchestra.run.smithi114.stdout: "object": { 2024-04-23T10:32:04.676 INFO:teuthology.orchestra.run.smithi114.stdout: "is_auth": true, 2024-04-23T10:32:04.676 INFO:teuthology.orchestra.run.smithi114.stdout: "auth_state": { 2024-04-23T10:32:04.676 INFO:teuthology.orchestra.run.smithi114.stdout: "replicas": { 2024-04-23T10:32:04.676 INFO:teuthology.orchestra.run.smithi114.stdout: "1": 1, 2024-04-23T10:32:04.676 INFO:teuthology.orchestra.run.smithi114.stdout: "2": 1 2024-04-23T10:32:04.676 INFO:teuthology.orchestra.run.smithi114.stdout: } 2024-04-23T10:32:04.676 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-23T10:32:04.676 INFO:teuthology.orchestra.run.smithi114.stdout: "replica_state": { 2024-04-23T10:32:04.676 INFO:teuthology.orchestra.run.smithi114.stdout: "authority": [ 2024-04-23T10:32:04.676 INFO:teuthology.orchestra.run.smithi114.stdout: 0, 2024-04-23T10:32:04.676 INFO:teuthology.orchestra.run.smithi114.stdout: -2 2024-04-23T10:32:04.677 INFO:teuthology.orchestra.run.smithi114.stdout: ], 2024-04-23T10:32:04.677 INFO:teuthology.orchestra.run.smithi114.stdout: "replica_nonce": 0 2024-04-23T10:32:04.677 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-23T10:32:04.677 INFO:teuthology.orchestra.run.smithi114.stdout: "auth_pins": 3, 2024-04-23T10:32:04.677 INFO:teuthology.orchestra.run.smithi114.stdout: "is_frozen": false, 2024-04-23T10:32:04.677 INFO:teuthology.orchestra.run.smithi114.stdout: "is_freezing": false, 2024-04-23T10:32:04.677 INFO:teuthology.orchestra.run.smithi114.stdout: "pins": { 2024-04-23T10:32:04.677 INFO:teuthology.orchestra.run.smithi114.stdout: "dirtyscattered": 0, 2024-04-23T10:32:04.677 INFO:teuthology.orchestra.run.smithi114.stdout: "request": 1, 2024-04-23T10:32:04.677 INFO:teuthology.orchestra.run.smithi114.stdout: "lock": 1, 2024-04-23T10:32:04.677 INFO:teuthology.orchestra.run.smithi114.stdout: "importing": 0, 2024-04-23T10:32:04.677 INFO:teuthology.orchestra.run.smithi114.stdout: "dirfrag": 1, 2024-04-23T10:32:04.677 INFO:teuthology.orchestra.run.smithi114.stdout: "caps": 1, 2024-04-23T10:32:04.677 INFO:teuthology.orchestra.run.smithi114.stdout: "dirtyparent": 0, 2024-04-23T10:32:04.677 INFO:teuthology.orchestra.run.smithi114.stdout: "replicated": 1, 2024-04-23T10:32:04.677 INFO:teuthology.orchestra.run.smithi114.stdout: "dirty": 0, 2024-04-23T10:32:04.678 INFO:teuthology.orchestra.run.smithi114.stdout: "waiter": 0, 2024-04-23T10:32:04.678 INFO:teuthology.orchestra.run.smithi114.stdout: "authpin": 1 2024-04-23T10:32:04.678 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-23T10:32:04.678 INFO:teuthology.orchestra.run.smithi114.stdout: "nref": 6 2024-04-23T10:32:04.678 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-23T10:32:04.678 INFO:teuthology.orchestra.run.smithi114.stdout: "object_string": "[inode 0x10000000240 [...7,head] /volumes/qa/sv_1/b206794e-7622-4ab8-8943-f3ae4e65b7a3/client.0/tmp/clients/client0/~dmtmp/SEED/ auth{1=1,2=1} v45576 ap=3 f(v0 m2024-04-23T10:27:52.196392+0000 3=3+0) n(v2 rc2024-04-23T10:31:17.246728+0000 b3244032 4=3+1) old_inodes=4 (inest mix) (iquiesce lock x=1 by request(mds.0:169 nref=3)) caps={15312=pAsLsXsFs/pAsLsXsFsxcral@32} mcw={2=AsLsXsFs} | dirtyscattered=0 request=1 lock=1 importing=0 dirfrag=1 caps=1 dirtyparent=0 replicated=1 dirty=0 waiter=0 authpin=1 0x5571cf56b180]", 2024-04-23T10:32:04.678 INFO:teuthology.orchestra.run.smithi114.stdout: "lock": { 2024-04-23T10:32:04.678 INFO:teuthology.orchestra.run.smithi114.stdout: "gather_set": [], 2024-04-23T10:32:04.678 INFO:teuthology.orchestra.run.smithi114.stdout: "state": "lock", 2024-04-23T10:32:04.678 INFO:teuthology.orchestra.run.smithi114.stdout: "type": "iquiesce", 2024-04-23T10:32:04.678 INFO:teuthology.orchestra.run.smithi114.stdout: "is_leased": false, 2024-04-23T10:32:04.678 INFO:teuthology.orchestra.run.smithi114.stdout: "num_rdlocks": 0, 2024-04-23T10:32:04.678 INFO:teuthology.orchestra.run.smithi114.stdout: "num_wrlocks": 0, 2024-04-23T10:32:04.678 INFO:teuthology.orchestra.run.smithi114.stdout: "num_xlocks": 1, 2024-04-23T10:32:04.679 INFO:teuthology.orchestra.run.smithi114.stdout: "xlock_by": { 2024-04-23T10:32:04.679 INFO:teuthology.orchestra.run.smithi114.stdout: "reqid": { 2024-04-23T10:32:04.679 INFO:teuthology.orchestra.run.smithi114.stdout: "entity": { 2024-04-23T10:32:04.679 INFO:teuthology.orchestra.run.smithi114.stdout: "type": "mds", 2024-04-23T10:32:04.679 INFO:teuthology.orchestra.run.smithi114.stdout: "num": 0 2024-04-23T10:32:04.679 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-23T10:32:04.679 INFO:teuthology.orchestra.run.smithi114.stdout: "tid": 169 2024-04-23T10:32:04.679 INFO:teuthology.orchestra.run.smithi114.stdout: } 2024-04-23T10:32:04.679 INFO:teuthology.orchestra.run.smithi114.stdout: } 2024-04-23T10:32:04.679 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-23T10:32:04.679 INFO:teuthology.orchestra.run.smithi114.stdout: "flags": 4, 2024-04-23T10:32:04.679 INFO:teuthology.orchestra.run.smithi114.stdout: "wrlock_target": -1 2024-04-23T10:32:04.679 INFO:teuthology.orchestra.run.smithi114.stdout: } 2024-04-23T10:32:04.679 INFO:teuthology.orchestra.run.smithi114.stdout: ] 2024-04-23T10:32:04.679 INFO:teuthology.orchestra.run.smithi114.stdout: } 2024-04-23T10:32:04.680 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-23T10:32:04.680 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-04-23T10:32:04.680 INFO:teuthology.orchestra.run.smithi114.stdout: "description": "internal op quiesce_inode:mds.0:194 fp=#0x1000000021e", 2024-04-23T10:32:04.680 INFO:teuthology.orchestra.run.smithi114.stdout: "initiated_at": "2024-04-23T10:31:21.483427+0000", 2024-04-23T10:32:04.680 INFO:teuthology.orchestra.run.smithi114.stdout: "age": 42.400962466000003, 2024-04-23T10:32:04.680 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 42.402994505000002, 2024-04-23T10:32:04.680 INFO:teuthology.orchestra.run.smithi114.stdout: "continuous": true, 2024-04-23T10:32:04.680 INFO:teuthology.orchestra.run.smithi114.stdout: "type_data": { 2024-04-23T10:32:04.680 INFO:teuthology.orchestra.run.smithi114.stdout: "result": -2147483648, 2024-04-23T10:32:04.680 INFO:teuthology.orchestra.run.smithi114.stdout: "flag_point": "quiesce complete for non-auth inode", 2024-04-23T10:32:04.680 INFO:teuthology.orchestra.run.smithi114.stdout: "reqid": { 2024-04-23T10:32:04.680 INFO:teuthology.orchestra.run.smithi114.stdout: "entity": { 2024-04-23T10:32:04.680 INFO:teuthology.orchestra.run.smithi114.stdout: "type": "mds", 2024-04-23T10:32:04.680 INFO:teuthology.orchestra.run.smithi114.stdout: "num": 0 2024-04-23T10:32:04.680 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-23T10:32:04.681 INFO:teuthology.orchestra.run.smithi114.stdout: "tid": 194 2024-04-23T10:32:04.681 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-23T10:32:04.681 INFO:teuthology.orchestra.run.smithi114.stdout: "op_type": "internal_op", 2024-04-23T10:32:04.681 INFO:teuthology.orchestra.run.smithi114.stdout: "internal_op": 5385, 2024-04-23T10:32:04.681 INFO:teuthology.orchestra.run.smithi114.stdout: "op_name": "quiesce_inode", 2024-04-23T10:32:04.681 INFO:teuthology.orchestra.run.smithi114.stdout: "events": [ 2024-04-23T10:32:04.681 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-04-23T10:32:04.681 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-04-23T10:31:21.483427+0000", 2024-04-23T10:32:04.681 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "initiated" 2024-04-23T10:32:04.681 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-23T10:32:04.681 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-04-23T10:32:04.681 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-04-23T10:31:21.483427+0000", 2024-04-23T10:32:04.681 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "header_read" 2024-04-23T10:32:04.681 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-23T10:32:04.681 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-04-23T10:32:04.682 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-04-23T10:31:21.483427+0000", 2024-04-23T10:32:04.682 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "throttled" 2024-04-23T10:32:04.682 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-23T10:32:04.682 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-04-23T10:32:04.682 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-04-23T10:31:21.483427+0000", 2024-04-23T10:32:04.682 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "all_read" 2024-04-23T10:32:04.682 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-23T10:32:04.682 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-04-23T10:32:04.682 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-04-23T10:31:21.483427+0000", 2024-04-23T10:32:04.682 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "dispatched" 2024-04-23T10:32:04.682 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-23T10:32:04.682 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-04-23T10:32:04.682 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-04-23T10:31:21.483789+0000", 2024-04-23T10:32:04.682 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "requesting remote authpins" 2024-04-23T10:32:04.683 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-23T10:32:04.683 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-04-23T10:32:04.683 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-04-23T10:31:21.507506+0000", 2024-04-23T10:32:04.683 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "acquired locks" 2024-04-23T10:32:04.683 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-23T10:32:04.683 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-04-23T10:32:04.683 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-04-23T10:31:21.507515+0000", 2024-04-23T10:32:04.683 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "quiesce complete for non-auth inode" 2024-04-23T10:32:04.683 INFO:teuthology.orchestra.run.smithi114.stdout: } 2024-04-23T10:32:04.683 INFO:teuthology.orchestra.run.smithi114.stdout: ], 2024-04-23T10:32:04.683 INFO:teuthology.orchestra.run.smithi114.stdout: "locks": [ 2024-04-23T10:32:04.683 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-04-23T10:32:04.683 INFO:teuthology.orchestra.run.smithi114.stdout: "object": { 2024-04-23T10:32:04.683 INFO:teuthology.orchestra.run.smithi114.stdout: "is_auth": false, 2024-04-23T10:32:04.683 INFO:teuthology.orchestra.run.smithi114.stdout: "auth_state": { 2024-04-23T10:32:04.684 INFO:teuthology.orchestra.run.smithi114.stdout: "replicas": {} 2024-04-23T10:32:04.684 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-23T10:32:04.684 INFO:teuthology.orchestra.run.smithi114.stdout: "replica_state": { 2024-04-23T10:32:04.684 INFO:teuthology.orchestra.run.smithi114.stdout: "authority": [ 2024-04-23T10:32:04.684 INFO:teuthology.orchestra.run.smithi114.stdout: 2, 2024-04-23T10:32:04.684 INFO:teuthology.orchestra.run.smithi114.stdout: -2 2024-04-23T10:32:04.684 INFO:teuthology.orchestra.run.smithi114.stdout: ], 2024-04-23T10:32:04.684 INFO:teuthology.orchestra.run.smithi114.stdout: "replica_nonce": 1 2024-04-23T10:32:04.684 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-23T10:32:04.684 INFO:teuthology.orchestra.run.smithi114.stdout: "auth_pins": 0, 2024-04-23T10:32:04.684 INFO:teuthology.orchestra.run.smithi114.stdout: "is_frozen": false, 2024-04-23T10:32:04.684 INFO:teuthology.orchestra.run.smithi114.stdout: "is_freezing": false, 2024-04-23T10:32:04.684 INFO:teuthology.orchestra.run.smithi114.stdout: "pins": { 2024-04-23T10:32:04.684 INFO:teuthology.orchestra.run.smithi114.stdout: "request": 1, 2024-04-23T10:32:04.684 INFO:teuthology.orchestra.run.smithi114.stdout: "lock": 1, 2024-04-23T10:32:04.685 INFO:teuthology.orchestra.run.smithi114.stdout: "caps": 0, 2024-04-23T10:32:04.685 INFO:teuthology.orchestra.run.smithi114.stdout: "exportingcaps": 0, 2024-04-23T10:32:04.685 INFO:teuthology.orchestra.run.smithi114.stdout: "dirtyparent": 0, 2024-04-23T10:32:04.685 INFO:teuthology.orchestra.run.smithi114.stdout: "dirty": 0, 2024-04-23T10:32:04.685 INFO:teuthology.orchestra.run.smithi114.stdout: "authpin": 0, 2024-04-23T10:32:04.685 INFO:teuthology.orchestra.run.smithi114.stdout: "tempexporting": 0 2024-04-23T10:32:04.685 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-23T10:32:04.685 INFO:teuthology.orchestra.run.smithi114.stdout: "nref": 2 2024-04-23T10:32:04.685 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-23T10:32:04.685 INFO:teuthology.orchestra.run.smithi114.stdout: "object_string": "[inode 0x1000000021e [7,head] /volumes/qa/sv_1/b206794e-7622-4ab8-8943-f3ae4e65b7a3/client.0/tmp/clients/client0/~dmtmp/COREL/GRAPHIC2.CDR rep@2.1 v7254 s=260096 n(v0 rc2024-04-23T10:27:52.043396+0000 b260096 1=1+0)/n(v0 1=1+0) (iquiesce lock x=1 by request(mds.0:194 nref=3)) | request=1 lock=1 caps=0 exportingcaps=0 dirtyparent=0 dirty=0 authpin=0 tempexporting=0 0x5571cf562c00]", 2024-04-23T10:32:04.685 INFO:teuthology.orchestra.run.smithi114.stdout: "lock": { 2024-04-23T10:32:04.685 INFO:teuthology.orchestra.run.smithi114.stdout: "gather_set": [], 2024-04-23T10:32:04.685 INFO:teuthology.orchestra.run.smithi114.stdout: "state": "lock", 2024-04-23T10:32:04.685 INFO:teuthology.orchestra.run.smithi114.stdout: "type": "iquiesce", 2024-04-23T10:32:04.685 INFO:teuthology.orchestra.run.smithi114.stdout: "is_leased": false, 2024-04-23T10:32:04.686 INFO:teuthology.orchestra.run.smithi114.stdout: "num_rdlocks": 0, 2024-04-23T10:32:04.686 INFO:teuthology.orchestra.run.smithi114.stdout: "num_wrlocks": 0, 2024-04-23T10:32:04.686 INFO:teuthology.orchestra.run.smithi114.stdout: "num_xlocks": 1, 2024-04-23T10:32:04.686 INFO:teuthology.orchestra.run.smithi114.stdout: "xlock_by": { 2024-04-23T10:32:04.686 INFO:teuthology.orchestra.run.smithi114.stdout: "reqid": { 2024-04-23T10:32:04.686 INFO:teuthology.orchestra.run.smithi114.stdout: "entity": { 2024-04-23T10:32:04.686 INFO:teuthology.orchestra.run.smithi114.stdout: "type": "mds", 2024-04-23T10:32:04.686 INFO:teuthology.orchestra.run.smithi114.stdout: "num": 0 2024-04-23T10:32:04.686 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-23T10:32:04.686 INFO:teuthology.orchestra.run.smithi114.stdout: "tid": 194 2024-04-23T10:32:04.686 INFO:teuthology.orchestra.run.smithi114.stdout: } 2024-04-23T10:32:04.686 INFO:teuthology.orchestra.run.smithi114.stdout: } 2024-04-23T10:32:04.686 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-23T10:32:04.686 INFO:teuthology.orchestra.run.smithi114.stdout: "flags": 4, 2024-04-23T10:32:04.687 INFO:teuthology.orchestra.run.smithi114.stdout: "wrlock_target": -1 2024-04-23T10:32:04.687 INFO:teuthology.orchestra.run.smithi114.stdout: } 2024-04-23T10:32:04.687 INFO:teuthology.orchestra.run.smithi114.stdout: ] 2024-04-23T10:32:04.687 INFO:teuthology.orchestra.run.smithi114.stdout: } 2024-04-23T10:32:04.687 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-23T10:32:04.687 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-04-23T10:32:04.687 INFO:teuthology.orchestra.run.smithi114.stdout: "description": "peer_request:mds.1:104", 2024-04-23T10:32:04.687 INFO:teuthology.orchestra.run.smithi114.stdout: "initiated_at": "2024-04-23T10:31:21.447409+0000", 2024-04-23T10:32:04.687 INFO:teuthology.orchestra.run.smithi114.stdout: "age": 42.436980587999997, 2024-04-23T10:32:04.687 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 42.439085126999998, 2024-04-23T10:32:04.687 INFO:teuthology.orchestra.run.smithi114.stdout: "continuous": false, 2024-04-23T10:32:04.687 INFO:teuthology.orchestra.run.smithi114.stdout: "type_data": { 2024-04-23T10:32:04.687 INFO:teuthology.orchestra.run.smithi114.stdout: "result": -2147483648, 2024-04-23T10:32:04.687 INFO:teuthology.orchestra.run.smithi114.stdout: "flag_point": "dispatched", 2024-04-23T10:32:04.687 INFO:teuthology.orchestra.run.smithi114.stdout: "reqid": { 2024-04-23T10:32:04.687 INFO:teuthology.orchestra.run.smithi114.stdout: "entity": { 2024-04-23T10:32:04.688 INFO:teuthology.orchestra.run.smithi114.stdout: "type": "mds", 2024-04-23T10:32:04.688 INFO:teuthology.orchestra.run.smithi114.stdout: "num": 1 2024-04-23T10:32:04.688 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-23T10:32:04.688 INFO:teuthology.orchestra.run.smithi114.stdout: "tid": 104 2024-04-23T10:32:04.688 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-23T10:32:04.688 INFO:teuthology.orchestra.run.smithi114.stdout: "op_type": "peer_request", 2024-04-23T10:32:04.688 INFO:teuthology.orchestra.run.smithi114.stdout: "leader_info": { 2024-04-23T10:32:04.688 INFO:teuthology.orchestra.run.smithi114.stdout: "leader": "1" 2024-04-23T10:32:04.688 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-23T10:32:04.688 INFO:teuthology.orchestra.run.smithi114.stdout: "events": [ 2024-04-23T10:32:04.688 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-04-23T10:32:04.688 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-04-23T10:31:21.447409+0000", 2024-04-23T10:32:04.688 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "initiated" 2024-04-23T10:32:04.688 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-23T10:32:04.688 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-04-23T10:32:04.689 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-04-23T10:31:21.447409+0000", 2024-04-23T10:32:04.689 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "header_read" 2024-04-23T10:32:04.689 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-23T10:32:04.689 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-04-23T10:32:04.689 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-04-23T10:31:21.447409+0000", 2024-04-23T10:32:04.689 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "throttled" 2024-04-23T10:32:04.689 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-23T10:32:04.689 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-04-23T10:32:04.689 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-04-23T10:31:21.447417+0000", 2024-04-23T10:32:04.689 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "all_read" 2024-04-23T10:32:04.689 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-23T10:32:04.689 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-04-23T10:32:04.689 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-04-23T10:31:21.448013+0000", 2024-04-23T10:32:04.689 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "dispatched" 2024-04-23T10:32:04.689 INFO:teuthology.orchestra.run.smithi114.stdout: } 2024-04-23T10:32:04.690 INFO:teuthology.orchestra.run.smithi114.stdout: ], 2024-04-23T10:32:04.690 INFO:teuthology.orchestra.run.smithi114.stdout: "locks": [] 2024-04-23T10:32:04.690 INFO:teuthology.orchestra.run.smithi114.stdout: } 2024-04-23T10:32:04.690 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-23T10:32:04.690 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-04-23T10:32:04.690 INFO:teuthology.orchestra.run.smithi114.stdout: "description": "internal op quiesce_inode:mds.0:170 fp=#0x10000000221", 2024-04-23T10:32:04.690 INFO:teuthology.orchestra.run.smithi114.stdout: "initiated_at": "2024-04-23T10:31:21.459896+0000", 2024-04-23T10:32:04.690 INFO:teuthology.orchestra.run.smithi114.stdout: "age": 42.424492903000001, 2024-04-23T10:32:04.690 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 42.426618873000002, 2024-04-23T10:32:04.690 INFO:teuthology.orchestra.run.smithi114.stdout: "continuous": true, 2024-04-23T10:32:04.690 INFO:teuthology.orchestra.run.smithi114.stdout: "type_data": { 2024-04-23T10:32:04.690 INFO:teuthology.orchestra.run.smithi114.stdout: "result": -2147483648, 2024-04-23T10:32:04.690 INFO:teuthology.orchestra.run.smithi114.stdout: "flag_point": "quiesce complete", 2024-04-23T10:32:04.690 INFO:teuthology.orchestra.run.smithi114.stdout: "reqid": { 2024-04-23T10:32:04.691 INFO:teuthology.orchestra.run.smithi114.stdout: "entity": { 2024-04-23T10:32:04.691 INFO:teuthology.orchestra.run.smithi114.stdout: "type": "mds", 2024-04-23T10:32:04.691 INFO:teuthology.orchestra.run.smithi114.stdout: "num": 0 2024-04-23T10:32:04.691 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-23T10:32:04.691 INFO:teuthology.orchestra.run.smithi114.stdout: "tid": 170 2024-04-23T10:32:04.691 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-23T10:32:04.691 INFO:teuthology.orchestra.run.smithi114.stdout: "op_type": "internal_op", 2024-04-23T10:32:04.691 INFO:teuthology.orchestra.run.smithi114.stdout: "internal_op": 5385, 2024-04-23T10:32:04.691 INFO:teuthology.orchestra.run.smithi114.stdout: "op_name": "quiesce_inode", 2024-04-23T10:32:04.691 INFO:teuthology.orchestra.run.smithi114.stdout: "events": [ 2024-04-23T10:32:04.691 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-04-23T10:32:04.691 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-04-23T10:31:21.459896+0000", 2024-04-23T10:32:04.691 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "initiated" 2024-04-23T10:32:04.691 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-23T10:32:04.691 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-04-23T10:32:04.691 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-04-23T10:31:21.459896+0000", 2024-04-23T10:32:04.692 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "header_read" 2024-04-23T10:32:04.692 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-23T10:32:04.692 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-04-23T10:32:04.692 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-04-23T10:31:21.459896+0000", 2024-04-23T10:32:04.692 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "throttled" 2024-04-23T10:32:04.692 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-23T10:32:04.692 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-04-23T10:32:04.692 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-04-23T10:31:21.459896+0000", 2024-04-23T10:32:04.692 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "all_read" 2024-04-23T10:32:04.692 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-23T10:32:04.692 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-04-23T10:32:04.692 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-04-23T10:31:21.459896+0000", 2024-04-23T10:32:04.692 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "dispatched" 2024-04-23T10:32:04.693 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-23T10:32:04.693 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-04-23T10:32:04.693 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-04-23T10:31:21.461535+0000", 2024-04-23T10:32:04.693 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "failed to xlock, waiting" 2024-04-23T10:32:04.693 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-23T10:32:04.693 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-04-23T10:32:04.693 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-04-23T10:31:21.467090+0000", 2024-04-23T10:32:04.693 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "acquired locks" 2024-04-23T10:32:04.693 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-23T10:32:04.693 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-04-23T10:32:04.693 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-04-23T10:31:21.479758+0000", 2024-04-23T10:32:04.693 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "quiesce complete" 2024-04-23T10:32:04.693 INFO:teuthology.orchestra.run.smithi114.stdout: } 2024-04-23T10:32:04.693 INFO:teuthology.orchestra.run.smithi114.stdout: ], 2024-04-23T10:32:04.693 INFO:teuthology.orchestra.run.smithi114.stdout: "locks": [ 2024-04-23T10:32:04.694 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-04-23T10:32:04.694 INFO:teuthology.orchestra.run.smithi114.stdout: "object": { 2024-04-23T10:32:04.694 INFO:teuthology.orchestra.run.smithi114.stdout: "is_auth": true, 2024-04-23T10:32:04.694 INFO:teuthology.orchestra.run.smithi114.stdout: "auth_state": { 2024-04-23T10:32:04.694 INFO:teuthology.orchestra.run.smithi114.stdout: "replicas": { 2024-04-23T10:32:04.694 INFO:teuthology.orchestra.run.smithi114.stdout: "1": 1, 2024-04-23T10:32:04.694 INFO:teuthology.orchestra.run.smithi114.stdout: "2": 1 2024-04-23T10:32:04.694 INFO:teuthology.orchestra.run.smithi114.stdout: } 2024-04-23T10:32:04.694 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-23T10:32:04.694 INFO:teuthology.orchestra.run.smithi114.stdout: "replica_state": { 2024-04-23T10:32:04.694 INFO:teuthology.orchestra.run.smithi114.stdout: "authority": [ 2024-04-23T10:32:04.694 INFO:teuthology.orchestra.run.smithi114.stdout: 0, 2024-04-23T10:32:04.694 INFO:teuthology.orchestra.run.smithi114.stdout: -2 2024-04-23T10:32:04.694 INFO:teuthology.orchestra.run.smithi114.stdout: ], 2024-04-23T10:32:04.695 INFO:teuthology.orchestra.run.smithi114.stdout: "replica_nonce": 0 2024-04-23T10:32:04.695 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-23T10:32:04.695 INFO:teuthology.orchestra.run.smithi114.stdout: "auth_pins": 3, 2024-04-23T10:32:04.695 INFO:teuthology.orchestra.run.smithi114.stdout: "is_frozen": false, 2024-04-23T10:32:04.695 INFO:teuthology.orchestra.run.smithi114.stdout: "is_freezing": false, 2024-04-23T10:32:04.695 INFO:teuthology.orchestra.run.smithi114.stdout: "pins": { 2024-04-23T10:32:04.695 INFO:teuthology.orchestra.run.smithi114.stdout: "request": 1, 2024-04-23T10:32:04.695 INFO:teuthology.orchestra.run.smithi114.stdout: "lock": 1, 2024-04-23T10:32:04.695 INFO:teuthology.orchestra.run.smithi114.stdout: "dirfrag": 1, 2024-04-23T10:32:04.695 INFO:teuthology.orchestra.run.smithi114.stdout: "caps": 1, 2024-04-23T10:32:04.695 INFO:teuthology.orchestra.run.smithi114.stdout: "dirtyrstat": 0, 2024-04-23T10:32:04.695 INFO:teuthology.orchestra.run.smithi114.stdout: "dirtyparent": 0, 2024-04-23T10:32:04.695 INFO:teuthology.orchestra.run.smithi114.stdout: "replicated": 1, 2024-04-23T10:32:04.695 INFO:teuthology.orchestra.run.smithi114.stdout: "dirty": 0, 2024-04-23T10:32:04.695 INFO:teuthology.orchestra.run.smithi114.stdout: "waiter": 0, 2024-04-23T10:32:04.696 INFO:teuthology.orchestra.run.smithi114.stdout: "authpin": 1 2024-04-23T10:32:04.696 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-23T10:32:04.696 INFO:teuthology.orchestra.run.smithi114.stdout: "nref": 6 2024-04-23T10:32:04.696 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-23T10:32:04.696 INFO:teuthology.orchestra.run.smithi114.stdout: "object_string": "[inode 0x10000000221 [...7,head] /volumes/qa/sv_1/b206794e-7622-4ab8-8943-f3ae4e65b7a3/client.0/tmp/clients/client0/~dmtmp/WORDPRO/ auth{1=1,2=1} v45561 ap=3 f(v0 m2024-04-23T10:31:17.760716+0000 3=3+0) n(v0 rc2024-04-23T10:31:17.760716+0000 b780288 4=3+1) old_inodes=4 (inest mix) (iquiesce lock x=1 by request(mds.0:170 nref=3)) caps={15312=pAsLsXsFs/pAsLsXsFsxcral@9160},l=15312 | request=1 lock=1 dirfrag=1 caps=1 dirtyrstat=0 dirtyparent=0 replicated=1 dirty=0 waiter=0 authpin=1 0x5571cf4d4b00]", 2024-04-23T10:32:04.696 INFO:teuthology.orchestra.run.smithi114.stdout: "lock": { 2024-04-23T10:32:04.696 INFO:teuthology.orchestra.run.smithi114.stdout: "gather_set": [], 2024-04-23T10:32:04.696 INFO:teuthology.orchestra.run.smithi114.stdout: "state": "lock", 2024-04-23T10:32:04.696 INFO:teuthology.orchestra.run.smithi114.stdout: "type": "iquiesce", 2024-04-23T10:32:04.696 INFO:teuthology.orchestra.run.smithi114.stdout: "is_leased": false, 2024-04-23T10:32:04.696 INFO:teuthology.orchestra.run.smithi114.stdout: "num_rdlocks": 0, 2024-04-23T10:32:04.696 INFO:teuthology.orchestra.run.smithi114.stdout: "num_wrlocks": 0, 2024-04-23T10:32:04.696 INFO:teuthology.orchestra.run.smithi114.stdout: "num_xlocks": 1, 2024-04-23T10:32:04.696 INFO:teuthology.orchestra.run.smithi114.stdout: "xlock_by": { 2024-04-23T10:32:04.696 INFO:teuthology.orchestra.run.smithi114.stdout: "reqid": { 2024-04-23T10:32:04.697 INFO:teuthology.orchestra.run.smithi114.stdout: "entity": { 2024-04-23T10:32:04.697 INFO:teuthology.orchestra.run.smithi114.stdout: "type": "mds", 2024-04-23T10:32:04.697 INFO:teuthology.orchestra.run.smithi114.stdout: "num": 0 2024-04-23T10:32:04.697 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-23T10:32:04.697 INFO:teuthology.orchestra.run.smithi114.stdout: "tid": 170 2024-04-23T10:32:04.697 INFO:teuthology.orchestra.run.smithi114.stdout: } 2024-04-23T10:32:04.697 INFO:teuthology.orchestra.run.smithi114.stdout: } 2024-04-23T10:32:04.697 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-23T10:32:04.697 INFO:teuthology.orchestra.run.smithi114.stdout: "flags": 4, 2024-04-23T10:32:04.697 INFO:teuthology.orchestra.run.smithi114.stdout: "wrlock_target": -1 2024-04-23T10:32:04.697 INFO:teuthology.orchestra.run.smithi114.stdout: } 2024-04-23T10:32:04.697 INFO:teuthology.orchestra.run.smithi114.stdout: ] 2024-04-23T10:32:04.697 INFO:teuthology.orchestra.run.smithi114.stdout: } 2024-04-23T10:32:04.697 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-23T10:32:04.698 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-04-23T10:32:04.698 INFO:teuthology.orchestra.run.smithi114.stdout: "description": "internal op quiesce_inode:mds.0:195 fp=#0x10000000218", 2024-04-23T10:32:04.698 INFO:teuthology.orchestra.run.smithi114.stdout: "initiated_at": "2024-04-23T10:31:21.483450+0000", 2024-04-23T10:32:04.698 INFO:teuthology.orchestra.run.smithi114.stdout: "age": 42.400938924999998, 2024-04-23T10:32:04.698 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 42.403132233000001, 2024-04-23T10:32:04.698 INFO:teuthology.orchestra.run.smithi114.stdout: "continuous": true, 2024-04-23T10:32:04.698 INFO:teuthology.orchestra.run.smithi114.stdout: "type_data": { 2024-04-23T10:32:04.698 INFO:teuthology.orchestra.run.smithi114.stdout: "result": -2147483648, 2024-04-23T10:32:04.698 INFO:teuthology.orchestra.run.smithi114.stdout: "flag_point": "quiesce complete for non-auth inode", 2024-04-23T10:32:04.698 INFO:teuthology.orchestra.run.smithi114.stdout: "reqid": { 2024-04-23T10:32:04.698 INFO:teuthology.orchestra.run.smithi114.stdout: "entity": { 2024-04-23T10:32:04.698 INFO:teuthology.orchestra.run.smithi114.stdout: "type": "mds", 2024-04-23T10:32:04.698 INFO:teuthology.orchestra.run.smithi114.stdout: "num": 0 2024-04-23T10:32:04.698 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-23T10:32:04.699 INFO:teuthology.orchestra.run.smithi114.stdout: "tid": 195 2024-04-23T10:32:04.699 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-23T10:32:04.699 INFO:teuthology.orchestra.run.smithi114.stdout: "op_type": "internal_op", 2024-04-23T10:32:04.699 INFO:teuthology.orchestra.run.smithi114.stdout: "internal_op": 5385, 2024-04-23T10:32:04.699 INFO:teuthology.orchestra.run.smithi114.stdout: "op_name": "quiesce_inode", 2024-04-23T10:32:04.699 INFO:teuthology.orchestra.run.smithi114.stdout: "events": [ 2024-04-23T10:32:04.699 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-04-23T10:32:04.699 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-04-23T10:31:21.483450+0000", 2024-04-23T10:32:04.699 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "initiated" 2024-04-23T10:32:04.699 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-23T10:32:04.699 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-04-23T10:32:04.699 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-04-23T10:31:21.483450+0000", 2024-04-23T10:32:04.699 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "header_read" 2024-04-23T10:32:04.699 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-23T10:32:04.699 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-04-23T10:32:04.700 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-04-23T10:31:21.483450+0000", 2024-04-23T10:32:04.700 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "throttled" 2024-04-23T10:32:04.700 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-23T10:32:04.700 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-04-23T10:32:04.700 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-04-23T10:31:21.483450+0000", 2024-04-23T10:32:04.700 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "all_read" 2024-04-23T10:32:04.700 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-23T10:32:04.700 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-04-23T10:32:04.700 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-04-23T10:31:21.483450+0000", 2024-04-23T10:32:04.700 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "dispatched" 2024-04-23T10:32:04.700 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-23T10:32:04.700 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-04-23T10:32:04.700 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-04-23T10:31:21.483840+0000", 2024-04-23T10:32:04.701 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "requesting remote authpins" 2024-04-23T10:32:04.701 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-23T10:32:04.701 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-04-23T10:32:04.701 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-04-23T10:31:21.507655+0000", 2024-04-23T10:32:04.701 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "acquired locks" 2024-04-23T10:32:04.701 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-23T10:32:04.701 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-04-23T10:32:04.701 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-04-23T10:31:21.507665+0000", 2024-04-23T10:32:04.701 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "quiesce complete for non-auth inode" 2024-04-23T10:32:04.701 INFO:teuthology.orchestra.run.smithi114.stdout: } 2024-04-23T10:32:04.701 INFO:teuthology.orchestra.run.smithi114.stdout: ], 2024-04-23T10:32:04.701 INFO:teuthology.orchestra.run.smithi114.stdout: "locks": [ 2024-04-23T10:32:04.701 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-04-23T10:32:04.701 INFO:teuthology.orchestra.run.smithi114.stdout: "object": { 2024-04-23T10:32:04.701 INFO:teuthology.orchestra.run.smithi114.stdout: "is_auth": false, 2024-04-23T10:32:04.702 INFO:teuthology.orchestra.run.smithi114.stdout: "auth_state": { 2024-04-23T10:32:04.702 INFO:teuthology.orchestra.run.smithi114.stdout: "replicas": {} 2024-04-23T10:32:04.702 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-23T10:32:04.702 INFO:teuthology.orchestra.run.smithi114.stdout: "replica_state": { 2024-04-23T10:32:04.702 INFO:teuthology.orchestra.run.smithi114.stdout: "authority": [ 2024-04-23T10:32:04.702 INFO:teuthology.orchestra.run.smithi114.stdout: 2, 2024-04-23T10:32:04.702 INFO:teuthology.orchestra.run.smithi114.stdout: -2 2024-04-23T10:32:04.702 INFO:teuthology.orchestra.run.smithi114.stdout: ], 2024-04-23T10:32:04.702 INFO:teuthology.orchestra.run.smithi114.stdout: "replica_nonce": 1 2024-04-23T10:32:04.702 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-23T10:32:04.702 INFO:teuthology.orchestra.run.smithi114.stdout: "auth_pins": 0, 2024-04-23T10:32:04.702 INFO:teuthology.orchestra.run.smithi114.stdout: "is_frozen": false, 2024-04-23T10:32:04.702 INFO:teuthology.orchestra.run.smithi114.stdout: "is_freezing": false, 2024-04-23T10:32:04.702 INFO:teuthology.orchestra.run.smithi114.stdout: "pins": { 2024-04-23T10:32:04.702 INFO:teuthology.orchestra.run.smithi114.stdout: "request": 1, 2024-04-23T10:32:04.703 INFO:teuthology.orchestra.run.smithi114.stdout: "lock": 1, 2024-04-23T10:32:04.703 INFO:teuthology.orchestra.run.smithi114.stdout: "caps": 0, 2024-04-23T10:32:04.703 INFO:teuthology.orchestra.run.smithi114.stdout: "exportingcaps": 0, 2024-04-23T10:32:04.703 INFO:teuthology.orchestra.run.smithi114.stdout: "dirtyparent": 0, 2024-04-23T10:32:04.703 INFO:teuthology.orchestra.run.smithi114.stdout: "dirty": 0, 2024-04-23T10:32:04.703 INFO:teuthology.orchestra.run.smithi114.stdout: "authpin": 0, 2024-04-23T10:32:04.703 INFO:teuthology.orchestra.run.smithi114.stdout: "tempexporting": 0 2024-04-23T10:32:04.703 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-23T10:32:04.703 INFO:teuthology.orchestra.run.smithi114.stdout: "nref": 2 2024-04-23T10:32:04.703 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-23T10:32:04.703 INFO:teuthology.orchestra.run.smithi114.stdout: "object_string": "[inode 0x10000000218 [7,head] /volumes/qa/sv_1/b206794e-7622-4ab8-8943-f3ae4e65b7a3/client.0/tmp/clients/client0/~dmtmp/COREL/ARTISTIC.ACL rep@2.1 v7240 s=260096 n(v0 rc2024-04-23T10:27:52.014396+0000 b260096 1=1+0)/n(v0 1=1+0) (iquiesce lock x=1 by request(mds.0:195 nref=3)) | request=1 lock=1 caps=0 exportingcaps=0 dirtyparent=0 dirty=0 authpin=0 tempexporting=0 0x5571cf560b00]", 2024-04-23T10:32:04.703 INFO:teuthology.orchestra.run.smithi114.stdout: "lock": { 2024-04-23T10:32:04.703 INFO:teuthology.orchestra.run.smithi114.stdout: "gather_set": [], 2024-04-23T10:32:04.703 INFO:teuthology.orchestra.run.smithi114.stdout: "state": "lock", 2024-04-23T10:32:04.703 INFO:teuthology.orchestra.run.smithi114.stdout: "type": "iquiesce", 2024-04-23T10:32:04.704 INFO:teuthology.orchestra.run.smithi114.stdout: "is_leased": false, 2024-04-23T10:32:04.704 INFO:teuthology.orchestra.run.smithi114.stdout: "num_rdlocks": 0, 2024-04-23T10:32:04.704 INFO:teuthology.orchestra.run.smithi114.stdout: "num_wrlocks": 0, 2024-04-23T10:32:04.704 INFO:teuthology.orchestra.run.smithi114.stdout: "num_xlocks": 1, 2024-04-23T10:32:04.704 INFO:teuthology.orchestra.run.smithi114.stdout: "xlock_by": { 2024-04-23T10:32:04.704 INFO:teuthology.orchestra.run.smithi114.stdout: "reqid": { 2024-04-23T10:32:04.704 INFO:teuthology.orchestra.run.smithi114.stdout: "entity": { 2024-04-23T10:32:04.704 INFO:teuthology.orchestra.run.smithi114.stdout: "type": "mds", 2024-04-23T10:32:04.704 INFO:teuthology.orchestra.run.smithi114.stdout: "num": 0 2024-04-23T10:32:04.704 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-23T10:32:04.704 INFO:teuthology.orchestra.run.smithi114.stdout: "tid": 195 2024-04-23T10:32:04.704 INFO:teuthology.orchestra.run.smithi114.stdout: } 2024-04-23T10:32:04.704 INFO:teuthology.orchestra.run.smithi114.stdout: } 2024-04-23T10:32:04.704 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-23T10:32:04.704 INFO:teuthology.orchestra.run.smithi114.stdout: "flags": 4, 2024-04-23T10:32:04.705 INFO:teuthology.orchestra.run.smithi114.stdout: "wrlock_target": -1 2024-04-23T10:32:04.705 INFO:teuthology.orchestra.run.smithi114.stdout: } 2024-04-23T10:32:04.705 INFO:teuthology.orchestra.run.smithi114.stdout: ] 2024-04-23T10:32:04.705 INFO:teuthology.orchestra.run.smithi114.stdout: } 2024-04-23T10:32:04.705 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-23T10:32:04.705 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-04-23T10:32:04.705 INFO:teuthology.orchestra.run.smithi114.stdout: "description": "peer_request:mds.2:109", 2024-04-23T10:32:04.705 INFO:teuthology.orchestra.run.smithi114.stdout: "initiated_at": "2024-04-23T10:31:21.447475+0000", 2024-04-23T10:32:04.705 INFO:teuthology.orchestra.run.smithi114.stdout: "age": 42.436913812999997, 2024-04-23T10:32:04.705 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 42.439168008999999, 2024-04-23T10:32:04.705 INFO:teuthology.orchestra.run.smithi114.stdout: "continuous": false, 2024-04-23T10:32:04.705 INFO:teuthology.orchestra.run.smithi114.stdout: "type_data": { 2024-04-23T10:32:04.705 INFO:teuthology.orchestra.run.smithi114.stdout: "result": -2147483648, 2024-04-23T10:32:04.705 INFO:teuthology.orchestra.run.smithi114.stdout: "flag_point": "dispatched", 2024-04-23T10:32:04.705 INFO:teuthology.orchestra.run.smithi114.stdout: "reqid": { 2024-04-23T10:32:04.706 INFO:teuthology.orchestra.run.smithi114.stdout: "entity": { 2024-04-23T10:32:04.706 INFO:teuthology.orchestra.run.smithi114.stdout: "type": "mds", 2024-04-23T10:32:04.706 INFO:teuthology.orchestra.run.smithi114.stdout: "num": 2 2024-04-23T10:32:04.706 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-23T10:32:04.706 INFO:teuthology.orchestra.run.smithi114.stdout: "tid": 109 2024-04-23T10:32:04.706 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-23T10:32:04.706 INFO:teuthology.orchestra.run.smithi114.stdout: "op_type": "peer_request", 2024-04-23T10:32:04.706 INFO:teuthology.orchestra.run.smithi114.stdout: "leader_info": { 2024-04-23T10:32:04.706 INFO:teuthology.orchestra.run.smithi114.stdout: "leader": "2" 2024-04-23T10:32:04.706 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-23T10:32:04.706 INFO:teuthology.orchestra.run.smithi114.stdout: "events": [ 2024-04-23T10:32:04.706 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-04-23T10:32:04.706 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-04-23T10:31:21.447475+0000", 2024-04-23T10:32:04.706 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "initiated" 2024-04-23T10:32:04.707 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-23T10:32:04.707 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-04-23T10:32:04.707 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-04-23T10:31:21.447475+0000", 2024-04-23T10:32:04.707 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "header_read" 2024-04-23T10:32:04.707 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-23T10:32:04.707 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-04-23T10:32:04.707 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-04-23T10:31:21.447476+0000", 2024-04-23T10:32:04.707 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "throttled" 2024-04-23T10:32:04.707 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-23T10:32:04.707 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-04-23T10:32:04.707 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-04-23T10:31:21.447482+0000", 2024-04-23T10:32:04.707 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "all_read" 2024-04-23T10:32:04.707 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-23T10:32:04.707 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-04-23T10:32:04.707 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-04-23T10:31:21.448091+0000", 2024-04-23T10:32:04.708 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "dispatched" 2024-04-23T10:32:04.708 INFO:teuthology.orchestra.run.smithi114.stdout: } 2024-04-23T10:32:04.708 INFO:teuthology.orchestra.run.smithi114.stdout: ], 2024-04-23T10:32:04.708 INFO:teuthology.orchestra.run.smithi114.stdout: "locks": [] 2024-04-23T10:32:04.708 INFO:teuthology.orchestra.run.smithi114.stdout: } 2024-04-23T10:32:04.708 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-23T10:32:04.708 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-04-23T10:32:04.708 INFO:teuthology.orchestra.run.smithi114.stdout: "description": "internal op quiesce_inode:mds.0:171 fp=#0x10000000229", 2024-04-23T10:32:04.708 INFO:teuthology.orchestra.run.smithi114.stdout: "initiated_at": "2024-04-23T10:31:21.459914+0000", 2024-04-23T10:32:04.708 INFO:teuthology.orchestra.run.smithi114.stdout: "age": 42.424475653000002, 2024-04-23T10:32:04.708 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 42.426751056000001, 2024-04-23T10:32:04.708 INFO:teuthology.orchestra.run.smithi114.stdout: "continuous": true, 2024-04-23T10:32:04.708 INFO:teuthology.orchestra.run.smithi114.stdout: "type_data": { 2024-04-23T10:32:04.708 INFO:teuthology.orchestra.run.smithi114.stdout: "result": -2147483648, 2024-04-23T10:32:04.708 INFO:teuthology.orchestra.run.smithi114.stdout: "flag_point": "quiesce complete", 2024-04-23T10:32:04.709 INFO:teuthology.orchestra.run.smithi114.stdout: "reqid": { 2024-04-23T10:32:04.709 INFO:teuthology.orchestra.run.smithi114.stdout: "entity": { 2024-04-23T10:32:04.709 INFO:teuthology.orchestra.run.smithi114.stdout: "type": "mds", 2024-04-23T10:32:04.709 INFO:teuthology.orchestra.run.smithi114.stdout: "num": 0 2024-04-23T10:32:04.709 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-23T10:32:04.709 INFO:teuthology.orchestra.run.smithi114.stdout: "tid": 171 2024-04-23T10:32:04.709 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-23T10:32:04.709 INFO:teuthology.orchestra.run.smithi114.stdout: "op_type": "internal_op", 2024-04-23T10:32:04.709 INFO:teuthology.orchestra.run.smithi114.stdout: "internal_op": 5385, 2024-04-23T10:32:04.709 INFO:teuthology.orchestra.run.smithi114.stdout: "op_name": "quiesce_inode", 2024-04-23T10:32:04.709 INFO:teuthology.orchestra.run.smithi114.stdout: "events": [ 2024-04-23T10:32:04.709 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-04-23T10:32:04.709 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-04-23T10:31:21.459914+0000", 2024-04-23T10:32:04.709 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "initiated" 2024-04-23T10:32:04.709 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-23T10:32:04.710 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-04-23T10:32:04.710 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-04-23T10:31:21.459914+0000", 2024-04-23T10:32:04.710 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "header_read" 2024-04-23T10:32:04.710 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-23T10:32:04.710 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-04-23T10:32:04.710 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-04-23T10:31:21.459914+0000", 2024-04-23T10:32:04.710 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "throttled" 2024-04-23T10:32:04.710 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-23T10:32:04.710 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-04-23T10:32:04.710 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-04-23T10:31:21.459914+0000", 2024-04-23T10:32:04.710 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "all_read" 2024-04-23T10:32:04.710 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-23T10:32:04.710 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-04-23T10:32:04.710 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-04-23T10:31:21.459914+0000", 2024-04-23T10:32:04.711 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "dispatched" 2024-04-23T10:32:04.711 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-23T10:32:04.711 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-04-23T10:32:04.711 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-04-23T10:31:21.461888+0000", 2024-04-23T10:32:04.711 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "failed to xlock, waiting" 2024-04-23T10:32:04.711 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-23T10:32:04.711 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-04-23T10:32:04.711 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-04-23T10:31:21.472768+0000", 2024-04-23T10:32:04.711 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "acquired locks" 2024-04-23T10:32:04.711 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-23T10:32:04.711 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-04-23T10:32:04.711 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-04-23T10:31:21.473165+0000", 2024-04-23T10:32:04.711 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "quiesce complete" 2024-04-23T10:32:04.711 INFO:teuthology.orchestra.run.smithi114.stdout: } 2024-04-23T10:32:04.712 INFO:teuthology.orchestra.run.smithi114.stdout: ], 2024-04-23T10:32:04.712 INFO:teuthology.orchestra.run.smithi114.stdout: "locks": [ 2024-04-23T10:32:04.712 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-04-23T10:32:04.712 INFO:teuthology.orchestra.run.smithi114.stdout: "object": { 2024-04-23T10:32:04.712 INFO:teuthology.orchestra.run.smithi114.stdout: "is_auth": true, 2024-04-23T10:32:04.712 INFO:teuthology.orchestra.run.smithi114.stdout: "auth_state": { 2024-04-23T10:32:04.712 INFO:teuthology.orchestra.run.smithi114.stdout: "replicas": { 2024-04-23T10:32:04.712 INFO:teuthology.orchestra.run.smithi114.stdout: "1": 1, 2024-04-23T10:32:04.712 INFO:teuthology.orchestra.run.smithi114.stdout: "2": 1 2024-04-23T10:32:04.712 INFO:teuthology.orchestra.run.smithi114.stdout: } 2024-04-23T10:32:04.712 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-23T10:32:04.712 INFO:teuthology.orchestra.run.smithi114.stdout: "replica_state": { 2024-04-23T10:32:04.712 INFO:teuthology.orchestra.run.smithi114.stdout: "authority": [ 2024-04-23T10:32:04.712 INFO:teuthology.orchestra.run.smithi114.stdout: 0, 2024-04-23T10:32:04.712 INFO:teuthology.orchestra.run.smithi114.stdout: -2 2024-04-23T10:32:04.713 INFO:teuthology.orchestra.run.smithi114.stdout: ], 2024-04-23T10:32:04.713 INFO:teuthology.orchestra.run.smithi114.stdout: "replica_nonce": 0 2024-04-23T10:32:04.713 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-23T10:32:04.713 INFO:teuthology.orchestra.run.smithi114.stdout: "auth_pins": 4, 2024-04-23T10:32:04.713 INFO:teuthology.orchestra.run.smithi114.stdout: "is_frozen": false, 2024-04-23T10:32:04.713 INFO:teuthology.orchestra.run.smithi114.stdout: "is_freezing": false, 2024-04-23T10:32:04.713 INFO:teuthology.orchestra.run.smithi114.stdout: "pins": { 2024-04-23T10:32:04.713 INFO:teuthology.orchestra.run.smithi114.stdout: "dirtyscattered": 0, 2024-04-23T10:32:04.713 INFO:teuthology.orchestra.run.smithi114.stdout: "request": 1, 2024-04-23T10:32:04.713 INFO:teuthology.orchestra.run.smithi114.stdout: "lock": 1, 2024-04-23T10:32:04.713 INFO:teuthology.orchestra.run.smithi114.stdout: "dirfrag": 1, 2024-04-23T10:32:04.713 INFO:teuthology.orchestra.run.smithi114.stdout: "caps": 1, 2024-04-23T10:32:04.713 INFO:teuthology.orchestra.run.smithi114.stdout: "dirtyrstat": 0, 2024-04-23T10:32:04.713 INFO:teuthology.orchestra.run.smithi114.stdout: "dirtyparent": 0, 2024-04-23T10:32:04.714 INFO:teuthology.orchestra.run.smithi114.stdout: "replicated": 1, 2024-04-23T10:32:04.714 INFO:teuthology.orchestra.run.smithi114.stdout: "dirty": 0, 2024-04-23T10:32:04.714 INFO:teuthology.orchestra.run.smithi114.stdout: "waiter": 0, 2024-04-23T10:32:04.714 INFO:teuthology.orchestra.run.smithi114.stdout: "authpin": 1, 2024-04-23T10:32:04.714 INFO:teuthology.orchestra.run.smithi114.stdout: "discoverbase": 0 2024-04-23T10:32:04.714 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-23T10:32:04.714 INFO:teuthology.orchestra.run.smithi114.stdout: "nref": 6 2024-04-23T10:32:04.714 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-23T10:32:04.714 INFO:teuthology.orchestra.run.smithi114.stdout: "object_string": "[inode 0x10000000229 [...7,head] /volumes/qa/sv_1/b206794e-7622-4ab8-8943-f3ae4e65b7a3/client.0/tmp/clients/client0/~dmtmp/PARADOX/ auth{1=1,2=1} v45567 ap=4 f(v1976 m2024-04-23T10:31:16.541744+0000 24=24+0) n(v19 rc2024-04-23T10:31:16.541744+0000 b5201920 25=24+1) old_inodes=4 (inest mix) (iquiesce lock x=1 by request(mds.0:171 nref=3)) caps={15312=pAsLsXsFs/pAsLsXsFsxcral@5318} mcw={1=AsLsXsFs} | dirtyscattered=0 request=1 lock=1 dirfrag=1 caps=1 dirtyrstat=0 dirtyparent=0 replicated=1 dirty=0 waiter=0 authpin=1 discoverbase=0 0x5571cf588680]", 2024-04-23T10:32:04.714 INFO:teuthology.orchestra.run.smithi114.stdout: "lock": { 2024-04-23T10:32:04.714 INFO:teuthology.orchestra.run.smithi114.stdout: "gather_set": [], 2024-04-23T10:32:04.714 INFO:teuthology.orchestra.run.smithi114.stdout: "state": "lock", 2024-04-23T10:32:04.714 INFO:teuthology.orchestra.run.smithi114.stdout: "type": "iquiesce", 2024-04-23T10:32:04.714 INFO:teuthology.orchestra.run.smithi114.stdout: "is_leased": false, 2024-04-23T10:32:04.715 INFO:teuthology.orchestra.run.smithi114.stdout: "num_rdlocks": 0, 2024-04-23T10:32:04.715 INFO:teuthology.orchestra.run.smithi114.stdout: "num_wrlocks": 0, 2024-04-23T10:32:04.715 INFO:teuthology.orchestra.run.smithi114.stdout: "num_xlocks": 1, 2024-04-23T10:32:04.715 INFO:teuthology.orchestra.run.smithi114.stdout: "xlock_by": { 2024-04-23T10:32:04.715 INFO:teuthology.orchestra.run.smithi114.stdout: "reqid": { 2024-04-23T10:32:04.715 INFO:teuthology.orchestra.run.smithi114.stdout: "entity": { 2024-04-23T10:32:04.715 INFO:teuthology.orchestra.run.smithi114.stdout: "type": "mds", 2024-04-23T10:32:04.715 INFO:teuthology.orchestra.run.smithi114.stdout: "num": 0 2024-04-23T10:32:04.715 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-23T10:32:04.715 INFO:teuthology.orchestra.run.smithi114.stdout: "tid": 171 2024-04-23T10:32:04.715 INFO:teuthology.orchestra.run.smithi114.stdout: } 2024-04-23T10:32:04.715 INFO:teuthology.orchestra.run.smithi114.stdout: } 2024-04-23T10:32:04.715 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-23T10:32:04.715 INFO:teuthology.orchestra.run.smithi114.stdout: "flags": 4, 2024-04-23T10:32:04.715 INFO:teuthology.orchestra.run.smithi114.stdout: "wrlock_target": -1 2024-04-23T10:32:04.715 INFO:teuthology.orchestra.run.smithi114.stdout: } 2024-04-23T10:32:04.716 INFO:teuthology.orchestra.run.smithi114.stdout: ] 2024-04-23T10:32:04.716 INFO:teuthology.orchestra.run.smithi114.stdout: } 2024-04-23T10:32:04.716 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-23T10:32:04.716 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-04-23T10:32:04.716 INFO:teuthology.orchestra.run.smithi114.stdout: "description": "internal op quiesce_inode:mds.0:196 fp=#0x1000000021c", 2024-04-23T10:32:04.716 INFO:teuthology.orchestra.run.smithi114.stdout: "initiated_at": "2024-04-23T10:31:21.483464+0000", 2024-04-23T10:32:04.716 INFO:teuthology.orchestra.run.smithi114.stdout: "age": 42.400924844000002, 2024-04-23T10:32:04.716 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 42.403269293999998, 2024-04-23T10:32:04.716 INFO:teuthology.orchestra.run.smithi114.stdout: "continuous": true, 2024-04-23T10:32:04.716 INFO:teuthology.orchestra.run.smithi114.stdout: "type_data": { 2024-04-23T10:32:04.716 INFO:teuthology.orchestra.run.smithi114.stdout: "result": -2147483648, 2024-04-23T10:32:04.716 INFO:teuthology.orchestra.run.smithi114.stdout: "flag_point": "quiesce complete for non-auth inode", 2024-04-23T10:32:04.716 INFO:teuthology.orchestra.run.smithi114.stdout: "reqid": { 2024-04-23T10:32:04.716 INFO:teuthology.orchestra.run.smithi114.stdout: "entity": { 2024-04-23T10:32:04.716 INFO:teuthology.orchestra.run.smithi114.stdout: "type": "mds", 2024-04-23T10:32:04.717 INFO:teuthology.orchestra.run.smithi114.stdout: "num": 0 2024-04-23T10:32:04.717 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-23T10:32:04.717 INFO:teuthology.orchestra.run.smithi114.stdout: "tid": 196 2024-04-23T10:32:04.717 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-23T10:32:04.717 INFO:teuthology.orchestra.run.smithi114.stdout: "op_type": "internal_op", 2024-04-23T10:32:04.717 INFO:teuthology.orchestra.run.smithi114.stdout: "internal_op": 5385, 2024-04-23T10:32:04.717 INFO:teuthology.orchestra.run.smithi114.stdout: "op_name": "quiesce_inode", 2024-04-23T10:32:04.717 INFO:teuthology.orchestra.run.smithi114.stdout: "events": [ 2024-04-23T10:32:04.717 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-04-23T10:32:04.717 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-04-23T10:31:21.483464+0000", 2024-04-23T10:32:04.717 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "initiated" 2024-04-23T10:32:04.717 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-23T10:32:04.717 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-04-23T10:32:04.717 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-04-23T10:31:21.483464+0000", 2024-04-23T10:32:04.717 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "header_read" 2024-04-23T10:32:04.717 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-23T10:32:04.718 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-04-23T10:32:04.718 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-04-23T10:31:21.483464+0000", 2024-04-23T10:32:04.718 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "throttled" 2024-04-23T10:32:04.718 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-23T10:32:04.718 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-04-23T10:32:04.718 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-04-23T10:31:21.483464+0000", 2024-04-23T10:32:04.718 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "all_read" 2024-04-23T10:32:04.718 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-23T10:32:04.718 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-04-23T10:32:04.718 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-04-23T10:31:21.483464+0000", 2024-04-23T10:32:04.718 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "dispatched" 2024-04-23T10:32:04.718 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-23T10:32:04.718 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-04-23T10:32:04.718 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-04-23T10:31:21.483894+0000", 2024-04-23T10:32:04.718 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "requesting remote authpins" 2024-04-23T10:32:04.719 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-23T10:32:04.719 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-04-23T10:32:04.719 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-04-23T10:31:21.507796+0000", 2024-04-23T10:32:04.719 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "acquired locks" 2024-04-23T10:32:04.719 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-23T10:32:04.719 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-04-23T10:32:04.719 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-04-23T10:31:21.507805+0000", 2024-04-23T10:32:04.719 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "quiesce complete for non-auth inode" 2024-04-23T10:32:04.719 INFO:teuthology.orchestra.run.smithi114.stdout: } 2024-04-23T10:32:04.719 INFO:teuthology.orchestra.run.smithi114.stdout: ], 2024-04-23T10:32:04.719 INFO:teuthology.orchestra.run.smithi114.stdout: "locks": [ 2024-04-23T10:32:04.719 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-04-23T10:32:04.719 INFO:teuthology.orchestra.run.smithi114.stdout: "object": { 2024-04-23T10:32:04.719 INFO:teuthology.orchestra.run.smithi114.stdout: "is_auth": false, 2024-04-23T10:32:04.719 INFO:teuthology.orchestra.run.smithi114.stdout: "auth_state": { 2024-04-23T10:32:04.720 INFO:teuthology.orchestra.run.smithi114.stdout: "replicas": {} 2024-04-23T10:32:04.720 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-23T10:32:04.720 INFO:teuthology.orchestra.run.smithi114.stdout: "replica_state": { 2024-04-23T10:32:04.720 INFO:teuthology.orchestra.run.smithi114.stdout: "authority": [ 2024-04-23T10:32:04.720 INFO:teuthology.orchestra.run.smithi114.stdout: 2, 2024-04-23T10:32:04.720 INFO:teuthology.orchestra.run.smithi114.stdout: -2 2024-04-23T10:32:04.720 INFO:teuthology.orchestra.run.smithi114.stdout: ], 2024-04-23T10:32:04.720 INFO:teuthology.orchestra.run.smithi114.stdout: "replica_nonce": 1 2024-04-23T10:32:04.720 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-23T10:32:04.720 INFO:teuthology.orchestra.run.smithi114.stdout: "auth_pins": 0, 2024-04-23T10:32:04.720 INFO:teuthology.orchestra.run.smithi114.stdout: "is_frozen": false, 2024-04-23T10:32:04.720 INFO:teuthology.orchestra.run.smithi114.stdout: "is_freezing": false, 2024-04-23T10:32:04.720 INFO:teuthology.orchestra.run.smithi114.stdout: "pins": { 2024-04-23T10:32:04.720 INFO:teuthology.orchestra.run.smithi114.stdout: "request": 1, 2024-04-23T10:32:04.720 INFO:teuthology.orchestra.run.smithi114.stdout: "lock": 1, 2024-04-23T10:32:04.721 INFO:teuthology.orchestra.run.smithi114.stdout: "caps": 0, 2024-04-23T10:32:04.721 INFO:teuthology.orchestra.run.smithi114.stdout: "exportingcaps": 0, 2024-04-23T10:32:04.721 INFO:teuthology.orchestra.run.smithi114.stdout: "dirtyparent": 0, 2024-04-23T10:32:04.721 INFO:teuthology.orchestra.run.smithi114.stdout: "dirty": 0, 2024-04-23T10:32:04.721 INFO:teuthology.orchestra.run.smithi114.stdout: "authpin": 0, 2024-04-23T10:32:04.721 INFO:teuthology.orchestra.run.smithi114.stdout: "tempexporting": 0 2024-04-23T10:32:04.721 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-23T10:32:04.721 INFO:teuthology.orchestra.run.smithi114.stdout: "nref": 2 2024-04-23T10:32:04.721 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-23T10:32:04.721 INFO:teuthology.orchestra.run.smithi114.stdout: "object_string": "[inode 0x1000000021c [7,head] /volumes/qa/sv_1/b206794e-7622-4ab8-8943-f3ae4e65b7a3/client.0/tmp/clients/client0/~dmtmp/COREL/SIGN2.CPT rep@2.1 v7250 s=260096 n(v0 rc2024-04-23T10:27:52.033396+0000 b260096 1=1+0)/n(v0 1=1+0) (iquiesce lock x=1 by request(mds.0:196 nref=3)) | request=1 lock=1 caps=0 exportingcaps=0 dirtyparent=0 dirty=0 authpin=0 tempexporting=0 0x5571cf562100]", 2024-04-23T10:32:04.721 INFO:teuthology.orchestra.run.smithi114.stdout: "lock": { 2024-04-23T10:32:04.721 INFO:teuthology.orchestra.run.smithi114.stdout: "gather_set": [], 2024-04-23T10:32:04.721 INFO:teuthology.orchestra.run.smithi114.stdout: "state": "lock", 2024-04-23T10:32:04.721 INFO:teuthology.orchestra.run.smithi114.stdout: "type": "iquiesce", 2024-04-23T10:32:04.722 INFO:teuthology.orchestra.run.smithi114.stdout: "is_leased": false, 2024-04-23T10:32:04.722 INFO:teuthology.orchestra.run.smithi114.stdout: "num_rdlocks": 0, 2024-04-23T10:32:04.722 INFO:teuthology.orchestra.run.smithi114.stdout: "num_wrlocks": 0, 2024-04-23T10:32:04.722 INFO:teuthology.orchestra.run.smithi114.stdout: "num_xlocks": 1, 2024-04-23T10:32:04.722 INFO:teuthology.orchestra.run.smithi114.stdout: "xlock_by": { 2024-04-23T10:32:04.722 INFO:teuthology.orchestra.run.smithi114.stdout: "reqid": { 2024-04-23T10:32:04.722 INFO:teuthology.orchestra.run.smithi114.stdout: "entity": { 2024-04-23T10:32:04.722 INFO:teuthology.orchestra.run.smithi114.stdout: "type": "mds", 2024-04-23T10:32:04.722 INFO:teuthology.orchestra.run.smithi114.stdout: "num": 0 2024-04-23T10:32:04.722 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-23T10:32:04.722 INFO:teuthology.orchestra.run.smithi114.stdout: "tid": 196 2024-04-23T10:32:04.722 INFO:teuthology.orchestra.run.smithi114.stdout: } 2024-04-23T10:32:04.722 INFO:teuthology.orchestra.run.smithi114.stdout: } 2024-04-23T10:32:04.722 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-23T10:32:04.723 INFO:teuthology.orchestra.run.smithi114.stdout: "flags": 4, 2024-04-23T10:32:04.723 INFO:teuthology.orchestra.run.smithi114.stdout: "wrlock_target": -1 2024-04-23T10:32:04.723 INFO:teuthology.orchestra.run.smithi114.stdout: } 2024-04-23T10:32:04.723 INFO:teuthology.orchestra.run.smithi114.stdout: ] 2024-04-23T10:32:04.723 INFO:teuthology.orchestra.run.smithi114.stdout: } 2024-04-23T10:32:04.723 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-23T10:32:04.723 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-04-23T10:32:04.723 INFO:teuthology.orchestra.run.smithi114.stdout: "description": "internal op quiesce_inode:mds.0:153 fp=#0x100000001fa", 2024-04-23T10:32:04.723 INFO:teuthology.orchestra.run.smithi114.stdout: "initiated_at": "2024-04-23T10:31:21.448695+0000", 2024-04-23T10:32:04.723 INFO:teuthology.orchestra.run.smithi114.stdout: "age": 42.435693784999998, 2024-04-23T10:32:04.723 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 42.438098199000002, 2024-04-23T10:32:04.723 INFO:teuthology.orchestra.run.smithi114.stdout: "continuous": true, 2024-04-23T10:32:04.723 INFO:teuthology.orchestra.run.smithi114.stdout: "type_data": { 2024-04-23T10:32:04.723 INFO:teuthology.orchestra.run.smithi114.stdout: "result": -2147483648, 2024-04-23T10:32:04.723 INFO:teuthology.orchestra.run.smithi114.stdout: "flag_point": "quiesce complete", 2024-04-23T10:32:04.723 INFO:teuthology.orchestra.run.smithi114.stdout: "reqid": { 2024-04-23T10:32:04.724 INFO:teuthology.orchestra.run.smithi114.stdout: "entity": { 2024-04-23T10:32:04.724 INFO:teuthology.orchestra.run.smithi114.stdout: "type": "mds", 2024-04-23T10:32:04.724 INFO:teuthology.orchestra.run.smithi114.stdout: "num": 0 2024-04-23T10:32:04.724 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-23T10:32:04.724 INFO:teuthology.orchestra.run.smithi114.stdout: "tid": 153 2024-04-23T10:32:04.724 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-23T10:32:04.724 INFO:teuthology.orchestra.run.smithi114.stdout: "op_type": "internal_op", 2024-04-23T10:32:04.724 INFO:teuthology.orchestra.run.smithi114.stdout: "internal_op": 5385, 2024-04-23T10:32:04.724 INFO:teuthology.orchestra.run.smithi114.stdout: "op_name": "quiesce_inode", 2024-04-23T10:32:04.724 INFO:teuthology.orchestra.run.smithi114.stdout: "events": [ 2024-04-23T10:32:04.724 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-04-23T10:32:04.724 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-04-23T10:31:21.448695+0000", 2024-04-23T10:32:04.724 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "initiated" 2024-04-23T10:32:04.724 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-23T10:32:04.725 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-04-23T10:32:04.725 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-04-23T10:31:21.448695+0000", 2024-04-23T10:32:04.725 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "header_read" 2024-04-23T10:32:04.725 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-23T10:32:04.725 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-04-23T10:32:04.725 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-04-23T10:31:21.448695+0000", 2024-04-23T10:32:04.725 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "throttled" 2024-04-23T10:32:04.725 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-23T10:32:04.725 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-04-23T10:32:04.725 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-04-23T10:31:21.448695+0000", 2024-04-23T10:32:04.725 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "all_read" 2024-04-23T10:32:04.725 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-23T10:32:04.725 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-04-23T10:32:04.725 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-04-23T10:31:21.448695+0000", 2024-04-23T10:32:04.725 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "dispatched" 2024-04-23T10:32:04.726 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-23T10:32:04.726 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-04-23T10:32:04.726 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-04-23T10:31:21.448924+0000", 2024-04-23T10:32:04.726 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "failed to xlock, waiting" 2024-04-23T10:32:04.726 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-23T10:32:04.726 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-04-23T10:32:04.726 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-04-23T10:31:21.449710+0000", 2024-04-23T10:32:04.726 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "acquired locks" 2024-04-23T10:32:04.726 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-23T10:32:04.726 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-04-23T10:32:04.726 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-04-23T10:31:21.509989+0000", 2024-04-23T10:32:04.726 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "quiesce complete" 2024-04-23T10:32:04.726 INFO:teuthology.orchestra.run.smithi114.stdout: } 2024-04-23T10:32:04.726 INFO:teuthology.orchestra.run.smithi114.stdout: ], 2024-04-23T10:32:04.727 INFO:teuthology.orchestra.run.smithi114.stdout: "locks": [ 2024-04-23T10:32:04.727 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-04-23T10:32:04.727 INFO:teuthology.orchestra.run.smithi114.stdout: "object": { 2024-04-23T10:32:04.727 INFO:teuthology.orchestra.run.smithi114.stdout: "is_auth": true, 2024-04-23T10:32:04.727 INFO:teuthology.orchestra.run.smithi114.stdout: "auth_state": { 2024-04-23T10:32:04.727 INFO:teuthology.orchestra.run.smithi114.stdout: "replicas": { 2024-04-23T10:32:04.727 INFO:teuthology.orchestra.run.smithi114.stdout: "1": 1, 2024-04-23T10:32:04.727 INFO:teuthology.orchestra.run.smithi114.stdout: "2": 1 2024-04-23T10:32:04.727 INFO:teuthology.orchestra.run.smithi114.stdout: } 2024-04-23T10:32:04.727 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-23T10:32:04.727 INFO:teuthology.orchestra.run.smithi114.stdout: "replica_state": { 2024-04-23T10:32:04.727 INFO:teuthology.orchestra.run.smithi114.stdout: "authority": [ 2024-04-23T10:32:04.727 INFO:teuthology.orchestra.run.smithi114.stdout: 0, 2024-04-23T10:32:04.727 INFO:teuthology.orchestra.run.smithi114.stdout: -2 2024-04-23T10:32:04.727 INFO:teuthology.orchestra.run.smithi114.stdout: ], 2024-04-23T10:32:04.728 INFO:teuthology.orchestra.run.smithi114.stdout: "replica_nonce": 0 2024-04-23T10:32:04.728 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-23T10:32:04.728 INFO:teuthology.orchestra.run.smithi114.stdout: "auth_pins": 3, 2024-04-23T10:32:04.728 INFO:teuthology.orchestra.run.smithi114.stdout: "is_frozen": false, 2024-04-23T10:32:04.728 INFO:teuthology.orchestra.run.smithi114.stdout: "is_freezing": false, 2024-04-23T10:32:04.728 INFO:teuthology.orchestra.run.smithi114.stdout: "pins": { 2024-04-23T10:32:04.728 INFO:teuthology.orchestra.run.smithi114.stdout: "dirtyscattered": 1, 2024-04-23T10:32:04.728 INFO:teuthology.orchestra.run.smithi114.stdout: "request": 1, 2024-04-23T10:32:04.728 INFO:teuthology.orchestra.run.smithi114.stdout: "lock": 1, 2024-04-23T10:32:04.728 INFO:teuthology.orchestra.run.smithi114.stdout: "dirfrag": 1, 2024-04-23T10:32:04.728 INFO:teuthology.orchestra.run.smithi114.stdout: "caps": 0, 2024-04-23T10:32:04.728 INFO:teuthology.orchestra.run.smithi114.stdout: "dirtyrstat": 0, 2024-04-23T10:32:04.728 INFO:teuthology.orchestra.run.smithi114.stdout: "dirtyparent": 0, 2024-04-23T10:32:04.728 INFO:teuthology.orchestra.run.smithi114.stdout: "replicated": 1, 2024-04-23T10:32:04.728 INFO:teuthology.orchestra.run.smithi114.stdout: "dirty": 1, 2024-04-23T10:32:04.729 INFO:teuthology.orchestra.run.smithi114.stdout: "waiter": 0, 2024-04-23T10:32:04.729 INFO:teuthology.orchestra.run.smithi114.stdout: "authpin": 1 2024-04-23T10:32:04.729 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-23T10:32:04.729 INFO:teuthology.orchestra.run.smithi114.stdout: "nref": 7 2024-04-23T10:32:04.729 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-23T10:32:04.729 INFO:teuthology.orchestra.run.smithi114.stdout: "object_string": "[inode 0x100000001fa [...7,head] /volumes/qa/sv_1/ auth{1=1,2=1} v436 ap=3 snaprealm=0x5571cf174000 f(v0 m2024-04-23T10:24:49.735055+0000 2=1+1) n(v29 rc2024-04-23T10:31:17.764716+0000 b21075191 80=64+16) old_inodes=5 (inest mix dirty) (iquiesce lock x=1 by request(mds.0:153 nref=3)) | dirtyscattered=1 request=1 lock=1 dirfrag=1 caps=0 dirtyrstat=0 dirtyparent=0 replicated=1 dirty=1 waiter=0 authpin=1 0x5571cf1dfb80]", 2024-04-23T10:32:04.729 INFO:teuthology.orchestra.run.smithi114.stdout: "lock": { 2024-04-23T10:32:04.729 INFO:teuthology.orchestra.run.smithi114.stdout: "gather_set": [], 2024-04-23T10:32:04.729 INFO:teuthology.orchestra.run.smithi114.stdout: "state": "lock", 2024-04-23T10:32:04.729 INFO:teuthology.orchestra.run.smithi114.stdout: "type": "iquiesce", 2024-04-23T10:32:04.729 INFO:teuthology.orchestra.run.smithi114.stdout: "is_leased": false, 2024-04-23T10:32:04.729 INFO:teuthology.orchestra.run.smithi114.stdout: "num_rdlocks": 0, 2024-04-23T10:32:04.729 INFO:teuthology.orchestra.run.smithi114.stdout: "num_wrlocks": 0, 2024-04-23T10:32:04.729 INFO:teuthology.orchestra.run.smithi114.stdout: "num_xlocks": 1, 2024-04-23T10:32:04.729 INFO:teuthology.orchestra.run.smithi114.stdout: "xlock_by": { 2024-04-23T10:32:04.730 INFO:teuthology.orchestra.run.smithi114.stdout: "reqid": { 2024-04-23T10:32:04.730 INFO:teuthology.orchestra.run.smithi114.stdout: "entity": { 2024-04-23T10:32:04.730 INFO:teuthology.orchestra.run.smithi114.stdout: "type": "mds", 2024-04-23T10:32:04.730 INFO:teuthology.orchestra.run.smithi114.stdout: "num": 0 2024-04-23T10:32:04.730 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-23T10:32:04.730 INFO:teuthology.orchestra.run.smithi114.stdout: "tid": 153 2024-04-23T10:32:04.730 INFO:teuthology.orchestra.run.smithi114.stdout: } 2024-04-23T10:32:04.730 INFO:teuthology.orchestra.run.smithi114.stdout: } 2024-04-23T10:32:04.730 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-23T10:32:04.730 INFO:teuthology.orchestra.run.smithi114.stdout: "flags": 4, 2024-04-23T10:32:04.730 INFO:teuthology.orchestra.run.smithi114.stdout: "wrlock_target": -1 2024-04-23T10:32:04.730 INFO:teuthology.orchestra.run.smithi114.stdout: } 2024-04-23T10:32:04.730 INFO:teuthology.orchestra.run.smithi114.stdout: ] 2024-04-23T10:32:04.730 INFO:teuthology.orchestra.run.smithi114.stdout: } 2024-04-23T10:32:04.731 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-23T10:32:04.731 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-04-23T10:32:04.731 INFO:teuthology.orchestra.run.smithi114.stdout: "description": "internal op quiesce_inode:mds.0:172 fp=#0x10000000225", 2024-04-23T10:32:04.731 INFO:teuthology.orchestra.run.smithi114.stdout: "initiated_at": "2024-04-23T10:31:21.459931+0000", 2024-04-23T10:32:04.731 INFO:teuthology.orchestra.run.smithi114.stdout: "age": 42.424458305000002, 2024-04-23T10:32:04.731 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 42.426927628000001, 2024-04-23T10:32:04.731 INFO:teuthology.orchestra.run.smithi114.stdout: "continuous": true, 2024-04-23T10:32:04.731 INFO:teuthology.orchestra.run.smithi114.stdout: "type_data": { 2024-04-23T10:32:04.731 INFO:teuthology.orchestra.run.smithi114.stdout: "result": -2147483648, 2024-04-23T10:32:04.731 INFO:teuthology.orchestra.run.smithi114.stdout: "flag_point": "quiesce complete", 2024-04-23T10:32:04.731 INFO:teuthology.orchestra.run.smithi114.stdout: "reqid": { 2024-04-23T10:32:04.731 INFO:teuthology.orchestra.run.smithi114.stdout: "entity": { 2024-04-23T10:32:04.731 INFO:teuthology.orchestra.run.smithi114.stdout: "type": "mds", 2024-04-23T10:32:04.732 INFO:teuthology.orchestra.run.smithi114.stdout: "num": 0 2024-04-23T10:32:04.732 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-23T10:32:04.732 INFO:teuthology.orchestra.run.smithi114.stdout: "tid": 172 2024-04-23T10:32:04.732 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-23T10:32:04.732 INFO:teuthology.orchestra.run.smithi114.stdout: "op_type": "internal_op", 2024-04-23T10:32:04.732 INFO:teuthology.orchestra.run.smithi114.stdout: "internal_op": 5385, 2024-04-23T10:32:04.732 INFO:teuthology.orchestra.run.smithi114.stdout: "op_name": "quiesce_inode", 2024-04-23T10:32:04.732 INFO:teuthology.orchestra.run.smithi114.stdout: "events": [ 2024-04-23T10:32:04.732 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-04-23T10:32:04.732 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-04-23T10:31:21.459931+0000", 2024-04-23T10:32:04.732 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "initiated" 2024-04-23T10:32:04.732 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-23T10:32:04.732 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-04-23T10:32:04.732 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-04-23T10:31:21.459931+0000", 2024-04-23T10:32:04.732 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "header_read" 2024-04-23T10:32:04.733 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-23T10:32:04.733 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-04-23T10:32:04.733 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-04-23T10:31:21.459931+0000", 2024-04-23T10:32:04.733 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "throttled" 2024-04-23T10:32:04.733 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-23T10:32:04.733 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-04-23T10:32:04.733 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-04-23T10:31:21.459931+0000", 2024-04-23T10:32:04.733 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "all_read" 2024-04-23T10:32:04.733 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-23T10:32:04.733 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-04-23T10:32:04.733 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-04-23T10:31:21.459931+0000", 2024-04-23T10:32:04.733 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "dispatched" 2024-04-23T10:32:04.733 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-23T10:32:04.733 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-04-23T10:32:04.733 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-04-23T10:31:21.462138+0000", 2024-04-23T10:32:04.734 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "failed to xlock, waiting" 2024-04-23T10:32:04.734 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-23T10:32:04.734 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-04-23T10:32:04.734 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-04-23T10:31:21.486438+0000", 2024-04-23T10:32:04.734 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "acquired locks" 2024-04-23T10:32:04.734 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-23T10:32:04.734 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-04-23T10:32:04.734 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-04-23T10:31:21.486778+0000", 2024-04-23T10:32:04.734 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "quiesce complete" 2024-04-23T10:32:04.734 INFO:teuthology.orchestra.run.smithi114.stdout: } 2024-04-23T10:32:04.734 INFO:teuthology.orchestra.run.smithi114.stdout: ], 2024-04-23T10:32:04.734 INFO:teuthology.orchestra.run.smithi114.stdout: "locks": [ 2024-04-23T10:32:04.734 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-04-23T10:32:04.734 INFO:teuthology.orchestra.run.smithi114.stdout: "object": { 2024-04-23T10:32:04.734 INFO:teuthology.orchestra.run.smithi114.stdout: "is_auth": true, 2024-04-23T10:32:04.735 INFO:teuthology.orchestra.run.smithi114.stdout: "auth_state": { 2024-04-23T10:32:04.735 INFO:teuthology.orchestra.run.smithi114.stdout: "replicas": { 2024-04-23T10:32:04.735 INFO:teuthology.orchestra.run.smithi114.stdout: "1": 1, 2024-04-23T10:32:04.735 INFO:teuthology.orchestra.run.smithi114.stdout: "2": 1 2024-04-23T10:32:04.735 INFO:teuthology.orchestra.run.smithi114.stdout: } 2024-04-23T10:32:04.735 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-23T10:32:04.735 INFO:teuthology.orchestra.run.smithi114.stdout: "replica_state": { 2024-04-23T10:32:04.735 INFO:teuthology.orchestra.run.smithi114.stdout: "authority": [ 2024-04-23T10:32:04.735 INFO:teuthology.orchestra.run.smithi114.stdout: 0, 2024-04-23T10:32:04.735 INFO:teuthology.orchestra.run.smithi114.stdout: -2 2024-04-23T10:32:04.735 INFO:teuthology.orchestra.run.smithi114.stdout: ], 2024-04-23T10:32:04.735 INFO:teuthology.orchestra.run.smithi114.stdout: "replica_nonce": 0 2024-04-23T10:32:04.735 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-23T10:32:04.736 INFO:teuthology.orchestra.run.smithi114.stdout: "auth_pins": 3, 2024-04-23T10:32:04.736 INFO:teuthology.orchestra.run.smithi114.stdout: "is_frozen": false, 2024-04-23T10:32:04.736 INFO:teuthology.orchestra.run.smithi114.stdout: "is_freezing": false, 2024-04-23T10:32:04.736 INFO:teuthology.orchestra.run.smithi114.stdout: "pins": { 2024-04-23T10:32:04.736 INFO:teuthology.orchestra.run.smithi114.stdout: "dirtyscattered": 0, 2024-04-23T10:32:04.736 INFO:teuthology.orchestra.run.smithi114.stdout: "request": 1, 2024-04-23T10:32:04.736 INFO:teuthology.orchestra.run.smithi114.stdout: "lock": 1, 2024-04-23T10:32:04.736 INFO:teuthology.orchestra.run.smithi114.stdout: "dirfrag": 1, 2024-04-23T10:32:04.736 INFO:teuthology.orchestra.run.smithi114.stdout: "caps": 1, 2024-04-23T10:32:04.736 INFO:teuthology.orchestra.run.smithi114.stdout: "dirtyrstat": 0, 2024-04-23T10:32:04.736 INFO:teuthology.orchestra.run.smithi114.stdout: "dirtyparent": 0, 2024-04-23T10:32:04.736 INFO:teuthology.orchestra.run.smithi114.stdout: "replicated": 1, 2024-04-23T10:32:04.736 INFO:teuthology.orchestra.run.smithi114.stdout: "dirty": 0, 2024-04-23T10:32:04.736 INFO:teuthology.orchestra.run.smithi114.stdout: "waiter": 0, 2024-04-23T10:32:04.736 INFO:teuthology.orchestra.run.smithi114.stdout: "authpin": 1 2024-04-23T10:32:04.737 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-23T10:32:04.737 INFO:teuthology.orchestra.run.smithi114.stdout: "nref": 6 2024-04-23T10:32:04.737 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-23T10:32:04.737 INFO:teuthology.orchestra.run.smithi114.stdout: "object_string": "[inode 0x10000000225 [...7,head] /volumes/qa/sv_1/b206794e-7622-4ab8-8943-f3ae4e65b7a3/client.0/tmp/clients/client0/~dmtmp/PM/ auth{1=1,2=1} v45574 ap=3 f(v2926 m2024-04-23T10:31:17.764716+0000 3=3+0) n(v17 rc2024-04-23T10:31:17.764716+0000 b780288 4=3+1) old_inodes=4 (inest mix) (iquiesce lock x=1 by request(mds.0:172 nref=3)) caps={15312=pAsLsXsFs/pAsLsXsFsxcral@7614} mcw={2=AsLsXsFs} | dirtyscattered=0 request=1 lock=1 dirfrag=1 caps=1 dirtyrstat=0 dirtyparent=0 replicated=1 dirty=0 waiter=0 authpin=1 0x5571cf587080]", 2024-04-23T10:32:04.737 INFO:teuthology.orchestra.run.smithi114.stdout: "lock": { 2024-04-23T10:32:04.737 INFO:teuthology.orchestra.run.smithi114.stdout: "gather_set": [], 2024-04-23T10:32:04.737 INFO:teuthology.orchestra.run.smithi114.stdout: "state": "lock", 2024-04-23T10:32:04.737 INFO:teuthology.orchestra.run.smithi114.stdout: "type": "iquiesce", 2024-04-23T10:32:04.737 INFO:teuthology.orchestra.run.smithi114.stdout: "is_leased": false, 2024-04-23T10:32:04.737 INFO:teuthology.orchestra.run.smithi114.stdout: "num_rdlocks": 0, 2024-04-23T10:32:04.737 INFO:teuthology.orchestra.run.smithi114.stdout: "num_wrlocks": 0, 2024-04-23T10:32:04.737 INFO:teuthology.orchestra.run.smithi114.stdout: "num_xlocks": 1, 2024-04-23T10:32:04.737 INFO:teuthology.orchestra.run.smithi114.stdout: "xlock_by": { 2024-04-23T10:32:04.737 INFO:teuthology.orchestra.run.smithi114.stdout: "reqid": { 2024-04-23T10:32:04.737 INFO:teuthology.orchestra.run.smithi114.stdout: "entity": { 2024-04-23T10:32:04.738 INFO:teuthology.orchestra.run.smithi114.stdout: "type": "mds", 2024-04-23T10:32:04.738 INFO:teuthology.orchestra.run.smithi114.stdout: "num": 0 2024-04-23T10:32:04.738 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-23T10:32:04.738 INFO:teuthology.orchestra.run.smithi114.stdout: "tid": 172 2024-04-23T10:32:04.738 INFO:teuthology.orchestra.run.smithi114.stdout: } 2024-04-23T10:32:04.738 INFO:teuthology.orchestra.run.smithi114.stdout: } 2024-04-23T10:32:04.738 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-23T10:32:04.738 INFO:teuthology.orchestra.run.smithi114.stdout: "flags": 4, 2024-04-23T10:32:04.738 INFO:teuthology.orchestra.run.smithi114.stdout: "wrlock_target": -1 2024-04-23T10:32:04.738 INFO:teuthology.orchestra.run.smithi114.stdout: } 2024-04-23T10:32:04.738 INFO:teuthology.orchestra.run.smithi114.stdout: ] 2024-04-23T10:32:04.738 INFO:teuthology.orchestra.run.smithi114.stdout: } 2024-04-23T10:32:04.738 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-23T10:32:04.738 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-04-23T10:32:04.738 INFO:teuthology.orchestra.run.smithi114.stdout: "description": "internal op quiesce_inode:mds.0:197 fp=#0x10000000217", 2024-04-23T10:32:04.739 INFO:teuthology.orchestra.run.smithi114.stdout: "initiated_at": "2024-04-23T10:31:21.483478+0000", 2024-04-23T10:32:04.739 INFO:teuthology.orchestra.run.smithi114.stdout: "age": 42.400911235000002, 2024-04-23T10:32:04.739 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 42.403449692000002, 2024-04-23T10:32:04.739 INFO:teuthology.orchestra.run.smithi114.stdout: "continuous": true, 2024-04-23T10:32:04.739 INFO:teuthology.orchestra.run.smithi114.stdout: "type_data": { 2024-04-23T10:32:04.739 INFO:teuthology.orchestra.run.smithi114.stdout: "result": -2147483648, 2024-04-23T10:32:04.739 INFO:teuthology.orchestra.run.smithi114.stdout: "flag_point": "quiesce complete for non-auth inode", 2024-04-23T10:32:04.739 INFO:teuthology.orchestra.run.smithi114.stdout: "reqid": { 2024-04-23T10:32:04.739 INFO:teuthology.orchestra.run.smithi114.stdout: "entity": { 2024-04-23T10:32:04.739 INFO:teuthology.orchestra.run.smithi114.stdout: "type": "mds", 2024-04-23T10:32:04.739 INFO:teuthology.orchestra.run.smithi114.stdout: "num": 0 2024-04-23T10:32:04.739 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-23T10:32:04.739 INFO:teuthology.orchestra.run.smithi114.stdout: "tid": 197 2024-04-23T10:32:04.739 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-23T10:32:04.739 INFO:teuthology.orchestra.run.smithi114.stdout: "op_type": "internal_op", 2024-04-23T10:32:04.739 INFO:teuthology.orchestra.run.smithi114.stdout: "internal_op": 5385, 2024-04-23T10:32:04.740 INFO:teuthology.orchestra.run.smithi114.stdout: "op_name": "quiesce_inode", 2024-04-23T10:32:04.740 INFO:teuthology.orchestra.run.smithi114.stdout: "events": [ 2024-04-23T10:32:04.740 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-04-23T10:32:04.740 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-04-23T10:31:21.483478+0000", 2024-04-23T10:32:04.740 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "initiated" 2024-04-23T10:32:04.740 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-23T10:32:04.740 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-04-23T10:32:04.740 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-04-23T10:31:21.483478+0000", 2024-04-23T10:32:04.740 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "header_read" 2024-04-23T10:32:04.740 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-23T10:32:04.740 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-04-23T10:32:04.740 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-04-23T10:31:21.483478+0000", 2024-04-23T10:32:04.740 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "throttled" 2024-04-23T10:32:04.740 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-23T10:32:04.740 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-04-23T10:32:04.741 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-04-23T10:31:21.483478+0000", 2024-04-23T10:32:04.741 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "all_read" 2024-04-23T10:32:04.741 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-23T10:32:04.741 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-04-23T10:32:04.741 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-04-23T10:31:21.483478+0000", 2024-04-23T10:32:04.741 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "dispatched" 2024-04-23T10:32:04.741 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-23T10:32:04.741 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-04-23T10:32:04.741 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-04-23T10:31:21.483944+0000", 2024-04-23T10:32:04.741 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "requesting remote authpins" 2024-04-23T10:32:04.741 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-23T10:32:04.741 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-04-23T10:32:04.741 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-04-23T10:31:21.507889+0000", 2024-04-23T10:32:04.741 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "acquired locks" 2024-04-23T10:32:04.741 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-23T10:32:04.742 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-04-23T10:32:04.742 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-04-23T10:31:21.507899+0000", 2024-04-23T10:32:04.742 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "quiesce complete for non-auth inode" 2024-04-23T10:32:04.742 INFO:teuthology.orchestra.run.smithi114.stdout: } 2024-04-23T10:32:04.742 INFO:teuthology.orchestra.run.smithi114.stdout: ], 2024-04-23T10:32:04.742 INFO:teuthology.orchestra.run.smithi114.stdout: "locks": [ 2024-04-23T10:32:04.742 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-04-23T10:32:04.742 INFO:teuthology.orchestra.run.smithi114.stdout: "object": { 2024-04-23T10:32:04.742 INFO:teuthology.orchestra.run.smithi114.stdout: "is_auth": false, 2024-04-23T10:32:04.742 INFO:teuthology.orchestra.run.smithi114.stdout: "auth_state": { 2024-04-23T10:32:04.742 INFO:teuthology.orchestra.run.smithi114.stdout: "replicas": {} 2024-04-23T10:32:04.742 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-23T10:32:04.742 INFO:teuthology.orchestra.run.smithi114.stdout: "replica_state": { 2024-04-23T10:32:04.742 INFO:teuthology.orchestra.run.smithi114.stdout: "authority": [ 2024-04-23T10:32:04.742 INFO:teuthology.orchestra.run.smithi114.stdout: 2, 2024-04-23T10:32:04.743 INFO:teuthology.orchestra.run.smithi114.stdout: -2 2024-04-23T10:32:04.743 INFO:teuthology.orchestra.run.smithi114.stdout: ], 2024-04-23T10:32:04.743 INFO:teuthology.orchestra.run.smithi114.stdout: "replica_nonce": 1 2024-04-23T10:32:04.743 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-23T10:32:04.743 INFO:teuthology.orchestra.run.smithi114.stdout: "auth_pins": 0, 2024-04-23T10:32:04.743 INFO:teuthology.orchestra.run.smithi114.stdout: "is_frozen": false, 2024-04-23T10:32:04.743 INFO:teuthology.orchestra.run.smithi114.stdout: "is_freezing": false, 2024-04-23T10:32:04.743 INFO:teuthology.orchestra.run.smithi114.stdout: "pins": { 2024-04-23T10:32:04.743 INFO:teuthology.orchestra.run.smithi114.stdout: "request": 1, 2024-04-23T10:32:04.743 INFO:teuthology.orchestra.run.smithi114.stdout: "lock": 1, 2024-04-23T10:32:04.743 INFO:teuthology.orchestra.run.smithi114.stdout: "caps": 0, 2024-04-23T10:32:04.743 INFO:teuthology.orchestra.run.smithi114.stdout: "exportingcaps": 0, 2024-04-23T10:32:04.743 INFO:teuthology.orchestra.run.smithi114.stdout: "dirtyparent": 0, 2024-04-23T10:32:04.743 INFO:teuthology.orchestra.run.smithi114.stdout: "dirty": 0, 2024-04-23T10:32:04.743 INFO:teuthology.orchestra.run.smithi114.stdout: "authpin": 0, 2024-04-23T10:32:04.743 INFO:teuthology.orchestra.run.smithi114.stdout: "tempexporting": 0 2024-04-23T10:32:04.744 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-23T10:32:04.744 INFO:teuthology.orchestra.run.smithi114.stdout: "nref": 2 2024-04-23T10:32:04.744 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-23T10:32:04.744 INFO:teuthology.orchestra.run.smithi114.stdout: "object_string": "[inode 0x10000000217 [7,head] /volumes/qa/sv_1/b206794e-7622-4ab8-8943-f3ae4e65b7a3/client.0/tmp/clients/client0/~dmtmp/COREL/CDRBARS.CFG rep@2.1 v7236 s=260096 n(v0 rc2024-04-23T10:29:16.529476+0000 b260096 1=1+0)/n(v0 1=1+0) (iquiesce lock x=1 by request(mds.0:197 nref=3)) | request=1 lock=1 caps=0 exportingcaps=0 dirtyparent=0 dirty=0 authpin=0 tempexporting=0 0x5571cf560580]", 2024-04-23T10:32:04.744 INFO:teuthology.orchestra.run.smithi114.stdout: "lock": { 2024-04-23T10:32:04.744 INFO:teuthology.orchestra.run.smithi114.stdout: "gather_set": [], 2024-04-23T10:32:04.744 INFO:teuthology.orchestra.run.smithi114.stdout: "state": "lock", 2024-04-23T10:32:04.744 INFO:teuthology.orchestra.run.smithi114.stdout: "type": "iquiesce", 2024-04-23T10:32:04.744 INFO:teuthology.orchestra.run.smithi114.stdout: "is_leased": false, 2024-04-23T10:32:04.744 INFO:teuthology.orchestra.run.smithi114.stdout: "num_rdlocks": 0, 2024-04-23T10:32:04.744 INFO:teuthology.orchestra.run.smithi114.stdout: "num_wrlocks": 0, 2024-04-23T10:32:04.744 INFO:teuthology.orchestra.run.smithi114.stdout: "num_xlocks": 1, 2024-04-23T10:32:04.744 INFO:teuthology.orchestra.run.smithi114.stdout: "xlock_by": { 2024-04-23T10:32:04.744 INFO:teuthology.orchestra.run.smithi114.stdout: "reqid": { 2024-04-23T10:32:04.744 INFO:teuthology.orchestra.run.smithi114.stdout: "entity": { 2024-04-23T10:32:04.745 INFO:teuthology.orchestra.run.smithi114.stdout: "type": "mds", 2024-04-23T10:32:04.745 INFO:teuthology.orchestra.run.smithi114.stdout: "num": 0 2024-04-23T10:32:04.745 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-23T10:32:04.745 INFO:teuthology.orchestra.run.smithi114.stdout: "tid": 197 2024-04-23T10:32:04.745 INFO:teuthology.orchestra.run.smithi114.stdout: } 2024-04-23T10:32:04.745 INFO:teuthology.orchestra.run.smithi114.stdout: } 2024-04-23T10:32:04.745 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-23T10:32:04.745 INFO:teuthology.orchestra.run.smithi114.stdout: "flags": 4, 2024-04-23T10:32:04.745 INFO:teuthology.orchestra.run.smithi114.stdout: "wrlock_target": -1 2024-04-23T10:32:04.745 INFO:teuthology.orchestra.run.smithi114.stdout: } 2024-04-23T10:32:04.745 INFO:teuthology.orchestra.run.smithi114.stdout: ] 2024-04-23T10:32:04.745 INFO:teuthology.orchestra.run.smithi114.stdout: } 2024-04-23T10:32:04.745 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-23T10:32:04.745 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-04-23T10:32:04.745 INFO:teuthology.orchestra.run.smithi114.stdout: "description": "internal op quiesce_inode:mds.0:154 fp=#0x10000000002", 2024-04-23T10:32:04.746 INFO:teuthology.orchestra.run.smithi114.stdout: "initiated_at": "2024-04-23T10:31:21.448714+0000", 2024-04-23T10:32:04.746 INFO:teuthology.orchestra.run.smithi114.stdout: "age": 42.435675556, 2024-04-23T10:32:04.746 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 42.438274481999997, 2024-04-23T10:32:04.746 INFO:teuthology.orchestra.run.smithi114.stdout: "continuous": true, 2024-04-23T10:32:04.746 INFO:teuthology.orchestra.run.smithi114.stdout: "type_data": { 2024-04-23T10:32:04.746 INFO:teuthology.orchestra.run.smithi114.stdout: "result": -2147483648, 2024-04-23T10:32:04.746 INFO:teuthology.orchestra.run.smithi114.stdout: "flag_point": "quiesce complete", 2024-04-23T10:32:04.746 INFO:teuthology.orchestra.run.smithi114.stdout: "reqid": { 2024-04-23T10:32:04.746 INFO:teuthology.orchestra.run.smithi114.stdout: "entity": { 2024-04-23T10:32:04.746 INFO:teuthology.orchestra.run.smithi114.stdout: "type": "mds", 2024-04-23T10:32:04.746 INFO:teuthology.orchestra.run.smithi114.stdout: "num": 0 2024-04-23T10:32:04.746 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-23T10:32:04.746 INFO:teuthology.orchestra.run.smithi114.stdout: "tid": 154 2024-04-23T10:32:04.746 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-23T10:32:04.747 INFO:teuthology.orchestra.run.smithi114.stdout: "op_type": "internal_op", 2024-04-23T10:32:04.747 INFO:teuthology.orchestra.run.smithi114.stdout: "internal_op": 5385, 2024-04-23T10:32:04.747 INFO:teuthology.orchestra.run.smithi114.stdout: "op_name": "quiesce_inode", 2024-04-23T10:32:04.747 INFO:teuthology.orchestra.run.smithi114.stdout: "events": [ 2024-04-23T10:32:04.747 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-04-23T10:32:04.747 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-04-23T10:31:21.448714+0000", 2024-04-23T10:32:04.747 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "initiated" 2024-04-23T10:32:04.747 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-23T10:32:04.747 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-04-23T10:32:04.747 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-04-23T10:31:21.448714+0000", 2024-04-23T10:32:04.747 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "header_read" 2024-04-23T10:32:04.747 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-23T10:32:04.747 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-04-23T10:32:04.747 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-04-23T10:31:21.448714+0000", 2024-04-23T10:32:04.747 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "throttled" 2024-04-23T10:32:04.747 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-23T10:32:04.748 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-04-23T10:32:04.748 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-04-23T10:31:21.448714+0000", 2024-04-23T10:32:04.748 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "all_read" 2024-04-23T10:32:04.748 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-23T10:32:04.748 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-04-23T10:32:04.748 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-04-23T10:31:21.448714+0000", 2024-04-23T10:32:04.748 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "dispatched" 2024-04-23T10:32:04.748 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-23T10:32:04.748 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-04-23T10:32:04.748 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-04-23T10:31:21.449116+0000", 2024-04-23T10:32:04.748 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "failed to xlock, waiting" 2024-04-23T10:32:04.748 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-23T10:32:04.748 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-04-23T10:32:04.748 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-04-23T10:31:21.450377+0000", 2024-04-23T10:32:04.748 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "acquired locks" 2024-04-23T10:32:04.749 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-23T10:32:04.749 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-04-23T10:32:04.749 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-04-23T10:31:21.450570+0000", 2024-04-23T10:32:04.749 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "quiesce complete" 2024-04-23T10:32:04.749 INFO:teuthology.orchestra.run.smithi114.stdout: } 2024-04-23T10:32:04.749 INFO:teuthology.orchestra.run.smithi114.stdout: ], 2024-04-23T10:32:04.749 INFO:teuthology.orchestra.run.smithi114.stdout: "locks": [ 2024-04-23T10:32:04.749 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-04-23T10:32:04.749 INFO:teuthology.orchestra.run.smithi114.stdout: "object": { 2024-04-23T10:32:04.749 INFO:teuthology.orchestra.run.smithi114.stdout: "is_auth": true, 2024-04-23T10:32:04.749 INFO:teuthology.orchestra.run.smithi114.stdout: "auth_state": { 2024-04-23T10:32:04.749 INFO:teuthology.orchestra.run.smithi114.stdout: "replicas": { 2024-04-23T10:32:04.749 INFO:teuthology.orchestra.run.smithi114.stdout: "1": 1 2024-04-23T10:32:04.749 INFO:teuthology.orchestra.run.smithi114.stdout: } 2024-04-23T10:32:04.749 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-23T10:32:04.750 INFO:teuthology.orchestra.run.smithi114.stdout: "replica_state": { 2024-04-23T10:32:04.750 INFO:teuthology.orchestra.run.smithi114.stdout: "authority": [ 2024-04-23T10:32:04.750 INFO:teuthology.orchestra.run.smithi114.stdout: 0, 2024-04-23T10:32:04.750 INFO:teuthology.orchestra.run.smithi114.stdout: -2 2024-04-23T10:32:04.750 INFO:teuthology.orchestra.run.smithi114.stdout: ], 2024-04-23T10:32:04.750 INFO:teuthology.orchestra.run.smithi114.stdout: "replica_nonce": 0 2024-04-23T10:32:04.750 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-23T10:32:04.750 INFO:teuthology.orchestra.run.smithi114.stdout: "auth_pins": 2, 2024-04-23T10:32:04.750 INFO:teuthology.orchestra.run.smithi114.stdout: "is_frozen": false, 2024-04-23T10:32:04.750 INFO:teuthology.orchestra.run.smithi114.stdout: "is_freezing": false, 2024-04-23T10:32:04.750 INFO:teuthology.orchestra.run.smithi114.stdout: "pins": { 2024-04-23T10:32:04.750 INFO:teuthology.orchestra.run.smithi114.stdout: "request": 1, 2024-04-23T10:32:04.750 INFO:teuthology.orchestra.run.smithi114.stdout: "lock": 1, 2024-04-23T10:32:04.750 INFO:teuthology.orchestra.run.smithi114.stdout: "dirfrag": 1, 2024-04-23T10:32:04.751 INFO:teuthology.orchestra.run.smithi114.stdout: "caps": 0, 2024-04-23T10:32:04.751 INFO:teuthology.orchestra.run.smithi114.stdout: "dirtyparent": 0, 2024-04-23T10:32:04.751 INFO:teuthology.orchestra.run.smithi114.stdout: "replicated": 1, 2024-04-23T10:32:04.751 INFO:teuthology.orchestra.run.smithi114.stdout: "dirty": 0, 2024-04-23T10:32:04.751 INFO:teuthology.orchestra.run.smithi114.stdout: "waiter": 0, 2024-04-23T10:32:04.751 INFO:teuthology.orchestra.run.smithi114.stdout: "authpin": 1 2024-04-23T10:32:04.751 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-23T10:32:04.751 INFO:teuthology.orchestra.run.smithi114.stdout: "nref": 5 2024-04-23T10:32:04.751 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-23T10:32:04.751 INFO:teuthology.orchestra.run.smithi114.stdout: "object_string": "[inode 0x10000000002 [...2,head] /volumes/qa/sv_0/ auth{1=1} v34 ap=2 snaprealm=0x5571cf174480 f(v0 m2024-04-23T10:24:49.040186+0000 2=1+1) n(v0 rc2024-04-23T10:24:49.040186+0000 b117 3=1+2) (inest mix) (iquiesce lock x=1 by request(mds.0:154 nref=3)) | request=1 lock=1 dirfrag=1 caps=0 dirtyparent=0 replicated=1 dirty=0 waiter=0 authpin=1 0x5571cf18fb80]", 2024-04-23T10:32:04.751 INFO:teuthology.orchestra.run.smithi114.stdout: "lock": { 2024-04-23T10:32:04.751 INFO:teuthology.orchestra.run.smithi114.stdout: "gather_set": [], 2024-04-23T10:32:04.751 INFO:teuthology.orchestra.run.smithi114.stdout: "state": "lock", 2024-04-23T10:32:04.751 INFO:teuthology.orchestra.run.smithi114.stdout: "type": "iquiesce", 2024-04-23T10:32:04.752 INFO:teuthology.orchestra.run.smithi114.stdout: "is_leased": false, 2024-04-23T10:32:04.752 INFO:teuthology.orchestra.run.smithi114.stdout: "num_rdlocks": 0, 2024-04-23T10:32:04.752 INFO:teuthology.orchestra.run.smithi114.stdout: "num_wrlocks": 0, 2024-04-23T10:32:04.752 INFO:teuthology.orchestra.run.smithi114.stdout: "num_xlocks": 1, 2024-04-23T10:32:04.752 INFO:teuthology.orchestra.run.smithi114.stdout: "xlock_by": { 2024-04-23T10:32:04.752 INFO:teuthology.orchestra.run.smithi114.stdout: "reqid": { 2024-04-23T10:32:04.752 INFO:teuthology.orchestra.run.smithi114.stdout: "entity": { 2024-04-23T10:32:04.752 INFO:teuthology.orchestra.run.smithi114.stdout: "type": "mds", 2024-04-23T10:32:04.752 INFO:teuthology.orchestra.run.smithi114.stdout: "num": 0 2024-04-23T10:32:04.752 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-23T10:32:04.752 INFO:teuthology.orchestra.run.smithi114.stdout: "tid": 154 2024-04-23T10:32:04.752 INFO:teuthology.orchestra.run.smithi114.stdout: } 2024-04-23T10:32:04.752 INFO:teuthology.orchestra.run.smithi114.stdout: } 2024-04-23T10:32:04.752 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-23T10:32:04.753 INFO:teuthology.orchestra.run.smithi114.stdout: "flags": 4, 2024-04-23T10:32:04.753 INFO:teuthology.orchestra.run.smithi114.stdout: "wrlock_target": -1 2024-04-23T10:32:04.753 INFO:teuthology.orchestra.run.smithi114.stdout: } 2024-04-23T10:32:04.753 INFO:teuthology.orchestra.run.smithi114.stdout: ] 2024-04-23T10:32:04.753 INFO:teuthology.orchestra.run.smithi114.stdout: } 2024-04-23T10:32:04.753 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-23T10:32:04.753 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-04-23T10:32:04.753 INFO:teuthology.orchestra.run.smithi114.stdout: "description": "internal op quiesce_inode:mds.0:173 fp=#0x1000000020f", 2024-04-23T10:32:04.753 INFO:teuthology.orchestra.run.smithi114.stdout: "initiated_at": "2024-04-23T10:31:21.459950+0000", 2024-04-23T10:32:04.753 INFO:teuthology.orchestra.run.smithi114.stdout: "age": 42.424439393, 2024-04-23T10:32:04.753 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 42.427099659, 2024-04-23T10:32:04.753 INFO:teuthology.orchestra.run.smithi114.stdout: "continuous": true, 2024-04-23T10:32:04.753 INFO:teuthology.orchestra.run.smithi114.stdout: "type_data": { 2024-04-23T10:32:04.753 INFO:teuthology.orchestra.run.smithi114.stdout: "result": -2147483648, 2024-04-23T10:32:04.753 INFO:teuthology.orchestra.run.smithi114.stdout: "flag_point": "quiesce complete", 2024-04-23T10:32:04.754 INFO:teuthology.orchestra.run.smithi114.stdout: "reqid": { 2024-04-23T10:32:04.754 INFO:teuthology.orchestra.run.smithi114.stdout: "entity": { 2024-04-23T10:32:04.754 INFO:teuthology.orchestra.run.smithi114.stdout: "type": "mds", 2024-04-23T10:32:04.754 INFO:teuthology.orchestra.run.smithi114.stdout: "num": 0 2024-04-23T10:32:04.754 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-23T10:32:04.754 INFO:teuthology.orchestra.run.smithi114.stdout: "tid": 173 2024-04-23T10:32:04.754 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-23T10:32:04.754 INFO:teuthology.orchestra.run.smithi114.stdout: "op_type": "internal_op", 2024-04-23T10:32:04.754 INFO:teuthology.orchestra.run.smithi114.stdout: "internal_op": 5385, 2024-04-23T10:32:04.754 INFO:teuthology.orchestra.run.smithi114.stdout: "op_name": "quiesce_inode", 2024-04-23T10:32:04.754 INFO:teuthology.orchestra.run.smithi114.stdout: "events": [ 2024-04-23T10:32:04.754 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-04-23T10:32:04.754 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-04-23T10:31:21.459950+0000", 2024-04-23T10:32:04.754 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "initiated" 2024-04-23T10:32:04.754 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-23T10:32:04.755 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-04-23T10:32:04.755 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-04-23T10:31:21.459950+0000", 2024-04-23T10:32:04.755 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "header_read" 2024-04-23T10:32:04.755 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-23T10:32:04.755 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-04-23T10:32:04.755 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-04-23T10:31:21.459950+0000", 2024-04-23T10:32:04.755 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "throttled" 2024-04-23T10:32:04.755 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-23T10:32:04.755 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-04-23T10:32:04.755 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-04-23T10:31:21.459950+0000", 2024-04-23T10:32:04.755 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "all_read" 2024-04-23T10:32:04.755 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-23T10:32:04.755 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-04-23T10:32:04.755 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-04-23T10:31:21.459950+0000", 2024-04-23T10:32:04.756 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "dispatched" 2024-04-23T10:32:04.756 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-23T10:32:04.756 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-04-23T10:32:04.756 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-04-23T10:31:21.462404+0000", 2024-04-23T10:32:04.756 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "failed to xlock, waiting" 2024-04-23T10:32:04.756 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-23T10:32:04.756 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-04-23T10:32:04.756 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-04-23T10:31:21.489527+0000", 2024-04-23T10:32:04.756 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "acquired locks" 2024-04-23T10:32:04.756 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-23T10:32:04.756 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-04-23T10:32:04.756 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-04-23T10:31:21.489939+0000", 2024-04-23T10:32:04.756 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "quiesce complete" 2024-04-23T10:32:04.756 INFO:teuthology.orchestra.run.smithi114.stdout: } 2024-04-23T10:32:04.756 INFO:teuthology.orchestra.run.smithi114.stdout: ], 2024-04-23T10:32:04.757 INFO:teuthology.orchestra.run.smithi114.stdout: "locks": [ 2024-04-23T10:32:04.757 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-04-23T10:32:04.757 INFO:teuthology.orchestra.run.smithi114.stdout: "object": { 2024-04-23T10:32:04.757 INFO:teuthology.orchestra.run.smithi114.stdout: "is_auth": true, 2024-04-23T10:32:04.757 INFO:teuthology.orchestra.run.smithi114.stdout: "auth_state": { 2024-04-23T10:32:04.757 INFO:teuthology.orchestra.run.smithi114.stdout: "replicas": { 2024-04-23T10:32:04.757 INFO:teuthology.orchestra.run.smithi114.stdout: "1": 1, 2024-04-23T10:32:04.757 INFO:teuthology.orchestra.run.smithi114.stdout: "2": 1 2024-04-23T10:32:04.757 INFO:teuthology.orchestra.run.smithi114.stdout: } 2024-04-23T10:32:04.757 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-23T10:32:04.757 INFO:teuthology.orchestra.run.smithi114.stdout: "replica_state": { 2024-04-23T10:32:04.757 INFO:teuthology.orchestra.run.smithi114.stdout: "authority": [ 2024-04-23T10:32:04.757 INFO:teuthology.orchestra.run.smithi114.stdout: 0, 2024-04-23T10:32:04.757 INFO:teuthology.orchestra.run.smithi114.stdout: -2 2024-04-23T10:32:04.757 INFO:teuthology.orchestra.run.smithi114.stdout: ], 2024-04-23T10:32:04.758 INFO:teuthology.orchestra.run.smithi114.stdout: "replica_nonce": 0 2024-04-23T10:32:04.758 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-23T10:32:04.758 INFO:teuthology.orchestra.run.smithi114.stdout: "auth_pins": 3, 2024-04-23T10:32:04.758 INFO:teuthology.orchestra.run.smithi114.stdout: "is_frozen": false, 2024-04-23T10:32:04.758 INFO:teuthology.orchestra.run.smithi114.stdout: "is_freezing": false, 2024-04-23T10:32:04.758 INFO:teuthology.orchestra.run.smithi114.stdout: "pins": { 2024-04-23T10:32:04.758 INFO:teuthology.orchestra.run.smithi114.stdout: "dirtyscattered": 0, 2024-04-23T10:32:04.758 INFO:teuthology.orchestra.run.smithi114.stdout: "request": 1, 2024-04-23T10:32:04.758 INFO:teuthology.orchestra.run.smithi114.stdout: "lock": 1, 2024-04-23T10:32:04.758 INFO:teuthology.orchestra.run.smithi114.stdout: "dirfrag": 1, 2024-04-23T10:32:04.758 INFO:teuthology.orchestra.run.smithi114.stdout: "caps": 1, 2024-04-23T10:32:04.758 INFO:teuthology.orchestra.run.smithi114.stdout: "dirtyparent": 0, 2024-04-23T10:32:04.758 INFO:teuthology.orchestra.run.smithi114.stdout: "replicated": 1, 2024-04-23T10:32:04.758 INFO:teuthology.orchestra.run.smithi114.stdout: "dirty": 0, 2024-04-23T10:32:04.759 INFO:teuthology.orchestra.run.smithi114.stdout: "waiter": 0, 2024-04-23T10:32:04.759 INFO:teuthology.orchestra.run.smithi114.stdout: "authpin": 1 2024-04-23T10:32:04.759 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-23T10:32:04.759 INFO:teuthology.orchestra.run.smithi114.stdout: "nref": 6 2024-04-23T10:32:04.759 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-23T10:32:04.759 INFO:teuthology.orchestra.run.smithi114.stdout: "object_string": "[inode 0x1000000020f [...7,head] /volumes/qa/sv_1/b206794e-7622-4ab8-8943-f3ae4e65b7a3/client.0/tmp/clients/client0/~dmtmp/PWRPNT/ auth{1=1,2=1} v45572 ap=3 f(v1293 m2024-04-23T10:31:17.718717+0000 5=5+0) n(v18 rc2024-04-23T10:31:17.718717+0000 b1300480 6=5+1) old_inodes=4 (inest mix) (iquiesce lock x=1 by request(mds.0:173 nref=3)) caps={15312=pAsLsXsFs/pAsLsXsFsxcral@3438} mcw={2=AsLsXsFs} | dirtyscattered=0 request=1 lock=1 dirfrag=1 caps=1 dirtyparent=0 replicated=1 dirty=0 waiter=0 authpin=1 0x5571cf53f600]", 2024-04-23T10:32:04.759 INFO:teuthology.orchestra.run.smithi114.stdout: "lock": { 2024-04-23T10:32:04.759 INFO:teuthology.orchestra.run.smithi114.stdout: "gather_set": [], 2024-04-23T10:32:04.759 INFO:teuthology.orchestra.run.smithi114.stdout: "state": "lock", 2024-04-23T10:32:04.759 INFO:teuthology.orchestra.run.smithi114.stdout: "type": "iquiesce", 2024-04-23T10:32:04.759 INFO:teuthology.orchestra.run.smithi114.stdout: "is_leased": false, 2024-04-23T10:32:04.759 INFO:teuthology.orchestra.run.smithi114.stdout: "num_rdlocks": 0, 2024-04-23T10:32:04.759 INFO:teuthology.orchestra.run.smithi114.stdout: "num_wrlocks": 0, 2024-04-23T10:32:04.759 INFO:teuthology.orchestra.run.smithi114.stdout: "num_xlocks": 1, 2024-04-23T10:32:04.759 INFO:teuthology.orchestra.run.smithi114.stdout: "xlock_by": { 2024-04-23T10:32:04.760 INFO:teuthology.orchestra.run.smithi114.stdout: "reqid": { 2024-04-23T10:32:04.760 INFO:teuthology.orchestra.run.smithi114.stdout: "entity": { 2024-04-23T10:32:04.760 INFO:teuthology.orchestra.run.smithi114.stdout: "type": "mds", 2024-04-23T10:32:04.760 INFO:teuthology.orchestra.run.smithi114.stdout: "num": 0 2024-04-23T10:32:04.760 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-23T10:32:04.760 INFO:teuthology.orchestra.run.smithi114.stdout: "tid": 173 2024-04-23T10:32:04.760 INFO:teuthology.orchestra.run.smithi114.stdout: } 2024-04-23T10:32:04.760 INFO:teuthology.orchestra.run.smithi114.stdout: } 2024-04-23T10:32:04.760 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-23T10:32:04.760 INFO:teuthology.orchestra.run.smithi114.stdout: "flags": 4, 2024-04-23T10:32:04.760 INFO:teuthology.orchestra.run.smithi114.stdout: "wrlock_target": -1 2024-04-23T10:32:04.760 INFO:teuthology.orchestra.run.smithi114.stdout: } 2024-04-23T10:32:04.760 INFO:teuthology.orchestra.run.smithi114.stdout: ] 2024-04-23T10:32:04.760 INFO:teuthology.orchestra.run.smithi114.stdout: } 2024-04-23T10:32:04.760 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-23T10:32:04.761 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-04-23T10:32:04.761 INFO:teuthology.orchestra.run.smithi114.stdout: "description": "internal op quiesce_inode:mds.0:198 fp=#0x1000000021d", 2024-04-23T10:32:04.761 INFO:teuthology.orchestra.run.smithi114.stdout: "initiated_at": "2024-04-23T10:31:21.483491+0000", 2024-04-23T10:32:04.761 INFO:teuthology.orchestra.run.smithi114.stdout: "age": 42.400898220999999, 2024-04-23T10:32:04.761 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 42.403625163999997, 2024-04-23T10:32:04.761 INFO:teuthology.orchestra.run.smithi114.stdout: "continuous": true, 2024-04-23T10:32:04.761 INFO:teuthology.orchestra.run.smithi114.stdout: "type_data": { 2024-04-23T10:32:04.761 INFO:teuthology.orchestra.run.smithi114.stdout: "result": -2147483648, 2024-04-23T10:32:04.761 INFO:teuthology.orchestra.run.smithi114.stdout: "flag_point": "quiesce complete for non-auth inode", 2024-04-23T10:32:04.761 INFO:teuthology.orchestra.run.smithi114.stdout: "reqid": { 2024-04-23T10:32:04.761 INFO:teuthology.orchestra.run.smithi114.stdout: "entity": { 2024-04-23T10:32:04.761 INFO:teuthology.orchestra.run.smithi114.stdout: "type": "mds", 2024-04-23T10:32:04.761 INFO:teuthology.orchestra.run.smithi114.stdout: "num": 0 2024-04-23T10:32:04.761 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-23T10:32:04.762 INFO:teuthology.orchestra.run.smithi114.stdout: "tid": 198 2024-04-23T10:32:04.762 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-23T10:32:04.762 INFO:teuthology.orchestra.run.smithi114.stdout: "op_type": "internal_op", 2024-04-23T10:32:04.762 INFO:teuthology.orchestra.run.smithi114.stdout: "internal_op": 5385, 2024-04-23T10:32:04.762 INFO:teuthology.orchestra.run.smithi114.stdout: "op_name": "quiesce_inode", 2024-04-23T10:32:04.762 INFO:teuthology.orchestra.run.smithi114.stdout: "events": [ 2024-04-23T10:32:04.762 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-04-23T10:32:04.762 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-04-23T10:31:21.483491+0000", 2024-04-23T10:32:04.762 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "initiated" 2024-04-23T10:32:04.762 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-23T10:32:04.762 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-04-23T10:32:04.762 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-04-23T10:31:21.483491+0000", 2024-04-23T10:32:04.762 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "header_read" 2024-04-23T10:32:04.762 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-23T10:32:04.762 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-04-23T10:32:04.763 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-04-23T10:31:21.483491+0000", 2024-04-23T10:32:04.763 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "throttled" 2024-04-23T10:32:04.763 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-23T10:32:04.763 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-04-23T10:32:04.763 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-04-23T10:31:21.483491+0000", 2024-04-23T10:32:04.763 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "all_read" 2024-04-23T10:32:04.763 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-23T10:32:04.763 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-04-23T10:32:04.763 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-04-23T10:31:21.483491+0000", 2024-04-23T10:32:04.763 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "dispatched" 2024-04-23T10:32:04.763 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-23T10:32:04.763 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-04-23T10:32:04.763 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-04-23T10:31:21.483994+0000", 2024-04-23T10:32:04.763 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "requesting remote authpins" 2024-04-23T10:32:04.764 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-23T10:32:04.764 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-04-23T10:32:04.764 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-04-23T10:31:21.507982+0000", 2024-04-23T10:32:04.764 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "acquired locks" 2024-04-23T10:32:04.764 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-23T10:32:04.764 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-04-23T10:32:04.764 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-04-23T10:31:21.507996+0000", 2024-04-23T10:32:04.764 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "quiesce complete for non-auth inode" 2024-04-23T10:32:04.764 INFO:teuthology.orchestra.run.smithi114.stdout: } 2024-04-23T10:32:04.764 INFO:teuthology.orchestra.run.smithi114.stdout: ], 2024-04-23T10:32:04.764 INFO:teuthology.orchestra.run.smithi114.stdout: "locks": [ 2024-04-23T10:32:04.764 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-04-23T10:32:04.764 INFO:teuthology.orchestra.run.smithi114.stdout: "object": { 2024-04-23T10:32:04.764 INFO:teuthology.orchestra.run.smithi114.stdout: "is_auth": false, 2024-04-23T10:32:04.764 INFO:teuthology.orchestra.run.smithi114.stdout: "auth_state": { 2024-04-23T10:32:04.765 INFO:teuthology.orchestra.run.smithi114.stdout: "replicas": {} 2024-04-23T10:32:04.765 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-23T10:32:04.765 INFO:teuthology.orchestra.run.smithi114.stdout: "replica_state": { 2024-04-23T10:32:04.765 INFO:teuthology.orchestra.run.smithi114.stdout: "authority": [ 2024-04-23T10:32:04.765 INFO:teuthology.orchestra.run.smithi114.stdout: 2, 2024-04-23T10:32:04.765 INFO:teuthology.orchestra.run.smithi114.stdout: -2 2024-04-23T10:32:04.765 INFO:teuthology.orchestra.run.smithi114.stdout: ], 2024-04-23T10:32:04.765 INFO:teuthology.orchestra.run.smithi114.stdout: "replica_nonce": 1 2024-04-23T10:32:04.765 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-23T10:32:04.765 INFO:teuthology.orchestra.run.smithi114.stdout: "auth_pins": 0, 2024-04-23T10:32:04.765 INFO:teuthology.orchestra.run.smithi114.stdout: "is_frozen": false, 2024-04-23T10:32:04.765 INFO:teuthology.orchestra.run.smithi114.stdout: "is_freezing": false, 2024-04-23T10:32:04.765 INFO:teuthology.orchestra.run.smithi114.stdout: "pins": { 2024-04-23T10:32:04.765 INFO:teuthology.orchestra.run.smithi114.stdout: "request": 1, 2024-04-23T10:32:04.766 INFO:teuthology.orchestra.run.smithi114.stdout: "lock": 1, 2024-04-23T10:32:04.766 INFO:teuthology.orchestra.run.smithi114.stdout: "caps": 0, 2024-04-23T10:32:04.766 INFO:teuthology.orchestra.run.smithi114.stdout: "exportingcaps": 0, 2024-04-23T10:32:04.766 INFO:teuthology.orchestra.run.smithi114.stdout: "dirtyparent": 0, 2024-04-23T10:32:04.766 INFO:teuthology.orchestra.run.smithi114.stdout: "dirty": 0, 2024-04-23T10:32:04.766 INFO:teuthology.orchestra.run.smithi114.stdout: "authpin": 0, 2024-04-23T10:32:04.766 INFO:teuthology.orchestra.run.smithi114.stdout: "tempexporting": 0 2024-04-23T10:32:04.766 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-23T10:32:04.766 INFO:teuthology.orchestra.run.smithi114.stdout: "nref": 2 2024-04-23T10:32:04.766 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-23T10:32:04.766 INFO:teuthology.orchestra.run.smithi114.stdout: "object_string": "[inode 0x1000000021d [7,head] /volumes/qa/sv_1/b206794e-7622-4ab8-8943-f3ae4e65b7a3/client.0/tmp/clients/client0/~dmtmp/COREL/TRAIN.CDR rep@2.1 v7252 s=260096 n(v0 rc2024-04-23T10:27:52.038396+0000 b260096 1=1+0)/n(v0 1=1+0) (iquiesce lock x=1 by request(mds.0:198 nref=3)) | request=1 lock=1 caps=0 exportingcaps=0 dirtyparent=0 dirty=0 authpin=0 tempexporting=0 0x5571cf562680]", 2024-04-23T10:32:04.766 INFO:teuthology.orchestra.run.smithi114.stdout: "lock": { 2024-04-23T10:32:04.766 INFO:teuthology.orchestra.run.smithi114.stdout: "gather_set": [], 2024-04-23T10:32:04.766 INFO:teuthology.orchestra.run.smithi114.stdout: "state": "lock", 2024-04-23T10:32:04.767 INFO:teuthology.orchestra.run.smithi114.stdout: "type": "iquiesce", 2024-04-23T10:32:04.767 INFO:teuthology.orchestra.run.smithi114.stdout: "is_leased": false, 2024-04-23T10:32:04.767 INFO:teuthology.orchestra.run.smithi114.stdout: "num_rdlocks": 0, 2024-04-23T10:32:04.767 INFO:teuthology.orchestra.run.smithi114.stdout: "num_wrlocks": 0, 2024-04-23T10:32:04.767 INFO:teuthology.orchestra.run.smithi114.stdout: "num_xlocks": 1, 2024-04-23T10:32:04.767 INFO:teuthology.orchestra.run.smithi114.stdout: "xlock_by": { 2024-04-23T10:32:04.767 INFO:teuthology.orchestra.run.smithi114.stdout: "reqid": { 2024-04-23T10:32:04.767 INFO:teuthology.orchestra.run.smithi114.stdout: "entity": { 2024-04-23T10:32:04.767 INFO:teuthology.orchestra.run.smithi114.stdout: "type": "mds", 2024-04-23T10:32:04.767 INFO:teuthology.orchestra.run.smithi114.stdout: "num": 0 2024-04-23T10:32:04.767 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-23T10:32:04.767 INFO:teuthology.orchestra.run.smithi114.stdout: "tid": 198 2024-04-23T10:32:04.767 INFO:teuthology.orchestra.run.smithi114.stdout: } 2024-04-23T10:32:04.767 INFO:teuthology.orchestra.run.smithi114.stdout: } 2024-04-23T10:32:04.767 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-23T10:32:04.768 INFO:teuthology.orchestra.run.smithi114.stdout: "flags": 4, 2024-04-23T10:32:04.768 INFO:teuthology.orchestra.run.smithi114.stdout: "wrlock_target": -1 2024-04-23T10:32:04.768 INFO:teuthology.orchestra.run.smithi114.stdout: } 2024-04-23T10:32:04.768 INFO:teuthology.orchestra.run.smithi114.stdout: ] 2024-04-23T10:32:04.768 INFO:teuthology.orchestra.run.smithi114.stdout: } 2024-04-23T10:32:04.768 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-23T10:32:04.768 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-04-23T10:32:04.768 INFO:teuthology.orchestra.run.smithi114.stdout: "description": "peer_request:mds.1:105", 2024-04-23T10:32:04.768 INFO:teuthology.orchestra.run.smithi114.stdout: "initiated_at": "2024-04-23T10:31:21.448402+0000", 2024-04-23T10:32:04.768 INFO:teuthology.orchestra.run.smithi114.stdout: "age": 42.435987691000001, 2024-04-23T10:32:04.768 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 42.438773701000002, 2024-04-23T10:32:04.768 INFO:teuthology.orchestra.run.smithi114.stdout: "continuous": false, 2024-04-23T10:32:04.768 INFO:teuthology.orchestra.run.smithi114.stdout: "type_data": { 2024-04-23T10:32:04.768 INFO:teuthology.orchestra.run.smithi114.stdout: "result": -2147483648, 2024-04-23T10:32:04.768 INFO:teuthology.orchestra.run.smithi114.stdout: "flag_point": "dispatched", 2024-04-23T10:32:04.769 INFO:teuthology.orchestra.run.smithi114.stdout: "reqid": { 2024-04-23T10:32:04.769 INFO:teuthology.orchestra.run.smithi114.stdout: "entity": { 2024-04-23T10:32:04.769 INFO:teuthology.orchestra.run.smithi114.stdout: "type": "mds", 2024-04-23T10:32:04.769 INFO:teuthology.orchestra.run.smithi114.stdout: "num": 1 2024-04-23T10:32:04.769 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-23T10:32:04.769 INFO:teuthology.orchestra.run.smithi114.stdout: "tid": 105 2024-04-23T10:32:04.769 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-23T10:32:04.769 INFO:teuthology.orchestra.run.smithi114.stdout: "op_type": "peer_request", 2024-04-23T10:32:04.769 INFO:teuthology.orchestra.run.smithi114.stdout: "leader_info": { 2024-04-23T10:32:04.769 INFO:teuthology.orchestra.run.smithi114.stdout: "leader": "1" 2024-04-23T10:32:04.769 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-23T10:32:04.769 INFO:teuthology.orchestra.run.smithi114.stdout: "events": [ 2024-04-23T10:32:04.769 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-04-23T10:32:04.769 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-04-23T10:31:21.448402+0000", 2024-04-23T10:32:04.769 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "initiated" 2024-04-23T10:32:04.769 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-23T10:32:04.770 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-04-23T10:32:04.770 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-04-23T10:31:21.448402+0000", 2024-04-23T10:32:04.770 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "header_read" 2024-04-23T10:32:04.770 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-23T10:32:04.770 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-04-23T10:32:04.770 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-04-23T10:31:21.448402+0000", 2024-04-23T10:32:04.770 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "throttled" 2024-04-23T10:32:04.770 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-23T10:32:04.770 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-04-23T10:32:04.770 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-04-23T10:31:21.448411+0000", 2024-04-23T10:32:04.770 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "all_read" 2024-04-23T10:32:04.770 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-23T10:32:04.770 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-04-23T10:32:04.770 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-04-23T10:31:21.449159+0000", 2024-04-23T10:32:04.771 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "dispatched" 2024-04-23T10:32:04.771 INFO:teuthology.orchestra.run.smithi114.stdout: } 2024-04-23T10:32:04.771 INFO:teuthology.orchestra.run.smithi114.stdout: ], 2024-04-23T10:32:04.771 INFO:teuthology.orchestra.run.smithi114.stdout: "locks": [] 2024-04-23T10:32:04.771 INFO:teuthology.orchestra.run.smithi114.stdout: } 2024-04-23T10:32:04.771 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-23T10:32:04.771 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-04-23T10:32:04.771 INFO:teuthology.orchestra.run.smithi114.stdout: "description": "internal op quiesce_inode:mds.0:174 fp=#0x1000000023e", 2024-04-23T10:32:04.771 INFO:teuthology.orchestra.run.smithi114.stdout: "initiated_at": "2024-04-23T10:31:21.459967+0000", 2024-04-23T10:32:04.771 INFO:teuthology.orchestra.run.smithi114.stdout: "age": 42.424422245999999, 2024-04-23T10:32:04.771 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 42.427230668, 2024-04-23T10:32:04.771 INFO:teuthology.orchestra.run.smithi114.stdout: "continuous": true, 2024-04-23T10:32:04.771 INFO:teuthology.orchestra.run.smithi114.stdout: "type_data": { 2024-04-23T10:32:04.771 INFO:teuthology.orchestra.run.smithi114.stdout: "result": -2147483648, 2024-04-23T10:32:04.771 INFO:teuthology.orchestra.run.smithi114.stdout: "flag_point": "quiesce complete", 2024-04-23T10:32:04.772 INFO:teuthology.orchestra.run.smithi114.stdout: "reqid": { 2024-04-23T10:32:04.772 INFO:teuthology.orchestra.run.smithi114.stdout: "entity": { 2024-04-23T10:32:04.772 INFO:teuthology.orchestra.run.smithi114.stdout: "type": "mds", 2024-04-23T10:32:04.772 INFO:teuthology.orchestra.run.smithi114.stdout: "num": 0 2024-04-23T10:32:04.772 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-23T10:32:04.772 INFO:teuthology.orchestra.run.smithi114.stdout: "tid": 174 2024-04-23T10:32:04.772 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-23T10:32:04.772 INFO:teuthology.orchestra.run.smithi114.stdout: "op_type": "internal_op", 2024-04-23T10:32:04.772 INFO:teuthology.orchestra.run.smithi114.stdout: "internal_op": 5385, 2024-04-23T10:32:04.772 INFO:teuthology.orchestra.run.smithi114.stdout: "op_name": "quiesce_inode", 2024-04-23T10:32:04.772 INFO:teuthology.orchestra.run.smithi114.stdout: "events": [ 2024-04-23T10:32:04.772 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-04-23T10:32:04.772 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-04-23T10:31:21.459967+0000", 2024-04-23T10:32:04.772 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "initiated" 2024-04-23T10:32:04.772 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-23T10:32:04.773 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-04-23T10:32:04.773 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-04-23T10:31:21.459967+0000", 2024-04-23T10:32:04.773 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "header_read" 2024-04-23T10:32:04.773 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-23T10:32:04.773 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-04-23T10:32:04.773 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-04-23T10:31:21.459967+0000", 2024-04-23T10:32:04.773 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "throttled" 2024-04-23T10:32:04.773 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-23T10:32:04.773 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-04-23T10:32:04.773 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-04-23T10:31:21.459967+0000", 2024-04-23T10:32:04.773 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "all_read" 2024-04-23T10:32:04.773 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-23T10:32:04.773 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-04-23T10:32:04.773 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-04-23T10:31:21.459967+0000", 2024-04-23T10:32:04.773 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "dispatched" 2024-04-23T10:32:04.774 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-23T10:32:04.774 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-04-23T10:32:04.774 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-04-23T10:31:21.462675+0000", 2024-04-23T10:32:04.774 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "failed to xlock, waiting" 2024-04-23T10:32:04.774 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-23T10:32:04.774 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-04-23T10:32:04.774 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-04-23T10:31:21.471011+0000", 2024-04-23T10:32:04.774 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "acquired locks" 2024-04-23T10:32:04.774 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-23T10:32:04.774 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-04-23T10:32:04.774 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-04-23T10:31:21.507174+0000", 2024-04-23T10:32:04.774 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "quiesce complete" 2024-04-23T10:32:04.774 INFO:teuthology.orchestra.run.smithi114.stdout: } 2024-04-23T10:32:04.774 INFO:teuthology.orchestra.run.smithi114.stdout: ], 2024-04-23T10:32:04.774 INFO:teuthology.orchestra.run.smithi114.stdout: "locks": [ 2024-04-23T10:32:04.774 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-04-23T10:32:04.775 INFO:teuthology.orchestra.run.smithi114.stdout: "object": { 2024-04-23T10:32:04.775 INFO:teuthology.orchestra.run.smithi114.stdout: "is_auth": true, 2024-04-23T10:32:04.775 INFO:teuthology.orchestra.run.smithi114.stdout: "auth_state": { 2024-04-23T10:32:04.775 INFO:teuthology.orchestra.run.smithi114.stdout: "replicas": { 2024-04-23T10:32:04.775 INFO:teuthology.orchestra.run.smithi114.stdout: "1": 1, 2024-04-23T10:32:04.775 INFO:teuthology.orchestra.run.smithi114.stdout: "2": 1 2024-04-23T10:32:04.775 INFO:teuthology.orchestra.run.smithi114.stdout: } 2024-04-23T10:32:04.775 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-23T10:32:04.775 INFO:teuthology.orchestra.run.smithi114.stdout: "replica_state": { 2024-04-23T10:32:04.775 INFO:teuthology.orchestra.run.smithi114.stdout: "authority": [ 2024-04-23T10:32:04.775 INFO:teuthology.orchestra.run.smithi114.stdout: 0, 2024-04-23T10:32:04.775 INFO:teuthology.orchestra.run.smithi114.stdout: -2 2024-04-23T10:32:04.775 INFO:teuthology.orchestra.run.smithi114.stdout: ], 2024-04-23T10:32:04.775 INFO:teuthology.orchestra.run.smithi114.stdout: "replica_nonce": 0 2024-04-23T10:32:04.775 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-23T10:32:04.775 INFO:teuthology.orchestra.run.smithi114.stdout: "auth_pins": 3, 2024-04-23T10:32:04.776 INFO:teuthology.orchestra.run.smithi114.stdout: "is_frozen": false, 2024-04-23T10:32:04.776 INFO:teuthology.orchestra.run.smithi114.stdout: "is_freezing": false, 2024-04-23T10:32:04.776 INFO:teuthology.orchestra.run.smithi114.stdout: "pins": { 2024-04-23T10:32:04.776 INFO:teuthology.orchestra.run.smithi114.stdout: "request": 1, 2024-04-23T10:32:04.776 INFO:teuthology.orchestra.run.smithi114.stdout: "lock": 1, 2024-04-23T10:32:04.776 INFO:teuthology.orchestra.run.smithi114.stdout: "dirfrag": 1, 2024-04-23T10:32:04.776 INFO:teuthology.orchestra.run.smithi114.stdout: "caps": 1, 2024-04-23T10:32:04.776 INFO:teuthology.orchestra.run.smithi114.stdout: "dirtyparent": 0, 2024-04-23T10:32:04.776 INFO:teuthology.orchestra.run.smithi114.stdout: "replicated": 1, 2024-04-23T10:32:04.776 INFO:teuthology.orchestra.run.smithi114.stdout: "dirty": 0, 2024-04-23T10:32:04.776 INFO:teuthology.orchestra.run.smithi114.stdout: "waiter": 0, 2024-04-23T10:32:04.776 INFO:teuthology.orchestra.run.smithi114.stdout: "authpin": 1 2024-04-23T10:32:04.776 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-23T10:32:04.776 INFO:teuthology.orchestra.run.smithi114.stdout: "nref": 6 2024-04-23T10:32:04.776 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-23T10:32:04.777 INFO:teuthology.orchestra.run.smithi114.stdout: "object_string": "[inode 0x1000000023e [...7,head] /volumes/qa/sv_1/b206794e-7622-4ab8-8943-f3ae4e65b7a3/client.0/tmp/clients/client0/~dmtmp/ACCESS/ auth{1=1,2=1} v45582 ap=3 f(v0 m2024-04-23T10:31:17.039732+0000 4=4+0) n(v0 rc2024-04-23T10:31:17.039732+0000 b363650 5=4+1) old_inodes=4 (inest mix) (iquiesce lock x=1 by request(mds.0:174 nref=3)) caps={15312=pAsLsXsFs/pAsLsXsFsxcral@2942},l=15312 | request=1 lock=1 dirfrag=1 caps=1 dirtyparent=0 replicated=1 dirty=0 waiter=0 authpin=1 0x5571cf56a680]", 2024-04-23T10:32:04.777 INFO:teuthology.orchestra.run.smithi114.stdout: "lock": { 2024-04-23T10:32:04.777 INFO:teuthology.orchestra.run.smithi114.stdout: "gather_set": [], 2024-04-23T10:32:04.777 INFO:teuthology.orchestra.run.smithi114.stdout: "state": "lock", 2024-04-23T10:32:04.777 INFO:teuthology.orchestra.run.smithi114.stdout: "type": "iquiesce", 2024-04-23T10:32:04.777 INFO:teuthology.orchestra.run.smithi114.stdout: "is_leased": false, 2024-04-23T10:32:04.777 INFO:teuthology.orchestra.run.smithi114.stdout: "num_rdlocks": 0, 2024-04-23T10:32:04.777 INFO:teuthology.orchestra.run.smithi114.stdout: "num_wrlocks": 0, 2024-04-23T10:32:04.777 INFO:teuthology.orchestra.run.smithi114.stdout: "num_xlocks": 1, 2024-04-23T10:32:04.777 INFO:teuthology.orchestra.run.smithi114.stdout: "xlock_by": { 2024-04-23T10:32:04.777 INFO:teuthology.orchestra.run.smithi114.stdout: "reqid": { 2024-04-23T10:32:04.777 INFO:teuthology.orchestra.run.smithi114.stdout: "entity": { 2024-04-23T10:32:04.777 INFO:teuthology.orchestra.run.smithi114.stdout: "type": "mds", 2024-04-23T10:32:04.777 INFO:teuthology.orchestra.run.smithi114.stdout: "num": 0 2024-04-23T10:32:04.777 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-23T10:32:04.777 INFO:teuthology.orchestra.run.smithi114.stdout: "tid": 174 2024-04-23T10:32:04.778 INFO:teuthology.orchestra.run.smithi114.stdout: } 2024-04-23T10:32:04.778 INFO:teuthology.orchestra.run.smithi114.stdout: } 2024-04-23T10:32:04.778 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-23T10:32:04.778 INFO:teuthology.orchestra.run.smithi114.stdout: "flags": 4, 2024-04-23T10:32:04.778 INFO:teuthology.orchestra.run.smithi114.stdout: "wrlock_target": -1 2024-04-23T10:32:04.778 INFO:teuthology.orchestra.run.smithi114.stdout: } 2024-04-23T10:32:04.778 INFO:teuthology.orchestra.run.smithi114.stdout: ] 2024-04-23T10:32:04.778 INFO:teuthology.orchestra.run.smithi114.stdout: } 2024-04-23T10:32:04.778 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-23T10:32:04.778 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-04-23T10:32:04.778 INFO:teuthology.orchestra.run.smithi114.stdout: "description": "internal op quiesce_inode:mds.0:199 fp=#0x10000000219", 2024-04-23T10:32:04.778 INFO:teuthology.orchestra.run.smithi114.stdout: "initiated_at": "2024-04-23T10:31:21.483505+0000", 2024-04-23T10:32:04.778 INFO:teuthology.orchestra.run.smithi114.stdout: "age": 42.400884478000002, 2024-04-23T10:32:04.778 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 42.403757959000004, 2024-04-23T10:32:04.778 INFO:teuthology.orchestra.run.smithi114.stdout: "continuous": true, 2024-04-23T10:32:04.779 INFO:teuthology.orchestra.run.smithi114.stdout: "type_data": { 2024-04-23T10:32:04.779 INFO:teuthology.orchestra.run.smithi114.stdout: "result": -2147483648, 2024-04-23T10:32:04.779 INFO:teuthology.orchestra.run.smithi114.stdout: "flag_point": "quiesce complete for non-auth inode", 2024-04-23T10:32:04.779 INFO:teuthology.orchestra.run.smithi114.stdout: "reqid": { 2024-04-23T10:32:04.779 INFO:teuthology.orchestra.run.smithi114.stdout: "entity": { 2024-04-23T10:32:04.779 INFO:teuthology.orchestra.run.smithi114.stdout: "type": "mds", 2024-04-23T10:32:04.779 INFO:teuthology.orchestra.run.smithi114.stdout: "num": 0 2024-04-23T10:32:04.779 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-23T10:32:04.779 INFO:teuthology.orchestra.run.smithi114.stdout: "tid": 199 2024-04-23T10:32:04.779 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-23T10:32:04.779 INFO:teuthology.orchestra.run.smithi114.stdout: "op_type": "internal_op", 2024-04-23T10:32:04.779 INFO:teuthology.orchestra.run.smithi114.stdout: "internal_op": 5385, 2024-04-23T10:32:04.779 INFO:teuthology.orchestra.run.smithi114.stdout: "op_name": "quiesce_inode", 2024-04-23T10:32:04.779 INFO:teuthology.orchestra.run.smithi114.stdout: "events": [ 2024-04-23T10:32:04.779 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-04-23T10:32:04.779 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-04-23T10:31:21.483505+0000", 2024-04-23T10:32:04.780 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "initiated" 2024-04-23T10:32:04.780 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-23T10:32:04.780 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-04-23T10:32:04.780 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-04-23T10:31:21.483505+0000", 2024-04-23T10:32:04.780 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "header_read" 2024-04-23T10:32:04.780 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-23T10:32:04.780 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-04-23T10:32:04.780 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-04-23T10:31:21.483505+0000", 2024-04-23T10:32:04.780 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "throttled" 2024-04-23T10:32:04.780 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-23T10:32:04.780 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-04-23T10:32:04.780 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-04-23T10:31:21.483505+0000", 2024-04-23T10:32:04.780 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "all_read" 2024-04-23T10:32:04.780 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-23T10:32:04.780 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-04-23T10:32:04.781 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-04-23T10:31:21.483505+0000", 2024-04-23T10:32:04.781 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "dispatched" 2024-04-23T10:32:04.781 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-23T10:32:04.781 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-04-23T10:32:04.781 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-04-23T10:31:21.484047+0000", 2024-04-23T10:32:04.781 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "requesting remote authpins" 2024-04-23T10:32:04.781 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-23T10:32:04.781 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-04-23T10:32:04.781 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-04-23T10:31:21.508085+0000", 2024-04-23T10:32:04.781 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "acquired locks" 2024-04-23T10:32:04.781 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-23T10:32:04.781 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-04-23T10:32:04.781 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-04-23T10:31:21.508095+0000", 2024-04-23T10:32:04.781 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "quiesce complete for non-auth inode" 2024-04-23T10:32:04.781 INFO:teuthology.orchestra.run.smithi114.stdout: } 2024-04-23T10:32:04.781 INFO:teuthology.orchestra.run.smithi114.stdout: ], 2024-04-23T10:32:04.782 INFO:teuthology.orchestra.run.smithi114.stdout: "locks": [ 2024-04-23T10:32:04.782 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-04-23T10:32:04.782 INFO:teuthology.orchestra.run.smithi114.stdout: "object": { 2024-04-23T10:32:04.782 INFO:teuthology.orchestra.run.smithi114.stdout: "is_auth": false, 2024-04-23T10:32:04.782 INFO:teuthology.orchestra.run.smithi114.stdout: "auth_state": { 2024-04-23T10:32:04.782 INFO:teuthology.orchestra.run.smithi114.stdout: "replicas": {} 2024-04-23T10:32:04.782 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-23T10:32:04.782 INFO:teuthology.orchestra.run.smithi114.stdout: "replica_state": { 2024-04-23T10:32:04.782 INFO:teuthology.orchestra.run.smithi114.stdout: "authority": [ 2024-04-23T10:32:04.782 INFO:teuthology.orchestra.run.smithi114.stdout: 2, 2024-04-23T10:32:04.782 INFO:teuthology.orchestra.run.smithi114.stdout: -2 2024-04-23T10:32:04.782 INFO:teuthology.orchestra.run.smithi114.stdout: ], 2024-04-23T10:32:04.782 INFO:teuthology.orchestra.run.smithi114.stdout: "replica_nonce": 1 2024-04-23T10:32:04.782 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-23T10:32:04.783 INFO:teuthology.orchestra.run.smithi114.stdout: "auth_pins": 0, 2024-04-23T10:32:04.783 INFO:teuthology.orchestra.run.smithi114.stdout: "is_frozen": false, 2024-04-23T10:32:04.783 INFO:teuthology.orchestra.run.smithi114.stdout: "is_freezing": false, 2024-04-23T10:32:04.783 INFO:teuthology.orchestra.run.smithi114.stdout: "pins": { 2024-04-23T10:32:04.783 INFO:teuthology.orchestra.run.smithi114.stdout: "request": 1, 2024-04-23T10:32:04.783 INFO:teuthology.orchestra.run.smithi114.stdout: "lock": 1, 2024-04-23T10:32:04.783 INFO:teuthology.orchestra.run.smithi114.stdout: "caps": 0, 2024-04-23T10:32:04.783 INFO:teuthology.orchestra.run.smithi114.stdout: "exportingcaps": 0, 2024-04-23T10:32:04.783 INFO:teuthology.orchestra.run.smithi114.stdout: "dirtyparent": 0, 2024-04-23T10:32:04.783 INFO:teuthology.orchestra.run.smithi114.stdout: "dirty": 0, 2024-04-23T10:32:04.783 INFO:teuthology.orchestra.run.smithi114.stdout: "authpin": 0, 2024-04-23T10:32:04.783 INFO:teuthology.orchestra.run.smithi114.stdout: "tempexporting": 0 2024-04-23T10:32:04.783 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-23T10:32:04.783 INFO:teuthology.orchestra.run.smithi114.stdout: "nref": 2 2024-04-23T10:32:04.783 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-23T10:32:04.783 INFO:teuthology.orchestra.run.smithi114.stdout: "object_string": "[inode 0x10000000219 [7,head] /volumes/qa/sv_1/b206794e-7622-4ab8-8943-f3ae4e65b7a3/client.0/tmp/clients/client0/~dmtmp/COREL/GRAPHIC1.CDR rep@2.1 v7244 s=260096 n(v0 rc2024-04-23T10:27:52.019396+0000 b260096 1=1+0)/n(v0 1=1+0) (iquiesce lock x=1 by request(mds.0:199 nref=3)) | request=1 lock=1 caps=0 exportingcaps=0 dirtyparent=0 dirty=0 authpin=0 tempexporting=0 0x5571cf561080]", 2024-04-23T10:32:04.784 INFO:teuthology.orchestra.run.smithi114.stdout: "lock": { 2024-04-23T10:32:04.784 INFO:teuthology.orchestra.run.smithi114.stdout: "gather_set": [], 2024-04-23T10:32:04.784 INFO:teuthology.orchestra.run.smithi114.stdout: "state": "lock", 2024-04-23T10:32:04.784 INFO:teuthology.orchestra.run.smithi114.stdout: "type": "iquiesce", 2024-04-23T10:32:04.784 INFO:teuthology.orchestra.run.smithi114.stdout: "is_leased": false, 2024-04-23T10:32:04.784 INFO:teuthology.orchestra.run.smithi114.stdout: "num_rdlocks": 0, 2024-04-23T10:32:04.784 INFO:teuthology.orchestra.run.smithi114.stdout: "num_wrlocks": 0, 2024-04-23T10:32:04.784 INFO:teuthology.orchestra.run.smithi114.stdout: "num_xlocks": 1, 2024-04-23T10:32:04.784 INFO:teuthology.orchestra.run.smithi114.stdout: "xlock_by": { 2024-04-23T10:32:04.784 INFO:teuthology.orchestra.run.smithi114.stdout: "reqid": { 2024-04-23T10:32:04.784 INFO:teuthology.orchestra.run.smithi114.stdout: "entity": { 2024-04-23T10:32:04.784 INFO:teuthology.orchestra.run.smithi114.stdout: "type": "mds", 2024-04-23T10:32:04.784 INFO:teuthology.orchestra.run.smithi114.stdout: "num": 0 2024-04-23T10:32:04.784 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-23T10:32:04.784 INFO:teuthology.orchestra.run.smithi114.stdout: "tid": 199 2024-04-23T10:32:04.785 INFO:teuthology.orchestra.run.smithi114.stdout: } 2024-04-23T10:32:04.785 INFO:teuthology.orchestra.run.smithi114.stdout: } 2024-04-23T10:32:04.785 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-23T10:32:04.785 INFO:teuthology.orchestra.run.smithi114.stdout: "flags": 4, 2024-04-23T10:32:04.785 INFO:teuthology.orchestra.run.smithi114.stdout: "wrlock_target": -1 2024-04-23T10:32:04.785 INFO:teuthology.orchestra.run.smithi114.stdout: } 2024-04-23T10:32:04.785 INFO:teuthology.orchestra.run.smithi114.stdout: ] 2024-04-23T10:32:04.785 INFO:teuthology.orchestra.run.smithi114.stdout: } 2024-04-23T10:32:04.785 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-23T10:32:04.785 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-04-23T10:32:04.785 INFO:teuthology.orchestra.run.smithi114.stdout: "description": "peer_request:mds.2:110", 2024-04-23T10:32:04.785 INFO:teuthology.orchestra.run.smithi114.stdout: "initiated_at": "2024-04-23T10:31:21.448540+0000", 2024-04-23T10:32:04.785 INFO:teuthology.orchestra.run.smithi114.stdout: "age": 42.435849642000001, 2024-04-23T10:32:04.785 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 42.438782058999998, 2024-04-23T10:32:04.785 INFO:teuthology.orchestra.run.smithi114.stdout: "continuous": false, 2024-04-23T10:32:04.786 INFO:teuthology.orchestra.run.smithi114.stdout: "type_data": { 2024-04-23T10:32:04.786 INFO:teuthology.orchestra.run.smithi114.stdout: "result": -2147483648, 2024-04-23T10:32:04.786 INFO:teuthology.orchestra.run.smithi114.stdout: "flag_point": "dispatched", 2024-04-23T10:32:04.786 INFO:teuthology.orchestra.run.smithi114.stdout: "reqid": { 2024-04-23T10:32:04.786 INFO:teuthology.orchestra.run.smithi114.stdout: "entity": { 2024-04-23T10:32:04.786 INFO:teuthology.orchestra.run.smithi114.stdout: "type": "mds", 2024-04-23T10:32:04.786 INFO:teuthology.orchestra.run.smithi114.stdout: "num": 2 2024-04-23T10:32:04.786 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-23T10:32:04.786 INFO:teuthology.orchestra.run.smithi114.stdout: "tid": 110 2024-04-23T10:32:04.786 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-23T10:32:04.786 INFO:teuthology.orchestra.run.smithi114.stdout: "op_type": "peer_request", 2024-04-23T10:32:04.786 INFO:teuthology.orchestra.run.smithi114.stdout: "leader_info": { 2024-04-23T10:32:04.786 INFO:teuthology.orchestra.run.smithi114.stdout: "leader": "2" 2024-04-23T10:32:04.786 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-23T10:32:04.787 INFO:teuthology.orchestra.run.smithi114.stdout: "events": [ 2024-04-23T10:32:04.787 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-04-23T10:32:04.787 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-04-23T10:31:21.448540+0000", 2024-04-23T10:32:04.787 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "initiated" 2024-04-23T10:32:04.787 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-23T10:32:04.787 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-04-23T10:32:04.787 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-04-23T10:31:21.448540+0000", 2024-04-23T10:32:04.787 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "header_read" 2024-04-23T10:32:04.787 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-23T10:32:04.787 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-04-23T10:32:04.787 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-04-23T10:31:21.448541+0000", 2024-04-23T10:32:04.787 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "throttled" 2024-04-23T10:32:04.787 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-23T10:32:04.787 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-04-23T10:32:04.788 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-04-23T10:31:21.448549+0000", 2024-04-23T10:32:04.788 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "all_read" 2024-04-23T10:32:04.788 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-23T10:32:04.788 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-04-23T10:32:04.788 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-04-23T10:31:21.449238+0000", 2024-04-23T10:32:04.788 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "dispatched" 2024-04-23T10:32:04.788 INFO:teuthology.orchestra.run.smithi114.stdout: } 2024-04-23T10:32:04.788 INFO:teuthology.orchestra.run.smithi114.stdout: ], 2024-04-23T10:32:04.788 INFO:teuthology.orchestra.run.smithi114.stdout: "locks": [] 2024-04-23T10:32:04.788 INFO:teuthology.orchestra.run.smithi114.stdout: } 2024-04-23T10:32:04.788 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-23T10:32:04.788 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-04-23T10:32:04.788 INFO:teuthology.orchestra.run.smithi114.stdout: "description": "peer_request:mds.2:116", 2024-04-23T10:32:04.788 INFO:teuthology.orchestra.run.smithi114.stdout: "initiated_at": "2024-04-23T10:31:21.457073+0000", 2024-04-23T10:32:04.788 INFO:teuthology.orchestra.run.smithi114.stdout: "age": 42.427316136999998, 2024-04-23T10:32:04.789 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 42.430268882999997, 2024-04-23T10:32:04.789 INFO:teuthology.orchestra.run.smithi114.stdout: "continuous": false, 2024-04-23T10:32:04.789 INFO:teuthology.orchestra.run.smithi114.stdout: "type_data": { 2024-04-23T10:32:04.789 INFO:teuthology.orchestra.run.smithi114.stdout: "result": -2147483648, 2024-04-23T10:32:04.789 INFO:teuthology.orchestra.run.smithi114.stdout: "flag_point": "dispatched", 2024-04-23T10:32:04.789 INFO:teuthology.orchestra.run.smithi114.stdout: "reqid": { 2024-04-23T10:32:04.789 INFO:teuthology.orchestra.run.smithi114.stdout: "entity": { 2024-04-23T10:32:04.789 INFO:teuthology.orchestra.run.smithi114.stdout: "type": "mds", 2024-04-23T10:32:04.789 INFO:teuthology.orchestra.run.smithi114.stdout: "num": 2 2024-04-23T10:32:04.789 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-23T10:32:04.789 INFO:teuthology.orchestra.run.smithi114.stdout: "tid": 116 2024-04-23T10:32:04.789 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-23T10:32:04.789 INFO:teuthology.orchestra.run.smithi114.stdout: "op_type": "peer_request", 2024-04-23T10:32:04.789 INFO:teuthology.orchestra.run.smithi114.stdout: "leader_info": { 2024-04-23T10:32:04.789 INFO:teuthology.orchestra.run.smithi114.stdout: "leader": "2" 2024-04-23T10:32:04.790 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-23T10:32:04.790 INFO:teuthology.orchestra.run.smithi114.stdout: "events": [ 2024-04-23T10:32:04.790 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-04-23T10:32:04.790 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-04-23T10:31:21.457073+0000", 2024-04-23T10:32:04.790 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "initiated" 2024-04-23T10:32:04.790 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-23T10:32:04.790 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-04-23T10:32:04.790 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-04-23T10:31:21.457073+0000", 2024-04-23T10:32:04.790 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "header_read" 2024-04-23T10:32:04.790 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-23T10:32:04.790 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-04-23T10:32:04.790 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-04-23T10:31:21.457074+0000", 2024-04-23T10:32:04.790 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "throttled" 2024-04-23T10:32:04.790 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-23T10:32:04.790 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-04-23T10:32:04.791 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-04-23T10:31:21.457080+0000", 2024-04-23T10:32:04.791 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "all_read" 2024-04-23T10:32:04.791 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-23T10:32:04.791 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-04-23T10:32:04.791 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-04-23T10:31:21.462839+0000", 2024-04-23T10:32:04.791 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "dispatched" 2024-04-23T10:32:04.791 INFO:teuthology.orchestra.run.smithi114.stdout: } 2024-04-23T10:32:04.791 INFO:teuthology.orchestra.run.smithi114.stdout: ], 2024-04-23T10:32:04.791 INFO:teuthology.orchestra.run.smithi114.stdout: "locks": [] 2024-04-23T10:32:04.791 INFO:teuthology.orchestra.run.smithi114.stdout: } 2024-04-23T10:32:04.791 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-23T10:32:04.791 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-04-23T10:32:04.791 INFO:teuthology.orchestra.run.smithi114.stdout: "description": "internal op quiesce_inode:mds.0:200 fp=#0x10000000216", 2024-04-23T10:32:04.791 INFO:teuthology.orchestra.run.smithi114.stdout: "initiated_at": "2024-04-23T10:31:21.483518+0000", 2024-04-23T10:32:04.791 INFO:teuthology.orchestra.run.smithi114.stdout: "age": 42.400871037999998, 2024-04-23T10:32:04.792 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 42.403846166000001, 2024-04-23T10:32:04.792 INFO:teuthology.orchestra.run.smithi114.stdout: "continuous": true, 2024-04-23T10:32:04.792 INFO:teuthology.orchestra.run.smithi114.stdout: "type_data": { 2024-04-23T10:32:04.792 INFO:teuthology.orchestra.run.smithi114.stdout: "result": -2147483648, 2024-04-23T10:32:04.792 INFO:teuthology.orchestra.run.smithi114.stdout: "flag_point": "quiesce complete for non-auth inode", 2024-04-23T10:32:04.792 INFO:teuthology.orchestra.run.smithi114.stdout: "reqid": { 2024-04-23T10:32:04.792 INFO:teuthology.orchestra.run.smithi114.stdout: "entity": { 2024-04-23T10:32:04.792 INFO:teuthology.orchestra.run.smithi114.stdout: "type": "mds", 2024-04-23T10:32:04.792 INFO:teuthology.orchestra.run.smithi114.stdout: "num": 0 2024-04-23T10:32:04.792 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-23T10:32:04.792 INFO:teuthology.orchestra.run.smithi114.stdout: "tid": 200 2024-04-23T10:32:04.792 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-23T10:32:04.792 INFO:teuthology.orchestra.run.smithi114.stdout: "op_type": "internal_op", 2024-04-23T10:32:04.792 INFO:teuthology.orchestra.run.smithi114.stdout: "internal_op": 5385, 2024-04-23T10:32:04.793 INFO:teuthology.orchestra.run.smithi114.stdout: "op_name": "quiesce_inode", 2024-04-23T10:32:04.793 INFO:teuthology.orchestra.run.smithi114.stdout: "events": [ 2024-04-23T10:32:04.793 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-04-23T10:32:04.793 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-04-23T10:31:21.483518+0000", 2024-04-23T10:32:04.793 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "initiated" 2024-04-23T10:32:04.793 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-23T10:32:04.793 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-04-23T10:32:04.793 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-04-23T10:31:21.483518+0000", 2024-04-23T10:32:04.793 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "header_read" 2024-04-23T10:32:04.793 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-23T10:32:04.793 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-04-23T10:32:04.793 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-04-23T10:31:21.483518+0000", 2024-04-23T10:32:04.793 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "throttled" 2024-04-23T10:32:04.793 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-23T10:32:04.793 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-04-23T10:32:04.793 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-04-23T10:31:21.483518+0000", 2024-04-23T10:32:04.794 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "all_read" 2024-04-23T10:32:04.794 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-23T10:32:04.794 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-04-23T10:32:04.794 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-04-23T10:31:21.483518+0000", 2024-04-23T10:32:04.794 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "dispatched" 2024-04-23T10:32:04.794 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-23T10:32:04.794 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-04-23T10:32:04.794 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-04-23T10:31:21.484100+0000", 2024-04-23T10:32:04.794 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "requesting remote authpins" 2024-04-23T10:32:04.794 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-23T10:32:04.794 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-04-23T10:32:04.794 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-04-23T10:31:21.508178+0000", 2024-04-23T10:32:04.794 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "acquired locks" 2024-04-23T10:32:04.794 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-23T10:32:04.795 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-04-23T10:32:04.795 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-04-23T10:31:21.508187+0000", 2024-04-23T10:32:04.795 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "quiesce complete for non-auth inode" 2024-04-23T10:32:04.795 INFO:teuthology.orchestra.run.smithi114.stdout: } 2024-04-23T10:32:04.795 INFO:teuthology.orchestra.run.smithi114.stdout: ], 2024-04-23T10:32:04.795 INFO:teuthology.orchestra.run.smithi114.stdout: "locks": [ 2024-04-23T10:32:04.795 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-04-23T10:32:04.795 INFO:teuthology.orchestra.run.smithi114.stdout: "object": { 2024-04-23T10:32:04.795 INFO:teuthology.orchestra.run.smithi114.stdout: "is_auth": false, 2024-04-23T10:32:04.795 INFO:teuthology.orchestra.run.smithi114.stdout: "auth_state": { 2024-04-23T10:32:04.795 INFO:teuthology.orchestra.run.smithi114.stdout: "replicas": {} 2024-04-23T10:32:04.795 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-23T10:32:04.795 INFO:teuthology.orchestra.run.smithi114.stdout: "replica_state": { 2024-04-23T10:32:04.795 INFO:teuthology.orchestra.run.smithi114.stdout: "authority": [ 2024-04-23T10:32:04.796 INFO:teuthology.orchestra.run.smithi114.stdout: 2, 2024-04-23T10:32:04.796 INFO:teuthology.orchestra.run.smithi114.stdout: -2 2024-04-23T10:32:04.796 INFO:teuthology.orchestra.run.smithi114.stdout: ], 2024-04-23T10:32:04.796 INFO:teuthology.orchestra.run.smithi114.stdout: "replica_nonce": 1 2024-04-23T10:32:04.796 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-23T10:32:04.796 INFO:teuthology.orchestra.run.smithi114.stdout: "auth_pins": 0, 2024-04-23T10:32:04.796 INFO:teuthology.orchestra.run.smithi114.stdout: "is_frozen": false, 2024-04-23T10:32:04.796 INFO:teuthology.orchestra.run.smithi114.stdout: "is_freezing": false, 2024-04-23T10:32:04.796 INFO:teuthology.orchestra.run.smithi114.stdout: "pins": { 2024-04-23T10:32:04.796 INFO:teuthology.orchestra.run.smithi114.stdout: "request": 1, 2024-04-23T10:32:04.796 INFO:teuthology.orchestra.run.smithi114.stdout: "lock": 1, 2024-04-23T10:32:04.796 INFO:teuthology.orchestra.run.smithi114.stdout: "caps": 0, 2024-04-23T10:32:04.796 INFO:teuthology.orchestra.run.smithi114.stdout: "exportingcaps": 0, 2024-04-23T10:32:04.796 INFO:teuthology.orchestra.run.smithi114.stdout: "dirtyparent": 0, 2024-04-23T10:32:04.796 INFO:teuthology.orchestra.run.smithi114.stdout: "dirty": 0, 2024-04-23T10:32:04.797 INFO:teuthology.orchestra.run.smithi114.stdout: "authpin": 0, 2024-04-23T10:32:04.797 INFO:teuthology.orchestra.run.smithi114.stdout: "tempexporting": 0 2024-04-23T10:32:04.797 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-23T10:32:04.797 INFO:teuthology.orchestra.run.smithi114.stdout: "nref": 2 2024-04-23T10:32:04.797 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-23T10:32:04.797 INFO:teuthology.orchestra.run.smithi114.stdout: "object_string": "[inode 0x10000000216 [7,head] /volumes/qa/sv_1/b206794e-7622-4ab8-8943-f3ae4e65b7a3/client.0/tmp/clients/client0/~dmtmp/COREL/CDRROLS.CFG rep@2.1 v7238 s=260096 n(v0 rc2024-04-23T10:29:16.539476+0000 b260096 1=1+0)/n(v0 1=1+0) (iquiesce lock x=1 by request(mds.0:200 nref=3)) | request=1 lock=1 caps=0 exportingcaps=0 dirtyparent=0 dirty=0 authpin=0 tempexporting=0 0x5571cf560000]", 2024-04-23T10:32:04.797 INFO:teuthology.orchestra.run.smithi114.stdout: "lock": { 2024-04-23T10:32:04.797 INFO:teuthology.orchestra.run.smithi114.stdout: "gather_set": [], 2024-04-23T10:32:04.797 INFO:teuthology.orchestra.run.smithi114.stdout: "state": "lock", 2024-04-23T10:32:04.797 INFO:teuthology.orchestra.run.smithi114.stdout: "type": "iquiesce", 2024-04-23T10:32:04.797 INFO:teuthology.orchestra.run.smithi114.stdout: "is_leased": false, 2024-04-23T10:32:04.797 INFO:teuthology.orchestra.run.smithi114.stdout: "num_rdlocks": 0, 2024-04-23T10:32:04.798 INFO:teuthology.orchestra.run.smithi114.stdout: "num_wrlocks": 0, 2024-04-23T10:32:04.798 INFO:teuthology.orchestra.run.smithi114.stdout: "num_xlocks": 1, 2024-04-23T10:32:04.798 INFO:teuthology.orchestra.run.smithi114.stdout: "xlock_by": { 2024-04-23T10:32:04.798 INFO:teuthology.orchestra.run.smithi114.stdout: "reqid": { 2024-04-23T10:32:04.798 INFO:teuthology.orchestra.run.smithi114.stdout: "entity": { 2024-04-23T10:32:04.798 INFO:teuthology.orchestra.run.smithi114.stdout: "type": "mds", 2024-04-23T10:32:04.798 INFO:teuthology.orchestra.run.smithi114.stdout: "num": 0 2024-04-23T10:32:04.798 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-23T10:32:04.798 INFO:teuthology.orchestra.run.smithi114.stdout: "tid": 200 2024-04-23T10:32:04.798 INFO:teuthology.orchestra.run.smithi114.stdout: } 2024-04-23T10:32:04.798 INFO:teuthology.orchestra.run.smithi114.stdout: } 2024-04-23T10:32:04.798 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-23T10:32:04.798 INFO:teuthology.orchestra.run.smithi114.stdout: "flags": 4, 2024-04-23T10:32:04.799 INFO:teuthology.orchestra.run.smithi114.stdout: "wrlock_target": -1 2024-04-23T10:32:04.799 INFO:teuthology.orchestra.run.smithi114.stdout: } 2024-04-23T10:32:04.799 INFO:teuthology.orchestra.run.smithi114.stdout: ] 2024-04-23T10:32:04.799 INFO:teuthology.orchestra.run.smithi114.stdout: } 2024-04-23T10:32:04.799 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-23T10:32:04.799 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-04-23T10:32:04.799 INFO:teuthology.orchestra.run.smithi114.stdout: "description": "peer_request:mds.1:106", 2024-04-23T10:32:04.799 INFO:teuthology.orchestra.run.smithi114.stdout: "initiated_at": "2024-04-23T10:31:21.448440+0000", 2024-04-23T10:32:04.799 INFO:teuthology.orchestra.run.smithi114.stdout: "age": 42.435949223000001, 2024-04-23T10:32:04.799 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 42.438983038000003, 2024-04-23T10:32:04.799 INFO:teuthology.orchestra.run.smithi114.stdout: "continuous": false, 2024-04-23T10:32:04.799 INFO:teuthology.orchestra.run.smithi114.stdout: "type_data": { 2024-04-23T10:32:04.799 INFO:teuthology.orchestra.run.smithi114.stdout: "result": -2147483648, 2024-04-23T10:32:04.799 INFO:teuthology.orchestra.run.smithi114.stdout: "flag_point": "dispatched", 2024-04-23T10:32:04.799 INFO:teuthology.orchestra.run.smithi114.stdout: "reqid": { 2024-04-23T10:32:04.800 INFO:teuthology.orchestra.run.smithi114.stdout: "entity": { 2024-04-23T10:32:04.800 INFO:teuthology.orchestra.run.smithi114.stdout: "type": "mds", 2024-04-23T10:32:04.800 INFO:teuthology.orchestra.run.smithi114.stdout: "num": 1 2024-04-23T10:32:04.800 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-23T10:32:04.800 INFO:teuthology.orchestra.run.smithi114.stdout: "tid": 106 2024-04-23T10:32:04.800 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-23T10:32:04.800 INFO:teuthology.orchestra.run.smithi114.stdout: "op_type": "peer_request", 2024-04-23T10:32:04.800 INFO:teuthology.orchestra.run.smithi114.stdout: "leader_info": { 2024-04-23T10:32:04.800 INFO:teuthology.orchestra.run.smithi114.stdout: "leader": "1" 2024-04-23T10:32:04.800 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-23T10:32:04.800 INFO:teuthology.orchestra.run.smithi114.stdout: "events": [ 2024-04-23T10:32:04.800 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-04-23T10:32:04.800 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-04-23T10:31:21.448440+0000", 2024-04-23T10:32:04.800 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "initiated" 2024-04-23T10:32:04.801 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-23T10:32:04.801 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-04-23T10:32:04.801 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-04-23T10:31:21.448440+0000", 2024-04-23T10:32:04.801 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "header_read" 2024-04-23T10:32:04.801 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-23T10:32:04.801 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-04-23T10:32:04.801 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-04-23T10:31:21.448441+0000", 2024-04-23T10:32:04.801 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "throttled" 2024-04-23T10:32:04.801 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-23T10:32:04.801 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-04-23T10:32:04.801 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-04-23T10:31:21.448454+0000", 2024-04-23T10:32:04.801 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "all_read" 2024-04-23T10:32:04.801 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-23T10:32:04.801 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-04-23T10:32:04.801 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-04-23T10:31:21.449319+0000", 2024-04-23T10:32:04.801 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "dispatched" 2024-04-23T10:32:04.802 INFO:teuthology.orchestra.run.smithi114.stdout: } 2024-04-23T10:32:04.802 INFO:teuthology.orchestra.run.smithi114.stdout: ], 2024-04-23T10:32:04.802 INFO:teuthology.orchestra.run.smithi114.stdout: "locks": [] 2024-04-23T10:32:04.802 INFO:teuthology.orchestra.run.smithi114.stdout: } 2024-04-23T10:32:04.802 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-23T10:32:04.802 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-04-23T10:32:04.802 INFO:teuthology.orchestra.run.smithi114.stdout: "description": "internal op quiesce_inode:mds.0:175 fp=#0x1000000020d", 2024-04-23T10:32:04.802 INFO:teuthology.orchestra.run.smithi114.stdout: "initiated_at": "2024-04-23T10:31:21.463518+0000", 2024-04-23T10:32:04.802 INFO:teuthology.orchestra.run.smithi114.stdout: "age": 42.420870845000003, 2024-04-23T10:32:04.802 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 42.423949573000002, 2024-04-23T10:32:04.802 INFO:teuthology.orchestra.run.smithi114.stdout: "continuous": true, 2024-04-23T10:32:04.802 INFO:teuthology.orchestra.run.smithi114.stdout: "type_data": { 2024-04-23T10:32:04.802 INFO:teuthology.orchestra.run.smithi114.stdout: "result": -2147483648, 2024-04-23T10:32:04.803 INFO:teuthology.orchestra.run.smithi114.stdout: "flag_point": "quiesce complete", 2024-04-23T10:32:04.803 INFO:teuthology.orchestra.run.smithi114.stdout: "reqid": { 2024-04-23T10:32:04.803 INFO:teuthology.orchestra.run.smithi114.stdout: "entity": { 2024-04-23T10:32:04.803 INFO:teuthology.orchestra.run.smithi114.stdout: "type": "mds", 2024-04-23T10:32:04.803 INFO:teuthology.orchestra.run.smithi114.stdout: "num": 0 2024-04-23T10:32:04.803 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-23T10:32:04.803 INFO:teuthology.orchestra.run.smithi114.stdout: "tid": 175 2024-04-23T10:32:04.803 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-23T10:32:04.803 INFO:teuthology.orchestra.run.smithi114.stdout: "op_type": "internal_op", 2024-04-23T10:32:04.803 INFO:teuthology.orchestra.run.smithi114.stdout: "internal_op": 5385, 2024-04-23T10:32:04.803 INFO:teuthology.orchestra.run.smithi114.stdout: "op_name": "quiesce_inode", 2024-04-23T10:32:04.803 INFO:teuthology.orchestra.run.smithi114.stdout: "events": [ 2024-04-23T10:32:04.803 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-04-23T10:32:04.803 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-04-23T10:31:21.463518+0000", 2024-04-23T10:32:04.804 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "initiated" 2024-04-23T10:32:04.804 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-23T10:32:04.804 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-04-23T10:32:04.804 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-04-23T10:31:21.463518+0000", 2024-04-23T10:32:04.804 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "header_read" 2024-04-23T10:32:04.804 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-23T10:32:04.804 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-04-23T10:32:04.804 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-04-23T10:31:21.463518+0000", 2024-04-23T10:32:04.804 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "throttled" 2024-04-23T10:32:04.804 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-23T10:32:04.804 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-04-23T10:32:04.804 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-04-23T10:31:21.463518+0000", 2024-04-23T10:32:04.804 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "all_read" 2024-04-23T10:32:04.804 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-23T10:32:04.804 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-04-23T10:32:04.804 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-04-23T10:31:21.463518+0000", 2024-04-23T10:32:04.805 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "dispatched" 2024-04-23T10:32:04.805 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-23T10:32:04.805 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-04-23T10:32:04.805 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-04-23T10:31:21.463813+0000", 2024-04-23T10:32:04.805 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "failed to xlock, waiting" 2024-04-23T10:32:04.805 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-23T10:32:04.805 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-04-23T10:32:04.805 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-04-23T10:31:21.474631+0000", 2024-04-23T10:32:04.805 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "failed to xlock, waiting" 2024-04-23T10:32:04.805 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-23T10:32:04.805 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-04-23T10:32:04.805 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-04-23T10:31:21.485239+0000", 2024-04-23T10:32:04.805 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "acquired locks" 2024-04-23T10:32:04.805 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-23T10:32:04.805 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-04-23T10:32:04.806 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-04-23T10:31:21.485479+0000", 2024-04-23T10:32:04.806 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "quiesce complete" 2024-04-23T10:32:04.806 INFO:teuthology.orchestra.run.smithi114.stdout: } 2024-04-23T10:32:04.806 INFO:teuthology.orchestra.run.smithi114.stdout: ], 2024-04-23T10:32:04.806 INFO:teuthology.orchestra.run.smithi114.stdout: "locks": [ 2024-04-23T10:32:04.806 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-04-23T10:32:04.806 INFO:teuthology.orchestra.run.smithi114.stdout: "object": { 2024-04-23T10:32:04.806 INFO:teuthology.orchestra.run.smithi114.stdout: "is_auth": true, 2024-04-23T10:32:04.806 INFO:teuthology.orchestra.run.smithi114.stdout: "auth_state": { 2024-04-23T10:32:04.806 INFO:teuthology.orchestra.run.smithi114.stdout: "replicas": {} 2024-04-23T10:32:04.806 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-23T10:32:04.806 INFO:teuthology.orchestra.run.smithi114.stdout: "replica_state": { 2024-04-23T10:32:04.806 INFO:teuthology.orchestra.run.smithi114.stdout: "authority": [ 2024-04-23T10:32:04.806 INFO:teuthology.orchestra.run.smithi114.stdout: 0, 2024-04-23T10:32:04.807 INFO:teuthology.orchestra.run.smithi114.stdout: -2 2024-04-23T10:32:04.807 INFO:teuthology.orchestra.run.smithi114.stdout: ], 2024-04-23T10:32:04.807 INFO:teuthology.orchestra.run.smithi114.stdout: "replica_nonce": 0 2024-04-23T10:32:04.807 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-23T10:32:04.807 INFO:teuthology.orchestra.run.smithi114.stdout: "auth_pins": 1, 2024-04-23T10:32:04.807 INFO:teuthology.orchestra.run.smithi114.stdout: "is_frozen": false, 2024-04-23T10:32:04.807 INFO:teuthology.orchestra.run.smithi114.stdout: "is_freezing": false, 2024-04-23T10:32:04.807 INFO:teuthology.orchestra.run.smithi114.stdout: "pins": { 2024-04-23T10:32:04.807 INFO:teuthology.orchestra.run.smithi114.stdout: "ptrwaiter": 0, 2024-04-23T10:32:04.807 INFO:teuthology.orchestra.run.smithi114.stdout: "request": 1, 2024-04-23T10:32:04.807 INFO:teuthology.orchestra.run.smithi114.stdout: "lock": 1, 2024-04-23T10:32:04.807 INFO:teuthology.orchestra.run.smithi114.stdout: "caps": 0, 2024-04-23T10:32:04.807 INFO:teuthology.orchestra.run.smithi114.stdout: "needsnapflush": 0, 2024-04-23T10:32:04.807 INFO:teuthology.orchestra.run.smithi114.stdout: "dirtyparent": 0, 2024-04-23T10:32:04.808 INFO:teuthology.orchestra.run.smithi114.stdout: "dirty": 0, 2024-04-23T10:32:04.808 INFO:teuthology.orchestra.run.smithi114.stdout: "waiter": 0, 2024-04-23T10:32:04.808 INFO:teuthology.orchestra.run.smithi114.stdout: "authpin": 1 2024-04-23T10:32:04.808 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-23T10:32:04.808 INFO:teuthology.orchestra.run.smithi114.stdout: "nref": 3 2024-04-23T10:32:04.808 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-23T10:32:04.808 INFO:teuthology.orchestra.run.smithi114.stdout: "object_string": "[inode 0x1000000020d [7,head] /volumes/qa/sv_1/b206794e-7622-4ab8-8943-f3ae4e65b7a3/client.0/tmp/clients/client0/~dmtmp/EXCEL/PCMAGCD.XLS auth v8692 ap=1 s=260096 n(v0 rc2024-04-23T10:31:16.585743+0000 b260096 1=1+0) (iquiesce lock x=1 by request(mds.0:175 nref=3)) | ptrwaiter=0 request=1 lock=1 caps=0 needsnapflush=0 dirtyparent=0 dirty=0 waiter=0 authpin=1 0x5571cf53eb00]", 2024-04-23T10:32:04.808 INFO:teuthology.orchestra.run.smithi114.stdout: "lock": { 2024-04-23T10:32:04.808 INFO:teuthology.orchestra.run.smithi114.stdout: "gather_set": [], 2024-04-23T10:32:04.808 INFO:teuthology.orchestra.run.smithi114.stdout: "state": "lock", 2024-04-23T10:32:04.808 INFO:teuthology.orchestra.run.smithi114.stdout: "type": "iquiesce", 2024-04-23T10:32:04.808 INFO:teuthology.orchestra.run.smithi114.stdout: "is_leased": false, 2024-04-23T10:32:04.808 INFO:teuthology.orchestra.run.smithi114.stdout: "num_rdlocks": 0, 2024-04-23T10:32:04.808 INFO:teuthology.orchestra.run.smithi114.stdout: "num_wrlocks": 0, 2024-04-23T10:32:04.808 INFO:teuthology.orchestra.run.smithi114.stdout: "num_xlocks": 1, 2024-04-23T10:32:04.809 INFO:teuthology.orchestra.run.smithi114.stdout: "xlock_by": { 2024-04-23T10:32:04.809 INFO:teuthology.orchestra.run.smithi114.stdout: "reqid": { 2024-04-23T10:32:04.809 INFO:teuthology.orchestra.run.smithi114.stdout: "entity": { 2024-04-23T10:32:04.809 INFO:teuthology.orchestra.run.smithi114.stdout: "type": "mds", 2024-04-23T10:32:04.809 INFO:teuthology.orchestra.run.smithi114.stdout: "num": 0 2024-04-23T10:32:04.809 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-23T10:32:04.809 INFO:teuthology.orchestra.run.smithi114.stdout: "tid": 175 2024-04-23T10:32:04.809 INFO:teuthology.orchestra.run.smithi114.stdout: } 2024-04-23T10:32:04.809 INFO:teuthology.orchestra.run.smithi114.stdout: } 2024-04-23T10:32:04.809 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-23T10:32:04.809 INFO:teuthology.orchestra.run.smithi114.stdout: "flags": 4, 2024-04-23T10:32:04.809 INFO:teuthology.orchestra.run.smithi114.stdout: "wrlock_target": -1 2024-04-23T10:32:04.809 INFO:teuthology.orchestra.run.smithi114.stdout: } 2024-04-23T10:32:04.809 INFO:teuthology.orchestra.run.smithi114.stdout: ] 2024-04-23T10:32:04.809 INFO:teuthology.orchestra.run.smithi114.stdout: } 2024-04-23T10:32:04.809 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-23T10:32:04.810 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-04-23T10:32:04.810 INFO:teuthology.orchestra.run.smithi114.stdout: "description": "peer_request:mds.1:115", 2024-04-23T10:32:04.810 INFO:teuthology.orchestra.run.smithi114.stdout: "initiated_at": "2024-04-23T10:31:21.464762+0000", 2024-04-23T10:32:04.810 INFO:teuthology.orchestra.run.smithi114.stdout: "age": 42.419627384999998, 2024-04-23T10:32:04.810 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 42.422772530000003, 2024-04-23T10:32:04.810 INFO:teuthology.orchestra.run.smithi114.stdout: "continuous": false, 2024-04-23T10:32:04.810 INFO:teuthology.orchestra.run.smithi114.stdout: "type_data": { 2024-04-23T10:32:04.810 INFO:teuthology.orchestra.run.smithi114.stdout: "result": -2147483648, 2024-04-23T10:32:04.810 INFO:teuthology.orchestra.run.smithi114.stdout: "flag_point": "dispatched", 2024-04-23T10:32:04.810 INFO:teuthology.orchestra.run.smithi114.stdout: "reqid": { 2024-04-23T10:32:04.810 INFO:teuthology.orchestra.run.smithi114.stdout: "entity": { 2024-04-23T10:32:04.810 INFO:teuthology.orchestra.run.smithi114.stdout: "type": "mds", 2024-04-23T10:32:04.810 INFO:teuthology.orchestra.run.smithi114.stdout: "num": 1 2024-04-23T10:32:04.810 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-23T10:32:04.810 INFO:teuthology.orchestra.run.smithi114.stdout: "tid": 115 2024-04-23T10:32:04.811 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-23T10:32:04.811 INFO:teuthology.orchestra.run.smithi114.stdout: "op_type": "peer_request", 2024-04-23T10:32:04.811 INFO:teuthology.orchestra.run.smithi114.stdout: "leader_info": { 2024-04-23T10:32:04.811 INFO:teuthology.orchestra.run.smithi114.stdout: "leader": "1" 2024-04-23T10:32:04.811 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-23T10:32:04.811 INFO:teuthology.orchestra.run.smithi114.stdout: "events": [ 2024-04-23T10:32:04.811 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-04-23T10:32:04.811 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-04-23T10:31:21.464762+0000", 2024-04-23T10:32:04.811 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "initiated" 2024-04-23T10:32:04.811 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-23T10:32:04.811 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-04-23T10:32:04.811 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-04-23T10:31:21.464762+0000", 2024-04-23T10:32:04.811 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "header_read" 2024-04-23T10:32:04.811 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-23T10:32:04.811 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-04-23T10:32:04.812 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-04-23T10:31:21.464762+0000", 2024-04-23T10:32:04.812 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "throttled" 2024-04-23T10:32:04.812 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-23T10:32:04.812 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-04-23T10:32:04.812 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-04-23T10:31:21.464767+0000", 2024-04-23T10:32:04.812 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "all_read" 2024-04-23T10:32:04.812 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-23T10:32:04.812 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-04-23T10:32:04.812 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-04-23T10:31:21.481469+0000", 2024-04-23T10:32:04.812 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "dispatched" 2024-04-23T10:32:04.812 INFO:teuthology.orchestra.run.smithi114.stdout: } 2024-04-23T10:32:04.812 INFO:teuthology.orchestra.run.smithi114.stdout: ], 2024-04-23T10:32:04.812 INFO:teuthology.orchestra.run.smithi114.stdout: "locks": [] 2024-04-23T10:32:04.812 INFO:teuthology.orchestra.run.smithi114.stdout: } 2024-04-23T10:32:04.813 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-23T10:32:04.813 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-04-23T10:32:04.813 INFO:teuthology.orchestra.run.smithi114.stdout: "description": "internal op quiesce_inode:mds.0:155 fp=#0x100000001fb", 2024-04-23T10:32:04.813 INFO:teuthology.orchestra.run.smithi114.stdout: "initiated_at": "2024-04-23T10:31:21.449954+0000", 2024-04-23T10:32:04.813 INFO:teuthology.orchestra.run.smithi114.stdout: "age": 42.434434795000001, 2024-04-23T10:32:04.813 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 42.437603787, 2024-04-23T10:32:04.813 INFO:teuthology.orchestra.run.smithi114.stdout: "continuous": true, 2024-04-23T10:32:04.813 INFO:teuthology.orchestra.run.smithi114.stdout: "type_data": { 2024-04-23T10:32:04.813 INFO:teuthology.orchestra.run.smithi114.stdout: "result": -2147483648, 2024-04-23T10:32:04.813 INFO:teuthology.orchestra.run.smithi114.stdout: "flag_point": "quiesce complete", 2024-04-23T10:32:04.813 INFO:teuthology.orchestra.run.smithi114.stdout: "reqid": { 2024-04-23T10:32:04.813 INFO:teuthology.orchestra.run.smithi114.stdout: "entity": { 2024-04-23T10:32:04.813 INFO:teuthology.orchestra.run.smithi114.stdout: "type": "mds", 2024-04-23T10:32:04.813 INFO:teuthology.orchestra.run.smithi114.stdout: "num": 0 2024-04-23T10:32:04.814 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-23T10:32:04.814 INFO:teuthology.orchestra.run.smithi114.stdout: "tid": 155 2024-04-23T10:32:04.814 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-23T10:32:04.814 INFO:teuthology.orchestra.run.smithi114.stdout: "op_type": "internal_op", 2024-04-23T10:32:04.814 INFO:teuthology.orchestra.run.smithi114.stdout: "internal_op": 5385, 2024-04-23T10:32:04.814 INFO:teuthology.orchestra.run.smithi114.stdout: "op_name": "quiesce_inode", 2024-04-23T10:32:04.814 INFO:teuthology.orchestra.run.smithi114.stdout: "events": [ 2024-04-23T10:32:04.814 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-04-23T10:32:04.814 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-04-23T10:31:21.449954+0000", 2024-04-23T10:32:04.814 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "initiated" 2024-04-23T10:32:04.814 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-23T10:32:04.814 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-04-23T10:32:04.814 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-04-23T10:31:21.449954+0000", 2024-04-23T10:32:04.814 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "header_read" 2024-04-23T10:32:04.815 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-23T10:32:04.815 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-04-23T10:32:04.815 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-04-23T10:31:21.449954+0000", 2024-04-23T10:32:04.815 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "throttled" 2024-04-23T10:32:04.815 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-23T10:32:04.815 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-04-23T10:32:04.815 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-04-23T10:31:21.449954+0000", 2024-04-23T10:32:04.815 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "all_read" 2024-04-23T10:32:04.815 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-23T10:32:04.815 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-04-23T10:32:04.815 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-04-23T10:31:21.449954+0000", 2024-04-23T10:32:04.815 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "dispatched" 2024-04-23T10:32:04.815 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-23T10:32:04.815 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-04-23T10:32:04.815 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-04-23T10:31:21.450125+0000", 2024-04-23T10:32:04.815 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "failed to xlock, waiting" 2024-04-23T10:32:04.816 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-23T10:32:04.816 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-04-23T10:32:04.816 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-04-23T10:31:21.451103+0000", 2024-04-23T10:32:04.816 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "acquired locks" 2024-04-23T10:32:04.816 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-23T10:32:04.816 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-04-23T10:32:04.816 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-04-23T10:31:21.509964+0000", 2024-04-23T10:32:04.816 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "quiesce complete" 2024-04-23T10:32:04.816 INFO:teuthology.orchestra.run.smithi114.stdout: } 2024-04-23T10:32:04.816 INFO:teuthology.orchestra.run.smithi114.stdout: ], 2024-04-23T10:32:04.816 INFO:teuthology.orchestra.run.smithi114.stdout: "locks": [ 2024-04-23T10:32:04.816 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-04-23T10:32:04.816 INFO:teuthology.orchestra.run.smithi114.stdout: "object": { 2024-04-23T10:32:04.816 INFO:teuthology.orchestra.run.smithi114.stdout: "is_auth": true, 2024-04-23T10:32:04.817 INFO:teuthology.orchestra.run.smithi114.stdout: "auth_state": { 2024-04-23T10:32:04.817 INFO:teuthology.orchestra.run.smithi114.stdout: "replicas": { 2024-04-23T10:32:04.817 INFO:teuthology.orchestra.run.smithi114.stdout: "1": 1, 2024-04-23T10:32:04.817 INFO:teuthology.orchestra.run.smithi114.stdout: "2": 1 2024-04-23T10:32:04.817 INFO:teuthology.orchestra.run.smithi114.stdout: } 2024-04-23T10:32:04.817 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-23T10:32:04.817 INFO:teuthology.orchestra.run.smithi114.stdout: "replica_state": { 2024-04-23T10:32:04.817 INFO:teuthology.orchestra.run.smithi114.stdout: "authority": [ 2024-04-23T10:32:04.817 INFO:teuthology.orchestra.run.smithi114.stdout: 0, 2024-04-23T10:32:04.817 INFO:teuthology.orchestra.run.smithi114.stdout: -2 2024-04-23T10:32:04.817 INFO:teuthology.orchestra.run.smithi114.stdout: ], 2024-04-23T10:32:04.817 INFO:teuthology.orchestra.run.smithi114.stdout: "replica_nonce": 0 2024-04-23T10:32:04.817 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-23T10:32:04.817 INFO:teuthology.orchestra.run.smithi114.stdout: "auth_pins": 3, 2024-04-23T10:32:04.817 INFO:teuthology.orchestra.run.smithi114.stdout: "is_frozen": false, 2024-04-23T10:32:04.818 INFO:teuthology.orchestra.run.smithi114.stdout: "is_freezing": false, 2024-04-23T10:32:04.818 INFO:teuthology.orchestra.run.smithi114.stdout: "pins": { 2024-04-23T10:32:04.818 INFO:teuthology.orchestra.run.smithi114.stdout: "dirtyscattered": 1, 2024-04-23T10:32:04.818 INFO:teuthology.orchestra.run.smithi114.stdout: "request": 1, 2024-04-23T10:32:04.818 INFO:teuthology.orchestra.run.smithi114.stdout: "lock": 1, 2024-04-23T10:32:04.818 INFO:teuthology.orchestra.run.smithi114.stdout: "dirfrag": 1, 2024-04-23T10:32:04.818 INFO:teuthology.orchestra.run.smithi114.stdout: "caps": 1, 2024-04-23T10:32:04.818 INFO:teuthology.orchestra.run.smithi114.stdout: "dirtyrstat": 1, 2024-04-23T10:32:04.818 INFO:teuthology.orchestra.run.smithi114.stdout: "dirtyparent": 0, 2024-04-23T10:32:04.818 INFO:teuthology.orchestra.run.smithi114.stdout: "replicated": 1, 2024-04-23T10:32:04.818 INFO:teuthology.orchestra.run.smithi114.stdout: "dirty": 1, 2024-04-23T10:32:04.818 INFO:teuthology.orchestra.run.smithi114.stdout: "waiter": 0, 2024-04-23T10:32:04.818 INFO:teuthology.orchestra.run.smithi114.stdout: "authpin": 1 2024-04-23T10:32:04.818 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-23T10:32:04.818 INFO:teuthology.orchestra.run.smithi114.stdout: "nref": 9 2024-04-23T10:32:04.819 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-23T10:32:04.819 INFO:teuthology.orchestra.run.smithi114.stdout: "object_string": "[inode 0x100000001fb [...7,head] /volumes/qa/sv_1/b206794e-7622-4ab8-8943-f3ae4e65b7a3/ auth{1=1,2=1} v426 ap=3 f(v0 m2024-04-23T10:26:48.475842+0000 1=0+1) n(v25 rc2024-04-23T10:31:17.764716+0000 b21075074 78=63+15)/n(v24 rc2024-04-23T10:31:17.764716+0000 b21075074 78=63+15) old_inodes=5 (inest mix dirty) (iquiesce lock x=1 by request(mds.0:155 nref=3)) caps={15312=pAsLsXs/-@18},l=15312 | dirtyscattered=1 request=1 lock=1 dirfrag=1 caps=1 dirtyrstat=1 dirtyparent=0 replicated=1 dirty=1 waiter=0 authpin=1 0x5571cf1e0100]", 2024-04-23T10:32:04.819 INFO:teuthology.orchestra.run.smithi114.stdout: "lock": { 2024-04-23T10:32:04.819 INFO:teuthology.orchestra.run.smithi114.stdout: "gather_set": [], 2024-04-23T10:32:04.819 INFO:teuthology.orchestra.run.smithi114.stdout: "state": "lock", 2024-04-23T10:32:04.819 INFO:teuthology.orchestra.run.smithi114.stdout: "type": "iquiesce", 2024-04-23T10:32:04.819 INFO:teuthology.orchestra.run.smithi114.stdout: "is_leased": false, 2024-04-23T10:32:04.819 INFO:teuthology.orchestra.run.smithi114.stdout: "num_rdlocks": 0, 2024-04-23T10:32:04.819 INFO:teuthology.orchestra.run.smithi114.stdout: "num_wrlocks": 0, 2024-04-23T10:32:04.819 INFO:teuthology.orchestra.run.smithi114.stdout: "num_xlocks": 1, 2024-04-23T10:32:04.819 INFO:teuthology.orchestra.run.smithi114.stdout: "xlock_by": { 2024-04-23T10:32:04.819 INFO:teuthology.orchestra.run.smithi114.stdout: "reqid": { 2024-04-23T10:32:04.819 INFO:teuthology.orchestra.run.smithi114.stdout: "entity": { 2024-04-23T10:32:04.819 INFO:teuthology.orchestra.run.smithi114.stdout: "type": "mds", 2024-04-23T10:32:04.819 INFO:teuthology.orchestra.run.smithi114.stdout: "num": 0 2024-04-23T10:32:04.820 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-23T10:32:04.820 INFO:teuthology.orchestra.run.smithi114.stdout: "tid": 155 2024-04-23T10:32:04.820 INFO:teuthology.orchestra.run.smithi114.stdout: } 2024-04-23T10:32:04.820 INFO:teuthology.orchestra.run.smithi114.stdout: } 2024-04-23T10:32:04.820 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-23T10:32:04.820 INFO:teuthology.orchestra.run.smithi114.stdout: "flags": 4, 2024-04-23T10:32:04.820 INFO:teuthology.orchestra.run.smithi114.stdout: "wrlock_target": -1 2024-04-23T10:32:04.820 INFO:teuthology.orchestra.run.smithi114.stdout: } 2024-04-23T10:32:04.820 INFO:teuthology.orchestra.run.smithi114.stdout: ] 2024-04-23T10:32:04.820 INFO:teuthology.orchestra.run.smithi114.stdout: } 2024-04-23T10:32:04.820 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-23T10:32:04.820 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-04-23T10:32:04.820 INFO:teuthology.orchestra.run.smithi114.stdout: "description": "internal op quiesce_inode:mds.0:176 fp=#0x1000000020e", 2024-04-23T10:32:04.820 INFO:teuthology.orchestra.run.smithi114.stdout: "initiated_at": "2024-04-23T10:31:21.463538+0000", 2024-04-23T10:32:04.820 INFO:teuthology.orchestra.run.smithi114.stdout: "age": 42.420850881, 2024-04-23T10:32:04.821 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 42.424124407000001, 2024-04-23T10:32:04.821 INFO:teuthology.orchestra.run.smithi114.stdout: "continuous": true, 2024-04-23T10:32:04.821 INFO:teuthology.orchestra.run.smithi114.stdout: "type_data": { 2024-04-23T10:32:04.821 INFO:teuthology.orchestra.run.smithi114.stdout: "result": -2147483648, 2024-04-23T10:32:04.821 INFO:teuthology.orchestra.run.smithi114.stdout: "flag_point": "quiesce complete", 2024-04-23T10:32:04.821 INFO:teuthology.orchestra.run.smithi114.stdout: "reqid": { 2024-04-23T10:32:04.821 INFO:teuthology.orchestra.run.smithi114.stdout: "entity": { 2024-04-23T10:32:04.821 INFO:teuthology.orchestra.run.smithi114.stdout: "type": "mds", 2024-04-23T10:32:04.821 INFO:teuthology.orchestra.run.smithi114.stdout: "num": 0 2024-04-23T10:32:04.821 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-23T10:32:04.821 INFO:teuthology.orchestra.run.smithi114.stdout: "tid": 176 2024-04-23T10:32:04.821 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-23T10:32:04.821 INFO:teuthology.orchestra.run.smithi114.stdout: "op_type": "internal_op", 2024-04-23T10:32:04.821 INFO:teuthology.orchestra.run.smithi114.stdout: "internal_op": 5385, 2024-04-23T10:32:04.821 INFO:teuthology.orchestra.run.smithi114.stdout: "op_name": "quiesce_inode", 2024-04-23T10:32:04.821 INFO:teuthology.orchestra.run.smithi114.stdout: "events": [ 2024-04-23T10:32:04.822 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-04-23T10:32:04.822 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-04-23T10:31:21.463538+0000", 2024-04-23T10:32:04.822 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "initiated" 2024-04-23T10:32:04.822 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-23T10:32:04.822 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-04-23T10:32:04.822 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-04-23T10:31:21.463538+0000", 2024-04-23T10:32:04.822 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "header_read" 2024-04-23T10:32:04.822 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-23T10:32:04.822 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-04-23T10:32:04.822 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-04-23T10:31:21.463538+0000", 2024-04-23T10:32:04.822 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "throttled" 2024-04-23T10:32:04.822 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-23T10:32:04.822 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-04-23T10:32:04.822 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-04-23T10:31:21.463538+0000", 2024-04-23T10:32:04.823 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "all_read" 2024-04-23T10:32:04.823 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-23T10:32:04.823 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-04-23T10:32:04.823 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-04-23T10:31:21.463538+0000", 2024-04-23T10:32:04.823 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "dispatched" 2024-04-23T10:32:04.823 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-23T10:32:04.823 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-04-23T10:32:04.823 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-04-23T10:31:21.464062+0000", 2024-04-23T10:32:04.823 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "failed to xlock, waiting" 2024-04-23T10:32:04.823 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-23T10:32:04.823 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-04-23T10:32:04.823 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-04-23T10:31:21.476699+0000", 2024-04-23T10:32:04.823 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "failed to xlock, waiting" 2024-04-23T10:32:04.823 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-23T10:32:04.823 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-04-23T10:32:04.823 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-04-23T10:31:21.488211+0000", 2024-04-23T10:32:04.824 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "acquired locks" 2024-04-23T10:32:04.824 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-23T10:32:04.824 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-04-23T10:32:04.824 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-04-23T10:31:21.488419+0000", 2024-04-23T10:32:04.824 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "quiesce complete" 2024-04-23T10:32:04.824 INFO:teuthology.orchestra.run.smithi114.stdout: } 2024-04-23T10:32:04.824 INFO:teuthology.orchestra.run.smithi114.stdout: ], 2024-04-23T10:32:04.824 INFO:teuthology.orchestra.run.smithi114.stdout: "locks": [ 2024-04-23T10:32:04.824 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-04-23T10:32:04.824 INFO:teuthology.orchestra.run.smithi114.stdout: "object": { 2024-04-23T10:32:04.824 INFO:teuthology.orchestra.run.smithi114.stdout: "is_auth": true, 2024-04-23T10:32:04.824 INFO:teuthology.orchestra.run.smithi114.stdout: "auth_state": { 2024-04-23T10:32:04.824 INFO:teuthology.orchestra.run.smithi114.stdout: "replicas": {} 2024-04-23T10:32:04.824 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-23T10:32:04.824 INFO:teuthology.orchestra.run.smithi114.stdout: "replica_state": { 2024-04-23T10:32:04.825 INFO:teuthology.orchestra.run.smithi114.stdout: "authority": [ 2024-04-23T10:32:04.825 INFO:teuthology.orchestra.run.smithi114.stdout: 0, 2024-04-23T10:32:04.825 INFO:teuthology.orchestra.run.smithi114.stdout: -2 2024-04-23T10:32:04.825 INFO:teuthology.orchestra.run.smithi114.stdout: ], 2024-04-23T10:32:04.825 INFO:teuthology.orchestra.run.smithi114.stdout: "replica_nonce": 0 2024-04-23T10:32:04.825 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-23T10:32:04.825 INFO:teuthology.orchestra.run.smithi114.stdout: "auth_pins": 1, 2024-04-23T10:32:04.825 INFO:teuthology.orchestra.run.smithi114.stdout: "is_frozen": false, 2024-04-23T10:32:04.825 INFO:teuthology.orchestra.run.smithi114.stdout: "is_freezing": false, 2024-04-23T10:32:04.825 INFO:teuthology.orchestra.run.smithi114.stdout: "pins": { 2024-04-23T10:32:04.825 INFO:teuthology.orchestra.run.smithi114.stdout: "ptrwaiter": 0, 2024-04-23T10:32:04.825 INFO:teuthology.orchestra.run.smithi114.stdout: "request": 1, 2024-04-23T10:32:04.825 INFO:teuthology.orchestra.run.smithi114.stdout: "lock": 1, 2024-04-23T10:32:04.825 INFO:teuthology.orchestra.run.smithi114.stdout: "caps": 0, 2024-04-23T10:32:04.825 INFO:teuthology.orchestra.run.smithi114.stdout: "needsnapflush": 0, 2024-04-23T10:32:04.826 INFO:teuthology.orchestra.run.smithi114.stdout: "dirtyparent": 0, 2024-04-23T10:32:04.826 INFO:teuthology.orchestra.run.smithi114.stdout: "dirty": 0, 2024-04-23T10:32:04.826 INFO:teuthology.orchestra.run.smithi114.stdout: "waiter": 0, 2024-04-23T10:32:04.826 INFO:teuthology.orchestra.run.smithi114.stdout: "authpin": 1 2024-04-23T10:32:04.826 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-23T10:32:04.826 INFO:teuthology.orchestra.run.smithi114.stdout: "nref": 3 2024-04-23T10:32:04.826 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-23T10:32:04.826 INFO:teuthology.orchestra.run.smithi114.stdout: "object_string": "[inode 0x1000000020e [7,head] /volumes/qa/sv_1/b206794e-7622-4ab8-8943-f3ae4e65b7a3/client.0/tmp/clients/client0/~dmtmp/EXCEL/SALES.XLS auth v8694 ap=1 s=260096 n(v0 rc2024-04-23T10:31:16.692740+0000 b260096 1=1+0) (iquiesce lock x=1 by request(mds.0:176 nref=3)) | ptrwaiter=0 request=1 lock=1 caps=0 needsnapflush=0 dirtyparent=0 dirty=0 waiter=0 authpin=1 0x5571cf53f080]", 2024-04-23T10:32:04.826 INFO:teuthology.orchestra.run.smithi114.stdout: "lock": { 2024-04-23T10:32:04.826 INFO:teuthology.orchestra.run.smithi114.stdout: "gather_set": [], 2024-04-23T10:32:04.826 INFO:teuthology.orchestra.run.smithi114.stdout: "state": "lock", 2024-04-23T10:32:04.826 INFO:teuthology.orchestra.run.smithi114.stdout: "type": "iquiesce", 2024-04-23T10:32:04.826 INFO:teuthology.orchestra.run.smithi114.stdout: "is_leased": false, 2024-04-23T10:32:04.826 INFO:teuthology.orchestra.run.smithi114.stdout: "num_rdlocks": 0, 2024-04-23T10:32:04.826 INFO:teuthology.orchestra.run.smithi114.stdout: "num_wrlocks": 0, 2024-04-23T10:32:04.827 INFO:teuthology.orchestra.run.smithi114.stdout: "num_xlocks": 1, 2024-04-23T10:32:04.827 INFO:teuthology.orchestra.run.smithi114.stdout: "xlock_by": { 2024-04-23T10:32:04.827 INFO:teuthology.orchestra.run.smithi114.stdout: "reqid": { 2024-04-23T10:32:04.827 INFO:teuthology.orchestra.run.smithi114.stdout: "entity": { 2024-04-23T10:32:04.827 INFO:teuthology.orchestra.run.smithi114.stdout: "type": "mds", 2024-04-23T10:32:04.827 INFO:teuthology.orchestra.run.smithi114.stdout: "num": 0 2024-04-23T10:32:04.827 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-23T10:32:04.827 INFO:teuthology.orchestra.run.smithi114.stdout: "tid": 176 2024-04-23T10:32:04.827 INFO:teuthology.orchestra.run.smithi114.stdout: } 2024-04-23T10:32:04.827 INFO:teuthology.orchestra.run.smithi114.stdout: } 2024-04-23T10:32:04.827 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-23T10:32:04.827 INFO:teuthology.orchestra.run.smithi114.stdout: "flags": 4, 2024-04-23T10:32:04.827 INFO:teuthology.orchestra.run.smithi114.stdout: "wrlock_target": -1 2024-04-23T10:32:04.827 INFO:teuthology.orchestra.run.smithi114.stdout: } 2024-04-23T10:32:04.828 INFO:teuthology.orchestra.run.smithi114.stdout: ] 2024-04-23T10:32:04.828 INFO:teuthology.orchestra.run.smithi114.stdout: } 2024-04-23T10:32:04.828 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-23T10:32:04.828 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-04-23T10:32:04.828 INFO:teuthology.orchestra.run.smithi114.stdout: "description": "peer_request:mds.2:120", 2024-04-23T10:32:04.828 INFO:teuthology.orchestra.run.smithi114.stdout: "initiated_at": "2024-04-23T10:31:21.463567+0000", 2024-04-23T10:32:04.828 INFO:teuthology.orchestra.run.smithi114.stdout: "age": 42.420821902999997, 2024-04-23T10:32:04.828 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 42.424164765, 2024-04-23T10:32:04.828 INFO:teuthology.orchestra.run.smithi114.stdout: "continuous": false, 2024-04-23T10:32:04.828 INFO:teuthology.orchestra.run.smithi114.stdout: "type_data": { 2024-04-23T10:32:04.828 INFO:teuthology.orchestra.run.smithi114.stdout: "result": -2147483648, 2024-04-23T10:32:04.828 INFO:teuthology.orchestra.run.smithi114.stdout: "flag_point": "dispatched", 2024-04-23T10:32:04.828 INFO:teuthology.orchestra.run.smithi114.stdout: "reqid": { 2024-04-23T10:32:04.828 INFO:teuthology.orchestra.run.smithi114.stdout: "entity": { 2024-04-23T10:32:04.828 INFO:teuthology.orchestra.run.smithi114.stdout: "type": "mds", 2024-04-23T10:32:04.829 INFO:teuthology.orchestra.run.smithi114.stdout: "num": 2 2024-04-23T10:32:04.829 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-23T10:32:04.829 INFO:teuthology.orchestra.run.smithi114.stdout: "tid": 120 2024-04-23T10:32:04.829 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-23T10:32:04.829 INFO:teuthology.orchestra.run.smithi114.stdout: "op_type": "peer_request", 2024-04-23T10:32:04.829 INFO:teuthology.orchestra.run.smithi114.stdout: "leader_info": { 2024-04-23T10:32:04.829 INFO:teuthology.orchestra.run.smithi114.stdout: "leader": "2" 2024-04-23T10:32:04.829 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-23T10:32:04.829 INFO:teuthology.orchestra.run.smithi114.stdout: "events": [ 2024-04-23T10:32:04.829 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-04-23T10:32:04.829 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-04-23T10:31:21.463567+0000", 2024-04-23T10:32:04.829 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "initiated" 2024-04-23T10:32:04.829 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-23T10:32:04.829 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-04-23T10:32:04.829 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-04-23T10:31:21.463567+0000", 2024-04-23T10:32:04.830 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "header_read" 2024-04-23T10:32:04.830 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-23T10:32:04.830 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-04-23T10:32:04.830 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-04-23T10:31:21.463568+0000", 2024-04-23T10:32:04.830 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "throttled" 2024-04-23T10:32:04.830 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-23T10:32:04.830 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-04-23T10:32:04.830 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-04-23T10:31:21.463573+0000", 2024-04-23T10:32:04.830 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "all_read" 2024-04-23T10:32:04.830 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-23T10:32:04.830 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-04-23T10:32:04.830 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-04-23T10:31:21.484620+0000", 2024-04-23T10:32:04.830 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "dispatched" 2024-04-23T10:32:04.830 INFO:teuthology.orchestra.run.smithi114.stdout: } 2024-04-23T10:32:04.831 INFO:teuthology.orchestra.run.smithi114.stdout: ], 2024-04-23T10:32:04.831 INFO:teuthology.orchestra.run.smithi114.stdout: "locks": [] 2024-04-23T10:32:04.831 INFO:teuthology.orchestra.run.smithi114.stdout: } 2024-04-23T10:32:04.831 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-23T10:32:04.831 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-04-23T10:32:04.831 INFO:teuthology.orchestra.run.smithi114.stdout: "description": "peer_request:mds.2:111", 2024-04-23T10:32:04.831 INFO:teuthology.orchestra.run.smithi114.stdout: "initiated_at": "2024-04-23T10:31:21.449682+0000", 2024-04-23T10:32:04.831 INFO:teuthology.orchestra.run.smithi114.stdout: "age": 42.434707566, 2024-04-23T10:32:04.831 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 42.438100112999997, 2024-04-23T10:32:04.831 INFO:teuthology.orchestra.run.smithi114.stdout: "continuous": false, 2024-04-23T10:32:04.831 INFO:teuthology.orchestra.run.smithi114.stdout: "type_data": { 2024-04-23T10:32:04.831 INFO:teuthology.orchestra.run.smithi114.stdout: "result": -2147483648, 2024-04-23T10:32:04.831 INFO:teuthology.orchestra.run.smithi114.stdout: "flag_point": "dispatched", 2024-04-23T10:32:04.831 INFO:teuthology.orchestra.run.smithi114.stdout: "reqid": { 2024-04-23T10:32:04.831 INFO:teuthology.orchestra.run.smithi114.stdout: "entity": { 2024-04-23T10:32:04.832 INFO:teuthology.orchestra.run.smithi114.stdout: "type": "mds", 2024-04-23T10:32:04.832 INFO:teuthology.orchestra.run.smithi114.stdout: "num": 2 2024-04-23T10:32:04.832 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-23T10:32:04.832 INFO:teuthology.orchestra.run.smithi114.stdout: "tid": 111 2024-04-23T10:32:04.832 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-23T10:32:04.832 INFO:teuthology.orchestra.run.smithi114.stdout: "op_type": "peer_request", 2024-04-23T10:32:04.832 INFO:teuthology.orchestra.run.smithi114.stdout: "leader_info": { 2024-04-23T10:32:04.832 INFO:teuthology.orchestra.run.smithi114.stdout: "leader": "2" 2024-04-23T10:32:04.832 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-23T10:32:04.832 INFO:teuthology.orchestra.run.smithi114.stdout: "events": [ 2024-04-23T10:32:04.832 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-04-23T10:32:04.832 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-04-23T10:31:21.449682+0000", 2024-04-23T10:32:04.832 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "initiated" 2024-04-23T10:32:04.833 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-23T10:32:04.833 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-04-23T10:32:04.833 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-04-23T10:31:21.449682+0000", 2024-04-23T10:32:04.833 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "header_read" 2024-04-23T10:32:04.833 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-23T10:32:04.833 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-04-23T10:32:04.833 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-04-23T10:31:21.449682+0000", 2024-04-23T10:32:04.833 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "throttled" 2024-04-23T10:32:04.833 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-23T10:32:04.833 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-04-23T10:32:04.833 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-04-23T10:31:21.449689+0000", 2024-04-23T10:32:04.833 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "all_read" 2024-04-23T10:32:04.833 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-23T10:32:04.833 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-04-23T10:32:04.833 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-04-23T10:31:21.450602+0000", 2024-04-23T10:32:04.833 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "dispatched" 2024-04-23T10:32:04.834 INFO:teuthology.orchestra.run.smithi114.stdout: } 2024-04-23T10:32:04.834 INFO:teuthology.orchestra.run.smithi114.stdout: ], 2024-04-23T10:32:04.834 INFO:teuthology.orchestra.run.smithi114.stdout: "locks": [] 2024-04-23T10:32:04.834 INFO:teuthology.orchestra.run.smithi114.stdout: } 2024-04-23T10:32:04.834 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-23T10:32:04.834 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-04-23T10:32:04.834 INFO:teuthology.orchestra.run.smithi114.stdout: "description": "peer_request:mds.1:112", 2024-04-23T10:32:04.834 INFO:teuthology.orchestra.run.smithi114.stdout: "initiated_at": "2024-04-23T10:31:21.457115+0000", 2024-04-23T10:32:04.834 INFO:teuthology.orchestra.run.smithi114.stdout: "age": 42.427273952999997, 2024-04-23T10:32:04.834 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 42.430707071999997, 2024-04-23T10:32:04.834 INFO:teuthology.orchestra.run.smithi114.stdout: "continuous": false, 2024-04-23T10:32:04.834 INFO:teuthology.orchestra.run.smithi114.stdout: "type_data": { 2024-04-23T10:32:04.834 INFO:teuthology.orchestra.run.smithi114.stdout: "result": -2147483648, 2024-04-23T10:32:04.834 INFO:teuthology.orchestra.run.smithi114.stdout: "flag_point": "dispatched", 2024-04-23T10:32:04.834 INFO:teuthology.orchestra.run.smithi114.stdout: "reqid": { 2024-04-23T10:32:04.835 INFO:teuthology.orchestra.run.smithi114.stdout: "entity": { 2024-04-23T10:32:04.835 INFO:teuthology.orchestra.run.smithi114.stdout: "type": "mds", 2024-04-23T10:32:04.835 INFO:teuthology.orchestra.run.smithi114.stdout: "num": 1 2024-04-23T10:32:04.835 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-23T10:32:04.835 INFO:teuthology.orchestra.run.smithi114.stdout: "tid": 112 2024-04-23T10:32:04.835 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-23T10:32:04.835 INFO:teuthology.orchestra.run.smithi114.stdout: "op_type": "peer_request", 2024-04-23T10:32:04.835 INFO:teuthology.orchestra.run.smithi114.stdout: "leader_info": { 2024-04-23T10:32:04.835 INFO:teuthology.orchestra.run.smithi114.stdout: "leader": "1" 2024-04-23T10:32:04.835 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-23T10:32:04.835 INFO:teuthology.orchestra.run.smithi114.stdout: "events": [ 2024-04-23T10:32:04.835 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-04-23T10:32:04.835 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-04-23T10:31:21.457115+0000", 2024-04-23T10:32:04.835 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "initiated" 2024-04-23T10:32:04.835 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-23T10:32:04.835 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-04-23T10:32:04.836 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-04-23T10:31:21.457115+0000", 2024-04-23T10:32:04.836 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "header_read" 2024-04-23T10:32:04.836 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-23T10:32:04.836 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-04-23T10:32:04.836 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-04-23T10:31:21.457116+0000", 2024-04-23T10:32:04.836 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "throttled" 2024-04-23T10:32:04.836 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-23T10:32:04.836 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-04-23T10:32:04.836 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-04-23T10:31:21.457124+0000", 2024-04-23T10:32:04.836 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "all_read" 2024-04-23T10:32:04.836 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-23T10:32:04.836 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-04-23T10:32:04.836 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-04-23T10:31:21.464095+0000", 2024-04-23T10:32:04.836 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "dispatched" 2024-04-23T10:32:04.836 INFO:teuthology.orchestra.run.smithi114.stdout: } 2024-04-23T10:32:04.837 INFO:teuthology.orchestra.run.smithi114.stdout: ], 2024-04-23T10:32:04.837 INFO:teuthology.orchestra.run.smithi114.stdout: "locks": [] 2024-04-23T10:32:04.837 INFO:teuthology.orchestra.run.smithi114.stdout: } 2024-04-23T10:32:04.837 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-23T10:32:04.837 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-04-23T10:32:04.837 INFO:teuthology.orchestra.run.smithi114.stdout: "description": "peer_request:mds.1:116", 2024-04-23T10:32:04.837 INFO:teuthology.orchestra.run.smithi114.stdout: "initiated_at": "2024-04-23T10:31:21.464842+0000", 2024-04-23T10:32:04.837 INFO:teuthology.orchestra.run.smithi114.stdout: "age": 42.419547631, 2024-04-23T10:32:04.837 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 42.423019656000001, 2024-04-23T10:32:04.837 INFO:teuthology.orchestra.run.smithi114.stdout: "continuous": false, 2024-04-23T10:32:04.837 INFO:teuthology.orchestra.run.smithi114.stdout: "type_data": { 2024-04-23T10:32:04.837 INFO:teuthology.orchestra.run.smithi114.stdout: "result": -2147483648, 2024-04-23T10:32:04.837 INFO:teuthology.orchestra.run.smithi114.stdout: "flag_point": "dispatched", 2024-04-23T10:32:04.837 INFO:teuthology.orchestra.run.smithi114.stdout: "reqid": { 2024-04-23T10:32:04.837 INFO:teuthology.orchestra.run.smithi114.stdout: "entity": { 2024-04-23T10:32:04.837 INFO:teuthology.orchestra.run.smithi114.stdout: "type": "mds", 2024-04-23T10:32:04.838 INFO:teuthology.orchestra.run.smithi114.stdout: "num": 1 2024-04-23T10:32:04.838 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-23T10:32:04.838 INFO:teuthology.orchestra.run.smithi114.stdout: "tid": 116 2024-04-23T10:32:04.838 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-23T10:32:04.838 INFO:teuthology.orchestra.run.smithi114.stdout: "op_type": "peer_request", 2024-04-23T10:32:04.838 INFO:teuthology.orchestra.run.smithi114.stdout: "leader_info": { 2024-04-23T10:32:04.838 INFO:teuthology.orchestra.run.smithi114.stdout: "leader": "1" 2024-04-23T10:32:04.838 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-23T10:32:04.838 INFO:teuthology.orchestra.run.smithi114.stdout: "events": [ 2024-04-23T10:32:04.838 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-04-23T10:32:04.838 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-04-23T10:31:21.464842+0000", 2024-04-23T10:32:04.838 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "initiated" 2024-04-23T10:32:04.838 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-23T10:32:04.838 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-04-23T10:32:04.838 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-04-23T10:31:21.464842+0000", 2024-04-23T10:32:04.839 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "header_read" 2024-04-23T10:32:04.839 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-23T10:32:04.839 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-04-23T10:32:04.839 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-04-23T10:31:21.464842+0000", 2024-04-23T10:32:04.839 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "throttled" 2024-04-23T10:32:04.839 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-23T10:32:04.839 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-04-23T10:32:04.839 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-04-23T10:31:21.464847+0000", 2024-04-23T10:32:04.839 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "all_read" 2024-04-23T10:32:04.839 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-23T10:32:04.839 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-04-23T10:32:04.839 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-04-23T10:31:21.487773+0000", 2024-04-23T10:32:04.839 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "dispatched" 2024-04-23T10:32:04.839 INFO:teuthology.orchestra.run.smithi114.stdout: } 2024-04-23T10:32:04.839 INFO:teuthology.orchestra.run.smithi114.stdout: ], 2024-04-23T10:32:04.840 INFO:teuthology.orchestra.run.smithi114.stdout: "locks": [] 2024-04-23T10:32:04.840 INFO:teuthology.orchestra.run.smithi114.stdout: } 2024-04-23T10:32:04.840 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-23T10:32:04.840 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-04-23T10:32:04.840 INFO:teuthology.orchestra.run.smithi114.stdout: "description": "peer_request:mds.1:107", 2024-04-23T10:32:04.840 INFO:teuthology.orchestra.run.smithi114.stdout: "initiated_at": "2024-04-23T10:31:21.449527+0000", 2024-04-23T10:32:04.840 INFO:teuthology.orchestra.run.smithi114.stdout: "age": 42.434862643000002, 2024-04-23T10:32:04.840 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 42.438378360000002, 2024-04-23T10:32:04.840 INFO:teuthology.orchestra.run.smithi114.stdout: "continuous": false, 2024-04-23T10:32:04.840 INFO:teuthology.orchestra.run.smithi114.stdout: "type_data": { 2024-04-23T10:32:04.840 INFO:teuthology.orchestra.run.smithi114.stdout: "result": -2147483648, 2024-04-23T10:32:04.840 INFO:teuthology.orchestra.run.smithi114.stdout: "flag_point": "dispatched", 2024-04-23T10:32:04.840 INFO:teuthology.orchestra.run.smithi114.stdout: "reqid": { 2024-04-23T10:32:04.840 INFO:teuthology.orchestra.run.smithi114.stdout: "entity": { 2024-04-23T10:32:04.840 INFO:teuthology.orchestra.run.smithi114.stdout: "type": "mds", 2024-04-23T10:32:04.841 INFO:teuthology.orchestra.run.smithi114.stdout: "num": 1 2024-04-23T10:32:04.841 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-23T10:32:04.841 INFO:teuthology.orchestra.run.smithi114.stdout: "tid": 107 2024-04-23T10:32:04.841 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-23T10:32:04.841 INFO:teuthology.orchestra.run.smithi114.stdout: "op_type": "peer_request", 2024-04-23T10:32:04.841 INFO:teuthology.orchestra.run.smithi114.stdout: "leader_info": { 2024-04-23T10:32:04.841 INFO:teuthology.orchestra.run.smithi114.stdout: "leader": "1" 2024-04-23T10:32:04.841 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-23T10:32:04.841 INFO:teuthology.orchestra.run.smithi114.stdout: "events": [ 2024-04-23T10:32:04.841 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-04-23T10:32:04.841 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-04-23T10:31:21.449527+0000", 2024-04-23T10:32:04.841 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "initiated" 2024-04-23T10:32:04.841 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-23T10:32:04.841 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-04-23T10:32:04.841 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-04-23T10:31:21.449527+0000", 2024-04-23T10:32:04.842 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "header_read" 2024-04-23T10:32:04.842 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-23T10:32:04.842 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-04-23T10:32:04.842 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-04-23T10:31:21.449527+0000", 2024-04-23T10:32:04.842 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "throttled" 2024-04-23T10:32:04.842 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-23T10:32:04.842 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-04-23T10:32:04.842 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-04-23T10:31:21.449535+0000", 2024-04-23T10:32:04.842 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "all_read" 2024-04-23T10:32:04.842 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-23T10:32:04.842 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-04-23T10:32:04.842 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-04-23T10:31:21.450675+0000", 2024-04-23T10:32:04.842 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "dispatched" 2024-04-23T10:32:04.843 INFO:teuthology.orchestra.run.smithi114.stdout: } 2024-04-23T10:32:04.843 INFO:teuthology.orchestra.run.smithi114.stdout: ], 2024-04-23T10:32:04.843 INFO:teuthology.orchestra.run.smithi114.stdout: "locks": [] 2024-04-23T10:32:04.843 INFO:teuthology.orchestra.run.smithi114.stdout: } 2024-04-23T10:32:04.843 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-23T10:32:04.843 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-04-23T10:32:04.843 INFO:teuthology.orchestra.run.smithi114.stdout: "description": "internal op quiesce_inode:mds.0:177 fp=#0x10000000222", 2024-04-23T10:32:04.843 INFO:teuthology.orchestra.run.smithi114.stdout: "initiated_at": "2024-04-23T10:31:21.467357+0000", 2024-04-23T10:32:04.843 INFO:teuthology.orchestra.run.smithi114.stdout: "age": 42.417032163000002, 2024-04-23T10:32:04.843 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 42.420584648999998, 2024-04-23T10:32:04.843 INFO:teuthology.orchestra.run.smithi114.stdout: "continuous": true, 2024-04-23T10:32:04.843 INFO:teuthology.orchestra.run.smithi114.stdout: "type_data": { 2024-04-23T10:32:04.843 INFO:teuthology.orchestra.run.smithi114.stdout: "result": -2147483648, 2024-04-23T10:32:04.843 INFO:teuthology.orchestra.run.smithi114.stdout: "flag_point": "quiesce complete", 2024-04-23T10:32:04.843 INFO:teuthology.orchestra.run.smithi114.stdout: "reqid": { 2024-04-23T10:32:04.844 INFO:teuthology.orchestra.run.smithi114.stdout: "entity": { 2024-04-23T10:32:04.844 INFO:teuthology.orchestra.run.smithi114.stdout: "type": "mds", 2024-04-23T10:32:04.844 INFO:teuthology.orchestra.run.smithi114.stdout: "num": 0 2024-04-23T10:32:04.844 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-23T10:32:04.844 INFO:teuthology.orchestra.run.smithi114.stdout: "tid": 177 2024-04-23T10:32:04.844 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-23T10:32:04.844 INFO:teuthology.orchestra.run.smithi114.stdout: "op_type": "internal_op", 2024-04-23T10:32:04.844 INFO:teuthology.orchestra.run.smithi114.stdout: "internal_op": 5385, 2024-04-23T10:32:04.844 INFO:teuthology.orchestra.run.smithi114.stdout: "op_name": "quiesce_inode", 2024-04-23T10:32:04.844 INFO:teuthology.orchestra.run.smithi114.stdout: "events": [ 2024-04-23T10:32:04.844 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-04-23T10:32:04.844 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-04-23T10:31:21.467357+0000", 2024-04-23T10:32:04.844 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "initiated" 2024-04-23T10:32:04.844 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-23T10:32:04.845 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-04-23T10:32:04.845 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-04-23T10:31:21.467357+0000", 2024-04-23T10:32:04.845 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "header_read" 2024-04-23T10:32:04.845 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-23T10:32:04.845 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-04-23T10:32:04.845 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-04-23T10:31:21.467357+0000", 2024-04-23T10:32:04.845 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "throttled" 2024-04-23T10:32:04.845 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-23T10:32:04.845 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-04-23T10:32:04.845 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-04-23T10:31:21.467357+0000", 2024-04-23T10:32:04.845 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "all_read" 2024-04-23T10:32:04.845 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-23T10:32:04.845 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-04-23T10:32:04.845 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-04-23T10:31:21.467357+0000", 2024-04-23T10:32:04.845 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "dispatched" 2024-04-23T10:32:04.846 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-23T10:32:04.846 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-04-23T10:32:04.846 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-04-23T10:31:21.467553+0000", 2024-04-23T10:32:04.846 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "failed to xlock, waiting" 2024-04-23T10:32:04.846 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-23T10:32:04.846 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-04-23T10:32:04.846 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-04-23T10:31:21.478020+0000", 2024-04-23T10:32:04.846 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "acquired locks" 2024-04-23T10:32:04.846 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-23T10:32:04.846 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-04-23T10:32:04.846 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-04-23T10:31:21.478129+0000", 2024-04-23T10:32:04.846 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "quiesce complete" 2024-04-23T10:32:04.846 INFO:teuthology.orchestra.run.smithi114.stdout: } 2024-04-23T10:32:04.846 INFO:teuthology.orchestra.run.smithi114.stdout: ], 2024-04-23T10:32:04.847 INFO:teuthology.orchestra.run.smithi114.stdout: "locks": [ 2024-04-23T10:32:04.847 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-04-23T10:32:04.847 INFO:teuthology.orchestra.run.smithi114.stdout: "object": { 2024-04-23T10:32:04.847 INFO:teuthology.orchestra.run.smithi114.stdout: "is_auth": true, 2024-04-23T10:32:04.847 INFO:teuthology.orchestra.run.smithi114.stdout: "auth_state": { 2024-04-23T10:32:04.847 INFO:teuthology.orchestra.run.smithi114.stdout: "replicas": {} 2024-04-23T10:32:04.847 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-23T10:32:04.847 INFO:teuthology.orchestra.run.smithi114.stdout: "replica_state": { 2024-04-23T10:32:04.847 INFO:teuthology.orchestra.run.smithi114.stdout: "authority": [ 2024-04-23T10:32:04.847 INFO:teuthology.orchestra.run.smithi114.stdout: 0, 2024-04-23T10:32:04.847 INFO:teuthology.orchestra.run.smithi114.stdout: -2 2024-04-23T10:32:04.847 INFO:teuthology.orchestra.run.smithi114.stdout: ], 2024-04-23T10:32:04.847 INFO:teuthology.orchestra.run.smithi114.stdout: "replica_nonce": 0 2024-04-23T10:32:04.847 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-23T10:32:04.847 INFO:teuthology.orchestra.run.smithi114.stdout: "auth_pins": 1, 2024-04-23T10:32:04.847 INFO:teuthology.orchestra.run.smithi114.stdout: "is_frozen": false, 2024-04-23T10:32:04.847 INFO:teuthology.orchestra.run.smithi114.stdout: "is_freezing": false, 2024-04-23T10:32:04.848 INFO:teuthology.orchestra.run.smithi114.stdout: "pins": { 2024-04-23T10:32:04.848 INFO:teuthology.orchestra.run.smithi114.stdout: "ptrwaiter": 0, 2024-04-23T10:32:04.848 INFO:teuthology.orchestra.run.smithi114.stdout: "request": 1, 2024-04-23T10:32:04.848 INFO:teuthology.orchestra.run.smithi114.stdout: "lock": 1, 2024-04-23T10:32:04.848 INFO:teuthology.orchestra.run.smithi114.stdout: "caps": 0, 2024-04-23T10:32:04.848 INFO:teuthology.orchestra.run.smithi114.stdout: "needsnapflush": 0, 2024-04-23T10:32:04.848 INFO:teuthology.orchestra.run.smithi114.stdout: "dirtyparent": 0, 2024-04-23T10:32:04.848 INFO:teuthology.orchestra.run.smithi114.stdout: "dirty": 0, 2024-04-23T10:32:04.848 INFO:teuthology.orchestra.run.smithi114.stdout: "waiter": 0, 2024-04-23T10:32:04.848 INFO:teuthology.orchestra.run.smithi114.stdout: "authpin": 1 2024-04-23T10:32:04.848 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-23T10:32:04.848 INFO:teuthology.orchestra.run.smithi114.stdout: "nref": 3 2024-04-23T10:32:04.848 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-23T10:32:04.848 INFO:teuthology.orchestra.run.smithi114.stdout: "object_string": "[inode 0x10000000222 [7,head] /volumes/qa/sv_1/b206794e-7622-4ab8-8943-f3ae4e65b7a3/client.0/tmp/clients/client0/~dmtmp/WORDPRO/BENCHS.LWP auth v8110 ap=1 s=260096 n(v0 rc2024-04-23T10:27:52.058395+0000 b260096 1=1+0) (iquiesce lock x=1 by request(mds.0:177 nref=3)) | ptrwaiter=0 request=1 lock=1 caps=0 needsnapflush=0 dirtyparent=0 dirty=0 waiter=0 authpin=1 0x5571cf586000]", 2024-04-23T10:32:04.849 INFO:teuthology.orchestra.run.smithi114.stdout: "lock": { 2024-04-23T10:32:04.849 INFO:teuthology.orchestra.run.smithi114.stdout: "gather_set": [], 2024-04-23T10:32:04.849 INFO:teuthology.orchestra.run.smithi114.stdout: "state": "lock", 2024-04-23T10:32:04.849 INFO:teuthology.orchestra.run.smithi114.stdout: "type": "iquiesce", 2024-04-23T10:32:04.849 INFO:teuthology.orchestra.run.smithi114.stdout: "is_leased": false, 2024-04-23T10:32:04.849 INFO:teuthology.orchestra.run.smithi114.stdout: "num_rdlocks": 0, 2024-04-23T10:32:04.849 INFO:teuthology.orchestra.run.smithi114.stdout: "num_wrlocks": 0, 2024-04-23T10:32:04.849 INFO:teuthology.orchestra.run.smithi114.stdout: "num_xlocks": 1, 2024-04-23T10:32:04.849 INFO:teuthology.orchestra.run.smithi114.stdout: "xlock_by": { 2024-04-23T10:32:04.849 INFO:teuthology.orchestra.run.smithi114.stdout: "reqid": { 2024-04-23T10:32:04.849 INFO:teuthology.orchestra.run.smithi114.stdout: "entity": { 2024-04-23T10:32:04.849 INFO:teuthology.orchestra.run.smithi114.stdout: "type": "mds", 2024-04-23T10:32:04.849 INFO:teuthology.orchestra.run.smithi114.stdout: "num": 0 2024-04-23T10:32:04.849 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-23T10:32:04.849 INFO:teuthology.orchestra.run.smithi114.stdout: "tid": 177 2024-04-23T10:32:04.850 INFO:teuthology.orchestra.run.smithi114.stdout: } 2024-04-23T10:32:04.850 INFO:teuthology.orchestra.run.smithi114.stdout: } 2024-04-23T10:32:04.850 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-23T10:32:04.850 INFO:teuthology.orchestra.run.smithi114.stdout: "flags": 4, 2024-04-23T10:32:04.850 INFO:teuthology.orchestra.run.smithi114.stdout: "wrlock_target": -1 2024-04-23T10:32:04.850 INFO:teuthology.orchestra.run.smithi114.stdout: } 2024-04-23T10:32:04.850 INFO:teuthology.orchestra.run.smithi114.stdout: ] 2024-04-23T10:32:04.850 INFO:teuthology.orchestra.run.smithi114.stdout: } 2024-04-23T10:32:04.850 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-23T10:32:04.850 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-04-23T10:32:04.851 INFO:teuthology.orchestra.run.smithi114.stdout: "description": "peer_request:mds.2:121", 2024-04-23T10:32:04.851 INFO:teuthology.orchestra.run.smithi114.stdout: "initiated_at": "2024-04-23T10:31:21.463630+0000", 2024-04-23T10:32:04.851 INFO:teuthology.orchestra.run.smithi114.stdout: "age": 42.420758773999999, 2024-04-23T10:32:04.851 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 42.424385663999999, 2024-04-23T10:32:04.851 INFO:teuthology.orchestra.run.smithi114.stdout: "continuous": false, 2024-04-23T10:32:04.851 INFO:teuthology.orchestra.run.smithi114.stdout: "type_data": { 2024-04-23T10:32:04.851 INFO:teuthology.orchestra.run.smithi114.stdout: "result": -2147483648, 2024-04-23T10:32:04.851 INFO:teuthology.orchestra.run.smithi114.stdout: "flag_point": "dispatched", 2024-04-23T10:32:04.851 INFO:teuthology.orchestra.run.smithi114.stdout: "reqid": { 2024-04-23T10:32:04.851 INFO:teuthology.orchestra.run.smithi114.stdout: "entity": { 2024-04-23T10:32:04.851 INFO:teuthology.orchestra.run.smithi114.stdout: "type": "mds", 2024-04-23T10:32:04.851 INFO:teuthology.orchestra.run.smithi114.stdout: "num": 2 2024-04-23T10:32:04.852 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-23T10:32:04.852 INFO:teuthology.orchestra.run.smithi114.stdout: "tid": 121 2024-04-23T10:32:04.852 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-23T10:32:04.852 INFO:teuthology.orchestra.run.smithi114.stdout: "op_type": "peer_request", 2024-04-23T10:32:04.852 INFO:teuthology.orchestra.run.smithi114.stdout: "leader_info": { 2024-04-23T10:32:04.852 INFO:teuthology.orchestra.run.smithi114.stdout: "leader": "2" 2024-04-23T10:32:04.852 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-23T10:32:04.852 INFO:teuthology.orchestra.run.smithi114.stdout: "events": [ 2024-04-23T10:32:04.852 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-04-23T10:32:04.852 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-04-23T10:31:21.463630+0000", 2024-04-23T10:32:04.852 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "initiated" 2024-04-23T10:32:04.852 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-23T10:32:04.853 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-04-23T10:32:04.853 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-04-23T10:31:21.463630+0000", 2024-04-23T10:32:04.853 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "header_read" 2024-04-23T10:32:04.853 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-23T10:32:04.853 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-04-23T10:32:04.853 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-04-23T10:31:21.463631+0000", 2024-04-23T10:32:04.853 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "throttled" 2024-04-23T10:32:04.853 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-23T10:32:04.853 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-04-23T10:32:04.853 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-04-23T10:31:21.463636+0000", 2024-04-23T10:32:04.853 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "all_read" 2024-04-23T10:32:04.854 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-23T10:32:04.854 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-04-23T10:32:04.854 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-04-23T10:31:21.488755+0000", 2024-04-23T10:32:04.854 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "dispatched" 2024-04-23T10:32:04.854 INFO:teuthology.orchestra.run.smithi114.stdout: } 2024-04-23T10:32:04.854 INFO:teuthology.orchestra.run.smithi114.stdout: ], 2024-04-23T10:32:04.854 INFO:teuthology.orchestra.run.smithi114.stdout: "locks": [] 2024-04-23T10:32:04.854 INFO:teuthology.orchestra.run.smithi114.stdout: } 2024-04-23T10:32:04.854 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-23T10:32:04.854 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-04-23T10:32:04.854 INFO:teuthology.orchestra.run.smithi114.stdout: "description": "internal op quiesce_inode:mds.0:156 fp=#0x10000000005", 2024-04-23T10:32:04.855 INFO:teuthology.orchestra.run.smithi114.stdout: "initiated_at": "2024-04-23T10:31:21.451325+0000", 2024-04-23T10:32:04.855 INFO:teuthology.orchestra.run.smithi114.stdout: "age": 42.433063761, 2024-04-23T10:32:04.855 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 42.436722033000002, 2024-04-23T10:32:04.855 INFO:teuthology.orchestra.run.smithi114.stdout: "continuous": true, 2024-04-23T10:32:04.855 INFO:teuthology.orchestra.run.smithi114.stdout: "type_data": { 2024-04-23T10:32:04.855 INFO:teuthology.orchestra.run.smithi114.stdout: "result": -2147483648, 2024-04-23T10:32:04.855 INFO:teuthology.orchestra.run.smithi114.stdout: "flag_point": "quiesce complete", 2024-04-23T10:32:04.855 INFO:teuthology.orchestra.run.smithi114.stdout: "reqid": { 2024-04-23T10:32:04.855 INFO:teuthology.orchestra.run.smithi114.stdout: "entity": { 2024-04-23T10:32:04.855 INFO:teuthology.orchestra.run.smithi114.stdout: "type": "mds", 2024-04-23T10:32:04.855 INFO:teuthology.orchestra.run.smithi114.stdout: "num": 0 2024-04-23T10:32:04.856 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-23T10:32:04.856 INFO:teuthology.orchestra.run.smithi114.stdout: "tid": 156 2024-04-23T10:32:04.856 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-23T10:32:04.856 INFO:teuthology.orchestra.run.smithi114.stdout: "op_type": "internal_op", 2024-04-23T10:32:04.856 INFO:teuthology.orchestra.run.smithi114.stdout: "internal_op": 5385, 2024-04-23T10:32:04.856 INFO:teuthology.orchestra.run.smithi114.stdout: "op_name": "quiesce_inode", 2024-04-23T10:32:04.856 INFO:teuthology.orchestra.run.smithi114.stdout: "events": [ 2024-04-23T10:32:04.856 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-04-23T10:32:04.856 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-04-23T10:31:21.451325+0000", 2024-04-23T10:32:04.856 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "initiated" 2024-04-23T10:32:04.857 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-23T10:32:04.857 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-04-23T10:32:04.857 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-04-23T10:31:21.451325+0000", 2024-04-23T10:32:04.857 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "header_read" 2024-04-23T10:32:04.857 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-23T10:32:04.857 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-04-23T10:32:04.857 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-04-23T10:31:21.451325+0000", 2024-04-23T10:32:04.857 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "throttled" 2024-04-23T10:32:04.857 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-23T10:32:04.857 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-04-23T10:32:04.857 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-04-23T10:31:21.451325+0000", 2024-04-23T10:32:04.857 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "all_read" 2024-04-23T10:32:04.858 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-23T10:32:04.858 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-04-23T10:32:04.858 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-04-23T10:31:21.451325+0000", 2024-04-23T10:32:04.858 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "dispatched" 2024-04-23T10:32:04.858 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-23T10:32:04.858 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-04-23T10:32:04.858 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-04-23T10:31:21.451535+0000", 2024-04-23T10:32:04.858 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "failed to xlock, waiting" 2024-04-23T10:32:04.858 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-23T10:32:04.858 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-04-23T10:32:04.859 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-04-23T10:31:21.452316+0000", 2024-04-23T10:32:04.859 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "acquired locks" 2024-04-23T10:32:04.859 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-23T10:32:04.859 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-04-23T10:32:04.859 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-04-23T10:31:21.509937+0000", 2024-04-23T10:32:04.859 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "quiesce complete" 2024-04-23T10:32:04.859 INFO:teuthology.orchestra.run.smithi114.stdout: } 2024-04-23T10:32:04.859 INFO:teuthology.orchestra.run.smithi114.stdout: ], 2024-04-23T10:32:04.859 INFO:teuthology.orchestra.run.smithi114.stdout: "locks": [ 2024-04-23T10:32:04.859 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-04-23T10:32:04.859 INFO:teuthology.orchestra.run.smithi114.stdout: "object": { 2024-04-23T10:32:04.860 INFO:teuthology.orchestra.run.smithi114.stdout: "is_auth": true, 2024-04-23T10:32:04.860 INFO:teuthology.orchestra.run.smithi114.stdout: "auth_state": { 2024-04-23T10:32:04.860 INFO:teuthology.orchestra.run.smithi114.stdout: "replicas": { 2024-04-23T10:32:04.860 INFO:teuthology.orchestra.run.smithi114.stdout: "1": 1, 2024-04-23T10:32:04.860 INFO:teuthology.orchestra.run.smithi114.stdout: "2": 1 2024-04-23T10:32:04.860 INFO:teuthology.orchestra.run.smithi114.stdout: } 2024-04-23T10:32:04.860 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-23T10:32:04.860 INFO:teuthology.orchestra.run.smithi114.stdout: "replica_state": { 2024-04-23T10:32:04.860 INFO:teuthology.orchestra.run.smithi114.stdout: "authority": [ 2024-04-23T10:32:04.860 INFO:teuthology.orchestra.run.smithi114.stdout: 0, 2024-04-23T10:32:04.861 INFO:teuthology.orchestra.run.smithi114.stdout: -2 2024-04-23T10:32:04.861 INFO:teuthology.orchestra.run.smithi114.stdout: ], 2024-04-23T10:32:04.861 INFO:teuthology.orchestra.run.smithi114.stdout: "replica_nonce": 0 2024-04-23T10:32:04.861 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-23T10:32:04.861 INFO:teuthology.orchestra.run.smithi114.stdout: "auth_pins": 3, 2024-04-23T10:32:04.861 INFO:teuthology.orchestra.run.smithi114.stdout: "is_frozen": false, 2024-04-23T10:32:04.861 INFO:teuthology.orchestra.run.smithi114.stdout: "is_freezing": false, 2024-04-23T10:32:04.861 INFO:teuthology.orchestra.run.smithi114.stdout: "pins": { 2024-04-23T10:32:04.861 INFO:teuthology.orchestra.run.smithi114.stdout: "ptrwaiter": 0, 2024-04-23T10:32:04.861 INFO:teuthology.orchestra.run.smithi114.stdout: "dirtyscattered": 1, 2024-04-23T10:32:04.862 INFO:teuthology.orchestra.run.smithi114.stdout: "request": 1, 2024-04-23T10:32:04.862 INFO:teuthology.orchestra.run.smithi114.stdout: "lock": 1, 2024-04-23T10:32:04.862 INFO:teuthology.orchestra.run.smithi114.stdout: "dirfrag": 1, 2024-04-23T10:32:04.862 INFO:teuthology.orchestra.run.smithi114.stdout: "caps": 1, 2024-04-23T10:32:04.862 INFO:teuthology.orchestra.run.smithi114.stdout: "dirtyrstat": 0, 2024-04-23T10:32:04.862 INFO:teuthology.orchestra.run.smithi114.stdout: "dirtyparent": 0, 2024-04-23T10:32:04.862 INFO:teuthology.orchestra.run.smithi114.stdout: "replicated": 1, 2024-04-23T10:32:04.862 INFO:teuthology.orchestra.run.smithi114.stdout: "dirty": 1, 2024-04-23T10:32:04.862 INFO:teuthology.orchestra.run.smithi114.stdout: "waiter": 0, 2024-04-23T10:32:04.862 INFO:teuthology.orchestra.run.smithi114.stdout: "authpin": 1 2024-04-23T10:32:04.862 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-23T10:32:04.862 INFO:teuthology.orchestra.run.smithi114.stdout: "nref": 8 2024-04-23T10:32:04.862 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-23T10:32:04.863 INFO:teuthology.orchestra.run.smithi114.stdout: "object_string": "[inode 0x10000000005 [...7,head] /volumes/qa/sv_1/b206794e-7622-4ab8-8943-f3ae4e65b7a3/client.0/ auth{1=1,2=1} v423 ap=3 f(v0 m2024-04-23T10:27:42.521612+0000 1=0+1) n(v32 rc2024-04-23T10:31:17.764716+0000 b21075074 77=63+14) old_inodes=5 (inest mix dirty) (iquiesce lock x=1 by request(mds.0:156 nref=3)) caps={15312=pAsLsXs/-@3},l=15312 | ptrwaiter=0 dirtyscattered=1 request=1 lock=1 dirfrag=1 caps=1 dirtyrstat=0 dirtyparent=0 replicated=1 dirty=1 waiter=0 authpin=1 0x5571cf340b00]", 2024-04-23T10:32:04.863 INFO:teuthology.orchestra.run.smithi114.stdout: "lock": { 2024-04-23T10:32:04.863 INFO:teuthology.orchestra.run.smithi114.stdout: "gather_set": [], 2024-04-23T10:32:04.863 INFO:teuthology.orchestra.run.smithi114.stdout: "state": "lock", 2024-04-23T10:32:04.863 INFO:teuthology.orchestra.run.smithi114.stdout: "type": "iquiesce", 2024-04-23T10:32:04.863 INFO:teuthology.orchestra.run.smithi114.stdout: "is_leased": false, 2024-04-23T10:32:04.863 INFO:teuthology.orchestra.run.smithi114.stdout: "num_rdlocks": 0, 2024-04-23T10:32:04.863 INFO:teuthology.orchestra.run.smithi114.stdout: "num_wrlocks": 0, 2024-04-23T10:32:04.863 INFO:teuthology.orchestra.run.smithi114.stdout: "num_xlocks": 1, 2024-04-23T10:32:04.863 INFO:teuthology.orchestra.run.smithi114.stdout: "xlock_by": { 2024-04-23T10:32:04.863 INFO:teuthology.orchestra.run.smithi114.stdout: "reqid": { 2024-04-23T10:32:04.863 INFO:teuthology.orchestra.run.smithi114.stdout: "entity": { 2024-04-23T10:32:04.863 INFO:teuthology.orchestra.run.smithi114.stdout: "type": "mds", 2024-04-23T10:32:04.863 INFO:teuthology.orchestra.run.smithi114.stdout: "num": 0 2024-04-23T10:32:04.863 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-23T10:32:04.863 INFO:teuthology.orchestra.run.smithi114.stdout: "tid": 156 2024-04-23T10:32:04.864 INFO:teuthology.orchestra.run.smithi114.stdout: } 2024-04-23T10:32:04.864 INFO:teuthology.orchestra.run.smithi114.stdout: } 2024-04-23T10:32:04.864 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-23T10:32:04.864 INFO:teuthology.orchestra.run.smithi114.stdout: "flags": 4, 2024-04-23T10:32:04.864 INFO:teuthology.orchestra.run.smithi114.stdout: "wrlock_target": -1 2024-04-23T10:32:04.864 INFO:teuthology.orchestra.run.smithi114.stdout: } 2024-04-23T10:32:04.864 INFO:teuthology.orchestra.run.smithi114.stdout: ] 2024-04-23T10:32:04.864 INFO:teuthology.orchestra.run.smithi114.stdout: } 2024-04-23T10:32:04.864 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-23T10:32:04.864 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-04-23T10:32:04.864 INFO:teuthology.orchestra.run.smithi114.stdout: "description": "internal op quiesce_inode:mds.0:178 fp=#0x10000000223", 2024-04-23T10:32:04.864 INFO:teuthology.orchestra.run.smithi114.stdout: "initiated_at": "2024-04-23T10:31:21.467375+0000", 2024-04-23T10:32:04.864 INFO:teuthology.orchestra.run.smithi114.stdout: "age": 42.417014410999997, 2024-04-23T10:32:04.864 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 42.420749837999999, 2024-04-23T10:32:04.864 INFO:teuthology.orchestra.run.smithi114.stdout: "continuous": true, 2024-04-23T10:32:04.865 INFO:teuthology.orchestra.run.smithi114.stdout: "type_data": { 2024-04-23T10:32:04.865 INFO:teuthology.orchestra.run.smithi114.stdout: "result": -2147483648, 2024-04-23T10:32:04.865 INFO:teuthology.orchestra.run.smithi114.stdout: "flag_point": "quiesce complete", 2024-04-23T10:32:04.865 INFO:teuthology.orchestra.run.smithi114.stdout: "reqid": { 2024-04-23T10:32:04.865 INFO:teuthology.orchestra.run.smithi114.stdout: "entity": { 2024-04-23T10:32:04.865 INFO:teuthology.orchestra.run.smithi114.stdout: "type": "mds", 2024-04-23T10:32:04.865 INFO:teuthology.orchestra.run.smithi114.stdout: "num": 0 2024-04-23T10:32:04.865 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-23T10:32:04.865 INFO:teuthology.orchestra.run.smithi114.stdout: "tid": 178 2024-04-23T10:32:04.865 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-23T10:32:04.865 INFO:teuthology.orchestra.run.smithi114.stdout: "op_type": "internal_op", 2024-04-23T10:32:04.865 INFO:teuthology.orchestra.run.smithi114.stdout: "internal_op": 5385, 2024-04-23T10:32:04.865 INFO:teuthology.orchestra.run.smithi114.stdout: "op_name": "quiesce_inode", 2024-04-23T10:32:04.865 INFO:teuthology.orchestra.run.smithi114.stdout: "events": [ 2024-04-23T10:32:04.865 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-04-23T10:32:04.865 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-04-23T10:31:21.467375+0000", 2024-04-23T10:32:04.865 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "initiated" 2024-04-23T10:32:04.866 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-23T10:32:04.866 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-04-23T10:32:04.866 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-04-23T10:31:21.467375+0000", 2024-04-23T10:32:04.866 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "header_read" 2024-04-23T10:32:04.866 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-23T10:32:04.866 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-04-23T10:32:04.866 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-04-23T10:31:21.467375+0000", 2024-04-23T10:32:04.866 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "throttled" 2024-04-23T10:32:04.866 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-23T10:32:04.866 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-04-23T10:32:04.866 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-04-23T10:31:21.467375+0000", 2024-04-23T10:32:04.866 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "all_read" 2024-04-23T10:32:04.866 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-23T10:32:04.866 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-04-23T10:32:04.866 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-04-23T10:31:21.467375+0000", 2024-04-23T10:32:04.867 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "dispatched" 2024-04-23T10:32:04.867 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-23T10:32:04.867 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-04-23T10:32:04.867 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-04-23T10:31:21.467706+0000", 2024-04-23T10:32:04.867 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "acquired locks" 2024-04-23T10:32:04.867 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-23T10:32:04.867 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-04-23T10:32:04.867 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-04-23T10:31:21.467818+0000", 2024-04-23T10:32:04.867 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "quiesce complete" 2024-04-23T10:32:04.867 INFO:teuthology.orchestra.run.smithi114.stdout: } 2024-04-23T10:32:04.867 INFO:teuthology.orchestra.run.smithi114.stdout: ], 2024-04-23T10:32:04.867 INFO:teuthology.orchestra.run.smithi114.stdout: "locks": [ 2024-04-23T10:32:04.867 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-04-23T10:32:04.867 INFO:teuthology.orchestra.run.smithi114.stdout: "object": { 2024-04-23T10:32:04.867 INFO:teuthology.orchestra.run.smithi114.stdout: "is_auth": true, 2024-04-23T10:32:04.868 INFO:teuthology.orchestra.run.smithi114.stdout: "auth_state": { 2024-04-23T10:32:04.868 INFO:teuthology.orchestra.run.smithi114.stdout: "replicas": {} 2024-04-23T10:32:04.868 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-23T10:32:04.868 INFO:teuthology.orchestra.run.smithi114.stdout: "replica_state": { 2024-04-23T10:32:04.868 INFO:teuthology.orchestra.run.smithi114.stdout: "authority": [ 2024-04-23T10:32:04.868 INFO:teuthology.orchestra.run.smithi114.stdout: 0, 2024-04-23T10:32:04.868 INFO:teuthology.orchestra.run.smithi114.stdout: -2 2024-04-23T10:32:04.868 INFO:teuthology.orchestra.run.smithi114.stdout: ], 2024-04-23T10:32:04.868 INFO:teuthology.orchestra.run.smithi114.stdout: "replica_nonce": 0 2024-04-23T10:32:04.868 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-23T10:32:04.868 INFO:teuthology.orchestra.run.smithi114.stdout: "auth_pins": 1, 2024-04-23T10:32:04.868 INFO:teuthology.orchestra.run.smithi114.stdout: "is_frozen": false, 2024-04-23T10:32:04.868 INFO:teuthology.orchestra.run.smithi114.stdout: "is_freezing": false, 2024-04-23T10:32:04.868 INFO:teuthology.orchestra.run.smithi114.stdout: "pins": { 2024-04-23T10:32:04.869 INFO:teuthology.orchestra.run.smithi114.stdout: "ptrwaiter": 0, 2024-04-23T10:32:04.869 INFO:teuthology.orchestra.run.smithi114.stdout: "request": 1, 2024-04-23T10:32:04.869 INFO:teuthology.orchestra.run.smithi114.stdout: "lock": 1, 2024-04-23T10:32:04.869 INFO:teuthology.orchestra.run.smithi114.stdout: "caps": 0, 2024-04-23T10:32:04.869 INFO:teuthology.orchestra.run.smithi114.stdout: "needsnapflush": 0, 2024-04-23T10:32:04.869 INFO:teuthology.orchestra.run.smithi114.stdout: "dirtyparent": 0, 2024-04-23T10:32:04.869 INFO:teuthology.orchestra.run.smithi114.stdout: "dirty": 0, 2024-04-23T10:32:04.869 INFO:teuthology.orchestra.run.smithi114.stdout: "waiter": 0, 2024-04-23T10:32:04.869 INFO:teuthology.orchestra.run.smithi114.stdout: "authpin": 1 2024-04-23T10:32:04.869 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-23T10:32:04.869 INFO:teuthology.orchestra.run.smithi114.stdout: "nref": 3 2024-04-23T10:32:04.869 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-23T10:32:04.869 INFO:teuthology.orchestra.run.smithi114.stdout: "object_string": "[inode 0x10000000223 [7,head] /volumes/qa/sv_1/b206794e-7622-4ab8-8943-f3ae4e65b7a3/client.0/tmp/clients/client0/~dmtmp/WORDPRO/RESULTS.XLS auth v8108 ap=1 s=260096 n(v0 rc2024-04-23T10:27:52.063395+0000 b260096 1=1+0) (iquiesce lock x=1 by request(mds.0:178 nref=3)) | ptrwaiter=0 request=1 lock=1 caps=0 needsnapflush=0 dirtyparent=0 dirty=0 waiter=0 authpin=1 0x5571cf586580]", 2024-04-23T10:32:04.869 INFO:teuthology.orchestra.run.smithi114.stdout: "lock": { 2024-04-23T10:32:04.869 INFO:teuthology.orchestra.run.smithi114.stdout: "gather_set": [], 2024-04-23T10:32:04.870 INFO:teuthology.orchestra.run.smithi114.stdout: "state": "lock", 2024-04-23T10:32:04.870 INFO:teuthology.orchestra.run.smithi114.stdout: "type": "iquiesce", 2024-04-23T10:32:04.870 INFO:teuthology.orchestra.run.smithi114.stdout: "is_leased": false, 2024-04-23T10:32:04.870 INFO:teuthology.orchestra.run.smithi114.stdout: "num_rdlocks": 0, 2024-04-23T10:32:04.870 INFO:teuthology.orchestra.run.smithi114.stdout: "num_wrlocks": 0, 2024-04-23T10:32:04.870 INFO:teuthology.orchestra.run.smithi114.stdout: "num_xlocks": 1, 2024-04-23T10:32:04.870 INFO:teuthology.orchestra.run.smithi114.stdout: "xlock_by": { 2024-04-23T10:32:04.870 INFO:teuthology.orchestra.run.smithi114.stdout: "reqid": { 2024-04-23T10:32:04.870 INFO:teuthology.orchestra.run.smithi114.stdout: "entity": { 2024-04-23T10:32:04.870 INFO:teuthology.orchestra.run.smithi114.stdout: "type": "mds", 2024-04-23T10:32:04.870 INFO:teuthology.orchestra.run.smithi114.stdout: "num": 0 2024-04-23T10:32:04.870 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-23T10:32:04.870 INFO:teuthology.orchestra.run.smithi114.stdout: "tid": 178 2024-04-23T10:32:04.870 INFO:teuthology.orchestra.run.smithi114.stdout: } 2024-04-23T10:32:04.870 INFO:teuthology.orchestra.run.smithi114.stdout: } 2024-04-23T10:32:04.871 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-23T10:32:04.871 INFO:teuthology.orchestra.run.smithi114.stdout: "flags": 4, 2024-04-23T10:32:04.871 INFO:teuthology.orchestra.run.smithi114.stdout: "wrlock_target": -1 2024-04-23T10:32:04.871 INFO:teuthology.orchestra.run.smithi114.stdout: } 2024-04-23T10:32:04.871 INFO:teuthology.orchestra.run.smithi114.stdout: ] 2024-04-23T10:32:04.871 INFO:teuthology.orchestra.run.smithi114.stdout: } 2024-04-23T10:32:04.871 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-23T10:32:04.871 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-04-23T10:32:04.871 INFO:teuthology.orchestra.run.smithi114.stdout: "description": "peer_request:mds.1:117", 2024-04-23T10:32:04.871 INFO:teuthology.orchestra.run.smithi114.stdout: "initiated_at": "2024-04-23T10:31:21.464907+0000", 2024-04-23T10:32:04.871 INFO:teuthology.orchestra.run.smithi114.stdout: "age": 42.419482221000003, 2024-04-23T10:32:04.871 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 42.423281594999999, 2024-04-23T10:32:04.871 INFO:teuthology.orchestra.run.smithi114.stdout: "continuous": false, 2024-04-23T10:32:04.871 INFO:teuthology.orchestra.run.smithi114.stdout: "type_data": { 2024-04-23T10:32:04.871 INFO:teuthology.orchestra.run.smithi114.stdout: "result": -2147483648, 2024-04-23T10:32:04.872 INFO:teuthology.orchestra.run.smithi114.stdout: "flag_point": "dispatched", 2024-04-23T10:32:04.872 INFO:teuthology.orchestra.run.smithi114.stdout: "reqid": { 2024-04-23T10:32:04.872 INFO:teuthology.orchestra.run.smithi114.stdout: "entity": { 2024-04-23T10:32:04.872 INFO:teuthology.orchestra.run.smithi114.stdout: "type": "mds", 2024-04-23T10:32:04.872 INFO:teuthology.orchestra.run.smithi114.stdout: "num": 1 2024-04-23T10:32:04.872 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-23T10:32:04.872 INFO:teuthology.orchestra.run.smithi114.stdout: "tid": 117 2024-04-23T10:32:04.872 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-23T10:32:04.872 INFO:teuthology.orchestra.run.smithi114.stdout: "op_type": "peer_request", 2024-04-23T10:32:04.872 INFO:teuthology.orchestra.run.smithi114.stdout: "leader_info": { 2024-04-23T10:32:04.872 INFO:teuthology.orchestra.run.smithi114.stdout: "leader": "1" 2024-04-23T10:32:04.872 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-23T10:32:04.872 INFO:teuthology.orchestra.run.smithi114.stdout: "events": [ 2024-04-23T10:32:04.872 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-04-23T10:32:04.872 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-04-23T10:31:21.464907+0000", 2024-04-23T10:32:04.873 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "initiated" 2024-04-23T10:32:04.873 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-23T10:32:04.873 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-04-23T10:32:04.873 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-04-23T10:31:21.464907+0000", 2024-04-23T10:32:04.873 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "header_read" 2024-04-23T10:32:04.873 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-23T10:32:04.873 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-04-23T10:32:04.873 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-04-23T10:31:21.464907+0000", 2024-04-23T10:32:04.873 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "throttled" 2024-04-23T10:32:04.873 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-23T10:32:04.873 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-04-23T10:32:04.873 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-04-23T10:31:21.464912+0000", 2024-04-23T10:32:04.873 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "all_read" 2024-04-23T10:32:04.873 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-23T10:32:04.873 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-04-23T10:32:04.873 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-04-23T10:31:21.492009+0000", 2024-04-23T10:32:04.874 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "dispatched" 2024-04-23T10:32:04.874 INFO:teuthology.orchestra.run.smithi114.stdout: } 2024-04-23T10:32:04.874 INFO:teuthology.orchestra.run.smithi114.stdout: ], 2024-04-23T10:32:04.874 INFO:teuthology.orchestra.run.smithi114.stdout: "locks": [] 2024-04-23T10:32:04.874 INFO:teuthology.orchestra.run.smithi114.stdout: } 2024-04-23T10:32:04.874 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-23T10:32:04.874 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-04-23T10:32:04.874 INFO:teuthology.orchestra.run.smithi114.stdout: "description": "peer_request:mds.2:112", 2024-04-23T10:32:04.874 INFO:teuthology.orchestra.run.smithi114.stdout: "initiated_at": "2024-04-23T10:31:21.450958+0000", 2024-04-23T10:32:04.874 INFO:teuthology.orchestra.run.smithi114.stdout: "age": 42.433431548999998, 2024-04-23T10:32:04.874 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 42.437252534999999, 2024-04-23T10:32:04.874 INFO:teuthology.orchestra.run.smithi114.stdout: "continuous": false, 2024-04-23T10:32:04.874 INFO:teuthology.orchestra.run.smithi114.stdout: "type_data": { 2024-04-23T10:32:04.874 INFO:teuthology.orchestra.run.smithi114.stdout: "result": -2147483648, 2024-04-23T10:32:04.874 INFO:teuthology.orchestra.run.smithi114.stdout: "flag_point": "dispatched", 2024-04-23T10:32:04.874 INFO:teuthology.orchestra.run.smithi114.stdout: "reqid": { 2024-04-23T10:32:04.875 INFO:teuthology.orchestra.run.smithi114.stdout: "entity": { 2024-04-23T10:32:04.875 INFO:teuthology.orchestra.run.smithi114.stdout: "type": "mds", 2024-04-23T10:32:04.875 INFO:teuthology.orchestra.run.smithi114.stdout: "num": 2 2024-04-23T10:32:04.875 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-23T10:32:04.875 INFO:teuthology.orchestra.run.smithi114.stdout: "tid": 112 2024-04-23T10:32:04.875 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-23T10:32:04.875 INFO:teuthology.orchestra.run.smithi114.stdout: "op_type": "peer_request", 2024-04-23T10:32:04.875 INFO:teuthology.orchestra.run.smithi114.stdout: "leader_info": { 2024-04-23T10:32:04.875 INFO:teuthology.orchestra.run.smithi114.stdout: "leader": "2" 2024-04-23T10:32:04.875 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-23T10:32:04.875 INFO:teuthology.orchestra.run.smithi114.stdout: "events": [ 2024-04-23T10:32:04.875 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-04-23T10:32:04.875 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-04-23T10:31:21.450958+0000", 2024-04-23T10:32:04.875 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "initiated" 2024-04-23T10:32:04.875 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-23T10:32:04.875 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-04-23T10:32:04.876 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-04-23T10:31:21.450958+0000", 2024-04-23T10:32:04.876 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "header_read" 2024-04-23T10:32:04.876 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-23T10:32:04.876 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-04-23T10:32:04.876 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-04-23T10:31:21.450958+0000", 2024-04-23T10:32:04.876 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "throttled" 2024-04-23T10:32:04.876 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-23T10:32:04.876 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-04-23T10:32:04.876 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-04-23T10:31:21.450965+0000", 2024-04-23T10:32:04.876 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "all_read" 2024-04-23T10:32:04.876 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-23T10:32:04.876 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-04-23T10:32:04.876 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-04-23T10:31:21.451700+0000", 2024-04-23T10:32:04.876 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "dispatched" 2024-04-23T10:32:04.876 INFO:teuthology.orchestra.run.smithi114.stdout: } 2024-04-23T10:32:04.876 INFO:teuthology.orchestra.run.smithi114.stdout: ], 2024-04-23T10:32:04.876 INFO:teuthology.orchestra.run.smithi114.stdout: "locks": [] 2024-04-23T10:32:04.877 INFO:teuthology.orchestra.run.smithi114.stdout: } 2024-04-23T10:32:04.877 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-23T10:32:04.877 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-04-23T10:32:04.877 INFO:teuthology.orchestra.run.smithi114.stdout: "description": "internal op quiesce_inode:mds.0:179 fp=#0x10000000224", 2024-04-23T10:32:04.877 INFO:teuthology.orchestra.run.smithi114.stdout: "initiated_at": "2024-04-23T10:31:21.467394+0000", 2024-04-23T10:32:04.877 INFO:teuthology.orchestra.run.smithi114.stdout: "age": 42.416995122000003, 2024-04-23T10:32:04.877 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 42.420840339000002, 2024-04-23T10:32:04.877 INFO:teuthology.orchestra.run.smithi114.stdout: "continuous": true, 2024-04-23T10:32:04.877 INFO:teuthology.orchestra.run.smithi114.stdout: "type_data": { 2024-04-23T10:32:04.877 INFO:teuthology.orchestra.run.smithi114.stdout: "result": -2147483648, 2024-04-23T10:32:04.877 INFO:teuthology.orchestra.run.smithi114.stdout: "flag_point": "quiesce complete", 2024-04-23T10:32:04.877 INFO:teuthology.orchestra.run.smithi114.stdout: "reqid": { 2024-04-23T10:32:04.877 INFO:teuthology.orchestra.run.smithi114.stdout: "entity": { 2024-04-23T10:32:04.877 INFO:teuthology.orchestra.run.smithi114.stdout: "type": "mds", 2024-04-23T10:32:04.877 INFO:teuthology.orchestra.run.smithi114.stdout: "num": 0 2024-04-23T10:32:04.877 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-23T10:32:04.878 INFO:teuthology.orchestra.run.smithi114.stdout: "tid": 179 2024-04-23T10:32:04.878 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-23T10:32:04.878 INFO:teuthology.orchestra.run.smithi114.stdout: "op_type": "internal_op", 2024-04-23T10:32:04.878 INFO:teuthology.orchestra.run.smithi114.stdout: "internal_op": 5385, 2024-04-23T10:32:04.878 INFO:teuthology.orchestra.run.smithi114.stdout: "op_name": "quiesce_inode", 2024-04-23T10:32:04.878 INFO:teuthology.orchestra.run.smithi114.stdout: "events": [ 2024-04-23T10:32:04.878 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-04-23T10:32:04.878 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-04-23T10:31:21.467394+0000", 2024-04-23T10:32:04.878 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "initiated" 2024-04-23T10:32:04.878 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-23T10:32:04.878 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-04-23T10:32:04.878 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-04-23T10:31:21.467394+0000", 2024-04-23T10:32:04.878 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "header_read" 2024-04-23T10:32:04.878 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-23T10:32:04.879 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-04-23T10:32:04.879 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-04-23T10:31:21.467394+0000", 2024-04-23T10:32:04.879 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "throttled" 2024-04-23T10:32:04.879 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-23T10:32:04.879 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-04-23T10:32:04.879 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-04-23T10:31:21.467394+0000", 2024-04-23T10:32:04.879 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "all_read" 2024-04-23T10:32:04.879 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-23T10:32:04.879 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-04-23T10:32:04.879 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-04-23T10:31:21.467394+0000", 2024-04-23T10:32:04.879 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "dispatched" 2024-04-23T10:32:04.879 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-23T10:32:04.879 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-04-23T10:32:04.879 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-04-23T10:31:21.467972+0000", 2024-04-23T10:32:04.879 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "failed to xlock, waiting" 2024-04-23T10:32:04.880 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-23T10:32:04.880 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-04-23T10:32:04.880 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-04-23T10:31:21.479506+0000", 2024-04-23T10:32:04.880 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "acquired locks" 2024-04-23T10:32:04.880 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-23T10:32:04.880 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-04-23T10:32:04.880 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-04-23T10:31:21.479719+0000", 2024-04-23T10:32:04.880 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "quiesce complete" 2024-04-23T10:32:04.880 INFO:teuthology.orchestra.run.smithi114.stdout: } 2024-04-23T10:32:04.880 INFO:teuthology.orchestra.run.smithi114.stdout: ], 2024-04-23T10:32:04.880 INFO:teuthology.orchestra.run.smithi114.stdout: "locks": [ 2024-04-23T10:32:04.880 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-04-23T10:32:04.880 INFO:teuthology.orchestra.run.smithi114.stdout: "object": { 2024-04-23T10:32:04.881 INFO:teuthology.orchestra.run.smithi114.stdout: "is_auth": true, 2024-04-23T10:32:04.881 INFO:teuthology.orchestra.run.smithi114.stdout: "auth_state": { 2024-04-23T10:32:04.881 INFO:teuthology.orchestra.run.smithi114.stdout: "replicas": {} 2024-04-23T10:32:04.881 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-23T10:32:04.881 INFO:teuthology.orchestra.run.smithi114.stdout: "replica_state": { 2024-04-23T10:32:04.881 INFO:teuthology.orchestra.run.smithi114.stdout: "authority": [ 2024-04-23T10:32:04.881 INFO:teuthology.orchestra.run.smithi114.stdout: 0, 2024-04-23T10:32:04.881 INFO:teuthology.orchestra.run.smithi114.stdout: -2 2024-04-23T10:32:04.881 INFO:teuthology.orchestra.run.smithi114.stdout: ], 2024-04-23T10:32:04.881 INFO:teuthology.orchestra.run.smithi114.stdout: "replica_nonce": 0 2024-04-23T10:32:04.882 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-23T10:32:04.882 INFO:teuthology.orchestra.run.smithi114.stdout: "auth_pins": 1, 2024-04-23T10:32:04.882 INFO:teuthology.orchestra.run.smithi114.stdout: "is_frozen": false, 2024-04-23T10:32:04.882 INFO:teuthology.orchestra.run.smithi114.stdout: "is_freezing": false, 2024-04-23T10:32:04.882 INFO:teuthology.orchestra.run.smithi114.stdout: "pins": { 2024-04-23T10:32:04.882 INFO:teuthology.orchestra.run.smithi114.stdout: "ptrwaiter": 0, 2024-04-23T10:32:04.882 INFO:teuthology.orchestra.run.smithi114.stdout: "request": 1, 2024-04-23T10:32:04.882 INFO:teuthology.orchestra.run.smithi114.stdout: "lock": 1, 2024-04-23T10:32:04.882 INFO:teuthology.orchestra.run.smithi114.stdout: "caps": 0, 2024-04-23T10:32:04.882 INFO:teuthology.orchestra.run.smithi114.stdout: "needsnapflush": 0, 2024-04-23T10:32:04.882 INFO:teuthology.orchestra.run.smithi114.stdout: "dirtyparent": 0, 2024-04-23T10:32:04.882 INFO:teuthology.orchestra.run.smithi114.stdout: "dirty": 0, 2024-04-23T10:32:04.882 INFO:teuthology.orchestra.run.smithi114.stdout: "waiter": 0, 2024-04-23T10:32:04.882 INFO:teuthology.orchestra.run.smithi114.stdout: "authpin": 1 2024-04-23T10:32:04.883 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-23T10:32:04.883 INFO:teuthology.orchestra.run.smithi114.stdout: "nref": 3 2024-04-23T10:32:04.883 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-23T10:32:04.883 INFO:teuthology.orchestra.run.smithi114.stdout: "object_string": "[inode 0x10000000224 [7,head] /volumes/qa/sv_1/b206794e-7622-4ab8-8943-f3ae4e65b7a3/client.0/tmp/clients/client0/~dmtmp/WORDPRO/NEWS1_1.LWP auth v8112 ap=1 s=260096 n(v0 rc2024-04-23T10:27:52.068395+0000 b260096 1=1+0) (iquiesce lock x=1 by request(mds.0:179 nref=3)) | ptrwaiter=0 request=1 lock=1 caps=0 needsnapflush=0 dirtyparent=0 dirty=0 waiter=0 authpin=1 0x5571cf586b00]", 2024-04-23T10:32:04.883 INFO:teuthology.orchestra.run.smithi114.stdout: "lock": { 2024-04-23T10:32:04.883 INFO:teuthology.orchestra.run.smithi114.stdout: "gather_set": [], 2024-04-23T10:32:04.883 INFO:teuthology.orchestra.run.smithi114.stdout: "state": "lock", 2024-04-23T10:32:04.883 INFO:teuthology.orchestra.run.smithi114.stdout: "type": "iquiesce", 2024-04-23T10:32:04.883 INFO:teuthology.orchestra.run.smithi114.stdout: "is_leased": false, 2024-04-23T10:32:04.883 INFO:teuthology.orchestra.run.smithi114.stdout: "num_rdlocks": 0, 2024-04-23T10:32:04.884 INFO:teuthology.orchestra.run.smithi114.stdout: "num_wrlocks": 0, 2024-04-23T10:32:04.884 INFO:teuthology.orchestra.run.smithi114.stdout: "num_xlocks": 1, 2024-04-23T10:32:04.884 INFO:teuthology.orchestra.run.smithi114.stdout: "xlock_by": { 2024-04-23T10:32:04.884 INFO:teuthology.orchestra.run.smithi114.stdout: "reqid": { 2024-04-23T10:32:04.884 INFO:teuthology.orchestra.run.smithi114.stdout: "entity": { 2024-04-23T10:32:04.884 INFO:teuthology.orchestra.run.smithi114.stdout: "type": "mds", 2024-04-23T10:32:04.884 INFO:teuthology.orchestra.run.smithi114.stdout: "num": 0 2024-04-23T10:32:04.884 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-23T10:32:04.884 INFO:teuthology.orchestra.run.smithi114.stdout: "tid": 179 2024-04-23T10:32:04.884 INFO:teuthology.orchestra.run.smithi114.stdout: } 2024-04-23T10:32:04.884 INFO:teuthology.orchestra.run.smithi114.stdout: } 2024-04-23T10:32:04.884 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-23T10:32:04.884 INFO:teuthology.orchestra.run.smithi114.stdout: "flags": 4, 2024-04-23T10:32:04.884 INFO:teuthology.orchestra.run.smithi114.stdout: "wrlock_target": -1 2024-04-23T10:32:04.885 INFO:teuthology.orchestra.run.smithi114.stdout: } 2024-04-23T10:32:04.885 INFO:teuthology.orchestra.run.smithi114.stdout: ] 2024-04-23T10:32:04.885 INFO:teuthology.orchestra.run.smithi114.stdout: } 2024-04-23T10:32:04.885 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-23T10:32:04.885 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-04-23T10:32:04.885 INFO:teuthology.orchestra.run.smithi114.stdout: "description": "peer_request:mds.2:122", 2024-04-23T10:32:04.885 INFO:teuthology.orchestra.run.smithi114.stdout: "initiated_at": "2024-04-23T10:31:21.463703+0000", 2024-04-23T10:32:04.885 INFO:teuthology.orchestra.run.smithi114.stdout: "age": 42.420686615999998, 2024-04-23T10:32:04.885 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 42.424595052999997, 2024-04-23T10:32:04.885 INFO:teuthology.orchestra.run.smithi114.stdout: "continuous": false, 2024-04-23T10:32:04.885 INFO:teuthology.orchestra.run.smithi114.stdout: "type_data": { 2024-04-23T10:32:04.886 INFO:teuthology.orchestra.run.smithi114.stdout: "result": -2147483648, 2024-04-23T10:32:04.886 INFO:teuthology.orchestra.run.smithi114.stdout: "flag_point": "dispatched", 2024-04-23T10:32:04.886 INFO:teuthology.orchestra.run.smithi114.stdout: "reqid": { 2024-04-23T10:32:04.886 INFO:teuthology.orchestra.run.smithi114.stdout: "entity": { 2024-04-23T10:32:04.886 INFO:teuthology.orchestra.run.smithi114.stdout: "type": "mds", 2024-04-23T10:32:04.886 INFO:teuthology.orchestra.run.smithi114.stdout: "num": 2 2024-04-23T10:32:04.886 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-23T10:32:04.886 INFO:teuthology.orchestra.run.smithi114.stdout: "tid": 122 2024-04-23T10:32:04.886 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-23T10:32:04.886 INFO:teuthology.orchestra.run.smithi114.stdout: "op_type": "peer_request", 2024-04-23T10:32:04.886 INFO:teuthology.orchestra.run.smithi114.stdout: "leader_info": { 2024-04-23T10:32:04.886 INFO:teuthology.orchestra.run.smithi114.stdout: "leader": "2" 2024-04-23T10:32:04.886 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-23T10:32:04.886 INFO:teuthology.orchestra.run.smithi114.stdout: "events": [ 2024-04-23T10:32:04.887 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-04-23T10:32:04.887 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-04-23T10:31:21.463703+0000", 2024-04-23T10:32:04.887 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "initiated" 2024-04-23T10:32:04.887 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-23T10:32:04.887 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-04-23T10:32:04.887 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-04-23T10:31:21.463703+0000", 2024-04-23T10:32:04.887 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "header_read" 2024-04-23T10:32:04.887 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-23T10:32:04.887 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-04-23T10:32:04.887 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-04-23T10:31:21.463703+0000", 2024-04-23T10:32:04.887 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "throttled" 2024-04-23T10:32:04.887 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-23T10:32:04.887 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-04-23T10:32:04.887 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-04-23T10:31:21.463708+0000", 2024-04-23T10:32:04.888 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "all_read" 2024-04-23T10:32:04.888 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-23T10:32:04.888 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-04-23T10:32:04.888 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-04-23T10:31:21.492926+0000", 2024-04-23T10:32:04.888 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "dispatched" 2024-04-23T10:32:04.888 INFO:teuthology.orchestra.run.smithi114.stdout: } 2024-04-23T10:32:04.888 INFO:teuthology.orchestra.run.smithi114.stdout: ], 2024-04-23T10:32:04.888 INFO:teuthology.orchestra.run.smithi114.stdout: "locks": [] 2024-04-23T10:32:04.888 INFO:teuthology.orchestra.run.smithi114.stdout: } 2024-04-23T10:32:04.888 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-23T10:32:04.888 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-04-23T10:32:04.888 INFO:teuthology.orchestra.run.smithi114.stdout: "description": "peer_request:mds.1:108", 2024-04-23T10:32:04.888 INFO:teuthology.orchestra.run.smithi114.stdout: "initiated_at": "2024-04-23T10:31:21.450989+0000", 2024-04-23T10:32:04.889 INFO:teuthology.orchestra.run.smithi114.stdout: "age": 42.433400542000001, 2024-04-23T10:32:04.889 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 42.437330357, 2024-04-23T10:32:04.889 INFO:teuthology.orchestra.run.smithi114.stdout: "continuous": false, 2024-04-23T10:32:04.889 INFO:teuthology.orchestra.run.smithi114.stdout: "type_data": { 2024-04-23T10:32:04.889 INFO:teuthology.orchestra.run.smithi114.stdout: "result": -2147483648, 2024-04-23T10:32:04.889 INFO:teuthology.orchestra.run.smithi114.stdout: "flag_point": "dispatched", 2024-04-23T10:32:04.889 INFO:teuthology.orchestra.run.smithi114.stdout: "reqid": { 2024-04-23T10:32:04.889 INFO:teuthology.orchestra.run.smithi114.stdout: "entity": { 2024-04-23T10:32:04.889 INFO:teuthology.orchestra.run.smithi114.stdout: "type": "mds", 2024-04-23T10:32:04.889 INFO:teuthology.orchestra.run.smithi114.stdout: "num": 1 2024-04-23T10:32:04.889 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-23T10:32:04.890 INFO:teuthology.orchestra.run.smithi114.stdout: "tid": 108 2024-04-23T10:32:04.890 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-23T10:32:04.890 INFO:teuthology.orchestra.run.smithi114.stdout: "op_type": "peer_request", 2024-04-23T10:32:04.890 INFO:teuthology.orchestra.run.smithi114.stdout: "leader_info": { 2024-04-23T10:32:04.890 INFO:teuthology.orchestra.run.smithi114.stdout: "leader": "1" 2024-04-23T10:32:04.890 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-23T10:32:04.890 INFO:teuthology.orchestra.run.smithi114.stdout: "events": [ 2024-04-23T10:32:04.890 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-04-23T10:32:04.890 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-04-23T10:31:21.450989+0000", 2024-04-23T10:32:04.890 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "initiated" 2024-04-23T10:32:04.890 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-23T10:32:04.891 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-04-23T10:32:04.891 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-04-23T10:31:21.450989+0000", 2024-04-23T10:32:04.891 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "header_read" 2024-04-23T10:32:04.891 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-23T10:32:04.891 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-04-23T10:32:04.891 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-04-23T10:31:21.450990+0000", 2024-04-23T10:32:04.891 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "throttled" 2024-04-23T10:32:04.891 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-23T10:32:04.891 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-04-23T10:32:04.891 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-04-23T10:31:21.450998+0000", 2024-04-23T10:32:04.892 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "all_read" 2024-04-23T10:32:04.892 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-23T10:32:04.892 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-04-23T10:32:04.892 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-04-23T10:31:21.451852+0000", 2024-04-23T10:32:04.892 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "dispatched" 2024-04-23T10:32:04.892 INFO:teuthology.orchestra.run.smithi114.stdout: } 2024-04-23T10:32:04.892 INFO:teuthology.orchestra.run.smithi114.stdout: ], 2024-04-23T10:32:04.892 INFO:teuthology.orchestra.run.smithi114.stdout: "locks": [] 2024-04-23T10:32:04.892 INFO:teuthology.orchestra.run.smithi114.stdout: } 2024-04-23T10:32:04.892 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-23T10:32:04.892 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-04-23T10:32:04.893 INFO:teuthology.orchestra.run.smithi114.stdout: "description": "internal op quiesce_inode:mds.0:180 fp=#0x1000000020a", 2024-04-23T10:32:04.893 INFO:teuthology.orchestra.run.smithi114.stdout: "initiated_at": "2024-04-23T10:31:21.468566+0000", 2024-04-23T10:32:04.893 INFO:teuthology.orchestra.run.smithi114.stdout: "age": 42.415823480999997, 2024-04-23T10:32:04.893 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 42.419776714999998, 2024-04-23T10:32:04.893 INFO:teuthology.orchestra.run.smithi114.stdout: "continuous": true, 2024-04-23T10:32:04.893 INFO:teuthology.orchestra.run.smithi114.stdout: "type_data": { 2024-04-23T10:32:04.893 INFO:teuthology.orchestra.run.smithi114.stdout: "result": -2147483648, 2024-04-23T10:32:04.893 INFO:teuthology.orchestra.run.smithi114.stdout: "flag_point": "quiesce complete", 2024-04-23T10:32:04.893 INFO:teuthology.orchestra.run.smithi114.stdout: "reqid": { 2024-04-23T10:32:04.893 INFO:teuthology.orchestra.run.smithi114.stdout: "entity": { 2024-04-23T10:32:04.893 INFO:teuthology.orchestra.run.smithi114.stdout: "type": "mds", 2024-04-23T10:32:04.893 INFO:teuthology.orchestra.run.smithi114.stdout: "num": 0 2024-04-23T10:32:04.894 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-23T10:32:04.894 INFO:teuthology.orchestra.run.smithi114.stdout: "tid": 180 2024-04-23T10:32:04.894 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-23T10:32:04.894 INFO:teuthology.orchestra.run.smithi114.stdout: "op_type": "internal_op", 2024-04-23T10:32:04.894 INFO:teuthology.orchestra.run.smithi114.stdout: "internal_op": 5385, 2024-04-23T10:32:04.894 INFO:teuthology.orchestra.run.smithi114.stdout: "op_name": "quiesce_inode", 2024-04-23T10:32:04.894 INFO:teuthology.orchestra.run.smithi114.stdout: "events": [ 2024-04-23T10:32:04.894 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-04-23T10:32:04.894 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-04-23T10:31:21.468566+0000", 2024-04-23T10:32:04.895 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "initiated" 2024-04-23T10:32:04.895 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-23T10:32:04.895 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-04-23T10:32:04.895 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-04-23T10:31:21.468566+0000", 2024-04-23T10:32:04.895 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "header_read" 2024-04-23T10:32:04.895 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-23T10:32:04.895 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-04-23T10:32:04.895 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-04-23T10:31:21.468566+0000", 2024-04-23T10:32:04.895 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "throttled" 2024-04-23T10:32:04.895 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-23T10:32:04.895 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-04-23T10:32:04.896 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-04-23T10:31:21.468566+0000", 2024-04-23T10:32:04.896 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "all_read" 2024-04-23T10:32:04.896 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-23T10:32:04.896 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-04-23T10:32:04.896 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-04-23T10:31:21.468566+0000", 2024-04-23T10:32:04.896 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "dispatched" 2024-04-23T10:32:04.896 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-23T10:32:04.896 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-04-23T10:32:04.896 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-04-23T10:31:21.468784+0000", 2024-04-23T10:32:04.896 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "failed to xlock, waiting" 2024-04-23T10:32:04.896 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-23T10:32:04.896 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-04-23T10:32:04.896 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-04-23T10:31:21.481424+0000", 2024-04-23T10:32:04.896 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "failed to xlock, waiting" 2024-04-23T10:32:04.896 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-23T10:32:04.897 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-04-23T10:32:04.897 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-04-23T10:31:21.490419+0000", 2024-04-23T10:32:04.897 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "acquired locks" 2024-04-23T10:32:04.897 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-23T10:32:04.897 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-04-23T10:32:04.897 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-04-23T10:31:21.490640+0000", 2024-04-23T10:32:04.897 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "quiesce complete" 2024-04-23T10:32:04.897 INFO:teuthology.orchestra.run.smithi114.stdout: } 2024-04-23T10:32:04.897 INFO:teuthology.orchestra.run.smithi114.stdout: ], 2024-04-23T10:32:04.897 INFO:teuthology.orchestra.run.smithi114.stdout: "locks": [ 2024-04-23T10:32:04.897 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-04-23T10:32:04.897 INFO:teuthology.orchestra.run.smithi114.stdout: "object": { 2024-04-23T10:32:04.897 INFO:teuthology.orchestra.run.smithi114.stdout: "is_auth": true, 2024-04-23T10:32:04.897 INFO:teuthology.orchestra.run.smithi114.stdout: "auth_state": { 2024-04-23T10:32:04.897 INFO:teuthology.orchestra.run.smithi114.stdout: "replicas": {} 2024-04-23T10:32:04.898 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-23T10:32:04.898 INFO:teuthology.orchestra.run.smithi114.stdout: "replica_state": { 2024-04-23T10:32:04.898 INFO:teuthology.orchestra.run.smithi114.stdout: "authority": [ 2024-04-23T10:32:04.898 INFO:teuthology.orchestra.run.smithi114.stdout: 0, 2024-04-23T10:32:04.898 INFO:teuthology.orchestra.run.smithi114.stdout: -2 2024-04-23T10:32:04.898 INFO:teuthology.orchestra.run.smithi114.stdout: ], 2024-04-23T10:32:04.898 INFO:teuthology.orchestra.run.smithi114.stdout: "replica_nonce": 0 2024-04-23T10:32:04.898 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-23T10:32:04.898 INFO:teuthology.orchestra.run.smithi114.stdout: "auth_pins": 1, 2024-04-23T10:32:04.898 INFO:teuthology.orchestra.run.smithi114.stdout: "is_frozen": false, 2024-04-23T10:32:04.898 INFO:teuthology.orchestra.run.smithi114.stdout: "is_freezing": false, 2024-04-23T10:32:04.898 INFO:teuthology.orchestra.run.smithi114.stdout: "pins": { 2024-04-23T10:32:04.898 INFO:teuthology.orchestra.run.smithi114.stdout: "ptrwaiter": 0, 2024-04-23T10:32:04.898 INFO:teuthology.orchestra.run.smithi114.stdout: "request": 1, 2024-04-23T10:32:04.899 INFO:teuthology.orchestra.run.smithi114.stdout: "lock": 1, 2024-04-23T10:32:04.899 INFO:teuthology.orchestra.run.smithi114.stdout: "caps": 0, 2024-04-23T10:32:04.899 INFO:teuthology.orchestra.run.smithi114.stdout: "needsnapflush": 0, 2024-04-23T10:32:04.899 INFO:teuthology.orchestra.run.smithi114.stdout: "dirtyparent": 0, 2024-04-23T10:32:04.899 INFO:teuthology.orchestra.run.smithi114.stdout: "dirty": 0, 2024-04-23T10:32:04.899 INFO:teuthology.orchestra.run.smithi114.stdout: "waiter": 0, 2024-04-23T10:32:04.899 INFO:teuthology.orchestra.run.smithi114.stdout: "authpin": 1 2024-04-23T10:32:04.899 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-23T10:32:04.899 INFO:teuthology.orchestra.run.smithi114.stdout: "nref": 3 2024-04-23T10:32:04.899 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-23T10:32:04.899 INFO:teuthology.orchestra.run.smithi114.stdout: "object_string": "[inode 0x1000000020a [6,head] /volumes/qa/sv_1/b206794e-7622-4ab8-8943-f3ae4e65b7a3/client.0/tmp/clients/client0/~dmtmp/WORD/BASEMACH.DOC auth v5584 ap=1 s=260096 n(v0 rc2024-04-23T10:27:51.959398+0000 b260096 1=1+0) (iquiesce lock x=1 by request(mds.0:180 nref=3)) | ptrwaiter=0 request=1 lock=1 caps=0 needsnapflush=0 dirtyparent=0 dirty=0 waiter=0 authpin=1 0x5571cf4d7700]", 2024-04-23T10:32:04.899 INFO:teuthology.orchestra.run.smithi114.stdout: "lock": { 2024-04-23T10:32:04.899 INFO:teuthology.orchestra.run.smithi114.stdout: "gather_set": [], 2024-04-23T10:32:04.899 INFO:teuthology.orchestra.run.smithi114.stdout: "state": "lock", 2024-04-23T10:32:04.899 INFO:teuthology.orchestra.run.smithi114.stdout: "type": "iquiesce", 2024-04-23T10:32:04.900 INFO:teuthology.orchestra.run.smithi114.stdout: "is_leased": false, 2024-04-23T10:32:04.900 INFO:teuthology.orchestra.run.smithi114.stdout: "num_rdlocks": 0, 2024-04-23T10:32:04.900 INFO:teuthology.orchestra.run.smithi114.stdout: "num_wrlocks": 0, 2024-04-23T10:32:04.900 INFO:teuthology.orchestra.run.smithi114.stdout: "num_xlocks": 1, 2024-04-23T10:32:04.900 INFO:teuthology.orchestra.run.smithi114.stdout: "xlock_by": { 2024-04-23T10:32:04.900 INFO:teuthology.orchestra.run.smithi114.stdout: "reqid": { 2024-04-23T10:32:04.900 INFO:teuthology.orchestra.run.smithi114.stdout: "entity": { 2024-04-23T10:32:04.900 INFO:teuthology.orchestra.run.smithi114.stdout: "type": "mds", 2024-04-23T10:32:04.900 INFO:teuthology.orchestra.run.smithi114.stdout: "num": 0 2024-04-23T10:32:04.900 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-23T10:32:04.900 INFO:teuthology.orchestra.run.smithi114.stdout: "tid": 180 2024-04-23T10:32:04.900 INFO:teuthology.orchestra.run.smithi114.stdout: } 2024-04-23T10:32:04.900 INFO:teuthology.orchestra.run.smithi114.stdout: } 2024-04-23T10:32:04.900 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-23T10:32:04.900 INFO:teuthology.orchestra.run.smithi114.stdout: "flags": 4, 2024-04-23T10:32:04.900 INFO:teuthology.orchestra.run.smithi114.stdout: "wrlock_target": -1 2024-04-23T10:32:04.901 INFO:teuthology.orchestra.run.smithi114.stdout: } 2024-04-23T10:32:04.901 INFO:teuthology.orchestra.run.smithi114.stdout: ] 2024-04-23T10:32:04.901 INFO:teuthology.orchestra.run.smithi114.stdout: } 2024-04-23T10:32:04.901 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-23T10:32:04.901 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-04-23T10:32:04.901 INFO:teuthology.orchestra.run.smithi114.stdout: "description": "peer_request:mds.1:118", 2024-04-23T10:32:04.901 INFO:teuthology.orchestra.run.smithi114.stdout: "initiated_at": "2024-04-23T10:31:21.464992+0000", 2024-04-23T10:32:04.901 INFO:teuthology.orchestra.run.smithi114.stdout: "age": 42.419397154999999, 2024-04-23T10:32:04.901 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 42.423539978000001, 2024-04-23T10:32:04.901 INFO:teuthology.orchestra.run.smithi114.stdout: "continuous": false, 2024-04-23T10:32:04.901 INFO:teuthology.orchestra.run.smithi114.stdout: "type_data": { 2024-04-23T10:32:04.901 INFO:teuthology.orchestra.run.smithi114.stdout: "result": -2147483648, 2024-04-23T10:32:04.901 INFO:teuthology.orchestra.run.smithi114.stdout: "flag_point": "dispatched", 2024-04-23T10:32:04.901 INFO:teuthology.orchestra.run.smithi114.stdout: "reqid": { 2024-04-23T10:32:04.901 INFO:teuthology.orchestra.run.smithi114.stdout: "entity": { 2024-04-23T10:32:04.902 INFO:teuthology.orchestra.run.smithi114.stdout: "type": "mds", 2024-04-23T10:32:04.902 INFO:teuthology.orchestra.run.smithi114.stdout: "num": 1 2024-04-23T10:32:04.902 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-23T10:32:04.902 INFO:teuthology.orchestra.run.smithi114.stdout: "tid": 118 2024-04-23T10:32:04.902 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-23T10:32:04.902 INFO:teuthology.orchestra.run.smithi114.stdout: "op_type": "peer_request", 2024-04-23T10:32:04.902 INFO:teuthology.orchestra.run.smithi114.stdout: "leader_info": { 2024-04-23T10:32:04.902 INFO:teuthology.orchestra.run.smithi114.stdout: "leader": "1" 2024-04-23T10:32:04.902 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-23T10:32:04.902 INFO:teuthology.orchestra.run.smithi114.stdout: "events": [ 2024-04-23T10:32:04.902 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-04-23T10:32:04.902 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-04-23T10:31:21.464992+0000", 2024-04-23T10:32:04.902 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "initiated" 2024-04-23T10:32:04.902 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-23T10:32:04.903 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-04-23T10:32:04.903 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-04-23T10:31:21.464992+0000", 2024-04-23T10:32:04.903 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "header_read" 2024-04-23T10:32:04.903 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-23T10:32:04.903 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-04-23T10:32:04.903 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-04-23T10:31:21.464993+0000", 2024-04-23T10:32:04.903 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "throttled" 2024-04-23T10:32:04.903 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-23T10:32:04.903 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-04-23T10:32:04.903 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-04-23T10:31:21.464998+0000", 2024-04-23T10:32:04.903 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "all_read" 2024-04-23T10:32:04.903 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-23T10:32:04.903 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-04-23T10:32:04.903 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-04-23T10:31:21.493737+0000", 2024-04-23T10:32:04.903 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "dispatched" 2024-04-23T10:32:04.903 INFO:teuthology.orchestra.run.smithi114.stdout: } 2024-04-23T10:32:04.904 INFO:teuthology.orchestra.run.smithi114.stdout: ], 2024-04-23T10:32:04.904 INFO:teuthology.orchestra.run.smithi114.stdout: "locks": [] 2024-04-23T10:32:04.904 INFO:teuthology.orchestra.run.smithi114.stdout: } 2024-04-23T10:32:04.904 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-23T10:32:04.904 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-04-23T10:32:04.904 INFO:teuthology.orchestra.run.smithi114.stdout: "description": "internal op quiesce_inode:mds.0:157 fp=#0x10000000006", 2024-04-23T10:32:04.904 INFO:teuthology.orchestra.run.smithi114.stdout: "initiated_at": "2024-04-23T10:31:21.452552+0000", 2024-04-23T10:32:04.904 INFO:teuthology.orchestra.run.smithi114.stdout: "age": 42.431837444000003, 2024-04-23T10:32:04.904 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 42.436003264999997, 2024-04-23T10:32:04.904 INFO:teuthology.orchestra.run.smithi114.stdout: "continuous": true, 2024-04-23T10:32:04.904 INFO:teuthology.orchestra.run.smithi114.stdout: "type_data": { 2024-04-23T10:32:04.904 INFO:teuthology.orchestra.run.smithi114.stdout: "result": -2147483648, 2024-04-23T10:32:04.904 INFO:teuthology.orchestra.run.smithi114.stdout: "flag_point": "quiesce complete", 2024-04-23T10:32:04.904 INFO:teuthology.orchestra.run.smithi114.stdout: "reqid": { 2024-04-23T10:32:04.904 INFO:teuthology.orchestra.run.smithi114.stdout: "entity": { 2024-04-23T10:32:04.904 INFO:teuthology.orchestra.run.smithi114.stdout: "type": "mds", 2024-04-23T10:32:04.905 INFO:teuthology.orchestra.run.smithi114.stdout: "num": 0 2024-04-23T10:32:04.905 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-23T10:32:04.905 INFO:teuthology.orchestra.run.smithi114.stdout: "tid": 157 2024-04-23T10:32:04.905 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-23T10:32:04.905 INFO:teuthology.orchestra.run.smithi114.stdout: "op_type": "internal_op", 2024-04-23T10:32:04.905 INFO:teuthology.orchestra.run.smithi114.stdout: "internal_op": 5385, 2024-04-23T10:32:04.905 INFO:teuthology.orchestra.run.smithi114.stdout: "op_name": "quiesce_inode", 2024-04-23T10:32:04.905 INFO:teuthology.orchestra.run.smithi114.stdout: "events": [ 2024-04-23T10:32:04.905 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-04-23T10:32:04.905 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-04-23T10:31:21.452552+0000", 2024-04-23T10:32:04.905 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "initiated" 2024-04-23T10:32:04.905 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-23T10:32:04.905 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-04-23T10:32:04.905 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-04-23T10:31:21.452552+0000", 2024-04-23T10:32:04.906 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "header_read" 2024-04-23T10:32:04.906 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-23T10:32:04.906 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-04-23T10:32:04.906 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-04-23T10:31:21.452552+0000", 2024-04-23T10:32:04.906 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "throttled" 2024-04-23T10:32:04.906 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-23T10:32:04.906 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-04-23T10:32:04.906 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-04-23T10:31:21.452552+0000", 2024-04-23T10:32:04.906 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "all_read" 2024-04-23T10:32:04.906 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-23T10:32:04.906 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-04-23T10:32:04.906 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-04-23T10:31:21.452552+0000", 2024-04-23T10:32:04.906 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "dispatched" 2024-04-23T10:32:04.906 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-23T10:32:04.906 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-04-23T10:32:04.907 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-04-23T10:31:21.452764+0000", 2024-04-23T10:32:04.907 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "failed to xlock, waiting" 2024-04-23T10:32:04.907 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-23T10:32:04.907 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-04-23T10:32:04.907 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-04-23T10:31:21.453845+0000", 2024-04-23T10:32:04.907 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "acquired locks" 2024-04-23T10:32:04.907 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-23T10:32:04.907 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-04-23T10:32:04.907 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-04-23T10:31:21.509910+0000", 2024-04-23T10:32:04.907 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "quiesce complete" 2024-04-23T10:32:04.907 INFO:teuthology.orchestra.run.smithi114.stdout: } 2024-04-23T10:32:04.907 INFO:teuthology.orchestra.run.smithi114.stdout: ], 2024-04-23T10:32:04.907 INFO:teuthology.orchestra.run.smithi114.stdout: "locks": [ 2024-04-23T10:32:04.907 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-04-23T10:32:04.907 INFO:teuthology.orchestra.run.smithi114.stdout: "object": { 2024-04-23T10:32:04.908 INFO:teuthology.orchestra.run.smithi114.stdout: "is_auth": true, 2024-04-23T10:32:04.908 INFO:teuthology.orchestra.run.smithi114.stdout: "auth_state": { 2024-04-23T10:32:04.908 INFO:teuthology.orchestra.run.smithi114.stdout: "replicas": { 2024-04-23T10:32:04.908 INFO:teuthology.orchestra.run.smithi114.stdout: "1": 1, 2024-04-23T10:32:04.908 INFO:teuthology.orchestra.run.smithi114.stdout: "2": 1 2024-04-23T10:32:04.908 INFO:teuthology.orchestra.run.smithi114.stdout: } 2024-04-23T10:32:04.908 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-23T10:32:04.908 INFO:teuthology.orchestra.run.smithi114.stdout: "replica_state": { 2024-04-23T10:32:04.908 INFO:teuthology.orchestra.run.smithi114.stdout: "authority": [ 2024-04-23T10:32:04.908 INFO:teuthology.orchestra.run.smithi114.stdout: 0, 2024-04-23T10:32:04.908 INFO:teuthology.orchestra.run.smithi114.stdout: -2 2024-04-23T10:32:04.908 INFO:teuthology.orchestra.run.smithi114.stdout: ], 2024-04-23T10:32:04.908 INFO:teuthology.orchestra.run.smithi114.stdout: "replica_nonce": 0 2024-04-23T10:32:04.908 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-23T10:32:04.908 INFO:teuthology.orchestra.run.smithi114.stdout: "auth_pins": 3, 2024-04-23T10:32:04.908 INFO:teuthology.orchestra.run.smithi114.stdout: "is_frozen": false, 2024-04-23T10:32:04.909 INFO:teuthology.orchestra.run.smithi114.stdout: "is_freezing": false, 2024-04-23T10:32:04.909 INFO:teuthology.orchestra.run.smithi114.stdout: "pins": { 2024-04-23T10:32:04.909 INFO:teuthology.orchestra.run.smithi114.stdout: "dirtyscattered": 0, 2024-04-23T10:32:04.909 INFO:teuthology.orchestra.run.smithi114.stdout: "request": 1, 2024-04-23T10:32:04.909 INFO:teuthology.orchestra.run.smithi114.stdout: "lock": 1, 2024-04-23T10:32:04.909 INFO:teuthology.orchestra.run.smithi114.stdout: "dirfrag": 1, 2024-04-23T10:32:04.909 INFO:teuthology.orchestra.run.smithi114.stdout: "caps": 1, 2024-04-23T10:32:04.909 INFO:teuthology.orchestra.run.smithi114.stdout: "dirtyrstat": 1, 2024-04-23T10:32:04.909 INFO:teuthology.orchestra.run.smithi114.stdout: "dirtyparent": 0, 2024-04-23T10:32:04.909 INFO:teuthology.orchestra.run.smithi114.stdout: "replicated": 1, 2024-04-23T10:32:04.909 INFO:teuthology.orchestra.run.smithi114.stdout: "dirty": 1, 2024-04-23T10:32:04.909 INFO:teuthology.orchestra.run.smithi114.stdout: "waiter": 0, 2024-04-23T10:32:04.909 INFO:teuthology.orchestra.run.smithi114.stdout: "authpin": 1 2024-04-23T10:32:04.909 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-23T10:32:04.909 INFO:teuthology.orchestra.run.smithi114.stdout: "nref": 8 2024-04-23T10:32:04.909 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-23T10:32:04.909 INFO:teuthology.orchestra.run.smithi114.stdout: "object_string": "[inode 0x10000000006 [...7,head] /volumes/qa/sv_1/b206794e-7622-4ab8-8943-f3ae4e65b7a3/client.0/tmp/ auth{1=1,2=1} v417 ap=3 f(v0 m2024-04-23T10:27:51.919399+0000 1=0+1) n(v30 rc2024-04-23T10:31:17.764716+0000 b21075074 76=63+13) old_inodes=4 (inest mix) (iquiesce lock x=1 by request(mds.0:157 nref=3)) caps={15312=pAsLsXs/-@9},l=15312 | dirtyscattered=0 request=1 lock=1 dirfrag=1 caps=1 dirtyrstat=1 dirtyparent=0 replicated=1 dirty=1 waiter=0 authpin=1 0x5571cf4d6100]", 2024-04-23T10:32:04.910 INFO:teuthology.orchestra.run.smithi114.stdout: "lock": { 2024-04-23T10:32:04.910 INFO:teuthology.orchestra.run.smithi114.stdout: "gather_set": [], 2024-04-23T10:32:04.910 INFO:teuthology.orchestra.run.smithi114.stdout: "state": "lock", 2024-04-23T10:32:04.910 INFO:teuthology.orchestra.run.smithi114.stdout: "type": "iquiesce", 2024-04-23T10:32:04.910 INFO:teuthology.orchestra.run.smithi114.stdout: "is_leased": false, 2024-04-23T10:32:04.910 INFO:teuthology.orchestra.run.smithi114.stdout: "num_rdlocks": 0, 2024-04-23T10:32:04.910 INFO:teuthology.orchestra.run.smithi114.stdout: "num_wrlocks": 0, 2024-04-23T10:32:04.910 INFO:teuthology.orchestra.run.smithi114.stdout: "num_xlocks": 1, 2024-04-23T10:32:04.910 INFO:teuthology.orchestra.run.smithi114.stdout: "xlock_by": { 2024-04-23T10:32:04.910 INFO:teuthology.orchestra.run.smithi114.stdout: "reqid": { 2024-04-23T10:32:04.910 INFO:teuthology.orchestra.run.smithi114.stdout: "entity": { 2024-04-23T10:32:04.910 INFO:teuthology.orchestra.run.smithi114.stdout: "type": "mds", 2024-04-23T10:32:04.910 INFO:teuthology.orchestra.run.smithi114.stdout: "num": 0 2024-04-23T10:32:04.910 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-23T10:32:04.911 INFO:teuthology.orchestra.run.smithi114.stdout: "tid": 157 2024-04-23T10:32:04.911 INFO:teuthology.orchestra.run.smithi114.stdout: } 2024-04-23T10:32:04.911 INFO:teuthology.orchestra.run.smithi114.stdout: } 2024-04-23T10:32:04.911 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-23T10:32:04.911 INFO:teuthology.orchestra.run.smithi114.stdout: "flags": 4, 2024-04-23T10:32:04.911 INFO:teuthology.orchestra.run.smithi114.stdout: "wrlock_target": -1 2024-04-23T10:32:04.911 INFO:teuthology.orchestra.run.smithi114.stdout: } 2024-04-23T10:32:04.911 INFO:teuthology.orchestra.run.smithi114.stdout: ] 2024-04-23T10:32:04.911 INFO:teuthology.orchestra.run.smithi114.stdout: } 2024-04-23T10:32:04.911 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-23T10:32:04.911 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-04-23T10:32:04.911 INFO:teuthology.orchestra.run.smithi114.stdout: "description": "internal op quiesce_inode:mds.0:181 fp=#0x1000000020b", 2024-04-23T10:32:04.911 INFO:teuthology.orchestra.run.smithi114.stdout: "initiated_at": "2024-04-23T10:31:21.468582+0000", 2024-04-23T10:32:04.911 INFO:teuthology.orchestra.run.smithi114.stdout: "age": 42.415806846999999, 2024-04-23T10:32:04.911 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 42.420045846999997, 2024-04-23T10:32:04.912 INFO:teuthology.orchestra.run.smithi114.stdout: "continuous": true, 2024-04-23T10:32:04.912 INFO:teuthology.orchestra.run.smithi114.stdout: "type_data": { 2024-04-23T10:32:04.912 INFO:teuthology.orchestra.run.smithi114.stdout: "result": -2147483648, 2024-04-23T10:32:04.912 INFO:teuthology.orchestra.run.smithi114.stdout: "flag_point": "quiesce complete", 2024-04-23T10:32:04.912 INFO:teuthology.orchestra.run.smithi114.stdout: "reqid": { 2024-04-23T10:32:04.912 INFO:teuthology.orchestra.run.smithi114.stdout: "entity": { 2024-04-23T10:32:04.912 INFO:teuthology.orchestra.run.smithi114.stdout: "type": "mds", 2024-04-23T10:32:04.912 INFO:teuthology.orchestra.run.smithi114.stdout: "num": 0 2024-04-23T10:32:04.912 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-23T10:32:04.912 INFO:teuthology.orchestra.run.smithi114.stdout: "tid": 181 2024-04-23T10:32:04.912 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-23T10:32:04.912 INFO:teuthology.orchestra.run.smithi114.stdout: "op_type": "internal_op", 2024-04-23T10:32:04.912 INFO:teuthology.orchestra.run.smithi114.stdout: "internal_op": 5385, 2024-04-23T10:32:04.912 INFO:teuthology.orchestra.run.smithi114.stdout: "op_name": "quiesce_inode", 2024-04-23T10:32:04.912 INFO:teuthology.orchestra.run.smithi114.stdout: "events": [ 2024-04-23T10:32:04.913 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-04-23T10:32:04.913 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-04-23T10:31:21.468582+0000", 2024-04-23T10:32:04.913 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "initiated" 2024-04-23T10:32:04.913 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-23T10:32:04.913 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-04-23T10:32:04.913 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-04-23T10:31:21.468582+0000", 2024-04-23T10:32:04.913 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "header_read" 2024-04-23T10:32:04.913 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-23T10:32:04.913 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-04-23T10:32:04.913 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-04-23T10:31:21.468582+0000", 2024-04-23T10:32:04.913 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "throttled" 2024-04-23T10:32:04.913 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-23T10:32:04.913 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-04-23T10:32:04.913 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-04-23T10:31:21.468582+0000", 2024-04-23T10:32:04.913 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "all_read" 2024-04-23T10:32:04.914 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-23T10:32:04.914 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-04-23T10:32:04.914 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-04-23T10:31:21.468582+0000", 2024-04-23T10:32:04.914 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "dispatched" 2024-04-23T10:32:04.914 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-23T10:32:04.914 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-04-23T10:32:04.914 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-04-23T10:31:21.468953+0000", 2024-04-23T10:32:04.914 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "failed to xlock, waiting" 2024-04-23T10:32:04.914 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-23T10:32:04.914 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-04-23T10:32:04.914 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-04-23T10:31:21.487718+0000", 2024-04-23T10:32:04.914 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "failed to xlock, waiting" 2024-04-23T10:32:04.914 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-23T10:32:04.914 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-04-23T10:32:04.915 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-04-23T10:31:21.492458+0000", 2024-04-23T10:32:04.915 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "acquired locks" 2024-04-23T10:32:04.915 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-23T10:32:04.915 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-04-23T10:32:04.915 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-04-23T10:31:21.492666+0000", 2024-04-23T10:32:04.915 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "quiesce complete" 2024-04-23T10:32:04.915 INFO:teuthology.orchestra.run.smithi114.stdout: } 2024-04-23T10:32:04.915 INFO:teuthology.orchestra.run.smithi114.stdout: ], 2024-04-23T10:32:04.915 INFO:teuthology.orchestra.run.smithi114.stdout: "locks": [ 2024-04-23T10:32:04.915 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-04-23T10:32:04.915 INFO:teuthology.orchestra.run.smithi114.stdout: "object": { 2024-04-23T10:32:04.915 INFO:teuthology.orchestra.run.smithi114.stdout: "is_auth": true, 2024-04-23T10:32:04.915 INFO:teuthology.orchestra.run.smithi114.stdout: "auth_state": { 2024-04-23T10:32:04.915 INFO:teuthology.orchestra.run.smithi114.stdout: "replicas": {} 2024-04-23T10:32:04.915 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-23T10:32:04.915 INFO:teuthology.orchestra.run.smithi114.stdout: "replica_state": { 2024-04-23T10:32:04.916 INFO:teuthology.orchestra.run.smithi114.stdout: "authority": [ 2024-04-23T10:32:04.916 INFO:teuthology.orchestra.run.smithi114.stdout: 0, 2024-04-23T10:32:04.916 INFO:teuthology.orchestra.run.smithi114.stdout: -2 2024-04-23T10:32:04.916 INFO:teuthology.orchestra.run.smithi114.stdout: ], 2024-04-23T10:32:04.916 INFO:teuthology.orchestra.run.smithi114.stdout: "replica_nonce": 0 2024-04-23T10:32:04.916 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-23T10:32:04.916 INFO:teuthology.orchestra.run.smithi114.stdout: "auth_pins": 1, 2024-04-23T10:32:04.916 INFO:teuthology.orchestra.run.smithi114.stdout: "is_frozen": false, 2024-04-23T10:32:04.916 INFO:teuthology.orchestra.run.smithi114.stdout: "is_freezing": false, 2024-04-23T10:32:04.916 INFO:teuthology.orchestra.run.smithi114.stdout: "pins": { 2024-04-23T10:32:04.916 INFO:teuthology.orchestra.run.smithi114.stdout: "ptrwaiter": 0, 2024-04-23T10:32:04.916 INFO:teuthology.orchestra.run.smithi114.stdout: "request": 1, 2024-04-23T10:32:04.916 INFO:teuthology.orchestra.run.smithi114.stdout: "lock": 1, 2024-04-23T10:32:04.916 INFO:teuthology.orchestra.run.smithi114.stdout: "caps": 0, 2024-04-23T10:32:04.917 INFO:teuthology.orchestra.run.smithi114.stdout: "needsnapflush": 0, 2024-04-23T10:32:04.917 INFO:teuthology.orchestra.run.smithi114.stdout: "dirtyparent": 0, 2024-04-23T10:32:04.917 INFO:teuthology.orchestra.run.smithi114.stdout: "dirty": 0, 2024-04-23T10:32:04.917 INFO:teuthology.orchestra.run.smithi114.stdout: "waiter": 0, 2024-04-23T10:32:04.917 INFO:teuthology.orchestra.run.smithi114.stdout: "authpin": 1 2024-04-23T10:32:04.917 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-23T10:32:04.917 INFO:teuthology.orchestra.run.smithi114.stdout: "nref": 3 2024-04-23T10:32:04.917 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-23T10:32:04.917 INFO:teuthology.orchestra.run.smithi114.stdout: "object_string": "[inode 0x1000000020b [6,head] /volumes/qa/sv_1/b206794e-7622-4ab8-8943-f3ae4e65b7a3/client.0/tmp/clients/client0/~dmtmp/WORD/FACTS.DOC auth v5586 ap=1 s=260096 n(v0 rc2024-04-23T10:27:51.965397+0000 b260096 1=1+0) (iquiesce lock x=1 by request(mds.0:181 nref=3)) | ptrwaiter=0 request=1 lock=1 caps=0 needsnapflush=0 dirtyparent=0 dirty=0 waiter=0 authpin=1 0x5571cf53e000]", 2024-04-23T10:32:04.917 INFO:teuthology.orchestra.run.smithi114.stdout: "lock": { 2024-04-23T10:32:04.917 INFO:teuthology.orchestra.run.smithi114.stdout: "gather_set": [], 2024-04-23T10:32:04.917 INFO:teuthology.orchestra.run.smithi114.stdout: "state": "lock", 2024-04-23T10:32:04.917 INFO:teuthology.orchestra.run.smithi114.stdout: "type": "iquiesce", 2024-04-23T10:32:04.917 INFO:teuthology.orchestra.run.smithi114.stdout: "is_leased": false, 2024-04-23T10:32:04.918 INFO:teuthology.orchestra.run.smithi114.stdout: "num_rdlocks": 0, 2024-04-23T10:32:04.918 INFO:teuthology.orchestra.run.smithi114.stdout: "num_wrlocks": 0, 2024-04-23T10:32:04.918 INFO:teuthology.orchestra.run.smithi114.stdout: "num_xlocks": 1, 2024-04-23T10:32:04.918 INFO:teuthology.orchestra.run.smithi114.stdout: "xlock_by": { 2024-04-23T10:32:04.918 INFO:teuthology.orchestra.run.smithi114.stdout: "reqid": { 2024-04-23T10:32:04.918 INFO:teuthology.orchestra.run.smithi114.stdout: "entity": { 2024-04-23T10:32:04.918 INFO:teuthology.orchestra.run.smithi114.stdout: "type": "mds", 2024-04-23T10:32:04.918 INFO:teuthology.orchestra.run.smithi114.stdout: "num": 0 2024-04-23T10:32:04.918 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-23T10:32:04.918 INFO:teuthology.orchestra.run.smithi114.stdout: "tid": 181 2024-04-23T10:32:04.918 INFO:teuthology.orchestra.run.smithi114.stdout: } 2024-04-23T10:32:04.918 INFO:teuthology.orchestra.run.smithi114.stdout: } 2024-04-23T10:32:04.918 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-23T10:32:04.919 INFO:teuthology.orchestra.run.smithi114.stdout: "flags": 4, 2024-04-23T10:32:04.919 INFO:teuthology.orchestra.run.smithi114.stdout: "wrlock_target": -1 2024-04-23T10:32:04.919 INFO:teuthology.orchestra.run.smithi114.stdout: } 2024-04-23T10:32:04.919 INFO:teuthology.orchestra.run.smithi114.stdout: ] 2024-04-23T10:32:04.919 INFO:teuthology.orchestra.run.smithi114.stdout: } 2024-04-23T10:32:04.919 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-23T10:32:04.919 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-04-23T10:32:04.919 INFO:teuthology.orchestra.run.smithi114.stdout: "description": "peer_request:mds.2:123", 2024-04-23T10:32:04.919 INFO:teuthology.orchestra.run.smithi114.stdout: "initiated_at": "2024-04-23T10:31:21.463782+0000", 2024-04-23T10:32:04.919 INFO:teuthology.orchestra.run.smithi114.stdout: "age": 42.420607275999998, 2024-04-23T10:32:04.919 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 42.424913517999997, 2024-04-23T10:32:04.919 INFO:teuthology.orchestra.run.smithi114.stdout: "continuous": false, 2024-04-23T10:32:04.919 INFO:teuthology.orchestra.run.smithi114.stdout: "type_data": { 2024-04-23T10:32:04.919 INFO:teuthology.orchestra.run.smithi114.stdout: "result": -2147483648, 2024-04-23T10:32:04.919 INFO:teuthology.orchestra.run.smithi114.stdout: "flag_point": "dispatched", 2024-04-23T10:32:04.920 INFO:teuthology.orchestra.run.smithi114.stdout: "reqid": { 2024-04-23T10:32:04.920 INFO:teuthology.orchestra.run.smithi114.stdout: "entity": { 2024-04-23T10:32:04.920 INFO:teuthology.orchestra.run.smithi114.stdout: "type": "mds", 2024-04-23T10:32:04.920 INFO:teuthology.orchestra.run.smithi114.stdout: "num": 2 2024-04-23T10:32:04.920 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-23T10:32:04.920 INFO:teuthology.orchestra.run.smithi114.stdout: "tid": 123 2024-04-23T10:32:04.920 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-23T10:32:04.920 INFO:teuthology.orchestra.run.smithi114.stdout: "op_type": "peer_request", 2024-04-23T10:32:04.920 INFO:teuthology.orchestra.run.smithi114.stdout: "leader_info": { 2024-04-23T10:32:04.920 INFO:teuthology.orchestra.run.smithi114.stdout: "leader": "2" 2024-04-23T10:32:04.920 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-23T10:32:04.920 INFO:teuthology.orchestra.run.smithi114.stdout: "events": [ 2024-04-23T10:32:04.920 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-04-23T10:32:04.920 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-04-23T10:31:21.463782+0000", 2024-04-23T10:32:04.921 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "initiated" 2024-04-23T10:32:04.921 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-23T10:32:04.921 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-04-23T10:32:04.921 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-04-23T10:31:21.463782+0000", 2024-04-23T10:32:04.921 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "header_read" 2024-04-23T10:32:04.921 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-23T10:32:04.921 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-04-23T10:32:04.921 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-04-23T10:31:21.463782+0000", 2024-04-23T10:32:04.921 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "throttled" 2024-04-23T10:32:04.921 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-23T10:32:04.921 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-04-23T10:32:04.921 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-04-23T10:31:21.463788+0000", 2024-04-23T10:32:04.921 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "all_read" 2024-04-23T10:32:04.921 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-23T10:32:04.922 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-04-23T10:32:04.922 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-04-23T10:31:21.493829+0000", 2024-04-23T10:32:04.922 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "dispatched" 2024-04-23T10:32:04.922 INFO:teuthology.orchestra.run.smithi114.stdout: } 2024-04-23T10:32:04.922 INFO:teuthology.orchestra.run.smithi114.stdout: ], 2024-04-23T10:32:04.922 INFO:teuthology.orchestra.run.smithi114.stdout: "locks": [] 2024-04-23T10:32:04.922 INFO:teuthology.orchestra.run.smithi114.stdout: } 2024-04-23T10:32:04.922 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-23T10:32:04.922 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-04-23T10:32:04.922 INFO:teuthology.orchestra.run.smithi114.stdout: "description": "peer_request:mds.2:113", 2024-04-23T10:32:04.922 INFO:teuthology.orchestra.run.smithi114.stdout: "initiated_at": "2024-04-23T10:31:21.452144+0000", 2024-04-23T10:32:04.922 INFO:teuthology.orchestra.run.smithi114.stdout: "age": 42.432245659000003, 2024-04-23T10:32:04.922 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 42.436573430999999, 2024-04-23T10:32:04.922 INFO:teuthology.orchestra.run.smithi114.stdout: "continuous": false, 2024-04-23T10:32:04.923 INFO:teuthology.orchestra.run.smithi114.stdout: "type_data": { 2024-04-23T10:32:04.923 INFO:teuthology.orchestra.run.smithi114.stdout: "result": -2147483648, 2024-04-23T10:32:04.923 INFO:teuthology.orchestra.run.smithi114.stdout: "flag_point": "dispatched", 2024-04-23T10:32:04.923 INFO:teuthology.orchestra.run.smithi114.stdout: "reqid": { 2024-04-23T10:32:04.923 INFO:teuthology.orchestra.run.smithi114.stdout: "entity": { 2024-04-23T10:32:04.923 INFO:teuthology.orchestra.run.smithi114.stdout: "type": "mds", 2024-04-23T10:32:04.923 INFO:teuthology.orchestra.run.smithi114.stdout: "num": 2 2024-04-23T10:32:04.923 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-23T10:32:04.923 INFO:teuthology.orchestra.run.smithi114.stdout: "tid": 113 2024-04-23T10:32:04.923 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-23T10:32:04.923 INFO:teuthology.orchestra.run.smithi114.stdout: "op_type": "peer_request", 2024-04-23T10:32:04.923 INFO:teuthology.orchestra.run.smithi114.stdout: "leader_info": { 2024-04-23T10:32:04.923 INFO:teuthology.orchestra.run.smithi114.stdout: "leader": "2" 2024-04-23T10:32:04.923 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-23T10:32:04.924 INFO:teuthology.orchestra.run.smithi114.stdout: "events": [ 2024-04-23T10:32:04.924 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-04-23T10:32:04.924 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-04-23T10:31:21.452144+0000", 2024-04-23T10:32:04.924 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "initiated" 2024-04-23T10:32:04.924 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-23T10:32:04.924 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-04-23T10:32:04.924 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-04-23T10:31:21.452144+0000", 2024-04-23T10:32:04.924 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "header_read" 2024-04-23T10:32:04.924 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-23T10:32:04.924 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-04-23T10:32:04.924 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-04-23T10:31:21.452144+0000", 2024-04-23T10:32:04.924 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "throttled" 2024-04-23T10:32:04.924 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-23T10:32:04.924 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-04-23T10:32:04.925 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-04-23T10:31:21.452151+0000", 2024-04-23T10:32:04.925 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "all_read" 2024-04-23T10:32:04.925 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-23T10:32:04.925 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-04-23T10:32:04.925 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-04-23T10:31:21.453185+0000", 2024-04-23T10:32:04.925 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "dispatched" 2024-04-23T10:32:04.925 INFO:teuthology.orchestra.run.smithi114.stdout: } 2024-04-23T10:32:04.925 INFO:teuthology.orchestra.run.smithi114.stdout: ], 2024-04-23T10:32:04.925 INFO:teuthology.orchestra.run.smithi114.stdout: "locks": [] 2024-04-23T10:32:04.925 INFO:teuthology.orchestra.run.smithi114.stdout: } 2024-04-23T10:32:04.925 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-23T10:32:04.925 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-04-23T10:32:04.925 INFO:teuthology.orchestra.run.smithi114.stdout: "description": "internal op quiesce_inode:mds.0:182 fp=#0x10000000209", 2024-04-23T10:32:04.925 INFO:teuthology.orchestra.run.smithi114.stdout: "initiated_at": "2024-04-23T10:31:21.468596+0000", 2024-04-23T10:32:04.926 INFO:teuthology.orchestra.run.smithi114.stdout: "age": 42.415792758000002, 2024-04-23T10:32:04.926 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 42.420158164, 2024-04-23T10:32:04.926 INFO:teuthology.orchestra.run.smithi114.stdout: "continuous": true, 2024-04-23T10:32:04.926 INFO:teuthology.orchestra.run.smithi114.stdout: "type_data": { 2024-04-23T10:32:04.926 INFO:teuthology.orchestra.run.smithi114.stdout: "result": -2147483648, 2024-04-23T10:32:04.926 INFO:teuthology.orchestra.run.smithi114.stdout: "flag_point": "quiesce complete", 2024-04-23T10:32:04.926 INFO:teuthology.orchestra.run.smithi114.stdout: "reqid": { 2024-04-23T10:32:04.926 INFO:teuthology.orchestra.run.smithi114.stdout: "entity": { 2024-04-23T10:32:04.926 INFO:teuthology.orchestra.run.smithi114.stdout: "type": "mds", 2024-04-23T10:32:04.926 INFO:teuthology.orchestra.run.smithi114.stdout: "num": 0 2024-04-23T10:32:04.926 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-23T10:32:04.926 INFO:teuthology.orchestra.run.smithi114.stdout: "tid": 182 2024-04-23T10:32:04.926 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-23T10:32:04.926 INFO:teuthology.orchestra.run.smithi114.stdout: "op_type": "internal_op", 2024-04-23T10:32:04.927 INFO:teuthology.orchestra.run.smithi114.stdout: "internal_op": 5385, 2024-04-23T10:32:04.927 INFO:teuthology.orchestra.run.smithi114.stdout: "op_name": "quiesce_inode", 2024-04-23T10:32:04.927 INFO:teuthology.orchestra.run.smithi114.stdout: "events": [ 2024-04-23T10:32:04.927 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-04-23T10:32:04.927 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-04-23T10:31:21.468596+0000", 2024-04-23T10:32:04.927 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "initiated" 2024-04-23T10:32:04.927 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-23T10:32:04.927 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-04-23T10:32:04.927 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-04-23T10:31:21.468596+0000", 2024-04-23T10:32:04.927 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "header_read" 2024-04-23T10:32:04.927 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-23T10:32:04.927 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-04-23T10:32:04.927 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-04-23T10:31:21.468596+0000", 2024-04-23T10:32:04.927 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "throttled" 2024-04-23T10:32:04.927 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-23T10:32:04.928 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-04-23T10:32:04.928 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-04-23T10:31:21.468596+0000", 2024-04-23T10:32:04.928 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "all_read" 2024-04-23T10:32:04.928 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-23T10:32:04.928 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-04-23T10:32:04.928 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-04-23T10:31:21.468596+0000", 2024-04-23T10:32:04.928 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "dispatched" 2024-04-23T10:32:04.928 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-23T10:32:04.928 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-04-23T10:32:04.928 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-04-23T10:31:21.469164+0000", 2024-04-23T10:32:04.928 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "failed to xlock, waiting" 2024-04-23T10:32:04.928 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-23T10:32:04.928 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-04-23T10:32:04.928 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-04-23T10:31:21.491959+0000", 2024-04-23T10:32:04.929 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "failed to xlock, waiting" 2024-04-23T10:32:04.929 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-23T10:32:04.929 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-04-23T10:32:04.929 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-04-23T10:31:21.496928+0000", 2024-04-23T10:32:04.929 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "acquired locks" 2024-04-23T10:32:04.929 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-23T10:32:04.929 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-04-23T10:32:04.929 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-04-23T10:31:21.497138+0000", 2024-04-23T10:32:04.929 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "quiesce complete" 2024-04-23T10:32:04.929 INFO:teuthology.orchestra.run.smithi114.stdout: } 2024-04-23T10:32:04.929 INFO:teuthology.orchestra.run.smithi114.stdout: ], 2024-04-23T10:32:04.929 INFO:teuthology.orchestra.run.smithi114.stdout: "locks": [ 2024-04-23T10:32:04.929 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-04-23T10:32:04.929 INFO:teuthology.orchestra.run.smithi114.stdout: "object": { 2024-04-23T10:32:04.929 INFO:teuthology.orchestra.run.smithi114.stdout: "is_auth": true, 2024-04-23T10:32:04.930 INFO:teuthology.orchestra.run.smithi114.stdout: "auth_state": { 2024-04-23T10:32:04.930 INFO:teuthology.orchestra.run.smithi114.stdout: "replicas": {} 2024-04-23T10:32:04.930 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-23T10:32:04.930 INFO:teuthology.orchestra.run.smithi114.stdout: "replica_state": { 2024-04-23T10:32:04.930 INFO:teuthology.orchestra.run.smithi114.stdout: "authority": [ 2024-04-23T10:32:04.930 INFO:teuthology.orchestra.run.smithi114.stdout: 0, 2024-04-23T10:32:04.930 INFO:teuthology.orchestra.run.smithi114.stdout: -2 2024-04-23T10:32:04.930 INFO:teuthology.orchestra.run.smithi114.stdout: ], 2024-04-23T10:32:04.930 INFO:teuthology.orchestra.run.smithi114.stdout: "replica_nonce": 0 2024-04-23T10:32:04.930 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-23T10:32:04.930 INFO:teuthology.orchestra.run.smithi114.stdout: "auth_pins": 1, 2024-04-23T10:32:04.930 INFO:teuthology.orchestra.run.smithi114.stdout: "is_frozen": false, 2024-04-23T10:32:04.930 INFO:teuthology.orchestra.run.smithi114.stdout: "is_freezing": false, 2024-04-23T10:32:04.930 INFO:teuthology.orchestra.run.smithi114.stdout: "pins": { 2024-04-23T10:32:04.930 INFO:teuthology.orchestra.run.smithi114.stdout: "ptrwaiter": 0, 2024-04-23T10:32:04.931 INFO:teuthology.orchestra.run.smithi114.stdout: "request": 1, 2024-04-23T10:32:04.931 INFO:teuthology.orchestra.run.smithi114.stdout: "lock": 1, 2024-04-23T10:32:04.931 INFO:teuthology.orchestra.run.smithi114.stdout: "caps": 0, 2024-04-23T10:32:04.931 INFO:teuthology.orchestra.run.smithi114.stdout: "needsnapflush": 0, 2024-04-23T10:32:04.931 INFO:teuthology.orchestra.run.smithi114.stdout: "dirtyparent": 0, 2024-04-23T10:32:04.931 INFO:teuthology.orchestra.run.smithi114.stdout: "dirty": 0, 2024-04-23T10:32:04.931 INFO:teuthology.orchestra.run.smithi114.stdout: "waiter": 0, 2024-04-23T10:32:04.931 INFO:teuthology.orchestra.run.smithi114.stdout: "authpin": 1 2024-04-23T10:32:04.931 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-23T10:32:04.931 INFO:teuthology.orchestra.run.smithi114.stdout: "nref": 3 2024-04-23T10:32:04.931 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-23T10:32:04.931 INFO:teuthology.orchestra.run.smithi114.stdout: "object_string": "[inode 0x10000000209 [6,head] /volumes/qa/sv_1/b206794e-7622-4ab8-8943-f3ae4e65b7a3/client.0/tmp/clients/client0/~dmtmp/WORD/CHAP10.DOC auth v5588 ap=1 s=260096 n(v0 rc2024-04-23T10:27:51.955398+0000 b260096 1=1+0) (iquiesce lock x=1 by request(mds.0:182 nref=3)) | ptrwaiter=0 request=1 lock=1 caps=0 needsnapflush=0 dirtyparent=0 dirty=0 waiter=0 authpin=1 0x5571cf343700]", 2024-04-23T10:32:04.931 INFO:teuthology.orchestra.run.smithi114.stdout: "lock": { 2024-04-23T10:32:04.931 INFO:teuthology.orchestra.run.smithi114.stdout: "gather_set": [], 2024-04-23T10:32:04.932 INFO:teuthology.orchestra.run.smithi114.stdout: "state": "lock", 2024-04-23T10:32:04.932 INFO:teuthology.orchestra.run.smithi114.stdout: "type": "iquiesce", 2024-04-23T10:32:04.932 INFO:teuthology.orchestra.run.smithi114.stdout: "is_leased": false, 2024-04-23T10:32:04.932 INFO:teuthology.orchestra.run.smithi114.stdout: "num_rdlocks": 0, 2024-04-23T10:32:04.932 INFO:teuthology.orchestra.run.smithi114.stdout: "num_wrlocks": 0, 2024-04-23T10:32:04.932 INFO:teuthology.orchestra.run.smithi114.stdout: "num_xlocks": 1, 2024-04-23T10:32:04.932 INFO:teuthology.orchestra.run.smithi114.stdout: "xlock_by": { 2024-04-23T10:32:04.932 INFO:teuthology.orchestra.run.smithi114.stdout: "reqid": { 2024-04-23T10:32:04.932 INFO:teuthology.orchestra.run.smithi114.stdout: "entity": { 2024-04-23T10:32:04.932 INFO:teuthology.orchestra.run.smithi114.stdout: "type": "mds", 2024-04-23T10:32:04.932 INFO:teuthology.orchestra.run.smithi114.stdout: "num": 0 2024-04-23T10:32:04.932 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-23T10:32:04.932 INFO:teuthology.orchestra.run.smithi114.stdout: "tid": 182 2024-04-23T10:32:04.932 INFO:teuthology.orchestra.run.smithi114.stdout: } 2024-04-23T10:32:04.932 INFO:teuthology.orchestra.run.smithi114.stdout: } 2024-04-23T10:32:04.933 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-23T10:32:04.933 INFO:teuthology.orchestra.run.smithi114.stdout: "flags": 4, 2024-04-23T10:32:04.933 INFO:teuthology.orchestra.run.smithi114.stdout: "wrlock_target": -1 2024-04-23T10:32:04.933 INFO:teuthology.orchestra.run.smithi114.stdout: } 2024-04-23T10:32:04.933 INFO:teuthology.orchestra.run.smithi114.stdout: ] 2024-04-23T10:32:04.933 INFO:teuthology.orchestra.run.smithi114.stdout: } 2024-04-23T10:32:04.933 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-23T10:32:04.933 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-04-23T10:32:04.933 INFO:teuthology.orchestra.run.smithi114.stdout: "description": "peer_request:mds.1:119", 2024-04-23T10:32:04.933 INFO:teuthology.orchestra.run.smithi114.stdout: "initiated_at": "2024-04-23T10:31:21.465063+0000", 2024-04-23T10:32:04.933 INFO:teuthology.orchestra.run.smithi114.stdout: "age": 42.419326462000001, 2024-04-23T10:32:04.933 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 42.423754004000003, 2024-04-23T10:32:04.933 INFO:teuthology.orchestra.run.smithi114.stdout: "continuous": false, 2024-04-23T10:32:04.933 INFO:teuthology.orchestra.run.smithi114.stdout: "type_data": { 2024-04-23T10:32:04.933 INFO:teuthology.orchestra.run.smithi114.stdout: "result": -2147483648, 2024-04-23T10:32:04.934 INFO:teuthology.orchestra.run.smithi114.stdout: "flag_point": "dispatched", 2024-04-23T10:32:04.934 INFO:teuthology.orchestra.run.smithi114.stdout: "reqid": { 2024-04-23T10:32:04.934 INFO:teuthology.orchestra.run.smithi114.stdout: "entity": { 2024-04-23T10:32:04.934 INFO:teuthology.orchestra.run.smithi114.stdout: "type": "mds", 2024-04-23T10:32:04.934 INFO:teuthology.orchestra.run.smithi114.stdout: "num": 1 2024-04-23T10:32:04.934 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-23T10:32:04.934 INFO:teuthology.orchestra.run.smithi114.stdout: "tid": 119 2024-04-23T10:32:04.934 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-23T10:32:04.934 INFO:teuthology.orchestra.run.smithi114.stdout: "op_type": "peer_request", 2024-04-23T10:32:04.934 INFO:teuthology.orchestra.run.smithi114.stdout: "leader_info": { 2024-04-23T10:32:04.934 INFO:teuthology.orchestra.run.smithi114.stdout: "leader": "1" 2024-04-23T10:32:04.934 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-23T10:32:04.934 INFO:teuthology.orchestra.run.smithi114.stdout: "events": [ 2024-04-23T10:32:04.934 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-04-23T10:32:04.934 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-04-23T10:31:21.465063+0000", 2024-04-23T10:32:04.935 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "initiated" 2024-04-23T10:32:04.935 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-23T10:32:04.935 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-04-23T10:32:04.935 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-04-23T10:31:21.465063+0000", 2024-04-23T10:32:04.935 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "header_read" 2024-04-23T10:32:04.935 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-23T10:32:04.935 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-04-23T10:32:04.935 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-04-23T10:31:21.465063+0000", 2024-04-23T10:32:04.935 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "throttled" 2024-04-23T10:32:04.935 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-23T10:32:04.935 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-04-23T10:32:04.935 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-04-23T10:31:21.465068+0000", 2024-04-23T10:32:04.935 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "all_read" 2024-04-23T10:32:04.935 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-23T10:32:04.936 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-04-23T10:32:04.936 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-04-23T10:31:21.495290+0000", 2024-04-23T10:32:04.936 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "dispatched" 2024-04-23T10:32:04.936 INFO:teuthology.orchestra.run.smithi114.stdout: } 2024-04-23T10:32:04.936 INFO:teuthology.orchestra.run.smithi114.stdout: ], 2024-04-23T10:32:04.936 INFO:teuthology.orchestra.run.smithi114.stdout: "locks": [] 2024-04-23T10:32:04.936 INFO:teuthology.orchestra.run.smithi114.stdout: } 2024-04-23T10:32:04.936 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-23T10:32:04.936 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-04-23T10:32:04.936 INFO:teuthology.orchestra.run.smithi114.stdout: "description": "peer_request:mds.1:109", 2024-04-23T10:32:04.936 INFO:teuthology.orchestra.run.smithi114.stdout: "initiated_at": "2024-04-23T10:31:21.452180+0000", 2024-04-23T10:32:04.936 INFO:teuthology.orchestra.run.smithi114.stdout: "age": 42.432209415999999, 2024-04-23T10:32:04.936 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 42.436658680000001, 2024-04-23T10:32:04.937 INFO:teuthology.orchestra.run.smithi114.stdout: "continuous": false, 2024-04-23T10:32:04.937 INFO:teuthology.orchestra.run.smithi114.stdout: "type_data": { 2024-04-23T10:32:04.937 INFO:teuthology.orchestra.run.smithi114.stdout: "result": -2147483648, 2024-04-23T10:32:04.937 INFO:teuthology.orchestra.run.smithi114.stdout: "flag_point": "dispatched", 2024-04-23T10:32:04.937 INFO:teuthology.orchestra.run.smithi114.stdout: "reqid": { 2024-04-23T10:32:04.937 INFO:teuthology.orchestra.run.smithi114.stdout: "entity": { 2024-04-23T10:32:04.937 INFO:teuthology.orchestra.run.smithi114.stdout: "type": "mds", 2024-04-23T10:32:04.937 INFO:teuthology.orchestra.run.smithi114.stdout: "num": 1 2024-04-23T10:32:04.937 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-23T10:32:04.937 INFO:teuthology.orchestra.run.smithi114.stdout: "tid": 109 2024-04-23T10:32:04.937 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-23T10:32:04.937 INFO:teuthology.orchestra.run.smithi114.stdout: "op_type": "peer_request", 2024-04-23T10:32:04.937 INFO:teuthology.orchestra.run.smithi114.stdout: "leader_info": { 2024-04-23T10:32:04.937 INFO:teuthology.orchestra.run.smithi114.stdout: "leader": "1" 2024-04-23T10:32:04.938 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-23T10:32:04.938 INFO:teuthology.orchestra.run.smithi114.stdout: "events": [ 2024-04-23T10:32:04.938 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-04-23T10:32:04.938 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-04-23T10:31:21.452180+0000", 2024-04-23T10:32:04.938 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "initiated" 2024-04-23T10:32:04.938 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-23T10:32:04.938 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-04-23T10:32:04.938 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-04-23T10:31:21.452180+0000", 2024-04-23T10:32:04.938 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "header_read" 2024-04-23T10:32:04.938 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-23T10:32:04.938 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-04-23T10:32:04.938 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-04-23T10:31:21.452180+0000", 2024-04-23T10:32:04.938 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "throttled" 2024-04-23T10:32:04.938 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-23T10:32:04.939 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-04-23T10:32:04.939 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-04-23T10:31:21.452187+0000", 2024-04-23T10:32:04.939 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "all_read" 2024-04-23T10:32:04.939 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-23T10:32:04.939 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-04-23T10:32:04.939 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-04-23T10:31:21.453429+0000", 2024-04-23T10:32:04.939 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "dispatched" 2024-04-23T10:32:04.939 INFO:teuthology.orchestra.run.smithi114.stdout: } 2024-04-23T10:32:04.939 INFO:teuthology.orchestra.run.smithi114.stdout: ], 2024-04-23T10:32:04.939 INFO:teuthology.orchestra.run.smithi114.stdout: "locks": [] 2024-04-23T10:32:04.939 INFO:teuthology.orchestra.run.smithi114.stdout: } 2024-04-23T10:32:04.939 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-23T10:32:04.939 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-04-23T10:32:04.939 INFO:teuthology.orchestra.run.smithi114.stdout: "description": "internal op quiesce_inode:mds.0:183 fp=#0x10000000241", 2024-04-23T10:32:04.939 INFO:teuthology.orchestra.run.smithi114.stdout: "initiated_at": "2024-04-23T10:31:21.469882+0000", 2024-04-23T10:32:04.940 INFO:teuthology.orchestra.run.smithi114.stdout: "age": 42.414506783, 2024-04-23T10:32:04.940 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 42.418976442000002, 2024-04-23T10:32:04.940 INFO:teuthology.orchestra.run.smithi114.stdout: "continuous": true, 2024-04-23T10:32:04.940 INFO:teuthology.orchestra.run.smithi114.stdout: "type_data": { 2024-04-23T10:32:04.940 INFO:teuthology.orchestra.run.smithi114.stdout: "result": -2147483648, 2024-04-23T10:32:04.940 INFO:teuthology.orchestra.run.smithi114.stdout: "flag_point": "quiesce complete", 2024-04-23T10:32:04.940 INFO:teuthology.orchestra.run.smithi114.stdout: "reqid": { 2024-04-23T10:32:04.940 INFO:teuthology.orchestra.run.smithi114.stdout: "entity": { 2024-04-23T10:32:04.940 INFO:teuthology.orchestra.run.smithi114.stdout: "type": "mds", 2024-04-23T10:32:04.940 INFO:teuthology.orchestra.run.smithi114.stdout: "num": 0 2024-04-23T10:32:04.940 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-23T10:32:04.940 INFO:teuthology.orchestra.run.smithi114.stdout: "tid": 183 2024-04-23T10:32:04.940 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-23T10:32:04.940 INFO:teuthology.orchestra.run.smithi114.stdout: "op_type": "internal_op", 2024-04-23T10:32:04.941 INFO:teuthology.orchestra.run.smithi114.stdout: "internal_op": 5385, 2024-04-23T10:32:04.941 INFO:teuthology.orchestra.run.smithi114.stdout: "op_name": "quiesce_inode", 2024-04-23T10:32:04.941 INFO:teuthology.orchestra.run.smithi114.stdout: "events": [ 2024-04-23T10:32:04.941 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-04-23T10:32:04.941 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-04-23T10:31:21.469882+0000", 2024-04-23T10:32:04.941 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "initiated" 2024-04-23T10:32:04.941 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-23T10:32:04.941 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-04-23T10:32:04.941 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-04-23T10:31:21.469882+0000", 2024-04-23T10:32:04.941 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "header_read" 2024-04-23T10:32:04.941 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-23T10:32:04.941 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-04-23T10:32:04.941 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-04-23T10:31:21.469882+0000", 2024-04-23T10:32:04.941 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "throttled" 2024-04-23T10:32:04.942 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-23T10:32:04.942 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-04-23T10:32:04.942 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-04-23T10:31:21.469882+0000", 2024-04-23T10:32:04.942 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "all_read" 2024-04-23T10:32:04.942 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-23T10:32:04.942 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-04-23T10:32:04.942 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-04-23T10:31:21.469882+0000", 2024-04-23T10:32:04.942 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "dispatched" 2024-04-23T10:32:04.942 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-23T10:32:04.942 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-04-23T10:32:04.942 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-04-23T10:31:21.470153+0000", 2024-04-23T10:32:04.942 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "failed to xlock, waiting" 2024-04-23T10:32:04.942 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-23T10:32:04.942 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-04-23T10:32:04.943 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-04-23T10:31:21.508862+0000", 2024-04-23T10:32:04.943 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "acquired locks" 2024-04-23T10:32:04.943 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-23T10:32:04.943 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-04-23T10:32:04.943 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-04-23T10:31:21.509069+0000", 2024-04-23T10:32:04.943 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "quiesce complete" 2024-04-23T10:32:04.943 INFO:teuthology.orchestra.run.smithi114.stdout: } 2024-04-23T10:32:04.943 INFO:teuthology.orchestra.run.smithi114.stdout: ], 2024-04-23T10:32:04.943 INFO:teuthology.orchestra.run.smithi114.stdout: "locks": [ 2024-04-23T10:32:04.943 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-04-23T10:32:04.943 INFO:teuthology.orchestra.run.smithi114.stdout: "object": { 2024-04-23T10:32:04.943 INFO:teuthology.orchestra.run.smithi114.stdout: "is_auth": true, 2024-04-23T10:32:04.943 INFO:teuthology.orchestra.run.smithi114.stdout: "auth_state": { 2024-04-23T10:32:04.943 INFO:teuthology.orchestra.run.smithi114.stdout: "replicas": { 2024-04-23T10:32:04.943 INFO:teuthology.orchestra.run.smithi114.stdout: "2": 1 2024-04-23T10:32:04.944 INFO:teuthology.orchestra.run.smithi114.stdout: } 2024-04-23T10:32:04.944 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-23T10:32:04.944 INFO:teuthology.orchestra.run.smithi114.stdout: "replica_state": { 2024-04-23T10:32:04.944 INFO:teuthology.orchestra.run.smithi114.stdout: "authority": [ 2024-04-23T10:32:04.944 INFO:teuthology.orchestra.run.smithi114.stdout: 0, 2024-04-23T10:32:04.944 INFO:teuthology.orchestra.run.smithi114.stdout: -2 2024-04-23T10:32:04.944 INFO:teuthology.orchestra.run.smithi114.stdout: ], 2024-04-23T10:32:04.944 INFO:teuthology.orchestra.run.smithi114.stdout: "replica_nonce": 0 2024-04-23T10:32:04.944 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-23T10:32:04.944 INFO:teuthology.orchestra.run.smithi114.stdout: "auth_pins": 2, 2024-04-23T10:32:04.944 INFO:teuthology.orchestra.run.smithi114.stdout: "is_frozen": false, 2024-04-23T10:32:04.944 INFO:teuthology.orchestra.run.smithi114.stdout: "is_freezing": false, 2024-04-23T10:32:04.944 INFO:teuthology.orchestra.run.smithi114.stdout: "pins": { 2024-04-23T10:32:04.944 INFO:teuthology.orchestra.run.smithi114.stdout: "ptrwaiter": 0, 2024-04-23T10:32:04.945 INFO:teuthology.orchestra.run.smithi114.stdout: "request": 1, 2024-04-23T10:32:04.945 INFO:teuthology.orchestra.run.smithi114.stdout: "lock": 1, 2024-04-23T10:32:04.945 INFO:teuthology.orchestra.run.smithi114.stdout: "importingcaps": 0, 2024-04-23T10:32:04.945 INFO:teuthology.orchestra.run.smithi114.stdout: "caps": 0, 2024-04-23T10:32:04.945 INFO:teuthology.orchestra.run.smithi114.stdout: "replicated": 1, 2024-04-23T10:32:04.945 INFO:teuthology.orchestra.run.smithi114.stdout: "dirty": 0, 2024-04-23T10:32:04.945 INFO:teuthology.orchestra.run.smithi114.stdout: "waiter": 0, 2024-04-23T10:32:04.945 INFO:teuthology.orchestra.run.smithi114.stdout: "authpin": 1 2024-04-23T10:32:04.945 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-23T10:32:04.945 INFO:teuthology.orchestra.run.smithi114.stdout: "nref": 4 2024-04-23T10:32:04.945 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-23T10:32:04.945 INFO:teuthology.orchestra.run.smithi114.stdout: "object_string": "[inode 0x10000000241 [7,head] /volumes/qa/sv_1/b206794e-7622-4ab8-8943-f3ae4e65b7a3/client.0/tmp/clients/client0/~dmtmp/SEED/SMALL.FIL auth{2=1} v646 ap=2 s=327680 n(v0 rc2024-04-23T10:31:17.226728+0000 b327680 1=1+0) (iquiesce lock x=1 by request(mds.0:183 nref=3)) | ptrwaiter=0 request=1 lock=1 importingcaps=0 caps=0 replicated=1 dirty=0 waiter=0 authpin=1 0x5571cf30f180]", 2024-04-23T10:32:04.945 INFO:teuthology.orchestra.run.smithi114.stdout: "lock": { 2024-04-23T10:32:04.945 INFO:teuthology.orchestra.run.smithi114.stdout: "gather_set": [], 2024-04-23T10:32:04.945 INFO:teuthology.orchestra.run.smithi114.stdout: "state": "lock", 2024-04-23T10:32:04.946 INFO:teuthology.orchestra.run.smithi114.stdout: "type": "iquiesce", 2024-04-23T10:32:04.946 INFO:teuthology.orchestra.run.smithi114.stdout: "is_leased": false, 2024-04-23T10:32:04.946 INFO:teuthology.orchestra.run.smithi114.stdout: "num_rdlocks": 0, 2024-04-23T10:32:04.946 INFO:teuthology.orchestra.run.smithi114.stdout: "num_wrlocks": 0, 2024-04-23T10:32:04.946 INFO:teuthology.orchestra.run.smithi114.stdout: "num_xlocks": 1, 2024-04-23T10:32:04.946 INFO:teuthology.orchestra.run.smithi114.stdout: "xlock_by": { 2024-04-23T10:32:04.946 INFO:teuthology.orchestra.run.smithi114.stdout: "reqid": { 2024-04-23T10:32:04.946 INFO:teuthology.orchestra.run.smithi114.stdout: "entity": { 2024-04-23T10:32:04.946 INFO:teuthology.orchestra.run.smithi114.stdout: "type": "mds", 2024-04-23T10:32:04.946 INFO:teuthology.orchestra.run.smithi114.stdout: "num": 0 2024-04-23T10:32:04.946 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-23T10:32:04.946 INFO:teuthology.orchestra.run.smithi114.stdout: "tid": 183 2024-04-23T10:32:04.946 INFO:teuthology.orchestra.run.smithi114.stdout: } 2024-04-23T10:32:04.946 INFO:teuthology.orchestra.run.smithi114.stdout: } 2024-04-23T10:32:04.947 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-23T10:32:04.947 INFO:teuthology.orchestra.run.smithi114.stdout: "flags": 4, 2024-04-23T10:32:04.947 INFO:teuthology.orchestra.run.smithi114.stdout: "wrlock_target": -1 2024-04-23T10:32:04.947 INFO:teuthology.orchestra.run.smithi114.stdout: } 2024-04-23T10:32:04.947 INFO:teuthology.orchestra.run.smithi114.stdout: ] 2024-04-23T10:32:04.947 INFO:teuthology.orchestra.run.smithi114.stdout: } 2024-04-23T10:32:04.947 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-23T10:32:04.947 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-04-23T10:32:04.947 INFO:teuthology.orchestra.run.smithi114.stdout: "description": "peer_request:mds.2:124", 2024-04-23T10:32:04.947 INFO:teuthology.orchestra.run.smithi114.stdout: "initiated_at": "2024-04-23T10:31:21.463891+0000", 2024-04-23T10:32:04.947 INFO:teuthology.orchestra.run.smithi114.stdout: "age": 42.420498346999999, 2024-04-23T10:32:04.947 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 42.425028378, 2024-04-23T10:32:04.947 INFO:teuthology.orchestra.run.smithi114.stdout: "continuous": false, 2024-04-23T10:32:04.948 INFO:teuthology.orchestra.run.smithi114.stdout: "type_data": { 2024-04-23T10:32:04.948 INFO:teuthology.orchestra.run.smithi114.stdout: "result": -2147483648, 2024-04-23T10:32:04.948 INFO:teuthology.orchestra.run.smithi114.stdout: "flag_point": "dispatched", 2024-04-23T10:32:04.948 INFO:teuthology.orchestra.run.smithi114.stdout: "reqid": { 2024-04-23T10:32:04.948 INFO:teuthology.orchestra.run.smithi114.stdout: "entity": { 2024-04-23T10:32:04.948 INFO:teuthology.orchestra.run.smithi114.stdout: "type": "mds", 2024-04-23T10:32:04.948 INFO:teuthology.orchestra.run.smithi114.stdout: "num": 2 2024-04-23T10:32:04.948 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-23T10:32:04.948 INFO:teuthology.orchestra.run.smithi114.stdout: "tid": 124 2024-04-23T10:32:04.948 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-23T10:32:04.948 INFO:teuthology.orchestra.run.smithi114.stdout: "op_type": "peer_request", 2024-04-23T10:32:04.948 INFO:teuthology.orchestra.run.smithi114.stdout: "leader_info": { 2024-04-23T10:32:04.948 INFO:teuthology.orchestra.run.smithi114.stdout: "leader": "2" 2024-04-23T10:32:04.948 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-23T10:32:04.948 INFO:teuthology.orchestra.run.smithi114.stdout: "events": [ 2024-04-23T10:32:04.949 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-04-23T10:32:04.949 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-04-23T10:31:21.463891+0000", 2024-04-23T10:32:04.949 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "initiated" 2024-04-23T10:32:04.949 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-23T10:32:04.949 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-04-23T10:32:04.949 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-04-23T10:31:21.463891+0000", 2024-04-23T10:32:04.949 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "header_read" 2024-04-23T10:32:04.949 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-23T10:32:04.949 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-04-23T10:32:04.949 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-04-23T10:31:21.463891+0000", 2024-04-23T10:32:04.949 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "throttled" 2024-04-23T10:32:04.949 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-23T10:32:04.949 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-04-23T10:32:04.949 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-04-23T10:31:21.463896+0000", 2024-04-23T10:32:04.950 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "all_read" 2024-04-23T10:32:04.950 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-23T10:32:04.950 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-04-23T10:32:04.950 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-04-23T10:31:21.495370+0000", 2024-04-23T10:32:04.950 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "dispatched" 2024-04-23T10:32:04.950 INFO:teuthology.orchestra.run.smithi114.stdout: } 2024-04-23T10:32:04.950 INFO:teuthology.orchestra.run.smithi114.stdout: ], 2024-04-23T10:32:04.950 INFO:teuthology.orchestra.run.smithi114.stdout: "locks": [] 2024-04-23T10:32:04.950 INFO:teuthology.orchestra.run.smithi114.stdout: } 2024-04-23T10:32:04.950 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-23T10:32:04.950 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-04-23T10:32:04.950 INFO:teuthology.orchestra.run.smithi114.stdout: "description": "internal op quiesce_inode:mds.0:158 fp=#0x10000000007", 2024-04-23T10:32:04.950 INFO:teuthology.orchestra.run.smithi114.stdout: "initiated_at": "2024-04-23T10:31:21.454050+0000", 2024-04-23T10:32:04.950 INFO:teuthology.orchestra.run.smithi114.stdout: "age": 42.430339050999997, 2024-04-23T10:32:04.951 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 42.434891911999998, 2024-04-23T10:32:04.951 INFO:teuthology.orchestra.run.smithi114.stdout: "continuous": true, 2024-04-23T10:32:04.951 INFO:teuthology.orchestra.run.smithi114.stdout: "type_data": { 2024-04-23T10:32:04.951 INFO:teuthology.orchestra.run.smithi114.stdout: "result": -2147483648, 2024-04-23T10:32:04.951 INFO:teuthology.orchestra.run.smithi114.stdout: "flag_point": "quiesce complete", 2024-04-23T10:32:04.951 INFO:teuthology.orchestra.run.smithi114.stdout: "reqid": { 2024-04-23T10:32:04.951 INFO:teuthology.orchestra.run.smithi114.stdout: "entity": { 2024-04-23T10:32:04.951 INFO:teuthology.orchestra.run.smithi114.stdout: "type": "mds", 2024-04-23T10:32:04.951 INFO:teuthology.orchestra.run.smithi114.stdout: "num": 0 2024-04-23T10:32:04.951 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-23T10:32:04.951 INFO:teuthology.orchestra.run.smithi114.stdout: "tid": 158 2024-04-23T10:32:04.951 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-23T10:32:04.951 INFO:teuthology.orchestra.run.smithi114.stdout: "op_type": "internal_op", 2024-04-23T10:32:04.951 INFO:teuthology.orchestra.run.smithi114.stdout: "internal_op": 5385, 2024-04-23T10:32:04.951 INFO:teuthology.orchestra.run.smithi114.stdout: "op_name": "quiesce_inode", 2024-04-23T10:32:04.952 INFO:teuthology.orchestra.run.smithi114.stdout: "events": [ 2024-04-23T10:32:04.952 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-04-23T10:32:04.952 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-04-23T10:31:21.454050+0000", 2024-04-23T10:32:04.952 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "initiated" 2024-04-23T10:32:04.952 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-23T10:32:04.952 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-04-23T10:32:04.952 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-04-23T10:31:21.454050+0000", 2024-04-23T10:32:04.952 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "header_read" 2024-04-23T10:32:04.952 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-23T10:32:04.952 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-04-23T10:32:04.952 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-04-23T10:31:21.454050+0000", 2024-04-23T10:32:04.952 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "throttled" 2024-04-23T10:32:04.952 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-23T10:32:04.952 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-04-23T10:32:04.952 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-04-23T10:31:21.454050+0000", 2024-04-23T10:32:04.953 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "all_read" 2024-04-23T10:32:04.953 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-23T10:32:04.953 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-04-23T10:32:04.953 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-04-23T10:31:21.454050+0000", 2024-04-23T10:32:04.953 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "dispatched" 2024-04-23T10:32:04.953 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-23T10:32:04.953 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-04-23T10:32:04.953 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-04-23T10:31:21.454326+0000", 2024-04-23T10:32:04.953 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "failed to xlock, waiting" 2024-04-23T10:32:04.953 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-23T10:32:04.953 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-04-23T10:32:04.953 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-04-23T10:31:21.455522+0000", 2024-04-23T10:32:04.953 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "acquired locks" 2024-04-23T10:32:04.953 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-23T10:32:04.953 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-04-23T10:32:04.954 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-04-23T10:31:21.509882+0000", 2024-04-23T10:32:04.954 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "quiesce complete" 2024-04-23T10:32:04.954 INFO:teuthology.orchestra.run.smithi114.stdout: } 2024-04-23T10:32:04.954 INFO:teuthology.orchestra.run.smithi114.stdout: ], 2024-04-23T10:32:04.954 INFO:teuthology.orchestra.run.smithi114.stdout: "locks": [ 2024-04-23T10:32:04.954 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-04-23T10:32:04.954 INFO:teuthology.orchestra.run.smithi114.stdout: "object": { 2024-04-23T10:32:04.954 INFO:teuthology.orchestra.run.smithi114.stdout: "is_auth": true, 2024-04-23T10:32:04.954 INFO:teuthology.orchestra.run.smithi114.stdout: "auth_state": { 2024-04-23T10:32:04.954 INFO:teuthology.orchestra.run.smithi114.stdout: "replicas": { 2024-04-23T10:32:04.954 INFO:teuthology.orchestra.run.smithi114.stdout: "1": 1, 2024-04-23T10:32:04.954 INFO:teuthology.orchestra.run.smithi114.stdout: "2": 1 2024-04-23T10:32:04.954 INFO:teuthology.orchestra.run.smithi114.stdout: } 2024-04-23T10:32:04.954 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-23T10:32:04.955 INFO:teuthology.orchestra.run.smithi114.stdout: "replica_state": { 2024-04-23T10:32:04.955 INFO:teuthology.orchestra.run.smithi114.stdout: "authority": [ 2024-04-23T10:32:04.955 INFO:teuthology.orchestra.run.smithi114.stdout: 0, 2024-04-23T10:32:04.955 INFO:teuthology.orchestra.run.smithi114.stdout: -2 2024-04-23T10:32:04.955 INFO:teuthology.orchestra.run.smithi114.stdout: ], 2024-04-23T10:32:04.955 INFO:teuthology.orchestra.run.smithi114.stdout: "replica_nonce": 0 2024-04-23T10:32:04.955 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-23T10:32:04.955 INFO:teuthology.orchestra.run.smithi114.stdout: "auth_pins": 3, 2024-04-23T10:32:04.955 INFO:teuthology.orchestra.run.smithi114.stdout: "is_frozen": false, 2024-04-23T10:32:04.955 INFO:teuthology.orchestra.run.smithi114.stdout: "is_freezing": false, 2024-04-23T10:32:04.955 INFO:teuthology.orchestra.run.smithi114.stdout: "pins": { 2024-04-23T10:32:04.955 INFO:teuthology.orchestra.run.smithi114.stdout: "dirtyscattered": 0, 2024-04-23T10:32:04.955 INFO:teuthology.orchestra.run.smithi114.stdout: "request": 1, 2024-04-23T10:32:04.955 INFO:teuthology.orchestra.run.smithi114.stdout: "lock": 1, 2024-04-23T10:32:04.955 INFO:teuthology.orchestra.run.smithi114.stdout: "dirfrag": 1, 2024-04-23T10:32:04.955 INFO:teuthology.orchestra.run.smithi114.stdout: "caps": 1, 2024-04-23T10:32:04.956 INFO:teuthology.orchestra.run.smithi114.stdout: "dirtyrstat": 0, 2024-04-23T10:32:04.956 INFO:teuthology.orchestra.run.smithi114.stdout: "dirtyparent": 0, 2024-04-23T10:32:04.956 INFO:teuthology.orchestra.run.smithi114.stdout: "replicated": 1, 2024-04-23T10:32:04.956 INFO:teuthology.orchestra.run.smithi114.stdout: "dirty": 1, 2024-04-23T10:32:04.956 INFO:teuthology.orchestra.run.smithi114.stdout: "waiter": 0, 2024-04-23T10:32:04.956 INFO:teuthology.orchestra.run.smithi114.stdout: "authpin": 1 2024-04-23T10:32:04.956 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-23T10:32:04.956 INFO:teuthology.orchestra.run.smithi114.stdout: "nref": 7 2024-04-23T10:32:04.956 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-23T10:32:04.956 INFO:teuthology.orchestra.run.smithi114.stdout: "object_string": "[inode 0x10000000007 [...7,head] /volumes/qa/sv_1/b206794e-7622-4ab8-8943-f3ae4e65b7a3/client.0/tmp/clients/ auth{1=1,2=1} v419 ap=3 f(v0 m2024-04-23T10:27:51.919399+0000 1=0+1) n(v30 rc2024-04-23T10:31:17.764716+0000 b21075074 75=63+12) old_inodes=4 (inest mix) (iquiesce lock x=1 by request(mds.0:158 nref=3)) caps={15312=pAsLsXsFs/pAsLsXsFs@13},l=15312 | dirtyscattered=0 request=1 lock=1 dirfrag=1 caps=1 dirtyrstat=0 dirtyparent=0 replicated=1 dirty=1 waiter=0 authpin=1 0x5571cf1de580]", 2024-04-23T10:32:04.956 INFO:teuthology.orchestra.run.smithi114.stdout: "lock": { 2024-04-23T10:32:04.956 INFO:teuthology.orchestra.run.smithi114.stdout: "gather_set": [], 2024-04-23T10:32:04.956 INFO:teuthology.orchestra.run.smithi114.stdout: "state": "lock", 2024-04-23T10:32:04.956 INFO:teuthology.orchestra.run.smithi114.stdout: "type": "iquiesce", 2024-04-23T10:32:04.957 INFO:teuthology.orchestra.run.smithi114.stdout: "is_leased": false, 2024-04-23T10:32:04.957 INFO:teuthology.orchestra.run.smithi114.stdout: "num_rdlocks": 0, 2024-04-23T10:32:04.957 INFO:teuthology.orchestra.run.smithi114.stdout: "num_wrlocks": 0, 2024-04-23T10:32:04.957 INFO:teuthology.orchestra.run.smithi114.stdout: "num_xlocks": 1, 2024-04-23T10:32:04.957 INFO:teuthology.orchestra.run.smithi114.stdout: "xlock_by": { 2024-04-23T10:32:04.957 INFO:teuthology.orchestra.run.smithi114.stdout: "reqid": { 2024-04-23T10:32:04.957 INFO:teuthology.orchestra.run.smithi114.stdout: "entity": { 2024-04-23T10:32:04.957 INFO:teuthology.orchestra.run.smithi114.stdout: "type": "mds", 2024-04-23T10:32:04.957 INFO:teuthology.orchestra.run.smithi114.stdout: "num": 0 2024-04-23T10:32:04.957 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-23T10:32:04.957 INFO:teuthology.orchestra.run.smithi114.stdout: "tid": 158 2024-04-23T10:32:04.957 INFO:teuthology.orchestra.run.smithi114.stdout: } 2024-04-23T10:32:04.957 INFO:teuthology.orchestra.run.smithi114.stdout: } 2024-04-23T10:32:04.957 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-23T10:32:04.957 INFO:teuthology.orchestra.run.smithi114.stdout: "flags": 4, 2024-04-23T10:32:04.958 INFO:teuthology.orchestra.run.smithi114.stdout: "wrlock_target": -1 2024-04-23T10:32:04.958 INFO:teuthology.orchestra.run.smithi114.stdout: } 2024-04-23T10:32:04.958 INFO:teuthology.orchestra.run.smithi114.stdout: ] 2024-04-23T10:32:04.958 INFO:teuthology.orchestra.run.smithi114.stdout: } 2024-04-23T10:32:04.958 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-23T10:32:04.958 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-04-23T10:32:04.958 INFO:teuthology.orchestra.run.smithi114.stdout: "description": "internal op quiesce_inode:mds.0:184 fp=#0x10000000242", 2024-04-23T10:32:04.958 INFO:teuthology.orchestra.run.smithi114.stdout: "initiated_at": "2024-04-23T10:31:21.469904+0000", 2024-04-23T10:32:04.958 INFO:teuthology.orchestra.run.smithi114.stdout: "age": 42.414485661000001, 2024-04-23T10:32:04.958 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 42.419104969000003, 2024-04-23T10:32:04.958 INFO:teuthology.orchestra.run.smithi114.stdout: "continuous": true, 2024-04-23T10:32:04.958 INFO:teuthology.orchestra.run.smithi114.stdout: "type_data": { 2024-04-23T10:32:04.958 INFO:teuthology.orchestra.run.smithi114.stdout: "result": -2147483648, 2024-04-23T10:32:04.959 INFO:teuthology.orchestra.run.smithi114.stdout: "flag_point": "quiesce complete", 2024-04-23T10:32:04.959 INFO:teuthology.orchestra.run.smithi114.stdout: "reqid": { 2024-04-23T10:32:04.959 INFO:teuthology.orchestra.run.smithi114.stdout: "entity": { 2024-04-23T10:32:04.959 INFO:teuthology.orchestra.run.smithi114.stdout: "type": "mds", 2024-04-23T10:32:04.959 INFO:teuthology.orchestra.run.smithi114.stdout: "num": 0 2024-04-23T10:32:04.959 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-23T10:32:04.959 INFO:teuthology.orchestra.run.smithi114.stdout: "tid": 184 2024-04-23T10:32:04.959 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-23T10:32:04.959 INFO:teuthology.orchestra.run.smithi114.stdout: "op_type": "internal_op", 2024-04-23T10:32:04.959 INFO:teuthology.orchestra.run.smithi114.stdout: "internal_op": 5385, 2024-04-23T10:32:04.959 INFO:teuthology.orchestra.run.smithi114.stdout: "op_name": "quiesce_inode", 2024-04-23T10:32:04.959 INFO:teuthology.orchestra.run.smithi114.stdout: "events": [ 2024-04-23T10:32:04.959 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-04-23T10:32:04.960 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-04-23T10:31:21.469904+0000", 2024-04-23T10:32:04.960 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "initiated" 2024-04-23T10:32:04.960 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-23T10:32:04.960 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-04-23T10:32:04.960 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-04-23T10:31:21.469904+0000", 2024-04-23T10:32:04.960 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "header_read" 2024-04-23T10:32:04.960 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-23T10:32:04.960 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-04-23T10:32:04.960 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-04-23T10:31:21.469904+0000", 2024-04-23T10:32:04.960 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "throttled" 2024-04-23T10:32:04.960 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-23T10:32:04.960 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-04-23T10:32:04.960 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-04-23T10:31:21.469904+0000", 2024-04-23T10:32:04.960 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "all_read" 2024-04-23T10:32:04.961 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-23T10:32:04.961 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-04-23T10:32:04.961 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-04-23T10:31:21.469904+0000", 2024-04-23T10:32:04.961 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "dispatched" 2024-04-23T10:32:04.961 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-23T10:32:04.961 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-04-23T10:32:04.961 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-04-23T10:31:21.470297+0000", 2024-04-23T10:32:04.961 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "failed to xlock, waiting" 2024-04-23T10:32:04.961 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-23T10:32:04.961 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-04-23T10:32:04.961 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-04-23T10:31:21.495245+0000", 2024-04-23T10:32:04.961 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "failed to xlock, waiting" 2024-04-23T10:32:04.961 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-23T10:32:04.961 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-04-23T10:32:04.961 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-04-23T10:31:21.500681+0000", 2024-04-23T10:32:04.962 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "acquired locks" 2024-04-23T10:32:04.962 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-23T10:32:04.962 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-04-23T10:32:04.962 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-04-23T10:31:21.500803+0000", 2024-04-23T10:32:04.962 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "quiesce complete" 2024-04-23T10:32:04.962 INFO:teuthology.orchestra.run.smithi114.stdout: } 2024-04-23T10:32:04.962 INFO:teuthology.orchestra.run.smithi114.stdout: ], 2024-04-23T10:32:04.962 INFO:teuthology.orchestra.run.smithi114.stdout: "locks": [ 2024-04-23T10:32:04.962 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-04-23T10:32:04.962 INFO:teuthology.orchestra.run.smithi114.stdout: "object": { 2024-04-23T10:32:04.962 INFO:teuthology.orchestra.run.smithi114.stdout: "is_auth": true, 2024-04-23T10:32:04.962 INFO:teuthology.orchestra.run.smithi114.stdout: "auth_state": { 2024-04-23T10:32:04.962 INFO:teuthology.orchestra.run.smithi114.stdout: "replicas": { 2024-04-23T10:32:04.962 INFO:teuthology.orchestra.run.smithi114.stdout: "2": 1 2024-04-23T10:32:04.963 INFO:teuthology.orchestra.run.smithi114.stdout: } 2024-04-23T10:32:04.963 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-23T10:32:04.963 INFO:teuthology.orchestra.run.smithi114.stdout: "replica_state": { 2024-04-23T10:32:04.963 INFO:teuthology.orchestra.run.smithi114.stdout: "authority": [ 2024-04-23T10:32:04.963 INFO:teuthology.orchestra.run.smithi114.stdout: 0, 2024-04-23T10:32:04.963 INFO:teuthology.orchestra.run.smithi114.stdout: -2 2024-04-23T10:32:04.963 INFO:teuthology.orchestra.run.smithi114.stdout: ], 2024-04-23T10:32:04.963 INFO:teuthology.orchestra.run.smithi114.stdout: "replica_nonce": 0 2024-04-23T10:32:04.963 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-23T10:32:04.963 INFO:teuthology.orchestra.run.smithi114.stdout: "auth_pins": 2, 2024-04-23T10:32:04.963 INFO:teuthology.orchestra.run.smithi114.stdout: "is_frozen": false, 2024-04-23T10:32:04.963 INFO:teuthology.orchestra.run.smithi114.stdout: "is_freezing": false, 2024-04-23T10:32:04.963 INFO:teuthology.orchestra.run.smithi114.stdout: "pins": { 2024-04-23T10:32:04.963 INFO:teuthology.orchestra.run.smithi114.stdout: "ptrwaiter": 0, 2024-04-23T10:32:04.963 INFO:teuthology.orchestra.run.smithi114.stdout: "request": 1, 2024-04-23T10:32:04.964 INFO:teuthology.orchestra.run.smithi114.stdout: "lock": 1, 2024-04-23T10:32:04.964 INFO:teuthology.orchestra.run.smithi114.stdout: "importingcaps": 0, 2024-04-23T10:32:04.964 INFO:teuthology.orchestra.run.smithi114.stdout: "caps": 0, 2024-04-23T10:32:04.964 INFO:teuthology.orchestra.run.smithi114.stdout: "needsnapflush": 0, 2024-04-23T10:32:04.964 INFO:teuthology.orchestra.run.smithi114.stdout: "replicated": 1, 2024-04-23T10:32:04.964 INFO:teuthology.orchestra.run.smithi114.stdout: "dirty": 0, 2024-04-23T10:32:04.964 INFO:teuthology.orchestra.run.smithi114.stdout: "waiter": 0, 2024-04-23T10:32:04.964 INFO:teuthology.orchestra.run.smithi114.stdout: "authpin": 1 2024-04-23T10:32:04.964 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-23T10:32:04.964 INFO:teuthology.orchestra.run.smithi114.stdout: "nref": 4 2024-04-23T10:32:04.964 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-23T10:32:04.964 INFO:teuthology.orchestra.run.smithi114.stdout: "object_string": "[inode 0x10000000242 [7,head] /volumes/qa/sv_1/b206794e-7622-4ab8-8943-f3ae4e65b7a3/client.0/tmp/clients/client0/~dmtmp/SEED/MEDIUM.FIL auth{2=1} v650 ap=2 s=1048576 n(v0 rc2024-04-23T10:31:17.227728+0000 b1048576 1=1+0) (iquiesce lock x=1 by request(mds.0:184 nref=3)) | ptrwaiter=0 request=1 lock=1 importingcaps=0 caps=0 needsnapflush=0 replicated=1 dirty=0 waiter=0 authpin=1 0x5571cf5cec00]", 2024-04-23T10:32:04.964 INFO:teuthology.orchestra.run.smithi114.stdout: "lock": { 2024-04-23T10:32:04.964 INFO:teuthology.orchestra.run.smithi114.stdout: "gather_set": [], 2024-04-23T10:32:04.965 INFO:teuthology.orchestra.run.smithi114.stdout: "state": "lock", 2024-04-23T10:32:04.965 INFO:teuthology.orchestra.run.smithi114.stdout: "type": "iquiesce", 2024-04-23T10:32:04.965 INFO:teuthology.orchestra.run.smithi114.stdout: "is_leased": false, 2024-04-23T10:32:04.965 INFO:teuthology.orchestra.run.smithi114.stdout: "num_rdlocks": 0, 2024-04-23T10:32:04.965 INFO:teuthology.orchestra.run.smithi114.stdout: "num_wrlocks": 0, 2024-04-23T10:32:04.965 INFO:teuthology.orchestra.run.smithi114.stdout: "num_xlocks": 1, 2024-04-23T10:32:04.965 INFO:teuthology.orchestra.run.smithi114.stdout: "xlock_by": { 2024-04-23T10:32:04.965 INFO:teuthology.orchestra.run.smithi114.stdout: "reqid": { 2024-04-23T10:32:04.965 INFO:teuthology.orchestra.run.smithi114.stdout: "entity": { 2024-04-23T10:32:04.965 INFO:teuthology.orchestra.run.smithi114.stdout: "type": "mds", 2024-04-23T10:32:04.965 INFO:teuthology.orchestra.run.smithi114.stdout: "num": 0 2024-04-23T10:32:04.965 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-23T10:32:04.965 INFO:teuthology.orchestra.run.smithi114.stdout: "tid": 184 2024-04-23T10:32:04.965 INFO:teuthology.orchestra.run.smithi114.stdout: } 2024-04-23T10:32:04.965 INFO:teuthology.orchestra.run.smithi114.stdout: } 2024-04-23T10:32:04.965 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-23T10:32:04.966 INFO:teuthology.orchestra.run.smithi114.stdout: "flags": 4, 2024-04-23T10:32:04.966 INFO:teuthology.orchestra.run.smithi114.stdout: "wrlock_target": -1 2024-04-23T10:32:04.966 INFO:teuthology.orchestra.run.smithi114.stdout: } 2024-04-23T10:32:04.966 INFO:teuthology.orchestra.run.smithi114.stdout: ] 2024-04-23T10:32:04.966 INFO:teuthology.orchestra.run.smithi114.stdout: } 2024-04-23T10:32:04.966 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-23T10:32:04.966 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-04-23T10:32:04.966 INFO:teuthology.orchestra.run.smithi114.stdout: "description": "peer_request:mds.1:120", 2024-04-23T10:32:04.966 INFO:teuthology.orchestra.run.smithi114.stdout: "initiated_at": "2024-04-23T10:31:21.465132+0000", 2024-04-23T10:32:04.966 INFO:teuthology.orchestra.run.smithi114.stdout: "age": 42.419257543999997, 2024-04-23T10:32:04.966 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 42.423939896999997, 2024-04-23T10:32:04.966 INFO:teuthology.orchestra.run.smithi114.stdout: "continuous": false, 2024-04-23T10:32:04.966 INFO:teuthology.orchestra.run.smithi114.stdout: "type_data": { 2024-04-23T10:32:04.966 INFO:teuthology.orchestra.run.smithi114.stdout: "result": -2147483648, 2024-04-23T10:32:04.966 INFO:teuthology.orchestra.run.smithi114.stdout: "flag_point": "dispatched", 2024-04-23T10:32:04.967 INFO:teuthology.orchestra.run.smithi114.stdout: "reqid": { 2024-04-23T10:32:04.967 INFO:teuthology.orchestra.run.smithi114.stdout: "entity": { 2024-04-23T10:32:04.967 INFO:teuthology.orchestra.run.smithi114.stdout: "type": "mds", 2024-04-23T10:32:04.967 INFO:teuthology.orchestra.run.smithi114.stdout: "num": 1 2024-04-23T10:32:04.967 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-23T10:32:04.967 INFO:teuthology.orchestra.run.smithi114.stdout: "tid": 120 2024-04-23T10:32:04.967 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-23T10:32:04.967 INFO:teuthology.orchestra.run.smithi114.stdout: "op_type": "peer_request", 2024-04-23T10:32:04.967 INFO:teuthology.orchestra.run.smithi114.stdout: "leader_info": { 2024-04-23T10:32:04.967 INFO:teuthology.orchestra.run.smithi114.stdout: "leader": "1" 2024-04-23T10:32:04.967 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-23T10:32:04.967 INFO:teuthology.orchestra.run.smithi114.stdout: "events": [ 2024-04-23T10:32:04.967 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-04-23T10:32:04.967 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-04-23T10:31:21.465132+0000", 2024-04-23T10:32:04.967 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "initiated" 2024-04-23T10:32:04.967 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-23T10:32:04.968 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-04-23T10:32:04.968 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-04-23T10:31:21.465132+0000", 2024-04-23T10:32:04.968 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "header_read" 2024-04-23T10:32:04.968 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-23T10:32:04.968 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-04-23T10:32:04.968 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-04-23T10:31:21.465132+0000", 2024-04-23T10:32:04.968 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "throttled" 2024-04-23T10:32:04.968 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-23T10:32:04.968 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-04-23T10:32:04.968 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-04-23T10:31:21.465137+0000", 2024-04-23T10:32:04.968 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "all_read" 2024-04-23T10:32:04.968 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-23T10:32:04.968 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-04-23T10:32:04.968 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-04-23T10:31:21.496283+0000", 2024-04-23T10:32:04.968 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "dispatched" 2024-04-23T10:32:04.969 INFO:teuthology.orchestra.run.smithi114.stdout: } 2024-04-23T10:32:04.969 INFO:teuthology.orchestra.run.smithi114.stdout: ], 2024-04-23T10:32:04.969 INFO:teuthology.orchestra.run.smithi114.stdout: "locks": [] 2024-04-23T10:32:04.969 INFO:teuthology.orchestra.run.smithi114.stdout: } 2024-04-23T10:32:04.969 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-23T10:32:04.969 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-04-23T10:32:04.969 INFO:teuthology.orchestra.run.smithi114.stdout: "description": "peer_request:mds.2:114", 2024-04-23T10:32:04.969 INFO:teuthology.orchestra.run.smithi114.stdout: "initiated_at": "2024-04-23T10:31:21.453566+0000", 2024-04-23T10:32:04.969 INFO:teuthology.orchestra.run.smithi114.stdout: "age": 42.430822908000003, 2024-04-23T10:32:04.969 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 42.435527239000002, 2024-04-23T10:32:04.969 INFO:teuthology.orchestra.run.smithi114.stdout: "continuous": false, 2024-04-23T10:32:04.969 INFO:teuthology.orchestra.run.smithi114.stdout: "type_data": { 2024-04-23T10:32:04.969 INFO:teuthology.orchestra.run.smithi114.stdout: "result": -2147483648, 2024-04-23T10:32:04.969 INFO:teuthology.orchestra.run.smithi114.stdout: "flag_point": "dispatched", 2024-04-23T10:32:04.969 INFO:teuthology.orchestra.run.smithi114.stdout: "reqid": { 2024-04-23T10:32:04.970 INFO:teuthology.orchestra.run.smithi114.stdout: "entity": { 2024-04-23T10:32:04.970 INFO:teuthology.orchestra.run.smithi114.stdout: "type": "mds", 2024-04-23T10:32:04.970 INFO:teuthology.orchestra.run.smithi114.stdout: "num": 2 2024-04-23T10:32:04.970 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-23T10:32:04.970 INFO:teuthology.orchestra.run.smithi114.stdout: "tid": 114 2024-04-23T10:32:04.970 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-23T10:32:04.970 INFO:teuthology.orchestra.run.smithi114.stdout: "op_type": "peer_request", 2024-04-23T10:32:04.970 INFO:teuthology.orchestra.run.smithi114.stdout: "leader_info": { 2024-04-23T10:32:04.970 INFO:teuthology.orchestra.run.smithi114.stdout: "leader": "2" 2024-04-23T10:32:04.970 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-23T10:32:04.970 INFO:teuthology.orchestra.run.smithi114.stdout: "events": [ 2024-04-23T10:32:04.970 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-04-23T10:32:04.970 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-04-23T10:31:21.453566+0000", 2024-04-23T10:32:04.970 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "initiated" 2024-04-23T10:32:04.971 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-23T10:32:04.971 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-04-23T10:32:04.971 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-04-23T10:31:21.453566+0000", 2024-04-23T10:32:04.971 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "header_read" 2024-04-23T10:32:04.971 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-23T10:32:04.971 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-04-23T10:32:04.971 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-04-23T10:31:21.453567+0000", 2024-04-23T10:32:04.971 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "throttled" 2024-04-23T10:32:04.971 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-23T10:32:04.971 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-04-23T10:32:04.971 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-04-23T10:31:21.453575+0000", 2024-04-23T10:32:04.971 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "all_read" 2024-04-23T10:32:04.971 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-23T10:32:04.971 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-04-23T10:32:04.972 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-04-23T10:31:21.454920+0000", 2024-04-23T10:32:04.972 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "dispatched" 2024-04-23T10:32:04.972 INFO:teuthology.orchestra.run.smithi114.stdout: } 2024-04-23T10:32:04.972 INFO:teuthology.orchestra.run.smithi114.stdout: ], 2024-04-23T10:32:04.972 INFO:teuthology.orchestra.run.smithi114.stdout: "locks": [] 2024-04-23T10:32:04.972 INFO:teuthology.orchestra.run.smithi114.stdout: } 2024-04-23T10:32:04.972 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-23T10:32:04.972 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-04-23T10:32:04.972 INFO:teuthology.orchestra.run.smithi114.stdout: "description": "internal op quiesce_inode:mds.0:185 fp=#0x10000000243", 2024-04-23T10:32:04.972 INFO:teuthology.orchestra.run.smithi114.stdout: "initiated_at": "2024-04-23T10:31:21.469925+0000", 2024-04-23T10:32:04.972 INFO:teuthology.orchestra.run.smithi114.stdout: "age": 42.414464531, 2024-04-23T10:32:04.972 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 42.419189344000003, 2024-04-23T10:32:04.972 INFO:teuthology.orchestra.run.smithi114.stdout: "continuous": true, 2024-04-23T10:32:04.972 INFO:teuthology.orchestra.run.smithi114.stdout: "type_data": { 2024-04-23T10:32:04.972 INFO:teuthology.orchestra.run.smithi114.stdout: "result": -2147483648, 2024-04-23T10:32:04.973 INFO:teuthology.orchestra.run.smithi114.stdout: "flag_point": "quiesce complete", 2024-04-23T10:32:04.973 INFO:teuthology.orchestra.run.smithi114.stdout: "reqid": { 2024-04-23T10:32:04.973 INFO:teuthology.orchestra.run.smithi114.stdout: "entity": { 2024-04-23T10:32:04.973 INFO:teuthology.orchestra.run.smithi114.stdout: "type": "mds", 2024-04-23T10:32:04.973 INFO:teuthology.orchestra.run.smithi114.stdout: "num": 0 2024-04-23T10:32:04.973 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-23T10:32:04.973 INFO:teuthology.orchestra.run.smithi114.stdout: "tid": 185 2024-04-23T10:32:04.973 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-23T10:32:04.973 INFO:teuthology.orchestra.run.smithi114.stdout: "op_type": "internal_op", 2024-04-23T10:32:04.973 INFO:teuthology.orchestra.run.smithi114.stdout: "internal_op": 5385, 2024-04-23T10:32:04.973 INFO:teuthology.orchestra.run.smithi114.stdout: "op_name": "quiesce_inode", 2024-04-23T10:32:04.973 INFO:teuthology.orchestra.run.smithi114.stdout: "events": [ 2024-04-23T10:32:04.973 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-04-23T10:32:04.973 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-04-23T10:31:21.469925+0000", 2024-04-23T10:32:04.974 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "initiated" 2024-04-23T10:32:04.974 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-23T10:32:04.974 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-04-23T10:32:04.974 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-04-23T10:31:21.469925+0000", 2024-04-23T10:32:04.974 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "header_read" 2024-04-23T10:32:04.974 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-23T10:32:04.974 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-04-23T10:32:04.974 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-04-23T10:31:21.469925+0000", 2024-04-23T10:32:04.974 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "throttled" 2024-04-23T10:32:04.974 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-23T10:32:04.974 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-04-23T10:32:04.974 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-04-23T10:31:21.469925+0000", 2024-04-23T10:32:04.974 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "all_read" 2024-04-23T10:32:04.975 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-23T10:32:04.975 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-04-23T10:32:04.975 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-04-23T10:31:21.469925+0000", 2024-04-23T10:32:04.975 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "dispatched" 2024-04-23T10:32:04.975 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-23T10:32:04.975 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-04-23T10:32:04.975 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-04-23T10:31:21.470469+0000", 2024-04-23T10:32:04.975 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "failed to xlock, waiting" 2024-04-23T10:32:04.975 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-23T10:32:04.975 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-04-23T10:32:04.975 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-04-23T10:31:21.509433+0000", 2024-04-23T10:32:04.975 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "acquired locks" 2024-04-23T10:32:04.975 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-23T10:32:04.975 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-04-23T10:32:04.976 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-04-23T10:31:21.509734+0000", 2024-04-23T10:32:04.976 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "quiesce complete" 2024-04-23T10:32:04.976 INFO:teuthology.orchestra.run.smithi114.stdout: } 2024-04-23T10:32:04.976 INFO:teuthology.orchestra.run.smithi114.stdout: ], 2024-04-23T10:32:04.976 INFO:teuthology.orchestra.run.smithi114.stdout: "locks": [ 2024-04-23T10:32:04.976 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-04-23T10:32:04.976 INFO:teuthology.orchestra.run.smithi114.stdout: "object": { 2024-04-23T10:32:04.976 INFO:teuthology.orchestra.run.smithi114.stdout: "is_auth": true, 2024-04-23T10:32:04.976 INFO:teuthology.orchestra.run.smithi114.stdout: "auth_state": { 2024-04-23T10:32:04.976 INFO:teuthology.orchestra.run.smithi114.stdout: "replicas": { 2024-04-23T10:32:04.976 INFO:teuthology.orchestra.run.smithi114.stdout: "2": 1 2024-04-23T10:32:04.976 INFO:teuthology.orchestra.run.smithi114.stdout: } 2024-04-23T10:32:04.976 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-23T10:32:04.977 INFO:teuthology.orchestra.run.smithi114.stdout: "replica_state": { 2024-04-23T10:32:04.977 INFO:teuthology.orchestra.run.smithi114.stdout: "authority": [ 2024-04-23T10:32:04.977 INFO:teuthology.orchestra.run.smithi114.stdout: 0, 2024-04-23T10:32:04.977 INFO:teuthology.orchestra.run.smithi114.stdout: -2 2024-04-23T10:32:04.977 INFO:teuthology.orchestra.run.smithi114.stdout: ], 2024-04-23T10:32:04.977 INFO:teuthology.orchestra.run.smithi114.stdout: "replica_nonce": 0 2024-04-23T10:32:04.977 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-23T10:32:04.977 INFO:teuthology.orchestra.run.smithi114.stdout: "auth_pins": 2, 2024-04-23T10:32:04.977 INFO:teuthology.orchestra.run.smithi114.stdout: "is_frozen": false, 2024-04-23T10:32:04.977 INFO:teuthology.orchestra.run.smithi114.stdout: "is_freezing": false, 2024-04-23T10:32:04.977 INFO:teuthology.orchestra.run.smithi114.stdout: "pins": { 2024-04-23T10:32:04.977 INFO:teuthology.orchestra.run.smithi114.stdout: "ptrwaiter": 0, 2024-04-23T10:32:04.977 INFO:teuthology.orchestra.run.smithi114.stdout: "request": 1, 2024-04-23T10:32:04.977 INFO:teuthology.orchestra.run.smithi114.stdout: "lock": 1, 2024-04-23T10:32:04.978 INFO:teuthology.orchestra.run.smithi114.stdout: "importingcaps": 0, 2024-04-23T10:32:04.978 INFO:teuthology.orchestra.run.smithi114.stdout: "caps": 0, 2024-04-23T10:32:04.978 INFO:teuthology.orchestra.run.smithi114.stdout: "replicated": 1, 2024-04-23T10:32:04.978 INFO:teuthology.orchestra.run.smithi114.stdout: "dirty": 0, 2024-04-23T10:32:04.978 INFO:teuthology.orchestra.run.smithi114.stdout: "waiter": 0, 2024-04-23T10:32:04.978 INFO:teuthology.orchestra.run.smithi114.stdout: "authpin": 1 2024-04-23T10:32:04.978 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-23T10:32:04.978 INFO:teuthology.orchestra.run.smithi114.stdout: "nref": 4 2024-04-23T10:32:04.978 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-23T10:32:04.978 INFO:teuthology.orchestra.run.smithi114.stdout: "object_string": "[inode 0x10000000243 [7,head] /volumes/qa/sv_1/b206794e-7622-4ab8-8943-f3ae4e65b7a3/client.0/tmp/clients/client0/~dmtmp/SEED/LARGE.FIL auth{2=1} v648 ap=2 s=1867776 n(v0 rc2024-04-23T10:31:17.246728+0000 b1867776 1=1+0) (iquiesce lock x=1 by request(mds.0:185 nref=3)) | ptrwaiter=0 request=1 lock=1 importingcaps=0 caps=0 replicated=1 dirty=0 waiter=0 authpin=1 0x5571cf5d8b00]", 2024-04-23T10:32:04.978 INFO:teuthology.orchestra.run.smithi114.stdout: "lock": { 2024-04-23T10:32:04.978 INFO:teuthology.orchestra.run.smithi114.stdout: "gather_set": [], 2024-04-23T10:32:04.978 INFO:teuthology.orchestra.run.smithi114.stdout: "state": "lock", 2024-04-23T10:32:04.978 INFO:teuthology.orchestra.run.smithi114.stdout: "type": "iquiesce", 2024-04-23T10:32:04.979 INFO:teuthology.orchestra.run.smithi114.stdout: "is_leased": false, 2024-04-23T10:32:04.979 INFO:teuthology.orchestra.run.smithi114.stdout: "num_rdlocks": 0, 2024-04-23T10:32:04.979 INFO:teuthology.orchestra.run.smithi114.stdout: "num_wrlocks": 0, 2024-04-23T10:32:04.979 INFO:teuthology.orchestra.run.smithi114.stdout: "num_xlocks": 1, 2024-04-23T10:32:04.979 INFO:teuthology.orchestra.run.smithi114.stdout: "xlock_by": { 2024-04-23T10:32:04.979 INFO:teuthology.orchestra.run.smithi114.stdout: "reqid": { 2024-04-23T10:32:04.979 INFO:teuthology.orchestra.run.smithi114.stdout: "entity": { 2024-04-23T10:32:04.979 INFO:teuthology.orchestra.run.smithi114.stdout: "type": "mds", 2024-04-23T10:32:04.979 INFO:teuthology.orchestra.run.smithi114.stdout: "num": 0 2024-04-23T10:32:04.979 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-23T10:32:04.979 INFO:teuthology.orchestra.run.smithi114.stdout: "tid": 185 2024-04-23T10:32:04.979 INFO:teuthology.orchestra.run.smithi114.stdout: } 2024-04-23T10:32:04.979 INFO:teuthology.orchestra.run.smithi114.stdout: } 2024-04-23T10:32:04.979 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-23T10:32:04.979 INFO:teuthology.orchestra.run.smithi114.stdout: "flags": 4, 2024-04-23T10:32:04.980 INFO:teuthology.orchestra.run.smithi114.stdout: "wrlock_target": -1 2024-04-23T10:32:04.980 INFO:teuthology.orchestra.run.smithi114.stdout: } 2024-04-23T10:32:04.980 INFO:teuthology.orchestra.run.smithi114.stdout: ] 2024-04-23T10:32:04.980 INFO:teuthology.orchestra.run.smithi114.stdout: } 2024-04-23T10:32:04.980 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-23T10:32:04.980 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-04-23T10:32:04.980 INFO:teuthology.orchestra.run.smithi114.stdout: "description": "peer_request:mds.2:125", 2024-04-23T10:32:04.980 INFO:teuthology.orchestra.run.smithi114.stdout: "initiated_at": "2024-04-23T10:31:21.463955+0000", 2024-04-23T10:32:04.980 INFO:teuthology.orchestra.run.smithi114.stdout: "age": 42.420433768999999, 2024-04-23T10:32:04.980 INFO:teuthology.orchestra.run.smithi114.stdout: "duration": 42.425218538000003, 2024-04-23T10:32:04.980 INFO:teuthology.orchestra.run.smithi114.stdout: "continuous": false, 2024-04-23T10:32:04.980 INFO:teuthology.orchestra.run.smithi114.stdout: "type_data": { 2024-04-23T10:32:04.980 INFO:teuthology.orchestra.run.smithi114.stdout: "result": -2147483648, 2024-04-23T10:32:04.980 INFO:teuthology.orchestra.run.smithi114.stdout: "flag_point": "dispatched", 2024-04-23T10:32:04.980 INFO:teuthology.orchestra.run.smithi114.stdout: "reqid": { 2024-04-23T10:32:04.981 INFO:teuthology.orchestra.run.smithi114.stdout: "entity": { 2024-04-23T10:32:04.981 INFO:teuthology.orchestra.run.smithi114.stdout: "type": "mds", 2024-04-23T10:32:04.981 INFO:teuthology.orchestra.run.smithi114.stdout: "num": 2 2024-04-23T10:32:04.981 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-23T10:32:04.981 INFO:teuthology.orchestra.run.smithi114.stdout: "tid": 125 2024-04-23T10:32:04.981 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-23T10:32:04.981 INFO:teuthology.orchestra.run.smithi114.stdout: "op_type": "peer_request", 2024-04-23T10:32:04.981 INFO:teuthology.orchestra.run.smithi114.stdout: "leader_info": { 2024-04-23T10:32:04.981 INFO:teuthology.orchestra.run.smithi114.stdout: "leader": "2" 2024-04-23T10:32:04.981 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-23T10:32:04.981 INFO:teuthology.orchestra.run.smithi114.stdout: "events": [ 2024-04-23T10:32:04.981 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-04-23T10:32:04.981 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-04-23T10:31:21.463955+0000", 2024-04-23T10:32:04.981 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "initiated" 2024-04-23T10:32:04.982 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-23T10:32:04.982 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-04-23T10:32:04.982 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-04-23T10:31:21.463955+0000", 2024-04-23T10:32:04.982 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "header_read" 2024-04-23T10:32:04.982 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-23T10:32:04.982 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-04-23T10:32:04.982 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-04-23T10:31:21.463956+0000", 2024-04-23T10:32:04.982 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "throttled" 2024-04-23T10:32:04.982 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-23T10:32:04.982 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-04-23T10:32:04.982 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-04-23T10:31:21.463961+0000", 2024-04-23T10:32:04.982 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "all_read" 2024-04-23T10:32:04.982 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-23T10:32:04.982 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-04-23T10:32:04.982 INFO:teuthology.orchestra.run.smithi114.stdout: "time": "2024-04-23T10:31:21.496373+0000", 2024-04-23T10:32:04.983 INFO:teuthology.orchestra.run.smithi114.stdout: "event": "dispatched" 2024-04-23T10:32:04.983 INFO:teuthology.orchestra.run.smithi114.stdout: } 2024-04-23T10:32:04.983 INFO:teuthology.orchestra.run.smithi114.stdout: ], 2024-04-23T10:32:04.983 INFO:teuthology.orchestra.run.smithi114.stdout: "locks": [] 2024-04-23T10:32:04.983 INFO:teuthology.orchestra.run.smithi114.stdout: } 2024-04-23T10:32:04.983 INFO:teuthology.orchestra.run.smithi114.stdout: } 2024-04-23T10:32:04.983 INFO:teuthology.orchestra.run.smithi114.stdout: ], 2024-04-23T10:32:04.983 INFO:teuthology.orchestra.run.smithi114.stdout: "num_ops": 95 2024-04-23T10:32:04.983 INFO:teuthology.orchestra.run.smithi114.stdout:} 2024-04-23T10:32:04.986 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 23 10:32:04 smithi204 ceph-mon[32649]: pgmap v168: 97 pgs: 97 active+clean; 78 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 72 KiB/s rd, 786 KiB/s wr, 55 op/s 2024-04-23T10:32:04.986 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 23 10:32:04 smithi204 ceph-mon[32649]: from='client.? 172.21.15.114:0/2509710607' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-23T10:32:04.986 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 23 10:32:04 smithi119 ceph-mon[32608]: pgmap v168: 97 pgs: 97 active+clean; 78 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 72 KiB/s rd, 786 KiB/s wr, 55 op/s 2024-04-23T10:32:04.986 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 23 10:32:04 smithi119 ceph-mon[32608]: from='client.? 172.21.15.114:0/2509710607' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-23T10:32:04.996 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:32:03.891+0000 7f0fe81f4740 1 Processor -- start 2024-04-23T10:32:04.996 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:32:03.891+0000 7f0fe81f4740 1 -- start start 2024-04-23T10:32:04.996 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:32:03.891+0000 7f0fe81f4740 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55c8d0ee0ba0 0x55c8d0ee9e80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T10:32:04.996 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:32:03.891+0000 7f0fe81f4740 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x55c8d0ee14c0 0x55c8d0f02cc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T10:32:04.996 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:32:03.891+0000 7f0fe81f4740 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x55c8d0ef00a0 0x55c8d0f0aff0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T10:32:04.996 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:32:03.891+0000 7f0fe81f4740 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x55c8d0f68d70 con 0x55c8d0ee0ba0 2024-04-23T10:32:04.996 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:32:03.891+0000 7f0fe81f4740 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 -- 0x55c8d0f60e70 con 0x55c8d0ef00a0 2024-04-23T10:32:04.996 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:32:03.891+0000 7f0fe81f4740 1 -- --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_getmap magic: 0 -- 0x55c8d0ee0100 con 0x55c8d0ee14c0 2024-04-23T10:32:04.997 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:32:03.891+0000 7f0fdeffd640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55c8d0ee0ba0 0x55c8d0ee9e80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T10:32:04.997 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:32:03.891+0000 7f0fdeffd640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55c8d0ee0ba0 0x55c8d0ee9e80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.114:52088/0 (socket says 172.21.15.114:52088) 2024-04-23T10:32:04.997 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:32:03.891+0000 7f0fdeffd640 1 -- 172.21.15.114:0/3161633864 learned_addr learned my addr 172.21.15.114:0/3161633864 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-04-23T10:32:04.997 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:32:03.891+0000 7f0fde7fc640 1 --2- 172.21.15.114:0/3161633864 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x55c8d0ee14c0 0x55c8d0f02cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T10:32:04.997 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:32:03.891+0000 7f0fdeffd640 1 -- 172.21.15.114:0/3161633864 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x55c8d0ee14c0 msgr2=0x55c8d0f02cc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T10:32:04.997 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:32:03.891+0000 7f0fdeffd640 1 --2- 172.21.15.114:0/3161633864 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x55c8d0ee14c0 0x55c8d0f02cc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:32:04.997 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:32:03.891+0000 7f0fdf7fe640 1 --2- 172.21.15.114:0/3161633864 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x55c8d0ef00a0 0x55c8d0f0aff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T10:32:04.997 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:32:03.891+0000 7f0fdeffd640 1 -- 172.21.15.114:0/3161633864 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x55c8d0ef00a0 msgr2=0x55c8d0f0aff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T10:32:04.997 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:32:03.891+0000 7f0fdeffd640 1 --2- 172.21.15.114:0/3161633864 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x55c8d0ef00a0 0x55c8d0f0aff0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:32:04.997 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:32:03.891+0000 7f0fdeffd640 1 -- 172.21.15.114:0/3161633864 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55c8d0f679e0 con 0x55c8d0ee0ba0 2024-04-23T10:32:04.997 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:32:03.891+0000 7f0fde7fc640 1 --2- 172.21.15.114:0/3161633864 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x55c8d0ee14c0 0x55c8d0f02cc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-23T10:32:04.997 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:32:03.891+0000 7f0fdeffd640 1 --2- 172.21.15.114:0/3161633864 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55c8d0ee0ba0 0x55c8d0ee9e80 secure :-1 s=READY pgs=726 cs=0 l=1 rev1=1 crypto rx=0x7f0fd0061010 tx=0x7f0fd005f040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3a954648d21e6b7e server_cookie=0 in_seq=0 out_seq=0 2024-04-23T10:32:04.997 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:32:03.892+0000 7f0fc57fa640 1 -- 172.21.15.114:0/3161633864 <== mon.0 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0fd0077040 con 0x55c8d0ee0ba0 2024-04-23T10:32:04.997 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:32:03.892+0000 7f0fc57fa640 1 -- 172.21.15.114:0/3161633864 <== mon.0 v2:172.21.15.114:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f0fd0036040 con 0x55c8d0ee0ba0 2024-04-23T10:32:04.998 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:32:03.892+0000 7f0fc57fa640 1 -- 172.21.15.114:0/3161633864 <== mon.0 v2:172.21.15.114:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0fd0003c00 con 0x55c8d0ee0ba0 2024-04-23T10:32:04.998 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:32:03.892+0000 7f0fe81f4740 1 -- 172.21.15.114:0/3161633864 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55c8d0ee0ba0 msgr2=0x55c8d0ee9e80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T10:32:04.998 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:32:03.892+0000 7f0fe81f4740 1 --2- 172.21.15.114:0/3161633864 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55c8d0ee0ba0 0x55c8d0ee9e80 secure :-1 s=READY pgs=726 cs=0 l=1 rev1=1 crypto rx=0x7f0fd0061010 tx=0x7f0fd005f040 comp rx=0 tx=0).stop 2024-04-23T10:32:04.998 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:32:03.892+0000 7f0fe81f4740 1 -- 172.21.15.114:0/3161633864 shutdown_connections 2024-04-23T10:32:04.998 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:32:03.892+0000 7f0fe81f4740 1 --2- 172.21.15.114:0/3161633864 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x55c8d0ef00a0 0x55c8d0f0aff0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:32:04.998 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:32:03.892+0000 7f0fe81f4740 1 --2- 172.21.15.114:0/3161633864 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x55c8d0ee14c0 0x55c8d0f02cc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:32:04.998 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:32:03.892+0000 7f0fe81f4740 1 --2- 172.21.15.114:0/3161633864 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55c8d0ee0ba0 0x55c8d0ee9e80 unknown :-1 s=CLOSED pgs=726 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:32:04.998 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:32:03.892+0000 7f0fe81f4740 1 -- 172.21.15.114:0/3161633864 >> 172.21.15.114:0/3161633864 conn(0x55c8d0edcfc0 msgr2=0x55c8d0ee8460 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T10:32:04.998 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:32:03.892+0000 7f0fe81f4740 1 -- 172.21.15.114:0/3161633864 shutdown_connections 2024-04-23T10:32:04.998 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:32:03.892+0000 7f0fe81f4740 1 -- 172.21.15.114:0/3161633864 wait complete. 2024-04-23T10:32:04.998 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:32:03.892+0000 7f0fe81f4740 1 -- start start 2024-04-23T10:32:04.998 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:32:03.893+0000 7f0fe81f4740 1 Processor -- start 2024-04-23T10:32:04.998 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:32:03.893+0000 7f0fe81f4740 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x55c8d0ee0ba0 0x55c8d0ee9e80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T10:32:04.998 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:32:03.893+0000 7f0fe81f4740 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55c8d0ee14c0 0x55c8d0f02cc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T10:32:04.998 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:32:03.893+0000 7f0fe81f4740 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x55c8d0ef00a0 0x55c8d0eea600 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T10:32:04.999 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:32:03.893+0000 7f0fe81f4740 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x55c8d0ee9830 con 0x55c8d0ee14c0 2024-04-23T10:32:04.999 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:32:03.893+0000 7f0fe81f4740 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 -- 0x55c8d0f6ab70 con 0x55c8d0ee0ba0 2024-04-23T10:32:04.999 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:32:03.893+0000 7f0fe81f4740 1 -- --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_getmap magic: 0 -- 0x55c8d0edc4d0 con 0x55c8d0ef00a0 2024-04-23T10:32:04.999 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:32:03.893+0000 7f0fe81f4740 10 client.0 _ll_register_callbacks cb 0x7f0fd0005f60 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-23T10:32:04.999 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:32:03.893+0000 7f0fde7fc640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55c8d0ee14c0 0x55c8d0f02cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T10:32:04.999 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:32:03.893+0000 7f0fde7fc640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55c8d0ee14c0 0x55c8d0f02cc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.114:52102/0 (socket says 172.21.15.114:52102) 2024-04-23T10:32:04.999 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:32:03.893+0000 7f0fde7fc640 1 -- 172.21.15.114:0/4207429892 learned_addr learned my addr 172.21.15.114:0/4207429892 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-04-23T10:32:04.999 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:32:03.893+0000 7f0fdeffd640 1 --2- 172.21.15.114:0/4207429892 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x55c8d0ee0ba0 0x55c8d0ee9e80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T10:32:04.999 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:32:03.893+0000 7f0fdf7fe640 1 --2- 172.21.15.114:0/4207429892 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x55c8d0ef00a0 0x55c8d0eea600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T10:32:04.999 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:32:03.893+0000 7f0fde7fc640 1 -- 172.21.15.114:0/4207429892 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x55c8d0ef00a0 msgr2=0x55c8d0eea600 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T10:32:04.999 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:32:03.893+0000 7f0fde7fc640 1 --2- 172.21.15.114:0/4207429892 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x55c8d0ef00a0 0x55c8d0eea600 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:32:04.999 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:32:03.893+0000 7f0fde7fc640 1 -- 172.21.15.114:0/4207429892 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x55c8d0ee0ba0 msgr2=0x55c8d0ee9e80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T10:32:04.999 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:32:03.893+0000 7f0fde7fc640 1 --2- 172.21.15.114:0/4207429892 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x55c8d0ee0ba0 0x55c8d0ee9e80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:32:05.000 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:32:03.893+0000 7f0fde7fc640 1 -- 172.21.15.114:0/4207429892 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55c8d0f61fb0 con 0x55c8d0ee14c0 2024-04-23T10:32:05.000 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:32:03.893+0000 7f0fdf7fe640 1 --2- 172.21.15.114:0/4207429892 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x55c8d0ef00a0 0x55c8d0eea600 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-23T10:32:05.000 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:32:03.893+0000 7f0fdeffd640 1 --2- 172.21.15.114:0/4207429892 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x55c8d0ee0ba0 0x55c8d0ee9e80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-23T10:32:05.000 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:32:03.893+0000 7f0fde7fc640 1 --2- 172.21.15.114:0/4207429892 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55c8d0ee14c0 0x55c8d0f02cc0 secure :-1 s=READY pgs=727 cs=0 l=1 rev1=1 crypto rx=0x7f0fd4011cf0 tx=0x7f0fd4011fb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=418ba9c96a4238ee server_cookie=0 in_seq=0 out_seq=0 2024-04-23T10:32:05.000 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:32:03.893+0000 7f0fc2ff5640 10 client.0 ms_handle_connect on v2:172.21.15.114:3300/0 2024-04-23T10:32:05.000 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:32:03.893+0000 7f0fc2ff5640 1 -- 172.21.15.114:0/4207429892 <== mon.0 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0fd4064040 con 0x55c8d0ee14c0 2024-04-23T10:32:05.000 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:32:03.893+0000 7f0fc2ff5640 1 -- 172.21.15.114:0/4207429892 <== mon.0 v2:172.21.15.114:3300/0 2 ==== osd_map(94..94 src has 1..94) ==== 8585+0+0 (secure 0 0 0) 0x7f0fd405c270 con 0x55c8d0ee14c0 2024-04-23T10:32:05.000 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:32:03.893+0000 7f0fe81f4740 1 -- 172.21.15.114:0/4207429892 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f0fd0036040 con 0x55c8d0ee14c0 2024-04-23T10:32:05.000 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:32:03.893+0000 7f0fc2ff5640 1 -- 172.21.15.114:0/4207429892 <== mon.0 v2:172.21.15.114:3300/0 3 ==== mon_get_version_reply(handle=1 version=30) ==== 24+0+0 (secure 0 0 0) 0x7f0fd402f040 con 0x55c8d0ee14c0 2024-04-23T10:32:05.000 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:32:03.893+0000 7f0fe81f4740 10 client.16362 fetch_fsmap learned FSMap version 30 2024-04-23T10:32:05.000 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:32:03.893+0000 7f0fe81f4740 1 -- 172.21.15.114:0/4207429892 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({config=0+,fsmap=30,monmap=4+}) -- 0x55c8d0f61fb0 con 0x55c8d0ee14c0 2024-04-23T10:32:05.000 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:32:03.894+0000 7f0fc2ff5640 1 -- 172.21.15.114:0/4207429892 <== mon.0 v2:172.21.15.114:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f0fd0036040 con 0x55c8d0ee14c0 2024-04-23T10:32:05.000 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:32:03.894+0000 7f0fc2ff5640 1 -- 172.21.15.114:0/4207429892 <== mon.0 v2:172.21.15.114:3300/0 5 ==== fsmap(e 30) ==== 8338+0+0 (secure 0 0 0) 0x7f0fd4004260 con 0x55c8d0ee14c0 2024-04-23T10:32:05.000 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:32:03.894+0000 7f0fe81f4740 10 client.16362 fetch_fsmap finished waiting for FSMap version 30 2024-04-23T10:32:05.001 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:32:03.894+0000 7f0fe81f4740 10 client.16362 resolve_mds: resolved name 'i' to daemon mds.i 2024-04-23T10:32:05.001 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:32:03.894+0000 7f0fe81f4740 20 client.16362 populate_metadata read hostname 'smithi114' 2024-04-23T10:32:05.001 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:32:03.894+0000 7f0fe81f4740 1 --2- 172.21.15.114:0/4207429892 >> [v2:172.21.15.204:6838/2396114218,v1:172.21.15.204:6839/2396114218] conn(0x55c8d0f0a310 0x55c8d0eed3c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T10:32:05.001 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:32:03.894+0000 7f0fe81f4740 4 client.16362 mds_command: new command op to 25537 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-23T10:32:05.001 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:32:03.894+0000 7f0fe81f4740 1 -- 172.21.15.114:0/4207429892 --> [v2:172.21.15.204:6838/2396114218,v1:172.21.15.204:6839/2396114218] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55c8d0f7d480 con 0x55c8d0f0a310 2024-04-23T10:32:05.001 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:32:03.894+0000 7f0fdeffd640 1 --2- 172.21.15.114:0/4207429892 >> [v2:172.21.15.204:6838/2396114218,v1:172.21.15.204:6839/2396114218] conn(0x55c8d0f0a310 0x55c8d0eed3c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T10:32:05.001 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:32:03.895+0000 7f0fc2ff5640 10 client.16362 ms_handle_connect on v2:172.21.15.204:6838/2396114218 2024-04-23T10:32:05.001 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:32:03.895+0000 7f0fdeffd640 1 --2- 172.21.15.114:0/4207429892 >> [v2:172.21.15.204:6838/2396114218,v1:172.21.15.204:6839/2396114218] conn(0x55c8d0f0a310 0x55c8d0eed3c0 crc :-1 s=READY pgs=11 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.25537 client_cookie=9eec51f35e41b972 server_cookie=e21d3cc85291a3e2 in_seq=0 out_seq=0 2024-04-23T10:32:05.001 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:32:03.895+0000 7f0fc2ff5640 1 -- 172.21.15.114:0/4207429892 <== mds.25537 v2:172.21.15.204:6838/2396114218 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x55c8d0f7d480 con 0x55c8d0f0a310 2024-04-23T10:32:05.001 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:32:03.895+0000 7f0fc2ff5640 10 client.16362 handle_command_reply: tid=0 2024-04-23T10:32:05.001 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:32:03.895+0000 7f0fe81f4740 1 client.16362 shutdown 2024-04-23T10:32:05.001 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:32:03.895+0000 7f0fe81f4740 1 -- 172.21.15.114:0/4207429892 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55c8d0ee14c0 msgr2=0x55c8d0f02cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T10:32:05.001 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:32:03.895+0000 7f0fe81f4740 1 --2- 172.21.15.114:0/4207429892 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55c8d0ee14c0 0x55c8d0f02cc0 secure :-1 s=READY pgs=727 cs=0 l=1 rev1=1 crypto rx=0x7f0fd4011cf0 tx=0x7f0fd4011fb0 comp rx=0 tx=0).stop 2024-04-23T10:32:05.001 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:32:03.895+0000 7f0fe81f4740 1 -- 172.21.15.114:0/4207429892 shutdown_connections 2024-04-23T10:32:05.001 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:32:03.895+0000 7f0fe81f4740 1 --2- 172.21.15.114:0/4207429892 >> [v2:172.21.15.204:6838/2396114218,v1:172.21.15.204:6839/2396114218] conn(0x55c8d0f0a310 0x55c8d0eed3c0 crc :-1 s=READY pgs=11 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:32:05.002 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:32:03.895+0000 7f0fe81f4740 1 --2- 172.21.15.114:0/4207429892 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x55c8d0ef00a0 0x55c8d0eea600 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:32:05.002 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:32:03.895+0000 7f0fe81f4740 1 --2- 172.21.15.114:0/4207429892 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55c8d0ee14c0 0x55c8d0f02cc0 unknown :-1 s=CLOSED pgs=727 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:32:05.002 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:32:03.895+0000 7f0fe81f4740 1 --2- 172.21.15.114:0/4207429892 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x55c8d0ee0ba0 0x55c8d0ee9e80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:32:05.002 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:32:03.895+0000 7f0fc2ff5640 0 client.16362 ms_handle_reset on v2:172.21.15.204:6838/2396114218 2024-04-23T10:32:05.002 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:32:03.895+0000 7f0fe81f4740 1 -- 172.21.15.114:0/4207429892 >> 172.21.15.114:0/4207429892 conn(0x55c8d0edcfc0 msgr2=0x55c8d0efb1f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T10:32:05.002 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:32:03.896+0000 7f0fe81f4740 1 -- 172.21.15.114:0/4207429892 shutdown_connections 2024-04-23T10:32:05.002 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:32:03.896+0000 7f0fe81f4740 1 -- 172.21.15.114:0/4207429892 wait complete. 2024-04-23T10:32:05.002 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:32:03.896+0000 7f0fe81f4740 20 client.16362 trim_cache size 0 max 16384 2024-04-23T10:32:05.002 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:32:03.911+0000 7f0fe81f4740 1 Processor -- start 2024-04-23T10:32:05.002 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:32:03.911+0000 7f0fe81f4740 1 -- start start 2024-04-23T10:32:05.003 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:32:03.911+0000 7f0fe81f4740 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x55c8d0ee0ba0 0x55c8d0eea600 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T10:32:05.003 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:32:03.911+0000 7f0fe81f4740 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55c8d0ee14c0 0x55c8d0eed3c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T10:32:05.003 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:32:03.911+0000 7f0fe81f4740 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x55c8d0ef00a0 0x55c8d0f02cc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T10:32:05.003 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:32:03.911+0000 7f0fe81f4740 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x55c8d0f7d300 con 0x55c8d0ee14c0 2024-04-23T10:32:05.003 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:32:03.911+0000 7f0fe81f4740 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 -- 0x55c8d0f0fc90 con 0x55c8d0ee0ba0 2024-04-23T10:32:05.003 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:32:03.911+0000 7f0fe81f4740 1 -- --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_getmap magic: 0 -- 0x55c8d0ee9360 con 0x55c8d0ef00a0 2024-04-23T10:32:05.003 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:32:03.911+0000 7f0fde7fc640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55c8d0ee14c0 0x55c8d0eed3c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T10:32:05.003 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:32:03.911+0000 7f0fde7fc640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55c8d0ee14c0 0x55c8d0eed3c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.114:52116/0 (socket says 172.21.15.114:52116) 2024-04-23T10:32:05.003 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:32:03.911+0000 7f0fde7fc640 1 -- 172.21.15.114:0/2957864438 learned_addr learned my addr 172.21.15.114:0/2957864438 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-04-23T10:32:05.003 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:32:03.911+0000 7f0fdeffd640 1 --2- 172.21.15.114:0/2957864438 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x55c8d0ee0ba0 0x55c8d0eea600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T10:32:05.003 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:32:03.911+0000 7f0fde7fc640 1 -- 172.21.15.114:0/2957864438 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x55c8d0ef00a0 msgr2=0x55c8d0f02cc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T10:32:05.003 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:32:03.911+0000 7f0fde7fc640 1 --2- 172.21.15.114:0/2957864438 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x55c8d0ef00a0 0x55c8d0f02cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:32:05.003 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:32:03.911+0000 7f0fde7fc640 1 -- 172.21.15.114:0/2957864438 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x55c8d0ee0ba0 msgr2=0x55c8d0eea600 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T10:32:05.003 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:32:03.911+0000 7f0fde7fc640 1 --2- 172.21.15.114:0/2957864438 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x55c8d0ee0ba0 0x55c8d0eea600 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:32:05.003 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:32:03.911+0000 7f0fde7fc640 1 -- 172.21.15.114:0/2957864438 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55c8d0f61fb0 con 0x55c8d0ee14c0 2024-04-23T10:32:05.004 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:32:03.911+0000 7f0fde7fc640 1 --2- 172.21.15.114:0/2957864438 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55c8d0ee14c0 0x55c8d0eed3c0 secure :-1 s=READY pgs=728 cs=0 l=1 rev1=1 crypto rx=0x7f0fd4036a40 tx=0x7f0fd4005af0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e0cf9449b4e6327b server_cookie=0 in_seq=0 out_seq=0 2024-04-23T10:32:05.004 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:32:03.911+0000 7f0fc4ff9640 1 -- 172.21.15.114:0/2957864438 <== mon.0 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0fd402f070 con 0x55c8d0ee14c0 2024-04-23T10:32:05.004 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:32:03.912+0000 7f0fc4ff9640 1 -- 172.21.15.114:0/2957864438 <== mon.0 v2:172.21.15.114:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f0fd40047c0 con 0x55c8d0ee14c0 2024-04-23T10:32:05.004 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:32:03.912+0000 7f0fc4ff9640 1 -- 172.21.15.114:0/2957864438 <== mon.0 v2:172.21.15.114:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0fd405c020 con 0x55c8d0ee14c0 2024-04-23T10:32:05.004 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:32:03.912+0000 7f0fe81f4740 1 -- 172.21.15.114:0/2957864438 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55c8d0ee14c0 msgr2=0x55c8d0eed3c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T10:32:05.004 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:32:03.912+0000 7f0fe81f4740 1 --2- 172.21.15.114:0/2957864438 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55c8d0ee14c0 0x55c8d0eed3c0 secure :-1 s=READY pgs=728 cs=0 l=1 rev1=1 crypto rx=0x7f0fd4036a40 tx=0x7f0fd4005af0 comp rx=0 tx=0).stop 2024-04-23T10:32:05.004 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:32:03.912+0000 7f0fe81f4740 1 -- 172.21.15.114:0/2957864438 shutdown_connections 2024-04-23T10:32:05.004 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:32:03.912+0000 7f0fe81f4740 1 --2- 172.21.15.114:0/2957864438 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x55c8d0ef00a0 0x55c8d0f02cc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:32:05.004 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:32:03.912+0000 7f0fe81f4740 1 --2- 172.21.15.114:0/2957864438 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55c8d0ee14c0 0x55c8d0eed3c0 unknown :-1 s=CLOSED pgs=728 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:32:05.004 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:32:03.912+0000 7f0fe81f4740 1 --2- 172.21.15.114:0/2957864438 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x55c8d0ee0ba0 0x55c8d0eea600 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:32:05.004 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:32:03.912+0000 7f0fe81f4740 1 -- 172.21.15.114:0/2957864438 >> 172.21.15.114:0/2957864438 conn(0x55c8d0edcfc0 msgr2=0x55c8d0edc0a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T10:32:05.004 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:32:03.912+0000 7f0fe81f4740 1 -- 172.21.15.114:0/2957864438 shutdown_connections 2024-04-23T10:32:05.004 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:32:03.912+0000 7f0fe81f4740 1 -- 172.21.15.114:0/2957864438 wait complete. 2024-04-23T10:32:05.004 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:32:03.912+0000 7f0fe81f4740 1 -- start start 2024-04-23T10:32:05.004 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:32:03.912+0000 7f0fe81f4740 1 Processor -- start 2024-04-23T10:32:05.005 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:32:03.913+0000 7f0fe81f4740 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x55c8d0ee0ba0 0x55c8d0eea600 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T10:32:05.005 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:32:03.913+0000 7f0fe81f4740 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x55c8d0ee14c0 0x55c8d0f02cc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T10:32:05.005 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:32:03.913+0000 7f0fe81f4740 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55c8d0ef00a0 0x55c8d0eed3c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T10:32:05.005 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:32:03.913+0000 7f0fe81f4740 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x55c8d0eed910 con 0x55c8d0ef00a0 2024-04-23T10:32:05.005 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:32:03.913+0000 7f0fe81f4740 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 -- 0x55c8d0eed240 con 0x55c8d0ee0ba0 2024-04-23T10:32:05.005 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:32:03.913+0000 7f0fe81f4740 1 -- --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_getmap magic: 0 -- 0x55c8d0f6ac10 con 0x55c8d0ee14c0 2024-04-23T10:32:05.005 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:32:03.913+0000 7f0fe81f4740 10 client.0 _ll_register_callbacks cb 0x7f0fd4043040 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-23T10:32:05.005 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:32:03.913+0000 7f0fdf7fe640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55c8d0ef00a0 0x55c8d0eed3c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T10:32:05.005 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:32:03.913+0000 7f0fdf7fe640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55c8d0ef00a0 0x55c8d0eed3c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.114:52124/0 (socket says 172.21.15.114:52124) 2024-04-23T10:32:05.005 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:32:03.913+0000 7f0fdf7fe640 1 -- 172.21.15.114:0/3043853894 learned_addr learned my addr 172.21.15.114:0/3043853894 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-04-23T10:32:05.005 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:32:03.913+0000 7f0fdeffd640 1 --2- 172.21.15.114:0/3043853894 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x55c8d0ee0ba0 0x55c8d0eea600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T10:32:05.005 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:32:03.913+0000 7f0fde7fc640 1 --2- 172.21.15.114:0/3043853894 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x55c8d0ee14c0 0x55c8d0f02cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T10:32:05.005 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:32:03.913+0000 7f0fdf7fe640 1 -- 172.21.15.114:0/3043853894 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x55c8d0ee14c0 msgr2=0x55c8d0f02cc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T10:32:05.005 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:32:03.913+0000 7f0fdf7fe640 1 --2- 172.21.15.114:0/3043853894 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x55c8d0ee14c0 0x55c8d0f02cc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:32:05.006 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:32:03.913+0000 7f0fdf7fe640 1 -- 172.21.15.114:0/3043853894 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x55c8d0ee0ba0 msgr2=0x55c8d0eea600 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T10:32:05.006 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:32:03.913+0000 7f0fdf7fe640 1 --2- 172.21.15.114:0/3043853894 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x55c8d0ee0ba0 0x55c8d0eea600 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:32:05.006 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:32:03.913+0000 7f0fdf7fe640 1 -- 172.21.15.114:0/3043853894 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55c8d0f679e0 con 0x55c8d0ef00a0 2024-04-23T10:32:05.006 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:32:03.913+0000 7f0fde7fc640 1 --2- 172.21.15.114:0/3043853894 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x55c8d0ee14c0 0x55c8d0f02cc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-23T10:32:05.006 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:32:03.913+0000 7f0fdf7fe640 1 --2- 172.21.15.114:0/3043853894 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55c8d0ef00a0 0x55c8d0eed3c0 secure :-1 s=READY pgs=729 cs=0 l=1 rev1=1 crypto rx=0x7f0fd809c2b0 tx=0x7f0fd809bfd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2881ec023153eb6d server_cookie=0 in_seq=0 out_seq=0 2024-04-23T10:32:05.006 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:32:03.913+0000 7f0fc2ff5640 10 client.0 ms_handle_connect on v2:172.21.15.114:3300/0 2024-04-23T10:32:05.006 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:32:03.913+0000 7f0fc2ff5640 1 -- 172.21.15.114:0/3043853894 <== mon.0 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0fd808e070 con 0x55c8d0ef00a0 2024-04-23T10:32:05.006 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:32:03.913+0000 7f0fc2ff5640 1 -- 172.21.15.114:0/3043853894 <== mon.0 v2:172.21.15.114:3300/0 2 ==== osd_map(94..94 src has 1..94) ==== 8585+0+0 (secure 0 0 0) 0x7f0fd80d0130 con 0x55c8d0ef00a0 2024-04-23T10:32:05.006 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:32:03.913+0000 7f0fe81f4740 1 -- 172.21.15.114:0/3043853894 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f0fd40047c0 con 0x55c8d0ef00a0 2024-04-23T10:32:05.006 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:32:03.913+0000 7f0fc2ff5640 1 -- 172.21.15.114:0/3043853894 <== mon.0 v2:172.21.15.114:3300/0 3 ==== mon_get_version_reply(handle=1 version=30) ==== 24+0+0 (secure 0 0 0) 0x7f0fd80ad070 con 0x55c8d0ef00a0 2024-04-23T10:32:05.006 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:32:03.913+0000 7f0fe81f4740 10 client.16368 fetch_fsmap learned FSMap version 30 2024-04-23T10:32:05.006 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:32:03.913+0000 7f0fe81f4740 1 -- 172.21.15.114:0/3043853894 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({config=0+,fsmap=30,monmap=4+}) -- 0x55c8d0f09070 con 0x55c8d0ef00a0 2024-04-23T10:32:05.006 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:32:03.914+0000 7f0fc2ff5640 1 -- 172.21.15.114:0/3043853894 <== mon.0 v2:172.21.15.114:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f0fd40047c0 con 0x55c8d0ef00a0 2024-04-23T10:32:05.006 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:32:03.914+0000 7f0fc2ff5640 1 -- 172.21.15.114:0/3043853894 <== mon.0 v2:172.21.15.114:3300/0 5 ==== fsmap(e 30) ==== 8338+0+0 (secure 0 0 0) 0x7f0fd809b470 con 0x55c8d0ef00a0 2024-04-23T10:32:05.006 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:32:03.914+0000 7f0fe81f4740 10 client.16368 fetch_fsmap finished waiting for FSMap version 30 2024-04-23T10:32:05.007 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:32:03.914+0000 7f0fe81f4740 10 client.16368 resolve_mds: resolved name 'i' to daemon mds.i 2024-04-23T10:32:05.007 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:32:03.914+0000 7f0fe81f4740 20 client.16368 populate_metadata read hostname 'smithi114' 2024-04-23T10:32:05.007 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:32:03.914+0000 7f0fe81f4740 1 --2- 172.21.15.114:0/3043853894 >> [v2:172.21.15.204:6838/2396114218,v1:172.21.15.204:6839/2396114218] conn(0x55c8d0efb1f0 0x55c8d0f7b6f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T10:32:05.007 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:32:03.914+0000 7f0fe81f4740 4 client.16368 mds_command: new command op to 25537 tid=0 multi_id=0 [{"prefix": "ops", "flags": ["locks"]}] 2024-04-23T10:32:05.007 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:32:03.914+0000 7f0fe81f4740 1 -- 172.21.15.114:0/3043853894 --> [v2:172.21.15.204:6838/2396114218,v1:172.21.15.204:6839/2396114218] -- command(tid 0: {"prefix": "ops", "flags": ["locks"]}) -- 0x55c8d0ee7470 con 0x55c8d0efb1f0 2024-04-23T10:32:05.007 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:32:03.914+0000 7f0fdeffd640 1 --2- 172.21.15.114:0/3043853894 >> [v2:172.21.15.204:6838/2396114218,v1:172.21.15.204:6839/2396114218] conn(0x55c8d0efb1f0 0x55c8d0f7b6f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T10:32:05.007 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:32:03.914+0000 7f0fc2ff5640 10 client.16368 ms_handle_connect on v2:172.21.15.204:6838/2396114218 2024-04-23T10:32:05.007 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:32:03.914+0000 7f0fdeffd640 1 --2- 172.21.15.114:0/3043853894 >> [v2:172.21.15.204:6838/2396114218,v1:172.21.15.204:6839/2396114218] conn(0x55c8d0efb1f0 0x55c8d0f7b6f0 crc :-1 s=READY pgs=12 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.25537 client_cookie=3902939b4c47fb3a server_cookie=16380f051be5873 in_seq=0 out_seq=0 2024-04-23T10:32:05.007 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:32:03.914+0000 7f0fc2ff5640 1 -- 172.21.15.114:0/3043853894 <== mds.25537 v2:172.21.15.204:6838/2396114218 1 ==== command_reply(tid 0: 0 ) ==== 8+0+36 (crc 0 0 0) 0x7f0fd0036040 con 0x55c8d0efb1f0 2024-04-23T10:32:05.007 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:32:03.914+0000 7f0fc2ff5640 10 client.16368 handle_command_reply: tid=0 2024-04-23T10:32:05.007 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:32:03.915+0000 7f0fe81f4740 1 client.16368 shutdown 2024-04-23T10:32:05.007 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:32:03.915+0000 7f0fe81f4740 1 -- 172.21.15.114:0/3043853894 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55c8d0ef00a0 msgr2=0x55c8d0eed3c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T10:32:05.007 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:32:03.915+0000 7f0fe81f4740 1 --2- 172.21.15.114:0/3043853894 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55c8d0ef00a0 0x55c8d0eed3c0 secure :-1 s=READY pgs=729 cs=0 l=1 rev1=1 crypto rx=0x7f0fd809c2b0 tx=0x7f0fd809bfd0 comp rx=0 tx=0).stop 2024-04-23T10:32:05.007 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:32:03.915+0000 7f0fe81f4740 1 -- 172.21.15.114:0/3043853894 shutdown_connections 2024-04-23T10:32:05.007 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:32:03.915+0000 7f0fe81f4740 1 --2- 172.21.15.114:0/3043853894 >> [v2:172.21.15.204:6838/2396114218,v1:172.21.15.204:6839/2396114218] conn(0x55c8d0efb1f0 0x55c8d0f7b6f0 crc :-1 s=READY pgs=12 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:32:05.008 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:32:03.915+0000 7f0fe81f4740 1 --2- 172.21.15.114:0/3043853894 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55c8d0ef00a0 0x55c8d0eed3c0 unknown :-1 s=CLOSED pgs=729 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:32:05.008 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:32:03.915+0000 7f0fc2ff5640 0 client.16368 ms_handle_reset on v2:172.21.15.204:6838/2396114218 2024-04-23T10:32:05.008 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:32:03.915+0000 7f0fe81f4740 1 --2- 172.21.15.114:0/3043853894 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x55c8d0ee14c0 0x55c8d0f02cc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:32:05.008 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:32:03.915+0000 7f0fe81f4740 1 --2- 172.21.15.114:0/3043853894 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x55c8d0ee0ba0 0x55c8d0eea600 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:32:05.008 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:32:03.915+0000 7f0fe81f4740 1 -- 172.21.15.114:0/3043853894 >> 172.21.15.114:0/3043853894 conn(0x55c8d0edcfc0 msgr2=0x55c8d0f66c10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T10:32:05.008 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:32:03.915+0000 7f0fe81f4740 1 -- 172.21.15.114:0/3043853894 shutdown_connections 2024-04-23T10:32:05.008 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:32:03.915+0000 7f0fe81f4740 1 -- 172.21.15.114:0/3043853894 wait complete. 2024-04-23T10:32:05.008 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:32:03.915+0000 7f0fe81f4740 20 client.16368 trim_cache size 0 max 16384 2024-04-23T10:32:05.008 DEBUG:teuthology.orchestra.run:got remote process result: 38 2024-04-23T10:32:05.009 INFO:teuthology.orchestra.run.smithi114.stdout:mds.i: { 2024-04-23T10:32:05.009 INFO:teuthology.orchestra.run.smithi114.stdout: "ops": [], 2024-04-23T10:32:05.009 INFO:teuthology.orchestra.run.smithi114.stdout: "num_ops": 0 2024-04-23T10:32:05.009 INFO:teuthology.orchestra.run.smithi114.stdout:} 2024-04-23T10:32:05.011 INFO:tasks.quiescer.fs.[cephfs]: mds.c: { "ops": [], "num_ops": 0 } mds.k: { "ops": [], "num_ops": 0 } mds.e: { "ops": [ { "description": "peer_request:mds.0:190", "initiated_at": "2024-04-23T10:31:21.483611+0000", "age": 42.345133371000003, "duration": 42.345151907999998, "continuous": false, "type_data": { "result": -2147483648, "flag_point": "dispatched", "reqid": { "entity": { "type": "mds", "num": 0 }, "tid": 190 }, "op_type": "peer_request", "leader_info": { "leader": "0" }, "events": [ { "time": "2024-04-23T10:31:21.483611+0000", "event": "initiated" }, { "time": "2024-04-23T10:31:21.483611+0000", "event": "header_read" }, { "time": "2024-04-23T10:31:21.483612+0000", "event": "throttled" }, { "time": "2024-04-23T10:31:21.483618+0000", "event": "all_read" }, { "time": "2024-04-23T10:31:21.484647+0000", "event": "dispatched" } ], "locks": [] } }, { "description": "internal op quiesce_path:mds.2:106 fp=#0x1", "initiated_at": "2024-04-23T10:31:21.445229+0000", "age": 42.383516229999998, "duration": 42.383564112000002, "continuous": true, "type_data": { "result": 0, "flag_point": "quiesce complete", "reqid": { "entity": { "type": "mds", "num": 2 }, "tid": 106 }, "op_type": "internal_op", "internal_op": 5384, "op_name": "quiesce_path", "events": [ { "time": "2024-04-23T10:31:21.445229+0000", "event": "initiated" }, { "time": "2024-04-23T10:31:21.445229+0000", "event": "header_read" }, { "time": "2024-04-23T10:31:21.445229+0000", "event": "throttled" }, { "time": "2024-04-23T10:31:21.445229+0000", "event": "all_read" }, { "time": "2024-04-23T10:31:21.445229+0000", "event": "dispatched" }, { "time": "2024-04-23T10:31:21.509505+0000", "event": "quiesce complete" } ], "locks": [] } }, { "description": "peer_request:mds.0:191", "initiated_at": "2024-04-23T10:31:21.483666+0000", "age": 42.345078950000001, "duration": 42.345151399000002, "continuous": false, "type_data": { "result": -2147483648, "flag_point": "dispatched", "reqid": { "entity": { "type": "mds", "num": 0 }, "tid": 191 }, "op_type": "peer_request", "leader_info": { "leader": "0" }, "events": [ { "time": "2024-04-23T10:31:21.483666+0000", "event": "initiated" }, { "time": "2024-04-23T10:31:21.483666+0000", "event": "header_read" }, { "time": "2024-04-23T10:31:21.483666+0000", "event": "throttled" }, { "time": "2024-04-23T10:31:21.483671+0000", "event": "all_read" }, { "time": "2024-04-23T10:31:21.485256+0000", "event": "dispatched" } ], "locks": [] } }, { "description": "internal op quiesce_inode:mds.2:107 fp=#0x1", "initiated_at": "2024-04-23T10:31:21.445284+0000", "age": 42.383460773000003, "duration": 42.383555897999997, "continuous": true, "type_data": { "result": -2147483648, "flag_point": "quiesce complete for non-auth inode", "reqid": { "entity": { "type": "mds", "num": 2 }, "tid": 107 }, "op_type": "internal_op", "internal_op": 5385, "op_name": "quiesce_inode", "events": [ { "time": "2024-04-23T10:31:21.445284+0000", "event": "initiated" }, { "time": "2024-04-23T10:31:21.445284+0000", "event": "header_read" }, { "time": "2024-04-23T10:31:21.445284+0000", "event": "throttled" }, { "time": "2024-04-23T10:31:21.445284+0000", "event": "all_read" }, { "time": "2024-04-23T10:31:21.445284+0000", "event": "dispatched" }, { "time": "2024-04-23T10:31:21.445398+0000", "event": "requesting remote authpins" }, { "time": "2024-04-23T10:31:21.446094+0000", "event": "acquired locks" }, { "time": "2024-04-23T10:31:21.509469+0000", "event": "quiesce complete for non-auth inode" } ], "locks": [ { "object": { "is_auth": false, "auth_state": { "replicas": {} }, "replica_state": { "authority": [ 0, -2 ], "replica_nonce": 1 }, "auth_pins": 0, "is_frozen": false, "is_freezing": false, "pins": { "request": 1, "lock": 1, "dirfrag": 1, "caps": 0, "dirwaiter": 0, "waiter": 0, "discoverbase": 0 }, "nref": 3 }, "object_string": "[inode 0x1 [...7,head] / rep@0.1 v187 snaprealm=0x55c68b7b0480 f(v0 m2024-04-23T10:24:48.676117+0000 1=0+1) n(v27 rc2024-04-23T10:31:17.764716+0000 b21075308 rs5 88=67+21)/n(v0 rc2024-04-23T10:24:44.319412+0000 1=0+1) (inest mix) (iquiesce lock x=1 by request(mds.2:107 nref=3)) | request=1 lock=1 dirfrag=1 caps=0 dirwaiter=0 waiter=0 discoverbase=0 0x55c68ba05080]", "lock": { "gather_set": [], "state": "lock", "type": "iquiesce", "is_leased": false, "num_rdlocks": 0, "num_wrlocks": 0, "num_xlocks": 1, "xlock_by": { "reqid": { "entity": { "type": "mds", "num": 2 }, "tid": 107 } } }, "flags": 4, "wrlock_target": -1 } ] } }, { "description": "peer_request:mds.0:192", "initiated_at": "2024-04-23T10:31:21.483707+0000", "age": 42.345037998999999, "duration": 42.345206638000001, "continuous": false, "type_data": { "result": -2147483648, "flag_point": "dispatched", "reqid": { "entity": { "type": "mds", "num": 0 }, "tid": 192 }, "op_type": "peer_request", "leader_info": { "leader": "0" }, "events": [ { "time": "2024-04-23T10:31:21.483707+0000", "event": "initiated" }, { "time": "2024-04-23T10:31:21.483707+0000", "event": "header_read" }, { "time": "2024-04-23T10:31:21.483707+0000", "event": "throttled" }, { "time": "2024-04-23T10:31:21.483712+0000", "event": "all_read" }, { "time": "2024-04-23T10:31:21.485846+0000", "event": "dispatched" } ], "locks": [] } }, { "description": "internal op quiesce_inode:mds.2:108 fp=#0x10000000000", "initiated_at": "2024-04-23T10:31:21.446108+0000", "age": 42.382636726000001, "duration": 42.382827669000001, "continuous": true, "type_data": { "result": -2147483648, "flag_point": "quiesce complete for non-auth inode", "reqid": { "entity": { "type": "mds", "num": 2 }, "tid": 108 }, "op_type": "internal_op", "internal_op": 5385, "op_name": "quiesce_inode", "events": [ { "time": "2024-04-23T10:31:21.446108+0000", "event": "initiated" }, { "time": "2024-04-23T10:31:21.446108+0000", "event": "header_read" }, { "time": "2024-04-23T10:31:21.446108+0000", "event": "throttled" }, { "time": "2024-04-23T10:31:21.446108+0000", "event": "all_read" }, { "time": "2024-04-23T10:31:21.446108+0000", "event": "dispatched" }, { "time": "2024-04-23T10:31:21.446194+0000", "event": "requesting remote authpins" }, { "time": "2024-04-23T10:31:21.447335+0000", "event": "acquired locks" }, { "time": "2024-04-23T10:31:21.509440+0000", "event": "quiesce complete for non-auth inode" } ], "locks": [ { "object": { "is_auth": false, "auth_state": { "replicas": {} }, "replica_state": { "authority": [ 0, -2 ], "replica_nonce": 1 }, "auth_pins": 0, "is_frozen": false, "is_freezing": false, "pins": { "request": 1, "lock": 1, "dirfrag": 1, "dirwaiter": 0, "discoverbase": 0 }, "nref": 3 }, "object_string": "[inode 0x10000000000 [...7,head] /volumes/ rep@0.1 v404 f(v0 m2024-04-23T10:24:49.736808+0000 3=2+1) n(v29 rc2024-04-23T10:31:17.764716+0000 b21075308 87=67+20)/n(v0 1=0+1) (inest mix) (iquiesce lock x=1 by request(mds.2:108 nref=3)) | request=1 lock=1 dirfrag=1 dirwaiter=0 discoverbase=0 0x55c68ba05600]", "lock": { "gather_set": [], "state": "lock", "type": "iquiesce", "is_leased": false, "num_rdlocks": 0, "num_wrlocks": 0, "num_xlocks": 1, "xlock_by": { "reqid": { "entity": { "type": "mds", "num": 2 }, "tid": 108 } } }, "flags": 4, "wrlock_target": -1 } ] } }, { "description": "peer_request:mds.0:193", "initiated_at": "2024-04-23T10:31:21.483754+0000", "age": 42.344990783, "duration": 42.345240236999999, "continuous": false, "type_data": { "result": -2147483648, "flag_point": "dispatched", "reqid": { "entity": { "type": "mds", "num": 0 }, "tid": 193 }, "op_type": "peer_request", "leader_info": { "leader": "0" }, "events": [ { "time": "2024-04-23T10:31:21.483754+0000", "event": "initiated" }, { "time": "2024-04-23T10:31:21.483754+0000", "event": "header_read" }, { "time": "2024-04-23T10:31:21.483754+0000", "event": "throttled" }, { "time": "2024-04-23T10:31:21.483759+0000", "event": "all_read" }, { "time": "2024-04-23T10:31:21.486610+0000", "event": "dispatched" } ], "locks": [] } }, { "description": "internal op quiesce_inode:mds.2:109 fp=#0x10000000001", "initiated_at": "2024-04-23T10:31:21.447349+0000", "age": 42.381396215999999, "duration": 42.381670167999999, "continuous": true, "type_data": { "result": -2147483648, "flag_point": "quiesce complete for non-auth inode", "reqid": { "entity": { "type": "mds", "num": 2 }, "tid": 109 }, "op_type": "internal_op", "internal_op": 5385, "op_name": "quiesce_inode", "events": [ { "time": "2024-04-23T10:31:21.447349+0000", "event": "initiated" }, { "time": "2024-04-23T10:31:21.447349+0000", "event": "header_read" }, { "time": "2024-04-23T10:31:21.447349+0000", "event": "throttled" }, { "time": "2024-04-23T10:31:21.447349+0000", "event": "all_read" }, { "time": "2024-04-23T10:31:21.447349+0000", "event": "dispatched" }, { "time": "2024-04-23T10:31:21.447418+0000", "event": "requesting remote authpins" }, { "time": "2024-04-23T10:31:21.448374+0000", "event": "acquired locks" }, { "time": "2024-04-23T10:31:21.509412+0000", "event": "quiesce complete for non-auth inode" } ], "locks": [ { "object": { "is_auth": false, "auth_state": { "replicas": {} }, "replica_state": { "authority": [ 0, -2 ], "replica_nonce": 1 }, "auth_pins": 0, "is_frozen": false, "is_freezing": false, "pins": { "request": 1, "lock": 1, "dirfrag": 1, "discoverbase": 0 }, "nref": 3 }, "object_string": "[inode 0x10000000001 [...7,head] /volumes/qa/ rep@0.1 v416 f(v0 m2024-04-23T10:24:49.714717+0000 2=0+2) n(v30 rc2024-04-23T10:31:17.764716+0000 b21075308 84=65+19)/n(v0 rc2024-04-23T10:24:49.023727+0000 3=0+3) (inest mix) (iquiesce lock x=1 by request(mds.2:109 nref=3)) | request=1 lock=1 dirfrag=1 discoverbase=0 0x55c68ba05b80]", "lock": { "gather_set": [], "state": "lock", "type": "iquiesce", "is_leased": false, "num_rdlocks": 0, "num_wrlocks": 0, "num_xlocks": 1, "xlock_by": { "reqid": { "entity": { "type": "mds", "num": 2 }, "tid": 109 } } }, "flags": 4, "wrlock_target": -1 } ] } }, { "description": "peer_request:mds.0:194", "initiated_at": "2024-04-23T10:31:21.483806+0000", "age": 42.344938784, "duration": 42.345270112999998, "continuous": false, "type_data": { "result": -2147483648, "flag_point": "dispatched", "reqid": { "entity": { "type": "mds", "num": 0 }, "tid": 194 }, "op_type": "peer_request", "leader_info": { "leader": "0" }, "events": [ { "time": "2024-04-23T10:31:21.483806+0000", "event": "initiated" }, { "time": "2024-04-23T10:31:21.483806+0000", "event": "header_read" }, { "time": "2024-04-23T10:31:21.483806+0000", "event": "throttled" }, { "time": "2024-04-23T10:31:21.483811+0000", "event": "all_read" }, { "time": "2024-04-23T10:31:21.487168+0000", "event": "dispatched" } ], "locks": [] } }, { "description": "internal op quiesce_inode:mds.2:110 fp=#0x100000001fa", "initiated_at": "2024-04-23T10:31:21.448394+0000", "age": 42.380351050000002, "duration": 42.380704240999997, "continuous": true, "type_data": { "result": -2147483648, "flag_point": "quiesce complete for non-auth inode", "reqid": { "entity": { "type": "mds", "num": 2 }, "tid": 110 }, "op_type": "internal_op", "internal_op": 5385, "op_name": "quiesce_inode", "events": [ { "time": "2024-04-23T10:31:21.448394+0000", "event": "initiated" }, { "time": "2024-04-23T10:31:21.448394+0000", "event": "header_read" }, { "time": "2024-04-23T10:31:21.448394+0000", "event": "throttled" }, { "time": "2024-04-23T10:31:21.448394+0000", "event": "all_read" }, { "time": "2024-04-23T10:31:21.448394+0000", "event": "dispatched" }, { "time": "2024-04-23T10:31:21.448493+0000", "event": "requesting remote authpins" }, { "time": "2024-04-23T10:31:21.449533+0000", "event": "acquired locks" }, { "time": "2024-04-23T10:31:21.509389+0000", "event": "quiesce complete for non-auth inode" } ], "locks": [ { "object": { "is_auth": false, "auth_state": { "replicas": {} }, "replica_state": { "authority": [ 0, -2 ], "replica_nonce": 1 }, "auth_pins": 0, "is_frozen": false, "is_freezing": false, "pins": { "request": 1, "lock": 1, "dirfrag": 1, "caps": 1, "dirwaiter": 0, "discoverbase": 0 }, "nref": 4 }, "object_string": "[inode 0x100000001fa [...7,head] /volumes/qa/sv_1/ rep@0.1 v434 snaprealm=0x55c68b7b06c0 f(v0 m2024-04-23T10:24:49.735055+0000 2=1+1) n(v29 rc2024-04-23T10:31:17.764716+0000 b21075191 80=64+16)/n(v0 rc2024-04-23T10:24:49.717706+0000 2=0+2) (inest mix) (iquiesce lock x=1 by request(mds.2:110 nref=3)) caps={15312=pAsLsXs/-@2} | request=1 lock=1 dirfrag=1 caps=1 dirwaiter=0 discoverbase=0 0x55c68ba07700]", "lock": { "gather_set": [], "state": "lock", "type": "iquiesce", "is_leased": false, "num_rdlocks": 0, "num_wrlocks": 0, "num_xlocks": 1, "xlock_by": { "reqid": { "entity": { "type": "mds", "num": 2 }, "tid": 110 } } }, "flags": 4, "wrlock_target": -1 } ] } }, { "description": "peer_request:mds.0:195", "initiated_at": "2024-04-23T10:31:21.483857+0000", "age": 42.344887266999997, "duration": 42.345311197000001, "continuous": false, "type_data": { "result": -2147483648, "flag_point": "dispatched", "reqid": { "entity": { "type": "mds", "num": 0 }, "tid": 195 }, "op_type": "peer_request", "leader_info": { "leader": "0" }, "events": [ { "time": "2024-04-23T10:31:21.483857+0000", "event": "initiated" }, { "time": "2024-04-23T10:31:21.483857+0000", "event": "header_read" }, { "time": "2024-04-23T10:31:21.483858+0000", "event": "throttled" }, { "time": "2024-04-23T10:31:21.483862+0000", "event": "all_read" }, { "time": "2024-04-23T10:31:21.487796+0000", "event": "dispatched" } ], "locks": [] } }, { "description": "internal op quiesce_inode:mds.2:111 fp=#0x100000001fb", "initiated_at": "2024-04-23T10:31:21.449549+0000", "age": 42.379195478, "duration": 42.379643676999997, "continuous": true, "type_data": { "result": -2147483648, "flag_point": "quiesce complete for non-auth inode", "reqid": { "entity": { "type": "mds", "num": 2 }, "tid": 111 }, "op_type": "internal_op", "internal_op": 5385, "op_name": "quiesce_inode", "events": [ { "time": "2024-04-23T10:31:21.449549+0000", "event": "initiated" }, { "time": "2024-04-23T10:31:21.449549+0000", "event": "header_read" }, { "time": "2024-04-23T10:31:21.449549+0000", "event": "throttled" }, { "time": "2024-04-23T10:31:21.449549+0000", "event": "all_read" }, { "time": "2024-04-23T10:31:21.449549+0000", "event": "dispatched" }, { "time": "2024-04-23T10:31:21.449635+0000", "event": "requesting remote authpins" }, { "time": "2024-04-23T10:31:21.450800+0000", "event": "acquired locks" }, { "time": "2024-04-23T10:31:21.509364+0000", "event": "quiesce complete for non-auth inode" } ], "locks": [ { "object": { "is_auth": false, "auth_state": { "replicas": {} }, "replica_state": { "authority": [ 0, -2 ], "replica_nonce": 1 }, "auth_pins": 0, "is_frozen": false, "is_freezing": false, "pins": { "request": 1, "lock": 1, "dirfrag": 1, "dirwaiter": 0, "discoverbase": 0 }, "nref": 3 }, "object_string": "[inode 0x100000001fb [...7,head] /volumes/qa/sv_1/b206794e-7622-4ab8-8943-f3ae4e65b7a3/ rep@0.1 v426 f(v0 m2024-04-23T10:26:48.475842+0000 1=0+1) n(v25 rc2024-04-23T10:31:17.764716+0000 b21075074 78=63+15)/n(v0 rc2024-04-23T10:26:48.480842+0000 2=0+2) old_inodes=1 (inest mix) (iquiesce lock x=1 by request(mds.2:111 nref=3)) | request=1 lock=1 dirfrag=1 dirwaiter=0 discoverbase=0 0x55c68b9d1600]", "lock": { "gather_set": [], "state": "lock", "type": "iquiesce", "is_leased": false, "num_rdlocks": 0, "num_wrlocks": 0, "num_xlocks": 1, "xlock_by": { "reqid": { "entity": { "type": "mds", "num": 2 }, "tid": 111 } } }, "flags": 4, "wrlock_target": -1 } ] } }, { "description": "peer_request:mds.0:196", "initiated_at": "2024-04-23T10:31:21.483910+0000", "age": 42.344834759000001, "duration": 42.345370498999998, "continuous": false, "type_data": { "result": -2147483648, "flag_point": "dispatched", "reqid": { "entity": { "type": "mds", "num": 0 }, "tid": 196 }, "op_type": "peer_request", "leader_info": { "leader": "0" }, "events": [ { "time": "2024-04-23T10:31:21.483910+0000", "event": "initiated" }, { "time": "2024-04-23T10:31:21.483910+0000", "event": "header_read" }, { "time": "2024-04-23T10:31:21.483911+0000", "event": "throttled" }, { "time": "2024-04-23T10:31:21.483915+0000", "event": "all_read" }, { "time": "2024-04-23T10:31:21.488559+0000", "event": "dispatched" } ], "locks": [] } }, { "description": "internal op quiesce_inode:mds.2:112 fp=#0x10000000005", "initiated_at": "2024-04-23T10:31:21.450819+0000", "age": 42.377925865000002, "duration": 42.378484041, "continuous": true, "type_data": { "result": -2147483648, "flag_point": "quiesce complete for non-auth inode", "reqid": { "entity": { "type": "mds", "num": 2 }, "tid": 112 }, "op_type": "internal_op", "internal_op": 5385, "op_name": "quiesce_inode", "events": [ { "time": "2024-04-23T10:31:21.450819+0000", "event": "initiated" }, { "time": "2024-04-23T10:31:21.450819+0000", "event": "header_read" }, { "time": "2024-04-23T10:31:21.450819+0000", "event": "throttled" }, { "time": "2024-04-23T10:31:21.450819+0000", "event": "all_read" }, { "time": "2024-04-23T10:31:21.450819+0000", "event": "dispatched" }, { "time": "2024-04-23T10:31:21.450899+0000", "event": "requesting remote authpins" }, { "time": "2024-04-23T10:31:21.451997+0000", "event": "acquired locks" }, { "time": "2024-04-23T10:31:21.509342+0000", "event": "quiesce complete for non-auth inode" } ], "locks": [ { "object": { "is_auth": false, "auth_state": { "replicas": {} }, "replica_state": { "authority": [ 0, -2 ], "replica_nonce": 1 }, "auth_pins": 0, "is_frozen": false, "is_freezing": false, "pins": { "request": 1, "lock": 1, "dirfrag": 1 }, "nref": 3 }, "object_string": "[inode 0x10000000005 [...7,head] /volumes/qa/sv_1/b206794e-7622-4ab8-8943-f3ae4e65b7a3/client.0/ rep@0.1 v421 f(v0 m2024-04-23T10:27:42.521612+0000 1=0+1) n(v32 rc2024-04-23T10:31:17.764716+0000 b21075074 77=63+14)/n(v0 rc2024-04-23T10:27:42.521612+0000 2=0+2) old_inodes=1 (inest mix) (iquiesce lock x=1 by request(mds.2:112 nref=3)) | request=1 lock=1 dirfrag=1 0x55c68b9d1b80]", "lock": { "gather_set": [], "state": "lock", "type": "iquiesce", "is_leased": false, "num_rdlocks": 0, "num_wrlocks": 0, "num_xlocks": 1, "xlock_by": { "reqid": { "entity": { "type": "mds", "num": 2 }, "tid": 112 } } }, "flags": 4, "wrlock_target": -1 } ] } }, { "description": "peer_request:mds.0:197", "initiated_at": "2024-04-23T10:31:21.483962+0000", "age": 42.344782944000002, "duration": 42.345402452000002, "continuous": false, "type_data": { "result": -2147483648, "flag_point": "dispatched", "reqid": { "entity": { "type": "mds", "num": 0 }, "tid": 197 }, "op_type": "peer_request", "leader_info": { "leader": "0" }, "events": [ { "time": "2024-04-23T10:31:21.483962+0000", "event": "initiated" }, { "time": "2024-04-23T10:31:21.483962+0000", "event": "header_read" }, { "time": "2024-04-23T10:31:21.483962+0000", "event": "throttled" }, { "time": "2024-04-23T10:31:21.483966+0000", "event": "all_read" }, { "time": "2024-04-23T10:31:21.488644+0000", "event": "dispatched" } ], "locks": [] } }, { "description": "internal op quiesce_inode:mds.2:113 fp=#0x10000000006", "initiated_at": "2024-04-23T10:31:21.452009+0000", "age": 42.376735879000002, "duration": 42.377379026, "continuous": true, "type_data": { "result": -2147483648, "flag_point": "quiesce complete for non-auth inode", "reqid": { "entity": { "type": "mds", "num": 2 }, "tid": 113 }, "op_type": "internal_op", "internal_op": 5385, "op_name": "quiesce_inode", "events": [ { "time": "2024-04-23T10:31:21.452009+0000", "event": "initiated" }, { "time": "2024-04-23T10:31:21.452009+0000", "event": "header_read" }, { "time": "2024-04-23T10:31:21.452009+0000", "event": "throttled" }, { "time": "2024-04-23T10:31:21.452009+0000", "event": "all_read" }, { "time": "2024-04-23T10:31:21.452009+0000", "event": "dispatched" }, { "time": "2024-04-23T10:31:21.452087+0000", "event": "requesting remote authpins" }, { "time": "2024-04-23T10:31:21.453420+0000", "event": "acquired locks" }, { "time": "2024-04-23T10:31:21.509319+0000", "event": "quiesce complete for non-auth inode" } ], "locks": [ { "object": { "is_auth": false, "auth_state": { "replicas": {} }, "replica_state": { "authority": [ 0, -2 ], "replica_nonce": 1 }, "auth_pins": 0, "is_frozen": false, "is_freezing": false, "pins": { "request": 1, "lock": 1, "dirfrag": 1 }, "nref": 3 }, "object_string": "[inode 0x10000000006 [...7,head] /volumes/qa/sv_1/b206794e-7622-4ab8-8943-f3ae4e65b7a3/client.0/tmp/ rep@0.1 v408 f(v0 m2024-04-23T10:27:51.919399+0000 1=0+1) n(v30 rc2024-04-23T10:31:17.764716+0000 b21075074 76=63+13)/n(v0 rc2024-04-23T10:27:51.919399+0000 2=0+2) (inest mix) (iquiesce lock x=1 by request(mds.2:113 nref=3)) | request=1 lock=1 dirfrag=1 0x55c68b9d2100]", "lock": { "gather_set": [], "state": "lock", "type": "iquiesce", "is_leased": false, "num_rdlocks": 0, "num_wrlocks": 0, "num_xlocks": 1, "xlock_by": { "reqid": { "entity": { "type": "mds", "num": 2 }, "tid": 113 } } }, "flags": 4, "wrlock_target": -1 } ] } }, { "description": "peer_request:mds.0:198", "initiated_at": "2024-04-23T10:31:21.484024+0000", "age": 42.344720899000002, "duration": 42.345440551999999, "continuous": false, "type_data": { "result": -2147483648, "flag_point": "dispatched", "reqid": { "entity": { "type": "mds", "num": 0 }, "tid": 198 }, "op_type": "peer_request", "leader_info": { "leader": "0" }, "events": [ { "time": "2024-04-23T10:31:21.484024+0000", "event": "initiated" }, { "time": "2024-04-23T10:31:21.484024+0000", "event": "header_read" }, { "time": "2024-04-23T10:31:21.484025+0000", "event": "throttled" }, { "time": "2024-04-23T10:31:21.484032+0000", "event": "all_read" }, { "time": "2024-04-23T10:31:21.488737+0000", "event": "dispatched" } ], "locks": [] } }, { "description": "internal op quiesce_inode:mds.2:114 fp=#0x10000000007", "initiated_at": "2024-04-23T10:31:21.453434+0000", "age": 42.375310267000003, "duration": 42.376054576000001, "continuous": true, "type_data": { "result": -2147483648, "flag_point": "quiesce complete for non-auth inode", "reqid": { "entity": { "type": "mds", "num": 2 }, "tid": 114 }, "op_type": "internal_op", "internal_op": 5385, "op_name": "quiesce_inode", "events": [ { "time": "2024-04-23T10:31:21.453434+0000", "event": "initiated" }, { "time": "2024-04-23T10:31:21.453434+0000", "event": "header_read" }, { "time": "2024-04-23T10:31:21.453434+0000", "event": "throttled" }, { "time": "2024-04-23T10:31:21.453434+0000", "event": "all_read" }, { "time": "2024-04-23T10:31:21.453434+0000", "event": "dispatched" }, { "time": "2024-04-23T10:31:21.453528+0000", "event": "requesting remote authpins" }, { "time": "2024-04-23T10:31:21.455137+0000", "event": "acquired locks" }, { "time": "2024-04-23T10:31:21.509290+0000", "event": "quiesce complete for non-auth inode" } ], "locks": [ { "object": { "is_auth": false, "auth_state": { "replicas": {} }, "replica_state": { "authority": [ 0, -2 ], "replica_nonce": 1 }, "auth_pins": 0, "is_frozen": false, "is_freezing": false, "pins": { "request": 1, "lock": 1, "dirfrag": 1 }, "nref": 3 }, "object_string": "[inode 0x10000000007 [...7,head] /volumes/qa/sv_1/b206794e-7622-4ab8-8943-f3ae4e65b7a3/client.0/tmp/clients/ rep@0.1 v412 f(v0 m2024-04-23T10:27:51.919399+0000 1=0+1) n(v30 rc2024-04-23T10:31:17.764716+0000 b21075074 75=63+12)/n(v0 rc2024-04-23T10:27:51.919399+0000 2=0+2) (inest mix) (iquiesce lock x=1 by request(mds.2:114 nref=3)) | request=1 lock=1 dirfrag=1 0x55c68b9d2680]", "lock": { "gather_set": [], "state": "lock", "type": "iquiesce", "is_leased": false, "num_rdlocks": 0, "num_wrlocks": 0, "num_xlocks": 1, "xlock_by": { "reqid": { "entity": { "type": "mds", "num": 2 }, "tid": 114 } } }, "flags": 4, "wrlock_target": -1 } ] } }, { "description": "peer_request:mds.0:199", "initiated_at": "2024-04-23T10:31:21.484068+0000", "age": 42.344677247999996, "duration": 42.345484798000001, "continuous": false, "type_data": { "result": -2147483648, "flag_point": "dispatched", "reqid": { "entity": { "type": "mds", "num": 0 }, "tid": 199 }, "op_type": "peer_request", "leader_info": { "leader": "0" }, "events": [ { "time": "2024-04-23T10:31:21.484068+0000", "event": "initiated" }, { "time": "2024-04-23T10:31:21.484068+0000", "event": "header_read" }, { "time": "2024-04-23T10:31:21.484068+0000", "event": "throttled" }, { "time": "2024-04-23T10:31:21.484073+0000", "event": "all_read" }, { "time": "2024-04-23T10:31:21.488833+0000", "event": "dispatched" } ], "locks": [] } }, { "description": "internal op quiesce_inode:mds.2:115 fp=#0x10000000008", "initiated_at": "2024-04-23T10:31:21.455152+0000", "age": 42.373593233999998, "duration": 42.374422787, "continuous": true, "type_data": { "result": -2147483648, "flag_point": "quiesce complete for non-auth inode", "reqid": { "entity": { "type": "mds", "num": 2 }, "tid": 115 }, "op_type": "internal_op", "internal_op": 5385, "op_name": "quiesce_inode", "events": [ { "time": "2024-04-23T10:31:21.455152+0000", "event": "initiated" }, { "time": "2024-04-23T10:31:21.455152+0000", "event": "header_read" }, { "time": "2024-04-23T10:31:21.455152+0000", "event": "throttled" }, { "time": "2024-04-23T10:31:21.455152+0000", "event": "all_read" }, { "time": "2024-04-23T10:31:21.455152+0000", "event": "dispatched" }, { "time": "2024-04-23T10:31:21.455229+0000", "event": "requesting remote authpins" }, { "time": "2024-04-23T10:31:21.456936+0000", "event": "acquired locks" }, { "time": "2024-04-23T10:31:21.509277+0000", "event": "quiesce complete for non-auth inode" } ], "locks": [ { "object": { "is_auth": false, "auth_state": { "replicas": {} }, "replica_state": { "authority": [ 0, -2 ], "replica_nonce": 1 }, "auth_pins": 0, "is_frozen": false, "is_freezing": false, "pins": { "request": 1, "lock": 1, "dirfrag": 1, "dirwaiter": 0, "discoverbase": 0 }, "nref": 3 }, "object_string": "[inode 0x10000000008 [...7,head] /volumes/qa/sv_1/b206794e-7622-4ab8-8943-f3ae4e65b7a3/client.0/tmp/clients/client0/ rep@0.1 v456 f(v0 m2024-04-23T10:27:51.950398+0000 6=5+1) n(v30 rc2024-04-23T10:31:17.764716+0000 b21075074 74=63+11)/n(v0 rc2024-04-23T10:27:51.920398+0000 2=0+2) (inest mix) (iquiesce lock x=1 by request(mds.2:115 nref=3)) | request=1 lock=1 dirfrag=1 dirwaiter=0 discoverbase=0 0x55c68b9d2c00]", "lock": { "gather_set": [], "state": "lock", "type": "iquiesce", "is_leased": false, "num_rdlocks": 0, "num_wrlocks": 0, "num_xlocks": 1, "xlock_by": { "reqid": { "entity": { "type": "mds", "num": 2 }, "tid": 115 } } }, "flags": 4, "wrlock_target": -1 } ] } }, { "description": "peer_request:mds.0:200", "initiated_at": "2024-04-23T10:31:21.484125+0000", "age": 42.344619600000001, "duration": 42.345512378999999, "continuous": false, "type_data": { "result": -2147483648, "flag_point": "dispatched", "reqid": { "entity": { "type": "mds", "num": 0 }, "tid": 200 }, "op_type": "peer_request", "leader_info": { "leader": "0" }, "events": [ { "time": "2024-04-23T10:31:21.484125+0000", "event": "initiated" }, { "time": "2024-04-23T10:31:21.484125+0000", "event": "header_read" }, { "time": "2024-04-23T10:31:21.484126+0000", "event": "throttled" }, { "time": "2024-04-23T10:31:21.484131+0000", "event": "all_read" }, { "time": "2024-04-23T10:31:21.488931+0000", "event": "dispatched" } ], "locks": [] } }, { "description": "internal op quiesce_inode:mds.2:116 fp=#0x10000000009", "initiated_at": "2024-04-23T10:31:21.456950+0000", "age": 42.371794465999997, "duration": 42.372710093000002, "continuous": true, "type_data": { "result": -2147483648, "flag_point": "quiesce complete for non-auth inode", "reqid": { "entity": { "type": "mds", "num": 2 }, "tid": 116 }, "op_type": "internal_op", "internal_op": 5385, "op_name": "quiesce_inode", "events": [ { "time": "2024-04-23T10:31:21.456950+0000", "event": "initiated" }, { "time": "2024-04-23T10:31:21.456950+0000", "event": "header_read" }, { "time": "2024-04-23T10:31:21.456950+0000", "event": "throttled" }, { "time": "2024-04-23T10:31:21.456950+0000", "event": "all_read" }, { "time": "2024-04-23T10:31:21.456950+0000", "event": "dispatched" }, { "time": "2024-04-23T10:31:21.457023+0000", "event": "requesting remote authpins" }, { "time": "2024-04-23T10:31:21.463073+0000", "event": "acquired locks" }, { "time": "2024-04-23T10:31:21.509262+0000", "event": "quiesce complete for non-auth inode" } ], "locks": [ { "object": { "is_auth": false, "auth_state": { "replicas": {} }, "replica_state": { "authority": [ 0, -2 ], "replica_nonce": 1 }, "auth_pins": 0, "is_frozen": false, "is_freezing": false, "pins": { "request": 1, "lock": 1, "dirfrag": 1, "discoverbase": 0 }, "nref": 3 }, "object_string": "[inode 0x10000000009 [...7,head] /volumes/qa/sv_1/b206794e-7622-4ab8-8943-f3ae4e65b7a3/client.0/tmp/clients/client0/~dmtmp/ rep@0.1 v15176 f(v0 m2024-04-23T10:27:52.186392+0000 9=0+9) n(v31 rc2024-04-23T10:31:17.764716+0000 b15832194 68=58+10)/n(v0 rc2024-04-23T10:27:51.953398+0000 2=0+2) (inest mix) (iquiesce lock x=1 by request(mds.2:116 nref=3)) | request=1 lock=1 dirfrag=1 discoverbase=0 0x55c68b9d3180]", "lock": { "gather_set": [], "state": "lock", "type": "iquiesce", "is_leased": false, "num_rdlocks": 0, "num_wrlocks": 0, "num_xlocks": 1, "xlock_by": { "reqid": { "entity": { "type": "mds", "num": 2 }, "tid": 116 } } }, "flags": 4, "wrlock_target": -1 } ] } }, { "description": "internal op quiesce_inode:mds.2:137 fp=#0x10000000241", "initiated_at": "2024-04-23T10:31:21.489235+0000", "age": 42.339509425000003, "duration": 42.340487074999999, "continuous": true, "type_data": { "result": -2147483648, "flag_point": "quiesce complete for non-auth inode", "reqid": { "entity": { "type": "mds", "num": 2 }, "tid": 137 }, "op_type": "internal_op", "internal_op": 5385, "op_name": "quiesce_inode", "events": [ { "time": "2024-04-23T10:31:21.489235+0000", "event": "initiated" }, { "time": "2024-04-23T10:31:21.489235+0000", "event": "header_read" }, { "time": "2024-04-23T10:31:21.489235+0000", "event": "throttled" }, { "time": "2024-04-23T10:31:21.489235+0000", "event": "all_read" }, { "time": "2024-04-23T10:31:21.489235+0000", "event": "dispatched" }, { "time": "2024-04-23T10:31:21.489345+0000", "event": "requesting remote authpins" }, { "time": "2024-04-23T10:31:21.508959+0000", "event": "acquired locks" }, { "time": "2024-04-23T10:31:21.508974+0000", "event": "quiesce complete for non-auth inode" } ], "locks": [ { "object": { "is_auth": false, "auth_state": { "replicas": {} }, "replica_state": { "authority": [ 0, -2 ], "replica_nonce": 1 }, "auth_pins": 0, "is_frozen": false, "is_freezing": false, "pins": { "ptrwaiter": 0, "request": 1, "lock": 1, "importingcaps": 0, "caps": 0, "exportingcaps": 0, "replicated": 0, "dirty": 0, "authpin": 0, "tempexporting": 0 }, "nref": 2 }, "object_string": "[inode 0x10000000241 [7,head] /volumes/qa/sv_1/b206794e-7622-4ab8-8943-f3ae4e65b7a3/client.0/tmp/clients/client0/~dmtmp/SEED/SMALL.FIL rep@0.1 v646 s=327680 n(v0 rc2024-04-23T10:29:16.539476+0000 b327680 1=1+0)/n(v0 rc2024-04-23T10:28:22.485704+0000 b327680 1=1+0) (iquiesce lock x=1 by request(mds.2:137 nref=3)) | ptrwaiter=0 request=1 lock=1 importingcaps=0 caps=0 exportingcaps=0 replicated=0 dirty=0 authpin=0 tempexporting=0 0x55c68bc80c00]", "lock": { "gather_set": [], "state": "lock", "type": "iquiesce", "is_leased": false, "num_rdlocks": 0, "num_wrlocks": 0, "num_xlocks": 1, "xlock_by": { "reqid": { "entity": { "type": "mds", "num": 2 }, "tid": 137 } } }, "flags": 4, "wrlock_target": -1 } ] } }, { "description": "internal op quiesce_inode:mds.2:117 fp=#0x10000000215", "initiated_at": "2024-04-23T10:31:21.463094+0000", "age": 42.365650399000003, "duration": 42.366730732999997, "continuous": true, "type_data": { "result": -2147483648, "flag_point": "quiesce complete for non-auth inode", "reqid": { "entity": { "type": "mds", "num": 2 }, "tid": 117 }, "op_type": "internal_op", "internal_op": 5385, "op_name": "quiesce_inode", "events": [ { "time": "2024-04-23T10:31:21.463094+0000", "event": "initiated" }, { "time": "2024-04-23T10:31:21.463094+0000", "event": "header_read" }, { "time": "2024-04-23T10:31:21.463094+0000", "event": "throttled" }, { "time": "2024-04-23T10:31:21.463094+0000", "event": "all_read" }, { "time": "2024-04-23T10:31:21.463094+0000", "event": "dispatched" }, { "time": "2024-04-23T10:31:21.463297+0000", "event": "requesting remote authpins" }, { "time": "2024-04-23T10:31:21.477857+0000", "event": "acquired locks" }, { "time": "2024-04-23T10:31:21.506084+0000", "event": "quiesce complete for non-auth inode" } ], "locks": [ { "object": { "is_auth": false, "auth_state": { "replicas": {} }, "replica_state": { "authority": [ 0, -2 ], "replica_nonce": 1 }, "auth_pins": 0, "is_frozen": false, "is_freezing": false, "pins": { "dirtyscattered": 0, "request": 1, "lock": 1, "importing": 0, "dirfrag": 1, "caps": 1, "dirwaiter": 0, "waiter": 0, "discoverbase": 0 }, "nref": 4 }, "object_string": "[inode 0x10000000215 [...7,head] /volumes/qa/sv_1/b206794e-7622-4ab8-8943-f3ae4e65b7a3/client.0/tmp/clients/client0/~dmtmp/COREL/ rep@0.1 v45589 f(v1232 m2024-04-23T10:31:17.745716+0000 11=11+0) n(v19 rc2024-04-23T10:31:17.745716+0000 b2861056 12=11+1)/n(v0 rc2024-04-23T10:27:52.004397+0000 2=1+1) (inest mix) (iquiesce lock x=1 by request(mds.2:117 nref=3)) caps={15312=pAsLsXsFs/pAsLsXsFsxcral@5564} | dirtyscattered=0 request=1 lock=1 importing=0 dirfrag=1 caps=1 dirwaiter=0 waiter=0 discoverbase=0 0x55c68bbfa580]", "lock": { "gather_set": [], "state": "lock", "type": "iquiesce", "is_leased": false, "num_rdlocks": 0, "num_wrlocks": 0, "num_xlocks": 1, "xlock_by": { "reqid": { "entity": { "type": "mds", "num": 2 }, "tid": 117 } } }, "flags": 4, "wrlock_target": -1 } ] } }, { "description": "internal op quiesce_inode:mds.2:138 fp=#0x10000000242", "initiated_at": "2024-04-23T10:31:21.489251+0000", "age": 42.339494180999999, "duration": 42.340645232999996, "continuous": true, "type_data": { "result": -2147483648, "flag_point": "quiesce complete for non-auth inode", "reqid": { "entity": { "type": "mds", "num": 2 }, "tid": 138 }, "op_type": "internal_op", "internal_op": 5385, "op_name": "quiesce_inode", "events": [ { "time": "2024-04-23T10:31:21.489251+0000", "event": "initiated" }, { "time": "2024-04-23T10:31:21.489251+0000", "event": "header_read" }, { "time": "2024-04-23T10:31:21.489251+0000", "event": "throttled" }, { "time": "2024-04-23T10:31:21.489251+0000", "event": "all_read" }, { "time": "2024-04-23T10:31:21.489251+0000", "event": "dispatched" }, { "time": "2024-04-23T10:31:21.489406+0000", "event": "requesting remote authpins" }, { "time": "2024-04-23T10:31:21.509077+0000", "event": "acquired locks" }, { "time": "2024-04-23T10:31:21.509083+0000", "event": "quiesce complete for non-auth inode" } ], "locks": [ { "object": { "is_auth": false, "auth_state": { "replicas": {} }, "replica_state": { "authority": [ 0, -2 ], "replica_nonce": 1 }, "auth_pins": 0, "is_frozen": false, "is_freezing": false, "pins": { "ptrwaiter": 0, "request": 1, "lock": 1, "importingcaps": 0, "caps": 0, "exportingcaps": 0, "replicated": 0, "dirty": 0, "authpin": 0, "tempexporting": 0 }, "nref": 2 }, "object_string": "[inode 0x10000000242 [7,head] /volumes/qa/sv_1/b206794e-7622-4ab8-8943-f3ae4e65b7a3/client.0/tmp/clients/client0/~dmtmp/SEED/MEDIUM.FIL rep@0.1 v650 s=1048576 n(v0 rc2024-04-23T10:29:16.539476+0000 b1048576 1=1+0)/n(v0 rc2024-04-23T10:28:22.485704+0000 b1048576 1=1+0) (iquiesce lock x=1 by request(mds.2:138 nref=3)) | ptrwaiter=0 request=1 lock=1 importingcaps=0 caps=0 exportingcaps=0 replicated=0 dirty=0 authpin=0 tempexporting=0 0x55c68bc81180]", "lock": { "gather_set": [], "state": "lock", "type": "iquiesce", "is_leased": false, "num_rdlocks": 0, "num_wrlocks": 0, "num_xlocks": 1, "xlock_by": { "reqid": { "entity": { "type": "mds", "num": 2 }, "tid": 138 } } }, "flags": 4, "wrlock_target": -1 } ] } }, { "description": "internal op quiesce_inode:mds.2:118 fp=#0x1000000020c", "initiated_at": "2024-04-23T10:31:21.463112+0000", "age": 42.365632798, "duration": 42.366852076999997, "continuous": true, "type_data": { "result": -2147483648, "flag_point": "quiesce complete for non-auth inode", "reqid": { "entity": { "type": "mds", "num": 2 }, "tid": 118 }, "op_type": "internal_op", "internal_op": 5385, "op_name": "quiesce_inode", "events": [ { "time": "2024-04-23T10:31:21.463112+0000", "event": "initiated" }, { "time": "2024-04-23T10:31:21.463112+0000", "event": "header_read" }, { "time": "2024-04-23T10:31:21.463112+0000", "event": "throttled" }, { "time": "2024-04-23T10:31:21.463112+0000", "event": "all_read" }, { "time": "2024-04-23T10:31:21.463112+0000", "event": "dispatched" }, { "time": "2024-04-23T10:31:21.463360+0000", "event": "requesting remote authpins" }, { "time": "2024-04-23T10:31:21.481931+0000", "event": "acquired locks" }, { "time": "2024-04-23T10:31:21.481945+0000", "event": "quiesce complete for non-auth inode" } ], "locks": [ { "object": { "is_auth": false, "auth_state": { "replicas": {} }, "replica_state": { "authority": [ 0, -2 ], "replica_nonce": 1 }, "auth_pins": 0, "is_frozen": false, "is_freezing": false, "pins": { "request": 1, "lock": 1, "dirwaiter": 0, "discoverbase": 0 }, "nref": 2 }, "object_string": "[inode 0x1000000020c [...7,head] /volumes/qa/sv_1/b206794e-7622-4ab8-8943-f3ae4e65b7a3/client.0/tmp/clients/client0/~dmtmp/EXCEL/ rep@0.1 v45489 f(v0 m2024-04-23T10:31:17.692717+0000 2=2+0) n(v0 rc2024-04-23T10:27:51.972397+0000 3=2+1)/n(v0 rc2024-04-23T10:27:51.967397+0000 2=1+1) (inest mix) (iquiesce lock x=1 by request(mds.2:118 nref=3)) | request=1 lock=1 dirwaiter=0 discoverbase=0 0x55c68b9d0000]", "lock": { "gather_set": [], "state": "lock", "type": "iquiesce", "is_leased": false, "num_rdlocks": 0, "num_wrlocks": 0, "num_xlocks": 1, "xlock_by": { "reqid": { "entity": { "type": "mds", "num": 2 }, "tid": 118 } } }, "flags": 4, "wrlock_target": -1 } ] } }, { "description": "internal op quiesce_inode:mds.2:139 fp=#0x10000000243", "initiated_at": "2024-04-23T10:31:21.489269+0000", "age": 42.339475419999999, "duration": 42.340755917000003, "continuous": true, "type_data": { "result": -2147483648, "flag_point": "quiesce complete for non-auth inode", "reqid": { "entity": { "type": "mds", "num": 2 }, "tid": 139 }, "op_type": "internal_op", "internal_op": 5385, "op_name": "quiesce_inode", "events": [ { "time": "2024-04-23T10:31:21.489269+0000", "event": "initiated" }, { "time": "2024-04-23T10:31:21.489269+0000", "event": "header_read" }, { "time": "2024-04-23T10:31:21.489269+0000", "event": "throttled" }, { "time": "2024-04-23T10:31:21.489269+0000", "event": "all_read" }, { "time": "2024-04-23T10:31:21.489269+0000", "event": "dispatched" }, { "time": "2024-04-23T10:31:21.489494+0000", "event": "requesting remote authpins" }, { "time": "2024-04-23T10:31:21.509204+0000", "event": "acquired locks" }, { "time": "2024-04-23T10:31:21.509216+0000", "event": "quiesce complete for non-auth inode" } ], "locks": [ { "object": { "is_auth": false, "auth_state": { "replicas": {} }, "replica_state": { "authority": [ 0, -2 ], "replica_nonce": 1 }, "auth_pins": 0, "is_frozen": false, "is_freezing": false, "pins": { "ptrwaiter": 0, "request": 1, "lock": 1, "importingcaps": 0, "caps": 0, "exportingcaps": 0, "replicated": 0, "dirty": 0, "authpin": 0, "tempexporting": 0 }, "nref": 2 }, "object_string": "[inode 0x10000000243 [7,head] /volumes/qa/sv_1/b206794e-7622-4ab8-8943-f3ae4e65b7a3/client.0/tmp/clients/client0/~dmtmp/SEED/LARGE.FIL rep@0.1 v648 s=1867776 n(v0 rc2024-04-23T10:29:16.539476+0000 b1867776 1=1+0)/n(v0 rc2024-04-23T10:28:22.485704+0000 b1867776 1=1+0) (iquiesce lock x=1 by request(mds.2:139 nref=3)) | ptrwaiter=0 request=1 lock=1 importingcaps=0 caps=0 exportingcaps=0 replicated=0 dirty=0 authpin=0 tempexporting=0 0x55c68bc81700]", "lock": { "gather_set": [], "state": "lock", "type": "iquiesce", "is_leased": false, "num_rdlocks": 0, "num_wrlocks": 0, "num_xlocks": 1, "xlock_by": { "reqid": { "entity": { "type": "mds", "num": 2 }, "tid": 139 } } }, "flags": 4, "wrlock_target": -1 } ] } }, { "description": "internal op quiesce_inode:mds.2:119 fp=#0x10000000208", "initiated_at": "2024-04-23T10:31:21.463127+0000", "age": 42.365618232000003, "duration": 42.366964119999999, "continuous": true, "type_data": { "result": -2147483648, "flag_point": "quiesce complete for non-auth inode", "reqid": { "entity": { "type": "mds", "num": 2 }, "tid": 119 }, "op_type": "internal_op", "internal_op": 5385, "op_name": "quiesce_inode", "events": [ { "time": "2024-04-23T10:31:21.463127+0000", "event": "initiated" }, { "time": "2024-04-23T10:31:21.463127+0000", "event": "header_read" }, { "time": "2024-04-23T10:31:21.463127+0000", "event": "throttled" }, { "time": "2024-04-23T10:31:21.463127+0000", "event": "all_read" }, { "time": "2024-04-23T10:31:21.463127+0000", "event": "dispatched" }, { "time": "2024-04-23T10:31:21.463424+0000", "event": "requesting remote authpins" }, { "time": "2024-04-23T10:31:21.482852+0000", "event": "acquired locks" }, { "time": "2024-04-23T10:31:21.482861+0000", "event": "quiesce complete for non-auth inode" } ], "locks": [ { "object": { "is_auth": false, "auth_state": { "replicas": {} }, "replica_state": { "authority": [ 0, -2 ], "replica_nonce": 1 }, "auth_pins": 0, "is_frozen": false, "is_freezing": false, "pins": { "request": 1, "lock": 1, "dirfrag": 0, "caps": 1, "dirwaiter": 0, "discoverbase": 0 }, "nref": 3 }, "object_string": "[inode 0x10000000208 [...7,head] /volumes/qa/sv_1/b206794e-7622-4ab8-8943-f3ae4e65b7a3/client.0/tmp/clients/client0/~dmtmp/WORD/ rep@0.1 v45514 f(v0 m2024-04-23T10:31:17.675718+0000 3=3+0) n(v0 rc2024-04-23T10:28:39.366321+0000 b780288 4=3+1) old_inodes=1 (inest mix) (iquiesce lock x=1 by request(mds.2:119 nref=3)) caps={15312=pAsLsXsFs/pAsLsXsFsxcral@1378} | request=1 lock=1 dirfrag=0 caps=1 dirwaiter=0 discoverbase=0 0x55c68c112100]", "lock": { "gather_set": [], "state": "lock", "type": "iquiesce", "is_leased": false, "num_rdlocks": 0, "num_wrlocks": 0, "num_xlocks": 1, "xlock_by": { "reqid": { "entity": { "type": "mds", "num": 2 }, "tid": 119 } } }, "flags": 4, "wrlock_target": -1 } ] } }, { "description": "internal op quiesce_inode:mds.2:140 fp=#0x10000000226", "initiated_at": "2024-04-23T10:31:21.494635+0000", "age": 42.334109484000003, "duration": 42.335519542999997, "continuous": true, "type_data": { "result": -2147483648, "flag_point": "quiesce complete", "reqid": { "entity": { "type": "mds", "num": 2 }, "tid": 140 }, "op_type": "internal_op", "internal_op": 5385, "op_name": "quiesce_inode", "events": [ { "time": "2024-04-23T10:31:21.494635+0000", "event": "initiated" }, { "time": "2024-04-23T10:31:21.494635+0000", "event": "header_read" }, { "time": "2024-04-23T10:31:21.494635+0000", "event": "throttled" }, { "time": "2024-04-23T10:31:21.494635+0000", "event": "all_read" }, { "time": "2024-04-23T10:31:21.494635+0000", "event": "dispatched" }, { "time": "2024-04-23T10:31:21.494909+0000", "event": "acquired locks" }, { "time": "2024-04-23T10:31:21.495384+0000", "event": "quiesce complete" } ], "locks": [ { "object": { "is_auth": true, "auth_state": { "replicas": {} }, "replica_state": { "authority": [ 2, -2 ], "replica_nonce": 0 }, "auth_pins": 1, "is_frozen": false, "is_freezing": false, "pins": { "ptrwaiter": 0, "request": 1, "lock": 1, "importingcaps": 0, "caps": 0, "needsnapflush": 0, "dirtyrstat": 0, "replicated": 0, "dirty": 1, "waiter": 0, "authpin": 1 }, "nref": 4 }, "object_string": "[inode 0x10000000226 [7,head] /volumes/qa/sv_1/b206794e-7622-4ab8-8943-f3ae4e65b7a3/client.0/tmp/clients/client0/~dmtmp/PM/T1.XLS auth v16150 ap=1 s=260096 n(v0 rc2024-04-23T10:27:52.072395+0000 b260096 1=1+0) (iquiesce lock x=1 by request(mds.2:140 nref=3)) | ptrwaiter=0 request=1 lock=1 importingcaps=0 caps=0 needsnapflush=0 dirtyrstat=0 replicated=0 dirty=1 waiter=0 authpin=1 0x55c68bc4e680]", "lock": { "gather_set": [], "state": "lock", "type": "iquiesce", "is_leased": false, "num_rdlocks": 0, "num_wrlocks": 0, "num_xlocks": 1, "xlock_by": { "reqid": { "entity": { "type": "mds", "num": 2 }, "tid": 140 } } }, "flags": 4, "wrlock_target": -1 } ] } }, { "description": "internal op quiesce_inode:mds.2:120 fp=#0x10000000240", "initiated_at": "2024-04-23T10:31:21.463144+0000", "age": 42.365600370000003, "duration": 42.367073902000001, "continuous": true, "type_data": { "result": -2147483648, "flag_point": "quiesce complete for non-auth inode", "reqid": { "entity": { "type": "mds", "num": 2 }, "tid": 120 }, "op_type": "internal_op", "internal_op": 5385, "op_name": "quiesce_inode", "events": [ { "time": "2024-04-23T10:31:21.463144+0000", "event": "initiated" }, { "time": "2024-04-23T10:31:21.463144+0000", "event": "header_read" }, { "time": "2024-04-23T10:31:21.463144+0000", "event": "throttled" }, { "time": "2024-04-23T10:31:21.463144+0000", "event": "all_read" }, { "time": "2024-04-23T10:31:21.463144+0000", "event": "dispatched" }, { "time": "2024-04-23T10:31:21.463539+0000", "event": "requesting remote authpins" }, { "time": "2024-04-23T10:31:21.489203+0000", "event": "acquired locks" }, { "time": "2024-04-23T10:31:21.509244+0000", "event": "quiesce complete for non-auth inode" } ], "locks": [ { "object": { "is_auth": false, "auth_state": { "replicas": {} }, "replica_state": { "authority": [ 0, -2 ], "replica_nonce": 1 }, "auth_pins": 0, "is_frozen": false, "is_freezing": false, "pins": { "dirtyscattered": 0, "request": 1, "lock": 1, "importing": 0, "dirfrag": 1, "caps": 1, "dirwaiter": 0, "waiter": 0, "discoverbase": 0 }, "nref": 4 }, "object_string": "[inode 0x10000000240 [...7,head] /volumes/qa/sv_1/b206794e-7622-4ab8-8943-f3ae4e65b7a3/client.0/tmp/clients/client0/~dmtmp/SEED/ rep@0.1 v45361 f(v0 m2024-04-23T10:27:52.196392+0000 3=3+0) n(v2 rc2024-04-23T10:28:22.485704+0000 b3244032 4=3+1)/n(v0 rc2024-04-23T10:27:52.187392+0000 2=1+1) (inest mix) (iquiesce lock x=1 by request(mds.2:120 nref=3)) caps={15312=pAsLsXsFs/pAsLsXsFsxcral@15} | dirtyscattered=0 request=1 lock=1 importing=0 dirfrag=1 caps=1 dirwaiter=0 waiter=0 discoverbase=0 0x55c68bbfbb80]", "lock": { "gather_set": [], "state": "lock", "type": "iquiesce", "is_leased": false, "num_rdlocks": 0, "num_wrlocks": 0, "num_xlocks": 1, "xlock_by": { "reqid": { "entity": { "type": "mds", "num": 2 }, "tid": 120 } } }, "flags": 4, "wrlock_target": -1 } ] } }, { "description": "internal op quiesce_inode:mds.2:141 fp=#0x10000000227", "initiated_at": "2024-04-23T10:31:21.494648+0000", "age": 42.334096457999998, "duration": 42.335637937999998, "continuous": true, "type_data": { "result": -2147483648, "flag_point": "quiesce complete", "reqid": { "entity": { "type": "mds", "num": 2 }, "tid": 141 }, "op_type": "internal_op", "internal_op": 5385, "op_name": "quiesce_inode", "events": [ { "time": "2024-04-23T10:31:21.494648+0000", "event": "initiated" }, { "time": "2024-04-23T10:31:21.494648+0000", "event": "header_read" }, { "time": "2024-04-23T10:31:21.494648+0000", "event": "throttled" }, { "time": "2024-04-23T10:31:21.494648+0000", "event": "all_read" }, { "time": "2024-04-23T10:31:21.494648+0000", "event": "dispatched" }, { "time": "2024-04-23T10:31:21.495697+0000", "event": "failed to xlock, waiting" }, { "time": "2024-04-23T10:31:21.496927+0000", "event": "acquired locks" }, { "time": "2024-04-23T10:31:21.497138+0000", "event": "quiesce complete" } ], "locks": [ { "object": { "is_auth": true, "auth_state": { "replicas": {} }, "replica_state": { "authority": [ 2, -2 ], "replica_nonce": 0 }, "auth_pins": 1, "is_frozen": false, "is_freezing": false, "pins": { "ptrwaiter": 0, "request": 1, "lock": 1, "importingcaps": 0, "caps": 0, "replicated": 0, "dirty": 1, "waiter": 0, "authpin": 1 }, "nref": 4 }, "object_string": "[inode 0x10000000227 [7,head] /volumes/qa/sv_1/b206794e-7622-4ab8-8943-f3ae4e65b7a3/client.0/tmp/clients/client0/~dmtmp/PM/EVENTS.DOC auth v16153 ap=1 s=260096 n(v0 rc2024-04-23T10:27:52.076395+0000 b260096 1=1+0) (iquiesce lock x=1 by request(mds.2:141 nref=3)) | ptrwaiter=0 request=1 lock=1 importingcaps=0 caps=0 replicated=0 dirty=1 waiter=0 authpin=1 0x55c68bc4ec00]", "lock": { "gather_set": [], "state": "lock", "type": "iquiesce", "is_leased": false, "num_rdlocks": 0, "num_wrlocks": 0, "num_xlocks": 1, "xlock_by": { "reqid": { "entity": { "type": "mds", "num": 2 }, "tid": 141 } } }, "flags": 4, "wrlock_target": -1 } ] } }, { "description": "internal op quiesce_inode:mds.2:121 fp=#0x10000000221", "initiated_at": "2024-04-23T10:31:21.463157+0000", "age": 42.365587916999999, "duration": 42.367192140999997, "continuous": true, "type_data": { "result": -2147483648, "flag_point": "quiesce complete for non-auth inode", "reqid": { "entity": { "type": "mds", "num": 2 }, "tid": 121 }, "op_type": "internal_op", "internal_op": 5385, "op_name": "quiesce_inode", "events": [ { "time": "2024-04-23T10:31:21.463157+0000", "event": "initiated" }, { "time": "2024-04-23T10:31:21.463157+0000", "event": "header_read" }, { "time": "2024-04-23T10:31:21.463157+0000", "event": "throttled" }, { "time": "2024-04-23T10:31:21.463157+0000", "event": "all_read" }, { "time": "2024-04-23T10:31:21.463157+0000", "event": "dispatched" }, { "time": "2024-04-23T10:31:21.463611+0000", "event": "requesting remote authpins" }, { "time": "2024-04-23T10:31:21.490252+0000", "event": "acquired locks" }, { "time": "2024-04-23T10:31:21.490264+0000", "event": "quiesce complete for non-auth inode" } ], "locks": [ { "object": { "is_auth": false, "auth_state": { "replicas": {} }, "replica_state": { "authority": [ 0, -2 ], "replica_nonce": 1 }, "auth_pins": 0, "is_frozen": false, "is_freezing": false, "pins": { "request": 1, "lock": 1, "dirwaiter": 0, "discoverbase": 0 }, "nref": 2 }, "object_string": "[inode 0x10000000221 [...7,head] /volumes/qa/sv_1/b206794e-7622-4ab8-8943-f3ae4e65b7a3/client.0/tmp/clients/client0/~dmtmp/WORDPRO/ rep@0.1 v45516 f(v0 m2024-04-23T10:31:17.760716+0000 3=3+0) n(v0 rc2024-04-23T10:28:42.855241+0000 b780288 4=3+1) old_inodes=1 (inest mix) (iquiesce lock x=1 by request(mds.2:121 nref=3)) | request=1 lock=1 dirwaiter=0 discoverbase=0 0x55c68c0d6c00]", "lock": { "gather_set": [], "state": "lock", "type": "iquiesce", "is_leased": false, "num_rdlocks": 0, "num_wrlocks": 0, "num_xlocks": 1, "xlock_by": { "reqid": { "entity": { "type": "mds", "num": 2 }, "tid": 121 } } }, "flags": 4, "wrlock_target": -1 } ] } }, { "description": "internal op quiesce_inode:mds.2:142 fp=#0x10000000228", "initiated_at": "2024-04-23T10:31:21.494670+0000", "age": 42.334074563999998, "duration": 42.335740854999997, "continuous": true, "type_data": { "result": -2147483648, "flag_point": "quiesce complete", "reqid": { "entity": { "type": "mds", "num": 2 }, "tid": 142 }, "op_type": "internal_op", "internal_op": 5385, "op_name": "quiesce_inode", "events": [ { "time": "2024-04-23T10:31:21.494670+0000", "event": "initiated" }, { "time": "2024-04-23T10:31:21.494670+0000", "event": "header_read" }, { "time": "2024-04-23T10:31:21.494670+0000", "event": "throttled" }, { "time": "2024-04-23T10:31:21.494670+0000", "event": "all_read" }, { "time": "2024-04-23T10:31:21.494670+0000", "event": "dispatched" }, { "time": "2024-04-23T10:31:21.495979+0000", "event": "failed to xlock, waiting" }, { "time": "2024-04-23T10:31:21.497990+0000", "event": "acquired locks" }, { "time": "2024-04-23T10:31:21.498222+0000", "event": "quiesce complete" } ], "locks": [ { "object": { "is_auth": true, "auth_state": { "replicas": {} }, "replica_state": { "authority": [ 2, -2 ], "replica_nonce": 0 }, "auth_pins": 1, "is_frozen": false, "is_freezing": false, "pins": { "ptrwaiter": 0, "request": 1, "lock": 1, "importingcaps": 0, "caps": 0, "replicated": 0, "dirty": 1, "waiter": 0, "authpin": 1 }, "nref": 4 }, "object_string": "[inode 0x10000000228 [7,head] /volumes/qa/sv_1/b206794e-7622-4ab8-8943-f3ae4e65b7a3/client.0/tmp/clients/client0/~dmtmp/PM/MOVED.DOC auth v16155 ap=1 s=260096 n(v0 rc2024-04-23T10:27:52.081395+0000 b260096 1=1+0) (iquiesce lock x=1 by request(mds.2:142 nref=3)) | ptrwaiter=0 request=1 lock=1 importingcaps=0 caps=0 replicated=0 dirty=1 waiter=0 authpin=1 0x55c68bc4f180]", "lock": { "gather_set": [], "state": "lock", "type": "iquiesce", "is_leased": false, "num_rdlocks": 0, "num_wrlocks": 0, "num_xlocks": 1, "xlock_by": { "reqid": { "entity": { "type": "mds", "num": 2 }, "tid": 142 } } }, "flags": 4, "wrlock_target": -1 } ] } }, { "description": "internal op quiesce_inode:mds.2:122 fp=#0x10000000229", "initiated_at": "2024-04-23T10:31:21.463170+0000", "age": 42.365574952999999, "duration": 42.367315818999998, "continuous": true, "type_data": { "result": -2147483648, "flag_point": "quiesce complete for non-auth inode", "reqid": { "entity": { "type": "mds", "num": 2 }, "tid": 122 }, "op_type": "internal_op", "internal_op": 5385, "op_name": "quiesce_inode", "events": [ { "time": "2024-04-23T10:31:21.463170+0000", "event": "initiated" }, { "time": "2024-04-23T10:31:21.463170+0000", "event": "header_read" }, { "time": "2024-04-23T10:31:21.463170+0000", "event": "throttled" }, { "time": "2024-04-23T10:31:21.463170+0000", "event": "all_read" }, { "time": "2024-04-23T10:31:21.463170+0000", "event": "dispatched" }, { "time": "2024-04-23T10:31:21.463682+0000", "event": "requesting remote authpins" }, { "time": "2024-04-23T10:31:21.493342+0000", "event": "acquired locks" }, { "time": "2024-04-23T10:31:21.493351+0000", "event": "quiesce complete for non-auth inode" } ], "locks": [ { "object": { "is_auth": false, "auth_state": { "replicas": {} }, "replica_state": { "authority": [ 0, -2 ], "replica_nonce": 1 }, "auth_pins": 0, "is_frozen": false, "is_freezing": false, "pins": { "request": 1, "lock": 1, "dirfrag": 0, "dirwaiter": 0, "discoverbase": 0 }, "nref": 2 }, "object_string": "[inode 0x10000000229 [...7,head] /volumes/qa/sv_1/b206794e-7622-4ab8-8943-f3ae4e65b7a3/client.0/tmp/clients/client0/~dmtmp/PARADOX/ rep@0.1 v45567 f(v1976 m2024-04-23T10:31:16.541744+0000 24=24+0) n(v19 rc2024-04-23T10:31:16.541744+0000 b5201920 25=24+1)/n(v0 rc2024-04-23T10:27:52.084395+0000 2=1+1) (inest mix) (iquiesce lock x=1 by request(mds.2:122 nref=3)) | request=1 lock=1 dirfrag=0 dirwaiter=0 discoverbase=0 0x55c68bbfb600]", "lock": { "gather_set": [], "state": "lock", "type": "iquiesce", "is_leased": false, "num_rdlocks": 0, "num_wrlocks": 0, "num_xlocks": 1, "xlock_by": { "reqid": { "entity": { "type": "mds", "num": 2 }, "tid": 122 } } }, "flags": 4, "wrlock_target": -1 } ] } }, { "description": "internal op quiesce_inode:mds.2:143 fp=#0x10000000210", "initiated_at": "2024-04-23T10:31:21.498646+0000", "age": 42.330098722999999, "duration": 42.331903072000003, "continuous": true, "type_data": { "result": -2147483648, "flag_point": "quiesce complete", "reqid": { "entity": { "type": "mds", "num": 2 }, "tid": 143 }, "op_type": "internal_op", "internal_op": 5385, "op_name": "quiesce_inode", "events": [ { "time": "2024-04-23T10:31:21.498646+0000", "event": "initiated" }, { "time": "2024-04-23T10:31:21.498646+0000", "event": "header_read" }, { "time": "2024-04-23T10:31:21.498646+0000", "event": "throttled" }, { "time": "2024-04-23T10:31:21.498646+0000", "event": "all_read" }, { "time": "2024-04-23T10:31:21.498646+0000", "event": "dispatched" }, { "time": "2024-04-23T10:31:21.498985+0000", "event": "failed to xlock, waiting" }, { "time": "2024-04-23T10:31:21.500907+0000", "event": "failed to xlock, waiting" }, { "time": "2024-04-23T10:31:21.506656+0000", "event": "acquired locks" }, { "time": "2024-04-23T10:31:21.506825+0000", "event": "quiesce complete" } ], "locks": [ { "object": { "is_auth": true, "auth_state": { "replicas": {} }, "replica_state": { "authority": [ 2, -2 ], "replica_nonce": 0 }, "auth_pins": 1, "is_frozen": false, "is_freezing": false, "pins": { "ptrwaiter": 0, "request": 1, "lock": 1, "importingcaps": 0, "caps": 0, "needsnapflush": 0, "replicated": 0, "dirty": 1, "waiter": 0, "authpin": 1 }, "nref": 4 }, "object_string": "[inode 0x10000000210 [7,head] /volumes/qa/sv_1/b206794e-7622-4ab8-8943-f3ae4e65b7a3/client.0/tmp/clients/client0/~dmtmp/PWRPNT/PCBENCHM.PPT auth v12260 ap=1 s=260096 n(v0 rc2024-04-23T10:31:15.901758+0000 b260096 1=1+0) (iquiesce lock x=1 by request(mds.2:143 nref=3)) | ptrwaiter=0 request=1 lock=1 importingcaps=0 caps=0 needsnapflush=0 replicated=0 dirty=1 waiter=0 authpin=1 0x55c68bc7e000]", "lock": { "gather_set": [], "state": "lock", "type": "iquiesce", "is_leased": false, "num_rdlocks": 0, "num_wrlocks": 0, "num_xlocks": 1, "xlock_by": { "reqid": { "entity": { "type": "mds", "num": 2 }, "tid": 143 } } }, "flags": 4, "wrlock_target": -1 } ] } }, { "description": "internal op quiesce_inode:mds.2:123 fp=#0x10000000225", "initiated_at": "2024-04-23T10:31:21.463187+0000", "age": 42.365558229999998, "duration": 42.367428386, "continuous": true, "type_data": { "result": -2147483648, "flag_point": "quiesce complete for non-auth inode", "reqid": { "entity": { "type": "mds", "num": 2 }, "tid": 123 }, "op_type": "internal_op", "internal_op": 5385, "op_name": "quiesce_inode", "events": [ { "time": "2024-04-23T10:31:21.463187+0000", "event": "initiated" }, { "time": "2024-04-23T10:31:21.463187+0000", "event": "header_read" }, { "time": "2024-04-23T10:31:21.463187+0000", "event": "throttled" }, { "time": "2024-04-23T10:31:21.463187+0000", "event": "all_read" }, { "time": "2024-04-23T10:31:21.463187+0000", "event": "dispatched" }, { "time": "2024-04-23T10:31:21.463762+0000", "event": "requesting remote authpins" }, { "time": "2024-04-23T10:31:21.494626+0000", "event": "acquired locks" }, { "time": "2024-04-23T10:31:21.498260+0000", "event": "quiesce complete for non-auth inode" } ], "locks": [ { "object": { "is_auth": false, "auth_state": { "replicas": {} }, "replica_state": { "authority": [ 0, -2 ], "replica_nonce": 1 }, "auth_pins": 0, "is_frozen": false, "is_freezing": false, "pins": { "dirtyscattered": 0, "request": 1, "lock": 1, "importing": 0, "dirfrag": 1, "caps": 1, "dirwaiter": 0, "waiter": 0, "discoverbase": 0 }, "nref": 4 }, "object_string": "[inode 0x10000000225 [...7,head] /volumes/qa/sv_1/b206794e-7622-4ab8-8943-f3ae4e65b7a3/client.0/tmp/clients/client0/~dmtmp/PM/ rep@0.1 v45574 f(v2926 m2024-04-23T10:31:17.764716+0000 3=3+0) n(v17 rc2024-04-23T10:31:17.764716+0000 b780288 4=3+1)/n(v0 rc2024-04-23T10:27:52.071395+0000 2=1+1) (inest mix) (iquiesce lock x=1 by request(mds.2:123 nref=3)) caps={15312=pAsLsXsFs/pAsLsXsFsxcral@20013} | dirtyscattered=0 request=1 lock=1 importing=0 dirfrag=1 caps=1 dirwaiter=0 waiter=0 discoverbase=0 0x55c68bbfb080]", "lock": { "gather_set": [], "state": "lock", "type": "iquiesce", "is_leased": false, "num_rdlocks": 0, "num_wrlocks": 0, "num_xlocks": 1, "xlock_by": { "reqid": { "entity": { "type": "mds", "num": 2 }, "tid": 123 } } }, "flags": 4, "wrlock_target": -1 } ] } }, { "description": "internal op quiesce_inode:mds.2:144 fp=#0x10000000211", "initiated_at": "2024-04-23T10:31:21.498664+0000", "age": 42.330080453999997, "duration": 42.332019819999999, "continuous": true, "type_data": { "result": -2147483648, "flag_point": "quiesce complete", "reqid": { "entity": { "type": "mds", "num": 2 }, "tid": 144 }, "op_type": "internal_op", "internal_op": 5385, "op_name": "quiesce_inode", "events": [ { "time": "2024-04-23T10:31:21.498664+0000", "event": "initiated" }, { "time": "2024-04-23T10:31:21.498664+0000", "event": "header_read" }, { "time": "2024-04-23T10:31:21.498664+0000", "event": "throttled" }, { "time": "2024-04-23T10:31:21.498664+0000", "event": "all_read" }, { "time": "2024-04-23T10:31:21.498664+0000", "event": "dispatched" }, { "time": "2024-04-23T10:31:21.499221+0000", "event": "failed to xlock, waiting" }, { "time": "2024-04-23T10:31:21.502033+0000", "event": "failed to xlock, waiting" }, { "time": "2024-04-23T10:31:21.507099+0000", "event": "acquired locks" }, { "time": "2024-04-23T10:31:21.507266+0000", "event": "quiesce complete" } ], "locks": [ { "object": { "is_auth": true, "auth_state": { "replicas": {} }, "replica_state": { "authority": [ 2, -2 ], "replica_nonce": 0 }, "auth_pins": 1, "is_frozen": false, "is_freezing": false, "pins": { "ptrwaiter": 0, "request": 1, "lock": 1, "importingcaps": 0, "caps": 0, "replicated": 0, "dirty": 1, "waiter": 0, "authpin": 1 }, "nref": 4 }, "object_string": "[inode 0x10000000211 [4,head] /volumes/qa/sv_1/b206794e-7622-4ab8-8943-f3ae4e65b7a3/client.0/tmp/clients/client0/~dmtmp/PWRPNT/ZD16.BMP auth v12262 ap=1 s=260096 n(v0 rc2024-04-23T10:27:51.983397+0000 b260096 1=1+0) (iquiesce lock x=1 by request(mds.2:144 nref=3)) | ptrwaiter=0 request=1 lock=1 importingcaps=0 caps=0 replicated=0 dirty=1 waiter=0 authpin=1 0x55c68bc7e580]", "lock": { "gather_set": [], "state": "lock", "type": "iquiesce", "is_leased": false, "num_rdlocks": 0, "num_wrlocks": 0, "num_xlocks": 1, "xlock_by": { "reqid": { "entity": { "type": "mds", "num": 2 }, "tid": 144 } } }, "flags": 4, "wrlock_target": -1 } ] } }, { "description": "internal op quiesce_inode:mds.2:124 fp=#0x1000000020f", "initiated_at": "2024-04-23T10:31:21.463203+0000", "age": 42.365541929000003, "duration": 42.367729769, "continuous": true, "type_data": { "result": -2147483648, "flag_point": "quiesce complete for non-auth inode", "reqid": { "entity": { "type": "mds", "num": 2 }, "tid": 124 }, "op_type": "internal_op", "internal_op": 5385, "op_name": "quiesce_inode", "events": [ { "time": "2024-04-23T10:31:21.463203+0000", "event": "initiated" }, { "time": "2024-04-23T10:31:21.463203+0000", "event": "header_read" }, { "time": "2024-04-23T10:31:21.463203+0000", "event": "throttled" }, { "time": "2024-04-23T10:31:21.463203+0000", "event": "all_read" }, { "time": "2024-04-23T10:31:21.463203+0000", "event": "dispatched" }, { "time": "2024-04-23T10:31:21.463869+0000", "event": "requesting remote authpins" }, { "time": "2024-04-23T10:31:21.498627+0000", "event": "acquired locks" }, { "time": "2024-04-23T10:31:21.508694+0000", "event": "quiesce complete for non-auth inode" } ], "locks": [ { "object": { "is_auth": false, "auth_state": { "replicas": {} }, "replica_state": { "authority": [ 0, -2 ], "replica_nonce": 1 }, "auth_pins": 0, "is_frozen": false, "is_freezing": false, "pins": { "dirtyscattered": 0, "request": 1, "lock": 1, "importing": 0, "dirfrag": 1, "caps": 1, "dirwaiter": 0, "waiter": 0, "discoverbase": 0 }, "nref": 4 }, "object_string": "[inode 0x1000000020f [...7,head] /volumes/qa/sv_1/b206794e-7622-4ab8-8943-f3ae4e65b7a3/client.0/tmp/clients/client0/~dmtmp/PWRPNT/ rep@0.1 v45572 f(v1293 m2024-04-23T10:31:17.718717+0000 5=5+0) n(v18 rc2024-04-23T10:31:17.718717+0000 b1300480 6=5+1)/n(v0 rc2024-04-23T10:27:51.976397+0000 2=1+1) (inest mix) (iquiesce lock x=1 by request(mds.2:124 nref=3)) caps={15312=pAsLsXsFs/pAsLsXsFsxcral@6268} | dirtyscattered=0 request=1 lock=1 importing=0 dirfrag=1 caps=1 dirwaiter=0 waiter=0 discoverbase=0 0x55c68bbfa000]", "lock": { "gather_set": [], "state": "lock", "type": "iquiesce", "is_leased": false, "num_rdlocks": 0, "num_wrlocks": 0, "num_xlocks": 1, "xlock_by": { "reqid": { "entity": { "type": "mds", "num": 2 }, "tid": 124 } } }, "flags": 4, "wrlock_target": -1 } ] } }, { "description": "internal op quiesce_inode:mds.2:145 fp=#0x10000000214", "initiated_at": "2024-04-23T10:31:21.498682+0000", "age": 42.330062400000003, "duration": 42.332320840000001, "continuous": true, "type_data": { "result": -2147483648, "flag_point": "quiesce complete", "reqid": { "entity": { "type": "mds", "num": 2 }, "tid": 145 }, "op_type": "internal_op", "internal_op": 5385, "op_name": "quiesce_inode", "events": [ { "time": "2024-04-23T10:31:21.498682+0000", "event": "initiated" }, { "time": "2024-04-23T10:31:21.498682+0000", "event": "header_read" }, { "time": "2024-04-23T10:31:21.498682+0000", "event": "throttled" }, { "time": "2024-04-23T10:31:21.498682+0000", "event": "all_read" }, { "time": "2024-04-23T10:31:21.498682+0000", "event": "dispatched" }, { "time": "2024-04-23T10:31:21.499441+0000", "event": "failed to xlock, waiting" }, { "time": "2024-04-23T10:31:21.502467+0000", "event": "acquired locks" }, { "time": "2024-04-23T10:31:21.502711+0000", "event": "quiesce complete" } ], "locks": [ { "object": { "is_auth": true, "auth_state": { "replicas": {} }, "replica_state": { "authority": [ 2, -2 ], "replica_nonce": 0 }, "auth_pins": 1, "is_frozen": false, "is_freezing": false, "pins": { "ptrwaiter": 0, "request": 1, "lock": 1, "importingcaps": 0, "caps": 0, "needsnapflush": 0, "replicated": 0, "dirty": 1, "waiter": 0, "authpin": 1 }, "nref": 4 }, "object_string": "[inode 0x10000000214 [7,head] /volumes/qa/sv_1/b206794e-7622-4ab8-8943-f3ae4e65b7a3/client.0/tmp/clients/client0/~dmtmp/PWRPNT/TRIDOTS.POT auth v12245 ap=1 s=260096 n(v0 rc2024-04-23T10:27:52.001397+0000 b260096 1=1+0) (iquiesce lock x=1 by request(mds.2:145 nref=3)) | ptrwaiter=0 request=1 lock=1 importingcaps=0 caps=0 needsnapflush=0 replicated=0 dirty=1 waiter=0 authpin=1 0x55c68bc7f080]", "lock": { "gather_set": [], "state": "lock", "type": "iquiesce", "is_leased": false, "num_rdlocks": 0, "num_wrlocks": 0, "num_xlocks": 1, "xlock_by": { "reqid": { "entity": { "type": "mds", "num": 2 }, "tid": 145 } } }, "flags": 4, "wrlock_target": -1 } ] } }, { "description": "internal op quiesce_inode:mds.2:125 fp=#0x1000000023e", "initiated_at": "2024-04-23T10:31:21.463216+0000", "age": 42.365528842000003, "duration": 42.367851899999998, "continuous": true, "type_data": { "result": -2147483648, "flag_point": "quiesce complete for non-auth inode", "reqid": { "entity": { "type": "mds", "num": 2 }, "tid": 125 }, "op_type": "internal_op", "internal_op": 5385, "op_name": "quiesce_inode", "events": [ { "time": "2024-04-23T10:31:21.463216+0000", "event": "initiated" }, { "time": "2024-04-23T10:31:21.463216+0000", "event": "header_read" }, { "time": "2024-04-23T10:31:21.463216+0000", "event": "throttled" }, { "time": "2024-04-23T10:31:21.463216+0000", "event": "all_read" }, { "time": "2024-04-23T10:31:21.463216+0000", "event": "dispatched" }, { "time": "2024-04-23T10:31:21.463936+0000", "event": "requesting remote authpins" }, { "time": "2024-04-23T10:31:21.501193+0000", "event": "acquired locks" }, { "time": "2024-04-23T10:31:21.501205+0000", "event": "quiesce complete for non-auth inode" } ], "locks": [ { "object": { "is_auth": false, "auth_state": { "replicas": {} }, "replica_state": { "authority": [ 0, -2 ], "replica_nonce": 1 }, "auth_pins": 0, "is_frozen": false, "is_freezing": false, "pins": { "request": 1, "lock": 1, "dirwaiter": 0, "discoverbase": 0 }, "nref": 2 }, "object_string": "[inode 0x1000000023e [...7,head] /volumes/qa/sv_1/b206794e-7622-4ab8-8943-f3ae4e65b7a3/client.0/tmp/clients/client0/~dmtmp/ACCESS/ rep@0.1 v45357 f(v0 m2024-04-23T10:31:17.039732+0000 4=4+0) n(v0 rc2024-04-23T10:28:42.888241+0000 b260096 2=1+1) old_inodes=1 (inest mix) (iquiesce lock x=1 by request(mds.2:125 nref=3)) | request=1 lock=1 dirwaiter=0 discoverbase=0 0x55c68c138580]", "lock": { "gather_set": [], "state": "lock", "type": "iquiesce", "is_leased": false, "num_rdlocks": 0, "num_wrlocks": 0, "num_xlocks": 1, "xlock_by": { "reqid": { "entity": { "type": "mds", "num": 2 }, "tid": 125 } } }, "flags": 4, "wrlock_target": -1 } ] } }, { "description": "internal op quiesce_inode:mds.2:146 fp=#0x10000000213", "initiated_at": "2024-04-23T10:31:21.498698+0000", "age": 42.330046457000002, "duration": 42.332431438, "continuous": true, "type_data": { "result": -2147483648, "flag_point": "quiesce complete", "reqid": { "entity": { "type": "mds", "num": 2 }, "tid": 146 }, "op_type": "internal_op", "internal_op": 5385, "op_name": "quiesce_inode", "events": [ { "time": "2024-04-23T10:31:21.498698+0000", "event": "initiated" }, { "time": "2024-04-23T10:31:21.498698+0000", "event": "header_read" }, { "time": "2024-04-23T10:31:21.498698+0000", "event": "throttled" }, { "time": "2024-04-23T10:31:21.498698+0000", "event": "all_read" }, { "time": "2024-04-23T10:31:21.498698+0000", "event": "dispatched" }, { "time": "2024-04-23T10:31:21.499644+0000", "event": "failed to xlock, waiting" }, { "time": "2024-04-23T10:31:21.503691+0000", "event": "failed to xlock, waiting" }, { "time": "2024-04-23T10:31:21.507743+0000", "event": "acquired locks" }, { "time": "2024-04-23T10:31:21.508049+0000", "event": "quiesce complete" } ], "locks": [ { "object": { "is_auth": true, "auth_state": { "replicas": {} }, "replica_state": { "authority": [ 2, -2 ], "replica_nonce": 0 }, "auth_pins": 1, "is_frozen": false, "is_freezing": false, "pins": { "ptrwaiter": 0, "request": 1, "lock": 1, "importingcaps": 0, "caps": 0, "needsnapflush": 0, "replicated": 0, "dirty": 1, "waiter": 0, "authpin": 1 }, "nref": 4 }, "object_string": "[inode 0x10000000213 [7,head] /volumes/qa/sv_1/b206794e-7622-4ab8-8943-f3ae4e65b7a3/client.0/tmp/clients/client0/~dmtmp/PWRPNT/TIPS.PPT auth v12268 ap=1 s=260096 n(v0 rc2024-04-23T10:31:16.010756+0000 b260096 1=1+0) (iquiesce lock x=1 by request(mds.2:146 nref=3)) | ptrwaiter=0 request=1 lock=1 importingcaps=0 caps=0 needsnapflush=0 replicated=0 dirty=1 waiter=0 authpin=1 0x55c68bc7f600]", "lock": { "gather_set": [], "state": "lock", "type": "iquiesce", "is_leased": false, "num_rdlocks": 0, "num_wrlocks": 0, "num_xlocks": 1, "xlock_by": { "reqid": { "entity": { "type": "mds", "num": 2 }, "tid": 146 } } }, "flags": 4, "wrlock_target": -1 } ] } }, { "description": "internal op quiesce_inode:mds.2:126 fp=#0x1000000021f", "initiated_at": "2024-04-23T10:31:21.477876+0000", "age": 42.350869230999997, "duration": 42.353319935999998, "continuous": true, "type_data": { "result": -2147483648, "flag_point": "quiesce complete", "reqid": { "entity": { "type": "mds", "num": 2 }, "tid": 126 }, "op_type": "internal_op", "internal_op": 5385, "op_name": "quiesce_inode", "events": [ { "time": "2024-04-23T10:31:21.477876+0000", "event": "initiated" }, { "time": "2024-04-23T10:31:21.477876+0000", "event": "header_read" }, { "time": "2024-04-23T10:31:21.477876+0000", "event": "throttled" }, { "time": "2024-04-23T10:31:21.477876+0000", "event": "all_read" }, { "time": "2024-04-23T10:31:21.477876+0000", "event": "dispatched" }, { "time": "2024-04-23T10:31:21.478331+0000", "event": "failed to xlock, waiting" }, { "time": "2024-04-23T10:31:21.481526+0000", "event": "acquired locks" }, { "time": "2024-04-23T10:31:21.481777+0000", "event": "quiesce complete" } ], "locks": [ { "object": { "is_auth": true, "auth_state": { "replicas": { "0": 1 } }, "replica_state": { "authority": [ 2, -2 ], "replica_nonce": 0 }, "auth_pins": 2, "is_frozen": false, "is_freezing": false, "pins": { "ptrwaiter": 0, "request": 1, "lock": 1, "importingcaps": 0, "caps": 0, "replicated": 1, "dirty": 1, "waiter": 0, "authpin": 1 }, "nref": 5 }, "object_string": "[inode 0x1000000021f [7,head] /volumes/qa/sv_1/b206794e-7622-4ab8-8943-f3ae4e65b7a3/client.0/tmp/clients/client0/~dmtmp/COREL/ROSE.CPT auth{0=1} v7256 ap=2 s=260096 n(v0 rc2024-04-23T10:27:52.049396+0000 b260096 1=1+0) (iquiesce lock x=1 by request(mds.2:126 nref=3)) | ptrwaiter=0 request=1 lock=1 importingcaps=0 caps=0 replicated=1 dirty=1 waiter=0 authpin=1 0x55c68bbfc680]", "lock": { "gather_set": [], "state": "lock", "type": "iquiesce", "is_leased": false, "num_rdlocks": 0, "num_wrlocks": 0, "num_xlocks": 1, "xlock_by": { "reqid": { "entity": { "type": "mds", "num": 2 }, "tid": 126 } } }, "flags": 4, "wrlock_target": -1 } ] } }, { "description": "internal op quiesce_inode:mds.2:147 fp=#0x10000000212", "initiated_at": "2024-04-23T10:31:21.498718+0000", "age": 42.330026959000001, "duration": 42.332541767999999, "continuous": true, "type_data": { "result": -2147483648, "flag_point": "quiesce complete", "reqid": { "entity": { "type": "mds", "num": 2 }, "tid": 147 }, "op_type": "internal_op", "internal_op": 5385, "op_name": "quiesce_inode", "events": [ { "time": "2024-04-23T10:31:21.498718+0000", "event": "initiated" }, { "time": "2024-04-23T10:31:21.498718+0000", "event": "header_read" }, { "time": "2024-04-23T10:31:21.498718+0000", "event": "throttled" }, { "time": "2024-04-23T10:31:21.498718+0000", "event": "all_read" }, { "time": "2024-04-23T10:31:21.498718+0000", "event": "dispatched" }, { "time": "2024-04-23T10:31:21.499826+0000", "event": "failed to xlock, waiting" }, { "time": "2024-04-23T10:31:21.504506+0000", "event": "failed to xlock, waiting" }, { "time": "2024-04-23T10:31:21.508468+0000", "event": "acquired locks" }, { "time": "2024-04-23T10:31:21.508653+0000", "event": "quiesce complete" } ], "locks": [ { "object": { "is_auth": true, "auth_state": { "replicas": {} }, "replica_state": { "authority": [ 2, -2 ], "replica_nonce": 0 }, "auth_pins": 1, "is_frozen": false, "is_freezing": false, "pins": { "ptrwaiter": 0, "request": 1, "lock": 1, "importingcaps": 0, "caps": 0, "replicated": 0, "dirty": 1, "waiter": 0, "authpin": 1 }, "nref": 4 }, "object_string": "[inode 0x10000000212 [4,head] /volumes/qa/sv_1/b206794e-7622-4ab8-8943-f3ae4e65b7a3/client.0/tmp/clients/client0/~dmtmp/PWRPNT/PPTOOLS1.PPA auth v12266 ap=1 s=260096 n(v0 rc2024-04-23T10:27:51.988397+0000 b260096 1=1+0) (iquiesce lock x=1 by request(mds.2:147 nref=3)) | ptrwaiter=0 request=1 lock=1 importingcaps=0 caps=0 replicated=0 dirty=1 waiter=0 authpin=1 0x55c68bc7fb80]", "lock": { "gather_set": [], "state": "lock", "type": "iquiesce", "is_leased": false, "num_rdlocks": 0, "num_wrlocks": 0, "num_xlocks": 1, "xlock_by": { "reqid": { "entity": { "type": "mds", "num": 2 }, "tid": 147 } } }, "flags": 4, "wrlock_target": -1 } ] } }, { "description": "internal op quiesce_inode:mds.2:127 fp=#0x1000000021b", "initiated_at": "2024-04-23T10:31:21.477896+0000", "age": 42.350849224000001, "duration": 42.353428264999998, "continuous": true, "type_data": { "result": -2147483648, "flag_point": "quiesce complete", "reqid": { "entity": { "type": "mds", "num": 2 }, "tid": 127 }, "op_type": "internal_op", "internal_op": 5385, "op_name": "quiesce_inode", "events": [ { "time": "2024-04-23T10:31:21.477896+0000", "event": "initiated" }, { "time": "2024-04-23T10:31:21.477896+0000", "event": "header_read" }, { "time": "2024-04-23T10:31:21.477896+0000", "event": "throttled" }, { "time": "2024-04-23T10:31:21.477896+0000", "event": "all_read" }, { "time": "2024-04-23T10:31:21.477896+0000", "event": "dispatched" }, { "time": "2024-04-23T10:31:21.478669+0000", "event": "failed to xlock, waiting" }, { "time": "2024-04-23T10:31:21.482509+0000", "event": "acquired locks" }, { "time": "2024-04-23T10:31:21.482729+0000", "event": "quiesce complete" } ], "locks": [ { "object": { "is_auth": true, "auth_state": { "replicas": { "0": 1 } }, "replica_state": { "authority": [ 2, -2 ], "replica_nonce": 0 }, "auth_pins": 2, "is_frozen": false, "is_freezing": false, "pins": { "ptrwaiter": 0, "request": 1, "lock": 1, "importingcaps": 0, "caps": 0, "replicated": 1, "dirty": 1, "waiter": 0, "authpin": 1 }, "nref": 5 }, "object_string": "[inode 0x1000000021b [7,head] /volumes/qa/sv_1/b206794e-7622-4ab8-8943-f3ae4e65b7a3/client.0/tmp/clients/client0/~dmtmp/COREL/TEAPOT2.CPT auth{0=1} v7248 ap=2 s=260096 n(v0 rc2024-04-23T10:27:52.028396+0000 b260096 1=1+0) (iquiesce lock x=1 by request(mds.2:127 nref=3)) | ptrwaiter=0 request=1 lock=1 importingcaps=0 caps=0 replicated=1 dirty=1 waiter=0 authpin=1 0x55c68bbfcc00]", "lock": { "gather_set": [], "state": "lock", "type": "iquiesce", "is_leased": false, "num_rdlocks": 0, "num_wrlocks": 0, "num_xlocks": 1, "xlock_by": { "reqid": { "entity": { "type": "mds", "num": 2 }, "tid": 127 } } }, "flags": 4, "wrlock_target": -1 } ] } }, { "description": "internal op quiesce_inode:mds.2:128 fp=#0x1000000021a", "initiated_at": "2024-04-23T10:31:21.477913+0000", "age": 42.350832021000002, "duration": 42.353474875000003, "continuous": true, "type_data": { "result": -2147483648, "flag_point": "quiesce complete", "reqid": { "entity": { "type": "mds", "num": 2 }, "tid": 128 }, "op_type": "internal_op", "internal_op": 5385, "op_name": "quiesce_inode", "events": [ { "time": "2024-04-23T10:31:21.477913+0000", "event": "initiated" }, { "time": "2024-04-23T10:31:21.477913+0000", "event": "header_read" }, { "time": "2024-04-23T10:31:21.477913+0000", "event": "throttled" }, { "time": "2024-04-23T10:31:21.477913+0000", "event": "all_read" }, { "time": "2024-04-23T10:31:21.477913+0000", "event": "dispatched" }, { "time": "2024-04-23T10:31:21.479061+0000", "event": "failed to xlock, waiting" }, { "time": "2024-04-23T10:31:21.483283+0000", "event": "acquired locks" }, { "time": "2024-04-23T10:31:21.483442+0000", "event": "quiesce complete" } ], "locks": [ { "object": { "is_auth": true, "auth_state": { "replicas": { "0": 1 } }, "replica_state": { "authority": [ 2, -2 ], "replica_nonce": 0 }, "auth_pins": 2, "is_frozen": false, "is_freezing": false, "pins": { "ptrwaiter": 0, "request": 1, "lock": 1, "importingcaps": 0, "caps": 0, "replicated": 1, "dirty": 1, "waiter": 0, "authpin": 1 }, "nref": 5 }, "object_string": "[inode 0x1000000021a [7,head] /volumes/qa/sv_1/b206794e-7622-4ab8-8943-f3ae4e65b7a3/client.0/tmp/clients/client0/~dmtmp/COREL/MAILBOX2.CPT auth{0=1} v7246 ap=2 s=260096 n(v0 rc2024-04-23T10:27:52.023396+0000 b260096 1=1+0) (iquiesce lock x=1 by request(mds.2:128 nref=3)) | ptrwaiter=0 request=1 lock=1 importingcaps=0 caps=0 replicated=1 dirty=1 waiter=0 authpin=1 0x55c68bbfd180]", "lock": { "gather_set": [], "state": "lock", "type": "iquiesce", "is_leased": false, "num_rdlocks": 0, "num_wrlocks": 0, "num_xlocks": 1, "xlock_by": { "reqid": { "entity": { "type": "mds", "num": 2 }, "tid": 128 } } }, "flags": 4, "wrlock_target": -1 } ] } }, { "description": "internal op quiesce_inode:mds.2:129 fp=#0x10000000220", "initiated_at": "2024-04-23T10:31:21.477929+0000", "age": 42.350815931, "duration": 42.353530345999999, "continuous": true, "type_data": { "result": -2147483648, "flag_point": "quiesce complete", "reqid": { "entity": { "type": "mds", "num": 2 }, "tid": 129 }, "op_type": "internal_op", "internal_op": 5385, "op_name": "quiesce_inode", "events": [ { "time": "2024-04-23T10:31:21.477929+0000", "event": "initiated" }, { "time": "2024-04-23T10:31:21.477929+0000", "event": "header_read" }, { "time": "2024-04-23T10:31:21.477929+0000", "event": "throttled" }, { "time": "2024-04-23T10:31:21.477929+0000", "event": "all_read" }, { "time": "2024-04-23T10:31:21.477929+0000", "event": "dispatched" }, { "time": "2024-04-23T10:31:21.479343+0000", "event": "failed to xlock, waiting" }, { "time": "2024-04-23T10:31:21.483782+0000", "event": "acquired locks" }, { "time": "2024-04-23T10:31:21.483892+0000", "event": "quiesce complete" } ], "locks": [ { "object": { "is_auth": true, "auth_state": { "replicas": { "0": 1 } }, "replica_state": { "authority": [ 2, -2 ], "replica_nonce": 0 }, "auth_pins": 2, "is_frozen": false, "is_freezing": false, "pins": { "ptrwaiter": 0, "request": 1, "lock": 1, "importingcaps": 0, "caps": 0, "needsnapflush": 0, "replicated": 1, "dirty": 1, "waiter": 0, "authpin": 1 }, "nref": 5 }, "object_string": "[inode 0x10000000220 [7,head] /volumes/qa/sv_1/b206794e-7622-4ab8-8943-f3ae4e65b7a3/client.0/tmp/clients/client0/~dmtmp/COREL/CORELDRW.CDT auth{0=1} v7242 ap=2 s=260096 n(v0 rc2024-04-23T10:27:52.054395+0000 b260096 1=1+0) (iquiesce lock x=1 by request(mds.2:129 nref=3)) | ptrwaiter=0 request=1 lock=1 importingcaps=0 caps=0 needsnapflush=0 replicated=1 dirty=1 waiter=0 authpin=1 0x55c68bbfd700]", "lock": { "gather_set": [], "state": "lock", "type": "iquiesce", "is_leased": false, "num_rdlocks": 0, "num_wrlocks": 0, "num_xlocks": 1, "xlock_by": { "reqid": { "entity": { "type": "mds", "num": 2 }, "tid": 129 } } }, "flags": 4, "wrlock_target": -1 } ] } }, { "description": "internal op quiesce_inode:mds.2:130 fp=#0x1000000021e", "initiated_at": "2024-04-23T10:31:21.477946+0000", "age": 42.350799174999999, "duration": 42.353584402999999, "continuous": true, "type_data": { "result": -2147483648, "flag_point": "quiesce complete", "reqid": { "entity": { "type": "mds", "num": 2 }, "tid": 130 }, "op_type": "internal_op", "internal_op": 5385, "op_name": "quiesce_inode", "events": [ { "time": "2024-04-23T10:31:21.477946+0000", "event": "initiated" }, { "time": "2024-04-23T10:31:21.477946+0000", "event": "header_read" }, { "time": "2024-04-23T10:31:21.477946+0000", "event": "throttled" }, { "time": "2024-04-23T10:31:21.477946+0000", "event": "all_read" }, { "time": "2024-04-23T10:31:21.477946+0000", "event": "dispatched" }, { "time": "2024-04-23T10:31:21.479632+0000", "event": "failed to xlock, waiting" }, { "time": "2024-04-23T10:31:21.484483+0000", "event": "acquired locks" }, { "time": "2024-04-23T10:31:21.484617+0000", "event": "quiesce complete" } ], "locks": [ { "object": { "is_auth": true, "auth_state": { "replicas": { "0": 1 } }, "replica_state": { "authority": [ 2, -2 ], "replica_nonce": 0 }, "auth_pins": 2, "is_frozen": false, "is_freezing": false, "pins": { "ptrwaiter": 0, "request": 1, "lock": 1, "importingcaps": 0, "caps": 0, "needsnapflush": 0, "replicated": 1, "dirty": 1, "waiter": 0, "authpin": 1 }, "nref": 5 }, "object_string": "[inode 0x1000000021e [7,head] /volumes/qa/sv_1/b206794e-7622-4ab8-8943-f3ae4e65b7a3/client.0/tmp/clients/client0/~dmtmp/COREL/GRAPHIC2.CDR auth{0=1} v7254 ap=2 s=260096 n(v0 rc2024-04-23T10:27:52.043396+0000 b260096 1=1+0) (iquiesce lock x=1 by request(mds.2:130 nref=3)) | ptrwaiter=0 request=1 lock=1 importingcaps=0 caps=0 needsnapflush=0 replicated=1 dirty=1 waiter=0 authpin=1 0x55c68bc4c000]", "lock": { "gather_set": [], "state": "lock", "type": "iquiesce", "is_leased": false, "num_rdlocks": 0, "num_wrlocks": 0, "num_xlocks": 1, "xlock_by": { "reqid": { "entity": { "type": "mds", "num": 2 }, "tid": 130 } } }, "flags": 4, "wrlock_target": -1 } ] } }, { "description": "internal op quiesce_inode:mds.2:131 fp=#0x10000000218", "initiated_at": "2024-04-23T10:31:21.477962+0000", "age": 42.350782817000002, "duration": 42.353634391, "continuous": true, "type_data": { "result": -2147483648, "flag_point": "quiesce complete", "reqid": { "entity": { "type": "mds", "num": 2 }, "tid": 131 }, "op_type": "internal_op", "internal_op": 5385, "op_name": "quiesce_inode", "events": [ { "time": "2024-04-23T10:31:21.477962+0000", "event": "initiated" }, { "time": "2024-04-23T10:31:21.477962+0000", "event": "header_read" }, { "time": "2024-04-23T10:31:21.477962+0000", "event": "throttled" }, { "time": "2024-04-23T10:31:21.477962+0000", "event": "all_read" }, { "time": "2024-04-23T10:31:21.477962+0000", "event": "dispatched" }, { "time": "2024-04-23T10:31:21.479962+0000", "event": "failed to xlock, waiting" }, { "time": "2024-04-23T10:31:21.485114+0000", "event": "acquired locks" }, { "time": "2024-04-23T10:31:21.485237+0000", "event": "quiesce complete" } ], "locks": [ { "object": { "is_auth": true, "auth_state": { "replicas": { "0": 1 } }, "replica_state": { "authority": [ 2, -2 ], "replica_nonce": 0 }, "auth_pins": 2, "is_frozen": false, "is_freezing": false, "pins": { "ptrwaiter": 0, "request": 1, "lock": 1, "importingcaps": 0, "caps": 0, "needsnapflush": 0, "replicated": 1, "dirty": 1, "waiter": 0, "authpin": 1 }, "nref": 5 }, "object_string": "[inode 0x10000000218 [7,head] /volumes/qa/sv_1/b206794e-7622-4ab8-8943-f3ae4e65b7a3/client.0/tmp/clients/client0/~dmtmp/COREL/ARTISTIC.ACL auth{0=1} v7240 ap=2 s=260096 n(v0 rc2024-04-23T10:27:52.014396+0000 b260096 1=1+0) (iquiesce lock x=1 by request(mds.2:131 nref=3)) | ptrwaiter=0 request=1 lock=1 importingcaps=0 caps=0 needsnapflush=0 replicated=1 dirty=1 waiter=0 authpin=1 0x55c68bc4c580]", "lock": { "gather_set": [], "state": "lock", "type": "iquiesce", "is_leased": false, "num_rdlocks": 0, "num_wrlocks": 0, "num_xlocks": 1, "xlock_by": { "reqid": { "entity": { "type": "mds", "num": 2 }, "tid": 131 } } }, "flags": 4, "wrlock_target": -1 } ] } }, { "description": "internal op quiesce_inode:mds.2:132 fp=#0x1000000021c", "initiated_at": "2024-04-23T10:31:21.477991+0000", "age": 42.350753672000003, "duration": 42.353669670000002, "continuous": true, "type_data": { "result": -2147483648, "flag_point": "quiesce complete", "reqid": { "entity": { "type": "mds", "num": 2 }, "tid": 132 }, "op_type": "internal_op", "internal_op": 5385, "op_name": "quiesce_inode", "events": [ { "time": "2024-04-23T10:31:21.477991+0000", "event": "initiated" }, { "time": "2024-04-23T10:31:21.477991+0000", "event": "header_read" }, { "time": "2024-04-23T10:31:21.477991+0000", "event": "throttled" }, { "time": "2024-04-23T10:31:21.477991+0000", "event": "all_read" }, { "time": "2024-04-23T10:31:21.477991+0000", "event": "dispatched" }, { "time": "2024-04-23T10:31:21.480251+0000", "event": "failed to xlock, waiting" }, { "time": "2024-04-23T10:31:21.485677+0000", "event": "acquired locks" }, { "time": "2024-04-23T10:31:21.485823+0000", "event": "quiesce complete" } ], "locks": [ { "object": { "is_auth": true, "auth_state": { "replicas": { "0": 1 } }, "replica_state": { "authority": [ 2, -2 ], "replica_nonce": 0 }, "auth_pins": 2, "is_frozen": false, "is_freezing": false, "pins": { "ptrwaiter": 0, "request": 1, "lock": 1, "importingcaps": 0, "caps": 0, "replicated": 1, "dirty": 1, "waiter": 0, "authpin": 1 }, "nref": 5 }, "object_string": "[inode 0x1000000021c [7,head] /volumes/qa/sv_1/b206794e-7622-4ab8-8943-f3ae4e65b7a3/client.0/tmp/clients/client0/~dmtmp/COREL/SIGN2.CPT auth{0=1} v7250 ap=2 s=260096 n(v0 rc2024-04-23T10:27:52.033396+0000 b260096 1=1+0) (iquiesce lock x=1 by request(mds.2:132 nref=3)) | ptrwaiter=0 request=1 lock=1 importingcaps=0 caps=0 replicated=1 dirty=1 waiter=0 authpin=1 0x55c68bc4cb00]", "lock": { "gather_set": [], "state": "lock", "type": "iquiesce", "is_leased": false, "num_rdlocks": 0, "num_wrlocks": 0, "num_xlocks": 1, "xlock_by": { "reqid": { "entity": { "type": "mds", "num": 2 }, "tid": 132 } } }, "flags": 4, "wrlock_target": -1 } ] } }, { "description": "internal op quiesce_inode:mds.2:133 fp=#0x10000000217", "initiated_at": "2024-04-23T10:31:21.478006+0000", "age": 42.350738485000001, "duration": 42.353734189999997, "continuous": true, "type_data": { "result": -2147483648, "flag_point": "quiesce complete", "reqid": { "entity": { "type": "mds", "num": 2 }, "tid": 133 }, "op_type": "internal_op", "internal_op": 5385, "op_name": "quiesce_inode", "events": [ { "time": "2024-04-23T10:31:21.478006+0000", "event": "initiated" }, { "time": "2024-04-23T10:31:21.478006+0000", "event": "header_read" }, { "time": "2024-04-23T10:31:21.478006+0000", "event": "throttled" }, { "time": "2024-04-23T10:31:21.478006+0000", "event": "all_read" }, { "time": "2024-04-23T10:31:21.478006+0000", "event": "dispatched" }, { "time": "2024-04-23T10:31:21.480540+0000", "event": "failed to xlock, waiting" }, { "time": "2024-04-23T10:31:21.486581+0000", "event": "failed to xlock, waiting" }, { "time": "2024-04-23T10:31:21.505374+0000", "event": "acquired locks" }, { "time": "2024-04-23T10:31:21.505510+0000", "event": "quiesce complete" } ], "locks": [ { "object": { "is_auth": true, "auth_state": { "replicas": { "0": 1 } }, "replica_state": { "authority": [ 2, -2 ], "replica_nonce": 0 }, "auth_pins": 2, "is_frozen": false, "is_freezing": false, "pins": { "ptrwaiter": 0, "request": 1, "lock": 1, "importingcaps": 0, "caps": 0, "needsnapflush": 0, "replicated": 1, "dirty": 1, "waiter": 0, "authpin": 1 }, "nref": 5 }, "object_string": "[inode 0x10000000217 [7,head] /volumes/qa/sv_1/b206794e-7622-4ab8-8943-f3ae4e65b7a3/client.0/tmp/clients/client0/~dmtmp/COREL/CDRBARS.CFG auth{0=1} v7236 ap=2 s=260096 n(v0 rc2024-04-23T10:31:17.211728+0000 b260096 1=1+0) (iquiesce lock x=1 by request(mds.2:133 nref=3)) | ptrwaiter=0 request=1 lock=1 importingcaps=0 caps=0 needsnapflush=0 replicated=1 dirty=1 waiter=0 authpin=1 0x55c68bc4d080]", "lock": { "gather_set": [], "state": "lock", "type": "iquiesce", "is_leased": false, "num_rdlocks": 0, "num_wrlocks": 0, "num_xlocks": 1, "xlock_by": { "reqid": { "entity": { "type": "mds", "num": 2 }, "tid": 133 } } }, "flags": 4, "wrlock_target": -1 } ] } }, { "description": "internal op quiesce_inode:mds.2:134 fp=#0x1000000021d", "initiated_at": "2024-04-23T10:31:21.478022+0000", "age": 42.350722484000002, "duration": 42.353781849000001, "continuous": true, "type_data": { "result": -2147483648, "flag_point": "quiesce complete", "reqid": { "entity": { "type": "mds", "num": 2 }, "tid": 134 }, "op_type": "internal_op", "internal_op": 5385, "op_name": "quiesce_inode", "events": [ { "time": "2024-04-23T10:31:21.478022+0000", "event": "initiated" }, { "time": "2024-04-23T10:31:21.478022+0000", "event": "header_read" }, { "time": "2024-04-23T10:31:21.478022+0000", "event": "throttled" }, { "time": "2024-04-23T10:31:21.478022+0000", "event": "all_read" }, { "time": "2024-04-23T10:31:21.478022+0000", "event": "dispatched" }, { "time": "2024-04-23T10:31:21.480740+0000", "event": "failed to xlock, waiting" }, { "time": "2024-04-23T10:31:21.487025+0000", "event": "acquired locks" }, { "time": "2024-04-23T10:31:21.487146+0000", "event": "quiesce complete" } ], "locks": [ { "object": { "is_auth": true, "auth_state": { "replicas": { "0": 1 } }, "replica_state": { "authority": [ 2, -2 ], "replica_nonce": 0 }, "auth_pins": 2, "is_frozen": false, "is_freezing": false, "pins": { "ptrwaiter": 0, "request": 1, "lock": 1, "importingcaps": 0, "caps": 0, "replicated": 1, "dirty": 1, "waiter": 0, "authpin": 1 }, "nref": 5 }, "object_string": "[inode 0x1000000021d [7,head] /volumes/qa/sv_1/b206794e-7622-4ab8-8943-f3ae4e65b7a3/client.0/tmp/clients/client0/~dmtmp/COREL/TRAIN.CDR auth{0=1} v7252 ap=2 s=260096 n(v0 rc2024-04-23T10:27:52.038396+0000 b260096 1=1+0) (iquiesce lock x=1 by request(mds.2:134 nref=3)) | ptrwaiter=0 request=1 lock=1 importingcaps=0 caps=0 replicated=1 dirty=1 waiter=0 authpin=1 0x55c68bc4d600]", "lock": { "gather_set": [], "state": "lock", "type": "iquiesce", "is_leased": false, "num_rdlocks": 0, "num_wrlocks": 0, "num_xlocks": 1, "xlock_by": { "reqid": { "entity": { "type": "mds", "num": 2 }, "tid": 134 } } }, "flags": 4, "wrlock_target": -1 } ] } }, { "description": "internal op quiesce_inode:mds.2:135 fp=#0x10000000219", "initiated_at": "2024-04-23T10:31:21.478038+0000", "age": 42.350706348999999, "duration": 42.353826496000003, "continuous": true, "type_data": { "result": -2147483648, "flag_point": "quiesce complete", "reqid": { "entity": { "type": "mds", "num": 2 }, "tid": 135 }, "op_type": "internal_op", "internal_op": 5385, "op_name": "quiesce_inode", "events": [ { "time": "2024-04-23T10:31:21.478038+0000", "event": "initiated" }, { "time": "2024-04-23T10:31:21.478038+0000", "event": "header_read" }, { "time": "2024-04-23T10:31:21.478038+0000", "event": "throttled" }, { "time": "2024-04-23T10:31:21.478038+0000", "event": "all_read" }, { "time": "2024-04-23T10:31:21.478038+0000", "event": "dispatched" }, { "time": "2024-04-23T10:31:21.480933+0000", "event": "failed to xlock, waiting" }, { "time": "2024-04-23T10:31:21.487514+0000", "event": "acquired locks" }, { "time": "2024-04-23T10:31:21.487768+0000", "event": "quiesce complete" } ], "locks": [ { "object": { "is_auth": true, "auth_state": { "replicas": { "0": 1 } }, "replica_state": { "authority": [ 2, -2 ], "replica_nonce": 0 }, "auth_pins": 2, "is_frozen": false, "is_freezing": false, "pins": { "ptrwaiter": 0, "request": 1, "lock": 1, "importingcaps": 0, "caps": 0, "needsnapflush": 0, "replicated": 1, "dirty": 1, "waiter": 0, "authpin": 1 }, "nref": 5 }, "object_string": "[inode 0x10000000219 [7,head] /volumes/qa/sv_1/b206794e-7622-4ab8-8943-f3ae4e65b7a3/client.0/tmp/clients/client0/~dmtmp/COREL/GRAPHIC1.CDR auth{0=1} v7244 ap=2 s=260096 n(v0 rc2024-04-23T10:27:52.019396+0000 b260096 1=1+0) (iquiesce lock x=1 by request(mds.2:135 nref=3)) | ptrwaiter=0 request=1 lock=1 importingcaps=0 caps=0 needsnapflush=0 replicated=1 dirty=1 waiter=0 authpin=1 0x55c68bc4db80]", "lock": { "gather_set": [], "state": "lock", "type": "iquiesce", "is_leased": false, "num_rdlocks": 0, "num_wrlocks": 0, "num_xlocks": 1, "xlock_by": { "reqid": { "entity": { "type": "mds", "num": 2 }, "tid": 135 } } }, "flags": 4, "wrlock_target": -1 } ] } }, { "description": "internal op quiesce_inode:mds.2:136 fp=#0x10000000216", "initiated_at": "2024-04-23T10:31:21.478055+0000", "age": 42.350689918999997, "duration": 42.353873546999999, "continuous": true, "type_data": { "result": -2147483648, "flag_point": "quiesce complete", "reqid": { "entity": { "type": "mds", "num": 2 }, "tid": 136 }, "op_type": "internal_op", "internal_op": 5385, "op_name": "quiesce_inode", "events": [ { "time": "2024-04-23T10:31:21.478055+0000", "event": "initiated" }, { "time": "2024-04-23T10:31:21.478055+0000", "event": "header_read" }, { "time": "2024-04-23T10:31:21.478055+0000", "event": "throttled" }, { "time": "2024-04-23T10:31:21.478055+0000", "event": "all_read" }, { "time": "2024-04-23T10:31:21.478055+0000", "event": "dispatched" }, { "time": "2024-04-23T10:31:21.481114+0000", "event": "failed to xlock, waiting" }, { "time": "2024-04-23T10:31:21.488531+0000", "event": "failed to xlock, waiting" }, { "time": "2024-04-23T10:31:21.505820+0000", "event": "acquired locks" }, { "time": "2024-04-23T10:31:21.506034+0000", "event": "quiesce complete" } ], "locks": [ { "object": { "is_auth": true, "auth_state": { "replicas": { "0": 1 } }, "replica_state": { "authority": [ 2, -2 ], "replica_nonce": 0 }, "auth_pins": 2, "is_frozen": false, "is_freezing": false, "pins": { "ptrwaiter": 0, "request": 1, "lock": 1, "importingcaps": 0, "caps": 0, "needsnapflush": 0, "replicated": 1, "dirty": 1, "waiter": 0, "authpin": 1 }, "nref": 5 }, "object_string": "[inode 0x10000000216 [7,head] /volumes/qa/sv_1/b206794e-7622-4ab8-8943-f3ae4e65b7a3/client.0/tmp/clients/client0/~dmtmp/COREL/CDRROLS.CFG auth{0=1} v7238 ap=2 s=260096 n(v0 rc2024-04-23T10:31:17.221728+0000 b260096 1=1+0) (iquiesce lock x=1 by request(mds.2:136 nref=3)) | ptrwaiter=0 request=1 lock=1 importingcaps=0 caps=0 needsnapflush=0 replicated=1 dirty=1 waiter=0 authpin=1 0x55c68bc4e100]", "lock": { "gather_set": [], "state": "lock", "type": "iquiesce", "is_leased": false, "num_rdlocks": 0, "num_wrlocks": 0, "num_xlocks": 1, "xlock_by": { "reqid": { "entity": { "type": "mds", "num": 2 }, "tid": 136 } } }, "flags": 4, "wrlock_target": -1 } ] } } ], "num_ops": 53 } mds.h: { "ops": [ { "description": "internal op quiesce_inode:mds.1:131 fp=#0x10000000233", "initiated_at": "2024-04-23T10:31:21.494203+0000", "age": 42.363435672999998, "duration": 42.363456708999998, "continuous": true, "type_data": { "result": -2147483648, "flag_point": "failed to authpin, dir is being fragmented", "reqid": { "entity": { "type": "mds", "num": 1 }, "tid": 131 }, "op_type": "internal_op", "internal_op": 5385, "op_name": "quiesce_inode", "events": [ { "time": "2024-04-23T10:31:21.494203+0000", "event": "initiated" }, { "time": "2024-04-23T10:31:21.494203+0000", "event": "header_read" }, { "time": "2024-04-23T10:31:21.494203+0000", "event": "throttled" }, { "time": "2024-04-23T10:31:21.494203+0000", "event": "all_read" }, { "time": "2024-04-23T10:31:21.494203+0000", "event": "dispatched" }, { "time": "2024-04-23T10:31:21.495293+0000", "event": "failed to authpin, dir is being fragmented" } ], "locks": [] } }, { "description": "client_request(client.15312:74454 unlink #0x10000000229/COURSES.VAL 2024-04-23T10:31:17.777716+0000 caller_uid=1000, caller_gid=1316{6,36,1000,1316,})", "initiated_at": "2024-04-23T10:31:17.778339+0000", "age": 46.079299241999998, "duration": 46.079362150999998, "continuous": false, "type_data": { "result": -2147483648, "flag_point": "failed to authpin, dir is being fragmented", "reqid": { "entity": { "type": "client", "num": 15312 }, "tid": 74454 }, "op_type": "client_request", "events": [ { "time": "2024-04-23T10:31:17.778339+0000", "event": "initiated" }, { "time": "2024-04-23T10:31:17.778339+0000", "event": "header_read" }, { "time": "2024-04-23T10:31:17.778340+0000", "event": "throttled" }, { "time": "2024-04-23T10:31:17.778345+0000", "event": "all_read" }, { "time": "2024-04-23T10:31:17.778370+0000", "event": "dispatched" }, { "time": "2024-04-23T10:31:17.778657+0000", "event": "failed to authpin, dir is being fragmented" } ], "locks": [] } }, { "description": "internal op quiesce_inode:mds.1:132 fp=#0x10000000239", "initiated_at": "2024-04-23T10:31:21.494238+0000", "age": 42.363400931999998, "duration": 42.363489846999997, "continuous": true, "type_data": { "result": -2147483648, "flag_point": "failed to authpin, dir is being fragmented", "reqid": { "entity": { "type": "mds", "num": 1 }, "tid": 132 }, "op_type": "internal_op", "internal_op": 5385, "op_name": "quiesce_inode", "events": [ { "time": "2024-04-23T10:31:21.494238+0000", "event": "initiated" }, { "time": "2024-04-23T10:31:21.494238+0000", "event": "header_read" }, { "time": "2024-04-23T10:31:21.494238+0000", "event": "throttled" }, { "time": "2024-04-23T10:31:21.494238+0000", "event": "all_read" }, { "time": "2024-04-23T10:31:21.494238+0000", "event": "dispatched" }, { "time": "2024-04-23T10:31:21.495383+0000", "event": "failed to authpin, dir is being fragmented" } ], "locks": [] } }, { "description": "internal op quiesce_path:mds.1:101 fp=#0x1", "initiated_at": "2024-04-23T10:31:21.445231+0000", "age": 42.412407909000002, "duration": 42.412538318999999, "continuous": true, "type_data": { "result": -2147483648, "flag_point": "dispatched", "reqid": { "entity": { "type": "mds", "num": 1 }, "tid": 101 }, "op_type": "internal_op", "internal_op": 5384, "op_name": "quiesce_path", "events": [ { "time": "2024-04-23T10:31:21.445231+0000", "event": "initiated" }, { "time": "2024-04-23T10:31:21.445231+0000", "event": "header_read" }, { "time": "2024-04-23T10:31:21.445231+0000", "event": "throttled" }, { "time": "2024-04-23T10:31:21.445231+0000", "event": "all_read" }, { "time": "2024-04-23T10:31:21.445231+0000", "event": "dispatched" } ], "locks": [] } }, { "description": "internal op quiesce_inode:mds.1:133 fp=#0x1000000023d", "initiated_at": "2024-04-23T10:31:21.494254+0000", "age": 42.363384936000003, "duration": 42.363535954, "continuous": true, "type_data": { "result": -2147483648, "flag_point": "failed to authpin, dir is being fragmented", "reqid": { "entity": { "type": "mds", "num": 1 }, "tid": 133 }, "op_type": "internal_op", "internal_op": 5385, "op_name": "quiesce_inode", "events": [ { "time": "2024-04-23T10:31:21.494254+0000", "event": "initiated" }, { "time": "2024-04-23T10:31:21.494254+0000", "event": "header_read" }, { "time": "2024-04-23T10:31:21.494254+0000", "event": "throttled" }, { "time": "2024-04-23T10:31:21.494254+0000", "event": "all_read" }, { "time": "2024-04-23T10:31:21.494254+0000", "event": "dispatched" }, { "time": "2024-04-23T10:31:21.495478+0000", "event": "failed to authpin, dir is being fragmented" } ], "locks": [] } }, { "description": "internal op quiesce_inode:mds.1:102 fp=#0x1", "initiated_at": "2024-04-23T10:31:21.445286+0000", "age": 42.412352829, "duration": 42.412528283, "continuous": true, "type_data": { "result": -2147483648, "flag_point": "acquired locks", "reqid": { "entity": { "type": "mds", "num": 1 }, "tid": 102 }, "op_type": "internal_op", "internal_op": 5385, "op_name": "quiesce_inode", "events": [ { "time": "2024-04-23T10:31:21.445286+0000", "event": "initiated" }, { "time": "2024-04-23T10:31:21.445286+0000", "event": "header_read" }, { "time": "2024-04-23T10:31:21.445286+0000", "event": "throttled" }, { "time": "2024-04-23T10:31:21.445286+0000", "event": "all_read" }, { "time": "2024-04-23T10:31:21.445286+0000", "event": "dispatched" }, { "time": "2024-04-23T10:31:21.445409+0000", "event": "requesting remote authpins" }, { "time": "2024-04-23T10:31:21.446064+0000", "event": "acquired locks" } ], "locks": [ { "object": { "is_auth": false, "auth_state": { "replicas": {} }, "replica_state": { "authority": [ 0, -2 ], "replica_nonce": 1 }, "auth_pins": 0, "is_frozen": false, "is_freezing": false, "pins": { "request": 1, "lock": 1, "dirfrag": 1, "caps": 1, "dirwaiter": 0, "discoverbase": 0 }, "nref": 4 }, "object_string": "[inode 0x1 [...7,head] / rep@0.1 v187 snaprealm=0x55e71896c480 f(v0 m2024-04-23T10:24:48.676117+0000 1=0+1) n(v27 rc2024-04-23T10:31:17.764716+0000 b21075308 rs5 88=67+21)/n(v0 rc2024-04-23T10:24:44.319412+0000 1=0+1) (inest mix) (iquiesce lock x=1 by request(mds.1:102 nref=4)) caps={15576=pAsLsXs/-@4} | request=1 lock=1 dirfrag=1 caps=1 dirwaiter=0 discoverbase=0 0x55e718955600]", "lock": { "gather_set": [], "state": "lock", "type": "iquiesce", "is_leased": false, "num_rdlocks": 0, "num_wrlocks": 0, "num_xlocks": 1, "xlock_by": { "reqid": { "entity": { "type": "mds", "num": 1 }, "tid": 102 } } }, "flags": 4, "wrlock_target": -1 } ] } }, { "description": "internal op quiesce_inode:mds.1:134 fp=#0x10000000234", "initiated_at": "2024-04-23T10:31:21.494267+0000", "age": 42.363371192999999, "duration": 42.363609945, "continuous": true, "type_data": { "result": -2147483648, "flag_point": "failed to authpin, dir is being fragmented", "reqid": { "entity": { "type": "mds", "num": 1 }, "tid": 134 }, "op_type": "internal_op", "internal_op": 5385, "op_name": "quiesce_inode", "events": [ { "time": "2024-04-23T10:31:21.494267+0000", "event": "initiated" }, { "time": "2024-04-23T10:31:21.494267+0000", "event": "header_read" }, { "time": "2024-04-23T10:31:21.494267+0000", "event": "throttled" }, { "time": "2024-04-23T10:31:21.494267+0000", "event": "all_read" }, { "time": "2024-04-23T10:31:21.494267+0000", "event": "dispatched" }, { "time": "2024-04-23T10:31:21.495566+0000", "event": "failed to authpin, dir is being fragmented" } ], "locks": [] } }, { "description": "internal op quiesce_inode:mds.1:103 fp=#0x10000000000", "initiated_at": "2024-04-23T10:31:21.446077+0000", "age": 42.411561650000003, "duration": 42.411824176000003, "continuous": true, "type_data": { "result": -2147483648, "flag_point": "acquired locks", "reqid": { "entity": { "type": "mds", "num": 1 }, "tid": 103 }, "op_type": "internal_op", "internal_op": 5385, "op_name": "quiesce_inode", "events": [ { "time": "2024-04-23T10:31:21.446077+0000", "event": "initiated" }, { "time": "2024-04-23T10:31:21.446077+0000", "event": "header_read" }, { "time": "2024-04-23T10:31:21.446077+0000", "event": "throttled" }, { "time": "2024-04-23T10:31:21.446077+0000", "event": "all_read" }, { "time": "2024-04-23T10:31:21.446077+0000", "event": "dispatched" }, { "time": "2024-04-23T10:31:21.446137+0000", "event": "requesting remote authpins" }, { "time": "2024-04-23T10:31:21.447260+0000", "event": "acquired locks" } ], "locks": [ { "object": { "is_auth": false, "auth_state": { "replicas": {} }, "replica_state": { "authority": [ 0, -2 ], "replica_nonce": 1 }, "auth_pins": 0, "is_frozen": false, "is_freezing": false, "pins": { "request": 1, "lock": 1, "dirfrag": 1, "caps": 0 }, "nref": 3 }, "object_string": "[inode 0x10000000000 [...7,head] /volumes/ rep@0.1 v404 f(v0 m2024-04-23T10:24:49.736808+0000 3=2+1) n(v29 rc2024-04-23T10:31:17.764716+0000 b21075308 87=67+20)/n(v0 rc2024-04-23T10:24:48.685915+0000 2=0+2) (inest mix) (iquiesce lock x=1 by request(mds.1:103 nref=4)) | request=1 lock=1 dirfrag=1 caps=0 0x55e718955b80]", "lock": { "gather_set": [], "state": "lock", "type": "iquiesce", "is_leased": false, "num_rdlocks": 0, "num_wrlocks": 0, "num_xlocks": 1, "xlock_by": { "reqid": { "entity": { "type": "mds", "num": 1 }, "tid": 103 } } }, "flags": 4, "wrlock_target": -1 } ] } }, { "description": "internal op quiesce_inode:mds.1:135 fp=#0x1000000022a", "initiated_at": "2024-04-23T10:31:21.494283+0000", "age": 42.363355947999999, "duration": 42.363674441999997, "continuous": true, "type_data": { "result": -2147483648, "flag_point": "failed to authpin, dir is being fragmented", "reqid": { "entity": { "type": "mds", "num": 1 }, "tid": 135 }, "op_type": "internal_op", "internal_op": 5385, "op_name": "quiesce_inode", "events": [ { "time": "2024-04-23T10:31:21.494283+0000", "event": "initiated" }, { "time": "2024-04-23T10:31:21.494283+0000", "event": "header_read" }, { "time": "2024-04-23T10:31:21.494283+0000", "event": "throttled" }, { "time": "2024-04-23T10:31:21.494283+0000", "event": "all_read" }, { "time": "2024-04-23T10:31:21.494283+0000", "event": "dispatched" }, { "time": "2024-04-23T10:31:21.495661+0000", "event": "failed to authpin, dir is being fragmented" } ], "locks": [] } }, { "description": "internal op quiesce_inode:mds.1:104 fp=#0x10000000001", "initiated_at": "2024-04-23T10:31:21.447285+0000", "age": 42.410353696000001, "duration": 42.410696027, "continuous": true, "type_data": { "result": -2147483648, "flag_point": "acquired locks", "reqid": { "entity": { "type": "mds", "num": 1 }, "tid": 104 }, "op_type": "internal_op", "internal_op": 5385, "op_name": "quiesce_inode", "events": [ { "time": "2024-04-23T10:31:21.447285+0000", "event": "initiated" }, { "time": "2024-04-23T10:31:21.447285+0000", "event": "header_read" }, { "time": "2024-04-23T10:31:21.447285+0000", "event": "throttled" }, { "time": "2024-04-23T10:31:21.447285+0000", "event": "all_read" }, { "time": "2024-04-23T10:31:21.447285+0000", "event": "dispatched" }, { "time": "2024-04-23T10:31:21.447364+0000", "event": "requesting remote authpins" }, { "time": "2024-04-23T10:31:21.448241+0000", "event": "acquired locks" } ], "locks": [ { "object": { "is_auth": false, "auth_state": { "replicas": {} }, "replica_state": { "authority": [ 0, -2 ], "replica_nonce": 1 }, "auth_pins": 0, "is_frozen": false, "is_freezing": false, "pins": { "request": 1, "lock": 1, "dirfrag": 1, "discoverbase": 0 }, "nref": 3 }, "object_string": "[inode 0x10000000001 [...7,head] /volumes/qa/ rep@0.1 v416 f(v0 m2024-04-23T10:24:49.714717+0000 2=0+2) n(v30 rc2024-04-23T10:31:17.764716+0000 b21075308 84=65+19)/n(v0 rc2024-04-23T10:24:49.023727+0000 3=0+3) (inest mix) (iquiesce lock x=1 by request(mds.1:104 nref=4)) | request=1 lock=1 dirfrag=1 discoverbase=0 0x55e718956100]", "lock": { "gather_set": [], "state": "lock", "type": "iquiesce", "is_leased": false, "num_rdlocks": 0, "num_wrlocks": 0, "num_xlocks": 1, "xlock_by": { "reqid": { "entity": { "type": "mds", "num": 1 }, "tid": 104 } } }, "flags": 4, "wrlock_target": -1 } ] } }, { "description": "internal op quiesce_inode:mds.1:136 fp=#0x1000000022d", "initiated_at": "2024-04-23T10:31:21.494296+0000", "age": 42.363342252999999, "duration": 42.363740145000001, "continuous": true, "type_data": { "result": -2147483648, "flag_point": "failed to authpin, dir is being fragmented", "reqid": { "entity": { "type": "mds", "num": 1 }, "tid": 136 }, "op_type": "internal_op", "internal_op": 5385, "op_name": "quiesce_inode", "events": [ { "time": "2024-04-23T10:31:21.494296+0000", "event": "initiated" }, { "time": "2024-04-23T10:31:21.494296+0000", "event": "header_read" }, { "time": "2024-04-23T10:31:21.494296+0000", "event": "throttled" }, { "time": "2024-04-23T10:31:21.494296+0000", "event": "all_read" }, { "time": "2024-04-23T10:31:21.494296+0000", "event": "dispatched" }, { "time": "2024-04-23T10:31:21.495757+0000", "event": "failed to authpin, dir is being fragmented" } ], "locks": [] } }, { "description": "internal op quiesce_inode:mds.1:105 fp=#0x100000001fa", "initiated_at": "2024-04-23T10:31:21.448256+0000", "age": 42.409382190999999, "duration": 42.409805163999998, "continuous": true, "type_data": { "result": -2147483648, "flag_point": "acquired locks", "reqid": { "entity": { "type": "mds", "num": 1 }, "tid": 105 }, "op_type": "internal_op", "internal_op": 5385, "op_name": "quiesce_inode", "events": [ { "time": "2024-04-23T10:31:21.448256+0000", "event": "initiated" }, { "time": "2024-04-23T10:31:21.448256+0000", "event": "header_read" }, { "time": "2024-04-23T10:31:21.448256+0000", "event": "throttled" }, { "time": "2024-04-23T10:31:21.448256+0000", "event": "all_read" }, { "time": "2024-04-23T10:31:21.448256+0000", "event": "dispatched" }, { "time": "2024-04-23T10:31:21.448346+0000", "event": "requesting remote authpins" }, { "time": "2024-04-23T10:31:21.449369+0000", "event": "acquired locks" } ], "locks": [ { "object": { "is_auth": false, "auth_state": { "replicas": {} }, "replica_state": { "authority": [ 0, -2 ], "replica_nonce": 1 }, "auth_pins": 0, "is_frozen": false, "is_freezing": false, "pins": { "request": 1, "lock": 1, "dirfrag": 1, "dirwaiter": 0, "discoverbase": 0 }, "nref": 3 }, "object_string": "[inode 0x100000001fa [...7,head] /volumes/qa/sv_1/ rep@0.1 v434 snaprealm=0x55e71896c6c0 f(v0 m2024-04-23T10:24:49.735055+0000 2=1+1) n(v29 rc2024-04-23T10:31:17.764716+0000 b21075191 80=64+16)/n(v0 rc2024-04-23T10:24:49.717706+0000 2=0+2) (inest mix) (iquiesce lock x=1 by request(mds.1:105 nref=4)) | request=1 lock=1 dirfrag=1 dirwaiter=0 discoverbase=0 0x55e7186f0000]", "lock": { "gather_set": [], "state": "lock", "type": "iquiesce", "is_leased": false, "num_rdlocks": 0, "num_wrlocks": 0, "num_xlocks": 1, "xlock_by": { "reqid": { "entity": { "type": "mds", "num": 1 }, "tid": 105 } } }, "flags": 4, "wrlock_target": -1 } ] } }, { "description": "internal op quiesce_inode:mds.1:137 fp=#0x10000000238", "initiated_at": "2024-04-23T10:31:21.494310+0000", "age": 42.363328478, "duration": 42.363808689999999, "continuous": true, "type_data": { "result": -2147483648, "flag_point": "failed to authpin, dir is being fragmented", "reqid": { "entity": { "type": "mds", "num": 1 }, "tid": 137 }, "op_type": "internal_op", "internal_op": 5385, "op_name": "quiesce_inode", "events": [ { "time": "2024-04-23T10:31:21.494310+0000", "event": "initiated" }, { "time": "2024-04-23T10:31:21.494310+0000", "event": "header_read" }, { "time": "2024-04-23T10:31:21.494310+0000", "event": "throttled" }, { "time": "2024-04-23T10:31:21.494310+0000", "event": "all_read" }, { "time": "2024-04-23T10:31:21.494310+0000", "event": "dispatched" }, { "time": "2024-04-23T10:31:21.495867+0000", "event": "failed to authpin, dir is being fragmented" } ], "locks": [] } }, { "description": "internal op quiesce_inode:mds.1:106 fp=#0x10000000002", "initiated_at": "2024-04-23T10:31:21.448273+0000", "age": 42.409365283, "duration": 42.409890222999998, "continuous": true, "type_data": { "result": -2147483648, "flag_point": "quiesce complete for non-auth inode", "reqid": { "entity": { "type": "mds", "num": 1 }, "tid": 106 }, "op_type": "internal_op", "internal_op": 5385, "op_name": "quiesce_inode", "events": [ { "time": "2024-04-23T10:31:21.448273+0000", "event": "initiated" }, { "time": "2024-04-23T10:31:21.448273+0000", "event": "header_read" }, { "time": "2024-04-23T10:31:21.448273+0000", "event": "throttled" }, { "time": "2024-04-23T10:31:21.448273+0000", "event": "all_read" }, { "time": "2024-04-23T10:31:21.448273+0000", "event": "dispatched" }, { "time": "2024-04-23T10:31:21.448407+0000", "event": "requesting remote authpins" }, { "time": "2024-04-23T10:31:21.449578+0000", "event": "acquired locks" }, { "time": "2024-04-23T10:31:21.449586+0000", "event": "quiesce complete for non-auth inode" } ], "locks": [ { "object": { "is_auth": false, "auth_state": { "replicas": {} }, "replica_state": { "authority": [ 0, -2 ], "replica_nonce": 1 }, "auth_pins": 0, "is_frozen": false, "is_freezing": false, "pins": { "request": 1, "lock": 1, "dirwaiter": 0, "discoverbase": 0 }, "nref": 2 }, "object_string": "[inode 0x10000000002 [...2,head] /volumes/qa/sv_0/ rep@0.1 v34 snaprealm=0x55e71896c240 f(v0 m2024-04-23T10:24:49.040186+0000 2=1+1) n(v0 rc2024-04-23T10:24:49.031387+0000 2=0+2)/n(v0 rc2024-04-23T10:24:49.023727+0000 2=0+2) (inest mix) (iquiesce lock x=1 by request(mds.1:106 nref=3)) | request=1 lock=1 dirwaiter=0 discoverbase=0 0x55e718957700]", "lock": { "gather_set": [], "state": "lock", "type": "iquiesce", "is_leased": false, "num_rdlocks": 0, "num_wrlocks": 0, "num_xlocks": 1, "xlock_by": { "reqid": { "entity": { "type": "mds", "num": 1 }, "tid": 106 } } }, "flags": 4, "wrlock_target": -1 } ] } }, { "description": "internal op quiesce_inode:mds.1:138 fp=#0x10000000235", "initiated_at": "2024-04-23T10:31:21.494330+0000", "age": 42.363308988, "duration": 42.363892266999997, "continuous": true, "type_data": { "result": -2147483648, "flag_point": "failed to authpin, dir is being fragmented", "reqid": { "entity": { "type": "mds", "num": 1 }, "tid": 138 }, "op_type": "internal_op", "internal_op": 5385, "op_name": "quiesce_inode", "events": [ { "time": "2024-04-23T10:31:21.494330+0000", "event": "initiated" }, { "time": "2024-04-23T10:31:21.494330+0000", "event": "header_read" }, { "time": "2024-04-23T10:31:21.494330+0000", "event": "throttled" }, { "time": "2024-04-23T10:31:21.494330+0000", "event": "all_read" }, { "time": "2024-04-23T10:31:21.494330+0000", "event": "dispatched" }, { "time": "2024-04-23T10:31:21.495977+0000", "event": "failed to authpin, dir is being fragmented" } ], "locks": [] } }, { "description": "internal op quiesce_inode:mds.1:107 fp=#0x100000001fb", "initiated_at": "2024-04-23T10:31:21.449385+0000", "age": 42.408253705999996, "duration": 42.408860955000002, "continuous": true, "type_data": { "result": -2147483648, "flag_point": "acquired locks", "reqid": { "entity": { "type": "mds", "num": 1 }, "tid": 107 }, "op_type": "internal_op", "internal_op": 5385, "op_name": "quiesce_inode", "events": [ { "time": "2024-04-23T10:31:21.449385+0000", "event": "initiated" }, { "time": "2024-04-23T10:31:21.449385+0000", "event": "header_read" }, { "time": "2024-04-23T10:31:21.449385+0000", "event": "throttled" }, { "time": "2024-04-23T10:31:21.449385+0000", "event": "all_read" }, { "time": "2024-04-23T10:31:21.449385+0000", "event": "dispatched" }, { "time": "2024-04-23T10:31:21.449471+0000", "event": "requesting remote authpins" }, { "time": "2024-04-23T10:31:21.450864+0000", "event": "acquired locks" } ], "locks": [ { "object": { "is_auth": false, "auth_state": { "replicas": {} }, "replica_state": { "authority": [ 0, -2 ], "replica_nonce": 1 }, "auth_pins": 0, "is_frozen": false, "is_freezing": false, "pins": { "request": 1, "lock": 1, "dirfrag": 1, "dirwaiter": 0, "discoverbase": 0 }, "nref": 3 }, "object_string": "[inode 0x100000001fb [...7,head] /volumes/qa/sv_1/b206794e-7622-4ab8-8943-f3ae4e65b7a3/ rep@0.1 v426 f(v0 m2024-04-23T10:26:48.475842+0000 1=0+1) n(v25 rc2024-04-23T10:31:17.764716+0000 b21075074 78=63+15)/n(v0 rc2024-04-23T10:26:48.480842+0000 2=0+2) old_inodes=1 (inest mix) (iquiesce lock x=1 by request(mds.1:107 nref=4)) | request=1 lock=1 dirfrag=1 dirwaiter=0 discoverbase=0 0x55e7186f1080]", "lock": { "gather_set": [], "state": "lock", "type": "iquiesce", "is_leased": false, "num_rdlocks": 0, "num_wrlocks": 0, "num_xlocks": 1, "xlock_by": { "reqid": { "entity": { "type": "mds", "num": 1 }, "tid": 107 } } }, "flags": 4, "wrlock_target": -1 } ] } }, { "description": "internal op quiesce_inode:mds.1:139 fp=#0x1000000023a", "initiated_at": "2024-04-23T10:31:21.494345+0000", "age": 42.363293798000001, "duration": 42.363957581999998, "continuous": true, "type_data": { "result": -2147483648, "flag_point": "failed to authpin, dir is being fragmented", "reqid": { "entity": { "type": "mds", "num": 1 }, "tid": 139 }, "op_type": "internal_op", "internal_op": 5385, "op_name": "quiesce_inode", "events": [ { "time": "2024-04-23T10:31:21.494345+0000", "event": "initiated" }, { "time": "2024-04-23T10:31:21.494345+0000", "event": "header_read" }, { "time": "2024-04-23T10:31:21.494345+0000", "event": "throttled" }, { "time": "2024-04-23T10:31:21.494345+0000", "event": "all_read" }, { "time": "2024-04-23T10:31:21.494345+0000", "event": "dispatched" }, { "time": "2024-04-23T10:31:21.496076+0000", "event": "failed to authpin, dir is being fragmented" } ], "locks": [] } }, { "description": "internal op quiesce_inode:mds.1:108 fp=#0x10000000005", "initiated_at": "2024-04-23T10:31:21.450873+0000", "age": 42.406765493999998, "duration": 42.407454528000002, "continuous": true, "type_data": { "result": -2147483648, "flag_point": "acquired locks", "reqid": { "entity": { "type": "mds", "num": 1 }, "tid": 108 }, "op_type": "internal_op", "internal_op": 5385, "op_name": "quiesce_inode", "events": [ { "time": "2024-04-23T10:31:21.450873+0000", "event": "initiated" }, { "time": "2024-04-23T10:31:21.450873+0000", "event": "header_read" }, { "time": "2024-04-23T10:31:21.450873+0000", "event": "throttled" }, { "time": "2024-04-23T10:31:21.450873+0000", "event": "all_read" }, { "time": "2024-04-23T10:31:21.450873+0000", "event": "dispatched" }, { "time": "2024-04-23T10:31:21.450934+0000", "event": "requesting remote authpins" }, { "time": "2024-04-23T10:31:21.452061+0000", "event": "acquired locks" } ], "locks": [ { "object": { "is_auth": false, "auth_state": { "replicas": {} }, "replica_state": { "authority": [ 0, -2 ], "replica_nonce": 1 }, "auth_pins": 0, "is_frozen": false, "is_freezing": false, "pins": { "request": 1, "lock": 1, "dirfrag": 1 }, "nref": 3 }, "object_string": "[inode 0x10000000005 [...7,head] /volumes/qa/sv_1/b206794e-7622-4ab8-8943-f3ae4e65b7a3/client.0/ rep@0.1 v421 f(v0 m2024-04-23T10:27:42.521612+0000 1=0+1) n(v32 rc2024-04-23T10:31:17.764716+0000 b21075074 77=63+14)/n(v0 rc2024-04-23T10:27:42.521612+0000 2=0+2) old_inodes=1 (inest mix) (iquiesce lock x=1 by request(mds.1:108 nref=4)) | request=1 lock=1 dirfrag=1 0x55e7186f0b00]", "lock": { "gather_set": [], "state": "lock", "type": "iquiesce", "is_leased": false, "num_rdlocks": 0, "num_wrlocks": 0, "num_xlocks": 1, "xlock_by": { "reqid": { "entity": { "type": "mds", "num": 1 }, "tid": 108 } } }, "flags": 4, "wrlock_target": -1 } ] } }, { "description": "internal op quiesce_inode:mds.1:140 fp=#0x10000000236", "initiated_at": "2024-04-23T10:31:21.494359+0000", "age": 42.363279785000003, "duration": 42.364023766000003, "continuous": true, "type_data": { "result": -2147483648, "flag_point": "failed to authpin, dir is being fragmented", "reqid": { "entity": { "type": "mds", "num": 1 }, "tid": 140 }, "op_type": "internal_op", "internal_op": 5385, "op_name": "quiesce_inode", "events": [ { "time": "2024-04-23T10:31:21.494359+0000", "event": "initiated" }, { "time": "2024-04-23T10:31:21.494359+0000", "event": "header_read" }, { "time": "2024-04-23T10:31:21.494359+0000", "event": "throttled" }, { "time": "2024-04-23T10:31:21.494359+0000", "event": "all_read" }, { "time": "2024-04-23T10:31:21.494359+0000", "event": "dispatched" }, { "time": "2024-04-23T10:31:21.496168+0000", "event": "failed to authpin, dir is being fragmented" } ], "locks": [] } }, { "description": "internal op quiesce_inode:mds.1:109 fp=#0x10000000006", "initiated_at": "2024-04-23T10:31:21.452069+0000", "age": 42.405569727, "duration": 42.406337188000002, "continuous": true, "type_data": { "result": -2147483648, "flag_point": "acquired locks", "reqid": { "entity": { "type": "mds", "num": 1 }, "tid": 109 }, "op_type": "internal_op", "internal_op": 5385, "op_name": "quiesce_inode", "events": [ { "time": "2024-04-23T10:31:21.452069+0000", "event": "initiated" }, { "time": "2024-04-23T10:31:21.452069+0000", "event": "header_read" }, { "time": "2024-04-23T10:31:21.452069+0000", "event": "throttled" }, { "time": "2024-04-23T10:31:21.452069+0000", "event": "all_read" }, { "time": "2024-04-23T10:31:21.452069+0000", "event": "dispatched" }, { "time": "2024-04-23T10:31:21.452141+0000", "event": "requesting remote authpins" }, { "time": "2024-04-23T10:31:21.453664+0000", "event": "acquired locks" } ], "locks": [ { "object": { "is_auth": false, "auth_state": { "replicas": {} }, "replica_state": { "authority": [ 0, -2 ], "replica_nonce": 1 }, "auth_pins": 0, "is_frozen": false, "is_freezing": false, "pins": { "request": 1, "lock": 1, "dirfrag": 1 }, "nref": 3 }, "object_string": "[inode 0x10000000006 [...7,head] /volumes/qa/sv_1/b206794e-7622-4ab8-8943-f3ae4e65b7a3/client.0/tmp/ rep@0.1 v408 f(v0 m2024-04-23T10:27:51.919399+0000 1=0+1) n(v30 rc2024-04-23T10:31:17.764716+0000 b21075074 76=63+13)/n(v0 rc2024-04-23T10:27:51.919399+0000 2=0+2) (inest mix) (iquiesce lock x=1 by request(mds.1:109 nref=4)) | request=1 lock=1 dirfrag=1 0x55e7186f0580]", "lock": { "gather_set": [], "state": "lock", "type": "iquiesce", "is_leased": false, "num_rdlocks": 0, "num_wrlocks": 0, "num_xlocks": 1, "xlock_by": { "reqid": { "entity": { "type": "mds", "num": 1 }, "tid": 109 } } }, "flags": 4, "wrlock_target": -1 } ] } }, { "description": "internal op quiesce_inode:mds.1:141 fp=#0x10000000232", "initiated_at": "2024-04-23T10:31:21.494372+0000", "age": 42.363266496999998, "duration": 42.364115597000001, "continuous": true, "type_data": { "result": -2147483648, "flag_point": "failed to authpin, dir is being fragmented", "reqid": { "entity": { "type": "mds", "num": 1 }, "tid": 141 }, "op_type": "internal_op", "internal_op": 5385, "op_name": "quiesce_inode", "events": [ { "time": "2024-04-23T10:31:21.494372+0000", "event": "initiated" }, { "time": "2024-04-23T10:31:21.494372+0000", "event": "header_read" }, { "time": "2024-04-23T10:31:21.494372+0000", "event": "throttled" }, { "time": "2024-04-23T10:31:21.494372+0000", "event": "all_read" }, { "time": "2024-04-23T10:31:21.494372+0000", "event": "dispatched" }, { "time": "2024-04-23T10:31:21.496272+0000", "event": "failed to authpin, dir is being fragmented" } ], "locks": [] } }, { "description": "internal op quiesce_inode:mds.1:110 fp=#0x10000000007", "initiated_at": "2024-04-23T10:31:21.453671+0000", "age": 42.403967201999997, "duration": 42.404843812999999, "continuous": true, "type_data": { "result": -2147483648, "flag_point": "acquired locks", "reqid": { "entity": { "type": "mds", "num": 1 }, "tid": 110 }, "op_type": "internal_op", "internal_op": 5385, "op_name": "quiesce_inode", "events": [ { "time": "2024-04-23T10:31:21.453671+0000", "event": "initiated" }, { "time": "2024-04-23T10:31:21.453671+0000", "event": "header_read" }, { "time": "2024-04-23T10:31:21.453671+0000", "event": "throttled" }, { "time": "2024-04-23T10:31:21.453671+0000", "event": "all_read" }, { "time": "2024-04-23T10:31:21.453671+0000", "event": "dispatched" }, { "time": "2024-04-23T10:31:21.453719+0000", "event": "requesting remote authpins" }, { "time": "2024-04-23T10:31:21.455232+0000", "event": "acquired locks" } ], "locks": [ { "object": { "is_auth": false, "auth_state": { "replicas": {} }, "replica_state": { "authority": [ 0, -2 ], "replica_nonce": 1 }, "auth_pins": 0, "is_frozen": false, "is_freezing": false, "pins": { "request": 1, "lock": 1, "dirfrag": 1 }, "nref": 3 }, "object_string": "[inode 0x10000000007 [...7,head] /volumes/qa/sv_1/b206794e-7622-4ab8-8943-f3ae4e65b7a3/client.0/tmp/clients/ rep@0.1 v412 f(v0 m2024-04-23T10:27:51.919399+0000 1=0+1) n(v30 rc2024-04-23T10:31:17.764716+0000 b21075074 75=63+12)/n(v0 rc2024-04-23T10:27:51.919399+0000 2=0+2) (inest mix) (iquiesce lock x=1 by request(mds.1:110 nref=4)) | request=1 lock=1 dirfrag=1 0x55e7186f3180]", "lock": { "gather_set": [], "state": "lock", "type": "iquiesce", "is_leased": false, "num_rdlocks": 0, "num_wrlocks": 0, "num_xlocks": 1, "xlock_by": { "reqid": { "entity": { "type": "mds", "num": 1 }, "tid": 110 } } }, "flags": 4, "wrlock_target": -1 } ] } }, { "description": "internal op quiesce_inode:mds.1:142 fp=#0x200000006f8", "initiated_at": "2024-04-23T10:31:21.494389+0000", "age": 42.363249875000001, "duration": 42.364184604000002, "continuous": true, "type_data": { "result": -2147483648, "flag_point": "failed to authpin, dir is being fragmented", "reqid": { "entity": { "type": "mds", "num": 1 }, "tid": 142 }, "op_type": "internal_op", "internal_op": 5385, "op_name": "quiesce_inode", "events": [ { "time": "2024-04-23T10:31:21.494389+0000", "event": "initiated" }, { "time": "2024-04-23T10:31:21.494389+0000", "event": "header_read" }, { "time": "2024-04-23T10:31:21.494389+0000", "event": "throttled" }, { "time": "2024-04-23T10:31:21.494389+0000", "event": "all_read" }, { "time": "2024-04-23T10:31:21.494389+0000", "event": "dispatched" }, { "time": "2024-04-23T10:31:21.496355+0000", "event": "failed to authpin, dir is being fragmented" } ], "locks": [] } }, { "description": "internal op quiesce_inode:mds.1:111 fp=#0x10000000008", "initiated_at": "2024-04-23T10:31:21.455245+0000", "age": 42.402393187000001, "duration": 42.403352562000002, "continuous": true, "type_data": { "result": -2147483648, "flag_point": "acquired locks", "reqid": { "entity": { "type": "mds", "num": 1 }, "tid": 111 }, "op_type": "internal_op", "internal_op": 5385, "op_name": "quiesce_inode", "events": [ { "time": "2024-04-23T10:31:21.455245+0000", "event": "initiated" }, { "time": "2024-04-23T10:31:21.455245+0000", "event": "header_read" }, { "time": "2024-04-23T10:31:21.455245+0000", "event": "throttled" }, { "time": "2024-04-23T10:31:21.455245+0000", "event": "all_read" }, { "time": "2024-04-23T10:31:21.455245+0000", "event": "dispatched" }, { "time": "2024-04-23T10:31:21.455296+0000", "event": "requesting remote authpins" }, { "time": "2024-04-23T10:31:21.456965+0000", "event": "acquired locks" } ], "locks": [ { "object": { "is_auth": false, "auth_state": { "replicas": {} }, "replica_state": { "authority": [ 0, -2 ], "replica_nonce": 1 }, "auth_pins": 0, "is_frozen": false, "is_freezing": false, "pins": { "request": 1, "lock": 1, "dirfrag": 1, "dirwaiter": 0, "discoverbase": 0 }, "nref": 3 }, "object_string": "[inode 0x10000000008 [...7,head] /volumes/qa/sv_1/b206794e-7622-4ab8-8943-f3ae4e65b7a3/client.0/tmp/clients/client0/ rep@0.1 v456 f(v0 m2024-04-23T10:27:51.950398+0000 6=5+1) n(v30 rc2024-04-23T10:31:17.764716+0000 b21075074 74=63+11)/n(v0 rc2024-04-23T10:27:51.920398+0000 2=0+2) (inest mix) (iquiesce lock x=1 by request(mds.1:111 nref=4)) | request=1 lock=1 dirfrag=1 dirwaiter=0 discoverbase=0 0x55e7186f3700]", "lock": { "gather_set": [], "state": "lock", "type": "iquiesce", "is_leased": false, "num_rdlocks": 0, "num_wrlocks": 0, "num_xlocks": 1, "xlock_by": { "reqid": { "entity": { "type": "mds", "num": 1 }, "tid": 111 } } }, "flags": 4, "wrlock_target": -1 } ] } }, { "description": "internal op quiesce_inode:mds.1:143 fp=#0x10000000230", "initiated_at": "2024-04-23T10:31:21.494413+0000", "age": 42.363225432999997, "duration": 42.364282203999998, "continuous": true, "type_data": { "result": -2147483648, "flag_point": "failed to authpin, dir is being fragmented", "reqid": { "entity": { "type": "mds", "num": 1 }, "tid": 143 }, "op_type": "internal_op", "internal_op": 5385, "op_name": "quiesce_inode", "events": [ { "time": "2024-04-23T10:31:21.494413+0000", "event": "initiated" }, { "time": "2024-04-23T10:31:21.494413+0000", "event": "header_read" }, { "time": "2024-04-23T10:31:21.494413+0000", "event": "throttled" }, { "time": "2024-04-23T10:31:21.494413+0000", "event": "all_read" }, { "time": "2024-04-23T10:31:21.494413+0000", "event": "dispatched" }, { "time": "2024-04-23T10:31:21.496473+0000", "event": "failed to authpin, dir is being fragmented" } ], "locks": [] } }, { "description": "internal op quiesce_inode:mds.1:112 fp=#0x10000000009", "initiated_at": "2024-04-23T10:31:21.456973+0000", "age": 42.400665381000003, "duration": 42.401747045999997, "continuous": true, "type_data": { "result": -2147483648, "flag_point": "acquired locks", "reqid": { "entity": { "type": "mds", "num": 1 }, "tid": 112 }, "op_type": "internal_op", "internal_op": 5385, "op_name": "quiesce_inode", "events": [ { "time": "2024-04-23T10:31:21.456973+0000", "event": "initiated" }, { "time": "2024-04-23T10:31:21.456973+0000", "event": "header_read" }, { "time": "2024-04-23T10:31:21.456973+0000", "event": "throttled" }, { "time": "2024-04-23T10:31:21.456973+0000", "event": "all_read" }, { "time": "2024-04-23T10:31:21.456973+0000", "event": "dispatched" }, { "time": "2024-04-23T10:31:21.457057+0000", "event": "requesting remote authpins" }, { "time": "2024-04-23T10:31:21.464343+0000", "event": "acquired locks" } ], "locks": [ { "object": { "is_auth": false, "auth_state": { "replicas": {} }, "replica_state": { "authority": [ 0, -2 ], "replica_nonce": 1 }, "auth_pins": 0, "is_frozen": false, "is_freezing": false, "pins": { "request": 1, "lock": 1, "dirfrag": 1, "discoverbase": 0 }, "nref": 3 }, "object_string": "[inode 0x10000000009 [...7,head] /volumes/qa/sv_1/b206794e-7622-4ab8-8943-f3ae4e65b7a3/client.0/tmp/clients/client0/~dmtmp/ rep@0.1 v15176 f(v0 m2024-04-23T10:27:52.186392+0000 9=0+9) n(v31 rc2024-04-23T10:31:17.764716+0000 b15832194 68=58+10)/n(v0 rc2024-04-23T10:27:51.953398+0000 2=0+2) (inest mix) (iquiesce lock x=1 by request(mds.1:112 nref=4)) | request=1 lock=1 dirfrag=1 discoverbase=0 0x55e718b90000]", "lock": { "gather_set": [], "state": "lock", "type": "iquiesce", "is_leased": false, "num_rdlocks": 0, "num_wrlocks": 0, "num_xlocks": 1, "xlock_by": { "reqid": { "entity": { "type": "mds", "num": 1 }, "tid": 112 } } }, "flags": 4, "wrlock_target": -1 } ] } }, { "description": "internal op quiesce_inode:mds.1:144 fp=#0x1000000022e", "initiated_at": "2024-04-23T10:31:21.494428+0000", "age": 42.363210180999999, "duration": 42.364371300999998, "continuous": true, "type_data": { "result": -2147483648, "flag_point": "failed to authpin, dir is being fragmented", "reqid": { "entity": { "type": "mds", "num": 1 }, "tid": 144 }, "op_type": "internal_op", "internal_op": 5385, "op_name": "quiesce_inode", "events": [ { "time": "2024-04-23T10:31:21.494428+0000", "event": "initiated" }, { "time": "2024-04-23T10:31:21.494428+0000", "event": "header_read" }, { "time": "2024-04-23T10:31:21.494428+0000", "event": "throttled" }, { "time": "2024-04-23T10:31:21.494428+0000", "event": "all_read" }, { "time": "2024-04-23T10:31:21.494428+0000", "event": "dispatched" }, { "time": "2024-04-23T10:31:21.496570+0000", "event": "failed to authpin, dir is being fragmented" } ], "locks": [] } }, { "description": "internal op quiesce_inode:mds.1:113 fp=#0x10000000215", "initiated_at": "2024-04-23T10:31:21.464355+0000", "age": 42.393283500999999, "duration": 42.394488328999998, "continuous": true, "type_data": { "result": -2147483648, "flag_point": "quiesce complete for non-auth inode", "reqid": { "entity": { "type": "mds", "num": 1 }, "tid": 113 }, "op_type": "internal_op", "internal_op": 5385, "op_name": "quiesce_inode", "events": [ { "time": "2024-04-23T10:31:21.464355+0000", "event": "initiated" }, { "time": "2024-04-23T10:31:21.464355+0000", "event": "header_read" }, { "time": "2024-04-23T10:31:21.464355+0000", "event": "throttled" }, { "time": "2024-04-23T10:31:21.464355+0000", "event": "all_read" }, { "time": "2024-04-23T10:31:21.464355+0000", "event": "dispatched" }, { "time": "2024-04-23T10:31:21.464598+0000", "event": "requesting remote authpins" }, { "time": "2024-04-23T10:31:21.478568+0000", "event": "acquired locks" }, { "time": "2024-04-23T10:31:21.478582+0000", "event": "quiesce complete for non-auth inode" } ], "locks": [ { "object": { "is_auth": false, "auth_state": { "replicas": {} }, "replica_state": { "authority": [ 0, -2 ], "replica_nonce": 1 }, "auth_pins": 0, "is_frozen": false, "is_freezing": false, "pins": { "request": 1, "lock": 1, "dirfrag": 0, "dirwaiter": 0, "discoverbase": 0 }, "nref": 2 }, "object_string": "[inode 0x10000000215 [...7,head] /volumes/qa/sv_1/b206794e-7622-4ab8-8943-f3ae4e65b7a3/client.0/tmp/clients/client0/~dmtmp/COREL/ rep@0.1 v45589 f(v1232 m2024-04-23T10:31:17.745716+0000 11=11+0) n(v19 rc2024-04-23T10:31:17.745716+0000 b2861056 12=11+1)/n(v0 rc2024-04-23T10:27:52.004397+0000 2=1+1) (inest mix) (iquiesce lock x=1 by request(mds.1:113 nref=3)) | request=1 lock=1 dirfrag=0 dirwaiter=0 discoverbase=0 0x55e718b91600]", "lock": { "gather_set": [], "state": "lock", "type": "iquiesce", "is_leased": false, "num_rdlocks": 0, "num_wrlocks": 0, "num_xlocks": 1, "xlock_by": { "reqid": { "entity": { "type": "mds", "num": 1 }, "tid": 113 } } }, "flags": 4, "wrlock_target": -1 } ] } }, { "description": "internal op quiesce_inode:mds.1:145 fp=#0x200000006f1", "initiated_at": "2024-04-23T10:31:21.494440+0000", "age": 42.363198320000002, "duration": 42.364471997999999, "continuous": true, "type_data": { "result": -2147483648, "flag_point": "failed to authpin, dir is being fragmented", "reqid": { "entity": { "type": "mds", "num": 1 }, "tid": 145 }, "op_type": "internal_op", "internal_op": 5385, "op_name": "quiesce_inode", "events": [ { "time": "2024-04-23T10:31:21.494440+0000", "event": "initiated" }, { "time": "2024-04-23T10:31:21.494440+0000", "event": "header_read" }, { "time": "2024-04-23T10:31:21.494440+0000", "event": "throttled" }, { "time": "2024-04-23T10:31:21.494440+0000", "event": "all_read" }, { "time": "2024-04-23T10:31:21.494440+0000", "event": "dispatched" }, { "time": "2024-04-23T10:31:21.496627+0000", "event": "failed to authpin, dir is being fragmented" } ], "locks": [] } }, { "description": "internal op quiesce_inode:mds.1:114 fp=#0x1000000020c", "initiated_at": "2024-04-23T10:31:21.464372+0000", "age": 42.393267004999998, "duration": 42.394574124999998, "continuous": true, "type_data": { "result": -2147483648, "flag_point": "quiesce complete for non-auth inode", "reqid": { "entity": { "type": "mds", "num": 1 }, "tid": 114 }, "op_type": "internal_op", "internal_op": 5385, "op_name": "quiesce_inode", "events": [ { "time": "2024-04-23T10:31:21.464372+0000", "event": "initiated" }, { "time": "2024-04-23T10:31:21.464372+0000", "event": "header_read" }, { "time": "2024-04-23T10:31:21.464372+0000", "event": "throttled" }, { "time": "2024-04-23T10:31:21.464372+0000", "event": "all_read" }, { "time": "2024-04-23T10:31:21.464372+0000", "event": "dispatched" }, { "time": "2024-04-23T10:31:21.464671+0000", "event": "requesting remote authpins" }, { "time": "2024-04-23T10:31:21.480456+0000", "event": "acquired locks" }, { "time": "2024-04-23T10:31:21.480474+0000", "event": "quiesce complete for non-auth inode" } ], "locks": [ { "object": { "is_auth": false, "auth_state": { "replicas": {} }, "replica_state": { "authority": [ 0, -2 ], "replica_nonce": 1 }, "auth_pins": 0, "is_frozen": false, "is_freezing": false, "pins": { "request": 1, "lock": 1, "dirwaiter": 0, "discoverbase": 0 }, "nref": 2 }, "object_string": "[inode 0x1000000020c [...7,head] /volumes/qa/sv_1/b206794e-7622-4ab8-8943-f3ae4e65b7a3/client.0/tmp/clients/client0/~dmtmp/EXCEL/ rep@0.1 v45489 f(v0 m2024-04-23T10:31:17.692717+0000 2=2+0) n(v0 rc2024-04-23T10:27:51.972397+0000 3=2+1)/n(v0 rc2024-04-23T10:27:51.967397+0000 2=1+1) (inest mix) (iquiesce lock x=1 by request(mds.1:114 nref=3)) | request=1 lock=1 dirwaiter=0 discoverbase=0 0x55e718b90b00]", "lock": { "gather_set": [], "state": "lock", "type": "iquiesce", "is_leased": false, "num_rdlocks": 0, "num_wrlocks": 0, "num_xlocks": 1, "xlock_by": { "reqid": { "entity": { "type": "mds", "num": 1 }, "tid": 114 } } }, "flags": 4, "wrlock_target": -1 } ] } }, { "description": "internal op quiesce_inode:mds.1:115 fp=#0x10000000208", "initiated_at": "2024-04-23T10:31:21.464386+0000", "age": 42.393252167999997, "duration": 42.394623408000001, "continuous": true, "type_data": { "result": -2147483648, "flag_point": "quiesce complete for non-auth inode", "reqid": { "entity": { "type": "mds", "num": 1 }, "tid": 115 }, "op_type": "internal_op", "internal_op": 5385, "op_name": "quiesce_inode", "events": [ { "time": "2024-04-23T10:31:21.464386+0000", "event": "initiated" }, { "time": "2024-04-23T10:31:21.464386+0000", "event": "header_read" }, { "time": "2024-04-23T10:31:21.464386+0000", "event": "throttled" }, { "time": "2024-04-23T10:31:21.464386+0000", "event": "all_read" }, { "time": "2024-04-23T10:31:21.464386+0000", "event": "dispatched" }, { "time": "2024-04-23T10:31:21.464740+0000", "event": "requesting remote authpins" }, { "time": "2024-04-23T10:31:21.484765+0000", "event": "acquired locks" }, { "time": "2024-04-23T10:31:21.484777+0000", "event": "quiesce complete for non-auth inode" } ], "locks": [ { "object": { "is_auth": false, "auth_state": { "replicas": {} }, "replica_state": { "authority": [ 0, -2 ], "replica_nonce": 1 }, "auth_pins": 0, "is_frozen": false, "is_freezing": false, "pins": { "request": 1, "lock": 1, "dirwaiter": 0, "discoverbase": 0 }, "nref": 2 }, "object_string": "[inode 0x10000000208 [...7,head] /volumes/qa/sv_1/b206794e-7622-4ab8-8943-f3ae4e65b7a3/client.0/tmp/clients/client0/~dmtmp/WORD/ rep@0.1 v45514 f(v0 m2024-04-23T10:31:17.675718+0000 3=3+0) n(v0 rc2024-04-23T10:27:51.958398+0000 3=2+1)/n(v0 rc2024-04-23T10:27:51.954398+0000 2=1+1) (inest mix) (iquiesce lock x=1 by request(mds.1:115 nref=3)) | request=1 lock=1 dirwaiter=0 discoverbase=0 0x55e718b90580]", "lock": { "gather_set": [], "state": "lock", "type": "iquiesce", "is_leased": false, "num_rdlocks": 0, "num_wrlocks": 0, "num_xlocks": 1, "xlock_by": { "reqid": { "entity": { "type": "mds", "num": 1 }, "tid": 115 } } }, "flags": 4, "wrlock_target": -1 } ] } }, { "description": "internal op quiesce_inode:mds.1:116 fp=#0x10000000240", "initiated_at": "2024-04-23T10:31:21.464402+0000", "age": 42.393236377000001, "duration": 42.394668606000003, "continuous": true, "type_data": { "result": -2147483648, "flag_point": "quiesce complete for non-auth inode", "reqid": { "entity": { "type": "mds", "num": 1 }, "tid": 116 }, "op_type": "internal_op", "internal_op": 5385, "op_name": "quiesce_inode", "events": [ { "time": "2024-04-23T10:31:21.464402+0000", "event": "initiated" }, { "time": "2024-04-23T10:31:21.464402+0000", "event": "header_read" }, { "time": "2024-04-23T10:31:21.464402+0000", "event": "throttled" }, { "time": "2024-04-23T10:31:21.464402+0000", "event": "all_read" }, { "time": "2024-04-23T10:31:21.464402+0000", "event": "dispatched" }, { "time": "2024-04-23T10:31:21.464820+0000", "event": "requesting remote authpins" }, { "time": "2024-04-23T10:31:21.488980+0000", "event": "acquired locks" }, { "time": "2024-04-23T10:31:21.488992+0000", "event": "quiesce complete for non-auth inode" } ], "locks": [ { "object": { "is_auth": false, "auth_state": { "replicas": {} }, "replica_state": { "authority": [ 0, -2 ], "replica_nonce": 1 }, "auth_pins": 0, "is_frozen": false, "is_freezing": false, "pins": { "request": 1, "lock": 1, "dirfrag": 0, "dirwaiter": 0, "discoverbase": 0 }, "nref": 2 }, "object_string": "[inode 0x10000000240 [...7,head] /volumes/qa/sv_1/b206794e-7622-4ab8-8943-f3ae4e65b7a3/client.0/tmp/clients/client0/~dmtmp/SEED/ rep@0.1 v45361 f(v0 m2024-04-23T10:27:52.196392+0000 3=3+0) n(v2 rc2024-04-23T10:28:22.485704+0000 b3244032 4=3+1)/n(v0 rc2024-04-23T10:27:52.187392+0000 2=1+1) (inest mix) (iquiesce lock x=1 by request(mds.1:116 nref=3)) | request=1 lock=1 dirfrag=0 dirwaiter=0 discoverbase=0 0x55e718b92c00]", "lock": { "gather_set": [], "state": "lock", "type": "iquiesce", "is_leased": false, "num_rdlocks": 0, "num_wrlocks": 0, "num_xlocks": 1, "xlock_by": { "reqid": { "entity": { "type": "mds", "num": 1 }, "tid": 116 } } }, "flags": 4, "wrlock_target": -1 } ] } }, { "description": "internal op quiesce_inode:mds.1:117 fp=#0x10000000221", "initiated_at": "2024-04-23T10:31:21.464418+0000", "age": 42.393221013999998, "duration": 42.394714657999998, "continuous": true, "type_data": { "result": -2147483648, "flag_point": "quiesce complete for non-auth inode", "reqid": { "entity": { "type": "mds", "num": 1 }, "tid": 117 }, "op_type": "internal_op", "internal_op": 5385, "op_name": "quiesce_inode", "events": [ { "time": "2024-04-23T10:31:21.464418+0000", "event": "initiated" }, { "time": "2024-04-23T10:31:21.464418+0000", "event": "header_read" }, { "time": "2024-04-23T10:31:21.464418+0000", "event": "throttled" }, { "time": "2024-04-23T10:31:21.464418+0000", "event": "all_read" }, { "time": "2024-04-23T10:31:21.464418+0000", "event": "dispatched" }, { "time": "2024-04-23T10:31:21.464887+0000", "event": "requesting remote authpins" }, { "time": "2024-04-23T10:31:21.493110+0000", "event": "acquired locks" }, { "time": "2024-04-23T10:31:21.493124+0000", "event": "quiesce complete for non-auth inode" } ], "locks": [ { "object": { "is_auth": false, "auth_state": { "replicas": {} }, "replica_state": { "authority": [ 0, -2 ], "replica_nonce": 1 }, "auth_pins": 0, "is_frozen": false, "is_freezing": false, "pins": { "request": 1, "lock": 1, "dirwaiter": 0, "discoverbase": 0 }, "nref": 2 }, "object_string": "[inode 0x10000000221 [...7,head] /volumes/qa/sv_1/b206794e-7622-4ab8-8943-f3ae4e65b7a3/client.0/tmp/clients/client0/~dmtmp/WORDPRO/ rep@0.1 v45516 f(v0 m2024-04-23T10:31:17.760716+0000 3=3+0) n(v0 rc2024-04-23T10:27:52.061395+0000 3=2+1)/n(v0 rc2024-04-23T10:27:52.057395+0000 2=1+1) (inest mix) (iquiesce lock x=1 by request(mds.1:117 nref=3)) | request=1 lock=1 dirwaiter=0 discoverbase=0 0x55e718b91b80]", "lock": { "gather_set": [], "state": "lock", "type": "iquiesce", "is_leased": false, "num_rdlocks": 0, "num_wrlocks": 0, "num_xlocks": 1, "xlock_by": { "reqid": { "entity": { "type": "mds", "num": 1 }, "tid": 117 } } }, "flags": 4, "wrlock_target": -1 } ] } }, { "description": "internal op quiesce_inode:mds.1:118 fp=#0x10000000229", "initiated_at": "2024-04-23T10:31:21.464436+0000", "age": 42.393202465999998, "duration": 42.394761938000002, "continuous": true, "type_data": { "result": -2147483648, "flag_point": "acquired locks", "reqid": { "entity": { "type": "mds", "num": 1 }, "tid": 118 }, "op_type": "internal_op", "internal_op": 5385, "op_name": "quiesce_inode", "events": [ { "time": "2024-04-23T10:31:21.464436+0000", "event": "initiated" }, { "time": "2024-04-23T10:31:21.464436+0000", "event": "header_read" }, { "time": "2024-04-23T10:31:21.464436+0000", "event": "throttled" }, { "time": "2024-04-23T10:31:21.464436+0000", "event": "all_read" }, { "time": "2024-04-23T10:31:21.464436+0000", "event": "dispatched" }, { "time": "2024-04-23T10:31:21.464971+0000", "event": "requesting remote authpins" }, { "time": "2024-04-23T10:31:21.494019+0000", "event": "acquired locks" } ], "locks": [ { "object": { "is_auth": false, "auth_state": { "replicas": {} }, "replica_state": { "authority": [ 0, -2 ], "replica_nonce": 1 }, "auth_pins": 0, "is_frozen": false, "is_freezing": false, "pins": { "dirtyscattered": 0, "request": 2, "lock": 1, "importing": 0, "dirfrag": 1, "caps": 1, "dirwaiter": 0, "waiter": 1, "discoverbase": 0 }, "nref": 6 }, "object_string": "[inode 0x10000000229 [...7,head] /volumes/qa/sv_1/b206794e-7622-4ab8-8943-f3ae4e65b7a3/client.0/tmp/clients/client0/~dmtmp/PARADOX/ rep@0.1 v45567 f(v1976 m2024-04-23T10:31:16.541744+0000 24=24+0) n(v19 rc2024-04-23T10:31:16.541744+0000 b5201920 25=24+1)/n(v0 rc2024-04-23T10:27:52.084395+0000 2=1+1) (inest mix) (iquiesce lock x=1 by request(mds.1:118 nref=4)) caps={15312=pAsLsXsFs/pAsLsXsFsxcral@10631} | dirtyscattered=0 request=2 lock=1 importing=0 dirfrag=1 caps=1 dirwaiter=0 waiter=1 discoverbase=0 0x55e718b92680]", "lock": { "gather_set": [], "state": "lock", "type": "iquiesce", "is_leased": false, "num_rdlocks": 0, "num_wrlocks": 0, "num_xlocks": 1, "xlock_by": { "reqid": { "entity": { "type": "mds", "num": 1 }, "tid": 118 } } }, "flags": 4, "wrlock_target": -1 } ] } }, { "description": "internal op quiesce_inode:mds.1:119 fp=#0x10000000225", "initiated_at": "2024-04-23T10:31:21.464459+0000", "age": 42.393179863, "duration": 42.394805542999997, "continuous": true, "type_data": { "result": -2147483648, "flag_point": "quiesce complete for non-auth inode", "reqid": { "entity": { "type": "mds", "num": 1 }, "tid": 119 }, "op_type": "internal_op", "internal_op": 5385, "op_name": "quiesce_inode", "events": [ { "time": "2024-04-23T10:31:21.464459+0000", "event": "initiated" }, { "time": "2024-04-23T10:31:21.464459+0000", "event": "header_read" }, { "time": "2024-04-23T10:31:21.464459+0000", "event": "throttled" }, { "time": "2024-04-23T10:31:21.464459+0000", "event": "all_read" }, { "time": "2024-04-23T10:31:21.464459+0000", "event": "dispatched" }, { "time": "2024-04-23T10:31:21.465043+0000", "event": "requesting remote authpins" }, { "time": "2024-04-23T10:31:21.496741+0000", "event": "acquired locks" }, { "time": "2024-04-23T10:31:21.496754+0000", "event": "quiesce complete for non-auth inode" } ], "locks": [ { "object": { "is_auth": false, "auth_state": { "replicas": {} }, "replica_state": { "authority": [ 0, -2 ], "replica_nonce": 1 }, "auth_pins": 0, "is_frozen": false, "is_freezing": false, "pins": { "request": 1, "lock": 1, "dirfrag": 0, "dirwaiter": 0, "discoverbase": 0 }, "nref": 2 }, "object_string": "[inode 0x10000000225 [...7,head] /volumes/qa/sv_1/b206794e-7622-4ab8-8943-f3ae4e65b7a3/client.0/tmp/clients/client0/~dmtmp/PM/ rep@0.1 v45574 f(v2926 m2024-04-23T10:31:17.764716+0000 3=3+0) n(v17 rc2024-04-23T10:31:17.764716+0000 b780288 4=3+1)/n(v0 rc2024-04-23T10:27:52.071395+0000 2=1+1) (inest mix) (iquiesce lock x=1 by request(mds.1:119 nref=3)) | request=1 lock=1 dirfrag=0 dirwaiter=0 discoverbase=0 0x55e718b92100]", "lock": { "gather_set": [], "state": "lock", "type": "iquiesce", "is_leased": false, "num_rdlocks": 0, "num_wrlocks": 0, "num_xlocks": 1, "xlock_by": { "reqid": { "entity": { "type": "mds", "num": 1 }, "tid": 119 } } }, "flags": 4, "wrlock_target": -1 } ] } }, { "description": "internal op quiesce_inode:mds.1:120 fp=#0x1000000020f", "initiated_at": "2024-04-23T10:31:21.464470+0000", "age": 42.393168217000003, "duration": 42.394857037000001, "continuous": true, "type_data": { "result": -2147483648, "flag_point": "quiesce complete for non-auth inode", "reqid": { "entity": { "type": "mds", "num": 1 }, "tid": 120 }, "op_type": "internal_op", "internal_op": 5385, "op_name": "quiesce_inode", "events": [ { "time": "2024-04-23T10:31:21.464470+0000", "event": "initiated" }, { "time": "2024-04-23T10:31:21.464470+0000", "event": "header_read" }, { "time": "2024-04-23T10:31:21.464470+0000", "event": "throttled" }, { "time": "2024-04-23T10:31:21.464470+0000", "event": "all_read" }, { "time": "2024-04-23T10:31:21.464470+0000", "event": "dispatched" }, { "time": "2024-04-23T10:31:21.465112+0000", "event": "requesting remote authpins" }, { "time": "2024-04-23T10:31:21.496858+0000", "event": "acquired locks" }, { "time": "2024-04-23T10:31:21.496882+0000", "event": "quiesce complete for non-auth inode" } ], "locks": [ { "object": { "is_auth": false, "auth_state": { "replicas": {} }, "replica_state": { "authority": [ 0, -2 ], "replica_nonce": 1 }, "auth_pins": 0, "is_frozen": false, "is_freezing": false, "pins": { "request": 1, "lock": 1, "dirfrag": 0, "dirwaiter": 0, "discoverbase": 0 }, "nref": 2 }, "object_string": "[inode 0x1000000020f [...7,head] /volumes/qa/sv_1/b206794e-7622-4ab8-8943-f3ae4e65b7a3/client.0/tmp/clients/client0/~dmtmp/PWRPNT/ rep@0.1 v45572 f(v1293 m2024-04-23T10:31:17.718717+0000 5=5+0) n(v18 rc2024-04-23T10:31:17.718717+0000 b1300480 6=5+1)/n(v0 rc2024-04-23T10:27:51.976397+0000 2=1+1) (inest mix) (iquiesce lock x=1 by request(mds.1:120 nref=3)) | request=1 lock=1 dirfrag=0 dirwaiter=0 discoverbase=0 0x55e718b91080]", "lock": { "gather_set": [], "state": "lock", "type": "iquiesce", "is_leased": false, "num_rdlocks": 0, "num_wrlocks": 0, "num_xlocks": 1, "xlock_by": { "reqid": { "entity": { "type": "mds", "num": 1 }, "tid": 120 } } }, "flags": 4, "wrlock_target": -1 } ] } }, { "description": "internal op quiesce_inode:mds.1:121 fp=#0x1000000023e", "initiated_at": "2024-04-23T10:31:21.464493+0000", "age": 42.393145695000001, "duration": 42.394896596000002, "continuous": true, "type_data": { "result": -2147483648, "flag_point": "quiesce complete for non-auth inode", "reqid": { "entity": { "type": "mds", "num": 1 }, "tid": 121 }, "op_type": "internal_op", "internal_op": 5385, "op_name": "quiesce_inode", "events": [ { "time": "2024-04-23T10:31:21.464493+0000", "event": "initiated" }, { "time": "2024-04-23T10:31:21.464493+0000", "event": "header_read" }, { "time": "2024-04-23T10:31:21.464493+0000", "event": "throttled" }, { "time": "2024-04-23T10:31:21.464493+0000", "event": "all_read" }, { "time": "2024-04-23T10:31:21.464493+0000", "event": "dispatched" }, { "time": "2024-04-23T10:31:21.465182+0000", "event": "requesting remote authpins" }, { "time": "2024-04-23T10:31:21.498658+0000", "event": "acquired locks" }, { "time": "2024-04-23T10:31:21.498671+0000", "event": "quiesce complete for non-auth inode" } ], "locks": [ { "object": { "is_auth": false, "auth_state": { "replicas": {} }, "replica_state": { "authority": [ 0, -2 ], "replica_nonce": 1 }, "auth_pins": 0, "is_frozen": false, "is_freezing": false, "pins": { "request": 1, "lock": 1, "dirwaiter": 0, "discoverbase": 0 }, "nref": 2 }, "object_string": "[inode 0x1000000023e [...7,head] /volumes/qa/sv_1/b206794e-7622-4ab8-8943-f3ae4e65b7a3/client.0/tmp/clients/client0/~dmtmp/ACCESS/ rep@0.1 v45357 f(v0 m2024-04-23T10:31:17.039732+0000 4=4+0) n(v0 rc2024-04-23T10:27:52.903376+0000 b260096 3=2+1)/n(v0 rc2024-04-23T10:27:52.181393+0000 2=1+1) (inest mix) (iquiesce lock x=1 by request(mds.1:121 nref=3)) | request=1 lock=1 dirwaiter=0 discoverbase=0 0x55e718b93180]", "lock": { "gather_set": [], "state": "lock", "type": "iquiesce", "is_leased": false, "num_rdlocks": 0, "num_wrlocks": 0, "num_xlocks": 1, "xlock_by": { "reqid": { "entity": { "type": "mds", "num": 1 }, "tid": 121 } } }, "flags": 4, "wrlock_target": -1 } ] } }, { "description": "internal op quiesce_inode:mds.1:122 fp=#0x1000000022b", "initiated_at": "2024-04-23T10:31:21.494037+0000", "age": 42.363601457999998, "duration": 42.365420164, "continuous": true, "type_data": { "result": -2147483648, "flag_point": "failed to authpin, dir is being fragmented", "reqid": { "entity": { "type": "mds", "num": 1 }, "tid": 122 }, "op_type": "internal_op", "internal_op": 5385, "op_name": "quiesce_inode", "events": [ { "time": "2024-04-23T10:31:21.494037+0000", "event": "initiated" }, { "time": "2024-04-23T10:31:21.494037+0000", "event": "header_read" }, { "time": "2024-04-23T10:31:21.494037+0000", "event": "throttled" }, { "time": "2024-04-23T10:31:21.494037+0000", "event": "all_read" }, { "time": "2024-04-23T10:31:21.494037+0000", "event": "dispatched" }, { "time": "2024-04-23T10:31:21.494549+0000", "event": "failed to authpin, dir is being fragmented" } ], "locks": [] } }, { "description": "internal op quiesce_inode:mds.1:123 fp=#0x200000006f9", "initiated_at": "2024-04-23T10:31:21.494053+0000", "age": 42.363585348999997, "duration": 42.365431334999997, "continuous": true, "type_data": { "result": -2147483648, "flag_point": "failed to authpin, dir is being fragmented", "reqid": { "entity": { "type": "mds", "num": 1 }, "tid": 123 }, "op_type": "internal_op", "internal_op": 5385, "op_name": "quiesce_inode", "events": [ { "time": "2024-04-23T10:31:21.494053+0000", "event": "initiated" }, { "time": "2024-04-23T10:31:21.494053+0000", "event": "header_read" }, { "time": "2024-04-23T10:31:21.494053+0000", "event": "throttled" }, { "time": "2024-04-23T10:31:21.494053+0000", "event": "all_read" }, { "time": "2024-04-23T10:31:21.494053+0000", "event": "dispatched" }, { "time": "2024-04-23T10:31:21.494624+0000", "event": "failed to authpin, dir is being fragmented" } ], "locks": [] } }, { "description": "internal op quiesce_inode:mds.1:124 fp=#0x1000000022c", "initiated_at": "2024-04-23T10:31:21.494068+0000", "age": 42.363570078999999, "duration": 42.365441021000002, "continuous": true, "type_data": { "result": -2147483648, "flag_point": "failed to authpin, dir is being fragmented", "reqid": { "entity": { "type": "mds", "num": 1 }, "tid": 124 }, "op_type": "internal_op", "internal_op": 5385, "op_name": "quiesce_inode", "events": [ { "time": "2024-04-23T10:31:21.494068+0000", "event": "initiated" }, { "time": "2024-04-23T10:31:21.494068+0000", "event": "header_read" }, { "time": "2024-04-23T10:31:21.494068+0000", "event": "throttled" }, { "time": "2024-04-23T10:31:21.494068+0000", "event": "all_read" }, { "time": "2024-04-23T10:31:21.494068+0000", "event": "dispatched" }, { "time": "2024-04-23T10:31:21.494716+0000", "event": "failed to authpin, dir is being fragmented" } ], "locks": [] } }, { "description": "internal op quiesce_inode:mds.1:125 fp=#0x10000000231", "initiated_at": "2024-04-23T10:31:21.494090+0000", "age": 42.363548352000002, "duration": 42.365445117999997, "continuous": true, "type_data": { "result": -2147483648, "flag_point": "failed to authpin, dir is being fragmented", "reqid": { "entity": { "type": "mds", "num": 1 }, "tid": 125 }, "op_type": "internal_op", "internal_op": 5385, "op_name": "quiesce_inode", "events": [ { "time": "2024-04-23T10:31:21.494090+0000", "event": "initiated" }, { "time": "2024-04-23T10:31:21.494090+0000", "event": "header_read" }, { "time": "2024-04-23T10:31:21.494090+0000", "event": "throttled" }, { "time": "2024-04-23T10:31:21.494090+0000", "event": "all_read" }, { "time": "2024-04-23T10:31:21.494090+0000", "event": "dispatched" }, { "time": "2024-04-23T10:31:21.494806+0000", "event": "failed to authpin, dir is being fragmented" } ], "locks": [] } }, { "description": "internal op fragmentdir:mds.1:100", "initiated_at": "2024-04-23T10:31:16.892500+0000", "age": 46.965138816, "duration": 46.967060001999997, "continuous": false, "type_data": { "result": -2147483648, "flag_point": "failed to acquire quiesce lock", "reqid": { "entity": { "type": "mds", "num": 1 }, "tid": 100 }, "op_type": "internal_op", "internal_op": 5376, "op_name": "fragmentdir", "events": [ { "time": "2024-04-23T10:31:16.892500+0000", "event": "initiated" }, { "time": "2024-04-23T10:31:16.892500+0000", "event": "header_read" }, { "time": "2024-04-23T10:31:16.892500+0000", "event": "throttled" }, { "time": "2024-04-23T10:31:16.892500+0000", "event": "all_read" }, { "time": "2024-04-23T10:31:16.892500+0000", "event": "dispatched" }, { "time": "2024-04-23T10:31:22.426076+0000", "event": "requesting remote authpins" }, { "time": "2024-04-23T10:31:22.426703+0000", "event": "failed to acquire quiesce lock" } ], "locks": [] } }, { "description": "internal op quiesce_inode:mds.1:126 fp=#0x10000000237", "initiated_at": "2024-04-23T10:31:21.494106+0000", "age": 42.363532720000002, "duration": 42.365481694000003, "continuous": true, "type_data": { "result": -2147483648, "flag_point": "failed to authpin, dir is being fragmented", "reqid": { "entity": { "type": "mds", "num": 1 }, "tid": 126 }, "op_type": "internal_op", "internal_op": 5385, "op_name": "quiesce_inode", "events": [ { "time": "2024-04-23T10:31:21.494106+0000", "event": "initiated" }, { "time": "2024-04-23T10:31:21.494106+0000", "event": "header_read" }, { "time": "2024-04-23T10:31:21.494106+0000", "event": "throttled" }, { "time": "2024-04-23T10:31:21.494106+0000", "event": "all_read" }, { "time": "2024-04-23T10:31:21.494106+0000", "event": "dispatched" }, { "time": "2024-04-23T10:31:21.494900+0000", "event": "failed to authpin, dir is being fragmented" } ], "locks": [] } }, { "description": "internal op quiesce_inode:mds.1:127 fp=#0x200000006f3", "initiated_at": "2024-04-23T10:31:21.494119+0000", "age": 42.363519453000002, "duration": 42.365492904, "continuous": true, "type_data": { "result": -2147483648, "flag_point": "failed to authpin, dir is being fragmented", "reqid": { "entity": { "type": "mds", "num": 1 }, "tid": 127 }, "op_type": "internal_op", "internal_op": 5385, "op_name": "quiesce_inode", "events": [ { "time": "2024-04-23T10:31:21.494119+0000", "event": "initiated" }, { "time": "2024-04-23T10:31:21.494119+0000", "event": "header_read" }, { "time": "2024-04-23T10:31:21.494119+0000", "event": "throttled" }, { "time": "2024-04-23T10:31:21.494119+0000", "event": "all_read" }, { "time": "2024-04-23T10:31:21.494119+0000", "event": "dispatched" }, { "time": "2024-04-23T10:31:21.494960+0000", "event": "failed to authpin, dir is being fragmented" } ], "locks": [] } }, { "description": "internal op quiesce_inode:mds.1:128 fp=#0x1000000022f", "initiated_at": "2024-04-23T10:31:21.494154+0000", "age": 42.363484606999997, "duration": 42.365482450000002, "continuous": true, "type_data": { "result": -2147483648, "flag_point": "failed to authpin, dir is being fragmented", "reqid": { "entity": { "type": "mds", "num": 1 }, "tid": 128 }, "op_type": "internal_op", "internal_op": 5385, "op_name": "quiesce_inode", "events": [ { "time": "2024-04-23T10:31:21.494154+0000", "event": "initiated" }, { "time": "2024-04-23T10:31:21.494154+0000", "event": "header_read" }, { "time": "2024-04-23T10:31:21.494154+0000", "event": "throttled" }, { "time": "2024-04-23T10:31:21.494154+0000", "event": "all_read" }, { "time": "2024-04-23T10:31:21.494154+0000", "event": "dispatched" }, { "time": "2024-04-23T10:31:21.495048+0000", "event": "failed to authpin, dir is being fragmented" } ], "locks": [] } }, { "description": "internal op quiesce_inode:mds.1:129 fp=#0x1000000023b", "initiated_at": "2024-04-23T10:31:21.494170+0000", "age": 42.363468840000003, "duration": 42.365558710000002, "continuous": true, "type_data": { "result": -2147483648, "flag_point": "failed to authpin, dir is being fragmented", "reqid": { "entity": { "type": "mds", "num": 1 }, "tid": 129 }, "op_type": "internal_op", "internal_op": 5385, "op_name": "quiesce_inode", "events": [ { "time": "2024-04-23T10:31:21.494170+0000", "event": "initiated" }, { "time": "2024-04-23T10:31:21.494170+0000", "event": "header_read" }, { "time": "2024-04-23T10:31:21.494170+0000", "event": "throttled" }, { "time": "2024-04-23T10:31:21.494170+0000", "event": "all_read" }, { "time": "2024-04-23T10:31:21.494170+0000", "event": "dispatched" }, { "time": "2024-04-23T10:31:21.495132+0000", "event": "failed to authpin, dir is being fragmented" } ], "locks": [] } }, { "description": "internal op quiesce_inode:mds.1:130 fp=#0x1000000023c", "initiated_at": "2024-04-23T10:31:21.494185+0000", "age": 42.363453972000002, "duration": 42.365585023000001, "continuous": true, "type_data": { "result": -2147483648, "flag_point": "failed to authpin, dir is being fragmented", "reqid": { "entity": { "type": "mds", "num": 1 }, "tid": 130 }, "op_type": "internal_op", "internal_op": 5385, "op_name": "quiesce_inode", "events": [ { "time": "2024-04-23T10:31:21.494185+0000", "event": "initiated" }, { "time": "2024-04-23T10:31:21.494185+0000", "event": "header_read" }, { "time": "2024-04-23T10:31:21.494185+0000", "event": "throttled" }, { "time": "2024-04-23T10:31:21.494185+0000", "event": "all_read" }, { "time": "2024-04-23T10:31:21.494185+0000", "event": "dispatched" }, { "time": "2024-04-23T10:31:21.495209+0000", "event": "failed to authpin, dir is being fragmented" } ], "locks": [] } } ], "num_ops": 47 } mds.b: { "ops": [ { "description": "peer_request:mds.1:110", "initiated_at": "2024-04-23T10:31:21.453770+0000", "age": 42.430619086999997, "duration": 42.430634918000003, "continuous": false, "type_data": { "result": -2147483648, "flag_point": "dispatched", "reqid": { "entity": { "type": "mds", "num": 1 }, "tid": 110 }, "op_type": "peer_request", "leader_info": { "leader": "1" }, "events": [ { "time": "2024-04-23T10:31:21.453770+0000", "event": "initiated" }, { "time": "2024-04-23T10:31:21.453770+0000", "event": "header_read" }, { "time": "2024-04-23T10:31:21.453771+0000", "event": "throttled" }, { "time": "2024-04-23T10:31:21.453778+0000", "event": "all_read" }, { "time": "2024-04-23T10:31:21.454984+0000", "event": "dispatched" } ], "locks": [] } }, { "description": "internal op quiesce_inode:mds.0:186 fp=#0x100000014db", "initiated_at": "2024-04-23T10:31:21.471262+0000", "age": 42.413126943999998, "duration": 42.413170811000001, "continuous": true, "type_data": { "result": -2147483648, "flag_point": "quiesce complete", "reqid": { "entity": { "type": "mds", "num": 0 }, "tid": 186 }, "op_type": "internal_op", "internal_op": 5385, "op_name": "quiesce_inode", "events": [ { "time": "2024-04-23T10:31:21.471262+0000", "event": "initiated" }, { "time": "2024-04-23T10:31:21.471262+0000", "event": "header_read" }, { "time": "2024-04-23T10:31:21.471262+0000", "event": "throttled" }, { "time": "2024-04-23T10:31:21.471262+0000", "event": "all_read" }, { "time": "2024-04-23T10:31:21.471262+0000", "event": "dispatched" }, { "time": "2024-04-23T10:31:21.471540+0000", "event": "failed to xlock, waiting" }, { "time": "2024-04-23T10:31:21.498297+0000", "event": "failed to xlock, waiting" }, { "time": "2024-04-23T10:31:21.503163+0000", "event": "acquired locks" }, { "time": "2024-04-23T10:31:21.503306+0000", "event": "quiesce complete" } ], "locks": [ { "object": { "is_auth": true, "auth_state": { "replicas": {} }, "replica_state": { "authority": [ 0, -2 ], "replica_nonce": 0 }, "auth_pins": 1, "is_frozen": false, "is_freezing": false, "pins": { "ptrwaiter": 0, "request": 1, "lock": 1, "caps": 0, "dirtyparent": 0, "dirty": 0, "waiter": 0, "authpin": 1 }, "nref": 3 }, "object_string": "[inode 0x100000014db [7,head] /volumes/qa/sv_1/b206794e-7622-4ab8-8943-f3ae4e65b7a3/client.0/tmp/clients/client0/~dmtmp/ACCESS/INV.PRN auth v3326 ap=1 s=62591 n(v0 rc2024-04-23T10:31:17.015733+0000 b62591 1=1+0) (iquiesce lock x=1 by request(mds.0:186 nref=3)) | ptrwaiter=0 request=1 lock=1 caps=0 dirtyparent=0 dirty=0 waiter=0 authpin=1 0x5571cf822100]", "lock": { "gather_set": [], "state": "lock", "type": "iquiesce", "is_leased": false, "num_rdlocks": 0, "num_wrlocks": 0, "num_xlocks": 1, "xlock_by": { "reqid": { "entity": { "type": "mds", "num": 0 }, "tid": 186 } } }, "flags": 4, "wrlock_target": -1 } ] } }, { "description": "peer_request:mds.1:121", "initiated_at": "2024-04-23T10:31:21.465203+0000", "age": 42.419186437, "duration": 42.419321269000001, "continuous": false, "type_data": { "result": -2147483648, "flag_point": "dispatched", "reqid": { "entity": { "type": "mds", "num": 1 }, "tid": 121 }, "op_type": "peer_request", "leader_info": { "leader": "1" }, "events": [ { "time": "2024-04-23T10:31:21.465203+0000", "event": "initiated" }, { "time": "2024-04-23T10:31:21.465203+0000", "event": "header_read" }, { "time": "2024-04-23T10:31:21.465203+0000", "event": "throttled" }, { "time": "2024-04-23T10:31:21.465208+0000", "event": "all_read" }, { "time": "2024-04-23T10:31:21.498343+0000", "event": "dispatched" } ], "locks": [] } }, { "description": "internal op quiesce_inode:mds.0:159 fp=#0x10000000008", "initiated_at": "2024-04-23T10:31:21.455845+0000", "age": 42.428544623999997, "duration": 42.428702510999997, "continuous": true, "type_data": { "result": -2147483648, "flag_point": "quiesce complete", "reqid": { "entity": { "type": "mds", "num": 0 }, "tid": 159 }, "op_type": "internal_op", "internal_op": 5385, "op_name": "quiesce_inode", "events": [ { "time": "2024-04-23T10:31:21.455845+0000", "event": "initiated" }, { "time": "2024-04-23T10:31:21.455845+0000", "event": "header_read" }, { "time": "2024-04-23T10:31:21.455845+0000", "event": "throttled" }, { "time": "2024-04-23T10:31:21.455845+0000", "event": "all_read" }, { "time": "2024-04-23T10:31:21.455845+0000", "event": "dispatched" }, { "time": "2024-04-23T10:31:21.456206+0000", "event": "failed to xlock, waiting" }, { "time": "2024-04-23T10:31:21.457273+0000", "event": "acquired locks" }, { "time": "2024-04-23T10:31:21.509852+0000", "event": "quiesce complete" } ], "locks": [ { "object": { "is_auth": true, "auth_state": { "replicas": { "1": 1, "2": 1 } }, "replica_state": { "authority": [ 0, -2 ], "replica_nonce": 0 }, "auth_pins": 3, "is_frozen": false, "is_freezing": false, "pins": { "dirtyscattered": 0, "request": 1, "lock": 1, "dirfrag": 1, "caps": 1, "dirtyrstat": 0, "dirtyparent": 0, "replicated": 1, "dirty": 1, "waiter": 0, "authpin": 1 }, "nref": 7 }, "object_string": "[inode 0x10000000008 [...7,head] /volumes/qa/sv_1/b206794e-7622-4ab8-8943-f3ae4e65b7a3/client.0/tmp/clients/client0/ auth{1=1,2=1} v456 ap=3 f(v0 m2024-04-23T10:27:51.950398+0000 6=5+1) n(v30 rc2024-04-23T10:31:17.764716+0000 b21075074 74=63+11) old_inodes=4 (inest mix) (iquiesce lock x=1 by request(mds.0:159 nref=3)) caps={15312=pAsLsXsFs/pAsLsXsFs@29},l=15312 | dirtyscattered=0 request=1 lock=1 dirfrag=1 caps=1 dirtyrstat=0 dirtyparent=0 replicated=1 dirty=1 waiter=0 authpin=1 0x5571cf4d4580]", "lock": { "gather_set": [], "state": "lock", "type": "iquiesce", "is_leased": false, "num_rdlocks": 0, "num_wrlocks": 0, "num_xlocks": 1, "xlock_by": { "reqid": { "entity": { "type": "mds", "num": 0 }, "tid": 159 } } }, "flags": 4, "wrlock_target": -1 } ] } }, { "description": "internal op quiesce_inode:mds.0:187 fp=#0x1000000023f", "initiated_at": "2024-04-23T10:31:21.471282+0000", "age": 42.413107613000001, "duration": 42.413338134, "continuous": true, "type_data": { "result": -2147483648, "flag_point": "quiesce complete", "reqid": { "entity": { "type": "mds", "num": 0 }, "tid": 187 }, "op_type": "internal_op", "internal_op": 5385, "op_name": "quiesce_inode", "events": [ { "time": "2024-04-23T10:31:21.471282+0000", "event": "initiated" }, { "time": "2024-04-23T10:31:21.471282+0000", "event": "header_read" }, { "time": "2024-04-23T10:31:21.471282+0000", "event": "throttled" }, { "time": "2024-04-23T10:31:21.471282+0000", "event": "all_read" }, { "time": "2024-04-23T10:31:21.471282+0000", "event": "dispatched" }, { "time": "2024-04-23T10:31:21.471808+0000", "event": "failed to xlock, waiting" }, { "time": "2024-04-23T10:31:21.500182+0000", "event": "failed to xlock, waiting" }, { "time": "2024-04-23T10:31:21.503967+0000", "event": "acquired locks" }, { "time": "2024-04-23T10:31:21.504156+0000", "event": "quiesce complete" } ], "locks": [ { "object": { "is_auth": true, "auth_state": { "replicas": {} }, "replica_state": { "authority": [ 0, -2 ], "replica_nonce": 0 }, "auth_pins": 1, "is_frozen": false, "is_freezing": false, "pins": { "ptrwaiter": 0, "request": 1, "lock": 1, "caps": 0, "needsnapflush": 0, "dirtyparent": 0, "dirty": 0, "waiter": 0, "authpin": 1 }, "nref": 3 }, "object_string": "[inode 0x1000000023f [7,head] /volumes/qa/sv_1/b206794e-7622-4ab8-8943-f3ae4e65b7a3/client.0/tmp/clients/client0/~dmtmp/ACCESS/FASTENER.MDB auth v3324 ap=1 s=260096 n(v0 rc2024-04-23T10:31:17.038732+0000 b260096 1=1+0) (iquiesce lock x=1 by request(mds.0:187 nref=3)) | ptrwaiter=0 request=1 lock=1 caps=0 needsnapflush=0 dirtyparent=0 dirty=0 waiter=0 authpin=1 0x5571cf56ac00]", "lock": { "gather_set": [], "state": "lock", "type": "iquiesce", "is_leased": false, "num_rdlocks": 0, "num_wrlocks": 0, "num_xlocks": 1, "xlock_by": { "reqid": { "entity": { "type": "mds", "num": 0 }, "tid": 187 } } }, "flags": 4, "wrlock_target": -1 } ] } }, { "description": "peer_request:mds.2:137", "initiated_at": "2024-04-23T10:31:21.489495+0000", "age": 42.394893967999998, "duration": 42.395189960000003, "continuous": false, "type_data": { "result": -2147483648, "flag_point": "dispatched", "reqid": { "entity": { "type": "mds", "num": 2 }, "tid": 137 }, "op_type": "peer_request", "leader_info": { "leader": "2" }, "events": [ { "time": "2024-04-23T10:31:21.489495+0000", "event": "initiated" }, { "time": "2024-04-23T10:31:21.489495+0000", "event": "header_read" }, { "time": "2024-04-23T10:31:21.489496+0000", "event": "throttled" }, { "time": "2024-04-23T10:31:21.489503+0000", "event": "all_read" }, { "time": "2024-04-23T10:31:21.508236+0000", "event": "dispatched" } ], "locks": [] } }, { "description": "peer_request:mds.2:115", "initiated_at": "2024-04-23T10:31:21.455271+0000", "age": 42.429118481000003, "duration": 42.429435904000002, "continuous": false, "type_data": { "result": -2147483648, "flag_point": "dispatched", "reqid": { "entity": { "type": "mds", "num": 2 }, "tid": 115 }, "op_type": "peer_request", "leader_info": { "leader": "2" }, "events": [ { "time": "2024-04-23T10:31:21.455271+0000", "event": "initiated" }, { "time": "2024-04-23T10:31:21.455271+0000", "event": "header_read" }, { "time": "2024-04-23T10:31:21.455271+0000", "event": "throttled" }, { "time": "2024-04-23T10:31:21.455278+0000", "event": "all_read" }, { "time": "2024-04-23T10:31:21.456696+0000", "event": "dispatched" } ], "locks": [] } }, { "description": "internal op quiesce_inode:mds.0:188 fp=#0x100000014dd", "initiated_at": "2024-04-23T10:31:21.471294+0000", "age": 42.413095052999999, "duration": 42.413434293000002, "continuous": true, "type_data": { "result": -2147483648, "flag_point": "quiesce complete", "reqid": { "entity": { "type": "mds", "num": 0 }, "tid": 188 }, "op_type": "internal_op", "internal_op": 5385, "op_name": "quiesce_inode", "events": [ { "time": "2024-04-23T10:31:21.471294+0000", "event": "initiated" }, { "time": "2024-04-23T10:31:21.471294+0000", "event": "header_read" }, { "time": "2024-04-23T10:31:21.471294+0000", "event": "throttled" }, { "time": "2024-04-23T10:31:21.471294+0000", "event": "all_read" }, { "time": "2024-04-23T10:31:21.471294+0000", "event": "dispatched" }, { "time": "2024-04-23T10:31:21.472009+0000", "event": "failed to xlock, waiting" }, { "time": "2024-04-23T10:31:21.502362+0000", "event": "failed to xlock, waiting" }, { "time": "2024-04-23T10:31:21.505667+0000", "event": "acquired locks" }, { "time": "2024-04-23T10:31:21.505865+0000", "event": "quiesce complete" } ], "locks": [ { "object": { "is_auth": true, "auth_state": { "replicas": {} }, "replica_state": { "authority": [ 0, -2 ], "replica_nonce": 0 }, "auth_pins": 1, "is_frozen": false, "is_freezing": false, "pins": { "ptrwaiter": 0, "request": 1, "lock": 1, "caps": 0, "dirtyparent": 0, "dirty": 0, "waiter": 0, "authpin": 1 }, "nref": 3 }, "object_string": "[inode 0x100000014dd [7,head] /volumes/qa/sv_1/b206794e-7622-4ab8-8943-f3ae4e65b7a3/client.0/tmp/clients/client0/~dmtmp/ACCESS/SALES.PRN auth v3330 ap=1 s=10845 n(v0 rc2024-04-23T10:31:17.026733+0000 b10845 1=1+0) (iquiesce lock x=1 by request(mds.0:188 nref=3)) | ptrwaiter=0 request=1 lock=1 caps=0 dirtyparent=0 dirty=0 waiter=0 authpin=1 0x5571cf5cf180]", "lock": { "gather_set": [], "state": "lock", "type": "iquiesce", "is_leased": false, "num_rdlocks": 0, "num_wrlocks": 0, "num_xlocks": 1, "xlock_by": { "reqid": { "entity": { "type": "mds", "num": 0 }, "tid": 188 } } }, "flags": 4, "wrlock_target": -1 } ] } }, { "description": "peer_request:mds.2:138", "initiated_at": "2024-04-23T10:31:21.489509+0000", "age": 42.394880688000001, "duration": 42.395288555999997, "continuous": false, "type_data": { "result": -2147483648, "flag_point": "dispatched", "reqid": { "entity": { "type": "mds", "num": 2 }, "tid": 138 }, "op_type": "peer_request", "leader_info": { "leader": "2" }, "events": [ { "time": "2024-04-23T10:31:21.489509+0000", "event": "initiated" }, { "time": "2024-04-23T10:31:21.489509+0000", "event": "header_read" }, { "time": "2024-04-23T10:31:21.489509+0000", "event": "throttled" }, { "time": "2024-04-23T10:31:21.489511+0000", "event": "all_read" }, { "time": "2024-04-23T10:31:21.508324+0000", "event": "dispatched" } ], "locks": [] } }, { "description": "peer_request:mds.1:111", "initiated_at": "2024-04-23T10:31:21.455336+0000", "age": 42.429053641000003, "duration": 42.429483251999997, "continuous": false, "type_data": { "result": -2147483648, "flag_point": "dispatched", "reqid": { "entity": { "type": "mds", "num": 1 }, "tid": 111 }, "op_type": "peer_request", "leader_info": { "leader": "1" }, "events": [ { "time": "2024-04-23T10:31:21.455336+0000", "event": "initiated" }, { "time": "2024-04-23T10:31:21.455336+0000", "event": "header_read" }, { "time": "2024-04-23T10:31:21.455336+0000", "event": "throttled" }, { "time": "2024-04-23T10:31:21.455341+0000", "event": "all_read" }, { "time": "2024-04-23T10:31:21.456770+0000", "event": "dispatched" } ], "locks": [] } }, { "description": "internal op quiesce_inode:mds.0:189 fp=#0x100000014df", "initiated_at": "2024-04-23T10:31:21.471306+0000", "age": 42.413082793999997, "duration": 42.413536282000003, "continuous": true, "type_data": { "result": -2147483648, "flag_point": "quiesce complete", "reqid": { "entity": { "type": "mds", "num": 0 }, "tid": 189 }, "op_type": "internal_op", "internal_op": 5385, "op_name": "quiesce_inode", "events": [ { "time": "2024-04-23T10:31:21.471306+0000", "event": "initiated" }, { "time": "2024-04-23T10:31:21.471306+0000", "event": "header_read" }, { "time": "2024-04-23T10:31:21.471306+0000", "event": "throttled" }, { "time": "2024-04-23T10:31:21.471306+0000", "event": "all_read" }, { "time": "2024-04-23T10:31:21.471306+0000", "event": "dispatched" }, { "time": "2024-04-23T10:31:21.472207+0000", "event": "failed to xlock, waiting" }, { "time": "2024-04-23T10:31:21.505006+0000", "event": "failed to xlock, waiting" }, { "time": "2024-04-23T10:31:21.507040+0000", "event": "acquired locks" }, { "time": "2024-04-23T10:31:21.507152+0000", "event": "quiesce complete" } ], "locks": [ { "object": { "is_auth": true, "auth_state": { "replicas": {} }, "replica_state": { "authority": [ 0, -2 ], "replica_nonce": 0 }, "auth_pins": 1, "is_frozen": false, "is_freezing": false, "pins": { "ptrwaiter": 0, "request": 1, "lock": 1, "caps": 0, "dirtyparent": 0, "dirty": 0, "waiter": 0, "authpin": 1 }, "nref": 3 }, "object_string": "[inode 0x100000014df [7,head] /volumes/qa/sv_1/b206794e-7622-4ab8-8943-f3ae4e65b7a3/client.0/tmp/clients/client0/~dmtmp/ACCESS/LABELS.PRN auth v3328 ap=1 s=30118 n(v0 rc2024-04-23T10:31:17.038732+0000 b30118 1=1+0) (iquiesce lock x=1 by request(mds.0:189 nref=3)) | ptrwaiter=0 request=1 lock=1 caps=0 dirtyparent=0 dirty=0 waiter=0 authpin=1 0x5571cfaaf180]", "lock": { "gather_set": [], "state": "lock", "type": "iquiesce", "is_leased": false, "num_rdlocks": 0, "num_wrlocks": 0, "num_xlocks": 1, "xlock_by": { "reqid": { "entity": { "type": "mds", "num": 0 }, "tid": 189 } } }, "flags": 4, "wrlock_target": -1 } ] } }, { "description": "peer_request:mds.2:139", "initiated_at": "2024-04-23T10:31:21.489583+0000", "age": 42.394806473000003, "duration": 42.395343420000003, "continuous": false, "type_data": { "result": -2147483648, "flag_point": "dispatched", "reqid": { "entity": { "type": "mds", "num": 2 }, "tid": 139 }, "op_type": "peer_request", "leader_info": { "leader": "2" }, "events": [ { "time": "2024-04-23T10:31:21.489583+0000", "event": "initiated" }, { "time": "2024-04-23T10:31:21.489583+0000", "event": "header_read" }, { "time": "2024-04-23T10:31:21.489583+0000", "event": "throttled" }, { "time": "2024-04-23T10:31:21.489589+0000", "event": "all_read" }, { "time": "2024-04-23T10:31:21.508404+0000", "event": "dispatched" } ], "locks": [] } }, { "description": "internal op quiesce_inode:mds.0:160 fp=#0x10000000009", "initiated_at": "2024-04-23T10:31:21.457565+0000", "age": 42.426823845000001, "duration": 42.427383431999999, "continuous": true, "type_data": { "result": -2147483648, "flag_point": "quiesce complete", "reqid": { "entity": { "type": "mds", "num": 0 }, "tid": 160 }, "op_type": "internal_op", "internal_op": 5385, "op_name": "quiesce_inode", "events": [ { "time": "2024-04-23T10:31:21.457565+0000", "event": "initiated" }, { "time": "2024-04-23T10:31:21.457565+0000", "event": "header_read" }, { "time": "2024-04-23T10:31:21.457565+0000", "event": "throttled" }, { "time": "2024-04-23T10:31:21.457565+0000", "event": "all_read" }, { "time": "2024-04-23T10:31:21.457565+0000", "event": "dispatched" }, { "time": "2024-04-23T10:31:21.457783+0000", "event": "failed to xlock, waiting" }, { "time": "2024-04-23T10:31:21.459494+0000", "event": "acquired locks" }, { "time": "2024-04-23T10:31:21.509817+0000", "event": "quiesce complete" } ], "locks": [ { "object": { "is_auth": true, "auth_state": { "replicas": { "1": 1, "2": 1 } }, "replica_state": { "authority": [ 0, -2 ], "replica_nonce": 0 }, "auth_pins": 3, "is_frozen": false, "is_freezing": false, "pins": { "dirtyscattered": 0, "request": 1, "lock": 1, "dirfrag": 1, "caps": 1, "dirtyrstat": 0, "dirtyparent": 0, "replicated": 1, "dirty": 0, "waiter": 0, "authpin": 1 }, "nref": 6 }, "object_string": "[inode 0x10000000009 [...7,head] /volumes/qa/sv_1/b206794e-7622-4ab8-8943-f3ae4e65b7a3/client.0/tmp/clients/client0/~dmtmp/ auth{1=1,2=1} v15178 ap=3 f(v0 m2024-04-23T10:27:52.186392+0000 9=0+9) n(v31 rc2024-04-23T10:31:17.764716+0000 b15832194 68=58+10) old_inodes=4 (inest mix) (iquiesce lock x=1 by request(mds.0:160 nref=3)) caps={15312=pAsLsXsFs/pAsLsXsFsxcral@56},l=15312 | dirtyscattered=0 request=1 lock=1 dirfrag=1 caps=1 dirtyrstat=0 dirtyparent=0 replicated=1 dirty=0 waiter=0 authpin=1 0x5571cf341080]", "lock": { "gather_set": [], "state": "lock", "type": "iquiesce", "is_leased": false, "num_rdlocks": 0, "num_wrlocks": 0, "num_xlocks": 1, "xlock_by": { "reqid": { "entity": { "type": "mds", "num": 0 }, "tid": 160 } } }, "flags": 4, "wrlock_target": -1 } ] } }, { "description": "peer_request:mds.2:117", "initiated_at": "2024-04-23T10:31:21.463330+0000", "age": 42.421059130000003, "duration": 42.421691584999998, "continuous": false, "type_data": { "result": -2147483648, "flag_point": "dispatched", "reqid": { "entity": { "type": "mds", "num": 2 }, "tid": 117 }, "op_type": "peer_request", "leader_info": { "leader": "2" }, "events": [ { "time": "2024-04-23T10:31:21.463330+0000", "event": "initiated" }, { "time": "2024-04-23T10:31:21.463330+0000", "event": "header_read" }, { "time": "2024-04-23T10:31:21.463331+0000", "event": "throttled" }, { "time": "2024-04-23T10:31:21.463338+0000", "event": "all_read" }, { "time": "2024-04-23T10:31:21.477419+0000", "event": "dispatched" } ], "locks": [] } }, { "description": "peer_request:mds.1:100", "initiated_at": "2024-04-23T10:31:22.426164+0000", "age": 41.458225577, "duration": 41.458880985999997, "continuous": false, "type_data": { "result": -2147483648, "flag_point": "dispatched", "reqid": { "entity": { "type": "mds", "num": 1 }, "tid": 100 }, "op_type": "peer_request", "leader_info": { "leader": "1" }, "events": [ { "time": "2024-04-23T10:31:22.426164+0000", "event": "initiated" }, { "time": "2024-04-23T10:31:22.426164+0000", "event": "header_read" }, { "time": "2024-04-23T10:31:22.426165+0000", "event": "throttled" }, { "time": "2024-04-23T10:31:22.426175+0000", "event": "all_read" }, { "time": "2024-04-23T10:31:22.426205+0000", "event": "dispatched" } ], "locks": [] } }, { "description": "internal op quiesce_inode:mds.0:161 fp=#0x10000000204", "initiated_at": "2024-04-23T10:31:21.457577+0000", "age": 42.426811915999998, "duration": 42.427489895999997, "continuous": true, "type_data": { "result": -2147483648, "flag_point": "quiesce complete", "reqid": { "entity": { "type": "mds", "num": 0 }, "tid": 161 }, "op_type": "internal_op", "internal_op": 5385, "op_name": "quiesce_inode", "events": [ { "time": "2024-04-23T10:31:21.457577+0000", "event": "initiated" }, { "time": "2024-04-23T10:31:21.457577+0000", "event": "header_read" }, { "time": "2024-04-23T10:31:21.457577+0000", "event": "throttled" }, { "time": "2024-04-23T10:31:21.457577+0000", "event": "all_read" }, { "time": "2024-04-23T10:31:21.457577+0000", "event": "dispatched" }, { "time": "2024-04-23T10:31:21.457900+0000", "event": "acquired locks" }, { "time": "2024-04-23T10:31:21.457989+0000", "event": "quiesce complete" } ], "locks": [ { "object": { "is_auth": true, "auth_state": { "replicas": {} }, "replica_state": { "authority": [ 0, -2 ], "replica_nonce": 0 }, "auth_pins": 1, "is_frozen": false, "is_freezing": false, "pins": { "ptrwaiter": 0, "request": 1, "lock": 1, "caps": 0, "dirtyparent": 0, "dirty": 0, "authpin": 1 }, "nref": 3 }, "object_string": "[inode 0x10000000204 [4,head] /volumes/qa/sv_1/b206794e-7622-4ab8-8943-f3ae4e65b7a3/client.0/tmp/clients/client0/filler.001 auth v483 ap=1 s=1048576 n(v0 rc2024-04-23T10:27:51.936398+0000 b1048576 1=1+0) (iquiesce lock x=1 by request(mds.0:161 nref=3)) | ptrwaiter=0 request=1 lock=1 caps=0 dirtyparent=0 dirty=0 authpin=1 0x5571cf341b80]", "lock": { "gather_set": [], "state": "lock", "type": "iquiesce", "is_leased": false, "num_rdlocks": 0, "num_wrlocks": 0, "num_xlocks": 1, "xlock_by": { "reqid": { "entity": { "type": "mds", "num": 0 }, "tid": 161 } } }, "flags": 4, "wrlock_target": -1 } ] } }, { "description": "peer_request:mds.1:113", "initiated_at": "2024-04-23T10:31:21.464643+0000", "age": 42.419745823, "duration": 42.420481959999996, "continuous": false, "type_data": { "result": -2147483648, "flag_point": "dispatched", "reqid": { "entity": { "type": "mds", "num": 1 }, "tid": 113 }, "op_type": "peer_request", "leader_info": { "leader": "1" }, "events": [ { "time": "2024-04-23T10:31:21.464643+0000", "event": "initiated" }, { "time": "2024-04-23T10:31:21.464643+0000", "event": "header_read" }, { "time": "2024-04-23T10:31:21.464644+0000", "event": "throttled" }, { "time": "2024-04-23T10:31:21.464652+0000", "event": "all_read" }, { "time": "2024-04-23T10:31:21.478154+0000", "event": "dispatched" } ], "locks": [] } }, { "description": "internal op quiesce_path:mds.0:149 fp=#0x1", "initiated_at": "2024-04-23T10:31:21.445093+0000", "age": 42.439296069999997, "duration": 42.440054617999998, "continuous": true, "type_data": { "result": 0, "flag_point": "quiesce complete", "reqid": { "entity": { "type": "mds", "num": 0 }, "tid": 149 }, "op_type": "internal_op", "internal_op": 5384, "op_name": "quiesce_path", "events": [ { "time": "2024-04-23T10:31:21.445093+0000", "event": "initiated" }, { "time": "2024-04-23T10:31:21.445093+0000", "event": "header_read" }, { "time": "2024-04-23T10:31:21.445093+0000", "event": "throttled" }, { "time": "2024-04-23T10:31:21.445093+0000", "event": "all_read" }, { "time": "2024-04-23T10:31:21.445093+0000", "event": "dispatched" }, { "time": "2024-04-23T10:31:21.510121+0000", "event": "quiesce complete" } ], "locks": [] } }, { "description": "internal op quiesce_inode:mds.0:162 fp=#0x10000000205", "initiated_at": "2024-04-23T10:31:21.457585+0000", "age": 42.426803874000001, "duration": 42.427589140999999, "continuous": true, "type_data": { "result": -2147483648, "flag_point": "quiesce complete", "reqid": { "entity": { "type": "mds", "num": 0 }, "tid": 162 }, "op_type": "internal_op", "internal_op": 5385, "op_name": "quiesce_inode", "events": [ { "time": "2024-04-23T10:31:21.457585+0000", "event": "initiated" }, { "time": "2024-04-23T10:31:21.457585+0000", "event": "header_read" }, { "time": "2024-04-23T10:31:21.457585+0000", "event": "throttled" }, { "time": "2024-04-23T10:31:21.457585+0000", "event": "all_read" }, { "time": "2024-04-23T10:31:21.457585+0000", "event": "dispatched" }, { "time": "2024-04-23T10:31:21.458102+0000", "event": "acquired locks" }, { "time": "2024-04-23T10:31:21.458185+0000", "event": "quiesce complete" } ], "locks": [ { "object": { "is_auth": true, "auth_state": { "replicas": {} }, "replica_state": { "authority": [ 0, -2 ], "replica_nonce": 0 }, "auth_pins": 1, "is_frozen": false, "is_freezing": false, "pins": { "ptrwaiter": 0, "request": 1, "lock": 1, "caps": 0, "dirtyparent": 0, "dirty": 0, "authpin": 1 }, "nref": 3 }, "object_string": "[inode 0x10000000205 [4,head] /volumes/qa/sv_1/b206794e-7622-4ab8-8943-f3ae4e65b7a3/client.0/tmp/clients/client0/filler.002 auth v479 ap=1 s=1048576 n(v0 rc2024-04-23T10:27:51.942398+0000 b1048576 1=1+0) (iquiesce lock x=1 by request(mds.0:162 nref=3)) | ptrwaiter=0 request=1 lock=1 caps=0 dirtyparent=0 dirty=0 authpin=1 0x5571cf342100]", "lock": { "gather_set": [], "state": "lock", "type": "iquiesce", "is_leased": false, "num_rdlocks": 0, "num_wrlocks": 0, "num_xlocks": 1, "xlock_by": { "reqid": { "entity": { "type": "mds", "num": 0 }, "tid": 162 } } }, "flags": 4, "wrlock_target": -1 } ] } }, { "description": "peer_request:mds.2:118", "initiated_at": "2024-04-23T10:31:21.463407+0000", "age": 42.420982723000002, "duration": 42.421825890000001, "continuous": false, "type_data": { "result": -2147483648, "flag_point": "dispatched", "reqid": { "entity": { "type": "mds", "num": 2 }, "tid": 118 }, "op_type": "peer_request", "leader_info": { "leader": "2" }, "events": [ { "time": "2024-04-23T10:31:21.463407+0000", "event": "initiated" }, { "time": "2024-04-23T10:31:21.463407+0000", "event": "header_read" }, { "time": "2024-04-23T10:31:21.463407+0000", "event": "throttled" }, { "time": "2024-04-23T10:31:21.463415+0000", "event": "all_read" }, { "time": "2024-04-23T10:31:21.478325+0000", "event": "dispatched" } ], "locks": [] } }, { "description": "internal op quiesce_inode:mds.0:150 fp=#0x1", "initiated_at": "2024-04-23T10:31:21.445156+0000", "age": 42.439233471000001, "duration": 42.440098505000002, "continuous": true, "type_data": { "result": -2147483648, "flag_point": "quiesce complete", "reqid": { "entity": { "type": "mds", "num": 0 }, "tid": 150 }, "op_type": "internal_op", "internal_op": 5385, "op_name": "quiesce_inode", "events": [ { "time": "2024-04-23T10:31:21.445156+0000", "event": "initiated" }, { "time": "2024-04-23T10:31:21.445156+0000", "event": "header_read" }, { "time": "2024-04-23T10:31:21.445156+0000", "event": "throttled" }, { "time": "2024-04-23T10:31:21.445156+0000", "event": "all_read" }, { "time": "2024-04-23T10:31:21.445156+0000", "event": "dispatched" }, { "time": "2024-04-23T10:31:21.445579+0000", "event": "failed to xlock, waiting" }, { "time": "2024-04-23T10:31:21.446412+0000", "event": "acquired locks" }, { "time": "2024-04-23T10:31:21.510071+0000", "event": "quiesce complete" } ], "locks": [ { "object": { "is_auth": true, "auth_state": { "replicas": { "1": 1, "2": 1 } }, "replica_state": { "authority": [ 0, -2 ], "replica_nonce": 0 }, "auth_pins": 3, "is_frozen": false, "is_freezing": false, "pins": { "dirtyscattered": 1, "request": 1, "lock": 1, "dirfrag": 1, "caps": 1, "dirtyparent": 0, "replicated": 1, "dirty": 1, "waiter": 0, "authpin": 1 }, "nref": 8 }, "object_string": "[inode 0x1 [...7,head] / auth{1=1,2=1} v187 ap=3 snaprealm=0x5571ce2bf8c0 f(v0 m2024-04-23T10:24:48.676117+0000 1=0+1) n(v27 rc2024-04-23T10:32:00.511998+0000 b21075308 rs6 88=67+21)/n(v0 rc2024-04-23T10:24:44.319412+0000 1=0+1) old_inodes=5 (inest mix dirty) (iquiesce lock x=1 by request(mds.0:150 nref=3)) caps={15312=pAsLsXs/-@2,15576=pAsLsXsFs/-@31} | dirtyscattered=1 request=1 lock=1 dirfrag=1 caps=1 dirtyparent=0 replicated=1 dirty=1 waiter=0 authpin=1 0x5571cef23b80]", "lock": { "gather_set": [], "state": "lock", "type": "iquiesce", "is_leased": false, "num_rdlocks": 0, "num_wrlocks": 0, "num_xlocks": 1, "xlock_by": { "reqid": { "entity": { "type": "mds", "num": 0 }, "tid": 150 } } }, "flags": 4, "wrlock_target": -1 } ] } }, { "description": "internal op quiesce_inode:mds.0:163 fp=#0x10000000207", "initiated_at": "2024-04-23T10:31:21.457592+0000", "age": 42.426797319999999, "duration": 42.427741163999997, "continuous": true, "type_data": { "result": -2147483648, "flag_point": "quiesce complete", "reqid": { "entity": { "type": "mds", "num": 0 }, "tid": 163 }, "op_type": "internal_op", "internal_op": 5385, "op_name": "quiesce_inode", "events": [ { "time": "2024-04-23T10:31:21.457592+0000", "event": "initiated" }, { "time": "2024-04-23T10:31:21.457592+0000", "event": "header_read" }, { "time": "2024-04-23T10:31:21.457592+0000", "event": "throttled" }, { "time": "2024-04-23T10:31:21.457592+0000", "event": "all_read" }, { "time": "2024-04-23T10:31:21.457592+0000", "event": "dispatched" }, { "time": "2024-04-23T10:31:21.458283+0000", "event": "acquired locks" }, { "time": "2024-04-23T10:31:21.458355+0000", "event": "quiesce complete" } ], "locks": [ { "object": { "is_auth": true, "auth_state": { "replicas": {} }, "replica_state": { "authority": [ 0, -2 ], "replica_nonce": 0 }, "auth_pins": 1, "is_frozen": false, "is_freezing": false, "pins": { "ptrwaiter": 0, "request": 1, "lock": 1, "caps": 0, "dirtyparent": 0, "dirty": 0, "waiter": 0, "authpin": 1 }, "nref": 3 }, "object_string": "[inode 0x10000000207 [4,head] /volumes/qa/sv_1/b206794e-7622-4ab8-8943-f3ae4e65b7a3/client.0/tmp/clients/client0/filler.004 auth v487 ap=1 s=1048576 n(v0 rc2024-04-23T10:27:51.952398+0000 b1048576 1=1+0) (iquiesce lock x=1 by request(mds.0:163 nref=3)) | ptrwaiter=0 request=1 lock=1 caps=0 dirtyparent=0 dirty=0 waiter=0 authpin=1 0x5571cf342c00]", "lock": { "gather_set": [], "state": "lock", "type": "iquiesce", "is_leased": false, "num_rdlocks": 0, "num_wrlocks": 0, "num_xlocks": 1, "xlock_by": { "reqid": { "entity": { "type": "mds", "num": 0 }, "tid": 163 } } }, "flags": 4, "wrlock_target": -1 } ] } }, { "description": "peer_request:mds.1:114", "initiated_at": "2024-04-23T10:31:21.464702+0000", "age": 42.419687234000001, "duration": 42.420722597000001, "continuous": false, "type_data": { "result": -2147483648, "flag_point": "dispatched", "reqid": { "entity": { "type": "mds", "num": 1 }, "tid": 114 }, "op_type": "peer_request", "leader_info": { "leader": "1" }, "events": [ { "time": "2024-04-23T10:31:21.464702+0000", "event": "initiated" }, { "time": "2024-04-23T10:31:21.464702+0000", "event": "header_read" }, { "time": "2024-04-23T10:31:21.464702+0000", "event": "throttled" }, { "time": "2024-04-23T10:31:21.464708+0000", "event": "all_read" }, { "time": "2024-04-23T10:31:21.479803+0000", "event": "dispatched" } ], "locks": [] } }, { "description": "peer_request:mds.1:102", "initiated_at": "2024-04-23T10:31:21.445466+0000", "age": 42.438923181, "duration": 42.439989287000003, "continuous": false, "type_data": { "result": -2147483648, "flag_point": "dispatched", "reqid": { "entity": { "type": "mds", "num": 1 }, "tid": 102 }, "op_type": "peer_request", "leader_info": { "leader": "1" }, "events": [ { "time": "2024-04-23T10:31:21.445466+0000", "event": "initiated" }, { "time": "2024-04-23T10:31:21.445466+0000", "event": "header_read" }, { "time": "2024-04-23T10:31:21.445467+0000", "event": "throttled" }, { "time": "2024-04-23T10:31:21.445474+0000", "event": "all_read" }, { "time": "2024-04-23T10:31:21.445755+0000", "event": "dispatched" } ], "locks": [] } }, { "description": "internal op quiesce_inode:mds.0:164 fp=#0x10000000206", "initiated_at": "2024-04-23T10:31:21.457599+0000", "age": 42.426790124999997, "duration": 42.42787938, "continuous": true, "type_data": { "result": -2147483648, "flag_point": "quiesce complete", "reqid": { "entity": { "type": "mds", "num": 0 }, "tid": 164 }, "op_type": "internal_op", "internal_op": 5385, "op_name": "quiesce_inode", "events": [ { "time": "2024-04-23T10:31:21.457599+0000", "event": "initiated" }, { "time": "2024-04-23T10:31:21.457599+0000", "event": "header_read" }, { "time": "2024-04-23T10:31:21.457599+0000", "event": "throttled" }, { "time": "2024-04-23T10:31:21.457599+0000", "event": "all_read" }, { "time": "2024-04-23T10:31:21.457599+0000", "event": "dispatched" }, { "time": "2024-04-23T10:31:21.458477+0000", "event": "acquired locks" }, { "time": "2024-04-23T10:31:21.458559+0000", "event": "quiesce complete" } ], "locks": [ { "object": { "is_auth": true, "auth_state": { "replicas": {} }, "replica_state": { "authority": [ 0, -2 ], "replica_nonce": 0 }, "auth_pins": 1, "is_frozen": false, "is_freezing": false, "pins": { "ptrwaiter": 0, "request": 1, "lock": 1, "caps": 0, "dirtyparent": 0, "dirty": 0, "authpin": 1 }, "nref": 3 }, "object_string": "[inode 0x10000000206 [4,head] /volumes/qa/sv_1/b206794e-7622-4ab8-8943-f3ae4e65b7a3/client.0/tmp/clients/client0/filler.003 auth v469 ap=1 s=1048576 n(v0 rc2024-04-23T10:27:51.947398+0000 b1048576 1=1+0) (iquiesce lock x=1 by request(mds.0:164 nref=3)) | ptrwaiter=0 request=1 lock=1 caps=0 dirtyparent=0 dirty=0 authpin=1 0x5571cf342680]", "lock": { "gather_set": [], "state": "lock", "type": "iquiesce", "is_leased": false, "num_rdlocks": 0, "num_wrlocks": 0, "num_xlocks": 1, "xlock_by": { "reqid": { "entity": { "type": "mds", "num": 0 }, "tid": 164 } } }, "flags": 4, "wrlock_target": -1 } ] } }, { "description": "peer_request:mds.2:119", "initiated_at": "2024-04-23T10:31:21.463515+0000", "age": 42.420874140999999, "duration": 42.422023795000001, "continuous": false, "type_data": { "result": -2147483648, "flag_point": "dispatched", "reqid": { "entity": { "type": "mds", "num": 2 }, "tid": 119 }, "op_type": "peer_request", "leader_info": { "leader": "2" }, "events": [ { "time": "2024-04-23T10:31:21.463515+0000", "event": "initiated" }, { "time": "2024-04-23T10:31:21.463515+0000", "event": "header_read" }, { "time": "2024-04-23T10:31:21.463516+0000", "event": "throttled" }, { "time": "2024-04-23T10:31:21.463523+0000", "event": "all_read" }, { "time": "2024-04-23T10:31:21.480061+0000", "event": "dispatched" } ], "locks": [] } }, { "description": "peer_request:mds.2:107", "initiated_at": "2024-04-23T10:31:21.445440+0000", "age": 42.438949735999998, "duration": 42.440122483000003, "continuous": false, "type_data": { "result": -2147483648, "flag_point": "dispatched", "reqid": { "entity": { "type": "mds", "num": 2 }, "tid": 107 }, "op_type": "peer_request", "leader_info": { "leader": "2" }, "events": [ { "time": "2024-04-23T10:31:21.445440+0000", "event": "initiated" }, { "time": "2024-04-23T10:31:21.445440+0000", "event": "header_read" }, { "time": "2024-04-23T10:31:21.445440+0000", "event": "throttled" }, { "time": "2024-04-23T10:31:21.445459+0000", "event": "all_read" }, { "time": "2024-04-23T10:31:21.445874+0000", "event": "dispatched" } ], "locks": [] } }, { "description": "internal op quiesce_inode:mds.0:165 fp=#0x1000000000a", "initiated_at": "2024-04-23T10:31:21.457606+0000", "age": 42.426783227999998, "duration": 42.427978015000001, "continuous": true, "type_data": { "result": -2147483648, "flag_point": "quiesce complete", "reqid": { "entity": { "type": "mds", "num": 0 }, "tid": 165 }, "op_type": "internal_op", "internal_op": 5385, "op_name": "quiesce_inode", "events": [ { "time": "2024-04-23T10:31:21.457606+0000", "event": "initiated" }, { "time": "2024-04-23T10:31:21.457606+0000", "event": "header_read" }, { "time": "2024-04-23T10:31:21.457606+0000", "event": "throttled" }, { "time": "2024-04-23T10:31:21.457606+0000", "event": "all_read" }, { "time": "2024-04-23T10:31:21.457606+0000", "event": "dispatched" }, { "time": "2024-04-23T10:31:21.458673+0000", "event": "acquired locks" }, { "time": "2024-04-23T10:31:21.458757+0000", "event": "quiesce complete" } ], "locks": [ { "object": { "is_auth": true, "auth_state": { "replicas": {} }, "replica_state": { "authority": [ 0, -2 ], "replica_nonce": 0 }, "auth_pins": 1, "is_frozen": false, "is_freezing": false, "pins": { "ptrwaiter": 0, "request": 1, "lock": 1, "caps": 0, "dirtyparent": 0, "dirty": 0, "authpin": 1 }, "nref": 3 }, "object_string": "[inode 0x1000000000a [4,head] /volumes/qa/sv_1/b206794e-7622-4ab8-8943-f3ae4e65b7a3/client.0/tmp/clients/client0/filler.000 auth v473 ap=1 s=1048576 n(v0 rc2024-04-23T10:27:51.931398+0000 b1048576 1=1+0) (iquiesce lock x=1 by request(mds.0:165 nref=3)) | ptrwaiter=0 request=1 lock=1 caps=0 dirtyparent=0 dirty=0 authpin=1 0x5571cf341600]", "lock": { "gather_set": [], "state": "lock", "type": "iquiesce", "is_leased": false, "num_rdlocks": 0, "num_wrlocks": 0, "num_xlocks": 1, "xlock_by": { "reqid": { "entity": { "type": "mds", "num": 0 }, "tid": 165 } } }, "flags": 4, "wrlock_target": -1 } ] } }, { "description": "internal op quiesce_inode:mds.0:190 fp=#0x1000000021f", "initiated_at": "2024-04-23T10:31:21.483372+0000", "age": 42.401016972999997, "duration": 42.402270540000004, "continuous": true, "type_data": { "result": -2147483648, "flag_point": "quiesce complete for non-auth inode", "reqid": { "entity": { "type": "mds", "num": 0 }, "tid": 190 }, "op_type": "internal_op", "internal_op": 5385, "op_name": "quiesce_inode", "events": [ { "time": "2024-04-23T10:31:21.483372+0000", "event": "initiated" }, { "time": "2024-04-23T10:31:21.483372+0000", "event": "header_read" }, { "time": "2024-04-23T10:31:21.483372+0000", "event": "throttled" }, { "time": "2024-04-23T10:31:21.483372+0000", "event": "all_read" }, { "time": "2024-04-23T10:31:21.483372+0000", "event": "dispatched" }, { "time": "2024-04-23T10:31:21.483581+0000", "event": "requesting remote authpins" }, { "time": "2024-04-23T10:31:21.506357+0000", "event": "acquired locks" }, { "time": "2024-04-23T10:31:21.506366+0000", "event": "quiesce complete for non-auth inode" } ], "locks": [ { "object": { "is_auth": false, "auth_state": { "replicas": {} }, "replica_state": { "authority": [ 2, -2 ], "replica_nonce": 1 }, "auth_pins": 0, "is_frozen": false, "is_freezing": false, "pins": { "request": 1, "lock": 1, "caps": 0, "exportingcaps": 0, "dirtyparent": 0, "dirty": 0, "authpin": 0, "tempexporting": 0 }, "nref": 2 }, "object_string": "[inode 0x1000000021f [7,head] /volumes/qa/sv_1/b206794e-7622-4ab8-8943-f3ae4e65b7a3/client.0/tmp/clients/client0/~dmtmp/COREL/ROSE.CPT rep@2.1 v7256 s=260096 n(v0 rc2024-04-23T10:27:52.049396+0000 b260096 1=1+0)/n(v0 1=1+0) (iquiesce lock x=1 by request(mds.0:190 nref=3)) | request=1 lock=1 caps=0 exportingcaps=0 dirtyparent=0 dirty=0 authpin=0 tempexporting=0 0x5571cf563180]", "lock": { "gather_set": [], "state": "lock", "type": "iquiesce", "is_leased": false, "num_rdlocks": 0, "num_wrlocks": 0, "num_xlocks": 1, "xlock_by": { "reqid": { "entity": { "type": "mds", "num": 0 }, "tid": 190 } } }, "flags": 4, "wrlock_target": -1 } ] } }, { "description": "internal op quiesce_inode:mds.0:151 fp=#0x10000000000", "initiated_at": "2024-04-23T10:31:21.446733+0000", "age": 42.437656287999999, "duration": 42.438973836999999, "continuous": true, "type_data": { "result": -2147483648, "flag_point": "quiesce complete", "reqid": { "entity": { "type": "mds", "num": 0 }, "tid": 151 }, "op_type": "internal_op", "internal_op": 5385, "op_name": "quiesce_inode", "events": [ { "time": "2024-04-23T10:31:21.446733+0000", "event": "initiated" }, { "time": "2024-04-23T10:31:21.446733+0000", "event": "header_read" }, { "time": "2024-04-23T10:31:21.446733+0000", "event": "throttled" }, { "time": "2024-04-23T10:31:21.446733+0000", "event": "all_read" }, { "time": "2024-04-23T10:31:21.446733+0000", "event": "dispatched" }, { "time": "2024-04-23T10:31:21.446918+0000", "event": "failed to xlock, waiting" }, { "time": "2024-04-23T10:31:21.447563+0000", "event": "acquired locks" }, { "time": "2024-04-23T10:31:21.510037+0000", "event": "quiesce complete" } ], "locks": [ { "object": { "is_auth": true, "auth_state": { "replicas": { "1": 1, "2": 1 } }, "replica_state": { "authority": [ 0, -2 ], "replica_nonce": 0 }, "auth_pins": 3, "is_frozen": false, "is_freezing": false, "pins": { "dirtyscattered": 1, "request": 1, "lock": 1, "dirfrag": 1, "caps": 0, "dirtyrstat": 0, "dirtyparent": 0, "replicated": 1, "dirty": 1, "waiter": 0, "authpin": 1 }, "nref": 7 }, "object_string": "[inode 0x10000000000 [...7,head] /volumes/ auth{1=1,2=1} v406 ap=3 f(v0 m2024-04-23T10:24:49.736808+0000 3=2+1) n(v29 rc2024-04-23T10:31:17.764716+0000 b21075308 87=67+20) old_inodes=5 (inest mix dirty) (iquiesce lock x=1 by request(mds.0:151 nref=3)) | dirtyscattered=1 request=1 lock=1 dirfrag=1 caps=0 dirtyrstat=0 dirtyparent=0 replicated=1 dirty=1 waiter=0 authpin=1 0x5571cf18f080]", "lock": { "gather_set": [], "state": "lock", "type": "iquiesce", "is_leased": false, "num_rdlocks": 0, "num_wrlocks": 0, "num_xlocks": 1, "xlock_by": { "reqid": { "entity": { "type": "mds", "num": 0 }, "tid": 151 } } }, "flags": 4, "wrlock_target": -1 } ] } }, { "description": "internal op quiesce_inode:mds.0:166 fp=#0x10000000215", "initiated_at": "2024-04-23T10:31:21.459820+0000", "age": 42.424569529000003, "duration": 42.425971025000003, "continuous": true, "type_data": { "result": -2147483648, "flag_point": "quiesce complete", "reqid": { "entity": { "type": "mds", "num": 0 }, "tid": 166 }, "op_type": "internal_op", "internal_op": 5385, "op_name": "quiesce_inode", "events": [ { "time": "2024-04-23T10:31:21.459820+0000", "event": "initiated" }, { "time": "2024-04-23T10:31:21.459820+0000", "event": "header_read" }, { "time": "2024-04-23T10:31:21.459820+0000", "event": "throttled" }, { "time": "2024-04-23T10:31:21.459820+0000", "event": "all_read" }, { "time": "2024-04-23T10:31:21.459820+0000", "event": "dispatched" }, { "time": "2024-04-23T10:31:21.460243+0000", "event": "failed to xlock, waiting" }, { "time": "2024-04-23T10:31:21.482957+0000", "event": "acquired locks" }, { "time": "2024-04-23T10:31:21.508224+0000", "event": "quiesce complete" } ], "locks": [ { "object": { "is_auth": true, "auth_state": { "replicas": { "1": 1, "2": 1 } }, "replica_state": { "authority": [ 0, -2 ], "replica_nonce": 0 }, "auth_pins": 3, "is_frozen": false, "is_freezing": false, "pins": { "dirtyscattered": 0, "request": 1, "lock": 1, "dirfrag": 1, "caps": 1, "dirtyrstat": 0, "dirtyparent": 0, "replicated": 1, "dirty": 0, "waiter": 0, "authpin": 1 }, "nref": 6 }, "object_string": "[inode 0x10000000215 [...7,head] /volumes/qa/sv_1/b206794e-7622-4ab8-8943-f3ae4e65b7a3/client.0/tmp/clients/client0/~dmtmp/COREL/ auth{1=1,2=1} v45589 ap=3 f(v1232 m2024-04-23T10:31:17.745716+0000 11=11+0) n(v19 rc2024-04-23T10:31:17.745716+0000 b2861056 12=11+1) old_inodes=4 (inest mix) (iquiesce lock x=1 by request(mds.0:166 nref=3)) caps={15312=pAsLsXsFs/pAsLsXsFsxcral@3233} mcw={2=AsLsXsFs} | dirtyscattered=0 request=1 lock=1 dirfrag=1 caps=1 dirtyrstat=0 dirtyparent=0 replicated=1 dirty=0 waiter=0 authpin=1 0x5571cf541700]", "lock": { "gather_set": [], "state": "lock", "type": "iquiesce", "is_leased": false, "num_rdlocks": 0, "num_wrlocks": 0, "num_xlocks": 1, "xlock_by": { "reqid": { "entity": { "type": "mds", "num": 0 }, "tid": 166 } } }, "flags": 4, "wrlock_target": -1 } ] } }, { "description": "internal op quiesce_inode:mds.0:191 fp=#0x1000000021b", "initiated_at": "2024-04-23T10:31:21.483386+0000", "age": 42.401002789000003, "duration": 42.402473235999999, "continuous": true, "type_data": { "result": -2147483648, "flag_point": "quiesce complete for non-auth inode", "reqid": { "entity": { "type": "mds", "num": 0 }, "tid": 191 }, "op_type": "internal_op", "internal_op": 5385, "op_name": "quiesce_inode", "events": [ { "time": "2024-04-23T10:31:21.483386+0000", "event": "initiated" }, { "time": "2024-04-23T10:31:21.483386+0000", "event": "header_read" }, { "time": "2024-04-23T10:31:21.483386+0000", "event": "throttled" }, { "time": "2024-04-23T10:31:21.483386+0000", "event": "all_read" }, { "time": "2024-04-23T10:31:21.483386+0000", "event": "dispatched" }, { "time": "2024-04-23T10:31:21.483634+0000", "event": "requesting remote authpins" }, { "time": "2024-04-23T10:31:21.506519+0000", "event": "acquired locks" }, { "time": "2024-04-23T10:31:21.506527+0000", "event": "quiesce complete for non-auth inode" } ], "locks": [ { "object": { "is_auth": false, "auth_state": { "replicas": {} }, "replica_state": { "authority": [ 2, -2 ], "replica_nonce": 1 }, "auth_pins": 0, "is_frozen": false, "is_freezing": false, "pins": { "request": 1, "lock": 1, "caps": 0, "exportingcaps": 0, "dirtyparent": 0, "dirty": 0, "authpin": 0, "tempexporting": 0 }, "nref": 2 }, "object_string": "[inode 0x1000000021b [7,head] /volumes/qa/sv_1/b206794e-7622-4ab8-8943-f3ae4e65b7a3/client.0/tmp/clients/client0/~dmtmp/COREL/TEAPOT2.CPT rep@2.1 v7248 s=260096 n(v0 rc2024-04-23T10:27:52.028396+0000 b260096 1=1+0)/n(v0 1=1+0) (iquiesce lock x=1 by request(mds.0:191 nref=3)) | request=1 lock=1 caps=0 exportingcaps=0 dirtyparent=0 dirty=0 authpin=0 tempexporting=0 0x5571cf561b80]", "lock": { "gather_set": [], "state": "lock", "type": "iquiesce", "is_leased": false, "num_rdlocks": 0, "num_wrlocks": 0, "num_xlocks": 1, "xlock_by": { "reqid": { "entity": { "type": "mds", "num": 0 }, "tid": 191 } } }, "flags": 4, "wrlock_target": -1 } ] } }, { "description": "peer_request:mds.1:103", "initiated_at": "2024-04-23T10:31:21.446187+0000", "age": 42.438202668000002, "duration": 42.439732186999997, "continuous": false, "type_data": { "result": -2147483648, "flag_point": "dispatched", "reqid": { "entity": { "type": "mds", "num": 1 }, "tid": 103 }, "op_type": "peer_request", "leader_info": { "leader": "1" }, "events": [ { "time": "2024-04-23T10:31:21.446187+0000", "event": "initiated" }, { "time": "2024-04-23T10:31:21.446187+0000", "event": "header_read" }, { "time": "2024-04-23T10:31:21.446187+0000", "event": "throttled" }, { "time": "2024-04-23T10:31:21.446194+0000", "event": "all_read" }, { "time": "2024-04-23T10:31:21.447001+0000", "event": "dispatched" } ], "locks": [] } }, { "description": "internal op quiesce_inode:mds.0:167 fp=#0x1000000020c", "initiated_at": "2024-04-23T10:31:21.459843+0000", "age": 42.424546264, "duration": 42.426096846, "continuous": true, "type_data": { "result": -2147483648, "flag_point": "quiesce complete", "reqid": { "entity": { "type": "mds", "num": 0 }, "tid": 167 }, "op_type": "internal_op", "internal_op": 5385, "op_name": "quiesce_inode", "events": [ { "time": "2024-04-23T10:31:21.459843+0000", "event": "initiated" }, { "time": "2024-04-23T10:31:21.459843+0000", "event": "header_read" }, { "time": "2024-04-23T10:31:21.459843+0000", "event": "throttled" }, { "time": "2024-04-23T10:31:21.459843+0000", "event": "all_read" }, { "time": "2024-04-23T10:31:21.459843+0000", "event": "dispatched" }, { "time": "2024-04-23T10:31:21.460564+0000", "event": "failed to xlock, waiting" }, { "time": "2024-04-23T10:31:21.463248+0000", "event": "acquired locks" }, { "time": "2024-04-23T10:31:21.488473+0000", "event": "quiesce complete" } ], "locks": [ { "object": { "is_auth": true, "auth_state": { "replicas": { "1": 1, "2": 1 } }, "replica_state": { "authority": [ 0, -2 ], "replica_nonce": 0 }, "auth_pins": 3, "is_frozen": false, "is_freezing": false, "pins": { "request": 1, "lock": 1, "dirfrag": 1, "caps": 1, "dirtyrstat": 0, "dirtyparent": 0, "replicated": 1, "dirty": 0, "waiter": 0, "authpin": 1 }, "nref": 6 }, "object_string": "[inode 0x1000000020c [...7,head] /volumes/qa/sv_1/b206794e-7622-4ab8-8943-f3ae4e65b7a3/client.0/tmp/clients/client0/~dmtmp/EXCEL/ auth{1=1,2=1} v45586 ap=3 f(v0 m2024-04-23T10:31:17.692717+0000 2=2+0) n(v0 rc2024-04-23T10:31:17.692717+0000 b520192 3=2+1) old_inodes=4 (inest mix) (iquiesce lock x=1 by request(mds.0:167 nref=3)) caps={15312=pAsLsXsFs/pAsLsXsFsxcral@9632},l=15312 | request=1 lock=1 dirfrag=1 caps=1 dirtyrstat=0 dirtyparent=0 replicated=1 dirty=0 waiter=0 authpin=1 0x5571cf53e580]", "lock": { "gather_set": [], "state": "lock", "type": "iquiesce", "is_leased": false, "num_rdlocks": 0, "num_wrlocks": 0, "num_xlocks": 1, "xlock_by": { "reqid": { "entity": { "type": "mds", "num": 0 }, "tid": 167 } } }, "flags": 4, "wrlock_target": -1 } ] } }, { "description": "internal op quiesce_inode:mds.0:192 fp=#0x1000000021a", "initiated_at": "2024-04-23T10:31:21.483400+0000", "age": 42.400989170999999, "duration": 42.402606259000002, "continuous": true, "type_data": { "result": -2147483648, "flag_point": "quiesce complete for non-auth inode", "reqid": { "entity": { "type": "mds", "num": 0 }, "tid": 192 }, "op_type": "internal_op", "internal_op": 5385, "op_name": "quiesce_inode", "events": [ { "time": "2024-04-23T10:31:21.483400+0000", "event": "initiated" }, { "time": "2024-04-23T10:31:21.483400+0000", "event": "header_read" }, { "time": "2024-04-23T10:31:21.483400+0000", "event": "throttled" }, { "time": "2024-04-23T10:31:21.483400+0000", "event": "all_read" }, { "time": "2024-04-23T10:31:21.483400+0000", "event": "dispatched" }, { "time": "2024-04-23T10:31:21.483686+0000", "event": "requesting remote authpins" }, { "time": "2024-04-23T10:31:21.506653+0000", "event": "acquired locks" }, { "time": "2024-04-23T10:31:21.506681+0000", "event": "quiesce complete for non-auth inode" } ], "locks": [ { "object": { "is_auth": false, "auth_state": { "replicas": {} }, "replica_state": { "authority": [ 2, -2 ], "replica_nonce": 1 }, "auth_pins": 0, "is_frozen": false, "is_freezing": false, "pins": { "request": 1, "lock": 1, "caps": 0, "exportingcaps": 0, "dirtyparent": 0, "dirty": 0, "authpin": 0, "tempexporting": 0 }, "nref": 2 }, "object_string": "[inode 0x1000000021a [7,head] /volumes/qa/sv_1/b206794e-7622-4ab8-8943-f3ae4e65b7a3/client.0/tmp/clients/client0/~dmtmp/COREL/MAILBOX2.CPT rep@2.1 v7246 s=260096 n(v0 rc2024-04-23T10:27:52.023396+0000 b260096 1=1+0)/n(v0 1=1+0) (iquiesce lock x=1 by request(mds.0:192 nref=3)) | request=1 lock=1 caps=0 exportingcaps=0 dirtyparent=0 dirty=0 authpin=0 tempexporting=0 0x5571cf561600]", "lock": { "gather_set": [], "state": "lock", "type": "iquiesce", "is_leased": false, "num_rdlocks": 0, "num_wrlocks": 0, "num_xlocks": 1, "xlock_by": { "reqid": { "entity": { "type": "mds", "num": 0 }, "tid": 192 } } }, "flags": 4, "wrlock_target": -1 } ] } }, { "description": "peer_request:mds.2:108", "initiated_at": "2024-04-23T10:31:21.446235+0000", "age": 42.438153837999998, "duration": 42.439831419999997, "continuous": false, "type_data": { "result": -2147483648, "flag_point": "dispatched", "reqid": { "entity": { "type": "mds", "num": 2 }, "tid": 108 }, "op_type": "peer_request", "leader_info": { "leader": "2" }, "events": [ { "time": "2024-04-23T10:31:21.446235+0000", "event": "initiated" }, { "time": "2024-04-23T10:31:21.446235+0000", "event": "header_read" }, { "time": "2024-04-23T10:31:21.446236+0000", "event": "throttled" }, { "time": "2024-04-23T10:31:21.446243+0000", "event": "all_read" }, { "time": "2024-04-23T10:31:21.447104+0000", "event": "dispatched" } ], "locks": [] } }, { "description": "internal op quiesce_inode:mds.0:168 fp=#0x10000000208", "initiated_at": "2024-04-23T10:31:21.459861+0000", "age": 42.424527853999997, "duration": 42.426226503999999, "continuous": true, "type_data": { "result": -2147483648, "flag_point": "quiesce complete", "reqid": { "entity": { "type": "mds", "num": 0 }, "tid": 168 }, "op_type": "internal_op", "internal_op": 5385, "op_name": "quiesce_inode", "events": [ { "time": "2024-04-23T10:31:21.459861+0000", "event": "initiated" }, { "time": "2024-04-23T10:31:21.459861+0000", "event": "header_read" }, { "time": "2024-04-23T10:31:21.459861+0000", "event": "throttled" }, { "time": "2024-04-23T10:31:21.459861+0000", "event": "all_read" }, { "time": "2024-04-23T10:31:21.459861+0000", "event": "dispatched" }, { "time": "2024-04-23T10:31:21.460889+0000", "event": "failed to xlock, waiting" }, { "time": "2024-04-23T10:31:21.468246+0000", "event": "acquired locks" }, { "time": "2024-04-23T10:31:21.497176+0000", "event": "quiesce complete" } ], "locks": [ { "object": { "is_auth": true, "auth_state": { "replicas": { "1": 1, "2": 1 } }, "replica_state": { "authority": [ 0, -2 ], "replica_nonce": 0 }, "auth_pins": 3, "is_frozen": false, "is_freezing": false, "pins": { "request": 1, "lock": 1, "dirfrag": 1, "caps": 1, "dirtyrstat": 0, "dirtyparent": 0, "replicated": 1, "dirty": 0, "waiter": 0, "authpin": 1 }, "nref": 6 }, "object_string": "[inode 0x10000000208 [...7,head] /volumes/qa/sv_1/b206794e-7622-4ab8-8943-f3ae4e65b7a3/client.0/tmp/clients/client0/~dmtmp/WORD/ auth{1=1,2=1} v45541 ap=3 f(v0 m2024-04-23T10:31:17.675718+0000 3=3+0) n(v0 rc2024-04-23T10:31:17.675718+0000 b780288 4=3+1) old_inodes=4 (inest mix) (iquiesce lock x=1 by request(mds.0:168 nref=3)) caps={15312=pAsLsXsFs/pAsLsXsFsxcral@6600} mcw={2=AsLsXsFs} | request=1 lock=1 dirfrag=1 caps=1 dirtyrstat=0 dirtyparent=0 replicated=1 dirty=0 waiter=0 authpin=1 0x5571cf343180]", "lock": { "gather_set": [], "state": "lock", "type": "iquiesce", "is_leased": false, "num_rdlocks": 0, "num_wrlocks": 0, "num_xlocks": 1, "xlock_by": { "reqid": { "entity": { "type": "mds", "num": 0 }, "tid": 168 } } }, "flags": 4, "wrlock_target": -1 } ] } }, { "description": "internal op quiesce_inode:mds.0:193 fp=#0x10000000220", "initiated_at": "2024-04-23T10:31:21.483415+0000", "age": 42.400974654000002, "duration": 42.402740053999999, "continuous": true, "type_data": { "result": -2147483648, "flag_point": "quiesce complete for non-auth inode", "reqid": { "entity": { "type": "mds", "num": 0 }, "tid": 193 }, "op_type": "internal_op", "internal_op": 5385, "op_name": "quiesce_inode", "events": [ { "time": "2024-04-23T10:31:21.483415+0000", "event": "initiated" }, { "time": "2024-04-23T10:31:21.483415+0000", "event": "header_read" }, { "time": "2024-04-23T10:31:21.483415+0000", "event": "throttled" }, { "time": "2024-04-23T10:31:21.483415+0000", "event": "all_read" }, { "time": "2024-04-23T10:31:21.483415+0000", "event": "dispatched" }, { "time": "2024-04-23T10:31:21.483737+0000", "event": "requesting remote authpins" }, { "time": "2024-04-23T10:31:21.507336+0000", "event": "acquired locks" }, { "time": "2024-04-23T10:31:21.507347+0000", "event": "quiesce complete for non-auth inode" } ], "locks": [ { "object": { "is_auth": false, "auth_state": { "replicas": {} }, "replica_state": { "authority": [ 2, -2 ], "replica_nonce": 1 }, "auth_pins": 0, "is_frozen": false, "is_freezing": false, "pins": { "request": 1, "lock": 1, "caps": 0, "exportingcaps": 0, "dirtyparent": 0, "dirty": 0, "authpin": 0, "tempexporting": 0 }, "nref": 2 }, "object_string": "[inode 0x10000000220 [7,head] /volumes/qa/sv_1/b206794e-7622-4ab8-8943-f3ae4e65b7a3/client.0/tmp/clients/client0/~dmtmp/COREL/CORELDRW.CDT rep@2.1 v7242 s=260096 n(v0 rc2024-04-23T10:27:52.054395+0000 b260096 1=1+0)/n(v0 1=1+0) (iquiesce lock x=1 by request(mds.0:193 nref=3)) | request=1 lock=1 caps=0 exportingcaps=0 dirtyparent=0 dirty=0 authpin=0 tempexporting=0 0x5571cf563700]", "lock": { "gather_set": [], "state": "lock", "type": "iquiesce", "is_leased": false, "num_rdlocks": 0, "num_wrlocks": 0, "num_xlocks": 1, "xlock_by": { "reqid": { "entity": { "type": "mds", "num": 0 }, "tid": 193 } } }, "flags": 4, "wrlock_target": -1 } ] } }, { "description": "internal op quiesce_inode:mds.0:152 fp=#0x10000000001", "initiated_at": "2024-04-23T10:31:21.447800+0000", "age": 42.436588751000002, "duration": 42.438416095999997, "continuous": true, "type_data": { "result": -2147483648, "flag_point": "quiesce complete", "reqid": { "entity": { "type": "mds", "num": 0 }, "tid": 152 }, "op_type": "internal_op", "internal_op": 5385, "op_name": "quiesce_inode", "events": [ { "time": "2024-04-23T10:31:21.447800+0000", "event": "initiated" }, { "time": "2024-04-23T10:31:21.447800+0000", "event": "header_read" }, { "time": "2024-04-23T10:31:21.447800+0000", "event": "throttled" }, { "time": "2024-04-23T10:31:21.447800+0000", "event": "all_read" }, { "time": "2024-04-23T10:31:21.447800+0000", "event": "dispatched" }, { "time": "2024-04-23T10:31:21.447978+0000", "event": "failed to xlock, waiting" }, { "time": "2024-04-23T10:31:21.448441+0000", "event": "acquired locks" }, { "time": "2024-04-23T10:31:21.510014+0000", "event": "quiesce complete" } ], "locks": [ { "object": { "is_auth": true, "auth_state": { "replicas": { "1": 1, "2": 1 } }, "replica_state": { "authority": [ 0, -2 ], "replica_nonce": 0 }, "auth_pins": 3, "is_frozen": false, "is_freezing": false, "pins": { "dirtyscattered": 1, "request": 1, "lock": 1, "dirfrag": 1, "caps": 0, "dirtyrstat": 0, "dirtyparent": 0, "replicated": 1, "dirty": 1, "waiter": 0, "authpin": 1 }, "nref": 7 }, "object_string": "[inode 0x10000000001 [...7,head] /volumes/qa/ auth{1=1,2=1} v418 ap=3 f(v0 m2024-04-23T10:24:49.714717+0000 2=0+2) n(v30 rc2024-04-23T10:31:17.764716+0000 b21075308 84=65+19) old_inodes=5 (inest mix dirty) (iquiesce lock x=1 by request(mds.0:152 nref=3)) | dirtyscattered=1 request=1 lock=1 dirfrag=1 caps=0 dirtyrstat=0 dirtyparent=0 replicated=1 dirty=1 waiter=0 authpin=1 0x5571cf18f600]", "lock": { "gather_set": [], "state": "lock", "type": "iquiesce", "is_leased": false, "num_rdlocks": 0, "num_wrlocks": 0, "num_xlocks": 1, "xlock_by": { "reqid": { "entity": { "type": "mds", "num": 0 }, "tid": 152 } } }, "flags": 4, "wrlock_target": -1 } ] } }, { "description": "internal op quiesce_inode:mds.0:169 fp=#0x10000000240", "initiated_at": "2024-04-23T10:31:21.459879+0000", "age": 42.424509798000003, "duration": 42.426401943000002, "continuous": true, "type_data": { "result": -2147483648, "flag_point": "quiesce complete", "reqid": { "entity": { "type": "mds", "num": 0 }, "tid": 169 }, "op_type": "internal_op", "internal_op": 5385, "op_name": "quiesce_inode", "events": [ { "time": "2024-04-23T10:31:21.459879+0000", "event": "initiated" }, { "time": "2024-04-23T10:31:21.459879+0000", "event": "header_read" }, { "time": "2024-04-23T10:31:21.459879+0000", "event": "throttled" }, { "time": "2024-04-23T10:31:21.459879+0000", "event": "all_read" }, { "time": "2024-04-23T10:31:21.459879+0000", "event": "dispatched" }, { "time": "2024-04-23T10:31:21.461207+0000", "event": "failed to xlock, waiting" }, { "time": "2024-04-23T10:31:21.469589+0000", "event": "acquired locks" }, { "time": "2024-04-23T10:31:21.509773+0000", "event": "quiesce complete" } ], "locks": [ { "object": { "is_auth": true, "auth_state": { "replicas": { "1": 1, "2": 1 } }, "replica_state": { "authority": [ 0, -2 ], "replica_nonce": 0 }, "auth_pins": 3, "is_frozen": false, "is_freezing": false, "pins": { "dirtyscattered": 0, "request": 1, "lock": 1, "importing": 0, "dirfrag": 1, "caps": 1, "dirtyparent": 0, "replicated": 1, "dirty": 0, "waiter": 0, "authpin": 1 }, "nref": 6 }, "object_string": "[inode 0x10000000240 [...7,head] /volumes/qa/sv_1/b206794e-7622-4ab8-8943-f3ae4e65b7a3/client.0/tmp/clients/client0/~dmtmp/SEED/ auth{1=1,2=1} v45576 ap=3 f(v0 m2024-04-23T10:27:52.196392+0000 3=3+0) n(v2 rc2024-04-23T10:31:17.246728+0000 b3244032 4=3+1) old_inodes=4 (inest mix) (iquiesce lock x=1 by request(mds.0:169 nref=3)) caps={15312=pAsLsXsFs/pAsLsXsFsxcral@32} mcw={2=AsLsXsFs} | dirtyscattered=0 request=1 lock=1 importing=0 dirfrag=1 caps=1 dirtyparent=0 replicated=1 dirty=0 waiter=0 authpin=1 0x5571cf56b180]", "lock": { "gather_set": [], "state": "lock", "type": "iquiesce", "is_leased": false, "num_rdlocks": 0, "num_wrlocks": 0, "num_xlocks": 1, "xlock_by": { "reqid": { "entity": { "type": "mds", "num": 0 }, "tid": 169 } } }, "flags": 4, "wrlock_target": -1 } ] } }, { "description": "internal op quiesce_inode:mds.0:194 fp=#0x1000000021e", "initiated_at": "2024-04-23T10:31:21.483427+0000", "age": 42.400962466000003, "duration": 42.402994505000002, "continuous": true, "type_data": { "result": -2147483648, "flag_point": "quiesce complete for non-auth inode", "reqid": { "entity": { "type": "mds", "num": 0 }, "tid": 194 }, "op_type": "internal_op", "internal_op": 5385, "op_name": "quiesce_inode", "events": [ { "time": "2024-04-23T10:31:21.483427+0000", "event": "initiated" }, { "time": "2024-04-23T10:31:21.483427+0000", "event": "header_read" }, { "time": "2024-04-23T10:31:21.483427+0000", "event": "throttled" }, { "time": "2024-04-23T10:31:21.483427+0000", "event": "all_read" }, { "time": "2024-04-23T10:31:21.483427+0000", "event": "dispatched" }, { "time": "2024-04-23T10:31:21.483789+0000", "event": "requesting remote authpins" }, { "time": "2024-04-23T10:31:21.507506+0000", "event": "acquired locks" }, { "time": "2024-04-23T10:31:21.507515+0000", "event": "quiesce complete for non-auth inode" } ], "locks": [ { "object": { "is_auth": false, "auth_state": { "replicas": {} }, "replica_state": { "authority": [ 2, -2 ], "replica_nonce": 1 }, "auth_pins": 0, "is_frozen": false, "is_freezing": false, "pins": { "request": 1, "lock": 1, "caps": 0, "exportingcaps": 0, "dirtyparent": 0, "dirty": 0, "authpin": 0, "tempexporting": 0 }, "nref": 2 }, "object_string": "[inode 0x1000000021e [7,head] /volumes/qa/sv_1/b206794e-7622-4ab8-8943-f3ae4e65b7a3/client.0/tmp/clients/client0/~dmtmp/COREL/GRAPHIC2.CDR rep@2.1 v7254 s=260096 n(v0 rc2024-04-23T10:27:52.043396+0000 b260096 1=1+0)/n(v0 1=1+0) (iquiesce lock x=1 by request(mds.0:194 nref=3)) | request=1 lock=1 caps=0 exportingcaps=0 dirtyparent=0 dirty=0 authpin=0 tempexporting=0 0x5571cf562c00]", "lock": { "gather_set": [], "state": "lock", "type": "iquiesce", "is_leased": false, "num_rdlocks": 0, "num_wrlocks": 0, "num_xlocks": 1, "xlock_by": { "reqid": { "entity": { "type": "mds", "num": 0 }, "tid": 194 } } }, "flags": 4, "wrlock_target": -1 } ] } }, { "description": "peer_request:mds.1:104", "initiated_at": "2024-04-23T10:31:21.447409+0000", "age": 42.436980587999997, "duration": 42.439085126999998, "continuous": false, "type_data": { "result": -2147483648, "flag_point": "dispatched", "reqid": { "entity": { "type": "mds", "num": 1 }, "tid": 104 }, "op_type": "peer_request", "leader_info": { "leader": "1" }, "events": [ { "time": "2024-04-23T10:31:21.447409+0000", "event": "initiated" }, { "time": "2024-04-23T10:31:21.447409+0000", "event": "header_read" }, { "time": "2024-04-23T10:31:21.447409+0000", "event": "throttled" }, { "time": "2024-04-23T10:31:21.447417+0000", "event": "all_read" }, { "time": "2024-04-23T10:31:21.448013+0000", "event": "dispatched" } ], "locks": [] } }, { "description": "internal op quiesce_inode:mds.0:170 fp=#0x10000000221", "initiated_at": "2024-04-23T10:31:21.459896+0000", "age": 42.424492903000001, "duration": 42.426618873000002, "continuous": true, "type_data": { "result": -2147483648, "flag_point": "quiesce complete", "reqid": { "entity": { "type": "mds", "num": 0 }, "tid": 170 }, "op_type": "internal_op", "internal_op": 5385, "op_name": "quiesce_inode", "events": [ { "time": "2024-04-23T10:31:21.459896+0000", "event": "initiated" }, { "time": "2024-04-23T10:31:21.459896+0000", "event": "header_read" }, { "time": "2024-04-23T10:31:21.459896+0000", "event": "throttled" }, { "time": "2024-04-23T10:31:21.459896+0000", "event": "all_read" }, { "time": "2024-04-23T10:31:21.459896+0000", "event": "dispatched" }, { "time": "2024-04-23T10:31:21.461535+0000", "event": "failed to xlock, waiting" }, { "time": "2024-04-23T10:31:21.467090+0000", "event": "acquired locks" }, { "time": "2024-04-23T10:31:21.479758+0000", "event": "quiesce complete" } ], "locks": [ { "object": { "is_auth": true, "auth_state": { "replicas": { "1": 1, "2": 1 } }, "replica_state": { "authority": [ 0, -2 ], "replica_nonce": 0 }, "auth_pins": 3, "is_frozen": false, "is_freezing": false, "pins": { "request": 1, "lock": 1, "dirfrag": 1, "caps": 1, "dirtyrstat": 0, "dirtyparent": 0, "replicated": 1, "dirty": 0, "waiter": 0, "authpin": 1 }, "nref": 6 }, "object_string": "[inode 0x10000000221 [...7,head] /volumes/qa/sv_1/b206794e-7622-4ab8-8943-f3ae4e65b7a3/client.0/tmp/clients/client0/~dmtmp/WORDPRO/ auth{1=1,2=1} v45561 ap=3 f(v0 m2024-04-23T10:31:17.760716+0000 3=3+0) n(v0 rc2024-04-23T10:31:17.760716+0000 b780288 4=3+1) old_inodes=4 (inest mix) (iquiesce lock x=1 by request(mds.0:170 nref=3)) caps={15312=pAsLsXsFs/pAsLsXsFsxcral@9160},l=15312 | request=1 lock=1 dirfrag=1 caps=1 dirtyrstat=0 dirtyparent=0 replicated=1 dirty=0 waiter=0 authpin=1 0x5571cf4d4b00]", "lock": { "gather_set": [], "state": "lock", "type": "iquiesce", "is_leased": false, "num_rdlocks": 0, "num_wrlocks": 0, "num_xlocks": 1, "xlock_by": { "reqid": { "entity": { "type": "mds", "num": 0 }, "tid": 170 } } }, "flags": 4, "wrlock_target": -1 } ] } }, { "description": "internal op quiesce_inode:mds.0:195 fp=#0x10000000218", "initiated_at": "2024-04-23T10:31:21.483450+0000", "age": 42.400938924999998, "duration": 42.403132233000001, "continuous": true, "type_data": { "result": -2147483648, "flag_point": "quiesce complete for non-auth inode", "reqid": { "entity": { "type": "mds", "num": 0 }, "tid": 195 }, "op_type": "internal_op", "internal_op": 5385, "op_name": "quiesce_inode", "events": [ { "time": "2024-04-23T10:31:21.483450+0000", "event": "initiated" }, { "time": "2024-04-23T10:31:21.483450+0000", "event": "header_read" }, { "time": "2024-04-23T10:31:21.483450+0000", "event": "throttled" }, { "time": "2024-04-23T10:31:21.483450+0000", "event": "all_read" }, { "time": "2024-04-23T10:31:21.483450+0000", "event": "dispatched" }, { "time": "2024-04-23T10:31:21.483840+0000", "event": "requesting remote authpins" }, { "time": "2024-04-23T10:31:21.507655+0000", "event": "acquired locks" }, { "time": "2024-04-23T10:31:21.507665+0000", "event": "quiesce complete for non-auth inode" } ], "locks": [ { "object": { "is_auth": false, "auth_state": { "replicas": {} }, "replica_state": { "authority": [ 2, -2 ], "replica_nonce": 1 }, "auth_pins": 0, "is_frozen": false, "is_freezing": false, "pins": { "request": 1, "lock": 1, "caps": 0, "exportingcaps": 0, "dirtyparent": 0, "dirty": 0, "authpin": 0, "tempexporting": 0 }, "nref": 2 }, "object_string": "[inode 0x10000000218 [7,head] /volumes/qa/sv_1/b206794e-7622-4ab8-8943-f3ae4e65b7a3/client.0/tmp/clients/client0/~dmtmp/COREL/ARTISTIC.ACL rep@2.1 v7240 s=260096 n(v0 rc2024-04-23T10:27:52.014396+0000 b260096 1=1+0)/n(v0 1=1+0) (iquiesce lock x=1 by request(mds.0:195 nref=3)) | request=1 lock=1 caps=0 exportingcaps=0 dirtyparent=0 dirty=0 authpin=0 tempexporting=0 0x5571cf560b00]", "lock": { "gather_set": [], "state": "lock", "type": "iquiesce", "is_leased": false, "num_rdlocks": 0, "num_wrlocks": 0, "num_xlocks": 1, "xlock_by": { "reqid": { "entity": { "type": "mds", "num": 0 }, "tid": 195 } } }, "flags": 4, "wrlock_target": -1 } ] } }, { "description": "peer_request:mds.2:109", "initiated_at": "2024-04-23T10:31:21.447475+0000", "age": 42.436913812999997, "duration": 42.439168008999999, "continuous": false, "type_data": { "result": -2147483648, "flag_point": "dispatched", "reqid": { "entity": { "type": "mds", "num": 2 }, "tid": 109 }, "op_type": "peer_request", "leader_info": { "leader": "2" }, "events": [ { "time": "2024-04-23T10:31:21.447475+0000", "event": "initiated" }, { "time": "2024-04-23T10:31:21.447475+0000", "event": "header_read" }, { "time": "2024-04-23T10:31:21.447476+0000", "event": "throttled" }, { "time": "2024-04-23T10:31:21.447482+0000", "event": "all_read" }, { "time": "2024-04-23T10:31:21.448091+0000", "event": "dispatched" } ], "locks": [] } }, { "description": "internal op quiesce_inode:mds.0:171 fp=#0x10000000229", "initiated_at": "2024-04-23T10:31:21.459914+0000", "age": 42.424475653000002, "duration": 42.426751056000001, "continuous": true, "type_data": { "result": -2147483648, "flag_point": "quiesce complete", "reqid": { "entity": { "type": "mds", "num": 0 }, "tid": 171 }, "op_type": "internal_op", "internal_op": 5385, "op_name": "quiesce_inode", "events": [ { "time": "2024-04-23T10:31:21.459914+0000", "event": "initiated" }, { "time": "2024-04-23T10:31:21.459914+0000", "event": "header_read" }, { "time": "2024-04-23T10:31:21.459914+0000", "event": "throttled" }, { "time": "2024-04-23T10:31:21.459914+0000", "event": "all_read" }, { "time": "2024-04-23T10:31:21.459914+0000", "event": "dispatched" }, { "time": "2024-04-23T10:31:21.461888+0000", "event": "failed to xlock, waiting" }, { "time": "2024-04-23T10:31:21.472768+0000", "event": "acquired locks" }, { "time": "2024-04-23T10:31:21.473165+0000", "event": "quiesce complete" } ], "locks": [ { "object": { "is_auth": true, "auth_state": { "replicas": { "1": 1, "2": 1 } }, "replica_state": { "authority": [ 0, -2 ], "replica_nonce": 0 }, "auth_pins": 4, "is_frozen": false, "is_freezing": false, "pins": { "dirtyscattered": 0, "request": 1, "lock": 1, "dirfrag": 1, "caps": 1, "dirtyrstat": 0, "dirtyparent": 0, "replicated": 1, "dirty": 0, "waiter": 0, "authpin": 1, "discoverbase": 0 }, "nref": 6 }, "object_string": "[inode 0x10000000229 [...7,head] /volumes/qa/sv_1/b206794e-7622-4ab8-8943-f3ae4e65b7a3/client.0/tmp/clients/client0/~dmtmp/PARADOX/ auth{1=1,2=1} v45567 ap=4 f(v1976 m2024-04-23T10:31:16.541744+0000 24=24+0) n(v19 rc2024-04-23T10:31:16.541744+0000 b5201920 25=24+1) old_inodes=4 (inest mix) (iquiesce lock x=1 by request(mds.0:171 nref=3)) caps={15312=pAsLsXsFs/pAsLsXsFsxcral@5318} mcw={1=AsLsXsFs} | dirtyscattered=0 request=1 lock=1 dirfrag=1 caps=1 dirtyrstat=0 dirtyparent=0 replicated=1 dirty=0 waiter=0 authpin=1 discoverbase=0 0x5571cf588680]", "lock": { "gather_set": [], "state": "lock", "type": "iquiesce", "is_leased": false, "num_rdlocks": 0, "num_wrlocks": 0, "num_xlocks": 1, "xlock_by": { "reqid": { "entity": { "type": "mds", "num": 0 }, "tid": 171 } } }, "flags": 4, "wrlock_target": -1 } ] } }, { "description": "internal op quiesce_inode:mds.0:196 fp=#0x1000000021c", "initiated_at": "2024-04-23T10:31:21.483464+0000", "age": 42.400924844000002, "duration": 42.403269293999998, "continuous": true, "type_data": { "result": -2147483648, "flag_point": "quiesce complete for non-auth inode", "reqid": { "entity": { "type": "mds", "num": 0 }, "tid": 196 }, "op_type": "internal_op", "internal_op": 5385, "op_name": "quiesce_inode", "events": [ { "time": "2024-04-23T10:31:21.483464+0000", "event": "initiated" }, { "time": "2024-04-23T10:31:21.483464+0000", "event": "header_read" }, { "time": "2024-04-23T10:31:21.483464+0000", "event": "throttled" }, { "time": "2024-04-23T10:31:21.483464+0000", "event": "all_read" }, { "time": "2024-04-23T10:31:21.483464+0000", "event": "dispatched" }, { "time": "2024-04-23T10:31:21.483894+0000", "event": "requesting remote authpins" }, { "time": "2024-04-23T10:31:21.507796+0000", "event": "acquired locks" }, { "time": "2024-04-23T10:31:21.507805+0000", "event": "quiesce complete for non-auth inode" } ], "locks": [ { "object": { "is_auth": false, "auth_state": { "replicas": {} }, "replica_state": { "authority": [ 2, -2 ], "replica_nonce": 1 }, "auth_pins": 0, "is_frozen": false, "is_freezing": false, "pins": { "request": 1, "lock": 1, "caps": 0, "exportingcaps": 0, "dirtyparent": 0, "dirty": 0, "authpin": 0, "tempexporting": 0 }, "nref": 2 }, "object_string": "[inode 0x1000000021c [7,head] /volumes/qa/sv_1/b206794e-7622-4ab8-8943-f3ae4e65b7a3/client.0/tmp/clients/client0/~dmtmp/COREL/SIGN2.CPT rep@2.1 v7250 s=260096 n(v0 rc2024-04-23T10:27:52.033396+0000 b260096 1=1+0)/n(v0 1=1+0) (iquiesce lock x=1 by request(mds.0:196 nref=3)) | request=1 lock=1 caps=0 exportingcaps=0 dirtyparent=0 dirty=0 authpin=0 tempexporting=0 0x5571cf562100]", "lock": { "gather_set": [], "state": "lock", "type": "iquiesce", "is_leased": false, "num_rdlocks": 0, "num_wrlocks": 0, "num_xlocks": 1, "xlock_by": { "reqid": { "entity": { "type": "mds", "num": 0 }, "tid": 196 } } }, "flags": 4, "wrlock_target": -1 } ] } }, { "description": "internal op quiesce_inode:mds.0:153 fp=#0x100000001fa", "initiated_at": "2024-04-23T10:31:21.448695+0000", "age": 42.435693784999998, "duration": 42.438098199000002, "continuous": true, "type_data": { "result": -2147483648, "flag_point": "quiesce complete", "reqid": { "entity": { "type": "mds", "num": 0 }, "tid": 153 }, "op_type": "internal_op", "internal_op": 5385, "op_name": "quiesce_inode", "events": [ { "time": "2024-04-23T10:31:21.448695+0000", "event": "initiated" }, { "time": "2024-04-23T10:31:21.448695+0000", "event": "header_read" }, { "time": "2024-04-23T10:31:21.448695+0000", "event": "throttled" }, { "time": "2024-04-23T10:31:21.448695+0000", "event": "all_read" }, { "time": "2024-04-23T10:31:21.448695+0000", "event": "dispatched" }, { "time": "2024-04-23T10:31:21.448924+0000", "event": "failed to xlock, waiting" }, { "time": "2024-04-23T10:31:21.449710+0000", "event": "acquired locks" }, { "time": "2024-04-23T10:31:21.509989+0000", "event": "quiesce complete" } ], "locks": [ { "object": { "is_auth": true, "auth_state": { "replicas": { "1": 1, "2": 1 } }, "replica_state": { "authority": [ 0, -2 ], "replica_nonce": 0 }, "auth_pins": 3, "is_frozen": false, "is_freezing": false, "pins": { "dirtyscattered": 1, "request": 1, "lock": 1, "dirfrag": 1, "caps": 0, "dirtyrstat": 0, "dirtyparent": 0, "replicated": 1, "dirty": 1, "waiter": 0, "authpin": 1 }, "nref": 7 }, "object_string": "[inode 0x100000001fa [...7,head] /volumes/qa/sv_1/ auth{1=1,2=1} v436 ap=3 snaprealm=0x5571cf174000 f(v0 m2024-04-23T10:24:49.735055+0000 2=1+1) n(v29 rc2024-04-23T10:31:17.764716+0000 b21075191 80=64+16) old_inodes=5 (inest mix dirty) (iquiesce lock x=1 by request(mds.0:153 nref=3)) | dirtyscattered=1 request=1 lock=1 dirfrag=1 caps=0 dirtyrstat=0 dirtyparent=0 replicated=1 dirty=1 waiter=0 authpin=1 0x5571cf1dfb80]", "lock": { "gather_set": [], "state": "lock", "type": "iquiesce", "is_leased": false, "num_rdlocks": 0, "num_wrlocks": 0, "num_xlocks": 1, "xlock_by": { "reqid": { "entity": { "type": "mds", "num": 0 }, "tid": 153 } } }, "flags": 4, "wrlock_target": -1 } ] } }, { "description": "internal op quiesce_inode:mds.0:172 fp=#0x10000000225", "initiated_at": "2024-04-23T10:31:21.459931+0000", "age": 42.424458305000002, "duration": 42.426927628000001, "continuous": true, "type_data": { "result": -2147483648, "flag_point": "quiesce complete", "reqid": { "entity": { "type": "mds", "num": 0 }, "tid": 172 }, "op_type": "internal_op", "internal_op": 5385, "op_name": "quiesce_inode", "events": [ { "time": "2024-04-23T10:31:21.459931+0000", "event": "initiated" }, { "time": "2024-04-23T10:31:21.459931+0000", "event": "header_read" }, { "time": "2024-04-23T10:31:21.459931+0000", "event": "throttled" }, { "time": "2024-04-23T10:31:21.459931+0000", "event": "all_read" }, { "time": "2024-04-23T10:31:21.459931+0000", "event": "dispatched" }, { "time": "2024-04-23T10:31:21.462138+0000", "event": "failed to xlock, waiting" }, { "time": "2024-04-23T10:31:21.486438+0000", "event": "acquired locks" }, { "time": "2024-04-23T10:31:21.486778+0000", "event": "quiesce complete" } ], "locks": [ { "object": { "is_auth": true, "auth_state": { "replicas": { "1": 1, "2": 1 } }, "replica_state": { "authority": [ 0, -2 ], "replica_nonce": 0 }, "auth_pins": 3, "is_frozen": false, "is_freezing": false, "pins": { "dirtyscattered": 0, "request": 1, "lock": 1, "dirfrag": 1, "caps": 1, "dirtyrstat": 0, "dirtyparent": 0, "replicated": 1, "dirty": 0, "waiter": 0, "authpin": 1 }, "nref": 6 }, "object_string": "[inode 0x10000000225 [...7,head] /volumes/qa/sv_1/b206794e-7622-4ab8-8943-f3ae4e65b7a3/client.0/tmp/clients/client0/~dmtmp/PM/ auth{1=1,2=1} v45574 ap=3 f(v2926 m2024-04-23T10:31:17.764716+0000 3=3+0) n(v17 rc2024-04-23T10:31:17.764716+0000 b780288 4=3+1) old_inodes=4 (inest mix) (iquiesce lock x=1 by request(mds.0:172 nref=3)) caps={15312=pAsLsXsFs/pAsLsXsFsxcral@7614} mcw={2=AsLsXsFs} | dirtyscattered=0 request=1 lock=1 dirfrag=1 caps=1 dirtyrstat=0 dirtyparent=0 replicated=1 dirty=0 waiter=0 authpin=1 0x5571cf587080]", "lock": { "gather_set": [], "state": "lock", "type": "iquiesce", "is_leased": false, "num_rdlocks": 0, "num_wrlocks": 0, "num_xlocks": 1, "xlock_by": { "reqid": { "entity": { "type": "mds", "num": 0 }, "tid": 172 } } }, "flags": 4, "wrlock_target": -1 } ] } }, { "description": "internal op quiesce_inode:mds.0:197 fp=#0x10000000217", "initiated_at": "2024-04-23T10:31:21.483478+0000", "age": 42.400911235000002, "duration": 42.403449692000002, "continuous": true, "type_data": { "result": -2147483648, "flag_point": "quiesce complete for non-auth inode", "reqid": { "entity": { "type": "mds", "num": 0 }, "tid": 197 }, "op_type": "internal_op", "internal_op": 5385, "op_name": "quiesce_inode", "events": [ { "time": "2024-04-23T10:31:21.483478+0000", "event": "initiated" }, { "time": "2024-04-23T10:31:21.483478+0000", "event": "header_read" }, { "time": "2024-04-23T10:31:21.483478+0000", "event": "throttled" }, { "time": "2024-04-23T10:31:21.483478+0000", "event": "all_read" }, { "time": "2024-04-23T10:31:21.483478+0000", "event": "dispatched" }, { "time": "2024-04-23T10:31:21.483944+0000", "event": "requesting remote authpins" }, { "time": "2024-04-23T10:31:21.507889+0000", "event": "acquired locks" }, { "time": "2024-04-23T10:31:21.507899+0000", "event": "quiesce complete for non-auth inode" } ], "locks": [ { "object": { "is_auth": false, "auth_state": { "replicas": {} }, "replica_state": { "authority": [ 2, -2 ], "replica_nonce": 1 }, "auth_pins": 0, "is_frozen": false, "is_freezing": false, "pins": { "request": 1, "lock": 1, "caps": 0, "exportingcaps": 0, "dirtyparent": 0, "dirty": 0, "authpin": 0, "tempexporting": 0 }, "nref": 2 }, "object_string": "[inode 0x10000000217 [7,head] /volumes/qa/sv_1/b206794e-7622-4ab8-8943-f3ae4e65b7a3/client.0/tmp/clients/client0/~dmtmp/COREL/CDRBARS.CFG rep@2.1 v7236 s=260096 n(v0 rc2024-04-23T10:29:16.529476+0000 b260096 1=1+0)/n(v0 1=1+0) (iquiesce lock x=1 by request(mds.0:197 nref=3)) | request=1 lock=1 caps=0 exportingcaps=0 dirtyparent=0 dirty=0 authpin=0 tempexporting=0 0x5571cf560580]", "lock": { "gather_set": [], "state": "lock", "type": "iquiesce", "is_leased": false, "num_rdlocks": 0, "num_wrlocks": 0, "num_xlocks": 1, "xlock_by": { "reqid": { "entity": { "type": "mds", "num": 0 }, "tid": 197 } } }, "flags": 4, "wrlock_target": -1 } ] } }, { "description": "internal op quiesce_inode:mds.0:154 fp=#0x10000000002", "initiated_at": "2024-04-23T10:31:21.448714+0000", "age": 42.435675556, "duration": 42.438274481999997, "continuous": true, "type_data": { "result": -2147483648, "flag_point": "quiesce complete", "reqid": { "entity": { "type": "mds", "num": 0 }, "tid": 154 }, "op_type": "internal_op", "internal_op": 5385, "op_name": "quiesce_inode", "events": [ { "time": "2024-04-23T10:31:21.448714+0000", "event": "initiated" }, { "time": "2024-04-23T10:31:21.448714+0000", "event": "header_read" }, { "time": "2024-04-23T10:31:21.448714+0000", "event": "throttled" }, { "time": "2024-04-23T10:31:21.448714+0000", "event": "all_read" }, { "time": "2024-04-23T10:31:21.448714+0000", "event": "dispatched" }, { "time": "2024-04-23T10:31:21.449116+0000", "event": "failed to xlock, waiting" }, { "time": "2024-04-23T10:31:21.450377+0000", "event": "acquired locks" }, { "time": "2024-04-23T10:31:21.450570+0000", "event": "quiesce complete" } ], "locks": [ { "object": { "is_auth": true, "auth_state": { "replicas": { "1": 1 } }, "replica_state": { "authority": [ 0, -2 ], "replica_nonce": 0 }, "auth_pins": 2, "is_frozen": false, "is_freezing": false, "pins": { "request": 1, "lock": 1, "dirfrag": 1, "caps": 0, "dirtyparent": 0, "replicated": 1, "dirty": 0, "waiter": 0, "authpin": 1 }, "nref": 5 }, "object_string": "[inode 0x10000000002 [...2,head] /volumes/qa/sv_0/ auth{1=1} v34 ap=2 snaprealm=0x5571cf174480 f(v0 m2024-04-23T10:24:49.040186+0000 2=1+1) n(v0 rc2024-04-23T10:24:49.040186+0000 b117 3=1+2) (inest mix) (iquiesce lock x=1 by request(mds.0:154 nref=3)) | request=1 lock=1 dirfrag=1 caps=0 dirtyparent=0 replicated=1 dirty=0 waiter=0 authpin=1 0x5571cf18fb80]", "lock": { "gather_set": [], "state": "lock", "type": "iquiesce", "is_leased": false, "num_rdlocks": 0, "num_wrlocks": 0, "num_xlocks": 1, "xlock_by": { "reqid": { "entity": { "type": "mds", "num": 0 }, "tid": 154 } } }, "flags": 4, "wrlock_target": -1 } ] } }, { "description": "internal op quiesce_inode:mds.0:173 fp=#0x1000000020f", "initiated_at": "2024-04-23T10:31:21.459950+0000", "age": 42.424439393, "duration": 42.427099659, "continuous": true, "type_data": { "result": -2147483648, "flag_point": "quiesce complete", "reqid": { "entity": { "type": "mds", "num": 0 }, "tid": 173 }, "op_type": "internal_op", "internal_op": 5385, "op_name": "quiesce_inode", "events": [ { "time": "2024-04-23T10:31:21.459950+0000", "event": "initiated" }, { "time": "2024-04-23T10:31:21.459950+0000", "event": "header_read" }, { "time": "2024-04-23T10:31:21.459950+0000", "event": "throttled" }, { "time": "2024-04-23T10:31:21.459950+0000", "event": "all_read" }, { "time": "2024-04-23T10:31:21.459950+0000", "event": "dispatched" }, { "time": "2024-04-23T10:31:21.462404+0000", "event": "failed to xlock, waiting" }, { "time": "2024-04-23T10:31:21.489527+0000", "event": "acquired locks" }, { "time": "2024-04-23T10:31:21.489939+0000", "event": "quiesce complete" } ], "locks": [ { "object": { "is_auth": true, "auth_state": { "replicas": { "1": 1, "2": 1 } }, "replica_state": { "authority": [ 0, -2 ], "replica_nonce": 0 }, "auth_pins": 3, "is_frozen": false, "is_freezing": false, "pins": { "dirtyscattered": 0, "request": 1, "lock": 1, "dirfrag": 1, "caps": 1, "dirtyparent": 0, "replicated": 1, "dirty": 0, "waiter": 0, "authpin": 1 }, "nref": 6 }, "object_string": "[inode 0x1000000020f [...7,head] /volumes/qa/sv_1/b206794e-7622-4ab8-8943-f3ae4e65b7a3/client.0/tmp/clients/client0/~dmtmp/PWRPNT/ auth{1=1,2=1} v45572 ap=3 f(v1293 m2024-04-23T10:31:17.718717+0000 5=5+0) n(v18 rc2024-04-23T10:31:17.718717+0000 b1300480 6=5+1) old_inodes=4 (inest mix) (iquiesce lock x=1 by request(mds.0:173 nref=3)) caps={15312=pAsLsXsFs/pAsLsXsFsxcral@3438} mcw={2=AsLsXsFs} | dirtyscattered=0 request=1 lock=1 dirfrag=1 caps=1 dirtyparent=0 replicated=1 dirty=0 waiter=0 authpin=1 0x5571cf53f600]", "lock": { "gather_set": [], "state": "lock", "type": "iquiesce", "is_leased": false, "num_rdlocks": 0, "num_wrlocks": 0, "num_xlocks": 1, "xlock_by": { "reqid": { "entity": { "type": "mds", "num": 0 }, "tid": 173 } } }, "flags": 4, "wrlock_target": -1 } ] } }, { "description": "internal op quiesce_inode:mds.0:198 fp=#0x1000000021d", "initiated_at": "2024-04-23T10:31:21.483491+0000", "age": 42.400898220999999, "duration": 42.403625163999997, "continuous": true, "type_data": { "result": -2147483648, "flag_point": "quiesce complete for non-auth inode", "reqid": { "entity": { "type": "mds", "num": 0 }, "tid": 198 }, "op_type": "internal_op", "internal_op": 5385, "op_name": "quiesce_inode", "events": [ { "time": "2024-04-23T10:31:21.483491+0000", "event": "initiated" }, { "time": "2024-04-23T10:31:21.483491+0000", "event": "header_read" }, { "time": "2024-04-23T10:31:21.483491+0000", "event": "throttled" }, { "time": "2024-04-23T10:31:21.483491+0000", "event": "all_read" }, { "time": "2024-04-23T10:31:21.483491+0000", "event": "dispatched" }, { "time": "2024-04-23T10:31:21.483994+0000", "event": "requesting remote authpins" }, { "time": "2024-04-23T10:31:21.507982+0000", "event": "acquired locks" }, { "time": "2024-04-23T10:31:21.507996+0000", "event": "quiesce complete for non-auth inode" } ], "locks": [ { "object": { "is_auth": false, "auth_state": { "replicas": {} }, "replica_state": { "authority": [ 2, -2 ], "replica_nonce": 1 }, "auth_pins": 0, "is_frozen": false, "is_freezing": false, "pins": { "request": 1, "lock": 1, "caps": 0, "exportingcaps": 0, "dirtyparent": 0, "dirty": 0, "authpin": 0, "tempexporting": 0 }, "nref": 2 }, "object_string": "[inode 0x1000000021d [7,head] /volumes/qa/sv_1/b206794e-7622-4ab8-8943-f3ae4e65b7a3/client.0/tmp/clients/client0/~dmtmp/COREL/TRAIN.CDR rep@2.1 v7252 s=260096 n(v0 rc2024-04-23T10:27:52.038396+0000 b260096 1=1+0)/n(v0 1=1+0) (iquiesce lock x=1 by request(mds.0:198 nref=3)) | request=1 lock=1 caps=0 exportingcaps=0 dirtyparent=0 dirty=0 authpin=0 tempexporting=0 0x5571cf562680]", "lock": { "gather_set": [], "state": "lock", "type": "iquiesce", "is_leased": false, "num_rdlocks": 0, "num_wrlocks": 0, "num_xlocks": 1, "xlock_by": { "reqid": { "entity": { "type": "mds", "num": 0 }, "tid": 198 } } }, "flags": 4, "wrlock_target": -1 } ] } }, { "description": "peer_request:mds.1:105", "initiated_at": "2024-04-23T10:31:21.448402+0000", "age": 42.435987691000001, "duration": 42.438773701000002, "continuous": false, "type_data": { "result": -2147483648, "flag_point": "dispatched", "reqid": { "entity": { "type": "mds", "num": 1 }, "tid": 105 }, "op_type": "peer_request", "leader_info": { "leader": "1" }, "events": [ { "time": "2024-04-23T10:31:21.448402+0000", "event": "initiated" }, { "time": "2024-04-23T10:31:21.448402+0000", "event": "header_read" }, { "time": "2024-04-23T10:31:21.448402+0000", "event": "throttled" }, { "time": "2024-04-23T10:31:21.448411+0000", "event": "all_read" }, { "time": "2024-04-23T10:31:21.449159+0000", "event": "dispatched" } ], "locks": [] } }, { "description": "internal op quiesce_inode:mds.0:174 fp=#0x1000000023e", "initiated_at": "2024-04-23T10:31:21.459967+0000", "age": 42.424422245999999, "duration": 42.427230668, "continuous": true, "type_data": { "result": -2147483648, "flag_point": "quiesce complete", "reqid": { "entity": { "type": "mds", "num": 0 }, "tid": 174 }, "op_type": "internal_op", "internal_op": 5385, "op_name": "quiesce_inode", "events": [ { "time": "2024-04-23T10:31:21.459967+0000", "event": "initiated" }, { "time": "2024-04-23T10:31:21.459967+0000", "event": "header_read" }, { "time": "2024-04-23T10:31:21.459967+0000", "event": "throttled" }, { "time": "2024-04-23T10:31:21.459967+0000", "event": "all_read" }, { "time": "2024-04-23T10:31:21.459967+0000", "event": "dispatched" }, { "time": "2024-04-23T10:31:21.462675+0000", "event": "failed to xlock, waiting" }, { "time": "2024-04-23T10:31:21.471011+0000", "event": "acquired locks" }, { "time": "2024-04-23T10:31:21.507174+0000", "event": "quiesce complete" } ], "locks": [ { "object": { "is_auth": true, "auth_state": { "replicas": { "1": 1, "2": 1 } }, "replica_state": { "authority": [ 0, -2 ], "replica_nonce": 0 }, "auth_pins": 3, "is_frozen": false, "is_freezing": false, "pins": { "request": 1, "lock": 1, "dirfrag": 1, "caps": 1, "dirtyparent": 0, "replicated": 1, "dirty": 0, "waiter": 0, "authpin": 1 }, "nref": 6 }, "object_string": "[inode 0x1000000023e [...7,head] /volumes/qa/sv_1/b206794e-7622-4ab8-8943-f3ae4e65b7a3/client.0/tmp/clients/client0/~dmtmp/ACCESS/ auth{1=1,2=1} v45582 ap=3 f(v0 m2024-04-23T10:31:17.039732+0000 4=4+0) n(v0 rc2024-04-23T10:31:17.039732+0000 b363650 5=4+1) old_inodes=4 (inest mix) (iquiesce lock x=1 by request(mds.0:174 nref=3)) caps={15312=pAsLsXsFs/pAsLsXsFsxcral@2942},l=15312 | request=1 lock=1 dirfrag=1 caps=1 dirtyparent=0 replicated=1 dirty=0 waiter=0 authpin=1 0x5571cf56a680]", "lock": { "gather_set": [], "state": "lock", "type": "iquiesce", "is_leased": false, "num_rdlocks": 0, "num_wrlocks": 0, "num_xlocks": 1, "xlock_by": { "reqid": { "entity": { "type": "mds", "num": 0 }, "tid": 174 } } }, "flags": 4, "wrlock_target": -1 } ] } }, { "description": "internal op quiesce_inode:mds.0:199 fp=#0x10000000219", "initiated_at": "2024-04-23T10:31:21.483505+0000", "age": 42.400884478000002, "duration": 42.403757959000004, "continuous": true, "type_data": { "result": -2147483648, "flag_point": "quiesce complete for non-auth inode", "reqid": { "entity": { "type": "mds", "num": 0 }, "tid": 199 }, "op_type": "internal_op", "internal_op": 5385, "op_name": "quiesce_inode", "events": [ { "time": "2024-04-23T10:31:21.483505+0000", "event": "initiated" }, { "time": "2024-04-23T10:31:21.483505+0000", "event": "header_read" }, { "time": "2024-04-23T10:31:21.483505+0000", "event": "throttled" }, { "time": "2024-04-23T10:31:21.483505+0000", "event": "all_read" }, { "time": "2024-04-23T10:31:21.483505+0000", "event": "dispatched" }, { "time": "2024-04-23T10:31:21.484047+0000", "event": "requesting remote authpins" }, { "time": "2024-04-23T10:31:21.508085+0000", "event": "acquired locks" }, { "time": "2024-04-23T10:31:21.508095+0000", "event": "quiesce complete for non-auth inode" } ], "locks": [ { "object": { "is_auth": false, "auth_state": { "replicas": {} }, "replica_state": { "authority": [ 2, -2 ], "replica_nonce": 1 }, "auth_pins": 0, "is_frozen": false, "is_freezing": false, "pins": { "request": 1, "lock": 1, "caps": 0, "exportingcaps": 0, "dirtyparent": 0, "dirty": 0, "authpin": 0, "tempexporting": 0 }, "nref": 2 }, "object_string": "[inode 0x10000000219 [7,head] /volumes/qa/sv_1/b206794e-7622-4ab8-8943-f3ae4e65b7a3/client.0/tmp/clients/client0/~dmtmp/COREL/GRAPHIC1.CDR rep@2.1 v7244 s=260096 n(v0 rc2024-04-23T10:27:52.019396+0000 b260096 1=1+0)/n(v0 1=1+0) (iquiesce lock x=1 by request(mds.0:199 nref=3)) | request=1 lock=1 caps=0 exportingcaps=0 dirtyparent=0 dirty=0 authpin=0 tempexporting=0 0x5571cf561080]", "lock": { "gather_set": [], "state": "lock", "type": "iquiesce", "is_leased": false, "num_rdlocks": 0, "num_wrlocks": 0, "num_xlocks": 1, "xlock_by": { "reqid": { "entity": { "type": "mds", "num": 0 }, "tid": 199 } } }, "flags": 4, "wrlock_target": -1 } ] } }, { "description": "peer_request:mds.2:110", "initiated_at": "2024-04-23T10:31:21.448540+0000", "age": 42.435849642000001, "duration": 42.438782058999998, "continuous": false, "type_data": { "result": -2147483648, "flag_point": "dispatched", "reqid": { "entity": { "type": "mds", "num": 2 }, "tid": 110 }, "op_type": "peer_request", "leader_info": { "leader": "2" }, "events": [ { "time": "2024-04-23T10:31:21.448540+0000", "event": "initiated" }, { "time": "2024-04-23T10:31:21.448540+0000", "event": "header_read" }, { "time": "2024-04-23T10:31:21.448541+0000", "event": "throttled" }, { "time": "2024-04-23T10:31:21.448549+0000", "event": "all_read" }, { "time": "2024-04-23T10:31:21.449238+0000", "event": "dispatched" } ], "locks": [] } }, { "description": "peer_request:mds.2:116", "initiated_at": "2024-04-23T10:31:21.457073+0000", "age": 42.427316136999998, "duration": 42.430268882999997, "continuous": false, "type_data": { "result": -2147483648, "flag_point": "dispatched", "reqid": { "entity": { "type": "mds", "num": 2 }, "tid": 116 }, "op_type": "peer_request", "leader_info": { "leader": "2" }, "events": [ { "time": "2024-04-23T10:31:21.457073+0000", "event": "initiated" }, { "time": "2024-04-23T10:31:21.457073+0000", "event": "header_read" }, { "time": "2024-04-23T10:31:21.457074+0000", "event": "throttled" }, { "time": "2024-04-23T10:31:21.457080+0000", "event": "all_read" }, { "time": "2024-04-23T10:31:21.462839+0000", "event": "dispatched" } ], "locks": [] } }, { "description": "internal op quiesce_inode:mds.0:200 fp=#0x10000000216", "initiated_at": "2024-04-23T10:31:21.483518+0000", "age": 42.400871037999998, "duration": 42.403846166000001, "continuous": true, "type_data": { "result": -2147483648, "flag_point": "quiesce complete for non-auth inode", "reqid": { "entity": { "type": "mds", "num": 0 }, "tid": 200 }, "op_type": "internal_op", "internal_op": 5385, "op_name": "quiesce_inode", "events": [ { "time": "2024-04-23T10:31:21.483518+0000", "event": "initiated" }, { "time": "2024-04-23T10:31:21.483518+0000", "event": "header_read" }, { "time": "2024-04-23T10:31:21.483518+0000", "event": "throttled" }, { "time": "2024-04-23T10:31:21.483518+0000", "event": "all_read" }, { "time": "2024-04-23T10:31:21.483518+0000", "event": "dispatched" }, { "time": "2024-04-23T10:31:21.484100+0000", "event": "requesting remote authpins" }, { "time": "2024-04-23T10:31:21.508178+0000", "event": "acquired locks" }, { "time": "2024-04-23T10:31:21.508187+0000", "event": "quiesce complete for non-auth inode" } ], "locks": [ { "object": { "is_auth": false, "auth_state": { "replicas": {} }, "replica_state": { "authority": [ 2, -2 ], "replica_nonce": 1 }, "auth_pins": 0, "is_frozen": false, "is_freezing": false, "pins": { "request": 1, "lock": 1, "caps": 0, "exportingcaps": 0, "dirtyparent": 0, "dirty": 0, "authpin": 0, "tempexporting": 0 }, "nref": 2 }, "object_string": "[inode 0x10000000216 [7,head] /volumes/qa/sv_1/b206794e-7622-4ab8-8943-f3ae4e65b7a3/client.0/tmp/clients/client0/~dmtmp/COREL/CDRROLS.CFG rep@2.1 v7238 s=260096 n(v0 rc2024-04-23T10:29:16.539476+0000 b260096 1=1+0)/n(v0 1=1+0) (iquiesce lock x=1 by request(mds.0:200 nref=3)) | request=1 lock=1 caps=0 exportingcaps=0 dirtyparent=0 dirty=0 authpin=0 tempexporting=0 0x5571cf560000]", "lock": { "gather_set": [], "state": "lock", "type": "iquiesce", "is_leased": false, "num_rdlocks": 0, "num_wrlocks": 0, "num_xlocks": 1, "xlock_by": { "reqid": { "entity": { "type": "mds", "num": 0 }, "tid": 200 } } }, "flags": 4, "wrlock_target": -1 } ] } }, { "description": "peer_request:mds.1:106", "initiated_at": "2024-04-23T10:31:21.448440+0000", "age": 42.435949223000001, "duration": 42.438983038000003, "continuous": false, "type_data": { "result": -2147483648, "flag_point": "dispatched", "reqid": { "entity": { "type": "mds", "num": 1 }, "tid": 106 }, "op_type": "peer_request", "leader_info": { "leader": "1" }, "events": [ { "time": "2024-04-23T10:31:21.448440+0000", "event": "initiated" }, { "time": "2024-04-23T10:31:21.448440+0000", "event": "header_read" }, { "time": "2024-04-23T10:31:21.448441+0000", "event": "throttled" }, { "time": "2024-04-23T10:31:21.448454+0000", "event": "all_read" }, { "time": "2024-04-23T10:31:21.449319+0000", "event": "dispatched" } ], "locks": [] } }, { "description": "internal op quiesce_inode:mds.0:175 fp=#0x1000000020d", "initiated_at": "2024-04-23T10:31:21.463518+0000", "age": 42.420870845000003, "duration": 42.423949573000002, "continuous": true, "type_data": { "result": -2147483648, "flag_point": "quiesce complete", "reqid": { "entity": { "type": "mds", "num": 0 }, "tid": 175 }, "op_type": "internal_op", "internal_op": 5385, "op_name": "quiesce_inode", "events": [ { "time": "2024-04-23T10:31:21.463518+0000", "event": "initiated" }, { "time": "2024-04-23T10:31:21.463518+0000", "event": "header_read" }, { "time": "2024-04-23T10:31:21.463518+0000", "event": "throttled" }, { "time": "2024-04-23T10:31:21.463518+0000", "event": "all_read" }, { "time": "2024-04-23T10:31:21.463518+0000", "event": "dispatched" }, { "time": "2024-04-23T10:31:21.463813+0000", "event": "failed to xlock, waiting" }, { "time": "2024-04-23T10:31:21.474631+0000", "event": "failed to xlock, waiting" }, { "time": "2024-04-23T10:31:21.485239+0000", "event": "acquired locks" }, { "time": "2024-04-23T10:31:21.485479+0000", "event": "quiesce complete" } ], "locks": [ { "object": { "is_auth": true, "auth_state": { "replicas": {} }, "replica_state": { "authority": [ 0, -2 ], "replica_nonce": 0 }, "auth_pins": 1, "is_frozen": false, "is_freezing": false, "pins": { "ptrwaiter": 0, "request": 1, "lock": 1, "caps": 0, "needsnapflush": 0, "dirtyparent": 0, "dirty": 0, "waiter": 0, "authpin": 1 }, "nref": 3 }, "object_string": "[inode 0x1000000020d [7,head] /volumes/qa/sv_1/b206794e-7622-4ab8-8943-f3ae4e65b7a3/client.0/tmp/clients/client0/~dmtmp/EXCEL/PCMAGCD.XLS auth v8692 ap=1 s=260096 n(v0 rc2024-04-23T10:31:16.585743+0000 b260096 1=1+0) (iquiesce lock x=1 by request(mds.0:175 nref=3)) | ptrwaiter=0 request=1 lock=1 caps=0 needsnapflush=0 dirtyparent=0 dirty=0 waiter=0 authpin=1 0x5571cf53eb00]", "lock": { "gather_set": [], "state": "lock", "type": "iquiesce", "is_leased": false, "num_rdlocks": 0, "num_wrlocks": 0, "num_xlocks": 1, "xlock_by": { "reqid": { "entity": { "type": "mds", "num": 0 }, "tid": 175 } } }, "flags": 4, "wrlock_target": -1 } ] } }, { "description": "peer_request:mds.1:115", "initiated_at": "2024-04-23T10:31:21.464762+0000", "age": 42.419627384999998, "duration": 42.422772530000003, "continuous": false, "type_data": { "result": -2147483648, "flag_point": "dispatched", "reqid": { "entity": { "type": "mds", "num": 1 }, "tid": 115 }, "op_type": "peer_request", "leader_info": { "leader": "1" }, "events": [ { "time": "2024-04-23T10:31:21.464762+0000", "event": "initiated" }, { "time": "2024-04-23T10:31:21.464762+0000", "event": "header_read" }, { "time": "2024-04-23T10:31:21.464762+0000", "event": "throttled" }, { "time": "2024-04-23T10:31:21.464767+0000", "event": "all_read" }, { "time": "2024-04-23T10:31:21.481469+0000", "event": "dispatched" } ], "locks": [] } }, { "description": "internal op quiesce_inode:mds.0:155 fp=#0x100000001fb", "initiated_at": "2024-04-23T10:31:21.449954+0000", "age": 42.434434795000001, "duration": 42.437603787, "continuous": true, "type_data": { "result": -2147483648, "flag_point": "quiesce complete", "reqid": { "entity": { "type": "mds", "num": 0 }, "tid": 155 }, "op_type": "internal_op", "internal_op": 5385, "op_name": "quiesce_inode", "events": [ { "time": "2024-04-23T10:31:21.449954+0000", "event": "initiated" }, { "time": "2024-04-23T10:31:21.449954+0000", "event": "header_read" }, { "time": "2024-04-23T10:31:21.449954+0000", "event": "throttled" }, { "time": "2024-04-23T10:31:21.449954+0000", "event": "all_read" }, { "time": "2024-04-23T10:31:21.449954+0000", "event": "dispatched" }, { "time": "2024-04-23T10:31:21.450125+0000", "event": "failed to xlock, waiting" }, { "time": "2024-04-23T10:31:21.451103+0000", "event": "acquired locks" }, { "time": "2024-04-23T10:31:21.509964+0000", "event": "quiesce complete" } ], "locks": [ { "object": { "is_auth": true, "auth_state": { "replicas": { "1": 1, "2": 1 } }, "replica_state": { "authority": [ 0, -2 ], "replica_nonce": 0 }, "auth_pins": 3, "is_frozen": false, "is_freezing": false, "pins": { "dirtyscattered": 1, "request": 1, "lock": 1, "dirfrag": 1, "caps": 1, "dirtyrstat": 1, "dirtyparent": 0, "replicated": 1, "dirty": 1, "waiter": 0, "authpin": 1 }, "nref": 9 }, "object_string": "[inode 0x100000001fb [...7,head] /volumes/qa/sv_1/b206794e-7622-4ab8-8943-f3ae4e65b7a3/ auth{1=1,2=1} v426 ap=3 f(v0 m2024-04-23T10:26:48.475842+0000 1=0+1) n(v25 rc2024-04-23T10:31:17.764716+0000 b21075074 78=63+15)/n(v24 rc2024-04-23T10:31:17.764716+0000 b21075074 78=63+15) old_inodes=5 (inest mix dirty) (iquiesce lock x=1 by request(mds.0:155 nref=3)) caps={15312=pAsLsXs/-@18},l=15312 | dirtyscattered=1 request=1 lock=1 dirfrag=1 caps=1 dirtyrstat=1 dirtyparent=0 replicated=1 dirty=1 waiter=0 authpin=1 0x5571cf1e0100]", "lock": { "gather_set": [], "state": "lock", "type": "iquiesce", "is_leased": false, "num_rdlocks": 0, "num_wrlocks": 0, "num_xlocks": 1, "xlock_by": { "reqid": { "entity": { "type": "mds", "num": 0 }, "tid": 155 } } }, "flags": 4, "wrlock_target": -1 } ] } }, { "description": "internal op quiesce_inode:mds.0:176 fp=#0x1000000020e", "initiated_at": "2024-04-23T10:31:21.463538+0000", "age": 42.420850881, "duration": 42.424124407000001, "continuous": true, "type_data": { "result": -2147483648, "flag_point": "quiesce complete", "reqid": { "entity": { "type": "mds", "num": 0 }, "tid": 176 }, "op_type": "internal_op", "internal_op": 5385, "op_name": "quiesce_inode", "events": [ { "time": "2024-04-23T10:31:21.463538+0000", "event": "initiated" }, { "time": "2024-04-23T10:31:21.463538+0000", "event": "header_read" }, { "time": "2024-04-23T10:31:21.463538+0000", "event": "throttled" }, { "time": "2024-04-23T10:31:21.463538+0000", "event": "all_read" }, { "time": "2024-04-23T10:31:21.463538+0000", "event": "dispatched" }, { "time": "2024-04-23T10:31:21.464062+0000", "event": "failed to xlock, waiting" }, { "time": "2024-04-23T10:31:21.476699+0000", "event": "failed to xlock, waiting" }, { "time": "2024-04-23T10:31:21.488211+0000", "event": "acquired locks" }, { "time": "2024-04-23T10:31:21.488419+0000", "event": "quiesce complete" } ], "locks": [ { "object": { "is_auth": true, "auth_state": { "replicas": {} }, "replica_state": { "authority": [ 0, -2 ], "replica_nonce": 0 }, "auth_pins": 1, "is_frozen": false, "is_freezing": false, "pins": { "ptrwaiter": 0, "request": 1, "lock": 1, "caps": 0, "needsnapflush": 0, "dirtyparent": 0, "dirty": 0, "waiter": 0, "authpin": 1 }, "nref": 3 }, "object_string": "[inode 0x1000000020e [7,head] /volumes/qa/sv_1/b206794e-7622-4ab8-8943-f3ae4e65b7a3/client.0/tmp/clients/client0/~dmtmp/EXCEL/SALES.XLS auth v8694 ap=1 s=260096 n(v0 rc2024-04-23T10:31:16.692740+0000 b260096 1=1+0) (iquiesce lock x=1 by request(mds.0:176 nref=3)) | ptrwaiter=0 request=1 lock=1 caps=0 needsnapflush=0 dirtyparent=0 dirty=0 waiter=0 authpin=1 0x5571cf53f080]", "lock": { "gather_set": [], "state": "lock", "type": "iquiesce", "is_leased": false, "num_rdlocks": 0, "num_wrlocks": 0, "num_xlocks": 1, "xlock_by": { "reqid": { "entity": { "type": "mds", "num": 0 }, "tid": 176 } } }, "flags": 4, "wrlock_target": -1 } ] } }, { "description": "peer_request:mds.2:120", "initiated_at": "2024-04-23T10:31:21.463567+0000", "age": 42.420821902999997, "duration": 42.424164765, "continuous": false, "type_data": { "result": -2147483648, "flag_point": "dispatched", "reqid": { "entity": { "type": "mds", "num": 2 }, "tid": 120 }, "op_type": "peer_request", "leader_info": { "leader": "2" }, "events": [ { "time": "2024-04-23T10:31:21.463567+0000", "event": "initiated" }, { "time": "2024-04-23T10:31:21.463567+0000", "event": "header_read" }, { "time": "2024-04-23T10:31:21.463568+0000", "event": "throttled" }, { "time": "2024-04-23T10:31:21.463573+0000", "event": "all_read" }, { "time": "2024-04-23T10:31:21.484620+0000", "event": "dispatched" } ], "locks": [] } }, { "description": "peer_request:mds.2:111", "initiated_at": "2024-04-23T10:31:21.449682+0000", "age": 42.434707566, "duration": 42.438100112999997, "continuous": false, "type_data": { "result": -2147483648, "flag_point": "dispatched", "reqid": { "entity": { "type": "mds", "num": 2 }, "tid": 111 }, "op_type": "peer_request", "leader_info": { "leader": "2" }, "events": [ { "time": "2024-04-23T10:31:21.449682+0000", "event": "initiated" }, { "time": "2024-04-23T10:31:21.449682+0000", "event": "header_read" }, { "time": "2024-04-23T10:31:21.449682+0000", "event": "throttled" }, { "time": "2024-04-23T10:31:21.449689+0000", "event": "all_read" }, { "time": "2024-04-23T10:31:21.450602+0000", "event": "dispatched" } ], "locks": [] } }, { "description": "peer_request:mds.1:112", "initiated_at": "2024-04-23T10:31:21.457115+0000", "age": 42.427273952999997, "duration": 42.430707071999997, "continuous": false, "type_data": { "result": -2147483648, "flag_point": "dispatched", "reqid": { "entity": { "type": "mds", "num": 1 }, "tid": 112 }, "op_type": "peer_request", "leader_info": { "leader": "1" }, "events": [ { "time": "2024-04-23T10:31:21.457115+0000", "event": "initiated" }, { "time": "2024-04-23T10:31:21.457115+0000", "event": "header_read" }, { "time": "2024-04-23T10:31:21.457116+0000", "event": "throttled" }, { "time": "2024-04-23T10:31:21.457124+0000", "event": "all_read" }, { "time": "2024-04-23T10:31:21.464095+0000", "event": "dispatched" } ], "locks": [] } }, { "description": "peer_request:mds.1:116", "initiated_at": "2024-04-23T10:31:21.464842+0000", "age": 42.419547631, "duration": 42.423019656000001, "continuous": false, "type_data": { "result": -2147483648, "flag_point": "dispatched", "reqid": { "entity": { "type": "mds", "num": 1 }, "tid": 116 }, "op_type": "peer_request", "leader_info": { "leader": "1" }, "events": [ { "time": "2024-04-23T10:31:21.464842+0000", "event": "initiated" }, { "time": "2024-04-23T10:31:21.464842+0000", "event": "header_read" }, { "time": "2024-04-23T10:31:21.464842+0000", "event": "throttled" }, { "time": "2024-04-23T10:31:21.464847+0000", "event": "all_read" }, { "time": "2024-04-23T10:31:21.487773+0000", "event": "dispatched" } ], "locks": [] } }, { "description": "peer_request:mds.1:107", "initiated_at": "2024-04-23T10:31:21.449527+0000", "age": 42.434862643000002, "duration": 42.438378360000002, "continuous": false, "type_data": { "result": -2147483648, "flag_point": "dispatched", "reqid": { "entity": { "type": "mds", "num": 1 }, "tid": 107 }, "op_type": "peer_request", "leader_info": { "leader": "1" }, "events": [ { "time": "2024-04-23T10:31:21.449527+0000", "event": "initiated" }, { "time": "2024-04-23T10:31:21.449527+0000", "event": "header_read" }, { "time": "2024-04-23T10:31:21.449527+0000", "event": "throttled" }, { "time": "2024-04-23T10:31:21.449535+0000", "event": "all_read" }, { "time": "2024-04-23T10:31:21.450675+0000", "event": "dispatched" } ], "locks": [] } }, { "description": "internal op quiesce_inode:mds.0:177 fp=#0x10000000222", "initiated_at": "2024-04-23T10:31:21.467357+0000", "age": 42.417032163000002, "duration": 42.420584648999998, "continuous": true, "type_data": { "result": -2147483648, "flag_point": "quiesce complete", "reqid": { "entity": { "type": "mds", "num": 0 }, "tid": 177 }, "op_type": "internal_op", "internal_op": 5385, "op_name": "quiesce_inode", "events": [ { "time": "2024-04-23T10:31:21.467357+0000", "event": "initiated" }, { "time": "2024-04-23T10:31:21.467357+0000", "event": "header_read" }, { "time": "2024-04-23T10:31:21.467357+0000", "event": "throttled" }, { "time": "2024-04-23T10:31:21.467357+0000", "event": "all_read" }, { "time": "2024-04-23T10:31:21.467357+0000", "event": "dispatched" }, { "time": "2024-04-23T10:31:21.467553+0000", "event": "failed to xlock, waiting" }, { "time": "2024-04-23T10:31:21.478020+0000", "event": "acquired locks" }, { "time": "2024-04-23T10:31:21.478129+0000", "event": "quiesce complete" } ], "locks": [ { "object": { "is_auth": true, "auth_state": { "replicas": {} }, "replica_state": { "authority": [ 0, -2 ], "replica_nonce": 0 }, "auth_pins": 1, "is_frozen": false, "is_freezing": false, "pins": { "ptrwaiter": 0, "request": 1, "lock": 1, "caps": 0, "needsnapflush": 0, "dirtyparent": 0, "dirty": 0, "waiter": 0, "authpin": 1 }, "nref": 3 }, "object_string": "[inode 0x10000000222 [7,head] /volumes/qa/sv_1/b206794e-7622-4ab8-8943-f3ae4e65b7a3/client.0/tmp/clients/client0/~dmtmp/WORDPRO/BENCHS.LWP auth v8110 ap=1 s=260096 n(v0 rc2024-04-23T10:27:52.058395+0000 b260096 1=1+0) (iquiesce lock x=1 by request(mds.0:177 nref=3)) | ptrwaiter=0 request=1 lock=1 caps=0 needsnapflush=0 dirtyparent=0 dirty=0 waiter=0 authpin=1 0x5571cf586000]", "lock": { "gather_set": [], "state": "lock", "type": "iquiesce", "is_leased": false, "num_rdlocks": 0, "num_wrlocks": 0, "num_xlocks": 1, "xlock_by": { "reqid": { "entity": { "type": "mds", "num": 0 }, "tid": 177 } } }, "flags": 4, "wrlock_target": -1 } ] } }, { "description": "peer_request:mds.2:121", "initiated_at": "2024-04-23T10:31:21.463630+0000", "age": 42.420758773999999, "duration": 42.424385663999999, "continuous": false, "type_data": { "result": -2147483648, "flag_point": "dispatched", "reqid": { "entity": { "type": "mds", "num": 2 }, "tid": 121 }, "op_type": "peer_request", "leader_info": { "leader": "2" }, "events": [ { "time": "2024-04-23T10:31:21.463630+0000", "event": "initiated" }, { "time": "2024-04-23T10:31:21.463630+0000", "event": "header_read" }, { "time": "2024-04-23T10:31:21.463631+0000", "event": "throttled" }, { "time": "2024-04-23T10:31:21.463636+0000", "event": "all_read" }, { "time": "2024-04-23T10:31:21.488755+0000", "event": "dispatched" } ], "locks": [] } }, { "description": "internal op quiesce_inode:mds.0:156 fp=#0x10000000005", "initiated_at": "2024-04-23T10:31:21.451325+0000", "age": 42.433063761, "duration": 42.436722033000002, "continuous": true, "type_data": { "result": -2147483648, "flag_point": "quiesce complete", "reqid": { "entity": { "type": "mds", "num": 0 }, "tid": 156 }, "op_type": "internal_op", "internal_op": 5385, "op_name": "quiesce_inode", "events": [ { "time": "2024-04-23T10:31:21.451325+0000", "event": "initiated" }, { "time": "2024-04-23T10:31:21.451325+0000", "event": "header_read" }, { "time": "2024-04-23T10:31:21.451325+0000", "event": "throttled" }, { "time": "2024-04-23T10:31:21.451325+0000", "event": "all_read" }, { "time": "2024-04-23T10:31:21.451325+0000", "event": "dispatched" }, { "time": "2024-04-23T10:31:21.451535+0000", "event": "failed to xlock, waiting" }, { "time": "2024-04-23T10:31:21.452316+0000", "event": "acquired locks" }, { "time": "2024-04-23T10:31:21.509937+0000", "event": "quiesce complete" } ], "locks": [ { "object": { "is_auth": true, "auth_state": { "replicas": { "1": 1, "2": 1 } }, "replica_state": { "authority": [ 0, -2 ], "replica_nonce": 0 }, "auth_pins": 3, "is_frozen": false, "is_freezing": false, "pins": { "ptrwaiter": 0, "dirtyscattered": 1, "request": 1, "lock": 1, "dirfrag": 1, "caps": 1, "dirtyrstat": 0, "dirtyparent": 0, "replicated": 1, "dirty": 1, "waiter": 0, "authpin": 1 }, "nref": 8 }, "object_string": "[inode 0x10000000005 [...7,head] /volumes/qa/sv_1/b206794e-7622-4ab8-8943-f3ae4e65b7a3/client.0/ auth{1=1,2=1} v423 ap=3 f(v0 m2024-04-23T10:27:42.521612+0000 1=0+1) n(v32 rc2024-04-23T10:31:17.764716+0000 b21075074 77=63+14) old_inodes=5 (inest mix dirty) (iquiesce lock x=1 by request(mds.0:156 nref=3)) caps={15312=pAsLsXs/-@3},l=15312 | ptrwaiter=0 dirtyscattered=1 request=1 lock=1 dirfrag=1 caps=1 dirtyrstat=0 dirtyparent=0 replicated=1 dirty=1 waiter=0 authpin=1 0x5571cf340b00]", "lock": { "gather_set": [], "state": "lock", "type": "iquiesce", "is_leased": false, "num_rdlocks": 0, "num_wrlocks": 0, "num_xlocks": 1, "xlock_by": { "reqid": { "entity": { "type": "mds", "num": 0 }, "tid": 156 } } }, "flags": 4, "wrlock_target": -1 } ] } }, { "description": "internal op quiesce_inode:mds.0:178 fp=#0x10000000223", "initiated_at": "2024-04-23T10:31:21.467375+0000", "age": 42.417014410999997, "duration": 42.420749837999999, "continuous": true, "type_data": { "result": -2147483648, "flag_point": "quiesce complete", "reqid": { "entity": { "type": "mds", "num": 0 }, "tid": 178 }, "op_type": "internal_op", "internal_op": 5385, "op_name": "quiesce_inode", "events": [ { "time": "2024-04-23T10:31:21.467375+0000", "event": "initiated" }, { "time": "2024-04-23T10:31:21.467375+0000", "event": "header_read" }, { "time": "2024-04-23T10:31:21.467375+0000", "event": "throttled" }, { "time": "2024-04-23T10:31:21.467375+0000", "event": "all_read" }, { "time": "2024-04-23T10:31:21.467375+0000", "event": "dispatched" }, { "time": "2024-04-23T10:31:21.467706+0000", "event": "acquired locks" }, { "time": "2024-04-23T10:31:21.467818+0000", "event": "quiesce complete" } ], "locks": [ { "object": { "is_auth": true, "auth_state": { "replicas": {} }, "replica_state": { "authority": [ 0, -2 ], "replica_nonce": 0 }, "auth_pins": 1, "is_frozen": false, "is_freezing": false, "pins": { "ptrwaiter": 0, "request": 1, "lock": 1, "caps": 0, "needsnapflush": 0, "dirtyparent": 0, "dirty": 0, "waiter": 0, "authpin": 1 }, "nref": 3 }, "object_string": "[inode 0x10000000223 [7,head] /volumes/qa/sv_1/b206794e-7622-4ab8-8943-f3ae4e65b7a3/client.0/tmp/clients/client0/~dmtmp/WORDPRO/RESULTS.XLS auth v8108 ap=1 s=260096 n(v0 rc2024-04-23T10:27:52.063395+0000 b260096 1=1+0) (iquiesce lock x=1 by request(mds.0:178 nref=3)) | ptrwaiter=0 request=1 lock=1 caps=0 needsnapflush=0 dirtyparent=0 dirty=0 waiter=0 authpin=1 0x5571cf586580]", "lock": { "gather_set": [], "state": "lock", "type": "iquiesce", "is_leased": false, "num_rdlocks": 0, "num_wrlocks": 0, "num_xlocks": 1, "xlock_by": { "reqid": { "entity": { "type": "mds", "num": 0 }, "tid": 178 } } }, "flags": 4, "wrlock_target": -1 } ] } }, { "description": "peer_request:mds.1:117", "initiated_at": "2024-04-23T10:31:21.464907+0000", "age": 42.419482221000003, "duration": 42.423281594999999, "continuous": false, "type_data": { "result": -2147483648, "flag_point": "dispatched", "reqid": { "entity": { "type": "mds", "num": 1 }, "tid": 117 }, "op_type": "peer_request", "leader_info": { "leader": "1" }, "events": [ { "time": "2024-04-23T10:31:21.464907+0000", "event": "initiated" }, { "time": "2024-04-23T10:31:21.464907+0000", "event": "header_read" }, { "time": "2024-04-23T10:31:21.464907+0000", "event": "throttled" }, { "time": "2024-04-23T10:31:21.464912+0000", "event": "all_read" }, { "time": "2024-04-23T10:31:21.492009+0000", "event": "dispatched" } ], "locks": [] } }, { "description": "peer_request:mds.2:112", "initiated_at": "2024-04-23T10:31:21.450958+0000", "age": 42.433431548999998, "duration": 42.437252534999999, "continuous": false, "type_data": { "result": -2147483648, "flag_point": "dispatched", "reqid": { "entity": { "type": "mds", "num": 2 }, "tid": 112 }, "op_type": "peer_request", "leader_info": { "leader": "2" }, "events": [ { "time": "2024-04-23T10:31:21.450958+0000", "event": "initiated" }, { "time": "2024-04-23T10:31:21.450958+0000", "event": "header_read" }, { "time": "2024-04-23T10:31:21.450958+0000", "event": "throttled" }, { "time": "2024-04-23T10:31:21.450965+0000", "event": "all_read" }, { "time": "2024-04-23T10:31:21.451700+0000", "event": "dispatched" } ], "locks": [] } }, { "description": "internal op quiesce_inode:mds.0:179 fp=#0x10000000224", "initiated_at": "2024-04-23T10:31:21.467394+0000", "age": 42.416995122000003, "duration": 42.420840339000002, "continuous": true, "type_data": { "result": -2147483648, "flag_point": "quiesce complete", "reqid": { "entity": { "type": "mds", "num": 0 }, "tid": 179 }, "op_type": "internal_op", "internal_op": 5385, "op_name": "quiesce_inode", "events": [ { "time": "2024-04-23T10:31:21.467394+0000", "event": "initiated" }, { "time": "2024-04-23T10:31:21.467394+0000", "event": "header_read" }, { "time": "2024-04-23T10:31:21.467394+0000", "event": "throttled" }, { "time": "2024-04-23T10:31:21.467394+0000", "event": "all_read" }, { "time": "2024-04-23T10:31:21.467394+0000", "event": "dispatched" }, { "time": "2024-04-23T10:31:21.467972+0000", "event": "failed to xlock, waiting" }, { "time": "2024-04-23T10:31:21.479506+0000", "event": "acquired locks" }, { "time": "2024-04-23T10:31:21.479719+0000", "event": "quiesce complete" } ], "locks": [ { "object": { "is_auth": true, "auth_state": { "replicas": {} }, "replica_state": { "authority": [ 0, -2 ], "replica_nonce": 0 }, "auth_pins": 1, "is_frozen": false, "is_freezing": false, "pins": { "ptrwaiter": 0, "request": 1, "lock": 1, "caps": 0, "needsnapflush": 0, "dirtyparent": 0, "dirty": 0, "waiter": 0, "authpin": 1 }, "nref": 3 }, "object_string": "[inode 0x10000000224 [7,head] /volumes/qa/sv_1/b206794e-7622-4ab8-8943-f3ae4e65b7a3/client.0/tmp/clients/client0/~dmtmp/WORDPRO/NEWS1_1.LWP auth v8112 ap=1 s=260096 n(v0 rc2024-04-23T10:27:52.068395+0000 b260096 1=1+0) (iquiesce lock x=1 by request(mds.0:179 nref=3)) | ptrwaiter=0 request=1 lock=1 caps=0 needsnapflush=0 dirtyparent=0 dirty=0 waiter=0 authpin=1 0x5571cf586b00]", "lock": { "gather_set": [], "state": "lock", "type": "iquiesce", "is_leased": false, "num_rdlocks": 0, "num_wrlocks": 0, "num_xlocks": 1, "xlock_by": { "reqid": { "entity": { "type": "mds", "num": 0 }, "tid": 179 } } }, "flags": 4, "wrlock_target": -1 } ] } }, { "description": "peer_request:mds.2:122", "initiated_at": "2024-04-23T10:31:21.463703+0000", "age": 42.420686615999998, "duration": 42.424595052999997, "continuous": false, "type_data": { "result": -2147483648, "flag_point": "dispatched", "reqid": { "entity": { "type": "mds", "num": 2 }, "tid": 122 }, "op_type": "peer_request", "leader_info": { "leader": "2" }, "events": [ { "time": "2024-04-23T10:31:21.463703+0000", "event": "initiated" }, { "time": "2024-04-23T10:31:21.463703+0000", "event": "header_read" }, { "time": "2024-04-23T10:31:21.463703+0000", "event": "throttled" }, { "time": "2024-04-23T10:31:21.463708+0000", "event": "all_read" }, { "time": "2024-04-23T10:31:21.492926+0000", "event": "dispatched" } ], "locks": [] } }, { "description": "peer_request:mds.1:108", "initiated_at": "2024-04-23T10:31:21.450989+0000", "age": 42.433400542000001, "duration": 42.437330357, "continuous": false, "type_data": { "result": -2147483648, "flag_point": "dispatched", "reqid": { "entity": { "type": "mds", "num": 1 }, "tid": 108 }, "op_type": "peer_request", "leader_info": { "leader": "1" }, "events": [ { "time": "2024-04-23T10:31:21.450989+0000", "event": "initiated" }, { "time": "2024-04-23T10:31:21.450989+0000", "event": "header_read" }, { "time": "2024-04-23T10:31:21.450990+0000", "event": "throttled" }, { "time": "2024-04-23T10:31:21.450998+0000", "event": "all_read" }, { "time": "2024-04-23T10:31:21.451852+0000", "event": "dispatched" } ], "locks": [] } }, { "description": "internal op quiesce_inode:mds.0:180 fp=#0x1000000020a", "initiated_at": "2024-04-23T10:31:21.468566+0000", "age": 42.415823480999997, "duration": 42.419776714999998, "continuous": true, "type_data": { "result": -2147483648, "flag_point": "quiesce complete", "reqid": { "entity": { "type": "mds", "num": 0 }, "tid": 180 }, "op_type": "internal_op", "internal_op": 5385, "op_name": "quiesce_inode", "events": [ { "time": "2024-04-23T10:31:21.468566+0000", "event": "initiated" }, { "time": "2024-04-23T10:31:21.468566+0000", "event": "header_read" }, { "time": "2024-04-23T10:31:21.468566+0000", "event": "throttled" }, { "time": "2024-04-23T10:31:21.468566+0000", "event": "all_read" }, { "time": "2024-04-23T10:31:21.468566+0000", "event": "dispatched" }, { "time": "2024-04-23T10:31:21.468784+0000", "event": "failed to xlock, waiting" }, { "time": "2024-04-23T10:31:21.481424+0000", "event": "failed to xlock, waiting" }, { "time": "2024-04-23T10:31:21.490419+0000", "event": "acquired locks" }, { "time": "2024-04-23T10:31:21.490640+0000", "event": "quiesce complete" } ], "locks": [ { "object": { "is_auth": true, "auth_state": { "replicas": {} }, "replica_state": { "authority": [ 0, -2 ], "replica_nonce": 0 }, "auth_pins": 1, "is_frozen": false, "is_freezing": false, "pins": { "ptrwaiter": 0, "request": 1, "lock": 1, "caps": 0, "needsnapflush": 0, "dirtyparent": 0, "dirty": 0, "waiter": 0, "authpin": 1 }, "nref": 3 }, "object_string": "[inode 0x1000000020a [6,head] /volumes/qa/sv_1/b206794e-7622-4ab8-8943-f3ae4e65b7a3/client.0/tmp/clients/client0/~dmtmp/WORD/BASEMACH.DOC auth v5584 ap=1 s=260096 n(v0 rc2024-04-23T10:27:51.959398+0000 b260096 1=1+0) (iquiesce lock x=1 by request(mds.0:180 nref=3)) | ptrwaiter=0 request=1 lock=1 caps=0 needsnapflush=0 dirtyparent=0 dirty=0 waiter=0 authpin=1 0x5571cf4d7700]", "lock": { "gather_set": [], "state": "lock", "type": "iquiesce", "is_leased": false, "num_rdlocks": 0, "num_wrlocks": 0, "num_xlocks": 1, "xlock_by": { "reqid": { "entity": { "type": "mds", "num": 0 }, "tid": 180 } } }, "flags": 4, "wrlock_target": -1 } ] } }, { "description": "peer_request:mds.1:118", "initiated_at": "2024-04-23T10:31:21.464992+0000", "age": 42.419397154999999, "duration": 42.423539978000001, "continuous": false, "type_data": { "result": -2147483648, "flag_point": "dispatched", "reqid": { "entity": { "type": "mds", "num": 1 }, "tid": 118 }, "op_type": "peer_request", "leader_info": { "leader": "1" }, "events": [ { "time": "2024-04-23T10:31:21.464992+0000", "event": "initiated" }, { "time": "2024-04-23T10:31:21.464992+0000", "event": "header_read" }, { "time": "2024-04-23T10:31:21.464993+0000", "event": "throttled" }, { "time": "2024-04-23T10:31:21.464998+0000", "event": "all_read" }, { "time": "2024-04-23T10:31:21.493737+0000", "event": "dispatched" } ], "locks": [] } }, { "description": "internal op quiesce_inode:mds.0:157 fp=#0x10000000006", "initiated_at": "2024-04-23T10:31:21.452552+0000", "age": 42.431837444000003, "duration": 42.436003264999997, "continuous": true, "type_data": { "result": -2147483648, "flag_point": "quiesce complete", "reqid": { "entity": { "type": "mds", "num": 0 }, "tid": 157 }, "op_type": "internal_op", "internal_op": 5385, "op_name": "quiesce_inode", "events": [ { "time": "2024-04-23T10:31:21.452552+0000", "event": "initiated" }, { "time": "2024-04-23T10:31:21.452552+0000", "event": "header_read" }, { "time": "2024-04-23T10:31:21.452552+0000", "event": "throttled" }, { "time": "2024-04-23T10:31:21.452552+0000", "event": "all_read" }, { "time": "2024-04-23T10:31:21.452552+0000", "event": "dispatched" }, { "time": "2024-04-23T10:31:21.452764+0000", "event": "failed to xlock, waiting" }, { "time": "2024-04-23T10:31:21.453845+0000", "event": "acquired locks" }, { "time": "2024-04-23T10:31:21.509910+0000", "event": "quiesce complete" } ], "locks": [ { "object": { "is_auth": true, "auth_state": { "replicas": { "1": 1, "2": 1 } }, "replica_state": { "authority": [ 0, -2 ], "replica_nonce": 0 }, "auth_pins": 3, "is_frozen": false, "is_freezing": false, "pins": { "dirtyscattered": 0, "request": 1, "lock": 1, "dirfrag": 1, "caps": 1, "dirtyrstat": 1, "dirtyparent": 0, "replicated": 1, "dirty": 1, "waiter": 0, "authpin": 1 }, "nref": 8 }, "object_string": "[inode 0x10000000006 [...7,head] /volumes/qa/sv_1/b206794e-7622-4ab8-8943-f3ae4e65b7a3/client.0/tmp/ auth{1=1,2=1} v417 ap=3 f(v0 m2024-04-23T10:27:51.919399+0000 1=0+1) n(v30 rc2024-04-23T10:31:17.764716+0000 b21075074 76=63+13) old_inodes=4 (inest mix) (iquiesce lock x=1 by request(mds.0:157 nref=3)) caps={15312=pAsLsXs/-@9},l=15312 | dirtyscattered=0 request=1 lock=1 dirfrag=1 caps=1 dirtyrstat=1 dirtyparent=0 replicated=1 dirty=1 waiter=0 authpin=1 0x5571cf4d6100]", "lock": { "gather_set": [], "state": "lock", "type": "iquiesce", "is_leased": false, "num_rdlocks": 0, "num_wrlocks": 0, "num_xlocks": 1, "xlock_by": { "reqid": { "entity": { "type": "mds", "num": 0 }, "tid": 157 } } }, "flags": 4, "wrlock_target": -1 } ] } }, { "description": "internal op quiesce_inode:mds.0:181 fp=#0x1000000020b", "initiated_at": "2024-04-23T10:31:21.468582+0000", "age": 42.415806846999999, "duration": 42.420045846999997, "continuous": true, "type_data": { "result": -2147483648, "flag_point": "quiesce complete", "reqid": { "entity": { "type": "mds", "num": 0 }, "tid": 181 }, "op_type": "internal_op", "internal_op": 5385, "op_name": "quiesce_inode", "events": [ { "time": "2024-04-23T10:31:21.468582+0000", "event": "initiated" }, { "time": "2024-04-23T10:31:21.468582+0000", "event": "header_read" }, { "time": "2024-04-23T10:31:21.468582+0000", "event": "throttled" }, { "time": "2024-04-23T10:31:21.468582+0000", "event": "all_read" }, { "time": "2024-04-23T10:31:21.468582+0000", "event": "dispatched" }, { "time": "2024-04-23T10:31:21.468953+0000", "event": "failed to xlock, waiting" }, { "time": "2024-04-23T10:31:21.487718+0000", "event": "failed to xlock, waiting" }, { "time": "2024-04-23T10:31:21.492458+0000", "event": "acquired locks" }, { "time": "2024-04-23T10:31:21.492666+0000", "event": "quiesce complete" } ], "locks": [ { "object": { "is_auth": true, "auth_state": { "replicas": {} }, "replica_state": { "authority": [ 0, -2 ], "replica_nonce": 0 }, "auth_pins": 1, "is_frozen": false, "is_freezing": false, "pins": { "ptrwaiter": 0, "request": 1, "lock": 1, "caps": 0, "needsnapflush": 0, "dirtyparent": 0, "dirty": 0, "waiter": 0, "authpin": 1 }, "nref": 3 }, "object_string": "[inode 0x1000000020b [6,head] /volumes/qa/sv_1/b206794e-7622-4ab8-8943-f3ae4e65b7a3/client.0/tmp/clients/client0/~dmtmp/WORD/FACTS.DOC auth v5586 ap=1 s=260096 n(v0 rc2024-04-23T10:27:51.965397+0000 b260096 1=1+0) (iquiesce lock x=1 by request(mds.0:181 nref=3)) | ptrwaiter=0 request=1 lock=1 caps=0 needsnapflush=0 dirtyparent=0 dirty=0 waiter=0 authpin=1 0x5571cf53e000]", "lock": { "gather_set": [], "state": "lock", "type": "iquiesce", "is_leased": false, "num_rdlocks": 0, "num_wrlocks": 0, "num_xlocks": 1, "xlock_by": { "reqid": { "entity": { "type": "mds", "num": 0 }, "tid": 181 } } }, "flags": 4, "wrlock_target": -1 } ] } }, { "description": "peer_request:mds.2:123", "initiated_at": "2024-04-23T10:31:21.463782+0000", "age": 42.420607275999998, "duration": 42.424913517999997, "continuous": false, "type_data": { "result": -2147483648, "flag_point": "dispatched", "reqid": { "entity": { "type": "mds", "num": 2 }, "tid": 123 }, "op_type": "peer_request", "leader_info": { "leader": "2" }, "events": [ { "time": "2024-04-23T10:31:21.463782+0000", "event": "initiated" }, { "time": "2024-04-23T10:31:21.463782+0000", "event": "header_read" }, { "time": "2024-04-23T10:31:21.463782+0000", "event": "throttled" }, { "time": "2024-04-23T10:31:21.463788+0000", "event": "all_read" }, { "time": "2024-04-23T10:31:21.493829+0000", "event": "dispatched" } ], "locks": [] } }, { "description": "peer_request:mds.2:113", "initiated_at": "2024-04-23T10:31:21.452144+0000", "age": 42.432245659000003, "duration": 42.436573430999999, "continuous": false, "type_data": { "result": -2147483648, "flag_point": "dispatched", "reqid": { "entity": { "type": "mds", "num": 2 }, "tid": 113 }, "op_type": "peer_request", "leader_info": { "leader": "2" }, "events": [ { "time": "2024-04-23T10:31:21.452144+0000", "event": "initiated" }, { "time": "2024-04-23T10:31:21.452144+0000", "event": "header_read" }, { "time": "2024-04-23T10:31:21.452144+0000", "event": "throttled" }, { "time": "2024-04-23T10:31:21.452151+0000", "event": "all_read" }, { "time": "2024-04-23T10:31:21.453185+0000", "event": "dispatched" } ], "locks": [] } }, { "description": "internal op quiesce_inode:mds.0:182 fp=#0x10000000209", "initiated_at": "2024-04-23T10:31:21.468596+0000", "age": 42.415792758000002, "duration": 42.420158164, "continuous": true, "type_data": { "result": -2147483648, "flag_point": "quiesce complete", "reqid": { "entity": { "type": "mds", "num": 0 }, "tid": 182 }, "op_type": "internal_op", "internal_op": 5385, "op_name": "quiesce_inode", "events": [ { "time": "2024-04-23T10:31:21.468596+0000", "event": "initiated" }, { "time": "2024-04-23T10:31:21.468596+0000", "event": "header_read" }, { "time": "2024-04-23T10:31:21.468596+0000", "event": "throttled" }, { "time": "2024-04-23T10:31:21.468596+0000", "event": "all_read" }, { "time": "2024-04-23T10:31:21.468596+0000", "event": "dispatched" }, { "time": "2024-04-23T10:31:21.469164+0000", "event": "failed to xlock, waiting" }, { "time": "2024-04-23T10:31:21.491959+0000", "event": "failed to xlock, waiting" }, { "time": "2024-04-23T10:31:21.496928+0000", "event": "acquired locks" }, { "time": "2024-04-23T10:31:21.497138+0000", "event": "quiesce complete" } ], "locks": [ { "object": { "is_auth": true, "auth_state": { "replicas": {} }, "replica_state": { "authority": [ 0, -2 ], "replica_nonce": 0 }, "auth_pins": 1, "is_frozen": false, "is_freezing": false, "pins": { "ptrwaiter": 0, "request": 1, "lock": 1, "caps": 0, "needsnapflush": 0, "dirtyparent": 0, "dirty": 0, "waiter": 0, "authpin": 1 }, "nref": 3 }, "object_string": "[inode 0x10000000209 [6,head] /volumes/qa/sv_1/b206794e-7622-4ab8-8943-f3ae4e65b7a3/client.0/tmp/clients/client0/~dmtmp/WORD/CHAP10.DOC auth v5588 ap=1 s=260096 n(v0 rc2024-04-23T10:27:51.955398+0000 b260096 1=1+0) (iquiesce lock x=1 by request(mds.0:182 nref=3)) | ptrwaiter=0 request=1 lock=1 caps=0 needsnapflush=0 dirtyparent=0 dirty=0 waiter=0 authpin=1 0x5571cf343700]", "lock": { "gather_set": [], "state": "lock", "type": "iquiesce", "is_leased": false, "num_rdlocks": 0, "num_wrlocks": 0, "num_xlocks": 1, "xlock_by": { "reqid": { "entity": { "type": "mds", "num": 0 }, "tid": 182 } } }, "flags": 4, "wrlock_target": -1 } ] } }, { "description": "peer_request:mds.1:119", "initiated_at": "2024-04-23T10:31:21.465063+0000", "age": 42.419326462000001, "duration": 42.423754004000003, "continuous": false, "type_data": { "result": -2147483648, "flag_point": "dispatched", "reqid": { "entity": { "type": "mds", "num": 1 }, "tid": 119 }, "op_type": "peer_request", "leader_info": { "leader": "1" }, "events": [ { "time": "2024-04-23T10:31:21.465063+0000", "event": "initiated" }, { "time": "2024-04-23T10:31:21.465063+0000", "event": "header_read" }, { "time": "2024-04-23T10:31:21.465063+0000", "event": "throttled" }, { "time": "2024-04-23T10:31:21.465068+0000", "event": "all_read" }, { "time": "2024-04-23T10:31:21.495290+0000", "event": "dispatched" } ], "locks": [] } }, { "description": "peer_request:mds.1:109", "initiated_at": "2024-04-23T10:31:21.452180+0000", "age": 42.432209415999999, "duration": 42.436658680000001, "continuous": false, "type_data": { "result": -2147483648, "flag_point": "dispatched", "reqid": { "entity": { "type": "mds", "num": 1 }, "tid": 109 }, "op_type": "peer_request", "leader_info": { "leader": "1" }, "events": [ { "time": "2024-04-23T10:31:21.452180+0000", "event": "initiated" }, { "time": "2024-04-23T10:31:21.452180+0000", "event": "header_read" }, { "time": "2024-04-23T10:31:21.452180+0000", "event": "throttled" }, { "time": "2024-04-23T10:31:21.452187+0000", "event": "all_read" }, { "time": "2024-04-23T10:31:21.453429+0000", "event": "dispatched" } ], "locks": [] } }, { "description": "internal op quiesce_inode:mds.0:183 fp=#0x10000000241", "initiated_at": "2024-04-23T10:31:21.469882+0000", "age": 42.414506783, "duration": 42.418976442000002, "continuous": true, "type_data": { "result": -2147483648, "flag_point": "quiesce complete", "reqid": { "entity": { "type": "mds", "num": 0 }, "tid": 183 }, "op_type": "internal_op", "internal_op": 5385, "op_name": "quiesce_inode", "events": [ { "time": "2024-04-23T10:31:21.469882+0000", "event": "initiated" }, { "time": "2024-04-23T10:31:21.469882+0000", "event": "header_read" }, { "time": "2024-04-23T10:31:21.469882+0000", "event": "throttled" }, { "time": "2024-04-23T10:31:21.469882+0000", "event": "all_read" }, { "time": "2024-04-23T10:31:21.469882+0000", "event": "dispatched" }, { "time": "2024-04-23T10:31:21.470153+0000", "event": "failed to xlock, waiting" }, { "time": "2024-04-23T10:31:21.508862+0000", "event": "acquired locks" }, { "time": "2024-04-23T10:31:21.509069+0000", "event": "quiesce complete" } ], "locks": [ { "object": { "is_auth": true, "auth_state": { "replicas": { "2": 1 } }, "replica_state": { "authority": [ 0, -2 ], "replica_nonce": 0 }, "auth_pins": 2, "is_frozen": false, "is_freezing": false, "pins": { "ptrwaiter": 0, "request": 1, "lock": 1, "importingcaps": 0, "caps": 0, "replicated": 1, "dirty": 0, "waiter": 0, "authpin": 1 }, "nref": 4 }, "object_string": "[inode 0x10000000241 [7,head] /volumes/qa/sv_1/b206794e-7622-4ab8-8943-f3ae4e65b7a3/client.0/tmp/clients/client0/~dmtmp/SEED/SMALL.FIL auth{2=1} v646 ap=2 s=327680 n(v0 rc2024-04-23T10:31:17.226728+0000 b327680 1=1+0) (iquiesce lock x=1 by request(mds.0:183 nref=3)) | ptrwaiter=0 request=1 lock=1 importingcaps=0 caps=0 replicated=1 dirty=0 waiter=0 authpin=1 0x5571cf30f180]", "lock": { "gather_set": [], "state": "lock", "type": "iquiesce", "is_leased": false, "num_rdlocks": 0, "num_wrlocks": 0, "num_xlocks": 1, "xlock_by": { "reqid": { "entity": { "type": "mds", "num": 0 }, "tid": 183 } } }, "flags": 4, "wrlock_target": -1 } ] } }, { "description": "peer_request:mds.2:124", "initiated_at": "2024-04-23T10:31:21.463891+0000", "age": 42.420498346999999, "duration": 42.425028378, "continuous": false, "type_data": { "result": -2147483648, "flag_point": "dispatched", "reqid": { "entity": { "type": "mds", "num": 2 }, "tid": 124 }, "op_type": "peer_request", "leader_info": { "leader": "2" }, "events": [ { "time": "2024-04-23T10:31:21.463891+0000", "event": "initiated" }, { "time": "2024-04-23T10:31:21.463891+0000", "event": "header_read" }, { "time": "2024-04-23T10:31:21.463891+0000", "event": "throttled" }, { "time": "2024-04-23T10:31:21.463896+0000", "event": "all_read" }, { "time": "2024-04-23T10:31:21.495370+0000", "event": "dispatched" } ], "locks": [] } }, { "description": "internal op quiesce_inode:mds.0:158 fp=#0x10000000007", "initiated_at": "2024-04-23T10:31:21.454050+0000", "age": 42.430339050999997, "duration": 42.434891911999998, "continuous": true, "type_data": { "result": -2147483648, "flag_point": "quiesce complete", "reqid": { "entity": { "type": "mds", "num": 0 }, "tid": 158 }, "op_type": "internal_op", "internal_op": 5385, "op_name": "quiesce_inode", "events": [ { "time": "2024-04-23T10:31:21.454050+0000", "event": "initiated" }, { "time": "2024-04-23T10:31:21.454050+0000", "event": "header_read" }, { "time": "2024-04-23T10:31:21.454050+0000", "event": "throttled" }, { "time": "2024-04-23T10:31:21.454050+0000", "event": "all_read" }, { "time": "2024-04-23T10:31:21.454050+0000", "event": "dispatched" }, { "time": "2024-04-23T10:31:21.454326+0000", "event": "failed to xlock, waiting" }, { "time": "2024-04-23T10:31:21.455522+0000", "event": "acquired locks" }, { "time": "2024-04-23T10:31:21.509882+0000", "event": "quiesce complete" } ], "locks": [ { "object": { "is_auth": true, "auth_state": { "replicas": { "1": 1, "2": 1 } }, "replica_state": { "authority": [ 0, -2 ], "replica_nonce": 0 }, "auth_pins": 3, "is_frozen": false, "is_freezing": false, "pins": { "dirtyscattered": 0, "request": 1, "lock": 1, "dirfrag": 1, "caps": 1, "dirtyrstat": 0, "dirtyparent": 0, "replicated": 1, "dirty": 1, "waiter": 0, "authpin": 1 }, "nref": 7 }, "object_string": "[inode 0x10000000007 [...7,head] /volumes/qa/sv_1/b206794e-7622-4ab8-8943-f3ae4e65b7a3/client.0/tmp/clients/ auth{1=1,2=1} v419 ap=3 f(v0 m2024-04-23T10:27:51.919399+0000 1=0+1) n(v30 rc2024-04-23T10:31:17.764716+0000 b21075074 75=63+12) old_inodes=4 (inest mix) (iquiesce lock x=1 by request(mds.0:158 nref=3)) caps={15312=pAsLsXsFs/pAsLsXsFs@13},l=15312 | dirtyscattered=0 request=1 lock=1 dirfrag=1 caps=1 dirtyrstat=0 dirtyparent=0 replicated=1 dirty=1 waiter=0 authpin=1 0x5571cf1de580]", "lock": { "gather_set": [], "state": "lock", "type": "iquiesce", "is_leased": false, "num_rdlocks": 0, "num_wrlocks": 0, "num_xlocks": 1, "xlock_by": { "reqid": { "entity": { "type": "mds", "num": 0 }, "tid": 158 } } }, "flags": 4, "wrlock_target": -1 } ] } }, { "description": "internal op quiesce_inode:mds.0:184 fp=#0x10000000242", "initiated_at": "2024-04-23T10:31:21.469904+0000", "age": 42.414485661000001, "duration": 42.419104969000003, "continuous": true, "type_data": { "result": -2147483648, "flag_point": "quiesce complete", "reqid": { "entity": { "type": "mds", "num": 0 }, "tid": 184 }, "op_type": "internal_op", "internal_op": 5385, "op_name": "quiesce_inode", "events": [ { "time": "2024-04-23T10:31:21.469904+0000", "event": "initiated" }, { "time": "2024-04-23T10:31:21.469904+0000", "event": "header_read" }, { "time": "2024-04-23T10:31:21.469904+0000", "event": "throttled" }, { "time": "2024-04-23T10:31:21.469904+0000", "event": "all_read" }, { "time": "2024-04-23T10:31:21.469904+0000", "event": "dispatched" }, { "time": "2024-04-23T10:31:21.470297+0000", "event": "failed to xlock, waiting" }, { "time": "2024-04-23T10:31:21.495245+0000", "event": "failed to xlock, waiting" }, { "time": "2024-04-23T10:31:21.500681+0000", "event": "acquired locks" }, { "time": "2024-04-23T10:31:21.500803+0000", "event": "quiesce complete" } ], "locks": [ { "object": { "is_auth": true, "auth_state": { "replicas": { "2": 1 } }, "replica_state": { "authority": [ 0, -2 ], "replica_nonce": 0 }, "auth_pins": 2, "is_frozen": false, "is_freezing": false, "pins": { "ptrwaiter": 0, "request": 1, "lock": 1, "importingcaps": 0, "caps": 0, "needsnapflush": 0, "replicated": 1, "dirty": 0, "waiter": 0, "authpin": 1 }, "nref": 4 }, "object_string": "[inode 0x10000000242 [7,head] /volumes/qa/sv_1/b206794e-7622-4ab8-8943-f3ae4e65b7a3/client.0/tmp/clients/client0/~dmtmp/SEED/MEDIUM.FIL auth{2=1} v650 ap=2 s=1048576 n(v0 rc2024-04-23T10:31:17.227728+0000 b1048576 1=1+0) (iquiesce lock x=1 by request(mds.0:184 nref=3)) | ptrwaiter=0 request=1 lock=1 importingcaps=0 caps=0 needsnapflush=0 replicated=1 dirty=0 waiter=0 authpin=1 0x5571cf5cec00]", "lock": { "gather_set": [], "state": "lock", "type": "iquiesce", "is_leased": false, "num_rdlocks": 0, "num_wrlocks": 0, "num_xlocks": 1, "xlock_by": { "reqid": { "entity": { "type": "mds", "num": 0 }, "tid": 184 } } }, "flags": 4, "wrlock_target": -1 } ] } }, { "description": "peer_request:mds.1:120", "initiated_at": "2024-04-23T10:31:21.465132+0000", "age": 42.419257543999997, "duration": 42.423939896999997, "continuous": false, "type_data": { "result": -2147483648, "flag_point": "dispatched", "reqid": { "entity": { "type": "mds", "num": 1 }, "tid": 120 }, "op_type": "peer_request", "leader_info": { "leader": "1" }, "events": [ { "time": "2024-04-23T10:31:21.465132+0000", "event": "initiated" }, { "time": "2024-04-23T10:31:21.465132+0000", "event": "header_read" }, { "time": "2024-04-23T10:31:21.465132+0000", "event": "throttled" }, { "time": "2024-04-23T10:31:21.465137+0000", "event": "all_read" }, { "time": "2024-04-23T10:31:21.496283+0000", "event": "dispatched" } ], "locks": [] } }, { "description": "peer_request:mds.2:114", "initiated_at": "2024-04-23T10:31:21.453566+0000", "age": 42.430822908000003, "duration": 42.435527239000002, "continuous": false, "type_data": { "result": -2147483648, "flag_point": "dispatched", "reqid": { "entity": { "type": "mds", "num": 2 }, "tid": 114 }, "op_type": "peer_request", "leader_info": { "leader": "2" }, "events": [ { "time": "2024-04-23T10:31:21.453566+0000", "event": "initiated" }, { "time": "2024-04-23T10:31:21.453566+0000", "event": "header_read" }, { "time": "2024-04-23T10:31:21.453567+0000", "event": "throttled" }, { "time": "2024-04-23T10:31:21.453575+0000", "event": "all_read" }, { "time": "2024-04-23T10:31:21.454920+0000", "event": "dispatched" } ], "locks": [] } }, { "description": "internal op quiesce_inode:mds.0:185 fp=#0x10000000243", "initiated_at": "2024-04-23T10:31:21.469925+0000", "age": 42.414464531, "duration": 42.419189344000003, "continuous": true, "type_data": { "result": -2147483648, "flag_point": "quiesce complete", "reqid": { "entity": { "type": "mds", "num": 0 }, "tid": 185 }, "op_type": "internal_op", "internal_op": 5385, "op_name": "quiesce_inode", "events": [ { "time": "2024-04-23T10:31:21.469925+0000", "event": "initiated" }, { "time": "2024-04-23T10:31:21.469925+0000", "event": "header_read" }, { "time": "2024-04-23T10:31:21.469925+0000", "event": "throttled" }, { "time": "2024-04-23T10:31:21.469925+0000", "event": "all_read" }, { "time": "2024-04-23T10:31:21.469925+0000", "event": "dispatched" }, { "time": "2024-04-23T10:31:21.470469+0000", "event": "failed to xlock, waiting" }, { "time": "2024-04-23T10:31:21.509433+0000", "event": "acquired locks" }, { "time": "2024-04-23T10:31:21.509734+0000", "event": "quiesce complete" } ], "locks": [ { "object": { "is_auth": true, "auth_state": { "replicas": { "2": 1 } }, "replica_state": { "authority": [ 0, -2 ], "replica_nonce": 0 }, "auth_pins": 2, "is_frozen": false, "is_freezing": false, "pins": { "ptrwaiter": 0, "request": 1, "lock": 1, "importingcaps": 0, "caps": 0, "replicated": 1, "dirty": 0, "waiter": 0, "authpin": 1 }, "nref": 4 }, "object_string": "[inode 0x10000000243 [7,head] /volumes/qa/sv_1/b206794e-7622-4ab8-8943-f3ae4e65b7a3/client.0/tmp/clients/client0/~dmtmp/SEED/LARGE.FIL auth{2=1} v648 ap=2 s=1867776 n(v0 rc2024-04-23T10:31:17.246728+0000 b1867776 1=1+0) (iquiesce lock x=1 by request(mds.0:185 nref=3)) | ptrwaiter=0 request=1 lock=1 importingcaps=0 caps=0 replicated=1 dirty=0 waiter=0 authpin=1 0x5571cf5d8b00]", "lock": { "gather_set": [], "state": "lock", "type": "iquiesce", "is_leased": false, "num_rdlocks": 0, "num_wrlocks": 0, "num_xlocks": 1, "xlock_by": { "reqid": { "entity": { "type": "mds", "num": 0 }, "tid": 185 } } }, "flags": 4, "wrlock_target": -1 } ] } }, { "description": "peer_request:mds.2:125", "initiated_at": "2024-04-23T10:31:21.463955+0000", "age": 42.420433768999999, "duration": 42.425218538000003, "continuous": false, "type_data": { "result": -2147483648, "flag_point": "dispatched", "reqid": { "entity": { "type": "mds", "num": 2 }, "tid": 125 }, "op_type": "peer_request", "leader_info": { "leader": "2" }, "events": [ { "time": "2024-04-23T10:31:21.463955+0000", "event": "initiated" }, { "time": "2024-04-23T10:31:21.463955+0000", "event": "header_read" }, { "time": "2024-04-23T10:31:21.463956+0000", "event": "throttled" }, { "time": "2024-04-23T10:31:21.463961+0000", "event": "all_read" }, { "time": "2024-04-23T10:31:21.496373+0000", "event": "dispatched" } ], "locks": [] } } ], "num_ops": 95 } mds.i: { "ops": [], "num_ops": 0 } 2024-04-23T10:32:05.015 DEBUG:teuthology.orchestra.run.smithi114:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-23T10:32:05.015 INFO:tasks.workunit.client.0.smithi114.stdout: 1 305661 5.21 MB/sec execute 132 sec latency 46146.508 ms 2024-04-23T10:32:05.016 INFO:journalctl@ceph.mon.a.smithi114.stdout:Apr 23 10:32:04 smithi114 ceph-mon[29255]: pgmap v168: 97 pgs: 97 active+clean; 78 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 72 KiB/s rd, 786 KiB/s wr, 55 op/s 2024-04-23T10:32:05.016 INFO:journalctl@ceph.mon.a.smithi114.stdout:Apr 23 10:32:04 smithi114 ceph-mon[29255]: from='client.? 172.21.15.114:0/2509710607' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-23T10:32:05.016 INFO:tasks.workunit.client.0.smithi114.stdout: 1 305661 5.17 MB/sec execute 133 sec latency 47146.618 ms 2024-04-23T10:32:05.121 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:32:05.122+0000 7f54d9c67640 1 -- 172.21.15.114:0/2235811939 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f54d41010a0 msgr2=0x7f54d4101480 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T10:32:05.122 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:32:05.122+0000 7f54d9c67640 1 --2- 172.21.15.114:0/2235811939 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f54d41010a0 0x7f54d4101480 secure :-1 s=READY pgs=730 cs=0 l=1 rev1=1 crypto rx=0x7f54c4009900 tx=0x7f54c402a410 comp rx=0 tx=0).stop 2024-04-23T10:32:05.122 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:32:05.122+0000 7f54d9c67640 1 -- 172.21.15.114:0/2235811939 shutdown_connections 2024-04-23T10:32:05.122 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:32:05.122+0000 7f54d9c67640 1 --2- 172.21.15.114:0/2235811939 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f54d4106170 0x7f54d410dd40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:32:05.122 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:32:05.122+0000 7f54d9c67640 1 --2- 172.21.15.114:0/2235811939 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f54d4101a50 0x7f54d4105a90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:32:05.122 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:32:05.122+0000 7f54d9c67640 1 --2- 172.21.15.114:0/2235811939 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f54d41010a0 0x7f54d4101480 unknown :-1 s=CLOSED pgs=730 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:32:05.122 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:32:05.122+0000 7f54d9c67640 1 -- 172.21.15.114:0/2235811939 >> 172.21.15.114:0/2235811939 conn(0x7f54d4077f50 msgr2=0x7f54d4078360 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T10:32:05.122 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:32:05.122+0000 7f54d9c67640 1 -- 172.21.15.114:0/2235811939 shutdown_connections 2024-04-23T10:32:05.122 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:32:05.122+0000 7f54d9c67640 1 -- 172.21.15.114:0/2235811939 wait complete. 2024-04-23T10:32:05.122 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:32:05.123+0000 7f54d9c67640 1 Processor -- start 2024-04-23T10:32:05.122 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:32:05.123+0000 7f54d9c67640 1 -- start start 2024-04-23T10:32:05.124 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:32:05.123+0000 7f54d9c67640 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f54d41010a0 0x7f54d4197650 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T10:32:05.124 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:32:05.123+0000 7f54d9c67640 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f54d4106170 0x7f54d4197b90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T10:32:05.124 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:32:05.123+0000 7f54d9c67640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f54d419bf10 0x7f54d419c3c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T10:32:05.125 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:32:05.123+0000 7f54d9c67640 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x7f54d41110d0 con 0x7f54d419bf10 2024-04-23T10:32:05.125 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:32:05.123+0000 7f54d9c67640 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 -- 0x7f54d4110f50 con 0x7f54d4106170 2024-04-23T10:32:05.125 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:32:05.123+0000 7f54d9c67640 1 -- --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_getmap magic: 0 -- 0x7f54d4111250 con 0x7f54d41010a0 2024-04-23T10:32:05.125 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:32:05.123+0000 7f54d3fff640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f54d419bf10 0x7f54d419c3c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T10:32:05.125 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:32:05.123+0000 7f54d3fff640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f54d419bf10 0x7f54d419c3c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.114:52134/0 (socket says 172.21.15.114:52134) 2024-04-23T10:32:05.125 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:32:05.123+0000 7f54d3fff640 1 -- 172.21.15.114:0/2628387773 learned_addr learned my addr 172.21.15.114:0/2628387773 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-04-23T10:32:05.125 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:32:05.123+0000 7f54d37fe640 1 --2- 172.21.15.114:0/2628387773 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f54d41010a0 0x7f54d4197650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T10:32:05.125 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:32:05.123+0000 7f54d2ffd640 1 --2- 172.21.15.114:0/2628387773 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f54d4106170 0x7f54d4197b90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T10:32:05.125 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:32:05.124+0000 7f54d3fff640 1 -- 172.21.15.114:0/2628387773 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f54d41010a0 msgr2=0x7f54d4197650 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T10:32:05.125 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:32:05.124+0000 7f54d3fff640 1 --2- 172.21.15.114:0/2628387773 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f54d41010a0 0x7f54d4197650 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:32:05.125 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:32:05.124+0000 7f54d3fff640 1 -- 172.21.15.114:0/2628387773 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f54d4106170 msgr2=0x7f54d4197b90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T10:32:05.125 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:32:05.124+0000 7f54d3fff640 1 --2- 172.21.15.114:0/2628387773 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f54d4106170 0x7f54d4197b90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:32:05.125 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:32:05.124+0000 7f54d3fff640 1 -- 172.21.15.114:0/2628387773 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f54d419ca00 con 0x7f54d419bf10 2024-04-23T10:32:05.125 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:32:05.124+0000 7f54d3fff640 1 --2- 172.21.15.114:0/2628387773 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f54d419bf10 0x7f54d419c3c0 secure :-1 s=READY pgs=731 cs=0 l=1 rev1=1 crypto rx=0x7f54cc00cad0 tx=0x7f54cc00cf90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T10:32:05.125 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:32:05.124+0000 7f54d0ff9640 1 -- 172.21.15.114:0/2628387773 <== mon.0 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f54cc00ae20 con 0x7f54d419bf10 2024-04-23T10:32:05.126 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:32:05.124+0000 7f54d0ff9640 1 -- 172.21.15.114:0/2628387773 <== mon.0 v2:172.21.15.114:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f54cc005020 con 0x7f54d419bf10 2024-04-23T10:32:05.126 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:32:05.124+0000 7f54d9c67640 1 -- 172.21.15.114:0/2628387773 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f54d419ccf0 con 0x7f54d419bf10 2024-04-23T10:32:05.126 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:32:05.124+0000 7f54d0ff9640 1 -- 172.21.15.114:0/2628387773 <== mon.0 v2:172.21.15.114:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f54cc00f6b0 con 0x7f54d419bf10 2024-04-23T10:32:05.126 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:32:05.124+0000 7f54d9c67640 1 -- 172.21.15.114:0/2628387773 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f54d41a4490 con 0x7f54d419bf10 2024-04-23T10:32:05.126 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:32:05.125+0000 7f54d9c67640 1 -- 172.21.15.114:0/2628387773 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f54d4068d70 con 0x7f54d419bf10 2024-04-23T10:32:05.127 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:32:05.126+0000 7f54d0ff9640 1 -- 172.21.15.114:0/2628387773 <== mon.0 v2:172.21.15.114:3300/0 4 ==== mgrmap(e 26) ==== 138778+0+0 (secure 0 0 0) 0x7f54cc0051d0 con 0x7f54d419bf10 2024-04-23T10:32:05.127 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:32:05.127+0000 7f54d0ff9640 1 --2- 172.21.15.114:0/2628387773 >> [v2:172.21.15.114:6800/1395693466,v1:172.21.15.114:6801/1395693466] conn(0x7f54bc0a3d20 0x7f54bc0a61e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T10:32:05.127 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:32:05.127+0000 7f54d37fe640 1 --2- 172.21.15.114:0/2628387773 >> [v2:172.21.15.114:6800/1395693466,v1:172.21.15.114:6801/1395693466] conn(0x7f54bc0a3d20 0x7f54bc0a61e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T10:32:05.127 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:32:05.127+0000 7f54d0ff9640 1 -- 172.21.15.114:0/2628387773 <== mon.0 v2:172.21.15.114:3300/0 5 ==== osd_map(94..94 src has 1..94) ==== 8585+0+0 (secure 0 0 0) 0x7f54cc0d07f0 con 0x7f54d419bf10 2024-04-23T10:32:05.127 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:32:05.127+0000 7f54d37fe640 1 --2- 172.21.15.114:0/2628387773 >> [v2:172.21.15.114:6800/1395693466,v1:172.21.15.114:6801/1395693466] conn(0x7f54bc0a3d20 0x7f54bc0a61e0 secure :-1 s=READY pgs=107 cs=0 l=1 rev1=1 crypto rx=0x7f54c4031440 tx=0x7f54c4005e80 comp rx=0 tx=0).ready entity=mgr.15291 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T10:32:05.130 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:32:05.132+0000 7f54d0ff9640 1 -- 172.21.15.114:0/2628387773 <== mon.0 v2:172.21.15.114:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f54cc014030 con 0x7f54d419bf10 2024-04-23T10:32:05.347 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:32:05.348+0000 7f54d9c67640 1 -- 172.21.15.114:0/2628387773 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f54d4101a50 con 0x7f54d419bf10 2024-04-23T10:32:05.348 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:32:05.349+0000 7f54d0ff9640 1 -- 172.21.15.114:0/2628387773 <== mon.0 v2:172.21.15.114:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 30 v30) ==== 94+0+11884 (secure 0 0 0) 0x7f54cc09c800 con 0x7f54d419bf10 2024-04-23T10:32:05.348 INFO:teuthology.orchestra.run.smithi114.stdout: 2024-04-23T10:32:05.348 INFO:teuthology.orchestra.run.smithi114.stdout:{"epoch":30,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14547,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.114:6835/663493580","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6834","nonce":663493580},{"type":"v1","addr":"172.21.15.114:6835","nonce":663493580}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14565,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.114:6837/1439252154","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6836","nonce":1439252154},{"type":"v1","addr":"172.21.15.114:6837","nonce":1439252154}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5},{"gid":14634,"name":"g","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.114:6839/3849796939","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6838","nonce":3849796939},{"type":"v1","addr":"172.21.15.114:6839","nonce":3849796939}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":11},{"gid":14646,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.114:6841/3544523891","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6840","nonce":3544523891},{"type":"v1","addr":"172.21.15.114:6841","nonce":3544523891}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":12},{"gid":24385,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.204:6833/3866561754","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6832","nonce":3866561754},{"type":"v1","addr":"172.21.15.204:6833","nonce":3866561754}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2},{"gid":24427,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.204:6835/2533793172","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6834","nonce":2533793172},{"type":"v1","addr":"172.21.15.204:6835","nonce":2533793172}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":7}],"filesystems":[{"mdsmap":{"epoch":30,"flags":306,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":true},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-04-23T10:24:43.383158+0000","modified":"2024-04-23T10:31:33.134629+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":3,"in":[0,1,2],"up":{"mds_0":24545,"mds_1":24509,"mds_2":24479},"failed":[],"damaged":[],"stopped":[],"info":{"gid_24439":{"gid":24439,"name":"c","rank":2,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.204:6837/513822179","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6836","nonce":513822179},{"type":"v1","addr":"172.21.15.204:6837","nonce":513822179}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24443":{"gid":24443,"name":"k","rank":1,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.119:6833/783244660","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6832","nonce":783244660},{"type":"v1","addr":"172.21.15.119:6833","nonce":783244660}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24479":{"gid":24479,"name":"e","rank":2,"incarnation":19,"state":"up:active","state_seq":9,"addr":"172.21.15.119:6835/2033916709","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6834","nonce":2033916709},{"type":"v1","addr":"172.21.15.119:6835","nonce":2033916709}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24509":{"gid":24509,"name":"h","rank":1,"incarnation":17,"state":"up:active","state_seq":6,"addr":"172.21.15.119:6837/1872363128","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6836","nonce":1872363128},{"type":"v1","addr":"172.21.15.119:6837","nonce":1872363128}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24545":{"gid":24545,"name":"b","rank":0,"incarnation":14,"state":"up:active","state_seq":3,"addr":"172.21.15.119:6839/937297340","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6838","nonce":937297340},{"type":"v1","addr":"172.21.15.119:6839","nonce":937297340}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_25537":{"gid":25537,"name":"i","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.204:6839/2396114218","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6838","nonce":2396114218},{"type":"v1","addr":"172.21.15.204:6839","nonce":2396114218}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":24545,"qdb_cluster":[24479,24509,24545]},"id":1}]} 2024-04-23T10:32:05.348 INFO:teuthology.orchestra.run.smithi114.stderr:dumped fsmap epoch 30 2024-04-23T10:32:05.350 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:32:05.351+0000 7f54d9c67640 1 -- 172.21.15.114:0/2628387773 >> [v2:172.21.15.114:6800/1395693466,v1:172.21.15.114:6801/1395693466] conn(0x7f54bc0a3d20 msgr2=0x7f54bc0a61e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T10:32:05.350 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:32:05.351+0000 7f54d9c67640 1 --2- 172.21.15.114:0/2628387773 >> [v2:172.21.15.114:6800/1395693466,v1:172.21.15.114:6801/1395693466] conn(0x7f54bc0a3d20 0x7f54bc0a61e0 secure :-1 s=READY pgs=107 cs=0 l=1 rev1=1 crypto rx=0x7f54c4031440 tx=0x7f54c4005e80 comp rx=0 tx=0).stop 2024-04-23T10:32:05.350 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:32:05.351+0000 7f54d9c67640 1 -- 172.21.15.114:0/2628387773 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f54d419bf10 msgr2=0x7f54d419c3c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T10:32:05.350 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:32:05.351+0000 7f54d9c67640 1 --2- 172.21.15.114:0/2628387773 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f54d419bf10 0x7f54d419c3c0 secure :-1 s=READY pgs=731 cs=0 l=1 rev1=1 crypto rx=0x7f54cc00cad0 tx=0x7f54cc00cf90 comp rx=0 tx=0).stop 2024-04-23T10:32:05.350 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:32:05.351+0000 7f54d9c67640 1 -- 172.21.15.114:0/2628387773 shutdown_connections 2024-04-23T10:32:05.350 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:32:05.351+0000 7f54d9c67640 1 --2- 172.21.15.114:0/2628387773 >> [v2:172.21.15.114:6800/1395693466,v1:172.21.15.114:6801/1395693466] conn(0x7f54bc0a3d20 0x7f54bc0a61e0 unknown :-1 s=CLOSED pgs=107 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:32:05.350 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:32:05.351+0000 7f54d9c67640 1 --2- 172.21.15.114:0/2628387773 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f54d419bf10 0x7f54d419c3c0 unknown :-1 s=CLOSED pgs=731 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:32:05.350 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:32:05.351+0000 7f54d9c67640 1 --2- 172.21.15.114:0/2628387773 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f54d4106170 0x7f54d4197b90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:32:05.350 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:32:05.351+0000 7f54d9c67640 1 --2- 172.21.15.114:0/2628387773 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f54d41010a0 0x7f54d4197650 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:32:05.350 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:32:05.351+0000 7f54d9c67640 1 -- 172.21.15.114:0/2628387773 >> 172.21.15.114:0/2628387773 conn(0x7f54d4077f50 msgr2=0x7f54d40ff0c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T10:32:05.351 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:32:05.351+0000 7f54d9c67640 1 -- 172.21.15.114:0/2628387773 shutdown_connections 2024-04-23T10:32:05.351 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:32:05.351+0000 7f54d9c67640 1 -- 172.21.15.114:0/2628387773 wait complete. 2024-04-23T10:32:05.361 DEBUG:tasks.quiescer.fs.[cephfs]:Running ceph command: 'tell mds.24545 quiesce db --set-id 11138c86 --await-for 42.0' 2024-04-23T10:32:05.361 DEBUG:teuthology.orchestra.run.smithi114:> timeout 75 sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.24545 quiesce db --set-id 11138c86 --await-for 42.0 2024-04-23T10:32:05.469 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:32:05.467+0000 7f347e8a7640 1 -- 172.21.15.114:0/4262332703 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f3478107270 msgr2=0x7f347810ee40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T10:32:05.469 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:32:05.467+0000 7f347e8a7640 1 --2- 172.21.15.114:0/4262332703 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f3478107270 0x7f347810ee40 secure :-1 s=READY pgs=732 cs=0 l=1 rev1=1 crypto rx=0x7f3470009ca0 tx=0x7f347002a8c0 comp rx=0 tx=0).stop 2024-04-23T10:32:05.469 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:32:05.467+0000 7f347e8a7640 1 -- 172.21.15.114:0/4262332703 shutdown_connections 2024-04-23T10:32:05.469 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:32:05.467+0000 7f347e8a7640 1 --2- 172.21.15.114:0/4262332703 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f3478107270 0x7f347810ee40 unknown :-1 s=CLOSED pgs=732 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:32:05.469 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:32:05.467+0000 7f347e8a7640 1 --2- 172.21.15.114:0/4262332703 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f347806b1d0 0x7f3478106b90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:32:05.469 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:32:05.467+0000 7f347e8a7640 1 --2- 172.21.15.114:0/4262332703 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f347806a820 0x7f347806ac00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:32:05.469 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:32:05.467+0000 7f347e8a7640 1 -- 172.21.15.114:0/4262332703 >> 172.21.15.114:0/4262332703 conn(0x7f3478077eb0 msgr2=0x7f34780782c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T10:32:05.469 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:32:05.467+0000 7f347e8a7640 1 -- 172.21.15.114:0/4262332703 shutdown_connections 2024-04-23T10:32:05.469 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:32:05.467+0000 7f347e8a7640 1 -- 172.21.15.114:0/4262332703 wait complete. 2024-04-23T10:32:05.469 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:32:05.468+0000 7f347e8a7640 1 Processor -- start 2024-04-23T10:32:05.469 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:32:05.468+0000 7f347e8a7640 1 -- start start 2024-04-23T10:32:05.469 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:32:05.468+0000 7f347e8a7640 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f347806a820 0x7f34781986f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T10:32:05.469 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:32:05.468+0000 7f347e8a7640 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f3478107270 0x7f3478198c30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T10:32:05.470 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:32:05.468+0000 7f347e8a7640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f347819cfb0 0x7f347819d460 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T10:32:05.470 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:32:05.468+0000 7f347e8a7640 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x7f34781121d0 con 0x7f347819cfb0 2024-04-23T10:32:05.470 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:32:05.468+0000 7f347e8a7640 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 -- 0x7f3478112050 con 0x7f3478107270 2024-04-23T10:32:05.470 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:32:05.468+0000 7f347e8a7640 1 -- --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_getmap magic: 0 -- 0x7f3478112350 con 0x7f347806a820 2024-04-23T10:32:05.470 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:32:05.468+0000 7f347ce44640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f347819cfb0 0x7f347819d460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T10:32:05.470 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:32:05.468+0000 7f347ce44640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f347819cfb0 0x7f347819d460 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.114:52158/0 (socket says 172.21.15.114:52158) 2024-04-23T10:32:05.470 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:32:05.468+0000 7f347ce44640 1 -- 172.21.15.114:0/2283858768 learned_addr learned my addr 172.21.15.114:0/2283858768 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-04-23T10:32:05.470 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:32:05.469+0000 7f3477fff640 1 --2- 172.21.15.114:0/2283858768 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f347806a820 0x7f34781986f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T10:32:05.470 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:32:05.469+0000 7f34777fe640 1 --2- 172.21.15.114:0/2283858768 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f3478107270 0x7f3478198c30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T10:32:05.470 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:32:05.469+0000 7f347ce44640 1 -- 172.21.15.114:0/2283858768 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f347806a820 msgr2=0x7f34781986f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T10:32:05.470 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:32:05.469+0000 7f347ce44640 1 --2- 172.21.15.114:0/2283858768 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f347806a820 0x7f34781986f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:32:05.470 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:32:05.469+0000 7f347ce44640 1 -- 172.21.15.114:0/2283858768 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f3478107270 msgr2=0x7f3478198c30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T10:32:05.471 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:32:05.469+0000 7f347ce44640 1 --2- 172.21.15.114:0/2283858768 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f3478107270 0x7f3478198c30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:32:05.471 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:32:05.469+0000 7f347ce44640 1 -- 172.21.15.114:0/2283858768 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f347819daa0 con 0x7f347819cfb0 2024-04-23T10:32:05.471 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:32:05.469+0000 7f34777fe640 1 --2- 172.21.15.114:0/2283858768 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f3478107270 0x7f3478198c30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-23T10:32:05.471 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:32:05.469+0000 7f347ce44640 1 --2- 172.21.15.114:0/2283858768 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f347819cfb0 0x7f347819d460 secure :-1 s=READY pgs=733 cs=0 l=1 rev1=1 crypto rx=0x7f34700098d0 tx=0x7f34700317b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T10:32:05.471 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:32:05.469+0000 7f34757fa640 1 -- 172.21.15.114:0/2283858768 <== mon.0 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3470003b70 con 0x7f347819cfb0 2024-04-23T10:32:05.471 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:32:05.469+0000 7f34757fa640 1 -- 172.21.15.114:0/2283858768 <== mon.0 v2:172.21.15.114:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f3470003d10 con 0x7f347819cfb0 2024-04-23T10:32:05.471 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:32:05.469+0000 7f347e8a7640 1 -- 172.21.15.114:0/2283858768 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f347819dd30 con 0x7f347819cfb0 2024-04-23T10:32:05.471 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:32:05.469+0000 7f34757fa640 1 -- 172.21.15.114:0/2283858768 <== mon.0 v2:172.21.15.114:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f347002fa20 con 0x7f347819cfb0 2024-04-23T10:32:05.471 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:32:05.469+0000 7f347e8a7640 1 -- 172.21.15.114:0/2283858768 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f34781a5550 con 0x7f347819cfb0 2024-04-23T10:32:05.478 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:32:05.471+0000 7f34757fa640 1 -- 172.21.15.114:0/2283858768 <== mon.0 v2:172.21.15.114:3300/0 4 ==== mgrmap(e 26) ==== 138778+0+0 (secure 0 0 0) 0x7f347002fc40 con 0x7f347819cfb0 2024-04-23T10:32:05.478 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:32:05.472+0000 7f34815bf740 1 Processor -- start 2024-04-23T10:32:05.478 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:32:05.472+0000 7f34815bf740 1 -- start start 2024-04-23T10:32:05.479 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:32:05.472+0000 7f34815bf740 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x5638da617b60 0x5638da617f40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T10:32:05.479 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:32:05.472+0000 7f34815bf740 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x5638da618480 0x5638da622610 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T10:32:05.479 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:32:05.472+0000 7f34815bf740 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x5638da622b50 0x5638da624f40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T10:32:05.479 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:32:05.472+0000 7f34815bf740 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x5638da57d380 con 0x5638da617b60 2024-04-23T10:32:05.479 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:32:05.472+0000 7f34815bf740 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 -- 0x5638da519d90 con 0x5638da622b50 2024-04-23T10:32:05.479 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:32:05.472+0000 7f34815bf740 1 -- --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_getmap magic: 0 -- 0x5638da579340 con 0x5638da618480 2024-04-23T10:32:05.479 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:32:05.472+0000 7f34757fa640 1 --2- 172.21.15.114:0/2283858768 >> [v2:172.21.15.114:6800/1395693466,v1:172.21.15.114:6801/1395693466] conn(0x7f34600a3d20 0x7f34600a61e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T10:32:05.479 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:32:05.472+0000 7f34777fe640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x5638da617b60 0x5638da617f40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T10:32:05.479 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:32:05.472+0000 7f34777fe640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x5638da617b60 0x5638da617f40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.114:52162/0 (socket says 172.21.15.114:52162) 2024-04-23T10:32:05.479 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:32:05.472+0000 7f34777fe640 1 -- 172.21.15.114:0/437848922 learned_addr learned my addr 172.21.15.114:0/437848922 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-04-23T10:32:05.479 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:32:05.472+0000 7f347ce44640 1 --2- 172.21.15.114:0/2283858768 >> [v2:172.21.15.114:6800/1395693466,v1:172.21.15.114:6801/1395693466] conn(0x7f34600a3d20 0x7f34600a61e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T10:32:05.479 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:32:05.472+0000 7f3477fff640 1 --2- 172.21.15.114:0/437848922 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x5638da618480 0x5638da622610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T10:32:05.479 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:32:05.472+0000 7f34777fe640 1 --2- 172.21.15.114:0/437848922 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x5638da622b50 0x5638da624f40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T10:32:05.479 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:32:05.472+0000 7f34757fa640 1 -- 172.21.15.114:0/2283858768 <== mon.0 v2:172.21.15.114:3300/0 5 ==== osd_map(94..94 src has 1..94) ==== 8585+0+0 (secure 0 0 0) 0x7f34700f1c70 con 0x7f347819cfb0 2024-04-23T10:32:05.480 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:32:05.472+0000 7f34777fe640 1 -- 172.21.15.114:0/437848922 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x5638da618480 msgr2=0x5638da622610 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T10:32:05.480 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:32:05.472+0000 7f34777fe640 1 --2- 172.21.15.114:0/437848922 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x5638da618480 0x5638da622610 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:32:05.480 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:32:05.472+0000 7f34777fe640 1 -- 172.21.15.114:0/437848922 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x5638da622b50 msgr2=0x5638da624f40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T10:32:05.480 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:32:05.472+0000 7f34777fe640 1 --2- 172.21.15.114:0/437848922 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x5638da622b50 0x5638da624f40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:32:05.480 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:32:05.473+0000 7f34777fe640 1 -- 172.21.15.114:0/437848922 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5638da5444d0 con 0x5638da617b60 2024-04-23T10:32:05.480 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:32:05.473+0000 7f3477fff640 1 --2- 172.21.15.114:0/437848922 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x5638da618480 0x5638da622610 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-23T10:32:05.480 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:32:05.473+0000 7f34777fe640 1 --2- 172.21.15.114:0/437848922 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x5638da617b60 0x5638da617f40 secure :-1 s=READY pgs=734 cs=0 l=1 rev1=1 crypto rx=0x7f346c009de0 tx=0x7f346c009590 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c3fca77b70bff5b9 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T10:32:05.480 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:32:05.473+0000 7f347ce44640 1 --2- 172.21.15.114:0/2283858768 >> [v2:172.21.15.114:6800/1395693466,v1:172.21.15.114:6801/1395693466] conn(0x7f34600a3d20 0x7f34600a61e0 secure :-1 s=READY pgs=108 cs=0 l=1 rev1=1 crypto rx=0x7f34700bd980 tx=0x7f347008f6c0 comp rx=0 tx=0).ready entity=mgr.15291 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T10:32:05.480 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:32:05.473+0000 7f345e7fc640 1 -- 172.21.15.114:0/437848922 <== mon.0 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f346c017070 con 0x5638da617b60 2024-04-23T10:32:05.480 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:32:05.473+0000 7f345e7fc640 1 -- 172.21.15.114:0/437848922 <== mon.0 v2:172.21.15.114:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f346c0054d0 con 0x5638da617b60 2024-04-23T10:32:05.480 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:32:05.473+0000 7f345e7fc640 1 -- 172.21.15.114:0/437848922 <== mon.0 v2:172.21.15.114:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f346c013400 con 0x5638da617b60 2024-04-23T10:32:05.480 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:32:05.474+0000 7f34815bf740 1 -- 172.21.15.114:0/437848922 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x5638da617b60 msgr2=0x5638da617f40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T10:32:05.480 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:32:05.474+0000 7f34815bf740 1 --2- 172.21.15.114:0/437848922 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x5638da617b60 0x5638da617f40 secure :-1 s=READY pgs=734 cs=0 l=1 rev1=1 crypto rx=0x7f346c009de0 tx=0x7f346c009590 comp rx=0 tx=0).stop 2024-04-23T10:32:05.480 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:32:05.474+0000 7f34815bf740 1 -- 172.21.15.114:0/437848922 shutdown_connections 2024-04-23T10:32:05.480 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:32:05.474+0000 7f34815bf740 1 --2- 172.21.15.114:0/437848922 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x5638da622b50 0x5638da624f40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:32:05.481 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:32:05.474+0000 7f34815bf740 1 --2- 172.21.15.114:0/437848922 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x5638da618480 0x5638da622610 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:32:05.481 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:32:05.474+0000 7f34815bf740 1 --2- 172.21.15.114:0/437848922 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x5638da617b60 0x5638da617f40 unknown :-1 s=CLOSED pgs=734 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:32:05.481 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:32:05.474+0000 7f34815bf740 1 -- 172.21.15.114:0/437848922 >> 172.21.15.114:0/437848922 conn(0x5638da613e80 msgr2=0x5638da614260 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T10:32:05.481 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:32:05.474+0000 7f34815bf740 1 -- 172.21.15.114:0/437848922 shutdown_connections 2024-04-23T10:32:05.481 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:32:05.474+0000 7f34815bf740 1 -- 172.21.15.114:0/437848922 wait complete. 2024-04-23T10:32:05.481 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:32:05.474+0000 7f34815bf740 1 -- start start 2024-04-23T10:32:05.481 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:32:05.474+0000 7f34815bf740 1 Processor -- start 2024-04-23T10:32:05.481 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:32:05.474+0000 7f34815bf740 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x5638da617b60 0x5638da622540 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T10:32:05.481 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:32:05.474+0000 7f34815bf740 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x5638da618480 0x5638da61f830 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T10:32:05.481 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:32:05.474+0000 7f34815bf740 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x5638da622b50 0x5638da61fe40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T10:32:05.481 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:32:05.474+0000 7f34815bf740 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x5638da625940 con 0x5638da617b60 2024-04-23T10:32:05.481 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:32:05.474+0000 7f34815bf740 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 -- 0x5638da517670 con 0x5638da622b50 2024-04-23T10:32:05.481 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:32:05.474+0000 7f34815bf740 1 -- --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_getmap magic: 0 -- 0x5638da625ac0 con 0x5638da618480 2024-04-23T10:32:05.481 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:32:05.475+0000 7f34815bf740 10 client.0 _ll_register_callbacks cb 0x5638da517060 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-23T10:32:05.481 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:32:05.475+0000 7f34777fe640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x5638da617b60 0x5638da622540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T10:32:05.482 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:32:05.475+0000 7f34777fe640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x5638da617b60 0x5638da622540 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.114:52172/0 (socket says 172.21.15.114:52172) 2024-04-23T10:32:05.482 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:32:05.475+0000 7f34777fe640 1 -- 172.21.15.114:0/2104770368 learned_addr learned my addr 172.21.15.114:0/2104770368 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-04-23T10:32:05.482 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:32:05.475+0000 7f34777fe640 1 --2- 172.21.15.114:0/2104770368 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x5638da622b50 0x5638da61fe40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T10:32:05.482 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:32:05.475+0000 7f3477fff640 1 --2- 172.21.15.114:0/2104770368 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x5638da618480 0x5638da61f830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T10:32:05.482 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:32:05.475+0000 7f34777fe640 1 -- 172.21.15.114:0/2104770368 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x5638da618480 msgr2=0x5638da61f830 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T10:32:05.482 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:32:05.475+0000 7f34777fe640 1 --2- 172.21.15.114:0/2104770368 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x5638da618480 0x5638da61f830 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:32:05.482 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:32:05.475+0000 7f34777fe640 1 -- 172.21.15.114:0/2104770368 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x5638da622b50 msgr2=0x5638da61fe40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T10:32:05.482 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:32:05.475+0000 7f34777fe640 1 --2- 172.21.15.114:0/2104770368 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x5638da622b50 0x5638da61fe40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:32:05.482 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:32:05.475+0000 7f34777fe640 1 -- 172.21.15.114:0/2104770368 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5638da35bae0 con 0x5638da617b60 2024-04-23T10:32:05.482 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:32:05.475+0000 7f34777fe640 1 --2- 172.21.15.114:0/2104770368 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x5638da617b60 0x5638da622540 secure :-1 s=READY pgs=735 cs=0 l=1 rev1=1 crypto rx=0x7f346c0057a0 tx=0x7f346c005f60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5c90a7d7c7b571fc server_cookie=0 in_seq=0 out_seq=0 2024-04-23T10:32:05.482 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:32:05.475+0000 7f34577fe640 10 client.0 ms_handle_connect on v2:172.21.15.114:3300/0 2024-04-23T10:32:05.482 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:32:05.475+0000 7f34577fe640 1 -- 172.21.15.114:0/2104770368 <== mon.0 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f346c017040 con 0x5638da617b60 2024-04-23T10:32:05.482 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:32:05.475+0000 7f34577fe640 1 -- 172.21.15.114:0/2104770368 <== mon.0 v2:172.21.15.114:3300/0 2 ==== osd_map(94..94 src has 1..94) ==== 8585+0+0 (secure 0 0 0) 0x7f346c0048d0 con 0x5638da617b60 2024-04-23T10:32:05.483 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:32:05.475+0000 7f34815bf740 1 -- 172.21.15.114:0/2104770368 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f346c0054d0 con 0x5638da617b60 2024-04-23T10:32:05.483 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:32:05.475+0000 7f34577fe640 1 -- 172.21.15.114:0/2104770368 <== mon.0 v2:172.21.15.114:3300/0 3 ==== mon_get_version_reply(handle=1 version=30) ==== 24+0+0 (secure 0 0 0) 0x7f346c033050 con 0x5638da617b60 2024-04-23T10:32:05.483 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:32:05.475+0000 7f34815bf740 10 client.16386 fetch_fsmap learned FSMap version 30 2024-04-23T10:32:05.483 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:32:05.475+0000 7f34815bf740 1 -- 172.21.15.114:0/2104770368 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({config=0+,fsmap=30,monmap=4+}) -- 0x5638da3bc270 con 0x5638da617b60 2024-04-23T10:32:05.483 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:32:05.475+0000 7f34577fe640 1 -- 172.21.15.114:0/2104770368 <== mon.0 v2:172.21.15.114:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f346c0054d0 con 0x5638da617b60 2024-04-23T10:32:05.483 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:32:05.476+0000 7f34577fe640 1 -- 172.21.15.114:0/2104770368 <== mon.0 v2:172.21.15.114:3300/0 5 ==== fsmap(e 30) ==== 8338+0+0 (secure 0 0 0) 0x7f346c013400 con 0x5638da617b60 2024-04-23T10:32:05.483 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:32:05.476+0000 7f34815bf740 10 client.16386 fetch_fsmap finished waiting for FSMap version 30 2024-04-23T10:32:05.483 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:32:05.476+0000 7f34815bf740 10 client.16386 resolve_mds: validated gid 24545 aka daemon mds.b 2024-04-23T10:32:05.483 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:32:05.476+0000 7f34815bf740 20 client.16386 populate_metadata read hostname 'smithi114' 2024-04-23T10:32:05.483 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:32:05.476+0000 7f34815bf740 1 --2- 172.21.15.114:0/2104770368 >> [v2:172.21.15.119:6838/937297340,v1:172.21.15.119:6839/937297340] conn(0x5638da632c20 0x5638da635010 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T10:32:05.483 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:32:05.476+0000 7f34815bf740 4 client.16386 mds_command: new command op to 24545 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-23T10:32:05.483 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:32:05.476+0000 7f34815bf740 1 -- 172.21.15.114:0/2104770368 --> [v2:172.21.15.119:6838/937297340,v1:172.21.15.119:6839/937297340] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x5638da58ada0 con 0x5638da632c20 2024-04-23T10:32:05.483 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:32:05.476+0000 7f3477fff640 1 --2- 172.21.15.114:0/2104770368 >> [v2:172.21.15.119:6838/937297340,v1:172.21.15.119:6839/937297340] conn(0x5638da632c20 0x5638da635010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T10:32:05.483 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:32:05.476+0000 7f3477fff640 1 --2- 172.21.15.114:0/2104770368 >> [v2:172.21.15.119:6838/937297340,v1:172.21.15.119:6839/937297340] conn(0x5638da632c20 0x5638da635010 crc :-1 s=READY pgs=108 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=bcd6c8387e8f645 server_cookie=ebe59d9fb00bf2b6 in_seq=0 out_seq=0 2024-04-23T10:32:05.483 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:32:05.476+0000 7f34577fe640 10 client.16386 ms_handle_connect on v2:172.21.15.119:6838/937297340 2024-04-23T10:32:05.484 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:32:05.477+0000 7f34577fe640 1 -- 172.21.15.114:0/2104770368 <== mds.0 v2:172.21.15.119:6838/937297340 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x5638da58ada0 con 0x5638da632c20 2024-04-23T10:32:05.484 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:32:05.477+0000 7f34577fe640 10 client.16386 handle_command_reply: tid=0 2024-04-23T10:32:05.484 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:32:05.477+0000 7f34815bf740 1 client.16386 shutdown 2024-04-23T10:32:05.484 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:32:05.477+0000 7f34815bf740 1 -- 172.21.15.114:0/2104770368 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x5638da617b60 msgr2=0x5638da622540 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T10:32:05.484 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:32:05.477+0000 7f34815bf740 1 --2- 172.21.15.114:0/2104770368 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x5638da617b60 0x5638da622540 secure :-1 s=READY pgs=735 cs=0 l=1 rev1=1 crypto rx=0x7f346c0057a0 tx=0x7f346c005f60 comp rx=0 tx=0).stop 2024-04-23T10:32:05.484 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:32:05.477+0000 7f34815bf740 1 -- 172.21.15.114:0/2104770368 shutdown_connections 2024-04-23T10:32:05.484 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:32:05.477+0000 7f34815bf740 1 --2- 172.21.15.114:0/2104770368 >> [v2:172.21.15.119:6838/937297340,v1:172.21.15.119:6839/937297340] conn(0x5638da632c20 0x5638da635010 crc :-1 s=READY pgs=108 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:32:05.484 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:32:05.477+0000 7f34815bf740 1 --2- 172.21.15.114:0/2104770368 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x5638da622b50 0x5638da61fe40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:32:05.484 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:32:05.477+0000 7f34577fe640 0 client.16386 ms_handle_reset on v2:172.21.15.119:6838/937297340 2024-04-23T10:32:05.484 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:32:05.477+0000 7f34815bf740 1 --2- 172.21.15.114:0/2104770368 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x5638da618480 0x5638da61f830 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:32:05.484 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:32:05.477+0000 7f34815bf740 1 --2- 172.21.15.114:0/2104770368 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x5638da617b60 0x5638da622540 unknown :-1 s=CLOSED pgs=735 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:32:05.484 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:32:05.477+0000 7f34815bf740 1 -- 172.21.15.114:0/2104770368 >> 172.21.15.114:0/2104770368 conn(0x5638da613e80 msgr2=0x5638da619b80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T10:32:05.484 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:32:05.477+0000 7f34815bf740 1 -- 172.21.15.114:0/2104770368 shutdown_connections 2024-04-23T10:32:05.484 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:32:05.477+0000 7f34815bf740 1 -- 172.21.15.114:0/2104770368 wait complete. 2024-04-23T10:32:05.484 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:32:05.477+0000 7f34815bf740 20 client.16386 trim_cache size 0 max 16384 2024-04-23T10:32:05.498 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:32:05.494+0000 7f34815bf740 1 Processor -- start 2024-04-23T10:32:05.498 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:32:05.494+0000 7f34815bf740 1 -- start start 2024-04-23T10:32:05.498 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:32:05.495+0000 7f34815bf740 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x5638da617b60 0x5638da61e860 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T10:32:05.498 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:32:05.495+0000 7f34815bf740 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x5638da618480 0x5638da622070 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T10:32:05.498 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:32:05.495+0000 7f34815bf740 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x5638da622b50 0x5638da63a6a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T10:32:05.498 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:32:05.495+0000 7f34815bf740 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x7f34781b0550 con 0x5638da618480 2024-04-23T10:32:05.498 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:32:05.495+0000 7f34815bf740 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 -- 0x7f34781ade20 con 0x5638da617b60 2024-04-23T10:32:05.498 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:32:05.495+0000 7f34815bf740 1 -- --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_getmap magic: 0 -- 0x7f34781abc70 con 0x5638da622b50 2024-04-23T10:32:05.498 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:32:05.495+0000 7f3477fff640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x5638da618480 0x5638da622070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T10:32:05.498 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:32:05.495+0000 7f3477fff640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x5638da618480 0x5638da622070 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.114:52180/0 (socket says 172.21.15.114:52180) 2024-04-23T10:32:05.498 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:32:05.495+0000 7f3477fff640 1 -- 172.21.15.114:0/2909510110 learned_addr learned my addr 172.21.15.114:0/2909510110 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-04-23T10:32:05.498 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:32:05.495+0000 7f34777fe640 1 --2- 172.21.15.114:0/2909510110 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x5638da617b60 0x5638da61e860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T10:32:05.498 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:32:05.495+0000 7f3477fff640 1 -- 172.21.15.114:0/2909510110 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x5638da622b50 msgr2=0x5638da63a6a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T10:32:05.499 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:32:05.495+0000 7f3477fff640 1 --2- 172.21.15.114:0/2909510110 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x5638da622b50 0x5638da63a6a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:32:05.499 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:32:05.495+0000 7f3477fff640 1 -- 172.21.15.114:0/2909510110 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x5638da617b60 msgr2=0x5638da61e860 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T10:32:05.499 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:32:05.495+0000 7f3477fff640 1 --2- 172.21.15.114:0/2909510110 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x5638da617b60 0x5638da61e860 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:32:05.499 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:32:05.495+0000 7f3477fff640 1 -- 172.21.15.114:0/2909510110 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5638da6182b0 con 0x5638da618480 2024-04-23T10:32:05.499 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:32:05.495+0000 7f34777fe640 1 --2- 172.21.15.114:0/2909510110 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x5638da617b60 0x5638da61e860 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-23T10:32:05.499 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:32:05.495+0000 7f3477fff640 1 --2- 172.21.15.114:0/2909510110 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x5638da618480 0x5638da622070 secure :-1 s=READY pgs=736 cs=0 l=1 rev1=1 crypto rx=0x5638da61f090 tx=0x7f3468005f90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e492c0a32b02d4b6 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T10:32:05.499 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:32:05.495+0000 7f345dffb640 1 -- 172.21.15.114:0/2909510110 <== mon.0 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3468004250 con 0x5638da618480 2024-04-23T10:32:05.499 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:32:05.495+0000 7f345dffb640 1 -- 172.21.15.114:0/2909510110 <== mon.0 v2:172.21.15.114:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f34680043f0 con 0x5638da618480 2024-04-23T10:32:05.499 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:32:05.495+0000 7f345dffb640 1 -- 172.21.15.114:0/2909510110 <== mon.0 v2:172.21.15.114:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3468004da0 con 0x5638da618480 2024-04-23T10:32:05.499 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:32:05.496+0000 7f34815bf740 1 -- 172.21.15.114:0/2909510110 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x5638da618480 msgr2=0x5638da622070 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T10:32:05.499 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:32:05.496+0000 7f34815bf740 1 --2- 172.21.15.114:0/2909510110 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x5638da618480 0x5638da622070 secure :-1 s=READY pgs=736 cs=0 l=1 rev1=1 crypto rx=0x5638da61f090 tx=0x7f3468005f90 comp rx=0 tx=0).stop 2024-04-23T10:32:05.499 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:32:05.496+0000 7f34815bf740 1 -- 172.21.15.114:0/2909510110 shutdown_connections 2024-04-23T10:32:05.499 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:32:05.496+0000 7f34815bf740 1 --2- 172.21.15.114:0/2909510110 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x5638da622b50 0x5638da63a6a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:32:05.499 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:32:05.496+0000 7f34815bf740 1 --2- 172.21.15.114:0/2909510110 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x5638da618480 0x5638da622070 unknown :-1 s=CLOSED pgs=736 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:32:05.499 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:32:05.496+0000 7f34815bf740 1 --2- 172.21.15.114:0/2909510110 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x5638da617b60 0x5638da61e860 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:32:05.500 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:32:05.496+0000 7f34815bf740 1 -- 172.21.15.114:0/2909510110 >> 172.21.15.114:0/2909510110 conn(0x5638da613e80 msgr2=0x5638da613050 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T10:32:05.500 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:32:05.496+0000 7f34815bf740 1 -- 172.21.15.114:0/2909510110 shutdown_connections 2024-04-23T10:32:05.500 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:32:05.496+0000 7f34815bf740 1 -- 172.21.15.114:0/2909510110 wait complete. 2024-04-23T10:32:05.500 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:32:05.496+0000 7f34815bf740 1 -- start start 2024-04-23T10:32:05.500 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:32:05.496+0000 7f34815bf740 1 Processor -- start 2024-04-23T10:32:05.500 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:32:05.497+0000 7f34815bf740 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x5638da617b60 0x5638da62af70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T10:32:05.500 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:32:05.497+0000 7f34815bf740 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x5638da618480 0x5638da625d30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T10:32:05.500 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:32:05.497+0000 7f34815bf740 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x5638da622b50 0x5638da626270 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T10:32:05.500 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:32:05.497+0000 7f34815bf740 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x7f34781ac2e0 con 0x5638da622b50 2024-04-23T10:32:05.500 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:32:05.497+0000 7f34815bf740 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 -- 0x7f34781ad750 con 0x5638da617b60 2024-04-23T10:32:05.500 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:32:05.497+0000 7f34815bf740 1 -- --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_getmap magic: 0 -- 0x7f34781ac9b0 con 0x5638da618480 2024-04-23T10:32:05.500 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:32:05.497+0000 7f34815bf740 10 client.0 _ll_register_callbacks cb 0x7f346c010d90 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-23T10:32:05.500 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:32:05.497+0000 7f34777fe640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x5638da622b50 0x5638da626270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T10:32:05.500 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:32:05.497+0000 7f34777fe640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x5638da622b50 0x5638da626270 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.114:52190/0 (socket says 172.21.15.114:52190) 2024-04-23T10:32:05.500 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:32:05.497+0000 7f34777fe640 1 -- 172.21.15.114:0/1636762081 learned_addr learned my addr 172.21.15.114:0/1636762081 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-04-23T10:32:05.501 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:32:05.497+0000 7f3477fff640 1 --2- 172.21.15.114:0/1636762081 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x5638da618480 0x5638da625d30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T10:32:05.501 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:32:05.497+0000 7f34777fe640 1 -- 172.21.15.114:0/1636762081 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x5638da618480 msgr2=0x5638da625d30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T10:32:05.501 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:32:05.497+0000 7f34777fe640 1 --2- 172.21.15.114:0/1636762081 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x5638da618480 0x5638da625d30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:32:05.501 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:32:05.497+0000 7f34777fe640 1 -- 172.21.15.114:0/1636762081 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x5638da617b60 msgr2=0x5638da62af70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T10:32:05.501 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:32:05.497+0000 7f34777fe640 1 --2- 172.21.15.114:0/1636762081 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x5638da617b60 0x5638da62af70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:32:05.501 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:32:05.497+0000 7f34777fe640 1 -- 172.21.15.114:0/1636762081 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5638da24f3d0 con 0x5638da622b50 2024-04-23T10:32:05.501 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:32:05.497+0000 7f34777fe640 1 --2- 172.21.15.114:0/1636762081 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x5638da622b50 0x5638da626270 secure :-1 s=READY pgs=737 cs=0 l=1 rev1=1 crypto rx=0x7f346c0096a0 tx=0x7f346c009560 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9b536002103d3172 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T10:32:05.501 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:32:05.497+0000 7f34577fe640 10 client.0 ms_handle_connect on v2:172.21.15.114:3300/0 2024-04-23T10:32:05.501 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:32:05.497+0000 7f34577fe640 1 -- 172.21.15.114:0/1636762081 <== mon.0 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f346c017070 con 0x5638da622b50 2024-04-23T10:32:05.501 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:32:05.497+0000 7f34577fe640 1 -- 172.21.15.114:0/1636762081 <== mon.0 v2:172.21.15.114:3300/0 2 ==== osd_map(94..94 src has 1..94) ==== 8585+0+0 (secure 0 0 0) 0x7f346c013400 con 0x5638da622b50 2024-04-23T10:32:05.501 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:32:05.497+0000 7f34815bf740 1 -- 172.21.15.114:0/1636762081 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f34680043f0 con 0x5638da622b50 2024-04-23T10:32:05.501 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:32:05.497+0000 7f34577fe640 1 -- 172.21.15.114:0/1636762081 <== mon.0 v2:172.21.15.114:3300/0 3 ==== mon_get_version_reply(handle=1 version=30) ==== 24+0+0 (secure 0 0 0) 0x7f346c013600 con 0x5638da622b50 2024-04-23T10:32:05.501 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:32:05.497+0000 7f34815bf740 10 client.16392 fetch_fsmap learned FSMap version 30 2024-04-23T10:32:05.501 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:32:05.497+0000 7f34815bf740 1 -- 172.21.15.114:0/1636762081 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({config=0+,fsmap=30,monmap=4+}) -- 0x5638da50be50 con 0x5638da622b50 2024-04-23T10:32:05.501 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:32:05.498+0000 7f34577fe640 1 -- 172.21.15.114:0/1636762081 <== mon.0 v2:172.21.15.114:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f34680043f0 con 0x5638da622b50 2024-04-23T10:32:05.502 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:32:05.498+0000 7f34577fe640 1 -- 172.21.15.114:0/1636762081 <== mon.0 v2:172.21.15.114:3300/0 5 ==== fsmap(e 30) ==== 8338+0+0 (secure 0 0 0) 0x7f346c02c5b0 con 0x5638da622b50 2024-04-23T10:32:05.502 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:32:05.498+0000 7f34815bf740 10 client.16392 fetch_fsmap finished waiting for FSMap version 30 2024-04-23T10:32:05.502 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:32:05.498+0000 7f34815bf740 10 client.16392 resolve_mds: validated gid 24545 aka daemon mds.b 2024-04-23T10:32:05.502 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:32:05.498+0000 7f34815bf740 20 client.16392 populate_metadata read hostname 'smithi114' 2024-04-23T10:32:05.502 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:32:05.498+0000 7f34815bf740 1 --2- 172.21.15.114:0/1636762081 >> [v2:172.21.15.119:6838/937297340,v1:172.21.15.119:6839/937297340] conn(0x5638da699f30 0x5638da69c350 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T10:32:05.502 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:32:05.498+0000 7f34815bf740 4 client.16392 mds_command: new command op to 24545 tid=0 multi_id=0 [{"prefix": "quiesce db", "set_id": "11138c86", "await_for": 42.0}] 2024-04-23T10:32:05.502 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:32:05.498+0000 7f34815bf740 1 -- 172.21.15.114:0/1636762081 --> [v2:172.21.15.119:6838/937297340,v1:172.21.15.119:6839/937297340] -- command(tid 0: {"prefix": "quiesce db", "set_id": "11138c86", "await_for": 42.0}) -- 0x5638da584c10 con 0x5638da699f30 2024-04-23T10:32:05.502 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:32:05.498+0000 7f3477fff640 1 --2- 172.21.15.114:0/1636762081 >> [v2:172.21.15.119:6838/937297340,v1:172.21.15.119:6839/937297340] conn(0x5638da699f30 0x5638da69c350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T10:32:05.502 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:32:05.499+0000 7f34577fe640 10 client.16392 ms_handle_connect on v2:172.21.15.119:6838/937297340 2024-04-23T10:32:05.502 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:32:05.499+0000 7f3477fff640 1 --2- 172.21.15.114:0/1636762081 >> [v2:172.21.15.119:6838/937297340,v1:172.21.15.119:6839/937297340] conn(0x5638da699f30 0x5638da69c350 crc :-1 s=READY pgs=109 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=6eb55855cca144aa server_cookie=7ebf33ea7ad7cb8a in_seq=0 out_seq=0 2024-04-23T10:32:05.922 INFO:tasks.workunit.client.0.smithi114.stdout: 1 305661 5.13 MB/sec execute 134 sec latency 48146.670 ms 2024-04-23T10:32:06.492 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 23 10:32:06 smithi119 ceph-mon[32608]: pgmap v169: 97 pgs: 97 active+clean; 78 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 44 KiB/s rd, 83 KiB/s wr, 30 op/s 2024-04-23T10:32:06.492 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 23 10:32:06 smithi119 ceph-mon[32608]: from='client.? 172.21.15.114:0/2628387773' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-23T10:32:06.504 INFO:journalctl@ceph.mon.a.smithi114.stdout:Apr 23 10:32:06 smithi114 ceph-mon[29255]: pgmap v169: 97 pgs: 97 active+clean; 78 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 44 KiB/s rd, 83 KiB/s wr, 30 op/s 2024-04-23T10:32:06.504 INFO:journalctl@ceph.mon.a.smithi114.stdout:Apr 23 10:32:06 smithi114 ceph-mon[29255]: from='client.? 172.21.15.114:0/2628387773' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-23T10:32:06.577 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 23 10:32:06 smithi204 ceph-mon[32649]: pgmap v169: 97 pgs: 97 active+clean; 78 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 44 KiB/s rd, 83 KiB/s wr, 30 op/s 2024-04-23T10:32:06.578 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 23 10:32:06 smithi204 ceph-mon[32649]: from='client.? 172.21.15.114:0/2628387773' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-23T10:32:06.922 INFO:tasks.workunit.client.0.smithi114.stdout: 1 305661 5.09 MB/sec execute 135 sec latency 49146.779 ms 2024-04-23T10:32:07.922 INFO:tasks.workunit.client.0.smithi114.stdout: 1 305661 5.06 MB/sec execute 136 sec latency 50146.892 ms 2024-04-23T10:32:08.492 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 23 10:32:08 smithi119 ceph-mon[32608]: pgmap v170: 97 pgs: 97 active+clean; 78 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 45 KiB/s rd, 83 KiB/s wr, 33 op/s 2024-04-23T10:32:08.504 INFO:journalctl@ceph.mon.a.smithi114.stdout:Apr 23 10:32:08 smithi114 ceph-mon[29255]: pgmap v170: 97 pgs: 97 active+clean; 78 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 45 KiB/s rd, 83 KiB/s wr, 33 op/s 2024-04-23T10:32:08.578 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 23 10:32:08 smithi204 ceph-mon[32649]: pgmap v170: 97 pgs: 97 active+clean; 78 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 45 KiB/s rd, 83 KiB/s wr, 33 op/s 2024-04-23T10:32:08.923 INFO:tasks.workunit.client.0.smithi114.stdout: 1 305661 5.02 MB/sec execute 137 sec latency 51146.945 ms 2024-04-23T10:32:09.923 INFO:tasks.workunit.client.0.smithi114.stdout: 1 305661 4.98 MB/sec execute 138 sec latency 52146.979 ms 2024-04-23T10:32:10.492 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 23 10:32:10 smithi119 ceph-mon[32608]: pgmap v171: 97 pgs: 97 active+clean; 78 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 22 KiB/s rd, 36 KiB/s wr, 25 op/s 2024-04-23T10:32:10.504 INFO:journalctl@ceph.mon.a.smithi114.stdout:Apr 23 10:32:10 smithi114 ceph-mon[29255]: pgmap v171: 97 pgs: 97 active+clean; 78 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 22 KiB/s rd, 36 KiB/s wr, 25 op/s 2024-04-23T10:32:10.577 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 23 10:32:10 smithi204 ceph-mon[32649]: pgmap v171: 97 pgs: 97 active+clean; 78 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 22 KiB/s rd, 36 KiB/s wr, 25 op/s 2024-04-23T10:32:10.923 INFO:tasks.workunit.client.0.smithi114.stdout: 1 305661 4.95 MB/sec execute 139 sec latency 53147.113 ms 2024-04-23T10:32:11.768 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:32:11.766+0000 7f65882a7640 1 -- 172.21.15.114:0/3973886584 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f6580103aa0 msgr2=0x7f658010bf80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T10:32:11.768 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:32:11.766+0000 7f65882a7640 1 --2- 172.21.15.114:0/3973886584 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f6580103aa0 0x7f658010bf80 secure :-1 s=READY pgs=738 cs=0 l=1 rev1=1 crypto rx=0x7f6570009900 tx=0x7f657002a480 comp rx=0 tx=0).stop 2024-04-23T10:32:11.768 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:32:11.766+0000 7f65882a7640 1 -- 172.21.15.114:0/3973886584 shutdown_connections 2024-04-23T10:32:11.768 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:32:11.766+0000 7f65882a7640 1 --2- 172.21.15.114:0/3973886584 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f658010c550 0x7f658010e9d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:32:11.768 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:32:11.766+0000 7f65882a7640 1 --2- 172.21.15.114:0/3973886584 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f6580103aa0 0x7f658010bf80 unknown :-1 s=CLOSED pgs=738 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:32:11.768 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:32:11.766+0000 7f65882a7640 1 --2- 172.21.15.114:0/3973886584 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f6580103180 0x7f6580103560 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:32:11.768 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:32:11.766+0000 7f65882a7640 1 -- 172.21.15.114:0/3973886584 >> 172.21.15.114:0/3973886584 conn(0x7f6580077fd0 msgr2=0x7f65800783e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T10:32:11.769 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:32:11.766+0000 7f65882a7640 1 -- 172.21.15.114:0/3973886584 shutdown_connections 2024-04-23T10:32:11.769 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:32:11.766+0000 7f65882a7640 1 -- 172.21.15.114:0/3973886584 wait complete. 2024-04-23T10:32:11.769 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:32:11.767+0000 7f65882a7640 1 Processor -- start 2024-04-23T10:32:11.769 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:32:11.767+0000 7f65882a7640 1 -- start start 2024-04-23T10:32:11.769 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:32:11.767+0000 7f65882a7640 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f6580103180 0x7f6580198670 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T10:32:11.769 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:32:11.767+0000 7f65882a7640 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f6580103aa0 0x7f6580198bb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T10:32:11.769 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:32:11.767+0000 7f65882a7640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f658010c550 0x7f658019cf30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T10:32:11.769 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:32:11.767+0000 7f65882a7640 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x7f65801121d0 con 0x7f658010c550 2024-04-23T10:32:11.769 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:32:11.767+0000 7f65882a7640 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 -- 0x7f6580112050 con 0x7f6580103aa0 2024-04-23T10:32:11.769 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:32:11.767+0000 7f65882a7640 1 -- --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_getmap magic: 0 -- 0x7f6580112350 con 0x7f6580103180 2024-04-23T10:32:11.769 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:32:11.767+0000 7f6586844640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f658010c550 0x7f658019cf30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T10:32:11.769 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:32:11.767+0000 7f6586844640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f658010c550 0x7f658019cf30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.114:50470/0 (socket says 172.21.15.114:50470) 2024-04-23T10:32:11.769 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:32:11.767+0000 7f6586844640 1 -- 172.21.15.114:0/3325722966 learned_addr learned my addr 172.21.15.114:0/3325722966 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-04-23T10:32:11.769 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:32:11.767+0000 7f6585842640 1 --2- 172.21.15.114:0/3325722966 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f6580103aa0 0x7f6580198bb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T10:32:11.770 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:32:11.767+0000 7f6586043640 1 --2- 172.21.15.114:0/3325722966 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f6580103180 0x7f6580198670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T10:32:11.770 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:32:11.768+0000 7f6586844640 1 -- 172.21.15.114:0/3325722966 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f6580103180 msgr2=0x7f6580198670 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T10:32:11.770 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:32:11.768+0000 7f6586844640 1 --2- 172.21.15.114:0/3325722966 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f6580103180 0x7f6580198670 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:32:11.770 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:32:11.768+0000 7f6586844640 1 -- 172.21.15.114:0/3325722966 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f6580103aa0 msgr2=0x7f6580198bb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T10:32:11.770 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:32:11.768+0000 7f6586844640 1 --2- 172.21.15.114:0/3325722966 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f6580103aa0 0x7f6580198bb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:32:11.770 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:32:11.768+0000 7f6586844640 1 -- 172.21.15.114:0/3325722966 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f658019d660 con 0x7f658010c550 2024-04-23T10:32:11.770 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:32:11.768+0000 7f6586844640 1 --2- 172.21.15.114:0/3325722966 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f658010c550 0x7f658019cf30 secure :-1 s=READY pgs=739 cs=0 l=1 rev1=1 crypto rx=0x7f657800cce0 tx=0x7f657800a620 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T10:32:11.770 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:32:11.768+0000 7f65777fe640 1 -- 172.21.15.114:0/3325722966 <== mon.0 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f657800f450 con 0x7f658010c550 2024-04-23T10:32:11.770 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:32:11.768+0000 7f65777fe640 1 -- 172.21.15.114:0/3325722966 <== mon.0 v2:172.21.15.114:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f657800faa0 con 0x7f658010c550 2024-04-23T10:32:11.770 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:32:11.768+0000 7f65882a7640 1 -- 172.21.15.114:0/3325722966 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f658019d950 con 0x7f658010c550 2024-04-23T10:32:11.770 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:32:11.768+0000 7f65777fe640 1 -- 172.21.15.114:0/3325722966 <== mon.0 v2:172.21.15.114:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6578018b70 con 0x7f658010c550 2024-04-23T10:32:11.770 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:32:11.768+0000 7f65882a7640 1 -- 172.21.15.114:0/3325722966 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f65801a51e0 con 0x7f658010c550 2024-04-23T10:32:11.777 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:32:11.770+0000 7f65777fe640 1 -- 172.21.15.114:0/3325722966 <== mon.0 v2:172.21.15.114:3300/0 4 ==== mgrmap(e 26) ==== 138778+0+0 (secure 0 0 0) 0x7f6578018d10 con 0x7f658010c550 2024-04-23T10:32:11.777 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:32:11.771+0000 7f658afc0740 1 Processor -- start 2024-04-23T10:32:11.777 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:32:11.771+0000 7f658afc0740 1 -- start start 2024-04-23T10:32:11.777 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:32:11.771+0000 7f658afc0740 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x56362f26a620 0x56362f26aa00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T10:32:11.777 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:32:11.771+0000 7f658afc0740 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x56362f26af40 0x56362f2752c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T10:32:11.777 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:32:11.771+0000 7f658afc0740 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x56362f275800 0x56362f277bf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T10:32:11.778 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:32:11.771+0000 7f658afc0740 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x56362f1cf970 con 0x56362f26a620 2024-04-23T10:32:11.778 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:32:11.771+0000 7f658afc0740 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 -- 0x56362f16c4a0 con 0x56362f26af40 2024-04-23T10:32:11.778 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:32:11.771+0000 7f658afc0740 1 -- --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_getmap magic: 0 -- 0x56362f1cb930 con 0x56362f275800 2024-04-23T10:32:11.778 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:32:11.771+0000 7f65777fe640 1 --2- 172.21.15.114:0/3325722966 >> [v2:172.21.15.114:6800/1395693466,v1:172.21.15.114:6801/1395693466] conn(0x7f656c0a3dc0 0x7f656c0a6280 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T10:32:11.778 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:32:11.772+0000 7f6585842640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x56362f26a620 0x56362f26aa00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T10:32:11.778 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:32:11.772+0000 7f6585842640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x56362f26a620 0x56362f26aa00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.114:50474/0 (socket says 172.21.15.114:50474) 2024-04-23T10:32:11.778 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:32:11.772+0000 7f6586844640 1 --2- 172.21.15.114:0/3325722966 >> [v2:172.21.15.114:6800/1395693466,v1:172.21.15.114:6801/1395693466] conn(0x7f656c0a3dc0 0x7f656c0a6280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T10:32:11.778 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:32:11.772+0000 7f6585842640 1 -- 172.21.15.114:0/4159159148 learned_addr learned my addr 172.21.15.114:0/4159159148 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-04-23T10:32:11.778 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:32:11.772+0000 7f6586043640 1 --2- 172.21.15.114:0/4159159148 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x56362f26af40 0x56362f2752c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T10:32:11.778 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:32:11.772+0000 7f6585842640 1 --2- 172.21.15.114:0/4159159148 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x56362f275800 0x56362f277bf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T10:32:11.778 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:32:11.772+0000 7f65777fe640 1 -- 172.21.15.114:0/3325722966 <== mon.0 v2:172.21.15.114:3300/0 5 ==== osd_map(94..94 src has 1..94) ==== 8585+0+0 (secure 0 0 0) 0x7f65780d0c50 con 0x7f658010c550 2024-04-23T10:32:11.778 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:32:11.772+0000 7f6585842640 1 -- 172.21.15.114:0/4159159148 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x56362f275800 msgr2=0x56362f277bf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T10:32:11.778 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:32:11.772+0000 7f6585842640 1 --2- 172.21.15.114:0/4159159148 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x56362f275800 0x56362f277bf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:32:11.778 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:32:11.772+0000 7f6585842640 1 -- 172.21.15.114:0/4159159148 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x56362f26af40 msgr2=0x56362f2752c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T10:32:11.779 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:32:11.772+0000 7f6585842640 1 --2- 172.21.15.114:0/4159159148 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x56362f26af40 0x56362f2752c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:32:11.779 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:32:11.772+0000 7f6585842640 1 -- 172.21.15.114:0/4159159148 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x56362eee2210 con 0x56362f26a620 2024-04-23T10:32:11.779 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:32:11.772+0000 7f6586844640 1 --2- 172.21.15.114:0/3325722966 >> [v2:172.21.15.114:6800/1395693466,v1:172.21.15.114:6801/1395693466] conn(0x7f656c0a3dc0 0x7f656c0a6280 secure :-1 s=READY pgs=109 cs=0 l=1 rev1=1 crypto rx=0x7f65780ce410 tx=0x7f657806e5f0 comp rx=0 tx=0).ready entity=mgr.15291 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T10:32:11.779 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:32:11.772+0000 7f6585842640 1 --2- 172.21.15.114:0/4159159148 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x56362f26a620 0x56362f26aa00 secure :-1 s=READY pgs=740 cs=0 l=1 rev1=1 crypto rx=0x56362f267cc0 tx=0x7f6570005e60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=df426d668732ca46 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T10:32:11.779 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:32:11.772+0000 7f6574bfe640 1 -- 172.21.15.114:0/4159159148 <== mon.0 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6570044020 con 0x56362f26a620 2024-04-23T10:32:11.779 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:32:11.772+0000 7f6574bfe640 1 -- 172.21.15.114:0/4159159148 <== mon.0 v2:172.21.15.114:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f65700317f0 con 0x56362f26a620 2024-04-23T10:32:11.779 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:32:11.772+0000 7f6574bfe640 1 -- 172.21.15.114:0/4159159148 <== mon.0 v2:172.21.15.114:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6570035040 con 0x56362f26a620 2024-04-23T10:32:11.779 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:32:11.773+0000 7f658afc0740 1 -- 172.21.15.114:0/4159159148 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x56362f26a620 msgr2=0x56362f26aa00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T10:32:11.779 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:32:11.773+0000 7f658afc0740 1 --2- 172.21.15.114:0/4159159148 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x56362f26a620 0x56362f26aa00 secure :-1 s=READY pgs=740 cs=0 l=1 rev1=1 crypto rx=0x56362f267cc0 tx=0x7f6570005e60 comp rx=0 tx=0).stop 2024-04-23T10:32:11.779 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:32:11.773+0000 7f658afc0740 1 -- 172.21.15.114:0/4159159148 shutdown_connections 2024-04-23T10:32:11.779 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:32:11.773+0000 7f658afc0740 1 --2- 172.21.15.114:0/4159159148 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x56362f275800 0x56362f277bf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:32:11.779 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:32:11.773+0000 7f658afc0740 1 --2- 172.21.15.114:0/4159159148 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x56362f26af40 0x56362f2752c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:32:11.779 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:32:11.773+0000 7f658afc0740 1 --2- 172.21.15.114:0/4159159148 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x56362f26a620 0x56362f26aa00 unknown :-1 s=CLOSED pgs=740 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:32:11.779 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:32:11.773+0000 7f658afc0740 1 -- 172.21.15.114:0/4159159148 >> 172.21.15.114:0/4159159148 conn(0x56362f266b30 msgr2=0x56362f266f10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T10:32:11.779 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:32:11.773+0000 7f658afc0740 1 -- 172.21.15.114:0/4159159148 shutdown_connections 2024-04-23T10:32:11.779 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:32:11.773+0000 7f658afc0740 1 -- 172.21.15.114:0/4159159148 wait complete. 2024-04-23T10:32:11.780 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:32:11.774+0000 7f658afc0740 1 -- start start 2024-04-23T10:32:11.780 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:32:11.774+0000 7f658afc0740 1 Processor -- start 2024-04-23T10:32:11.780 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:32:11.774+0000 7f658afc0740 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x56362f26a620 0x56362f275140 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T10:32:11.780 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:32:11.774+0000 7f658afc0740 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x56362f26af40 0x56362f272430 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T10:32:11.780 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:32:11.774+0000 7f658afc0740 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x56362f275800 0x56362f272970 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T10:32:11.780 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:32:11.774+0000 7f658afc0740 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x56362f2783d0 con 0x56362f26a620 2024-04-23T10:32:11.780 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:32:11.774+0000 7f658afc0740 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 -- 0x56362f16a0b0 con 0x56362f275800 2024-04-23T10:32:11.780 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:32:11.774+0000 7f658afc0740 1 -- --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_getmap magic: 0 -- 0x56362f278550 con 0x56362f26af40 2024-04-23T10:32:11.780 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:32:11.774+0000 7f658afc0740 10 client.0 _ll_register_callbacks cb 0x56362f157d30 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-23T10:32:11.780 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:32:11.774+0000 7f6585842640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x56362f26a620 0x56362f275140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T10:32:11.780 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:32:11.774+0000 7f6585842640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x56362f26a620 0x56362f275140 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.114:50482/0 (socket says 172.21.15.114:50482) 2024-04-23T10:32:11.780 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:32:11.774+0000 7f6585842640 1 -- 172.21.15.114:0/1660652003 learned_addr learned my addr 172.21.15.114:0/1660652003 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-04-23T10:32:11.780 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:32:11.774+0000 7f6585842640 1 --2- 172.21.15.114:0/1660652003 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x56362f275800 0x56362f272970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T10:32:11.780 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:32:11.774+0000 7f6586043640 1 --2- 172.21.15.114:0/1660652003 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x56362f26af40 0x56362f272430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T10:32:11.780 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:32:11.774+0000 7f6585842640 1 -- 172.21.15.114:0/1660652003 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x56362f26af40 msgr2=0x56362f272430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T10:32:11.780 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:32:11.774+0000 7f6585842640 1 --2- 172.21.15.114:0/1660652003 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x56362f26af40 0x56362f272430 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:32:11.781 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:32:11.774+0000 7f6585842640 1 -- 172.21.15.114:0/1660652003 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x56362f275800 msgr2=0x56362f272970 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T10:32:11.781 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:32:11.774+0000 7f6585842640 1 --2- 172.21.15.114:0/1660652003 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x56362f275800 0x56362f272970 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:32:11.781 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:32:11.774+0000 7f6585842640 1 -- 172.21.15.114:0/1660652003 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({osdmap=0}) -- 0x56362efae620 con 0x56362f26a620 2024-04-23T10:32:11.781 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:32:11.774+0000 7f6585842640 1 --2- 172.21.15.114:0/1660652003 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x56362f26a620 0x56362f275140 secure :-1 s=READY pgs=741 cs=0 l=1 rev1=1 crypto rx=0x7f65700057b0 tx=0x7f65700054d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=daed7cec4f9692eb server_cookie=0 in_seq=0 out_seq=0 2024-04-23T10:32:11.781 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:32:11.774+0000 7f6569ffb640 10 client.0 ms_handle_connect on v2:172.21.15.114:3300/0 2024-04-23T10:32:11.781 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:32:11.775+0000 7f6569ffb640 1 -- 172.21.15.114:0/1660652003 <== mon.0 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f657004e070 con 0x56362f26a620 2024-04-23T10:32:11.781 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:32:11.775+0000 7f6569ffb640 1 -- 172.21.15.114:0/1660652003 <== mon.0 v2:172.21.15.114:3300/0 2 ==== osd_map(94..94 src has 1..94) ==== 8585+0+0 (secure 0 0 0) 0x7f65700037f0 con 0x56362f26a620 2024-04-23T10:32:11.781 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:32:11.775+0000 7f658afc0740 1 -- 172.21.15.114:0/1660652003 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f65700317f0 con 0x56362f26a620 2024-04-23T10:32:11.781 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:32:11.775+0000 7f6569ffb640 1 -- 172.21.15.114:0/1660652003 <== mon.0 v2:172.21.15.114:3300/0 3 ==== mon_get_version_reply(handle=1 version=30) ==== 24+0+0 (secure 0 0 0) 0x7f6570044070 con 0x56362f26a620 2024-04-23T10:32:11.781 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:32:11.775+0000 7f658afc0740 10 client.16404 fetch_fsmap learned FSMap version 30 2024-04-23T10:32:11.781 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:32:11.775+0000 7f658afc0740 1 -- 172.21.15.114:0/1660652003 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({config=0+,fsmap=30,monmap=4+}) -- 0x56362f284320 con 0x56362f26a620 2024-04-23T10:32:11.781 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:32:11.775+0000 7f6569ffb640 1 -- 172.21.15.114:0/1660652003 <== mon.0 v2:172.21.15.114:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f65700317f0 con 0x56362f26a620 2024-04-23T10:32:11.781 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:32:11.775+0000 7f6569ffb640 1 -- 172.21.15.114:0/1660652003 <== mon.0 v2:172.21.15.114:3300/0 5 ==== fsmap(e 30) ==== 8338+0+0 (secure 0 0 0) 0x7f657004a7b0 con 0x56362f26a620 2024-04-23T10:32:11.781 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:32:11.775+0000 7f658afc0740 10 client.16404 fetch_fsmap finished waiting for FSMap version 30 2024-04-23T10:32:11.781 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:32:11.775+0000 7f658afc0740 10 client.16404 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-04-23T10:32:11.781 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:32:11.775+0000 7f658afc0740 20 client.16404 populate_metadata read hostname 'smithi114' 2024-04-23T10:32:11.782 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:32:11.775+0000 7f658afc0740 1 --2- 172.21.15.114:0/1660652003 >> [v2:172.21.15.119:6838/937297340,v1:172.21.15.119:6839/937297340] conn(0x56362f285670 0x56362f287a60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T10:32:11.782 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:32:11.775+0000 7f658afc0740 4 client.16404 mds_command: new command op to 24545 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-23T10:32:11.782 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:32:11.775+0000 7f658afc0740 1 -- 172.21.15.114:0/1660652003 --> [v2:172.21.15.119:6838/937297340,v1:172.21.15.119:6839/937297340] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x56362f1dd390 con 0x56362f285670 2024-04-23T10:32:11.782 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:32:11.775+0000 7f6586043640 1 --2- 172.21.15.114:0/1660652003 >> [v2:172.21.15.119:6838/937297340,v1:172.21.15.119:6839/937297340] conn(0x56362f285670 0x56362f287a60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T10:32:11.782 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:32:11.776+0000 7f6569ffb640 10 client.16404 ms_handle_connect on v2:172.21.15.119:6838/937297340 2024-04-23T10:32:11.782 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:32:11.776+0000 7f6586043640 1 --2- 172.21.15.114:0/1660652003 >> [v2:172.21.15.119:6838/937297340,v1:172.21.15.119:6839/937297340] conn(0x56362f285670 0x56362f287a60 crc :-1 s=READY pgs=110 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=b91703a3478682cf server_cookie=19c6ef153fd978cf in_seq=0 out_seq=0 2024-04-23T10:32:11.782 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:32:11.776+0000 7f6569ffb640 1 -- 172.21.15.114:0/1660652003 <== mds.0 v2:172.21.15.119:6838/937297340 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x56362f1dd390 con 0x56362f285670 2024-04-23T10:32:11.782 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:32:11.776+0000 7f6569ffb640 10 client.16404 handle_command_reply: tid=0 2024-04-23T10:32:11.782 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:32:11.777+0000 7f658afc0740 1 client.16404 shutdown 2024-04-23T10:32:11.782 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:32:11.777+0000 7f658afc0740 1 -- 172.21.15.114:0/1660652003 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x56362f26a620 msgr2=0x56362f275140 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T10:32:11.782 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:32:11.777+0000 7f658afc0740 1 --2- 172.21.15.114:0/1660652003 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x56362f26a620 0x56362f275140 secure :-1 s=READY pgs=741 cs=0 l=1 rev1=1 crypto rx=0x7f65700057b0 tx=0x7f65700054d0 comp rx=0 tx=0).stop 2024-04-23T10:32:11.782 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:32:11.777+0000 7f658afc0740 1 -- 172.21.15.114:0/1660652003 shutdown_connections 2024-04-23T10:32:11.782 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:32:11.777+0000 7f658afc0740 1 --2- 172.21.15.114:0/1660652003 >> [v2:172.21.15.119:6838/937297340,v1:172.21.15.119:6839/937297340] conn(0x56362f285670 0x56362f287a60 crc :-1 s=READY pgs=110 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:32:11.782 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:32:11.777+0000 7f658afc0740 1 --2- 172.21.15.114:0/1660652003 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x56362f275800 0x56362f272970 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:32:11.782 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:32:11.777+0000 7f658afc0740 1 --2- 172.21.15.114:0/1660652003 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x56362f26af40 0x56362f272430 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:32:11.783 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:32:11.777+0000 7f6569ffb640 0 client.16404 ms_handle_reset on v2:172.21.15.119:6838/937297340 2024-04-23T10:32:11.783 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:32:11.777+0000 7f658afc0740 1 --2- 172.21.15.114:0/1660652003 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x56362f26a620 0x56362f275140 unknown :-1 s=CLOSED pgs=741 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:32:11.783 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:32:11.777+0000 7f658afc0740 1 -- 172.21.15.114:0/1660652003 >> 172.21.15.114:0/1660652003 conn(0x56362f266b30 msgr2=0x56362f26c8b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T10:32:11.783 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:32:11.777+0000 7f658afc0740 1 -- 172.21.15.114:0/1660652003 shutdown_connections 2024-04-23T10:32:11.783 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:32:11.777+0000 7f658afc0740 1 -- 172.21.15.114:0/1660652003 wait complete. 2024-04-23T10:32:11.783 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:32:11.777+0000 7f658afc0740 20 client.16404 trim_cache size 0 max 16384 2024-04-23T10:32:11.795 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:32:11.792+0000 7f658afc0740 1 Processor -- start 2024-04-23T10:32:11.795 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:32:11.792+0000 7f658afc0740 1 -- start start 2024-04-23T10:32:11.795 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:32:11.792+0000 7f658afc0740 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x56362f26a620 0x56362f265ef0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T10:32:11.795 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:32:11.792+0000 7f658afc0740 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x56362f26af40 0x56362f28aa90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T10:32:11.795 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:32:11.792+0000 7f658afc0740 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x56362f275800 0x56362f28afd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T10:32:11.795 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:32:11.792+0000 7f658afc0740 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x7f65801b03d0 con 0x56362f26af40 2024-04-23T10:32:11.796 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:32:11.792+0000 7f658afc0740 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 -- 0x7f65801adca0 con 0x56362f275800 2024-04-23T10:32:11.796 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:32:11.792+0000 7f658afc0740 1 -- --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_getmap magic: 0 -- 0x7f65801abaf0 con 0x56362f26a620 2024-04-23T10:32:11.796 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:32:11.792+0000 7f6586043640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x56362f26af40 0x56362f28aa90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T10:32:11.796 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:32:11.792+0000 7f6586043640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x56362f26af40 0x56362f28aa90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.114:50492/0 (socket says 172.21.15.114:50492) 2024-04-23T10:32:11.796 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:32:11.792+0000 7f6586043640 1 -- 172.21.15.114:0/3106469227 learned_addr learned my addr 172.21.15.114:0/3106469227 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-04-23T10:32:11.796 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:32:11.792+0000 7f6585842640 1 --2- 172.21.15.114:0/3106469227 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x56362f275800 0x56362f28afd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T10:32:11.796 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:32:11.792+0000 7f6585842640 1 --2- 172.21.15.114:0/3106469227 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x56362f26a620 0x56362f265ef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T10:32:11.796 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:32:11.792+0000 7f6586043640 1 -- 172.21.15.114:0/3106469227 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x56362f26a620 msgr2=0x56362f265ef0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T10:32:11.796 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:32:11.792+0000 7f6586043640 1 --2- 172.21.15.114:0/3106469227 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x56362f26a620 0x56362f265ef0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:32:11.796 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:32:11.792+0000 7f6586043640 1 -- 172.21.15.114:0/3106469227 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x56362f275800 msgr2=0x56362f28afd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T10:32:11.796 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:32:11.792+0000 7f6586043640 1 --2- 172.21.15.114:0/3106469227 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x56362f275800 0x56362f28afd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:32:11.796 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:32:11.792+0000 7f6586043640 1 -- 172.21.15.114:0/3106469227 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x56362f278450 con 0x56362f26af40 2024-04-23T10:32:11.796 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:32:11.792+0000 7f6585842640 1 --2- 172.21.15.114:0/3106469227 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x56362f26a620 0x56362f265ef0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-23T10:32:11.796 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:32:11.792+0000 7f6586043640 1 --2- 172.21.15.114:0/3106469227 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x56362f26af40 0x56362f28aa90 secure :-1 s=READY pgs=742 cs=0 l=1 rev1=1 crypto rx=0x56362f278680 tx=0x7f657c00b4d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=38b701960ab7783 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T10:32:11.796 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:32:11.793+0000 7f656bfff640 1 -- 172.21.15.114:0/3106469227 <== mon.0 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f657c01f070 con 0x56362f26af40 2024-04-23T10:32:11.797 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:32:11.793+0000 7f656bfff640 1 -- 172.21.15.114:0/3106469227 <== mon.0 v2:172.21.15.114:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f657c01bdc0 con 0x56362f26af40 2024-04-23T10:32:11.797 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:32:11.793+0000 7f656bfff640 1 -- 172.21.15.114:0/3106469227 <== mon.0 v2:172.21.15.114:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f657c004730 con 0x56362f26af40 2024-04-23T10:32:11.797 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:32:11.793+0000 7f658afc0740 1 -- 172.21.15.114:0/3106469227 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x56362f26af40 msgr2=0x56362f28aa90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T10:32:11.797 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:32:11.793+0000 7f658afc0740 1 --2- 172.21.15.114:0/3106469227 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x56362f26af40 0x56362f28aa90 secure :-1 s=READY pgs=742 cs=0 l=1 rev1=1 crypto rx=0x56362f278680 tx=0x7f657c00b4d0 comp rx=0 tx=0).stop 2024-04-23T10:32:11.797 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:32:11.793+0000 7f658afc0740 1 -- 172.21.15.114:0/3106469227 shutdown_connections 2024-04-23T10:32:11.797 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:32:11.793+0000 7f658afc0740 1 --2- 172.21.15.114:0/3106469227 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x56362f275800 0x56362f28afd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:32:11.797 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:32:11.793+0000 7f658afc0740 1 --2- 172.21.15.114:0/3106469227 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x56362f26af40 0x56362f28aa90 unknown :-1 s=CLOSED pgs=742 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:32:11.797 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:32:11.793+0000 7f658afc0740 1 --2- 172.21.15.114:0/3106469227 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x56362f26a620 0x56362f265ef0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:32:11.797 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:32:11.793+0000 7f658afc0740 1 -- 172.21.15.114:0/3106469227 >> 172.21.15.114:0/3106469227 conn(0x56362f266b30 msgr2=0x56362f281030 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T10:32:11.797 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:32:11.793+0000 7f658afc0740 1 -- 172.21.15.114:0/3106469227 shutdown_connections 2024-04-23T10:32:11.797 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:32:11.793+0000 7f658afc0740 1 -- 172.21.15.114:0/3106469227 wait complete. 2024-04-23T10:32:11.797 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:32:11.793+0000 7f658afc0740 1 -- start start 2024-04-23T10:32:11.797 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:32:11.794+0000 7f658afc0740 1 Processor -- start 2024-04-23T10:32:11.797 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:32:11.794+0000 7f658afc0740 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x56362f26a620 0x56362f2e3f90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T10:32:11.797 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:32:11.794+0000 7f658afc0740 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x56362f26af40 0x56362f2e44d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T10:32:11.798 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:32:11.794+0000 7f658afc0740 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x56362f275800 0x56362f28a120 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T10:32:11.798 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:32:11.794+0000 7f658afc0740 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x7f65801ac160 con 0x56362f26af40 2024-04-23T10:32:11.798 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:32:11.794+0000 7f658afc0740 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 -- 0x7f65801ad5d0 con 0x56362f275800 2024-04-23T10:32:11.798 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:32:11.794+0000 7f658afc0740 1 -- --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_getmap magic: 0 -- 0x7f65801ac830 con 0x56362f26a620 2024-04-23T10:32:11.798 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:32:11.794+0000 7f658afc0740 10 client.0 _ll_register_callbacks cb 0x7f6570032cf0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-23T10:32:11.798 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:32:11.794+0000 7f6586043640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x56362f26af40 0x56362f2e44d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T10:32:11.798 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:32:11.794+0000 7f6586043640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x56362f26af40 0x56362f2e44d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.114:50498/0 (socket says 172.21.15.114:50498) 2024-04-23T10:32:11.798 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:32:11.794+0000 7f6586043640 1 -- 172.21.15.114:0/143034566 learned_addr learned my addr 172.21.15.114:0/143034566 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-04-23T10:32:11.798 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:32:11.794+0000 7f6585842640 1 --2- 172.21.15.114:0/143034566 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x56362f26a620 0x56362f2e3f90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T10:32:11.798 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:32:11.794+0000 7f6585842640 1 --2- 172.21.15.114:0/143034566 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x56362f275800 0x56362f28a120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T10:32:11.798 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:32:11.794+0000 7f6586043640 1 -- 172.21.15.114:0/143034566 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x56362f26a620 msgr2=0x56362f2e3f90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T10:32:11.798 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:32:11.794+0000 7f6586043640 1 --2- 172.21.15.114:0/143034566 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x56362f26a620 0x56362f2e3f90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:32:11.799 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:32:11.794+0000 7f6586043640 1 -- 172.21.15.114:0/143034566 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x56362f275800 msgr2=0x56362f28a120 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T10:32:11.799 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:32:11.794+0000 7f6586043640 1 --2- 172.21.15.114:0/143034566 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x56362f275800 0x56362f28a120 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:32:11.799 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:32:11.794+0000 7f6586043640 1 -- 172.21.15.114:0/143034566 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({osdmap=0}) -- 0x56362f28a810 con 0x56362f26af40 2024-04-23T10:32:11.799 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:32:11.794+0000 7f6586043640 1 --2- 172.21.15.114:0/143034566 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x56362f26af40 0x56362f2e44d0 secure :-1 s=READY pgs=743 cs=0 l=1 rev1=1 crypto rx=0x7f657c00b370 tx=0x7f657c01b420 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8581a5d6ef5949a3 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T10:32:11.799 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:32:11.794+0000 7f6569ffb640 10 client.0 ms_handle_connect on v2:172.21.15.114:3300/0 2024-04-23T10:32:11.799 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:32:11.795+0000 7f6569ffb640 1 -- 172.21.15.114:0/143034566 <== mon.0 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f657c01f040 con 0x56362f26af40 2024-04-23T10:32:11.799 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:32:11.795+0000 7f6569ffb640 1 -- 172.21.15.114:0/143034566 <== mon.0 v2:172.21.15.114:3300/0 2 ==== osd_map(94..94 src has 1..94) ==== 8585+0+0 (secure 0 0 0) 0x7f657c01b8a0 con 0x56362f26af40 2024-04-23T10:32:11.799 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:32:11.795+0000 7f658afc0740 1 -- 172.21.15.114:0/143034566 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f657c01bdc0 con 0x56362f26af40 2024-04-23T10:32:11.799 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:32:11.795+0000 7f6569ffb640 1 -- 172.21.15.114:0/143034566 <== mon.0 v2:172.21.15.114:3300/0 3 ==== mon_get_version_reply(handle=1 version=30) ==== 24+0+0 (secure 0 0 0) 0x7f657c01baa0 con 0x56362f26af40 2024-04-23T10:32:11.799 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:32:11.795+0000 7f658afc0740 10 client.16410 fetch_fsmap learned FSMap version 30 2024-04-23T10:32:11.799 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:32:11.795+0000 7f658afc0740 1 -- 172.21.15.114:0/143034566 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({config=0+,fsmap=30,monmap=4+}) -- 0x56362f2eb710 con 0x56362f26af40 2024-04-23T10:32:11.799 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:32:11.795+0000 7f6569ffb640 1 -- 172.21.15.114:0/143034566 <== mon.0 v2:172.21.15.114:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f657c01bdc0 con 0x56362f26af40 2024-04-23T10:32:11.799 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:32:11.795+0000 7f6569ffb640 1 -- 172.21.15.114:0/143034566 <== mon.0 v2:172.21.15.114:3300/0 5 ==== fsmap(e 30) ==== 8338+0+0 (secure 0 0 0) 0x7f657c004ad0 con 0x56362f26af40 2024-04-23T10:32:11.799 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:32:11.795+0000 7f658afc0740 10 client.16410 fetch_fsmap finished waiting for FSMap version 30 2024-04-23T10:32:11.799 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:32:11.795+0000 7f658afc0740 10 client.16410 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-04-23T10:32:11.800 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:32:11.795+0000 7f658afc0740 20 client.16410 populate_metadata read hostname 'smithi114' 2024-04-23T10:32:11.800 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:32:11.795+0000 7f658afc0740 1 --2- 172.21.15.114:0/143034566 >> [v2:172.21.15.119:6838/937297340,v1:172.21.15.119:6839/937297340] conn(0x56362f2ebee0 0x56362f2ee300 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T10:32:11.800 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:32:11.795+0000 7f658afc0740 4 client.16410 mds_command: new command op to 24545 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-04-23T10:32:11.800 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:32:11.795+0000 7f658afc0740 1 -- 172.21.15.114:0/143034566 --> [v2:172.21.15.119:6838/937297340,v1:172.21.15.119:6839/937297340] -- command(tid 0: {"prefix": "flush journal"}) -- 0x56362f1d7200 con 0x56362f2ebee0 2024-04-23T10:32:11.800 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:32:11.795+0000 7f6585842640 1 --2- 172.21.15.114:0/143034566 >> [v2:172.21.15.119:6838/937297340,v1:172.21.15.119:6839/937297340] conn(0x56362f2ebee0 0x56362f2ee300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T10:32:11.800 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:32:11.796+0000 7f6569ffb640 10 client.16410 ms_handle_connect on v2:172.21.15.119:6838/937297340 2024-04-23T10:32:11.800 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:32:11.796+0000 7f6585842640 1 --2- 172.21.15.114:0/143034566 >> [v2:172.21.15.119:6838/937297340,v1:172.21.15.119:6839/937297340] conn(0x56362f2ebee0 0x56362f2ee300 crc :-1 s=READY pgs=111 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=694fef93d8b56f67 server_cookie=7e77a22bc6932eb0 in_seq=0 out_seq=0 2024-04-23T10:32:11.812 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:32:11.812+0000 7f6569ffb640 1 -- 172.21.15.114:0/143034566 <== mds.0 v2:172.21.15.119:6838/937297340 1 ==== command_reply(tid 0: 0 ) ==== 8+0+44 (crc 0 0 0) 0x56362f1d7200 con 0x56362f2ebee0 2024-04-23T10:32:11.812 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:32:11.812+0000 7f6569ffb640 10 client.16410 handle_command_reply: tid=0 2024-04-23T10:32:11.812 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:32:11.812+0000 7f658afc0740 1 client.16410 shutdown 2024-04-23T10:32:11.812 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:32:11.813+0000 7f658afc0740 1 -- 172.21.15.114:0/143034566 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x56362f26af40 msgr2=0x56362f2e44d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T10:32:11.812 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:32:11.813+0000 7f658afc0740 1 --2- 172.21.15.114:0/143034566 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x56362f26af40 0x56362f2e44d0 secure :-1 s=READY pgs=743 cs=0 l=1 rev1=1 crypto rx=0x7f657c00b370 tx=0x7f657c01b420 comp rx=0 tx=0).stop 2024-04-23T10:32:11.812 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:32:11.813+0000 7f658afc0740 1 -- 172.21.15.114:0/143034566 shutdown_connections 2024-04-23T10:32:11.812 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:32:11.813+0000 7f658afc0740 1 --2- 172.21.15.114:0/143034566 >> [v2:172.21.15.119:6838/937297340,v1:172.21.15.119:6839/937297340] conn(0x56362f2ebee0 0x56362f2ee300 crc :-1 s=READY pgs=111 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:32:11.812 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:32:11.813+0000 7f658afc0740 1 --2- 172.21.15.114:0/143034566 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x56362f275800 0x56362f28a120 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:32:11.813 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:32:11.813+0000 7f658afc0740 1 --2- 172.21.15.114:0/143034566 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x56362f26af40 0x56362f2e44d0 unknown :-1 s=CLOSED pgs=743 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:32:11.813 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:32:11.813+0000 7f6569ffb640 0 client.16410 ms_handle_reset on v2:172.21.15.119:6838/937297340 2024-04-23T10:32:11.813 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:32:11.813+0000 7f658afc0740 1 --2- 172.21.15.114:0/143034566 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x56362f26a620 0x56362f2e3f90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:32:11.813 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:32:11.813+0000 7f658afc0740 1 -- 172.21.15.114:0/143034566 >> 172.21.15.114:0/143034566 conn(0x56362f266b30 msgr2=0x56362f271cc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T10:32:11.813 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:32:11.813+0000 7f658afc0740 1 -- 172.21.15.114:0/143034566 shutdown_connections 2024-04-23T10:32:11.813 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:32:11.813+0000 7f658afc0740 1 -- 172.21.15.114:0/143034566 wait complete. 2024-04-23T10:32:11.813 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:32:11.813+0000 7f658afc0740 20 client.16410 trim_cache size 0 max 16384 2024-04-23T10:32:11.813 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stdout:{ 2024-04-23T10:32:11.813 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stdout: "message": "", 2024-04-23T10:32:11.813 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stdout: "return_code": 0 2024-04-23T10:32:11.813 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stdout:} 2024-04-23T10:32:11.814 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:32:11.813+0000 7f65882a7640 1 -- 172.21.15.114:0/3325722966 >> [v2:172.21.15.114:6800/1395693466,v1:172.21.15.114:6801/1395693466] conn(0x7f656c0a3dc0 msgr2=0x7f656c0a6280 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T10:32:11.815 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:32:11.813+0000 7f65882a7640 1 --2- 172.21.15.114:0/3325722966 >> [v2:172.21.15.114:6800/1395693466,v1:172.21.15.114:6801/1395693466] conn(0x7f656c0a3dc0 0x7f656c0a6280 secure :-1 s=READY pgs=109 cs=0 l=1 rev1=1 crypto rx=0x7f65780ce410 tx=0x7f657806e5f0 comp rx=0 tx=0).stop 2024-04-23T10:32:11.815 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:32:11.813+0000 7f65882a7640 1 -- 172.21.15.114:0/3325722966 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f658010c550 msgr2=0x7f658019cf30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T10:32:11.815 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:32:11.813+0000 7f65882a7640 1 --2- 172.21.15.114:0/3325722966 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f658010c550 0x7f658019cf30 secure :-1 s=READY pgs=739 cs=0 l=1 rev1=1 crypto rx=0x7f657800cce0 tx=0x7f657800a620 comp rx=0 tx=0).stop 2024-04-23T10:32:11.815 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:32:11.813+0000 7f65882a7640 1 -- 172.21.15.114:0/3325722966 shutdown_connections 2024-04-23T10:32:11.815 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:32:11.813+0000 7f65882a7640 1 --2- 172.21.15.114:0/3325722966 >> [v2:172.21.15.114:6800/1395693466,v1:172.21.15.114:6801/1395693466] conn(0x7f656c0a3dc0 0x7f656c0a6280 unknown :-1 s=CLOSED pgs=109 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:32:11.815 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:32:11.813+0000 7f65882a7640 1 --2- 172.21.15.114:0/3325722966 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f658010c550 0x7f658019cf30 unknown :-1 s=CLOSED pgs=739 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:32:11.815 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:32:11.813+0000 7f65882a7640 1 --2- 172.21.15.114:0/3325722966 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f6580103aa0 0x7f6580198bb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:32:11.815 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:32:11.813+0000 7f65882a7640 1 --2- 172.21.15.114:0/3325722966 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f6580103180 0x7f6580198670 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:32:11.815 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:32:11.813+0000 7f65882a7640 1 -- 172.21.15.114:0/3325722966 >> 172.21.15.114:0/3325722966 conn(0x7f6580077fd0 msgr2=0x7f65800fde80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T10:32:11.815 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:32:11.813+0000 7f65882a7640 1 -- 172.21.15.114:0/3325722966 shutdown_connections 2024-04-23T10:32:11.815 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:32:11.813+0000 7f65882a7640 1 -- 172.21.15.114:0/3325722966 wait complete. 2024-04-23T10:32:11.923 INFO:tasks.workunit.client.0.smithi114.stdout: 1 305661 4.91 MB/sec execute 140 sec latency 54147.230 ms 2024-04-23T10:32:12.492 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 23 10:32:12 smithi119 ceph-mon[32608]: pgmap v172: 97 pgs: 97 active+clean; 78 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 22 KiB/s rd, 36 KiB/s wr, 25 op/s 2024-04-23T10:32:12.504 INFO:journalctl@ceph.mon.a.smithi114.stdout:Apr 23 10:32:12 smithi114 ceph-mon[29255]: pgmap v172: 97 pgs: 97 active+clean; 78 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 22 KiB/s rd, 36 KiB/s wr, 25 op/s 2024-04-23T10:32:12.577 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 23 10:32:12 smithi204 ceph-mon[32649]: pgmap v172: 97 pgs: 97 active+clean; 78 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 22 KiB/s rd, 36 KiB/s wr, 25 op/s 2024-04-23T10:32:12.923 INFO:tasks.workunit.client.0.smithi114.stdout: 1 305661 4.88 MB/sec execute 141 sec latency 55147.332 ms 2024-04-23T10:32:13.923 INFO:tasks.workunit.client.0.smithi114.stdout: 1 305661 4.84 MB/sec execute 142 sec latency 56147.445 ms 2024-04-23T10:32:14.492 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 23 10:32:14 smithi119 ceph-mon[32608]: pgmap v173: 97 pgs: 97 active+clean; 78 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 26 KiB/s rd, 39 KiB/s wr, 28 op/s 2024-04-23T10:32:14.504 INFO:journalctl@ceph.mon.a.smithi114.stdout:Apr 23 10:32:14 smithi114 ceph-mon[29255]: pgmap v173: 97 pgs: 97 active+clean; 78 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 26 KiB/s rd, 39 KiB/s wr, 28 op/s 2024-04-23T10:32:14.577 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 23 10:32:14 smithi204 ceph-mon[32649]: pgmap v173: 97 pgs: 97 active+clean; 78 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 26 KiB/s rd, 39 KiB/s wr, 28 op/s 2024-04-23T10:32:14.923 INFO:tasks.workunit.client.0.smithi114.stdout: 1 305661 4.81 MB/sec execute 143 sec latency 57147.580 ms 2024-04-23T10:32:15.923 INFO:tasks.workunit.client.0.smithi114.stdout: 1 305661 4.78 MB/sec execute 144 sec latency 58147.682 ms 2024-04-23T10:32:16.492 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 23 10:32:16 smithi119 ceph-mon[32608]: pgmap v174: 97 pgs: 97 active+clean; 78 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 5.1 KiB/s rd, 2.7 KiB/s wr, 5 op/s 2024-04-23T10:32:16.504 INFO:journalctl@ceph.mon.a.smithi114.stdout:Apr 23 10:32:16 smithi114 ceph-mon[29255]: pgmap v174: 97 pgs: 97 active+clean; 78 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 5.1 KiB/s rd, 2.7 KiB/s wr, 5 op/s 2024-04-23T10:32:16.578 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 23 10:32:16 smithi204 ceph-mon[32649]: pgmap v174: 97 pgs: 97 active+clean; 78 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 5.1 KiB/s rd, 2.7 KiB/s wr, 5 op/s 2024-04-23T10:32:16.925 INFO:tasks.workunit.client.0.smithi114.stdout: 1 305661 4.74 MB/sec execute 145 sec latency 59147.804 ms 2024-04-23T10:32:17.924 INFO:tasks.workunit.client.0.smithi114.stdout: 1 305661 4.71 MB/sec execute 146 sec latency 60147.906 ms 2024-04-23T10:32:18.492 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 23 10:32:18 smithi119 ceph-mon[32608]: pgmap v175: 97 pgs: 97 active+clean; 78 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 12 KiB/s wr, 11 op/s 2024-04-23T10:32:18.504 INFO:journalctl@ceph.mon.a.smithi114.stdout:Apr 23 10:32:18 smithi114 ceph-mon[29255]: pgmap v175: 97 pgs: 97 active+clean; 78 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 12 KiB/s wr, 11 op/s 2024-04-23T10:32:18.577 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 23 10:32:18 smithi204 ceph-mon[32649]: pgmap v175: 97 pgs: 97 active+clean; 78 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 12 KiB/s wr, 11 op/s 2024-04-23T10:32:18.924 INFO:tasks.workunit.client.0.smithi114.stdout: 1 305661 4.68 MB/sec execute 147 sec latency 61148.029 ms 2024-04-23T10:32:19.924 INFO:tasks.workunit.client.0.smithi114.stdout: 1 305661 4.65 MB/sec execute 148 sec latency 62148.141 ms 2024-04-23T10:32:20.491 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 23 10:32:20 smithi119 ceph-mon[32608]: pgmap v176: 97 pgs: 97 active+clean; 78 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 11 KiB/s rd, 12 KiB/s wr, 8 op/s 2024-04-23T10:32:20.504 INFO:journalctl@ceph.mon.a.smithi114.stdout:Apr 23 10:32:20 smithi114 ceph-mon[29255]: pgmap v176: 97 pgs: 97 active+clean; 78 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 11 KiB/s rd, 12 KiB/s wr, 8 op/s 2024-04-23T10:32:20.578 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 23 10:32:20 smithi204 ceph-mon[32649]: pgmap v176: 97 pgs: 97 active+clean; 78 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 11 KiB/s rd, 12 KiB/s wr, 8 op/s 2024-04-23T10:32:20.924 INFO:tasks.workunit.client.0.smithi114.stdout: 1 305661 4.62 MB/sec execute 149 sec latency 63148.236 ms 2024-04-23T10:32:21.443 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:32:21.444+0000 7f34577fe640 1 -- 172.21.15.114:0/1636762081 <== mds.0 v2:172.21.15.119:6838/937297340 1 ==== command_reply(tid 0: -110 ) ==== 8+0+592 (crc 0 0 0) 0x5638da584c10 con 0x5638da699f30 2024-04-23T10:32:21.444 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:32:21.444+0000 7f34577fe640 10 client.16392 handle_command_reply: tid=0 2024-04-23T10:32:21.444 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:32:21.444+0000 7f34815bf740 1 client.16392 shutdown 2024-04-23T10:32:21.444 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:32:21.444+0000 7f34815bf740 1 -- 172.21.15.114:0/1636762081 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x5638da622b50 msgr2=0x5638da626270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T10:32:21.444 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:32:21.444+0000 7f34815bf740 1 --2- 172.21.15.114:0/1636762081 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x5638da622b50 0x5638da626270 secure :-1 s=READY pgs=737 cs=0 l=1 rev1=1 crypto rx=0x7f346c0096a0 tx=0x7f346c009560 comp rx=0 tx=0).stop 2024-04-23T10:32:21.444 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:32:21.444+0000 7f34815bf740 1 -- 172.21.15.114:0/1636762081 shutdown_connections 2024-04-23T10:32:21.444 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:32:21.444+0000 7f34815bf740 1 --2- 172.21.15.114:0/1636762081 >> [v2:172.21.15.119:6838/937297340,v1:172.21.15.119:6839/937297340] conn(0x5638da699f30 0x5638da69c350 crc :-1 s=READY pgs=109 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:32:21.444 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:32:21.444+0000 7f34815bf740 1 --2- 172.21.15.114:0/1636762081 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x5638da622b50 0x5638da626270 unknown :-1 s=CLOSED pgs=737 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:32:21.444 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:32:21.444+0000 7f34815bf740 1 --2- 172.21.15.114:0/1636762081 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x5638da618480 0x5638da625d30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:32:21.444 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:32:21.444+0000 7f34815bf740 1 --2- 172.21.15.114:0/1636762081 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x5638da617b60 0x5638da62af70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:32:21.444 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:32:21.444+0000 7f34577fe640 0 client.16392 ms_handle_reset on v2:172.21.15.119:6838/937297340 2024-04-23T10:32:21.444 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:32:21.444+0000 7f34815bf740 1 -- 172.21.15.114:0/1636762081 >> 172.21.15.114:0/1636762081 conn(0x5638da613e80 msgr2=0x5638da61e860 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T10:32:21.444 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:32:21.444+0000 7f34815bf740 1 -- 172.21.15.114:0/1636762081 shutdown_connections 2024-04-23T10:32:21.444 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:32:21.444+0000 7f34815bf740 1 -- 172.21.15.114:0/1636762081 wait complete. 2024-04-23T10:32:21.444 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:32:21.444+0000 7f34815bf740 20 client.16392 trim_cache size 0 max 16384 2024-04-23T10:32:21.445 INFO:teuthology.orchestra.run.smithi114.stdout:{ 2024-04-23T10:32:21.445 INFO:teuthology.orchestra.run.smithi114.stdout: "epoch": 30, 2024-04-23T10:32:21.445 INFO:teuthology.orchestra.run.smithi114.stdout: "leader": 24545, 2024-04-23T10:32:21.445 INFO:teuthology.orchestra.run.smithi114.stdout: "set_version": 18, 2024-04-23T10:32:21.445 INFO:teuthology.orchestra.run.smithi114.stdout: "sets": { 2024-04-23T10:32:21.445 INFO:teuthology.orchestra.run.smithi114.stdout: "11138c86": { 2024-04-23T10:32:21.445 INFO:teuthology.orchestra.run.smithi114.stdout: "version": 18, 2024-04-23T10:32:21.445 INFO:teuthology.orchestra.run.smithi114.stdout: "age_ref": 0.0, 2024-04-23T10:32:21.445 INFO:teuthology.orchestra.run.smithi114.stdout: "state": { 2024-04-23T10:32:21.445 INFO:teuthology.orchestra.run.smithi114.stdout: "name": "TIMEDOUT", 2024-04-23T10:32:21.445 INFO:teuthology.orchestra.run.smithi114.stdout: "age": 0.0 2024-04-23T10:32:21.445 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-23T10:32:21.445 INFO:teuthology.orchestra.run.smithi114.stdout: "timeout": 60.0, 2024-04-23T10:32:21.446 INFO:teuthology.orchestra.run.smithi114.stdout: "expiration": 82.6, 2024-04-23T10:32:21.446 INFO:teuthology.orchestra.run.smithi114.stdout: "members": { 2024-04-23T10:32:21.446 INFO:teuthology.orchestra.run.smithi114.stdout: "file:/": { 2024-04-23T10:32:21.446 INFO:teuthology.orchestra.run.smithi114.stdout: "excluded": false, 2024-04-23T10:32:21.446 INFO:teuthology.orchestra.run.smithi114.stdout: "state": { 2024-04-23T10:32:21.446 INFO:teuthology.orchestra.run.smithi114.stdout: "name": "QUIESCING", 2024-04-23T10:32:21.446 INFO:teuthology.orchestra.run.smithi114.stdout: "age": 60.0 2024-04-23T10:32:21.446 INFO:teuthology.orchestra.run.smithi114.stdout: } 2024-04-23T10:32:21.446 INFO:teuthology.orchestra.run.smithi114.stdout: } 2024-04-23T10:32:21.446 INFO:teuthology.orchestra.run.smithi114.stdout: } 2024-04-23T10:32:21.446 INFO:teuthology.orchestra.run.smithi114.stdout: } 2024-04-23T10:32:21.446 INFO:teuthology.orchestra.run.smithi114.stdout: } 2024-04-23T10:32:21.446 INFO:teuthology.orchestra.run.smithi114.stdout:} 2024-04-23T10:32:21.447 INFO:teuthology.orchestra.run.smithi114.stderr:Error ETIMEDOUT: 2024-04-23T10:32:21.447 DEBUG:teuthology.orchestra.run:got remote process result: 110 2024-04-23T10:32:21.448 ERROR:tasks.quiescer.fs.[cephfs]:Couldn't quiesce root with rc: 110 (ETIMEDOUT), stdout: { "epoch": 30, "leader": 24545, "set_version": 18, "sets": { "11138c86": { "version": 18, "age_ref": 0.0, "state": { "name": "TIMEDOUT", "age": 0.0 }, "timeout": 60.0, "expiration": 82.6, "members": { "file:/": { "excluded": false, "state": { "name": "QUIESCING", "age": 60.0 } } } } } } 2024-04-23T10:32:21.448 ERROR:tasks.quiescer.fs.[cephfs]:exception: Traceback (most recent call last): File "/home/teuthworker/src/github.com_ceph_ceph_0a6c3ed699031b80a2b419e7e795368719871394/qa/tasks/quiescer.py", line 300, in _run self.do_quiesce(d) File "/home/teuthworker/src/github.com_ceph_ceph_0a6c3ed699031b80a2b419e7e795368719871394/qa/tasks/quiescer.py", line 255, in do_quiesce raise RuntimeError(f"Error quiescing root: {rcinfo}") RuntimeError: Error quiescing root: 110 (ETIMEDOUT) 2024-04-23T10:32:21.924 INFO:tasks.workunit.client.0.smithi114.stdout: 1 306475 4.62 MB/sec execute 150 sec latency 63699.123 ms 2024-04-23T10:32:22.491 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 23 10:32:22 smithi119 ceph-mon[32608]: pgmap v177: 97 pgs: 97 active+clean; 78 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 11 KiB/s rd, 12 KiB/s wr, 8 op/s 2024-04-23T10:32:22.504 INFO:journalctl@ceph.mon.a.smithi114.stdout:Apr 23 10:32:22 smithi114 ceph-mon[29255]: pgmap v177: 97 pgs: 97 active+clean; 78 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 11 KiB/s rd, 12 KiB/s wr, 8 op/s 2024-04-23T10:32:22.577 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 23 10:32:22 smithi204 ceph-mon[32649]: pgmap v177: 97 pgs: 97 active+clean; 78 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 11 KiB/s rd, 12 KiB/s wr, 8 op/s 2024-04-23T10:32:22.924 INFO:tasks.workunit.client.0.smithi114.stdout: 1 308813 4.68 MB/sec execute 151 sec latency 14.524 ms 2024-04-23T10:32:23.924 INFO:tasks.workunit.client.0.smithi114.stdout: 1 310119 4.68 MB/sec execute 152 sec latency 18.848 ms 2024-04-23T10:32:24.491 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 23 10:32:24 smithi119 ceph-mon[32608]: pgmap v178: 97 pgs: 97 active+clean; 78 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 14 KiB/s rd, 13 KiB/s wr, 11 op/s 2024-04-23T10:32:24.504 INFO:journalctl@ceph.mon.a.smithi114.stdout:Apr 23 10:32:24 smithi114 ceph-mon[29255]: pgmap v178: 97 pgs: 97 active+clean; 78 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 14 KiB/s rd, 13 KiB/s wr, 11 op/s 2024-04-23T10:32:24.578 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 23 10:32:24 smithi204 ceph-mon[32649]: pgmap v178: 97 pgs: 97 active+clean; 78 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 14 KiB/s rd, 13 KiB/s wr, 11 op/s 2024-04-23T10:32:24.904 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:32:24.903+0000 7f4751a9a640 1 -- 172.21.15.114:0/2687449478 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f474c100720 msgr2=0x7f474c100b00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T10:32:24.904 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:32:24.903+0000 7f4751a9a640 1 --2- 172.21.15.114:0/2687449478 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f474c100720 0x7f474c100b00 secure :-1 s=READY pgs=73 cs=0 l=1 rev1=1 crypto rx=0x7f473c009860 tx=0x7f473c02a340 comp rx=0 tx=0).stop 2024-04-23T10:32:24.904 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:32:24.903+0000 7f4751a9a640 1 -- 172.21.15.114:0/2687449478 shutdown_connections 2024-04-23T10:32:24.905 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:32:24.903+0000 7f4751a9a640 1 --2- 172.21.15.114:0/2687449478 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f474c107330 0x7f474c10eeb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:32:24.905 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:32:24.903+0000 7f4751a9a640 1 --2- 172.21.15.114:0/2687449478 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f474c1010d0 0x7f474c106c00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:32:24.905 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:32:24.903+0000 7f4751a9a640 1 --2- 172.21.15.114:0/2687449478 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f474c100720 0x7f474c100b00 unknown :-1 s=CLOSED pgs=73 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:32:24.905 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:32:24.903+0000 7f4751a9a640 1 -- 172.21.15.114:0/2687449478 >> 172.21.15.114:0/2687449478 conn(0x7f474c077390 msgr2=0x7f474c0fe760 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T10:32:24.905 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:32:24.903+0000 7f4751a9a640 1 -- 172.21.15.114:0/2687449478 shutdown_connections 2024-04-23T10:32:24.905 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:32:24.903+0000 7f4751a9a640 1 -- 172.21.15.114:0/2687449478 wait complete. 2024-04-23T10:32:24.905 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:32:24.904+0000 7f4751a9a640 1 Processor -- start 2024-04-23T10:32:24.905 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:32:24.904+0000 7f4751a9a640 1 -- start start 2024-04-23T10:32:24.905 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:32:24.904+0000 7f4751a9a640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f474c100720 0x7f474c1987a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T10:32:24.905 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:32:24.904+0000 7f4751a9a640 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f474c107330 0x7f474c198ce0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T10:32:24.905 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:32:24.904+0000 7f4751a9a640 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f474c19d010 0x7f474c19d4c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T10:32:24.905 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:32:24.904+0000 7f4751a9a640 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x7f474c112220 con 0x7f474c100720 2024-04-23T10:32:24.905 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:32:24.904+0000 7f4751a9a640 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 -- 0x7f474c1120a0 con 0x7f474c19d010 2024-04-23T10:32:24.905 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:32:24.904+0000 7f4751a9a640 1 -- --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_getmap magic: 0 -- 0x7f474c1123a0 con 0x7f474c107330 2024-04-23T10:32:24.906 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:32:24.904+0000 7f474b7fe640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f474c100720 0x7f474c1987a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T10:32:24.906 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:32:24.904+0000 7f474b7fe640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f474c100720 0x7f474c1987a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.114:39256/0 (socket says 172.21.15.114:39256) 2024-04-23T10:32:24.906 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:32:24.904+0000 7f474b7fe640 1 -- 172.21.15.114:0/469003199 learned_addr learned my addr 172.21.15.114:0/469003199 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-04-23T10:32:24.906 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:32:24.904+0000 7f474affd640 1 --2- 172.21.15.114:0/469003199 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f474c107330 0x7f474c198ce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T10:32:24.906 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:32:24.904+0000 7f474bfff640 1 --2- 172.21.15.114:0/469003199 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f474c19d010 0x7f474c19d4c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T10:32:24.906 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:32:24.905+0000 7f474b7fe640 1 -- 172.21.15.114:0/469003199 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f474c107330 msgr2=0x7f474c198ce0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T10:32:24.906 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:32:24.905+0000 7f474b7fe640 1 --2- 172.21.15.114:0/469003199 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f474c107330 0x7f474c198ce0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:32:24.906 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:32:24.905+0000 7f474b7fe640 1 -- 172.21.15.114:0/469003199 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f474c19d010 msgr2=0x7f474c19d4c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T10:32:24.906 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:32:24.905+0000 7f474b7fe640 1 --2- 172.21.15.114:0/469003199 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f474c19d010 0x7f474c19d4c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:32:24.906 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:32:24.905+0000 7f474b7fe640 1 -- 172.21.15.114:0/469003199 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f474c19db00 con 0x7f474c100720 2024-04-23T10:32:24.906 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:32:24.905+0000 7f474bfff640 1 --2- 172.21.15.114:0/469003199 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f474c19d010 0x7f474c19d4c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-23T10:32:24.906 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:32:24.905+0000 7f474b7fe640 1 --2- 172.21.15.114:0/469003199 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f474c100720 0x7f474c1987a0 secure :-1 s=READY pgs=744 cs=0 l=1 rev1=1 crypto rx=0x7f473c0303f0 tx=0x7f473c032da0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T10:32:24.906 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:32:24.905+0000 7f4748ff9640 1 -- 172.21.15.114:0/469003199 <== mon.0 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f473c040020 con 0x7f474c100720 2024-04-23T10:32:24.907 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:32:24.905+0000 7f4751a9a640 1 -- 172.21.15.114:0/469003199 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f474c19dd90 con 0x7f474c100720 2024-04-23T10:32:24.907 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:32:24.905+0000 7f4748ff9640 1 -- 172.21.15.114:0/469003199 <== mon.0 v2:172.21.15.114:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f473c030dd0 con 0x7f474c100720 2024-04-23T10:32:24.907 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:32:24.905+0000 7f4748ff9640 1 -- 172.21.15.114:0/469003199 <== mon.0 v2:172.21.15.114:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f473c037040 con 0x7f474c100720 2024-04-23T10:32:24.907 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:32:24.905+0000 7f4751a9a640 1 -- 172.21.15.114:0/469003199 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f474c1a5550 con 0x7f474c100720 2024-04-23T10:32:24.914 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:32:24.907+0000 7f4748ff9640 1 -- 172.21.15.114:0/469003199 <== mon.0 v2:172.21.15.114:3300/0 4 ==== mgrmap(e 26) ==== 138778+0+0 (secure 0 0 0) 0x7f473c043020 con 0x7f474c100720 2024-04-23T10:32:24.914 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:32:24.908+0000 7f47547b5740 1 Processor -- start 2024-04-23T10:32:24.914 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:32:24.908+0000 7f47547b5740 1 -- start start 2024-04-23T10:32:24.914 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:32:24.908+0000 7f47547b5740 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x56434d6d5500 0x56434d6d58e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T10:32:24.914 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:32:24.908+0000 7f47547b5740 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x56434d6d5e20 0x56434d6e46d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T10:32:24.914 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:32:24.908+0000 7f4748ff9640 1 --2- 172.21.15.114:0/469003199 >> [v2:172.21.15.114:6800/1395693466,v1:172.21.15.114:6801/1395693466] conn(0x7f47340a3dc0 0x7f47340a6280 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T10:32:24.914 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:32:24.908+0000 7f47547b5740 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x56434d6e4c10 0x56434d6e7000 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T10:32:24.914 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:32:24.908+0000 7f47547b5740 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x56434d63a970 con 0x56434d6d5e20 2024-04-23T10:32:24.914 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:32:24.908+0000 7f47547b5740 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 -- 0x56434d5d74a0 con 0x56434d6e4c10 2024-04-23T10:32:24.914 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:32:24.908+0000 7f47547b5740 1 -- --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_getmap magic: 0 -- 0x56434d636930 con 0x56434d6d5500 2024-04-23T10:32:24.915 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:32:24.908+0000 7f474bfff640 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x56434d6d5500 0x56434d6d58e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T10:32:24.915 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:32:24.908+0000 7f474b7fe640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x56434d6d5e20 0x56434d6e46d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T10:32:24.915 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:32:24.908+0000 7f474affd640 1 --2- 172.21.15.114:0/469003199 >> [v2:172.21.15.114:6800/1395693466,v1:172.21.15.114:6801/1395693466] conn(0x7f47340a3dc0 0x7f47340a6280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T10:32:24.915 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:32:24.908+0000 7f474bfff640 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x56434d6d5500 0x56434d6d58e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.204:3300/0 says I am v2:172.21.15.114:53380/0 (socket says 172.21.15.114:53380) 2024-04-23T10:32:24.915 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:32:24.908+0000 7f474bfff640 1 -- 172.21.15.114:0/4273922955 learned_addr learned my addr 172.21.15.114:0/4273922955 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-04-23T10:32:24.915 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:32:24.908+0000 7f474bfff640 1 --2- 172.21.15.114:0/4273922955 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x56434d6e4c10 0x56434d6e7000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T10:32:24.915 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:32:24.908+0000 7f4748ff9640 1 -- 172.21.15.114:0/469003199 <== mon.0 v2:172.21.15.114:3300/0 5 ==== osd_map(94..94 src has 1..94) ==== 8585+0+0 (secure 0 0 0) 0x7f473c0f0670 con 0x7f474c100720 2024-04-23T10:32:24.915 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:32:24.908+0000 7f474bfff640 1 -- 172.21.15.114:0/4273922955 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x56434d6e4c10 msgr2=0x56434d6e7000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T10:32:24.915 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:32:24.908+0000 7f474bfff640 1 --2- 172.21.15.114:0/4273922955 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x56434d6e4c10 0x56434d6e7000 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:32:24.915 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:32:24.908+0000 7f474bfff640 1 -- 172.21.15.114:0/4273922955 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x56434d6d5e20 msgr2=0x56434d6e46d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T10:32:24.915 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:32:24.908+0000 7f474bfff640 1 --2- 172.21.15.114:0/4273922955 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x56434d6d5e20 0x56434d6e46d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:32:24.915 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:32:24.908+0000 7f474bfff640 1 -- 172.21.15.114:0/4273922955 --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x56434d34d210 con 0x56434d6d5500 2024-04-23T10:32:24.916 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:32:24.909+0000 7f474bfff640 1 --2- 172.21.15.114:0/4273922955 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x56434d6d5500 0x56434d6d58e0 secure :-1 s=READY pgs=74 cs=0 l=1 rev1=1 crypto rx=0x7f474400efd0 tx=0x7f4744010040 comp rx=0 tx=0).ready entity=mon.1 client_cookie=3f007d0f7623d8e7 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T10:32:24.916 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:32:24.909+0000 7f474affd640 1 --2- 172.21.15.114:0/469003199 >> [v2:172.21.15.114:6800/1395693466,v1:172.21.15.114:6801/1395693466] conn(0x7f47340a3dc0 0x7f47340a6280 secure :-1 s=READY pgs=110 cs=0 l=1 rev1=1 crypto rx=0x7f474c104b10 tx=0x7f4740005f90 comp rx=0 tx=0).ready entity=mgr.15291 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T10:32:24.916 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:32:24.909+0000 7f4731ffb640 1 -- 172.21.15.114:0/4273922955 <== mon.1 v2:172.21.15.204:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4744003970 con 0x56434d6d5500 2024-04-23T10:32:24.916 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:32:24.909+0000 7f4731ffb640 1 -- 172.21.15.114:0/4273922955 <== mon.1 v2:172.21.15.204:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f4744004030 con 0x56434d6d5500 2024-04-23T10:32:24.916 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:32:24.909+0000 7f4731ffb640 1 -- 172.21.15.114:0/4273922955 <== mon.1 v2:172.21.15.204:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f474401d670 con 0x56434d6d5500 2024-04-23T10:32:24.916 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:32:24.909+0000 7f47547b5740 1 -- 172.21.15.114:0/4273922955 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x56434d6d5500 msgr2=0x56434d6d58e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T10:32:24.916 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:32:24.909+0000 7f47547b5740 1 --2- 172.21.15.114:0/4273922955 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x56434d6d5500 0x56434d6d58e0 secure :-1 s=READY pgs=74 cs=0 l=1 rev1=1 crypto rx=0x7f474400efd0 tx=0x7f4744010040 comp rx=0 tx=0).stop 2024-04-23T10:32:24.916 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:32:24.910+0000 7f47547b5740 1 -- 172.21.15.114:0/4273922955 shutdown_connections 2024-04-23T10:32:24.916 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:32:24.910+0000 7f47547b5740 1 --2- 172.21.15.114:0/4273922955 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x56434d6e4c10 0x56434d6e7000 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:32:24.916 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:32:24.910+0000 7f47547b5740 1 --2- 172.21.15.114:0/4273922955 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x56434d6d5e20 0x56434d6e46d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:32:24.916 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:32:24.910+0000 7f47547b5740 1 --2- 172.21.15.114:0/4273922955 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x56434d6d5500 0x56434d6d58e0 unknown :-1 s=CLOSED pgs=74 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:32:24.916 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:32:24.910+0000 7f47547b5740 1 -- 172.21.15.114:0/4273922955 >> 172.21.15.114:0/4273922955 conn(0x56434d6d1b30 msgr2=0x56434d6d1f10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T10:32:24.916 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:32:24.910+0000 7f47547b5740 1 -- 172.21.15.114:0/4273922955 shutdown_connections 2024-04-23T10:32:24.916 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:32:24.910+0000 7f47547b5740 1 -- 172.21.15.114:0/4273922955 wait complete. 2024-04-23T10:32:24.917 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:32:24.910+0000 7f47547b5740 1 -- start start 2024-04-23T10:32:24.917 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:32:24.910+0000 7f47547b5740 1 Processor -- start 2024-04-23T10:32:24.917 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:32:24.910+0000 7f47547b5740 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x56434d6d5500 0x56434d6e4560 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T10:32:24.917 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:32:24.910+0000 7f47547b5740 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x56434d6d5e20 0x56434d6e1850 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T10:32:24.917 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:32:24.910+0000 7f47547b5740 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x56434d6e4c10 0x56434d6e1db0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T10:32:24.917 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:32:24.910+0000 7f47547b5740 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x56434d6e77e0 con 0x56434d6e4c10 2024-04-23T10:32:24.917 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:32:24.910+0000 7f47547b5740 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 -- 0x56434d5d50b0 con 0x56434d6d5500 2024-04-23T10:32:24.917 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:32:24.910+0000 7f47547b5740 1 -- --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_getmap magic: 0 -- 0x56434d6e7960 con 0x56434d6d5e20 2024-04-23T10:32:24.917 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:32:24.910+0000 7f47547b5740 10 client.0 _ll_register_callbacks cb 0x56434d5c2d30 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-23T10:32:24.917 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:32:24.910+0000 7f474bfff640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x56434d6e4c10 0x56434d6e1db0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T10:32:24.917 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:32:24.911+0000 7f474bfff640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x56434d6e4c10 0x56434d6e1db0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.114:39260/0 (socket says 172.21.15.114:39260) 2024-04-23T10:32:24.917 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:32:24.911+0000 7f474bfff640 1 -- 172.21.15.114:0/820896308 learned_addr learned my addr 172.21.15.114:0/820896308 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-04-23T10:32:24.917 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:32:24.911+0000 7f474affd640 1 --2- 172.21.15.114:0/820896308 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x56434d6d5e20 0x56434d6e1850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T10:32:24.917 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:32:24.911+0000 7f474b7fe640 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x56434d6d5500 0x56434d6e4560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T10:32:24.918 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:32:24.911+0000 7f474bfff640 1 -- 172.21.15.114:0/820896308 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x56434d6d5e20 msgr2=0x56434d6e1850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T10:32:24.918 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:32:24.911+0000 7f474bfff640 1 --2- 172.21.15.114:0/820896308 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x56434d6d5e20 0x56434d6e1850 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:32:24.918 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:32:24.911+0000 7f474bfff640 1 -- 172.21.15.114:0/820896308 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x56434d6d5500 msgr2=0x56434d6e4560 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T10:32:24.918 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:32:24.911+0000 7f474bfff640 1 --2- 172.21.15.114:0/820896308 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x56434d6d5500 0x56434d6e4560 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:32:24.918 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:32:24.911+0000 7f474bfff640 1 -- 172.21.15.114:0/820896308 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({osdmap=0}) -- 0x56434d419620 con 0x56434d6e4c10 2024-04-23T10:32:24.918 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:32:24.911+0000 7f474b7fe640 1 --2- 172.21.15.114:0/820896308 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x56434d6d5500 0x56434d6e4560 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-23T10:32:24.918 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:32:24.911+0000 7f474bfff640 1 --2- 172.21.15.114:0/820896308 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x56434d6e4c10 0x56434d6e1db0 secure :-1 s=READY pgs=745 cs=0 l=1 rev1=1 crypto rx=0x7f474401f040 tx=0x7f4744004cf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2d103cf355e2ea69 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T10:32:24.918 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:32:24.911+0000 7f472affd640 10 client.0 ms_handle_connect on v2:172.21.15.114:3300/0 2024-04-23T10:32:24.918 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:32:24.911+0000 7f472affd640 1 -- 172.21.15.114:0/820896308 <== mon.0 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f474402c020 con 0x56434d6e4c10 2024-04-23T10:32:24.918 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:32:24.911+0000 7f472affd640 1 -- 172.21.15.114:0/820896308 <== mon.0 v2:172.21.15.114:3300/0 2 ==== osd_map(94..94 src has 1..94) ==== 8585+0+0 (secure 0 0 0) 0x7f4744003b80 con 0x56434d6e4c10 2024-04-23T10:32:24.918 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:32:24.911+0000 7f47547b5740 1 -- 172.21.15.114:0/820896308 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f4744004030 con 0x56434d6e4c10 2024-04-23T10:32:24.918 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:32:24.911+0000 7f472affd640 1 -- 172.21.15.114:0/820896308 <== mon.0 v2:172.21.15.114:3300/0 3 ==== mon_get_version_reply(handle=1 version=30) ==== 24+0+0 (secure 0 0 0) 0x7f4744037050 con 0x56434d6e4c10 2024-04-23T10:32:24.918 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:32:24.911+0000 7f47547b5740 10 client.16422 fetch_fsmap learned FSMap version 30 2024-04-23T10:32:24.919 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:32:24.911+0000 7f47547b5740 1 -- 172.21.15.114:0/820896308 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({config=0+,fsmap=30,monmap=4+}) -- 0x56434d6f37f0 con 0x56434d6e4c10 2024-04-23T10:32:24.919 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:32:24.911+0000 7f472affd640 1 -- 172.21.15.114:0/820896308 <== mon.0 v2:172.21.15.114:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f4744004030 con 0x56434d6e4c10 2024-04-23T10:32:24.919 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:32:24.912+0000 7f472affd640 1 -- 172.21.15.114:0/820896308 <== mon.0 v2:172.21.15.114:3300/0 5 ==== fsmap(e 30) ==== 8338+0+0 (secure 0 0 0) 0x7f474400b040 con 0x56434d6e4c10 2024-04-23T10:32:24.919 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:32:24.912+0000 7f47547b5740 10 client.16422 fetch_fsmap finished waiting for FSMap version 30 2024-04-23T10:32:24.919 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:32:24.912+0000 7f47547b5740 10 client.16422 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-04-23T10:32:24.919 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:32:24.912+0000 7f47547b5740 20 client.16422 populate_metadata read hostname 'smithi114' 2024-04-23T10:32:24.919 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:32:24.912+0000 7f47547b5740 1 --2- 172.21.15.114:0/820896308 >> [v2:172.21.15.119:6838/937297340,v1:172.21.15.119:6839/937297340] conn(0x56434d6f4c50 0x56434d6f7040 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T10:32:24.919 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:32:24.912+0000 7f47547b5740 4 client.16422 mds_command: new command op to 24545 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-23T10:32:24.919 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:32:24.912+0000 7f47547b5740 1 -- 172.21.15.114:0/820896308 --> [v2:172.21.15.119:6838/937297340,v1:172.21.15.119:6839/937297340] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x56434d648390 con 0x56434d6f4c50 2024-04-23T10:32:24.919 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:32:24.912+0000 7f474b7fe640 1 --2- 172.21.15.114:0/820896308 >> [v2:172.21.15.119:6838/937297340,v1:172.21.15.119:6839/937297340] conn(0x56434d6f4c50 0x56434d6f7040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T10:32:24.919 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:32:24.912+0000 7f474b7fe640 1 --2- 172.21.15.114:0/820896308 >> [v2:172.21.15.119:6838/937297340,v1:172.21.15.119:6839/937297340] conn(0x56434d6f4c50 0x56434d6f7040 crc :-1 s=READY pgs=112 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=b1aa56290556a648 server_cookie=c52687553466f849 in_seq=0 out_seq=0 2024-04-23T10:32:24.919 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:32:24.912+0000 7f472affd640 10 client.16422 ms_handle_connect on v2:172.21.15.119:6838/937297340 2024-04-23T10:32:24.919 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:32:24.913+0000 7f472affd640 1 -- 172.21.15.114:0/820896308 <== mds.0 v2:172.21.15.119:6838/937297340 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x56434d648390 con 0x56434d6f4c50 2024-04-23T10:32:24.919 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:32:24.913+0000 7f472affd640 10 client.16422 handle_command_reply: tid=0 2024-04-23T10:32:24.919 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:32:24.913+0000 7f47547b5740 1 client.16422 shutdown 2024-04-23T10:32:24.920 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:32:24.913+0000 7f47547b5740 1 -- 172.21.15.114:0/820896308 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x56434d6e4c10 msgr2=0x56434d6e1db0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T10:32:24.920 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:32:24.913+0000 7f47547b5740 1 --2- 172.21.15.114:0/820896308 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x56434d6e4c10 0x56434d6e1db0 secure :-1 s=READY pgs=745 cs=0 l=1 rev1=1 crypto rx=0x7f474401f040 tx=0x7f4744004cf0 comp rx=0 tx=0).stop 2024-04-23T10:32:24.920 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:32:24.913+0000 7f47547b5740 1 -- 172.21.15.114:0/820896308 shutdown_connections 2024-04-23T10:32:24.920 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:32:24.913+0000 7f47547b5740 1 --2- 172.21.15.114:0/820896308 >> [v2:172.21.15.119:6838/937297340,v1:172.21.15.119:6839/937297340] conn(0x56434d6f4c50 0x56434d6f7040 crc :-1 s=READY pgs=112 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:32:24.920 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:32:24.913+0000 7f47547b5740 1 --2- 172.21.15.114:0/820896308 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x56434d6e4c10 0x56434d6e1db0 unknown :-1 s=CLOSED pgs=745 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:32:24.920 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:32:24.913+0000 7f47547b5740 1 --2- 172.21.15.114:0/820896308 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x56434d6d5e20 0x56434d6e1850 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:32:24.920 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:32:24.913+0000 7f472affd640 0 client.16422 ms_handle_reset on v2:172.21.15.119:6838/937297340 2024-04-23T10:32:24.920 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:32:24.913+0000 7f47547b5740 1 --2- 172.21.15.114:0/820896308 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x56434d6d5500 0x56434d6e4560 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:32:24.920 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:32:24.913+0000 7f47547b5740 1 -- 172.21.15.114:0/820896308 >> 172.21.15.114:0/820896308 conn(0x56434d6d1b30 msgr2=0x56434d6d7920 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T10:32:24.920 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:32:24.913+0000 7f47547b5740 1 -- 172.21.15.114:0/820896308 shutdown_connections 2024-04-23T10:32:24.920 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:32:24.913+0000 7f47547b5740 1 -- 172.21.15.114:0/820896308 wait complete. 2024-04-23T10:32:24.920 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:32:24.913+0000 7f47547b5740 20 client.16422 trim_cache size 0 max 16384 2024-04-23T10:32:24.924 INFO:tasks.workunit.client.0.smithi114.stdout: 1 312315 4.74 MB/sec execute 153 sec latency 13.594 ms 2024-04-23T10:32:24.929 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:32:24.928+0000 7f47547b5740 1 Processor -- start 2024-04-23T10:32:24.929 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:32:24.928+0000 7f47547b5740 1 -- start start 2024-04-23T10:32:24.929 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:32:24.929+0000 7f47547b5740 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x56434d6d5500 0x56434d6f0740 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T10:32:24.929 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:32:24.929+0000 7f47547b5740 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x56434d6d5e20 0x56434d6dcd80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T10:32:24.929 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:32:24.929+0000 7f47547b5740 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x56434d6e4c10 0x56434d6e4090 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T10:32:24.929 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:32:24.929+0000 7f47547b5740 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x7f474c1ae370 con 0x56434d6d5e20 2024-04-23T10:32:24.929 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:32:24.929+0000 7f47547b5740 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 -- 0x7f474c1abc40 con 0x56434d6e4c10 2024-04-23T10:32:24.930 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:32:24.929+0000 7f47547b5740 1 -- --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_getmap magic: 0 -- 0x7f474c1a9a70 con 0x56434d6d5500 2024-04-23T10:32:24.930 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:32:24.929+0000 7f474affd640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x56434d6d5e20 0x56434d6dcd80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T10:32:24.930 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:32:24.929+0000 7f474affd640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x56434d6d5e20 0x56434d6dcd80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.114:39274/0 (socket says 172.21.15.114:39274) 2024-04-23T10:32:24.930 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:32:24.929+0000 7f474affd640 1 -- 172.21.15.114:0/1332760034 learned_addr learned my addr 172.21.15.114:0/1332760034 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-04-23T10:32:24.930 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:32:24.929+0000 7f474bfff640 1 --2- 172.21.15.114:0/1332760034 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x56434d6e4c10 0x56434d6e4090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T10:32:24.930 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:32:24.929+0000 7f474b7fe640 1 --2- 172.21.15.114:0/1332760034 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x56434d6d5500 0x56434d6f0740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T10:32:24.930 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:32:24.929+0000 7f474affd640 1 -- 172.21.15.114:0/1332760034 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x56434d6d5500 msgr2=0x56434d6f0740 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T10:32:24.930 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:32:24.929+0000 7f474affd640 1 --2- 172.21.15.114:0/1332760034 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x56434d6d5500 0x56434d6f0740 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:32:24.930 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:32:24.929+0000 7f474affd640 1 -- 172.21.15.114:0/1332760034 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x56434d6e4c10 msgr2=0x56434d6e4090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T10:32:24.930 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:32:24.929+0000 7f474affd640 1 --2- 172.21.15.114:0/1332760034 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x56434d6e4c10 0x56434d6e4090 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:32:24.930 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:32:24.929+0000 7f474affd640 1 -- 172.21.15.114:0/1332760034 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x56434d6d1960 con 0x56434d6d5e20 2024-04-23T10:32:24.930 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:32:24.929+0000 7f474b7fe640 1 --2- 172.21.15.114:0/1332760034 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x56434d6d5500 0x56434d6f0740 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-23T10:32:24.930 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:32:24.929+0000 7f474affd640 1 --2- 172.21.15.114:0/1332760034 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x56434d6d5e20 0x56434d6dcd80 secure :-1 s=READY pgs=746 cs=0 l=1 rev1=1 crypto rx=0x7f4740004390 tx=0x7f4740014000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=cff5ac6d544345f5 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T10:32:24.930 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:32:24.929+0000 7f47317fa640 1 -- 172.21.15.114:0/1332760034 <== mon.0 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4740003a70 con 0x56434d6d5e20 2024-04-23T10:32:24.931 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:32:24.929+0000 7f47317fa640 1 -- 172.21.15.114:0/1332760034 <== mon.0 v2:172.21.15.114:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f4740003c10 con 0x56434d6d5e20 2024-04-23T10:32:24.931 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:32:24.929+0000 7f47317fa640 1 -- 172.21.15.114:0/1332760034 <== mon.0 v2:172.21.15.114:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f474001c670 con 0x56434d6d5e20 2024-04-23T10:32:24.931 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:32:24.930+0000 7f47547b5740 1 -- 172.21.15.114:0/1332760034 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x56434d6d5e20 msgr2=0x56434d6dcd80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T10:32:24.931 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:32:24.930+0000 7f47547b5740 1 --2- 172.21.15.114:0/1332760034 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x56434d6d5e20 0x56434d6dcd80 secure :-1 s=READY pgs=746 cs=0 l=1 rev1=1 crypto rx=0x7f4740004390 tx=0x7f4740014000 comp rx=0 tx=0).stop 2024-04-23T10:32:24.931 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:32:24.930+0000 7f47547b5740 1 -- 172.21.15.114:0/1332760034 shutdown_connections 2024-04-23T10:32:24.931 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:32:24.930+0000 7f47547b5740 1 --2- 172.21.15.114:0/1332760034 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x56434d6e4c10 0x56434d6e4090 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:32:24.931 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:32:24.930+0000 7f47547b5740 1 --2- 172.21.15.114:0/1332760034 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x56434d6d5e20 0x56434d6dcd80 unknown :-1 s=CLOSED pgs=746 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:32:24.931 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:32:24.930+0000 7f47547b5740 1 --2- 172.21.15.114:0/1332760034 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x56434d6d5500 0x56434d6f0740 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:32:24.931 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:32:24.930+0000 7f47547b5740 1 -- 172.21.15.114:0/1332760034 >> 172.21.15.114:0/1332760034 conn(0x56434d6d1b30 msgr2=0x56434d6dc670 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T10:32:24.931 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:32:24.930+0000 7f47547b5740 1 -- 172.21.15.114:0/1332760034 shutdown_connections 2024-04-23T10:32:24.931 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:32:24.930+0000 7f47547b5740 1 -- 172.21.15.114:0/1332760034 wait complete. 2024-04-23T10:32:24.931 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:32:24.930+0000 7f47547b5740 1 -- start start 2024-04-23T10:32:24.935 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:32:24.931+0000 7f47547b5740 1 Processor -- start 2024-04-23T10:32:24.935 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:32:24.931+0000 7f47547b5740 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x56434d6d5500 0x56434d6ece40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T10:32:24.935 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:32:24.931+0000 7f47547b5740 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x56434d6d5e20 0x56434d6dded0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T10:32:24.935 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:32:24.931+0000 7f47547b5740 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x56434d6e4c10 0x56434d6de410 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T10:32:24.935 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:32:24.931+0000 7f47547b5740 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x7f474c1aa100 con 0x56434d6d5500 2024-04-23T10:32:24.935 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:32:24.931+0000 7f47547b5740 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 -- 0x7f474c1ab570 con 0x56434d6e4c10 2024-04-23T10:32:24.935 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:32:24.931+0000 7f47547b5740 1 -- --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_getmap magic: 0 -- 0x7f474c1aa7d0 con 0x56434d6d5e20 2024-04-23T10:32:24.935 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:32:24.931+0000 7f47547b5740 10 client.0 _ll_register_callbacks cb 0x7f474400eed0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-23T10:32:24.935 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:32:24.931+0000 7f474b7fe640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x56434d6d5500 0x56434d6ece40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T10:32:24.935 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:32:24.931+0000 7f474b7fe640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x56434d6d5500 0x56434d6ece40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.114:39284/0 (socket says 172.21.15.114:39284) 2024-04-23T10:32:24.935 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:32:24.931+0000 7f474b7fe640 1 -- 172.21.15.114:0/3409291138 learned_addr learned my addr 172.21.15.114:0/3409291138 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-04-23T10:32:24.935 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:32:24.931+0000 7f474bfff640 1 --2- 172.21.15.114:0/3409291138 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x56434d6e4c10 0x56434d6de410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T10:32:24.935 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:32:24.931+0000 7f474affd640 1 --2- 172.21.15.114:0/3409291138 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x56434d6d5e20 0x56434d6dded0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T10:32:24.936 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:32:24.931+0000 7f474b7fe640 1 -- 172.21.15.114:0/3409291138 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x56434d6d5e20 msgr2=0x56434d6dded0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T10:32:24.936 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:32:24.931+0000 7f474b7fe640 1 --2- 172.21.15.114:0/3409291138 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x56434d6d5e20 0x56434d6dded0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:32:24.936 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:32:24.931+0000 7f474b7fe640 1 -- 172.21.15.114:0/3409291138 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x56434d6e4c10 msgr2=0x56434d6de410 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T10:32:24.936 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:32:24.931+0000 7f474b7fe640 1 --2- 172.21.15.114:0/3409291138 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x56434d6e4c10 0x56434d6de410 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:32:24.936 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:32:24.931+0000 7f474b7fe640 1 -- 172.21.15.114:0/3409291138 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({osdmap=0}) -- 0x56434d6e7000 con 0x56434d6d5500 2024-04-23T10:32:24.936 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:32:24.931+0000 7f474affd640 1 --2- 172.21.15.114:0/3409291138 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x56434d6d5e20 0x56434d6dded0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-23T10:32:24.936 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:32:24.931+0000 7f474b7fe640 1 --2- 172.21.15.114:0/3409291138 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x56434d6d5500 0x56434d6ece40 secure :-1 s=READY pgs=747 cs=0 l=1 rev1=1 crypto rx=0x7f473c0bc670 tx=0x7f473c0b82a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=726a6ba687db0f3c server_cookie=0 in_seq=0 out_seq=0 2024-04-23T10:32:24.936 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:32:24.931+0000 7f472affd640 10 client.0 ms_handle_connect on v2:172.21.15.114:3300/0 2024-04-23T10:32:24.936 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:32:24.932+0000 7f472affd640 1 -- 172.21.15.114:0/3409291138 <== mon.0 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f473c0328c0 con 0x56434d6d5500 2024-04-23T10:32:24.936 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:32:24.932+0000 7f472affd640 1 -- 172.21.15.114:0/3409291138 <== mon.0 v2:172.21.15.114:3300/0 2 ==== osd_map(94..94 src has 1..94) ==== 8585+0+0 (secure 0 0 0) 0x7f473c030450 con 0x56434d6d5500 2024-04-23T10:32:24.936 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:32:24.932+0000 7f47547b5740 1 -- 172.21.15.114:0/3409291138 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f4740003c10 con 0x56434d6d5500 2024-04-23T10:32:24.936 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:32:24.932+0000 7f472affd640 1 -- 172.21.15.114:0/3409291138 <== mon.0 v2:172.21.15.114:3300/0 3 ==== mon_get_version_reply(handle=1 version=30) ==== 24+0+0 (secure 0 0 0) 0x7f473c0d5e60 con 0x56434d6d5500 2024-04-23T10:32:24.936 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:32:24.932+0000 7f47547b5740 10 client.16428 fetch_fsmap learned FSMap version 30 2024-04-23T10:32:24.937 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:32:24.932+0000 7f47547b5740 1 -- 172.21.15.114:0/3409291138 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({config=0+,fsmap=30,monmap=4+}) -- 0x56434d75aa20 con 0x56434d6d5500 2024-04-23T10:32:24.937 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:32:24.932+0000 7f472affd640 1 -- 172.21.15.114:0/3409291138 <== mon.0 v2:172.21.15.114:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f473c0d2020 con 0x56434d6d5500 2024-04-23T10:32:24.937 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:32:24.932+0000 7f472affd640 1 -- 172.21.15.114:0/3409291138 <== mon.0 v2:172.21.15.114:3300/0 5 ==== fsmap(e 30) ==== 8338+0+0 (secure 0 0 0) 0x7f473c0c4ec0 con 0x56434d6d5500 2024-04-23T10:32:24.937 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:32:24.932+0000 7f47547b5740 10 client.16428 fetch_fsmap finished waiting for FSMap version 30 2024-04-23T10:32:24.937 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:32:24.932+0000 7f47547b5740 10 client.16428 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-04-23T10:32:24.937 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:32:24.932+0000 7f47547b5740 20 client.16428 populate_metadata read hostname 'smithi114' 2024-04-23T10:32:24.937 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:32:24.932+0000 7f47547b5740 1 --2- 172.21.15.114:0/3409291138 >> [v2:172.21.15.119:6838/937297340,v1:172.21.15.119:6839/937297340] conn(0x56434d75b170 0x56434d75d590 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T10:32:24.937 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:32:24.932+0000 7f47547b5740 4 client.16428 mds_command: new command op to 24545 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-04-23T10:32:24.937 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:32:24.932+0000 7f47547b5740 1 -- 172.21.15.114:0/3409291138 --> [v2:172.21.15.119:6838/937297340,v1:172.21.15.119:6839/937297340] -- command(tid 0: {"prefix": "flush journal"}) -- 0x56434d642200 con 0x56434d75b170 2024-04-23T10:32:24.937 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:32:24.932+0000 7f474affd640 1 --2- 172.21.15.114:0/3409291138 >> [v2:172.21.15.119:6838/937297340,v1:172.21.15.119:6839/937297340] conn(0x56434d75b170 0x56434d75d590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T10:32:24.937 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:32:24.933+0000 7f472affd640 10 client.16428 ms_handle_connect on v2:172.21.15.119:6838/937297340 2024-04-23T10:32:24.937 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:32:24.933+0000 7f474affd640 1 --2- 172.21.15.114:0/3409291138 >> [v2:172.21.15.119:6838/937297340,v1:172.21.15.119:6839/937297340] conn(0x56434d75b170 0x56434d75d590 crc :-1 s=READY pgs=113 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=aa18466f97122665 server_cookie=d6e7001753d57a38 in_seq=0 out_seq=0 2024-04-23T10:32:24.968 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:32:24.968+0000 7f472affd640 1 -- 172.21.15.114:0/3409291138 <== mds.0 v2:172.21.15.119:6838/937297340 1 ==== command_reply(tid 0: 0 ) ==== 8+0+44 (crc 0 0 0) 0x56434d642200 con 0x56434d75b170 2024-04-23T10:32:24.968 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:32:24.968+0000 7f472affd640 10 client.16428 handle_command_reply: tid=0 2024-04-23T10:32:24.968 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:32:24.968+0000 7f47547b5740 1 client.16428 shutdown 2024-04-23T10:32:24.968 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:32:24.968+0000 7f47547b5740 1 -- 172.21.15.114:0/3409291138 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x56434d6d5500 msgr2=0x56434d6ece40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T10:32:24.968 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:32:24.968+0000 7f47547b5740 1 --2- 172.21.15.114:0/3409291138 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x56434d6d5500 0x56434d6ece40 secure :-1 s=READY pgs=747 cs=0 l=1 rev1=1 crypto rx=0x7f473c0bc670 tx=0x7f473c0b82a0 comp rx=0 tx=0).stop 2024-04-23T10:32:24.968 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:32:24.968+0000 7f47547b5740 1 -- 172.21.15.114:0/3409291138 shutdown_connections 2024-04-23T10:32:24.968 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:32:24.968+0000 7f47547b5740 1 --2- 172.21.15.114:0/3409291138 >> [v2:172.21.15.119:6838/937297340,v1:172.21.15.119:6839/937297340] conn(0x56434d75b170 0x56434d75d590 crc :-1 s=READY pgs=113 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:32:24.968 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:32:24.968+0000 7f47547b5740 1 --2- 172.21.15.114:0/3409291138 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x56434d6e4c10 0x56434d6de410 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:32:24.968 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:32:24.968+0000 7f47547b5740 1 --2- 172.21.15.114:0/3409291138 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x56434d6d5e20 0x56434d6dded0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:32:24.968 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:32:24.968+0000 7f47547b5740 1 --2- 172.21.15.114:0/3409291138 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x56434d6d5500 0x56434d6ece40 unknown :-1 s=CLOSED pgs=747 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:32:24.968 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:32:24.968+0000 7f472affd640 0 client.16428 ms_handle_reset on v2:172.21.15.119:6838/937297340 2024-04-23T10:32:24.968 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:32:24.968+0000 7f47547b5740 1 -- 172.21.15.114:0/3409291138 >> 172.21.15.114:0/3409291138 conn(0x56434d6d1b30 msgr2=0x56434d6d0420 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T10:32:24.968 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:32:24.968+0000 7f47547b5740 1 -- 172.21.15.114:0/3409291138 shutdown_connections 2024-04-23T10:32:24.969 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:32:24.968+0000 7f47547b5740 1 -- 172.21.15.114:0/3409291138 wait complete. 2024-04-23T10:32:24.969 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:32:24.969+0000 7f47547b5740 20 client.16428 trim_cache size 0 max 16384 2024-04-23T10:32:24.969 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stdout:{ 2024-04-23T10:32:24.969 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stdout: "message": "", 2024-04-23T10:32:24.969 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stdout: "return_code": 0 2024-04-23T10:32:24.969 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stdout:} 2024-04-23T10:32:24.970 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:32:24.969+0000 7f4751a9a640 1 -- 172.21.15.114:0/469003199 >> [v2:172.21.15.114:6800/1395693466,v1:172.21.15.114:6801/1395693466] conn(0x7f47340a3dc0 msgr2=0x7f47340a6280 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T10:32:24.970 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:32:24.969+0000 7f4751a9a640 1 --2- 172.21.15.114:0/469003199 >> [v2:172.21.15.114:6800/1395693466,v1:172.21.15.114:6801/1395693466] conn(0x7f47340a3dc0 0x7f47340a6280 secure :-1 s=READY pgs=110 cs=0 l=1 rev1=1 crypto rx=0x7f474c104b10 tx=0x7f4740005f90 comp rx=0 tx=0).stop 2024-04-23T10:32:24.970 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:32:24.969+0000 7f4751a9a640 1 -- 172.21.15.114:0/469003199 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f474c100720 msgr2=0x7f474c1987a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T10:32:24.970 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:32:24.969+0000 7f4751a9a640 1 --2- 172.21.15.114:0/469003199 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f474c100720 0x7f474c1987a0 secure :-1 s=READY pgs=744 cs=0 l=1 rev1=1 crypto rx=0x7f473c0303f0 tx=0x7f473c032da0 comp rx=0 tx=0).stop 2024-04-23T10:32:24.970 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:32:24.969+0000 7f4751a9a640 1 -- 172.21.15.114:0/469003199 shutdown_connections 2024-04-23T10:32:24.970 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:32:24.969+0000 7f4751a9a640 1 --2- 172.21.15.114:0/469003199 >> [v2:172.21.15.114:6800/1395693466,v1:172.21.15.114:6801/1395693466] conn(0x7f47340a3dc0 0x7f47340a6280 unknown :-1 s=CLOSED pgs=110 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:32:24.971 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:32:24.969+0000 7f4751a9a640 1 --2- 172.21.15.114:0/469003199 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f474c19d010 0x7f474c19d4c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:32:24.971 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:32:24.969+0000 7f4751a9a640 1 --2- 172.21.15.114:0/469003199 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f474c107330 0x7f474c198ce0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:32:24.971 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:32:24.969+0000 7f4751a9a640 1 --2- 172.21.15.114:0/469003199 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f474c100720 0x7f474c1987a0 unknown :-1 s=CLOSED pgs=744 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:32:24.971 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:32:24.969+0000 7f4751a9a640 1 -- 172.21.15.114:0/469003199 >> 172.21.15.114:0/469003199 conn(0x7f474c077390 msgr2=0x7f474c0fe760 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T10:32:24.971 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:32:24.969+0000 7f4751a9a640 1 -- 172.21.15.114:0/469003199 shutdown_connections 2024-04-23T10:32:24.971 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:32:24.969+0000 7f4751a9a640 1 -- 172.21.15.114:0/469003199 wait complete. 2024-04-23T10:32:25.924 INFO:tasks.workunit.client.0.smithi114.stdout: 1 313674 4.75 MB/sec execute 154 sec latency 25.517 ms 2024-04-23T10:32:26.491 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 23 10:32:26 smithi119 ceph-mon[32608]: pgmap v179: 97 pgs: 97 active+clean; 78 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 10 KiB/s rd, 11 KiB/s wr, 8 op/s 2024-04-23T10:32:26.503 INFO:journalctl@ceph.mon.a.smithi114.stdout:Apr 23 10:32:26 smithi114 ceph-mon[29255]: pgmap v179: 97 pgs: 97 active+clean; 78 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 10 KiB/s rd, 11 KiB/s wr, 8 op/s 2024-04-23T10:32:26.577 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 23 10:32:26 smithi204 ceph-mon[32649]: pgmap v179: 97 pgs: 97 active+clean; 78 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 10 KiB/s rd, 11 KiB/s wr, 8 op/s 2024-04-23T10:32:26.924 INFO:tasks.workunit.client.0.smithi114.stdout: 1 316054 4.81 MB/sec execute 155 sec latency 16.109 ms 2024-04-23T10:32:27.924 INFO:tasks.workunit.client.0.smithi114.stdout: 1 317684 4.81 MB/sec execute 156 sec latency 20.649 ms 2024-04-23T10:32:28.491 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 23 10:32:28 smithi119 ceph-mon[32608]: pgmap v180: 97 pgs: 97 active+clean; 86 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 MiB/s rd, 787 KiB/s wr, 122 op/s 2024-04-23T10:32:28.504 INFO:journalctl@ceph.mon.a.smithi114.stdout:Apr 23 10:32:28 smithi114 ceph-mon[29255]: pgmap v180: 97 pgs: 97 active+clean; 86 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 MiB/s rd, 787 KiB/s wr, 122 op/s 2024-04-23T10:32:28.577 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 23 10:32:28 smithi204 ceph-mon[32649]: pgmap v180: 97 pgs: 97 active+clean; 86 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 MiB/s rd, 787 KiB/s wr, 122 op/s 2024-04-23T10:32:28.924 INFO:tasks.workunit.client.0.smithi114.stdout: 1 319704 4.87 MB/sec execute 157 sec latency 16.918 ms 2024-04-23T10:32:29.924 INFO:tasks.workunit.client.0.smithi114.stdout: 1 321701 4.88 MB/sec execute 158 sec latency 21.368 ms 2024-04-23T10:32:30.491 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 23 10:32:30 smithi119 ceph-mon[32608]: pgmap v181: 97 pgs: 97 active+clean; 86 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 MiB/s rd, 778 KiB/s wr, 116 op/s 2024-04-23T10:32:30.503 INFO:journalctl@ceph.mon.a.smithi114.stdout:Apr 23 10:32:30 smithi114 ceph-mon[29255]: pgmap v181: 97 pgs: 97 active+clean; 86 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 MiB/s rd, 778 KiB/s wr, 116 op/s 2024-04-23T10:32:30.577 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 23 10:32:30 smithi204 ceph-mon[32649]: pgmap v181: 97 pgs: 97 active+clean; 86 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 MiB/s rd, 778 KiB/s wr, 116 op/s 2024-04-23T10:32:30.924 INFO:tasks.workunit.client.0.smithi114.stdout: 1 323471 4.93 MB/sec execute 159 sec latency 12.691 ms 2024-04-23T10:32:31.924 INFO:tasks.workunit.client.0.smithi114.stdout: 1 326016 4.98 MB/sec execute 160 sec latency 17.419 ms 2024-04-23T10:32:32.491 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 23 10:32:32 smithi119 ceph-mon[32608]: pgmap v182: 97 pgs: 97 active+clean; 86 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 MiB/s rd, 778 KiB/s wr, 116 op/s 2024-04-23T10:32:32.504 INFO:journalctl@ceph.mon.a.smithi114.stdout:Apr 23 10:32:32 smithi114 ceph-mon[29255]: pgmap v182: 97 pgs: 97 active+clean; 86 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 MiB/s rd, 778 KiB/s wr, 116 op/s 2024-04-23T10:32:32.577 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 23 10:32:32 smithi204 ceph-mon[32649]: pgmap v182: 97 pgs: 97 active+clean; 86 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 MiB/s rd, 778 KiB/s wr, 116 op/s 2024-04-23T10:32:32.924 INFO:tasks.workunit.client.0.smithi114.stdout: 1 327521 5.00 MB/sec execute 161 sec latency 24.461 ms 2024-04-23T10:32:33.925 INFO:tasks.workunit.client.0.smithi114.stdout: 1 330121 5.07 MB/sec execute 162 sec latency 16.726 ms 2024-04-23T10:32:34.493 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 23 10:32:34 smithi119 ceph-mon[32608]: pgmap v183: 97 pgs: 97 active+clean; 94 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.4 MiB/s rd, 1.5 MiB/s wr, 212 op/s 2024-04-23T10:32:34.504 INFO:journalctl@ceph.mon.a.smithi114.stdout:Apr 23 10:32:34 smithi114 ceph-mon[29255]: pgmap v183: 97 pgs: 97 active+clean; 94 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.4 MiB/s rd, 1.5 MiB/s wr, 212 op/s 2024-04-23T10:32:34.577 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 23 10:32:34 smithi204 ceph-mon[32649]: pgmap v183: 97 pgs: 97 active+clean; 94 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.4 MiB/s rd, 1.5 MiB/s wr, 212 op/s 2024-04-23T10:32:34.924 INFO:tasks.workunit.client.0.smithi114.stdout: 1 331394 5.07 MB/sec execute 163 sec latency 23.122 ms 2024-04-23T10:32:35.924 INFO:tasks.workunit.client.0.smithi114.stdout: 1 333632 5.12 MB/sec execute 164 sec latency 40.271 ms 2024-04-23T10:32:36.491 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 23 10:32:36 smithi119 ceph-mon[32608]: pgmap v184: 97 pgs: 97 active+clean; 94 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.4 MiB/s rd, 1.5 MiB/s wr, 209 op/s 2024-04-23T10:32:36.503 INFO:journalctl@ceph.mon.a.smithi114.stdout:Apr 23 10:32:36 smithi114 ceph-mon[29255]: pgmap v184: 97 pgs: 97 active+clean; 94 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.4 MiB/s rd, 1.5 MiB/s wr, 209 op/s 2024-04-23T10:32:36.577 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 23 10:32:36 smithi204 ceph-mon[32649]: pgmap v184: 97 pgs: 97 active+clean; 94 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.4 MiB/s rd, 1.5 MiB/s wr, 209 op/s 2024-04-23T10:32:36.924 INFO:tasks.workunit.client.0.smithi114.stdout: 1 334896 5.13 MB/sec execute 165 sec latency 17.341 ms 2024-04-23T10:32:37.925 INFO:tasks.workunit.client.0.smithi114.stdout: 1 337232 5.18 MB/sec execute 166 sec latency 18.201 ms 2024-04-23T10:32:38.060 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:32:38.061+0000 7f4c81708640 1 -- 172.21.15.114:0/975927869 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f4c7c1021d0 msgr2=0x7f4c7c1025b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T10:32:38.060 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:32:38.061+0000 7f4c81708640 1 --2- 172.21.15.114:0/975927869 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f4c7c1021d0 0x7f4c7c1025b0 secure :-1 s=READY pgs=748 cs=0 l=1 rev1=1 crypto rx=0x7f4c6c009900 tx=0x7f4c6c02a410 comp rx=0 tx=0).stop 2024-04-23T10:32:38.061 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:32:38.061+0000 7f4c81708640 1 -- 172.21.15.114:0/975927869 shutdown_connections 2024-04-23T10:32:38.061 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:32:38.061+0000 7f4c81708640 1 --2- 172.21.15.114:0/975927869 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f4c7c1072a0 0x7f4c7c10ee70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:32:38.061 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:32:38.061+0000 7f4c81708640 1 --2- 172.21.15.114:0/975927869 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f4c7c102b80 0x7f4c7c106bc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:32:38.061 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:32:38.061+0000 7f4c81708640 1 --2- 172.21.15.114:0/975927869 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f4c7c1021d0 0x7f4c7c1025b0 unknown :-1 s=CLOSED pgs=748 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:32:38.061 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:32:38.061+0000 7f4c81708640 1 -- 172.21.15.114:0/975927869 >> 172.21.15.114:0/975927869 conn(0x7f4c7c0793e0 msgr2=0x7f4c7c0797f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T10:32:38.061 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:32:38.061+0000 7f4c81708640 1 -- 172.21.15.114:0/975927869 shutdown_connections 2024-04-23T10:32:38.061 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:32:38.061+0000 7f4c81708640 1 -- 172.21.15.114:0/975927869 wait complete. 2024-04-23T10:32:38.061 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:32:38.062+0000 7f4c81708640 1 Processor -- start 2024-04-23T10:32:38.061 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:32:38.062+0000 7f4c81708640 1 -- start start 2024-04-23T10:32:38.063 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:32:38.062+0000 7f4c81708640 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f4c7c1021d0 0x7f4c7c1986c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T10:32:38.063 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:32:38.062+0000 7f4c81708640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f4c7c1072a0 0x7f4c7c198c00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T10:32:38.063 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:32:38.062+0000 7f4c81708640 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f4c7c19cf80 0x7f4c7c19d430 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T10:32:38.063 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:32:38.062+0000 7f4c81708640 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x7f4c7c112200 con 0x7f4c7c1072a0 2024-04-23T10:32:38.063 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:32:38.062+0000 7f4c81708640 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 -- 0x7f4c7c112080 con 0x7f4c7c1021d0 2024-04-23T10:32:38.063 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:32:38.062+0000 7f4c81708640 1 -- --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_getmap magic: 0 -- 0x7f4c7c112380 con 0x7f4c7c19cf80 2024-04-23T10:32:38.063 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:32:38.062+0000 7f4c7a7fc640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f4c7c1072a0 0x7f4c7c198c00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T10:32:38.064 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:32:38.062+0000 7f4c7a7fc640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f4c7c1072a0 0x7f4c7c198c00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.114:60544/0 (socket says 172.21.15.114:60544) 2024-04-23T10:32:38.064 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:32:38.062+0000 7f4c7a7fc640 1 -- 172.21.15.114:0/2875059613 learned_addr learned my addr 172.21.15.114:0/2875059613 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-04-23T10:32:38.064 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:32:38.062+0000 7f4c7affd640 1 --2- 172.21.15.114:0/2875059613 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f4c7c1021d0 0x7f4c7c1986c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T10:32:38.064 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:32:38.062+0000 7f4c7b7fe640 1 --2- 172.21.15.114:0/2875059613 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f4c7c19cf80 0x7f4c7c19d430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T10:32:38.064 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:32:38.062+0000 7f4c7a7fc640 1 -- 172.21.15.114:0/2875059613 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f4c7c19cf80 msgr2=0x7f4c7c19d430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T10:32:38.064 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:32:38.062+0000 7f4c7a7fc640 1 --2- 172.21.15.114:0/2875059613 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f4c7c19cf80 0x7f4c7c19d430 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:32:38.064 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:32:38.062+0000 7f4c7a7fc640 1 -- 172.21.15.114:0/2875059613 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f4c7c1021d0 msgr2=0x7f4c7c1986c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T10:32:38.064 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:32:38.062+0000 7f4c7a7fc640 1 --2- 172.21.15.114:0/2875059613 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f4c7c1021d0 0x7f4c7c1986c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:32:38.064 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:32:38.062+0000 7f4c7a7fc640 1 -- 172.21.15.114:0/2875059613 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f4c7c19da70 con 0x7f4c7c1072a0 2024-04-23T10:32:38.064 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:32:38.063+0000 7f4c7a7fc640 1 --2- 172.21.15.114:0/2875059613 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f4c7c1072a0 0x7f4c7c198c00 secure :-1 s=READY pgs=749 cs=0 l=1 rev1=1 crypto rx=0x7f4c7000e9f0 tx=0x7f4c7000eeb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T10:32:38.064 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:32:38.063+0000 7f4c63fff640 1 -- 172.21.15.114:0/2875059613 <== mon.0 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4c7000ccc0 con 0x7f4c7c1072a0 2024-04-23T10:32:38.064 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:32:38.063+0000 7f4c63fff640 1 -- 172.21.15.114:0/2875059613 <== mon.0 v2:172.21.15.114:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f4c700054d0 con 0x7f4c7c1072a0 2024-04-23T10:32:38.065 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:32:38.063+0000 7f4c81708640 1 -- 172.21.15.114:0/2875059613 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f4c7c19dd60 con 0x7f4c7c1072a0 2024-04-23T10:32:38.065 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:32:38.063+0000 7f4c63fff640 1 -- 172.21.15.114:0/2875059613 <== mon.0 v2:172.21.15.114:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4c70010430 con 0x7f4c7c1072a0 2024-04-23T10:32:38.065 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:32:38.063+0000 7f4c81708640 1 -- 172.21.15.114:0/2875059613 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f4c7c1a5500 con 0x7f4c7c1072a0 2024-04-23T10:32:38.071 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:32:38.065+0000 7f4c63fff640 1 -- 172.21.15.114:0/2875059613 <== mon.0 v2:172.21.15.114:3300/0 4 ==== mgrmap(e 26) ==== 138778+0+0 (secure 0 0 0) 0x7f4c700046c0 con 0x7f4c7c1072a0 2024-04-23T10:32:38.071 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:32:38.065+0000 7f4c84427740 1 Processor -- start 2024-04-23T10:32:38.071 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:32:38.065+0000 7f4c84427740 1 -- start start 2024-04-23T10:32:38.072 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:32:38.066+0000 7f4c84427740 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x562139bae620 0x562139baea00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T10:32:38.072 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:32:38.066+0000 7f4c84427740 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x562139baef40 0x562139bb92c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T10:32:38.072 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:32:38.066+0000 7f4c84427740 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x562139bb9800 0x562139bbbbf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T10:32:38.072 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:32:38.066+0000 7f4c84427740 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x562139b13970 con 0x562139bae620 2024-04-23T10:32:38.072 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:32:38.066+0000 7f4c84427740 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 -- 0x562139ab04a0 con 0x562139bb9800 2024-04-23T10:32:38.072 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:32:38.066+0000 7f4c84427740 1 -- --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_getmap magic: 0 -- 0x562139b0f930 con 0x562139baef40 2024-04-23T10:32:38.072 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:32:38.066+0000 7f4c63fff640 1 --2- 172.21.15.114:0/2875059613 >> [v2:172.21.15.114:6800/1395693466,v1:172.21.15.114:6801/1395693466] conn(0x7f4c640a3dc0 0x7f4c640a6280 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T10:32:38.072 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:32:38.066+0000 7f4c7b7fe640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x562139bae620 0x562139baea00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T10:32:38.072 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:32:38.066+0000 7f4c7b7fe640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x562139bae620 0x562139baea00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.114:60560/0 (socket says 172.21.15.114:60560) 2024-04-23T10:32:38.072 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:32:38.066+0000 7f4c7a7fc640 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x562139bb9800 0x562139bbbbf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T10:32:38.072 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:32:38.066+0000 7f4c7b7fe640 1 -- 172.21.15.114:0/1670671668 learned_addr learned my addr 172.21.15.114:0/1670671668 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-04-23T10:32:38.072 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:32:38.066+0000 7f4c7b7fe640 1 --2- 172.21.15.114:0/2875059613 >> [v2:172.21.15.114:6800/1395693466,v1:172.21.15.114:6801/1395693466] conn(0x7f4c640a3dc0 0x7f4c640a6280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T10:32:38.072 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:32:38.066+0000 7f4c7affd640 1 --2- 172.21.15.114:0/1670671668 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x562139baef40 0x562139bb92c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T10:32:38.072 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:32:38.066+0000 7f4c63fff640 1 -- 172.21.15.114:0/2875059613 <== mon.0 v2:172.21.15.114:3300/0 5 ==== osd_map(94..94 src has 1..94) ==== 8585+0+0 (secure 0 0 0) 0x7f4c700d0b90 con 0x7f4c7c1072a0 2024-04-23T10:32:38.073 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:32:38.066+0000 7f4c7b7fe640 1 -- 172.21.15.114:0/1670671668 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x562139baef40 msgr2=0x562139bb92c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T10:32:38.073 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:32:38.066+0000 7f4c7b7fe640 1 --2- 172.21.15.114:0/1670671668 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x562139baef40 0x562139bb92c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:32:38.073 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:32:38.066+0000 7f4c7b7fe640 1 -- 172.21.15.114:0/1670671668 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x562139bb9800 msgr2=0x562139bbbbf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T10:32:38.073 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:32:38.066+0000 7f4c7b7fe640 1 --2- 172.21.15.114:0/1670671668 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x562139bb9800 0x562139bbbbf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:32:38.073 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:32:38.066+0000 7f4c7b7fe640 1 -- 172.21.15.114:0/1670671668 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x562139826210 con 0x562139bae620 2024-04-23T10:32:38.073 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:32:38.066+0000 7f4c7a7fc640 1 --2- 172.21.15.114:0/1670671668 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x562139bb9800 0x562139bbbbf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-23T10:32:38.073 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:32:38.066+0000 7f4c7b7fe640 1 --2- 172.21.15.114:0/1670671668 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x562139bae620 0x562139baea00 secure :-1 s=READY pgs=750 cs=0 l=1 rev1=1 crypto rx=0x7f4c74011da0 tx=0x7f4c7400f790 comp rx=0 tx=0).ready entity=mon.0 client_cookie=abf4ca3cf8426a35 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T10:32:38.073 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:32:38.066+0000 7f4c7b7fe640 1 --2- 172.21.15.114:0/2875059613 >> [v2:172.21.15.114:6800/1395693466,v1:172.21.15.114:6801/1395693466] conn(0x7f4c640a3dc0 0x7f4c640a6280 secure :-1 s=READY pgs=111 cs=0 l=1 rev1=1 crypto rx=0x7f4c74009fd0 tx=0x7f4c7401e000 comp rx=0 tx=0).ready entity=mgr.15291 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T10:32:38.073 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:32:38.067+0000 7f4c617fa640 1 -- 172.21.15.114:0/1670671668 <== mon.0 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4c74022070 con 0x562139bae620 2024-04-23T10:32:38.073 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:32:38.067+0000 7f4c617fa640 1 -- 172.21.15.114:0/1670671668 <== mon.0 v2:172.21.15.114:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f4c74009db0 con 0x562139bae620 2024-04-23T10:32:38.073 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:32:38.067+0000 7f4c617fa640 1 -- 172.21.15.114:0/1670671668 <== mon.0 v2:172.21.15.114:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4c74004050 con 0x562139bae620 2024-04-23T10:32:38.073 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:32:38.067+0000 7f4c84427740 1 -- 172.21.15.114:0/1670671668 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x562139bae620 msgr2=0x562139baea00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T10:32:38.073 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:32:38.067+0000 7f4c84427740 1 --2- 172.21.15.114:0/1670671668 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x562139bae620 0x562139baea00 secure :-1 s=READY pgs=750 cs=0 l=1 rev1=1 crypto rx=0x7f4c74011da0 tx=0x7f4c7400f790 comp rx=0 tx=0).stop 2024-04-23T10:32:38.073 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:32:38.067+0000 7f4c84427740 1 -- 172.21.15.114:0/1670671668 shutdown_connections 2024-04-23T10:32:38.074 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:32:38.067+0000 7f4c84427740 1 --2- 172.21.15.114:0/1670671668 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x562139bb9800 0x562139bbbbf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:32:38.074 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:32:38.067+0000 7f4c84427740 1 --2- 172.21.15.114:0/1670671668 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x562139baef40 0x562139bb92c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:32:38.074 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:32:38.067+0000 7f4c84427740 1 --2- 172.21.15.114:0/1670671668 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x562139bae620 0x562139baea00 unknown :-1 s=CLOSED pgs=750 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:32:38.074 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:32:38.067+0000 7f4c84427740 1 -- 172.21.15.114:0/1670671668 >> 172.21.15.114:0/1670671668 conn(0x562139baab30 msgr2=0x562139baaf10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T10:32:38.074 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:32:38.067+0000 7f4c84427740 1 -- 172.21.15.114:0/1670671668 shutdown_connections 2024-04-23T10:32:38.074 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:32:38.067+0000 7f4c84427740 1 -- 172.21.15.114:0/1670671668 wait complete. 2024-04-23T10:32:38.074 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:32:38.068+0000 7f4c84427740 1 -- start start 2024-04-23T10:32:38.074 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:32:38.068+0000 7f4c84427740 1 Processor -- start 2024-04-23T10:32:38.074 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:32:38.068+0000 7f4c84427740 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x562139bae620 0x562139bb9230 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T10:32:38.074 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:32:38.068+0000 7f4c84427740 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x562139baef40 0x562139bb6520 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T10:32:38.074 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:32:38.068+0000 7f4c84427740 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x562139bb9800 0x562139bb6c60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T10:32:38.074 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:32:38.068+0000 7f4c84427740 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x562139bbc3d0 con 0x562139baef40 2024-04-23T10:32:38.074 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:32:38.068+0000 7f4c84427740 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 -- 0x562139aae0b0 con 0x562139bae620 2024-04-23T10:32:38.074 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:32:38.068+0000 7f4c84427740 1 -- --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_getmap magic: 0 -- 0x562139bbc550 con 0x562139bb9800 2024-04-23T10:32:38.075 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:32:38.068+0000 7f4c84427740 10 client.0 _ll_register_callbacks cb 0x562139a9bd30 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-23T10:32:38.075 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:32:38.068+0000 7f4c7a7fc640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x562139baef40 0x562139bb6520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T10:32:38.075 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:32:38.068+0000 7f4c7affd640 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x562139bae620 0x562139bb9230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T10:32:38.075 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:32:38.068+0000 7f4c7a7fc640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x562139baef40 0x562139bb6520 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.114:60570/0 (socket says 172.21.15.114:60570) 2024-04-23T10:32:38.075 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:32:38.068+0000 7f4c7a7fc640 1 -- 172.21.15.114:0/3681726756 learned_addr learned my addr 172.21.15.114:0/3681726756 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-04-23T10:32:38.075 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:32:38.068+0000 7f4c7affd640 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x562139bae620 0x562139bb9230 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.119:3300/0 says I am v2:172.21.15.114:44860/0 (socket says 172.21.15.114:44860) 2024-04-23T10:32:38.075 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:32:38.068+0000 7f4c7b7fe640 1 --2- 172.21.15.114:0/3681726756 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x562139bb9800 0x562139bb6c60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T10:32:38.075 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:32:38.068+0000 7f4c7a7fc640 1 -- 172.21.15.114:0/3681726756 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x562139bb9800 msgr2=0x562139bb6c60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T10:32:38.075 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:32:38.068+0000 7f4c7a7fc640 1 --2- 172.21.15.114:0/3681726756 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x562139bb9800 0x562139bb6c60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:32:38.075 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:32:38.068+0000 7f4c7a7fc640 1 -- 172.21.15.114:0/3681726756 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x562139bae620 msgr2=0x562139bb9230 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T10:32:38.075 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:32:38.068+0000 7f4c7a7fc640 1 --2- 172.21.15.114:0/3681726756 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x562139bae620 0x562139bb9230 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:32:38.075 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:32:38.068+0000 7f4c7a7fc640 1 -- 172.21.15.114:0/3681726756 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5621398f2620 con 0x562139baef40 2024-04-23T10:32:38.075 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:32:38.068+0000 7f4c7affd640 1 --2- 172.21.15.114:0/3681726756 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x562139bae620 0x562139bb9230 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-23T10:32:38.075 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:32:38.068+0000 7f4c7a7fc640 1 --2- 172.21.15.114:0/3681726756 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x562139baef40 0x562139bb6520 secure :-1 s=READY pgs=751 cs=0 l=1 rev1=1 crypto rx=0x7f4c7000e9c0 tx=0x7f4c7006c3d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=10b53ab3916d252d server_cookie=0 in_seq=0 out_seq=0 2024-04-23T10:32:38.076 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:32:38.068+0000 7f4c5eff5640 10 client.0 ms_handle_connect on v2:172.21.15.114:3300/0 2024-04-23T10:32:38.076 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:32:38.069+0000 7f4c5eff5640 1 -- 172.21.15.114:0/3681726756 <== mon.0 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4c7009caf0 con 0x562139baef40 2024-04-23T10:32:38.076 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:32:38.069+0000 7f4c5eff5640 1 -- 172.21.15.114:0/3681726756 <== mon.0 v2:172.21.15.114:3300/0 2 ==== osd_map(94..94 src has 1..94) ==== 8585+0+0 (secure 0 0 0) 0x7f4c7008c0b0 con 0x562139baef40 2024-04-23T10:32:38.076 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:32:38.069+0000 7f4c84427740 1 -- 172.21.15.114:0/3681726756 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f4c74009db0 con 0x562139baef40 2024-04-23T10:32:38.076 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:32:38.069+0000 7f4c5eff5640 1 -- 172.21.15.114:0/3681726756 <== mon.0 v2:172.21.15.114:3300/0 3 ==== mon_get_version_reply(handle=1 version=30) ==== 24+0+0 (secure 0 0 0) 0x7f4c7008c2d0 con 0x562139baef40 2024-04-23T10:32:38.076 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:32:38.069+0000 7f4c84427740 10 client.16440 fetch_fsmap learned FSMap version 30 2024-04-23T10:32:38.076 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:32:38.069+0000 7f4c84427740 1 -- 172.21.15.114:0/3681726756 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({config=0+,fsmap=30,monmap=4+}) -- 0x562139bc83b0 con 0x562139baef40 2024-04-23T10:32:38.076 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:32:38.069+0000 7f4c5eff5640 1 -- 172.21.15.114:0/3681726756 <== mon.0 v2:172.21.15.114:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f4c74009db0 con 0x562139baef40 2024-04-23T10:32:38.076 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:32:38.069+0000 7f4c5eff5640 1 -- 172.21.15.114:0/3681726756 <== mon.0 v2:172.21.15.114:3300/0 5 ==== fsmap(e 30) ==== 8338+0+0 (secure 0 0 0) 0x7f4c7006e0d0 con 0x562139baef40 2024-04-23T10:32:38.076 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:32:38.069+0000 7f4c84427740 10 client.16440 fetch_fsmap finished waiting for FSMap version 30 2024-04-23T10:32:38.076 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:32:38.069+0000 7f4c84427740 10 client.16440 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-04-23T10:32:38.076 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:32:38.069+0000 7f4c84427740 20 client.16440 populate_metadata read hostname 'smithi114' 2024-04-23T10:32:38.076 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:32:38.069+0000 7f4c84427740 1 --2- 172.21.15.114:0/3681726756 >> [v2:172.21.15.119:6838/937297340,v1:172.21.15.119:6839/937297340] conn(0x562139bc9820 0x562139bcbc10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T10:32:38.076 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:32:38.069+0000 7f4c84427740 4 client.16440 mds_command: new command op to 24545 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-23T10:32:38.077 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:32:38.069+0000 7f4c84427740 1 -- 172.21.15.114:0/3681726756 --> [v2:172.21.15.119:6838/937297340,v1:172.21.15.119:6839/937297340] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x562139b21390 con 0x562139bc9820 2024-04-23T10:32:38.077 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:32:38.069+0000 7f4c7affd640 1 --2- 172.21.15.114:0/3681726756 >> [v2:172.21.15.119:6838/937297340,v1:172.21.15.119:6839/937297340] conn(0x562139bc9820 0x562139bcbc10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T10:32:38.077 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:32:38.070+0000 7f4c7affd640 1 --2- 172.21.15.114:0/3681726756 >> [v2:172.21.15.119:6838/937297340,v1:172.21.15.119:6839/937297340] conn(0x562139bc9820 0x562139bcbc10 crc :-1 s=READY pgs=114 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=df1463c36af55653 server_cookie=84a35eaf70af80d4 in_seq=0 out_seq=0 2024-04-23T10:32:38.077 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:32:38.070+0000 7f4c5eff5640 10 client.16440 ms_handle_connect on v2:172.21.15.119:6838/937297340 2024-04-23T10:32:38.077 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:32:38.071+0000 7f4c5eff5640 1 -- 172.21.15.114:0/3681726756 <== mds.0 v2:172.21.15.119:6838/937297340 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x562139b21390 con 0x562139bc9820 2024-04-23T10:32:38.077 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:32:38.071+0000 7f4c5eff5640 10 client.16440 handle_command_reply: tid=0 2024-04-23T10:32:38.077 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:32:38.071+0000 7f4c84427740 1 client.16440 shutdown 2024-04-23T10:32:38.077 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:32:38.071+0000 7f4c84427740 1 -- 172.21.15.114:0/3681726756 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x562139baef40 msgr2=0x562139bb6520 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T10:32:38.077 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:32:38.071+0000 7f4c84427740 1 --2- 172.21.15.114:0/3681726756 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x562139baef40 0x562139bb6520 secure :-1 s=READY pgs=751 cs=0 l=1 rev1=1 crypto rx=0x7f4c7000e9c0 tx=0x7f4c7006c3d0 comp rx=0 tx=0).stop 2024-04-23T10:32:38.077 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:32:38.071+0000 7f4c84427740 1 -- 172.21.15.114:0/3681726756 shutdown_connections 2024-04-23T10:32:38.077 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:32:38.071+0000 7f4c84427740 1 --2- 172.21.15.114:0/3681726756 >> [v2:172.21.15.119:6838/937297340,v1:172.21.15.119:6839/937297340] conn(0x562139bc9820 0x562139bcbc10 crc :-1 s=READY pgs=114 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:32:38.077 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:32:38.071+0000 7f4c84427740 1 --2- 172.21.15.114:0/3681726756 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x562139bb9800 0x562139bb6c60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:32:38.077 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:32:38.071+0000 7f4c84427740 1 --2- 172.21.15.114:0/3681726756 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x562139baef40 0x562139bb6520 unknown :-1 s=CLOSED pgs=751 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:32:38.077 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:32:38.071+0000 7f4c5eff5640 0 client.16440 ms_handle_reset on v2:172.21.15.119:6838/937297340 2024-04-23T10:32:38.078 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:32:38.071+0000 7f4c84427740 1 --2- 172.21.15.114:0/3681726756 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x562139bae620 0x562139bb9230 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:32:38.078 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:32:38.071+0000 7f4c84427740 1 -- 172.21.15.114:0/3681726756 >> 172.21.15.114:0/3681726756 conn(0x562139baab30 msgr2=0x562139bb0860 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T10:32:38.078 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:32:38.071+0000 7f4c84427740 1 -- 172.21.15.114:0/3681726756 shutdown_connections 2024-04-23T10:32:38.078 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:32:38.071+0000 7f4c84427740 1 -- 172.21.15.114:0/3681726756 wait complete. 2024-04-23T10:32:38.078 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:32:38.071+0000 7f4c84427740 20 client.16440 trim_cache size 0 max 16384 2024-04-23T10:32:38.090 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:32:38.086+0000 7f4c84427740 1 Processor -- start 2024-04-23T10:32:38.090 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:32:38.086+0000 7f4c84427740 1 -- start start 2024-04-23T10:32:38.090 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:32:38.087+0000 7f4c84427740 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x562139bae620 0x562139bc5290 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T10:32:38.090 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:32:38.087+0000 7f4c84427740 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x562139baef40 0x562139bb0aa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T10:32:38.090 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:32:38.087+0000 7f4c84427740 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x562139bb9800 0x562139baa430 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T10:32:38.090 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:32:38.087+0000 7f4c84427740 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x7f4c7c1b0470 con 0x562139baef40 2024-04-23T10:32:38.090 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:32:38.087+0000 7f4c84427740 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 -- 0x7f4c7c1add40 con 0x562139bb9800 2024-04-23T10:32:38.090 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:32:38.087+0000 7f4c84427740 1 -- --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_getmap magic: 0 -- 0x7f4c7c1abb90 con 0x562139bae620 2024-04-23T10:32:38.090 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:32:38.087+0000 7f4c7a7fc640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x562139baef40 0x562139bb0aa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T10:32:38.090 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:32:38.087+0000 7f4c7a7fc640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x562139baef40 0x562139bb0aa0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.114:60572/0 (socket says 172.21.15.114:60572) 2024-04-23T10:32:38.090 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:32:38.087+0000 7f4c7a7fc640 1 -- 172.21.15.114:0/837080414 learned_addr learned my addr 172.21.15.114:0/837080414 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-04-23T10:32:38.090 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:32:38.087+0000 7f4c7b7fe640 1 --2- 172.21.15.114:0/837080414 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x562139bb9800 0x562139baa430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T10:32:38.090 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:32:38.087+0000 7f4c7affd640 1 --2- 172.21.15.114:0/837080414 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x562139bae620 0x562139bc5290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T10:32:38.090 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:32:38.087+0000 7f4c7a7fc640 1 -- 172.21.15.114:0/837080414 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x562139bae620 msgr2=0x562139bc5290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T10:32:38.091 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:32:38.087+0000 7f4c7a7fc640 1 --2- 172.21.15.114:0/837080414 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x562139bae620 0x562139bc5290 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:32:38.091 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:32:38.087+0000 7f4c7a7fc640 1 -- 172.21.15.114:0/837080414 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x562139bb9800 msgr2=0x562139baa430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T10:32:38.091 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:32:38.087+0000 7f4c7a7fc640 1 --2- 172.21.15.114:0/837080414 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x562139bb9800 0x562139baa430 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:32:38.091 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:32:38.087+0000 7f4c7a7fc640 1 -- 172.21.15.114:0/837080414 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x562139bb6fd0 con 0x562139baef40 2024-04-23T10:32:38.091 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:32:38.087+0000 7f4c7affd640 1 --2- 172.21.15.114:0/837080414 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x562139bae620 0x562139bc5290 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-23T10:32:38.091 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:32:38.087+0000 7f4c7a7fc640 1 --2- 172.21.15.114:0/837080414 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x562139baef40 0x562139bb0aa0 secure :-1 s=READY pgs=752 cs=0 l=1 rev1=1 crypto rx=0x7f4c7009ada0 tx=0x7f4c70009770 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b573ae7fa5be76ab server_cookie=0 in_seq=0 out_seq=0 2024-04-23T10:32:38.091 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:32:38.087+0000 7f4c60ff9640 1 -- 172.21.15.114:0/837080414 <== mon.0 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4c7009caf0 con 0x562139baef40 2024-04-23T10:32:38.091 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:32:38.087+0000 7f4c60ff9640 1 -- 172.21.15.114:0/837080414 <== mon.0 v2:172.21.15.114:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f4c700a0ea0 con 0x562139baef40 2024-04-23T10:32:38.091 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:32:38.087+0000 7f4c60ff9640 1 -- 172.21.15.114:0/837080414 <== mon.0 v2:172.21.15.114:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4c700a52d0 con 0x562139baef40 2024-04-23T10:32:38.091 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:32:38.088+0000 7f4c84427740 1 -- 172.21.15.114:0/837080414 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x562139baef40 msgr2=0x562139bb0aa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T10:32:38.091 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:32:38.088+0000 7f4c84427740 1 --2- 172.21.15.114:0/837080414 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x562139baef40 0x562139bb0aa0 secure :-1 s=READY pgs=752 cs=0 l=1 rev1=1 crypto rx=0x7f4c7009ada0 tx=0x7f4c70009770 comp rx=0 tx=0).stop 2024-04-23T10:32:38.091 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:32:38.088+0000 7f4c84427740 1 -- 172.21.15.114:0/837080414 shutdown_connections 2024-04-23T10:32:38.091 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:32:38.088+0000 7f4c84427740 1 --2- 172.21.15.114:0/837080414 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x562139bb9800 0x562139baa430 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:32:38.092 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:32:38.088+0000 7f4c84427740 1 --2- 172.21.15.114:0/837080414 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x562139baef40 0x562139bb0aa0 unknown :-1 s=CLOSED pgs=752 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:32:38.092 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:32:38.088+0000 7f4c84427740 1 --2- 172.21.15.114:0/837080414 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x562139bae620 0x562139bc5290 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:32:38.092 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:32:38.088+0000 7f4c84427740 1 -- 172.21.15.114:0/837080414 >> 172.21.15.114:0/837080414 conn(0x562139baab30 msgr2=0x562139bb19d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T10:32:38.092 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:32:38.088+0000 7f4c84427740 1 -- 172.21.15.114:0/837080414 shutdown_connections 2024-04-23T10:32:38.092 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:32:38.088+0000 7f4c84427740 1 -- 172.21.15.114:0/837080414 wait complete. 2024-04-23T10:32:38.092 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:32:38.088+0000 7f4c84427740 1 -- start start 2024-04-23T10:32:38.092 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:32:38.088+0000 7f4c84427740 1 Processor -- start 2024-04-23T10:32:38.092 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:32:38.089+0000 7f4c84427740 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x562139bae620 0x562139bb3e80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T10:32:38.092 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:32:38.089+0000 7f4c84427740 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x562139baef40 0x562139bb27b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T10:32:38.092 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:32:38.089+0000 7f4c84427740 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x562139bb9800 0x562139bb2cf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T10:32:38.092 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:32:38.089+0000 7f4c84427740 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x7f4c7c1ac200 con 0x562139bb9800 2024-04-23T10:32:38.092 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:32:38.089+0000 7f4c84427740 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 -- 0x7f4c7c1ad670 con 0x562139baef40 2024-04-23T10:32:38.092 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:32:38.089+0000 7f4c84427740 1 -- --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_getmap magic: 0 -- 0x7f4c7c1ac8d0 con 0x562139bae620 2024-04-23T10:32:38.092 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:32:38.089+0000 7f4c84427740 10 client.0 _ll_register_callbacks cb 0x7f4c7009c630 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-23T10:32:38.093 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:32:38.089+0000 7f4c7b7fe640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x562139bb9800 0x562139bb2cf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T10:32:38.093 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:32:38.089+0000 7f4c7b7fe640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x562139bb9800 0x562139bb2cf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.114:60588/0 (socket says 172.21.15.114:60588) 2024-04-23T10:32:38.093 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:32:38.089+0000 7f4c7b7fe640 1 -- 172.21.15.114:0/789347517 learned_addr learned my addr 172.21.15.114:0/789347517 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-04-23T10:32:38.093 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:32:38.089+0000 7f4c7a7fc640 1 --2- 172.21.15.114:0/789347517 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x562139baef40 0x562139bb27b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T10:32:38.093 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:32:38.089+0000 7f4c7affd640 1 --2- 172.21.15.114:0/789347517 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x562139bae620 0x562139bb3e80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T10:32:38.093 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:32:38.089+0000 7f4c7b7fe640 1 -- 172.21.15.114:0/789347517 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x562139bae620 msgr2=0x562139bb3e80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T10:32:38.093 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:32:38.089+0000 7f4c7b7fe640 1 --2- 172.21.15.114:0/789347517 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x562139bae620 0x562139bb3e80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:32:38.093 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:32:38.089+0000 7f4c7b7fe640 1 -- 172.21.15.114:0/789347517 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x562139baef40 msgr2=0x562139bb27b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T10:32:38.093 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:32:38.089+0000 7f4c7b7fe640 1 --2- 172.21.15.114:0/789347517 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x562139baef40 0x562139bb27b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:32:38.093 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:32:38.089+0000 7f4c7b7fe640 1 -- 172.21.15.114:0/789347517 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({osdmap=0}) -- 0x562139bb32b0 con 0x562139bb9800 2024-04-23T10:32:38.093 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:32:38.089+0000 7f4c7b7fe640 1 --2- 172.21.15.114:0/789347517 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x562139bb9800 0x562139bb2cf0 secure :-1 s=READY pgs=753 cs=0 l=1 rev1=1 crypto rx=0x7f4c74014070 tx=0x7f4c74009db0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a7b820fbb41d362 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T10:32:38.093 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:32:38.089+0000 7f4c5eff5640 10 client.0 ms_handle_connect on v2:172.21.15.114:3300/0 2024-04-23T10:32:38.093 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:32:38.089+0000 7f4c5eff5640 1 -- 172.21.15.114:0/789347517 <== mon.0 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4c74032070 con 0x562139bb9800 2024-04-23T10:32:38.094 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:32:38.089+0000 7f4c5eff5640 1 -- 172.21.15.114:0/789347517 <== mon.0 v2:172.21.15.114:3300/0 2 ==== osd_map(94..94 src has 1..94) ==== 8585+0+0 (secure 0 0 0) 0x7f4c74003c50 con 0x562139bb9800 2024-04-23T10:32:38.094 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:32:38.089+0000 7f4c84427740 1 -- 172.21.15.114:0/789347517 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f4c700a0ea0 con 0x562139bb9800 2024-04-23T10:32:38.094 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:32:38.089+0000 7f4c5eff5640 1 -- 172.21.15.114:0/789347517 <== mon.0 v2:172.21.15.114:3300/0 3 ==== mon_get_version_reply(handle=1 version=30) ==== 24+0+0 (secure 0 0 0) 0x7f4c74022040 con 0x562139bb9800 2024-04-23T10:32:38.094 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:32:38.089+0000 7f4c84427740 10 client.16446 fetch_fsmap learned FSMap version 30 2024-04-23T10:32:38.094 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:32:38.089+0000 7f4c84427740 1 -- 172.21.15.114:0/789347517 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({config=0+,fsmap=30,monmap=4+}) -- 0x562139c34ac0 con 0x562139bb9800 2024-04-23T10:32:38.094 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:32:38.090+0000 7f4c5eff5640 1 -- 172.21.15.114:0/789347517 <== mon.0 v2:172.21.15.114:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f4c700a0ea0 con 0x562139bb9800 2024-04-23T10:32:38.094 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:32:38.090+0000 7f4c5eff5640 1 -- 172.21.15.114:0/789347517 <== mon.0 v2:172.21.15.114:3300/0 5 ==== fsmap(e 30) ==== 8338+0+0 (secure 0 0 0) 0x7f4c74035020 con 0x562139bb9800 2024-04-23T10:32:38.094 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:32:38.090+0000 7f4c84427740 10 client.16446 fetch_fsmap finished waiting for FSMap version 30 2024-04-23T10:32:38.094 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:32:38.090+0000 7f4c84427740 10 client.16446 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-04-23T10:32:38.094 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:32:38.090+0000 7f4c84427740 20 client.16446 populate_metadata read hostname 'smithi114' 2024-04-23T10:32:38.094 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:32:38.090+0000 7f4c84427740 1 --2- 172.21.15.114:0/789347517 >> [v2:172.21.15.119:6838/937297340,v1:172.21.15.119:6839/937297340] conn(0x562139c35120 0x562139c37540 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T10:32:38.094 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:32:38.090+0000 7f4c84427740 4 client.16446 mds_command: new command op to 24545 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-04-23T10:32:38.094 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:32:38.090+0000 7f4c84427740 1 -- 172.21.15.114:0/789347517 --> [v2:172.21.15.119:6838/937297340,v1:172.21.15.119:6839/937297340] -- command(tid 0: {"prefix": "flush journal"}) -- 0x562139b1b200 con 0x562139c35120 2024-04-23T10:32:38.094 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:32:38.090+0000 7f4c7affd640 1 --2- 172.21.15.114:0/789347517 >> [v2:172.21.15.119:6838/937297340,v1:172.21.15.119:6839/937297340] conn(0x562139c35120 0x562139c37540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T10:32:38.095 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:32:38.090+0000 7f4c5eff5640 10 client.16446 ms_handle_connect on v2:172.21.15.119:6838/937297340 2024-04-23T10:32:38.095 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:32:38.090+0000 7f4c7affd640 1 --2- 172.21.15.114:0/789347517 >> [v2:172.21.15.119:6838/937297340,v1:172.21.15.119:6839/937297340] conn(0x562139c35120 0x562139c37540 crc :-1 s=READY pgs=115 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=f8363228fb594b97 server_cookie=2849656eb41f8a10 in_seq=0 out_seq=0 2024-04-23T10:32:38.148 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:32:38.148+0000 7f4c5eff5640 1 -- 172.21.15.114:0/789347517 <== mds.0 v2:172.21.15.119:6838/937297340 1 ==== command_reply(tid 0: 0 ) ==== 8+0+44 (crc 0 0 0) 0x562139b1b200 con 0x562139c35120 2024-04-23T10:32:38.148 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:32:38.148+0000 7f4c5eff5640 10 client.16446 handle_command_reply: tid=0 2024-04-23T10:32:38.148 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:32:38.148+0000 7f4c84427740 1 client.16446 shutdown 2024-04-23T10:32:38.148 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:32:38.149+0000 7f4c84427740 1 -- 172.21.15.114:0/789347517 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x562139bb9800 msgr2=0x562139bb2cf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T10:32:38.148 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:32:38.149+0000 7f4c84427740 1 --2- 172.21.15.114:0/789347517 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x562139bb9800 0x562139bb2cf0 secure :-1 s=READY pgs=753 cs=0 l=1 rev1=1 crypto rx=0x7f4c74014070 tx=0x7f4c74009db0 comp rx=0 tx=0).stop 2024-04-23T10:32:38.148 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:32:38.149+0000 7f4c84427740 1 -- 172.21.15.114:0/789347517 shutdown_connections 2024-04-23T10:32:38.148 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:32:38.149+0000 7f4c84427740 1 --2- 172.21.15.114:0/789347517 >> [v2:172.21.15.119:6838/937297340,v1:172.21.15.119:6839/937297340] conn(0x562139c35120 0x562139c37540 crc :-1 s=READY pgs=115 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:32:38.148 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:32:38.149+0000 7f4c84427740 1 --2- 172.21.15.114:0/789347517 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x562139bb9800 0x562139bb2cf0 unknown :-1 s=CLOSED pgs=753 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:32:38.148 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:32:38.149+0000 7f4c84427740 1 --2- 172.21.15.114:0/789347517 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x562139baef40 0x562139bb27b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:32:38.149 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:32:38.149+0000 7f4c84427740 1 --2- 172.21.15.114:0/789347517 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x562139bae620 0x562139bb3e80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:32:38.149 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:32:38.149+0000 7f4c5eff5640 0 client.16446 ms_handle_reset on v2:172.21.15.119:6838/937297340 2024-04-23T10:32:38.149 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:32:38.149+0000 7f4c84427740 1 -- 172.21.15.114:0/789347517 >> 172.21.15.114:0/789347517 conn(0x562139baab30 msgr2=0x562139bc4280 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T10:32:38.149 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:32:38.149+0000 7f4c84427740 1 -- 172.21.15.114:0/789347517 shutdown_connections 2024-04-23T10:32:38.149 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:32:38.149+0000 7f4c84427740 1 -- 172.21.15.114:0/789347517 wait complete. 2024-04-23T10:32:38.149 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:32:38.149+0000 7f4c84427740 20 client.16446 trim_cache size 0 max 16384 2024-04-23T10:32:38.149 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stdout:{ 2024-04-23T10:32:38.149 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stdout: "message": "", 2024-04-23T10:32:38.149 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stdout: "return_code": 0 2024-04-23T10:32:38.149 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stdout:} 2024-04-23T10:32:38.150 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:32:38.149+0000 7f4c81708640 1 -- 172.21.15.114:0/2875059613 >> [v2:172.21.15.114:6800/1395693466,v1:172.21.15.114:6801/1395693466] conn(0x7f4c640a3dc0 msgr2=0x7f4c640a6280 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T10:32:38.150 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:32:38.149+0000 7f4c81708640 1 --2- 172.21.15.114:0/2875059613 >> [v2:172.21.15.114:6800/1395693466,v1:172.21.15.114:6801/1395693466] conn(0x7f4c640a3dc0 0x7f4c640a6280 secure :-1 s=READY pgs=111 cs=0 l=1 rev1=1 crypto rx=0x7f4c74009fd0 tx=0x7f4c7401e000 comp rx=0 tx=0).stop 2024-04-23T10:32:38.150 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:32:38.149+0000 7f4c81708640 1 -- 172.21.15.114:0/2875059613 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f4c7c1072a0 msgr2=0x7f4c7c198c00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T10:32:38.150 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:32:38.149+0000 7f4c81708640 1 --2- 172.21.15.114:0/2875059613 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f4c7c1072a0 0x7f4c7c198c00 secure :-1 s=READY pgs=749 cs=0 l=1 rev1=1 crypto rx=0x7f4c7000e9f0 tx=0x7f4c7000eeb0 comp rx=0 tx=0).stop 2024-04-23T10:32:38.150 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:32:38.149+0000 7f4c81708640 1 -- 172.21.15.114:0/2875059613 shutdown_connections 2024-04-23T10:32:38.151 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:32:38.149+0000 7f4c81708640 1 --2- 172.21.15.114:0/2875059613 >> [v2:172.21.15.114:6800/1395693466,v1:172.21.15.114:6801/1395693466] conn(0x7f4c640a3dc0 0x7f4c640a6280 unknown :-1 s=CLOSED pgs=111 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:32:38.151 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:32:38.149+0000 7f4c81708640 1 --2- 172.21.15.114:0/2875059613 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f4c7c19cf80 0x7f4c7c19d430 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:32:38.151 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:32:38.149+0000 7f4c81708640 1 --2- 172.21.15.114:0/2875059613 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f4c7c1072a0 0x7f4c7c198c00 unknown :-1 s=CLOSED pgs=749 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:32:38.151 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:32:38.149+0000 7f4c81708640 1 --2- 172.21.15.114:0/2875059613 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f4c7c1021d0 0x7f4c7c1986c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:32:38.151 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:32:38.149+0000 7f4c81708640 1 -- 172.21.15.114:0/2875059613 >> 172.21.15.114:0/2875059613 conn(0x7f4c7c0793e0 msgr2=0x7f4c7c0ffe10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T10:32:38.151 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:32:38.150+0000 7f4c81708640 1 -- 172.21.15.114:0/2875059613 shutdown_connections 2024-04-23T10:32:38.151 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:32:38.150+0000 7f4c81708640 1 -- 172.21.15.114:0/2875059613 wait complete. 2024-04-23T10:32:38.491 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 23 10:32:38 smithi119 ceph-mon[32608]: pgmap v185: 97 pgs: 97 active+clean; 103 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 4.8 MiB/s rd, 2.4 MiB/s wr, 304 op/s 2024-04-23T10:32:38.504 INFO:journalctl@ceph.mon.a.smithi114.stdout:Apr 23 10:32:38 smithi114 ceph-mon[29255]: pgmap v185: 97 pgs: 97 active+clean; 103 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 4.8 MiB/s rd, 2.4 MiB/s wr, 304 op/s 2024-04-23T10:32:38.577 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 23 10:32:38 smithi204 ceph-mon[32649]: pgmap v185: 97 pgs: 97 active+clean; 103 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 4.8 MiB/s rd, 2.4 MiB/s wr, 304 op/s 2024-04-23T10:32:38.924 INFO:tasks.workunit.client.0.smithi114.stdout: 1 338470 5.18 MB/sec execute 167 sec latency 87.347 ms 2024-04-23T10:32:39.924 INFO:tasks.workunit.client.0.smithi114.stdout: 1 340882 5.23 MB/sec execute 168 sec latency 12.582 ms 2024-04-23T10:32:40.491 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 23 10:32:40 smithi119 ceph-mon[32608]: pgmap v186: 97 pgs: 97 active+clean; 103 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 2.7 MiB/s rd, 1.6 MiB/s wr, 190 op/s 2024-04-23T10:32:40.504 INFO:journalctl@ceph.mon.a.smithi114.stdout:Apr 23 10:32:40 smithi114 ceph-mon[29255]: pgmap v186: 97 pgs: 97 active+clean; 103 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 2.7 MiB/s rd, 1.6 MiB/s wr, 190 op/s 2024-04-23T10:32:40.577 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 23 10:32:40 smithi204 ceph-mon[32649]: pgmap v186: 97 pgs: 97 active+clean; 103 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 2.7 MiB/s rd, 1.6 MiB/s wr, 190 op/s 2024-04-23T10:32:40.924 INFO:tasks.workunit.client.0.smithi114.stdout: 1 342186 5.23 MB/sec execute 169 sec latency 21.562 ms 2024-04-23T10:32:41.925 INFO:tasks.workunit.client.0.smithi114.stdout: 1 344379 5.28 MB/sec execute 170 sec latency 16.949 ms 2024-04-23T10:32:42.491 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 23 10:32:42 smithi119 ceph-mon[32608]: pgmap v187: 97 pgs: 97 active+clean; 103 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 2.7 MiB/s rd, 1.6 MiB/s wr, 190 op/s 2024-04-23T10:32:42.503 INFO:journalctl@ceph.mon.a.smithi114.stdout:Apr 23 10:32:42 smithi114 ceph-mon[29255]: pgmap v187: 97 pgs: 97 active+clean; 103 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 2.7 MiB/s rd, 1.6 MiB/s wr, 190 op/s 2024-04-23T10:32:42.577 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 23 10:32:42 smithi204 ceph-mon[32649]: pgmap v187: 97 pgs: 97 active+clean; 103 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 2.7 MiB/s rd, 1.6 MiB/s wr, 190 op/s 2024-04-23T10:32:42.925 INFO:tasks.workunit.client.0.smithi114.stdout: 1 345762 5.28 MB/sec execute 171 sec latency 17.526 ms 2024-04-23T10:32:43.925 INFO:tasks.workunit.client.0.smithi114.stdout: 1 348048 5.33 MB/sec execute 172 sec latency 13.957 ms 2024-04-23T10:32:44.491 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 23 10:32:44 smithi119 ceph-mon[32608]: pgmap v188: 97 pgs: 97 active+clean; 98 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 4.1 MiB/s rd, 2.4 MiB/s wr, 318 op/s 2024-04-23T10:32:44.503 INFO:journalctl@ceph.mon.a.smithi114.stdout:Apr 23 10:32:44 smithi114 ceph-mon[29255]: pgmap v188: 97 pgs: 97 active+clean; 98 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 4.1 MiB/s rd, 2.4 MiB/s wr, 318 op/s 2024-04-23T10:32:44.578 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 23 10:32:44 smithi204 ceph-mon[32649]: pgmap v188: 97 pgs: 97 active+clean; 98 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 4.1 MiB/s rd, 2.4 MiB/s wr, 318 op/s 2024-04-23T10:32:44.925 INFO:tasks.workunit.client.0.smithi114.stdout: 1 349472 5.33 MB/sec execute 173 sec latency 19.948 ms 2024-04-23T10:32:45.925 INFO:tasks.workunit.client.0.smithi114.stdout: 1 351558 5.38 MB/sec execute 174 sec latency 15.293 ms 2024-04-23T10:32:46.491 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 23 10:32:46 smithi119 ceph-mon[32608]: pgmap v189: 97 pgs: 97 active+clean; 98 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 2.8 MiB/s rd, 1.6 MiB/s wr, 221 op/s 2024-04-23T10:32:46.504 INFO:journalctl@ceph.mon.a.smithi114.stdout:Apr 23 10:32:46 smithi114 ceph-mon[29255]: pgmap v189: 97 pgs: 97 active+clean; 98 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 2.8 MiB/s rd, 1.6 MiB/s wr, 221 op/s 2024-04-23T10:32:46.577 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 23 10:32:46 smithi204 ceph-mon[32649]: pgmap v189: 97 pgs: 97 active+clean; 98 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 2.8 MiB/s rd, 1.6 MiB/s wr, 221 op/s 2024-04-23T10:32:46.925 INFO:tasks.workunit.client.0.smithi114.stdout: 1 353114 5.38 MB/sec execute 175 sec latency 19.874 ms 2024-04-23T10:32:47.925 INFO:tasks.workunit.client.0.smithi114.stdout: 1 355172 5.43 MB/sec execute 176 sec latency 16.418 ms 2024-04-23T10:32:48.491 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 23 10:32:48 smithi119 ceph-mon[32608]: pgmap v190: 97 pgs: 97 active+clean; 106 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 4.0 MiB/s rd, 2.3 MiB/s wr, 310 op/s 2024-04-23T10:32:48.503 INFO:journalctl@ceph.mon.a.smithi114.stdout:Apr 23 10:32:48 smithi114 ceph-mon[29255]: pgmap v190: 97 pgs: 97 active+clean; 106 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 4.0 MiB/s rd, 2.3 MiB/s wr, 310 op/s 2024-04-23T10:32:48.577 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 23 10:32:48 smithi204 ceph-mon[32649]: pgmap v190: 97 pgs: 97 active+clean; 106 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 4.0 MiB/s rd, 2.3 MiB/s wr, 310 op/s 2024-04-23T10:32:48.925 INFO:tasks.workunit.client.0.smithi114.stdout: 1 357258 5.44 MB/sec execute 177 sec latency 19.825 ms 2024-04-23T10:32:49.492 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 23 10:32:49 smithi119 ceph-mon[32608]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=e=up:active} 3 up:standby-replay 6 up:standby 2024-04-23T10:32:49.504 INFO:journalctl@ceph.mon.a.smithi114.stdout:Apr 23 10:32:49 smithi114 ceph-mon[29255]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=e=up:active} 3 up:standby-replay 6 up:standby 2024-04-23T10:32:49.578 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 23 10:32:49 smithi204 ceph-mon[32649]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=e=up:active} 3 up:standby-replay 6 up:standby 2024-04-23T10:32:49.925 INFO:tasks.workunit.client.0.smithi114.stdout: 1 358994 5.48 MB/sec execute 178 sec latency 19.589 ms 2024-04-23T10:32:50.492 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 23 10:32:50 smithi119 ceph-mon[32608]: pgmap v191: 97 pgs: 97 active+clean; 106 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 2.6 MiB/s rd, 1.5 MiB/s wr, 216 op/s 2024-04-23T10:32:50.492 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 23 10:32:50 smithi119 ceph-mon[32608]: from='mgr.15291 172.21.15.114:0/1095789187' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-23T10:32:50.503 INFO:journalctl@ceph.mon.a.smithi114.stdout:Apr 23 10:32:50 smithi114 ceph-mon[29255]: pgmap v191: 97 pgs: 97 active+clean; 106 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 2.6 MiB/s rd, 1.5 MiB/s wr, 216 op/s 2024-04-23T10:32:50.504 INFO:journalctl@ceph.mon.a.smithi114.stdout:Apr 23 10:32:50 smithi114 ceph-mon[29255]: from='mgr.15291 172.21.15.114:0/1095789187' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-23T10:32:50.577 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 23 10:32:50 smithi204 ceph-mon[32649]: pgmap v191: 97 pgs: 97 active+clean; 106 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 2.6 MiB/s rd, 1.5 MiB/s wr, 216 op/s 2024-04-23T10:32:50.577 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 23 10:32:50 smithi204 ceph-mon[32649]: from='mgr.15291 172.21.15.114:0/1095789187' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-23T10:32:50.925 INFO:tasks.workunit.client.0.smithi114.stdout: 1 360667 5.48 MB/sec execute 179 sec latency 16.344 ms 2024-04-23T10:32:51.242 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:32:51.240+0000 7fd08af36640 1 -- 172.21.15.114:0/3747231188 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fd084101310 msgr2=0x7fd0841016f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T10:32:51.242 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:32:51.240+0000 7fd08af36640 1 --2- 172.21.15.114:0/3747231188 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fd084101310 0x7fd0841016f0 secure :-1 s=READY pgs=754 cs=0 l=1 rev1=1 crypto rx=0x7fd078009900 tx=0x7fd07802a410 comp rx=0 tx=0).stop 2024-04-23T10:32:51.242 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:32:51.241+0000 7fd08af36640 1 -- 172.21.15.114:0/3747231188 shutdown_connections 2024-04-23T10:32:51.242 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:32:51.241+0000 7fd08af36640 1 --2- 172.21.15.114:0/3747231188 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fd084106190 0x7fd08410dd80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:32:51.243 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:32:51.241+0000 7fd08af36640 1 --2- 172.21.15.114:0/3747231188 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fd084101cc0 0x7fd084105ab0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:32:51.243 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:32:51.241+0000 7fd08af36640 1 --2- 172.21.15.114:0/3747231188 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fd084101310 0x7fd0841016f0 unknown :-1 s=CLOSED pgs=754 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:32:51.243 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:32:51.241+0000 7fd08af36640 1 -- 172.21.15.114:0/3747231188 >> 172.21.15.114:0/3747231188 conn(0x7fd084077cc0 msgr2=0x7fd0840780d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T10:32:51.243 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:32:51.241+0000 7fd08af36640 1 -- 172.21.15.114:0/3747231188 shutdown_connections 2024-04-23T10:32:51.243 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:32:51.241+0000 7fd08af36640 1 -- 172.21.15.114:0/3747231188 wait complete. 2024-04-23T10:32:51.243 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:32:51.241+0000 7fd08af36640 1 Processor -- start 2024-04-23T10:32:51.243 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:32:51.241+0000 7fd08af36640 1 -- start start 2024-04-23T10:32:51.243 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:32:51.241+0000 7fd08af36640 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fd084101310 0x7fd0841975b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T10:32:51.243 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:32:51.241+0000 7fd08af36640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fd084106190 0x7fd084197af0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T10:32:51.243 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:32:51.241+0000 7fd08af36640 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fd08419be70 0x7fd08419c320 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T10:32:51.243 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:32:51.241+0000 7fd08af36640 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x7fd0841110f0 con 0x7fd084106190 2024-04-23T10:32:51.243 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:32:51.241+0000 7fd08af36640 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 -- 0x7fd084110f70 con 0x7fd08419be70 2024-04-23T10:32:51.243 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:32:51.241+0000 7fd08af36640 1 -- --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_getmap magic: 0 -- 0x7fd084111270 con 0x7fd084101310 2024-04-23T10:32:51.243 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:32:51.242+0000 7fd083fff640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fd084106190 0x7fd084197af0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T10:32:51.243 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:32:51.242+0000 7fd083fff640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fd084106190 0x7fd084197af0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.114:54280/0 (socket says 172.21.15.114:54280) 2024-04-23T10:32:51.243 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:32:51.242+0000 7fd083fff640 1 -- 172.21.15.114:0/3418716458 learned_addr learned my addr 172.21.15.114:0/3418716458 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-04-23T10:32:51.244 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:32:51.242+0000 7fd088cd2640 1 --2- 172.21.15.114:0/3418716458 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fd084101310 0x7fd0841975b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T10:32:51.244 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:32:51.242+0000 7fd0894d3640 1 --2- 172.21.15.114:0/3418716458 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fd08419be70 0x7fd08419c320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T10:32:51.244 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:32:51.242+0000 7fd083fff640 1 -- 172.21.15.114:0/3418716458 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fd084101310 msgr2=0x7fd0841975b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T10:32:51.244 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:32:51.242+0000 7fd083fff640 1 --2- 172.21.15.114:0/3418716458 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fd084101310 0x7fd0841975b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:32:51.244 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:32:51.242+0000 7fd083fff640 1 -- 172.21.15.114:0/3418716458 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fd08419be70 msgr2=0x7fd08419c320 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T10:32:51.244 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:32:51.242+0000 7fd083fff640 1 --2- 172.21.15.114:0/3418716458 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fd08419be70 0x7fd08419c320 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:32:51.244 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:32:51.242+0000 7fd083fff640 1 -- 172.21.15.114:0/3418716458 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd08419c960 con 0x7fd084106190 2024-04-23T10:32:51.244 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:32:51.242+0000 7fd083fff640 1 --2- 172.21.15.114:0/3418716458 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fd084106190 0x7fd084197af0 secure :-1 s=READY pgs=755 cs=0 l=1 rev1=1 crypto rx=0x7fd07000e9c0 tx=0x7fd07000ee80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T10:32:51.244 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:32:51.243+0000 7fd081ffb640 1 -- 172.21.15.114:0/3418716458 <== mon.0 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd07000cd80 con 0x7fd084106190 2024-04-23T10:32:51.244 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:32:51.243+0000 7fd081ffb640 1 -- 172.21.15.114:0/3418716458 <== mon.0 v2:172.21.15.114:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fd0700054d0 con 0x7fd084106190 2024-04-23T10:32:51.244 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:32:51.243+0000 7fd08af36640 1 -- 172.21.15.114:0/3418716458 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fd08419cc50 con 0x7fd084106190 2024-04-23T10:32:51.244 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:32:51.243+0000 7fd081ffb640 1 -- 172.21.15.114:0/3418716458 <== mon.0 v2:172.21.15.114:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd070010430 con 0x7fd084106190 2024-04-23T10:32:51.244 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:32:51.243+0000 7fd08af36640 1 -- 172.21.15.114:0/3418716458 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fd0841a4490 con 0x7fd084106190 2024-04-23T10:32:51.247 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:32:51.245+0000 7fd081ffb640 1 -- 172.21.15.114:0/3418716458 <== mon.0 v2:172.21.15.114:3300/0 4 ==== mgrmap(e 26) ==== 138778+0+0 (secure 0 0 0) 0x7fd070004fd0 con 0x7fd084106190 2024-04-23T10:32:51.247 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:32:51.246+0000 7fd08dc52740 1 Processor -- start 2024-04-23T10:32:51.247 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:32:51.246+0000 7fd08dc52740 1 -- start start 2024-04-23T10:32:51.247 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:32:51.246+0000 7fd081ffb640 1 --2- 172.21.15.114:0/3418716458 >> [v2:172.21.15.114:6800/1395693466,v1:172.21.15.114:6801/1395693466] conn(0x7fd0740a3e40 0x7fd0740a6300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T10:32:51.247 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:32:51.246+0000 7fd08dc52740 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x5610aa90b620 0x5610aa90ba00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T10:32:51.247 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:32:51.246+0000 7fd08dc52740 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x5610aa90bf40 0x5610aa9162c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T10:32:51.247 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:32:51.246+0000 7fd0894d3640 1 --2- 172.21.15.114:0/3418716458 >> [v2:172.21.15.114:6800/1395693466,v1:172.21.15.114:6801/1395693466] conn(0x7fd0740a3e40 0x7fd0740a6300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T10:32:51.247 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:32:51.246+0000 7fd08dc52740 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x5610aa916800 0x5610aa918bf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T10:32:51.247 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:32:51.246+0000 7fd08dc52740 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x5610aa870970 con 0x5610aa916800 2024-04-23T10:32:51.248 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:32:51.246+0000 7fd08dc52740 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 -- 0x5610aa80d4a0 con 0x5610aa90b620 2024-04-23T10:32:51.248 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:32:51.246+0000 7fd08dc52740 1 -- --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_getmap magic: 0 -- 0x5610aa86c930 con 0x5610aa90bf40 2024-04-23T10:32:51.248 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:32:51.246+0000 7fd081ffb640 1 -- 172.21.15.114:0/3418716458 <== mon.0 v2:172.21.15.114:3300/0 5 ==== osd_map(94..94 src has 1..94) ==== 8585+0+0 (secure 0 0 0) 0x7fd0700d0b90 con 0x7fd084106190 2024-04-23T10:32:51.248 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:32:51.246+0000 7fd088cd2640 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x5610aa90b620 0x5610aa90ba00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T10:32:51.248 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:32:51.246+0000 7fd088cd2640 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x5610aa90b620 0x5610aa90ba00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.119:3300/0 says I am v2:172.21.15.114:42434/0 (socket says 172.21.15.114:42434) 2024-04-23T10:32:51.248 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:32:51.246+0000 7fd088cd2640 1 -- 172.21.15.114:0/3404925977 learned_addr learned my addr 172.21.15.114:0/3404925977 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-04-23T10:32:51.248 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:32:51.246+0000 7fd0894d3640 1 --2- 172.21.15.114:0/3404925977 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x5610aa916800 0x5610aa918bf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T10:32:51.248 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:32:51.246+0000 7fd083fff640 1 --2- 172.21.15.114:0/3404925977 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x5610aa90bf40 0x5610aa9162c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T10:32:51.248 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:32:51.246+0000 7fd088cd2640 1 -- 172.21.15.114:0/3404925977 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x5610aa90bf40 msgr2=0x5610aa9162c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T10:32:51.248 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:32:51.246+0000 7fd088cd2640 1 --2- 172.21.15.114:0/3404925977 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x5610aa90bf40 0x5610aa9162c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:32:51.248 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:32:51.246+0000 7fd088cd2640 1 -- 172.21.15.114:0/3404925977 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x5610aa916800 msgr2=0x5610aa918bf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T10:32:51.248 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:32:51.246+0000 7fd088cd2640 1 --2- 172.21.15.114:0/3404925977 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x5610aa916800 0x5610aa918bf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:32:51.248 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:32:51.246+0000 7fd088cd2640 1 -- 172.21.15.114:0/3404925977 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5610aa583210 con 0x5610aa90b620 2024-04-23T10:32:51.248 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:32:51.246+0000 7fd083fff640 1 --2- 172.21.15.114:0/3404925977 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x5610aa90bf40 0x5610aa9162c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-23T10:32:51.248 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:32:51.247+0000 7fd0894d3640 1 --2- 172.21.15.114:0/3418716458 >> [v2:172.21.15.114:6800/1395693466,v1:172.21.15.114:6801/1395693466] conn(0x7fd0740a3e40 0x7fd0740a6300 secure :-1 s=READY pgs=112 cs=0 l=1 rev1=1 crypto rx=0x7fd07c005fd0 tx=0x7fd07c009dd0 comp rx=0 tx=0).ready entity=mgr.15291 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T10:32:51.248 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:32:51.247+0000 7fd088cd2640 1 --2- 172.21.15.114:0/3404925977 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x5610aa90b620 0x5610aa90ba00 secure :-1 s=READY pgs=74 cs=0 l=1 rev1=1 crypto rx=0x7fd078009900 tx=0x7fd078032b20 comp rx=0 tx=0).ready entity=mon.2 client_cookie=ec28edee3599552c server_cookie=0 in_seq=0 out_seq=0 2024-04-23T10:32:51.249 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:32:51.247+0000 7fd06effd640 1 -- 172.21.15.114:0/3404925977 <== mon.2 v2:172.21.15.119:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd078040070 con 0x5610aa90b620 2024-04-23T10:32:51.249 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:32:51.247+0000 7fd06effd640 1 -- 172.21.15.114:0/3404925977 <== mon.2 v2:172.21.15.119:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fd078004030 con 0x5610aa90b620 2024-04-23T10:32:51.249 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:32:51.247+0000 7fd06effd640 1 -- 172.21.15.114:0/3404925977 <== mon.2 v2:172.21.15.119:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd078035040 con 0x5610aa90b620 2024-04-23T10:32:51.249 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:32:51.248+0000 7fd08dc52740 1 -- 172.21.15.114:0/3404925977 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x5610aa90b620 msgr2=0x5610aa90ba00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T10:32:51.249 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:32:51.248+0000 7fd08dc52740 1 --2- 172.21.15.114:0/3404925977 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x5610aa90b620 0x5610aa90ba00 secure :-1 s=READY pgs=74 cs=0 l=1 rev1=1 crypto rx=0x7fd078009900 tx=0x7fd078032b20 comp rx=0 tx=0).stop 2024-04-23T10:32:51.249 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:32:51.248+0000 7fd08dc52740 1 -- 172.21.15.114:0/3404925977 shutdown_connections 2024-04-23T10:32:51.249 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:32:51.248+0000 7fd08dc52740 1 --2- 172.21.15.114:0/3404925977 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x5610aa916800 0x5610aa918bf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:32:51.249 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:32:51.248+0000 7fd08dc52740 1 --2- 172.21.15.114:0/3404925977 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x5610aa90bf40 0x5610aa9162c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:32:51.249 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:32:51.248+0000 7fd08dc52740 1 --2- 172.21.15.114:0/3404925977 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x5610aa90b620 0x5610aa90ba00 unknown :-1 s=CLOSED pgs=74 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:32:51.249 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:32:51.248+0000 7fd08dc52740 1 -- 172.21.15.114:0/3404925977 >> 172.21.15.114:0/3404925977 conn(0x5610aa907b30 msgr2=0x5610aa907f10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T10:32:51.249 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:32:51.248+0000 7fd08dc52740 1 -- 172.21.15.114:0/3404925977 shutdown_connections 2024-04-23T10:32:51.249 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:32:51.248+0000 7fd08dc52740 1 -- 172.21.15.114:0/3404925977 wait complete. 2024-04-23T10:32:51.249 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:32:51.248+0000 7fd08dc52740 1 -- start start 2024-04-23T10:32:51.249 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:32:51.248+0000 7fd08dc52740 1 Processor -- start 2024-04-23T10:32:51.253 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:32:51.249+0000 7fd08dc52740 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x5610aa90b620 0x5610aa9161a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T10:32:51.253 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:32:51.249+0000 7fd08dc52740 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x5610aa90bf40 0x5610aa913490 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T10:32:51.253 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:32:51.249+0000 7fd08dc52740 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x5610aa916800 0x5610aa913a30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T10:32:51.253 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:32:51.249+0000 7fd08dc52740 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x5610aa9193d0 con 0x5610aa90bf40 2024-04-23T10:32:51.253 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:32:51.249+0000 7fd08dc52740 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 -- 0x5610aa80b0b0 con 0x5610aa916800 2024-04-23T10:32:51.253 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:32:51.249+0000 7fd08dc52740 1 -- --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_getmap magic: 0 -- 0x5610aa919550 con 0x5610aa90b620 2024-04-23T10:32:51.253 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:32:51.249+0000 7fd083fff640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x5610aa90bf40 0x5610aa913490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T10:32:51.253 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:32:51.249+0000 7fd083fff640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x5610aa90bf40 0x5610aa913490 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.114:54308/0 (socket says 172.21.15.114:54308) 2024-04-23T10:32:51.254 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:32:51.249+0000 7fd083fff640 1 -- 172.21.15.114:0/3413779069 learned_addr learned my addr 172.21.15.114:0/3413779069 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-04-23T10:32:51.254 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:32:51.249+0000 7fd088cd2640 1 --2- 172.21.15.114:0/3413779069 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x5610aa90b620 0x5610aa9161a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T10:32:51.254 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:32:51.249+0000 7fd08dc52740 10 client.0 _ll_register_callbacks cb 0x5610aa7f8d30 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-23T10:32:51.254 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:32:51.249+0000 7fd0894d3640 1 --2- 172.21.15.114:0/3413779069 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x5610aa916800 0x5610aa913a30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T10:32:51.254 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:32:51.249+0000 7fd083fff640 1 -- 172.21.15.114:0/3413779069 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x5610aa90b620 msgr2=0x5610aa9161a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T10:32:51.254 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:32:51.249+0000 7fd083fff640 1 --2- 172.21.15.114:0/3413779069 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x5610aa90b620 0x5610aa9161a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:32:51.254 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:32:51.249+0000 7fd083fff640 1 -- 172.21.15.114:0/3413779069 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x5610aa916800 msgr2=0x5610aa913a30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T10:32:51.257 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:32:51.249+0000 7fd083fff640 1 --2- 172.21.15.114:0/3413779069 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x5610aa916800 0x5610aa913a30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:32:51.257 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:32:51.249+0000 7fd083fff640 1 -- 172.21.15.114:0/3413779069 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5610aa64f620 con 0x5610aa90bf40 2024-04-23T10:32:51.257 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:32:51.249+0000 7fd083fff640 1 --2- 172.21.15.114:0/3413779069 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x5610aa90bf40 0x5610aa913490 secure :-1 s=READY pgs=756 cs=0 l=1 rev1=1 crypto rx=0x7fd07009ca80 tx=0x7fd0700046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2cf3f011525d0af8 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T10:32:51.257 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:32:51.249+0000 7fd063fff640 10 client.0 ms_handle_connect on v2:172.21.15.114:3300/0 2024-04-23T10:32:51.257 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:32:51.249+0000 7fd063fff640 1 -- 172.21.15.114:0/3413779069 <== mon.0 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd07009caf0 con 0x5610aa90bf40 2024-04-23T10:32:51.257 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:32:51.249+0000 7fd063fff640 1 -- 172.21.15.114:0/3413779069 <== mon.0 v2:172.21.15.114:3300/0 2 ==== osd_map(94..94 src has 1..94) ==== 8585+0+0 (secure 0 0 0) 0x7fd070099160 con 0x5610aa90bf40 2024-04-23T10:32:51.257 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:32:51.249+0000 7fd08dc52740 1 -- 172.21.15.114:0/3413779069 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fd078004030 con 0x5610aa90bf40 2024-04-23T10:32:51.257 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:32:51.249+0000 7fd063fff640 1 -- 172.21.15.114:0/3413779069 <== mon.0 v2:172.21.15.114:3300/0 3 ==== mon_get_version_reply(handle=1 version=31) ==== 24+0+0 (secure 0 0 0) 0x7fd0700a0ea0 con 0x5610aa90bf40 2024-04-23T10:32:51.257 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:32:51.250+0000 7fd08dc52740 10 client.16458 fetch_fsmap learned FSMap version 31 2024-04-23T10:32:51.257 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:32:51.250+0000 7fd08dc52740 1 -- 172.21.15.114:0/3413779069 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({config=0+,fsmap=31,monmap=4+}) -- 0x5610aa924720 con 0x5610aa90bf40 2024-04-23T10:32:51.257 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:32:51.250+0000 7fd063fff640 1 -- 172.21.15.114:0/3413779069 <== mon.0 v2:172.21.15.114:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fd0700a5760 con 0x5610aa90bf40 2024-04-23T10:32:51.258 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:32:51.250+0000 7fd063fff640 1 -- 172.21.15.114:0/3413779069 <== mon.0 v2:172.21.15.114:3300/0 5 ==== fsmap(e 31) ==== 8342+0+0 (secure 0 0 0) 0x7fd070081030 con 0x5610aa90bf40 2024-04-23T10:32:51.258 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:32:51.250+0000 7fd08dc52740 10 client.16458 fetch_fsmap finished waiting for FSMap version 31 2024-04-23T10:32:51.258 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:32:51.250+0000 7fd08dc52740 10 client.16458 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-04-23T10:32:51.258 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:32:51.250+0000 7fd08dc52740 20 client.16458 populate_metadata read hostname 'smithi114' 2024-04-23T10:32:51.258 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:32:51.250+0000 7fd08dc52740 1 --2- 172.21.15.114:0/3413779069 >> [v2:172.21.15.119:6838/937297340,v1:172.21.15.119:6839/937297340] conn(0x5610aa9267f0 0x5610aa928be0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T10:32:51.258 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:32:51.250+0000 7fd08dc52740 4 client.16458 mds_command: new command op to 24545 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-23T10:32:51.258 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:32:51.250+0000 7fd08dc52740 1 -- 172.21.15.114:0/3413779069 --> [v2:172.21.15.119:6838/937297340,v1:172.21.15.119:6839/937297340] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x7fd078004030 con 0x5610aa9267f0 2024-04-23T10:32:51.258 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:32:51.250+0000 7fd088cd2640 1 --2- 172.21.15.114:0/3413779069 >> [v2:172.21.15.119:6838/937297340,v1:172.21.15.119:6839/937297340] conn(0x5610aa9267f0 0x5610aa928be0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T10:32:51.258 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:32:51.250+0000 7fd063fff640 10 client.16458 ms_handle_connect on v2:172.21.15.119:6838/937297340 2024-04-23T10:32:51.258 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:32:51.250+0000 7fd088cd2640 1 --2- 172.21.15.114:0/3413779069 >> [v2:172.21.15.119:6838/937297340,v1:172.21.15.119:6839/937297340] conn(0x5610aa9267f0 0x5610aa928be0 crc :-1 s=READY pgs=116 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=8405dad728af9a1d server_cookie=678340ea162a3410 in_seq=0 out_seq=0 2024-04-23T10:32:51.258 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:32:51.251+0000 7fd063fff640 1 -- 172.21.15.114:0/3413779069 <== mds.0 v2:172.21.15.119:6838/937297340 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x7fd078004030 con 0x5610aa9267f0 2024-04-23T10:32:51.258 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:32:51.251+0000 7fd063fff640 10 client.16458 handle_command_reply: tid=0 2024-04-23T10:32:51.258 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:32:51.251+0000 7fd08dc52740 1 client.16458 shutdown 2024-04-23T10:32:51.258 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:32:51.251+0000 7fd08dc52740 1 -- 172.21.15.114:0/3413779069 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x5610aa90bf40 msgr2=0x5610aa913490 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T10:32:51.258 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:32:51.251+0000 7fd08dc52740 1 --2- 172.21.15.114:0/3413779069 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x5610aa90bf40 0x5610aa913490 secure :-1 s=READY pgs=756 cs=0 l=1 rev1=1 crypto rx=0x7fd07009ca80 tx=0x7fd0700046c0 comp rx=0 tx=0).stop 2024-04-23T10:32:51.258 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:32:51.251+0000 7fd08dc52740 1 -- 172.21.15.114:0/3413779069 shutdown_connections 2024-04-23T10:32:51.259 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:32:51.251+0000 7fd08dc52740 1 --2- 172.21.15.114:0/3413779069 >> [v2:172.21.15.119:6838/937297340,v1:172.21.15.119:6839/937297340] conn(0x5610aa9267f0 0x5610aa928be0 crc :-1 s=READY pgs=116 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:32:51.259 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:32:51.251+0000 7fd08dc52740 1 --2- 172.21.15.114:0/3413779069 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x5610aa916800 0x5610aa913a30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:32:51.259 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:32:51.251+0000 7fd063fff640 0 client.16458 ms_handle_reset on v2:172.21.15.119:6838/937297340 2024-04-23T10:32:51.259 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:32:51.251+0000 7fd08dc52740 1 --2- 172.21.15.114:0/3413779069 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x5610aa90bf40 0x5610aa913490 unknown :-1 s=CLOSED pgs=756 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:32:51.259 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:32:51.251+0000 7fd08dc52740 1 --2- 172.21.15.114:0/3413779069 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x5610aa90b620 0x5610aa9161a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:32:51.259 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:32:51.251+0000 7fd08dc52740 1 -- 172.21.15.114:0/3413779069 >> 172.21.15.114:0/3413779069 conn(0x5610aa907b30 msgr2=0x5610aa90d860 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T10:32:51.259 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:32:51.252+0000 7fd08dc52740 1 -- 172.21.15.114:0/3413779069 shutdown_connections 2024-04-23T10:32:51.259 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:32:51.252+0000 7fd08dc52740 1 -- 172.21.15.114:0/3413779069 wait complete. 2024-04-23T10:32:51.259 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:32:51.252+0000 7fd08dc52740 20 client.16458 trim_cache size 0 max 16384 2024-04-23T10:32:51.271 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:32:51.267+0000 7fd08dc52740 1 Processor -- start 2024-04-23T10:32:51.271 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:32:51.267+0000 7fd08dc52740 1 -- start start 2024-04-23T10:32:51.271 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:32:51.267+0000 7fd08dc52740 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x5610aa90b620 0x5610aa922290 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T10:32:51.271 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:32:51.267+0000 7fd08dc52740 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x5610aa90bf40 0x5610aa915cd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T10:32:51.271 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:32:51.267+0000 7fd08dc52740 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x5610aa916800 0x5610aa92e4d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T10:32:51.271 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:32:51.267+0000 7fd08dc52740 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x7fd0841ae3c0 con 0x5610aa90bf40 2024-04-23T10:32:51.271 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:32:51.267+0000 7fd08dc52740 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 -- 0x7fd0841abc60 con 0x5610aa90b620 2024-04-23T10:32:51.272 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:32:51.267+0000 7fd08dc52740 1 -- --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_getmap magic: 0 -- 0x7fd0841a9ab0 con 0x5610aa916800 2024-04-23T10:32:51.272 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:32:51.267+0000 7fd083fff640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x5610aa90bf40 0x5610aa915cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T10:32:51.272 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:32:51.267+0000 7fd0894d3640 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x5610aa916800 0x5610aa92e4d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T10:32:51.272 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:32:51.267+0000 7fd088cd2640 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x5610aa90b620 0x5610aa922290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T10:32:51.272 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:32:51.267+0000 7fd083fff640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x5610aa90bf40 0x5610aa915cd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.114:54324/0 (socket says 172.21.15.114:54324) 2024-04-23T10:32:51.272 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:32:51.267+0000 7fd083fff640 1 -- 172.21.15.114:0/48848864 learned_addr learned my addr 172.21.15.114:0/48848864 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-04-23T10:32:51.272 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:32:51.268+0000 7fd083fff640 1 -- 172.21.15.114:0/48848864 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x5610aa916800 msgr2=0x5610aa92e4d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T10:32:51.272 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:32:51.268+0000 7fd083fff640 1 --2- 172.21.15.114:0/48848864 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x5610aa916800 0x5610aa92e4d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:32:51.272 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:32:51.268+0000 7fd083fff640 1 -- 172.21.15.114:0/48848864 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x5610aa90b620 msgr2=0x5610aa922290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T10:32:51.272 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:32:51.268+0000 7fd083fff640 1 --2- 172.21.15.114:0/48848864 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x5610aa90b620 0x5610aa922290 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:32:51.272 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:32:51.268+0000 7fd083fff640 1 -- 172.21.15.114:0/48848864 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5610aa928f50 con 0x5610aa90bf40 2024-04-23T10:32:51.272 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:32:51.268+0000 7fd0894d3640 1 --2- 172.21.15.114:0/48848864 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x5610aa916800 0x5610aa92e4d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-23T10:32:51.272 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:32:51.268+0000 7fd088cd2640 1 --2- 172.21.15.114:0/48848864 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x5610aa90b620 0x5610aa922290 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-23T10:32:51.272 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:32:51.268+0000 7fd083fff640 1 --2- 172.21.15.114:0/48848864 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x5610aa90bf40 0x5610aa915cd0 secure :-1 s=READY pgs=757 cs=0 l=1 rev1=1 crypto rx=0x7fd07000c6a0 tx=0x7fd07000efd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8389ae10f9e15662 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T10:32:51.273 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:32:51.268+0000 7fd06e7fc640 1 -- 172.21.15.114:0/48848864 <== mon.0 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd07009caf0 con 0x5610aa90bf40 2024-04-23T10:32:51.273 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:32:51.268+0000 7fd06e7fc640 1 -- 172.21.15.114:0/48848864 <== mon.0 v2:172.21.15.114:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fd0700a5760 con 0x5610aa90bf40 2024-04-23T10:32:51.273 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:32:51.268+0000 7fd06e7fc640 1 -- 172.21.15.114:0/48848864 <== mon.0 v2:172.21.15.114:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd0700a52d0 con 0x5610aa90bf40 2024-04-23T10:32:51.273 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:32:51.269+0000 7fd08dc52740 1 -- 172.21.15.114:0/48848864 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x5610aa90bf40 msgr2=0x5610aa915cd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T10:32:51.273 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:32:51.269+0000 7fd08dc52740 1 --2- 172.21.15.114:0/48848864 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x5610aa90bf40 0x5610aa915cd0 secure :-1 s=READY pgs=757 cs=0 l=1 rev1=1 crypto rx=0x7fd07000c6a0 tx=0x7fd07000efd0 comp rx=0 tx=0).stop 2024-04-23T10:32:51.273 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:32:51.269+0000 7fd08dc52740 1 -- 172.21.15.114:0/48848864 shutdown_connections 2024-04-23T10:32:51.273 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:32:51.269+0000 7fd08dc52740 1 --2- 172.21.15.114:0/48848864 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x5610aa916800 0x5610aa92e4d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:32:51.273 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:32:51.269+0000 7fd08dc52740 1 --2- 172.21.15.114:0/48848864 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x5610aa90bf40 0x5610aa915cd0 unknown :-1 s=CLOSED pgs=757 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:32:51.273 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:32:51.269+0000 7fd08dc52740 1 --2- 172.21.15.114:0/48848864 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x5610aa90b620 0x5610aa922290 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:32:51.273 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:32:51.269+0000 7fd08dc52740 1 -- 172.21.15.114:0/48848864 >> 172.21.15.114:0/48848864 conn(0x5610aa907b30 msgr2=0x5610aa90eaf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T10:32:51.273 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:32:51.269+0000 7fd08dc52740 1 -- 172.21.15.114:0/48848864 shutdown_connections 2024-04-23T10:32:51.273 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:32:51.269+0000 7fd08dc52740 1 -- 172.21.15.114:0/48848864 wait complete. 2024-04-23T10:32:51.273 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:32:51.269+0000 7fd08dc52740 1 -- start start 2024-04-23T10:32:51.273 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:32:51.269+0000 7fd08dc52740 1 Processor -- start 2024-04-23T10:32:51.274 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:32:51.269+0000 7fd08dc52740 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x5610aa90b620 0x5610aa91ea30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T10:32:51.274 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:32:51.269+0000 7fd08dc52740 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x5610aa90bf40 0x5610aa919da0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T10:32:51.274 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:32:51.269+0000 7fd08dc52740 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x5610aa916800 0x5610aa91a2e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T10:32:51.274 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:32:51.269+0000 7fd08dc52740 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x7fd0841aa120 con 0x5610aa90b620 2024-04-23T10:32:51.274 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:32:51.269+0000 7fd08dc52740 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 -- 0x7fd0841ab590 con 0x5610aa916800 2024-04-23T10:32:51.274 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:32:51.269+0000 7fd08dc52740 1 -- --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_getmap magic: 0 -- 0x7fd0841aa7f0 con 0x5610aa90bf40 2024-04-23T10:32:51.274 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:32:51.269+0000 7fd08dc52740 10 client.0 _ll_register_callbacks cb 0x7fd07009c630 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-23T10:32:51.274 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:32:51.270+0000 7fd088cd2640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x5610aa90b620 0x5610aa91ea30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T10:32:51.274 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:32:51.270+0000 7fd088cd2640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x5610aa90b620 0x5610aa91ea30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.114:54332/0 (socket says 172.21.15.114:54332) 2024-04-23T10:32:51.274 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:32:51.270+0000 7fd088cd2640 1 -- 172.21.15.114:0/3959003687 learned_addr learned my addr 172.21.15.114:0/3959003687 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-04-23T10:32:51.274 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:32:51.270+0000 7fd0894d3640 1 --2- 172.21.15.114:0/3959003687 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x5610aa916800 0x5610aa91a2e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T10:32:51.274 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:32:51.270+0000 7fd083fff640 1 --2- 172.21.15.114:0/3959003687 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x5610aa90bf40 0x5610aa919da0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T10:32:51.274 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:32:51.270+0000 7fd088cd2640 1 -- 172.21.15.114:0/3959003687 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x5610aa90bf40 msgr2=0x5610aa919da0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T10:32:51.274 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:32:51.270+0000 7fd088cd2640 1 --2- 172.21.15.114:0/3959003687 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x5610aa90bf40 0x5610aa919da0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:32:51.275 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:32:51.270+0000 7fd088cd2640 1 -- 172.21.15.114:0/3959003687 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x5610aa916800 msgr2=0x5610aa91a2e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T10:32:51.275 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:32:51.270+0000 7fd088cd2640 1 --2- 172.21.15.114:0/3959003687 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x5610aa916800 0x5610aa91a2e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:32:51.275 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:32:51.270+0000 7fd088cd2640 1 -- 172.21.15.114:0/3959003687 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5610aa913da0 con 0x5610aa90b620 2024-04-23T10:32:51.275 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:32:51.270+0000 7fd088cd2640 1 --2- 172.21.15.114:0/3959003687 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x5610aa90b620 0x5610aa91ea30 secure :-1 s=READY pgs=758 cs=0 l=1 rev1=1 crypto rx=0x7fd07802a930 tx=0x7fd078031400 comp rx=0 tx=0).ready entity=mon.0 client_cookie=715439e5dbfb2afa server_cookie=0 in_seq=0 out_seq=0 2024-04-23T10:32:51.275 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:32:51.270+0000 7fd063fff640 10 client.0 ms_handle_connect on v2:172.21.15.114:3300/0 2024-04-23T10:32:51.275 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:32:51.270+0000 7fd063fff640 1 -- 172.21.15.114:0/3959003687 <== mon.0 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd078040040 con 0x5610aa90b620 2024-04-23T10:32:51.275 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:32:51.270+0000 7fd063fff640 1 -- 172.21.15.114:0/3959003687 <== mon.0 v2:172.21.15.114:3300/0 2 ==== osd_map(94..94 src has 1..94) ==== 8585+0+0 (secure 0 0 0) 0x7fd0780316a0 con 0x5610aa90b620 2024-04-23T10:32:51.275 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:32:51.270+0000 7fd08dc52740 1 -- 172.21.15.114:0/3959003687 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fd0700a5760 con 0x5610aa90b620 2024-04-23T10:32:51.275 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:32:51.270+0000 7fd063fff640 1 -- 172.21.15.114:0/3959003687 <== mon.0 v2:172.21.15.114:3300/0 3 ==== mon_get_version_reply(handle=1 version=31) ==== 24+0+0 (secure 0 0 0) 0x7fd07805e050 con 0x5610aa90b620 2024-04-23T10:32:51.276 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:32:51.270+0000 7fd08dc52740 10 client.16464 fetch_fsmap learned FSMap version 31 2024-04-23T10:32:51.276 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:32:51.270+0000 7fd08dc52740 1 -- 172.21.15.114:0/3959003687 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({config=0+,fsmap=31,monmap=4+}) -- 0x5610aa98ca40 con 0x5610aa90b620 2024-04-23T10:32:51.276 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:32:51.270+0000 7fd063fff640 1 -- 172.21.15.114:0/3959003687 <== mon.0 v2:172.21.15.114:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fd0700a5760 con 0x5610aa90b620 2024-04-23T10:32:51.276 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:32:51.271+0000 7fd063fff640 1 -- 172.21.15.114:0/3959003687 <== mon.0 v2:172.21.15.114:3300/0 5 ==== fsmap(e 31) ==== 8342+0+0 (secure 0 0 0) 0x7fd07804e070 con 0x5610aa90b620 2024-04-23T10:32:51.276 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:32:51.271+0000 7fd08dc52740 10 client.16464 fetch_fsmap finished waiting for FSMap version 31 2024-04-23T10:32:51.276 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:32:51.271+0000 7fd08dc52740 10 client.16464 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-04-23T10:32:51.276 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:32:51.271+0000 7fd08dc52740 20 client.16464 populate_metadata read hostname 'smithi114' 2024-04-23T10:32:51.276 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:32:51.271+0000 7fd08dc52740 1 --2- 172.21.15.114:0/3959003687 >> [v2:172.21.15.119:6838/937297340,v1:172.21.15.119:6839/937297340] conn(0x5610aa9267f0 0x5610aa98ede0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T10:32:51.276 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:32:51.271+0000 7fd08dc52740 4 client.16464 mds_command: new command op to 24545 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-04-23T10:32:51.276 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:32:51.271+0000 7fd08dc52740 1 -- 172.21.15.114:0/3959003687 --> [v2:172.21.15.119:6838/937297340,v1:172.21.15.119:6839/937297340] -- command(tid 0: {"prefix": "flush journal"}) -- 0x5610aa87e390 con 0x5610aa9267f0 2024-04-23T10:32:51.276 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:32:51.271+0000 7fd083fff640 1 --2- 172.21.15.114:0/3959003687 >> [v2:172.21.15.119:6838/937297340,v1:172.21.15.119:6839/937297340] conn(0x5610aa9267f0 0x5610aa98ede0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T10:32:51.277 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:32:51.271+0000 7fd083fff640 1 --2- 172.21.15.114:0/3959003687 >> [v2:172.21.15.119:6838/937297340,v1:172.21.15.119:6839/937297340] conn(0x5610aa9267f0 0x5610aa98ede0 crc :-1 s=READY pgs=117 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=31159e5f404b5bc1 server_cookie=1527689847ea9cef in_seq=0 out_seq=0 2024-04-23T10:32:51.277 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:32:51.271+0000 7fd063fff640 10 client.16464 ms_handle_connect on v2:172.21.15.119:6838/937297340 2024-04-23T10:32:51.335 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:32:51.335+0000 7fd063fff640 1 -- 172.21.15.114:0/3959003687 <== mds.0 v2:172.21.15.119:6838/937297340 1 ==== command_reply(tid 0: 0 ) ==== 8+0+44 (crc 0 0 0) 0x5610aa87e390 con 0x5610aa9267f0 2024-04-23T10:32:51.335 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:32:51.335+0000 7fd063fff640 10 client.16464 handle_command_reply: tid=0 2024-04-23T10:32:51.335 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:32:51.335+0000 7fd08dc52740 1 client.16464 shutdown 2024-04-23T10:32:51.335 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:32:51.336+0000 7fd08dc52740 1 -- 172.21.15.114:0/3959003687 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x5610aa90b620 msgr2=0x5610aa91ea30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T10:32:51.335 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:32:51.336+0000 7fd08dc52740 1 --2- 172.21.15.114:0/3959003687 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x5610aa90b620 0x5610aa91ea30 secure :-1 s=READY pgs=758 cs=0 l=1 rev1=1 crypto rx=0x7fd07802a930 tx=0x7fd078031400 comp rx=0 tx=0).stop 2024-04-23T10:32:51.335 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:32:51.336+0000 7fd08dc52740 1 -- 172.21.15.114:0/3959003687 shutdown_connections 2024-04-23T10:32:51.335 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:32:51.336+0000 7fd08dc52740 1 --2- 172.21.15.114:0/3959003687 >> [v2:172.21.15.119:6838/937297340,v1:172.21.15.119:6839/937297340] conn(0x5610aa9267f0 0x5610aa98ede0 crc :-1 s=READY pgs=117 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:32:51.335 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:32:51.336+0000 7fd08dc52740 1 --2- 172.21.15.114:0/3959003687 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x5610aa916800 0x5610aa91a2e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:32:51.335 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:32:51.336+0000 7fd08dc52740 1 --2- 172.21.15.114:0/3959003687 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x5610aa90bf40 0x5610aa919da0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:32:51.335 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:32:51.336+0000 7fd063fff640 0 client.16464 ms_handle_reset on v2:172.21.15.119:6838/937297340 2024-04-23T10:32:51.335 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:32:51.336+0000 7fd08dc52740 1 --2- 172.21.15.114:0/3959003687 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x5610aa90b620 0x5610aa91ea30 unknown :-1 s=CLOSED pgs=758 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:32:51.336 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:32:51.336+0000 7fd08dc52740 1 -- 172.21.15.114:0/3959003687 >> 172.21.15.114:0/3959003687 conn(0x5610aa907b30 msgr2=0x5610aa921280 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T10:32:51.336 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:32:51.336+0000 7fd08dc52740 1 -- 172.21.15.114:0/3959003687 shutdown_connections 2024-04-23T10:32:51.336 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:32:51.336+0000 7fd08dc52740 1 -- 172.21.15.114:0/3959003687 wait complete. 2024-04-23T10:32:51.336 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:32:51.336+0000 7fd08dc52740 20 client.16464 trim_cache size 0 max 16384 2024-04-23T10:32:51.336 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stdout:{ 2024-04-23T10:32:51.336 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stdout: "message": "", 2024-04-23T10:32:51.336 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stdout: "return_code": 0 2024-04-23T10:32:51.336 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stdout:} 2024-04-23T10:32:51.337 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:32:51.337+0000 7fd08af36640 1 -- 172.21.15.114:0/3418716458 >> [v2:172.21.15.114:6800/1395693466,v1:172.21.15.114:6801/1395693466] conn(0x7fd0740a3e40 msgr2=0x7fd0740a6300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T10:32:51.337 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:32:51.337+0000 7fd08af36640 1 --2- 172.21.15.114:0/3418716458 >> [v2:172.21.15.114:6800/1395693466,v1:172.21.15.114:6801/1395693466] conn(0x7fd0740a3e40 0x7fd0740a6300 secure :-1 s=READY pgs=112 cs=0 l=1 rev1=1 crypto rx=0x7fd07c005fd0 tx=0x7fd07c009dd0 comp rx=0 tx=0).stop 2024-04-23T10:32:51.337 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:32:51.337+0000 7fd08af36640 1 -- 172.21.15.114:0/3418716458 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fd084106190 msgr2=0x7fd084197af0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T10:32:51.337 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:32:51.337+0000 7fd08af36640 1 --2- 172.21.15.114:0/3418716458 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fd084106190 0x7fd084197af0 secure :-1 s=READY pgs=755 cs=0 l=1 rev1=1 crypto rx=0x7fd07000e9c0 tx=0x7fd07000ee80 comp rx=0 tx=0).stop 2024-04-23T10:32:51.337 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:32:51.337+0000 7fd08af36640 1 -- 172.21.15.114:0/3418716458 shutdown_connections 2024-04-23T10:32:51.337 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:32:51.337+0000 7fd08af36640 1 --2- 172.21.15.114:0/3418716458 >> [v2:172.21.15.114:6800/1395693466,v1:172.21.15.114:6801/1395693466] conn(0x7fd0740a3e40 0x7fd0740a6300 unknown :-1 s=CLOSED pgs=112 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:32:51.337 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:32:51.337+0000 7fd08af36640 1 --2- 172.21.15.114:0/3418716458 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fd08419be70 0x7fd08419c320 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:32:51.338 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:32:51.337+0000 7fd08af36640 1 --2- 172.21.15.114:0/3418716458 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fd084106190 0x7fd084197af0 unknown :-1 s=CLOSED pgs=755 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:32:51.338 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:32:51.337+0000 7fd08af36640 1 --2- 172.21.15.114:0/3418716458 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fd084101310 0x7fd0841975b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:32:51.338 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:32:51.337+0000 7fd08af36640 1 -- 172.21.15.114:0/3418716458 >> 172.21.15.114:0/3418716458 conn(0x7fd084077cc0 msgr2=0x7fd084100830 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T10:32:51.338 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:32:51.337+0000 7fd08af36640 1 -- 172.21.15.114:0/3418716458 shutdown_connections 2024-04-23T10:32:51.338 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:32:51.338+0000 7fd08af36640 1 -- 172.21.15.114:0/3418716458 wait complete. 2024-04-23T10:32:51.492 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 23 10:32:51 smithi119 ceph-mon[32608]: from='mgr.15291 172.21.15.114:0/1095789187' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-23T10:32:51.492 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 23 10:32:51 smithi119 ceph-mon[32608]: from='mgr.15291 172.21.15.114:0/1095789187' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-23T10:32:51.492 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 23 10:32:51 smithi119 ceph-mon[32608]: from='mgr.15291 172.21.15.114:0/1095789187' entity='mgr.x' 2024-04-23T10:32:51.504 INFO:journalctl@ceph.mon.a.smithi114.stdout:Apr 23 10:32:51 smithi114 ceph-mon[29255]: from='mgr.15291 172.21.15.114:0/1095789187' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-23T10:32:51.504 INFO:journalctl@ceph.mon.a.smithi114.stdout:Apr 23 10:32:51 smithi114 ceph-mon[29255]: from='mgr.15291 172.21.15.114:0/1095789187' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-23T10:32:51.504 INFO:journalctl@ceph.mon.a.smithi114.stdout:Apr 23 10:32:51 smithi114 ceph-mon[29255]: from='mgr.15291 172.21.15.114:0/1095789187' entity='mgr.x' 2024-04-23T10:32:51.578 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 23 10:32:51 smithi204 ceph-mon[32649]: from='mgr.15291 172.21.15.114:0/1095789187' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-23T10:32:51.578 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 23 10:32:51 smithi204 ceph-mon[32649]: from='mgr.15291 172.21.15.114:0/1095789187' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-23T10:32:51.578 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 23 10:32:51 smithi204 ceph-mon[32649]: from='mgr.15291 172.21.15.114:0/1095789187' entity='mgr.x' 2024-04-23T10:32:51.925 INFO:tasks.workunit.client.0.smithi114.stdout: 1 362263 5.52 MB/sec execute 180 sec latency 81.384 ms 2024-04-23T10:32:52.491 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 23 10:32:52 smithi119 ceph-mon[32608]: pgmap v192: 97 pgs: 97 active+clean; 106 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 2.6 MiB/s rd, 1.5 MiB/s wr, 216 op/s 2024-04-23T10:32:52.504 INFO:journalctl@ceph.mon.a.smithi114.stdout:Apr 23 10:32:52 smithi114 ceph-mon[29255]: pgmap v192: 97 pgs: 97 active+clean; 106 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 2.6 MiB/s rd, 1.5 MiB/s wr, 216 op/s 2024-04-23T10:32:52.577 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 23 10:32:52 smithi204 ceph-mon[32649]: pgmap v192: 97 pgs: 97 active+clean; 106 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 2.6 MiB/s rd, 1.5 MiB/s wr, 216 op/s 2024-04-23T10:32:52.925 INFO:tasks.workunit.client.0.smithi114.stdout: 1 364199 5.53 MB/sec execute 181 sec latency 36.871 ms 2024-04-23T10:32:53.504 INFO:journalctl@ceph.mon.a.smithi114.stdout:Apr 23 10:32:53 smithi114 ceph-mon[29255]: pgmap v193: 97 pgs: 97 active+clean; 113 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 4.0 MiB/s rd, 2.3 MiB/s wr, 345 op/s 2024-04-23T10:32:53.578 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 23 10:32:53 smithi204 ceph-mon[32649]: pgmap v193: 97 pgs: 97 active+clean; 113 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 4.0 MiB/s rd, 2.3 MiB/s wr, 345 op/s 2024-04-23T10:32:53.741 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 23 10:32:53 smithi119 ceph-mon[32608]: pgmap v193: 97 pgs: 97 active+clean; 113 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 4.0 MiB/s rd, 2.3 MiB/s wr, 345 op/s 2024-04-23T10:32:53.925 INFO:tasks.workunit.client.0.smithi114.stdout: 1 365937 5.56 MB/sec execute 182 sec latency 12.952 ms 2024-04-23T10:32:54.925 INFO:tasks.workunit.client.0.smithi114.stdout: 1 368271 5.60 MB/sec execute 183 sec latency 18.584 ms 2024-04-23T10:32:55.741 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 23 10:32:55 smithi119 ceph-mon[32608]: pgmap v194: 97 pgs: 97 active+clean; 113 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 2.6 MiB/s rd, 1.5 MiB/s wr, 217 op/s 2024-04-23T10:32:55.753 INFO:journalctl@ceph.mon.a.smithi114.stdout:Apr 23 10:32:55 smithi114 ceph-mon[29255]: pgmap v194: 97 pgs: 97 active+clean; 113 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 2.6 MiB/s rd, 1.5 MiB/s wr, 217 op/s 2024-04-23T10:32:55.827 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 23 10:32:55 smithi204 ceph-mon[32649]: pgmap v194: 97 pgs: 97 active+clean; 113 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 2.6 MiB/s rd, 1.5 MiB/s wr, 217 op/s 2024-04-23T10:32:55.925 INFO:tasks.workunit.client.0.smithi114.stdout: 1 369810 5.61 MB/sec execute 184 sec latency 20.169 ms 2024-04-23T10:32:56.925 INFO:tasks.workunit.client.0.smithi114.stdout: 1 372436 5.67 MB/sec execute 185 sec latency 19.993 ms 2024-04-23T10:32:57.741 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 23 10:32:57 smithi119 ceph-mon[32608]: pgmap v195: 97 pgs: 97 active+clean; 108 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 4.0 MiB/s rd, 2.3 MiB/s wr, 324 op/s 2024-04-23T10:32:57.753 INFO:journalctl@ceph.mon.a.smithi114.stdout:Apr 23 10:32:57 smithi114 ceph-mon[29255]: pgmap v195: 97 pgs: 97 active+clean; 108 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 4.0 MiB/s rd, 2.3 MiB/s wr, 324 op/s 2024-04-23T10:32:57.827 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 23 10:32:57 smithi204 ceph-mon[32649]: pgmap v195: 97 pgs: 97 active+clean; 108 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 4.0 MiB/s rd, 2.3 MiB/s wr, 324 op/s 2024-04-23T10:32:57.925 INFO:tasks.workunit.client.0.smithi114.stdout: 1 373500 5.66 MB/sec execute 186 sec latency 54.901 ms 2024-04-23T10:32:58.925 INFO:tasks.workunit.client.0.smithi114.stdout: 1 376121 5.72 MB/sec execute 187 sec latency 19.616 ms 2024-04-23T10:32:59.742 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 23 10:32:59 smithi119 ceph-mon[32608]: pgmap v196: 97 pgs: 97 active+clean; 108 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 2.8 MiB/s rd, 1.6 MiB/s wr, 235 op/s 2024-04-23T10:32:59.754 INFO:journalctl@ceph.mon.a.smithi114.stdout:Apr 23 10:32:59 smithi114 ceph-mon[29255]: pgmap v196: 97 pgs: 97 active+clean; 108 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 2.8 MiB/s rd, 1.6 MiB/s wr, 235 op/s 2024-04-23T10:32:59.827 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 23 10:32:59 smithi204 ceph-mon[32649]: pgmap v196: 97 pgs: 97 active+clean; 108 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 2.8 MiB/s rd, 1.6 MiB/s wr, 235 op/s 2024-04-23T10:32:59.925 INFO:tasks.workunit.client.0.smithi114.stdout: 1 377493 5.72 MB/sec execute 188 sec latency 15.586 ms 2024-04-23T10:33:00.926 INFO:tasks.workunit.client.0.smithi114.stdout: 1 379750 5.76 MB/sec execute 189 sec latency 75.260 ms 2024-04-23T10:33:01.741 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 23 10:33:01 smithi119 ceph-mon[32608]: pgmap v197: 97 pgs: 97 active+clean; 108 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 2.8 MiB/s rd, 1.6 MiB/s wr, 235 op/s 2024-04-23T10:33:01.753 INFO:journalctl@ceph.mon.a.smithi114.stdout:Apr 23 10:33:01 smithi114 ceph-mon[29255]: pgmap v197: 97 pgs: 97 active+clean; 108 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 2.8 MiB/s rd, 1.6 MiB/s wr, 235 op/s 2024-04-23T10:33:01.827 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 23 10:33:01 smithi204 ceph-mon[32649]: pgmap v197: 97 pgs: 97 active+clean; 108 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 2.8 MiB/s rd, 1.6 MiB/s wr, 235 op/s 2024-04-23T10:33:01.926 INFO:tasks.workunit.client.0.smithi114.stdout: 1 380795 5.74 MB/sec execute 190 sec latency 30.130 ms 2024-04-23T10:33:02.926 INFO:tasks.workunit.client.0.smithi114.stdout: 1 383168 5.80 MB/sec execute 191 sec latency 26.320 ms 2024-04-23T10:33:03.742 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 23 10:33:03 smithi119 ceph-mon[32608]: pgmap v198: 97 pgs: 97 active+clean; 114 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 4.1 MiB/s rd, 3.1 MiB/s wr, 381 op/s 2024-04-23T10:33:03.753 INFO:journalctl@ceph.mon.a.smithi114.stdout:Apr 23 10:33:03 smithi114 ceph-mon[29255]: pgmap v198: 97 pgs: 97 active+clean; 114 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 4.1 MiB/s rd, 3.1 MiB/s wr, 381 op/s 2024-04-23T10:33:03.827 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 23 10:33:03 smithi204 ceph-mon[32649]: pgmap v198: 97 pgs: 97 active+clean; 114 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 4.1 MiB/s rd, 3.1 MiB/s wr, 381 op/s 2024-04-23T10:33:03.926 INFO:tasks.workunit.client.0.smithi114.stdout: 1 384017 5.78 MB/sec execute 192 sec latency 81.903 ms 2024-04-23T10:33:04.427 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:33:04.427+0000 7f3fe7fff640 1 -- 172.21.15.114:0/3008040600 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f3fe8107080 msgr2=0x7f3fe810ec50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T10:33:04.427 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:33:04.427+0000 7f3fe7fff640 1 --2- 172.21.15.114:0/3008040600 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f3fe8107080 0x7f3fe810ec50 secure :-1 s=READY pgs=759 cs=0 l=1 rev1=1 crypto rx=0x7f3fe0009ca0 tx=0x7f3fe002a8c0 comp rx=0 tx=0).stop 2024-04-23T10:33:04.427 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:33:04.427+0000 7f3fe7fff640 1 -- 172.21.15.114:0/3008040600 shutdown_connections 2024-04-23T10:33:04.427 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:33:04.427+0000 7f3fe7fff640 1 --2- 172.21.15.114:0/3008040600 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f3fe8107080 0x7f3fe810ec50 unknown :-1 s=CLOSED pgs=759 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:33:04.427 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:33:04.427+0000 7f3fe7fff640 1 --2- 172.21.15.114:0/3008040600 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f3fe8100e70 0x7f3fe81069a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:33:04.427 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:33:04.427+0000 7f3fe7fff640 1 --2- 172.21.15.114:0/3008040600 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f3fe81004c0 0x7f3fe81008a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:33:04.427 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:33:04.427+0000 7f3fe7fff640 1 -- 172.21.15.114:0/3008040600 >> 172.21.15.114:0/3008040600 conn(0x7f3fe80771d0 msgr2=0x7f3fe80fe560 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T10:33:04.428 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:33:04.427+0000 7f3fe7fff640 1 -- 172.21.15.114:0/3008040600 shutdown_connections 2024-04-23T10:33:04.428 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:33:04.427+0000 7f3fe7fff640 1 -- 172.21.15.114:0/3008040600 wait complete. 2024-04-23T10:33:04.428 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:33:04.428+0000 7f3fe7fff640 1 Processor -- start 2024-04-23T10:33:04.428 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:33:04.428+0000 7f3fe7fff640 1 -- start start 2024-04-23T10:33:04.430 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:33:04.428+0000 7f3fe7fff640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f3fe81004c0 0x7f3fe81985b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T10:33:04.430 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:33:04.428+0000 7f3fe7fff640 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f3fe8107080 0x7f3fe8198af0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T10:33:04.430 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:33:04.428+0000 7f3fe7fff640 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f3fe819ce20 0x7f3fe819d2d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T10:33:04.430 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:33:04.428+0000 7f3fe7fff640 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x7f3fe8111ff0 con 0x7f3fe81004c0 2024-04-23T10:33:04.430 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:33:04.428+0000 7f3fe7fff640 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 -- 0x7f3fe8111e70 con 0x7f3fe8107080 2024-04-23T10:33:04.430 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:33:04.428+0000 7f3fe7fff640 1 -- --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_getmap magic: 0 -- 0x7f3fe8112170 con 0x7f3fe819ce20 2024-04-23T10:33:04.431 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:33:04.429+0000 7f3fe6ffd640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f3fe81004c0 0x7f3fe81985b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T10:33:04.431 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:33:04.429+0000 7f3fe67fc640 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f3fe8107080 0x7f3fe8198af0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T10:33:04.431 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:33:04.429+0000 7f3fe67fc640 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f3fe8107080 0x7f3fe8198af0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.119:3300/0 says I am v2:172.21.15.114:40196/0 (socket says 172.21.15.114:40196) 2024-04-23T10:33:04.431 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:33:04.429+0000 7f3fe67fc640 1 -- 172.21.15.114:0/1441986308 learned_addr learned my addr 172.21.15.114:0/1441986308 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-04-23T10:33:04.431 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:33:04.429+0000 7f3fe77fe640 1 --2- 172.21.15.114:0/1441986308 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f3fe819ce20 0x7f3fe819d2d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T10:33:04.431 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:33:04.429+0000 7f3fe67fc640 1 -- 172.21.15.114:0/1441986308 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f3fe819ce20 msgr2=0x7f3fe819d2d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T10:33:04.431 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:33:04.429+0000 7f3fe67fc640 1 --2- 172.21.15.114:0/1441986308 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f3fe819ce20 0x7f3fe819d2d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:33:04.431 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:33:04.429+0000 7f3fe67fc640 1 -- 172.21.15.114:0/1441986308 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f3fe81004c0 msgr2=0x7f3fe81985b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T10:33:04.431 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:33:04.429+0000 7f3fe67fc640 1 --2- 172.21.15.114:0/1441986308 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f3fe81004c0 0x7f3fe81985b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:33:04.431 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:33:04.429+0000 7f3fe67fc640 1 -- 172.21.15.114:0/1441986308 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f3fe819d910 con 0x7f3fe8107080 2024-04-23T10:33:04.431 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:33:04.429+0000 7f3fe6ffd640 1 --2- 172.21.15.114:0/1441986308 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f3fe81004c0 0x7f3fe81985b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-23T10:33:04.431 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:33:04.429+0000 7f3fe67fc640 1 --2- 172.21.15.114:0/1441986308 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f3fe8107080 0x7f3fe8198af0 secure :-1 s=READY pgs=75 cs=0 l=1 rev1=1 crypto rx=0x7f3fdc00e9f0 tx=0x7f3fdc00eeb0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T10:33:04.431 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:33:04.429+0000 7f3fec9fe640 1 -- 172.21.15.114:0/1441986308 <== mon.2 v2:172.21.15.119:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3fdc00ccc0 con 0x7f3fe8107080 2024-04-23T10:33:04.432 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:33:04.429+0000 7f3fec9fe640 1 -- 172.21.15.114:0/1441986308 <== mon.2 v2:172.21.15.119:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f3fdc0054d0 con 0x7f3fe8107080 2024-04-23T10:33:04.432 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:33:04.429+0000 7f3fe7fff640 1 -- 172.21.15.114:0/1441986308 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f3fe819dc00 con 0x7f3fe8107080 2024-04-23T10:33:04.432 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:33:04.429+0000 7f3fec9fe640 1 -- 172.21.15.114:0/1441986308 <== mon.2 v2:172.21.15.119:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3fdc010430 con 0x7f3fe8107080 2024-04-23T10:33:04.432 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:33:04.429+0000 7f3fe7fff640 1 -- 172.21.15.114:0/1441986308 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f3fe81a53a0 con 0x7f3fe8107080 2024-04-23T10:33:04.438 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:33:04.432+0000 7f3fec9fe640 1 -- 172.21.15.114:0/1441986308 <== mon.2 v2:172.21.15.119:3300/0 4 ==== mgrmap(e 26) ==== 138778+0+0 (secure 0 0 0) 0x7f3fdc0046c0 con 0x7f3fe8107080 2024-04-23T10:33:04.439 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:33:04.432+0000 7f3ff07ea740 1 Processor -- start 2024-04-23T10:33:04.439 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:33:04.432+0000 7f3ff07ea740 1 -- start start 2024-04-23T10:33:04.439 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:33:04.432+0000 7f3ff07ea740 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x56112cd786e0 0x56112cd78ac0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T10:33:04.439 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:33:04.432+0000 7f3ff07ea740 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x56112cd79000 0x56112cd87890 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T10:33:04.439 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:33:04.432+0000 7f3ff07ea740 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x56112cd87dd0 0x56112cd8a1c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T10:33:04.439 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:33:04.432+0000 7f3ff07ea740 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x56112ccdd970 con 0x56112cd79000 2024-04-23T10:33:04.439 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:33:04.432+0000 7f3ff07ea740 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 -- 0x56112cc7a4a0 con 0x56112cd786e0 2024-04-23T10:33:04.439 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:33:04.432+0000 7f3ff07ea740 1 -- --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_getmap magic: 0 -- 0x56112ccd9930 con 0x56112cd87dd0 2024-04-23T10:33:04.439 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:33:04.433+0000 7f3fe6ffd640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x56112cd79000 0x56112cd87890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T10:33:04.439 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:33:04.433+0000 7f3fe77fe640 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x56112cd786e0 0x56112cd78ac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T10:33:04.439 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:33:04.433+0000 7f3fe6ffd640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x56112cd79000 0x56112cd87890 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.114:52226/0 (socket says 172.21.15.114:52226) 2024-04-23T10:33:04.439 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:33:04.433+0000 7f3fe6ffd640 1 -- 172.21.15.114:0/4293840536 learned_addr learned my addr 172.21.15.114:0/4293840536 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-04-23T10:33:04.439 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:33:04.433+0000 7f3fec9fe640 1 --2- 172.21.15.114:0/1441986308 >> [v2:172.21.15.114:6800/1395693466,v1:172.21.15.114:6801/1395693466] conn(0x7f3fd00a3d10 0x7f3fd00a61d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T10:33:04.440 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:33:04.433+0000 7f3fe67fc640 1 --2- 172.21.15.114:0/4293840536 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x56112cd87dd0 0x56112cd8a1c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T10:33:04.440 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:33:04.433+0000 7f3fe77fe640 1 -- 172.21.15.114:0/4293840536 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x56112cd87dd0 msgr2=0x56112cd8a1c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T10:33:04.440 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:33:04.433+0000 7f3fe77fe640 1 --2- 172.21.15.114:0/4293840536 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x56112cd87dd0 0x56112cd8a1c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:33:04.440 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:33:04.433+0000 7f3fe77fe640 1 -- 172.21.15.114:0/4293840536 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x56112cd79000 msgr2=0x56112cd87890 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T10:33:04.440 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:33:04.433+0000 7f3fe77fe640 1 --2- 172.21.15.114:0/4293840536 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x56112cd79000 0x56112cd87890 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:33:04.440 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:33:04.433+0000 7f3fe77fe640 1 -- 172.21.15.114:0/4293840536 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x56112c9f0210 con 0x56112cd786e0 2024-04-23T10:33:04.440 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:33:04.433+0000 7f3fe67fc640 1 --2- 172.21.15.114:0/4293840536 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x56112cd87dd0 0x56112cd8a1c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-23T10:33:04.440 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:33:04.433+0000 7f3fe77fe640 1 --2- 172.21.15.114:0/1441986308 >> [v2:172.21.15.114:6800/1395693466,v1:172.21.15.114:6801/1395693466] conn(0x7f3fd00a3d10 0x7f3fd00a61d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T10:33:04.440 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:33:04.433+0000 7f3fec9fe640 1 -- 172.21.15.114:0/1441986308 <== mon.2 v2:172.21.15.119:3300/0 5 ==== osd_map(95..95 src has 1..95) ==== 8673+0+0 (secure 0 0 0) 0x7f3fdc0d0bd0 con 0x7f3fe8107080 2024-04-23T10:33:04.440 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:33:04.433+0000 7f3fe77fe640 1 --2- 172.21.15.114:0/4293840536 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x56112cd786e0 0x56112cd78ac0 secure :-1 s=READY pgs=76 cs=0 l=1 rev1=1 crypto rx=0x7f3fe0005e30 tx=0x7f3fe0005fa0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=a38e692026f87835 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T10:33:04.440 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:33:04.433+0000 7f3fcdffb640 1 -- 172.21.15.114:0/4293840536 <== mon.2 v2:172.21.15.119:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3fe0042040 con 0x56112cd786e0 2024-04-23T10:33:04.440 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:33:04.433+0000 7f3fcdffb640 1 -- 172.21.15.114:0/4293840536 <== mon.2 v2:172.21.15.119:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f3fe0036070 con 0x56112cd786e0 2024-04-23T10:33:04.440 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:33:04.433+0000 7f3fe77fe640 1 --2- 172.21.15.114:0/1441986308 >> [v2:172.21.15.114:6800/1395693466,v1:172.21.15.114:6801/1395693466] conn(0x7f3fd00a3d10 0x7f3fd00a61d0 secure :-1 s=READY pgs=113 cs=0 l=1 rev1=1 crypto rx=0x7f3fe0049000 tx=0x7f3fe004e000 comp rx=0 tx=0).ready entity=mgr.15291 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T10:33:04.441 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:33:04.433+0000 7f3fcdffb640 1 -- 172.21.15.114:0/4293840536 <== mon.2 v2:172.21.15.119:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3fe0004600 con 0x56112cd786e0 2024-04-23T10:33:04.441 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:33:04.434+0000 7f3ff07ea740 1 -- 172.21.15.114:0/4293840536 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x56112cd786e0 msgr2=0x56112cd78ac0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T10:33:04.441 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:33:04.434+0000 7f3ff07ea740 1 --2- 172.21.15.114:0/4293840536 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x56112cd786e0 0x56112cd78ac0 secure :-1 s=READY pgs=76 cs=0 l=1 rev1=1 crypto rx=0x7f3fe0005e30 tx=0x7f3fe0005fa0 comp rx=0 tx=0).stop 2024-04-23T10:33:04.441 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:33:04.434+0000 7f3ff07ea740 1 -- 172.21.15.114:0/4293840536 shutdown_connections 2024-04-23T10:33:04.441 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:33:04.434+0000 7f3ff07ea740 1 --2- 172.21.15.114:0/4293840536 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x56112cd87dd0 0x56112cd8a1c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:33:04.441 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:33:04.434+0000 7f3ff07ea740 1 --2- 172.21.15.114:0/4293840536 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x56112cd79000 0x56112cd87890 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:33:04.441 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:33:04.434+0000 7f3ff07ea740 1 --2- 172.21.15.114:0/4293840536 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x56112cd786e0 0x56112cd78ac0 unknown :-1 s=CLOSED pgs=76 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:33:04.441 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:33:04.434+0000 7f3ff07ea740 1 -- 172.21.15.114:0/4293840536 >> 172.21.15.114:0/4293840536 conn(0x56112cd74cf0 msgr2=0x56112cd750d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T10:33:04.441 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:33:04.434+0000 7f3ff07ea740 1 -- 172.21.15.114:0/4293840536 shutdown_connections 2024-04-23T10:33:04.441 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:33:04.434+0000 7f3ff07ea740 1 -- 172.21.15.114:0/4293840536 wait complete. 2024-04-23T10:33:04.441 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:33:04.435+0000 7f3ff07ea740 1 -- start start 2024-04-23T10:33:04.441 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:33:04.435+0000 7f3ff07ea740 1 Processor -- start 2024-04-23T10:33:04.442 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:33:04.435+0000 7f3ff07ea740 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x56112cd786e0 0x56112cd87780 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T10:33:04.442 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:33:04.435+0000 7f3ff07ea740 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x56112cd79000 0x56112cd84a70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T10:33:04.442 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:33:04.435+0000 7f3ff07ea740 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x56112cd87dd0 0x56112cd85010 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T10:33:04.442 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:33:04.435+0000 7f3ff07ea740 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x56112cd8aa00 con 0x56112cd79000 2024-04-23T10:33:04.442 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:33:04.435+0000 7f3ff07ea740 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 -- 0x56112cc780b0 con 0x56112cd786e0 2024-04-23T10:33:04.442 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:33:04.435+0000 7f3ff07ea740 1 -- --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_getmap magic: 0 -- 0x56112cd8ab80 con 0x56112cd87dd0 2024-04-23T10:33:04.442 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:33:04.435+0000 7f3ff07ea740 10 client.0 _ll_register_callbacks cb 0x56112cc65d30 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-23T10:33:04.442 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:33:04.435+0000 7f3fe67fc640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x56112cd79000 0x56112cd84a70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T10:33:04.442 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:33:04.435+0000 7f3fe6ffd640 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x56112cd786e0 0x56112cd87780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T10:33:04.442 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:33:04.435+0000 7f3fe67fc640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x56112cd79000 0x56112cd84a70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.114:52236/0 (socket says 172.21.15.114:52236) 2024-04-23T10:33:04.442 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:33:04.435+0000 7f3fe67fc640 1 -- 172.21.15.114:0/792715499 learned_addr learned my addr 172.21.15.114:0/792715499 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-04-23T10:33:04.442 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:33:04.435+0000 7f3fe77fe640 1 --2- 172.21.15.114:0/792715499 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x56112cd87dd0 0x56112cd85010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T10:33:04.442 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:33:04.436+0000 7f3fe6ffd640 1 -- 172.21.15.114:0/792715499 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x56112cd87dd0 msgr2=0x56112cd85010 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T10:33:04.443 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:33:04.436+0000 7f3fe6ffd640 1 --2- 172.21.15.114:0/792715499 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x56112cd87dd0 0x56112cd85010 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:33:04.443 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:33:04.436+0000 7f3fe6ffd640 1 -- 172.21.15.114:0/792715499 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x56112cd79000 msgr2=0x56112cd84a70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T10:33:04.443 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:33:04.436+0000 7f3fe6ffd640 1 --2- 172.21.15.114:0/792715499 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x56112cd79000 0x56112cd84a70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:33:04.443 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:33:04.436+0000 7f3fe6ffd640 1 -- 172.21.15.114:0/792715499 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({osdmap=0}) -- 0x56112cabc620 con 0x56112cd786e0 2024-04-23T10:33:04.443 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:33:04.436+0000 7f3fe67fc640 1 --2- 172.21.15.114:0/792715499 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x56112cd79000 0x56112cd84a70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-23T10:33:04.443 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:33:04.436+0000 7f3fe6ffd640 1 --2- 172.21.15.114:0/792715499 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x56112cd786e0 0x56112cd87780 secure :-1 s=READY pgs=77 cs=0 l=1 rev1=1 crypto rx=0x56112cd7b3d0 tx=0x7f3fd800f790 comp rx=0 tx=0).ready entity=mon.2 client_cookie=c6f003e453622874 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T10:33:04.443 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:33:04.436+0000 7f3fc6ffd640 10 client.0 ms_handle_connect on v2:172.21.15.119:3300/0 2024-04-23T10:33:04.443 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:33:04.436+0000 7f3fc6ffd640 1 -- 172.21.15.114:0/792715499 <== mon.2 v2:172.21.15.119:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3fd80120d0 con 0x56112cd786e0 2024-04-23T10:33:04.443 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:33:04.436+0000 7f3fc6ffd640 1 -- 172.21.15.114:0/792715499 <== mon.2 v2:172.21.15.119:3300/0 2 ==== osd_map(95..95 src has 1..95) ==== 8673+0+0 (secure 0 0 0) 0x7f3fd8004030 con 0x56112cd786e0 2024-04-23T10:33:04.443 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:33:04.436+0000 7f3ff07ea740 1 -- 172.21.15.114:0/792715499 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f3fe0036070 con 0x56112cd786e0 2024-04-23T10:33:04.443 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:33:04.436+0000 7f3fc6ffd640 1 -- 172.21.15.114:0/792715499 <== mon.2 v2:172.21.15.119:3300/0 3 ==== mon_get_version_reply(handle=1 version=31) ==== 24+0+0 (secure 0 0 0) 0x7f3fd8012270 con 0x56112cd786e0 2024-04-23T10:33:04.443 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:33:04.436+0000 7f3ff07ea740 10 client.25847 fetch_fsmap learned FSMap version 31 2024-04-23T10:33:04.443 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:33:04.436+0000 7f3ff07ea740 1 -- 172.21.15.114:0/792715499 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({config=0+,fsmap=31,monmap=4+}) -- 0x56112cd96840 con 0x56112cd786e0 2024-04-23T10:33:04.443 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:33:04.436+0000 7f3fc6ffd640 1 -- 172.21.15.114:0/792715499 <== mon.2 v2:172.21.15.119:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f3fe0036070 con 0x56112cd786e0 2024-04-23T10:33:04.444 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:33:04.437+0000 7f3fc6ffd640 1 -- 172.21.15.114:0/792715499 <== mon.2 v2:172.21.15.119:3300/0 5 ==== fsmap(e 31) ==== 8342+0+0 (secure 0 0 0) 0x7f3fd801c460 con 0x56112cd786e0 2024-04-23T10:33:04.444 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:33:04.437+0000 7f3ff07ea740 10 client.25847 fetch_fsmap finished waiting for FSMap version 31 2024-04-23T10:33:04.444 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:33:04.437+0000 7f3ff07ea740 10 client.25847 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-04-23T10:33:04.444 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:33:04.437+0000 7f3ff07ea740 20 client.25847 populate_metadata read hostname 'smithi114' 2024-04-23T10:33:04.444 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:33:04.437+0000 7f3ff07ea740 1 --2- 172.21.15.114:0/792715499 >> [v2:172.21.15.119:6838/937297340,v1:172.21.15.119:6839/937297340] conn(0x56112cd97d00 0x56112cd9a0f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T10:33:04.444 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:33:04.437+0000 7f3ff07ea740 4 client.25847 mds_command: new command op to 24545 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-23T10:33:04.444 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:33:04.437+0000 7f3ff07ea740 1 -- 172.21.15.114:0/792715499 --> [v2:172.21.15.119:6838/937297340,v1:172.21.15.119:6839/937297340] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x56112cceb390 con 0x56112cd97d00 2024-04-23T10:33:04.444 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:33:04.437+0000 7f3fe67fc640 1 --2- 172.21.15.114:0/792715499 >> [v2:172.21.15.119:6838/937297340,v1:172.21.15.119:6839/937297340] conn(0x56112cd97d00 0x56112cd9a0f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T10:33:04.444 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:33:04.437+0000 7f3fc6ffd640 10 client.25847 ms_handle_connect on v2:172.21.15.119:6838/937297340 2024-04-23T10:33:04.444 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:33:04.437+0000 7f3fe67fc640 1 --2- 172.21.15.114:0/792715499 >> [v2:172.21.15.119:6838/937297340,v1:172.21.15.119:6839/937297340] conn(0x56112cd97d00 0x56112cd9a0f0 crc :-1 s=READY pgs=118 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=21eeb3c474fde8c2 server_cookie=9e095f077a58bc51 in_seq=0 out_seq=0 2024-04-23T10:33:04.444 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:33:04.438+0000 7f3fc6ffd640 1 -- 172.21.15.114:0/792715499 <== mds.0 v2:172.21.15.119:6838/937297340 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x56112cceb390 con 0x56112cd97d00 2024-04-23T10:33:04.444 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:33:04.438+0000 7f3fc6ffd640 10 client.25847 handle_command_reply: tid=0 2024-04-23T10:33:04.444 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:33:04.438+0000 7f3ff07ea740 1 client.25847 shutdown 2024-04-23T10:33:04.444 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:33:04.438+0000 7f3ff07ea740 1 -- 172.21.15.114:0/792715499 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x56112cd786e0 msgr2=0x56112cd87780 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T10:33:04.445 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:33:04.438+0000 7f3ff07ea740 1 --2- 172.21.15.114:0/792715499 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x56112cd786e0 0x56112cd87780 secure :-1 s=READY pgs=77 cs=0 l=1 rev1=1 crypto rx=0x56112cd7b3d0 tx=0x7f3fd800f790 comp rx=0 tx=0).stop 2024-04-23T10:33:04.445 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:33:04.438+0000 7f3ff07ea740 1 -- 172.21.15.114:0/792715499 shutdown_connections 2024-04-23T10:33:04.445 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:33:04.438+0000 7f3ff07ea740 1 --2- 172.21.15.114:0/792715499 >> [v2:172.21.15.119:6838/937297340,v1:172.21.15.119:6839/937297340] conn(0x56112cd97d00 0x56112cd9a0f0 crc :-1 s=READY pgs=118 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:33:04.445 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:33:04.438+0000 7f3ff07ea740 1 --2- 172.21.15.114:0/792715499 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x56112cd87dd0 0x56112cd85010 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:33:04.445 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:33:04.438+0000 7f3fc6ffd640 0 client.25847 ms_handle_reset on v2:172.21.15.119:6838/937297340 2024-04-23T10:33:04.445 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:33:04.438+0000 7f3ff07ea740 1 --2- 172.21.15.114:0/792715499 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x56112cd79000 0x56112cd84a70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:33:04.445 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:33:04.438+0000 7f3ff07ea740 1 --2- 172.21.15.114:0/792715499 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x56112cd786e0 0x56112cd87780 unknown :-1 s=CLOSED pgs=77 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:33:04.445 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:33:04.438+0000 7f3ff07ea740 1 -- 172.21.15.114:0/792715499 >> 172.21.15.114:0/792715499 conn(0x56112cd74cf0 msgr2=0x56112cd7ab60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T10:33:04.445 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:33:04.438+0000 7f3ff07ea740 1 -- 172.21.15.114:0/792715499 shutdown_connections 2024-04-23T10:33:04.445 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:33:04.438+0000 7f3ff07ea740 1 -- 172.21.15.114:0/792715499 wait complete. 2024-04-23T10:33:04.445 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:33:04.438+0000 7f3ff07ea740 20 client.25847 trim_cache size 0 max 16384 2024-04-23T10:33:04.458 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:33:04.454+0000 7f3ff07ea740 1 Processor -- start 2024-04-23T10:33:04.459 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:33:04.454+0000 7f3ff07ea740 1 -- start start 2024-04-23T10:33:04.459 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:33:04.454+0000 7f3ff07ea740 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x56112cd786e0 0x56112cd938c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T10:33:04.459 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:33:04.454+0000 7f3ff07ea740 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x56112cd79000 0x56112cd7ff40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T10:33:04.459 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:33:04.454+0000 7f3ff07ea740 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x56112cd87dd0 0x56112cd872b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T10:33:04.459 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:33:04.454+0000 7f3ff07ea740 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x7f3fe81ae2c0 con 0x56112cd87dd0 2024-04-23T10:33:04.459 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:33:04.454+0000 7f3ff07ea740 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 -- 0x7f3fe81abb60 con 0x56112cd79000 2024-04-23T10:33:04.459 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:33:04.454+0000 7f3ff07ea740 1 -- --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_getmap magic: 0 -- 0x7f3fe81a99d0 con 0x56112cd786e0 2024-04-23T10:33:04.459 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:33:04.455+0000 7f3fe77fe640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x56112cd87dd0 0x56112cd872b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T10:33:04.459 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:33:04.455+0000 7f3fe77fe640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x56112cd87dd0 0x56112cd872b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.114:52248/0 (socket says 172.21.15.114:52248) 2024-04-23T10:33:04.459 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:33:04.455+0000 7f3fe77fe640 1 -- 172.21.15.114:0/971558177 learned_addr learned my addr 172.21.15.114:0/971558177 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-04-23T10:33:04.460 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:33:04.455+0000 7f3fe6ffd640 1 --2- 172.21.15.114:0/971558177 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x56112cd786e0 0x56112cd938c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T10:33:04.460 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:33:04.455+0000 7f3fe67fc640 1 --2- 172.21.15.114:0/971558177 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x56112cd79000 0x56112cd7ff40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T10:33:04.460 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:33:04.455+0000 7f3fe77fe640 1 -- 172.21.15.114:0/971558177 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x56112cd786e0 msgr2=0x56112cd938c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T10:33:04.460 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:33:04.455+0000 7f3fe77fe640 1 --2- 172.21.15.114:0/971558177 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x56112cd786e0 0x56112cd938c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:33:04.460 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:33:04.455+0000 7f3fe77fe640 1 -- 172.21.15.114:0/971558177 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x56112cd79000 msgr2=0x56112cd7ff40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T10:33:04.460 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:33:04.455+0000 7f3fe77fe640 1 --2- 172.21.15.114:0/971558177 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x56112cd79000 0x56112cd7ff40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:33:04.460 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:33:04.455+0000 7f3fe77fe640 1 -- 172.21.15.114:0/971558177 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x56112cd85380 con 0x56112cd87dd0 2024-04-23T10:33:04.460 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:33:04.455+0000 7f3fe6ffd640 1 --2- 172.21.15.114:0/971558177 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x56112cd786e0 0x56112cd938c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-23T10:33:04.460 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:33:04.455+0000 7f3fe77fe640 1 --2- 172.21.15.114:0/971558177 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x56112cd87dd0 0x56112cd872b0 secure :-1 s=READY pgs=760 cs=0 l=1 rev1=1 crypto rx=0x7f3fe0009430 tx=0x7f3fe0005c30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=96cd3111176e4824 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T10:33:04.460 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:33:04.455+0000 7f3fcd7fa640 1 -- 172.21.15.114:0/971558177 <== mon.0 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3fe0042030 con 0x56112cd87dd0 2024-04-23T10:33:04.460 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:33:04.455+0000 7f3fcd7fa640 1 -- 172.21.15.114:0/971558177 <== mon.0 v2:172.21.15.114:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f3fe0036050 con 0x56112cd87dd0 2024-04-23T10:33:04.460 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:33:04.455+0000 7f3fcd7fa640 1 -- 172.21.15.114:0/971558177 <== mon.0 v2:172.21.15.114:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3fe002f660 con 0x56112cd87dd0 2024-04-23T10:33:04.460 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:33:04.456+0000 7f3ff07ea740 1 -- 172.21.15.114:0/971558177 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x56112cd87dd0 msgr2=0x56112cd872b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T10:33:04.461 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:33:04.456+0000 7f3ff07ea740 1 --2- 172.21.15.114:0/971558177 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x56112cd87dd0 0x56112cd872b0 secure :-1 s=READY pgs=760 cs=0 l=1 rev1=1 crypto rx=0x7f3fe0009430 tx=0x7f3fe0005c30 comp rx=0 tx=0).stop 2024-04-23T10:33:04.461 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:33:04.456+0000 7f3ff07ea740 1 -- 172.21.15.114:0/971558177 shutdown_connections 2024-04-23T10:33:04.461 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:33:04.456+0000 7f3ff07ea740 1 --2- 172.21.15.114:0/971558177 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x56112cd87dd0 0x56112cd872b0 unknown :-1 s=CLOSED pgs=760 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:33:04.461 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:33:04.456+0000 7f3ff07ea740 1 --2- 172.21.15.114:0/971558177 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x56112cd79000 0x56112cd7ff40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:33:04.461 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:33:04.456+0000 7f3ff07ea740 1 --2- 172.21.15.114:0/971558177 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x56112cd786e0 0x56112cd938c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:33:04.461 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:33:04.456+0000 7f3ff07ea740 1 -- 172.21.15.114:0/971558177 >> 172.21.15.114:0/971558177 conn(0x56112cd74cf0 msgr2=0x56112cd7f830 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T10:33:04.461 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:33:04.456+0000 7f3ff07ea740 1 -- 172.21.15.114:0/971558177 shutdown_connections 2024-04-23T10:33:04.461 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:33:04.456+0000 7f3ff07ea740 1 -- 172.21.15.114:0/971558177 wait complete. 2024-04-23T10:33:04.461 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:33:04.456+0000 7f3ff07ea740 1 -- start start 2024-04-23T10:33:04.461 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:33:04.457+0000 7f3ff07ea740 1 Processor -- start 2024-04-23T10:33:04.461 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:33:04.457+0000 7f3ff07ea740 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x56112cd786e0 0x56112cd8fdd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T10:33:04.461 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:33:04.457+0000 7f3ff07ea740 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x56112cd79000 0x56112cd90310 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T10:33:04.461 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:33:04.457+0000 7f3ff07ea740 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x56112cd87dd0 0x56112cd810f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T10:33:04.462 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:33:04.457+0000 7f3ff07ea740 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x7f3fe81aa020 con 0x56112cd786e0 2024-04-23T10:33:04.462 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:33:04.457+0000 7f3ff07ea740 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 -- 0x7f3fe81ab490 con 0x56112cd79000 2024-04-23T10:33:04.462 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:33:04.457+0000 7f3ff07ea740 1 -- --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_getmap magic: 0 -- 0x7f3fe81aa6f0 con 0x56112cd87dd0 2024-04-23T10:33:04.462 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:33:04.457+0000 7f3ff07ea740 10 client.0 _ll_register_callbacks cb 0x7f3fd8012930 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-23T10:33:04.462 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:33:04.457+0000 7f3fe6ffd640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x56112cd786e0 0x56112cd8fdd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T10:33:04.462 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:33:04.457+0000 7f3fe6ffd640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x56112cd786e0 0x56112cd8fdd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.114:52262/0 (socket says 172.21.15.114:52262) 2024-04-23T10:33:04.462 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:33:04.457+0000 7f3fe6ffd640 1 -- 172.21.15.114:0/1256873895 learned_addr learned my addr 172.21.15.114:0/1256873895 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-04-23T10:33:04.462 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:33:04.457+0000 7f3fe67fc640 1 --2- 172.21.15.114:0/1256873895 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x56112cd79000 0x56112cd90310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T10:33:04.462 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:33:04.457+0000 7f3fe77fe640 1 --2- 172.21.15.114:0/1256873895 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x56112cd87dd0 0x56112cd810f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T10:33:04.462 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:33:04.457+0000 7f3fe6ffd640 1 -- 172.21.15.114:0/1256873895 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x56112cd87dd0 msgr2=0x56112cd810f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T10:33:04.462 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:33:04.457+0000 7f3fe6ffd640 1 --2- 172.21.15.114:0/1256873895 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x56112cd87dd0 0x56112cd810f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:33:04.462 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:33:04.457+0000 7f3fe6ffd640 1 -- 172.21.15.114:0/1256873895 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x56112cd79000 msgr2=0x56112cd90310 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T10:33:04.462 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:33:04.457+0000 7f3fe6ffd640 1 --2- 172.21.15.114:0/1256873895 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x56112cd79000 0x56112cd90310 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:33:04.463 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:33:04.457+0000 7f3fe6ffd640 1 -- 172.21.15.114:0/1256873895 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({osdmap=0}) -- 0x56112cd78e30 con 0x56112cd786e0 2024-04-23T10:33:04.463 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:33:04.457+0000 7f3fe77fe640 1 --2- 172.21.15.114:0/1256873895 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x56112cd87dd0 0x56112cd810f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-23T10:33:04.463 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:33:04.457+0000 7f3fe6ffd640 1 --2- 172.21.15.114:0/1256873895 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x56112cd786e0 0x56112cd8fdd0 secure :-1 s=READY pgs=761 cs=0 l=1 rev1=1 crypto rx=0x56112cd7b3d0 tx=0x7f3fd80128a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a4d704b32d91ab04 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T10:33:04.463 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:33:04.457+0000 7f3fc6ffd640 10 client.0 ms_handle_connect on v2:172.21.15.114:3300/0 2024-04-23T10:33:04.463 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:33:04.457+0000 7f3fc6ffd640 1 -- 172.21.15.114:0/1256873895 <== mon.0 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3fd8000e60 con 0x56112cd786e0 2024-04-23T10:33:04.463 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:33:04.457+0000 7f3fc6ffd640 1 -- 172.21.15.114:0/1256873895 <== mon.0 v2:172.21.15.114:3300/0 2 ==== osd_map(95..95 src has 1..95) ==== 8673+0+0 (secure 0 0 0) 0x7f3fd80052d0 con 0x56112cd786e0 2024-04-23T10:33:04.463 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:33:04.457+0000 7f3ff07ea740 1 -- 172.21.15.114:0/1256873895 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f3fe0036050 con 0x56112cd786e0 2024-04-23T10:33:04.463 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:33:04.458+0000 7f3fc6ffd640 1 -- 172.21.15.114:0/1256873895 <== mon.0 v2:172.21.15.114:3300/0 3 ==== mon_get_version_reply(handle=1 version=31) ==== 24+0+0 (secure 0 0 0) 0x7f3fd80054d0 con 0x56112cd786e0 2024-04-23T10:33:04.463 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:33:04.458+0000 7f3ff07ea740 10 client.16482 fetch_fsmap learned FSMap version 31 2024-04-23T10:33:04.463 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:33:04.458+0000 7f3ff07ea740 1 -- 172.21.15.114:0/1256873895 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({config=0+,fsmap=31,monmap=4+}) -- 0x56112cdfdb80 con 0x56112cd786e0 2024-04-23T10:33:04.463 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:33:04.458+0000 7f3fc6ffd640 1 -- 172.21.15.114:0/1256873895 <== mon.0 v2:172.21.15.114:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f3fe0036050 con 0x56112cd786e0 2024-04-23T10:33:04.463 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:33:04.458+0000 7f3fc6ffd640 1 -- 172.21.15.114:0/1256873895 <== mon.0 v2:172.21.15.114:3300/0 5 ==== fsmap(e 31) ==== 8342+0+0 (secure 0 0 0) 0x7f3fd802e4e0 con 0x56112cd786e0 2024-04-23T10:33:04.463 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:33:04.458+0000 7f3ff07ea740 10 client.16482 fetch_fsmap finished waiting for FSMap version 31 2024-04-23T10:33:04.463 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:33:04.458+0000 7f3ff07ea740 10 client.16482 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-04-23T10:33:04.463 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:33:04.458+0000 7f3ff07ea740 20 client.16482 populate_metadata read hostname 'smithi114' 2024-04-23T10:33:04.464 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:33:04.458+0000 7f3ff07ea740 1 --2- 172.21.15.114:0/1256873895 >> [v2:172.21.15.119:6838/937297340,v1:172.21.15.119:6839/937297340] conn(0x56112cdfe2d0 0x56112ce006f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T10:33:04.464 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:33:04.458+0000 7f3ff07ea740 4 client.16482 mds_command: new command op to 24545 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-04-23T10:33:04.464 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:33:04.458+0000 7f3ff07ea740 1 -- 172.21.15.114:0/1256873895 --> [v2:172.21.15.119:6838/937297340,v1:172.21.15.119:6839/937297340] -- command(tid 0: {"prefix": "flush journal"}) -- 0x56112cce5200 con 0x56112cdfe2d0 2024-04-23T10:33:04.464 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:33:04.458+0000 7f3fe67fc640 1 --2- 172.21.15.114:0/1256873895 >> [v2:172.21.15.119:6838/937297340,v1:172.21.15.119:6839/937297340] conn(0x56112cdfe2d0 0x56112ce006f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T10:33:04.464 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:33:04.459+0000 7f3fe67fc640 1 --2- 172.21.15.114:0/1256873895 >> [v2:172.21.15.119:6838/937297340,v1:172.21.15.119:6839/937297340] conn(0x56112cdfe2d0 0x56112ce006f0 crc :-1 s=READY pgs=119 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=9d86100322f4d8a3 server_cookie=8a13c2660632bca9 in_seq=0 out_seq=0 2024-04-23T10:33:04.464 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:33:04.459+0000 7f3fc6ffd640 10 client.16482 ms_handle_connect on v2:172.21.15.119:6838/937297340 2024-04-23T10:33:04.519 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:33:04.519+0000 7f3fc6ffd640 1 -- 172.21.15.114:0/1256873895 <== mds.0 v2:172.21.15.119:6838/937297340 1 ==== command_reply(tid 0: 0 ) ==== 8+0+44 (crc 0 0 0) 0x56112cce5200 con 0x56112cdfe2d0 2024-04-23T10:33:04.519 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:33:04.519+0000 7f3fc6ffd640 10 client.16482 handle_command_reply: tid=0 2024-04-23T10:33:04.519 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:33:04.519+0000 7f3ff07ea740 1 client.16482 shutdown 2024-04-23T10:33:04.519 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:33:04.519+0000 7f3ff07ea740 1 -- 172.21.15.114:0/1256873895 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x56112cd786e0 msgr2=0x56112cd8fdd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T10:33:04.519 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:33:04.519+0000 7f3ff07ea740 1 --2- 172.21.15.114:0/1256873895 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x56112cd786e0 0x56112cd8fdd0 secure :-1 s=READY pgs=761 cs=0 l=1 rev1=1 crypto rx=0x56112cd7b3d0 tx=0x7f3fd80128a0 comp rx=0 tx=0).stop 2024-04-23T10:33:04.519 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:33:04.519+0000 7f3ff07ea740 1 -- 172.21.15.114:0/1256873895 shutdown_connections 2024-04-23T10:33:04.519 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:33:04.519+0000 7f3ff07ea740 1 --2- 172.21.15.114:0/1256873895 >> [v2:172.21.15.119:6838/937297340,v1:172.21.15.119:6839/937297340] conn(0x56112cdfe2d0 0x56112ce006f0 crc :-1 s=READY pgs=119 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:33:04.519 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:33:04.519+0000 7f3ff07ea740 1 --2- 172.21.15.114:0/1256873895 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x56112cd87dd0 0x56112cd810f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:33:04.519 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:33:04.519+0000 7f3ff07ea740 1 --2- 172.21.15.114:0/1256873895 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x56112cd79000 0x56112cd90310 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:33:04.519 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:33:04.519+0000 7f3ff07ea740 1 --2- 172.21.15.114:0/1256873895 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x56112cd786e0 0x56112cd8fdd0 unknown :-1 s=CLOSED pgs=761 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:33:04.520 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:33:04.519+0000 7f3ff07ea740 1 -- 172.21.15.114:0/1256873895 >> 172.21.15.114:0/1256873895 conn(0x56112cd74cf0 msgr2=0x56112cd800c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T10:33:04.520 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:33:04.519+0000 7f3fc6ffd640 0 client.16482 ms_handle_reset on v2:172.21.15.119:6838/937297340 2024-04-23T10:33:04.520 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:33:04.519+0000 7f3ff07ea740 1 -- 172.21.15.114:0/1256873895 shutdown_connections 2024-04-23T10:33:04.520 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:33:04.519+0000 7f3ff07ea740 1 -- 172.21.15.114:0/1256873895 wait complete. 2024-04-23T10:33:04.520 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:33:04.519+0000 7f3ff07ea740 20 client.16482 trim_cache size 0 max 16384 2024-04-23T10:33:04.520 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stdout:{ 2024-04-23T10:33:04.520 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stdout: "message": "", 2024-04-23T10:33:04.520 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stdout: "return_code": 0 2024-04-23T10:33:04.520 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stdout:} 2024-04-23T10:33:04.521 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:33:04.520+0000 7f3fe7fff640 1 -- 172.21.15.114:0/1441986308 >> [v2:172.21.15.114:6800/1395693466,v1:172.21.15.114:6801/1395693466] conn(0x7f3fd00a3d10 msgr2=0x7f3fd00a61d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T10:33:04.521 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:33:04.520+0000 7f3fe7fff640 1 --2- 172.21.15.114:0/1441986308 >> [v2:172.21.15.114:6800/1395693466,v1:172.21.15.114:6801/1395693466] conn(0x7f3fd00a3d10 0x7f3fd00a61d0 secure :-1 s=READY pgs=113 cs=0 l=1 rev1=1 crypto rx=0x7f3fe0049000 tx=0x7f3fe004e000 comp rx=0 tx=0).stop 2024-04-23T10:33:04.521 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:33:04.520+0000 7f3fe7fff640 1 -- 172.21.15.114:0/1441986308 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f3fe8107080 msgr2=0x7f3fe8198af0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T10:33:04.521 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:33:04.520+0000 7f3fe7fff640 1 --2- 172.21.15.114:0/1441986308 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f3fe8107080 0x7f3fe8198af0 secure :-1 s=READY pgs=75 cs=0 l=1 rev1=1 crypto rx=0x7f3fdc00e9f0 tx=0x7f3fdc00eeb0 comp rx=0 tx=0).stop 2024-04-23T10:33:04.522 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:33:04.520+0000 7f3fe7fff640 1 -- 172.21.15.114:0/1441986308 shutdown_connections 2024-04-23T10:33:04.522 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:33:04.520+0000 7f3fe7fff640 1 --2- 172.21.15.114:0/1441986308 >> [v2:172.21.15.114:6800/1395693466,v1:172.21.15.114:6801/1395693466] conn(0x7f3fd00a3d10 0x7f3fd00a61d0 unknown :-1 s=CLOSED pgs=113 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:33:04.522 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:33:04.520+0000 7f3fe7fff640 1 --2- 172.21.15.114:0/1441986308 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f3fe819ce20 0x7f3fe819d2d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:33:04.522 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:33:04.520+0000 7f3fe7fff640 1 --2- 172.21.15.114:0/1441986308 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f3fe8107080 0x7f3fe8198af0 unknown :-1 s=CLOSED pgs=75 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:33:04.522 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:33:04.520+0000 7f3fe7fff640 1 --2- 172.21.15.114:0/1441986308 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f3fe81004c0 0x7f3fe81985b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:33:04.522 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:33:04.520+0000 7f3fe7fff640 1 -- 172.21.15.114:0/1441986308 >> 172.21.15.114:0/1441986308 conn(0x7f3fe80771d0 msgr2=0x7f3fe80fe560 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T10:33:04.522 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:33:04.520+0000 7f3fe7fff640 1 -- 172.21.15.114:0/1441986308 shutdown_connections 2024-04-23T10:33:04.522 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:33:04.520+0000 7f3fe7fff640 1 -- 172.21.15.114:0/1441986308 wait complete. 2024-04-23T10:33:04.741 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 23 10:33:04 smithi119 ceph-mon[32608]: osdmap e95: 12 total, 12 up, 12 in 2024-04-23T10:33:04.754 INFO:journalctl@ceph.mon.a.smithi114.stdout:Apr 23 10:33:04 smithi114 ceph-mon[29255]: osdmap e95: 12 total, 12 up, 12 in 2024-04-23T10:33:04.827 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 23 10:33:04 smithi204 ceph-mon[32649]: osdmap e95: 12 total, 12 up, 12 in 2024-04-23T10:33:04.929 INFO:tasks.workunit.client.0.smithi114.stdout: 1 386115 5.81 MB/sec execute 193 sec latency 41.257 ms 2024-04-23T10:33:05.742 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 23 10:33:05 smithi119 ceph-mon[32608]: pgmap v200: 97 pgs: 97 active+clean; 114 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 3.2 MiB/s rd, 2.9 MiB/s wr, 303 op/s 2024-04-23T10:33:05.753 INFO:journalctl@ceph.mon.a.smithi114.stdout:Apr 23 10:33:05 smithi114 ceph-mon[29255]: pgmap v200: 97 pgs: 97 active+clean; 114 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 3.2 MiB/s rd, 2.9 MiB/s wr, 303 op/s 2024-04-23T10:33:05.827 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 23 10:33:05 smithi204 ceph-mon[32649]: pgmap v200: 97 pgs: 97 active+clean; 114 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 3.2 MiB/s rd, 2.9 MiB/s wr, 303 op/s 2024-04-23T10:33:05.926 INFO:tasks.workunit.client.0.smithi114.stdout: 1 387600 5.82 MB/sec execute 194 sec latency 18.046 ms 2024-04-23T10:33:06.926 INFO:tasks.workunit.client.0.smithi114.stdout: 1 390292 5.87 MB/sec execute 195 sec latency 16.211 ms 2024-04-23T10:33:07.741 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 23 10:33:07 smithi119 ceph-mon[32608]: pgmap v201: 97 pgs: 97 active+clean; 110 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 3.2 MiB/s rd, 3.7 MiB/s wr, 355 op/s 2024-04-23T10:33:07.754 INFO:journalctl@ceph.mon.a.smithi114.stdout:Apr 23 10:33:07 smithi114 ceph-mon[29255]: pgmap v201: 97 pgs: 97 active+clean; 110 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 3.2 MiB/s rd, 3.7 MiB/s wr, 355 op/s 2024-04-23T10:33:07.827 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 23 10:33:07 smithi204 ceph-mon[32649]: pgmap v201: 97 pgs: 97 active+clean; 110 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 3.2 MiB/s rd, 3.7 MiB/s wr, 355 op/s 2024-04-23T10:33:07.926 INFO:tasks.workunit.client.0.smithi114.stdout: 1 391327 5.86 MB/sec execute 196 sec latency 29.581 ms 2024-04-23T10:33:08.926 INFO:tasks.workunit.client.0.smithi114.stdout: 1 393922 5.91 MB/sec execute 197 sec latency 28.183 ms 2024-04-23T10:33:09.742 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 23 10:33:09 smithi119 ceph-mon[32608]: pgmap v202: 97 pgs: 97 active+clean; 110 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 3.2 MiB/s rd, 3.7 MiB/s wr, 355 op/s 2024-04-23T10:33:09.742 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 23 10:33:09 smithi119 ceph-mon[32608]: osdmap e96: 12 total, 12 up, 12 in 2024-04-23T10:33:09.754 INFO:journalctl@ceph.mon.a.smithi114.stdout:Apr 23 10:33:09 smithi114 ceph-mon[29255]: pgmap v202: 97 pgs: 97 active+clean; 110 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 3.2 MiB/s rd, 3.7 MiB/s wr, 355 op/s 2024-04-23T10:33:09.754 INFO:journalctl@ceph.mon.a.smithi114.stdout:Apr 23 10:33:09 smithi114 ceph-mon[29255]: osdmap e96: 12 total, 12 up, 12 in 2024-04-23T10:33:09.827 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 23 10:33:09 smithi204 ceph-mon[32649]: pgmap v202: 97 pgs: 97 active+clean; 110 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 3.2 MiB/s rd, 3.7 MiB/s wr, 355 op/s 2024-04-23T10:33:09.827 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 23 10:33:09 smithi204 ceph-mon[32649]: osdmap e96: 12 total, 12 up, 12 in 2024-04-23T10:33:09.926 INFO:tasks.workunit.client.0.smithi114.stdout: 1 395273 5.91 MB/sec execute 198 sec latency 19.738 ms 2024-04-23T10:33:10.926 INFO:tasks.workunit.client.0.smithi114.stdout: 1 397524 5.95 MB/sec execute 199 sec latency 13.874 ms 2024-04-23T10:33:11.742 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 23 10:33:11 smithi119 ceph-mon[32608]: pgmap v204: 97 pgs: 97 active+clean; 110 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 MiB/s rd, 2.3 MiB/s wr, 225 op/s 2024-04-23T10:33:11.754 INFO:journalctl@ceph.mon.a.smithi114.stdout:Apr 23 10:33:11 smithi114 ceph-mon[29255]: pgmap v204: 97 pgs: 97 active+clean; 110 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 MiB/s rd, 2.3 MiB/s wr, 225 op/s 2024-04-23T10:33:11.827 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 23 10:33:11 smithi204 ceph-mon[32649]: pgmap v204: 97 pgs: 97 active+clean; 110 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 MiB/s rd, 2.3 MiB/s wr, 225 op/s 2024-04-23T10:33:11.926 INFO:tasks.workunit.client.0.smithi114.stdout: 1 398958 5.95 MB/sec execute 200 sec latency 15.399 ms 2024-04-23T10:33:12.926 INFO:tasks.workunit.client.0.smithi114.stdout: 1 401249 5.99 MB/sec execute 201 sec latency 16.638 ms 2024-04-23T10:33:13.742 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 23 10:33:13 smithi119 ceph-mon[32608]: pgmap v205: 97 pgs: 97 active+clean; 116 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 MiB/s rd, 4.0 MiB/s wr, 351 op/s 2024-04-23T10:33:13.754 INFO:journalctl@ceph.mon.a.smithi114.stdout:Apr 23 10:33:13 smithi114 ceph-mon[29255]: pgmap v205: 97 pgs: 97 active+clean; 116 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 MiB/s rd, 4.0 MiB/s wr, 351 op/s 2024-04-23T10:33:13.828 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 23 10:33:13 smithi204 ceph-mon[32649]: pgmap v205: 97 pgs: 97 active+clean; 116 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 MiB/s rd, 4.0 MiB/s wr, 351 op/s 2024-04-23T10:33:13.926 INFO:tasks.workunit.client.0.smithi114.stdout: 1 402838 5.99 MB/sec execute 202 sec latency 35.796 ms 2024-04-23T10:33:14.926 INFO:tasks.workunit.client.0.smithi114.stdout: 1 404947 6.03 MB/sec execute 203 sec latency 11.331 ms 2024-04-23T10:33:15.742 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 23 10:33:15 smithi119 ceph-mon[32608]: pgmap v206: 97 pgs: 97 active+clean; 116 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 MiB/s rd, 3.6 MiB/s wr, 314 op/s 2024-04-23T10:33:15.753 INFO:journalctl@ceph.mon.a.smithi114.stdout:Apr 23 10:33:15 smithi114 ceph-mon[29255]: pgmap v206: 97 pgs: 97 active+clean; 116 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 MiB/s rd, 3.6 MiB/s wr, 314 op/s 2024-04-23T10:33:15.827 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 23 10:33:15 smithi204 ceph-mon[32649]: pgmap v206: 97 pgs: 97 active+clean; 116 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 MiB/s rd, 3.6 MiB/s wr, 314 op/s 2024-04-23T10:33:15.926 INFO:tasks.workunit.client.0.smithi114.stdout: 1 406435 6.02 MB/sec execute 204 sec latency 20.542 ms 2024-04-23T10:33:16.926 INFO:tasks.workunit.client.0.smithi114.stdout: 1 408360 6.06 MB/sec execute 205 sec latency 67.114 ms 2024-04-23T10:33:17.621 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:33:17.609+0000 7ff1dfa43640 1 -- 172.21.15.114:0/4079068800 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7ff1d8102b70 msgr2=0x7ff1d8106bb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T10:33:17.621 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:33:17.609+0000 7ff1dfa43640 1 --2- 172.21.15.114:0/4079068800 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7ff1d8102b70 0x7ff1d8106bb0 secure :-1 s=READY pgs=762 cs=0 l=1 rev1=1 crypto rx=0x7ff1c8009900 tx=0x7ff1c802a410 comp rx=0 tx=0).stop 2024-04-23T10:33:17.622 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:33:17.609+0000 7ff1dfa43640 1 -- 172.21.15.114:0/4079068800 shutdown_connections 2024-04-23T10:33:17.622 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:33:17.609+0000 7ff1dfa43640 1 --2- 172.21.15.114:0/4079068800 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7ff1d8107290 0x7ff1d810ee60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:33:17.622 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:33:17.609+0000 7ff1dfa43640 1 --2- 172.21.15.114:0/4079068800 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7ff1d8102b70 0x7ff1d8106bb0 unknown :-1 s=CLOSED pgs=762 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:33:17.622 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:33:17.609+0000 7ff1dfa43640 1 --2- 172.21.15.114:0/4079068800 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7ff1d81021c0 0x7ff1d81025a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:33:17.622 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:33:17.609+0000 7ff1dfa43640 1 -- 172.21.15.114:0/4079068800 >> 172.21.15.114:0/4079068800 conn(0x7ff1d80fda40 msgr2=0x7ff1d80ffe60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T10:33:17.622 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:33:17.609+0000 7ff1dfa43640 1 -- 172.21.15.114:0/4079068800 shutdown_connections 2024-04-23T10:33:17.622 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:33:17.609+0000 7ff1dfa43640 1 -- 172.21.15.114:0/4079068800 wait complete. 2024-04-23T10:33:17.622 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:33:17.610+0000 7ff1dfa43640 1 Processor -- start 2024-04-23T10:33:17.622 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:33:17.610+0000 7ff1dfa43640 1 -- start start 2024-04-23T10:33:17.623 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:33:17.610+0000 7ff1dfa43640 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7ff1d81021c0 0x7ff1d8198710 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T10:33:17.623 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:33:17.610+0000 7ff1dfa43640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7ff1d8107290 0x7ff1d8198c50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T10:33:17.623 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:33:17.610+0000 7ff1dfa43640 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7ff1d819cfd0 0x7ff1d819d480 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T10:33:17.623 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:33:17.610+0000 7ff1dfa43640 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x7ff1d81121b0 con 0x7ff1d8107290 2024-04-23T10:33:17.623 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:33:17.610+0000 7ff1dfa43640 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 -- 0x7ff1d8112030 con 0x7ff1d819cfd0 2024-04-23T10:33:17.623 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:33:17.610+0000 7ff1dfa43640 1 -- --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_getmap magic: 0 -- 0x7ff1d8112330 con 0x7ff1d81021c0 2024-04-23T10:33:17.623 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:33:17.610+0000 7ff1dcfde640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7ff1d8107290 0x7ff1d8198c50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T10:33:17.623 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:33:17.610+0000 7ff1dcfde640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7ff1d8107290 0x7ff1d8198c50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.114:50782/0 (socket says 172.21.15.114:50782) 2024-04-23T10:33:17.623 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:33:17.610+0000 7ff1dcfde640 1 -- 172.21.15.114:0/1219875962 learned_addr learned my addr 172.21.15.114:0/1219875962 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-04-23T10:33:17.624 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:33:17.610+0000 7ff1dd7df640 1 --2- 172.21.15.114:0/1219875962 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7ff1d81021c0 0x7ff1d8198710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T10:33:17.624 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:33:17.610+0000 7ff1ddfe0640 1 --2- 172.21.15.114:0/1219875962 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7ff1d819cfd0 0x7ff1d819d480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T10:33:17.624 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:33:17.611+0000 7ff1dcfde640 1 -- 172.21.15.114:0/1219875962 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7ff1d81021c0 msgr2=0x7ff1d8198710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T10:33:17.624 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:33:17.611+0000 7ff1dcfde640 1 --2- 172.21.15.114:0/1219875962 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7ff1d81021c0 0x7ff1d8198710 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:33:17.624 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:33:17.611+0000 7ff1dcfde640 1 -- 172.21.15.114:0/1219875962 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7ff1d819cfd0 msgr2=0x7ff1d819d480 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T10:33:17.624 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:33:17.611+0000 7ff1dcfde640 1 --2- 172.21.15.114:0/1219875962 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7ff1d819cfd0 0x7ff1d819d480 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:33:17.624 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:33:17.611+0000 7ff1dcfde640 1 -- 172.21.15.114:0/1219875962 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff1d819dac0 con 0x7ff1d8107290 2024-04-23T10:33:17.625 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:33:17.611+0000 7ff1ddfe0640 1 --2- 172.21.15.114:0/1219875962 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7ff1d819cfd0 0x7ff1d819d480 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-23T10:33:17.625 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:33:17.611+0000 7ff1dcfde640 1 --2- 172.21.15.114:0/1219875962 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7ff1d8107290 0x7ff1d8198c50 secure :-1 s=READY pgs=763 cs=0 l=1 rev1=1 crypto rx=0x7ff1c8006010 tx=0x7ff1c8004060 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T10:33:17.625 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:33:17.611+0000 7ff1ce7fc640 1 -- 172.21.15.114:0/1219875962 <== mon.0 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff1c803f030 con 0x7ff1d8107290 2024-04-23T10:33:17.625 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:33:17.611+0000 7ff1ce7fc640 1 -- 172.21.15.114:0/1219875962 <== mon.0 v2:172.21.15.114:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7ff1c8003aa0 con 0x7ff1d8107290 2024-04-23T10:33:17.625 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:33:17.611+0000 7ff1dfa43640 1 -- 172.21.15.114:0/1219875962 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7ff1d819dd50 con 0x7ff1d8107290 2024-04-23T10:33:17.625 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:33:17.611+0000 7ff1ce7fc640 1 -- 172.21.15.114:0/1219875962 <== mon.0 v2:172.21.15.114:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff1c8035070 con 0x7ff1d8107290 2024-04-23T10:33:17.625 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:33:17.611+0000 7ff1dfa43640 1 -- 172.21.15.114:0/1219875962 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7ff1d81a5510 con 0x7ff1d8107290 2024-04-23T10:33:17.625 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:33:17.613+0000 7ff1ce7fc640 1 -- 172.21.15.114:0/1219875962 <== mon.0 v2:172.21.15.114:3300/0 4 ==== mgrmap(e 26) ==== 138778+0+0 (secure 0 0 0) 0x7ff1c8042030 con 0x7ff1d8107290 2024-04-23T10:33:17.625 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:33:17.614+0000 7ff1e2836740 1 Processor -- start 2024-04-23T10:33:17.625 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:33:17.614+0000 7ff1e2836740 1 -- start start 2024-04-23T10:33:17.626 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:33:17.614+0000 7ff1ce7fc640 1 --2- 172.21.15.114:0/1219875962 >> [v2:172.21.15.114:6800/1395693466,v1:172.21.15.114:6801/1395693466] conn(0x7ff1c40a3dc0 0x7ff1c40a6280 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T10:33:17.626 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:33:17.614+0000 7ff1e2836740 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x55591fa76620 0x55591fa76a00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T10:33:17.626 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:33:17.614+0000 7ff1e2836740 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55591fa76f40 0x55591fa812c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T10:33:17.626 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:33:17.614+0000 7ff1e2836740 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x55591fa81800 0x55591fa83bf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T10:33:17.626 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:33:17.614+0000 7ff1e2836740 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x55591f9db970 con 0x55591fa76f40 2024-04-23T10:33:17.626 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:33:17.614+0000 7ff1e2836740 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 -- 0x55591f9784a0 con 0x55591fa81800 2024-04-23T10:33:17.626 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:33:17.614+0000 7ff1e2836740 1 -- --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_getmap magic: 0 -- 0x55591f9d7930 con 0x55591fa76620 2024-04-23T10:33:17.626 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:33:17.614+0000 7ff1ddfe0640 1 --2- 172.21.15.114:0/1219875962 >> [v2:172.21.15.114:6800/1395693466,v1:172.21.15.114:6801/1395693466] conn(0x7ff1c40a3dc0 0x7ff1c40a6280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T10:33:17.626 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:33:17.614+0000 7ff1dcfde640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55591fa76f40 0x55591fa812c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T10:33:17.626 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:33:17.614+0000 7ff1dd7df640 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x55591fa76620 0x55591fa76a00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T10:33:17.627 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:33:17.614+0000 7ff1ce7fc640 1 -- 172.21.15.114:0/1219875962 <== mon.0 v2:172.21.15.114:3300/0 5 ==== osd_map(96..96 src has 1..96) ==== 8629+0+0 (secure 0 0 0) 0x7ff1c80f0c10 con 0x7ff1d8107290 2024-04-23T10:33:17.627 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:33:17.614+0000 7ff1dd7df640 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x55591fa76620 0x55591fa76a00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.204:3300/0 says I am v2:172.21.15.114:51292/0 (socket says 172.21.15.114:51292) 2024-04-23T10:33:17.627 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:33:17.614+0000 7ff1dd7df640 1 -- 172.21.15.114:0/3310973517 learned_addr learned my addr 172.21.15.114:0/3310973517 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-04-23T10:33:17.627 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:33:17.614+0000 7ff1dcfde640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55591fa76f40 0x55591fa812c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.114:50796/0 (socket says 172.21.15.114:50796) 2024-04-23T10:33:17.627 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:33:17.614+0000 7ff1ddfe0640 1 --2- 172.21.15.114:0/3310973517 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x55591fa81800 0x55591fa83bf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T10:33:17.627 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:33:17.614+0000 7ff1dd7df640 1 -- 172.21.15.114:0/3310973517 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x55591fa81800 msgr2=0x55591fa83bf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T10:33:17.627 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:33:17.614+0000 7ff1dd7df640 1 --2- 172.21.15.114:0/3310973517 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x55591fa81800 0x55591fa83bf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:33:17.627 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:33:17.614+0000 7ff1dd7df640 1 -- 172.21.15.114:0/3310973517 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55591fa76f40 msgr2=0x55591fa812c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T10:33:17.627 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:33:17.614+0000 7ff1dd7df640 1 --2- 172.21.15.114:0/3310973517 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55591fa76f40 0x55591fa812c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:33:17.627 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:33:17.614+0000 7ff1dd7df640 1 -- 172.21.15.114:0/3310973517 --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55591f6ee210 con 0x55591fa76620 2024-04-23T10:33:17.628 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:33:17.614+0000 7ff1ddfe0640 1 --2- 172.21.15.114:0/3310973517 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x55591fa81800 0x55591fa83bf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-23T10:33:17.628 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:33:17.614+0000 7ff1ddfe0640 1 --2- 172.21.15.114:0/1219875962 >> [v2:172.21.15.114:6800/1395693466,v1:172.21.15.114:6801/1395693466] conn(0x7ff1c40a3dc0 0x7ff1c40a6280 secure :-1 s=READY pgs=114 cs=0 l=1 rev1=1 crypto rx=0x55591fa73f50 tx=0x7ff1d0009e70 comp rx=0 tx=0).ready entity=mgr.15291 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T10:33:17.628 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:33:17.615+0000 7ff1dd7df640 1 --2- 172.21.15.114:0/3310973517 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x55591fa76620 0x55591fa76a00 secure :-1 s=READY pgs=75 cs=0 l=1 rev1=1 crypto rx=0x7ff1d4009cf0 tx=0x7ff1d4011810 comp rx=0 tx=0).ready entity=mon.1 client_cookie=97ddc84be39d314a server_cookie=0 in_seq=0 out_seq=0 2024-04-23T10:33:17.628 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:33:17.615+0000 7ff1c37fe640 1 -- 172.21.15.114:0/3310973517 <== mon.1 v2:172.21.15.204:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff1d4004230 con 0x55591fa76620 2024-04-23T10:33:17.628 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:33:17.615+0000 7ff1c37fe640 1 -- 172.21.15.114:0/3310973517 <== mon.1 v2:172.21.15.204:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7ff1d40043d0 con 0x55591fa76620 2024-04-23T10:33:17.628 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:33:17.615+0000 7ff1c37fe640 1 -- 172.21.15.114:0/3310973517 <== mon.1 v2:172.21.15.204:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff1d4019880 con 0x55591fa76620 2024-04-23T10:33:17.628 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:33:17.615+0000 7ff1e2836740 1 -- 172.21.15.114:0/3310973517 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x55591fa76620 msgr2=0x55591fa76a00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T10:33:17.628 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:33:17.615+0000 7ff1e2836740 1 --2- 172.21.15.114:0/3310973517 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x55591fa76620 0x55591fa76a00 secure :-1 s=READY pgs=75 cs=0 l=1 rev1=1 crypto rx=0x7ff1d4009cf0 tx=0x7ff1d4011810 comp rx=0 tx=0).stop 2024-04-23T10:33:17.628 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:33:17.615+0000 7ff1e2836740 1 -- 172.21.15.114:0/3310973517 shutdown_connections 2024-04-23T10:33:17.628 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:33:17.615+0000 7ff1e2836740 1 --2- 172.21.15.114:0/3310973517 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x55591fa81800 0x55591fa83bf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:33:17.628 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:33:17.615+0000 7ff1e2836740 1 --2- 172.21.15.114:0/3310973517 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55591fa76f40 0x55591fa812c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:33:17.629 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:33:17.615+0000 7ff1e2836740 1 --2- 172.21.15.114:0/3310973517 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x55591fa76620 0x55591fa76a00 unknown :-1 s=CLOSED pgs=75 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:33:17.629 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:33:17.615+0000 7ff1e2836740 1 -- 172.21.15.114:0/3310973517 >> 172.21.15.114:0/3310973517 conn(0x55591fa72b30 msgr2=0x55591fa72f10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T10:33:17.629 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:33:17.616+0000 7ff1e2836740 1 -- 172.21.15.114:0/3310973517 shutdown_connections 2024-04-23T10:33:17.629 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:33:17.616+0000 7ff1e2836740 1 -- 172.21.15.114:0/3310973517 wait complete. 2024-04-23T10:33:17.629 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:33:17.616+0000 7ff1e2836740 1 -- start start 2024-04-23T10:33:17.629 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:33:17.616+0000 7ff1e2836740 1 Processor -- start 2024-04-23T10:33:17.629 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:33:17.616+0000 7ff1e2836740 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x55591fa76620 0x55591fa81080 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T10:33:17.629 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:33:17.616+0000 7ff1e2836740 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55591fa76f40 0x55591fa7e3e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T10:33:17.629 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:33:17.616+0000 7ff1e2836740 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x55591fa81800 0x55591fa7e920 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T10:33:17.629 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:33:17.616+0000 7ff1e2836740 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x55591fa84430 con 0x55591fa76f40 2024-04-23T10:33:17.629 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:33:17.616+0000 7ff1e2836740 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 -- 0x55591f9760b0 con 0x55591fa76620 2024-04-23T10:33:17.629 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:33:17.616+0000 7ff1e2836740 1 -- --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_getmap magic: 0 -- 0x55591fa845b0 con 0x55591fa81800 2024-04-23T10:33:17.630 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:33:17.616+0000 7ff1e2836740 10 client.0 _ll_register_callbacks cb 0x55591f963d30 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-23T10:33:17.630 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:33:17.616+0000 7ff1dcfde640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55591fa76f40 0x55591fa7e3e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T10:33:17.630 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:33:17.616+0000 7ff1dcfde640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55591fa76f40 0x55591fa7e3e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.114:50810/0 (socket says 172.21.15.114:50810) 2024-04-23T10:33:17.630 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:33:17.616+0000 7ff1dd7df640 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x55591fa76620 0x55591fa81080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T10:33:17.630 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:33:17.616+0000 7ff1dcfde640 1 -- 172.21.15.114:0/3250286981 learned_addr learned my addr 172.21.15.114:0/3250286981 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-04-23T10:33:17.630 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:33:17.616+0000 7ff1ddfe0640 1 --2- 172.21.15.114:0/3250286981 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x55591fa81800 0x55591fa7e920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T10:33:17.630 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:33:17.617+0000 7ff1dcfde640 1 -- 172.21.15.114:0/3250286981 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x55591fa81800 msgr2=0x55591fa7e920 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T10:33:17.630 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:33:17.617+0000 7ff1dcfde640 1 --2- 172.21.15.114:0/3250286981 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x55591fa81800 0x55591fa7e920 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:33:17.630 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:33:17.617+0000 7ff1dcfde640 1 -- 172.21.15.114:0/3250286981 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x55591fa76620 msgr2=0x55591fa81080 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T10:33:17.630 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:33:17.617+0000 7ff1dcfde640 1 --2- 172.21.15.114:0/3250286981 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x55591fa76620 0x55591fa81080 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:33:17.631 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:33:17.617+0000 7ff1dcfde640 1 -- 172.21.15.114:0/3250286981 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55591f7ba620 con 0x55591fa76f40 2024-04-23T10:33:17.631 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:33:17.617+0000 7ff1dcfde640 1 --2- 172.21.15.114:0/3250286981 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55591fa76f40 0x55591fa7e3e0 secure :-1 s=READY pgs=764 cs=0 l=1 rev1=1 crypto rx=0x7ff1c80b95c0 tx=0x7ff1c80b8fc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2b7d258b9c63ecbb server_cookie=0 in_seq=0 out_seq=0 2024-04-23T10:33:17.631 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:33:17.617+0000 7ff1c0ff9640 10 client.0 ms_handle_connect on v2:172.21.15.114:3300/0 2024-04-23T10:33:17.631 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:33:17.617+0000 7ff1c0ff9640 1 -- 172.21.15.114:0/3250286981 <== mon.0 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff1c8049040 con 0x55591fa76f40 2024-04-23T10:33:17.631 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:33:17.617+0000 7ff1c0ff9640 1 -- 172.21.15.114:0/3250286981 <== mon.0 v2:172.21.15.114:3300/0 2 ==== osd_map(96..96 src has 1..96) ==== 8629+0+0 (secure 0 0 0) 0x7ff1c80ee570 con 0x55591fa76f40 2024-04-23T10:33:17.631 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:33:17.617+0000 7ff1e2836740 1 -- 172.21.15.114:0/3250286981 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7ff1d40043d0 con 0x55591fa76f40 2024-04-23T10:33:17.631 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:33:17.617+0000 7ff1c0ff9640 1 -- 172.21.15.114:0/3250286981 <== mon.0 v2:172.21.15.114:3300/0 3 ==== mon_get_version_reply(handle=1 version=31) ==== 24+0+0 (secure 0 0 0) 0x7ff1c803b460 con 0x55591fa76f40 2024-04-23T10:33:17.631 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:33:17.617+0000 7ff1e2836740 10 client.16494 fetch_fsmap learned FSMap version 31 2024-04-23T10:33:17.631 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:33:17.617+0000 7ff1e2836740 1 -- 172.21.15.114:0/3250286981 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({config=0+,fsmap=31,monmap=4+}) -- 0x55591fa902e0 con 0x55591fa76f40 2024-04-23T10:33:17.631 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:33:17.617+0000 7ff1c0ff9640 1 -- 172.21.15.114:0/3250286981 <== mon.0 v2:172.21.15.114:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7ff1d40043d0 con 0x55591fa76f40 2024-04-23T10:33:17.632 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:33:17.617+0000 7ff1c0ff9640 1 -- 172.21.15.114:0/3250286981 <== mon.0 v2:172.21.15.114:3300/0 5 ==== fsmap(e 31) ==== 8342+0+0 (secure 0 0 0) 0x7ff1c80b9260 con 0x55591fa76f40 2024-04-23T10:33:17.632 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:33:17.617+0000 7ff1e2836740 10 client.16494 fetch_fsmap finished waiting for FSMap version 31 2024-04-23T10:33:17.632 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:33:17.618+0000 7ff1e2836740 10 client.16494 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-04-23T10:33:17.632 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:33:17.618+0000 7ff1e2836740 20 client.16494 populate_metadata read hostname 'smithi114' 2024-04-23T10:33:17.632 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:33:17.618+0000 7ff1e2836740 1 --2- 172.21.15.114:0/3250286981 >> [v2:172.21.15.119:6838/937297340,v1:172.21.15.119:6839/937297340] conn(0x55591fa918d0 0x55591fa93d60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T10:33:17.632 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:33:17.618+0000 7ff1e2836740 4 client.16494 mds_command: new command op to 24545 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-23T10:33:17.632 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:33:17.618+0000 7ff1e2836740 1 -- 172.21.15.114:0/3250286981 --> [v2:172.21.15.119:6838/937297340,v1:172.21.15.119:6839/937297340] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55591f9e9390 con 0x55591fa918d0 2024-04-23T10:33:17.632 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:33:17.618+0000 7ff1dd7df640 1 --2- 172.21.15.114:0/3250286981 >> [v2:172.21.15.119:6838/937297340,v1:172.21.15.119:6839/937297340] conn(0x55591fa918d0 0x55591fa93d60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T10:33:17.632 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:33:17.618+0000 7ff1dd7df640 1 --2- 172.21.15.114:0/3250286981 >> [v2:172.21.15.119:6838/937297340,v1:172.21.15.119:6839/937297340] conn(0x55591fa918d0 0x55591fa93d60 crc :-1 s=READY pgs=120 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=3bf9e61c36fdda8f server_cookie=831e21edd214a930 in_seq=0 out_seq=0 2024-04-23T10:33:17.632 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:33:17.618+0000 7ff1c0ff9640 10 client.16494 ms_handle_connect on v2:172.21.15.119:6838/937297340 2024-04-23T10:33:17.633 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:33:17.619+0000 7ff1c0ff9640 1 -- 172.21.15.114:0/3250286981 <== mds.0 v2:172.21.15.119:6838/937297340 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x55591f9e9390 con 0x55591fa918d0 2024-04-23T10:33:17.633 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:33:17.619+0000 7ff1c0ff9640 10 client.16494 handle_command_reply: tid=0 2024-04-23T10:33:17.633 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:33:17.619+0000 7ff1e2836740 1 client.16494 shutdown 2024-04-23T10:33:17.633 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:33:17.619+0000 7ff1e2836740 1 -- 172.21.15.114:0/3250286981 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55591fa76f40 msgr2=0x55591fa7e3e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T10:33:17.633 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:33:17.619+0000 7ff1e2836740 1 --2- 172.21.15.114:0/3250286981 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55591fa76f40 0x55591fa7e3e0 secure :-1 s=READY pgs=764 cs=0 l=1 rev1=1 crypto rx=0x7ff1c80b95c0 tx=0x7ff1c80b8fc0 comp rx=0 tx=0).stop 2024-04-23T10:33:17.633 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:33:17.619+0000 7ff1e2836740 1 -- 172.21.15.114:0/3250286981 shutdown_connections 2024-04-23T10:33:17.633 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:33:17.619+0000 7ff1e2836740 1 --2- 172.21.15.114:0/3250286981 >> [v2:172.21.15.119:6838/937297340,v1:172.21.15.119:6839/937297340] conn(0x55591fa918d0 0x55591fa93d60 crc :-1 s=READY pgs=120 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:33:17.633 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:33:17.619+0000 7ff1e2836740 1 --2- 172.21.15.114:0/3250286981 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x55591fa81800 0x55591fa7e920 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:33:17.634 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:33:17.619+0000 7ff1c0ff9640 0 client.16494 ms_handle_reset on v2:172.21.15.119:6838/937297340 2024-04-23T10:33:17.634 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:33:17.619+0000 7ff1e2836740 1 --2- 172.21.15.114:0/3250286981 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55591fa76f40 0x55591fa7e3e0 unknown :-1 s=CLOSED pgs=764 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:33:17.634 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:33:17.619+0000 7ff1e2836740 1 --2- 172.21.15.114:0/3250286981 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x55591fa76620 0x55591fa81080 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:33:17.634 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:33:17.619+0000 7ff1e2836740 1 -- 172.21.15.114:0/3250286981 >> 172.21.15.114:0/3250286981 conn(0x55591fa72b30 msgr2=0x55591fa788b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T10:33:17.634 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:33:17.619+0000 7ff1e2836740 1 -- 172.21.15.114:0/3250286981 shutdown_connections 2024-04-23T10:33:17.634 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:33:17.619+0000 7ff1e2836740 1 -- 172.21.15.114:0/3250286981 wait complete. 2024-04-23T10:33:17.634 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:33:17.619+0000 7ff1e2836740 20 client.16494 trim_cache size 0 max 16384 2024-04-23T10:33:17.640 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:33:17.634+0000 7ff1e2836740 1 Processor -- start 2024-04-23T10:33:17.640 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:33:17.634+0000 7ff1e2836740 1 -- start start 2024-04-23T10:33:17.641 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:33:17.634+0000 7ff1e2836740 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x55591fa76620 0x55591fa8d2f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T10:33:17.641 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:33:17.634+0000 7ff1e2836740 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55591fa76f40 0x55591fa79af0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T10:33:17.641 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:33:17.634+0000 7ff1e2836740 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x55591fa81800 0x55591fa7dcc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T10:33:17.641 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:33:17.634+0000 7ff1e2836740 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x7ff1d81b05b0 con 0x55591fa76f40 2024-04-23T10:33:17.641 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:33:17.634+0000 7ff1e2836740 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 -- 0x7ff1d81ade50 con 0x55591fa76620 2024-04-23T10:33:17.641 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:33:17.634+0000 7ff1e2836740 1 -- --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_getmap magic: 0 -- 0x7ff1d81abcc0 con 0x55591fa81800 2024-04-23T10:33:17.641 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:33:17.634+0000 7ff1dcfde640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55591fa76f40 0x55591fa79af0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T10:33:17.641 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:33:17.634+0000 7ff1dcfde640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55591fa76f40 0x55591fa79af0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.114:50820/0 (socket says 172.21.15.114:50820) 2024-04-23T10:33:17.641 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:33:17.634+0000 7ff1dcfde640 1 -- 172.21.15.114:0/1444400057 learned_addr learned my addr 172.21.15.114:0/1444400057 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-04-23T10:33:17.641 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:33:17.634+0000 7ff1dd7df640 1 --2- 172.21.15.114:0/1444400057 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x55591fa76620 0x55591fa8d2f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T10:33:17.641 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:33:17.634+0000 7ff1ddfe0640 1 --2- 172.21.15.114:0/1444400057 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x55591fa81800 0x55591fa7dcc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T10:33:17.642 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:33:17.635+0000 7ff1dcfde640 1 -- 172.21.15.114:0/1444400057 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x55591fa81800 msgr2=0x55591fa7dcc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T10:33:17.642 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:33:17.635+0000 7ff1dcfde640 1 --2- 172.21.15.114:0/1444400057 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x55591fa81800 0x55591fa7dcc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:33:17.642 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:33:17.635+0000 7ff1dcfde640 1 -- 172.21.15.114:0/1444400057 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x55591fa76620 msgr2=0x55591fa8d2f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T10:33:17.642 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:33:17.635+0000 7ff1dcfde640 1 --2- 172.21.15.114:0/1444400057 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x55591fa76620 0x55591fa8d2f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:33:17.642 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:33:17.635+0000 7ff1dcfde640 1 -- 172.21.15.114:0/1444400057 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55591fa76d70 con 0x55591fa76f40 2024-04-23T10:33:17.642 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:33:17.635+0000 7ff1dcfde640 1 --2- 172.21.15.114:0/1444400057 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55591fa76f40 0x55591fa79af0 secure :-1 s=READY pgs=765 cs=0 l=1 rev1=1 crypto rx=0x7ff1c8097330 tx=0x7ff1c80bb100 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4f628ff6e22a5a6d server_cookie=0 in_seq=0 out_seq=0 2024-04-23T10:33:17.642 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:33:17.635+0000 7ff1c2ffd640 1 -- 172.21.15.114:0/1444400057 <== mon.0 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff1c80d5070 con 0x55591fa76f40 2024-04-23T10:33:17.642 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:33:17.635+0000 7ff1c2ffd640 1 -- 172.21.15.114:0/1444400057 <== mon.0 v2:172.21.15.114:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7ff1c80c57b0 con 0x55591fa76f40 2024-04-23T10:33:17.642 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:33:17.635+0000 7ff1c2ffd640 1 -- 172.21.15.114:0/1444400057 <== mon.0 v2:172.21.15.114:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff1c80bce50 con 0x55591fa76f40 2024-04-23T10:33:17.642 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:33:17.636+0000 7ff1e2836740 1 -- 172.21.15.114:0/1444400057 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55591fa76f40 msgr2=0x55591fa79af0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T10:33:17.642 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:33:17.636+0000 7ff1e2836740 1 --2- 172.21.15.114:0/1444400057 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55591fa76f40 0x55591fa79af0 secure :-1 s=READY pgs=765 cs=0 l=1 rev1=1 crypto rx=0x7ff1c8097330 tx=0x7ff1c80bb100 comp rx=0 tx=0).stop 2024-04-23T10:33:17.643 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:33:17.636+0000 7ff1e2836740 1 -- 172.21.15.114:0/1444400057 shutdown_connections 2024-04-23T10:33:17.643 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:33:17.636+0000 7ff1e2836740 1 --2- 172.21.15.114:0/1444400057 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x55591fa81800 0x55591fa7dcc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:33:17.643 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:33:17.636+0000 7ff1e2836740 1 --2- 172.21.15.114:0/1444400057 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55591fa76f40 0x55591fa79af0 unknown :-1 s=CLOSED pgs=765 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:33:17.643 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:33:17.636+0000 7ff1e2836740 1 --2- 172.21.15.114:0/1444400057 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x55591fa76620 0x55591fa8d2f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:33:17.643 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:33:17.636+0000 7ff1e2836740 1 -- 172.21.15.114:0/1444400057 >> 172.21.15.114:0/1444400057 conn(0x55591fa72b30 msgr2=0x55591fa7fc10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T10:33:17.643 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:33:17.636+0000 7ff1e2836740 1 -- 172.21.15.114:0/1444400057 shutdown_connections 2024-04-23T10:33:17.643 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:33:17.636+0000 7ff1e2836740 1 -- 172.21.15.114:0/1444400057 wait complete. 2024-04-23T10:33:17.643 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:33:17.636+0000 7ff1e2836740 1 -- start start 2024-04-23T10:33:17.643 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:33:17.636+0000 7ff1e2836740 1 Processor -- start 2024-04-23T10:33:17.643 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:33:17.636+0000 7ff1e2836740 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x55591fa76620 0x55591faf00e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T10:33:17.643 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:33:17.636+0000 7ff1e2836740 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55591fa76f40 0x55591faf0620 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T10:33:17.644 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:33:17.636+0000 7ff1e2836740 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x55591fa81800 0x55591fa84bb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T10:33:17.644 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:33:17.636+0000 7ff1e2836740 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x7ff1d81ac310 con 0x55591fa76f40 2024-04-23T10:33:17.644 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:33:17.636+0000 7ff1e2836740 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 -- 0x7ff1d81ad780 con 0x55591fa81800 2024-04-23T10:33:17.644 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:33:17.636+0000 7ff1e2836740 1 -- --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_getmap magic: 0 -- 0x7ff1d81ac9e0 con 0x55591fa76620 2024-04-23T10:33:17.644 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:33:17.637+0000 7ff1e2836740 10 client.0 _ll_register_callbacks cb 0x7ff1c80bca50 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-23T10:33:17.644 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:33:17.637+0000 7ff1dcfde640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55591fa76f40 0x55591faf0620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T10:33:17.644 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:33:17.637+0000 7ff1dd7df640 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x55591fa76620 0x55591faf00e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T10:33:17.644 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:33:17.637+0000 7ff1ddfe0640 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x55591fa81800 0x55591fa84bb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T10:33:17.644 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:33:17.637+0000 7ff1dcfde640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55591fa76f40 0x55591faf0620 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.114:50836/0 (socket says 172.21.15.114:50836) 2024-04-23T10:33:17.644 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:33:17.637+0000 7ff1dd7df640 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x55591fa76620 0x55591faf00e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.204:3300/0 says I am v2:172.21.15.114:51318/0 (socket says 172.21.15.114:51318) 2024-04-23T10:33:17.644 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:33:17.637+0000 7ff1dcfde640 1 -- 172.21.15.114:0/1963610582 learned_addr learned my addr 172.21.15.114:0/1963610582 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-04-23T10:33:17.644 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:33:17.637+0000 7ff1ddfe0640 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x55591fa81800 0x55591fa84bb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.119:3300/0 says I am v2:172.21.15.114:44778/0 (socket says 172.21.15.114:44778) 2024-04-23T10:33:17.645 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:33:17.637+0000 7ff1dcfde640 1 -- 172.21.15.114:0/1963610582 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x55591fa76620 msgr2=0x55591faf00e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T10:33:17.645 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:33:17.637+0000 7ff1dcfde640 1 --2- 172.21.15.114:0/1963610582 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x55591fa76620 0x55591faf00e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:33:17.645 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:33:17.637+0000 7ff1dcfde640 1 -- 172.21.15.114:0/1963610582 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x55591fa81800 msgr2=0x55591fa84bb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T10:33:17.645 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:33:17.637+0000 7ff1dcfde640 1 --2- 172.21.15.114:0/1963610582 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x55591fa81800 0x55591fa84bb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:33:17.645 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:33:17.637+0000 7ff1dcfde640 1 -- 172.21.15.114:0/1963610582 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55591fa852a0 con 0x55591fa76f40 2024-04-23T10:33:17.645 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:33:17.637+0000 7ff1dcfde640 1 --2- 172.21.15.114:0/1963610582 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55591fa76f40 0x55591faf0620 secure :-1 s=READY pgs=766 cs=0 l=1 rev1=1 crypto rx=0x7ff1c80c5280 tx=0x7ff1c8000bc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a0d254501df40147 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T10:33:17.645 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:33:17.637+0000 7ff1c0ff9640 10 client.0 ms_handle_connect on v2:172.21.15.114:3300/0 2024-04-23T10:33:17.645 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:33:17.637+0000 7ff1c0ff9640 1 -- 172.21.15.114:0/1963610582 <== mon.0 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff1c80d5070 con 0x55591fa76f40 2024-04-23T10:33:17.645 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:33:17.637+0000 7ff1c0ff9640 1 -- 172.21.15.114:0/1963610582 <== mon.0 v2:172.21.15.114:3300/0 2 ==== osd_map(96..96 src has 1..96) ==== 8629+0+0 (secure 0 0 0) 0x7ff1c80bd050 con 0x55591fa76f40 2024-04-23T10:33:17.645 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:33:17.637+0000 7ff1e2836740 1 -- 172.21.15.114:0/1963610582 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7ff1c80c57b0 con 0x55591fa76f40 2024-04-23T10:33:17.645 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:33:17.637+0000 7ff1c0ff9640 1 -- 172.21.15.114:0/1963610582 <== mon.0 v2:172.21.15.114:3300/0 3 ==== mon_get_version_reply(handle=1 version=31) ==== 24+0+0 (secure 0 0 0) 0x7ff1c80bc550 con 0x55591fa76f40 2024-04-23T10:33:17.645 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:33:17.637+0000 7ff1e2836740 10 client.16500 fetch_fsmap learned FSMap version 31 2024-04-23T10:33:17.646 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:33:17.637+0000 7ff1e2836740 1 -- 172.21.15.114:0/1963610582 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({config=0+,fsmap=31,monmap=4+}) -- 0x55591fa852a0 con 0x55591fa76f40 2024-04-23T10:33:17.646 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:33:17.638+0000 7ff1c0ff9640 1 -- 172.21.15.114:0/1963610582 <== mon.0 v2:172.21.15.114:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7ff1c80c57b0 con 0x55591fa76f40 2024-04-23T10:33:17.646 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:33:17.638+0000 7ff1c0ff9640 1 -- 172.21.15.114:0/1963610582 <== mon.0 v2:172.21.15.114:3300/0 5 ==== fsmap(e 31) ==== 8342+0+0 (secure 0 0 0) 0x7ff1c803bcd0 con 0x55591fa76f40 2024-04-23T10:33:17.646 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:33:17.638+0000 7ff1e2836740 10 client.16500 fetch_fsmap finished waiting for FSMap version 31 2024-04-23T10:33:17.646 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:33:17.638+0000 7ff1e2836740 10 client.16500 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-04-23T10:33:17.646 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:33:17.638+0000 7ff1e2836740 20 client.16500 populate_metadata read hostname 'smithi114' 2024-04-23T10:33:17.646 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:33:17.638+0000 7ff1e2836740 1 --2- 172.21.15.114:0/1963610582 >> [v2:172.21.15.119:6838/937297340,v1:172.21.15.119:6839/937297340] conn(0x55591fa918d0 0x55591faf9ba0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T10:33:17.646 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:33:17.638+0000 7ff1e2836740 4 client.16500 mds_command: new command op to 24545 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-04-23T10:33:17.646 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:33:17.638+0000 7ff1e2836740 1 -- 172.21.15.114:0/1963610582 --> [v2:172.21.15.119:6838/937297340,v1:172.21.15.119:6839/937297340] -- command(tid 0: {"prefix": "flush journal"}) -- 0x55591f9e3200 con 0x55591fa918d0 2024-04-23T10:33:17.646 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:33:17.638+0000 7ff1dd7df640 1 --2- 172.21.15.114:0/1963610582 >> [v2:172.21.15.119:6838/937297340,v1:172.21.15.119:6839/937297340] conn(0x55591fa918d0 0x55591faf9ba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T10:33:17.646 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:33:17.638+0000 7ff1dd7df640 1 --2- 172.21.15.114:0/1963610582 >> [v2:172.21.15.119:6838/937297340,v1:172.21.15.119:6839/937297340] conn(0x55591fa918d0 0x55591faf9ba0 crc :-1 s=READY pgs=121 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=b660ed8852d63 server_cookie=29b270e05d583077 in_seq=0 out_seq=0 2024-04-23T10:33:17.646 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:33:17.638+0000 7ff1c0ff9640 10 client.16500 ms_handle_connect on v2:172.21.15.119:6838/937297340 2024-04-23T10:33:17.740 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:33:17.740+0000 7ff1c0ff9640 1 -- 172.21.15.114:0/1963610582 <== mds.0 v2:172.21.15.119:6838/937297340 1 ==== command_reply(tid 0: 0 ) ==== 8+0+44 (crc 0 0 0) 0x55591f9e3200 con 0x55591fa918d0 2024-04-23T10:33:17.740 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:33:17.740+0000 7ff1c0ff9640 10 client.16500 handle_command_reply: tid=0 2024-04-23T10:33:17.740 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:33:17.740+0000 7ff1e2836740 1 client.16500 shutdown 2024-04-23T10:33:17.741 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:33:17.740+0000 7ff1e2836740 1 -- 172.21.15.114:0/1963610582 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55591fa76f40 msgr2=0x55591faf0620 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T10:33:17.741 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:33:17.740+0000 7ff1e2836740 1 --2- 172.21.15.114:0/1963610582 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55591fa76f40 0x55591faf0620 secure :-1 s=READY pgs=766 cs=0 l=1 rev1=1 crypto rx=0x7ff1c80c5280 tx=0x7ff1c8000bc0 comp rx=0 tx=0).stop 2024-04-23T10:33:17.741 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:33:17.740+0000 7ff1e2836740 1 -- 172.21.15.114:0/1963610582 shutdown_connections 2024-04-23T10:33:17.741 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:33:17.740+0000 7ff1e2836740 1 --2- 172.21.15.114:0/1963610582 >> [v2:172.21.15.119:6838/937297340,v1:172.21.15.119:6839/937297340] conn(0x55591fa918d0 0x55591faf9ba0 crc :-1 s=READY pgs=121 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:33:17.741 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:33:17.740+0000 7ff1e2836740 1 --2- 172.21.15.114:0/1963610582 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x55591fa81800 0x55591fa84bb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:33:17.741 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:33:17.740+0000 7ff1e2836740 1 --2- 172.21.15.114:0/1963610582 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55591fa76f40 0x55591faf0620 unknown :-1 s=CLOSED pgs=766 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:33:17.741 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:33:17.740+0000 7ff1e2836740 1 --2- 172.21.15.114:0/1963610582 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x55591fa76620 0x55591faf00e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:33:17.741 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:33:17.740+0000 7ff1e2836740 1 -- 172.21.15.114:0/1963610582 >> 172.21.15.114:0/1963610582 conn(0x55591fa72b30 msgr2=0x55591fa7fc10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T10:33:17.741 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:33:17.740+0000 7ff1c0ff9640 0 client.16500 ms_handle_reset on v2:172.21.15.119:6838/937297340 2024-04-23T10:33:17.741 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:33:17.741+0000 7ff1e2836740 1 -- 172.21.15.114:0/1963610582 shutdown_connections 2024-04-23T10:33:17.742 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:33:17.741+0000 7ff1e2836740 1 -- 172.21.15.114:0/1963610582 wait complete. 2024-04-23T10:33:17.742 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:33:17.741+0000 7ff1e2836740 20 client.16500 trim_cache size 0 max 16384 2024-04-23T10:33:17.742 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stdout:{ 2024-04-23T10:33:17.742 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stdout: "message": "", 2024-04-23T10:33:17.742 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stdout: "return_code": 0 2024-04-23T10:33:17.742 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stdout:} 2024-04-23T10:33:17.743 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:33:17.741+0000 7ff1dfa43640 1 -- 172.21.15.114:0/1219875962 >> [v2:172.21.15.114:6800/1395693466,v1:172.21.15.114:6801/1395693466] conn(0x7ff1c40a3dc0 msgr2=0x7ff1c40a6280 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T10:33:17.743 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:33:17.741+0000 7ff1dfa43640 1 --2- 172.21.15.114:0/1219875962 >> [v2:172.21.15.114:6800/1395693466,v1:172.21.15.114:6801/1395693466] conn(0x7ff1c40a3dc0 0x7ff1c40a6280 secure :-1 s=READY pgs=114 cs=0 l=1 rev1=1 crypto rx=0x55591fa73f50 tx=0x7ff1d0009e70 comp rx=0 tx=0).stop 2024-04-23T10:33:17.744 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:33:17.741+0000 7ff1dfa43640 1 -- 172.21.15.114:0/1219875962 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7ff1d8107290 msgr2=0x7ff1d8198c50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T10:33:17.744 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:33:17.741+0000 7ff1dfa43640 1 --2- 172.21.15.114:0/1219875962 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7ff1d8107290 0x7ff1d8198c50 secure :-1 s=READY pgs=763 cs=0 l=1 rev1=1 crypto rx=0x7ff1c8006010 tx=0x7ff1c8004060 comp rx=0 tx=0).stop 2024-04-23T10:33:17.744 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:33:17.741+0000 7ff1dfa43640 1 -- 172.21.15.114:0/1219875962 shutdown_connections 2024-04-23T10:33:17.744 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:33:17.741+0000 7ff1dfa43640 1 --2- 172.21.15.114:0/1219875962 >> [v2:172.21.15.114:6800/1395693466,v1:172.21.15.114:6801/1395693466] conn(0x7ff1c40a3dc0 0x7ff1c40a6280 unknown :-1 s=CLOSED pgs=114 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:33:17.744 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:33:17.741+0000 7ff1dfa43640 1 --2- 172.21.15.114:0/1219875962 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7ff1d819cfd0 0x7ff1d819d480 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:33:17.744 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:33:17.741+0000 7ff1dfa43640 1 --2- 172.21.15.114:0/1219875962 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7ff1d8107290 0x7ff1d8198c50 unknown :-1 s=CLOSED pgs=763 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:33:17.744 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:33:17.741+0000 7ff1dfa43640 1 --2- 172.21.15.114:0/1219875962 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7ff1d81021c0 0x7ff1d8198710 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:33:17.744 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:33:17.741+0000 7ff1dfa43640 1 -- 172.21.15.114:0/1219875962 >> 172.21.15.114:0/1219875962 conn(0x7ff1d80fda40 msgr2=0x7ff1d80fed80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T10:33:17.744 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:33:17.742+0000 7ff1dfa43640 1 -- 172.21.15.114:0/1219875962 shutdown_connections 2024-04-23T10:33:17.744 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:33:17.742+0000 7ff1dfa43640 1 -- 172.21.15.114:0/1219875962 wait complete. 2024-04-23T10:33:17.745 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 23 10:33:17 smithi119 ceph-mon[32608]: pgmap v207: 97 pgs: 97 active+clean; 126 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 MiB/s rd, 3.6 MiB/s wr, 271 op/s 2024-04-23T10:33:17.754 INFO:journalctl@ceph.mon.a.smithi114.stdout:Apr 23 10:33:17 smithi114 ceph-mon[29255]: pgmap v207: 97 pgs: 97 active+clean; 126 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 MiB/s rd, 3.6 MiB/s wr, 271 op/s 2024-04-23T10:33:17.829 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 23 10:33:17 smithi204 ceph-mon[32649]: pgmap v207: 97 pgs: 97 active+clean; 126 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 MiB/s rd, 3.6 MiB/s wr, 271 op/s 2024-04-23T10:33:17.926 INFO:tasks.workunit.client.0.smithi114.stdout: 1 410041 6.06 MB/sec execute 206 sec latency 52.124 ms 2024-04-23T10:33:18.926 INFO:tasks.workunit.client.0.smithi114.stdout: 1 411893 6.09 MB/sec execute 207 sec latency 52.309 ms 2024-04-23T10:33:19.742 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 23 10:33:19 smithi119 ceph-mon[32608]: pgmap v208: 97 pgs: 97 active+clean; 126 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 MiB/s rd, 3.6 MiB/s wr, 271 op/s 2024-04-23T10:33:19.754 INFO:journalctl@ceph.mon.a.smithi114.stdout:Apr 23 10:33:19 smithi114 ceph-mon[29255]: pgmap v208: 97 pgs: 97 active+clean; 126 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 MiB/s rd, 3.6 MiB/s wr, 271 op/s 2024-04-23T10:33:19.828 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 23 10:33:19 smithi204 ceph-mon[32649]: pgmap v208: 97 pgs: 97 active+clean; 126 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 MiB/s rd, 3.6 MiB/s wr, 271 op/s 2024-04-23T10:33:19.926 INFO:tasks.workunit.client.0.smithi114.stdout: 1 413785 6.10 MB/sec execute 208 sec latency 25.795 ms 2024-04-23T10:33:20.926 INFO:tasks.workunit.client.0.smithi114.stdout: 1 415542 6.13 MB/sec execute 209 sec latency 16.262 ms 2024-04-23T10:33:21.741 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 23 10:33:21 smithi119 ceph-mon[32608]: pgmap v209: 97 pgs: 97 active+clean; 126 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 2.8 MiB/s rd, 3.0 MiB/s wr, 226 op/s 2024-04-23T10:33:21.753 INFO:journalctl@ceph.mon.a.smithi114.stdout:Apr 23 10:33:21 smithi114 ceph-mon[29255]: pgmap v209: 97 pgs: 97 active+clean; 126 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 2.8 MiB/s rd, 3.0 MiB/s wr, 226 op/s 2024-04-23T10:33:21.827 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 23 10:33:21 smithi204 ceph-mon[32649]: pgmap v209: 97 pgs: 97 active+clean; 126 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 2.8 MiB/s rd, 3.0 MiB/s wr, 226 op/s 2024-04-23T10:33:21.926 INFO:tasks.workunit.client.0.smithi114.stdout: 1 417409 6.13 MB/sec execute 210 sec latency 37.643 ms 2024-04-23T10:33:22.926 INFO:tasks.workunit.client.0.smithi114.stdout: 1 419141 6.16 MB/sec execute 211 sec latency 11.906 ms 2024-04-23T10:33:23.742 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 23 10:33:23 smithi119 ceph-mon[32608]: pgmap v210: 97 pgs: 97 active+clean; 121 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 4.0 MiB/s rd, 4.5 MiB/s wr, 377 op/s 2024-04-23T10:33:23.754 INFO:journalctl@ceph.mon.a.smithi114.stdout:Apr 23 10:33:23 smithi114 ceph-mon[29255]: pgmap v210: 97 pgs: 97 active+clean; 121 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 4.0 MiB/s rd, 4.5 MiB/s wr, 377 op/s 2024-04-23T10:33:23.827 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 23 10:33:23 smithi204 ceph-mon[32649]: pgmap v210: 97 pgs: 97 active+clean; 121 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 4.0 MiB/s rd, 4.5 MiB/s wr, 377 op/s 2024-04-23T10:33:23.926 INFO:tasks.workunit.client.0.smithi114.stdout: 1 421000 6.16 MB/sec execute 212 sec latency 36.852 ms 2024-04-23T10:33:24.926 INFO:tasks.workunit.client.0.smithi114.stdout: 1 422733 6.19 MB/sec execute 213 sec latency 19.759 ms 2024-04-23T10:33:25.742 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 23 10:33:25 smithi119 ceph-mon[32608]: pgmap v211: 97 pgs: 97 active+clean; 121 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 2.7 MiB/s rd, 3.1 MiB/s wr, 265 op/s 2024-04-23T10:33:25.754 INFO:journalctl@ceph.mon.a.smithi114.stdout:Apr 23 10:33:25 smithi114 ceph-mon[29255]: pgmap v211: 97 pgs: 97 active+clean; 121 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 2.7 MiB/s rd, 3.1 MiB/s wr, 265 op/s 2024-04-23T10:33:25.827 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 23 10:33:25 smithi204 ceph-mon[32649]: pgmap v211: 97 pgs: 97 active+clean; 121 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 2.7 MiB/s rd, 3.1 MiB/s wr, 265 op/s 2024-04-23T10:33:25.926 INFO:tasks.workunit.client.0.smithi114.stdout: 1 425055 6.22 MB/sec execute 214 sec latency 19.368 ms 2024-04-23T10:33:26.927 INFO:tasks.workunit.client.0.smithi114.stdout: 1 426597 6.23 MB/sec execute 215 sec latency 20.663 ms 2024-04-23T10:33:27.742 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 23 10:33:27 smithi119 ceph-mon[32608]: pgmap v212: 97 pgs: 97 active+clean; 129 MiB data, 1.7 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 MiB/s rd, 4.6 MiB/s wr, 380 op/s 2024-04-23T10:33:27.754 INFO:journalctl@ceph.mon.a.smithi114.stdout:Apr 23 10:33:27 smithi114 ceph-mon[29255]: pgmap v212: 97 pgs: 97 active+clean; 129 MiB data, 1.7 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 MiB/s rd, 4.6 MiB/s wr, 380 op/s 2024-04-23T10:33:27.828 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 23 10:33:27 smithi204 ceph-mon[32649]: pgmap v212: 97 pgs: 97 active+clean; 129 MiB data, 1.7 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 MiB/s rd, 4.6 MiB/s wr, 380 op/s 2024-04-23T10:33:27.926 INFO:tasks.workunit.client.0.smithi114.stdout: 1 428441 6.24 MB/sec execute 216 sec latency 22.074 ms 2024-04-23T10:33:28.927 INFO:tasks.workunit.client.0.smithi114.stdout: 1 429904 6.25 MB/sec execute 217 sec latency 35.744 ms 2024-04-23T10:33:29.741 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 23 10:33:29 smithi119 ceph-mon[32608]: pgmap v213: 97 pgs: 97 active+clean; 129 MiB data, 1.7 GiB used, 1.0 TiB / 1.0 TiB avail; 2.7 MiB/s rd, 3.1 MiB/s wr, 266 op/s 2024-04-23T10:33:29.754 INFO:journalctl@ceph.mon.a.smithi114.stdout:Apr 23 10:33:29 smithi114 ceph-mon[29255]: pgmap v213: 97 pgs: 97 active+clean; 129 MiB data, 1.7 GiB used, 1.0 TiB / 1.0 TiB avail; 2.7 MiB/s rd, 3.1 MiB/s wr, 266 op/s 2024-04-23T10:33:29.827 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 23 10:33:29 smithi204 ceph-mon[32649]: pgmap v213: 97 pgs: 97 active+clean; 129 MiB data, 1.7 GiB used, 1.0 TiB / 1.0 TiB avail; 2.7 MiB/s rd, 3.1 MiB/s wr, 266 op/s 2024-04-23T10:33:29.926 INFO:tasks.workunit.client.0.smithi114.stdout: 1 432572 6.29 MB/sec execute 218 sec latency 15.880 ms 2024-04-23T10:33:30.839 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:33:30.839+0000 7fef5818e640 1 -- 172.21.15.114:0/4139674775 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fef50101aa0 msgr2=0x7fef50105a90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T10:33:30.839 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:33:30.839+0000 7fef5818e640 1 --2- 172.21.15.114:0/4139674775 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fef50101aa0 0x7fef50105a90 secure :-1 s=READY pgs=767 cs=0 l=1 rev1=1 crypto rx=0x7fef40009900 tx=0x7fef4002a410 comp rx=0 tx=0).stop 2024-04-23T10:33:30.839 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:33:30.839+0000 7fef5818e640 1 -- 172.21.15.114:0/4139674775 shutdown_connections 2024-04-23T10:33:30.839 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:33:30.839+0000 7fef5818e640 1 --2- 172.21.15.114:0/4139674775 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fef50106170 0x7fef5010dd40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:33:30.839 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:33:30.839+0000 7fef5818e640 1 --2- 172.21.15.114:0/4139674775 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fef50101aa0 0x7fef50105a90 unknown :-1 s=CLOSED pgs=767 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:33:30.839 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:33:30.839+0000 7fef5818e640 1 --2- 172.21.15.114:0/4139674775 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fef501010f0 0x7fef501014d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:33:30.839 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:33:30.839+0000 7fef5818e640 1 -- 172.21.15.114:0/4139674775 >> 172.21.15.114:0/4139674775 conn(0x7fef50077fd0 msgr2=0x7fef500783e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T10:33:30.839 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:33:30.840+0000 7fef5818e640 1 -- 172.21.15.114:0/4139674775 shutdown_connections 2024-04-23T10:33:30.839 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:33:30.840+0000 7fef5818e640 1 -- 172.21.15.114:0/4139674775 wait complete. 2024-04-23T10:33:30.841 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:33:30.841+0000 7fef5818e640 1 Processor -- start 2024-04-23T10:33:30.841 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:33:30.841+0000 7fef5818e640 1 -- start start 2024-04-23T10:33:30.841 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:33:30.841+0000 7fef5818e640 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fef501010f0 0x7fef501986c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T10:33:30.841 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:33:30.841+0000 7fef5818e640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fef50106170 0x7fef50198c00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T10:33:30.841 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:33:30.841+0000 7fef5818e640 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fef5019cf80 0x7fef5019d430 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T10:33:30.842 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:33:30.841+0000 7fef5818e640 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x7fef50112220 con 0x7fef50106170 2024-04-23T10:33:30.842 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:33:30.841+0000 7fef5818e640 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 -- 0x7fef501120a0 con 0x7fef501010f0 2024-04-23T10:33:30.842 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:33:30.841+0000 7fef5818e640 1 -- --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_getmap magic: 0 -- 0x7fef501123a0 con 0x7fef5019cf80 2024-04-23T10:33:30.842 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:33:30.841+0000 7fef55f2a640 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fef501010f0 0x7fef501986c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T10:33:30.842 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:33:30.841+0000 7fef55729640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fef50106170 0x7fef50198c00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T10:33:30.842 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:33:30.841+0000 7fef55729640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fef50106170 0x7fef50198c00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.114:42936/0 (socket says 172.21.15.114:42936) 2024-04-23T10:33:30.842 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:33:30.841+0000 7fef5672b640 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fef5019cf80 0x7fef5019d430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T10:33:30.842 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:33:30.841+0000 7fef55f2a640 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fef501010f0 0x7fef501986c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.119:3300/0 says I am v2:172.21.15.114:36532/0 (socket says 172.21.15.114:36532) 2024-04-23T10:33:30.842 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:33:30.841+0000 7fef55729640 1 -- 172.21.15.114:0/170215612 learned_addr learned my addr 172.21.15.114:0/170215612 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-04-23T10:33:30.842 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:33:30.842+0000 7fef55729640 1 -- 172.21.15.114:0/170215612 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fef5019cf80 msgr2=0x7fef5019d430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T10:33:30.842 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:33:30.842+0000 7fef55729640 1 --2- 172.21.15.114:0/170215612 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fef5019cf80 0x7fef5019d430 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:33:30.842 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:33:30.842+0000 7fef55729640 1 -- 172.21.15.114:0/170215612 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fef501010f0 msgr2=0x7fef501986c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T10:33:30.842 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:33:30.842+0000 7fef55729640 1 --2- 172.21.15.114:0/170215612 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fef501010f0 0x7fef501986c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:33:30.842 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:33:30.842+0000 7fef55729640 1 -- 172.21.15.114:0/170215612 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fef5019da70 con 0x7fef50106170 2024-04-23T10:33:30.843 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:33:30.842+0000 7fef55729640 1 --2- 172.21.15.114:0/170215612 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fef50106170 0x7fef50198c00 secure :-1 s=READY pgs=768 cs=0 l=1 rev1=1 crypto rx=0x7fef40005ab0 tx=0x7fef4003ac10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T10:33:30.843 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:33:30.842+0000 7fef46ffd640 1 -- 172.21.15.114:0/170215612 <== mon.0 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fef40035070 con 0x7fef50106170 2024-04-23T10:33:30.843 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:33:30.842+0000 7fef5818e640 1 -- 172.21.15.114:0/170215612 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fef5019dd00 con 0x7fef50106170 2024-04-23T10:33:30.843 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:33:30.842+0000 7fef46ffd640 1 -- 172.21.15.114:0/170215612 <== mon.0 v2:172.21.15.114:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fef40003cb0 con 0x7fef50106170 2024-04-23T10:33:30.843 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:33:30.842+0000 7fef46ffd640 1 -- 172.21.15.114:0/170215612 <== mon.0 v2:172.21.15.114:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fef40031dd0 con 0x7fef50106170 2024-04-23T10:33:30.843 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:33:30.842+0000 7fef5818e640 1 -- 172.21.15.114:0/170215612 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fef50110030 con 0x7fef50106170 2024-04-23T10:33:30.845 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:33:30.846+0000 7fef5af80740 1 Processor -- start 2024-04-23T10:33:30.845 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:33:30.846+0000 7fef46ffd640 1 -- 172.21.15.114:0/170215612 <== mon.0 v2:172.21.15.114:3300/0 4 ==== mgrmap(e 26) ==== 138778+0+0 (secure 0 0 0) 0x7fef40031530 con 0x7fef50106170 2024-04-23T10:33:30.845 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:33:30.846+0000 7fef5af80740 1 -- start start 2024-04-23T10:33:30.845 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:33:30.846+0000 7fef5af80740 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x55f0993e0620 0x55f0993e0a00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T10:33:30.845 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:33:30.846+0000 7fef5af80740 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x55f0993e0f40 0x55f0993eb2c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T10:33:30.845 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:33:30.846+0000 7fef5af80740 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55f0993eb800 0x55f0993edbf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T10:33:30.846 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:33:30.846+0000 7fef5af80740 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x55f099345970 con 0x55f0993eb800 2024-04-23T10:33:30.846 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:33:30.846+0000 7fef5af80740 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 -- 0x55f0992e24a0 con 0x55f0993e0620 2024-04-23T10:33:30.846 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:33:30.846+0000 7fef5af80740 1 -- --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_getmap magic: 0 -- 0x55f099341930 con 0x55f0993e0f40 2024-04-23T10:33:30.846 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:33:30.847+0000 7fef55729640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55f0993eb800 0x55f0993edbf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T10:33:30.846 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:33:30.847+0000 7fef55f2a640 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x55f0993e0f40 0x55f0993eb2c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T10:33:30.846 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:33:30.847+0000 7fef55f2a640 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x55f0993e0f40 0x55f0993eb2c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.204:3300/0 says I am v2:172.21.15.114:59584/0 (socket says 172.21.15.114:59584) 2024-04-23T10:33:30.846 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:33:30.847+0000 7fef55f2a640 1 -- 172.21.15.114:0/2253192259 learned_addr learned my addr 172.21.15.114:0/2253192259 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-04-23T10:33:30.848 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:33:30.847+0000 7fef55729640 1 -- 172.21.15.114:0/2253192259 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x55f0993e0f40 msgr2=0x55f0993eb2c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T10:33:30.848 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:33:30.847+0000 7fef55729640 1 --2- 172.21.15.114:0/2253192259 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x55f0993e0f40 0x55f0993eb2c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:33:30.848 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:33:30.847+0000 7fef5672b640 1 --2- 172.21.15.114:0/2253192259 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x55f0993e0620 0x55f0993e0a00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T10:33:30.848 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:33:30.847+0000 7fef55729640 1 -- 172.21.15.114:0/2253192259 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x55f0993e0620 msgr2=0x55f0993e0a00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T10:33:30.848 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:33:30.847+0000 7fef55729640 1 --2- 172.21.15.114:0/2253192259 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x55f0993e0620 0x55f0993e0a00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:33:30.848 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:33:30.847+0000 7fef55729640 1 -- 172.21.15.114:0/2253192259 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55f099058210 con 0x55f0993eb800 2024-04-23T10:33:30.848 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:33:30.847+0000 7fef55f2a640 1 --2- 172.21.15.114:0/2253192259 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x55f0993e0f40 0x55f0993eb2c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-23T10:33:30.848 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:33:30.847+0000 7fef5672b640 1 --2- 172.21.15.114:0/2253192259 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x55f0993e0620 0x55f0993e0a00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-23T10:33:30.848 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:33:30.847+0000 7fef46ffd640 1 --2- 172.21.15.114:0/170215612 >> [v2:172.21.15.114:6800/1395693466,v1:172.21.15.114:6801/1395693466] conn(0x7fef3c0a3dc0 0x7fef3c0a6280 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T10:33:30.849 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:33:30.847+0000 7fef55729640 1 --2- 172.21.15.114:0/2253192259 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55f0993eb800 0x55f0993edbf0 secure :-1 s=READY pgs=769 cs=0 l=1 rev1=1 crypto rx=0x7fef400f21c0 tx=0x7fef400f8be0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=31c6dc94265c3523 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T10:33:30.849 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:33:30.848+0000 7fef5672b640 1 --2- 172.21.15.114:0/170215612 >> [v2:172.21.15.114:6800/1395693466,v1:172.21.15.114:6801/1395693466] conn(0x7fef3c0a3dc0 0x7fef3c0a6280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T10:33:30.849 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:33:30.848+0000 7fef46ffd640 1 -- 172.21.15.114:0/170215612 <== mon.0 v2:172.21.15.114:3300/0 5 ==== osd_map(96..96 src has 1..96) ==== 8629+0+0 (secure 0 0 0) 0x7fef400f4b40 con 0x7fef50106170 2024-04-23T10:33:30.849 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:33:30.848+0000 7fef3bfff640 1 -- 172.21.15.114:0/2253192259 <== mon.0 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fef400bccb0 con 0x55f0993eb800 2024-04-23T10:33:30.849 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:33:30.848+0000 7fef3bfff640 1 -- 172.21.15.114:0/2253192259 <== mon.0 v2:172.21.15.114:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fef400c1110 con 0x55f0993eb800 2024-04-23T10:33:30.849 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:33:30.849+0000 7fef3bfff640 1 -- 172.21.15.114:0/2253192259 <== mon.0 v2:172.21.15.114:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fef40003e60 con 0x55f0993eb800 2024-04-23T10:33:30.849 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:33:30.849+0000 7fef5af80740 1 -- 172.21.15.114:0/2253192259 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55f0993eb800 msgr2=0x55f0993edbf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T10:33:30.849 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:33:30.849+0000 7fef5672b640 1 --2- 172.21.15.114:0/170215612 >> [v2:172.21.15.114:6800/1395693466,v1:172.21.15.114:6801/1395693466] conn(0x7fef3c0a3dc0 0x7fef3c0a6280 secure :-1 s=READY pgs=115 cs=0 l=1 rev1=1 crypto rx=0x7fef48005fd0 tx=0x7fef48009500 comp rx=0 tx=0).ready entity=mgr.15291 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T10:33:30.849 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:33:30.849+0000 7fef5af80740 1 --2- 172.21.15.114:0/2253192259 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55f0993eb800 0x55f0993edbf0 secure :-1 s=READY pgs=769 cs=0 l=1 rev1=1 crypto rx=0x7fef400f21c0 tx=0x7fef400f8be0 comp rx=0 tx=0).stop 2024-04-23T10:33:30.849 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:33:30.849+0000 7fef5af80740 1 -- 172.21.15.114:0/2253192259 shutdown_connections 2024-04-23T10:33:30.849 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:33:30.849+0000 7fef5af80740 1 --2- 172.21.15.114:0/2253192259 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55f0993eb800 0x55f0993edbf0 unknown :-1 s=CLOSED pgs=769 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:33:30.849 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:33:30.849+0000 7fef5af80740 1 --2- 172.21.15.114:0/2253192259 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x55f0993e0f40 0x55f0993eb2c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:33:30.849 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:33:30.849+0000 7fef5af80740 1 --2- 172.21.15.114:0/2253192259 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x55f0993e0620 0x55f0993e0a00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:33:30.849 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:33:30.849+0000 7fef5af80740 1 -- 172.21.15.114:0/2253192259 >> 172.21.15.114:0/2253192259 conn(0x55f0993dcb30 msgr2=0x55f0993dcf10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T10:33:30.849 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:33:30.849+0000 7fef5af80740 1 -- 172.21.15.114:0/2253192259 shutdown_connections 2024-04-23T10:33:30.849 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:33:30.849+0000 7fef5af80740 1 -- 172.21.15.114:0/2253192259 wait complete. 2024-04-23T10:33:30.852 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:33:30.849+0000 7fef5af80740 1 -- start start 2024-04-23T10:33:30.852 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:33:30.850+0000 7fef5af80740 1 Processor -- start 2024-04-23T10:33:30.852 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:33:30.850+0000 7fef5af80740 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x55f0993e0620 0x55f0993eb050 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T10:33:30.852 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:33:30.850+0000 7fef5af80740 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55f0993e0f40 0x55f0993e8340 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T10:33:30.852 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:33:30.850+0000 7fef5af80740 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x55f0993eb800 0x55f0993e8880 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T10:33:30.852 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:33:30.850+0000 7fef5af80740 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x55f0993ee1f0 con 0x55f0993e0f40 2024-04-23T10:33:30.852 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:33:30.850+0000 7fef5af80740 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 -- 0x55f0992e00b0 con 0x55f0993eb800 2024-04-23T10:33:30.853 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:33:30.850+0000 7fef5af80740 1 -- --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_getmap magic: 0 -- 0x55f0993ee370 con 0x55f0993e0620 2024-04-23T10:33:30.853 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:33:30.850+0000 7fef5af80740 10 client.0 _ll_register_callbacks cb 0x55f0992cdd30 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-23T10:33:30.853 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:33:30.850+0000 7fef55729640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55f0993e0f40 0x55f0993e8340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T10:33:30.853 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:33:30.850+0000 7fef55729640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55f0993e0f40 0x55f0993e8340 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.114:42952/0 (socket says 172.21.15.114:42952) 2024-04-23T10:33:30.853 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:33:30.850+0000 7fef55729640 1 -- 172.21.15.114:0/586255605 learned_addr learned my addr 172.21.15.114:0/586255605 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-04-23T10:33:30.853 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:33:30.851+0000 7fef5672b640 1 --2- 172.21.15.114:0/586255605 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x55f0993eb800 0x55f0993e8880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T10:33:30.853 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:33:30.851+0000 7fef55f2a640 1 --2- 172.21.15.114:0/586255605 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x55f0993e0620 0x55f0993eb050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T10:33:30.853 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:33:30.851+0000 7fef55f2a640 1 -- 172.21.15.114:0/586255605 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x55f0993eb800 msgr2=0x55f0993e8880 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T10:33:30.853 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:33:30.851+0000 7fef55f2a640 1 --2- 172.21.15.114:0/586255605 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x55f0993eb800 0x55f0993e8880 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:33:30.853 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:33:30.851+0000 7fef55f2a640 1 -- 172.21.15.114:0/586255605 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55f0993e0f40 msgr2=0x55f0993e8340 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T10:33:30.853 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:33:30.851+0000 7fef55f2a640 1 --2- 172.21.15.114:0/586255605 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55f0993e0f40 0x55f0993e8340 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:33:30.853 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:33:30.851+0000 7fef55f2a640 1 -- 172.21.15.114:0/586255605 --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55f099124620 con 0x55f0993e0620 2024-04-23T10:33:30.853 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:33:30.851+0000 7fef55729640 1 --2- 172.21.15.114:0/586255605 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55f0993e0f40 0x55f0993e8340 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-23T10:33:30.853 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:33:30.851+0000 7fef55f2a640 1 --2- 172.21.15.114:0/586255605 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x55f0993e0620 0x55f0993eb050 secure :-1 s=READY pgs=76 cs=0 l=1 rev1=1 crypto rx=0x7fef4c0126b0 tx=0x7fef4c012b70 comp rx=0 tx=0).ready entity=mon.1 client_cookie=6addfd667cdc002 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T10:33:30.853 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:33:30.851+0000 7fef397fa640 10 client.0 ms_handle_connect on v2:172.21.15.204:3300/0 2024-04-23T10:33:30.854 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:33:30.851+0000 7fef397fa640 1 -- 172.21.15.114:0/586255605 <== mon.1 v2:172.21.15.204:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fef4c004020 con 0x55f0993e0620 2024-04-23T10:33:30.854 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:33:30.852+0000 7fef397fa640 1 -- 172.21.15.114:0/586255605 <== mon.1 v2:172.21.15.204:3300/0 2 ==== osd_map(96..96 src has 1..96) ==== 8629+0+0 (secure 0 0 0) 0x7fef4c0041c0 con 0x55f0993e0620 2024-04-23T10:33:30.854 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:33:30.852+0000 7fef5af80740 1 -- 172.21.15.114:0/586255605 --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fef400c1110 con 0x55f0993e0620 2024-04-23T10:33:30.854 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:33:30.852+0000 7fef397fa640 1 -- 172.21.15.114:0/586255605 <== mon.1 v2:172.21.15.204:3300/0 3 ==== mon_get_version_reply(handle=1 version=31) ==== 24+0+0 (secure 0 0 0) 0x7fef4c0043c0 con 0x55f0993e0620 2024-04-23T10:33:30.854 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:33:30.852+0000 7fef5af80740 10 client.25774 fetch_fsmap learned FSMap version 31 2024-04-23T10:33:30.854 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:33:30.852+0000 7fef5af80740 1 -- 172.21.15.114:0/586255605 --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_subscribe({config=0+,fsmap=31,monmap=4+}) -- 0x55f099402830 con 0x55f0993e0620 2024-04-23T10:33:30.854 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:33:30.852+0000 7fef397fa640 1 -- 172.21.15.114:0/586255605 <== mon.1 v2:172.21.15.204:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fef400c1110 con 0x55f0993e0620 2024-04-23T10:33:30.854 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:33:30.852+0000 7fef397fa640 1 -- 172.21.15.114:0/586255605 <== mon.1 v2:172.21.15.204:3300/0 5 ==== fsmap(e 31) ==== 8342+0+0 (secure 0 0 0) 0x7fef4c003a30 con 0x55f0993e0620 2024-04-23T10:33:30.854 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:33:30.852+0000 7fef5af80740 10 client.25774 fetch_fsmap finished waiting for FSMap version 31 2024-04-23T10:33:30.854 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:33:30.852+0000 7fef5af80740 10 client.25774 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-04-23T10:33:30.854 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:33:30.852+0000 7fef5af80740 20 client.25774 populate_metadata read hostname 'smithi114' 2024-04-23T10:33:30.854 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:33:30.852+0000 7fef5af80740 1 --2- 172.21.15.114:0/586255605 >> [v2:172.21.15.119:6838/937297340,v1:172.21.15.119:6839/937297340] conn(0x55f099403cf0 0x55f099406160 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T10:33:30.854 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:33:30.852+0000 7fef5af80740 4 client.25774 mds_command: new command op to 24545 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-23T10:33:30.854 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:33:30.852+0000 7fef5af80740 1 -- 172.21.15.114:0/586255605 --> [v2:172.21.15.119:6838/937297340,v1:172.21.15.119:6839/937297340] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55f099353390 con 0x55f099403cf0 2024-04-23T10:33:30.854 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:33:30.853+0000 7fef55729640 1 --2- 172.21.15.114:0/586255605 >> [v2:172.21.15.119:6838/937297340,v1:172.21.15.119:6839/937297340] conn(0x55f099403cf0 0x55f099406160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T10:33:30.855 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:33:30.853+0000 7fef55729640 1 --2- 172.21.15.114:0/586255605 >> [v2:172.21.15.119:6838/937297340,v1:172.21.15.119:6839/937297340] conn(0x55f099403cf0 0x55f099406160 crc :-1 s=READY pgs=122 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=7d97610bc46cfa8a server_cookie=dccd5b33e90b6c02 in_seq=0 out_seq=0 2024-04-23T10:33:30.855 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:33:30.853+0000 7fef397fa640 10 client.25774 ms_handle_connect on v2:172.21.15.119:6838/937297340 2024-04-23T10:33:30.855 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:33:30.854+0000 7fef397fa640 1 -- 172.21.15.114:0/586255605 <== mds.0 v2:172.21.15.119:6838/937297340 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x55f099353390 con 0x55f099403cf0 2024-04-23T10:33:30.856 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:33:30.854+0000 7fef397fa640 10 client.25774 handle_command_reply: tid=0 2024-04-23T10:33:30.856 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:33:30.854+0000 7fef5af80740 1 client.25774 shutdown 2024-04-23T10:33:30.856 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:33:30.854+0000 7fef5af80740 1 -- 172.21.15.114:0/586255605 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x55f0993e0620 msgr2=0x55f0993eb050 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T10:33:30.856 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:33:30.854+0000 7fef5af80740 1 --2- 172.21.15.114:0/586255605 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x55f0993e0620 0x55f0993eb050 secure :-1 s=READY pgs=76 cs=0 l=1 rev1=1 crypto rx=0x7fef4c0126b0 tx=0x7fef4c012b70 comp rx=0 tx=0).stop 2024-04-23T10:33:30.856 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:33:30.854+0000 7fef5af80740 1 -- 172.21.15.114:0/586255605 shutdown_connections 2024-04-23T10:33:30.856 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:33:30.854+0000 7fef5af80740 1 --2- 172.21.15.114:0/586255605 >> [v2:172.21.15.119:6838/937297340,v1:172.21.15.119:6839/937297340] conn(0x55f099403cf0 0x55f099406160 crc :-1 s=READY pgs=122 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:33:30.856 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:33:30.854+0000 7fef5af80740 1 --2- 172.21.15.114:0/586255605 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x55f0993eb800 0x55f0993e8880 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:33:30.856 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:33:30.854+0000 7fef5af80740 1 --2- 172.21.15.114:0/586255605 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55f0993e0f40 0x55f0993e8340 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:33:30.856 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:33:30.854+0000 7fef5af80740 1 --2- 172.21.15.114:0/586255605 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x55f0993e0620 0x55f0993eb050 unknown :-1 s=CLOSED pgs=76 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:33:30.856 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:33:30.854+0000 7fef397fa640 0 client.25774 ms_handle_reset on v2:172.21.15.119:6838/937297340 2024-04-23T10:33:30.856 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:33:30.854+0000 7fef5af80740 1 -- 172.21.15.114:0/586255605 >> 172.21.15.114:0/586255605 conn(0x55f0993dcb30 msgr2=0x55f0993e2860 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T10:33:30.856 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:33:30.854+0000 7fef5af80740 1 -- 172.21.15.114:0/586255605 shutdown_connections 2024-04-23T10:33:30.856 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:33:30.854+0000 7fef5af80740 1 -- 172.21.15.114:0/586255605 wait complete. 2024-04-23T10:33:30.856 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:33:30.854+0000 7fef5af80740 20 client.25774 trim_cache size 0 max 16384 2024-04-23T10:33:30.871 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:33:30.870+0000 7fef5af80740 1 Processor -- start 2024-04-23T10:33:30.871 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:33:30.870+0000 7fef5af80740 1 -- start start 2024-04-23T10:33:30.871 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:33:30.871+0000 7fef5af80740 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x55f0993e0620 0x55f0993eab80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T10:33:30.871 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:33:30.871+0000 7fef5af80740 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55f0993e0f40 0x55f09940ba70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T10:33:30.872 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:33:30.871+0000 7fef5af80740 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x55f0993eb800 0x55f09940bfb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T10:33:30.872 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:33:30.871+0000 7fef5af80740 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x7fef501ae410 con 0x55f0993e0f40 2024-04-23T10:33:30.872 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:33:30.871+0000 7fef5af80740 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 -- 0x7fef501abcb0 con 0x55f0993eb800 2024-04-23T10:33:30.872 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:33:30.871+0000 7fef5af80740 1 -- --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_getmap magic: 0 -- 0x7fef501a9b20 con 0x55f0993e0620 2024-04-23T10:33:30.872 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:33:30.871+0000 7fef55729640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55f0993e0f40 0x55f09940ba70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T10:33:30.872 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:33:30.871+0000 7fef5672b640 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x55f0993eb800 0x55f09940bfb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T10:33:30.872 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:33:30.871+0000 7fef55f2a640 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x55f0993e0620 0x55f0993eab80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T10:33:30.872 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:33:30.871+0000 7fef55729640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55f0993e0f40 0x55f09940ba70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.114:42956/0 (socket says 172.21.15.114:42956) 2024-04-23T10:33:30.872 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:33:30.871+0000 7fef5672b640 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x55f0993eb800 0x55f09940bfb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.119:3300/0 says I am v2:172.21.15.114:36556/0 (socket says 172.21.15.114:36556) 2024-04-23T10:33:30.872 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:33:30.871+0000 7fef55729640 1 -- 172.21.15.114:0/1034657514 learned_addr learned my addr 172.21.15.114:0/1034657514 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-04-23T10:33:30.872 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:33:30.871+0000 7fef55729640 1 -- 172.21.15.114:0/1034657514 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x55f0993e0620 msgr2=0x55f0993eab80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T10:33:30.872 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:33:30.871+0000 7fef55729640 1 --2- 172.21.15.114:0/1034657514 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x55f0993e0620 0x55f0993eab80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:33:30.872 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:33:30.871+0000 7fef55729640 1 -- 172.21.15.114:0/1034657514 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x55f0993eb800 msgr2=0x55f09940bfb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T10:33:30.872 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:33:30.871+0000 7fef55729640 1 --2- 172.21.15.114:0/1034657514 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x55f0993eb800 0x55f09940bfb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:33:30.873 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:33:30.871+0000 7fef55729640 1 -- 172.21.15.114:0/1034657514 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55f0994009b0 con 0x55f0993e0f40 2024-04-23T10:33:30.873 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:33:30.871+0000 7fef5672b640 1 --2- 172.21.15.114:0/1034657514 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x55f0993eb800 0x55f09940bfb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-23T10:33:30.873 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:33:30.871+0000 7fef55729640 1 --2- 172.21.15.114:0/1034657514 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55f0993e0f40 0x55f09940ba70 secure :-1 s=READY pgs=770 cs=0 l=1 rev1=1 crypto rx=0x7fef400f2220 tx=0x7fef400314a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=fec1034a687dea7c server_cookie=0 in_seq=0 out_seq=0 2024-04-23T10:33:30.873 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:33:30.871+0000 7fef3b7fe640 1 -- 172.21.15.114:0/1034657514 <== mon.0 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fef400c0370 con 0x55f0993e0f40 2024-04-23T10:33:30.873 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:33:30.871+0000 7fef3b7fe640 1 -- 172.21.15.114:0/1034657514 <== mon.0 v2:172.21.15.114:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fef400c1110 con 0x55f0993e0f40 2024-04-23T10:33:30.873 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:33:30.871+0000 7fef3b7fe640 1 -- 172.21.15.114:0/1034657514 <== mon.0 v2:172.21.15.114:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fef400c0550 con 0x55f0993e0f40 2024-04-23T10:33:30.873 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:33:30.872+0000 7fef5af80740 1 -- 172.21.15.114:0/1034657514 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55f0993e0f40 msgr2=0x55f09940ba70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T10:33:30.873 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:33:30.872+0000 7fef5af80740 1 --2- 172.21.15.114:0/1034657514 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55f0993e0f40 0x55f09940ba70 secure :-1 s=READY pgs=770 cs=0 l=1 rev1=1 crypto rx=0x7fef400f2220 tx=0x7fef400314a0 comp rx=0 tx=0).stop 2024-04-23T10:33:30.873 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:33:30.872+0000 7fef5af80740 1 -- 172.21.15.114:0/1034657514 shutdown_connections 2024-04-23T10:33:30.873 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:33:30.872+0000 7fef5af80740 1 --2- 172.21.15.114:0/1034657514 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x55f0993eb800 0x55f09940bfb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:33:30.873 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:33:30.872+0000 7fef5af80740 1 --2- 172.21.15.114:0/1034657514 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55f0993e0f40 0x55f09940ba70 unknown :-1 s=CLOSED pgs=770 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:33:30.873 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:33:30.872+0000 7fef5af80740 1 --2- 172.21.15.114:0/1034657514 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x55f0993e0620 0x55f0993eab80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:33:30.873 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:33:30.872+0000 7fef5af80740 1 -- 172.21.15.114:0/1034657514 >> 172.21.15.114:0/1034657514 conn(0x55f0993dcb30 msgr2=0x55f0993dbcb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T10:33:30.873 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:33:30.872+0000 7fef5af80740 1 -- 172.21.15.114:0/1034657514 shutdown_connections 2024-04-23T10:33:30.873 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:33:30.872+0000 7fef5af80740 1 -- 172.21.15.114:0/1034657514 wait complete. 2024-04-23T10:33:30.873 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:33:30.872+0000 7fef5af80740 1 -- start start 2024-04-23T10:33:30.874 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:33:30.873+0000 7fef5af80740 1 Processor -- start 2024-04-23T10:33:30.876 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:33:30.873+0000 7fef5af80740 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55f0993e0620 0x55f0993f3320 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T10:33:30.876 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:33:30.873+0000 7fef5af80740 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x55f0993e0f40 0x55f0993f3860 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T10:33:30.876 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:33:30.873+0000 7fef5af80740 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x55f0993eb800 0x55f0993f07b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T10:33:30.877 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:33:30.873+0000 7fef5af80740 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x7fef501aa170 con 0x55f0993e0620 2024-04-23T10:33:30.877 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:33:30.873+0000 7fef5af80740 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 -- 0x7fef501ab5e0 con 0x55f0993eb800 2024-04-23T10:33:30.877 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:33:30.873+0000 7fef5af80740 1 -- --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_getmap magic: 0 -- 0x7fef501aa840 con 0x55f0993e0f40 2024-04-23T10:33:30.877 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:33:30.873+0000 7fef5af80740 10 client.0 _ll_register_callbacks cb 0x7fef4c009ea0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-23T10:33:30.877 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:33:30.873+0000 7fef55f2a640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55f0993e0620 0x55f0993f3320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T10:33:30.877 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:33:30.873+0000 7fef55f2a640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55f0993e0620 0x55f0993f3320 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.114:42970/0 (socket says 172.21.15.114:42970) 2024-04-23T10:33:30.877 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:33:30.873+0000 7fef55f2a640 1 -- 172.21.15.114:0/868899621 learned_addr learned my addr 172.21.15.114:0/868899621 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-04-23T10:33:30.877 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:33:30.873+0000 7fef5672b640 1 --2- 172.21.15.114:0/868899621 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x55f0993eb800 0x55f0993f07b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T10:33:30.877 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:33:30.873+0000 7fef55729640 1 --2- 172.21.15.114:0/868899621 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x55f0993e0f40 0x55f0993f3860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T10:33:30.877 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:33:30.873+0000 7fef55f2a640 1 -- 172.21.15.114:0/868899621 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x55f0993e0f40 msgr2=0x55f0993f3860 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T10:33:30.877 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:33:30.873+0000 7fef55f2a640 1 --2- 172.21.15.114:0/868899621 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x55f0993e0f40 0x55f0993f3860 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:33:30.877 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:33:30.873+0000 7fef55f2a640 1 -- 172.21.15.114:0/868899621 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x55f0993eb800 msgr2=0x55f0993f07b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T10:33:30.877 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:33:30.873+0000 7fef55f2a640 1 --2- 172.21.15.114:0/868899621 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x55f0993eb800 0x55f0993f07b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:33:30.877 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:33:30.873+0000 7fef55f2a640 1 -- 172.21.15.114:0/868899621 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55f0993f0f40 con 0x55f0993e0620 2024-04-23T10:33:30.878 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:33:30.873+0000 7fef55729640 1 --2- 172.21.15.114:0/868899621 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x55f0993e0f40 0x55f0993f3860 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-23T10:33:30.878 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:33:30.873+0000 7fef55f2a640 1 --2- 172.21.15.114:0/868899621 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55f0993e0620 0x55f0993f3320 secure :-1 s=READY pgs=771 cs=0 l=1 rev1=1 crypto rx=0x7fef4c012420 tx=0x7fef4c005820 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7d7b603ae7e9cb03 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T10:33:30.878 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:33:30.873+0000 7fef397fa640 10 client.0 ms_handle_connect on v2:172.21.15.114:3300/0 2024-04-23T10:33:30.878 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:33:30.874+0000 7fef397fa640 1 -- 172.21.15.114:0/868899621 <== mon.0 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fef4c028040 con 0x55f0993e0620 2024-04-23T10:33:30.878 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:33:30.874+0000 7fef397fa640 1 -- 172.21.15.114:0/868899621 <== mon.0 v2:172.21.15.114:3300/0 2 ==== osd_map(96..96 src has 1..96) ==== 8629+0+0 (secure 0 0 0) 0x7fef4c004100 con 0x55f0993e0620 2024-04-23T10:33:30.878 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:33:30.874+0000 7fef5af80740 1 -- 172.21.15.114:0/868899621 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fef400c1110 con 0x55f0993e0620 2024-04-23T10:33:30.878 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:33:30.874+0000 7fef397fa640 1 -- 172.21.15.114:0/868899621 <== mon.0 v2:172.21.15.114:3300/0 3 ==== mon_get_version_reply(handle=1 version=31) ==== 24+0+0 (secure 0 0 0) 0x7fef4c039050 con 0x55f0993e0620 2024-04-23T10:33:30.878 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:33:30.874+0000 7fef5af80740 10 client.16518 fetch_fsmap learned FSMap version 31 2024-04-23T10:33:30.878 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:33:30.874+0000 7fef5af80740 1 -- 172.21.15.114:0/868899621 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({config=0+,fsmap=31,monmap=4+}) -- 0x55f099469d40 con 0x55f0993e0620 2024-04-23T10:33:30.878 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:33:30.874+0000 7fef397fa640 1 -- 172.21.15.114:0/868899621 <== mon.0 v2:172.21.15.114:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fef400c1110 con 0x55f0993e0620 2024-04-23T10:33:30.878 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:33:30.874+0000 7fef397fa640 1 -- 172.21.15.114:0/868899621 <== mon.0 v2:172.21.15.114:3300/0 5 ==== fsmap(e 31) ==== 8342+0+0 (secure 0 0 0) 0x7fef4c034030 con 0x55f0993e0620 2024-04-23T10:33:30.878 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:33:30.874+0000 7fef5af80740 10 client.16518 fetch_fsmap finished waiting for FSMap version 31 2024-04-23T10:33:30.878 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:33:30.874+0000 7fef5af80740 10 client.16518 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-04-23T10:33:30.878 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:33:30.874+0000 7fef5af80740 20 client.16518 populate_metadata read hostname 'smithi114' 2024-04-23T10:33:30.878 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:33:30.874+0000 7fef5af80740 1 --2- 172.21.15.114:0/868899621 >> [v2:172.21.15.119:6838/937297340,v1:172.21.15.119:6839/937297340] conn(0x55f09946a510 0x55f09946c930 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T10:33:30.878 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:33:30.874+0000 7fef5af80740 4 client.16518 mds_command: new command op to 24545 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-04-23T10:33:30.879 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:33:30.874+0000 7fef5af80740 1 -- 172.21.15.114:0/868899621 --> [v2:172.21.15.119:6838/937297340,v1:172.21.15.119:6839/937297340] -- command(tid 0: {"prefix": "flush journal"}) -- 0x55f09934d200 con 0x55f09946a510 2024-04-23T10:33:30.879 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:33:30.874+0000 7fef55729640 1 --2- 172.21.15.114:0/868899621 >> [v2:172.21.15.119:6838/937297340,v1:172.21.15.119:6839/937297340] conn(0x55f09946a510 0x55f09946c930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T10:33:30.879 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:33:30.875+0000 7fef397fa640 10 client.16518 ms_handle_connect on v2:172.21.15.119:6838/937297340 2024-04-23T10:33:30.879 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:33:30.875+0000 7fef55729640 1 --2- 172.21.15.114:0/868899621 >> [v2:172.21.15.119:6838/937297340,v1:172.21.15.119:6839/937297340] conn(0x55f09946a510 0x55f09946c930 crc :-1 s=READY pgs=123 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=16b5b8158dbb965 server_cookie=5ff27b7cd14a1226 in_seq=0 out_seq=0 2024-04-23T10:33:30.926 INFO:tasks.workunit.client.0.smithi114.stdout: 1 433824 6.29 MB/sec execute 219 sec latency 22.621 ms 2024-04-23T10:33:30.935 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:33:30.936+0000 7fef397fa640 1 -- 172.21.15.114:0/868899621 <== mds.0 v2:172.21.15.119:6838/937297340 1 ==== command_reply(tid 0: 0 ) ==== 8+0+44 (crc 0 0 0) 0x55f09934d200 con 0x55f09946a510 2024-04-23T10:33:30.935 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:33:30.936+0000 7fef397fa640 10 client.16518 handle_command_reply: tid=0 2024-04-23T10:33:30.935 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:33:30.936+0000 7fef5af80740 1 client.16518 shutdown 2024-04-23T10:33:30.935 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:33:30.936+0000 7fef5af80740 1 -- 172.21.15.114:0/868899621 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55f0993e0620 msgr2=0x55f0993f3320 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T10:33:30.936 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:33:30.936+0000 7fef5af80740 1 --2- 172.21.15.114:0/868899621 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55f0993e0620 0x55f0993f3320 secure :-1 s=READY pgs=771 cs=0 l=1 rev1=1 crypto rx=0x7fef4c012420 tx=0x7fef4c005820 comp rx=0 tx=0).stop 2024-04-23T10:33:30.936 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:33:30.936+0000 7fef5af80740 1 -- 172.21.15.114:0/868899621 shutdown_connections 2024-04-23T10:33:30.936 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:33:30.936+0000 7fef5af80740 1 --2- 172.21.15.114:0/868899621 >> [v2:172.21.15.119:6838/937297340,v1:172.21.15.119:6839/937297340] conn(0x55f09946a510 0x55f09946c930 crc :-1 s=READY pgs=123 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:33:30.936 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:33:30.936+0000 7fef5af80740 1 --2- 172.21.15.114:0/868899621 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x55f0993eb800 0x55f0993f07b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:33:30.936 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:33:30.936+0000 7fef397fa640 0 client.16518 ms_handle_reset on v2:172.21.15.119:6838/937297340 2024-04-23T10:33:30.936 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:33:30.936+0000 7fef5af80740 1 --2- 172.21.15.114:0/868899621 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x55f0993e0f40 0x55f0993f3860 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:33:30.936 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:33:30.936+0000 7fef5af80740 1 --2- 172.21.15.114:0/868899621 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55f0993e0620 0x55f0993f3320 unknown :-1 s=CLOSED pgs=771 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:33:30.936 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:33:30.936+0000 7fef5af80740 1 -- 172.21.15.114:0/868899621 >> 172.21.15.114:0/868899621 conn(0x55f0993dcb30 msgr2=0x55f0993e1fd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T10:33:30.936 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:33:30.936+0000 7fef5af80740 1 -- 172.21.15.114:0/868899621 shutdown_connections 2024-04-23T10:33:30.936 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:33:30.936+0000 7fef5af80740 1 -- 172.21.15.114:0/868899621 wait complete. 2024-04-23T10:33:30.936 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:33:30.936+0000 7fef5af80740 20 client.16518 trim_cache size 0 max 16384 2024-04-23T10:33:30.937 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stdout:{ 2024-04-23T10:33:30.937 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stdout: "message": "", 2024-04-23T10:33:30.937 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stdout: "return_code": 0 2024-04-23T10:33:30.937 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stdout:} 2024-04-23T10:33:30.937 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:33:30.937+0000 7fef5818e640 1 -- 172.21.15.114:0/170215612 >> [v2:172.21.15.114:6800/1395693466,v1:172.21.15.114:6801/1395693466] conn(0x7fef3c0a3dc0 msgr2=0x7fef3c0a6280 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T10:33:30.938 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:33:30.937+0000 7fef5818e640 1 --2- 172.21.15.114:0/170215612 >> [v2:172.21.15.114:6800/1395693466,v1:172.21.15.114:6801/1395693466] conn(0x7fef3c0a3dc0 0x7fef3c0a6280 secure :-1 s=READY pgs=115 cs=0 l=1 rev1=1 crypto rx=0x7fef48005fd0 tx=0x7fef48009500 comp rx=0 tx=0).stop 2024-04-23T10:33:30.938 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:33:30.937+0000 7fef5818e640 1 -- 172.21.15.114:0/170215612 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fef50106170 msgr2=0x7fef50198c00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T10:33:30.938 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:33:30.937+0000 7fef5818e640 1 --2- 172.21.15.114:0/170215612 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fef50106170 0x7fef50198c00 secure :-1 s=READY pgs=768 cs=0 l=1 rev1=1 crypto rx=0x7fef40005ab0 tx=0x7fef4003ac10 comp rx=0 tx=0).stop 2024-04-23T10:33:30.938 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:33:30.937+0000 7fef5818e640 1 -- 172.21.15.114:0/170215612 shutdown_connections 2024-04-23T10:33:30.938 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:33:30.937+0000 7fef5818e640 1 --2- 172.21.15.114:0/170215612 >> [v2:172.21.15.114:6800/1395693466,v1:172.21.15.114:6801/1395693466] conn(0x7fef3c0a3dc0 0x7fef3c0a6280 unknown :-1 s=CLOSED pgs=115 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:33:30.938 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:33:30.937+0000 7fef5818e640 1 --2- 172.21.15.114:0/170215612 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fef5019cf80 0x7fef5019d430 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:33:30.938 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:33:30.937+0000 7fef5818e640 1 --2- 172.21.15.114:0/170215612 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fef50106170 0x7fef50198c00 unknown :-1 s=CLOSED pgs=768 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:33:30.938 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:33:30.937+0000 7fef5818e640 1 --2- 172.21.15.114:0/170215612 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fef501010f0 0x7fef501986c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:33:30.938 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:33:30.937+0000 7fef5818e640 1 -- 172.21.15.114:0/170215612 >> 172.21.15.114:0/170215612 conn(0x7fef50077fd0 msgr2=0x7fef500fef80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T10:33:30.938 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:33:30.937+0000 7fef5818e640 1 -- 172.21.15.114:0/170215612 shutdown_connections 2024-04-23T10:33:30.938 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:33:30.937+0000 7fef5818e640 1 -- 172.21.15.114:0/170215612 wait complete. 2024-04-23T10:33:31.741 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 23 10:33:31 smithi119 ceph-mon[32608]: pgmap v214: 97 pgs: 97 active+clean; 129 MiB data, 1.7 GiB used, 1.0 TiB / 1.0 TiB avail; 2.7 MiB/s rd, 3.1 MiB/s wr, 266 op/s 2024-04-23T10:33:31.754 INFO:journalctl@ceph.mon.a.smithi114.stdout:Apr 23 10:33:31 smithi114 ceph-mon[29255]: pgmap v214: 97 pgs: 97 active+clean; 129 MiB data, 1.7 GiB used, 1.0 TiB / 1.0 TiB avail; 2.7 MiB/s rd, 3.1 MiB/s wr, 266 op/s 2024-04-23T10:33:31.827 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 23 10:33:31 smithi204 ceph-mon[32649]: pgmap v214: 97 pgs: 97 active+clean; 129 MiB data, 1.7 GiB used, 1.0 TiB / 1.0 TiB avail; 2.7 MiB/s rd, 3.1 MiB/s wr, 266 op/s 2024-04-23T10:33:31.926 INFO:tasks.workunit.client.0.smithi114.stdout: 1 436292 6.33 MB/sec execute 220 sec latency 67.875 ms 2024-04-23T10:33:32.926 INFO:tasks.workunit.client.0.smithi114.stdout: 1 437565 6.32 MB/sec execute 221 sec latency 19.611 ms 2024-04-23T10:33:33.741 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 23 10:33:33 smithi119 ceph-mon[32608]: pgmap v215: 97 pgs: 97 active+clean; 124 MiB data, 1.7 GiB used, 1.0 TiB / 1.0 TiB avail; 4.4 MiB/s rd, 4.4 MiB/s wr, 419 op/s 2024-04-23T10:33:33.753 INFO:journalctl@ceph.mon.a.smithi114.stdout:Apr 23 10:33:33 smithi114 ceph-mon[29255]: pgmap v215: 97 pgs: 97 active+clean; 124 MiB data, 1.7 GiB used, 1.0 TiB / 1.0 TiB avail; 4.4 MiB/s rd, 4.4 MiB/s wr, 419 op/s 2024-04-23T10:33:33.827 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 23 10:33:33 smithi204 ceph-mon[32649]: pgmap v215: 97 pgs: 97 active+clean; 124 MiB data, 1.7 GiB used, 1.0 TiB / 1.0 TiB avail; 4.4 MiB/s rd, 4.4 MiB/s wr, 419 op/s 2024-04-23T10:33:33.926 INFO:tasks.workunit.client.0.smithi114.stdout: 1 439974 6.36 MB/sec execute 222 sec latency 34.632 ms 2024-04-23T10:33:34.927 INFO:tasks.workunit.client.0.smithi114.stdout: 1 441290 6.37 MB/sec execute 223 sec latency 19.463 ms 2024-04-23T10:33:35.741 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 23 10:33:35 smithi119 ceph-mon[32608]: pgmap v216: 97 pgs: 97 active+clean; 124 MiB data, 1.7 GiB used, 1.0 TiB / 1.0 TiB avail; 3.2 MiB/s rd, 2.9 MiB/s wr, 268 op/s 2024-04-23T10:33:35.773 INFO:journalctl@ceph.mon.a.smithi114.stdout:Apr 23 10:33:35 smithi114 ceph-mon[29255]: pgmap v216: 97 pgs: 97 active+clean; 124 MiB data, 1.7 GiB used, 1.0 TiB / 1.0 TiB avail; 3.2 MiB/s rd, 2.9 MiB/s wr, 268 op/s 2024-04-23T10:33:35.827 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 23 10:33:35 smithi204 ceph-mon[32649]: pgmap v216: 97 pgs: 97 active+clean; 124 MiB data, 1.7 GiB used, 1.0 TiB / 1.0 TiB avail; 3.2 MiB/s rd, 2.9 MiB/s wr, 268 op/s 2024-04-23T10:33:35.926 INFO:tasks.workunit.client.0.smithi114.stdout: 1 443624 6.40 MB/sec execute 224 sec latency 82.365 ms 2024-04-23T10:33:36.927 INFO:tasks.workunit.client.0.smithi114.stdout: 1 444860 6.39 MB/sec execute 225 sec latency 22.683 ms 2024-04-23T10:33:37.741 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 23 10:33:37 smithi119 ceph-mon[32608]: pgmap v217: 97 pgs: 97 active+clean; 131 MiB data, 1.7 GiB used, 1.0 TiB / 1.0 TiB avail; 4.4 MiB/s rd, 4.2 MiB/s wr, 377 op/s 2024-04-23T10:33:37.753 INFO:journalctl@ceph.mon.a.smithi114.stdout:Apr 23 10:33:37 smithi114 ceph-mon[29255]: pgmap v217: 97 pgs: 97 active+clean; 131 MiB data, 1.7 GiB used, 1.0 TiB / 1.0 TiB avail; 4.4 MiB/s rd, 4.2 MiB/s wr, 377 op/s 2024-04-23T10:33:37.827 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 23 10:33:37 smithi204 ceph-mon[32649]: pgmap v217: 97 pgs: 97 active+clean; 131 MiB data, 1.7 GiB used, 1.0 TiB / 1.0 TiB avail; 4.4 MiB/s rd, 4.2 MiB/s wr, 377 op/s 2024-04-23T10:33:37.927 INFO:tasks.workunit.client.0.smithi114.stdout: 1 447210 6.43 MB/sec execute 226 sec latency 14.803 ms 2024-04-23T10:33:38.927 INFO:tasks.workunit.client.0.smithi114.stdout: 1 448434 6.42 MB/sec execute 227 sec latency 40.591 ms 2024-04-23T10:33:39.742 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 23 10:33:39 smithi119 ceph-mon[32608]: pgmap v218: 97 pgs: 97 active+clean; 131 MiB data, 1.7 GiB used, 1.0 TiB / 1.0 TiB avail; 3.0 MiB/s rd, 2.7 MiB/s wr, 262 op/s 2024-04-23T10:33:39.754 INFO:journalctl@ceph.mon.a.smithi114.stdout:Apr 23 10:33:39 smithi114 ceph-mon[29255]: pgmap v218: 97 pgs: 97 active+clean; 131 MiB data, 1.7 GiB used, 1.0 TiB / 1.0 TiB avail; 3.0 MiB/s rd, 2.7 MiB/s wr, 262 op/s 2024-04-23T10:33:39.827 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 23 10:33:39 smithi204 ceph-mon[32649]: pgmap v218: 97 pgs: 97 active+clean; 131 MiB data, 1.7 GiB used, 1.0 TiB / 1.0 TiB avail; 3.0 MiB/s rd, 2.7 MiB/s wr, 262 op/s 2024-04-23T10:33:39.927 INFO:tasks.workunit.client.0.smithi114.stdout: 1 450767 6.45 MB/sec execute 228 sec latency 20.762 ms 2024-04-23T10:33:40.927 INFO:tasks.workunit.client.0.smithi114.stdout: 1 452187 6.45 MB/sec execute 229 sec latency 16.035 ms 2024-04-23T10:33:41.741 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 23 10:33:41 smithi119 ceph-mon[32608]: pgmap v219: 97 pgs: 97 active+clean; 131 MiB data, 1.7 GiB used, 1.0 TiB / 1.0 TiB avail; 3.0 MiB/s rd, 2.7 MiB/s wr, 262 op/s 2024-04-23T10:33:41.753 INFO:journalctl@ceph.mon.a.smithi114.stdout:Apr 23 10:33:41 smithi114 ceph-mon[29255]: pgmap v219: 97 pgs: 97 active+clean; 131 MiB data, 1.7 GiB used, 1.0 TiB / 1.0 TiB avail; 3.0 MiB/s rd, 2.7 MiB/s wr, 262 op/s 2024-04-23T10:33:41.827 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 23 10:33:41 smithi204 ceph-mon[32649]: pgmap v219: 97 pgs: 97 active+clean; 131 MiB data, 1.7 GiB used, 1.0 TiB / 1.0 TiB avail; 3.0 MiB/s rd, 2.7 MiB/s wr, 262 op/s 2024-04-23T10:33:41.927 INFO:tasks.workunit.client.0.smithi114.stdout: 1 454364 6.48 MB/sec execute 230 sec latency 47.517 ms 2024-04-23T10:33:42.927 INFO:tasks.workunit.client.0.smithi114.stdout: 1 455687 6.48 MB/sec execute 231 sec latency 20.720 ms 2024-04-23T10:33:43.742 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 23 10:33:43 smithi119 ceph-mon[32608]: pgmap v220: 97 pgs: 97 active+clean; 138 MiB data, 1.7 GiB used, 1.0 TiB / 1.0 TiB avail; 4.5 MiB/s rd, 4.1 MiB/s wr, 378 op/s 2024-04-23T10:33:43.754 INFO:journalctl@ceph.mon.a.smithi114.stdout:Apr 23 10:33:43 smithi114 ceph-mon[29255]: pgmap v220: 97 pgs: 97 active+clean; 138 MiB data, 1.7 GiB used, 1.0 TiB / 1.0 TiB avail; 4.5 MiB/s rd, 4.1 MiB/s wr, 378 op/s 2024-04-23T10:33:43.827 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 23 10:33:43 smithi204 ceph-mon[32649]: pgmap v220: 97 pgs: 97 active+clean; 138 MiB data, 1.7 GiB used, 1.0 TiB / 1.0 TiB avail; 4.5 MiB/s rd, 4.1 MiB/s wr, 378 op/s 2024-04-23T10:33:43.927 INFO:tasks.workunit.client.0.smithi114.stdout: 1 457806 6.51 MB/sec execute 232 sec latency 34.596 ms 2024-04-23T10:33:44.029 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:33:44.027+0000 7fe894ca7640 1 -- 172.21.15.114:0/3734441336 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fe890107280 msgr2=0x7fe89010ee50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T10:33:44.029 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:33:44.027+0000 7fe894ca7640 1 --2- 172.21.15.114:0/3734441336 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fe890107280 0x7fe89010ee50 secure :-1 s=READY pgs=772 cs=0 l=1 rev1=1 crypto rx=0x7fe888009ca0 tx=0x7fe88802a8c0 comp rx=0 tx=0).stop 2024-04-23T10:33:44.029 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:33:44.027+0000 7fe894ca7640 1 -- 172.21.15.114:0/3734441336 shutdown_connections 2024-04-23T10:33:44.029 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:33:44.027+0000 7fe894ca7640 1 --2- 172.21.15.114:0/3734441336 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fe890107280 0x7fe89010ee50 unknown :-1 s=CLOSED pgs=772 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:33:44.029 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:33:44.027+0000 7fe894ca7640 1 --2- 172.21.15.114:0/3734441336 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fe89006b350 0x7fe890106ba0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:33:44.029 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:33:44.027+0000 7fe894ca7640 1 --2- 172.21.15.114:0/3734441336 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fe89006a9a0 0x7fe89006ad80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:33:44.029 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:33:44.027+0000 7fe894ca7640 1 -- 172.21.15.114:0/3734441336 >> 172.21.15.114:0/3734441336 conn(0x7fe890077c30 msgr2=0x7fe890078040 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T10:33:44.029 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:33:44.027+0000 7fe894ca7640 1 -- 172.21.15.114:0/3734441336 shutdown_connections 2024-04-23T10:33:44.029 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:33:44.027+0000 7fe894ca7640 1 -- 172.21.15.114:0/3734441336 wait complete. 2024-04-23T10:33:44.030 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:33:44.028+0000 7fe894ca7640 1 Processor -- start 2024-04-23T10:33:44.030 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:33:44.028+0000 7fe894ca7640 1 -- start start 2024-04-23T10:33:44.030 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:33:44.029+0000 7fe894ca7640 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fe89006a9a0 0x7fe890198680 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T10:33:44.030 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:33:44.029+0000 7fe894ca7640 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fe890107280 0x7fe890198bc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T10:33:44.030 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:33:44.029+0000 7fe894ca7640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fe89019cf40 0x7fe89019d3f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T10:33:44.030 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:33:44.029+0000 7fe894ca7640 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x7fe890112240 con 0x7fe89019cf40 2024-04-23T10:33:44.030 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:33:44.029+0000 7fe894ca7640 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 -- 0x7fe8901120c0 con 0x7fe89006a9a0 2024-04-23T10:33:44.030 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:33:44.029+0000 7fe894ca7640 1 -- --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_getmap magic: 0 -- 0x7fe8901123c0 con 0x7fe890107280 2024-04-23T10:33:44.030 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:33:44.029+0000 7fe88ed9d640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fe89019cf40 0x7fe89019d3f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T10:33:44.030 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:33:44.029+0000 7fe88ed9d640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fe89019cf40 0x7fe89019d3f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.114:57988/0 (socket says 172.21.15.114:57988) 2024-04-23T10:33:44.030 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:33:44.029+0000 7fe88e59c640 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fe89006a9a0 0x7fe890198680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T10:33:44.031 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:33:44.029+0000 7fe88ed9d640 1 -- 172.21.15.114:0/849372087 learned_addr learned my addr 172.21.15.114:0/849372087 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-04-23T10:33:44.031 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:33:44.029+0000 7fe885d9b640 1 --2- 172.21.15.114:0/849372087 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fe890107280 0x7fe890198bc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T10:33:44.031 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:33:44.029+0000 7fe88ed9d640 1 -- 172.21.15.114:0/849372087 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fe890107280 msgr2=0x7fe890198bc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T10:33:44.031 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:33:44.029+0000 7fe88ed9d640 1 --2- 172.21.15.114:0/849372087 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fe890107280 0x7fe890198bc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:33:44.031 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:33:44.029+0000 7fe88ed9d640 1 -- 172.21.15.114:0/849372087 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fe89006a9a0 msgr2=0x7fe890198680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T10:33:44.031 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:33:44.029+0000 7fe88ed9d640 1 --2- 172.21.15.114:0/849372087 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fe89006a9a0 0x7fe890198680 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:33:44.031 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:33:44.029+0000 7fe88ed9d640 1 -- 172.21.15.114:0/849372087 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe89019da30 con 0x7fe89019cf40 2024-04-23T10:33:44.031 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:33:44.029+0000 7fe88e59c640 1 --2- 172.21.15.114:0/849372087 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fe89006a9a0 0x7fe890198680 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-23T10:33:44.031 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:33:44.029+0000 7fe88ed9d640 1 --2- 172.21.15.114:0/849372087 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fe89019cf40 0x7fe89019d3f0 secure :-1 s=READY pgs=773 cs=0 l=1 rev1=1 crypto rx=0x7fe8880098d0 tx=0x7fe8880317b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T10:33:44.031 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:33:44.029+0000 7fe887fff640 1 -- 172.21.15.114:0/849372087 <== mon.0 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe888003b70 con 0x7fe89019cf40 2024-04-23T10:33:44.031 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:33:44.029+0000 7fe894ca7640 1 -- 172.21.15.114:0/849372087 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fe89019dcc0 con 0x7fe89019cf40 2024-04-23T10:33:44.032 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:33:44.029+0000 7fe887fff640 1 -- 172.21.15.114:0/849372087 <== mon.0 v2:172.21.15.114:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fe888003d10 con 0x7fe89019cf40 2024-04-23T10:33:44.032 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:33:44.029+0000 7fe894ca7640 1 -- 172.21.15.114:0/849372087 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fe8901a5510 con 0x7fe89019cf40 2024-04-23T10:33:44.032 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:33:44.030+0000 7fe887fff640 1 -- 172.21.15.114:0/849372087 <== mon.0 v2:172.21.15.114:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe88802f970 con 0x7fe89019cf40 2024-04-23T10:33:44.039 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:33:44.032+0000 7fe887fff640 1 -- 172.21.15.114:0/849372087 <== mon.0 v2:172.21.15.114:3300/0 4 ==== mgrmap(e 26) ==== 138778+0+0 (secure 0 0 0) 0x7fe88802fb10 con 0x7fe89019cf40 2024-04-23T10:33:44.039 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:33:44.032+0000 7fe887fff640 1 --2- 172.21.15.114:0/849372087 >> [v2:172.21.15.114:6800/1395693466,v1:172.21.15.114:6801/1395693466] conn(0x7fe87c0a3dc0 0x7fe87c0a6280 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T10:33:44.039 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:33:44.032+0000 7fe88e59c640 1 --2- 172.21.15.114:0/849372087 >> [v2:172.21.15.114:6800/1395693466,v1:172.21.15.114:6801/1395693466] conn(0x7fe87c0a3dc0 0x7fe87c0a6280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T10:33:44.039 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:33:44.032+0000 7fe887fff640 1 -- 172.21.15.114:0/849372087 <== mon.0 v2:172.21.15.114:3300/0 5 ==== osd_map(96..96 src has 1..96) ==== 8629+0+0 (secure 0 0 0) 0x7fe8880f1c20 con 0x7fe89019cf40 2024-04-23T10:33:44.039 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:33:44.032+0000 7fe8979bf740 1 Processor -- start 2024-04-23T10:33:44.039 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:33:44.033+0000 7fe88e59c640 1 --2- 172.21.15.114:0/849372087 >> [v2:172.21.15.114:6800/1395693466,v1:172.21.15.114:6801/1395693466] conn(0x7fe87c0a3dc0 0x7fe87c0a6280 secure :-1 s=READY pgs=116 cs=0 l=1 rev1=1 crypto rx=0x7fe880009fc0 tx=0x7fe880005f90 comp rx=0 tx=0).ready entity=mgr.15291 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T10:33:44.040 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:33:44.033+0000 7fe8979bf740 1 -- start start 2024-04-23T10:33:44.040 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:33:44.033+0000 7fe8979bf740 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55995da13620 0x55995da13a00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T10:33:44.040 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:33:44.033+0000 7fe8979bf740 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x55995da13f40 0x55995da1e2c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T10:33:44.040 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:33:44.033+0000 7fe8979bf740 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x55995da1e800 0x55995da20bf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T10:33:44.040 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:33:44.033+0000 7fe8979bf740 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x55995d978970 con 0x55995da13620 2024-04-23T10:33:44.040 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:33:44.033+0000 7fe8979bf740 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 -- 0x55995d9154a0 con 0x55995da13f40 2024-04-23T10:33:44.040 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:33:44.033+0000 7fe8979bf740 1 -- --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_getmap magic: 0 -- 0x55995d974930 con 0x55995da1e800 2024-04-23T10:33:44.040 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:33:44.033+0000 7fe88e59c640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55995da13620 0x55995da13a00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T10:33:44.040 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:33:44.033+0000 7fe885d9b640 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x55995da13f40 0x55995da1e2c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T10:33:44.040 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:33:44.033+0000 7fe88e59c640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55995da13620 0x55995da13a00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.114:57992/0 (socket says 172.21.15.114:57992) 2024-04-23T10:33:44.040 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:33:44.033+0000 7fe88e59c640 1 -- 172.21.15.114:0/1131674923 learned_addr learned my addr 172.21.15.114:0/1131674923 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-04-23T10:33:44.040 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:33:44.033+0000 7fe88ed9d640 1 --2- 172.21.15.114:0/1131674923 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x55995da1e800 0x55995da20bf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T10:33:44.040 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:33:44.033+0000 7fe885d9b640 1 -- 172.21.15.114:0/1131674923 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x55995da1e800 msgr2=0x55995da20bf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T10:33:44.040 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:33:44.033+0000 7fe885d9b640 1 --2- 172.21.15.114:0/1131674923 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x55995da1e800 0x55995da20bf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:33:44.041 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:33:44.033+0000 7fe885d9b640 1 -- 172.21.15.114:0/1131674923 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55995da13620 msgr2=0x55995da13a00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T10:33:44.041 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:33:44.033+0000 7fe885d9b640 1 --2- 172.21.15.114:0/1131674923 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55995da13620 0x55995da13a00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:33:44.041 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:33:44.033+0000 7fe885d9b640 1 -- 172.21.15.114:0/1131674923 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55995d68b210 con 0x55995da13f40 2024-04-23T10:33:44.041 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:33:44.033+0000 7fe88e59c640 1 --2- 172.21.15.114:0/1131674923 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55995da13620 0x55995da13a00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-23T10:33:44.041 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:33:44.034+0000 7fe885d9b640 1 --2- 172.21.15.114:0/1131674923 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x55995da13f40 0x55995da1e2c0 secure :-1 s=READY pgs=78 cs=0 l=1 rev1=1 crypto rx=0x7fe89006a590 tx=0x7fe878009e00 comp rx=0 tx=0).ready entity=mon.2 client_cookie=6084a01c44464757 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T10:33:44.041 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:33:44.034+0000 7fe884d99640 1 -- 172.21.15.114:0/1131674923 <== mon.2 v2:172.21.15.119:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe878017070 con 0x55995da13f40 2024-04-23T10:33:44.041 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:33:44.034+0000 7fe884d99640 1 -- 172.21.15.114:0/1131674923 <== mon.2 v2:172.21.15.119:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fe878013470 con 0x55995da13f40 2024-04-23T10:33:44.041 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:33:44.034+0000 7fe884d99640 1 -- 172.21.15.114:0/1131674923 <== mon.2 v2:172.21.15.119:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe878012630 con 0x55995da13f40 2024-04-23T10:33:44.041 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:33:44.035+0000 7fe8979bf740 1 -- 172.21.15.114:0/1131674923 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x55995da13f40 msgr2=0x55995da1e2c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T10:33:44.041 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:33:44.035+0000 7fe8979bf740 1 --2- 172.21.15.114:0/1131674923 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x55995da13f40 0x55995da1e2c0 secure :-1 s=READY pgs=78 cs=0 l=1 rev1=1 crypto rx=0x7fe89006a590 tx=0x7fe878009e00 comp rx=0 tx=0).stop 2024-04-23T10:33:44.041 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:33:44.035+0000 7fe8979bf740 1 -- 172.21.15.114:0/1131674923 shutdown_connections 2024-04-23T10:33:44.041 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:33:44.035+0000 7fe8979bf740 1 --2- 172.21.15.114:0/1131674923 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x55995da1e800 0x55995da20bf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:33:44.041 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:33:44.035+0000 7fe8979bf740 1 --2- 172.21.15.114:0/1131674923 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x55995da13f40 0x55995da1e2c0 unknown :-1 s=CLOSED pgs=78 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:33:44.041 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:33:44.035+0000 7fe8979bf740 1 --2- 172.21.15.114:0/1131674923 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55995da13620 0x55995da13a00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:33:44.042 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:33:44.035+0000 7fe8979bf740 1 -- 172.21.15.114:0/1131674923 >> 172.21.15.114:0/1131674923 conn(0x55995da0fb30 msgr2=0x55995da0ff10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T10:33:44.042 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:33:44.035+0000 7fe8979bf740 1 -- 172.21.15.114:0/1131674923 shutdown_connections 2024-04-23T10:33:44.042 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:33:44.035+0000 7fe8979bf740 1 -- 172.21.15.114:0/1131674923 wait complete. 2024-04-23T10:33:44.042 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:33:44.035+0000 7fe8979bf740 1 -- start start 2024-04-23T10:33:44.042 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:33:44.035+0000 7fe8979bf740 1 Processor -- start 2024-04-23T10:33:44.042 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:33:44.035+0000 7fe8979bf740 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x55995da13620 0x55995da1e0f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T10:33:44.042 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:33:44.035+0000 7fe8979bf740 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x55995da13f40 0x55995da1b3e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T10:33:44.042 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:33:44.035+0000 7fe8979bf740 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55995da1e800 0x55995da1b920 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T10:33:44.042 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:33:44.035+0000 7fe8979bf740 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x55995da21430 con 0x55995da1e800 2024-04-23T10:33:44.042 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:33:44.035+0000 7fe8979bf740 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 -- 0x55995d9130b0 con 0x55995da13f40 2024-04-23T10:33:44.042 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:33:44.035+0000 7fe8979bf740 1 -- --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_getmap magic: 0 -- 0x55995da215b0 con 0x55995da13620 2024-04-23T10:33:44.042 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:33:44.035+0000 7fe8979bf740 10 client.0 _ll_register_callbacks cb 0x55995d900d30 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-23T10:33:44.042 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:33:44.036+0000 7fe88e59c640 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x55995da13620 0x55995da1e0f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T10:33:44.043 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:33:44.036+0000 7fe88ed9d640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55995da1e800 0x55995da1b920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T10:33:44.043 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:33:44.036+0000 7fe88e59c640 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x55995da13620 0x55995da1e0f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.204:3300/0 says I am v2:172.21.15.114:34830/0 (socket says 172.21.15.114:34830) 2024-04-23T10:33:44.043 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:33:44.036+0000 7fe88e59c640 1 -- 172.21.15.114:0/3253222370 learned_addr learned my addr 172.21.15.114:0/3253222370 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-04-23T10:33:44.043 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:33:44.036+0000 7fe885d9b640 1 --2- 172.21.15.114:0/3253222370 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x55995da13f40 0x55995da1b3e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T10:33:44.043 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:33:44.036+0000 7fe88ed9d640 1 -- 172.21.15.114:0/3253222370 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x55995da13620 msgr2=0x55995da1e0f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T10:33:44.043 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:33:44.036+0000 7fe88ed9d640 1 --2- 172.21.15.114:0/3253222370 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x55995da13620 0x55995da1e0f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:33:44.043 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:33:44.036+0000 7fe88ed9d640 1 -- 172.21.15.114:0/3253222370 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x55995da13f40 msgr2=0x55995da1b3e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T10:33:44.043 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:33:44.036+0000 7fe88ed9d640 1 --2- 172.21.15.114:0/3253222370 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x55995da13f40 0x55995da1b3e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:33:44.043 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:33:44.036+0000 7fe88ed9d640 1 -- 172.21.15.114:0/3253222370 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55995d757620 con 0x55995da1e800 2024-04-23T10:33:44.043 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:33:44.036+0000 7fe885d9b640 1 --2- 172.21.15.114:0/3253222370 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x55995da13f40 0x55995da1b3e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-23T10:33:44.043 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:33:44.036+0000 7fe88ed9d640 1 --2- 172.21.15.114:0/3253222370 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55995da1e800 0x55995da1b920 secure :-1 s=READY pgs=774 cs=0 l=1 rev1=1 crypto rx=0x7fe8880ef410 tx=0x7fe8880bd860 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5bc907ade70ba70b server_cookie=0 in_seq=0 out_seq=0 2024-04-23T10:33:44.043 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:33:44.036+0000 7fe875ffb640 10 client.0 ms_handle_connect on v2:172.21.15.114:3300/0 2024-04-23T10:33:44.043 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:33:44.036+0000 7fe875ffb640 1 -- 172.21.15.114:0/3253222370 <== mon.0 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe8880d6070 con 0x55995da1e800 2024-04-23T10:33:44.044 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:33:44.036+0000 7fe875ffb640 1 -- 172.21.15.114:0/3253222370 <== mon.0 v2:172.21.15.114:3300/0 2 ==== osd_map(96..96 src has 1..96) ==== 8629+0+0 (secure 0 0 0) 0x7fe8880ba2d0 con 0x55995da1e800 2024-04-23T10:33:44.044 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:33:44.036+0000 7fe8979bf740 1 -- 172.21.15.114:0/3253222370 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fe878013470 con 0x55995da1e800 2024-04-23T10:33:44.044 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:33:44.036+0000 7fe875ffb640 1 -- 172.21.15.114:0/3253222370 <== mon.0 v2:172.21.15.114:3300/0 3 ==== mon_get_version_reply(handle=1 version=31) ==== 24+0+0 (secure 0 0 0) 0x7fe8880c2010 con 0x55995da1e800 2024-04-23T10:33:44.044 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:33:44.036+0000 7fe8979bf740 10 client.16530 fetch_fsmap learned FSMap version 31 2024-04-23T10:33:44.044 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:33:44.036+0000 7fe8979bf740 1 -- 172.21.15.114:0/3253222370 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({config=0+,fsmap=31,monmap=4+}) -- 0x55995da21430 con 0x55995da1e800 2024-04-23T10:33:44.044 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:33:44.037+0000 7fe875ffb640 1 -- 172.21.15.114:0/3253222370 <== mon.0 v2:172.21.15.114:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fe878013470 con 0x55995da1e800 2024-04-23T10:33:44.044 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:33:44.037+0000 7fe875ffb640 1 -- 172.21.15.114:0/3253222370 <== mon.0 v2:172.21.15.114:3300/0 5 ==== fsmap(e 31) ==== 8342+0+0 (secure 0 0 0) 0x7fe8880bdec0 con 0x55995da1e800 2024-04-23T10:33:44.044 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:33:44.037+0000 7fe8979bf740 10 client.16530 fetch_fsmap finished waiting for FSMap version 31 2024-04-23T10:33:44.044 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:33:44.037+0000 7fe8979bf740 10 client.16530 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-04-23T10:33:44.044 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:33:44.037+0000 7fe8979bf740 20 client.16530 populate_metadata read hostname 'smithi114' 2024-04-23T10:33:44.044 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:33:44.037+0000 7fe8979bf740 1 --2- 172.21.15.114:0/3253222370 >> [v2:172.21.15.119:6838/937297340,v1:172.21.15.119:6839/937297340] conn(0x55995da2e770 0x55995da30b60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T10:33:44.044 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:33:44.037+0000 7fe8979bf740 4 client.16530 mds_command: new command op to 24545 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-23T10:33:44.044 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:33:44.037+0000 7fe8979bf740 1 -- 172.21.15.114:0/3253222370 --> [v2:172.21.15.119:6838/937297340,v1:172.21.15.119:6839/937297340] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55995d986390 con 0x55995da2e770 2024-04-23T10:33:44.044 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:33:44.037+0000 7fe88e59c640 1 --2- 172.21.15.114:0/3253222370 >> [v2:172.21.15.119:6838/937297340,v1:172.21.15.119:6839/937297340] conn(0x55995da2e770 0x55995da30b60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T10:33:44.044 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:33:44.037+0000 7fe88e59c640 1 --2- 172.21.15.114:0/3253222370 >> [v2:172.21.15.119:6838/937297340,v1:172.21.15.119:6839/937297340] conn(0x55995da2e770 0x55995da30b60 crc :-1 s=READY pgs=124 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=bc2c07c6ecb7e2bb server_cookie=a1d7957619cf494f in_seq=0 out_seq=0 2024-04-23T10:33:44.045 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:33:44.037+0000 7fe875ffb640 10 client.16530 ms_handle_connect on v2:172.21.15.119:6838/937297340 2024-04-23T10:33:44.045 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:33:44.038+0000 7fe875ffb640 1 -- 172.21.15.114:0/3253222370 <== mds.0 v2:172.21.15.119:6838/937297340 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x55995d986390 con 0x55995da2e770 2024-04-23T10:33:44.045 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:33:44.038+0000 7fe875ffb640 10 client.16530 handle_command_reply: tid=0 2024-04-23T10:33:44.045 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:33:44.038+0000 7fe8979bf740 1 client.16530 shutdown 2024-04-23T10:33:44.045 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:33:44.038+0000 7fe8979bf740 1 -- 172.21.15.114:0/3253222370 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55995da1e800 msgr2=0x55995da1b920 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T10:33:44.045 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:33:44.038+0000 7fe8979bf740 1 --2- 172.21.15.114:0/3253222370 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55995da1e800 0x55995da1b920 secure :-1 s=READY pgs=774 cs=0 l=1 rev1=1 crypto rx=0x7fe8880ef410 tx=0x7fe8880bd860 comp rx=0 tx=0).stop 2024-04-23T10:33:44.045 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:33:44.038+0000 7fe8979bf740 1 -- 172.21.15.114:0/3253222370 shutdown_connections 2024-04-23T10:33:44.045 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:33:44.038+0000 7fe8979bf740 1 --2- 172.21.15.114:0/3253222370 >> [v2:172.21.15.119:6838/937297340,v1:172.21.15.119:6839/937297340] conn(0x55995da2e770 0x55995da30b60 crc :-1 s=READY pgs=124 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:33:44.045 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:33:44.038+0000 7fe8979bf740 1 --2- 172.21.15.114:0/3253222370 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55995da1e800 0x55995da1b920 unknown :-1 s=CLOSED pgs=774 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:33:44.045 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:33:44.038+0000 7fe8979bf740 1 --2- 172.21.15.114:0/3253222370 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x55995da13f40 0x55995da1b3e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:33:44.045 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:33:44.038+0000 7fe875ffb640 0 client.16530 ms_handle_reset on v2:172.21.15.119:6838/937297340 2024-04-23T10:33:44.045 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:33:44.038+0000 7fe8979bf740 1 --2- 172.21.15.114:0/3253222370 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x55995da13620 0x55995da1e0f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:33:44.045 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:33:44.038+0000 7fe8979bf740 1 -- 172.21.15.114:0/3253222370 >> 172.21.15.114:0/3253222370 conn(0x55995da0fb30 msgr2=0x55995da158b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T10:33:44.045 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:33:44.038+0000 7fe8979bf740 1 -- 172.21.15.114:0/3253222370 shutdown_connections 2024-04-23T10:33:44.045 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:33:44.038+0000 7fe8979bf740 1 -- 172.21.15.114:0/3253222370 wait complete. 2024-04-23T10:33:44.046 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:33:44.038+0000 7fe8979bf740 20 client.16530 trim_cache size 0 max 16384 2024-04-23T10:33:44.057 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:33:44.053+0000 7fe8979bf740 1 Processor -- start 2024-04-23T10:33:44.057 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:33:44.053+0000 7fe8979bf740 1 -- start start 2024-04-23T10:33:44.057 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:33:44.054+0000 7fe8979bf740 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x55995da0fb30 0x55995da2a2f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T10:33:44.057 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:33:44.054+0000 7fe8979bf740 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55995da13620 0x55995da1dc20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T10:33:44.057 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:33:44.054+0000 7fe8979bf740 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x55995da13f40 0x55995da36470 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T10:33:44.058 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:33:44.054+0000 7fe8979bf740 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x7fe8901ae400 con 0x55995da13620 2024-04-23T10:33:44.058 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:33:44.054+0000 7fe8979bf740 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 -- 0x7fe8901abca0 con 0x55995da13f40 2024-04-23T10:33:44.058 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:33:44.054+0000 7fe8979bf740 1 -- --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_getmap magic: 0 -- 0x7fe8901a9b10 con 0x55995da0fb30 2024-04-23T10:33:44.058 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:33:44.054+0000 7fe885d9b640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55995da13620 0x55995da1dc20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T10:33:44.058 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:33:44.054+0000 7fe885d9b640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55995da13620 0x55995da1dc20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.114:58014/0 (socket says 172.21.15.114:58014) 2024-04-23T10:33:44.058 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:33:44.054+0000 7fe885d9b640 1 -- 172.21.15.114:0/3697665689 learned_addr learned my addr 172.21.15.114:0/3697665689 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-04-23T10:33:44.058 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:33:44.054+0000 7fe88e59c640 1 --2- 172.21.15.114:0/3697665689 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x55995da0fb30 0x55995da2a2f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T10:33:44.058 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:33:44.054+0000 7fe88ed9d640 1 --2- 172.21.15.114:0/3697665689 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x55995da13f40 0x55995da36470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T10:33:44.058 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:33:44.054+0000 7fe885d9b640 1 -- 172.21.15.114:0/3697665689 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x55995da0fb30 msgr2=0x55995da2a2f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T10:33:44.058 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:33:44.054+0000 7fe885d9b640 1 --2- 172.21.15.114:0/3697665689 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x55995da0fb30 0x55995da2a2f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:33:44.058 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:33:44.054+0000 7fe885d9b640 1 -- 172.21.15.114:0/3697665689 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x55995da13f40 msgr2=0x55995da36470 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T10:33:44.058 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:33:44.054+0000 7fe885d9b640 1 --2- 172.21.15.114:0/3697665689 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x55995da13f40 0x55995da36470 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:33:44.058 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:33:44.054+0000 7fe885d9b640 1 -- 172.21.15.114:0/3697665689 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55995da311e0 con 0x55995da13620 2024-04-23T10:33:44.059 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:33:44.054+0000 7fe885d9b640 1 --2- 172.21.15.114:0/3697665689 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55995da13620 0x55995da1dc20 secure :-1 s=READY pgs=775 cs=0 l=1 rev1=1 crypto rx=0x7fe878013ce0 tx=0x7fe878013d10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f07c5af9a683c16f server_cookie=0 in_seq=0 out_seq=0 2024-04-23T10:33:44.059 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:33:44.054+0000 7fe877fff640 1 -- 172.21.15.114:0/3697665689 <== mon.0 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe878017040 con 0x55995da13620 2024-04-23T10:33:44.059 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:33:44.054+0000 7fe877fff640 1 -- 172.21.15.114:0/3697665689 <== mon.0 v2:172.21.15.114:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fe878013470 con 0x55995da13620 2024-04-23T10:33:44.059 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:33:44.054+0000 7fe877fff640 1 -- 172.21.15.114:0/3697665689 <== mon.0 v2:172.21.15.114:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe87801c950 con 0x55995da13620 2024-04-23T10:33:44.059 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:33:44.055+0000 7fe8979bf740 1 -- 172.21.15.114:0/3697665689 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55995da13620 msgr2=0x55995da1dc20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T10:33:44.059 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:33:44.055+0000 7fe8979bf740 1 --2- 172.21.15.114:0/3697665689 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55995da13620 0x55995da1dc20 secure :-1 s=READY pgs=775 cs=0 l=1 rev1=1 crypto rx=0x7fe878013ce0 tx=0x7fe878013d10 comp rx=0 tx=0).stop 2024-04-23T10:33:44.059 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:33:44.055+0000 7fe8979bf740 1 -- 172.21.15.114:0/3697665689 shutdown_connections 2024-04-23T10:33:44.059 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:33:44.055+0000 7fe8979bf740 1 --2- 172.21.15.114:0/3697665689 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x55995da13f40 0x55995da36470 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:33:44.059 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:33:44.055+0000 7fe8979bf740 1 --2- 172.21.15.114:0/3697665689 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55995da13620 0x55995da1dc20 unknown :-1 s=CLOSED pgs=775 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:33:44.059 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:33:44.055+0000 7fe8979bf740 1 --2- 172.21.15.114:0/3697665689 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x55995da0fb30 0x55995da2a2f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:33:44.059 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:33:44.055+0000 7fe8979bf740 1 -- 172.21.15.114:0/3697665689 >> 172.21.15.114:0/3697665689 conn(0x55995da1ba80 msgr2=0x55995da169d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T10:33:44.059 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:33:44.055+0000 7fe8979bf740 1 -- 172.21.15.114:0/3697665689 shutdown_connections 2024-04-23T10:33:44.059 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:33:44.055+0000 7fe8979bf740 1 -- 172.21.15.114:0/3697665689 wait complete. 2024-04-23T10:33:44.060 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:33:44.055+0000 7fe8979bf740 1 -- start start 2024-04-23T10:33:44.060 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:33:44.055+0000 7fe8979bf740 1 Processor -- start 2024-04-23T10:33:44.060 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:33:44.056+0000 7fe8979bf740 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x55995da0fb30 0x55995da18730 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T10:33:44.060 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:33:44.056+0000 7fe8979bf740 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55995da13620 0x55995da18c70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T10:33:44.060 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:33:44.056+0000 7fe8979bf740 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x55995da13f40 0x55995da93ce0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T10:33:44.060 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:33:44.056+0000 7fe8979bf740 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x7fe8901aa160 con 0x55995da13620 2024-04-23T10:33:44.060 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:33:44.056+0000 7fe8979bf740 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 -- 0x7fe8901ab5d0 con 0x55995da13f40 2024-04-23T10:33:44.060 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:33:44.056+0000 7fe8979bf740 1 -- --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_getmap magic: 0 -- 0x7fe8901aa830 con 0x55995da0fb30 2024-04-23T10:33:44.060 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:33:44.056+0000 7fe8979bf740 10 client.0 _ll_register_callbacks cb 0x7fe8880bda30 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-23T10:33:44.060 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:33:44.056+0000 7fe885d9b640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55995da13620 0x55995da18c70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T10:33:44.060 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:33:44.056+0000 7fe885d9b640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55995da13620 0x55995da18c70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.114:58020/0 (socket says 172.21.15.114:58020) 2024-04-23T10:33:44.060 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:33:44.056+0000 7fe885d9b640 1 -- 172.21.15.114:0/2436082216 learned_addr learned my addr 172.21.15.114:0/2436082216 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-04-23T10:33:44.060 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:33:44.056+0000 7fe88e59c640 1 --2- 172.21.15.114:0/2436082216 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x55995da0fb30 0x55995da18730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T10:33:44.060 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:33:44.056+0000 7fe88ed9d640 1 --2- 172.21.15.114:0/2436082216 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x55995da13f40 0x55995da93ce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T10:33:44.061 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:33:44.056+0000 7fe885d9b640 1 -- 172.21.15.114:0/2436082216 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x55995da0fb30 msgr2=0x55995da18730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T10:33:44.061 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:33:44.056+0000 7fe885d9b640 1 --2- 172.21.15.114:0/2436082216 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x55995da0fb30 0x55995da18730 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:33:44.061 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:33:44.056+0000 7fe885d9b640 1 -- 172.21.15.114:0/2436082216 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x55995da13f40 msgr2=0x55995da93ce0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T10:33:44.061 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:33:44.056+0000 7fe885d9b640 1 --2- 172.21.15.114:0/2436082216 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x55995da13f40 0x55995da93ce0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:33:44.061 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:33:44.056+0000 7fe885d9b640 1 -- 172.21.15.114:0/2436082216 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55995da149f0 con 0x55995da13620 2024-04-23T10:33:44.061 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:33:44.056+0000 7fe885d9b640 1 --2- 172.21.15.114:0/2436082216 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55995da13620 0x55995da18c70 secure :-1 s=READY pgs=776 cs=0 l=1 rev1=1 crypto rx=0x7fe878008010 tx=0x7fe87800ecb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c14651f9da49b55c server_cookie=0 in_seq=0 out_seq=0 2024-04-23T10:33:44.061 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:33:44.056+0000 7fe875ffb640 10 client.0 ms_handle_connect on v2:172.21.15.114:3300/0 2024-04-23T10:33:44.061 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:33:44.056+0000 7fe875ffb640 1 -- 172.21.15.114:0/2436082216 <== mon.0 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe878017040 con 0x55995da13620 2024-04-23T10:33:44.061 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:33:44.056+0000 7fe875ffb640 1 -- 172.21.15.114:0/2436082216 <== mon.0 v2:172.21.15.114:3300/0 2 ==== osd_map(96..96 src has 1..96) ==== 8629+0+0 (secure 0 0 0) 0x7fe87801c7e0 con 0x55995da13620 2024-04-23T10:33:44.061 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:33:44.056+0000 7fe8979bf740 1 -- 172.21.15.114:0/2436082216 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fe878013470 con 0x55995da13620 2024-04-23T10:33:44.061 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:33:44.056+0000 7fe875ffb640 1 -- 172.21.15.114:0/2436082216 <== mon.0 v2:172.21.15.114:3300/0 3 ==== mon_get_version_reply(handle=1 version=31) ==== 24+0+0 (secure 0 0 0) 0x7fe87801ca60 con 0x55995da13620 2024-04-23T10:33:44.061 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:33:44.057+0000 7fe8979bf740 10 client.16536 fetch_fsmap learned FSMap version 31 2024-04-23T10:33:44.061 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:33:44.057+0000 7fe8979bf740 1 -- 172.21.15.114:0/2436082216 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({config=0+,fsmap=31,monmap=4+}) -- 0x55995da952c0 con 0x55995da13620 2024-04-23T10:33:44.062 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:33:44.057+0000 7fe875ffb640 1 -- 172.21.15.114:0/2436082216 <== mon.0 v2:172.21.15.114:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fe878013470 con 0x55995da13620 2024-04-23T10:33:44.062 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:33:44.057+0000 7fe875ffb640 1 -- 172.21.15.114:0/2436082216 <== mon.0 v2:172.21.15.114:3300/0 5 ==== fsmap(e 31) ==== 8342+0+0 (secure 0 0 0) 0x7fe878030c40 con 0x55995da13620 2024-04-23T10:33:44.062 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:33:44.057+0000 7fe8979bf740 10 client.16536 fetch_fsmap finished waiting for FSMap version 31 2024-04-23T10:33:44.062 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:33:44.057+0000 7fe8979bf740 10 client.16536 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-04-23T10:33:44.062 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:33:44.057+0000 7fe8979bf740 20 client.16536 populate_metadata read hostname 'smithi114' 2024-04-23T10:33:44.062 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:33:44.057+0000 7fe8979bf740 1 --2- 172.21.15.114:0/2436082216 >> [v2:172.21.15.119:6838/937297340,v1:172.21.15.119:6839/937297340] conn(0x55995da1e800 0x55995da975b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T10:33:44.062 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:33:44.057+0000 7fe8979bf740 4 client.16536 mds_command: new command op to 24545 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-04-23T10:33:44.062 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:33:44.057+0000 7fe8979bf740 1 -- 172.21.15.114:0/2436082216 --> [v2:172.21.15.119:6838/937297340,v1:172.21.15.119:6839/937297340] -- command(tid 0: {"prefix": "flush journal"}) -- 0x55995d980200 con 0x55995da1e800 2024-04-23T10:33:44.062 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:33:44.057+0000 7fe88e59c640 1 --2- 172.21.15.114:0/2436082216 >> [v2:172.21.15.119:6838/937297340,v1:172.21.15.119:6839/937297340] conn(0x55995da1e800 0x55995da975b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T10:33:44.062 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:33:44.057+0000 7fe875ffb640 10 client.16536 ms_handle_connect on v2:172.21.15.119:6838/937297340 2024-04-23T10:33:44.062 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:33:44.057+0000 7fe88e59c640 1 --2- 172.21.15.114:0/2436082216 >> [v2:172.21.15.119:6838/937297340,v1:172.21.15.119:6839/937297340] conn(0x55995da1e800 0x55995da975b0 crc :-1 s=READY pgs=125 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=ec78656876b58e67 server_cookie=9cab9945190b4ad7 in_seq=0 out_seq=0 2024-04-23T10:33:44.113 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:33:44.114+0000 7fe875ffb640 1 -- 172.21.15.114:0/2436082216 <== mds.0 v2:172.21.15.119:6838/937297340 1 ==== command_reply(tid 0: 0 ) ==== 8+0+44 (crc 0 0 0) 0x55995d980200 con 0x55995da1e800 2024-04-23T10:33:44.113 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:33:44.114+0000 7fe875ffb640 10 client.16536 handle_command_reply: tid=0 2024-04-23T10:33:44.113 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:33:44.114+0000 7fe8979bf740 1 client.16536 shutdown 2024-04-23T10:33:44.113 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:33:44.114+0000 7fe8979bf740 1 -- 172.21.15.114:0/2436082216 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55995da13620 msgr2=0x55995da18c70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T10:33:44.113 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:33:44.114+0000 7fe8979bf740 1 --2- 172.21.15.114:0/2436082216 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55995da13620 0x55995da18c70 secure :-1 s=READY pgs=776 cs=0 l=1 rev1=1 crypto rx=0x7fe878008010 tx=0x7fe87800ecb0 comp rx=0 tx=0).stop 2024-04-23T10:33:44.114 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:33:44.114+0000 7fe8979bf740 1 -- 172.21.15.114:0/2436082216 shutdown_connections 2024-04-23T10:33:44.114 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:33:44.114+0000 7fe8979bf740 1 --2- 172.21.15.114:0/2436082216 >> [v2:172.21.15.119:6838/937297340,v1:172.21.15.119:6839/937297340] conn(0x55995da1e800 0x55995da975b0 crc :-1 s=READY pgs=125 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:33:44.114 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:33:44.114+0000 7fe8979bf740 1 --2- 172.21.15.114:0/2436082216 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x55995da13f40 0x55995da93ce0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:33:44.114 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:33:44.114+0000 7fe8979bf740 1 --2- 172.21.15.114:0/2436082216 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55995da13620 0x55995da18c70 unknown :-1 s=CLOSED pgs=776 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:33:44.114 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:33:44.114+0000 7fe8979bf740 1 --2- 172.21.15.114:0/2436082216 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x55995da0fb30 0x55995da18730 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:33:44.114 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:33:44.114+0000 7fe8979bf740 1 -- 172.21.15.114:0/2436082216 >> 172.21.15.114:0/2436082216 conn(0x55995da1ba80 msgr2=0x55995da16120 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T10:33:44.114 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:33:44.114+0000 7fe875ffb640 0 client.16536 ms_handle_reset on v2:172.21.15.119:6838/937297340 2024-04-23T10:33:44.114 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:33:44.114+0000 7fe8979bf740 1 -- 172.21.15.114:0/2436082216 shutdown_connections 2024-04-23T10:33:44.114 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:33:44.114+0000 7fe8979bf740 1 -- 172.21.15.114:0/2436082216 wait complete. 2024-04-23T10:33:44.114 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:33:44.114+0000 7fe8979bf740 20 client.16536 trim_cache size 0 max 16384 2024-04-23T10:33:44.115 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stdout:{ 2024-04-23T10:33:44.115 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stdout: "message": "", 2024-04-23T10:33:44.115 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stdout: "return_code": 0 2024-04-23T10:33:44.115 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stdout:} 2024-04-23T10:33:44.116 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:33:44.115+0000 7fe894ca7640 1 -- 172.21.15.114:0/849372087 >> [v2:172.21.15.114:6800/1395693466,v1:172.21.15.114:6801/1395693466] conn(0x7fe87c0a3dc0 msgr2=0x7fe87c0a6280 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T10:33:44.116 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:33:44.115+0000 7fe894ca7640 1 --2- 172.21.15.114:0/849372087 >> [v2:172.21.15.114:6800/1395693466,v1:172.21.15.114:6801/1395693466] conn(0x7fe87c0a3dc0 0x7fe87c0a6280 secure :-1 s=READY pgs=116 cs=0 l=1 rev1=1 crypto rx=0x7fe880009fc0 tx=0x7fe880005f90 comp rx=0 tx=0).stop 2024-04-23T10:33:44.116 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:33:44.115+0000 7fe894ca7640 1 -- 172.21.15.114:0/849372087 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fe89019cf40 msgr2=0x7fe89019d3f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T10:33:44.116 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:33:44.115+0000 7fe894ca7640 1 --2- 172.21.15.114:0/849372087 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fe89019cf40 0x7fe89019d3f0 secure :-1 s=READY pgs=773 cs=0 l=1 rev1=1 crypto rx=0x7fe8880098d0 tx=0x7fe8880317b0 comp rx=0 tx=0).stop 2024-04-23T10:33:44.116 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:33:44.115+0000 7fe894ca7640 1 -- 172.21.15.114:0/849372087 shutdown_connections 2024-04-23T10:33:44.116 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:33:44.115+0000 7fe894ca7640 1 --2- 172.21.15.114:0/849372087 >> [v2:172.21.15.114:6800/1395693466,v1:172.21.15.114:6801/1395693466] conn(0x7fe87c0a3dc0 0x7fe87c0a6280 unknown :-1 s=CLOSED pgs=116 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:33:44.116 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:33:44.115+0000 7fe894ca7640 1 --2- 172.21.15.114:0/849372087 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fe89019cf40 0x7fe89019d3f0 unknown :-1 s=CLOSED pgs=773 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:33:44.116 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:33:44.115+0000 7fe894ca7640 1 --2- 172.21.15.114:0/849372087 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fe890107280 0x7fe890198bc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:33:44.116 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:33:44.115+0000 7fe894ca7640 1 --2- 172.21.15.114:0/849372087 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fe89006a9a0 0x7fe890198680 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:33:44.116 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:33:44.115+0000 7fe894ca7640 1 -- 172.21.15.114:0/849372087 >> 172.21.15.114:0/849372087 conn(0x7fe890077c30 msgr2=0x7fe8900ff1b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T10:33:44.116 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:33:44.116+0000 7fe894ca7640 1 -- 172.21.15.114:0/849372087 shutdown_connections 2024-04-23T10:33:44.116 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:33:44.116+0000 7fe894ca7640 1 -- 172.21.15.114:0/849372087 wait complete. 2024-04-23T10:33:44.927 INFO:tasks.workunit.client.0.smithi114.stdout: 1 458435 6.50 MB/sec execute 233 sec latency 94.499 ms 2024-04-23T10:33:45.742 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 23 10:33:45 smithi119 ceph-mon[32608]: pgmap v221: 97 pgs: 97 active+clean; 138 MiB data, 1.7 GiB used, 1.0 TiB / 1.0 TiB avail; 2.8 MiB/s rd, 2.7 MiB/s wr, 225 op/s 2024-04-23T10:33:45.753 INFO:journalctl@ceph.mon.a.smithi114.stdout:Apr 23 10:33:45 smithi114 ceph-mon[29255]: pgmap v221: 97 pgs: 97 active+clean; 138 MiB data, 1.7 GiB used, 1.0 TiB / 1.0 TiB avail; 2.8 MiB/s rd, 2.7 MiB/s wr, 225 op/s 2024-04-23T10:33:45.827 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 23 10:33:45 smithi204 ceph-mon[32649]: pgmap v221: 97 pgs: 97 active+clean; 138 MiB data, 1.7 GiB used, 1.0 TiB / 1.0 TiB avail; 2.8 MiB/s rd, 2.7 MiB/s wr, 225 op/s 2024-04-23T10:33:45.927 INFO:tasks.workunit.client.0.smithi114.stdout: 1 460164 6.57 MB/sec execute 234 sec latency 16.422 ms 2024-04-23T10:33:46.927 INFO:tasks.workunit.client.0.smithi114.stdout: 1 462349 6.60 MB/sec execute 235 sec latency 17.509 ms 2024-04-23T10:33:47.742 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 23 10:33:47 smithi119 ceph-mon[32608]: pgmap v222: 97 pgs: 97 active+clean; 132 MiB data, 1.7 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 MiB/s rd, 4.1 MiB/s wr, 363 op/s 2024-04-23T10:33:47.753 INFO:journalctl@ceph.mon.a.smithi114.stdout:Apr 23 10:33:47 smithi114 ceph-mon[29255]: pgmap v222: 97 pgs: 97 active+clean; 132 MiB data, 1.7 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 MiB/s rd, 4.1 MiB/s wr, 363 op/s 2024-04-23T10:33:47.827 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 23 10:33:47 smithi204 ceph-mon[32649]: pgmap v222: 97 pgs: 97 active+clean; 132 MiB data, 1.7 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 MiB/s rd, 4.1 MiB/s wr, 363 op/s 2024-04-23T10:33:47.927 INFO:tasks.workunit.client.0.smithi114.stdout: 1 464664 6.62 MB/sec execute 236 sec latency 20.540 ms 2024-04-23T10:33:48.927 INFO:tasks.workunit.client.0.smithi114.stdout: 1 466167 6.62 MB/sec execute 237 sec latency 59.611 ms 2024-04-23T10:33:49.742 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 23 10:33:49 smithi119 ceph-mon[32608]: pgmap v223: 97 pgs: 97 active+clean; 132 MiB data, 1.7 GiB used, 1.0 TiB / 1.0 TiB avail; 2.4 MiB/s rd, 2.8 MiB/s wr, 253 op/s 2024-04-23T10:33:49.754 INFO:journalctl@ceph.mon.a.smithi114.stdout:Apr 23 10:33:49 smithi114 ceph-mon[29255]: pgmap v223: 97 pgs: 97 active+clean; 132 MiB data, 1.7 GiB used, 1.0 TiB / 1.0 TiB avail; 2.4 MiB/s rd, 2.8 MiB/s wr, 253 op/s 2024-04-23T10:33:49.827 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 23 10:33:49 smithi204 ceph-mon[32649]: pgmap v223: 97 pgs: 97 active+clean; 132 MiB data, 1.7 GiB used, 1.0 TiB / 1.0 TiB avail; 2.4 MiB/s rd, 2.8 MiB/s wr, 253 op/s 2024-04-23T10:33:49.927 INFO:tasks.workunit.client.0.smithi114.stdout: 1 468655 6.65 MB/sec execute 238 sec latency 18.887 ms 2024-04-23T10:33:50.927 INFO:tasks.workunit.client.0.smithi114.stdout: 1 470252 6.66 MB/sec execute 239 sec latency 19.234 ms 2024-04-23T10:33:51.742 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 23 10:33:51 smithi119 ceph-mon[32608]: pgmap v224: 97 pgs: 97 active+clean; 132 MiB data, 1.7 GiB used, 1.0 TiB / 1.0 TiB avail; 2.4 MiB/s rd, 2.8 MiB/s wr, 253 op/s 2024-04-23T10:33:51.742 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 23 10:33:51 smithi119 ceph-mon[32608]: from='mgr.15291 172.21.15.114:0/1095789187' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-23T10:33:51.753 INFO:journalctl@ceph.mon.a.smithi114.stdout:Apr 23 10:33:51 smithi114 ceph-mon[29255]: pgmap v224: 97 pgs: 97 active+clean; 132 MiB data, 1.7 GiB used, 1.0 TiB / 1.0 TiB avail; 2.4 MiB/s rd, 2.8 MiB/s wr, 253 op/s 2024-04-23T10:33:51.754 INFO:journalctl@ceph.mon.a.smithi114.stdout:Apr 23 10:33:51 smithi114 ceph-mon[29255]: from='mgr.15291 172.21.15.114:0/1095789187' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-23T10:33:51.827 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 23 10:33:51 smithi204 ceph-mon[32649]: pgmap v224: 97 pgs: 97 active+clean; 132 MiB data, 1.7 GiB used, 1.0 TiB / 1.0 TiB avail; 2.4 MiB/s rd, 2.8 MiB/s wr, 253 op/s 2024-04-23T10:33:51.827 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 23 10:33:51 smithi204 ceph-mon[32649]: from='mgr.15291 172.21.15.114:0/1095789187' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-23T10:33:51.927 INFO:tasks.workunit.client.0.smithi114.stdout: 1 472787 6.70 MB/sec execute 240 sec latency 18.334 ms 2024-04-23T10:33:52.742 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 23 10:33:52 smithi119 ceph-mon[32608]: from='mgr.15291 172.21.15.114:0/1095789187' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-23T10:33:52.742 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 23 10:33:52 smithi119 ceph-mon[32608]: from='mgr.15291 172.21.15.114:0/1095789187' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-23T10:33:52.742 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 23 10:33:52 smithi119 ceph-mon[32608]: from='mgr.15291 172.21.15.114:0/1095789187' entity='mgr.x' 2024-04-23T10:33:52.753 INFO:journalctl@ceph.mon.a.smithi114.stdout:Apr 23 10:33:52 smithi114 ceph-mon[29255]: from='mgr.15291 172.21.15.114:0/1095789187' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-23T10:33:52.754 INFO:journalctl@ceph.mon.a.smithi114.stdout:Apr 23 10:33:52 smithi114 ceph-mon[29255]: from='mgr.15291 172.21.15.114:0/1095789187' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-23T10:33:52.754 INFO:journalctl@ceph.mon.a.smithi114.stdout:Apr 23 10:33:52 smithi114 ceph-mon[29255]: from='mgr.15291 172.21.15.114:0/1095789187' entity='mgr.x' 2024-04-23T10:33:52.828 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 23 10:33:52 smithi204 ceph-mon[32649]: from='mgr.15291 172.21.15.114:0/1095789187' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-23T10:33:52.828 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 23 10:33:52 smithi204 ceph-mon[32649]: from='mgr.15291 172.21.15.114:0/1095789187' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-23T10:33:52.828 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 23 10:33:52 smithi204 ceph-mon[32649]: from='mgr.15291 172.21.15.114:0/1095789187' entity='mgr.x' 2024-04-23T10:33:52.927 INFO:tasks.workunit.client.0.smithi114.stdout: 1 474315 6.69 MB/sec execute 241 sec latency 14.843 ms 2024-04-23T10:33:53.927 INFO:tasks.workunit.client.0.smithi114.stdout: 1 476533 6.72 MB/sec execute 242 sec latency 33.262 ms 2024-04-23T10:33:54.077 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 23 10:33:53 smithi204 ceph-mon[32649]: pgmap v225: 97 pgs: 97 active+clean; 140 MiB data, 1.8 GiB used, 1.0 TiB / 1.0 TiB avail; 3.0 MiB/s rd, 4.3 MiB/s wr, 371 op/s 2024-04-23T10:33:54.241 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 23 10:33:53 smithi119 ceph-mon[32608]: pgmap v225: 97 pgs: 97 active+clean; 140 MiB data, 1.8 GiB used, 1.0 TiB / 1.0 TiB avail; 3.0 MiB/s rd, 4.3 MiB/s wr, 371 op/s 2024-04-23T10:33:54.254 INFO:journalctl@ceph.mon.a.smithi114.stdout:Apr 23 10:33:53 smithi114 ceph-mon[29255]: pgmap v225: 97 pgs: 97 active+clean; 140 MiB data, 1.8 GiB used, 1.0 TiB / 1.0 TiB avail; 3.0 MiB/s rd, 4.3 MiB/s wr, 371 op/s 2024-04-23T10:33:54.927 INFO:tasks.workunit.client.0.smithi114.stdout: 1 478545 6.72 MB/sec execute 243 sec latency 53.540 ms 2024-04-23T10:33:55.928 INFO:tasks.workunit.client.0.smithi114.stdout: 1 480275 6.75 MB/sec execute 244 sec latency 23.744 ms 2024-04-23T10:33:56.078 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 23 10:33:55 smithi204 ceph-mon[32649]: pgmap v226: 97 pgs: 97 active+clean; 140 MiB data, 1.8 GiB used, 1.0 TiB / 1.0 TiB avail; 1.5 MiB/s rd, 3.0 MiB/s wr, 256 op/s 2024-04-23T10:33:56.241 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 23 10:33:55 smithi119 ceph-mon[32608]: pgmap v226: 97 pgs: 97 active+clean; 140 MiB data, 1.8 GiB used, 1.0 TiB / 1.0 TiB avail; 1.5 MiB/s rd, 3.0 MiB/s wr, 256 op/s 2024-04-23T10:33:56.254 INFO:journalctl@ceph.mon.a.smithi114.stdout:Apr 23 10:33:55 smithi114 ceph-mon[29255]: pgmap v226: 97 pgs: 97 active+clean; 140 MiB data, 1.8 GiB used, 1.0 TiB / 1.0 TiB avail; 1.5 MiB/s rd, 3.0 MiB/s wr, 256 op/s 2024-04-23T10:33:56.928 INFO:tasks.workunit.client.0.smithi114.stdout: 1 482669 6.77 MB/sec execute 245 sec latency 29.567 ms 2024-04-23T10:33:57.209 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:33:57.208+0000 7f81459d8640 1 -- 172.21.15.114:0/3196349466 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f8140100450 msgr2=0x7f81401008d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T10:33:57.209 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:33:57.208+0000 7f81459d8640 1 --2- 172.21.15.114:0/3196349466 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f8140100450 0x7f81401008d0 secure :-1 s=READY pgs=79 cs=0 l=1 rev1=1 crypto rx=0x7f8128009860 tx=0x7f812802a340 comp rx=0 tx=0).stop 2024-04-23T10:33:57.209 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:33:57.208+0000 7f81459d8640 1 -- 172.21.15.114:0/3196349466 shutdown_connections 2024-04-23T10:33:57.210 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:33:57.208+0000 7f81459d8640 1 --2- 172.21.15.114:0/3196349466 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f8140100e10 0x7f814010ede0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:33:57.210 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:33:57.208+0000 7f81459d8640 1 --2- 172.21.15.114:0/3196349466 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f8140100450 0x7f81401008d0 unknown :-1 s=CLOSED pgs=79 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:33:57.210 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:33:57.208+0000 7f81459d8640 1 --2- 172.21.15.114:0/3196349466 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f8140106f30 0x7f8140107310 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:33:57.210 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:33:57.208+0000 7f81459d8640 1 -- 172.21.15.114:0/3196349466 >> 172.21.15.114:0/3196349466 conn(0x7f81400773e0 msgr2=0x7f81400fe760 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T10:33:57.210 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:33:57.208+0000 7f81459d8640 1 -- 172.21.15.114:0/3196349466 shutdown_connections 2024-04-23T10:33:57.210 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:33:57.208+0000 7f81459d8640 1 -- 172.21.15.114:0/3196349466 wait complete. 2024-04-23T10:33:57.210 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:33:57.209+0000 7f81459d8640 1 Processor -- start 2024-04-23T10:33:57.210 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:33:57.209+0000 7f81459d8640 1 -- start start 2024-04-23T10:33:57.210 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:33:57.209+0000 7f81459d8640 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f8140100450 0x7f8140198730 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T10:33:57.210 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:33:57.209+0000 7f81459d8640 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f8140100e10 0x7f8140198c70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T10:33:57.210 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:33:57.209+0000 7f81459d8640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f8140106f30 0x7f814019cfa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T10:33:57.210 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:33:57.209+0000 7f81459d8640 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x7f8140112270 con 0x7f8140106f30 2024-04-23T10:33:57.210 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:33:57.209+0000 7f81459d8640 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 -- 0x7f81401120f0 con 0x7f8140100e10 2024-04-23T10:33:57.211 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:33:57.209+0000 7f81459d8640 1 -- --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_getmap magic: 0 -- 0x7f81401123f0 con 0x7f8140100450 2024-04-23T10:33:57.211 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:33:57.209+0000 7f8137fff640 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f8140100e10 0x7f8140198c70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T10:33:57.211 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:33:57.209+0000 7f813f7fe640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f8140106f30 0x7f814019cfa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T10:33:57.211 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:33:57.209+0000 7f8137fff640 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f8140100e10 0x7f8140198c70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.119:3300/0 says I am v2:172.21.15.114:60268/0 (socket says 172.21.15.114:60268) 2024-04-23T10:33:57.211 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:33:57.209+0000 7f8137fff640 1 -- 172.21.15.114:0/3277871574 learned_addr learned my addr 172.21.15.114:0/3277871574 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-04-23T10:33:57.211 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:33:57.209+0000 7f813effd640 1 --2- 172.21.15.114:0/3277871574 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f8140100450 0x7f8140198730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T10:33:57.211 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:33:57.210+0000 7f8137fff640 1 -- 172.21.15.114:0/3277871574 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f8140100450 msgr2=0x7f8140198730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T10:33:57.211 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:33:57.210+0000 7f8137fff640 1 --2- 172.21.15.114:0/3277871574 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f8140100450 0x7f8140198730 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:33:57.211 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:33:57.210+0000 7f8137fff640 1 -- 172.21.15.114:0/3277871574 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f8140106f30 msgr2=0x7f814019cfa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T10:33:57.211 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:33:57.210+0000 7f8137fff640 1 --2- 172.21.15.114:0/3277871574 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f8140106f30 0x7f814019cfa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:33:57.211 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:33:57.210+0000 7f8137fff640 1 -- 172.21.15.114:0/3277871574 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f814019d6d0 con 0x7f8140100e10 2024-04-23T10:33:57.211 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:33:57.210+0000 7f813effd640 1 --2- 172.21.15.114:0/3277871574 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f8140100450 0x7f8140198730 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-23T10:33:57.211 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:33:57.210+0000 7f813f7fe640 1 --2- 172.21.15.114:0/3277871574 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f8140106f30 0x7f814019cfa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-23T10:33:57.211 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:33:57.210+0000 7f8137fff640 1 --2- 172.21.15.114:0/3277871574 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f8140100e10 0x7f8140198c70 secure :-1 s=READY pgs=80 cs=0 l=1 rev1=1 crypto rx=0x7f8128009980 tx=0x7f8128032dc0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T10:33:57.212 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:33:57.210+0000 7f81449d6640 1 -- 172.21.15.114:0/3277871574 <== mon.2 v2:172.21.15.119:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8128040020 con 0x7f8140100e10 2024-04-23T10:33:57.212 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:33:57.210+0000 7f81459d8640 1 -- 172.21.15.114:0/3277871574 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f814019d960 con 0x7f8140100e10 2024-04-23T10:33:57.212 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:33:57.210+0000 7f81449d6640 1 -- 172.21.15.114:0/3277871574 <== mon.2 v2:172.21.15.119:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f8128030dc0 con 0x7f8140100e10 2024-04-23T10:33:57.212 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:33:57.210+0000 7f81449d6640 1 -- 172.21.15.114:0/3277871574 <== mon.2 v2:172.21.15.119:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8128037040 con 0x7f8140100e10 2024-04-23T10:33:57.212 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:33:57.210+0000 7f81459d8640 1 -- 172.21.15.114:0/3277871574 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f81401a5250 con 0x7f8140100e10 2024-04-23T10:33:57.218 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:33:57.212+0000 7f81487c6740 1 Processor -- start 2024-04-23T10:33:57.218 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:33:57.212+0000 7f81449d6640 1 -- 172.21.15.114:0/3277871574 <== mon.2 v2:172.21.15.119:3300/0 4 ==== mgrmap(e 26) ==== 138778+0+0 (secure 0 0 0) 0x7f8128043020 con 0x7f8140100e10 2024-04-23T10:33:57.218 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:33:57.212+0000 7f81487c6740 1 -- start start 2024-04-23T10:33:57.218 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:33:57.213+0000 7f81487c6740 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x564f5efa6620 0x564f5efa6a00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T10:33:57.219 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:33:57.213+0000 7f81487c6740 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x564f5efa6f40 0x564f5efb12c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T10:33:57.219 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:33:57.213+0000 7f81487c6740 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x564f5efb1800 0x564f5efb3bf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T10:33:57.219 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:33:57.213+0000 7f81487c6740 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x564f5ef0b970 con 0x564f5efb1800 2024-04-23T10:33:57.219 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:33:57.213+0000 7f81487c6740 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 -- 0x564f5eea84a0 con 0x564f5efa6f40 2024-04-23T10:33:57.219 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:33:57.213+0000 7f81487c6740 1 -- --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_getmap magic: 0 -- 0x564f5ef07930 con 0x564f5efa6620 2024-04-23T10:33:57.219 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:33:57.213+0000 7f8137fff640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x564f5efb1800 0x564f5efb3bf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T10:33:57.219 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:33:57.213+0000 7f813effd640 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x564f5efa6f40 0x564f5efb12c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T10:33:57.219 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:33:57.213+0000 7f813effd640 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x564f5efa6f40 0x564f5efb12c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.119:3300/0 says I am v2:172.21.15.114:60274/0 (socket says 172.21.15.114:60274) 2024-04-23T10:33:57.219 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:33:57.213+0000 7f8137fff640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x564f5efb1800 0x564f5efb3bf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.114:36666/0 (socket says 172.21.15.114:36666) 2024-04-23T10:33:57.219 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:33:57.213+0000 7f813effd640 1 -- 172.21.15.114:0/3684201603 learned_addr learned my addr 172.21.15.114:0/3684201603 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-04-23T10:33:57.219 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:33:57.213+0000 7f813f7fe640 1 --2- 172.21.15.114:0/3684201603 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x564f5efa6620 0x564f5efa6a00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T10:33:57.219 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:33:57.213+0000 7f813effd640 1 -- 172.21.15.114:0/3684201603 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x564f5efa6620 msgr2=0x564f5efa6a00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T10:33:57.219 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:33:57.213+0000 7f813effd640 1 --2- 172.21.15.114:0/3684201603 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x564f5efa6620 0x564f5efa6a00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:33:57.219 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:33:57.213+0000 7f813effd640 1 -- 172.21.15.114:0/3684201603 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x564f5efb1800 msgr2=0x564f5efb3bf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T10:33:57.219 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:33:57.213+0000 7f813effd640 1 --2- 172.21.15.114:0/3684201603 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x564f5efb1800 0x564f5efb3bf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:33:57.220 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:33:57.213+0000 7f813effd640 1 -- 172.21.15.114:0/3684201603 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x564f5ec1e210 con 0x564f5efa6f40 2024-04-23T10:33:57.220 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:33:57.213+0000 7f81449d6640 1 --2- 172.21.15.114:0/3277871574 >> [v2:172.21.15.114:6800/1395693466,v1:172.21.15.114:6801/1395693466] conn(0x7f812c0a3dc0 0x7f812c0a6280 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T10:33:57.220 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:33:57.213+0000 7f8137fff640 1 --2- 172.21.15.114:0/3684201603 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x564f5efb1800 0x564f5efb3bf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-23T10:33:57.220 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:33:57.213+0000 7f813f7fe640 1 --2- 172.21.15.114:0/3277871574 >> [v2:172.21.15.114:6800/1395693466,v1:172.21.15.114:6801/1395693466] conn(0x7f812c0a3dc0 0x7f812c0a6280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T10:33:57.220 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:33:57.213+0000 7f813effd640 1 --2- 172.21.15.114:0/3684201603 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x564f5efa6f40 0x564f5efb12c0 secure :-1 s=READY pgs=81 cs=0 l=1 rev1=1 crypto rx=0x7f8130009d50 tx=0x7f81300117b0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=4e94e45ae7e5ea82 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T10:33:57.220 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:33:57.213+0000 7f81449d6640 1 -- 172.21.15.114:0/3277871574 <== mon.2 v2:172.21.15.119:3300/0 5 ==== osd_map(96..96 src has 1..96) ==== 8629+0+0 (secure 0 0 0) 0x7f81280f07a0 con 0x7f8140100e10 2024-04-23T10:33:57.220 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:33:57.213+0000 7f8135ffb640 1 -- 172.21.15.114:0/3684201603 <== mon.2 v2:172.21.15.119:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8130012650 con 0x564f5efa6f40 2024-04-23T10:33:57.220 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:33:57.214+0000 7f8135ffb640 1 -- 172.21.15.114:0/3684201603 <== mon.2 v2:172.21.15.119:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f8130009e40 con 0x564f5efa6f40 2024-04-23T10:33:57.220 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:33:57.214+0000 7f8135ffb640 1 -- 172.21.15.114:0/3684201603 <== mon.2 v2:172.21.15.119:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f813000fbf0 con 0x564f5efa6f40 2024-04-23T10:33:57.220 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:33:57.214+0000 7f81487c6740 1 -- 172.21.15.114:0/3684201603 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x564f5efa6f40 msgr2=0x564f5efb12c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T10:33:57.220 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:33:57.214+0000 7f813f7fe640 1 --2- 172.21.15.114:0/3277871574 >> [v2:172.21.15.114:6800/1395693466,v1:172.21.15.114:6801/1395693466] conn(0x7f812c0a3dc0 0x7f812c0a6280 secure :-1 s=READY pgs=117 cs=0 l=1 rev1=1 crypto rx=0x7f8138005fd0 tx=0x7f81380095c0 comp rx=0 tx=0).ready entity=mgr.15291 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T10:33:57.220 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:33:57.214+0000 7f81487c6740 1 --2- 172.21.15.114:0/3684201603 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x564f5efa6f40 0x564f5efb12c0 secure :-1 s=READY pgs=81 cs=0 l=1 rev1=1 crypto rx=0x7f8130009d50 tx=0x7f81300117b0 comp rx=0 tx=0).stop 2024-04-23T10:33:57.220 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:33:57.214+0000 7f81487c6740 1 -- 172.21.15.114:0/3684201603 shutdown_connections 2024-04-23T10:33:57.220 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:33:57.214+0000 7f81487c6740 1 --2- 172.21.15.114:0/3684201603 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x564f5efb1800 0x564f5efb3bf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:33:57.220 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:33:57.214+0000 7f81487c6740 1 --2- 172.21.15.114:0/3684201603 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x564f5efa6f40 0x564f5efb12c0 unknown :-1 s=CLOSED pgs=81 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:33:57.221 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:33:57.214+0000 7f81487c6740 1 --2- 172.21.15.114:0/3684201603 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x564f5efa6620 0x564f5efa6a00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:33:57.221 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:33:57.214+0000 7f81487c6740 1 -- 172.21.15.114:0/3684201603 >> 172.21.15.114:0/3684201603 conn(0x564f5efa2b30 msgr2=0x564f5efa2f10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T10:33:57.221 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:33:57.214+0000 7f81487c6740 1 -- 172.21.15.114:0/3684201603 shutdown_connections 2024-04-23T10:33:57.221 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:33:57.214+0000 7f81487c6740 1 -- 172.21.15.114:0/3684201603 wait complete. 2024-04-23T10:33:57.221 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:33:57.215+0000 7f81487c6740 1 -- start start 2024-04-23T10:33:57.221 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:33:57.215+0000 7f81487c6740 1 Processor -- start 2024-04-23T10:33:57.221 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:33:57.215+0000 7f81487c6740 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x564f5efa6620 0x564f5efb1130 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T10:33:57.221 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:33:57.215+0000 7f81487c6740 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x564f5efa6f40 0x564f5efae420 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T10:33:57.221 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:33:57.215+0000 7f81487c6740 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x564f5efb1800 0x564f5efae960 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T10:33:57.221 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:33:57.215+0000 7f81487c6740 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x564f5efb4430 con 0x564f5efb1800 2024-04-23T10:33:57.221 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:33:57.215+0000 7f81487c6740 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 -- 0x564f5eea60b0 con 0x564f5efa6f40 2024-04-23T10:33:57.221 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:33:57.215+0000 7f81487c6740 1 -- --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_getmap magic: 0 -- 0x564f5efb45b0 con 0x564f5efa6620 2024-04-23T10:33:57.221 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:33:57.215+0000 7f81487c6740 10 client.0 _ll_register_callbacks cb 0x564f5ee93d30 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-23T10:33:57.221 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:33:57.215+0000 7f813f7fe640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x564f5efb1800 0x564f5efae960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T10:33:57.221 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:33:57.215+0000 7f813f7fe640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x564f5efb1800 0x564f5efae960 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.114:36668/0 (socket says 172.21.15.114:36668) 2024-04-23T10:33:57.222 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:33:57.215+0000 7f8137fff640 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x564f5efa6f40 0x564f5efae420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T10:33:57.222 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:33:57.215+0000 7f813f7fe640 1 -- 172.21.15.114:0/4197502361 learned_addr learned my addr 172.21.15.114:0/4197502361 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-04-23T10:33:57.222 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:33:57.215+0000 7f813effd640 1 --2- 172.21.15.114:0/4197502361 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x564f5efa6620 0x564f5efb1130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T10:33:57.222 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:33:57.215+0000 7f8137fff640 1 -- 172.21.15.114:0/4197502361 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x564f5efa6620 msgr2=0x564f5efb1130 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T10:33:57.222 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:33:57.215+0000 7f8137fff640 1 --2- 172.21.15.114:0/4197502361 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x564f5efa6620 0x564f5efb1130 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:33:57.222 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:33:57.215+0000 7f8137fff640 1 -- 172.21.15.114:0/4197502361 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x564f5efb1800 msgr2=0x564f5efae960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T10:33:57.222 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:33:57.215+0000 7f8137fff640 1 --2- 172.21.15.114:0/4197502361 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x564f5efb1800 0x564f5efae960 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:33:57.222 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:33:57.215+0000 7f8137fff640 1 -- 172.21.15.114:0/4197502361 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({osdmap=0}) -- 0x564f5ecea620 con 0x564f5efa6f40 2024-04-23T10:33:57.222 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:33:57.215+0000 7f813f7fe640 1 --2- 172.21.15.114:0/4197502361 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x564f5efb1800 0x564f5efae960 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-23T10:33:57.222 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:33:57.216+0000 7f8137fff640 1 --2- 172.21.15.114:0/4197502361 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x564f5efa6f40 0x564f5efae420 secure :-1 s=READY pgs=82 cs=0 l=1 rev1=1 crypto rx=0x7f81280edf90 tx=0x7f8128004040 comp rx=0 tx=0).ready entity=mon.2 client_cookie=ef3cd5e9c9105b2d server_cookie=0 in_seq=0 out_seq=0 2024-04-23T10:33:57.222 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:33:57.216+0000 7f811effd640 10 client.0 ms_handle_connect on v2:172.21.15.119:3300/0 2024-04-23T10:33:57.222 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:33:57.216+0000 7f811effd640 1 -- 172.21.15.114:0/4197502361 <== mon.2 v2:172.21.15.119:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f81280b8a80 con 0x564f5efa6f40 2024-04-23T10:33:57.222 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:33:57.216+0000 7f811effd640 1 -- 172.21.15.114:0/4197502361 <== mon.2 v2:172.21.15.119:3300/0 2 ==== osd_map(96..96 src has 1..96) ==== 8629+0+0 (secure 0 0 0) 0x7f81280b8e50 con 0x564f5efa6f40 2024-04-23T10:33:57.223 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:33:57.216+0000 7f81487c6740 1 -- 172.21.15.114:0/4197502361 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x564f5ef19390 con 0x564f5efa6f40 2024-04-23T10:33:57.223 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:33:57.216+0000 7f811effd640 1 -- 172.21.15.114:0/4197502361 <== mon.2 v2:172.21.15.119:3300/0 3 ==== mon_get_version_reply(handle=1 version=31) ==== 24+0+0 (secure 0 0 0) 0x7f81280bc140 con 0x564f5efa6f40 2024-04-23T10:33:57.223 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:33:57.216+0000 7f81487c6740 10 client.25916 fetch_fsmap learned FSMap version 31 2024-04-23T10:33:57.223 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:33:57.216+0000 7f81487c6740 1 -- 172.21.15.114:0/4197502361 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({config=0+,fsmap=31,monmap=4+}) -- 0x564f5efc0340 con 0x564f5efa6f40 2024-04-23T10:33:57.223 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:33:57.216+0000 7f811effd640 1 -- 172.21.15.114:0/4197502361 <== mon.2 v2:172.21.15.119:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x564f5ef19390 con 0x564f5efa6f40 2024-04-23T10:33:57.223 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:33:57.216+0000 7f811effd640 1 -- 172.21.15.114:0/4197502361 <== mon.2 v2:172.21.15.119:3300/0 5 ==== fsmap(e 31) ==== 8342+0+0 (secure 0 0 0) 0x7f81280040d0 con 0x564f5efa6f40 2024-04-23T10:33:57.223 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:33:57.216+0000 7f81487c6740 10 client.25916 fetch_fsmap finished waiting for FSMap version 31 2024-04-23T10:33:57.223 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:33:57.216+0000 7f81487c6740 10 client.25916 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-04-23T10:33:57.223 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:33:57.216+0000 7f81487c6740 20 client.25916 populate_metadata read hostname 'smithi114' 2024-04-23T10:33:57.223 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:33:57.216+0000 7f81487c6740 1 --2- 172.21.15.114:0/4197502361 >> [v2:172.21.15.119:6838/937297340,v1:172.21.15.119:6839/937297340] conn(0x564f5efc17f0 0x564f5efc3c00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T10:33:57.223 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:33:57.216+0000 7f81487c6740 4 client.25916 mds_command: new command op to 24545 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-23T10:33:57.223 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:33:57.216+0000 7f81487c6740 1 -- 172.21.15.114:0/4197502361 --> [v2:172.21.15.119:6838/937297340,v1:172.21.15.119:6839/937297340] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x564f5ef13200 con 0x564f5efc17f0 2024-04-23T10:33:57.223 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:33:57.217+0000 7f813effd640 1 --2- 172.21.15.114:0/4197502361 >> [v2:172.21.15.119:6838/937297340,v1:172.21.15.119:6839/937297340] conn(0x564f5efc17f0 0x564f5efc3c00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T10:33:57.223 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:33:57.217+0000 7f811effd640 10 client.25916 ms_handle_connect on v2:172.21.15.119:6838/937297340 2024-04-23T10:33:57.224 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:33:57.217+0000 7f813effd640 1 --2- 172.21.15.114:0/4197502361 >> [v2:172.21.15.119:6838/937297340,v1:172.21.15.119:6839/937297340] conn(0x564f5efc17f0 0x564f5efc3c00 crc :-1 s=READY pgs=126 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=2a85113262447f03 server_cookie=82027eb9db93df05 in_seq=0 out_seq=0 2024-04-23T10:33:57.224 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:33:57.217+0000 7f811effd640 1 -- 172.21.15.114:0/4197502361 <== mds.0 v2:172.21.15.119:6838/937297340 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x564f5ef13200 con 0x564f5efc17f0 2024-04-23T10:33:57.224 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:33:57.217+0000 7f811effd640 10 client.25916 handle_command_reply: tid=0 2024-04-23T10:33:57.224 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:33:57.217+0000 7f81487c6740 1 client.25916 shutdown 2024-04-23T10:33:57.224 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:33:57.218+0000 7f81487c6740 1 -- 172.21.15.114:0/4197502361 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x564f5efa6f40 msgr2=0x564f5efae420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T10:33:57.224 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:33:57.218+0000 7f81487c6740 1 --2- 172.21.15.114:0/4197502361 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x564f5efa6f40 0x564f5efae420 secure :-1 s=READY pgs=82 cs=0 l=1 rev1=1 crypto rx=0x7f81280edf90 tx=0x7f8128004040 comp rx=0 tx=0).stop 2024-04-23T10:33:57.224 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:33:57.218+0000 7f81487c6740 1 -- 172.21.15.114:0/4197502361 shutdown_connections 2024-04-23T10:33:57.224 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:33:57.218+0000 7f81487c6740 1 --2- 172.21.15.114:0/4197502361 >> [v2:172.21.15.119:6838/937297340,v1:172.21.15.119:6839/937297340] conn(0x564f5efc17f0 0x564f5efc3c00 crc :-1 s=READY pgs=126 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:33:57.224 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:33:57.218+0000 7f81487c6740 1 --2- 172.21.15.114:0/4197502361 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x564f5efb1800 0x564f5efae960 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:33:57.224 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:33:57.218+0000 7f81487c6740 1 --2- 172.21.15.114:0/4197502361 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x564f5efa6f40 0x564f5efae420 unknown :-1 s=CLOSED pgs=82 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:33:57.224 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:33:57.218+0000 7f81487c6740 1 --2- 172.21.15.114:0/4197502361 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x564f5efa6620 0x564f5efb1130 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:33:57.224 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:33:57.218+0000 7f811effd640 0 client.25916 ms_handle_reset on v2:172.21.15.119:6838/937297340 2024-04-23T10:33:57.224 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:33:57.218+0000 7f81487c6740 1 -- 172.21.15.114:0/4197502361 >> 172.21.15.114:0/4197502361 conn(0x564f5efa2b30 msgr2=0x564f5efa8860 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T10:33:57.224 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:33:57.218+0000 7f81487c6740 1 -- 172.21.15.114:0/4197502361 shutdown_connections 2024-04-23T10:33:57.224 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:33:57.218+0000 7f81487c6740 1 -- 172.21.15.114:0/4197502361 wait complete. 2024-04-23T10:33:57.224 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:33:57.218+0000 7f81487c6740 20 client.25916 trim_cache size 0 max 16384 2024-04-23T10:33:57.233 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:33:57.233+0000 7f81487c6740 1 Processor -- start 2024-04-23T10:33:57.233 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:33:57.233+0000 7f81487c6740 1 -- start start 2024-04-23T10:33:57.233 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:33:57.233+0000 7f81487c6740 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x564f5efa6620 0x564f5efbd2f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T10:33:57.233 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:33:57.233+0000 7f81487c6740 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x564f5efa6f40 0x564f5efa98e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T10:33:57.233 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:33:57.233+0000 7f81487c6740 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x564f5efb1800 0x564f5efb0c60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T10:33:57.233 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:33:57.233+0000 7f81487c6740 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x7f81401b0510 con 0x564f5efb1800 2024-04-23T10:33:57.233 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:33:57.233+0000 7f81487c6740 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 -- 0x7f81401addb0 con 0x564f5efa6f40 2024-04-23T10:33:57.233 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:33:57.233+0000 7f81487c6740 1 -- --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_getmap magic: 0 -- 0x7f81401abc20 con 0x564f5efa6620 2024-04-23T10:33:57.233 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:33:57.233+0000 7f813f7fe640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x564f5efb1800 0x564f5efb0c60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T10:33:57.233 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:33:57.233+0000 7f8137fff640 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x564f5efa6f40 0x564f5efa98e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T10:33:57.234 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:33:57.233+0000 7f813f7fe640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x564f5efb1800 0x564f5efb0c60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.114:36670/0 (socket says 172.21.15.114:36670) 2024-04-23T10:33:57.234 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:33:57.233+0000 7f813f7fe640 1 -- 172.21.15.114:0/3519489335 learned_addr learned my addr 172.21.15.114:0/3519489335 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-04-23T10:33:57.234 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:33:57.233+0000 7f813effd640 1 --2- 172.21.15.114:0/3519489335 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x564f5efa6620 0x564f5efbd2f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T10:33:57.234 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:33:57.233+0000 7f813f7fe640 1 -- 172.21.15.114:0/3519489335 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x564f5efa6620 msgr2=0x564f5efbd2f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T10:33:57.234 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:33:57.233+0000 7f813f7fe640 1 --2- 172.21.15.114:0/3519489335 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x564f5efa6620 0x564f5efbd2f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:33:57.234 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:33:57.233+0000 7f813f7fe640 1 -- 172.21.15.114:0/3519489335 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x564f5efa6f40 msgr2=0x564f5efa98e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T10:33:57.234 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:33:57.233+0000 7f813f7fe640 1 --2- 172.21.15.114:0/3519489335 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x564f5efa6f40 0x564f5efa98e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:33:57.234 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:33:57.233+0000 7f813f7fe640 1 -- 172.21.15.114:0/3519489335 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x564f5efa6d70 con 0x564f5efb1800 2024-04-23T10:33:57.234 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:33:57.234+0000 7f813effd640 1 --2- 172.21.15.114:0/3519489335 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x564f5efa6620 0x564f5efbd2f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-23T10:33:57.234 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:33:57.234+0000 7f8137fff640 1 --2- 172.21.15.114:0/3519489335 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x564f5efa6f40 0x564f5efa98e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-23T10:33:57.234 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:33:57.234+0000 7f813f7fe640 1 --2- 172.21.15.114:0/3519489335 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x564f5efb1800 0x564f5efb0c60 secure :-1 s=READY pgs=777 cs=0 l=1 rev1=1 crypto rx=0x7f8138009a60 tx=0x7f8138019000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2cb29aa250c28b9f server_cookie=0 in_seq=0 out_seq=0 2024-04-23T10:33:57.234 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:33:57.234+0000 7f81357fa640 1 -- 172.21.15.114:0/3519489335 <== mon.0 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f813800ee60 con 0x564f5efb1800 2024-04-23T10:33:57.234 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:33:57.234+0000 7f81357fa640 1 -- 172.21.15.114:0/3519489335 <== mon.0 v2:172.21.15.114:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f813802a070 con 0x564f5efb1800 2024-04-23T10:33:57.234 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:33:57.234+0000 7f81357fa640 1 -- 172.21.15.114:0/3519489335 <== mon.0 v2:172.21.15.114:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f813802fac0 con 0x564f5efb1800 2024-04-23T10:33:57.238 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:33:57.235+0000 7f81487c6740 1 -- 172.21.15.114:0/3519489335 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x564f5efb1800 msgr2=0x564f5efb0c60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T10:33:57.238 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:33:57.235+0000 7f81487c6740 1 --2- 172.21.15.114:0/3519489335 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x564f5efb1800 0x564f5efb0c60 secure :-1 s=READY pgs=777 cs=0 l=1 rev1=1 crypto rx=0x7f8138009a60 tx=0x7f8138019000 comp rx=0 tx=0).stop 2024-04-23T10:33:57.238 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:33:57.235+0000 7f81487c6740 1 -- 172.21.15.114:0/3519489335 shutdown_connections 2024-04-23T10:33:57.238 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:33:57.235+0000 7f81487c6740 1 --2- 172.21.15.114:0/3519489335 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x564f5efb1800 0x564f5efb0c60 unknown :-1 s=CLOSED pgs=777 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:33:57.238 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:33:57.235+0000 7f81487c6740 1 --2- 172.21.15.114:0/3519489335 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x564f5efa6f40 0x564f5efa98e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:33:57.238 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:33:57.235+0000 7f81487c6740 1 --2- 172.21.15.114:0/3519489335 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x564f5efa6620 0x564f5efbd2f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:33:57.238 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:33:57.235+0000 7f81487c6740 1 -- 172.21.15.114:0/3519489335 >> 172.21.15.114:0/3519489335 conn(0x564f5efa2b30 msgr2=0x564f5efa91d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T10:33:57.238 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:33:57.235+0000 7f81487c6740 1 -- 172.21.15.114:0/3519489335 shutdown_connections 2024-04-23T10:33:57.238 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:33:57.235+0000 7f81487c6740 1 -- 172.21.15.114:0/3519489335 wait complete. 2024-04-23T10:33:57.238 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:33:57.235+0000 7f81487c6740 1 -- start start 2024-04-23T10:33:57.238 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:33:57.235+0000 7f81487c6740 1 Processor -- start 2024-04-23T10:33:57.239 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:33:57.235+0000 7f81487c6740 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x564f5efa6620 0x564f5efb9c30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T10:33:57.239 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:33:57.235+0000 7f81487c6740 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x564f5efa6f40 0x564f5efaadf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T10:33:57.239 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:33:57.235+0000 7f81487c6740 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x564f5efb1800 0x564f5efab330 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T10:33:57.239 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:33:57.235+0000 7f81487c6740 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x7f81401ac270 con 0x564f5efa6f40 2024-04-23T10:33:57.239 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:33:57.235+0000 7f81487c6740 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 -- 0x7f81401ad6e0 con 0x564f5efa6620 2024-04-23T10:33:57.239 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:33:57.235+0000 7f81487c6740 1 -- --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_getmap magic: 0 -- 0x7f81401ac940 con 0x564f5efb1800 2024-04-23T10:33:57.239 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:33:57.235+0000 7f81487c6740 10 client.0 _ll_register_callbacks cb 0x7f8128005f30 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-23T10:33:57.239 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:33:57.235+0000 7f8137fff640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x564f5efa6f40 0x564f5efaadf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T10:33:57.239 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:33:57.236+0000 7f8137fff640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x564f5efa6f40 0x564f5efaadf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.114:36674/0 (socket says 172.21.15.114:36674) 2024-04-23T10:33:57.239 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:33:57.236+0000 7f8137fff640 1 -- 172.21.15.114:0/3056830422 learned_addr learned my addr 172.21.15.114:0/3056830422 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-04-23T10:33:57.239 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:33:57.236+0000 7f813effd640 1 --2- 172.21.15.114:0/3056830422 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x564f5efa6620 0x564f5efb9c30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T10:33:57.239 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:33:57.236+0000 7f813f7fe640 1 --2- 172.21.15.114:0/3056830422 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x564f5efb1800 0x564f5efab330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T10:33:57.239 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:33:57.236+0000 7f8137fff640 1 -- 172.21.15.114:0/3056830422 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x564f5efb1800 msgr2=0x564f5efab330 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T10:33:57.239 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:33:57.236+0000 7f8137fff640 1 --2- 172.21.15.114:0/3056830422 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x564f5efb1800 0x564f5efab330 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:33:57.239 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:33:57.236+0000 7f8137fff640 1 -- 172.21.15.114:0/3056830422 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x564f5efa6620 msgr2=0x564f5efb9c30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T10:33:57.240 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:33:57.236+0000 7f8137fff640 1 --2- 172.21.15.114:0/3056830422 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x564f5efa6620 0x564f5efb9c30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:33:57.240 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:33:57.236+0000 7f8137fff640 1 -- 172.21.15.114:0/3056830422 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({osdmap=0}) -- 0x564f5efabb30 con 0x564f5efa6f40 2024-04-23T10:33:57.240 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:33:57.236+0000 7f8137fff640 1 --2- 172.21.15.114:0/3056830422 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x564f5efa6f40 0x564f5efaadf0 secure :-1 s=READY pgs=778 cs=0 l=1 rev1=1 crypto rx=0x7f81280bc770 tx=0x7f8128004470 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c0f9bfbd15527a11 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T10:33:57.240 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:33:57.236+0000 7f811effd640 10 client.0 ms_handle_connect on v2:172.21.15.114:3300/0 2024-04-23T10:33:57.240 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:33:57.236+0000 7f811effd640 1 -- 172.21.15.114:0/3056830422 <== mon.0 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f81280bc320 con 0x564f5efa6f40 2024-04-23T10:33:57.240 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:33:57.236+0000 7f811effd640 1 -- 172.21.15.114:0/3056830422 <== mon.0 v2:172.21.15.114:3300/0 2 ==== osd_map(96..96 src has 1..96) ==== 8629+0+0 (secure 0 0 0) 0x7f81280bdb10 con 0x564f5efa6f40 2024-04-23T10:33:57.240 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:33:57.236+0000 7f81487c6740 1 -- 172.21.15.114:0/3056830422 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f813802a070 con 0x564f5efa6f40 2024-04-23T10:33:57.240 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:33:57.236+0000 7f811effd640 1 -- 172.21.15.114:0/3056830422 <== mon.0 v2:172.21.15.114:3300/0 3 ==== mon_get_version_reply(handle=1 version=31) ==== 24+0+0 (secure 0 0 0) 0x7f81280bc5b0 con 0x564f5efa6f40 2024-04-23T10:33:57.240 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:33:57.236+0000 7f81487c6740 10 client.16554 fetch_fsmap learned FSMap version 31 2024-04-23T10:33:57.240 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:33:57.236+0000 7f81487c6740 1 -- 172.21.15.114:0/3056830422 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({config=0+,fsmap=31,monmap=4+}) -- 0x564f5f027d60 con 0x564f5efa6f40 2024-04-23T10:33:57.240 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:33:57.236+0000 7f811effd640 1 -- 172.21.15.114:0/3056830422 <== mon.0 v2:172.21.15.114:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f813802a070 con 0x564f5efa6f40 2024-04-23T10:33:57.240 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:33:57.237+0000 7f811effd640 1 -- 172.21.15.114:0/3056830422 <== mon.0 v2:172.21.15.114:3300/0 5 ==== fsmap(e 31) ==== 8342+0+0 (secure 0 0 0) 0x7f81280bb6a0 con 0x564f5efa6f40 2024-04-23T10:33:57.240 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:33:57.237+0000 7f81487c6740 10 client.16554 fetch_fsmap finished waiting for FSMap version 31 2024-04-23T10:33:57.240 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:33:57.237+0000 7f81487c6740 10 client.16554 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-04-23T10:33:57.241 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:33:57.237+0000 7f81487c6740 20 client.16554 populate_metadata read hostname 'smithi114' 2024-04-23T10:33:57.241 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:33:57.237+0000 7f81487c6740 1 --2- 172.21.15.114:0/3056830422 >> [v2:172.21.15.119:6838/937297340,v1:172.21.15.119:6839/937297340] conn(0x564f5f028320 0x564f5f02a740 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T10:33:57.241 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:33:57.237+0000 7f81487c6740 4 client.16554 mds_command: new command op to 24545 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-04-23T10:33:57.241 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:33:57.237+0000 7f81487c6740 1 -- 172.21.15.114:0/3056830422 --> [v2:172.21.15.119:6838/937297340,v1:172.21.15.119:6839/937297340] -- command(tid 0: {"prefix": "flush journal"}) -- 0x564f5ee8f5f0 con 0x564f5f028320 2024-04-23T10:33:57.241 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:33:57.237+0000 7f813effd640 1 --2- 172.21.15.114:0/3056830422 >> [v2:172.21.15.119:6838/937297340,v1:172.21.15.119:6839/937297340] conn(0x564f5f028320 0x564f5f02a740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T10:33:57.241 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:33:57.237+0000 7f813effd640 1 --2- 172.21.15.114:0/3056830422 >> [v2:172.21.15.119:6838/937297340,v1:172.21.15.119:6839/937297340] conn(0x564f5f028320 0x564f5f02a740 crc :-1 s=READY pgs=127 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=1b1fade88d269c2e server_cookie=3c9bd862c4c7f716 in_seq=0 out_seq=0 2024-04-23T10:33:57.241 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:33:57.237+0000 7f811effd640 10 client.16554 ms_handle_connect on v2:172.21.15.119:6838/937297340 2024-04-23T10:33:57.307 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:33:57.308+0000 7f811effd640 1 -- 172.21.15.114:0/3056830422 <== mds.0 v2:172.21.15.119:6838/937297340 1 ==== command_reply(tid 0: 0 ) ==== 8+0+44 (crc 0 0 0) 0x564f5ee8f5f0 con 0x564f5f028320 2024-04-23T10:33:57.307 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:33:57.308+0000 7f811effd640 10 client.16554 handle_command_reply: tid=0 2024-04-23T10:33:57.307 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:33:57.308+0000 7f81487c6740 1 client.16554 shutdown 2024-04-23T10:33:57.307 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:33:57.308+0000 7f81487c6740 1 -- 172.21.15.114:0/3056830422 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x564f5efa6f40 msgr2=0x564f5efaadf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T10:33:57.308 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:33:57.308+0000 7f81487c6740 1 --2- 172.21.15.114:0/3056830422 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x564f5efa6f40 0x564f5efaadf0 secure :-1 s=READY pgs=778 cs=0 l=1 rev1=1 crypto rx=0x7f81280bc770 tx=0x7f8128004470 comp rx=0 tx=0).stop 2024-04-23T10:33:57.308 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:33:57.308+0000 7f81487c6740 1 -- 172.21.15.114:0/3056830422 shutdown_connections 2024-04-23T10:33:57.308 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:33:57.308+0000 7f81487c6740 1 --2- 172.21.15.114:0/3056830422 >> [v2:172.21.15.119:6838/937297340,v1:172.21.15.119:6839/937297340] conn(0x564f5f028320 0x564f5f02a740 crc :-1 s=READY pgs=127 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:33:57.308 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:33:57.308+0000 7f81487c6740 1 --2- 172.21.15.114:0/3056830422 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x564f5efb1800 0x564f5efab330 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:33:57.308 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:33:57.308+0000 7f81487c6740 1 --2- 172.21.15.114:0/3056830422 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x564f5efa6f40 0x564f5efaadf0 unknown :-1 s=CLOSED pgs=778 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:33:57.308 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:33:57.308+0000 7f81487c6740 1 --2- 172.21.15.114:0/3056830422 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x564f5efa6620 0x564f5efb9c30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:33:57.308 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:33:57.308+0000 7f81487c6740 1 -- 172.21.15.114:0/3056830422 >> 172.21.15.114:0/3056830422 conn(0x564f5efa2b30 msgr2=0x564f5efa8860 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T10:33:57.308 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:33:57.308+0000 7f811effd640 0 client.16554 ms_handle_reset on v2:172.21.15.119:6838/937297340 2024-04-23T10:33:57.308 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:33:57.308+0000 7f81487c6740 1 -- 172.21.15.114:0/3056830422 shutdown_connections 2024-04-23T10:33:57.308 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:33:57.308+0000 7f81487c6740 1 -- 172.21.15.114:0/3056830422 wait complete. 2024-04-23T10:33:57.308 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:33:57.308+0000 7f81487c6740 20 client.16554 trim_cache size 0 max 16384 2024-04-23T10:33:57.309 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stdout:{ 2024-04-23T10:33:57.309 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stdout: "message": "", 2024-04-23T10:33:57.309 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stdout: "return_code": 0 2024-04-23T10:33:57.309 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stdout:} 2024-04-23T10:33:57.309 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:33:57.309+0000 7f81459d8640 1 -- 172.21.15.114:0/3277871574 >> [v2:172.21.15.114:6800/1395693466,v1:172.21.15.114:6801/1395693466] conn(0x7f812c0a3dc0 msgr2=0x7f812c0a6280 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T10:33:57.310 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:33:57.309+0000 7f81459d8640 1 --2- 172.21.15.114:0/3277871574 >> [v2:172.21.15.114:6800/1395693466,v1:172.21.15.114:6801/1395693466] conn(0x7f812c0a3dc0 0x7f812c0a6280 secure :-1 s=READY pgs=117 cs=0 l=1 rev1=1 crypto rx=0x7f8138005fd0 tx=0x7f81380095c0 comp rx=0 tx=0).stop 2024-04-23T10:33:57.310 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:33:57.309+0000 7f81459d8640 1 -- 172.21.15.114:0/3277871574 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f8140100e10 msgr2=0x7f8140198c70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T10:33:57.310 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:33:57.309+0000 7f81459d8640 1 --2- 172.21.15.114:0/3277871574 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f8140100e10 0x7f8140198c70 secure :-1 s=READY pgs=80 cs=0 l=1 rev1=1 crypto rx=0x7f8128009980 tx=0x7f8128032dc0 comp rx=0 tx=0).stop 2024-04-23T10:33:57.310 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:33:57.309+0000 7f81459d8640 1 -- 172.21.15.114:0/3277871574 shutdown_connections 2024-04-23T10:33:57.310 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:33:57.309+0000 7f81459d8640 1 --2- 172.21.15.114:0/3277871574 >> [v2:172.21.15.114:6800/1395693466,v1:172.21.15.114:6801/1395693466] conn(0x7f812c0a3dc0 0x7f812c0a6280 unknown :-1 s=CLOSED pgs=117 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:33:57.310 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:33:57.309+0000 7f81459d8640 1 --2- 172.21.15.114:0/3277871574 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f8140106f30 0x7f814019cfa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:33:57.310 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:33:57.309+0000 7f81459d8640 1 --2- 172.21.15.114:0/3277871574 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f8140100e10 0x7f8140198c70 unknown :-1 s=CLOSED pgs=80 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:33:57.310 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:33:57.309+0000 7f81459d8640 1 --2- 172.21.15.114:0/3277871574 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f8140100450 0x7f8140198730 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:33:57.310 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:33:57.309+0000 7f81459d8640 1 -- 172.21.15.114:0/3277871574 >> 172.21.15.114:0/3277871574 conn(0x7f81400773e0 msgr2=0x7f81400fe760 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T10:33:57.310 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:33:57.309+0000 7f81459d8640 1 -- 172.21.15.114:0/3277871574 shutdown_connections 2024-04-23T10:33:57.310 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:33:57.309+0000 7f81459d8640 1 -- 172.21.15.114:0/3277871574 wait complete. 2024-04-23T10:33:57.927 INFO:tasks.workunit.client.0.smithi114.stdout: 1 483940 6.77 MB/sec execute 246 sec latency 58.446 ms 2024-04-23T10:33:58.077 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 23 10:33:57 smithi204 ceph-mon[32649]: pgmap v227: 97 pgs: 97 active+clean; 147 MiB data, 1.8 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 MiB/s rd, 4.4 MiB/s wr, 369 op/s 2024-04-23T10:33:58.242 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 23 10:33:57 smithi119 ceph-mon[32608]: pgmap v227: 97 pgs: 97 active+clean; 147 MiB data, 1.8 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 MiB/s rd, 4.4 MiB/s wr, 369 op/s 2024-04-23T10:33:58.254 INFO:journalctl@ceph.mon.a.smithi114.stdout:Apr 23 10:33:57 smithi114 ceph-mon[29255]: pgmap v227: 97 pgs: 97 active+clean; 147 MiB data, 1.8 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 MiB/s rd, 4.4 MiB/s wr, 369 op/s 2024-04-23T10:33:58.928 INFO:tasks.workunit.client.0.smithi114.stdout: 1 486573 6.80 MB/sec execute 247 sec latency 15.870 ms 2024-04-23T10:33:59.928 INFO:tasks.workunit.client.0.smithi114.stdout: 1 487739 6.79 MB/sec execute 248 sec latency 42.406 ms 2024-04-23T10:34:00.077 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 23 10:33:59 smithi204 ceph-mon[32649]: pgmap v228: 97 pgs: 97 active+clean; 147 MiB data, 1.8 GiB used, 1.0 TiB / 1.0 TiB avail; 1.3 MiB/s rd, 3.0 MiB/s wr, 231 op/s 2024-04-23T10:34:00.241 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 23 10:33:59 smithi119 ceph-mon[32608]: pgmap v228: 97 pgs: 97 active+clean; 147 MiB data, 1.8 GiB used, 1.0 TiB / 1.0 TiB avail; 1.3 MiB/s rd, 3.0 MiB/s wr, 231 op/s 2024-04-23T10:34:00.253 INFO:journalctl@ceph.mon.a.smithi114.stdout:Apr 23 10:33:59 smithi114 ceph-mon[29255]: pgmap v228: 97 pgs: 97 active+clean; 147 MiB data, 1.8 GiB used, 1.0 TiB / 1.0 TiB avail; 1.3 MiB/s rd, 3.0 MiB/s wr, 231 op/s 2024-04-23T10:34:00.928 INFO:tasks.workunit.client.0.smithi114.stdout: 1 490014 6.82 MB/sec execute 249 sec latency 47.646 ms 2024-04-23T10:34:01.928 INFO:tasks.workunit.client.0.smithi114.stdout: 1 491043 6.81 MB/sec execute 250 sec latency 81.022 ms 2024-04-23T10:34:02.043 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 23 10:34:01 smithi119 ceph-mon[32608]: pgmap v229: 97 pgs: 97 active+clean; 147 MiB data, 1.8 GiB used, 1.0 TiB / 1.0 TiB avail; 1.3 MiB/s rd, 3.0 MiB/s wr, 231 op/s 2024-04-23T10:34:02.077 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 23 10:34:01 smithi204 ceph-mon[32649]: pgmap v229: 97 pgs: 97 active+clean; 147 MiB data, 1.8 GiB used, 1.0 TiB / 1.0 TiB avail; 1.3 MiB/s rd, 3.0 MiB/s wr, 231 op/s 2024-04-23T10:34:02.254 INFO:journalctl@ceph.mon.a.smithi114.stdout:Apr 23 10:34:01 smithi114 ceph-mon[29255]: pgmap v229: 97 pgs: 97 active+clean; 147 MiB data, 1.8 GiB used, 1.0 TiB / 1.0 TiB avail; 1.3 MiB/s rd, 3.0 MiB/s wr, 231 op/s 2024-04-23T10:34:02.928 INFO:tasks.workunit.client.0.smithi114.stdout: 1 493658 6.84 MB/sec execute 251 sec latency 16.443 ms 2024-04-23T10:34:03.928 INFO:tasks.workunit.client.0.smithi114.stdout: 1 494925 6.84 MB/sec execute 252 sec latency 36.224 ms 2024-04-23T10:34:04.077 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 23 10:34:03 smithi204 ceph-mon[32649]: pgmap v230: 97 pgs: 97 active+clean; 142 MiB data, 1.8 GiB used, 1.0 TiB / 1.0 TiB avail; 1.8 MiB/s rd, 4.6 MiB/s wr, 418 op/s 2024-04-23T10:34:04.242 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 23 10:34:03 smithi119 ceph-mon[32608]: pgmap v230: 97 pgs: 97 active+clean; 142 MiB data, 1.8 GiB used, 1.0 TiB / 1.0 TiB avail; 1.8 MiB/s rd, 4.6 MiB/s wr, 418 op/s 2024-04-23T10:34:04.254 INFO:journalctl@ceph.mon.a.smithi114.stdout:Apr 23 10:34:03 smithi114 ceph-mon[29255]: pgmap v230: 97 pgs: 97 active+clean; 142 MiB data, 1.8 GiB used, 1.0 TiB / 1.0 TiB avail; 1.8 MiB/s rd, 4.6 MiB/s wr, 418 op/s 2024-04-23T10:34:04.928 INFO:tasks.workunit.client.0.smithi114.stdout: 1 496872 6.86 MB/sec execute 253 sec latency 82.087 ms 2024-04-23T10:34:05.078 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 23 10:34:04 smithi204 ceph-mon[32649]: osdmap e97: 12 total, 12 up, 12 in 2024-04-23T10:34:05.241 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 23 10:34:04 smithi119 ceph-mon[32608]: osdmap e97: 12 total, 12 up, 12 in 2024-04-23T10:34:05.253 INFO:journalctl@ceph.mon.a.smithi114.stdout:Apr 23 10:34:04 smithi114 ceph-mon[29255]: osdmap e97: 12 total, 12 up, 12 in 2024-04-23T10:34:05.928 INFO:tasks.workunit.client.0.smithi114.stdout: 1 498380 6.86 MB/sec execute 254 sec latency 21.552 ms 2024-04-23T10:34:06.077 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 23 10:34:05 smithi204 ceph-mon[32649]: pgmap v232: 97 pgs: 97 active+clean; 142 MiB data, 1.8 GiB used, 1.0 TiB / 1.0 TiB avail; 1.4 MiB/s rd, 3.6 MiB/s wr, 360 op/s 2024-04-23T10:34:06.241 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 23 10:34:05 smithi119 ceph-mon[32608]: pgmap v232: 97 pgs: 97 active+clean; 142 MiB data, 1.8 GiB used, 1.0 TiB / 1.0 TiB avail; 1.4 MiB/s rd, 3.6 MiB/s wr, 360 op/s 2024-04-23T10:34:06.261 INFO:journalctl@ceph.mon.a.smithi114.stdout:Apr 23 10:34:05 smithi114 ceph-mon[29255]: pgmap v232: 97 pgs: 97 active+clean; 142 MiB data, 1.8 GiB used, 1.0 TiB / 1.0 TiB avail; 1.4 MiB/s rd, 3.6 MiB/s wr, 360 op/s 2024-04-23T10:34:06.928 INFO:tasks.workunit.client.0.smithi114.stdout: 1 501146 6.90 MB/sec execute 255 sec latency 17.947 ms 2024-04-23T10:34:07.928 INFO:tasks.workunit.client.0.smithi114.stdout: 1 502471 6.90 MB/sec execute 256 sec latency 16.201 ms 2024-04-23T10:34:08.078 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 23 10:34:07 smithi204 ceph-mon[32649]: pgmap v233: 97 pgs: 97 active+clean; 149 MiB data, 1.9 GiB used, 1.0 TiB / 1.0 TiB avail; 1.3 MiB/s rd, 3.7 MiB/s wr, 359 op/s 2024-04-23T10:34:08.242 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 23 10:34:07 smithi119 ceph-mon[32608]: pgmap v233: 97 pgs: 97 active+clean; 149 MiB data, 1.9 GiB used, 1.0 TiB / 1.0 TiB avail; 1.3 MiB/s rd, 3.7 MiB/s wr, 359 op/s 2024-04-23T10:34:08.254 INFO:journalctl@ceph.mon.a.smithi114.stdout:Apr 23 10:34:07 smithi114 ceph-mon[29255]: pgmap v233: 97 pgs: 97 active+clean; 149 MiB data, 1.9 GiB used, 1.0 TiB / 1.0 TiB avail; 1.3 MiB/s rd, 3.7 MiB/s wr, 359 op/s 2024-04-23T10:34:08.928 INFO:tasks.workunit.client.0.smithi114.stdout: 1 504816 6.92 MB/sec execute 257 sec latency 16.053 ms 2024-04-23T10:34:09.742 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 23 10:34:09 smithi119 ceph-mon[32608]: pgmap v234: 97 pgs: 97 active+clean; 149 MiB data, 1.9 GiB used, 1.0 TiB / 1.0 TiB avail; 1.3 MiB/s rd, 3.7 MiB/s wr, 359 op/s 2024-04-23T10:34:09.742 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 23 10:34:09 smithi119 ceph-mon[32608]: osdmap e98: 12 total, 12 up, 12 in 2024-04-23T10:34:09.753 INFO:journalctl@ceph.mon.a.smithi114.stdout:Apr 23 10:34:09 smithi114 ceph-mon[29255]: pgmap v234: 97 pgs: 97 active+clean; 149 MiB data, 1.9 GiB used, 1.0 TiB / 1.0 TiB avail; 1.3 MiB/s rd, 3.7 MiB/s wr, 359 op/s 2024-04-23T10:34:09.753 INFO:journalctl@ceph.mon.a.smithi114.stdout:Apr 23 10:34:09 smithi114 ceph-mon[29255]: osdmap e98: 12 total, 12 up, 12 in 2024-04-23T10:34:09.827 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 23 10:34:09 smithi204 ceph-mon[32649]: pgmap v234: 97 pgs: 97 active+clean; 149 MiB data, 1.9 GiB used, 1.0 TiB / 1.0 TiB avail; 1.3 MiB/s rd, 3.7 MiB/s wr, 359 op/s 2024-04-23T10:34:09.827 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 23 10:34:09 smithi204 ceph-mon[32649]: osdmap e98: 12 total, 12 up, 12 in 2024-04-23T10:34:09.928 INFO:tasks.workunit.client.0.smithi114.stdout: 1 506184 6.92 MB/sec execute 258 sec latency 51.230 ms 2024-04-23T10:34:10.402 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:34:10.399+0000 7f4d35abb640 1 -- 172.21.15.114:0/1299733194 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f4d30101130 msgr2=0x7f4d301015b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T10:34:10.402 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:34:10.399+0000 7f4d35abb640 1 --2- 172.21.15.114:0/1299733194 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f4d30101130 0x7f4d301015b0 secure :-1 s=READY pgs=779 cs=0 l=1 rev1=1 crypto rx=0x7f4d24009930 tx=0x7f4d2402a420 comp rx=0 tx=0).stop 2024-04-23T10:34:10.402 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:34:10.399+0000 7f4d35abb640 1 -- 172.21.15.114:0/1299733194 shutdown_connections 2024-04-23T10:34:10.402 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:34:10.399+0000 7f4d35abb640 1 --2- 172.21.15.114:0/1299733194 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f4d30101af0 0x7f4d3010dca0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:34:10.402 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:34:10.399+0000 7f4d35abb640 1 --2- 172.21.15.114:0/1299733194 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f4d30101130 0x7f4d301015b0 unknown :-1 s=CLOSED pgs=779 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:34:10.403 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:34:10.399+0000 7f4d35abb640 1 --2- 172.21.15.114:0/1299733194 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f4d30106f90 0x7f4d30107370 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:34:10.403 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:34:10.399+0000 7f4d35abb640 1 -- 172.21.15.114:0/1299733194 >> 172.21.15.114:0/1299733194 conn(0x7f4d30077c30 msgr2=0x7f4d30078040 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T10:34:10.403 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:34:10.399+0000 7f4d35abb640 1 -- 172.21.15.114:0/1299733194 shutdown_connections 2024-04-23T10:34:10.403 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:34:10.399+0000 7f4d35abb640 1 -- 172.21.15.114:0/1299733194 wait complete. 2024-04-23T10:34:10.403 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:34:10.400+0000 7f4d35abb640 1 Processor -- start 2024-04-23T10:34:10.403 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:34:10.400+0000 7f4d35abb640 1 -- start start 2024-04-23T10:34:10.403 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:34:10.400+0000 7f4d35abb640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f4d30101130 0x7f4d301986c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T10:34:10.403 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:34:10.400+0000 7f4d35abb640 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f4d30101af0 0x7f4d30198c00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T10:34:10.403 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:34:10.400+0000 7f4d35abb640 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f4d30106f90 0x7f4d3019cf80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T10:34:10.403 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:34:10.400+0000 7f4d35abb640 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x7f4d30111070 con 0x7f4d30101130 2024-04-23T10:34:10.404 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:34:10.400+0000 7f4d35abb640 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 -- 0x7f4d30110ef0 con 0x7f4d30106f90 2024-04-23T10:34:10.404 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:34:10.400+0000 7f4d35abb640 1 -- --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_getmap magic: 0 -- 0x7f4d301111f0 con 0x7f4d30101af0 2024-04-23T10:34:10.404 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:34:10.400+0000 7f4d2f7fe640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f4d30101130 0x7f4d301986c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T10:34:10.404 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:34:10.400+0000 7f4d2f7fe640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f4d30101130 0x7f4d301986c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.114:48222/0 (socket says 172.21.15.114:48222) 2024-04-23T10:34:10.404 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:34:10.400+0000 7f4d2f7fe640 1 -- 172.21.15.114:0/3287179793 learned_addr learned my addr 172.21.15.114:0/3287179793 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-04-23T10:34:10.404 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:34:10.400+0000 7f4d2ffff640 1 --2- 172.21.15.114:0/3287179793 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f4d30106f90 0x7f4d3019cf80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T10:34:10.404 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:34:10.401+0000 7f4d2effd640 1 --2- 172.21.15.114:0/3287179793 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f4d30101af0 0x7f4d30198c00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T10:34:10.404 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:34:10.401+0000 7f4d2f7fe640 1 -- 172.21.15.114:0/3287179793 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f4d30101af0 msgr2=0x7f4d30198c00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T10:34:10.404 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:34:10.401+0000 7f4d2f7fe640 1 --2- 172.21.15.114:0/3287179793 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f4d30101af0 0x7f4d30198c00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:34:10.404 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:34:10.401+0000 7f4d2f7fe640 1 -- 172.21.15.114:0/3287179793 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f4d30106f90 msgr2=0x7f4d3019cf80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T10:34:10.404 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:34:10.401+0000 7f4d2f7fe640 1 --2- 172.21.15.114:0/3287179793 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f4d30106f90 0x7f4d3019cf80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:34:10.405 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:34:10.401+0000 7f4d2f7fe640 1 -- 172.21.15.114:0/3287179793 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f4d3019d6b0 con 0x7f4d30101130 2024-04-23T10:34:10.405 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:34:10.401+0000 7f4d2f7fe640 1 --2- 172.21.15.114:0/3287179793 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f4d30101130 0x7f4d301986c0 secure :-1 s=READY pgs=780 cs=0 l=1 rev1=1 crypto rx=0x7f4d2000e9f0 tx=0x7f4d2000eeb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T10:34:10.405 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:34:10.401+0000 7f4d2cff9640 1 -- 172.21.15.114:0/3287179793 <== mon.0 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4d2000cd80 con 0x7f4d30101130 2024-04-23T10:34:10.405 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:34:10.401+0000 7f4d2cff9640 1 -- 172.21.15.114:0/3287179793 <== mon.0 v2:172.21.15.114:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f4d200108e0 con 0x7f4d30101130 2024-04-23T10:34:10.405 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:34:10.401+0000 7f4d35abb640 1 -- 172.21.15.114:0/3287179793 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f4d3019d9a0 con 0x7f4d30101130 2024-04-23T10:34:10.405 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:34:10.401+0000 7f4d2cff9640 1 -- 172.21.15.114:0/3287179793 <== mon.0 v2:172.21.15.114:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4d20018980 con 0x7f4d30101130 2024-04-23T10:34:10.405 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:34:10.401+0000 7f4d35abb640 1 -- 172.21.15.114:0/3287179793 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f4d301a5230 con 0x7f4d30101130 2024-04-23T10:34:10.413 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:34:10.403+0000 7f4d2cff9640 1 -- 172.21.15.114:0/3287179793 <== mon.0 v2:172.21.15.114:3300/0 4 ==== mgrmap(e 26) ==== 138778+0+0 (secure 0 0 0) 0x7f4d20010430 con 0x7f4d30101130 2024-04-23T10:34:10.414 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:34:10.404+0000 7f4d387cc740 1 Processor -- start 2024-04-23T10:34:10.414 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:34:10.404+0000 7f4d387cc740 1 -- start start 2024-04-23T10:34:10.414 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:34:10.404+0000 7f4d2cff9640 1 --2- 172.21.15.114:0/3287179793 >> [v2:172.21.15.114:6800/1395693466,v1:172.21.15.114:6801/1395693466] conn(0x7f4d180a3dc0 0x7f4d180a6280 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T10:34:10.414 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:34:10.404+0000 7f4d387cc740 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x555587e08620 0x555587e08a00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T10:34:10.414 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:34:10.404+0000 7f4d387cc740 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x555587e08f40 0x555587e132c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T10:34:10.414 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:34:10.404+0000 7f4d2ffff640 1 --2- 172.21.15.114:0/3287179793 >> [v2:172.21.15.114:6800/1395693466,v1:172.21.15.114:6801/1395693466] conn(0x7f4d180a3dc0 0x7f4d180a6280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T10:34:10.414 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:34:10.404+0000 7f4d387cc740 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x555587e13800 0x555587e15bf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T10:34:10.415 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:34:10.404+0000 7f4d387cc740 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x555587d6d970 con 0x555587e08620 2024-04-23T10:34:10.415 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:34:10.404+0000 7f4d387cc740 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 -- 0x555587d0a4a0 con 0x555587e08f40 2024-04-23T10:34:10.415 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:34:10.404+0000 7f4d387cc740 1 -- --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_getmap magic: 0 -- 0x555587d69930 con 0x555587e13800 2024-04-23T10:34:10.415 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:34:10.404+0000 7f4d2cff9640 1 -- 172.21.15.114:0/3287179793 <== mon.0 v2:172.21.15.114:3300/0 5 ==== osd_map(98..98 src has 1..98) ==== 8629+0+0 (secure 0 0 0) 0x7f4d200d0b00 con 0x7f4d30101130 2024-04-23T10:34:10.415 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:34:10.404+0000 7f4d2effd640 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x555587e08f40 0x555587e132c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T10:34:10.415 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:34:10.404+0000 7f4d2f7fe640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x555587e08620 0x555587e08a00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T10:34:10.415 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:34:10.404+0000 7f4d2effd640 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x555587e08f40 0x555587e132c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.119:3300/0 says I am v2:172.21.15.114:38468/0 (socket says 172.21.15.114:38468) 2024-04-23T10:34:10.415 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:34:10.404+0000 7f4d2ffff640 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x555587e13800 0x555587e15bf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T10:34:10.415 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:34:10.404+0000 7f4d2f7fe640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x555587e08620 0x555587e08a00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.114:48226/0 (socket says 172.21.15.114:48226) 2024-04-23T10:34:10.415 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:34:10.404+0000 7f4d2effd640 1 -- 172.21.15.114:0/3074930611 learned_addr learned my addr 172.21.15.114:0/3074930611 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-04-23T10:34:10.415 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:34:10.404+0000 7f4d2f7fe640 1 -- 172.21.15.114:0/3074930611 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x555587e13800 msgr2=0x555587e15bf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T10:34:10.416 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:34:10.404+0000 7f4d2f7fe640 1 --2- 172.21.15.114:0/3074930611 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x555587e13800 0x555587e15bf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:34:10.416 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:34:10.404+0000 7f4d2f7fe640 1 -- 172.21.15.114:0/3074930611 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x555587e08f40 msgr2=0x555587e132c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T10:34:10.416 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:34:10.404+0000 7f4d2f7fe640 1 --2- 172.21.15.114:0/3074930611 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x555587e08f40 0x555587e132c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:34:10.416 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:34:10.404+0000 7f4d2f7fe640 1 -- 172.21.15.114:0/3074930611 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x555587a80210 con 0x555587e08620 2024-04-23T10:34:10.416 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:34:10.404+0000 7f4d2effd640 1 --2- 172.21.15.114:0/3074930611 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x555587e08f40 0x555587e132c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-23T10:34:10.416 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:34:10.404+0000 7f4d2ffff640 1 --2- 172.21.15.114:0/3074930611 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x555587e13800 0x555587e15bf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-23T10:34:10.416 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:34:10.404+0000 7f4d2ffff640 1 --2- 172.21.15.114:0/3287179793 >> [v2:172.21.15.114:6800/1395693466,v1:172.21.15.114:6801/1395693466] conn(0x7f4d180a3dc0 0x7f4d180a6280 secure :-1 s=READY pgs=118 cs=0 l=1 rev1=1 crypto rx=0x7f4d28005fd0 tx=0x7f4d28004410 comp rx=0 tx=0).ready entity=mgr.15291 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T10:34:10.416 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:34:10.404+0000 7f4d2f7fe640 1 --2- 172.21.15.114:0/3074930611 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x555587e08620 0x555587e08a00 secure :-1 s=READY pgs=781 cs=0 l=1 rev1=1 crypto rx=0x7f4d200ce1e0 tx=0x7f4d20010720 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9fc071cda7e3147 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T10:34:10.416 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:34:10.405+0000 7f4d15ffb640 1 -- 172.21.15.114:0/3074930611 <== mon.0 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4d200a0de0 con 0x555587e08620 2024-04-23T10:34:10.416 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:34:10.405+0000 7f4d15ffb640 1 -- 172.21.15.114:0/3074930611 <== mon.0 v2:172.21.15.114:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f4d200a56a0 con 0x555587e08620 2024-04-23T10:34:10.416 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:34:10.405+0000 7f4d15ffb640 1 -- 172.21.15.114:0/3074930611 <== mon.0 v2:172.21.15.114:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4d20018b20 con 0x555587e08620 2024-04-23T10:34:10.417 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:34:10.405+0000 7f4d387cc740 1 -- 172.21.15.114:0/3074930611 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x555587e08620 msgr2=0x555587e08a00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T10:34:10.417 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:34:10.405+0000 7f4d387cc740 1 --2- 172.21.15.114:0/3074930611 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x555587e08620 0x555587e08a00 secure :-1 s=READY pgs=781 cs=0 l=1 rev1=1 crypto rx=0x7f4d200ce1e0 tx=0x7f4d20010720 comp rx=0 tx=0).stop 2024-04-23T10:34:10.417 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:34:10.405+0000 7f4d387cc740 1 -- 172.21.15.114:0/3074930611 shutdown_connections 2024-04-23T10:34:10.417 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:34:10.405+0000 7f4d387cc740 1 --2- 172.21.15.114:0/3074930611 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x555587e13800 0x555587e15bf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:34:10.417 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:34:10.405+0000 7f4d387cc740 1 --2- 172.21.15.114:0/3074930611 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x555587e08f40 0x555587e132c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:34:10.417 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:34:10.405+0000 7f4d387cc740 1 --2- 172.21.15.114:0/3074930611 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x555587e08620 0x555587e08a00 unknown :-1 s=CLOSED pgs=781 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:34:10.417 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:34:10.405+0000 7f4d387cc740 1 -- 172.21.15.114:0/3074930611 >> 172.21.15.114:0/3074930611 conn(0x555587e04b30 msgr2=0x555587e04f10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T10:34:10.417 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:34:10.405+0000 7f4d387cc740 1 -- 172.21.15.114:0/3074930611 shutdown_connections 2024-04-23T10:34:10.417 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:34:10.405+0000 7f4d387cc740 1 -- 172.21.15.114:0/3074930611 wait complete. 2024-04-23T10:34:10.417 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:34:10.406+0000 7f4d387cc740 1 -- start start 2024-04-23T10:34:10.417 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:34:10.406+0000 7f4d387cc740 1 Processor -- start 2024-04-23T10:34:10.418 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:34:10.406+0000 7f4d387cc740 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x555587e08620 0x555587e130b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T10:34:10.418 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:34:10.406+0000 7f4d387cc740 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x555587e08f40 0x555587e10410 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T10:34:10.418 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:34:10.406+0000 7f4d387cc740 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x555587e13800 0x555587e10950 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T10:34:10.418 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:34:10.406+0000 7f4d387cc740 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x555587e163d0 con 0x555587e08620 2024-04-23T10:34:10.418 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:34:10.406+0000 7f4d387cc740 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 -- 0x555587d080b0 con 0x555587e08f40 2024-04-23T10:34:10.418 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:34:10.406+0000 7f4d387cc740 1 -- --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_getmap magic: 0 -- 0x555587e16550 con 0x555587e13800 2024-04-23T10:34:10.418 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:34:10.406+0000 7f4d387cc740 10 client.0 _ll_register_callbacks cb 0x555587cf5d30 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-23T10:34:10.418 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:34:10.406+0000 7f4d2f7fe640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x555587e08620 0x555587e130b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T10:34:10.418 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:34:10.406+0000 7f4d2f7fe640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x555587e08620 0x555587e130b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.114:48242/0 (socket says 172.21.15.114:48242) 2024-04-23T10:34:10.418 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:34:10.406+0000 7f4d2f7fe640 1 -- 172.21.15.114:0/1638075031 learned_addr learned my addr 172.21.15.114:0/1638075031 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-04-23T10:34:10.419 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:34:10.406+0000 7f4d2ffff640 1 --2- 172.21.15.114:0/1638075031 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x555587e13800 0x555587e10950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T10:34:10.419 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:34:10.406+0000 7f4d2effd640 1 --2- 172.21.15.114:0/1638075031 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x555587e08f40 0x555587e10410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T10:34:10.419 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:34:10.407+0000 7f4d2f7fe640 1 -- 172.21.15.114:0/1638075031 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x555587e13800 msgr2=0x555587e10950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T10:34:10.419 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:34:10.407+0000 7f4d2f7fe640 1 --2- 172.21.15.114:0/1638075031 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x555587e13800 0x555587e10950 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:34:10.419 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:34:10.407+0000 7f4d2f7fe640 1 -- 172.21.15.114:0/1638075031 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x555587e08f40 msgr2=0x555587e10410 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T10:34:10.419 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:34:10.407+0000 7f4d2f7fe640 1 --2- 172.21.15.114:0/1638075031 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x555587e08f40 0x555587e10410 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:34:10.419 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:34:10.407+0000 7f4d2f7fe640 1 -- 172.21.15.114:0/1638075031 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({osdmap=0}) -- 0x555587b4c620 con 0x555587e08620 2024-04-23T10:34:10.419 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:34:10.407+0000 7f4d2ffff640 1 --2- 172.21.15.114:0/1638075031 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x555587e13800 0x555587e10950 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-23T10:34:10.419 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:34:10.407+0000 7f4d2effd640 1 --2- 172.21.15.114:0/1638075031 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x555587e08f40 0x555587e10410 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-23T10:34:10.419 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:34:10.407+0000 7f4d2f7fe640 1 --2- 172.21.15.114:0/1638075031 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x555587e08620 0x555587e130b0 secure :-1 s=READY pgs=782 cs=0 l=1 rev1=1 crypto rx=0x7f4d200994b0 tx=0x7f4d20000bc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d1d55874d3f2d88c server_cookie=0 in_seq=0 out_seq=0 2024-04-23T10:34:10.420 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:34:10.407+0000 7f4d0effd640 10 client.0 ms_handle_connect on v2:172.21.15.114:3300/0 2024-04-23T10:34:10.420 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:34:10.407+0000 7f4d0effd640 1 -- 172.21.15.114:0/1638075031 <== mon.0 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4d200b5070 con 0x555587e08620 2024-04-23T10:34:10.420 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:34:10.407+0000 7f4d0effd640 1 -- 172.21.15.114:0/1638075031 <== mon.0 v2:172.21.15.114:3300/0 2 ==== osd_map(98..98 src has 1..98) ==== 8629+0+0 (secure 0 0 0) 0x7f4d2009bf30 con 0x555587e08620 2024-04-23T10:34:10.420 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:34:10.407+0000 7f4d387cc740 1 -- 172.21.15.114:0/1638075031 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f4d200a56a0 con 0x555587e08620 2024-04-23T10:34:10.420 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:34:10.407+0000 7f4d0effd640 1 -- 172.21.15.114:0/1638075031 <== mon.0 v2:172.21.15.114:3300/0 3 ==== mon_get_version_reply(handle=1 version=31) ==== 24+0+0 (secure 0 0 0) 0x7f4d2009c130 con 0x555587e08620 2024-04-23T10:34:10.420 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:34:10.407+0000 7f4d387cc740 10 client.16566 fetch_fsmap learned FSMap version 31 2024-04-23T10:34:10.420 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:34:10.407+0000 7f4d387cc740 1 -- 172.21.15.114:0/1638075031 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({config=0+,fsmap=31,monmap=4+}) -- 0x555587e222f0 con 0x555587e08620 2024-04-23T10:34:10.420 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:34:10.407+0000 7f4d0effd640 1 -- 172.21.15.114:0/1638075031 <== mon.0 v2:172.21.15.114:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f4d200a56a0 con 0x555587e08620 2024-04-23T10:34:10.420 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:34:10.407+0000 7f4d0effd640 1 -- 172.21.15.114:0/1638075031 <== mon.0 v2:172.21.15.114:3300/0 5 ==== fsmap(e 31) ==== 8342+0+0 (secure 0 0 0) 0x7f4d2009b9a0 con 0x555587e08620 2024-04-23T10:34:10.420 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:34:10.407+0000 7f4d387cc740 10 client.16566 fetch_fsmap finished waiting for FSMap version 31 2024-04-23T10:34:10.420 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:34:10.407+0000 7f4d387cc740 10 client.16566 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-04-23T10:34:10.421 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:34:10.407+0000 7f4d387cc740 20 client.16566 populate_metadata read hostname 'smithi114' 2024-04-23T10:34:10.421 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:34:10.407+0000 7f4d387cc740 1 --2- 172.21.15.114:0/1638075031 >> [v2:172.21.15.119:6838/937297340,v1:172.21.15.119:6839/937297340] conn(0x555587e23640 0x555587e25a30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T10:34:10.421 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:34:10.407+0000 7f4d387cc740 4 client.16566 mds_command: new command op to 24545 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-23T10:34:10.421 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:34:10.407+0000 7f4d387cc740 1 -- 172.21.15.114:0/1638075031 --> [v2:172.21.15.119:6838/937297340,v1:172.21.15.119:6839/937297340] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x555587d7b390 con 0x555587e23640 2024-04-23T10:34:10.421 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:34:10.408+0000 7f4d2effd640 1 --2- 172.21.15.114:0/1638075031 >> [v2:172.21.15.119:6838/937297340,v1:172.21.15.119:6839/937297340] conn(0x555587e23640 0x555587e25a30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T10:34:10.421 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:34:10.408+0000 7f4d2effd640 1 --2- 172.21.15.114:0/1638075031 >> [v2:172.21.15.119:6838/937297340,v1:172.21.15.119:6839/937297340] conn(0x555587e23640 0x555587e25a30 crc :-1 s=READY pgs=128 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=5fd07ede6331f0b2 server_cookie=32c577b539573258 in_seq=0 out_seq=0 2024-04-23T10:34:10.421 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:34:10.408+0000 7f4d0effd640 10 client.16566 ms_handle_connect on v2:172.21.15.119:6838/937297340 2024-04-23T10:34:10.421 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:34:10.408+0000 7f4d0effd640 1 -- 172.21.15.114:0/1638075031 <== mds.0 v2:172.21.15.119:6838/937297340 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x555587d7b390 con 0x555587e23640 2024-04-23T10:34:10.421 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:34:10.408+0000 7f4d0effd640 10 client.16566 handle_command_reply: tid=0 2024-04-23T10:34:10.421 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:34:10.408+0000 7f4d387cc740 1 client.16566 shutdown 2024-04-23T10:34:10.421 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:34:10.409+0000 7f4d387cc740 1 -- 172.21.15.114:0/1638075031 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x555587e08620 msgr2=0x555587e130b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T10:34:10.422 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:34:10.409+0000 7f4d387cc740 1 --2- 172.21.15.114:0/1638075031 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x555587e08620 0x555587e130b0 secure :-1 s=READY pgs=782 cs=0 l=1 rev1=1 crypto rx=0x7f4d200994b0 tx=0x7f4d20000bc0 comp rx=0 tx=0).stop 2024-04-23T10:34:10.422 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:34:10.409+0000 7f4d387cc740 1 -- 172.21.15.114:0/1638075031 shutdown_connections 2024-04-23T10:34:10.422 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:34:10.409+0000 7f4d387cc740 1 --2- 172.21.15.114:0/1638075031 >> [v2:172.21.15.119:6838/937297340,v1:172.21.15.119:6839/937297340] conn(0x555587e23640 0x555587e25a30 crc :-1 s=READY pgs=128 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:34:10.422 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:34:10.409+0000 7f4d387cc740 1 --2- 172.21.15.114:0/1638075031 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x555587e13800 0x555587e10950 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:34:10.422 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:34:10.409+0000 7f4d0effd640 0 client.16566 ms_handle_reset on v2:172.21.15.119:6838/937297340 2024-04-23T10:34:10.422 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:34:10.409+0000 7f4d387cc740 1 --2- 172.21.15.114:0/1638075031 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x555587e08f40 0x555587e10410 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:34:10.422 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:34:10.409+0000 7f4d387cc740 1 --2- 172.21.15.114:0/1638075031 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x555587e08620 0x555587e130b0 unknown :-1 s=CLOSED pgs=782 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:34:10.422 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:34:10.409+0000 7f4d387cc740 1 -- 172.21.15.114:0/1638075031 >> 172.21.15.114:0/1638075031 conn(0x555587e04b30 msgr2=0x555587e0a8b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T10:34:10.422 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:34:10.409+0000 7f4d387cc740 1 -- 172.21.15.114:0/1638075031 shutdown_connections 2024-04-23T10:34:10.423 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:34:10.409+0000 7f4d387cc740 1 -- 172.21.15.114:0/1638075031 wait complete. 2024-04-23T10:34:10.423 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:34:10.409+0000 7f4d387cc740 20 client.16566 trim_cache size 0 max 16384 2024-04-23T10:34:10.428 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:34:10.424+0000 7f4d387cc740 1 Processor -- start 2024-04-23T10:34:10.428 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:34:10.424+0000 7f4d387cc740 1 -- start start 2024-04-23T10:34:10.429 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:34:10.424+0000 7f4d387cc740 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x555587e08620 0x555587e1f290 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T10:34:10.429 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:34:10.424+0000 7f4d387cc740 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x555587e08f40 0x555587e12c50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T10:34:10.429 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:34:10.424+0000 7f4d387cc740 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x555587e13800 0x555587e2b090 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T10:34:10.429 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:34:10.424+0000 7f4d387cc740 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x7f4d301b0640 con 0x555587e08f40 2024-04-23T10:34:10.429 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:34:10.424+0000 7f4d387cc740 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 -- 0x7f4d301adee0 con 0x555587e13800 2024-04-23T10:34:10.429 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:34:10.424+0000 7f4d387cc740 1 -- --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_getmap magic: 0 -- 0x7f4d301abd50 con 0x555587e08620 2024-04-23T10:34:10.429 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:34:10.424+0000 7f4d2effd640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x555587e08f40 0x555587e12c50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T10:34:10.429 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:34:10.424+0000 7f4d2effd640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x555587e08f40 0x555587e12c50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.114:48256/0 (socket says 172.21.15.114:48256) 2024-04-23T10:34:10.429 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:34:10.424+0000 7f4d2effd640 1 -- 172.21.15.114:0/2291758897 learned_addr learned my addr 172.21.15.114:0/2291758897 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-04-23T10:34:10.429 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:34:10.424+0000 7f4d2ffff640 1 --2- 172.21.15.114:0/2291758897 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x555587e13800 0x555587e2b090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T10:34:10.429 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:34:10.424+0000 7f4d2f7fe640 1 --2- 172.21.15.114:0/2291758897 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x555587e08620 0x555587e1f290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T10:34:10.429 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:34:10.424+0000 7f4d2effd640 1 -- 172.21.15.114:0/2291758897 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x555587e08620 msgr2=0x555587e1f290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T10:34:10.430 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:34:10.424+0000 7f4d2effd640 1 --2- 172.21.15.114:0/2291758897 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x555587e08620 0x555587e1f290 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:34:10.430 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:34:10.424+0000 7f4d2effd640 1 -- 172.21.15.114:0/2291758897 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x555587e13800 msgr2=0x555587e2b090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T10:34:10.430 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:34:10.424+0000 7f4d2effd640 1 --2- 172.21.15.114:0/2291758897 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x555587e13800 0x555587e2b090 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:34:10.430 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:34:10.424+0000 7f4d2effd640 1 -- 172.21.15.114:0/2291758897 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x555587e0e980 con 0x555587e08f40 2024-04-23T10:34:10.430 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:34:10.425+0000 7f4d2effd640 1 --2- 172.21.15.114:0/2291758897 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x555587e08f40 0x555587e12c50 secure :-1 s=READY pgs=783 cs=0 l=1 rev1=1 crypto rx=0x7f4d24031440 tx=0x7f4d24005fd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=432c09ad73afe027 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T10:34:10.430 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:34:10.425+0000 7f4d157fa640 1 -- 172.21.15.114:0/2291758897 <== mon.0 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4d2404b070 con 0x555587e08f40 2024-04-23T10:34:10.430 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:34:10.425+0000 7f4d157fa640 1 -- 172.21.15.114:0/2291758897 <== mon.0 v2:172.21.15.114:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f4d24046dc0 con 0x555587e08f40 2024-04-23T10:34:10.430 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:34:10.425+0000 7f4d157fa640 1 -- 172.21.15.114:0/2291758897 <== mon.0 v2:172.21.15.114:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4d24035040 con 0x555587e08f40 2024-04-23T10:34:10.430 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:34:10.425+0000 7f4d387cc740 1 -- 172.21.15.114:0/2291758897 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x555587e08f40 msgr2=0x555587e12c50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T10:34:10.430 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:34:10.425+0000 7f4d387cc740 1 --2- 172.21.15.114:0/2291758897 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x555587e08f40 0x555587e12c50 secure :-1 s=READY pgs=783 cs=0 l=1 rev1=1 crypto rx=0x7f4d24031440 tx=0x7f4d24005fd0 comp rx=0 tx=0).stop 2024-04-23T10:34:10.431 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:34:10.425+0000 7f4d387cc740 1 -- 172.21.15.114:0/2291758897 shutdown_connections 2024-04-23T10:34:10.431 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:34:10.425+0000 7f4d387cc740 1 --2- 172.21.15.114:0/2291758897 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x555587e13800 0x555587e2b090 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:34:10.431 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:34:10.425+0000 7f4d387cc740 1 --2- 172.21.15.114:0/2291758897 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x555587e08f40 0x555587e12c50 unknown :-1 s=CLOSED pgs=783 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:34:10.431 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:34:10.425+0000 7f4d387cc740 1 --2- 172.21.15.114:0/2291758897 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x555587e08620 0x555587e1f290 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:34:10.431 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:34:10.425+0000 7f4d387cc740 1 -- 172.21.15.114:0/2291758897 >> 172.21.15.114:0/2291758897 conn(0x555587e04b30 msgr2=0x555587e103d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T10:34:10.431 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:34:10.426+0000 7f4d387cc740 1 -- 172.21.15.114:0/2291758897 shutdown_connections 2024-04-23T10:34:10.431 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:34:10.426+0000 7f4d387cc740 1 -- 172.21.15.114:0/2291758897 wait complete. 2024-04-23T10:34:10.431 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:34:10.426+0000 7f4d387cc740 1 -- start start 2024-04-23T10:34:10.431 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:34:10.426+0000 7f4d387cc740 1 Processor -- start 2024-04-23T10:34:10.431 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:34:10.426+0000 7f4d387cc740 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x555587e08620 0x555587e1b7a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T10:34:10.431 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:34:10.426+0000 7f4d387cc740 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x555587e08f40 0x555587e1bce0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T10:34:10.431 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:34:10.426+0000 7f4d387cc740 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x555587e13800 0x555587e0c8a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T10:34:10.432 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:34:10.426+0000 7f4d387cc740 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x7f4d301ac3a0 con 0x555587e13800 2024-04-23T10:34:10.432 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:34:10.426+0000 7f4d387cc740 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 -- 0x7f4d301ad810 con 0x555587e08f40 2024-04-23T10:34:10.432 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:34:10.426+0000 7f4d387cc740 1 -- --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_getmap magic: 0 -- 0x7f4d301aca70 con 0x555587e08620 2024-04-23T10:34:10.432 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:34:10.426+0000 7f4d387cc740 10 client.0 _ll_register_callbacks cb 0x7f4d2009c710 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-23T10:34:10.432 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:34:10.426+0000 7f4d2ffff640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x555587e13800 0x555587e0c8a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T10:34:10.432 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:34:10.426+0000 7f4d2effd640 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x555587e08f40 0x555587e1bce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T10:34:10.432 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:34:10.426+0000 7f4d2ffff640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x555587e13800 0x555587e0c8a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.114:48260/0 (socket says 172.21.15.114:48260) 2024-04-23T10:34:10.432 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:34:10.426+0000 7f4d2ffff640 1 -- 172.21.15.114:0/1061292869 learned_addr learned my addr 172.21.15.114:0/1061292869 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-04-23T10:34:10.432 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:34:10.426+0000 7f4d2f7fe640 1 --2- 172.21.15.114:0/1061292869 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x555587e08620 0x555587e1b7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T10:34:10.432 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:34:10.427+0000 7f4d2effd640 1 -- 172.21.15.114:0/1061292869 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x555587e08620 msgr2=0x555587e1b7a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T10:34:10.432 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:34:10.427+0000 7f4d2effd640 1 --2- 172.21.15.114:0/1061292869 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x555587e08620 0x555587e1b7a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:34:10.433 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:34:10.427+0000 7f4d2effd640 1 -- 172.21.15.114:0/1061292869 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x555587e13800 msgr2=0x555587e0c8a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T10:34:10.433 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:34:10.427+0000 7f4d2effd640 1 --2- 172.21.15.114:0/1061292869 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x555587e13800 0x555587e0c8a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:34:10.433 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:34:10.427+0000 7f4d2effd640 1 -- 172.21.15.114:0/1061292869 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({osdmap=0}) -- 0x555587e0d0d0 con 0x555587e08f40 2024-04-23T10:34:10.433 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:34:10.427+0000 7f4d2ffff640 1 --2- 172.21.15.114:0/1061292869 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x555587e13800 0x555587e0c8a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-23T10:34:10.433 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:34:10.427+0000 7f4d2f7fe640 1 --2- 172.21.15.114:0/1061292869 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x555587e08620 0x555587e1b7a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-23T10:34:10.433 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:34:10.427+0000 7f4d2effd640 1 --2- 172.21.15.114:0/1061292869 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x555587e08f40 0x555587e1bce0 secure :-1 s=READY pgs=83 cs=0 l=1 rev1=1 crypto rx=0x7f4d24031440 tx=0x7f4d2402a420 comp rx=0 tx=0).ready entity=mon.2 client_cookie=fea01023828aac14 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T10:34:10.433 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:34:10.427+0000 7f4d0effd640 10 client.0 ms_handle_connect on v2:172.21.15.119:3300/0 2024-04-23T10:34:10.433 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:34:10.427+0000 7f4d0effd640 1 -- 172.21.15.114:0/1061292869 <== mon.2 v2:172.21.15.119:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4d24035070 con 0x555587e08f40 2024-04-23T10:34:10.433 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:34:10.427+0000 7f4d387cc740 1 -- 172.21.15.114:0/1061292869 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f4d24046dc0 con 0x555587e08f40 2024-04-23T10:34:10.433 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:34:10.427+0000 7f4d0effd640 1 -- 172.21.15.114:0/1061292869 <== mon.2 v2:172.21.15.119:3300/0 2 ==== osd_map(98..98 src has 1..98) ==== 8629+0+0 (secure 0 0 0) 0x7f4d24004380 con 0x555587e08f40 2024-04-23T10:34:10.433 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:34:10.427+0000 7f4d0effd640 1 -- 172.21.15.114:0/1061292869 <== mon.2 v2:172.21.15.119:3300/0 3 ==== mon_get_version_reply(handle=1 version=31) ==== 24+0+0 (secure 0 0 0) 0x7f4d2405c050 con 0x555587e08f40 2024-04-23T10:34:10.433 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:34:10.427+0000 7f4d387cc740 10 client.25940 fetch_fsmap learned FSMap version 31 2024-04-23T10:34:10.434 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:34:10.427+0000 7f4d387cc740 1 -- 172.21.15.114:0/1061292869 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({config=0+,fsmap=31,monmap=4+}) -- 0x555587e89430 con 0x555587e08f40 2024-04-23T10:34:10.434 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:34:10.428+0000 7f4d0effd640 1 -- 172.21.15.114:0/1061292869 <== mon.2 v2:172.21.15.119:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f4d24046dc0 con 0x555587e08f40 2024-04-23T10:34:10.434 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:34:10.428+0000 7f4d0effd640 1 -- 172.21.15.114:0/1061292869 <== mon.2 v2:172.21.15.119:3300/0 5 ==== fsmap(e 31) ==== 8342+0+0 (secure 0 0 0) 0x7f4d24057030 con 0x555587e08f40 2024-04-23T10:34:10.434 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:34:10.428+0000 7f4d387cc740 10 client.25940 fetch_fsmap finished waiting for FSMap version 31 2024-04-23T10:34:10.434 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:34:10.428+0000 7f4d387cc740 10 client.25940 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-04-23T10:34:10.434 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:34:10.428+0000 7f4d387cc740 20 client.25940 populate_metadata read hostname 'smithi114' 2024-04-23T10:34:10.434 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:34:10.428+0000 7f4d387cc740 1 --2- 172.21.15.114:0/1061292869 >> [v2:172.21.15.119:6838/937297340,v1:172.21.15.119:6839/937297340] conn(0x555587e89ad0 0x555587e8bef0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T10:34:10.434 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:34:10.428+0000 7f4d387cc740 4 client.25940 mds_command: new command op to 24545 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-04-23T10:34:10.434 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:34:10.428+0000 7f4d387cc740 1 -- 172.21.15.114:0/1061292869 --> [v2:172.21.15.119:6838/937297340,v1:172.21.15.119:6839/937297340] -- command(tid 0: {"prefix": "flush journal"}) -- 0x555587d75200 con 0x555587e89ad0 2024-04-23T10:34:10.434 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:34:10.428+0000 7f4d2f7fe640 1 --2- 172.21.15.114:0/1061292869 >> [v2:172.21.15.119:6838/937297340,v1:172.21.15.119:6839/937297340] conn(0x555587e89ad0 0x555587e8bef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T10:34:10.434 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:34:10.428+0000 7f4d0effd640 10 client.25940 ms_handle_connect on v2:172.21.15.119:6838/937297340 2024-04-23T10:34:10.435 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:34:10.428+0000 7f4d2f7fe640 1 --2- 172.21.15.114:0/1061292869 >> [v2:172.21.15.119:6838/937297340,v1:172.21.15.119:6839/937297340] conn(0x555587e89ad0 0x555587e8bef0 crc :-1 s=READY pgs=129 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=9c6c8ddc43fb433c server_cookie=1fa38c5b5619c70e in_seq=0 out_seq=0 2024-04-23T10:34:10.500 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:34:10.500+0000 7f4d0effd640 1 -- 172.21.15.114:0/1061292869 <== mds.0 v2:172.21.15.119:6838/937297340 1 ==== command_reply(tid 0: 0 ) ==== 8+0+44 (crc 0 0 0) 0x555587d75200 con 0x555587e89ad0 2024-04-23T10:34:10.500 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:34:10.500+0000 7f4d0effd640 10 client.25940 handle_command_reply: tid=0 2024-04-23T10:34:10.500 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:34:10.500+0000 7f4d387cc740 1 client.25940 shutdown 2024-04-23T10:34:10.500 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:34:10.500+0000 7f4d387cc740 1 -- 172.21.15.114:0/1061292869 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x555587e08f40 msgr2=0x555587e1bce0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T10:34:10.500 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:34:10.500+0000 7f4d387cc740 1 --2- 172.21.15.114:0/1061292869 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x555587e08f40 0x555587e1bce0 secure :-1 s=READY pgs=83 cs=0 l=1 rev1=1 crypto rx=0x7f4d24031440 tx=0x7f4d2402a420 comp rx=0 tx=0).stop 2024-04-23T10:34:10.501 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:34:10.500+0000 7f4d387cc740 1 -- 172.21.15.114:0/1061292869 shutdown_connections 2024-04-23T10:34:10.501 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:34:10.500+0000 7f4d387cc740 1 --2- 172.21.15.114:0/1061292869 >> [v2:172.21.15.119:6838/937297340,v1:172.21.15.119:6839/937297340] conn(0x555587e89ad0 0x555587e8bef0 crc :-1 s=READY pgs=129 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:34:10.501 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:34:10.500+0000 7f4d387cc740 1 --2- 172.21.15.114:0/1061292869 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x555587e13800 0x555587e0c8a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:34:10.501 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:34:10.500+0000 7f4d387cc740 1 --2- 172.21.15.114:0/1061292869 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x555587e08f40 0x555587e1bce0 unknown :-1 s=CLOSED pgs=83 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:34:10.501 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:34:10.500+0000 7f4d0effd640 0 client.25940 ms_handle_reset on v2:172.21.15.119:6838/937297340 2024-04-23T10:34:10.501 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:34:10.500+0000 7f4d387cc740 1 --2- 172.21.15.114:0/1061292869 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x555587e08620 0x555587e1b7a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:34:10.501 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:34:10.501+0000 7f4d387cc740 1 -- 172.21.15.114:0/1061292869 >> 172.21.15.114:0/1061292869 conn(0x555587e04b30 msgr2=0x555587e03420 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T10:34:10.501 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:34:10.501+0000 7f4d387cc740 1 -- 172.21.15.114:0/1061292869 shutdown_connections 2024-04-23T10:34:10.501 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:34:10.501+0000 7f4d387cc740 1 -- 172.21.15.114:0/1061292869 wait complete. 2024-04-23T10:34:10.501 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:34:10.501+0000 7f4d387cc740 20 client.25940 trim_cache size 0 max 16384 2024-04-23T10:34:10.502 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stdout:{ 2024-04-23T10:34:10.502 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stdout: "message": "", 2024-04-23T10:34:10.502 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stdout: "return_code": 0 2024-04-23T10:34:10.502 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stdout:} 2024-04-23T10:34:10.503 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:34:10.501+0000 7f4d35abb640 1 -- 172.21.15.114:0/3287179793 >> [v2:172.21.15.114:6800/1395693466,v1:172.21.15.114:6801/1395693466] conn(0x7f4d180a3dc0 msgr2=0x7f4d180a6280 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T10:34:10.503 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:34:10.501+0000 7f4d35abb640 1 --2- 172.21.15.114:0/3287179793 >> [v2:172.21.15.114:6800/1395693466,v1:172.21.15.114:6801/1395693466] conn(0x7f4d180a3dc0 0x7f4d180a6280 secure :-1 s=READY pgs=118 cs=0 l=1 rev1=1 crypto rx=0x7f4d28005fd0 tx=0x7f4d28004410 comp rx=0 tx=0).stop 2024-04-23T10:34:10.504 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:34:10.501+0000 7f4d35abb640 1 -- 172.21.15.114:0/3287179793 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f4d30101130 msgr2=0x7f4d301986c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T10:34:10.504 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:34:10.501+0000 7f4d35abb640 1 --2- 172.21.15.114:0/3287179793 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f4d30101130 0x7f4d301986c0 secure :-1 s=READY pgs=780 cs=0 l=1 rev1=1 crypto rx=0x7f4d2000e9f0 tx=0x7f4d2000eeb0 comp rx=0 tx=0).stop 2024-04-23T10:34:10.504 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:34:10.502+0000 7f4d35abb640 1 -- 172.21.15.114:0/3287179793 shutdown_connections 2024-04-23T10:34:10.504 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:34:10.502+0000 7f4d35abb640 1 --2- 172.21.15.114:0/3287179793 >> [v2:172.21.15.114:6800/1395693466,v1:172.21.15.114:6801/1395693466] conn(0x7f4d180a3dc0 0x7f4d180a6280 unknown :-1 s=CLOSED pgs=118 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:34:10.504 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:34:10.502+0000 7f4d35abb640 1 --2- 172.21.15.114:0/3287179793 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f4d30106f90 0x7f4d3019cf80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:34:10.504 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:34:10.502+0000 7f4d35abb640 1 --2- 172.21.15.114:0/3287179793 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f4d30101af0 0x7f4d30198c00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:34:10.504 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:34:10.502+0000 7f4d35abb640 1 --2- 172.21.15.114:0/3287179793 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f4d30101130 0x7f4d301986c0 unknown :-1 s=CLOSED pgs=780 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:34:10.504 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:34:10.502+0000 7f4d35abb640 1 -- 172.21.15.114:0/3287179793 >> 172.21.15.114:0/3287179793 conn(0x7f4d30077c30 msgr2=0x7f4d30105220 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T10:34:10.504 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:34:10.502+0000 7f4d35abb640 1 -- 172.21.15.114:0/3287179793 shutdown_connections 2024-04-23T10:34:10.504 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:34:10.502+0000 7f4d35abb640 1 -- 172.21.15.114:0/3287179793 wait complete. 2024-04-23T10:34:10.928 INFO:tasks.workunit.client.0.smithi114.stdout: 1 508319 6.94 MB/sec execute 259 sec latency 40.399 ms 2024-04-23T10:34:11.741 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 23 10:34:11 smithi119 ceph-mon[32608]: pgmap v236: 97 pgs: 97 active+clean; 149 MiB data, 1.9 GiB used, 1.0 TiB / 1.0 TiB avail; 826 KiB/s rd, 2.3 MiB/s wr, 169 op/s 2024-04-23T10:34:11.753 INFO:journalctl@ceph.mon.a.smithi114.stdout:Apr 23 10:34:11 smithi114 ceph-mon[29255]: pgmap v236: 97 pgs: 97 active+clean; 149 MiB data, 1.9 GiB used, 1.0 TiB / 1.0 TiB avail; 826 KiB/s rd, 2.3 MiB/s wr, 169 op/s 2024-04-23T10:34:11.827 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 23 10:34:11 smithi204 ceph-mon[32649]: pgmap v236: 97 pgs: 97 active+clean; 149 MiB data, 1.9 GiB used, 1.0 TiB / 1.0 TiB avail; 826 KiB/s rd, 2.3 MiB/s wr, 169 op/s 2024-04-23T10:34:11.929 INFO:tasks.workunit.client.0.smithi114.stdout: 1 510098 6.94 MB/sec execute 260 sec latency 17.274 ms 2024-04-23T10:34:12.928 INFO:tasks.workunit.client.0.smithi114.stdout: 1 512133 6.97 MB/sec execute 261 sec latency 10.314 ms 2024-04-23T10:34:13.741 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 23 10:34:13 smithi119 ceph-mon[32608]: pgmap v237: 97 pgs: 97 active+clean; 145 MiB data, 1.9 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 MiB/s rd, 4.1 MiB/s wr, 367 op/s 2024-04-23T10:34:13.753 INFO:journalctl@ceph.mon.a.smithi114.stdout:Apr 23 10:34:13 smithi114 ceph-mon[29255]: pgmap v237: 97 pgs: 97 active+clean; 145 MiB data, 1.9 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 MiB/s rd, 4.1 MiB/s wr, 367 op/s 2024-04-23T10:34:13.827 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 23 10:34:13 smithi204 ceph-mon[32649]: pgmap v237: 97 pgs: 97 active+clean; 145 MiB data, 1.9 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 MiB/s rd, 4.1 MiB/s wr, 367 op/s 2024-04-23T10:34:13.928 INFO:tasks.workunit.client.0.smithi114.stdout: 1 514496 6.99 MB/sec execute 262 sec latency 32.632 ms 2024-04-23T10:34:14.928 INFO:tasks.workunit.client.0.smithi114.stdout: 1 515846 6.99 MB/sec execute 263 sec latency 66.252 ms 2024-04-23T10:34:15.742 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 23 10:34:15 smithi119 ceph-mon[32608]: pgmap v238: 97 pgs: 97 active+clean; 145 MiB data, 1.9 GiB used, 1.0 TiB / 1.0 TiB avail; 1.4 MiB/s rd, 3.5 MiB/s wr, 315 op/s 2024-04-23T10:34:15.753 INFO:journalctl@ceph.mon.a.smithi114.stdout:Apr 23 10:34:15 smithi114 ceph-mon[29255]: pgmap v238: 97 pgs: 97 active+clean; 145 MiB data, 1.9 GiB used, 1.0 TiB / 1.0 TiB avail; 1.4 MiB/s rd, 3.5 MiB/s wr, 315 op/s 2024-04-23T10:34:15.827 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 23 10:34:15 smithi204 ceph-mon[32649]: pgmap v238: 97 pgs: 97 active+clean; 145 MiB data, 1.9 GiB used, 1.0 TiB / 1.0 TiB avail; 1.4 MiB/s rd, 3.5 MiB/s wr, 315 op/s 2024-04-23T10:34:15.928 INFO:tasks.workunit.client.0.smithi114.stdout: 1 518297 7.01 MB/sec execute 264 sec latency 70.201 ms 2024-04-23T10:34:16.928 INFO:tasks.workunit.client.0.smithi114.stdout: 1 519851 7.01 MB/sec execute 265 sec latency 16.592 ms 2024-04-23T10:34:17.741 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 23 10:34:17 smithi119 ceph-mon[32608]: pgmap v239: 97 pgs: 97 active+clean; 153 MiB data, 1.9 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 MiB/s rd, 3.6 MiB/s wr, 314 op/s 2024-04-23T10:34:17.753 INFO:journalctl@ceph.mon.a.smithi114.stdout:Apr 23 10:34:17 smithi114 ceph-mon[29255]: pgmap v239: 97 pgs: 97 active+clean; 153 MiB data, 1.9 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 MiB/s rd, 3.6 MiB/s wr, 314 op/s 2024-04-23T10:34:17.827 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 23 10:34:17 smithi204 ceph-mon[32649]: pgmap v239: 97 pgs: 97 active+clean; 153 MiB data, 1.9 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 MiB/s rd, 3.6 MiB/s wr, 314 op/s 2024-04-23T10:34:17.928 INFO:tasks.workunit.client.0.smithi114.stdout: 1 522450 7.05 MB/sec execute 266 sec latency 15.643 ms 2024-04-23T10:34:18.928 INFO:tasks.workunit.client.0.smithi114.stdout: 1 523906 7.04 MB/sec execute 267 sec latency 21.180 ms 2024-04-23T10:34:19.741 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 23 10:34:19 smithi119 ceph-mon[32608]: pgmap v240: 97 pgs: 97 active+clean; 153 MiB data, 1.9 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 MiB/s rd, 3.6 MiB/s wr, 314 op/s 2024-04-23T10:34:19.753 INFO:journalctl@ceph.mon.a.smithi114.stdout:Apr 23 10:34:19 smithi114 ceph-mon[29255]: pgmap v240: 97 pgs: 97 active+clean; 153 MiB data, 1.9 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 MiB/s rd, 3.6 MiB/s wr, 314 op/s 2024-04-23T10:34:19.827 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 23 10:34:19 smithi204 ceph-mon[32649]: pgmap v240: 97 pgs: 97 active+clean; 153 MiB data, 1.9 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 MiB/s rd, 3.6 MiB/s wr, 314 op/s 2024-04-23T10:34:19.928 INFO:tasks.workunit.client.0.smithi114.stdout: 1 526102 7.07 MB/sec execute 268 sec latency 80.909 ms 2024-04-23T10:34:20.928 INFO:tasks.workunit.client.0.smithi114.stdout: 1 527743 7.06 MB/sec execute 269 sec latency 24.039 ms 2024-04-23T10:34:21.742 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 23 10:34:21 smithi119 ceph-mon[32608]: pgmap v241: 97 pgs: 97 active+clean; 153 MiB data, 1.9 GiB used, 1.0 TiB / 1.0 TiB avail; 1.3 MiB/s rd, 3.0 MiB/s wr, 262 op/s 2024-04-23T10:34:21.754 INFO:journalctl@ceph.mon.a.smithi114.stdout:Apr 23 10:34:21 smithi114 ceph-mon[29255]: pgmap v241: 97 pgs: 97 active+clean; 153 MiB data, 1.9 GiB used, 1.0 TiB / 1.0 TiB avail; 1.3 MiB/s rd, 3.0 MiB/s wr, 262 op/s 2024-04-23T10:34:21.827 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 23 10:34:21 smithi204 ceph-mon[32649]: pgmap v241: 97 pgs: 97 active+clean; 153 MiB data, 1.9 GiB used, 1.0 TiB / 1.0 TiB avail; 1.3 MiB/s rd, 3.0 MiB/s wr, 262 op/s 2024-04-23T10:34:21.928 INFO:tasks.workunit.client.0.smithi114.stdout: 1 529813 7.09 MB/sec execute 270 sec latency 13.887 ms 2024-04-23T10:34:22.929 INFO:tasks.workunit.client.0.smithi114.stdout: 1 531942 7.09 MB/sec execute 271 sec latency 18.258 ms 2024-04-23T10:34:23.594 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:34:23.593+0000 7f2257a74640 1 -- 172.21.15.114:0/1488224359 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f2250107290 msgr2=0x7f225010ee60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T10:34:23.594 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:34:23.593+0000 7f2257a74640 1 --2- 172.21.15.114:0/1488224359 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f2250107290 0x7f225010ee60 secure :-1 s=READY pgs=784 cs=0 l=1 rev1=1 crypto rx=0x7f2248009ca0 tx=0x7f224802a8c0 comp rx=0 tx=0).stop 2024-04-23T10:34:23.594 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:34:23.593+0000 7f2257a74640 1 -- 172.21.15.114:0/1488224359 shutdown_connections 2024-04-23T10:34:23.594 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:34:23.593+0000 7f2257a74640 1 --2- 172.21.15.114:0/1488224359 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f2250107290 0x7f225010ee60 unknown :-1 s=CLOSED pgs=784 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:34:23.594 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:34:23.593+0000 7f2257a74640 1 --2- 172.21.15.114:0/1488224359 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f2250102b70 0x7f2250106bb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:34:23.594 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:34:23.593+0000 7f2257a74640 1 --2- 172.21.15.114:0/1488224359 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f22501021c0 0x7f22501025a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:34:23.595 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:34:23.593+0000 7f2257a74640 1 -- 172.21.15.114:0/1488224359 >> 172.21.15.114:0/1488224359 conn(0x7f22500fda20 msgr2=0x7f22500ffe40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T10:34:23.595 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:34:23.593+0000 7f2257a74640 1 -- 172.21.15.114:0/1488224359 shutdown_connections 2024-04-23T10:34:23.595 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:34:23.593+0000 7f2257a74640 1 -- 172.21.15.114:0/1488224359 wait complete. 2024-04-23T10:34:23.595 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:34:23.594+0000 7f2257a74640 1 Processor -- start 2024-04-23T10:34:23.595 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:34:23.594+0000 7f2257a74640 1 -- start start 2024-04-23T10:34:23.595 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:34:23.594+0000 7f2257a74640 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f22501021c0 0x7f2250198740 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T10:34:23.595 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:34:23.594+0000 7f2257a74640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f2250107290 0x7f2250198c80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T10:34:23.595 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:34:23.594+0000 7f2257a74640 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f225019cfb0 0x7f225019d460 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T10:34:23.595 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:34:23.594+0000 7f2257a74640 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x7f2250112200 con 0x7f2250107290 2024-04-23T10:34:23.595 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:34:23.594+0000 7f2257a74640 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 -- 0x7f2250112080 con 0x7f225019cfb0 2024-04-23T10:34:23.595 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:34:23.594+0000 7f2257a74640 1 -- --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_getmap magic: 0 -- 0x7f2250112380 con 0x7f22501021c0 2024-04-23T10:34:23.595 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:34:23.594+0000 7f225500f640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f2250107290 0x7f2250198c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T10:34:23.595 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:34:23.594+0000 7f225500f640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f2250107290 0x7f2250198c80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.114:58394/0 (socket says 172.21.15.114:58394) 2024-04-23T10:34:23.595 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:34:23.594+0000 7f225500f640 1 -- 172.21.15.114:0/29798126 learned_addr learned my addr 172.21.15.114:0/29798126 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-04-23T10:34:23.596 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:34:23.594+0000 7f2255810640 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f22501021c0 0x7f2250198740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T10:34:23.596 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:34:23.594+0000 7f2256011640 1 --2- 172.21.15.114:0/29798126 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f225019cfb0 0x7f225019d460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T10:34:23.596 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:34:23.594+0000 7f225500f640 1 -- 172.21.15.114:0/29798126 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f22501021c0 msgr2=0x7f2250198740 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T10:34:23.596 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:34:23.594+0000 7f225500f640 1 --2- 172.21.15.114:0/29798126 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f22501021c0 0x7f2250198740 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:34:23.596 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:34:23.594+0000 7f225500f640 1 -- 172.21.15.114:0/29798126 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f225019cfb0 msgr2=0x7f225019d460 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T10:34:23.596 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:34:23.594+0000 7f225500f640 1 --2- 172.21.15.114:0/29798126 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f225019cfb0 0x7f225019d460 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:34:23.596 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:34:23.594+0000 7f225500f640 1 -- 172.21.15.114:0/29798126 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f225019daa0 con 0x7f2250107290 2024-04-23T10:34:23.596 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:34:23.595+0000 7f225500f640 1 --2- 172.21.15.114:0/29798126 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f2250107290 0x7f2250198c80 secure :-1 s=READY pgs=785 cs=0 l=1 rev1=1 crypto rx=0x7f224000e9f0 tx=0x7f224000eeb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T10:34:23.596 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:34:23.595+0000 7f2246ffd640 1 -- 172.21.15.114:0/29798126 <== mon.0 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f224000ccc0 con 0x7f2250107290 2024-04-23T10:34:23.596 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:34:23.595+0000 7f2257a74640 1 -- 172.21.15.114:0/29798126 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f225019dd90 con 0x7f2250107290 2024-04-23T10:34:23.596 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:34:23.595+0000 7f2246ffd640 1 -- 172.21.15.114:0/29798126 <== mon.0 v2:172.21.15.114:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f22400054d0 con 0x7f2250107290 2024-04-23T10:34:23.596 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:34:23.595+0000 7f2246ffd640 1 -- 172.21.15.114:0/29798126 <== mon.0 v2:172.21.15.114:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2240010430 con 0x7f2250107290 2024-04-23T10:34:23.596 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:34:23.595+0000 7f2257a74640 1 -- 172.21.15.114:0/29798126 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f22501a5530 con 0x7f2250107290 2024-04-23T10:34:23.603 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:34:23.597+0000 7f2246ffd640 1 -- 172.21.15.114:0/29798126 <== mon.0 v2:172.21.15.114:3300/0 4 ==== mgrmap(e 26) ==== 138778+0+0 (secure 0 0 0) 0x7f22400046c0 con 0x7f2250107290 2024-04-23T10:34:23.603 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:34:23.597+0000 7f225a781740 1 Processor -- start 2024-04-23T10:34:23.603 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:34:23.598+0000 7f225a781740 1 -- start start 2024-04-23T10:34:23.603 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:34:23.598+0000 7f225a781740 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x5584cb778620 0x5584cb778a00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T10:34:23.603 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:34:23.598+0000 7f225a781740 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x5584cb778f40 0x5584cb7832c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T10:34:23.603 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:34:23.598+0000 7f225a781740 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x5584cb783800 0x5584cb785bf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T10:34:23.603 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:34:23.598+0000 7f225a781740 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x5584cb6dd970 con 0x5584cb778f40 2024-04-23T10:34:23.603 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:34:23.598+0000 7f225a781740 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 -- 0x5584cb67a4a0 con 0x5584cb783800 2024-04-23T10:34:23.603 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:34:23.598+0000 7f225a781740 1 -- --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_getmap magic: 0 -- 0x5584cb6d9930 con 0x5584cb778620 2024-04-23T10:34:23.604 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:34:23.598+0000 7f2246ffd640 1 --2- 172.21.15.114:0/29798126 >> [v2:172.21.15.114:6800/1395693466,v1:172.21.15.114:6801/1395693466] conn(0x7f223c0a3dc0 0x7f223c0a6280 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T10:34:23.604 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:34:23.598+0000 7f2255810640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x5584cb778f40 0x5584cb7832c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T10:34:23.604 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:34:23.598+0000 7f225500f640 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x5584cb783800 0x5584cb785bf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T10:34:23.604 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:34:23.598+0000 7f2255810640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x5584cb778f40 0x5584cb7832c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.114:58402/0 (socket says 172.21.15.114:58402) 2024-04-23T10:34:23.604 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:34:23.598+0000 7f2255810640 1 -- 172.21.15.114:0/2584306981 learned_addr learned my addr 172.21.15.114:0/2584306981 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-04-23T10:34:23.604 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:34:23.598+0000 7f2256011640 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x5584cb778620 0x5584cb778a00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T10:34:23.604 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:34:23.598+0000 7f2256011640 1 --2- 172.21.15.114:0/29798126 >> [v2:172.21.15.114:6800/1395693466,v1:172.21.15.114:6801/1395693466] conn(0x7f223c0a3dc0 0x7f223c0a6280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T10:34:23.604 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:34:23.598+0000 7f2246ffd640 1 -- 172.21.15.114:0/29798126 <== mon.0 v2:172.21.15.114:3300/0 5 ==== osd_map(98..98 src has 1..98) ==== 8629+0+0 (secure 0 0 0) 0x7f22400d0ba0 con 0x7f2250107290 2024-04-23T10:34:23.604 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:34:23.598+0000 7f2256011640 1 -- 172.21.15.114:0/2584306981 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x5584cb783800 msgr2=0x5584cb785bf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T10:34:23.604 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:34:23.598+0000 7f2256011640 1 --2- 172.21.15.114:0/2584306981 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x5584cb783800 0x5584cb785bf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:34:23.604 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:34:23.598+0000 7f2256011640 1 -- 172.21.15.114:0/2584306981 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x5584cb778f40 msgr2=0x5584cb7832c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T10:34:23.604 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:34:23.598+0000 7f2256011640 1 --2- 172.21.15.114:0/2584306981 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x5584cb778f40 0x5584cb7832c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:34:23.604 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:34:23.598+0000 7f2256011640 1 -- 172.21.15.114:0/2584306981 --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5584cb3f0210 con 0x5584cb778620 2024-04-23T10:34:23.604 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:34:23.599+0000 7f2256011640 1 --2- 172.21.15.114:0/2584306981 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x5584cb778620 0x5584cb778a00 secure :-1 s=READY pgs=77 cs=0 l=1 rev1=1 crypto rx=0x7f2248006010 tx=0x7f2248032f90 comp rx=0 tx=0).ready entity=mon.1 client_cookie=a59541f73de92d6f server_cookie=0 in_seq=0 out_seq=0 2024-04-23T10:34:23.605 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:34:23.599+0000 7f2256011640 1 --2- 172.21.15.114:0/29798126 >> [v2:172.21.15.114:6800/1395693466,v1:172.21.15.114:6801/1395693466] conn(0x7f223c0a3dc0 0x7f223c0a6280 secure :-1 s=READY pgs=119 cs=0 l=1 rev1=1 crypto rx=0x7f2248005e30 tx=0x7f224803b000 comp rx=0 tx=0).ready entity=mgr.15291 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T10:34:23.605 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:34:23.599+0000 7f223bfff640 1 -- 172.21.15.114:0/2584306981 <== mon.1 v2:172.21.15.204:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f224802f350 con 0x5584cb778620 2024-04-23T10:34:23.605 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:34:23.599+0000 7f223bfff640 1 -- 172.21.15.114:0/2584306981 <== mon.1 v2:172.21.15.204:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f2248036040 con 0x5584cb778620 2024-04-23T10:34:23.605 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:34:23.599+0000 7f223bfff640 1 -- 172.21.15.114:0/2584306981 <== mon.1 v2:172.21.15.204:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f224802f660 con 0x5584cb778620 2024-04-23T10:34:23.605 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:34:23.600+0000 7f225a781740 1 -- 172.21.15.114:0/2584306981 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x5584cb778620 msgr2=0x5584cb778a00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T10:34:23.605 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:34:23.600+0000 7f225a781740 1 --2- 172.21.15.114:0/2584306981 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x5584cb778620 0x5584cb778a00 secure :-1 s=READY pgs=77 cs=0 l=1 rev1=1 crypto rx=0x7f2248006010 tx=0x7f2248032f90 comp rx=0 tx=0).stop 2024-04-23T10:34:23.605 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:34:23.600+0000 7f225a781740 1 -- 172.21.15.114:0/2584306981 shutdown_connections 2024-04-23T10:34:23.605 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:34:23.600+0000 7f225a781740 1 --2- 172.21.15.114:0/2584306981 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x5584cb783800 0x5584cb785bf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:34:23.605 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:34:23.600+0000 7f225a781740 1 --2- 172.21.15.114:0/2584306981 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x5584cb778f40 0x5584cb7832c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:34:23.605 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:34:23.600+0000 7f225a781740 1 --2- 172.21.15.114:0/2584306981 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x5584cb778620 0x5584cb778a00 unknown :-1 s=CLOSED pgs=77 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:34:23.605 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:34:23.600+0000 7f225a781740 1 -- 172.21.15.114:0/2584306981 >> 172.21.15.114:0/2584306981 conn(0x5584cb774b30 msgr2=0x5584cb774f10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T10:34:23.605 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:34:23.600+0000 7f225a781740 1 -- 172.21.15.114:0/2584306981 shutdown_connections 2024-04-23T10:34:23.605 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:34:23.600+0000 7f225a781740 1 -- 172.21.15.114:0/2584306981 wait complete. 2024-04-23T10:34:23.605 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:34:23.600+0000 7f225a781740 1 -- start start 2024-04-23T10:34:23.605 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:34:23.600+0000 7f225a781740 1 Processor -- start 2024-04-23T10:34:23.605 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:34:23.601+0000 7f225a781740 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x5584cb778620 0x5584cb783200 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T10:34:23.606 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:34:23.601+0000 7f225a781740 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x5584cb778f40 0x5584cb7804f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T10:34:23.606 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:34:23.601+0000 7f225a781740 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x5584cb783800 0x5584cb780b00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T10:34:23.606 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:34:23.601+0000 7f225a781740 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x5584cb786250 con 0x5584cb778f40 2024-04-23T10:34:23.606 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:34:23.601+0000 7f225a781740 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 -- 0x5584cb6780b0 con 0x5584cb783800 2024-04-23T10:34:23.606 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:34:23.601+0000 7f225a781740 1 -- --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_getmap magic: 0 -- 0x5584cb7863d0 con 0x5584cb778620 2024-04-23T10:34:23.606 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:34:23.601+0000 7f225a781740 10 client.0 _ll_register_callbacks cb 0x5584cb665d30 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-23T10:34:23.606 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:34:23.601+0000 7f225500f640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x5584cb778f40 0x5584cb7804f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T10:34:23.606 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:34:23.601+0000 7f225500f640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x5584cb778f40 0x5584cb7804f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.114:58410/0 (socket says 172.21.15.114:58410) 2024-04-23T10:34:23.606 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:34:23.601+0000 7f225500f640 1 -- 172.21.15.114:0/3640626414 learned_addr learned my addr 172.21.15.114:0/3640626414 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-04-23T10:34:23.606 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:34:23.601+0000 7f2256011640 1 --2- 172.21.15.114:0/3640626414 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x5584cb783800 0x5584cb780b00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T10:34:23.606 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:34:23.601+0000 7f2255810640 1 --2- 172.21.15.114:0/3640626414 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x5584cb778620 0x5584cb783200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T10:34:23.606 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:34:23.601+0000 7f225500f640 1 -- 172.21.15.114:0/3640626414 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x5584cb778620 msgr2=0x5584cb783200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T10:34:23.606 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:34:23.601+0000 7f225500f640 1 --2- 172.21.15.114:0/3640626414 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x5584cb778620 0x5584cb783200 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:34:23.606 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:34:23.601+0000 7f225500f640 1 -- 172.21.15.114:0/3640626414 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x5584cb783800 msgr2=0x5584cb780b00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T10:34:23.607 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:34:23.601+0000 7f225500f640 1 --2- 172.21.15.114:0/3640626414 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x5584cb783800 0x5584cb780b00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:34:23.607 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:34:23.601+0000 7f225500f640 1 -- 172.21.15.114:0/3640626414 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5584cb4bc620 con 0x5584cb778f40 2024-04-23T10:34:23.607 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:34:23.601+0000 7f225500f640 1 --2- 172.21.15.114:0/3640626414 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x5584cb778f40 0x5584cb7804f0 secure :-1 s=READY pgs=786 cs=0 l=1 rev1=1 crypto rx=0x7f2240098f30 tx=0x7f224006c3b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=fb929ab4bd91fdb server_cookie=0 in_seq=0 out_seq=0 2024-04-23T10:34:23.607 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:34:23.601+0000 7f22397fa640 10 client.0 ms_handle_connect on v2:172.21.15.114:3300/0 2024-04-23T10:34:23.607 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:34:23.601+0000 7f22397fa640 1 -- 172.21.15.114:0/3640626414 <== mon.0 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f224009cad0 con 0x5584cb778f40 2024-04-23T10:34:23.607 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:34:23.601+0000 7f22397fa640 1 -- 172.21.15.114:0/3640626414 <== mon.0 v2:172.21.15.114:3300/0 2 ==== osd_map(98..98 src has 1..98) ==== 8629+0+0 (secure 0 0 0) 0x7f224009add0 con 0x5584cb778f40 2024-04-23T10:34:23.607 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:34:23.601+0000 7f225a781740 1 -- 172.21.15.114:0/3640626414 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f2248036040 con 0x5584cb778f40 2024-04-23T10:34:23.607 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:34:23.602+0000 7f22397fa640 1 -- 172.21.15.114:0/3640626414 <== mon.0 v2:172.21.15.114:3300/0 3 ==== mon_get_version_reply(handle=1 version=31) ==== 24+0+0 (secure 0 0 0) 0x7f22400a0e80 con 0x5584cb778f40 2024-04-23T10:34:23.607 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:34:23.602+0000 7f225a781740 10 client.16584 fetch_fsmap learned FSMap version 31 2024-04-23T10:34:23.607 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:34:23.602+0000 7f225a781740 1 -- 172.21.15.114:0/3640626414 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({config=0+,fsmap=31,monmap=4+}) -- 0x5584cb792200 con 0x5584cb778f40 2024-04-23T10:34:23.607 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:34:23.602+0000 7f22397fa640 1 -- 172.21.15.114:0/3640626414 <== mon.0 v2:172.21.15.114:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f2248036040 con 0x5584cb778f40 2024-04-23T10:34:23.607 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:34:23.602+0000 7f22397fa640 1 -- 172.21.15.114:0/3640626414 <== mon.0 v2:172.21.15.114:3300/0 5 ==== fsmap(e 31) ==== 8342+0+0 (secure 0 0 0) 0x7f22400a52b0 con 0x5584cb778f40 2024-04-23T10:34:23.607 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:34:23.602+0000 7f225a781740 10 client.16584 fetch_fsmap finished waiting for FSMap version 31 2024-04-23T10:34:23.607 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:34:23.602+0000 7f225a781740 10 client.16584 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-04-23T10:34:23.607 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:34:23.602+0000 7f225a781740 20 client.16584 populate_metadata read hostname 'smithi114' 2024-04-23T10:34:23.607 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:34:23.602+0000 7f225a781740 1 --2- 172.21.15.114:0/3640626414 >> [v2:172.21.15.119:6838/937297340,v1:172.21.15.119:6839/937297340] conn(0x5584cb793670 0x5584cb795a60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T10:34:23.608 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:34:23.602+0000 7f225a781740 4 client.16584 mds_command: new command op to 24545 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-23T10:34:23.608 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:34:23.602+0000 7f225a781740 1 -- 172.21.15.114:0/3640626414 --> [v2:172.21.15.119:6838/937297340,v1:172.21.15.119:6839/937297340] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x5584cb6eb390 con 0x5584cb793670 2024-04-23T10:34:23.608 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:34:23.602+0000 7f2255810640 1 --2- 172.21.15.114:0/3640626414 >> [v2:172.21.15.119:6838/937297340,v1:172.21.15.119:6839/937297340] conn(0x5584cb793670 0x5584cb795a60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T10:34:23.608 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:34:23.602+0000 7f2255810640 1 --2- 172.21.15.114:0/3640626414 >> [v2:172.21.15.119:6838/937297340,v1:172.21.15.119:6839/937297340] conn(0x5584cb793670 0x5584cb795a60 crc :-1 s=READY pgs=130 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=7623a96666002c8e server_cookie=e19bd9c67780fec9 in_seq=0 out_seq=0 2024-04-23T10:34:23.608 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:34:23.602+0000 7f22397fa640 10 client.16584 ms_handle_connect on v2:172.21.15.119:6838/937297340 2024-04-23T10:34:23.608 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:34:23.603+0000 7f22397fa640 1 -- 172.21.15.114:0/3640626414 <== mds.0 v2:172.21.15.119:6838/937297340 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x5584cb6eb390 con 0x5584cb793670 2024-04-23T10:34:23.608 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:34:23.603+0000 7f22397fa640 10 client.16584 handle_command_reply: tid=0 2024-04-23T10:34:23.608 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:34:23.603+0000 7f225a781740 1 client.16584 shutdown 2024-04-23T10:34:23.608 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:34:23.603+0000 7f225a781740 1 -- 172.21.15.114:0/3640626414 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x5584cb778f40 msgr2=0x5584cb7804f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T10:34:23.608 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:34:23.603+0000 7f225a781740 1 --2- 172.21.15.114:0/3640626414 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x5584cb778f40 0x5584cb7804f0 secure :-1 s=READY pgs=786 cs=0 l=1 rev1=1 crypto rx=0x7f2240098f30 tx=0x7f224006c3b0 comp rx=0 tx=0).stop 2024-04-23T10:34:23.608 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:34:23.603+0000 7f225a781740 1 -- 172.21.15.114:0/3640626414 shutdown_connections 2024-04-23T10:34:23.608 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:34:23.603+0000 7f225a781740 1 --2- 172.21.15.114:0/3640626414 >> [v2:172.21.15.119:6838/937297340,v1:172.21.15.119:6839/937297340] conn(0x5584cb793670 0x5584cb795a60 crc :-1 s=READY pgs=130 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:34:23.608 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:34:23.603+0000 7f225a781740 1 --2- 172.21.15.114:0/3640626414 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x5584cb783800 0x5584cb780b00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:34:23.609 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:34:23.603+0000 7f225a781740 1 --2- 172.21.15.114:0/3640626414 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x5584cb778f40 0x5584cb7804f0 unknown :-1 s=CLOSED pgs=786 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:34:23.609 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:34:23.603+0000 7f22397fa640 0 client.16584 ms_handle_reset on v2:172.21.15.119:6838/937297340 2024-04-23T10:34:23.609 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:34:23.603+0000 7f225a781740 1 --2- 172.21.15.114:0/3640626414 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x5584cb778620 0x5584cb783200 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:34:23.609 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:34:23.603+0000 7f225a781740 1 -- 172.21.15.114:0/3640626414 >> 172.21.15.114:0/3640626414 conn(0x5584cb774b30 msgr2=0x5584cb77a8b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T10:34:23.609 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:34:23.604+0000 7f225a781740 1 -- 172.21.15.114:0/3640626414 shutdown_connections 2024-04-23T10:34:23.609 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:34:23.604+0000 7f225a781740 1 -- 172.21.15.114:0/3640626414 wait complete. 2024-04-23T10:34:23.609 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:34:23.604+0000 7f225a781740 20 client.16584 trim_cache size 0 max 16384 2024-04-23T10:34:23.620 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:34:23.619+0000 7f225a781740 1 Processor -- start 2024-04-23T10:34:23.620 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:34:23.619+0000 7f225a781740 1 -- start start 2024-04-23T10:34:23.620 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:34:23.619+0000 7f225a781740 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x5584cb778620 0x5584cb78f110 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T10:34:23.620 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:34:23.619+0000 7f225a781740 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x5584cb778f40 0x5584cb77a8b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T10:34:23.620 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:34:23.619+0000 7f225a781740 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x5584cb783800 0x5584cb782d30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T10:34:23.620 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:34:23.619+0000 7f225a781740 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x7f22501b0550 con 0x5584cb783800 2024-04-23T10:34:23.621 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:34:23.619+0000 7f225a781740 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 -- 0x7f22501addf0 con 0x5584cb778f40 2024-04-23T10:34:23.621 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:34:23.619+0000 7f225a781740 1 -- --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_getmap magic: 0 -- 0x7f22501abc60 con 0x5584cb778620 2024-04-23T10:34:23.621 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:34:23.620+0000 7f2256011640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x5584cb783800 0x5584cb782d30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T10:34:23.621 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:34:23.620+0000 7f2255810640 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x5584cb778620 0x5584cb78f110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T10:34:23.621 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:34:23.620+0000 7f225500f640 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x5584cb778f40 0x5584cb77a8b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T10:34:23.621 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:34:23.620+0000 7f2256011640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x5584cb783800 0x5584cb782d30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.114:58426/0 (socket says 172.21.15.114:58426) 2024-04-23T10:34:23.621 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:34:23.620+0000 7f2256011640 1 -- 172.21.15.114:0/1663042206 learned_addr learned my addr 172.21.15.114:0/1663042206 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-04-23T10:34:23.621 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:34:23.620+0000 7f2256011640 1 -- 172.21.15.114:0/1663042206 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x5584cb778620 msgr2=0x5584cb78f110 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T10:34:23.621 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:34:23.620+0000 7f2256011640 1 --2- 172.21.15.114:0/1663042206 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x5584cb778620 0x5584cb78f110 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:34:23.621 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:34:23.620+0000 7f2256011640 1 -- 172.21.15.114:0/1663042206 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x5584cb778f40 msgr2=0x5584cb77a8b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T10:34:23.621 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:34:23.620+0000 7f2256011640 1 --2- 172.21.15.114:0/1663042206 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x5584cb778f40 0x5584cb77a8b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:34:23.621 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:34:23.620+0000 7f2256011640 1 -- 172.21.15.114:0/1663042206 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5584cb796020 con 0x5584cb783800 2024-04-23T10:34:23.621 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:34:23.620+0000 7f2255810640 1 --2- 172.21.15.114:0/1663042206 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x5584cb778620 0x5584cb78f110 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-23T10:34:23.622 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:34:23.620+0000 7f2256011640 1 --2- 172.21.15.114:0/1663042206 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x5584cb783800 0x5584cb782d30 secure :-1 s=READY pgs=787 cs=0 l=1 rev1=1 crypto rx=0x7f22480046a0 tx=0x7f22480046d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=60e938825b56873a server_cookie=0 in_seq=0 out_seq=0 2024-04-23T10:34:23.622 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:34:23.620+0000 7f223b7fe640 1 -- 172.21.15.114:0/1663042206 <== mon.0 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f224805f070 con 0x5584cb783800 2024-04-23T10:34:23.622 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:34:23.620+0000 7f223b7fe640 1 -- 172.21.15.114:0/1663042206 <== mon.0 v2:172.21.15.114:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f2248036040 con 0x5584cb783800 2024-04-23T10:34:23.622 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:34:23.620+0000 7f223b7fe640 1 -- 172.21.15.114:0/1663042206 <== mon.0 v2:172.21.15.114:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f224802f8c0 con 0x5584cb783800 2024-04-23T10:34:23.622 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:34:23.621+0000 7f225a781740 1 -- 172.21.15.114:0/1663042206 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x5584cb783800 msgr2=0x5584cb782d30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T10:34:23.622 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:34:23.621+0000 7f225a781740 1 --2- 172.21.15.114:0/1663042206 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x5584cb783800 0x5584cb782d30 secure :-1 s=READY pgs=787 cs=0 l=1 rev1=1 crypto rx=0x7f22480046a0 tx=0x7f22480046d0 comp rx=0 tx=0).stop 2024-04-23T10:34:23.622 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:34:23.621+0000 7f225a781740 1 -- 172.21.15.114:0/1663042206 shutdown_connections 2024-04-23T10:34:23.622 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:34:23.621+0000 7f225a781740 1 --2- 172.21.15.114:0/1663042206 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x5584cb783800 0x5584cb782d30 unknown :-1 s=CLOSED pgs=787 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:34:23.622 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:34:23.621+0000 7f225a781740 1 --2- 172.21.15.114:0/1663042206 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x5584cb778f40 0x5584cb77a8b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:34:23.622 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:34:23.621+0000 7f225a781740 1 --2- 172.21.15.114:0/1663042206 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x5584cb778620 0x5584cb78f110 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:34:23.622 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:34:23.621+0000 7f225a781740 1 -- 172.21.15.114:0/1663042206 >> 172.21.15.114:0/1663042206 conn(0x5584cb774b30 msgr2=0x5584cb77b140 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T10:34:23.622 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:34:23.621+0000 7f225a781740 1 -- 172.21.15.114:0/1663042206 shutdown_connections 2024-04-23T10:34:23.623 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:34:23.621+0000 7f225a781740 1 -- 172.21.15.114:0/1663042206 wait complete. 2024-04-23T10:34:23.623 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:34:23.621+0000 7f225a781740 1 -- start start 2024-04-23T10:34:23.623 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:34:23.621+0000 7f225a781740 1 Processor -- start 2024-04-23T10:34:23.625 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:34:23.622+0000 7f225a781740 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x5584cb778620 0x5584cb78b7e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T10:34:23.625 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:34:23.622+0000 7f225a781740 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x5584cb778f40 0x5584cb7885c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T10:34:23.625 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:34:23.622+0000 7f225a781740 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x5584cb783800 0x5584cb788b00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T10:34:23.625 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:34:23.622+0000 7f225a781740 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x7f22501ac2b0 con 0x5584cb778620 2024-04-23T10:34:23.626 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:34:23.622+0000 7f225a781740 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 -- 0x7f22501ad720 con 0x5584cb783800 2024-04-23T10:34:23.626 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:34:23.622+0000 7f225a781740 1 -- --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_getmap magic: 0 -- 0x7f22501ac980 con 0x5584cb778f40 2024-04-23T10:34:23.626 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:34:23.622+0000 7f225a781740 10 client.0 _ll_register_callbacks cb 0x7f224009c700 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-23T10:34:23.626 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:34:23.622+0000 7f2255810640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x5584cb778620 0x5584cb78b7e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T10:34:23.626 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:34:23.622+0000 7f2255810640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x5584cb778620 0x5584cb78b7e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.114:58436/0 (socket says 172.21.15.114:58436) 2024-04-23T10:34:23.626 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:34:23.622+0000 7f2255810640 1 -- 172.21.15.114:0/993828073 learned_addr learned my addr 172.21.15.114:0/993828073 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-04-23T10:34:23.626 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:34:23.622+0000 7f2256011640 1 --2- 172.21.15.114:0/993828073 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x5584cb783800 0x5584cb788b00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T10:34:23.626 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:34:23.622+0000 7f2255810640 1 -- 172.21.15.114:0/993828073 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x5584cb778f40 msgr2=0x5584cb7885c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T10:34:23.626 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:34:23.622+0000 7f2255810640 1 --2- 172.21.15.114:0/993828073 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x5584cb778f40 0x5584cb7885c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:34:23.626 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:34:23.622+0000 7f2255810640 1 -- 172.21.15.114:0/993828073 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x5584cb783800 msgr2=0x5584cb788b00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T10:34:23.626 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:34:23.622+0000 7f2255810640 1 --2- 172.21.15.114:0/993828073 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x5584cb783800 0x5584cb788b00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:34:23.626 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:34:23.622+0000 7f2255810640 1 -- 172.21.15.114:0/993828073 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5584cb7890c0 con 0x5584cb778620 2024-04-23T10:34:23.626 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:34:23.622+0000 7f2256011640 1 --2- 172.21.15.114:0/993828073 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x5584cb783800 0x5584cb788b00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-23T10:34:23.626 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:34:23.622+0000 7f2255810640 1 --2- 172.21.15.114:0/993828073 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x5584cb778620 0x5584cb78b7e0 secure :-1 s=READY pgs=788 cs=0 l=1 rev1=1 crypto rx=0x7f224c000c00 tx=0x7f224c0044d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5c5b069dc6b42901 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T10:34:23.627 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:34:23.622+0000 7f22397fa640 10 client.0 ms_handle_connect on v2:172.21.15.114:3300/0 2024-04-23T10:34:23.627 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:34:23.622+0000 7f22397fa640 1 -- 172.21.15.114:0/993828073 <== mon.0 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f224c020110 con 0x5584cb778620 2024-04-23T10:34:23.627 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:34:23.623+0000 7f22397fa640 1 -- 172.21.15.114:0/993828073 <== mon.0 v2:172.21.15.114:3300/0 2 ==== osd_map(98..98 src has 1..98) ==== 8629+0+0 (secure 0 0 0) 0x7f224c0202b0 con 0x5584cb778620 2024-04-23T10:34:23.627 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:34:23.623+0000 7f225a781740 1 -- 172.21.15.114:0/993828073 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f2248036040 con 0x5584cb778620 2024-04-23T10:34:23.627 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:34:23.623+0000 7f22397fa640 1 -- 172.21.15.114:0/993828073 <== mon.0 v2:172.21.15.114:3300/0 3 ==== mon_get_version_reply(handle=1 version=31) ==== 24+0+0 (secure 0 0 0) 0x7f224c030050 con 0x5584cb778620 2024-04-23T10:34:23.627 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:34:23.623+0000 7f225a781740 10 client.16590 fetch_fsmap learned FSMap version 31 2024-04-23T10:34:23.627 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:34:23.623+0000 7f225a781740 1 -- 172.21.15.114:0/993828073 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({config=0+,fsmap=31,monmap=4+}) -- 0x5584cb7fa950 con 0x5584cb778620 2024-04-23T10:34:23.627 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:34:23.623+0000 7f22397fa640 1 -- 172.21.15.114:0/993828073 <== mon.0 v2:172.21.15.114:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f2248036040 con 0x5584cb778620 2024-04-23T10:34:23.627 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:34:23.623+0000 7f22397fa640 1 -- 172.21.15.114:0/993828073 <== mon.0 v2:172.21.15.114:3300/0 5 ==== fsmap(e 31) ==== 8342+0+0 (secure 0 0 0) 0x7f224c010ba0 con 0x5584cb778620 2024-04-23T10:34:23.627 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:34:23.623+0000 7f225a781740 10 client.16590 fetch_fsmap finished waiting for FSMap version 31 2024-04-23T10:34:23.627 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:34:23.623+0000 7f225a781740 10 client.16590 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-04-23T10:34:23.627 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:34:23.623+0000 7f225a781740 20 client.16590 populate_metadata read hostname 'smithi114' 2024-04-23T10:34:23.627 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:34:23.623+0000 7f225a781740 1 --2- 172.21.15.114:0/993828073 >> [v2:172.21.15.119:6838/937297340,v1:172.21.15.119:6839/937297340] conn(0x5584cb7fb0d0 0x5584cb7fd4f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T10:34:23.627 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:34:23.623+0000 7f225a781740 4 client.16590 mds_command: new command op to 24545 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-04-23T10:34:23.627 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:34:23.623+0000 7f225a781740 1 -- 172.21.15.114:0/993828073 --> [v2:172.21.15.119:6838/937297340,v1:172.21.15.119:6839/937297340] -- command(tid 0: {"prefix": "flush journal"}) -- 0x5584cb6e5200 con 0x5584cb7fb0d0 2024-04-23T10:34:23.627 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:34:23.623+0000 7f225500f640 1 --2- 172.21.15.114:0/993828073 >> [v2:172.21.15.119:6838/937297340,v1:172.21.15.119:6839/937297340] conn(0x5584cb7fb0d0 0x5584cb7fd4f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T10:34:23.628 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:34:23.624+0000 7f225500f640 1 --2- 172.21.15.114:0/993828073 >> [v2:172.21.15.119:6838/937297340,v1:172.21.15.119:6839/937297340] conn(0x5584cb7fb0d0 0x5584cb7fd4f0 crc :-1 s=READY pgs=131 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=39e5f0b166a5f8b server_cookie=5e61784fa8bf1e89 in_seq=0 out_seq=0 2024-04-23T10:34:23.628 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:34:23.624+0000 7f22397fa640 10 client.16590 ms_handle_connect on v2:172.21.15.119:6838/937297340 2024-04-23T10:34:23.692 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:34:23.693+0000 7f22397fa640 1 -- 172.21.15.114:0/993828073 <== mds.0 v2:172.21.15.119:6838/937297340 1 ==== command_reply(tid 0: 0 ) ==== 8+0+44 (crc 0 0 0) 0x5584cb6e5200 con 0x5584cb7fb0d0 2024-04-23T10:34:23.692 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:34:23.693+0000 7f22397fa640 10 client.16590 handle_command_reply: tid=0 2024-04-23T10:34:23.692 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:34:23.693+0000 7f225a781740 1 client.16590 shutdown 2024-04-23T10:34:23.692 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:34:23.693+0000 7f225a781740 1 -- 172.21.15.114:0/993828073 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x5584cb778620 msgr2=0x5584cb78b7e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T10:34:23.692 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:34:23.693+0000 7f225a781740 1 --2- 172.21.15.114:0/993828073 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x5584cb778620 0x5584cb78b7e0 secure :-1 s=READY pgs=788 cs=0 l=1 rev1=1 crypto rx=0x7f224c000c00 tx=0x7f224c0044d0 comp rx=0 tx=0).stop 2024-04-23T10:34:23.692 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:34:23.693+0000 7f225a781740 1 -- 172.21.15.114:0/993828073 shutdown_connections 2024-04-23T10:34:23.692 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:34:23.693+0000 7f225a781740 1 --2- 172.21.15.114:0/993828073 >> [v2:172.21.15.119:6838/937297340,v1:172.21.15.119:6839/937297340] conn(0x5584cb7fb0d0 0x5584cb7fd4f0 crc :-1 s=READY pgs=131 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:34:23.692 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:34:23.693+0000 7f225a781740 1 --2- 172.21.15.114:0/993828073 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x5584cb783800 0x5584cb788b00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:34:23.692 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:34:23.693+0000 7f225a781740 1 --2- 172.21.15.114:0/993828073 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x5584cb778f40 0x5584cb7885c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:34:23.692 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:34:23.693+0000 7f22397fa640 0 client.16590 ms_handle_reset on v2:172.21.15.119:6838/937297340 2024-04-23T10:34:23.692 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:34:23.693+0000 7f225a781740 1 --2- 172.21.15.114:0/993828073 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x5584cb778620 0x5584cb78b7e0 unknown :-1 s=CLOSED pgs=788 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:34:23.693 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:34:23.693+0000 7f225a781740 1 -- 172.21.15.114:0/993828073 >> 172.21.15.114:0/993828073 conn(0x5584cb774b30 msgr2=0x5584cb77b9d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T10:34:23.693 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:34:23.694+0000 7f225a781740 1 -- 172.21.15.114:0/993828073 shutdown_connections 2024-04-23T10:34:23.693 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:34:23.694+0000 7f225a781740 1 -- 172.21.15.114:0/993828073 wait complete. 2024-04-23T10:34:23.693 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:34:23.694+0000 7f225a781740 20 client.16590 trim_cache size 0 max 16384 2024-04-23T10:34:23.694 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stdout:{ 2024-04-23T10:34:23.694 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stdout: "message": "", 2024-04-23T10:34:23.694 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stdout: "return_code": 0 2024-04-23T10:34:23.694 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stdout:} 2024-04-23T10:34:23.695 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:34:23.694+0000 7f2257a74640 1 -- 172.21.15.114:0/29798126 >> [v2:172.21.15.114:6800/1395693466,v1:172.21.15.114:6801/1395693466] conn(0x7f223c0a3dc0 msgr2=0x7f223c0a6280 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T10:34:23.695 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:34:23.694+0000 7f2257a74640 1 --2- 172.21.15.114:0/29798126 >> [v2:172.21.15.114:6800/1395693466,v1:172.21.15.114:6801/1395693466] conn(0x7f223c0a3dc0 0x7f223c0a6280 secure :-1 s=READY pgs=119 cs=0 l=1 rev1=1 crypto rx=0x7f2248005e30 tx=0x7f224803b000 comp rx=0 tx=0).stop 2024-04-23T10:34:23.695 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:34:23.694+0000 7f2257a74640 1 -- 172.21.15.114:0/29798126 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f2250107290 msgr2=0x7f2250198c80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T10:34:23.695 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:34:23.694+0000 7f2257a74640 1 --2- 172.21.15.114:0/29798126 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f2250107290 0x7f2250198c80 secure :-1 s=READY pgs=785 cs=0 l=1 rev1=1 crypto rx=0x7f224000e9f0 tx=0x7f224000eeb0 comp rx=0 tx=0).stop 2024-04-23T10:34:23.695 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:34:23.694+0000 7f2257a74640 1 -- 172.21.15.114:0/29798126 shutdown_connections 2024-04-23T10:34:23.695 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:34:23.694+0000 7f2257a74640 1 --2- 172.21.15.114:0/29798126 >> [v2:172.21.15.114:6800/1395693466,v1:172.21.15.114:6801/1395693466] conn(0x7f223c0a3dc0 0x7f223c0a6280 unknown :-1 s=CLOSED pgs=119 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:34:23.695 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:34:23.694+0000 7f2257a74640 1 --2- 172.21.15.114:0/29798126 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f225019cfb0 0x7f225019d460 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:34:23.695 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:34:23.694+0000 7f2257a74640 1 --2- 172.21.15.114:0/29798126 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f2250107290 0x7f2250198c80 unknown :-1 s=CLOSED pgs=785 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:34:23.695 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:34:23.694+0000 7f2257a74640 1 --2- 172.21.15.114:0/29798126 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f22501021c0 0x7f2250198740 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:34:23.695 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:34:23.694+0000 7f2257a74640 1 -- 172.21.15.114:0/29798126 >> 172.21.15.114:0/29798126 conn(0x7f22500fda20 msgr2=0x7f22500ffce0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T10:34:23.695 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:34:23.694+0000 7f2257a74640 1 -- 172.21.15.114:0/29798126 shutdown_connections 2024-04-23T10:34:23.695 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:34:23.694+0000 7f2257a74640 1 -- 172.21.15.114:0/29798126 wait complete. 2024-04-23T10:34:23.741 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 23 10:34:23 smithi119 ceph-mon[32608]: pgmap v242: 97 pgs: 97 active+clean; 180 MiB data, 2.0 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 MiB/s rd, 5.9 MiB/s wr, 376 op/s 2024-04-23T10:34:23.754 INFO:journalctl@ceph.mon.a.smithi114.stdout:Apr 23 10:34:23 smithi114 ceph-mon[29255]: pgmap v242: 97 pgs: 97 active+clean; 180 MiB data, 2.0 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 MiB/s rd, 5.9 MiB/s wr, 376 op/s 2024-04-23T10:34:23.828 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 23 10:34:23 smithi204 ceph-mon[32649]: pgmap v242: 97 pgs: 97 active+clean; 180 MiB data, 2.0 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 MiB/s rd, 5.9 MiB/s wr, 376 op/s 2024-04-23T10:34:23.928 INFO:tasks.workunit.client.0.smithi114.stdout: 1 533439 7.11 MB/sec execute 272 sec latency 111.715 ms 2024-04-23T10:34:24.929 INFO:tasks.workunit.client.0.smithi114.stdout: 1 535356 7.11 MB/sec execute 273 sec latency 65.807 ms 2024-04-23T10:34:25.742 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 23 10:34:25 smithi119 ceph-mon[32608]: pgmap v243: 97 pgs: 97 active+clean; 180 MiB data, 2.0 GiB used, 1.0 TiB / 1.0 TiB avail; 1.3 MiB/s rd, 4.6 MiB/s wr, 226 op/s 2024-04-23T10:34:25.754 INFO:journalctl@ceph.mon.a.smithi114.stdout:Apr 23 10:34:25 smithi114 ceph-mon[29255]: pgmap v243: 97 pgs: 97 active+clean; 180 MiB data, 2.0 GiB used, 1.0 TiB / 1.0 TiB avail; 1.3 MiB/s rd, 4.6 MiB/s wr, 226 op/s 2024-04-23T10:34:25.828 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 23 10:34:25 smithi204 ceph-mon[32649]: pgmap v243: 97 pgs: 97 active+clean; 180 MiB data, 2.0 GiB used, 1.0 TiB / 1.0 TiB avail; 1.3 MiB/s rd, 4.6 MiB/s wr, 226 op/s 2024-04-23T10:34:25.929 INFO:tasks.workunit.client.0.smithi114.stdout: 1 537042 7.12 MB/sec execute 274 sec latency 17.025 ms 2024-04-23T10:34:26.929 INFO:tasks.workunit.client.0.smithi114.stdout: 1 539420 7.14 MB/sec execute 275 sec latency 18.574 ms 2024-04-23T10:34:27.742 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 23 10:34:27 smithi119 ceph-mon[32608]: pgmap v244: 97 pgs: 97 active+clean; 171 MiB data, 2.0 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 MiB/s rd, 6.1 MiB/s wr, 379 op/s 2024-04-23T10:34:27.754 INFO:journalctl@ceph.mon.a.smithi114.stdout:Apr 23 10:34:27 smithi114 ceph-mon[29255]: pgmap v244: 97 pgs: 97 active+clean; 171 MiB data, 2.0 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 MiB/s rd, 6.1 MiB/s wr, 379 op/s 2024-04-23T10:34:27.828 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 23 10:34:27 smithi204 ceph-mon[32649]: pgmap v244: 97 pgs: 97 active+clean; 171 MiB data, 2.0 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 MiB/s rd, 6.1 MiB/s wr, 379 op/s 2024-04-23T10:34:27.929 INFO:tasks.workunit.client.0.smithi114.stdout: 1 540958 7.14 MB/sec execute 276 sec latency 14.512 ms 2024-04-23T10:34:28.929 INFO:tasks.workunit.client.0.smithi114.stdout: 1 543585 7.18 MB/sec execute 277 sec latency 22.093 ms 2024-04-23T10:34:29.742 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 23 10:34:29 smithi119 ceph-mon[32608]: pgmap v245: 97 pgs: 97 active+clean; 171 MiB data, 2.0 GiB used, 1.0 TiB / 1.0 TiB avail; 1.3 MiB/s rd, 4.5 MiB/s wr, 268 op/s 2024-04-23T10:34:29.754 INFO:journalctl@ceph.mon.a.smithi114.stdout:Apr 23 10:34:29 smithi114 ceph-mon[29255]: pgmap v245: 97 pgs: 97 active+clean; 171 MiB data, 2.0 GiB used, 1.0 TiB / 1.0 TiB avail; 1.3 MiB/s rd, 4.5 MiB/s wr, 268 op/s 2024-04-23T10:34:29.827 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 23 10:34:29 smithi204 ceph-mon[32649]: pgmap v245: 97 pgs: 97 active+clean; 171 MiB data, 2.0 GiB used, 1.0 TiB / 1.0 TiB avail; 1.3 MiB/s rd, 4.5 MiB/s wr, 268 op/s 2024-04-23T10:34:29.929 INFO:tasks.workunit.client.0.smithi114.stdout: 1 544741 7.17 MB/sec execute 278 sec latency 33.965 ms 2024-04-23T10:34:30.929 INFO:tasks.workunit.client.0.smithi114.stdout: 1 547217 7.20 MB/sec execute 279 sec latency 39.174 ms 2024-04-23T10:34:31.741 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 23 10:34:31 smithi119 ceph-mon[32608]: pgmap v246: 97 pgs: 97 active+clean; 171 MiB data, 2.0 GiB used, 1.0 TiB / 1.0 TiB avail; 1.3 MiB/s rd, 4.5 MiB/s wr, 268 op/s 2024-04-23T10:34:31.754 INFO:journalctl@ceph.mon.a.smithi114.stdout:Apr 23 10:34:31 smithi114 ceph-mon[29255]: pgmap v246: 97 pgs: 97 active+clean; 171 MiB data, 2.0 GiB used, 1.0 TiB / 1.0 TiB avail; 1.3 MiB/s rd, 4.5 MiB/s wr, 268 op/s 2024-04-23T10:34:31.827 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 23 10:34:31 smithi204 ceph-mon[32649]: pgmap v246: 97 pgs: 97 active+clean; 171 MiB data, 2.0 GiB used, 1.0 TiB / 1.0 TiB avail; 1.3 MiB/s rd, 4.5 MiB/s wr, 268 op/s 2024-04-23T10:34:31.929 INFO:tasks.workunit.client.0.smithi114.stdout: 1 548600 7.19 MB/sec execute 280 sec latency 17.011 ms 2024-04-23T10:34:32.929 INFO:tasks.workunit.client.0.smithi114.stdout: 1 550939 7.22 MB/sec execute 281 sec latency 13.685 ms 2024-04-23T10:34:33.742 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 23 10:34:33 smithi119 ceph-mon[32608]: pgmap v247: 97 pgs: 97 active+clean; 178 MiB data, 2.1 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 MiB/s rd, 6.0 MiB/s wr, 377 op/s 2024-04-23T10:34:33.753 INFO:journalctl@ceph.mon.a.smithi114.stdout:Apr 23 10:34:33 smithi114 ceph-mon[29255]: pgmap v247: 97 pgs: 97 active+clean; 178 MiB data, 2.1 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 MiB/s rd, 6.0 MiB/s wr, 377 op/s 2024-04-23T10:34:33.827 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 23 10:34:33 smithi204 ceph-mon[32649]: pgmap v247: 97 pgs: 97 active+clean; 178 MiB data, 2.1 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 MiB/s rd, 6.0 MiB/s wr, 377 op/s 2024-04-23T10:34:33.929 INFO:tasks.workunit.client.0.smithi114.stdout: 1 552593 7.21 MB/sec execute 282 sec latency 16.771 ms 2024-04-23T10:34:34.929 INFO:tasks.workunit.client.0.smithi114.stdout: 1 554608 7.24 MB/sec execute 283 sec latency 36.989 ms 2024-04-23T10:34:35.741 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 23 10:34:35 smithi119 ceph-mon[32608]: pgmap v248: 97 pgs: 97 active+clean; 178 MiB data, 2.1 GiB used, 1.0 TiB / 1.0 TiB avail; 1.3 MiB/s rd, 3.0 MiB/s wr, 262 op/s 2024-04-23T10:34:35.754 INFO:journalctl@ceph.mon.a.smithi114.stdout:Apr 23 10:34:35 smithi114 ceph-mon[29255]: pgmap v248: 97 pgs: 97 active+clean; 178 MiB data, 2.1 GiB used, 1.0 TiB / 1.0 TiB avail; 1.3 MiB/s rd, 3.0 MiB/s wr, 262 op/s 2024-04-23T10:34:35.827 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 23 10:34:35 smithi204 ceph-mon[32649]: pgmap v248: 97 pgs: 97 active+clean; 178 MiB data, 2.1 GiB used, 1.0 TiB / 1.0 TiB avail; 1.3 MiB/s rd, 3.0 MiB/s wr, 262 op/s 2024-04-23T10:34:35.929 INFO:tasks.workunit.client.0.smithi114.stdout: 1 556590 7.23 MB/sec execute 284 sec latency 29.175 ms 2024-04-23T10:34:36.787 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:34:36.786+0000 7f5bfefad640 1 -- 172.21.15.114:0/749645048 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f5bf81010f0 msgr2=0x7f5bf81014d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T10:34:36.787 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:34:36.786+0000 7f5bfefad640 1 --2- 172.21.15.114:0/749645048 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f5bf81010f0 0x7f5bf81014d0 secure :-1 s=READY pgs=789 cs=0 l=1 rev1=1 crypto rx=0x7f5bec009900 tx=0x7f5bec02a410 comp rx=0 tx=0).stop 2024-04-23T10:34:36.787 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:34:36.786+0000 7f5bfefad640 1 -- 172.21.15.114:0/749645048 shutdown_connections 2024-04-23T10:34:36.787 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:34:36.786+0000 7f5bfefad640 1 --2- 172.21.15.114:0/749645048 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f5bf8106170 0x7f5bf810dd40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:34:36.787 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:34:36.786+0000 7f5bfefad640 1 --2- 172.21.15.114:0/749645048 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f5bf8101aa0 0x7f5bf8105a90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:34:36.787 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:34:36.786+0000 7f5bfefad640 1 --2- 172.21.15.114:0/749645048 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f5bf81010f0 0x7f5bf81014d0 unknown :-1 s=CLOSED pgs=789 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:34:36.787 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:34:36.786+0000 7f5bfefad640 1 -- 172.21.15.114:0/749645048 >> 172.21.15.114:0/749645048 conn(0x7f5bf8077fd0 msgr2=0x7f5bf80783e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T10:34:36.787 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:34:36.786+0000 7f5bfefad640 1 -- 172.21.15.114:0/749645048 shutdown_connections 2024-04-23T10:34:36.788 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:34:36.786+0000 7f5bfefad640 1 -- 172.21.15.114:0/749645048 wait complete. 2024-04-23T10:34:36.788 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:34:36.786+0000 7f5bfefad640 1 Processor -- start 2024-04-23T10:34:36.788 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:34:36.786+0000 7f5bfefad640 1 -- start start 2024-04-23T10:34:36.788 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:34:36.786+0000 7f5bfefad640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f5bf81010f0 0x7f5bf8196550 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T10:34:36.788 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:34:36.786+0000 7f5bfefad640 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f5bf8106170 0x7f5bf8196a90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T10:34:36.788 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:34:36.787+0000 7f5bfefad640 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f5bf819ae10 0x7f5bf819b2c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T10:34:36.788 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:34:36.787+0000 7f5bfefad640 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x7f5bf810ffd0 con 0x7f5bf81010f0 2024-04-23T10:34:36.788 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:34:36.787+0000 7f5bfefad640 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 -- 0x7f5bf810fe50 con 0x7f5bf8106170 2024-04-23T10:34:36.788 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:34:36.787+0000 7f5bfefad640 1 -- --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_getmap magic: 0 -- 0x7f5bf8110150 con 0x7f5bf819ae10 2024-04-23T10:34:36.788 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:34:36.787+0000 7f5bfcd49640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f5bf81010f0 0x7f5bf8196550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T10:34:36.788 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:34:36.787+0000 7f5bfcd49640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f5bf81010f0 0x7f5bf8196550 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.114:45942/0 (socket says 172.21.15.114:45942) 2024-04-23T10:34:36.788 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:34:36.787+0000 7f5bfcd49640 1 -- 172.21.15.114:0/3204731123 learned_addr learned my addr 172.21.15.114:0/3204731123 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-04-23T10:34:36.788 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:34:36.787+0000 7f5bf7fff640 1 --2- 172.21.15.114:0/3204731123 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f5bf8106170 0x7f5bf8196a90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T10:34:36.789 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:34:36.787+0000 7f5bfd54a640 1 --2- 172.21.15.114:0/3204731123 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f5bf819ae10 0x7f5bf819b2c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T10:34:36.789 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:34:36.787+0000 7f5bfcd49640 1 -- 172.21.15.114:0/3204731123 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f5bf819ae10 msgr2=0x7f5bf819b2c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T10:34:36.789 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:34:36.787+0000 7f5bfcd49640 1 --2- 172.21.15.114:0/3204731123 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f5bf819ae10 0x7f5bf819b2c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:34:36.789 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:34:36.787+0000 7f5bfcd49640 1 -- 172.21.15.114:0/3204731123 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f5bf8106170 msgr2=0x7f5bf8196a90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T10:34:36.789 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:34:36.787+0000 7f5bfcd49640 1 --2- 172.21.15.114:0/3204731123 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f5bf8106170 0x7f5bf8196a90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:34:36.789 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:34:36.787+0000 7f5bfcd49640 1 -- 172.21.15.114:0/3204731123 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f5bf819b900 con 0x7f5bf81010f0 2024-04-23T10:34:36.789 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:34:36.787+0000 7f5bfd54a640 1 --2- 172.21.15.114:0/3204731123 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f5bf819ae10 0x7f5bf819b2c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-23T10:34:36.789 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:34:36.787+0000 7f5bfcd49640 1 --2- 172.21.15.114:0/3204731123 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f5bf81010f0 0x7f5bf8196550 secure :-1 s=READY pgs=790 cs=0 l=1 rev1=1 crypto rx=0x7f5bec005940 tx=0x7f5bec0040c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T10:34:36.789 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:34:36.787+0000 7f5bf5ffb640 1 -- 172.21.15.114:0/3204731123 <== mon.0 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5bec03f030 con 0x7f5bf81010f0 2024-04-23T10:34:36.789 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:34:36.787+0000 7f5bf5ffb640 1 -- 172.21.15.114:0/3204731123 <== mon.0 v2:172.21.15.114:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f5bec0044c0 con 0x7f5bf81010f0 2024-04-23T10:34:36.789 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:34:36.787+0000 7f5bfefad640 1 -- 172.21.15.114:0/3204731123 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f5bf819bb90 con 0x7f5bf81010f0 2024-04-23T10:34:36.789 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:34:36.787+0000 7f5bf5ffb640 1 -- 172.21.15.114:0/3204731123 <== mon.0 v2:172.21.15.114:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5bec035070 con 0x7f5bf81010f0 2024-04-23T10:34:36.789 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:34:36.787+0000 7f5bfefad640 1 -- 172.21.15.114:0/3204731123 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f5bf81a3350 con 0x7f5bf81010f0 2024-04-23T10:34:36.796 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:34:36.790+0000 7f5bf5ffb640 1 -- 172.21.15.114:0/3204731123 <== mon.0 v2:172.21.15.114:3300/0 4 ==== mgrmap(e 26) ==== 138778+0+0 (secure 0 0 0) 0x7f5bec042030 con 0x7f5bf81010f0 2024-04-23T10:34:36.796 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:34:36.790+0000 7f5c01da5740 1 Processor -- start 2024-04-23T10:34:36.796 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:34:36.790+0000 7f5c01da5740 1 -- start start 2024-04-23T10:34:36.796 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:34:36.790+0000 7f5bf5ffb640 1 --2- 172.21.15.114:0/3204731123 >> [v2:172.21.15.114:6800/1395693466,v1:172.21.15.114:6801/1395693466] conn(0x7f5be80a3dc0 0x7f5be80a6280 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T10:34:36.796 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:34:36.790+0000 7f5c01da5740 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x55b08e22c620 0x55b08e22ca00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T10:34:36.796 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:34:36.790+0000 7f5c01da5740 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x55b08e22cf40 0x55b08e2372c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T10:34:36.796 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:34:36.790+0000 7f5c01da5740 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55b08e237800 0x55b08e239bf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T10:34:36.797 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:34:36.790+0000 7f5c01da5740 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x55b08e191970 con 0x55b08e237800 2024-04-23T10:34:36.797 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:34:36.790+0000 7f5c01da5740 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 -- 0x55b08e12e4a0 con 0x55b08e22c620 2024-04-23T10:34:36.797 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:34:36.790+0000 7f5c01da5740 1 -- --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_getmap magic: 0 -- 0x55b08e18d930 con 0x55b08e22cf40 2024-04-23T10:34:36.797 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:34:36.790+0000 7f5bfd54a640 1 --2- 172.21.15.114:0/3204731123 >> [v2:172.21.15.114:6800/1395693466,v1:172.21.15.114:6801/1395693466] conn(0x7f5be80a3dc0 0x7f5be80a6280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T10:34:36.797 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:34:36.790+0000 7f5bfcd49640 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x55b08e22c620 0x55b08e22ca00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T10:34:36.797 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:34:36.791+0000 7f5bfd54a640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55b08e237800 0x55b08e239bf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T10:34:36.797 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:34:36.791+0000 7f5bf7fff640 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x55b08e22cf40 0x55b08e2372c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T10:34:36.797 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:34:36.791+0000 7f5bfd54a640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55b08e237800 0x55b08e239bf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.114:45956/0 (socket says 172.21.15.114:45956) 2024-04-23T10:34:36.797 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:34:36.791+0000 7f5bfd54a640 1 -- 172.21.15.114:0/1609242100 learned_addr learned my addr 172.21.15.114:0/1609242100 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-04-23T10:34:36.797 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:34:36.791+0000 7f5bf5ffb640 1 -- 172.21.15.114:0/3204731123 <== mon.0 v2:172.21.15.114:3300/0 5 ==== osd_map(98..98 src has 1..98) ==== 8629+0+0 (secure 0 0 0) 0x7f5bec0f0c60 con 0x7f5bf81010f0 2024-04-23T10:34:36.797 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:34:36.791+0000 7f5bfcd49640 1 -- 172.21.15.114:0/1609242100 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x55b08e22cf40 msgr2=0x55b08e2372c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T10:34:36.797 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:34:36.791+0000 7f5bfcd49640 1 --2- 172.21.15.114:0/1609242100 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x55b08e22cf40 0x55b08e2372c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:34:36.797 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:34:36.791+0000 7f5bfcd49640 1 -- 172.21.15.114:0/1609242100 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55b08e237800 msgr2=0x55b08e239bf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T10:34:36.797 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:34:36.791+0000 7f5bfcd49640 1 --2- 172.21.15.114:0/1609242100 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55b08e237800 0x55b08e239bf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:34:36.797 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:34:36.791+0000 7f5bfcd49640 1 -- 172.21.15.114:0/1609242100 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55b08dea4210 con 0x55b08e22c620 2024-04-23T10:34:36.798 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:34:36.791+0000 7f5bfd54a640 1 --2- 172.21.15.114:0/1609242100 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55b08e237800 0x55b08e239bf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-23T10:34:36.798 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:34:36.791+0000 7f5bfd54a640 1 --2- 172.21.15.114:0/3204731123 >> [v2:172.21.15.114:6800/1395693466,v1:172.21.15.114:6801/1395693466] conn(0x7f5be80a3dc0 0x7f5be80a6280 secure :-1 s=READY pgs=120 cs=0 l=1 rev1=1 crypto rx=0x55b08e229f50 tx=0x7f5bf0004ab0 comp rx=0 tx=0).ready entity=mgr.15291 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T10:34:36.798 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:34:36.791+0000 7f5bfcd49640 1 --2- 172.21.15.114:0/1609242100 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x55b08e22c620 0x55b08e22ca00 secure :-1 s=READY pgs=84 cs=0 l=1 rev1=1 crypto rx=0x7f5bec0ee340 tx=0x7f5bec071d80 comp rx=0 tx=0).ready entity=mon.2 client_cookie=ab2d72b7bcd9cc51 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T10:34:36.798 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:34:36.791+0000 7f5be2ffd640 1 -- 172.21.15.114:0/1609242100 <== mon.2 v2:172.21.15.119:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5bec049070 con 0x55b08e22c620 2024-04-23T10:34:36.798 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:34:36.791+0000 7f5be2ffd640 1 -- 172.21.15.114:0/1609242100 <== mon.2 v2:172.21.15.119:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f5bec0bcb90 con 0x55b08e22c620 2024-04-23T10:34:36.798 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:34:36.791+0000 7f5be2ffd640 1 -- 172.21.15.114:0/1609242100 <== mon.2 v2:172.21.15.119:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5bec0ee5c0 con 0x55b08e22c620 2024-04-23T10:34:36.798 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:34:36.792+0000 7f5c01da5740 1 -- 172.21.15.114:0/1609242100 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x55b08e22c620 msgr2=0x55b08e22ca00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T10:34:36.798 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:34:36.792+0000 7f5c01da5740 1 --2- 172.21.15.114:0/1609242100 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x55b08e22c620 0x55b08e22ca00 secure :-1 s=READY pgs=84 cs=0 l=1 rev1=1 crypto rx=0x7f5bec0ee340 tx=0x7f5bec071d80 comp rx=0 tx=0).stop 2024-04-23T10:34:36.798 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:34:36.792+0000 7f5c01da5740 1 -- 172.21.15.114:0/1609242100 shutdown_connections 2024-04-23T10:34:36.798 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:34:36.792+0000 7f5c01da5740 1 --2- 172.21.15.114:0/1609242100 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55b08e237800 0x55b08e239bf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:34:36.798 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:34:36.792+0000 7f5c01da5740 1 --2- 172.21.15.114:0/1609242100 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x55b08e22cf40 0x55b08e2372c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:34:36.798 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:34:36.792+0000 7f5c01da5740 1 --2- 172.21.15.114:0/1609242100 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x55b08e22c620 0x55b08e22ca00 unknown :-1 s=CLOSED pgs=84 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:34:36.798 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:34:36.792+0000 7f5c01da5740 1 -- 172.21.15.114:0/1609242100 >> 172.21.15.114:0/1609242100 conn(0x55b08e228b30 msgr2=0x55b08e228f10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T10:34:36.798 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:34:36.792+0000 7f5c01da5740 1 -- 172.21.15.114:0/1609242100 shutdown_connections 2024-04-23T10:34:36.798 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:34:36.792+0000 7f5c01da5740 1 -- 172.21.15.114:0/1609242100 wait complete. 2024-04-23T10:34:36.799 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:34:36.792+0000 7f5c01da5740 1 -- start start 2024-04-23T10:34:36.799 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:34:36.792+0000 7f5c01da5740 1 Processor -- start 2024-04-23T10:34:36.799 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:34:36.793+0000 7f5c01da5740 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x55b08e22c620 0x55b08e2371d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T10:34:36.799 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:34:36.793+0000 7f5c01da5740 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55b08e22cf40 0x55b08e2344c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T10:34:36.799 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:34:36.793+0000 7f5c01da5740 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x55b08e237800 0x55b08e234ad0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T10:34:36.799 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:34:36.793+0000 7f5c01da5740 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x55b08e23a1f0 con 0x55b08e22cf40 2024-04-23T10:34:36.799 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:34:36.793+0000 7f5c01da5740 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 -- 0x55b08e12c0b0 con 0x55b08e22c620 2024-04-23T10:34:36.799 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:34:36.793+0000 7f5c01da5740 1 -- --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_getmap magic: 0 -- 0x55b08e23a370 con 0x55b08e237800 2024-04-23T10:34:36.799 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:34:36.793+0000 7f5c01da5740 10 client.0 _ll_register_callbacks cb 0x55b08e119d30 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-23T10:34:36.799 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:34:36.793+0000 7f5bfcd49640 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x55b08e22c620 0x55b08e2371d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T10:34:36.799 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:34:36.793+0000 7f5bf7fff640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55b08e22cf40 0x55b08e2344c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T10:34:36.799 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:34:36.793+0000 7f5bfcd49640 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x55b08e22c620 0x55b08e2371d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.119:3300/0 says I am v2:172.21.15.114:42710/0 (socket says 172.21.15.114:42710) 2024-04-23T10:34:36.799 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:34:36.793+0000 7f5bfcd49640 1 -- 172.21.15.114:0/415529311 learned_addr learned my addr 172.21.15.114:0/415529311 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-04-23T10:34:36.799 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:34:36.793+0000 7f5bfd54a640 1 --2- 172.21.15.114:0/415529311 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x55b08e237800 0x55b08e234ad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T10:34:36.799 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:34:36.793+0000 7f5bfcd49640 1 -- 172.21.15.114:0/415529311 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x55b08e237800 msgr2=0x55b08e234ad0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T10:34:36.800 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:34:36.793+0000 7f5bfcd49640 1 --2- 172.21.15.114:0/415529311 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x55b08e237800 0x55b08e234ad0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:34:36.800 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:34:36.793+0000 7f5bfcd49640 1 -- 172.21.15.114:0/415529311 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55b08e22cf40 msgr2=0x55b08e2344c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T10:34:36.800 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:34:36.793+0000 7f5bfcd49640 1 --2- 172.21.15.114:0/415529311 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55b08e22cf40 0x55b08e2344c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:34:36.800 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:34:36.793+0000 7f5bfcd49640 1 -- 172.21.15.114:0/415529311 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55b08df70620 con 0x55b08e22c620 2024-04-23T10:34:36.800 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:34:36.793+0000 7f5bfcd49640 1 --2- 172.21.15.114:0/415529311 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x55b08e22c620 0x55b08e2371d0 secure :-1 s=READY pgs=85 cs=0 l=1 rev1=1 crypto rx=0x7f5bec03d040 tx=0x7f5bec03b4f0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=715694b01121999 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T10:34:36.800 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:34:36.793+0000 7f5bd7fff640 10 client.0 ms_handle_connect on v2:172.21.15.119:3300/0 2024-04-23T10:34:36.800 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:34:36.793+0000 7f5bd7fff640 1 -- 172.21.15.114:0/415529311 <== mon.2 v2:172.21.15.119:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5bec049040 con 0x55b08e22c620 2024-04-23T10:34:36.800 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:34:36.793+0000 7f5bd7fff640 1 -- 172.21.15.114:0/415529311 <== mon.2 v2:172.21.15.119:3300/0 2 ==== osd_map(98..98 src has 1..98) ==== 8629+0+0 (secure 0 0 0) 0x7f5bec0b9200 con 0x55b08e22c620 2024-04-23T10:34:36.800 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:34:36.793+0000 7f5c01da5740 1 -- 172.21.15.114:0/415529311 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f5bec0bcb90 con 0x55b08e22c620 2024-04-23T10:34:36.800 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:34:36.794+0000 7f5bd7fff640 1 -- 172.21.15.114:0/415529311 <== mon.2 v2:172.21.15.119:3300/0 3 ==== mon_get_version_reply(handle=1 version=31) ==== 24+0+0 (secure 0 0 0) 0x7f5bec0c5800 con 0x55b08e22c620 2024-04-23T10:34:36.800 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:34:36.794+0000 7f5c01da5740 10 client.25970 fetch_fsmap learned FSMap version 31 2024-04-23T10:34:36.800 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:34:36.794+0000 7f5c01da5740 1 -- 172.21.15.114:0/415529311 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({config=0+,fsmap=31,monmap=4+}) -- 0x55b08e23a1f0 con 0x55b08e22c620 2024-04-23T10:34:36.800 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:34:36.794+0000 7f5bd7fff640 1 -- 172.21.15.114:0/415529311 <== mon.2 v2:172.21.15.119:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f5bec0bcb90 con 0x55b08e22c620 2024-04-23T10:34:36.800 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:34:36.794+0000 7f5bd7fff640 1 -- 172.21.15.114:0/415529311 <== mon.2 v2:172.21.15.119:3300/0 5 ==== fsmap(e 31) ==== 8342+0+0 (secure 0 0 0) 0x7f5bec0d5290 con 0x55b08e22c620 2024-04-23T10:34:36.801 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:34:36.794+0000 7f5c01da5740 10 client.25970 fetch_fsmap finished waiting for FSMap version 31 2024-04-23T10:34:36.801 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:34:36.794+0000 7f5c01da5740 10 client.25970 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-04-23T10:34:36.801 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:34:36.794+0000 7f5c01da5740 20 client.25970 populate_metadata read hostname 'smithi114' 2024-04-23T10:34:36.801 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:34:36.794+0000 7f5c01da5740 1 --2- 172.21.15.114:0/415529311 >> [v2:172.21.15.119:6838/937297340,v1:172.21.15.119:6839/937297340] conn(0x55b08e247450 0x55b08e2498b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T10:34:36.801 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:34:36.794+0000 7f5c01da5740 4 client.25970 mds_command: new command op to 24545 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-23T10:34:36.801 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:34:36.794+0000 7f5c01da5740 1 -- 172.21.15.114:0/415529311 --> [v2:172.21.15.119:6838/937297340,v1:172.21.15.119:6839/937297340] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55b08e19f390 con 0x55b08e247450 2024-04-23T10:34:36.801 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:34:36.794+0000 7f5bf7fff640 1 --2- 172.21.15.114:0/415529311 >> [v2:172.21.15.119:6838/937297340,v1:172.21.15.119:6839/937297340] conn(0x55b08e247450 0x55b08e2498b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T10:34:36.801 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:34:36.794+0000 7f5bf7fff640 1 --2- 172.21.15.114:0/415529311 >> [v2:172.21.15.119:6838/937297340,v1:172.21.15.119:6839/937297340] conn(0x55b08e247450 0x55b08e2498b0 crc :-1 s=READY pgs=132 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=b366126616ee48b1 server_cookie=c7649e86a0380365 in_seq=0 out_seq=0 2024-04-23T10:34:36.801 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:34:36.794+0000 7f5bd7fff640 10 client.25970 ms_handle_connect on v2:172.21.15.119:6838/937297340 2024-04-23T10:34:36.801 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:34:36.795+0000 7f5bd7fff640 1 -- 172.21.15.114:0/415529311 <== mds.0 v2:172.21.15.119:6838/937297340 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x55b08e19f390 con 0x55b08e247450 2024-04-23T10:34:36.801 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:34:36.795+0000 7f5bd7fff640 10 client.25970 handle_command_reply: tid=0 2024-04-23T10:34:36.801 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:34:36.795+0000 7f5c01da5740 1 client.25970 shutdown 2024-04-23T10:34:36.801 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:34:36.796+0000 7f5c01da5740 1 -- 172.21.15.114:0/415529311 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x55b08e22c620 msgr2=0x55b08e2371d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T10:34:36.801 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:34:36.796+0000 7f5c01da5740 1 --2- 172.21.15.114:0/415529311 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x55b08e22c620 0x55b08e2371d0 secure :-1 s=READY pgs=85 cs=0 l=1 rev1=1 crypto rx=0x7f5bec03d040 tx=0x7f5bec03b4f0 comp rx=0 tx=0).stop 2024-04-23T10:34:36.802 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:34:36.796+0000 7f5c01da5740 1 -- 172.21.15.114:0/415529311 shutdown_connections 2024-04-23T10:34:36.802 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:34:36.796+0000 7f5c01da5740 1 --2- 172.21.15.114:0/415529311 >> [v2:172.21.15.119:6838/937297340,v1:172.21.15.119:6839/937297340] conn(0x55b08e247450 0x55b08e2498b0 crc :-1 s=READY pgs=132 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:34:36.802 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:34:36.796+0000 7f5c01da5740 1 --2- 172.21.15.114:0/415529311 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x55b08e237800 0x55b08e234ad0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:34:36.802 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:34:36.796+0000 7f5bd7fff640 0 client.25970 ms_handle_reset on v2:172.21.15.119:6838/937297340 2024-04-23T10:34:36.802 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:34:36.796+0000 7f5c01da5740 1 --2- 172.21.15.114:0/415529311 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55b08e22cf40 0x55b08e2344c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:34:36.802 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:34:36.796+0000 7f5c01da5740 1 --2- 172.21.15.114:0/415529311 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x55b08e22c620 0x55b08e2371d0 unknown :-1 s=CLOSED pgs=85 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:34:36.802 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:34:36.796+0000 7f5c01da5740 1 -- 172.21.15.114:0/415529311 >> 172.21.15.114:0/415529311 conn(0x55b08e228b30 msgr2=0x55b08e22e8b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T10:34:36.802 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:34:36.796+0000 7f5c01da5740 1 -- 172.21.15.114:0/415529311 shutdown_connections 2024-04-23T10:34:36.802 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:34:36.796+0000 7f5c01da5740 1 -- 172.21.15.114:0/415529311 wait complete. 2024-04-23T10:34:36.802 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:34:36.796+0000 7f5c01da5740 20 client.25970 trim_cache size 0 max 16384 2024-04-23T10:34:36.811 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:34:36.811+0000 7f5c01da5740 1 Processor -- start 2024-04-23T10:34:36.811 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:34:36.811+0000 7f5c01da5740 1 -- start start 2024-04-23T10:34:36.811 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:34:36.811+0000 7f5c01da5740 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x55b08e22c620 0x55b08e243000 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T10:34:36.812 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:34:36.811+0000 7f5c01da5740 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x55b08e22cf40 0x55b08e236d00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T10:34:36.812 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:34:36.811+0000 7f5c01da5740 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55b08e237800 0x55b08e24f180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T10:34:36.812 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:34:36.811+0000 7f5c01da5740 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x7f5bf81ae3c0 con 0x55b08e237800 2024-04-23T10:34:36.812 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:34:36.811+0000 7f5c01da5740 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 -- 0x7f5bf81abc60 con 0x55b08e22cf40 2024-04-23T10:34:36.812 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:34:36.811+0000 7f5c01da5740 1 -- --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_getmap magic: 0 -- 0x7f5bf81a9ad0 con 0x55b08e22c620 2024-04-23T10:34:36.812 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:34:36.811+0000 7f5bfd54a640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55b08e237800 0x55b08e24f180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T10:34:36.812 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:34:36.811+0000 7f5bfd54a640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55b08e237800 0x55b08e24f180 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.114:45976/0 (socket says 172.21.15.114:45976) 2024-04-23T10:34:36.812 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:34:36.811+0000 7f5bfd54a640 1 -- 172.21.15.114:0/1186452217 learned_addr learned my addr 172.21.15.114:0/1186452217 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-04-23T10:34:36.812 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:34:36.811+0000 7f5bf7fff640 1 --2- 172.21.15.114:0/1186452217 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x55b08e22cf40 0x55b08e236d00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T10:34:36.812 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:34:36.811+0000 7f5bfcd49640 1 --2- 172.21.15.114:0/1186452217 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x55b08e22c620 0x55b08e243000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T10:34:36.812 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:34:36.811+0000 7f5bfd54a640 1 -- 172.21.15.114:0/1186452217 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x55b08e22c620 msgr2=0x55b08e243000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T10:34:36.812 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:34:36.811+0000 7f5bfd54a640 1 --2- 172.21.15.114:0/1186452217 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x55b08e22c620 0x55b08e243000 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:34:36.812 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:34:36.811+0000 7f5bfd54a640 1 -- 172.21.15.114:0/1186452217 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x55b08e22cf40 msgr2=0x55b08e236d00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T10:34:36.812 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:34:36.811+0000 7f5bfd54a640 1 --2- 172.21.15.114:0/1186452217 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x55b08e22cf40 0x55b08e236d00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:34:36.812 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:34:36.811+0000 7f5bfd54a640 1 -- 172.21.15.114:0/1186452217 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55b08e22d8f0 con 0x55b08e237800 2024-04-23T10:34:36.813 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:34:36.811+0000 7f5bfd54a640 1 --2- 172.21.15.114:0/1186452217 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55b08e237800 0x55b08e24f180 secure :-1 s=READY pgs=791 cs=0 l=1 rev1=1 crypto rx=0x7f5bf0012b20 tx=0x7f5bf0017000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=214331437fa26d6e server_cookie=0 in_seq=0 out_seq=0 2024-04-23T10:34:36.813 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:34:36.812+0000 7f5be27fc640 1 -- 172.21.15.114:0/1186452217 <== mon.0 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5bf0012df0 con 0x55b08e237800 2024-04-23T10:34:36.813 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:34:36.812+0000 7f5be27fc640 1 -- 172.21.15.114:0/1186452217 <== mon.0 v2:172.21.15.114:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f5bf0026070 con 0x55b08e237800 2024-04-23T10:34:36.813 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:34:36.812+0000 7f5be27fc640 1 -- 172.21.15.114:0/1186452217 <== mon.0 v2:172.21.15.114:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5bf002b900 con 0x55b08e237800 2024-04-23T10:34:36.813 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:34:36.812+0000 7f5c01da5740 1 -- 172.21.15.114:0/1186452217 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55b08e237800 msgr2=0x55b08e24f180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T10:34:36.813 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:34:36.812+0000 7f5c01da5740 1 --2- 172.21.15.114:0/1186452217 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55b08e237800 0x55b08e24f180 secure :-1 s=READY pgs=791 cs=0 l=1 rev1=1 crypto rx=0x7f5bf0012b20 tx=0x7f5bf0017000 comp rx=0 tx=0).stop 2024-04-23T10:34:36.813 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:34:36.812+0000 7f5c01da5740 1 -- 172.21.15.114:0/1186452217 shutdown_connections 2024-04-23T10:34:36.813 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:34:36.812+0000 7f5c01da5740 1 --2- 172.21.15.114:0/1186452217 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55b08e237800 0x55b08e24f180 unknown :-1 s=CLOSED pgs=791 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:34:36.813 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:34:36.812+0000 7f5c01da5740 1 --2- 172.21.15.114:0/1186452217 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x55b08e22cf40 0x55b08e236d00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:34:36.813 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:34:36.812+0000 7f5c01da5740 1 --2- 172.21.15.114:0/1186452217 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x55b08e22c620 0x55b08e243000 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:34:36.813 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:34:36.812+0000 7f5c01da5740 1 -- 172.21.15.114:0/1186452217 >> 172.21.15.114:0/1186452217 conn(0x55b08e228b30 msgr2=0x55b08e22f9d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T10:34:36.813 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:34:36.812+0000 7f5c01da5740 1 -- 172.21.15.114:0/1186452217 shutdown_connections 2024-04-23T10:34:36.813 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:34:36.812+0000 7f5c01da5740 1 -- 172.21.15.114:0/1186452217 wait complete. 2024-04-23T10:34:36.813 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:34:36.813+0000 7f5c01da5740 1 -- start start 2024-04-23T10:34:36.813 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:34:36.813+0000 7f5c01da5740 1 Processor -- start 2024-04-23T10:34:36.816 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:34:36.813+0000 7f5c01da5740 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55b08e22c620 0x55b08e23b380 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T10:34:36.816 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:34:36.813+0000 7f5c01da5740 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x55b08e22cf40 0x55b08e23b8c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T10:34:36.816 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:34:36.813+0000 7f5c01da5740 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x55b08e237800 0x55b08e2ac9f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T10:34:36.816 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:34:36.813+0000 7f5c01da5740 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x7f5bf81aa120 con 0x55b08e22c620 2024-04-23T10:34:36.816 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:34:36.813+0000 7f5c01da5740 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 -- 0x7f5bf81ab590 con 0x55b08e237800 2024-04-23T10:34:36.816 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:34:36.813+0000 7f5c01da5740 1 -- --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_getmap magic: 0 -- 0x7f5bf81aa7f0 con 0x55b08e22cf40 2024-04-23T10:34:36.816 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:34:36.813+0000 7f5c01da5740 10 client.0 _ll_register_callbacks cb 0x7f5bec0b8ee0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-23T10:34:36.816 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:34:36.813+0000 7f5bfcd49640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55b08e22c620 0x55b08e23b380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T10:34:36.816 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:34:36.813+0000 7f5bfcd49640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55b08e22c620 0x55b08e23b380 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.114:45992/0 (socket says 172.21.15.114:45992) 2024-04-23T10:34:36.817 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:34:36.813+0000 7f5bfcd49640 1 -- 172.21.15.114:0/2971339845 learned_addr learned my addr 172.21.15.114:0/2971339845 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-04-23T10:34:36.817 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:34:36.813+0000 7f5bf7fff640 1 --2- 172.21.15.114:0/2971339845 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x55b08e22cf40 0x55b08e23b8c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T10:34:36.817 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:34:36.813+0000 7f5bfd54a640 1 --2- 172.21.15.114:0/2971339845 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x55b08e237800 0x55b08e2ac9f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T10:34:36.817 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:34:36.813+0000 7f5bfcd49640 1 -- 172.21.15.114:0/2971339845 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x55b08e22cf40 msgr2=0x55b08e23b8c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T10:34:36.817 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:34:36.813+0000 7f5bfcd49640 1 --2- 172.21.15.114:0/2971339845 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x55b08e22cf40 0x55b08e23b8c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:34:36.817 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:34:36.813+0000 7f5bfcd49640 1 -- 172.21.15.114:0/2971339845 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x55b08e237800 msgr2=0x55b08e2ac9f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T10:34:36.817 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:34:36.813+0000 7f5bfcd49640 1 --2- 172.21.15.114:0/2971339845 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x55b08e237800 0x55b08e2ac9f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:34:36.817 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:34:36.813+0000 7f5bfcd49640 1 -- 172.21.15.114:0/2971339845 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55b08e2acf30 con 0x55b08e22c620 2024-04-23T10:34:36.817 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:34:36.814+0000 7f5bfcd49640 1 --2- 172.21.15.114:0/2971339845 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55b08e22c620 0x55b08e23b380 secure :-1 s=READY pgs=792 cs=0 l=1 rev1=1 crypto rx=0x7f5bec0b9560 tx=0x7f5bec0bc790 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9d69c33f9f7e9d83 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T10:34:36.817 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:34:36.814+0000 7f5bd7fff640 10 client.0 ms_handle_connect on v2:172.21.15.114:3300/0 2024-04-23T10:34:36.817 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:34:36.814+0000 7f5bd7fff640 1 -- 172.21.15.114:0/2971339845 <== mon.0 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5bec049040 con 0x55b08e22c620 2024-04-23T10:34:36.817 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:34:36.814+0000 7f5bd7fff640 1 -- 172.21.15.114:0/2971339845 <== mon.0 v2:172.21.15.114:3300/0 2 ==== osd_map(98..98 src has 1..98) ==== 8629+0+0 (secure 0 0 0) 0x7f5bec0bce40 con 0x55b08e22c620 2024-04-23T10:34:36.817 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:34:36.814+0000 7f5c01da5740 1 -- 172.21.15.114:0/2971339845 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f5bf0026070 con 0x55b08e22c620 2024-04-23T10:34:36.817 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:34:36.814+0000 7f5bd7fff640 1 -- 172.21.15.114:0/2971339845 <== mon.0 v2:172.21.15.114:3300/0 3 ==== mon_get_version_reply(handle=1 version=31) ==== 24+0+0 (secure 0 0 0) 0x7f5bec0bd040 con 0x55b08e22c620 2024-04-23T10:34:36.817 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:34:36.814+0000 7f5c01da5740 10 client.16608 fetch_fsmap learned FSMap version 31 2024-04-23T10:34:36.817 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:34:36.814+0000 7f5c01da5740 1 -- 172.21.15.114:0/2971339845 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({config=0+,fsmap=31,monmap=4+}) -- 0x55b08e2acf30 con 0x55b08e22c620 2024-04-23T10:34:36.818 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:34:36.814+0000 7f5bd7fff640 1 -- 172.21.15.114:0/2971339845 <== mon.0 v2:172.21.15.114:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f5bf0026070 con 0x55b08e22c620 2024-04-23T10:34:36.818 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:34:36.814+0000 7f5bd7fff640 1 -- 172.21.15.114:0/2971339845 <== mon.0 v2:172.21.15.114:3300/0 5 ==== fsmap(e 31) ==== 8342+0+0 (secure 0 0 0) 0x7f5bec0b8ae0 con 0x55b08e22c620 2024-04-23T10:34:36.818 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:34:36.814+0000 7f5c01da5740 10 client.16608 fetch_fsmap finished waiting for FSMap version 31 2024-04-23T10:34:36.818 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:34:36.814+0000 7f5c01da5740 10 client.16608 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-04-23T10:34:36.818 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:34:36.814+0000 7f5c01da5740 20 client.16608 populate_metadata read hostname 'smithi114' 2024-04-23T10:34:36.818 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:34:36.814+0000 7f5c01da5740 1 --2- 172.21.15.114:0/2971339845 >> [v2:172.21.15.119:6838/937297340,v1:172.21.15.119:6839/937297340] conn(0x55b08e2ae480 0x55b08e2b08a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T10:34:36.818 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:34:36.814+0000 7f5c01da5740 4 client.16608 mds_command: new command op to 24545 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-04-23T10:34:36.818 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:34:36.814+0000 7f5c01da5740 1 -- 172.21.15.114:0/2971339845 --> [v2:172.21.15.119:6838/937297340,v1:172.21.15.119:6839/937297340] -- command(tid 0: {"prefix": "flush journal"}) -- 0x55b08e199200 con 0x55b08e2ae480 2024-04-23T10:34:36.818 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:34:36.814+0000 7f5bf7fff640 1 --2- 172.21.15.114:0/2971339845 >> [v2:172.21.15.119:6838/937297340,v1:172.21.15.119:6839/937297340] conn(0x55b08e2ae480 0x55b08e2b08a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T10:34:36.818 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:34:36.815+0000 7f5bf7fff640 1 --2- 172.21.15.114:0/2971339845 >> [v2:172.21.15.119:6838/937297340,v1:172.21.15.119:6839/937297340] conn(0x55b08e2ae480 0x55b08e2b08a0 crc :-1 s=READY pgs=133 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=8193b21e08d92721 server_cookie=9b3234408d4286e1 in_seq=0 out_seq=0 2024-04-23T10:34:36.818 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:34:36.815+0000 7f5bd7fff640 10 client.16608 ms_handle_connect on v2:172.21.15.119:6838/937297340 2024-04-23T10:34:36.898 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:34:36.898+0000 7f5bd7fff640 1 -- 172.21.15.114:0/2971339845 <== mds.0 v2:172.21.15.119:6838/937297340 1 ==== command_reply(tid 0: 0 ) ==== 8+0+44 (crc 0 0 0) 0x55b08e199200 con 0x55b08e2ae480 2024-04-23T10:34:36.898 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:34:36.898+0000 7f5bd7fff640 10 client.16608 handle_command_reply: tid=0 2024-04-23T10:34:36.898 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:34:36.899+0000 7f5c01da5740 1 client.16608 shutdown 2024-04-23T10:34:36.898 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:34:36.899+0000 7f5c01da5740 1 -- 172.21.15.114:0/2971339845 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55b08e22c620 msgr2=0x55b08e23b380 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T10:34:36.898 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:34:36.899+0000 7f5c01da5740 1 --2- 172.21.15.114:0/2971339845 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55b08e22c620 0x55b08e23b380 secure :-1 s=READY pgs=792 cs=0 l=1 rev1=1 crypto rx=0x7f5bec0b9560 tx=0x7f5bec0bc790 comp rx=0 tx=0).stop 2024-04-23T10:34:36.898 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:34:36.899+0000 7f5c01da5740 1 -- 172.21.15.114:0/2971339845 shutdown_connections 2024-04-23T10:34:36.898 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:34:36.899+0000 7f5c01da5740 1 --2- 172.21.15.114:0/2971339845 >> [v2:172.21.15.119:6838/937297340,v1:172.21.15.119:6839/937297340] conn(0x55b08e2ae480 0x55b08e2b08a0 crc :-1 s=READY pgs=133 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:34:36.898 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:34:36.899+0000 7f5c01da5740 1 --2- 172.21.15.114:0/2971339845 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x55b08e237800 0x55b08e2ac9f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:34:36.899 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:34:36.899+0000 7f5c01da5740 1 --2- 172.21.15.114:0/2971339845 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x55b08e22cf40 0x55b08e23b8c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:34:36.899 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:34:36.899+0000 7f5c01da5740 1 --2- 172.21.15.114:0/2971339845 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55b08e22c620 0x55b08e23b380 unknown :-1 s=CLOSED pgs=792 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:34:36.899 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:34:36.899+0000 7f5bd7fff640 0 client.16608 ms_handle_reset on v2:172.21.15.119:6838/937297340 2024-04-23T10:34:36.899 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:34:36.899+0000 7f5c01da5740 1 -- 172.21.15.114:0/2971339845 >> 172.21.15.114:0/2971339845 conn(0x55b08e228b30 msgr2=0x55b08e22f120 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T10:34:36.899 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:34:36.899+0000 7f5c01da5740 1 -- 172.21.15.114:0/2971339845 shutdown_connections 2024-04-23T10:34:36.899 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:34:36.899+0000 7f5c01da5740 1 -- 172.21.15.114:0/2971339845 wait complete. 2024-04-23T10:34:36.899 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:34:36.899+0000 7f5c01da5740 20 client.16608 trim_cache size 0 max 16384 2024-04-23T10:34:36.899 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stdout:{ 2024-04-23T10:34:36.899 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stdout: "message": "", 2024-04-23T10:34:36.899 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stdout: "return_code": 0 2024-04-23T10:34:36.899 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stdout:} 2024-04-23T10:34:36.900 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:34:36.900+0000 7f5bfefad640 1 -- 172.21.15.114:0/3204731123 >> [v2:172.21.15.114:6800/1395693466,v1:172.21.15.114:6801/1395693466] conn(0x7f5be80a3dc0 msgr2=0x7f5be80a6280 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T10:34:36.900 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:34:36.900+0000 7f5bfefad640 1 --2- 172.21.15.114:0/3204731123 >> [v2:172.21.15.114:6800/1395693466,v1:172.21.15.114:6801/1395693466] conn(0x7f5be80a3dc0 0x7f5be80a6280 secure :-1 s=READY pgs=120 cs=0 l=1 rev1=1 crypto rx=0x55b08e229f50 tx=0x7f5bf0004ab0 comp rx=0 tx=0).stop 2024-04-23T10:34:36.900 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:34:36.900+0000 7f5bfefad640 1 -- 172.21.15.114:0/3204731123 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f5bf81010f0 msgr2=0x7f5bf8196550 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T10:34:36.900 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:34:36.900+0000 7f5bfefad640 1 --2- 172.21.15.114:0/3204731123 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f5bf81010f0 0x7f5bf8196550 secure :-1 s=READY pgs=790 cs=0 l=1 rev1=1 crypto rx=0x7f5bec005940 tx=0x7f5bec0040c0 comp rx=0 tx=0).stop 2024-04-23T10:34:36.900 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:34:36.900+0000 7f5bfefad640 1 -- 172.21.15.114:0/3204731123 shutdown_connections 2024-04-23T10:34:36.901 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:34:36.900+0000 7f5bfefad640 1 --2- 172.21.15.114:0/3204731123 >> [v2:172.21.15.114:6800/1395693466,v1:172.21.15.114:6801/1395693466] conn(0x7f5be80a3dc0 0x7f5be80a6280 unknown :-1 s=CLOSED pgs=120 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:34:36.901 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:34:36.900+0000 7f5bfefad640 1 --2- 172.21.15.114:0/3204731123 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f5bf819ae10 0x7f5bf819b2c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:34:36.901 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:34:36.900+0000 7f5bfefad640 1 --2- 172.21.15.114:0/3204731123 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f5bf8106170 0x7f5bf8196a90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:34:36.901 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:34:36.900+0000 7f5bfefad640 1 --2- 172.21.15.114:0/3204731123 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f5bf81010f0 0x7f5bf8196550 unknown :-1 s=CLOSED pgs=790 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:34:36.901 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:34:36.900+0000 7f5bfefad640 1 -- 172.21.15.114:0/3204731123 >> 172.21.15.114:0/3204731123 conn(0x7f5bf8077fd0 msgr2=0x7f5bf80ff0e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T10:34:36.901 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:34:36.900+0000 7f5bfefad640 1 -- 172.21.15.114:0/3204731123 shutdown_connections 2024-04-23T10:34:36.901 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:34:36.900+0000 7f5bfefad640 1 -- 172.21.15.114:0/3204731123 wait complete. 2024-04-23T10:34:36.929 INFO:tasks.workunit.client.0.smithi114.stdout: 1 558325 7.25 MB/sec execute 285 sec latency 20.504 ms 2024-04-23T10:34:37.742 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 23 10:34:37 smithi119 ceph-mon[32608]: pgmap v249: 97 pgs: 97 active+clean; 186 MiB data, 2.1 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 MiB/s rd, 4.6 MiB/s wr, 375 op/s 2024-04-23T10:34:37.754 INFO:journalctl@ceph.mon.a.smithi114.stdout:Apr 23 10:34:37 smithi114 ceph-mon[29255]: pgmap v249: 97 pgs: 97 active+clean; 186 MiB data, 2.1 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 MiB/s rd, 4.6 MiB/s wr, 375 op/s 2024-04-23T10:34:37.827 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 23 10:34:37 smithi204 ceph-mon[32649]: pgmap v249: 97 pgs: 97 active+clean; 186 MiB data, 2.1 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 MiB/s rd, 4.6 MiB/s wr, 375 op/s 2024-04-23T10:34:37.929 INFO:tasks.workunit.client.0.smithi114.stdout: 1 560670 7.27 MB/sec execute 286 sec latency 74.454 ms 2024-04-23T10:34:38.929 INFO:tasks.workunit.client.0.smithi114.stdout: 1 562346 7.27 MB/sec execute 287 sec latency 20.979 ms 2024-04-23T10:34:39.742 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 23 10:34:39 smithi119 ceph-mon[32608]: pgmap v250: 97 pgs: 97 active+clean; 186 MiB data, 2.1 GiB used, 1.0 TiB / 1.0 TiB avail; 1.4 MiB/s rd, 3.0 MiB/s wr, 222 op/s 2024-04-23T10:34:39.753 INFO:journalctl@ceph.mon.a.smithi114.stdout:Apr 23 10:34:39 smithi114 ceph-mon[29255]: pgmap v250: 97 pgs: 97 active+clean; 186 MiB data, 2.1 GiB used, 1.0 TiB / 1.0 TiB avail; 1.4 MiB/s rd, 3.0 MiB/s wr, 222 op/s 2024-04-23T10:34:39.827 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 23 10:34:39 smithi204 ceph-mon[32649]: pgmap v250: 97 pgs: 97 active+clean; 186 MiB data, 2.1 GiB used, 1.0 TiB / 1.0 TiB avail; 1.4 MiB/s rd, 3.0 MiB/s wr, 222 op/s 2024-04-23T10:34:39.929 INFO:tasks.workunit.client.0.smithi114.stdout: 1 564942 7.30 MB/sec execute 288 sec latency 21.509 ms 2024-04-23T10:34:40.929 INFO:tasks.workunit.client.0.smithi114.stdout: 1 566219 7.30 MB/sec execute 289 sec latency 58.739 ms 2024-04-23T10:34:41.741 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 23 10:34:41 smithi119 ceph-mon[32608]: pgmap v251: 97 pgs: 97 active+clean; 186 MiB data, 2.1 GiB used, 1.0 TiB / 1.0 TiB avail; 1.4 MiB/s rd, 3.0 MiB/s wr, 222 op/s 2024-04-23T10:34:41.754 INFO:journalctl@ceph.mon.a.smithi114.stdout:Apr 23 10:34:41 smithi114 ceph-mon[29255]: pgmap v251: 97 pgs: 97 active+clean; 186 MiB data, 2.1 GiB used, 1.0 TiB / 1.0 TiB avail; 1.4 MiB/s rd, 3.0 MiB/s wr, 222 op/s 2024-04-23T10:34:41.827 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 23 10:34:41 smithi204 ceph-mon[32649]: pgmap v251: 97 pgs: 97 active+clean; 186 MiB data, 2.1 GiB used, 1.0 TiB / 1.0 TiB avail; 1.4 MiB/s rd, 3.0 MiB/s wr, 222 op/s 2024-04-23T10:34:41.929 INFO:tasks.workunit.client.0.smithi114.stdout: 1 568580 7.32 MB/sec execute 290 sec latency 29.054 ms 2024-04-23T10:34:42.929 INFO:tasks.workunit.client.0.smithi114.stdout: 1 570036 7.32 MB/sec execute 291 sec latency 14.983 ms 2024-04-23T10:34:43.741 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 23 10:34:43 smithi119 ceph-mon[32608]: pgmap v252: 97 pgs: 97 active+clean; 182 MiB data, 2.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 MiB/s rd, 4.4 MiB/s wr, 376 op/s 2024-04-23T10:34:43.753 INFO:journalctl@ceph.mon.a.smithi114.stdout:Apr 23 10:34:43 smithi114 ceph-mon[29255]: pgmap v252: 97 pgs: 97 active+clean; 182 MiB data, 2.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 MiB/s rd, 4.4 MiB/s wr, 376 op/s 2024-04-23T10:34:43.827 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 23 10:34:43 smithi204 ceph-mon[32649]: pgmap v252: 97 pgs: 97 active+clean; 182 MiB data, 2.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 MiB/s rd, 4.4 MiB/s wr, 376 op/s 2024-04-23T10:34:43.929 INFO:tasks.workunit.client.0.smithi114.stdout: 1 572295 7.34 MB/sec execute 292 sec latency 13.833 ms 2024-04-23T10:34:44.929 INFO:tasks.workunit.client.0.smithi114.stdout: 1 574317 7.34 MB/sec execute 293 sec latency 19.122 ms 2024-04-23T10:34:45.742 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 23 10:34:45 smithi119 ceph-mon[32608]: pgmap v253: 97 pgs: 97 active+clean; 182 MiB data, 2.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.3 MiB/s rd, 2.9 MiB/s wr, 267 op/s 2024-04-23T10:34:45.753 INFO:journalctl@ceph.mon.a.smithi114.stdout:Apr 23 10:34:45 smithi114 ceph-mon[29255]: pgmap v253: 97 pgs: 97 active+clean; 182 MiB data, 2.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.3 MiB/s rd, 2.9 MiB/s wr, 267 op/s 2024-04-23T10:34:45.827 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 23 10:34:45 smithi204 ceph-mon[32649]: pgmap v253: 97 pgs: 97 active+clean; 182 MiB data, 2.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.3 MiB/s rd, 2.9 MiB/s wr, 267 op/s 2024-04-23T10:34:45.929 INFO:tasks.workunit.client.0.smithi114.stdout: 1 575974 7.35 MB/sec execute 294 sec latency 63.131 ms 2024-04-23T10:34:46.929 INFO:tasks.workunit.client.0.smithi114.stdout: 1 577819 7.35 MB/sec execute 295 sec latency 63.739 ms 2024-04-23T10:34:47.741 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 23 10:34:47 smithi119 ceph-mon[32608]: pgmap v254: 97 pgs: 97 active+clean; 189 MiB data, 2.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 MiB/s rd, 4.5 MiB/s wr, 374 op/s 2024-04-23T10:34:47.753 INFO:journalctl@ceph.mon.a.smithi114.stdout:Apr 23 10:34:47 smithi114 ceph-mon[29255]: pgmap v254: 97 pgs: 97 active+clean; 189 MiB data, 2.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 MiB/s rd, 4.5 MiB/s wr, 374 op/s 2024-04-23T10:34:47.827 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 23 10:34:47 smithi204 ceph-mon[32649]: pgmap v254: 97 pgs: 97 active+clean; 189 MiB data, 2.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 MiB/s rd, 4.5 MiB/s wr, 374 op/s 2024-04-23T10:34:47.930 INFO:tasks.workunit.client.0.smithi114.stdout: 1 579589 7.37 MB/sec execute 296 sec latency 16.449 ms 2024-04-23T10:34:48.930 INFO:tasks.workunit.client.0.smithi114.stdout: 1 581981 7.39 MB/sec execute 297 sec latency 23.124 ms 2024-04-23T10:34:49.742 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 23 10:34:49 smithi119 ceph-mon[32608]: pgmap v255: 97 pgs: 97 active+clean; 189 MiB data, 2.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.3 MiB/s rd, 2.9 MiB/s wr, 261 op/s 2024-04-23T10:34:49.753 INFO:journalctl@ceph.mon.a.smithi114.stdout:Apr 23 10:34:49 smithi114 ceph-mon[29255]: pgmap v255: 97 pgs: 97 active+clean; 189 MiB data, 2.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.3 MiB/s rd, 2.9 MiB/s wr, 261 op/s 2024-04-23T10:34:49.827 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 23 10:34:49 smithi204 ceph-mon[32649]: pgmap v255: 97 pgs: 97 active+clean; 189 MiB data, 2.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.3 MiB/s rd, 2.9 MiB/s wr, 261 op/s 2024-04-23T10:34:49.930 INFO:tasks.workunit.client.0.smithi114.stdout: 1 583595 7.39 MB/sec execute 298 sec latency 16.703 ms 2024-04-23T10:34:50.008 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:34:50.009+0000 7feabf7fe640 1 -- 172.21.15.114:0/1712537153 <== mon.1 v2:172.21.15.204:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7feab002c040 con 0x7feac0106390 2024-04-23T10:34:50.008 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:34:50.009+0000 7feac7091640 1 -- 172.21.15.114:0/1712537153 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7feac0106390 msgr2=0x7feac0106810 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T10:34:50.009 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:34:50.009+0000 7feac7091640 1 --2- 172.21.15.114:0/1712537153 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7feac0106390 0x7feac0106810 secure :-1 s=READY pgs=78 cs=0 l=1 rev1=1 crypto rx=0x7feab0009ca0 tx=0x7feab002a950 comp rx=0 tx=0).stop 2024-04-23T10:34:50.009 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:34:50.009+0000 7feac7091640 1 -- 172.21.15.114:0/1712537153 shutdown_connections 2024-04-23T10:34:50.009 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:34:50.009+0000 7feac7091640 1 --2- 172.21.15.114:0/1712537153 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7feac0106390 0x7feac0106810 unknown :-1 s=CLOSED pgs=78 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:34:50.009 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:34:50.009+0000 7feac7091640 1 --2- 172.21.15.114:0/1712537153 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7feac01018b0 0x7feac0105d70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:34:50.009 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:34:50.009+0000 7feac7091640 1 --2- 172.21.15.114:0/1712537153 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7feac0100f90 0x7feac0101370 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:34:50.009 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:34:50.009+0000 7feac7091640 1 -- 172.21.15.114:0/1712537153 >> 172.21.15.114:0/1712537153 conn(0x7feac0078060 msgr2=0x7feac0078470 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T10:34:50.009 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:34:50.010+0000 7feac7091640 1 -- 172.21.15.114:0/1712537153 shutdown_connections 2024-04-23T10:34:50.009 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:34:50.010+0000 7feac7091640 1 -- 172.21.15.114:0/1712537153 wait complete. 2024-04-23T10:34:50.011 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:34:50.010+0000 7feac7091640 1 Processor -- start 2024-04-23T10:34:50.011 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:34:50.010+0000 7feac7091640 1 -- start start 2024-04-23T10:34:50.011 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:34:50.011+0000 7feac7091640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7feac0100f90 0x7feac01a4880 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T10:34:50.011 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:34:50.011+0000 7feac7091640 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7feac01018b0 0x7feac01a4dc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T10:34:50.011 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:34:50.011+0000 7feac7091640 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7feac0106390 0x7feac019ea30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T10:34:50.012 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:34:50.011+0000 7feac7091640 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x7feac0118810 con 0x7feac0100f90 2024-04-23T10:34:50.012 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:34:50.011+0000 7feac7091640 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 -- 0x7feac0118690 con 0x7feac01018b0 2024-04-23T10:34:50.012 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:34:50.011+0000 7feac7091640 1 -- --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_getmap magic: 0 -- 0x7feac0118990 con 0x7feac0106390 2024-04-23T10:34:50.012 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:34:50.011+0000 7feac4e2d640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7feac0100f90 0x7feac01a4880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T10:34:50.012 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:34:50.011+0000 7feac4e2d640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7feac0100f90 0x7feac01a4880 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.114:36846/0 (socket says 172.21.15.114:36846) 2024-04-23T10:34:50.012 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:34:50.011+0000 7feac4e2d640 1 -- 172.21.15.114:0/4088759243 learned_addr learned my addr 172.21.15.114:0/4088759243 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-04-23T10:34:50.012 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:34:50.011+0000 7feac562e640 1 --2- 172.21.15.114:0/4088759243 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7feac0106390 0x7feac019ea30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T10:34:50.012 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:34:50.011+0000 7feabffff640 1 --2- 172.21.15.114:0/4088759243 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7feac01018b0 0x7feac01a4dc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T10:34:50.012 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:34:50.011+0000 7feac4e2d640 1 -- 172.21.15.114:0/4088759243 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7feac0106390 msgr2=0x7feac019ea30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T10:34:50.012 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:34:50.011+0000 7feac4e2d640 1 --2- 172.21.15.114:0/4088759243 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7feac0106390 0x7feac019ea30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:34:50.012 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:34:50.011+0000 7feac4e2d640 1 -- 172.21.15.114:0/4088759243 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7feac01018b0 msgr2=0x7feac01a4dc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T10:34:50.012 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:34:50.011+0000 7feac4e2d640 1 --2- 172.21.15.114:0/4088759243 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7feac01018b0 0x7feac01a4dc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:34:50.012 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:34:50.011+0000 7feac4e2d640 1 -- 172.21.15.114:0/4088759243 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7feac019f1b0 con 0x7feac0100f90 2024-04-23T10:34:50.013 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:34:50.011+0000 7feac4e2d640 1 --2- 172.21.15.114:0/4088759243 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7feac0100f90 0x7feac01a4880 secure :-1 s=READY pgs=793 cs=0 l=1 rev1=1 crypto rx=0x7feab400e9f0 tx=0x7feab400eeb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T10:34:50.013 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:34:50.011+0000 7feabdffb640 1 -- 172.21.15.114:0/4088759243 <== mon.0 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7feab400cd80 con 0x7feac0100f90 2024-04-23T10:34:50.013 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:34:50.011+0000 7feac7091640 1 -- 172.21.15.114:0/4088759243 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7feac019f4a0 con 0x7feac0100f90 2024-04-23T10:34:50.013 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:34:50.012+0000 7feac7091640 1 -- 172.21.15.114:0/4088759243 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7feac01ab740 con 0x7feac0100f90 2024-04-23T10:34:50.013 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:34:50.012+0000 7feabdffb640 1 -- 172.21.15.114:0/4088759243 <== mon.0 v2:172.21.15.114:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7feab40108e0 con 0x7feac0100f90 2024-04-23T10:34:50.013 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:34:50.012+0000 7feabdffb640 1 -- 172.21.15.114:0/4088759243 <== mon.0 v2:172.21.15.114:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7feab40188e0 con 0x7feac0100f90 2024-04-23T10:34:50.015 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:34:50.015+0000 7feac9dab740 1 Processor -- start 2024-04-23T10:34:50.015 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:34:50.015+0000 7feac9dab740 1 -- start start 2024-04-23T10:34:50.015 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:34:50.015+0000 7feabdffb640 1 -- 172.21.15.114:0/4088759243 <== mon.0 v2:172.21.15.114:3300/0 4 ==== mgrmap(e 26) ==== 138778+0+0 (secure 0 0 0) 0x7feab4010430 con 0x7feac0100f90 2024-04-23T10:34:50.015 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:34:50.015+0000 7feac9dab740 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x55b813819620 0x55b813819a00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T10:34:50.015 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:34:50.015+0000 7feac9dab740 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x55b813819f40 0x55b8138242c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T10:34:50.015 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:34:50.015+0000 7feac9dab740 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55b813824800 0x55b813826bf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T10:34:50.015 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:34:50.015+0000 7feac9dab740 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x55b81377e970 con 0x55b813824800 2024-04-23T10:34:50.015 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:34:50.015+0000 7feac9dab740 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 -- 0x55b81371b4a0 con 0x55b813819620 2024-04-23T10:34:50.015 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:34:50.015+0000 7feac9dab740 1 -- --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_getmap magic: 0 -- 0x55b81377a930 con 0x55b813819f40 2024-04-23T10:34:50.015 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:34:50.016+0000 7feac562e640 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x55b813819620 0x55b813819a00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T10:34:50.015 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:34:50.016+0000 7feac562e640 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x55b813819620 0x55b813819a00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.119:3300/0 says I am v2:172.21.15.114:58282/0 (socket says 172.21.15.114:58282) 2024-04-23T10:34:50.015 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:34:50.016+0000 7feac562e640 1 -- 172.21.15.114:0/1211470653 learned_addr learned my addr 172.21.15.114:0/1211470653 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-04-23T10:34:50.016 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:34:50.016+0000 7feabdffb640 1 --2- 172.21.15.114:0/4088759243 >> [v2:172.21.15.114:6800/1395693466,v1:172.21.15.114:6801/1395693466] conn(0x7feaa80a3e40 0x7feaa80a6300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T10:34:50.016 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:34:50.016+0000 7feabffff640 1 --2- 172.21.15.114:0/1211470653 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55b813824800 0x55b813826bf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T10:34:50.016 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:34:50.016+0000 7feabdffb640 1 -- 172.21.15.114:0/4088759243 <== mon.0 v2:172.21.15.114:3300/0 5 ==== osd_map(98..98 src has 1..98) ==== 8629+0+0 (secure 0 0 0) 0x7feab40d0a40 con 0x7feac0100f90 2024-04-23T10:34:50.016 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:34:50.016+0000 7feabffff640 1 -- 172.21.15.114:0/1211470653 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x55b813819f40 msgr2=0x55b8138242c0 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2024-04-23T10:34:50.016 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:34:50.016+0000 7feabffff640 1 --2- 172.21.15.114:0/1211470653 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x55b813819f40 0x55b8138242c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:34:50.016 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:34:50.016+0000 7feabffff640 1 -- 172.21.15.114:0/1211470653 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x55b813819620 msgr2=0x55b813819a00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T10:34:50.016 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:34:50.016+0000 7feabffff640 1 --2- 172.21.15.114:0/1211470653 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x55b813819620 0x55b813819a00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:34:50.016 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:34:50.016+0000 7feabffff640 1 -- 172.21.15.114:0/1211470653 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55b813491210 con 0x55b813824800 2024-04-23T10:34:50.017 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:34:50.017+0000 7feabffff640 1 --2- 172.21.15.114:0/1211470653 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55b813824800 0x55b813826bf0 secure :-1 s=READY pgs=794 cs=0 l=1 rev1=1 crypto rx=0x7feac01a0220 tx=0x7feab8011810 comp rx=0 tx=0).ready entity=mon.0 client_cookie=553a2b3266ecb21e server_cookie=0 in_seq=0 out_seq=0 2024-04-23T10:34:50.017 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:34:50.017+0000 7feaa6ffd640 1 -- 172.21.15.114:0/1211470653 <== mon.0 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7feab8004050 con 0x55b813824800 2024-04-23T10:34:50.017 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:34:50.017+0000 7feaa6ffd640 1 -- 172.21.15.114:0/1211470653 <== mon.0 v2:172.21.15.114:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7feab80054d0 con 0x55b813824800 2024-04-23T10:34:50.017 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:34:50.017+0000 7feaa6ffd640 1 -- 172.21.15.114:0/1211470653 <== mon.0 v2:172.21.15.114:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7feab8004c40 con 0x55b813824800 2024-04-23T10:34:50.018 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:34:50.018+0000 7feac562e640 1 --2- 172.21.15.114:0/4088759243 >> [v2:172.21.15.114:6800/1395693466,v1:172.21.15.114:6801/1395693466] conn(0x7feaa80a3e40 0x7feaa80a6300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T10:34:50.018 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:34:50.018+0000 7feac9dab740 1 -- 172.21.15.114:0/1211470653 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55b813824800 msgr2=0x55b813826bf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T10:34:50.018 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:34:50.018+0000 7feac9dab740 1 --2- 172.21.15.114:0/1211470653 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55b813824800 0x55b813826bf0 secure :-1 s=READY pgs=794 cs=0 l=1 rev1=1 crypto rx=0x7feac01a0220 tx=0x7feab8011810 comp rx=0 tx=0).stop 2024-04-23T10:34:50.018 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:34:50.018+0000 7feac9dab740 1 -- 172.21.15.114:0/1211470653 shutdown_connections 2024-04-23T10:34:50.019 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:34:50.018+0000 7feac9dab740 1 --2- 172.21.15.114:0/1211470653 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55b813824800 0x55b813826bf0 unknown :-1 s=CLOSED pgs=794 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:34:50.019 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:34:50.018+0000 7feac9dab740 1 --2- 172.21.15.114:0/1211470653 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x55b813819f40 0x55b8138242c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:34:50.019 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:34:50.018+0000 7feac9dab740 1 --2- 172.21.15.114:0/1211470653 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x55b813819620 0x55b813819a00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:34:50.019 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:34:50.018+0000 7feac9dab740 1 -- 172.21.15.114:0/1211470653 >> 172.21.15.114:0/1211470653 conn(0x55b813815b30 msgr2=0x55b813815f10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T10:34:50.019 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:34:50.018+0000 7feac562e640 1 --2- 172.21.15.114:0/4088759243 >> [v2:172.21.15.114:6800/1395693466,v1:172.21.15.114:6801/1395693466] conn(0x7feaa80a3e40 0x7feaa80a6300 secure :-1 s=READY pgs=121 cs=0 l=1 rev1=1 crypto rx=0x7feab002ae70 tx=0x7feab002a950 comp rx=0 tx=0).ready entity=mgr.15291 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T10:34:50.019 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:34:50.018+0000 7feac9dab740 1 -- 172.21.15.114:0/1211470653 shutdown_connections 2024-04-23T10:34:50.019 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:34:50.019+0000 7feac9dab740 1 -- 172.21.15.114:0/1211470653 wait complete. 2024-04-23T10:34:50.019 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:34:50.019+0000 7feac9dab740 1 -- start start 2024-04-23T10:34:50.019 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:34:50.019+0000 7feac9dab740 1 Processor -- start 2024-04-23T10:34:50.019 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:34:50.020+0000 7feac9dab740 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x55b813819620 0x55b813824040 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T10:34:50.019 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:34:50.020+0000 7feac9dab740 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x55b813819f40 0x55b813821330 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T10:34:50.019 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:34:50.020+0000 7feac9dab740 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55b813824800 0x55b813821870 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T10:34:50.019 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:34:50.020+0000 7feac9dab740 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x55b813827430 con 0x55b813824800 2024-04-23T10:34:50.019 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:34:50.020+0000 7feac9dab740 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 -- 0x55b8137190b0 con 0x55b813819f40 2024-04-23T10:34:50.020 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:34:50.020+0000 7feac9dab740 1 -- --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_getmap magic: 0 -- 0x55b8138275b0 con 0x55b813819620 2024-04-23T10:34:50.020 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:34:50.020+0000 7feac9dab740 10 client.0 _ll_register_callbacks cb 0x55b813706d30 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-23T10:34:50.021 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:34:50.020+0000 7feac562e640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55b813824800 0x55b813821870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T10:34:50.021 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:34:50.020+0000 7feac4e2d640 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x55b813819620 0x55b813824040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T10:34:50.021 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:34:50.020+0000 7feac562e640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55b813824800 0x55b813821870 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.114:36860/0 (socket says 172.21.15.114:36860) 2024-04-23T10:34:50.021 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:34:50.020+0000 7feac562e640 1 -- 172.21.15.114:0/2817210111 learned_addr learned my addr 172.21.15.114:0/2817210111 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-04-23T10:34:50.021 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:34:50.020+0000 7feac562e640 1 -- 172.21.15.114:0/2817210111 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x55b813819620 msgr2=0x55b813824040 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T10:34:50.021 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:34:50.020+0000 7feac562e640 1 --2- 172.21.15.114:0/2817210111 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x55b813819620 0x55b813824040 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:34:50.021 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:34:50.020+0000 7feac562e640 1 -- 172.21.15.114:0/2817210111 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x55b813819f40 msgr2=0x55b813821330 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T10:34:50.021 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:34:50.020+0000 7feac562e640 1 --2- 172.21.15.114:0/2817210111 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x55b813819f40 0x55b813821330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:34:50.021 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:34:50.020+0000 7feac562e640 1 -- 172.21.15.114:0/2817210111 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55b81355d620 con 0x55b813824800 2024-04-23T10:34:50.022 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:34:50.021+0000 7feac562e640 1 --2- 172.21.15.114:0/2817210111 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55b813824800 0x55b813821870 secure :-1 s=READY pgs=795 cs=0 l=1 rev1=1 crypto rx=0x7feab002f730 tx=0x7feab00095c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=49d9301bfc678e71 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T10:34:50.022 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:34:50.021+0000 7fea9ffff640 10 client.0 ms_handle_connect on v2:172.21.15.114:3300/0 2024-04-23T10:34:50.022 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:34:50.021+0000 7fea9ffff640 1 -- 172.21.15.114:0/2817210111 <== mon.0 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7feab0043050 con 0x55b813824800 2024-04-23T10:34:50.022 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:34:50.022+0000 7fea9ffff640 1 -- 172.21.15.114:0/2817210111 <== mon.0 v2:172.21.15.114:3300/0 2 ==== osd_map(98..98 src has 1..98) ==== 8629+0+0 (secure 0 0 0) 0x7feab0004240 con 0x55b813824800 2024-04-23T10:34:50.022 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:34:50.022+0000 7feac9dab740 1 -- 172.21.15.114:0/2817210111 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7feab80054d0 con 0x55b813824800 2024-04-23T10:34:50.022 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:34:50.022+0000 7fea9ffff640 1 -- 172.21.15.114:0/2817210111 <== mon.0 v2:172.21.15.114:3300/0 3 ==== mon_get_version_reply(handle=1 version=31) ==== 24+0+0 (secure 0 0 0) 0x7feab0034040 con 0x55b813824800 2024-04-23T10:34:50.022 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:34:50.022+0000 7feac9dab740 10 client.16620 fetch_fsmap learned FSMap version 31 2024-04-23T10:34:50.022 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:34:50.022+0000 7feac9dab740 1 -- 172.21.15.114:0/2817210111 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({config=0+,fsmap=31,monmap=4+}) -- 0x55b813833310 con 0x55b813824800 2024-04-23T10:34:50.023 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:34:50.022+0000 7fea9ffff640 1 -- 172.21.15.114:0/2817210111 <== mon.0 v2:172.21.15.114:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7feab80054d0 con 0x55b813824800 2024-04-23T10:34:50.023 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:34:50.023+0000 7fea9ffff640 1 -- 172.21.15.114:0/2817210111 <== mon.0 v2:172.21.15.114:3300/0 5 ==== fsmap(e 31) ==== 8342+0+0 (secure 0 0 0) 0x7feab0030ad0 con 0x55b813824800 2024-04-23T10:34:50.023 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:34:50.023+0000 7feac9dab740 10 client.16620 fetch_fsmap finished waiting for FSMap version 31 2024-04-23T10:34:50.023 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:34:50.023+0000 7feac9dab740 10 client.16620 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-04-23T10:34:50.024 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:34:50.023+0000 7feac9dab740 20 client.16620 populate_metadata read hostname 'smithi114' 2024-04-23T10:34:50.024 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:34:50.023+0000 7feac9dab740 1 --2- 172.21.15.114:0/2817210111 >> [v2:172.21.15.119:6838/937297340,v1:172.21.15.119:6839/937297340] conn(0x55b813834820 0x55b813836c70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T10:34:50.024 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:34:50.023+0000 7feac9dab740 4 client.16620 mds_command: new command op to 24545 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-23T10:34:50.024 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:34:50.023+0000 7feac9dab740 1 -- 172.21.15.114:0/2817210111 --> [v2:172.21.15.119:6838/937297340,v1:172.21.15.119:6839/937297340] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55b81378c390 con 0x55b813834820 2024-04-23T10:34:50.024 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:34:50.023+0000 7feac4e2d640 1 --2- 172.21.15.114:0/2817210111 >> [v2:172.21.15.119:6838/937297340,v1:172.21.15.119:6839/937297340] conn(0x55b813834820 0x55b813836c70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T10:34:50.024 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:34:50.024+0000 7feac4e2d640 1 --2- 172.21.15.114:0/2817210111 >> [v2:172.21.15.119:6838/937297340,v1:172.21.15.119:6839/937297340] conn(0x55b813834820 0x55b813836c70 crc :-1 s=READY pgs=134 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=5733ec37c6e4503c server_cookie=2ffae8f943bfb923 in_seq=0 out_seq=0 2024-04-23T10:34:50.024 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:34:50.024+0000 7fea9ffff640 10 client.16620 ms_handle_connect on v2:172.21.15.119:6838/937297340 2024-04-23T10:34:50.025 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:34:50.025+0000 7fea9ffff640 1 -- 172.21.15.114:0/2817210111 <== mds.0 v2:172.21.15.119:6838/937297340 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x55b81378c390 con 0x55b813834820 2024-04-23T10:34:50.025 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:34:50.025+0000 7fea9ffff640 10 client.16620 handle_command_reply: tid=0 2024-04-23T10:34:50.025 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:34:50.025+0000 7feac9dab740 1 client.16620 shutdown 2024-04-23T10:34:50.025 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:34:50.025+0000 7feac9dab740 1 -- 172.21.15.114:0/2817210111 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55b813824800 msgr2=0x55b813821870 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T10:34:50.025 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:34:50.025+0000 7feac9dab740 1 --2- 172.21.15.114:0/2817210111 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55b813824800 0x55b813821870 secure :-1 s=READY pgs=795 cs=0 l=1 rev1=1 crypto rx=0x7feab002f730 tx=0x7feab00095c0 comp rx=0 tx=0).stop 2024-04-23T10:34:50.025 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:34:50.025+0000 7feac9dab740 1 -- 172.21.15.114:0/2817210111 shutdown_connections 2024-04-23T10:34:50.025 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:34:50.025+0000 7feac9dab740 1 --2- 172.21.15.114:0/2817210111 >> [v2:172.21.15.119:6838/937297340,v1:172.21.15.119:6839/937297340] conn(0x55b813834820 0x55b813836c70 crc :-1 s=READY pgs=134 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:34:50.026 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:34:50.025+0000 7feac9dab740 1 --2- 172.21.15.114:0/2817210111 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55b813824800 0x55b813821870 unknown :-1 s=CLOSED pgs=795 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:34:50.026 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:34:50.025+0000 7feac9dab740 1 --2- 172.21.15.114:0/2817210111 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x55b813819f40 0x55b813821330 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:34:50.026 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:34:50.025+0000 7fea9ffff640 0 client.16620 ms_handle_reset on v2:172.21.15.119:6838/937297340 2024-04-23T10:34:50.026 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:34:50.025+0000 7feac9dab740 1 --2- 172.21.15.114:0/2817210111 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x55b813819620 0x55b813824040 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:34:50.026 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:34:50.025+0000 7feac9dab740 1 -- 172.21.15.114:0/2817210111 >> 172.21.15.114:0/2817210111 conn(0x55b813815b30 msgr2=0x55b81381b860 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T10:34:50.026 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:34:50.026+0000 7feac9dab740 1 -- 172.21.15.114:0/2817210111 shutdown_connections 2024-04-23T10:34:50.026 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:34:50.026+0000 7feac9dab740 1 -- 172.21.15.114:0/2817210111 wait complete. 2024-04-23T10:34:50.026 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:34:50.026+0000 7feac9dab740 20 client.16620 trim_cache size 0 max 16384 2024-04-23T10:34:50.041 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:34:50.041+0000 7feac9dab740 1 Processor -- start 2024-04-23T10:34:50.041 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:34:50.041+0000 7feac9dab740 1 -- start start 2024-04-23T10:34:50.041 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:34:50.041+0000 7feac9dab740 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x55b813815b30 0x55b81381b6e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T10:34:50.041 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:34:50.041+0000 7feac9dab740 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x55b813819620 0x55b813823b70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T10:34:50.041 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:34:50.041+0000 7feac9dab740 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55b813819f40 0x55b81383c540 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T10:34:50.041 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:34:50.041+0000 7feac9dab740 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x7feaac011300 con 0x55b813819f40 2024-04-23T10:34:50.041 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:34:50.041+0000 7feac9dab740 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 -- 0x7feaac00eba0 con 0x55b813815b30 2024-04-23T10:34:50.042 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:34:50.041+0000 7feac9dab740 1 -- --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_getmap magic: 0 -- 0x7feaac00ca10 con 0x55b813819620 2024-04-23T10:34:50.042 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:34:50.042+0000 7feac562e640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55b813819f40 0x55b81383c540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T10:34:50.042 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:34:50.042+0000 7feac4e2d640 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x55b813815b30 0x55b81381b6e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T10:34:50.042 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:34:50.042+0000 7feac562e640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55b813819f40 0x55b81383c540 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.114:36872/0 (socket says 172.21.15.114:36872) 2024-04-23T10:34:50.042 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:34:50.042+0000 7feac562e640 1 -- 172.21.15.114:0/963788284 learned_addr learned my addr 172.21.15.114:0/963788284 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-04-23T10:34:50.042 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:34:50.042+0000 7feabffff640 1 --2- 172.21.15.114:0/963788284 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x55b813819620 0x55b813823b70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T10:34:50.042 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:34:50.042+0000 7feac4e2d640 1 -- 172.21.15.114:0/963788284 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x55b813819620 msgr2=0x55b813823b70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T10:34:50.042 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:34:50.042+0000 7feac4e2d640 1 --2- 172.21.15.114:0/963788284 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x55b813819620 0x55b813823b70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:34:50.042 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:34:50.042+0000 7feac4e2d640 1 -- 172.21.15.114:0/963788284 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55b813819f40 msgr2=0x55b81383c540 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T10:34:50.042 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:34:50.042+0000 7feac4e2d640 1 --2- 172.21.15.114:0/963788284 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55b813819f40 0x55b81383c540 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:34:50.042 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:34:50.042+0000 7feac4e2d640 1 -- 172.21.15.114:0/963788284 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55b813811500 con 0x55b813815b30 2024-04-23T10:34:50.042 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:34:50.042+0000 7feac4e2d640 1 --2- 172.21.15.114:0/963788284 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x55b813815b30 0x55b81381b6e0 secure :-1 s=READY pgs=86 cs=0 l=1 rev1=1 crypto rx=0x7feab40ce120 tx=0x7feab4009700 comp rx=0 tx=0).ready entity=mon.2 client_cookie=9fe757f19ae2b185 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T10:34:50.043 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:34:50.042+0000 7feaa67fc640 1 -- 172.21.15.114:0/963788284 <== mon.2 v2:172.21.15.119:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7feab40a55e0 con 0x55b813815b30 2024-04-23T10:34:50.043 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:34:50.042+0000 7feaa67fc640 1 -- 172.21.15.114:0/963788284 <== mon.2 v2:172.21.15.119:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7feab409ad20 con 0x55b813815b30 2024-04-23T10:34:50.043 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:34:50.042+0000 7feaa67fc640 1 -- 172.21.15.114:0/963788284 <== mon.2 v2:172.21.15.119:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7feab409c0d0 con 0x55b813815b30 2024-04-23T10:34:50.046 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:34:50.043+0000 7feac9dab740 1 -- 172.21.15.114:0/963788284 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x55b813815b30 msgr2=0x55b81381b6e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T10:34:50.046 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:34:50.043+0000 7feac9dab740 1 --2- 172.21.15.114:0/963788284 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x55b813815b30 0x55b81381b6e0 secure :-1 s=READY pgs=86 cs=0 l=1 rev1=1 crypto rx=0x7feab40ce120 tx=0x7feab4009700 comp rx=0 tx=0).stop 2024-04-23T10:34:50.046 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:34:50.043+0000 7feac9dab740 1 -- 172.21.15.114:0/963788284 shutdown_connections 2024-04-23T10:34:50.046 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:34:50.043+0000 7feac9dab740 1 --2- 172.21.15.114:0/963788284 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55b813819f40 0x55b81383c540 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:34:50.046 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:34:50.043+0000 7feac9dab740 1 --2- 172.21.15.114:0/963788284 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x55b813819620 0x55b813823b70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:34:50.046 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:34:50.043+0000 7feac9dab740 1 --2- 172.21.15.114:0/963788284 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x55b813815b30 0x55b81381b6e0 unknown :-1 s=CLOSED pgs=86 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:34:50.046 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:34:50.043+0000 7feac9dab740 1 -- 172.21.15.114:0/963788284 >> 172.21.15.114:0/963788284 conn(0x55b8138219d0 msgr2=0x55b81381afd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T10:34:50.046 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:34:50.043+0000 7feac9dab740 1 -- 172.21.15.114:0/963788284 shutdown_connections 2024-04-23T10:34:50.047 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:34:50.043+0000 7feac9dab740 1 -- 172.21.15.114:0/963788284 wait complete. 2024-04-23T10:34:50.047 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:34:50.043+0000 7feac9dab740 1 -- start start 2024-04-23T10:34:50.047 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:34:50.044+0000 7feac9dab740 1 Processor -- start 2024-04-23T10:34:50.047 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:34:50.044+0000 7feac9dab740 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55b813815b30 0x55b81382c880 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T10:34:50.047 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:34:50.044+0000 7feac9dab740 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x55b813819620 0x55b81382cdc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T10:34:50.047 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:34:50.044+0000 7feac9dab740 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x55b813819f40 0x55b813829850 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T10:34:50.047 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:34:50.044+0000 7feac9dab740 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x7feaac00d060 con 0x55b813815b30 2024-04-23T10:34:50.047 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:34:50.044+0000 7feac9dab740 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 -- 0x7feaac00e4d0 con 0x55b813819620 2024-04-23T10:34:50.047 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:34:50.044+0000 7feac9dab740 1 -- --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_getmap magic: 0 -- 0x7feaac00d730 con 0x55b813819f40 2024-04-23T10:34:50.047 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:34:50.044+0000 7feac9dab740 10 client.0 _ll_register_callbacks cb 0x7feab0003c80 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-23T10:34:50.047 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:34:50.044+0000 7feac4e2d640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55b813815b30 0x55b81382c880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T10:34:50.047 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:34:50.044+0000 7feabffff640 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x55b813819620 0x55b81382cdc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T10:34:50.047 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:34:50.044+0000 7feac4e2d640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55b813815b30 0x55b81382c880 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.114:36876/0 (socket says 172.21.15.114:36876) 2024-04-23T10:34:50.047 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:34:50.044+0000 7feac4e2d640 1 -- 172.21.15.114:0/2847533808 learned_addr learned my addr 172.21.15.114:0/2847533808 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-04-23T10:34:50.048 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:34:50.044+0000 7feac562e640 1 --2- 172.21.15.114:0/2847533808 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x55b813819f40 0x55b813829850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T10:34:50.048 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:34:50.044+0000 7feac4e2d640 1 -- 172.21.15.114:0/2847533808 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x55b813819f40 msgr2=0x55b813829850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T10:34:50.048 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:34:50.044+0000 7feac4e2d640 1 --2- 172.21.15.114:0/2847533808 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x55b813819f40 0x55b813829850 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:34:50.048 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:34:50.044+0000 7feac4e2d640 1 -- 172.21.15.114:0/2847533808 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x55b813819620 msgr2=0x55b81382cdc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T10:34:50.048 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:34:50.044+0000 7feac4e2d640 1 --2- 172.21.15.114:0/2847533808 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x55b813819620 0x55b81382cdc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:34:50.048 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:34:50.044+0000 7feac4e2d640 1 -- 172.21.15.114:0/2847533808 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55b813829e90 con 0x55b813815b30 2024-04-23T10:34:50.048 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:34:50.044+0000 7feabffff640 1 --2- 172.21.15.114:0/2847533808 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x55b813819620 0x55b81382cdc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-23T10:34:50.048 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:34:50.045+0000 7feac4e2d640 1 --2- 172.21.15.114:0/2847533808 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55b813815b30 0x55b81382c880 secure :-1 s=READY pgs=796 cs=0 l=1 rev1=1 crypto rx=0x7feab400c6a0 tx=0x7feab400c840 comp rx=0 tx=0).ready entity=mon.0 client_cookie=49663b5807ec8153 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T10:34:50.048 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:34:50.045+0000 7fea9ffff640 10 client.0 ms_handle_connect on v2:172.21.15.114:3300/0 2024-04-23T10:34:50.048 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:34:50.045+0000 7fea9ffff640 1 -- 172.21.15.114:0/2847533808 <== mon.0 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7feab40a55e0 con 0x55b813815b30 2024-04-23T10:34:50.048 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:34:50.045+0000 7fea9ffff640 1 -- 172.21.15.114:0/2847533808 <== mon.0 v2:172.21.15.114:3300/0 2 ==== osd_map(98..98 src has 1..98) ==== 8629+0+0 (secure 0 0 0) 0x7feab4099090 con 0x55b813815b30 2024-04-23T10:34:50.048 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:34:50.045+0000 7feac9dab740 1 -- 172.21.15.114:0/2847533808 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7feab409ad20 con 0x55b813815b30 2024-04-23T10:34:50.048 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:34:50.045+0000 7fea9ffff640 1 -- 172.21.15.114:0/2847533808 <== mon.0 v2:172.21.15.114:3300/0 3 ==== mon_get_version_reply(handle=1 version=31) ==== 24+0+0 (secure 0 0 0) 0x7feab40bbb30 con 0x55b813815b30 2024-04-23T10:34:50.048 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:34:50.045+0000 7feac9dab740 10 client.16626 fetch_fsmap learned FSMap version 31 2024-04-23T10:34:50.048 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:34:50.045+0000 7feac9dab740 1 -- 172.21.15.114:0/2847533808 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({config=0+,fsmap=31,monmap=4+}) -- 0x55b81389b710 con 0x55b813815b30 2024-04-23T10:34:50.048 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:34:50.045+0000 7fea9ffff640 1 -- 172.21.15.114:0/2847533808 <== mon.0 v2:172.21.15.114:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7feab409ad20 con 0x55b813815b30 2024-04-23T10:34:50.049 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:34:50.045+0000 7fea9ffff640 1 -- 172.21.15.114:0/2847533808 <== mon.0 v2:172.21.15.114:3300/0 5 ==== fsmap(e 31) ==== 8342+0+0 (secure 0 0 0) 0x7feab4098770 con 0x55b813815b30 2024-04-23T10:34:50.049 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:34:50.045+0000 7feac9dab740 10 client.16626 fetch_fsmap finished waiting for FSMap version 31 2024-04-23T10:34:50.049 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:34:50.045+0000 7feac9dab740 10 client.16626 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-04-23T10:34:50.049 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:34:50.045+0000 7feac9dab740 20 client.16626 populate_metadata read hostname 'smithi114' 2024-04-23T10:34:50.049 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:34:50.045+0000 7feac9dab740 1 --2- 172.21.15.114:0/2847533808 >> [v2:172.21.15.119:6838/937297340,v1:172.21.15.119:6839/937297340] conn(0x55b813824800 0x55b81389dea0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T10:34:50.049 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:34:50.045+0000 7feac9dab740 4 client.16626 mds_command: new command op to 24545 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-04-23T10:34:50.049 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:34:50.045+0000 7feac9dab740 1 -- 172.21.15.114:0/2847533808 --> [v2:172.21.15.119:6838/937297340,v1:172.21.15.119:6839/937297340] -- command(tid 0: {"prefix": "flush journal"}) -- 0x55b813786200 con 0x55b813824800 2024-04-23T10:34:50.049 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:34:50.045+0000 7feabffff640 1 --2- 172.21.15.114:0/2847533808 >> [v2:172.21.15.119:6838/937297340,v1:172.21.15.119:6839/937297340] conn(0x55b813824800 0x55b81389dea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T10:34:50.049 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:34:50.046+0000 7fea9ffff640 10 client.16626 ms_handle_connect on v2:172.21.15.119:6838/937297340 2024-04-23T10:34:50.049 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:34:50.046+0000 7feabffff640 1 --2- 172.21.15.114:0/2847533808 >> [v2:172.21.15.119:6838/937297340,v1:172.21.15.119:6839/937297340] conn(0x55b813824800 0x55b81389dea0 crc :-1 s=READY pgs=135 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=3220bfdc3fa601f5 server_cookie=f577cf4943b855ac in_seq=0 out_seq=0 2024-04-23T10:34:50.112 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:34:50.113+0000 7fea9ffff640 1 -- 172.21.15.114:0/2847533808 <== mds.0 v2:172.21.15.119:6838/937297340 1 ==== command_reply(tid 0: 0 ) ==== 8+0+44 (crc 0 0 0) 0x55b813786200 con 0x55b813824800 2024-04-23T10:34:50.112 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:34:50.113+0000 7fea9ffff640 10 client.16626 handle_command_reply: tid=0 2024-04-23T10:34:50.113 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:34:50.113+0000 7feac9dab740 1 client.16626 shutdown 2024-04-23T10:34:50.113 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:34:50.113+0000 7feac9dab740 1 -- 172.21.15.114:0/2847533808 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55b813815b30 msgr2=0x55b81382c880 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T10:34:50.113 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:34:50.113+0000 7feac9dab740 1 --2- 172.21.15.114:0/2847533808 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55b813815b30 0x55b81382c880 secure :-1 s=READY pgs=796 cs=0 l=1 rev1=1 crypto rx=0x7feab400c6a0 tx=0x7feab400c840 comp rx=0 tx=0).stop 2024-04-23T10:34:50.113 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:34:50.113+0000 7feac9dab740 1 -- 172.21.15.114:0/2847533808 shutdown_connections 2024-04-23T10:34:50.113 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:34:50.113+0000 7feac9dab740 1 --2- 172.21.15.114:0/2847533808 >> [v2:172.21.15.119:6838/937297340,v1:172.21.15.119:6839/937297340] conn(0x55b813824800 0x55b81389dea0 crc :-1 s=READY pgs=135 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:34:50.113 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:34:50.113+0000 7feac9dab740 1 --2- 172.21.15.114:0/2847533808 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x55b813819f40 0x55b813829850 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:34:50.113 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:34:50.113+0000 7feac9dab740 1 --2- 172.21.15.114:0/2847533808 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x55b813819620 0x55b81382cdc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:34:50.113 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:34:50.113+0000 7feac9dab740 1 --2- 172.21.15.114:0/2847533808 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55b813815b30 0x55b81382c880 unknown :-1 s=CLOSED pgs=796 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:34:50.113 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:34:50.113+0000 7fea9ffff640 0 client.16626 ms_handle_reset on v2:172.21.15.119:6838/937297340 2024-04-23T10:34:50.113 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:34:50.113+0000 7feac9dab740 1 -- 172.21.15.114:0/2847533808 >> 172.21.15.114:0/2847533808 conn(0x55b8138219d0 msgr2=0x55b81381b860 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T10:34:50.113 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:34:50.113+0000 7feac9dab740 1 -- 172.21.15.114:0/2847533808 shutdown_connections 2024-04-23T10:34:50.113 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:34:50.113+0000 7feac9dab740 1 -- 172.21.15.114:0/2847533808 wait complete. 2024-04-23T10:34:50.113 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:34:50.113+0000 7feac9dab740 20 client.16626 trim_cache size 0 max 16384 2024-04-23T10:34:50.114 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stdout:{ 2024-04-23T10:34:50.114 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stdout: "message": "", 2024-04-23T10:34:50.114 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stdout: "return_code": 0 2024-04-23T10:34:50.114 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stdout:} 2024-04-23T10:34:50.115 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:34:50.114+0000 7feac7091640 1 -- 172.21.15.114:0/4088759243 >> [v2:172.21.15.114:6800/1395693466,v1:172.21.15.114:6801/1395693466] conn(0x7feaa80a3e40 msgr2=0x7feaa80a6300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T10:34:50.115 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:34:50.114+0000 7feac7091640 1 --2- 172.21.15.114:0/4088759243 >> [v2:172.21.15.114:6800/1395693466,v1:172.21.15.114:6801/1395693466] conn(0x7feaa80a3e40 0x7feaa80a6300 secure :-1 s=READY pgs=121 cs=0 l=1 rev1=1 crypto rx=0x7feab002ae70 tx=0x7feab002a950 comp rx=0 tx=0).stop 2024-04-23T10:34:50.115 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:34:50.114+0000 7feac7091640 1 -- 172.21.15.114:0/4088759243 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7feac0100f90 msgr2=0x7feac01a4880 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T10:34:50.115 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:34:50.114+0000 7feac7091640 1 --2- 172.21.15.114:0/4088759243 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7feac0100f90 0x7feac01a4880 secure :-1 s=READY pgs=793 cs=0 l=1 rev1=1 crypto rx=0x7feab400e9f0 tx=0x7feab400eeb0 comp rx=0 tx=0).stop 2024-04-23T10:34:50.115 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:34:50.114+0000 7feac7091640 1 -- 172.21.15.114:0/4088759243 shutdown_connections 2024-04-23T10:34:50.115 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:34:50.114+0000 7feac7091640 1 --2- 172.21.15.114:0/4088759243 >> [v2:172.21.15.114:6800/1395693466,v1:172.21.15.114:6801/1395693466] conn(0x7feaa80a3e40 0x7feaa80a6300 unknown :-1 s=CLOSED pgs=121 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:34:50.115 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:34:50.114+0000 7feac7091640 1 --2- 172.21.15.114:0/4088759243 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7feac0106390 0x7feac019ea30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:34:50.115 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:34:50.114+0000 7feac7091640 1 --2- 172.21.15.114:0/4088759243 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7feac01018b0 0x7feac01a4dc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:34:50.115 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:34:50.114+0000 7feac7091640 1 --2- 172.21.15.114:0/4088759243 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7feac0100f90 0x7feac01a4880 unknown :-1 s=CLOSED pgs=793 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:34:50.115 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:34:50.114+0000 7feac7091640 1 -- 172.21.15.114:0/4088759243 >> 172.21.15.114:0/4088759243 conn(0x7feac0078060 msgr2=0x7feac0103d40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T10:34:50.115 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:34:50.114+0000 7feac7091640 1 -- 172.21.15.114:0/4088759243 shutdown_connections 2024-04-23T10:34:50.115 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:34:50.114+0000 7feac7091640 1 -- 172.21.15.114:0/4088759243 wait complete. 2024-04-23T10:34:50.930 INFO:tasks.workunit.client.0.smithi114.stdout: 1 585511 7.40 MB/sec execute 299 sec latency 58.746 ms 2024-04-23T10:34:51.742 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 23 10:34:51 smithi119 ceph-mon[32608]: pgmap v256: 97 pgs: 97 active+clean; 189 MiB data, 2.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.3 MiB/s rd, 2.9 MiB/s wr, 261 op/s 2024-04-23T10:34:51.753 INFO:journalctl@ceph.mon.a.smithi114.stdout:Apr 23 10:34:51 smithi114 ceph-mon[29255]: pgmap v256: 97 pgs: 97 active+clean; 189 MiB data, 2.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.3 MiB/s rd, 2.9 MiB/s wr, 261 op/s 2024-04-23T10:34:51.827 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 23 10:34:51 smithi204 ceph-mon[32649]: pgmap v256: 97 pgs: 97 active+clean; 189 MiB data, 2.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.3 MiB/s rd, 2.9 MiB/s wr, 261 op/s 2024-04-23T10:34:51.930 INFO:tasks.workunit.client.0.smithi114.stdout: 1 586974 7.40 MB/sec execute 300 sec latency 28.659 ms 2024-04-23T10:34:52.741 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 23 10:34:52 smithi119 ceph-mon[32608]: from='mgr.15291 172.21.15.114:0/1095789187' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-23T10:34:52.753 INFO:journalctl@ceph.mon.a.smithi114.stdout:Apr 23 10:34:52 smithi114 ceph-mon[29255]: from='mgr.15291 172.21.15.114:0/1095789187' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-23T10:34:52.827 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 23 10:34:52 smithi204 ceph-mon[32649]: from='mgr.15291 172.21.15.114:0/1095789187' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-23T10:34:52.930 INFO:tasks.workunit.client.0.smithi114.stdout: 1 589582 7.43 MB/sec execute 301 sec latency 20.654 ms 2024-04-23T10:34:53.741 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 23 10:34:53 smithi119 ceph-mon[32608]: pgmap v257: 97 pgs: 97 active+clean; 186 MiB data, 2.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 MiB/s rd, 4.5 MiB/s wr, 411 op/s 2024-04-23T10:34:53.742 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 23 10:34:53 smithi119 ceph-mon[32608]: from='mgr.15291 172.21.15.114:0/1095789187' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-23T10:34:53.742 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 23 10:34:53 smithi119 ceph-mon[32608]: from='mgr.15291 172.21.15.114:0/1095789187' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-23T10:34:53.742 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 23 10:34:53 smithi119 ceph-mon[32608]: from='mgr.15291 172.21.15.114:0/1095789187' entity='mgr.x' 2024-04-23T10:34:53.754 INFO:journalctl@ceph.mon.a.smithi114.stdout:Apr 23 10:34:53 smithi114 ceph-mon[29255]: pgmap v257: 97 pgs: 97 active+clean; 186 MiB data, 2.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 MiB/s rd, 4.5 MiB/s wr, 411 op/s 2024-04-23T10:34:53.754 INFO:journalctl@ceph.mon.a.smithi114.stdout:Apr 23 10:34:53 smithi114 ceph-mon[29255]: from='mgr.15291 172.21.15.114:0/1095789187' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-23T10:34:53.754 INFO:journalctl@ceph.mon.a.smithi114.stdout:Apr 23 10:34:53 smithi114 ceph-mon[29255]: from='mgr.15291 172.21.15.114:0/1095789187' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-23T10:34:53.754 INFO:journalctl@ceph.mon.a.smithi114.stdout:Apr 23 10:34:53 smithi114 ceph-mon[29255]: from='mgr.15291 172.21.15.114:0/1095789187' entity='mgr.x' 2024-04-23T10:34:53.827 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 23 10:34:53 smithi204 ceph-mon[32649]: pgmap v257: 97 pgs: 97 active+clean; 186 MiB data, 2.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 MiB/s rd, 4.5 MiB/s wr, 411 op/s 2024-04-23T10:34:53.828 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 23 10:34:53 smithi204 ceph-mon[32649]: from='mgr.15291 172.21.15.114:0/1095789187' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-23T10:34:53.828 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 23 10:34:53 smithi204 ceph-mon[32649]: from='mgr.15291 172.21.15.114:0/1095789187' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-23T10:34:53.828 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 23 10:34:53 smithi204 ceph-mon[32649]: from='mgr.15291 172.21.15.114:0/1095789187' entity='mgr.x' 2024-04-23T10:34:53.930 INFO:tasks.workunit.client.0.smithi114.stdout: 1 590853 7.42 MB/sec execute 302 sec latency 51.324 ms 2024-04-23T10:34:54.930 INFO:tasks.workunit.client.0.smithi114.stdout: 1 593413 7.45 MB/sec execute 303 sec latency 15.690 ms 2024-04-23T10:34:55.742 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 23 10:34:55 smithi119 ceph-mon[32608]: pgmap v258: 97 pgs: 97 active+clean; 186 MiB data, 2.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.3 MiB/s rd, 3.1 MiB/s wr, 257 op/s 2024-04-23T10:34:55.753 INFO:journalctl@ceph.mon.a.smithi114.stdout:Apr 23 10:34:55 smithi114 ceph-mon[29255]: pgmap v258: 97 pgs: 97 active+clean; 186 MiB data, 2.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.3 MiB/s rd, 3.1 MiB/s wr, 257 op/s 2024-04-23T10:34:55.827 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 23 10:34:55 smithi204 ceph-mon[32649]: pgmap v258: 97 pgs: 97 active+clean; 186 MiB data, 2.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.3 MiB/s rd, 3.1 MiB/s wr, 257 op/s 2024-04-23T10:34:55.930 INFO:tasks.workunit.client.0.smithi114.stdout: 1 594813 7.44 MB/sec execute 304 sec latency 69.461 ms 2024-04-23T10:34:56.930 INFO:tasks.workunit.client.0.smithi114.stdout: 1 597076 7.47 MB/sec execute 305 sec latency 26.782 ms 2024-04-23T10:34:57.742 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 23 10:34:57 smithi119 ceph-mon[32608]: pgmap v259: 97 pgs: 97 active+clean; 191 MiB data, 2.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 MiB/s rd, 4.7 MiB/s wr, 367 op/s 2024-04-23T10:34:57.753 INFO:journalctl@ceph.mon.a.smithi114.stdout:Apr 23 10:34:57 smithi114 ceph-mon[29255]: pgmap v259: 97 pgs: 97 active+clean; 191 MiB data, 2.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 MiB/s rd, 4.7 MiB/s wr, 367 op/s 2024-04-23T10:34:57.827 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 23 10:34:57 smithi204 ceph-mon[32649]: pgmap v259: 97 pgs: 97 active+clean; 191 MiB data, 2.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 MiB/s rd, 4.7 MiB/s wr, 367 op/s 2024-04-23T10:34:57.930 INFO:tasks.workunit.client.0.smithi114.stdout: 1 599040 7.46 MB/sec execute 306 sec latency 17.695 ms 2024-04-23T10:34:58.930 INFO:tasks.workunit.client.0.smithi114.stdout: 1 600848 7.48 MB/sec execute 307 sec latency 12.571 ms 2024-04-23T10:34:59.742 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 23 10:34:59 smithi119 ceph-mon[32608]: pgmap v260: 97 pgs: 97 active+clean; 191 MiB data, 2.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.3 MiB/s rd, 3.1 MiB/s wr, 260 op/s 2024-04-23T10:34:59.753 INFO:journalctl@ceph.mon.a.smithi114.stdout:Apr 23 10:34:59 smithi114 ceph-mon[29255]: pgmap v260: 97 pgs: 97 active+clean; 191 MiB data, 2.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.3 MiB/s rd, 3.1 MiB/s wr, 260 op/s 2024-04-23T10:34:59.827 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 23 10:34:59 smithi204 ceph-mon[32649]: pgmap v260: 97 pgs: 97 active+clean; 191 MiB data, 2.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.3 MiB/s rd, 3.1 MiB/s wr, 260 op/s 2024-04-23T10:34:59.930 INFO:tasks.workunit.client.0.smithi114.stdout: 1 603263 7.50 MB/sec execute 308 sec latency 15.658 ms 2024-04-23T10:35:00.930 INFO:tasks.workunit.client.0.smithi114.stdout: 1 604594 7.49 MB/sec execute 309 sec latency 75.264 ms 2024-04-23T10:35:01.742 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 23 10:35:01 smithi119 ceph-mon[32608]: pgmap v261: 97 pgs: 97 active+clean; 191 MiB data, 2.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.3 MiB/s rd, 3.1 MiB/s wr, 260 op/s 2024-04-23T10:35:01.754 INFO:journalctl@ceph.mon.a.smithi114.stdout:Apr 23 10:35:01 smithi114 ceph-mon[29255]: pgmap v261: 97 pgs: 97 active+clean; 191 MiB data, 2.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.3 MiB/s rd, 3.1 MiB/s wr, 260 op/s 2024-04-23T10:35:01.827 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 23 10:35:01 smithi204 ceph-mon[32649]: pgmap v261: 97 pgs: 97 active+clean; 191 MiB data, 2.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.3 MiB/s rd, 3.1 MiB/s wr, 260 op/s 2024-04-23T10:35:01.930 INFO:tasks.workunit.client.0.smithi114.stdout: 1 606005 7.49 MB/sec execute 310 sec latency 35.260 ms 2024-04-23T10:35:02.930 INFO:tasks.workunit.client.0.smithi114.stdout: 1 607817 7.51 MB/sec execute 311 sec latency 11.794 ms 2024-04-23T10:35:03.204 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:35:03.202+0000 7f13cb1fe640 1 -- 172.21.15.114:0/126622672 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f13c41010f0 msgr2=0x7f13c41014d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T10:35:03.204 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:35:03.203+0000 7f13cb1fe640 1 --2- 172.21.15.114:0/126622672 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f13c41010f0 0x7f13c41014d0 secure :-1 s=READY pgs=797 cs=0 l=1 rev1=1 crypto rx=0x7f13b8009900 tx=0x7f13b802a410 comp rx=0 tx=0).stop 2024-04-23T10:35:03.205 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:35:03.203+0000 7f13cb1fe640 1 -- 172.21.15.114:0/126622672 shutdown_connections 2024-04-23T10:35:03.205 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:35:03.203+0000 7f13cb1fe640 1 --2- 172.21.15.114:0/126622672 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f13c4106170 0x7f13c410dd40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:35:03.205 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:35:03.203+0000 7f13cb1fe640 1 --2- 172.21.15.114:0/126622672 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f13c4101aa0 0x7f13c4105a90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:35:03.205 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:35:03.203+0000 7f13cb1fe640 1 --2- 172.21.15.114:0/126622672 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f13c41010f0 0x7f13c41014d0 unknown :-1 s=CLOSED pgs=797 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:35:03.205 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:35:03.203+0000 7f13cb1fe640 1 -- 172.21.15.114:0/126622672 >> 172.21.15.114:0/126622672 conn(0x7f13c4077fd0 msgr2=0x7f13c40783e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T10:35:03.205 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:35:03.203+0000 7f13cb1fe640 1 -- 172.21.15.114:0/126622672 shutdown_connections 2024-04-23T10:35:03.205 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:35:03.203+0000 7f13cb1fe640 1 -- 172.21.15.114:0/126622672 wait complete. 2024-04-23T10:35:03.205 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:35:03.203+0000 7f13cb1fe640 1 Processor -- start 2024-04-23T10:35:03.205 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:35:03.203+0000 7f13cb1fe640 1 -- start start 2024-04-23T10:35:03.205 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:35:03.203+0000 7f13cb1fe640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f13c41010f0 0x7f13c4102b30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T10:35:03.205 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:35:03.203+0000 7f13cb1fe640 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f13c4106170 0x7f13c4103070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T10:35:03.205 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:35:03.203+0000 7f13cb1fe640 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f13c41035b0 0x7f13c41a6840 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T10:35:03.205 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:35:03.203+0000 7f13cb1fe640 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x7f13c41110b0 con 0x7f13c41010f0 2024-04-23T10:35:03.206 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:35:03.203+0000 7f13cb1fe640 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 -- 0x7f13c4110f30 con 0x7f13c41035b0 2024-04-23T10:35:03.206 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:35:03.203+0000 7f13cb1fe640 1 -- --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_getmap magic: 0 -- 0x7f13c4111230 con 0x7f13c4106170 2024-04-23T10:35:03.206 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:35:03.204+0000 7f13c8f9a640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f13c41010f0 0x7f13c4102b30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T10:35:03.206 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:35:03.204+0000 7f13c8f9a640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f13c41010f0 0x7f13c4102b30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.114:53880/0 (socket says 172.21.15.114:53880) 2024-04-23T10:35:03.206 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:35:03.204+0000 7f13c8f9a640 1 -- 172.21.15.114:0/109278636 learned_addr learned my addr 172.21.15.114:0/109278636 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-04-23T10:35:03.206 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:35:03.204+0000 7f13c979b640 1 --2- 172.21.15.114:0/109278636 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f13c41035b0 0x7f13c41a6840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T10:35:03.206 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:35:03.204+0000 7f13c3fff640 1 --2- 172.21.15.114:0/109278636 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f13c4106170 0x7f13c4103070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T10:35:03.206 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:35:03.204+0000 7f13c979b640 1 -- 172.21.15.114:0/109278636 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f13c4106170 msgr2=0x7f13c4103070 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T10:35:03.206 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:35:03.204+0000 7f13c979b640 1 --2- 172.21.15.114:0/109278636 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f13c4106170 0x7f13c4103070 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:35:03.206 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:35:03.204+0000 7f13c979b640 1 -- 172.21.15.114:0/109278636 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f13c41010f0 msgr2=0x7f13c4102b30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T10:35:03.206 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:35:03.204+0000 7f13c979b640 1 --2- 172.21.15.114:0/109278636 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f13c41010f0 0x7f13c4102b30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:35:03.206 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:35:03.204+0000 7f13c979b640 1 -- 172.21.15.114:0/109278636 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f13c41a6f40 con 0x7f13c41035b0 2024-04-23T10:35:03.207 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:35:03.204+0000 7f13c8f9a640 1 --2- 172.21.15.114:0/109278636 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f13c41010f0 0x7f13c4102b30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-23T10:35:03.207 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:35:03.204+0000 7f13c979b640 1 --2- 172.21.15.114:0/109278636 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f13c41035b0 0x7f13c41a6840 secure :-1 s=READY pgs=87 cs=0 l=1 rev1=1 crypto rx=0x7f13bc00da10 tx=0x7f13bc00ded0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T10:35:03.207 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:35:03.204+0000 7f13c1ffb640 1 -- 172.21.15.114:0/109278636 <== mon.2 v2:172.21.15.119:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f13bc00bd70 con 0x7f13c41035b0 2024-04-23T10:35:03.207 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:35:03.204+0000 7f13c1ffb640 1 -- 172.21.15.114:0/109278636 <== mon.2 v2:172.21.15.119:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f13bc003e20 con 0x7f13c41035b0 2024-04-23T10:35:03.207 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:35:03.204+0000 7f13cb1fe640 1 -- 172.21.15.114:0/109278636 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f13c41a7230 con 0x7f13c41035b0 2024-04-23T10:35:03.207 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:35:03.204+0000 7f13c1ffb640 1 -- 172.21.15.114:0/109278636 <== mon.2 v2:172.21.15.119:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f13bc0106c0 con 0x7f13c41035b0 2024-04-23T10:35:03.207 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:35:03.205+0000 7f13cb1fe640 1 -- 172.21.15.114:0/109278636 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f13c41a7730 con 0x7f13c41035b0 2024-04-23T10:35:03.214 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:35:03.207+0000 7f13c1ffb640 1 -- 172.21.15.114:0/109278636 <== mon.2 v2:172.21.15.119:3300/0 4 ==== mgrmap(e 26) ==== 138778+0+0 (secure 0 0 0) 0x7f13bc004270 con 0x7f13c41035b0 2024-04-23T10:35:03.215 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:35:03.207+0000 7f13cdfee740 1 Processor -- start 2024-04-23T10:35:03.215 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:35:03.207+0000 7f13cdfee740 1 -- start start 2024-04-23T10:35:03.215 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:35:03.207+0000 7f13cdfee740 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x5603ace4a620 0x5603ace4aa00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T10:35:03.215 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:35:03.207+0000 7f13cdfee740 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x5603ace4af40 0x5603ace552c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T10:35:03.215 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:35:03.207+0000 7f13cdfee740 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x5603ace55800 0x5603ace57bf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T10:35:03.215 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:35:03.207+0000 7f13cdfee740 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x5603acdaf970 con 0x5603ace55800 2024-04-23T10:35:03.215 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:35:03.207+0000 7f13cdfee740 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 -- 0x5603acd4c4a0 con 0x5603ace4a620 2024-04-23T10:35:03.215 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:35:03.207+0000 7f13cdfee740 1 -- --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_getmap magic: 0 -- 0x5603acdab930 con 0x5603ace4af40 2024-04-23T10:35:03.215 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:35:03.207+0000 7f13c3fff640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x5603ace55800 0x5603ace57bf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T10:35:03.215 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:35:03.207+0000 7f13c8f9a640 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x5603ace4af40 0x5603ace552c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T10:35:03.216 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:35:03.207+0000 7f13c3fff640 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x5603ace4a620 0x5603ace4aa00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T10:35:03.216 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:35:03.208+0000 7f13c8f9a640 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x5603ace4af40 0x5603ace552c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.204:3300/0 says I am v2:172.21.15.114:41096/0 (socket says 172.21.15.114:41096) 2024-04-23T10:35:03.216 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:35:03.208+0000 7f13c8f9a640 1 -- 172.21.15.114:0/3635919563 learned_addr learned my addr 172.21.15.114:0/3635919563 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-04-23T10:35:03.216 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:35:03.208+0000 7f13c1ffb640 1 --2- 172.21.15.114:0/109278636 >> [v2:172.21.15.114:6800/1395693466,v1:172.21.15.114:6801/1395693466] conn(0x7f13b40a3df0 0x7f13b40a62b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T10:35:03.216 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:35:03.208+0000 7f13c3fff640 1 -- 172.21.15.114:0/3635919563 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x5603ace4af40 msgr2=0x5603ace552c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T10:35:03.216 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:35:03.208+0000 7f13c3fff640 1 --2- 172.21.15.114:0/3635919563 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x5603ace4af40 0x5603ace552c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:35:03.216 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:35:03.208+0000 7f13c979b640 1 --2- 172.21.15.114:0/109278636 >> [v2:172.21.15.114:6800/1395693466,v1:172.21.15.114:6801/1395693466] conn(0x7f13b40a3df0 0x7f13b40a62b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T10:35:03.216 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:35:03.208+0000 7f13c3fff640 1 -- 172.21.15.114:0/3635919563 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x5603ace55800 msgr2=0x5603ace57bf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T10:35:03.216 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:35:03.208+0000 7f13c3fff640 1 --2- 172.21.15.114:0/3635919563 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x5603ace55800 0x5603ace57bf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:35:03.217 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:35:03.208+0000 7f13c3fff640 1 -- 172.21.15.114:0/3635919563 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5603acac2210 con 0x5603ace4a620 2024-04-23T10:35:03.217 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:35:03.208+0000 7f13c8f9a640 1 --2- 172.21.15.114:0/3635919563 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x5603ace4af40 0x5603ace552c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-23T10:35:03.217 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:35:03.208+0000 7f13c1ffb640 1 -- 172.21.15.114:0/109278636 <== mon.2 v2:172.21.15.119:3300/0 5 ==== osd_map(98..98 src has 1..98) ==== 8629+0+0 (secure 0 0 0) 0x7f13bc0d0b70 con 0x7f13c41035b0 2024-04-23T10:35:03.217 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:35:03.208+0000 7f13c3fff640 1 --2- 172.21.15.114:0/3635919563 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x5603ace4a620 0x5603ace4aa00 secure :-1 s=READY pgs=88 cs=0 l=1 rev1=1 crypto rx=0x7f13c41045c0 tx=0x7f13b0005f90 comp rx=0 tx=0).ready entity=mon.2 client_cookie=7178e7e9aba142a2 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T10:35:03.217 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:35:03.208+0000 7f13c979b640 1 --2- 172.21.15.114:0/109278636 >> [v2:172.21.15.114:6800/1395693466,v1:172.21.15.114:6801/1395693466] conn(0x7f13b40a3df0 0x7f13b40a62b0 secure :-1 s=READY pgs=122 cs=0 l=1 rev1=1 crypto rx=0x7f13bc0ce850 tx=0x7f13bc06e4e0 comp rx=0 tx=0).ready entity=mgr.15291 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T10:35:03.217 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:35:03.208+0000 7f13aeffd640 1 -- 172.21.15.114:0/3635919563 <== mon.2 v2:172.21.15.119:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f13b0016070 con 0x5603ace4a620 2024-04-23T10:35:03.217 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:35:03.208+0000 7f13aeffd640 1 -- 172.21.15.114:0/3635919563 <== mon.2 v2:172.21.15.119:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f13b00047a0 con 0x5603ace4a620 2024-04-23T10:35:03.217 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:35:03.208+0000 7f13aeffd640 1 -- 172.21.15.114:0/3635919563 <== mon.2 v2:172.21.15.119:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f13b0019040 con 0x5603ace4a620 2024-04-23T10:35:03.217 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:35:03.209+0000 7f13cdfee740 1 -- 172.21.15.114:0/3635919563 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x5603ace4a620 msgr2=0x5603ace4aa00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T10:35:03.217 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:35:03.209+0000 7f13cdfee740 1 --2- 172.21.15.114:0/3635919563 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x5603ace4a620 0x5603ace4aa00 secure :-1 s=READY pgs=88 cs=0 l=1 rev1=1 crypto rx=0x7f13c41045c0 tx=0x7f13b0005f90 comp rx=0 tx=0).stop 2024-04-23T10:35:03.217 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:35:03.209+0000 7f13cdfee740 1 -- 172.21.15.114:0/3635919563 shutdown_connections 2024-04-23T10:35:03.218 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:35:03.209+0000 7f13cdfee740 1 --2- 172.21.15.114:0/3635919563 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x5603ace55800 0x5603ace57bf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:35:03.218 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:35:03.209+0000 7f13cdfee740 1 --2- 172.21.15.114:0/3635919563 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x5603ace4af40 0x5603ace552c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:35:03.218 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:35:03.209+0000 7f13cdfee740 1 --2- 172.21.15.114:0/3635919563 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x5603ace4a620 0x5603ace4aa00 unknown :-1 s=CLOSED pgs=88 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:35:03.218 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:35:03.209+0000 7f13cdfee740 1 -- 172.21.15.114:0/3635919563 >> 172.21.15.114:0/3635919563 conn(0x5603ace46b30 msgr2=0x5603ace46f10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T10:35:03.218 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:35:03.209+0000 7f13cdfee740 1 -- 172.21.15.114:0/3635919563 shutdown_connections 2024-04-23T10:35:03.218 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:35:03.209+0000 7f13cdfee740 1 -- 172.21.15.114:0/3635919563 wait complete. 2024-04-23T10:35:03.218 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:35:03.209+0000 7f13cdfee740 1 -- start start 2024-04-23T10:35:03.218 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:35:03.209+0000 7f13cdfee740 1 Processor -- start 2024-04-23T10:35:03.219 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:35:03.210+0000 7f13cdfee740 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x5603ace4a620 0x5603ace55170 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T10:35:03.219 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:35:03.210+0000 7f13cdfee740 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x5603ace4af40 0x5603ace52460 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T10:35:03.219 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:35:03.210+0000 7f13cdfee740 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x5603ace55800 0x5603ace529c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T10:35:03.219 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:35:03.210+0000 7f13cdfee740 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x5603ace58430 con 0x5603ace4a620 2024-04-23T10:35:03.219 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:35:03.210+0000 7f13cdfee740 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 -- 0x5603acd4a0b0 con 0x5603ace55800 2024-04-23T10:35:03.219 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:35:03.210+0000 7f13cdfee740 1 -- --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_getmap magic: 0 -- 0x5603ace585b0 con 0x5603ace4af40 2024-04-23T10:35:03.219 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:35:03.210+0000 7f13cdfee740 10 client.0 _ll_register_callbacks cb 0x5603acd37d30 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-23T10:35:03.219 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:35:03.210+0000 7f13c3fff640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x5603ace4a620 0x5603ace55170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T10:35:03.219 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:35:03.210+0000 7f13c3fff640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x5603ace4a620 0x5603ace55170 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.114:53898/0 (socket says 172.21.15.114:53898) 2024-04-23T10:35:03.219 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:35:03.210+0000 7f13c3fff640 1 -- 172.21.15.114:0/3932035763 learned_addr learned my addr 172.21.15.114:0/3932035763 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-04-23T10:35:03.219 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:35:03.210+0000 7f13c8f9a640 1 --2- 172.21.15.114:0/3932035763 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x5603ace4af40 0x5603ace52460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T10:35:03.219 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:35:03.210+0000 7f13c3fff640 1 --2- 172.21.15.114:0/3932035763 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x5603ace55800 0x5603ace529c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T10:35:03.220 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:35:03.210+0000 7f13c3fff640 1 -- 172.21.15.114:0/3932035763 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x5603ace4af40 msgr2=0x5603ace52460 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T10:35:03.220 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:35:03.210+0000 7f13c3fff640 1 --2- 172.21.15.114:0/3932035763 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x5603ace4af40 0x5603ace52460 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:35:03.220 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:35:03.210+0000 7f13c3fff640 1 -- 172.21.15.114:0/3932035763 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x5603ace55800 msgr2=0x5603ace529c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T10:35:03.220 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:35:03.210+0000 7f13c3fff640 1 --2- 172.21.15.114:0/3932035763 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x5603ace55800 0x5603ace529c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:35:03.220 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:35:03.210+0000 7f13c3fff640 1 -- 172.21.15.114:0/3932035763 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5603acb8e620 con 0x5603ace4a620 2024-04-23T10:35:03.220 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:35:03.210+0000 7f13c8f9a640 1 --2- 172.21.15.114:0/3932035763 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x5603ace4af40 0x5603ace52460 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-23T10:35:03.220 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:35:03.210+0000 7f13c3fff640 1 --2- 172.21.15.114:0/3932035763 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x5603ace4a620 0x5603ace55170 secure :-1 s=READY pgs=798 cs=0 l=1 rev1=1 crypto rx=0x7f13b0005890 tx=0x7f13b00046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b454aeb475e3a3d7 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T10:35:03.220 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:35:03.210+0000 7f13a3fff640 10 client.0 ms_handle_connect on v2:172.21.15.114:3300/0 2024-04-23T10:35:03.220 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:35:03.210+0000 7f13a3fff640 1 -- 172.21.15.114:0/3932035763 <== mon.0 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f13b0016050 con 0x5603ace4a620 2024-04-23T10:35:03.221 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:35:03.210+0000 7f13a3fff640 1 -- 172.21.15.114:0/3932035763 <== mon.0 v2:172.21.15.114:3300/0 2 ==== osd_map(98..98 src has 1..98) ==== 8629+0+0 (secure 0 0 0) 0x7f13b0005410 con 0x5603ace4a620 2024-04-23T10:35:03.221 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:35:03.210+0000 7f13cdfee740 1 -- 172.21.15.114:0/3932035763 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f13b00047a0 con 0x5603ace4a620 2024-04-23T10:35:03.221 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:35:03.211+0000 7f13a3fff640 1 -- 172.21.15.114:0/3932035763 <== mon.0 v2:172.21.15.114:3300/0 3 ==== mon_get_version_reply(handle=1 version=31) ==== 24+0+0 (secure 0 0 0) 0x7f13b0034050 con 0x5603ace4a620 2024-04-23T10:35:03.221 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:35:03.211+0000 7f13cdfee740 10 client.16638 fetch_fsmap learned FSMap version 31 2024-04-23T10:35:03.221 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:35:03.211+0000 7f13cdfee740 1 -- 172.21.15.114:0/3932035763 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({config=0+,fsmap=31,monmap=4+}) -- 0x5603ace64360 con 0x5603ace4a620 2024-04-23T10:35:03.221 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:35:03.211+0000 7f13a3fff640 1 -- 172.21.15.114:0/3932035763 <== mon.0 v2:172.21.15.114:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f13b00047a0 con 0x5603ace4a620 2024-04-23T10:35:03.221 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:35:03.211+0000 7f13a3fff640 1 -- 172.21.15.114:0/3932035763 <== mon.0 v2:172.21.15.114:3300/0 5 ==== fsmap(e 31) ==== 8342+0+0 (secure 0 0 0) 0x7f13b000fb10 con 0x5603ace4a620 2024-04-23T10:35:03.221 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:35:03.211+0000 7f13cdfee740 10 client.16638 fetch_fsmap finished waiting for FSMap version 31 2024-04-23T10:35:03.221 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:35:03.211+0000 7f13cdfee740 10 client.16638 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-04-23T10:35:03.221 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:35:03.211+0000 7f13cdfee740 20 client.16638 populate_metadata read hostname 'smithi114' 2024-04-23T10:35:03.222 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:35:03.211+0000 7f13cdfee740 1 --2- 172.21.15.114:0/3932035763 >> [v2:172.21.15.119:6838/937297340,v1:172.21.15.119:6839/937297340] conn(0x5603ace65760 0x5603ace67b50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T10:35:03.222 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:35:03.211+0000 7f13cdfee740 4 client.16638 mds_command: new command op to 24545 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-23T10:35:03.222 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:35:03.211+0000 7f13cdfee740 1 -- 172.21.15.114:0/3932035763 --> [v2:172.21.15.119:6838/937297340,v1:172.21.15.119:6839/937297340] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x5603acdbd390 con 0x5603ace65760 2024-04-23T10:35:03.222 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:35:03.211+0000 7f13c8f9a640 1 --2- 172.21.15.114:0/3932035763 >> [v2:172.21.15.119:6838/937297340,v1:172.21.15.119:6839/937297340] conn(0x5603ace65760 0x5603ace67b50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T10:35:03.222 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:35:03.212+0000 7f13a3fff640 10 client.16638 ms_handle_connect on v2:172.21.15.119:6838/937297340 2024-04-23T10:35:03.222 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:35:03.212+0000 7f13c8f9a640 1 --2- 172.21.15.114:0/3932035763 >> [v2:172.21.15.119:6838/937297340,v1:172.21.15.119:6839/937297340] conn(0x5603ace65760 0x5603ace67b50 crc :-1 s=READY pgs=136 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=b82aed0533b51e31 server_cookie=5ccae6342674ce75 in_seq=0 out_seq=0 2024-04-23T10:35:03.222 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:35:03.212+0000 7f13a3fff640 1 -- 172.21.15.114:0/3932035763 <== mds.0 v2:172.21.15.119:6838/937297340 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x5603acdbd390 con 0x5603ace65760 2024-04-23T10:35:03.222 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:35:03.212+0000 7f13a3fff640 10 client.16638 handle_command_reply: tid=0 2024-04-23T10:35:03.222 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:35:03.212+0000 7f13cdfee740 1 client.16638 shutdown 2024-04-23T10:35:03.223 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:35:03.212+0000 7f13cdfee740 1 -- 172.21.15.114:0/3932035763 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x5603ace4a620 msgr2=0x5603ace55170 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T10:35:03.223 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:35:03.212+0000 7f13cdfee740 1 --2- 172.21.15.114:0/3932035763 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x5603ace4a620 0x5603ace55170 secure :-1 s=READY pgs=798 cs=0 l=1 rev1=1 crypto rx=0x7f13b0005890 tx=0x7f13b00046c0 comp rx=0 tx=0).stop 2024-04-23T10:35:03.223 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:35:03.212+0000 7f13cdfee740 1 -- 172.21.15.114:0/3932035763 shutdown_connections 2024-04-23T10:35:03.223 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:35:03.212+0000 7f13cdfee740 1 --2- 172.21.15.114:0/3932035763 >> [v2:172.21.15.119:6838/937297340,v1:172.21.15.119:6839/937297340] conn(0x5603ace65760 0x5603ace67b50 crc :-1 s=READY pgs=136 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:35:03.223 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:35:03.212+0000 7f13cdfee740 1 --2- 172.21.15.114:0/3932035763 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x5603ace55800 0x5603ace529c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:35:03.223 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:35:03.212+0000 7f13a3fff640 0 client.16638 ms_handle_reset on v2:172.21.15.119:6838/937297340 2024-04-23T10:35:03.223 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:35:03.212+0000 7f13cdfee740 1 --2- 172.21.15.114:0/3932035763 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x5603ace4af40 0x5603ace52460 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:35:03.223 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:35:03.212+0000 7f13cdfee740 1 --2- 172.21.15.114:0/3932035763 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x5603ace4a620 0x5603ace55170 unknown :-1 s=CLOSED pgs=798 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:35:03.224 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:35:03.212+0000 7f13cdfee740 1 -- 172.21.15.114:0/3932035763 >> 172.21.15.114:0/3932035763 conn(0x5603ace46b30 msgr2=0x5603ace4c860 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T10:35:03.224 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:35:03.213+0000 7f13cdfee740 1 -- 172.21.15.114:0/3932035763 shutdown_connections 2024-04-23T10:35:03.224 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:35:03.213+0000 7f13cdfee740 1 -- 172.21.15.114:0/3932035763 wait complete. 2024-04-23T10:35:03.224 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:35:03.213+0000 7f13cdfee740 20 client.16638 trim_cache size 0 max 16384 2024-04-23T10:35:03.232 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:35:03.228+0000 7f13cdfee740 1 Processor -- start 2024-04-23T10:35:03.232 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:35:03.228+0000 7f13cdfee740 1 -- start start 2024-04-23T10:35:03.232 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:35:03.228+0000 7f13cdfee740 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x5603ace4a620 0x5603ace54ca0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T10:35:03.232 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:35:03.228+0000 7f13cdfee740 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x5603ace4af40 0x5603ace6d150 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T10:35:03.232 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:35:03.228+0000 7f13cdfee740 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x5603ace55800 0x5603ace6d690 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T10:35:03.232 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:35:03.228+0000 7f13cdfee740 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x7f13c41b27f0 con 0x5603ace55800 2024-04-23T10:35:03.232 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:35:03.228+0000 7f13cdfee740 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 -- 0x7f13c41b0090 con 0x5603ace4af40 2024-04-23T10:35:03.233 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:35:03.228+0000 7f13cdfee740 1 -- --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_getmap magic: 0 -- 0x7f13c41adf00 con 0x5603ace4a620 2024-04-23T10:35:03.233 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:35:03.228+0000 7f13c3fff640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x5603ace55800 0x5603ace6d690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T10:35:03.233 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:35:03.228+0000 7f13c3fff640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x5603ace55800 0x5603ace6d690 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.114:53900/0 (socket says 172.21.15.114:53900) 2024-04-23T10:35:03.233 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:35:03.228+0000 7f13c3fff640 1 -- 172.21.15.114:0/681165385 learned_addr learned my addr 172.21.15.114:0/681165385 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-04-23T10:35:03.233 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:35:03.228+0000 7f13c8f9a640 1 --2- 172.21.15.114:0/681165385 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x5603ace4af40 0x5603ace6d150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T10:35:03.233 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:35:03.228+0000 7f13c3fff640 1 --2- 172.21.15.114:0/681165385 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x5603ace4a620 0x5603ace54ca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T10:35:03.233 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:35:03.229+0000 7f13c3fff640 1 -- 172.21.15.114:0/681165385 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x5603ace4a620 msgr2=0x5603ace54ca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T10:35:03.233 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:35:03.229+0000 7f13c3fff640 1 --2- 172.21.15.114:0/681165385 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x5603ace4a620 0x5603ace54ca0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:35:03.233 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:35:03.229+0000 7f13c3fff640 1 -- 172.21.15.114:0/681165385 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x5603ace4af40 msgr2=0x5603ace6d150 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T10:35:03.233 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:35:03.229+0000 7f13c3fff640 1 --2- 172.21.15.114:0/681165385 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x5603ace4af40 0x5603ace6d150 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:35:03.233 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:35:03.229+0000 7f13c3fff640 1 -- 172.21.15.114:0/681165385 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5603ace4ad30 con 0x5603ace55800 2024-04-23T10:35:03.233 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:35:03.229+0000 7f13c8f9a640 1 --2- 172.21.15.114:0/681165385 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x5603ace4af40 0x5603ace6d150 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-23T10:35:03.234 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:35:03.229+0000 7f13c3fff640 1 --2- 172.21.15.114:0/681165385 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x5603ace55800 0x5603ace6d690 secure :-1 s=READY pgs=799 cs=0 l=1 rev1=1 crypto rx=0x7f13b00104f0 tx=0x7f13b0009930 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ccde9a81216c0eec server_cookie=0 in_seq=0 out_seq=0 2024-04-23T10:35:03.234 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:35:03.229+0000 7f13ae7fc640 1 -- 172.21.15.114:0/681165385 <== mon.0 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f13b0016070 con 0x5603ace55800 2024-04-23T10:35:03.234 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:35:03.229+0000 7f13ae7fc640 1 -- 172.21.15.114:0/681165385 <== mon.0 v2:172.21.15.114:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f13b002a040 con 0x5603ace55800 2024-04-23T10:35:03.234 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:35:03.229+0000 7f13ae7fc640 1 -- 172.21.15.114:0/681165385 <== mon.0 v2:172.21.15.114:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f13b000fac0 con 0x5603ace55800 2024-04-23T10:35:03.234 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:35:03.229+0000 7f13cdfee740 1 -- 172.21.15.114:0/681165385 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x5603ace55800 msgr2=0x5603ace6d690 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T10:35:03.234 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:35:03.229+0000 7f13cdfee740 1 --2- 172.21.15.114:0/681165385 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x5603ace55800 0x5603ace6d690 secure :-1 s=READY pgs=799 cs=0 l=1 rev1=1 crypto rx=0x7f13b00104f0 tx=0x7f13b0009930 comp rx=0 tx=0).stop 2024-04-23T10:35:03.234 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:35:03.230+0000 7f13cdfee740 1 -- 172.21.15.114:0/681165385 shutdown_connections 2024-04-23T10:35:03.234 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:35:03.230+0000 7f13cdfee740 1 --2- 172.21.15.114:0/681165385 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x5603ace55800 0x5603ace6d690 unknown :-1 s=CLOSED pgs=799 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:35:03.234 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:35:03.230+0000 7f13cdfee740 1 --2- 172.21.15.114:0/681165385 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x5603ace4af40 0x5603ace6d150 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:35:03.234 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:35:03.230+0000 7f13cdfee740 1 --2- 172.21.15.114:0/681165385 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x5603ace4a620 0x5603ace54ca0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:35:03.234 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:35:03.230+0000 7f13cdfee740 1 -- 172.21.15.114:0/681165385 >> 172.21.15.114:0/681165385 conn(0x5603ace46b30 msgr2=0x5603ace4c860 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T10:35:03.234 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:35:03.230+0000 7f13cdfee740 1 -- 172.21.15.114:0/681165385 shutdown_connections 2024-04-23T10:35:03.235 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:35:03.230+0000 7f13cdfee740 1 -- 172.21.15.114:0/681165385 wait complete. 2024-04-23T10:35:03.235 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:35:03.230+0000 7f13cdfee740 1 -- start start 2024-04-23T10:35:03.235 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:35:03.230+0000 7f13cdfee740 1 Processor -- start 2024-04-23T10:35:03.235 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:35:03.230+0000 7f13cdfee740 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x5603ace4a620 0x5603ace587c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T10:35:03.235 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:35:03.230+0000 7f13cdfee740 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x5603ace4af40 0x5603ace58d00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T10:35:03.235 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:35:03.230+0000 7f13cdfee740 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x5603ace55800 0x5603ace59240 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T10:35:03.235 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:35:03.230+0000 7f13cdfee740 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x7f13c41ae550 con 0x5603ace4af40 2024-04-23T10:35:03.235 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:35:03.230+0000 7f13cdfee740 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 -- 0x7f13c41af9c0 con 0x5603ace4a620 2024-04-23T10:35:03.235 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:35:03.230+0000 7f13cdfee740 1 -- --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_getmap magic: 0 -- 0x7f13c41aec20 con 0x5603ace55800 2024-04-23T10:35:03.235 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:35:03.230+0000 7f13cdfee740 10 client.0 _ll_register_callbacks cb 0x7f13b00107a0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-23T10:35:03.236 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:35:03.230+0000 7f13c8f9a640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x5603ace4af40 0x5603ace58d00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T10:35:03.236 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:35:03.230+0000 7f13c8f9a640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x5603ace4af40 0x5603ace58d00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.114:53904/0 (socket says 172.21.15.114:53904) 2024-04-23T10:35:03.236 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:35:03.230+0000 7f13c8f9a640 1 -- 172.21.15.114:0/3484408920 learned_addr learned my addr 172.21.15.114:0/3484408920 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-04-23T10:35:03.236 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:35:03.231+0000 7f13c3fff640 1 --2- 172.21.15.114:0/3484408920 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x5603ace4a620 0x5603ace587c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T10:35:03.236 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:35:03.231+0000 7f13c3fff640 1 --2- 172.21.15.114:0/3484408920 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x5603ace55800 0x5603ace59240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T10:35:03.236 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:35:03.231+0000 7f13c8f9a640 1 -- 172.21.15.114:0/3484408920 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x5603ace55800 msgr2=0x5603ace59240 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T10:35:03.236 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:35:03.231+0000 7f13c8f9a640 1 --2- 172.21.15.114:0/3484408920 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x5603ace55800 0x5603ace59240 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:35:03.236 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:35:03.231+0000 7f13c8f9a640 1 -- 172.21.15.114:0/3484408920 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x5603ace4a620 msgr2=0x5603ace587c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T10:35:03.236 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:35:03.231+0000 7f13c8f9a640 1 --2- 172.21.15.114:0/3484408920 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x5603ace4a620 0x5603ace587c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:35:03.236 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:35:03.231+0000 7f13c8f9a640 1 -- 172.21.15.114:0/3484408920 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5603acec9a60 con 0x5603ace4af40 2024-04-23T10:35:03.237 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:35:03.231+0000 7f13c8f9a640 1 --2- 172.21.15.114:0/3484408920 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x5603ace4af40 0x5603ace58d00 secure :-1 s=READY pgs=800 cs=0 l=1 rev1=1 crypto rx=0x7f13b80096c0 tx=0x7f13b80043d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d2c534ffd72544a server_cookie=0 in_seq=0 out_seq=0 2024-04-23T10:35:03.237 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:35:03.231+0000 7f13a3fff640 10 client.0 ms_handle_connect on v2:172.21.15.114:3300/0 2024-04-23T10:35:03.237 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:35:03.231+0000 7f13a3fff640 1 -- 172.21.15.114:0/3484408920 <== mon.0 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f13b8035040 con 0x5603ace4af40 2024-04-23T10:35:03.237 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:35:03.231+0000 7f13a3fff640 1 -- 172.21.15.114:0/3484408920 <== mon.0 v2:172.21.15.114:3300/0 2 ==== osd_map(98..98 src has 1..98) ==== 8629+0+0 (secure 0 0 0) 0x7f13b804a830 con 0x5603ace4af40 2024-04-23T10:35:03.237 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:35:03.231+0000 7f13cdfee740 1 -- 172.21.15.114:0/3484408920 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f13b002a040 con 0x5603ace4af40 2024-04-23T10:35:03.237 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:35:03.231+0000 7f13a3fff640 1 -- 172.21.15.114:0/3484408920 <== mon.0 v2:172.21.15.114:3300/0 3 ==== mon_get_version_reply(handle=1 version=31) ==== 24+0+0 (secure 0 0 0) 0x7f13b8058050 con 0x5603ace4af40 2024-04-23T10:35:03.237 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:35:03.231+0000 7f13cdfee740 10 client.16644 fetch_fsmap learned FSMap version 31 2024-04-23T10:35:03.237 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:35:03.231+0000 7f13cdfee740 1 -- 172.21.15.114:0/3484408920 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({config=0+,fsmap=31,monmap=4+}) -- 0x5603acecade0 con 0x5603ace4af40 2024-04-23T10:35:03.237 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:35:03.231+0000 7f13a3fff640 1 -- 172.21.15.114:0/3484408920 <== mon.0 v2:172.21.15.114:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f13b002a040 con 0x5603ace4af40 2024-04-23T10:35:03.237 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:35:03.232+0000 7f13a3fff640 1 -- 172.21.15.114:0/3484408920 <== mon.0 v2:172.21.15.114:3300/0 5 ==== fsmap(e 31) ==== 8342+0+0 (secure 0 0 0) 0x7f13b8004bb0 con 0x5603ace4af40 2024-04-23T10:35:03.237 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:35:03.232+0000 7f13cdfee740 10 client.16644 fetch_fsmap finished waiting for FSMap version 31 2024-04-23T10:35:03.237 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:35:03.232+0000 7f13cdfee740 10 client.16644 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-04-23T10:35:03.238 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:35:03.232+0000 7f13cdfee740 20 client.16644 populate_metadata read hostname 'smithi114' 2024-04-23T10:35:03.238 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:35:03.232+0000 7f13cdfee740 1 --2- 172.21.15.114:0/3484408920 >> [v2:172.21.15.119:6838/937297340,v1:172.21.15.119:6839/937297340] conn(0x5603acecb590 0x5603acecd9b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T10:35:03.238 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:35:03.232+0000 7f13cdfee740 4 client.16644 mds_command: new command op to 24545 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-04-23T10:35:03.238 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:35:03.232+0000 7f13cdfee740 1 -- 172.21.15.114:0/3484408920 --> [v2:172.21.15.119:6838/937297340,v1:172.21.15.119:6839/937297340] -- command(tid 0: {"prefix": "flush journal"}) -- 0x5603acdb7200 con 0x5603acecb590 2024-04-23T10:35:03.238 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:35:03.232+0000 7f13c3fff640 1 --2- 172.21.15.114:0/3484408920 >> [v2:172.21.15.119:6838/937297340,v1:172.21.15.119:6839/937297340] conn(0x5603acecb590 0x5603acecd9b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T10:35:03.238 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:35:03.232+0000 7f13c3fff640 1 --2- 172.21.15.114:0/3484408920 >> [v2:172.21.15.119:6838/937297340,v1:172.21.15.119:6839/937297340] conn(0x5603acecb590 0x5603acecd9b0 crc :-1 s=READY pgs=137 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=213e2d04ed02f632 server_cookie=2578322dd66c7362 in_seq=0 out_seq=0 2024-04-23T10:35:03.238 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:35:03.232+0000 7f13a3fff640 10 client.16644 ms_handle_connect on v2:172.21.15.119:6838/937297340 2024-04-23T10:35:03.306 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:35:03.306+0000 7f13a3fff640 1 -- 172.21.15.114:0/3484408920 <== mds.0 v2:172.21.15.119:6838/937297340 1 ==== command_reply(tid 0: 0 ) ==== 8+0+44 (crc 0 0 0) 0x5603acdb7200 con 0x5603acecb590 2024-04-23T10:35:03.306 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:35:03.306+0000 7f13a3fff640 10 client.16644 handle_command_reply: tid=0 2024-04-23T10:35:03.306 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:35:03.306+0000 7f13cdfee740 1 client.16644 shutdown 2024-04-23T10:35:03.306 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:35:03.306+0000 7f13cdfee740 1 -- 172.21.15.114:0/3484408920 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x5603ace4af40 msgr2=0x5603ace58d00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T10:35:03.306 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:35:03.306+0000 7f13cdfee740 1 --2- 172.21.15.114:0/3484408920 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x5603ace4af40 0x5603ace58d00 secure :-1 s=READY pgs=800 cs=0 l=1 rev1=1 crypto rx=0x7f13b80096c0 tx=0x7f13b80043d0 comp rx=0 tx=0).stop 2024-04-23T10:35:03.306 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:35:03.306+0000 7f13cdfee740 1 -- 172.21.15.114:0/3484408920 shutdown_connections 2024-04-23T10:35:03.306 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:35:03.306+0000 7f13cdfee740 1 --2- 172.21.15.114:0/3484408920 >> [v2:172.21.15.119:6838/937297340,v1:172.21.15.119:6839/937297340] conn(0x5603acecb590 0x5603acecd9b0 crc :-1 s=READY pgs=137 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:35:03.306 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:35:03.306+0000 7f13cdfee740 1 --2- 172.21.15.114:0/3484408920 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x5603ace55800 0x5603ace59240 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:35:03.306 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:35:03.306+0000 7f13cdfee740 1 --2- 172.21.15.114:0/3484408920 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x5603ace4af40 0x5603ace58d00 unknown :-1 s=CLOSED pgs=800 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:35:03.306 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:35:03.306+0000 7f13cdfee740 1 --2- 172.21.15.114:0/3484408920 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x5603ace4a620 0x5603ace587c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:35:03.306 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:35:03.306+0000 7f13a3fff640 0 client.16644 ms_handle_reset on v2:172.21.15.119:6838/937297340 2024-04-23T10:35:03.307 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:35:03.306+0000 7f13cdfee740 1 -- 172.21.15.114:0/3484408920 >> 172.21.15.114:0/3484408920 conn(0x5603ace46b30 msgr2=0x5603ace45420 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T10:35:03.307 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:35:03.307+0000 7f13cdfee740 1 -- 172.21.15.114:0/3484408920 shutdown_connections 2024-04-23T10:35:03.307 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:35:03.307+0000 7f13cdfee740 1 -- 172.21.15.114:0/3484408920 wait complete. 2024-04-23T10:35:03.307 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:35:03.307+0000 7f13cdfee740 20 client.16644 trim_cache size 0 max 16384 2024-04-23T10:35:03.307 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stdout:{ 2024-04-23T10:35:03.307 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stdout: "message": "", 2024-04-23T10:35:03.307 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stdout: "return_code": 0 2024-04-23T10:35:03.307 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stdout:} 2024-04-23T10:35:03.308 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:35:03.307+0000 7f13cb1fe640 1 -- 172.21.15.114:0/109278636 >> [v2:172.21.15.114:6800/1395693466,v1:172.21.15.114:6801/1395693466] conn(0x7f13b40a3df0 msgr2=0x7f13b40a62b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T10:35:03.308 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:35:03.307+0000 7f13cb1fe640 1 --2- 172.21.15.114:0/109278636 >> [v2:172.21.15.114:6800/1395693466,v1:172.21.15.114:6801/1395693466] conn(0x7f13b40a3df0 0x7f13b40a62b0 secure :-1 s=READY pgs=122 cs=0 l=1 rev1=1 crypto rx=0x7f13bc0ce850 tx=0x7f13bc06e4e0 comp rx=0 tx=0).stop 2024-04-23T10:35:03.309 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:35:03.307+0000 7f13cb1fe640 1 -- 172.21.15.114:0/109278636 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f13c41035b0 msgr2=0x7f13c41a6840 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T10:35:03.309 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:35:03.307+0000 7f13cb1fe640 1 --2- 172.21.15.114:0/109278636 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f13c41035b0 0x7f13c41a6840 secure :-1 s=READY pgs=87 cs=0 l=1 rev1=1 crypto rx=0x7f13bc00da10 tx=0x7f13bc00ded0 comp rx=0 tx=0).stop 2024-04-23T10:35:03.309 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:35:03.308+0000 7f13cb1fe640 1 -- 172.21.15.114:0/109278636 shutdown_connections 2024-04-23T10:35:03.309 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:35:03.308+0000 7f13cb1fe640 1 --2- 172.21.15.114:0/109278636 >> [v2:172.21.15.114:6800/1395693466,v1:172.21.15.114:6801/1395693466] conn(0x7f13b40a3df0 0x7f13b40a62b0 unknown :-1 s=CLOSED pgs=122 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:35:03.309 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:35:03.308+0000 7f13cb1fe640 1 --2- 172.21.15.114:0/109278636 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f13c41035b0 0x7f13c41a6840 unknown :-1 s=CLOSED pgs=87 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:35:03.309 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:35:03.308+0000 7f13cb1fe640 1 --2- 172.21.15.114:0/109278636 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f13c4106170 0x7f13c4103070 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:35:03.309 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:35:03.308+0000 7f13cb1fe640 1 --2- 172.21.15.114:0/109278636 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f13c41010f0 0x7f13c4102b30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:35:03.309 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:35:03.308+0000 7f13cb1fe640 1 -- 172.21.15.114:0/109278636 >> 172.21.15.114:0/109278636 conn(0x7f13c4077fd0 msgr2=0x7f13c4069930 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T10:35:03.309 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:35:03.308+0000 7f13cb1fe640 1 -- 172.21.15.114:0/109278636 shutdown_connections 2024-04-23T10:35:03.309 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:35:03.308+0000 7f13cb1fe640 1 -- 172.21.15.114:0/109278636 wait complete. 2024-04-23T10:35:03.742 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 23 10:35:03 smithi119 ceph-mon[32608]: pgmap v262: 97 pgs: 97 active+clean; 199 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 MiB/s rd, 4.6 MiB/s wr, 400 op/s 2024-04-23T10:35:03.753 INFO:journalctl@ceph.mon.a.smithi114.stdout:Apr 23 10:35:03 smithi114 ceph-mon[29255]: pgmap v262: 97 pgs: 97 active+clean; 199 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 MiB/s rd, 4.6 MiB/s wr, 400 op/s 2024-04-23T10:35:03.828 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 23 10:35:03 smithi204 ceph-mon[32649]: pgmap v262: 97 pgs: 97 active+clean; 199 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 MiB/s rd, 4.6 MiB/s wr, 400 op/s 2024-04-23T10:35:03.930 INFO:tasks.workunit.client.0.smithi114.stdout: 1 608993 7.50 MB/sec execute 312 sec latency 43.942 ms 2024-04-23T10:35:04.742 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 23 10:35:04 smithi119 ceph-mon[32608]: osdmap e99: 12 total, 12 up, 12 in 2024-04-23T10:35:04.753 INFO:journalctl@ceph.mon.a.smithi114.stdout:Apr 23 10:35:04 smithi114 ceph-mon[29255]: osdmap e99: 12 total, 12 up, 12 in 2024-04-23T10:35:04.827 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 23 10:35:04 smithi204 ceph-mon[32649]: osdmap e99: 12 total, 12 up, 12 in 2024-04-23T10:35:04.930 INFO:tasks.workunit.client.0.smithi114.stdout: 1 611115 7.52 MB/sec execute 313 sec latency 165.855 ms 2024-04-23T10:35:05.742 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 23 10:35:05 smithi119 ceph-mon[32608]: pgmap v264: 97 pgs: 97 active+clean; 199 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 MiB/s rd, 3.6 MiB/s wr, 299 op/s 2024-04-23T10:35:05.755 INFO:journalctl@ceph.mon.a.smithi114.stdout:Apr 23 10:35:05 smithi114 ceph-mon[29255]: pgmap v264: 97 pgs: 97 active+clean; 199 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 MiB/s rd, 3.6 MiB/s wr, 299 op/s 2024-04-23T10:35:05.828 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 23 10:35:05 smithi204 ceph-mon[32649]: pgmap v264: 97 pgs: 97 active+clean; 199 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 MiB/s rd, 3.6 MiB/s wr, 299 op/s 2024-04-23T10:35:05.931 INFO:tasks.workunit.client.0.smithi114.stdout: 1 612366 7.51 MB/sec execute 314 sec latency 76.410 ms 2024-04-23T10:35:06.931 INFO:tasks.workunit.client.0.smithi114.stdout: 1 614663 7.53 MB/sec execute 315 sec latency 27.364 ms 2024-04-23T10:35:07.754 INFO:journalctl@ceph.mon.a.smithi114.stdout:Apr 23 10:35:07 smithi114 ceph-mon[29255]: pgmap v265: 97 pgs: 97 active+clean; 189 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 MiB/s rd, 3.5 MiB/s wr, 339 op/s 2024-04-23T10:35:07.827 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 23 10:35:07 smithi204 ceph-mon[32649]: pgmap v265: 97 pgs: 97 active+clean; 189 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 MiB/s rd, 3.5 MiB/s wr, 339 op/s 2024-04-23T10:35:07.931 INFO:tasks.workunit.client.0.smithi114.stdout: 1 616046 7.52 MB/sec execute 316 sec latency 20.095 ms 2024-04-23T10:35:07.991 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 23 10:35:07 smithi119 ceph-mon[32608]: pgmap v265: 97 pgs: 97 active+clean; 189 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 MiB/s rd, 3.5 MiB/s wr, 339 op/s 2024-04-23T10:35:08.754 INFO:journalctl@ceph.mon.a.smithi114.stdout:Apr 23 10:35:08 smithi114 ceph-mon[29255]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=e=up:active} 3 up:standby-replay 6 up:standby 2024-04-23T10:35:08.754 INFO:journalctl@ceph.mon.a.smithi114.stdout:Apr 23 10:35:08 smithi114 ceph-mon[29255]: osdmap e100: 12 total, 12 up, 12 in 2024-04-23T10:35:08.827 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 23 10:35:08 smithi204 ceph-mon[32649]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=e=up:active} 3 up:standby-replay 6 up:standby 2024-04-23T10:35:08.827 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 23 10:35:08 smithi204 ceph-mon[32649]: osdmap e100: 12 total, 12 up, 12 in 2024-04-23T10:35:08.931 INFO:tasks.workunit.client.0.smithi114.stdout: 1 617926 7.54 MB/sec execute 317 sec latency 356.734 ms 2024-04-23T10:35:08.991 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 23 10:35:08 smithi119 ceph-mon[32608]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=e=up:active} 3 up:standby-replay 6 up:standby 2024-04-23T10:35:08.992 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 23 10:35:08 smithi119 ceph-mon[32608]: osdmap e100: 12 total, 12 up, 12 in 2024-04-23T10:35:09.828 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 23 10:35:09 smithi204 ceph-mon[32649]: pgmap v266: 97 pgs: 97 active+clean; 189 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 MiB/s rd, 3.5 MiB/s wr, 339 op/s 2024-04-23T10:35:09.931 INFO:tasks.workunit.client.0.smithi114.stdout: 1 617926 7.52 MB/sec execute 318 sec latency 1356.825 ms 2024-04-23T10:35:09.992 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 23 10:35:09 smithi119 ceph-mon[32608]: pgmap v266: 97 pgs: 97 active+clean; 189 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 MiB/s rd, 3.5 MiB/s wr, 339 op/s 2024-04-23T10:35:10.004 INFO:journalctl@ceph.mon.a.smithi114.stdout:Apr 23 10:35:09 smithi114 ceph-mon[29255]: pgmap v266: 97 pgs: 97 active+clean; 189 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 MiB/s rd, 3.5 MiB/s wr, 339 op/s 2024-04-23T10:35:10.931 INFO:tasks.workunit.client.0.smithi114.stdout: 1 617926 7.49 MB/sec execute 319 sec latency 2356.927 ms 2024-04-23T10:35:11.828 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 23 10:35:11 smithi204 ceph-mon[32649]: pgmap v268: 97 pgs: 97 active+clean; 189 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail; 901 KiB/s rd, 2.2 MiB/s wr, 214 op/s 2024-04-23T10:35:11.931 INFO:tasks.workunit.client.0.smithi114.stdout: 1 617926 7.47 MB/sec execute 320 sec latency 3357.063 ms 2024-04-23T10:35:11.992 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 23 10:35:11 smithi119 ceph-mon[32608]: pgmap v268: 97 pgs: 97 active+clean; 189 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail; 901 KiB/s rd, 2.2 MiB/s wr, 214 op/s 2024-04-23T10:35:12.004 INFO:journalctl@ceph.mon.a.smithi114.stdout:Apr 23 10:35:11 smithi114 ceph-mon[29255]: pgmap v268: 97 pgs: 97 active+clean; 189 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail; 901 KiB/s rd, 2.2 MiB/s wr, 214 op/s 2024-04-23T10:35:12.931 INFO:tasks.workunit.client.0.smithi114.stdout: 1 617926 7.45 MB/sec execute 321 sec latency 4357.181 ms 2024-04-23T10:35:13.828 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 23 10:35:13 smithi204 ceph-mon[32649]: pgmap v269: 97 pgs: 97 active+clean; 201 MiB data, 2.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.3 MiB/s rd, 3.4 MiB/s wr, 271 op/s 2024-04-23T10:35:13.931 INFO:tasks.workunit.client.0.smithi114.stdout: 1 617926 7.42 MB/sec execute 322 sec latency 5357.261 ms 2024-04-23T10:35:13.992 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 23 10:35:13 smithi119 ceph-mon[32608]: pgmap v269: 97 pgs: 97 active+clean; 201 MiB data, 2.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.3 MiB/s rd, 3.4 MiB/s wr, 271 op/s 2024-04-23T10:35:14.004 INFO:journalctl@ceph.mon.a.smithi114.stdout:Apr 23 10:35:13 smithi114 ceph-mon[29255]: pgmap v269: 97 pgs: 97 active+clean; 201 MiB data, 2.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.3 MiB/s rd, 3.4 MiB/s wr, 271 op/s 2024-04-23T10:35:14.931 INFO:tasks.workunit.client.0.smithi114.stdout: 1 617926 7.40 MB/sec execute 323 sec latency 6357.296 ms 2024-04-23T10:35:15.827 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 23 10:35:15 smithi204 ceph-mon[32649]: pgmap v270: 97 pgs: 97 active+clean; 201 MiB data, 2.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 MiB/s rd, 3.0 MiB/s wr, 242 op/s 2024-04-23T10:35:15.931 INFO:tasks.workunit.client.0.smithi114.stdout: 1 617926 7.38 MB/sec execute 324 sec latency 7357.401 ms 2024-04-23T10:35:15.992 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 23 10:35:15 smithi119 ceph-mon[32608]: pgmap v270: 97 pgs: 97 active+clean; 201 MiB data, 2.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 MiB/s rd, 3.0 MiB/s wr, 242 op/s 2024-04-23T10:35:16.003 INFO:journalctl@ceph.mon.a.smithi114.stdout:Apr 23 10:35:15 smithi114 ceph-mon[29255]: pgmap v270: 97 pgs: 97 active+clean; 201 MiB data, 2.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 MiB/s rd, 3.0 MiB/s wr, 242 op/s 2024-04-23T10:35:16.406 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:35:16.405+0000 7efeef2cf640 1 -- 172.21.15.114:0/264167034 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7efee8107290 msgr2=0x7efee810ee60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T10:35:16.406 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:35:16.405+0000 7efeef2cf640 1 --2- 172.21.15.114:0/264167034 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7efee8107290 0x7efee810ee60 secure :-1 s=READY pgs=801 cs=0 l=1 rev1=1 crypto rx=0x7efee0009ca0 tx=0x7efee002a8c0 comp rx=0 tx=0).stop 2024-04-23T10:35:16.406 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:35:16.405+0000 7efeef2cf640 1 -- 172.21.15.114:0/264167034 shutdown_connections 2024-04-23T10:35:16.406 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:35:16.405+0000 7efeef2cf640 1 --2- 172.21.15.114:0/264167034 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7efee8107290 0x7efee810ee60 unknown :-1 s=CLOSED pgs=801 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:35:16.406 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:35:16.405+0000 7efeef2cf640 1 --2- 172.21.15.114:0/264167034 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7efee8102b70 0x7efee8106bb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:35:16.406 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:35:16.405+0000 7efeef2cf640 1 --2- 172.21.15.114:0/264167034 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7efee81021c0 0x7efee81025a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:35:16.407 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:35:16.405+0000 7efeef2cf640 1 -- 172.21.15.114:0/264167034 >> 172.21.15.114:0/264167034 conn(0x7efee80fda20 msgr2=0x7efee80ffe40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T10:35:16.407 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:35:16.405+0000 7efeef2cf640 1 -- 172.21.15.114:0/264167034 shutdown_connections 2024-04-23T10:35:16.407 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:35:16.405+0000 7efeef2cf640 1 -- 172.21.15.114:0/264167034 wait complete. 2024-04-23T10:35:16.407 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:35:16.406+0000 7efeef2cf640 1 Processor -- start 2024-04-23T10:35:16.407 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:35:16.406+0000 7efeef2cf640 1 -- start start 2024-04-23T10:35:16.407 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:35:16.406+0000 7efeef2cf640 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7efee81021c0 0x7efee81985c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T10:35:16.407 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:35:16.406+0000 7efeef2cf640 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7efee8107290 0x7efee8198b00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T10:35:16.407 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:35:16.406+0000 7efeef2cf640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7efee819ce80 0x7efee819d330 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T10:35:16.407 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:35:16.406+0000 7efeef2cf640 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x7efee81121e0 con 0x7efee819ce80 2024-04-23T10:35:16.407 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:35:16.406+0000 7efeef2cf640 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 -- 0x7efee8112060 con 0x7efee81021c0 2024-04-23T10:35:16.407 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:35:16.406+0000 7efeef2cf640 1 -- --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_getmap magic: 0 -- 0x7efee8112360 con 0x7efee8107290 2024-04-23T10:35:16.407 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:35:16.406+0000 7efeed86c640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7efee819ce80 0x7efee819d330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T10:35:16.407 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:35:16.406+0000 7efeed86c640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7efee819ce80 0x7efee819d330 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.114:57518/0 (socket says 172.21.15.114:57518) 2024-04-23T10:35:16.407 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:35:16.406+0000 7efeed86c640 1 -- 172.21.15.114:0/2851140285 learned_addr learned my addr 172.21.15.114:0/2851140285 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-04-23T10:35:16.408 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:35:16.406+0000 7efeec86a640 1 --2- 172.21.15.114:0/2851140285 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7efee8107290 0x7efee8198b00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T10:35:16.408 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:35:16.406+0000 7efeed06b640 1 --2- 172.21.15.114:0/2851140285 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7efee81021c0 0x7efee81985c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T10:35:16.408 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:35:16.406+0000 7efeed86c640 1 -- 172.21.15.114:0/2851140285 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7efee8107290 msgr2=0x7efee8198b00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T10:35:16.408 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:35:16.407+0000 7efeed86c640 1 --2- 172.21.15.114:0/2851140285 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7efee8107290 0x7efee8198b00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:35:16.408 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:35:16.407+0000 7efeed86c640 1 -- 172.21.15.114:0/2851140285 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7efee81021c0 msgr2=0x7efee81985c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T10:35:16.408 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:35:16.407+0000 7efeed86c640 1 --2- 172.21.15.114:0/2851140285 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7efee81021c0 0x7efee81985c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:35:16.408 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:35:16.407+0000 7efeed86c640 1 -- 172.21.15.114:0/2851140285 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7efee819d970 con 0x7efee819ce80 2024-04-23T10:35:16.408 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:35:16.407+0000 7efeed06b640 1 --2- 172.21.15.114:0/2851140285 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7efee81021c0 0x7efee81985c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-23T10:35:16.408 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:35:16.407+0000 7efeed86c640 1 --2- 172.21.15.114:0/2851140285 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7efee819ce80 0x7efee819d330 secure :-1 s=READY pgs=802 cs=0 l=1 rev1=1 crypto rx=0x7efee00098d0 tx=0x7efee00317b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T10:35:16.408 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:35:16.407+0000 7efeda7fc640 1 -- 172.21.15.114:0/2851140285 <== mon.0 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7efee0003b70 con 0x7efee819ce80 2024-04-23T10:35:16.408 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:35:16.407+0000 7efeda7fc640 1 -- 172.21.15.114:0/2851140285 <== mon.0 v2:172.21.15.114:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7efee0003d10 con 0x7efee819ce80 2024-04-23T10:35:16.408 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:35:16.407+0000 7efeef2cf640 1 -- 172.21.15.114:0/2851140285 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7efee819dc00 con 0x7efee819ce80 2024-04-23T10:35:16.408 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:35:16.407+0000 7efeda7fc640 1 -- 172.21.15.114:0/2851140285 <== mon.0 v2:172.21.15.114:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7efee002fa20 con 0x7efee819ce80 2024-04-23T10:35:16.408 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:35:16.407+0000 7efeef2cf640 1 -- 172.21.15.114:0/2851140285 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7efee81a5420 con 0x7efee819ce80 2024-04-23T10:35:16.414 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:35:16.409+0000 7efeda7fc640 1 -- 172.21.15.114:0/2851140285 <== mon.0 v2:172.21.15.114:3300/0 4 ==== mgrmap(e 26) ==== 138778+0+0 (secure 0 0 0) 0x7efee002fc40 con 0x7efee819ce80 2024-04-23T10:35:16.414 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:35:16.410+0000 7efef1feb740 1 Processor -- start 2024-04-23T10:35:16.414 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:35:16.410+0000 7efef1feb740 1 -- start start 2024-04-23T10:35:16.414 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:35:16.410+0000 7efef1feb740 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x55a35ff4b620 0x55a35ff4ba00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T10:35:16.414 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:35:16.410+0000 7efef1feb740 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x55a35ff4bf40 0x55a35ff562c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T10:35:16.414 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:35:16.410+0000 7efef1feb740 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55a35ff56800 0x55a35ff58bf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T10:35:16.414 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:35:16.410+0000 7efef1feb740 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x55a35feb0970 con 0x55a35ff56800 2024-04-23T10:35:16.414 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:35:16.410+0000 7efef1feb740 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 -- 0x55a35fe4d4a0 con 0x55a35ff4b620 2024-04-23T10:35:16.414 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:35:16.410+0000 7efef1feb740 1 -- --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_getmap magic: 0 -- 0x55a35feac930 con 0x55a35ff4bf40 2024-04-23T10:35:16.414 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:35:16.410+0000 7efeda7fc640 1 --2- 172.21.15.114:0/2851140285 >> [v2:172.21.15.114:6800/1395693466,v1:172.21.15.114:6801/1395693466] conn(0x7efed40a3dc0 0x7efed40a6280 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T10:35:16.415 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:35:16.411+0000 7efeec86a640 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x55a35ff4b620 0x55a35ff4ba00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T10:35:16.415 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:35:16.411+0000 7efeed86c640 1 --2- 172.21.15.114:0/2851140285 >> [v2:172.21.15.114:6800/1395693466,v1:172.21.15.114:6801/1395693466] conn(0x7efed40a3dc0 0x7efed40a6280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T10:35:16.415 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:35:16.411+0000 7efeec86a640 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x55a35ff4b620 0x55a35ff4ba00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.119:3300/0 says I am v2:172.21.15.114:57962/0 (socket says 172.21.15.114:57962) 2024-04-23T10:35:16.415 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:35:16.411+0000 7efeec86a640 1 -- 172.21.15.114:0/4263870965 learned_addr learned my addr 172.21.15.114:0/4263870965 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-04-23T10:35:16.415 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:35:16.411+0000 7efeed06b640 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x55a35ff4bf40 0x55a35ff562c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T10:35:16.415 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:35:16.411+0000 7efeec86a640 1 --2- 172.21.15.114:0/4263870965 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55a35ff56800 0x55a35ff58bf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T10:35:16.415 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:35:16.411+0000 7efeda7fc640 1 -- 172.21.15.114:0/2851140285 <== mon.0 v2:172.21.15.114:3300/0 5 ==== osd_map(100..100 src has 1..100) ==== 8629+0+0 (secure 0 0 0) 0x7efee00f1c20 con 0x7efee819ce80 2024-04-23T10:35:16.415 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:35:16.411+0000 7efeec86a640 1 -- 172.21.15.114:0/4263870965 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x55a35ff4bf40 msgr2=0x55a35ff562c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T10:35:16.415 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:35:16.411+0000 7efeec86a640 1 --2- 172.21.15.114:0/4263870965 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x55a35ff4bf40 0x55a35ff562c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:35:16.415 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:35:16.411+0000 7efeec86a640 1 -- 172.21.15.114:0/4263870965 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55a35ff56800 msgr2=0x55a35ff58bf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T10:35:16.415 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:35:16.411+0000 7efeec86a640 1 --2- 172.21.15.114:0/4263870965 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55a35ff56800 0x55a35ff58bf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:35:16.415 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:35:16.411+0000 7efeec86a640 1 -- 172.21.15.114:0/4263870965 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55a35fbc3210 con 0x55a35ff4b620 2024-04-23T10:35:16.415 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:35:16.411+0000 7efeec86a640 1 --2- 172.21.15.114:0/4263870965 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x55a35ff4b620 0x55a35ff4ba00 secure :-1 s=READY pgs=89 cs=0 l=1 rev1=1 crypto rx=0x7efee4009560 tx=0x7efee40046c0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=8f2e77ab2aad76d7 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T10:35:16.415 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:35:16.411+0000 7efeed86c640 1 --2- 172.21.15.114:0/2851140285 >> [v2:172.21.15.114:6800/1395693466,v1:172.21.15.114:6801/1395693466] conn(0x7efed40a3dc0 0x7efed40a6280 secure :-1 s=READY pgs=123 cs=0 l=1 rev1=1 crypto rx=0x7efee00ef4c0 tx=0x7efee008f6a0 comp rx=0 tx=0).ready entity=mgr.15291 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T10:35:16.415 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:35:16.411+0000 7efed37fe640 1 -- 172.21.15.114:0/4263870965 <== mon.2 v2:172.21.15.119:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7efee4017cb0 con 0x55a35ff4b620 2024-04-23T10:35:16.416 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:35:16.411+0000 7efed37fe640 1 -- 172.21.15.114:0/4263870965 <== mon.2 v2:172.21.15.119:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7efee4017e50 con 0x55a35ff4b620 2024-04-23T10:35:16.416 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:35:16.411+0000 7efed37fe640 1 -- 172.21.15.114:0/4263870965 <== mon.2 v2:172.21.15.119:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7efee4011aa0 con 0x55a35ff4b620 2024-04-23T10:35:16.416 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:35:16.412+0000 7efef1feb740 1 -- 172.21.15.114:0/4263870965 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x55a35ff4b620 msgr2=0x55a35ff4ba00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T10:35:16.416 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:35:16.412+0000 7efef1feb740 1 --2- 172.21.15.114:0/4263870965 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x55a35ff4b620 0x55a35ff4ba00 secure :-1 s=READY pgs=89 cs=0 l=1 rev1=1 crypto rx=0x7efee4009560 tx=0x7efee40046c0 comp rx=0 tx=0).stop 2024-04-23T10:35:16.416 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:35:16.412+0000 7efef1feb740 1 -- 172.21.15.114:0/4263870965 shutdown_connections 2024-04-23T10:35:16.416 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:35:16.412+0000 7efef1feb740 1 --2- 172.21.15.114:0/4263870965 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55a35ff56800 0x55a35ff58bf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:35:16.416 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:35:16.412+0000 7efef1feb740 1 --2- 172.21.15.114:0/4263870965 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x55a35ff4bf40 0x55a35ff562c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:35:16.416 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:35:16.412+0000 7efef1feb740 1 --2- 172.21.15.114:0/4263870965 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x55a35ff4b620 0x55a35ff4ba00 unknown :-1 s=CLOSED pgs=89 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:35:16.416 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:35:16.412+0000 7efef1feb740 1 -- 172.21.15.114:0/4263870965 >> 172.21.15.114:0/4263870965 conn(0x55a35ff47b30 msgr2=0x55a35ff47f10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T10:35:16.416 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:35:16.412+0000 7efef1feb740 1 -- 172.21.15.114:0/4263870965 shutdown_connections 2024-04-23T10:35:16.416 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:35:16.412+0000 7efef1feb740 1 -- 172.21.15.114:0/4263870965 wait complete. 2024-04-23T10:35:16.416 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:35:16.412+0000 7efef1feb740 1 -- start start 2024-04-23T10:35:16.416 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:35:16.413+0000 7efef1feb740 1 Processor -- start 2024-04-23T10:35:16.416 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:35:16.413+0000 7efef1feb740 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55a35ff4b620 0x55a35ff56190 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T10:35:16.416 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:35:16.413+0000 7efef1feb740 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x55a35ff4bf40 0x55a35ff53480 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T10:35:16.416 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:35:16.413+0000 7efef1feb740 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x55a35ff56800 0x55a35ff53a00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T10:35:16.417 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:35:16.413+0000 7efef1feb740 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x55a35ff59430 con 0x55a35ff4b620 2024-04-23T10:35:16.417 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:35:16.413+0000 7efef1feb740 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 -- 0x55a35fe4b0b0 con 0x55a35ff4bf40 2024-04-23T10:35:16.417 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:35:16.413+0000 7efef1feb740 1 -- --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_getmap magic: 0 -- 0x55a35ff595b0 con 0x55a35ff56800 2024-04-23T10:35:16.417 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:35:16.413+0000 7efef1feb740 10 client.0 _ll_register_callbacks cb 0x55a35fe38d30 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-23T10:35:16.417 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:35:16.413+0000 7efeec86a640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55a35ff4b620 0x55a35ff56190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T10:35:16.417 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:35:16.413+0000 7efeec86a640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55a35ff4b620 0x55a35ff56190 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.114:57542/0 (socket says 172.21.15.114:57542) 2024-04-23T10:35:16.417 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:35:16.413+0000 7efeec86a640 1 -- 172.21.15.114:0/2185354512 learned_addr learned my addr 172.21.15.114:0/2185354512 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-04-23T10:35:16.417 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:35:16.413+0000 7efeed06b640 1 --2- 172.21.15.114:0/2185354512 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x55a35ff4bf40 0x55a35ff53480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T10:35:16.417 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:35:16.413+0000 7efeec86a640 1 --2- 172.21.15.114:0/2185354512 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x55a35ff56800 0x55a35ff53a00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T10:35:16.417 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:35:16.413+0000 7efeec86a640 1 -- 172.21.15.114:0/2185354512 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x55a35ff56800 msgr2=0x55a35ff53a00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T10:35:16.417 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:35:16.413+0000 7efeec86a640 1 --2- 172.21.15.114:0/2185354512 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x55a35ff56800 0x55a35ff53a00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:35:16.417 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:35:16.413+0000 7efeec86a640 1 -- 172.21.15.114:0/2185354512 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x55a35ff4bf40 msgr2=0x55a35ff53480 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T10:35:16.417 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:35:16.413+0000 7efeec86a640 1 --2- 172.21.15.114:0/2185354512 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x55a35ff4bf40 0x55a35ff53480 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:35:16.417 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:35:16.413+0000 7efeec86a640 1 -- 172.21.15.114:0/2185354512 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55a35fc8f620 con 0x55a35ff4b620 2024-04-23T10:35:16.417 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:35:16.414+0000 7efeed06b640 1 --2- 172.21.15.114:0/2185354512 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x55a35ff4bf40 0x55a35ff53480 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-23T10:35:16.418 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:35:16.414+0000 7efeec86a640 1 --2- 172.21.15.114:0/2185354512 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55a35ff4b620 0x55a35ff56190 secure :-1 s=READY pgs=803 cs=0 l=1 rev1=1 crypto rx=0x7efee40045e0 tx=0x7efee4005710 comp rx=0 tx=0).ready entity=mon.0 client_cookie=810ebc16d88431ed server_cookie=0 in_seq=0 out_seq=0 2024-04-23T10:35:16.418 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:35:16.414+0000 7efed0ff9640 10 client.0 ms_handle_connect on v2:172.21.15.114:3300/0 2024-04-23T10:35:16.418 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:35:16.414+0000 7efed0ff9640 1 -- 172.21.15.114:0/2185354512 <== mon.0 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7efee4005460 con 0x55a35ff4b620 2024-04-23T10:35:16.418 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:35:16.414+0000 7efed0ff9640 1 -- 172.21.15.114:0/2185354512 <== mon.0 v2:172.21.15.114:3300/0 2 ==== osd_map(100..100 src has 1..100) ==== 8629+0+0 (secure 0 0 0) 0x7efee4017050 con 0x55a35ff4b620 2024-04-23T10:35:16.418 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:35:16.414+0000 7efef1feb740 1 -- 172.21.15.114:0/2185354512 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7efee4017e50 con 0x55a35ff4b620 2024-04-23T10:35:16.418 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:35:16.414+0000 7efed0ff9640 1 -- 172.21.15.114:0/2185354512 <== mon.0 v2:172.21.15.114:3300/0 3 ==== mon_get_version_reply(handle=1 version=32) ==== 24+0+0 (secure 0 0 0) 0x7efee4037050 con 0x55a35ff4b620 2024-04-23T10:35:16.418 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:35:16.414+0000 7efef1feb740 10 client.16656 fetch_fsmap learned FSMap version 32 2024-04-23T10:35:16.418 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:35:16.414+0000 7efef1feb740 1 -- 172.21.15.114:0/2185354512 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({config=0+,fsmap=32,monmap=4+}) -- 0x55a35ff653c0 con 0x55a35ff4b620 2024-04-23T10:35:16.418 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:35:16.414+0000 7efed0ff9640 1 -- 172.21.15.114:0/2185354512 <== mon.0 v2:172.21.15.114:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7efee4017e50 con 0x55a35ff4b620 2024-04-23T10:35:16.418 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:35:16.414+0000 7efed0ff9640 1 -- 172.21.15.114:0/2185354512 <== mon.0 v2:172.21.15.114:3300/0 5 ==== fsmap(e 32) ==== 8338+0+0 (secure 0 0 0) 0x7efee40048f0 con 0x55a35ff4b620 2024-04-23T10:35:16.418 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:35:16.414+0000 7efef1feb740 10 client.16656 fetch_fsmap finished waiting for FSMap version 32 2024-04-23T10:35:16.418 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:35:16.414+0000 7efef1feb740 10 client.16656 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-04-23T10:35:16.418 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:35:16.414+0000 7efef1feb740 20 client.16656 populate_metadata read hostname 'smithi114' 2024-04-23T10:35:16.418 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:35:16.414+0000 7efef1feb740 1 --2- 172.21.15.114:0/2185354512 >> [v2:172.21.15.119:6838/937297340,v1:172.21.15.119:6839/937297340] conn(0x55a35ff66710 0x55a35ff68b00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T10:35:16.418 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:35:16.414+0000 7efef1feb740 4 client.16656 mds_command: new command op to 24545 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-23T10:35:16.419 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:35:16.414+0000 7efef1feb740 1 -- 172.21.15.114:0/2185354512 --> [v2:172.21.15.119:6838/937297340,v1:172.21.15.119:6839/937297340] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55a35febe390 con 0x55a35ff66710 2024-04-23T10:35:16.419 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:35:16.415+0000 7efeed06b640 1 --2- 172.21.15.114:0/2185354512 >> [v2:172.21.15.119:6838/937297340,v1:172.21.15.119:6839/937297340] conn(0x55a35ff66710 0x55a35ff68b00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T10:35:16.419 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:35:16.415+0000 7efeed06b640 1 --2- 172.21.15.114:0/2185354512 >> [v2:172.21.15.119:6838/937297340,v1:172.21.15.119:6839/937297340] conn(0x55a35ff66710 0x55a35ff68b00 crc :-1 s=READY pgs=138 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=16d1babfeb30948f server_cookie=b0273178f9a7f9db in_seq=0 out_seq=0 2024-04-23T10:35:16.419 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:35:16.415+0000 7efed0ff9640 10 client.16656 ms_handle_connect on v2:172.21.15.119:6838/937297340 2024-04-23T10:35:16.420 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:35:16.416+0000 7efed0ff9640 1 -- 172.21.15.114:0/2185354512 <== mds.0 v2:172.21.15.119:6838/937297340 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x55a35febe390 con 0x55a35ff66710 2024-04-23T10:35:16.420 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:35:16.416+0000 7efed0ff9640 10 client.16656 handle_command_reply: tid=0 2024-04-23T10:35:16.420 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:35:16.416+0000 7efef1feb740 1 client.16656 shutdown 2024-04-23T10:35:16.420 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:35:16.416+0000 7efef1feb740 1 -- 172.21.15.114:0/2185354512 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55a35ff4b620 msgr2=0x55a35ff56190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T10:35:16.420 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:35:16.416+0000 7efef1feb740 1 --2- 172.21.15.114:0/2185354512 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55a35ff4b620 0x55a35ff56190 secure :-1 s=READY pgs=803 cs=0 l=1 rev1=1 crypto rx=0x7efee40045e0 tx=0x7efee4005710 comp rx=0 tx=0).stop 2024-04-23T10:35:16.420 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:35:16.416+0000 7efef1feb740 1 -- 172.21.15.114:0/2185354512 shutdown_connections 2024-04-23T10:35:16.420 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:35:16.416+0000 7efef1feb740 1 --2- 172.21.15.114:0/2185354512 >> [v2:172.21.15.119:6838/937297340,v1:172.21.15.119:6839/937297340] conn(0x55a35ff66710 0x55a35ff68b00 crc :-1 s=READY pgs=138 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:35:16.420 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:35:16.416+0000 7efef1feb740 1 --2- 172.21.15.114:0/2185354512 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x55a35ff56800 0x55a35ff53a00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:35:16.420 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:35:16.416+0000 7efef1feb740 1 --2- 172.21.15.114:0/2185354512 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x55a35ff4bf40 0x55a35ff53480 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:35:16.420 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:35:16.416+0000 7efed0ff9640 0 client.16656 ms_handle_reset on v2:172.21.15.119:6838/937297340 2024-04-23T10:35:16.420 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:35:16.416+0000 7efef1feb740 1 --2- 172.21.15.114:0/2185354512 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55a35ff4b620 0x55a35ff56190 unknown :-1 s=CLOSED pgs=803 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:35:16.421 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:35:16.416+0000 7efef1feb740 1 -- 172.21.15.114:0/2185354512 >> 172.21.15.114:0/2185354512 conn(0x55a35ff47b30 msgr2=0x55a35ff4d860 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T10:35:16.421 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:35:16.416+0000 7efef1feb740 1 -- 172.21.15.114:0/2185354512 shutdown_connections 2024-04-23T10:35:16.421 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:35:16.416+0000 7efef1feb740 1 -- 172.21.15.114:0/2185354512 wait complete. 2024-04-23T10:35:16.421 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:35:16.416+0000 7efef1feb740 20 client.16656 trim_cache size 0 max 16384 2024-04-23T10:35:16.435 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:35:16.431+0000 7efef1feb740 1 Processor -- start 2024-04-23T10:35:16.435 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:35:16.431+0000 7efef1feb740 1 -- start start 2024-04-23T10:35:16.435 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:35:16.431+0000 7efef1feb740 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55a35ff4b620 0x55a35ff622f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T10:35:16.435 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:35:16.431+0000 7efef1feb740 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x55a35ff4bf40 0x55a35ff46b30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T10:35:16.435 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:35:16.431+0000 7efef1feb740 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x55a35ff56800 0x55a35ff55cc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T10:35:16.435 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:35:16.431+0000 7efef1feb740 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x7efee81ae320 con 0x55a35ff4b620 2024-04-23T10:35:16.435 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:35:16.431+0000 7efef1feb740 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 -- 0x7efee81abbf0 con 0x55a35ff4bf40 2024-04-23T10:35:16.435 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:35:16.431+0000 7efef1feb740 1 -- --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_getmap magic: 0 -- 0x7efee81a9a80 con 0x55a35ff56800 2024-04-23T10:35:16.435 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:35:16.431+0000 7efeec86a640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55a35ff4b620 0x55a35ff622f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T10:35:16.435 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:35:16.431+0000 7efeec86a640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55a35ff4b620 0x55a35ff622f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.114:57558/0 (socket says 172.21.15.114:57558) 2024-04-23T10:35:16.436 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:35:16.431+0000 7efeec86a640 1 -- 172.21.15.114:0/1227619982 learned_addr learned my addr 172.21.15.114:0/1227619982 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-04-23T10:35:16.436 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:35:16.431+0000 7efeed06b640 1 --2- 172.21.15.114:0/1227619982 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x55a35ff4bf40 0x55a35ff46b30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T10:35:16.436 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:35:16.432+0000 7efeec86a640 1 --2- 172.21.15.114:0/1227619982 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x55a35ff56800 0x55a35ff55cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T10:35:16.436 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:35:16.432+0000 7efeec86a640 1 -- 172.21.15.114:0/1227619982 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x55a35ff56800 msgr2=0x55a35ff55cc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T10:35:16.436 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:35:16.432+0000 7efeec86a640 1 --2- 172.21.15.114:0/1227619982 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x55a35ff56800 0x55a35ff55cc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:35:16.436 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:35:16.432+0000 7efeec86a640 1 -- 172.21.15.114:0/1227619982 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x55a35ff4bf40 msgr2=0x55a35ff46b30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T10:35:16.436 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:35:16.432+0000 7efeec86a640 1 --2- 172.21.15.114:0/1227619982 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x55a35ff4bf40 0x55a35ff46b30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:35:16.436 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:35:16.432+0000 7efeec86a640 1 -- 172.21.15.114:0/1227619982 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55a35ff52450 con 0x55a35ff4b620 2024-04-23T10:35:16.436 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:35:16.432+0000 7efeed06b640 1 --2- 172.21.15.114:0/1227619982 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x55a35ff4bf40 0x55a35ff46b30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-23T10:35:16.436 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:35:16.432+0000 7efeec86a640 1 --2- 172.21.15.114:0/1227619982 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55a35ff4b620 0x55a35ff622f0 secure :-1 s=READY pgs=804 cs=0 l=1 rev1=1 crypto rx=0x7efee4004e60 tx=0x7efee4005d90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a81a60989710af34 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T10:35:16.436 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:35:16.432+0000 7efed2ffd640 1 -- 172.21.15.114:0/1227619982 <== mon.0 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7efee4005460 con 0x55a35ff4b620 2024-04-23T10:35:16.436 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:35:16.432+0000 7efed2ffd640 1 -- 172.21.15.114:0/1227619982 <== mon.0 v2:172.21.15.114:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7efee40097a0 con 0x55a35ff4b620 2024-04-23T10:35:16.436 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:35:16.432+0000 7efed2ffd640 1 -- 172.21.15.114:0/1227619982 <== mon.0 v2:172.21.15.114:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7efee4011a70 con 0x55a35ff4b620 2024-04-23T10:35:16.436 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:35:16.432+0000 7efef1feb740 1 -- 172.21.15.114:0/1227619982 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55a35ff4b620 msgr2=0x55a35ff622f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T10:35:16.436 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:35:16.432+0000 7efef1feb740 1 --2- 172.21.15.114:0/1227619982 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55a35ff4b620 0x55a35ff622f0 secure :-1 s=READY pgs=804 cs=0 l=1 rev1=1 crypto rx=0x7efee4004e60 tx=0x7efee4005d90 comp rx=0 tx=0).stop 2024-04-23T10:35:16.437 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:35:16.433+0000 7efef1feb740 1 -- 172.21.15.114:0/1227619982 shutdown_connections 2024-04-23T10:35:16.437 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:35:16.433+0000 7efef1feb740 1 --2- 172.21.15.114:0/1227619982 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x55a35ff56800 0x55a35ff55cc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:35:16.437 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:35:16.433+0000 7efef1feb740 1 --2- 172.21.15.114:0/1227619982 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x55a35ff4bf40 0x55a35ff46b30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:35:16.437 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:35:16.433+0000 7efef1feb740 1 --2- 172.21.15.114:0/1227619982 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55a35ff4b620 0x55a35ff622f0 unknown :-1 s=CLOSED pgs=804 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:35:16.437 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:35:16.433+0000 7efef1feb740 1 -- 172.21.15.114:0/1227619982 >> 172.21.15.114:0/1227619982 conn(0x55a35ff47b30 msgr2=0x55a35ff46420 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T10:35:16.437 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:35:16.433+0000 7efef1feb740 1 -- 172.21.15.114:0/1227619982 shutdown_connections 2024-04-23T10:35:16.437 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:35:16.433+0000 7efef1feb740 1 -- 172.21.15.114:0/1227619982 wait complete. 2024-04-23T10:35:16.437 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:35:16.433+0000 7efef1feb740 1 -- start start 2024-04-23T10:35:16.437 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:35:16.433+0000 7efef1feb740 1 Processor -- start 2024-04-23T10:35:16.437 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:35:16.433+0000 7efef1feb740 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55a35ff4b620 0x55a35ff5e9f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T10:35:16.437 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:35:16.433+0000 7efef1feb740 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x55a35ff4bf40 0x55a35ff4fae0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T10:35:16.437 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:35:16.433+0000 7efef1feb740 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x55a35ff56800 0x55a35ff50020 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T10:35:16.437 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:35:16.433+0000 7efef1feb740 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x7efee81aa0d0 con 0x55a35ff4b620 2024-04-23T10:35:16.437 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:35:16.433+0000 7efef1feb740 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 -- 0x7efee81ab520 con 0x55a35ff56800 2024-04-23T10:35:16.437 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:35:16.433+0000 7efef1feb740 1 -- --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_getmap magic: 0 -- 0x7efee81aa780 con 0x55a35ff4bf40 2024-04-23T10:35:16.437 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:35:16.433+0000 7efef1feb740 10 client.0 _ll_register_callbacks cb 0x7efee40044a0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-23T10:35:16.438 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:35:16.433+0000 7efeec86a640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55a35ff4b620 0x55a35ff5e9f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T10:35:16.438 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:35:16.434+0000 7efeec86a640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55a35ff4b620 0x55a35ff5e9f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.114:57568/0 (socket says 172.21.15.114:57568) 2024-04-23T10:35:16.438 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:35:16.434+0000 7efeec86a640 1 -- 172.21.15.114:0/1547288496 learned_addr learned my addr 172.21.15.114:0/1547288496 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-04-23T10:35:16.438 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:35:16.434+0000 7efeed06b640 1 --2- 172.21.15.114:0/1547288496 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x55a35ff4bf40 0x55a35ff4fae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T10:35:16.438 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:35:16.434+0000 7efeec86a640 1 --2- 172.21.15.114:0/1547288496 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x55a35ff56800 0x55a35ff50020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T10:35:16.438 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:35:16.434+0000 7efeec86a640 1 -- 172.21.15.114:0/1547288496 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x55a35ff4bf40 msgr2=0x55a35ff4fae0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T10:35:16.438 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:35:16.434+0000 7efeec86a640 1 --2- 172.21.15.114:0/1547288496 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x55a35ff4bf40 0x55a35ff4fae0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:35:16.438 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:35:16.434+0000 7efeec86a640 1 -- 172.21.15.114:0/1547288496 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x55a35ff56800 msgr2=0x55a35ff50020 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T10:35:16.438 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:35:16.434+0000 7efeec86a640 1 --2- 172.21.15.114:0/1547288496 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x55a35ff56800 0x55a35ff50020 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:35:16.438 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:35:16.434+0000 7efeec86a640 1 -- 172.21.15.114:0/1547288496 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55a35ff53d70 con 0x55a35ff4b620 2024-04-23T10:35:16.438 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:35:16.434+0000 7efeed06b640 1 --2- 172.21.15.114:0/1547288496 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x55a35ff4bf40 0x55a35ff4fae0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-23T10:35:16.438 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:35:16.434+0000 7efeec86a640 1 --2- 172.21.15.114:0/1547288496 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55a35ff4b620 0x55a35ff5e9f0 secure :-1 s=READY pgs=805 cs=0 l=1 rev1=1 crypto rx=0x7efee4009f10 tx=0x7efee4009ea0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=915a84f1d634a498 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T10:35:16.438 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:35:16.434+0000 7efed0ff9640 10 client.0 ms_handle_connect on v2:172.21.15.114:3300/0 2024-04-23T10:35:16.438 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:35:16.434+0000 7efed0ff9640 1 -- 172.21.15.114:0/1547288496 <== mon.0 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7efee4017240 con 0x55a35ff4b620 2024-04-23T10:35:16.438 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:35:16.434+0000 7efed0ff9640 1 -- 172.21.15.114:0/1547288496 <== mon.0 v2:172.21.15.114:3300/0 2 ==== osd_map(100..100 src has 1..100) ==== 8629+0+0 (secure 0 0 0) 0x7efee4011810 con 0x55a35ff4b620 2024-04-23T10:35:16.439 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:35:16.434+0000 7efef1feb740 1 -- 172.21.15.114:0/1547288496 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7efee40097a0 con 0x55a35ff4b620 2024-04-23T10:35:16.439 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:35:16.434+0000 7efed0ff9640 1 -- 172.21.15.114:0/1547288496 <== mon.0 v2:172.21.15.114:3300/0 3 ==== mon_get_version_reply(handle=1 version=32) ==== 24+0+0 (secure 0 0 0) 0x7efee40173e0 con 0x55a35ff4b620 2024-04-23T10:35:16.439 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:35:16.434+0000 7efef1feb740 10 client.16662 fetch_fsmap learned FSMap version 32 2024-04-23T10:35:16.439 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:35:16.434+0000 7efef1feb740 1 -- 172.21.15.114:0/1547288496 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({config=0+,fsmap=32,monmap=4+}) -- 0x55a35ff68e70 con 0x55a35ff4b620 2024-04-23T10:35:16.439 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:35:16.434+0000 7efed0ff9640 1 -- 172.21.15.114:0/1547288496 <== mon.0 v2:172.21.15.114:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7efee40097a0 con 0x55a35ff4b620 2024-04-23T10:35:16.439 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:35:16.435+0000 7efed0ff9640 1 -- 172.21.15.114:0/1547288496 <== mon.0 v2:172.21.15.114:3300/0 5 ==== fsmap(e 32) ==== 8338+0+0 (secure 0 0 0) 0x7efee4003ba0 con 0x55a35ff4b620 2024-04-23T10:35:16.439 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:35:16.435+0000 7efef1feb740 10 client.16662 fetch_fsmap finished waiting for FSMap version 32 2024-04-23T10:35:16.439 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:35:16.435+0000 7efef1feb740 10 client.16662 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-04-23T10:35:16.439 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:35:16.435+0000 7efef1feb740 20 client.16662 populate_metadata read hostname 'smithi114' 2024-04-23T10:35:16.439 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:35:16.435+0000 7efef1feb740 1 --2- 172.21.15.114:0/1547288496 >> [v2:172.21.15.119:6838/937297340,v1:172.21.15.119:6839/937297340] conn(0x55a35ffcc9e0 0x55a35ffcedd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T10:35:16.439 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:35:16.435+0000 7efef1feb740 4 client.16662 mds_command: new command op to 24545 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-04-23T10:35:16.439 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:35:16.435+0000 7efef1feb740 1 -- 172.21.15.114:0/1547288496 --> [v2:172.21.15.119:6838/937297340,v1:172.21.15.119:6839/937297340] -- command(tid 0: {"prefix": "flush journal"}) -- 0x55a35feb8200 con 0x55a35ffcc9e0 2024-04-23T10:35:16.439 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:35:16.435+0000 7efeed06b640 1 --2- 172.21.15.114:0/1547288496 >> [v2:172.21.15.119:6838/937297340,v1:172.21.15.119:6839/937297340] conn(0x55a35ffcc9e0 0x55a35ffcedd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T10:35:16.439 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:35:16.435+0000 7efeed06b640 1 --2- 172.21.15.114:0/1547288496 >> [v2:172.21.15.119:6838/937297340,v1:172.21.15.119:6839/937297340] conn(0x55a35ffcc9e0 0x55a35ffcedd0 crc :-1 s=READY pgs=139 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=c21a2de0010e3a57 server_cookie=830dba4ce84f5195 in_seq=0 out_seq=0 2024-04-23T10:35:16.439 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:35:16.435+0000 7efed0ff9640 10 client.16662 ms_handle_connect on v2:172.21.15.119:6838/937297340 2024-04-23T10:35:16.493 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:35:16.493+0000 7efed0ff9640 1 -- 172.21.15.114:0/1547288496 <== mds.0 v2:172.21.15.119:6838/937297340 1 ==== command_reply(tid 0: 0 ) ==== 8+0+44 (crc 0 0 0) 0x55a35feb8200 con 0x55a35ffcc9e0 2024-04-23T10:35:16.493 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:35:16.493+0000 7efed0ff9640 10 client.16662 handle_command_reply: tid=0 2024-04-23T10:35:16.493 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:35:16.493+0000 7efef1feb740 1 client.16662 shutdown 2024-04-23T10:35:16.493 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:35:16.493+0000 7efef1feb740 1 -- 172.21.15.114:0/1547288496 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55a35ff4b620 msgr2=0x55a35ff5e9f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T10:35:16.493 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:35:16.493+0000 7efef1feb740 1 --2- 172.21.15.114:0/1547288496 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55a35ff4b620 0x55a35ff5e9f0 secure :-1 s=READY pgs=805 cs=0 l=1 rev1=1 crypto rx=0x7efee4009f10 tx=0x7efee4009ea0 comp rx=0 tx=0).stop 2024-04-23T10:35:16.493 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:35:16.493+0000 7efef1feb740 1 -- 172.21.15.114:0/1547288496 shutdown_connections 2024-04-23T10:35:16.493 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:35:16.493+0000 7efef1feb740 1 --2- 172.21.15.114:0/1547288496 >> [v2:172.21.15.119:6838/937297340,v1:172.21.15.119:6839/937297340] conn(0x55a35ffcc9e0 0x55a35ffcedd0 crc :-1 s=READY pgs=139 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:35:16.493 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:35:16.493+0000 7efed0ff9640 0 client.16662 ms_handle_reset on v2:172.21.15.119:6838/937297340 2024-04-23T10:35:16.493 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:35:16.493+0000 7efef1feb740 1 --2- 172.21.15.114:0/1547288496 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x55a35ff56800 0x55a35ff50020 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:35:16.493 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:35:16.493+0000 7efef1feb740 1 --2- 172.21.15.114:0/1547288496 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x55a35ff4bf40 0x55a35ff4fae0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:35:16.493 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:35:16.493+0000 7efef1feb740 1 --2- 172.21.15.114:0/1547288496 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55a35ff4b620 0x55a35ff5e9f0 unknown :-1 s=CLOSED pgs=805 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:35:16.493 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:35:16.493+0000 7efef1feb740 1 -- 172.21.15.114:0/1547288496 >> 172.21.15.114:0/1547288496 conn(0x55a35ff47b30 msgr2=0x55a35ff4e9d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T10:35:16.493 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:35:16.493+0000 7efef1feb740 1 -- 172.21.15.114:0/1547288496 shutdown_connections 2024-04-23T10:35:16.493 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:35:16.493+0000 7efef1feb740 1 -- 172.21.15.114:0/1547288496 wait complete. 2024-04-23T10:35:16.494 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:35:16.493+0000 7efef1feb740 20 client.16662 trim_cache size 0 max 16384 2024-04-23T10:35:16.494 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stdout:{ 2024-04-23T10:35:16.494 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stdout: "message": "", 2024-04-23T10:35:16.494 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stdout: "return_code": 0 2024-04-23T10:35:16.494 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stdout:} 2024-04-23T10:35:16.495 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:35:16.494+0000 7efeef2cf640 1 -- 172.21.15.114:0/2851140285 >> [v2:172.21.15.114:6800/1395693466,v1:172.21.15.114:6801/1395693466] conn(0x7efed40a3dc0 msgr2=0x7efed40a6280 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T10:35:16.495 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:35:16.494+0000 7efeef2cf640 1 --2- 172.21.15.114:0/2851140285 >> [v2:172.21.15.114:6800/1395693466,v1:172.21.15.114:6801/1395693466] conn(0x7efed40a3dc0 0x7efed40a6280 secure :-1 s=READY pgs=123 cs=0 l=1 rev1=1 crypto rx=0x7efee00ef4c0 tx=0x7efee008f6a0 comp rx=0 tx=0).stop 2024-04-23T10:35:16.495 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:35:16.494+0000 7efeef2cf640 1 -- 172.21.15.114:0/2851140285 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7efee819ce80 msgr2=0x7efee819d330 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T10:35:16.495 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:35:16.494+0000 7efeef2cf640 1 --2- 172.21.15.114:0/2851140285 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7efee819ce80 0x7efee819d330 secure :-1 s=READY pgs=802 cs=0 l=1 rev1=1 crypto rx=0x7efee00098d0 tx=0x7efee00317b0 comp rx=0 tx=0).stop 2024-04-23T10:35:16.495 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:35:16.494+0000 7efeef2cf640 1 -- 172.21.15.114:0/2851140285 shutdown_connections 2024-04-23T10:35:16.495 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:35:16.494+0000 7efeef2cf640 1 --2- 172.21.15.114:0/2851140285 >> [v2:172.21.15.114:6800/1395693466,v1:172.21.15.114:6801/1395693466] conn(0x7efed40a3dc0 0x7efed40a6280 unknown :-1 s=CLOSED pgs=123 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:35:16.495 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:35:16.494+0000 7efeef2cf640 1 --2- 172.21.15.114:0/2851140285 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7efee819ce80 0x7efee819d330 unknown :-1 s=CLOSED pgs=802 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:35:16.495 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:35:16.494+0000 7efeef2cf640 1 --2- 172.21.15.114:0/2851140285 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7efee8107290 0x7efee8198b00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:35:16.495 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:35:16.494+0000 7efeef2cf640 1 --2- 172.21.15.114:0/2851140285 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7efee81021c0 0x7efee81985c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:35:16.495 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:35:16.494+0000 7efeef2cf640 1 -- 172.21.15.114:0/2851140285 >> 172.21.15.114:0/2851140285 conn(0x7efee80fda20 msgr2=0x7efee80fe4e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T10:35:16.495 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:35:16.494+0000 7efeef2cf640 1 -- 172.21.15.114:0/2851140285 shutdown_connections 2024-04-23T10:35:16.495 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:35:16.494+0000 7efeef2cf640 1 -- 172.21.15.114:0/2851140285 wait complete. 2024-04-23T10:35:16.931 INFO:tasks.workunit.client.0.smithi114.stdout: 1 617926 7.35 MB/sec execute 325 sec latency 8357.495 ms 2024-04-23T10:35:17.828 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 23 10:35:17 smithi204 ceph-mon[32649]: pgmap v271: 97 pgs: 97 active+clean; 203 MiB data, 2.3 GiB used, 1.0 TiB / 1.0 TiB avail; 572 KiB/s rd, 2.4 MiB/s wr, 110 op/s 2024-04-23T10:35:17.932 INFO:tasks.workunit.client.0.smithi114.stdout: 1 617926 7.33 MB/sec execute 326 sec latency 9357.622 ms 2024-04-23T10:35:17.992 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 23 10:35:17 smithi119 ceph-mon[32608]: pgmap v271: 97 pgs: 97 active+clean; 203 MiB data, 2.3 GiB used, 1.0 TiB / 1.0 TiB avail; 572 KiB/s rd, 2.4 MiB/s wr, 110 op/s 2024-04-23T10:35:18.004 INFO:journalctl@ceph.mon.a.smithi114.stdout:Apr 23 10:35:17 smithi114 ceph-mon[29255]: pgmap v271: 97 pgs: 97 active+clean; 203 MiB data, 2.3 GiB used, 1.0 TiB / 1.0 TiB avail; 572 KiB/s rd, 2.4 MiB/s wr, 110 op/s 2024-04-23T10:35:18.932 INFO:tasks.workunit.client.0.smithi114.stdout: 1 617926 7.31 MB/sec execute 327 sec latency 10357.772 ms 2024-04-23T10:35:19.806 INFO:journalctl@ceph.mon.a.smithi114.stdout:Apr 23 10:35:19 smithi114 ceph-mon[29255]: pgmap v272: 97 pgs: 97 active+clean; 203 MiB data, 2.3 GiB used, 1.0 TiB / 1.0 TiB avail; 572 KiB/s rd, 2.4 MiB/s wr, 110 op/s 2024-04-23T10:35:19.827 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 23 10:35:19 smithi204 ceph-mon[32649]: pgmap v272: 97 pgs: 97 active+clean; 203 MiB data, 2.3 GiB used, 1.0 TiB / 1.0 TiB avail; 572 KiB/s rd, 2.4 MiB/s wr, 110 op/s 2024-04-23T10:35:19.932 INFO:tasks.workunit.client.0.smithi114.stdout: 1 618603 7.29 MB/sec execute 328 sec latency 10637.376 ms 2024-04-23T10:35:19.991 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 23 10:35:19 smithi119 ceph-mon[32608]: pgmap v272: 97 pgs: 97 active+clean; 203 MiB data, 2.3 GiB used, 1.0 TiB / 1.0 TiB avail; 572 KiB/s rd, 2.4 MiB/s wr, 110 op/s 2024-04-23T10:35:20.932 INFO:tasks.workunit.client.0.smithi114.stdout: 1 620995 7.31 MB/sec execute 329 sec latency 21.127 ms 2024-04-23T10:35:21.827 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 23 10:35:21 smithi204 ceph-mon[32649]: pgmap v273: 97 pgs: 97 active+clean; 203 MiB data, 2.3 GiB used, 1.0 TiB / 1.0 TiB avail; 478 KiB/s rd, 2.0 MiB/s wr, 92 op/s 2024-04-23T10:35:21.932 INFO:tasks.workunit.client.0.smithi114.stdout: 1 622194 7.31 MB/sec execute 330 sec latency 41.336 ms 2024-04-23T10:35:21.991 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 23 10:35:21 smithi119 ceph-mon[32608]: pgmap v273: 97 pgs: 97 active+clean; 203 MiB data, 2.3 GiB used, 1.0 TiB / 1.0 TiB avail; 478 KiB/s rd, 2.0 MiB/s wr, 92 op/s 2024-04-23T10:35:22.003 INFO:journalctl@ceph.mon.a.smithi114.stdout:Apr 23 10:35:21 smithi114 ceph-mon[29255]: pgmap v273: 97 pgs: 97 active+clean; 203 MiB data, 2.3 GiB used, 1.0 TiB / 1.0 TiB avail; 478 KiB/s rd, 2.0 MiB/s wr, 92 op/s 2024-04-23T10:35:22.932 INFO:tasks.workunit.client.0.smithi114.stdout: 1 624403 7.32 MB/sec execute 331 sec latency 68.124 ms 2024-04-23T10:35:23.827 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 23 10:35:23 smithi204 ceph-mon[32649]: pgmap v274: 97 pgs: 97 active+clean; 191 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail; 790 KiB/s rd, 3.1 MiB/s wr, 181 op/s 2024-04-23T10:35:23.932 INFO:tasks.workunit.client.0.smithi114.stdout: 1 625799 7.32 MB/sec execute 332 sec latency 20.835 ms 2024-04-23T10:35:23.991 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 23 10:35:23 smithi119 ceph-mon[32608]: pgmap v274: 97 pgs: 97 active+clean; 191 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail; 790 KiB/s rd, 3.1 MiB/s wr, 181 op/s 2024-04-23T10:35:24.003 INFO:journalctl@ceph.mon.a.smithi114.stdout:Apr 23 10:35:23 smithi114 ceph-mon[29255]: pgmap v274: 97 pgs: 97 active+clean; 191 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail; 790 KiB/s rd, 3.1 MiB/s wr, 181 op/s 2024-04-23T10:35:24.932 INFO:tasks.workunit.client.0.smithi114.stdout: 1 628097 7.33 MB/sec execute 333 sec latency 20.641 ms 2024-04-23T10:35:25.827 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 23 10:35:25 smithi204 ceph-mon[32649]: pgmap v275: 97 pgs: 97 active+clean; 191 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail; 382 KiB/s rd, 2.1 MiB/s wr, 122 op/s 2024-04-23T10:35:25.828 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 23 10:35:25 smithi204 ceph-mon[32649]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=e=up:active} 3 up:standby-replay 6 up:standby 2024-04-23T10:35:25.932 INFO:tasks.workunit.client.0.smithi114.stdout: 1 628563 7.33 MB/sec execute 334 sec latency 903.157 ms 2024-04-23T10:35:25.992 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 23 10:35:25 smithi119 ceph-mon[32608]: pgmap v275: 97 pgs: 97 active+clean; 191 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail; 382 KiB/s rd, 2.1 MiB/s wr, 122 op/s 2024-04-23T10:35:25.992 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 23 10:35:25 smithi119 ceph-mon[32608]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=e=up:active} 3 up:standby-replay 6 up:standby 2024-04-23T10:35:26.004 INFO:journalctl@ceph.mon.a.smithi114.stdout:Apr 23 10:35:25 smithi114 ceph-mon[29255]: pgmap v275: 97 pgs: 97 active+clean; 191 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail; 382 KiB/s rd, 2.1 MiB/s wr, 122 op/s 2024-04-23T10:35:26.004 INFO:journalctl@ceph.mon.a.smithi114.stdout:Apr 23 10:35:25 smithi114 ceph-mon[29255]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=e=up:active} 3 up:standby-replay 6 up:standby 2024-04-23T10:35:26.932 INFO:tasks.workunit.client.0.smithi114.stdout: 1 628563 7.31 MB/sec execute 335 sec latency 1903.279 ms 2024-04-23T10:35:27.828 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 23 10:35:27 smithi204 ceph-mon[32649]: pgmap v276: 97 pgs: 97 active+clean; 201 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail; 948 KiB/s rd, 3.2 MiB/s wr, 204 op/s 2024-04-23T10:35:27.932 INFO:tasks.workunit.client.0.smithi114.stdout: 1 628563 7.28 MB/sec execute 336 sec latency 2903.377 ms 2024-04-23T10:35:27.992 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 23 10:35:27 smithi119 ceph-mon[32608]: pgmap v276: 97 pgs: 97 active+clean; 201 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail; 948 KiB/s rd, 3.2 MiB/s wr, 204 op/s 2024-04-23T10:35:28.004 INFO:journalctl@ceph.mon.a.smithi114.stdout:Apr 23 10:35:27 smithi114 ceph-mon[29255]: pgmap v276: 97 pgs: 97 active+clean; 201 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail; 948 KiB/s rd, 3.2 MiB/s wr, 204 op/s 2024-04-23T10:35:28.932 INFO:tasks.workunit.client.0.smithi114.stdout: 1 628563 7.26 MB/sec execute 337 sec latency 3903.519 ms 2024-04-23T10:35:29.590 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:35:29.589+0000 7f27521fe640 1 -- 172.21.15.114:0/3379034803 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f274c1010f0 msgr2=0x7f274c1014d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T10:35:29.590 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:35:29.589+0000 7f27521fe640 1 --2- 172.21.15.114:0/3379034803 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f274c1010f0 0x7f274c1014d0 secure :-1 s=READY pgs=806 cs=0 l=1 rev1=1 crypto rx=0x7f273c009900 tx=0x7f273c02a410 comp rx=0 tx=0).stop 2024-04-23T10:35:29.590 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:35:29.589+0000 7f27521fe640 1 -- 172.21.15.114:0/3379034803 shutdown_connections 2024-04-23T10:35:29.590 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:35:29.589+0000 7f27521fe640 1 --2- 172.21.15.114:0/3379034803 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f274c106170 0x7f274c10dd40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:35:29.590 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:35:29.589+0000 7f27521fe640 1 --2- 172.21.15.114:0/3379034803 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f274c101aa0 0x7f274c105a90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:35:29.590 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:35:29.589+0000 7f27521fe640 1 --2- 172.21.15.114:0/3379034803 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f274c1010f0 0x7f274c1014d0 unknown :-1 s=CLOSED pgs=806 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:35:29.591 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:35:29.589+0000 7f27521fe640 1 -- 172.21.15.114:0/3379034803 >> 172.21.15.114:0/3379034803 conn(0x7f274c077fd0 msgr2=0x7f274c0783e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T10:35:29.591 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:35:29.589+0000 7f27521fe640 1 -- 172.21.15.114:0/3379034803 shutdown_connections 2024-04-23T10:35:29.591 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:35:29.589+0000 7f27521fe640 1 -- 172.21.15.114:0/3379034803 wait complete. 2024-04-23T10:35:29.591 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:35:29.590+0000 7f27521fe640 1 Processor -- start 2024-04-23T10:35:29.591 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:35:29.590+0000 7f27521fe640 1 -- start start 2024-04-23T10:35:29.591 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:35:29.590+0000 7f27521fe640 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f274c1010f0 0x7f274c199590 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T10:35:29.591 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:35:29.590+0000 7f27521fe640 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f274c106170 0x7f274c199ad0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T10:35:29.591 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:35:29.590+0000 7f27521fe640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f274c19a010 0x7f274c1a4610 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T10:35:29.591 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:35:29.590+0000 7f27521fe640 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x7f274c1110d0 con 0x7f274c19a010 2024-04-23T10:35:29.591 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:35:29.590+0000 7f27521fe640 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 -- 0x7f274c110f50 con 0x7f274c1010f0 2024-04-23T10:35:29.591 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:35:29.590+0000 7f27521fe640 1 -- --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_getmap magic: 0 -- 0x7f274c111250 con 0x7f274c106170 2024-04-23T10:35:29.591 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:35:29.590+0000 7f274bfff640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f274c19a010 0x7f274c1a4610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T10:35:29.591 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:35:29.590+0000 7f274b7fe640 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f274c1010f0 0x7f274c199590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T10:35:29.592 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:35:29.590+0000 7f274affd640 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f274c106170 0x7f274c199ad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T10:35:29.592 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:35:29.590+0000 7f274b7fe640 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f274c1010f0 0x7f274c199590 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.119:3300/0 says I am v2:172.21.15.114:35820/0 (socket says 172.21.15.114:35820) 2024-04-23T10:35:29.592 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:35:29.590+0000 7f274bfff640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f274c19a010 0x7f274c1a4610 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.114:53872/0 (socket says 172.21.15.114:53872) 2024-04-23T10:35:29.592 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:35:29.590+0000 7f274b7fe640 1 -- 172.21.15.114:0/4261043619 learned_addr learned my addr 172.21.15.114:0/4261043619 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-04-23T10:35:29.592 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:35:29.591+0000 7f274affd640 1 -- 172.21.15.114:0/4261043619 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f274c1010f0 msgr2=0x7f274c199590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T10:35:29.592 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:35:29.591+0000 7f274affd640 1 --2- 172.21.15.114:0/4261043619 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f274c1010f0 0x7f274c199590 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:35:29.592 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:35:29.591+0000 7f274affd640 1 -- 172.21.15.114:0/4261043619 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f274c19a010 msgr2=0x7f274c1a4610 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T10:35:29.592 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:35:29.591+0000 7f274affd640 1 --2- 172.21.15.114:0/4261043619 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f274c19a010 0x7f274c1a4610 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:35:29.592 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:35:29.591+0000 7f274affd640 1 -- 172.21.15.114:0/4261043619 --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f274c1a4d90 con 0x7f274c106170 2024-04-23T10:35:29.592 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:35:29.591+0000 7f274bfff640 1 --2- 172.21.15.114:0/4261043619 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f274c19a010 0x7f274c1a4610 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-23T10:35:29.592 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:35:29.591+0000 7f274affd640 1 --2- 172.21.15.114:0/4261043619 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f274c106170 0x7f274c199ad0 secure :-1 s=READY pgs=79 cs=0 l=1 rev1=1 crypto rx=0x7f274000e9c0 tx=0x7f274000ee80 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T10:35:29.592 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:35:29.591+0000 7f2748ff9640 1 -- 172.21.15.114:0/4261043619 <== mon.1 v2:172.21.15.204:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f274000ccc0 con 0x7f274c106170 2024-04-23T10:35:29.592 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:35:29.591+0000 7f27521fe640 1 -- 172.21.15.114:0/4261043619 --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f274c1a5080 con 0x7f274c106170 2024-04-23T10:35:29.592 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:35:29.591+0000 7f2748ff9640 1 -- 172.21.15.114:0/4261043619 <== mon.1 v2:172.21.15.204:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f2740005020 con 0x7f274c106170 2024-04-23T10:35:29.593 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:35:29.591+0000 7f2748ff9640 1 -- 172.21.15.114:0/4261043619 <== mon.1 v2:172.21.15.204:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2740004a00 con 0x7f274c106170 2024-04-23T10:35:29.593 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:35:29.591+0000 7f27521fe640 1 -- 172.21.15.114:0/4261043619 --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f274c1a55d0 con 0x7f274c106170 2024-04-23T10:35:29.600 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:35:29.593+0000 7f2748ff9640 1 -- 172.21.15.114:0/4261043619 <== mon.1 v2:172.21.15.204:3300/0 4 ==== mgrmap(e 26) ==== 138778+0+0 (secure 0 0 0) 0x7f27400051d0 con 0x7f274c106170 2024-04-23T10:35:29.600 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:35:29.593+0000 7f2754ff2740 1 Processor -- start 2024-04-23T10:35:29.600 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:35:29.593+0000 7f2754ff2740 1 -- start start 2024-04-23T10:35:29.600 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:35:29.594+0000 7f2754ff2740 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x5617c94f4620 0x5617c94f4a00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T10:35:29.600 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:35:29.594+0000 7f2754ff2740 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x5617c94f4f40 0x5617c94ff2c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T10:35:29.600 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:35:29.594+0000 7f2754ff2740 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x5617c94ff800 0x5617c9501bf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T10:35:29.600 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:35:29.594+0000 7f2754ff2740 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x5617c9459970 con 0x5617c94ff800 2024-04-23T10:35:29.600 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:35:29.594+0000 7f2754ff2740 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 -- 0x5617c93f64a0 con 0x5617c94f4f40 2024-04-23T10:35:29.600 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:35:29.594+0000 7f2754ff2740 1 -- --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_getmap magic: 0 -- 0x5617c9455930 con 0x5617c94f4620 2024-04-23T10:35:29.600 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:35:29.594+0000 7f274b7fe640 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x5617c94f4f40 0x5617c94ff2c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T10:35:29.600 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:35:29.594+0000 7f274affd640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x5617c94ff800 0x5617c9501bf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T10:35:29.601 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:35:29.594+0000 7f274b7fe640 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x5617c94f4f40 0x5617c94ff2c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.119:3300/0 says I am v2:172.21.15.114:35832/0 (socket says 172.21.15.114:35832) 2024-04-23T10:35:29.601 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:35:29.594+0000 7f274b7fe640 1 -- 172.21.15.114:0/2615602069 learned_addr learned my addr 172.21.15.114:0/2615602069 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-04-23T10:35:29.601 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:35:29.594+0000 7f274bfff640 1 --2- 172.21.15.114:0/2615602069 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x5617c94f4620 0x5617c94f4a00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T10:35:29.601 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:35:29.594+0000 7f274b7fe640 1 -- 172.21.15.114:0/2615602069 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x5617c94f4620 msgr2=0x5617c94f4a00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T10:35:29.601 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:35:29.594+0000 7f274b7fe640 1 --2- 172.21.15.114:0/2615602069 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x5617c94f4620 0x5617c94f4a00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:35:29.601 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:35:29.594+0000 7f274b7fe640 1 -- 172.21.15.114:0/2615602069 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x5617c94ff800 msgr2=0x5617c9501bf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T10:35:29.601 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:35:29.594+0000 7f274b7fe640 1 --2- 172.21.15.114:0/2615602069 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x5617c94ff800 0x5617c9501bf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:35:29.601 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:35:29.594+0000 7f2748ff9640 1 --2- 172.21.15.114:0/4261043619 >> [v2:172.21.15.114:6800/1395693466,v1:172.21.15.114:6801/1395693466] conn(0x7f27340a3dc0 0x7f27340a6280 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T10:35:29.601 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:35:29.594+0000 7f274b7fe640 1 -- 172.21.15.114:0/2615602069 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5617c916c210 con 0x5617c94f4f40 2024-04-23T10:35:29.601 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:35:29.594+0000 7f274bfff640 1 --2- 172.21.15.114:0/4261043619 >> [v2:172.21.15.114:6800/1395693466,v1:172.21.15.114:6801/1395693466] conn(0x7f27340a3dc0 0x7f27340a6280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T10:35:29.601 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:35:29.594+0000 7f2748ff9640 1 -- 172.21.15.114:0/4261043619 <== mon.1 v2:172.21.15.204:3300/0 5 ==== osd_map(100..100 src has 1..100) ==== 8629+0+0 (secure 0 0 0) 0x7f27400d0870 con 0x7f274c106170 2024-04-23T10:35:29.601 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:35:29.595+0000 7f274b7fe640 1 --2- 172.21.15.114:0/2615602069 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x5617c94f4f40 0x5617c94ff2c0 secure :-1 s=READY pgs=90 cs=0 l=1 rev1=1 crypto rx=0x7f273c000c00 tx=0x7f273c031c40 comp rx=0 tx=0).ready entity=mon.2 client_cookie=1ce73d6a2f459f96 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T10:35:29.601 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:35:29.595+0000 7f2731ffb640 1 -- 172.21.15.114:0/2615602069 <== mon.2 v2:172.21.15.119:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f273c040070 con 0x5617c94f4f40 2024-04-23T10:35:29.601 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:35:29.595+0000 7f2731ffb640 1 -- 172.21.15.114:0/2615602069 <== mon.2 v2:172.21.15.119:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f273c0316a0 con 0x5617c94f4f40 2024-04-23T10:35:29.602 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:35:29.595+0000 7f2731ffb640 1 -- 172.21.15.114:0/2615602069 <== mon.2 v2:172.21.15.119:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f273c035040 con 0x5617c94f4f40 2024-04-23T10:35:29.602 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:35:29.595+0000 7f2754ff2740 1 -- 172.21.15.114:0/2615602069 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x5617c94f4f40 msgr2=0x5617c94ff2c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T10:35:29.602 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:35:29.595+0000 7f274bfff640 1 --2- 172.21.15.114:0/4261043619 >> [v2:172.21.15.114:6800/1395693466,v1:172.21.15.114:6801/1395693466] conn(0x7f27340a3dc0 0x7f27340a6280 secure :-1 s=READY pgs=124 cs=0 l=1 rev1=1 crypto rx=0x7f2744005fd0 tx=0x7f27440096a0 comp rx=0 tx=0).ready entity=mgr.15291 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T10:35:29.602 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:35:29.596+0000 7f2754ff2740 1 --2- 172.21.15.114:0/2615602069 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x5617c94f4f40 0x5617c94ff2c0 secure :-1 s=READY pgs=90 cs=0 l=1 rev1=1 crypto rx=0x7f273c000c00 tx=0x7f273c031c40 comp rx=0 tx=0).stop 2024-04-23T10:35:29.602 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:35:29.596+0000 7f2754ff2740 1 -- 172.21.15.114:0/2615602069 shutdown_connections 2024-04-23T10:35:29.602 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:35:29.596+0000 7f2754ff2740 1 --2- 172.21.15.114:0/2615602069 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x5617c94ff800 0x5617c9501bf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:35:29.602 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:35:29.596+0000 7f2754ff2740 1 --2- 172.21.15.114:0/2615602069 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x5617c94f4f40 0x5617c94ff2c0 unknown :-1 s=CLOSED pgs=90 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:35:29.602 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:35:29.596+0000 7f2754ff2740 1 --2- 172.21.15.114:0/2615602069 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x5617c94f4620 0x5617c94f4a00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:35:29.602 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:35:29.596+0000 7f2754ff2740 1 -- 172.21.15.114:0/2615602069 >> 172.21.15.114:0/2615602069 conn(0x5617c94f0b30 msgr2=0x5617c94f0f10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T10:35:29.602 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:35:29.596+0000 7f2754ff2740 1 -- 172.21.15.114:0/2615602069 shutdown_connections 2024-04-23T10:35:29.602 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:35:29.596+0000 7f2754ff2740 1 -- 172.21.15.114:0/2615602069 wait complete. 2024-04-23T10:35:29.602 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:35:29.596+0000 7f2754ff2740 1 -- start start 2024-04-23T10:35:29.602 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:35:29.596+0000 7f2754ff2740 1 Processor -- start 2024-04-23T10:35:29.603 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:35:29.597+0000 7f2754ff2740 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x5617c94f4620 0x5617c94ff150 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T10:35:29.603 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:35:29.597+0000 7f2754ff2740 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x5617c94f4f40 0x5617c94fc440 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T10:35:29.603 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:35:29.597+0000 7f2754ff2740 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x5617c94ff800 0x5617c94fc9a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T10:35:29.603 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:35:29.597+0000 7f2754ff2740 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x5617c95023d0 con 0x5617c94ff800 2024-04-23T10:35:29.603 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:35:29.597+0000 7f2754ff2740 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 -- 0x5617c93f40b0 con 0x5617c94f4f40 2024-04-23T10:35:29.603 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:35:29.597+0000 7f2754ff2740 1 -- --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_getmap magic: 0 -- 0x5617c9502550 con 0x5617c94f4620 2024-04-23T10:35:29.603 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:35:29.597+0000 7f2754ff2740 10 client.0 _ll_register_callbacks cb 0x5617c93e1d30 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-23T10:35:29.603 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:35:29.597+0000 7f274bfff640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x5617c94ff800 0x5617c94fc9a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T10:35:29.603 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:35:29.597+0000 7f274b7fe640 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x5617c94f4620 0x5617c94ff150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T10:35:29.603 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:35:29.597+0000 7f274affd640 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x5617c94f4f40 0x5617c94fc440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T10:35:29.603 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:35:29.597+0000 7f274b7fe640 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x5617c94f4620 0x5617c94ff150 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.204:3300/0 says I am v2:172.21.15.114:57248/0 (socket says 172.21.15.114:57248) 2024-04-23T10:35:29.603 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:35:29.597+0000 7f274b7fe640 1 -- 172.21.15.114:0/1078608070 learned_addr learned my addr 172.21.15.114:0/1078608070 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-04-23T10:35:29.603 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:35:29.597+0000 7f274affd640 1 -- 172.21.15.114:0/1078608070 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x5617c94f4620 msgr2=0x5617c94ff150 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T10:35:29.603 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:35:29.597+0000 7f274affd640 1 --2- 172.21.15.114:0/1078608070 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x5617c94f4620 0x5617c94ff150 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:35:29.604 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:35:29.597+0000 7f274affd640 1 -- 172.21.15.114:0/1078608070 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x5617c94ff800 msgr2=0x5617c94fc9a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T10:35:29.604 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:35:29.597+0000 7f274affd640 1 --2- 172.21.15.114:0/1078608070 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x5617c94ff800 0x5617c94fc9a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:35:29.604 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:35:29.597+0000 7f274affd640 1 -- 172.21.15.114:0/1078608070 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5617c9238620 con 0x5617c94f4f40 2024-04-23T10:35:29.604 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:35:29.597+0000 7f274affd640 1 --2- 172.21.15.114:0/1078608070 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x5617c94f4f40 0x5617c94fc440 secure :-1 s=READY pgs=91 cs=0 l=1 rev1=1 crypto rx=0x7f27400cdef0 tx=0x7f2740098aa0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=169c21d5ee72a2b4 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T10:35:29.604 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:35:29.597+0000 7f272affd640 10 client.0 ms_handle_connect on v2:172.21.15.119:3300/0 2024-04-23T10:35:29.604 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:35:29.597+0000 7f272affd640 1 -- 172.21.15.114:0/1078608070 <== mon.2 v2:172.21.15.119:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f274001c070 con 0x5617c94f4f40 2024-04-23T10:35:29.604 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:35:29.597+0000 7f2754ff2740 1 -- 172.21.15.114:0/1078608070 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f273c0316a0 con 0x5617c94f4f40 2024-04-23T10:35:29.604 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:35:29.597+0000 7f272affd640 1 -- 172.21.15.114:0/1078608070 <== mon.2 v2:172.21.15.119:3300/0 2 ==== osd_map(100..100 src has 1..100) ==== 8629+0+0 (secure 0 0 0) 0x7f27400ce170 con 0x5617c94f4f40 2024-04-23T10:35:29.604 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:35:29.598+0000 7f272affd640 1 -- 172.21.15.114:0/1078608070 <== mon.2 v2:172.21.15.119:3300/0 3 ==== mon_get_version_reply(handle=1 version=33) ==== 24+0+0 (secure 0 0 0) 0x7f27400ce370 con 0x5617c94f4f40 2024-04-23T10:35:29.604 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:35:29.598+0000 7f2754ff2740 10 client.26039 fetch_fsmap learned FSMap version 33 2024-04-23T10:35:29.604 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:35:29.598+0000 7f2754ff2740 1 -- 172.21.15.114:0/1078608070 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({config=0+,fsmap=33,monmap=4+}) -- 0x5617c950e3b0 con 0x5617c94f4f40 2024-04-23T10:35:29.604 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:35:29.598+0000 7f272affd640 1 -- 172.21.15.114:0/1078608070 <== mon.2 v2:172.21.15.119:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f273c0316a0 con 0x5617c94f4f40 2024-04-23T10:35:29.604 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:35:29.598+0000 7f272affd640 1 -- 172.21.15.114:0/1078608070 <== mon.2 v2:172.21.15.119:3300/0 5 ==== fsmap(e 33) ==== 8342+0+0 (secure 0 0 0) 0x7f27400a4f20 con 0x5617c94f4f40 2024-04-23T10:35:29.604 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:35:29.598+0000 7f2754ff2740 10 client.26039 fetch_fsmap finished waiting for FSMap version 33 2024-04-23T10:35:29.604 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:35:29.598+0000 7f2754ff2740 10 client.26039 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-04-23T10:35:29.604 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:35:29.598+0000 7f2754ff2740 20 client.26039 populate_metadata read hostname 'smithi114' 2024-04-23T10:35:29.605 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:35:29.598+0000 7f2754ff2740 1 --2- 172.21.15.114:0/1078608070 >> [v2:172.21.15.119:6838/937297340,v1:172.21.15.119:6839/937297340] conn(0x5617c950f870 0x5617c9511c60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T10:35:29.605 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:35:29.598+0000 7f2754ff2740 4 client.26039 mds_command: new command op to 24545 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-23T10:35:29.605 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:35:29.598+0000 7f2754ff2740 1 -- 172.21.15.114:0/1078608070 --> [v2:172.21.15.119:6838/937297340,v1:172.21.15.119:6839/937297340] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x5617c9467390 con 0x5617c950f870 2024-04-23T10:35:29.605 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:35:29.598+0000 7f274b7fe640 1 --2- 172.21.15.114:0/1078608070 >> [v2:172.21.15.119:6838/937297340,v1:172.21.15.119:6839/937297340] conn(0x5617c950f870 0x5617c9511c60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T10:35:29.605 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:35:29.599+0000 7f274b7fe640 1 --2- 172.21.15.114:0/1078608070 >> [v2:172.21.15.119:6838/937297340,v1:172.21.15.119:6839/937297340] conn(0x5617c950f870 0x5617c9511c60 crc :-1 s=READY pgs=140 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=84edc85f1a46dc5b server_cookie=22070824ef33ae40 in_seq=0 out_seq=0 2024-04-23T10:35:29.605 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:35:29.599+0000 7f272affd640 10 client.26039 ms_handle_connect on v2:172.21.15.119:6838/937297340 2024-04-23T10:35:29.605 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:35:29.599+0000 7f272affd640 1 -- 172.21.15.114:0/1078608070 <== mds.0 v2:172.21.15.119:6838/937297340 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x5617c9467390 con 0x5617c950f870 2024-04-23T10:35:29.605 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:35:29.599+0000 7f272affd640 10 client.26039 handle_command_reply: tid=0 2024-04-23T10:35:29.605 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:35:29.599+0000 7f2754ff2740 1 client.26039 shutdown 2024-04-23T10:35:29.605 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:35:29.599+0000 7f2754ff2740 1 -- 172.21.15.114:0/1078608070 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x5617c94f4f40 msgr2=0x5617c94fc440 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T10:35:29.605 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:35:29.599+0000 7f2754ff2740 1 --2- 172.21.15.114:0/1078608070 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x5617c94f4f40 0x5617c94fc440 secure :-1 s=READY pgs=91 cs=0 l=1 rev1=1 crypto rx=0x7f27400cdef0 tx=0x7f2740098aa0 comp rx=0 tx=0).stop 2024-04-23T10:35:29.605 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:35:29.599+0000 7f2754ff2740 1 -- 172.21.15.114:0/1078608070 shutdown_connections 2024-04-23T10:35:29.605 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:35:29.599+0000 7f2754ff2740 1 --2- 172.21.15.114:0/1078608070 >> [v2:172.21.15.119:6838/937297340,v1:172.21.15.119:6839/937297340] conn(0x5617c950f870 0x5617c9511c60 crc :-1 s=READY pgs=140 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:35:29.605 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:35:29.600+0000 7f272affd640 0 client.26039 ms_handle_reset on v2:172.21.15.119:6838/937297340 2024-04-23T10:35:29.605 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:35:29.600+0000 7f2754ff2740 1 --2- 172.21.15.114:0/1078608070 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x5617c94ff800 0x5617c94fc9a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:35:29.606 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:35:29.600+0000 7f2754ff2740 1 --2- 172.21.15.114:0/1078608070 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x5617c94f4f40 0x5617c94fc440 unknown :-1 s=CLOSED pgs=91 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:35:29.606 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:35:29.600+0000 7f2754ff2740 1 --2- 172.21.15.114:0/1078608070 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x5617c94f4620 0x5617c94ff150 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:35:29.606 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:35:29.600+0000 7f2754ff2740 1 -- 172.21.15.114:0/1078608070 >> 172.21.15.114:0/1078608070 conn(0x5617c94f0b30 msgr2=0x5617c94f6860 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T10:35:29.606 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:35:29.600+0000 7f2754ff2740 1 -- 172.21.15.114:0/1078608070 shutdown_connections 2024-04-23T10:35:29.606 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:35:29.600+0000 7f2754ff2740 1 -- 172.21.15.114:0/1078608070 wait complete. 2024-04-23T10:35:29.606 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:35:29.600+0000 7f2754ff2740 20 client.26039 trim_cache size 0 max 16384 2024-04-23T10:35:29.619 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:35:29.615+0000 7f2754ff2740 1 Processor -- start 2024-04-23T10:35:29.619 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:35:29.615+0000 7f2754ff2740 1 -- start start 2024-04-23T10:35:29.619 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:35:29.616+0000 7f2754ff2740 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x5617c94f4620 0x5617c94f7af0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T10:35:29.619 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:35:29.616+0000 7f2754ff2740 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x5617c94f4f40 0x5617c94efb30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T10:35:29.619 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:35:29.616+0000 7f2754ff2740 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x5617c94ff800 0x5617c94fec80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T10:35:29.619 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:35:29.616+0000 7f2754ff2740 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x7f274c1b05a0 con 0x5617c94f4f40 2024-04-23T10:35:29.619 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:35:29.616+0000 7f2754ff2740 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 -- 0x7f274c1ade70 con 0x5617c94f4620 2024-04-23T10:35:29.619 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:35:29.616+0000 7f2754ff2740 1 -- --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_getmap magic: 0 -- 0x7f274c1abce0 con 0x5617c94ff800 2024-04-23T10:35:29.619 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:35:29.616+0000 7f274affd640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x5617c94f4f40 0x5617c94efb30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T10:35:29.620 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:35:29.616+0000 7f274affd640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x5617c94f4f40 0x5617c94efb30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.114:53894/0 (socket says 172.21.15.114:53894) 2024-04-23T10:35:29.620 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:35:29.616+0000 7f274affd640 1 -- 172.21.15.114:0/3695563858 learned_addr learned my addr 172.21.15.114:0/3695563858 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-04-23T10:35:29.620 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:35:29.616+0000 7f274b7fe640 1 --2- 172.21.15.114:0/3695563858 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x5617c94f4620 0x5617c94f7af0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T10:35:29.620 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:35:29.616+0000 7f274bfff640 1 --2- 172.21.15.114:0/3695563858 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x5617c94ff800 0x5617c94fec80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T10:35:29.620 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:35:29.616+0000 7f274affd640 1 -- 172.21.15.114:0/3695563858 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x5617c94ff800 msgr2=0x5617c94fec80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T10:35:29.620 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:35:29.616+0000 7f274affd640 1 --2- 172.21.15.114:0/3695563858 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x5617c94ff800 0x5617c94fec80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:35:29.620 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:35:29.616+0000 7f274affd640 1 -- 172.21.15.114:0/3695563858 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x5617c94f4620 msgr2=0x5617c94f7af0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T10:35:29.620 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:35:29.616+0000 7f274affd640 1 --2- 172.21.15.114:0/3695563858 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x5617c94f4620 0x5617c94f7af0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:35:29.620 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:35:29.616+0000 7f274affd640 1 -- 172.21.15.114:0/3695563858 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5617c94fcd10 con 0x5617c94f4f40 2024-04-23T10:35:29.620 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:35:29.616+0000 7f274b7fe640 1 --2- 172.21.15.114:0/3695563858 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x5617c94f4620 0x5617c94f7af0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-23T10:35:29.620 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:35:29.616+0000 7f274affd640 1 --2- 172.21.15.114:0/3695563858 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x5617c94f4f40 0x5617c94efb30 secure :-1 s=READY pgs=807 cs=0 l=1 rev1=1 crypto rx=0x7f27400990c0 tx=0x7f27400cdfd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=64c575eeac6ed20d server_cookie=0 in_seq=0 out_seq=0 2024-04-23T10:35:29.620 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:35:29.616+0000 7f27317fa640 1 -- 172.21.15.114:0/3695563858 <== mon.0 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f274009ce40 con 0x5617c94f4f40 2024-04-23T10:35:29.620 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:35:29.616+0000 7f27317fa640 1 -- 172.21.15.114:0/3695563858 <== mon.0 v2:172.21.15.114:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f27400d3e50 con 0x5617c94f4f40 2024-04-23T10:35:29.620 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:35:29.616+0000 7f27317fa640 1 -- 172.21.15.114:0/3695563858 <== mon.0 v2:172.21.15.114:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f27400ce170 con 0x5617c94f4f40 2024-04-23T10:35:29.621 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:35:29.617+0000 7f2754ff2740 1 -- 172.21.15.114:0/3695563858 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x5617c94f4f40 msgr2=0x5617c94efb30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T10:35:29.621 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:35:29.617+0000 7f2754ff2740 1 --2- 172.21.15.114:0/3695563858 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x5617c94f4f40 0x5617c94efb30 secure :-1 s=READY pgs=807 cs=0 l=1 rev1=1 crypto rx=0x7f27400990c0 tx=0x7f27400cdfd0 comp rx=0 tx=0).stop 2024-04-23T10:35:29.621 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:35:29.617+0000 7f2754ff2740 1 -- 172.21.15.114:0/3695563858 shutdown_connections 2024-04-23T10:35:29.621 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:35:29.617+0000 7f2754ff2740 1 --2- 172.21.15.114:0/3695563858 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x5617c94ff800 0x5617c94fec80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:35:29.621 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:35:29.617+0000 7f2754ff2740 1 --2- 172.21.15.114:0/3695563858 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x5617c94f4f40 0x5617c94efb30 unknown :-1 s=CLOSED pgs=807 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:35:29.621 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:35:29.617+0000 7f2754ff2740 1 --2- 172.21.15.114:0/3695563858 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x5617c94f4620 0x5617c94f7af0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:35:29.621 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:35:29.617+0000 7f2754ff2740 1 -- 172.21.15.114:0/3695563858 >> 172.21.15.114:0/3695563858 conn(0x5617c94f0b30 msgr2=0x5617c94ef420 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T10:35:29.621 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:35:29.617+0000 7f2754ff2740 1 -- 172.21.15.114:0/3695563858 shutdown_connections 2024-04-23T10:35:29.621 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:35:29.617+0000 7f2754ff2740 1 -- 172.21.15.114:0/3695563858 wait complete. 2024-04-23T10:35:29.621 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:35:29.618+0000 7f2754ff2740 1 -- start start 2024-04-23T10:35:29.621 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:35:29.618+0000 7f2754ff2740 1 Processor -- start 2024-04-23T10:35:29.621 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:35:29.618+0000 7f2754ff2740 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x5617c94f4620 0x5617c9503140 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T10:35:29.621 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:35:29.618+0000 7f2754ff2740 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x5617c94f4f40 0x5617c9503680 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T10:35:29.621 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:35:29.618+0000 7f2754ff2740 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x5617c94ff800 0x5617c9503bc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T10:35:29.621 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:35:29.618+0000 7f2754ff2740 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x7f274c1ac330 con 0x5617c94f4f40 2024-04-23T10:35:29.622 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:35:29.618+0000 7f2754ff2740 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 -- 0x7f274c1ad7a0 con 0x5617c94ff800 2024-04-23T10:35:29.622 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:35:29.618+0000 7f2754ff2740 1 -- --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_getmap magic: 0 -- 0x7f274c1aca00 con 0x5617c94f4620 2024-04-23T10:35:29.622 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:35:29.618+0000 7f2754ff2740 10 client.0 _ll_register_callbacks cb 0x7f2740098320 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-23T10:35:29.622 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:35:29.618+0000 7f274affd640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x5617c94f4f40 0x5617c9503680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T10:35:29.622 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:35:29.618+0000 7f274affd640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x5617c94f4f40 0x5617c9503680 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.114:53908/0 (socket says 172.21.15.114:53908) 2024-04-23T10:35:29.622 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:35:29.618+0000 7f274affd640 1 -- 172.21.15.114:0/129615249 learned_addr learned my addr 172.21.15.114:0/129615249 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-04-23T10:35:29.622 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:35:29.618+0000 7f274bfff640 1 --2- 172.21.15.114:0/129615249 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x5617c94ff800 0x5617c9503bc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T10:35:29.622 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:35:29.618+0000 7f274b7fe640 1 --2- 172.21.15.114:0/129615249 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x5617c94f4620 0x5617c9503140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T10:35:29.622 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:35:29.619+0000 7f274affd640 1 -- 172.21.15.114:0/129615249 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x5617c94f4620 msgr2=0x5617c9503140 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T10:35:29.622 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:35:29.619+0000 7f274affd640 1 --2- 172.21.15.114:0/129615249 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x5617c94f4620 0x5617c9503140 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:35:29.622 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:35:29.619+0000 7f274affd640 1 -- 172.21.15.114:0/129615249 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x5617c94ff800 msgr2=0x5617c9503bc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T10:35:29.622 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:35:29.619+0000 7f274affd640 1 --2- 172.21.15.114:0/129615249 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x5617c94ff800 0x5617c9503bc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:35:29.622 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:35:29.619+0000 7f274affd640 1 -- 172.21.15.114:0/129615249 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5617c9574de0 con 0x5617c94f4f40 2024-04-23T10:35:29.623 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:35:29.619+0000 7f274affd640 1 --2- 172.21.15.114:0/129615249 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x5617c94f4f40 0x5617c9503680 secure :-1 s=READY pgs=808 cs=0 l=1 rev1=1 crypto rx=0x7f274000c6a0 tx=0x7f274000efb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9669f58f334bcfd0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T10:35:29.623 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:35:29.619+0000 7f272affd640 10 client.0 ms_handle_connect on v2:172.21.15.114:3300/0 2024-04-23T10:35:29.623 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:35:29.619+0000 7f272affd640 1 -- 172.21.15.114:0/129615249 <== mon.0 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f27400ce370 con 0x5617c94f4f40 2024-04-23T10:35:29.623 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:35:29.619+0000 7f272affd640 1 -- 172.21.15.114:0/129615249 <== mon.0 v2:172.21.15.114:3300/0 2 ==== osd_map(100..100 src has 1..100) ==== 8629+0+0 (secure 0 0 0) 0x7f274009bb90 con 0x5617c94f4f40 2024-04-23T10:35:29.623 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:35:29.619+0000 7f2754ff2740 1 -- 172.21.15.114:0/129615249 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f27400d3e50 con 0x5617c94f4f40 2024-04-23T10:35:29.623 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:35:29.619+0000 7f272affd640 1 -- 172.21.15.114:0/129615249 <== mon.0 v2:172.21.15.114:3300/0 3 ==== mon_get_version_reply(handle=1 version=33) ==== 24+0+0 (secure 0 0 0) 0x7f2740004cc0 con 0x5617c94f4f40 2024-04-23T10:35:29.623 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:35:29.619+0000 7f2754ff2740 10 client.16680 fetch_fsmap learned FSMap version 33 2024-04-23T10:35:29.623 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:35:29.619+0000 7f2754ff2740 1 -- 172.21.15.114:0/129615249 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({config=0+,fsmap=33,monmap=4+}) -- 0x5617c9576010 con 0x5617c94f4f40 2024-04-23T10:35:29.623 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:35:29.619+0000 7f272affd640 1 -- 172.21.15.114:0/129615249 <== mon.0 v2:172.21.15.114:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f27400d3e50 con 0x5617c94f4f40 2024-04-23T10:35:29.623 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:35:29.619+0000 7f272affd640 1 -- 172.21.15.114:0/129615249 <== mon.0 v2:172.21.15.114:3300/0 5 ==== fsmap(e 33) ==== 8342+0+0 (secure 0 0 0) 0x7f2740098650 con 0x5617c94f4f40 2024-04-23T10:35:29.623 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:35:29.620+0000 7f2754ff2740 10 client.16680 fetch_fsmap finished waiting for FSMap version 33 2024-04-23T10:35:29.623 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:35:29.620+0000 7f2754ff2740 10 client.16680 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-04-23T10:35:29.623 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:35:29.620+0000 7f2754ff2740 20 client.16680 populate_metadata read hostname 'smithi114' 2024-04-23T10:35:29.623 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:35:29.620+0000 7f2754ff2740 1 --2- 172.21.15.114:0/129615249 >> [v2:172.21.15.119:6838/937297340,v1:172.21.15.119:6839/937297340] conn(0x5617c950f870 0x5617c9578490 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T10:35:29.623 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:35:29.620+0000 7f2754ff2740 4 client.16680 mds_command: new command op to 24545 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-04-23T10:35:29.624 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:35:29.620+0000 7f2754ff2740 1 -- 172.21.15.114:0/129615249 --> [v2:172.21.15.119:6838/937297340,v1:172.21.15.119:6839/937297340] -- command(tid 0: {"prefix": "flush journal"}) -- 0x5617c9461200 con 0x5617c950f870 2024-04-23T10:35:29.624 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:35:29.620+0000 7f274b7fe640 1 --2- 172.21.15.114:0/129615249 >> [v2:172.21.15.119:6838/937297340,v1:172.21.15.119:6839/937297340] conn(0x5617c950f870 0x5617c9578490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T10:35:29.624 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:35:29.620+0000 7f272affd640 10 client.16680 ms_handle_connect on v2:172.21.15.119:6838/937297340 2024-04-23T10:35:29.624 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:35:29.620+0000 7f274b7fe640 1 --2- 172.21.15.114:0/129615249 >> [v2:172.21.15.119:6838/937297340,v1:172.21.15.119:6839/937297340] conn(0x5617c950f870 0x5617c9578490 crc :-1 s=READY pgs=141 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=621af485a6ecabf7 server_cookie=df3b3e63d810272 in_seq=0 out_seq=0 2024-04-23T10:35:29.666 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:35:29.666+0000 7f272affd640 1 -- 172.21.15.114:0/129615249 <== mds.0 v2:172.21.15.119:6838/937297340 1 ==== command_reply(tid 0: 0 ) ==== 8+0+44 (crc 0 0 0) 0x5617c9461200 con 0x5617c950f870 2024-04-23T10:35:29.666 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:35:29.666+0000 7f272affd640 10 client.16680 handle_command_reply: tid=0 2024-04-23T10:35:29.666 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:35:29.666+0000 7f2754ff2740 1 client.16680 shutdown 2024-04-23T10:35:29.666 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:35:29.666+0000 7f2754ff2740 1 -- 172.21.15.114:0/129615249 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x5617c94f4f40 msgr2=0x5617c9503680 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T10:35:29.666 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:35:29.666+0000 7f2754ff2740 1 --2- 172.21.15.114:0/129615249 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x5617c94f4f40 0x5617c9503680 secure :-1 s=READY pgs=808 cs=0 l=1 rev1=1 crypto rx=0x7f274000c6a0 tx=0x7f274000efb0 comp rx=0 tx=0).stop 2024-04-23T10:35:29.666 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:35:29.666+0000 7f2754ff2740 1 -- 172.21.15.114:0/129615249 shutdown_connections 2024-04-23T10:35:29.666 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:35:29.666+0000 7f2754ff2740 1 --2- 172.21.15.114:0/129615249 >> [v2:172.21.15.119:6838/937297340,v1:172.21.15.119:6839/937297340] conn(0x5617c950f870 0x5617c9578490 crc :-1 s=READY pgs=141 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:35:29.666 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:35:29.666+0000 7f272affd640 0 client.16680 ms_handle_reset on v2:172.21.15.119:6838/937297340 2024-04-23T10:35:29.666 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:35:29.666+0000 7f2754ff2740 1 --2- 172.21.15.114:0/129615249 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x5617c94ff800 0x5617c9503bc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:35:29.666 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:35:29.666+0000 7f2754ff2740 1 --2- 172.21.15.114:0/129615249 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x5617c94f4f40 0x5617c9503680 unknown :-1 s=CLOSED pgs=808 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:35:29.666 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:35:29.666+0000 7f2754ff2740 1 --2- 172.21.15.114:0/129615249 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x5617c94f4620 0x5617c9503140 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:35:29.666 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:35:29.666+0000 7f2754ff2740 1 -- 172.21.15.114:0/129615249 >> 172.21.15.114:0/129615249 conn(0x5617c94f0b30 msgr2=0x5617c950a280 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T10:35:29.667 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:35:29.666+0000 7f2754ff2740 1 -- 172.21.15.114:0/129615249 shutdown_connections 2024-04-23T10:35:29.667 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:35:29.666+0000 7f2754ff2740 1 -- 172.21.15.114:0/129615249 wait complete. 2024-04-23T10:35:29.667 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:35:29.666+0000 7f2754ff2740 20 client.16680 trim_cache size 0 max 16384 2024-04-23T10:35:29.667 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stdout:{ 2024-04-23T10:35:29.667 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stdout: "message": "", 2024-04-23T10:35:29.667 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stdout: "return_code": 0 2024-04-23T10:35:29.667 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stdout:} 2024-04-23T10:35:29.668 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:35:29.667+0000 7f27521fe640 1 -- 172.21.15.114:0/4261043619 >> [v2:172.21.15.114:6800/1395693466,v1:172.21.15.114:6801/1395693466] conn(0x7f27340a3dc0 msgr2=0x7f27340a6280 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T10:35:29.669 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:35:29.667+0000 7f27521fe640 1 --2- 172.21.15.114:0/4261043619 >> [v2:172.21.15.114:6800/1395693466,v1:172.21.15.114:6801/1395693466] conn(0x7f27340a3dc0 0x7f27340a6280 secure :-1 s=READY pgs=124 cs=0 l=1 rev1=1 crypto rx=0x7f2744005fd0 tx=0x7f27440096a0 comp rx=0 tx=0).stop 2024-04-23T10:35:29.669 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:35:29.667+0000 7f27521fe640 1 -- 172.21.15.114:0/4261043619 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f274c106170 msgr2=0x7f274c199ad0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T10:35:29.669 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:35:29.667+0000 7f27521fe640 1 --2- 172.21.15.114:0/4261043619 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f274c106170 0x7f274c199ad0 secure :-1 s=READY pgs=79 cs=0 l=1 rev1=1 crypto rx=0x7f274000e9c0 tx=0x7f274000ee80 comp rx=0 tx=0).stop 2024-04-23T10:35:29.669 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:35:29.667+0000 7f27521fe640 1 -- 172.21.15.114:0/4261043619 shutdown_connections 2024-04-23T10:35:29.669 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:35:29.667+0000 7f27521fe640 1 --2- 172.21.15.114:0/4261043619 >> [v2:172.21.15.114:6800/1395693466,v1:172.21.15.114:6801/1395693466] conn(0x7f27340a3dc0 0x7f27340a6280 unknown :-1 s=CLOSED pgs=124 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:35:29.669 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:35:29.667+0000 7f27521fe640 1 --2- 172.21.15.114:0/4261043619 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f274c19a010 0x7f274c1a4610 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:35:29.669 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:35:29.667+0000 7f27521fe640 1 --2- 172.21.15.114:0/4261043619 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f274c106170 0x7f274c199ad0 unknown :-1 s=CLOSED pgs=79 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:35:29.669 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:35:29.667+0000 7f27521fe640 1 --2- 172.21.15.114:0/4261043619 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f274c1010f0 0x7f274c199590 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:35:29.669 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:35:29.667+0000 7f27521fe640 1 -- 172.21.15.114:0/4261043619 >> 172.21.15.114:0/4261043619 conn(0x7f274c077fd0 msgr2=0x7f274c0ff030 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T10:35:29.669 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:35:29.667+0000 7f27521fe640 1 -- 172.21.15.114:0/4261043619 shutdown_connections 2024-04-23T10:35:29.669 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:35:29.667+0000 7f27521fe640 1 -- 172.21.15.114:0/4261043619 wait complete. 2024-04-23T10:35:29.827 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 23 10:35:29 smithi204 ceph-mon[32649]: pgmap v277: 97 pgs: 97 active+clean; 201 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail; 879 KiB/s rd, 2.3 MiB/s wr, 171 op/s 2024-04-23T10:35:29.932 INFO:tasks.workunit.client.0.smithi114.stdout: 1 629169 7.25 MB/sec execute 338 sec latency 4157.437 ms 2024-04-23T10:35:29.991 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 23 10:35:29 smithi119 ceph-mon[32608]: pgmap v277: 97 pgs: 97 active+clean; 201 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail; 879 KiB/s rd, 2.3 MiB/s wr, 171 op/s 2024-04-23T10:35:30.004 INFO:journalctl@ceph.mon.a.smithi114.stdout:Apr 23 10:35:29 smithi114 ceph-mon[29255]: pgmap v277: 97 pgs: 97 active+clean; 201 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail; 879 KiB/s rd, 2.3 MiB/s wr, 171 op/s 2024-04-23T10:35:30.932 INFO:tasks.workunit.client.0.smithi114.stdout: 1 631199 7.25 MB/sec execute 339 sec latency 23.960 ms 2024-04-23T10:35:31.828 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 23 10:35:31 smithi204 ceph-mon[32649]: pgmap v278: 97 pgs: 97 active+clean; 201 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail; 879 KiB/s rd, 2.3 MiB/s wr, 171 op/s 2024-04-23T10:35:31.933 INFO:tasks.workunit.client.0.smithi114.stdout: 1 632146 7.25 MB/sec execute 340 sec latency 677.724 ms 2024-04-23T10:35:31.992 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 23 10:35:31 smithi119 ceph-mon[32608]: pgmap v278: 97 pgs: 97 active+clean; 201 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail; 879 KiB/s rd, 2.3 MiB/s wr, 171 op/s 2024-04-23T10:35:32.004 INFO:journalctl@ceph.mon.a.smithi114.stdout:Apr 23 10:35:31 smithi114 ceph-mon[29255]: pgmap v278: 97 pgs: 97 active+clean; 201 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail; 879 KiB/s rd, 2.3 MiB/s wr, 171 op/s 2024-04-23T10:35:32.933 INFO:tasks.workunit.client.0.smithi114.stdout: 1 632146 7.23 MB/sec execute 341 sec latency 1677.760 ms 2024-04-23T10:35:33.794 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 23 10:35:33 smithi119 ceph-mon[32608]: pgmap v279: 97 pgs: 97 active+clean; 200 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail; 1.1 MiB/s rd, 3.2 MiB/s wr, 251 op/s 2024-04-23T10:35:33.828 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 23 10:35:33 smithi204 ceph-mon[32649]: pgmap v279: 97 pgs: 97 active+clean; 200 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail; 1.1 MiB/s rd, 3.2 MiB/s wr, 251 op/s 2024-04-23T10:35:33.933 INFO:tasks.workunit.client.0.smithi114.stdout: 1 632235 7.21 MB/sec execute 342 sec latency 2195.183 ms 2024-04-23T10:35:34.004 INFO:journalctl@ceph.mon.a.smithi114.stdout:Apr 23 10:35:33 smithi114 ceph-mon[29255]: pgmap v279: 97 pgs: 97 active+clean; 200 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail; 1.1 MiB/s rd, 3.2 MiB/s wr, 251 op/s 2024-04-23T10:35:34.933 INFO:tasks.workunit.client.0.smithi114.stdout: 1 632302 7.19 MB/sec execute 343 sec latency 666.128 ms 2024-04-23T10:35:35.827 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 23 10:35:35 smithi204 ceph-mon[32649]: pgmap v280: 97 pgs: 97 active+clean; 200 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail; 862 KiB/s rd, 2.1 MiB/s wr, 161 op/s 2024-04-23T10:35:35.933 INFO:tasks.workunit.client.0.smithi114.stdout: 1 632302 7.17 MB/sec execute 344 sec latency 1652.234 ms 2024-04-23T10:35:35.991 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 23 10:35:35 smithi119 ceph-mon[32608]: pgmap v280: 97 pgs: 97 active+clean; 200 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail; 862 KiB/s rd, 2.1 MiB/s wr, 161 op/s 2024-04-23T10:35:36.003 INFO:journalctl@ceph.mon.a.smithi114.stdout:Apr 23 10:35:35 smithi114 ceph-mon[29255]: pgmap v280: 97 pgs: 97 active+clean; 200 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail; 862 KiB/s rd, 2.1 MiB/s wr, 161 op/s 2024-04-23T10:35:36.933 INFO:tasks.workunit.client.0.smithi114.stdout: 1 632302 7.15 MB/sec execute 345 sec latency 2652.324 ms 2024-04-23T10:35:37.828 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 23 10:35:37 smithi204 ceph-mon[32649]: pgmap v281: 97 pgs: 97 active+clean; 198 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail; 963 KiB/s rd, 2.7 MiB/s wr, 194 op/s 2024-04-23T10:35:37.933 INFO:tasks.workunit.client.0.smithi114.stdout: 1 632302 7.13 MB/sec execute 346 sec latency 3652.454 ms 2024-04-23T10:35:37.992 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 23 10:35:37 smithi119 ceph-mon[32608]: pgmap v281: 97 pgs: 97 active+clean; 198 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail; 963 KiB/s rd, 2.7 MiB/s wr, 194 op/s 2024-04-23T10:35:38.004 INFO:journalctl@ceph.mon.a.smithi114.stdout:Apr 23 10:35:37 smithi114 ceph-mon[29255]: pgmap v281: 97 pgs: 97 active+clean; 198 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail; 963 KiB/s rd, 2.7 MiB/s wr, 194 op/s 2024-04-23T10:35:38.933 INFO:tasks.workunit.client.0.smithi114.stdout: 1 632302 7.11 MB/sec execute 347 sec latency 4652.551 ms 2024-04-23T10:35:39.828 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 23 10:35:39 smithi204 ceph-mon[32649]: pgmap v282: 97 pgs: 97 active+clean; 198 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail; 397 KiB/s rd, 1.5 MiB/s wr, 112 op/s 2024-04-23T10:35:39.933 INFO:tasks.workunit.client.0.smithi114.stdout: 1 633069 7.09 MB/sec execute 348 sec latency 4905.625 ms 2024-04-23T10:35:39.991 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 23 10:35:39 smithi119 ceph-mon[32608]: pgmap v282: 97 pgs: 97 active+clean; 198 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail; 397 KiB/s rd, 1.5 MiB/s wr, 112 op/s 2024-04-23T10:35:40.003 INFO:journalctl@ceph.mon.a.smithi114.stdout:Apr 23 10:35:39 smithi114 ceph-mon[29255]: pgmap v282: 97 pgs: 97 active+clean; 198 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail; 397 KiB/s rd, 1.5 MiB/s wr, 112 op/s 2024-04-23T10:35:40.933 INFO:tasks.workunit.client.0.smithi114.stdout: 1 635780 7.12 MB/sec execute 349 sec latency 15.870 ms 2024-04-23T10:35:41.827 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 23 10:35:41 smithi204 ceph-mon[32649]: pgmap v283: 97 pgs: 97 active+clean; 198 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail; 397 KiB/s rd, 1.5 MiB/s wr, 112 op/s 2024-04-23T10:35:41.933 INFO:tasks.workunit.client.0.smithi114.stdout: 1 635780 7.10 MB/sec execute 350 sec latency 1015.935 ms 2024-04-23T10:35:41.992 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 23 10:35:41 smithi119 ceph-mon[32608]: pgmap v283: 97 pgs: 97 active+clean; 198 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail; 397 KiB/s rd, 1.5 MiB/s wr, 112 op/s 2024-04-23T10:35:42.004 INFO:journalctl@ceph.mon.a.smithi114.stdout:Apr 23 10:35:41 smithi114 ceph-mon[29255]: pgmap v283: 97 pgs: 97 active+clean; 198 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail; 397 KiB/s rd, 1.5 MiB/s wr, 112 op/s 2024-04-23T10:35:42.766 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:35:42.765+0000 7fecce930640 1 -- 172.21.15.114:0/3999179095 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fecc8102b80 msgr2=0x7fecc8106bc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T10:35:42.767 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:35:42.765+0000 7fecce930640 1 --2- 172.21.15.114:0/3999179095 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fecc8102b80 0x7fecc8106bc0 secure :-1 s=READY pgs=809 cs=0 l=1 rev1=1 crypto rx=0x7fecbc009900 tx=0x7fecbc02a410 comp rx=0 tx=0).stop 2024-04-23T10:35:42.767 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:35:42.765+0000 7fecce930640 1 -- 172.21.15.114:0/3999179095 shutdown_connections 2024-04-23T10:35:42.767 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:35:42.765+0000 7fecce930640 1 --2- 172.21.15.114:0/3999179095 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fecc81072a0 0x7fecc810ee70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:35:42.767 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:35:42.765+0000 7fecce930640 1 --2- 172.21.15.114:0/3999179095 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fecc8102b80 0x7fecc8106bc0 unknown :-1 s=CLOSED pgs=809 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:35:42.767 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:35:42.765+0000 7fecce930640 1 --2- 172.21.15.114:0/3999179095 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fecc81021d0 0x7fecc81025b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:35:42.767 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:35:42.765+0000 7fecce930640 1 -- 172.21.15.114:0/3999179095 >> 172.21.15.114:0/3999179095 conn(0x7fecc80793e0 msgr2=0x7fecc80797f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T10:35:42.767 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:35:42.765+0000 7fecce930640 1 -- 172.21.15.114:0/3999179095 shutdown_connections 2024-04-23T10:35:42.767 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:35:42.765+0000 7fecce930640 1 -- 172.21.15.114:0/3999179095 wait complete. 2024-04-23T10:35:42.767 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:35:42.766+0000 7fecce930640 1 Processor -- start 2024-04-23T10:35:42.767 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:35:42.766+0000 7fecce930640 1 -- start start 2024-04-23T10:35:42.767 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:35:42.766+0000 7fecce930640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fecc81021d0 0x7fecc8074250 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T10:35:42.767 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:35:42.766+0000 7fecce930640 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fecc81072a0 0x7fecc8074790 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T10:35:42.767 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:35:42.766+0000 7fecce930640 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fecc8074cd0 0x7fecc81a6890 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T10:35:42.768 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:35:42.766+0000 7fecce930640 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x7fecc81121c0 con 0x7fecc81021d0 2024-04-23T10:35:42.768 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:35:42.766+0000 7fecce930640 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 -- 0x7fecc8112040 con 0x7fecc81072a0 2024-04-23T10:35:42.768 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:35:42.766+0000 7fecce930640 1 -- --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_getmap magic: 0 -- 0x7fecc8112340 con 0x7fecc8074cd0 2024-04-23T10:35:42.768 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:35:42.767+0000 7fecc7fff640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fecc81021d0 0x7fecc8074250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T10:35:42.768 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:35:42.767+0000 7fecc7fff640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fecc81021d0 0x7fecc8074250 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.114:60842/0 (socket says 172.21.15.114:60842) 2024-04-23T10:35:42.768 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:35:42.767+0000 7fecc7fff640 1 -- 172.21.15.114:0/652575493 learned_addr learned my addr 172.21.15.114:0/652575493 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-04-23T10:35:42.768 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:35:42.767+0000 7fecccecd640 1 --2- 172.21.15.114:0/652575493 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fecc8074cd0 0x7fecc81a6890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T10:35:42.768 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:35:42.767+0000 7fecc77fe640 1 --2- 172.21.15.114:0/652575493 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fecc81072a0 0x7fecc8074790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T10:35:42.768 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:35:42.767+0000 7fecc7fff640 1 -- 172.21.15.114:0/652575493 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fecc8074cd0 msgr2=0x7fecc81a6890 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T10:35:42.768 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:35:42.767+0000 7fecc7fff640 1 --2- 172.21.15.114:0/652575493 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fecc8074cd0 0x7fecc81a6890 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:35:42.768 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:35:42.767+0000 7fecc7fff640 1 -- 172.21.15.114:0/652575493 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fecc81072a0 msgr2=0x7fecc8074790 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T10:35:42.768 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:35:42.767+0000 7fecc7fff640 1 --2- 172.21.15.114:0/652575493 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fecc81072a0 0x7fecc8074790 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:35:42.768 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:35:42.767+0000 7fecc7fff640 1 -- 172.21.15.114:0/652575493 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fecc81a6f90 con 0x7fecc81021d0 2024-04-23T10:35:42.768 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:35:42.767+0000 7fecc7fff640 1 --2- 172.21.15.114:0/652575493 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fecc81021d0 0x7fecc8074250 secure :-1 s=READY pgs=810 cs=0 l=1 rev1=1 crypto rx=0x7fecb800c890 tx=0x7fecb800cd50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T10:35:42.768 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:35:42.767+0000 7fecc57fa640 1 -- 172.21.15.114:0/652575493 <== mon.0 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fecb8004220 con 0x7fecc81021d0 2024-04-23T10:35:42.769 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:35:42.767+0000 7fecc57fa640 1 -- 172.21.15.114:0/652575493 <== mon.0 v2:172.21.15.114:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fecb80043c0 con 0x7fecc81021d0 2024-04-23T10:35:42.769 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:35:42.767+0000 7fecce930640 1 -- 172.21.15.114:0/652575493 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fecc81a7280 con 0x7fecc81021d0 2024-04-23T10:35:42.769 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:35:42.767+0000 7fecc57fa640 1 -- 172.21.15.114:0/652575493 <== mon.0 v2:172.21.15.114:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fecb8004a00 con 0x7fecc81021d0 2024-04-23T10:35:42.769 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:35:42.767+0000 7fecce930640 1 -- 172.21.15.114:0/652575493 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fecc81a77a0 con 0x7fecc81021d0 2024-04-23T10:35:42.775 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:35:42.770+0000 7fecc57fa640 1 -- 172.21.15.114:0/652575493 <== mon.0 v2:172.21.15.114:3300/0 4 ==== mgrmap(e 26) ==== 138778+0+0 (secure 0 0 0) 0x7fecb80079b0 con 0x7fecc81021d0 2024-04-23T10:35:42.776 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:35:42.770+0000 7fecd164a740 1 Processor -- start 2024-04-23T10:35:42.776 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:35:42.770+0000 7fecd164a740 1 -- start start 2024-04-23T10:35:42.776 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:35:42.770+0000 7fecd164a740 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x56401832b620 0x56401832ba00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T10:35:42.776 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:35:42.770+0000 7fecd164a740 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x56401832bf40 0x5640183362c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T10:35:42.776 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:35:42.770+0000 7fecd164a740 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x564018336800 0x564018338bf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T10:35:42.776 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:35:42.770+0000 7fecd164a740 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x564018290970 con 0x56401832b620 2024-04-23T10:35:42.776 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:35:42.770+0000 7fecd164a740 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 -- 0x56401822d4a0 con 0x564018336800 2024-04-23T10:35:42.776 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:35:42.770+0000 7fecd164a740 1 -- --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_getmap magic: 0 -- 0x56401828c930 con 0x56401832bf40 2024-04-23T10:35:42.776 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:35:42.770+0000 7fecc57fa640 1 --2- 172.21.15.114:0/652575493 >> [v2:172.21.15.114:6800/1395693466,v1:172.21.15.114:6801/1395693466] conn(0x7fecb00a3df0 0x7fecb00a62b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T10:35:42.776 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:35:42.770+0000 7fecccecd640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x56401832b620 0x56401832ba00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T10:35:42.776 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:35:42.770+0000 7fecccecd640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x56401832b620 0x56401832ba00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.114:60858/0 (socket says 172.21.15.114:60858) 2024-04-23T10:35:42.776 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:35:42.770+0000 7fecccecd640 1 -- 172.21.15.114:0/178975218 learned_addr learned my addr 172.21.15.114:0/178975218 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-04-23T10:35:42.776 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:35:42.770+0000 7fecc7fff640 1 --2- 172.21.15.114:0/178975218 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x56401832bf40 0x5640183362c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T10:35:42.776 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:35:42.770+0000 7fecc77fe640 1 --2- 172.21.15.114:0/178975218 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x564018336800 0x564018338bf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T10:35:42.777 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:35:42.770+0000 7fecccecd640 1 --2- 172.21.15.114:0/652575493 >> [v2:172.21.15.114:6800/1395693466,v1:172.21.15.114:6801/1395693466] conn(0x7fecb00a3df0 0x7fecb00a62b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T10:35:42.777 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:35:42.771+0000 7fecc57fa640 1 -- 172.21.15.114:0/652575493 <== mon.0 v2:172.21.15.114:3300/0 5 ==== osd_map(100..100 src has 1..100) ==== 8629+0+0 (secure 0 0 0) 0x7fecb80d0570 con 0x7fecc81021d0 2024-04-23T10:35:42.777 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:35:42.771+0000 7fecccecd640 1 -- 172.21.15.114:0/178975218 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x56401832bf40 msgr2=0x5640183362c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T10:35:42.777 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:35:42.771+0000 7fecccecd640 1 --2- 172.21.15.114:0/178975218 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x56401832bf40 0x5640183362c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:35:42.777 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:35:42.771+0000 7fecccecd640 1 -- 172.21.15.114:0/178975218 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x564018336800 msgr2=0x564018338bf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T10:35:42.777 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:35:42.771+0000 7fecccecd640 1 --2- 172.21.15.114:0/178975218 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x564018336800 0x564018338bf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:35:42.777 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:35:42.771+0000 7fecccecd640 1 -- 172.21.15.114:0/178975218 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x564017fa3210 con 0x56401832b620 2024-04-23T10:35:42.777 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:35:42.771+0000 7fecccecd640 1 --2- 172.21.15.114:0/178975218 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x56401832b620 0x56401832ba00 secure :-1 s=READY pgs=811 cs=0 l=1 rev1=1 crypto rx=0x7fecc0005f40 tx=0x7fecc0014e50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a4258388252952e0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T10:35:42.777 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:35:42.771+0000 7fecae7fc640 1 -- 172.21.15.114:0/178975218 <== mon.0 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fecc0004d20 con 0x56401832b620 2024-04-23T10:35:42.777 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:35:42.771+0000 7fecae7fc640 1 -- 172.21.15.114:0/178975218 <== mon.0 v2:172.21.15.114:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fecc0023070 con 0x56401832b620 2024-04-23T10:35:42.777 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:35:42.771+0000 7fecae7fc640 1 -- 172.21.15.114:0/178975218 <== mon.0 v2:172.21.15.114:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fecc001d810 con 0x56401832b620 2024-04-23T10:35:42.777 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:35:42.771+0000 7fecccecd640 1 --2- 172.21.15.114:0/652575493 >> [v2:172.21.15.114:6800/1395693466,v1:172.21.15.114:6801/1395693466] conn(0x7fecb00a3df0 0x7fecb00a62b0 secure :-1 s=READY pgs=125 cs=0 l=1 rev1=1 crypto rx=0x7fecc0022000 tx=0x7fecc0020000 comp rx=0 tx=0).ready entity=mgr.15291 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T10:35:42.777 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:35:42.772+0000 7fecd164a740 1 -- 172.21.15.114:0/178975218 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x56401832b620 msgr2=0x56401832ba00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T10:35:42.777 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:35:42.772+0000 7fecd164a740 1 --2- 172.21.15.114:0/178975218 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x56401832b620 0x56401832ba00 secure :-1 s=READY pgs=811 cs=0 l=1 rev1=1 crypto rx=0x7fecc0005f40 tx=0x7fecc0014e50 comp rx=0 tx=0).stop 2024-04-23T10:35:42.777 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:35:42.772+0000 7fecd164a740 1 -- 172.21.15.114:0/178975218 shutdown_connections 2024-04-23T10:35:42.778 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:35:42.772+0000 7fecd164a740 1 --2- 172.21.15.114:0/178975218 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x564018336800 0x564018338bf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:35:42.778 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:35:42.772+0000 7fecd164a740 1 --2- 172.21.15.114:0/178975218 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x56401832bf40 0x5640183362c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:35:42.778 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:35:42.772+0000 7fecd164a740 1 --2- 172.21.15.114:0/178975218 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x56401832b620 0x56401832ba00 unknown :-1 s=CLOSED pgs=811 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:35:42.778 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:35:42.772+0000 7fecd164a740 1 -- 172.21.15.114:0/178975218 >> 172.21.15.114:0/178975218 conn(0x564018327b30 msgr2=0x564018327f10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T10:35:42.778 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:35:42.772+0000 7fecd164a740 1 -- 172.21.15.114:0/178975218 shutdown_connections 2024-04-23T10:35:42.778 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:35:42.772+0000 7fecd164a740 1 -- 172.21.15.114:0/178975218 wait complete. 2024-04-23T10:35:42.778 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:35:42.772+0000 7fecd164a740 1 -- start start 2024-04-23T10:35:42.778 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:35:42.772+0000 7fecd164a740 1 Processor -- start 2024-04-23T10:35:42.778 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:35:42.773+0000 7fecd164a740 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x56401832b620 0x564018336160 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T10:35:42.778 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:35:42.773+0000 7fecd164a740 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x56401832bf40 0x564018333450 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T10:35:42.778 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:35:42.773+0000 7fecd164a740 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x564018336800 0x5640183339b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T10:35:42.778 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:35:42.773+0000 7fecd164a740 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x564018339430 con 0x56401832bf40 2024-04-23T10:35:42.778 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:35:42.773+0000 7fecd164a740 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 -- 0x56401822b0b0 con 0x56401832b620 2024-04-23T10:35:42.778 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:35:42.773+0000 7fecd164a740 1 -- --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_getmap magic: 0 -- 0x5640183395b0 con 0x564018336800 2024-04-23T10:35:42.779 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:35:42.773+0000 7fecd164a740 10 client.0 _ll_register_callbacks cb 0x564018218d30 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-23T10:35:42.779 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:35:42.773+0000 7fecc77fe640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x56401832bf40 0x564018333450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T10:35:42.779 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:35:42.773+0000 7fecc77fe640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x56401832bf40 0x564018333450 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.114:60868/0 (socket says 172.21.15.114:60868) 2024-04-23T10:35:42.779 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:35:42.773+0000 7fecc77fe640 1 -- 172.21.15.114:0/3759397048 learned_addr learned my addr 172.21.15.114:0/3759397048 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-04-23T10:35:42.779 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:35:42.773+0000 7fecccecd640 1 --2- 172.21.15.114:0/3759397048 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x564018336800 0x5640183339b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T10:35:42.779 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:35:42.773+0000 7fecc7fff640 1 --2- 172.21.15.114:0/3759397048 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x56401832b620 0x564018336160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T10:35:42.779 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:35:42.773+0000 7fecc77fe640 1 -- 172.21.15.114:0/3759397048 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x564018336800 msgr2=0x5640183339b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T10:35:42.779 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:35:42.773+0000 7fecc77fe640 1 --2- 172.21.15.114:0/3759397048 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x564018336800 0x5640183339b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:35:42.779 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:35:42.773+0000 7fecc77fe640 1 -- 172.21.15.114:0/3759397048 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x56401832b620 msgr2=0x564018336160 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T10:35:42.779 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:35:42.773+0000 7fecc77fe640 1 --2- 172.21.15.114:0/3759397048 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x56401832b620 0x564018336160 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:35:42.779 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:35:42.773+0000 7fecc77fe640 1 -- 172.21.15.114:0/3759397048 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({osdmap=0}) -- 0x56401806f620 con 0x56401832bf40 2024-04-23T10:35:42.779 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:35:42.773+0000 7fecc7fff640 1 --2- 172.21.15.114:0/3759397048 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x56401832b620 0x564018336160 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-23T10:35:42.779 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:35:42.773+0000 7fecc77fe640 1 --2- 172.21.15.114:0/3759397048 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x56401832bf40 0x564018333450 secure :-1 s=READY pgs=812 cs=0 l=1 rev1=1 crypto rx=0x7fecbc009900 tx=0x7fecbc031740 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7a50327e650719c6 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T10:35:42.779 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:35:42.773+0000 7feca77fe640 10 client.0 ms_handle_connect on v2:172.21.15.114:3300/0 2024-04-23T10:35:42.779 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:35:42.773+0000 7feca77fe640 1 -- 172.21.15.114:0/3759397048 <== mon.0 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fecbc035040 con 0x56401832bf40 2024-04-23T10:35:42.780 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:35:42.773+0000 7feca77fe640 1 -- 172.21.15.114:0/3759397048 <== mon.0 v2:172.21.15.114:3300/0 2 ==== osd_map(100..100 src has 1..100) ==== 8629+0+0 (secure 0 0 0) 0x7fecbc004240 con 0x56401832bf40 2024-04-23T10:35:42.780 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:35:42.773+0000 7fecd164a740 1 -- 172.21.15.114:0/3759397048 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fecc0023070 con 0x56401832bf40 2024-04-23T10:35:42.780 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:35:42.774+0000 7feca77fe640 1 -- 172.21.15.114:0/3759397048 <== mon.0 v2:172.21.15.114:3300/0 3 ==== mon_get_version_reply(handle=1 version=33) ==== 24+0+0 (secure 0 0 0) 0x7fecbc004440 con 0x56401832bf40 2024-04-23T10:35:42.780 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:35:42.774+0000 7fecd164a740 10 client.16692 fetch_fsmap learned FSMap version 33 2024-04-23T10:35:42.780 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:35:42.774+0000 7fecd164a740 1 -- 172.21.15.114:0/3759397048 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({config=0+,fsmap=33,monmap=4+}) -- 0x564018345360 con 0x56401832bf40 2024-04-23T10:35:42.780 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:35:42.774+0000 7feca77fe640 1 -- 172.21.15.114:0/3759397048 <== mon.0 v2:172.21.15.114:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fecc0023070 con 0x56401832bf40 2024-04-23T10:35:42.780 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:35:42.774+0000 7feca77fe640 1 -- 172.21.15.114:0/3759397048 <== mon.0 v2:172.21.15.114:3300/0 5 ==== fsmap(e 33) ==== 8342+0+0 (secure 0 0 0) 0x7fecbc004b80 con 0x56401832bf40 2024-04-23T10:35:42.780 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:35:42.774+0000 7fecd164a740 10 client.16692 fetch_fsmap finished waiting for FSMap version 33 2024-04-23T10:35:42.780 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:35:42.774+0000 7fecd164a740 10 client.16692 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-04-23T10:35:42.780 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:35:42.774+0000 7fecd164a740 20 client.16692 populate_metadata read hostname 'smithi114' 2024-04-23T10:35:42.780 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:35:42.774+0000 7fecd164a740 1 --2- 172.21.15.114:0/3759397048 >> [v2:172.21.15.119:6838/937297340,v1:172.21.15.119:6839/937297340] conn(0x564018346820 0x564018348c30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T10:35:42.780 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:35:42.774+0000 7fecd164a740 4 client.16692 mds_command: new command op to 24545 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-23T10:35:42.780 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:35:42.774+0000 7fecd164a740 1 -- 172.21.15.114:0/3759397048 --> [v2:172.21.15.119:6838/937297340,v1:172.21.15.119:6839/937297340] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x56401829e390 con 0x564018346820 2024-04-23T10:35:42.780 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:35:42.774+0000 7fecc7fff640 1 --2- 172.21.15.114:0/3759397048 >> [v2:172.21.15.119:6838/937297340,v1:172.21.15.119:6839/937297340] conn(0x564018346820 0x564018348c30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T10:35:42.780 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:35:42.775+0000 7fecc7fff640 1 --2- 172.21.15.114:0/3759397048 >> [v2:172.21.15.119:6838/937297340,v1:172.21.15.119:6839/937297340] conn(0x564018346820 0x564018348c30 crc :-1 s=READY pgs=142 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=a761f6148340584f server_cookie=a4c2a88cbc87544e in_seq=0 out_seq=0 2024-04-23T10:35:42.781 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:35:42.775+0000 7feca77fe640 10 client.16692 ms_handle_connect on v2:172.21.15.119:6838/937297340 2024-04-23T10:35:42.781 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:35:42.775+0000 7feca77fe640 1 -- 172.21.15.114:0/3759397048 <== mds.0 v2:172.21.15.119:6838/937297340 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x56401829e390 con 0x564018346820 2024-04-23T10:35:42.781 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:35:42.775+0000 7feca77fe640 10 client.16692 handle_command_reply: tid=0 2024-04-23T10:35:42.781 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:35:42.775+0000 7fecd164a740 1 client.16692 shutdown 2024-04-23T10:35:42.781 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:35:42.776+0000 7fecd164a740 1 -- 172.21.15.114:0/3759397048 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x56401832bf40 msgr2=0x564018333450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T10:35:42.781 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:35:42.776+0000 7fecd164a740 1 --2- 172.21.15.114:0/3759397048 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x56401832bf40 0x564018333450 secure :-1 s=READY pgs=812 cs=0 l=1 rev1=1 crypto rx=0x7fecbc009900 tx=0x7fecbc031740 comp rx=0 tx=0).stop 2024-04-23T10:35:42.781 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:35:42.776+0000 7fecd164a740 1 -- 172.21.15.114:0/3759397048 shutdown_connections 2024-04-23T10:35:42.781 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:35:42.776+0000 7fecd164a740 1 --2- 172.21.15.114:0/3759397048 >> [v2:172.21.15.119:6838/937297340,v1:172.21.15.119:6839/937297340] conn(0x564018346820 0x564018348c30 crc :-1 s=READY pgs=142 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:35:42.781 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:35:42.776+0000 7fecd164a740 1 --2- 172.21.15.114:0/3759397048 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x564018336800 0x5640183339b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:35:42.781 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:35:42.776+0000 7fecd164a740 1 --2- 172.21.15.114:0/3759397048 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x56401832bf40 0x564018333450 unknown :-1 s=CLOSED pgs=812 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:35:42.781 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:35:42.776+0000 7feca77fe640 0 client.16692 ms_handle_reset on v2:172.21.15.119:6838/937297340 2024-04-23T10:35:42.781 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:35:42.776+0000 7fecd164a740 1 --2- 172.21.15.114:0/3759397048 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x56401832b620 0x564018336160 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:35:42.781 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:35:42.776+0000 7fecd164a740 1 -- 172.21.15.114:0/3759397048 >> 172.21.15.114:0/3759397048 conn(0x564018327b30 msgr2=0x56401832d860 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T10:35:42.781 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:35:42.776+0000 7fecd164a740 1 -- 172.21.15.114:0/3759397048 shutdown_connections 2024-04-23T10:35:42.781 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:35:42.776+0000 7fecd164a740 1 -- 172.21.15.114:0/3759397048 wait complete. 2024-04-23T10:35:42.782 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:35:42.776+0000 7fecd164a740 20 client.16692 trim_cache size 0 max 16384 2024-04-23T10:35:42.794 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:35:42.790+0000 7fecd164a740 1 Processor -- start 2024-04-23T10:35:42.794 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:35:42.790+0000 7fecd164a740 1 -- start start 2024-04-23T10:35:42.794 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:35:42.791+0000 7fecd164a740 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x56401832b620 0x5640183422f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T10:35:42.795 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:35:42.791+0000 7fecd164a740 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x56401832bf40 0x56401834e540 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T10:35:42.795 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:35:42.791+0000 7fecd164a740 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x564018336800 0x56401834ea80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T10:35:42.795 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:35:42.791+0000 7fecd164a740 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x7fecc81b2880 con 0x56401832b620 2024-04-23T10:35:42.795 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:35:42.791+0000 7fecd164a740 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 -- 0x7fecc81b0150 con 0x564018336800 2024-04-23T10:35:42.795 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:35:42.791+0000 7fecd164a740 1 -- --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_getmap magic: 0 -- 0x7fecc81adfc0 con 0x56401832bf40 2024-04-23T10:35:42.795 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:35:42.791+0000 7fecc7fff640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x56401832b620 0x5640183422f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T10:35:42.795 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:35:42.791+0000 7fecc7fff640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x56401832b620 0x5640183422f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.114:60874/0 (socket says 172.21.15.114:60874) 2024-04-23T10:35:42.795 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:35:42.791+0000 7fecc7fff640 1 -- 172.21.15.114:0/1435065017 learned_addr learned my addr 172.21.15.114:0/1435065017 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-04-23T10:35:42.795 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:35:42.791+0000 7fecc7fff640 1 -- 172.21.15.114:0/1435065017 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x56401832bf40 msgr2=0x56401834e540 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T10:35:42.795 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:35:42.791+0000 7fecc77fe640 1 --2- 172.21.15.114:0/1435065017 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x56401832bf40 0x56401834e540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T10:35:42.796 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:35:42.791+0000 7fecc7fff640 1 --2- 172.21.15.114:0/1435065017 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x56401832bf40 0x56401834e540 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:35:42.796 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:35:42.791+0000 7fecc7fff640 1 -- 172.21.15.114:0/1435065017 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x564018336800 msgr2=0x56401834ea80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T10:35:42.796 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:35:42.791+0000 7fecc7fff640 1 --2- 172.21.15.114:0/1435065017 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x564018336800 0x56401834ea80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:35:42.796 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:35:42.791+0000 7fecc7fff640 1 -- 172.21.15.114:0/1435065017 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x564018332400 con 0x56401832b620 2024-04-23T10:35:42.796 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:35:42.791+0000 7fecc7fff640 1 --2- 172.21.15.114:0/1435065017 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x56401832b620 0x5640183422f0 secure :-1 s=READY pgs=813 cs=0 l=1 rev1=1 crypto rx=0x7fecb809c430 tx=0x7fecb8004570 comp rx=0 tx=0).ready entity=mon.0 client_cookie=47f4f54b26559916 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T10:35:42.796 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:35:42.791+0000 7fecadffb640 1 -- 172.21.15.114:0/1435065017 <== mon.0 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fecb801d040 con 0x56401832b620 2024-04-23T10:35:42.796 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:35:42.791+0000 7fecadffb640 1 -- 172.21.15.114:0/1435065017 <== mon.0 v2:172.21.15.114:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fecb80cd040 con 0x56401832b620 2024-04-23T10:35:42.796 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:35:42.791+0000 7fecadffb640 1 -- 172.21.15.114:0/1435065017 <== mon.0 v2:172.21.15.114:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fecb80cded0 con 0x56401832b620 2024-04-23T10:35:42.796 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:35:42.792+0000 7fecd164a740 1 -- 172.21.15.114:0/1435065017 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x56401832b620 msgr2=0x5640183422f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T10:35:42.796 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:35:42.792+0000 7fecd164a740 1 --2- 172.21.15.114:0/1435065017 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x56401832b620 0x5640183422f0 secure :-1 s=READY pgs=813 cs=0 l=1 rev1=1 crypto rx=0x7fecb809c430 tx=0x7fecb8004570 comp rx=0 tx=0).stop 2024-04-23T10:35:42.796 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:35:42.792+0000 7fecd164a740 1 -- 172.21.15.114:0/1435065017 shutdown_connections 2024-04-23T10:35:42.796 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:35:42.792+0000 7fecd164a740 1 --2- 172.21.15.114:0/1435065017 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x564018336800 0x56401834ea80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:35:42.796 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:35:42.792+0000 7fecd164a740 1 --2- 172.21.15.114:0/1435065017 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x56401832bf40 0x56401834e540 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:35:42.796 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:35:42.792+0000 7fecd164a740 1 --2- 172.21.15.114:0/1435065017 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x56401832b620 0x5640183422f0 unknown :-1 s=CLOSED pgs=813 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:35:42.797 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:35:42.792+0000 7fecd164a740 1 -- 172.21.15.114:0/1435065017 >> 172.21.15.114:0/1435065017 conn(0x564018327b30 msgr2=0x564018326cb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T10:35:42.797 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:35:42.792+0000 7fecd164a740 1 -- 172.21.15.114:0/1435065017 shutdown_connections 2024-04-23T10:35:42.797 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:35:42.792+0000 7fecd164a740 1 -- 172.21.15.114:0/1435065017 wait complete. 2024-04-23T10:35:42.797 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:35:42.792+0000 7fecd164a740 1 -- start start 2024-04-23T10:35:42.797 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:35:42.793+0000 7fecd164a740 1 Processor -- start 2024-04-23T10:35:42.797 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:35:42.793+0000 7fecd164a740 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x56401832b620 0x56401833ca00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T10:35:42.797 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:35:42.793+0000 7fecd164a740 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x56401832bf40 0x56401833cf40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T10:35:42.797 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:35:42.793+0000 7fecd164a740 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x564018336800 0x5640183b45d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T10:35:42.797 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:35:42.793+0000 7fecd164a740 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x7fecc81ae610 con 0x56401832bf40 2024-04-23T10:35:42.797 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:35:42.793+0000 7fecd164a740 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 -- 0x7fecc81afa80 con 0x564018336800 2024-04-23T10:35:42.797 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:35:42.793+0000 7fecd164a740 1 -- --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_getmap magic: 0 -- 0x7fecc81aece0 con 0x56401832b620 2024-04-23T10:35:42.797 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:35:42.793+0000 7fecd164a740 10 client.0 _ll_register_callbacks cb 0x7fecbc005e20 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-23T10:35:42.797 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:35:42.793+0000 7fecc77fe640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x56401832bf40 0x56401833cf40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T10:35:42.797 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:35:42.793+0000 7fecc77fe640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x56401832bf40 0x56401833cf40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.114:60890/0 (socket says 172.21.15.114:60890) 2024-04-23T10:35:42.798 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:35:42.793+0000 7fecc77fe640 1 -- 172.21.15.114:0/2067235035 learned_addr learned my addr 172.21.15.114:0/2067235035 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-04-23T10:35:42.798 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:35:42.793+0000 7fecc7fff640 1 --2- 172.21.15.114:0/2067235035 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x56401832b620 0x56401833ca00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T10:35:42.798 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:35:42.793+0000 7fecc77fe640 1 -- 172.21.15.114:0/2067235035 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x56401832b620 msgr2=0x56401833ca00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T10:35:42.798 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:35:42.793+0000 7fecccecd640 1 --2- 172.21.15.114:0/2067235035 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x564018336800 0x5640183b45d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T10:35:42.798 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:35:42.793+0000 7fecc77fe640 1 --2- 172.21.15.114:0/2067235035 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x56401832b620 0x56401833ca00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:35:42.798 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:35:42.793+0000 7fecc77fe640 1 -- 172.21.15.114:0/2067235035 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x564018336800 msgr2=0x5640183b45d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T10:35:42.798 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:35:42.793+0000 7fecc77fe640 1 --2- 172.21.15.114:0/2067235035 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x564018336800 0x5640183b45d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:35:42.798 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:35:42.793+0000 7fecc77fe640 1 -- 172.21.15.114:0/2067235035 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({osdmap=0}) -- 0x56401832bd70 con 0x56401832bf40 2024-04-23T10:35:42.798 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:35:42.793+0000 7fecc77fe640 1 --2- 172.21.15.114:0/2067235035 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x56401832bf40 0x56401833cf40 secure :-1 s=READY pgs=814 cs=0 l=1 rev1=1 crypto rx=0x7fecbc0036e0 tx=0x7fecbc0319f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=bf7766a0f20adf20 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T10:35:42.798 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:35:42.793+0000 7feca77fe640 10 client.0 ms_handle_connect on v2:172.21.15.114:3300/0 2024-04-23T10:35:42.798 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:35:42.793+0000 7feca77fe640 1 -- 172.21.15.114:0/2067235035 <== mon.0 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fecbc035050 con 0x56401832bf40 2024-04-23T10:35:42.798 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:35:42.794+0000 7feca77fe640 1 -- 172.21.15.114:0/2067235035 <== mon.0 v2:172.21.15.114:3300/0 2 ==== osd_map(100..100 src has 1..100) ==== 8629+0+0 (secure 0 0 0) 0x7fecbc0048c0 con 0x56401832bf40 2024-04-23T10:35:42.798 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:35:42.794+0000 7fecd164a740 1 -- 172.21.15.114:0/2067235035 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fecb80cd040 con 0x56401832bf40 2024-04-23T10:35:42.798 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:35:42.794+0000 7feca77fe640 1 -- 172.21.15.114:0/2067235035 <== mon.0 v2:172.21.15.114:3300/0 3 ==== mon_get_version_reply(handle=1 version=33) ==== 24+0+0 (secure 0 0 0) 0x7fecbc005610 con 0x56401832bf40 2024-04-23T10:35:42.798 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:35:42.794+0000 7fecd164a740 10 client.16698 fetch_fsmap learned FSMap version 33 2024-04-23T10:35:42.799 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:35:42.794+0000 7fecd164a740 1 -- 172.21.15.114:0/2067235035 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({config=0+,fsmap=33,monmap=4+}) -- 0x5640183b5ce0 con 0x56401832bf40 2024-04-23T10:35:42.799 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:35:42.794+0000 7feca77fe640 1 -- 172.21.15.114:0/2067235035 <== mon.0 v2:172.21.15.114:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fecb80cd040 con 0x56401832bf40 2024-04-23T10:35:42.799 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:35:42.794+0000 7feca77fe640 1 -- 172.21.15.114:0/2067235035 <== mon.0 v2:172.21.15.114:3300/0 5 ==== fsmap(e 33) ==== 8342+0+0 (secure 0 0 0) 0x7fecbc003d20 con 0x56401832bf40 2024-04-23T10:35:42.799 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:35:42.794+0000 7fecd164a740 10 client.16698 fetch_fsmap finished waiting for FSMap version 33 2024-04-23T10:35:42.799 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:35:42.794+0000 7fecd164a740 10 client.16698 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-04-23T10:35:42.799 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:35:42.794+0000 7fecd164a740 20 client.16698 populate_metadata read hostname 'smithi114' 2024-04-23T10:35:42.799 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:35:42.794+0000 7fecd164a740 1 --2- 172.21.15.114:0/2067235035 >> [v2:172.21.15.119:6838/937297340,v1:172.21.15.119:6839/937297340] conn(0x5640183b6430 0x5640183b8850 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T10:35:42.799 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:35:42.794+0000 7fecd164a740 4 client.16698 mds_command: new command op to 24545 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-04-23T10:35:42.799 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:35:42.794+0000 7fecd164a740 1 -- 172.21.15.114:0/2067235035 --> [v2:172.21.15.119:6838/937297340,v1:172.21.15.119:6839/937297340] -- command(tid 0: {"prefix": "flush journal"}) -- 0x564018298200 con 0x5640183b6430 2024-04-23T10:35:42.799 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:35:42.794+0000 7fecc7fff640 1 --2- 172.21.15.114:0/2067235035 >> [v2:172.21.15.119:6838/937297340,v1:172.21.15.119:6839/937297340] conn(0x5640183b6430 0x5640183b8850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T10:35:42.799 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:35:42.795+0000 7feca77fe640 10 client.16698 ms_handle_connect on v2:172.21.15.119:6838/937297340 2024-04-23T10:35:42.799 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:35:42.795+0000 7fecc7fff640 1 --2- 172.21.15.114:0/2067235035 >> [v2:172.21.15.119:6838/937297340,v1:172.21.15.119:6839/937297340] conn(0x5640183b6430 0x5640183b8850 crc :-1 s=READY pgs=143 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=83dda39c88ccb38b server_cookie=8ec8cd2407071669 in_seq=0 out_seq=0 2024-04-23T10:35:42.849 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:35:42.849+0000 7feca77fe640 1 -- 172.21.15.114:0/2067235035 <== mds.0 v2:172.21.15.119:6838/937297340 1 ==== command_reply(tid 0: 0 ) ==== 8+0+44 (crc 0 0 0) 0x564018298200 con 0x5640183b6430 2024-04-23T10:35:42.849 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:35:42.849+0000 7feca77fe640 10 client.16698 handle_command_reply: tid=0 2024-04-23T10:35:42.849 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:35:42.849+0000 7fecd164a740 1 client.16698 shutdown 2024-04-23T10:35:42.849 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:35:42.850+0000 7fecd164a740 1 -- 172.21.15.114:0/2067235035 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x56401832bf40 msgr2=0x56401833cf40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T10:35:42.849 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:35:42.850+0000 7fecd164a740 1 --2- 172.21.15.114:0/2067235035 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x56401832bf40 0x56401833cf40 secure :-1 s=READY pgs=814 cs=0 l=1 rev1=1 crypto rx=0x7fecbc0036e0 tx=0x7fecbc0319f0 comp rx=0 tx=0).stop 2024-04-23T10:35:42.849 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:35:42.850+0000 7fecd164a740 1 -- 172.21.15.114:0/2067235035 shutdown_connections 2024-04-23T10:35:42.849 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:35:42.850+0000 7fecd164a740 1 --2- 172.21.15.114:0/2067235035 >> [v2:172.21.15.119:6838/937297340,v1:172.21.15.119:6839/937297340] conn(0x5640183b6430 0x5640183b8850 crc :-1 s=READY pgs=143 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:35:42.849 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:35:42.850+0000 7fecd164a740 1 --2- 172.21.15.114:0/2067235035 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x564018336800 0x5640183b45d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:35:42.849 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:35:42.850+0000 7fecd164a740 1 --2- 172.21.15.114:0/2067235035 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x56401832bf40 0x56401833cf40 unknown :-1 s=CLOSED pgs=814 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:35:42.849 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:35:42.850+0000 7feca77fe640 0 client.16698 ms_handle_reset on v2:172.21.15.119:6838/937297340 2024-04-23T10:35:42.849 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:35:42.850+0000 7fecd164a740 1 --2- 172.21.15.114:0/2067235035 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x56401832b620 0x56401833ca00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:35:42.850 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:35:42.850+0000 7fecd164a740 1 -- 172.21.15.114:0/2067235035 >> 172.21.15.114:0/2067235035 conn(0x564018327b30 msgr2=0x5640183412e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T10:35:42.850 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:35:42.850+0000 7fecd164a740 1 -- 172.21.15.114:0/2067235035 shutdown_connections 2024-04-23T10:35:42.850 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:35:42.850+0000 7fecd164a740 1 -- 172.21.15.114:0/2067235035 wait complete. 2024-04-23T10:35:42.850 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:35:42.850+0000 7fecd164a740 20 client.16698 trim_cache size 0 max 16384 2024-04-23T10:35:42.850 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stdout:{ 2024-04-23T10:35:42.850 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stdout: "message": "", 2024-04-23T10:35:42.850 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stdout: "return_code": 0 2024-04-23T10:35:42.850 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stdout:} 2024-04-23T10:35:42.851 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:35:42.850+0000 7fecce930640 1 -- 172.21.15.114:0/652575493 >> [v2:172.21.15.114:6800/1395693466,v1:172.21.15.114:6801/1395693466] conn(0x7fecb00a3df0 msgr2=0x7fecb00a62b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T10:35:42.851 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:35:42.850+0000 7fecce930640 1 --2- 172.21.15.114:0/652575493 >> [v2:172.21.15.114:6800/1395693466,v1:172.21.15.114:6801/1395693466] conn(0x7fecb00a3df0 0x7fecb00a62b0 secure :-1 s=READY pgs=125 cs=0 l=1 rev1=1 crypto rx=0x7fecc0022000 tx=0x7fecc0020000 comp rx=0 tx=0).stop 2024-04-23T10:35:42.851 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:35:42.850+0000 7fecce930640 1 -- 172.21.15.114:0/652575493 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fecc81021d0 msgr2=0x7fecc8074250 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T10:35:42.851 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:35:42.850+0000 7fecce930640 1 --2- 172.21.15.114:0/652575493 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fecc81021d0 0x7fecc8074250 secure :-1 s=READY pgs=810 cs=0 l=1 rev1=1 crypto rx=0x7fecb800c890 tx=0x7fecb800cd50 comp rx=0 tx=0).stop 2024-04-23T10:35:42.851 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:35:42.851+0000 7fecce930640 1 -- 172.21.15.114:0/652575493 shutdown_connections 2024-04-23T10:35:42.851 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:35:42.851+0000 7fecce930640 1 --2- 172.21.15.114:0/652575493 >> [v2:172.21.15.114:6800/1395693466,v1:172.21.15.114:6801/1395693466] conn(0x7fecb00a3df0 0x7fecb00a62b0 unknown :-1 s=CLOSED pgs=125 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:35:42.851 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:35:42.851+0000 7fecce930640 1 --2- 172.21.15.114:0/652575493 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fecc8074cd0 0x7fecc81a6890 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:35:42.852 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:35:42.851+0000 7fecce930640 1 --2- 172.21.15.114:0/652575493 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fecc81072a0 0x7fecc8074790 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:35:42.852 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:35:42.851+0000 7fecce930640 1 --2- 172.21.15.114:0/652575493 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fecc81021d0 0x7fecc8074250 unknown :-1 s=CLOSED pgs=810 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:35:42.852 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:35:42.851+0000 7fecce930640 1 -- 172.21.15.114:0/652575493 >> 172.21.15.114:0/652575493 conn(0x7fecc80793e0 msgr2=0x7fecc80ffe00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T10:35:42.852 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:35:42.851+0000 7fecce930640 1 -- 172.21.15.114:0/652575493 shutdown_connections 2024-04-23T10:35:42.852 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:35:42.851+0000 7fecce930640 1 -- 172.21.15.114:0/652575493 wait complete. 2024-04-23T10:35:42.933 INFO:tasks.workunit.client.0.smithi114.stdout: 1 635780 7.08 MB/sec execute 351 sec latency 2016.012 ms 2024-04-23T10:35:43.828 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 23 10:35:43 smithi204 ceph-mon[32649]: pgmap v284: 97 pgs: 97 active+clean; 195 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail; 676 KiB/s rd, 1.9 MiB/s wr, 159 op/s 2024-04-23T10:35:43.934 INFO:tasks.workunit.client.0.smithi114.stdout: 1 635780 7.06 MB/sec execute 352 sec latency 3016.142 ms 2024-04-23T10:35:43.992 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 23 10:35:43 smithi119 ceph-mon[32608]: pgmap v284: 97 pgs: 97 active+clean; 195 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail; 676 KiB/s rd, 1.9 MiB/s wr, 159 op/s 2024-04-23T10:35:44.003 INFO:journalctl@ceph.mon.a.smithi114.stdout:Apr 23 10:35:43 smithi114 ceph-mon[29255]: pgmap v284: 97 pgs: 97 active+clean; 195 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail; 676 KiB/s rd, 1.9 MiB/s wr, 159 op/s 2024-04-23T10:35:44.934 INFO:tasks.workunit.client.0.smithi114.stdout: 1 635847 7.04 MB/sec execute 353 sec latency 3267.118 ms 2024-04-23T10:35:45.828 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 23 10:35:45 smithi204 ceph-mon[32649]: pgmap v285: 97 pgs: 97 active+clean; 195 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail; 381 KiB/s rd, 970 KiB/s wr, 79 op/s 2024-04-23T10:35:45.934 INFO:tasks.workunit.client.0.smithi114.stdout: 1 635847 7.02 MB/sec execute 354 sec latency 1652.620 ms 2024-04-23T10:35:45.992 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 23 10:35:45 smithi119 ceph-mon[32608]: pgmap v285: 97 pgs: 97 active+clean; 195 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail; 381 KiB/s rd, 970 KiB/s wr, 79 op/s 2024-04-23T10:35:46.004 INFO:journalctl@ceph.mon.a.smithi114.stdout:Apr 23 10:35:45 smithi114 ceph-mon[29255]: pgmap v285: 97 pgs: 97 active+clean; 195 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail; 381 KiB/s rd, 970 KiB/s wr, 79 op/s 2024-04-23T10:35:46.934 INFO:tasks.workunit.client.0.smithi114.stdout: 1 635847 7.00 MB/sec execute 355 sec latency 2652.694 ms 2024-04-23T10:35:47.827 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 23 10:35:47 smithi204 ceph-mon[32649]: pgmap v286: 97 pgs: 97 active+clean; 191 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail; 440 KiB/s rd, 1.4 MiB/s wr, 119 op/s 2024-04-23T10:35:47.934 INFO:tasks.workunit.client.0.smithi114.stdout: 1 635847 6.98 MB/sec execute 356 sec latency 3652.801 ms 2024-04-23T10:35:47.992 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 23 10:35:47 smithi119 ceph-mon[32608]: pgmap v286: 97 pgs: 97 active+clean; 191 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail; 440 KiB/s rd, 1.4 MiB/s wr, 119 op/s 2024-04-23T10:35:48.004 INFO:journalctl@ceph.mon.a.smithi114.stdout:Apr 23 10:35:47 smithi114 ceph-mon[29255]: pgmap v286: 97 pgs: 97 active+clean; 191 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail; 440 KiB/s rd, 1.4 MiB/s wr, 119 op/s 2024-04-23T10:35:48.934 INFO:tasks.workunit.client.0.smithi114.stdout: 1 635847 6.96 MB/sec execute 357 sec latency 4652.900 ms 2024-04-23T10:35:49.827 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 23 10:35:49 smithi204 ceph-mon[32649]: pgmap v287: 97 pgs: 97 active+clean; 191 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail; 339 KiB/s rd, 851 KiB/s wr, 86 op/s 2024-04-23T10:35:49.934 INFO:tasks.workunit.client.0.smithi114.stdout: 1 636622 6.95 MB/sec execute 358 sec latency 4897.164 ms 2024-04-23T10:35:49.991 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 23 10:35:49 smithi119 ceph-mon[32608]: pgmap v287: 97 pgs: 97 active+clean; 191 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail; 339 KiB/s rd, 851 KiB/s wr, 86 op/s 2024-04-23T10:35:50.003 INFO:journalctl@ceph.mon.a.smithi114.stdout:Apr 23 10:35:49 smithi114 ceph-mon[29255]: pgmap v287: 97 pgs: 97 active+clean; 191 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail; 339 KiB/s rd, 851 KiB/s wr, 86 op/s 2024-04-23T10:35:50.934 INFO:tasks.workunit.client.0.smithi114.stdout: 1 639252 6.98 MB/sec execute 359 sec latency 16.337 ms 2024-04-23T10:35:51.828 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 23 10:35:51 smithi204 ceph-mon[32649]: pgmap v288: 97 pgs: 97 active+clean; 191 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail; 339 KiB/s rd, 851 KiB/s wr, 86 op/s 2024-04-23T10:35:51.934 INFO:tasks.workunit.client.0.smithi114.stdout: 1 639326 6.96 MB/sec execute 360 sec latency 968.559 ms 2024-04-23T10:35:51.992 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 23 10:35:51 smithi119 ceph-mon[32608]: pgmap v288: 97 pgs: 97 active+clean; 191 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail; 339 KiB/s rd, 851 KiB/s wr, 86 op/s 2024-04-23T10:35:52.004 INFO:journalctl@ceph.mon.a.smithi114.stdout:Apr 23 10:35:51 smithi114 ceph-mon[29255]: pgmap v288: 97 pgs: 97 active+clean; 191 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail; 339 KiB/s rd, 851 KiB/s wr, 86 op/s 2024-04-23T10:35:52.935 INFO:tasks.workunit.client.0.smithi114.stdout: 1 639326 6.94 MB/sec execute 361 sec latency 1968.589 ms 2024-04-23T10:35:53.828 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 23 10:35:53 smithi204 ceph-mon[32649]: pgmap v289: 97 pgs: 97 active+clean; 194 MiB data, 2.5 GiB used, 1.0 TiB / 1.0 TiB avail; 612 KiB/s rd, 1.1 MiB/s wr, 120 op/s 2024-04-23T10:35:53.828 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 23 10:35:53 smithi204 ceph-mon[32649]: from='mgr.15291 172.21.15.114:0/1095789187' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-23T10:35:53.828 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 23 10:35:53 smithi204 ceph-mon[32649]: from='mgr.15291 172.21.15.114:0/1095789187' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-23T10:35:53.828 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 23 10:35:53 smithi204 ceph-mon[32649]: from='mgr.15291 172.21.15.114:0/1095789187' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-23T10:35:53.828 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 23 10:35:53 smithi204 ceph-mon[32649]: from='mgr.15291 172.21.15.114:0/1095789187' entity='mgr.x' 2024-04-23T10:35:53.935 INFO:tasks.workunit.client.0.smithi114.stdout: 1 639326 6.92 MB/sec execute 362 sec latency 2968.673 ms 2024-04-23T10:35:53.992 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 23 10:35:53 smithi119 ceph-mon[32608]: pgmap v289: 97 pgs: 97 active+clean; 194 MiB data, 2.5 GiB used, 1.0 TiB / 1.0 TiB avail; 612 KiB/s rd, 1.1 MiB/s wr, 120 op/s 2024-04-23T10:35:53.992 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 23 10:35:53 smithi119 ceph-mon[32608]: from='mgr.15291 172.21.15.114:0/1095789187' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-23T10:35:53.992 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 23 10:35:53 smithi119 ceph-mon[32608]: from='mgr.15291 172.21.15.114:0/1095789187' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-23T10:35:53.992 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 23 10:35:53 smithi119 ceph-mon[32608]: from='mgr.15291 172.21.15.114:0/1095789187' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-23T10:35:53.992 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 23 10:35:53 smithi119 ceph-mon[32608]: from='mgr.15291 172.21.15.114:0/1095789187' entity='mgr.x' 2024-04-23T10:35:54.004 INFO:journalctl@ceph.mon.a.smithi114.stdout:Apr 23 10:35:53 smithi114 ceph-mon[29255]: pgmap v289: 97 pgs: 97 active+clean; 194 MiB data, 2.5 GiB used, 1.0 TiB / 1.0 TiB avail; 612 KiB/s rd, 1.1 MiB/s wr, 120 op/s 2024-04-23T10:35:54.004 INFO:journalctl@ceph.mon.a.smithi114.stdout:Apr 23 10:35:53 smithi114 ceph-mon[29255]: from='mgr.15291 172.21.15.114:0/1095789187' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-23T10:35:54.004 INFO:journalctl@ceph.mon.a.smithi114.stdout:Apr 23 10:35:53 smithi114 ceph-mon[29255]: from='mgr.15291 172.21.15.114:0/1095789187' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-23T10:35:54.004 INFO:journalctl@ceph.mon.a.smithi114.stdout:Apr 23 10:35:53 smithi114 ceph-mon[29255]: from='mgr.15291 172.21.15.114:0/1095789187' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-23T10:35:54.004 INFO:journalctl@ceph.mon.a.smithi114.stdout:Apr 23 10:35:53 smithi114 ceph-mon[29255]: from='mgr.15291 172.21.15.114:0/1095789187' entity='mgr.x' 2024-04-23T10:35:54.935 INFO:tasks.workunit.client.0.smithi114.stdout: 1 639393 6.90 MB/sec execute 363 sec latency 3216.339 ms 2024-04-23T10:35:55.742 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 23 10:35:55 smithi119 ceph-mon[32608]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=e=up:active} 3 up:standby-replay 6 up:standby 2024-04-23T10:35:55.742 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 23 10:35:55 smithi119 ceph-mon[32608]: pgmap v290: 97 pgs: 97 active+clean; 194 MiB data, 2.5 GiB used, 1.0 TiB / 1.0 TiB avail; 332 KiB/s rd, 798 KiB/s wr, 73 op/s 2024-04-23T10:35:55.754 INFO:journalctl@ceph.mon.a.smithi114.stdout:Apr 23 10:35:55 smithi114 ceph-mon[29255]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=e=up:active} 3 up:standby-replay 6 up:standby 2024-04-23T10:35:55.754 INFO:journalctl@ceph.mon.a.smithi114.stdout:Apr 23 10:35:55 smithi114 ceph-mon[29255]: pgmap v290: 97 pgs: 97 active+clean; 194 MiB data, 2.5 GiB used, 1.0 TiB / 1.0 TiB avail; 332 KiB/s rd, 798 KiB/s wr, 73 op/s 2024-04-23T10:35:55.827 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 23 10:35:55 smithi204 ceph-mon[32649]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=e=up:active} 3 up:standby-replay 6 up:standby 2024-04-23T10:35:55.828 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 23 10:35:55 smithi204 ceph-mon[32649]: pgmap v290: 97 pgs: 97 active+clean; 194 MiB data, 2.5 GiB used, 1.0 TiB / 1.0 TiB avail; 332 KiB/s rd, 798 KiB/s wr, 73 op/s 2024-04-23T10:35:55.934 INFO:tasks.workunit.client.0.smithi114.stdout: 1 639393 6.88 MB/sec execute 364 sec latency 1664.343 ms 2024-04-23T10:35:55.944 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:35:55.945+0000 7fac1469a640 1 -- 172.21.15.114:0/2817567792 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fac0c106170 msgr2=0x7fac0c10dd40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T10:35:55.944 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:35:55.945+0000 7fac1469a640 1 --2- 172.21.15.114:0/2817567792 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fac0c106170 0x7fac0c10dd40 secure :-1 s=READY pgs=815 cs=0 l=1 rev1=1 crypto rx=0x7fac04009900 tx=0x7fac0402a410 comp rx=0 tx=0).stop 2024-04-23T10:35:55.944 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:35:55.945+0000 7fac1469a640 1 -- 172.21.15.114:0/2817567792 shutdown_connections 2024-04-23T10:35:55.944 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:35:55.945+0000 7fac1469a640 1 --2- 172.21.15.114:0/2817567792 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fac0c106170 0x7fac0c10dd40 unknown :-1 s=CLOSED pgs=815 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:35:55.944 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:35:55.945+0000 7fac1469a640 1 --2- 172.21.15.114:0/2817567792 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fac0c101aa0 0x7fac0c105a90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:35:55.944 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:35:55.945+0000 7fac1469a640 1 --2- 172.21.15.114:0/2817567792 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fac0c1010f0 0x7fac0c1014d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:35:55.944 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:35:55.945+0000 7fac1469a640 1 -- 172.21.15.114:0/2817567792 >> 172.21.15.114:0/2817567792 conn(0x7fac0c077fd0 msgr2=0x7fac0c0783e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T10:35:55.945 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:35:55.945+0000 7fac1469a640 1 -- 172.21.15.114:0/2817567792 shutdown_connections 2024-04-23T10:35:55.945 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:35:55.945+0000 7fac1469a640 1 -- 172.21.15.114:0/2817567792 wait complete. 2024-04-23T10:35:55.945 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:35:55.945+0000 7fac1469a640 1 Processor -- start 2024-04-23T10:35:55.945 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:35:55.945+0000 7fac1469a640 1 -- start start 2024-04-23T10:35:55.947 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:35:55.946+0000 7fac1469a640 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fac0c1010f0 0x7fac0c1986c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T10:35:55.947 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:35:55.946+0000 7fac1469a640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fac0c106170 0x7fac0c198c00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T10:35:55.947 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:35:55.946+0000 7fac1469a640 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fac0c19cf80 0x7fac0c19d430 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T10:35:55.947 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:35:55.946+0000 7fac1469a640 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x7fac0c1121e0 con 0x7fac0c106170 2024-04-23T10:35:55.947 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:35:55.946+0000 7fac1469a640 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 -- 0x7fac0c112060 con 0x7fac0c19cf80 2024-04-23T10:35:55.947 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:35:55.946+0000 7fac1469a640 1 -- --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_getmap magic: 0 -- 0x7fac0c112360 con 0x7fac0c1010f0 2024-04-23T10:35:55.947 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:35:55.946+0000 7fac11c35640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fac0c106170 0x7fac0c198c00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T10:35:55.947 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:35:55.946+0000 7fac11c35640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fac0c106170 0x7fac0c198c00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.114:42990/0 (socket says 172.21.15.114:42990) 2024-04-23T10:35:55.947 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:35:55.946+0000 7fac11c35640 1 -- 172.21.15.114:0/2080589150 learned_addr learned my addr 172.21.15.114:0/2080589150 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-04-23T10:35:55.947 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:35:55.946+0000 7fac12436640 1 --2- 172.21.15.114:0/2080589150 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fac0c1010f0 0x7fac0c1986c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T10:35:55.947 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:35:55.946+0000 7fac11c35640 1 -- 172.21.15.114:0/2080589150 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fac0c1010f0 msgr2=0x7fac0c1986c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T10:35:55.947 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:35:55.946+0000 7fac11c35640 1 --2- 172.21.15.114:0/2080589150 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fac0c1010f0 0x7fac0c1986c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:35:55.948 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:35:55.946+0000 7fac11c35640 1 -- 172.21.15.114:0/2080589150 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fac0c19cf80 msgr2=0x7fac0c19d430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T10:35:55.948 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:35:55.946+0000 7fac12c37640 1 --2- 172.21.15.114:0/2080589150 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fac0c19cf80 0x7fac0c19d430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T10:35:55.948 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:35:55.946+0000 7fac11c35640 1 --2- 172.21.15.114:0/2080589150 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fac0c19cf80 0x7fac0c19d430 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:35:55.948 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:35:55.946+0000 7fac11c35640 1 -- 172.21.15.114:0/2080589150 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fac0c19da70 con 0x7fac0c106170 2024-04-23T10:35:55.948 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:35:55.946+0000 7fac12c37640 1 --2- 172.21.15.114:0/2080589150 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fac0c19cf80 0x7fac0c19d430 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-23T10:35:55.948 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:35:55.946+0000 7fac11c35640 1 --2- 172.21.15.114:0/2080589150 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fac0c106170 0x7fac0c198c00 secure :-1 s=READY pgs=816 cs=0 l=1 rev1=1 crypto rx=0x7fabfc00d740 tx=0x7fabfc00dc00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T10:35:55.948 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:35:55.946+0000 7fac037fe640 1 -- 172.21.15.114:0/2080589150 <== mon.0 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fabfc017070 con 0x7fac0c106170 2024-04-23T10:35:55.948 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:35:55.947+0000 7fac037fe640 1 -- 172.21.15.114:0/2080589150 <== mon.0 v2:172.21.15.114:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fabfc00de50 con 0x7fac0c106170 2024-04-23T10:35:55.948 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:35:55.947+0000 7fac1469a640 1 -- 172.21.15.114:0/2080589150 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fac0c19dd60 con 0x7fac0c106170 2024-04-23T10:35:55.948 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:35:55.947+0000 7fac037fe640 1 -- 172.21.15.114:0/2080589150 <== mon.0 v2:172.21.15.114:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fabfc004ce0 con 0x7fac0c106170 2024-04-23T10:35:55.948 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:35:55.947+0000 7fac1469a640 1 -- 172.21.15.114:0/2080589150 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fac0c1a5500 con 0x7fac0c106170 2024-04-23T10:35:55.955 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:35:55.949+0000 7fac037fe640 1 -- 172.21.15.114:0/2080589150 <== mon.0 v2:172.21.15.114:3300/0 4 ==== mgrmap(e 26) ==== 138778+0+0 (secure 0 0 0) 0x7fabfc004e80 con 0x7fac0c106170 2024-04-23T10:35:55.955 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:35:55.949+0000 7fac173b3740 1 Processor -- start 2024-04-23T10:35:55.955 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:35:55.949+0000 7fac173b3740 1 -- start start 2024-04-23T10:35:55.955 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:35:55.950+0000 7fac037fe640 1 --2- 172.21.15.114:0/2080589150 >> [v2:172.21.15.114:6800/1395693466,v1:172.21.15.114:6801/1395693466] conn(0x7fabf80a3dc0 0x7fabf80a6280 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T10:35:55.955 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:35:55.950+0000 7fac173b3740 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55cd913e4520 0x55cd913e4900 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T10:35:55.955 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:35:55.950+0000 7fac173b3740 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x55cd913e4e40 0x55cd913f36d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T10:35:55.955 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:35:55.950+0000 7fac173b3740 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x55cd913f3c10 0x55cd913f6000 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T10:35:55.955 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:35:55.950+0000 7fac173b3740 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x55cd91349970 con 0x55cd913e4520 2024-04-23T10:35:55.955 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:35:55.950+0000 7fac173b3740 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 -- 0x55cd912e64a0 con 0x55cd913f3c10 2024-04-23T10:35:55.955 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:35:55.950+0000 7fac173b3740 1 -- --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_getmap magic: 0 -- 0x55cd91345930 con 0x55cd913e4e40 2024-04-23T10:35:55.955 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:35:55.950+0000 7fac12c37640 1 --2- 172.21.15.114:0/2080589150 >> [v2:172.21.15.114:6800/1395693466,v1:172.21.15.114:6801/1395693466] conn(0x7fabf80a3dc0 0x7fabf80a6280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T10:35:55.956 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:35:55.950+0000 7fac12436640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55cd913e4520 0x55cd913e4900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T10:35:55.956 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:35:55.950+0000 7fac037fe640 1 -- 172.21.15.114:0/2080589150 <== mon.0 v2:172.21.15.114:3300/0 5 ==== osd_map(100..100 src has 1..100) ==== 8629+0+0 (secure 0 0 0) 0x7fabfc0d0240 con 0x7fac0c106170 2024-04-23T10:35:55.956 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:35:55.950+0000 7fac12436640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55cd913e4520 0x55cd913e4900 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.114:43006/0 (socket says 172.21.15.114:43006) 2024-04-23T10:35:55.956 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:35:55.950+0000 7fac12436640 1 -- 172.21.15.114:0/1402852907 learned_addr learned my addr 172.21.15.114:0/1402852907 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-04-23T10:35:55.956 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:35:55.950+0000 7fac11c35640 1 --2- 172.21.15.114:0/1402852907 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x55cd913e4e40 0x55cd913f36d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T10:35:55.956 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:35:55.950+0000 7fac12c37640 1 --2- 172.21.15.114:0/1402852907 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x55cd913f3c10 0x55cd913f6000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T10:35:55.956 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:35:55.950+0000 7fac12436640 1 -- 172.21.15.114:0/1402852907 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x55cd913e4e40 msgr2=0x55cd913f36d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T10:35:55.956 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:35:55.950+0000 7fac12436640 1 --2- 172.21.15.114:0/1402852907 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x55cd913e4e40 0x55cd913f36d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:35:55.956 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:35:55.950+0000 7fac12436640 1 -- 172.21.15.114:0/1402852907 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x55cd913f3c10 msgr2=0x55cd913f6000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T10:35:55.956 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:35:55.950+0000 7fac12436640 1 --2- 172.21.15.114:0/1402852907 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x55cd913f3c10 0x55cd913f6000 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:35:55.956 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:35:55.950+0000 7fac12436640 1 -- 172.21.15.114:0/1402852907 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55cd9105c210 con 0x55cd913e4520 2024-04-23T10:35:55.956 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:35:55.950+0000 7fac12c37640 1 --2- 172.21.15.114:0/1402852907 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x55cd913f3c10 0x55cd913f6000 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-23T10:35:55.956 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:35:55.950+0000 7fac12c37640 1 --2- 172.21.15.114:0/2080589150 >> [v2:172.21.15.114:6800/1395693466,v1:172.21.15.114:6801/1395693466] conn(0x7fabf80a3dc0 0x7fabf80a6280 secure :-1 s=READY pgs=126 cs=0 l=1 rev1=1 crypto rx=0x7fac04006010 tx=0x7fac04032d20 comp rx=0 tx=0).ready entity=mgr.15291 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T10:35:55.956 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:35:55.950+0000 7fac12436640 1 --2- 172.21.15.114:0/1402852907 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55cd913e4520 0x55cd913e4900 secure :-1 s=READY pgs=817 cs=0 l=1 rev1=1 crypto rx=0x7fac08009f60 tx=0x7fac08009f90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2f997c9b4d224b93 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T10:35:55.957 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:35:55.951+0000 7fac00ff9640 1 -- 172.21.15.114:0/1402852907 <== mon.0 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fac08016070 con 0x55cd913e4520 2024-04-23T10:35:55.957 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:35:55.951+0000 7fac00ff9640 1 -- 172.21.15.114:0/1402852907 <== mon.0 v2:172.21.15.114:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fac08012900 con 0x55cd913e4520 2024-04-23T10:35:55.957 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:35:55.951+0000 7fac00ff9640 1 -- 172.21.15.114:0/1402852907 <== mon.0 v2:172.21.15.114:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fac08011920 con 0x55cd913e4520 2024-04-23T10:35:55.957 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:35:55.951+0000 7fac173b3740 1 -- 172.21.15.114:0/1402852907 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55cd913e4520 msgr2=0x55cd913e4900 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T10:35:55.957 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:35:55.951+0000 7fac173b3740 1 --2- 172.21.15.114:0/1402852907 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55cd913e4520 0x55cd913e4900 secure :-1 s=READY pgs=817 cs=0 l=1 rev1=1 crypto rx=0x7fac08009f60 tx=0x7fac08009f90 comp rx=0 tx=0).stop 2024-04-23T10:35:55.957 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:35:55.951+0000 7fac173b3740 1 -- 172.21.15.114:0/1402852907 shutdown_connections 2024-04-23T10:35:55.957 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:35:55.951+0000 7fac173b3740 1 --2- 172.21.15.114:0/1402852907 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x55cd913f3c10 0x55cd913f6000 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:35:55.957 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:35:55.951+0000 7fac173b3740 1 --2- 172.21.15.114:0/1402852907 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x55cd913e4e40 0x55cd913f36d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:35:55.957 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:35:55.951+0000 7fac173b3740 1 --2- 172.21.15.114:0/1402852907 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55cd913e4520 0x55cd913e4900 unknown :-1 s=CLOSED pgs=817 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:35:55.957 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:35:55.951+0000 7fac173b3740 1 -- 172.21.15.114:0/1402852907 >> 172.21.15.114:0/1402852907 conn(0x55cd913e0b30 msgr2=0x55cd913e0f10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T10:35:55.957 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:35:55.951+0000 7fac173b3740 1 -- 172.21.15.114:0/1402852907 shutdown_connections 2024-04-23T10:35:55.957 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:35:55.951+0000 7fac173b3740 1 -- 172.21.15.114:0/1402852907 wait complete. 2024-04-23T10:35:55.957 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:35:55.952+0000 7fac173b3740 1 -- start start 2024-04-23T10:35:55.957 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:35:55.952+0000 7fac173b3740 1 Processor -- start 2024-04-23T10:35:55.957 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:35:55.952+0000 7fac173b3740 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x55cd913e4520 0x55cd913f34c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T10:35:55.957 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:35:55.952+0000 7fac173b3740 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x55cd913e4e40 0x55cd913f07b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T10:35:55.958 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:35:55.952+0000 7fac173b3740 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55cd913f3c10 0x55cd913f0cf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T10:35:55.958 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:35:55.952+0000 7fac173b3740 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x55cd913f6840 con 0x55cd913f3c10 2024-04-23T10:35:55.958 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:35:55.952+0000 7fac173b3740 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 -- 0x55cd912e40b0 con 0x55cd913e4e40 2024-04-23T10:35:55.958 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:35:55.952+0000 7fac173b3740 1 -- --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_getmap magic: 0 -- 0x55cd913f69c0 con 0x55cd913e4520 2024-04-23T10:35:55.958 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:35:55.952+0000 7fac173b3740 10 client.0 _ll_register_callbacks cb 0x55cd912d1d30 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-23T10:35:55.958 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:35:55.952+0000 7fac12c37640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55cd913f3c10 0x55cd913f0cf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T10:35:55.958 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:35:55.952+0000 7fac12c37640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55cd913f3c10 0x55cd913f0cf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.114:43012/0 (socket says 172.21.15.114:43012) 2024-04-23T10:35:55.958 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:35:55.952+0000 7fac12c37640 1 -- 172.21.15.114:0/4215735448 learned_addr learned my addr 172.21.15.114:0/4215735448 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-04-23T10:35:55.958 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:35:55.952+0000 7fac12436640 1 --2- 172.21.15.114:0/4215735448 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x55cd913e4520 0x55cd913f34c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T10:35:55.958 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:35:55.952+0000 7fac11c35640 1 --2- 172.21.15.114:0/4215735448 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x55cd913e4e40 0x55cd913f07b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T10:35:55.958 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:35:55.953+0000 7fac12c37640 1 -- 172.21.15.114:0/4215735448 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x55cd913e4520 msgr2=0x55cd913f34c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T10:35:55.958 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:35:55.953+0000 7fac12c37640 1 --2- 172.21.15.114:0/4215735448 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x55cd913e4520 0x55cd913f34c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:35:55.958 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:35:55.953+0000 7fac12c37640 1 -- 172.21.15.114:0/4215735448 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x55cd913e4e40 msgr2=0x55cd913f07b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T10:35:55.958 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:35:55.953+0000 7fac12c37640 1 --2- 172.21.15.114:0/4215735448 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x55cd913e4e40 0x55cd913f07b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:35:55.959 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:35:55.953+0000 7fac12c37640 1 -- 172.21.15.114:0/4215735448 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55cd91128620 con 0x55cd913f3c10 2024-04-23T10:35:55.959 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:35:55.953+0000 7fac12436640 1 --2- 172.21.15.114:0/4215735448 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x55cd913e4520 0x55cd913f34c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-23T10:35:55.959 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:35:55.953+0000 7fac12c37640 1 --2- 172.21.15.114:0/4215735448 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55cd913f3c10 0x55cd913f0cf0 secure :-1 s=READY pgs=818 cs=0 l=1 rev1=1 crypto rx=0x7fac04033000 tx=0x7fac0402a410 comp rx=0 tx=0).ready entity=mon.0 client_cookie=46e6fdb4b29a0300 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T10:35:55.959 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:35:55.953+0000 7fabf5ffb640 10 client.0 ms_handle_connect on v2:172.21.15.114:3300/0 2024-04-23T10:35:55.959 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:35:55.953+0000 7fabf5ffb640 1 -- 172.21.15.114:0/4215735448 <== mon.0 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fac04035050 con 0x55cd913f3c10 2024-04-23T10:35:55.959 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:35:55.953+0000 7fabf5ffb640 1 -- 172.21.15.114:0/4215735448 <== mon.0 v2:172.21.15.114:3300/0 2 ==== osd_map(100..100 src has 1..100) ==== 8629+0+0 (secure 0 0 0) 0x7fac04005240 con 0x55cd913f3c10 2024-04-23T10:35:55.959 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:35:55.953+0000 7fac173b3740 1 -- 172.21.15.114:0/4215735448 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fac08012900 con 0x55cd913f3c10 2024-04-23T10:35:55.959 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:35:55.953+0000 7fabf5ffb640 1 -- 172.21.15.114:0/4215735448 <== mon.0 v2:172.21.15.114:3300/0 3 ==== mon_get_version_reply(handle=1 version=34) ==== 24+0+0 (secure 0 0 0) 0x7fac04057050 con 0x55cd913f3c10 2024-04-23T10:35:55.959 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:35:55.953+0000 7fac173b3740 10 client.16710 fetch_fsmap learned FSMap version 34 2024-04-23T10:35:55.959 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:35:55.953+0000 7fac173b3740 1 -- 172.21.15.114:0/4215735448 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({config=0+,fsmap=34,monmap=4+}) -- 0x55cd914026f0 con 0x55cd913f3c10 2024-04-23T10:35:55.959 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:35:55.953+0000 7fabf5ffb640 1 -- 172.21.15.114:0/4215735448 <== mon.0 v2:172.21.15.114:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fac08012900 con 0x55cd913f3c10 2024-04-23T10:35:55.959 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:35:55.953+0000 7fabf5ffb640 1 -- 172.21.15.114:0/4215735448 <== mon.0 v2:172.21.15.114:3300/0 5 ==== fsmap(e 34) ==== 8346+0+0 (secure 0 0 0) 0x7fac04003890 con 0x55cd913f3c10 2024-04-23T10:35:55.959 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:35:55.953+0000 7fac173b3740 10 client.16710 fetch_fsmap finished waiting for FSMap version 34 2024-04-23T10:35:55.959 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:35:55.953+0000 7fac173b3740 10 client.16710 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-04-23T10:35:55.959 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:35:55.953+0000 7fac173b3740 20 client.16710 populate_metadata read hostname 'smithi114' 2024-04-23T10:35:55.960 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:35:55.954+0000 7fac173b3740 1 --2- 172.21.15.114:0/4215735448 >> [v2:172.21.15.119:6838/937297340,v1:172.21.15.119:6839/937297340] conn(0x55cd91403c00 0x55cd91406010 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T10:35:55.960 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:35:55.954+0000 7fac173b3740 4 client.16710 mds_command: new command op to 24545 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-23T10:35:55.960 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:35:55.954+0000 7fac173b3740 1 -- 172.21.15.114:0/4215735448 --> [v2:172.21.15.119:6838/937297340,v1:172.21.15.119:6839/937297340] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55cd91357390 con 0x55cd91403c00 2024-04-23T10:35:55.960 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:35:55.954+0000 7fac12436640 1 --2- 172.21.15.114:0/4215735448 >> [v2:172.21.15.119:6838/937297340,v1:172.21.15.119:6839/937297340] conn(0x55cd91403c00 0x55cd91406010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T10:35:55.960 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:35:55.954+0000 7fac12436640 1 --2- 172.21.15.114:0/4215735448 >> [v2:172.21.15.119:6838/937297340,v1:172.21.15.119:6839/937297340] conn(0x55cd91403c00 0x55cd91406010 crc :-1 s=READY pgs=144 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=f81dbf0c71606fc1 server_cookie=5b410d1c2a84c3f1 in_seq=0 out_seq=0 2024-04-23T10:35:55.960 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:35:55.954+0000 7fabf5ffb640 10 client.16710 ms_handle_connect on v2:172.21.15.119:6838/937297340 2024-04-23T10:35:55.960 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:35:55.955+0000 7fabf5ffb640 1 -- 172.21.15.114:0/4215735448 <== mds.0 v2:172.21.15.119:6838/937297340 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x55cd91357390 con 0x55cd91403c00 2024-04-23T10:35:55.960 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:35:55.955+0000 7fabf5ffb640 10 client.16710 handle_command_reply: tid=0 2024-04-23T10:35:55.960 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:35:55.955+0000 7fac173b3740 1 client.16710 shutdown 2024-04-23T10:35:55.960 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:35:55.955+0000 7fac173b3740 1 -- 172.21.15.114:0/4215735448 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55cd913f3c10 msgr2=0x55cd913f0cf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T10:35:55.960 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:35:55.955+0000 7fac173b3740 1 --2- 172.21.15.114:0/4215735448 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55cd913f3c10 0x55cd913f0cf0 secure :-1 s=READY pgs=818 cs=0 l=1 rev1=1 crypto rx=0x7fac04033000 tx=0x7fac0402a410 comp rx=0 tx=0).stop 2024-04-23T10:35:55.960 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:35:55.955+0000 7fac173b3740 1 -- 172.21.15.114:0/4215735448 shutdown_connections 2024-04-23T10:35:55.960 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:35:55.955+0000 7fac173b3740 1 --2- 172.21.15.114:0/4215735448 >> [v2:172.21.15.119:6838/937297340,v1:172.21.15.119:6839/937297340] conn(0x55cd91403c00 0x55cd91406010 crc :-1 s=READY pgs=144 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:35:55.960 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:35:55.955+0000 7fac173b3740 1 --2- 172.21.15.114:0/4215735448 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55cd913f3c10 0x55cd913f0cf0 unknown :-1 s=CLOSED pgs=818 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:35:55.960 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:35:55.955+0000 7fac173b3740 1 --2- 172.21.15.114:0/4215735448 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x55cd913e4e40 0x55cd913f07b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:35:55.960 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:35:55.955+0000 7fac173b3740 1 --2- 172.21.15.114:0/4215735448 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x55cd913e4520 0x55cd913f34c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:35:55.961 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:35:55.955+0000 7fabf5ffb640 0 client.16710 ms_handle_reset on v2:172.21.15.119:6838/937297340 2024-04-23T10:35:55.961 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:35:55.955+0000 7fac173b3740 1 -- 172.21.15.114:0/4215735448 >> 172.21.15.114:0/4215735448 conn(0x55cd913e0b30 msgr2=0x55cd913e69a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T10:35:55.961 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:35:55.955+0000 7fac173b3740 1 -- 172.21.15.114:0/4215735448 shutdown_connections 2024-04-23T10:35:55.961 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:35:55.955+0000 7fac173b3740 1 -- 172.21.15.114:0/4215735448 wait complete. 2024-04-23T10:35:55.961 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:35:55.955+0000 7fac173b3740 20 client.16710 trim_cache size 0 max 16384 2024-04-23T10:35:55.974 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:35:55.970+0000 7fac173b3740 1 Processor -- start 2024-04-23T10:35:55.974 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:35:55.970+0000 7fac173b3740 1 -- start start 2024-04-23T10:35:55.974 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:35:55.970+0000 7fac173b3740 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x55cd913e0b30 0x55cd913ff700 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T10:35:55.974 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:35:55.970+0000 7fac173b3740 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55cd913e4520 0x55cd913ebd80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T10:35:55.974 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:35:55.970+0000 7fac173b3740 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x55cd913e4e40 0x55cd913f2ff0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T10:35:55.974 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:35:55.970+0000 7fac173b3740 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x7fac0c1ae440 con 0x55cd913e4520 2024-04-23T10:35:55.974 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:35:55.970+0000 7fac173b3740 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 -- 0x7fac0c1abce0 con 0x55cd913e4e40 2024-04-23T10:35:55.974 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:35:55.970+0000 7fac173b3740 1 -- --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_getmap magic: 0 -- 0x7fac0c1a9b50 con 0x55cd913e0b30 2024-04-23T10:35:55.974 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:35:55.970+0000 7fac11c35640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55cd913e4520 0x55cd913ebd80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T10:35:55.974 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:35:55.970+0000 7fac11c35640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55cd913e4520 0x55cd913ebd80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.114:43022/0 (socket says 172.21.15.114:43022) 2024-04-23T10:35:55.974 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:35:55.970+0000 7fac11c35640 1 -- 172.21.15.114:0/2594690210 learned_addr learned my addr 172.21.15.114:0/2594690210 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-04-23T10:35:55.974 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:35:55.971+0000 7fac12c37640 1 --2- 172.21.15.114:0/2594690210 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x55cd913e4e40 0x55cd913f2ff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T10:35:55.975 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:35:55.971+0000 7fac12436640 1 --2- 172.21.15.114:0/2594690210 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x55cd913e0b30 0x55cd913ff700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T10:35:55.975 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:35:55.971+0000 7fac11c35640 1 -- 172.21.15.114:0/2594690210 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x55cd913e0b30 msgr2=0x55cd913ff700 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T10:35:55.975 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:35:55.971+0000 7fac11c35640 1 --2- 172.21.15.114:0/2594690210 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x55cd913e0b30 0x55cd913ff700 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:35:55.975 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:35:55.971+0000 7fac11c35640 1 -- 172.21.15.114:0/2594690210 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x55cd913e4e40 msgr2=0x55cd913f2ff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T10:35:55.975 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:35:55.971+0000 7fac11c35640 1 --2- 172.21.15.114:0/2594690210 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x55cd913e4e40 0x55cd913f2ff0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:35:55.975 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:35:55.971+0000 7fac11c35640 1 -- 172.21.15.114:0/2594690210 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55cd913e5380 con 0x55cd913e4520 2024-04-23T10:35:55.975 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:35:55.971+0000 7fac12c37640 1 --2- 172.21.15.114:0/2594690210 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x55cd913e4e40 0x55cd913f2ff0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-23T10:35:55.975 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:35:55.971+0000 7fac11c35640 1 --2- 172.21.15.114:0/2594690210 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55cd913e4520 0x55cd913ebd80 secure :-1 s=READY pgs=819 cs=0 l=1 rev1=1 crypto rx=0x7fabfc09be70 tx=0x7fabfc09c170 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7d857858952c1dba server_cookie=0 in_seq=0 out_seq=0 2024-04-23T10:35:55.975 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:35:55.971+0000 7fabf7fff640 1 -- 172.21.15.114:0/2594690210 <== mon.0 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fabfc020040 con 0x55cd913e4520 2024-04-23T10:35:55.975 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:35:55.971+0000 7fabf7fff640 1 -- 172.21.15.114:0/2594690210 <== mon.0 v2:172.21.15.114:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fabfc09a210 con 0x55cd913e4520 2024-04-23T10:35:55.975 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:35:55.971+0000 7fabf7fff640 1 -- 172.21.15.114:0/2594690210 <== mon.0 v2:172.21.15.114:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fabfc0987e0 con 0x55cd913e4520 2024-04-23T10:35:55.975 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:35:55.972+0000 7fac173b3740 1 -- 172.21.15.114:0/2594690210 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55cd913e4520 msgr2=0x55cd913ebd80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T10:35:55.975 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:35:55.972+0000 7fac173b3740 1 --2- 172.21.15.114:0/2594690210 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55cd913e4520 0x55cd913ebd80 secure :-1 s=READY pgs=819 cs=0 l=1 rev1=1 crypto rx=0x7fabfc09be70 tx=0x7fabfc09c170 comp rx=0 tx=0).stop 2024-04-23T10:35:55.975 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:35:55.972+0000 7fac173b3740 1 -- 172.21.15.114:0/2594690210 shutdown_connections 2024-04-23T10:35:55.975 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:35:55.972+0000 7fac173b3740 1 --2- 172.21.15.114:0/2594690210 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x55cd913e4e40 0x55cd913f2ff0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:35:55.976 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:35:55.972+0000 7fac173b3740 1 --2- 172.21.15.114:0/2594690210 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55cd913e4520 0x55cd913ebd80 unknown :-1 s=CLOSED pgs=819 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:35:55.976 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:35:55.972+0000 7fac173b3740 1 --2- 172.21.15.114:0/2594690210 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x55cd913e0b30 0x55cd913ff700 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:35:55.976 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:35:55.972+0000 7fac173b3740 1 -- 172.21.15.114:0/2594690210 >> 172.21.15.114:0/2594690210 conn(0x55cd913f0e50 msgr2=0x55cd913eb670 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T10:35:55.976 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:35:55.972+0000 7fac173b3740 1 -- 172.21.15.114:0/2594690210 shutdown_connections 2024-04-23T10:35:55.976 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:35:55.972+0000 7fac173b3740 1 -- 172.21.15.114:0/2594690210 wait complete. 2024-04-23T10:35:55.976 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:35:55.972+0000 7fac173b3740 1 -- start start 2024-04-23T10:35:55.976 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:35:55.972+0000 7fac173b3740 1 Processor -- start 2024-04-23T10:35:55.976 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:35:55.972+0000 7fac173b3740 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55cd913e0b30 0x55cd913ed660 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T10:35:55.976 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:35:55.972+0000 7fac173b3740 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x55cd913e4520 0x55cd913edba0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T10:35:55.976 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:35:55.972+0000 7fac173b3740 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x55cd913e4e40 0x55cd913ee0e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T10:35:55.976 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:35:55.972+0000 7fac173b3740 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x7fac0c1aa1a0 con 0x55cd913e0b30 2024-04-23T10:35:55.976 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:35:55.972+0000 7fac173b3740 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 -- 0x7fac0c1ab610 con 0x55cd913e4e40 2024-04-23T10:35:55.976 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:35:55.972+0000 7fac173b3740 1 -- --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_getmap magic: 0 -- 0x7fac0c1aa870 con 0x55cd913e4520 2024-04-23T10:35:55.976 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:35:55.972+0000 7fac173b3740 10 client.0 _ll_register_callbacks cb 0x7fac04031950 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-23T10:35:55.976 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:35:55.972+0000 7fac12436640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55cd913e0b30 0x55cd913ed660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T10:35:55.977 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:35:55.972+0000 7fac12436640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55cd913e0b30 0x55cd913ed660 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.114:43038/0 (socket says 172.21.15.114:43038) 2024-04-23T10:35:55.977 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:35:55.972+0000 7fac12436640 1 -- 172.21.15.114:0/3890574400 learned_addr learned my addr 172.21.15.114:0/3890574400 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-04-23T10:35:55.977 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:35:55.972+0000 7fac12c37640 1 --2- 172.21.15.114:0/3890574400 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x55cd913e4e40 0x55cd913ee0e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T10:35:55.977 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:35:55.972+0000 7fac11c35640 1 --2- 172.21.15.114:0/3890574400 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x55cd913e4520 0x55cd913edba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T10:35:55.977 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:35:55.973+0000 7fac12436640 1 -- 172.21.15.114:0/3890574400 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x55cd913e4520 msgr2=0x55cd913edba0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T10:35:55.977 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:35:55.973+0000 7fac12436640 1 --2- 172.21.15.114:0/3890574400 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x55cd913e4520 0x55cd913edba0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:35:55.977 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:35:55.973+0000 7fac12436640 1 -- 172.21.15.114:0/3890574400 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x55cd913e4e40 msgr2=0x55cd913ee0e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T10:35:55.977 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:35:55.973+0000 7fac12436640 1 --2- 172.21.15.114:0/3890574400 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x55cd913e4e40 0x55cd913ee0e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:35:55.977 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:35:55.973+0000 7fac12436640 1 -- 172.21.15.114:0/3890574400 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55cd913e1d60 con 0x55cd913e0b30 2024-04-23T10:35:55.977 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:35:55.973+0000 7fac11c35640 1 --2- 172.21.15.114:0/3890574400 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x55cd913e4520 0x55cd913edba0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-23T10:35:55.977 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:35:55.973+0000 7fac12436640 1 --2- 172.21.15.114:0/3890574400 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55cd913e0b30 0x55cd913ed660 secure :-1 s=READY pgs=820 cs=0 l=1 rev1=1 crypto rx=0x7fac080114f0 tx=0x7fac08011520 comp rx=0 tx=0).ready entity=mon.0 client_cookie=94b26552c56d2710 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T10:35:55.977 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:35:55.973+0000 7fabf5ffb640 10 client.0 ms_handle_connect on v2:172.21.15.114:3300/0 2024-04-23T10:35:55.977 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:35:55.973+0000 7fabf5ffb640 1 -- 172.21.15.114:0/3890574400 <== mon.0 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fac08016040 con 0x55cd913e0b30 2024-04-23T10:35:55.977 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:35:55.973+0000 7fabf5ffb640 1 -- 172.21.15.114:0/3890574400 <== mon.0 v2:172.21.15.114:3300/0 2 ==== osd_map(100..100 src has 1..100) ==== 8629+0+0 (secure 0 0 0) 0x7fac08010c90 con 0x55cd913e0b30 2024-04-23T10:35:55.977 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:35:55.973+0000 7fac173b3740 1 -- 172.21.15.114:0/3890574400 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fabfc09a210 con 0x55cd913e0b30 2024-04-23T10:35:55.977 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:35:55.973+0000 7fabf5ffb640 1 -- 172.21.15.114:0/3890574400 <== mon.0 v2:172.21.15.114:3300/0 3 ==== mon_get_version_reply(handle=1 version=34) ==== 24+0+0 (secure 0 0 0) 0x7fac08003d80 con 0x55cd913e0b30 2024-04-23T10:35:55.978 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:35:55.973+0000 7fac173b3740 10 client.16716 fetch_fsmap learned FSMap version 34 2024-04-23T10:35:55.978 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:35:55.973+0000 7fac173b3740 1 -- 172.21.15.114:0/3890574400 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({config=0+,fsmap=34,monmap=4+}) -- 0x55cd9146a3d0 con 0x55cd913e0b30 2024-04-23T10:35:55.978 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:35:55.973+0000 7fabf5ffb640 1 -- 172.21.15.114:0/3890574400 <== mon.0 v2:172.21.15.114:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fabfc09a210 con 0x55cd913e0b30 2024-04-23T10:35:55.978 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:35:55.973+0000 7fabf5ffb640 1 -- 172.21.15.114:0/3890574400 <== mon.0 v2:172.21.15.114:3300/0 5 ==== fsmap(e 34) ==== 8346+0+0 (secure 0 0 0) 0x7fac0800f040 con 0x55cd913e0b30 2024-04-23T10:35:55.978 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:35:55.973+0000 7fac173b3740 10 client.16716 fetch_fsmap finished waiting for FSMap version 34 2024-04-23T10:35:55.978 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:35:55.973+0000 7fac173b3740 10 client.16716 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-04-23T10:35:55.978 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:35:55.973+0000 7fac173b3740 20 client.16716 populate_metadata read hostname 'smithi114' 2024-04-23T10:35:55.978 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:35:55.973+0000 7fac173b3740 1 --2- 172.21.15.114:0/3890574400 >> [v2:172.21.15.119:6838/937297340,v1:172.21.15.119:6839/937297340] conn(0x55cd913f3c10 0x55cd9146c7c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T10:35:55.978 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:35:55.973+0000 7fac173b3740 4 client.16716 mds_command: new command op to 24545 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-04-23T10:35:55.978 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:35:55.973+0000 7fac173b3740 1 -- 172.21.15.114:0/3890574400 --> [v2:172.21.15.119:6838/937297340,v1:172.21.15.119:6839/937297340] -- command(tid 0: {"prefix": "flush journal"}) -- 0x55cd91351200 con 0x55cd913f3c10 2024-04-23T10:35:55.978 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:35:55.974+0000 7fac11c35640 1 --2- 172.21.15.114:0/3890574400 >> [v2:172.21.15.119:6838/937297340,v1:172.21.15.119:6839/937297340] conn(0x55cd913f3c10 0x55cd9146c7c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T10:35:55.978 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:35:55.974+0000 7fabf5ffb640 10 client.16716 ms_handle_connect on v2:172.21.15.119:6838/937297340 2024-04-23T10:35:55.978 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:35:55.974+0000 7fac11c35640 1 --2- 172.21.15.114:0/3890574400 >> [v2:172.21.15.119:6838/937297340,v1:172.21.15.119:6839/937297340] conn(0x55cd913f3c10 0x55cd9146c7c0 crc :-1 s=READY pgs=145 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=4b413824a0963bd7 server_cookie=64b5d8e9aa91df8b in_seq=0 out_seq=0 2024-04-23T10:35:56.935 INFO:tasks.workunit.client.0.smithi114.stdout: 1 639393 6.86 MB/sec execute 365 sec latency 2664.409 ms 2024-04-23T10:35:57.742 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 23 10:35:57 smithi119 ceph-mon[32608]: pgmap v291: 97 pgs: 97 active+clean; 194 MiB data, 2.5 GiB used, 1.0 TiB / 1.0 TiB avail; 398 KiB/s rd, 1.2 MiB/s wr, 106 op/s 2024-04-23T10:35:57.753 INFO:journalctl@ceph.mon.a.smithi114.stdout:Apr 23 10:35:57 smithi114 ceph-mon[29255]: pgmap v291: 97 pgs: 97 active+clean; 194 MiB data, 2.5 GiB used, 1.0 TiB / 1.0 TiB avail; 398 KiB/s rd, 1.2 MiB/s wr, 106 op/s 2024-04-23T10:35:57.828 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 23 10:35:57 smithi204 ceph-mon[32649]: pgmap v291: 97 pgs: 97 active+clean; 194 MiB data, 2.5 GiB used, 1.0 TiB / 1.0 TiB avail; 398 KiB/s rd, 1.2 MiB/s wr, 106 op/s 2024-04-23T10:35:57.935 INFO:tasks.workunit.client.0.smithi114.stdout: 1 639393 6.84 MB/sec execute 366 sec latency 3664.455 ms 2024-04-23T10:35:58.522 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:35:58.522+0000 7fabf5ffb640 1 -- 172.21.15.114:0/3890574400 <== mds.0 v2:172.21.15.119:6838/937297340 1 ==== command_reply(tid 0: 0 ) ==== 8+0+44 (crc 0 0 0) 0x55cd91351200 con 0x55cd913f3c10 2024-04-23T10:35:58.522 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:35:58.522+0000 7fabf5ffb640 10 client.16716 handle_command_reply: tid=0 2024-04-23T10:35:58.522 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:35:58.522+0000 7fac173b3740 1 client.16716 shutdown 2024-04-23T10:35:58.522 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:35:58.523+0000 7fac173b3740 1 -- 172.21.15.114:0/3890574400 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55cd913e0b30 msgr2=0x55cd913ed660 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T10:35:58.522 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:35:58.523+0000 7fac173b3740 1 --2- 172.21.15.114:0/3890574400 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55cd913e0b30 0x55cd913ed660 secure :-1 s=READY pgs=820 cs=0 l=1 rev1=1 crypto rx=0x7fac080114f0 tx=0x7fac08011520 comp rx=0 tx=0).stop 2024-04-23T10:35:58.522 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:35:58.523+0000 7fac173b3740 1 -- 172.21.15.114:0/3890574400 shutdown_connections 2024-04-23T10:35:58.522 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:35:58.523+0000 7fac173b3740 1 --2- 172.21.15.114:0/3890574400 >> [v2:172.21.15.119:6838/937297340,v1:172.21.15.119:6839/937297340] conn(0x55cd913f3c10 0x55cd9146c7c0 crc :-1 s=READY pgs=145 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:35:58.522 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:35:58.523+0000 7fac173b3740 1 --2- 172.21.15.114:0/3890574400 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x55cd913e4e40 0x55cd913ee0e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:35:58.523 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:35:58.523+0000 7fabf5ffb640 0 client.16716 ms_handle_reset on v2:172.21.15.119:6838/937297340 2024-04-23T10:35:58.523 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:35:58.523+0000 7fac173b3740 1 --2- 172.21.15.114:0/3890574400 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x55cd913e4520 0x55cd913edba0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:35:58.523 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:35:58.523+0000 7fac173b3740 1 --2- 172.21.15.114:0/3890574400 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55cd913e0b30 0x55cd913ed660 unknown :-1 s=CLOSED pgs=820 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:35:58.523 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:35:58.523+0000 7fac173b3740 1 -- 172.21.15.114:0/3890574400 >> 172.21.15.114:0/3890574400 conn(0x55cd913f0e50 msgr2=0x55cd913df420 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T10:35:58.523 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:35:58.523+0000 7fac173b3740 1 -- 172.21.15.114:0/3890574400 shutdown_connections 2024-04-23T10:35:58.523 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:35:58.523+0000 7fac173b3740 1 -- 172.21.15.114:0/3890574400 wait complete. 2024-04-23T10:35:58.523 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:35:58.523+0000 7fac173b3740 20 client.16716 trim_cache size 0 max 16384 2024-04-23T10:35:58.523 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stdout:{ 2024-04-23T10:35:58.523 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stdout: "message": "", 2024-04-23T10:35:58.523 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stdout: "return_code": 0 2024-04-23T10:35:58.523 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stdout:} 2024-04-23T10:35:58.524 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:35:58.524+0000 7fac1469a640 1 -- 172.21.15.114:0/2080589150 >> [v2:172.21.15.114:6800/1395693466,v1:172.21.15.114:6801/1395693466] conn(0x7fabf80a3dc0 msgr2=0x7fabf80a6280 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T10:35:58.524 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:35:58.524+0000 7fac1469a640 1 --2- 172.21.15.114:0/2080589150 >> [v2:172.21.15.114:6800/1395693466,v1:172.21.15.114:6801/1395693466] conn(0x7fabf80a3dc0 0x7fabf80a6280 secure :-1 s=READY pgs=126 cs=0 l=1 rev1=1 crypto rx=0x7fac04006010 tx=0x7fac04032d20 comp rx=0 tx=0).stop 2024-04-23T10:35:58.524 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:35:58.524+0000 7fac1469a640 1 -- 172.21.15.114:0/2080589150 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fac0c106170 msgr2=0x7fac0c198c00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T10:35:58.524 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:35:58.524+0000 7fac1469a640 1 --2- 172.21.15.114:0/2080589150 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fac0c106170 0x7fac0c198c00 secure :-1 s=READY pgs=816 cs=0 l=1 rev1=1 crypto rx=0x7fabfc00d740 tx=0x7fabfc00dc00 comp rx=0 tx=0).stop 2024-04-23T10:35:58.525 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:35:58.524+0000 7fac1469a640 1 -- 172.21.15.114:0/2080589150 shutdown_connections 2024-04-23T10:35:58.525 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:35:58.524+0000 7fac1469a640 1 --2- 172.21.15.114:0/2080589150 >> [v2:172.21.15.114:6800/1395693466,v1:172.21.15.114:6801/1395693466] conn(0x7fabf80a3dc0 0x7fabf80a6280 unknown :-1 s=CLOSED pgs=126 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:35:58.525 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:35:58.524+0000 7fac1469a640 1 --2- 172.21.15.114:0/2080589150 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fac0c19cf80 0x7fac0c19d430 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:35:58.525 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:35:58.524+0000 7fac1469a640 1 --2- 172.21.15.114:0/2080589150 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fac0c106170 0x7fac0c198c00 unknown :-1 s=CLOSED pgs=816 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:35:58.525 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:35:58.524+0000 7fac1469a640 1 --2- 172.21.15.114:0/2080589150 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fac0c1010f0 0x7fac0c1986c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:35:58.525 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:35:58.524+0000 7fac1469a640 1 -- 172.21.15.114:0/2080589150 >> 172.21.15.114:0/2080589150 conn(0x7fac0c077fd0 msgr2=0x7fac0c0fef50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T10:35:58.525 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:35:58.524+0000 7fac1469a640 1 -- 172.21.15.114:0/2080589150 shutdown_connections 2024-04-23T10:35:58.525 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:35:58.524+0000 7fac1469a640 1 -- 172.21.15.114:0/2080589150 wait complete. 2024-04-23T10:35:58.935 INFO:tasks.workunit.client.0.smithi114.stdout: 1 639646 6.83 MB/sec execute 367 sec latency 4231.591 ms 2024-04-23T10:35:59.742 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 23 10:35:59 smithi119 ceph-mon[32608]: pgmap v292: 97 pgs: 97 active+clean; 194 MiB data, 2.5 GiB used, 1.0 TiB / 1.0 TiB avail; 338 KiB/s rd, 717 KiB/s wr, 66 op/s 2024-04-23T10:35:59.742 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 23 10:35:59 smithi119 ceph-mon[32608]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=e=up:active} 3 up:standby-replay 6 up:standby 2024-04-23T10:35:59.753 INFO:journalctl@ceph.mon.a.smithi114.stdout:Apr 23 10:35:59 smithi114 ceph-mon[29255]: pgmap v292: 97 pgs: 97 active+clean; 194 MiB data, 2.5 GiB used, 1.0 TiB / 1.0 TiB avail; 338 KiB/s rd, 717 KiB/s wr, 66 op/s 2024-04-23T10:35:59.753 INFO:journalctl@ceph.mon.a.smithi114.stdout:Apr 23 10:35:59 smithi114 ceph-mon[29255]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=e=up:active} 3 up:standby-replay 6 up:standby 2024-04-23T10:35:59.828 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 23 10:35:59 smithi204 ceph-mon[32649]: pgmap v292: 97 pgs: 97 active+clean; 194 MiB data, 2.5 GiB used, 1.0 TiB / 1.0 TiB avail; 338 KiB/s rd, 717 KiB/s wr, 66 op/s 2024-04-23T10:35:59.828 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 23 10:35:59 smithi204 ceph-mon[32649]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=e=up:active} 3 up:standby-replay 6 up:standby 2024-04-23T10:35:59.935 INFO:tasks.workunit.client.0.smithi114.stdout: 1 641631 6.83 MB/sec execute 368 sec latency 19.024 ms 2024-04-23T10:36:00.935 INFO:tasks.workunit.client.0.smithi114.stdout: 1 642872 6.84 MB/sec execute 369 sec latency 545.496 ms 2024-04-23T10:36:01.742 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 23 10:36:01 smithi119 ceph-mon[32608]: pgmap v293: 97 pgs: 97 active+clean; 194 MiB data, 2.5 GiB used, 1.0 TiB / 1.0 TiB avail; 338 KiB/s rd, 717 KiB/s wr, 66 op/s 2024-04-23T10:36:01.754 INFO:journalctl@ceph.mon.a.smithi114.stdout:Apr 23 10:36:01 smithi114 ceph-mon[29255]: pgmap v293: 97 pgs: 97 active+clean; 194 MiB data, 2.5 GiB used, 1.0 TiB / 1.0 TiB avail; 338 KiB/s rd, 717 KiB/s wr, 66 op/s 2024-04-23T10:36:01.828 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 23 10:36:01 smithi204 ceph-mon[32649]: pgmap v293: 97 pgs: 97 active+clean; 194 MiB data, 2.5 GiB used, 1.0 TiB / 1.0 TiB avail; 338 KiB/s rd, 717 KiB/s wr, 66 op/s 2024-04-23T10:36:01.935 INFO:tasks.workunit.client.0.smithi114.stdout: 1 642872 6.82 MB/sec execute 370 sec latency 1545.593 ms 2024-04-23T10:36:02.935 INFO:tasks.workunit.client.0.smithi114.stdout: 1 642872 6.80 MB/sec execute 371 sec latency 2545.634 ms 2024-04-23T10:36:03.754 INFO:journalctl@ceph.mon.a.smithi114.stdout:Apr 23 10:36:03 smithi114 ceph-mon[29255]: pgmap v294: 97 pgs: 97 active+clean; 193 MiB data, 2.5 GiB used, 1.0 TiB / 1.0 TiB avail; 647 KiB/s rd, 1.5 MiB/s wr, 121 op/s 2024-04-23T10:36:03.754 INFO:journalctl@ceph.mon.a.smithi114.stdout:Apr 23 10:36:03 smithi114 ceph-mon[29255]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=e=up:active} 3 up:standby-replay 6 up:standby 2024-04-23T10:36:03.828 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 23 10:36:03 smithi204 ceph-mon[32649]: pgmap v294: 97 pgs: 97 active+clean; 193 MiB data, 2.5 GiB used, 1.0 TiB / 1.0 TiB avail; 647 KiB/s rd, 1.5 MiB/s wr, 121 op/s 2024-04-23T10:36:03.828 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 23 10:36:03 smithi204 ceph-mon[32649]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=e=up:active} 3 up:standby-replay 6 up:standby 2024-04-23T10:36:03.935 INFO:tasks.workunit.client.0.smithi114.stdout: 1 642872 6.79 MB/sec execute 372 sec latency 3545.755 ms 2024-04-23T10:36:03.992 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 23 10:36:03 smithi119 ceph-mon[32608]: pgmap v294: 97 pgs: 97 active+clean; 193 MiB data, 2.5 GiB used, 1.0 TiB / 1.0 TiB avail; 647 KiB/s rd, 1.5 MiB/s wr, 121 op/s 2024-04-23T10:36:03.992 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 23 10:36:03 smithi119 ceph-mon[32608]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=e=up:active} 3 up:standby-replay 6 up:standby 2024-04-23T10:36:04.935 INFO:tasks.workunit.client.0.smithi114.stdout: 1 642918 6.77 MB/sec execute 373 sec latency 3791.760 ms 2024-04-23T10:36:05.754 INFO:journalctl@ceph.mon.a.smithi114.stdout:Apr 23 10:36:05 smithi114 ceph-mon[29255]: pgmap v295: 97 pgs: 97 active+clean; 193 MiB data, 2.5 GiB used, 1.0 TiB / 1.0 TiB avail; 374 KiB/s rd, 1.2 MiB/s wr, 87 op/s 2024-04-23T10:36:05.827 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 23 10:36:05 smithi204 ceph-mon[32649]: pgmap v295: 97 pgs: 97 active+clean; 193 MiB data, 2.5 GiB used, 1.0 TiB / 1.0 TiB avail; 374 KiB/s rd, 1.2 MiB/s wr, 87 op/s 2024-04-23T10:36:05.935 INFO:tasks.workunit.client.0.smithi114.stdout: 1 642918 6.75 MB/sec execute 374 sec latency 1675.194 ms 2024-04-23T10:36:05.992 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 23 10:36:05 smithi119 ceph-mon[32608]: pgmap v295: 97 pgs: 97 active+clean; 193 MiB data, 2.5 GiB used, 1.0 TiB / 1.0 TiB avail; 374 KiB/s rd, 1.2 MiB/s wr, 87 op/s 2024-04-23T10:36:06.935 INFO:tasks.workunit.client.0.smithi114.stdout: 1 642918 6.73 MB/sec execute 375 sec latency 2675.256 ms 2024-04-23T10:36:07.754 INFO:journalctl@ceph.mon.a.smithi114.stdout:Apr 23 10:36:07 smithi114 ceph-mon[29255]: pgmap v296: 97 pgs: 97 active+clean; 193 MiB data, 2.5 GiB used, 1.0 TiB / 1.0 TiB avail; 425 KiB/s rd, 1.5 MiB/s wr, 119 op/s 2024-04-23T10:36:07.828 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 23 10:36:07 smithi204 ceph-mon[32649]: pgmap v296: 97 pgs: 97 active+clean; 193 MiB data, 2.5 GiB used, 1.0 TiB / 1.0 TiB avail; 425 KiB/s rd, 1.5 MiB/s wr, 119 op/s 2024-04-23T10:36:07.936 INFO:tasks.workunit.client.0.smithi114.stdout: 1 642918 6.71 MB/sec execute 376 sec latency 3675.304 ms 2024-04-23T10:36:07.992 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 23 10:36:07 smithi119 ceph-mon[32608]: pgmap v296: 97 pgs: 97 active+clean; 193 MiB data, 2.5 GiB used, 1.0 TiB / 1.0 TiB avail; 425 KiB/s rd, 1.5 MiB/s wr, 119 op/s 2024-04-23T10:36:08.936 INFO:tasks.workunit.client.0.smithi114.stdout: 1 642918 6.70 MB/sec execute 377 sec latency 4675.463 ms 2024-04-23T10:36:09.754 INFO:journalctl@ceph.mon.a.smithi114.stdout:Apr 23 10:36:09 smithi114 ceph-mon[29255]: pgmap v297: 97 pgs: 97 active+clean; 193 MiB data, 2.5 GiB used, 1.0 TiB / 1.0 TiB avail; 359 KiB/s rd, 1.1 MiB/s wr, 86 op/s 2024-04-23T10:36:09.828 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 23 10:36:09 smithi204 ceph-mon[32649]: pgmap v297: 97 pgs: 97 active+clean; 193 MiB data, 2.5 GiB used, 1.0 TiB / 1.0 TiB avail; 359 KiB/s rd, 1.1 MiB/s wr, 86 op/s 2024-04-23T10:36:09.936 INFO:tasks.workunit.client.0.smithi114.stdout: 1 642939 6.68 MB/sec execute 378 sec latency 4928.841 ms 2024-04-23T10:36:09.992 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 23 10:36:09 smithi119 ceph-mon[32608]: pgmap v297: 97 pgs: 97 active+clean; 193 MiB data, 2.5 GiB used, 1.0 TiB / 1.0 TiB avail; 359 KiB/s rd, 1.1 MiB/s wr, 86 op/s 2024-04-23T10:36:10.936 INFO:tasks.workunit.client.0.smithi114.stdout: 1 642939 6.66 MB/sec execute 379 sec latency 1726.817 ms 2024-04-23T10:36:11.623 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:36:11.622+0000 7f1636fc6640 1 -- 172.21.15.114:0/3133348335 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f163010c5b0 msgr2=0x7f163010e9e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T10:36:11.623 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:36:11.622+0000 7f1636fc6640 1 --2- 172.21.15.114:0/3133348335 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f163010c5b0 0x7f163010e9e0 secure :-1 s=READY pgs=821 cs=0 l=1 rev1=1 crypto rx=0x7f1628009ca0 tx=0x7f162802a950 comp rx=0 tx=0).stop 2024-04-23T10:36:11.623 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:36:11.622+0000 7f1636fc6640 1 -- 172.21.15.114:0/3133348335 shutdown_connections 2024-04-23T10:36:11.623 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:36:11.622+0000 7f1636fc6640 1 --2- 172.21.15.114:0/3133348335 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f163010c5b0 0x7f163010e9e0 unknown :-1 s=CLOSED pgs=821 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:36:11.623 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:36:11.622+0000 7f1636fc6640 1 --2- 172.21.15.114:0/3133348335 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f1630103b00 0x7f163010bfe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:36:11.623 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:36:11.622+0000 7f1636fc6640 1 --2- 172.21.15.114:0/3133348335 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f16301031e0 0x7f16301035c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:36:11.624 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:36:11.622+0000 7f1636fc6640 1 -- 172.21.15.114:0/3133348335 >> 172.21.15.114:0/3133348335 conn(0x7f1630077f60 msgr2=0x7f1630078370 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T10:36:11.624 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:36:11.622+0000 7f1636fc6640 1 -- 172.21.15.114:0/3133348335 shutdown_connections 2024-04-23T10:36:11.624 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:36:11.622+0000 7f1636fc6640 1 -- 172.21.15.114:0/3133348335 wait complete. 2024-04-23T10:36:11.624 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:36:11.622+0000 7f1636fc6640 1 Processor -- start 2024-04-23T10:36:11.624 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:36:11.623+0000 7f1636fc6640 1 -- start start 2024-04-23T10:36:11.624 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:36:11.623+0000 7f1636fc6640 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f16301031e0 0x7f16301986b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T10:36:11.624 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:36:11.623+0000 7f1636fc6640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f1630103b00 0x7f1630198bf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T10:36:11.624 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:36:11.623+0000 7f1636fc6640 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f163010c5b0 0x7f163019cf20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T10:36:11.624 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:36:11.623+0000 7f1636fc6640 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x7f16301121d0 con 0x7f1630103b00 2024-04-23T10:36:11.624 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:36:11.623+0000 7f1636fc6640 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 -- 0x7f1630112050 con 0x7f16301031e0 2024-04-23T10:36:11.624 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:36:11.623+0000 7f1636fc6640 1 -- --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_getmap magic: 0 -- 0x7f1630112350 con 0x7f163010c5b0 2024-04-23T10:36:11.624 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:36:11.623+0000 7f162ffff640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f1630103b00 0x7f1630198bf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T10:36:11.624 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:36:11.623+0000 7f162ffff640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f1630103b00 0x7f1630198bf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.114:43322/0 (socket says 172.21.15.114:43322) 2024-04-23T10:36:11.625 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:36:11.623+0000 7f162ffff640 1 -- 172.21.15.114:0/1992239175 learned_addr learned my addr 172.21.15.114:0/1992239175 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-04-23T10:36:11.625 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:36:11.623+0000 7f1634d62640 1 --2- 172.21.15.114:0/1992239175 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f16301031e0 0x7f16301986b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T10:36:11.625 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:36:11.623+0000 7f1635563640 1 --2- 172.21.15.114:0/1992239175 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f163010c5b0 0x7f163019cf20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T10:36:11.625 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:36:11.623+0000 7f162ffff640 1 -- 172.21.15.114:0/1992239175 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f163010c5b0 msgr2=0x7f163019cf20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T10:36:11.625 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:36:11.623+0000 7f162ffff640 1 --2- 172.21.15.114:0/1992239175 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f163010c5b0 0x7f163019cf20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:36:11.625 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:36:11.623+0000 7f162ffff640 1 -- 172.21.15.114:0/1992239175 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f16301031e0 msgr2=0x7f16301986b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T10:36:11.625 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:36:11.623+0000 7f162ffff640 1 --2- 172.21.15.114:0/1992239175 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f16301031e0 0x7f16301986b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:36:11.625 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:36:11.623+0000 7f162ffff640 1 -- 172.21.15.114:0/1992239175 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f163019d650 con 0x7f1630103b00 2024-04-23T10:36:11.625 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:36:11.623+0000 7f162ffff640 1 --2- 172.21.15.114:0/1992239175 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f1630103b00 0x7f1630198bf0 secure :-1 s=READY pgs=822 cs=0 l=1 rev1=1 crypto rx=0x7f162000d770 tx=0x7f162000dc30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T10:36:11.625 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:36:11.624+0000 7f162dffb640 1 -- 172.21.15.114:0/1992239175 <== mon.0 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1620017070 con 0x7f1630103b00 2024-04-23T10:36:11.625 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:36:11.624+0000 7f1636fc6640 1 -- 172.21.15.114:0/1992239175 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f163019d940 con 0x7f1630103b00 2024-04-23T10:36:11.625 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:36:11.624+0000 7f162dffb640 1 -- 172.21.15.114:0/1992239175 <== mon.0 v2:172.21.15.114:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f16200054d0 con 0x7f1630103b00 2024-04-23T10:36:11.625 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:36:11.624+0000 7f162dffb640 1 -- 172.21.15.114:0/1992239175 <== mon.0 v2:172.21.15.114:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1620004c30 con 0x7f1630103b00 2024-04-23T10:36:11.626 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:36:11.624+0000 7f1636fc6640 1 -- 172.21.15.114:0/1992239175 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f16301a51d0 con 0x7f1630103b00 2024-04-23T10:36:11.633 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:36:11.626+0000 7f162dffb640 1 -- 172.21.15.114:0/1992239175 <== mon.0 v2:172.21.15.114:3300/0 4 ==== mgrmap(e 26) ==== 138778+0+0 (secure 0 0 0) 0x7f1620004e10 con 0x7f1630103b00 2024-04-23T10:36:11.633 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:36:11.626+0000 7f1639dbc740 1 Processor -- start 2024-04-23T10:36:11.633 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:36:11.626+0000 7f162dffb640 1 --2- 172.21.15.114:0/1992239175 >> [v2:172.21.15.114:6800/1395693466,v1:172.21.15.114:6801/1395693466] conn(0x7f161c0a3dc0 0x7f161c0a6280 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T10:36:11.633 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:36:11.626+0000 7f1639dbc740 1 -- start start 2024-04-23T10:36:11.633 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:36:11.627+0000 7f162dffb640 1 -- 172.21.15.114:0/1992239175 <== mon.0 v2:172.21.15.114:3300/0 5 ==== osd_map(100..100 src has 1..100) ==== 8629+0+0 (secure 0 0 0) 0x7f16200d0180 con 0x7f1630103b00 2024-04-23T10:36:11.633 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:36:11.627+0000 7f1635563640 1 --2- 172.21.15.114:0/1992239175 >> [v2:172.21.15.114:6800/1395693466,v1:172.21.15.114:6801/1395693466] conn(0x7f161c0a3dc0 0x7f161c0a6280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T10:36:11.634 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:36:11.627+0000 7f1639dbc740 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x5651cd4a8540 0x5651cd4a8920 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T10:36:11.634 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:36:11.627+0000 7f1639dbc740 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x5651cd4a8e60 0x5651cd4b76d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T10:36:11.634 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:36:11.627+0000 7f1639dbc740 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x5651cd4b7c10 0x5651cd4ba000 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T10:36:11.634 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:36:11.627+0000 7f1639dbc740 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x5651cd40d970 con 0x5651cd4a8540 2024-04-23T10:36:11.634 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:36:11.627+0000 7f1639dbc740 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 -- 0x5651cd3aa4a0 con 0x5651cd4b7c10 2024-04-23T10:36:11.634 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:36:11.627+0000 7f1639dbc740 1 -- --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_getmap magic: 0 -- 0x5651cd409930 con 0x5651cd4a8e60 2024-04-23T10:36:11.634 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:36:11.627+0000 7f1634d62640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x5651cd4a8540 0x5651cd4a8920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T10:36:11.634 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:36:11.627+0000 7f1634d62640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x5651cd4a8540 0x5651cd4a8920 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.114:43326/0 (socket says 172.21.15.114:43326) 2024-04-23T10:36:11.634 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:36:11.627+0000 7f1634d62640 1 -- 172.21.15.114:0/3792496624 learned_addr learned my addr 172.21.15.114:0/3792496624 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-04-23T10:36:11.634 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:36:11.627+0000 7f162ffff640 1 --2- 172.21.15.114:0/3792496624 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x5651cd4a8e60 0x5651cd4b76d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T10:36:11.634 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:36:11.627+0000 7f1635563640 1 --2- 172.21.15.114:0/3792496624 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x5651cd4b7c10 0x5651cd4ba000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T10:36:11.634 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:36:11.627+0000 7f1634d62640 1 -- 172.21.15.114:0/3792496624 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x5651cd4a8e60 msgr2=0x5651cd4b76d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T10:36:11.634 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:36:11.627+0000 7f1634d62640 1 --2- 172.21.15.114:0/3792496624 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x5651cd4a8e60 0x5651cd4b76d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:36:11.634 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:36:11.627+0000 7f1634d62640 1 -- 172.21.15.114:0/3792496624 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x5651cd4b7c10 msgr2=0x5651cd4ba000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T10:36:11.635 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:36:11.627+0000 7f1634d62640 1 --2- 172.21.15.114:0/3792496624 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x5651cd4b7c10 0x5651cd4ba000 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:36:11.635 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:36:11.627+0000 7f1634d62640 1 -- 172.21.15.114:0/3792496624 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5651cd120210 con 0x5651cd4a8540 2024-04-23T10:36:11.635 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:36:11.627+0000 7f1635563640 1 --2- 172.21.15.114:0/3792496624 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x5651cd4b7c10 0x5651cd4ba000 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-23T10:36:11.635 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:36:11.627+0000 7f1635563640 1 --2- 172.21.15.114:0/1992239175 >> [v2:172.21.15.114:6800/1395693466,v1:172.21.15.114:6801/1395693466] conn(0x7f161c0a3dc0 0x7f161c0a6280 secure :-1 s=READY pgs=127 cs=0 l=1 rev1=1 crypto rx=0x7f1628005e30 tx=0x7f16280099a0 comp rx=0 tx=0).ready entity=mgr.15291 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T10:36:11.635 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:36:11.627+0000 7f1634d62640 1 --2- 172.21.15.114:0/3792496624 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x5651cd4a8540 0x5651cd4a8920 secure :-1 s=READY pgs=823 cs=0 l=1 rev1=1 crypto rx=0x7f161800bd10 tx=0x7f16180107a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=74fb333bdbd64395 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T10:36:11.635 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:36:11.628+0000 7f1626ffd640 1 -- 172.21.15.114:0/3792496624 <== mon.0 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1618004030 con 0x5651cd4a8540 2024-04-23T10:36:11.635 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:36:11.628+0000 7f1626ffd640 1 -- 172.21.15.114:0/3792496624 <== mon.0 v2:172.21.15.114:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f16180035f0 con 0x5651cd4a8540 2024-04-23T10:36:11.635 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:36:11.628+0000 7f1626ffd640 1 -- 172.21.15.114:0/3792496624 <== mon.0 v2:172.21.15.114:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1618011770 con 0x5651cd4a8540 2024-04-23T10:36:11.635 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:36:11.628+0000 7f1639dbc740 1 -- 172.21.15.114:0/3792496624 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x5651cd4a8540 msgr2=0x5651cd4a8920 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T10:36:11.635 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:36:11.628+0000 7f1639dbc740 1 --2- 172.21.15.114:0/3792496624 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x5651cd4a8540 0x5651cd4a8920 secure :-1 s=READY pgs=823 cs=0 l=1 rev1=1 crypto rx=0x7f161800bd10 tx=0x7f16180107a0 comp rx=0 tx=0).stop 2024-04-23T10:36:11.635 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:36:11.628+0000 7f1639dbc740 1 -- 172.21.15.114:0/3792496624 shutdown_connections 2024-04-23T10:36:11.635 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:36:11.628+0000 7f1639dbc740 1 --2- 172.21.15.114:0/3792496624 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x5651cd4b7c10 0x5651cd4ba000 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:36:11.635 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:36:11.628+0000 7f1639dbc740 1 --2- 172.21.15.114:0/3792496624 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x5651cd4a8e60 0x5651cd4b76d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:36:11.635 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:36:11.628+0000 7f1639dbc740 1 --2- 172.21.15.114:0/3792496624 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x5651cd4a8540 0x5651cd4a8920 unknown :-1 s=CLOSED pgs=823 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:36:11.636 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:36:11.628+0000 7f1639dbc740 1 -- 172.21.15.114:0/3792496624 >> 172.21.15.114:0/3792496624 conn(0x5651cd4a4b30 msgr2=0x5651cd4a4f10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T10:36:11.636 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:36:11.628+0000 7f1639dbc740 1 -- 172.21.15.114:0/3792496624 shutdown_connections 2024-04-23T10:36:11.636 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:36:11.628+0000 7f1639dbc740 1 -- 172.21.15.114:0/3792496624 wait complete. 2024-04-23T10:36:11.636 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:36:11.629+0000 7f1639dbc740 1 -- start start 2024-04-23T10:36:11.636 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:36:11.629+0000 7f1639dbc740 1 Processor -- start 2024-04-23T10:36:11.636 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:36:11.629+0000 7f1639dbc740 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x5651cd4a8540 0x5651cd4b7590 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T10:36:11.636 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:36:11.629+0000 7f1639dbc740 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x5651cd4a8e60 0x5651cd4b4880 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T10:36:11.636 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:36:11.629+0000 7f1639dbc740 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x5651cd4b7c10 0x5651cd4b4e00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T10:36:11.636 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:36:11.629+0000 7f1639dbc740 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x5651cd4ba840 con 0x5651cd4a8540 2024-04-23T10:36:11.636 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:36:11.629+0000 7f1639dbc740 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 -- 0x5651cd3a80b0 con 0x5651cd4b7c10 2024-04-23T10:36:11.636 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:36:11.629+0000 7f1639dbc740 1 -- --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_getmap magic: 0 -- 0x5651cd4ba9c0 con 0x5651cd4a8e60 2024-04-23T10:36:11.636 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:36:11.629+0000 7f1639dbc740 10 client.0 _ll_register_callbacks cb 0x5651cd395d30 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-23T10:36:11.636 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:36:11.629+0000 7f1634d62640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x5651cd4a8540 0x5651cd4b7590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T10:36:11.636 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:36:11.629+0000 7f1634d62640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x5651cd4a8540 0x5651cd4b7590 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.114:43334/0 (socket says 172.21.15.114:43334) 2024-04-23T10:36:11.637 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:36:11.629+0000 7f1634d62640 1 -- 172.21.15.114:0/3079130610 learned_addr learned my addr 172.21.15.114:0/3079130610 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-04-23T10:36:11.637 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:36:11.629+0000 7f162ffff640 1 --2- 172.21.15.114:0/3079130610 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x5651cd4a8e60 0x5651cd4b4880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T10:36:11.637 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:36:11.630+0000 7f1635563640 1 --2- 172.21.15.114:0/3079130610 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x5651cd4b7c10 0x5651cd4b4e00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T10:36:11.637 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:36:11.630+0000 7f1634d62640 1 -- 172.21.15.114:0/3079130610 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x5651cd4a8e60 msgr2=0x5651cd4b4880 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T10:36:11.637 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:36:11.630+0000 7f1634d62640 1 --2- 172.21.15.114:0/3079130610 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x5651cd4a8e60 0x5651cd4b4880 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:36:11.637 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:36:11.630+0000 7f1634d62640 1 -- 172.21.15.114:0/3079130610 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x5651cd4b7c10 msgr2=0x5651cd4b4e00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T10:36:11.637 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:36:11.630+0000 7f1634d62640 1 --2- 172.21.15.114:0/3079130610 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x5651cd4b7c10 0x5651cd4b4e00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:36:11.637 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:36:11.630+0000 7f1634d62640 1 -- 172.21.15.114:0/3079130610 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5651cd1ec620 con 0x5651cd4a8540 2024-04-23T10:36:11.637 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:36:11.630+0000 7f1635563640 1 --2- 172.21.15.114:0/3079130610 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x5651cd4b7c10 0x5651cd4b4e00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-23T10:36:11.637 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:36:11.630+0000 7f1634d62640 1 --2- 172.21.15.114:0/3079130610 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x5651cd4a8540 0x5651cd4b7590 secure :-1 s=READY pgs=824 cs=0 l=1 rev1=1 crypto rx=0x7f161800b620 tx=0x7f161800ba20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b0f778bd4a54a1fe server_cookie=0 in_seq=0 out_seq=0 2024-04-23T10:36:11.637 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:36:11.630+0000 7f160ffff640 10 client.0 ms_handle_connect on v2:172.21.15.114:3300/0 2024-04-23T10:36:11.637 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:36:11.630+0000 7f160ffff640 1 -- 172.21.15.114:0/3079130610 <== mon.0 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1618029060 con 0x5651cd4a8540 2024-04-23T10:36:11.637 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:36:11.630+0000 7f160ffff640 1 -- 172.21.15.114:0/3079130610 <== mon.0 v2:172.21.15.114:3300/0 2 ==== osd_map(100..100 src has 1..100) ==== 8629+0+0 (secure 0 0 0) 0x7f1618003ab0 con 0x5651cd4a8540 2024-04-23T10:36:11.638 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:36:11.630+0000 7f1639dbc740 1 -- 172.21.15.114:0/3079130610 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f16180035f0 con 0x5651cd4a8540 2024-04-23T10:36:11.638 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:36:11.630+0000 7f160ffff640 1 -- 172.21.15.114:0/3079130610 <== mon.0 v2:172.21.15.114:3300/0 3 ==== mon_get_version_reply(handle=1 version=36) ==== 24+0+0 (secure 0 0 0) 0x7f1618004000 con 0x5651cd4a8540 2024-04-23T10:36:11.638 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:36:11.630+0000 7f1639dbc740 10 client.16728 fetch_fsmap learned FSMap version 36 2024-04-23T10:36:11.638 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:36:11.630+0000 7f1639dbc740 1 -- 172.21.15.114:0/3079130610 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({config=0+,fsmap=36,monmap=4+}) -- 0x5651cd4c6700 con 0x5651cd4a8540 2024-04-23T10:36:11.638 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:36:11.630+0000 7f160ffff640 1 -- 172.21.15.114:0/3079130610 <== mon.0 v2:172.21.15.114:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f16180035f0 con 0x5651cd4a8540 2024-04-23T10:36:11.638 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:36:11.630+0000 7f160ffff640 1 -- 172.21.15.114:0/3079130610 <== mon.0 v2:172.21.15.114:3300/0 5 ==== fsmap(e 36) ==== 8354+0+0 (secure 0 0 0) 0x7f1618004850 con 0x5651cd4a8540 2024-04-23T10:36:11.638 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:36:11.631+0000 7f1639dbc740 10 client.16728 fetch_fsmap finished waiting for FSMap version 36 2024-04-23T10:36:11.638 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:36:11.631+0000 7f1639dbc740 10 client.16728 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-04-23T10:36:11.638 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:36:11.631+0000 7f1639dbc740 20 client.16728 populate_metadata read hostname 'smithi114' 2024-04-23T10:36:11.638 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:36:11.631+0000 7f1639dbc740 1 --2- 172.21.15.114:0/3079130610 >> [v2:172.21.15.119:6838/937297340,v1:172.21.15.119:6839/937297340] conn(0x5651cd4c7b10 0x5651cd4c9f00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T10:36:11.638 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:36:11.631+0000 7f1639dbc740 4 client.16728 mds_command: new command op to 24545 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-23T10:36:11.638 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:36:11.631+0000 7f1639dbc740 1 -- 172.21.15.114:0/3079130610 --> [v2:172.21.15.119:6838/937297340,v1:172.21.15.119:6839/937297340] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x5651cd41b390 con 0x5651cd4c7b10 2024-04-23T10:36:11.638 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:36:11.631+0000 7f162ffff640 1 --2- 172.21.15.114:0/3079130610 >> [v2:172.21.15.119:6838/937297340,v1:172.21.15.119:6839/937297340] conn(0x5651cd4c7b10 0x5651cd4c9f00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T10:36:11.638 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:36:11.631+0000 7f162ffff640 1 --2- 172.21.15.114:0/3079130610 >> [v2:172.21.15.119:6838/937297340,v1:172.21.15.119:6839/937297340] conn(0x5651cd4c7b10 0x5651cd4c9f00 crc :-1 s=READY pgs=146 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=10cbc1061eac309d server_cookie=a8b056b37f59a49c in_seq=0 out_seq=0 2024-04-23T10:36:11.639 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:36:11.631+0000 7f160ffff640 10 client.16728 ms_handle_connect on v2:172.21.15.119:6838/937297340 2024-04-23T10:36:11.639 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:36:11.632+0000 7f160ffff640 1 -- 172.21.15.114:0/3079130610 <== mds.0 v2:172.21.15.119:6838/937297340 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x5651cd41b390 con 0x5651cd4c7b10 2024-04-23T10:36:11.639 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:36:11.632+0000 7f160ffff640 10 client.16728 handle_command_reply: tid=0 2024-04-23T10:36:11.639 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:36:11.632+0000 7f1639dbc740 1 client.16728 shutdown 2024-04-23T10:36:11.639 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:36:11.632+0000 7f1639dbc740 1 -- 172.21.15.114:0/3079130610 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x5651cd4a8540 msgr2=0x5651cd4b7590 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T10:36:11.639 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:36:11.632+0000 7f1639dbc740 1 --2- 172.21.15.114:0/3079130610 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x5651cd4a8540 0x5651cd4b7590 secure :-1 s=READY pgs=824 cs=0 l=1 rev1=1 crypto rx=0x7f161800b620 tx=0x7f161800ba20 comp rx=0 tx=0).stop 2024-04-23T10:36:11.639 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:36:11.632+0000 7f1639dbc740 1 -- 172.21.15.114:0/3079130610 shutdown_connections 2024-04-23T10:36:11.639 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:36:11.632+0000 7f1639dbc740 1 --2- 172.21.15.114:0/3079130610 >> [v2:172.21.15.119:6838/937297340,v1:172.21.15.119:6839/937297340] conn(0x5651cd4c7b10 0x5651cd4c9f00 crc :-1 s=READY pgs=146 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:36:11.639 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:36:11.632+0000 7f1639dbc740 1 --2- 172.21.15.114:0/3079130610 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x5651cd4b7c10 0x5651cd4b4e00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:36:11.639 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:36:11.632+0000 7f1639dbc740 1 --2- 172.21.15.114:0/3079130610 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x5651cd4a8e60 0x5651cd4b4880 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:36:11.640 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:36:11.632+0000 7f160ffff640 0 client.16728 ms_handle_reset on v2:172.21.15.119:6838/937297340 2024-04-23T10:36:11.640 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:36:11.632+0000 7f1639dbc740 1 --2- 172.21.15.114:0/3079130610 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x5651cd4a8540 0x5651cd4b7590 unknown :-1 s=CLOSED pgs=824 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:36:11.640 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:36:11.632+0000 7f1639dbc740 1 -- 172.21.15.114:0/3079130610 >> 172.21.15.114:0/3079130610 conn(0x5651cd4a4b30 msgr2=0x5651cd4aa9a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T10:36:11.640 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:36:11.632+0000 7f1639dbc740 1 -- 172.21.15.114:0/3079130610 shutdown_connections 2024-04-23T10:36:11.640 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:36:11.632+0000 7f1639dbc740 1 -- 172.21.15.114:0/3079130610 wait complete. 2024-04-23T10:36:11.640 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:36:11.632+0000 7f1639dbc740 20 client.16728 trim_cache size 0 max 16384 2024-04-23T10:36:11.651 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:36:11.647+0000 7f1639dbc740 1 Processor -- start 2024-04-23T10:36:11.651 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:36:11.647+0000 7f1639dbc740 1 -- start start 2024-04-23T10:36:11.651 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:36:11.647+0000 7f1639dbc740 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x5651cd4a8540 0x5651cd4c3700 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T10:36:11.651 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:36:11.647+0000 7f1639dbc740 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x5651cd4a8e60 0x5651cd4b70c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T10:36:11.651 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:36:11.647+0000 7f1639dbc740 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x5651cd4b7c10 0x5651cd4cf4d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T10:36:11.651 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:36:11.647+0000 7f1639dbc740 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x7f16301ae3b0 con 0x5651cd4a8e60 2024-04-23T10:36:11.651 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:36:11.647+0000 7f1639dbc740 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 -- 0x7f16301abbf0 con 0x5651cd4a8540 2024-04-23T10:36:11.651 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:36:11.647+0000 7f1639dbc740 1 -- --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_getmap magic: 0 -- 0x7f16301a9a60 con 0x5651cd4b7c10 2024-04-23T10:36:11.651 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:36:11.647+0000 7f162ffff640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x5651cd4a8e60 0x5651cd4b70c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T10:36:11.651 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:36:11.647+0000 7f162ffff640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x5651cd4a8e60 0x5651cd4b70c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.114:43342/0 (socket says 172.21.15.114:43342) 2024-04-23T10:36:11.651 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:36:11.647+0000 7f162ffff640 1 -- 172.21.15.114:0/1566458053 learned_addr learned my addr 172.21.15.114:0/1566458053 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-04-23T10:36:11.652 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:36:11.648+0000 7f162ffff640 1 -- 172.21.15.114:0/1566458053 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x5651cd4b7c10 msgr2=0x5651cd4cf4d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T10:36:11.652 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:36:11.648+0000 7f162ffff640 1 --2- 172.21.15.114:0/1566458053 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x5651cd4b7c10 0x5651cd4cf4d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:36:11.652 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:36:11.648+0000 7f1634d62640 1 --2- 172.21.15.114:0/1566458053 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x5651cd4a8540 0x5651cd4c3700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T10:36:11.652 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:36:11.648+0000 7f162ffff640 1 -- 172.21.15.114:0/1566458053 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x5651cd4a8540 msgr2=0x5651cd4c3700 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T10:36:11.652 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:36:11.648+0000 7f162ffff640 1 --2- 172.21.15.114:0/1566458053 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x5651cd4a8540 0x5651cd4c3700 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:36:11.652 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:36:11.648+0000 7f162ffff640 1 -- 172.21.15.114:0/1566458053 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5651cd4a02e0 con 0x5651cd4a8e60 2024-04-23T10:36:11.652 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:36:11.648+0000 7f162ffff640 1 --2- 172.21.15.114:0/1566458053 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x5651cd4a8e60 0x5651cd4b70c0 secure :-1 s=READY pgs=825 cs=0 l=1 rev1=1 crypto rx=0x7f1620098bd0 tx=0x7f1620004710 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b92a32749e073ff2 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T10:36:11.652 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:36:11.648+0000 7f16267fc640 1 -- 172.21.15.114:0/1566458053 <== mon.0 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f16200b4070 con 0x5651cd4a8e60 2024-04-23T10:36:11.652 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:36:11.648+0000 7f16267fc640 1 -- 172.21.15.114:0/1566458053 <== mon.0 v2:172.21.15.114:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f162009a2a0 con 0x5651cd4a8e60 2024-04-23T10:36:11.652 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:36:11.648+0000 7f16267fc640 1 -- 172.21.15.114:0/1566458053 <== mon.0 v2:172.21.15.114:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1620020040 con 0x5651cd4a8e60 2024-04-23T10:36:11.652 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:36:11.648+0000 7f1639dbc740 1 -- 172.21.15.114:0/1566458053 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x5651cd4a8e60 msgr2=0x5651cd4b70c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T10:36:11.652 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:36:11.648+0000 7f1639dbc740 1 --2- 172.21.15.114:0/1566458053 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x5651cd4a8e60 0x5651cd4b70c0 secure :-1 s=READY pgs=825 cs=0 l=1 rev1=1 crypto rx=0x7f1620098bd0 tx=0x7f1620004710 comp rx=0 tx=0).stop 2024-04-23T10:36:11.652 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:36:11.649+0000 7f1639dbc740 1 -- 172.21.15.114:0/1566458053 shutdown_connections 2024-04-23T10:36:11.653 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:36:11.649+0000 7f1639dbc740 1 --2- 172.21.15.114:0/1566458053 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x5651cd4b7c10 0x5651cd4cf4d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:36:11.653 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:36:11.649+0000 7f1639dbc740 1 --2- 172.21.15.114:0/1566458053 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x5651cd4a8e60 0x5651cd4b70c0 unknown :-1 s=CLOSED pgs=825 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:36:11.653 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:36:11.649+0000 7f1639dbc740 1 --2- 172.21.15.114:0/1566458053 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x5651cd4a8540 0x5651cd4c3700 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:36:11.653 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:36:11.649+0000 7f1639dbc740 1 -- 172.21.15.114:0/1566458053 >> 172.21.15.114:0/1566458053 conn(0x5651cd4a4b30 msgr2=0x5651cd4b39e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T10:36:11.653 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:36:11.649+0000 7f1639dbc740 1 -- 172.21.15.114:0/1566458053 shutdown_connections 2024-04-23T10:36:11.653 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:36:11.649+0000 7f1639dbc740 1 -- 172.21.15.114:0/1566458053 wait complete. 2024-04-23T10:36:11.653 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:36:11.649+0000 7f1639dbc740 1 -- start start 2024-04-23T10:36:11.653 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:36:11.649+0000 7f1639dbc740 1 Processor -- start 2024-04-23T10:36:11.653 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:36:11.649+0000 7f1639dbc740 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x5651cd4a8540 0x5651cd4b2560 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T10:36:11.653 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:36:11.649+0000 7f1639dbc740 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x5651cd4a8e60 0x5651cd4bfc10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T10:36:11.653 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:36:11.649+0000 7f1639dbc740 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x5651cd4b7c10 0x5651cd4b0af0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T10:36:11.653 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:36:11.649+0000 7f1639dbc740 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x7f16301aa0b0 con 0x5651cd4b7c10 2024-04-23T10:36:11.653 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:36:11.649+0000 7f1639dbc740 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 -- 0x7f16301ab520 con 0x5651cd4a8540 2024-04-23T10:36:11.653 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:36:11.649+0000 7f1639dbc740 1 -- --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_getmap magic: 0 -- 0x7f16301aa780 con 0x5651cd4a8e60 2024-04-23T10:36:11.654 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:36:11.649+0000 7f1639dbc740 10 client.0 _ll_register_callbacks cb 0x7f161800b8e0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-23T10:36:11.654 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:36:11.649+0000 7f1635563640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x5651cd4b7c10 0x5651cd4b0af0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T10:36:11.654 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:36:11.650+0000 7f1635563640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x5651cd4b7c10 0x5651cd4b0af0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.114:43354/0 (socket says 172.21.15.114:43354) 2024-04-23T10:36:11.654 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:36:11.650+0000 7f1635563640 1 -- 172.21.15.114:0/1762125541 learned_addr learned my addr 172.21.15.114:0/1762125541 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-04-23T10:36:11.654 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:36:11.650+0000 7f162ffff640 1 --2- 172.21.15.114:0/1762125541 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x5651cd4a8e60 0x5651cd4bfc10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T10:36:11.654 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:36:11.650+0000 7f1634d62640 1 --2- 172.21.15.114:0/1762125541 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x5651cd4a8540 0x5651cd4b2560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T10:36:11.654 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:36:11.650+0000 7f1635563640 1 -- 172.21.15.114:0/1762125541 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x5651cd4a8e60 msgr2=0x5651cd4bfc10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T10:36:11.654 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:36:11.650+0000 7f1635563640 1 --2- 172.21.15.114:0/1762125541 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x5651cd4a8e60 0x5651cd4bfc10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:36:11.654 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:36:11.650+0000 7f1635563640 1 -- 172.21.15.114:0/1762125541 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x5651cd4a8540 msgr2=0x5651cd4b2560 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T10:36:11.654 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:36:11.650+0000 7f1635563640 1 --2- 172.21.15.114:0/1762125541 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x5651cd4a8540 0x5651cd4b2560 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:36:11.654 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:36:11.650+0000 7f1635563640 1 -- 172.21.15.114:0/1762125541 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5651cd4b1030 con 0x5651cd4b7c10 2024-04-23T10:36:11.654 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:36:11.650+0000 7f1635563640 1 --2- 172.21.15.114:0/1762125541 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x5651cd4b7c10 0x5651cd4b0af0 secure :-1 s=READY pgs=826 cs=0 l=1 rev1=1 crypto rx=0x7f1628009ca0 tx=0x7f16280095c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=dbd9c1bf7b1a71db server_cookie=0 in_seq=0 out_seq=0 2024-04-23T10:36:11.654 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:36:11.650+0000 7f160ffff640 10 client.0 ms_handle_connect on v2:172.21.15.114:3300/0 2024-04-23T10:36:11.655 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:36:11.650+0000 7f160ffff640 1 -- 172.21.15.114:0/1762125541 <== mon.0 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1628034040 con 0x5651cd4b7c10 2024-04-23T10:36:11.655 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:36:11.650+0000 7f160ffff640 1 -- 172.21.15.114:0/1762125541 <== mon.0 v2:172.21.15.114:3300/0 2 ==== osd_map(100..100 src has 1..100) ==== 8629+0+0 (secure 0 0 0) 0x7f1628004450 con 0x5651cd4b7c10 2024-04-23T10:36:11.655 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:36:11.650+0000 7f1639dbc740 1 -- 172.21.15.114:0/1762125541 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f162009a2a0 con 0x5651cd4b7c10 2024-04-23T10:36:11.655 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:36:11.650+0000 7f160ffff640 1 -- 172.21.15.114:0/1762125541 <== mon.0 v2:172.21.15.114:3300/0 3 ==== mon_get_version_reply(handle=1 version=36) ==== 24+0+0 (secure 0 0 0) 0x7f1628004650 con 0x5651cd4b7c10 2024-04-23T10:36:11.655 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:36:11.650+0000 7f1639dbc740 10 client.16734 fetch_fsmap learned FSMap version 36 2024-04-23T10:36:11.655 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:36:11.650+0000 7f1639dbc740 1 -- 172.21.15.114:0/1762125541 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({config=0+,fsmap=36,monmap=4+}) -- 0x5651cd52d400 con 0x5651cd4b7c10 2024-04-23T10:36:11.655 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:36:11.651+0000 7f160ffff640 1 -- 172.21.15.114:0/1762125541 <== mon.0 v2:172.21.15.114:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f162009a2a0 con 0x5651cd4b7c10 2024-04-23T10:36:11.655 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:36:11.651+0000 7f160ffff640 1 -- 172.21.15.114:0/1762125541 <== mon.0 v2:172.21.15.114:3300/0 5 ==== fsmap(e 36) ==== 8354+0+0 (secure 0 0 0) 0x7f162803a430 con 0x5651cd4b7c10 2024-04-23T10:36:11.655 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:36:11.651+0000 7f1639dbc740 10 client.16734 fetch_fsmap finished waiting for FSMap version 36 2024-04-23T10:36:11.655 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:36:11.651+0000 7f1639dbc740 10 client.16734 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-04-23T10:36:11.655 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:36:11.651+0000 7f1639dbc740 20 client.16734 populate_metadata read hostname 'smithi114' 2024-04-23T10:36:11.655 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:36:11.651+0000 7f1639dbc740 1 --2- 172.21.15.114:0/1762125541 >> [v2:172.21.15.119:6838/937297340,v1:172.21.15.119:6839/937297340] conn(0x5651cd52dbe0 0x5651cd530000 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T10:36:11.655 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:36:11.651+0000 7f1639dbc740 4 client.16734 mds_command: new command op to 24545 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-04-23T10:36:11.655 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:36:11.651+0000 7f1639dbc740 1 -- 172.21.15.114:0/1762125541 --> [v2:172.21.15.119:6838/937297340,v1:172.21.15.119:6839/937297340] -- command(tid 0: {"prefix": "flush journal"}) -- 0x5651cd415200 con 0x5651cd52dbe0 2024-04-23T10:36:11.655 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:36:11.651+0000 7f1634d62640 1 --2- 172.21.15.114:0/1762125541 >> [v2:172.21.15.119:6838/937297340,v1:172.21.15.119:6839/937297340] conn(0x5651cd52dbe0 0x5651cd530000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T10:36:11.656 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:36:11.651+0000 7f160ffff640 10 client.16734 ms_handle_connect on v2:172.21.15.119:6838/937297340 2024-04-23T10:36:11.656 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:36:11.651+0000 7f1634d62640 1 --2- 172.21.15.114:0/1762125541 >> [v2:172.21.15.119:6838/937297340,v1:172.21.15.119:6839/937297340] conn(0x5651cd52dbe0 0x5651cd530000 crc :-1 s=READY pgs=147 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=2ef1695d9ab400fa server_cookie=445f9c7ab0b737a4 in_seq=0 out_seq=0 2024-04-23T10:36:11.688 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:36:11.688+0000 7f160ffff640 1 -- 172.21.15.114:0/1762125541 <== mds.0 v2:172.21.15.119:6838/937297340 1 ==== command_reply(tid 0: 0 ) ==== 8+0+44 (crc 0 0 0) 0x5651cd415200 con 0x5651cd52dbe0 2024-04-23T10:36:11.688 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:36:11.688+0000 7f160ffff640 10 client.16734 handle_command_reply: tid=0 2024-04-23T10:36:11.688 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:36:11.688+0000 7f1639dbc740 1 client.16734 shutdown 2024-04-23T10:36:11.688 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:36:11.688+0000 7f1639dbc740 1 -- 172.21.15.114:0/1762125541 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x5651cd4b7c10 msgr2=0x5651cd4b0af0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T10:36:11.688 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:36:11.688+0000 7f1639dbc740 1 --2- 172.21.15.114:0/1762125541 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x5651cd4b7c10 0x5651cd4b0af0 secure :-1 s=READY pgs=826 cs=0 l=1 rev1=1 crypto rx=0x7f1628009ca0 tx=0x7f16280095c0 comp rx=0 tx=0).stop 2024-04-23T10:36:11.688 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:36:11.689+0000 7f1639dbc740 1 -- 172.21.15.114:0/1762125541 shutdown_connections 2024-04-23T10:36:11.689 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:36:11.689+0000 7f1639dbc740 1 --2- 172.21.15.114:0/1762125541 >> [v2:172.21.15.119:6838/937297340,v1:172.21.15.119:6839/937297340] conn(0x5651cd52dbe0 0x5651cd530000 crc :-1 s=READY pgs=147 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:36:11.689 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:36:11.689+0000 7f1639dbc740 1 --2- 172.21.15.114:0/1762125541 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x5651cd4b7c10 0x5651cd4b0af0 unknown :-1 s=CLOSED pgs=826 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:36:11.689 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:36:11.689+0000 7f1639dbc740 1 --2- 172.21.15.114:0/1762125541 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x5651cd4a8e60 0x5651cd4bfc10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:36:11.689 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:36:11.689+0000 7f160ffff640 0 client.16734 ms_handle_reset on v2:172.21.15.119:6838/937297340 2024-04-23T10:36:11.689 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:36:11.689+0000 7f1639dbc740 1 --2- 172.21.15.114:0/1762125541 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x5651cd4a8540 0x5651cd4b2560 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:36:11.689 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:36:11.689+0000 7f1639dbc740 1 -- 172.21.15.114:0/1762125541 >> 172.21.15.114:0/1762125541 conn(0x5651cd4a4b30 msgr2=0x5651cd4af670 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T10:36:11.689 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:36:11.689+0000 7f1639dbc740 1 -- 172.21.15.114:0/1762125541 shutdown_connections 2024-04-23T10:36:11.689 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:36:11.689+0000 7f1639dbc740 1 -- 172.21.15.114:0/1762125541 wait complete. 2024-04-23T10:36:11.689 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:36:11.689+0000 7f1639dbc740 20 client.16734 trim_cache size 0 max 16384 2024-04-23T10:36:11.689 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stdout:{ 2024-04-23T10:36:11.689 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stdout: "message": "", 2024-04-23T10:36:11.690 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stdout: "return_code": 0 2024-04-23T10:36:11.690 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stdout:} 2024-04-23T10:36:11.690 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:36:11.689+0000 7f1636fc6640 1 -- 172.21.15.114:0/1992239175 >> [v2:172.21.15.114:6800/1395693466,v1:172.21.15.114:6801/1395693466] conn(0x7f161c0a3dc0 msgr2=0x7f161c0a6280 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T10:36:11.691 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:36:11.689+0000 7f1636fc6640 1 --2- 172.21.15.114:0/1992239175 >> [v2:172.21.15.114:6800/1395693466,v1:172.21.15.114:6801/1395693466] conn(0x7f161c0a3dc0 0x7f161c0a6280 secure :-1 s=READY pgs=127 cs=0 l=1 rev1=1 crypto rx=0x7f1628005e30 tx=0x7f16280099a0 comp rx=0 tx=0).stop 2024-04-23T10:36:11.691 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:36:11.689+0000 7f1636fc6640 1 -- 172.21.15.114:0/1992239175 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f1630103b00 msgr2=0x7f1630198bf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T10:36:11.691 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:36:11.689+0000 7f1636fc6640 1 --2- 172.21.15.114:0/1992239175 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f1630103b00 0x7f1630198bf0 secure :-1 s=READY pgs=822 cs=0 l=1 rev1=1 crypto rx=0x7f162000d770 tx=0x7f162000dc30 comp rx=0 tx=0).stop 2024-04-23T10:36:11.691 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:36:11.689+0000 7f1636fc6640 1 -- 172.21.15.114:0/1992239175 shutdown_connections 2024-04-23T10:36:11.691 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:36:11.689+0000 7f1636fc6640 1 --2- 172.21.15.114:0/1992239175 >> [v2:172.21.15.114:6800/1395693466,v1:172.21.15.114:6801/1395693466] conn(0x7f161c0a3dc0 0x7f161c0a6280 unknown :-1 s=CLOSED pgs=127 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:36:11.691 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:36:11.689+0000 7f1636fc6640 1 --2- 172.21.15.114:0/1992239175 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f163010c5b0 0x7f163019cf20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:36:11.691 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:36:11.689+0000 7f1636fc6640 1 --2- 172.21.15.114:0/1992239175 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f1630103b00 0x7f1630198bf0 unknown :-1 s=CLOSED pgs=822 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:36:11.691 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:36:11.689+0000 7f1636fc6640 1 --2- 172.21.15.114:0/1992239175 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f16301031e0 0x7f16301986b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:36:11.691 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:36:11.689+0000 7f1636fc6640 1 -- 172.21.15.114:0/1992239175 >> 172.21.15.114:0/1992239175 conn(0x7f1630077f60 msgr2=0x7f16300fdf20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T10:36:11.691 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:36:11.689+0000 7f1636fc6640 1 -- 172.21.15.114:0/1992239175 shutdown_connections 2024-04-23T10:36:11.691 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:36:11.689+0000 7f1636fc6640 1 -- 172.21.15.114:0/1992239175 wait complete. 2024-04-23T10:36:11.827 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 23 10:36:11 smithi204 ceph-mon[32649]: pgmap v298: 97 pgs: 97 active+clean; 193 MiB data, 2.5 GiB used, 1.0 TiB / 1.0 TiB avail; 359 KiB/s rd, 1.1 MiB/s wr, 86 op/s 2024-04-23T10:36:11.936 INFO:tasks.workunit.client.0.smithi114.stdout: 1 642939 6.64 MB/sec execute 380 sec latency 2726.901 ms 2024-04-23T10:36:11.992 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 23 10:36:11 smithi119 ceph-mon[32608]: pgmap v298: 97 pgs: 97 active+clean; 193 MiB data, 2.5 GiB used, 1.0 TiB / 1.0 TiB avail; 359 KiB/s rd, 1.1 MiB/s wr, 86 op/s 2024-04-23T10:36:12.003 INFO:journalctl@ceph.mon.a.smithi114.stdout:Apr 23 10:36:11 smithi114 ceph-mon[29255]: pgmap v298: 97 pgs: 97 active+clean; 193 MiB data, 2.5 GiB used, 1.0 TiB / 1.0 TiB avail; 359 KiB/s rd, 1.1 MiB/s wr, 86 op/s 2024-04-23T10:36:12.936 INFO:tasks.workunit.client.0.smithi114.stdout: 1 643070 6.63 MB/sec execute 381 sec latency 3219.563 ms 2024-04-23T10:36:13.828 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 23 10:36:13 smithi204 ceph-mon[32649]: pgmap v299: 97 pgs: 97 active+clean; 193 MiB data, 2.5 GiB used, 1.0 TiB / 1.0 TiB avail; 381 KiB/s rd, 1.1 MiB/s wr, 94 op/s 2024-04-23T10:36:13.936 INFO:tasks.workunit.client.0.smithi114.stdout: 1 643956 6.62 MB/sec execute 382 sec latency 35.874 ms 2024-04-23T10:36:13.991 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 23 10:36:13 smithi119 ceph-mon[32608]: pgmap v299: 97 pgs: 97 active+clean; 193 MiB data, 2.5 GiB used, 1.0 TiB / 1.0 TiB avail; 381 KiB/s rd, 1.1 MiB/s wr, 94 op/s 2024-04-23T10:36:14.003 INFO:journalctl@ceph.mon.a.smithi114.stdout:Apr 23 10:36:13 smithi114 ceph-mon[29255]: pgmap v299: 97 pgs: 97 active+clean; 193 MiB data, 2.5 GiB used, 1.0 TiB / 1.0 TiB avail; 381 KiB/s rd, 1.1 MiB/s wr, 94 op/s 2024-04-23T10:36:14.827 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 23 10:36:14 smithi204 ceph-mon[32649]: osdmap e101: 12 total, 12 up, 12 in 2024-04-23T10:36:14.936 INFO:tasks.workunit.client.0.smithi114.stdout: 1 646298 6.64 MB/sec execute 383 sec latency 29.945 ms 2024-04-23T10:36:14.991 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 23 10:36:14 smithi119 ceph-mon[32608]: osdmap e101: 12 total, 12 up, 12 in 2024-04-23T10:36:15.004 INFO:journalctl@ceph.mon.a.smithi114.stdout:Apr 23 10:36:14 smithi114 ceph-mon[29255]: osdmap e101: 12 total, 12 up, 12 in 2024-04-23T10:36:15.827 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 23 10:36:15 smithi204 ceph-mon[32649]: pgmap v301: 97 pgs: 97 active+clean; 193 MiB data, 2.5 GiB used, 1.0 TiB / 1.0 TiB avail; 86 KiB/s rd, 417 KiB/s wr, 47 op/s 2024-04-23T10:36:15.936 INFO:tasks.workunit.client.0.smithi114.stdout: 1 646417 6.62 MB/sec execute 384 sec latency 936.021 ms 2024-04-23T10:36:15.992 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 23 10:36:15 smithi119 ceph-mon[32608]: pgmap v301: 97 pgs: 97 active+clean; 193 MiB data, 2.5 GiB used, 1.0 TiB / 1.0 TiB avail; 86 KiB/s rd, 417 KiB/s wr, 47 op/s 2024-04-23T10:36:16.004 INFO:journalctl@ceph.mon.a.smithi114.stdout:Apr 23 10:36:15 smithi114 ceph-mon[29255]: pgmap v301: 97 pgs: 97 active+clean; 193 MiB data, 2.5 GiB used, 1.0 TiB / 1.0 TiB avail; 86 KiB/s rd, 417 KiB/s wr, 47 op/s 2024-04-23T10:36:16.936 INFO:tasks.workunit.client.0.smithi114.stdout: 1 646417 6.61 MB/sec execute 385 sec latency 1936.077 ms 2024-04-23T10:36:17.828 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 23 10:36:17 smithi204 ceph-mon[32649]: pgmap v302: 97 pgs: 97 active+clean; 195 MiB data, 2.5 GiB used, 1.0 TiB / 1.0 TiB avail; 361 KiB/s rd, 854 KiB/s wr, 100 op/s 2024-04-23T10:36:17.936 INFO:tasks.workunit.client.0.smithi114.stdout: 1 646417 6.59 MB/sec execute 386 sec latency 2936.165 ms 2024-04-23T10:36:17.991 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 23 10:36:17 smithi119 ceph-mon[32608]: pgmap v302: 97 pgs: 97 active+clean; 195 MiB data, 2.5 GiB used, 1.0 TiB / 1.0 TiB avail; 361 KiB/s rd, 854 KiB/s wr, 100 op/s 2024-04-23T10:36:18.004 INFO:journalctl@ceph.mon.a.smithi114.stdout:Apr 23 10:36:17 smithi114 ceph-mon[29255]: pgmap v302: 97 pgs: 97 active+clean; 195 MiB data, 2.5 GiB used, 1.0 TiB / 1.0 TiB avail; 361 KiB/s rd, 854 KiB/s wr, 100 op/s 2024-04-23T10:36:18.828 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 23 10:36:18 smithi204 ceph-mon[32649]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=e=up:active} 3 up:standby-replay 6 up:standby 2024-04-23T10:36:18.828 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 23 10:36:18 smithi204 ceph-mon[32649]: osdmap e102: 12 total, 12 up, 12 in 2024-04-23T10:36:18.936 INFO:tasks.workunit.client.0.smithi114.stdout: 1 646417 6.57 MB/sec execute 387 sec latency 3936.296 ms 2024-04-23T10:36:18.992 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 23 10:36:18 smithi119 ceph-mon[32608]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=e=up:active} 3 up:standby-replay 6 up:standby 2024-04-23T10:36:18.992 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 23 10:36:18 smithi119 ceph-mon[32608]: osdmap e102: 12 total, 12 up, 12 in 2024-04-23T10:36:19.004 INFO:journalctl@ceph.mon.a.smithi114.stdout:Apr 23 10:36:18 smithi114 ceph-mon[29255]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=e=up:active} 3 up:standby-replay 6 up:standby 2024-04-23T10:36:19.004 INFO:journalctl@ceph.mon.a.smithi114.stdout:Apr 23 10:36:18 smithi114 ceph-mon[29255]: osdmap e102: 12 total, 12 up, 12 in 2024-04-23T10:36:19.828 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 23 10:36:19 smithi204 ceph-mon[32649]: pgmap v303: 97 pgs: 97 active+clean; 195 MiB data, 2.5 GiB used, 1.0 TiB / 1.0 TiB avail; 361 KiB/s rd, 854 KiB/s wr, 100 op/s 2024-04-23T10:36:19.905 INFO:journalctl@ceph.mon.a.smithi114.stdout:Apr 23 10:36:19 smithi114 ceph-mon[29255]: pgmap v303: 97 pgs: 97 active+clean; 195 MiB data, 2.5 GiB used, 1.0 TiB / 1.0 TiB avail; 361 KiB/s rd, 854 KiB/s wr, 100 op/s 2024-04-23T10:36:19.936 INFO:tasks.workunit.client.0.smithi114.stdout: 1 646484 6.55 MB/sec execute 388 sec latency 4184.293 ms 2024-04-23T10:36:19.992 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 23 10:36:19 smithi119 ceph-mon[32608]: pgmap v303: 97 pgs: 97 active+clean; 195 MiB data, 2.5 GiB used, 1.0 TiB / 1.0 TiB avail; 361 KiB/s rd, 854 KiB/s wr, 100 op/s 2024-04-23T10:36:20.937 INFO:tasks.workunit.client.0.smithi114.stdout: 1 646484 6.54 MB/sec execute 389 sec latency 1648.716 ms 2024-04-23T10:36:21.828 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 23 10:36:21 smithi204 ceph-mon[32649]: pgmap v305: 97 pgs: 97 active+clean; 195 MiB data, 2.5 GiB used, 1.0 TiB / 1.0 TiB avail; 419 KiB/s rd, 996 KiB/s wr, 112 op/s 2024-04-23T10:36:21.937 INFO:tasks.workunit.client.0.smithi114.stdout: 1 646484 6.52 MB/sec execute 390 sec latency 2648.834 ms 2024-04-23T10:36:21.992 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 23 10:36:21 smithi119 ceph-mon[32608]: pgmap v305: 97 pgs: 97 active+clean; 195 MiB data, 2.5 GiB used, 1.0 TiB / 1.0 TiB avail; 419 KiB/s rd, 996 KiB/s wr, 112 op/s 2024-04-23T10:36:22.003 INFO:journalctl@ceph.mon.a.smithi114.stdout:Apr 23 10:36:21 smithi114 ceph-mon[29255]: pgmap v305: 97 pgs: 97 active+clean; 195 MiB data, 2.5 GiB used, 1.0 TiB / 1.0 TiB avail; 419 KiB/s rd, 996 KiB/s wr, 112 op/s 2024-04-23T10:36:22.937 INFO:tasks.workunit.client.0.smithi114.stdout: 1 646484 6.50 MB/sec execute 391 sec latency 3648.928 ms 2024-04-23T10:36:23.827 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 23 10:36:23 smithi204 ceph-mon[32649]: pgmap v306: 97 pgs: 97 active+clean; 195 MiB data, 2.5 GiB used, 1.0 TiB / 1.0 TiB avail; 421 KiB/s rd, 947 KiB/s wr, 114 op/s 2024-04-23T10:36:23.937 INFO:tasks.workunit.client.0.smithi114.stdout: 1 646484 6.49 MB/sec execute 392 sec latency 4648.965 ms 2024-04-23T10:36:23.992 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 23 10:36:23 smithi119 ceph-mon[32608]: pgmap v306: 97 pgs: 97 active+clean; 195 MiB data, 2.5 GiB used, 1.0 TiB / 1.0 TiB avail; 421 KiB/s rd, 947 KiB/s wr, 114 op/s 2024-04-23T10:36:24.004 INFO:journalctl@ceph.mon.a.smithi114.stdout:Apr 23 10:36:23 smithi114 ceph-mon[29255]: pgmap v306: 97 pgs: 97 active+clean; 195 MiB data, 2.5 GiB used, 1.0 TiB / 1.0 TiB avail; 421 KiB/s rd, 947 KiB/s wr, 114 op/s 2024-04-23T10:36:24.780 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:36:24.781+0000 7f1320f4a640 1 -- 172.21.15.114:0/2115844076 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f131c106190 msgr2=0x7f131c10dd80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T10:36:24.781 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:36:24.781+0000 7f1320f4a640 1 --2- 172.21.15.114:0/2115844076 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f131c106190 0x7f131c10dd80 secure :-1 s=READY pgs=827 cs=0 l=1 rev1=1 crypto rx=0x7f1314009ca0 tx=0x7f131402a8c0 comp rx=0 tx=0).stop 2024-04-23T10:36:24.781 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:36:24.781+0000 7f1320f4a640 1 -- 172.21.15.114:0/2115844076 shutdown_connections 2024-04-23T10:36:24.781 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:36:24.781+0000 7f1320f4a640 1 --2- 172.21.15.114:0/2115844076 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f131c106190 0x7f131c10dd80 unknown :-1 s=CLOSED pgs=827 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:36:24.781 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:36:24.781+0000 7f1320f4a640 1 --2- 172.21.15.114:0/2115844076 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f131c101cc0 0x7f131c105ab0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:36:24.781 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:36:24.781+0000 7f1320f4a640 1 --2- 172.21.15.114:0/2115844076 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f131c101310 0x7f131c1016f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:36:24.781 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:36:24.781+0000 7f1320f4a640 1 -- 172.21.15.114:0/2115844076 >> 172.21.15.114:0/2115844076 conn(0x7f131c077cc0 msgr2=0x7f131c0780d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T10:36:24.781 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:36:24.781+0000 7f1320f4a640 1 -- 172.21.15.114:0/2115844076 shutdown_connections 2024-04-23T10:36:24.781 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:36:24.781+0000 7f1320f4a640 1 -- 172.21.15.114:0/2115844076 wait complete. 2024-04-23T10:36:24.781 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:36:24.782+0000 7f1320f4a640 1 Processor -- start 2024-04-23T10:36:24.781 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:36:24.782+0000 7f1320f4a640 1 -- start start 2024-04-23T10:36:24.783 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:36:24.782+0000 7f1320f4a640 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f131c101310 0x7f131c10e710 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T10:36:24.783 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:36:24.782+0000 7f1320f4a640 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f131c106190 0x7f131c10ec50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T10:36:24.783 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:36:24.782+0000 7f1320f4a640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f131c10f190 0x7f131c1a6840 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T10:36:24.783 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:36:24.782+0000 7f1320f4a640 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x7f131c112230 con 0x7f131c10f190 2024-04-23T10:36:24.783 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:36:24.782+0000 7f1320f4a640 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 -- 0x7f131c1120b0 con 0x7f131c106190 2024-04-23T10:36:24.783 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:36:24.782+0000 7f1320f4a640 1 -- --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_getmap magic: 0 -- 0x7f131c1123b0 con 0x7f131c101310 2024-04-23T10:36:24.783 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:36:24.782+0000 7f131ad9d640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f131c10f190 0x7f131c1a6840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T10:36:24.783 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:36:24.782+0000 7f1313fff640 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f131c106190 0x7f131c10ec50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T10:36:24.783 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:36:24.782+0000 7f131a59c640 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f131c101310 0x7f131c10e710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T10:36:24.783 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:36:24.782+0000 7f1313fff640 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f131c106190 0x7f131c10ec50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.119:3300/0 says I am v2:172.21.15.114:56766/0 (socket says 172.21.15.114:56766) 2024-04-23T10:36:24.783 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:36:24.782+0000 7f131ad9d640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f131c10f190 0x7f131c1a6840 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.114:39494/0 (socket says 172.21.15.114:39494) 2024-04-23T10:36:24.783 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:36:24.782+0000 7f1313fff640 1 -- 172.21.15.114:0/4228009606 learned_addr learned my addr 172.21.15.114:0/4228009606 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-04-23T10:36:24.784 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:36:24.783+0000 7f1313fff640 1 -- 172.21.15.114:0/4228009606 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f131c101310 msgr2=0x7f131c10e710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T10:36:24.784 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:36:24.783+0000 7f1313fff640 1 --2- 172.21.15.114:0/4228009606 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f131c101310 0x7f131c10e710 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:36:24.784 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:36:24.783+0000 7f1313fff640 1 -- 172.21.15.114:0/4228009606 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f131c10f190 msgr2=0x7f131c1a6840 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T10:36:24.784 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:36:24.783+0000 7f1313fff640 1 --2- 172.21.15.114:0/4228009606 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f131c10f190 0x7f131c1a6840 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:36:24.784 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:36:24.783+0000 7f1313fff640 1 -- 172.21.15.114:0/4228009606 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f131c1a6f40 con 0x7f131c106190 2024-04-23T10:36:24.784 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:36:24.783+0000 7f131ad9d640 1 --2- 172.21.15.114:0/4228009606 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f131c10f190 0x7f131c1a6840 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-23T10:36:24.784 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:36:24.783+0000 7f1313fff640 1 --2- 172.21.15.114:0/4228009606 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f131c106190 0x7f131c10ec50 secure :-1 s=READY pgs=92 cs=0 l=1 rev1=1 crypto rx=0x7f130400c970 tx=0x7f130400ce30 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T10:36:24.784 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:36:24.783+0000 7f13137fe640 1 -- 172.21.15.114:0/4228009606 <== mon.2 v2:172.21.15.119:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1304007bb0 con 0x7f131c106190 2024-04-23T10:36:24.784 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:36:24.783+0000 7f1320f4a640 1 -- 172.21.15.114:0/4228009606 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f131c1a7230 con 0x7f131c106190 2024-04-23T10:36:24.784 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:36:24.783+0000 7f13137fe640 1 -- 172.21.15.114:0/4228009606 <== mon.2 v2:172.21.15.119:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f1304007d50 con 0x7f131c106190 2024-04-23T10:36:24.784 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:36:24.783+0000 7f13137fe640 1 -- 172.21.15.114:0/4228009606 <== mon.2 v2:172.21.15.119:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1304018730 con 0x7f131c106190 2024-04-23T10:36:24.784 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:36:24.783+0000 7f1320f4a640 1 -- 172.21.15.114:0/4228009606 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f131c1a7780 con 0x7f131c106190 2024-04-23T10:36:24.789 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:36:24.785+0000 7f13137fe640 1 -- 172.21.15.114:0/4228009606 <== mon.2 v2:172.21.15.119:3300/0 4 ==== mgrmap(e 26) ==== 138778+0+0 (secure 0 0 0) 0x7f130400f990 con 0x7f131c106190 2024-04-23T10:36:24.789 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:36:24.786+0000 7f1323c5b740 1 Processor -- start 2024-04-23T10:36:24.789 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:36:24.786+0000 7f1323c5b740 1 -- start start 2024-04-23T10:36:24.789 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:36:24.786+0000 7f13137fe640 1 --2- 172.21.15.114:0/4228009606 >> [v2:172.21.15.114:6800/1395693466,v1:172.21.15.114:6801/1395693466] conn(0x7f13080a3df0 0x7f13080a62b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T10:36:24.789 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:36:24.786+0000 7f1323c5b740 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x557cec129620 0x557cec129a00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T10:36:24.790 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:36:24.786+0000 7f1323c5b740 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x557cec129f40 0x557cec1342c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T10:36:24.790 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:36:24.786+0000 7f1323c5b740 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x557cec134800 0x557cec136bf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T10:36:24.790 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:36:24.786+0000 7f1323c5b740 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x557cec08e970 con 0x557cec129620 2024-04-23T10:36:24.790 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:36:24.786+0000 7f1323c5b740 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 -- 0x557cec02b4a0 con 0x557cec134800 2024-04-23T10:36:24.790 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:36:24.786+0000 7f1323c5b740 1 -- --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_getmap magic: 0 -- 0x557cec08a930 con 0x557cec129f40 2024-04-23T10:36:24.790 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:36:24.786+0000 7f131a59c640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x557cec129620 0x557cec129a00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T10:36:24.790 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:36:24.786+0000 7f131ad9d640 1 --2- 172.21.15.114:0/4228009606 >> [v2:172.21.15.114:6800/1395693466,v1:172.21.15.114:6801/1395693466] conn(0x7f13080a3df0 0x7f13080a62b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T10:36:24.790 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:36:24.786+0000 7f131a59c640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x557cec129620 0x557cec129a00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.114:39510/0 (socket says 172.21.15.114:39510) 2024-04-23T10:36:24.790 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:36:24.786+0000 7f131a59c640 1 -- 172.21.15.114:0/3204216676 learned_addr learned my addr 172.21.15.114:0/3204216676 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-04-23T10:36:24.790 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:36:24.786+0000 7f131ad9d640 1 --2- 172.21.15.114:0/3204216676 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x557cec134800 0x557cec136bf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T10:36:24.790 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:36:24.786+0000 7f1313fff640 1 --2- 172.21.15.114:0/3204216676 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x557cec129f40 0x557cec1342c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T10:36:24.790 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:36:24.786+0000 7f13137fe640 1 -- 172.21.15.114:0/4228009606 <== mon.2 v2:172.21.15.119:3300/0 5 ==== osd_map(102..102 src has 1..102) ==== 8629+0+0 (secure 0 0 0) 0x7f13040d0750 con 0x7f131c106190 2024-04-23T10:36:24.790 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:36:24.787+0000 7f131a59c640 1 -- 172.21.15.114:0/3204216676 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x557cec129f40 msgr2=0x557cec1342c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T10:36:24.790 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:36:24.787+0000 7f131a59c640 1 --2- 172.21.15.114:0/3204216676 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x557cec129f40 0x557cec1342c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:36:24.790 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:36:24.787+0000 7f131a59c640 1 -- 172.21.15.114:0/3204216676 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x557cec134800 msgr2=0x557cec136bf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T10:36:24.791 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:36:24.787+0000 7f131a59c640 1 --2- 172.21.15.114:0/3204216676 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x557cec134800 0x557cec136bf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:36:24.791 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:36:24.787+0000 7f131a59c640 1 -- 172.21.15.114:0/3204216676 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x557cebda1210 con 0x557cec129620 2024-04-23T10:36:24.791 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:36:24.787+0000 7f1313fff640 1 --2- 172.21.15.114:0/3204216676 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x557cec129f40 0x557cec1342c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-23T10:36:24.791 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:36:24.787+0000 7f131ad9d640 1 --2- 172.21.15.114:0/4228009606 >> [v2:172.21.15.114:6800/1395693466,v1:172.21.15.114:6801/1395693466] conn(0x7f13080a3df0 0x7f13080a62b0 secure :-1 s=READY pgs=128 cs=0 l=1 rev1=1 crypto rx=0x7f1314006010 tx=0x7f131402f4a0 comp rx=0 tx=0).ready entity=mgr.15291 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T10:36:24.791 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:36:24.787+0000 7f131a59c640 1 --2- 172.21.15.114:0/3204216676 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x557cec129620 0x557cec129a00 secure :-1 s=READY pgs=828 cs=0 l=1 rev1=1 crypto rx=0x7f130c009cf0 tx=0x7f130c011810 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ea038b45f54ed669 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T10:36:24.791 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:36:24.787+0000 7f1310ff9640 1 -- 172.21.15.114:0/3204216676 <== mon.0 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f130c004230 con 0x557cec129620 2024-04-23T10:36:24.791 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:36:24.787+0000 7f1310ff9640 1 -- 172.21.15.114:0/3204216676 <== mon.0 v2:172.21.15.114:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f130c0043d0 con 0x557cec129620 2024-04-23T10:36:24.791 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:36:24.787+0000 7f1310ff9640 1 -- 172.21.15.114:0/3204216676 <== mon.0 v2:172.21.15.114:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f130c0197c0 con 0x557cec129620 2024-04-23T10:36:24.791 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:36:24.788+0000 7f1323c5b740 1 -- 172.21.15.114:0/3204216676 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x557cec129620 msgr2=0x557cec129a00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T10:36:24.791 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:36:24.788+0000 7f1323c5b740 1 --2- 172.21.15.114:0/3204216676 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x557cec129620 0x557cec129a00 secure :-1 s=READY pgs=828 cs=0 l=1 rev1=1 crypto rx=0x7f130c009cf0 tx=0x7f130c011810 comp rx=0 tx=0).stop 2024-04-23T10:36:24.791 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:36:24.788+0000 7f1323c5b740 1 -- 172.21.15.114:0/3204216676 shutdown_connections 2024-04-23T10:36:24.791 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:36:24.788+0000 7f1323c5b740 1 --2- 172.21.15.114:0/3204216676 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x557cec134800 0x557cec136bf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:36:24.791 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:36:24.788+0000 7f1323c5b740 1 --2- 172.21.15.114:0/3204216676 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x557cec129f40 0x557cec1342c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:36:24.791 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:36:24.788+0000 7f1323c5b740 1 --2- 172.21.15.114:0/3204216676 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x557cec129620 0x557cec129a00 unknown :-1 s=CLOSED pgs=828 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:36:24.791 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:36:24.788+0000 7f1323c5b740 1 -- 172.21.15.114:0/3204216676 >> 172.21.15.114:0/3204216676 conn(0x557cec125b30 msgr2=0x557cec125f10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T10:36:24.792 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:36:24.788+0000 7f1323c5b740 1 -- 172.21.15.114:0/3204216676 shutdown_connections 2024-04-23T10:36:24.792 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:36:24.788+0000 7f1323c5b740 1 -- 172.21.15.114:0/3204216676 wait complete. 2024-04-23T10:36:24.792 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:36:24.788+0000 7f1323c5b740 1 -- start start 2024-04-23T10:36:24.792 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:36:24.788+0000 7f1323c5b740 1 Processor -- start 2024-04-23T10:36:24.792 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:36:24.788+0000 7f1323c5b740 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x557cec129620 0x557cec134110 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T10:36:24.792 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:36:24.788+0000 7f1323c5b740 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x557cec129f40 0x557cec131400 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T10:36:24.792 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:36:24.788+0000 7f1323c5b740 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x557cec134800 0x557cec131940 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T10:36:24.792 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:36:24.788+0000 7f1323c5b740 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x557cec137430 con 0x557cec129f40 2024-04-23T10:36:24.792 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:36:24.788+0000 7f1323c5b740 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 -- 0x557cec0290b0 con 0x557cec134800 2024-04-23T10:36:24.792 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:36:24.788+0000 7f1323c5b740 1 -- --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_getmap magic: 0 -- 0x557cec1375b0 con 0x557cec129620 2024-04-23T10:36:24.792 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:36:24.788+0000 7f1323c5b740 10 client.0 _ll_register_callbacks cb 0x557cec016d30 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-23T10:36:24.792 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:36:24.789+0000 7f1313fff640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x557cec129f40 0x557cec131400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T10:36:24.792 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:36:24.789+0000 7f131a59c640 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x557cec129620 0x557cec134110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T10:36:24.792 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:36:24.789+0000 7f1313fff640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x557cec129f40 0x557cec131400 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.114:39520/0 (socket says 172.21.15.114:39520) 2024-04-23T10:36:24.792 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:36:24.789+0000 7f1313fff640 1 -- 172.21.15.114:0/1207833143 learned_addr learned my addr 172.21.15.114:0/1207833143 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-04-23T10:36:24.793 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:36:24.789+0000 7f131ad9d640 1 --2- 172.21.15.114:0/1207833143 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x557cec134800 0x557cec131940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T10:36:24.793 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:36:24.789+0000 7f1313fff640 1 -- 172.21.15.114:0/1207833143 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x557cec129620 msgr2=0x557cec134110 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T10:36:24.793 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:36:24.789+0000 7f1313fff640 1 --2- 172.21.15.114:0/1207833143 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x557cec129620 0x557cec134110 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:36:24.793 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:36:24.789+0000 7f1313fff640 1 -- 172.21.15.114:0/1207833143 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x557cec134800 msgr2=0x557cec131940 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T10:36:24.793 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:36:24.789+0000 7f1313fff640 1 --2- 172.21.15.114:0/1207833143 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x557cec134800 0x557cec131940 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:36:24.793 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:36:24.789+0000 7f1313fff640 1 -- 172.21.15.114:0/1207833143 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({osdmap=0}) -- 0x557cebe6d620 con 0x557cec129f40 2024-04-23T10:36:24.793 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:36:24.789+0000 7f131a59c640 1 --2- 172.21.15.114:0/1207833143 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x557cec129620 0x557cec134110 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-23T10:36:24.793 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:36:24.789+0000 7f1313fff640 1 --2- 172.21.15.114:0/1207833143 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x557cec129f40 0x557cec131400 secure :-1 s=READY pgs=829 cs=0 l=1 rev1=1 crypto rx=0x7f1304099050 tx=0x7f130409c1c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b97e24cdab4ed902 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T10:36:24.793 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:36:24.789+0000 7f1301ffb640 10 client.0 ms_handle_connect on v2:172.21.15.114:3300/0 2024-04-23T10:36:24.793 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:36:24.789+0000 7f1301ffb640 1 -- 172.21.15.114:0/1207833143 <== mon.0 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f130409cd80 con 0x557cec129f40 2024-04-23T10:36:24.793 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:36:24.789+0000 7f1301ffb640 1 -- 172.21.15.114:0/1207833143 <== mon.0 v2:172.21.15.114:3300/0 2 ==== osd_map(102..102 src has 1..102) ==== 8629+0+0 (secure 0 0 0) 0x7f13040c75f0 con 0x557cec129f40 2024-04-23T10:36:24.793 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:36:24.789+0000 7f1323c5b740 1 -- 172.21.15.114:0/1207833143 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f130c0043d0 con 0x557cec129f40 2024-04-23T10:36:24.793 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:36:24.789+0000 7f1301ffb640 1 -- 172.21.15.114:0/1207833143 <== mon.0 v2:172.21.15.114:3300/0 3 ==== mon_get_version_reply(handle=1 version=37) ==== 24+0+0 (secure 0 0 0) 0x7f130409c680 con 0x557cec129f40 2024-04-23T10:36:24.793 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:36:24.789+0000 7f1323c5b740 10 client.16746 fetch_fsmap learned FSMap version 37 2024-04-23T10:36:24.793 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:36:24.789+0000 7f1323c5b740 1 -- 172.21.15.114:0/1207833143 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({config=0+,fsmap=37,monmap=4+}) -- 0x557cec143370 con 0x557cec129f40 2024-04-23T10:36:24.793 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:36:24.789+0000 7f1301ffb640 1 -- 172.21.15.114:0/1207833143 <== mon.0 v2:172.21.15.114:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f130c0043d0 con 0x557cec129f40 2024-04-23T10:36:24.794 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:36:24.790+0000 7f1301ffb640 1 -- 172.21.15.114:0/1207833143 <== mon.0 v2:172.21.15.114:3300/0 5 ==== fsmap(e 37) ==== 8358+0+0 (secure 0 0 0) 0x7f1304020030 con 0x557cec129f40 2024-04-23T10:36:24.794 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:36:24.790+0000 7f1323c5b740 10 client.16746 fetch_fsmap finished waiting for FSMap version 37 2024-04-23T10:36:24.794 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:36:24.790+0000 7f1323c5b740 10 client.16746 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-04-23T10:36:24.794 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:36:24.790+0000 7f1323c5b740 20 client.16746 populate_metadata read hostname 'smithi114' 2024-04-23T10:36:24.794 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:36:24.790+0000 7f1323c5b740 1 --2- 172.21.15.114:0/1207833143 >> [v2:172.21.15.119:6838/937297340,v1:172.21.15.119:6839/937297340] conn(0x557cec144830 0x557cec146c40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T10:36:24.794 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:36:24.790+0000 7f1323c5b740 4 client.16746 mds_command: new command op to 24545 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-23T10:36:24.794 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:36:24.790+0000 7f1323c5b740 1 -- 172.21.15.114:0/1207833143 --> [v2:172.21.15.119:6838/937297340,v1:172.21.15.119:6839/937297340] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x557cec09c390 con 0x557cec144830 2024-04-23T10:36:24.794 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:36:24.790+0000 7f131a59c640 1 --2- 172.21.15.114:0/1207833143 >> [v2:172.21.15.119:6838/937297340,v1:172.21.15.119:6839/937297340] conn(0x557cec144830 0x557cec146c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T10:36:24.794 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:36:24.790+0000 7f1301ffb640 10 client.16746 ms_handle_connect on v2:172.21.15.119:6838/937297340 2024-04-23T10:36:24.794 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:36:24.790+0000 7f131a59c640 1 --2- 172.21.15.114:0/1207833143 >> [v2:172.21.15.119:6838/937297340,v1:172.21.15.119:6839/937297340] conn(0x557cec144830 0x557cec146c40 crc :-1 s=READY pgs=148 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=50656c98a3e4c6a5 server_cookie=f2d89de50b6d9e6e in_seq=0 out_seq=0 2024-04-23T10:36:24.795 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:36:24.792+0000 7f1301ffb640 1 -- 172.21.15.114:0/1207833143 <== mds.0 v2:172.21.15.119:6838/937297340 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x557cec09c390 con 0x557cec144830 2024-04-23T10:36:24.795 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:36:24.792+0000 7f1301ffb640 10 client.16746 handle_command_reply: tid=0 2024-04-23T10:36:24.796 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:36:24.792+0000 7f1323c5b740 1 client.16746 shutdown 2024-04-23T10:36:24.796 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:36:24.792+0000 7f1323c5b740 1 -- 172.21.15.114:0/1207833143 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x557cec129f40 msgr2=0x557cec131400 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T10:36:24.796 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:36:24.792+0000 7f1323c5b740 1 --2- 172.21.15.114:0/1207833143 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x557cec129f40 0x557cec131400 secure :-1 s=READY pgs=829 cs=0 l=1 rev1=1 crypto rx=0x7f1304099050 tx=0x7f130409c1c0 comp rx=0 tx=0).stop 2024-04-23T10:36:24.796 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:36:24.792+0000 7f1323c5b740 1 -- 172.21.15.114:0/1207833143 shutdown_connections 2024-04-23T10:36:24.796 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:36:24.792+0000 7f1323c5b740 1 --2- 172.21.15.114:0/1207833143 >> [v2:172.21.15.119:6838/937297340,v1:172.21.15.119:6839/937297340] conn(0x557cec144830 0x557cec146c40 crc :-1 s=READY pgs=148 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:36:24.796 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:36:24.792+0000 7f1323c5b740 1 --2- 172.21.15.114:0/1207833143 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x557cec134800 0x557cec131940 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:36:24.796 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:36:24.792+0000 7f1301ffb640 0 client.16746 ms_handle_reset on v2:172.21.15.119:6838/937297340 2024-04-23T10:36:24.796 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:36:24.792+0000 7f1323c5b740 1 --2- 172.21.15.114:0/1207833143 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x557cec129f40 0x557cec131400 unknown :-1 s=CLOSED pgs=829 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:36:24.796 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:36:24.792+0000 7f1323c5b740 1 --2- 172.21.15.114:0/1207833143 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x557cec129620 0x557cec134110 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:36:24.796 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:36:24.792+0000 7f1323c5b740 1 -- 172.21.15.114:0/1207833143 >> 172.21.15.114:0/1207833143 conn(0x557cec125b30 msgr2=0x557cec12b860 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T10:36:24.796 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:36:24.792+0000 7f1323c5b740 1 -- 172.21.15.114:0/1207833143 shutdown_connections 2024-04-23T10:36:24.796 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:36:24.792+0000 7f1323c5b740 1 -- 172.21.15.114:0/1207833143 wait complete. 2024-04-23T10:36:24.796 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:36:24.792+0000 7f1323c5b740 20 client.16746 trim_cache size 0 max 16384 2024-04-23T10:36:24.810 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:36:24.807+0000 7f1323c5b740 1 Processor -- start 2024-04-23T10:36:24.810 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:36:24.807+0000 7f1323c5b740 1 -- start start 2024-04-23T10:36:24.810 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:36:24.807+0000 7f1323c5b740 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x557cec129620 0x557cec1402f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T10:36:24.810 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:36:24.807+0000 7f1323c5b740 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x557cec129f40 0x557cec124b30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T10:36:24.810 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:36:24.807+0000 7f1323c5b740 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x557cec134800 0x557cec14c550 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T10:36:24.810 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:36:24.807+0000 7f1323c5b740 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x7f131c1b2870 con 0x557cec134800 2024-04-23T10:36:24.810 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:36:24.807+0000 7f1323c5b740 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 -- 0x7f131c1b0080 con 0x557cec129f40 2024-04-23T10:36:24.810 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:36:24.807+0000 7f1323c5b740 1 -- --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_getmap magic: 0 -- 0x7f131c1adef0 con 0x557cec129620 2024-04-23T10:36:24.810 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:36:24.807+0000 7f131ad9d640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x557cec134800 0x557cec14c550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T10:36:24.810 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:36:24.807+0000 7f131ad9d640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x557cec134800 0x557cec14c550 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.114:39536/0 (socket says 172.21.15.114:39536) 2024-04-23T10:36:24.810 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:36:24.807+0000 7f131ad9d640 1 -- 172.21.15.114:0/1424090904 learned_addr learned my addr 172.21.15.114:0/1424090904 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-04-23T10:36:24.810 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:36:24.807+0000 7f1313fff640 1 --2- 172.21.15.114:0/1424090904 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x557cec129f40 0x557cec124b30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T10:36:24.811 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:36:24.807+0000 7f131a59c640 1 --2- 172.21.15.114:0/1424090904 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x557cec129620 0x557cec1402f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T10:36:24.811 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:36:24.808+0000 7f131ad9d640 1 -- 172.21.15.114:0/1424090904 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x557cec129620 msgr2=0x557cec1402f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T10:36:24.811 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:36:24.808+0000 7f131ad9d640 1 --2- 172.21.15.114:0/1424090904 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x557cec129620 0x557cec1402f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:36:24.811 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:36:24.808+0000 7f131ad9d640 1 -- 172.21.15.114:0/1424090904 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x557cec129f40 msgr2=0x557cec124b30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T10:36:24.811 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:36:24.808+0000 7f131ad9d640 1 --2- 172.21.15.114:0/1424090904 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x557cec129f40 0x557cec124b30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:36:24.811 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:36:24.808+0000 7f131ad9d640 1 -- 172.21.15.114:0/1424090904 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x557cec1472c0 con 0x557cec134800 2024-04-23T10:36:24.811 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:36:24.808+0000 7f131ad9d640 1 --2- 172.21.15.114:0/1424090904 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x557cec134800 0x557cec14c550 secure :-1 s=READY pgs=830 cs=0 l=1 rev1=1 crypto rx=0x7f1314035000 tx=0x7f131404c000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d0a235ef50808c3a server_cookie=0 in_seq=0 out_seq=0 2024-04-23T10:36:24.811 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:36:24.808+0000 7f1303fff640 1 -- 172.21.15.114:0/1424090904 <== mon.0 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1314004040 con 0x557cec134800 2024-04-23T10:36:24.811 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:36:24.808+0000 7f1303fff640 1 -- 172.21.15.114:0/1424090904 <== mon.0 v2:172.21.15.114:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f1314004690 con 0x557cec134800 2024-04-23T10:36:24.811 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:36:24.808+0000 7f1303fff640 1 -- 172.21.15.114:0/1424090904 <== mon.0 v2:172.21.15.114:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f131402fd00 con 0x557cec134800 2024-04-23T10:36:24.811 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:36:24.808+0000 7f1323c5b740 1 -- 172.21.15.114:0/1424090904 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x557cec134800 msgr2=0x557cec14c550 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T10:36:24.811 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:36:24.808+0000 7f1323c5b740 1 --2- 172.21.15.114:0/1424090904 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x557cec134800 0x557cec14c550 secure :-1 s=READY pgs=830 cs=0 l=1 rev1=1 crypto rx=0x7f1314035000 tx=0x7f131404c000 comp rx=0 tx=0).stop 2024-04-23T10:36:24.811 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:36:24.808+0000 7f1323c5b740 1 -- 172.21.15.114:0/1424090904 shutdown_connections 2024-04-23T10:36:24.811 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:36:24.808+0000 7f1323c5b740 1 --2- 172.21.15.114:0/1424090904 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x557cec134800 0x557cec14c550 unknown :-1 s=CLOSED pgs=830 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:36:24.811 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:36:24.808+0000 7f1323c5b740 1 --2- 172.21.15.114:0/1424090904 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x557cec129f40 0x557cec124b30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:36:24.812 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:36:24.808+0000 7f1323c5b740 1 --2- 172.21.15.114:0/1424090904 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x557cec129620 0x557cec1402f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:36:24.812 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:36:24.808+0000 7f1323c5b740 1 -- 172.21.15.114:0/1424090904 >> 172.21.15.114:0/1424090904 conn(0x557cec125b30 msgr2=0x557cec124420 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T10:36:24.812 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:36:24.808+0000 7f1323c5b740 1 -- 172.21.15.114:0/1424090904 shutdown_connections 2024-04-23T10:36:24.812 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:36:24.808+0000 7f1323c5b740 1 -- 172.21.15.114:0/1424090904 wait complete. 2024-04-23T10:36:24.812 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:36:24.809+0000 7f1323c5b740 1 -- start start 2024-04-23T10:36:24.812 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:36:24.809+0000 7f1323c5b740 1 Processor -- start 2024-04-23T10:36:24.812 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:36:24.809+0000 7f1323c5b740 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x557cec129620 0x557cec1399c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T10:36:24.812 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:36:24.809+0000 7f1323c5b740 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x557cec129f40 0x557cec139f00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T10:36:24.812 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:36:24.809+0000 7f1323c5b740 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x557cec134800 0x557cec1a8dc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T10:36:24.812 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:36:24.809+0000 7f1323c5b740 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x7f131c1ae540 con 0x557cec129620 2024-04-23T10:36:24.812 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:36:24.809+0000 7f1323c5b740 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 -- 0x7f131c1af9b0 con 0x557cec129f40 2024-04-23T10:36:24.812 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:36:24.809+0000 7f1323c5b740 1 -- --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_getmap magic: 0 -- 0x7f131c1aec10 con 0x557cec134800 2024-04-23T10:36:24.812 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:36:24.809+0000 7f1323c5b740 10 client.0 _ll_register_callbacks cb 0x7f1304087110 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-23T10:36:24.812 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:36:24.809+0000 7f131a59c640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x557cec129620 0x557cec1399c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T10:36:24.812 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:36:24.809+0000 7f131a59c640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x557cec129620 0x557cec1399c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.114:39552/0 (socket says 172.21.15.114:39552) 2024-04-23T10:36:24.812 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:36:24.809+0000 7f131a59c640 1 -- 172.21.15.114:0/1660996188 learned_addr learned my addr 172.21.15.114:0/1660996188 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-04-23T10:36:24.813 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:36:24.809+0000 7f1313fff640 1 --2- 172.21.15.114:0/1660996188 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x557cec129f40 0x557cec139f00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T10:36:24.813 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:36:24.809+0000 7f131ad9d640 1 --2- 172.21.15.114:0/1660996188 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x557cec134800 0x557cec1a8dc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T10:36:24.813 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:36:24.809+0000 7f131a59c640 1 -- 172.21.15.114:0/1660996188 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x557cec134800 msgr2=0x557cec1a8dc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T10:36:24.813 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:36:24.809+0000 7f131a59c640 1 --2- 172.21.15.114:0/1660996188 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x557cec134800 0x557cec1a8dc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:36:24.813 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:36:24.809+0000 7f131a59c640 1 -- 172.21.15.114:0/1660996188 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x557cec129f40 msgr2=0x557cec139f00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T10:36:24.813 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:36:24.809+0000 7f131a59c640 1 --2- 172.21.15.114:0/1660996188 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x557cec129f40 0x557cec139f00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:36:24.813 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:36:24.809+0000 7f131a59c640 1 -- 172.21.15.114:0/1660996188 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({osdmap=0}) -- 0x557cec1a9300 con 0x557cec129620 2024-04-23T10:36:24.813 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:36:24.810+0000 7f131a59c640 1 --2- 172.21.15.114:0/1660996188 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x557cec129620 0x557cec1399c0 secure :-1 s=READY pgs=831 cs=0 l=1 rev1=1 crypto rx=0x7f130c00a010 tx=0x7f130c0052f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5dc02d2070a613a8 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T10:36:24.813 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:36:24.810+0000 7f1301ffb640 10 client.0 ms_handle_connect on v2:172.21.15.114:3300/0 2024-04-23T10:36:24.813 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:36:24.810+0000 7f1301ffb640 1 -- 172.21.15.114:0/1660996188 <== mon.0 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f130c02bd50 con 0x557cec129620 2024-04-23T10:36:24.813 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:36:24.810+0000 7f1301ffb640 1 -- 172.21.15.114:0/1660996188 <== mon.0 v2:172.21.15.114:3300/0 2 ==== osd_map(102..102 src has 1..102) ==== 8629+0+0 (secure 0 0 0) 0x7f130c02b280 con 0x557cec129620 2024-04-23T10:36:24.813 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:36:24.810+0000 7f1323c5b740 1 -- 172.21.15.114:0/1660996188 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f1314004690 con 0x557cec129620 2024-04-23T10:36:24.813 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:36:24.810+0000 7f1301ffb640 1 -- 172.21.15.114:0/1660996188 <== mon.0 v2:172.21.15.114:3300/0 3 ==== mon_get_version_reply(handle=1 version=37) ==== 24+0+0 (secure 0 0 0) 0x7f130c02b4a0 con 0x557cec129620 2024-04-23T10:36:24.813 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:36:24.810+0000 7f1323c5b740 10 client.16752 fetch_fsmap learned FSMap version 37 2024-04-23T10:36:24.813 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:36:24.810+0000 7f1323c5b740 1 -- 172.21.15.114:0/1660996188 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({config=0+,fsmap=37,monmap=4+}) -- 0x557cec1aa610 con 0x557cec129620 2024-04-23T10:36:24.813 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:36:24.810+0000 7f1301ffb640 1 -- 172.21.15.114:0/1660996188 <== mon.0 v2:172.21.15.114:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f1314004690 con 0x557cec129620 2024-04-23T10:36:24.814 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:36:24.810+0000 7f1301ffb640 1 -- 172.21.15.114:0/1660996188 <== mon.0 v2:172.21.15.114:3300/0 5 ==== fsmap(e 37) ==== 8358+0+0 (secure 0 0 0) 0x7f130c02b640 con 0x557cec129620 2024-04-23T10:36:24.814 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:36:24.810+0000 7f1323c5b740 10 client.16752 fetch_fsmap finished waiting for FSMap version 37 2024-04-23T10:36:24.814 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:36:24.810+0000 7f1323c5b740 10 client.16752 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-04-23T10:36:24.814 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:36:24.810+0000 7f1323c5b740 20 client.16752 populate_metadata read hostname 'smithi114' 2024-04-23T10:36:24.814 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:36:24.810+0000 7f1323c5b740 1 --2- 172.21.15.114:0/1660996188 >> [v2:172.21.15.119:6838/937297340,v1:172.21.15.119:6839/937297340] conn(0x557cec1aabd0 0x557cec1acff0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T10:36:24.814 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:36:24.810+0000 7f1323c5b740 4 client.16752 mds_command: new command op to 24545 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-04-23T10:36:24.814 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:36:24.810+0000 7f1323c5b740 1 -- 172.21.15.114:0/1660996188 --> [v2:172.21.15.119:6838/937297340,v1:172.21.15.119:6839/937297340] -- command(tid 0: {"prefix": "flush journal"}) -- 0x557cec131cd0 con 0x557cec1aabd0 2024-04-23T10:36:24.814 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:36:24.810+0000 7f1313fff640 1 --2- 172.21.15.114:0/1660996188 >> [v2:172.21.15.119:6838/937297340,v1:172.21.15.119:6839/937297340] conn(0x557cec1aabd0 0x557cec1acff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T10:36:24.814 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:36:24.811+0000 7f1301ffb640 10 client.16752 ms_handle_connect on v2:172.21.15.119:6838/937297340 2024-04-23T10:36:24.814 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:36:24.811+0000 7f1313fff640 1 --2- 172.21.15.114:0/1660996188 >> [v2:172.21.15.119:6838/937297340,v1:172.21.15.119:6839/937297340] conn(0x557cec1aabd0 0x557cec1acff0 crc :-1 s=READY pgs=149 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=abf19593d362b658 server_cookie=4a8d5095dd17606e in_seq=0 out_seq=0 2024-04-23T10:36:24.859 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:36:24.860+0000 7f1301ffb640 1 -- 172.21.15.114:0/1660996188 <== mds.0 v2:172.21.15.119:6838/937297340 1 ==== command_reply(tid 0: 0 ) ==== 8+0+44 (crc 0 0 0) 0x557cec131cd0 con 0x557cec1aabd0 2024-04-23T10:36:24.859 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:36:24.860+0000 7f1301ffb640 10 client.16752 handle_command_reply: tid=0 2024-04-23T10:36:24.859 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:36:24.860+0000 7f1323c5b740 1 client.16752 shutdown 2024-04-23T10:36:24.859 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:36:24.860+0000 7f1323c5b740 1 -- 172.21.15.114:0/1660996188 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x557cec129620 msgr2=0x557cec1399c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T10:36:24.860 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:36:24.860+0000 7f1323c5b740 1 --2- 172.21.15.114:0/1660996188 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x557cec129620 0x557cec1399c0 secure :-1 s=READY pgs=831 cs=0 l=1 rev1=1 crypto rx=0x7f130c00a010 tx=0x7f130c0052f0 comp rx=0 tx=0).stop 2024-04-23T10:36:24.860 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:36:24.860+0000 7f1323c5b740 1 -- 172.21.15.114:0/1660996188 shutdown_connections 2024-04-23T10:36:24.860 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:36:24.860+0000 7f1323c5b740 1 --2- 172.21.15.114:0/1660996188 >> [v2:172.21.15.119:6838/937297340,v1:172.21.15.119:6839/937297340] conn(0x557cec1aabd0 0x557cec1acff0 crc :-1 s=READY pgs=149 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:36:24.860 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:36:24.860+0000 7f1323c5b740 1 --2- 172.21.15.114:0/1660996188 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x557cec134800 0x557cec1a8dc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:36:24.860 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:36:24.860+0000 7f1323c5b740 1 --2- 172.21.15.114:0/1660996188 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x557cec129f40 0x557cec139f00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:36:24.860 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:36:24.860+0000 7f1323c5b740 1 --2- 172.21.15.114:0/1660996188 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x557cec129620 0x557cec1399c0 unknown :-1 s=CLOSED pgs=831 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:36:24.860 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:36:24.860+0000 7f1301ffb640 0 client.16752 ms_handle_reset on v2:172.21.15.119:6838/937297340 2024-04-23T10:36:24.860 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:36:24.860+0000 7f1323c5b740 1 -- 172.21.15.114:0/1660996188 >> 172.21.15.114:0/1660996188 conn(0x557cec125b30 msgr2=0x557cec124cb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T10:36:24.860 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:36:24.860+0000 7f1323c5b740 1 -- 172.21.15.114:0/1660996188 shutdown_connections 2024-04-23T10:36:24.860 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:36:24.860+0000 7f1323c5b740 1 -- 172.21.15.114:0/1660996188 wait complete. 2024-04-23T10:36:24.860 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:36:24.860+0000 7f1323c5b740 20 client.16752 trim_cache size 0 max 16384 2024-04-23T10:36:24.860 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stdout:{ 2024-04-23T10:36:24.861 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stdout: "message": "", 2024-04-23T10:36:24.861 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stdout: "return_code": 0 2024-04-23T10:36:24.861 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stdout:} 2024-04-23T10:36:24.861 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:36:24.861+0000 7f1320f4a640 1 -- 172.21.15.114:0/4228009606 >> [v2:172.21.15.114:6800/1395693466,v1:172.21.15.114:6801/1395693466] conn(0x7f13080a3df0 msgr2=0x7f13080a62b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T10:36:24.861 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:36:24.861+0000 7f1320f4a640 1 --2- 172.21.15.114:0/4228009606 >> [v2:172.21.15.114:6800/1395693466,v1:172.21.15.114:6801/1395693466] conn(0x7f13080a3df0 0x7f13080a62b0 secure :-1 s=READY pgs=128 cs=0 l=1 rev1=1 crypto rx=0x7f1314006010 tx=0x7f131402f4a0 comp rx=0 tx=0).stop 2024-04-23T10:36:24.861 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:36:24.861+0000 7f1320f4a640 1 -- 172.21.15.114:0/4228009606 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f131c106190 msgr2=0x7f131c10ec50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T10:36:24.862 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:36:24.861+0000 7f1320f4a640 1 --2- 172.21.15.114:0/4228009606 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f131c106190 0x7f131c10ec50 secure :-1 s=READY pgs=92 cs=0 l=1 rev1=1 crypto rx=0x7f130400c970 tx=0x7f130400ce30 comp rx=0 tx=0).stop 2024-04-23T10:36:24.862 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:36:24.861+0000 7f1320f4a640 1 -- 172.21.15.114:0/4228009606 shutdown_connections 2024-04-23T10:36:24.862 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:36:24.861+0000 7f1320f4a640 1 --2- 172.21.15.114:0/4228009606 >> [v2:172.21.15.114:6800/1395693466,v1:172.21.15.114:6801/1395693466] conn(0x7f13080a3df0 0x7f13080a62b0 unknown :-1 s=CLOSED pgs=128 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:36:24.862 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:36:24.861+0000 7f1320f4a640 1 --2- 172.21.15.114:0/4228009606 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f131c10f190 0x7f131c1a6840 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:36:24.862 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:36:24.861+0000 7f1320f4a640 1 --2- 172.21.15.114:0/4228009606 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f131c106190 0x7f131c10ec50 unknown :-1 s=CLOSED pgs=92 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:36:24.862 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:36:24.861+0000 7f1320f4a640 1 --2- 172.21.15.114:0/4228009606 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f131c101310 0x7f131c10e710 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:36:24.862 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:36:24.861+0000 7f1320f4a640 1 -- 172.21.15.114:0/4228009606 >> 172.21.15.114:0/4228009606 conn(0x7f131c077cc0 msgr2=0x7f131c0ff0b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T10:36:24.862 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:36:24.861+0000 7f1320f4a640 1 -- 172.21.15.114:0/4228009606 shutdown_connections 2024-04-23T10:36:24.862 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:36:24.861+0000 7f1320f4a640 1 -- 172.21.15.114:0/4228009606 wait complete. 2024-04-23T10:36:24.937 INFO:tasks.workunit.client.0.smithi114.stdout: 1 647087 6.48 MB/sec execute 393 sec latency 4903.522 ms 2024-04-23T10:36:25.827 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 23 10:36:25 smithi204 ceph-mon[32649]: pgmap v307: 97 pgs: 97 active+clean; 195 MiB data, 2.5 GiB used, 1.0 TiB / 1.0 TiB avail; 375 KiB/s rd, 843 KiB/s wr, 101 op/s 2024-04-23T10:36:25.937 INFO:tasks.workunit.client.0.smithi114.stdout: 1 649362 6.49 MB/sec execute 394 sec latency 16.929 ms 2024-04-23T10:36:25.991 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 23 10:36:25 smithi119 ceph-mon[32608]: pgmap v307: 97 pgs: 97 active+clean; 195 MiB data, 2.5 GiB used, 1.0 TiB / 1.0 TiB avail; 375 KiB/s rd, 843 KiB/s wr, 101 op/s 2024-04-23T10:36:26.004 INFO:journalctl@ceph.mon.a.smithi114.stdout:Apr 23 10:36:25 smithi114 ceph-mon[29255]: pgmap v307: 97 pgs: 97 active+clean; 195 MiB data, 2.5 GiB used, 1.0 TiB / 1.0 TiB avail; 375 KiB/s rd, 843 KiB/s wr, 101 op/s 2024-04-23T10:36:26.937 INFO:tasks.workunit.client.0.smithi114.stdout: 1 649964 6.49 MB/sec execute 395 sec latency 785.506 ms 2024-04-23T10:36:27.827 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 23 10:36:27 smithi204 ceph-mon[32649]: pgmap v308: 97 pgs: 97 active+clean; 197 MiB data, 2.5 GiB used, 1.0 TiB / 1.0 TiB avail; 394 KiB/s rd, 986 KiB/s wr, 79 op/s 2024-04-23T10:36:27.937 INFO:tasks.workunit.client.0.smithi114.stdout: 1 649964 6.47 MB/sec execute 396 sec latency 1785.614 ms 2024-04-23T10:36:27.992 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 23 10:36:27 smithi119 ceph-mon[32608]: pgmap v308: 97 pgs: 97 active+clean; 197 MiB data, 2.5 GiB used, 1.0 TiB / 1.0 TiB avail; 394 KiB/s rd, 986 KiB/s wr, 79 op/s 2024-04-23T10:36:28.004 INFO:journalctl@ceph.mon.a.smithi114.stdout:Apr 23 10:36:27 smithi114 ceph-mon[29255]: pgmap v308: 97 pgs: 97 active+clean; 197 MiB data, 2.5 GiB used, 1.0 TiB / 1.0 TiB avail; 394 KiB/s rd, 986 KiB/s wr, 79 op/s 2024-04-23T10:36:28.937 INFO:tasks.workunit.client.0.smithi114.stdout: 1 649964 6.45 MB/sec execute 397 sec latency 2785.726 ms 2024-04-23T10:36:29.828 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 23 10:36:29 smithi204 ceph-mon[32649]: pgmap v309: 97 pgs: 97 active+clean; 197 MiB data, 2.5 GiB used, 1.0 TiB / 1.0 TiB avail; 394 KiB/s rd, 986 KiB/s wr, 79 op/s 2024-04-23T10:36:29.937 INFO:tasks.workunit.client.0.smithi114.stdout: 1 650031 6.44 MB/sec execute 398 sec latency 3030.948 ms 2024-04-23T10:36:29.992 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 23 10:36:29 smithi119 ceph-mon[32608]: pgmap v309: 97 pgs: 97 active+clean; 197 MiB data, 2.5 GiB used, 1.0 TiB / 1.0 TiB avail; 394 KiB/s rd, 986 KiB/s wr, 79 op/s 2024-04-23T10:36:30.004 INFO:journalctl@ceph.mon.a.smithi114.stdout:Apr 23 10:36:29 smithi114 ceph-mon[29255]: pgmap v309: 97 pgs: 97 active+clean; 197 MiB data, 2.5 GiB used, 1.0 TiB / 1.0 TiB avail; 394 KiB/s rd, 986 KiB/s wr, 79 op/s 2024-04-23T10:36:30.937 INFO:tasks.workunit.client.0.smithi114.stdout: 1 650031 6.42 MB/sec execute 399 sec latency 1652.573 ms 2024-04-23T10:36:31.827 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 23 10:36:31 smithi204 ceph-mon[32649]: pgmap v310: 97 pgs: 97 active+clean; 197 MiB data, 2.5 GiB used, 1.0 TiB / 1.0 TiB avail; 329 KiB/s rd, 824 KiB/s wr, 66 op/s 2024-04-23T10:36:31.937 INFO:tasks.workunit.client.0.smithi114.stdout: 1 650031 6.41 MB/sec execute 400 sec latency 2652.662 ms 2024-04-23T10:36:31.992 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 23 10:36:31 smithi119 ceph-mon[32608]: pgmap v310: 97 pgs: 97 active+clean; 197 MiB data, 2.5 GiB used, 1.0 TiB / 1.0 TiB avail; 329 KiB/s rd, 824 KiB/s wr, 66 op/s 2024-04-23T10:36:32.004 INFO:journalctl@ceph.mon.a.smithi114.stdout:Apr 23 10:36:31 smithi114 ceph-mon[29255]: pgmap v310: 97 pgs: 97 active+clean; 197 MiB data, 2.5 GiB used, 1.0 TiB / 1.0 TiB avail; 329 KiB/s rd, 824 KiB/s wr, 66 op/s 2024-04-23T10:36:32.937 INFO:tasks.workunit.client.0.smithi114.stdout: 1 650031 6.39 MB/sec execute 401 sec latency 3652.711 ms 2024-04-23T10:36:33.828 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 23 10:36:33 smithi204 ceph-mon[32649]: pgmap v311: 97 pgs: 97 active+clean; 197 MiB data, 2.5 GiB used, 1.0 TiB / 1.0 TiB avail; 356 KiB/s rd, 861 KiB/s wr, 76 op/s 2024-04-23T10:36:33.938 INFO:tasks.workunit.client.0.smithi114.stdout: 1 650031 6.37 MB/sec execute 402 sec latency 4652.802 ms 2024-04-23T10:36:33.992 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 23 10:36:33 smithi119 ceph-mon[32608]: pgmap v311: 97 pgs: 97 active+clean; 197 MiB data, 2.5 GiB used, 1.0 TiB / 1.0 TiB avail; 356 KiB/s rd, 861 KiB/s wr, 76 op/s 2024-04-23T10:36:34.004 INFO:journalctl@ceph.mon.a.smithi114.stdout:Apr 23 10:36:33 smithi114 ceph-mon[29255]: pgmap v311: 97 pgs: 97 active+clean; 197 MiB data, 2.5 GiB used, 1.0 TiB / 1.0 TiB avail; 356 KiB/s rd, 861 KiB/s wr, 76 op/s 2024-04-23T10:36:34.938 INFO:tasks.workunit.client.0.smithi114.stdout: 1 650794 6.36 MB/sec execute 403 sec latency 4895.633 ms 2024-04-23T10:36:35.827 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 23 10:36:35 smithi204 ceph-mon[32649]: pgmap v312: 97 pgs: 97 active+clean; 197 MiB data, 2.5 GiB used, 1.0 TiB / 1.0 TiB avail; 322 KiB/s rd, 822 KiB/s wr, 66 op/s 2024-04-23T10:36:35.938 INFO:tasks.workunit.client.0.smithi114.stdout: 1 653233 6.38 MB/sec execute 404 sec latency 13.915 ms 2024-04-23T10:36:35.991 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 23 10:36:35 smithi119 ceph-mon[32608]: pgmap v312: 97 pgs: 97 active+clean; 197 MiB data, 2.5 GiB used, 1.0 TiB / 1.0 TiB avail; 322 KiB/s rd, 822 KiB/s wr, 66 op/s 2024-04-23T10:36:36.003 INFO:journalctl@ceph.mon.a.smithi114.stdout:Apr 23 10:36:35 smithi114 ceph-mon[29255]: pgmap v312: 97 pgs: 97 active+clean; 197 MiB data, 2.5 GiB used, 1.0 TiB / 1.0 TiB avail; 322 KiB/s rd, 822 KiB/s wr, 66 op/s 2024-04-23T10:36:36.938 INFO:tasks.workunit.client.0.smithi114.stdout: 1 653509 6.37 MB/sec execute 405 sec latency 869.615 ms 2024-04-23T10:36:37.828 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 23 10:36:37 smithi204 ceph-mon[32649]: pgmap v313: 97 pgs: 97 active+clean; 201 MiB data, 2.5 GiB used, 1.0 TiB / 1.0 TiB avail; 616 KiB/s rd, 1.3 MiB/s wr, 112 op/s 2024-04-23T10:36:37.938 INFO:tasks.workunit.client.0.smithi114.stdout: 1 653509 6.36 MB/sec execute 406 sec latency 1869.690 ms 2024-04-23T10:36:37.955 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:36:37.956+0000 7f40b4de1640 1 -- 172.21.15.114:0/14280679 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f40b01072a0 msgr2=0x7f40b010ee70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T10:36:37.956 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:36:37.956+0000 7f40b4de1640 1 --2- 172.21.15.114:0/14280679 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f40b01072a0 0x7f40b010ee70 secure :-1 s=READY pgs=832 cs=0 l=1 rev1=1 crypto rx=0x7f40a8009ca0 tx=0x7f40a802a8c0 comp rx=0 tx=0).stop 2024-04-23T10:36:37.956 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:36:37.956+0000 7f40b4de1640 1 -- 172.21.15.114:0/14280679 shutdown_connections 2024-04-23T10:36:37.956 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:36:37.956+0000 7f40b4de1640 1 --2- 172.21.15.114:0/14280679 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f40b01072a0 0x7f40b010ee70 unknown :-1 s=CLOSED pgs=832 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:36:37.956 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:36:37.956+0000 7f40b4de1640 1 --2- 172.21.15.114:0/14280679 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f40b0102b80 0x7f40b0106bc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:36:37.956 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:36:37.956+0000 7f40b4de1640 1 --2- 172.21.15.114:0/14280679 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f40b01021d0 0x7f40b01025b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:36:37.956 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:36:37.956+0000 7f40b4de1640 1 -- 172.21.15.114:0/14280679 >> 172.21.15.114:0/14280679 conn(0x7f40b00793e0 msgr2=0x7f40b00797f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T10:36:37.956 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:36:37.956+0000 7f40b4de1640 1 -- 172.21.15.114:0/14280679 shutdown_connections 2024-04-23T10:36:37.956 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:36:37.957+0000 7f40b4de1640 1 -- 172.21.15.114:0/14280679 wait complete. 2024-04-23T10:36:37.958 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:36:37.957+0000 7f40b4de1640 1 Processor -- start 2024-04-23T10:36:37.958 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:36:37.957+0000 7f40b4de1640 1 -- start start 2024-04-23T10:36:37.959 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:36:37.957+0000 7f40b4de1640 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f40b01021d0 0x7f40b01985d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T10:36:37.959 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:36:37.957+0000 7f40b4de1640 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f40b01072a0 0x7f40b0198b10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T10:36:37.959 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:36:37.957+0000 7f40b4de1640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f40b019ce90 0x7f40b019d340 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T10:36:37.959 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:36:37.957+0000 7f40b4de1640 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x7f40b0112210 con 0x7f40b019ce90 2024-04-23T10:36:37.959 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:36:37.957+0000 7f40b4de1640 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 -- 0x7f40b0112090 con 0x7f40b01021d0 2024-04-23T10:36:37.959 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:36:37.957+0000 7f40b4de1640 1 -- --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_getmap magic: 0 -- 0x7f40b0112390 con 0x7f40b01072a0 2024-04-23T10:36:37.959 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:36:37.958+0000 7f40aed9d640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f40b019ce90 0x7f40b019d340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T10:36:37.959 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:36:37.958+0000 7f40aed9d640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f40b019ce90 0x7f40b019d340 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.114:34480/0 (socket says 172.21.15.114:34480) 2024-04-23T10:36:37.959 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:36:37.958+0000 7f40aed9d640 1 -- 172.21.15.114:0/1027537292 learned_addr learned my addr 172.21.15.114:0/1027537292 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-04-23T10:36:37.959 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:36:37.958+0000 7f40a7fff640 1 --2- 172.21.15.114:0/1027537292 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f40b01072a0 0x7f40b0198b10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T10:36:37.959 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:36:37.958+0000 7f40ae59c640 1 --2- 172.21.15.114:0/1027537292 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f40b01021d0 0x7f40b01985d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T10:36:37.959 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:36:37.958+0000 7f40aed9d640 1 -- 172.21.15.114:0/1027537292 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f40b01072a0 msgr2=0x7f40b0198b10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T10:36:37.959 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:36:37.958+0000 7f40aed9d640 1 --2- 172.21.15.114:0/1027537292 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f40b01072a0 0x7f40b0198b10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:36:37.960 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:36:37.958+0000 7f40aed9d640 1 -- 172.21.15.114:0/1027537292 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f40b01021d0 msgr2=0x7f40b01985d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T10:36:37.960 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:36:37.958+0000 7f40aed9d640 1 --2- 172.21.15.114:0/1027537292 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f40b01021d0 0x7f40b01985d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:36:37.960 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:36:37.958+0000 7f40aed9d640 1 -- 172.21.15.114:0/1027537292 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f40b019d980 con 0x7f40b019ce90 2024-04-23T10:36:37.960 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:36:37.958+0000 7f40aed9d640 1 --2- 172.21.15.114:0/1027537292 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f40b019ce90 0x7f40b019d340 secure :-1 s=READY pgs=833 cs=0 l=1 rev1=1 crypto rx=0x7f40a80098d0 tx=0x7f40a80317b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T10:36:37.960 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:36:37.958+0000 7f40a77fe640 1 -- 172.21.15.114:0/1027537292 <== mon.0 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f40a8003b70 con 0x7f40b019ce90 2024-04-23T10:36:37.960 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:36:37.958+0000 7f40a77fe640 1 -- 172.21.15.114:0/1027537292 <== mon.0 v2:172.21.15.114:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f40a8003d10 con 0x7f40b019ce90 2024-04-23T10:36:37.960 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:36:37.958+0000 7f40b4de1640 1 -- 172.21.15.114:0/1027537292 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f40b019dc10 con 0x7f40b019ce90 2024-04-23T10:36:37.960 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:36:37.958+0000 7f40a77fe640 1 -- 172.21.15.114:0/1027537292 <== mon.0 v2:172.21.15.114:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f40a802fa20 con 0x7f40b019ce90 2024-04-23T10:36:37.960 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:36:37.958+0000 7f40b4de1640 1 -- 172.21.15.114:0/1027537292 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f40b01a5430 con 0x7f40b019ce90 2024-04-23T10:36:37.967 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:36:37.961+0000 7f40a77fe640 1 -- 172.21.15.114:0/1027537292 <== mon.0 v2:172.21.15.114:3300/0 4 ==== mgrmap(e 26) ==== 138778+0+0 (secure 0 0 0) 0x7f40a802fc40 con 0x7f40b019ce90 2024-04-23T10:36:37.967 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:36:37.961+0000 7f40b7bcb740 1 Processor -- start 2024-04-23T10:36:37.967 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:36:37.961+0000 7f40a77fe640 1 --2- 172.21.15.114:0/1027537292 >> [v2:172.21.15.114:6800/1395693466,v1:172.21.15.114:6801/1395693466] conn(0x7f409c0a3dc0 0x7f409c0a6280 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T10:36:37.967 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:36:37.961+0000 7f40b7bcb740 1 -- start start 2024-04-23T10:36:37.967 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:36:37.961+0000 7f40a77fe640 1 -- 172.21.15.114:0/1027537292 <== mon.0 v2:172.21.15.114:3300/0 5 ==== osd_map(102..102 src has 1..102) ==== 8629+0+0 (secure 0 0 0) 0x7f40a80f1d00 con 0x7f40b019ce90 2024-04-23T10:36:37.967 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:36:37.961+0000 7f40b7bcb740 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x56255adca620 0x56255adcaa00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T10:36:37.967 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:36:37.961+0000 7f40b7bcb740 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x56255adcaf40 0x56255add52c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T10:36:37.967 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:36:37.962+0000 7f40a7fff640 1 --2- 172.21.15.114:0/1027537292 >> [v2:172.21.15.114:6800/1395693466,v1:172.21.15.114:6801/1395693466] conn(0x7f409c0a3dc0 0x7f409c0a6280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T10:36:37.968 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:36:37.962+0000 7f40b7bcb740 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x56255add5800 0x56255add7bf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T10:36:37.968 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:36:37.962+0000 7f40b7bcb740 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x56255ad2f970 con 0x56255add5800 2024-04-23T10:36:37.968 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:36:37.962+0000 7f40b7bcb740 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 -- 0x56255accc4a0 con 0x56255adcaf40 2024-04-23T10:36:37.968 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:36:37.962+0000 7f40b7bcb740 1 -- --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_getmap magic: 0 -- 0x56255ad2b930 con 0x56255adca620 2024-04-23T10:36:37.968 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:36:37.962+0000 7f40aed9d640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x56255add5800 0x56255add7bf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T10:36:37.968 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:36:37.962+0000 7f40aed9d640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x56255add5800 0x56255add7bf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.114:34496/0 (socket says 172.21.15.114:34496) 2024-04-23T10:36:37.968 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:36:37.962+0000 7f40aed9d640 1 -- 172.21.15.114:0/887160254 learned_addr learned my addr 172.21.15.114:0/887160254 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-04-23T10:36:37.968 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:36:37.962+0000 7f40ae59c640 1 --2- 172.21.15.114:0/887160254 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x56255adca620 0x56255adcaa00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T10:36:37.968 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:36:37.962+0000 7f40a7fff640 1 --2- 172.21.15.114:0/887160254 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x56255adcaf40 0x56255add52c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T10:36:37.968 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:36:37.962+0000 7f40a7fff640 1 --2- 172.21.15.114:0/1027537292 >> [v2:172.21.15.114:6800/1395693466,v1:172.21.15.114:6801/1395693466] conn(0x7f409c0a3dc0 0x7f409c0a6280 secure :-1 s=READY pgs=129 cs=0 l=1 rev1=1 crypto rx=0x7f4098005940 tx=0x7f409800af00 comp rx=0 tx=0).ready entity=mgr.15291 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T10:36:37.968 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:36:37.962+0000 7f40aed9d640 1 -- 172.21.15.114:0/887160254 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x56255adca620 msgr2=0x56255adcaa00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T10:36:37.968 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:36:37.962+0000 7f40aed9d640 1 --2- 172.21.15.114:0/887160254 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x56255adca620 0x56255adcaa00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:36:37.968 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:36:37.962+0000 7f40aed9d640 1 -- 172.21.15.114:0/887160254 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x56255adcaf40 msgr2=0x56255add52c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T10:36:37.968 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:36:37.962+0000 7f40aed9d640 1 --2- 172.21.15.114:0/887160254 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x56255adcaf40 0x56255add52c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:36:37.969 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:36:37.962+0000 7f40aed9d640 1 -- 172.21.15.114:0/887160254 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x56255aa42210 con 0x56255add5800 2024-04-23T10:36:37.969 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:36:37.962+0000 7f40aed9d640 1 --2- 172.21.15.114:0/887160254 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x56255add5800 0x56255add7bf0 secure :-1 s=READY pgs=834 cs=0 l=1 rev1=1 crypto rx=0x7f40a80ba1a0 tx=0x7f40a80bd940 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1dd682eee5028658 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T10:36:37.969 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:36:37.962+0000 7f40a4ff9640 1 -- 172.21.15.114:0/887160254 <== mon.0 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f40a80c20f0 con 0x56255add5800 2024-04-23T10:36:37.969 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:36:37.962+0000 7f40a4ff9640 1 -- 172.21.15.114:0/887160254 <== mon.0 v2:172.21.15.114:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f40a8036070 con 0x56255add5800 2024-04-23T10:36:37.969 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:36:37.962+0000 7f40a4ff9640 1 -- 172.21.15.114:0/887160254 <== mon.0 v2:172.21.15.114:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f40a80bdfa0 con 0x56255add5800 2024-04-23T10:36:37.969 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:36:37.963+0000 7f40b7bcb740 1 -- 172.21.15.114:0/887160254 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x56255add5800 msgr2=0x56255add7bf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T10:36:37.969 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:36:37.963+0000 7f40b7bcb740 1 --2- 172.21.15.114:0/887160254 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x56255add5800 0x56255add7bf0 secure :-1 s=READY pgs=834 cs=0 l=1 rev1=1 crypto rx=0x7f40a80ba1a0 tx=0x7f40a80bd940 comp rx=0 tx=0).stop 2024-04-23T10:36:37.969 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:36:37.963+0000 7f40b7bcb740 1 -- 172.21.15.114:0/887160254 shutdown_connections 2024-04-23T10:36:37.969 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:36:37.963+0000 7f40b7bcb740 1 --2- 172.21.15.114:0/887160254 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x56255add5800 0x56255add7bf0 unknown :-1 s=CLOSED pgs=834 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:36:37.969 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:36:37.963+0000 7f40b7bcb740 1 --2- 172.21.15.114:0/887160254 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x56255adcaf40 0x56255add52c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:36:37.969 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:36:37.963+0000 7f40b7bcb740 1 --2- 172.21.15.114:0/887160254 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x56255adca620 0x56255adcaa00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:36:37.969 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:36:37.963+0000 7f40b7bcb740 1 -- 172.21.15.114:0/887160254 >> 172.21.15.114:0/887160254 conn(0x56255adc6b30 msgr2=0x56255adc6f10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T10:36:37.969 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:36:37.963+0000 7f40b7bcb740 1 -- 172.21.15.114:0/887160254 shutdown_connections 2024-04-23T10:36:37.969 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:36:37.963+0000 7f40b7bcb740 1 -- 172.21.15.114:0/887160254 wait complete. 2024-04-23T10:36:37.969 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:36:37.964+0000 7f40b7bcb740 1 -- start start 2024-04-23T10:36:37.970 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:36:37.964+0000 7f40b7bcb740 1 Processor -- start 2024-04-23T10:36:37.970 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:36:37.964+0000 7f40b7bcb740 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x56255adca620 0x56255add5140 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T10:36:37.970 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:36:37.964+0000 7f40b7bcb740 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x56255adcaf40 0x56255add2430 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T10:36:37.970 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:36:37.964+0000 7f40b7bcb740 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x56255add5800 0x56255add2970 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T10:36:37.970 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:36:37.964+0000 7f40b7bcb740 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x56255ad2f970 con 0x56255adcaf40 2024-04-23T10:36:37.970 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:36:37.964+0000 7f40b7bcb740 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 -- 0x56255acca0b0 con 0x56255adca620 2024-04-23T10:36:37.970 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:36:37.964+0000 7f40b7bcb740 1 -- --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_getmap magic: 0 -- 0x56255add81f0 con 0x56255add5800 2024-04-23T10:36:37.970 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:36:37.964+0000 7f40b7bcb740 10 client.0 _ll_register_callbacks cb 0x56255acb7d30 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-23T10:36:37.970 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:36:37.964+0000 7f40a7fff640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x56255adcaf40 0x56255add2430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T10:36:37.970 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:36:37.964+0000 7f40a7fff640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x56255adcaf40 0x56255add2430 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.114:34500/0 (socket says 172.21.15.114:34500) 2024-04-23T10:36:37.970 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:36:37.964+0000 7f40a7fff640 1 -- 172.21.15.114:0/61558874 learned_addr learned my addr 172.21.15.114:0/61558874 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-04-23T10:36:37.970 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:36:37.964+0000 7f40ae59c640 1 --2- 172.21.15.114:0/61558874 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x56255adca620 0x56255add5140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T10:36:37.970 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:36:37.964+0000 7f40aed9d640 1 --2- 172.21.15.114:0/61558874 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x56255add5800 0x56255add2970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T10:36:37.971 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:36:37.964+0000 7f40a7fff640 1 -- 172.21.15.114:0/61558874 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x56255add5800 msgr2=0x56255add2970 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T10:36:37.971 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:36:37.964+0000 7f40a7fff640 1 --2- 172.21.15.114:0/61558874 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x56255add5800 0x56255add2970 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:36:37.971 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:36:37.964+0000 7f40a7fff640 1 -- 172.21.15.114:0/61558874 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x56255adca620 msgr2=0x56255add5140 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T10:36:37.971 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:36:37.964+0000 7f40a7fff640 1 --2- 172.21.15.114:0/61558874 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x56255adca620 0x56255add5140 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:36:37.971 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:36:37.964+0000 7f40a7fff640 1 -- 172.21.15.114:0/61558874 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({osdmap=0}) -- 0x56255ab0e620 con 0x56255adcaf40 2024-04-23T10:36:37.971 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:36:37.964+0000 7f40ae59c640 1 --2- 172.21.15.114:0/61558874 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x56255adca620 0x56255add5140 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-23T10:36:37.971 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:36:37.964+0000 7f40aed9d640 1 --2- 172.21.15.114:0/61558874 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x56255add5800 0x56255add2970 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-23T10:36:37.971 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:36:37.965+0000 7f40a7fff640 1 --2- 172.21.15.114:0/61558874 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x56255adcaf40 0x56255add2430 secure :-1 s=READY pgs=835 cs=0 l=1 rev1=1 crypto rx=0x7f40b006b630 tx=0x7f4098004730 comp rx=0 tx=0).ready entity=mon.0 client_cookie=28acf318426439f5 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T10:36:37.971 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:36:37.965+0000 7f4095ffb640 10 client.0 ms_handle_connect on v2:172.21.15.114:3300/0 2024-04-23T10:36:37.971 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:36:37.965+0000 7f4095ffb640 1 -- 172.21.15.114:0/61558874 <== mon.0 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4098022070 con 0x56255adcaf40 2024-04-23T10:36:37.971 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:36:37.965+0000 7f4095ffb640 1 -- 172.21.15.114:0/61558874 <== mon.0 v2:172.21.15.114:3300/0 2 ==== osd_map(102..102 src has 1..102) ==== 8629+0+0 (secure 0 0 0) 0x7f4098011a10 con 0x56255adcaf40 2024-04-23T10:36:37.971 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:36:37.965+0000 7f40b7bcb740 1 -- 172.21.15.114:0/61558874 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f40a8036070 con 0x56255adcaf40 2024-04-23T10:36:37.971 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:36:37.965+0000 7f4095ffb640 1 -- 172.21.15.114:0/61558874 <== mon.0 v2:172.21.15.114:3300/0 3 ==== mon_get_version_reply(handle=1 version=37) ==== 24+0+0 (secure 0 0 0) 0x7f4098011c10 con 0x56255adcaf40 2024-04-23T10:36:37.972 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:36:37.965+0000 7f40b7bcb740 10 client.16764 fetch_fsmap learned FSMap version 37 2024-04-23T10:36:37.972 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:36:37.965+0000 7f40b7bcb740 1 -- 172.21.15.114:0/61558874 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({config=0+,fsmap=37,monmap=4+}) -- 0x56255ade3f60 con 0x56255adcaf40 2024-04-23T10:36:37.972 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:36:37.965+0000 7f4095ffb640 1 -- 172.21.15.114:0/61558874 <== mon.0 v2:172.21.15.114:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f40a8036070 con 0x56255adcaf40 2024-04-23T10:36:37.972 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:36:37.965+0000 7f4095ffb640 1 -- 172.21.15.114:0/61558874 <== mon.0 v2:172.21.15.114:3300/0 5 ==== fsmap(e 37) ==== 8358+0+0 (secure 0 0 0) 0x7f409802f420 con 0x56255adcaf40 2024-04-23T10:36:37.972 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:36:37.965+0000 7f40b7bcb740 10 client.16764 fetch_fsmap finished waiting for FSMap version 37 2024-04-23T10:36:37.972 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:36:37.965+0000 7f40b7bcb740 10 client.16764 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-04-23T10:36:37.972 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:36:37.965+0000 7f40b7bcb740 20 client.16764 populate_metadata read hostname 'smithi114' 2024-04-23T10:36:37.972 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:36:37.965+0000 7f40b7bcb740 1 --2- 172.21.15.114:0/61558874 >> [v2:172.21.15.119:6838/937297340,v1:172.21.15.119:6839/937297340] conn(0x56255ade54c0 0x56255ade7910 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T10:36:37.972 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:36:37.965+0000 7f40b7bcb740 4 client.16764 mds_command: new command op to 24545 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-23T10:36:37.972 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:36:37.965+0000 7f40b7bcb740 1 -- 172.21.15.114:0/61558874 --> [v2:172.21.15.119:6838/937297340,v1:172.21.15.119:6839/937297340] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x56255ad3d390 con 0x56255ade54c0 2024-04-23T10:36:37.972 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:36:37.966+0000 7f40ae59c640 1 --2- 172.21.15.114:0/61558874 >> [v2:172.21.15.119:6838/937297340,v1:172.21.15.119:6839/937297340] conn(0x56255ade54c0 0x56255ade7910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T10:36:37.972 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:36:37.966+0000 7f40ae59c640 1 --2- 172.21.15.114:0/61558874 >> [v2:172.21.15.119:6838/937297340,v1:172.21.15.119:6839/937297340] conn(0x56255ade54c0 0x56255ade7910 crc :-1 s=READY pgs=150 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=66e383e52a4400fa server_cookie=b8ec67fa6d8e08c9 in_seq=0 out_seq=0 2024-04-23T10:36:37.972 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:36:37.966+0000 7f4095ffb640 10 client.16764 ms_handle_connect on v2:172.21.15.119:6838/937297340 2024-04-23T10:36:37.972 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:36:37.967+0000 7f4095ffb640 1 -- 172.21.15.114:0/61558874 <== mds.0 v2:172.21.15.119:6838/937297340 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x56255ad3d390 con 0x56255ade54c0 2024-04-23T10:36:37.972 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:36:37.967+0000 7f4095ffb640 10 client.16764 handle_command_reply: tid=0 2024-04-23T10:36:37.973 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:36:37.967+0000 7f40b7bcb740 1 client.16764 shutdown 2024-04-23T10:36:37.973 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:36:37.967+0000 7f40b7bcb740 1 -- 172.21.15.114:0/61558874 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x56255adcaf40 msgr2=0x56255add2430 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T10:36:37.973 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:36:37.967+0000 7f40b7bcb740 1 --2- 172.21.15.114:0/61558874 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x56255adcaf40 0x56255add2430 secure :-1 s=READY pgs=835 cs=0 l=1 rev1=1 crypto rx=0x7f40b006b630 tx=0x7f4098004730 comp rx=0 tx=0).stop 2024-04-23T10:36:37.973 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:36:37.967+0000 7f40b7bcb740 1 -- 172.21.15.114:0/61558874 shutdown_connections 2024-04-23T10:36:37.973 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:36:37.967+0000 7f40b7bcb740 1 --2- 172.21.15.114:0/61558874 >> [v2:172.21.15.119:6838/937297340,v1:172.21.15.119:6839/937297340] conn(0x56255ade54c0 0x56255ade7910 crc :-1 s=READY pgs=150 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:36:37.973 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:36:37.967+0000 7f40b7bcb740 1 --2- 172.21.15.114:0/61558874 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x56255add5800 0x56255add2970 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:36:37.973 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:36:37.967+0000 7f4095ffb640 0 client.16764 ms_handle_reset on v2:172.21.15.119:6838/937297340 2024-04-23T10:36:37.973 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:36:37.967+0000 7f40b7bcb740 1 --2- 172.21.15.114:0/61558874 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x56255adcaf40 0x56255add2430 unknown :-1 s=CLOSED pgs=835 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:36:37.973 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:36:37.967+0000 7f40b7bcb740 1 --2- 172.21.15.114:0/61558874 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x56255adca620 0x56255add5140 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:36:37.973 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:36:37.967+0000 7f40b7bcb740 1 -- 172.21.15.114:0/61558874 >> 172.21.15.114:0/61558874 conn(0x56255adc6b30 msgr2=0x56255adc5cb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T10:36:37.973 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:36:37.967+0000 7f40b7bcb740 1 -- 172.21.15.114:0/61558874 shutdown_connections 2024-04-23T10:36:37.973 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:36:37.967+0000 7f40b7bcb740 1 -- 172.21.15.114:0/61558874 wait complete. 2024-04-23T10:36:37.973 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:36:37.967+0000 7f40b7bcb740 20 client.16764 trim_cache size 0 max 16384 2024-04-23T10:36:37.985 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:36:37.981+0000 7f40b7bcb740 1 Processor -- start 2024-04-23T10:36:37.985 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:36:37.982+0000 7f40b7bcb740 1 -- start start 2024-04-23T10:36:37.985 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:36:37.982+0000 7f40b7bcb740 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x56255adca620 0x56255add4c70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T10:36:37.986 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:36:37.982+0000 7f40b7bcb740 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x56255adcaf40 0x56255aded220 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T10:36:37.986 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:36:37.982+0000 7f40b7bcb740 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x56255add5800 0x56255aded760 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T10:36:37.986 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:36:37.982+0000 7f40b7bcb740 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x7f40b01b05b0 con 0x56255adca620 2024-04-23T10:36:37.986 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:36:37.982+0000 7f40b7bcb740 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 -- 0x7f40b01addc0 con 0x56255add5800 2024-04-23T10:36:37.986 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:36:37.982+0000 7f40b7bcb740 1 -- --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_getmap magic: 0 -- 0x7f40b01abc30 con 0x56255adcaf40 2024-04-23T10:36:37.986 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:36:37.982+0000 7f40ae59c640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x56255adca620 0x56255add4c70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T10:36:37.986 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:36:37.982+0000 7f40ae59c640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x56255adca620 0x56255add4c70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.114:34510/0 (socket says 172.21.15.114:34510) 2024-04-23T10:36:37.986 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:36:37.982+0000 7f40ae59c640 1 -- 172.21.15.114:0/96007216 learned_addr learned my addr 172.21.15.114:0/96007216 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-04-23T10:36:37.986 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:36:37.982+0000 7f40a7fff640 1 --2- 172.21.15.114:0/96007216 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x56255adcaf40 0x56255aded220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T10:36:37.986 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:36:37.982+0000 7f40aed9d640 1 --2- 172.21.15.114:0/96007216 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x56255add5800 0x56255aded760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T10:36:37.986 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:36:37.982+0000 7f40ae59c640 1 -- 172.21.15.114:0/96007216 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x56255adcaf40 msgr2=0x56255aded220 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T10:36:37.986 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:36:37.982+0000 7f40ae59c640 1 --2- 172.21.15.114:0/96007216 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x56255adcaf40 0x56255aded220 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:36:37.986 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:36:37.982+0000 7f40ae59c640 1 -- 172.21.15.114:0/96007216 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x56255add5800 msgr2=0x56255aded760 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T10:36:37.986 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:36:37.982+0000 7f40ae59c640 1 --2- 172.21.15.114:0/96007216 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x56255add5800 0x56255aded760 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:36:37.987 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:36:37.982+0000 7f40ae59c640 1 -- 172.21.15.114:0/96007216 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x56255ade7f90 con 0x56255adca620 2024-04-23T10:36:37.987 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:36:37.982+0000 7f40ae59c640 1 --2- 172.21.15.114:0/96007216 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x56255adca620 0x56255add4c70 secure :-1 s=READY pgs=836 cs=0 l=1 rev1=1 crypto rx=0x7f40a000bfd0 tx=0x7f40a0005c50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6006773a10e2cf82 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T10:36:37.987 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:36:37.982+0000 7f4097fff640 1 -- 172.21.15.114:0/96007216 <== mon.0 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f40a0013da0 con 0x56255adca620 2024-04-23T10:36:37.987 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:36:37.982+0000 7f4097fff640 1 -- 172.21.15.114:0/96007216 <== mon.0 v2:172.21.15.114:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f40a001e030 con 0x56255adca620 2024-04-23T10:36:37.987 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:36:37.982+0000 7f4097fff640 1 -- 172.21.15.114:0/96007216 <== mon.0 v2:172.21.15.114:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f40a0011aa0 con 0x56255adca620 2024-04-23T10:36:37.987 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:36:37.983+0000 7f40b7bcb740 1 -- 172.21.15.114:0/96007216 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x56255adca620 msgr2=0x56255add4c70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T10:36:37.987 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:36:37.983+0000 7f40b7bcb740 1 --2- 172.21.15.114:0/96007216 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x56255adca620 0x56255add4c70 secure :-1 s=READY pgs=836 cs=0 l=1 rev1=1 crypto rx=0x7f40a000bfd0 tx=0x7f40a0005c50 comp rx=0 tx=0).stop 2024-04-23T10:36:37.987 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:36:37.983+0000 7f40b7bcb740 1 -- 172.21.15.114:0/96007216 shutdown_connections 2024-04-23T10:36:37.987 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:36:37.983+0000 7f40b7bcb740 1 --2- 172.21.15.114:0/96007216 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x56255add5800 0x56255aded760 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:36:37.987 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:36:37.983+0000 7f40b7bcb740 1 --2- 172.21.15.114:0/96007216 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x56255adcaf40 0x56255aded220 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:36:37.987 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:36:37.983+0000 7f40b7bcb740 1 --2- 172.21.15.114:0/96007216 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x56255adca620 0x56255add4c70 unknown :-1 s=CLOSED pgs=836 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:36:37.987 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:36:37.983+0000 7f40b7bcb740 1 -- 172.21.15.114:0/96007216 >> 172.21.15.114:0/96007216 conn(0x56255adc6b30 msgr2=0x56255add23f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T10:36:37.987 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:36:37.983+0000 7f40b7bcb740 1 -- 172.21.15.114:0/96007216 shutdown_connections 2024-04-23T10:36:37.987 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:36:37.983+0000 7f40b7bcb740 1 -- 172.21.15.114:0/96007216 wait complete. 2024-04-23T10:36:37.987 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:36:37.983+0000 7f40b7bcb740 1 -- start start 2024-04-23T10:36:37.988 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:36:37.983+0000 7f40b7bcb740 1 Processor -- start 2024-04-23T10:36:37.988 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:36:37.984+0000 7f40b7bcb740 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x56255adca620 0x56255addd7c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T10:36:37.988 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:36:37.984+0000 7f40b7bcb740 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x56255adcaf40 0x56255add8b30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T10:36:37.988 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:36:37.984+0000 7f40b7bcb740 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x56255add5800 0x56255add9070 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T10:36:37.988 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:36:37.984+0000 7f40b7bcb740 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x7f40b01ac280 con 0x56255add5800 2024-04-23T10:36:37.988 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:36:37.984+0000 7f40b7bcb740 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 -- 0x7f40b01ad6f0 con 0x56255adca620 2024-04-23T10:36:37.988 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:36:37.984+0000 7f40b7bcb740 1 -- --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_getmap magic: 0 -- 0x7f40b01ac950 con 0x56255adcaf40 2024-04-23T10:36:37.988 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:36:37.984+0000 7f40b7bcb740 10 client.0 _ll_register_callbacks cb 0x7f409800ef30 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-23T10:36:37.988 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:36:37.984+0000 7f40aed9d640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x56255add5800 0x56255add9070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T10:36:37.988 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:36:37.984+0000 7f40aed9d640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x56255add5800 0x56255add9070 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.114:34514/0 (socket says 172.21.15.114:34514) 2024-04-23T10:36:37.988 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:36:37.984+0000 7f40aed9d640 1 -- 172.21.15.114:0/4010463306 learned_addr learned my addr 172.21.15.114:0/4010463306 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-04-23T10:36:37.988 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:36:37.984+0000 7f40a7fff640 1 --2- 172.21.15.114:0/4010463306 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x56255adcaf40 0x56255add8b30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T10:36:37.988 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:36:37.984+0000 7f40ae59c640 1 --2- 172.21.15.114:0/4010463306 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x56255adca620 0x56255addd7c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T10:36:37.988 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:36:37.984+0000 7f40aed9d640 1 -- 172.21.15.114:0/4010463306 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x56255adcaf40 msgr2=0x56255add8b30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T10:36:37.989 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:36:37.984+0000 7f40aed9d640 1 --2- 172.21.15.114:0/4010463306 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x56255adcaf40 0x56255add8b30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:36:37.989 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:36:37.984+0000 7f40aed9d640 1 -- 172.21.15.114:0/4010463306 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x56255adca620 msgr2=0x56255addd7c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T10:36:37.989 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:36:37.984+0000 7f40aed9d640 1 --2- 172.21.15.114:0/4010463306 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x56255adca620 0x56255addd7c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:36:37.989 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:36:37.984+0000 7f40aed9d640 1 -- 172.21.15.114:0/4010463306 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({osdmap=0}) -- 0x56255add98a0 con 0x56255add5800 2024-04-23T10:36:37.989 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:36:37.984+0000 7f40aed9d640 1 --2- 172.21.15.114:0/4010463306 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x56255add5800 0x56255add9070 secure :-1 s=READY pgs=837 cs=0 l=1 rev1=1 crypto rx=0x7f40a80ef9e0 tx=0x7f40a80bdc20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4eef5931cbd5b605 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T10:36:37.989 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:36:37.984+0000 7f4095ffb640 10 client.0 ms_handle_connect on v2:172.21.15.114:3300/0 2024-04-23T10:36:37.989 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:36:37.984+0000 7f4095ffb640 1 -- 172.21.15.114:0/4010463306 <== mon.0 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f40a80c20f0 con 0x56255add5800 2024-04-23T10:36:37.989 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:36:37.984+0000 7f4095ffb640 1 -- 172.21.15.114:0/4010463306 <== mon.0 v2:172.21.15.114:3300/0 2 ==== osd_map(102..102 src has 1..102) ==== 8629+0+0 (secure 0 0 0) 0x7f40a80ef660 con 0x56255add5800 2024-04-23T10:36:37.989 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:36:37.984+0000 7f40b7bcb740 1 -- 172.21.15.114:0/4010463306 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f40a001e030 con 0x56255add5800 2024-04-23T10:36:37.989 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:36:37.985+0000 7f4095ffb640 1 -- 172.21.15.114:0/4010463306 <== mon.0 v2:172.21.15.114:3300/0 3 ==== mon_get_version_reply(handle=1 version=37) ==== 24+0+0 (secure 0 0 0) 0x7f40a80c69b0 con 0x56255add5800 2024-04-23T10:36:37.989 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:36:37.985+0000 7f40b7bcb740 10 client.16770 fetch_fsmap learned FSMap version 37 2024-04-23T10:36:37.989 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:36:37.985+0000 7f40b7bcb740 1 -- 172.21.15.114:0/4010463306 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({config=0+,fsmap=37,monmap=4+}) -- 0x56255ae54310 con 0x56255add5800 2024-04-23T10:36:37.989 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:36:37.985+0000 7f4095ffb640 1 -- 172.21.15.114:0/4010463306 <== mon.0 v2:172.21.15.114:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f40a001e030 con 0x56255add5800 2024-04-23T10:36:37.989 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:36:37.985+0000 7f4095ffb640 1 -- 172.21.15.114:0/4010463306 <== mon.0 v2:172.21.15.114:3300/0 5 ==== fsmap(e 37) ==== 8358+0+0 (secure 0 0 0) 0x7f40a80c6520 con 0x56255add5800 2024-04-23T10:36:37.990 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:36:37.985+0000 7f40b7bcb740 10 client.16770 fetch_fsmap finished waiting for FSMap version 37 2024-04-23T10:36:37.990 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:36:37.985+0000 7f40b7bcb740 10 client.16770 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-04-23T10:36:37.990 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:36:37.985+0000 7f40b7bcb740 20 client.16770 populate_metadata read hostname 'smithi114' 2024-04-23T10:36:37.990 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:36:37.985+0000 7f40b7bcb740 1 --2- 172.21.15.114:0/4010463306 >> [v2:172.21.15.119:6838/937297340,v1:172.21.15.119:6839/937297340] conn(0x56255ae54ac0 0x56255ae56ee0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T10:36:37.990 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:36:37.985+0000 7f40b7bcb740 4 client.16770 mds_command: new command op to 24545 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-04-23T10:36:37.990 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:36:37.985+0000 7f40b7bcb740 1 -- 172.21.15.114:0/4010463306 --> [v2:172.21.15.119:6838/937297340,v1:172.21.15.119:6839/937297340] -- command(tid 0: {"prefix": "flush journal"}) -- 0x56255ad37200 con 0x56255ae54ac0 2024-04-23T10:36:37.990 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:36:37.985+0000 7f40ae59c640 1 --2- 172.21.15.114:0/4010463306 >> [v2:172.21.15.119:6838/937297340,v1:172.21.15.119:6839/937297340] conn(0x56255ae54ac0 0x56255ae56ee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T10:36:37.990 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:36:37.986+0000 7f4095ffb640 10 client.16770 ms_handle_connect on v2:172.21.15.119:6838/937297340 2024-04-23T10:36:37.990 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:36:37.986+0000 7f40ae59c640 1 --2- 172.21.15.114:0/4010463306 >> [v2:172.21.15.119:6838/937297340,v1:172.21.15.119:6839/937297340] conn(0x56255ae54ac0 0x56255ae56ee0 crc :-1 s=READY pgs=151 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=b265dc9b3157d9fc server_cookie=68eb4e6b3d271fcb in_seq=0 out_seq=0 2024-04-23T10:36:37.991 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 23 10:36:37 smithi119 ceph-mon[32608]: pgmap v313: 97 pgs: 97 active+clean; 201 MiB data, 2.5 GiB used, 1.0 TiB / 1.0 TiB avail; 616 KiB/s rd, 1.3 MiB/s wr, 112 op/s 2024-04-23T10:36:38.003 INFO:journalctl@ceph.mon.a.smithi114.stdout:Apr 23 10:36:37 smithi114 ceph-mon[29255]: pgmap v313: 97 pgs: 97 active+clean; 201 MiB data, 2.5 GiB used, 1.0 TiB / 1.0 TiB avail; 616 KiB/s rd, 1.3 MiB/s wr, 112 op/s 2024-04-23T10:36:38.028 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:36:38.028+0000 7f4095ffb640 1 -- 172.21.15.114:0/4010463306 <== mds.0 v2:172.21.15.119:6838/937297340 1 ==== command_reply(tid 0: 0 ) ==== 8+0+44 (crc 0 0 0) 0x56255ad37200 con 0x56255ae54ac0 2024-04-23T10:36:38.028 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:36:38.028+0000 7f4095ffb640 10 client.16770 handle_command_reply: tid=0 2024-04-23T10:36:38.028 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:36:38.028+0000 7f40b7bcb740 1 client.16770 shutdown 2024-04-23T10:36:38.028 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:36:38.029+0000 7f40b7bcb740 1 -- 172.21.15.114:0/4010463306 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x56255add5800 msgr2=0x56255add9070 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T10:36:38.028 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:36:38.029+0000 7f40b7bcb740 1 --2- 172.21.15.114:0/4010463306 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x56255add5800 0x56255add9070 secure :-1 s=READY pgs=837 cs=0 l=1 rev1=1 crypto rx=0x7f40a80ef9e0 tx=0x7f40a80bdc20 comp rx=0 tx=0).stop 2024-04-23T10:36:38.028 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:36:38.029+0000 7f40b7bcb740 1 -- 172.21.15.114:0/4010463306 shutdown_connections 2024-04-23T10:36:38.028 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:36:38.029+0000 7f40b7bcb740 1 --2- 172.21.15.114:0/4010463306 >> [v2:172.21.15.119:6838/937297340,v1:172.21.15.119:6839/937297340] conn(0x56255ae54ac0 0x56255ae56ee0 crc :-1 s=READY pgs=151 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:36:38.029 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:36:38.029+0000 7f40b7bcb740 1 --2- 172.21.15.114:0/4010463306 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x56255add5800 0x56255add9070 unknown :-1 s=CLOSED pgs=837 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:36:38.029 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:36:38.029+0000 7f4095ffb640 0 client.16770 ms_handle_reset on v2:172.21.15.119:6838/937297340 2024-04-23T10:36:38.029 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:36:38.029+0000 7f40b7bcb740 1 --2- 172.21.15.114:0/4010463306 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x56255adcaf40 0x56255add8b30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:36:38.029 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:36:38.029+0000 7f40b7bcb740 1 --2- 172.21.15.114:0/4010463306 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x56255adca620 0x56255addd7c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:36:38.029 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:36:38.029+0000 7f40b7bcb740 1 -- 172.21.15.114:0/4010463306 >> 172.21.15.114:0/4010463306 conn(0x56255adc6b30 msgr2=0x56255adc5420 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T10:36:38.029 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:36:38.029+0000 7f40b7bcb740 1 -- 172.21.15.114:0/4010463306 shutdown_connections 2024-04-23T10:36:38.029 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:36:38.029+0000 7f40b7bcb740 1 -- 172.21.15.114:0/4010463306 wait complete. 2024-04-23T10:36:38.029 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:36:38.029+0000 7f40b7bcb740 20 client.16770 trim_cache size 0 max 16384 2024-04-23T10:36:38.029 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stdout:{ 2024-04-23T10:36:38.029 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stdout: "message": "", 2024-04-23T10:36:38.029 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stdout: "return_code": 0 2024-04-23T10:36:38.030 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stdout:} 2024-04-23T10:36:38.030 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:36:38.029+0000 7f40b4de1640 1 -- 172.21.15.114:0/1027537292 >> [v2:172.21.15.114:6800/1395693466,v1:172.21.15.114:6801/1395693466] conn(0x7f409c0a3dc0 msgr2=0x7f409c0a6280 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T10:36:38.030 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:36:38.029+0000 7f40b4de1640 1 --2- 172.21.15.114:0/1027537292 >> [v2:172.21.15.114:6800/1395693466,v1:172.21.15.114:6801/1395693466] conn(0x7f409c0a3dc0 0x7f409c0a6280 secure :-1 s=READY pgs=129 cs=0 l=1 rev1=1 crypto rx=0x7f4098005940 tx=0x7f409800af00 comp rx=0 tx=0).stop 2024-04-23T10:36:38.031 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:36:38.029+0000 7f40b4de1640 1 -- 172.21.15.114:0/1027537292 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f40b019ce90 msgr2=0x7f40b019d340 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T10:36:38.031 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:36:38.029+0000 7f40b4de1640 1 --2- 172.21.15.114:0/1027537292 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f40b019ce90 0x7f40b019d340 secure :-1 s=READY pgs=833 cs=0 l=1 rev1=1 crypto rx=0x7f40a80098d0 tx=0x7f40a80317b0 comp rx=0 tx=0).stop 2024-04-23T10:36:38.031 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:36:38.029+0000 7f40b4de1640 1 -- 172.21.15.114:0/1027537292 shutdown_connections 2024-04-23T10:36:38.031 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:36:38.029+0000 7f40b4de1640 1 --2- 172.21.15.114:0/1027537292 >> [v2:172.21.15.114:6800/1395693466,v1:172.21.15.114:6801/1395693466] conn(0x7f409c0a3dc0 0x7f409c0a6280 unknown :-1 s=CLOSED pgs=129 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:36:38.031 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:36:38.029+0000 7f40b4de1640 1 --2- 172.21.15.114:0/1027537292 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f40b019ce90 0x7f40b019d340 unknown :-1 s=CLOSED pgs=833 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:36:38.031 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:36:38.029+0000 7f40b4de1640 1 --2- 172.21.15.114:0/1027537292 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f40b01072a0 0x7f40b0198b10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:36:38.031 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:36:38.029+0000 7f40b4de1640 1 --2- 172.21.15.114:0/1027537292 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f40b01021d0 0x7f40b01985d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:36:38.031 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:36:38.029+0000 7f40b4de1640 1 -- 172.21.15.114:0/1027537292 >> 172.21.15.114:0/1027537292 conn(0x7f40b00793e0 msgr2=0x7f40b00ffbe0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T10:36:38.031 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:36:38.030+0000 7f40b4de1640 1 -- 172.21.15.114:0/1027537292 shutdown_connections 2024-04-23T10:36:38.031 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:36:38.030+0000 7f40b4de1640 1 -- 172.21.15.114:0/1027537292 wait complete. 2024-04-23T10:36:38.938 INFO:tasks.workunit.client.0.smithi114.stdout: 1 653509 6.34 MB/sec execute 407 sec latency 2869.796 ms 2024-04-23T10:36:39.827 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 23 10:36:39 smithi204 ceph-mon[32649]: pgmap v314: 97 pgs: 97 active+clean; 201 MiB data, 2.5 GiB used, 1.0 TiB / 1.0 TiB avail; 321 KiB/s rd, 504 KiB/s wr, 55 op/s 2024-04-23T10:36:39.938 INFO:tasks.workunit.client.0.smithi114.stdout: 1 653576 6.33 MB/sec execute 408 sec latency 3110.208 ms 2024-04-23T10:36:39.992 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 23 10:36:39 smithi119 ceph-mon[32608]: pgmap v314: 97 pgs: 97 active+clean; 201 MiB data, 2.5 GiB used, 1.0 TiB / 1.0 TiB avail; 321 KiB/s rd, 504 KiB/s wr, 55 op/s 2024-04-23T10:36:40.003 INFO:journalctl@ceph.mon.a.smithi114.stdout:Apr 23 10:36:39 smithi114 ceph-mon[29255]: pgmap v314: 97 pgs: 97 active+clean; 201 MiB data, 2.5 GiB used, 1.0 TiB / 1.0 TiB avail; 321 KiB/s rd, 504 KiB/s wr, 55 op/s 2024-04-23T10:36:40.938 INFO:tasks.workunit.client.0.smithi114.stdout: 1 653576 6.31 MB/sec execute 409 sec latency 1659.267 ms 2024-04-23T10:36:41.828 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 23 10:36:41 smithi204 ceph-mon[32649]: pgmap v315: 97 pgs: 97 active+clean; 201 MiB data, 2.5 GiB used, 1.0 TiB / 1.0 TiB avail; 321 KiB/s rd, 504 KiB/s wr, 55 op/s 2024-04-23T10:36:41.938 INFO:tasks.workunit.client.0.smithi114.stdout: 1 653576 6.30 MB/sec execute 410 sec latency 2659.393 ms 2024-04-23T10:36:41.991 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 23 10:36:41 smithi119 ceph-mon[32608]: pgmap v315: 97 pgs: 97 active+clean; 201 MiB data, 2.5 GiB used, 1.0 TiB / 1.0 TiB avail; 321 KiB/s rd, 504 KiB/s wr, 55 op/s 2024-04-23T10:36:42.004 INFO:journalctl@ceph.mon.a.smithi114.stdout:Apr 23 10:36:41 smithi114 ceph-mon[29255]: pgmap v315: 97 pgs: 97 active+clean; 201 MiB data, 2.5 GiB used, 1.0 TiB / 1.0 TiB avail; 321 KiB/s rd, 504 KiB/s wr, 55 op/s 2024-04-23T10:36:42.938 INFO:tasks.workunit.client.0.smithi114.stdout: 1 653576 6.28 MB/sec execute 411 sec latency 3659.492 ms 2024-04-23T10:36:43.938 INFO:tasks.workunit.client.0.smithi114.stdout: 1 653576 6.27 MB/sec execute 412 sec latency 4659.608 ms 2024-04-23T10:36:43.992 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 23 10:36:43 smithi119 ceph-mon[32608]: pgmap v316: 97 pgs: 97 active+clean; 197 MiB data, 2.5 GiB used, 1.0 TiB / 1.0 TiB avail; 358 KiB/s rd, 551 KiB/s wr, 80 op/s 2024-04-23T10:36:44.003 INFO:journalctl@ceph.mon.a.smithi114.stdout:Apr 23 10:36:43 smithi114 ceph-mon[29255]: pgmap v316: 97 pgs: 97 active+clean; 197 MiB data, 2.5 GiB used, 1.0 TiB / 1.0 TiB avail; 358 KiB/s rd, 551 KiB/s wr, 80 op/s 2024-04-23T10:36:44.078 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 23 10:36:43 smithi204 ceph-mon[32649]: pgmap v316: 97 pgs: 97 active+clean; 197 MiB data, 2.5 GiB used, 1.0 TiB / 1.0 TiB avail; 358 KiB/s rd, 551 KiB/s wr, 80 op/s 2024-04-23T10:36:44.938 INFO:tasks.workunit.client.0.smithi114.stdout: 1 654314 6.26 MB/sec execute 413 sec latency 4902.851 ms 2024-04-23T10:36:45.938 INFO:tasks.workunit.client.0.smithi114.stdout: 1 656788 6.27 MB/sec execute 414 sec latency 15.139 ms 2024-04-23T10:36:45.991 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 23 10:36:45 smithi119 ceph-mon[32608]: pgmap v317: 97 pgs: 97 active+clean; 197 MiB data, 2.5 GiB used, 1.0 TiB / 1.0 TiB avail; 331 KiB/s rd, 512 KiB/s wr, 70 op/s 2024-04-23T10:36:46.003 INFO:journalctl@ceph.mon.a.smithi114.stdout:Apr 23 10:36:45 smithi114 ceph-mon[29255]: pgmap v317: 97 pgs: 97 active+clean; 197 MiB data, 2.5 GiB used, 1.0 TiB / 1.0 TiB avail; 331 KiB/s rd, 512 KiB/s wr, 70 op/s 2024-04-23T10:36:46.077 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 23 10:36:45 smithi204 ceph-mon[32649]: pgmap v317: 97 pgs: 97 active+clean; 197 MiB data, 2.5 GiB used, 1.0 TiB / 1.0 TiB avail; 331 KiB/s rd, 512 KiB/s wr, 70 op/s 2024-04-23T10:36:46.939 INFO:tasks.workunit.client.0.smithi114.stdout: 1 657054 6.27 MB/sec execute 415 sec latency 881.112 ms 2024-04-23T10:36:47.940 INFO:tasks.workunit.client.0.smithi114.stdout: 1 657054 6.25 MB/sec execute 416 sec latency 1881.160 ms 2024-04-23T10:36:47.991 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 23 10:36:47 smithi119 ceph-mon[32608]: pgmap v318: 97 pgs: 97 active+clean; 201 MiB data, 2.5 GiB used, 1.0 TiB / 1.0 TiB avail; 521 KiB/s rd, 840 KiB/s wr, 107 op/s 2024-04-23T10:36:48.004 INFO:journalctl@ceph.mon.a.smithi114.stdout:Apr 23 10:36:47 smithi114 ceph-mon[29255]: pgmap v318: 97 pgs: 97 active+clean; 201 MiB data, 2.5 GiB used, 1.0 TiB / 1.0 TiB avail; 521 KiB/s rd, 840 KiB/s wr, 107 op/s 2024-04-23T10:36:48.078 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 23 10:36:47 smithi204 ceph-mon[32649]: pgmap v318: 97 pgs: 97 active+clean; 201 MiB data, 2.5 GiB used, 1.0 TiB / 1.0 TiB avail; 521 KiB/s rd, 840 KiB/s wr, 107 op/s 2024-04-23T10:36:48.939 INFO:tasks.workunit.client.0.smithi114.stdout: 1 657054 6.24 MB/sec execute 417 sec latency 2881.255 ms 2024-04-23T10:36:49.939 INFO:tasks.workunit.client.0.smithi114.stdout: 1 657121 6.22 MB/sec execute 418 sec latency 3127.289 ms 2024-04-23T10:36:49.992 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 23 10:36:49 smithi119 ceph-mon[32608]: pgmap v319: 97 pgs: 97 active+clean; 201 MiB data, 2.5 GiB used, 1.0 TiB / 1.0 TiB avail; 227 KiB/s rd, 375 KiB/s wr, 61 op/s 2024-04-23T10:36:50.004 INFO:journalctl@ceph.mon.a.smithi114.stdout:Apr 23 10:36:49 smithi114 ceph-mon[29255]: pgmap v319: 97 pgs: 97 active+clean; 201 MiB data, 2.5 GiB used, 1.0 TiB / 1.0 TiB avail; 227 KiB/s rd, 375 KiB/s wr, 61 op/s 2024-04-23T10:36:50.078 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 23 10:36:49 smithi204 ceph-mon[32649]: pgmap v319: 97 pgs: 97 active+clean; 201 MiB data, 2.5 GiB used, 1.0 TiB / 1.0 TiB avail; 227 KiB/s rd, 375 KiB/s wr, 61 op/s 2024-04-23T10:36:50.939 INFO:tasks.workunit.client.0.smithi114.stdout: 1 657121 6.21 MB/sec execute 419 sec latency 1658.107 ms 2024-04-23T10:36:51.128 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:36:51.127+0000 7f04f1f51640 1 -- 172.21.15.114:0/695534114 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f04ec101b60 msgr2=0x7f04ec10dc70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T10:36:51.128 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:36:51.127+0000 7f04f1f51640 1 --2- 172.21.15.114:0/695534114 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f04ec101b60 0x7f04ec10dc70 secure :-1 s=READY pgs=838 cs=0 l=1 rev1=1 crypto rx=0x7f04e4009ca0 tx=0x7f04e402a8b0 comp rx=0 tx=0).stop 2024-04-23T10:36:51.128 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:36:51.127+0000 7f04f1f51640 1 -- 172.21.15.114:0/695534114 shutdown_connections 2024-04-23T10:36:51.128 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:36:51.127+0000 7f04f1f51640 1 --2- 172.21.15.114:0/695534114 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f04ec101b60 0x7f04ec10dc70 unknown :-1 s=CLOSED pgs=838 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:36:51.129 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:36:51.127+0000 7f04f1f51640 1 --2- 172.21.15.114:0/695534114 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f04ec1011a0 0x7f04ec101620 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:36:51.129 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:36:51.127+0000 7f04f1f51640 1 --2- 172.21.15.114:0/695534114 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f04ec106fb0 0x7f04ec107390 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:36:51.129 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:36:51.127+0000 7f04f1f51640 1 -- 172.21.15.114:0/695534114 >> 172.21.15.114:0/695534114 conn(0x7f04ec076c50 msgr2=0x7f04ec077060 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T10:36:51.129 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:36:51.127+0000 7f04f1f51640 1 -- 172.21.15.114:0/695534114 shutdown_connections 2024-04-23T10:36:51.129 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:36:51.127+0000 7f04f1f51640 1 -- 172.21.15.114:0/695534114 wait complete. 2024-04-23T10:36:51.129 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:36:51.127+0000 7f04f1f51640 1 Processor -- start 2024-04-23T10:36:51.129 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:36:51.128+0000 7f04f1f51640 1 -- start start 2024-04-23T10:36:51.129 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:36:51.128+0000 7f04f1f51640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f04ec1011a0 0x7f04ec1993e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T10:36:51.129 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:36:51.128+0000 7f04f1f51640 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f04ec101b60 0x7f04ec199920 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T10:36:51.129 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:36:51.128+0000 7f04f1f51640 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f04ec106fb0 0x7f04ec1a4460 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T10:36:51.129 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:36:51.128+0000 7f04f1f51640 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x7f04ec111080 con 0x7f04ec1011a0 2024-04-23T10:36:51.129 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:36:51.128+0000 7f04f1f51640 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 -- 0x7f04ec110f00 con 0x7f04ec101b60 2024-04-23T10:36:51.129 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:36:51.128+0000 7f04f1f51640 1 -- --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_getmap magic: 0 -- 0x7f04ec111200 con 0x7f04ec106fb0 2024-04-23T10:36:51.130 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:36:51.128+0000 7f04eb7fe640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f04ec1011a0 0x7f04ec1993e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T10:36:51.130 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:36:51.128+0000 7f04eb7fe640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f04ec1011a0 0x7f04ec1993e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.114:37502/0 (socket says 172.21.15.114:37502) 2024-04-23T10:36:51.130 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:36:51.128+0000 7f04eb7fe640 1 -- 172.21.15.114:0/3330750689 learned_addr learned my addr 172.21.15.114:0/3330750689 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-04-23T10:36:51.130 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:36:51.128+0000 7f04e3fff640 1 --2- 172.21.15.114:0/3330750689 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f04ec101b60 0x7f04ec199920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T10:36:51.130 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:36:51.128+0000 7f04ebfff640 1 --2- 172.21.15.114:0/3330750689 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f04ec106fb0 0x7f04ec1a4460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T10:36:51.130 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:36:51.128+0000 7f04eb7fe640 1 -- 172.21.15.114:0/3330750689 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f04ec106fb0 msgr2=0x7f04ec1a4460 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T10:36:51.130 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:36:51.128+0000 7f04eb7fe640 1 --2- 172.21.15.114:0/3330750689 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f04ec106fb0 0x7f04ec1a4460 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:36:51.130 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:36:51.128+0000 7f04eb7fe640 1 -- 172.21.15.114:0/3330750689 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f04ec101b60 msgr2=0x7f04ec199920 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T10:36:51.130 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:36:51.128+0000 7f04eb7fe640 1 --2- 172.21.15.114:0/3330750689 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f04ec101b60 0x7f04ec199920 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:36:51.130 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:36:51.128+0000 7f04eb7fe640 1 -- 172.21.15.114:0/3330750689 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f04ec1a49a0 con 0x7f04ec1011a0 2024-04-23T10:36:51.130 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:36:51.128+0000 7f04eb7fe640 1 --2- 172.21.15.114:0/3330750689 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f04ec1011a0 0x7f04ec1993e0 secure :-1 s=READY pgs=839 cs=0 l=1 rev1=1 crypto rx=0x7f04dc00e9f0 tx=0x7f04dc00edf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T10:36:51.130 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:36:51.128+0000 7f04e97fa640 1 -- 172.21.15.114:0/3330750689 <== mon.0 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f04dc00cd80 con 0x7f04ec1011a0 2024-04-23T10:36:51.130 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:36:51.129+0000 7f04e97fa640 1 -- 172.21.15.114:0/3330750689 <== mon.0 v2:172.21.15.114:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f04dc0054d0 con 0x7f04ec1011a0 2024-04-23T10:36:51.131 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:36:51.129+0000 7f04f1f51640 1 -- 172.21.15.114:0/3330750689 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f04ec1a4b70 con 0x7f04ec1011a0 2024-04-23T10:36:51.131 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:36:51.129+0000 7f04e97fa640 1 -- 172.21.15.114:0/3330750689 <== mon.0 v2:172.21.15.114:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f04dc010690 con 0x7f04ec1011a0 2024-04-23T10:36:51.131 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:36:51.129+0000 7f04f1f51640 1 -- 172.21.15.114:0/3330750689 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f04ec1a5040 con 0x7f04ec1011a0 2024-04-23T10:36:51.137 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:36:51.131+0000 7f04e97fa640 1 -- 172.21.15.114:0/3330750689 <== mon.0 v2:172.21.15.114:3300/0 4 ==== mgrmap(e 26) ==== 138778+0+0 (secure 0 0 0) 0x7f04dc004e90 con 0x7f04ec1011a0 2024-04-23T10:36:51.137 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:36:51.131+0000 7f04f4c60740 1 Processor -- start 2024-04-23T10:36:51.137 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:36:51.131+0000 7f04f4c60740 1 -- start start 2024-04-23T10:36:51.137 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:36:51.131+0000 7f04f4c60740 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x55ded7c20a30 0x55ded7c20e10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T10:36:51.137 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:36:51.131+0000 7f04f4c60740 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x55ded7c21350 0x55ded7c2b6d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T10:36:51.137 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:36:51.131+0000 7f04f4c60740 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55ded7c2bc10 0x55ded7c2e000 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T10:36:51.137 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:36:51.131+0000 7f04e97fa640 1 --2- 172.21.15.114:0/3330750689 >> [v2:172.21.15.114:6800/1395693466,v1:172.21.15.114:6801/1395693466] conn(0x7f04d80a3dc0 0x7f04d80a6280 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T10:36:51.137 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:36:51.131+0000 7f04f4c60740 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x55ded7b81970 con 0x55ded7c2bc10 2024-04-23T10:36:51.137 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:36:51.131+0000 7f04f4c60740 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 -- 0x55ded7b1e4a0 con 0x55ded7c20a30 2024-04-23T10:36:51.137 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:36:51.131+0000 7f04f4c60740 1 -- --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_getmap magic: 0 -- 0x55ded7b7d930 con 0x55ded7c21350 2024-04-23T10:36:51.138 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:36:51.131+0000 7f04ebfff640 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x55ded7c20a30 0x55ded7c20e10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T10:36:51.138 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:36:51.131+0000 7f04e3fff640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55ded7c2bc10 0x55ded7c2e000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T10:36:51.138 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:36:51.131+0000 7f04ebfff640 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x55ded7c20a30 0x55ded7c20e10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.119:3300/0 says I am v2:172.21.15.114:36680/0 (socket says 172.21.15.114:36680) 2024-04-23T10:36:51.138 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:36:51.131+0000 7f04e3fff640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55ded7c2bc10 0x55ded7c2e000 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.114:37514/0 (socket says 172.21.15.114:37514) 2024-04-23T10:36:51.138 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:36:51.132+0000 7f04ebfff640 1 -- 172.21.15.114:0/16131557 learned_addr learned my addr 172.21.15.114:0/16131557 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-04-23T10:36:51.138 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:36:51.132+0000 7f04ebfff640 1 --2- 172.21.15.114:0/3330750689 >> [v2:172.21.15.114:6800/1395693466,v1:172.21.15.114:6801/1395693466] conn(0x7f04d80a3dc0 0x7f04d80a6280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T10:36:51.138 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:36:51.132+0000 7f04e97fa640 1 -- 172.21.15.114:0/3330750689 <== mon.0 v2:172.21.15.114:3300/0 5 ==== osd_map(102..102 src has 1..102) ==== 8629+0+0 (secure 0 0 0) 0x7f04dc0d0d30 con 0x7f04ec1011a0 2024-04-23T10:36:51.138 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:36:51.132+0000 7f04eb7fe640 1 --2- 172.21.15.114:0/16131557 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x55ded7c21350 0x55ded7c2b6d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T10:36:51.138 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:36:51.132+0000 7f04e3fff640 1 -- 172.21.15.114:0/16131557 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x55ded7c21350 msgr2=0x55ded7c2b6d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T10:36:51.138 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:36:51.132+0000 7f04e3fff640 1 --2- 172.21.15.114:0/16131557 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x55ded7c21350 0x55ded7c2b6d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:36:51.138 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:36:51.132+0000 7f04e3fff640 1 -- 172.21.15.114:0/16131557 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x55ded7c20a30 msgr2=0x55ded7c20e10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T10:36:51.138 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:36:51.132+0000 7f04e3fff640 1 --2- 172.21.15.114:0/16131557 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x55ded7c20a30 0x55ded7c20e10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:36:51.138 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:36:51.132+0000 7f04e3fff640 1 -- 172.21.15.114:0/16131557 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55ded7894210 con 0x55ded7c2bc10 2024-04-23T10:36:51.139 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:36:51.132+0000 7f04e3fff640 1 --2- 172.21.15.114:0/16131557 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55ded7c2bc10 0x55ded7c2e000 secure :-1 s=READY pgs=840 cs=0 l=1 rev1=1 crypto rx=0x7f04d400bbb0 tx=0x7f04d400f730 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5104317810880e63 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T10:36:51.139 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:36:51.132+0000 7f04ebfff640 1 --2- 172.21.15.114:0/3330750689 >> [v2:172.21.15.114:6800/1395693466,v1:172.21.15.114:6801/1395693466] conn(0x7f04d80a3dc0 0x7f04d80a6280 secure :-1 s=READY pgs=130 cs=0 l=1 rev1=1 crypto rx=0x7f04e4004af0 tx=0x7f04e402a8b0 comp rx=0 tx=0).ready entity=mgr.15291 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T10:36:51.139 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:36:51.132+0000 7f04e1ffb640 1 -- 172.21.15.114:0/16131557 <== mon.0 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f04d4015070 con 0x55ded7c2bc10 2024-04-23T10:36:51.139 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:36:51.132+0000 7f04e1ffb640 1 -- 172.21.15.114:0/16131557 <== mon.0 v2:172.21.15.114:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f04d40035f0 con 0x55ded7c2bc10 2024-04-23T10:36:51.139 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:36:51.132+0000 7f04e1ffb640 1 -- 172.21.15.114:0/16131557 <== mon.0 v2:172.21.15.114:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f04d40116a0 con 0x55ded7c2bc10 2024-04-23T10:36:51.139 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:36:51.133+0000 7f04f4c60740 1 -- 172.21.15.114:0/16131557 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55ded7c2bc10 msgr2=0x55ded7c2e000 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T10:36:51.139 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:36:51.133+0000 7f04f4c60740 1 --2- 172.21.15.114:0/16131557 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55ded7c2bc10 0x55ded7c2e000 secure :-1 s=READY pgs=840 cs=0 l=1 rev1=1 crypto rx=0x7f04d400bbb0 tx=0x7f04d400f730 comp rx=0 tx=0).stop 2024-04-23T10:36:51.139 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:36:51.133+0000 7f04f4c60740 1 -- 172.21.15.114:0/16131557 shutdown_connections 2024-04-23T10:36:51.139 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:36:51.133+0000 7f04f4c60740 1 --2- 172.21.15.114:0/16131557 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55ded7c2bc10 0x55ded7c2e000 unknown :-1 s=CLOSED pgs=840 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:36:51.139 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:36:51.133+0000 7f04f4c60740 1 --2- 172.21.15.114:0/16131557 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x55ded7c21350 0x55ded7c2b6d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:36:51.139 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:36:51.133+0000 7f04f4c60740 1 --2- 172.21.15.114:0/16131557 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x55ded7c20a30 0x55ded7c20e10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:36:51.139 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:36:51.133+0000 7f04f4c60740 1 -- 172.21.15.114:0/16131557 >> 172.21.15.114:0/16131557 conn(0x55ded7c1cf40 msgr2=0x55ded7c1d320 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T10:36:51.139 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:36:51.133+0000 7f04f4c60740 1 -- 172.21.15.114:0/16131557 shutdown_connections 2024-04-23T10:36:51.139 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:36:51.133+0000 7f04f4c60740 1 -- 172.21.15.114:0/16131557 wait complete. 2024-04-23T10:36:51.139 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:36:51.133+0000 7f04f4c60740 1 -- start start 2024-04-23T10:36:51.140 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:36:51.134+0000 7f04f4c60740 1 Processor -- start 2024-04-23T10:36:51.140 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:36:51.134+0000 7f04f4c60740 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55ded7c20a30 0x55ded7c2b520 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T10:36:51.140 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:36:51.134+0000 7f04f4c60740 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x55ded7c21350 0x55ded7c28880 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T10:36:51.140 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:36:51.134+0000 7f04f4c60740 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x55ded7c2bc10 0x55ded7c28e00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T10:36:51.140 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:36:51.134+0000 7f04f4c60740 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x55ded7c2e840 con 0x55ded7c20a30 2024-04-23T10:36:51.140 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:36:51.134+0000 7f04f4c60740 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 -- 0x55ded7b1c0b0 con 0x55ded7c21350 2024-04-23T10:36:51.140 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:36:51.134+0000 7f04f4c60740 1 -- --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_getmap magic: 0 -- 0x55ded7c2e9c0 con 0x55ded7c2bc10 2024-04-23T10:36:51.140 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:36:51.134+0000 7f04f4c60740 10 client.0 _ll_register_callbacks cb 0x55ded7b09d30 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-23T10:36:51.140 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:36:51.134+0000 7f04eb7fe640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55ded7c20a30 0x55ded7c2b520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T10:36:51.140 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:36:51.134+0000 7f04eb7fe640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55ded7c20a30 0x55ded7c2b520 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.114:37530/0 (socket says 172.21.15.114:37530) 2024-04-23T10:36:51.140 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:36:51.134+0000 7f04eb7fe640 1 -- 172.21.15.114:0/2431107955 learned_addr learned my addr 172.21.15.114:0/2431107955 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-04-23T10:36:51.140 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:36:51.134+0000 7f04e3fff640 1 --2- 172.21.15.114:0/2431107955 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x55ded7c21350 0x55ded7c28880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T10:36:51.140 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:36:51.134+0000 7f04ebfff640 1 --2- 172.21.15.114:0/2431107955 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x55ded7c2bc10 0x55ded7c28e00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T10:36:51.141 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:36:51.134+0000 7f04eb7fe640 1 -- 172.21.15.114:0/2431107955 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x55ded7c2bc10 msgr2=0x55ded7c28e00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T10:36:51.141 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:36:51.134+0000 7f04eb7fe640 1 --2- 172.21.15.114:0/2431107955 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x55ded7c2bc10 0x55ded7c28e00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:36:51.141 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:36:51.134+0000 7f04eb7fe640 1 -- 172.21.15.114:0/2431107955 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x55ded7c21350 msgr2=0x55ded7c28880 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T10:36:51.141 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:36:51.134+0000 7f04eb7fe640 1 --2- 172.21.15.114:0/2431107955 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x55ded7c21350 0x55ded7c28880 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:36:51.141 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:36:51.134+0000 7f04eb7fe640 1 -- 172.21.15.114:0/2431107955 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55ded7960620 con 0x55ded7c20a30 2024-04-23T10:36:51.141 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:36:51.134+0000 7f04eb7fe640 1 --2- 172.21.15.114:0/2431107955 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55ded7c20a30 0x55ded7c2b520 secure :-1 s=READY pgs=841 cs=0 l=1 rev1=1 crypto rx=0x7f04dc09cbf0 tx=0x7f04dc09b170 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7dc653fd5850c909 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T10:36:51.141 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:36:51.134+0000 7f04caffd640 10 client.0 ms_handle_connect on v2:172.21.15.114:3300/0 2024-04-23T10:36:51.141 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:36:51.135+0000 7f04caffd640 1 -- 172.21.15.114:0/2431107955 <== mon.0 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f04dc0b1020 con 0x55ded7c20a30 2024-04-23T10:36:51.141 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:36:51.135+0000 7f04caffd640 1 -- 172.21.15.114:0/2431107955 <== mon.0 v2:172.21.15.114:3300/0 2 ==== osd_map(102..102 src has 1..102) ==== 8629+0+0 (secure 0 0 0) 0x7f04dc0992d0 con 0x55ded7c20a30 2024-04-23T10:36:51.141 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:36:51.135+0000 7f04f4c60740 1 -- 172.21.15.114:0/2431107955 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f04d40035f0 con 0x55ded7c20a30 2024-04-23T10:36:51.141 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:36:51.135+0000 7f04caffd640 1 -- 172.21.15.114:0/2431107955 <== mon.0 v2:172.21.15.114:3300/0 3 ==== mon_get_version_reply(handle=1 version=37) ==== 24+0+0 (secure 0 0 0) 0x7f04dc09cc60 con 0x55ded7c20a30 2024-04-23T10:36:51.141 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:36:51.135+0000 7f04f4c60740 10 client.16782 fetch_fsmap learned FSMap version 37 2024-04-23T10:36:51.141 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:36:51.135+0000 7f04f4c60740 1 -- 172.21.15.114:0/2431107955 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({config=0+,fsmap=37,monmap=4+}) -- 0x55ded7c3a790 con 0x55ded7c20a30 2024-04-23T10:36:51.141 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:36:51.135+0000 7f04caffd640 1 -- 172.21.15.114:0/2431107955 <== mon.0 v2:172.21.15.114:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f04d40035f0 con 0x55ded7c20a30 2024-04-23T10:36:51.142 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:36:51.135+0000 7f04caffd640 1 -- 172.21.15.114:0/2431107955 <== mon.0 v2:172.21.15.114:3300/0 5 ==== fsmap(e 37) ==== 8358+0+0 (secure 0 0 0) 0x7f04dc0ad050 con 0x55ded7c20a30 2024-04-23T10:36:51.142 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:36:51.135+0000 7f04f4c60740 10 client.16782 fetch_fsmap finished waiting for FSMap version 37 2024-04-23T10:36:51.142 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:36:51.135+0000 7f04f4c60740 10 client.16782 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-04-23T10:36:51.142 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:36:51.135+0000 7f04f4c60740 20 client.16782 populate_metadata read hostname 'smithi114' 2024-04-23T10:36:51.142 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:36:51.135+0000 7f04f4c60740 1 --2- 172.21.15.114:0/2431107955 >> [v2:172.21.15.119:6838/937297340,v1:172.21.15.119:6839/937297340] conn(0x55ded7c3bc50 0x55ded7c3e080 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T10:36:51.142 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:36:51.135+0000 7f04f4c60740 4 client.16782 mds_command: new command op to 24545 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-23T10:36:51.142 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:36:51.135+0000 7f04f4c60740 1 -- 172.21.15.114:0/2431107955 --> [v2:172.21.15.119:6838/937297340,v1:172.21.15.119:6839/937297340] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55ded7b8f390 con 0x55ded7c3bc50 2024-04-23T10:36:51.142 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:36:51.136+0000 7f04e3fff640 1 --2- 172.21.15.114:0/2431107955 >> [v2:172.21.15.119:6838/937297340,v1:172.21.15.119:6839/937297340] conn(0x55ded7c3bc50 0x55ded7c3e080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T10:36:51.142 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:36:51.136+0000 7f04caffd640 10 client.16782 ms_handle_connect on v2:172.21.15.119:6838/937297340 2024-04-23T10:36:51.142 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:36:51.136+0000 7f04e3fff640 1 --2- 172.21.15.114:0/2431107955 >> [v2:172.21.15.119:6838/937297340,v1:172.21.15.119:6839/937297340] conn(0x55ded7c3bc50 0x55ded7c3e080 crc :-1 s=READY pgs=152 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=3d082a1c51e983a4 server_cookie=37b495bb02d2b57e in_seq=0 out_seq=0 2024-04-23T10:36:51.142 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:36:51.137+0000 7f04caffd640 1 -- 172.21.15.114:0/2431107955 <== mds.0 v2:172.21.15.119:6838/937297340 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x55ded7b8f390 con 0x55ded7c3bc50 2024-04-23T10:36:51.142 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:36:51.137+0000 7f04caffd640 10 client.16782 handle_command_reply: tid=0 2024-04-23T10:36:51.142 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:36:51.137+0000 7f04f4c60740 1 client.16782 shutdown 2024-04-23T10:36:51.142 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:36:51.137+0000 7f04f4c60740 1 -- 172.21.15.114:0/2431107955 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55ded7c20a30 msgr2=0x55ded7c2b520 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T10:36:51.143 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:36:51.137+0000 7f04f4c60740 1 --2- 172.21.15.114:0/2431107955 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55ded7c20a30 0x55ded7c2b520 secure :-1 s=READY pgs=841 cs=0 l=1 rev1=1 crypto rx=0x7f04dc09cbf0 tx=0x7f04dc09b170 comp rx=0 tx=0).stop 2024-04-23T10:36:51.143 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:36:51.137+0000 7f04f4c60740 1 -- 172.21.15.114:0/2431107955 shutdown_connections 2024-04-23T10:36:51.143 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:36:51.137+0000 7f04f4c60740 1 --2- 172.21.15.114:0/2431107955 >> [v2:172.21.15.119:6838/937297340,v1:172.21.15.119:6839/937297340] conn(0x55ded7c3bc50 0x55ded7c3e080 crc :-1 s=READY pgs=152 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:36:51.143 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:36:51.137+0000 7f04f4c60740 1 --2- 172.21.15.114:0/2431107955 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x55ded7c2bc10 0x55ded7c28e00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:36:51.143 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:36:51.137+0000 7f04f4c60740 1 --2- 172.21.15.114:0/2431107955 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x55ded7c21350 0x55ded7c28880 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:36:51.143 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:36:51.137+0000 7f04caffd640 0 client.16782 ms_handle_reset on v2:172.21.15.119:6838/937297340 2024-04-23T10:36:51.143 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:36:51.137+0000 7f04f4c60740 1 --2- 172.21.15.114:0/2431107955 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55ded7c20a30 0x55ded7c2b520 unknown :-1 s=CLOSED pgs=841 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:36:51.143 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:36:51.137+0000 7f04f4c60740 1 -- 172.21.15.114:0/2431107955 >> 172.21.15.114:0/2431107955 conn(0x55ded7c1cf40 msgr2=0x55ded7c17cb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T10:36:51.143 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:36:51.137+0000 7f04f4c60740 1 -- 172.21.15.114:0/2431107955 shutdown_connections 2024-04-23T10:36:51.143 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:36:51.137+0000 7f04f4c60740 1 -- 172.21.15.114:0/2431107955 wait complete. 2024-04-23T10:36:51.143 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:36:51.137+0000 7f04f4c60740 20 client.16782 trim_cache size 0 max 16384 2024-04-23T10:36:51.155 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:36:51.152+0000 7f04f4c60740 1 Processor -- start 2024-04-23T10:36:51.155 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:36:51.152+0000 7f04f4c60740 1 -- start start 2024-04-23T10:36:51.155 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:36:51.152+0000 7f04f4c60740 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x55ded7c20a30 0x55ded7c37700 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T10:36:51.156 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:36:51.152+0000 7f04f4c60740 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x55ded7c21350 0x55ded7c17b30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T10:36:51.156 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:36:51.152+0000 7f04f4c60740 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55ded7c2bc10 0x55ded7c43990 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T10:36:51.156 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:36:51.152+0000 7f04f4c60740 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x7f04ec1ae3e0 con 0x55ded7c2bc10 2024-04-23T10:36:51.156 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:36:51.152+0000 7f04f4c60740 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 -- 0x7f04ec1abbf0 con 0x55ded7c20a30 2024-04-23T10:36:51.156 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:36:51.152+0000 7f04f4c60740 1 -- --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_getmap magic: 0 -- 0x7f04ec1a9a60 con 0x55ded7c21350 2024-04-23T10:36:51.156 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:36:51.152+0000 7f04ebfff640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55ded7c2bc10 0x55ded7c43990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T10:36:51.156 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:36:51.152+0000 7f04ebfff640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55ded7c2bc10 0x55ded7c43990 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.114:37532/0 (socket says 172.21.15.114:37532) 2024-04-23T10:36:51.156 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:36:51.152+0000 7f04ebfff640 1 -- 172.21.15.114:0/107822649 learned_addr learned my addr 172.21.15.114:0/107822649 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-04-23T10:36:51.156 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:36:51.152+0000 7f04eb7fe640 1 --2- 172.21.15.114:0/107822649 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x55ded7c20a30 0x55ded7c37700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T10:36:51.156 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:36:51.152+0000 7f04e3fff640 1 --2- 172.21.15.114:0/107822649 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x55ded7c21350 0x55ded7c17b30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T10:36:51.156 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:36:51.152+0000 7f04ebfff640 1 -- 172.21.15.114:0/107822649 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x55ded7c21350 msgr2=0x55ded7c17b30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T10:36:51.156 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:36:51.152+0000 7f04ebfff640 1 --2- 172.21.15.114:0/107822649 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x55ded7c21350 0x55ded7c17b30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:36:51.156 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:36:51.152+0000 7f04ebfff640 1 -- 172.21.15.114:0/107822649 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x55ded7c20a30 msgr2=0x55ded7c37700 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T10:36:51.156 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:36:51.152+0000 7f04ebfff640 1 --2- 172.21.15.114:0/107822649 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x55ded7c20a30 0x55ded7c37700 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:36:51.157 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:36:51.153+0000 7f04ebfff640 1 -- 172.21.15.114:0/107822649 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55ded7c21180 con 0x55ded7c2bc10 2024-04-23T10:36:51.157 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:36:51.153+0000 7f04eb7fe640 1 --2- 172.21.15.114:0/107822649 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x55ded7c20a30 0x55ded7c37700 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-23T10:36:51.157 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:36:51.153+0000 7f04ebfff640 1 --2- 172.21.15.114:0/107822649 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55ded7c2bc10 0x55ded7c43990 secure :-1 s=READY pgs=842 cs=0 l=1 rev1=1 crypto rx=0x7f04e4009430 tx=0x7f04e4039000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=61ba72bc289d0ad5 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T10:36:51.157 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:36:51.153+0000 7f04e17fa640 1 -- 172.21.15.114:0/107822649 <== mon.0 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f04e4003a30 con 0x55ded7c2bc10 2024-04-23T10:36:51.157 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:36:51.153+0000 7f04e17fa640 1 -- 172.21.15.114:0/107822649 <== mon.0 v2:172.21.15.114:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f04e4036040 con 0x55ded7c2bc10 2024-04-23T10:36:51.157 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:36:51.153+0000 7f04e17fa640 1 -- 172.21.15.114:0/107822649 <== mon.0 v2:172.21.15.114:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f04e404f6a0 con 0x55ded7c2bc10 2024-04-23T10:36:51.157 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:36:51.153+0000 7f04f4c60740 1 -- 172.21.15.114:0/107822649 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55ded7c2bc10 msgr2=0x55ded7c43990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T10:36:51.157 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:36:51.153+0000 7f04f4c60740 1 --2- 172.21.15.114:0/107822649 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55ded7c2bc10 0x55ded7c43990 secure :-1 s=READY pgs=842 cs=0 l=1 rev1=1 crypto rx=0x7f04e4009430 tx=0x7f04e4039000 comp rx=0 tx=0).stop 2024-04-23T10:36:51.157 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:36:51.154+0000 7f04f4c60740 1 -- 172.21.15.114:0/107822649 shutdown_connections 2024-04-23T10:36:51.157 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:36:51.154+0000 7f04f4c60740 1 --2- 172.21.15.114:0/107822649 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55ded7c2bc10 0x55ded7c43990 unknown :-1 s=CLOSED pgs=842 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:36:51.157 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:36:51.154+0000 7f04f4c60740 1 --2- 172.21.15.114:0/107822649 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x55ded7c21350 0x55ded7c17b30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:36:51.157 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:36:51.154+0000 7f04f4c60740 1 --2- 172.21.15.114:0/107822649 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x55ded7c20a30 0x55ded7c37700 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:36:51.157 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:36:51.154+0000 7f04f4c60740 1 -- 172.21.15.114:0/107822649 >> 172.21.15.114:0/107822649 conn(0x55ded7c1cf40 msgr2=0x55ded7c17420 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T10:36:51.157 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:36:51.154+0000 7f04f4c60740 1 -- 172.21.15.114:0/107822649 shutdown_connections 2024-04-23T10:36:51.157 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:36:51.154+0000 7f04f4c60740 1 -- 172.21.15.114:0/107822649 wait complete. 2024-04-23T10:36:51.157 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:36:51.154+0000 7f04f4c60740 1 -- start start 2024-04-23T10:36:51.158 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:36:51.154+0000 7f04f4c60740 1 Processor -- start 2024-04-23T10:36:51.158 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:36:51.154+0000 7f04f4c60740 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x55ded7c20a30 0x55ded7c33e10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T10:36:51.158 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:36:51.154+0000 7f04f4c60740 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x55ded7c21350 0x55ded7c311a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T10:36:51.158 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:36:51.154+0000 7f04f4c60740 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55ded7c2bc10 0x55ded7c316e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T10:36:51.158 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:36:51.154+0000 7f04f4c60740 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x7f04ec1aa0b0 con 0x55ded7c2bc10 2024-04-23T10:36:51.158 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:36:51.154+0000 7f04f4c60740 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 -- 0x7f04ec1ab520 con 0x55ded7c20a30 2024-04-23T10:36:51.158 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:36:51.154+0000 7f04f4c60740 1 -- --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_getmap magic: 0 -- 0x7f04ec1aa780 con 0x55ded7c21350 2024-04-23T10:36:51.158 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:36:51.154+0000 7f04f4c60740 10 client.0 _ll_register_callbacks cb 0x7f04dc098fb0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-23T10:36:51.158 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:36:51.154+0000 7f04ebfff640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55ded7c2bc10 0x55ded7c316e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T10:36:51.158 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:36:51.154+0000 7f04ebfff640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55ded7c2bc10 0x55ded7c316e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.114:37546/0 (socket says 172.21.15.114:37546) 2024-04-23T10:36:51.158 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:36:51.154+0000 7f04ebfff640 1 -- 172.21.15.114:0/1368202803 learned_addr learned my addr 172.21.15.114:0/1368202803 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-04-23T10:36:51.158 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:36:51.155+0000 7f04eb7fe640 1 --2- 172.21.15.114:0/1368202803 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x55ded7c20a30 0x55ded7c33e10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T10:36:51.158 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:36:51.155+0000 7f04e3fff640 1 --2- 172.21.15.114:0/1368202803 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x55ded7c21350 0x55ded7c311a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T10:36:51.158 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:36:51.155+0000 7f04ebfff640 1 -- 172.21.15.114:0/1368202803 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x55ded7c21350 msgr2=0x55ded7c311a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T10:36:51.159 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:36:51.155+0000 7f04ebfff640 1 --2- 172.21.15.114:0/1368202803 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x55ded7c21350 0x55ded7c311a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:36:51.159 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:36:51.155+0000 7f04ebfff640 1 -- 172.21.15.114:0/1368202803 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x55ded7c20a30 msgr2=0x55ded7c33e10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T10:36:51.159 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:36:51.155+0000 7f04ebfff640 1 --2- 172.21.15.114:0/1368202803 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x55ded7c20a30 0x55ded7c33e10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:36:51.159 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:36:51.155+0000 7f04ebfff640 1 -- 172.21.15.114:0/1368202803 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55ded7c31d20 con 0x55ded7c2bc10 2024-04-23T10:36:51.159 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:36:51.155+0000 7f04ebfff640 1 --2- 172.21.15.114:0/1368202803 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55ded7c2bc10 0x55ded7c316e0 secure :-1 s=READY pgs=843 cs=0 l=1 rev1=1 crypto rx=0x7f04e4009a60 tx=0x7f04e4009360 comp rx=0 tx=0).ready entity=mon.0 client_cookie=fb3ce2adc79c6424 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T10:36:51.159 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:36:51.155+0000 7f04caffd640 10 client.0 ms_handle_connect on v2:172.21.15.114:3300/0 2024-04-23T10:36:51.159 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:36:51.155+0000 7f04caffd640 1 -- 172.21.15.114:0/1368202803 <== mon.0 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f04e4047040 con 0x55ded7c2bc10 2024-04-23T10:36:51.159 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:36:51.155+0000 7f04f4c60740 1 -- 172.21.15.114:0/1368202803 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f04e4036040 con 0x55ded7c2bc10 2024-04-23T10:36:51.159 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:36:51.155+0000 7f04caffd640 1 -- 172.21.15.114:0/1368202803 <== mon.0 v2:172.21.15.114:3300/0 2 ==== osd_map(102..102 src has 1..102) ==== 8629+0+0 (secure 0 0 0) 0x7f04e4004450 con 0x55ded7c2bc10 2024-04-23T10:36:51.159 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:36:51.155+0000 7f04caffd640 1 -- 172.21.15.114:0/1368202803 <== mon.0 v2:172.21.15.114:3300/0 3 ==== mon_get_version_reply(handle=1 version=37) ==== 24+0+0 (secure 0 0 0) 0x7f04e4060470 con 0x55ded7c2bc10 2024-04-23T10:36:51.159 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:36:51.155+0000 7f04f4c60740 10 client.16788 fetch_fsmap learned FSMap version 37 2024-04-23T10:36:51.159 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:36:51.155+0000 7f04f4c60740 1 -- 172.21.15.114:0/1368202803 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({config=0+,fsmap=37,monmap=4+}) -- 0x55ded7ca2050 con 0x55ded7c2bc10 2024-04-23T10:36:51.159 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:36:51.155+0000 7f04caffd640 1 -- 172.21.15.114:0/1368202803 <== mon.0 v2:172.21.15.114:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f04e4036040 con 0x55ded7c2bc10 2024-04-23T10:36:51.159 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:36:51.156+0000 7f04caffd640 1 -- 172.21.15.114:0/1368202803 <== mon.0 v2:172.21.15.114:3300/0 5 ==== fsmap(e 37) ==== 8358+0+0 (secure 0 0 0) 0x7f04e404f440 con 0x55ded7c2bc10 2024-04-23T10:36:51.160 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:36:51.156+0000 7f04f4c60740 10 client.16788 fetch_fsmap finished waiting for FSMap version 37 2024-04-23T10:36:51.160 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:36:51.156+0000 7f04f4c60740 10 client.16788 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-04-23T10:36:51.160 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:36:51.156+0000 7f04f4c60740 20 client.16788 populate_metadata read hostname 'smithi114' 2024-04-23T10:36:51.160 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:36:51.156+0000 7f04f4c60740 1 --2- 172.21.15.114:0/1368202803 >> [v2:172.21.15.119:6838/937297340,v1:172.21.15.119:6839/937297340] conn(0x55ded7ca2790 0x55ded7ca4bb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T10:36:51.160 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:36:51.156+0000 7f04f4c60740 4 client.16788 mds_command: new command op to 24545 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-04-23T10:36:51.160 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:36:51.156+0000 7f04f4c60740 1 -- 172.21.15.114:0/1368202803 --> [v2:172.21.15.119:6838/937297340,v1:172.21.15.119:6839/937297340] -- command(tid 0: {"prefix": "flush journal"}) -- 0x55ded7b89200 con 0x55ded7ca2790 2024-04-23T10:36:51.160 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:36:51.156+0000 7f04eb7fe640 1 --2- 172.21.15.114:0/1368202803 >> [v2:172.21.15.119:6838/937297340,v1:172.21.15.119:6839/937297340] conn(0x55ded7ca2790 0x55ded7ca4bb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T10:36:51.160 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:36:51.156+0000 7f04caffd640 10 client.16788 ms_handle_connect on v2:172.21.15.119:6838/937297340 2024-04-23T10:36:51.160 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:36:51.156+0000 7f04eb7fe640 1 --2- 172.21.15.114:0/1368202803 >> [v2:172.21.15.119:6838/937297340,v1:172.21.15.119:6839/937297340] conn(0x55ded7ca2790 0x55ded7ca4bb0 crc :-1 s=READY pgs=153 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=3d1da712d711c67b server_cookie=40289c0e53998e74 in_seq=0 out_seq=0 2024-04-23T10:36:51.195 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:36:51.196+0000 7f04caffd640 1 -- 172.21.15.114:0/1368202803 <== mds.0 v2:172.21.15.119:6838/937297340 1 ==== command_reply(tid 0: 0 ) ==== 8+0+44 (crc 0 0 0) 0x55ded7b89200 con 0x55ded7ca2790 2024-04-23T10:36:51.196 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:36:51.196+0000 7f04caffd640 10 client.16788 handle_command_reply: tid=0 2024-04-23T10:36:51.196 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:36:51.196+0000 7f04f4c60740 1 client.16788 shutdown 2024-04-23T10:36:51.196 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:36:51.196+0000 7f04f4c60740 1 -- 172.21.15.114:0/1368202803 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55ded7c2bc10 msgr2=0x55ded7c316e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T10:36:51.196 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:36:51.196+0000 7f04f4c60740 1 --2- 172.21.15.114:0/1368202803 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55ded7c2bc10 0x55ded7c316e0 secure :-1 s=READY pgs=843 cs=0 l=1 rev1=1 crypto rx=0x7f04e4009a60 tx=0x7f04e4009360 comp rx=0 tx=0).stop 2024-04-23T10:36:51.196 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:36:51.196+0000 7f04f4c60740 1 -- 172.21.15.114:0/1368202803 shutdown_connections 2024-04-23T10:36:51.196 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:36:51.196+0000 7f04f4c60740 1 --2- 172.21.15.114:0/1368202803 >> [v2:172.21.15.119:6838/937297340,v1:172.21.15.119:6839/937297340] conn(0x55ded7ca2790 0x55ded7ca4bb0 crc :-1 s=READY pgs=153 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:36:51.196 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:36:51.196+0000 7f04f4c60740 1 --2- 172.21.15.114:0/1368202803 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55ded7c2bc10 0x55ded7c316e0 unknown :-1 s=CLOSED pgs=843 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:36:51.196 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:36:51.196+0000 7f04caffd640 0 client.16788 ms_handle_reset on v2:172.21.15.119:6838/937297340 2024-04-23T10:36:51.196 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:36:51.196+0000 7f04f4c60740 1 --2- 172.21.15.114:0/1368202803 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x55ded7c21350 0x55ded7c311a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:36:51.196 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:36:51.196+0000 7f04f4c60740 1 --2- 172.21.15.114:0/1368202803 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x55ded7c20a30 0x55ded7c33e10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:36:51.196 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:36:51.196+0000 7f04f4c60740 1 -- 172.21.15.114:0/1368202803 >> 172.21.15.114:0/1368202803 conn(0x55ded7c1cf40 msgr2=0x55ded7c17cb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T10:36:51.196 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:36:51.196+0000 7f04f4c60740 1 -- 172.21.15.114:0/1368202803 shutdown_connections 2024-04-23T10:36:51.196 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:36:51.196+0000 7f04f4c60740 1 -- 172.21.15.114:0/1368202803 wait complete. 2024-04-23T10:36:51.196 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:36:51.196+0000 7f04f4c60740 20 client.16788 trim_cache size 0 max 16384 2024-04-23T10:36:51.197 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stdout:{ 2024-04-23T10:36:51.197 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stdout: "message": "", 2024-04-23T10:36:51.197 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stdout: "return_code": 0 2024-04-23T10:36:51.197 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stdout:} 2024-04-23T10:36:51.198 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:36:51.197+0000 7f04f1f51640 1 -- 172.21.15.114:0/3330750689 >> [v2:172.21.15.114:6800/1395693466,v1:172.21.15.114:6801/1395693466] conn(0x7f04d80a3dc0 msgr2=0x7f04d80a6280 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T10:36:51.198 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:36:51.197+0000 7f04f1f51640 1 --2- 172.21.15.114:0/3330750689 >> [v2:172.21.15.114:6800/1395693466,v1:172.21.15.114:6801/1395693466] conn(0x7f04d80a3dc0 0x7f04d80a6280 secure :-1 s=READY pgs=130 cs=0 l=1 rev1=1 crypto rx=0x7f04e4004af0 tx=0x7f04e402a8b0 comp rx=0 tx=0).stop 2024-04-23T10:36:51.198 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:36:51.197+0000 7f04f1f51640 1 -- 172.21.15.114:0/3330750689 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f04ec1011a0 msgr2=0x7f04ec1993e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T10:36:51.198 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:36:51.197+0000 7f04f1f51640 1 --2- 172.21.15.114:0/3330750689 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f04ec1011a0 0x7f04ec1993e0 secure :-1 s=READY pgs=839 cs=0 l=1 rev1=1 crypto rx=0x7f04dc00e9f0 tx=0x7f04dc00edf0 comp rx=0 tx=0).stop 2024-04-23T10:36:51.198 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:36:51.197+0000 7f04f1f51640 1 -- 172.21.15.114:0/3330750689 shutdown_connections 2024-04-23T10:36:51.198 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:36:51.197+0000 7f04f1f51640 1 --2- 172.21.15.114:0/3330750689 >> [v2:172.21.15.114:6800/1395693466,v1:172.21.15.114:6801/1395693466] conn(0x7f04d80a3dc0 0x7f04d80a6280 unknown :-1 s=CLOSED pgs=130 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:36:51.198 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:36:51.197+0000 7f04f1f51640 1 --2- 172.21.15.114:0/3330750689 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f04ec106fb0 0x7f04ec1a4460 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:36:51.198 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:36:51.197+0000 7f04f1f51640 1 --2- 172.21.15.114:0/3330750689 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f04ec101b60 0x7f04ec199920 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:36:51.198 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:36:51.197+0000 7f04f1f51640 1 --2- 172.21.15.114:0/3330750689 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f04ec1011a0 0x7f04ec1993e0 unknown :-1 s=CLOSED pgs=839 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:36:51.198 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:36:51.197+0000 7f04f1f51640 1 -- 172.21.15.114:0/3330750689 >> 172.21.15.114:0/3330750689 conn(0x7f04ec076c50 msgr2=0x7f04ec0ffdf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T10:36:51.199 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:36:51.197+0000 7f04f1f51640 1 -- 172.21.15.114:0/3330750689 shutdown_connections 2024-04-23T10:36:51.199 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:36:51.197+0000 7f04f1f51640 1 -- 172.21.15.114:0/3330750689 wait complete. 2024-04-23T10:36:51.939 INFO:tasks.workunit.client.0.smithi114.stdout: 1 657121 6.19 MB/sec execute 420 sec latency 2658.195 ms 2024-04-23T10:36:51.992 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 23 10:36:51 smithi119 ceph-mon[32608]: pgmap v320: 97 pgs: 97 active+clean; 201 MiB data, 2.5 GiB used, 1.0 TiB / 1.0 TiB avail; 227 KiB/s rd, 375 KiB/s wr, 61 op/s 2024-04-23T10:36:52.003 INFO:journalctl@ceph.mon.a.smithi114.stdout:Apr 23 10:36:51 smithi114 ceph-mon[29255]: pgmap v320: 97 pgs: 97 active+clean; 201 MiB data, 2.5 GiB used, 1.0 TiB / 1.0 TiB avail; 227 KiB/s rd, 375 KiB/s wr, 61 op/s 2024-04-23T10:36:52.077 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 23 10:36:51 smithi204 ceph-mon[32649]: pgmap v320: 97 pgs: 97 active+clean; 201 MiB data, 2.5 GiB used, 1.0 TiB / 1.0 TiB avail; 227 KiB/s rd, 375 KiB/s wr, 61 op/s 2024-04-23T10:36:52.939 INFO:tasks.workunit.client.0.smithi114.stdout: 1 657121 6.18 MB/sec execute 421 sec latency 3658.330 ms 2024-04-23T10:36:53.939 INFO:tasks.workunit.client.0.smithi114.stdout: 1 657121 6.16 MB/sec execute 422 sec latency 4658.368 ms 2024-04-23T10:36:53.991 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 23 10:36:53 smithi119 ceph-mon[32608]: pgmap v321: 97 pgs: 97 active+clean; 201 MiB data, 2.5 GiB used, 1.0 TiB / 1.0 TiB avail; 318 KiB/s rd, 656 KiB/s wr, 87 op/s 2024-04-23T10:36:53.991 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 23 10:36:53 smithi119 ceph-mon[32608]: from='mgr.15291 172.21.15.114:0/1095789187' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-23T10:36:54.004 INFO:journalctl@ceph.mon.a.smithi114.stdout:Apr 23 10:36:53 smithi114 ceph-mon[29255]: pgmap v321: 97 pgs: 97 active+clean; 201 MiB data, 2.5 GiB used, 1.0 TiB / 1.0 TiB avail; 318 KiB/s rd, 656 KiB/s wr, 87 op/s 2024-04-23T10:36:54.004 INFO:journalctl@ceph.mon.a.smithi114.stdout:Apr 23 10:36:53 smithi114 ceph-mon[29255]: from='mgr.15291 172.21.15.114:0/1095789187' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-23T10:36:54.077 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 23 10:36:53 smithi204 ceph-mon[32649]: pgmap v321: 97 pgs: 97 active+clean; 201 MiB data, 2.5 GiB used, 1.0 TiB / 1.0 TiB avail; 318 KiB/s rd, 656 KiB/s wr, 87 op/s 2024-04-23T10:36:54.078 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 23 10:36:53 smithi204 ceph-mon[32649]: from='mgr.15291 172.21.15.114:0/1095789187' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-23T10:36:54.939 INFO:tasks.workunit.client.0.smithi114.stdout: 1 657895 6.15 MB/sec execute 423 sec latency 4900.731 ms 2024-04-23T10:36:55.939 INFO:tasks.workunit.client.0.smithi114.stdout: 1 660149 6.17 MB/sec execute 424 sec latency 17.308 ms 2024-04-23T10:36:56.492 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 23 10:36:56 smithi119 ceph-mon[32608]: pgmap v322: 97 pgs: 97 active+clean; 201 MiB data, 2.5 GiB used, 1.0 TiB / 1.0 TiB avail; 281 KiB/s rd, 610 KiB/s wr, 62 op/s 2024-04-23T10:36:56.492 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 23 10:36:56 smithi119 ceph-mon[32608]: from='mgr.15291 172.21.15.114:0/1095789187' entity='mgr.x' 2024-04-23T10:36:56.492 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 23 10:36:56 smithi119 ceph-mon[32608]: from='mgr.15291 172.21.15.114:0/1095789187' entity='mgr.x' 2024-04-23T10:36:56.492 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 23 10:36:56 smithi119 ceph-mon[32608]: from='mgr.15291 172.21.15.114:0/1095789187' entity='mgr.x' 2024-04-23T10:36:56.492 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 23 10:36:56 smithi119 ceph-mon[32608]: from='mgr.15291 172.21.15.114:0/1095789187' entity='mgr.x' 2024-04-23T10:36:56.492 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 23 10:36:56 smithi119 ceph-mon[32608]: from='mgr.15291 172.21.15.114:0/1095789187' entity='mgr.x' 2024-04-23T10:36:56.492 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 23 10:36:56 smithi119 ceph-mon[32608]: from='mgr.15291 172.21.15.114:0/1095789187' entity='mgr.x' 2024-04-23T10:36:56.493 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 23 10:36:56 smithi119 ceph-mon[32608]: from='mgr.15291 172.21.15.114:0/1095789187' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-04-23T10:36:56.493 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 23 10:36:56 smithi119 ceph-mon[32608]: from='mgr.15291 172.21.15.114:0/1095789187' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-04-23T10:36:56.493 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 23 10:36:56 smithi119 ceph-mon[32608]: from='mgr.15291 172.21.15.114:0/1095789187' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2024-04-23T10:36:56.493 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 23 10:36:56 smithi119 ceph-mon[32608]: from='mgr.15291 172.21.15.114:0/1095789187' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2024-04-23T10:36:56.493 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 23 10:36:56 smithi119 ceph-mon[32608]: from='mgr.15291 172.21.15.114:0/1095789187' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-04-23T10:36:56.493 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 23 10:36:56 smithi119 ceph-mon[32608]: from='mgr.15291 172.21.15.114:0/1095789187' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-04-23T10:36:56.493 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 23 10:36:56 smithi119 ceph-mon[32608]: from='mgr.15291 172.21.15.114:0/1095789187' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-04-23T10:36:56.493 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 23 10:36:56 smithi119 ceph-mon[32608]: from='mgr.15291 172.21.15.114:0/1095789187' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-04-23T10:36:56.493 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 23 10:36:56 smithi119 ceph-mon[32608]: from='mgr.15291 172.21.15.114:0/1095789187' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-04-23T10:36:56.493 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 23 10:36:56 smithi119 ceph-mon[32608]: from='mgr.15291 172.21.15.114:0/1095789187' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-04-23T10:36:56.493 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 23 10:36:56 smithi119 ceph-mon[32608]: from='mgr.15291 172.21.15.114:0/1095789187' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"}]: dispatch 2024-04-23T10:36:56.493 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 23 10:36:56 smithi119 ceph-mon[32608]: from='mgr.15291 172.21.15.114:0/1095789187' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2024-04-23T10:36:56.493 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 23 10:36:56 smithi119 ceph-mon[32608]: from='mgr.15291 172.21.15.114:0/1095789187' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-23T10:36:56.493 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 23 10:36:56 smithi119 ceph-mon[32608]: from='mgr.15291 172.21.15.114:0/1095789187' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-23T10:36:56.493 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 23 10:36:56 smithi119 ceph-mon[32608]: from='mgr.15291 172.21.15.114:0/1095789187' entity='mgr.x' 2024-04-23T10:36:56.504 INFO:journalctl@ceph.mon.a.smithi114.stdout:Apr 23 10:36:56 smithi114 ceph-mon[29255]: pgmap v322: 97 pgs: 97 active+clean; 201 MiB data, 2.5 GiB used, 1.0 TiB / 1.0 TiB avail; 281 KiB/s rd, 610 KiB/s wr, 62 op/s 2024-04-23T10:36:56.504 INFO:journalctl@ceph.mon.a.smithi114.stdout:Apr 23 10:36:56 smithi114 ceph-mon[29255]: from='mgr.15291 172.21.15.114:0/1095789187' entity='mgr.x' 2024-04-23T10:36:56.504 INFO:journalctl@ceph.mon.a.smithi114.stdout:Apr 23 10:36:56 smithi114 ceph-mon[29255]: from='mgr.15291 172.21.15.114:0/1095789187' entity='mgr.x' 2024-04-23T10:36:56.504 INFO:journalctl@ceph.mon.a.smithi114.stdout:Apr 23 10:36:56 smithi114 ceph-mon[29255]: from='mgr.15291 172.21.15.114:0/1095789187' entity='mgr.x' 2024-04-23T10:36:56.504 INFO:journalctl@ceph.mon.a.smithi114.stdout:Apr 23 10:36:56 smithi114 ceph-mon[29255]: from='mgr.15291 172.21.15.114:0/1095789187' entity='mgr.x' 2024-04-23T10:36:56.504 INFO:journalctl@ceph.mon.a.smithi114.stdout:Apr 23 10:36:56 smithi114 ceph-mon[29255]: from='mgr.15291 172.21.15.114:0/1095789187' entity='mgr.x' 2024-04-23T10:36:56.504 INFO:journalctl@ceph.mon.a.smithi114.stdout:Apr 23 10:36:56 smithi114 ceph-mon[29255]: from='mgr.15291 172.21.15.114:0/1095789187' entity='mgr.x' 2024-04-23T10:36:56.504 INFO:journalctl@ceph.mon.a.smithi114.stdout:Apr 23 10:36:56 smithi114 ceph-mon[29255]: from='mgr.15291 172.21.15.114:0/1095789187' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-04-23T10:36:56.504 INFO:journalctl@ceph.mon.a.smithi114.stdout:Apr 23 10:36:56 smithi114 ceph-mon[29255]: from='mgr.15291 172.21.15.114:0/1095789187' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-04-23T10:36:56.504 INFO:journalctl@ceph.mon.a.smithi114.stdout:Apr 23 10:36:56 smithi114 ceph-mon[29255]: from='mgr.15291 172.21.15.114:0/1095789187' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2024-04-23T10:36:56.504 INFO:journalctl@ceph.mon.a.smithi114.stdout:Apr 23 10:36:56 smithi114 ceph-mon[29255]: from='mgr.15291 172.21.15.114:0/1095789187' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2024-04-23T10:36:56.505 INFO:journalctl@ceph.mon.a.smithi114.stdout:Apr 23 10:36:56 smithi114 ceph-mon[29255]: from='mgr.15291 172.21.15.114:0/1095789187' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-04-23T10:36:56.505 INFO:journalctl@ceph.mon.a.smithi114.stdout:Apr 23 10:36:56 smithi114 ceph-mon[29255]: from='mgr.15291 172.21.15.114:0/1095789187' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-04-23T10:36:56.505 INFO:journalctl@ceph.mon.a.smithi114.stdout:Apr 23 10:36:56 smithi114 ceph-mon[29255]: from='mgr.15291 172.21.15.114:0/1095789187' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-04-23T10:36:56.505 INFO:journalctl@ceph.mon.a.smithi114.stdout:Apr 23 10:36:56 smithi114 ceph-mon[29255]: from='mgr.15291 172.21.15.114:0/1095789187' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-04-23T10:36:56.505 INFO:journalctl@ceph.mon.a.smithi114.stdout:Apr 23 10:36:56 smithi114 ceph-mon[29255]: from='mgr.15291 172.21.15.114:0/1095789187' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-04-23T10:36:56.505 INFO:journalctl@ceph.mon.a.smithi114.stdout:Apr 23 10:36:56 smithi114 ceph-mon[29255]: from='mgr.15291 172.21.15.114:0/1095789187' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-04-23T10:36:56.505 INFO:journalctl@ceph.mon.a.smithi114.stdout:Apr 23 10:36:56 smithi114 ceph-mon[29255]: from='mgr.15291 172.21.15.114:0/1095789187' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"}]: dispatch 2024-04-23T10:36:56.505 INFO:journalctl@ceph.mon.a.smithi114.stdout:Apr 23 10:36:56 smithi114 ceph-mon[29255]: from='mgr.15291 172.21.15.114:0/1095789187' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2024-04-23T10:36:56.505 INFO:journalctl@ceph.mon.a.smithi114.stdout:Apr 23 10:36:56 smithi114 ceph-mon[29255]: from='mgr.15291 172.21.15.114:0/1095789187' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-23T10:36:56.505 INFO:journalctl@ceph.mon.a.smithi114.stdout:Apr 23 10:36:56 smithi114 ceph-mon[29255]: from='mgr.15291 172.21.15.114:0/1095789187' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-23T10:36:56.505 INFO:journalctl@ceph.mon.a.smithi114.stdout:Apr 23 10:36:56 smithi114 ceph-mon[29255]: from='mgr.15291 172.21.15.114:0/1095789187' entity='mgr.x' 2024-04-23T10:36:56.578 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 23 10:36:56 smithi204 ceph-mon[32649]: pgmap v322: 97 pgs: 97 active+clean; 201 MiB data, 2.5 GiB used, 1.0 TiB / 1.0 TiB avail; 281 KiB/s rd, 610 KiB/s wr, 62 op/s 2024-04-23T10:36:56.578 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 23 10:36:56 smithi204 ceph-mon[32649]: from='mgr.15291 172.21.15.114:0/1095789187' entity='mgr.x' 2024-04-23T10:36:56.578 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 23 10:36:56 smithi204 ceph-mon[32649]: from='mgr.15291 172.21.15.114:0/1095789187' entity='mgr.x' 2024-04-23T10:36:56.578 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 23 10:36:56 smithi204 ceph-mon[32649]: from='mgr.15291 172.21.15.114:0/1095789187' entity='mgr.x' 2024-04-23T10:36:56.578 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 23 10:36:56 smithi204 ceph-mon[32649]: from='mgr.15291 172.21.15.114:0/1095789187' entity='mgr.x' 2024-04-23T10:36:56.578 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 23 10:36:56 smithi204 ceph-mon[32649]: from='mgr.15291 172.21.15.114:0/1095789187' entity='mgr.x' 2024-04-23T10:36:56.578 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 23 10:36:56 smithi204 ceph-mon[32649]: from='mgr.15291 172.21.15.114:0/1095789187' entity='mgr.x' 2024-04-23T10:36:56.578 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 23 10:36:56 smithi204 ceph-mon[32649]: from='mgr.15291 172.21.15.114:0/1095789187' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-04-23T10:36:56.578 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 23 10:36:56 smithi204 ceph-mon[32649]: from='mgr.15291 172.21.15.114:0/1095789187' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-04-23T10:36:56.578 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 23 10:36:56 smithi204 ceph-mon[32649]: from='mgr.15291 172.21.15.114:0/1095789187' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2024-04-23T10:36:56.578 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 23 10:36:56 smithi204 ceph-mon[32649]: from='mgr.15291 172.21.15.114:0/1095789187' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2024-04-23T10:36:56.578 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 23 10:36:56 smithi204 ceph-mon[32649]: from='mgr.15291 172.21.15.114:0/1095789187' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-04-23T10:36:56.579 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 23 10:36:56 smithi204 ceph-mon[32649]: from='mgr.15291 172.21.15.114:0/1095789187' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-04-23T10:36:56.579 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 23 10:36:56 smithi204 ceph-mon[32649]: from='mgr.15291 172.21.15.114:0/1095789187' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-04-23T10:36:56.579 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 23 10:36:56 smithi204 ceph-mon[32649]: from='mgr.15291 172.21.15.114:0/1095789187' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-04-23T10:36:56.579 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 23 10:36:56 smithi204 ceph-mon[32649]: from='mgr.15291 172.21.15.114:0/1095789187' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-04-23T10:36:56.579 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 23 10:36:56 smithi204 ceph-mon[32649]: from='mgr.15291 172.21.15.114:0/1095789187' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-04-23T10:36:56.579 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 23 10:36:56 smithi204 ceph-mon[32649]: from='mgr.15291 172.21.15.114:0/1095789187' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"}]: dispatch 2024-04-23T10:36:56.579 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 23 10:36:56 smithi204 ceph-mon[32649]: from='mgr.15291 172.21.15.114:0/1095789187' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2024-04-23T10:36:56.579 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 23 10:36:56 smithi204 ceph-mon[32649]: from='mgr.15291 172.21.15.114:0/1095789187' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-23T10:36:56.579 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 23 10:36:56 smithi204 ceph-mon[32649]: from='mgr.15291 172.21.15.114:0/1095789187' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-23T10:36:56.579 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 23 10:36:56 smithi204 ceph-mon[32649]: from='mgr.15291 172.21.15.114:0/1095789187' entity='mgr.x' 2024-04-23T10:36:56.939 INFO:tasks.workunit.client.0.smithi114.stdout: 1 660599 6.16 MB/sec execute 425 sec latency 803.506 ms 2024-04-23T10:36:57.492 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 23 10:36:57 smithi119 ceph-mon[32608]: Adjusting osd_memory_target on smithi204 to 176.2M 2024-04-23T10:36:57.492 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 23 10:36:57 smithi119 ceph-mon[32608]: Unable to set osd_memory_target on smithi204 to 184759500: error parsing value: Value '184759500' is below minimum 939524096 2024-04-23T10:36:57.492 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 23 10:36:57 smithi119 ceph-mon[32608]: Adjusting osd_memory_target on smithi114 to 176.2M 2024-04-23T10:36:57.492 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 23 10:36:57 smithi119 ceph-mon[32608]: Adjusting osd_memory_target on smithi119 to 176.2M 2024-04-23T10:36:57.492 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 23 10:36:57 smithi119 ceph-mon[32608]: Unable to set osd_memory_target on smithi114 to 184769536: error parsing value: Value '184769536' is below minimum 939524096 2024-04-23T10:36:57.493 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 23 10:36:57 smithi119 ceph-mon[32608]: Unable to set osd_memory_target on smithi119 to 184770252: error parsing value: Value '184770252' is below minimum 939524096 2024-04-23T10:36:57.504 INFO:journalctl@ceph.mon.a.smithi114.stdout:Apr 23 10:36:57 smithi114 ceph-mon[29255]: Adjusting osd_memory_target on smithi204 to 176.2M 2024-04-23T10:36:57.504 INFO:journalctl@ceph.mon.a.smithi114.stdout:Apr 23 10:36:57 smithi114 ceph-mon[29255]: Unable to set osd_memory_target on smithi204 to 184759500: error parsing value: Value '184759500' is below minimum 939524096 2024-04-23T10:36:57.504 INFO:journalctl@ceph.mon.a.smithi114.stdout:Apr 23 10:36:57 smithi114 ceph-mon[29255]: Adjusting osd_memory_target on smithi114 to 176.2M 2024-04-23T10:36:57.504 INFO:journalctl@ceph.mon.a.smithi114.stdout:Apr 23 10:36:57 smithi114 ceph-mon[29255]: Adjusting osd_memory_target on smithi119 to 176.2M 2024-04-23T10:36:57.504 INFO:journalctl@ceph.mon.a.smithi114.stdout:Apr 23 10:36:57 smithi114 ceph-mon[29255]: Unable to set osd_memory_target on smithi114 to 184769536: error parsing value: Value '184769536' is below minimum 939524096 2024-04-23T10:36:57.504 INFO:journalctl@ceph.mon.a.smithi114.stdout:Apr 23 10:36:57 smithi114 ceph-mon[29255]: Unable to set osd_memory_target on smithi119 to 184770252: error parsing value: Value '184770252' is below minimum 939524096 2024-04-23T10:36:57.578 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 23 10:36:57 smithi204 ceph-mon[32649]: Adjusting osd_memory_target on smithi204 to 176.2M 2024-04-23T10:36:57.578 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 23 10:36:57 smithi204 ceph-mon[32649]: Unable to set osd_memory_target on smithi204 to 184759500: error parsing value: Value '184759500' is below minimum 939524096 2024-04-23T10:36:57.578 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 23 10:36:57 smithi204 ceph-mon[32649]: Adjusting osd_memory_target on smithi114 to 176.2M 2024-04-23T10:36:57.578 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 23 10:36:57 smithi204 ceph-mon[32649]: Adjusting osd_memory_target on smithi119 to 176.2M 2024-04-23T10:36:57.578 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 23 10:36:57 smithi204 ceph-mon[32649]: Unable to set osd_memory_target on smithi114 to 184769536: error parsing value: Value '184769536' is below minimum 939524096 2024-04-23T10:36:57.578 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 23 10:36:57 smithi204 ceph-mon[32649]: Unable to set osd_memory_target on smithi119 to 184770252: error parsing value: Value '184770252' is below minimum 939524096 2024-04-23T10:36:57.939 INFO:tasks.workunit.client.0.smithi114.stdout: 1 660599 6.15 MB/sec execute 426 sec latency 1803.597 ms 2024-04-23T10:36:58.492 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 23 10:36:58 smithi119 ceph-mon[32608]: pgmap v323: 97 pgs: 97 active+clean; 204 MiB data, 2.5 GiB used, 1.0 TiB / 1.0 TiB avail; 493 KiB/s rd, 942 KiB/s wr, 99 op/s 2024-04-23T10:36:58.503 INFO:journalctl@ceph.mon.a.smithi114.stdout:Apr 23 10:36:58 smithi114 ceph-mon[29255]: pgmap v323: 97 pgs: 97 active+clean; 204 MiB data, 2.5 GiB used, 1.0 TiB / 1.0 TiB avail; 493 KiB/s rd, 942 KiB/s wr, 99 op/s 2024-04-23T10:36:58.577 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 23 10:36:58 smithi204 ceph-mon[32649]: pgmap v323: 97 pgs: 97 active+clean; 204 MiB data, 2.5 GiB used, 1.0 TiB / 1.0 TiB avail; 493 KiB/s rd, 942 KiB/s wr, 99 op/s 2024-04-23T10:36:58.939 INFO:tasks.workunit.client.0.smithi114.stdout: 1 660599 6.14 MB/sec execute 427 sec latency 2803.646 ms 2024-04-23T10:36:59.940 INFO:tasks.workunit.client.0.smithi114.stdout: 1 660666 6.12 MB/sec execute 428 sec latency 3052.068 ms 2024-04-23T10:37:00.492 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 23 10:37:00 smithi119 ceph-mon[32608]: pgmap v324: 97 pgs: 97 active+clean; 204 MiB data, 2.5 GiB used, 1.0 TiB / 1.0 TiB avail; 303 KiB/s rd, 613 KiB/s wr, 62 op/s 2024-04-23T10:37:00.504 INFO:journalctl@ceph.mon.a.smithi114.stdout:Apr 23 10:37:00 smithi114 ceph-mon[29255]: pgmap v324: 97 pgs: 97 active+clean; 204 MiB data, 2.5 GiB used, 1.0 TiB / 1.0 TiB avail; 303 KiB/s rd, 613 KiB/s wr, 62 op/s 2024-04-23T10:37:00.577 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 23 10:37:00 smithi204 ceph-mon[32649]: pgmap v324: 97 pgs: 97 active+clean; 204 MiB data, 2.5 GiB used, 1.0 TiB / 1.0 TiB avail; 303 KiB/s rd, 613 KiB/s wr, 62 op/s 2024-04-23T10:37:00.940 INFO:tasks.workunit.client.0.smithi114.stdout: 1 660666 6.11 MB/sec execute 429 sec latency 1652.745 ms 2024-04-23T10:37:01.940 INFO:tasks.workunit.client.0.smithi114.stdout: 1 660666 6.09 MB/sec execute 430 sec latency 2652.822 ms 2024-04-23T10:37:02.492 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 23 10:37:02 smithi119 ceph-mon[32608]: pgmap v325: 97 pgs: 97 active+clean; 204 MiB data, 2.5 GiB used, 1.0 TiB / 1.0 TiB avail; 303 KiB/s rd, 613 KiB/s wr, 62 op/s 2024-04-23T10:37:02.504 INFO:journalctl@ceph.mon.a.smithi114.stdout:Apr 23 10:37:02 smithi114 ceph-mon[29255]: pgmap v325: 97 pgs: 97 active+clean; 204 MiB data, 2.5 GiB used, 1.0 TiB / 1.0 TiB avail; 303 KiB/s rd, 613 KiB/s wr, 62 op/s 2024-04-23T10:37:02.577 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 23 10:37:02 smithi204 ceph-mon[32649]: pgmap v325: 97 pgs: 97 active+clean; 204 MiB data, 2.5 GiB used, 1.0 TiB / 1.0 TiB avail; 303 KiB/s rd, 613 KiB/s wr, 62 op/s 2024-04-23T10:37:02.940 INFO:tasks.workunit.client.0.smithi114.stdout: 1 660666 6.08 MB/sec execute 431 sec latency 3652.908 ms 2024-04-23T10:37:03.940 INFO:tasks.workunit.client.0.smithi114.stdout: 1 660666 6.06 MB/sec execute 432 sec latency 4652.964 ms 2024-04-23T10:37:04.313 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:37:04.311+0000 7f51c9eb9640 1 -- 172.21.15.114:0/50301339 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f51c4100d60 msgr2=0x7f51c4101140 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T10:37:04.313 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:37:04.311+0000 7f51c9eb9640 1 --2- 172.21.15.114:0/50301339 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f51c4100d60 0x7f51c4101140 secure :-1 s=READY pgs=844 cs=0 l=1 rev1=1 crypto rx=0x7f51b4009900 tx=0x7f51b402a410 comp rx=0 tx=0).stop 2024-04-23T10:37:04.313 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:37:04.312+0000 7f51c9eb9640 1 -- 172.21.15.114:0/50301339 shutdown_connections 2024-04-23T10:37:04.313 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:37:04.312+0000 7f51c9eb9640 1 --2- 172.21.15.114:0/50301339 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f51c410e910 0x7f51c4110d00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:37:04.313 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:37:04.312+0000 7f51c9eb9640 1 --2- 172.21.15.114:0/50301339 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f51c4101680 0x7f51c410e3d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:37:04.313 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:37:04.312+0000 7f51c9eb9640 1 --2- 172.21.15.114:0/50301339 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f51c4100d60 0x7f51c4101140 unknown :-1 s=CLOSED pgs=844 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:37:04.313 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:37:04.312+0000 7f51c9eb9640 1 -- 172.21.15.114:0/50301339 >> 172.21.15.114:0/50301339 conn(0x7f51c4077fd0 msgr2=0x7f51c40783e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T10:37:04.313 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:37:04.312+0000 7f51c9eb9640 1 -- 172.21.15.114:0/50301339 shutdown_connections 2024-04-23T10:37:04.313 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:37:04.312+0000 7f51c9eb9640 1 -- 172.21.15.114:0/50301339 wait complete. 2024-04-23T10:37:04.314 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:37:04.312+0000 7f51c9eb9640 1 Processor -- start 2024-04-23T10:37:04.314 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:37:04.312+0000 7f51c9eb9640 1 -- start start 2024-04-23T10:37:04.314 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:37:04.313+0000 7f51c9eb9640 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f51c4100d60 0x7f51c41a0630 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T10:37:04.314 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:37:04.313+0000 7f51c9eb9640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f51c4101680 0x7f51c41a0b70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T10:37:04.314 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:37:04.313+0000 7f51c9eb9640 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f51c410e910 0x7f51c419a740 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T10:37:04.314 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:37:04.313+0000 7f51c9eb9640 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x7f51c41143e0 con 0x7f51c4101680 2024-04-23T10:37:04.314 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:37:04.313+0000 7f51c9eb9640 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 -- 0x7f51c4114260 con 0x7f51c4100d60 2024-04-23T10:37:04.314 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:37:04.313+0000 7f51c9eb9640 1 -- --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_getmap magic: 0 -- 0x7f51c4114560 con 0x7f51c410e910 2024-04-23T10:37:04.314 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:37:04.313+0000 7f51c2ffd640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f51c4101680 0x7f51c41a0b70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T10:37:04.314 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:37:04.313+0000 7f51c2ffd640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f51c4101680 0x7f51c41a0b70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.114:58206/0 (socket says 172.21.15.114:58206) 2024-04-23T10:37:04.314 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:37:04.313+0000 7f51c2ffd640 1 -- 172.21.15.114:0/3568019809 learned_addr learned my addr 172.21.15.114:0/3568019809 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-04-23T10:37:04.314 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:37:04.313+0000 7f51c37fe640 1 --2- 172.21.15.114:0/3568019809 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f51c4100d60 0x7f51c41a0630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T10:37:04.314 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:37:04.313+0000 7f51c3fff640 1 --2- 172.21.15.114:0/3568019809 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f51c410e910 0x7f51c419a740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T10:37:04.315 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:37:04.313+0000 7f51c2ffd640 1 -- 172.21.15.114:0/3568019809 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f51c410e910 msgr2=0x7f51c419a740 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T10:37:04.315 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:37:04.313+0000 7f51c2ffd640 1 --2- 172.21.15.114:0/3568019809 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f51c410e910 0x7f51c419a740 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:37:04.315 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:37:04.313+0000 7f51c2ffd640 1 -- 172.21.15.114:0/3568019809 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f51c4100d60 msgr2=0x7f51c41a0630 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T10:37:04.315 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:37:04.313+0000 7f51c2ffd640 1 --2- 172.21.15.114:0/3568019809 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f51c4100d60 0x7f51c41a0630 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:37:04.315 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:37:04.313+0000 7f51c2ffd640 1 -- 172.21.15.114:0/3568019809 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f51c419b000 con 0x7f51c4101680 2024-04-23T10:37:04.315 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:37:04.313+0000 7f51c37fe640 1 --2- 172.21.15.114:0/3568019809 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f51c4100d60 0x7f51c41a0630 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-23T10:37:04.315 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:37:04.313+0000 7f51c2ffd640 1 --2- 172.21.15.114:0/3568019809 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f51c4101680 0x7f51c41a0b70 secure :-1 s=READY pgs=845 cs=0 l=1 rev1=1 crypto rx=0x7f51b800e9f0 tx=0x7f51b800edf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T10:37:04.315 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:37:04.313+0000 7f51c0ff9640 1 -- 172.21.15.114:0/3568019809 <== mon.0 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f51b800cd80 con 0x7f51c4101680 2024-04-23T10:37:04.315 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:37:04.313+0000 7f51c0ff9640 1 -- 172.21.15.114:0/3568019809 <== mon.0 v2:172.21.15.114:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f51b80054d0 con 0x7f51c4101680 2024-04-23T10:37:04.315 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:37:04.313+0000 7f51c0ff9640 1 -- 172.21.15.114:0/3568019809 <== mon.0 v2:172.21.15.114:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f51b8010690 con 0x7f51c4101680 2024-04-23T10:37:04.315 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:37:04.313+0000 7f51c9eb9640 1 -- 172.21.15.114:0/3568019809 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f51c419b2f0 con 0x7f51c4101680 2024-04-23T10:37:04.315 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:37:04.313+0000 7f51c9eb9640 1 -- 172.21.15.114:0/3568019809 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f51c41a75b0 con 0x7f51c4101680 2024-04-23T10:37:04.322 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:37:04.316+0000 7f51c0ff9640 1 -- 172.21.15.114:0/3568019809 <== mon.0 v2:172.21.15.114:3300/0 4 ==== mgrmap(e 26) ==== 138778+0+0 (secure 0 0 0) 0x7f51b8004e90 con 0x7f51c4101680 2024-04-23T10:37:04.322 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:37:04.316+0000 7f51ccbcd740 1 Processor -- start 2024-04-23T10:37:04.322 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:37:04.316+0000 7f51ccbcd740 1 -- start start 2024-04-23T10:37:04.322 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:37:04.316+0000 7f51c0ff9640 1 --2- 172.21.15.114:0/3568019809 >> [v2:172.21.15.114:6800/1395693466,v1:172.21.15.114:6801/1395693466] conn(0x7f51ac0a3dc0 0x7f51ac0a6280 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T10:37:04.322 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:37:04.316+0000 7f51ccbcd740 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x55693663c620 0x55693663ca00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T10:37:04.322 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:37:04.316+0000 7f51ccbcd740 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55693663cf40 0x5569366472c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T10:37:04.322 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:37:04.316+0000 7f51ccbcd740 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x556936647800 0x556936649bf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T10:37:04.323 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:37:04.316+0000 7f51ccbcd740 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x5569365a1970 con 0x55693663cf40 2024-04-23T10:37:04.323 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:37:04.316+0000 7f51ccbcd740 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 -- 0x55693653e4a0 con 0x55693663c620 2024-04-23T10:37:04.323 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:37:04.316+0000 7f51ccbcd740 1 -- --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_getmap magic: 0 -- 0x55693659d930 con 0x556936647800 2024-04-23T10:37:04.323 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:37:04.316+0000 7f51c3fff640 1 --2- 172.21.15.114:0/3568019809 >> [v2:172.21.15.114:6800/1395693466,v1:172.21.15.114:6801/1395693466] conn(0x7f51ac0a3dc0 0x7f51ac0a6280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T10:37:04.323 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:37:04.316+0000 7f51c2ffd640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55693663cf40 0x5569366472c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T10:37:04.323 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:37:04.316+0000 7f51c37fe640 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x55693663c620 0x55693663ca00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T10:37:04.323 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:37:04.317+0000 7f51c37fe640 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x55693663c620 0x55693663ca00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.119:3300/0 says I am v2:172.21.15.114:39572/0 (socket says 172.21.15.114:39572) 2024-04-23T10:37:04.323 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:37:04.317+0000 7f51c37fe640 1 -- 172.21.15.114:0/1631191503 learned_addr learned my addr 172.21.15.114:0/1631191503 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-04-23T10:37:04.323 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:37:04.317+0000 7f51c2ffd640 1 --2- 172.21.15.114:0/1631191503 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55693663cf40 0x5569366472c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.114:58210/0 (socket says 172.21.15.114:58210) 2024-04-23T10:37:04.323 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:37:04.317+0000 7f51c3fff640 1 --2- 172.21.15.114:0/1631191503 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x556936647800 0x556936649bf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T10:37:04.323 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:37:04.317+0000 7f51c0ff9640 1 -- 172.21.15.114:0/3568019809 <== mon.0 v2:172.21.15.114:3300/0 5 ==== osd_map(102..102 src has 1..102) ==== 8629+0+0 (secure 0 0 0) 0x7f51b80d0c60 con 0x7f51c4101680 2024-04-23T10:37:04.323 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:37:04.317+0000 7f51c37fe640 1 -- 172.21.15.114:0/1631191503 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x556936647800 msgr2=0x556936649bf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T10:37:04.323 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:37:04.317+0000 7f51c37fe640 1 --2- 172.21.15.114:0/1631191503 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x556936647800 0x556936649bf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:37:04.324 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:37:04.317+0000 7f51c37fe640 1 -- 172.21.15.114:0/1631191503 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55693663cf40 msgr2=0x5569366472c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T10:37:04.324 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:37:04.317+0000 7f51c37fe640 1 --2- 172.21.15.114:0/1631191503 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55693663cf40 0x5569366472c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:37:04.324 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:37:04.317+0000 7f51c37fe640 1 -- 172.21.15.114:0/1631191503 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5569362b4210 con 0x55693663c620 2024-04-23T10:37:04.324 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:37:04.317+0000 7f51c2ffd640 1 --2- 172.21.15.114:0/1631191503 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55693663cf40 0x5569366472c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-23T10:37:04.324 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:37:04.317+0000 7f51c3fff640 1 --2- 172.21.15.114:0/3568019809 >> [v2:172.21.15.114:6800/1395693466,v1:172.21.15.114:6801/1395693466] conn(0x7f51ac0a3dc0 0x7f51ac0a6280 secure :-1 s=READY pgs=131 cs=0 l=1 rev1=1 crypto rx=0x7f51bc009fd0 tx=0x7f51bc009d80 comp rx=0 tx=0).ready entity=mgr.15291 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T10:37:04.324 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:37:04.317+0000 7f51c37fe640 1 --2- 172.21.15.114:0/1631191503 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x55693663c620 0x55693663ca00 secure :-1 s=READY pgs=93 cs=0 l=1 rev1=1 crypto rx=0x7f51b4009900 tx=0x7f51b4005e80 comp rx=0 tx=0).ready entity=mon.2 client_cookie=f3ab020171792503 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T10:37:04.324 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:37:04.317+0000 7f51a9ffb640 1 -- 172.21.15.114:0/1631191503 <== mon.2 v2:172.21.15.119:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f51b4040070 con 0x55693663c620 2024-04-23T10:37:04.324 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:37:04.317+0000 7f51a9ffb640 1 -- 172.21.15.114:0/1631191503 <== mon.2 v2:172.21.15.119:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f51b40317e0 con 0x55693663c620 2024-04-23T10:37:04.324 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:37:04.317+0000 7f51a9ffb640 1 -- 172.21.15.114:0/1631191503 <== mon.2 v2:172.21.15.119:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f51b4035040 con 0x55693663c620 2024-04-23T10:37:04.324 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:37:04.318+0000 7f51ccbcd740 1 -- 172.21.15.114:0/1631191503 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x55693663c620 msgr2=0x55693663ca00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T10:37:04.324 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:37:04.318+0000 7f51ccbcd740 1 --2- 172.21.15.114:0/1631191503 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x55693663c620 0x55693663ca00 secure :-1 s=READY pgs=93 cs=0 l=1 rev1=1 crypto rx=0x7f51b4009900 tx=0x7f51b4005e80 comp rx=0 tx=0).stop 2024-04-23T10:37:04.324 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:37:04.318+0000 7f51ccbcd740 1 -- 172.21.15.114:0/1631191503 shutdown_connections 2024-04-23T10:37:04.324 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:37:04.318+0000 7f51ccbcd740 1 --2- 172.21.15.114:0/1631191503 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x556936647800 0x556936649bf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:37:04.324 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:37:04.318+0000 7f51ccbcd740 1 --2- 172.21.15.114:0/1631191503 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55693663cf40 0x5569366472c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:37:04.324 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:37:04.318+0000 7f51ccbcd740 1 --2- 172.21.15.114:0/1631191503 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x55693663c620 0x55693663ca00 unknown :-1 s=CLOSED pgs=93 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:37:04.325 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:37:04.318+0000 7f51ccbcd740 1 -- 172.21.15.114:0/1631191503 >> 172.21.15.114:0/1631191503 conn(0x556936638b30 msgr2=0x556936638f10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T10:37:04.325 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:37:04.318+0000 7f51ccbcd740 1 -- 172.21.15.114:0/1631191503 shutdown_connections 2024-04-23T10:37:04.325 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:37:04.318+0000 7f51ccbcd740 1 -- 172.21.15.114:0/1631191503 wait complete. 2024-04-23T10:37:04.325 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:37:04.318+0000 7f51ccbcd740 1 -- start start 2024-04-23T10:37:04.325 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:37:04.319+0000 7f51ccbcd740 1 Processor -- start 2024-04-23T10:37:04.325 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:37:04.319+0000 7f51ccbcd740 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x55693663c620 0x556936647170 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T10:37:04.325 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:37:04.319+0000 7f51ccbcd740 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x55693663cf40 0x556936644460 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T10:37:04.325 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:37:04.319+0000 7f51ccbcd740 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x556936647800 0x5569366449c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T10:37:04.325 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:37:04.319+0000 7f51ccbcd740 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x55693664a3d0 con 0x556936647800 2024-04-23T10:37:04.325 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:37:04.319+0000 7f51ccbcd740 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 -- 0x55693653c0b0 con 0x55693663c620 2024-04-23T10:37:04.325 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:37:04.319+0000 7f51ccbcd740 1 -- --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_getmap magic: 0 -- 0x55693664a550 con 0x55693663cf40 2024-04-23T10:37:04.325 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:37:04.319+0000 7f51ccbcd740 10 client.0 _ll_register_callbacks cb 0x556936529d30 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-23T10:37:04.325 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:37:04.319+0000 7f51c3fff640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x556936647800 0x5569366449c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T10:37:04.325 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:37:04.319+0000 7f51c2ffd640 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x55693663cf40 0x556936644460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T10:37:04.325 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:37:04.319+0000 7f51c37fe640 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x55693663c620 0x556936647170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T10:37:04.326 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:37:04.319+0000 7f51c3fff640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x556936647800 0x5569366449c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.114:58222/0 (socket says 172.21.15.114:58222) 2024-04-23T10:37:04.326 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:37:04.319+0000 7f51c37fe640 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x55693663c620 0x556936647170 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.119:3300/0 says I am v2:172.21.15.114:39576/0 (socket says 172.21.15.114:39576) 2024-04-23T10:37:04.326 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:37:04.319+0000 7f51c2ffd640 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x55693663cf40 0x556936644460 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.204:3300/0 says I am v2:172.21.15.114:33146/0 (socket says 172.21.15.114:33146) 2024-04-23T10:37:04.326 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:37:04.319+0000 7f51c3fff640 1 -- 172.21.15.114:0/780065535 learned_addr learned my addr 172.21.15.114:0/780065535 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-04-23T10:37:04.326 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:37:04.319+0000 7f51c3fff640 1 -- 172.21.15.114:0/780065535 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x55693663cf40 msgr2=0x556936644460 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T10:37:04.326 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:37:04.319+0000 7f51c3fff640 1 --2- 172.21.15.114:0/780065535 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x55693663cf40 0x556936644460 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:37:04.326 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:37:04.319+0000 7f51c3fff640 1 -- 172.21.15.114:0/780065535 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x55693663c620 msgr2=0x556936647170 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T10:37:04.326 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:37:04.319+0000 7f51c3fff640 1 --2- 172.21.15.114:0/780065535 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x55693663c620 0x556936647170 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:37:04.326 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:37:04.319+0000 7f51c3fff640 1 -- 172.21.15.114:0/780065535 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({osdmap=0}) -- 0x556936380620 con 0x556936647800 2024-04-23T10:37:04.326 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:37:04.319+0000 7f51c37fe640 1 --2- 172.21.15.114:0/780065535 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x55693663c620 0x556936647170 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-23T10:37:04.326 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:37:04.319+0000 7f51c3fff640 1 --2- 172.21.15.114:0/780065535 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x556936647800 0x5569366449c0 secure :-1 s=READY pgs=846 cs=0 l=1 rev1=1 crypto rx=0x7f51bc00fec0 tx=0x7f51bc01d000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3a9e79e38a2f65b0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T10:37:04.326 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:37:04.319+0000 7f51a2ffd640 10 client.0 ms_handle_connect on v2:172.21.15.114:3300/0 2024-04-23T10:37:04.326 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:37:04.320+0000 7f51a2ffd640 1 -- 172.21.15.114:0/780065535 <== mon.0 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f51bc013330 con 0x556936647800 2024-04-23T10:37:04.326 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:37:04.320+0000 7f51a2ffd640 1 -- 172.21.15.114:0/780065535 <== mon.0 v2:172.21.15.114:3300/0 2 ==== osd_map(102..102 src has 1..102) ==== 8629+0+0 (secure 0 0 0) 0x7f51bc01c040 con 0x556936647800 2024-04-23T10:37:04.326 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:37:04.320+0000 7f51ccbcd740 1 -- 172.21.15.114:0/780065535 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f51b40317e0 con 0x556936647800 2024-04-23T10:37:04.327 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:37:04.320+0000 7f51a2ffd640 1 -- 172.21.15.114:0/780065535 <== mon.0 v2:172.21.15.114:3300/0 3 ==== mon_get_version_reply(handle=1 version=37) ==== 24+0+0 (secure 0 0 0) 0x7f51bc02a430 con 0x556936647800 2024-04-23T10:37:04.327 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:37:04.320+0000 7f51ccbcd740 10 client.16800 fetch_fsmap learned FSMap version 37 2024-04-23T10:37:04.327 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:37:04.320+0000 7f51ccbcd740 1 -- 172.21.15.114:0/780065535 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({config=0+,fsmap=37,monmap=4+}) -- 0x556936656300 con 0x556936647800 2024-04-23T10:37:04.327 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:37:04.320+0000 7f51a2ffd640 1 -- 172.21.15.114:0/780065535 <== mon.0 v2:172.21.15.114:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f51b40317e0 con 0x556936647800 2024-04-23T10:37:04.327 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:37:04.320+0000 7f51a2ffd640 1 -- 172.21.15.114:0/780065535 <== mon.0 v2:172.21.15.114:3300/0 5 ==== fsmap(e 37) ==== 8358+0+0 (secure 0 0 0) 0x7f51bc020050 con 0x556936647800 2024-04-23T10:37:04.327 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:37:04.320+0000 7f51ccbcd740 10 client.16800 fetch_fsmap finished waiting for FSMap version 37 2024-04-23T10:37:04.327 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:37:04.320+0000 7f51ccbcd740 10 client.16800 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-04-23T10:37:04.327 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:37:04.320+0000 7f51ccbcd740 20 client.16800 populate_metadata read hostname 'smithi114' 2024-04-23T10:37:04.327 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:37:04.320+0000 7f51ccbcd740 1 --2- 172.21.15.114:0/780065535 >> [v2:172.21.15.119:6838/937297340,v1:172.21.15.119:6839/937297340] conn(0x556936657830 0x556936659c40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T10:37:04.327 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:37:04.320+0000 7f51ccbcd740 4 client.16800 mds_command: new command op to 24545 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-23T10:37:04.327 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:37:04.320+0000 7f51ccbcd740 1 -- 172.21.15.114:0/780065535 --> [v2:172.21.15.119:6838/937297340,v1:172.21.15.119:6839/937297340] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x5569365af390 con 0x556936657830 2024-04-23T10:37:04.327 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:37:04.320+0000 7f51c37fe640 1 --2- 172.21.15.114:0/780065535 >> [v2:172.21.15.119:6838/937297340,v1:172.21.15.119:6839/937297340] conn(0x556936657830 0x556936659c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T10:37:04.327 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:37:04.321+0000 7f51a2ffd640 10 client.16800 ms_handle_connect on v2:172.21.15.119:6838/937297340 2024-04-23T10:37:04.327 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:37:04.321+0000 7f51c37fe640 1 --2- 172.21.15.114:0/780065535 >> [v2:172.21.15.119:6838/937297340,v1:172.21.15.119:6839/937297340] conn(0x556936657830 0x556936659c40 crc :-1 s=READY pgs=154 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=d66c2a4054e6a494 server_cookie=6b318f4b58a8c486 in_seq=0 out_seq=0 2024-04-23T10:37:04.328 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:37:04.321+0000 7f51a2ffd640 1 -- 172.21.15.114:0/780065535 <== mds.0 v2:172.21.15.119:6838/937297340 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x5569365af390 con 0x556936657830 2024-04-23T10:37:04.328 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:37:04.321+0000 7f51a2ffd640 10 client.16800 handle_command_reply: tid=0 2024-04-23T10:37:04.328 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:37:04.321+0000 7f51ccbcd740 1 client.16800 shutdown 2024-04-23T10:37:04.328 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:37:04.321+0000 7f51ccbcd740 1 -- 172.21.15.114:0/780065535 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x556936647800 msgr2=0x5569366449c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T10:37:04.328 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:37:04.321+0000 7f51ccbcd740 1 --2- 172.21.15.114:0/780065535 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x556936647800 0x5569366449c0 secure :-1 s=READY pgs=846 cs=0 l=1 rev1=1 crypto rx=0x7f51bc00fec0 tx=0x7f51bc01d000 comp rx=0 tx=0).stop 2024-04-23T10:37:04.328 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:37:04.321+0000 7f51ccbcd740 1 -- 172.21.15.114:0/780065535 shutdown_connections 2024-04-23T10:37:04.328 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:37:04.321+0000 7f51ccbcd740 1 --2- 172.21.15.114:0/780065535 >> [v2:172.21.15.119:6838/937297340,v1:172.21.15.119:6839/937297340] conn(0x556936657830 0x556936659c40 crc :-1 s=READY pgs=154 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:37:04.328 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:37:04.321+0000 7f51ccbcd740 1 --2- 172.21.15.114:0/780065535 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x556936647800 0x5569366449c0 unknown :-1 s=CLOSED pgs=846 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:37:04.328 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:37:04.321+0000 7f51ccbcd740 1 --2- 172.21.15.114:0/780065535 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x55693663cf40 0x556936644460 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:37:04.328 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:37:04.321+0000 7f51a2ffd640 0 client.16800 ms_handle_reset on v2:172.21.15.119:6838/937297340 2024-04-23T10:37:04.328 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:37:04.321+0000 7f51ccbcd740 1 --2- 172.21.15.114:0/780065535 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x55693663c620 0x556936647170 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:37:04.328 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:37:04.321+0000 7f51ccbcd740 1 -- 172.21.15.114:0/780065535 >> 172.21.15.114:0/780065535 conn(0x556936638b30 msgr2=0x55693663e8b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T10:37:04.328 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:37:04.322+0000 7f51ccbcd740 1 -- 172.21.15.114:0/780065535 shutdown_connections 2024-04-23T10:37:04.328 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:37:04.322+0000 7f51ccbcd740 1 -- 172.21.15.114:0/780065535 wait complete. 2024-04-23T10:37:04.328 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:37:04.322+0000 7f51ccbcd740 20 client.16800 trim_cache size 0 max 16384 2024-04-23T10:37:04.340 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:37:04.337+0000 7f51ccbcd740 1 Processor -- start 2024-04-23T10:37:04.340 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:37:04.337+0000 7f51ccbcd740 1 -- start start 2024-04-23T10:37:04.340 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:37:04.337+0000 7f51ccbcd740 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x55693663c620 0x556936653290 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T10:37:04.340 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:37:04.337+0000 7f51ccbcd740 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x55693663cf40 0x556936637b30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T10:37:04.340 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:37:04.337+0000 7f51ccbcd740 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x556936647800 0x55693665f550 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T10:37:04.340 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:37:04.337+0000 7f51ccbcd740 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x7f51c41b2840 con 0x556936647800 2024-04-23T10:37:04.340 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:37:04.337+0000 7f51ccbcd740 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 -- 0x7f51c41b0050 con 0x55693663c620 2024-04-23T10:37:04.340 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:37:04.337+0000 7f51ccbcd740 1 -- --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_getmap magic: 0 -- 0x7f51c41adec0 con 0x55693663cf40 2024-04-23T10:37:04.340 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:37:04.337+0000 7f51c3fff640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x556936647800 0x55693665f550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T10:37:04.341 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:37:04.337+0000 7f51c3fff640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x556936647800 0x55693665f550 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.114:58228/0 (socket says 172.21.15.114:58228) 2024-04-23T10:37:04.341 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:37:04.337+0000 7f51c3fff640 1 -- 172.21.15.114:0/849770726 learned_addr learned my addr 172.21.15.114:0/849770726 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-04-23T10:37:04.341 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:37:04.337+0000 7f51c37fe640 1 --2- 172.21.15.114:0/849770726 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x55693663c620 0x556936653290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T10:37:04.341 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:37:04.337+0000 7f51c2ffd640 1 --2- 172.21.15.114:0/849770726 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x55693663cf40 0x556936637b30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T10:37:04.341 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:37:04.337+0000 7f51c3fff640 1 -- 172.21.15.114:0/849770726 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x55693663cf40 msgr2=0x556936637b30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T10:37:04.341 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:37:04.337+0000 7f51c3fff640 1 --2- 172.21.15.114:0/849770726 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x55693663cf40 0x556936637b30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:37:04.341 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:37:04.337+0000 7f51c3fff640 1 -- 172.21.15.114:0/849770726 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x55693663c620 msgr2=0x556936653290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T10:37:04.341 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:37:04.337+0000 7f51c3fff640 1 --2- 172.21.15.114:0/849770726 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x55693663c620 0x556936653290 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:37:04.341 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:37:04.337+0000 7f51c3fff640 1 -- 172.21.15.114:0/849770726 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5569366429b0 con 0x556936647800 2024-04-23T10:37:04.341 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:37:04.337+0000 7f51c3fff640 1 --2- 172.21.15.114:0/849770726 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x556936647800 0x55693665f550 secure :-1 s=READY pgs=847 cs=0 l=1 rev1=1 crypto rx=0x7f51bc00f8c0 tx=0x7f51bc005bd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=afb7d730de450899 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T10:37:04.341 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:37:04.337+0000 7f51a97fa640 1 -- 172.21.15.114:0/849770726 <== mon.0 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f51bc036020 con 0x556936647800 2024-04-23T10:37:04.341 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:37:04.337+0000 7f51a97fa640 1 -- 172.21.15.114:0/849770726 <== mon.0 v2:172.21.15.114:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f51bc013330 con 0x556936647800 2024-04-23T10:37:04.341 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:37:04.337+0000 7f51a97fa640 1 -- 172.21.15.114:0/849770726 <== mon.0 v2:172.21.15.114:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f51bc003970 con 0x556936647800 2024-04-23T10:37:04.341 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:37:04.338+0000 7f51ccbcd740 1 -- 172.21.15.114:0/849770726 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x556936647800 msgr2=0x55693665f550 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T10:37:04.342 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:37:04.338+0000 7f51ccbcd740 1 --2- 172.21.15.114:0/849770726 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x556936647800 0x55693665f550 secure :-1 s=READY pgs=847 cs=0 l=1 rev1=1 crypto rx=0x7f51bc00f8c0 tx=0x7f51bc005bd0 comp rx=0 tx=0).stop 2024-04-23T10:37:04.342 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:37:04.338+0000 7f51ccbcd740 1 -- 172.21.15.114:0/849770726 shutdown_connections 2024-04-23T10:37:04.342 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:37:04.338+0000 7f51ccbcd740 1 --2- 172.21.15.114:0/849770726 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x556936647800 0x55693665f550 unknown :-1 s=CLOSED pgs=847 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:37:04.342 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:37:04.338+0000 7f51ccbcd740 1 --2- 172.21.15.114:0/849770726 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x55693663cf40 0x556936637b30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:37:04.342 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:37:04.338+0000 7f51ccbcd740 1 --2- 172.21.15.114:0/849770726 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x55693663c620 0x556936653290 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:37:04.342 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:37:04.338+0000 7f51ccbcd740 1 -- 172.21.15.114:0/849770726 >> 172.21.15.114:0/849770726 conn(0x556936638b30 msgr2=0x556936637420 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T10:37:04.342 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:37:04.338+0000 7f51ccbcd740 1 -- 172.21.15.114:0/849770726 shutdown_connections 2024-04-23T10:37:04.342 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:37:04.338+0000 7f51ccbcd740 1 -- 172.21.15.114:0/849770726 wait complete. 2024-04-23T10:37:04.342 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:37:04.338+0000 7f51ccbcd740 1 -- start start 2024-04-23T10:37:04.342 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:37:04.339+0000 7f51ccbcd740 1 Processor -- start 2024-04-23T10:37:04.342 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:37:04.339+0000 7f51ccbcd740 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x55693663c620 0x55693664f980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T10:37:04.342 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:37:04.339+0000 7f51ccbcd740 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x55693663cf40 0x55693664cd10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T10:37:04.342 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:37:04.339+0000 7f51ccbcd740 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x556936647800 0x55693664d250 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T10:37:04.342 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:37:04.339+0000 7f51ccbcd740 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x7f51c41ae510 con 0x556936647800 2024-04-23T10:37:04.343 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:37:04.339+0000 7f51ccbcd740 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 -- 0x7f51c41af980 con 0x55693663cf40 2024-04-23T10:37:04.343 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:37:04.339+0000 7f51ccbcd740 1 -- --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_getmap magic: 0 -- 0x7f51c41aebe0 con 0x55693663c620 2024-04-23T10:37:04.343 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:37:04.339+0000 7f51ccbcd740 10 client.0 _ll_register_callbacks cb 0x7f51bc00fb70 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-23T10:37:04.343 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:37:04.339+0000 7f51c3fff640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x556936647800 0x55693664d250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T10:37:04.343 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:37:04.339+0000 7f51c3fff640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x556936647800 0x55693664d250 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.114:58238/0 (socket says 172.21.15.114:58238) 2024-04-23T10:37:04.343 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:37:04.339+0000 7f51c3fff640 1 -- 172.21.15.114:0/2981283072 learned_addr learned my addr 172.21.15.114:0/2981283072 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-04-23T10:37:04.343 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:37:04.339+0000 7f51c37fe640 1 --2- 172.21.15.114:0/2981283072 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x55693663c620 0x55693664f980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T10:37:04.343 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:37:04.339+0000 7f51c2ffd640 1 --2- 172.21.15.114:0/2981283072 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x55693663cf40 0x55693664cd10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T10:37:04.343 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:37:04.339+0000 7f51c3fff640 1 -- 172.21.15.114:0/2981283072 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x55693663c620 msgr2=0x55693664f980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T10:37:04.343 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:37:04.339+0000 7f51c3fff640 1 --2- 172.21.15.114:0/2981283072 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x55693663c620 0x55693664f980 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:37:04.343 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:37:04.339+0000 7f51c3fff640 1 -- 172.21.15.114:0/2981283072 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x55693663cf40 msgr2=0x55693664cd10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T10:37:04.343 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:37:04.339+0000 7f51c3fff640 1 --2- 172.21.15.114:0/2981283072 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x55693663cf40 0x55693664cd10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:37:04.343 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:37:04.339+0000 7f51c3fff640 1 -- 172.21.15.114:0/2981283072 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55693664d890 con 0x556936647800 2024-04-23T10:37:04.343 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:37:04.339+0000 7f51c3fff640 1 --2- 172.21.15.114:0/2981283072 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x556936647800 0x55693664d250 secure :-1 s=READY pgs=848 cs=0 l=1 rev1=1 crypto rx=0x7f51bc022040 tx=0x7f51bc009500 comp rx=0 tx=0).ready entity=mon.0 client_cookie=129ae3712328e666 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T10:37:04.344 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:37:04.339+0000 7f51a2ffd640 10 client.0 ms_handle_connect on v2:172.21.15.114:3300/0 2024-04-23T10:37:04.344 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:37:04.339+0000 7f51a2ffd640 1 -- 172.21.15.114:0/2981283072 <== mon.0 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f51bc036070 con 0x556936647800 2024-04-23T10:37:04.344 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:37:04.339+0000 7f51a2ffd640 1 -- 172.21.15.114:0/2981283072 <== mon.0 v2:172.21.15.114:3300/0 2 ==== osd_map(102..102 src has 1..102) ==== 8629+0+0 (secure 0 0 0) 0x7f51bc02c320 con 0x556936647800 2024-04-23T10:37:04.344 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:37:04.340+0000 7f51ccbcd740 1 -- 172.21.15.114:0/2981283072 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f51bc013330 con 0x556936647800 2024-04-23T10:37:04.344 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:37:04.340+0000 7f51a2ffd640 1 -- 172.21.15.114:0/2981283072 <== mon.0 v2:172.21.15.114:3300/0 3 ==== mon_get_version_reply(handle=1 version=37) ==== 24+0+0 (secure 0 0 0) 0x7f51bc03f050 con 0x556936647800 2024-04-23T10:37:04.344 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:37:04.340+0000 7f51ccbcd740 10 client.16806 fetch_fsmap learned FSMap version 37 2024-04-23T10:37:04.344 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:37:04.340+0000 7f51ccbcd740 1 -- 172.21.15.114:0/2981283072 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({config=0+,fsmap=37,monmap=4+}) -- 0x5569366bdbf0 con 0x556936647800 2024-04-23T10:37:04.344 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:37:04.340+0000 7f51a2ffd640 1 -- 172.21.15.114:0/2981283072 <== mon.0 v2:172.21.15.114:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f51bc013330 con 0x556936647800 2024-04-23T10:37:04.344 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:37:04.340+0000 7f51a2ffd640 1 -- 172.21.15.114:0/2981283072 <== mon.0 v2:172.21.15.114:3300/0 5 ==== fsmap(e 37) ==== 8358+0+0 (secure 0 0 0) 0x7f51bc039020 con 0x556936647800 2024-04-23T10:37:04.344 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:37:04.340+0000 7f51ccbcd740 10 client.16806 fetch_fsmap finished waiting for FSMap version 37 2024-04-23T10:37:04.344 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:37:04.340+0000 7f51ccbcd740 10 client.16806 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-04-23T10:37:04.344 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:37:04.340+0000 7f51ccbcd740 20 client.16806 populate_metadata read hostname 'smithi114' 2024-04-23T10:37:04.344 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:37:04.340+0000 7f51ccbcd740 1 --2- 172.21.15.114:0/2981283072 >> [v2:172.21.15.119:6838/937297340,v1:172.21.15.119:6839/937297340] conn(0x5569366be250 0x5569366c0670 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T10:37:04.344 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:37:04.340+0000 7f51ccbcd740 4 client.16806 mds_command: new command op to 24545 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-04-23T10:37:04.344 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:37:04.340+0000 7f51ccbcd740 1 -- 172.21.15.114:0/2981283072 --> [v2:172.21.15.119:6838/937297340,v1:172.21.15.119:6839/937297340] -- command(tid 0: {"prefix": "flush journal"}) -- 0x5569365a9200 con 0x5569366be250 2024-04-23T10:37:04.345 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:37:04.340+0000 7f51c37fe640 1 --2- 172.21.15.114:0/2981283072 >> [v2:172.21.15.119:6838/937297340,v1:172.21.15.119:6839/937297340] conn(0x5569366be250 0x5569366c0670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T10:37:04.345 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:37:04.341+0000 7f51c37fe640 1 --2- 172.21.15.114:0/2981283072 >> [v2:172.21.15.119:6838/937297340,v1:172.21.15.119:6839/937297340] conn(0x5569366be250 0x5569366c0670 crc :-1 s=READY pgs=155 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=8b069fa5139deed8 server_cookie=ab0bde2f0423033a in_seq=0 out_seq=0 2024-04-23T10:37:04.345 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:37:04.341+0000 7f51a2ffd640 10 client.16806 ms_handle_connect on v2:172.21.15.119:6838/937297340 2024-04-23T10:37:04.374 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:37:04.375+0000 7f51a2ffd640 1 -- 172.21.15.114:0/2981283072 <== mds.0 v2:172.21.15.119:6838/937297340 1 ==== command_reply(tid 0: 0 ) ==== 8+0+44 (crc 0 0 0) 0x5569365a9200 con 0x5569366be250 2024-04-23T10:37:04.374 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:37:04.375+0000 7f51a2ffd640 10 client.16806 handle_command_reply: tid=0 2024-04-23T10:37:04.374 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:37:04.375+0000 7f51ccbcd740 1 client.16806 shutdown 2024-04-23T10:37:04.374 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:37:04.375+0000 7f51ccbcd740 1 -- 172.21.15.114:0/2981283072 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x556936647800 msgr2=0x55693664d250 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T10:37:04.375 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:37:04.375+0000 7f51ccbcd740 1 --2- 172.21.15.114:0/2981283072 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x556936647800 0x55693664d250 secure :-1 s=READY pgs=848 cs=0 l=1 rev1=1 crypto rx=0x7f51bc022040 tx=0x7f51bc009500 comp rx=0 tx=0).stop 2024-04-23T10:37:04.375 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:37:04.375+0000 7f51ccbcd740 1 -- 172.21.15.114:0/2981283072 shutdown_connections 2024-04-23T10:37:04.375 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:37:04.375+0000 7f51ccbcd740 1 --2- 172.21.15.114:0/2981283072 >> [v2:172.21.15.119:6838/937297340,v1:172.21.15.119:6839/937297340] conn(0x5569366be250 0x5569366c0670 crc :-1 s=READY pgs=155 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:37:04.375 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:37:04.375+0000 7f51ccbcd740 1 --2- 172.21.15.114:0/2981283072 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x556936647800 0x55693664d250 unknown :-1 s=CLOSED pgs=848 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:37:04.375 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:37:04.375+0000 7f51ccbcd740 1 --2- 172.21.15.114:0/2981283072 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x55693663cf40 0x55693664cd10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:37:04.375 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:37:04.375+0000 7f51ccbcd740 1 --2- 172.21.15.114:0/2981283072 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x55693663c620 0x55693664f980 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:37:04.375 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:37:04.375+0000 7f51ccbcd740 1 -- 172.21.15.114:0/2981283072 >> 172.21.15.114:0/2981283072 conn(0x556936638b30 msgr2=0x556936637cb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T10:37:04.375 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:37:04.375+0000 7f51a2ffd640 0 client.16806 ms_handle_reset on v2:172.21.15.119:6838/937297340 2024-04-23T10:37:04.375 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:37:04.375+0000 7f51ccbcd740 1 -- 172.21.15.114:0/2981283072 shutdown_connections 2024-04-23T10:37:04.375 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:37:04.375+0000 7f51ccbcd740 1 -- 172.21.15.114:0/2981283072 wait complete. 2024-04-23T10:37:04.375 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:37:04.375+0000 7f51ccbcd740 20 client.16806 trim_cache size 0 max 16384 2024-04-23T10:37:04.376 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stdout:{ 2024-04-23T10:37:04.376 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stdout: "message": "", 2024-04-23T10:37:04.376 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stdout: "return_code": 0 2024-04-23T10:37:04.376 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stdout:} 2024-04-23T10:37:04.376 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:37:04.376+0000 7f51c9eb9640 1 -- 172.21.15.114:0/3568019809 >> [v2:172.21.15.114:6800/1395693466,v1:172.21.15.114:6801/1395693466] conn(0x7f51ac0a3dc0 msgr2=0x7f51ac0a6280 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T10:37:04.377 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:37:04.376+0000 7f51c9eb9640 1 --2- 172.21.15.114:0/3568019809 >> [v2:172.21.15.114:6800/1395693466,v1:172.21.15.114:6801/1395693466] conn(0x7f51ac0a3dc0 0x7f51ac0a6280 secure :-1 s=READY pgs=131 cs=0 l=1 rev1=1 crypto rx=0x7f51bc009fd0 tx=0x7f51bc009d80 comp rx=0 tx=0).stop 2024-04-23T10:37:04.377 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:37:04.376+0000 7f51c9eb9640 1 -- 172.21.15.114:0/3568019809 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f51c4101680 msgr2=0x7f51c41a0b70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T10:37:04.377 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:37:04.376+0000 7f51c9eb9640 1 --2- 172.21.15.114:0/3568019809 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f51c4101680 0x7f51c41a0b70 secure :-1 s=READY pgs=845 cs=0 l=1 rev1=1 crypto rx=0x7f51b800e9f0 tx=0x7f51b800edf0 comp rx=0 tx=0).stop 2024-04-23T10:37:04.377 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:37:04.376+0000 7f51c9eb9640 1 -- 172.21.15.114:0/3568019809 shutdown_connections 2024-04-23T10:37:04.377 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:37:04.376+0000 7f51c9eb9640 1 --2- 172.21.15.114:0/3568019809 >> [v2:172.21.15.114:6800/1395693466,v1:172.21.15.114:6801/1395693466] conn(0x7f51ac0a3dc0 0x7f51ac0a6280 unknown :-1 s=CLOSED pgs=131 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:37:04.377 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:37:04.376+0000 7f51c9eb9640 1 --2- 172.21.15.114:0/3568019809 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f51c410e910 0x7f51c419a740 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:37:04.377 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:37:04.376+0000 7f51c9eb9640 1 --2- 172.21.15.114:0/3568019809 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f51c4101680 0x7f51c41a0b70 unknown :-1 s=CLOSED pgs=845 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:37:04.377 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:37:04.376+0000 7f51c9eb9640 1 --2- 172.21.15.114:0/3568019809 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f51c4100d60 0x7f51c41a0630 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:37:04.377 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:37:04.376+0000 7f51c9eb9640 1 -- 172.21.15.114:0/3568019809 >> 172.21.15.114:0/3568019809 conn(0x7f51c4077fd0 msgr2=0x7f51c410ffe0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T10:37:04.377 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:37:04.376+0000 7f51c9eb9640 1 -- 172.21.15.114:0/3568019809 shutdown_connections 2024-04-23T10:37:04.377 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:37:04.376+0000 7f51c9eb9640 1 -- 172.21.15.114:0/3568019809 wait complete. 2024-04-23T10:37:04.492 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 23 10:37:04 smithi119 ceph-mon[32608]: pgmap v326: 97 pgs: 97 active+clean; 204 MiB data, 2.5 GiB used, 1.0 TiB / 1.0 TiB avail; 396 KiB/s rd, 878 KiB/s wr, 80 op/s 2024-04-23T10:37:04.503 INFO:journalctl@ceph.mon.a.smithi114.stdout:Apr 23 10:37:04 smithi114 ceph-mon[29255]: pgmap v326: 97 pgs: 97 active+clean; 204 MiB data, 2.5 GiB used, 1.0 TiB / 1.0 TiB avail; 396 KiB/s rd, 878 KiB/s wr, 80 op/s 2024-04-23T10:37:04.577 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 23 10:37:04 smithi204 ceph-mon[32649]: pgmap v326: 97 pgs: 97 active+clean; 204 MiB data, 2.5 GiB used, 1.0 TiB / 1.0 TiB avail; 396 KiB/s rd, 878 KiB/s wr, 80 op/s 2024-04-23T10:37:04.940 INFO:tasks.workunit.client.0.smithi114.stdout: 1 660946 6.05 MB/sec execute 433 sec latency 4897.840 ms 2024-04-23T10:37:05.940 INFO:tasks.workunit.client.0.smithi114.stdout: 1 662574 6.05 MB/sec execute 434 sec latency 48.217 ms 2024-04-23T10:37:06.492 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 23 10:37:06 smithi119 ceph-mon[32608]: pgmap v327: 97 pgs: 97 active+clean; 204 MiB data, 2.5 GiB used, 1.0 TiB / 1.0 TiB avail; 305 KiB/s rd, 597 KiB/s wr, 55 op/s 2024-04-23T10:37:06.504 INFO:journalctl@ceph.mon.a.smithi114.stdout:Apr 23 10:37:06 smithi114 ceph-mon[29255]: pgmap v327: 97 pgs: 97 active+clean; 204 MiB data, 2.5 GiB used, 1.0 TiB / 1.0 TiB avail; 305 KiB/s rd, 597 KiB/s wr, 55 op/s 2024-04-23T10:37:06.578 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 23 10:37:06 smithi204 ceph-mon[32649]: pgmap v327: 97 pgs: 97 active+clean; 204 MiB data, 2.5 GiB used, 1.0 TiB / 1.0 TiB avail; 305 KiB/s rd, 597 KiB/s wr, 55 op/s 2024-04-23T10:37:06.940 INFO:tasks.workunit.client.0.smithi114.stdout: 1 664146 6.07 MB/sec execute 435 sec latency 431.952 ms 2024-04-23T10:37:07.940 INFO:tasks.workunit.client.0.smithi114.stdout: 1 664146 6.05 MB/sec execute 436 sec latency 1432.052 ms 2024-04-23T10:37:08.492 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 23 10:37:08 smithi119 ceph-mon[32608]: pgmap v328: 97 pgs: 97 active+clean; 203 MiB data, 2.5 GiB used, 1.0 TiB / 1.0 TiB avail; 516 KiB/s rd, 1.0 MiB/s wr, 139 op/s 2024-04-23T10:37:08.504 INFO:journalctl@ceph.mon.a.smithi114.stdout:Apr 23 10:37:08 smithi114 ceph-mon[29255]: pgmap v328: 97 pgs: 97 active+clean; 203 MiB data, 2.5 GiB used, 1.0 TiB / 1.0 TiB avail; 516 KiB/s rd, 1.0 MiB/s wr, 139 op/s 2024-04-23T10:37:08.577 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 23 10:37:08 smithi204 ceph-mon[32649]: pgmap v328: 97 pgs: 97 active+clean; 203 MiB data, 2.5 GiB used, 1.0 TiB / 1.0 TiB avail; 516 KiB/s rd, 1.0 MiB/s wr, 139 op/s 2024-04-23T10:37:08.940 INFO:tasks.workunit.client.0.smithi114.stdout: 1 664146 6.04 MB/sec execute 437 sec latency 2432.143 ms 2024-04-23T10:37:09.940 INFO:tasks.workunit.client.0.smithi114.stdout: 1 664213 6.02 MB/sec execute 438 sec latency 2692.537 ms 2024-04-23T10:37:10.492 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 23 10:37:10 smithi119 ceph-mon[32608]: pgmap v329: 97 pgs: 97 active+clean; 203 MiB data, 2.5 GiB used, 1.0 TiB / 1.0 TiB avail; 304 KiB/s rd, 741 KiB/s wr, 101 op/s 2024-04-23T10:37:10.492 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 23 10:37:10 smithi119 ceph-mon[32608]: osdmap e103: 12 total, 12 up, 12 in 2024-04-23T10:37:10.504 INFO:journalctl@ceph.mon.a.smithi114.stdout:Apr 23 10:37:10 smithi114 ceph-mon[29255]: pgmap v329: 97 pgs: 97 active+clean; 203 MiB data, 2.5 GiB used, 1.0 TiB / 1.0 TiB avail; 304 KiB/s rd, 741 KiB/s wr, 101 op/s 2024-04-23T10:37:10.504 INFO:journalctl@ceph.mon.a.smithi114.stdout:Apr 23 10:37:10 smithi114 ceph-mon[29255]: osdmap e103: 12 total, 12 up, 12 in 2024-04-23T10:37:10.578 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 23 10:37:10 smithi204 ceph-mon[32649]: pgmap v329: 97 pgs: 97 active+clean; 203 MiB data, 2.5 GiB used, 1.0 TiB / 1.0 TiB avail; 304 KiB/s rd, 741 KiB/s wr, 101 op/s 2024-04-23T10:37:10.578 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 23 10:37:10 smithi204 ceph-mon[32649]: osdmap e103: 12 total, 12 up, 12 in 2024-04-23T10:37:10.940 INFO:tasks.workunit.client.0.smithi114.stdout: 1 664213 6.01 MB/sec execute 439 sec latency 1523.851 ms 2024-04-23T10:37:11.941 INFO:tasks.workunit.client.0.smithi114.stdout: 1 664213 6.00 MB/sec execute 440 sec latency 2523.965 ms 2024-04-23T10:37:12.492 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 23 10:37:12 smithi119 ceph-mon[32608]: pgmap v331: 97 pgs: 97 active+clean; 203 MiB data, 2.5 GiB used, 1.0 TiB / 1.0 TiB avail; 364 KiB/s rd, 889 KiB/s wr, 122 op/s 2024-04-23T10:37:12.504 INFO:journalctl@ceph.mon.a.smithi114.stdout:Apr 23 10:37:12 smithi114 ceph-mon[29255]: pgmap v331: 97 pgs: 97 active+clean; 203 MiB data, 2.5 GiB used, 1.0 TiB / 1.0 TiB avail; 364 KiB/s rd, 889 KiB/s wr, 122 op/s 2024-04-23T10:37:12.578 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 23 10:37:12 smithi204 ceph-mon[32649]: pgmap v331: 97 pgs: 97 active+clean; 203 MiB data, 2.5 GiB used, 1.0 TiB / 1.0 TiB avail; 364 KiB/s rd, 889 KiB/s wr, 122 op/s 2024-04-23T10:37:12.941 INFO:tasks.workunit.client.0.smithi114.stdout: 1 664213 5.98 MB/sec execute 441 sec latency 3524.005 ms 2024-04-23T10:37:13.941 INFO:tasks.workunit.client.0.smithi114.stdout: 1 664213 5.97 MB/sec execute 442 sec latency 4524.057 ms 2024-04-23T10:37:14.492 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 23 10:37:14 smithi119 ceph-mon[32608]: pgmap v332: 97 pgs: 97 active+clean; 203 MiB data, 2.6 GiB used, 1.0 TiB / 1.0 TiB avail; 401 KiB/s rd, 1016 KiB/s wr, 124 op/s 2024-04-23T10:37:14.492 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 23 10:37:14 smithi119 ceph-mon[32608]: osdmap e104: 12 total, 12 up, 12 in 2024-04-23T10:37:14.503 INFO:journalctl@ceph.mon.a.smithi114.stdout:Apr 23 10:37:14 smithi114 ceph-mon[29255]: pgmap v332: 97 pgs: 97 active+clean; 203 MiB data, 2.6 GiB used, 1.0 TiB / 1.0 TiB avail; 401 KiB/s rd, 1016 KiB/s wr, 124 op/s 2024-04-23T10:37:14.504 INFO:journalctl@ceph.mon.a.smithi114.stdout:Apr 23 10:37:14 smithi114 ceph-mon[29255]: osdmap e104: 12 total, 12 up, 12 in 2024-04-23T10:37:14.578 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 23 10:37:14 smithi204 ceph-mon[32649]: pgmap v332: 97 pgs: 97 active+clean; 203 MiB data, 2.6 GiB used, 1.0 TiB / 1.0 TiB avail; 401 KiB/s rd, 1016 KiB/s wr, 124 op/s 2024-04-23T10:37:14.578 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 23 10:37:14 smithi204 ceph-mon[32649]: osdmap e104: 12 total, 12 up, 12 in 2024-04-23T10:37:14.941 INFO:tasks.workunit.client.0.smithi114.stdout: 1 665017 5.96 MB/sec execute 443 sec latency 4765.112 ms 2024-04-23T10:37:15.941 INFO:tasks.workunit.client.0.smithi114.stdout: 1 667681 5.99 MB/sec execute 444 sec latency 14.652 ms 2024-04-23T10:37:16.492 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 23 10:37:16 smithi119 ceph-mon[32608]: pgmap v334: 97 pgs: 97 active+clean; 203 MiB data, 2.6 GiB used, 1.0 TiB / 1.0 TiB avail; 187 KiB/s rd, 556 KiB/s wr, 29 op/s 2024-04-23T10:37:16.503 INFO:journalctl@ceph.mon.a.smithi114.stdout:Apr 23 10:37:16 smithi114 ceph-mon[29255]: pgmap v334: 97 pgs: 97 active+clean; 203 MiB data, 2.6 GiB used, 1.0 TiB / 1.0 TiB avail; 187 KiB/s rd, 556 KiB/s wr, 29 op/s 2024-04-23T10:37:16.577 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 23 10:37:16 smithi204 ceph-mon[32649]: pgmap v334: 97 pgs: 97 active+clean; 203 MiB data, 2.6 GiB used, 1.0 TiB / 1.0 TiB avail; 187 KiB/s rd, 556 KiB/s wr, 29 op/s 2024-04-23T10:37:16.941 INFO:tasks.workunit.client.0.smithi114.stdout: 1 667690 5.97 MB/sec execute 445 sec latency 991.373 ms 2024-04-23T10:37:17.473 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:37:17.472+0000 7f54ed9f0640 1 -- 172.21.15.114:0/2598601919 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f54e810c550 msgr2=0x7f54e810e9d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T10:37:17.473 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:37:17.472+0000 7f54ed9f0640 1 --2- 172.21.15.114:0/2598601919 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f54e810c550 0x7f54e810e9d0 secure :-1 s=READY pgs=849 cs=0 l=1 rev1=1 crypto rx=0x7f54e0009ca0 tx=0x7f54e002a950 comp rx=0 tx=0).stop 2024-04-23T10:37:17.473 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:37:17.472+0000 7f54ed9f0640 1 -- 172.21.15.114:0/2598601919 shutdown_connections 2024-04-23T10:37:17.474 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:37:17.472+0000 7f54ed9f0640 1 --2- 172.21.15.114:0/2598601919 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f54e810c550 0x7f54e810e9d0 unknown :-1 s=CLOSED pgs=849 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:37:17.474 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:37:17.472+0000 7f54ed9f0640 1 --2- 172.21.15.114:0/2598601919 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f54e8103aa0 0x7f54e810bf80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:37:17.474 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:37:17.472+0000 7f54ed9f0640 1 --2- 172.21.15.114:0/2598601919 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f54e8103180 0x7f54e8103560 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:37:17.474 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:37:17.472+0000 7f54ed9f0640 1 -- 172.21.15.114:0/2598601919 >> 172.21.15.114:0/2598601919 conn(0x7f54e8077fd0 msgr2=0x7f54e80783e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T10:37:17.474 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:37:17.472+0000 7f54ed9f0640 1 -- 172.21.15.114:0/2598601919 shutdown_connections 2024-04-23T10:37:17.474 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:37:17.472+0000 7f54ed9f0640 1 -- 172.21.15.114:0/2598601919 wait complete. 2024-04-23T10:37:17.474 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:37:17.473+0000 7f54ed9f0640 1 Processor -- start 2024-04-23T10:37:17.474 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:37:17.473+0000 7f54ed9f0640 1 -- start start 2024-04-23T10:37:17.474 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:37:17.473+0000 7f54ed9f0640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f54e8103180 0x7f54e8198660 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T10:37:17.474 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:37:17.473+0000 7f54ed9f0640 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f54e8103aa0 0x7f54e8198ba0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T10:37:17.474 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:37:17.473+0000 7f54ed9f0640 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f54e810c550 0x7f54e819cf20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T10:37:17.474 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:37:17.473+0000 7f54ed9f0640 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x7f54e81121c0 con 0x7f54e8103180 2024-04-23T10:37:17.474 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:37:17.473+0000 7f54ed9f0640 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 -- 0x7f54e8112040 con 0x7f54e8103aa0 2024-04-23T10:37:17.474 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:37:17.473+0000 7f54ed9f0640 1 -- --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_getmap magic: 0 -- 0x7f54e8112340 con 0x7f54e810c550 2024-04-23T10:37:17.475 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:37:17.473+0000 7f54e6ffd640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f54e8103180 0x7f54e8198660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T10:37:17.475 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:37:17.473+0000 7f54e6ffd640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f54e8103180 0x7f54e8198660 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.114:48522/0 (socket says 172.21.15.114:48522) 2024-04-23T10:37:17.475 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:37:17.473+0000 7f54e6ffd640 1 -- 172.21.15.114:0/2042188437 learned_addr learned my addr 172.21.15.114:0/2042188437 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-04-23T10:37:17.475 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:37:17.473+0000 7f54e77fe640 1 --2- 172.21.15.114:0/2042188437 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f54e810c550 0x7f54e819cf20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T10:37:17.475 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:37:17.473+0000 7f54dffff640 1 --2- 172.21.15.114:0/2042188437 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f54e8103aa0 0x7f54e8198ba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T10:37:17.475 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:37:17.474+0000 7f54e6ffd640 1 -- 172.21.15.114:0/2042188437 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f54e810c550 msgr2=0x7f54e819cf20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T10:37:17.475 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:37:17.474+0000 7f54e6ffd640 1 --2- 172.21.15.114:0/2042188437 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f54e810c550 0x7f54e819cf20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:37:17.475 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:37:17.474+0000 7f54e6ffd640 1 -- 172.21.15.114:0/2042188437 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f54e8103aa0 msgr2=0x7f54e8198ba0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T10:37:17.475 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:37:17.474+0000 7f54e6ffd640 1 --2- 172.21.15.114:0/2042188437 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f54e8103aa0 0x7f54e8198ba0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:37:17.475 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:37:17.474+0000 7f54e6ffd640 1 -- 172.21.15.114:0/2042188437 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f54e819d650 con 0x7f54e8103180 2024-04-23T10:37:17.475 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:37:17.474+0000 7f54dffff640 1 --2- 172.21.15.114:0/2042188437 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f54e8103aa0 0x7f54e8198ba0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-23T10:37:17.475 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:37:17.474+0000 7f54e6ffd640 1 --2- 172.21.15.114:0/2042188437 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f54e8103180 0x7f54e8198660 secure :-1 s=READY pgs=850 cs=0 l=1 rev1=1 crypto rx=0x7f54d800e9f0 tx=0x7f54d800edf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T10:37:17.475 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:37:17.474+0000 7f54ec9ee640 1 -- 172.21.15.114:0/2042188437 <== mon.0 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f54d800cd80 con 0x7f54e8103180 2024-04-23T10:37:17.475 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:37:17.474+0000 7f54ec9ee640 1 -- 172.21.15.114:0/2042188437 <== mon.0 v2:172.21.15.114:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f54d80054d0 con 0x7f54e8103180 2024-04-23T10:37:17.476 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:37:17.474+0000 7f54ed9f0640 1 -- 172.21.15.114:0/2042188437 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f54e819d940 con 0x7f54e8103180 2024-04-23T10:37:17.476 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:37:17.474+0000 7f54ec9ee640 1 -- 172.21.15.114:0/2042188437 <== mon.0 v2:172.21.15.114:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f54d8010690 con 0x7f54e8103180 2024-04-23T10:37:17.476 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:37:17.474+0000 7f54ed9f0640 1 -- 172.21.15.114:0/2042188437 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f54e81a51d0 con 0x7f54e8103180 2024-04-23T10:37:17.483 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:37:17.476+0000 7f54ec9ee640 1 -- 172.21.15.114:0/2042188437 <== mon.0 v2:172.21.15.114:3300/0 4 ==== mgrmap(e 26) ==== 138778+0+0 (secure 0 0 0) 0x7f54d8004e90 con 0x7f54e8103180 2024-04-23T10:37:17.483 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:37:17.477+0000 7f54f07de740 1 Processor -- start 2024-04-23T10:37:17.483 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:37:17.477+0000 7f54f07de740 1 -- start start 2024-04-23T10:37:17.483 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:37:17.477+0000 7f54f07de740 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x558c6fb35620 0x558c6fb35a00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T10:37:17.483 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:37:17.477+0000 7f54f07de740 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x558c6fb35f40 0x558c6fb402c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T10:37:17.484 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:37:17.477+0000 7f54ec9ee640 1 --2- 172.21.15.114:0/2042188437 >> [v2:172.21.15.114:6800/1395693466,v1:172.21.15.114:6801/1395693466] conn(0x7f54d40a3dc0 0x7f54d40a6280 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T10:37:17.484 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:37:17.477+0000 7f54f07de740 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x558c6fb40800 0x558c6fb42bf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T10:37:17.484 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:37:17.477+0000 7f54f07de740 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x558c6fa9a970 con 0x558c6fb35f40 2024-04-23T10:37:17.484 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:37:17.477+0000 7f54f07de740 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 -- 0x558c6fa374a0 con 0x558c6fb40800 2024-04-23T10:37:17.484 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:37:17.477+0000 7f54f07de740 1 -- --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_getmap magic: 0 -- 0x558c6fa96930 con 0x558c6fb35620 2024-04-23T10:37:17.484 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:37:17.477+0000 7f54dffff640 1 --2- 172.21.15.114:0/2042188437 >> [v2:172.21.15.114:6800/1395693466,v1:172.21.15.114:6801/1395693466] conn(0x7f54d40a3dc0 0x7f54d40a6280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T10:37:17.484 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:37:17.477+0000 7f54e6ffd640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x558c6fb35f40 0x558c6fb402c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T10:37:17.484 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:37:17.477+0000 7f54e6ffd640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x558c6fb35f40 0x558c6fb402c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.114:48536/0 (socket says 172.21.15.114:48536) 2024-04-23T10:37:17.484 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:37:17.477+0000 7f54e6ffd640 1 -- 172.21.15.114:0/3014612208 learned_addr learned my addr 172.21.15.114:0/3014612208 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-04-23T10:37:17.484 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:37:17.477+0000 7f54e77fe640 1 --2- 172.21.15.114:0/3014612208 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x558c6fb40800 0x558c6fb42bf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T10:37:17.484 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:37:17.477+0000 7f54ec9ee640 1 -- 172.21.15.114:0/2042188437 <== mon.0 v2:172.21.15.114:3300/0 5 ==== osd_map(104..104 src has 1..104) ==== 8629+0+0 (secure 0 0 0) 0x7f54d80d0d10 con 0x7f54e8103180 2024-04-23T10:37:17.484 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:37:17.477+0000 7f54e77fe640 1 --2- 172.21.15.114:0/3014612208 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x558c6fb35620 0x558c6fb35a00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T10:37:17.484 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:37:17.477+0000 7f54e6ffd640 1 -- 172.21.15.114:0/3014612208 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x558c6fb35620 msgr2=0x558c6fb35a00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T10:37:17.485 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:37:17.477+0000 7f54e6ffd640 1 --2- 172.21.15.114:0/3014612208 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x558c6fb35620 0x558c6fb35a00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:37:17.485 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:37:17.477+0000 7f54e6ffd640 1 -- 172.21.15.114:0/3014612208 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x558c6fb40800 msgr2=0x558c6fb42bf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T10:37:17.485 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:37:17.477+0000 7f54e6ffd640 1 --2- 172.21.15.114:0/3014612208 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x558c6fb40800 0x558c6fb42bf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:37:17.485 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:37:17.477+0000 7f54e6ffd640 1 -- 172.21.15.114:0/3014612208 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x558c6f7ad210 con 0x558c6fb35f40 2024-04-23T10:37:17.485 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:37:17.477+0000 7f54e77fe640 1 --2- 172.21.15.114:0/3014612208 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x558c6fb40800 0x558c6fb42bf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-23T10:37:17.485 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:37:17.477+0000 7f54dffff640 1 --2- 172.21.15.114:0/2042188437 >> [v2:172.21.15.114:6800/1395693466,v1:172.21.15.114:6801/1395693466] conn(0x7f54d40a3dc0 0x7f54d40a6280 secure :-1 s=READY pgs=132 cs=0 l=1 rev1=1 crypto rx=0x7f54e810e210 tx=0x7f54d0005f90 comp rx=0 tx=0).ready entity=mgr.15291 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T10:37:17.485 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:37:17.478+0000 7f54e6ffd640 1 --2- 172.21.15.114:0/3014612208 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x558c6fb35f40 0x558c6fb402c0 secure :-1 s=READY pgs=851 cs=0 l=1 rev1=1 crypto rx=0x7f54d809cbd0 tx=0x7f54d8004a90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f766915f000d725e server_cookie=0 in_seq=0 out_seq=0 2024-04-23T10:37:17.485 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:37:17.478+0000 7f54ddffb640 1 -- 172.21.15.114:0/3014612208 <== mon.0 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f54d809c780 con 0x558c6fb35f40 2024-04-23T10:37:17.485 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:37:17.478+0000 7f54ddffb640 1 -- 172.21.15.114:0/3014612208 <== mon.0 v2:172.21.15.114:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f54d8087030 con 0x558c6fb35f40 2024-04-23T10:37:17.485 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:37:17.478+0000 7f54ddffb640 1 -- 172.21.15.114:0/3014612208 <== mon.0 v2:172.21.15.114:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f54d8005180 con 0x558c6fb35f40 2024-04-23T10:37:17.485 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:37:17.478+0000 7f54f07de740 1 -- 172.21.15.114:0/3014612208 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x558c6fb35f40 msgr2=0x558c6fb402c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T10:37:17.485 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:37:17.478+0000 7f54f07de740 1 --2- 172.21.15.114:0/3014612208 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x558c6fb35f40 0x558c6fb402c0 secure :-1 s=READY pgs=851 cs=0 l=1 rev1=1 crypto rx=0x7f54d809cbd0 tx=0x7f54d8004a90 comp rx=0 tx=0).stop 2024-04-23T10:37:17.485 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:37:17.478+0000 7f54f07de740 1 -- 172.21.15.114:0/3014612208 shutdown_connections 2024-04-23T10:37:17.485 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:37:17.478+0000 7f54f07de740 1 --2- 172.21.15.114:0/3014612208 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x558c6fb40800 0x558c6fb42bf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:37:17.486 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:37:17.478+0000 7f54f07de740 1 --2- 172.21.15.114:0/3014612208 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x558c6fb35f40 0x558c6fb402c0 unknown :-1 s=CLOSED pgs=851 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:37:17.486 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:37:17.478+0000 7f54f07de740 1 --2- 172.21.15.114:0/3014612208 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x558c6fb35620 0x558c6fb35a00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:37:17.486 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:37:17.478+0000 7f54f07de740 1 -- 172.21.15.114:0/3014612208 >> 172.21.15.114:0/3014612208 conn(0x558c6fb31b30 msgr2=0x558c6fb31f10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T10:37:17.486 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:37:17.479+0000 7f54f07de740 1 -- 172.21.15.114:0/3014612208 shutdown_connections 2024-04-23T10:37:17.486 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:37:17.479+0000 7f54f07de740 1 -- 172.21.15.114:0/3014612208 wait complete. 2024-04-23T10:37:17.486 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:37:17.479+0000 7f54f07de740 1 -- start start 2024-04-23T10:37:17.486 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:37:17.479+0000 7f54f07de740 1 Processor -- start 2024-04-23T10:37:17.486 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:37:17.479+0000 7f54f07de740 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x558c6fb35620 0x558c6fb40230 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T10:37:17.486 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:37:17.479+0000 7f54f07de740 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x558c6fb35f40 0x558c6fb3d520 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T10:37:17.486 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:37:17.479+0000 7f54f07de740 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x558c6fb40800 0x558c6fb3dc60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T10:37:17.486 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:37:17.479+0000 7f54f07de740 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x558c6fb433d0 con 0x558c6fb40800 2024-04-23T10:37:17.486 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:37:17.479+0000 7f54f07de740 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 -- 0x558c6fa350b0 con 0x558c6fb35f40 2024-04-23T10:37:17.486 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:37:17.479+0000 7f54f07de740 1 -- --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_getmap magic: 0 -- 0x558c6fb43550 con 0x558c6fb35620 2024-04-23T10:37:17.486 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:37:17.479+0000 7f54f07de740 10 client.0 _ll_register_callbacks cb 0x558c6fa22d30 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-23T10:37:17.487 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:37:17.480+0000 7f54e77fe640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x558c6fb40800 0x558c6fb3dc60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T10:37:17.487 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:37:17.480+0000 7f54e77fe640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x558c6fb40800 0x558c6fb3dc60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.114:48546/0 (socket says 172.21.15.114:48546) 2024-04-23T10:37:17.487 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:37:17.480+0000 7f54e77fe640 1 -- 172.21.15.114:0/3118370804 learned_addr learned my addr 172.21.15.114:0/3118370804 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-04-23T10:37:17.487 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:37:17.480+0000 7f54e6ffd640 1 --2- 172.21.15.114:0/3118370804 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x558c6fb35620 0x558c6fb40230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T10:37:17.487 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:37:17.480+0000 7f54dffff640 1 --2- 172.21.15.114:0/3118370804 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x558c6fb35f40 0x558c6fb3d520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T10:37:17.487 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:37:17.480+0000 7f54e77fe640 1 -- 172.21.15.114:0/3118370804 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x558c6fb35620 msgr2=0x558c6fb40230 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T10:37:17.487 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:37:17.480+0000 7f54e77fe640 1 --2- 172.21.15.114:0/3118370804 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x558c6fb35620 0x558c6fb40230 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:37:17.487 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:37:17.480+0000 7f54e77fe640 1 -- 172.21.15.114:0/3118370804 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x558c6fb35f40 msgr2=0x558c6fb3d520 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T10:37:17.487 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:37:17.480+0000 7f54e77fe640 1 --2- 172.21.15.114:0/3118370804 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x558c6fb35f40 0x558c6fb3d520 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:37:17.487 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:37:17.480+0000 7f54e77fe640 1 -- 172.21.15.114:0/3118370804 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({osdmap=0}) -- 0x558c6f879620 con 0x558c6fb40800 2024-04-23T10:37:17.487 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:37:17.480+0000 7f54e6ffd640 1 --2- 172.21.15.114:0/3118370804 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x558c6fb35620 0x558c6fb40230 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-23T10:37:17.487 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:37:17.480+0000 7f54dffff640 1 --2- 172.21.15.114:0/3118370804 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x558c6fb35f40 0x558c6fb3d520 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-23T10:37:17.487 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:37:17.480+0000 7f54e77fe640 1 --2- 172.21.15.114:0/3118370804 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x558c6fb40800 0x558c6fb3dc60 secure :-1 s=READY pgs=852 cs=0 l=1 rev1=1 crypto rx=0x7f54e0006010 tx=0x7f54e002fe20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ba51e7484509c0c4 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T10:37:17.487 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:37:17.480+0000 7f54c6ffd640 10 client.0 ms_handle_connect on v2:172.21.15.114:3300/0 2024-04-23T10:37:17.488 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:37:17.480+0000 7f54c6ffd640 1 -- 172.21.15.114:0/3118370804 <== mon.0 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f54e003e030 con 0x558c6fb40800 2024-04-23T10:37:17.488 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:37:17.480+0000 7f54f07de740 1 -- 172.21.15.114:0/3118370804 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f54d8087030 con 0x558c6fb40800 2024-04-23T10:37:17.488 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:37:17.480+0000 7f54c6ffd640 1 -- 172.21.15.114:0/3118370804 <== mon.0 v2:172.21.15.114:3300/0 2 ==== osd_map(104..104 src has 1..104) ==== 8629+0+0 (secure 0 0 0) 0x7f54e0003b70 con 0x558c6fb40800 2024-04-23T10:37:17.488 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:37:17.480+0000 7f54c6ffd640 1 -- 172.21.15.114:0/3118370804 <== mon.0 v2:172.21.15.114:3300/0 3 ==== mon_get_version_reply(handle=1 version=37) ==== 24+0+0 (secure 0 0 0) 0x7f54e0034040 con 0x558c6fb40800 2024-04-23T10:37:17.488 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:37:17.480+0000 7f54f07de740 10 client.16818 fetch_fsmap learned FSMap version 37 2024-04-23T10:37:17.488 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:37:17.480+0000 7f54f07de740 1 -- 172.21.15.114:0/3118370804 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({config=0+,fsmap=37,monmap=4+}) -- 0x558c6fb4f3b0 con 0x558c6fb40800 2024-04-23T10:37:17.488 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:37:17.481+0000 7f54c6ffd640 1 -- 172.21.15.114:0/3118370804 <== mon.0 v2:172.21.15.114:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f54d8087030 con 0x558c6fb40800 2024-04-23T10:37:17.488 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:37:17.481+0000 7f54c6ffd640 1 -- 172.21.15.114:0/3118370804 <== mon.0 v2:172.21.15.114:3300/0 5 ==== fsmap(e 37) ==== 8358+0+0 (secure 0 0 0) 0x7f54e0048020 con 0x558c6fb40800 2024-04-23T10:37:17.488 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:37:17.481+0000 7f54f07de740 10 client.16818 fetch_fsmap finished waiting for FSMap version 37 2024-04-23T10:37:17.488 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:37:17.481+0000 7f54f07de740 10 client.16818 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-04-23T10:37:17.488 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:37:17.481+0000 7f54f07de740 20 client.16818 populate_metadata read hostname 'smithi114' 2024-04-23T10:37:17.488 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:37:17.481+0000 7f54f07de740 1 --2- 172.21.15.114:0/3118370804 >> [v2:172.21.15.119:6838/937297340,v1:172.21.15.119:6839/937297340] conn(0x558c6fb50910 0x558c6fb52d00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T10:37:17.488 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:37:17.481+0000 7f54f07de740 4 client.16818 mds_command: new command op to 24545 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-23T10:37:17.488 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:37:17.481+0000 7f54f07de740 1 -- 172.21.15.114:0/3118370804 --> [v2:172.21.15.119:6838/937297340,v1:172.21.15.119:6839/937297340] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x558c6faa8390 con 0x558c6fb50910 2024-04-23T10:37:17.488 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:37:17.481+0000 7f54e6ffd640 1 --2- 172.21.15.114:0/3118370804 >> [v2:172.21.15.119:6838/937297340,v1:172.21.15.119:6839/937297340] conn(0x558c6fb50910 0x558c6fb52d00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T10:37:17.489 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:37:17.482+0000 7f54c6ffd640 10 client.16818 ms_handle_connect on v2:172.21.15.119:6838/937297340 2024-04-23T10:37:17.489 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:37:17.482+0000 7f54e6ffd640 1 --2- 172.21.15.114:0/3118370804 >> [v2:172.21.15.119:6838/937297340,v1:172.21.15.119:6839/937297340] conn(0x558c6fb50910 0x558c6fb52d00 crc :-1 s=READY pgs=156 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=8cbae4b351105249 server_cookie=46c108b33b416bea in_seq=0 out_seq=0 2024-04-23T10:37:17.489 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:37:17.482+0000 7f54c6ffd640 1 -- 172.21.15.114:0/3118370804 <== mds.0 v2:172.21.15.119:6838/937297340 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x558c6faa8390 con 0x558c6fb50910 2024-04-23T10:37:17.489 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:37:17.482+0000 7f54c6ffd640 10 client.16818 handle_command_reply: tid=0 2024-04-23T10:37:17.489 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:37:17.482+0000 7f54f07de740 1 client.16818 shutdown 2024-04-23T10:37:17.489 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:37:17.483+0000 7f54f07de740 1 -- 172.21.15.114:0/3118370804 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x558c6fb40800 msgr2=0x558c6fb3dc60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T10:37:17.489 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:37:17.483+0000 7f54f07de740 1 --2- 172.21.15.114:0/3118370804 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x558c6fb40800 0x558c6fb3dc60 secure :-1 s=READY pgs=852 cs=0 l=1 rev1=1 crypto rx=0x7f54e0006010 tx=0x7f54e002fe20 comp rx=0 tx=0).stop 2024-04-23T10:37:17.489 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:37:17.483+0000 7f54f07de740 1 -- 172.21.15.114:0/3118370804 shutdown_connections 2024-04-23T10:37:17.489 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:37:17.483+0000 7f54f07de740 1 --2- 172.21.15.114:0/3118370804 >> [v2:172.21.15.119:6838/937297340,v1:172.21.15.119:6839/937297340] conn(0x558c6fb50910 0x558c6fb52d00 crc :-1 s=READY pgs=156 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:37:17.489 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:37:17.483+0000 7f54f07de740 1 --2- 172.21.15.114:0/3118370804 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x558c6fb40800 0x558c6fb3dc60 unknown :-1 s=CLOSED pgs=852 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:37:17.489 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:37:17.483+0000 7f54c6ffd640 0 client.16818 ms_handle_reset on v2:172.21.15.119:6838/937297340 2024-04-23T10:37:17.489 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:37:17.483+0000 7f54f07de740 1 --2- 172.21.15.114:0/3118370804 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x558c6fb35f40 0x558c6fb3d520 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:37:17.489 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:37:17.483+0000 7f54f07de740 1 --2- 172.21.15.114:0/3118370804 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x558c6fb35620 0x558c6fb40230 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:37:17.489 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:37:17.483+0000 7f54f07de740 1 -- 172.21.15.114:0/3118370804 >> 172.21.15.114:0/3118370804 conn(0x558c6fb31b30 msgr2=0x558c6fb378b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T10:37:17.490 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:37:17.483+0000 7f54f07de740 1 -- 172.21.15.114:0/3118370804 shutdown_connections 2024-04-23T10:37:17.490 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:37:17.483+0000 7f54f07de740 1 -- 172.21.15.114:0/3118370804 wait complete. 2024-04-23T10:37:17.490 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:37:17.483+0000 7f54f07de740 20 client.16818 trim_cache size 0 max 16384 2024-04-23T10:37:17.501 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:37:17.497+0000 7f54f07de740 1 Processor -- start 2024-04-23T10:37:17.501 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:37:17.497+0000 7f54f07de740 1 -- start start 2024-04-23T10:37:17.501 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:37:17.498+0000 7f54f07de740 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x558c6fb35620 0x558c6fb4c290 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T10:37:17.501 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:37:17.498+0000 7f54f07de740 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x558c6fb35f40 0x558c6fb31430 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T10:37:17.502 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:37:17.498+0000 7f54f07de740 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x558c6fb40800 0x558c6fb3fd60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T10:37:17.502 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:37:17.498+0000 7f54f07de740 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x7f54e81b0670 con 0x558c6fb35620 2024-04-23T10:37:17.502 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:37:17.498+0000 7f54f07de740 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 -- 0x7f54e81ade80 con 0x558c6fb35f40 2024-04-23T10:37:17.502 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:37:17.498+0000 7f54f07de740 1 -- --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_getmap magic: 0 -- 0x7f54e81abcf0 con 0x558c6fb40800 2024-04-23T10:37:17.502 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:37:17.498+0000 7f54e6ffd640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x558c6fb35620 0x558c6fb4c290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T10:37:17.502 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:37:17.498+0000 7f54e6ffd640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x558c6fb35620 0x558c6fb4c290 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.114:48552/0 (socket says 172.21.15.114:48552) 2024-04-23T10:37:17.502 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:37:17.498+0000 7f54e6ffd640 1 -- 172.21.15.114:0/64454386 learned_addr learned my addr 172.21.15.114:0/64454386 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-04-23T10:37:17.502 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:37:17.498+0000 7f54dffff640 1 --2- 172.21.15.114:0/64454386 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x558c6fb35f40 0x558c6fb31430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T10:37:17.502 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:37:17.498+0000 7f54e6ffd640 1 -- 172.21.15.114:0/64454386 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x558c6fb40800 msgr2=0x558c6fb3fd60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T10:37:17.502 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:37:17.498+0000 7f54e6ffd640 1 --2- 172.21.15.114:0/64454386 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x558c6fb40800 0x558c6fb3fd60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:37:17.502 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:37:17.498+0000 7f54e6ffd640 1 -- 172.21.15.114:0/64454386 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x558c6fb35f40 msgr2=0x558c6fb31430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T10:37:17.502 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:37:17.498+0000 7f54e6ffd640 1 --2- 172.21.15.114:0/64454386 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x558c6fb35f40 0x558c6fb31430 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:37:17.503 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:37:17.498+0000 7f54e6ffd640 1 -- 172.21.15.114:0/64454386 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x558c6fb53070 con 0x558c6fb35620 2024-04-23T10:37:17.503 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:37:17.498+0000 7f54e6ffd640 1 --2- 172.21.15.114:0/64454386 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x558c6fb35620 0x558c6fb4c290 secure :-1 s=READY pgs=853 cs=0 l=1 rev1=1 crypto rx=0x7f54d80990a0 tx=0x7f54d80bd040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e4b5bf901d93cc5c server_cookie=0 in_seq=0 out_seq=0 2024-04-23T10:37:17.503 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:37:17.498+0000 7f54dd7fa640 1 -- 172.21.15.114:0/64454386 <== mon.0 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f54d809c5a0 con 0x558c6fb35620 2024-04-23T10:37:17.503 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:37:17.498+0000 7f54dd7fa640 1 -- 172.21.15.114:0/64454386 <== mon.0 v2:172.21.15.114:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f54d80b5070 con 0x558c6fb35620 2024-04-23T10:37:17.503 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:37:17.498+0000 7f54dd7fa640 1 -- 172.21.15.114:0/64454386 <== mon.0 v2:172.21.15.114:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f54d809cc40 con 0x558c6fb35620 2024-04-23T10:37:17.503 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:37:17.499+0000 7f54f07de740 1 -- 172.21.15.114:0/64454386 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x558c6fb35620 msgr2=0x558c6fb4c290 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T10:37:17.503 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:37:17.499+0000 7f54f07de740 1 --2- 172.21.15.114:0/64454386 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x558c6fb35620 0x558c6fb4c290 secure :-1 s=READY pgs=853 cs=0 l=1 rev1=1 crypto rx=0x7f54d80990a0 tx=0x7f54d80bd040 comp rx=0 tx=0).stop 2024-04-23T10:37:17.503 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:37:17.499+0000 7f54f07de740 1 -- 172.21.15.114:0/64454386 shutdown_connections 2024-04-23T10:37:17.503 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:37:17.499+0000 7f54f07de740 1 --2- 172.21.15.114:0/64454386 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x558c6fb40800 0x558c6fb3fd60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:37:17.503 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:37:17.499+0000 7f54f07de740 1 --2- 172.21.15.114:0/64454386 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x558c6fb35f40 0x558c6fb31430 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:37:17.503 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:37:17.499+0000 7f54f07de740 1 --2- 172.21.15.114:0/64454386 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x558c6fb35620 0x558c6fb4c290 unknown :-1 s=CLOSED pgs=853 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:37:17.503 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:37:17.499+0000 7f54f07de740 1 -- 172.21.15.114:0/64454386 >> 172.21.15.114:0/64454386 conn(0x558c6fb31b30 msgr2=0x558c6fb38140 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T10:37:17.503 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:37:17.499+0000 7f54f07de740 1 -- 172.21.15.114:0/64454386 shutdown_connections 2024-04-23T10:37:17.503 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:37:17.499+0000 7f54f07de740 1 -- 172.21.15.114:0/64454386 wait complete. 2024-04-23T10:37:17.504 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:37:17.499+0000 7f54f07de740 1 -- start start 2024-04-23T10:37:17.504 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:37:17.499+0000 7f54f07de740 1 Processor -- start 2024-04-23T10:37:17.504 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:37:17.500+0000 7f54f07de740 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x558c6fb35620 0x558c6fb399a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T10:37:17.504 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:37:17.500+0000 7f54f07de740 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x558c6fb35f40 0x558c6fb39ee0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T10:37:17.504 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:37:17.500+0000 7f54f07de740 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x558c6fb40800 0x558c6fb3a620 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T10:37:17.504 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:37:17.500+0000 7f54f07de740 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x7f54e81ac340 con 0x558c6fb35f40 2024-04-23T10:37:17.504 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:37:17.500+0000 7f54f07de740 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 -- 0x7f54e81ad7b0 con 0x558c6fb35620 2024-04-23T10:37:17.504 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:37:17.500+0000 7f54f07de740 1 -- --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_getmap magic: 0 -- 0x7f54e81aca10 con 0x558c6fb40800 2024-04-23T10:37:17.504 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:37:17.500+0000 7f54f07de740 10 client.0 _ll_register_callbacks cb 0x558c6fa22d30 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-23T10:37:17.504 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:37:17.500+0000 7f54dffff640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x558c6fb35f40 0x558c6fb39ee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T10:37:17.504 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:37:17.500+0000 7f54dffff640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x558c6fb35f40 0x558c6fb39ee0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.114:48558/0 (socket says 172.21.15.114:48558) 2024-04-23T10:37:17.504 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:37:17.500+0000 7f54dffff640 1 -- 172.21.15.114:0/692319595 learned_addr learned my addr 172.21.15.114:0/692319595 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-04-23T10:37:17.504 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:37:17.500+0000 7f54e6ffd640 1 --2- 172.21.15.114:0/692319595 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x558c6fb35620 0x558c6fb399a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T10:37:17.504 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:37:17.500+0000 7f54e77fe640 1 --2- 172.21.15.114:0/692319595 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x558c6fb40800 0x558c6fb3a620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T10:37:17.505 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:37:17.500+0000 7f54dffff640 1 -- 172.21.15.114:0/692319595 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x558c6fb40800 msgr2=0x558c6fb3a620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T10:37:17.505 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:37:17.500+0000 7f54dffff640 1 --2- 172.21.15.114:0/692319595 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x558c6fb40800 0x558c6fb3a620 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:37:17.505 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:37:17.500+0000 7f54dffff640 1 -- 172.21.15.114:0/692319595 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x558c6fb35620 msgr2=0x558c6fb399a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T10:37:17.505 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:37:17.500+0000 7f54dffff640 1 --2- 172.21.15.114:0/692319595 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x558c6fb35620 0x558c6fb399a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:37:17.505 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:37:17.500+0000 7f54dffff640 1 -- 172.21.15.114:0/692319595 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({osdmap=0}) -- 0x558c6fb3ae20 con 0x558c6fb35f40 2024-04-23T10:37:17.505 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:37:17.500+0000 7f54e77fe640 1 --2- 172.21.15.114:0/692319595 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x558c6fb40800 0x558c6fb3a620 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-23T10:37:17.505 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:37:17.500+0000 7f54e6ffd640 1 --2- 172.21.15.114:0/692319595 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x558c6fb35620 0x558c6fb399a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-23T10:37:17.505 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:37:17.500+0000 7f54dffff640 1 --2- 172.21.15.114:0/692319595 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x558c6fb35f40 0x558c6fb39ee0 secure :-1 s=READY pgs=854 cs=0 l=1 rev1=1 crypto rx=0x7f54d0009fd0 tx=0x7f54d0004730 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2edeead11e9986ca server_cookie=0 in_seq=0 out_seq=0 2024-04-23T10:37:17.505 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:37:17.500+0000 7f54c6ffd640 10 client.0 ms_handle_connect on v2:172.21.15.114:3300/0 2024-04-23T10:37:17.505 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:37:17.500+0000 7f54c6ffd640 1 -- 172.21.15.114:0/692319595 <== mon.0 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f54d00129a0 con 0x558c6fb35f40 2024-04-23T10:37:17.505 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:37:17.500+0000 7f54c6ffd640 1 -- 172.21.15.114:0/692319595 <== mon.0 v2:172.21.15.114:3300/0 2 ==== osd_map(104..104 src has 1..104) ==== 8629+0+0 (secure 0 0 0) 0x7f54d0012b40 con 0x558c6fb35f40 2024-04-23T10:37:17.505 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:37:17.500+0000 7f54f07de740 1 -- 172.21.15.114:0/692319595 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f54d80b5070 con 0x558c6fb35f40 2024-04-23T10:37:17.505 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:37:17.501+0000 7f54c6ffd640 1 -- 172.21.15.114:0/692319595 <== mon.0 v2:172.21.15.114:3300/0 3 ==== mon_get_version_reply(handle=1 version=37) ==== 24+0+0 (secure 0 0 0) 0x7f54d0012d40 con 0x558c6fb35f40 2024-04-23T10:37:17.505 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:37:17.501+0000 7f54f07de740 10 client.16824 fetch_fsmap learned FSMap version 37 2024-04-23T10:37:17.505 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:37:17.501+0000 7f54f07de740 1 -- 172.21.15.114:0/692319595 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({config=0+,fsmap=37,monmap=4+}) -- 0x558c6fbbb7a0 con 0x558c6fb35f40 2024-04-23T10:37:17.506 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:37:17.501+0000 7f54c6ffd640 1 -- 172.21.15.114:0/692319595 <== mon.0 v2:172.21.15.114:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f54d80b5070 con 0x558c6fb35f40 2024-04-23T10:37:17.506 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:37:17.501+0000 7f54c6ffd640 1 -- 172.21.15.114:0/692319595 <== mon.0 v2:172.21.15.114:3300/0 5 ==== fsmap(e 37) ==== 8358+0+0 (secure 0 0 0) 0x7f54d0022440 con 0x558c6fb35f40 2024-04-23T10:37:17.506 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:37:17.501+0000 7f54f07de740 10 client.16824 fetch_fsmap finished waiting for FSMap version 37 2024-04-23T10:37:17.506 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:37:17.501+0000 7f54f07de740 10 client.16824 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-04-23T10:37:17.506 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:37:17.501+0000 7f54f07de740 20 client.16824 populate_metadata read hostname 'smithi114' 2024-04-23T10:37:17.506 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:37:17.501+0000 7f54f07de740 1 --2- 172.21.15.114:0/692319595 >> [v2:172.21.15.119:6838/937297340,v1:172.21.15.119:6839/937297340] conn(0x558c6fbbc030 0x558c6fbbe450 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T10:37:17.506 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:37:17.501+0000 7f54f07de740 4 client.16824 mds_command: new command op to 24545 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-04-23T10:37:17.506 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:37:17.501+0000 7f54f07de740 1 -- 172.21.15.114:0/692319595 --> [v2:172.21.15.119:6838/937297340,v1:172.21.15.119:6839/937297340] -- command(tid 0: {"prefix": "flush journal"}) -- 0x558c6faa2200 con 0x558c6fbbc030 2024-04-23T10:37:17.506 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:37:17.501+0000 7f54e6ffd640 1 --2- 172.21.15.114:0/692319595 >> [v2:172.21.15.119:6838/937297340,v1:172.21.15.119:6839/937297340] conn(0x558c6fbbc030 0x558c6fbbe450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T10:37:17.506 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:37:17.502+0000 7f54c6ffd640 10 client.16824 ms_handle_connect on v2:172.21.15.119:6838/937297340 2024-04-23T10:37:17.506 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:37:17.502+0000 7f54e6ffd640 1 --2- 172.21.15.114:0/692319595 >> [v2:172.21.15.119:6838/937297340,v1:172.21.15.119:6839/937297340] conn(0x558c6fbbc030 0x558c6fbbe450 crc :-1 s=READY pgs=157 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=fcb7fd1d33b91864 server_cookie=cf96d6d9318e8b26 in_seq=0 out_seq=0 2024-04-23T10:37:17.541 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:37:17.542+0000 7f54c6ffd640 1 -- 172.21.15.114:0/692319595 <== mds.0 v2:172.21.15.119:6838/937297340 1 ==== command_reply(tid 0: 0 ) ==== 8+0+44 (crc 0 0 0) 0x558c6faa2200 con 0x558c6fbbc030 2024-04-23T10:37:17.541 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:37:17.542+0000 7f54c6ffd640 10 client.16824 handle_command_reply: tid=0 2024-04-23T10:37:17.541 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:37:17.542+0000 7f54f07de740 1 client.16824 shutdown 2024-04-23T10:37:17.541 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:37:17.542+0000 7f54f07de740 1 -- 172.21.15.114:0/692319595 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x558c6fb35f40 msgr2=0x558c6fb39ee0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T10:37:17.541 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:37:17.542+0000 7f54f07de740 1 --2- 172.21.15.114:0/692319595 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x558c6fb35f40 0x558c6fb39ee0 secure :-1 s=READY pgs=854 cs=0 l=1 rev1=1 crypto rx=0x7f54d0009fd0 tx=0x7f54d0004730 comp rx=0 tx=0).stop 2024-04-23T10:37:17.542 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:37:17.542+0000 7f54f07de740 1 -- 172.21.15.114:0/692319595 shutdown_connections 2024-04-23T10:37:17.542 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:37:17.542+0000 7f54f07de740 1 --2- 172.21.15.114:0/692319595 >> [v2:172.21.15.119:6838/937297340,v1:172.21.15.119:6839/937297340] conn(0x558c6fbbc030 0x558c6fbbe450 crc :-1 s=READY pgs=157 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:37:17.542 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:37:17.542+0000 7f54f07de740 1 --2- 172.21.15.114:0/692319595 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x558c6fb40800 0x558c6fb3a620 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:37:17.542 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:37:17.542+0000 7f54f07de740 1 --2- 172.21.15.114:0/692319595 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x558c6fb35f40 0x558c6fb39ee0 unknown :-1 s=CLOSED pgs=854 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:37:17.542 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:37:17.542+0000 7f54c6ffd640 0 client.16824 ms_handle_reset on v2:172.21.15.119:6838/937297340 2024-04-23T10:37:17.542 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:37:17.542+0000 7f54f07de740 1 --2- 172.21.15.114:0/692319595 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x558c6fb35620 0x558c6fb399a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:37:17.542 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:37:17.542+0000 7f54f07de740 1 -- 172.21.15.114:0/692319595 >> 172.21.15.114:0/692319595 conn(0x558c6fb31b30 msgr2=0x558c6fb30420 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T10:37:17.542 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:37:17.542+0000 7f54f07de740 1 -- 172.21.15.114:0/692319595 shutdown_connections 2024-04-23T10:37:17.542 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:37:17.542+0000 7f54f07de740 1 -- 172.21.15.114:0/692319595 wait complete. 2024-04-23T10:37:17.542 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:37:17.542+0000 7f54f07de740 20 client.16824 trim_cache size 0 max 16384 2024-04-23T10:37:17.542 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stdout:{ 2024-04-23T10:37:17.543 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stdout: "message": "", 2024-04-23T10:37:17.543 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stdout: "return_code": 0 2024-04-23T10:37:17.543 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stdout:} 2024-04-23T10:37:17.543 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:37:17.543+0000 7f54ed9f0640 1 -- 172.21.15.114:0/2042188437 >> [v2:172.21.15.114:6800/1395693466,v1:172.21.15.114:6801/1395693466] conn(0x7f54d40a3dc0 msgr2=0x7f54d40a6280 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T10:37:17.544 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:37:17.543+0000 7f54ed9f0640 1 --2- 172.21.15.114:0/2042188437 >> [v2:172.21.15.114:6800/1395693466,v1:172.21.15.114:6801/1395693466] conn(0x7f54d40a3dc0 0x7f54d40a6280 secure :-1 s=READY pgs=132 cs=0 l=1 rev1=1 crypto rx=0x7f54e810e210 tx=0x7f54d0005f90 comp rx=0 tx=0).stop 2024-04-23T10:37:17.544 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:37:17.543+0000 7f54ed9f0640 1 -- 172.21.15.114:0/2042188437 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f54e8103180 msgr2=0x7f54e8198660 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T10:37:17.544 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:37:17.543+0000 7f54ed9f0640 1 --2- 172.21.15.114:0/2042188437 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f54e8103180 0x7f54e8198660 secure :-1 s=READY pgs=850 cs=0 l=1 rev1=1 crypto rx=0x7f54d800e9f0 tx=0x7f54d800edf0 comp rx=0 tx=0).stop 2024-04-23T10:37:17.544 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:37:17.543+0000 7f54ed9f0640 1 -- 172.21.15.114:0/2042188437 shutdown_connections 2024-04-23T10:37:17.544 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:37:17.543+0000 7f54ed9f0640 1 --2- 172.21.15.114:0/2042188437 >> [v2:172.21.15.114:6800/1395693466,v1:172.21.15.114:6801/1395693466] conn(0x7f54d40a3dc0 0x7f54d40a6280 unknown :-1 s=CLOSED pgs=132 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:37:17.544 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:37:17.543+0000 7f54ed9f0640 1 --2- 172.21.15.114:0/2042188437 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f54e810c550 0x7f54e819cf20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:37:17.544 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:37:17.543+0000 7f54ed9f0640 1 --2- 172.21.15.114:0/2042188437 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f54e8103aa0 0x7f54e8198ba0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:37:17.544 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:37:17.543+0000 7f54ed9f0640 1 --2- 172.21.15.114:0/2042188437 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f54e8103180 0x7f54e8198660 unknown :-1 s=CLOSED pgs=850 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:37:17.544 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:37:17.543+0000 7f54ed9f0640 1 -- 172.21.15.114:0/2042188437 >> 172.21.15.114:0/2042188437 conn(0x7f54e8077fd0 msgr2=0x7f54e80fde80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T10:37:17.544 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:37:17.543+0000 7f54ed9f0640 1 -- 172.21.15.114:0/2042188437 shutdown_connections 2024-04-23T10:37:17.544 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:37:17.543+0000 7f54ed9f0640 1 -- 172.21.15.114:0/2042188437 wait complete. 2024-04-23T10:37:17.941 INFO:tasks.workunit.client.0.smithi114.stdout: 1 667690 5.96 MB/sec execute 446 sec latency 1991.475 ms 2024-04-23T10:37:18.492 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 23 10:37:18 smithi119 ceph-mon[32608]: pgmap v335: 97 pgs: 97 active+clean; 207 MiB data, 2.6 GiB used, 1.0 TiB / 1.0 TiB avail; 569 KiB/s rd, 1.0 MiB/s wr, 88 op/s 2024-04-23T10:37:18.503 INFO:journalctl@ceph.mon.a.smithi114.stdout:Apr 23 10:37:18 smithi114 ceph-mon[29255]: pgmap v335: 97 pgs: 97 active+clean; 207 MiB data, 2.6 GiB used, 1.0 TiB / 1.0 TiB avail; 569 KiB/s rd, 1.0 MiB/s wr, 88 op/s 2024-04-23T10:37:18.578 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 23 10:37:18 smithi204 ceph-mon[32649]: pgmap v335: 97 pgs: 97 active+clean; 207 MiB data, 2.6 GiB used, 1.0 TiB / 1.0 TiB avail; 569 KiB/s rd, 1.0 MiB/s wr, 88 op/s 2024-04-23T10:37:18.941 INFO:tasks.workunit.client.0.smithi114.stdout: 1 667690 5.95 MB/sec execute 447 sec latency 2991.570 ms 2024-04-23T10:37:19.941 INFO:tasks.workunit.client.0.smithi114.stdout: 1 667757 5.93 MB/sec execute 448 sec latency 3243.558 ms 2024-04-23T10:37:20.492 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 23 10:37:20 smithi119 ceph-mon[32608]: pgmap v336: 97 pgs: 97 active+clean; 207 MiB data, 2.6 GiB used, 1.0 TiB / 1.0 TiB avail; 493 KiB/s rd, 929 KiB/s wr, 76 op/s 2024-04-23T10:37:20.504 INFO:journalctl@ceph.mon.a.smithi114.stdout:Apr 23 10:37:20 smithi114 ceph-mon[29255]: pgmap v336: 97 pgs: 97 active+clean; 207 MiB data, 2.6 GiB used, 1.0 TiB / 1.0 TiB avail; 493 KiB/s rd, 929 KiB/s wr, 76 op/s 2024-04-23T10:37:20.577 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 23 10:37:20 smithi204 ceph-mon[32649]: pgmap v336: 97 pgs: 97 active+clean; 207 MiB data, 2.6 GiB used, 1.0 TiB / 1.0 TiB avail; 493 KiB/s rd, 929 KiB/s wr, 76 op/s 2024-04-23T10:37:20.941 INFO:tasks.workunit.client.0.smithi114.stdout: 1 667757 5.92 MB/sec execute 449 sec latency 1656.484 ms 2024-04-23T10:37:21.941 INFO:tasks.workunit.client.0.smithi114.stdout: 1 667757 5.91 MB/sec execute 450 sec latency 2656.538 ms 2024-04-23T10:37:22.492 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 23 10:37:22 smithi119 ceph-mon[32608]: pgmap v337: 97 pgs: 97 active+clean; 207 MiB data, 2.6 GiB used, 1.0 TiB / 1.0 TiB avail; 456 KiB/s rd, 859 KiB/s wr, 70 op/s 2024-04-23T10:37:22.503 INFO:journalctl@ceph.mon.a.smithi114.stdout:Apr 23 10:37:22 smithi114 ceph-mon[29255]: pgmap v337: 97 pgs: 97 active+clean; 207 MiB data, 2.6 GiB used, 1.0 TiB / 1.0 TiB avail; 456 KiB/s rd, 859 KiB/s wr, 70 op/s 2024-04-23T10:37:22.577 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 23 10:37:22 smithi204 ceph-mon[32649]: pgmap v337: 97 pgs: 97 active+clean; 207 MiB data, 2.6 GiB used, 1.0 TiB / 1.0 TiB avail; 456 KiB/s rd, 859 KiB/s wr, 70 op/s 2024-04-23T10:37:22.942 INFO:tasks.workunit.client.0.smithi114.stdout: 1 667757 5.89 MB/sec execute 451 sec latency 3656.569 ms 2024-04-23T10:37:23.942 INFO:tasks.workunit.client.0.smithi114.stdout: 1 667757 5.88 MB/sec execute 452 sec latency 4656.651 ms 2024-04-23T10:37:24.491 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 23 10:37:24 smithi119 ceph-mon[32608]: pgmap v338: 97 pgs: 97 active+clean; 205 MiB data, 2.6 GiB used, 1.0 TiB / 1.0 TiB avail; 421 KiB/s rd, 1.1 MiB/s wr, 91 op/s 2024-04-23T10:37:24.504 INFO:journalctl@ceph.mon.a.smithi114.stdout:Apr 23 10:37:24 smithi114 ceph-mon[29255]: pgmap v338: 97 pgs: 97 active+clean; 205 MiB data, 2.6 GiB used, 1.0 TiB / 1.0 TiB avail; 421 KiB/s rd, 1.1 MiB/s wr, 91 op/s 2024-04-23T10:37:24.578 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 23 10:37:24 smithi204 ceph-mon[32649]: pgmap v338: 97 pgs: 97 active+clean; 205 MiB data, 2.6 GiB used, 1.0 TiB / 1.0 TiB avail; 421 KiB/s rd, 1.1 MiB/s wr, 91 op/s 2024-04-23T10:37:24.941 INFO:tasks.workunit.client.0.smithi114.stdout: 1 668546 5.87 MB/sec execute 453 sec latency 4895.544 ms 2024-04-23T10:37:25.941 INFO:tasks.workunit.client.0.smithi114.stdout: 1 671147 5.90 MB/sec execute 454 sec latency 23.138 ms 2024-04-23T10:37:26.491 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 23 10:37:26 smithi119 ceph-mon[32608]: pgmap v339: 97 pgs: 97 active+clean; 205 MiB data, 2.6 GiB used, 1.0 TiB / 1.0 TiB avail; 384 KiB/s rd, 998 KiB/s wr, 83 op/s 2024-04-23T10:37:26.503 INFO:journalctl@ceph.mon.a.smithi114.stdout:Apr 23 10:37:26 smithi114 ceph-mon[29255]: pgmap v339: 97 pgs: 97 active+clean; 205 MiB data, 2.6 GiB used, 1.0 TiB / 1.0 TiB avail; 384 KiB/s rd, 998 KiB/s wr, 83 op/s 2024-04-23T10:37:26.577 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 23 10:37:26 smithi204 ceph-mon[32649]: pgmap v339: 97 pgs: 97 active+clean; 205 MiB data, 2.6 GiB used, 1.0 TiB / 1.0 TiB avail; 384 KiB/s rd, 998 KiB/s wr, 83 op/s 2024-04-23T10:37:26.942 INFO:tasks.workunit.client.0.smithi114.stdout: 1 671236 5.88 MB/sec execute 455 sec latency 952.450 ms 2024-04-23T10:37:27.942 INFO:tasks.workunit.client.0.smithi114.stdout: 1 671236 5.87 MB/sec execute 456 sec latency 1952.531 ms 2024-04-23T10:37:28.492 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 23 10:37:28 smithi119 ceph-mon[32608]: pgmap v340: 97 pgs: 97 active+clean; 209 MiB data, 2.6 GiB used, 1.0 TiB / 1.0 TiB avail; 596 KiB/s rd, 1.2 MiB/s wr, 117 op/s 2024-04-23T10:37:28.504 INFO:journalctl@ceph.mon.a.smithi114.stdout:Apr 23 10:37:28 smithi114 ceph-mon[29255]: pgmap v340: 97 pgs: 97 active+clean; 209 MiB data, 2.6 GiB used, 1.0 TiB / 1.0 TiB avail; 596 KiB/s rd, 1.2 MiB/s wr, 117 op/s 2024-04-23T10:37:28.577 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 23 10:37:28 smithi204 ceph-mon[32649]: pgmap v340: 97 pgs: 97 active+clean; 209 MiB data, 2.6 GiB used, 1.0 TiB / 1.0 TiB avail; 596 KiB/s rd, 1.2 MiB/s wr, 117 op/s 2024-04-23T10:37:28.942 INFO:tasks.workunit.client.0.smithi114.stdout: 1 671236 5.86 MB/sec execute 457 sec latency 2952.566 ms 2024-04-23T10:37:29.942 INFO:tasks.workunit.client.0.smithi114.stdout: 1 671303 5.85 MB/sec execute 458 sec latency 3196.525 ms 2024-04-23T10:37:30.491 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 23 10:37:30 smithi119 ceph-mon[32608]: pgmap v341: 97 pgs: 97 active+clean; 209 MiB data, 2.6 GiB used, 1.0 TiB / 1.0 TiB avail; 341 KiB/s rd, 927 KiB/s wr, 78 op/s 2024-04-23T10:37:30.504 INFO:journalctl@ceph.mon.a.smithi114.stdout:Apr 23 10:37:30 smithi114 ceph-mon[29255]: pgmap v341: 97 pgs: 97 active+clean; 209 MiB data, 2.6 GiB used, 1.0 TiB / 1.0 TiB avail; 341 KiB/s rd, 927 KiB/s wr, 78 op/s 2024-04-23T10:37:30.577 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 23 10:37:30 smithi204 ceph-mon[32649]: pgmap v341: 97 pgs: 97 active+clean; 209 MiB data, 2.6 GiB used, 1.0 TiB / 1.0 TiB avail; 341 KiB/s rd, 927 KiB/s wr, 78 op/s 2024-04-23T10:37:30.639 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:37:30.640+0000 7f9a83ba1640 1 -- 172.21.15.114:0/1065890005 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f9a7c1010f0 msgr2=0x7f9a7c1014d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T10:37:30.639 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:37:30.640+0000 7f9a83ba1640 1 --2- 172.21.15.114:0/1065890005 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f9a7c1010f0 0x7f9a7c1014d0 secure :-1 s=READY pgs=855 cs=0 l=1 rev1=1 crypto rx=0x7f9a78009900 tx=0x7f9a7802a410 comp rx=0 tx=0).stop 2024-04-23T10:37:30.639 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:37:30.640+0000 7f9a83ba1640 1 -- 172.21.15.114:0/1065890005 shutdown_connections 2024-04-23T10:37:30.639 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:37:30.640+0000 7f9a83ba1640 1 --2- 172.21.15.114:0/1065890005 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f9a7c106170 0x7f9a7c10dd40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:37:30.639 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:37:30.640+0000 7f9a83ba1640 1 --2- 172.21.15.114:0/1065890005 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f9a7c101aa0 0x7f9a7c105a90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:37:30.639 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:37:30.640+0000 7f9a83ba1640 1 --2- 172.21.15.114:0/1065890005 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f9a7c1010f0 0x7f9a7c1014d0 unknown :-1 s=CLOSED pgs=855 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:37:30.640 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:37:30.640+0000 7f9a83ba1640 1 -- 172.21.15.114:0/1065890005 >> 172.21.15.114:0/1065890005 conn(0x7f9a7c077fd0 msgr2=0x7f9a7c0783e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T10:37:30.640 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:37:30.640+0000 7f9a83ba1640 1 -- 172.21.15.114:0/1065890005 shutdown_connections 2024-04-23T10:37:30.640 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:37:30.640+0000 7f9a83ba1640 1 -- 172.21.15.114:0/1065890005 wait complete. 2024-04-23T10:37:30.640 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:37:30.641+0000 7f9a83ba1640 1 Processor -- start 2024-04-23T10:37:30.640 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:37:30.641+0000 7f9a83ba1640 1 -- start start 2024-04-23T10:37:30.642 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:37:30.641+0000 7f9a83ba1640 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f9a7c1010f0 0x7f9a7c19e400 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T10:37:30.642 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:37:30.641+0000 7f9a83ba1640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f9a7c106170 0x7f9a7c19e940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T10:37:30.642 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:37:30.641+0000 7f9a83ba1640 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f9a7c1985b0 0x7f9a7c198a60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T10:37:30.642 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:37:30.641+0000 7f9a83ba1640 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x7f9a7c1121f0 con 0x7f9a7c106170 2024-04-23T10:37:30.642 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:37:30.641+0000 7f9a83ba1640 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 -- 0x7f9a7c112070 con 0x7f9a7c1010f0 2024-04-23T10:37:30.642 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:37:30.641+0000 7f9a83ba1640 1 -- --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_getmap magic: 0 -- 0x7f9a7c112370 con 0x7f9a7c1985b0 2024-04-23T10:37:30.642 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:37:30.641+0000 7f9a8113c640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f9a7c106170 0x7f9a7c19e940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T10:37:30.642 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:37:30.641+0000 7f9a8113c640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f9a7c106170 0x7f9a7c19e940 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.114:51490/0 (socket says 172.21.15.114:51490) 2024-04-23T10:37:30.642 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:37:30.641+0000 7f9a8113c640 1 -- 172.21.15.114:0/2450880004 learned_addr learned my addr 172.21.15.114:0/2450880004 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-04-23T10:37:30.642 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:37:30.641+0000 7f9a8213e640 1 --2- 172.21.15.114:0/2450880004 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f9a7c1985b0 0x7f9a7c198a60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T10:37:30.642 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:37:30.641+0000 7f9a8193d640 1 --2- 172.21.15.114:0/2450880004 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f9a7c1010f0 0x7f9a7c19e400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T10:37:30.642 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:37:30.641+0000 7f9a8113c640 1 -- 172.21.15.114:0/2450880004 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f9a7c1985b0 msgr2=0x7f9a7c198a60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T10:37:30.643 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:37:30.641+0000 7f9a8113c640 1 --2- 172.21.15.114:0/2450880004 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f9a7c1985b0 0x7f9a7c198a60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:37:30.643 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:37:30.641+0000 7f9a8113c640 1 -- 172.21.15.114:0/2450880004 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f9a7c1010f0 msgr2=0x7f9a7c19e400 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T10:37:30.643 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:37:30.641+0000 7f9a8113c640 1 --2- 172.21.15.114:0/2450880004 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f9a7c1010f0 0x7f9a7c19e400 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:37:30.643 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:37:30.641+0000 7f9a8113c640 1 -- 172.21.15.114:0/2450880004 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f9a7c1990d0 con 0x7f9a7c106170 2024-04-23T10:37:30.643 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:37:30.641+0000 7f9a8193d640 1 --2- 172.21.15.114:0/2450880004 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f9a7c1010f0 0x7f9a7c19e400 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-23T10:37:30.643 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:37:30.641+0000 7f9a8113c640 1 --2- 172.21.15.114:0/2450880004 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f9a7c106170 0x7f9a7c19e940 secure :-1 s=READY pgs=856 cs=0 l=1 rev1=1 crypto rx=0x7f9a6c00d740 tx=0x7f9a6c00dc00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T10:37:30.643 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:37:30.642+0000 7f9a72ffd640 1 -- 172.21.15.114:0/2450880004 <== mon.0 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9a6c017070 con 0x7f9a7c106170 2024-04-23T10:37:30.643 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:37:30.642+0000 7f9a83ba1640 1 -- 172.21.15.114:0/2450880004 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f9a7c1993c0 con 0x7f9a7c106170 2024-04-23T10:37:30.643 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:37:30.642+0000 7f9a72ffd640 1 -- 172.21.15.114:0/2450880004 <== mon.0 v2:172.21.15.114:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f9a6c00de50 con 0x7f9a7c106170 2024-04-23T10:37:30.643 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:37:30.642+0000 7f9a72ffd640 1 -- 172.21.15.114:0/2450880004 <== mon.0 v2:172.21.15.114:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9a6c004ce0 con 0x7f9a7c106170 2024-04-23T10:37:30.643 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:37:30.642+0000 7f9a83ba1640 1 -- 172.21.15.114:0/2450880004 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f9a7c1a5610 con 0x7f9a7c106170 2024-04-23T10:37:30.650 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:37:30.644+0000 7f9a72ffd640 1 -- 172.21.15.114:0/2450880004 <== mon.0 v2:172.21.15.114:3300/0 4 ==== mgrmap(e 26) ==== 138778+0+0 (secure 0 0 0) 0x7f9a6c004e80 con 0x7f9a7c106170 2024-04-23T10:37:30.650 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:37:30.644+0000 7f9a8698a740 1 Processor -- start 2024-04-23T10:37:30.650 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:37:30.645+0000 7f9a8698a740 1 -- start start 2024-04-23T10:37:30.650 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:37:30.645+0000 7f9a8698a740 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x5574ea762620 0x5574ea762a00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T10:37:30.650 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:37:30.645+0000 7f9a72ffd640 1 --2- 172.21.15.114:0/2450880004 >> [v2:172.21.15.114:6800/1395693466,v1:172.21.15.114:6801/1395693466] conn(0x7f9a680a3dc0 0x7f9a680a6280 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T10:37:30.650 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:37:30.645+0000 7f9a8698a740 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x5574ea762f40 0x5574ea76d2c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T10:37:30.650 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:37:30.645+0000 7f9a8698a740 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x5574ea76d800 0x5574ea76fbf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T10:37:30.650 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:37:30.645+0000 7f9a8698a740 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x5574ea6c7970 con 0x5574ea762620 2024-04-23T10:37:30.650 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:37:30.645+0000 7f9a8698a740 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 -- 0x5574ea6644a0 con 0x5574ea762f40 2024-04-23T10:37:30.651 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:37:30.645+0000 7f9a8698a740 1 -- --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_getmap magic: 0 -- 0x5574ea6c3930 con 0x5574ea76d800 2024-04-23T10:37:30.651 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:37:30.645+0000 7f9a8213e640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x5574ea762620 0x5574ea762a00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T10:37:30.651 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:37:30.645+0000 7f9a8193d640 1 --2- 172.21.15.114:0/2450880004 >> [v2:172.21.15.114:6800/1395693466,v1:172.21.15.114:6801/1395693466] conn(0x7f9a680a3dc0 0x7f9a680a6280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T10:37:30.651 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:37:30.645+0000 7f9a8213e640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x5574ea762620 0x5574ea762a00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.114:51502/0 (socket says 172.21.15.114:51502) 2024-04-23T10:37:30.651 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:37:30.645+0000 7f9a8213e640 1 -- 172.21.15.114:0/1436589100 learned_addr learned my addr 172.21.15.114:0/1436589100 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-04-23T10:37:30.651 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:37:30.645+0000 7f9a72ffd640 1 -- 172.21.15.114:0/2450880004 <== mon.0 v2:172.21.15.114:3300/0 5 ==== osd_map(104..104 src has 1..104) ==== 8629+0+0 (secure 0 0 0) 0x7f9a6c0d0310 con 0x7f9a7c106170 2024-04-23T10:37:30.651 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:37:30.645+0000 7f9a8213e640 1 --2- 172.21.15.114:0/1436589100 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x5574ea76d800 0x5574ea76fbf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T10:37:30.651 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:37:30.645+0000 7f9a8113c640 1 --2- 172.21.15.114:0/1436589100 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x5574ea762f40 0x5574ea76d2c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T10:37:30.651 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:37:30.645+0000 7f9a8213e640 1 -- 172.21.15.114:0/1436589100 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x5574ea76d800 msgr2=0x5574ea76fbf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T10:37:30.651 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:37:30.645+0000 7f9a8213e640 1 --2- 172.21.15.114:0/1436589100 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x5574ea76d800 0x5574ea76fbf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:37:30.651 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:37:30.645+0000 7f9a8213e640 1 -- 172.21.15.114:0/1436589100 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x5574ea762f40 msgr2=0x5574ea76d2c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T10:37:30.651 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:37:30.645+0000 7f9a8213e640 1 --2- 172.21.15.114:0/1436589100 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x5574ea762f40 0x5574ea76d2c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:37:30.651 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:37:30.645+0000 7f9a8213e640 1 -- 172.21.15.114:0/1436589100 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5574ea3da210 con 0x5574ea762620 2024-04-23T10:37:30.652 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:37:30.645+0000 7f9a8113c640 1 --2- 172.21.15.114:0/1436589100 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x5574ea762f40 0x5574ea76d2c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-23T10:37:30.652 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:37:30.646+0000 7f9a8193d640 1 --2- 172.21.15.114:0/2450880004 >> [v2:172.21.15.114:6800/1395693466,v1:172.21.15.114:6801/1395693466] conn(0x7f9a680a3dc0 0x7f9a680a6280 secure :-1 s=READY pgs=133 cs=0 l=1 rev1=1 crypto rx=0x7f9a78031440 tx=0x7f9a78005a20 comp rx=0 tx=0).ready entity=mgr.15291 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T10:37:30.652 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:37:30.646+0000 7f9a8213e640 1 --2- 172.21.15.114:0/1436589100 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x5574ea762620 0x5574ea762a00 secure :-1 s=READY pgs=857 cs=0 l=1 rev1=1 crypto rx=0x7f9a74012940 tx=0x7f9a74009730 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c6fca7b24edbec0d server_cookie=0 in_seq=0 out_seq=0 2024-04-23T10:37:30.652 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:37:30.646+0000 7f9a67fff640 1 -- 172.21.15.114:0/1436589100 <== mon.0 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9a74010770 con 0x5574ea762620 2024-04-23T10:37:30.652 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:37:30.646+0000 7f9a67fff640 1 -- 172.21.15.114:0/1436589100 <== mon.0 v2:172.21.15.114:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f9a74010dc0 con 0x5574ea762620 2024-04-23T10:37:30.652 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:37:30.646+0000 7f9a67fff640 1 -- 172.21.15.114:0/1436589100 <== mon.0 v2:172.21.15.114:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9a74011d20 con 0x5574ea762620 2024-04-23T10:37:30.652 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:37:30.647+0000 7f9a8698a740 1 -- 172.21.15.114:0/1436589100 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x5574ea762620 msgr2=0x5574ea762a00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T10:37:30.652 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:37:30.647+0000 7f9a8698a740 1 --2- 172.21.15.114:0/1436589100 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x5574ea762620 0x5574ea762a00 secure :-1 s=READY pgs=857 cs=0 l=1 rev1=1 crypto rx=0x7f9a74012940 tx=0x7f9a74009730 comp rx=0 tx=0).stop 2024-04-23T10:37:30.652 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:37:30.647+0000 7f9a8698a740 1 -- 172.21.15.114:0/1436589100 shutdown_connections 2024-04-23T10:37:30.652 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:37:30.647+0000 7f9a8698a740 1 --2- 172.21.15.114:0/1436589100 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x5574ea76d800 0x5574ea76fbf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:37:30.652 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:37:30.647+0000 7f9a8698a740 1 --2- 172.21.15.114:0/1436589100 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x5574ea762f40 0x5574ea76d2c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:37:30.652 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:37:30.647+0000 7f9a8698a740 1 --2- 172.21.15.114:0/1436589100 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x5574ea762620 0x5574ea762a00 unknown :-1 s=CLOSED pgs=857 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:37:30.652 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:37:30.647+0000 7f9a8698a740 1 -- 172.21.15.114:0/1436589100 >> 172.21.15.114:0/1436589100 conn(0x5574ea75eb30 msgr2=0x5574ea75ef10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T10:37:30.652 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:37:30.647+0000 7f9a8698a740 1 -- 172.21.15.114:0/1436589100 shutdown_connections 2024-04-23T10:37:30.653 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:37:30.647+0000 7f9a8698a740 1 -- 172.21.15.114:0/1436589100 wait complete. 2024-04-23T10:37:30.653 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:37:30.647+0000 7f9a8698a740 1 -- start start 2024-04-23T10:37:30.653 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:37:30.647+0000 7f9a8698a740 1 Processor -- start 2024-04-23T10:37:30.653 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:37:30.647+0000 7f9a8698a740 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x5574ea762620 0x5574ea76d0d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T10:37:30.653 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:37:30.648+0000 7f9a8698a740 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x5574ea762f40 0x5574ea76a3c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T10:37:30.653 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:37:30.648+0000 7f9a8698a740 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x5574ea76d800 0x5574ea76a900 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T10:37:30.653 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:37:30.648+0000 7f9a8698a740 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x5574ea7703d0 con 0x5574ea762f40 2024-04-23T10:37:30.653 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:37:30.648+0000 7f9a8698a740 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 -- 0x5574ea6620b0 con 0x5574ea762620 2024-04-23T10:37:30.653 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:37:30.648+0000 7f9a8698a740 1 -- --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_getmap magic: 0 -- 0x5574ea770550 con 0x5574ea76d800 2024-04-23T10:37:30.653 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:37:30.648+0000 7f9a8698a740 10 client.0 _ll_register_callbacks cb 0x5574ea64fd30 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-23T10:37:30.653 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:37:30.648+0000 7f9a8113c640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x5574ea762f40 0x5574ea76a3c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T10:37:30.653 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:37:30.648+0000 7f9a8113c640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x5574ea762f40 0x5574ea76a3c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.114:51516/0 (socket says 172.21.15.114:51516) 2024-04-23T10:37:30.653 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:37:30.648+0000 7f9a8113c640 1 -- 172.21.15.114:0/4223939080 learned_addr learned my addr 172.21.15.114:0/4223939080 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-04-23T10:37:30.654 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:37:30.648+0000 7f9a8213e640 1 --2- 172.21.15.114:0/4223939080 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x5574ea76d800 0x5574ea76a900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T10:37:30.654 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:37:30.648+0000 7f9a8193d640 1 --2- 172.21.15.114:0/4223939080 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x5574ea762620 0x5574ea76d0d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T10:37:30.654 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:37:30.648+0000 7f9a8113c640 1 -- 172.21.15.114:0/4223939080 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x5574ea76d800 msgr2=0x5574ea76a900 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T10:37:30.654 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:37:30.648+0000 7f9a8113c640 1 --2- 172.21.15.114:0/4223939080 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x5574ea76d800 0x5574ea76a900 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:37:30.654 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:37:30.648+0000 7f9a8113c640 1 -- 172.21.15.114:0/4223939080 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x5574ea762620 msgr2=0x5574ea76d0d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T10:37:30.654 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:37:30.648+0000 7f9a8113c640 1 --2- 172.21.15.114:0/4223939080 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x5574ea762620 0x5574ea76d0d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:37:30.654 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:37:30.648+0000 7f9a8113c640 1 -- 172.21.15.114:0/4223939080 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5574ea4a6620 con 0x5574ea762f40 2024-04-23T10:37:30.654 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:37:30.648+0000 7f9a8213e640 1 --2- 172.21.15.114:0/4223939080 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x5574ea76d800 0x5574ea76a900 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-23T10:37:30.654 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:37:30.648+0000 7f9a8113c640 1 --2- 172.21.15.114:0/4223939080 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x5574ea762f40 0x5574ea76a3c0 secure :-1 s=READY pgs=858 cs=0 l=1 rev1=1 crypto rx=0x7f9a6c009610 tx=0x7f9a6c09bc90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d01acb27ddb79784 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T10:37:30.654 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:37:30.648+0000 7f9a657fa640 10 client.0 ms_handle_connect on v2:172.21.15.114:3300/0 2024-04-23T10:37:30.654 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:37:30.648+0000 7f9a657fa640 1 -- 172.21.15.114:0/4223939080 <== mon.0 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9a6c020040 con 0x5574ea762f40 2024-04-23T10:37:30.654 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:37:30.648+0000 7f9a657fa640 1 -- 172.21.15.114:0/4223939080 <== mon.0 v2:172.21.15.114:3300/0 2 ==== osd_map(104..104 src has 1..104) ==== 8629+0+0 (secure 0 0 0) 0x7f9a6c0a0120 con 0x5574ea762f40 2024-04-23T10:37:30.654 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:37:30.648+0000 7f9a8698a740 1 -- 172.21.15.114:0/4223939080 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f9a74010dc0 con 0x5574ea762f40 2024-04-23T10:37:30.655 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:37:30.648+0000 7f9a657fa640 1 -- 172.21.15.114:0/4223939080 <== mon.0 v2:172.21.15.114:3300/0 3 ==== mon_get_version_reply(handle=1 version=37) ==== 24+0+0 (secure 0 0 0) 0x7f9a6c09c240 con 0x5574ea762f40 2024-04-23T10:37:30.655 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:37:30.648+0000 7f9a8698a740 10 client.16836 fetch_fsmap learned FSMap version 37 2024-04-23T10:37:30.655 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:37:30.648+0000 7f9a8698a740 1 -- 172.21.15.114:0/4223939080 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({config=0+,fsmap=37,monmap=4+}) -- 0x5574ea77c2d0 con 0x5574ea762f40 2024-04-23T10:37:30.655 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:37:30.649+0000 7f9a657fa640 1 -- 172.21.15.114:0/4223939080 <== mon.0 v2:172.21.15.114:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f9a74010dc0 con 0x5574ea762f40 2024-04-23T10:37:30.655 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:37:30.649+0000 7f9a657fa640 1 -- 172.21.15.114:0/4223939080 <== mon.0 v2:172.21.15.114:3300/0 5 ==== fsmap(e 37) ==== 8358+0+0 (secure 0 0 0) 0x7f9a6c0b0ce0 con 0x5574ea762f40 2024-04-23T10:37:30.655 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:37:30.649+0000 7f9a8698a740 10 client.16836 fetch_fsmap finished waiting for FSMap version 37 2024-04-23T10:37:30.655 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:37:30.649+0000 7f9a8698a740 10 client.16836 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-04-23T10:37:30.655 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:37:30.649+0000 7f9a8698a740 20 client.16836 populate_metadata read hostname 'smithi114' 2024-04-23T10:37:30.656 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:37:30.649+0000 7f9a8698a740 1 --2- 172.21.15.114:0/4223939080 >> [v2:172.21.15.119:6838/937297340,v1:172.21.15.119:6839/937297340] conn(0x5574ea77d790 0x5574ea77fbe0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T10:37:30.656 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:37:30.649+0000 7f9a8698a740 4 client.16836 mds_command: new command op to 24545 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-23T10:37:30.656 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:37:30.649+0000 7f9a8698a740 1 -- 172.21.15.114:0/4223939080 --> [v2:172.21.15.119:6838/937297340,v1:172.21.15.119:6839/937297340] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x5574ea6d5390 con 0x5574ea77d790 2024-04-23T10:37:30.656 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:37:30.649+0000 7f9a8193d640 1 --2- 172.21.15.114:0/4223939080 >> [v2:172.21.15.119:6838/937297340,v1:172.21.15.119:6839/937297340] conn(0x5574ea77d790 0x5574ea77fbe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T10:37:30.656 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:37:30.649+0000 7f9a657fa640 10 client.16836 ms_handle_connect on v2:172.21.15.119:6838/937297340 2024-04-23T10:37:30.656 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:37:30.649+0000 7f9a8193d640 1 --2- 172.21.15.114:0/4223939080 >> [v2:172.21.15.119:6838/937297340,v1:172.21.15.119:6839/937297340] conn(0x5574ea77d790 0x5574ea77fbe0 crc :-1 s=READY pgs=158 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=3e73cb6cce9257de server_cookie=c4c64d495ae20c6e in_seq=0 out_seq=0 2024-04-23T10:37:30.656 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:37:30.650+0000 7f9a657fa640 1 -- 172.21.15.114:0/4223939080 <== mds.0 v2:172.21.15.119:6838/937297340 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x5574ea6d5390 con 0x5574ea77d790 2024-04-23T10:37:30.656 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:37:30.650+0000 7f9a657fa640 10 client.16836 handle_command_reply: tid=0 2024-04-23T10:37:30.656 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:37:30.650+0000 7f9a8698a740 1 client.16836 shutdown 2024-04-23T10:37:30.656 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:37:30.650+0000 7f9a8698a740 1 -- 172.21.15.114:0/4223939080 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x5574ea762f40 msgr2=0x5574ea76a3c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T10:37:30.656 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:37:30.650+0000 7f9a8698a740 1 --2- 172.21.15.114:0/4223939080 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x5574ea762f40 0x5574ea76a3c0 secure :-1 s=READY pgs=858 cs=0 l=1 rev1=1 crypto rx=0x7f9a6c009610 tx=0x7f9a6c09bc90 comp rx=0 tx=0).stop 2024-04-23T10:37:30.656 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:37:30.650+0000 7f9a8698a740 1 -- 172.21.15.114:0/4223939080 shutdown_connections 2024-04-23T10:37:30.656 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:37:30.650+0000 7f9a8698a740 1 --2- 172.21.15.114:0/4223939080 >> [v2:172.21.15.119:6838/937297340,v1:172.21.15.119:6839/937297340] conn(0x5574ea77d790 0x5574ea77fbe0 crc :-1 s=READY pgs=158 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:37:30.656 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:37:30.650+0000 7f9a8698a740 1 --2- 172.21.15.114:0/4223939080 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x5574ea76d800 0x5574ea76a900 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:37:30.657 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:37:30.650+0000 7f9a8698a740 1 --2- 172.21.15.114:0/4223939080 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x5574ea762f40 0x5574ea76a3c0 unknown :-1 s=CLOSED pgs=858 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:37:30.657 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:37:30.650+0000 7f9a657fa640 0 client.16836 ms_handle_reset on v2:172.21.15.119:6838/937297340 2024-04-23T10:37:30.657 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:37:30.650+0000 7f9a8698a740 1 --2- 172.21.15.114:0/4223939080 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x5574ea762620 0x5574ea76d0d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:37:30.657 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:37:30.650+0000 7f9a8698a740 1 -- 172.21.15.114:0/4223939080 >> 172.21.15.114:0/4223939080 conn(0x5574ea75eb30 msgr2=0x5574ea7648b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T10:37:30.657 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:37:30.650+0000 7f9a8698a740 1 -- 172.21.15.114:0/4223939080 shutdown_connections 2024-04-23T10:37:30.657 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:37:30.650+0000 7f9a8698a740 1 -- 172.21.15.114:0/4223939080 wait complete. 2024-04-23T10:37:30.657 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:37:30.651+0000 7f9a8698a740 20 client.16836 trim_cache size 0 max 16384 2024-04-23T10:37:30.669 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:37:30.665+0000 7f9a8698a740 1 Processor -- start 2024-04-23T10:37:30.669 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:37:30.665+0000 7f9a8698a740 1 -- start start 2024-04-23T10:37:30.669 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:37:30.665+0000 7f9a8698a740 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x5574ea762620 0x5574ea779290 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T10:37:30.669 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:37:30.665+0000 7f9a8698a740 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x5574ea762f40 0x5574ea76cc00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T10:37:30.669 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:37:30.666+0000 7f9a8698a740 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x5574ea76d800 0x5574ea7854f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T10:37:30.669 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:37:30.666+0000 7f9a8698a740 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x7f9a7c1b0660 con 0x5574ea762620 2024-04-23T10:37:30.669 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:37:30.666+0000 7f9a8698a740 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 -- 0x7f9a7c1ade70 con 0x5574ea76d800 2024-04-23T10:37:30.669 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:37:30.666+0000 7f9a8698a740 1 -- --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_getmap magic: 0 -- 0x7f9a7c1abce0 con 0x5574ea762f40 2024-04-23T10:37:30.669 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:37:30.666+0000 7f9a8193d640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x5574ea762620 0x5574ea779290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T10:37:30.670 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:37:30.666+0000 7f9a8193d640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x5574ea762620 0x5574ea779290 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.114:51520/0 (socket says 172.21.15.114:51520) 2024-04-23T10:37:30.670 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:37:30.666+0000 7f9a8193d640 1 -- 172.21.15.114:0/973177193 learned_addr learned my addr 172.21.15.114:0/973177193 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-04-23T10:37:30.670 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:37:30.666+0000 7f9a8113c640 1 --2- 172.21.15.114:0/973177193 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x5574ea762f40 0x5574ea76cc00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T10:37:30.670 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:37:30.666+0000 7f9a8193d640 1 -- 172.21.15.114:0/973177193 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x5574ea762f40 msgr2=0x5574ea76cc00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T10:37:30.670 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:37:30.666+0000 7f9a8213e640 1 --2- 172.21.15.114:0/973177193 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x5574ea76d800 0x5574ea7854f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T10:37:30.670 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:37:30.666+0000 7f9a8193d640 1 --2- 172.21.15.114:0/973177193 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x5574ea762f40 0x5574ea76cc00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:37:30.670 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:37:30.666+0000 7f9a8193d640 1 -- 172.21.15.114:0/973177193 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x5574ea76d800 msgr2=0x5574ea7854f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T10:37:30.670 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:37:30.666+0000 7f9a8193d640 1 --2- 172.21.15.114:0/973177193 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x5574ea76d800 0x5574ea7854f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:37:30.670 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:37:30.666+0000 7f9a8193d640 1 -- 172.21.15.114:0/973177193 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5574ea75e430 con 0x5574ea762620 2024-04-23T10:37:30.670 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:37:30.666+0000 7f9a8113c640 1 --2- 172.21.15.114:0/973177193 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x5574ea762f40 0x5574ea76cc00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-23T10:37:30.670 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:37:30.666+0000 7f9a8193d640 1 --2- 172.21.15.114:0/973177193 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x5574ea762620 0x5574ea779290 secure :-1 s=READY pgs=859 cs=0 l=1 rev1=1 crypto rx=0x7f9a7802a930 tx=0x7f9a7803c000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=46d995aa527c9069 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T10:37:30.670 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:37:30.666+0000 7f9a677fe640 1 -- 172.21.15.114:0/973177193 <== mon.0 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9a78035050 con 0x5574ea762620 2024-04-23T10:37:30.671 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:37:30.666+0000 7f9a677fe640 1 -- 172.21.15.114:0/973177193 <== mon.0 v2:172.21.15.114:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f9a78031820 con 0x5574ea762620 2024-04-23T10:37:30.671 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:37:30.666+0000 7f9a677fe640 1 -- 172.21.15.114:0/973177193 <== mon.0 v2:172.21.15.114:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9a78005620 con 0x5574ea762620 2024-04-23T10:37:30.671 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:37:30.667+0000 7f9a8698a740 1 -- 172.21.15.114:0/973177193 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x5574ea762620 msgr2=0x5574ea779290 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T10:37:30.671 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:37:30.667+0000 7f9a8698a740 1 --2- 172.21.15.114:0/973177193 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x5574ea762620 0x5574ea779290 secure :-1 s=READY pgs=859 cs=0 l=1 rev1=1 crypto rx=0x7f9a7802a930 tx=0x7f9a7803c000 comp rx=0 tx=0).stop 2024-04-23T10:37:30.671 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:37:30.667+0000 7f9a8698a740 1 -- 172.21.15.114:0/973177193 shutdown_connections 2024-04-23T10:37:30.671 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:37:30.667+0000 7f9a8698a740 1 --2- 172.21.15.114:0/973177193 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x5574ea76d800 0x5574ea7854f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:37:30.671 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:37:30.667+0000 7f9a8698a740 1 --2- 172.21.15.114:0/973177193 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x5574ea762f40 0x5574ea76cc00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:37:30.671 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:37:30.667+0000 7f9a8698a740 1 --2- 172.21.15.114:0/973177193 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x5574ea762620 0x5574ea779290 unknown :-1 s=CLOSED pgs=859 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:37:30.671 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:37:30.667+0000 7f9a8698a740 1 -- 172.21.15.114:0/973177193 >> 172.21.15.114:0/973177193 conn(0x5574ea75eb30 msgr2=0x5574ea76a380 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T10:37:30.671 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:37:30.667+0000 7f9a8698a740 1 -- 172.21.15.114:0/973177193 shutdown_connections 2024-04-23T10:37:30.671 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:37:30.667+0000 7f9a8698a740 1 -- 172.21.15.114:0/973177193 wait complete. 2024-04-23T10:37:30.671 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:37:30.667+0000 7f9a8698a740 1 -- start start 2024-04-23T10:37:30.671 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:37:30.667+0000 7f9a8698a740 1 Processor -- start 2024-04-23T10:37:30.671 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:37:30.668+0000 7f9a8698a740 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x5574ea762620 0x5574ea775bc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T10:37:30.671 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:37:30.668+0000 7f9a8698a740 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x5574ea762f40 0x5574ea770f30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T10:37:30.671 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:37:30.668+0000 7f9a8698a740 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x5574ea76d800 0x5574ea771490 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T10:37:30.672 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:37:30.668+0000 7f9a8698a740 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x7f9a7c1ac330 con 0x5574ea762620 2024-04-23T10:37:30.672 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:37:30.668+0000 7f9a8698a740 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 -- 0x7f9a7c1ad7a0 con 0x5574ea76d800 2024-04-23T10:37:30.672 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:37:30.668+0000 7f9a8698a740 1 -- --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_getmap magic: 0 -- 0x7f9a7c1aca00 con 0x5574ea762f40 2024-04-23T10:37:30.672 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:37:30.668+0000 7f9a8698a740 10 client.0 _ll_register_callbacks cb 0x7f9a6c09be50 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-23T10:37:30.672 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:37:30.668+0000 7f9a8193d640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x5574ea762620 0x5574ea775bc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T10:37:30.672 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:37:30.668+0000 7f9a8193d640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x5574ea762620 0x5574ea775bc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.114:51530/0 (socket says 172.21.15.114:51530) 2024-04-23T10:37:30.672 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:37:30.668+0000 7f9a8193d640 1 -- 172.21.15.114:0/2534859169 learned_addr learned my addr 172.21.15.114:0/2534859169 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-04-23T10:37:30.672 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:37:30.668+0000 7f9a8213e640 1 --2- 172.21.15.114:0/2534859169 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x5574ea76d800 0x5574ea771490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T10:37:30.672 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:37:30.668+0000 7f9a8113c640 1 --2- 172.21.15.114:0/2534859169 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x5574ea762f40 0x5574ea770f30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T10:37:30.672 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:37:30.668+0000 7f9a8193d640 1 -- 172.21.15.114:0/2534859169 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x5574ea762f40 msgr2=0x5574ea770f30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T10:37:30.672 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:37:30.668+0000 7f9a8193d640 1 --2- 172.21.15.114:0/2534859169 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x5574ea762f40 0x5574ea770f30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:37:30.672 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:37:30.668+0000 7f9a8193d640 1 -- 172.21.15.114:0/2534859169 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x5574ea76d800 msgr2=0x5574ea771490 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T10:37:30.672 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:37:30.668+0000 7f9a8193d640 1 --2- 172.21.15.114:0/2534859169 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x5574ea76d800 0x5574ea771490 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:37:30.673 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:37:30.668+0000 7f9a8193d640 1 -- 172.21.15.114:0/2534859169 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5574ea771d10 con 0x5574ea762620 2024-04-23T10:37:30.673 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:37:30.668+0000 7f9a8213e640 1 --2- 172.21.15.114:0/2534859169 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x5574ea76d800 0x5574ea771490 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-23T10:37:30.673 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:37:30.668+0000 7f9a8193d640 1 --2- 172.21.15.114:0/2534859169 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x5574ea762620 0x5574ea775bc0 secure :-1 s=READY pgs=860 cs=0 l=1 rev1=1 crypto rx=0x7f9a78005740 tx=0x7f9a78005770 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4fcca69cca6c5c3 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T10:37:30.673 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:37:30.668+0000 7f9a657fa640 10 client.0 ms_handle_connect on v2:172.21.15.114:3300/0 2024-04-23T10:37:30.673 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:37:30.668+0000 7f9a657fa640 1 -- 172.21.15.114:0/2534859169 <== mon.0 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9a78035070 con 0x5574ea762620 2024-04-23T10:37:30.673 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:37:30.668+0000 7f9a8698a740 1 -- 172.21.15.114:0/2534859169 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f9a78031820 con 0x5574ea762620 2024-04-23T10:37:30.673 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:37:30.668+0000 7f9a657fa640 1 -- 172.21.15.114:0/2534859169 <== mon.0 v2:172.21.15.114:3300/0 2 ==== osd_map(104..104 src has 1..104) ==== 8629+0+0 (secure 0 0 0) 0x7f9a78053520 con 0x5574ea762620 2024-04-23T10:37:30.673 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:37:30.669+0000 7f9a657fa640 1 -- 172.21.15.114:0/2534859169 <== mon.0 v2:172.21.15.114:3300/0 3 ==== mon_get_version_reply(handle=1 version=37) ==== 24+0+0 (secure 0 0 0) 0x7f9a78053720 con 0x5574ea762620 2024-04-23T10:37:30.674 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:37:30.669+0000 7f9a8698a740 10 client.16842 fetch_fsmap learned FSMap version 37 2024-04-23T10:37:30.674 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:37:30.669+0000 7f9a8698a740 1 -- 172.21.15.114:0/2534859169 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({config=0+,fsmap=37,monmap=4+}) -- 0x5574ea7e3d50 con 0x5574ea762620 2024-04-23T10:37:30.674 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:37:30.669+0000 7f9a657fa640 1 -- 172.21.15.114:0/2534859169 <== mon.0 v2:172.21.15.114:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f9a78031820 con 0x5574ea762620 2024-04-23T10:37:30.674 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:37:30.669+0000 7f9a657fa640 1 -- 172.21.15.114:0/2534859169 <== mon.0 v2:172.21.15.114:3300/0 5 ==== fsmap(e 37) ==== 8358+0+0 (secure 0 0 0) 0x7f9a7803baf0 con 0x5574ea762620 2024-04-23T10:37:30.674 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:37:30.669+0000 7f9a8698a740 10 client.16842 fetch_fsmap finished waiting for FSMap version 37 2024-04-23T10:37:30.674 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:37:30.669+0000 7f9a8698a740 10 client.16842 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-04-23T10:37:30.674 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:37:30.669+0000 7f9a8698a740 20 client.16842 populate_metadata read hostname 'smithi114' 2024-04-23T10:37:30.674 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:37:30.669+0000 7f9a8698a740 1 --2- 172.21.15.114:0/2534859169 >> [v2:172.21.15.119:6838/937297340,v1:172.21.15.119:6839/937297340] conn(0x5574ea7e4420 0x5574ea7e6810 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T10:37:30.674 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:37:30.669+0000 7f9a8698a740 4 client.16842 mds_command: new command op to 24545 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-04-23T10:37:30.674 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:37:30.669+0000 7f9a8698a740 1 -- 172.21.15.114:0/2534859169 --> [v2:172.21.15.119:6838/937297340,v1:172.21.15.119:6839/937297340] -- command(tid 0: {"prefix": "flush journal"}) -- 0x5574ea6cf200 con 0x5574ea7e4420 2024-04-23T10:37:30.674 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:37:30.669+0000 7f9a8113c640 1 --2- 172.21.15.114:0/2534859169 >> [v2:172.21.15.119:6838/937297340,v1:172.21.15.119:6839/937297340] conn(0x5574ea7e4420 0x5574ea7e6810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T10:37:30.674 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:37:30.670+0000 7f9a657fa640 10 client.16842 ms_handle_connect on v2:172.21.15.119:6838/937297340 2024-04-23T10:37:30.674 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:37:30.670+0000 7f9a8113c640 1 --2- 172.21.15.114:0/2534859169 >> [v2:172.21.15.119:6838/937297340,v1:172.21.15.119:6839/937297340] conn(0x5574ea7e4420 0x5574ea7e6810 crc :-1 s=READY pgs=159 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=9b54686117d27b8f server_cookie=1f7906530738f093 in_seq=0 out_seq=0 2024-04-23T10:37:30.706 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:37:30.706+0000 7f9a657fa640 1 -- 172.21.15.114:0/2534859169 <== mds.0 v2:172.21.15.119:6838/937297340 1 ==== command_reply(tid 0: 0 ) ==== 8+0+44 (crc 0 0 0) 0x5574ea6cf200 con 0x5574ea7e4420 2024-04-23T10:37:30.706 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:37:30.706+0000 7f9a657fa640 10 client.16842 handle_command_reply: tid=0 2024-04-23T10:37:30.706 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:37:30.706+0000 7f9a8698a740 1 client.16842 shutdown 2024-04-23T10:37:30.706 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:37:30.707+0000 7f9a8698a740 1 -- 172.21.15.114:0/2534859169 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x5574ea762620 msgr2=0x5574ea775bc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T10:37:30.706 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:37:30.707+0000 7f9a8698a740 1 --2- 172.21.15.114:0/2534859169 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x5574ea762620 0x5574ea775bc0 secure :-1 s=READY pgs=860 cs=0 l=1 rev1=1 crypto rx=0x7f9a78005740 tx=0x7f9a78005770 comp rx=0 tx=0).stop 2024-04-23T10:37:30.707 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:37:30.707+0000 7f9a8698a740 1 -- 172.21.15.114:0/2534859169 shutdown_connections 2024-04-23T10:37:30.707 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:37:30.707+0000 7f9a8698a740 1 --2- 172.21.15.114:0/2534859169 >> [v2:172.21.15.119:6838/937297340,v1:172.21.15.119:6839/937297340] conn(0x5574ea7e4420 0x5574ea7e6810 crc :-1 s=READY pgs=159 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:37:30.707 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:37:30.707+0000 7f9a8698a740 1 --2- 172.21.15.114:0/2534859169 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x5574ea76d800 0x5574ea771490 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:37:30.707 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:37:30.707+0000 7f9a8698a740 1 --2- 172.21.15.114:0/2534859169 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x5574ea762f40 0x5574ea770f30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:37:30.707 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:37:30.707+0000 7f9a8698a740 1 --2- 172.21.15.114:0/2534859169 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x5574ea762620 0x5574ea775bc0 unknown :-1 s=CLOSED pgs=860 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:37:30.707 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:37:30.707+0000 7f9a657fa640 0 client.16842 ms_handle_reset on v2:172.21.15.119:6838/937297340 2024-04-23T10:37:30.707 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:37:30.707+0000 7f9a8698a740 1 -- 172.21.15.114:0/2534859169 >> 172.21.15.114:0/2534859169 conn(0x5574ea75eb30 msgr2=0x5574ea763fd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T10:37:30.707 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:37:30.707+0000 7f9a8698a740 1 -- 172.21.15.114:0/2534859169 shutdown_connections 2024-04-23T10:37:30.707 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:37:30.707+0000 7f9a8698a740 1 -- 172.21.15.114:0/2534859169 wait complete. 2024-04-23T10:37:30.707 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:37:30.707+0000 7f9a8698a740 20 client.16842 trim_cache size 0 max 16384 2024-04-23T10:37:30.707 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stdout:{ 2024-04-23T10:37:30.707 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stdout: "message": "", 2024-04-23T10:37:30.708 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stdout: "return_code": 0 2024-04-23T10:37:30.708 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stdout:} 2024-04-23T10:37:30.708 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:37:30.708+0000 7f9a83ba1640 1 -- 172.21.15.114:0/2450880004 >> [v2:172.21.15.114:6800/1395693466,v1:172.21.15.114:6801/1395693466] conn(0x7f9a680a3dc0 msgr2=0x7f9a680a6280 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T10:37:30.708 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:37:30.708+0000 7f9a83ba1640 1 --2- 172.21.15.114:0/2450880004 >> [v2:172.21.15.114:6800/1395693466,v1:172.21.15.114:6801/1395693466] conn(0x7f9a680a3dc0 0x7f9a680a6280 secure :-1 s=READY pgs=133 cs=0 l=1 rev1=1 crypto rx=0x7f9a78031440 tx=0x7f9a78005a20 comp rx=0 tx=0).stop 2024-04-23T10:37:30.708 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:37:30.708+0000 7f9a83ba1640 1 -- 172.21.15.114:0/2450880004 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f9a7c106170 msgr2=0x7f9a7c19e940 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T10:37:30.709 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:37:30.708+0000 7f9a83ba1640 1 --2- 172.21.15.114:0/2450880004 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f9a7c106170 0x7f9a7c19e940 secure :-1 s=READY pgs=856 cs=0 l=1 rev1=1 crypto rx=0x7f9a6c00d740 tx=0x7f9a6c00dc00 comp rx=0 tx=0).stop 2024-04-23T10:37:30.709 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:37:30.708+0000 7f9a83ba1640 1 -- 172.21.15.114:0/2450880004 shutdown_connections 2024-04-23T10:37:30.709 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:37:30.708+0000 7f9a83ba1640 1 --2- 172.21.15.114:0/2450880004 >> [v2:172.21.15.114:6800/1395693466,v1:172.21.15.114:6801/1395693466] conn(0x7f9a680a3dc0 0x7f9a680a6280 unknown :-1 s=CLOSED pgs=133 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:37:30.709 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:37:30.708+0000 7f9a83ba1640 1 --2- 172.21.15.114:0/2450880004 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f9a7c1985b0 0x7f9a7c198a60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:37:30.709 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:37:30.708+0000 7f9a83ba1640 1 --2- 172.21.15.114:0/2450880004 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f9a7c106170 0x7f9a7c19e940 unknown :-1 s=CLOSED pgs=856 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:37:30.709 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:37:30.708+0000 7f9a83ba1640 1 --2- 172.21.15.114:0/2450880004 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f9a7c1010f0 0x7f9a7c19e400 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:37:30.709 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:37:30.708+0000 7f9a83ba1640 1 -- 172.21.15.114:0/2450880004 >> 172.21.15.114:0/2450880004 conn(0x7f9a7c077fd0 msgr2=0x7f9a7c0ff170 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T10:37:30.709 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:37:30.708+0000 7f9a83ba1640 1 -- 172.21.15.114:0/2450880004 shutdown_connections 2024-04-23T10:37:30.709 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:37:30.708+0000 7f9a83ba1640 1 -- 172.21.15.114:0/2450880004 wait complete. 2024-04-23T10:37:30.942 INFO:tasks.workunit.client.0.smithi114.stdout: 1 671303 5.83 MB/sec execute 459 sec latency 1661.034 ms 2024-04-23T10:37:31.942 INFO:tasks.workunit.client.0.smithi114.stdout: 1 671303 5.82 MB/sec execute 460 sec latency 2661.077 ms 2024-04-23T10:37:32.492 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 23 10:37:32 smithi119 ceph-mon[32608]: pgmap v342: 97 pgs: 97 active+clean; 209 MiB data, 2.6 GiB used, 1.0 TiB / 1.0 TiB avail; 341 KiB/s rd, 927 KiB/s wr, 78 op/s 2024-04-23T10:37:32.504 INFO:journalctl@ceph.mon.a.smithi114.stdout:Apr 23 10:37:32 smithi114 ceph-mon[29255]: pgmap v342: 97 pgs: 97 active+clean; 209 MiB data, 2.6 GiB used, 1.0 TiB / 1.0 TiB avail; 341 KiB/s rd, 927 KiB/s wr, 78 op/s 2024-04-23T10:37:32.577 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 23 10:37:32 smithi204 ceph-mon[32649]: pgmap v342: 97 pgs: 97 active+clean; 209 MiB data, 2.6 GiB used, 1.0 TiB / 1.0 TiB avail; 341 KiB/s rd, 927 KiB/s wr, 78 op/s 2024-04-23T10:37:32.942 INFO:tasks.workunit.client.0.smithi114.stdout: 1 671303 5.81 MB/sec execute 461 sec latency 3661.167 ms 2024-04-23T10:37:33.943 INFO:tasks.workunit.client.0.smithi114.stdout: 1 671303 5.79 MB/sec execute 462 sec latency 4661.222 ms 2024-04-23T10:37:34.492 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 23 10:37:34 smithi119 ceph-mon[32608]: pgmap v343: 97 pgs: 97 active+clean; 205 MiB data, 2.6 GiB used, 1.0 TiB / 1.0 TiB avail; 419 KiB/s rd, 1022 KiB/s wr, 103 op/s 2024-04-23T10:37:34.504 INFO:journalctl@ceph.mon.a.smithi114.stdout:Apr 23 10:37:34 smithi114 ceph-mon[29255]: pgmap v343: 97 pgs: 97 active+clean; 205 MiB data, 2.6 GiB used, 1.0 TiB / 1.0 TiB avail; 419 KiB/s rd, 1022 KiB/s wr, 103 op/s 2024-04-23T10:37:34.578 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 23 10:37:34 smithi204 ceph-mon[32649]: pgmap v343: 97 pgs: 97 active+clean; 205 MiB data, 2.6 GiB used, 1.0 TiB / 1.0 TiB avail; 419 KiB/s rd, 1022 KiB/s wr, 103 op/s 2024-04-23T10:37:34.942 INFO:tasks.workunit.client.0.smithi114.stdout: 1 672034 5.79 MB/sec execute 463 sec latency 4906.817 ms 2024-04-23T10:37:35.942 INFO:tasks.workunit.client.0.smithi114.stdout: 1 674491 5.80 MB/sec execute 464 sec latency 16.277 ms 2024-04-23T10:37:36.492 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 23 10:37:36 smithi119 ceph-mon[32608]: pgmap v344: 97 pgs: 97 active+clean; 205 MiB data, 2.6 GiB used, 1.0 TiB / 1.0 TiB avail; 323 KiB/s rd, 454 KiB/s wr, 66 op/s 2024-04-23T10:37:36.504 INFO:journalctl@ceph.mon.a.smithi114.stdout:Apr 23 10:37:36 smithi114 ceph-mon[29255]: pgmap v344: 97 pgs: 97 active+clean; 205 MiB data, 2.6 GiB used, 1.0 TiB / 1.0 TiB avail; 323 KiB/s rd, 454 KiB/s wr, 66 op/s 2024-04-23T10:37:36.578 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 23 10:37:36 smithi204 ceph-mon[32649]: pgmap v344: 97 pgs: 97 active+clean; 205 MiB data, 2.6 GiB used, 1.0 TiB / 1.0 TiB avail; 323 KiB/s rd, 454 KiB/s wr, 66 op/s 2024-04-23T10:37:36.942 INFO:tasks.workunit.client.0.smithi114.stdout: 1 674835 5.80 MB/sec execute 465 sec latency 851.621 ms 2024-04-23T10:37:37.943 INFO:tasks.workunit.client.0.smithi114.stdout: 1 674835 5.79 MB/sec execute 466 sec latency 1851.749 ms 2024-04-23T10:37:38.492 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 23 10:37:38 smithi119 ceph-mon[32608]: pgmap v345: 97 pgs: 97 active+clean; 208 MiB data, 2.6 GiB used, 1.0 TiB / 1.0 TiB avail; 547 KiB/s rd, 782 KiB/s wr, 103 op/s 2024-04-23T10:37:38.504 INFO:journalctl@ceph.mon.a.smithi114.stdout:Apr 23 10:37:38 smithi114 ceph-mon[29255]: pgmap v345: 97 pgs: 97 active+clean; 208 MiB data, 2.6 GiB used, 1.0 TiB / 1.0 TiB avail; 547 KiB/s rd, 782 KiB/s wr, 103 op/s 2024-04-23T10:37:38.577 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 23 10:37:38 smithi204 ceph-mon[32649]: pgmap v345: 97 pgs: 97 active+clean; 208 MiB data, 2.6 GiB used, 1.0 TiB / 1.0 TiB avail; 547 KiB/s rd, 782 KiB/s wr, 103 op/s 2024-04-23T10:37:38.942 INFO:tasks.workunit.client.0.smithi114.stdout: 1 674835 5.77 MB/sec execute 467 sec latency 2851.863 ms 2024-04-23T10:37:39.943 INFO:tasks.workunit.client.0.smithi114.stdout: 1 674902 5.76 MB/sec execute 468 sec latency 3117.258 ms 2024-04-23T10:37:40.492 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 23 10:37:40 smithi119 ceph-mon[32608]: pgmap v346: 97 pgs: 97 active+clean; 208 MiB data, 2.6 GiB used, 1.0 TiB / 1.0 TiB avail; 302 KiB/s rd, 423 KiB/s wr, 62 op/s 2024-04-23T10:37:40.492 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 23 10:37:40 smithi119 ceph-mon[32608]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=e=up:active} 3 up:standby-replay 6 up:standby 2024-04-23T10:37:40.503 INFO:journalctl@ceph.mon.a.smithi114.stdout:Apr 23 10:37:40 smithi114 ceph-mon[29255]: pgmap v346: 97 pgs: 97 active+clean; 208 MiB data, 2.6 GiB used, 1.0 TiB / 1.0 TiB avail; 302 KiB/s rd, 423 KiB/s wr, 62 op/s 2024-04-23T10:37:40.504 INFO:journalctl@ceph.mon.a.smithi114.stdout:Apr 23 10:37:40 smithi114 ceph-mon[29255]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=e=up:active} 3 up:standby-replay 6 up:standby 2024-04-23T10:37:40.577 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 23 10:37:40 smithi204 ceph-mon[32649]: pgmap v346: 97 pgs: 97 active+clean; 208 MiB data, 2.6 GiB used, 1.0 TiB / 1.0 TiB avail; 302 KiB/s rd, 423 KiB/s wr, 62 op/s 2024-04-23T10:37:40.578 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 23 10:37:40 smithi204 ceph-mon[32649]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=e=up:active} 3 up:standby-replay 6 up:standby 2024-04-23T10:37:40.942 INFO:tasks.workunit.client.0.smithi114.stdout: 1 674902 5.75 MB/sec execute 469 sec latency 1636.441 ms 2024-04-23T10:37:41.943 INFO:tasks.workunit.client.0.smithi114.stdout: 1 674902 5.74 MB/sec execute 470 sec latency 2636.530 ms 2024-04-23T10:37:42.492 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 23 10:37:42 smithi119 ceph-mon[32608]: pgmap v347: 97 pgs: 97 active+clean; 208 MiB data, 2.6 GiB used, 1.0 TiB / 1.0 TiB avail; 302 KiB/s rd, 423 KiB/s wr, 62 op/s 2024-04-23T10:37:42.504 INFO:journalctl@ceph.mon.a.smithi114.stdout:Apr 23 10:37:42 smithi114 ceph-mon[29255]: pgmap v347: 97 pgs: 97 active+clean; 208 MiB data, 2.6 GiB used, 1.0 TiB / 1.0 TiB avail; 302 KiB/s rd, 423 KiB/s wr, 62 op/s 2024-04-23T10:37:42.578 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 23 10:37:42 smithi204 ceph-mon[32649]: pgmap v347: 97 pgs: 97 active+clean; 208 MiB data, 2.6 GiB used, 1.0 TiB / 1.0 TiB avail; 302 KiB/s rd, 423 KiB/s wr, 62 op/s 2024-04-23T10:37:42.943 INFO:tasks.workunit.client.0.smithi114.stdout: 1 674902 5.72 MB/sec execute 471 sec latency 3636.621 ms 2024-04-23T10:37:43.803 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:37:43.804+0000 7fb2ee7a1640 1 -- 172.21.15.114:0/926998419 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fb2e8102b70 msgr2=0x7fb2e8106bb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T10:37:43.803 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:37:43.804+0000 7fb2ee7a1640 1 --2- 172.21.15.114:0/926998419 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fb2e8102b70 0x7fb2e8106bb0 secure :-1 s=READY pgs=861 cs=0 l=1 rev1=1 crypto rx=0x7fb2dc009900 tx=0x7fb2dc02a410 comp rx=0 tx=0).stop 2024-04-23T10:37:43.803 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:37:43.804+0000 7fb2ee7a1640 1 -- 172.21.15.114:0/926998419 shutdown_connections 2024-04-23T10:37:43.803 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:37:43.804+0000 7fb2ee7a1640 1 --2- 172.21.15.114:0/926998419 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fb2e8107290 0x7fb2e810ee60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:37:43.803 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:37:43.804+0000 7fb2ee7a1640 1 --2- 172.21.15.114:0/926998419 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fb2e8102b70 0x7fb2e8106bb0 unknown :-1 s=CLOSED pgs=861 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:37:43.803 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:37:43.804+0000 7fb2ee7a1640 1 --2- 172.21.15.114:0/926998419 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fb2e81021c0 0x7fb2e81025a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:37:43.804 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:37:43.804+0000 7fb2ee7a1640 1 -- 172.21.15.114:0/926998419 >> 172.21.15.114:0/926998419 conn(0x7fb2e80fda20 msgr2=0x7fb2e80ffe40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T10:37:43.804 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:37:43.804+0000 7fb2ee7a1640 1 -- 172.21.15.114:0/926998419 shutdown_connections 2024-04-23T10:37:43.804 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:37:43.804+0000 7fb2ee7a1640 1 -- 172.21.15.114:0/926998419 wait complete. 2024-04-23T10:37:43.804 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:37:43.804+0000 7fb2ee7a1640 1 Processor -- start 2024-04-23T10:37:43.804 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:37:43.804+0000 7fb2ee7a1640 1 -- start start 2024-04-23T10:37:43.805 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:37:43.805+0000 7fb2ee7a1640 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fb2e81021c0 0x7fb2e8074250 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T10:37:43.806 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:37:43.805+0000 7fb2ee7a1640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fb2e8107290 0x7fb2e8074790 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T10:37:43.806 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:37:43.805+0000 7fb2ee7a1640 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fb2e8074cd0 0x7fb2e81a6820 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T10:37:43.806 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:37:43.805+0000 7fb2ee7a1640 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x7fb2e81121b0 con 0x7fb2e8107290 2024-04-23T10:37:43.806 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:37:43.805+0000 7fb2ee7a1640 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 -- 0x7fb2e8112030 con 0x7fb2e8074cd0 2024-04-23T10:37:43.806 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:37:43.805+0000 7fb2ee7a1640 1 -- --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_getmap magic: 0 -- 0x7fb2e8112330 con 0x7fb2e81021c0 2024-04-23T10:37:43.806 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:37:43.805+0000 7fb2e77fe640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fb2e8107290 0x7fb2e8074790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T10:37:43.806 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:37:43.805+0000 7fb2e77fe640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fb2e8107290 0x7fb2e8074790 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.114:60060/0 (socket says 172.21.15.114:60060) 2024-04-23T10:37:43.806 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:37:43.805+0000 7fb2e77fe640 1 -- 172.21.15.114:0/3760507729 learned_addr learned my addr 172.21.15.114:0/3760507729 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-04-23T10:37:43.806 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:37:43.805+0000 7fb2e7fff640 1 --2- 172.21.15.114:0/3760507729 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fb2e81021c0 0x7fb2e8074250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T10:37:43.806 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:37:43.805+0000 7fb2ecd3e640 1 --2- 172.21.15.114:0/3760507729 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fb2e8074cd0 0x7fb2e81a6820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T10:37:43.806 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:37:43.805+0000 7fb2e77fe640 1 -- 172.21.15.114:0/3760507729 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fb2e81021c0 msgr2=0x7fb2e8074250 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T10:37:43.807 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:37:43.805+0000 7fb2e77fe640 1 --2- 172.21.15.114:0/3760507729 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fb2e81021c0 0x7fb2e8074250 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:37:43.807 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:37:43.805+0000 7fb2e77fe640 1 -- 172.21.15.114:0/3760507729 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fb2e8074cd0 msgr2=0x7fb2e81a6820 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T10:37:43.807 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:37:43.805+0000 7fb2e77fe640 1 --2- 172.21.15.114:0/3760507729 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fb2e8074cd0 0x7fb2e81a6820 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:37:43.807 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:37:43.805+0000 7fb2e77fe640 1 -- 172.21.15.114:0/3760507729 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb2e81a6f20 con 0x7fb2e8107290 2024-04-23T10:37:43.807 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:37:43.805+0000 7fb2e77fe640 1 --2- 172.21.15.114:0/3760507729 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fb2e8107290 0x7fb2e8074790 secure :-1 s=READY pgs=862 cs=0 l=1 rev1=1 crypto rx=0x7fb2dc031a90 tx=0x7fb2dc004270 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T10:37:43.807 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:37:43.806+0000 7fb2e57fa640 1 -- 172.21.15.114:0/3760507729 <== mon.0 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb2dc03f030 con 0x7fb2e8107290 2024-04-23T10:37:43.807 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:37:43.806+0000 7fb2ee7a1640 1 -- 172.21.15.114:0/3760507729 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fb2e81a71b0 con 0x7fb2e8107290 2024-04-23T10:37:43.807 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:37:43.806+0000 7fb2e57fa640 1 -- 172.21.15.114:0/3760507729 <== mon.0 v2:172.21.15.114:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fb2dc003cb0 con 0x7fb2e8107290 2024-04-23T10:37:43.807 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:37:43.806+0000 7fb2e57fa640 1 -- 172.21.15.114:0/3760507729 <== mon.0 v2:172.21.15.114:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb2dc049060 con 0x7fb2e8107290 2024-04-23T10:37:43.807 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:37:43.806+0000 7fb2ee7a1640 1 -- 172.21.15.114:0/3760507729 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fb2e81a76a0 con 0x7fb2e8107290 2024-04-23T10:37:43.814 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:37:43.808+0000 7fb2e57fa640 1 -- 172.21.15.114:0/3760507729 <== mon.0 v2:172.21.15.114:3300/0 4 ==== mgrmap(e 26) ==== 138778+0+0 (secure 0 0 0) 0x7fb2dc042030 con 0x7fb2e8107290 2024-04-23T10:37:43.814 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:37:43.809+0000 7fb2f158b740 1 Processor -- start 2024-04-23T10:37:43.814 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:37:43.809+0000 7fb2f158b740 1 -- start start 2024-04-23T10:37:43.814 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:37:43.809+0000 7fb2f158b740 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x55970112e620 0x55970112ea00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T10:37:43.815 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:37:43.809+0000 7fb2f158b740 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55970112ef40 0x5597011392c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T10:37:43.815 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:37:43.809+0000 7fb2f158b740 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x559701139800 0x55970113bbf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T10:37:43.815 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:37:43.809+0000 7fb2f158b740 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x559701093970 con 0x55970112ef40 2024-04-23T10:37:43.815 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:37:43.809+0000 7fb2f158b740 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 -- 0x5597010304a0 con 0x55970112e620 2024-04-23T10:37:43.815 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:37:43.809+0000 7fb2f158b740 1 -- --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_getmap magic: 0 -- 0x55970108f930 con 0x559701139800 2024-04-23T10:37:43.815 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:37:43.809+0000 7fb2e7fff640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55970112ef40 0x5597011392c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T10:37:43.815 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:37:43.809+0000 7fb2e57fa640 1 --2- 172.21.15.114:0/3760507729 >> [v2:172.21.15.114:6800/1395693466,v1:172.21.15.114:6801/1395693466] conn(0x7fb2d00a3df0 0x7fb2d00a62b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T10:37:43.815 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:37:43.809+0000 7fb2e7fff640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55970112ef40 0x5597011392c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.114:60070/0 (socket says 172.21.15.114:60070) 2024-04-23T10:37:43.815 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:37:43.809+0000 7fb2e7fff640 1 -- 172.21.15.114:0/505181346 learned_addr learned my addr 172.21.15.114:0/505181346 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-04-23T10:37:43.815 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:37:43.809+0000 7fb2ecd3e640 1 --2- 172.21.15.114:0/505181346 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x55970112e620 0x55970112ea00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T10:37:43.815 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:37:43.809+0000 7fb2e77fe640 1 --2- 172.21.15.114:0/505181346 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x559701139800 0x55970113bbf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T10:37:43.815 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:37:43.809+0000 7fb2ecd3e640 1 --2- 172.21.15.114:0/3760507729 >> [v2:172.21.15.114:6800/1395693466,v1:172.21.15.114:6801/1395693466] conn(0x7fb2d00a3df0 0x7fb2d00a62b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T10:37:43.815 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:37:43.809+0000 7fb2e7fff640 1 -- 172.21.15.114:0/505181346 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x559701139800 msgr2=0x55970113bbf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T10:37:43.815 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:37:43.809+0000 7fb2e7fff640 1 --2- 172.21.15.114:0/505181346 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x559701139800 0x55970113bbf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:37:43.816 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:37:43.809+0000 7fb2e57fa640 1 -- 172.21.15.114:0/3760507729 <== mon.0 v2:172.21.15.114:3300/0 5 ==== osd_map(104..104 src has 1..104) ==== 8629+0+0 (secure 0 0 0) 0x7fb2dc0f1150 con 0x7fb2e8107290 2024-04-23T10:37:43.816 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:37:43.809+0000 7fb2e7fff640 1 -- 172.21.15.114:0/505181346 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x55970112e620 msgr2=0x55970112ea00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T10:37:43.816 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:37:43.809+0000 7fb2e7fff640 1 --2- 172.21.15.114:0/505181346 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x55970112e620 0x55970112ea00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:37:43.816 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:37:43.809+0000 7fb2e7fff640 1 -- 172.21.15.114:0/505181346 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x559700da6210 con 0x55970112ef40 2024-04-23T10:37:43.816 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:37:43.809+0000 7fb2ecd3e640 1 --2- 172.21.15.114:0/505181346 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x55970112e620 0x55970112ea00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-23T10:37:43.816 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:37:43.809+0000 7fb2e77fe640 1 --2- 172.21.15.114:0/505181346 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x559701139800 0x55970113bbf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-23T10:37:43.816 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:37:43.810+0000 7fb2e7fff640 1 --2- 172.21.15.114:0/505181346 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55970112ef40 0x5597011392c0 secure :-1 s=READY pgs=863 cs=0 l=1 rev1=1 crypto rx=0x7fb2d800cbf0 tx=0x7fb2d800e720 comp rx=0 tx=0).ready entity=mon.0 client_cookie=eae17c8ac7eca299 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T10:37:43.816 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:37:43.810+0000 7fb2ecd3e640 1 --2- 172.21.15.114:0/3760507729 >> [v2:172.21.15.114:6800/1395693466,v1:172.21.15.114:6801/1395693466] conn(0x7fb2d00a3df0 0x7fb2d00a62b0 secure :-1 s=READY pgs=134 cs=0 l=1 rev1=1 crypto rx=0x7fb2e0013540 tx=0x7fb2e0004ab0 comp rx=0 tx=0).ready entity=mgr.15291 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T10:37:43.816 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:37:43.810+0000 7fb2ce7fc640 1 -- 172.21.15.114:0/505181346 <== mon.0 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb2d8010630 con 0x55970112ef40 2024-04-23T10:37:43.816 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:37:43.810+0000 7fb2ce7fc640 1 -- 172.21.15.114:0/505181346 <== mon.0 v2:172.21.15.114:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fb2d8010c80 con 0x55970112ef40 2024-04-23T10:37:43.816 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:37:43.810+0000 7fb2ce7fc640 1 -- 172.21.15.114:0/505181346 <== mon.0 v2:172.21.15.114:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb2d8019b40 con 0x55970112ef40 2024-04-23T10:37:43.816 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:37:43.810+0000 7fb2f158b740 1 -- 172.21.15.114:0/505181346 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55970112ef40 msgr2=0x5597011392c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T10:37:43.816 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:37:43.810+0000 7fb2f158b740 1 --2- 172.21.15.114:0/505181346 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55970112ef40 0x5597011392c0 secure :-1 s=READY pgs=863 cs=0 l=1 rev1=1 crypto rx=0x7fb2d800cbf0 tx=0x7fb2d800e720 comp rx=0 tx=0).stop 2024-04-23T10:37:43.816 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:37:43.811+0000 7fb2f158b740 1 -- 172.21.15.114:0/505181346 shutdown_connections 2024-04-23T10:37:43.817 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:37:43.811+0000 7fb2f158b740 1 --2- 172.21.15.114:0/505181346 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x559701139800 0x55970113bbf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:37:43.817 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:37:43.811+0000 7fb2f158b740 1 --2- 172.21.15.114:0/505181346 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55970112ef40 0x5597011392c0 unknown :-1 s=CLOSED pgs=863 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:37:43.817 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:37:43.811+0000 7fb2f158b740 1 --2- 172.21.15.114:0/505181346 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x55970112e620 0x55970112ea00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:37:43.817 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:37:43.811+0000 7fb2f158b740 1 -- 172.21.15.114:0/505181346 >> 172.21.15.114:0/505181346 conn(0x55970112ab30 msgr2=0x55970112af10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T10:37:43.817 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:37:43.811+0000 7fb2f158b740 1 -- 172.21.15.114:0/505181346 shutdown_connections 2024-04-23T10:37:43.817 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:37:43.811+0000 7fb2f158b740 1 -- 172.21.15.114:0/505181346 wait complete. 2024-04-23T10:37:43.817 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:37:43.811+0000 7fb2f158b740 1 -- start start 2024-04-23T10:37:43.817 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:37:43.811+0000 7fb2f158b740 1 Processor -- start 2024-04-23T10:37:43.817 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:37:43.811+0000 7fb2f158b740 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55970112e620 0x5597011390b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T10:37:43.817 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:37:43.811+0000 7fb2f158b740 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x55970112ef40 0x5597011363a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T10:37:43.817 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:37:43.811+0000 7fb2f158b740 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x559701139800 0x5597011368e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T10:37:43.817 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:37:43.811+0000 7fb2f158b740 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x55970113c430 con 0x55970112e620 2024-04-23T10:37:43.817 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:37:43.811+0000 7fb2f158b740 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 -- 0x55970102e0b0 con 0x559701139800 2024-04-23T10:37:43.817 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:37:43.811+0000 7fb2f158b740 1 -- --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_getmap magic: 0 -- 0x55970113c5b0 con 0x55970112ef40 2024-04-23T10:37:43.818 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:37:43.811+0000 7fb2f158b740 10 client.0 _ll_register_callbacks cb 0x55970101bd30 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-23T10:37:43.818 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:37:43.812+0000 7fb2e7fff640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55970112e620 0x5597011390b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T10:37:43.818 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:37:43.812+0000 7fb2e7fff640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55970112e620 0x5597011390b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.114:60080/0 (socket says 172.21.15.114:60080) 2024-04-23T10:37:43.818 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:37:43.812+0000 7fb2e7fff640 1 -- 172.21.15.114:0/2804752047 learned_addr learned my addr 172.21.15.114:0/2804752047 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-04-23T10:37:43.818 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:37:43.812+0000 7fb2e77fe640 1 --2- 172.21.15.114:0/2804752047 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x55970112ef40 0x5597011363a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T10:37:43.818 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:37:43.812+0000 7fb2ecd3e640 1 --2- 172.21.15.114:0/2804752047 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x559701139800 0x5597011368e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T10:37:43.818 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:37:43.812+0000 7fb2e7fff640 1 -- 172.21.15.114:0/2804752047 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x55970112ef40 msgr2=0x5597011363a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T10:37:43.818 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:37:43.812+0000 7fb2e7fff640 1 --2- 172.21.15.114:0/2804752047 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x55970112ef40 0x5597011363a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:37:43.818 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:37:43.812+0000 7fb2e7fff640 1 -- 172.21.15.114:0/2804752047 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x559701139800 msgr2=0x5597011368e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T10:37:43.818 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:37:43.812+0000 7fb2e7fff640 1 --2- 172.21.15.114:0/2804752047 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x559701139800 0x5597011368e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:37:43.818 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:37:43.812+0000 7fb2e7fff640 1 -- 172.21.15.114:0/2804752047 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({osdmap=0}) -- 0x559700e72620 con 0x55970112e620 2024-04-23T10:37:43.818 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:37:43.812+0000 7fb2e77fe640 1 --2- 172.21.15.114:0/2804752047 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x55970112ef40 0x5597011363a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-23T10:37:43.818 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:37:43.812+0000 7fb2e7fff640 1 --2- 172.21.15.114:0/2804752047 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55970112e620 0x5597011390b0 secure :-1 s=READY pgs=864 cs=0 l=1 rev1=1 crypto rx=0x7fb2d800c960 tx=0x7fb2d80198c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=838f51e75e593443 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T10:37:43.818 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:37:43.812+0000 7fb2c77fe640 10 client.0 ms_handle_connect on v2:172.21.15.114:3300/0 2024-04-23T10:37:43.818 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:37:43.812+0000 7fb2c77fe640 1 -- 172.21.15.114:0/2804752047 <== mon.0 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb2d8019c70 con 0x55970112e620 2024-04-23T10:37:43.819 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:37:43.812+0000 7fb2c77fe640 1 -- 172.21.15.114:0/2804752047 <== mon.0 v2:172.21.15.114:3300/0 2 ==== osd_map(104..104 src has 1..104) ==== 8629+0+0 (secure 0 0 0) 0x7fb2d8019e10 con 0x55970112e620 2024-04-23T10:37:43.819 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:37:43.812+0000 7fb2f158b740 1 -- 172.21.15.114:0/2804752047 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fb2d8010c80 con 0x55970112e620 2024-04-23T10:37:43.819 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:37:43.812+0000 7fb2c77fe640 1 -- 172.21.15.114:0/2804752047 <== mon.0 v2:172.21.15.114:3300/0 3 ==== mon_get_version_reply(handle=1 version=38) ==== 24+0+0 (secure 0 0 0) 0x7fb2d8005050 con 0x55970112e620 2024-04-23T10:37:43.819 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:37:43.812+0000 7fb2f158b740 10 client.16854 fetch_fsmap learned FSMap version 38 2024-04-23T10:37:43.819 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:37:43.812+0000 7fb2f158b740 1 -- 172.21.15.114:0/2804752047 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({config=0+,fsmap=38,monmap=4+}) -- 0x559701148210 con 0x55970112e620 2024-04-23T10:37:43.819 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:37:43.813+0000 7fb2c77fe640 1 -- 172.21.15.114:0/2804752047 <== mon.0 v2:172.21.15.114:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fb2d8010c80 con 0x55970112e620 2024-04-23T10:37:43.819 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:37:43.813+0000 7fb2c77fe640 1 -- 172.21.15.114:0/2804752047 <== mon.0 v2:172.21.15.114:3300/0 5 ==== fsmap(e 38) ==== 8354+0+0 (secure 0 0 0) 0x7fb2d80046c0 con 0x55970112e620 2024-04-23T10:37:43.819 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:37:43.813+0000 7fb2f158b740 10 client.16854 fetch_fsmap finished waiting for FSMap version 38 2024-04-23T10:37:43.819 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:37:43.813+0000 7fb2f158b740 10 client.16854 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-04-23T10:37:43.819 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:37:43.813+0000 7fb2f158b740 20 client.16854 populate_metadata read hostname 'smithi114' 2024-04-23T10:37:43.819 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:37:43.813+0000 7fb2f158b740 1 --2- 172.21.15.114:0/2804752047 >> [v2:172.21.15.119:6838/937297340,v1:172.21.15.119:6839/937297340] conn(0x559701149610 0x55970114ba00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T10:37:43.819 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:37:43.813+0000 7fb2f158b740 4 client.16854 mds_command: new command op to 24545 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-23T10:37:43.819 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:37:43.813+0000 7fb2f158b740 1 -- 172.21.15.114:0/2804752047 --> [v2:172.21.15.119:6838/937297340,v1:172.21.15.119:6839/937297340] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x5597010a1390 con 0x559701149610 2024-04-23T10:37:43.819 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:37:43.813+0000 7fb2e77fe640 1 --2- 172.21.15.114:0/2804752047 >> [v2:172.21.15.119:6838/937297340,v1:172.21.15.119:6839/937297340] conn(0x559701149610 0x55970114ba00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T10:37:43.820 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:37:43.814+0000 7fb2c77fe640 10 client.16854 ms_handle_connect on v2:172.21.15.119:6838/937297340 2024-04-23T10:37:43.820 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:37:43.814+0000 7fb2e77fe640 1 --2- 172.21.15.114:0/2804752047 >> [v2:172.21.15.119:6838/937297340,v1:172.21.15.119:6839/937297340] conn(0x559701149610 0x55970114ba00 crc :-1 s=READY pgs=160 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=96b750a8df4631fe server_cookie=9baabb755d49bbaf in_seq=0 out_seq=0 2024-04-23T10:37:43.820 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:37:43.814+0000 7fb2c77fe640 1 -- 172.21.15.114:0/2804752047 <== mds.0 v2:172.21.15.119:6838/937297340 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x5597010a1390 con 0x559701149610 2024-04-23T10:37:43.820 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:37:43.814+0000 7fb2c77fe640 10 client.16854 handle_command_reply: tid=0 2024-04-23T10:37:43.820 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:37:43.814+0000 7fb2f158b740 1 client.16854 shutdown 2024-04-23T10:37:43.820 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:37:43.814+0000 7fb2f158b740 1 -- 172.21.15.114:0/2804752047 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55970112e620 msgr2=0x5597011390b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T10:37:43.820 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:37:43.814+0000 7fb2f158b740 1 --2- 172.21.15.114:0/2804752047 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55970112e620 0x5597011390b0 secure :-1 s=READY pgs=864 cs=0 l=1 rev1=1 crypto rx=0x7fb2d800c960 tx=0x7fb2d80198c0 comp rx=0 tx=0).stop 2024-04-23T10:37:43.820 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:37:43.814+0000 7fb2f158b740 1 -- 172.21.15.114:0/2804752047 shutdown_connections 2024-04-23T10:37:43.820 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:37:43.814+0000 7fb2f158b740 1 --2- 172.21.15.114:0/2804752047 >> [v2:172.21.15.119:6838/937297340,v1:172.21.15.119:6839/937297340] conn(0x559701149610 0x55970114ba00 crc :-1 s=READY pgs=160 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:37:43.820 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:37:43.815+0000 7fb2f158b740 1 --2- 172.21.15.114:0/2804752047 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x559701139800 0x5597011368e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:37:43.820 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:37:43.815+0000 7fb2c77fe640 0 client.16854 ms_handle_reset on v2:172.21.15.119:6838/937297340 2024-04-23T10:37:43.820 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:37:43.815+0000 7fb2f158b740 1 --2- 172.21.15.114:0/2804752047 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x55970112ef40 0x5597011363a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:37:43.820 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:37:43.815+0000 7fb2f158b740 1 --2- 172.21.15.114:0/2804752047 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55970112e620 0x5597011390b0 unknown :-1 s=CLOSED pgs=864 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:37:43.820 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:37:43.815+0000 7fb2f158b740 1 -- 172.21.15.114:0/2804752047 >> 172.21.15.114:0/2804752047 conn(0x55970112ab30 msgr2=0x5597011308b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T10:37:43.820 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:37:43.815+0000 7fb2f158b740 1 -- 172.21.15.114:0/2804752047 shutdown_connections 2024-04-23T10:37:43.821 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:37:43.815+0000 7fb2f158b740 1 -- 172.21.15.114:0/2804752047 wait complete. 2024-04-23T10:37:43.821 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:37:43.815+0000 7fb2f158b740 20 client.16854 trim_cache size 0 max 16384 2024-04-23T10:37:43.833 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:37:43.829+0000 7fb2f158b740 1 Processor -- start 2024-04-23T10:37:43.833 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:37:43.829+0000 7fb2f158b740 1 -- start start 2024-04-23T10:37:43.833 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:37:43.830+0000 7fb2f158b740 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x55970112e620 0x5597011452f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T10:37:43.833 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:37:43.830+0000 7fb2f158b740 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x55970112ef40 0x559701131af0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T10:37:43.833 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:37:43.830+0000 7fb2f158b740 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x559701139800 0x559701138be0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T10:37:43.833 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:37:43.830+0000 7fb2f158b740 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x7fb2e81b2850 con 0x559701139800 2024-04-23T10:37:43.834 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:37:43.830+0000 7fb2f158b740 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 -- 0x7fb2e81b0060 con 0x55970112ef40 2024-04-23T10:37:43.834 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:37:43.830+0000 7fb2f158b740 1 -- --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_getmap magic: 0 -- 0x7fb2e81aded0 con 0x55970112e620 2024-04-23T10:37:43.834 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:37:43.830+0000 7fb2ecd3e640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x559701139800 0x559701138be0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T10:37:43.834 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:37:43.830+0000 7fb2ecd3e640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x559701139800 0x559701138be0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.114:60088/0 (socket says 172.21.15.114:60088) 2024-04-23T10:37:43.834 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:37:43.830+0000 7fb2ecd3e640 1 -- 172.21.15.114:0/1276809501 learned_addr learned my addr 172.21.15.114:0/1276809501 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-04-23T10:37:43.834 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:37:43.830+0000 7fb2e77fe640 1 --2- 172.21.15.114:0/1276809501 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x55970112ef40 0x559701131af0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T10:37:43.834 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:37:43.830+0000 7fb2e7fff640 1 --2- 172.21.15.114:0/1276809501 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x55970112e620 0x5597011452f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T10:37:43.834 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:37:43.830+0000 7fb2ecd3e640 1 -- 172.21.15.114:0/1276809501 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x55970112e620 msgr2=0x5597011452f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T10:37:43.834 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:37:43.830+0000 7fb2ecd3e640 1 --2- 172.21.15.114:0/1276809501 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x55970112e620 0x5597011452f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:37:43.834 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:37:43.830+0000 7fb2ecd3e640 1 -- 172.21.15.114:0/1276809501 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x55970112ef40 msgr2=0x559701131af0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T10:37:43.834 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:37:43.830+0000 7fb2ecd3e640 1 --2- 172.21.15.114:0/1276809501 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x55970112ef40 0x559701131af0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:37:43.834 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:37:43.830+0000 7fb2ecd3e640 1 -- 172.21.15.114:0/1276809501 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5597011348f0 con 0x559701139800 2024-04-23T10:37:43.834 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:37:43.830+0000 7fb2ecd3e640 1 --2- 172.21.15.114:0/1276809501 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x559701139800 0x559701138be0 secure :-1 s=READY pgs=865 cs=0 l=1 rev1=1 crypto rx=0x7fb2e000e7d0 tx=0x7fb2e0009000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5ea70bd3d7584ef8 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T10:37:43.835 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:37:43.830+0000 7fb2cdffb640 1 -- 172.21.15.114:0/1276809501 <== mon.0 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb2e0017040 con 0x559701139800 2024-04-23T10:37:43.835 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:37:43.830+0000 7fb2cdffb640 1 -- 172.21.15.114:0/1276809501 <== mon.0 v2:172.21.15.114:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fb2e000ee80 con 0x559701139800 2024-04-23T10:37:43.835 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:37:43.830+0000 7fb2cdffb640 1 -- 172.21.15.114:0/1276809501 <== mon.0 v2:172.21.15.114:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb2e00206b0 con 0x559701139800 2024-04-23T10:37:43.835 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:37:43.831+0000 7fb2f158b740 1 -- 172.21.15.114:0/1276809501 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x559701139800 msgr2=0x559701138be0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T10:37:43.835 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:37:43.831+0000 7fb2f158b740 1 --2- 172.21.15.114:0/1276809501 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x559701139800 0x559701138be0 secure :-1 s=READY pgs=865 cs=0 l=1 rev1=1 crypto rx=0x7fb2e000e7d0 tx=0x7fb2e0009000 comp rx=0 tx=0).stop 2024-04-23T10:37:43.835 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:37:43.831+0000 7fb2f158b740 1 -- 172.21.15.114:0/1276809501 shutdown_connections 2024-04-23T10:37:43.835 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:37:43.831+0000 7fb2f158b740 1 --2- 172.21.15.114:0/1276809501 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x559701139800 0x559701138be0 unknown :-1 s=CLOSED pgs=865 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:37:43.835 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:37:43.831+0000 7fb2f158b740 1 --2- 172.21.15.114:0/1276809501 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x55970112ef40 0x559701131af0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:37:43.835 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:37:43.831+0000 7fb2f158b740 1 --2- 172.21.15.114:0/1276809501 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x55970112e620 0x5597011452f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:37:43.835 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:37:43.831+0000 7fb2f158b740 1 -- 172.21.15.114:0/1276809501 >> 172.21.15.114:0/1276809501 conn(0x55970112ab30 msgr2=0x559701129420 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T10:37:43.835 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:37:43.831+0000 7fb2f158b740 1 -- 172.21.15.114:0/1276809501 shutdown_connections 2024-04-23T10:37:43.835 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:37:43.831+0000 7fb2f158b740 1 -- 172.21.15.114:0/1276809501 wait complete. 2024-04-23T10:37:43.835 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:37:43.831+0000 7fb2f158b740 1 -- start start 2024-04-23T10:37:43.835 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:37:43.832+0000 7fb2f158b740 1 Processor -- start 2024-04-23T10:37:43.835 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:37:43.832+0000 7fb2f158b740 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x55970112e620 0x55970113c7c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T10:37:43.836 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:37:43.832+0000 7fb2f158b740 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55970112ef40 0x55970113cd00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T10:37:43.836 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:37:43.832+0000 7fb2f158b740 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x559701139800 0x5597011ada30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T10:37:43.836 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:37:43.832+0000 7fb2f158b740 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x7fb2e81ae520 con 0x55970112ef40 2024-04-23T10:37:43.836 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:37:43.832+0000 7fb2f158b740 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 -- 0x7fb2e81af990 con 0x559701139800 2024-04-23T10:37:43.836 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:37:43.832+0000 7fb2f158b740 1 -- --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_getmap magic: 0 -- 0x7fb2e81aebf0 con 0x55970112e620 2024-04-23T10:37:43.836 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:37:43.832+0000 7fb2f158b740 10 client.0 _ll_register_callbacks cb 0x7fb2d8010e30 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-23T10:37:43.836 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:37:43.832+0000 7fb2e77fe640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55970112ef40 0x55970113cd00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T10:37:43.836 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:37:43.832+0000 7fb2e77fe640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55970112ef40 0x55970113cd00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.114:60096/0 (socket says 172.21.15.114:60096) 2024-04-23T10:37:43.836 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:37:43.832+0000 7fb2e77fe640 1 -- 172.21.15.114:0/1875985785 learned_addr learned my addr 172.21.15.114:0/1875985785 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-04-23T10:37:43.836 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:37:43.832+0000 7fb2e7fff640 1 --2- 172.21.15.114:0/1875985785 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x55970112e620 0x55970113c7c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T10:37:43.836 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:37:43.832+0000 7fb2e77fe640 1 -- 172.21.15.114:0/1875985785 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x55970112e620 msgr2=0x55970113c7c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T10:37:43.836 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:37:43.832+0000 7fb2e77fe640 1 --2- 172.21.15.114:0/1875985785 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x55970112e620 0x55970113c7c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:37:43.836 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:37:43.832+0000 7fb2e77fe640 1 -- 172.21.15.114:0/1875985785 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x559701139800 msgr2=0x5597011ada30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T10:37:43.836 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:37:43.832+0000 7fb2e77fe640 1 --2- 172.21.15.114:0/1875985785 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x559701139800 0x5597011ada30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:37:43.837 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:37:43.832+0000 7fb2e77fe640 1 -- 172.21.15.114:0/1875985785 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5597011adf70 con 0x55970112ef40 2024-04-23T10:37:43.837 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:37:43.832+0000 7fb2e7fff640 1 --2- 172.21.15.114:0/1875985785 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x55970112e620 0x55970113c7c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-23T10:37:43.837 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:37:43.832+0000 7fb2e77fe640 1 --2- 172.21.15.114:0/1875985785 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55970112ef40 0x55970113cd00 secure :-1 s=READY pgs=866 cs=0 l=1 rev1=1 crypto rx=0x7fb2dc0bd060 tx=0x7fb2dc03b720 comp rx=0 tx=0).ready entity=mon.0 client_cookie=dd0accd46bf36626 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T10:37:43.837 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:37:43.832+0000 7fb2c77fe640 10 client.0 ms_handle_connect on v2:172.21.15.114:3300/0 2024-04-23T10:37:43.837 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:37:43.832+0000 7fb2c77fe640 1 -- 172.21.15.114:0/1875985785 <== mon.0 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb2dc035040 con 0x55970112ef40 2024-04-23T10:37:43.837 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:37:43.832+0000 7fb2c77fe640 1 -- 172.21.15.114:0/1875985785 <== mon.0 v2:172.21.15.114:3300/0 2 ==== osd_map(104..104 src has 1..104) ==== 8629+0+0 (secure 0 0 0) 0x7fb2dc0eeb50 con 0x55970112ef40 2024-04-23T10:37:43.837 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:37:43.832+0000 7fb2f158b740 1 -- 172.21.15.114:0/1875985785 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x55970109b200 con 0x55970112ef40 2024-04-23T10:37:43.837 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:37:43.833+0000 7fb2c77fe640 1 -- 172.21.15.114:0/1875985785 <== mon.0 v2:172.21.15.114:3300/0 3 ==== mon_get_version_reply(handle=1 version=38) ==== 24+0+0 (secure 0 0 0) 0x7fb2dc0c14d0 con 0x55970112ef40 2024-04-23T10:37:43.837 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:37:43.833+0000 7fb2f158b740 10 client.16860 fetch_fsmap learned FSMap version 38 2024-04-23T10:37:43.837 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:37:43.833+0000 7fb2f158b740 1 -- 172.21.15.114:0/1875985785 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({config=0+,fsmap=38,monmap=4+}) -- 0x5597011af0f0 con 0x55970112ef40 2024-04-23T10:37:43.837 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:37:43.833+0000 7fb2c77fe640 1 -- 172.21.15.114:0/1875985785 <== mon.0 v2:172.21.15.114:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x55970109b200 con 0x55970112ef40 2024-04-23T10:37:43.837 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:37:43.833+0000 7fb2c77fe640 1 -- 172.21.15.114:0/1875985785 <== mon.0 v2:172.21.15.114:3300/0 5 ==== fsmap(e 38) ==== 8354+0+0 (secure 0 0 0) 0x7fb2dc086020 con 0x55970112ef40 2024-04-23T10:37:43.837 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:37:43.833+0000 7fb2f158b740 10 client.16860 fetch_fsmap finished waiting for FSMap version 38 2024-04-23T10:37:43.837 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:37:43.833+0000 7fb2f158b740 10 client.16860 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-04-23T10:37:43.837 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:37:43.833+0000 7fb2f158b740 20 client.16860 populate_metadata read hostname 'smithi114' 2024-04-23T10:37:43.838 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:37:43.833+0000 7fb2f158b740 1 --2- 172.21.15.114:0/1875985785 >> [v2:172.21.15.119:6838/937297340,v1:172.21.15.119:6839/937297340] conn(0x5597011af8c0 0x5597011b1cb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T10:37:43.838 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:37:43.833+0000 7fb2f158b740 4 client.16860 mds_command: new command op to 24545 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-04-23T10:37:43.838 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:37:43.833+0000 7fb2f158b740 1 -- 172.21.15.114:0/1875985785 --> [v2:172.21.15.119:6838/937297340,v1:172.21.15.119:6839/937297340] -- command(tid 0: {"prefix": "flush journal"}) -- 0x5597010175f0 con 0x5597011af8c0 2024-04-23T10:37:43.838 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:37:43.833+0000 7fb2e7fff640 1 --2- 172.21.15.114:0/1875985785 >> [v2:172.21.15.119:6838/937297340,v1:172.21.15.119:6839/937297340] conn(0x5597011af8c0 0x5597011b1cb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T10:37:43.838 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:37:43.834+0000 7fb2c77fe640 10 client.16860 ms_handle_connect on v2:172.21.15.119:6838/937297340 2024-04-23T10:37:43.838 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:37:43.834+0000 7fb2e7fff640 1 --2- 172.21.15.114:0/1875985785 >> [v2:172.21.15.119:6838/937297340,v1:172.21.15.119:6839/937297340] conn(0x5597011af8c0 0x5597011b1cb0 crc :-1 s=READY pgs=161 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=318fc3c464bcda70 server_cookie=42cca277864b6829 in_seq=0 out_seq=0 2024-04-23T10:37:43.866 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:37:43.867+0000 7fb2c77fe640 1 -- 172.21.15.114:0/1875985785 <== mds.0 v2:172.21.15.119:6838/937297340 1 ==== command_reply(tid 0: 0 ) ==== 8+0+44 (crc 0 0 0) 0x5597010175f0 con 0x5597011af8c0 2024-04-23T10:37:43.867 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:37:43.867+0000 7fb2c77fe640 10 client.16860 handle_command_reply: tid=0 2024-04-23T10:37:43.867 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:37:43.867+0000 7fb2f158b740 1 client.16860 shutdown 2024-04-23T10:37:43.867 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:37:43.867+0000 7fb2f158b740 1 -- 172.21.15.114:0/1875985785 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55970112ef40 msgr2=0x55970113cd00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T10:37:43.867 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:37:43.867+0000 7fb2f158b740 1 --2- 172.21.15.114:0/1875985785 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55970112ef40 0x55970113cd00 secure :-1 s=READY pgs=866 cs=0 l=1 rev1=1 crypto rx=0x7fb2dc0bd060 tx=0x7fb2dc03b720 comp rx=0 tx=0).stop 2024-04-23T10:37:43.867 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:37:43.867+0000 7fb2f158b740 1 -- 172.21.15.114:0/1875985785 shutdown_connections 2024-04-23T10:37:43.867 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:37:43.867+0000 7fb2f158b740 1 --2- 172.21.15.114:0/1875985785 >> [v2:172.21.15.119:6838/937297340,v1:172.21.15.119:6839/937297340] conn(0x5597011af8c0 0x5597011b1cb0 crc :-1 s=READY pgs=161 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:37:43.867 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:37:43.867+0000 7fb2f158b740 1 --2- 172.21.15.114:0/1875985785 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x559701139800 0x5597011ada30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:37:43.867 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:37:43.867+0000 7fb2f158b740 1 --2- 172.21.15.114:0/1875985785 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55970112ef40 0x55970113cd00 unknown :-1 s=CLOSED pgs=866 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:37:43.867 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:37:43.867+0000 7fb2c77fe640 0 client.16860 ms_handle_reset on v2:172.21.15.119:6838/937297340 2024-04-23T10:37:43.867 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:37:43.867+0000 7fb2f158b740 1 --2- 172.21.15.114:0/1875985785 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x55970112e620 0x55970113c7c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:37:43.867 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:37:43.867+0000 7fb2f158b740 1 -- 172.21.15.114:0/1875985785 >> 172.21.15.114:0/1875985785 conn(0x55970112ab30 msgr2=0x5597011358e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T10:37:43.867 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:37:43.867+0000 7fb2f158b740 1 -- 172.21.15.114:0/1875985785 shutdown_connections 2024-04-23T10:37:43.867 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:37:43.867+0000 7fb2f158b740 1 -- 172.21.15.114:0/1875985785 wait complete. 2024-04-23T10:37:43.868 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:37:43.867+0000 7fb2f158b740 20 client.16860 trim_cache size 0 max 16384 2024-04-23T10:37:43.868 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stdout:{ 2024-04-23T10:37:43.868 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stdout: "message": "", 2024-04-23T10:37:43.868 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stdout: "return_code": 0 2024-04-23T10:37:43.868 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stdout:} 2024-04-23T10:37:43.869 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:37:43.868+0000 7fb2ee7a1640 1 -- 172.21.15.114:0/3760507729 >> [v2:172.21.15.114:6800/1395693466,v1:172.21.15.114:6801/1395693466] conn(0x7fb2d00a3df0 msgr2=0x7fb2d00a62b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T10:37:43.869 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:37:43.868+0000 7fb2ee7a1640 1 --2- 172.21.15.114:0/3760507729 >> [v2:172.21.15.114:6800/1395693466,v1:172.21.15.114:6801/1395693466] conn(0x7fb2d00a3df0 0x7fb2d00a62b0 secure :-1 s=READY pgs=134 cs=0 l=1 rev1=1 crypto rx=0x7fb2e0013540 tx=0x7fb2e0004ab0 comp rx=0 tx=0).stop 2024-04-23T10:37:43.869 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:37:43.868+0000 7fb2ee7a1640 1 -- 172.21.15.114:0/3760507729 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fb2e8107290 msgr2=0x7fb2e8074790 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T10:37:43.869 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:37:43.868+0000 7fb2ee7a1640 1 --2- 172.21.15.114:0/3760507729 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fb2e8107290 0x7fb2e8074790 secure :-1 s=READY pgs=862 cs=0 l=1 rev1=1 crypto rx=0x7fb2dc031a90 tx=0x7fb2dc004270 comp rx=0 tx=0).stop 2024-04-23T10:37:43.869 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:37:43.868+0000 7fb2ee7a1640 1 -- 172.21.15.114:0/3760507729 shutdown_connections 2024-04-23T10:37:43.869 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:37:43.868+0000 7fb2ee7a1640 1 --2- 172.21.15.114:0/3760507729 >> [v2:172.21.15.114:6800/1395693466,v1:172.21.15.114:6801/1395693466] conn(0x7fb2d00a3df0 0x7fb2d00a62b0 unknown :-1 s=CLOSED pgs=134 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:37:43.870 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:37:43.868+0000 7fb2ee7a1640 1 --2- 172.21.15.114:0/3760507729 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fb2e8074cd0 0x7fb2e81a6820 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:37:43.870 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:37:43.868+0000 7fb2ee7a1640 1 --2- 172.21.15.114:0/3760507729 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fb2e8107290 0x7fb2e8074790 unknown :-1 s=CLOSED pgs=862 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:37:43.870 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:37:43.868+0000 7fb2ee7a1640 1 --2- 172.21.15.114:0/3760507729 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fb2e81021c0 0x7fb2e8074250 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:37:43.870 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:37:43.868+0000 7fb2ee7a1640 1 -- 172.21.15.114:0/3760507729 >> 172.21.15.114:0/3760507729 conn(0x7fb2e80fda20 msgr2=0x7fb2e80ffce0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T10:37:43.870 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:37:43.868+0000 7fb2ee7a1640 1 -- 172.21.15.114:0/3760507729 shutdown_connections 2024-04-23T10:37:43.870 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:37:43.868+0000 7fb2ee7a1640 1 -- 172.21.15.114:0/3760507729 wait complete. 2024-04-23T10:37:43.943 INFO:tasks.workunit.client.0.smithi114.stdout: 1 674902 5.71 MB/sec execute 472 sec latency 4636.738 ms 2024-04-23T10:37:44.503 INFO:journalctl@ceph.mon.a.smithi114.stdout:Apr 23 10:37:44 smithi114 ceph-mon[29255]: pgmap v348: 97 pgs: 97 active+clean; 208 MiB data, 2.6 GiB used, 1.0 TiB / 1.0 TiB avail; 366 KiB/s rd, 686 KiB/s wr, 78 op/s 2024-04-23T10:37:44.577 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 23 10:37:44 smithi204 ceph-mon[32649]: pgmap v348: 97 pgs: 97 active+clean; 208 MiB data, 2.6 GiB used, 1.0 TiB / 1.0 TiB avail; 366 KiB/s rd, 686 KiB/s wr, 78 op/s 2024-04-23T10:37:44.741 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 23 10:37:44 smithi119 ceph-mon[32608]: pgmap v348: 97 pgs: 97 active+clean; 208 MiB data, 2.6 GiB used, 1.0 TiB / 1.0 TiB avail; 366 KiB/s rd, 686 KiB/s wr, 78 op/s 2024-04-23T10:37:44.943 INFO:tasks.workunit.client.0.smithi114.stdout: 1 675640 5.70 MB/sec execute 473 sec latency 4878.223 ms 2024-04-23T10:37:45.943 INFO:tasks.workunit.client.0.smithi114.stdout: 1 677961 5.72 MB/sec execute 474 sec latency 18.270 ms 2024-04-23T10:37:46.504 INFO:journalctl@ceph.mon.a.smithi114.stdout:Apr 23 10:37:46 smithi114 ceph-mon[29255]: pgmap v349: 97 pgs: 97 active+clean; 208 MiB data, 2.6 GiB used, 1.0 TiB / 1.0 TiB avail; 288 KiB/s rd, 591 KiB/s wr, 52 op/s 2024-04-23T10:37:46.578 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 23 10:37:46 smithi204 ceph-mon[32649]: pgmap v349: 97 pgs: 97 active+clean; 208 MiB data, 2.6 GiB used, 1.0 TiB / 1.0 TiB avail; 288 KiB/s rd, 591 KiB/s wr, 52 op/s 2024-04-23T10:37:46.742 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 23 10:37:46 smithi119 ceph-mon[32608]: pgmap v349: 97 pgs: 97 active+clean; 208 MiB data, 2.6 GiB used, 1.0 TiB / 1.0 TiB avail; 288 KiB/s rd, 591 KiB/s wr, 52 op/s 2024-04-23T10:37:46.943 INFO:tasks.workunit.client.0.smithi114.stdout: 1 678381 5.72 MB/sec execute 475 sec latency 822.766 ms 2024-04-23T10:37:47.944 INFO:tasks.workunit.client.0.smithi114.stdout: 1 678381 5.70 MB/sec execute 476 sec latency 1822.893 ms 2024-04-23T10:37:48.504 INFO:journalctl@ceph.mon.a.smithi114.stdout:Apr 23 10:37:48 smithi114 ceph-mon[29255]: pgmap v350: 97 pgs: 97 active+clean; 211 MiB data, 2.6 GiB used, 1.0 TiB / 1.0 TiB avail; 538 KiB/s rd, 1.0 MiB/s wr, 101 op/s 2024-04-23T10:37:48.578 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 23 10:37:48 smithi204 ceph-mon[32649]: pgmap v350: 97 pgs: 97 active+clean; 211 MiB data, 2.6 GiB used, 1.0 TiB / 1.0 TiB avail; 538 KiB/s rd, 1.0 MiB/s wr, 101 op/s 2024-04-23T10:37:48.742 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 23 10:37:48 smithi119 ceph-mon[32608]: pgmap v350: 97 pgs: 97 active+clean; 211 MiB data, 2.6 GiB used, 1.0 TiB / 1.0 TiB avail; 538 KiB/s rd, 1.0 MiB/s wr, 101 op/s 2024-04-23T10:37:48.944 INFO:tasks.workunit.client.0.smithi114.stdout: 1 678381 5.69 MB/sec execute 477 sec latency 2822.994 ms 2024-04-23T10:37:49.944 INFO:tasks.workunit.client.0.smithi114.stdout: 1 678448 5.68 MB/sec execute 478 sec latency 3066.129 ms 2024-04-23T10:37:50.504 INFO:journalctl@ceph.mon.a.smithi114.stdout:Apr 23 10:37:50 smithi114 ceph-mon[29255]: pgmap v351: 97 pgs: 97 active+clean; 211 MiB data, 2.6 GiB used, 1.0 TiB / 1.0 TiB avail; 314 KiB/s rd, 702 KiB/s wr, 64 op/s 2024-04-23T10:37:50.578 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 23 10:37:50 smithi204 ceph-mon[32649]: pgmap v351: 97 pgs: 97 active+clean; 211 MiB data, 2.6 GiB used, 1.0 TiB / 1.0 TiB avail; 314 KiB/s rd, 702 KiB/s wr, 64 op/s 2024-04-23T10:37:50.742 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 23 10:37:50 smithi119 ceph-mon[32608]: pgmap v351: 97 pgs: 97 active+clean; 211 MiB data, 2.6 GiB used, 1.0 TiB / 1.0 TiB avail; 314 KiB/s rd, 702 KiB/s wr, 64 op/s 2024-04-23T10:37:50.944 INFO:tasks.workunit.client.0.smithi114.stdout: 1 678448 5.67 MB/sec execute 479 sec latency 1656.141 ms 2024-04-23T10:37:51.944 INFO:tasks.workunit.client.0.smithi114.stdout: 1 678448 5.66 MB/sec execute 480 sec latency 2656.249 ms 2024-04-23T10:37:52.578 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 23 10:37:52 smithi204 ceph-mon[32649]: pgmap v352: 97 pgs: 97 active+clean; 211 MiB data, 2.6 GiB used, 1.0 TiB / 1.0 TiB avail; 314 KiB/s rd, 702 KiB/s wr, 64 op/s 2024-04-23T10:37:52.742 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 23 10:37:52 smithi119 ceph-mon[32608]: pgmap v352: 97 pgs: 97 active+clean; 211 MiB data, 2.6 GiB used, 1.0 TiB / 1.0 TiB avail; 314 KiB/s rd, 702 KiB/s wr, 64 op/s 2024-04-23T10:37:52.754 INFO:journalctl@ceph.mon.a.smithi114.stdout:Apr 23 10:37:52 smithi114 ceph-mon[29255]: pgmap v352: 97 pgs: 97 active+clean; 211 MiB data, 2.6 GiB used, 1.0 TiB / 1.0 TiB avail; 314 KiB/s rd, 702 KiB/s wr, 64 op/s 2024-04-23T10:37:52.943 INFO:tasks.workunit.client.0.smithi114.stdout: 1 678448 5.65 MB/sec execute 481 sec latency 3656.319 ms 2024-04-23T10:37:53.944 INFO:tasks.workunit.client.0.smithi114.stdout: 1 678448 5.63 MB/sec execute 482 sec latency 4656.425 ms 2024-04-23T10:37:54.578 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 23 10:37:54 smithi204 ceph-mon[32649]: pgmap v353: 97 pgs: 97 active+clean; 214 MiB data, 2.6 GiB used, 1.0 TiB / 1.0 TiB avail; 362 KiB/s rd, 957 KiB/s wr, 77 op/s 2024-04-23T10:37:54.741 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 23 10:37:54 smithi119 ceph-mon[32608]: pgmap v353: 97 pgs: 97 active+clean; 214 MiB data, 2.6 GiB used, 1.0 TiB / 1.0 TiB avail; 362 KiB/s rd, 957 KiB/s wr, 77 op/s 2024-04-23T10:37:54.753 INFO:journalctl@ceph.mon.a.smithi114.stdout:Apr 23 10:37:54 smithi114 ceph-mon[29255]: pgmap v353: 97 pgs: 97 active+clean; 214 MiB data, 2.6 GiB used, 1.0 TiB / 1.0 TiB avail; 362 KiB/s rd, 957 KiB/s wr, 77 op/s 2024-04-23T10:37:54.944 INFO:tasks.workunit.client.0.smithi114.stdout: 1 679212 5.63 MB/sec execute 483 sec latency 4900.140 ms 2024-04-23T10:37:55.944 INFO:tasks.workunit.client.0.smithi114.stdout: 1 681671 5.64 MB/sec execute 484 sec latency 14.051 ms 2024-04-23T10:37:56.577 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 23 10:37:56 smithi204 ceph-mon[32649]: pgmap v354: 97 pgs: 97 active+clean; 214 MiB data, 2.6 GiB used, 1.0 TiB / 1.0 TiB avail; 298 KiB/s rd, 694 KiB/s wr, 62 op/s 2024-04-23T10:37:56.578 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 23 10:37:56 smithi204 ceph-mon[32649]: from='mgr.15291 172.21.15.114:0/1095789187' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-23T10:37:56.742 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 23 10:37:56 smithi119 ceph-mon[32608]: pgmap v354: 97 pgs: 97 active+clean; 214 MiB data, 2.6 GiB used, 1.0 TiB / 1.0 TiB avail; 298 KiB/s rd, 694 KiB/s wr, 62 op/s 2024-04-23T10:37:56.742 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 23 10:37:56 smithi119 ceph-mon[32608]: from='mgr.15291 172.21.15.114:0/1095789187' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-23T10:37:56.754 INFO:journalctl@ceph.mon.a.smithi114.stdout:Apr 23 10:37:56 smithi114 ceph-mon[29255]: pgmap v354: 97 pgs: 97 active+clean; 214 MiB data, 2.6 GiB used, 1.0 TiB / 1.0 TiB avail; 298 KiB/s rd, 694 KiB/s wr, 62 op/s 2024-04-23T10:37:56.754 INFO:journalctl@ceph.mon.a.smithi114.stdout:Apr 23 10:37:56 smithi114 ceph-mon[29255]: from='mgr.15291 172.21.15.114:0/1095789187' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-23T10:37:56.944 INFO:tasks.workunit.client.0.smithi114.stdout: 1 681926 5.64 MB/sec execute 485 sec latency 894.784 ms 2024-04-23T10:37:56.955 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:37:56.956+0000 7f7318329640 1 -- 172.21.15.114:0/1307020946 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f73101010c0 msgr2=0x7f73101014a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T10:37:56.955 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:37:56.956+0000 7f7318329640 1 --2- 172.21.15.114:0/1307020946 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f73101010c0 0x7f73101014a0 secure :-1 s=READY pgs=867 cs=0 l=1 rev1=1 crypto rx=0x7f7300009900 tx=0x7f730002a410 comp rx=0 tx=0).stop 2024-04-23T10:37:56.955 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:37:56.956+0000 7f7318329640 1 -- 172.21.15.114:0/1307020946 shutdown_connections 2024-04-23T10:37:56.955 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:37:56.956+0000 7f7318329640 1 --2- 172.21.15.114:0/1307020946 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f7310106190 0x7f731010dd60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:37:56.955 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:37:56.956+0000 7f7318329640 1 --2- 172.21.15.114:0/1307020946 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f7310101a70 0x7f7310105ab0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:37:56.955 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:37:56.956+0000 7f7318329640 1 --2- 172.21.15.114:0/1307020946 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f73101010c0 0x7f73101014a0 unknown :-1 s=CLOSED pgs=867 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:37:56.955 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:37:56.956+0000 7f7318329640 1 -- 172.21.15.114:0/1307020946 >> 172.21.15.114:0/1307020946 conn(0x7f7310077fa0 msgr2=0x7f73100783b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T10:37:56.955 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:37:56.956+0000 7f7318329640 1 -- 172.21.15.114:0/1307020946 shutdown_connections 2024-04-23T10:37:56.955 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:37:56.956+0000 7f7318329640 1 -- 172.21.15.114:0/1307020946 wait complete. 2024-04-23T10:37:56.958 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:37:56.956+0000 7f7318329640 1 Processor -- start 2024-04-23T10:37:56.958 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:37:56.957+0000 7f7318329640 1 -- start start 2024-04-23T10:37:56.958 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:37:56.957+0000 7f7318329640 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f73101010c0 0x7f73101986d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T10:37:56.958 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:37:56.957+0000 7f7318329640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f7310106190 0x7f7310198c10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T10:37:56.958 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:37:56.957+0000 7f7318329640 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f731019cf90 0x7f731019d440 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T10:37:56.958 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:37:56.957+0000 7f7318329640 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x7f73101110f0 con 0x7f7310106190 2024-04-23T10:37:56.959 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:37:56.957+0000 7f7318329640 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 -- 0x7f7310110f70 con 0x7f731019cf90 2024-04-23T10:37:56.959 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:37:56.957+0000 7f7318329640 1 -- --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_getmap magic: 0 -- 0x7f7310111270 con 0x7f73101010c0 2024-04-23T10:37:56.959 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:37:56.957+0000 7f73158c4640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f7310106190 0x7f7310198c10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T10:37:56.959 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:37:56.957+0000 7f73158c4640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f7310106190 0x7f7310198c10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.114:36476/0 (socket says 172.21.15.114:36476) 2024-04-23T10:37:56.959 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:37:56.957+0000 7f73158c4640 1 -- 172.21.15.114:0/3091329814 learned_addr learned my addr 172.21.15.114:0/3091329814 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-04-23T10:37:56.959 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:37:56.957+0000 7f73168c6640 1 --2- 172.21.15.114:0/3091329814 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f731019cf90 0x7f731019d440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T10:37:56.959 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:37:56.957+0000 7f73160c5640 1 --2- 172.21.15.114:0/3091329814 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f73101010c0 0x7f73101986d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T10:37:56.959 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:37:56.957+0000 7f73158c4640 1 -- 172.21.15.114:0/3091329814 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f73101010c0 msgr2=0x7f73101986d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T10:37:56.959 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:37:56.957+0000 7f73158c4640 1 --2- 172.21.15.114:0/3091329814 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f73101010c0 0x7f73101986d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:37:56.959 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:37:56.957+0000 7f73158c4640 1 -- 172.21.15.114:0/3091329814 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f731019cf90 msgr2=0x7f731019d440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T10:37:56.959 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:37:56.957+0000 7f73158c4640 1 --2- 172.21.15.114:0/3091329814 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f731019cf90 0x7f731019d440 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:37:56.959 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:37:56.957+0000 7f73158c4640 1 -- 172.21.15.114:0/3091329814 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f731019da80 con 0x7f7310106190 2024-04-23T10:37:56.959 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:37:56.957+0000 7f73158c4640 1 --2- 172.21.15.114:0/3091329814 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f7310106190 0x7f7310198c10 secure :-1 s=READY pgs=868 cs=0 l=1 rev1=1 crypto rx=0x7f730c00d740 tx=0x7f730c00dc00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T10:37:56.960 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:37:56.958+0000 7f73077fe640 1 -- 172.21.15.114:0/3091329814 <== mon.0 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f730c017070 con 0x7f7310106190 2024-04-23T10:37:56.960 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:37:56.958+0000 7f73077fe640 1 -- 172.21.15.114:0/3091329814 <== mon.0 v2:172.21.15.114:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f730c00de50 con 0x7f7310106190 2024-04-23T10:37:56.960 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:37:56.958+0000 7f7318329640 1 -- 172.21.15.114:0/3091329814 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f731019dd70 con 0x7f7310106190 2024-04-23T10:37:56.960 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:37:56.958+0000 7f73077fe640 1 -- 172.21.15.114:0/3091329814 <== mon.0 v2:172.21.15.114:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f730c004ce0 con 0x7f7310106190 2024-04-23T10:37:56.960 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:37:56.958+0000 7f7318329640 1 -- 172.21.15.114:0/3091329814 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f73101a5510 con 0x7f7310106190 2024-04-23T10:37:56.967 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:37:56.960+0000 7f731b043740 1 Processor -- start 2024-04-23T10:37:56.967 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:37:56.960+0000 7f731b043740 1 -- start start 2024-04-23T10:37:56.967 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:37:56.961+0000 7f731b043740 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x5615e3a12620 0x5615e3a12a00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T10:37:56.967 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:37:56.961+0000 7f731b043740 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x5615e3a12f40 0x5615e3a1d2c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T10:37:56.968 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:37:56.961+0000 7f731b043740 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x5615e3a1d800 0x5615e3a1fbf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T10:37:56.968 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:37:56.961+0000 7f731b043740 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x5615e3977970 con 0x5615e3a1d800 2024-04-23T10:37:56.968 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:37:56.961+0000 7f731b043740 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 -- 0x5615e39144a0 con 0x5615e3a12f40 2024-04-23T10:37:56.968 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:37:56.961+0000 7f731b043740 1 -- --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_getmap magic: 0 -- 0x5615e3973930 con 0x5615e3a12620 2024-04-23T10:37:56.968 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:37:56.961+0000 7f73168c6640 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x5615e3a12620 0x5615e3a12a00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T10:37:56.968 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:37:56.961+0000 7f73160c5640 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x5615e3a12f40 0x5615e3a1d2c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T10:37:56.968 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:37:56.961+0000 7f73168c6640 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x5615e3a12620 0x5615e3a12a00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.204:3300/0 says I am v2:172.21.15.114:33388/0 (socket says 172.21.15.114:33388) 2024-04-23T10:37:56.968 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:37:56.961+0000 7f73168c6640 1 -- 172.21.15.114:0/3125912660 learned_addr learned my addr 172.21.15.114:0/3125912660 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-04-23T10:37:56.968 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:37:56.961+0000 7f73077fe640 1 -- 172.21.15.114:0/3091329814 <== mon.0 v2:172.21.15.114:3300/0 4 ==== mgrmap(e 26) ==== 138778+0+0 (secure 0 0 0) 0x7f730c004e80 con 0x7f7310106190 2024-04-23T10:37:56.968 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:37:56.961+0000 7f73160c5640 1 -- 172.21.15.114:0/3125912660 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x5615e3a12620 msgr2=0x5615e3a12a00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T10:37:56.968 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:37:56.961+0000 7f73160c5640 1 --2- 172.21.15.114:0/3125912660 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x5615e3a12620 0x5615e3a12a00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:37:56.968 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:37:56.961+0000 7f73160c5640 1 -- 172.21.15.114:0/3125912660 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x5615e3a1d800 msgr2=0x5615e3a1fbf0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-04-23T10:37:56.969 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:37:56.961+0000 7f73160c5640 1 --2- 172.21.15.114:0/3125912660 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x5615e3a1d800 0x5615e3a1fbf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:37:56.969 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:37:56.961+0000 7f73160c5640 1 -- 172.21.15.114:0/3125912660 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5615e368a210 con 0x5615e3a12f40 2024-04-23T10:37:56.969 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:37:56.961+0000 7f73168c6640 1 --2- 172.21.15.114:0/3125912660 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x5615e3a12620 0x5615e3a12a00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-23T10:37:56.969 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:37:56.961+0000 7f73160c5640 1 --2- 172.21.15.114:0/3125912660 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x5615e3a12f40 0x5615e3a1d2c0 secure :-1 s=READY pgs=94 cs=0 l=1 rev1=1 crypto rx=0x7f7300009230 tx=0x7f7300032bd0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=c53dcbc923281de9 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T10:37:56.969 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:37:56.962+0000 7f7304bfe640 1 -- 172.21.15.114:0/3125912660 <== mon.2 v2:172.21.15.119:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7300040070 con 0x5615e3a12f40 2024-04-23T10:37:56.969 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:37:56.962+0000 7f7304bfe640 1 -- 172.21.15.114:0/3125912660 <== mon.2 v2:172.21.15.119:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f7300004030 con 0x5615e3a12f40 2024-04-23T10:37:56.969 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:37:56.962+0000 7f7304bfe640 1 -- 172.21.15.114:0/3125912660 <== mon.2 v2:172.21.15.119:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7300035040 con 0x5615e3a12f40 2024-04-23T10:37:56.969 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:37:56.962+0000 7f73077fe640 1 --2- 172.21.15.114:0/3091329814 >> [v2:172.21.15.114:6800/1395693466,v1:172.21.15.114:6801/1395693466] conn(0x7f72fc0a3dc0 0x7f72fc0a6280 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T10:37:56.969 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:37:56.962+0000 7f73168c6640 1 --2- 172.21.15.114:0/3091329814 >> [v2:172.21.15.114:6800/1395693466,v1:172.21.15.114:6801/1395693466] conn(0x7f72fc0a3dc0 0x7f72fc0a6280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T10:37:56.969 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:37:56.962+0000 7f73077fe640 1 -- 172.21.15.114:0/3091329814 <== mon.0 v2:172.21.15.114:3300/0 5 ==== osd_map(104..104 src has 1..104) ==== 8629+0+0 (secure 0 0 0) 0x7f730c0d0170 con 0x7f7310106190 2024-04-23T10:37:56.969 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:37:56.962+0000 7f731b043740 1 -- 172.21.15.114:0/3125912660 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x5615e3a12f40 msgr2=0x5615e3a1d2c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T10:37:56.969 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:37:56.962+0000 7f73168c6640 1 --2- 172.21.15.114:0/3091329814 >> [v2:172.21.15.114:6800/1395693466,v1:172.21.15.114:6801/1395693466] conn(0x7f72fc0a3dc0 0x7f72fc0a6280 secure :-1 s=READY pgs=135 cs=0 l=1 rev1=1 crypto rx=0x7f7308009fd0 tx=0x7f7308009500 comp rx=0 tx=0).ready entity=mgr.15291 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T10:37:56.969 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:37:56.962+0000 7f731b043740 1 --2- 172.21.15.114:0/3125912660 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x5615e3a12f40 0x5615e3a1d2c0 secure :-1 s=READY pgs=94 cs=0 l=1 rev1=1 crypto rx=0x7f7300009230 tx=0x7f7300032bd0 comp rx=0 tx=0).stop 2024-04-23T10:37:56.970 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:37:56.962+0000 7f731b043740 1 -- 172.21.15.114:0/3125912660 shutdown_connections 2024-04-23T10:37:56.970 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:37:56.962+0000 7f731b043740 1 --2- 172.21.15.114:0/3125912660 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x5615e3a1d800 0x5615e3a1fbf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:37:56.970 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:37:56.962+0000 7f731b043740 1 --2- 172.21.15.114:0/3125912660 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x5615e3a12f40 0x5615e3a1d2c0 unknown :-1 s=CLOSED pgs=94 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:37:56.970 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:37:56.962+0000 7f731b043740 1 --2- 172.21.15.114:0/3125912660 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x5615e3a12620 0x5615e3a12a00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:37:56.970 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:37:56.962+0000 7f731b043740 1 -- 172.21.15.114:0/3125912660 >> 172.21.15.114:0/3125912660 conn(0x5615e3a0eb30 msgr2=0x5615e3a0ef10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T10:37:56.970 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:37:56.963+0000 7f731b043740 1 -- 172.21.15.114:0/3125912660 shutdown_connections 2024-04-23T10:37:56.970 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:37:56.963+0000 7f731b043740 1 -- 172.21.15.114:0/3125912660 wait complete. 2024-04-23T10:37:56.970 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:37:56.963+0000 7f731b043740 1 -- start start 2024-04-23T10:37:56.970 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:37:56.963+0000 7f731b043740 1 Processor -- start 2024-04-23T10:37:56.970 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:37:56.963+0000 7f731b043740 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x5615e3a12620 0x5615e3a1d190 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T10:37:56.970 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:37:56.963+0000 7f731b043740 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x5615e3a12f40 0x5615e3a1a480 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T10:37:56.970 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:37:56.963+0000 7f731b043740 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x5615e3a1d800 0x5615e3a1aa00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T10:37:56.971 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:37:56.963+0000 7f731b043740 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x5615e3a203d0 con 0x5615e3a12f40 2024-04-23T10:37:56.971 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:37:56.963+0000 7f731b043740 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 -- 0x5615e39120b0 con 0x5615e3a1d800 2024-04-23T10:37:56.971 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:37:56.963+0000 7f731b043740 1 -- --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_getmap magic: 0 -- 0x5615e3a20550 con 0x5615e3a12620 2024-04-23T10:37:56.971 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:37:56.963+0000 7f731b043740 10 client.0 _ll_register_callbacks cb 0x5615e38ffd30 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-23T10:37:56.971 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:37:56.963+0000 7f73158c4640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x5615e3a12f40 0x5615e3a1a480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T10:37:56.971 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:37:56.963+0000 7f73158c4640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x5615e3a12f40 0x5615e3a1a480 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.114:36498/0 (socket says 172.21.15.114:36498) 2024-04-23T10:37:56.971 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:37:56.963+0000 7f73158c4640 1 -- 172.21.15.114:0/121934418 learned_addr learned my addr 172.21.15.114:0/121934418 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-04-23T10:37:56.971 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:37:56.963+0000 7f73168c6640 1 --2- 172.21.15.114:0/121934418 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x5615e3a1d800 0x5615e3a1aa00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T10:37:56.971 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:37:56.963+0000 7f73160c5640 1 --2- 172.21.15.114:0/121934418 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x5615e3a12620 0x5615e3a1d190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T10:37:56.971 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:37:56.964+0000 7f73158c4640 1 -- 172.21.15.114:0/121934418 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x5615e3a12620 msgr2=0x5615e3a1d190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T10:37:56.972 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:37:56.964+0000 7f73158c4640 1 --2- 172.21.15.114:0/121934418 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x5615e3a12620 0x5615e3a1d190 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:37:56.972 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:37:56.964+0000 7f73158c4640 1 -- 172.21.15.114:0/121934418 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x5615e3a1d800 msgr2=0x5615e3a1aa00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T10:37:56.972 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:37:56.964+0000 7f73158c4640 1 --2- 172.21.15.114:0/121934418 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x5615e3a1d800 0x5615e3a1aa00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:37:56.972 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:37:56.964+0000 7f73158c4640 1 -- 172.21.15.114:0/121934418 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5615e3756620 con 0x5615e3a12f40 2024-04-23T10:37:56.972 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:37:56.964+0000 7f73158c4640 1 --2- 172.21.15.114:0/121934418 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x5615e3a12f40 0x5615e3a1a480 secure :-1 s=READY pgs=869 cs=0 l=1 rev1=1 crypto rx=0x7f730c0cd8b0 tx=0x7f730c09c150 comp rx=0 tx=0).ready entity=mon.0 client_cookie=348e737c9646d0b2 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T10:37:56.972 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:37:56.964+0000 7f72f9ffb640 10 client.0 ms_handle_connect on v2:172.21.15.114:3300/0 2024-04-23T10:37:56.972 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:37:56.964+0000 7f72f9ffb640 1 -- 172.21.15.114:0/121934418 <== mon.0 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f730c020040 con 0x5615e3a12f40 2024-04-23T10:37:56.972 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:37:56.964+0000 7f72f9ffb640 1 -- 172.21.15.114:0/121934418 <== mon.0 v2:172.21.15.114:3300/0 2 ==== osd_map(104..104 src has 1..104) ==== 8629+0+0 (secure 0 0 0) 0x7f730c0a00b0 con 0x5615e3a12f40 2024-04-23T10:37:56.972 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:37:56.964+0000 7f731b043740 1 -- 172.21.15.114:0/121934418 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f7300004030 con 0x5615e3a12f40 2024-04-23T10:37:56.972 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:37:56.964+0000 7f72f9ffb640 1 -- 172.21.15.114:0/121934418 <== mon.0 v2:172.21.15.114:3300/0 3 ==== mon_get_version_reply(handle=1 version=38) ==== 24+0+0 (secure 0 0 0) 0x7f730c09b5a0 con 0x5615e3a12f40 2024-04-23T10:37:56.972 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:37:56.964+0000 7f731b043740 10 client.16869 fetch_fsmap learned FSMap version 38 2024-04-23T10:37:56.972 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:37:56.964+0000 7f731b043740 1 -- 172.21.15.114:0/121934418 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({config=0+,fsmap=38,monmap=4+}) -- 0x5615e3a2c370 con 0x5615e3a12f40 2024-04-23T10:37:56.972 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:37:56.964+0000 7f72f9ffb640 1 -- 172.21.15.114:0/121934418 <== mon.0 v2:172.21.15.114:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f7300004030 con 0x5615e3a12f40 2024-04-23T10:37:56.973 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:37:56.965+0000 7f72f9ffb640 1 -- 172.21.15.114:0/121934418 <== mon.0 v2:172.21.15.114:3300/0 5 ==== fsmap(e 38) ==== 8354+0+0 (secure 0 0 0) 0x7f730c09b0a0 con 0x5615e3a12f40 2024-04-23T10:37:56.973 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:37:56.965+0000 7f731b043740 10 client.16869 fetch_fsmap finished waiting for FSMap version 38 2024-04-23T10:37:56.973 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:37:56.965+0000 7f731b043740 10 client.16869 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-04-23T10:37:56.973 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:37:56.965+0000 7f731b043740 20 client.16869 populate_metadata read hostname 'smithi114' 2024-04-23T10:37:56.973 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:37:56.965+0000 7f731b043740 1 --2- 172.21.15.114:0/121934418 >> [v2:172.21.15.119:6838/937297340,v1:172.21.15.119:6839/937297340] conn(0x5615e3a2d830 0x5615e3a2fc20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T10:37:56.973 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:37:56.965+0000 7f731b043740 4 client.16869 mds_command: new command op to 24545 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-23T10:37:56.973 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:37:56.965+0000 7f731b043740 1 -- 172.21.15.114:0/121934418 --> [v2:172.21.15.119:6838/937297340,v1:172.21.15.119:6839/937297340] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x5615e3985390 con 0x5615e3a2d830 2024-04-23T10:37:56.973 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:37:56.965+0000 7f73160c5640 1 --2- 172.21.15.114:0/121934418 >> [v2:172.21.15.119:6838/937297340,v1:172.21.15.119:6839/937297340] conn(0x5615e3a2d830 0x5615e3a2fc20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T10:37:56.973 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:37:56.965+0000 7f72f9ffb640 10 client.16869 ms_handle_connect on v2:172.21.15.119:6838/937297340 2024-04-23T10:37:56.973 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:37:56.965+0000 7f73160c5640 1 --2- 172.21.15.114:0/121934418 >> [v2:172.21.15.119:6838/937297340,v1:172.21.15.119:6839/937297340] conn(0x5615e3a2d830 0x5615e3a2fc20 crc :-1 s=READY pgs=162 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=28a6e946a99f2ee0 server_cookie=c3f32f9715a96022 in_seq=0 out_seq=0 2024-04-23T10:37:56.973 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:37:56.966+0000 7f72f9ffb640 1 -- 172.21.15.114:0/121934418 <== mds.0 v2:172.21.15.119:6838/937297340 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x5615e3985390 con 0x5615e3a2d830 2024-04-23T10:37:56.973 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:37:56.966+0000 7f72f9ffb640 10 client.16869 handle_command_reply: tid=0 2024-04-23T10:37:56.973 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:37:56.966+0000 7f731b043740 1 client.16869 shutdown 2024-04-23T10:37:56.973 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:37:56.966+0000 7f731b043740 1 -- 172.21.15.114:0/121934418 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x5615e3a12f40 msgr2=0x5615e3a1a480 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T10:37:56.973 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:37:56.966+0000 7f731b043740 1 --2- 172.21.15.114:0/121934418 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x5615e3a12f40 0x5615e3a1a480 secure :-1 s=READY pgs=869 cs=0 l=1 rev1=1 crypto rx=0x7f730c0cd8b0 tx=0x7f730c09c150 comp rx=0 tx=0).stop 2024-04-23T10:37:56.974 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:37:56.966+0000 7f731b043740 1 -- 172.21.15.114:0/121934418 shutdown_connections 2024-04-23T10:37:56.974 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:37:56.966+0000 7f731b043740 1 --2- 172.21.15.114:0/121934418 >> [v2:172.21.15.119:6838/937297340,v1:172.21.15.119:6839/937297340] conn(0x5615e3a2d830 0x5615e3a2fc20 crc :-1 s=READY pgs=162 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:37:56.974 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:37:56.966+0000 7f731b043740 1 --2- 172.21.15.114:0/121934418 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x5615e3a1d800 0x5615e3a1aa00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:37:56.974 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:37:56.966+0000 7f72f9ffb640 0 client.16869 ms_handle_reset on v2:172.21.15.119:6838/937297340 2024-04-23T10:37:56.974 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:37:56.966+0000 7f731b043740 1 --2- 172.21.15.114:0/121934418 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x5615e3a12f40 0x5615e3a1a480 unknown :-1 s=CLOSED pgs=869 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:37:56.974 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:37:56.966+0000 7f731b043740 1 --2- 172.21.15.114:0/121934418 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x5615e3a12620 0x5615e3a1d190 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:37:56.974 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:37:56.966+0000 7f731b043740 1 -- 172.21.15.114:0/121934418 >> 172.21.15.114:0/121934418 conn(0x5615e3a0eb30 msgr2=0x5615e3a0dcb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T10:37:56.974 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:37:56.966+0000 7f731b043740 1 -- 172.21.15.114:0/121934418 shutdown_connections 2024-04-23T10:37:56.974 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:37:56.966+0000 7f731b043740 1 -- 172.21.15.114:0/121934418 wait complete. 2024-04-23T10:37:56.974 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:37:56.966+0000 7f731b043740 20 client.16869 trim_cache size 0 max 16384 2024-04-23T10:37:56.985 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:37:56.981+0000 7f731b043740 1 Processor -- start 2024-04-23T10:37:56.985 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:37:56.981+0000 7f731b043740 1 -- start start 2024-04-23T10:37:56.985 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:37:56.981+0000 7f731b043740 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x5615e3a12620 0x5615e3a29290 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T10:37:56.985 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:37:56.981+0000 7f731b043740 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x5615e3a12f40 0x5615e3a0db30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T10:37:56.985 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:37:56.981+0000 7f731b043740 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x5615e3a1d800 0x5615e3a1ccc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T10:37:56.985 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:37:56.981+0000 7f731b043740 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x7f73101b0660 con 0x5615e3a12f40 2024-04-23T10:37:56.985 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:37:56.981+0000 7f731b043740 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 -- 0x7f73101ade70 con 0x5615e3a12620 2024-04-23T10:37:56.985 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:37:56.981+0000 7f731b043740 1 -- --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_getmap magic: 0 -- 0x7f73101abce0 con 0x5615e3a1d800 2024-04-23T10:37:56.985 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:37:56.981+0000 7f73158c4640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x5615e3a12f40 0x5615e3a0db30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T10:37:56.985 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:37:56.981+0000 7f73158c4640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x5615e3a12f40 0x5615e3a0db30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.114:36504/0 (socket says 172.21.15.114:36504) 2024-04-23T10:37:56.985 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:37:56.981+0000 7f73158c4640 1 -- 172.21.15.114:0/808924171 learned_addr learned my addr 172.21.15.114:0/808924171 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-04-23T10:37:56.985 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:37:56.981+0000 7f73160c5640 1 --2- 172.21.15.114:0/808924171 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x5615e3a12620 0x5615e3a29290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T10:37:56.986 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:37:56.981+0000 7f73168c6640 1 --2- 172.21.15.114:0/808924171 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x5615e3a1d800 0x5615e3a1ccc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T10:37:56.986 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:37:56.981+0000 7f73158c4640 1 -- 172.21.15.114:0/808924171 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x5615e3a1d800 msgr2=0x5615e3a1ccc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T10:37:56.986 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:37:56.981+0000 7f73158c4640 1 --2- 172.21.15.114:0/808924171 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x5615e3a1d800 0x5615e3a1ccc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:37:56.986 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:37:56.981+0000 7f73158c4640 1 -- 172.21.15.114:0/808924171 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x5615e3a12620 msgr2=0x5615e3a29290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T10:37:56.986 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:37:56.981+0000 7f73158c4640 1 --2- 172.21.15.114:0/808924171 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x5615e3a12620 0x5615e3a29290 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:37:56.986 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:37:56.981+0000 7f73158c4640 1 -- 172.21.15.114:0/808924171 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5615e3a19450 con 0x5615e3a12f40 2024-04-23T10:37:56.986 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:37:56.981+0000 7f73168c6640 1 --2- 172.21.15.114:0/808924171 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x5615e3a1d800 0x5615e3a1ccc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-23T10:37:56.986 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:37:56.981+0000 7f73160c5640 1 --2- 172.21.15.114:0/808924171 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x5615e3a12620 0x5615e3a29290 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-23T10:37:56.986 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:37:56.982+0000 7f73158c4640 1 --2- 172.21.15.114:0/808924171 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x5615e3a12f40 0x5615e3a0db30 secure :-1 s=READY pgs=870 cs=0 l=1 rev1=1 crypto rx=0x7f730c098b20 tx=0x7f730c005bf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=581612cec202bdca server_cookie=0 in_seq=0 out_seq=0 2024-04-23T10:37:56.986 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:37:56.982+0000 7f72fbfff640 1 -- 172.21.15.114:0/808924171 <== mon.0 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f730c014070 con 0x5615e3a12f40 2024-04-23T10:37:56.986 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:37:56.982+0000 7f72fbfff640 1 -- 172.21.15.114:0/808924171 <== mon.0 v2:172.21.15.114:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f730c0ba020 con 0x5615e3a12f40 2024-04-23T10:37:56.986 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:37:56.982+0000 7f72fbfff640 1 -- 172.21.15.114:0/808924171 <== mon.0 v2:172.21.15.114:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f730c020070 con 0x5615e3a12f40 2024-04-23T10:37:56.986 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:37:56.982+0000 7f731b043740 1 -- 172.21.15.114:0/808924171 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x5615e3a12f40 msgr2=0x5615e3a0db30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T10:37:56.987 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:37:56.982+0000 7f731b043740 1 --2- 172.21.15.114:0/808924171 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x5615e3a12f40 0x5615e3a0db30 secure :-1 s=READY pgs=870 cs=0 l=1 rev1=1 crypto rx=0x7f730c098b20 tx=0x7f730c005bf0 comp rx=0 tx=0).stop 2024-04-23T10:37:56.987 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:37:56.982+0000 7f731b043740 1 -- 172.21.15.114:0/808924171 shutdown_connections 2024-04-23T10:37:56.987 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:37:56.982+0000 7f731b043740 1 --2- 172.21.15.114:0/808924171 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x5615e3a1d800 0x5615e3a1ccc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:37:56.987 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:37:56.982+0000 7f731b043740 1 --2- 172.21.15.114:0/808924171 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x5615e3a12f40 0x5615e3a0db30 unknown :-1 s=CLOSED pgs=870 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:37:56.987 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:37:56.982+0000 7f731b043740 1 --2- 172.21.15.114:0/808924171 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x5615e3a12620 0x5615e3a29290 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:37:56.987 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:37:56.982+0000 7f731b043740 1 -- 172.21.15.114:0/808924171 >> 172.21.15.114:0/808924171 conn(0x5615e3a0eb30 msgr2=0x5615e3a0d420 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T10:37:56.987 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:37:56.983+0000 7f731b043740 1 -- 172.21.15.114:0/808924171 shutdown_connections 2024-04-23T10:37:56.987 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:37:56.983+0000 7f731b043740 1 -- 172.21.15.114:0/808924171 wait complete. 2024-04-23T10:37:56.987 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:37:56.983+0000 7f731b043740 1 -- start start 2024-04-23T10:37:56.987 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:37:56.983+0000 7f731b043740 1 Processor -- start 2024-04-23T10:37:56.987 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:37:56.983+0000 7f731b043740 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x5615e3a12620 0x5615e3a17130 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T10:37:56.987 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:37:56.983+0000 7f731b043740 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x5615e3a12f40 0x5615e3a17670 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T10:37:56.987 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:37:56.983+0000 7f731b043740 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x5615e3a1d800 0x5615e3a17bb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T10:37:56.988 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:37:56.983+0000 7f731b043740 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x7f73101ac330 con 0x5615e3a12f40 2024-04-23T10:37:56.988 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:37:56.983+0000 7f731b043740 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 -- 0x7f73101ad7a0 con 0x5615e3a1d800 2024-04-23T10:37:56.988 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:37:56.983+0000 7f731b043740 1 -- --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_getmap magic: 0 -- 0x7f73101aca00 con 0x5615e3a12620 2024-04-23T10:37:56.988 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:37:56.983+0000 7f731b043740 10 client.0 _ll_register_callbacks cb 0x7f730c09bc90 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-23T10:37:56.988 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:37:56.983+0000 7f73158c4640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x5615e3a12f40 0x5615e3a17670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T10:37:56.988 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:37:56.983+0000 7f73158c4640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x5615e3a12f40 0x5615e3a17670 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.114:36508/0 (socket says 172.21.15.114:36508) 2024-04-23T10:37:56.988 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:37:56.983+0000 7f73158c4640 1 -- 172.21.15.114:0/223767440 learned_addr learned my addr 172.21.15.114:0/223767440 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-04-23T10:37:56.988 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:37:56.983+0000 7f73160c5640 1 --2- 172.21.15.114:0/223767440 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x5615e3a12620 0x5615e3a17130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T10:37:56.988 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:37:56.983+0000 7f73168c6640 1 --2- 172.21.15.114:0/223767440 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x5615e3a1d800 0x5615e3a17bb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T10:37:56.988 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:37:56.983+0000 7f73158c4640 1 -- 172.21.15.114:0/223767440 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x5615e3a12620 msgr2=0x5615e3a17130 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T10:37:56.988 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:37:56.983+0000 7f73158c4640 1 --2- 172.21.15.114:0/223767440 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x5615e3a12620 0x5615e3a17130 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:37:56.988 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:37:56.983+0000 7f73158c4640 1 -- 172.21.15.114:0/223767440 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x5615e3a1d800 msgr2=0x5615e3a17bb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T10:37:56.988 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:37:56.983+0000 7f73158c4640 1 --2- 172.21.15.114:0/223767440 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x5615e3a1d800 0x5615e3a17bb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:37:56.989 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:37:56.984+0000 7f73158c4640 1 -- 172.21.15.114:0/223767440 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5615e3a12d70 con 0x5615e3a12f40 2024-04-23T10:37:56.989 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:37:56.984+0000 7f73158c4640 1 --2- 172.21.15.114:0/223767440 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x5615e3a12f40 0x5615e3a17670 secure :-1 s=READY pgs=871 cs=0 l=1 rev1=1 crypto rx=0x7f730c005c60 tx=0x7f730c0a1a70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7d7438fa3cc270d7 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T10:37:56.989 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:37:56.984+0000 7f72f9ffb640 10 client.0 ms_handle_connect on v2:172.21.15.114:3300/0 2024-04-23T10:37:56.989 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:37:56.984+0000 7f72f9ffb640 1 -- 172.21.15.114:0/223767440 <== mon.0 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f730c014050 con 0x5615e3a12f40 2024-04-23T10:37:56.989 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:37:56.984+0000 7f72f9ffb640 1 -- 172.21.15.114:0/223767440 <== mon.0 v2:172.21.15.114:3300/0 2 ==== osd_map(104..104 src has 1..104) ==== 8629+0+0 (secure 0 0 0) 0x7f730c09b010 con 0x5615e3a12f40 2024-04-23T10:37:56.989 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:37:56.984+0000 7f731b043740 1 -- 172.21.15.114:0/223767440 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f730c0ba020 con 0x5615e3a12f40 2024-04-23T10:37:56.989 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:37:56.984+0000 7f72f9ffb640 1 -- 172.21.15.114:0/223767440 <== mon.0 v2:172.21.15.114:3300/0 3 ==== mon_get_version_reply(handle=1 version=38) ==== 24+0+0 (secure 0 0 0) 0x7f730c020070 con 0x5615e3a12f40 2024-04-23T10:37:56.989 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:37:56.984+0000 7f731b043740 10 client.16875 fetch_fsmap learned FSMap version 38 2024-04-23T10:37:56.989 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:37:56.984+0000 7f731b043740 1 -- 172.21.15.114:0/223767440 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({config=0+,fsmap=38,monmap=4+}) -- 0x5615e3a1ad70 con 0x5615e3a12f40 2024-04-23T10:37:56.989 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:37:56.984+0000 7f72f9ffb640 1 -- 172.21.15.114:0/223767440 <== mon.0 v2:172.21.15.114:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f730c0ba020 con 0x5615e3a12f40 2024-04-23T10:37:56.989 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:37:56.984+0000 7f72f9ffb640 1 -- 172.21.15.114:0/223767440 <== mon.0 v2:172.21.15.114:3300/0 5 ==== fsmap(e 38) ==== 8354+0+0 (secure 0 0 0) 0x7f730c082030 con 0x5615e3a12f40 2024-04-23T10:37:56.989 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:37:56.984+0000 7f731b043740 10 client.16875 fetch_fsmap finished waiting for FSMap version 38 2024-04-23T10:37:56.989 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:37:56.984+0000 7f731b043740 10 client.16875 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-04-23T10:37:56.989 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:37:56.984+0000 7f731b043740 20 client.16875 populate_metadata read hostname 'smithi114' 2024-04-23T10:37:56.990 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:37:56.984+0000 7f731b043740 1 --2- 172.21.15.114:0/223767440 >> [v2:172.21.15.119:6838/937297340,v1:172.21.15.119:6839/937297340] conn(0x5615e3a940f0 0x5615e3a96510 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T10:37:56.990 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:37:56.984+0000 7f731b043740 4 client.16875 mds_command: new command op to 24545 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-04-23T10:37:56.990 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:37:56.984+0000 7f731b043740 1 -- 172.21.15.114:0/223767440 --> [v2:172.21.15.119:6838/937297340,v1:172.21.15.119:6839/937297340] -- command(tid 0: {"prefix": "flush journal"}) -- 0x5615e397f200 con 0x5615e3a940f0 2024-04-23T10:37:56.990 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:37:56.985+0000 7f73160c5640 1 --2- 172.21.15.114:0/223767440 >> [v2:172.21.15.119:6838/937297340,v1:172.21.15.119:6839/937297340] conn(0x5615e3a940f0 0x5615e3a96510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T10:37:56.990 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:37:56.985+0000 7f73160c5640 1 --2- 172.21.15.114:0/223767440 >> [v2:172.21.15.119:6838/937297340,v1:172.21.15.119:6839/937297340] conn(0x5615e3a940f0 0x5615e3a96510 crc :-1 s=READY pgs=163 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=17ff0ed37450dde7 server_cookie=cdc7fe81d30f7b26 in_seq=0 out_seq=0 2024-04-23T10:37:56.990 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:37:56.985+0000 7f72f9ffb640 10 client.16875 ms_handle_connect on v2:172.21.15.119:6838/937297340 2024-04-23T10:37:57.023 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:37:57.024+0000 7f72f9ffb640 1 -- 172.21.15.114:0/223767440 <== mds.0 v2:172.21.15.119:6838/937297340 1 ==== command_reply(tid 0: 0 ) ==== 8+0+44 (crc 0 0 0) 0x5615e397f200 con 0x5615e3a940f0 2024-04-23T10:37:57.023 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:37:57.024+0000 7f72f9ffb640 10 client.16875 handle_command_reply: tid=0 2024-04-23T10:37:57.023 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:37:57.024+0000 7f731b043740 1 client.16875 shutdown 2024-04-23T10:37:57.023 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:37:57.024+0000 7f731b043740 1 -- 172.21.15.114:0/223767440 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x5615e3a12f40 msgr2=0x5615e3a17670 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T10:37:57.023 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:37:57.024+0000 7f731b043740 1 --2- 172.21.15.114:0/223767440 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x5615e3a12f40 0x5615e3a17670 secure :-1 s=READY pgs=871 cs=0 l=1 rev1=1 crypto rx=0x7f730c005c60 tx=0x7f730c0a1a70 comp rx=0 tx=0).stop 2024-04-23T10:37:57.024 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:37:57.024+0000 7f731b043740 1 -- 172.21.15.114:0/223767440 shutdown_connections 2024-04-23T10:37:57.024 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:37:57.024+0000 7f731b043740 1 --2- 172.21.15.114:0/223767440 >> [v2:172.21.15.119:6838/937297340,v1:172.21.15.119:6839/937297340] conn(0x5615e3a940f0 0x5615e3a96510 crc :-1 s=READY pgs=163 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:37:57.024 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:37:57.024+0000 7f72f9ffb640 0 client.16875 ms_handle_reset on v2:172.21.15.119:6838/937297340 2024-04-23T10:37:57.024 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:37:57.024+0000 7f731b043740 1 --2- 172.21.15.114:0/223767440 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x5615e3a1d800 0x5615e3a17bb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:37:57.024 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:37:57.024+0000 7f731b043740 1 --2- 172.21.15.114:0/223767440 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x5615e3a12f40 0x5615e3a17670 unknown :-1 s=CLOSED pgs=871 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:37:57.024 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:37:57.024+0000 7f731b043740 1 --2- 172.21.15.114:0/223767440 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x5615e3a12620 0x5615e3a17130 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:37:57.024 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:37:57.024+0000 7f731b043740 1 -- 172.21.15.114:0/223767440 >> 172.21.15.114:0/223767440 conn(0x5615e3a0eb30 msgr2=0x5615e3a0dcb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T10:37:57.024 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:37:57.024+0000 7f731b043740 1 -- 172.21.15.114:0/223767440 shutdown_connections 2024-04-23T10:37:57.024 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:37:57.024+0000 7f731b043740 1 -- 172.21.15.114:0/223767440 wait complete. 2024-04-23T10:37:57.024 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:37:57.024+0000 7f731b043740 20 client.16875 trim_cache size 0 max 16384 2024-04-23T10:37:57.024 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stdout:{ 2024-04-23T10:37:57.024 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stdout: "message": "", 2024-04-23T10:37:57.025 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stdout: "return_code": 0 2024-04-23T10:37:57.025 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stdout:} 2024-04-23T10:37:57.026 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:37:57.025+0000 7f7318329640 1 -- 172.21.15.114:0/3091329814 >> [v2:172.21.15.114:6800/1395693466,v1:172.21.15.114:6801/1395693466] conn(0x7f72fc0a3dc0 msgr2=0x7f72fc0a6280 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T10:37:57.026 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:37:57.025+0000 7f7318329640 1 --2- 172.21.15.114:0/3091329814 >> [v2:172.21.15.114:6800/1395693466,v1:172.21.15.114:6801/1395693466] conn(0x7f72fc0a3dc0 0x7f72fc0a6280 secure :-1 s=READY pgs=135 cs=0 l=1 rev1=1 crypto rx=0x7f7308009fd0 tx=0x7f7308009500 comp rx=0 tx=0).stop 2024-04-23T10:37:57.026 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:37:57.025+0000 7f7318329640 1 -- 172.21.15.114:0/3091329814 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f7310106190 msgr2=0x7f7310198c10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T10:37:57.026 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:37:57.025+0000 7f7318329640 1 --2- 172.21.15.114:0/3091329814 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f7310106190 0x7f7310198c10 secure :-1 s=READY pgs=868 cs=0 l=1 rev1=1 crypto rx=0x7f730c00d740 tx=0x7f730c00dc00 comp rx=0 tx=0).stop 2024-04-23T10:37:57.026 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:37:57.025+0000 7f7318329640 1 -- 172.21.15.114:0/3091329814 shutdown_connections 2024-04-23T10:37:57.026 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:37:57.026+0000 7f7318329640 1 --2- 172.21.15.114:0/3091329814 >> [v2:172.21.15.114:6800/1395693466,v1:172.21.15.114:6801/1395693466] conn(0x7f72fc0a3dc0 0x7f72fc0a6280 unknown :-1 s=CLOSED pgs=135 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:37:57.026 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:37:57.026+0000 7f7318329640 1 --2- 172.21.15.114:0/3091329814 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f731019cf90 0x7f731019d440 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:37:57.027 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:37:57.026+0000 7f7318329640 1 --2- 172.21.15.114:0/3091329814 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f7310106190 0x7f7310198c10 unknown :-1 s=CLOSED pgs=868 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:37:57.027 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:37:57.026+0000 7f7318329640 1 --2- 172.21.15.114:0/3091329814 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f73101010c0 0x7f73101986d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:37:57.027 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:37:57.026+0000 7f7318329640 1 -- 172.21.15.114:0/3091329814 >> 172.21.15.114:0/3091329814 conn(0x7f7310077fa0 msgr2=0x7f731006a470 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T10:37:57.027 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:37:57.026+0000 7f7318329640 1 -- 172.21.15.114:0/3091329814 shutdown_connections 2024-04-23T10:37:57.027 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:37:57.026+0000 7f7318329640 1 -- 172.21.15.114:0/3091329814 wait complete. 2024-04-23T10:37:57.944 INFO:tasks.workunit.client.0.smithi114.stdout: 1 681926 5.63 MB/sec execute 486 sec latency 1894.866 ms 2024-04-23T10:37:57.992 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 23 10:37:57 smithi119 ceph-mon[32608]: pgmap v355: 97 pgs: 97 active+clean; 215 MiB data, 2.7 GiB used, 1.0 TiB / 1.0 TiB avail; 563 KiB/s rd, 1.0 MiB/s wr, 103 op/s 2024-04-23T10:37:57.992 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 23 10:37:57 smithi119 ceph-mon[32608]: from='mgr.15291 172.21.15.114:0/1095789187' entity='mgr.x' 2024-04-23T10:37:57.992 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 23 10:37:57 smithi119 ceph-mon[32608]: from='mgr.15291 172.21.15.114:0/1095789187' entity='mgr.x' 2024-04-23T10:37:57.992 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 23 10:37:57 smithi119 ceph-mon[32608]: from='mgr.15291 172.21.15.114:0/1095789187' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-23T10:37:57.992 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 23 10:37:57 smithi119 ceph-mon[32608]: from='mgr.15291 172.21.15.114:0/1095789187' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-23T10:37:57.992 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 23 10:37:57 smithi119 ceph-mon[32608]: from='mgr.15291 172.21.15.114:0/1095789187' entity='mgr.x' 2024-04-23T10:37:58.004 INFO:journalctl@ceph.mon.a.smithi114.stdout:Apr 23 10:37:57 smithi114 ceph-mon[29255]: pgmap v355: 97 pgs: 97 active+clean; 215 MiB data, 2.7 GiB used, 1.0 TiB / 1.0 TiB avail; 563 KiB/s rd, 1.0 MiB/s wr, 103 op/s 2024-04-23T10:37:58.004 INFO:journalctl@ceph.mon.a.smithi114.stdout:Apr 23 10:37:57 smithi114 ceph-mon[29255]: from='mgr.15291 172.21.15.114:0/1095789187' entity='mgr.x' 2024-04-23T10:37:58.004 INFO:journalctl@ceph.mon.a.smithi114.stdout:Apr 23 10:37:57 smithi114 ceph-mon[29255]: from='mgr.15291 172.21.15.114:0/1095789187' entity='mgr.x' 2024-04-23T10:37:58.004 INFO:journalctl@ceph.mon.a.smithi114.stdout:Apr 23 10:37:57 smithi114 ceph-mon[29255]: from='mgr.15291 172.21.15.114:0/1095789187' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-23T10:37:58.004 INFO:journalctl@ceph.mon.a.smithi114.stdout:Apr 23 10:37:57 smithi114 ceph-mon[29255]: from='mgr.15291 172.21.15.114:0/1095789187' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-23T10:37:58.004 INFO:journalctl@ceph.mon.a.smithi114.stdout:Apr 23 10:37:57 smithi114 ceph-mon[29255]: from='mgr.15291 172.21.15.114:0/1095789187' entity='mgr.x' 2024-04-23T10:37:58.077 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 23 10:37:57 smithi204 ceph-mon[32649]: pgmap v355: 97 pgs: 97 active+clean; 215 MiB data, 2.7 GiB used, 1.0 TiB / 1.0 TiB avail; 563 KiB/s rd, 1.0 MiB/s wr, 103 op/s 2024-04-23T10:37:58.077 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 23 10:37:57 smithi204 ceph-mon[32649]: from='mgr.15291 172.21.15.114:0/1095789187' entity='mgr.x' 2024-04-23T10:37:58.078 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 23 10:37:57 smithi204 ceph-mon[32649]: from='mgr.15291 172.21.15.114:0/1095789187' entity='mgr.x' 2024-04-23T10:37:58.078 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 23 10:37:57 smithi204 ceph-mon[32649]: from='mgr.15291 172.21.15.114:0/1095789187' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-23T10:37:58.078 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 23 10:37:57 smithi204 ceph-mon[32649]: from='mgr.15291 172.21.15.114:0/1095789187' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-23T10:37:58.078 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 23 10:37:57 smithi204 ceph-mon[32649]: from='mgr.15291 172.21.15.114:0/1095789187' entity='mgr.x' 2024-04-23T10:37:58.944 INFO:tasks.workunit.client.0.smithi114.stdout: 1 681926 5.61 MB/sec execute 487 sec latency 2894.951 ms 2024-04-23T10:37:59.944 INFO:tasks.workunit.client.0.smithi114.stdout: 1 681993 5.60 MB/sec execute 488 sec latency 3134.389 ms 2024-04-23T10:37:59.992 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 23 10:37:59 smithi119 ceph-mon[32608]: pgmap v356: 97 pgs: 97 active+clean; 215 MiB data, 2.7 GiB used, 1.0 TiB / 1.0 TiB avail; 313 KiB/s rd, 628 KiB/s wr, 54 op/s 2024-04-23T10:38:00.004 INFO:journalctl@ceph.mon.a.smithi114.stdout:Apr 23 10:37:59 smithi114 ceph-mon[29255]: pgmap v356: 97 pgs: 97 active+clean; 215 MiB data, 2.7 GiB used, 1.0 TiB / 1.0 TiB avail; 313 KiB/s rd, 628 KiB/s wr, 54 op/s 2024-04-23T10:38:00.078 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 23 10:37:59 smithi204 ceph-mon[32649]: pgmap v356: 97 pgs: 97 active+clean; 215 MiB data, 2.7 GiB used, 1.0 TiB / 1.0 TiB avail; 313 KiB/s rd, 628 KiB/s wr, 54 op/s 2024-04-23T10:38:00.944 INFO:tasks.workunit.client.0.smithi114.stdout: 1 681993 5.59 MB/sec execute 489 sec latency 1659.334 ms 2024-04-23T10:38:01.945 INFO:tasks.workunit.client.0.smithi114.stdout: 1 681993 5.58 MB/sec execute 490 sec latency 2659.435 ms 2024-04-23T10:38:01.992 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 23 10:38:01 smithi119 ceph-mon[32608]: pgmap v357: 97 pgs: 97 active+clean; 215 MiB data, 2.7 GiB used, 1.0 TiB / 1.0 TiB avail; 313 KiB/s rd, 628 KiB/s wr, 54 op/s 2024-04-23T10:38:02.004 INFO:journalctl@ceph.mon.a.smithi114.stdout:Apr 23 10:38:01 smithi114 ceph-mon[29255]: pgmap v357: 97 pgs: 97 active+clean; 215 MiB data, 2.7 GiB used, 1.0 TiB / 1.0 TiB avail; 313 KiB/s rd, 628 KiB/s wr, 54 op/s 2024-04-23T10:38:02.077 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 23 10:38:01 smithi204 ceph-mon[32649]: pgmap v357: 97 pgs: 97 active+clean; 215 MiB data, 2.7 GiB used, 1.0 TiB / 1.0 TiB avail; 313 KiB/s rd, 628 KiB/s wr, 54 op/s 2024-04-23T10:38:02.944 INFO:tasks.workunit.client.0.smithi114.stdout: 1 681993 5.57 MB/sec execute 491 sec latency 3659.506 ms 2024-04-23T10:38:03.880 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 23 10:38:03 smithi204 ceph-mon[32649]: pgmap v358: 97 pgs: 97 active+clean; 211 MiB data, 2.7 GiB used, 1.0 TiB / 1.0 TiB avail; 384 KiB/s rd, 887 KiB/s wr, 88 op/s 2024-04-23T10:38:03.945 INFO:tasks.workunit.client.0.smithi114.stdout: 1 681993 5.56 MB/sec execute 492 sec latency 4659.579 ms 2024-04-23T10:38:03.991 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 23 10:38:03 smithi119 ceph-mon[32608]: pgmap v358: 97 pgs: 97 active+clean; 211 MiB data, 2.7 GiB used, 1.0 TiB / 1.0 TiB avail; 384 KiB/s rd, 887 KiB/s wr, 88 op/s 2024-04-23T10:38:04.004 INFO:journalctl@ceph.mon.a.smithi114.stdout:Apr 23 10:38:03 smithi114 ceph-mon[29255]: pgmap v358: 97 pgs: 97 active+clean; 211 MiB data, 2.7 GiB used, 1.0 TiB / 1.0 TiB avail; 384 KiB/s rd, 887 KiB/s wr, 88 op/s 2024-04-23T10:38:04.945 INFO:tasks.workunit.client.0.smithi114.stdout: 1 682285 5.55 MB/sec execute 493 sec latency 4901.519 ms 2024-04-23T10:38:05.945 INFO:tasks.workunit.client.0.smithi114.stdout: 1 683948 5.55 MB/sec execute 494 sec latency 28.072 ms 2024-04-23T10:38:05.992 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 23 10:38:05 smithi119 ceph-mon[32608]: pgmap v359: 97 pgs: 97 active+clean; 211 MiB data, 2.7 GiB used, 1.0 TiB / 1.0 TiB avail; 336 KiB/s rd, 632 KiB/s wr, 75 op/s 2024-04-23T10:38:06.004 INFO:journalctl@ceph.mon.a.smithi114.stdout:Apr 23 10:38:05 smithi114 ceph-mon[29255]: pgmap v359: 97 pgs: 97 active+clean; 211 MiB data, 2.7 GiB used, 1.0 TiB / 1.0 TiB avail; 336 KiB/s rd, 632 KiB/s wr, 75 op/s 2024-04-23T10:38:06.078 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 23 10:38:05 smithi204 ceph-mon[32649]: pgmap v359: 97 pgs: 97 active+clean; 211 MiB data, 2.7 GiB used, 1.0 TiB / 1.0 TiB avail; 336 KiB/s rd, 632 KiB/s wr, 75 op/s 2024-04-23T10:38:06.945 INFO:tasks.workunit.client.0.smithi114.stdout: 1 685472 5.56 MB/sec execute 495 sec latency 444.303 ms 2024-04-23T10:38:07.945 INFO:tasks.workunit.client.0.smithi114.stdout: 1 685472 5.55 MB/sec execute 496 sec latency 1444.389 ms 2024-04-23T10:38:07.992 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 23 10:38:07 smithi119 ceph-mon[32608]: pgmap v360: 97 pgs: 97 active+clean; 214 MiB data, 2.7 GiB used, 1.0 TiB / 1.0 TiB avail; 547 KiB/s rd, 1.1 MiB/s wr, 150 op/s 2024-04-23T10:38:08.004 INFO:journalctl@ceph.mon.a.smithi114.stdout:Apr 23 10:38:07 smithi114 ceph-mon[29255]: pgmap v360: 97 pgs: 97 active+clean; 214 MiB data, 2.7 GiB used, 1.0 TiB / 1.0 TiB avail; 547 KiB/s rd, 1.1 MiB/s wr, 150 op/s 2024-04-23T10:38:08.078 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 23 10:38:07 smithi204 ceph-mon[32649]: pgmap v360: 97 pgs: 97 active+clean; 214 MiB data, 2.7 GiB used, 1.0 TiB / 1.0 TiB avail; 547 KiB/s rd, 1.1 MiB/s wr, 150 op/s 2024-04-23T10:38:08.945 INFO:tasks.workunit.client.0.smithi114.stdout: 1 685472 5.54 MB/sec execute 497 sec latency 2444.439 ms 2024-04-23T10:38:08.992 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 23 10:38:08 smithi119 ceph-mon[32608]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=e=up:active} 3 up:standby-replay 6 up:standby 2024-04-23T10:38:09.004 INFO:journalctl@ceph.mon.a.smithi114.stdout:Apr 23 10:38:08 smithi114 ceph-mon[29255]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=e=up:active} 3 up:standby-replay 6 up:standby 2024-04-23T10:38:09.078 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 23 10:38:08 smithi204 ceph-mon[32649]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=e=up:active} 3 up:standby-replay 6 up:standby 2024-04-23T10:38:09.945 INFO:tasks.workunit.client.0.smithi114.stdout: 1 685539 5.53 MB/sec execute 498 sec latency 2679.921 ms 2024-04-23T10:38:09.992 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 23 10:38:09 smithi119 ceph-mon[32608]: pgmap v361: 97 pgs: 97 active+clean; 214 MiB data, 2.7 GiB used, 1.0 TiB / 1.0 TiB avail; 282 KiB/s rd, 764 KiB/s wr, 108 op/s 2024-04-23T10:38:09.992 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 23 10:38:09 smithi119 ceph-mon[32608]: osdmap e105: 12 total, 12 up, 12 in 2024-04-23T10:38:10.003 INFO:journalctl@ceph.mon.a.smithi114.stdout:Apr 23 10:38:09 smithi114 ceph-mon[29255]: pgmap v361: 97 pgs: 97 active+clean; 214 MiB data, 2.7 GiB used, 1.0 TiB / 1.0 TiB avail; 282 KiB/s rd, 764 KiB/s wr, 108 op/s 2024-04-23T10:38:10.004 INFO:journalctl@ceph.mon.a.smithi114.stdout:Apr 23 10:38:09 smithi114 ceph-mon[29255]: osdmap e105: 12 total, 12 up, 12 in 2024-04-23T10:38:10.078 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 23 10:38:09 smithi204 ceph-mon[32649]: pgmap v361: 97 pgs: 97 active+clean; 214 MiB data, 2.7 GiB used, 1.0 TiB / 1.0 TiB avail; 282 KiB/s rd, 764 KiB/s wr, 108 op/s 2024-04-23T10:38:10.078 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 23 10:38:09 smithi204 ceph-mon[32649]: osdmap e105: 12 total, 12 up, 12 in 2024-04-23T10:38:10.117 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:38:10.116+0000 7fe65fc60640 1 -- 172.21.15.114:0/721940308 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fe6581021c0 msgr2=0x7fe6581025a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T10:38:10.117 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:38:10.116+0000 7fe65fc60640 1 --2- 172.21.15.114:0/721940308 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fe6581021c0 0x7fe6581025a0 secure :-1 s=READY pgs=872 cs=0 l=1 rev1=1 crypto rx=0x7fe648009900 tx=0x7fe64802a410 comp rx=0 tx=0).stop 2024-04-23T10:38:10.117 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:38:10.116+0000 7fe65fc60640 1 -- 172.21.15.114:0/721940308 shutdown_connections 2024-04-23T10:38:10.117 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:38:10.116+0000 7fe65fc60640 1 --2- 172.21.15.114:0/721940308 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fe658107290 0x7fe65810ee60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:38:10.117 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:38:10.116+0000 7fe65fc60640 1 --2- 172.21.15.114:0/721940308 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fe658102b70 0x7fe658106bb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:38:10.117 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:38:10.116+0000 7fe65fc60640 1 --2- 172.21.15.114:0/721940308 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fe6581021c0 0x7fe6581025a0 unknown :-1 s=CLOSED pgs=872 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:38:10.117 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:38:10.116+0000 7fe65fc60640 1 -- 172.21.15.114:0/721940308 >> 172.21.15.114:0/721940308 conn(0x7fe6580fda00 msgr2=0x7fe6580ffe20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T10:38:10.117 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:38:10.116+0000 7fe65fc60640 1 -- 172.21.15.114:0/721940308 shutdown_connections 2024-04-23T10:38:10.117 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:38:10.116+0000 7fe65fc60640 1 -- 172.21.15.114:0/721940308 wait complete. 2024-04-23T10:38:10.117 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:38:10.116+0000 7fe65fc60640 1 Processor -- start 2024-04-23T10:38:10.117 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:38:10.116+0000 7fe65fc60640 1 -- start start 2024-04-23T10:38:10.118 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:38:10.116+0000 7fe65fc60640 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fe6581021c0 0x7fe6581986e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T10:38:10.118 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:38:10.116+0000 7fe65fc60640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fe658107290 0x7fe658198c20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T10:38:10.118 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:38:10.116+0000 7fe65fc60640 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fe65819cfa0 0x7fe65819d450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T10:38:10.118 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:38:10.116+0000 7fe65fc60640 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x7fe658112200 con 0x7fe658107290 2024-04-23T10:38:10.118 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:38:10.117+0000 7fe65fc60640 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 -- 0x7fe658112080 con 0x7fe6581021c0 2024-04-23T10:38:10.118 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:38:10.117+0000 7fe65fc60640 1 -- --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_getmap magic: 0 -- 0x7fe658112380 con 0x7fe65819cfa0 2024-04-23T10:38:10.118 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:38:10.117+0000 7fe65d1fb640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fe658107290 0x7fe658198c20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T10:38:10.118 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:38:10.117+0000 7fe65d1fb640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fe658107290 0x7fe658198c20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.114:52430/0 (socket says 172.21.15.114:52430) 2024-04-23T10:38:10.118 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:38:10.117+0000 7fe65d1fb640 1 -- 172.21.15.114:0/1449004791 learned_addr learned my addr 172.21.15.114:0/1449004791 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-04-23T10:38:10.118 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:38:10.117+0000 7fe65d9fc640 1 --2- 172.21.15.114:0/1449004791 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fe6581021c0 0x7fe6581986e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T10:38:10.118 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:38:10.117+0000 7fe65e1fd640 1 --2- 172.21.15.114:0/1449004791 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fe65819cfa0 0x7fe65819d450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T10:38:10.118 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:38:10.117+0000 7fe65d1fb640 1 -- 172.21.15.114:0/1449004791 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fe65819cfa0 msgr2=0x7fe65819d450 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T10:38:10.118 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:38:10.117+0000 7fe65d1fb640 1 --2- 172.21.15.114:0/1449004791 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fe65819cfa0 0x7fe65819d450 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:38:10.119 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:38:10.117+0000 7fe65d1fb640 1 -- 172.21.15.114:0/1449004791 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fe6581021c0 msgr2=0x7fe6581986e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T10:38:10.119 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:38:10.117+0000 7fe65d1fb640 1 --2- 172.21.15.114:0/1449004791 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fe6581021c0 0x7fe6581986e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:38:10.119 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:38:10.117+0000 7fe65d1fb640 1 -- 172.21.15.114:0/1449004791 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe65819da90 con 0x7fe658107290 2024-04-23T10:38:10.119 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:38:10.117+0000 7fe65d1fb640 1 --2- 172.21.15.114:0/1449004791 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fe658107290 0x7fe658198c20 secure :-1 s=READY pgs=873 cs=0 l=1 rev1=1 crypto rx=0x7fe65400d770 tx=0x7fe65400dc30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T10:38:10.119 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:38:10.117+0000 7fe64effd640 1 -- 172.21.15.114:0/1449004791 <== mon.0 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe654017070 con 0x7fe658107290 2024-04-23T10:38:10.119 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:38:10.117+0000 7fe65fc60640 1 -- 172.21.15.114:0/1449004791 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fe65819dd80 con 0x7fe658107290 2024-04-23T10:38:10.119 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:38:10.117+0000 7fe64effd640 1 -- 172.21.15.114:0/1449004791 <== mon.0 v2:172.21.15.114:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fe654009c30 con 0x7fe658107290 2024-04-23T10:38:10.119 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:38:10.117+0000 7fe64effd640 1 -- 172.21.15.114:0/1449004791 <== mon.0 v2:172.21.15.114:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe65400f620 con 0x7fe658107290 2024-04-23T10:38:10.119 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:38:10.117+0000 7fe65fc60640 1 -- 172.21.15.114:0/1449004791 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fe6581a55c0 con 0x7fe658107290 2024-04-23T10:38:10.125 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:38:10.119+0000 7fe64effd640 1 -- 172.21.15.114:0/1449004791 <== mon.0 v2:172.21.15.114:3300/0 4 ==== mgrmap(e 26) ==== 138778+0+0 (secure 0 0 0) 0x7fe65401a070 con 0x7fe658107290 2024-04-23T10:38:10.125 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:38:10.120+0000 7fe662971740 1 Processor -- start 2024-04-23T10:38:10.125 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:38:10.120+0000 7fe662971740 1 -- start start 2024-04-23T10:38:10.126 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:38:10.120+0000 7fe662971740 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x5576e36b5620 0x5576e36b5a00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T10:38:10.126 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:38:10.120+0000 7fe662971740 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x5576e36b5f40 0x5576e36c02c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T10:38:10.126 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:38:10.120+0000 7fe662971740 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x5576e36c0800 0x5576e36c2bf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T10:38:10.126 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:38:10.120+0000 7fe662971740 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x5576e361a970 con 0x5576e36b5620 2024-04-23T10:38:10.126 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:38:10.120+0000 7fe662971740 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 -- 0x5576e35b74a0 con 0x5576e36b5f40 2024-04-23T10:38:10.126 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:38:10.120+0000 7fe662971740 1 -- --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_getmap magic: 0 -- 0x5576e3616930 con 0x5576e36c0800 2024-04-23T10:38:10.126 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:38:10.120+0000 7fe64effd640 1 --2- 172.21.15.114:0/1449004791 >> [v2:172.21.15.114:6800/1395693466,v1:172.21.15.114:6801/1395693466] conn(0x7fe6440a3dc0 0x7fe6440a6280 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T10:38:10.126 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:38:10.120+0000 7fe65e1fd640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x5576e36b5620 0x5576e36b5a00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T10:38:10.126 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:38:10.120+0000 7fe65e1fd640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x5576e36b5620 0x5576e36b5a00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.114:52438/0 (socket says 172.21.15.114:52438) 2024-04-23T10:38:10.126 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:38:10.120+0000 7fe65e1fd640 1 -- 172.21.15.114:0/1408500025 learned_addr learned my addr 172.21.15.114:0/1408500025 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-04-23T10:38:10.126 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:38:10.120+0000 7fe65e1fd640 1 --2- 172.21.15.114:0/1449004791 >> [v2:172.21.15.114:6800/1395693466,v1:172.21.15.114:6801/1395693466] conn(0x7fe6440a3dc0 0x7fe6440a6280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T10:38:10.126 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:38:10.120+0000 7fe65d9fc640 1 --2- 172.21.15.114:0/1408500025 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x5576e36b5f40 0x5576e36c02c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T10:38:10.126 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:38:10.120+0000 7fe65d1fb640 1 --2- 172.21.15.114:0/1408500025 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x5576e36c0800 0x5576e36c2bf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T10:38:10.127 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:38:10.120+0000 7fe64effd640 1 -- 172.21.15.114:0/1449004791 <== mon.0 v2:172.21.15.114:3300/0 5 ==== osd_map(105..105 src has 1..105) ==== 8673+0+0 (secure 0 0 0) 0x7fe6540d0410 con 0x7fe658107290 2024-04-23T10:38:10.127 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:38:10.121+0000 7fe65e1fd640 1 -- 172.21.15.114:0/1408500025 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x5576e36c0800 msgr2=0x5576e36c2bf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T10:38:10.127 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:38:10.121+0000 7fe65e1fd640 1 --2- 172.21.15.114:0/1408500025 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x5576e36c0800 0x5576e36c2bf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:38:10.127 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:38:10.121+0000 7fe65e1fd640 1 -- 172.21.15.114:0/1408500025 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x5576e36b5f40 msgr2=0x5576e36c02c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T10:38:10.127 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:38:10.121+0000 7fe65e1fd640 1 --2- 172.21.15.114:0/1408500025 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x5576e36b5f40 0x5576e36c02c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:38:10.127 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:38:10.121+0000 7fe65e1fd640 1 -- 172.21.15.114:0/1408500025 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5576e332d210 con 0x5576e36b5620 2024-04-23T10:38:10.127 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:38:10.121+0000 7fe65e1fd640 1 --2- 172.21.15.114:0/1408500025 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x5576e36b5620 0x5576e36b5a00 secure :-1 s=READY pgs=874 cs=0 l=1 rev1=1 crypto rx=0x7fe650012c50 tx=0x7fe650010740 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8f1714822c2e1a3 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T10:38:10.127 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:38:10.121+0000 7fe65e1fd640 1 --2- 172.21.15.114:0/1449004791 >> [v2:172.21.15.114:6800/1395693466,v1:172.21.15.114:6801/1395693466] conn(0x7fe6440a3dc0 0x7fe6440a6280 secure :-1 s=READY pgs=136 cs=0 l=1 rev1=1 crypto rx=0x7fe650010c40 tx=0x7fe650017000 comp rx=0 tx=0).ready entity=mgr.15291 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T10:38:10.127 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:38:10.121+0000 7fe643fff640 1 -- 172.21.15.114:0/1408500025 <== mon.0 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe650012e00 con 0x5576e36b5620 2024-04-23T10:38:10.127 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:38:10.121+0000 7fe643fff640 1 -- 172.21.15.114:0/1408500025 <== mon.0 v2:172.21.15.114:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fe650009c00 con 0x5576e36b5620 2024-04-23T10:38:10.127 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:38:10.121+0000 7fe643fff640 1 -- 172.21.15.114:0/1408500025 <== mon.0 v2:172.21.15.114:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe650005280 con 0x5576e36b5620 2024-04-23T10:38:10.127 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:38:10.122+0000 7fe662971740 1 -- 172.21.15.114:0/1408500025 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x5576e36b5620 msgr2=0x5576e36b5a00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T10:38:10.127 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:38:10.122+0000 7fe662971740 1 --2- 172.21.15.114:0/1408500025 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x5576e36b5620 0x5576e36b5a00 secure :-1 s=READY pgs=874 cs=0 l=1 rev1=1 crypto rx=0x7fe650012c50 tx=0x7fe650010740 comp rx=0 tx=0).stop 2024-04-23T10:38:10.127 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:38:10.122+0000 7fe662971740 1 -- 172.21.15.114:0/1408500025 shutdown_connections 2024-04-23T10:38:10.128 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:38:10.122+0000 7fe662971740 1 --2- 172.21.15.114:0/1408500025 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x5576e36c0800 0x5576e36c2bf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:38:10.128 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:38:10.122+0000 7fe662971740 1 --2- 172.21.15.114:0/1408500025 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x5576e36b5f40 0x5576e36c02c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:38:10.128 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:38:10.122+0000 7fe662971740 1 --2- 172.21.15.114:0/1408500025 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x5576e36b5620 0x5576e36b5a00 unknown :-1 s=CLOSED pgs=874 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:38:10.128 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:38:10.122+0000 7fe662971740 1 -- 172.21.15.114:0/1408500025 >> 172.21.15.114:0/1408500025 conn(0x5576e36b1b30 msgr2=0x5576e36b1f10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T10:38:10.128 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:38:10.122+0000 7fe662971740 1 -- 172.21.15.114:0/1408500025 shutdown_connections 2024-04-23T10:38:10.128 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:38:10.122+0000 7fe662971740 1 -- 172.21.15.114:0/1408500025 wait complete. 2024-04-23T10:38:10.128 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:38:10.122+0000 7fe662971740 1 -- start start 2024-04-23T10:38:10.128 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:38:10.122+0000 7fe662971740 1 Processor -- start 2024-04-23T10:38:10.128 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:38:10.123+0000 7fe662971740 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x5576e36b5620 0x5576e36c01b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T10:38:10.128 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:38:10.123+0000 7fe662971740 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x5576e36b5f40 0x5576e36bd4a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T10:38:10.128 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:38:10.123+0000 7fe662971740 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x5576e36c0800 0x5576e36bda40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T10:38:10.128 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:38:10.123+0000 7fe662971740 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x5576e36c33d0 con 0x5576e36c0800 2024-04-23T10:38:10.128 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:38:10.123+0000 7fe662971740 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 -- 0x5576e35b50b0 con 0x5576e36b5f40 2024-04-23T10:38:10.128 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:38:10.123+0000 7fe662971740 1 -- --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_getmap magic: 0 -- 0x5576e36c3550 con 0x5576e36b5620 2024-04-23T10:38:10.128 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:38:10.123+0000 7fe662971740 10 client.0 _ll_register_callbacks cb 0x5576e35a2d30 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-23T10:38:10.129 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:38:10.123+0000 7fe65e1fd640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x5576e36c0800 0x5576e36bda40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T10:38:10.129 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:38:10.123+0000 7fe65e1fd640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x5576e36c0800 0x5576e36bda40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.114:52448/0 (socket says 172.21.15.114:52448) 2024-04-23T10:38:10.129 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:38:10.123+0000 7fe65e1fd640 1 -- 172.21.15.114:0/2766489802 learned_addr learned my addr 172.21.15.114:0/2766489802 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-04-23T10:38:10.129 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:38:10.123+0000 7fe65d1fb640 1 --2- 172.21.15.114:0/2766489802 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x5576e36b5f40 0x5576e36bd4a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T10:38:10.129 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:38:10.123+0000 7fe65d9fc640 1 --2- 172.21.15.114:0/2766489802 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x5576e36b5620 0x5576e36c01b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T10:38:10.129 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:38:10.123+0000 7fe65e1fd640 1 -- 172.21.15.114:0/2766489802 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x5576e36b5620 msgr2=0x5576e36c01b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T10:38:10.129 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:38:10.123+0000 7fe65e1fd640 1 --2- 172.21.15.114:0/2766489802 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x5576e36b5620 0x5576e36c01b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:38:10.129 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:38:10.123+0000 7fe65e1fd640 1 -- 172.21.15.114:0/2766489802 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x5576e36b5f40 msgr2=0x5576e36bd4a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T10:38:10.129 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:38:10.123+0000 7fe65e1fd640 1 --2- 172.21.15.114:0/2766489802 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x5576e36b5f40 0x5576e36bd4a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:38:10.129 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:38:10.123+0000 7fe65e1fd640 1 -- 172.21.15.114:0/2766489802 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5576e33f9620 con 0x5576e36c0800 2024-04-23T10:38:10.129 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:38:10.123+0000 7fe65d9fc640 1 --2- 172.21.15.114:0/2766489802 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x5576e36b5620 0x5576e36c01b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-23T10:38:10.129 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:38:10.123+0000 7fe65e1fd640 1 --2- 172.21.15.114:0/2766489802 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x5576e36c0800 0x5576e36bda40 secure :-1 s=READY pgs=875 cs=0 l=1 rev1=1 crypto rx=0x7fe6500092f0 tx=0x7fe6500127e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6ec0890309c5ddb7 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T10:38:10.129 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:38:10.123+0000 7fe6417fa640 10 client.0 ms_handle_connect on v2:172.21.15.114:3300/0 2024-04-23T10:38:10.129 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:38:10.123+0000 7fe6417fa640 1 -- 172.21.15.114:0/2766489802 <== mon.0 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe65002a030 con 0x5576e36c0800 2024-04-23T10:38:10.130 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:38:10.123+0000 7fe6417fa640 1 -- 172.21.15.114:0/2766489802 <== mon.0 v2:172.21.15.114:3300/0 2 ==== osd_map(105..105 src has 1..105) ==== 8673+0+0 (secure 0 0 0) 0x7fe65001b050 con 0x5576e36c0800 2024-04-23T10:38:10.130 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:38:10.123+0000 7fe662971740 1 -- 172.21.15.114:0/2766489802 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x5576e3628390 con 0x5576e36c0800 2024-04-23T10:38:10.130 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:38:10.123+0000 7fe6417fa640 1 -- 172.21.15.114:0/2766489802 <== mon.0 v2:172.21.15.114:3300/0 3 ==== mon_get_version_reply(handle=1 version=39) ==== 24+0+0 (secure 0 0 0) 0x7fe65000cb40 con 0x5576e36c0800 2024-04-23T10:38:10.130 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:38:10.123+0000 7fe662971740 10 client.16887 fetch_fsmap learned FSMap version 39 2024-04-23T10:38:10.130 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:38:10.123+0000 7fe662971740 1 -- 172.21.15.114:0/2766489802 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({config=0+,fsmap=39,monmap=4+}) -- 0x5576e36cf390 con 0x5576e36c0800 2024-04-23T10:38:10.130 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:38:10.124+0000 7fe6417fa640 1 -- 172.21.15.114:0/2766489802 <== mon.0 v2:172.21.15.114:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x5576e3628390 con 0x5576e36c0800 2024-04-23T10:38:10.130 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:38:10.124+0000 7fe6417fa640 1 -- 172.21.15.114:0/2766489802 <== mon.0 v2:172.21.15.114:3300/0 5 ==== fsmap(e 39) ==== 8350+0+0 (secure 0 0 0) 0x7fe6500314c0 con 0x5576e36c0800 2024-04-23T10:38:10.130 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:38:10.124+0000 7fe662971740 10 client.16887 fetch_fsmap finished waiting for FSMap version 39 2024-04-23T10:38:10.130 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:38:10.124+0000 7fe662971740 10 client.16887 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-04-23T10:38:10.130 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:38:10.124+0000 7fe662971740 20 client.16887 populate_metadata read hostname 'smithi114' 2024-04-23T10:38:10.130 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:38:10.124+0000 7fe662971740 1 --2- 172.21.15.114:0/2766489802 >> [v2:172.21.15.119:6838/937297340,v1:172.21.15.119:6839/937297340] conn(0x5576e36d0800 0x5576e36d2bf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T10:38:10.130 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:38:10.124+0000 7fe662971740 4 client.16887 mds_command: new command op to 24545 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-23T10:38:10.130 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:38:10.124+0000 7fe662971740 1 -- 172.21.15.114:0/2766489802 --> [v2:172.21.15.119:6838/937297340,v1:172.21.15.119:6839/937297340] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x5576e3622200 con 0x5576e36d0800 2024-04-23T10:38:10.131 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:38:10.124+0000 7fe65d9fc640 1 --2- 172.21.15.114:0/2766489802 >> [v2:172.21.15.119:6838/937297340,v1:172.21.15.119:6839/937297340] conn(0x5576e36d0800 0x5576e36d2bf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T10:38:10.131 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:38:10.124+0000 7fe6417fa640 10 client.16887 ms_handle_connect on v2:172.21.15.119:6838/937297340 2024-04-23T10:38:10.131 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:38:10.124+0000 7fe65d9fc640 1 --2- 172.21.15.114:0/2766489802 >> [v2:172.21.15.119:6838/937297340,v1:172.21.15.119:6839/937297340] conn(0x5576e36d0800 0x5576e36d2bf0 crc :-1 s=READY pgs=164 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=d14611ba5956f690 server_cookie=7b87bd212b069724 in_seq=0 out_seq=0 2024-04-23T10:38:10.131 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:38:10.125+0000 7fe6417fa640 1 -- 172.21.15.114:0/2766489802 <== mds.0 v2:172.21.15.119:6838/937297340 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x5576e3622200 con 0x5576e36d0800 2024-04-23T10:38:10.131 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:38:10.125+0000 7fe6417fa640 10 client.16887 handle_command_reply: tid=0 2024-04-23T10:38:10.131 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:38:10.125+0000 7fe662971740 1 client.16887 shutdown 2024-04-23T10:38:10.131 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:38:10.125+0000 7fe662971740 1 -- 172.21.15.114:0/2766489802 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x5576e36c0800 msgr2=0x5576e36bda40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T10:38:10.131 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:38:10.125+0000 7fe662971740 1 --2- 172.21.15.114:0/2766489802 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x5576e36c0800 0x5576e36bda40 secure :-1 s=READY pgs=875 cs=0 l=1 rev1=1 crypto rx=0x7fe6500092f0 tx=0x7fe6500127e0 comp rx=0 tx=0).stop 2024-04-23T10:38:10.131 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:38:10.125+0000 7fe662971740 1 -- 172.21.15.114:0/2766489802 shutdown_connections 2024-04-23T10:38:10.131 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:38:10.125+0000 7fe662971740 1 --2- 172.21.15.114:0/2766489802 >> [v2:172.21.15.119:6838/937297340,v1:172.21.15.119:6839/937297340] conn(0x5576e36d0800 0x5576e36d2bf0 crc :-1 s=READY pgs=164 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:38:10.131 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:38:10.125+0000 7fe662971740 1 --2- 172.21.15.114:0/2766489802 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x5576e36c0800 0x5576e36bda40 unknown :-1 s=CLOSED pgs=875 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:38:10.131 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:38:10.125+0000 7fe662971740 1 --2- 172.21.15.114:0/2766489802 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x5576e36b5f40 0x5576e36bd4a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:38:10.131 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:38:10.125+0000 7fe662971740 1 --2- 172.21.15.114:0/2766489802 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x5576e36b5620 0x5576e36c01b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:38:10.131 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:38:10.125+0000 7fe6417fa640 0 client.16887 ms_handle_reset on v2:172.21.15.119:6838/937297340 2024-04-23T10:38:10.132 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:38:10.125+0000 7fe662971740 1 -- 172.21.15.114:0/2766489802 >> 172.21.15.114:0/2766489802 conn(0x5576e36b1b30 msgr2=0x5576e36b7860 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T10:38:10.132 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:38:10.125+0000 7fe662971740 1 -- 172.21.15.114:0/2766489802 shutdown_connections 2024-04-23T10:38:10.132 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:38:10.125+0000 7fe662971740 1 -- 172.21.15.114:0/2766489802 wait complete. 2024-04-23T10:38:10.132 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:38:10.125+0000 7fe662971740 20 client.16887 trim_cache size 0 max 16384 2024-04-23T10:38:10.141 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:38:10.141+0000 7fe662971740 1 Processor -- start 2024-04-23T10:38:10.141 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:38:10.141+0000 7fe662971740 1 -- start start 2024-04-23T10:38:10.141 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:38:10.141+0000 7fe662971740 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x5576e36b5620 0x5576e36cc290 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T10:38:10.141 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:38:10.141+0000 7fe662971740 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x5576e36b5f40 0x5576e36bfce0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T10:38:10.141 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:38:10.141+0000 7fe662971740 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x5576e36c0800 0x5576e36d81c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T10:38:10.141 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:38:10.141+0000 7fe662971740 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x7fe6581b06b0 con 0x5576e36b5f40 2024-04-23T10:38:10.141 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:38:10.141+0000 7fe662971740 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 -- 0x7fe6581adef0 con 0x5576e36b5620 2024-04-23T10:38:10.141 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:38:10.141+0000 7fe662971740 1 -- --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_getmap magic: 0 -- 0x7fe6581abd60 con 0x5576e36c0800 2024-04-23T10:38:10.141 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:38:10.141+0000 7fe65d1fb640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x5576e36b5f40 0x5576e36bfce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T10:38:10.141 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:38:10.141+0000 7fe65d1fb640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x5576e36b5f40 0x5576e36bfce0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.114:52454/0 (socket says 172.21.15.114:52454) 2024-04-23T10:38:10.142 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:38:10.141+0000 7fe65d1fb640 1 -- 172.21.15.114:0/3984449427 learned_addr learned my addr 172.21.15.114:0/3984449427 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-04-23T10:38:10.142 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:38:10.141+0000 7fe65d9fc640 1 --2- 172.21.15.114:0/3984449427 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x5576e36b5620 0x5576e36cc290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T10:38:10.142 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:38:10.141+0000 7fe65e1fd640 1 --2- 172.21.15.114:0/3984449427 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x5576e36c0800 0x5576e36d81c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T10:38:10.142 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:38:10.142+0000 7fe65d9fc640 1 -- 172.21.15.114:0/3984449427 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x5576e36c0800 msgr2=0x5576e36d81c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T10:38:10.142 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:38:10.142+0000 7fe65d9fc640 1 --2- 172.21.15.114:0/3984449427 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x5576e36c0800 0x5576e36d81c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:38:10.142 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:38:10.142+0000 7fe65d9fc640 1 -- 172.21.15.114:0/3984449427 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x5576e36b5f40 msgr2=0x5576e36bfce0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T10:38:10.142 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:38:10.142+0000 7fe65d9fc640 1 --2- 172.21.15.114:0/3984449427 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x5576e36b5f40 0x5576e36bfce0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:38:10.142 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:38:10.142+0000 7fe65d9fc640 1 -- 172.21.15.114:0/3984449427 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5576e36b5d70 con 0x5576e36b5620 2024-04-23T10:38:10.142 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:38:10.142+0000 7fe65d1fb640 1 --2- 172.21.15.114:0/3984449427 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x5576e36b5f40 0x5576e36bfce0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-23T10:38:10.142 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:38:10.142+0000 7fe65d9fc640 1 --2- 172.21.15.114:0/3984449427 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x5576e36b5620 0x5576e36cc290 secure :-1 s=READY pgs=95 cs=0 l=1 rev1=1 crypto rx=0x7fe648009230 tx=0x7fe648004180 comp rx=0 tx=0).ready entity=mon.2 client_cookie=742a7975d0703d83 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T10:38:10.142 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:38:10.142+0000 7fe6437fe640 1 -- 172.21.15.114:0/3984449427 <== mon.2 v2:172.21.15.119:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe64804a070 con 0x5576e36b5620 2024-04-23T10:38:10.142 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:38:10.142+0000 7fe6437fe640 1 -- 172.21.15.114:0/3984449427 <== mon.2 v2:172.21.15.119:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fe6480044b0 con 0x5576e36b5620 2024-04-23T10:38:10.142 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:38:10.142+0000 7fe6437fe640 1 -- 172.21.15.114:0/3984449427 <== mon.2 v2:172.21.15.119:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe648035040 con 0x5576e36b5620 2024-04-23T10:38:10.146 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:38:10.143+0000 7fe662971740 1 -- 172.21.15.114:0/3984449427 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x5576e36b5620 msgr2=0x5576e36cc290 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T10:38:10.146 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:38:10.143+0000 7fe662971740 1 --2- 172.21.15.114:0/3984449427 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x5576e36b5620 0x5576e36cc290 secure :-1 s=READY pgs=95 cs=0 l=1 rev1=1 crypto rx=0x7fe648009230 tx=0x7fe648004180 comp rx=0 tx=0).stop 2024-04-23T10:38:10.146 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:38:10.143+0000 7fe662971740 1 -- 172.21.15.114:0/3984449427 shutdown_connections 2024-04-23T10:38:10.146 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:38:10.143+0000 7fe662971740 1 --2- 172.21.15.114:0/3984449427 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x5576e36c0800 0x5576e36d81c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:38:10.146 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:38:10.143+0000 7fe662971740 1 --2- 172.21.15.114:0/3984449427 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x5576e36b5f40 0x5576e36bfce0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:38:10.146 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:38:10.143+0000 7fe662971740 1 --2- 172.21.15.114:0/3984449427 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x5576e36b5620 0x5576e36cc290 unknown :-1 s=CLOSED pgs=95 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:38:10.146 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:38:10.143+0000 7fe662971740 1 -- 172.21.15.114:0/3984449427 >> 172.21.15.114:0/3984449427 conn(0x5576e36b1b30 msgr2=0x5576e36b0cb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T10:38:10.146 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:38:10.143+0000 7fe662971740 1 -- 172.21.15.114:0/3984449427 shutdown_connections 2024-04-23T10:38:10.146 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:38:10.143+0000 7fe662971740 1 -- 172.21.15.114:0/3984449427 wait complete. 2024-04-23T10:38:10.146 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:38:10.143+0000 7fe662971740 1 -- start start 2024-04-23T10:38:10.146 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:38:10.143+0000 7fe662971740 1 Processor -- start 2024-04-23T10:38:10.146 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:38:10.144+0000 7fe662971740 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x5576e36b5620 0x5576e36c42e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T10:38:10.147 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:38:10.144+0000 7fe662971740 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x5576e36b5f40 0x5576e36c4820 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T10:38:10.147 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:38:10.144+0000 7fe662971740 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x5576e36c0800 0x5576e36c4d60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T10:38:10.147 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:38:10.144+0000 7fe662971740 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x7fe6581ac3b0 con 0x5576e36b5f40 2024-04-23T10:38:10.147 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:38:10.144+0000 7fe662971740 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 -- 0x7fe6581ad820 con 0x5576e36b5620 2024-04-23T10:38:10.147 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:38:10.144+0000 7fe662971740 1 -- --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_getmap magic: 0 -- 0x7fe6581aca80 con 0x5576e36c0800 2024-04-23T10:38:10.147 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:38:10.144+0000 7fe662971740 10 client.0 _ll_register_callbacks cb 0x7fe650012a20 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-23T10:38:10.147 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:38:10.144+0000 7fe65d1fb640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x5576e36b5f40 0x5576e36c4820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T10:38:10.147 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:38:10.144+0000 7fe65d9fc640 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x5576e36b5620 0x5576e36c42e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T10:38:10.147 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:38:10.144+0000 7fe65d1fb640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x5576e36b5f40 0x5576e36c4820 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.114:52470/0 (socket says 172.21.15.114:52470) 2024-04-23T10:38:10.147 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:38:10.144+0000 7fe65d1fb640 1 -- 172.21.15.114:0/1603079327 learned_addr learned my addr 172.21.15.114:0/1603079327 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-04-23T10:38:10.147 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:38:10.144+0000 7fe65e1fd640 1 --2- 172.21.15.114:0/1603079327 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x5576e36c0800 0x5576e36c4d60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T10:38:10.147 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:38:10.144+0000 7fe65d9fc640 1 -- 172.21.15.114:0/1603079327 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x5576e36c0800 msgr2=0x5576e36c4d60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T10:38:10.147 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:38:10.144+0000 7fe65d9fc640 1 --2- 172.21.15.114:0/1603079327 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x5576e36c0800 0x5576e36c4d60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:38:10.147 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:38:10.144+0000 7fe65d9fc640 1 -- 172.21.15.114:0/1603079327 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x5576e36b5f40 msgr2=0x5576e36c4820 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T10:38:10.147 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:38:10.144+0000 7fe65d9fc640 1 --2- 172.21.15.114:0/1603079327 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x5576e36b5f40 0x5576e36c4820 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:38:10.148 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:38:10.144+0000 7fe65d9fc640 1 -- 172.21.15.114:0/1603079327 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5576e3735b60 con 0x5576e36b5620 2024-04-23T10:38:10.148 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:38:10.144+0000 7fe65d1fb640 1 --2- 172.21.15.114:0/1603079327 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x5576e36b5f40 0x5576e36c4820 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-23T10:38:10.148 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:38:10.144+0000 7fe65d9fc640 1 --2- 172.21.15.114:0/1603079327 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x5576e36b5620 0x5576e36c42e0 secure :-1 s=READY pgs=96 cs=0 l=1 rev1=1 crypto rx=0x7fe6480096c0 tx=0x7fe648009260 comp rx=0 tx=0).ready entity=mon.2 client_cookie=8ce2063d6289a255 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T10:38:10.148 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:38:10.144+0000 7fe6417fa640 10 client.0 ms_handle_connect on v2:172.21.15.119:3300/0 2024-04-23T10:38:10.148 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:38:10.144+0000 7fe6417fa640 1 -- 172.21.15.114:0/1603079327 <== mon.2 v2:172.21.15.119:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe648035070 con 0x5576e36b5620 2024-04-23T10:38:10.148 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:38:10.144+0000 7fe6417fa640 1 -- 172.21.15.114:0/1603079327 <== mon.2 v2:172.21.15.119:3300/0 2 ==== osd_map(105..105 src has 1..105) ==== 8673+0+0 (secure 0 0 0) 0x7fe648045ce0 con 0x5576e36b5620 2024-04-23T10:38:10.148 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:38:10.144+0000 7fe662971740 1 -- 172.21.15.114:0/1603079327 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fe6480044b0 con 0x5576e36b5620 2024-04-23T10:38:10.148 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:38:10.145+0000 7fe6417fa640 1 -- 172.21.15.114:0/1603079327 <== mon.2 v2:172.21.15.119:3300/0 3 ==== mon_get_version_reply(handle=1 version=39) ==== 24+0+0 (secure 0 0 0) 0x7fe64804a040 con 0x5576e36b5620 2024-04-23T10:38:10.148 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:38:10.145+0000 7fe662971740 10 client.26225 fetch_fsmap learned FSMap version 39 2024-04-23T10:38:10.148 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:38:10.145+0000 7fe662971740 1 -- 172.21.15.114:0/1603079327 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({config=0+,fsmap=39,monmap=4+}) -- 0x5576e3736e50 con 0x5576e36b5620 2024-04-23T10:38:10.148 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:38:10.145+0000 7fe6417fa640 1 -- 172.21.15.114:0/1603079327 <== mon.2 v2:172.21.15.119:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fe6480044b0 con 0x5576e36b5620 2024-04-23T10:38:10.148 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:38:10.145+0000 7fe6417fa640 1 -- 172.21.15.114:0/1603079327 <== mon.2 v2:172.21.15.119:3300/0 5 ==== fsmap(e 39) ==== 8350+0+0 (secure 0 0 0) 0x7fe648056030 con 0x5576e36b5620 2024-04-23T10:38:10.148 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:38:10.145+0000 7fe662971740 10 client.26225 fetch_fsmap finished waiting for FSMap version 39 2024-04-23T10:38:10.148 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:38:10.145+0000 7fe662971740 10 client.26225 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-04-23T10:38:10.148 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:38:10.145+0000 7fe662971740 20 client.26225 populate_metadata read hostname 'smithi114' 2024-04-23T10:38:10.149 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:38:10.145+0000 7fe662971740 1 --2- 172.21.15.114:0/1603079327 >> [v2:172.21.15.119:6838/937297340,v1:172.21.15.119:6839/937297340] conn(0x5576e37375f0 0x5576e3739a10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T10:38:10.149 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:38:10.145+0000 7fe662971740 4 client.26225 mds_command: new command op to 24545 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-04-23T10:38:10.149 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:38:10.145+0000 7fe662971740 1 -- 172.21.15.114:0/1603079327 --> [v2:172.21.15.119:6838/937297340,v1:172.21.15.119:6839/937297340] -- command(tid 0: {"prefix": "flush journal"}) -- 0x5576e359e5f0 con 0x5576e37375f0 2024-04-23T10:38:10.149 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:38:10.145+0000 7fe65d1fb640 1 --2- 172.21.15.114:0/1603079327 >> [v2:172.21.15.119:6838/937297340,v1:172.21.15.119:6839/937297340] conn(0x5576e37375f0 0x5576e3739a10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T10:38:10.149 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:38:10.145+0000 7fe6417fa640 10 client.26225 ms_handle_connect on v2:172.21.15.119:6838/937297340 2024-04-23T10:38:10.149 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:38:10.145+0000 7fe65d1fb640 1 --2- 172.21.15.114:0/1603079327 >> [v2:172.21.15.119:6838/937297340,v1:172.21.15.119:6839/937297340] conn(0x5576e37375f0 0x5576e3739a10 crc :-1 s=READY pgs=165 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=6b6b8d2096502353 server_cookie=54f5486392f33501 in_seq=0 out_seq=0 2024-04-23T10:38:10.203 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:38:10.204+0000 7fe6417fa640 1 -- 172.21.15.114:0/1603079327 <== mds.0 v2:172.21.15.119:6838/937297340 1 ==== command_reply(tid 0: 0 ) ==== 8+0+44 (crc 0 0 0) 0x5576e359e5f0 con 0x5576e37375f0 2024-04-23T10:38:10.203 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:38:10.204+0000 7fe6417fa640 10 client.26225 handle_command_reply: tid=0 2024-04-23T10:38:10.203 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:38:10.204+0000 7fe662971740 1 client.26225 shutdown 2024-04-23T10:38:10.203 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:38:10.204+0000 7fe662971740 1 -- 172.21.15.114:0/1603079327 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x5576e36b5620 msgr2=0x5576e36c42e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T10:38:10.203 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:38:10.204+0000 7fe662971740 1 --2- 172.21.15.114:0/1603079327 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x5576e36b5620 0x5576e36c42e0 secure :-1 s=READY pgs=96 cs=0 l=1 rev1=1 crypto rx=0x7fe6480096c0 tx=0x7fe648009260 comp rx=0 tx=0).stop 2024-04-23T10:38:10.203 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:38:10.204+0000 7fe662971740 1 -- 172.21.15.114:0/1603079327 shutdown_connections 2024-04-23T10:38:10.203 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:38:10.204+0000 7fe662971740 1 --2- 172.21.15.114:0/1603079327 >> [v2:172.21.15.119:6838/937297340,v1:172.21.15.119:6839/937297340] conn(0x5576e37375f0 0x5576e3739a10 crc :-1 s=READY pgs=165 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:38:10.203 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:38:10.204+0000 7fe662971740 1 --2- 172.21.15.114:0/1603079327 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x5576e36c0800 0x5576e36c4d60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:38:10.203 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:38:10.204+0000 7fe662971740 1 --2- 172.21.15.114:0/1603079327 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x5576e36b5f40 0x5576e36c4820 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:38:10.204 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:38:10.204+0000 7fe662971740 1 --2- 172.21.15.114:0/1603079327 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x5576e36b5620 0x5576e36c42e0 unknown :-1 s=CLOSED pgs=96 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:38:10.204 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:38:10.204+0000 7fe6417fa640 0 client.26225 ms_handle_reset on v2:172.21.15.119:6838/937297340 2024-04-23T10:38:10.204 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:38:10.204+0000 7fe662971740 1 -- 172.21.15.114:0/1603079327 >> 172.21.15.114:0/1603079327 conn(0x5576e36b1b30 msgr2=0x5576e36cb280 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T10:38:10.204 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:38:10.204+0000 7fe662971740 1 -- 172.21.15.114:0/1603079327 shutdown_connections 2024-04-23T10:38:10.204 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:38:10.204+0000 7fe662971740 1 -- 172.21.15.114:0/1603079327 wait complete. 2024-04-23T10:38:10.204 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:38:10.204+0000 7fe662971740 20 client.26225 trim_cache size 0 max 16384 2024-04-23T10:38:10.204 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stdout:{ 2024-04-23T10:38:10.204 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stdout: "message": "", 2024-04-23T10:38:10.204 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stdout: "return_code": 0 2024-04-23T10:38:10.205 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stdout:} 2024-04-23T10:38:10.205 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:38:10.204+0000 7fe65fc60640 1 -- 172.21.15.114:0/1449004791 >> [v2:172.21.15.114:6800/1395693466,v1:172.21.15.114:6801/1395693466] conn(0x7fe6440a3dc0 msgr2=0x7fe6440a6280 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T10:38:10.205 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:38:10.204+0000 7fe65fc60640 1 --2- 172.21.15.114:0/1449004791 >> [v2:172.21.15.114:6800/1395693466,v1:172.21.15.114:6801/1395693466] conn(0x7fe6440a3dc0 0x7fe6440a6280 secure :-1 s=READY pgs=136 cs=0 l=1 rev1=1 crypto rx=0x7fe650010c40 tx=0x7fe650017000 comp rx=0 tx=0).stop 2024-04-23T10:38:10.206 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:38:10.204+0000 7fe65fc60640 1 -- 172.21.15.114:0/1449004791 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fe658107290 msgr2=0x7fe658198c20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T10:38:10.206 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:38:10.204+0000 7fe65fc60640 1 --2- 172.21.15.114:0/1449004791 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fe658107290 0x7fe658198c20 secure :-1 s=READY pgs=873 cs=0 l=1 rev1=1 crypto rx=0x7fe65400d770 tx=0x7fe65400dc30 comp rx=0 tx=0).stop 2024-04-23T10:38:10.206 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:38:10.204+0000 7fe65fc60640 1 -- 172.21.15.114:0/1449004791 shutdown_connections 2024-04-23T10:38:10.206 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:38:10.204+0000 7fe65fc60640 1 --2- 172.21.15.114:0/1449004791 >> [v2:172.21.15.114:6800/1395693466,v1:172.21.15.114:6801/1395693466] conn(0x7fe6440a3dc0 0x7fe6440a6280 unknown :-1 s=CLOSED pgs=136 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:38:10.206 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:38:10.204+0000 7fe65fc60640 1 --2- 172.21.15.114:0/1449004791 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fe65819cfa0 0x7fe65819d450 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:38:10.206 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:38:10.204+0000 7fe65fc60640 1 --2- 172.21.15.114:0/1449004791 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fe658107290 0x7fe658198c20 unknown :-1 s=CLOSED pgs=873 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:38:10.206 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:38:10.204+0000 7fe65fc60640 1 --2- 172.21.15.114:0/1449004791 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fe6581021c0 0x7fe6581986e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:38:10.206 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:38:10.205+0000 7fe65fc60640 1 -- 172.21.15.114:0/1449004791 >> 172.21.15.114:0/1449004791 conn(0x7fe6580fda00 msgr2=0x7fe6580fee20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T10:38:10.206 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:38:10.205+0000 7fe65fc60640 1 -- 172.21.15.114:0/1449004791 shutdown_connections 2024-04-23T10:38:10.206 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:38:10.205+0000 7fe65fc60640 1 -- 172.21.15.114:0/1449004791 wait complete. 2024-04-23T10:38:10.945 INFO:tasks.workunit.client.0.smithi114.stdout: 1 685539 5.52 MB/sec execute 499 sec latency 1675.609 ms 2024-04-23T10:38:11.945 INFO:tasks.workunit.client.0.smithi114.stdout: 1 685539 5.51 MB/sec execute 500 sec latency 2675.736 ms 2024-04-23T10:38:11.992 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 23 10:38:11 smithi119 ceph-mon[32608]: pgmap v363: 97 pgs: 97 active+clean; 214 MiB data, 2.7 GiB used, 1.0 TiB / 1.0 TiB avail; 339 KiB/s rd, 917 KiB/s wr, 130 op/s 2024-04-23T10:38:12.004 INFO:journalctl@ceph.mon.a.smithi114.stdout:Apr 23 10:38:11 smithi114 ceph-mon[29255]: pgmap v363: 97 pgs: 97 active+clean; 214 MiB data, 2.7 GiB used, 1.0 TiB / 1.0 TiB avail; 339 KiB/s rd, 917 KiB/s wr, 130 op/s 2024-04-23T10:38:12.078 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 23 10:38:11 smithi204 ceph-mon[32649]: pgmap v363: 97 pgs: 97 active+clean; 214 MiB data, 2.7 GiB used, 1.0 TiB / 1.0 TiB avail; 339 KiB/s rd, 917 KiB/s wr, 130 op/s 2024-04-23T10:38:12.945 INFO:tasks.workunit.client.0.smithi114.stdout: 1 685539 5.50 MB/sec execute 501 sec latency 3675.793 ms 2024-04-23T10:38:13.945 INFO:tasks.workunit.client.0.smithi114.stdout: 1 685539 5.49 MB/sec execute 502 sec latency 4675.892 ms 2024-04-23T10:38:13.992 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 23 10:38:13 smithi119 ceph-mon[32608]: pgmap v364: 97 pgs: 97 active+clean; 210 MiB data, 2.7 GiB used, 1.0 TiB / 1.0 TiB avail; 425 KiB/s rd, 947 KiB/s wr, 124 op/s 2024-04-23T10:38:13.992 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 23 10:38:13 smithi119 ceph-mon[32608]: osdmap e106: 12 total, 12 up, 12 in 2024-04-23T10:38:14.004 INFO:journalctl@ceph.mon.a.smithi114.stdout:Apr 23 10:38:13 smithi114 ceph-mon[29255]: pgmap v364: 97 pgs: 97 active+clean; 210 MiB data, 2.7 GiB used, 1.0 TiB / 1.0 TiB avail; 425 KiB/s rd, 947 KiB/s wr, 124 op/s 2024-04-23T10:38:14.004 INFO:journalctl@ceph.mon.a.smithi114.stdout:Apr 23 10:38:13 smithi114 ceph-mon[29255]: osdmap e106: 12 total, 12 up, 12 in 2024-04-23T10:38:14.078 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 23 10:38:13 smithi204 ceph-mon[32649]: pgmap v364: 97 pgs: 97 active+clean; 210 MiB data, 2.7 GiB used, 1.0 TiB / 1.0 TiB avail; 425 KiB/s rd, 947 KiB/s wr, 124 op/s 2024-04-23T10:38:14.078 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 23 10:38:13 smithi204 ceph-mon[32649]: osdmap e106: 12 total, 12 up, 12 in 2024-04-23T10:38:14.945 INFO:tasks.workunit.client.0.smithi114.stdout: 1 686226 5.48 MB/sec execute 503 sec latency 4919.823 ms 2024-04-23T10:38:14.992 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 23 10:38:14 smithi119 ceph-mon[32608]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=e=up:active} 3 up:standby-replay 6 up:standby 2024-04-23T10:38:15.004 INFO:journalctl@ceph.mon.a.smithi114.stdout:Apr 23 10:38:14 smithi114 ceph-mon[29255]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=e=up:active} 3 up:standby-replay 6 up:standby 2024-04-23T10:38:15.077 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 23 10:38:14 smithi204 ceph-mon[32649]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=e=up:active} 3 up:standby-replay 6 up:standby 2024-04-23T10:38:15.946 INFO:tasks.workunit.client.0.smithi114.stdout: 1 688573 5.49 MB/sec execute 504 sec latency 16.183 ms 2024-04-23T10:38:15.992 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 23 10:38:15 smithi119 ceph-mon[32608]: pgmap v366: 97 pgs: 97 active+clean; 210 MiB data, 2.7 GiB used, 1.0 TiB / 1.0 TiB avail; 214 KiB/s rd, 428 KiB/s wr, 43 op/s 2024-04-23T10:38:16.004 INFO:journalctl@ceph.mon.a.smithi114.stdout:Apr 23 10:38:15 smithi114 ceph-mon[29255]: pgmap v366: 97 pgs: 97 active+clean; 210 MiB data, 2.7 GiB used, 1.0 TiB / 1.0 TiB avail; 214 KiB/s rd, 428 KiB/s wr, 43 op/s 2024-04-23T10:38:16.077 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 23 10:38:15 smithi204 ceph-mon[32649]: pgmap v366: 97 pgs: 97 active+clean; 210 MiB data, 2.7 GiB used, 1.0 TiB / 1.0 TiB avail; 214 KiB/s rd, 428 KiB/s wr, 43 op/s 2024-04-23T10:38:16.946 INFO:tasks.workunit.client.0.smithi114.stdout: 1 689016 5.49 MB/sec execute 505 sec latency 807.599 ms 2024-04-23T10:38:17.946 INFO:tasks.workunit.client.0.smithi114.stdout: 1 689016 5.48 MB/sec execute 506 sec latency 1807.684 ms 2024-04-23T10:38:17.991 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 23 10:38:17 smithi119 ceph-mon[32608]: pgmap v367: 97 pgs: 97 active+clean; 212 MiB data, 2.7 GiB used, 1.0 TiB / 1.0 TiB avail; 584 KiB/s rd, 981 KiB/s wr, 101 op/s 2024-04-23T10:38:18.003 INFO:journalctl@ceph.mon.a.smithi114.stdout:Apr 23 10:38:17 smithi114 ceph-mon[29255]: pgmap v367: 97 pgs: 97 active+clean; 212 MiB data, 2.7 GiB used, 1.0 TiB / 1.0 TiB avail; 584 KiB/s rd, 981 KiB/s wr, 101 op/s 2024-04-23T10:38:18.077 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 23 10:38:17 smithi204 ceph-mon[32649]: pgmap v367: 97 pgs: 97 active+clean; 212 MiB data, 2.7 GiB used, 1.0 TiB / 1.0 TiB avail; 584 KiB/s rd, 981 KiB/s wr, 101 op/s 2024-04-23T10:38:18.946 INFO:tasks.workunit.client.0.smithi114.stdout: 1 689016 5.47 MB/sec execute 507 sec latency 2807.778 ms 2024-04-23T10:38:19.946 INFO:tasks.workunit.client.0.smithi114.stdout: 1 689083 5.46 MB/sec execute 508 sec latency 3050.675 ms 2024-04-23T10:38:19.991 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 23 10:38:19 smithi119 ceph-mon[32608]: pgmap v368: 97 pgs: 97 active+clean; 212 MiB data, 2.7 GiB used, 1.0 TiB / 1.0 TiB avail; 476 KiB/s rd, 800 KiB/s wr, 83 op/s 2024-04-23T10:38:20.003 INFO:journalctl@ceph.mon.a.smithi114.stdout:Apr 23 10:38:19 smithi114 ceph-mon[29255]: pgmap v368: 97 pgs: 97 active+clean; 212 MiB data, 2.7 GiB used, 1.0 TiB / 1.0 TiB avail; 476 KiB/s rd, 800 KiB/s wr, 83 op/s 2024-04-23T10:38:20.077 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 23 10:38:19 smithi204 ceph-mon[32649]: pgmap v368: 97 pgs: 97 active+clean; 212 MiB data, 2.7 GiB used, 1.0 TiB / 1.0 TiB avail; 476 KiB/s rd, 800 KiB/s wr, 83 op/s 2024-04-23T10:38:20.946 INFO:tasks.workunit.client.0.smithi114.stdout: 1 689083 5.45 MB/sec execute 509 sec latency 1655.208 ms 2024-04-23T10:38:21.946 INFO:tasks.workunit.client.0.smithi114.stdout: 1 689083 5.44 MB/sec execute 510 sec latency 2655.301 ms 2024-04-23T10:38:21.992 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 23 10:38:21 smithi119 ceph-mon[32608]: pgmap v369: 97 pgs: 97 active+clean; 212 MiB data, 2.7 GiB used, 1.0 TiB / 1.0 TiB avail; 467 KiB/s rd, 785 KiB/s wr, 81 op/s 2024-04-23T10:38:22.004 INFO:journalctl@ceph.mon.a.smithi114.stdout:Apr 23 10:38:21 smithi114 ceph-mon[29255]: pgmap v369: 97 pgs: 97 active+clean; 212 MiB data, 2.7 GiB used, 1.0 TiB / 1.0 TiB avail; 467 KiB/s rd, 785 KiB/s wr, 81 op/s 2024-04-23T10:38:22.077 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 23 10:38:21 smithi204 ceph-mon[32649]: pgmap v369: 97 pgs: 97 active+clean; 212 MiB data, 2.7 GiB used, 1.0 TiB / 1.0 TiB avail; 467 KiB/s rd, 785 KiB/s wr, 81 op/s 2024-04-23T10:38:22.946 INFO:tasks.workunit.client.0.smithi114.stdout: 1 689083 5.43 MB/sec execute 511 sec latency 3655.360 ms 2024-04-23T10:38:23.300 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:38:23.298+0000 7f5a37864640 1 -- 172.21.15.114:0/2545276122 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f5a301006c0 msgr2=0x7f5a30100aa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T10:38:23.300 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:38:23.298+0000 7f5a37864640 1 --2- 172.21.15.114:0/2545276122 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f5a301006c0 0x7f5a30100aa0 secure :-1 s=READY pgs=876 cs=0 l=1 rev1=1 crypto rx=0x7f5a2c009900 tx=0x7f5a2c02a410 comp rx=0 tx=0).stop 2024-04-23T10:38:23.300 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:38:23.298+0000 7f5a37864640 1 -- 172.21.15.114:0/2545276122 shutdown_connections 2024-04-23T10:38:23.300 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:38:23.298+0000 7f5a37864640 1 --2- 172.21.15.114:0/2545276122 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f5a30107280 0x7f5a3010ee50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:38:23.300 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:38:23.298+0000 7f5a37864640 1 --2- 172.21.15.114:0/2545276122 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f5a30101070 0x7f5a30106ba0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:38:23.300 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:38:23.298+0000 7f5a37864640 1 --2- 172.21.15.114:0/2545276122 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f5a301006c0 0x7f5a30100aa0 unknown :-1 s=CLOSED pgs=876 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:38:23.301 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:38:23.298+0000 7f5a37864640 1 -- 172.21.15.114:0/2545276122 >> 172.21.15.114:0/2545276122 conn(0x7f5a30077390 msgr2=0x7f5a300fe760 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T10:38:23.301 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:38:23.298+0000 7f5a37864640 1 -- 172.21.15.114:0/2545276122 shutdown_connections 2024-04-23T10:38:23.301 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:38:23.298+0000 7f5a37864640 1 -- 172.21.15.114:0/2545276122 wait complete. 2024-04-23T10:38:23.301 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:38:23.298+0000 7f5a37864640 1 Processor -- start 2024-04-23T10:38:23.301 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:38:23.298+0000 7f5a37864640 1 -- start start 2024-04-23T10:38:23.301 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:38:23.299+0000 7f5a37864640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f5a301006c0 0x7f5a30198730 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T10:38:23.301 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:38:23.299+0000 7f5a37864640 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f5a30107280 0x7f5a30198c70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T10:38:23.301 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:38:23.299+0000 7f5a37864640 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f5a3019cfa0 0x7f5a3019d450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T10:38:23.301 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:38:23.299+0000 7f5a37864640 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x7f5a301121f0 con 0x7f5a301006c0 2024-04-23T10:38:23.302 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:38:23.299+0000 7f5a37864640 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 -- 0x7f5a30112070 con 0x7f5a30107280 2024-04-23T10:38:23.302 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:38:23.299+0000 7f5a37864640 1 -- --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_getmap magic: 0 -- 0x7f5a30112370 con 0x7f5a3019cfa0 2024-04-23T10:38:23.302 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:38:23.299+0000 7f5a35600640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f5a301006c0 0x7f5a30198730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T10:38:23.302 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:38:23.299+0000 7f5a35600640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f5a301006c0 0x7f5a30198730 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.114:46766/0 (socket says 172.21.15.114:46766) 2024-04-23T10:38:23.302 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:38:23.299+0000 7f5a35600640 1 -- 172.21.15.114:0/4235522242 learned_addr learned my addr 172.21.15.114:0/4235522242 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-04-23T10:38:23.302 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:38:23.299+0000 7f5a34dff640 1 --2- 172.21.15.114:0/4235522242 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f5a30107280 0x7f5a30198c70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T10:38:23.302 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:38:23.299+0000 7f5a35e01640 1 --2- 172.21.15.114:0/4235522242 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f5a3019cfa0 0x7f5a3019d450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T10:38:23.302 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:38:23.299+0000 7f5a35600640 1 -- 172.21.15.114:0/4235522242 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f5a3019cfa0 msgr2=0x7f5a3019d450 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T10:38:23.302 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:38:23.299+0000 7f5a35600640 1 --2- 172.21.15.114:0/4235522242 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f5a3019cfa0 0x7f5a3019d450 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:38:23.303 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:38:23.299+0000 7f5a35600640 1 -- 172.21.15.114:0/4235522242 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f5a30107280 msgr2=0x7f5a30198c70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T10:38:23.303 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:38:23.299+0000 7f5a35600640 1 --2- 172.21.15.114:0/4235522242 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f5a30107280 0x7f5a30198c70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:38:23.303 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:38:23.299+0000 7f5a35600640 1 -- 172.21.15.114:0/4235522242 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f5a3019da90 con 0x7f5a301006c0 2024-04-23T10:38:23.303 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:38:23.299+0000 7f5a35600640 1 --2- 172.21.15.114:0/4235522242 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f5a301006c0 0x7f5a30198730 secure :-1 s=READY pgs=877 cs=0 l=1 rev1=1 crypto rx=0x7f5a2c005940 tx=0x7f5a2c004270 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T10:38:23.303 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:38:23.299+0000 7f5a267fc640 1 -- 172.21.15.114:0/4235522242 <== mon.0 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5a2c03f030 con 0x7f5a301006c0 2024-04-23T10:38:23.303 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:38:23.299+0000 7f5a267fc640 1 -- 172.21.15.114:0/4235522242 <== mon.0 v2:172.21.15.114:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f5a2c003aa0 con 0x7f5a301006c0 2024-04-23T10:38:23.303 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:38:23.299+0000 7f5a37864640 1 -- 172.21.15.114:0/4235522242 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f5a3019dd20 con 0x7f5a301006c0 2024-04-23T10:38:23.303 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:38:23.299+0000 7f5a267fc640 1 -- 172.21.15.114:0/4235522242 <== mon.0 v2:172.21.15.114:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5a2c035070 con 0x7f5a301006c0 2024-04-23T10:38:23.303 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:38:23.299+0000 7f5a37864640 1 -- 172.21.15.114:0/4235522242 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f5a301a54e0 con 0x7f5a301006c0 2024-04-23T10:38:23.313 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:38:23.302+0000 7f5a267fc640 1 -- 172.21.15.114:0/4235522242 <== mon.0 v2:172.21.15.114:3300/0 4 ==== mgrmap(e 26) ==== 138778+0+0 (secure 0 0 0) 0x7f5a2c042030 con 0x7f5a301006c0 2024-04-23T10:38:23.313 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:38:23.302+0000 7f5a3a579740 1 Processor -- start 2024-04-23T10:38:23.313 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:38:23.302+0000 7f5a3a579740 1 -- start start 2024-04-23T10:38:23.313 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:38:23.303+0000 7f5a3a579740 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55de933f7620 0x55de933f7a00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T10:38:23.313 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:38:23.303+0000 7f5a3a579740 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x55de933f7f40 0x55de934022c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T10:38:23.313 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:38:23.303+0000 7f5a3a579740 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x55de93402800 0x55de93404bf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T10:38:23.313 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:38:23.303+0000 7f5a3a579740 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x55de9335c970 con 0x55de933f7620 2024-04-23T10:38:23.313 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:38:23.303+0000 7f5a267fc640 1 --2- 172.21.15.114:0/4235522242 >> [v2:172.21.15.114:6800/1395693466,v1:172.21.15.114:6801/1395693466] conn(0x7f5a1c0a3dc0 0x7f5a1c0a6280 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T10:38:23.314 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:38:23.303+0000 7f5a3a579740 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 -- 0x55de932f94a0 con 0x55de933f7f40 2024-04-23T10:38:23.314 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:38:23.303+0000 7f5a3a579740 1 -- --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_getmap magic: 0 -- 0x55de93358930 con 0x55de93402800 2024-04-23T10:38:23.314 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:38:23.303+0000 7f5a35e01640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55de933f7620 0x55de933f7a00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T10:38:23.314 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:38:23.303+0000 7f5a35e01640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55de933f7620 0x55de933f7a00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.114:46774/0 (socket says 172.21.15.114:46774) 2024-04-23T10:38:23.314 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:38:23.303+0000 7f5a35e01640 1 -- 172.21.15.114:0/3049521195 learned_addr learned my addr 172.21.15.114:0/3049521195 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-04-23T10:38:23.314 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:38:23.303+0000 7f5a35e01640 1 --2- 172.21.15.114:0/4235522242 >> [v2:172.21.15.114:6800/1395693466,v1:172.21.15.114:6801/1395693466] conn(0x7f5a1c0a3dc0 0x7f5a1c0a6280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T10:38:23.314 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:38:23.303+0000 7f5a267fc640 1 -- 172.21.15.114:0/4235522242 <== mon.0 v2:172.21.15.114:3300/0 5 ==== osd_map(106..106 src has 1..106) ==== 8629+0+0 (secure 0 0 0) 0x7f5a2c0f0dd0 con 0x7f5a301006c0 2024-04-23T10:38:23.314 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:38:23.303+0000 7f5a34dff640 1 --2- 172.21.15.114:0/3049521195 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x55de93402800 0x55de93404bf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T10:38:23.315 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:38:23.303+0000 7f5a35600640 1 --2- 172.21.15.114:0/3049521195 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x55de933f7f40 0x55de934022c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T10:38:23.315 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:38:23.303+0000 7f5a35e01640 1 -- 172.21.15.114:0/3049521195 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x55de93402800 msgr2=0x55de93404bf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T10:38:23.315 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:38:23.303+0000 7f5a35e01640 1 --2- 172.21.15.114:0/3049521195 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x55de93402800 0x55de93404bf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:38:23.315 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:38:23.303+0000 7f5a35e01640 1 -- 172.21.15.114:0/3049521195 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x55de933f7f40 msgr2=0x55de934022c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T10:38:23.315 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:38:23.303+0000 7f5a35e01640 1 --2- 172.21.15.114:0/3049521195 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x55de933f7f40 0x55de934022c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:38:23.315 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:38:23.303+0000 7f5a35e01640 1 -- 172.21.15.114:0/3049521195 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55de9306f210 con 0x55de933f7620 2024-04-23T10:38:23.315 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:38:23.303+0000 7f5a34dff640 1 --2- 172.21.15.114:0/3049521195 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x55de93402800 0x55de93404bf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-23T10:38:23.315 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:38:23.303+0000 7f5a35e01640 1 --2- 172.21.15.114:0/4235522242 >> [v2:172.21.15.114:6800/1395693466,v1:172.21.15.114:6801/1395693466] conn(0x7f5a1c0a3dc0 0x7f5a1c0a6280 secure :-1 s=READY pgs=137 cs=0 l=1 rev1=1 crypto rx=0x7f5a28012b80 tx=0x7f5a280099b0 comp rx=0 tx=0).ready entity=mgr.15291 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T10:38:23.315 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:38:23.303+0000 7f5a35e01640 1 --2- 172.21.15.114:0/3049521195 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55de933f7620 0x55de933f7a00 secure :-1 s=READY pgs=878 cs=0 l=1 rev1=1 crypto rx=0x7f5a28012850 tx=0x7f5a2801c000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f9c6d38f71bf6853 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T10:38:23.315 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:38:23.304+0000 7f5a1b7fe640 1 -- 172.21.15.114:0/3049521195 <== mon.0 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5a28024070 con 0x55de933f7620 2024-04-23T10:38:23.316 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:38:23.304+0000 7f5a1b7fe640 1 -- 172.21.15.114:0/3049521195 <== mon.0 v2:172.21.15.114:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f5a2800db90 con 0x55de933f7620 2024-04-23T10:38:23.316 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:38:23.304+0000 7f5a1b7fe640 1 -- 172.21.15.114:0/3049521195 <== mon.0 v2:172.21.15.114:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5a280209d0 con 0x55de933f7620 2024-04-23T10:38:23.316 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:38:23.304+0000 7f5a3a579740 1 -- 172.21.15.114:0/3049521195 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55de933f7620 msgr2=0x55de933f7a00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T10:38:23.316 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:38:23.304+0000 7f5a3a579740 1 --2- 172.21.15.114:0/3049521195 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55de933f7620 0x55de933f7a00 secure :-1 s=READY pgs=878 cs=0 l=1 rev1=1 crypto rx=0x7f5a28012850 tx=0x7f5a2801c000 comp rx=0 tx=0).stop 2024-04-23T10:38:23.316 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:38:23.304+0000 7f5a3a579740 1 -- 172.21.15.114:0/3049521195 shutdown_connections 2024-04-23T10:38:23.316 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:38:23.304+0000 7f5a3a579740 1 --2- 172.21.15.114:0/3049521195 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x55de93402800 0x55de93404bf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:38:23.316 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:38:23.304+0000 7f5a3a579740 1 --2- 172.21.15.114:0/3049521195 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x55de933f7f40 0x55de934022c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:38:23.316 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:38:23.304+0000 7f5a3a579740 1 --2- 172.21.15.114:0/3049521195 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55de933f7620 0x55de933f7a00 unknown :-1 s=CLOSED pgs=878 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:38:23.316 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:38:23.304+0000 7f5a3a579740 1 -- 172.21.15.114:0/3049521195 >> 172.21.15.114:0/3049521195 conn(0x55de933f3b30 msgr2=0x55de933f3f10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T10:38:23.316 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:38:23.304+0000 7f5a3a579740 1 -- 172.21.15.114:0/3049521195 shutdown_connections 2024-04-23T10:38:23.316 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:38:23.304+0000 7f5a3a579740 1 -- 172.21.15.114:0/3049521195 wait complete. 2024-04-23T10:38:23.317 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:38:23.305+0000 7f5a3a579740 1 -- start start 2024-04-23T10:38:23.317 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:38:23.305+0000 7f5a3a579740 1 Processor -- start 2024-04-23T10:38:23.317 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:38:23.305+0000 7f5a3a579740 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x55de933f7620 0x55de93402160 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T10:38:23.317 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:38:23.305+0000 7f5a3a579740 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55de933f7f40 0x55de933ff450 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T10:38:23.317 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:38:23.305+0000 7f5a3a579740 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x55de93402800 0x55de933ff9b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T10:38:23.317 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:38:23.305+0000 7f5a3a579740 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x55de934053d0 con 0x55de933f7f40 2024-04-23T10:38:23.317 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:38:23.305+0000 7f5a3a579740 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 -- 0x55de932f70b0 con 0x55de933f7620 2024-04-23T10:38:23.317 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:38:23.305+0000 7f5a3a579740 1 -- --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_getmap magic: 0 -- 0x55de93405550 con 0x55de93402800 2024-04-23T10:38:23.317 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:38:23.305+0000 7f5a3a579740 10 client.0 _ll_register_callbacks cb 0x55de932e4d30 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-23T10:38:23.317 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:38:23.305+0000 7f5a34dff640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55de933f7f40 0x55de933ff450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T10:38:23.318 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:38:23.305+0000 7f5a34dff640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55de933f7f40 0x55de933ff450 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.114:46788/0 (socket says 172.21.15.114:46788) 2024-04-23T10:38:23.318 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:38:23.305+0000 7f5a34dff640 1 -- 172.21.15.114:0/3218000872 learned_addr learned my addr 172.21.15.114:0/3218000872 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-04-23T10:38:23.318 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:38:23.305+0000 7f5a35e01640 1 --2- 172.21.15.114:0/3218000872 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x55de93402800 0x55de933ff9b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T10:38:23.318 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:38:23.306+0000 7f5a35600640 1 --2- 172.21.15.114:0/3218000872 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x55de933f7620 0x55de93402160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T10:38:23.318 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:38:23.306+0000 7f5a34dff640 1 -- 172.21.15.114:0/3218000872 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x55de93402800 msgr2=0x55de933ff9b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T10:38:23.318 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:38:23.306+0000 7f5a34dff640 1 --2- 172.21.15.114:0/3218000872 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x55de93402800 0x55de933ff9b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:38:23.318 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:38:23.306+0000 7f5a34dff640 1 -- 172.21.15.114:0/3218000872 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x55de933f7620 msgr2=0x55de93402160 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T10:38:23.318 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:38:23.306+0000 7f5a34dff640 1 --2- 172.21.15.114:0/3218000872 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x55de933f7620 0x55de93402160 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:38:23.318 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:38:23.306+0000 7f5a34dff640 1 -- 172.21.15.114:0/3218000872 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55de9313b620 con 0x55de933f7f40 2024-04-23T10:38:23.319 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:38:23.306+0000 7f5a34dff640 1 --2- 172.21.15.114:0/3218000872 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55de933f7f40 0x55de933ff450 secure :-1 s=READY pgs=879 cs=0 l=1 rev1=1 crypto rx=0x7f5a200126e0 tx=0x7f5a20012ae0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=76e9577bc73a46 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T10:38:23.319 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:38:23.306+0000 7f5a18ff9640 10 client.0 ms_handle_connect on v2:172.21.15.114:3300/0 2024-04-23T10:38:23.319 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:38:23.306+0000 7f5a18ff9640 1 -- 172.21.15.114:0/3218000872 <== mon.0 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5a20004030 con 0x55de933f7f40 2024-04-23T10:38:23.319 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:38:23.306+0000 7f5a18ff9640 1 -- 172.21.15.114:0/3218000872 <== mon.0 v2:172.21.15.114:3300/0 2 ==== osd_map(106..106 src has 1..106) ==== 8629+0+0 (secure 0 0 0) 0x7f5a200041d0 con 0x55de933f7f40 2024-04-23T10:38:23.319 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:38:23.306+0000 7f5a3a579740 1 -- 172.21.15.114:0/3218000872 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f5a2800db90 con 0x55de933f7f40 2024-04-23T10:38:23.319 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:38:23.306+0000 7f5a18ff9640 1 -- 172.21.15.114:0/3218000872 <== mon.0 v2:172.21.15.114:3300/0 3 ==== mon_get_version_reply(handle=1 version=40) ==== 24+0+0 (secure 0 0 0) 0x7f5a20004410 con 0x55de933f7f40 2024-04-23T10:38:23.319 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:38:23.306+0000 7f5a3a579740 10 client.16905 fetch_fsmap learned FSMap version 40 2024-04-23T10:38:23.319 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:38:23.306+0000 7f5a3a579740 1 -- 172.21.15.114:0/3218000872 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({config=0+,fsmap=40,monmap=4+}) -- 0x55de934113a0 con 0x55de933f7f40 2024-04-23T10:38:23.319 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:38:23.306+0000 7f5a18ff9640 1 -- 172.21.15.114:0/3218000872 <== mon.0 v2:172.21.15.114:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f5a2800db90 con 0x55de933f7f40 2024-04-23T10:38:23.319 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:38:23.307+0000 7f5a18ff9640 1 -- 172.21.15.114:0/3218000872 <== mon.0 v2:172.21.15.114:3300/0 5 ==== fsmap(e 40) ==== 8346+0+0 (secure 0 0 0) 0x7f5a20003920 con 0x55de933f7f40 2024-04-23T10:38:23.319 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:38:23.307+0000 7f5a3a579740 10 client.16905 fetch_fsmap finished waiting for FSMap version 40 2024-04-23T10:38:23.320 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:38:23.307+0000 7f5a3a579740 10 client.16905 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-04-23T10:38:23.320 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:38:23.307+0000 7f5a3a579740 20 client.16905 populate_metadata read hostname 'smithi114' 2024-04-23T10:38:23.320 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:38:23.307+0000 7f5a3a579740 1 --2- 172.21.15.114:0/3218000872 >> [v2:172.21.15.119:6838/937297340,v1:172.21.15.119:6839/937297340] conn(0x55de934128b0 0x55de93414ca0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T10:38:23.320 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:38:23.307+0000 7f5a3a579740 4 client.16905 mds_command: new command op to 24545 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-23T10:38:23.320 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:38:23.307+0000 7f5a3a579740 1 -- 172.21.15.114:0/3218000872 --> [v2:172.21.15.119:6838/937297340,v1:172.21.15.119:6839/937297340] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55de9336a390 con 0x55de934128b0 2024-04-23T10:38:23.320 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:38:23.307+0000 7f5a35600640 1 --2- 172.21.15.114:0/3218000872 >> [v2:172.21.15.119:6838/937297340,v1:172.21.15.119:6839/937297340] conn(0x55de934128b0 0x55de93414ca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T10:38:23.320 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:38:23.307+0000 7f5a18ff9640 10 client.16905 ms_handle_connect on v2:172.21.15.119:6838/937297340 2024-04-23T10:38:23.320 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:38:23.307+0000 7f5a35600640 1 --2- 172.21.15.114:0/3218000872 >> [v2:172.21.15.119:6838/937297340,v1:172.21.15.119:6839/937297340] conn(0x55de934128b0 0x55de93414ca0 crc :-1 s=READY pgs=166 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=a1a6481250f4be34 server_cookie=3a0020325483e4e4 in_seq=0 out_seq=0 2024-04-23T10:38:23.320 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:38:23.308+0000 7f5a18ff9640 1 -- 172.21.15.114:0/3218000872 <== mds.0 v2:172.21.15.119:6838/937297340 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x55de9336a390 con 0x55de934128b0 2024-04-23T10:38:23.320 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:38:23.308+0000 7f5a18ff9640 10 client.16905 handle_command_reply: tid=0 2024-04-23T10:38:23.321 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:38:23.308+0000 7f5a3a579740 1 client.16905 shutdown 2024-04-23T10:38:23.321 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:38:23.308+0000 7f5a3a579740 1 -- 172.21.15.114:0/3218000872 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55de933f7f40 msgr2=0x55de933ff450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T10:38:23.321 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:38:23.308+0000 7f5a3a579740 1 --2- 172.21.15.114:0/3218000872 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55de933f7f40 0x55de933ff450 secure :-1 s=READY pgs=879 cs=0 l=1 rev1=1 crypto rx=0x7f5a200126e0 tx=0x7f5a20012ae0 comp rx=0 tx=0).stop 2024-04-23T10:38:23.321 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:38:23.308+0000 7f5a3a579740 1 -- 172.21.15.114:0/3218000872 shutdown_connections 2024-04-23T10:38:23.321 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:38:23.308+0000 7f5a3a579740 1 --2- 172.21.15.114:0/3218000872 >> [v2:172.21.15.119:6838/937297340,v1:172.21.15.119:6839/937297340] conn(0x55de934128b0 0x55de93414ca0 crc :-1 s=READY pgs=166 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:38:23.321 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:38:23.308+0000 7f5a3a579740 1 --2- 172.21.15.114:0/3218000872 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x55de93402800 0x55de933ff9b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:38:23.321 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:38:23.308+0000 7f5a3a579740 1 --2- 172.21.15.114:0/3218000872 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55de933f7f40 0x55de933ff450 unknown :-1 s=CLOSED pgs=879 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:38:23.321 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:38:23.308+0000 7f5a18ff9640 0 client.16905 ms_handle_reset on v2:172.21.15.119:6838/937297340 2024-04-23T10:38:23.321 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:38:23.308+0000 7f5a3a579740 1 --2- 172.21.15.114:0/3218000872 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x55de933f7620 0x55de93402160 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:38:23.321 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:38:23.308+0000 7f5a3a579740 1 -- 172.21.15.114:0/3218000872 >> 172.21.15.114:0/3218000872 conn(0x55de933f3b30 msgr2=0x55de933f9860 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T10:38:23.321 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:38:23.308+0000 7f5a3a579740 1 -- 172.21.15.114:0/3218000872 shutdown_connections 2024-04-23T10:38:23.322 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:38:23.308+0000 7f5a3a579740 1 -- 172.21.15.114:0/3218000872 wait complete. 2024-04-23T10:38:23.322 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:38:23.308+0000 7f5a3a579740 20 client.16905 trim_cache size 0 max 16384 2024-04-23T10:38:23.329 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:38:23.323+0000 7f5a3a579740 1 Processor -- start 2024-04-23T10:38:23.329 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:38:23.323+0000 7f5a3a579740 1 -- start start 2024-04-23T10:38:23.329 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:38:23.323+0000 7f5a3a579740 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x55de933f7620 0x55de9340e290 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T10:38:23.329 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:38:23.323+0000 7f5a3a579740 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55de933f7f40 0x55de93401c90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T10:38:23.329 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:38:23.324+0000 7f5a3a579740 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x55de93402800 0x55de9341a5b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T10:38:23.329 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:38:23.324+0000 7f5a3a579740 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x7f5a301b05d0 con 0x55de933f7f40 2024-04-23T10:38:23.329 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:38:23.324+0000 7f5a3a579740 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 -- 0x7f5a301ade40 con 0x55de933f7620 2024-04-23T10:38:23.329 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:38:23.324+0000 7f5a3a579740 1 -- --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_getmap magic: 0 -- 0x7f5a301abcb0 con 0x55de93402800 2024-04-23T10:38:23.330 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:38:23.324+0000 7f5a34dff640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55de933f7f40 0x55de93401c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T10:38:23.330 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:38:23.324+0000 7f5a34dff640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55de933f7f40 0x55de93401c90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.114:46792/0 (socket says 172.21.15.114:46792) 2024-04-23T10:38:23.330 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:38:23.324+0000 7f5a34dff640 1 -- 172.21.15.114:0/624476770 learned_addr learned my addr 172.21.15.114:0/624476770 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-04-23T10:38:23.330 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:38:23.324+0000 7f5a35e01640 1 --2- 172.21.15.114:0/624476770 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x55de93402800 0x55de9341a5b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T10:38:23.330 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:38:23.324+0000 7f5a35600640 1 --2- 172.21.15.114:0/624476770 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x55de933f7620 0x55de9340e290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T10:38:23.330 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:38:23.324+0000 7f5a34dff640 1 -- 172.21.15.114:0/624476770 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x55de93402800 msgr2=0x55de9341a5b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T10:38:23.330 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:38:23.324+0000 7f5a34dff640 1 --2- 172.21.15.114:0/624476770 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x55de93402800 0x55de9341a5b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:38:23.330 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:38:23.324+0000 7f5a34dff640 1 -- 172.21.15.114:0/624476770 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x55de933f7620 msgr2=0x55de9340e290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T10:38:23.330 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:38:23.324+0000 7f5a34dff640 1 --2- 172.21.15.114:0/624476770 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x55de933f7620 0x55de9340e290 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:38:23.330 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:38:23.324+0000 7f5a34dff640 1 -- 172.21.15.114:0/624476770 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55de933ffd20 con 0x55de933f7f40 2024-04-23T10:38:23.331 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:38:23.324+0000 7f5a35e01640 1 --2- 172.21.15.114:0/624476770 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x55de93402800 0x55de9341a5b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-23T10:38:23.331 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:38:23.324+0000 7f5a34dff640 1 --2- 172.21.15.114:0/624476770 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55de933f7f40 0x55de93401c90 secure :-1 s=READY pgs=880 cs=0 l=1 rev1=1 crypto rx=0x7f5a200059f0 tx=0x7f5a200098c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a9c1d8c043f5a2d1 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T10:38:23.331 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:38:23.324+0000 7f5a1affd640 1 -- 172.21.15.114:0/624476770 <== mon.0 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5a20004070 con 0x55de933f7f40 2024-04-23T10:38:23.331 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:38:23.324+0000 7f5a1affd640 1 -- 172.21.15.114:0/624476770 <== mon.0 v2:172.21.15.114:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f5a20004210 con 0x55de933f7f40 2024-04-23T10:38:23.331 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:38:23.324+0000 7f5a1affd640 1 -- 172.21.15.114:0/624476770 <== mon.0 v2:172.21.15.114:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5a2000fe30 con 0x55de933f7f40 2024-04-23T10:38:23.331 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:38:23.325+0000 7f5a3a579740 1 -- 172.21.15.114:0/624476770 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55de933f7f40 msgr2=0x55de93401c90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T10:38:23.331 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:38:23.325+0000 7f5a3a579740 1 --2- 172.21.15.114:0/624476770 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55de933f7f40 0x55de93401c90 secure :-1 s=READY pgs=880 cs=0 l=1 rev1=1 crypto rx=0x7f5a200059f0 tx=0x7f5a200098c0 comp rx=0 tx=0).stop 2024-04-23T10:38:23.331 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:38:23.325+0000 7f5a3a579740 1 -- 172.21.15.114:0/624476770 shutdown_connections 2024-04-23T10:38:23.331 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:38:23.325+0000 7f5a3a579740 1 --2- 172.21.15.114:0/624476770 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x55de93402800 0x55de9341a5b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:38:23.331 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:38:23.325+0000 7f5a3a579740 1 --2- 172.21.15.114:0/624476770 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55de933f7f40 0x55de93401c90 unknown :-1 s=CLOSED pgs=880 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:38:23.332 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:38:23.325+0000 7f5a3a579740 1 --2- 172.21.15.114:0/624476770 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x55de933f7620 0x55de9340e290 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:38:23.332 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:38:23.325+0000 7f5a3a579740 1 -- 172.21.15.114:0/624476770 >> 172.21.15.114:0/624476770 conn(0x55de933f3b30 msgr2=0x55de933f2cb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T10:38:23.332 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:38:23.325+0000 7f5a3a579740 1 -- 172.21.15.114:0/624476770 shutdown_connections 2024-04-23T10:38:23.332 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:38:23.325+0000 7f5a3a579740 1 -- 172.21.15.114:0/624476770 wait complete. 2024-04-23T10:38:23.332 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:38:23.325+0000 7f5a3a579740 1 -- start start 2024-04-23T10:38:23.332 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:38:23.325+0000 7f5a3a579740 1 Processor -- start 2024-04-23T10:38:23.332 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:38:23.326+0000 7f5a3a579740 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55de933f7620 0x55de933fc850 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T10:38:23.332 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:38:23.326+0000 7f5a3a579740 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x55de933f7f40 0x55de933fcd90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T10:38:23.332 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:38:23.326+0000 7f5a3a579740 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x55de93402800 0x55de93477e20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T10:38:23.332 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:38:23.326+0000 7f5a3a579740 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x7f5a301ac300 con 0x55de933f7620 2024-04-23T10:38:23.332 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:38:23.326+0000 7f5a3a579740 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 -- 0x7f5a301ad770 con 0x55de933f7f40 2024-04-23T10:38:23.333 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:38:23.326+0000 7f5a3a579740 1 -- --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_getmap magic: 0 -- 0x7f5a301ac9d0 con 0x55de93402800 2024-04-23T10:38:23.333 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:38:23.326+0000 7f5a3a579740 10 client.0 _ll_register_callbacks cb 0x7f5a20009ea0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-23T10:38:23.333 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:38:23.326+0000 7f5a35600640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55de933f7620 0x55de933fc850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T10:38:23.333 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:38:23.326+0000 7f5a35600640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55de933f7620 0x55de933fc850 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.114:46808/0 (socket says 172.21.15.114:46808) 2024-04-23T10:38:23.333 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:38:23.326+0000 7f5a35600640 1 -- 172.21.15.114:0/1476406004 learned_addr learned my addr 172.21.15.114:0/1476406004 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-04-23T10:38:23.333 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:38:23.326+0000 7f5a35e01640 1 --2- 172.21.15.114:0/1476406004 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x55de93402800 0x55de93477e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T10:38:23.333 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:38:23.326+0000 7f5a34dff640 1 --2- 172.21.15.114:0/1476406004 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x55de933f7f40 0x55de933fcd90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T10:38:23.333 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:38:23.326+0000 7f5a35600640 1 -- 172.21.15.114:0/1476406004 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x55de93402800 msgr2=0x55de93477e20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T10:38:23.333 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:38:23.326+0000 7f5a35600640 1 --2- 172.21.15.114:0/1476406004 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x55de93402800 0x55de93477e20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:38:23.333 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:38:23.326+0000 7f5a35600640 1 -- 172.21.15.114:0/1476406004 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x55de933f7f40 msgr2=0x55de933fcd90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T10:38:23.334 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:38:23.326+0000 7f5a35600640 1 --2- 172.21.15.114:0/1476406004 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x55de933f7f40 0x55de933fcd90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:38:23.334 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:38:23.326+0000 7f5a35600640 1 -- 172.21.15.114:0/1476406004 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55de93478360 con 0x55de933f7620 2024-04-23T10:38:23.334 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:38:23.326+0000 7f5a34dff640 1 --2- 172.21.15.114:0/1476406004 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x55de933f7f40 0x55de933fcd90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-23T10:38:23.334 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:38:23.326+0000 7f5a35e01640 1 --2- 172.21.15.114:0/1476406004 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x55de93402800 0x55de93477e20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-23T10:38:23.334 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:38:23.326+0000 7f5a35600640 1 --2- 172.21.15.114:0/1476406004 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55de933f7620 0x55de933fc850 secure :-1 s=READY pgs=881 cs=0 l=1 rev1=1 crypto rx=0x7f5a2c0bc9a0 tx=0x7f5a2c005e70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=df3415052bd35988 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T10:38:23.334 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:38:23.326+0000 7f5a18ff9640 10 client.0 ms_handle_connect on v2:172.21.15.114:3300/0 2024-04-23T10:38:23.334 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:38:23.326+0000 7f5a18ff9640 1 -- 172.21.15.114:0/1476406004 <== mon.0 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5a2c049040 con 0x55de933f7620 2024-04-23T10:38:23.334 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:38:23.326+0000 7f5a18ff9640 1 -- 172.21.15.114:0/1476406004 <== mon.0 v2:172.21.15.114:3300/0 2 ==== osd_map(106..106 src has 1..106) ==== 8629+0+0 (secure 0 0 0) 0x7f5a2c003c50 con 0x55de933f7620 2024-04-23T10:38:23.334 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:38:23.326+0000 7f5a3a579740 1 -- 172.21.15.114:0/1476406004 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f5a20004210 con 0x55de933f7620 2024-04-23T10:38:23.334 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:38:23.327+0000 7f5a18ff9640 1 -- 172.21.15.114:0/1476406004 <== mon.0 v2:172.21.15.114:3300/0 3 ==== mon_get_version_reply(handle=1 version=40) ==== 24+0+0 (secure 0 0 0) 0x7f5a2c0dbe60 con 0x55de933f7620 2024-04-23T10:38:23.335 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:38:23.327+0000 7f5a3a579740 10 client.16911 fetch_fsmap learned FSMap version 40 2024-04-23T10:38:23.335 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:38:23.327+0000 7f5a3a579740 1 -- 172.21.15.114:0/1476406004 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({config=0+,fsmap=40,monmap=4+}) -- 0x55de934795e0 con 0x55de933f7620 2024-04-23T10:38:23.335 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:38:23.327+0000 7f5a18ff9640 1 -- 172.21.15.114:0/1476406004 <== mon.0 v2:172.21.15.114:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f5a20004210 con 0x55de933f7620 2024-04-23T10:38:23.335 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:38:23.327+0000 7f5a18ff9640 1 -- 172.21.15.114:0/1476406004 <== mon.0 v2:172.21.15.114:3300/0 5 ==== fsmap(e 40) ==== 8346+0+0 (secure 0 0 0) 0x7f5a2c0b9310 con 0x55de933f7620 2024-04-23T10:38:23.335 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:38:23.327+0000 7f5a3a579740 10 client.16911 fetch_fsmap finished waiting for FSMap version 40 2024-04-23T10:38:23.335 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:38:23.327+0000 7f5a3a579740 10 client.16911 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-04-23T10:38:23.335 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:38:23.327+0000 7f5a3a579740 20 client.16911 populate_metadata read hostname 'smithi114' 2024-04-23T10:38:23.335 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:38:23.327+0000 7f5a3a579740 1 --2- 172.21.15.114:0/1476406004 >> [v2:172.21.15.119:6838/937297340,v1:172.21.15.119:6839/937297340] conn(0x55de93479d10 0x55de9347c150 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T10:38:23.335 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:38:23.327+0000 7f5a3a579740 4 client.16911 mds_command: new command op to 24545 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-04-23T10:38:23.335 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:38:23.327+0000 7f5a3a579740 1 -- 172.21.15.114:0/1476406004 --> [v2:172.21.15.119:6838/937297340,v1:172.21.15.119:6839/937297340] -- command(tid 0: {"prefix": "flush journal"}) -- 0x55de93364200 con 0x55de93479d10 2024-04-23T10:38:23.335 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:38:23.327+0000 7f5a34dff640 1 --2- 172.21.15.114:0/1476406004 >> [v2:172.21.15.119:6838/937297340,v1:172.21.15.119:6839/937297340] conn(0x55de93479d10 0x55de9347c150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T10:38:23.335 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:38:23.328+0000 7f5a18ff9640 10 client.16911 ms_handle_connect on v2:172.21.15.119:6838/937297340 2024-04-23T10:38:23.336 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:38:23.328+0000 7f5a34dff640 1 --2- 172.21.15.114:0/1476406004 >> [v2:172.21.15.119:6838/937297340,v1:172.21.15.119:6839/937297340] conn(0x55de93479d10 0x55de9347c150 crc :-1 s=READY pgs=167 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=175569177377c0e1 server_cookie=61b84e9d8fb43c in_seq=0 out_seq=0 2024-04-23T10:38:23.368 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:38:23.368+0000 7f5a18ff9640 1 -- 172.21.15.114:0/1476406004 <== mds.0 v2:172.21.15.119:6838/937297340 1 ==== command_reply(tid 0: 0 ) ==== 8+0+44 (crc 0 0 0) 0x55de93364200 con 0x55de93479d10 2024-04-23T10:38:23.368 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:38:23.368+0000 7f5a18ff9640 10 client.16911 handle_command_reply: tid=0 2024-04-23T10:38:23.368 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:38:23.368+0000 7f5a3a579740 1 client.16911 shutdown 2024-04-23T10:38:23.368 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:38:23.369+0000 7f5a3a579740 1 -- 172.21.15.114:0/1476406004 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55de933f7620 msgr2=0x55de933fc850 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T10:38:23.369 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:38:23.369+0000 7f5a3a579740 1 --2- 172.21.15.114:0/1476406004 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55de933f7620 0x55de933fc850 secure :-1 s=READY pgs=881 cs=0 l=1 rev1=1 crypto rx=0x7f5a2c0bc9a0 tx=0x7f5a2c005e70 comp rx=0 tx=0).stop 2024-04-23T10:38:23.369 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:38:23.369+0000 7f5a3a579740 1 -- 172.21.15.114:0/1476406004 shutdown_connections 2024-04-23T10:38:23.369 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:38:23.369+0000 7f5a3a579740 1 --2- 172.21.15.114:0/1476406004 >> [v2:172.21.15.119:6838/937297340,v1:172.21.15.119:6839/937297340] conn(0x55de93479d10 0x55de9347c150 crc :-1 s=READY pgs=167 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:38:23.369 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:38:23.369+0000 7f5a3a579740 1 --2- 172.21.15.114:0/1476406004 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x55de93402800 0x55de93477e20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:38:23.369 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:38:23.369+0000 7f5a3a579740 1 --2- 172.21.15.114:0/1476406004 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x55de933f7f40 0x55de933fcd90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:38:23.369 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:38:23.369+0000 7f5a3a579740 1 --2- 172.21.15.114:0/1476406004 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55de933f7620 0x55de933fc850 unknown :-1 s=CLOSED pgs=881 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:38:23.369 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:38:23.369+0000 7f5a18ff9640 0 client.16911 ms_handle_reset on v2:172.21.15.119:6838/937297340 2024-04-23T10:38:23.369 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:38:23.369+0000 7f5a3a579740 1 -- 172.21.15.114:0/1476406004 >> 172.21.15.114:0/1476406004 conn(0x55de933f3b30 msgr2=0x55de933fa140 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T10:38:23.369 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:38:23.369+0000 7f5a3a579740 1 -- 172.21.15.114:0/1476406004 shutdown_connections 2024-04-23T10:38:23.369 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:38:23.369+0000 7f5a3a579740 1 -- 172.21.15.114:0/1476406004 wait complete. 2024-04-23T10:38:23.370 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:38:23.369+0000 7f5a3a579740 20 client.16911 trim_cache size 0 max 16384 2024-04-23T10:38:23.370 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stdout:{ 2024-04-23T10:38:23.370 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stdout: "message": "", 2024-04-23T10:38:23.370 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stdout: "return_code": 0 2024-04-23T10:38:23.370 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stdout:} 2024-04-23T10:38:23.371 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:38:23.369+0000 7f5a37864640 1 -- 172.21.15.114:0/4235522242 >> [v2:172.21.15.114:6800/1395693466,v1:172.21.15.114:6801/1395693466] conn(0x7f5a1c0a3dc0 msgr2=0x7f5a1c0a6280 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T10:38:23.371 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:38:23.369+0000 7f5a37864640 1 --2- 172.21.15.114:0/4235522242 >> [v2:172.21.15.114:6800/1395693466,v1:172.21.15.114:6801/1395693466] conn(0x7f5a1c0a3dc0 0x7f5a1c0a6280 secure :-1 s=READY pgs=137 cs=0 l=1 rev1=1 crypto rx=0x7f5a28012b80 tx=0x7f5a280099b0 comp rx=0 tx=0).stop 2024-04-23T10:38:23.371 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:38:23.369+0000 7f5a37864640 1 -- 172.21.15.114:0/4235522242 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f5a301006c0 msgr2=0x7f5a30198730 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T10:38:23.372 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:38:23.369+0000 7f5a37864640 1 --2- 172.21.15.114:0/4235522242 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f5a301006c0 0x7f5a30198730 secure :-1 s=READY pgs=877 cs=0 l=1 rev1=1 crypto rx=0x7f5a2c005940 tx=0x7f5a2c004270 comp rx=0 tx=0).stop 2024-04-23T10:38:23.372 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:38:23.370+0000 7f5a37864640 1 -- 172.21.15.114:0/4235522242 shutdown_connections 2024-04-23T10:38:23.372 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:38:23.370+0000 7f5a37864640 1 --2- 172.21.15.114:0/4235522242 >> [v2:172.21.15.114:6800/1395693466,v1:172.21.15.114:6801/1395693466] conn(0x7f5a1c0a3dc0 0x7f5a1c0a6280 unknown :-1 s=CLOSED pgs=137 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:38:23.372 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:38:23.370+0000 7f5a37864640 1 --2- 172.21.15.114:0/4235522242 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f5a3019cfa0 0x7f5a3019d450 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:38:23.372 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:38:23.370+0000 7f5a37864640 1 --2- 172.21.15.114:0/4235522242 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f5a30107280 0x7f5a30198c70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:38:23.372 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:38:23.370+0000 7f5a37864640 1 --2- 172.21.15.114:0/4235522242 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f5a301006c0 0x7f5a30198730 unknown :-1 s=CLOSED pgs=877 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:38:23.372 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:38:23.370+0000 7f5a37864640 1 -- 172.21.15.114:0/4235522242 >> 172.21.15.114:0/4235522242 conn(0x7f5a30077390 msgr2=0x7f5a300fe760 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T10:38:23.372 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:38:23.370+0000 7f5a37864640 1 -- 172.21.15.114:0/4235522242 shutdown_connections 2024-04-23T10:38:23.372 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:38:23.370+0000 7f5a37864640 1 -- 172.21.15.114:0/4235522242 wait complete. 2024-04-23T10:38:23.946 INFO:tasks.workunit.client.0.smithi114.stdout: 1 689083 5.42 MB/sec execute 512 sec latency 4655.516 ms 2024-04-23T10:38:23.992 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 23 10:38:23 smithi119 ceph-mon[32608]: pgmap v370: 97 pgs: 97 active+clean; 218 MiB data, 2.7 GiB used, 1.0 TiB / 1.0 TiB avail; 360 KiB/s rd, 1.0 MiB/s wr, 66 op/s 2024-04-23T10:38:24.004 INFO:journalctl@ceph.mon.a.smithi114.stdout:Apr 23 10:38:23 smithi114 ceph-mon[29255]: pgmap v370: 97 pgs: 97 active+clean; 218 MiB data, 2.7 GiB used, 1.0 TiB / 1.0 TiB avail; 360 KiB/s rd, 1.0 MiB/s wr, 66 op/s 2024-04-23T10:38:24.078 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 23 10:38:23 smithi204 ceph-mon[32649]: pgmap v370: 97 pgs: 97 active+clean; 218 MiB data, 2.7 GiB used, 1.0 TiB / 1.0 TiB avail; 360 KiB/s rd, 1.0 MiB/s wr, 66 op/s 2024-04-23T10:38:24.946 INFO:tasks.workunit.client.0.smithi114.stdout: 1 689775 5.41 MB/sec execute 513 sec latency 4898.535 ms 2024-04-23T10:38:25.946 INFO:tasks.workunit.client.0.smithi114.stdout: 1 691959 5.42 MB/sec execute 514 sec latency 19.920 ms 2024-04-23T10:38:25.992 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 23 10:38:25 smithi119 ceph-mon[32608]: pgmap v371: 97 pgs: 97 active+clean; 218 MiB data, 2.7 GiB used, 1.0 TiB / 1.0 TiB avail; 328 KiB/s rd, 955 KiB/s wr, 60 op/s 2024-04-23T10:38:26.004 INFO:journalctl@ceph.mon.a.smithi114.stdout:Apr 23 10:38:25 smithi114 ceph-mon[29255]: pgmap v371: 97 pgs: 97 active+clean; 218 MiB data, 2.7 GiB used, 1.0 TiB / 1.0 TiB avail; 328 KiB/s rd, 955 KiB/s wr, 60 op/s 2024-04-23T10:38:26.078 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 23 10:38:25 smithi204 ceph-mon[32649]: pgmap v371: 97 pgs: 97 active+clean; 218 MiB data, 2.7 GiB used, 1.0 TiB / 1.0 TiB avail; 328 KiB/s rd, 955 KiB/s wr, 60 op/s 2024-04-23T10:38:26.946 INFO:tasks.workunit.client.0.smithi114.stdout: 1 692564 5.42 MB/sec execute 515 sec latency 771.910 ms 2024-04-23T10:38:27.947 INFO:tasks.workunit.client.0.smithi114.stdout: 1 692564 5.41 MB/sec execute 516 sec latency 1771.991 ms 2024-04-23T10:38:27.992 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 23 10:38:27 smithi119 ceph-mon[32608]: pgmap v372: 97 pgs: 97 active+clean; 219 MiB data, 2.7 GiB used, 1.0 TiB / 1.0 TiB avail; 585 KiB/s rd, 1.3 MiB/s wr, 110 op/s 2024-04-23T10:38:28.004 INFO:journalctl@ceph.mon.a.smithi114.stdout:Apr 23 10:38:27 smithi114 ceph-mon[29255]: pgmap v372: 97 pgs: 97 active+clean; 219 MiB data, 2.7 GiB used, 1.0 TiB / 1.0 TiB avail; 585 KiB/s rd, 1.3 MiB/s wr, 110 op/s 2024-04-23T10:38:28.078 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 23 10:38:27 smithi204 ceph-mon[32649]: pgmap v372: 97 pgs: 97 active+clean; 219 MiB data, 2.7 GiB used, 1.0 TiB / 1.0 TiB avail; 585 KiB/s rd, 1.3 MiB/s wr, 110 op/s 2024-04-23T10:38:28.947 INFO:tasks.workunit.client.0.smithi114.stdout: 1 692564 5.40 MB/sec execute 517 sec latency 2772.089 ms 2024-04-23T10:38:29.947 INFO:tasks.workunit.client.0.smithi114.stdout: 1 692631 5.39 MB/sec execute 518 sec latency 3007.826 ms 2024-04-23T10:38:29.992 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 23 10:38:29 smithi119 ceph-mon[32608]: pgmap v373: 97 pgs: 97 active+clean; 219 MiB data, 2.7 GiB used, 1.0 TiB / 1.0 TiB avail; 339 KiB/s rd, 912 KiB/s wr, 71 op/s 2024-04-23T10:38:30.004 INFO:journalctl@ceph.mon.a.smithi114.stdout:Apr 23 10:38:29 smithi114 ceph-mon[29255]: pgmap v373: 97 pgs: 97 active+clean; 219 MiB data, 2.7 GiB used, 1.0 TiB / 1.0 TiB avail; 339 KiB/s rd, 912 KiB/s wr, 71 op/s 2024-04-23T10:38:30.078 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 23 10:38:29 smithi204 ceph-mon[32649]: pgmap v373: 97 pgs: 97 active+clean; 219 MiB data, 2.7 GiB used, 1.0 TiB / 1.0 TiB avail; 339 KiB/s rd, 912 KiB/s wr, 71 op/s 2024-04-23T10:38:30.947 INFO:tasks.workunit.client.0.smithi114.stdout: 1 692631 5.38 MB/sec execute 519 sec latency 1654.791 ms 2024-04-23T10:38:31.947 INFO:tasks.workunit.client.0.smithi114.stdout: 1 692631 5.37 MB/sec execute 520 sec latency 2654.878 ms 2024-04-23T10:38:31.992 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 23 10:38:31 smithi119 ceph-mon[32608]: pgmap v374: 97 pgs: 97 active+clean; 219 MiB data, 2.7 GiB used, 1.0 TiB / 1.0 TiB avail; 339 KiB/s rd, 912 KiB/s wr, 71 op/s 2024-04-23T10:38:32.004 INFO:journalctl@ceph.mon.a.smithi114.stdout:Apr 23 10:38:31 smithi114 ceph-mon[29255]: pgmap v374: 97 pgs: 97 active+clean; 219 MiB data, 2.7 GiB used, 1.0 TiB / 1.0 TiB avail; 339 KiB/s rd, 912 KiB/s wr, 71 op/s 2024-04-23T10:38:32.078 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 23 10:38:31 smithi204 ceph-mon[32649]: pgmap v374: 97 pgs: 97 active+clean; 219 MiB data, 2.7 GiB used, 1.0 TiB / 1.0 TiB avail; 339 KiB/s rd, 912 KiB/s wr, 71 op/s 2024-04-23T10:38:32.947 INFO:tasks.workunit.client.0.smithi114.stdout: 1 692631 5.36 MB/sec execute 521 sec latency 3654.985 ms 2024-04-23T10:38:33.947 INFO:tasks.workunit.client.0.smithi114.stdout: 1 692631 5.35 MB/sec execute 522 sec latency 4655.108 ms 2024-04-23T10:38:33.992 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 23 10:38:33 smithi119 ceph-mon[32608]: pgmap v375: 97 pgs: 97 active+clean; 219 MiB data, 2.7 GiB used, 1.0 TiB / 1.0 TiB avail; 392 KiB/s rd, 947 KiB/s wr, 83 op/s 2024-04-23T10:38:33.992 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 23 10:38:33 smithi119 ceph-mon[32608]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=e=up:active} 3 up:standby-replay 6 up:standby 2024-04-23T10:38:34.004 INFO:journalctl@ceph.mon.a.smithi114.stdout:Apr 23 10:38:33 smithi114 ceph-mon[29255]: pgmap v375: 97 pgs: 97 active+clean; 219 MiB data, 2.7 GiB used, 1.0 TiB / 1.0 TiB avail; 392 KiB/s rd, 947 KiB/s wr, 83 op/s 2024-04-23T10:38:34.004 INFO:journalctl@ceph.mon.a.smithi114.stdout:Apr 23 10:38:33 smithi114 ceph-mon[29255]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=e=up:active} 3 up:standby-replay 6 up:standby 2024-04-23T10:38:34.078 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 23 10:38:33 smithi204 ceph-mon[32649]: pgmap v375: 97 pgs: 97 active+clean; 219 MiB data, 2.7 GiB used, 1.0 TiB / 1.0 TiB avail; 392 KiB/s rd, 947 KiB/s wr, 83 op/s 2024-04-23T10:38:34.078 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 23 10:38:33 smithi204 ceph-mon[32649]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=e=up:active} 3 up:standby-replay 6 up:standby 2024-04-23T10:38:34.947 INFO:tasks.workunit.client.0.smithi114.stdout: 1 693292 5.34 MB/sec execute 523 sec latency 4898.679 ms 2024-04-23T10:38:35.947 INFO:tasks.workunit.client.0.smithi114.stdout: 1 695496 5.35 MB/sec execute 524 sec latency 16.535 ms 2024-04-23T10:38:35.992 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 23 10:38:35 smithi119 ceph-mon[32608]: pgmap v376: 97 pgs: 97 active+clean; 219 MiB data, 2.7 GiB used, 1.0 TiB / 1.0 TiB avail; 338 KiB/s rd, 442 KiB/s wr, 66 op/s 2024-04-23T10:38:36.004 INFO:journalctl@ceph.mon.a.smithi114.stdout:Apr 23 10:38:35 smithi114 ceph-mon[29255]: pgmap v376: 97 pgs: 97 active+clean; 219 MiB data, 2.7 GiB used, 1.0 TiB / 1.0 TiB avail; 338 KiB/s rd, 442 KiB/s wr, 66 op/s 2024-04-23T10:38:36.078 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 23 10:38:35 smithi204 ceph-mon[32649]: pgmap v376: 97 pgs: 97 active+clean; 219 MiB data, 2.7 GiB used, 1.0 TiB / 1.0 TiB avail; 338 KiB/s rd, 442 KiB/s wr, 66 op/s 2024-04-23T10:38:36.457 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:38:36.455+0000 7f6c49b76640 1 -- 172.21.15.114:0/207384571 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f6c4410c5d0 msgr2=0x7f6c4410ea00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T10:38:36.457 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:38:36.455+0000 7f6c49b76640 1 --2- 172.21.15.114:0/207384571 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f6c4410c5d0 0x7f6c4410ea00 secure :-1 s=READY pgs=882 cs=0 l=1 rev1=1 crypto rx=0x7f6c3c009ca0 tx=0x7f6c3c02a950 comp rx=0 tx=0).stop 2024-04-23T10:38:36.457 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:38:36.455+0000 7f6c49b76640 1 -- 172.21.15.114:0/207384571 shutdown_connections 2024-04-23T10:38:36.457 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:38:36.455+0000 7f6c49b76640 1 --2- 172.21.15.114:0/207384571 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f6c4410c5d0 0x7f6c4410ea00 unknown :-1 s=CLOSED pgs=882 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:38:36.457 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:38:36.455+0000 7f6c49b76640 1 --2- 172.21.15.114:0/207384571 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f6c44103b20 0x7f6c4410c000 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:38:36.457 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:38:36.455+0000 7f6c49b76640 1 --2- 172.21.15.114:0/207384571 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f6c44103200 0x7f6c441035e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:38:36.457 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:38:36.455+0000 7f6c49b76640 1 -- 172.21.15.114:0/207384571 >> 172.21.15.114:0/207384571 conn(0x7f6c440771d0 msgr2=0x7f6c440775e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T10:38:36.457 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:38:36.455+0000 7f6c49b76640 1 -- 172.21.15.114:0/207384571 shutdown_connections 2024-04-23T10:38:36.458 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:38:36.455+0000 7f6c49b76640 1 -- 172.21.15.114:0/207384571 wait complete. 2024-04-23T10:38:36.458 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:38:36.456+0000 7f6c49b76640 1 Processor -- start 2024-04-23T10:38:36.458 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:38:36.456+0000 7f6c49b76640 1 -- start start 2024-04-23T10:38:36.458 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:38:36.456+0000 7f6c49b76640 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f6c44103200 0x7f6c44198590 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T10:38:36.458 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:38:36.456+0000 7f6c49b76640 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f6c44103b20 0x7f6c44198ad0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T10:38:36.458 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:38:36.456+0000 7f6c49b76640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f6c4410c5d0 0x7f6c4419ce50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T10:38:36.458 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:38:36.456+0000 7f6c49b76640 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x7f6c441121d0 con 0x7f6c4410c5d0 2024-04-23T10:38:36.458 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:38:36.456+0000 7f6c49b76640 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 -- 0x7f6c44112050 con 0x7f6c44103b20 2024-04-23T10:38:36.458 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:38:36.456+0000 7f6c49b76640 1 -- --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_getmap magic: 0 -- 0x7f6c44112350 con 0x7f6c44103200 2024-04-23T10:38:36.458 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:38:36.456+0000 7f6c43fff640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f6c4410c5d0 0x7f6c4419ce50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T10:38:36.459 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:38:36.456+0000 7f6c43fff640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f6c4410c5d0 0x7f6c4419ce50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.114:41962/0 (socket says 172.21.15.114:41962) 2024-04-23T10:38:36.459 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:38:36.456+0000 7f6c43fff640 1 -- 172.21.15.114:0/2265813900 learned_addr learned my addr 172.21.15.114:0/2265813900 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-04-23T10:38:36.459 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:38:36.456+0000 7f6c437fe640 1 --2- 172.21.15.114:0/2265813900 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f6c44103200 0x7f6c44198590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T10:38:36.459 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:38:36.456+0000 7f6c42ffd640 1 --2- 172.21.15.114:0/2265813900 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f6c44103b20 0x7f6c44198ad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T10:38:36.459 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:38:36.457+0000 7f6c43fff640 1 -- 172.21.15.114:0/2265813900 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f6c44103200 msgr2=0x7f6c44198590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T10:38:36.459 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:38:36.457+0000 7f6c43fff640 1 --2- 172.21.15.114:0/2265813900 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f6c44103200 0x7f6c44198590 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:38:36.459 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:38:36.457+0000 7f6c43fff640 1 -- 172.21.15.114:0/2265813900 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f6c44103b20 msgr2=0x7f6c44198ad0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T10:38:36.459 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:38:36.457+0000 7f6c43fff640 1 --2- 172.21.15.114:0/2265813900 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f6c44103b20 0x7f6c44198ad0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:38:36.459 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:38:36.457+0000 7f6c43fff640 1 -- 172.21.15.114:0/2265813900 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f6c4419d580 con 0x7f6c4410c5d0 2024-04-23T10:38:36.459 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:38:36.457+0000 7f6c43fff640 1 --2- 172.21.15.114:0/2265813900 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f6c4410c5d0 0x7f6c4419ce50 secure :-1 s=READY pgs=883 cs=0 l=1 rev1=1 crypto rx=0x7f6c3c02ae70 tx=0x7f6c3c02f5d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T10:38:36.460 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:38:36.457+0000 7f6c40ff9640 1 -- 172.21.15.114:0/2265813900 <== mon.0 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6c3c034040 con 0x7f6c4410c5d0 2024-04-23T10:38:36.460 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:38:36.457+0000 7f6c49b76640 1 -- 172.21.15.114:0/2265813900 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f6c4419d810 con 0x7f6c4410c5d0 2024-04-23T10:38:36.460 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:38:36.457+0000 7f6c49b76640 1 -- 172.21.15.114:0/2265813900 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f6c441a5100 con 0x7f6c4410c5d0 2024-04-23T10:38:36.460 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:38:36.457+0000 7f6c40ff9640 1 -- 172.21.15.114:0/2265813900 <== mon.0 v2:172.21.15.114:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f6c3c0308f0 con 0x7f6c4410c5d0 2024-04-23T10:38:36.460 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:38:36.457+0000 7f6c40ff9640 1 -- 172.21.15.114:0/2265813900 <== mon.0 v2:172.21.15.114:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6c3c0046f0 con 0x7f6c4410c5d0 2024-04-23T10:38:36.469 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:38:36.459+0000 7f6c40ff9640 1 -- 172.21.15.114:0/2265813900 <== mon.0 v2:172.21.15.114:3300/0 4 ==== mgrmap(e 26) ==== 138778+0+0 (secure 0 0 0) 0x7f6c3c037020 con 0x7f6c4410c5d0 2024-04-23T10:38:36.469 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:38:36.459+0000 7f6c4c96f740 1 Processor -- start 2024-04-23T10:38:36.469 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:38:36.459+0000 7f6c4c96f740 1 -- start start 2024-04-23T10:38:36.469 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:38:36.460+0000 7f6c4c96f740 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x55c8fd221620 0x55c8fd221a00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T10:38:36.469 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:38:36.460+0000 7f6c4c96f740 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55c8fd221f40 0x55c8fd22c2c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T10:38:36.469 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:38:36.460+0000 7f6c4c96f740 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x55c8fd22c800 0x55c8fd22ebf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T10:38:36.469 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:38:36.460+0000 7f6c4c96f740 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x55c8fd186970 con 0x55c8fd221f40 2024-04-23T10:38:36.469 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:38:36.460+0000 7f6c4c96f740 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 -- 0x55c8fd1234a0 con 0x55c8fd22c800 2024-04-23T10:38:36.469 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:38:36.460+0000 7f6c4c96f740 1 -- --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_getmap magic: 0 -- 0x55c8fd182930 con 0x55c8fd221620 2024-04-23T10:38:36.470 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:38:36.460+0000 7f6c40ff9640 1 --2- 172.21.15.114:0/2265813900 >> [v2:172.21.15.114:6800/1395693466,v1:172.21.15.114:6801/1395693466] conn(0x7f6c2c0a3e40 0x7f6c2c0a6300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T10:38:36.470 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:38:36.460+0000 7f6c437fe640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55c8fd221f40 0x55c8fd22c2c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T10:38:36.470 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:38:36.460+0000 7f6c437fe640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55c8fd221f40 0x55c8fd22c2c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.114:41974/0 (socket says 172.21.15.114:41974) 2024-04-23T10:38:36.470 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:38:36.460+0000 7f6c437fe640 1 -- 172.21.15.114:0/2058140868 learned_addr learned my addr 172.21.15.114:0/2058140868 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-04-23T10:38:36.470 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:38:36.460+0000 7f6c42ffd640 1 --2- 172.21.15.114:0/2058140868 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x55c8fd221620 0x55c8fd221a00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T10:38:36.470 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:38:36.460+0000 7f6c42ffd640 1 --2- 172.21.15.114:0/2058140868 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x55c8fd22c800 0x55c8fd22ebf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T10:38:36.470 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:38:36.460+0000 7f6c43fff640 1 --2- 172.21.15.114:0/2265813900 >> [v2:172.21.15.114:6800/1395693466,v1:172.21.15.114:6801/1395693466] conn(0x7f6c2c0a3e40 0x7f6c2c0a6300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T10:38:36.470 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:38:36.460+0000 7f6c40ff9640 1 -- 172.21.15.114:0/2265813900 <== mon.0 v2:172.21.15.114:3300/0 5 ==== osd_map(106..106 src has 1..106) ==== 8629+0+0 (secure 0 0 0) 0x7f6c3c0f15d0 con 0x7f6c4410c5d0 2024-04-23T10:38:36.471 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:38:36.460+0000 7f6c42ffd640 1 -- 172.21.15.114:0/2058140868 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x55c8fd22c800 msgr2=0x55c8fd22ebf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T10:38:36.471 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:38:36.460+0000 7f6c42ffd640 1 --2- 172.21.15.114:0/2058140868 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x55c8fd22c800 0x55c8fd22ebf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:38:36.471 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:38:36.460+0000 7f6c42ffd640 1 -- 172.21.15.114:0/2058140868 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55c8fd221f40 msgr2=0x55c8fd22c2c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T10:38:36.471 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:38:36.460+0000 7f6c42ffd640 1 --2- 172.21.15.114:0/2058140868 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55c8fd221f40 0x55c8fd22c2c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:38:36.471 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:38:36.460+0000 7f6c42ffd640 1 -- 172.21.15.114:0/2058140868 --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55c8fce99210 con 0x55c8fd221620 2024-04-23T10:38:36.471 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:38:36.460+0000 7f6c437fe640 1 --2- 172.21.15.114:0/2058140868 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55c8fd221f40 0x55c8fd22c2c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-23T10:38:36.471 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:38:36.460+0000 7f6c42ffd640 1 --2- 172.21.15.114:0/2058140868 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x55c8fd221620 0x55c8fd221a00 secure :-1 s=READY pgs=80 cs=0 l=1 rev1=1 crypto rx=0x7f6c3800afd0 tx=0x7f6c3800a680 comp rx=0 tx=0).ready entity=mon.1 client_cookie=7fc3853d7bb4c793 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T10:38:36.471 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:38:36.460+0000 7f6c43fff640 1 --2- 172.21.15.114:0/2265813900 >> [v2:172.21.15.114:6800/1395693466,v1:172.21.15.114:6801/1395693466] conn(0x7f6c2c0a3e40 0x7f6c2c0a6300 secure :-1 s=READY pgs=138 cs=0 l=1 rev1=1 crypto rx=0x7f6c3c0bd2b0 tx=0x7f6c3c0b2a40 comp rx=0 tx=0).ready entity=mgr.15291 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T10:38:36.471 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:38:36.461+0000 7f6c29ffb640 1 -- 172.21.15.114:0/2058140868 <== mon.1 v2:172.21.15.204:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6c38005020 con 0x55c8fd221620 2024-04-23T10:38:36.471 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:38:36.461+0000 7f6c29ffb640 1 -- 172.21.15.114:0/2058140868 <== mon.1 v2:172.21.15.204:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f6c380047a0 con 0x55c8fd221620 2024-04-23T10:38:36.472 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:38:36.461+0000 7f6c29ffb640 1 -- 172.21.15.114:0/2058140868 <== mon.1 v2:172.21.15.204:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6c3800f840 con 0x55c8fd221620 2024-04-23T10:38:36.472 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:38:36.461+0000 7f6c4c96f740 1 -- 172.21.15.114:0/2058140868 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x55c8fd221620 msgr2=0x55c8fd221a00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T10:38:36.472 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:38:36.461+0000 7f6c4c96f740 1 --2- 172.21.15.114:0/2058140868 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x55c8fd221620 0x55c8fd221a00 secure :-1 s=READY pgs=80 cs=0 l=1 rev1=1 crypto rx=0x7f6c3800afd0 tx=0x7f6c3800a680 comp rx=0 tx=0).stop 2024-04-23T10:38:36.472 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:38:36.461+0000 7f6c4c96f740 1 -- 172.21.15.114:0/2058140868 shutdown_connections 2024-04-23T10:38:36.472 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:38:36.461+0000 7f6c4c96f740 1 --2- 172.21.15.114:0/2058140868 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x55c8fd22c800 0x55c8fd22ebf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:38:36.472 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:38:36.461+0000 7f6c4c96f740 1 --2- 172.21.15.114:0/2058140868 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55c8fd221f40 0x55c8fd22c2c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:38:36.472 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:38:36.461+0000 7f6c4c96f740 1 --2- 172.21.15.114:0/2058140868 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x55c8fd221620 0x55c8fd221a00 unknown :-1 s=CLOSED pgs=80 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:38:36.472 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:38:36.461+0000 7f6c4c96f740 1 -- 172.21.15.114:0/2058140868 >> 172.21.15.114:0/2058140868 conn(0x55c8fd21db30 msgr2=0x55c8fd21df10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T10:38:36.472 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:38:36.461+0000 7f6c4c96f740 1 -- 172.21.15.114:0/2058140868 shutdown_connections 2024-04-23T10:38:36.472 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:38:36.461+0000 7f6c4c96f740 1 -- 172.21.15.114:0/2058140868 wait complete. 2024-04-23T10:38:36.472 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:38:36.462+0000 7f6c4c96f740 1 -- start start 2024-04-23T10:38:36.473 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:38:36.462+0000 7f6c4c96f740 1 Processor -- start 2024-04-23T10:38:36.473 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:38:36.462+0000 7f6c4c96f740 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x55c8fd221620 0x55c8fd22c130 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T10:38:36.473 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:38:36.462+0000 7f6c4c96f740 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x55c8fd221f40 0x55c8fd229420 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T10:38:36.473 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:38:36.462+0000 7f6c4c96f740 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55c8fd22c800 0x55c8fd229960 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T10:38:36.473 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:38:36.462+0000 7f6c4c96f740 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x55c8fd22f430 con 0x55c8fd22c800 2024-04-23T10:38:36.473 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:38:36.462+0000 7f6c4c96f740 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 -- 0x55c8fd1210b0 con 0x55c8fd221f40 2024-04-23T10:38:36.473 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:38:36.462+0000 7f6c4c96f740 1 -- --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_getmap magic: 0 -- 0x55c8fd22f5b0 con 0x55c8fd221620 2024-04-23T10:38:36.473 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:38:36.462+0000 7f6c4c96f740 10 client.0 _ll_register_callbacks cb 0x55c8fd10ed30 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-23T10:38:36.474 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:38:36.462+0000 7f6c42ffd640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55c8fd22c800 0x55c8fd229960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T10:38:36.474 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:38:36.462+0000 7f6c437fe640 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x55c8fd221f40 0x55c8fd229420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T10:38:36.474 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:38:36.462+0000 7f6c42ffd640 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x55c8fd221620 0x55c8fd22c130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T10:38:36.474 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:38:36.462+0000 7f6c437fe640 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x55c8fd221f40 0x55c8fd229420 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.119:3300/0 says I am v2:172.21.15.114:36916/0 (socket says 172.21.15.114:36916) 2024-04-23T10:38:36.474 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:38:36.462+0000 7f6c437fe640 1 -- 172.21.15.114:0/2690454726 learned_addr learned my addr 172.21.15.114:0/2690454726 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-04-23T10:38:36.474 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:38:36.462+0000 7f6c437fe640 1 -- 172.21.15.114:0/2690454726 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x55c8fd221620 msgr2=0x55c8fd22c130 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T10:38:36.474 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:38:36.463+0000 7f6c437fe640 1 --2- 172.21.15.114:0/2690454726 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x55c8fd221620 0x55c8fd22c130 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:38:36.474 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:38:36.463+0000 7f6c437fe640 1 -- 172.21.15.114:0/2690454726 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55c8fd22c800 msgr2=0x55c8fd229960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T10:38:36.474 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:38:36.463+0000 7f6c437fe640 1 --2- 172.21.15.114:0/2690454726 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55c8fd22c800 0x55c8fd229960 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:38:36.474 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:38:36.463+0000 7f6c437fe640 1 -- 172.21.15.114:0/2690454726 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55c8fcf65620 con 0x55c8fd221f40 2024-04-23T10:38:36.475 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:38:36.463+0000 7f6c42ffd640 1 --2- 172.21.15.114:0/2690454726 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x55c8fd221620 0x55c8fd22c130 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-23T10:38:36.475 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:38:36.463+0000 7f6c437fe640 1 --2- 172.21.15.114:0/2690454726 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x55c8fd221f40 0x55c8fd229420 secure :-1 s=READY pgs=97 cs=0 l=1 rev1=1 crypto rx=0x7f6c34011440 tx=0x7f6c34011900 comp rx=0 tx=0).ready entity=mon.2 client_cookie=a3e534f52a5d6320 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T10:38:36.475 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:38:36.463+0000 7f6c22ffd640 10 client.0 ms_handle_connect on v2:172.21.15.119:3300/0 2024-04-23T10:38:36.475 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:38:36.463+0000 7f6c22ffd640 1 -- 172.21.15.114:0/2690454726 <== mon.2 v2:172.21.15.119:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6c3401f060 con 0x55c8fd221f40 2024-04-23T10:38:36.475 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:38:36.463+0000 7f6c4c96f740 1 -- 172.21.15.114:0/2690454726 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f6c380047a0 con 0x55c8fd221f40 2024-04-23T10:38:36.475 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:38:36.463+0000 7f6c22ffd640 1 -- 172.21.15.114:0/2690454726 <== mon.2 v2:172.21.15.119:3300/0 2 ==== osd_map(106..106 src has 1..106) ==== 8629+0+0 (secure 0 0 0) 0x7f6c34010700 con 0x55c8fd221f40 2024-04-23T10:38:36.475 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:38:36.463+0000 7f6c22ffd640 1 -- 172.21.15.114:0/2690454726 <== mon.2 v2:172.21.15.119:3300/0 3 ==== mon_get_version_reply(handle=1 version=41) ==== 24+0+0 (secure 0 0 0) 0x7f6c34004020 con 0x55c8fd221f40 2024-04-23T10:38:36.475 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:38:36.463+0000 7f6c4c96f740 10 client.26249 fetch_fsmap learned FSMap version 41 2024-04-23T10:38:36.475 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:38:36.463+0000 7f6c4c96f740 1 -- 172.21.15.114:0/2690454726 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({config=0+,fsmap=41,monmap=4+}) -- 0x55c8fd23b430 con 0x55c8fd221f40 2024-04-23T10:38:36.475 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:38:36.463+0000 7f6c22ffd640 1 -- 172.21.15.114:0/2690454726 <== mon.2 v2:172.21.15.119:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f6c380047a0 con 0x55c8fd221f40 2024-04-23T10:38:36.475 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:38:36.464+0000 7f6c22ffd640 1 -- 172.21.15.114:0/2690454726 <== mon.2 v2:172.21.15.119:3300/0 5 ==== fsmap(e 41) ==== 8342+0+0 (secure 0 0 0) 0x7f6c34003860 con 0x55c8fd221f40 2024-04-23T10:38:36.476 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:38:36.464+0000 7f6c4c96f740 10 client.26249 fetch_fsmap finished waiting for FSMap version 41 2024-04-23T10:38:36.476 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:38:36.464+0000 7f6c4c96f740 10 client.26249 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-04-23T10:38:36.476 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:38:36.464+0000 7f6c4c96f740 20 client.26249 populate_metadata read hostname 'smithi114' 2024-04-23T10:38:36.476 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:38:36.464+0000 7f6c4c96f740 1 --2- 172.21.15.114:0/2690454726 >> [v2:172.21.15.119:6838/937297340,v1:172.21.15.119:6839/937297340] conn(0x55c8fd23c8f0 0x55c8fd23ed00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T10:38:36.476 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:38:36.464+0000 7f6c4c96f740 4 client.26249 mds_command: new command op to 24545 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-23T10:38:36.476 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:38:36.464+0000 7f6c4c96f740 1 -- 172.21.15.114:0/2690454726 --> [v2:172.21.15.119:6838/937297340,v1:172.21.15.119:6839/937297340] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55c8fd194390 con 0x55c8fd23c8f0 2024-04-23T10:38:36.476 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:38:36.464+0000 7f6c42ffd640 1 --2- 172.21.15.114:0/2690454726 >> [v2:172.21.15.119:6838/937297340,v1:172.21.15.119:6839/937297340] conn(0x55c8fd23c8f0 0x55c8fd23ed00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T10:38:36.476 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:38:36.464+0000 7f6c22ffd640 10 client.26249 ms_handle_connect on v2:172.21.15.119:6838/937297340 2024-04-23T10:38:36.476 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:38:36.464+0000 7f6c42ffd640 1 --2- 172.21.15.114:0/2690454726 >> [v2:172.21.15.119:6838/937297340,v1:172.21.15.119:6839/937297340] conn(0x55c8fd23c8f0 0x55c8fd23ed00 crc :-1 s=READY pgs=168 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=e82cb99bffec3b62 server_cookie=1b6d256ac7b2efb7 in_seq=0 out_seq=0 2024-04-23T10:38:36.476 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:38:36.465+0000 7f6c22ffd640 1 -- 172.21.15.114:0/2690454726 <== mds.0 v2:172.21.15.119:6838/937297340 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x55c8fd194390 con 0x55c8fd23c8f0 2024-04-23T10:38:36.477 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:38:36.465+0000 7f6c22ffd640 10 client.26249 handle_command_reply: tid=0 2024-04-23T10:38:36.477 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:38:36.465+0000 7f6c4c96f740 1 client.26249 shutdown 2024-04-23T10:38:36.477 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:38:36.465+0000 7f6c4c96f740 1 -- 172.21.15.114:0/2690454726 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x55c8fd221f40 msgr2=0x55c8fd229420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T10:38:36.477 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:38:36.465+0000 7f6c4c96f740 1 --2- 172.21.15.114:0/2690454726 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x55c8fd221f40 0x55c8fd229420 secure :-1 s=READY pgs=97 cs=0 l=1 rev1=1 crypto rx=0x7f6c34011440 tx=0x7f6c34011900 comp rx=0 tx=0).stop 2024-04-23T10:38:36.477 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:38:36.465+0000 7f6c4c96f740 1 -- 172.21.15.114:0/2690454726 shutdown_connections 2024-04-23T10:38:36.477 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:38:36.465+0000 7f6c4c96f740 1 --2- 172.21.15.114:0/2690454726 >> [v2:172.21.15.119:6838/937297340,v1:172.21.15.119:6839/937297340] conn(0x55c8fd23c8f0 0x55c8fd23ed00 crc :-1 s=READY pgs=168 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:38:36.477 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:38:36.465+0000 7f6c4c96f740 1 --2- 172.21.15.114:0/2690454726 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55c8fd22c800 0x55c8fd229960 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:38:36.477 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:38:36.465+0000 7f6c4c96f740 1 --2- 172.21.15.114:0/2690454726 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x55c8fd221f40 0x55c8fd229420 unknown :-1 s=CLOSED pgs=97 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:38:36.477 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:38:36.465+0000 7f6c4c96f740 1 --2- 172.21.15.114:0/2690454726 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x55c8fd221620 0x55c8fd22c130 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:38:36.478 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:38:36.465+0000 7f6c22ffd640 0 client.26249 ms_handle_reset on v2:172.21.15.119:6838/937297340 2024-04-23T10:38:36.478 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:38:36.465+0000 7f6c4c96f740 1 -- 172.21.15.114:0/2690454726 >> 172.21.15.114:0/2690454726 conn(0x55c8fd21db30 msgr2=0x55c8fd2238b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T10:38:36.478 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:38:36.465+0000 7f6c4c96f740 1 -- 172.21.15.114:0/2690454726 shutdown_connections 2024-04-23T10:38:36.478 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:38:36.465+0000 7f6c4c96f740 1 -- 172.21.15.114:0/2690454726 wait complete. 2024-04-23T10:38:36.478 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:38:36.465+0000 7f6c4c96f740 20 client.26249 trim_cache size 0 max 16384 2024-04-23T10:38:36.487 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:38:36.480+0000 7f6c4c96f740 1 Processor -- start 2024-04-23T10:38:36.488 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:38:36.480+0000 7f6c4c96f740 1 -- start start 2024-04-23T10:38:36.488 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:38:36.480+0000 7f6c4c96f740 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55c8fd221620 0x55c8fd2382f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T10:38:36.488 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:38:36.480+0000 7f6c4c96f740 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x55c8fd221f40 0x55c8fd21cb30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T10:38:36.488 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:38:36.480+0000 7f6c4c96f740 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x55c8fd22c800 0x55c8fd244610 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T10:38:36.488 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:38:36.480+0000 7f6c4c96f740 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x7f6c441b0590 con 0x55c8fd221620 2024-04-23T10:38:36.488 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:38:36.480+0000 7f6c4c96f740 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 -- 0x7f6c441ade30 con 0x55c8fd221f40 2024-04-23T10:38:36.488 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:38:36.480+0000 7f6c4c96f740 1 -- --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_getmap magic: 0 -- 0x7f6c441abca0 con 0x55c8fd22c800 2024-04-23T10:38:36.488 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:38:36.480+0000 7f6c42ffd640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55c8fd221620 0x55c8fd2382f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T10:38:36.489 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:38:36.480+0000 7f6c42ffd640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55c8fd221620 0x55c8fd2382f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.114:41986/0 (socket says 172.21.15.114:41986) 2024-04-23T10:38:36.489 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:38:36.480+0000 7f6c42ffd640 1 -- 172.21.15.114:0/1195607032 learned_addr learned my addr 172.21.15.114:0/1195607032 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-04-23T10:38:36.489 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:38:36.480+0000 7f6c437fe640 1 --2- 172.21.15.114:0/1195607032 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x55c8fd221f40 0x55c8fd21cb30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T10:38:36.489 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:38:36.481+0000 7f6c42ffd640 1 -- 172.21.15.114:0/1195607032 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x55c8fd22c800 msgr2=0x55c8fd244610 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T10:38:36.489 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:38:36.481+0000 7f6c42ffd640 1 --2- 172.21.15.114:0/1195607032 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x55c8fd22c800 0x55c8fd244610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:38:36.489 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:38:36.481+0000 7f6c42ffd640 1 -- 172.21.15.114:0/1195607032 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x55c8fd221f40 msgr2=0x55c8fd21cb30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T10:38:36.489 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:38:36.481+0000 7f6c42ffd640 1 --2- 172.21.15.114:0/1195607032 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x55c8fd221f40 0x55c8fd21cb30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:38:36.489 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:38:36.481+0000 7f6c42ffd640 1 -- 172.21.15.114:0/1195607032 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55c8fd221d70 con 0x55c8fd221620 2024-04-23T10:38:36.489 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:38:36.481+0000 7f6c42ffd640 1 --2- 172.21.15.114:0/1195607032 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55c8fd221620 0x55c8fd2382f0 secure :-1 s=READY pgs=884 cs=0 l=1 rev1=1 crypto rx=0x7f6c38011fd0 tx=0x7f6c3801a040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2fa3c4b61300b2c6 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T10:38:36.489 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:38:36.481+0000 7f6c297fa640 1 -- 172.21.15.114:0/1195607032 <== mon.0 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6c38028040 con 0x55c8fd221620 2024-04-23T10:38:36.489 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:38:36.481+0000 7f6c297fa640 1 -- 172.21.15.114:0/1195607032 <== mon.0 v2:172.21.15.114:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f6c38030530 con 0x55c8fd221620 2024-04-23T10:38:36.490 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:38:36.481+0000 7f6c297fa640 1 -- 172.21.15.114:0/1195607032 <== mon.0 v2:172.21.15.114:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6c3802e6e0 con 0x55c8fd221620 2024-04-23T10:38:36.490 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:38:36.481+0000 7f6c4c96f740 1 -- 172.21.15.114:0/1195607032 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55c8fd221620 msgr2=0x55c8fd2382f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T10:38:36.490 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:38:36.481+0000 7f6c4c96f740 1 --2- 172.21.15.114:0/1195607032 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55c8fd221620 0x55c8fd2382f0 secure :-1 s=READY pgs=884 cs=0 l=1 rev1=1 crypto rx=0x7f6c38011fd0 tx=0x7f6c3801a040 comp rx=0 tx=0).stop 2024-04-23T10:38:36.490 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:38:36.482+0000 7f6c4c96f740 1 -- 172.21.15.114:0/1195607032 shutdown_connections 2024-04-23T10:38:36.490 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:38:36.482+0000 7f6c4c96f740 1 --2- 172.21.15.114:0/1195607032 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x55c8fd22c800 0x55c8fd244610 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:38:36.490 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:38:36.482+0000 7f6c4c96f740 1 --2- 172.21.15.114:0/1195607032 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x55c8fd221f40 0x55c8fd21cb30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:38:36.490 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:38:36.482+0000 7f6c4c96f740 1 --2- 172.21.15.114:0/1195607032 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55c8fd221620 0x55c8fd2382f0 unknown :-1 s=CLOSED pgs=884 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:38:36.490 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:38:36.482+0000 7f6c4c96f740 1 -- 172.21.15.114:0/1195607032 >> 172.21.15.114:0/1195607032 conn(0x55c8fd21db30 msgr2=0x55c8fd21c420 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T10:38:36.490 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:38:36.482+0000 7f6c4c96f740 1 -- 172.21.15.114:0/1195607032 shutdown_connections 2024-04-23T10:38:36.490 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:38:36.482+0000 7f6c4c96f740 1 -- 172.21.15.114:0/1195607032 wait complete. 2024-04-23T10:38:36.490 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:38:36.482+0000 7f6c4c96f740 1 -- start start 2024-04-23T10:38:36.491 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:38:36.482+0000 7f6c4c96f740 1 Processor -- start 2024-04-23T10:38:36.491 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:38:36.482+0000 7f6c4c96f740 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x55c8fd221620 0x55c8fd2319d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T10:38:36.491 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:38:36.482+0000 7f6c4c96f740 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55c8fd221f40 0x55c8fd231f10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T10:38:36.491 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:38:36.482+0000 7f6c4c96f740 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x55c8fd22c800 0x55c8fd2a5290 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T10:38:36.491 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:38:36.482+0000 7f6c4c96f740 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x7f6c441ac2f0 con 0x55c8fd221f40 2024-04-23T10:38:36.491 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:38:36.482+0000 7f6c4c96f740 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 -- 0x7f6c441ad760 con 0x55c8fd221620 2024-04-23T10:38:36.491 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:38:36.482+0000 7f6c4c96f740 1 -- --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_getmap magic: 0 -- 0x7f6c441ac9c0 con 0x55c8fd22c800 2024-04-23T10:38:36.491 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:38:36.482+0000 7f6c4c96f740 10 client.0 _ll_register_callbacks cb 0x7f6c3400baf0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-23T10:38:36.491 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:38:36.482+0000 7f6c437fe640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55c8fd221f40 0x55c8fd231f10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T10:38:36.492 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:38:36.482+0000 7f6c437fe640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55c8fd221f40 0x55c8fd231f10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.114:41988/0 (socket says 172.21.15.114:41988) 2024-04-23T10:38:36.492 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:38:36.482+0000 7f6c437fe640 1 -- 172.21.15.114:0/738240773 learned_addr learned my addr 172.21.15.114:0/738240773 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-04-23T10:38:36.492 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:38:36.482+0000 7f6c42ffd640 1 --2- 172.21.15.114:0/738240773 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x55c8fd221620 0x55c8fd2319d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T10:38:36.492 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:38:36.483+0000 7f6c42ffd640 1 --2- 172.21.15.114:0/738240773 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x55c8fd22c800 0x55c8fd2a5290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T10:38:36.493 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:38:36.483+0000 7f6c437fe640 1 -- 172.21.15.114:0/738240773 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x55c8fd22c800 msgr2=0x55c8fd2a5290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T10:38:36.493 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:38:36.483+0000 7f6c437fe640 1 --2- 172.21.15.114:0/738240773 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x55c8fd22c800 0x55c8fd2a5290 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:38:36.493 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:38:36.483+0000 7f6c437fe640 1 -- 172.21.15.114:0/738240773 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x55c8fd221620 msgr2=0x55c8fd2319d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T10:38:36.493 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:38:36.483+0000 7f6c437fe640 1 --2- 172.21.15.114:0/738240773 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x55c8fd221620 0x55c8fd2319d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:38:36.493 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:38:36.483+0000 7f6c437fe640 1 -- 172.21.15.114:0/738240773 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55c8fd2a5800 con 0x55c8fd221f40 2024-04-23T10:38:36.493 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:38:36.483+0000 7f6c42ffd640 1 --2- 172.21.15.114:0/738240773 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x55c8fd22c800 0x55c8fd2a5290 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-23T10:38:36.493 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:38:36.483+0000 7f6c437fe640 1 --2- 172.21.15.114:0/738240773 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55c8fd221f40 0x55c8fd231f10 secure :-1 s=READY pgs=885 cs=0 l=1 rev1=1 crypto rx=0x55c8fd22a9f0 tx=0x7f6c3400b930 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9766f564f358050a server_cookie=0 in_seq=0 out_seq=0 2024-04-23T10:38:36.493 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:38:36.483+0000 7f6c22ffd640 10 client.0 ms_handle_connect on v2:172.21.15.114:3300/0 2024-04-23T10:38:36.493 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:38:36.483+0000 7f6c22ffd640 1 -- 172.21.15.114:0/738240773 <== mon.0 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6c3400b480 con 0x55c8fd221f40 2024-04-23T10:38:36.494 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:38:36.483+0000 7f6c4c96f740 1 -- 172.21.15.114:0/738240773 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f6c38030530 con 0x55c8fd221f40 2024-04-23T10:38:36.494 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:38:36.483+0000 7f6c22ffd640 1 -- 172.21.15.114:0/738240773 <== mon.0 v2:172.21.15.114:3300/0 2 ==== osd_map(106..106 src has 1..106) ==== 8629+0+0 (secure 0 0 0) 0x7f6c34003800 con 0x55c8fd221f40 2024-04-23T10:38:36.494 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:38:36.483+0000 7f6c22ffd640 1 -- 172.21.15.114:0/738240773 <== mon.0 v2:172.21.15.114:3300/0 3 ==== mon_get_version_reply(handle=1 version=41) ==== 24+0+0 (secure 0 0 0) 0x7f6c3401f020 con 0x55c8fd221f40 2024-04-23T10:38:36.494 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:38:36.483+0000 7f6c4c96f740 10 client.16929 fetch_fsmap learned FSMap version 41 2024-04-23T10:38:36.494 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:38:36.483+0000 7f6c4c96f740 1 -- 172.21.15.114:0/738240773 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({config=0+,fsmap=41,monmap=4+}) -- 0x55c8fd2a6b40 con 0x55c8fd221f40 2024-04-23T10:38:36.494 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:38:36.483+0000 7f6c22ffd640 1 -- 172.21.15.114:0/738240773 <== mon.0 v2:172.21.15.114:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f6c38030530 con 0x55c8fd221f40 2024-04-23T10:38:36.494 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:38:36.483+0000 7f6c22ffd640 1 -- 172.21.15.114:0/738240773 <== mon.0 v2:172.21.15.114:3300/0 5 ==== fsmap(e 41) ==== 8342+0+0 (secure 0 0 0) 0x7f6c34010820 con 0x55c8fd221f40 2024-04-23T10:38:36.494 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:38:36.484+0000 7f6c4c96f740 10 client.16929 fetch_fsmap finished waiting for FSMap version 41 2024-04-23T10:38:36.494 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:38:36.484+0000 7f6c4c96f740 10 client.16929 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-04-23T10:38:36.494 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:38:36.484+0000 7f6c4c96f740 20 client.16929 populate_metadata read hostname 'smithi114' 2024-04-23T10:38:36.495 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:38:36.484+0000 7f6c4c96f740 1 --2- 172.21.15.114:0/738240773 >> [v2:172.21.15.119:6838/937297340,v1:172.21.15.119:6839/937297340] conn(0x55c8fd2a7360 0x55c8fd2a9780 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T10:38:36.495 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:38:36.484+0000 7f6c4c96f740 4 client.16929 mds_command: new command op to 24545 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-04-23T10:38:36.495 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:38:36.484+0000 7f6c4c96f740 1 -- 172.21.15.114:0/738240773 --> [v2:172.21.15.119:6838/937297340,v1:172.21.15.119:6839/937297340] -- command(tid 0: {"prefix": "flush journal"}) -- 0x55c8fd229cf0 con 0x55c8fd2a7360 2024-04-23T10:38:36.495 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:38:36.484+0000 7f6c42ffd640 1 --2- 172.21.15.114:0/738240773 >> [v2:172.21.15.119:6838/937297340,v1:172.21.15.119:6839/937297340] conn(0x55c8fd2a7360 0x55c8fd2a9780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T10:38:36.495 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:38:36.484+0000 7f6c42ffd640 1 --2- 172.21.15.114:0/738240773 >> [v2:172.21.15.119:6838/937297340,v1:172.21.15.119:6839/937297340] conn(0x55c8fd2a7360 0x55c8fd2a9780 crc :-1 s=READY pgs=169 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=982de515f980e200 server_cookie=793aa9418f933607 in_seq=0 out_seq=0 2024-04-23T10:38:36.495 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:38:36.484+0000 7f6c22ffd640 10 client.16929 ms_handle_connect on v2:172.21.15.119:6838/937297340 2024-04-23T10:38:36.529 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:38:36.529+0000 7f6c22ffd640 1 -- 172.21.15.114:0/738240773 <== mds.0 v2:172.21.15.119:6838/937297340 1 ==== command_reply(tid 0: 0 ) ==== 8+0+44 (crc 0 0 0) 0x55c8fd229cf0 con 0x55c8fd2a7360 2024-04-23T10:38:36.529 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:38:36.529+0000 7f6c22ffd640 10 client.16929 handle_command_reply: tid=0 2024-04-23T10:38:36.529 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:38:36.530+0000 7f6c4c96f740 1 client.16929 shutdown 2024-04-23T10:38:36.529 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:38:36.530+0000 7f6c4c96f740 1 -- 172.21.15.114:0/738240773 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55c8fd221f40 msgr2=0x55c8fd231f10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T10:38:36.529 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:38:36.530+0000 7f6c4c96f740 1 --2- 172.21.15.114:0/738240773 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55c8fd221f40 0x55c8fd231f10 secure :-1 s=READY pgs=885 cs=0 l=1 rev1=1 crypto rx=0x55c8fd22a9f0 tx=0x7f6c3400b930 comp rx=0 tx=0).stop 2024-04-23T10:38:36.529 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:38:36.530+0000 7f6c4c96f740 1 -- 172.21.15.114:0/738240773 shutdown_connections 2024-04-23T10:38:36.530 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:38:36.530+0000 7f6c4c96f740 1 --2- 172.21.15.114:0/738240773 >> [v2:172.21.15.119:6838/937297340,v1:172.21.15.119:6839/937297340] conn(0x55c8fd2a7360 0x55c8fd2a9780 crc :-1 s=READY pgs=169 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:38:36.530 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:38:36.530+0000 7f6c4c96f740 1 --2- 172.21.15.114:0/738240773 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x55c8fd22c800 0x55c8fd2a5290 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:38:36.530 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:38:36.530+0000 7f6c4c96f740 1 --2- 172.21.15.114:0/738240773 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55c8fd221f40 0x55c8fd231f10 unknown :-1 s=CLOSED pgs=885 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:38:36.530 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:38:36.530+0000 7f6c22ffd640 0 client.16929 ms_handle_reset on v2:172.21.15.119:6838/937297340 2024-04-23T10:38:36.530 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:38:36.530+0000 7f6c4c96f740 1 --2- 172.21.15.114:0/738240773 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x55c8fd221620 0x55c8fd2319d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:38:36.530 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:38:36.530+0000 7f6c4c96f740 1 -- 172.21.15.114:0/738240773 >> 172.21.15.114:0/738240773 conn(0x55c8fd21db30 msgr2=0x55c8fd21c420 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T10:38:36.530 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:38:36.530+0000 7f6c4c96f740 1 -- 172.21.15.114:0/738240773 shutdown_connections 2024-04-23T10:38:36.530 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:38:36.530+0000 7f6c4c96f740 1 -- 172.21.15.114:0/738240773 wait complete. 2024-04-23T10:38:36.530 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:38:36.530+0000 7f6c4c96f740 20 client.16929 trim_cache size 0 max 16384 2024-04-23T10:38:36.531 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stdout:{ 2024-04-23T10:38:36.531 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stdout: "message": "", 2024-04-23T10:38:36.531 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stdout: "return_code": 0 2024-04-23T10:38:36.531 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stdout:} 2024-04-23T10:38:36.532 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:38:36.531+0000 7f6c49b76640 1 -- 172.21.15.114:0/2265813900 >> [v2:172.21.15.114:6800/1395693466,v1:172.21.15.114:6801/1395693466] conn(0x7f6c2c0a3e40 msgr2=0x7f6c2c0a6300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T10:38:36.532 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:38:36.531+0000 7f6c49b76640 1 --2- 172.21.15.114:0/2265813900 >> [v2:172.21.15.114:6800/1395693466,v1:172.21.15.114:6801/1395693466] conn(0x7f6c2c0a3e40 0x7f6c2c0a6300 secure :-1 s=READY pgs=138 cs=0 l=1 rev1=1 crypto rx=0x7f6c3c0bd2b0 tx=0x7f6c3c0b2a40 comp rx=0 tx=0).stop 2024-04-23T10:38:36.532 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:38:36.531+0000 7f6c49b76640 1 -- 172.21.15.114:0/2265813900 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f6c4410c5d0 msgr2=0x7f6c4419ce50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T10:38:36.532 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:38:36.531+0000 7f6c49b76640 1 --2- 172.21.15.114:0/2265813900 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f6c4410c5d0 0x7f6c4419ce50 secure :-1 s=READY pgs=883 cs=0 l=1 rev1=1 crypto rx=0x7f6c3c02ae70 tx=0x7f6c3c02f5d0 comp rx=0 tx=0).stop 2024-04-23T10:38:36.532 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:38:36.531+0000 7f6c49b76640 1 -- 172.21.15.114:0/2265813900 shutdown_connections 2024-04-23T10:38:36.532 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:38:36.531+0000 7f6c49b76640 1 --2- 172.21.15.114:0/2265813900 >> [v2:172.21.15.114:6800/1395693466,v1:172.21.15.114:6801/1395693466] conn(0x7f6c2c0a3e40 0x7f6c2c0a6300 unknown :-1 s=CLOSED pgs=138 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:38:36.532 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:38:36.531+0000 7f6c49b76640 1 --2- 172.21.15.114:0/2265813900 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f6c4410c5d0 0x7f6c4419ce50 unknown :-1 s=CLOSED pgs=883 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:38:36.532 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:38:36.531+0000 7f6c49b76640 1 --2- 172.21.15.114:0/2265813900 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f6c44103b20 0x7f6c44198ad0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:38:36.532 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:38:36.531+0000 7f6c49b76640 1 --2- 172.21.15.114:0/2265813900 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f6c44103200 0x7f6c44198590 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:38:36.533 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:38:36.531+0000 7f6c49b76640 1 -- 172.21.15.114:0/2265813900 >> 172.21.15.114:0/2265813900 conn(0x7f6c440771d0 msgr2=0x7f6c440fdc80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T10:38:36.533 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:38:36.531+0000 7f6c49b76640 1 -- 172.21.15.114:0/2265813900 shutdown_connections 2024-04-23T10:38:36.533 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:38:36.531+0000 7f6c49b76640 1 -- 172.21.15.114:0/2265813900 wait complete. 2024-04-23T10:38:36.947 INFO:tasks.workunit.client.0.smithi114.stdout: 1 696108 5.35 MB/sec execute 525 sec latency 761.921 ms 2024-04-23T10:38:37.948 INFO:tasks.workunit.client.0.smithi114.stdout: 1 696108 5.34 MB/sec execute 526 sec latency 1762.035 ms 2024-04-23T10:38:37.992 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 23 10:38:37 smithi119 ceph-mon[32608]: pgmap v377: 97 pgs: 97 active+clean; 222 MiB data, 2.7 GiB used, 1.0 TiB / 1.0 TiB avail; 579 KiB/s rd, 888 KiB/s wr, 106 op/s 2024-04-23T10:38:38.004 INFO:journalctl@ceph.mon.a.smithi114.stdout:Apr 23 10:38:37 smithi114 ceph-mon[29255]: pgmap v377: 97 pgs: 97 active+clean; 222 MiB data, 2.7 GiB used, 1.0 TiB / 1.0 TiB avail; 579 KiB/s rd, 888 KiB/s wr, 106 op/s 2024-04-23T10:38:38.078 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 23 10:38:37 smithi204 ceph-mon[32649]: pgmap v377: 97 pgs: 97 active+clean; 222 MiB data, 2.7 GiB used, 1.0 TiB / 1.0 TiB avail; 579 KiB/s rd, 888 KiB/s wr, 106 op/s 2024-04-23T10:38:38.948 INFO:tasks.workunit.client.0.smithi114.stdout: 1 696108 5.33 MB/sec execute 527 sec latency 2762.156 ms 2024-04-23T10:38:39.948 INFO:tasks.workunit.client.0.smithi114.stdout: 1 696175 5.32 MB/sec execute 528 sec latency 2998.608 ms 2024-04-23T10:38:39.992 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 23 10:38:39 smithi119 ceph-mon[32608]: pgmap v378: 97 pgs: 97 active+clean; 222 MiB data, 2.7 GiB used, 1.0 TiB / 1.0 TiB avail; 294 KiB/s rd, 481 KiB/s wr, 51 op/s 2024-04-23T10:38:39.992 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 23 10:38:39 smithi119 ceph-mon[32608]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=e=up:active} 3 up:standby-replay 6 up:standby 2024-04-23T10:38:40.004 INFO:journalctl@ceph.mon.a.smithi114.stdout:Apr 23 10:38:39 smithi114 ceph-mon[29255]: pgmap v378: 97 pgs: 97 active+clean; 222 MiB data, 2.7 GiB used, 1.0 TiB / 1.0 TiB avail; 294 KiB/s rd, 481 KiB/s wr, 51 op/s 2024-04-23T10:38:40.004 INFO:journalctl@ceph.mon.a.smithi114.stdout:Apr 23 10:38:39 smithi114 ceph-mon[29255]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=e=up:active} 3 up:standby-replay 6 up:standby 2024-04-23T10:38:40.078 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 23 10:38:39 smithi204 ceph-mon[32649]: pgmap v378: 97 pgs: 97 active+clean; 222 MiB data, 2.7 GiB used, 1.0 TiB / 1.0 TiB avail; 294 KiB/s rd, 481 KiB/s wr, 51 op/s 2024-04-23T10:38:40.078 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 23 10:38:39 smithi204 ceph-mon[32649]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=e=up:active} 3 up:standby-replay 6 up:standby 2024-04-23T10:38:40.948 INFO:tasks.workunit.client.0.smithi114.stdout: 1 696175 5.31 MB/sec execute 529 sec latency 1656.215 ms 2024-04-23T10:38:41.948 INFO:tasks.workunit.client.0.smithi114.stdout: 1 696175 5.30 MB/sec execute 530 sec latency 2656.292 ms 2024-04-23T10:38:41.992 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 23 10:38:41 smithi119 ceph-mon[32608]: pgmap v379: 97 pgs: 97 active+clean; 222 MiB data, 2.7 GiB used, 1.0 TiB / 1.0 TiB avail; 294 KiB/s rd, 481 KiB/s wr, 51 op/s 2024-04-23T10:38:42.004 INFO:journalctl@ceph.mon.a.smithi114.stdout:Apr 23 10:38:41 smithi114 ceph-mon[29255]: pgmap v379: 97 pgs: 97 active+clean; 222 MiB data, 2.7 GiB used, 1.0 TiB / 1.0 TiB avail; 294 KiB/s rd, 481 KiB/s wr, 51 op/s 2024-04-23T10:38:42.078 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 23 10:38:41 smithi204 ceph-mon[32649]: pgmap v379: 97 pgs: 97 active+clean; 222 MiB data, 2.7 GiB used, 1.0 TiB / 1.0 TiB avail; 294 KiB/s rd, 481 KiB/s wr, 51 op/s 2024-04-23T10:38:42.948 INFO:tasks.workunit.client.0.smithi114.stdout: 1 696175 5.29 MB/sec execute 531 sec latency 3656.333 ms 2024-04-23T10:38:43.948 INFO:tasks.workunit.client.0.smithi114.stdout: 1 696175 5.28 MB/sec execute 532 sec latency 4656.424 ms 2024-04-23T10:38:43.992 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 23 10:38:43 smithi119 ceph-mon[32608]: pgmap v380: 97 pgs: 97 active+clean; 221 MiB data, 2.7 GiB used, 1.0 TiB / 1.0 TiB avail; 356 KiB/s rd, 729 KiB/s wr, 81 op/s 2024-04-23T10:38:44.004 INFO:journalctl@ceph.mon.a.smithi114.stdout:Apr 23 10:38:43 smithi114 ceph-mon[29255]: pgmap v380: 97 pgs: 97 active+clean; 221 MiB data, 2.7 GiB used, 1.0 TiB / 1.0 TiB avail; 356 KiB/s rd, 729 KiB/s wr, 81 op/s 2024-04-23T10:38:44.078 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 23 10:38:43 smithi204 ceph-mon[32649]: pgmap v380: 97 pgs: 97 active+clean; 221 MiB data, 2.7 GiB used, 1.0 TiB / 1.0 TiB avail; 356 KiB/s rd, 729 KiB/s wr, 81 op/s 2024-04-23T10:38:44.948 INFO:tasks.workunit.client.0.smithi114.stdout: 1 696841 5.28 MB/sec execute 533 sec latency 4898.961 ms 2024-04-23T10:38:45.948 INFO:tasks.workunit.client.0.smithi114.stdout: 1 699033 5.29 MB/sec execute 534 sec latency 18.403 ms 2024-04-23T10:38:45.992 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 23 10:38:45 smithi119 ceph-mon[32608]: pgmap v381: 97 pgs: 97 active+clean; 221 MiB data, 2.7 GiB used, 1.0 TiB / 1.0 TiB avail; 303 KiB/s rd, 694 KiB/s wr, 69 op/s 2024-04-23T10:38:46.004 INFO:journalctl@ceph.mon.a.smithi114.stdout:Apr 23 10:38:45 smithi114 ceph-mon[29255]: pgmap v381: 97 pgs: 97 active+clean; 221 MiB data, 2.7 GiB used, 1.0 TiB / 1.0 TiB avail; 303 KiB/s rd, 694 KiB/s wr, 69 op/s 2024-04-23T10:38:46.078 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 23 10:38:45 smithi204 ceph-mon[32649]: pgmap v381: 97 pgs: 97 active+clean; 221 MiB data, 2.7 GiB used, 1.0 TiB / 1.0 TiB avail; 303 KiB/s rd, 694 KiB/s wr, 69 op/s 2024-04-23T10:38:46.948 INFO:tasks.workunit.client.0.smithi114.stdout: 1 699654 5.29 MB/sec execute 535 sec latency 750.612 ms 2024-04-23T10:38:47.948 INFO:tasks.workunit.client.0.smithi114.stdout: 1 699654 5.28 MB/sec execute 536 sec latency 1750.757 ms 2024-04-23T10:38:47.992 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 23 10:38:47 smithi119 ceph-mon[32608]: pgmap v382: 97 pgs: 97 active+clean; 222 MiB data, 2.7 GiB used, 1.0 TiB / 1.0 TiB avail; 568 KiB/s rd, 1.1 MiB/s wr, 117 op/s 2024-04-23T10:38:48.004 INFO:journalctl@ceph.mon.a.smithi114.stdout:Apr 23 10:38:47 smithi114 ceph-mon[29255]: pgmap v382: 97 pgs: 97 active+clean; 222 MiB data, 2.7 GiB used, 1.0 TiB / 1.0 TiB avail; 568 KiB/s rd, 1.1 MiB/s wr, 117 op/s 2024-04-23T10:38:48.078 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 23 10:38:47 smithi204 ceph-mon[32649]: pgmap v382: 97 pgs: 97 active+clean; 222 MiB data, 2.7 GiB used, 1.0 TiB / 1.0 TiB avail; 568 KiB/s rd, 1.1 MiB/s wr, 117 op/s 2024-04-23T10:38:48.948 INFO:tasks.workunit.client.0.smithi114.stdout: 1 699654 5.27 MB/sec execute 537 sec latency 2750.851 ms 2024-04-23T10:38:49.628 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:38:49.626+0000 7fbef075d640 1 -- 172.21.15.114:0/1196350171 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fbee810c570 msgr2=0x7fbee810e9e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T10:38:49.628 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:38:49.626+0000 7fbef075d640 1 --2- 172.21.15.114:0/1196350171 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fbee810c570 0x7fbee810e9e0 secure :-1 s=READY pgs=886 cs=0 l=1 rev1=1 crypto rx=0x7fbee0009ca0 tx=0x7fbee002a950 comp rx=0 tx=0).stop 2024-04-23T10:38:49.628 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:38:49.626+0000 7fbef075d640 1 -- 172.21.15.114:0/1196350171 shutdown_connections 2024-04-23T10:38:49.628 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:38:49.626+0000 7fbef075d640 1 --2- 172.21.15.114:0/1196350171 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fbee810c570 0x7fbee810e9e0 unknown :-1 s=CLOSED pgs=886 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:38:49.628 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:38:49.626+0000 7fbef075d640 1 --2- 172.21.15.114:0/1196350171 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fbee8103ac0 0x7fbee810bfa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:38:49.628 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:38:49.626+0000 7fbef075d640 1 --2- 172.21.15.114:0/1196350171 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fbee81031a0 0x7fbee8103580 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:38:49.628 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:38:49.626+0000 7fbef075d640 1 -- 172.21.15.114:0/1196350171 >> 172.21.15.114:0/1196350171 conn(0x7fbee8078060 msgr2=0x7fbee8078470 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T10:38:49.628 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:38:49.626+0000 7fbef075d640 1 -- 172.21.15.114:0/1196350171 shutdown_connections 2024-04-23T10:38:49.628 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:38:49.626+0000 7fbef075d640 1 -- 172.21.15.114:0/1196350171 wait complete. 2024-04-23T10:38:49.629 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:38:49.627+0000 7fbef075d640 1 Processor -- start 2024-04-23T10:38:49.629 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:38:49.627+0000 7fbef075d640 1 -- start start 2024-04-23T10:38:49.629 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:38:49.627+0000 7fbef075d640 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fbee81031a0 0x7fbee8100df0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T10:38:49.629 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:38:49.627+0000 7fbef075d640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fbee8103ac0 0x7fbee8101330 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T10:38:49.629 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:38:49.627+0000 7fbef075d640 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fbee810c570 0x7fbee8101870 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T10:38:49.629 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:38:49.627+0000 7fbef075d640 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x7fbee81121d0 con 0x7fbee8103ac0 2024-04-23T10:38:49.629 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:38:49.627+0000 7fbef075d640 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 -- 0x7fbee8112050 con 0x7fbee810c570 2024-04-23T10:38:49.629 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:38:49.627+0000 7fbef075d640 1 -- --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_getmap magic: 0 -- 0x7fbee8112350 con 0x7fbee81031a0 2024-04-23T10:38:49.629 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:38:49.627+0000 7fbeedcf8640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fbee8103ac0 0x7fbee8101330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T10:38:49.629 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:38:49.627+0000 7fbeedcf8640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fbee8103ac0 0x7fbee8101330 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.114:38216/0 (socket says 172.21.15.114:38216) 2024-04-23T10:38:49.629 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:38:49.627+0000 7fbeedcf8640 1 -- 172.21.15.114:0/1371192985 learned_addr learned my addr 172.21.15.114:0/1371192985 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-04-23T10:38:49.629 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:38:49.627+0000 7fbeee4f9640 1 --2- 172.21.15.114:0/1371192985 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fbee81031a0 0x7fbee8100df0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T10:38:49.629 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:38:49.627+0000 7fbeeecfa640 1 --2- 172.21.15.114:0/1371192985 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fbee810c570 0x7fbee8101870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T10:38:49.630 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:38:49.627+0000 7fbeedcf8640 1 -- 172.21.15.114:0/1371192985 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fbee81031a0 msgr2=0x7fbee8100df0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T10:38:49.630 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:38:49.627+0000 7fbeedcf8640 1 --2- 172.21.15.114:0/1371192985 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fbee81031a0 0x7fbee8100df0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:38:49.630 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:38:49.627+0000 7fbeedcf8640 1 -- 172.21.15.114:0/1371192985 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fbee810c570 msgr2=0x7fbee8101870 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T10:38:49.630 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:38:49.627+0000 7fbeedcf8640 1 --2- 172.21.15.114:0/1371192985 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fbee810c570 0x7fbee8101870 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:38:49.630 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:38:49.627+0000 7fbeedcf8640 1 -- 172.21.15.114:0/1371192985 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fbee81a6b10 con 0x7fbee8103ac0 2024-04-23T10:38:49.630 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:38:49.628+0000 7fbeedcf8640 1 --2- 172.21.15.114:0/1371192985 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fbee8103ac0 0x7fbee8101330 secure :-1 s=READY pgs=887 cs=0 l=1 rev1=1 crypto rx=0x7fbed800d950 tx=0x7fbed800de10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T10:38:49.630 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:38:49.628+0000 7fbedf7fe640 1 -- 172.21.15.114:0/1371192985 <== mon.0 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbed8017070 con 0x7fbee8103ac0 2024-04-23T10:38:49.630 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:38:49.628+0000 7fbedf7fe640 1 -- 172.21.15.114:0/1371192985 <== mon.0 v2:172.21.15.114:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fbed8009c30 con 0x7fbee8103ac0 2024-04-23T10:38:49.630 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:38:49.628+0000 7fbef075d640 1 -- 172.21.15.114:0/1371192985 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fbee81a6e00 con 0x7fbee8103ac0 2024-04-23T10:38:49.630 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:38:49.628+0000 7fbedf7fe640 1 -- 172.21.15.114:0/1371192985 <== mon.0 v2:172.21.15.114:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbed800f620 con 0x7fbee8103ac0 2024-04-23T10:38:49.630 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:38:49.628+0000 7fbef075d640 1 -- 172.21.15.114:0/1371192985 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fbee81a72c0 con 0x7fbee8103ac0 2024-04-23T10:38:49.637 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:38:49.630+0000 7fbedf7fe640 1 -- 172.21.15.114:0/1371192985 <== mon.0 v2:172.21.15.114:3300/0 4 ==== mgrmap(e 26) ==== 138778+0+0 (secure 0 0 0) 0x7fbed8010460 con 0x7fbee8103ac0 2024-04-23T10:38:49.637 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:38:49.630+0000 7fbef3474740 1 Processor -- start 2024-04-23T10:38:49.637 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:38:49.631+0000 7fbef3474740 1 -- start start 2024-04-23T10:38:49.638 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:38:49.631+0000 7fbedf7fe640 1 --2- 172.21.15.114:0/1371192985 >> [v2:172.21.15.114:6800/1395693466,v1:172.21.15.114:6801/1395693466] conn(0x7fbed40a3dc0 0x7fbed40a6280 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T10:38:49.638 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:38:49.631+0000 7fbef3474740 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x561b8125f620 0x561b8125fa00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T10:38:49.638 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:38:49.631+0000 7fbef3474740 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x561b8125ff40 0x561b8126a2c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T10:38:49.638 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:38:49.631+0000 7fbef3474740 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x561b8126a800 0x561b8126cbf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T10:38:49.638 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:38:49.631+0000 7fbef3474740 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x561b811c4970 con 0x561b8125f620 2024-04-23T10:38:49.638 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:38:49.631+0000 7fbef3474740 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 -- 0x561b811614a0 con 0x561b8126a800 2024-04-23T10:38:49.638 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:38:49.631+0000 7fbef3474740 1 -- --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_getmap magic: 0 -- 0x561b811c0930 con 0x561b8125ff40 2024-04-23T10:38:49.638 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:38:49.631+0000 7fbeeecfa640 1 --2- 172.21.15.114:0/1371192985 >> [v2:172.21.15.114:6800/1395693466,v1:172.21.15.114:6801/1395693466] conn(0x7fbed40a3dc0 0x7fbed40a6280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T10:38:49.638 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:38:49.631+0000 7fbeee4f9640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x561b8125f620 0x561b8125fa00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T10:38:49.638 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:38:49.631+0000 7fbeee4f9640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x561b8125f620 0x561b8125fa00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.114:38220/0 (socket says 172.21.15.114:38220) 2024-04-23T10:38:49.638 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:38:49.631+0000 7fbeee4f9640 1 -- 172.21.15.114:0/391211683 learned_addr learned my addr 172.21.15.114:0/391211683 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-04-23T10:38:49.638 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:38:49.631+0000 7fbeedcf8640 1 --2- 172.21.15.114:0/391211683 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x561b8125ff40 0x561b8126a2c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T10:38:49.639 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:38:49.631+0000 7fbedf7fe640 1 -- 172.21.15.114:0/1371192985 <== mon.0 v2:172.21.15.114:3300/0 5 ==== osd_map(106..106 src has 1..106) ==== 8629+0+0 (secure 0 0 0) 0x7fbed80d0430 con 0x7fbee8103ac0 2024-04-23T10:38:49.639 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:38:49.631+0000 7fbeeecfa640 1 --2- 172.21.15.114:0/391211683 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x561b8126a800 0x561b8126cbf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T10:38:49.639 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:38:49.631+0000 7fbeee4f9640 1 -- 172.21.15.114:0/391211683 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x561b8125ff40 msgr2=0x561b8126a2c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T10:38:49.639 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:38:49.631+0000 7fbeee4f9640 1 --2- 172.21.15.114:0/391211683 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x561b8125ff40 0x561b8126a2c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:38:49.639 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:38:49.631+0000 7fbeee4f9640 1 -- 172.21.15.114:0/391211683 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x561b8126a800 msgr2=0x561b8126cbf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T10:38:49.639 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:38:49.631+0000 7fbeee4f9640 1 --2- 172.21.15.114:0/391211683 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x561b8126a800 0x561b8126cbf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:38:49.639 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:38:49.631+0000 7fbeee4f9640 1 -- 172.21.15.114:0/391211683 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x561b80ed7210 con 0x561b8125f620 2024-04-23T10:38:49.639 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:38:49.632+0000 7fbeeecfa640 1 --2- 172.21.15.114:0/1371192985 >> [v2:172.21.15.114:6800/1395693466,v1:172.21.15.114:6801/1395693466] conn(0x7fbed40a3dc0 0x7fbed40a6280 secure :-1 s=READY pgs=139 cs=0 l=1 rev1=1 crypto rx=0x7fbee0005fa0 tx=0x7fbee002ffd0 comp rx=0 tx=0).ready entity=mgr.15291 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T10:38:49.639 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:38:49.632+0000 7fbeee4f9640 1 --2- 172.21.15.114:0/391211683 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x561b8125f620 0x561b8125fa00 secure :-1 s=READY pgs=888 cs=0 l=1 rev1=1 crypto rx=0x7fbee400b760 tx=0x7fbee400bc20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5d4d3049f80739b1 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T10:38:49.639 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:38:49.632+0000 7fbedcff9640 1 -- 172.21.15.114:0/391211683 <== mon.0 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbee4003d50 con 0x561b8125f620 2024-04-23T10:38:49.639 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:38:49.632+0000 7fbedcff9640 1 -- 172.21.15.114:0/391211683 <== mon.0 v2:172.21.15.114:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fbee4015070 con 0x561b8125f620 2024-04-23T10:38:49.639 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:38:49.632+0000 7fbedcff9640 1 -- 172.21.15.114:0/391211683 <== mon.0 v2:172.21.15.114:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbee4004cd0 con 0x561b8125f620 2024-04-23T10:38:49.639 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:38:49.633+0000 7fbef3474740 1 -- 172.21.15.114:0/391211683 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x561b8125f620 msgr2=0x561b8125fa00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T10:38:49.639 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:38:49.633+0000 7fbef3474740 1 --2- 172.21.15.114:0/391211683 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x561b8125f620 0x561b8125fa00 secure :-1 s=READY pgs=888 cs=0 l=1 rev1=1 crypto rx=0x7fbee400b760 tx=0x7fbee400bc20 comp rx=0 tx=0).stop 2024-04-23T10:38:49.639 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:38:49.633+0000 7fbef3474740 1 -- 172.21.15.114:0/391211683 shutdown_connections 2024-04-23T10:38:49.640 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:38:49.633+0000 7fbef3474740 1 --2- 172.21.15.114:0/391211683 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x561b8126a800 0x561b8126cbf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:38:49.640 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:38:49.633+0000 7fbef3474740 1 --2- 172.21.15.114:0/391211683 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x561b8125ff40 0x561b8126a2c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:38:49.640 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:38:49.633+0000 7fbef3474740 1 --2- 172.21.15.114:0/391211683 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x561b8125f620 0x561b8125fa00 unknown :-1 s=CLOSED pgs=888 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:38:49.640 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:38:49.633+0000 7fbef3474740 1 -- 172.21.15.114:0/391211683 >> 172.21.15.114:0/391211683 conn(0x561b8125bb30 msgr2=0x561b8125bf10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T10:38:49.640 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:38:49.633+0000 7fbef3474740 1 -- 172.21.15.114:0/391211683 shutdown_connections 2024-04-23T10:38:49.640 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:38:49.633+0000 7fbef3474740 1 -- 172.21.15.114:0/391211683 wait complete. 2024-04-23T10:38:49.640 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:38:49.633+0000 7fbef3474740 1 -- start start 2024-04-23T10:38:49.640 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:38:49.633+0000 7fbef3474740 1 Processor -- start 2024-04-23T10:38:49.640 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:38:49.633+0000 7fbef3474740 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x561b8125f620 0x561b8126a170 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T10:38:49.640 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:38:49.633+0000 7fbef3474740 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x561b8125ff40 0x561b81267460 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T10:38:49.640 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:38:49.633+0000 7fbef3474740 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x561b8126a800 0x561b812679c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T10:38:49.640 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:38:49.633+0000 7fbef3474740 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x561b8126d430 con 0x561b8125f620 2024-04-23T10:38:49.640 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:38:49.633+0000 7fbef3474740 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 -- 0x561b8115f0b0 con 0x561b8126a800 2024-04-23T10:38:49.640 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:38:49.633+0000 7fbef3474740 1 -- --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_getmap magic: 0 -- 0x561b8126d5b0 con 0x561b8125ff40 2024-04-23T10:38:49.641 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:38:49.634+0000 7fbef3474740 10 client.0 _ll_register_callbacks cb 0x561b8114cd30 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-23T10:38:49.641 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:38:49.634+0000 7fbeee4f9640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x561b8125f620 0x561b8126a170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T10:38:49.641 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:38:49.634+0000 7fbeee4f9640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x561b8125f620 0x561b8126a170 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.114:38228/0 (socket says 172.21.15.114:38228) 2024-04-23T10:38:49.641 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:38:49.634+0000 7fbeee4f9640 1 -- 172.21.15.114:0/856141312 learned_addr learned my addr 172.21.15.114:0/856141312 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-04-23T10:38:49.641 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:38:49.634+0000 7fbeedcf8640 1 --2- 172.21.15.114:0/856141312 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x561b8125ff40 0x561b81267460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T10:38:49.641 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:38:49.634+0000 7fbeeecfa640 1 --2- 172.21.15.114:0/856141312 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x561b8126a800 0x561b812679c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T10:38:49.641 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:38:49.634+0000 7fbeee4f9640 1 -- 172.21.15.114:0/856141312 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x561b8125ff40 msgr2=0x561b81267460 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T10:38:49.641 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:38:49.634+0000 7fbeee4f9640 1 --2- 172.21.15.114:0/856141312 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x561b8125ff40 0x561b81267460 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:38:49.641 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:38:49.634+0000 7fbeee4f9640 1 -- 172.21.15.114:0/856141312 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x561b8126a800 msgr2=0x561b812679c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T10:38:49.641 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:38:49.634+0000 7fbeee4f9640 1 --2- 172.21.15.114:0/856141312 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x561b8126a800 0x561b812679c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:38:49.641 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:38:49.634+0000 7fbeee4f9640 1 -- 172.21.15.114:0/856141312 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({osdmap=0}) -- 0x561b80fa3620 con 0x561b8125f620 2024-04-23T10:38:49.641 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:38:49.634+0000 7fbeee4f9640 1 --2- 172.21.15.114:0/856141312 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x561b8125f620 0x561b8126a170 secure :-1 s=READY pgs=889 cs=0 l=1 rev1=1 crypto rx=0x7fbee4006010 tx=0x7fbee4003d80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7f3934e08c9a36c9 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T10:38:49.641 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:38:49.634+0000 7fbed1ffb640 10 client.0 ms_handle_connect on v2:172.21.15.114:3300/0 2024-04-23T10:38:49.641 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:38:49.634+0000 7fbed1ffb640 1 -- 172.21.15.114:0/856141312 <== mon.0 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbee4028060 con 0x561b8125f620 2024-04-23T10:38:49.642 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:38:49.634+0000 7fbef3474740 1 -- 172.21.15.114:0/856141312 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fbee4015070 con 0x561b8125f620 2024-04-23T10:38:49.642 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:38:49.634+0000 7fbed1ffb640 1 -- 172.21.15.114:0/856141312 <== mon.0 v2:172.21.15.114:3300/0 2 ==== osd_map(106..106 src has 1..106) ==== 8629+0+0 (secure 0 0 0) 0x7fbee4011850 con 0x561b8125f620 2024-04-23T10:38:49.642 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:38:49.634+0000 7fbed1ffb640 1 -- 172.21.15.114:0/856141312 <== mon.0 v2:172.21.15.114:3300/0 3 ==== mon_get_version_reply(handle=1 version=42) ==== 24+0+0 (secure 0 0 0) 0x7fbee4011a50 con 0x561b8125f620 2024-04-23T10:38:49.642 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:38:49.634+0000 7fbef3474740 10 client.16941 fetch_fsmap learned FSMap version 42 2024-04-23T10:38:49.642 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:38:49.634+0000 7fbef3474740 1 -- 172.21.15.114:0/856141312 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({config=0+,fsmap=42,monmap=4+}) -- 0x561b81279360 con 0x561b8125f620 2024-04-23T10:38:49.642 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:38:49.635+0000 7fbed1ffb640 1 -- 172.21.15.114:0/856141312 <== mon.0 v2:172.21.15.114:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fbee4015070 con 0x561b8125f620 2024-04-23T10:38:49.642 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:38:49.635+0000 7fbed1ffb640 1 -- 172.21.15.114:0/856141312 <== mon.0 v2:172.21.15.114:3300/0 5 ==== fsmap(e 42) ==== 8338+0+0 (secure 0 0 0) 0x7fbee4022030 con 0x561b8125f620 2024-04-23T10:38:49.642 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:38:49.635+0000 7fbef3474740 10 client.16941 fetch_fsmap finished waiting for FSMap version 42 2024-04-23T10:38:49.642 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:38:49.635+0000 7fbef3474740 10 client.16941 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-04-23T10:38:49.642 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:38:49.635+0000 7fbef3474740 20 client.16941 populate_metadata read hostname 'smithi114' 2024-04-23T10:38:49.642 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:38:49.635+0000 7fbef3474740 1 --2- 172.21.15.114:0/856141312 >> [v2:172.21.15.119:6838/937297340,v1:172.21.15.119:6839/937297340] conn(0x561b8127a770 0x561b8127cb60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T10:38:49.642 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:38:49.635+0000 7fbef3474740 4 client.16941 mds_command: new command op to 24545 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-23T10:38:49.642 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:38:49.635+0000 7fbef3474740 1 -- 172.21.15.114:0/856141312 --> [v2:172.21.15.119:6838/937297340,v1:172.21.15.119:6839/937297340] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x561b811d2390 con 0x561b8127a770 2024-04-23T10:38:49.642 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:38:49.635+0000 7fbeedcf8640 1 --2- 172.21.15.114:0/856141312 >> [v2:172.21.15.119:6838/937297340,v1:172.21.15.119:6839/937297340] conn(0x561b8127a770 0x561b8127cb60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T10:38:49.643 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:38:49.635+0000 7fbed1ffb640 10 client.16941 ms_handle_connect on v2:172.21.15.119:6838/937297340 2024-04-23T10:38:49.643 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:38:49.635+0000 7fbeedcf8640 1 --2- 172.21.15.114:0/856141312 >> [v2:172.21.15.119:6838/937297340,v1:172.21.15.119:6839/937297340] conn(0x561b8127a770 0x561b8127cb60 crc :-1 s=READY pgs=170 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=7aa032926c315be0 server_cookie=f563d0af02a7ad6e in_seq=0 out_seq=0 2024-04-23T10:38:49.643 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:38:49.636+0000 7fbed1ffb640 1 -- 172.21.15.114:0/856141312 <== mds.0 v2:172.21.15.119:6838/937297340 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x561b811d2390 con 0x561b8127a770 2024-04-23T10:38:49.643 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:38:49.636+0000 7fbed1ffb640 10 client.16941 handle_command_reply: tid=0 2024-04-23T10:38:49.643 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:38:49.636+0000 7fbef3474740 1 client.16941 shutdown 2024-04-23T10:38:49.643 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:38:49.636+0000 7fbef3474740 1 -- 172.21.15.114:0/856141312 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x561b8125f620 msgr2=0x561b8126a170 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T10:38:49.643 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:38:49.636+0000 7fbef3474740 1 --2- 172.21.15.114:0/856141312 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x561b8125f620 0x561b8126a170 secure :-1 s=READY pgs=889 cs=0 l=1 rev1=1 crypto rx=0x7fbee4006010 tx=0x7fbee4003d80 comp rx=0 tx=0).stop 2024-04-23T10:38:49.643 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:38:49.636+0000 7fbef3474740 1 -- 172.21.15.114:0/856141312 shutdown_connections 2024-04-23T10:38:49.643 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:38:49.636+0000 7fbef3474740 1 --2- 172.21.15.114:0/856141312 >> [v2:172.21.15.119:6838/937297340,v1:172.21.15.119:6839/937297340] conn(0x561b8127a770 0x561b8127cb60 crc :-1 s=READY pgs=170 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:38:49.643 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:38:49.636+0000 7fbef3474740 1 --2- 172.21.15.114:0/856141312 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x561b8126a800 0x561b812679c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:38:49.643 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:38:49.636+0000 7fbef3474740 1 --2- 172.21.15.114:0/856141312 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x561b8125ff40 0x561b81267460 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:38:49.643 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:38:49.636+0000 7fbed1ffb640 0 client.16941 ms_handle_reset on v2:172.21.15.119:6838/937297340 2024-04-23T10:38:49.643 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:38:49.636+0000 7fbef3474740 1 --2- 172.21.15.114:0/856141312 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x561b8125f620 0x561b8126a170 unknown :-1 s=CLOSED pgs=889 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:38:49.644 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:38:49.636+0000 7fbef3474740 1 -- 172.21.15.114:0/856141312 >> 172.21.15.114:0/856141312 conn(0x561b8125bb30 msgr2=0x561b81261860 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T10:38:49.644 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:38:49.636+0000 7fbef3474740 1 -- 172.21.15.114:0/856141312 shutdown_connections 2024-04-23T10:38:49.644 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:38:49.636+0000 7fbef3474740 1 -- 172.21.15.114:0/856141312 wait complete. 2024-04-23T10:38:49.644 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:38:49.636+0000 7fbef3474740 20 client.16941 trim_cache size 0 max 16384 2024-04-23T10:38:49.655 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:38:49.651+0000 7fbef3474740 1 Processor -- start 2024-04-23T10:38:49.655 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:38:49.651+0000 7fbef3474740 1 -- start start 2024-04-23T10:38:49.655 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:38:49.651+0000 7fbef3474740 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x561b8125f620 0x561b81269ca0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T10:38:49.655 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:38:49.652+0000 7fbef3474740 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x561b8125ff40 0x561b8127fb80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T10:38:49.655 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:38:49.652+0000 7fbef3474740 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x561b8126a800 0x561b812800c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T10:38:49.655 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:38:49.652+0000 7fbef3474740 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x7fbee81b2770 con 0x561b8125ff40 2024-04-23T10:38:49.656 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:38:49.652+0000 7fbef3474740 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 -- 0x7fbee81b0040 con 0x561b8125f620 2024-04-23T10:38:49.656 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:38:49.652+0000 7fbef3474740 1 -- --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_getmap magic: 0 -- 0x7fbee81adeb0 con 0x561b8126a800 2024-04-23T10:38:49.656 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:38:49.652+0000 7fbeedcf8640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x561b8125ff40 0x561b8127fb80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T10:38:49.656 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:38:49.652+0000 7fbeedcf8640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x561b8125ff40 0x561b8127fb80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.114:38234/0 (socket says 172.21.15.114:38234) 2024-04-23T10:38:49.656 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:38:49.652+0000 7fbeedcf8640 1 -- 172.21.15.114:0/2882320875 learned_addr learned my addr 172.21.15.114:0/2882320875 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-04-23T10:38:49.656 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:38:49.652+0000 7fbeee4f9640 1 --2- 172.21.15.114:0/2882320875 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x561b8125f620 0x561b81269ca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T10:38:49.656 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:38:49.652+0000 7fbeeecfa640 1 --2- 172.21.15.114:0/2882320875 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x561b8126a800 0x561b812800c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T10:38:49.656 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:38:49.652+0000 7fbeedcf8640 1 -- 172.21.15.114:0/2882320875 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x561b8126a800 msgr2=0x561b812800c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T10:38:49.656 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:38:49.652+0000 7fbeedcf8640 1 --2- 172.21.15.114:0/2882320875 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x561b8126a800 0x561b812800c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:38:49.656 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:38:49.652+0000 7fbeedcf8640 1 -- 172.21.15.114:0/2882320875 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x561b8125f620 msgr2=0x561b81269ca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T10:38:49.656 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:38:49.652+0000 7fbeedcf8640 1 --2- 172.21.15.114:0/2882320875 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x561b8125f620 0x561b81269ca0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:38:49.656 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:38:49.652+0000 7fbeedcf8640 1 -- 172.21.15.114:0/2882320875 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x561b81267d30 con 0x561b8125ff40 2024-04-23T10:38:49.656 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:38:49.652+0000 7fbeeecfa640 1 --2- 172.21.15.114:0/2882320875 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x561b8126a800 0x561b812800c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-23T10:38:49.656 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:38:49.652+0000 7fbeedcf8640 1 --2- 172.21.15.114:0/2882320875 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x561b8125ff40 0x561b8127fb80 secure :-1 s=READY pgs=890 cs=0 l=1 rev1=1 crypto rx=0x7fbed80ce110 tx=0x7fbed800bf80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4e920290ed86481b server_cookie=0 in_seq=0 out_seq=0 2024-04-23T10:38:49.656 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:38:49.652+0000 7fbed3fff640 1 -- 172.21.15.114:0/2882320875 <== mon.0 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbed8020040 con 0x561b8125ff40 2024-04-23T10:38:49.657 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:38:49.652+0000 7fbed3fff640 1 -- 172.21.15.114:0/2882320875 <== mon.0 v2:172.21.15.114:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fbed806e480 con 0x561b8125ff40 2024-04-23T10:38:49.657 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:38:49.652+0000 7fbed3fff640 1 -- 172.21.15.114:0/2882320875 <== mon.0 v2:172.21.15.114:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbed80cdd90 con 0x561b8125ff40 2024-04-23T10:38:49.657 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:38:49.653+0000 7fbef3474740 1 -- 172.21.15.114:0/2882320875 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x561b8125ff40 msgr2=0x561b8127fb80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T10:38:49.657 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:38:49.653+0000 7fbef3474740 1 --2- 172.21.15.114:0/2882320875 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x561b8125ff40 0x561b8127fb80 secure :-1 s=READY pgs=890 cs=0 l=1 rev1=1 crypto rx=0x7fbed80ce110 tx=0x7fbed800bf80 comp rx=0 tx=0).stop 2024-04-23T10:38:49.657 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:38:49.653+0000 7fbef3474740 1 -- 172.21.15.114:0/2882320875 shutdown_connections 2024-04-23T10:38:49.657 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:38:49.653+0000 7fbef3474740 1 --2- 172.21.15.114:0/2882320875 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x561b8126a800 0x561b812800c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:38:49.657 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:38:49.653+0000 7fbef3474740 1 --2- 172.21.15.114:0/2882320875 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x561b8125ff40 0x561b8127fb80 unknown :-1 s=CLOSED pgs=890 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:38:49.657 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:38:49.653+0000 7fbef3474740 1 --2- 172.21.15.114:0/2882320875 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x561b8125f620 0x561b81269ca0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:38:49.657 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:38:49.653+0000 7fbef3474740 1 -- 172.21.15.114:0/2882320875 >> 172.21.15.114:0/2882320875 conn(0x561b8125bb30 msgr2=0x561b81276090 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T10:38:49.657 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:38:49.653+0000 7fbef3474740 1 -- 172.21.15.114:0/2882320875 shutdown_connections 2024-04-23T10:38:49.657 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:38:49.653+0000 7fbef3474740 1 -- 172.21.15.114:0/2882320875 wait complete. 2024-04-23T10:38:49.657 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:38:49.653+0000 7fbef3474740 1 -- start start 2024-04-23T10:38:49.657 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:38:49.653+0000 7fbef3474740 1 Processor -- start 2024-04-23T10:38:49.657 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:38:49.653+0000 7fbef3474740 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x561b8125f620 0x561b8127d6e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T10:38:49.657 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:38:49.653+0000 7fbef3474740 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x561b8125ff40 0x561b8127dc20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T10:38:49.658 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:38:49.653+0000 7fbef3474740 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x561b8126a800 0x561b8127e160 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T10:38:49.658 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:38:49.653+0000 7fbef3474740 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x7fbee81ae500 con 0x561b8125f620 2024-04-23T10:38:49.658 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:38:49.653+0000 7fbef3474740 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 -- 0x7fbee81af970 con 0x561b8126a800 2024-04-23T10:38:49.658 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:38:49.653+0000 7fbef3474740 1 -- --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_getmap magic: 0 -- 0x7fbee81aebd0 con 0x561b8125ff40 2024-04-23T10:38:49.658 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:38:49.653+0000 7fbef3474740 10 client.0 _ll_register_callbacks cb 0x7fbee4004750 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-23T10:38:49.658 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:38:49.654+0000 7fbeee4f9640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x561b8125f620 0x561b8127d6e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T10:38:49.658 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:38:49.654+0000 7fbeee4f9640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x561b8125f620 0x561b8127d6e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.114:38250/0 (socket says 172.21.15.114:38250) 2024-04-23T10:38:49.658 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:38:49.654+0000 7fbeee4f9640 1 -- 172.21.15.114:0/3396253872 learned_addr learned my addr 172.21.15.114:0/3396253872 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-04-23T10:38:49.658 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:38:49.654+0000 7fbeedcf8640 1 --2- 172.21.15.114:0/3396253872 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x561b8125ff40 0x561b8127dc20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T10:38:49.658 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:38:49.654+0000 7fbeeecfa640 1 --2- 172.21.15.114:0/3396253872 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x561b8126a800 0x561b8127e160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T10:38:49.658 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:38:49.654+0000 7fbeee4f9640 1 -- 172.21.15.114:0/3396253872 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x561b8125ff40 msgr2=0x561b8127dc20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T10:38:49.658 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:38:49.654+0000 7fbeee4f9640 1 --2- 172.21.15.114:0/3396253872 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x561b8125ff40 0x561b8127dc20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:38:49.658 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:38:49.654+0000 7fbeee4f9640 1 -- 172.21.15.114:0/3396253872 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x561b8126a800 msgr2=0x561b8127e160 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T10:38:49.658 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:38:49.654+0000 7fbeee4f9640 1 --2- 172.21.15.114:0/3396253872 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x561b8126a800 0x561b8127e160 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:38:49.659 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:38:49.654+0000 7fbeee4f9640 1 -- 172.21.15.114:0/3396253872 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({osdmap=0}) -- 0x561b812dfa50 con 0x561b8125f620 2024-04-23T10:38:49.659 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:38:49.654+0000 7fbeee4f9640 1 --2- 172.21.15.114:0/3396253872 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x561b8125f620 0x561b8127d6e0 secure :-1 s=READY pgs=891 cs=0 l=1 rev1=1 crypto rx=0x7fbee4004010 tx=0x7fbee4005af0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=230ab2b89a900f9 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T10:38:49.659 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:38:49.654+0000 7fbed1ffb640 10 client.0 ms_handle_connect on v2:172.21.15.114:3300/0 2024-04-23T10:38:49.659 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:38:49.654+0000 7fbed1ffb640 1 -- 172.21.15.114:0/3396253872 <== mon.0 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbee4038060 con 0x561b8125f620 2024-04-23T10:38:49.659 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:38:49.654+0000 7fbed1ffb640 1 -- 172.21.15.114:0/3396253872 <== mon.0 v2:172.21.15.114:3300/0 2 ==== osd_map(106..106 src has 1..106) ==== 8629+0+0 (secure 0 0 0) 0x7fbee4019420 con 0x561b8125f620 2024-04-23T10:38:49.659 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:38:49.654+0000 7fbef3474740 1 -- 172.21.15.114:0/3396253872 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fbed806e480 con 0x561b8125f620 2024-04-23T10:38:49.659 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:38:49.654+0000 7fbed1ffb640 1 -- 172.21.15.114:0/3396253872 <== mon.0 v2:172.21.15.114:3300/0 3 ==== mon_get_version_reply(handle=1 version=42) ==== 24+0+0 (secure 0 0 0) 0x7fbee4028070 con 0x561b8125f620 2024-04-23T10:38:49.659 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:38:49.654+0000 7fbef3474740 10 client.16947 fetch_fsmap learned FSMap version 42 2024-04-23T10:38:49.659 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:38:49.654+0000 7fbef3474740 1 -- 172.21.15.114:0/3396253872 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({config=0+,fsmap=42,monmap=4+}) -- 0x561b812e0cd0 con 0x561b8125f620 2024-04-23T10:38:49.659 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:38:49.654+0000 7fbed1ffb640 1 -- 172.21.15.114:0/3396253872 <== mon.0 v2:172.21.15.114:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fbed806e480 con 0x561b8125f620 2024-04-23T10:38:49.659 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:38:49.655+0000 7fbed1ffb640 1 -- 172.21.15.114:0/3396253872 <== mon.0 v2:172.21.15.114:3300/0 5 ==== fsmap(e 42) ==== 8338+0+0 (secure 0 0 0) 0x7fbee4032030 con 0x561b8125f620 2024-04-23T10:38:49.659 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:38:49.655+0000 7fbef3474740 10 client.16947 fetch_fsmap finished waiting for FSMap version 42 2024-04-23T10:38:49.659 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:38:49.655+0000 7fbef3474740 10 client.16947 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-04-23T10:38:49.659 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:38:49.655+0000 7fbef3474740 20 client.16947 populate_metadata read hostname 'smithi114' 2024-04-23T10:38:49.660 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:38:49.655+0000 7fbef3474740 1 --2- 172.21.15.114:0/3396253872 >> [v2:172.21.15.119:6838/937297340,v1:172.21.15.119:6839/937297340] conn(0x561b812e13f0 0x561b812e3810 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T10:38:49.660 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:38:49.655+0000 7fbef3474740 4 client.16947 mds_command: new command op to 24545 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-04-23T10:38:49.660 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:38:49.655+0000 7fbef3474740 1 -- 172.21.15.114:0/3396253872 --> [v2:172.21.15.119:6838/937297340,v1:172.21.15.119:6839/937297340] -- command(tid 0: {"prefix": "flush journal"}) -- 0x561b811cc200 con 0x561b812e13f0 2024-04-23T10:38:49.660 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:38:49.655+0000 7fbeedcf8640 1 --2- 172.21.15.114:0/3396253872 >> [v2:172.21.15.119:6838/937297340,v1:172.21.15.119:6839/937297340] conn(0x561b812e13f0 0x561b812e3810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T10:38:49.660 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:38:49.655+0000 7fbed1ffb640 10 client.16947 ms_handle_connect on v2:172.21.15.119:6838/937297340 2024-04-23T10:38:49.660 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:38:49.655+0000 7fbeedcf8640 1 --2- 172.21.15.114:0/3396253872 >> [v2:172.21.15.119:6838/937297340,v1:172.21.15.119:6839/937297340] conn(0x561b812e13f0 0x561b812e3810 crc :-1 s=READY pgs=171 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=3bcb69ee60026b13 server_cookie=5e29da0aa2ae0f5e in_seq=0 out_seq=0 2024-04-23T10:38:49.691 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:38:49.692+0000 7fbed1ffb640 1 -- 172.21.15.114:0/3396253872 <== mds.0 v2:172.21.15.119:6838/937297340 1 ==== command_reply(tid 0: 0 ) ==== 8+0+44 (crc 0 0 0) 0x561b811cc200 con 0x561b812e13f0 2024-04-23T10:38:49.691 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:38:49.692+0000 7fbed1ffb640 10 client.16947 handle_command_reply: tid=0 2024-04-23T10:38:49.691 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:38:49.692+0000 7fbef3474740 1 client.16947 shutdown 2024-04-23T10:38:49.692 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:38:49.693+0000 7fbef3474740 1 -- 172.21.15.114:0/3396253872 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x561b8125f620 msgr2=0x561b8127d6e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T10:38:49.692 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:38:49.693+0000 7fbef3474740 1 --2- 172.21.15.114:0/3396253872 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x561b8125f620 0x561b8127d6e0 secure :-1 s=READY pgs=891 cs=0 l=1 rev1=1 crypto rx=0x7fbee4004010 tx=0x7fbee4005af0 comp rx=0 tx=0).stop 2024-04-23T10:38:49.692 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:38:49.693+0000 7fbef3474740 1 -- 172.21.15.114:0/3396253872 shutdown_connections 2024-04-23T10:38:49.692 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:38:49.693+0000 7fbef3474740 1 --2- 172.21.15.114:0/3396253872 >> [v2:172.21.15.119:6838/937297340,v1:172.21.15.119:6839/937297340] conn(0x561b812e13f0 0x561b812e3810 crc :-1 s=READY pgs=171 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:38:49.692 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:38:49.693+0000 7fbef3474740 1 --2- 172.21.15.114:0/3396253872 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x561b8126a800 0x561b8127e160 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:38:49.692 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:38:49.693+0000 7fbef3474740 1 --2- 172.21.15.114:0/3396253872 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x561b8125ff40 0x561b8127dc20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:38:49.692 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:38:49.693+0000 7fbed1ffb640 0 client.16947 ms_handle_reset on v2:172.21.15.119:6838/937297340 2024-04-23T10:38:49.692 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:38:49.693+0000 7fbef3474740 1 --2- 172.21.15.114:0/3396253872 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x561b8125f620 0x561b8127d6e0 unknown :-1 s=CLOSED pgs=891 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:38:49.693 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:38:49.693+0000 7fbef3474740 1 -- 172.21.15.114:0/3396253872 >> 172.21.15.114:0/3396253872 conn(0x561b8125bb30 msgr2=0x561b8125a420 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T10:38:49.693 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:38:49.693+0000 7fbef3474740 1 -- 172.21.15.114:0/3396253872 shutdown_connections 2024-04-23T10:38:49.693 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:38:49.693+0000 7fbef3474740 1 -- 172.21.15.114:0/3396253872 wait complete. 2024-04-23T10:38:49.693 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:38:49.693+0000 7fbef3474740 20 client.16947 trim_cache size 0 max 16384 2024-04-23T10:38:49.693 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stdout:{ 2024-04-23T10:38:49.693 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stdout: "message": "", 2024-04-23T10:38:49.693 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stdout: "return_code": 0 2024-04-23T10:38:49.693 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stdout:} 2024-04-23T10:38:49.694 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:38:49.693+0000 7fbef075d640 1 -- 172.21.15.114:0/1371192985 >> [v2:172.21.15.114:6800/1395693466,v1:172.21.15.114:6801/1395693466] conn(0x7fbed40a3dc0 msgr2=0x7fbed40a6280 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T10:38:49.694 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:38:49.694+0000 7fbef075d640 1 --2- 172.21.15.114:0/1371192985 >> [v2:172.21.15.114:6800/1395693466,v1:172.21.15.114:6801/1395693466] conn(0x7fbed40a3dc0 0x7fbed40a6280 secure :-1 s=READY pgs=139 cs=0 l=1 rev1=1 crypto rx=0x7fbee0005fa0 tx=0x7fbee002ffd0 comp rx=0 tx=0).stop 2024-04-23T10:38:49.694 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:38:49.694+0000 7fbef075d640 1 -- 172.21.15.114:0/1371192985 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fbee8103ac0 msgr2=0x7fbee8101330 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T10:38:49.694 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:38:49.694+0000 7fbef075d640 1 --2- 172.21.15.114:0/1371192985 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fbee8103ac0 0x7fbee8101330 secure :-1 s=READY pgs=887 cs=0 l=1 rev1=1 crypto rx=0x7fbed800d950 tx=0x7fbed800de10 comp rx=0 tx=0).stop 2024-04-23T10:38:49.694 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:38:49.694+0000 7fbef075d640 1 -- 172.21.15.114:0/1371192985 shutdown_connections 2024-04-23T10:38:49.694 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:38:49.694+0000 7fbef075d640 1 --2- 172.21.15.114:0/1371192985 >> [v2:172.21.15.114:6800/1395693466,v1:172.21.15.114:6801/1395693466] conn(0x7fbed40a3dc0 0x7fbed40a6280 unknown :-1 s=CLOSED pgs=139 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:38:49.694 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:38:49.694+0000 7fbef075d640 1 --2- 172.21.15.114:0/1371192985 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fbee810c570 0x7fbee8101870 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:38:49.694 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:38:49.694+0000 7fbef075d640 1 --2- 172.21.15.114:0/1371192985 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fbee8103ac0 0x7fbee8101330 unknown :-1 s=CLOSED pgs=887 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:38:49.694 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:38:49.694+0000 7fbef075d640 1 --2- 172.21.15.114:0/1371192985 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fbee81031a0 0x7fbee8100df0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:38:49.694 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:38:49.694+0000 7fbef075d640 1 -- 172.21.15.114:0/1371192985 >> 172.21.15.114:0/1371192985 conn(0x7fbee8078060 msgr2=0x7fbee80fdea0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T10:38:49.695 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:38:49.694+0000 7fbef075d640 1 -- 172.21.15.114:0/1371192985 shutdown_connections 2024-04-23T10:38:49.695 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:38:49.694+0000 7fbef075d640 1 -- 172.21.15.114:0/1371192985 wait complete. 2024-04-23T10:38:49.948 INFO:tasks.workunit.client.0.smithi114.stdout: 1 699721 5.26 MB/sec execute 538 sec latency 2988.644 ms 2024-04-23T10:38:49.992 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 23 10:38:49 smithi119 ceph-mon[32608]: pgmap v383: 97 pgs: 97 active+clean; 222 MiB data, 2.7 GiB used, 1.0 TiB / 1.0 TiB avail; 326 KiB/s rd, 667 KiB/s wr, 77 op/s 2024-04-23T10:38:50.003 INFO:journalctl@ceph.mon.a.smithi114.stdout:Apr 23 10:38:49 smithi114 ceph-mon[29255]: pgmap v383: 97 pgs: 97 active+clean; 222 MiB data, 2.7 GiB used, 1.0 TiB / 1.0 TiB avail; 326 KiB/s rd, 667 KiB/s wr, 77 op/s 2024-04-23T10:38:50.077 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 23 10:38:49 smithi204 ceph-mon[32649]: pgmap v383: 97 pgs: 97 active+clean; 222 MiB data, 2.7 GiB used, 1.0 TiB / 1.0 TiB avail; 326 KiB/s rd, 667 KiB/s wr, 77 op/s 2024-04-23T10:38:50.949 INFO:tasks.workunit.client.0.smithi114.stdout: 1 699721 5.25 MB/sec execute 539 sec latency 1658.668 ms 2024-04-23T10:38:51.949 INFO:tasks.workunit.client.0.smithi114.stdout: 1 699721 5.24 MB/sec execute 540 sec latency 2658.748 ms 2024-04-23T10:38:51.992 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 23 10:38:51 smithi119 ceph-mon[32608]: pgmap v384: 97 pgs: 97 active+clean; 222 MiB data, 2.7 GiB used, 1.0 TiB / 1.0 TiB avail; 327 KiB/s rd, 667 KiB/s wr, 77 op/s 2024-04-23T10:38:52.004 INFO:journalctl@ceph.mon.a.smithi114.stdout:Apr 23 10:38:51 smithi114 ceph-mon[29255]: pgmap v384: 97 pgs: 97 active+clean; 222 MiB data, 2.7 GiB used, 1.0 TiB / 1.0 TiB avail; 327 KiB/s rd, 667 KiB/s wr, 77 op/s 2024-04-23T10:38:52.077 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 23 10:38:51 smithi204 ceph-mon[32649]: pgmap v384: 97 pgs: 97 active+clean; 222 MiB data, 2.7 GiB used, 1.0 TiB / 1.0 TiB avail; 327 KiB/s rd, 667 KiB/s wr, 77 op/s 2024-04-23T10:38:52.948 INFO:tasks.workunit.client.0.smithi114.stdout: 1 699721 5.23 MB/sec execute 541 sec latency 3658.807 ms 2024-04-23T10:38:53.949 INFO:tasks.workunit.client.0.smithi114.stdout: 1 699721 5.22 MB/sec execute 542 sec latency 4658.899 ms 2024-04-23T10:38:53.992 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 23 10:38:53 smithi119 ceph-mon[32608]: pgmap v385: 97 pgs: 97 active+clean; 220 MiB data, 2.7 GiB used, 1.0 TiB / 1.0 TiB avail; 394 KiB/s rd, 901 KiB/s wr, 101 op/s 2024-04-23T10:38:54.004 INFO:journalctl@ceph.mon.a.smithi114.stdout:Apr 23 10:38:53 smithi114 ceph-mon[29255]: pgmap v385: 97 pgs: 97 active+clean; 220 MiB data, 2.7 GiB used, 1.0 TiB / 1.0 TiB avail; 394 KiB/s rd, 901 KiB/s wr, 101 op/s 2024-04-23T10:38:54.078 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 23 10:38:53 smithi204 ceph-mon[32649]: pgmap v385: 97 pgs: 97 active+clean; 220 MiB data, 2.7 GiB used, 1.0 TiB / 1.0 TiB avail; 394 KiB/s rd, 901 KiB/s wr, 101 op/s 2024-04-23T10:38:54.949 INFO:tasks.workunit.client.0.smithi114.stdout: 1 700373 5.22 MB/sec execute 543 sec latency 4900.937 ms 2024-04-23T10:38:55.949 INFO:tasks.workunit.client.0.smithi114.stdout: 1 702604 5.23 MB/sec execute 544 sec latency 18.104 ms 2024-04-23T10:38:55.992 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 23 10:38:55 smithi119 ceph-mon[32608]: pgmap v386: 97 pgs: 97 active+clean; 220 MiB data, 2.7 GiB used, 1.0 TiB / 1.0 TiB avail; 332 KiB/s rd, 653 KiB/s wr, 71 op/s 2024-04-23T10:38:56.004 INFO:journalctl@ceph.mon.a.smithi114.stdout:Apr 23 10:38:55 smithi114 ceph-mon[29255]: pgmap v386: 97 pgs: 97 active+clean; 220 MiB data, 2.7 GiB used, 1.0 TiB / 1.0 TiB avail; 332 KiB/s rd, 653 KiB/s wr, 71 op/s 2024-04-23T10:38:56.077 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 23 10:38:55 smithi204 ceph-mon[32649]: pgmap v386: 97 pgs: 97 active+clean; 220 MiB data, 2.7 GiB used, 1.0 TiB / 1.0 TiB avail; 332 KiB/s rd, 653 KiB/s wr, 71 op/s 2024-04-23T10:38:56.949 INFO:tasks.workunit.client.0.smithi114.stdout: 1 703199 5.23 MB/sec execute 545 sec latency 769.486 ms 2024-04-23T10:38:57.949 INFO:tasks.workunit.client.0.smithi114.stdout: 1 703199 5.22 MB/sec execute 546 sec latency 1769.556 ms 2024-04-23T10:38:57.992 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 23 10:38:57 smithi119 ceph-mon[32608]: pgmap v387: 97 pgs: 97 active+clean; 221 MiB data, 2.8 GiB used, 1.0 TiB / 1.0 TiB avail; 576 KiB/s rd, 1.2 MiB/s wr, 115 op/s 2024-04-23T10:38:57.992 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 23 10:38:57 smithi119 ceph-mon[32608]: from='mgr.15291 172.21.15.114:0/1095789187' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-23T10:38:58.004 INFO:journalctl@ceph.mon.a.smithi114.stdout:Apr 23 10:38:57 smithi114 ceph-mon[29255]: pgmap v387: 97 pgs: 97 active+clean; 221 MiB data, 2.8 GiB used, 1.0 TiB / 1.0 TiB avail; 576 KiB/s rd, 1.2 MiB/s wr, 115 op/s 2024-04-23T10:38:58.004 INFO:journalctl@ceph.mon.a.smithi114.stdout:Apr 23 10:38:57 smithi114 ceph-mon[29255]: from='mgr.15291 172.21.15.114:0/1095789187' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-23T10:38:58.078 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 23 10:38:57 smithi204 ceph-mon[32649]: pgmap v387: 97 pgs: 97 active+clean; 221 MiB data, 2.8 GiB used, 1.0 TiB / 1.0 TiB avail; 576 KiB/s rd, 1.2 MiB/s wr, 115 op/s 2024-04-23T10:38:58.078 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 23 10:38:57 smithi204 ceph-mon[32649]: from='mgr.15291 172.21.15.114:0/1095789187' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-23T10:38:58.949 INFO:tasks.workunit.client.0.smithi114.stdout: 1 703199 5.21 MB/sec execute 547 sec latency 2769.613 ms 2024-04-23T10:38:59.242 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 23 10:38:58 smithi119 ceph-mon[32608]: from='mgr.15291 172.21.15.114:0/1095789187' entity='mgr.x' 2024-04-23T10:38:59.242 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 23 10:38:58 smithi119 ceph-mon[32608]: from='mgr.15291 172.21.15.114:0/1095789187' entity='mgr.x' 2024-04-23T10:38:59.242 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 23 10:38:58 smithi119 ceph-mon[32608]: from='mgr.15291 172.21.15.114:0/1095789187' entity='mgr.x' 2024-04-23T10:38:59.242 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 23 10:38:58 smithi119 ceph-mon[32608]: from='mgr.15291 172.21.15.114:0/1095789187' entity='mgr.x' 2024-04-23T10:38:59.242 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 23 10:38:58 smithi119 ceph-mon[32608]: from='mgr.15291 172.21.15.114:0/1095789187' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-23T10:38:59.243 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 23 10:38:58 smithi119 ceph-mon[32608]: from='mgr.15291 172.21.15.114:0/1095789187' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-23T10:38:59.243 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 23 10:38:58 smithi119 ceph-mon[32608]: from='mgr.15291 172.21.15.114:0/1095789187' entity='mgr.x' 2024-04-23T10:38:59.254 INFO:journalctl@ceph.mon.a.smithi114.stdout:Apr 23 10:38:58 smithi114 ceph-mon[29255]: from='mgr.15291 172.21.15.114:0/1095789187' entity='mgr.x' 2024-04-23T10:38:59.254 INFO:journalctl@ceph.mon.a.smithi114.stdout:Apr 23 10:38:58 smithi114 ceph-mon[29255]: from='mgr.15291 172.21.15.114:0/1095789187' entity='mgr.x' 2024-04-23T10:38:59.254 INFO:journalctl@ceph.mon.a.smithi114.stdout:Apr 23 10:38:58 smithi114 ceph-mon[29255]: from='mgr.15291 172.21.15.114:0/1095789187' entity='mgr.x' 2024-04-23T10:38:59.254 INFO:journalctl@ceph.mon.a.smithi114.stdout:Apr 23 10:38:58 smithi114 ceph-mon[29255]: from='mgr.15291 172.21.15.114:0/1095789187' entity='mgr.x' 2024-04-23T10:38:59.255 INFO:journalctl@ceph.mon.a.smithi114.stdout:Apr 23 10:38:58 smithi114 ceph-mon[29255]: from='mgr.15291 172.21.15.114:0/1095789187' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-23T10:38:59.255 INFO:journalctl@ceph.mon.a.smithi114.stdout:Apr 23 10:38:58 smithi114 ceph-mon[29255]: from='mgr.15291 172.21.15.114:0/1095789187' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-23T10:38:59.255 INFO:journalctl@ceph.mon.a.smithi114.stdout:Apr 23 10:38:58 smithi114 ceph-mon[29255]: from='mgr.15291 172.21.15.114:0/1095789187' entity='mgr.x' 2024-04-23T10:38:59.328 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 23 10:38:58 smithi204 ceph-mon[32649]: from='mgr.15291 172.21.15.114:0/1095789187' entity='mgr.x' 2024-04-23T10:38:59.328 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 23 10:38:58 smithi204 ceph-mon[32649]: from='mgr.15291 172.21.15.114:0/1095789187' entity='mgr.x' 2024-04-23T10:38:59.328 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 23 10:38:58 smithi204 ceph-mon[32649]: from='mgr.15291 172.21.15.114:0/1095789187' entity='mgr.x' 2024-04-23T10:38:59.328 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 23 10:38:58 smithi204 ceph-mon[32649]: from='mgr.15291 172.21.15.114:0/1095789187' entity='mgr.x' 2024-04-23T10:38:59.328 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 23 10:38:58 smithi204 ceph-mon[32649]: from='mgr.15291 172.21.15.114:0/1095789187' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-23T10:38:59.328 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 23 10:38:58 smithi204 ceph-mon[32649]: from='mgr.15291 172.21.15.114:0/1095789187' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-23T10:38:59.328 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 23 10:38:58 smithi204 ceph-mon[32649]: from='mgr.15291 172.21.15.114:0/1095789187' entity='mgr.x' 2024-04-23T10:38:59.949 INFO:tasks.workunit.client.0.smithi114.stdout: 1 703266 5.20 MB/sec execute 548 sec latency 3006.276 ms 2024-04-23T10:39:00.242 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 23 10:38:59 smithi119 ceph-mon[32608]: pgmap v388: 97 pgs: 97 active+clean; 221 MiB data, 2.8 GiB used, 1.0 TiB / 1.0 TiB avail; 311 KiB/s rd, 858 KiB/s wr, 67 op/s 2024-04-23T10:39:00.254 INFO:journalctl@ceph.mon.a.smithi114.stdout:Apr 23 10:38:59 smithi114 ceph-mon[29255]: pgmap v388: 97 pgs: 97 active+clean; 221 MiB data, 2.8 GiB used, 1.0 TiB / 1.0 TiB avail; 311 KiB/s rd, 858 KiB/s wr, 67 op/s 2024-04-23T10:39:00.327 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 23 10:38:59 smithi204 ceph-mon[32649]: pgmap v388: 97 pgs: 97 active+clean; 221 MiB data, 2.8 GiB used, 1.0 TiB / 1.0 TiB avail; 311 KiB/s rd, 858 KiB/s wr, 67 op/s 2024-04-23T10:39:00.949 INFO:tasks.workunit.client.0.smithi114.stdout: 1 703266 5.19 MB/sec execute 549 sec latency 1670.888 ms 2024-04-23T10:39:01.950 INFO:tasks.workunit.client.0.smithi114.stdout: 1 703266 5.18 MB/sec execute 550 sec latency 2670.945 ms 2024-04-23T10:39:02.242 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 23 10:39:01 smithi119 ceph-mon[32608]: pgmap v389: 97 pgs: 97 active+clean; 221 MiB data, 2.8 GiB used, 1.0 TiB / 1.0 TiB avail; 311 KiB/s rd, 858 KiB/s wr, 67 op/s 2024-04-23T10:39:02.254 INFO:journalctl@ceph.mon.a.smithi114.stdout:Apr 23 10:39:01 smithi114 ceph-mon[29255]: pgmap v389: 97 pgs: 97 active+clean; 221 MiB data, 2.8 GiB used, 1.0 TiB / 1.0 TiB avail; 311 KiB/s rd, 858 KiB/s wr, 67 op/s 2024-04-23T10:39:02.328 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 23 10:39:01 smithi204 ceph-mon[32649]: pgmap v389: 97 pgs: 97 active+clean; 221 MiB data, 2.8 GiB used, 1.0 TiB / 1.0 TiB avail; 311 KiB/s rd, 858 KiB/s wr, 67 op/s 2024-04-23T10:39:02.790 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:39:02.788+0000 7f44e07bb640 1 -- 172.21.15.114:0/1672070978 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f44d810e950 msgr2=0x7f44d8110d40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T10:39:02.790 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:39:02.788+0000 7f44e07bb640 1 --2- 172.21.15.114:0/1672070978 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f44d810e950 0x7f44d8110d40 secure :-1 s=READY pgs=892 cs=0 l=1 rev1=1 crypto rx=0x7f44d0009ca0 tx=0x7f44d002a8c0 comp rx=0 tx=0).stop 2024-04-23T10:39:02.790 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:39:02.788+0000 7f44e07bb640 1 -- 172.21.15.114:0/1672070978 shutdown_connections 2024-04-23T10:39:02.790 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:39:02.788+0000 7f44e07bb640 1 --2- 172.21.15.114:0/1672070978 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f44d810e950 0x7f44d8110d40 unknown :-1 s=CLOSED pgs=892 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:39:02.791 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:39:02.788+0000 7f44e07bb640 1 --2- 172.21.15.114:0/1672070978 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f44d81016c0 0x7f44d810e410 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:39:02.791 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:39:02.788+0000 7f44e07bb640 1 --2- 172.21.15.114:0/1672070978 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f44d8100da0 0x7f44d8101180 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:39:02.791 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:39:02.788+0000 7f44e07bb640 1 -- 172.21.15.114:0/1672070978 >> 172.21.15.114:0/1672070978 conn(0x7f44d8077fd0 msgr2=0x7f44d80783e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T10:39:02.791 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:39:02.788+0000 7f44e07bb640 1 -- 172.21.15.114:0/1672070978 shutdown_connections 2024-04-23T10:39:02.791 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:39:02.788+0000 7f44e07bb640 1 -- 172.21.15.114:0/1672070978 wait complete. 2024-04-23T10:39:02.791 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:39:02.789+0000 7f44e07bb640 1 Processor -- start 2024-04-23T10:39:02.791 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:39:02.789+0000 7f44e07bb640 1 -- start start 2024-04-23T10:39:02.791 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:39:02.789+0000 7f44e07bb640 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f44d8100da0 0x7f44d81a0610 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T10:39:02.791 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:39:02.789+0000 7f44e07bb640 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f44d81016c0 0x7f44d81a0b50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T10:39:02.791 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:39:02.789+0000 7f44e07bb640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f44d810e950 0x7f44d819a6d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T10:39:02.792 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:39:02.789+0000 7f44e07bb640 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x7f44d81143b0 con 0x7f44d810e950 2024-04-23T10:39:02.792 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:39:02.789+0000 7f44e07bb640 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 -- 0x7f44d8114230 con 0x7f44d8100da0 2024-04-23T10:39:02.792 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:39:02.789+0000 7f44e07bb640 1 -- --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_getmap magic: 0 -- 0x7f44d8114530 con 0x7f44d81016c0 2024-04-23T10:39:02.792 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:39:02.789+0000 7f44ded58640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f44d810e950 0x7f44d819a6d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T10:39:02.792 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:39:02.789+0000 7f44ded58640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f44d810e950 0x7f44d819a6d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.114:44744/0 (socket says 172.21.15.114:44744) 2024-04-23T10:39:02.792 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:39:02.789+0000 7f44ded58640 1 -- 172.21.15.114:0/3764529763 learned_addr learned my addr 172.21.15.114:0/3764529763 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-04-23T10:39:02.792 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:39:02.789+0000 7f44ddd56640 1 --2- 172.21.15.114:0/3764529763 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f44d81016c0 0x7f44d81a0b50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T10:39:02.792 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:39:02.789+0000 7f44de557640 1 --2- 172.21.15.114:0/3764529763 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f44d8100da0 0x7f44d81a0610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T10:39:02.792 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:39:02.789+0000 7f44ded58640 1 -- 172.21.15.114:0/3764529763 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f44d81016c0 msgr2=0x7f44d81a0b50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T10:39:02.792 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:39:02.789+0000 7f44ded58640 1 --2- 172.21.15.114:0/3764529763 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f44d81016c0 0x7f44d81a0b50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:39:02.792 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:39:02.789+0000 7f44ded58640 1 -- 172.21.15.114:0/3764529763 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f44d8100da0 msgr2=0x7f44d81a0610 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T10:39:02.793 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:39:02.789+0000 7f44ded58640 1 --2- 172.21.15.114:0/3764529763 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f44d8100da0 0x7f44d81a0610 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:39:02.793 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:39:02.789+0000 7f44ded58640 1 -- 172.21.15.114:0/3764529763 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f44d819af90 con 0x7f44d810e950 2024-04-23T10:39:02.793 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:39:02.789+0000 7f44de557640 1 --2- 172.21.15.114:0/3764529763 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f44d8100da0 0x7f44d81a0610 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-23T10:39:02.793 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:39:02.789+0000 7f44ded58640 1 --2- 172.21.15.114:0/3764529763 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f44d810e950 0x7f44d819a6d0 secure :-1 s=READY pgs=893 cs=0 l=1 rev1=1 crypto rx=0x7f44d00319f0 tx=0x7f44d0031a20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T10:39:02.793 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:39:02.790+0000 7f44cf7fe640 1 -- 172.21.15.114:0/3764529763 <== mon.0 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f44d0003bf0 con 0x7f44d810e950 2024-04-23T10:39:02.793 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:39:02.790+0000 7f44cf7fe640 1 -- 172.21.15.114:0/3764529763 <== mon.0 v2:172.21.15.114:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f44d0003d90 con 0x7f44d810e950 2024-04-23T10:39:02.793 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:39:02.790+0000 7f44e07bb640 1 -- 172.21.15.114:0/3764529763 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f44d819b220 con 0x7f44d810e950 2024-04-23T10:39:02.793 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:39:02.790+0000 7f44cf7fe640 1 -- 172.21.15.114:0/3764529763 <== mon.0 v2:172.21.15.114:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f44d002fc00 con 0x7f44d810e950 2024-04-23T10:39:02.793 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:39:02.790+0000 7f44e07bb640 1 -- 172.21.15.114:0/3764529763 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f44d81a7590 con 0x7f44d810e950 2024-04-23T10:39:02.802 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:39:02.793+0000 7f44cf7fe640 1 -- 172.21.15.114:0/3764529763 <== mon.0 v2:172.21.15.114:3300/0 4 ==== mgrmap(e 26) ==== 138778+0+0 (secure 0 0 0) 0x7f44d0004610 con 0x7f44d810e950 2024-04-23T10:39:02.803 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:39:02.793+0000 7f44e35ac740 1 Processor -- start 2024-04-23T10:39:02.803 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:39:02.793+0000 7f44e35ac740 1 -- start start 2024-04-23T10:39:02.803 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:39:02.793+0000 7f44e35ac740 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x558b90caa560 0x558b90caa940 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T10:39:02.803 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:39:02.793+0000 7f44e35ac740 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x558b90caae80 0x558b90cbdad0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T10:39:02.803 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:39:02.793+0000 7f44e35ac740 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x558b90cbe010 0x558b90cc0400 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T10:39:02.803 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:39:02.793+0000 7f44e35ac740 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x558b90c0f970 con 0x558b90caa560 2024-04-23T10:39:02.803 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:39:02.793+0000 7f44e35ac740 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 -- 0x558b90bac4a0 con 0x558b90cbe010 2024-04-23T10:39:02.803 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:39:02.793+0000 7f44e35ac740 1 -- --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_getmap magic: 0 -- 0x558b90c0b930 con 0x558b90caae80 2024-04-23T10:39:02.803 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:39:02.793+0000 7f44ddd56640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x558b90caa560 0x558b90caa940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T10:39:02.803 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:39:02.793+0000 7f44ddd56640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x558b90caa560 0x558b90caa940 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.114:44752/0 (socket says 172.21.15.114:44752) 2024-04-23T10:39:02.803 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:39:02.793+0000 7f44ddd56640 1 -- 172.21.15.114:0/1633529748 learned_addr learned my addr 172.21.15.114:0/1633529748 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-04-23T10:39:02.804 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:39:02.793+0000 7f44cf7fe640 1 --2- 172.21.15.114:0/3764529763 >> [v2:172.21.15.114:6800/1395693466,v1:172.21.15.114:6801/1395693466] conn(0x7f44c40a3dc0 0x7f44c40a6280 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T10:39:02.804 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:39:02.793+0000 7f44de557640 1 --2- 172.21.15.114:0/1633529748 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x558b90caae80 0x558b90cbdad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T10:39:02.804 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:39:02.794+0000 7f44ddd56640 1 --2- 172.21.15.114:0/1633529748 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x558b90cbe010 0x558b90cc0400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T10:39:02.804 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:39:02.794+0000 7f44ded58640 1 --2- 172.21.15.114:0/3764529763 >> [v2:172.21.15.114:6800/1395693466,v1:172.21.15.114:6801/1395693466] conn(0x7f44c40a3dc0 0x7f44c40a6280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T10:39:02.804 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:39:02.794+0000 7f44ddd56640 1 -- 172.21.15.114:0/1633529748 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x558b90caae80 msgr2=0x558b90cbdad0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T10:39:02.804 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:39:02.794+0000 7f44ddd56640 1 --2- 172.21.15.114:0/1633529748 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x558b90caae80 0x558b90cbdad0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:39:02.804 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:39:02.794+0000 7f44ddd56640 1 -- 172.21.15.114:0/1633529748 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x558b90cbe010 msgr2=0x558b90cc0400 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T10:39:02.804 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:39:02.794+0000 7f44ddd56640 1 --2- 172.21.15.114:0/1633529748 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x558b90cbe010 0x558b90cc0400 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:39:02.804 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:39:02.794+0000 7f44ddd56640 1 -- 172.21.15.114:0/1633529748 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x558b90922210 con 0x558b90caa560 2024-04-23T10:39:02.804 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:39:02.794+0000 7f44de557640 1 --2- 172.21.15.114:0/1633529748 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x558b90caae80 0x558b90cbdad0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-23T10:39:02.804 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:39:02.794+0000 7f44cf7fe640 1 -- 172.21.15.114:0/3764529763 <== mon.0 v2:172.21.15.114:3300/0 5 ==== osd_map(106..106 src has 1..106) ==== 8629+0+0 (secure 0 0 0) 0x7f44d00f1d30 con 0x7f44d810e950 2024-04-23T10:39:02.804 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:39:02.794+0000 7f44ddd56640 1 --2- 172.21.15.114:0/1633529748 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x558b90caa560 0x558b90caa940 secure :-1 s=READY pgs=894 cs=0 l=1 rev1=1 crypto rx=0x7f44c8012540 tx=0x7f44c800b7c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=dbb4a465725ae7b1 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T10:39:02.805 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:39:02.794+0000 7f44ded58640 1 --2- 172.21.15.114:0/3764529763 >> [v2:172.21.15.114:6800/1395693466,v1:172.21.15.114:6801/1395693466] conn(0x7f44c40a3dc0 0x7f44c40a6280 secure :-1 s=READY pgs=140 cs=0 l=1 rev1=1 crypto rx=0x7f44d00bda70 tx=0x7f44d00a88a0 comp rx=0 tx=0).ready entity=mgr.15291 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T10:39:02.805 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:39:02.794+0000 7f44ccff9640 1 -- 172.21.15.114:0/1633529748 <== mon.0 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f44c8004030 con 0x558b90caa560 2024-04-23T10:39:02.805 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:39:02.794+0000 7f44ccff9640 1 -- 172.21.15.114:0/1633529748 <== mon.0 v2:172.21.15.114:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f44c80041d0 con 0x558b90caa560 2024-04-23T10:39:02.805 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:39:02.794+0000 7f44ccff9640 1 -- 172.21.15.114:0/1633529748 <== mon.0 v2:172.21.15.114:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f44c800f6e0 con 0x558b90caa560 2024-04-23T10:39:02.805 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:39:02.795+0000 7f44e35ac740 1 -- 172.21.15.114:0/1633529748 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x558b90caa560 msgr2=0x558b90caa940 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T10:39:02.805 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:39:02.795+0000 7f44e35ac740 1 --2- 172.21.15.114:0/1633529748 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x558b90caa560 0x558b90caa940 secure :-1 s=READY pgs=894 cs=0 l=1 rev1=1 crypto rx=0x7f44c8012540 tx=0x7f44c800b7c0 comp rx=0 tx=0).stop 2024-04-23T10:39:02.805 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:39:02.795+0000 7f44e35ac740 1 -- 172.21.15.114:0/1633529748 shutdown_connections 2024-04-23T10:39:02.805 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:39:02.795+0000 7f44e35ac740 1 --2- 172.21.15.114:0/1633529748 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x558b90cbe010 0x558b90cc0400 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:39:02.805 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:39:02.795+0000 7f44e35ac740 1 --2- 172.21.15.114:0/1633529748 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x558b90caae80 0x558b90cbdad0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:39:02.805 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:39:02.795+0000 7f44e35ac740 1 --2- 172.21.15.114:0/1633529748 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x558b90caa560 0x558b90caa940 unknown :-1 s=CLOSED pgs=894 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:39:02.806 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:39:02.795+0000 7f44e35ac740 1 -- 172.21.15.114:0/1633529748 >> 172.21.15.114:0/1633529748 conn(0x558b90ca6b30 msgr2=0x558b90ca6f10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T10:39:02.806 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:39:02.795+0000 7f44e35ac740 1 -- 172.21.15.114:0/1633529748 shutdown_connections 2024-04-23T10:39:02.806 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:39:02.795+0000 7f44e35ac740 1 -- 172.21.15.114:0/1633529748 wait complete. 2024-04-23T10:39:02.806 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:39:02.795+0000 7f44e35ac740 1 -- start start 2024-04-23T10:39:02.806 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:39:02.795+0000 7f44e35ac740 1 Processor -- start 2024-04-23T10:39:02.806 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:39:02.796+0000 7f44e35ac740 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x558b90caa560 0x558b90cbd960 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T10:39:02.806 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:39:02.796+0000 7f44e35ac740 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x558b90caae80 0x558b90cbac50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T10:39:02.806 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:39:02.796+0000 7f44e35ac740 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x558b90cbe010 0x558b90cbb1b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T10:39:02.806 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:39:02.796+0000 7f44e35ac740 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x558b90cc0be0 con 0x558b90caa560 2024-04-23T10:39:02.806 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:39:02.796+0000 7f44e35ac740 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 -- 0x558b90baa0b0 con 0x558b90cbe010 2024-04-23T10:39:02.806 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:39:02.796+0000 7f44e35ac740 1 -- --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_getmap magic: 0 -- 0x558b90cc0d60 con 0x558b90caae80 2024-04-23T10:39:02.807 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:39:02.796+0000 7f44e35ac740 10 client.0 _ll_register_callbacks cb 0x558b90b97d30 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-23T10:39:02.807 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:39:02.796+0000 7f44ddd56640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x558b90caa560 0x558b90cbd960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T10:39:02.807 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:39:02.796+0000 7f44ddd56640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x558b90caa560 0x558b90cbd960 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.114:44764/0 (socket says 172.21.15.114:44764) 2024-04-23T10:39:02.807 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:39:02.796+0000 7f44ddd56640 1 -- 172.21.15.114:0/40972918 learned_addr learned my addr 172.21.15.114:0/40972918 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-04-23T10:39:02.807 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:39:02.796+0000 7f44de557640 1 --2- 172.21.15.114:0/40972918 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x558b90caae80 0x558b90cbac50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T10:39:02.807 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:39:02.796+0000 7f44ddd56640 1 -- 172.21.15.114:0/40972918 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x558b90caae80 msgr2=0x558b90cbac50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T10:39:02.807 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:39:02.796+0000 7f44ddd56640 1 --2- 172.21.15.114:0/40972918 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x558b90caae80 0x558b90cbac50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:39:02.807 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:39:02.796+0000 7f44ddd56640 1 -- 172.21.15.114:0/40972918 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x558b90cbe010 msgr2=0x558b90cbb1b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T10:39:02.807 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:39:02.796+0000 7f44ddd56640 1 --2- 172.21.15.114:0/40972918 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x558b90cbe010 0x558b90cbb1b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:39:02.807 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:39:02.796+0000 7f44ddd56640 1 -- 172.21.15.114:0/40972918 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({osdmap=0}) -- 0x558b909ee620 con 0x558b90caa560 2024-04-23T10:39:02.807 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:39:02.796+0000 7f44de557640 1 --2- 172.21.15.114:0/40972918 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x558b90caae80 0x558b90cbac50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-23T10:39:02.807 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:39:02.796+0000 7f44ddd56640 1 --2- 172.21.15.114:0/40972918 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x558b90caa560 0x558b90cbd960 secure :-1 s=READY pgs=895 cs=0 l=1 rev1=1 crypto rx=0x7f44c800b3d0 tx=0x7f44c800b930 comp rx=0 tx=0).ready entity=mon.0 client_cookie=466de0f615e237f3 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T10:39:02.808 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:39:02.796+0000 7f44c1ffb640 10 client.0 ms_handle_connect on v2:172.21.15.114:3300/0 2024-04-23T10:39:02.808 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:39:02.796+0000 7f44c1ffb640 1 -- 172.21.15.114:0/40972918 <== mon.0 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f44c80037f0 con 0x558b90caa560 2024-04-23T10:39:02.808 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:39:02.796+0000 7f44c1ffb640 1 -- 172.21.15.114:0/40972918 <== mon.0 v2:172.21.15.114:3300/0 2 ==== osd_map(106..106 src has 1..106) ==== 8629+0+0 (secure 0 0 0) 0x7f44c8003990 con 0x558b90caa560 2024-04-23T10:39:02.808 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:39:02.796+0000 7f44e35ac740 1 -- 172.21.15.114:0/40972918 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f44c80041d0 con 0x558b90caa560 2024-04-23T10:39:02.808 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:39:02.797+0000 7f44c1ffb640 1 -- 172.21.15.114:0/40972918 <== mon.0 v2:172.21.15.114:3300/0 3 ==== mon_get_version_reply(handle=1 version=42) ==== 24+0+0 (secure 0 0 0) 0x7f44c8005040 con 0x558b90caa560 2024-04-23T10:39:02.808 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:39:02.797+0000 7f44e35ac740 10 client.16959 fetch_fsmap learned FSMap version 42 2024-04-23T10:39:02.808 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:39:02.797+0000 7f44e35ac740 1 -- 172.21.15.114:0/40972918 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({config=0+,fsmap=42,monmap=4+}) -- 0x558b90cccbc0 con 0x558b90caa560 2024-04-23T10:39:02.808 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:39:02.797+0000 7f44c1ffb640 1 -- 172.21.15.114:0/40972918 <== mon.0 v2:172.21.15.114:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f44c80041d0 con 0x558b90caa560 2024-04-23T10:39:02.808 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:39:02.797+0000 7f44c1ffb640 1 -- 172.21.15.114:0/40972918 <== mon.0 v2:172.21.15.114:3300/0 5 ==== fsmap(e 42) ==== 8338+0+0 (secure 0 0 0) 0x7f44c8004a00 con 0x558b90caa560 2024-04-23T10:39:02.808 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:39:02.797+0000 7f44e35ac740 10 client.16959 fetch_fsmap finished waiting for FSMap version 42 2024-04-23T10:39:02.808 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:39:02.797+0000 7f44e35ac740 10 client.16959 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-04-23T10:39:02.808 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:39:02.797+0000 7f44e35ac740 20 client.16959 populate_metadata read hostname 'smithi114' 2024-04-23T10:39:02.809 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:39:02.797+0000 7f44e35ac740 1 --2- 172.21.15.114:0/40972918 >> [v2:172.21.15.119:6838/937297340,v1:172.21.15.119:6839/937297340] conn(0x558b90ccdfc0 0x558b90cd03b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T10:39:02.809 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:39:02.797+0000 7f44e35ac740 4 client.16959 mds_command: new command op to 24545 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-23T10:39:02.809 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:39:02.797+0000 7f44e35ac740 1 -- 172.21.15.114:0/40972918 --> [v2:172.21.15.119:6838/937297340,v1:172.21.15.119:6839/937297340] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x558b90c1d390 con 0x558b90ccdfc0 2024-04-23T10:39:02.809 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:39:02.797+0000 7f44de557640 1 --2- 172.21.15.114:0/40972918 >> [v2:172.21.15.119:6838/937297340,v1:172.21.15.119:6839/937297340] conn(0x558b90ccdfc0 0x558b90cd03b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T10:39:02.809 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:39:02.798+0000 7f44c1ffb640 10 client.16959 ms_handle_connect on v2:172.21.15.119:6838/937297340 2024-04-23T10:39:02.809 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:39:02.798+0000 7f44de557640 1 --2- 172.21.15.114:0/40972918 >> [v2:172.21.15.119:6838/937297340,v1:172.21.15.119:6839/937297340] conn(0x558b90ccdfc0 0x558b90cd03b0 crc :-1 s=READY pgs=172 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=ecec98380e4c6d59 server_cookie=366c138b893ec3f2 in_seq=0 out_seq=0 2024-04-23T10:39:02.809 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:39:02.798+0000 7f44c1ffb640 1 -- 172.21.15.114:0/40972918 <== mds.0 v2:172.21.15.119:6838/937297340 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x558b90c1d390 con 0x558b90ccdfc0 2024-04-23T10:39:02.809 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:39:02.798+0000 7f44c1ffb640 10 client.16959 handle_command_reply: tid=0 2024-04-23T10:39:02.809 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:39:02.798+0000 7f44e35ac740 1 client.16959 shutdown 2024-04-23T10:39:02.809 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:39:02.799+0000 7f44e35ac740 1 -- 172.21.15.114:0/40972918 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x558b90caa560 msgr2=0x558b90cbd960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T10:39:02.809 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:39:02.799+0000 7f44e35ac740 1 --2- 172.21.15.114:0/40972918 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x558b90caa560 0x558b90cbd960 secure :-1 s=READY pgs=895 cs=0 l=1 rev1=1 crypto rx=0x7f44c800b3d0 tx=0x7f44c800b930 comp rx=0 tx=0).stop 2024-04-23T10:39:02.810 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:39:02.799+0000 7f44e35ac740 1 -- 172.21.15.114:0/40972918 shutdown_connections 2024-04-23T10:39:02.810 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:39:02.799+0000 7f44e35ac740 1 --2- 172.21.15.114:0/40972918 >> [v2:172.21.15.119:6838/937297340,v1:172.21.15.119:6839/937297340] conn(0x558b90ccdfc0 0x558b90cd03b0 crc :-1 s=READY pgs=172 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:39:02.810 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:39:02.799+0000 7f44e35ac740 1 --2- 172.21.15.114:0/40972918 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x558b90cbe010 0x558b90cbb1b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:39:02.810 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:39:02.799+0000 7f44e35ac740 1 --2- 172.21.15.114:0/40972918 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x558b90caae80 0x558b90cbac50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:39:02.810 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:39:02.799+0000 7f44c1ffb640 0 client.16959 ms_handle_reset on v2:172.21.15.119:6838/937297340 2024-04-23T10:39:02.810 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:39:02.799+0000 7f44e35ac740 1 --2- 172.21.15.114:0/40972918 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x558b90caa560 0x558b90cbd960 unknown :-1 s=CLOSED pgs=895 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:39:02.810 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:39:02.799+0000 7f44e35ac740 1 -- 172.21.15.114:0/40972918 >> 172.21.15.114:0/40972918 conn(0x558b90ca6b30 msgr2=0x558b90cac8b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T10:39:02.810 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:39:02.799+0000 7f44e35ac740 1 -- 172.21.15.114:0/40972918 shutdown_connections 2024-04-23T10:39:02.810 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:39:02.799+0000 7f44e35ac740 1 -- 172.21.15.114:0/40972918 wait complete. 2024-04-23T10:39:02.810 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:39:02.799+0000 7f44e35ac740 20 client.16959 trim_cache size 0 max 16384 2024-04-23T10:39:02.817 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:39:02.813+0000 7f44e35ac740 1 Processor -- start 2024-04-23T10:39:02.818 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:39:02.813+0000 7f44e35ac740 1 -- start start 2024-04-23T10:39:02.818 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:39:02.814+0000 7f44e35ac740 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x558b90caa560 0x558b90ca5b30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T10:39:02.818 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:39:02.814+0000 7f44e35ac740 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x558b90caae80 0x558b90cbd490 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T10:39:02.818 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:39:02.814+0000 7f44e35ac740 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x558b90cbe010 0x558b90cd5af0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T10:39:02.818 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:39:02.814+0000 7f44e35ac740 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x7f44d81ae400 con 0x558b90caae80 2024-04-23T10:39:02.818 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:39:02.814+0000 7f44e35ac740 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 -- 0x7f44d81abcd0 con 0x558b90cbe010 2024-04-23T10:39:02.818 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:39:02.814+0000 7f44e35ac740 1 -- --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_getmap magic: 0 -- 0x7f44d81a9b40 con 0x558b90caa560 2024-04-23T10:39:02.818 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:39:02.814+0000 7f44de557640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x558b90caae80 0x558b90cbd490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T10:39:02.818 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:39:02.814+0000 7f44de557640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x558b90caae80 0x558b90cbd490 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.114:44770/0 (socket says 172.21.15.114:44770) 2024-04-23T10:39:02.819 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:39:02.814+0000 7f44de557640 1 -- 172.21.15.114:0/2471872552 learned_addr learned my addr 172.21.15.114:0/2471872552 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-04-23T10:39:02.819 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:39:02.814+0000 7f44ddd56640 1 --2- 172.21.15.114:0/2471872552 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x558b90caa560 0x558b90ca5b30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T10:39:02.819 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:39:02.814+0000 7f44ddd56640 1 --2- 172.21.15.114:0/2471872552 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x558b90cbe010 0x558b90cd5af0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T10:39:02.819 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:39:02.814+0000 7f44de557640 1 -- 172.21.15.114:0/2471872552 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x558b90caa560 msgr2=0x558b90ca5b30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T10:39:02.819 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:39:02.814+0000 7f44de557640 1 --2- 172.21.15.114:0/2471872552 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x558b90caa560 0x558b90ca5b30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:39:02.819 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:39:02.814+0000 7f44de557640 1 -- 172.21.15.114:0/2471872552 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x558b90cbe010 msgr2=0x558b90cd5af0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T10:39:02.819 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:39:02.814+0000 7f44de557640 1 --2- 172.21.15.114:0/2471872552 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x558b90cbe010 0x558b90cd5af0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:39:02.819 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:39:02.814+0000 7f44de557640 1 -- 172.21.15.114:0/2471872552 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x558b90cabaa0 con 0x558b90caae80 2024-04-23T10:39:02.819 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:39:02.814+0000 7f44de557640 1 --2- 172.21.15.114:0/2471872552 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x558b90caae80 0x558b90cbd490 secure :-1 s=READY pgs=896 cs=0 l=1 rev1=1 crypto rx=0x7f44d4005940 tx=0x7f44d40097d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d1ccc05f6ba3e1cc server_cookie=0 in_seq=0 out_seq=0 2024-04-23T10:39:02.819 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:39:02.814+0000 7f44c3fff640 1 -- 172.21.15.114:0/2471872552 <== mon.0 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f44d4013e80 con 0x558b90caae80 2024-04-23T10:39:02.819 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:39:02.814+0000 7f44c3fff640 1 -- 172.21.15.114:0/2471872552 <== mon.0 v2:172.21.15.114:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f44d401e030 con 0x558b90caae80 2024-04-23T10:39:02.819 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:39:02.814+0000 7f44c3fff640 1 -- 172.21.15.114:0/2471872552 <== mon.0 v2:172.21.15.114:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f44d4004730 con 0x558b90caae80 2024-04-23T10:39:02.820 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:39:02.815+0000 7f44e35ac740 1 -- 172.21.15.114:0/2471872552 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x558b90caae80 msgr2=0x558b90cbd490 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T10:39:02.820 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:39:02.815+0000 7f44e35ac740 1 --2- 172.21.15.114:0/2471872552 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x558b90caae80 0x558b90cbd490 secure :-1 s=READY pgs=896 cs=0 l=1 rev1=1 crypto rx=0x7f44d4005940 tx=0x7f44d40097d0 comp rx=0 tx=0).stop 2024-04-23T10:39:02.820 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:39:02.815+0000 7f44e35ac740 1 -- 172.21.15.114:0/2471872552 shutdown_connections 2024-04-23T10:39:02.820 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:39:02.815+0000 7f44e35ac740 1 --2- 172.21.15.114:0/2471872552 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x558b90cbe010 0x558b90cd5af0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:39:02.820 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:39:02.815+0000 7f44e35ac740 1 --2- 172.21.15.114:0/2471872552 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x558b90caae80 0x558b90cbd490 unknown :-1 s=CLOSED pgs=896 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:39:02.820 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:39:02.815+0000 7f44e35ac740 1 --2- 172.21.15.114:0/2471872552 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x558b90caa560 0x558b90ca5b30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:39:02.820 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:39:02.815+0000 7f44e35ac740 1 -- 172.21.15.114:0/2471872552 >> 172.21.15.114:0/2471872552 conn(0x558b90ca6b30 msgr2=0x558b90ca5420 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T10:39:02.820 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:39:02.815+0000 7f44e35ac740 1 -- 172.21.15.114:0/2471872552 shutdown_connections 2024-04-23T10:39:02.820 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:39:02.815+0000 7f44e35ac740 1 -- 172.21.15.114:0/2471872552 wait complete. 2024-04-23T10:39:02.820 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:39:02.815+0000 7f44e35ac740 1 -- start start 2024-04-23T10:39:02.820 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:39:02.815+0000 7f44e35ac740 1 Processor -- start 2024-04-23T10:39:02.820 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:39:02.816+0000 7f44e35ac740 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x558b90caa560 0x558b90cc64b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T10:39:02.821 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:39:02.816+0000 7f44e35ac740 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x558b90caae80 0x558b90cb77e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T10:39:02.821 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:39:02.816+0000 7f44e35ac740 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x558b90cbe010 0x558b90cb7df0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T10:39:02.821 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:39:02.816+0000 7f44e35ac740 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x7f44d81aa190 con 0x558b90caa560 2024-04-23T10:39:02.821 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:39:02.816+0000 7f44e35ac740 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 -- 0x7f44d81ab600 con 0x558b90cbe010 2024-04-23T10:39:02.821 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:39:02.816+0000 7f44e35ac740 1 -- --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_getmap magic: 0 -- 0x7f44d81aa860 con 0x558b90caae80 2024-04-23T10:39:02.821 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:39:02.816+0000 7f44e35ac740 10 client.0 _ll_register_callbacks cb 0x7f44c8004380 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-23T10:39:02.821 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:39:02.816+0000 7f44ddd56640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x558b90caa560 0x558b90cc64b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T10:39:02.821 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:39:02.816+0000 7f44ddd56640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x558b90caa560 0x558b90cc64b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.114:44780/0 (socket says 172.21.15.114:44780) 2024-04-23T10:39:02.821 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:39:02.816+0000 7f44ddd56640 1 -- 172.21.15.114:0/1499961791 learned_addr learned my addr 172.21.15.114:0/1499961791 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-04-23T10:39:02.821 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:39:02.816+0000 7f44ddd56640 1 -- 172.21.15.114:0/1499961791 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x558b90caae80 msgr2=0x558b90cb77e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T10:39:02.821 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:39:02.816+0000 7f44ddd56640 1 --2- 172.21.15.114:0/1499961791 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x558b90caae80 0x558b90cb77e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:39:02.821 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:39:02.816+0000 7f44ddd56640 1 -- 172.21.15.114:0/1499961791 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x558b90cbe010 msgr2=0x558b90cb7df0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T10:39:02.822 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:39:02.816+0000 7f44ddd56640 1 --2- 172.21.15.114:0/1499961791 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x558b90cbe010 0x558b90cb7df0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:39:02.822 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:39:02.816+0000 7f44ddd56640 1 -- 172.21.15.114:0/1499961791 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({osdmap=0}) -- 0x558b90cb8620 con 0x558b90caa560 2024-04-23T10:39:02.822 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:39:02.816+0000 7f44ddd56640 1 --2- 172.21.15.114:0/1499961791 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x558b90caa560 0x558b90cc64b0 secure :-1 s=READY pgs=897 cs=0 l=1 rev1=1 crypto rx=0x7f44c80042f0 tx=0x7f44c800bce0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8ca6c0797cb42b70 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T10:39:02.822 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:39:02.816+0000 7f44c1ffb640 10 client.0 ms_handle_connect on v2:172.21.15.114:3300/0 2024-04-23T10:39:02.822 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:39:02.816+0000 7f44c1ffb640 1 -- 172.21.15.114:0/1499961791 <== mon.0 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f44c8037070 con 0x558b90caa560 2024-04-23T10:39:02.822 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:39:02.816+0000 7f44c1ffb640 1 -- 172.21.15.114:0/1499961791 <== mon.0 v2:172.21.15.114:3300/0 2 ==== osd_map(106..106 src has 1..106) ==== 8629+0+0 (secure 0 0 0) 0x7f44c800b760 con 0x558b90caa560 2024-04-23T10:39:02.822 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:39:02.816+0000 7f44e35ac740 1 -- 172.21.15.114:0/1499961791 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f44d401e030 con 0x558b90caa560 2024-04-23T10:39:02.822 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:39:02.816+0000 7f44c1ffb640 1 -- 172.21.15.114:0/1499961791 <== mon.0 v2:172.21.15.114:3300/0 3 ==== mon_get_version_reply(handle=1 version=42) ==== 24+0+0 (secure 0 0 0) 0x7f44c8043050 con 0x558b90caa560 2024-04-23T10:39:02.822 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:39:02.816+0000 7f44e35ac740 10 client.16965 fetch_fsmap learned FSMap version 42 2024-04-23T10:39:02.822 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:39:02.816+0000 7f44e35ac740 1 -- 172.21.15.114:0/1499961791 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({config=0+,fsmap=42,monmap=4+}) -- 0x558b90d34470 con 0x558b90caa560 2024-04-23T10:39:02.822 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:39:02.817+0000 7f44c1ffb640 1 -- 172.21.15.114:0/1499961791 <== mon.0 v2:172.21.15.114:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f44d401e030 con 0x558b90caa560 2024-04-23T10:39:02.822 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:39:02.817+0000 7f44c1ffb640 1 -- 172.21.15.114:0/1499961791 <== mon.0 v2:172.21.15.114:3300/0 5 ==== fsmap(e 42) ==== 8338+0+0 (secure 0 0 0) 0x7f44c8004660 con 0x558b90caa560 2024-04-23T10:39:02.823 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:39:02.817+0000 7f44e35ac740 10 client.16965 fetch_fsmap finished waiting for FSMap version 42 2024-04-23T10:39:02.823 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:39:02.817+0000 7f44e35ac740 10 client.16965 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-04-23T10:39:02.823 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:39:02.817+0000 7f44e35ac740 20 client.16965 populate_metadata read hostname 'smithi114' 2024-04-23T10:39:02.823 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:39:02.817+0000 7f44e35ac740 1 --2- 172.21.15.114:0/1499961791 >> [v2:172.21.15.119:6838/937297340,v1:172.21.15.119:6839/937297340] conn(0x558b90d34ae0 0x558b90d36f00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T10:39:02.823 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:39:02.817+0000 7f44e35ac740 4 client.16965 mds_command: new command op to 24545 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-04-23T10:39:02.823 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:39:02.817+0000 7f44e35ac740 1 -- 172.21.15.114:0/1499961791 --> [v2:172.21.15.119:6838/937297340,v1:172.21.15.119:6839/937297340] -- command(tid 0: {"prefix": "flush journal"}) -- 0x558b90c17200 con 0x558b90d34ae0 2024-04-23T10:39:02.823 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:39:02.817+0000 7f44de557640 1 --2- 172.21.15.114:0/1499961791 >> [v2:172.21.15.119:6838/937297340,v1:172.21.15.119:6839/937297340] conn(0x558b90d34ae0 0x558b90d36f00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T10:39:02.823 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:39:02.818+0000 7f44c1ffb640 10 client.16965 ms_handle_connect on v2:172.21.15.119:6838/937297340 2024-04-23T10:39:02.823 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:39:02.818+0000 7f44de557640 1 --2- 172.21.15.114:0/1499961791 >> [v2:172.21.15.119:6838/937297340,v1:172.21.15.119:6839/937297340] conn(0x558b90d34ae0 0x558b90d36f00 crc :-1 s=READY pgs=173 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=be30b9816905b436 server_cookie=914d9ebf38ae00cc in_seq=0 out_seq=0 2024-04-23T10:39:02.858 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:39:02.859+0000 7f44c1ffb640 1 -- 172.21.15.114:0/1499961791 <== mds.0 v2:172.21.15.119:6838/937297340 1 ==== command_reply(tid 0: 0 ) ==== 8+0+44 (crc 0 0 0) 0x558b90c17200 con 0x558b90d34ae0 2024-04-23T10:39:02.859 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:39:02.859+0000 7f44c1ffb640 10 client.16965 handle_command_reply: tid=0 2024-04-23T10:39:02.859 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:39:02.859+0000 7f44e35ac740 1 client.16965 shutdown 2024-04-23T10:39:02.859 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:39:02.859+0000 7f44e35ac740 1 -- 172.21.15.114:0/1499961791 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x558b90caa560 msgr2=0x558b90cc64b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T10:39:02.859 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:39:02.859+0000 7f44e35ac740 1 --2- 172.21.15.114:0/1499961791 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x558b90caa560 0x558b90cc64b0 secure :-1 s=READY pgs=897 cs=0 l=1 rev1=1 crypto rx=0x7f44c80042f0 tx=0x7f44c800bce0 comp rx=0 tx=0).stop 2024-04-23T10:39:02.859 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:39:02.859+0000 7f44e35ac740 1 -- 172.21.15.114:0/1499961791 shutdown_connections 2024-04-23T10:39:02.859 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:39:02.859+0000 7f44e35ac740 1 --2- 172.21.15.114:0/1499961791 >> [v2:172.21.15.119:6838/937297340,v1:172.21.15.119:6839/937297340] conn(0x558b90d34ae0 0x558b90d36f00 crc :-1 s=READY pgs=173 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:39:02.859 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:39:02.859+0000 7f44e35ac740 1 --2- 172.21.15.114:0/1499961791 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x558b90cbe010 0x558b90cb7df0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:39:02.859 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:39:02.859+0000 7f44c1ffb640 0 client.16965 ms_handle_reset on v2:172.21.15.119:6838/937297340 2024-04-23T10:39:02.859 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:39:02.859+0000 7f44e35ac740 1 --2- 172.21.15.114:0/1499961791 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x558b90caae80 0x558b90cb77e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:39:02.859 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:39:02.859+0000 7f44e35ac740 1 --2- 172.21.15.114:0/1499961791 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x558b90caa560 0x558b90cc64b0 unknown :-1 s=CLOSED pgs=897 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:39:02.859 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:39:02.859+0000 7f44e35ac740 1 -- 172.21.15.114:0/1499961791 >> 172.21.15.114:0/1499961791 conn(0x558b90ca6b30 msgr2=0x558b90ca5420 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T10:39:02.859 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:39:02.859+0000 7f44e35ac740 1 -- 172.21.15.114:0/1499961791 shutdown_connections 2024-04-23T10:39:02.860 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:39:02.859+0000 7f44e35ac740 1 -- 172.21.15.114:0/1499961791 wait complete. 2024-04-23T10:39:02.860 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:39:02.859+0000 7f44e35ac740 20 client.16965 trim_cache size 0 max 16384 2024-04-23T10:39:02.860 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stdout:{ 2024-04-23T10:39:02.860 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stdout: "message": "", 2024-04-23T10:39:02.860 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stdout: "return_code": 0 2024-04-23T10:39:02.860 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stdout:} 2024-04-23T10:39:02.861 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:39:02.860+0000 7f44e07bb640 1 -- 172.21.15.114:0/3764529763 >> [v2:172.21.15.114:6800/1395693466,v1:172.21.15.114:6801/1395693466] conn(0x7f44c40a3dc0 msgr2=0x7f44c40a6280 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T10:39:02.861 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:39:02.860+0000 7f44e07bb640 1 --2- 172.21.15.114:0/3764529763 >> [v2:172.21.15.114:6800/1395693466,v1:172.21.15.114:6801/1395693466] conn(0x7f44c40a3dc0 0x7f44c40a6280 secure :-1 s=READY pgs=140 cs=0 l=1 rev1=1 crypto rx=0x7f44d00bda70 tx=0x7f44d00a88a0 comp rx=0 tx=0).stop 2024-04-23T10:39:02.861 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:39:02.860+0000 7f44e07bb640 1 -- 172.21.15.114:0/3764529763 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f44d810e950 msgr2=0x7f44d819a6d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T10:39:02.861 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:39:02.860+0000 7f44e07bb640 1 --2- 172.21.15.114:0/3764529763 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f44d810e950 0x7f44d819a6d0 secure :-1 s=READY pgs=893 cs=0 l=1 rev1=1 crypto rx=0x7f44d00319f0 tx=0x7f44d0031a20 comp rx=0 tx=0).stop 2024-04-23T10:39:02.862 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:39:02.860+0000 7f44e07bb640 1 -- 172.21.15.114:0/3764529763 shutdown_connections 2024-04-23T10:39:02.862 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:39:02.860+0000 7f44e07bb640 1 --2- 172.21.15.114:0/3764529763 >> [v2:172.21.15.114:6800/1395693466,v1:172.21.15.114:6801/1395693466] conn(0x7f44c40a3dc0 0x7f44c40a6280 unknown :-1 s=CLOSED pgs=140 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:39:02.862 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:39:02.860+0000 7f44e07bb640 1 --2- 172.21.15.114:0/3764529763 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f44d810e950 0x7f44d819a6d0 unknown :-1 s=CLOSED pgs=893 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:39:02.862 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:39:02.860+0000 7f44e07bb640 1 --2- 172.21.15.114:0/3764529763 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f44d81016c0 0x7f44d81a0b50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:39:02.862 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:39:02.860+0000 7f44e07bb640 1 --2- 172.21.15.114:0/3764529763 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f44d8100da0 0x7f44d81a0610 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:39:02.862 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:39:02.860+0000 7f44e07bb640 1 -- 172.21.15.114:0/3764529763 >> 172.21.15.114:0/3764529763 conn(0x7f44d8077fd0 msgr2=0x7f44d810fe00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T10:39:02.862 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:39:02.860+0000 7f44e07bb640 1 -- 172.21.15.114:0/3764529763 shutdown_connections 2024-04-23T10:39:02.862 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:39:02.860+0000 7f44e07bb640 1 -- 172.21.15.114:0/3764529763 wait complete. 2024-04-23T10:39:02.949 INFO:tasks.workunit.client.0.smithi114.stdout: 1 703266 5.17 MB/sec execute 551 sec latency 3671.082 ms 2024-04-23T10:39:03.950 INFO:tasks.workunit.client.0.smithi114.stdout: 1 703266 5.16 MB/sec execute 552 sec latency 4671.179 ms 2024-04-23T10:39:04.241 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 23 10:39:03 smithi119 ceph-mon[32608]: pgmap v390: 97 pgs: 97 active+clean; 221 MiB data, 2.7 GiB used, 1.0 TiB / 1.0 TiB avail; 363 KiB/s rd, 909 KiB/s wr, 79 op/s 2024-04-23T10:39:04.253 INFO:journalctl@ceph.mon.a.smithi114.stdout:Apr 23 10:39:03 smithi114 ceph-mon[29255]: pgmap v390: 97 pgs: 97 active+clean; 221 MiB data, 2.7 GiB used, 1.0 TiB / 1.0 TiB avail; 363 KiB/s rd, 909 KiB/s wr, 79 op/s 2024-04-23T10:39:04.327 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 23 10:39:03 smithi204 ceph-mon[32649]: pgmap v390: 97 pgs: 97 active+clean; 221 MiB data, 2.7 GiB used, 1.0 TiB / 1.0 TiB avail; 363 KiB/s rd, 909 KiB/s wr, 79 op/s 2024-04-23T10:39:04.950 INFO:tasks.workunit.client.0.smithi114.stdout: 1 703550 5.15 MB/sec execute 553 sec latency 4913.990 ms 2024-04-23T10:39:05.949 INFO:tasks.workunit.client.0.smithi114.stdout: 1 705045 5.16 MB/sec execute 554 sec latency 22.932 ms 2024-04-23T10:39:06.241 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 23 10:39:05 smithi119 ceph-mon[32608]: pgmap v391: 97 pgs: 97 active+clean; 221 MiB data, 2.7 GiB used, 1.0 TiB / 1.0 TiB avail; 296 KiB/s rd, 675 KiB/s wr, 56 op/s 2024-04-23T10:39:06.253 INFO:journalctl@ceph.mon.a.smithi114.stdout:Apr 23 10:39:05 smithi114 ceph-mon[29255]: pgmap v391: 97 pgs: 97 active+clean; 221 MiB data, 2.7 GiB used, 1.0 TiB / 1.0 TiB avail; 296 KiB/s rd, 675 KiB/s wr, 56 op/s 2024-04-23T10:39:06.327 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 23 10:39:05 smithi204 ceph-mon[32649]: pgmap v391: 97 pgs: 97 active+clean; 221 MiB data, 2.7 GiB used, 1.0 TiB / 1.0 TiB avail; 296 KiB/s rd, 675 KiB/s wr, 56 op/s 2024-04-23T10:39:06.950 INFO:tasks.workunit.client.0.smithi114.stdout: 1 706746 5.17 MB/sec execute 555 sec latency 379.849 ms 2024-04-23T10:39:07.950 INFO:tasks.workunit.client.0.smithi114.stdout: 1 706746 5.16 MB/sec execute 556 sec latency 1379.931 ms 2024-04-23T10:39:08.242 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 23 10:39:07 smithi119 ceph-mon[32608]: pgmap v392: 97 pgs: 97 active+clean; 227 MiB data, 2.8 GiB used, 1.0 TiB / 1.0 TiB avail; 540 KiB/s rd, 1.3 MiB/s wr, 144 op/s 2024-04-23T10:39:08.254 INFO:journalctl@ceph.mon.a.smithi114.stdout:Apr 23 10:39:07 smithi114 ceph-mon[29255]: pgmap v392: 97 pgs: 97 active+clean; 227 MiB data, 2.8 GiB used, 1.0 TiB / 1.0 TiB avail; 540 KiB/s rd, 1.3 MiB/s wr, 144 op/s 2024-04-23T10:39:08.328 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 23 10:39:07 smithi204 ceph-mon[32649]: pgmap v392: 97 pgs: 97 active+clean; 227 MiB data, 2.8 GiB used, 1.0 TiB / 1.0 TiB avail; 540 KiB/s rd, 1.3 MiB/s wr, 144 op/s 2024-04-23T10:39:08.950 INFO:tasks.workunit.client.0.smithi114.stdout: 1 706746 5.15 MB/sec execute 557 sec latency 2380.066 ms 2024-04-23T10:39:09.950 INFO:tasks.workunit.client.0.smithi114.stdout: 1 706813 5.14 MB/sec execute 558 sec latency 2632.076 ms 2024-04-23T10:39:10.254 INFO:journalctl@ceph.mon.a.smithi114.stdout:Apr 23 10:39:09 smithi114 ceph-mon[29255]: pgmap v393: 97 pgs: 97 active+clean; 227 MiB data, 2.8 GiB used, 1.0 TiB / 1.0 TiB avail; 296 KiB/s rd, 758 KiB/s wr, 100 op/s 2024-04-23T10:39:10.254 INFO:journalctl@ceph.mon.a.smithi114.stdout:Apr 23 10:39:09 smithi114 ceph-mon[29255]: osdmap e107: 12 total, 12 up, 12 in 2024-04-23T10:39:10.327 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 23 10:39:09 smithi204 ceph-mon[32649]: pgmap v393: 97 pgs: 97 active+clean; 227 MiB data, 2.8 GiB used, 1.0 TiB / 1.0 TiB avail; 296 KiB/s rd, 758 KiB/s wr, 100 op/s 2024-04-23T10:39:10.328 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 23 10:39:09 smithi204 ceph-mon[32649]: osdmap e107: 12 total, 12 up, 12 in 2024-04-23T10:39:10.492 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 23 10:39:09 smithi119 ceph-mon[32608]: pgmap v393: 97 pgs: 97 active+clean; 227 MiB data, 2.8 GiB used, 1.0 TiB / 1.0 TiB avail; 296 KiB/s rd, 758 KiB/s wr, 100 op/s 2024-04-23T10:39:10.492 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 23 10:39:09 smithi119 ceph-mon[32608]: osdmap e107: 12 total, 12 up, 12 in 2024-04-23T10:39:10.950 INFO:tasks.workunit.client.0.smithi114.stdout: 1 706813 5.13 MB/sec execute 559 sec latency 1536.653 ms 2024-04-23T10:39:11.951 INFO:tasks.workunit.client.0.smithi114.stdout: 1 706813 5.12 MB/sec execute 560 sec latency 2536.770 ms 2024-04-23T10:39:12.254 INFO:journalctl@ceph.mon.a.smithi114.stdout:Apr 23 10:39:12 smithi114 ceph-mon[29255]: pgmap v395: 97 pgs: 97 active+clean; 227 MiB data, 2.8 GiB used, 1.0 TiB / 1.0 TiB avail; 355 KiB/s rd, 910 KiB/s wr, 120 op/s 2024-04-23T10:39:12.328 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 23 10:39:12 smithi204 ceph-mon[32649]: pgmap v395: 97 pgs: 97 active+clean; 227 MiB data, 2.8 GiB used, 1.0 TiB / 1.0 TiB avail; 355 KiB/s rd, 910 KiB/s wr, 120 op/s 2024-04-23T10:39:12.492 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 23 10:39:12 smithi119 ceph-mon[32608]: pgmap v395: 97 pgs: 97 active+clean; 227 MiB data, 2.8 GiB used, 1.0 TiB / 1.0 TiB avail; 355 KiB/s rd, 910 KiB/s wr, 120 op/s 2024-04-23T10:39:12.950 INFO:tasks.workunit.client.0.smithi114.stdout: 1 706813 5.11 MB/sec execute 561 sec latency 3536.860 ms 2024-04-23T10:39:13.951 INFO:tasks.workunit.client.0.smithi114.stdout: 1 706813 5.10 MB/sec execute 562 sec latency 4536.986 ms 2024-04-23T10:39:14.254 INFO:journalctl@ceph.mon.a.smithi114.stdout:Apr 23 10:39:14 smithi114 ceph-mon[29255]: pgmap v396: 97 pgs: 97 active+clean; 203 MiB data, 2.7 GiB used, 1.0 TiB / 1.0 TiB avail; 451 KiB/s rd, 953 KiB/s wr, 127 op/s 2024-04-23T10:39:14.254 INFO:journalctl@ceph.mon.a.smithi114.stdout:Apr 23 10:39:14 smithi114 ceph-mon[29255]: osdmap e108: 12 total, 12 up, 12 in 2024-04-23T10:39:14.327 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 23 10:39:14 smithi204 ceph-mon[32649]: pgmap v396: 97 pgs: 97 active+clean; 203 MiB data, 2.7 GiB used, 1.0 TiB / 1.0 TiB avail; 451 KiB/s rd, 953 KiB/s wr, 127 op/s 2024-04-23T10:39:14.327 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 23 10:39:14 smithi204 ceph-mon[32649]: osdmap e108: 12 total, 12 up, 12 in 2024-04-23T10:39:14.492 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 23 10:39:14 smithi119 ceph-mon[32608]: pgmap v396: 97 pgs: 97 active+clean; 203 MiB data, 2.7 GiB used, 1.0 TiB / 1.0 TiB avail; 451 KiB/s rd, 953 KiB/s wr, 127 op/s 2024-04-23T10:39:14.492 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 23 10:39:14 smithi119 ceph-mon[32608]: osdmap e108: 12 total, 12 up, 12 in 2024-04-23T10:39:14.950 INFO:tasks.workunit.client.0.smithi114.stdout: 1 707547 5.10 MB/sec execute 563 sec latency 4772.009 ms 2024-04-23T10:39:15.952 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:39:15.949+0000 7f7c0ef19640 1 -- 172.21.15.114:0/3746806844 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f7c08106170 msgr2=0x7f7c0810dd40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T10:39:15.952 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:39:15.949+0000 7f7c0ef19640 1 --2- 172.21.15.114:0/3746806844 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f7c08106170 0x7f7c0810dd40 secure :-1 s=READY pgs=898 cs=0 l=1 rev1=1 crypto rx=0x7f7c00009ca0 tx=0x7f7c0002a8c0 comp rx=0 tx=0).stop 2024-04-23T10:39:15.952 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:39:15.949+0000 7f7c0ef19640 1 -- 172.21.15.114:0/3746806844 shutdown_connections 2024-04-23T10:39:15.952 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:39:15.949+0000 7f7c0ef19640 1 --2- 172.21.15.114:0/3746806844 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f7c08106170 0x7f7c0810dd40 unknown :-1 s=CLOSED pgs=898 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:39:15.952 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:39:15.949+0000 7f7c0ef19640 1 --2- 172.21.15.114:0/3746806844 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f7c08101aa0 0x7f7c08105a90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:39:15.952 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:39:15.949+0000 7f7c0ef19640 1 --2- 172.21.15.114:0/3746806844 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f7c081010f0 0x7f7c081014d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:39:15.953 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:39:15.949+0000 7f7c0ef19640 1 -- 172.21.15.114:0/3746806844 >> 172.21.15.114:0/3746806844 conn(0x7f7c08077fd0 msgr2=0x7f7c080783e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T10:39:15.953 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:39:15.949+0000 7f7c0ef19640 1 -- 172.21.15.114:0/3746806844 shutdown_connections 2024-04-23T10:39:15.953 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:39:15.949+0000 7f7c0ef19640 1 -- 172.21.15.114:0/3746806844 wait complete. 2024-04-23T10:39:15.953 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:39:15.949+0000 7f7c0ef19640 1 Processor -- start 2024-04-23T10:39:15.953 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:39:15.949+0000 7f7c0ef19640 1 -- start start 2024-04-23T10:39:15.953 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:39:15.949+0000 7f7c0ef19640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f7c081010f0 0x7f7c081986a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T10:39:15.953 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:39:15.949+0000 7f7c0ef19640 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f7c08106170 0x7f7c08198be0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T10:39:15.953 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:39:15.950+0000 7f7c0ef19640 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f7c0819cec0 0x7f7c0819d370 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T10:39:15.953 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:39:15.950+0000 7f7c0ef19640 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x7f7c081110c0 con 0x7f7c081010f0 2024-04-23T10:39:15.954 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:39:15.950+0000 7f7c0ef19640 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 -- 0x7f7c08110f40 con 0x7f7c0819cec0 2024-04-23T10:39:15.954 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:39:15.950+0000 7f7c0ef19640 1 -- --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_getmap magic: 0 -- 0x7f7c08111240 con 0x7f7c08106170 2024-04-23T10:39:15.954 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:39:15.950+0000 7f7c0ccb5640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f7c081010f0 0x7f7c081986a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T10:39:15.954 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:39:15.950+0000 7f7c0ccb5640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f7c081010f0 0x7f7c081986a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.114:50000/0 (socket says 172.21.15.114:50000) 2024-04-23T10:39:15.954 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:39:15.950+0000 7f7c0ccb5640 1 -- 172.21.15.114:0/2010360730 learned_addr learned my addr 172.21.15.114:0/2010360730 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-04-23T10:39:15.954 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:39:15.950+0000 7f7c07fff640 1 --2- 172.21.15.114:0/2010360730 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f7c08106170 0x7f7c08198be0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T10:39:15.954 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:39:15.950+0000 7f7c0d4b6640 1 --2- 172.21.15.114:0/2010360730 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f7c0819cec0 0x7f7c0819d370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T10:39:15.954 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:39:15.950+0000 7f7c0ccb5640 1 -- 172.21.15.114:0/2010360730 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f7c08106170 msgr2=0x7f7c08198be0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T10:39:15.954 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:39:15.950+0000 7f7c0ccb5640 1 --2- 172.21.15.114:0/2010360730 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f7c08106170 0x7f7c08198be0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:39:15.955 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:39:15.950+0000 7f7c0ccb5640 1 -- 172.21.15.114:0/2010360730 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f7c0819cec0 msgr2=0x7f7c0819d370 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T10:39:15.955 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:39:15.950+0000 7f7c0ccb5640 1 --2- 172.21.15.114:0/2010360730 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f7c0819cec0 0x7f7c0819d370 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:39:15.955 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:39:15.950+0000 7f7c0ccb5640 1 -- 172.21.15.114:0/2010360730 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f7c0819d9b0 con 0x7f7c081010f0 2024-04-23T10:39:15.955 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:39:15.950+0000 7f7c0ccb5640 1 --2- 172.21.15.114:0/2010360730 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f7c081010f0 0x7f7c081986a0 secure :-1 s=READY pgs=899 cs=0 l=1 rev1=1 crypto rx=0x7f7bf800e9f0 tx=0x7f7bf800eeb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T10:39:15.955 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:39:15.950+0000 7f7c05ffb640 1 -- 172.21.15.114:0/2010360730 <== mon.0 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7bf800cd80 con 0x7f7c081010f0 2024-04-23T10:39:15.955 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:39:15.950+0000 7f7c0ef19640 1 -- 172.21.15.114:0/2010360730 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f7c0819dca0 con 0x7f7c081010f0 2024-04-23T10:39:15.955 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:39:15.950+0000 7f7c05ffb640 1 -- 172.21.15.114:0/2010360730 <== mon.0 v2:172.21.15.114:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f7bf80108e0 con 0x7f7c081010f0 2024-04-23T10:39:15.955 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:39:15.950+0000 7f7c05ffb640 1 -- 172.21.15.114:0/2010360730 <== mon.0 v2:172.21.15.114:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7bf8018980 con 0x7f7c081010f0 2024-04-23T10:39:15.955 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:39:15.950+0000 7f7c0ef19640 1 -- 172.21.15.114:0/2010360730 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f7c081a5490 con 0x7f7c081010f0 2024-04-23T10:39:15.956 INFO:tasks.workunit.client.0.smithi114.stdout: 1 709933 5.11 MB/sec execute 564 sec latency 14.246 ms 2024-04-23T10:39:15.964 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:39:15.953+0000 7f7c05ffb640 1 -- 172.21.15.114:0/2010360730 <== mon.0 v2:172.21.15.114:3300/0 4 ==== mgrmap(e 26) ==== 138778+0+0 (secure 0 0 0) 0x7f7bf8010430 con 0x7f7c081010f0 2024-04-23T10:39:15.964 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:39:15.953+0000 7f7c11c2f740 1 Processor -- start 2024-04-23T10:39:15.964 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:39:15.953+0000 7f7c11c2f740 1 -- start start 2024-04-23T10:39:15.964 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:39:15.953+0000 7f7c05ffb640 1 --2- 172.21.15.114:0/2010360730 >> [v2:172.21.15.114:6800/1395693466,v1:172.21.15.114:6801/1395693466] conn(0x7f7bf00a3dc0 0x7f7bf00a6280 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T10:39:15.965 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:39:15.954+0000 7f7c11c2f740 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x564004b75620 0x564004b75a00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T10:39:15.965 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:39:15.954+0000 7f7c11c2f740 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x564004b75f40 0x564004b802c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T10:39:15.965 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:39:15.954+0000 7f7c11c2f740 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x564004b80800 0x564004b82bf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T10:39:15.965 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:39:15.954+0000 7f7c11c2f740 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x564004ada970 con 0x564004b80800 2024-04-23T10:39:15.965 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:39:15.954+0000 7f7c11c2f740 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 -- 0x564004a774a0 con 0x564004b75f40 2024-04-23T10:39:15.965 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:39:15.954+0000 7f7c11c2f740 1 -- --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_getmap magic: 0 -- 0x564004ad6930 con 0x564004b75620 2024-04-23T10:39:15.965 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:39:15.954+0000 7f7c05ffb640 1 -- 172.21.15.114:0/2010360730 <== mon.0 v2:172.21.15.114:3300/0 5 ==== osd_map(108..108 src has 1..108) ==== 8629+0+0 (secure 0 0 0) 0x7f7bf80d0b50 con 0x7f7c081010f0 2024-04-23T10:39:15.965 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:39:15.954+0000 7f7c0d4b6640 1 --2- 172.21.15.114:0/2010360730 >> [v2:172.21.15.114:6800/1395693466,v1:172.21.15.114:6801/1395693466] conn(0x7f7bf00a3dc0 0x7f7bf00a6280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T10:39:15.965 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:39:15.954+0000 7f7c07fff640 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x564004b75f40 0x564004b802c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T10:39:15.965 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:39:15.954+0000 7f7c07fff640 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x564004b75f40 0x564004b802c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.119:3300/0 says I am v2:172.21.15.114:34668/0 (socket says 172.21.15.114:34668) 2024-04-23T10:39:15.966 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:39:15.954+0000 7f7c07fff640 1 -- 172.21.15.114:0/2146489936 learned_addr learned my addr 172.21.15.114:0/2146489936 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-04-23T10:39:15.966 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:39:15.954+0000 7f7c0d4b6640 1 --2- 172.21.15.114:0/2146489936 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x564004b80800 0x564004b82bf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T10:39:15.966 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:39:15.954+0000 7f7c0ccb5640 1 --2- 172.21.15.114:0/2146489936 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x564004b75620 0x564004b75a00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T10:39:15.966 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:39:15.954+0000 7f7c07fff640 1 -- 172.21.15.114:0/2146489936 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x564004b75620 msgr2=0x564004b75a00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T10:39:15.966 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:39:15.954+0000 7f7c07fff640 1 --2- 172.21.15.114:0/2146489936 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x564004b75620 0x564004b75a00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:39:15.966 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:39:15.954+0000 7f7c07fff640 1 -- 172.21.15.114:0/2146489936 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x564004b80800 msgr2=0x564004b82bf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T10:39:15.966 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:39:15.954+0000 7f7c07fff640 1 --2- 172.21.15.114:0/2146489936 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x564004b80800 0x564004b82bf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:39:15.966 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:39:15.954+0000 7f7c07fff640 1 -- 172.21.15.114:0/2146489936 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5640047ed210 con 0x564004b75f40 2024-04-23T10:39:15.966 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:39:15.954+0000 7f7c0d4b6640 1 --2- 172.21.15.114:0/2010360730 >> [v2:172.21.15.114:6800/1395693466,v1:172.21.15.114:6801/1395693466] conn(0x7f7bf00a3dc0 0x7f7bf00a6280 secure :-1 s=READY pgs=141 cs=0 l=1 rev1=1 crypto rx=0x7f7c0002f500 tx=0x7f7c0002f490 comp rx=0 tx=0).ready entity=mgr.15291 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T10:39:15.966 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:39:15.954+0000 7f7c07fff640 1 --2- 172.21.15.114:0/2146489936 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x564004b75f40 0x564004b802c0 secure :-1 s=READY pgs=98 cs=0 l=1 rev1=1 crypto rx=0x7f7bfc009c20 tx=0x7f7bfc011810 comp rx=0 tx=0).ready entity=mon.2 client_cookie=2f457f7a3a52fb0d server_cookie=0 in_seq=0 out_seq=0 2024-04-23T10:39:15.967 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:39:15.954+0000 7f7beeffd640 1 -- 172.21.15.114:0/2146489936 <== mon.2 v2:172.21.15.119:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7bfc004050 con 0x564004b75f40 2024-04-23T10:39:15.967 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:39:15.955+0000 7f7beeffd640 1 -- 172.21.15.114:0/2146489936 <== mon.2 v2:172.21.15.119:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f7bfc0054d0 con 0x564004b75f40 2024-04-23T10:39:15.967 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:39:15.955+0000 7f7beeffd640 1 -- 172.21.15.114:0/2146489936 <== mon.2 v2:172.21.15.119:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7bfc004c40 con 0x564004b75f40 2024-04-23T10:39:15.967 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:39:15.955+0000 7f7c11c2f740 1 -- 172.21.15.114:0/2146489936 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x564004b75f40 msgr2=0x564004b802c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T10:39:15.967 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:39:15.955+0000 7f7c11c2f740 1 --2- 172.21.15.114:0/2146489936 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x564004b75f40 0x564004b802c0 secure :-1 s=READY pgs=98 cs=0 l=1 rev1=1 crypto rx=0x7f7bfc009c20 tx=0x7f7bfc011810 comp rx=0 tx=0).stop 2024-04-23T10:39:15.967 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:39:15.955+0000 7f7c11c2f740 1 -- 172.21.15.114:0/2146489936 shutdown_connections 2024-04-23T10:39:15.967 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:39:15.955+0000 7f7c11c2f740 1 --2- 172.21.15.114:0/2146489936 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x564004b80800 0x564004b82bf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:39:15.967 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:39:15.955+0000 7f7c11c2f740 1 --2- 172.21.15.114:0/2146489936 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x564004b75f40 0x564004b802c0 unknown :-1 s=CLOSED pgs=98 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:39:15.967 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:39:15.955+0000 7f7c11c2f740 1 --2- 172.21.15.114:0/2146489936 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x564004b75620 0x564004b75a00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:39:15.967 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:39:15.955+0000 7f7c11c2f740 1 -- 172.21.15.114:0/2146489936 >> 172.21.15.114:0/2146489936 conn(0x564004b71b30 msgr2=0x564004b71f10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T10:39:15.967 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:39:15.955+0000 7f7c11c2f740 1 -- 172.21.15.114:0/2146489936 shutdown_connections 2024-04-23T10:39:15.968 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:39:15.955+0000 7f7c11c2f740 1 -- 172.21.15.114:0/2146489936 wait complete. 2024-04-23T10:39:15.968 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:39:15.956+0000 7f7c11c2f740 1 -- start start 2024-04-23T10:39:15.968 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:39:15.956+0000 7f7c11c2f740 1 Processor -- start 2024-04-23T10:39:15.968 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:39:15.956+0000 7f7c11c2f740 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x564004b75620 0x564004b80180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T10:39:15.968 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:39:15.956+0000 7f7c11c2f740 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x564004b75f40 0x564004b7d470 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T10:39:15.968 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:39:15.956+0000 7f7c11c2f740 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x564004b80800 0x564004b7d9f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T10:39:15.968 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:39:15.956+0000 7f7c11c2f740 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x564004b83430 con 0x564004b75f40 2024-04-23T10:39:15.968 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:39:15.956+0000 7f7c11c2f740 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 -- 0x564004a750b0 con 0x564004b75620 2024-04-23T10:39:15.968 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:39:15.956+0000 7f7c11c2f740 1 -- --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_getmap magic: 0 -- 0x564004b835b0 con 0x564004b80800 2024-04-23T10:39:15.968 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:39:15.956+0000 7f7c11c2f740 10 client.0 _ll_register_callbacks cb 0x564004a62d30 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-23T10:39:15.968 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:39:15.956+0000 7f7c07fff640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x564004b75f40 0x564004b7d470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T10:39:15.969 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:39:15.956+0000 7f7c07fff640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x564004b75f40 0x564004b7d470 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.114:50014/0 (socket says 172.21.15.114:50014) 2024-04-23T10:39:15.969 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:39:15.956+0000 7f7c07fff640 1 -- 172.21.15.114:0/2364812741 learned_addr learned my addr 172.21.15.114:0/2364812741 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-04-23T10:39:15.969 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:39:15.956+0000 7f7c0ccb5640 1 --2- 172.21.15.114:0/2364812741 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x564004b75620 0x564004b80180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T10:39:15.969 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:39:15.956+0000 7f7c0d4b6640 1 --2- 172.21.15.114:0/2364812741 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x564004b80800 0x564004b7d9f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T10:39:15.969 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:39:15.957+0000 7f7c07fff640 1 -- 172.21.15.114:0/2364812741 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x564004b80800 msgr2=0x564004b7d9f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T10:39:15.969 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:39:15.957+0000 7f7c07fff640 1 --2- 172.21.15.114:0/2364812741 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x564004b80800 0x564004b7d9f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:39:15.969 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:39:15.957+0000 7f7c07fff640 1 -- 172.21.15.114:0/2364812741 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x564004b75620 msgr2=0x564004b80180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T10:39:15.969 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:39:15.957+0000 7f7c07fff640 1 --2- 172.21.15.114:0/2364812741 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x564004b75620 0x564004b80180 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:39:15.969 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:39:15.957+0000 7f7c07fff640 1 -- 172.21.15.114:0/2364812741 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5640048b9620 con 0x564004b75f40 2024-04-23T10:39:15.969 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:39:15.957+0000 7f7c0ccb5640 1 --2- 172.21.15.114:0/2364812741 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x564004b75620 0x564004b80180 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-23T10:39:15.970 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:39:15.957+0000 7f7c0d4b6640 1 --2- 172.21.15.114:0/2364812741 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x564004b80800 0x564004b7d9f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-23T10:39:15.970 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:39:15.957+0000 7f7c07fff640 1 --2- 172.21.15.114:0/2364812741 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x564004b75f40 0x564004b7d470 secure :-1 s=READY pgs=900 cs=0 l=1 rev1=1 crypto rx=0x7f7bfc009bf0 tx=0x7f7bfc005680 comp rx=0 tx=0).ready entity=mon.0 client_cookie=98161204419bea59 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T10:39:15.970 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:39:15.957+0000 7f7be7fff640 10 client.0 ms_handle_connect on v2:172.21.15.114:3300/0 2024-04-23T10:39:15.970 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:39:15.957+0000 7f7be7fff640 1 -- 172.21.15.114:0/2364812741 <== mon.0 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7bfc029070 con 0x564004b75f40 2024-04-23T10:39:15.970 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:39:15.957+0000 7f7be7fff640 1 -- 172.21.15.114:0/2364812741 <== mon.0 v2:172.21.15.114:3300/0 2 ==== osd_map(108..108 src has 1..108) ==== 8629+0+0 (secure 0 0 0) 0x7f7bfc003a00 con 0x564004b75f40 2024-04-23T10:39:15.970 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:39:15.957+0000 7f7c11c2f740 1 -- 172.21.15.114:0/2364812741 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f7bfc0054d0 con 0x564004b75f40 2024-04-23T10:39:15.970 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:39:15.957+0000 7f7be7fff640 1 -- 172.21.15.114:0/2364812741 <== mon.0 v2:172.21.15.114:3300/0 3 ==== mon_get_version_reply(handle=1 version=42) ==== 24+0+0 (secure 0 0 0) 0x7f7bfc035050 con 0x564004b75f40 2024-04-23T10:39:15.970 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:39:15.957+0000 7f7c11c2f740 10 client.16977 fetch_fsmap learned FSMap version 42 2024-04-23T10:39:15.970 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:39:15.957+0000 7f7c11c2f740 1 -- 172.21.15.114:0/2364812741 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({config=0+,fsmap=42,monmap=4+}) -- 0x564004b8f380 con 0x564004b75f40 2024-04-23T10:39:15.970 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:39:15.957+0000 7f7be7fff640 1 -- 172.21.15.114:0/2364812741 <== mon.0 v2:172.21.15.114:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f7bfc0054d0 con 0x564004b75f40 2024-04-23T10:39:15.971 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:39:15.957+0000 7f7be7fff640 1 -- 172.21.15.114:0/2364812741 <== mon.0 v2:172.21.15.114:3300/0 5 ==== fsmap(e 42) ==== 8338+0+0 (secure 0 0 0) 0x7f7bfc0051a0 con 0x564004b75f40 2024-04-23T10:39:15.971 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:39:15.957+0000 7f7c11c2f740 10 client.16977 fetch_fsmap finished waiting for FSMap version 42 2024-04-23T10:39:15.971 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:39:15.957+0000 7f7c11c2f740 10 client.16977 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-04-23T10:39:15.971 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:39:15.957+0000 7f7c11c2f740 20 client.16977 populate_metadata read hostname 'smithi114' 2024-04-23T10:39:15.971 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:39:15.957+0000 7f7c11c2f740 1 --2- 172.21.15.114:0/2364812741 >> [v2:172.21.15.119:6838/937297340,v1:172.21.15.119:6839/937297340] conn(0x564004b90790 0x564004b92b80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T10:39:15.971 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:39:15.957+0000 7f7c11c2f740 4 client.16977 mds_command: new command op to 24545 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-23T10:39:15.971 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:39:15.957+0000 7f7c11c2f740 1 -- 172.21.15.114:0/2364812741 --> [v2:172.21.15.119:6838/937297340,v1:172.21.15.119:6839/937297340] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x564004ae8390 con 0x564004b90790 2024-04-23T10:39:15.971 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:39:15.958+0000 7f7c0ccb5640 1 --2- 172.21.15.114:0/2364812741 >> [v2:172.21.15.119:6838/937297340,v1:172.21.15.119:6839/937297340] conn(0x564004b90790 0x564004b92b80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T10:39:15.971 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:39:15.958+0000 7f7be7fff640 10 client.16977 ms_handle_connect on v2:172.21.15.119:6838/937297340 2024-04-23T10:39:15.971 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:39:15.958+0000 7f7c0ccb5640 1 --2- 172.21.15.114:0/2364812741 >> [v2:172.21.15.119:6838/937297340,v1:172.21.15.119:6839/937297340] conn(0x564004b90790 0x564004b92b80 crc :-1 s=READY pgs=174 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=99b8eb260fcad435 server_cookie=6c4a791d6982d25a in_seq=0 out_seq=0 2024-04-23T10:39:15.971 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:39:15.959+0000 7f7be7fff640 1 -- 172.21.15.114:0/2364812741 <== mds.0 v2:172.21.15.119:6838/937297340 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x564004ae8390 con 0x564004b90790 2024-04-23T10:39:15.972 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:39:15.959+0000 7f7be7fff640 10 client.16977 handle_command_reply: tid=0 2024-04-23T10:39:15.972 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:39:15.959+0000 7f7c11c2f740 1 client.16977 shutdown 2024-04-23T10:39:15.972 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:39:15.959+0000 7f7c11c2f740 1 -- 172.21.15.114:0/2364812741 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x564004b75f40 msgr2=0x564004b7d470 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T10:39:15.972 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:39:15.959+0000 7f7c11c2f740 1 --2- 172.21.15.114:0/2364812741 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x564004b75f40 0x564004b7d470 secure :-1 s=READY pgs=900 cs=0 l=1 rev1=1 crypto rx=0x7f7bfc009bf0 tx=0x7f7bfc005680 comp rx=0 tx=0).stop 2024-04-23T10:39:15.972 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:39:15.959+0000 7f7c11c2f740 1 -- 172.21.15.114:0/2364812741 shutdown_connections 2024-04-23T10:39:15.972 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:39:15.959+0000 7f7c11c2f740 1 --2- 172.21.15.114:0/2364812741 >> [v2:172.21.15.119:6838/937297340,v1:172.21.15.119:6839/937297340] conn(0x564004b90790 0x564004b92b80 crc :-1 s=READY pgs=174 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:39:15.972 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:39:15.959+0000 7f7c11c2f740 1 --2- 172.21.15.114:0/2364812741 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x564004b80800 0x564004b7d9f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:39:15.972 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:39:15.959+0000 7f7c11c2f740 1 --2- 172.21.15.114:0/2364812741 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x564004b75f40 0x564004b7d470 unknown :-1 s=CLOSED pgs=900 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:39:15.972 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:39:15.959+0000 7f7c11c2f740 1 --2- 172.21.15.114:0/2364812741 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x564004b75620 0x564004b80180 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:39:15.972 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:39:15.959+0000 7f7c11c2f740 1 -- 172.21.15.114:0/2364812741 >> 172.21.15.114:0/2364812741 conn(0x564004b71b30 msgr2=0x564004b77860 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T10:39:15.972 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:39:15.959+0000 7f7be7fff640 0 client.16977 ms_handle_reset on v2:172.21.15.119:6838/937297340 2024-04-23T10:39:15.973 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:39:15.959+0000 7f7c11c2f740 1 -- 172.21.15.114:0/2364812741 shutdown_connections 2024-04-23T10:39:15.973 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:39:15.959+0000 7f7c11c2f740 1 -- 172.21.15.114:0/2364812741 wait complete. 2024-04-23T10:39:15.973 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:39:15.959+0000 7f7c11c2f740 20 client.16977 trim_cache size 0 max 16384 2024-04-23T10:39:15.981 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:39:15.975+0000 7f7c11c2f740 1 Processor -- start 2024-04-23T10:39:15.981 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:39:15.975+0000 7f7c11c2f740 1 -- start start 2024-04-23T10:39:15.981 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:39:15.975+0000 7f7c11c2f740 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x564004b75620 0x564004b8c2f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T10:39:15.981 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:39:15.975+0000 7f7c11c2f740 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x564004b80800 0x564004b70b30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T10:39:15.981 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:39:15.975+0000 7f7c11c2f740 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x564004b71070 0x564004b98340 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T10:39:15.981 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:39:15.975+0000 7f7c11c2f740 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x7f7c081b0530 con 0x564004b71070 2024-04-23T10:39:15.982 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:39:15.975+0000 7f7c11c2f740 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 -- 0x7f7c081ade00 con 0x564004b80800 2024-04-23T10:39:15.982 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:39:15.975+0000 7f7c11c2f740 1 -- --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_getmap magic: 0 -- 0x7f7c081abc70 con 0x564004b75620 2024-04-23T10:39:15.982 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:39:15.976+0000 7f7c0d4b6640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x564004b71070 0x564004b98340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T10:39:15.982 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:39:15.976+0000 7f7c0ccb5640 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x564004b75620 0x564004b8c2f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T10:39:15.982 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:39:15.976+0000 7f7c07fff640 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x564004b80800 0x564004b70b30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T10:39:15.982 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:39:15.976+0000 7f7c07fff640 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x564004b80800 0x564004b70b30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.119:3300/0 says I am v2:172.21.15.114:34692/0 (socket says 172.21.15.114:34692) 2024-04-23T10:39:15.982 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:39:15.976+0000 7f7c07fff640 1 -- 172.21.15.114:0/341943890 learned_addr learned my addr 172.21.15.114:0/341943890 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-04-23T10:39:15.982 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:39:15.976+0000 7f7c0ccb5640 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x564004b75620 0x564004b8c2f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.204:3300/0 says I am v2:172.21.15.114:43036/0 (socket says 172.21.15.114:43036) 2024-04-23T10:39:15.982 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:39:15.976+0000 7f7c07fff640 1 -- 172.21.15.114:0/341943890 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x564004b75620 msgr2=0x564004b8c2f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T10:39:15.982 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:39:15.976+0000 7f7c07fff640 1 --2- 172.21.15.114:0/341943890 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x564004b75620 0x564004b8c2f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:39:15.982 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:39:15.976+0000 7f7c07fff640 1 -- 172.21.15.114:0/341943890 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x564004b71070 msgr2=0x564004b98340 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T10:39:15.983 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:39:15.976+0000 7f7c07fff640 1 --2- 172.21.15.114:0/341943890 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x564004b71070 0x564004b98340 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:39:15.983 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:39:15.976+0000 7f7c07fff640 1 -- 172.21.15.114:0/341943890 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x564004b7c420 con 0x564004b80800 2024-04-23T10:39:15.983 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:39:15.976+0000 7f7c07fff640 1 --2- 172.21.15.114:0/341943890 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x564004b80800 0x564004b70b30 secure :-1 s=READY pgs=99 cs=0 l=1 rev1=1 crypto rx=0x7f7bfc023640 tx=0x7f7bfc009970 comp rx=0 tx=0).ready entity=mon.2 client_cookie=ca15df279b8717b2 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T10:39:15.983 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:39:15.976+0000 7f7bee7fc640 1 -- 172.21.15.114:0/341943890 <== mon.2 v2:172.21.15.119:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7bfc029050 con 0x564004b80800 2024-04-23T10:39:15.983 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:39:15.976+0000 7f7bee7fc640 1 -- 172.21.15.114:0/341943890 <== mon.2 v2:172.21.15.119:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f7bfc0238c0 con 0x564004b80800 2024-04-23T10:39:15.983 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:39:15.976+0000 7f7bee7fc640 1 -- 172.21.15.114:0/341943890 <== mon.2 v2:172.21.15.119:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7bfc003a40 con 0x564004b80800 2024-04-23T10:39:15.983 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:39:15.977+0000 7f7c11c2f740 1 -- 172.21.15.114:0/341943890 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x564004b80800 msgr2=0x564004b70b30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T10:39:15.983 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:39:15.977+0000 7f7c11c2f740 1 --2- 172.21.15.114:0/341943890 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x564004b80800 0x564004b70b30 secure :-1 s=READY pgs=99 cs=0 l=1 rev1=1 crypto rx=0x7f7bfc023640 tx=0x7f7bfc009970 comp rx=0 tx=0).stop 2024-04-23T10:39:15.983 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:39:15.977+0000 7f7c11c2f740 1 -- 172.21.15.114:0/341943890 shutdown_connections 2024-04-23T10:39:15.983 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:39:15.977+0000 7f7c11c2f740 1 --2- 172.21.15.114:0/341943890 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x564004b71070 0x564004b98340 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:39:15.983 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:39:15.977+0000 7f7c11c2f740 1 --2- 172.21.15.114:0/341943890 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x564004b80800 0x564004b70b30 unknown :-1 s=CLOSED pgs=99 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:39:15.983 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:39:15.977+0000 7f7c11c2f740 1 --2- 172.21.15.114:0/341943890 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x564004b75620 0x564004b8c2f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:39:15.984 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:39:15.977+0000 7f7c11c2f740 1 -- 172.21.15.114:0/341943890 >> 172.21.15.114:0/341943890 conn(0x564004b71b30 msgr2=0x564004b70420 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T10:39:15.984 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:39:15.977+0000 7f7c11c2f740 1 -- 172.21.15.114:0/341943890 shutdown_connections 2024-04-23T10:39:15.984 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:39:15.977+0000 7f7c11c2f740 1 -- 172.21.15.114:0/341943890 wait complete. 2024-04-23T10:39:15.984 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:39:15.977+0000 7f7c11c2f740 1 -- start start 2024-04-23T10:39:15.984 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:39:15.977+0000 7f7c11c2f740 1 Processor -- start 2024-04-23T10:39:15.984 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:39:15.978+0000 7f7c11c2f740 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x564004b71070 0x564004b88c30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T10:39:15.984 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:39:15.978+0000 7f7c11c2f740 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x564004b75620 0x564004b86010 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T10:39:15.984 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:39:15.978+0000 7f7c11c2f740 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x564004b80800 0x564004b86570 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T10:39:15.984 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:39:15.978+0000 7f7c11c2f740 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x7f7c081ac2c0 con 0x564004b71070 2024-04-23T10:39:15.984 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:39:15.978+0000 7f7c11c2f740 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 -- 0x7f7c081ad730 con 0x564004b80800 2024-04-23T10:39:15.984 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:39:15.978+0000 7f7c11c2f740 1 -- --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_getmap magic: 0 -- 0x7f7c081ac990 con 0x564004b75620 2024-04-23T10:39:15.984 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:39:15.978+0000 7f7c11c2f740 10 client.0 _ll_register_callbacks cb 0x7f7bfc009a70 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-23T10:39:15.985 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:39:15.978+0000 7f7c0ccb5640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x564004b71070 0x564004b88c30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T10:39:15.985 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:39:15.978+0000 7f7c0ccb5640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x564004b71070 0x564004b88c30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.114:50022/0 (socket says 172.21.15.114:50022) 2024-04-23T10:39:15.985 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:39:15.978+0000 7f7c0ccb5640 1 -- 172.21.15.114:0/4212148502 learned_addr learned my addr 172.21.15.114:0/4212148502 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-04-23T10:39:15.985 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:39:15.978+0000 7f7c0d4b6640 1 --2- 172.21.15.114:0/4212148502 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x564004b80800 0x564004b86570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T10:39:15.985 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:39:15.978+0000 7f7c07fff640 1 --2- 172.21.15.114:0/4212148502 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x564004b75620 0x564004b86010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T10:39:15.985 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:39:15.978+0000 7f7c0ccb5640 1 -- 172.21.15.114:0/4212148502 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x564004b75620 msgr2=0x564004b86010 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T10:39:15.985 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:39:15.978+0000 7f7c0ccb5640 1 --2- 172.21.15.114:0/4212148502 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x564004b75620 0x564004b86010 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:39:15.985 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:39:15.978+0000 7f7c0ccb5640 1 -- 172.21.15.114:0/4212148502 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x564004b80800 msgr2=0x564004b86570 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T10:39:15.985 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:39:15.978+0000 7f7c0ccb5640 1 --2- 172.21.15.114:0/4212148502 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x564004b80800 0x564004b86570 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:39:15.985 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:39:15.978+0000 7f7c0ccb5640 1 -- 172.21.15.114:0/4212148502 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({osdmap=0}) -- 0x564004b75d70 con 0x564004b71070 2024-04-23T10:39:15.985 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:39:15.978+0000 7f7c0ccb5640 1 --2- 172.21.15.114:0/4212148502 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x564004b71070 0x564004b88c30 secure :-1 s=READY pgs=901 cs=0 l=1 rev1=1 crypto rx=0x7f7bf809c830 tx=0x7f7bf8018420 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b37131af4b27f07c server_cookie=0 in_seq=0 out_seq=0 2024-04-23T10:39:15.986 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:39:15.978+0000 7f7be7fff640 10 client.0 ms_handle_connect on v2:172.21.15.114:3300/0 2024-04-23T10:39:15.986 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:39:15.978+0000 7f7be7fff640 1 -- 172.21.15.114:0/4212148502 <== mon.0 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7bf80bd0b0 con 0x564004b71070 2024-04-23T10:39:15.986 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:39:15.979+0000 7f7be7fff640 1 -- 172.21.15.114:0/4212148502 <== mon.0 v2:172.21.15.114:3300/0 2 ==== osd_map(108..108 src has 1..108) ==== 8629+0+0 (secure 0 0 0) 0x7f7bf80ce4b0 con 0x564004b71070 2024-04-23T10:39:15.986 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:39:15.979+0000 7f7c11c2f740 1 -- 172.21.15.114:0/4212148502 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f7bfc0238c0 con 0x564004b71070 2024-04-23T10:39:15.986 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:39:15.979+0000 7f7be7fff640 1 -- 172.21.15.114:0/4212148502 <== mon.0 v2:172.21.15.114:3300/0 3 ==== mon_get_version_reply(handle=1 version=42) ==== 24+0+0 (secure 0 0 0) 0x7f7bf80ce6b0 con 0x564004b71070 2024-04-23T10:39:15.986 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:39:15.979+0000 7f7c11c2f740 10 client.16983 fetch_fsmap learned FSMap version 42 2024-04-23T10:39:15.986 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:39:15.979+0000 7f7c11c2f740 1 -- 172.21.15.114:0/4212148502 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({config=0+,fsmap=42,monmap=4+}) -- 0x564004bf6ae0 con 0x564004b71070 2024-04-23T10:39:15.986 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:39:15.979+0000 7f7be7fff640 1 -- 172.21.15.114:0/4212148502 <== mon.0 v2:172.21.15.114:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f7bfc0238c0 con 0x564004b71070 2024-04-23T10:39:15.986 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:39:15.979+0000 7f7be7fff640 1 -- 172.21.15.114:0/4212148502 <== mon.0 v2:172.21.15.114:3300/0 5 ==== fsmap(e 42) ==== 8338+0+0 (secure 0 0 0) 0x7f7bf8086bd0 con 0x564004b71070 2024-04-23T10:39:15.986 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:39:15.979+0000 7f7c11c2f740 10 client.16983 fetch_fsmap finished waiting for FSMap version 42 2024-04-23T10:39:15.986 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:39:15.979+0000 7f7c11c2f740 10 client.16983 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-04-23T10:39:15.986 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:39:15.979+0000 7f7c11c2f740 20 client.16983 populate_metadata read hostname 'smithi114' 2024-04-23T10:39:15.987 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:39:15.979+0000 7f7c11c2f740 1 --2- 172.21.15.114:0/4212148502 >> [v2:172.21.15.119:6838/937297340,v1:172.21.15.119:6839/937297340] conn(0x564004bf7200 0x564004bf9610 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T10:39:15.987 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:39:15.979+0000 7f7c11c2f740 4 client.16983 mds_command: new command op to 24545 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-04-23T10:39:15.987 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:39:15.979+0000 7f7c11c2f740 1 -- 172.21.15.114:0/4212148502 --> [v2:172.21.15.119:6838/937297340,v1:172.21.15.119:6839/937297340] -- command(tid 0: {"prefix": "flush journal"}) -- 0x564004ae2200 con 0x564004bf7200 2024-04-23T10:39:15.987 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:39:15.979+0000 7f7c07fff640 1 --2- 172.21.15.114:0/4212148502 >> [v2:172.21.15.119:6838/937297340,v1:172.21.15.119:6839/937297340] conn(0x564004bf7200 0x564004bf9610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T10:39:15.987 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:39:15.979+0000 7f7be7fff640 10 client.16983 ms_handle_connect on v2:172.21.15.119:6838/937297340 2024-04-23T10:39:15.987 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:39:15.980+0000 7f7c07fff640 1 --2- 172.21.15.114:0/4212148502 >> [v2:172.21.15.119:6838/937297340,v1:172.21.15.119:6839/937297340] conn(0x564004bf7200 0x564004bf9610 crc :-1 s=READY pgs=175 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=658acb7ee0423e77 server_cookie=bc356ba7bd8ae4cf in_seq=0 out_seq=0 2024-04-23T10:39:16.022 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:39:16.022+0000 7f7be7fff640 1 -- 172.21.15.114:0/4212148502 <== mds.0 v2:172.21.15.119:6838/937297340 1 ==== command_reply(tid 0: 0 ) ==== 8+0+44 (crc 0 0 0) 0x564004ae2200 con 0x564004bf7200 2024-04-23T10:39:16.022 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:39:16.022+0000 7f7be7fff640 10 client.16983 handle_command_reply: tid=0 2024-04-23T10:39:16.022 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:39:16.022+0000 7f7c11c2f740 1 client.16983 shutdown 2024-04-23T10:39:16.022 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:39:16.022+0000 7f7c11c2f740 1 -- 172.21.15.114:0/4212148502 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x564004b71070 msgr2=0x564004b88c30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T10:39:16.022 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:39:16.022+0000 7f7c11c2f740 1 --2- 172.21.15.114:0/4212148502 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x564004b71070 0x564004b88c30 secure :-1 s=READY pgs=901 cs=0 l=1 rev1=1 crypto rx=0x7f7bf809c830 tx=0x7f7bf8018420 comp rx=0 tx=0).stop 2024-04-23T10:39:16.022 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:39:16.022+0000 7f7c11c2f740 1 -- 172.21.15.114:0/4212148502 shutdown_connections 2024-04-23T10:39:16.022 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:39:16.022+0000 7f7c11c2f740 1 --2- 172.21.15.114:0/4212148502 >> [v2:172.21.15.119:6838/937297340,v1:172.21.15.119:6839/937297340] conn(0x564004bf7200 0x564004bf9610 crc :-1 s=READY pgs=175 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:39:16.023 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:39:16.022+0000 7f7c11c2f740 1 --2- 172.21.15.114:0/4212148502 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x564004b80800 0x564004b86570 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:39:16.023 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:39:16.022+0000 7f7be7fff640 0 client.16983 ms_handle_reset on v2:172.21.15.119:6838/937297340 2024-04-23T10:39:16.023 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:39:16.022+0000 7f7c11c2f740 1 --2- 172.21.15.114:0/4212148502 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x564004b75620 0x564004b86010 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:39:16.023 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:39:16.022+0000 7f7c11c2f740 1 --2- 172.21.15.114:0/4212148502 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x564004b71070 0x564004b88c30 unknown :-1 s=CLOSED pgs=901 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:39:16.023 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:39:16.022+0000 7f7c11c2f740 1 -- 172.21.15.114:0/4212148502 >> 172.21.15.114:0/4212148502 conn(0x564004b71b30 msgr2=0x564004b8b2e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T10:39:16.023 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:39:16.022+0000 7f7c11c2f740 1 -- 172.21.15.114:0/4212148502 shutdown_connections 2024-04-23T10:39:16.023 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:39:16.022+0000 7f7c11c2f740 1 -- 172.21.15.114:0/4212148502 wait complete. 2024-04-23T10:39:16.023 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:39:16.022+0000 7f7c11c2f740 20 client.16983 trim_cache size 0 max 16384 2024-04-23T10:39:16.024 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stdout:{ 2024-04-23T10:39:16.024 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stdout: "message": "", 2024-04-23T10:39:16.024 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stdout: "return_code": 0 2024-04-23T10:39:16.024 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stdout:} 2024-04-23T10:39:16.024 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:39:16.022+0000 7f7c0ef19640 1 -- 172.21.15.114:0/2010360730 >> [v2:172.21.15.114:6800/1395693466,v1:172.21.15.114:6801/1395693466] conn(0x7f7bf00a3dc0 msgr2=0x7f7bf00a6280 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T10:39:16.025 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:39:16.022+0000 7f7c0ef19640 1 --2- 172.21.15.114:0/2010360730 >> [v2:172.21.15.114:6800/1395693466,v1:172.21.15.114:6801/1395693466] conn(0x7f7bf00a3dc0 0x7f7bf00a6280 secure :-1 s=READY pgs=141 cs=0 l=1 rev1=1 crypto rx=0x7f7c0002f500 tx=0x7f7c0002f490 comp rx=0 tx=0).stop 2024-04-23T10:39:16.025 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:39:16.022+0000 7f7c0ef19640 1 -- 172.21.15.114:0/2010360730 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f7c081010f0 msgr2=0x7f7c081986a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T10:39:16.025 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:39:16.022+0000 7f7c0ef19640 1 --2- 172.21.15.114:0/2010360730 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f7c081010f0 0x7f7c081986a0 secure :-1 s=READY pgs=899 cs=0 l=1 rev1=1 crypto rx=0x7f7bf800e9f0 tx=0x7f7bf800eeb0 comp rx=0 tx=0).stop 2024-04-23T10:39:16.025 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:39:16.022+0000 7f7c0ef19640 1 -- 172.21.15.114:0/2010360730 shutdown_connections 2024-04-23T10:39:16.025 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:39:16.022+0000 7f7c0ef19640 1 --2- 172.21.15.114:0/2010360730 >> [v2:172.21.15.114:6800/1395693466,v1:172.21.15.114:6801/1395693466] conn(0x7f7bf00a3dc0 0x7f7bf00a6280 unknown :-1 s=CLOSED pgs=141 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:39:16.025 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:39:16.022+0000 7f7c0ef19640 1 --2- 172.21.15.114:0/2010360730 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f7c0819cec0 0x7f7c0819d370 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:39:16.025 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:39:16.022+0000 7f7c0ef19640 1 --2- 172.21.15.114:0/2010360730 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f7c08106170 0x7f7c08198be0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:39:16.025 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:39:16.022+0000 7f7c0ef19640 1 --2- 172.21.15.114:0/2010360730 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f7c081010f0 0x7f7c081986a0 unknown :-1 s=CLOSED pgs=899 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:39:16.025 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:39:16.022+0000 7f7c0ef19640 1 -- 172.21.15.114:0/2010360730 >> 172.21.15.114:0/2010360730 conn(0x7f7c08077fd0 msgr2=0x7f7c080698e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T10:39:16.025 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:39:16.022+0000 7f7c0ef19640 1 -- 172.21.15.114:0/2010360730 shutdown_connections 2024-04-23T10:39:16.026 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:39:16.022+0000 7f7c0ef19640 1 -- 172.21.15.114:0/2010360730 wait complete. 2024-04-23T10:39:16.254 INFO:journalctl@ceph.mon.a.smithi114.stdout:Apr 23 10:39:16 smithi114 ceph-mon[29255]: pgmap v398: 97 pgs: 97 active+clean; 203 MiB data, 2.7 GiB used, 1.0 TiB / 1.0 TiB avail; 198 KiB/s rd, 130 KiB/s wr, 27 op/s 2024-04-23T10:39:16.328 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 23 10:39:16 smithi204 ceph-mon[32649]: pgmap v398: 97 pgs: 97 active+clean; 203 MiB data, 2.7 GiB used, 1.0 TiB / 1.0 TiB avail; 198 KiB/s rd, 130 KiB/s wr, 27 op/s 2024-04-23T10:39:16.492 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 23 10:39:16 smithi119 ceph-mon[32608]: pgmap v398: 97 pgs: 97 active+clean; 203 MiB data, 2.7 GiB used, 1.0 TiB / 1.0 TiB avail; 198 KiB/s rd, 130 KiB/s wr, 27 op/s 2024-04-23T10:39:16.951 INFO:tasks.workunit.client.0.smithi114.stdout: 1 710290 5.11 MB/sec execute 565 sec latency 803.001 ms 2024-04-23T10:39:17.951 INFO:tasks.workunit.client.0.smithi114.stdout: 1 710290 5.10 MB/sec execute 566 sec latency 1803.086 ms 2024-04-23T10:39:18.328 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 23 10:39:18 smithi204 ceph-mon[32649]: pgmap v399: 97 pgs: 97 active+clean; 202 MiB data, 2.7 GiB used, 1.0 TiB / 1.0 TiB avail; 628 KiB/s rd, 694 KiB/s wr, 107 op/s 2024-04-23T10:39:18.492 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 23 10:39:18 smithi119 ceph-mon[32608]: pgmap v399: 97 pgs: 97 active+clean; 202 MiB data, 2.7 GiB used, 1.0 TiB / 1.0 TiB avail; 628 KiB/s rd, 694 KiB/s wr, 107 op/s 2024-04-23T10:39:18.504 INFO:journalctl@ceph.mon.a.smithi114.stdout:Apr 23 10:39:18 smithi114 ceph-mon[29255]: pgmap v399: 97 pgs: 97 active+clean; 202 MiB data, 2.7 GiB used, 1.0 TiB / 1.0 TiB avail; 628 KiB/s rd, 694 KiB/s wr, 107 op/s 2024-04-23T10:39:18.951 INFO:tasks.workunit.client.0.smithi114.stdout: 1 710290 5.09 MB/sec execute 567 sec latency 2803.190 ms 2024-04-23T10:39:19.951 INFO:tasks.workunit.client.0.smithi114.stdout: 1 710357 5.08 MB/sec execute 568 sec latency 3043.345 ms 2024-04-23T10:39:20.328 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 23 10:39:20 smithi204 ceph-mon[32649]: pgmap v400: 97 pgs: 97 active+clean; 202 MiB data, 2.7 GiB used, 1.0 TiB / 1.0 TiB avail; 530 KiB/s rd, 586 KiB/s wr, 90 op/s 2024-04-23T10:39:20.492 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 23 10:39:20 smithi119 ceph-mon[32608]: pgmap v400: 97 pgs: 97 active+clean; 202 MiB data, 2.7 GiB used, 1.0 TiB / 1.0 TiB avail; 530 KiB/s rd, 586 KiB/s wr, 90 op/s 2024-04-23T10:39:20.504 INFO:journalctl@ceph.mon.a.smithi114.stdout:Apr 23 10:39:20 smithi114 ceph-mon[29255]: pgmap v400: 97 pgs: 97 active+clean; 202 MiB data, 2.7 GiB used, 1.0 TiB / 1.0 TiB avail; 530 KiB/s rd, 586 KiB/s wr, 90 op/s 2024-04-23T10:39:20.951 INFO:tasks.workunit.client.0.smithi114.stdout: 1 710357 5.07 MB/sec execute 569 sec latency 1642.091 ms 2024-04-23T10:39:21.951 INFO:tasks.workunit.client.0.smithi114.stdout: 1 710357 5.07 MB/sec execute 570 sec latency 2642.208 ms 2024-04-23T10:39:22.306 INFO:journalctl@ceph.mon.a.smithi114.stdout:Apr 23 10:39:22 smithi114 ceph-mon[29255]: pgmap v401: 97 pgs: 97 active+clean; 202 MiB data, 2.7 GiB used, 1.0 TiB / 1.0 TiB avail; 502 KiB/s rd, 555 KiB/s wr, 86 op/s 2024-04-23T10:39:22.327 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 23 10:39:22 smithi204 ceph-mon[32649]: pgmap v401: 97 pgs: 97 active+clean; 202 MiB data, 2.7 GiB used, 1.0 TiB / 1.0 TiB avail; 502 KiB/s rd, 555 KiB/s wr, 86 op/s 2024-04-23T10:39:22.491 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 23 10:39:22 smithi119 ceph-mon[32608]: pgmap v401: 97 pgs: 97 active+clean; 202 MiB data, 2.7 GiB used, 1.0 TiB / 1.0 TiB avail; 502 KiB/s rd, 555 KiB/s wr, 86 op/s 2024-04-23T10:39:22.951 INFO:tasks.workunit.client.0.smithi114.stdout: 1 710357 5.06 MB/sec execute 571 sec latency 3642.307 ms 2024-04-23T10:39:23.951 INFO:tasks.workunit.client.0.smithi114.stdout: 1 710357 5.05 MB/sec execute 572 sec latency 4642.425 ms 2024-04-23T10:39:24.327 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 23 10:39:24 smithi204 ceph-mon[32649]: pgmap v402: 97 pgs: 97 active+clean; 205 MiB data, 2.8 GiB used, 1.0 TiB / 1.0 TiB avail; 419 KiB/s rd, 1.0 MiB/s wr, 92 op/s 2024-04-23T10:39:24.491 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 23 10:39:24 smithi119 ceph-mon[32608]: pgmap v402: 97 pgs: 97 active+clean; 205 MiB data, 2.8 GiB used, 1.0 TiB / 1.0 TiB avail; 419 KiB/s rd, 1.0 MiB/s wr, 92 op/s 2024-04-23T10:39:24.504 INFO:journalctl@ceph.mon.a.smithi114.stdout:Apr 23 10:39:24 smithi114 ceph-mon[29255]: pgmap v402: 97 pgs: 97 active+clean; 205 MiB data, 2.8 GiB used, 1.0 TiB / 1.0 TiB avail; 419 KiB/s rd, 1.0 MiB/s wr, 92 op/s 2024-04-23T10:39:24.951 INFO:tasks.workunit.client.0.smithi114.stdout: 1 711107 5.04 MB/sec execute 573 sec latency 4876.720 ms 2024-04-23T10:39:25.952 INFO:tasks.workunit.client.0.smithi114.stdout: 1 713583 5.06 MB/sec execute 574 sec latency 14.756 ms 2024-04-23T10:39:26.327 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 23 10:39:26 smithi204 ceph-mon[32649]: pgmap v403: 97 pgs: 97 active+clean; 205 MiB data, 2.8 GiB used, 1.0 TiB / 1.0 TiB avail; 381 KiB/s rd, 937 KiB/s wr, 84 op/s 2024-04-23T10:39:26.491 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 23 10:39:26 smithi119 ceph-mon[32608]: pgmap v403: 97 pgs: 97 active+clean; 205 MiB data, 2.8 GiB used, 1.0 TiB / 1.0 TiB avail; 381 KiB/s rd, 937 KiB/s wr, 84 op/s 2024-04-23T10:39:26.504 INFO:journalctl@ceph.mon.a.smithi114.stdout:Apr 23 10:39:26 smithi114 ceph-mon[29255]: pgmap v403: 97 pgs: 97 active+clean; 205 MiB data, 2.8 GiB used, 1.0 TiB / 1.0 TiB avail; 381 KiB/s rd, 937 KiB/s wr, 84 op/s 2024-04-23T10:39:26.951 INFO:tasks.workunit.client.0.smithi114.stdout: 1 713837 5.05 MB/sec execute 575 sec latency 893.197 ms 2024-04-23T10:39:27.951 INFO:tasks.workunit.client.0.smithi114.stdout: 1 713837 5.05 MB/sec execute 576 sec latency 1893.237 ms 2024-04-23T10:39:28.328 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 23 10:39:28 smithi204 ceph-mon[32649]: pgmap v404: 97 pgs: 97 active+clean; 208 MiB data, 2.8 GiB used, 1.0 TiB / 1.0 TiB avail; 567 KiB/s rd, 1.2 MiB/s wr, 115 op/s 2024-04-23T10:39:28.491 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 23 10:39:28 smithi119 ceph-mon[32608]: pgmap v404: 97 pgs: 97 active+clean; 208 MiB data, 2.8 GiB used, 1.0 TiB / 1.0 TiB avail; 567 KiB/s rd, 1.2 MiB/s wr, 115 op/s 2024-04-23T10:39:28.504 INFO:journalctl@ceph.mon.a.smithi114.stdout:Apr 23 10:39:28 smithi114 ceph-mon[29255]: pgmap v404: 97 pgs: 97 active+clean; 208 MiB data, 2.8 GiB used, 1.0 TiB / 1.0 TiB avail; 567 KiB/s rd, 1.2 MiB/s wr, 115 op/s 2024-04-23T10:39:28.952 INFO:tasks.workunit.client.0.smithi114.stdout: 1 713837 5.04 MB/sec execute 577 sec latency 2893.363 ms 2024-04-23T10:39:29.119 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:39:29.117+0000 7f19ea631640 1 -- 172.21.15.114:0/1359964033 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f19e410c560 msgr2=0x7f19e410e9e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T10:39:29.119 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:39:29.117+0000 7f19ea631640 1 --2- 172.21.15.114:0/1359964033 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f19e410c560 0x7f19e410e9e0 secure :-1 s=READY pgs=902 cs=0 l=1 rev1=1 crypto rx=0x7f19dc009ca0 tx=0x7f19dc02a950 comp rx=0 tx=0).stop 2024-04-23T10:39:29.119 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:39:29.117+0000 7f19ea631640 1 -- 172.21.15.114:0/1359964033 shutdown_connections 2024-04-23T10:39:29.119 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:39:29.117+0000 7f19ea631640 1 --2- 172.21.15.114:0/1359964033 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f19e410c560 0x7f19e410e9e0 unknown :-1 s=CLOSED pgs=902 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:39:29.119 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:39:29.117+0000 7f19ea631640 1 --2- 172.21.15.114:0/1359964033 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f19e4103b40 0x7f19e410c020 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:39:29.119 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:39:29.117+0000 7f19ea631640 1 --2- 172.21.15.114:0/1359964033 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f19e4103220 0x7f19e4103600 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:39:29.119 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:39:29.117+0000 7f19ea631640 1 -- 172.21.15.114:0/1359964033 >> 172.21.15.114:0/1359964033 conn(0x7f19e4078060 msgr2=0x7f19e4078470 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T10:39:29.119 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:39:29.117+0000 7f19ea631640 1 -- 172.21.15.114:0/1359964033 shutdown_connections 2024-04-23T10:39:29.119 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:39:29.117+0000 7f19ea631640 1 -- 172.21.15.114:0/1359964033 wait complete. 2024-04-23T10:39:29.120 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:39:29.118+0000 7f19ea631640 1 Processor -- start 2024-04-23T10:39:29.120 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:39:29.118+0000 7f19ea631640 1 -- start start 2024-04-23T10:39:29.120 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:39:29.118+0000 7f19ea631640 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f19e4103220 0x7f19e41985f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T10:39:29.120 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:39:29.118+0000 7f19ea631640 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f19e4103b40 0x7f19e4198b30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T10:39:29.120 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:39:29.118+0000 7f19ea631640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f19e410c560 0x7f19e419ceb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T10:39:29.120 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:39:29.118+0000 7f19ea631640 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x7f19e41121d0 con 0x7f19e410c560 2024-04-23T10:39:29.120 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:39:29.118+0000 7f19ea631640 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 -- 0x7f19e4112050 con 0x7f19e4103b40 2024-04-23T10:39:29.120 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:39:29.118+0000 7f19ea631640 1 -- --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_getmap magic: 0 -- 0x7f19e4112350 con 0x7f19e4103220 2024-04-23T10:39:29.120 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:39:29.118+0000 7f19e8bce640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f19e410c560 0x7f19e419ceb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T10:39:29.120 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:39:29.118+0000 7f19e8bce640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f19e410c560 0x7f19e419ceb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.114:52056/0 (socket says 172.21.15.114:52056) 2024-04-23T10:39:29.120 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:39:29.118+0000 7f19e8bce640 1 -- 172.21.15.114:0/1610929039 learned_addr learned my addr 172.21.15.114:0/1610929039 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-04-23T10:39:29.121 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:39:29.118+0000 7f19e37fe640 1 --2- 172.21.15.114:0/1610929039 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f19e4103b40 0x7f19e4198b30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T10:39:29.121 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:39:29.118+0000 7f19e3fff640 1 --2- 172.21.15.114:0/1610929039 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f19e4103220 0x7f19e41985f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T10:39:29.121 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:39:29.118+0000 7f19e8bce640 1 -- 172.21.15.114:0/1610929039 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f19e4103220 msgr2=0x7f19e41985f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T10:39:29.121 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:39:29.118+0000 7f19e8bce640 1 --2- 172.21.15.114:0/1610929039 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f19e4103220 0x7f19e41985f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:39:29.121 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:39:29.118+0000 7f19e8bce640 1 -- 172.21.15.114:0/1610929039 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f19e4103b40 msgr2=0x7f19e4198b30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T10:39:29.121 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:39:29.118+0000 7f19e8bce640 1 --2- 172.21.15.114:0/1610929039 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f19e4103b40 0x7f19e4198b30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:39:29.121 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:39:29.119+0000 7f19e8bce640 1 -- 172.21.15.114:0/1610929039 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f19e419d5e0 con 0x7f19e410c560 2024-04-23T10:39:29.121 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:39:29.119+0000 7f19e8bce640 1 --2- 172.21.15.114:0/1610929039 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f19e410c560 0x7f19e419ceb0 secure :-1 s=READY pgs=903 cs=0 l=1 rev1=1 crypto rx=0x7f19dc004af0 tx=0x7f19dc02f5d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T10:39:29.121 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:39:29.119+0000 7f19e17fa640 1 -- 172.21.15.114:0/1610929039 <== mon.0 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f19dc034040 con 0x7f19e410c560 2024-04-23T10:39:29.121 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:39:29.119+0000 7f19ea631640 1 -- 172.21.15.114:0/1610929039 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f19e419d870 con 0x7f19e410c560 2024-04-23T10:39:29.121 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:39:29.119+0000 7f19e17fa640 1 -- 172.21.15.114:0/1610929039 <== mon.0 v2:172.21.15.114:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f19dc02fe80 con 0x7f19e410c560 2024-04-23T10:39:29.121 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:39:29.119+0000 7f19e17fa640 1 -- 172.21.15.114:0/1610929039 <== mon.0 v2:172.21.15.114:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f19dc0046a0 con 0x7f19e410c560 2024-04-23T10:39:29.121 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:39:29.119+0000 7f19ea631640 1 -- 172.21.15.114:0/1610929039 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f19e41a5160 con 0x7f19e410c560 2024-04-23T10:39:29.129 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:39:29.121+0000 7f19e17fa640 1 -- 172.21.15.114:0/1610929039 <== mon.0 v2:172.21.15.114:3300/0 4 ==== mgrmap(e 26) ==== 138778+0+0 (secure 0 0 0) 0x7f19dc037020 con 0x7f19e410c560 2024-04-23T10:39:29.129 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:39:29.122+0000 7f19ed425740 1 Processor -- start 2024-04-23T10:39:29.129 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:39:29.122+0000 7f19e17fa640 1 --2- 172.21.15.114:0/1610929039 >> [v2:172.21.15.114:6800/1395693466,v1:172.21.15.114:6801/1395693466] conn(0x7f19d00a3dc0 0x7f19d00a6280 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T10:39:29.129 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:39:29.122+0000 7f19ed425740 1 -- start start 2024-04-23T10:39:29.129 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:39:29.122+0000 7f19e17fa640 1 -- 172.21.15.114:0/1610929039 <== mon.0 v2:172.21.15.114:3300/0 5 ==== osd_map(108..108 src has 1..108) ==== 8629+0+0 (secure 0 0 0) 0x7f19dc0f16c0 con 0x7f19e410c560 2024-04-23T10:39:29.129 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:39:29.122+0000 7f19e37fe640 1 --2- 172.21.15.114:0/1610929039 >> [v2:172.21.15.114:6800/1395693466,v1:172.21.15.114:6801/1395693466] conn(0x7f19d00a3dc0 0x7f19d00a6280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T10:39:29.129 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:39:29.122+0000 7f19ed425740 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x559389676620 0x559389676a00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T10:39:29.129 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:39:29.122+0000 7f19ed425740 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x559389676f40 0x5593896812c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T10:39:29.129 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:39:29.122+0000 7f19ed425740 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x559389681800 0x559389683bf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T10:39:29.129 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:39:29.122+0000 7f19ed425740 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x5593895db970 con 0x559389676620 2024-04-23T10:39:29.130 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:39:29.122+0000 7f19ed425740 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 -- 0x5593895784a0 con 0x559389681800 2024-04-23T10:39:29.130 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:39:29.122+0000 7f19ed425740 1 -- --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_getmap magic: 0 -- 0x5593895d7930 con 0x559389676f40 2024-04-23T10:39:29.130 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:39:29.122+0000 7f19e3fff640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x559389676620 0x559389676a00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T10:39:29.130 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:39:29.122+0000 7f19e3fff640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x559389676620 0x559389676a00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.114:52060/0 (socket says 172.21.15.114:52060) 2024-04-23T10:39:29.130 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:39:29.122+0000 7f19e3fff640 1 -- 172.21.15.114:0/108145214 learned_addr learned my addr 172.21.15.114:0/108145214 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-04-23T10:39:29.130 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:39:29.122+0000 7f19e8bce640 1 --2- 172.21.15.114:0/108145214 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x559389681800 0x559389683bf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T10:39:29.130 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:39:29.122+0000 7f19e37fe640 1 --2- 172.21.15.114:0/108145214 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x559389676f40 0x5593896812c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T10:39:29.130 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:39:29.122+0000 7f19e37fe640 1 --2- 172.21.15.114:0/1610929039 >> [v2:172.21.15.114:6800/1395693466,v1:172.21.15.114:6801/1395693466] conn(0x7f19d00a3dc0 0x7f19d00a6280 secure :-1 s=READY pgs=142 cs=0 l=1 rev1=1 crypto rx=0x7f19d4005fd0 tx=0x7f19d400bf30 comp rx=0 tx=0).ready entity=mgr.15291 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T10:39:29.130 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:39:29.122+0000 7f19e3fff640 1 -- 172.21.15.114:0/108145214 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x559389676f40 msgr2=0x5593896812c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T10:39:29.130 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:39:29.122+0000 7f19e3fff640 1 --2- 172.21.15.114:0/108145214 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x559389676f40 0x5593896812c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:39:29.130 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:39:29.122+0000 7f19e3fff640 1 -- 172.21.15.114:0/108145214 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x559389681800 msgr2=0x559389683bf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T10:39:29.131 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:39:29.122+0000 7f19e3fff640 1 --2- 172.21.15.114:0/108145214 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x559389681800 0x559389683bf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:39:29.131 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:39:29.122+0000 7f19e3fff640 1 -- 172.21.15.114:0/108145214 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5593892ee210 con 0x559389676620 2024-04-23T10:39:29.131 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:39:29.123+0000 7f19e3fff640 1 --2- 172.21.15.114:0/108145214 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x559389676620 0x559389676a00 secure :-1 s=READY pgs=904 cs=0 l=1 rev1=1 crypto rx=0x7f19cc00bcb0 tx=0x7f19cc00f730 comp rx=0 tx=0).ready entity=mon.0 client_cookie=31aa5886868b209f server_cookie=0 in_seq=0 out_seq=0 2024-04-23T10:39:29.131 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:39:29.123+0000 7f19da7fc640 1 -- 172.21.15.114:0/108145214 <== mon.0 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f19cc015070 con 0x559389676620 2024-04-23T10:39:29.131 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:39:29.123+0000 7f19da7fc640 1 -- 172.21.15.114:0/108145214 <== mon.0 v2:172.21.15.114:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f19cc01a910 con 0x559389676620 2024-04-23T10:39:29.131 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:39:29.123+0000 7f19da7fc640 1 -- 172.21.15.114:0/108145214 <== mon.0 v2:172.21.15.114:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f19cc011920 con 0x559389676620 2024-04-23T10:39:29.131 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:39:29.123+0000 7f19ed425740 1 -- 172.21.15.114:0/108145214 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x559389676620 msgr2=0x559389676a00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T10:39:29.131 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:39:29.123+0000 7f19ed425740 1 --2- 172.21.15.114:0/108145214 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x559389676620 0x559389676a00 secure :-1 s=READY pgs=904 cs=0 l=1 rev1=1 crypto rx=0x7f19cc00bcb0 tx=0x7f19cc00f730 comp rx=0 tx=0).stop 2024-04-23T10:39:29.131 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:39:29.123+0000 7f19ed425740 1 -- 172.21.15.114:0/108145214 shutdown_connections 2024-04-23T10:39:29.131 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:39:29.123+0000 7f19ed425740 1 --2- 172.21.15.114:0/108145214 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x559389681800 0x559389683bf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:39:29.131 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:39:29.123+0000 7f19ed425740 1 --2- 172.21.15.114:0/108145214 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x559389676f40 0x5593896812c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:39:29.132 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:39:29.123+0000 7f19ed425740 1 --2- 172.21.15.114:0/108145214 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x559389676620 0x559389676a00 unknown :-1 s=CLOSED pgs=904 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:39:29.132 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:39:29.123+0000 7f19ed425740 1 -- 172.21.15.114:0/108145214 >> 172.21.15.114:0/108145214 conn(0x559389672b30 msgr2=0x559389672f10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T10:39:29.132 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:39:29.124+0000 7f19ed425740 1 -- 172.21.15.114:0/108145214 shutdown_connections 2024-04-23T10:39:29.132 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:39:29.124+0000 7f19ed425740 1 -- 172.21.15.114:0/108145214 wait complete. 2024-04-23T10:39:29.132 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:39:29.124+0000 7f19ed425740 1 -- start start 2024-04-23T10:39:29.132 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:39:29.124+0000 7f19ed425740 1 Processor -- start 2024-04-23T10:39:29.132 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:39:29.124+0000 7f19ed425740 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x559389676620 0x559389681170 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T10:39:29.132 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:39:29.124+0000 7f19ed425740 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x559389676f40 0x55938967e460 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T10:39:29.132 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:39:29.124+0000 7f19ed425740 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x559389681800 0x55938967e9c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T10:39:29.132 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:39:29.124+0000 7f19ed425740 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x559389684430 con 0x559389681800 2024-04-23T10:39:29.132 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:39:29.124+0000 7f19ed425740 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 -- 0x5593895760b0 con 0x559389676f40 2024-04-23T10:39:29.132 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:39:29.124+0000 7f19ed425740 1 -- --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_getmap magic: 0 -- 0x5593896845b0 con 0x559389676620 2024-04-23T10:39:29.132 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:39:29.124+0000 7f19ed425740 10 client.0 _ll_register_callbacks cb 0x559389563d30 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-23T10:39:29.133 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:39:29.124+0000 7f19e8bce640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x559389681800 0x55938967e9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T10:39:29.133 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:39:29.125+0000 7f19e3fff640 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x559389676620 0x559389681170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T10:39:29.133 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:39:29.125+0000 7f19e8bce640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x559389681800 0x55938967e9c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.114:52076/0 (socket says 172.21.15.114:52076) 2024-04-23T10:39:29.133 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:39:29.125+0000 7f19e8bce640 1 -- 172.21.15.114:0/3773365379 learned_addr learned my addr 172.21.15.114:0/3773365379 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-04-23T10:39:29.133 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:39:29.125+0000 7f19e37fe640 1 --2- 172.21.15.114:0/3773365379 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x559389676f40 0x55938967e460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T10:39:29.133 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:39:29.125+0000 7f19e8bce640 1 -- 172.21.15.114:0/3773365379 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x559389676620 msgr2=0x559389681170 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T10:39:29.133 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:39:29.125+0000 7f19e8bce640 1 --2- 172.21.15.114:0/3773365379 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x559389676620 0x559389681170 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:39:29.133 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:39:29.125+0000 7f19e8bce640 1 -- 172.21.15.114:0/3773365379 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x559389676f40 msgr2=0x55938967e460 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T10:39:29.133 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:39:29.125+0000 7f19e8bce640 1 --2- 172.21.15.114:0/3773365379 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x559389676f40 0x55938967e460 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:39:29.133 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:39:29.125+0000 7f19e8bce640 1 -- 172.21.15.114:0/3773365379 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5593893ba620 con 0x559389681800 2024-04-23T10:39:29.133 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:39:29.125+0000 7f19e8bce640 1 --2- 172.21.15.114:0/3773365379 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x559389681800 0x55938967e9c0 secure :-1 s=READY pgs=905 cs=0 l=1 rev1=1 crypto rx=0x7f19dc0b93e0 tx=0x7f19dc0bd0b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=627f9d96c6893e48 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T10:39:29.133 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:39:29.125+0000 7f19c37fe640 10 client.0 ms_handle_connect on v2:172.21.15.114:3300/0 2024-04-23T10:39:29.133 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:39:29.125+0000 7f19c37fe640 1 -- 172.21.15.114:0/3773365379 <== mon.0 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f19dc02c040 con 0x559389681800 2024-04-23T10:39:29.133 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:39:29.125+0000 7f19c37fe640 1 -- 172.21.15.114:0/3773365379 <== mon.0 v2:172.21.15.114:3300/0 2 ==== osd_map(108..108 src has 1..108) ==== 8629+0+0 (secure 0 0 0) 0x7f19dc0b9d70 con 0x559389681800 2024-04-23T10:39:29.134 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:39:29.125+0000 7f19ed425740 1 -- 172.21.15.114:0/3773365379 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f19cc01a910 con 0x559389681800 2024-04-23T10:39:29.134 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:39:29.125+0000 7f19c37fe640 1 -- 172.21.15.114:0/3773365379 <== mon.0 v2:172.21.15.114:3300/0 3 ==== mon_get_version_reply(handle=1 version=42) ==== 24+0+0 (secure 0 0 0) 0x7f19dc0bde00 con 0x559389681800 2024-04-23T10:39:29.134 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:39:29.125+0000 7f19ed425740 10 client.16995 fetch_fsmap learned FSMap version 42 2024-04-23T10:39:29.134 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:39:29.125+0000 7f19ed425740 1 -- 172.21.15.114:0/3773365379 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({config=0+,fsmap=42,monmap=4+}) -- 0x559389690360 con 0x559389681800 2024-04-23T10:39:29.134 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:39:29.125+0000 7f19c37fe640 1 -- 172.21.15.114:0/3773365379 <== mon.0 v2:172.21.15.114:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f19cc01a910 con 0x559389681800 2024-04-23T10:39:29.134 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:39:29.126+0000 7f19c37fe640 1 -- 172.21.15.114:0/3773365379 <== mon.0 v2:172.21.15.114:3300/0 5 ==== fsmap(e 42) ==== 8338+0+0 (secure 0 0 0) 0x7f19dc04d030 con 0x559389681800 2024-04-23T10:39:29.134 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:39:29.126+0000 7f19ed425740 10 client.16995 fetch_fsmap finished waiting for FSMap version 42 2024-04-23T10:39:29.134 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:39:29.126+0000 7f19ed425740 10 client.16995 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-04-23T10:39:29.134 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:39:29.126+0000 7f19ed425740 20 client.16995 populate_metadata read hostname 'smithi114' 2024-04-23T10:39:29.134 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:39:29.126+0000 7f19ed425740 1 --2- 172.21.15.114:0/3773365379 >> [v2:172.21.15.119:6838/937297340,v1:172.21.15.119:6839/937297340] conn(0x559389691840 0x559389693cc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T10:39:29.134 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:39:29.126+0000 7f19ed425740 4 client.16995 mds_command: new command op to 24545 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-23T10:39:29.134 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:39:29.126+0000 7f19ed425740 1 -- 172.21.15.114:0/3773365379 --> [v2:172.21.15.119:6838/937297340,v1:172.21.15.119:6839/937297340] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x5593895e9390 con 0x559389691840 2024-04-23T10:39:29.134 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:39:29.126+0000 7f19e3fff640 1 --2- 172.21.15.114:0/3773365379 >> [v2:172.21.15.119:6838/937297340,v1:172.21.15.119:6839/937297340] conn(0x559389691840 0x559389693cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T10:39:29.134 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:39:29.126+0000 7f19c37fe640 10 client.16995 ms_handle_connect on v2:172.21.15.119:6838/937297340 2024-04-23T10:39:29.135 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:39:29.126+0000 7f19e3fff640 1 --2- 172.21.15.114:0/3773365379 >> [v2:172.21.15.119:6838/937297340,v1:172.21.15.119:6839/937297340] conn(0x559389691840 0x559389693cc0 crc :-1 s=READY pgs=176 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=c857918eb8d1ab9 server_cookie=b3c9f80f52546897 in_seq=0 out_seq=0 2024-04-23T10:39:29.135 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:39:29.127+0000 7f19c37fe640 1 -- 172.21.15.114:0/3773365379 <== mds.0 v2:172.21.15.119:6838/937297340 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x5593895e9390 con 0x559389691840 2024-04-23T10:39:29.135 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:39:29.127+0000 7f19c37fe640 10 client.16995 handle_command_reply: tid=0 2024-04-23T10:39:29.135 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:39:29.127+0000 7f19ed425740 1 client.16995 shutdown 2024-04-23T10:39:29.135 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:39:29.127+0000 7f19ed425740 1 -- 172.21.15.114:0/3773365379 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x559389681800 msgr2=0x55938967e9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T10:39:29.135 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:39:29.127+0000 7f19ed425740 1 --2- 172.21.15.114:0/3773365379 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x559389681800 0x55938967e9c0 secure :-1 s=READY pgs=905 cs=0 l=1 rev1=1 crypto rx=0x7f19dc0b93e0 tx=0x7f19dc0bd0b0 comp rx=0 tx=0).stop 2024-04-23T10:39:29.135 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:39:29.127+0000 7f19ed425740 1 -- 172.21.15.114:0/3773365379 shutdown_connections 2024-04-23T10:39:29.135 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:39:29.127+0000 7f19ed425740 1 --2- 172.21.15.114:0/3773365379 >> [v2:172.21.15.119:6838/937297340,v1:172.21.15.119:6839/937297340] conn(0x559389691840 0x559389693cc0 crc :-1 s=READY pgs=176 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:39:29.135 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:39:29.127+0000 7f19ed425740 1 --2- 172.21.15.114:0/3773365379 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x559389681800 0x55938967e9c0 unknown :-1 s=CLOSED pgs=905 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:39:29.135 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:39:29.127+0000 7f19ed425740 1 --2- 172.21.15.114:0/3773365379 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x559389676f40 0x55938967e460 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:39:29.135 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:39:29.127+0000 7f19ed425740 1 --2- 172.21.15.114:0/3773365379 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x559389676620 0x559389681170 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:39:29.135 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:39:29.127+0000 7f19ed425740 1 -- 172.21.15.114:0/3773365379 >> 172.21.15.114:0/3773365379 conn(0x559389672b30 msgr2=0x559389678860 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T10:39:29.135 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:39:29.127+0000 7f19c37fe640 0 client.16995 ms_handle_reset on v2:172.21.15.119:6838/937297340 2024-04-23T10:39:29.135 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:39:29.127+0000 7f19ed425740 1 -- 172.21.15.114:0/3773365379 shutdown_connections 2024-04-23T10:39:29.136 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:39:29.127+0000 7f19ed425740 1 -- 172.21.15.114:0/3773365379 wait complete. 2024-04-23T10:39:29.136 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:39:29.127+0000 7f19ed425740 20 client.16995 trim_cache size 0 max 16384 2024-04-23T10:39:29.147 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:39:29.142+0000 7f19ed425740 1 Processor -- start 2024-04-23T10:39:29.147 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:39:29.142+0000 7f19ed425740 1 -- start start 2024-04-23T10:39:29.147 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:39:29.142+0000 7f19ed425740 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x559389676620 0x55938968d2f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T10:39:29.147 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:39:29.142+0000 7f19ed425740 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x559389676f40 0x559389680ca0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T10:39:29.147 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:39:29.142+0000 7f19ed425740 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x559389681800 0x5593896995d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T10:39:29.147 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:39:29.142+0000 7f19ed425740 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x7f19e41b0550 con 0x559389676f40 2024-04-23T10:39:29.147 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:39:29.142+0000 7f19ed425740 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 -- 0x7f19e41ade20 con 0x559389681800 2024-04-23T10:39:29.147 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:39:29.142+0000 7f19ed425740 1 -- --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_getmap magic: 0 -- 0x7f19e41abc90 con 0x559389676620 2024-04-23T10:39:29.147 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:39:29.142+0000 7f19e37fe640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x559389676f40 0x559389680ca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T10:39:29.147 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:39:29.142+0000 7f19e37fe640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x559389676f40 0x559389680ca0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.114:52082/0 (socket says 172.21.15.114:52082) 2024-04-23T10:39:29.147 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:39:29.142+0000 7f19e37fe640 1 -- 172.21.15.114:0/3489228934 learned_addr learned my addr 172.21.15.114:0/3489228934 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-04-23T10:39:29.147 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:39:29.142+0000 7f19e3fff640 1 --2- 172.21.15.114:0/3489228934 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x559389676620 0x55938968d2f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T10:39:29.148 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:39:29.143+0000 7f19e8bce640 1 --2- 172.21.15.114:0/3489228934 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x559389681800 0x5593896995d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T10:39:29.148 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:39:29.143+0000 7f19e37fe640 1 -- 172.21.15.114:0/3489228934 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x559389676620 msgr2=0x55938968d2f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T10:39:29.148 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:39:29.143+0000 7f19e37fe640 1 --2- 172.21.15.114:0/3489228934 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x559389676620 0x55938968d2f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:39:29.148 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:39:29.143+0000 7f19e37fe640 1 -- 172.21.15.114:0/3489228934 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x559389681800 msgr2=0x5593896995d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T10:39:29.148 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:39:29.143+0000 7f19e37fe640 1 --2- 172.21.15.114:0/3489228934 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x559389681800 0x5593896995d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:39:29.148 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:39:29.143+0000 7f19e37fe640 1 -- 172.21.15.114:0/3489228934 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55938967c9b0 con 0x559389676f40 2024-04-23T10:39:29.148 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:39:29.143+0000 7f19e37fe640 1 --2- 172.21.15.114:0/3489228934 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x559389676f40 0x559389680ca0 secure :-1 s=READY pgs=906 cs=0 l=1 rev1=1 crypto rx=0x7f19d4011440 tx=0x7f19d4014000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a4926e2380b2b6c3 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T10:39:29.148 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:39:29.143+0000 7f19d9ffb640 1 -- 172.21.15.114:0/3489228934 <== mon.0 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f19d4011af0 con 0x559389676f40 2024-04-23T10:39:29.148 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:39:29.143+0000 7f19d9ffb640 1 -- 172.21.15.114:0/3489228934 <== mon.0 v2:172.21.15.114:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f19d4011c90 con 0x559389676f40 2024-04-23T10:39:29.148 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:39:29.143+0000 7f19d9ffb640 1 -- 172.21.15.114:0/3489228934 <== mon.0 v2:172.21.15.114:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f19d401d670 con 0x559389676f40 2024-04-23T10:39:29.148 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:39:29.144+0000 7f19ed425740 1 -- 172.21.15.114:0/3489228934 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x559389676f40 msgr2=0x559389680ca0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T10:39:29.148 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:39:29.144+0000 7f19ed425740 1 --2- 172.21.15.114:0/3489228934 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x559389676f40 0x559389680ca0 secure :-1 s=READY pgs=906 cs=0 l=1 rev1=1 crypto rx=0x7f19d4011440 tx=0x7f19d4014000 comp rx=0 tx=0).stop 2024-04-23T10:39:29.148 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:39:29.144+0000 7f19ed425740 1 -- 172.21.15.114:0/3489228934 shutdown_connections 2024-04-23T10:39:29.149 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:39:29.144+0000 7f19ed425740 1 --2- 172.21.15.114:0/3489228934 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x559389681800 0x5593896995d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:39:29.149 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:39:29.144+0000 7f19ed425740 1 --2- 172.21.15.114:0/3489228934 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x559389676f40 0x559389680ca0 unknown :-1 s=CLOSED pgs=906 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:39:29.149 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:39:29.144+0000 7f19ed425740 1 --2- 172.21.15.114:0/3489228934 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x559389676620 0x55938968d2f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:39:29.149 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:39:29.144+0000 7f19ed425740 1 -- 172.21.15.114:0/3489228934 >> 172.21.15.114:0/3489228934 conn(0x559389672b30 msgr2=0x559389671420 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T10:39:29.149 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:39:29.144+0000 7f19ed425740 1 -- 172.21.15.114:0/3489228934 shutdown_connections 2024-04-23T10:39:29.149 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:39:29.144+0000 7f19ed425740 1 -- 172.21.15.114:0/3489228934 wait complete. 2024-04-23T10:39:29.149 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:39:29.144+0000 7f19ed425740 1 -- start start 2024-04-23T10:39:29.149 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:39:29.144+0000 7f19ed425740 1 Processor -- start 2024-04-23T10:39:29.149 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:39:29.144+0000 7f19ed425740 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x559389676620 0x5593896f4700 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T10:39:29.149 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:39:29.144+0000 7f19ed425740 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x559389676f40 0x5593896f4c40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T10:39:29.149 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:39:29.144+0000 7f19ed425740 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x559389681800 0x5593896eeb10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T10:39:29.149 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:39:29.144+0000 7f19ed425740 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x7f19e41ac2e0 con 0x559389681800 2024-04-23T10:39:29.149 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:39:29.144+0000 7f19ed425740 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 -- 0x7f19e41ad750 con 0x559389676f40 2024-04-23T10:39:29.149 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:39:29.144+0000 7f19ed425740 1 -- --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_getmap magic: 0 -- 0x7f19e41ac9b0 con 0x559389676620 2024-04-23T10:39:29.149 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:39:29.144+0000 7f19ed425740 10 client.0 _ll_register_callbacks cb 0x7f19dc0bd310 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-23T10:39:29.150 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:39:29.145+0000 7f19e8bce640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x559389681800 0x5593896eeb10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T10:39:29.150 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:39:29.145+0000 7f19e8bce640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x559389681800 0x5593896eeb10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.114:52088/0 (socket says 172.21.15.114:52088) 2024-04-23T10:39:29.150 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:39:29.145+0000 7f19e8bce640 1 -- 172.21.15.114:0/1761019200 learned_addr learned my addr 172.21.15.114:0/1761019200 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-04-23T10:39:29.150 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:39:29.145+0000 7f19e3fff640 1 --2- 172.21.15.114:0/1761019200 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x559389676620 0x5593896f4700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T10:39:29.150 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:39:29.145+0000 7f19e37fe640 1 --2- 172.21.15.114:0/1761019200 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x559389676f40 0x5593896f4c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T10:39:29.150 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:39:29.145+0000 7f19e8bce640 1 -- 172.21.15.114:0/1761019200 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x559389676620 msgr2=0x5593896f4700 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T10:39:29.150 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:39:29.145+0000 7f19e8bce640 1 --2- 172.21.15.114:0/1761019200 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x559389676620 0x5593896f4700 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:39:29.150 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:39:29.145+0000 7f19e8bce640 1 -- 172.21.15.114:0/1761019200 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x559389676f40 msgr2=0x5593896f4c40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T10:39:29.150 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:39:29.145+0000 7f19e8bce640 1 --2- 172.21.15.114:0/1761019200 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x559389676f40 0x5593896f4c40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:39:29.150 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:39:29.145+0000 7f19e8bce640 1 -- 172.21.15.114:0/1761019200 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5593896ef200 con 0x559389681800 2024-04-23T10:39:29.150 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:39:29.145+0000 7f19e37fe640 1 --2- 172.21.15.114:0/1761019200 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x559389676f40 0x5593896f4c40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-23T10:39:29.150 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:39:29.145+0000 7f19e8bce640 1 --2- 172.21.15.114:0/1761019200 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x559389681800 0x5593896eeb10 secure :-1 s=READY pgs=907 cs=0 l=1 rev1=1 crypto rx=0x7f19dc005fa0 tx=0x7f19dc0bdad0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=59e3bcaf05637a63 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T10:39:29.150 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:39:29.145+0000 7f19c37fe640 10 client.0 ms_handle_connect on v2:172.21.15.114:3300/0 2024-04-23T10:39:29.151 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:39:29.145+0000 7f19c37fe640 1 -- 172.21.15.114:0/1761019200 <== mon.0 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f19dc02c040 con 0x559389681800 2024-04-23T10:39:29.151 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:39:29.145+0000 7f19c37fe640 1 -- 172.21.15.114:0/1761019200 <== mon.0 v2:172.21.15.114:3300/0 2 ==== osd_map(108..108 src has 1..108) ==== 8629+0+0 (secure 0 0 0) 0x7f19dc0c5f30 con 0x559389681800 2024-04-23T10:39:29.151 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:39:29.145+0000 7f19ed425740 1 -- 172.21.15.114:0/1761019200 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f19d4011c90 con 0x559389681800 2024-04-23T10:39:29.151 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:39:29.145+0000 7f19c37fe640 1 -- 172.21.15.114:0/1761019200 <== mon.0 v2:172.21.15.114:3300/0 3 ==== mon_get_version_reply(handle=1 version=42) ==== 24+0+0 (secure 0 0 0) 0x7f19dc0c6130 con 0x559389681800 2024-04-23T10:39:29.151 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:39:29.145+0000 7f19ed425740 10 client.17001 fetch_fsmap learned FSMap version 42 2024-04-23T10:39:29.151 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:39:29.145+0000 7f19ed425740 1 -- 172.21.15.114:0/1761019200 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({config=0+,fsmap=42,monmap=4+}) -- 0x5593896ef200 con 0x559389681800 2024-04-23T10:39:29.151 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:39:29.145+0000 7f19c37fe640 1 -- 172.21.15.114:0/1761019200 <== mon.0 v2:172.21.15.114:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f19d4011c90 con 0x559389681800 2024-04-23T10:39:29.151 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:39:29.146+0000 7f19c37fe640 1 -- 172.21.15.114:0/1761019200 <== mon.0 v2:172.21.15.114:3300/0 5 ==== fsmap(e 42) ==== 8338+0+0 (secure 0 0 0) 0x7f19dc04d070 con 0x559389681800 2024-04-23T10:39:29.151 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:39:29.146+0000 7f19ed425740 10 client.17001 fetch_fsmap finished waiting for FSMap version 42 2024-04-23T10:39:29.151 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:39:29.146+0000 7f19ed425740 10 client.17001 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-04-23T10:39:29.151 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:39:29.146+0000 7f19ed425740 20 client.17001 populate_metadata read hostname 'smithi114' 2024-04-23T10:39:29.151 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:39:29.146+0000 7f19ed425740 1 --2- 172.21.15.114:0/1761019200 >> [v2:172.21.15.119:6838/937297340,v1:172.21.15.119:6839/937297340] conn(0x5593896fc270 0x5593896fe660 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T10:39:29.151 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:39:29.146+0000 7f19ed425740 4 client.17001 mds_command: new command op to 24545 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-04-23T10:39:29.151 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:39:29.146+0000 7f19ed425740 1 -- 172.21.15.114:0/1761019200 --> [v2:172.21.15.119:6838/937297340,v1:172.21.15.119:6839/937297340] -- command(tid 0: {"prefix": "flush journal"}) -- 0x5593895e3200 con 0x5593896fc270 2024-04-23T10:39:29.151 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:39:29.146+0000 7f19e3fff640 1 --2- 172.21.15.114:0/1761019200 >> [v2:172.21.15.119:6838/937297340,v1:172.21.15.119:6839/937297340] conn(0x5593896fc270 0x5593896fe660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T10:39:29.152 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:39:29.146+0000 7f19c37fe640 10 client.17001 ms_handle_connect on v2:172.21.15.119:6838/937297340 2024-04-23T10:39:29.152 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:39:29.146+0000 7f19e3fff640 1 --2- 172.21.15.114:0/1761019200 >> [v2:172.21.15.119:6838/937297340,v1:172.21.15.119:6839/937297340] conn(0x5593896fc270 0x5593896fe660 crc :-1 s=READY pgs=177 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=7da5595a6e86ecc9 server_cookie=b66a8b3ec0104584 in_seq=0 out_seq=0 2024-04-23T10:39:29.187 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:39:29.186+0000 7f19c37fe640 1 -- 172.21.15.114:0/1761019200 <== mds.0 v2:172.21.15.119:6838/937297340 1 ==== command_reply(tid 0: 0 ) ==== 8+0+44 (crc 0 0 0) 0x5593895e3200 con 0x5593896fc270 2024-04-23T10:39:29.187 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:39:29.186+0000 7f19c37fe640 10 client.17001 handle_command_reply: tid=0 2024-04-23T10:39:29.187 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:39:29.186+0000 7f19ed425740 1 client.17001 shutdown 2024-04-23T10:39:29.187 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:39:29.186+0000 7f19ed425740 1 -- 172.21.15.114:0/1761019200 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x559389681800 msgr2=0x5593896eeb10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T10:39:29.187 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:39:29.186+0000 7f19ed425740 1 --2- 172.21.15.114:0/1761019200 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x559389681800 0x5593896eeb10 secure :-1 s=READY pgs=907 cs=0 l=1 rev1=1 crypto rx=0x7f19dc005fa0 tx=0x7f19dc0bdad0 comp rx=0 tx=0).stop 2024-04-23T10:39:29.187 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:39:29.186+0000 7f19ed425740 1 -- 172.21.15.114:0/1761019200 shutdown_connections 2024-04-23T10:39:29.187 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:39:29.186+0000 7f19ed425740 1 --2- 172.21.15.114:0/1761019200 >> [v2:172.21.15.119:6838/937297340,v1:172.21.15.119:6839/937297340] conn(0x5593896fc270 0x5593896fe660 crc :-1 s=READY pgs=177 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:39:29.187 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:39:29.186+0000 7f19ed425740 1 --2- 172.21.15.114:0/1761019200 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x559389681800 0x5593896eeb10 unknown :-1 s=CLOSED pgs=907 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:39:29.187 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:39:29.186+0000 7f19c37fe640 0 client.17001 ms_handle_reset on v2:172.21.15.119:6838/937297340 2024-04-23T10:39:29.187 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:39:29.186+0000 7f19ed425740 1 --2- 172.21.15.114:0/1761019200 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x559389676f40 0x5593896f4c40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:39:29.187 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:39:29.186+0000 7f19ed425740 1 --2- 172.21.15.114:0/1761019200 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x559389676620 0x5593896f4700 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:39:29.187 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:39:29.186+0000 7f19ed425740 1 -- 172.21.15.114:0/1761019200 >> 172.21.15.114:0/1761019200 conn(0x559389672b30 msgr2=0x559389671cb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T10:39:29.187 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:39:29.186+0000 7f19ed425740 1 -- 172.21.15.114:0/1761019200 shutdown_connections 2024-04-23T10:39:29.188 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:39:29.186+0000 7f19ed425740 1 -- 172.21.15.114:0/1761019200 wait complete. 2024-04-23T10:39:29.188 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:39:29.187+0000 7f19ed425740 20 client.17001 trim_cache size 0 max 16384 2024-04-23T10:39:29.188 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stdout:{ 2024-04-23T10:39:29.188 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stdout: "message": "", 2024-04-23T10:39:29.188 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stdout: "return_code": 0 2024-04-23T10:39:29.188 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stdout:} 2024-04-23T10:39:29.189 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:39:29.187+0000 7f19ea631640 1 -- 172.21.15.114:0/1610929039 >> [v2:172.21.15.114:6800/1395693466,v1:172.21.15.114:6801/1395693466] conn(0x7f19d00a3dc0 msgr2=0x7f19d00a6280 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T10:39:29.189 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:39:29.187+0000 7f19ea631640 1 --2- 172.21.15.114:0/1610929039 >> [v2:172.21.15.114:6800/1395693466,v1:172.21.15.114:6801/1395693466] conn(0x7f19d00a3dc0 0x7f19d00a6280 secure :-1 s=READY pgs=142 cs=0 l=1 rev1=1 crypto rx=0x7f19d4005fd0 tx=0x7f19d400bf30 comp rx=0 tx=0).stop 2024-04-23T10:39:29.189 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:39:29.187+0000 7f19ea631640 1 -- 172.21.15.114:0/1610929039 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f19e410c560 msgr2=0x7f19e419ceb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T10:39:29.189 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:39:29.187+0000 7f19ea631640 1 --2- 172.21.15.114:0/1610929039 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f19e410c560 0x7f19e419ceb0 secure :-1 s=READY pgs=903 cs=0 l=1 rev1=1 crypto rx=0x7f19dc004af0 tx=0x7f19dc02f5d0 comp rx=0 tx=0).stop 2024-04-23T10:39:29.189 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:39:29.187+0000 7f19ea631640 1 -- 172.21.15.114:0/1610929039 shutdown_connections 2024-04-23T10:39:29.189 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:39:29.187+0000 7f19ea631640 1 --2- 172.21.15.114:0/1610929039 >> [v2:172.21.15.114:6800/1395693466,v1:172.21.15.114:6801/1395693466] conn(0x7f19d00a3dc0 0x7f19d00a6280 unknown :-1 s=CLOSED pgs=142 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:39:29.189 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:39:29.187+0000 7f19ea631640 1 --2- 172.21.15.114:0/1610929039 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f19e410c560 0x7f19e419ceb0 unknown :-1 s=CLOSED pgs=903 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:39:29.190 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:39:29.187+0000 7f19ea631640 1 --2- 172.21.15.114:0/1610929039 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f19e4103b40 0x7f19e4198b30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:39:29.190 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:39:29.187+0000 7f19ea631640 1 --2- 172.21.15.114:0/1610929039 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f19e4103220 0x7f19e41985f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:39:29.190 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:39:29.187+0000 7f19ea631640 1 -- 172.21.15.114:0/1610929039 >> 172.21.15.114:0/1610929039 conn(0x7f19e4078060 msgr2=0x7f19e40fddc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T10:39:29.190 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:39:29.187+0000 7f19ea631640 1 -- 172.21.15.114:0/1610929039 shutdown_connections 2024-04-23T10:39:29.190 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:39:29.187+0000 7f19ea631640 1 -- 172.21.15.114:0/1610929039 wait complete. 2024-04-23T10:39:29.952 INFO:tasks.workunit.client.0.smithi114.stdout: 1 713904 5.03 MB/sec execute 578 sec latency 3137.612 ms 2024-04-23T10:39:30.328 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 23 10:39:30 smithi204 ceph-mon[32649]: pgmap v405: 97 pgs: 97 active+clean; 208 MiB data, 2.8 GiB used, 1.0 TiB / 1.0 TiB avail; 281 KiB/s rd, 822 KiB/s wr, 62 op/s 2024-04-23T10:39:30.492 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 23 10:39:30 smithi119 ceph-mon[32608]: pgmap v405: 97 pgs: 97 active+clean; 208 MiB data, 2.8 GiB used, 1.0 TiB / 1.0 TiB avail; 281 KiB/s rd, 822 KiB/s wr, 62 op/s 2024-04-23T10:39:30.504 INFO:journalctl@ceph.mon.a.smithi114.stdout:Apr 23 10:39:30 smithi114 ceph-mon[29255]: pgmap v405: 97 pgs: 97 active+clean; 208 MiB data, 2.8 GiB used, 1.0 TiB / 1.0 TiB avail; 281 KiB/s rd, 822 KiB/s wr, 62 op/s 2024-04-23T10:39:30.952 INFO:tasks.workunit.client.0.smithi114.stdout: 1 713904 5.02 MB/sec execute 579 sec latency 1647.348 ms 2024-04-23T10:39:31.952 INFO:tasks.workunit.client.0.smithi114.stdout: 1 713904 5.01 MB/sec execute 580 sec latency 2647.486 ms 2024-04-23T10:39:32.328 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 23 10:39:32 smithi204 ceph-mon[32649]: pgmap v406: 97 pgs: 97 active+clean; 208 MiB data, 2.8 GiB used, 1.0 TiB / 1.0 TiB avail; 281 KiB/s rd, 822 KiB/s wr, 62 op/s 2024-04-23T10:39:32.491 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 23 10:39:32 smithi119 ceph-mon[32608]: pgmap v406: 97 pgs: 97 active+clean; 208 MiB data, 2.8 GiB used, 1.0 TiB / 1.0 TiB avail; 281 KiB/s rd, 822 KiB/s wr, 62 op/s 2024-04-23T10:39:32.503 INFO:journalctl@ceph.mon.a.smithi114.stdout:Apr 23 10:39:32 smithi114 ceph-mon[29255]: pgmap v406: 97 pgs: 97 active+clean; 208 MiB data, 2.8 GiB used, 1.0 TiB / 1.0 TiB avail; 281 KiB/s rd, 822 KiB/s wr, 62 op/s 2024-04-23T10:39:32.952 INFO:tasks.workunit.client.0.smithi114.stdout: 1 713904 5.00 MB/sec execute 581 sec latency 3647.574 ms 2024-04-23T10:39:33.952 INFO:tasks.workunit.client.0.smithi114.stdout: 1 713904 4.99 MB/sec execute 582 sec latency 4647.672 ms 2024-04-23T10:39:34.327 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 23 10:39:34 smithi204 ceph-mon[32649]: pgmap v407: 97 pgs: 97 active+clean; 205 MiB data, 2.8 GiB used, 1.0 TiB / 1.0 TiB avail; 397 KiB/s rd, 997 KiB/s wr, 89 op/s 2024-04-23T10:39:34.491 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 23 10:39:34 smithi119 ceph-mon[32608]: pgmap v407: 97 pgs: 97 active+clean; 205 MiB data, 2.8 GiB used, 1.0 TiB / 1.0 TiB avail; 397 KiB/s rd, 997 KiB/s wr, 89 op/s 2024-04-23T10:39:34.503 INFO:journalctl@ceph.mon.a.smithi114.stdout:Apr 23 10:39:34 smithi114 ceph-mon[29255]: pgmap v407: 97 pgs: 97 active+clean; 205 MiB data, 2.8 GiB used, 1.0 TiB / 1.0 TiB avail; 397 KiB/s rd, 997 KiB/s wr, 89 op/s 2024-04-23T10:39:34.952 INFO:tasks.workunit.client.0.smithi114.stdout: 1 714638 4.99 MB/sec execute 583 sec latency 4883.662 ms 2024-04-23T10:39:35.952 INFO:tasks.workunit.client.0.smithi114.stdout: 1 717038 5.00 MB/sec execute 584 sec latency 14.824 ms 2024-04-23T10:39:36.293 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 23 10:39:36 smithi119 ceph-mon[32608]: pgmap v408: 97 pgs: 97 active+clean; 205 MiB data, 2.8 GiB used, 1.0 TiB / 1.0 TiB avail; 334 KiB/s rd, 515 KiB/s wr, 65 op/s 2024-04-23T10:39:36.328 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 23 10:39:36 smithi204 ceph-mon[32649]: pgmap v408: 97 pgs: 97 active+clean; 205 MiB data, 2.8 GiB used, 1.0 TiB / 1.0 TiB avail; 334 KiB/s rd, 515 KiB/s wr, 65 op/s 2024-04-23T10:39:36.504 INFO:journalctl@ceph.mon.a.smithi114.stdout:Apr 23 10:39:36 smithi114 ceph-mon[29255]: pgmap v408: 97 pgs: 97 active+clean; 205 MiB data, 2.8 GiB used, 1.0 TiB / 1.0 TiB avail; 334 KiB/s rd, 515 KiB/s wr, 65 op/s 2024-04-23T10:39:36.952 INFO:tasks.workunit.client.0.smithi114.stdout: 1 717382 5.00 MB/sec execute 585 sec latency 845.377 ms 2024-04-23T10:39:37.953 INFO:tasks.workunit.client.0.smithi114.stdout: 1 717382 4.99 MB/sec execute 586 sec latency 1845.440 ms 2024-04-23T10:39:38.328 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 23 10:39:38 smithi204 ceph-mon[32649]: pgmap v409: 97 pgs: 97 active+clean; 208 MiB data, 2.8 GiB used, 1.0 TiB / 1.0 TiB avail; 523 KiB/s rd, 841 KiB/s wr, 100 op/s 2024-04-23T10:39:38.492 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 23 10:39:38 smithi119 ceph-mon[32608]: pgmap v409: 97 pgs: 97 active+clean; 208 MiB data, 2.8 GiB used, 1.0 TiB / 1.0 TiB avail; 523 KiB/s rd, 841 KiB/s wr, 100 op/s 2024-04-23T10:39:38.504 INFO:journalctl@ceph.mon.a.smithi114.stdout:Apr 23 10:39:38 smithi114 ceph-mon[29255]: pgmap v409: 97 pgs: 97 active+clean; 208 MiB data, 2.8 GiB used, 1.0 TiB / 1.0 TiB avail; 523 KiB/s rd, 841 KiB/s wr, 100 op/s 2024-04-23T10:39:38.954 INFO:tasks.workunit.client.0.smithi114.stdout: 1 717382 4.98 MB/sec execute 587 sec latency 2845.565 ms 2024-04-23T10:39:39.953 INFO:tasks.workunit.client.0.smithi114.stdout: 1 717449 4.98 MB/sec execute 588 sec latency 3091.005 ms 2024-04-23T10:39:40.327 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 23 10:39:40 smithi204 ceph-mon[32649]: pgmap v410: 97 pgs: 97 active+clean; 208 MiB data, 2.8 GiB used, 1.0 TiB / 1.0 TiB avail; 304 KiB/s rd, 501 KiB/s wr, 61 op/s 2024-04-23T10:39:40.491 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 23 10:39:40 smithi119 ceph-mon[32608]: pgmap v410: 97 pgs: 97 active+clean; 208 MiB data, 2.8 GiB used, 1.0 TiB / 1.0 TiB avail; 304 KiB/s rd, 501 KiB/s wr, 61 op/s 2024-04-23T10:39:40.504 INFO:journalctl@ceph.mon.a.smithi114.stdout:Apr 23 10:39:40 smithi114 ceph-mon[29255]: pgmap v410: 97 pgs: 97 active+clean; 208 MiB data, 2.8 GiB used, 1.0 TiB / 1.0 TiB avail; 304 KiB/s rd, 501 KiB/s wr, 61 op/s 2024-04-23T10:39:40.953 INFO:tasks.workunit.client.0.smithi114.stdout: 1 717449 4.97 MB/sec execute 589 sec latency 1639.213 ms 2024-04-23T10:39:41.953 INFO:tasks.workunit.client.0.smithi114.stdout: 1 717449 4.96 MB/sec execute 590 sec latency 2639.296 ms 2024-04-23T10:39:42.283 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:39:42.283+0000 7f7889120640 1 -- 172.21.15.114:0/1705523470 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f7884101cc0 msgr2=0x7f7884105ab0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T10:39:42.283 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:39:42.283+0000 7f7889120640 1 --2- 172.21.15.114:0/1705523470 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f7884101cc0 0x7f7884105ab0 secure :-1 s=READY pgs=908 cs=0 l=1 rev1=1 crypto rx=0x7f7874009900 tx=0x7f787402a3f0 comp rx=0 tx=0).stop 2024-04-23T10:39:42.283 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:39:42.283+0000 7f7889120640 1 -- 172.21.15.114:0/1705523470 shutdown_connections 2024-04-23T10:39:42.283 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:39:42.283+0000 7f7889120640 1 --2- 172.21.15.114:0/1705523470 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f7884106190 0x7f788410dd80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:39:42.283 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:39:42.283+0000 7f7889120640 1 --2- 172.21.15.114:0/1705523470 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f7884101cc0 0x7f7884105ab0 unknown :-1 s=CLOSED pgs=908 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:39:42.283 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:39:42.283+0000 7f7889120640 1 --2- 172.21.15.114:0/1705523470 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f7884101310 0x7f78841016f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:39:42.283 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:39:42.283+0000 7f7889120640 1 -- 172.21.15.114:0/1705523470 >> 172.21.15.114:0/1705523470 conn(0x7f7884077cc0 msgr2=0x7f78840780d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T10:39:42.283 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:39:42.283+0000 7f7889120640 1 -- 172.21.15.114:0/1705523470 shutdown_connections 2024-04-23T10:39:42.283 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:39:42.283+0000 7f7889120640 1 -- 172.21.15.114:0/1705523470 wait complete. 2024-04-23T10:39:42.283 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:39:42.284+0000 7f7889120640 1 Processor -- start 2024-04-23T10:39:42.285 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:39:42.284+0000 7f7889120640 1 -- start start 2024-04-23T10:39:42.286 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:39:42.285+0000 7f7889120640 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f7884101310 0x7f7884198630 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T10:39:42.286 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:39:42.285+0000 7f7889120640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f7884106190 0x7f7884198b70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T10:39:42.286 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:39:42.285+0000 7f7889120640 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f788419cef0 0x7f788419d3a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T10:39:42.286 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:39:42.285+0000 7f7889120640 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x7f78841121f0 con 0x7f7884106190 2024-04-23T10:39:42.286 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:39:42.285+0000 7f7889120640 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 -- 0x7f7884112070 con 0x7f788419cef0 2024-04-23T10:39:42.286 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:39:42.285+0000 7f7889120640 1 -- --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_getmap magic: 0 -- 0x7f7884112370 con 0x7f7884101310 2024-04-23T10:39:42.286 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:39:42.285+0000 7f788259c640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f7884106190 0x7f7884198b70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T10:39:42.286 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:39:42.285+0000 7f788259c640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f7884106190 0x7f7884198b70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.114:40256/0 (socket says 172.21.15.114:40256) 2024-04-23T10:39:42.286 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:39:42.285+0000 7f788259c640 1 -- 172.21.15.114:0/1083379445 learned_addr learned my addr 172.21.15.114:0/1083379445 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-04-23T10:39:42.286 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:39:42.285+0000 7f788359e640 1 --2- 172.21.15.114:0/1083379445 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f788419cef0 0x7f788419d3a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T10:39:42.286 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:39:42.285+0000 7f7882d9d640 1 --2- 172.21.15.114:0/1083379445 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f7884101310 0x7f7884198630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T10:39:42.286 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:39:42.285+0000 7f788259c640 1 -- 172.21.15.114:0/1083379445 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f7884101310 msgr2=0x7f7884198630 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T10:39:42.286 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:39:42.285+0000 7f788259c640 1 --2- 172.21.15.114:0/1083379445 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f7884101310 0x7f7884198630 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:39:42.287 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:39:42.285+0000 7f788259c640 1 -- 172.21.15.114:0/1083379445 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f788419cef0 msgr2=0x7f788419d3a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T10:39:42.287 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:39:42.285+0000 7f788259c640 1 --2- 172.21.15.114:0/1083379445 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f788419cef0 0x7f788419d3a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:39:42.287 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:39:42.285+0000 7f788259c640 1 -- 172.21.15.114:0/1083379445 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f788419d9e0 con 0x7f7884106190 2024-04-23T10:39:42.287 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:39:42.285+0000 7f788359e640 1 --2- 172.21.15.114:0/1083379445 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f788419cef0 0x7f788419d3a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-23T10:39:42.287 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:39:42.285+0000 7f788259c640 1 --2- 172.21.15.114:0/1083379445 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f7884106190 0x7f7884198b70 secure :-1 s=READY pgs=909 cs=0 l=1 rev1=1 crypto rx=0x7f7874031a90 tx=0x7f7874004270 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T10:39:42.287 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:39:42.285+0000 7f787bfff640 1 -- 172.21.15.114:0/1083379445 <== mon.0 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f787403f030 con 0x7f7884106190 2024-04-23T10:39:42.287 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:39:42.286+0000 7f7889120640 1 -- 172.21.15.114:0/1083379445 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f788419dc70 con 0x7f7884106190 2024-04-23T10:39:42.287 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:39:42.286+0000 7f787bfff640 1 -- 172.21.15.114:0/1083379445 <== mon.0 v2:172.21.15.114:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f7874003cb0 con 0x7f7884106190 2024-04-23T10:39:42.287 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:39:42.286+0000 7f787bfff640 1 -- 172.21.15.114:0/1083379445 <== mon.0 v2:172.21.15.114:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7874035070 con 0x7f7884106190 2024-04-23T10:39:42.287 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:39:42.286+0000 7f7889120640 1 -- 172.21.15.114:0/1083379445 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f78841a5490 con 0x7f7884106190 2024-04-23T10:39:42.294 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:39:42.288+0000 7f787bfff640 1 -- 172.21.15.114:0/1083379445 <== mon.0 v2:172.21.15.114:3300/0 4 ==== mgrmap(e 26) ==== 138778+0+0 (secure 0 0 0) 0x7f7874042030 con 0x7f7884106190 2024-04-23T10:39:42.294 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:39:42.288+0000 7f788be37740 1 Processor -- start 2024-04-23T10:39:42.294 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:39:42.288+0000 7f788be37740 1 -- start start 2024-04-23T10:39:42.294 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:39:42.288+0000 7f787bfff640 1 --2- 172.21.15.114:0/1083379445 >> [v2:172.21.15.114:6800/1395693466,v1:172.21.15.114:6801/1395693466] conn(0x7f78700a3dc0 0x7f78700a6280 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T10:39:42.294 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:39:42.288+0000 7f788359e640 1 --2- 172.21.15.114:0/1083379445 >> [v2:172.21.15.114:6800/1395693466,v1:172.21.15.114:6801/1395693466] conn(0x7f78700a3dc0 0x7f78700a6280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T10:39:42.294 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:39:42.289+0000 7f787bfff640 1 -- 172.21.15.114:0/1083379445 <== mon.0 v2:172.21.15.114:3300/0 5 ==== osd_map(108..108 src has 1..108) ==== 8629+0+0 (secure 0 0 0) 0x7f78740f0ed0 con 0x7f7884106190 2024-04-23T10:39:42.294 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:39:42.289+0000 7f788be37740 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x55f382c5e620 0x55f382c5ea00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T10:39:42.294 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:39:42.289+0000 7f788be37740 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55f382c5ef40 0x55f382c692c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T10:39:42.294 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:39:42.289+0000 7f788be37740 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x55f382c69800 0x55f382c6bbf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T10:39:42.295 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:39:42.289+0000 7f788be37740 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x55f382bc3970 con 0x55f382c5ef40 2024-04-23T10:39:42.295 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:39:42.289+0000 7f788be37740 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 -- 0x55f382b604a0 con 0x55f382c69800 2024-04-23T10:39:42.295 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:39:42.289+0000 7f788be37740 1 -- --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_getmap magic: 0 -- 0x55f382bbf930 con 0x55f382c5e620 2024-04-23T10:39:42.295 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:39:42.289+0000 7f788259c640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55f382c5ef40 0x55f382c692c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T10:39:42.295 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:39:42.289+0000 7f7882d9d640 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x55f382c5e620 0x55f382c5ea00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T10:39:42.295 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:39:42.289+0000 7f788259c640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55f382c5ef40 0x55f382c692c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.114:40270/0 (socket says 172.21.15.114:40270) 2024-04-23T10:39:42.295 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:39:42.289+0000 7f788259c640 1 -- 172.21.15.114:0/4107832725 learned_addr learned my addr 172.21.15.114:0/4107832725 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-04-23T10:39:42.295 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:39:42.289+0000 7f788359e640 1 --2- 172.21.15.114:0/1083379445 >> [v2:172.21.15.114:6800/1395693466,v1:172.21.15.114:6801/1395693466] conn(0x7f78700a3dc0 0x7f78700a6280 secure :-1 s=READY pgs=143 cs=0 l=1 rev1=1 crypto rx=0x7f787c005cc0 tx=0x7f787c005c50 comp rx=0 tx=0).ready entity=mgr.15291 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T10:39:42.295 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:39:42.289+0000 7f788359e640 1 --2- 172.21.15.114:0/4107832725 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x55f382c69800 0x55f382c6bbf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T10:39:42.295 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:39:42.289+0000 7f788259c640 1 -- 172.21.15.114:0/4107832725 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x55f382c5e620 msgr2=0x55f382c5ea00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T10:39:42.295 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:39:42.289+0000 7f788259c640 1 --2- 172.21.15.114:0/4107832725 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x55f382c5e620 0x55f382c5ea00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:39:42.295 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:39:42.289+0000 7f788259c640 1 -- 172.21.15.114:0/4107832725 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x55f382c69800 msgr2=0x55f382c6bbf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T10:39:42.295 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:39:42.289+0000 7f788259c640 1 --2- 172.21.15.114:0/4107832725 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x55f382c69800 0x55f382c6bbf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:39:42.296 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:39:42.289+0000 7f788259c640 1 -- 172.21.15.114:0/4107832725 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55f3828d6210 con 0x55f382c5ef40 2024-04-23T10:39:42.296 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:39:42.289+0000 7f788259c640 1 --2- 172.21.15.114:0/4107832725 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55f382c5ef40 0x55f382c692c0 secure :-1 s=READY pgs=910 cs=0 l=1 rev1=1 crypto rx=0x7f78740bcb00 tx=0x7f78740bcc90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3f1f25eaaf68fafa server_cookie=0 in_seq=0 out_seq=0 2024-04-23T10:39:42.296 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:39:42.290+0000 7f78797fa640 1 -- 172.21.15.114:0/4107832725 <== mon.0 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7874049040 con 0x55f382c5ef40 2024-04-23T10:39:42.296 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:39:42.290+0000 7f78797fa640 1 -- 172.21.15.114:0/4107832725 <== mon.0 v2:172.21.15.114:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f78740bce00 con 0x55f382c5ef40 2024-04-23T10:39:42.296 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:39:42.290+0000 7f78797fa640 1 -- 172.21.15.114:0/4107832725 <== mon.0 v2:172.21.15.114:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f78740b9470 con 0x55f382c5ef40 2024-04-23T10:39:42.296 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:39:42.290+0000 7f788be37740 1 -- 172.21.15.114:0/4107832725 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55f382c5ef40 msgr2=0x55f382c692c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T10:39:42.296 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:39:42.290+0000 7f788be37740 1 --2- 172.21.15.114:0/4107832725 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55f382c5ef40 0x55f382c692c0 secure :-1 s=READY pgs=910 cs=0 l=1 rev1=1 crypto rx=0x7f78740bcb00 tx=0x7f78740bcc90 comp rx=0 tx=0).stop 2024-04-23T10:39:42.296 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:39:42.290+0000 7f788be37740 1 -- 172.21.15.114:0/4107832725 shutdown_connections 2024-04-23T10:39:42.296 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:39:42.290+0000 7f788be37740 1 --2- 172.21.15.114:0/4107832725 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x55f382c69800 0x55f382c6bbf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:39:42.296 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:39:42.290+0000 7f788be37740 1 --2- 172.21.15.114:0/4107832725 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55f382c5ef40 0x55f382c692c0 unknown :-1 s=CLOSED pgs=910 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:39:42.296 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:39:42.290+0000 7f788be37740 1 --2- 172.21.15.114:0/4107832725 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x55f382c5e620 0x55f382c5ea00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:39:42.296 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:39:42.290+0000 7f788be37740 1 -- 172.21.15.114:0/4107832725 >> 172.21.15.114:0/4107832725 conn(0x55f382c5ab30 msgr2=0x55f382c5af10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T10:39:42.296 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:39:42.290+0000 7f788be37740 1 -- 172.21.15.114:0/4107832725 shutdown_connections 2024-04-23T10:39:42.296 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:39:42.290+0000 7f788be37740 1 -- 172.21.15.114:0/4107832725 wait complete. 2024-04-23T10:39:42.296 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:39:42.291+0000 7f788be37740 1 -- start start 2024-04-23T10:39:42.296 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:39:42.291+0000 7f788be37740 1 Processor -- start 2024-04-23T10:39:42.297 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:39:42.291+0000 7f788be37740 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x55f382c5e620 0x55f382c69150 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T10:39:42.297 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:39:42.291+0000 7f788be37740 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55f382c5ef40 0x55f382c66440 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T10:39:42.297 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:39:42.291+0000 7f788be37740 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x55f382c69800 0x55f382c669a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T10:39:42.297 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:39:42.291+0000 7f788be37740 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x55f382c6c1f0 con 0x55f382c5ef40 2024-04-23T10:39:42.297 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:39:42.291+0000 7f788be37740 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 -- 0x55f382b5e0b0 con 0x55f382c5e620 2024-04-23T10:39:42.297 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:39:42.291+0000 7f788be37740 1 -- --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_getmap magic: 0 -- 0x55f382c6c370 con 0x55f382c69800 2024-04-23T10:39:42.297 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:39:42.291+0000 7f788be37740 10 client.0 _ll_register_callbacks cb 0x55f382b4bd30 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-23T10:39:42.297 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:39:42.291+0000 7f788259c640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55f382c5ef40 0x55f382c66440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T10:39:42.297 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:39:42.291+0000 7f788259c640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55f382c5ef40 0x55f382c66440 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.114:40284/0 (socket says 172.21.15.114:40284) 2024-04-23T10:39:42.297 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:39:42.291+0000 7f788259c640 1 -- 172.21.15.114:0/1304040491 learned_addr learned my addr 172.21.15.114:0/1304040491 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-04-23T10:39:42.297 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:39:42.291+0000 7f788359e640 1 --2- 172.21.15.114:0/1304040491 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x55f382c69800 0x55f382c669a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T10:39:42.297 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:39:42.291+0000 7f7882d9d640 1 --2- 172.21.15.114:0/1304040491 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x55f382c5e620 0x55f382c69150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T10:39:42.297 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:39:42.291+0000 7f788259c640 1 -- 172.21.15.114:0/1304040491 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x55f382c69800 msgr2=0x55f382c669a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T10:39:42.297 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:39:42.291+0000 7f788259c640 1 --2- 172.21.15.114:0/1304040491 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x55f382c69800 0x55f382c669a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:39:42.297 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:39:42.291+0000 7f788259c640 1 -- 172.21.15.114:0/1304040491 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x55f382c5e620 msgr2=0x55f382c69150 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T10:39:42.298 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:39:42.291+0000 7f788259c640 1 --2- 172.21.15.114:0/1304040491 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x55f382c5e620 0x55f382c69150 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:39:42.298 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:39:42.291+0000 7f788259c640 1 -- 172.21.15.114:0/1304040491 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55f3829a2620 con 0x55f382c5ef40 2024-04-23T10:39:42.298 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:39:42.292+0000 7f788259c640 1 --2- 172.21.15.114:0/1304040491 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55f382c5ef40 0x55f382c66440 secure :-1 s=READY pgs=911 cs=0 l=1 rev1=1 crypto rx=0x7f78740dcfd0 tx=0x7f787402a3f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f325ef3e3267ad77 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T10:39:42.298 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:39:42.292+0000 7f786a7fc640 10 client.0 ms_handle_connect on v2:172.21.15.114:3300/0 2024-04-23T10:39:42.298 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:39:42.292+0000 7f786a7fc640 1 -- 172.21.15.114:0/1304040491 <== mon.0 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7874049070 con 0x55f382c5ef40 2024-04-23T10:39:42.298 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:39:42.292+0000 7f786a7fc640 1 -- 172.21.15.114:0/1304040491 <== mon.0 v2:172.21.15.114:3300/0 2 ==== osd_map(108..108 src has 1..108) ==== 8629+0+0 (secure 0 0 0) 0x7f78740b9470 con 0x55f382c5ef40 2024-04-23T10:39:42.298 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:39:42.292+0000 7f788be37740 1 -- 172.21.15.114:0/1304040491 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f78740bce00 con 0x55f382c5ef40 2024-04-23T10:39:42.298 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:39:42.292+0000 7f786a7fc640 1 -- 172.21.15.114:0/1304040491 <== mon.0 v2:172.21.15.114:3300/0 3 ==== mon_get_version_reply(handle=1 version=42) ==== 24+0+0 (secure 0 0 0) 0x7f78740c11b0 con 0x55f382c5ef40 2024-04-23T10:39:42.298 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:39:42.292+0000 7f788be37740 10 client.17013 fetch_fsmap learned FSMap version 42 2024-04-23T10:39:42.298 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:39:42.292+0000 7f788be37740 1 -- 172.21.15.114:0/1304040491 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({config=0+,fsmap=42,monmap=4+}) -- 0x55f382c6c1f0 con 0x55f382c5ef40 2024-04-23T10:39:42.298 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:39:42.292+0000 7f786a7fc640 1 -- 172.21.15.114:0/1304040491 <== mon.0 v2:172.21.15.114:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f78740bce00 con 0x55f382c5ef40 2024-04-23T10:39:42.298 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:39:42.292+0000 7f786a7fc640 1 -- 172.21.15.114:0/1304040491 <== mon.0 v2:172.21.15.114:3300/0 5 ==== fsmap(e 42) ==== 8338+0+0 (secure 0 0 0) 0x7f78740c55e0 con 0x55f382c5ef40 2024-04-23T10:39:42.298 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:39:42.292+0000 7f788be37740 10 client.17013 fetch_fsmap finished waiting for FSMap version 42 2024-04-23T10:39:42.298 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:39:42.292+0000 7f788be37740 10 client.17013 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-04-23T10:39:42.299 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:39:42.292+0000 7f788be37740 20 client.17013 populate_metadata read hostname 'smithi114' 2024-04-23T10:39:42.299 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:39:42.292+0000 7f788be37740 1 --2- 172.21.15.114:0/1304040491 >> [v2:172.21.15.119:6838/937297340,v1:172.21.15.119:6839/937297340] conn(0x55f382c79330 0x55f382c7b720 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T10:39:42.299 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:39:42.292+0000 7f788be37740 4 client.17013 mds_command: new command op to 24545 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-23T10:39:42.299 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:39:42.292+0000 7f788be37740 1 -- 172.21.15.114:0/1304040491 --> [v2:172.21.15.119:6838/937297340,v1:172.21.15.119:6839/937297340] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55f382bd1390 con 0x55f382c79330 2024-04-23T10:39:42.299 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:39:42.293+0000 7f7882d9d640 1 --2- 172.21.15.114:0/1304040491 >> [v2:172.21.15.119:6838/937297340,v1:172.21.15.119:6839/937297340] conn(0x55f382c79330 0x55f382c7b720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T10:39:42.299 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:39:42.293+0000 7f7882d9d640 1 --2- 172.21.15.114:0/1304040491 >> [v2:172.21.15.119:6838/937297340,v1:172.21.15.119:6839/937297340] conn(0x55f382c79330 0x55f382c7b720 crc :-1 s=READY pgs=178 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=8c90a82b27e0f928 server_cookie=6c9bbae295a7a49f in_seq=0 out_seq=0 2024-04-23T10:39:42.299 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:39:42.293+0000 7f786a7fc640 10 client.17013 ms_handle_connect on v2:172.21.15.119:6838/937297340 2024-04-23T10:39:42.299 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:39:42.294+0000 7f786a7fc640 1 -- 172.21.15.114:0/1304040491 <== mds.0 v2:172.21.15.119:6838/937297340 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x55f382bd1390 con 0x55f382c79330 2024-04-23T10:39:42.299 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:39:42.294+0000 7f786a7fc640 10 client.17013 handle_command_reply: tid=0 2024-04-23T10:39:42.299 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:39:42.294+0000 7f788be37740 1 client.17013 shutdown 2024-04-23T10:39:42.299 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:39:42.294+0000 7f788be37740 1 -- 172.21.15.114:0/1304040491 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55f382c5ef40 msgr2=0x55f382c66440 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T10:39:42.299 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:39:42.294+0000 7f788be37740 1 --2- 172.21.15.114:0/1304040491 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55f382c5ef40 0x55f382c66440 secure :-1 s=READY pgs=911 cs=0 l=1 rev1=1 crypto rx=0x7f78740dcfd0 tx=0x7f787402a3f0 comp rx=0 tx=0).stop 2024-04-23T10:39:42.299 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:39:42.294+0000 7f788be37740 1 -- 172.21.15.114:0/1304040491 shutdown_connections 2024-04-23T10:39:42.299 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:39:42.294+0000 7f788be37740 1 --2- 172.21.15.114:0/1304040491 >> [v2:172.21.15.119:6838/937297340,v1:172.21.15.119:6839/937297340] conn(0x55f382c79330 0x55f382c7b720 crc :-1 s=READY pgs=178 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:39:42.299 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:39:42.294+0000 7f788be37740 1 --2- 172.21.15.114:0/1304040491 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x55f382c69800 0x55f382c669a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:39:42.300 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:39:42.294+0000 7f788be37740 1 --2- 172.21.15.114:0/1304040491 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55f382c5ef40 0x55f382c66440 unknown :-1 s=CLOSED pgs=911 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:39:42.300 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:39:42.294+0000 7f786a7fc640 0 client.17013 ms_handle_reset on v2:172.21.15.119:6838/937297340 2024-04-23T10:39:42.300 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:39:42.294+0000 7f788be37740 1 --2- 172.21.15.114:0/1304040491 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x55f382c5e620 0x55f382c69150 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:39:42.300 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:39:42.294+0000 7f788be37740 1 -- 172.21.15.114:0/1304040491 >> 172.21.15.114:0/1304040491 conn(0x55f382c5ab30 msgr2=0x55f382c59cb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T10:39:42.300 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:39:42.294+0000 7f788be37740 1 -- 172.21.15.114:0/1304040491 shutdown_connections 2024-04-23T10:39:42.300 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:39:42.294+0000 7f788be37740 1 -- 172.21.15.114:0/1304040491 wait complete. 2024-04-23T10:39:42.300 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:39:42.294+0000 7f788be37740 20 client.17013 trim_cache size 0 max 16384 2024-04-23T10:39:42.308 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:39:42.309+0000 7f788be37740 1 Processor -- start 2024-04-23T10:39:42.308 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:39:42.309+0000 7f788be37740 1 -- start start 2024-04-23T10:39:42.313 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:39:42.309+0000 7f788be37740 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x55f382c5e620 0x55f382c68c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T10:39:42.313 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:39:42.309+0000 7f788be37740 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x55f382c5ef40 0x55f382c80fa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T10:39:42.313 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:39:42.309+0000 7f788be37740 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55f382c69800 0x55f382c814e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T10:39:42.313 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:39:42.309+0000 7f788be37740 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x7f78841b04e0 con 0x55f382c69800 2024-04-23T10:39:42.313 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:39:42.309+0000 7f788be37740 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 -- 0x7f78841addb0 con 0x55f382c5e620 2024-04-23T10:39:42.313 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:39:42.309+0000 7f788be37740 1 -- --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_getmap magic: 0 -- 0x7f78841abc20 con 0x55f382c5ef40 2024-04-23T10:39:42.313 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:39:42.310+0000 7f788359e640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55f382c69800 0x55f382c814e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T10:39:42.313 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:39:42.310+0000 7f788359e640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55f382c69800 0x55f382c814e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.114:40286/0 (socket says 172.21.15.114:40286) 2024-04-23T10:39:42.313 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:39:42.310+0000 7f788359e640 1 -- 172.21.15.114:0/3295432081 learned_addr learned my addr 172.21.15.114:0/3295432081 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-04-23T10:39:42.313 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:39:42.310+0000 7f788259c640 1 --2- 172.21.15.114:0/3295432081 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x55f382c5ef40 0x55f382c80fa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T10:39:42.313 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:39:42.310+0000 7f7882d9d640 1 --2- 172.21.15.114:0/3295432081 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x55f382c5e620 0x55f382c68c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T10:39:42.313 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:39:42.310+0000 7f788359e640 1 -- 172.21.15.114:0/3295432081 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x55f382c5ef40 msgr2=0x55f382c80fa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T10:39:42.313 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:39:42.310+0000 7f788359e640 1 --2- 172.21.15.114:0/3295432081 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x55f382c5ef40 0x55f382c80fa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:39:42.314 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:39:42.310+0000 7f788359e640 1 -- 172.21.15.114:0/3295432081 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x55f382c5e620 msgr2=0x55f382c68c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T10:39:42.314 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:39:42.310+0000 7f788359e640 1 --2- 172.21.15.114:0/3295432081 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x55f382c5e620 0x55f382c68c80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:39:42.314 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:39:42.310+0000 7f788359e640 1 -- 172.21.15.114:0/3295432081 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55f382c66d10 con 0x55f382c69800 2024-04-23T10:39:42.314 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:39:42.310+0000 7f788359e640 1 --2- 172.21.15.114:0/3295432081 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55f382c69800 0x55f382c814e0 secure :-1 s=READY pgs=912 cs=0 l=1 rev1=1 crypto rx=0x7f787c00ebb0 tx=0x7f787c011000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9bec6fa28f7ad7f4 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T10:39:42.314 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:39:42.310+0000 7f7878ff9640 1 -- 172.21.15.114:0/3295432081 <== mon.0 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f787c003b70 con 0x55f382c69800 2024-04-23T10:39:42.314 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:39:42.310+0000 7f7878ff9640 1 -- 172.21.15.114:0/3295432081 <== mon.0 v2:172.21.15.114:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f787c003d10 con 0x55f382c69800 2024-04-23T10:39:42.314 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:39:42.310+0000 7f7878ff9640 1 -- 172.21.15.114:0/3295432081 <== mon.0 v2:172.21.15.114:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f787c004690 con 0x55f382c69800 2024-04-23T10:39:42.314 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:39:42.311+0000 7f788be37740 1 -- 172.21.15.114:0/3295432081 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55f382c69800 msgr2=0x55f382c814e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T10:39:42.314 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:39:42.311+0000 7f788be37740 1 --2- 172.21.15.114:0/3295432081 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55f382c69800 0x55f382c814e0 secure :-1 s=READY pgs=912 cs=0 l=1 rev1=1 crypto rx=0x7f787c00ebb0 tx=0x7f787c011000 comp rx=0 tx=0).stop 2024-04-23T10:39:42.314 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:39:42.311+0000 7f788be37740 1 -- 172.21.15.114:0/3295432081 shutdown_connections 2024-04-23T10:39:42.314 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:39:42.311+0000 7f788be37740 1 --2- 172.21.15.114:0/3295432081 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55f382c69800 0x55f382c814e0 unknown :-1 s=CLOSED pgs=912 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:39:42.314 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:39:42.311+0000 7f788be37740 1 --2- 172.21.15.114:0/3295432081 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x55f382c5ef40 0x55f382c80fa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:39:42.314 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:39:42.311+0000 7f788be37740 1 --2- 172.21.15.114:0/3295432081 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x55f382c5e620 0x55f382c68c80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:39:42.314 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:39:42.311+0000 7f788be37740 1 -- 172.21.15.114:0/3295432081 >> 172.21.15.114:0/3295432081 conn(0x55f382c5ab30 msgr2=0x55f382c59cb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T10:39:42.315 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:39:42.311+0000 7f788be37740 1 -- 172.21.15.114:0/3295432081 shutdown_connections 2024-04-23T10:39:42.315 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:39:42.311+0000 7f788be37740 1 -- 172.21.15.114:0/3295432081 wait complete. 2024-04-23T10:39:42.315 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:39:42.311+0000 7f788be37740 1 -- start start 2024-04-23T10:39:42.315 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:39:42.311+0000 7f788be37740 1 Processor -- start 2024-04-23T10:39:42.315 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:39:42.312+0000 7f788be37740 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x55f382c5e620 0x55f382c6d300 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T10:39:42.315 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:39:42.312+0000 7f788be37740 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55f382c5ef40 0x55f382c6d840 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T10:39:42.315 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:39:42.312+0000 7f788be37740 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x55f382c69800 0x55f382ce2c20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T10:39:42.315 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:39:42.312+0000 7f788be37740 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x7f78841ac270 con 0x55f382c5ef40 2024-04-23T10:39:42.315 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:39:42.312+0000 7f788be37740 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 -- 0x7f78841ad6e0 con 0x55f382c5e620 2024-04-23T10:39:42.315 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:39:42.312+0000 7f788be37740 1 -- --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_getmap magic: 0 -- 0x7f78841ac940 con 0x55f382c69800 2024-04-23T10:39:42.315 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:39:42.312+0000 7f788be37740 10 client.0 _ll_register_callbacks cb 0x7f78740b90a0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-23T10:39:42.315 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:39:42.312+0000 7f788259c640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55f382c5ef40 0x55f382c6d840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T10:39:42.315 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:39:42.312+0000 7f788259c640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55f382c5ef40 0x55f382c6d840 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.114:40300/0 (socket says 172.21.15.114:40300) 2024-04-23T10:39:42.315 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:39:42.312+0000 7f788259c640 1 -- 172.21.15.114:0/2151293299 learned_addr learned my addr 172.21.15.114:0/2151293299 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-04-23T10:39:42.315 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:39:42.312+0000 7f788359e640 1 --2- 172.21.15.114:0/2151293299 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x55f382c69800 0x55f382ce2c20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T10:39:42.316 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:39:42.312+0000 7f7882d9d640 1 --2- 172.21.15.114:0/2151293299 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x55f382c5e620 0x55f382c6d300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T10:39:42.316 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:39:42.312+0000 7f788259c640 1 -- 172.21.15.114:0/2151293299 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x55f382c69800 msgr2=0x55f382ce2c20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T10:39:42.316 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:39:42.312+0000 7f788259c640 1 --2- 172.21.15.114:0/2151293299 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x55f382c69800 0x55f382ce2c20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:39:42.316 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:39:42.312+0000 7f788259c640 1 -- 172.21.15.114:0/2151293299 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x55f382c5e620 msgr2=0x55f382c6d300 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T10:39:42.316 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:39:42.312+0000 7f788259c640 1 --2- 172.21.15.114:0/2151293299 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x55f382c5e620 0x55f382c6d300 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:39:42.316 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:39:42.312+0000 7f788259c640 1 -- 172.21.15.114:0/2151293299 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55f382ce3160 con 0x55f382c5ef40 2024-04-23T10:39:42.316 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:39:42.312+0000 7f7882d9d640 1 --2- 172.21.15.114:0/2151293299 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x55f382c5e620 0x55f382c6d300 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-23T10:39:42.316 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:39:42.312+0000 7f788259c640 1 --2- 172.21.15.114:0/2151293299 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55f382c5ef40 0x55f382c6d840 secure :-1 s=READY pgs=913 cs=0 l=1 rev1=1 crypto rx=0x7f78740bcd90 tx=0x7f7874005cb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7634b96fa6236a39 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T10:39:42.316 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:39:42.312+0000 7f786a7fc640 10 client.0 ms_handle_connect on v2:172.21.15.114:3300/0 2024-04-23T10:39:42.316 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:39:42.312+0000 7f786a7fc640 1 -- 172.21.15.114:0/2151293299 <== mon.0 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7874049040 con 0x55f382c5ef40 2024-04-23T10:39:42.316 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:39:42.312+0000 7f786a7fc640 1 -- 172.21.15.114:0/2151293299 <== mon.0 v2:172.21.15.114:3300/0 2 ==== osd_map(108..108 src has 1..108) ==== 8629+0+0 (secure 0 0 0) 0x7f787403bc50 con 0x55f382c5ef40 2024-04-23T10:39:42.316 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:39:42.312+0000 7f788be37740 1 -- 172.21.15.114:0/2151293299 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f787c003d10 con 0x55f382c5ef40 2024-04-23T10:39:42.316 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:39:42.312+0000 7f786a7fc640 1 -- 172.21.15.114:0/2151293299 <== mon.0 v2:172.21.15.114:3300/0 3 ==== mon_get_version_reply(handle=1 version=42) ==== 24+0+0 (secure 0 0 0) 0x7f78740a6120 con 0x55f382c5ef40 2024-04-23T10:39:42.316 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:39:42.312+0000 7f788be37740 10 client.17019 fetch_fsmap learned FSMap version 42 2024-04-23T10:39:42.316 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:39:42.312+0000 7f788be37740 1 -- 172.21.15.114:0/2151293299 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({config=0+,fsmap=42,monmap=4+}) -- 0x55f382ce3160 con 0x55f382c5ef40 2024-04-23T10:39:42.317 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:39:42.313+0000 7f786a7fc640 1 -- 172.21.15.114:0/2151293299 <== mon.0 v2:172.21.15.114:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f787c003d10 con 0x55f382c5ef40 2024-04-23T10:39:42.317 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:39:42.313+0000 7f786a7fc640 1 -- 172.21.15.114:0/2151293299 <== mon.0 v2:172.21.15.114:3300/0 5 ==== fsmap(e 42) ==== 8338+0+0 (secure 0 0 0) 0x7f78740c55e0 con 0x55f382c5ef40 2024-04-23T10:39:42.317 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:39:42.313+0000 7f788be37740 10 client.17019 fetch_fsmap finished waiting for FSMap version 42 2024-04-23T10:39:42.317 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:39:42.313+0000 7f788be37740 10 client.17019 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-04-23T10:39:42.317 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:39:42.313+0000 7f788be37740 20 client.17019 populate_metadata read hostname 'smithi114' 2024-04-23T10:39:42.317 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:39:42.313+0000 7f788be37740 1 --2- 172.21.15.114:0/2151293299 >> [v2:172.21.15.119:6838/937297340,v1:172.21.15.119:6839/937297340] conn(0x55f382ce46b0 0x55f382ce6ad0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T10:39:42.317 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:39:42.313+0000 7f788be37740 4 client.17019 mds_command: new command op to 24545 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-04-23T10:39:42.317 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:39:42.313+0000 7f788be37740 1 -- 172.21.15.114:0/2151293299 --> [v2:172.21.15.119:6838/937297340,v1:172.21.15.119:6839/937297340] -- command(tid 0: {"prefix": "flush journal"}) -- 0x55f382bcb200 con 0x55f382ce46b0 2024-04-23T10:39:42.317 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:39:42.313+0000 7f7882d9d640 1 --2- 172.21.15.114:0/2151293299 >> [v2:172.21.15.119:6838/937297340,v1:172.21.15.119:6839/937297340] conn(0x55f382ce46b0 0x55f382ce6ad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T10:39:42.317 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:39:42.313+0000 7f7882d9d640 1 --2- 172.21.15.114:0/2151293299 >> [v2:172.21.15.119:6838/937297340,v1:172.21.15.119:6839/937297340] conn(0x55f382ce46b0 0x55f382ce6ad0 crc :-1 s=READY pgs=179 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=eb118027e7a746d2 server_cookie=bd56ed9236b57341 in_seq=0 out_seq=0 2024-04-23T10:39:42.317 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:39:42.313+0000 7f786a7fc640 10 client.17019 ms_handle_connect on v2:172.21.15.119:6838/937297340 2024-04-23T10:39:42.327 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 23 10:39:42 smithi204 ceph-mon[32649]: pgmap v411: 97 pgs: 97 active+clean; 208 MiB data, 2.8 GiB used, 1.0 TiB / 1.0 TiB avail; 304 KiB/s rd, 501 KiB/s wr, 61 op/s 2024-04-23T10:39:42.347 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:39:42.347+0000 7f786a7fc640 1 -- 172.21.15.114:0/2151293299 <== mds.0 v2:172.21.15.119:6838/937297340 1 ==== command_reply(tid 0: 0 ) ==== 8+0+44 (crc 0 0 0) 0x55f382bcb200 con 0x55f382ce46b0 2024-04-23T10:39:42.347 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:39:42.347+0000 7f786a7fc640 10 client.17019 handle_command_reply: tid=0 2024-04-23T10:39:42.347 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:39:42.347+0000 7f788be37740 1 client.17019 shutdown 2024-04-23T10:39:42.347 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:39:42.348+0000 7f788be37740 1 -- 172.21.15.114:0/2151293299 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55f382c5ef40 msgr2=0x55f382c6d840 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T10:39:42.347 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:39:42.348+0000 7f788be37740 1 --2- 172.21.15.114:0/2151293299 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55f382c5ef40 0x55f382c6d840 secure :-1 s=READY pgs=913 cs=0 l=1 rev1=1 crypto rx=0x7f78740bcd90 tx=0x7f7874005cb0 comp rx=0 tx=0).stop 2024-04-23T10:39:42.347 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:39:42.348+0000 7f788be37740 1 -- 172.21.15.114:0/2151293299 shutdown_connections 2024-04-23T10:39:42.347 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:39:42.348+0000 7f788be37740 1 --2- 172.21.15.114:0/2151293299 >> [v2:172.21.15.119:6838/937297340,v1:172.21.15.119:6839/937297340] conn(0x55f382ce46b0 0x55f382ce6ad0 crc :-1 s=READY pgs=179 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:39:42.347 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:39:42.348+0000 7f788be37740 1 --2- 172.21.15.114:0/2151293299 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x55f382c69800 0x55f382ce2c20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:39:42.347 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:39:42.348+0000 7f788be37740 1 --2- 172.21.15.114:0/2151293299 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55f382c5ef40 0x55f382c6d840 unknown :-1 s=CLOSED pgs=913 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:39:42.347 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:39:42.348+0000 7f788be37740 1 --2- 172.21.15.114:0/2151293299 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x55f382c5e620 0x55f382c6d300 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:39:42.347 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:39:42.348+0000 7f786a7fc640 0 client.17019 ms_handle_reset on v2:172.21.15.119:6838/937297340 2024-04-23T10:39:42.348 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:39:42.348+0000 7f788be37740 1 -- 172.21.15.114:0/2151293299 >> 172.21.15.114:0/2151293299 conn(0x55f382c5ab30 msgr2=0x55f382c74010 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T10:39:42.348 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:39:42.348+0000 7f788be37740 1 -- 172.21.15.114:0/2151293299 shutdown_connections 2024-04-23T10:39:42.348 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:39:42.348+0000 7f788be37740 1 -- 172.21.15.114:0/2151293299 wait complete. 2024-04-23T10:39:42.348 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:39:42.348+0000 7f788be37740 20 client.17019 trim_cache size 0 max 16384 2024-04-23T10:39:42.348 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stdout:{ 2024-04-23T10:39:42.348 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stdout: "message": "", 2024-04-23T10:39:42.348 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stdout: "return_code": 0 2024-04-23T10:39:42.348 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stdout:} 2024-04-23T10:39:42.349 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:39:42.348+0000 7f7889120640 1 -- 172.21.15.114:0/1083379445 >> [v2:172.21.15.114:6800/1395693466,v1:172.21.15.114:6801/1395693466] conn(0x7f78700a3dc0 msgr2=0x7f78700a6280 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T10:39:42.349 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:39:42.348+0000 7f7889120640 1 --2- 172.21.15.114:0/1083379445 >> [v2:172.21.15.114:6800/1395693466,v1:172.21.15.114:6801/1395693466] conn(0x7f78700a3dc0 0x7f78700a6280 secure :-1 s=READY pgs=143 cs=0 l=1 rev1=1 crypto rx=0x7f787c005cc0 tx=0x7f787c005c50 comp rx=0 tx=0).stop 2024-04-23T10:39:42.349 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:39:42.348+0000 7f7889120640 1 -- 172.21.15.114:0/1083379445 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f7884106190 msgr2=0x7f7884198b70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T10:39:42.349 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:39:42.348+0000 7f7889120640 1 --2- 172.21.15.114:0/1083379445 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f7884106190 0x7f7884198b70 secure :-1 s=READY pgs=909 cs=0 l=1 rev1=1 crypto rx=0x7f7874031a90 tx=0x7f7874004270 comp rx=0 tx=0).stop 2024-04-23T10:39:42.349 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:39:42.348+0000 7f7889120640 1 -- 172.21.15.114:0/1083379445 shutdown_connections 2024-04-23T10:39:42.349 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:39:42.348+0000 7f7889120640 1 --2- 172.21.15.114:0/1083379445 >> [v2:172.21.15.114:6800/1395693466,v1:172.21.15.114:6801/1395693466] conn(0x7f78700a3dc0 0x7f78700a6280 unknown :-1 s=CLOSED pgs=143 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:39:42.349 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:39:42.348+0000 7f7889120640 1 --2- 172.21.15.114:0/1083379445 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f788419cef0 0x7f788419d3a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:39:42.349 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:39:42.348+0000 7f7889120640 1 --2- 172.21.15.114:0/1083379445 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f7884106190 0x7f7884198b70 unknown :-1 s=CLOSED pgs=909 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:39:42.350 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:39:42.348+0000 7f7889120640 1 --2- 172.21.15.114:0/1083379445 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f7884101310 0x7f7884198630 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:39:42.350 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:39:42.348+0000 7f7889120640 1 -- 172.21.15.114:0/1083379445 >> 172.21.15.114:0/1083379445 conn(0x7f7884077cc0 msgr2=0x7f78840ff050 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T10:39:42.350 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:39:42.349+0000 7f7889120640 1 -- 172.21.15.114:0/1083379445 shutdown_connections 2024-04-23T10:39:42.350 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:39:42.349+0000 7f7889120640 1 -- 172.21.15.114:0/1083379445 wait complete. 2024-04-23T10:39:42.492 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 23 10:39:42 smithi119 ceph-mon[32608]: pgmap v411: 97 pgs: 97 active+clean; 208 MiB data, 2.8 GiB used, 1.0 TiB / 1.0 TiB avail; 304 KiB/s rd, 501 KiB/s wr, 61 op/s 2024-04-23T10:39:42.503 INFO:journalctl@ceph.mon.a.smithi114.stdout:Apr 23 10:39:42 smithi114 ceph-mon[29255]: pgmap v411: 97 pgs: 97 active+clean; 208 MiB data, 2.8 GiB used, 1.0 TiB / 1.0 TiB avail; 304 KiB/s rd, 501 KiB/s wr, 61 op/s 2024-04-23T10:39:42.953 INFO:tasks.workunit.client.0.smithi114.stdout: 1 717449 4.95 MB/sec execute 591 sec latency 3639.419 ms 2024-04-23T10:39:43.953 INFO:tasks.workunit.client.0.smithi114.stdout: 1 717449 4.94 MB/sec execute 592 sec latency 4639.491 ms 2024-04-23T10:39:44.327 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 23 10:39:44 smithi204 ceph-mon[32649]: pgmap v412: 97 pgs: 97 active+clean; 208 MiB data, 2.8 GiB used, 1.0 TiB / 1.0 TiB avail; 414 KiB/s rd, 853 KiB/s wr, 80 op/s 2024-04-23T10:39:44.491 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 23 10:39:44 smithi119 ceph-mon[32608]: pgmap v412: 97 pgs: 97 active+clean; 208 MiB data, 2.8 GiB used, 1.0 TiB / 1.0 TiB avail; 414 KiB/s rd, 853 KiB/s wr, 80 op/s 2024-04-23T10:39:44.503 INFO:journalctl@ceph.mon.a.smithi114.stdout:Apr 23 10:39:44 smithi114 ceph-mon[29255]: pgmap v412: 97 pgs: 97 active+clean; 208 MiB data, 2.8 GiB used, 1.0 TiB / 1.0 TiB avail; 414 KiB/s rd, 853 KiB/s wr, 80 op/s 2024-04-23T10:39:44.953 INFO:tasks.workunit.client.0.smithi114.stdout: 1 718126 4.94 MB/sec execute 593 sec latency 4873.879 ms 2024-04-23T10:39:45.953 INFO:tasks.workunit.client.0.smithi114.stdout: 1 720669 4.95 MB/sec execute 594 sec latency 13.783 ms 2024-04-23T10:39:46.327 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 23 10:39:46 smithi204 ceph-mon[32649]: pgmap v413: 97 pgs: 97 active+clean; 208 MiB data, 2.8 GiB used, 1.0 TiB / 1.0 TiB avail; 298 KiB/s rd, 678 KiB/s wr, 53 op/s 2024-04-23T10:39:46.491 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 23 10:39:46 smithi119 ceph-mon[32608]: pgmap v413: 97 pgs: 97 active+clean; 208 MiB data, 2.8 GiB used, 1.0 TiB / 1.0 TiB avail; 298 KiB/s rd, 678 KiB/s wr, 53 op/s 2024-04-23T10:39:46.503 INFO:journalctl@ceph.mon.a.smithi114.stdout:Apr 23 10:39:46 smithi114 ceph-mon[29255]: pgmap v413: 97 pgs: 97 active+clean; 208 MiB data, 2.8 GiB used, 1.0 TiB / 1.0 TiB avail; 298 KiB/s rd, 678 KiB/s wr, 53 op/s 2024-04-23T10:39:46.953 INFO:tasks.workunit.client.0.smithi114.stdout: 1 720928 4.95 MB/sec execute 595 sec latency 887.107 ms 2024-04-23T10:39:47.953 INFO:tasks.workunit.client.0.smithi114.stdout: 1 720928 4.94 MB/sec execute 596 sec latency 1887.251 ms 2024-04-23T10:39:48.328 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 23 10:39:48 smithi204 ceph-mon[32649]: pgmap v414: 97 pgs: 97 active+clean; 212 MiB data, 2.8 GiB used, 1.0 TiB / 1.0 TiB avail; 504 KiB/s rd, 1.0 MiB/s wr, 99 op/s 2024-04-23T10:39:48.492 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 23 10:39:48 smithi119 ceph-mon[32608]: pgmap v414: 97 pgs: 97 active+clean; 212 MiB data, 2.8 GiB used, 1.0 TiB / 1.0 TiB avail; 504 KiB/s rd, 1.0 MiB/s wr, 99 op/s 2024-04-23T10:39:48.504 INFO:journalctl@ceph.mon.a.smithi114.stdout:Apr 23 10:39:48 smithi114 ceph-mon[29255]: pgmap v414: 97 pgs: 97 active+clean; 212 MiB data, 2.8 GiB used, 1.0 TiB / 1.0 TiB avail; 504 KiB/s rd, 1.0 MiB/s wr, 99 op/s 2024-04-23T10:39:48.953 INFO:tasks.workunit.client.0.smithi114.stdout: 1 720928 4.93 MB/sec execute 597 sec latency 2887.298 ms 2024-04-23T10:39:49.953 INFO:tasks.workunit.client.0.smithi114.stdout: 1 720995 4.92 MB/sec execute 598 sec latency 3119.908 ms 2024-04-23T10:39:50.328 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 23 10:39:50 smithi204 ceph-mon[32649]: pgmap v415: 97 pgs: 97 active+clean; 212 MiB data, 2.8 GiB used, 1.0 TiB / 1.0 TiB avail; 316 KiB/s rd, 703 KiB/s wr, 65 op/s 2024-04-23T10:39:50.491 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 23 10:39:50 smithi119 ceph-mon[32608]: pgmap v415: 97 pgs: 97 active+clean; 212 MiB data, 2.8 GiB used, 1.0 TiB / 1.0 TiB avail; 316 KiB/s rd, 703 KiB/s wr, 65 op/s 2024-04-23T10:39:50.504 INFO:journalctl@ceph.mon.a.smithi114.stdout:Apr 23 10:39:50 smithi114 ceph-mon[29255]: pgmap v415: 97 pgs: 97 active+clean; 212 MiB data, 2.8 GiB used, 1.0 TiB / 1.0 TiB avail; 316 KiB/s rd, 703 KiB/s wr, 65 op/s 2024-04-23T10:39:50.954 INFO:tasks.workunit.client.0.smithi114.stdout: 1 720995 4.92 MB/sec execute 599 sec latency 1654.305 ms 2024-04-23T10:39:51.954 INFO:tasks.workunit.client.0.smithi114.stdout: 1 cleanup 600 sec 2024-04-23T10:39:52.328 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 23 10:39:52 smithi204 ceph-mon[32649]: pgmap v416: 97 pgs: 97 active+clean; 212 MiB data, 2.8 GiB used, 1.0 TiB / 1.0 TiB avail; 316 KiB/s rd, 703 KiB/s wr, 65 op/s 2024-04-23T10:39:52.492 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 23 10:39:52 smithi119 ceph-mon[32608]: pgmap v416: 97 pgs: 97 active+clean; 212 MiB data, 2.8 GiB used, 1.0 TiB / 1.0 TiB avail; 316 KiB/s rd, 703 KiB/s wr, 65 op/s 2024-04-23T10:39:52.504 INFO:journalctl@ceph.mon.a.smithi114.stdout:Apr 23 10:39:52 smithi114 ceph-mon[29255]: pgmap v416: 97 pgs: 97 active+clean; 212 MiB data, 2.8 GiB used, 1.0 TiB / 1.0 TiB avail; 316 KiB/s rd, 703 KiB/s wr, 65 op/s 2024-04-23T10:39:52.954 INFO:tasks.workunit.client.0.smithi114.stdout: 1 cleanup 601 sec 2024-04-23T10:39:53.954 INFO:tasks.workunit.client.0.smithi114.stdout: 1 cleanup 602 sec 2024-04-23T10:39:54.328 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 23 10:39:54 smithi204 ceph-mon[32649]: pgmap v417: 97 pgs: 97 active+clean; 212 MiB data, 2.8 GiB used, 1.0 TiB / 1.0 TiB avail; 427 KiB/s rd, 987 KiB/s wr, 84 op/s 2024-04-23T10:39:54.394 INFO:tasks.workunit.client.0.smithi114.stdout: 0 cleanup 602 sec 2024-04-23T10:39:54.394 INFO:tasks.workunit.client.0.smithi114.stdout: 2024-04-23T10:39:54.394 INFO:tasks.workunit.client.0.smithi114.stdout: Operation Count AvgLat MaxLat 2024-04-23T10:39:54.394 INFO:tasks.workunit.client.0.smithi114.stdout: ---------------------------------------- 2024-04-23T10:39:54.394 INFO:tasks.workunit.client.0.smithi114.stdout: NTCreateX 93778 1.288 111.701 2024-04-23T10:39:54.394 INFO:tasks.workunit.client.0.smithi114.stdout: Close 68928 0.002 0.091 2024-04-23T10:39:54.394 INFO:tasks.workunit.client.0.smithi114.stdout: Rename 3976 56.185 4919.807 2024-04-23T10:39:54.394 INFO:tasks.workunit.client.0.smithi114.stdout: Unlink 18906 5.199 63699.111 2024-04-23T10:39:54.395 INFO:tasks.workunit.client.0.smithi114.stdout: Deltree 2 92.898 134.573 2024-04-23T10:39:54.395 INFO:tasks.workunit.client.0.smithi114.stdout: Mkdir 1 0.005 0.005 2024-04-23T10:39:54.395 INFO:tasks.workunit.client.0.smithi114.stdout: Qpathinfo 85040 1.257 10637.366 2024-04-23T10:39:54.395 INFO:tasks.workunit.client.0.smithi114.stdout: Qfileinfo 14905 0.001 0.025 2024-04-23T10:39:54.395 INFO:tasks.workunit.client.0.smithi114.stdout: Qfsinfo 15590 0.131 4.851 2024-04-23T10:39:54.395 INFO:tasks.workunit.client.0.smithi114.stdout: Sfileinfo 7645 2.895 21963.405 2024-04-23T10:39:54.395 INFO:tasks.workunit.client.0.smithi114.stdout: Find 32888 0.161 69.446 2024-04-23T10:39:54.395 INFO:tasks.workunit.client.0.smithi114.stdout: WriteX 46719 0.130 81.016 2024-04-23T10:39:54.395 INFO:tasks.workunit.client.0.smithi114.stdout: ReadX 147218 0.068 671.022 2024-04-23T10:39:54.395 INFO:tasks.workunit.client.0.smithi114.stdout: LockX 306 0.821 9.100 2024-04-23T10:39:54.395 INFO:tasks.workunit.client.0.smithi114.stdout: UnlockX 306 0.687 1.884 2024-04-23T10:39:54.396 INFO:tasks.workunit.client.0.smithi114.stdout: Flush 6567 2.742 81.897 2024-04-23T10:39:54.396 INFO:tasks.workunit.client.0.smithi114.stdout: 2024-04-23T10:39:54.396 INFO:tasks.workunit.client.0.smithi114.stdout:Throughput 4.91634 MB/sec 1 clients 1 procs max_latency=63699.123 ms 2024-04-23T10:39:54.396 INFO:tasks.workunit.client.0.smithi114.stderr:+ dbench 10 2024-04-23T10:39:54.397 INFO:tasks.workunit.client.0.smithi114.stdout:dbench version 4.00 - Copyright Andrew Tridgell 1999-2004 2024-04-23T10:39:54.397 INFO:tasks.workunit.client.0.smithi114.stdout: 2024-04-23T10:39:54.397 INFO:tasks.workunit.client.0.smithi114.stdout:Running for 600 seconds with load '/usr/share/dbench/client.txt' and minimum warmup 120 secs 2024-04-23T10:39:54.397 INFO:tasks.workunit.client.0.smithi114.stdout:8 of 10 processes prepared for launch 0 sec 2024-04-23T10:39:54.492 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 23 10:39:54 smithi119 ceph-mon[32608]: pgmap v417: 97 pgs: 97 active+clean; 212 MiB data, 2.8 GiB used, 1.0 TiB / 1.0 TiB avail; 427 KiB/s rd, 987 KiB/s wr, 84 op/s 2024-04-23T10:39:54.495 INFO:tasks.workunit.client.0.smithi114.stdout:10 of 10 processes prepared for launch 0 sec 2024-04-23T10:39:54.495 INFO:tasks.workunit.client.0.smithi114.stdout:releasing clients 2024-04-23T10:39:54.503 INFO:journalctl@ceph.mon.a.smithi114.stdout:Apr 23 10:39:54 smithi114 ceph-mon[29255]: pgmap v417: 97 pgs: 97 active+clean; 212 MiB data, 2.8 GiB used, 1.0 TiB / 1.0 TiB avail; 427 KiB/s rd, 987 KiB/s wr, 84 op/s 2024-04-23T10:39:55.440 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:39:55.439+0000 7fdb94635640 1 -- 172.21.15.114:0/2631993038 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fdb8c101a70 msgr2=0x7fdb8c105ab0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T10:39:55.440 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:39:55.439+0000 7fdb94635640 1 --2- 172.21.15.114:0/2631993038 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fdb8c101a70 0x7fdb8c105ab0 secure :-1 s=READY pgs=914 cs=0 l=1 rev1=1 crypto rx=0x7fdb7c009900 tx=0x7fdb7c02a410 comp rx=0 tx=0).stop 2024-04-23T10:39:55.440 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:39:55.439+0000 7fdb94635640 1 -- 172.21.15.114:0/2631993038 shutdown_connections 2024-04-23T10:39:55.440 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:39:55.439+0000 7fdb94635640 1 --2- 172.21.15.114:0/2631993038 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fdb8c106190 0x7fdb8c10dd60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:39:55.440 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:39:55.439+0000 7fdb94635640 1 --2- 172.21.15.114:0/2631993038 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fdb8c101a70 0x7fdb8c105ab0 unknown :-1 s=CLOSED pgs=914 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:39:55.440 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:39:55.439+0000 7fdb94635640 1 --2- 172.21.15.114:0/2631993038 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fdb8c1010c0 0x7fdb8c1014a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:39:55.441 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:39:55.439+0000 7fdb94635640 1 -- 172.21.15.114:0/2631993038 >> 172.21.15.114:0/2631993038 conn(0x7fdb8c077f60 msgr2=0x7fdb8c078370 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T10:39:55.441 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:39:55.439+0000 7fdb94635640 1 -- 172.21.15.114:0/2631993038 shutdown_connections 2024-04-23T10:39:55.441 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:39:55.439+0000 7fdb94635640 1 -- 172.21.15.114:0/2631993038 wait complete. 2024-04-23T10:39:55.441 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:39:55.440+0000 7fdb94635640 1 Processor -- start 2024-04-23T10:39:55.441 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:39:55.440+0000 7fdb94635640 1 -- start start 2024-04-23T10:39:55.441 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:39:55.440+0000 7fdb94635640 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fdb8c1010c0 0x7fdb8c1a06b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T10:39:55.441 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:39:55.440+0000 7fdb94635640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fdb8c106190 0x7fdb8c1a0bf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T10:39:55.441 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:39:55.440+0000 7fdb94635640 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fdb8c19a770 0x7fdb8c19abf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T10:39:55.441 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:39:55.440+0000 7fdb94635640 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x7fdb8c111100 con 0x7fdb8c106190 2024-04-23T10:39:55.441 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:39:55.440+0000 7fdb94635640 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 -- 0x7fdb8c110f80 con 0x7fdb8c19a770 2024-04-23T10:39:55.441 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:39:55.440+0000 7fdb94635640 1 -- --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_getmap magic: 0 -- 0x7fdb8c111280 con 0x7fdb8c1010c0 2024-04-23T10:39:55.441 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:39:55.440+0000 7fdb91bd0640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fdb8c106190 0x7fdb8c1a0bf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T10:39:55.441 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:39:55.440+0000 7fdb91bd0640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fdb8c106190 0x7fdb8c1a0bf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.114:48152/0 (socket says 172.21.15.114:48152) 2024-04-23T10:39:55.442 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:39:55.440+0000 7fdb91bd0640 1 -- 172.21.15.114:0/2108536228 learned_addr learned my addr 172.21.15.114:0/2108536228 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-04-23T10:39:55.442 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:39:55.440+0000 7fdb92bd2640 1 --2- 172.21.15.114:0/2108536228 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fdb8c19a770 0x7fdb8c19abf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T10:39:55.442 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:39:55.440+0000 7fdb923d1640 1 --2- 172.21.15.114:0/2108536228 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fdb8c1010c0 0x7fdb8c1a06b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T10:39:55.442 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:39:55.441+0000 7fdb91bd0640 1 -- 172.21.15.114:0/2108536228 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fdb8c1010c0 msgr2=0x7fdb8c1a06b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T10:39:55.442 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:39:55.441+0000 7fdb91bd0640 1 --2- 172.21.15.114:0/2108536228 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fdb8c1010c0 0x7fdb8c1a06b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:39:55.442 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:39:55.441+0000 7fdb91bd0640 1 -- 172.21.15.114:0/2108536228 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fdb8c19a770 msgr2=0x7fdb8c19abf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T10:39:55.442 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:39:55.441+0000 7fdb91bd0640 1 --2- 172.21.15.114:0/2108536228 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fdb8c19a770 0x7fdb8c19abf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:39:55.442 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:39:55.441+0000 7fdb91bd0640 1 -- 172.21.15.114:0/2108536228 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fdb8c19b370 con 0x7fdb8c106190 2024-04-23T10:39:55.442 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:39:55.441+0000 7fdb91bd0640 1 --2- 172.21.15.114:0/2108536228 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fdb8c106190 0x7fdb8c1a0bf0 secure :-1 s=READY pgs=915 cs=0 l=1 rev1=1 crypto rx=0x7fdb7c005ab0 tx=0x7fdb7c003b90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T10:39:55.442 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:39:55.441+0000 7fdb837fe640 1 -- 172.21.15.114:0/2108536228 <== mon.0 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdb7c03f070 con 0x7fdb8c106190 2024-04-23T10:39:55.442 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:39:55.441+0000 7fdb837fe640 1 -- 172.21.15.114:0/2108536228 <== mon.0 v2:172.21.15.114:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fdb7c031b50 con 0x7fdb8c106190 2024-04-23T10:39:55.442 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:39:55.441+0000 7fdb94635640 1 -- 172.21.15.114:0/2108536228 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fdb8c19b600 con 0x7fdb8c106190 2024-04-23T10:39:55.442 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:39:55.441+0000 7fdb837fe640 1 -- 172.21.15.114:0/2108536228 <== mon.0 v2:172.21.15.114:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdb7c035030 con 0x7fdb8c106190 2024-04-23T10:39:55.442 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:39:55.441+0000 7fdb94635640 1 -- 172.21.15.114:0/2108536228 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fdb8c1a7740 con 0x7fdb8c106190 2024-04-23T10:39:55.448 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:39:55.443+0000 7fdb837fe640 1 -- 172.21.15.114:0/2108536228 <== mon.0 v2:172.21.15.114:3300/0 4 ==== mgrmap(e 26) ==== 138778+0+0 (secure 0 0 0) 0x7fdb7c042070 con 0x7fdb8c106190 2024-04-23T10:39:55.449 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:39:55.444+0000 7fdb9742f740 1 Processor -- start 2024-04-23T10:39:55.449 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:39:55.444+0000 7fdb9742f740 1 -- start start 2024-04-23T10:39:55.449 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:39:55.444+0000 7fdb9742f740 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x55b6bc5c0620 0x55b6bc5c0a00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T10:39:55.449 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:39:55.444+0000 7fdb9742f740 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x55b6bc5c0f40 0x55b6bc5cb2c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T10:39:55.449 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:39:55.444+0000 7fdb9742f740 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55b6bc5cb800 0x55b6bc5cdbf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T10:39:55.449 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:39:55.444+0000 7fdb9742f740 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x55b6bc525970 con 0x55b6bc5cb800 2024-04-23T10:39:55.449 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:39:55.444+0000 7fdb9742f740 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 -- 0x55b6bc4c24a0 con 0x55b6bc5c0f40 2024-04-23T10:39:55.449 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:39:55.444+0000 7fdb9742f740 1 -- --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_getmap magic: 0 -- 0x55b6bc521930 con 0x55b6bc5c0620 2024-04-23T10:39:55.449 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:39:55.444+0000 7fdb837fe640 1 --2- 172.21.15.114:0/2108536228 >> [v2:172.21.15.114:6800/1395693466,v1:172.21.15.114:6801/1395693466] conn(0x7fdb780a3dc0 0x7fdb780a6280 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T10:39:55.449 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:39:55.445+0000 7fdb91bd0640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55b6bc5cb800 0x55b6bc5cdbf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T10:39:55.449 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:39:55.445+0000 7fdb923d1640 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x55b6bc5c0f40 0x55b6bc5cb2c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T10:39:55.449 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:39:55.445+0000 7fdb923d1640 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x55b6bc5c0f40 0x55b6bc5cb2c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.119:3300/0 says I am v2:172.21.15.114:60274/0 (socket says 172.21.15.114:60274) 2024-04-23T10:39:55.450 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:39:55.445+0000 7fdb923d1640 1 -- 172.21.15.114:0/2344167909 learned_addr learned my addr 172.21.15.114:0/2344167909 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-04-23T10:39:55.450 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:39:55.445+0000 7fdb92bd2640 1 --2- 172.21.15.114:0/2108536228 >> [v2:172.21.15.114:6800/1395693466,v1:172.21.15.114:6801/1395693466] conn(0x7fdb780a3dc0 0x7fdb780a6280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T10:39:55.450 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:39:55.445+0000 7fdb92bd2640 1 --2- 172.21.15.114:0/2344167909 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x55b6bc5c0620 0x55b6bc5c0a00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T10:39:55.450 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:39:55.445+0000 7fdb837fe640 1 -- 172.21.15.114:0/2108536228 <== mon.0 v2:172.21.15.114:3300/0 5 ==== osd_map(108..108 src has 1..108) ==== 8629+0+0 (secure 0 0 0) 0x7fdb7c0f0f10 con 0x7fdb8c106190 2024-04-23T10:39:55.450 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:39:55.445+0000 7fdb923d1640 1 -- 172.21.15.114:0/2344167909 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x55b6bc5c0620 msgr2=0x55b6bc5c0a00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T10:39:55.450 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:39:55.445+0000 7fdb923d1640 1 --2- 172.21.15.114:0/2344167909 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x55b6bc5c0620 0x55b6bc5c0a00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:39:55.450 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:39:55.445+0000 7fdb923d1640 1 -- 172.21.15.114:0/2344167909 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55b6bc5cb800 msgr2=0x55b6bc5cdbf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T10:39:55.450 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:39:55.445+0000 7fdb923d1640 1 --2- 172.21.15.114:0/2344167909 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55b6bc5cb800 0x55b6bc5cdbf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:39:55.450 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:39:55.445+0000 7fdb923d1640 1 -- 172.21.15.114:0/2344167909 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55b6bc238210 con 0x55b6bc5c0f40 2024-04-23T10:39:55.450 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:39:55.445+0000 7fdb923d1640 1 --2- 172.21.15.114:0/2344167909 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x55b6bc5c0f40 0x55b6bc5cb2c0 secure :-1 s=READY pgs=100 cs=0 l=1 rev1=1 crypto rx=0x7fdb8800ac20 tx=0x7fdb8800f790 comp rx=0 tx=0).ready entity=mon.2 client_cookie=757511cb5bf4080d server_cookie=0 in_seq=0 out_seq=0 2024-04-23T10:39:55.450 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:39:55.445+0000 7fdb92bd2640 1 --2- 172.21.15.114:0/2108536228 >> [v2:172.21.15.114:6800/1395693466,v1:172.21.15.114:6801/1395693466] conn(0x7fdb780a3dc0 0x7fdb780a6280 secure :-1 s=READY pgs=144 cs=0 l=1 rev1=1 crypto rx=0x7fdb84009fd0 tx=0x7fdb84004ab0 comp rx=0 tx=0).ready entity=mgr.15291 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T10:39:55.450 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:39:55.446+0000 7fdb80bfe640 1 -- 172.21.15.114:0/2344167909 <== mon.2 v2:172.21.15.119:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdb8801d070 con 0x55b6bc5c0f40 2024-04-23T10:39:55.450 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:39:55.446+0000 7fdb80bfe640 1 -- 172.21.15.114:0/2344167909 <== mon.2 v2:172.21.15.119:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fdb88018af0 con 0x55b6bc5c0f40 2024-04-23T10:39:55.451 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:39:55.446+0000 7fdb80bfe640 1 -- 172.21.15.114:0/2344167909 <== mon.2 v2:172.21.15.119:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdb88010b40 con 0x55b6bc5c0f40 2024-04-23T10:39:55.451 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:39:55.446+0000 7fdb9742f740 1 -- 172.21.15.114:0/2344167909 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x55b6bc5c0f40 msgr2=0x55b6bc5cb2c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T10:39:55.451 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:39:55.446+0000 7fdb9742f740 1 --2- 172.21.15.114:0/2344167909 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x55b6bc5c0f40 0x55b6bc5cb2c0 secure :-1 s=READY pgs=100 cs=0 l=1 rev1=1 crypto rx=0x7fdb8800ac20 tx=0x7fdb8800f790 comp rx=0 tx=0).stop 2024-04-23T10:39:55.451 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:39:55.446+0000 7fdb9742f740 1 -- 172.21.15.114:0/2344167909 shutdown_connections 2024-04-23T10:39:55.451 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:39:55.446+0000 7fdb9742f740 1 --2- 172.21.15.114:0/2344167909 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55b6bc5cb800 0x55b6bc5cdbf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:39:55.451 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:39:55.446+0000 7fdb9742f740 1 --2- 172.21.15.114:0/2344167909 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x55b6bc5c0f40 0x55b6bc5cb2c0 unknown :-1 s=CLOSED pgs=100 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:39:55.451 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:39:55.446+0000 7fdb9742f740 1 --2- 172.21.15.114:0/2344167909 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x55b6bc5c0620 0x55b6bc5c0a00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:39:55.451 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:39:55.446+0000 7fdb9742f740 1 -- 172.21.15.114:0/2344167909 >> 172.21.15.114:0/2344167909 conn(0x55b6bc5bcb30 msgr2=0x55b6bc5bcf10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T10:39:55.451 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:39:55.446+0000 7fdb9742f740 1 -- 172.21.15.114:0/2344167909 shutdown_connections 2024-04-23T10:39:55.451 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:39:55.447+0000 7fdb9742f740 1 -- 172.21.15.114:0/2344167909 wait complete. 2024-04-23T10:39:55.451 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:39:55.447+0000 7fdb9742f740 1 -- start start 2024-04-23T10:39:55.451 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:39:55.447+0000 7fdb9742f740 1 Processor -- start 2024-04-23T10:39:55.451 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:39:55.447+0000 7fdb9742f740 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x55b6bc5c0620 0x55b6bc5cb1a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T10:39:55.451 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:39:55.447+0000 7fdb9742f740 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55b6bc5c0f40 0x55b6bc5c8490 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T10:39:55.452 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:39:55.447+0000 7fdb9742f740 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x55b6bc5cb800 0x55b6bc5c8a30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T10:39:55.452 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:39:55.447+0000 7fdb9742f740 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x55b6bc5ce430 con 0x55b6bc5c0f40 2024-04-23T10:39:55.452 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:39:55.447+0000 7fdb9742f740 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 -- 0x55b6bc4c00b0 con 0x55b6bc5cb800 2024-04-23T10:39:55.452 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:39:55.447+0000 7fdb9742f740 1 -- --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_getmap magic: 0 -- 0x55b6bc5ce5b0 con 0x55b6bc5c0620 2024-04-23T10:39:55.452 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:39:55.447+0000 7fdb9742f740 10 client.0 _ll_register_callbacks cb 0x55b6bc4add30 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-23T10:39:55.452 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:39:55.447+0000 7fdb91bd0640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55b6bc5c0f40 0x55b6bc5c8490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T10:39:55.452 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:39:55.447+0000 7fdb91bd0640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55b6bc5c0f40 0x55b6bc5c8490 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.114:48170/0 (socket says 172.21.15.114:48170) 2024-04-23T10:39:55.452 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:39:55.447+0000 7fdb91bd0640 1 -- 172.21.15.114:0/1771937076 learned_addr learned my addr 172.21.15.114:0/1771937076 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-04-23T10:39:55.452 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:39:55.447+0000 7fdb92bd2640 1 --2- 172.21.15.114:0/1771937076 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x55b6bc5cb800 0x55b6bc5c8a30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T10:39:55.452 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:39:55.447+0000 7fdb923d1640 1 --2- 172.21.15.114:0/1771937076 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x55b6bc5c0620 0x55b6bc5cb1a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T10:39:55.452 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:39:55.448+0000 7fdb91bd0640 1 -- 172.21.15.114:0/1771937076 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x55b6bc5c0620 msgr2=0x55b6bc5cb1a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T10:39:55.452 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:39:55.448+0000 7fdb91bd0640 1 --2- 172.21.15.114:0/1771937076 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x55b6bc5c0620 0x55b6bc5cb1a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:39:55.452 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:39:55.448+0000 7fdb91bd0640 1 -- 172.21.15.114:0/1771937076 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x55b6bc5cb800 msgr2=0x55b6bc5c8a30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T10:39:55.452 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:39:55.448+0000 7fdb91bd0640 1 --2- 172.21.15.114:0/1771937076 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x55b6bc5cb800 0x55b6bc5c8a30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:39:55.452 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:39:55.448+0000 7fdb91bd0640 1 -- 172.21.15.114:0/1771937076 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55b6bc304620 con 0x55b6bc5c0f40 2024-04-23T10:39:55.453 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:39:55.448+0000 7fdb91bd0640 1 --2- 172.21.15.114:0/1771937076 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55b6bc5c0f40 0x55b6bc5c8490 secure :-1 s=READY pgs=916 cs=0 l=1 rev1=1 crypto rx=0x7fdb7c0bce20 tx=0x7fdb7c031f50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7e1441f1cf9baf86 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T10:39:55.453 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:39:55.448+0000 7fdb75ffb640 10 client.0 ms_handle_connect on v2:172.21.15.114:3300/0 2024-04-23T10:39:55.453 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:39:55.448+0000 7fdb75ffb640 1 -- 172.21.15.114:0/1771937076 <== mon.0 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdb7c0d5020 con 0x55b6bc5c0f40 2024-04-23T10:39:55.453 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:39:55.448+0000 7fdb75ffb640 1 -- 172.21.15.114:0/1771937076 <== mon.0 v2:172.21.15.114:3300/0 2 ==== osd_map(108..108 src has 1..108) ==== 8629+0+0 (secure 0 0 0) 0x7fdb7c0b9500 con 0x55b6bc5c0f40 2024-04-23T10:39:55.453 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:39:55.448+0000 7fdb9742f740 1 -- 172.21.15.114:0/1771937076 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fdb88018af0 con 0x55b6bc5c0f40 2024-04-23T10:39:55.453 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:39:55.448+0000 7fdb75ffb640 1 -- 172.21.15.114:0/1771937076 <== mon.0 v2:172.21.15.114:3300/0 3 ==== mon_get_version_reply(handle=1 version=42) ==== 24+0+0 (secure 0 0 0) 0x7fdb7c049070 con 0x55b6bc5c0f40 2024-04-23T10:39:55.453 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:39:55.448+0000 7fdb9742f740 10 client.17031 fetch_fsmap learned FSMap version 42 2024-04-23T10:39:55.453 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:39:55.448+0000 7fdb9742f740 1 -- 172.21.15.114:0/1771937076 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({config=0+,fsmap=42,monmap=4+}) -- 0x55b6bc5da3a0 con 0x55b6bc5c0f40 2024-04-23T10:39:55.453 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:39:55.448+0000 7fdb75ffb640 1 -- 172.21.15.114:0/1771937076 <== mon.0 v2:172.21.15.114:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fdb88018af0 con 0x55b6bc5c0f40 2024-04-23T10:39:55.453 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:39:55.448+0000 7fdb75ffb640 1 -- 172.21.15.114:0/1771937076 <== mon.0 v2:172.21.15.114:3300/0 5 ==== fsmap(e 42) ==== 8338+0+0 (secure 0 0 0) 0x7fdb7c0eece0 con 0x55b6bc5c0f40 2024-04-23T10:39:55.453 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:39:55.448+0000 7fdb9742f740 10 client.17031 fetch_fsmap finished waiting for FSMap version 42 2024-04-23T10:39:55.453 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:39:55.448+0000 7fdb9742f740 10 client.17031 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-04-23T10:39:55.453 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:39:55.448+0000 7fdb9742f740 20 client.17031 populate_metadata read hostname 'smithi114' 2024-04-23T10:39:55.453 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:39:55.448+0000 7fdb9742f740 1 --2- 172.21.15.114:0/1771937076 >> [v2:172.21.15.119:6838/937297340,v1:172.21.15.119:6839/937297340] conn(0x55b6bc5db850 0x55b6bc5ddc80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T10:39:55.454 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:39:55.448+0000 7fdb9742f740 4 client.17031 mds_command: new command op to 24545 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-23T10:39:55.454 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:39:55.448+0000 7fdb9742f740 1 -- 172.21.15.114:0/1771937076 --> [v2:172.21.15.119:6838/937297340,v1:172.21.15.119:6839/937297340] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55b6bc533390 con 0x55b6bc5db850 2024-04-23T10:39:55.454 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:39:55.449+0000 7fdb923d1640 1 --2- 172.21.15.114:0/1771937076 >> [v2:172.21.15.119:6838/937297340,v1:172.21.15.119:6839/937297340] conn(0x55b6bc5db850 0x55b6bc5ddc80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T10:39:55.454 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:39:55.449+0000 7fdb923d1640 1 --2- 172.21.15.114:0/1771937076 >> [v2:172.21.15.119:6838/937297340,v1:172.21.15.119:6839/937297340] conn(0x55b6bc5db850 0x55b6bc5ddc80 crc :-1 s=READY pgs=180 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=cb34d8eec84c215e server_cookie=7bf6e6b62d6fdba8 in_seq=0 out_seq=0 2024-04-23T10:39:55.454 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:39:55.449+0000 7fdb75ffb640 10 client.17031 ms_handle_connect on v2:172.21.15.119:6838/937297340 2024-04-23T10:39:55.455 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:39:55.451+0000 7fdb75ffb640 1 -- 172.21.15.114:0/1771937076 <== mds.0 v2:172.21.15.119:6838/937297340 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x55b6bc533390 con 0x55b6bc5db850 2024-04-23T10:39:55.455 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:39:55.451+0000 7fdb75ffb640 10 client.17031 handle_command_reply: tid=0 2024-04-23T10:39:55.455 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:39:55.452+0000 7fdb9742f740 1 client.17031 shutdown 2024-04-23T10:39:55.455 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:39:55.452+0000 7fdb9742f740 1 -- 172.21.15.114:0/1771937076 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55b6bc5c0f40 msgr2=0x55b6bc5c8490 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T10:39:55.455 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:39:55.452+0000 7fdb9742f740 1 --2- 172.21.15.114:0/1771937076 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55b6bc5c0f40 0x55b6bc5c8490 secure :-1 s=READY pgs=916 cs=0 l=1 rev1=1 crypto rx=0x7fdb7c0bce20 tx=0x7fdb7c031f50 comp rx=0 tx=0).stop 2024-04-23T10:39:55.455 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:39:55.452+0000 7fdb9742f740 1 -- 172.21.15.114:0/1771937076 shutdown_connections 2024-04-23T10:39:55.455 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:39:55.452+0000 7fdb9742f740 1 --2- 172.21.15.114:0/1771937076 >> [v2:172.21.15.119:6838/937297340,v1:172.21.15.119:6839/937297340] conn(0x55b6bc5db850 0x55b6bc5ddc80 crc :-1 s=READY pgs=180 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:39:55.456 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:39:55.452+0000 7fdb9742f740 1 --2- 172.21.15.114:0/1771937076 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x55b6bc5cb800 0x55b6bc5c8a30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:39:55.456 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:39:55.452+0000 7fdb9742f740 1 --2- 172.21.15.114:0/1771937076 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55b6bc5c0f40 0x55b6bc5c8490 unknown :-1 s=CLOSED pgs=916 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:39:55.456 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:39:55.452+0000 7fdb75ffb640 0 client.17031 ms_handle_reset on v2:172.21.15.119:6838/937297340 2024-04-23T10:39:55.456 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:39:55.452+0000 7fdb9742f740 1 --2- 172.21.15.114:0/1771937076 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x55b6bc5c0620 0x55b6bc5cb1a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:39:55.456 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:39:55.452+0000 7fdb9742f740 1 -- 172.21.15.114:0/1771937076 >> 172.21.15.114:0/1771937076 conn(0x55b6bc5bcb30 msgr2=0x55b6bc5c2860 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T10:39:55.456 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:39:55.452+0000 7fdb9742f740 1 -- 172.21.15.114:0/1771937076 shutdown_connections 2024-04-23T10:39:55.456 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:39:55.452+0000 7fdb9742f740 1 -- 172.21.15.114:0/1771937076 wait complete. 2024-04-23T10:39:55.456 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:39:55.452+0000 7fdb9742f740 20 client.17031 trim_cache size 0 max 16384 2024-04-23T10:39:55.470 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:39:55.467+0000 7fdb9742f740 1 Processor -- start 2024-04-23T10:39:55.470 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:39:55.467+0000 7fdb9742f740 1 -- start start 2024-04-23T10:39:55.470 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:39:55.467+0000 7fdb9742f740 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55b6bc5c0620 0x55b6bc5d72f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T10:39:55.470 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:39:55.467+0000 7fdb9742f740 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x55b6bc5c0f40 0x55b6bc5c8450 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T10:39:55.471 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:39:55.467+0000 7fdb9742f740 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x55b6bc5cb800 0x55b6bc5bbb30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T10:39:55.471 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:39:55.467+0000 7fdb9742f740 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x7fdb8c1b2780 con 0x55b6bc5c0620 2024-04-23T10:39:55.471 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:39:55.467+0000 7fdb9742f740 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 -- 0x7fdb8c1b0050 con 0x55b6bc5c0f40 2024-04-23T10:39:55.471 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:39:55.467+0000 7fdb9742f740 1 -- --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_getmap magic: 0 -- 0x7fdb8c1adec0 con 0x55b6bc5cb800 2024-04-23T10:39:55.471 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:39:55.467+0000 7fdb923d1640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55b6bc5c0620 0x55b6bc5d72f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T10:39:55.471 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:39:55.467+0000 7fdb923d1640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55b6bc5c0620 0x55b6bc5d72f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.114:48172/0 (socket says 172.21.15.114:48172) 2024-04-23T10:39:55.471 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:39:55.467+0000 7fdb923d1640 1 -- 172.21.15.114:0/3622103000 learned_addr learned my addr 172.21.15.114:0/3622103000 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-04-23T10:39:55.471 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:39:55.467+0000 7fdb91bd0640 1 --2- 172.21.15.114:0/3622103000 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x55b6bc5c0f40 0x55b6bc5c8450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T10:39:55.471 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:39:55.468+0000 7fdb923d1640 1 -- 172.21.15.114:0/3622103000 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x55b6bc5cb800 msgr2=0x55b6bc5bbb30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T10:39:55.471 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:39:55.468+0000 7fdb923d1640 1 --2- 172.21.15.114:0/3622103000 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x55b6bc5cb800 0x55b6bc5bbb30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:39:55.471 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:39:55.468+0000 7fdb923d1640 1 -- 172.21.15.114:0/3622103000 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x55b6bc5c0f40 msgr2=0x55b6bc5c8450 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T10:39:55.471 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:39:55.468+0000 7fdb923d1640 1 --2- 172.21.15.114:0/3622103000 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x55b6bc5c0f40 0x55b6bc5c8450 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:39:55.471 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:39:55.468+0000 7fdb923d1640 1 -- 172.21.15.114:0/3622103000 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55b6bc5c8da0 con 0x55b6bc5c0620 2024-04-23T10:39:55.472 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:39:55.468+0000 7fdb923d1640 1 --2- 172.21.15.114:0/3622103000 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55b6bc5c0620 0x55b6bc5d72f0 secure :-1 s=READY pgs=917 cs=0 l=1 rev1=1 crypto rx=0x7fdb8800ac20 tx=0x7fdb8800a4b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=dcb693743a3eb010 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T10:39:55.472 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:39:55.468+0000 7fdb77fff640 1 -- 172.21.15.114:0/3622103000 <== mon.0 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdb8802c030 con 0x55b6bc5c0620 2024-04-23T10:39:55.472 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:39:55.468+0000 7fdb77fff640 1 -- 172.21.15.114:0/3622103000 <== mon.0 v2:172.21.15.114:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fdb8801e040 con 0x55b6bc5c0620 2024-04-23T10:39:55.472 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:39:55.468+0000 7fdb77fff640 1 -- 172.21.15.114:0/3622103000 <== mon.0 v2:172.21.15.114:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdb88028e60 con 0x55b6bc5c0620 2024-04-23T10:39:55.472 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:39:55.469+0000 7fdb9742f740 1 -- 172.21.15.114:0/3622103000 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55b6bc5c0620 msgr2=0x55b6bc5d72f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T10:39:55.472 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:39:55.469+0000 7fdb9742f740 1 --2- 172.21.15.114:0/3622103000 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55b6bc5c0620 0x55b6bc5d72f0 secure :-1 s=READY pgs=917 cs=0 l=1 rev1=1 crypto rx=0x7fdb8800ac20 tx=0x7fdb8800a4b0 comp rx=0 tx=0).stop 2024-04-23T10:39:55.472 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:39:55.469+0000 7fdb9742f740 1 -- 172.21.15.114:0/3622103000 shutdown_connections 2024-04-23T10:39:55.472 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:39:55.469+0000 7fdb9742f740 1 --2- 172.21.15.114:0/3622103000 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x55b6bc5cb800 0x55b6bc5bbb30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:39:55.472 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:39:55.469+0000 7fdb9742f740 1 --2- 172.21.15.114:0/3622103000 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x55b6bc5c0f40 0x55b6bc5c8450 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:39:55.472 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:39:55.469+0000 7fdb9742f740 1 --2- 172.21.15.114:0/3622103000 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55b6bc5c0620 0x55b6bc5d72f0 unknown :-1 s=CLOSED pgs=917 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:39:55.472 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:39:55.469+0000 7fdb9742f740 1 -- 172.21.15.114:0/3622103000 >> 172.21.15.114:0/3622103000 conn(0x55b6bc5bcb30 msgr2=0x55b6bc5bb420 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T10:39:55.472 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:39:55.469+0000 7fdb9742f740 1 -- 172.21.15.114:0/3622103000 shutdown_connections 2024-04-23T10:39:55.472 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:39:55.469+0000 7fdb9742f740 1 -- 172.21.15.114:0/3622103000 wait complete. 2024-04-23T10:39:55.472 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:39:55.469+0000 7fdb9742f740 1 -- start start 2024-04-23T10:39:55.472 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:39:55.469+0000 7fdb9742f740 1 Processor -- start 2024-04-23T10:39:55.473 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:39:55.469+0000 7fdb9742f740 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x55b6bc5c0620 0x55b6bc5d3940 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T10:39:55.473 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:39:55.469+0000 7fdb9742f740 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55b6bc5c0f40 0x55b6bc5d0cd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T10:39:55.473 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:39:55.469+0000 7fdb9742f740 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x55b6bc5cb800 0x55b6bc5d1210 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T10:39:55.473 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:39:55.469+0000 7fdb9742f740 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x7fdb8c1ae510 con 0x55b6bc5c0f40 2024-04-23T10:39:55.473 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:39:55.469+0000 7fdb9742f740 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 -- 0x7fdb8c1af980 con 0x55b6bc5c0620 2024-04-23T10:39:55.473 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:39:55.469+0000 7fdb9742f740 1 -- --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_getmap magic: 0 -- 0x7fdb8c1aebe0 con 0x55b6bc5cb800 2024-04-23T10:39:55.473 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:39:55.469+0000 7fdb9742f740 10 client.0 _ll_register_callbacks cb 0x7fdb7c009560 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-23T10:39:55.473 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:39:55.470+0000 7fdb91bd0640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55b6bc5c0f40 0x55b6bc5d0cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T10:39:55.473 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:39:55.470+0000 7fdb923d1640 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x55b6bc5c0620 0x55b6bc5d3940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T10:39:55.473 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:39:55.470+0000 7fdb91bd0640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55b6bc5c0f40 0x55b6bc5d0cd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.114:48174/0 (socket says 172.21.15.114:48174) 2024-04-23T10:39:55.473 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:39:55.470+0000 7fdb91bd0640 1 -- 172.21.15.114:0/154147397 learned_addr learned my addr 172.21.15.114:0/154147397 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-04-23T10:39:55.473 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:39:55.470+0000 7fdb92bd2640 1 --2- 172.21.15.114:0/154147397 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x55b6bc5cb800 0x55b6bc5d1210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T10:39:55.473 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:39:55.470+0000 7fdb91bd0640 1 -- 172.21.15.114:0/154147397 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x55b6bc5cb800 msgr2=0x55b6bc5d1210 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T10:39:55.473 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:39:55.470+0000 7fdb91bd0640 1 --2- 172.21.15.114:0/154147397 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x55b6bc5cb800 0x55b6bc5d1210 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:39:55.474 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:39:55.470+0000 7fdb91bd0640 1 -- 172.21.15.114:0/154147397 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x55b6bc5c0620 msgr2=0x55b6bc5d3940 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T10:39:55.474 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:39:55.470+0000 7fdb91bd0640 1 --2- 172.21.15.114:0/154147397 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x55b6bc5c0620 0x55b6bc5d3940 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:39:55.474 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:39:55.470+0000 7fdb91bd0640 1 -- 172.21.15.114:0/154147397 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55b6bc5d1780 con 0x55b6bc5c0f40 2024-04-23T10:39:55.474 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:39:55.470+0000 7fdb92bd2640 1 --2- 172.21.15.114:0/154147397 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x55b6bc5cb800 0x55b6bc5d1210 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-23T10:39:55.474 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:39:55.470+0000 7fdb91bd0640 1 --2- 172.21.15.114:0/154147397 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55b6bc5c0f40 0x55b6bc5d0cd0 secure :-1 s=READY pgs=918 cs=0 l=1 rev1=1 crypto rx=0x7fdb7c0ee8b0 tx=0x7fdb7c0eeb50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b23cbd089a4e8d9 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T10:39:55.474 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:39:55.470+0000 7fdb75ffb640 10 client.0 ms_handle_connect on v2:172.21.15.114:3300/0 2024-04-23T10:39:55.474 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:39:55.470+0000 7fdb75ffb640 1 -- 172.21.15.114:0/154147397 <== mon.0 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdb7c0d5040 con 0x55b6bc5c0f40 2024-04-23T10:39:55.474 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:39:55.470+0000 7fdb75ffb640 1 -- 172.21.15.114:0/154147397 <== mon.0 v2:172.21.15.114:3300/0 2 ==== osd_map(108..108 src has 1..108) ==== 8629+0+0 (secure 0 0 0) 0x7fdb7c0c57c0 con 0x55b6bc5c0f40 2024-04-23T10:39:55.474 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:39:55.470+0000 7fdb9742f740 1 -- 172.21.15.114:0/154147397 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fdb8801e040 con 0x55b6bc5c0f40 2024-04-23T10:39:55.474 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:39:55.470+0000 7fdb75ffb640 1 -- 172.21.15.114:0/154147397 <== mon.0 v2:172.21.15.114:3300/0 3 ==== mon_get_version_reply(handle=1 version=42) ==== 24+0+0 (secure 0 0 0) 0x7fdb7c0b9130 con 0x55b6bc5c0f40 2024-04-23T10:39:55.474 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:39:55.470+0000 7fdb9742f740 10 client.17037 fetch_fsmap learned FSMap version 42 2024-04-23T10:39:55.474 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:39:55.470+0000 7fdb9742f740 1 -- 172.21.15.114:0/154147397 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({config=0+,fsmap=42,monmap=4+}) -- 0x55b6bc641ba0 con 0x55b6bc5c0f40 2024-04-23T10:39:55.474 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:39:55.471+0000 7fdb75ffb640 1 -- 172.21.15.114:0/154147397 <== mon.0 v2:172.21.15.114:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fdb8801e040 con 0x55b6bc5c0f40 2024-04-23T10:39:55.474 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:39:55.471+0000 7fdb75ffb640 1 -- 172.21.15.114:0/154147397 <== mon.0 v2:172.21.15.114:3300/0 5 ==== fsmap(e 42) ==== 8338+0+0 (secure 0 0 0) 0x7fdb7c0eece0 con 0x55b6bc5c0f40 2024-04-23T10:39:55.475 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:39:55.471+0000 7fdb9742f740 10 client.17037 fetch_fsmap finished waiting for FSMap version 42 2024-04-23T10:39:55.475 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:39:55.471+0000 7fdb9742f740 10 client.17037 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-04-23T10:39:55.475 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:39:55.471+0000 7fdb9742f740 20 client.17037 populate_metadata read hostname 'smithi114' 2024-04-23T10:39:55.475 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:39:55.471+0000 7fdb9742f740 1 --2- 172.21.15.114:0/154147397 >> [v2:172.21.15.119:6838/937297340,v1:172.21.15.119:6839/937297340] conn(0x55b6bc6420f0 0x55b6bc644510 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T10:39:55.475 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:39:55.471+0000 7fdb9742f740 4 client.17037 mds_command: new command op to 24545 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-04-23T10:39:55.475 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:39:55.471+0000 7fdb9742f740 1 -- 172.21.15.114:0/154147397 --> [v2:172.21.15.119:6838/937297340,v1:172.21.15.119:6839/937297340] -- command(tid 0: {"prefix": "flush journal"}) -- 0x55b6bc52d200 con 0x55b6bc6420f0 2024-04-23T10:39:55.475 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:39:55.471+0000 7fdb923d1640 1 --2- 172.21.15.114:0/154147397 >> [v2:172.21.15.119:6838/937297340,v1:172.21.15.119:6839/937297340] conn(0x55b6bc6420f0 0x55b6bc644510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T10:39:55.475 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:39:55.471+0000 7fdb923d1640 1 --2- 172.21.15.114:0/154147397 >> [v2:172.21.15.119:6838/937297340,v1:172.21.15.119:6839/937297340] conn(0x55b6bc6420f0 0x55b6bc644510 crc :-1 s=READY pgs=181 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=57656ffaaba12fcd server_cookie=a337ee3b4d15c2aa in_seq=0 out_seq=0 2024-04-23T10:39:55.475 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:39:55.471+0000 7fdb75ffb640 10 client.17037 ms_handle_connect on v2:172.21.15.119:6838/937297340 2024-04-23T10:39:55.495 INFO:tasks.workunit.client.0.smithi114.stdout: 10 339 122.65 MB/sec warmup 1 sec latency 63.922 ms 2024-04-23T10:39:55.694 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:39:55.694+0000 7fdb75ffb640 1 -- 172.21.15.114:0/154147397 <== mds.0 v2:172.21.15.119:6838/937297340 1 ==== command_reply(tid 0: 0 ) ==== 8+0+44 (crc 0 0 0) 0x55b6bc52d200 con 0x55b6bc6420f0 2024-04-23T10:39:55.694 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:39:55.694+0000 7fdb75ffb640 10 client.17037 handle_command_reply: tid=0 2024-04-23T10:39:55.694 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:39:55.694+0000 7fdb9742f740 1 client.17037 shutdown 2024-04-23T10:39:55.694 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:39:55.694+0000 7fdb9742f740 1 -- 172.21.15.114:0/154147397 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55b6bc5c0f40 msgr2=0x55b6bc5d0cd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T10:39:55.694 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:39:55.694+0000 7fdb9742f740 1 --2- 172.21.15.114:0/154147397 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55b6bc5c0f40 0x55b6bc5d0cd0 secure :-1 s=READY pgs=918 cs=0 l=1 rev1=1 crypto rx=0x7fdb7c0ee8b0 tx=0x7fdb7c0eeb50 comp rx=0 tx=0).stop 2024-04-23T10:39:55.694 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:39:55.694+0000 7fdb9742f740 1 -- 172.21.15.114:0/154147397 shutdown_connections 2024-04-23T10:39:55.695 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:39:55.694+0000 7fdb9742f740 1 --2- 172.21.15.114:0/154147397 >> [v2:172.21.15.119:6838/937297340,v1:172.21.15.119:6839/937297340] conn(0x55b6bc6420f0 0x55b6bc644510 crc :-1 s=READY pgs=181 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:39:55.695 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:39:55.694+0000 7fdb9742f740 1 --2- 172.21.15.114:0/154147397 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x55b6bc5cb800 0x55b6bc5d1210 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:39:55.695 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:39:55.694+0000 7fdb9742f740 1 --2- 172.21.15.114:0/154147397 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55b6bc5c0f40 0x55b6bc5d0cd0 unknown :-1 s=CLOSED pgs=918 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:39:55.695 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:39:55.694+0000 7fdb75ffb640 0 client.17037 ms_handle_reset on v2:172.21.15.119:6838/937297340 2024-04-23T10:39:55.695 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:39:55.694+0000 7fdb9742f740 1 --2- 172.21.15.114:0/154147397 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x55b6bc5c0620 0x55b6bc5d3940 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:39:55.695 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:39:55.694+0000 7fdb9742f740 1 -- 172.21.15.114:0/154147397 >> 172.21.15.114:0/154147397 conn(0x55b6bc5bcb30 msgr2=0x55b6bc5c6930 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T10:39:55.695 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:39:55.694+0000 7fdb9742f740 1 -- 172.21.15.114:0/154147397 shutdown_connections 2024-04-23T10:39:55.695 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:39:55.694+0000 7fdb9742f740 1 -- 172.21.15.114:0/154147397 wait complete. 2024-04-23T10:39:55.695 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:39:55.694+0000 7fdb9742f740 20 client.17037 trim_cache size 0 max 16384 2024-04-23T10:39:55.695 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stdout:{ 2024-04-23T10:39:55.695 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stdout: "message": "", 2024-04-23T10:39:55.696 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stdout: "return_code": 0 2024-04-23T10:39:55.696 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stdout:} 2024-04-23T10:39:55.696 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:39:55.695+0000 7fdb94635640 1 -- 172.21.15.114:0/2108536228 >> [v2:172.21.15.114:6800/1395693466,v1:172.21.15.114:6801/1395693466] conn(0x7fdb780a3dc0 msgr2=0x7fdb780a6280 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T10:39:55.697 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:39:55.695+0000 7fdb94635640 1 --2- 172.21.15.114:0/2108536228 >> [v2:172.21.15.114:6800/1395693466,v1:172.21.15.114:6801/1395693466] conn(0x7fdb780a3dc0 0x7fdb780a6280 secure :-1 s=READY pgs=144 cs=0 l=1 rev1=1 crypto rx=0x7fdb84009fd0 tx=0x7fdb84004ab0 comp rx=0 tx=0).stop 2024-04-23T10:39:55.697 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:39:55.695+0000 7fdb94635640 1 -- 172.21.15.114:0/2108536228 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fdb8c106190 msgr2=0x7fdb8c1a0bf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T10:39:55.697 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:39:55.695+0000 7fdb94635640 1 --2- 172.21.15.114:0/2108536228 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fdb8c106190 0x7fdb8c1a0bf0 secure :-1 s=READY pgs=915 cs=0 l=1 rev1=1 crypto rx=0x7fdb7c005ab0 tx=0x7fdb7c003b90 comp rx=0 tx=0).stop 2024-04-23T10:39:55.697 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:39:55.695+0000 7fdb94635640 1 -- 172.21.15.114:0/2108536228 shutdown_connections 2024-04-23T10:39:55.697 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:39:55.695+0000 7fdb94635640 1 --2- 172.21.15.114:0/2108536228 >> [v2:172.21.15.114:6800/1395693466,v1:172.21.15.114:6801/1395693466] conn(0x7fdb780a3dc0 0x7fdb780a6280 unknown :-1 s=CLOSED pgs=144 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:39:55.697 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:39:55.695+0000 7fdb94635640 1 --2- 172.21.15.114:0/2108536228 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fdb8c19a770 0x7fdb8c19abf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:39:55.697 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:39:55.695+0000 7fdb94635640 1 --2- 172.21.15.114:0/2108536228 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fdb8c106190 0x7fdb8c1a0bf0 unknown :-1 s=CLOSED pgs=915 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:39:55.697 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:39:55.695+0000 7fdb94635640 1 --2- 172.21.15.114:0/2108536228 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fdb8c1010c0 0x7fdb8c1a06b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:39:55.697 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:39:55.695+0000 7fdb94635640 1 -- 172.21.15.114:0/2108536228 >> 172.21.15.114:0/2108536228 conn(0x7fdb8c077f60 msgr2=0x7fdb8c0fed80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T10:39:55.697 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:39:55.695+0000 7fdb94635640 1 -- 172.21.15.114:0/2108536228 shutdown_connections 2024-04-23T10:39:55.697 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:39:55.695+0000 7fdb94635640 1 -- 172.21.15.114:0/2108536228 wait complete. 2024-04-23T10:39:56.328 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 23 10:39:56 smithi204 ceph-mon[32649]: pgmap v418: 97 pgs: 97 active+clean; 212 MiB data, 2.8 GiB used, 1.0 TiB / 1.0 TiB avail; 317 KiB/s rd, 636 KiB/s wr, 65 op/s 2024-04-23T10:39:56.491 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 23 10:39:56 smithi119 ceph-mon[32608]: pgmap v418: 97 pgs: 97 active+clean; 212 MiB data, 2.8 GiB used, 1.0 TiB / 1.0 TiB avail; 317 KiB/s rd, 636 KiB/s wr, 65 op/s 2024-04-23T10:39:56.495 INFO:tasks.workunit.client.0.smithi114.stdout: 10 676 102.75 MB/sec warmup 2 sec latency 222.174 ms 2024-04-23T10:39:56.503 INFO:journalctl@ceph.mon.a.smithi114.stdout:Apr 23 10:39:56 smithi114 ceph-mon[29255]: pgmap v418: 97 pgs: 97 active+clean; 212 MiB data, 2.8 GiB used, 1.0 TiB / 1.0 TiB avail; 317 KiB/s rd, 636 KiB/s wr, 65 op/s 2024-04-23T10:39:57.495 INFO:tasks.workunit.client.0.smithi114.stdout: 10 1072 75.96 MB/sec warmup 3 sec latency 73.949 ms 2024-04-23T10:39:58.328 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 23 10:39:58 smithi204 ceph-mon[32649]: pgmap v419: 97 pgs: 97 active+clean; 211 MiB data, 2.8 GiB used, 1.0 TiB / 1.0 TiB avail; 501 KiB/s rd, 945 KiB/s wr, 178 op/s 2024-04-23T10:39:58.492 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 23 10:39:58 smithi119 ceph-mon[32608]: pgmap v419: 97 pgs: 97 active+clean; 211 MiB data, 2.8 GiB used, 1.0 TiB / 1.0 TiB avail; 501 KiB/s rd, 945 KiB/s wr, 178 op/s 2024-04-23T10:39:58.495 INFO:tasks.workunit.client.0.smithi114.stdout: 10 1539 64.57 MB/sec warmup 4 sec latency 90.213 ms 2024-04-23T10:39:58.503 INFO:journalctl@ceph.mon.a.smithi114.stdout:Apr 23 10:39:58 smithi114 ceph-mon[29255]: pgmap v419: 97 pgs: 97 active+clean; 211 MiB data, 2.8 GiB used, 1.0 TiB / 1.0 TiB avail; 501 KiB/s rd, 945 KiB/s wr, 178 op/s 2024-04-23T10:39:59.492 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 23 10:39:59 smithi119 ceph-mon[32608]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=e=up:active} 3 up:standby-replay 6 up:standby 2024-04-23T10:39:59.492 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 23 10:39:59 smithi119 ceph-mon[32608]: from='mgr.15291 172.21.15.114:0/1095789187' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-23T10:39:59.495 INFO:tasks.workunit.client.0.smithi114.stdout: 10 2276 55.49 MB/sec warmup 5 sec latency 136.955 ms 2024-04-23T10:39:59.504 INFO:journalctl@ceph.mon.a.smithi114.stdout:Apr 23 10:39:59 smithi114 ceph-mon[29255]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=e=up:active} 3 up:standby-replay 6 up:standby 2024-04-23T10:39:59.504 INFO:journalctl@ceph.mon.a.smithi114.stdout:Apr 23 10:39:59 smithi114 ceph-mon[29255]: from='mgr.15291 172.21.15.114:0/1095789187' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-23T10:39:59.578 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 23 10:39:59 smithi204 ceph-mon[32649]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=e=up:active} 3 up:standby-replay 6 up:standby 2024-04-23T10:39:59.578 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 23 10:39:59 smithi204 ceph-mon[32649]: from='mgr.15291 172.21.15.114:0/1095789187' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-23T10:40:00.492 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 23 10:40:00 smithi119 ceph-mon[32608]: pgmap v420: 97 pgs: 97 active+clean; 211 MiB data, 2.8 GiB used, 1.0 TiB / 1.0 TiB avail; 295 KiB/s rd, 594 KiB/s wr, 131 op/s 2024-04-23T10:40:00.492 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 23 10:40:00 smithi119 ceph-mon[32608]: from='mgr.15291 172.21.15.114:0/1095789187' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-23T10:40:00.492 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 23 10:40:00 smithi119 ceph-mon[32608]: from='mgr.15291 172.21.15.114:0/1095789187' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-23T10:40:00.492 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 23 10:40:00 smithi119 ceph-mon[32608]: from='mgr.15291 172.21.15.114:0/1095789187' entity='mgr.x' 2024-04-23T10:40:00.492 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 23 10:40:00 smithi119 ceph-mon[32608]: overall HEALTH_OK 2024-04-23T10:40:00.495 INFO:tasks.workunit.client.0.smithi114.stdout: 10 2704 52.54 MB/sec warmup 6 sec latency 120.818 ms 2024-04-23T10:40:00.504 INFO:journalctl@ceph.mon.a.smithi114.stdout:Apr 23 10:40:00 smithi114 ceph-mon[29255]: pgmap v420: 97 pgs: 97 active+clean; 211 MiB data, 2.8 GiB used, 1.0 TiB / 1.0 TiB avail; 295 KiB/s rd, 594 KiB/s wr, 131 op/s 2024-04-23T10:40:00.504 INFO:journalctl@ceph.mon.a.smithi114.stdout:Apr 23 10:40:00 smithi114 ceph-mon[29255]: from='mgr.15291 172.21.15.114:0/1095789187' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-23T10:40:00.504 INFO:journalctl@ceph.mon.a.smithi114.stdout:Apr 23 10:40:00 smithi114 ceph-mon[29255]: from='mgr.15291 172.21.15.114:0/1095789187' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-23T10:40:00.504 INFO:journalctl@ceph.mon.a.smithi114.stdout:Apr 23 10:40:00 smithi114 ceph-mon[29255]: from='mgr.15291 172.21.15.114:0/1095789187' entity='mgr.x' 2024-04-23T10:40:00.504 INFO:journalctl@ceph.mon.a.smithi114.stdout:Apr 23 10:40:00 smithi114 ceph-mon[29255]: overall HEALTH_OK 2024-04-23T10:40:00.578 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 23 10:40:00 smithi204 ceph-mon[32649]: pgmap v420: 97 pgs: 97 active+clean; 211 MiB data, 2.8 GiB used, 1.0 TiB / 1.0 TiB avail; 295 KiB/s rd, 594 KiB/s wr, 131 op/s 2024-04-23T10:40:00.578 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 23 10:40:00 smithi204 ceph-mon[32649]: from='mgr.15291 172.21.15.114:0/1095789187' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-23T10:40:00.578 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 23 10:40:00 smithi204 ceph-mon[32649]: from='mgr.15291 172.21.15.114:0/1095789187' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-23T10:40:00.578 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 23 10:40:00 smithi204 ceph-mon[32649]: from='mgr.15291 172.21.15.114:0/1095789187' entity='mgr.x' 2024-04-23T10:40:00.578 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 23 10:40:00 smithi204 ceph-mon[32649]: overall HEALTH_OK 2024-04-23T10:40:01.492 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 23 10:40:01 smithi119 ceph-mon[32608]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=e=up:active} 3 up:standby-replay 6 up:standby 2024-04-23T10:40:01.496 INFO:tasks.workunit.client.0.smithi114.stdout: 10 3153 48.67 MB/sec warmup 7 sec latency 113.803 ms 2024-04-23T10:40:01.504 INFO:journalctl@ceph.mon.a.smithi114.stdout:Apr 23 10:40:01 smithi114 ceph-mon[29255]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=e=up:active} 3 up:standby-replay 6 up:standby 2024-04-23T10:40:01.577 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 23 10:40:01 smithi204 ceph-mon[32649]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=e=up:active} 3 up:standby-replay 6 up:standby 2024-04-23T10:40:02.491 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 23 10:40:02 smithi119 ceph-mon[32608]: pgmap v421: 97 pgs: 97 active+clean; 211 MiB data, 2.8 GiB used, 1.0 TiB / 1.0 TiB avail; 295 KiB/s rd, 594 KiB/s wr, 131 op/s 2024-04-23T10:40:02.495 INFO:tasks.workunit.client.0.smithi114.stdout: 10 3519 47.15 MB/sec warmup 8 sec latency 87.753 ms 2024-04-23T10:40:02.503 INFO:journalctl@ceph.mon.a.smithi114.stdout:Apr 23 10:40:02 smithi114 ceph-mon[29255]: pgmap v421: 97 pgs: 97 active+clean; 211 MiB data, 2.8 GiB used, 1.0 TiB / 1.0 TiB avail; 295 KiB/s rd, 594 KiB/s wr, 131 op/s 2024-04-23T10:40:02.577 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 23 10:40:02 smithi204 ceph-mon[32649]: pgmap v421: 97 pgs: 97 active+clean; 211 MiB data, 2.8 GiB used, 1.0 TiB / 1.0 TiB avail; 295 KiB/s rd, 594 KiB/s wr, 131 op/s 2024-04-23T10:40:03.496 INFO:tasks.workunit.client.0.smithi114.stdout: 10 3765 43.48 MB/sec warmup 9 sec latency 51.677 ms 2024-04-23T10:40:04.380 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 23 10:40:04 smithi204 ceph-mon[32649]: pgmap v422: 97 pgs: 97 active+clean; 227 MiB data, 2.9 GiB used, 1.0 TiB / 1.0 TiB avail; 897 KiB/s rd, 2.4 MiB/s wr, 317 op/s 2024-04-23T10:40:04.491 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 23 10:40:04 smithi119 ceph-mon[32608]: pgmap v422: 97 pgs: 97 active+clean; 227 MiB data, 2.9 GiB used, 1.0 TiB / 1.0 TiB avail; 897 KiB/s rd, 2.4 MiB/s wr, 317 op/s 2024-04-23T10:40:04.496 INFO:tasks.workunit.client.0.smithi114.stdout: 10 3921 39.49 MB/sec warmup 10 sec latency 124.102 ms 2024-04-23T10:40:04.504 INFO:journalctl@ceph.mon.a.smithi114.stdout:Apr 23 10:40:04 smithi114 ceph-mon[29255]: pgmap v422: 97 pgs: 97 active+clean; 227 MiB data, 2.9 GiB used, 1.0 TiB / 1.0 TiB avail; 897 KiB/s rd, 2.4 MiB/s wr, 317 op/s 2024-04-23T10:40:05.492 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 23 10:40:05 smithi119 ceph-mon[32608]: osdmap e109: 12 total, 12 up, 12 in 2024-04-23T10:40:05.496 INFO:tasks.workunit.client.0.smithi114.stdout: 10 4126 36.39 MB/sec warmup 11 sec latency 124.153 ms 2024-04-23T10:40:05.504 INFO:journalctl@ceph.mon.a.smithi114.stdout:Apr 23 10:40:05 smithi114 ceph-mon[29255]: osdmap e109: 12 total, 12 up, 12 in 2024-04-23T10:40:05.577 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 23 10:40:05 smithi204 ceph-mon[32649]: osdmap e109: 12 total, 12 up, 12 in 2024-04-23T10:40:06.392 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 23 10:40:06 smithi119 ceph-mon[32608]: pgmap v424: 97 pgs: 97 active+clean; 227 MiB data, 2.9 GiB used, 1.0 TiB / 1.0 TiB avail; 943 KiB/s rd, 2.5 MiB/s wr, 357 op/s 2024-04-23T10:40:06.392 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 23 10:40:06 smithi119 ceph-mon[32608]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=e=up:active} 3 up:standby-replay 6 up:standby 2024-04-23T10:40:06.496 INFO:tasks.workunit.client.0.smithi114.stdout: 10 4382 33.96 MB/sec warmup 12 sec latency 245.618 ms 2024-04-23T10:40:06.503 INFO:journalctl@ceph.mon.a.smithi114.stdout:Apr 23 10:40:06 smithi114 ceph-mon[29255]: pgmap v424: 97 pgs: 97 active+clean; 227 MiB data, 2.9 GiB used, 1.0 TiB / 1.0 TiB avail; 943 KiB/s rd, 2.5 MiB/s wr, 357 op/s 2024-04-23T10:40:06.504 INFO:journalctl@ceph.mon.a.smithi114.stdout:Apr 23 10:40:06 smithi114 ceph-mon[29255]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=e=up:active} 3 up:standby-replay 6 up:standby 2024-04-23T10:40:06.577 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 23 10:40:06 smithi204 ceph-mon[32649]: pgmap v424: 97 pgs: 97 active+clean; 227 MiB data, 2.9 GiB used, 1.0 TiB / 1.0 TiB avail; 943 KiB/s rd, 2.5 MiB/s wr, 357 op/s 2024-04-23T10:40:06.577 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 23 10:40:06 smithi204 ceph-mon[32649]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=e=up:active} 3 up:standby-replay 6 up:standby 2024-04-23T10:40:07.496 INFO:tasks.workunit.client.0.smithi114.stdout: 10 4831 33.37 MB/sec warmup 13 sec latency 735.233 ms 2024-04-23T10:40:08.492 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 23 10:40:08 smithi119 ceph-mon[32608]: pgmap v425: 97 pgs: 97 active+clean; 243 MiB data, 3.0 GiB used, 1.0 TiB / 1.0 TiB avail; 1.5 MiB/s rd, 4.2 MiB/s wr, 463 op/s 2024-04-23T10:40:08.496 INFO:tasks.workunit.client.0.smithi114.stdout: 10 5465 32.33 MB/sec warmup 14 sec latency 1735.277 ms 2024-04-23T10:40:08.503 INFO:journalctl@ceph.mon.a.smithi114.stdout:Apr 23 10:40:08 smithi114 ceph-mon[29255]: pgmap v425: 97 pgs: 97 active+clean; 243 MiB data, 3.0 GiB used, 1.0 TiB / 1.0 TiB avail; 1.5 MiB/s rd, 4.2 MiB/s wr, 463 op/s 2024-04-23T10:40:08.577 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 23 10:40:08 smithi204 ceph-mon[32649]: pgmap v425: 97 pgs: 97 active+clean; 243 MiB data, 3.0 GiB used, 1.0 TiB / 1.0 TiB avail; 1.5 MiB/s rd, 4.2 MiB/s wr, 463 op/s 2024-04-23T10:40:08.786 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:40:08.787+0000 7fbaed144640 1 -- 172.21.15.114:0/290322489 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fbae8103b00 msgr2=0x7fbae810bfe0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T10:40:08.786 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:40:08.787+0000 7fbaed144640 1 --2- 172.21.15.114:0/290322489 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fbae8103b00 0x7fbae810bfe0 secure :-1 s=READY pgs=919 cs=0 l=1 rev1=1 crypto rx=0x7fbadc009900 tx=0x7fbadc02a510 comp rx=0 tx=0).stop 2024-04-23T10:40:08.786 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:40:08.787+0000 7fbaed144640 1 -- 172.21.15.114:0/290322489 shutdown_connections 2024-04-23T10:40:08.786 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:40:08.787+0000 7fbaed144640 1 --2- 172.21.15.114:0/290322489 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fbae810c520 0x7fbae810e9c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:40:08.786 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:40:08.787+0000 7fbaed144640 1 --2- 172.21.15.114:0/290322489 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fbae8103b00 0x7fbae810bfe0 unknown :-1 s=CLOSED pgs=919 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:40:08.786 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:40:08.787+0000 7fbaed144640 1 --2- 172.21.15.114:0/290322489 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fbae81031e0 0x7fbae81035c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:40:08.786 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:40:08.787+0000 7fbaed144640 1 -- 172.21.15.114:0/290322489 >> 172.21.15.114:0/290322489 conn(0x7fbae8077fd0 msgr2=0x7fbae80783e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T10:40:08.786 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:40:08.787+0000 7fbaed144640 1 -- 172.21.15.114:0/290322489 shutdown_connections 2024-04-23T10:40:08.787 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:40:08.787+0000 7fbaed144640 1 -- 172.21.15.114:0/290322489 wait complete. 2024-04-23T10:40:08.789 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:40:08.788+0000 7fbaed144640 1 Processor -- start 2024-04-23T10:40:08.789 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:40:08.788+0000 7fbaed144640 1 -- start start 2024-04-23T10:40:08.789 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:40:08.788+0000 7fbaed144640 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fbae81031e0 0x7fbae8198710 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T10:40:08.789 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:40:08.788+0000 7fbaed144640 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fbae8103b00 0x7fbae8198c50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T10:40:08.789 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:40:08.788+0000 7fbaed144640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fbae810c520 0x7fbae819cf80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T10:40:08.789 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:40:08.788+0000 7fbaed144640 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x7fbae81121b0 con 0x7fbae810c520 2024-04-23T10:40:08.789 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:40:08.788+0000 7fbaed144640 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 -- 0x7fbae8112030 con 0x7fbae81031e0 2024-04-23T10:40:08.789 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:40:08.788+0000 7fbaed144640 1 -- --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_getmap magic: 0 -- 0x7fbae8112330 con 0x7fbae8103b00 2024-04-23T10:40:08.790 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:40:08.788+0000 7fbae759e640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fbae810c520 0x7fbae819cf80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T10:40:08.790 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:40:08.788+0000 7fbae659c640 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fbae8103b00 0x7fbae8198c50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T10:40:08.790 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:40:08.788+0000 7fbae759e640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fbae810c520 0x7fbae819cf80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.114:60998/0 (socket says 172.21.15.114:60998) 2024-04-23T10:40:08.790 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:40:08.788+0000 7fbae759e640 1 -- 172.21.15.114:0/2187082834 learned_addr learned my addr 172.21.15.114:0/2187082834 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-04-23T10:40:08.790 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:40:08.788+0000 7fbae6d9d640 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fbae81031e0 0x7fbae8198710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T10:40:08.790 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:40:08.789+0000 7fbae659c640 1 -- 172.21.15.114:0/2187082834 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fbae81031e0 msgr2=0x7fbae8198710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T10:40:08.790 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:40:08.789+0000 7fbae659c640 1 --2- 172.21.15.114:0/2187082834 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fbae81031e0 0x7fbae8198710 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:40:08.790 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:40:08.789+0000 7fbae659c640 1 -- 172.21.15.114:0/2187082834 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fbae810c520 msgr2=0x7fbae819cf80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T10:40:08.790 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:40:08.789+0000 7fbae659c640 1 --2- 172.21.15.114:0/2187082834 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fbae810c520 0x7fbae819cf80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:40:08.790 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:40:08.789+0000 7fbae659c640 1 -- 172.21.15.114:0/2187082834 --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fbae819d6b0 con 0x7fbae8103b00 2024-04-23T10:40:08.790 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:40:08.789+0000 7fbae759e640 1 --2- 172.21.15.114:0/2187082834 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fbae810c520 0x7fbae819cf80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-23T10:40:08.790 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:40:08.789+0000 7fbae6d9d640 1 --2- 172.21.15.114:0/2187082834 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fbae81031e0 0x7fbae8198710 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-23T10:40:08.790 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:40:08.789+0000 7fbae659c640 1 --2- 172.21.15.114:0/2187082834 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fbae8103b00 0x7fbae8198c50 secure :-1 s=READY pgs=81 cs=0 l=1 rev1=1 crypto rx=0x7fbadc02aa30 tx=0x7fbadc031a20 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T10:40:08.790 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:40:08.789+0000 7fbacffff640 1 -- 172.21.15.114:0/2187082834 <== mon.1 v2:172.21.15.204:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbadc03f030 con 0x7fbae8103b00 2024-04-23T10:40:08.791 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:40:08.789+0000 7fbaed144640 1 -- 172.21.15.114:0/2187082834 --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fbae819d940 con 0x7fbae8103b00 2024-04-23T10:40:08.791 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:40:08.789+0000 7fbacffff640 1 -- 172.21.15.114:0/2187082834 <== mon.1 v2:172.21.15.204:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fbadc004490 con 0x7fbae8103b00 2024-04-23T10:40:08.791 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:40:08.789+0000 7fbacffff640 1 -- 172.21.15.114:0/2187082834 <== mon.1 v2:172.21.15.204:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbadc035070 con 0x7fbae8103b00 2024-04-23T10:40:08.791 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:40:08.789+0000 7fbaed144640 1 -- 172.21.15.114:0/2187082834 --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fbae81a5230 con 0x7fbae8103b00 2024-04-23T10:40:08.797 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:40:08.791+0000 7fbacffff640 1 -- 172.21.15.114:0/2187082834 <== mon.1 v2:172.21.15.204:3300/0 4 ==== mgrmap(e 26) ==== 138778+0+0 (secure 0 0 0) 0x7fbadc042030 con 0x7fbae8103b00 2024-04-23T10:40:08.797 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:40:08.791+0000 7fbaefe66740 1 Processor -- start 2024-04-23T10:40:08.797 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:40:08.791+0000 7fbaefe66740 1 -- start start 2024-04-23T10:40:08.797 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:40:08.792+0000 7fbaefe66740 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x56177bbb9620 0x56177bbb9a00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T10:40:08.798 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:40:08.792+0000 7fbaefe66740 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x56177bbb9f40 0x56177bbc42c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T10:40:08.798 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:40:08.792+0000 7fbaefe66740 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x56177bbc4800 0x56177bbc6bf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T10:40:08.798 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:40:08.792+0000 7fbaefe66740 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x56177bb1e970 con 0x56177bbb9620 2024-04-23T10:40:08.798 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:40:08.792+0000 7fbaefe66740 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 -- 0x56177babb4a0 con 0x56177bbb9f40 2024-04-23T10:40:08.798 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:40:08.792+0000 7fbaefe66740 1 -- --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_getmap magic: 0 -- 0x56177bb1a930 con 0x56177bbc4800 2024-04-23T10:40:08.798 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:40:08.792+0000 7fbae6d9d640 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x56177bbb9f40 0x56177bbc42c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T10:40:08.798 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:40:08.792+0000 7fbae759e640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x56177bbb9620 0x56177bbb9a00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T10:40:08.798 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:40:08.792+0000 7fbacffff640 1 --2- 172.21.15.114:0/2187082834 >> [v2:172.21.15.114:6800/1395693466,v1:172.21.15.114:6801/1395693466] conn(0x7fbad00a3dc0 0x7fbad00a6280 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T10:40:08.798 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:40:08.792+0000 7fbae6d9d640 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x56177bbb9f40 0x56177bbc42c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.119:3300/0 says I am v2:172.21.15.114:45558/0 (socket says 172.21.15.114:45558) 2024-04-23T10:40:08.798 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:40:08.792+0000 7fbae6d9d640 1 -- 172.21.15.114:0/1233538254 learned_addr learned my addr 172.21.15.114:0/1233538254 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-04-23T10:40:08.798 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:40:08.792+0000 7fbae759e640 1 --2- 172.21.15.114:0/2187082834 >> [v2:172.21.15.114:6800/1395693466,v1:172.21.15.114:6801/1395693466] conn(0x7fbad00a3dc0 0x7fbad00a6280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T10:40:08.798 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:40:08.792+0000 7fbae659c640 1 --2- 172.21.15.114:0/1233538254 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x56177bbc4800 0x56177bbc6bf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T10:40:08.798 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:40:08.792+0000 7fbae6d9d640 1 -- 172.21.15.114:0/1233538254 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x56177bbc4800 msgr2=0x56177bbc6bf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T10:40:08.798 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:40:08.792+0000 7fbae6d9d640 1 --2- 172.21.15.114:0/1233538254 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x56177bbc4800 0x56177bbc6bf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:40:08.798 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:40:08.792+0000 7fbae6d9d640 1 -- 172.21.15.114:0/1233538254 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x56177bbb9620 msgr2=0x56177bbb9a00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T10:40:08.799 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:40:08.792+0000 7fbae6d9d640 1 --2- 172.21.15.114:0/1233538254 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x56177bbb9620 0x56177bbb9a00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:40:08.799 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:40:08.792+0000 7fbacffff640 1 -- 172.21.15.114:0/2187082834 <== mon.1 v2:172.21.15.204:3300/0 5 ==== osd_map(110..110 src has 1..110) ==== 8629+0+0 (secure 0 0 0) 0x7fbadc0f0ae0 con 0x7fbae8103b00 2024-04-23T10:40:08.799 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:40:08.792+0000 7fbae6d9d640 1 -- 172.21.15.114:0/1233538254 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x56177b831210 con 0x56177bbb9f40 2024-04-23T10:40:08.799 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:40:08.792+0000 7fbae659c640 1 --2- 172.21.15.114:0/1233538254 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x56177bbc4800 0x56177bbc6bf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-23T10:40:08.799 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:40:08.792+0000 7fbae6d9d640 1 --2- 172.21.15.114:0/1233538254 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x56177bbb9f40 0x56177bbc42c0 secure :-1 s=READY pgs=101 cs=0 l=1 rev1=1 crypto rx=0x7fbad8011540 tx=0x7fbad8011a00 comp rx=0 tx=0).ready entity=mon.2 client_cookie=70a4b0e2d6210668 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T10:40:08.799 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:40:08.793+0000 7fbae759e640 1 --2- 172.21.15.114:0/2187082834 >> [v2:172.21.15.114:6800/1395693466,v1:172.21.15.114:6801/1395693466] conn(0x7fbad00a3dc0 0x7fbad00a6280 secure :-1 s=READY pgs=145 cs=0 l=1 rev1=1 crypto rx=0x7fbae0009aa0 tx=0x7fbae0015040 comp rx=0 tx=0).ready entity=mgr.15291 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T10:40:08.799 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:40:08.793+0000 7fbacd7fa640 1 -- 172.21.15.114:0/1233538254 <== mon.2 v2:172.21.15.119:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbad800f640 con 0x56177bbb9f40 2024-04-23T10:40:08.799 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:40:08.793+0000 7fbacd7fa640 1 -- 172.21.15.114:0/1233538254 <== mon.2 v2:172.21.15.119:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fbad800fc90 con 0x56177bbb9f40 2024-04-23T10:40:08.799 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:40:08.793+0000 7fbacd7fa640 1 -- 172.21.15.114:0/1233538254 <== mon.2 v2:172.21.15.119:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbad8019c80 con 0x56177bbb9f40 2024-04-23T10:40:08.799 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:40:08.793+0000 7fbaefe66740 1 -- 172.21.15.114:0/1233538254 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x56177bbb9f40 msgr2=0x56177bbc42c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T10:40:08.799 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:40:08.793+0000 7fbaefe66740 1 --2- 172.21.15.114:0/1233538254 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x56177bbb9f40 0x56177bbc42c0 secure :-1 s=READY pgs=101 cs=0 l=1 rev1=1 crypto rx=0x7fbad8011540 tx=0x7fbad8011a00 comp rx=0 tx=0).stop 2024-04-23T10:40:08.799 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:40:08.793+0000 7fbaefe66740 1 -- 172.21.15.114:0/1233538254 shutdown_connections 2024-04-23T10:40:08.799 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:40:08.793+0000 7fbaefe66740 1 --2- 172.21.15.114:0/1233538254 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x56177bbc4800 0x56177bbc6bf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:40:08.799 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:40:08.793+0000 7fbaefe66740 1 --2- 172.21.15.114:0/1233538254 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x56177bbb9f40 0x56177bbc42c0 unknown :-1 s=CLOSED pgs=101 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:40:08.799 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:40:08.793+0000 7fbaefe66740 1 --2- 172.21.15.114:0/1233538254 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x56177bbb9620 0x56177bbb9a00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:40:08.800 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:40:08.793+0000 7fbaefe66740 1 -- 172.21.15.114:0/1233538254 >> 172.21.15.114:0/1233538254 conn(0x56177bbb5b30 msgr2=0x56177bbb5f10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T10:40:08.800 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:40:08.793+0000 7fbaefe66740 1 -- 172.21.15.114:0/1233538254 shutdown_connections 2024-04-23T10:40:08.800 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:40:08.793+0000 7fbaefe66740 1 -- 172.21.15.114:0/1233538254 wait complete. 2024-04-23T10:40:08.800 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:40:08.794+0000 7fbaefe66740 1 -- start start 2024-04-23T10:40:08.800 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:40:08.794+0000 7fbaefe66740 1 Processor -- start 2024-04-23T10:40:08.800 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:40:08.794+0000 7fbaefe66740 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x56177bbb9620 0x56177bbc40c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T10:40:08.800 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:40:08.794+0000 7fbaefe66740 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x56177bbb9f40 0x56177bbc13b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T10:40:08.800 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:40:08.794+0000 7fbaefe66740 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x56177bbc4800 0x56177bbc18f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T10:40:08.800 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:40:08.794+0000 7fbaefe66740 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x56177bbc7430 con 0x56177bbb9620 2024-04-23T10:40:08.800 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:40:08.794+0000 7fbaefe66740 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 -- 0x56177bab90b0 con 0x56177bbb9f40 2024-04-23T10:40:08.800 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:40:08.794+0000 7fbaefe66740 1 -- --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_getmap magic: 0 -- 0x56177bbc75b0 con 0x56177bbc4800 2024-04-23T10:40:08.800 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:40:08.794+0000 7fbaefe66740 10 client.0 _ll_register_callbacks cb 0x56177baa6d30 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-23T10:40:08.800 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:40:08.794+0000 7fbae6d9d640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x56177bbb9620 0x56177bbc40c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T10:40:08.800 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:40:08.794+0000 7fbae659c640 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x56177bbb9f40 0x56177bbc13b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T10:40:08.800 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:40:08.794+0000 7fbae6d9d640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x56177bbb9620 0x56177bbc40c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.114:32792/0 (socket says 172.21.15.114:32792) 2024-04-23T10:40:08.800 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:40:08.794+0000 7fbae6d9d640 1 -- 172.21.15.114:0/2834542707 learned_addr learned my addr 172.21.15.114:0/2834542707 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-04-23T10:40:08.801 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:40:08.794+0000 7fbae759e640 1 --2- 172.21.15.114:0/2834542707 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x56177bbc4800 0x56177bbc18f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T10:40:08.801 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:40:08.794+0000 7fbae6d9d640 1 -- 172.21.15.114:0/2834542707 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x56177bbc4800 msgr2=0x56177bbc18f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T10:40:08.801 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:40:08.794+0000 7fbae6d9d640 1 --2- 172.21.15.114:0/2834542707 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x56177bbc4800 0x56177bbc18f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:40:08.801 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:40:08.794+0000 7fbae6d9d640 1 -- 172.21.15.114:0/2834542707 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x56177bbb9f40 msgr2=0x56177bbc13b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T10:40:08.801 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:40:08.794+0000 7fbae6d9d640 1 --2- 172.21.15.114:0/2834542707 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x56177bbb9f40 0x56177bbc13b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:40:08.801 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:40:08.794+0000 7fbae6d9d640 1 -- 172.21.15.114:0/2834542707 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({osdmap=0}) -- 0x56177b8fd620 con 0x56177bbb9620 2024-04-23T10:40:08.801 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:40:08.794+0000 7fbae659c640 1 --2- 172.21.15.114:0/2834542707 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x56177bbb9f40 0x56177bbc13b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-23T10:40:08.801 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:40:08.795+0000 7fbae6d9d640 1 --2- 172.21.15.114:0/2834542707 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x56177bbb9620 0x56177bbc40c0 secure :-1 s=READY pgs=920 cs=0 l=1 rev1=1 crypto rx=0x7fbad8005bc0 tx=0x7fbad800f8f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1f1c3e4872097c05 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T10:40:08.801 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:40:08.795+0000 7fbacaff5640 10 client.0 ms_handle_connect on v2:172.21.15.114:3300/0 2024-04-23T10:40:08.801 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:40:08.795+0000 7fbacaff5640 1 -- 172.21.15.114:0/2834542707 <== mon.0 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbad8026020 con 0x56177bbb9620 2024-04-23T10:40:08.801 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:40:08.795+0000 7fbacaff5640 1 -- 172.21.15.114:0/2834542707 <== mon.0 v2:172.21.15.114:3300/0 2 ==== osd_map(110..110 src has 1..110) ==== 8629+0+0 (secure 0 0 0) 0x7fbad8019b90 con 0x56177bbb9620 2024-04-23T10:40:08.801 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:40:08.795+0000 7fbaefe66740 1 -- 172.21.15.114:0/2834542707 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fbad800fc90 con 0x56177bbb9620 2024-04-23T10:40:08.801 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:40:08.795+0000 7fbacaff5640 1 -- 172.21.15.114:0/2834542707 <== mon.0 v2:172.21.15.114:3300/0 3 ==== mon_get_version_reply(handle=1 version=45) ==== 24+0+0 (secure 0 0 0) 0x7fbad8031050 con 0x56177bbb9620 2024-04-23T10:40:08.801 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:40:08.795+0000 7fbaefe66740 10 client.17049 fetch_fsmap learned FSMap version 45 2024-04-23T10:40:08.801 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:40:08.795+0000 7fbaefe66740 1 -- 172.21.15.114:0/2834542707 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({config=0+,fsmap=45,monmap=4+}) -- 0x56177bbd3230 con 0x56177bbb9620 2024-04-23T10:40:08.802 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:40:08.795+0000 7fbacaff5640 1 -- 172.21.15.114:0/2834542707 <== mon.0 v2:172.21.15.114:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fbad800fc90 con 0x56177bbb9620 2024-04-23T10:40:08.802 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:40:08.795+0000 7fbacaff5640 1 -- 172.21.15.114:0/2834542707 <== mon.0 v2:172.21.15.114:3300/0 5 ==== fsmap(e 45) ==== 8350+0+0 (secure 0 0 0) 0x7fbad80046c0 con 0x56177bbb9620 2024-04-23T10:40:08.802 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:40:08.795+0000 7fbaefe66740 10 client.17049 fetch_fsmap finished waiting for FSMap version 45 2024-04-23T10:40:08.802 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:40:08.795+0000 7fbaefe66740 10 client.17049 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-04-23T10:40:08.802 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:40:08.795+0000 7fbaefe66740 20 client.17049 populate_metadata read hostname 'smithi114' 2024-04-23T10:40:08.802 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:40:08.795+0000 7fbaefe66740 1 --2- 172.21.15.114:0/2834542707 >> [v2:172.21.15.119:6838/937297340,v1:172.21.15.119:6839/937297340] conn(0x56177bbd4690 0x56177bbd6a80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T10:40:08.802 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:40:08.795+0000 7fbaefe66740 4 client.17049 mds_command: new command op to 24545 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-23T10:40:08.802 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:40:08.795+0000 7fbaefe66740 1 -- 172.21.15.114:0/2834542707 --> [v2:172.21.15.119:6838/937297340,v1:172.21.15.119:6839/937297340] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x56177bb2c390 con 0x56177bbd4690 2024-04-23T10:40:08.802 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:40:08.796+0000 7fbae659c640 1 --2- 172.21.15.114:0/2834542707 >> [v2:172.21.15.119:6838/937297340,v1:172.21.15.119:6839/937297340] conn(0x56177bbd4690 0x56177bbd6a80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T10:40:08.802 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:40:08.796+0000 7fbae659c640 1 --2- 172.21.15.114:0/2834542707 >> [v2:172.21.15.119:6838/937297340,v1:172.21.15.119:6839/937297340] conn(0x56177bbd4690 0x56177bbd6a80 crc :-1 s=READY pgs=182 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=a3661472f7fb770a server_cookie=18d87074f01cbbf5 in_seq=0 out_seq=0 2024-04-23T10:40:08.802 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:40:08.796+0000 7fbacaff5640 10 client.17049 ms_handle_connect on v2:172.21.15.119:6838/937297340 2024-04-23T10:40:08.802 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:40:08.798+0000 7fbacaff5640 1 -- 172.21.15.114:0/2834542707 <== mds.0 v2:172.21.15.119:6838/937297340 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x56177bb2c390 con 0x56177bbd4690 2024-04-23T10:40:08.802 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:40:08.798+0000 7fbacaff5640 10 client.17049 handle_command_reply: tid=0 2024-04-23T10:40:08.802 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:40:08.798+0000 7fbaefe66740 1 client.17049 shutdown 2024-04-23T10:40:08.803 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:40:08.798+0000 7fbaefe66740 1 -- 172.21.15.114:0/2834542707 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x56177bbb9620 msgr2=0x56177bbc40c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T10:40:08.803 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:40:08.798+0000 7fbaefe66740 1 --2- 172.21.15.114:0/2834542707 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x56177bbb9620 0x56177bbc40c0 secure :-1 s=READY pgs=920 cs=0 l=1 rev1=1 crypto rx=0x7fbad8005bc0 tx=0x7fbad800f8f0 comp rx=0 tx=0).stop 2024-04-23T10:40:08.803 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:40:08.798+0000 7fbaefe66740 1 -- 172.21.15.114:0/2834542707 shutdown_connections 2024-04-23T10:40:08.803 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:40:08.798+0000 7fbaefe66740 1 --2- 172.21.15.114:0/2834542707 >> [v2:172.21.15.119:6838/937297340,v1:172.21.15.119:6839/937297340] conn(0x56177bbd4690 0x56177bbd6a80 crc :-1 s=READY pgs=182 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:40:08.803 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:40:08.798+0000 7fbaefe66740 1 --2- 172.21.15.114:0/2834542707 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x56177bbc4800 0x56177bbc18f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:40:08.803 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:40:08.798+0000 7fbaefe66740 1 --2- 172.21.15.114:0/2834542707 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x56177bbb9f40 0x56177bbc13b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:40:08.803 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:40:08.798+0000 7fbaefe66740 1 --2- 172.21.15.114:0/2834542707 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x56177bbb9620 0x56177bbc40c0 unknown :-1 s=CLOSED pgs=920 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:40:08.803 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:40:08.798+0000 7fbaefe66740 1 -- 172.21.15.114:0/2834542707 >> 172.21.15.114:0/2834542707 conn(0x56177bbb5b30 msgr2=0x56177bbbb8b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T10:40:08.803 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:40:08.798+0000 7fbacaff5640 0 client.17049 ms_handle_reset on v2:172.21.15.119:6838/937297340 2024-04-23T10:40:08.803 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:40:08.798+0000 7fbaefe66740 1 -- 172.21.15.114:0/2834542707 shutdown_connections 2024-04-23T10:40:08.803 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:40:08.798+0000 7fbaefe66740 1 -- 172.21.15.114:0/2834542707 wait complete. 2024-04-23T10:40:08.803 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:40:08.798+0000 7fbaefe66740 20 client.17049 trim_cache size 0 max 16384 2024-04-23T10:40:08.813 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:40:08.813+0000 7fbaefe66740 1 Processor -- start 2024-04-23T10:40:08.814 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:40:08.813+0000 7fbaefe66740 1 -- start start 2024-04-23T10:40:08.814 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:40:08.814+0000 7fbaefe66740 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x56177bbb9620 0x56177bbd02f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T10:40:08.814 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:40:08.814+0000 7fbaefe66740 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x56177bbb9f40 0x56177bbbcaf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T10:40:08.814 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:40:08.814+0000 7fbaefe66740 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x56177bbc4800 0x56177bbc3bf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T10:40:08.814 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:40:08.814+0000 7fbaefe66740 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x7fbae81b05d0 con 0x56177bbb9f40 2024-04-23T10:40:08.814 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:40:08.814+0000 7fbaefe66740 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 -- 0x7fbae81ade70 con 0x56177bbc4800 2024-04-23T10:40:08.814 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:40:08.814+0000 7fbaefe66740 1 -- --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_getmap magic: 0 -- 0x7fbae81abce0 con 0x56177bbb9620 2024-04-23T10:40:08.814 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:40:08.814+0000 7fbae659c640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x56177bbb9f40 0x56177bbbcaf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T10:40:08.814 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:40:08.814+0000 7fbae659c640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x56177bbb9f40 0x56177bbbcaf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.114:32794/0 (socket says 172.21.15.114:32794) 2024-04-23T10:40:08.814 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:40:08.814+0000 7fbae659c640 1 -- 172.21.15.114:0/3167027726 learned_addr learned my addr 172.21.15.114:0/3167027726 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-04-23T10:40:08.814 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:40:08.814+0000 7fbae759e640 1 --2- 172.21.15.114:0/3167027726 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x56177bbc4800 0x56177bbc3bf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T10:40:08.814 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:40:08.814+0000 7fbae659c640 1 -- 172.21.15.114:0/3167027726 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x56177bbb9620 msgr2=0x56177bbd02f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T10:40:08.815 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:40:08.814+0000 7fbae659c640 1 --2- 172.21.15.114:0/3167027726 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x56177bbb9620 0x56177bbd02f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:40:08.815 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:40:08.814+0000 7fbae659c640 1 -- 172.21.15.114:0/3167027726 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x56177bbc4800 msgr2=0x56177bbc3bf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T10:40:08.815 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:40:08.814+0000 7fbae659c640 1 --2- 172.21.15.114:0/3167027726 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x56177bbc4800 0x56177bbc3bf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:40:08.815 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:40:08.814+0000 7fbae659c640 1 -- 172.21.15.114:0/3167027726 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x56177bbbf900 con 0x56177bbb9f40 2024-04-23T10:40:08.815 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:40:08.814+0000 7fbae659c640 1 --2- 172.21.15.114:0/3167027726 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x56177bbb9f40 0x56177bbbcaf0 secure :-1 s=READY pgs=921 cs=0 l=1 rev1=1 crypto rx=0x7fbadc0ee1c0 tx=0x7fbadc036e70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=77be67810c02b15c server_cookie=0 in_seq=0 out_seq=0 2024-04-23T10:40:08.815 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:40:08.814+0000 7fbaccff9640 1 -- 172.21.15.114:0/3167027726 <== mon.0 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbadc049040 con 0x56177bbb9f40 2024-04-23T10:40:08.815 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:40:08.814+0000 7fbaccff9640 1 -- 172.21.15.114:0/3167027726 <== mon.0 v2:172.21.15.114:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fbadc0bca60 con 0x56177bbb9f40 2024-04-23T10:40:08.815 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:40:08.815+0000 7fbaccff9640 1 -- 172.21.15.114:0/3167027726 <== mon.0 v2:172.21.15.114:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbadc0b90d0 con 0x56177bbb9f40 2024-04-23T10:40:08.818 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:40:08.815+0000 7fbaefe66740 1 -- 172.21.15.114:0/3167027726 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x56177bbb9f40 msgr2=0x56177bbbcaf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T10:40:08.819 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:40:08.815+0000 7fbaefe66740 1 --2- 172.21.15.114:0/3167027726 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x56177bbb9f40 0x56177bbbcaf0 secure :-1 s=READY pgs=921 cs=0 l=1 rev1=1 crypto rx=0x7fbadc0ee1c0 tx=0x7fbadc036e70 comp rx=0 tx=0).stop 2024-04-23T10:40:08.819 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:40:08.815+0000 7fbaefe66740 1 -- 172.21.15.114:0/3167027726 shutdown_connections 2024-04-23T10:40:08.819 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:40:08.815+0000 7fbaefe66740 1 --2- 172.21.15.114:0/3167027726 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x56177bbc4800 0x56177bbc3bf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:40:08.819 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:40:08.815+0000 7fbaefe66740 1 --2- 172.21.15.114:0/3167027726 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x56177bbb9f40 0x56177bbbcaf0 unknown :-1 s=CLOSED pgs=921 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:40:08.819 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:40:08.815+0000 7fbaefe66740 1 --2- 172.21.15.114:0/3167027726 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x56177bbb9620 0x56177bbd02f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:40:08.819 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:40:08.815+0000 7fbaefe66740 1 -- 172.21.15.114:0/3167027726 >> 172.21.15.114:0/3167027726 conn(0x56177bbb5b30 msgr2=0x56177bbb4420 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T10:40:08.819 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:40:08.815+0000 7fbaefe66740 1 -- 172.21.15.114:0/3167027726 shutdown_connections 2024-04-23T10:40:08.819 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:40:08.815+0000 7fbaefe66740 1 -- 172.21.15.114:0/3167027726 wait complete. 2024-04-23T10:40:08.819 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:40:08.816+0000 7fbaefe66740 1 -- start start 2024-04-23T10:40:08.819 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:40:08.816+0000 7fbaefe66740 1 Processor -- start 2024-04-23T10:40:08.819 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:40:08.816+0000 7fbaefe66740 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x56177bbb9620 0x56177bbc77c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T10:40:08.819 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:40:08.816+0000 7fbaefe66740 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x56177bbb9f40 0x56177bbc7d00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T10:40:08.819 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:40:08.816+0000 7fbaefe66740 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x56177bbc4800 0x56177bc38a90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T10:40:08.819 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:40:08.816+0000 7fbaefe66740 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x7fbae81ac330 con 0x56177bbb9f40 2024-04-23T10:40:08.819 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:40:08.816+0000 7fbaefe66740 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 -- 0x7fbae81ad7a0 con 0x56177bbb9620 2024-04-23T10:40:08.820 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:40:08.816+0000 7fbaefe66740 1 -- --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_getmap magic: 0 -- 0x7fbae81aca00 con 0x56177bbc4800 2024-04-23T10:40:08.820 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:40:08.816+0000 7fbaefe66740 10 client.0 _ll_register_callbacks cb 0x7fbad8009d60 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-23T10:40:08.820 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:40:08.816+0000 7fbae659c640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x56177bbb9f40 0x56177bbc7d00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T10:40:08.820 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:40:08.816+0000 7fbae6d9d640 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x56177bbb9620 0x56177bbc77c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T10:40:08.820 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:40:08.816+0000 7fbae659c640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x56177bbb9f40 0x56177bbc7d00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.114:32802/0 (socket says 172.21.15.114:32802) 2024-04-23T10:40:08.820 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:40:08.816+0000 7fbae659c640 1 -- 172.21.15.114:0/2965504309 learned_addr learned my addr 172.21.15.114:0/2965504309 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-04-23T10:40:08.820 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:40:08.816+0000 7fbae759e640 1 --2- 172.21.15.114:0/2965504309 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x56177bbc4800 0x56177bc38a90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T10:40:08.820 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:40:08.816+0000 7fbae659c640 1 -- 172.21.15.114:0/2965504309 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x56177bbc4800 msgr2=0x56177bc38a90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T10:40:08.820 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:40:08.816+0000 7fbae659c640 1 --2- 172.21.15.114:0/2965504309 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x56177bbc4800 0x56177bc38a90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:40:08.820 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:40:08.816+0000 7fbae659c640 1 -- 172.21.15.114:0/2965504309 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x56177bbb9620 msgr2=0x56177bbc77c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T10:40:08.820 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:40:08.816+0000 7fbae659c640 1 --2- 172.21.15.114:0/2965504309 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x56177bbb9620 0x56177bbc77c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:40:08.820 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:40:08.816+0000 7fbae659c640 1 -- 172.21.15.114:0/2965504309 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({osdmap=0}) -- 0x56177bc38fd0 con 0x56177bbb9f40 2024-04-23T10:40:08.820 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:40:08.816+0000 7fbae6d9d640 1 --2- 172.21.15.114:0/2965504309 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x56177bbb9620 0x56177bbc77c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-23T10:40:08.820 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:40:08.816+0000 7fbae659c640 1 --2- 172.21.15.114:0/2965504309 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x56177bbb9f40 0x56177bbc7d00 secure :-1 s=READY pgs=922 cs=0 l=1 rev1=1 crypto rx=0x7fbadc0bc760 tx=0x7fbadc03b560 comp rx=0 tx=0).ready entity=mon.0 client_cookie=da33e9f223bc27e9 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T10:40:08.820 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:40:08.816+0000 7fbacaff5640 10 client.0 ms_handle_connect on v2:172.21.15.114:3300/0 2024-04-23T10:40:08.820 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:40:08.817+0000 7fbacaff5640 1 -- 172.21.15.114:0/2965504309 <== mon.0 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbadc049030 con 0x56177bbb9f40 2024-04-23T10:40:08.821 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:40:08.817+0000 7fbacaff5640 1 -- 172.21.15.114:0/2965504309 <== mon.0 v2:172.21.15.114:3300/0 2 ==== osd_map(110..110 src has 1..110) ==== 8629+0+0 (secure 0 0 0) 0x7fbadc003d30 con 0x56177bbb9f40 2024-04-23T10:40:08.821 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:40:08.817+0000 7fbaefe66740 1 -- 172.21.15.114:0/2965504309 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fbadc0bca60 con 0x56177bbb9f40 2024-04-23T10:40:08.821 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:40:08.817+0000 7fbacaff5640 1 -- 172.21.15.114:0/2965504309 <== mon.0 v2:172.21.15.114:3300/0 3 ==== mon_get_version_reply(handle=1 version=45) ==== 24+0+0 (secure 0 0 0) 0x7fbadc0c5620 con 0x56177bbb9f40 2024-04-23T10:40:08.821 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:40:08.817+0000 7fbaefe66740 10 client.17055 fetch_fsmap learned FSMap version 45 2024-04-23T10:40:08.821 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:40:08.817+0000 7fbaefe66740 1 -- 172.21.15.114:0/2965504309 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({config=0+,fsmap=45,monmap=4+}) -- 0x56177bc3a180 con 0x56177bbb9f40 2024-04-23T10:40:08.821 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:40:08.817+0000 7fbacaff5640 1 -- 172.21.15.114:0/2965504309 <== mon.0 v2:172.21.15.114:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fbadc0bca60 con 0x56177bbb9f40 2024-04-23T10:40:08.821 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:40:08.817+0000 7fbacaff5640 1 -- 172.21.15.114:0/2965504309 <== mon.0 v2:172.21.15.114:3300/0 5 ==== fsmap(e 45) ==== 8350+0+0 (secure 0 0 0) 0x7fbadc0bdce0 con 0x56177bbb9f40 2024-04-23T10:40:08.821 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:40:08.817+0000 7fbaefe66740 10 client.17055 fetch_fsmap finished waiting for FSMap version 45 2024-04-23T10:40:08.821 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:40:08.817+0000 7fbaefe66740 10 client.17055 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-04-23T10:40:08.821 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:40:08.817+0000 7fbaefe66740 20 client.17055 populate_metadata read hostname 'smithi114' 2024-04-23T10:40:08.821 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:40:08.817+0000 7fbaefe66740 1 --2- 172.21.15.114:0/2965504309 >> [v2:172.21.15.119:6838/937297340,v1:172.21.15.119:6839/937297340] conn(0x56177bc3a7c0 0x56177bc3cbe0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T10:40:08.821 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:40:08.817+0000 7fbaefe66740 4 client.17055 mds_command: new command op to 24545 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-04-23T10:40:08.821 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:40:08.817+0000 7fbaefe66740 1 -- 172.21.15.114:0/2965504309 --> [v2:172.21.15.119:6838/937297340,v1:172.21.15.119:6839/937297340] -- command(tid 0: {"prefix": "flush journal"}) -- 0x56177bb26200 con 0x56177bc3a7c0 2024-04-23T10:40:08.821 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:40:08.817+0000 7fbae6d9d640 1 --2- 172.21.15.114:0/2965504309 >> [v2:172.21.15.119:6838/937297340,v1:172.21.15.119:6839/937297340] conn(0x56177bc3a7c0 0x56177bc3cbe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T10:40:08.821 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:40:08.818+0000 7fbacaff5640 10 client.17055 ms_handle_connect on v2:172.21.15.119:6838/937297340 2024-04-23T10:40:08.821 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:40:08.818+0000 7fbae6d9d640 1 --2- 172.21.15.114:0/2965504309 >> [v2:172.21.15.119:6838/937297340,v1:172.21.15.119:6839/937297340] conn(0x56177bc3a7c0 0x56177bc3cbe0 crc :-1 s=READY pgs=183 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=93a0ab22e406c417 server_cookie=1cac0ef45565593 in_seq=0 out_seq=0 2024-04-23T10:40:09.222 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:40:09.222+0000 7fbacaff5640 1 -- 172.21.15.114:0/2965504309 <== mds.0 v2:172.21.15.119:6838/937297340 1 ==== command_reply(tid 0: 0 ) ==== 8+0+44 (crc 0 0 0) 0x56177bb26200 con 0x56177bc3a7c0 2024-04-23T10:40:09.222 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:40:09.222+0000 7fbacaff5640 10 client.17055 handle_command_reply: tid=0 2024-04-23T10:40:09.222 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:40:09.222+0000 7fbaefe66740 1 client.17055 shutdown 2024-04-23T10:40:09.222 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:40:09.222+0000 7fbaefe66740 1 -- 172.21.15.114:0/2965504309 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x56177bbb9f40 msgr2=0x56177bbc7d00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T10:40:09.223 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:40:09.222+0000 7fbaefe66740 1 --2- 172.21.15.114:0/2965504309 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x56177bbb9f40 0x56177bbc7d00 secure :-1 s=READY pgs=922 cs=0 l=1 rev1=1 crypto rx=0x7fbadc0bc760 tx=0x7fbadc03b560 comp rx=0 tx=0).stop 2024-04-23T10:40:09.223 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:40:09.223+0000 7fbaefe66740 1 -- 172.21.15.114:0/2965504309 shutdown_connections 2024-04-23T10:40:09.223 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:40:09.223+0000 7fbaefe66740 1 --2- 172.21.15.114:0/2965504309 >> [v2:172.21.15.119:6838/937297340,v1:172.21.15.119:6839/937297340] conn(0x56177bc3a7c0 0x56177bc3cbe0 crc :-1 s=READY pgs=183 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:40:09.223 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:40:09.223+0000 7fbaefe66740 1 --2- 172.21.15.114:0/2965504309 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x56177bbc4800 0x56177bc38a90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:40:09.223 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:40:09.223+0000 7fbaefe66740 1 --2- 172.21.15.114:0/2965504309 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x56177bbb9f40 0x56177bbc7d00 unknown :-1 s=CLOSED pgs=922 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:40:09.223 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:40:09.223+0000 7fbaefe66740 1 --2- 172.21.15.114:0/2965504309 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x56177bbb9620 0x56177bbc77c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:40:09.223 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:40:09.223+0000 7fbacaff5640 0 client.17055 ms_handle_reset on v2:172.21.15.119:6838/937297340 2024-04-23T10:40:09.223 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:40:09.223+0000 7fbaefe66740 1 -- 172.21.15.114:0/2965504309 >> 172.21.15.114:0/2965504309 conn(0x56177bbb5b30 msgr2=0x56177bbc08f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T10:40:09.223 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:40:09.223+0000 7fbaefe66740 1 -- 172.21.15.114:0/2965504309 shutdown_connections 2024-04-23T10:40:09.223 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:40:09.223+0000 7fbaefe66740 1 -- 172.21.15.114:0/2965504309 wait complete. 2024-04-23T10:40:09.223 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:40:09.223+0000 7fbaefe66740 20 client.17055 trim_cache size 0 max 16384 2024-04-23T10:40:09.223 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stdout:{ 2024-04-23T10:40:09.224 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stdout: "message": "", 2024-04-23T10:40:09.224 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stdout: "return_code": 0 2024-04-23T10:40:09.224 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stdout:} 2024-04-23T10:40:09.224 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:40:09.223+0000 7fbaed144640 1 -- 172.21.15.114:0/2187082834 >> [v2:172.21.15.114:6800/1395693466,v1:172.21.15.114:6801/1395693466] conn(0x7fbad00a3dc0 msgr2=0x7fbad00a6280 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T10:40:09.224 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:40:09.223+0000 7fbaed144640 1 --2- 172.21.15.114:0/2187082834 >> [v2:172.21.15.114:6800/1395693466,v1:172.21.15.114:6801/1395693466] conn(0x7fbad00a3dc0 0x7fbad00a6280 secure :-1 s=READY pgs=145 cs=0 l=1 rev1=1 crypto rx=0x7fbae0009aa0 tx=0x7fbae0015040 comp rx=0 tx=0).stop 2024-04-23T10:40:09.224 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:40:09.223+0000 7fbaed144640 1 -- 172.21.15.114:0/2187082834 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fbae8103b00 msgr2=0x7fbae8198c50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T10:40:09.225 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:40:09.223+0000 7fbaed144640 1 --2- 172.21.15.114:0/2187082834 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fbae8103b00 0x7fbae8198c50 secure :-1 s=READY pgs=81 cs=0 l=1 rev1=1 crypto rx=0x7fbadc02aa30 tx=0x7fbadc031a20 comp rx=0 tx=0).stop 2024-04-23T10:40:09.225 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:40:09.223+0000 7fbaed144640 1 -- 172.21.15.114:0/2187082834 shutdown_connections 2024-04-23T10:40:09.225 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:40:09.223+0000 7fbaed144640 1 --2- 172.21.15.114:0/2187082834 >> [v2:172.21.15.114:6800/1395693466,v1:172.21.15.114:6801/1395693466] conn(0x7fbad00a3dc0 0x7fbad00a6280 unknown :-1 s=CLOSED pgs=145 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:40:09.225 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:40:09.223+0000 7fbaed144640 1 --2- 172.21.15.114:0/2187082834 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fbae810c520 0x7fbae819cf80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:40:09.225 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:40:09.223+0000 7fbaed144640 1 --2- 172.21.15.114:0/2187082834 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fbae8103b00 0x7fbae8198c50 unknown :-1 s=CLOSED pgs=81 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:40:09.225 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:40:09.223+0000 7fbaed144640 1 --2- 172.21.15.114:0/2187082834 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fbae81031e0 0x7fbae8198710 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:40:09.225 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:40:09.223+0000 7fbaed144640 1 -- 172.21.15.114:0/2187082834 >> 172.21.15.114:0/2187082834 conn(0x7fbae8077fd0 msgr2=0x7fbae80fdfc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T10:40:09.225 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:40:09.223+0000 7fbaed144640 1 -- 172.21.15.114:0/2187082834 shutdown_connections 2024-04-23T10:40:09.225 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:40:09.224+0000 7fbaed144640 1 -- 172.21.15.114:0/2187082834 wait complete. 2024-04-23T10:40:09.496 INFO:tasks.workunit.client.0.smithi114.stdout: 10 5817 31.23 MB/sec warmup 15 sec latency 2453.889 ms 2024-04-23T10:40:09.754 INFO:journalctl@ceph.mon.a.smithi114.stdout:Apr 23 10:40:09 smithi114 ceph-mon[29255]: pgmap v426: 97 pgs: 97 active+clean; 243 MiB data, 3.0 GiB used, 1.0 TiB / 1.0 TiB avail; 1.5 MiB/s rd, 4.2 MiB/s wr, 463 op/s 2024-04-23T10:40:09.754 INFO:journalctl@ceph.mon.a.smithi114.stdout:Apr 23 10:40:09 smithi114 ceph-mon[29255]: osdmap e110: 12 total, 12 up, 12 in 2024-04-23T10:40:09.754 INFO:journalctl@ceph.mon.a.smithi114.stdout:Apr 23 10:40:09 smithi114 ceph-mon[29255]: daemon mds.i restarted 2024-04-23T10:40:09.828 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 23 10:40:09 smithi204 ceph-mon[32649]: pgmap v426: 97 pgs: 97 active+clean; 243 MiB data, 3.0 GiB used, 1.0 TiB / 1.0 TiB avail; 1.5 MiB/s rd, 4.2 MiB/s wr, 463 op/s 2024-04-23T10:40:09.828 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 23 10:40:09 smithi204 ceph-mon[32649]: osdmap e110: 12 total, 12 up, 12 in 2024-04-23T10:40:09.828 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 23 10:40:09 smithi204 ceph-mon[32649]: daemon mds.i restarted 2024-04-23T10:40:09.828 INFO:journalctl@ceph.mds.i.smithi204.stdout:Apr 23 10:40:09 smithi204 ceph-aa240572-0159-11ef-bc93-c7b262605968-mds-i[63304]: ignoring --setuser ceph since I am not root 2024-04-23T10:40:09.828 INFO:journalctl@ceph.mds.i.smithi204.stdout:Apr 23 10:40:09 smithi204 ceph-aa240572-0159-11ef-bc93-c7b262605968-mds-i[63304]: ignoring --setgroup ceph since I am not root 2024-04-23T10:40:09.828 INFO:journalctl@ceph.mds.i.smithi204.stdout:Apr 23 10:40:09 smithi204 ceph-mds[63308]: -- 172.21.15.204:0/1828699888 <== mon.1 v2:172.21.15.204:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 194+0+0 (secure 0 0 0) 0x5555e2583a40 con 0x5555e2557000 2024-04-23T10:40:09.828 INFO:journalctl@ceph.mds.i.smithi204.stdout:Apr 23 10:40:09 smithi204 ceph-aa240572-0159-11ef-bc93-c7b262605968-mds-i[63304]: starting mds.i at 2024-04-23T10:40:09.994 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 23 10:40:09 smithi119 ceph-mon[32608]: pgmap v426: 97 pgs: 97 active+clean; 243 MiB data, 3.0 GiB used, 1.0 TiB / 1.0 TiB avail; 1.5 MiB/s rd, 4.2 MiB/s wr, 463 op/s 2024-04-23T10:40:09.994 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 23 10:40:09 smithi119 ceph-mon[32608]: osdmap e110: 12 total, 12 up, 12 in 2024-04-23T10:40:09.994 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 23 10:40:09 smithi119 ceph-mon[32608]: daemon mds.i restarted 2024-04-23T10:40:10.496 INFO:tasks.workunit.client.0.smithi114.stdout: 10 6331 32.28 MB/sec warmup 16 sec latency 102.198 ms 2024-04-23T10:40:10.754 INFO:journalctl@ceph.mon.a.smithi114.stdout:Apr 23 10:40:10 smithi114 ceph-mon[29255]: osdmap e111: 12 total, 12 up, 12 in 2024-04-23T10:40:10.754 INFO:journalctl@ceph.mon.a.smithi114.stdout:Apr 23 10:40:10 smithi114 ceph-mon[29255]: mds.? [v2:172.21.15.204:6838/3251736831,v1:172.21.15.204:6839/3251736831] up:boot 2024-04-23T10:40:10.754 INFO:journalctl@ceph.mon.a.smithi114.stdout:Apr 23 10:40:10 smithi114 ceph-mon[29255]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=e=up:active} 2 up:standby-replay 7 up:standby 2024-04-23T10:40:10.754 INFO:journalctl@ceph.mon.a.smithi114.stdout:Apr 23 10:40:10 smithi114 ceph-mon[29255]: from='mgr.15291 172.21.15.114:0/1095789187' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "i"}]: dispatch 2024-04-23T10:40:10.754 INFO:journalctl@ceph.mon.a.smithi114.stdout:Apr 23 10:40:10 smithi114 ceph-mon[29255]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=e=up:active} 3 up:standby-replay 6 up:standby 2024-04-23T10:40:10.828 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 23 10:40:10 smithi204 ceph-mon[32649]: osdmap e111: 12 total, 12 up, 12 in 2024-04-23T10:40:10.828 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 23 10:40:10 smithi204 ceph-mon[32649]: mds.? [v2:172.21.15.204:6838/3251736831,v1:172.21.15.204:6839/3251736831] up:boot 2024-04-23T10:40:10.828 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 23 10:40:10 smithi204 ceph-mon[32649]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=e=up:active} 2 up:standby-replay 7 up:standby 2024-04-23T10:40:10.828 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 23 10:40:10 smithi204 ceph-mon[32649]: from='mgr.15291 172.21.15.114:0/1095789187' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "i"}]: dispatch 2024-04-23T10:40:10.828 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 23 10:40:10 smithi204 ceph-mon[32649]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=e=up:active} 3 up:standby-replay 6 up:standby 2024-04-23T10:40:10.991 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 23 10:40:10 smithi119 ceph-mon[32608]: osdmap e111: 12 total, 12 up, 12 in 2024-04-23T10:40:10.992 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 23 10:40:10 smithi119 ceph-mon[32608]: mds.? [v2:172.21.15.204:6838/3251736831,v1:172.21.15.204:6839/3251736831] up:boot 2024-04-23T10:40:10.992 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 23 10:40:10 smithi119 ceph-mon[32608]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=e=up:active} 2 up:standby-replay 7 up:standby 2024-04-23T10:40:10.992 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 23 10:40:10 smithi119 ceph-mon[32608]: from='mgr.15291 172.21.15.114:0/1095789187' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "i"}]: dispatch 2024-04-23T10:40:10.992 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 23 10:40:10 smithi119 ceph-mon[32608]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=e=up:active} 3 up:standby-replay 6 up:standby 2024-04-23T10:40:11.496 INFO:tasks.workunit.client.0.smithi114.stdout: 10 6853 33.01 MB/sec warmup 17 sec latency 82.186 ms 2024-04-23T10:40:12.492 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 23 10:40:12 smithi119 ceph-mon[32608]: pgmap v429: 97 pgs: 97 active+clean; 243 MiB data, 3.0 GiB used, 1.0 TiB / 1.0 TiB avail; 1.3 MiB/s rd, 3.2 MiB/s wr, 377 op/s 2024-04-23T10:40:12.496 INFO:tasks.workunit.client.0.smithi114.stdout: 10 7066 31.70 MB/sec warmup 18 sec latency 74.358 ms 2024-04-23T10:40:12.504 INFO:journalctl@ceph.mon.a.smithi114.stdout:Apr 23 10:40:12 smithi114 ceph-mon[29255]: pgmap v429: 97 pgs: 97 active+clean; 243 MiB data, 3.0 GiB used, 1.0 TiB / 1.0 TiB avail; 1.3 MiB/s rd, 3.2 MiB/s wr, 377 op/s 2024-04-23T10:40:12.578 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 23 10:40:12 smithi204 ceph-mon[32649]: pgmap v429: 97 pgs: 97 active+clean; 243 MiB data, 3.0 GiB used, 1.0 TiB / 1.0 TiB avail; 1.3 MiB/s rd, 3.2 MiB/s wr, 377 op/s 2024-04-23T10:40:13.496 INFO:tasks.workunit.client.0.smithi114.stdout: 10 7239 30.19 MB/sec warmup 19 sec latency 99.590 ms 2024-04-23T10:40:14.492 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 23 10:40:14 smithi119 ceph-mon[32608]: pgmap v430: 97 pgs: 97 active+clean; 234 MiB data, 3.0 GiB used, 1.0 TiB / 1.0 TiB avail; 1.9 MiB/s rd, 4.8 MiB/s wr, 757 op/s 2024-04-23T10:40:14.496 INFO:tasks.workunit.client.0.smithi114.stdout: 10 7450 28.82 MB/sec warmup 20 sec latency 147.436 ms 2024-04-23T10:40:14.503 INFO:journalctl@ceph.mon.a.smithi114.stdout:Apr 23 10:40:14 smithi114 ceph-mon[29255]: pgmap v430: 97 pgs: 97 active+clean; 234 MiB data, 3.0 GiB used, 1.0 TiB / 1.0 TiB avail; 1.9 MiB/s rd, 4.8 MiB/s wr, 757 op/s 2024-04-23T10:40:14.577 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 23 10:40:14 smithi204 ceph-mon[32649]: pgmap v430: 97 pgs: 97 active+clean; 234 MiB data, 3.0 GiB used, 1.0 TiB / 1.0 TiB avail; 1.9 MiB/s rd, 4.8 MiB/s wr, 757 op/s 2024-04-23T10:40:15.496 INFO:tasks.workunit.client.0.smithi114.stdout: 10 7740 27.92 MB/sec warmup 21 sec latency 177.425 ms 2024-04-23T10:40:16.492 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 23 10:40:16 smithi119 ceph-mon[32608]: pgmap v431: 97 pgs: 97 active+clean; 234 MiB data, 3.0 GiB used, 1.0 TiB / 1.0 TiB avail; 954 KiB/s rd, 2.2 MiB/s wr, 456 op/s 2024-04-23T10:40:16.496 INFO:tasks.workunit.client.0.smithi114.stdout: 10 8206 27.84 MB/sec warmup 22 sec latency 119.168 ms 2024-04-23T10:40:16.504 INFO:journalctl@ceph.mon.a.smithi114.stdout:Apr 23 10:40:16 smithi114 ceph-mon[29255]: pgmap v431: 97 pgs: 97 active+clean; 234 MiB data, 3.0 GiB used, 1.0 TiB / 1.0 TiB avail; 954 KiB/s rd, 2.2 MiB/s wr, 456 op/s 2024-04-23T10:40:16.578 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 23 10:40:16 smithi204 ceph-mon[32649]: pgmap v431: 97 pgs: 97 active+clean; 234 MiB data, 3.0 GiB used, 1.0 TiB / 1.0 TiB avail; 954 KiB/s rd, 2.2 MiB/s wr, 456 op/s 2024-04-23T10:40:17.497 INFO:tasks.workunit.client.0.smithi114.stdout: 10 8878 27.55 MB/sec warmup 23 sec latency 108.412 ms 2024-04-23T10:40:18.492 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 23 10:40:18 smithi119 ceph-mon[32608]: pgmap v432: 97 pgs: 97 active+clean; 245 MiB data, 3.0 GiB used, 1.0 TiB / 1.0 TiB avail; 1.8 MiB/s rd, 3.9 MiB/s wr, 581 op/s 2024-04-23T10:40:18.496 INFO:tasks.workunit.client.0.smithi114.stdout: 10 9344 27.43 MB/sec warmup 24 sec latency 103.801 ms 2024-04-23T10:40:18.503 INFO:journalctl@ceph.mon.a.smithi114.stdout:Apr 23 10:40:18 smithi114 ceph-mon[29255]: pgmap v432: 97 pgs: 97 active+clean; 245 MiB data, 3.0 GiB used, 1.0 TiB / 1.0 TiB avail; 1.8 MiB/s rd, 3.9 MiB/s wr, 581 op/s 2024-04-23T10:40:18.578 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 23 10:40:18 smithi204 ceph-mon[32649]: pgmap v432: 97 pgs: 97 active+clean; 245 MiB data, 3.0 GiB used, 1.0 TiB / 1.0 TiB avail; 1.8 MiB/s rd, 3.9 MiB/s wr, 581 op/s 2024-04-23T10:40:19.496 INFO:tasks.workunit.client.0.smithi114.stdout: 10 9803 28.16 MB/sec warmup 25 sec latency 243.309 ms 2024-04-23T10:40:20.492 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 23 10:40:20 smithi119 ceph-mon[32608]: pgmap v433: 97 pgs: 97 active+clean; 245 MiB data, 3.0 GiB used, 1.0 TiB / 1.0 TiB avail; 1.4 MiB/s rd, 3.2 MiB/s wr, 465 op/s 2024-04-23T10:40:20.492 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 23 10:40:20 smithi119 ceph-mon[32608]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=e=up:active} 3 up:standby-replay 6 up:standby 2024-04-23T10:40:20.496 INFO:tasks.workunit.client.0.smithi114.stdout: 10 10422 28.98 MB/sec warmup 26 sec latency 76.601 ms 2024-04-23T10:40:20.503 INFO:journalctl@ceph.mon.a.smithi114.stdout:Apr 23 10:40:20 smithi114 ceph-mon[29255]: pgmap v433: 97 pgs: 97 active+clean; 245 MiB data, 3.0 GiB used, 1.0 TiB / 1.0 TiB avail; 1.4 MiB/s rd, 3.2 MiB/s wr, 465 op/s 2024-04-23T10:40:20.504 INFO:journalctl@ceph.mon.a.smithi114.stdout:Apr 23 10:40:20 smithi114 ceph-mon[29255]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=e=up:active} 3 up:standby-replay 6 up:standby 2024-04-23T10:40:20.577 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 23 10:40:20 smithi204 ceph-mon[32649]: pgmap v433: 97 pgs: 97 active+clean; 245 MiB data, 3.0 GiB used, 1.0 TiB / 1.0 TiB avail; 1.4 MiB/s rd, 3.2 MiB/s wr, 465 op/s 2024-04-23T10:40:20.578 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 23 10:40:20 smithi204 ceph-mon[32649]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=e=up:active} 3 up:standby-replay 6 up:standby 2024-04-23T10:40:21.496 INFO:tasks.workunit.client.0.smithi114.stdout: 10 10720 28.27 MB/sec warmup 27 sec latency 111.407 ms 2024-04-23T10:40:22.319 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:40:22.318+0000 7fe4d5336640 1 -- 172.21.15.114:0/2825352535 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fe4d0101090 msgr2=0x7fe4d0106bc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T10:40:22.320 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:40:22.318+0000 7fe4d5336640 1 --2- 172.21.15.114:0/2825352535 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fe4d0101090 0x7fe4d0106bc0 secure :-1 s=READY pgs=923 cs=0 l=1 rev1=1 crypto rx=0x7fe4c4009900 tx=0x7fe4c402a410 comp rx=0 tx=0).stop 2024-04-23T10:40:22.320 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:40:22.318+0000 7fe4d5336640 1 -- 172.21.15.114:0/2825352535 shutdown_connections 2024-04-23T10:40:22.320 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:40:22.318+0000 7fe4d5336640 1 --2- 172.21.15.114:0/2825352535 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fe4d01072a0 0x7fe4d010ee70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:40:22.320 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:40:22.318+0000 7fe4d5336640 1 --2- 172.21.15.114:0/2825352535 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fe4d0101090 0x7fe4d0106bc0 unknown :-1 s=CLOSED pgs=923 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:40:22.320 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:40:22.318+0000 7fe4d5336640 1 --2- 172.21.15.114:0/2825352535 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fe4d01006e0 0x7fe4d0100ac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:40:22.320 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:40:22.318+0000 7fe4d5336640 1 -- 172.21.15.114:0/2825352535 >> 172.21.15.114:0/2825352535 conn(0x7fe4d0077390 msgr2=0x7fe4d00fe760 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T10:40:22.320 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:40:22.318+0000 7fe4d5336640 1 -- 172.21.15.114:0/2825352535 shutdown_connections 2024-04-23T10:40:22.320 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:40:22.318+0000 7fe4d5336640 1 -- 172.21.15.114:0/2825352535 wait complete. 2024-04-23T10:40:22.320 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:40:22.318+0000 7fe4d5336640 1 Processor -- start 2024-04-23T10:40:22.320 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:40:22.319+0000 7fe4d5336640 1 -- start start 2024-04-23T10:40:22.320 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:40:22.319+0000 7fe4d5336640 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fe4d01006e0 0x7fe4d0101650 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T10:40:22.320 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:40:22.319+0000 7fe4d5336640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fe4d01072a0 0x7fe4d0101b90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T10:40:22.321 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:40:22.319+0000 7fe4d5336640 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fe4d01020d0 0x7fe4d01a6930 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T10:40:22.321 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:40:22.319+0000 7fe4d5336640 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x7fe4d01121e0 con 0x7fe4d01072a0 2024-04-23T10:40:22.321 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:40:22.319+0000 7fe4d5336640 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 -- 0x7fe4d0112060 con 0x7fe4d01020d0 2024-04-23T10:40:22.321 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:40:22.319+0000 7fe4d5336640 1 -- --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_getmap magic: 0 -- 0x7fe4d0112360 con 0x7fe4d01006e0 2024-04-23T10:40:22.321 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:40:22.319+0000 7fe4ce7fc640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fe4d01072a0 0x7fe4d0101b90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T10:40:22.321 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:40:22.319+0000 7fe4ce7fc640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fe4d01072a0 0x7fe4d0101b90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.114:58586/0 (socket says 172.21.15.114:58586) 2024-04-23T10:40:22.321 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:40:22.319+0000 7fe4ce7fc640 1 -- 172.21.15.114:0/2553678170 learned_addr learned my addr 172.21.15.114:0/2553678170 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-04-23T10:40:22.321 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:40:22.319+0000 7fe4cf7fe640 1 --2- 172.21.15.114:0/2553678170 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fe4d01020d0 0x7fe4d01a6930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T10:40:22.321 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:40:22.319+0000 7fe4ceffd640 1 --2- 172.21.15.114:0/2553678170 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fe4d01006e0 0x7fe4d0101650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T10:40:22.321 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:40:22.319+0000 7fe4cf7fe640 1 -- 172.21.15.114:0/2553678170 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fe4d01006e0 msgr2=0x7fe4d0101650 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T10:40:22.321 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:40:22.319+0000 7fe4cf7fe640 1 --2- 172.21.15.114:0/2553678170 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fe4d01006e0 0x7fe4d0101650 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:40:22.321 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:40:22.319+0000 7fe4cf7fe640 1 -- 172.21.15.114:0/2553678170 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fe4d01072a0 msgr2=0x7fe4d0101b90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T10:40:22.322 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:40:22.319+0000 7fe4cf7fe640 1 --2- 172.21.15.114:0/2553678170 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fe4d01072a0 0x7fe4d0101b90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:40:22.322 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:40:22.319+0000 7fe4cf7fe640 1 -- 172.21.15.114:0/2553678170 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe4d01a6fd0 con 0x7fe4d01020d0 2024-04-23T10:40:22.322 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:40:22.319+0000 7fe4ce7fc640 1 --2- 172.21.15.114:0/2553678170 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fe4d01072a0 0x7fe4d0101b90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-23T10:40:22.322 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:40:22.320+0000 7fe4cf7fe640 1 --2- 172.21.15.114:0/2553678170 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fe4d01020d0 0x7fe4d01a6930 secure :-1 s=READY pgs=102 cs=0 l=1 rev1=1 crypto rx=0x7fe4c8009fd0 tx=0x7fe4c800ef10 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T10:40:22.322 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:40:22.320+0000 7fe4b7fff640 1 -- 172.21.15.114:0/2553678170 <== mon.2 v2:172.21.15.119:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe4c800ce20 con 0x7fe4d01020d0 2024-04-23T10:40:22.322 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:40:22.320+0000 7fe4b7fff640 1 -- 172.21.15.114:0/2553678170 <== mon.2 v2:172.21.15.119:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fe4c8005020 con 0x7fe4d01020d0 2024-04-23T10:40:22.322 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:40:22.320+0000 7fe4d5336640 1 -- 172.21.15.114:0/2553678170 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fe4d01a72c0 con 0x7fe4d01020d0 2024-04-23T10:40:22.322 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:40:22.320+0000 7fe4b7fff640 1 -- 172.21.15.114:0/2553678170 <== mon.2 v2:172.21.15.119:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe4c8010690 con 0x7fe4d01020d0 2024-04-23T10:40:22.322 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:40:22.320+0000 7fe4d5336640 1 -- 172.21.15.114:0/2553678170 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fe4d01a77c0 con 0x7fe4d01020d0 2024-04-23T10:40:22.328 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:40:22.323+0000 7fe4d8051740 1 Processor -- start 2024-04-23T10:40:22.329 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:40:22.323+0000 7fe4b7fff640 1 -- 172.21.15.114:0/2553678170 <== mon.2 v2:172.21.15.119:3300/0 4 ==== mgrmap(e 26) ==== 138778+0+0 (secure 0 0 0) 0x7fe4c80054d0 con 0x7fe4d01020d0 2024-04-23T10:40:22.329 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:40:22.323+0000 7fe4d8051740 1 -- start start 2024-04-23T10:40:22.329 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:40:22.323+0000 7fe4d8051740 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x56479d81c620 0x56479d81ca00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T10:40:22.329 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:40:22.323+0000 7fe4d8051740 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x56479d81cf40 0x56479d8272c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T10:40:22.329 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:40:22.323+0000 7fe4d8051740 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x56479d827800 0x56479d829bf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T10:40:22.329 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:40:22.323+0000 7fe4d8051740 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x56479d781970 con 0x56479d81cf40 2024-04-23T10:40:22.329 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:40:22.323+0000 7fe4d8051740 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 -- 0x56479d71e4a0 con 0x56479d81c620 2024-04-23T10:40:22.329 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:40:22.323+0000 7fe4d8051740 1 -- --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_getmap magic: 0 -- 0x56479d77d930 con 0x56479d827800 2024-04-23T10:40:22.329 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:40:22.323+0000 7fe4ceffd640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x56479d81cf40 0x56479d8272c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T10:40:22.329 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:40:22.323+0000 7fe4ceffd640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x56479d81cf40 0x56479d8272c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.114:58602/0 (socket says 172.21.15.114:58602) 2024-04-23T10:40:22.329 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:40:22.323+0000 7fe4ceffd640 1 -- 172.21.15.114:0/1370702002 learned_addr learned my addr 172.21.15.114:0/1370702002 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-04-23T10:40:22.329 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:40:22.323+0000 7fe4ce7fc640 1 --2- 172.21.15.114:0/1370702002 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x56479d81c620 0x56479d81ca00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T10:40:22.330 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:40:22.324+0000 7fe4ce7fc640 1 --2- 172.21.15.114:0/1370702002 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x56479d827800 0x56479d829bf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T10:40:22.330 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:40:22.324+0000 7fe4ceffd640 1 -- 172.21.15.114:0/1370702002 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x56479d827800 msgr2=0x56479d829bf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T10:40:22.330 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:40:22.324+0000 7fe4ceffd640 1 --2- 172.21.15.114:0/1370702002 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x56479d827800 0x56479d829bf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:40:22.330 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:40:22.324+0000 7fe4ceffd640 1 -- 172.21.15.114:0/1370702002 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x56479d81c620 msgr2=0x56479d81ca00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T10:40:22.330 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:40:22.324+0000 7fe4ceffd640 1 --2- 172.21.15.114:0/1370702002 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x56479d81c620 0x56479d81ca00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:40:22.330 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:40:22.324+0000 7fe4ceffd640 1 -- 172.21.15.114:0/1370702002 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x56479d494210 con 0x56479d81cf40 2024-04-23T10:40:22.330 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:40:22.324+0000 7fe4ce7fc640 1 --2- 172.21.15.114:0/1370702002 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x56479d81c620 0x56479d81ca00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-23T10:40:22.330 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:40:22.324+0000 7fe4b7fff640 1 --2- 172.21.15.114:0/2553678170 >> [v2:172.21.15.114:6800/1395693466,v1:172.21.15.114:6801/1395693466] conn(0x7fe4b80a3dc0 0x7fe4b80a6280 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T10:40:22.330 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:40:22.324+0000 7fe4ceffd640 1 --2- 172.21.15.114:0/1370702002 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x56479d81cf40 0x56479d8272c0 secure :-1 s=READY pgs=924 cs=0 l=1 rev1=1 crypto rx=0x7fe4c0009d20 tx=0x7fe4c0011810 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7a838cdebb1e9b60 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T10:40:22.330 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:40:22.324+0000 7fe4ce7fc640 1 --2- 172.21.15.114:0/2553678170 >> [v2:172.21.15.114:6800/1395693466,v1:172.21.15.114:6801/1395693466] conn(0x7fe4b80a3dc0 0x7fe4b80a6280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T10:40:22.330 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:40:22.324+0000 7fe4b7fff640 1 -- 172.21.15.114:0/2553678170 <== mon.2 v2:172.21.15.119:3300/0 5 ==== osd_map(111..111 src has 1..111) ==== 8585+0+0 (secure 0 0 0) 0x7fe4c80d0ec0 con 0x7fe4d01020d0 2024-04-23T10:40:22.331 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:40:22.324+0000 7fe4b57fa640 1 -- 172.21.15.114:0/1370702002 <== mon.0 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe4c001e070 con 0x56479d81cf40 2024-04-23T10:40:22.331 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:40:22.324+0000 7fe4b57fa640 1 -- 172.21.15.114:0/1370702002 <== mon.0 v2:172.21.15.114:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fe4c0019ae0 con 0x56479d81cf40 2024-04-23T10:40:22.331 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:40:22.324+0000 7fe4b57fa640 1 -- 172.21.15.114:0/1370702002 <== mon.0 v2:172.21.15.114:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe4c000ea80 con 0x56479d81cf40 2024-04-23T10:40:22.331 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:40:22.325+0000 7fe4d8051740 1 -- 172.21.15.114:0/1370702002 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x56479d81cf40 msgr2=0x56479d8272c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T10:40:22.331 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:40:22.325+0000 7fe4d8051740 1 --2- 172.21.15.114:0/1370702002 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x56479d81cf40 0x56479d8272c0 secure :-1 s=READY pgs=924 cs=0 l=1 rev1=1 crypto rx=0x7fe4c0009d20 tx=0x7fe4c0011810 comp rx=0 tx=0).stop 2024-04-23T10:40:22.331 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:40:22.325+0000 7fe4ce7fc640 1 --2- 172.21.15.114:0/2553678170 >> [v2:172.21.15.114:6800/1395693466,v1:172.21.15.114:6801/1395693466] conn(0x7fe4b80a3dc0 0x7fe4b80a6280 secure :-1 s=READY pgs=147 cs=0 l=1 rev1=1 crypto rx=0x7fe4c4032fd0 tx=0x7fe4c4005bc0 comp rx=0 tx=0).ready entity=mgr.15291 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T10:40:22.331 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:40:22.325+0000 7fe4d8051740 1 -- 172.21.15.114:0/1370702002 shutdown_connections 2024-04-23T10:40:22.331 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:40:22.325+0000 7fe4d8051740 1 --2- 172.21.15.114:0/1370702002 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x56479d827800 0x56479d829bf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:40:22.331 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:40:22.325+0000 7fe4d8051740 1 --2- 172.21.15.114:0/1370702002 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x56479d81cf40 0x56479d8272c0 unknown :-1 s=CLOSED pgs=924 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:40:22.331 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:40:22.325+0000 7fe4d8051740 1 --2- 172.21.15.114:0/1370702002 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x56479d81c620 0x56479d81ca00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:40:22.331 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:40:22.325+0000 7fe4d8051740 1 -- 172.21.15.114:0/1370702002 >> 172.21.15.114:0/1370702002 conn(0x56479d818b30 msgr2=0x56479d818f10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T10:40:22.331 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:40:22.325+0000 7fe4d8051740 1 -- 172.21.15.114:0/1370702002 shutdown_connections 2024-04-23T10:40:22.331 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:40:22.325+0000 7fe4d8051740 1 -- 172.21.15.114:0/1370702002 wait complete. 2024-04-23T10:40:22.332 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:40:22.325+0000 7fe4d8051740 1 -- start start 2024-04-23T10:40:22.332 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:40:22.326+0000 7fe4d8051740 1 Processor -- start 2024-04-23T10:40:22.332 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:40:22.326+0000 7fe4d8051740 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x56479d81c620 0x56479d8270f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T10:40:22.332 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:40:22.326+0000 7fe4d8051740 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x56479d81cf40 0x56479d8243e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T10:40:22.332 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:40:22.326+0000 7fe4d8051740 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x56479d827800 0x56479d824920 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T10:40:22.332 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:40:22.326+0000 7fe4d8051740 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x56479d82a430 con 0x56479d81c620 2024-04-23T10:40:22.332 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:40:22.326+0000 7fe4d8051740 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 -- 0x56479d71c0b0 con 0x56479d81cf40 2024-04-23T10:40:22.332 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:40:22.326+0000 7fe4d8051740 1 -- --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_getmap magic: 0 -- 0x56479d82a5b0 con 0x56479d827800 2024-04-23T10:40:22.332 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:40:22.326+0000 7fe4d8051740 10 client.0 _ll_register_callbacks cb 0x56479d709d30 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-23T10:40:22.333 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:40:22.326+0000 7fe4ceffd640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x56479d81c620 0x56479d8270f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T10:40:22.333 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:40:22.326+0000 7fe4ceffd640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x56479d81c620 0x56479d8270f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.114:58608/0 (socket says 172.21.15.114:58608) 2024-04-23T10:40:22.333 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:40:22.326+0000 7fe4ceffd640 1 -- 172.21.15.114:0/592947231 learned_addr learned my addr 172.21.15.114:0/592947231 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-04-23T10:40:22.333 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:40:22.326+0000 7fe4ce7fc640 1 --2- 172.21.15.114:0/592947231 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x56479d81cf40 0x56479d8243e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T10:40:22.333 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:40:22.326+0000 7fe4cf7fe640 1 --2- 172.21.15.114:0/592947231 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x56479d827800 0x56479d824920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T10:40:22.333 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:40:22.326+0000 7fe4ceffd640 1 -- 172.21.15.114:0/592947231 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x56479d827800 msgr2=0x56479d824920 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T10:40:22.333 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:40:22.326+0000 7fe4ceffd640 1 --2- 172.21.15.114:0/592947231 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x56479d827800 0x56479d824920 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:40:22.333 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:40:22.326+0000 7fe4ceffd640 1 -- 172.21.15.114:0/592947231 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x56479d81cf40 msgr2=0x56479d8243e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T10:40:22.333 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:40:22.326+0000 7fe4ceffd640 1 --2- 172.21.15.114:0/592947231 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x56479d81cf40 0x56479d8243e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:40:22.333 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:40:22.326+0000 7fe4ceffd640 1 -- 172.21.15.114:0/592947231 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({osdmap=0}) -- 0x56479d560620 con 0x56479d81c620 2024-04-23T10:40:22.333 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:40:22.326+0000 7fe4ceffd640 1 --2- 172.21.15.114:0/592947231 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x56479d81c620 0x56479d8270f0 secure :-1 s=READY pgs=925 cs=0 l=1 rev1=1 crypto rx=0x7fe4c00058c0 tx=0x7fe4c0004270 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8eda6d8b11342941 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T10:40:22.333 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:40:22.326+0000 7fe4b2ff5640 10 client.0 ms_handle_connect on v2:172.21.15.114:3300/0 2024-04-23T10:40:22.333 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:40:22.327+0000 7fe4b2ff5640 1 -- 172.21.15.114:0/592947231 <== mon.0 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe4c0027020 con 0x56479d81c620 2024-04-23T10:40:22.333 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:40:22.327+0000 7fe4b2ff5640 1 -- 172.21.15.114:0/592947231 <== mon.0 v2:172.21.15.114:3300/0 2 ==== osd_map(111..111 src has 1..111) ==== 8585+0+0 (secure 0 0 0) 0x7fe4c0019630 con 0x56479d81c620 2024-04-23T10:40:22.334 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:40:22.327+0000 7fe4d8051740 1 -- 172.21.15.114:0/592947231 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fe4c0019ae0 con 0x56479d81c620 2024-04-23T10:40:22.334 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:40:22.327+0000 7fe4b2ff5640 1 -- 172.21.15.114:0/592947231 <== mon.0 v2:172.21.15.114:3300/0 3 ==== mon_get_version_reply(handle=1 version=48) ==== 24+0+0 (secure 0 0 0) 0x7fe4c0032050 con 0x56479d81c620 2024-04-23T10:40:22.334 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:40:22.327+0000 7fe4d8051740 10 client.17073 fetch_fsmap learned FSMap version 48 2024-04-23T10:40:22.334 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:40:22.327+0000 7fe4d8051740 1 -- 172.21.15.114:0/592947231 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({config=0+,fsmap=48,monmap=4+}) -- 0x56479d836300 con 0x56479d81c620 2024-04-23T10:40:22.334 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:40:22.327+0000 7fe4b2ff5640 1 -- 172.21.15.114:0/592947231 <== mon.0 v2:172.21.15.114:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fe4c0019ae0 con 0x56479d81c620 2024-04-23T10:40:22.334 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:40:22.327+0000 7fe4b2ff5640 1 -- 172.21.15.114:0/592947231 <== mon.0 v2:172.21.15.114:3300/0 5 ==== fsmap(e 48) ==== 8354+0+0 (secure 0 0 0) 0x7fe4c000eb00 con 0x56479d81c620 2024-04-23T10:40:22.334 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:40:22.327+0000 7fe4d8051740 10 client.17073 fetch_fsmap finished waiting for FSMap version 48 2024-04-23T10:40:22.334 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:40:22.327+0000 7fe4d8051740 10 client.17073 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-04-23T10:40:22.334 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:40:22.327+0000 7fe4d8051740 20 client.17073 populate_metadata read hostname 'smithi114' 2024-04-23T10:40:22.334 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:40:22.327+0000 7fe4d8051740 1 --2- 172.21.15.114:0/592947231 >> [v2:172.21.15.119:6838/937297340,v1:172.21.15.119:6839/937297340] conn(0x56479d837700 0x56479d839af0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T10:40:22.334 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:40:22.327+0000 7fe4d8051740 4 client.17073 mds_command: new command op to 24545 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-23T10:40:22.334 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:40:22.327+0000 7fe4d8051740 1 -- 172.21.15.114:0/592947231 --> [v2:172.21.15.119:6838/937297340,v1:172.21.15.119:6839/937297340] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x56479d78f390 con 0x56479d837700 2024-04-23T10:40:22.334 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:40:22.327+0000 7fe4ce7fc640 1 --2- 172.21.15.114:0/592947231 >> [v2:172.21.15.119:6838/937297340,v1:172.21.15.119:6839/937297340] conn(0x56479d837700 0x56479d839af0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T10:40:22.335 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:40:22.328+0000 7fe4ce7fc640 1 --2- 172.21.15.114:0/592947231 >> [v2:172.21.15.119:6838/937297340,v1:172.21.15.119:6839/937297340] conn(0x56479d837700 0x56479d839af0 crc :-1 s=READY pgs=184 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=2976a179b0ec8773 server_cookie=bb0d97fd06b73331 in_seq=0 out_seq=0 2024-04-23T10:40:22.335 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:40:22.328+0000 7fe4b2ff5640 10 client.17073 ms_handle_connect on v2:172.21.15.119:6838/937297340 2024-04-23T10:40:22.336 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:40:22.332+0000 7fe4b2ff5640 1 -- 172.21.15.114:0/592947231 <== mds.0 v2:172.21.15.119:6838/937297340 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x56479d78f390 con 0x56479d837700 2024-04-23T10:40:22.336 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:40:22.332+0000 7fe4b2ff5640 10 client.17073 handle_command_reply: tid=0 2024-04-23T10:40:22.336 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:40:22.332+0000 7fe4d8051740 1 client.17073 shutdown 2024-04-23T10:40:22.336 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:40:22.332+0000 7fe4d8051740 1 -- 172.21.15.114:0/592947231 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x56479d81c620 msgr2=0x56479d8270f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T10:40:22.336 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:40:22.332+0000 7fe4d8051740 1 --2- 172.21.15.114:0/592947231 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x56479d81c620 0x56479d8270f0 secure :-1 s=READY pgs=925 cs=0 l=1 rev1=1 crypto rx=0x7fe4c00058c0 tx=0x7fe4c0004270 comp rx=0 tx=0).stop 2024-04-23T10:40:22.336 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:40:22.332+0000 7fe4d8051740 1 -- 172.21.15.114:0/592947231 shutdown_connections 2024-04-23T10:40:22.336 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:40:22.332+0000 7fe4d8051740 1 --2- 172.21.15.114:0/592947231 >> [v2:172.21.15.119:6838/937297340,v1:172.21.15.119:6839/937297340] conn(0x56479d837700 0x56479d839af0 crc :-1 s=READY pgs=184 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:40:22.336 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:40:22.332+0000 7fe4d8051740 1 --2- 172.21.15.114:0/592947231 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x56479d827800 0x56479d824920 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:40:22.336 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:40:22.332+0000 7fe4d8051740 1 --2- 172.21.15.114:0/592947231 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x56479d81cf40 0x56479d8243e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:40:22.336 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:40:22.332+0000 7fe4b2ff5640 0 client.17073 ms_handle_reset on v2:172.21.15.119:6838/937297340 2024-04-23T10:40:22.336 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:40:22.332+0000 7fe4d8051740 1 --2- 172.21.15.114:0/592947231 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x56479d81c620 0x56479d8270f0 unknown :-1 s=CLOSED pgs=925 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:40:22.337 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:40:22.332+0000 7fe4d8051740 1 -- 172.21.15.114:0/592947231 >> 172.21.15.114:0/592947231 conn(0x56479d818b30 msgr2=0x56479d81e8b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T10:40:22.337 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:40:22.332+0000 7fe4d8051740 1 -- 172.21.15.114:0/592947231 shutdown_connections 2024-04-23T10:40:22.337 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:40:22.332+0000 7fe4d8051740 1 -- 172.21.15.114:0/592947231 wait complete. 2024-04-23T10:40:22.337 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:40:22.332+0000 7fe4d8051740 20 client.17073 trim_cache size 0 max 16384 2024-04-23T10:40:22.350 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:40:22.349+0000 7fe4d8051740 1 Processor -- start 2024-04-23T10:40:22.350 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:40:22.349+0000 7fe4d8051740 1 -- start start 2024-04-23T10:40:22.350 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:40:22.350+0000 7fe4d8051740 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x56479d81c620 0x56479d8332f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T10:40:22.350 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:40:22.350+0000 7fe4d8051740 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x56479d81cf40 0x56479d83f260 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T10:40:22.351 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:40:22.350+0000 7fe4d8051740 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x56479d827800 0x56479d83f7a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T10:40:22.351 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:40:22.350+0000 7fe4d8051740 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x7fe4d01b2800 con 0x56479d81c620 2024-04-23T10:40:22.351 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:40:22.350+0000 7fe4d8051740 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 -- 0x7fe4d01b0070 con 0x56479d827800 2024-04-23T10:40:22.351 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:40:22.350+0000 7fe4d8051740 1 -- --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_getmap magic: 0 -- 0x7fe4d01adec0 con 0x56479d81cf40 2024-04-23T10:40:22.351 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:40:22.350+0000 7fe4ceffd640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x56479d81c620 0x56479d8332f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T10:40:22.351 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:40:22.350+0000 7fe4ceffd640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x56479d81c620 0x56479d8332f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.114:58616/0 (socket says 172.21.15.114:58616) 2024-04-23T10:40:22.351 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:40:22.350+0000 7fe4ceffd640 1 -- 172.21.15.114:0/2135590719 learned_addr learned my addr 172.21.15.114:0/2135590719 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-04-23T10:40:22.351 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:40:22.350+0000 7fe4cf7fe640 1 --2- 172.21.15.114:0/2135590719 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x56479d827800 0x56479d83f7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T10:40:22.351 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:40:22.350+0000 7fe4ce7fc640 1 --2- 172.21.15.114:0/2135590719 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x56479d81cf40 0x56479d83f260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T10:40:22.351 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:40:22.350+0000 7fe4ceffd640 1 -- 172.21.15.114:0/2135590719 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x56479d81cf40 msgr2=0x56479d83f260 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T10:40:22.352 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:40:22.350+0000 7fe4ceffd640 1 --2- 172.21.15.114:0/2135590719 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x56479d81cf40 0x56479d83f260 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:40:22.352 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:40:22.350+0000 7fe4ceffd640 1 -- 172.21.15.114:0/2135590719 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x56479d827800 msgr2=0x56479d83f7a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T10:40:22.352 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:40:22.350+0000 7fe4ceffd640 1 --2- 172.21.15.114:0/2135590719 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x56479d827800 0x56479d83f7a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:40:22.352 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:40:22.350+0000 7fe4ceffd640 1 -- 172.21.15.114:0/2135590719 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x56479d81ffe0 con 0x56479d81c620 2024-04-23T10:40:22.352 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:40:22.350+0000 7fe4ceffd640 1 --2- 172.21.15.114:0/2135590719 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x56479d81c620 0x56479d8332f0 secure :-1 s=READY pgs=926 cs=0 l=1 rev1=1 crypto rx=0x7fe4c0001eb0 tx=0x7fe4c0011e60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=628f5eb0dae75ddc server_cookie=0 in_seq=0 out_seq=0 2024-04-23T10:40:22.352 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:40:22.351+0000 7fe4b4ff9640 1 -- 172.21.15.114:0/2135590719 <== mon.0 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe4c0027070 con 0x56479d81c620 2024-04-23T10:40:22.352 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:40:22.351+0000 7fe4b4ff9640 1 -- 172.21.15.114:0/2135590719 <== mon.0 v2:172.21.15.114:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fe4c0015020 con 0x56479d81c620 2024-04-23T10:40:22.352 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:40:22.351+0000 7fe4b4ff9640 1 -- 172.21.15.114:0/2135590719 <== mon.0 v2:172.21.15.114:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe4c001e070 con 0x56479d81c620 2024-04-23T10:40:22.356 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:40:22.351+0000 7fe4d8051740 1 -- 172.21.15.114:0/2135590719 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x56479d81c620 msgr2=0x56479d8332f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T10:40:22.356 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:40:22.351+0000 7fe4d8051740 1 --2- 172.21.15.114:0/2135590719 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x56479d81c620 0x56479d8332f0 secure :-1 s=READY pgs=926 cs=0 l=1 rev1=1 crypto rx=0x7fe4c0001eb0 tx=0x7fe4c0011e60 comp rx=0 tx=0).stop 2024-04-23T10:40:22.356 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:40:22.351+0000 7fe4d8051740 1 -- 172.21.15.114:0/2135590719 shutdown_connections 2024-04-23T10:40:22.356 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:40:22.351+0000 7fe4d8051740 1 --2- 172.21.15.114:0/2135590719 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x56479d827800 0x56479d83f7a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:40:22.356 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:40:22.351+0000 7fe4d8051740 1 --2- 172.21.15.114:0/2135590719 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x56479d81cf40 0x56479d83f260 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:40:22.356 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:40:22.351+0000 7fe4d8051740 1 --2- 172.21.15.114:0/2135590719 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x56479d81c620 0x56479d8332f0 unknown :-1 s=CLOSED pgs=926 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:40:22.356 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:40:22.351+0000 7fe4d8051740 1 -- 172.21.15.114:0/2135590719 >> 172.21.15.114:0/2135590719 conn(0x56479d818b30 msgr2=0x56479d823920 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T10:40:22.356 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:40:22.351+0000 7fe4d8051740 1 -- 172.21.15.114:0/2135590719 shutdown_connections 2024-04-23T10:40:22.356 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:40:22.351+0000 7fe4d8051740 1 -- 172.21.15.114:0/2135590719 wait complete. 2024-04-23T10:40:22.356 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:40:22.352+0000 7fe4d8051740 1 -- start start 2024-04-23T10:40:22.356 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:40:22.352+0000 7fe4d8051740 1 Processor -- start 2024-04-23T10:40:22.356 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:40:22.352+0000 7fe4d8051740 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x56479d81c620 0x56479d82f8d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T10:40:22.356 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:40:22.352+0000 7fe4d8051740 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x56479d81cf40 0x56479d82cd90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T10:40:22.357 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:40:22.352+0000 7fe4d8051740 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x56479d827800 0x56479d82d2d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T10:40:22.357 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:40:22.352+0000 7fe4d8051740 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x7fe4d01ae530 con 0x56479d827800 2024-04-23T10:40:22.357 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:40:22.352+0000 7fe4d8051740 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 -- 0x7fe4d01af9a0 con 0x56479d81c620 2024-04-23T10:40:22.357 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:40:22.352+0000 7fe4d8051740 1 -- --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_getmap magic: 0 -- 0x7fe4d01aec00 con 0x56479d81cf40 2024-04-23T10:40:22.357 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:40:22.352+0000 7fe4d8051740 10 client.0 _ll_register_callbacks cb 0x7fe4c0009ed0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-23T10:40:22.357 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:40:22.352+0000 7fe4cf7fe640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x56479d827800 0x56479d82d2d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T10:40:22.357 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:40:22.352+0000 7fe4cf7fe640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x56479d827800 0x56479d82d2d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.114:58626/0 (socket says 172.21.15.114:58626) 2024-04-23T10:40:22.357 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:40:22.352+0000 7fe4cf7fe640 1 -- 172.21.15.114:0/37538909 learned_addr learned my addr 172.21.15.114:0/37538909 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-04-23T10:40:22.357 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:40:22.352+0000 7fe4ce7fc640 1 --2- 172.21.15.114:0/37538909 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x56479d81cf40 0x56479d82cd90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T10:40:22.357 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:40:22.352+0000 7fe4ceffd640 1 --2- 172.21.15.114:0/37538909 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x56479d81c620 0x56479d82f8d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T10:40:22.357 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:40:22.352+0000 7fe4cf7fe640 1 -- 172.21.15.114:0/37538909 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x56479d81cf40 msgr2=0x56479d82cd90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T10:40:22.357 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:40:22.352+0000 7fe4cf7fe640 1 --2- 172.21.15.114:0/37538909 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x56479d81cf40 0x56479d82cd90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:40:22.357 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:40:22.352+0000 7fe4cf7fe640 1 -- 172.21.15.114:0/37538909 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x56479d81c620 msgr2=0x56479d82f8d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T10:40:22.357 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:40:22.352+0000 7fe4cf7fe640 1 --2- 172.21.15.114:0/37538909 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x56479d81c620 0x56479d82f8d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:40:22.358 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:40:22.352+0000 7fe4cf7fe640 1 -- 172.21.15.114:0/37538909 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({osdmap=0}) -- 0x56479d82d810 con 0x56479d827800 2024-04-23T10:40:22.358 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:40:22.353+0000 7fe4cf7fe640 1 --2- 172.21.15.114:0/37538909 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x56479d827800 0x56479d82d2d0 secure :-1 s=READY pgs=927 cs=0 l=1 rev1=1 crypto rx=0x7fe4c809cb20 tx=0x7fe4c809ce20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8207a229fe32af10 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T10:40:22.358 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:40:22.353+0000 7fe4b2ff5640 10 client.0 ms_handle_connect on v2:172.21.15.114:3300/0 2024-04-23T10:40:22.358 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:40:22.353+0000 7fe4b2ff5640 1 -- 172.21.15.114:0/37538909 <== mon.0 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe4c80ba020 con 0x56479d827800 2024-04-23T10:40:22.358 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:40:22.353+0000 7fe4b2ff5640 1 -- 172.21.15.114:0/37538909 <== mon.0 v2:172.21.15.114:3300/0 2 ==== osd_map(111..111 src has 1..111) ==== 8585+0+0 (secure 0 0 0) 0x7fe4c80b6170 con 0x56479d827800 2024-04-23T10:40:22.358 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:40:22.353+0000 7fe4d8051740 1 -- 172.21.15.114:0/37538909 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fe4c0015020 con 0x56479d827800 2024-04-23T10:40:22.358 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:40:22.353+0000 7fe4b2ff5640 1 -- 172.21.15.114:0/37538909 <== mon.0 v2:172.21.15.114:3300/0 3 ==== mon_get_version_reply(handle=1 version=48) ==== 24+0+0 (secure 0 0 0) 0x7fe4c80b6370 con 0x56479d827800 2024-04-23T10:40:22.358 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:40:22.353+0000 7fe4d8051740 10 client.17079 fetch_fsmap learned FSMap version 48 2024-04-23T10:40:22.358 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:40:22.353+0000 7fe4d8051740 1 -- 172.21.15.114:0/37538909 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({config=0+,fsmap=48,monmap=4+}) -- 0x56479d89d8a0 con 0x56479d827800 2024-04-23T10:40:22.358 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:40:22.353+0000 7fe4b2ff5640 1 -- 172.21.15.114:0/37538909 <== mon.0 v2:172.21.15.114:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fe4c0015020 con 0x56479d827800 2024-04-23T10:40:22.358 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:40:22.353+0000 7fe4b2ff5640 1 -- 172.21.15.114:0/37538909 <== mon.0 v2:172.21.15.114:3300/0 5 ==== fsmap(e 48) ==== 8354+0+0 (secure 0 0 0) 0x7fe4c80a5ce0 con 0x56479d827800 2024-04-23T10:40:22.358 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:40:22.353+0000 7fe4d8051740 10 client.17079 fetch_fsmap finished waiting for FSMap version 48 2024-04-23T10:40:22.359 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:40:22.353+0000 7fe4d8051740 10 client.17079 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-04-23T10:40:22.359 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:40:22.353+0000 7fe4d8051740 20 client.17079 populate_metadata read hostname 'smithi114' 2024-04-23T10:40:22.359 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:40:22.353+0000 7fe4d8051740 1 --2- 172.21.15.114:0/37538909 >> [v2:172.21.15.119:6838/937297340,v1:172.21.15.119:6839/937297340] conn(0x56479d89e020 0x56479d8a0440 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T10:40:22.359 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:40:22.353+0000 7fe4d8051740 4 client.17079 mds_command: new command op to 24545 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-04-23T10:40:22.359 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:40:22.353+0000 7fe4d8051740 1 -- 172.21.15.114:0/37538909 --> [v2:172.21.15.119:6838/937297340,v1:172.21.15.119:6839/937297340] -- command(tid 0: {"prefix": "flush journal"}) -- 0x56479d789200 con 0x56479d89e020 2024-04-23T10:40:22.359 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:40:22.354+0000 7fe4ceffd640 1 --2- 172.21.15.114:0/37538909 >> [v2:172.21.15.119:6838/937297340,v1:172.21.15.119:6839/937297340] conn(0x56479d89e020 0x56479d8a0440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T10:40:22.359 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:40:22.354+0000 7fe4b2ff5640 10 client.17079 ms_handle_connect on v2:172.21.15.119:6838/937297340 2024-04-23T10:40:22.359 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:40:22.354+0000 7fe4ceffd640 1 --2- 172.21.15.114:0/37538909 >> [v2:172.21.15.119:6838/937297340,v1:172.21.15.119:6839/937297340] conn(0x56479d89e020 0x56479d8a0440 crc :-1 s=READY pgs=185 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=f86d0db69d8e2b36 server_cookie=86b420dd1ce4ec99 in_seq=0 out_seq=0 2024-04-23T10:40:22.492 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 23 10:40:22 smithi119 ceph-mon[32608]: pgmap v434: 97 pgs: 97 active+clean; 245 MiB data, 3.0 GiB used, 1.0 TiB / 1.0 TiB avail; 1.3 MiB/s rd, 2.9 MiB/s wr, 423 op/s 2024-04-23T10:40:22.496 INFO:tasks.workunit.client.0.smithi114.stdout: 10 11070 27.60 MB/sec warmup 28 sec latency 933.071 ms 2024-04-23T10:40:22.504 INFO:journalctl@ceph.mon.a.smithi114.stdout:Apr 23 10:40:22 smithi114 ceph-mon[29255]: pgmap v434: 97 pgs: 97 active+clean; 245 MiB data, 3.0 GiB used, 1.0 TiB / 1.0 TiB avail; 1.3 MiB/s rd, 2.9 MiB/s wr, 423 op/s 2024-04-23T10:40:22.577 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 23 10:40:22 smithi204 ceph-mon[32649]: pgmap v434: 97 pgs: 97 active+clean; 245 MiB data, 3.0 GiB used, 1.0 TiB / 1.0 TiB avail; 1.3 MiB/s rd, 2.9 MiB/s wr, 423 op/s 2024-04-23T10:40:23.496 INFO:tasks.workunit.client.0.smithi114.stdout: 10 11535 27.33 MB/sec warmup 29 sec latency 1933.098 ms 2024-04-23T10:40:23.556 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:40:23.556+0000 7fe4b2ff5640 1 -- 172.21.15.114:0/37538909 <== mds.0 v2:172.21.15.119:6838/937297340 1 ==== command_reply(tid 0: 0 ) ==== 8+0+44 (crc 0 0 0) 0x56479d789200 con 0x56479d89e020 2024-04-23T10:40:23.557 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:40:23.556+0000 7fe4b2ff5640 10 client.17079 handle_command_reply: tid=0 2024-04-23T10:40:23.557 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:40:23.556+0000 7fe4d8051740 1 client.17079 shutdown 2024-04-23T10:40:23.557 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:40:23.556+0000 7fe4d8051740 1 -- 172.21.15.114:0/37538909 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x56479d827800 msgr2=0x56479d82d2d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T10:40:23.557 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:40:23.556+0000 7fe4d8051740 1 --2- 172.21.15.114:0/37538909 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x56479d827800 0x56479d82d2d0 secure :-1 s=READY pgs=927 cs=0 l=1 rev1=1 crypto rx=0x7fe4c809cb20 tx=0x7fe4c809ce20 comp rx=0 tx=0).stop 2024-04-23T10:40:23.557 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:40:23.556+0000 7fe4d8051740 1 -- 172.21.15.114:0/37538909 shutdown_connections 2024-04-23T10:40:23.557 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:40:23.556+0000 7fe4d8051740 1 --2- 172.21.15.114:0/37538909 >> [v2:172.21.15.119:6838/937297340,v1:172.21.15.119:6839/937297340] conn(0x56479d89e020 0x56479d8a0440 crc :-1 s=READY pgs=185 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:40:23.557 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:40:23.556+0000 7fe4d8051740 1 --2- 172.21.15.114:0/37538909 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x56479d827800 0x56479d82d2d0 unknown :-1 s=CLOSED pgs=927 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:40:23.557 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:40:23.556+0000 7fe4b2ff5640 0 client.17079 ms_handle_reset on v2:172.21.15.119:6838/937297340 2024-04-23T10:40:23.557 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:40:23.556+0000 7fe4d8051740 1 --2- 172.21.15.114:0/37538909 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x56479d81cf40 0x56479d82cd90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:40:23.557 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:40:23.556+0000 7fe4d8051740 1 --2- 172.21.15.114:0/37538909 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x56479d81c620 0x56479d82f8d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:40:23.557 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:40:23.556+0000 7fe4d8051740 1 -- 172.21.15.114:0/37538909 >> 172.21.15.114:0/37538909 conn(0x56479d818b30 msgr2=0x56479d8241b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T10:40:23.557 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:40:23.557+0000 7fe4d8051740 1 -- 172.21.15.114:0/37538909 shutdown_connections 2024-04-23T10:40:23.557 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:40:23.557+0000 7fe4d8051740 1 -- 172.21.15.114:0/37538909 wait complete. 2024-04-23T10:40:23.557 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:40:23.557+0000 7fe4d8051740 20 client.17079 trim_cache size 0 max 16384 2024-04-23T10:40:23.558 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stdout:{ 2024-04-23T10:40:23.558 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stdout: "message": "", 2024-04-23T10:40:23.558 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stdout: "return_code": 0 2024-04-23T10:40:23.558 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stdout:} 2024-04-23T10:40:23.559 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:40:23.557+0000 7fe4d5336640 1 -- 172.21.15.114:0/2553678170 >> [v2:172.21.15.114:6800/1395693466,v1:172.21.15.114:6801/1395693466] conn(0x7fe4b80a3dc0 msgr2=0x7fe4b80a6280 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T10:40:23.559 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:40:23.557+0000 7fe4d5336640 1 --2- 172.21.15.114:0/2553678170 >> [v2:172.21.15.114:6800/1395693466,v1:172.21.15.114:6801/1395693466] conn(0x7fe4b80a3dc0 0x7fe4b80a6280 secure :-1 s=READY pgs=147 cs=0 l=1 rev1=1 crypto rx=0x7fe4c4032fd0 tx=0x7fe4c4005bc0 comp rx=0 tx=0).stop 2024-04-23T10:40:23.559 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:40:23.557+0000 7fe4d5336640 1 -- 172.21.15.114:0/2553678170 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fe4d01020d0 msgr2=0x7fe4d01a6930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T10:40:23.559 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:40:23.557+0000 7fe4d5336640 1 --2- 172.21.15.114:0/2553678170 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fe4d01020d0 0x7fe4d01a6930 secure :-1 s=READY pgs=102 cs=0 l=1 rev1=1 crypto rx=0x7fe4c8009fd0 tx=0x7fe4c800ef10 comp rx=0 tx=0).stop 2024-04-23T10:40:23.559 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:40:23.557+0000 7fe4d5336640 1 -- 172.21.15.114:0/2553678170 shutdown_connections 2024-04-23T10:40:23.559 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:40:23.557+0000 7fe4d5336640 1 --2- 172.21.15.114:0/2553678170 >> [v2:172.21.15.114:6800/1395693466,v1:172.21.15.114:6801/1395693466] conn(0x7fe4b80a3dc0 0x7fe4b80a6280 unknown :-1 s=CLOSED pgs=147 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:40:23.559 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:40:23.557+0000 7fe4d5336640 1 --2- 172.21.15.114:0/2553678170 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fe4d01020d0 0x7fe4d01a6930 unknown :-1 s=CLOSED pgs=102 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:40:23.559 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:40:23.557+0000 7fe4d5336640 1 --2- 172.21.15.114:0/2553678170 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fe4d01072a0 0x7fe4d0101b90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:40:23.559 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:40:23.557+0000 7fe4d5336640 1 --2- 172.21.15.114:0/2553678170 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fe4d01006e0 0x7fe4d0101650 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:40:23.559 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:40:23.557+0000 7fe4d5336640 1 -- 172.21.15.114:0/2553678170 >> 172.21.15.114:0/2553678170 conn(0x7fe4d0077390 msgr2=0x7fe4d00fe760 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T10:40:23.559 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:40:23.557+0000 7fe4d5336640 1 -- 172.21.15.114:0/2553678170 shutdown_connections 2024-04-23T10:40:23.560 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:40:23.557+0000 7fe4d5336640 1 -- 172.21.15.114:0/2553678170 wait complete. 2024-04-23T10:40:24.492 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 23 10:40:24 smithi119 ceph-mon[32608]: pgmap v435: 97 pgs: 97 active+clean; 263 MiB data, 3.1 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 MiB/s rd, 4.5 MiB/s wr, 493 op/s 2024-04-23T10:40:24.496 INFO:tasks.workunit.client.0.smithi114.stdout: 10 12181 27.77 MB/sec warmup 30 sec latency 2637.491 ms 2024-04-23T10:40:24.504 INFO:journalctl@ceph.mon.a.smithi114.stdout:Apr 23 10:40:24 smithi114 ceph-mon[29255]: pgmap v435: 97 pgs: 97 active+clean; 263 MiB data, 3.1 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 MiB/s rd, 4.5 MiB/s wr, 493 op/s 2024-04-23T10:40:24.578 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 23 10:40:24 smithi204 ceph-mon[32649]: pgmap v435: 97 pgs: 97 active+clean; 263 MiB data, 3.1 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 MiB/s rd, 4.5 MiB/s wr, 493 op/s 2024-04-23T10:40:25.492 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 23 10:40:25 smithi119 ceph-mon[32608]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=e=up:active} 3 up:standby-replay 6 up:standby 2024-04-23T10:40:25.496 INFO:tasks.workunit.client.0.smithi114.stdout: 10 13058 28.05 MB/sec warmup 31 sec latency 40.759 ms 2024-04-23T10:40:25.504 INFO:journalctl@ceph.mon.a.smithi114.stdout:Apr 23 10:40:25 smithi114 ceph-mon[29255]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=e=up:active} 3 up:standby-replay 6 up:standby 2024-04-23T10:40:25.578 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 23 10:40:25 smithi204 ceph-mon[32649]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=e=up:active} 3 up:standby-replay 6 up:standby 2024-04-23T10:40:26.492 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 23 10:40:26 smithi119 ceph-mon[32608]: pgmap v436: 97 pgs: 97 active+clean; 263 MiB data, 3.1 GiB used, 1.0 TiB / 1.0 TiB avail; 1.5 MiB/s rd, 3.0 MiB/s wr, 189 op/s 2024-04-23T10:40:26.497 INFO:tasks.workunit.client.0.smithi114.stdout: 10 13836 28.92 MB/sec warmup 32 sec latency 64.653 ms 2024-04-23T10:40:26.504 INFO:journalctl@ceph.mon.a.smithi114.stdout:Apr 23 10:40:26 smithi114 ceph-mon[29255]: pgmap v436: 97 pgs: 97 active+clean; 263 MiB data, 3.1 GiB used, 1.0 TiB / 1.0 TiB avail; 1.5 MiB/s rd, 3.0 MiB/s wr, 189 op/s 2024-04-23T10:40:26.577 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 23 10:40:26 smithi204 ceph-mon[32649]: pgmap v436: 97 pgs: 97 active+clean; 263 MiB data, 3.1 GiB used, 1.0 TiB / 1.0 TiB avail; 1.5 MiB/s rd, 3.0 MiB/s wr, 189 op/s 2024-04-23T10:40:27.497 INFO:tasks.workunit.client.0.smithi114.stdout: 10 14452 28.99 MB/sec warmup 33 sec latency 86.337 ms 2024-04-23T10:40:28.492 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 23 10:40:28 smithi119 ceph-mon[32608]: pgmap v437: 97 pgs: 97 active+clean; 267 MiB data, 3.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.8 MiB/s rd, 5.1 MiB/s wr, 500 op/s 2024-04-23T10:40:28.497 INFO:tasks.workunit.client.0.smithi114.stdout: 10 14845 28.68 MB/sec warmup 34 sec latency 358.785 ms 2024-04-23T10:40:28.503 INFO:journalctl@ceph.mon.a.smithi114.stdout:Apr 23 10:40:28 smithi114 ceph-mon[29255]: pgmap v437: 97 pgs: 97 active+clean; 267 MiB data, 3.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.8 MiB/s rd, 5.1 MiB/s wr, 500 op/s 2024-04-23T10:40:28.578 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 23 10:40:28 smithi204 ceph-mon[32649]: pgmap v437: 97 pgs: 97 active+clean; 267 MiB data, 3.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.8 MiB/s rd, 5.1 MiB/s wr, 500 op/s 2024-04-23T10:40:29.497 INFO:tasks.workunit.client.0.smithi114.stdout: 10 15313 28.65 MB/sec warmup 35 sec latency 378.691 ms 2024-04-23T10:40:30.492 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 23 10:40:30 smithi119 ceph-mon[32608]: pgmap v438: 97 pgs: 97 active+clean; 267 MiB data, 3.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 MiB/s rd, 3.9 MiB/s wr, 417 op/s 2024-04-23T10:40:30.497 INFO:tasks.workunit.client.0.smithi114.stdout: 10 15915 28.55 MB/sec warmup 36 sec latency 243.128 ms 2024-04-23T10:40:30.504 INFO:journalctl@ceph.mon.a.smithi114.stdout:Apr 23 10:40:30 smithi114 ceph-mon[29255]: pgmap v438: 97 pgs: 97 active+clean; 267 MiB data, 3.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 MiB/s rd, 3.9 MiB/s wr, 417 op/s 2024-04-23T10:40:30.578 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 23 10:40:30 smithi204 ceph-mon[32649]: pgmap v438: 97 pgs: 97 active+clean; 267 MiB data, 3.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 MiB/s rd, 3.9 MiB/s wr, 417 op/s 2024-04-23T10:40:31.497 INFO:tasks.workunit.client.0.smithi114.stdout: 10 16673 28.82 MB/sec warmup 37 sec latency 93.566 ms 2024-04-23T10:40:32.492 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 23 10:40:32 smithi119 ceph-mon[32608]: pgmap v439: 97 pgs: 97 active+clean; 267 MiB data, 3.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 MiB/s rd, 3.9 MiB/s wr, 417 op/s 2024-04-23T10:40:32.497 INFO:tasks.workunit.client.0.smithi114.stdout: 10 17407 29.39 MB/sec warmup 38 sec latency 61.089 ms 2024-04-23T10:40:32.504 INFO:journalctl@ceph.mon.a.smithi114.stdout:Apr 23 10:40:32 smithi114 ceph-mon[29255]: pgmap v439: 97 pgs: 97 active+clean; 267 MiB data, 3.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 MiB/s rd, 3.9 MiB/s wr, 417 op/s 2024-04-23T10:40:32.578 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 23 10:40:32 smithi204 ceph-mon[32649]: pgmap v439: 97 pgs: 97 active+clean; 267 MiB data, 3.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 MiB/s rd, 3.9 MiB/s wr, 417 op/s 2024-04-23T10:40:33.497 INFO:tasks.workunit.client.0.smithi114.stdout: 10 17959 29.43 MB/sec warmup 39 sec latency 258.858 ms 2024-04-23T10:40:34.492 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 23 10:40:34 smithi119 ceph-mon[32608]: pgmap v440: 97 pgs: 97 active+clean; 467 MiB data, 3.8 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 MiB/s rd, 21 MiB/s wr, 604 op/s 2024-04-23T10:40:34.497 INFO:tasks.workunit.client.0.smithi114.stdout: 10 18420 29.28 MB/sec warmup 40 sec latency 323.134 ms 2024-04-23T10:40:34.503 INFO:journalctl@ceph.mon.a.smithi114.stdout:Apr 23 10:40:34 smithi114 ceph-mon[29255]: pgmap v440: 97 pgs: 97 active+clean; 467 MiB data, 3.8 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 MiB/s rd, 21 MiB/s wr, 604 op/s 2024-04-23T10:40:34.578 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 23 10:40:34 smithi204 ceph-mon[32649]: pgmap v440: 97 pgs: 97 active+clean; 467 MiB data, 3.8 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 MiB/s rd, 21 MiB/s wr, 604 op/s 2024-04-23T10:40:35.497 INFO:tasks.workunit.client.0.smithi114.stdout: 10 18917 29.29 MB/sec warmup 41 sec latency 233.901 ms 2024-04-23T10:40:36.492 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 23 10:40:36 smithi119 ceph-mon[32608]: pgmap v441: 97 pgs: 97 active+clean; 467 MiB data, 3.8 GiB used, 1.0 TiB / 1.0 TiB avail; 3.2 MiB/s rd, 19 MiB/s wr, 498 op/s 2024-04-23T10:40:36.497 INFO:tasks.workunit.client.0.smithi114.stdout: 10 19581 29.14 MB/sec warmup 42 sec latency 52.518 ms 2024-04-23T10:40:36.503 INFO:journalctl@ceph.mon.a.smithi114.stdout:Apr 23 10:40:36 smithi114 ceph-mon[29255]: pgmap v441: 97 pgs: 97 active+clean; 467 MiB data, 3.8 GiB used, 1.0 TiB / 1.0 TiB avail; 3.2 MiB/s rd, 19 MiB/s wr, 498 op/s 2024-04-23T10:40:36.577 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 23 10:40:36 smithi204 ceph-mon[32649]: pgmap v441: 97 pgs: 97 active+clean; 467 MiB data, 3.8 GiB used, 1.0 TiB / 1.0 TiB avail; 3.2 MiB/s rd, 19 MiB/s wr, 498 op/s 2024-04-23T10:40:36.657 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:40:36.655+0000 7f1fec620640 1 -- 172.21.15.114:0/307706196 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f1fe41043c0 msgr2=0x7f1fe41047a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T10:40:36.657 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:40:36.655+0000 7f1fec620640 1 --2- 172.21.15.114:0/307706196 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f1fe41043c0 0x7f1fe41047a0 secure :-1 s=READY pgs=928 cs=0 l=1 rev1=1 crypto rx=0x7f1fe0009900 tx=0x7f1fe002a410 comp rx=0 tx=0).stop 2024-04-23T10:40:36.657 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:40:36.656+0000 7f1fec620640 1 -- 172.21.15.114:0/307706196 shutdown_connections 2024-04-23T10:40:36.657 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:40:36.656+0000 7f1fec620640 1 --2- 172.21.15.114:0/307706196 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f1fe4109490 0x7f1fe4111060 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:40:36.657 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:40:36.656+0000 7f1fec620640 1 --2- 172.21.15.114:0/307706196 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f1fe4104d70 0x7f1fe4108db0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:40:36.657 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:40:36.656+0000 7f1fec620640 1 --2- 172.21.15.114:0/307706196 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f1fe41043c0 0x7f1fe41047a0 unknown :-1 s=CLOSED pgs=928 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:40:36.657 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:40:36.656+0000 7f1fec620640 1 -- 172.21.15.114:0/307706196 >> 172.21.15.114:0/307706196 conn(0x7f1fe40ffc00 msgr2=0x7f1fe4102020 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T10:40:36.657 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:40:36.656+0000 7f1fec620640 1 -- 172.21.15.114:0/307706196 shutdown_connections 2024-04-23T10:40:36.657 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:40:36.656+0000 7f1fec620640 1 -- 172.21.15.114:0/307706196 wait complete. 2024-04-23T10:40:36.657 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:40:36.656+0000 7f1fec620640 1 Processor -- start 2024-04-23T10:40:36.658 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:40:36.656+0000 7f1fec620640 1 -- start start 2024-04-23T10:40:36.658 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:40:36.656+0000 7f1fec620640 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f1fe41043c0 0x7f1fe41a06a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T10:40:36.658 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:40:36.656+0000 7f1fec620640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f1fe4109490 0x7f1fe41a0be0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T10:40:36.658 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:40:36.656+0000 7f1fec620640 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f1fe419a7b0 0x7f1fe419ac30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T10:40:36.658 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:40:36.656+0000 7f1fec620640 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x7f1fe4114470 con 0x7f1fe4109490 2024-04-23T10:40:36.658 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:40:36.656+0000 7f1fec620640 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 -- 0x7f1fe41142f0 con 0x7f1fe41043c0 2024-04-23T10:40:36.658 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:40:36.656+0000 7f1fec620640 1 -- --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_getmap magic: 0 -- 0x7f1fe41145f0 con 0x7f1fe419a7b0 2024-04-23T10:40:36.658 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:40:36.657+0000 7f1fe9bbb640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f1fe4109490 0x7f1fe41a0be0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T10:40:36.658 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:40:36.657+0000 7f1fe9bbb640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f1fe4109490 0x7f1fe41a0be0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.114:37074/0 (socket says 172.21.15.114:37074) 2024-04-23T10:40:36.658 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:40:36.657+0000 7f1fea3bc640 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f1fe41043c0 0x7f1fe41a06a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T10:40:36.658 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:40:36.657+0000 7f1fe9bbb640 1 -- 172.21.15.114:0/1422528318 learned_addr learned my addr 172.21.15.114:0/1422528318 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-04-23T10:40:36.658 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:40:36.657+0000 7f1feabbd640 1 --2- 172.21.15.114:0/1422528318 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f1fe419a7b0 0x7f1fe419ac30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T10:40:36.658 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:40:36.657+0000 7f1fe9bbb640 1 -- 172.21.15.114:0/1422528318 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f1fe419a7b0 msgr2=0x7f1fe419ac30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T10:40:36.659 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:40:36.657+0000 7f1fe9bbb640 1 --2- 172.21.15.114:0/1422528318 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f1fe419a7b0 0x7f1fe419ac30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:40:36.659 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:40:36.657+0000 7f1fe9bbb640 1 -- 172.21.15.114:0/1422528318 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f1fe41043c0 msgr2=0x7f1fe41a06a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T10:40:36.659 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:40:36.657+0000 7f1fe9bbb640 1 --2- 172.21.15.114:0/1422528318 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f1fe41043c0 0x7f1fe41a06a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:40:36.659 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:40:36.657+0000 7f1fe9bbb640 1 -- 172.21.15.114:0/1422528318 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f1fe419b3b0 con 0x7f1fe4109490 2024-04-23T10:40:36.659 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:40:36.657+0000 7f1fea3bc640 1 --2- 172.21.15.114:0/1422528318 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f1fe41043c0 0x7f1fe41a06a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-23T10:40:36.659 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:40:36.657+0000 7f1fe9bbb640 1 --2- 172.21.15.114:0/1422528318 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f1fe4109490 0x7f1fe41a0be0 secure :-1 s=READY pgs=929 cs=0 l=1 rev1=1 crypto rx=0x7f1fd400e9f0 tx=0x7f1fd400eeb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T10:40:36.659 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:40:36.657+0000 7f1fdb7fe640 1 -- 172.21.15.114:0/1422528318 <== mon.0 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1fd400ccc0 con 0x7f1fe4109490 2024-04-23T10:40:36.659 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:40:36.657+0000 7f1fdb7fe640 1 -- 172.21.15.114:0/1422528318 <== mon.0 v2:172.21.15.114:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f1fd40054d0 con 0x7f1fe4109490 2024-04-23T10:40:36.659 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:40:36.657+0000 7f1fec620640 1 -- 172.21.15.114:0/1422528318 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f1fe41a7440 con 0x7f1fe4109490 2024-04-23T10:40:36.659 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:40:36.657+0000 7f1fec620640 1 -- 172.21.15.114:0/1422528318 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f1fe41a77f0 con 0x7f1fe4109490 2024-04-23T10:40:36.659 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:40:36.658+0000 7f1fdb7fe640 1 -- 172.21.15.114:0/1422528318 <== mon.0 v2:172.21.15.114:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1fd4018430 con 0x7f1fe4109490 2024-04-23T10:40:36.666 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:40:36.660+0000 7f1fdb7fe640 1 -- 172.21.15.114:0/1422528318 <== mon.0 v2:172.21.15.114:3300/0 4 ==== mgrmap(e 26) ==== 138778+0+0 (secure 0 0 0) 0x7f1fd40046c0 con 0x7f1fe4109490 2024-04-23T10:40:36.666 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:40:36.661+0000 7f1fef40b740 1 Processor -- start 2024-04-23T10:40:36.666 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:40:36.661+0000 7f1fdb7fe640 1 --2- 172.21.15.114:0/1422528318 >> [v2:172.21.15.114:6800/1395693466,v1:172.21.15.114:6801/1395693466] conn(0x7f1fd00a3dc0 0x7f1fd00a6280 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T10:40:36.666 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:40:36.661+0000 7f1feabbd640 1 --2- 172.21.15.114:0/1422528318 >> [v2:172.21.15.114:6800/1395693466,v1:172.21.15.114:6801/1395693466] conn(0x7f1fd00a3dc0 0x7f1fd00a6280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T10:40:36.666 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:40:36.661+0000 7f1fef40b740 1 -- start start 2024-04-23T10:40:36.666 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:40:36.661+0000 7f1fdb7fe640 1 -- 172.21.15.114:0/1422528318 <== mon.0 v2:172.21.15.114:3300/0 5 ==== osd_map(111..111 src has 1..111) ==== 8585+0+0 (secure 0 0 0) 0x7f1fd40d0cc0 con 0x7f1fe4109490 2024-04-23T10:40:36.667 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:40:36.661+0000 7f1fef40b740 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x561cbad0e620 0x561cbad0ea00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T10:40:36.667 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:40:36.661+0000 7f1fef40b740 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x561cbad0ef40 0x561cbad192c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T10:40:36.667 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:40:36.661+0000 7f1fef40b740 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x561cbad19800 0x561cbad1bbf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T10:40:36.667 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:40:36.661+0000 7f1fef40b740 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x561cbac73970 con 0x561cbad0e620 2024-04-23T10:40:36.667 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:40:36.661+0000 7f1fef40b740 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 -- 0x561cbac104a0 con 0x561cbad0ef40 2024-04-23T10:40:36.667 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:40:36.661+0000 7f1fef40b740 1 -- --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_getmap magic: 0 -- 0x561cbac6f930 con 0x561cbad19800 2024-04-23T10:40:36.667 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:40:36.661+0000 7f1fea3bc640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x561cbad0e620 0x561cbad0ea00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T10:40:36.667 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:40:36.661+0000 7f1fea3bc640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x561cbad0e620 0x561cbad0ea00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.114:37088/0 (socket says 172.21.15.114:37088) 2024-04-23T10:40:36.667 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:40:36.661+0000 7f1fea3bc640 1 -- 172.21.15.114:0/1973823191 learned_addr learned my addr 172.21.15.114:0/1973823191 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-04-23T10:40:36.667 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:40:36.661+0000 7f1feabbd640 1 --2- 172.21.15.114:0/1973823191 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x561cbad19800 0x561cbad1bbf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T10:40:36.667 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:40:36.661+0000 7f1fe9bbb640 1 --2- 172.21.15.114:0/1973823191 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x561cbad0ef40 0x561cbad192c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T10:40:36.667 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:40:36.662+0000 7f1feabbd640 1 --2- 172.21.15.114:0/1422528318 >> [v2:172.21.15.114:6800/1395693466,v1:172.21.15.114:6801/1395693466] conn(0x7f1fd00a3dc0 0x7f1fd00a6280 secure :-1 s=READY pgs=148 cs=0 l=1 rev1=1 crypto rx=0x7f1fdc009fb0 tx=0x7f1fdc00d040 comp rx=0 tx=0).ready entity=mgr.15291 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T10:40:36.667 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:40:36.662+0000 7f1fea3bc640 1 -- 172.21.15.114:0/1973823191 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x561cbad19800 msgr2=0x561cbad1bbf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T10:40:36.668 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:40:36.662+0000 7f1fea3bc640 1 --2- 172.21.15.114:0/1973823191 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x561cbad19800 0x561cbad1bbf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:40:36.668 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:40:36.662+0000 7f1fea3bc640 1 -- 172.21.15.114:0/1973823191 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x561cbad0ef40 msgr2=0x561cbad192c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T10:40:36.668 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:40:36.662+0000 7f1fea3bc640 1 --2- 172.21.15.114:0/1973823191 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x561cbad0ef40 0x561cbad192c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:40:36.668 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:40:36.662+0000 7f1fea3bc640 1 -- 172.21.15.114:0/1973823191 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x561cba986210 con 0x561cbad0e620 2024-04-23T10:40:36.668 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:40:36.662+0000 7f1fea3bc640 1 --2- 172.21.15.114:0/1973823191 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x561cbad0e620 0x561cbad0ea00 secure :-1 s=READY pgs=930 cs=0 l=1 rev1=1 crypto rx=0x7f1fe0009900 tx=0x7f1fe00314a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=17601c5a704adb4a server_cookie=0 in_seq=0 out_seq=0 2024-04-23T10:40:36.668 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:40:36.662+0000 7f1fd8bfe640 1 -- 172.21.15.114:0/1973823191 <== mon.0 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1fe0040070 con 0x561cbad0e620 2024-04-23T10:40:36.668 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:40:36.662+0000 7f1fd8bfe640 1 -- 172.21.15.114:0/1973823191 <== mon.0 v2:172.21.15.114:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f1fe0031890 con 0x561cbad0e620 2024-04-23T10:40:36.668 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:40:36.662+0000 7f1fd8bfe640 1 -- 172.21.15.114:0/1973823191 <== mon.0 v2:172.21.15.114:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1fe0035040 con 0x561cbad0e620 2024-04-23T10:40:36.668 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:40:36.663+0000 7f1fef40b740 1 -- 172.21.15.114:0/1973823191 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x561cbad0e620 msgr2=0x561cbad0ea00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T10:40:36.668 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:40:36.663+0000 7f1fef40b740 1 --2- 172.21.15.114:0/1973823191 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x561cbad0e620 0x561cbad0ea00 secure :-1 s=READY pgs=930 cs=0 l=1 rev1=1 crypto rx=0x7f1fe0009900 tx=0x7f1fe00314a0 comp rx=0 tx=0).stop 2024-04-23T10:40:36.668 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:40:36.663+0000 7f1fef40b740 1 -- 172.21.15.114:0/1973823191 shutdown_connections 2024-04-23T10:40:36.668 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:40:36.663+0000 7f1fef40b740 1 --2- 172.21.15.114:0/1973823191 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x561cbad19800 0x561cbad1bbf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:40:36.668 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:40:36.663+0000 7f1fef40b740 1 --2- 172.21.15.114:0/1973823191 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x561cbad0ef40 0x561cbad192c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:40:36.668 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:40:36.663+0000 7f1fef40b740 1 --2- 172.21.15.114:0/1973823191 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x561cbad0e620 0x561cbad0ea00 unknown :-1 s=CLOSED pgs=930 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:40:36.669 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:40:36.663+0000 7f1fef40b740 1 -- 172.21.15.114:0/1973823191 >> 172.21.15.114:0/1973823191 conn(0x561cbad0ab30 msgr2=0x561cbad0af10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T10:40:36.669 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:40:36.663+0000 7f1fef40b740 1 -- 172.21.15.114:0/1973823191 shutdown_connections 2024-04-23T10:40:36.669 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:40:36.663+0000 7f1fef40b740 1 -- 172.21.15.114:0/1973823191 wait complete. 2024-04-23T10:40:36.669 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:40:36.663+0000 7f1fef40b740 1 -- start start 2024-04-23T10:40:36.669 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:40:36.663+0000 7f1fef40b740 1 Processor -- start 2024-04-23T10:40:36.669 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:40:36.664+0000 7f1fef40b740 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x561cbad0e620 0x561cbad19110 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T10:40:36.669 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:40:36.664+0000 7f1fef40b740 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x561cbad0ef40 0x561cbad16400 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T10:40:36.669 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:40:36.664+0000 7f1fef40b740 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x561cbad19800 0x561cbad16940 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T10:40:36.669 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:40:36.664+0000 7f1fef40b740 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x561cbad1c3d0 con 0x561cbad0ef40 2024-04-23T10:40:36.669 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:40:36.664+0000 7f1fef40b740 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 -- 0x561cbac0e0b0 con 0x561cbad19800 2024-04-23T10:40:36.669 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:40:36.664+0000 7f1fef40b740 1 -- --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_getmap magic: 0 -- 0x561cbad1c550 con 0x561cbad0e620 2024-04-23T10:40:36.669 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:40:36.664+0000 7f1fef40b740 10 client.0 _ll_register_callbacks cb 0x561cbabfbd30 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-23T10:40:36.669 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:40:36.664+0000 7f1fe9bbb640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x561cbad0ef40 0x561cbad16400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T10:40:36.670 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:40:36.664+0000 7f1fe9bbb640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x561cbad0ef40 0x561cbad16400 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.114:37104/0 (socket says 172.21.15.114:37104) 2024-04-23T10:40:36.670 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:40:36.664+0000 7f1fe9bbb640 1 -- 172.21.15.114:0/4037108350 learned_addr learned my addr 172.21.15.114:0/4037108350 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-04-23T10:40:36.670 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:40:36.664+0000 7f1fea3bc640 1 --2- 172.21.15.114:0/4037108350 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x561cbad0e620 0x561cbad19110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T10:40:36.670 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:40:36.664+0000 7f1feabbd640 1 --2- 172.21.15.114:0/4037108350 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x561cbad19800 0x561cbad16940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T10:40:36.670 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:40:36.664+0000 7f1fe9bbb640 1 -- 172.21.15.114:0/4037108350 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x561cbad0e620 msgr2=0x561cbad19110 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T10:40:36.670 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:40:36.664+0000 7f1fe9bbb640 1 --2- 172.21.15.114:0/4037108350 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x561cbad0e620 0x561cbad19110 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:40:36.670 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:40:36.664+0000 7f1fe9bbb640 1 -- 172.21.15.114:0/4037108350 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x561cbad19800 msgr2=0x561cbad16940 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T10:40:36.670 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:40:36.664+0000 7f1fe9bbb640 1 --2- 172.21.15.114:0/4037108350 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x561cbad19800 0x561cbad16940 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:40:36.670 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:40:36.664+0000 7f1fe9bbb640 1 -- 172.21.15.114:0/4037108350 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({osdmap=0}) -- 0x561cbaa52620 con 0x561cbad0ef40 2024-04-23T10:40:36.670 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:40:36.664+0000 7f1fea3bc640 1 --2- 172.21.15.114:0/4037108350 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x561cbad0e620 0x561cbad19110 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-23T10:40:36.670 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:40:36.664+0000 7f1fe9bbb640 1 --2- 172.21.15.114:0/4037108350 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x561cbad0ef40 0x561cbad16400 secure :-1 s=READY pgs=931 cs=0 l=1 rev1=1 crypto rx=0x7f1fd40995f0 tx=0x7f1fd409d3e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f2011f5d07637241 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T10:40:36.670 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:40:36.664+0000 7f1fcdffb640 10 client.0 ms_handle_connect on v2:172.21.15.114:3300/0 2024-04-23T10:40:36.670 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:40:36.664+0000 7f1fcdffb640 1 -- 172.21.15.114:0/4037108350 <== mon.0 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1fd40b5070 con 0x561cbad0ef40 2024-04-23T10:40:36.671 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:40:36.664+0000 7f1fcdffb640 1 -- 172.21.15.114:0/4037108350 <== mon.0 v2:172.21.15.114:3300/0 2 ==== osd_map(111..111 src has 1..111) ==== 8585+0+0 (secure 0 0 0) 0x7f1fd40a5600 con 0x561cbad0ef40 2024-04-23T10:40:36.671 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:40:36.664+0000 7f1fef40b740 1 -- 172.21.15.114:0/4037108350 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f1fe0031890 con 0x561cbad0ef40 2024-04-23T10:40:36.671 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:40:36.664+0000 7f1fcdffb640 1 -- 172.21.15.114:0/4037108350 <== mon.0 v2:172.21.15.114:3300/0 3 ==== mon_get_version_reply(handle=1 version=49) ==== 24+0+0 (secure 0 0 0) 0x7f1fd40a5890 con 0x561cbad0ef40 2024-04-23T10:40:36.671 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:40:36.665+0000 7f1fef40b740 10 client.17091 fetch_fsmap learned FSMap version 49 2024-04-23T10:40:36.671 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:40:36.665+0000 7f1fef40b740 1 -- 172.21.15.114:0/4037108350 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({config=0+,fsmap=49,monmap=4+}) -- 0x561cbad28310 con 0x561cbad0ef40 2024-04-23T10:40:36.671 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:40:36.665+0000 7f1fcdffb640 1 -- 172.21.15.114:0/4037108350 <== mon.0 v2:172.21.15.114:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f1fe0031890 con 0x561cbad0ef40 2024-04-23T10:40:36.671 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:40:36.665+0000 7f1fcdffb640 1 -- 172.21.15.114:0/4037108350 <== mon.0 v2:172.21.15.114:3300/0 5 ==== fsmap(e 49) ==== 8358+0+0 (secure 0 0 0) 0x7f1fd4005070 con 0x561cbad0ef40 2024-04-23T10:40:36.671 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:40:36.665+0000 7f1fef40b740 10 client.17091 fetch_fsmap finished waiting for FSMap version 49 2024-04-23T10:40:36.671 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:40:36.665+0000 7f1fef40b740 10 client.17091 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-04-23T10:40:36.671 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:40:36.665+0000 7f1fef40b740 20 client.17091 populate_metadata read hostname 'smithi114' 2024-04-23T10:40:36.671 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:40:36.665+0000 7f1fef40b740 1 --2- 172.21.15.114:0/4037108350 >> [v2:172.21.15.119:6838/937297340,v1:172.21.15.119:6839/937297340] conn(0x561cbad29890 0x561cbad2bca0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T10:40:36.671 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:40:36.665+0000 7f1fef40b740 4 client.17091 mds_command: new command op to 24545 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-23T10:40:36.671 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:40:36.665+0000 7f1fef40b740 1 -- 172.21.15.114:0/4037108350 --> [v2:172.21.15.119:6838/937297340,v1:172.21.15.119:6839/937297340] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x561cbac81390 con 0x561cbad29890 2024-04-23T10:40:36.672 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:40:36.665+0000 7f1fea3bc640 1 --2- 172.21.15.114:0/4037108350 >> [v2:172.21.15.119:6838/937297340,v1:172.21.15.119:6839/937297340] conn(0x561cbad29890 0x561cbad2bca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T10:40:36.672 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:40:36.666+0000 7f1fcdffb640 10 client.17091 ms_handle_connect on v2:172.21.15.119:6838/937297340 2024-04-23T10:40:36.672 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:40:36.666+0000 7f1fea3bc640 1 --2- 172.21.15.114:0/4037108350 >> [v2:172.21.15.119:6838/937297340,v1:172.21.15.119:6839/937297340] conn(0x561cbad29890 0x561cbad2bca0 crc :-1 s=READY pgs=186 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=557f41a655628bed server_cookie=9fb5b106a41e3064 in_seq=0 out_seq=0 2024-04-23T10:40:36.673 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:40:36.668+0000 7f1fcdffb640 1 -- 172.21.15.114:0/4037108350 <== mds.0 v2:172.21.15.119:6838/937297340 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x561cbac81390 con 0x561cbad29890 2024-04-23T10:40:36.673 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:40:36.668+0000 7f1fcdffb640 10 client.17091 handle_command_reply: tid=0 2024-04-23T10:40:36.673 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:40:36.668+0000 7f1fef40b740 1 client.17091 shutdown 2024-04-23T10:40:36.673 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:40:36.668+0000 7f1fef40b740 1 -- 172.21.15.114:0/4037108350 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x561cbad0ef40 msgr2=0x561cbad16400 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T10:40:36.673 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:40:36.668+0000 7f1fef40b740 1 --2- 172.21.15.114:0/4037108350 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x561cbad0ef40 0x561cbad16400 secure :-1 s=READY pgs=931 cs=0 l=1 rev1=1 crypto rx=0x7f1fd40995f0 tx=0x7f1fd409d3e0 comp rx=0 tx=0).stop 2024-04-23T10:40:36.673 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:40:36.668+0000 7f1fef40b740 1 -- 172.21.15.114:0/4037108350 shutdown_connections 2024-04-23T10:40:36.673 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:40:36.668+0000 7f1fef40b740 1 --2- 172.21.15.114:0/4037108350 >> [v2:172.21.15.119:6838/937297340,v1:172.21.15.119:6839/937297340] conn(0x561cbad29890 0x561cbad2bca0 crc :-1 s=READY pgs=186 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:40:36.674 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:40:36.668+0000 7f1fef40b740 1 --2- 172.21.15.114:0/4037108350 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x561cbad19800 0x561cbad16940 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:40:36.674 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:40:36.668+0000 7f1fef40b740 1 --2- 172.21.15.114:0/4037108350 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x561cbad0ef40 0x561cbad16400 unknown :-1 s=CLOSED pgs=931 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:40:36.674 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:40:36.668+0000 7f1fef40b740 1 --2- 172.21.15.114:0/4037108350 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x561cbad0e620 0x561cbad19110 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:40:36.674 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:40:36.668+0000 7f1fef40b740 1 -- 172.21.15.114:0/4037108350 >> 172.21.15.114:0/4037108350 conn(0x561cbad0ab30 msgr2=0x561cbad108b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T10:40:36.674 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:40:36.668+0000 7f1fcdffb640 0 client.17091 ms_handle_reset on v2:172.21.15.119:6838/937297340 2024-04-23T10:40:36.674 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:40:36.668+0000 7f1fef40b740 1 -- 172.21.15.114:0/4037108350 shutdown_connections 2024-04-23T10:40:36.674 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:40:36.668+0000 7f1fef40b740 1 -- 172.21.15.114:0/4037108350 wait complete. 2024-04-23T10:40:36.674 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:40:36.668+0000 7f1fef40b740 20 client.17091 trim_cache size 0 max 16384 2024-04-23T10:40:36.684 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:40:36.684+0000 7f1fef40b740 1 Processor -- start 2024-04-23T10:40:36.685 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:40:36.684+0000 7f1fef40b740 1 -- start start 2024-04-23T10:40:36.685 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:40:36.685+0000 7f1fef40b740 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x561cbad0ab30 0x561cbad25290 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T10:40:36.685 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:40:36.685+0000 7f1fef40b740 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x561cbad0e620 0x561cbad09b30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T10:40:36.685 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:40:36.685+0000 7f1fef40b740 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x561cbad0ef40 0x561cbad315b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T10:40:36.685 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:40:36.685+0000 7f1fef40b740 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x7f1fe41b2850 con 0x561cbad0e620 2024-04-23T10:40:36.685 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:40:36.685+0000 7f1fef40b740 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 -- 0x7f1fe41b0090 con 0x561cbad0ef40 2024-04-23T10:40:36.685 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:40:36.685+0000 7f1fef40b740 1 -- --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_getmap magic: 0 -- 0x7f1fe41adee0 con 0x561cbad0ab30 2024-04-23T10:40:36.685 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:40:36.685+0000 7f1fe9bbb640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x561cbad0e620 0x561cbad09b30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T10:40:36.685 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:40:36.685+0000 7f1fe9bbb640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x561cbad0e620 0x561cbad09b30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.114:37112/0 (socket says 172.21.15.114:37112) 2024-04-23T10:40:36.685 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:40:36.685+0000 7f1fe9bbb640 1 -- 172.21.15.114:0/3110251715 learned_addr learned my addr 172.21.15.114:0/3110251715 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-04-23T10:40:36.685 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:40:36.685+0000 7f1fea3bc640 1 --2- 172.21.15.114:0/3110251715 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x561cbad0ab30 0x561cbad25290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T10:40:36.685 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:40:36.685+0000 7f1feabbd640 1 --2- 172.21.15.114:0/3110251715 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x561cbad0ef40 0x561cbad315b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T10:40:36.686 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:40:36.685+0000 7f1fe9bbb640 1 -- 172.21.15.114:0/3110251715 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x561cbad0ab30 msgr2=0x561cbad25290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T10:40:36.686 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:40:36.685+0000 7f1fe9bbb640 1 --2- 172.21.15.114:0/3110251715 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x561cbad0ab30 0x561cbad25290 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:40:36.686 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:40:36.685+0000 7f1fe9bbb640 1 -- 172.21.15.114:0/3110251715 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x561cbad0ef40 msgr2=0x561cbad315b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T10:40:36.686 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:40:36.685+0000 7f1fe9bbb640 1 --2- 172.21.15.114:0/3110251715 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x561cbad0ef40 0x561cbad315b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:40:36.686 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:40:36.685+0000 7f1fe9bbb640 1 -- 172.21.15.114:0/3110251715 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x561cbad0a5c0 con 0x561cbad0e620 2024-04-23T10:40:36.686 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:40:36.685+0000 7f1fe9bbb640 1 --2- 172.21.15.114:0/3110251715 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x561cbad0e620 0x561cbad09b30 secure :-1 s=READY pgs=932 cs=0 l=1 rev1=1 crypto rx=0x7f1fd409cbb0 tx=0x7f1fd40a1150 comp rx=0 tx=0).ready entity=mon.0 client_cookie=bce69a7d9bb9f161 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T10:40:36.686 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:40:36.686+0000 7f1fcffff640 1 -- 172.21.15.114:0/3110251715 <== mon.0 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1fd40b5030 con 0x561cbad0e620 2024-04-23T10:40:36.686 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:40:36.686+0000 7f1fcffff640 1 -- 172.21.15.114:0/3110251715 <== mon.0 v2:172.21.15.114:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f1fd4098970 con 0x561cbad0e620 2024-04-23T10:40:36.686 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:40:36.686+0000 7f1fcffff640 1 -- 172.21.15.114:0/3110251715 <== mon.0 v2:172.21.15.114:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1fd40051f0 con 0x561cbad0e620 2024-04-23T10:40:36.690 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:40:36.687+0000 7f1fef40b740 1 -- 172.21.15.114:0/3110251715 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x561cbad0e620 msgr2=0x561cbad09b30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T10:40:36.690 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:40:36.687+0000 7f1fef40b740 1 --2- 172.21.15.114:0/3110251715 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x561cbad0e620 0x561cbad09b30 secure :-1 s=READY pgs=932 cs=0 l=1 rev1=1 crypto rx=0x7f1fd409cbb0 tx=0x7f1fd40a1150 comp rx=0 tx=0).stop 2024-04-23T10:40:36.690 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:40:36.687+0000 7f1fef40b740 1 -- 172.21.15.114:0/3110251715 shutdown_connections 2024-04-23T10:40:36.690 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:40:36.687+0000 7f1fef40b740 1 --2- 172.21.15.114:0/3110251715 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x561cbad0ef40 0x561cbad315b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:40:36.690 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:40:36.687+0000 7f1fef40b740 1 --2- 172.21.15.114:0/3110251715 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x561cbad0e620 0x561cbad09b30 unknown :-1 s=CLOSED pgs=932 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:40:36.690 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:40:36.687+0000 7f1fef40b740 1 --2- 172.21.15.114:0/3110251715 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x561cbad0ab30 0x561cbad25290 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:40:36.690 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:40:36.687+0000 7f1fef40b740 1 -- 172.21.15.114:0/3110251715 >> 172.21.15.114:0/3110251715 conn(0x561cbad16aa0 msgr2=0x561cbad09420 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T10:40:36.690 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:40:36.687+0000 7f1fef40b740 1 -- 172.21.15.114:0/3110251715 shutdown_connections 2024-04-23T10:40:36.690 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:40:36.687+0000 7f1fef40b740 1 -- 172.21.15.114:0/3110251715 wait complete. 2024-04-23T10:40:36.690 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:40:36.687+0000 7f1fef40b740 1 -- start start 2024-04-23T10:40:36.690 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:40:36.687+0000 7f1fef40b740 1 Processor -- start 2024-04-23T10:40:36.690 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:40:36.688+0000 7f1fef40b740 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x561cbad0ab30 0x561cbad21830 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T10:40:36.691 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:40:36.688+0000 7f1fef40b740 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x561cbad0e620 0x561cbad1ecf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T10:40:36.691 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:40:36.688+0000 7f1fef40b740 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x561cbad0ef40 0x561cbad1f230 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T10:40:36.691 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:40:36.688+0000 7f1fef40b740 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x7f1fe41ae550 con 0x561cbad0e620 2024-04-23T10:40:36.691 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:40:36.688+0000 7f1fef40b740 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 -- 0x7f1fe41af9c0 con 0x561cbad0ef40 2024-04-23T10:40:36.691 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:40:36.688+0000 7f1fef40b740 1 -- --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_getmap magic: 0 -- 0x7f1fe41aec20 con 0x561cbad0ab30 2024-04-23T10:40:36.691 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:40:36.688+0000 7f1fef40b740 10 client.0 _ll_register_callbacks cb 0x7f1fd4005680 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-23T10:40:36.691 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:40:36.688+0000 7f1fea3bc640 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x561cbad0ab30 0x561cbad21830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T10:40:36.691 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:40:36.688+0000 7f1feabbd640 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x561cbad0ef40 0x561cbad1f230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T10:40:36.691 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:40:36.688+0000 7f1fe9bbb640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x561cbad0e620 0x561cbad1ecf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T10:40:36.691 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:40:36.688+0000 7f1fea3bc640 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x561cbad0ab30 0x561cbad21830 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.204:3300/0 says I am v2:172.21.15.114:33218/0 (socket says 172.21.15.114:33218) 2024-04-23T10:40:36.691 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:40:36.688+0000 7f1fea3bc640 1 -- 172.21.15.114:0/4146774621 learned_addr learned my addr 172.21.15.114:0/4146774621 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-04-23T10:40:36.691 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:40:36.688+0000 7f1fe9bbb640 1 -- 172.21.15.114:0/4146774621 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x561cbad0ab30 msgr2=0x561cbad21830 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T10:40:36.691 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:40:36.688+0000 7f1fe9bbb640 1 --2- 172.21.15.114:0/4146774621 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x561cbad0ab30 0x561cbad21830 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:40:36.691 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:40:36.688+0000 7f1fe9bbb640 1 -- 172.21.15.114:0/4146774621 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x561cbad0ef40 msgr2=0x561cbad1f230 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T10:40:36.692 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:40:36.688+0000 7f1fe9bbb640 1 --2- 172.21.15.114:0/4146774621 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x561cbad0ef40 0x561cbad1f230 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:40:36.692 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:40:36.688+0000 7f1fe9bbb640 1 -- 172.21.15.114:0/4146774621 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({osdmap=0}) -- 0x561cbad1f770 con 0x561cbad0e620 2024-04-23T10:40:36.692 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:40:36.688+0000 7f1fea3bc640 1 --2- 172.21.15.114:0/4146774621 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x561cbad0ab30 0x561cbad21830 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-23T10:40:36.692 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:40:36.688+0000 7f1fe9bbb640 1 --2- 172.21.15.114:0/4146774621 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x561cbad0e620 0x561cbad1ecf0 secure :-1 s=READY pgs=933 cs=0 l=1 rev1=1 crypto rx=0x561cbad0ab00 tx=0x7f1fd4005730 comp rx=0 tx=0).ready entity=mon.0 client_cookie=aac2d2168dd4fa5a server_cookie=0 in_seq=0 out_seq=0 2024-04-23T10:40:36.692 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:40:36.688+0000 7f1fcdffb640 10 client.0 ms_handle_connect on v2:172.21.15.114:3300/0 2024-04-23T10:40:36.692 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:40:36.688+0000 7f1fcdffb640 1 -- 172.21.15.114:0/4146774621 <== mon.0 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1fd40b5070 con 0x561cbad0e620 2024-04-23T10:40:36.692 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:40:36.689+0000 7f1fcdffb640 1 -- 172.21.15.114:0/4146774621 <== mon.0 v2:172.21.15.114:3300/0 2 ==== osd_map(111..111 src has 1..111) ==== 8585+0+0 (secure 0 0 0) 0x7f1fd409c070 con 0x561cbad0e620 2024-04-23T10:40:36.692 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:40:36.689+0000 7f1fef40b740 1 -- 172.21.15.114:0/4146774621 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f1fd4098970 con 0x561cbad0e620 2024-04-23T10:40:36.692 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:40:36.689+0000 7f1fcdffb640 1 -- 172.21.15.114:0/4146774621 <== mon.0 v2:172.21.15.114:3300/0 3 ==== mon_get_version_reply(handle=1 version=49) ==== 24+0+0 (secure 0 0 0) 0x7f1fd409c2f0 con 0x561cbad0e620 2024-04-23T10:40:36.692 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:40:36.689+0000 7f1fef40b740 10 client.17097 fetch_fsmap learned FSMap version 49 2024-04-23T10:40:36.692 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:40:36.689+0000 7f1fef40b740 1 -- 172.21.15.114:0/4146774621 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({config=0+,fsmap=49,monmap=4+}) -- 0x561cbad8fb50 con 0x561cbad0e620 2024-04-23T10:40:36.692 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:40:36.689+0000 7f1fcdffb640 1 -- 172.21.15.114:0/4146774621 <== mon.0 v2:172.21.15.114:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f1fd4098970 con 0x561cbad0e620 2024-04-23T10:40:36.692 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:40:36.689+0000 7f1fcdffb640 1 -- 172.21.15.114:0/4146774621 <== mon.0 v2:172.21.15.114:3300/0 5 ==== fsmap(e 49) ==== 8358+0+0 (secure 0 0 0) 0x7f1fd4084070 con 0x561cbad0e620 2024-04-23T10:40:36.692 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:40:36.689+0000 7f1fef40b740 10 client.17097 fetch_fsmap finished waiting for FSMap version 49 2024-04-23T10:40:36.693 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:40:36.689+0000 7f1fef40b740 10 client.17097 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-04-23T10:40:36.693 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:40:36.689+0000 7f1fef40b740 20 client.17097 populate_metadata read hostname 'smithi114' 2024-04-23T10:40:36.693 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:40:36.689+0000 7f1fef40b740 1 --2- 172.21.15.114:0/4146774621 >> [v2:172.21.15.119:6838/937297340,v1:172.21.15.119:6839/937297340] conn(0x561cbad19800 0x561cbad91ff0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T10:40:36.693 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:40:36.689+0000 7f1fef40b740 4 client.17097 mds_command: new command op to 24545 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-04-23T10:40:36.693 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:40:36.689+0000 7f1fef40b740 1 -- 172.21.15.114:0/4146774621 --> [v2:172.21.15.119:6838/937297340,v1:172.21.15.119:6839/937297340] -- command(tid 0: {"prefix": "flush journal"}) -- 0x561cbac7b200 con 0x561cbad19800 2024-04-23T10:40:36.693 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:40:36.689+0000 7f1fea3bc640 1 --2- 172.21.15.114:0/4146774621 >> [v2:172.21.15.119:6838/937297340,v1:172.21.15.119:6839/937297340] conn(0x561cbad19800 0x561cbad91ff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T10:40:36.693 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:40:36.690+0000 7f1fea3bc640 1 --2- 172.21.15.114:0/4146774621 >> [v2:172.21.15.119:6838/937297340,v1:172.21.15.119:6839/937297340] conn(0x561cbad19800 0x561cbad91ff0 crc :-1 s=READY pgs=187 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=a5618dda39cc45d8 server_cookie=7deb7765642b3d3f in_seq=0 out_seq=0 2024-04-23T10:40:36.693 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:40:36.690+0000 7f1fcdffb640 10 client.17097 ms_handle_connect on v2:172.21.15.119:6838/937297340 2024-04-23T10:40:37.063 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:40:37.064+0000 7f1fcdffb640 1 -- 172.21.15.114:0/4146774621 <== mds.0 v2:172.21.15.119:6838/937297340 1 ==== command_reply(tid 0: 0 ) ==== 8+0+44 (crc 0 0 0) 0x561cbac7b200 con 0x561cbad19800 2024-04-23T10:40:37.063 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:40:37.064+0000 7f1fcdffb640 10 client.17097 handle_command_reply: tid=0 2024-04-23T10:40:37.063 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:40:37.064+0000 7f1fef40b740 1 client.17097 shutdown 2024-04-23T10:40:37.064 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:40:37.064+0000 7f1fef40b740 1 -- 172.21.15.114:0/4146774621 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x561cbad0e620 msgr2=0x561cbad1ecf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T10:40:37.064 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:40:37.064+0000 7f1fef40b740 1 --2- 172.21.15.114:0/4146774621 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x561cbad0e620 0x561cbad1ecf0 secure :-1 s=READY pgs=933 cs=0 l=1 rev1=1 crypto rx=0x561cbad0ab00 tx=0x7f1fd4005730 comp rx=0 tx=0).stop 2024-04-23T10:40:37.064 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:40:37.064+0000 7f1fef40b740 1 -- 172.21.15.114:0/4146774621 shutdown_connections 2024-04-23T10:40:37.064 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:40:37.064+0000 7f1fef40b740 1 --2- 172.21.15.114:0/4146774621 >> [v2:172.21.15.119:6838/937297340,v1:172.21.15.119:6839/937297340] conn(0x561cbad19800 0x561cbad91ff0 crc :-1 s=READY pgs=187 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:40:37.064 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:40:37.064+0000 7f1fef40b740 1 --2- 172.21.15.114:0/4146774621 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x561cbad0ef40 0x561cbad1f230 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:40:37.064 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:40:37.064+0000 7f1fef40b740 1 --2- 172.21.15.114:0/4146774621 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x561cbad0e620 0x561cbad1ecf0 unknown :-1 s=CLOSED pgs=933 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:40:37.064 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:40:37.064+0000 7f1fcdffb640 0 client.17097 ms_handle_reset on v2:172.21.15.119:6838/937297340 2024-04-23T10:40:37.064 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:40:37.064+0000 7f1fef40b740 1 --2- 172.21.15.114:0/4146774621 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x561cbad0ab30 0x561cbad21830 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:40:37.064 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:40:37.064+0000 7f1fef40b740 1 -- 172.21.15.114:0/4146774621 >> 172.21.15.114:0/4146774621 conn(0x561cbad16aa0 msgr2=0x561cbad09420 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T10:40:37.064 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:40:37.064+0000 7f1fef40b740 1 -- 172.21.15.114:0/4146774621 shutdown_connections 2024-04-23T10:40:37.064 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:40:37.064+0000 7f1fef40b740 1 -- 172.21.15.114:0/4146774621 wait complete. 2024-04-23T10:40:37.064 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:40:37.064+0000 7f1fef40b740 20 client.17097 trim_cache size 0 max 16384 2024-04-23T10:40:37.065 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stdout:{ 2024-04-23T10:40:37.065 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stdout: "message": "", 2024-04-23T10:40:37.065 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stdout: "return_code": 0 2024-04-23T10:40:37.065 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stdout:} 2024-04-23T10:40:37.066 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:40:37.065+0000 7f1fec620640 1 -- 172.21.15.114:0/1422528318 >> [v2:172.21.15.114:6800/1395693466,v1:172.21.15.114:6801/1395693466] conn(0x7f1fd00a3dc0 msgr2=0x7f1fd00a6280 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T10:40:37.066 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:40:37.065+0000 7f1fec620640 1 --2- 172.21.15.114:0/1422528318 >> [v2:172.21.15.114:6800/1395693466,v1:172.21.15.114:6801/1395693466] conn(0x7f1fd00a3dc0 0x7f1fd00a6280 secure :-1 s=READY pgs=148 cs=0 l=1 rev1=1 crypto rx=0x7f1fdc009fb0 tx=0x7f1fdc00d040 comp rx=0 tx=0).stop 2024-04-23T10:40:37.066 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:40:37.065+0000 7f1fec620640 1 -- 172.21.15.114:0/1422528318 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f1fe4109490 msgr2=0x7f1fe41a0be0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T10:40:37.066 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:40:37.065+0000 7f1fec620640 1 --2- 172.21.15.114:0/1422528318 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f1fe4109490 0x7f1fe41a0be0 secure :-1 s=READY pgs=929 cs=0 l=1 rev1=1 crypto rx=0x7f1fd400e9f0 tx=0x7f1fd400eeb0 comp rx=0 tx=0).stop 2024-04-23T10:40:37.066 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:40:37.065+0000 7f1fec620640 1 -- 172.21.15.114:0/1422528318 shutdown_connections 2024-04-23T10:40:37.066 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:40:37.065+0000 7f1fec620640 1 --2- 172.21.15.114:0/1422528318 >> [v2:172.21.15.114:6800/1395693466,v1:172.21.15.114:6801/1395693466] conn(0x7f1fd00a3dc0 0x7f1fd00a6280 unknown :-1 s=CLOSED pgs=148 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:40:37.066 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:40:37.065+0000 7f1fec620640 1 --2- 172.21.15.114:0/1422528318 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f1fe419a7b0 0x7f1fe419ac30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:40:37.066 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:40:37.065+0000 7f1fec620640 1 --2- 172.21.15.114:0/1422528318 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f1fe4109490 0x7f1fe41a0be0 unknown :-1 s=CLOSED pgs=929 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:40:37.066 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:40:37.065+0000 7f1fec620640 1 --2- 172.21.15.114:0/1422528318 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f1fe41043c0 0x7f1fe41a06a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:40:37.066 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:40:37.065+0000 7f1fec620640 1 -- 172.21.15.114:0/1422528318 >> 172.21.15.114:0/1422528318 conn(0x7f1fe40ffc00 msgr2=0x7f1fe4100f90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T10:40:37.066 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:40:37.065+0000 7f1fec620640 1 -- 172.21.15.114:0/1422528318 shutdown_connections 2024-04-23T10:40:37.067 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:40:37.065+0000 7f1fec620640 1 -- 172.21.15.114:0/1422528318 wait complete. 2024-04-23T10:40:37.497 INFO:tasks.workunit.client.0.smithi114.stdout: 10 20056 29.04 MB/sec warmup 43 sec latency 176.432 ms 2024-04-23T10:40:38.492 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 23 10:40:38 smithi119 ceph-mon[32608]: pgmap v442: 97 pgs: 97 active+clean; 486 MiB data, 3.9 GiB used, 1.0 TiB / 1.0 TiB avail; 7.5 MiB/s rd, 23 MiB/s wr, 705 op/s 2024-04-23T10:40:38.497 INFO:tasks.workunit.client.0.smithi114.stdout: 10 20750 29.35 MB/sec warmup 44 sec latency 117.696 ms 2024-04-23T10:40:38.503 INFO:journalctl@ceph.mon.a.smithi114.stdout:Apr 23 10:40:38 smithi114 ceph-mon[29255]: pgmap v442: 97 pgs: 97 active+clean; 486 MiB data, 3.9 GiB used, 1.0 TiB / 1.0 TiB avail; 7.5 MiB/s rd, 23 MiB/s wr, 705 op/s 2024-04-23T10:40:38.577 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 23 10:40:38 smithi204 ceph-mon[32649]: pgmap v442: 97 pgs: 97 active+clean; 486 MiB data, 3.9 GiB used, 1.0 TiB / 1.0 TiB avail; 7.5 MiB/s rd, 23 MiB/s wr, 705 op/s 2024-04-23T10:40:39.497 INFO:tasks.workunit.client.0.smithi114.stdout: 10 21346 29.46 MB/sec warmup 45 sec latency 202.152 ms 2024-04-23T10:40:40.492 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 23 10:40:40 smithi119 ceph-mon[32608]: pgmap v443: 97 pgs: 97 active+clean; 486 MiB data, 3.9 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 MiB/s rd, 20 MiB/s wr, 393 op/s 2024-04-23T10:40:40.497 INFO:tasks.workunit.client.0.smithi114.stdout: 10 21866 29.49 MB/sec warmup 46 sec latency 135.412 ms 2024-04-23T10:40:40.503 INFO:journalctl@ceph.mon.a.smithi114.stdout:Apr 23 10:40:40 smithi114 ceph-mon[29255]: pgmap v443: 97 pgs: 97 active+clean; 486 MiB data, 3.9 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 MiB/s rd, 20 MiB/s wr, 393 op/s 2024-04-23T10:40:40.577 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 23 10:40:40 smithi204 ceph-mon[32649]: pgmap v443: 97 pgs: 97 active+clean; 486 MiB data, 3.9 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 MiB/s rd, 20 MiB/s wr, 393 op/s 2024-04-23T10:40:41.497 INFO:tasks.workunit.client.0.smithi114.stdout: 10 22391 29.34 MB/sec warmup 47 sec latency 67.827 ms 2024-04-23T10:40:42.491 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 23 10:40:42 smithi119 ceph-mon[32608]: pgmap v444: 97 pgs: 97 active+clean; 486 MiB data, 3.9 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 MiB/s rd, 20 MiB/s wr, 393 op/s 2024-04-23T10:40:42.497 INFO:tasks.workunit.client.0.smithi114.stdout: 10 23025 29.37 MB/sec warmup 48 sec latency 92.617 ms 2024-04-23T10:40:42.503 INFO:journalctl@ceph.mon.a.smithi114.stdout:Apr 23 10:40:42 smithi114 ceph-mon[29255]: pgmap v444: 97 pgs: 97 active+clean; 486 MiB data, 3.9 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 MiB/s rd, 20 MiB/s wr, 393 op/s 2024-04-23T10:40:42.577 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 23 10:40:42 smithi204 ceph-mon[32649]: pgmap v444: 97 pgs: 97 active+clean; 486 MiB data, 3.9 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 MiB/s rd, 20 MiB/s wr, 393 op/s 2024-04-23T10:40:43.497 INFO:tasks.workunit.client.0.smithi114.stdout: 10 23807 29.66 MB/sec warmup 49 sec latency 114.840 ms 2024-04-23T10:40:44.491 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 23 10:40:44 smithi119 ceph-mon[32608]: pgmap v445: 97 pgs: 97 active+clean; 487 MiB data, 3.9 GiB used, 1.0 TiB / 1.0 TiB avail; 11 MiB/s rd, 23 MiB/s wr, 725 op/s 2024-04-23T10:40:44.497 INFO:tasks.workunit.client.0.smithi114.stdout: 10 24273 29.62 MB/sec warmup 50 sec latency 236.000 ms 2024-04-23T10:40:44.503 INFO:journalctl@ceph.mon.a.smithi114.stdout:Apr 23 10:40:44 smithi114 ceph-mon[29255]: pgmap v445: 97 pgs: 97 active+clean; 487 MiB data, 3.9 GiB used, 1.0 TiB / 1.0 TiB avail; 11 MiB/s rd, 23 MiB/s wr, 725 op/s 2024-04-23T10:40:44.577 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 23 10:40:44 smithi204 ceph-mon[32649]: pgmap v445: 97 pgs: 97 active+clean; 487 MiB data, 3.9 GiB used, 1.0 TiB / 1.0 TiB avail; 11 MiB/s rd, 23 MiB/s wr, 725 op/s 2024-04-23T10:40:45.497 INFO:tasks.workunit.client.0.smithi114.stdout: 10 24907 29.80 MB/sec warmup 51 sec latency 112.899 ms 2024-04-23T10:40:46.491 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 23 10:40:46 smithi119 ceph-mon[32608]: pgmap v446: 97 pgs: 97 active+clean; 487 MiB data, 3.9 GiB used, 1.0 TiB / 1.0 TiB avail; 8.9 MiB/s rd, 5.9 MiB/s wr, 537 op/s 2024-04-23T10:40:46.497 INFO:tasks.workunit.client.0.smithi114.stdout: 10 25500 30.00 MB/sec warmup 52 sec latency 41.506 ms 2024-04-23T10:40:46.503 INFO:journalctl@ceph.mon.a.smithi114.stdout:Apr 23 10:40:46 smithi114 ceph-mon[29255]: pgmap v446: 97 pgs: 97 active+clean; 487 MiB data, 3.9 GiB used, 1.0 TiB / 1.0 TiB avail; 8.9 MiB/s rd, 5.9 MiB/s wr, 537 op/s 2024-04-23T10:40:46.577 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 23 10:40:46 smithi204 ceph-mon[32649]: pgmap v446: 97 pgs: 97 active+clean; 487 MiB data, 3.9 GiB used, 1.0 TiB / 1.0 TiB avail; 8.9 MiB/s rd, 5.9 MiB/s wr, 537 op/s 2024-04-23T10:40:47.497 INFO:tasks.workunit.client.0.smithi114.stdout: 10 26019 29.67 MB/sec warmup 53 sec latency 51.139 ms 2024-04-23T10:40:48.492 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 23 10:40:48 smithi119 ceph-mon[32608]: pgmap v447: 97 pgs: 97 active+clean; 527 MiB data, 4.1 GiB used, 1.0 TiB / 1.0 TiB avail; 13 MiB/s rd, 12 MiB/s wr, 766 op/s 2024-04-23T10:40:48.497 INFO:tasks.workunit.client.0.smithi114.stdout: 10 26650 29.70 MB/sec warmup 54 sec latency 88.409 ms 2024-04-23T10:40:48.503 INFO:journalctl@ceph.mon.a.smithi114.stdout:Apr 23 10:40:48 smithi114 ceph-mon[29255]: pgmap v447: 97 pgs: 97 active+clean; 527 MiB data, 4.1 GiB used, 1.0 TiB / 1.0 TiB avail; 13 MiB/s rd, 12 MiB/s wr, 766 op/s 2024-04-23T10:40:48.577 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 23 10:40:48 smithi204 ceph-mon[32649]: pgmap v447: 97 pgs: 97 active+clean; 527 MiB data, 4.1 GiB used, 1.0 TiB / 1.0 TiB avail; 13 MiB/s rd, 12 MiB/s wr, 766 op/s 2024-04-23T10:40:49.497 INFO:tasks.workunit.client.0.smithi114.stdout: 10 27142 29.83 MB/sec warmup 55 sec latency 278.487 ms 2024-04-23T10:40:50.162 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:40:50.163+0000 7ff60a507640 1 -- 172.21.15.114:0/4249174344 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7ff604106190 msgr2=0x7ff60410dd60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T10:40:50.162 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:40:50.163+0000 7ff60a507640 1 --2- 172.21.15.114:0/4249174344 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7ff604106190 0x7ff60410dd60 secure :-1 s=READY pgs=934 cs=0 l=1 rev1=1 crypto rx=0x7ff5fc009ca0 tx=0x7ff5fc02a8c0 comp rx=0 tx=0).stop 2024-04-23T10:40:50.162 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:40:50.163+0000 7ff60a507640 1 -- 172.21.15.114:0/4249174344 shutdown_connections 2024-04-23T10:40:50.162 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:40:50.163+0000 7ff60a507640 1 --2- 172.21.15.114:0/4249174344 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7ff604106190 0x7ff60410dd60 unknown :-1 s=CLOSED pgs=934 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:40:50.163 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:40:50.163+0000 7ff60a507640 1 --2- 172.21.15.114:0/4249174344 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7ff604101ac0 0x7ff604105ab0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:40:50.163 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:40:50.163+0000 7ff60a507640 1 --2- 172.21.15.114:0/4249174344 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7ff604101110 0x7ff6041014f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:40:50.163 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:40:50.163+0000 7ff60a507640 1 -- 172.21.15.114:0/4249174344 >> 172.21.15.114:0/4249174344 conn(0x7ff604077fd0 msgr2=0x7ff6040783e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T10:40:50.163 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:40:50.163+0000 7ff60a507640 1 -- 172.21.15.114:0/4249174344 shutdown_connections 2024-04-23T10:40:50.163 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:40:50.163+0000 7ff60a507640 1 -- 172.21.15.114:0/4249174344 wait complete. 2024-04-23T10:40:50.163 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:40:50.163+0000 7ff60a507640 1 Processor -- start 2024-04-23T10:40:50.163 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:40:50.163+0000 7ff60a507640 1 -- start start 2024-04-23T10:40:50.174 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:40:50.164+0000 7ff60a507640 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7ff604101110 0x7ff6041a05b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T10:40:50.175 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:40:50.164+0000 7ff60a507640 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7ff604106190 0x7ff6041a0af0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T10:40:50.175 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:40:50.164+0000 7ff60a507640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7ff60419a6c0 0x7ff60419ab40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T10:40:50.175 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:40:50.164+0000 7ff60a507640 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x7ff604111100 con 0x7ff60419a6c0 2024-04-23T10:40:50.175 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:40:50.164+0000 7ff60a507640 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 -- 0x7ff604110f80 con 0x7ff604106190 2024-04-23T10:40:50.175 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:40:50.164+0000 7ff60a507640 1 -- --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_getmap magic: 0 -- 0x7ff604111280 con 0x7ff604101110 2024-04-23T10:40:50.175 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:40:50.164+0000 7ff608aa4640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7ff60419a6c0 0x7ff60419ab40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T10:40:50.175 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:40:50.164+0000 7ff603fff640 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7ff604101110 0x7ff6041a05b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T10:40:50.175 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:40:50.164+0000 7ff608aa4640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7ff60419a6c0 0x7ff60419ab40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.114:42894/0 (socket says 172.21.15.114:42894) 2024-04-23T10:40:50.176 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:40:50.164+0000 7ff608aa4640 1 -- 172.21.15.114:0/2563505110 learned_addr learned my addr 172.21.15.114:0/2563505110 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-04-23T10:40:50.176 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:40:50.164+0000 7ff6037fe640 1 --2- 172.21.15.114:0/2563505110 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7ff604106190 0x7ff6041a0af0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T10:40:50.176 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:40:50.164+0000 7ff603fff640 1 -- 172.21.15.114:0/2563505110 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7ff604106190 msgr2=0x7ff6041a0af0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T10:40:50.176 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:40:50.164+0000 7ff603fff640 1 --2- 172.21.15.114:0/2563505110 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7ff604106190 0x7ff6041a0af0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:40:50.176 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:40:50.164+0000 7ff603fff640 1 -- 172.21.15.114:0/2563505110 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7ff60419a6c0 msgr2=0x7ff60419ab40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T10:40:50.176 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:40:50.164+0000 7ff603fff640 1 --2- 172.21.15.114:0/2563505110 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7ff60419a6c0 0x7ff60419ab40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:40:50.176 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:40:50.164+0000 7ff603fff640 1 -- 172.21.15.114:0/2563505110 --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff60419b2c0 con 0x7ff604101110 2024-04-23T10:40:50.176 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:40:50.164+0000 7ff608aa4640 1 --2- 172.21.15.114:0/2563505110 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7ff60419a6c0 0x7ff60419ab40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-23T10:40:50.176 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:40:50.164+0000 7ff603fff640 1 --2- 172.21.15.114:0/2563505110 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7ff604101110 0x7ff6041a05b0 secure :-1 s=READY pgs=84 cs=0 l=1 rev1=1 crypto rx=0x7ff5f400e9f0 tx=0x7ff5f400eeb0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T10:40:50.176 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:40:50.164+0000 7ff6017fa640 1 -- 172.21.15.114:0/2563505110 <== mon.1 v2:172.21.15.204:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff5f400ccc0 con 0x7ff604101110 2024-04-23T10:40:50.177 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:40:50.164+0000 7ff60a507640 1 -- 172.21.15.114:0/2563505110 --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7ff6041a7350 con 0x7ff604101110 2024-04-23T10:40:50.177 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:40:50.164+0000 7ff6017fa640 1 -- 172.21.15.114:0/2563505110 <== mon.1 v2:172.21.15.204:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7ff5f40108e0 con 0x7ff604101110 2024-04-23T10:40:50.177 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:40:50.164+0000 7ff6017fa640 1 -- 172.21.15.114:0/2563505110 <== mon.1 v2:172.21.15.204:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff5f4018880 con 0x7ff604101110 2024-04-23T10:40:50.177 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:40:50.164+0000 7ff60a507640 1 -- 172.21.15.114:0/2563505110 --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7ff6041a77b0 con 0x7ff604101110 2024-04-23T10:40:50.177 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:40:50.167+0000 7ff6017fa640 1 -- 172.21.15.114:0/2563505110 <== mon.1 v2:172.21.15.204:3300/0 4 ==== mgrmap(e 26) ==== 138778+0+0 (secure 0 0 0) 0x7ff5f4010430 con 0x7ff604101110 2024-04-23T10:40:50.177 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:40:50.167+0000 7ff60d21a740 1 Processor -- start 2024-04-23T10:40:50.177 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:40:50.167+0000 7ff60d21a740 1 -- start start 2024-04-23T10:40:50.177 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:40:50.167+0000 7ff60d21a740 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x5569ac834620 0x5569ac834a00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T10:40:50.177 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:40:50.167+0000 7ff60d21a740 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x5569ac834f40 0x5569ac83f2c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T10:40:50.177 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:40:50.167+0000 7ff60d21a740 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x5569ac83f800 0x5569ac841bf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T10:40:50.177 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:40:50.167+0000 7ff60d21a740 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x5569ac799970 con 0x5569ac834620 2024-04-23T10:40:50.177 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:40:50.167+0000 7ff60d21a740 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 -- 0x5569ac7364a0 con 0x5569ac83f800 2024-04-23T10:40:50.178 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:40:50.167+0000 7ff60d21a740 1 -- --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_getmap magic: 0 -- 0x5569ac795930 con 0x5569ac834f40 2024-04-23T10:40:50.178 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:40:50.167+0000 7ff6017fa640 1 --2- 172.21.15.114:0/2563505110 >> [v2:172.21.15.114:6800/1395693466,v1:172.21.15.114:6801/1395693466] conn(0x7ff5ec0a3dc0 0x7ff5ec0a6280 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T10:40:50.178 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:40:50.167+0000 7ff608aa4640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x5569ac834620 0x5569ac834a00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T10:40:50.178 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:40:50.167+0000 7ff608aa4640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x5569ac834620 0x5569ac834a00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.114:42902/0 (socket says 172.21.15.114:42902) 2024-04-23T10:40:50.178 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:40:50.167+0000 7ff608aa4640 1 -- 172.21.15.114:0/686366502 learned_addr learned my addr 172.21.15.114:0/686366502 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-04-23T10:40:50.178 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:40:50.167+0000 7ff603fff640 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x5569ac834f40 0x5569ac83f2c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T10:40:50.178 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:40:50.167+0000 7ff608aa4640 1 --2- 172.21.15.114:0/2563505110 >> [v2:172.21.15.114:6800/1395693466,v1:172.21.15.114:6801/1395693466] conn(0x7ff5ec0a3dc0 0x7ff5ec0a6280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T10:40:50.178 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:40:50.167+0000 7ff6037fe640 1 --2- 172.21.15.114:0/686366502 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x5569ac83f800 0x5569ac841bf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T10:40:50.179 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:40:50.167+0000 7ff6017fa640 1 -- 172.21.15.114:0/2563505110 <== mon.1 v2:172.21.15.204:3300/0 5 ==== osd_map(111..111 src has 1..111) ==== 8585+0+0 (secure 0 0 0) 0x7ff5f40d0ba0 con 0x7ff604101110 2024-04-23T10:40:50.179 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:40:50.168+0000 7ff608aa4640 1 -- 172.21.15.114:0/686366502 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x5569ac834f40 msgr2=0x5569ac83f2c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T10:40:50.179 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:40:50.168+0000 7ff608aa4640 1 --2- 172.21.15.114:0/686366502 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x5569ac834f40 0x5569ac83f2c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:40:50.179 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:40:50.168+0000 7ff608aa4640 1 -- 172.21.15.114:0/686366502 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x5569ac83f800 msgr2=0x5569ac841bf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T10:40:50.179 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:40:50.168+0000 7ff608aa4640 1 --2- 172.21.15.114:0/686366502 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x5569ac83f800 0x5569ac841bf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:40:50.179 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:40:50.168+0000 7ff608aa4640 1 -- 172.21.15.114:0/686366502 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5569ac4ac210 con 0x5569ac834620 2024-04-23T10:40:50.179 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:40:50.168+0000 7ff608aa4640 1 --2- 172.21.15.114:0/686366502 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x5569ac834620 0x5569ac834a00 secure :-1 s=READY pgs=935 cs=0 l=1 rev1=1 crypto rx=0x7ff5fc005fe0 tx=0x7ff5fc0317e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b7ed15197c56efb9 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T10:40:50.179 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:40:50.168+0000 7ff608aa4640 1 --2- 172.21.15.114:0/2563505110 >> [v2:172.21.15.114:6800/1395693466,v1:172.21.15.114:6801/1395693466] conn(0x7ff5ec0a3dc0 0x7ff5ec0a6280 secure :-1 s=READY pgs=149 cs=0 l=1 rev1=1 crypto rx=0x7ff5fc031850 tx=0x7ff5fc03b000 comp rx=0 tx=0).ready entity=mgr.15291 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T10:40:50.179 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:40:50.168+0000 7ff5ea7fc640 1 -- 172.21.15.114:0/686366502 <== mon.0 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff5fc004240 con 0x5569ac834620 2024-04-23T10:40:50.179 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:40:50.168+0000 7ff5ea7fc640 1 -- 172.21.15.114:0/686366502 <== mon.0 v2:172.21.15.114:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7ff5fc036040 con 0x5569ac834620 2024-04-23T10:40:50.180 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:40:50.168+0000 7ff5ea7fc640 1 -- 172.21.15.114:0/686366502 <== mon.0 v2:172.21.15.114:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff5fc02fe10 con 0x5569ac834620 2024-04-23T10:40:50.180 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:40:50.169+0000 7ff60d21a740 1 -- 172.21.15.114:0/686366502 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x5569ac834620 msgr2=0x5569ac834a00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T10:40:50.180 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:40:50.169+0000 7ff60d21a740 1 --2- 172.21.15.114:0/686366502 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x5569ac834620 0x5569ac834a00 secure :-1 s=READY pgs=935 cs=0 l=1 rev1=1 crypto rx=0x7ff5fc005fe0 tx=0x7ff5fc0317e0 comp rx=0 tx=0).stop 2024-04-23T10:40:50.180 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:40:50.169+0000 7ff60d21a740 1 -- 172.21.15.114:0/686366502 shutdown_connections 2024-04-23T10:40:50.180 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:40:50.169+0000 7ff60d21a740 1 --2- 172.21.15.114:0/686366502 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x5569ac83f800 0x5569ac841bf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:40:50.180 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:40:50.169+0000 7ff60d21a740 1 --2- 172.21.15.114:0/686366502 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x5569ac834f40 0x5569ac83f2c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:40:50.180 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:40:50.169+0000 7ff60d21a740 1 --2- 172.21.15.114:0/686366502 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x5569ac834620 0x5569ac834a00 unknown :-1 s=CLOSED pgs=935 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:40:50.180 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:40:50.169+0000 7ff60d21a740 1 -- 172.21.15.114:0/686366502 >> 172.21.15.114:0/686366502 conn(0x5569ac830b30 msgr2=0x5569ac830f10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T10:40:50.180 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:40:50.169+0000 7ff60d21a740 1 -- 172.21.15.114:0/686366502 shutdown_connections 2024-04-23T10:40:50.180 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:40:50.169+0000 7ff60d21a740 1 -- 172.21.15.114:0/686366502 wait complete. 2024-04-23T10:40:50.180 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:40:50.169+0000 7ff60d21a740 1 -- start start 2024-04-23T10:40:50.180 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:40:50.169+0000 7ff60d21a740 1 Processor -- start 2024-04-23T10:40:50.180 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:40:50.170+0000 7ff60d21a740 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x5569ac834620 0x5569ac83f200 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T10:40:50.181 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:40:50.170+0000 7ff60d21a740 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x5569ac834f40 0x5569ac83c4f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T10:40:50.181 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:40:50.170+0000 7ff60d21a740 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x5569ac83f800 0x5569ac83cb00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T10:40:50.181 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:40:50.170+0000 7ff60d21a740 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x5569ac842250 con 0x5569ac834f40 2024-04-23T10:40:50.181 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:40:50.170+0000 7ff60d21a740 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 -- 0x5569ac7340b0 con 0x5569ac834620 2024-04-23T10:40:50.181 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:40:50.170+0000 7ff60d21a740 1 -- --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_getmap magic: 0 -- 0x5569ac8423d0 con 0x5569ac83f800 2024-04-23T10:40:50.181 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:40:50.170+0000 7ff60d21a740 10 client.0 _ll_register_callbacks cb 0x5569ac721d30 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-23T10:40:50.181 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:40:50.170+0000 7ff6037fe640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x5569ac834f40 0x5569ac83c4f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T10:40:50.181 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:40:50.170+0000 7ff603fff640 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x5569ac834620 0x5569ac83f200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T10:40:50.181 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:40:50.170+0000 7ff6037fe640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x5569ac834f40 0x5569ac83c4f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.114:42914/0 (socket says 172.21.15.114:42914) 2024-04-23T10:40:50.182 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:40:50.170+0000 7ff6037fe640 1 -- 172.21.15.114:0/808264047 learned_addr learned my addr 172.21.15.114:0/808264047 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-04-23T10:40:50.182 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:40:50.170+0000 7ff608aa4640 1 --2- 172.21.15.114:0/808264047 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x5569ac83f800 0x5569ac83cb00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T10:40:50.182 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:40:50.170+0000 7ff6037fe640 1 -- 172.21.15.114:0/808264047 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x5569ac83f800 msgr2=0x5569ac83cb00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T10:40:50.182 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:40:50.170+0000 7ff6037fe640 1 --2- 172.21.15.114:0/808264047 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x5569ac83f800 0x5569ac83cb00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:40:50.182 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:40:50.170+0000 7ff6037fe640 1 -- 172.21.15.114:0/808264047 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x5569ac834620 msgr2=0x5569ac83f200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T10:40:50.182 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:40:50.170+0000 7ff6037fe640 1 --2- 172.21.15.114:0/808264047 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x5569ac834620 0x5569ac83f200 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:40:50.182 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:40:50.170+0000 7ff6037fe640 1 -- 172.21.15.114:0/808264047 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5569ac578620 con 0x5569ac834f40 2024-04-23T10:40:50.182 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:40:50.170+0000 7ff6037fe640 1 --2- 172.21.15.114:0/808264047 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x5569ac834f40 0x5569ac83c4f0 secure :-1 s=READY pgs=936 cs=0 l=1 rev1=1 crypto rx=0x7ff5f8012700 tx=0x7ff5f8012bc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=bbb9b6334bedd323 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T10:40:50.182 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:40:50.170+0000 7ff5e37fe640 10 client.0 ms_handle_connect on v2:172.21.15.114:3300/0 2024-04-23T10:40:50.182 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:40:50.170+0000 7ff5e37fe640 1 -- 172.21.15.114:0/808264047 <== mon.0 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff5f800f920 con 0x5569ac834f40 2024-04-23T10:40:50.183 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:40:50.170+0000 7ff5e37fe640 1 -- 172.21.15.114:0/808264047 <== mon.0 v2:172.21.15.114:3300/0 2 ==== osd_map(111..111 src has 1..111) ==== 8585+0+0 (secure 0 0 0) 0x7ff5f800fac0 con 0x5569ac834f40 2024-04-23T10:40:50.183 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:40:50.170+0000 7ff60d21a740 1 -- 172.21.15.114:0/808264047 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7ff5fc036040 con 0x5569ac834f40 2024-04-23T10:40:50.183 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:40:50.171+0000 7ff5e37fe640 1 -- 172.21.15.114:0/808264047 <== mon.0 v2:172.21.15.114:3300/0 3 ==== mon_get_version_reply(handle=1 version=49) ==== 24+0+0 (secure 0 0 0) 0x7ff5f800fcc0 con 0x5569ac834f40 2024-04-23T10:40:50.183 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:40:50.171+0000 7ff60d21a740 10 client.17109 fetch_fsmap learned FSMap version 49 2024-04-23T10:40:50.183 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:40:50.171+0000 7ff60d21a740 1 -- 172.21.15.114:0/808264047 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({config=0+,fsmap=49,monmap=4+}) -- 0x5569ac84e2b0 con 0x5569ac834f40 2024-04-23T10:40:50.183 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:40:50.171+0000 7ff5e37fe640 1 -- 172.21.15.114:0/808264047 <== mon.0 v2:172.21.15.114:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7ff5fc036040 con 0x5569ac834f40 2024-04-23T10:40:50.183 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:40:50.171+0000 7ff5e37fe640 1 -- 172.21.15.114:0/808264047 <== mon.0 v2:172.21.15.114:3300/0 5 ==== fsmap(e 49) ==== 8358+0+0 (secure 0 0 0) 0x7ff5f8010450 con 0x5569ac834f40 2024-04-23T10:40:50.183 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:40:50.171+0000 7ff60d21a740 10 client.17109 fetch_fsmap finished waiting for FSMap version 49 2024-04-23T10:40:50.183 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:40:50.171+0000 7ff60d21a740 10 client.17109 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-04-23T10:40:50.183 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:40:50.171+0000 7ff60d21a740 20 client.17109 populate_metadata read hostname 'smithi114' 2024-04-23T10:40:50.183 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:40:50.171+0000 7ff60d21a740 1 --2- 172.21.15.114:0/808264047 >> [v2:172.21.15.119:6838/937297340,v1:172.21.15.119:6839/937297340] conn(0x5569ac84f7c0 0x5569ac851c10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T10:40:50.183 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:40:50.171+0000 7ff60d21a740 4 client.17109 mds_command: new command op to 24545 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-23T10:40:50.184 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:40:50.171+0000 7ff60d21a740 1 -- 172.21.15.114:0/808264047 --> [v2:172.21.15.119:6838/937297340,v1:172.21.15.119:6839/937297340] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x5569ac7a7390 con 0x5569ac84f7c0 2024-04-23T10:40:50.184 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:40:50.171+0000 7ff603fff640 1 --2- 172.21.15.114:0/808264047 >> [v2:172.21.15.119:6838/937297340,v1:172.21.15.119:6839/937297340] conn(0x5569ac84f7c0 0x5569ac851c10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T10:40:50.184 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:40:50.172+0000 7ff5e37fe640 10 client.17109 ms_handle_connect on v2:172.21.15.119:6838/937297340 2024-04-23T10:40:50.184 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:40:50.172+0000 7ff603fff640 1 --2- 172.21.15.114:0/808264047 >> [v2:172.21.15.119:6838/937297340,v1:172.21.15.119:6839/937297340] conn(0x5569ac84f7c0 0x5569ac851c10 crc :-1 s=READY pgs=188 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=f6a2646c64585bfc server_cookie=2b4f616423c902a1 in_seq=0 out_seq=0 2024-04-23T10:40:50.184 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:40:50.173+0000 7ff5e37fe640 1 -- 172.21.15.114:0/808264047 <== mds.0 v2:172.21.15.119:6838/937297340 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x5569ac7a7390 con 0x5569ac84f7c0 2024-04-23T10:40:50.184 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:40:50.173+0000 7ff5e37fe640 10 client.17109 handle_command_reply: tid=0 2024-04-23T10:40:50.184 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:40:50.173+0000 7ff60d21a740 1 client.17109 shutdown 2024-04-23T10:40:50.184 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:40:50.174+0000 7ff60d21a740 1 -- 172.21.15.114:0/808264047 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x5569ac834f40 msgr2=0x5569ac83c4f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T10:40:50.184 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:40:50.174+0000 7ff60d21a740 1 --2- 172.21.15.114:0/808264047 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x5569ac834f40 0x5569ac83c4f0 secure :-1 s=READY pgs=936 cs=0 l=1 rev1=1 crypto rx=0x7ff5f8012700 tx=0x7ff5f8012bc0 comp rx=0 tx=0).stop 2024-04-23T10:40:50.184 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:40:50.174+0000 7ff60d21a740 1 -- 172.21.15.114:0/808264047 shutdown_connections 2024-04-23T10:40:50.184 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:40:50.174+0000 7ff60d21a740 1 --2- 172.21.15.114:0/808264047 >> [v2:172.21.15.119:6838/937297340,v1:172.21.15.119:6839/937297340] conn(0x5569ac84f7c0 0x5569ac851c10 crc :-1 s=READY pgs=188 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:40:50.184 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:40:50.174+0000 7ff60d21a740 1 --2- 172.21.15.114:0/808264047 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x5569ac83f800 0x5569ac83cb00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:40:50.184 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:40:50.174+0000 7ff60d21a740 1 --2- 172.21.15.114:0/808264047 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x5569ac834f40 0x5569ac83c4f0 unknown :-1 s=CLOSED pgs=936 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:40:50.185 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:40:50.174+0000 7ff5e37fe640 0 client.17109 ms_handle_reset on v2:172.21.15.119:6838/937297340 2024-04-23T10:40:50.185 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:40:50.174+0000 7ff60d21a740 1 --2- 172.21.15.114:0/808264047 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x5569ac834620 0x5569ac83f200 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:40:50.185 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:40:50.174+0000 7ff60d21a740 1 -- 172.21.15.114:0/808264047 >> 172.21.15.114:0/808264047 conn(0x5569ac830b30 msgr2=0x5569ac836860 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T10:40:50.185 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:40:50.174+0000 7ff60d21a740 1 -- 172.21.15.114:0/808264047 shutdown_connections 2024-04-23T10:40:50.185 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:40:50.174+0000 7ff60d21a740 1 -- 172.21.15.114:0/808264047 wait complete. 2024-04-23T10:40:50.185 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:40:50.174+0000 7ff60d21a740 20 client.17109 trim_cache size 0 max 16384 2024-04-23T10:40:50.192 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:40:50.191+0000 7ff60d21a740 1 Processor -- start 2024-04-23T10:40:50.192 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:40:50.191+0000 7ff60d21a740 1 -- start start 2024-04-23T10:40:50.192 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:40:50.191+0000 7ff60d21a740 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x5569ac834620 0x5569ac84b110 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T10:40:50.192 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:40:50.191+0000 7ff60d21a740 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x5569ac834f40 0x5569ac82fb30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T10:40:50.192 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:40:50.191+0000 7ff60d21a740 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x5569ac83f800 0x5569ac83ed30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T10:40:50.192 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:40:50.191+0000 7ff60d21a740 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x7ff6041b2720 con 0x5569ac834f40 2024-04-23T10:40:50.193 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:40:50.191+0000 7ff60d21a740 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 -- 0x7ff6041aff60 con 0x5569ac83f800 2024-04-23T10:40:50.193 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:40:50.191+0000 7ff60d21a740 1 -- --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_getmap magic: 0 -- 0x7ff6041addb0 con 0x5569ac834620 2024-04-23T10:40:50.193 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:40:50.191+0000 7ff6037fe640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x5569ac834f40 0x5569ac82fb30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T10:40:50.193 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:40:50.191+0000 7ff603fff640 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x5569ac834620 0x5569ac84b110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T10:40:50.193 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:40:50.191+0000 7ff6037fe640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x5569ac834f40 0x5569ac82fb30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.114:42918/0 (socket says 172.21.15.114:42918) 2024-04-23T10:40:50.193 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:40:50.191+0000 7ff6037fe640 1 -- 172.21.15.114:0/2926717909 learned_addr learned my addr 172.21.15.114:0/2926717909 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-04-23T10:40:50.193 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:40:50.192+0000 7ff608aa4640 1 --2- 172.21.15.114:0/2926717909 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x5569ac83f800 0x5569ac83ed30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T10:40:50.193 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:40:50.192+0000 7ff603fff640 1 -- 172.21.15.114:0/2926717909 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x5569ac83f800 msgr2=0x5569ac83ed30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T10:40:50.193 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:40:50.192+0000 7ff603fff640 1 --2- 172.21.15.114:0/2926717909 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x5569ac83f800 0x5569ac83ed30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:40:50.193 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:40:50.192+0000 7ff603fff640 1 -- 172.21.15.114:0/2926717909 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x5569ac834f40 msgr2=0x5569ac82fb30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T10:40:50.193 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:40:50.192+0000 7ff603fff640 1 --2- 172.21.15.114:0/2926717909 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x5569ac834f40 0x5569ac82fb30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:40:50.193 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:40:50.192+0000 7ff603fff640 1 -- 172.21.15.114:0/2926717909 --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5569ac84e0c0 con 0x5569ac834620 2024-04-23T10:40:50.194 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:40:50.192+0000 7ff608aa4640 1 --2- 172.21.15.114:0/2926717909 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x5569ac83f800 0x5569ac83ed30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-23T10:40:50.194 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:40:50.192+0000 7ff603fff640 1 --2- 172.21.15.114:0/2926717909 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x5569ac834620 0x5569ac84b110 secure :-1 s=READY pgs=85 cs=0 l=1 rev1=1 crypto rx=0x7ff5f40ce250 tx=0x7ff5f406f640 comp rx=0 tx=0).ready entity=mon.1 client_cookie=41144831b0cf7c29 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T10:40:50.194 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:40:50.192+0000 7ff5e9ffb640 1 -- 172.21.15.114:0/2926717909 <== mon.1 v2:172.21.15.204:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff5f409abf0 con 0x5569ac834620 2024-04-23T10:40:50.194 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:40:50.192+0000 7ff5e9ffb640 1 -- 172.21.15.114:0/2926717909 <== mon.1 v2:172.21.15.204:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7ff5f409ae50 con 0x5569ac834620 2024-04-23T10:40:50.194 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:40:50.192+0000 7ff5e9ffb640 1 -- 172.21.15.114:0/2926717909 <== mon.1 v2:172.21.15.204:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff5f4099350 con 0x5569ac834620 2024-04-23T10:40:50.194 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:40:50.193+0000 7ff60d21a740 1 -- 172.21.15.114:0/2926717909 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x5569ac834620 msgr2=0x5569ac84b110 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T10:40:50.194 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:40:50.193+0000 7ff60d21a740 1 --2- 172.21.15.114:0/2926717909 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x5569ac834620 0x5569ac84b110 secure :-1 s=READY pgs=85 cs=0 l=1 rev1=1 crypto rx=0x7ff5f40ce250 tx=0x7ff5f406f640 comp rx=0 tx=0).stop 2024-04-23T10:40:50.194 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:40:50.193+0000 7ff60d21a740 1 -- 172.21.15.114:0/2926717909 shutdown_connections 2024-04-23T10:40:50.194 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:40:50.193+0000 7ff60d21a740 1 --2- 172.21.15.114:0/2926717909 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x5569ac83f800 0x5569ac83ed30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:40:50.194 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:40:50.193+0000 7ff60d21a740 1 --2- 172.21.15.114:0/2926717909 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x5569ac834f40 0x5569ac82fb30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:40:50.194 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:40:50.193+0000 7ff60d21a740 1 --2- 172.21.15.114:0/2926717909 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x5569ac834620 0x5569ac84b110 unknown :-1 s=CLOSED pgs=85 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:40:50.194 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:40:50.193+0000 7ff60d21a740 1 -- 172.21.15.114:0/2926717909 >> 172.21.15.114:0/2926717909 conn(0x5569ac830b30 msgr2=0x5569ac82f420 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T10:40:50.195 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:40:50.193+0000 7ff60d21a740 1 -- 172.21.15.114:0/2926717909 shutdown_connections 2024-04-23T10:40:50.195 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:40:50.193+0000 7ff60d21a740 1 -- 172.21.15.114:0/2926717909 wait complete. 2024-04-23T10:40:50.198 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:40:50.194+0000 7ff60d21a740 1 -- start start 2024-04-23T10:40:50.198 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:40:50.194+0000 7ff60d21a740 1 Processor -- start 2024-04-23T10:40:50.198 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:40:50.194+0000 7ff60d21a740 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x5569ac834620 0x5569ac8394e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T10:40:50.198 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:40:50.194+0000 7ff60d21a740 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x5569ac834f40 0x5569ac839a20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T10:40:50.198 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:40:50.194+0000 7ff60d21a740 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x5569ac83f800 0x5569ac839f80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T10:40:50.198 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:40:50.194+0000 7ff60d21a740 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x7ff6041ae420 con 0x5569ac83f800 2024-04-23T10:40:50.198 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:40:50.194+0000 7ff60d21a740 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 -- 0x7ff6041af890 con 0x5569ac834f40 2024-04-23T10:40:50.198 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:40:50.195+0000 7ff60d21a740 1 -- --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_getmap magic: 0 -- 0x7ff6041aeaf0 con 0x5569ac834620 2024-04-23T10:40:50.198 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:40:50.195+0000 7ff60d21a740 10 client.0 _ll_register_callbacks cb 0x7ff5f8009de0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-23T10:40:50.199 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:40:50.195+0000 7ff603fff640 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x5569ac834620 0x5569ac8394e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T10:40:50.199 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:40:50.195+0000 7ff6037fe640 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x5569ac834f40 0x5569ac839a20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T10:40:50.199 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:40:50.195+0000 7ff603fff640 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x5569ac834620 0x5569ac8394e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.204:3300/0 says I am v2:172.21.15.114:60556/0 (socket says 172.21.15.114:60556) 2024-04-23T10:40:50.199 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:40:50.195+0000 7ff603fff640 1 -- 172.21.15.114:0/1437619240 learned_addr learned my addr 172.21.15.114:0/1437619240 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-04-23T10:40:50.199 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:40:50.195+0000 7ff608aa4640 1 --2- 172.21.15.114:0/1437619240 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x5569ac83f800 0x5569ac839f80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T10:40:50.199 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:40:50.195+0000 7ff603fff640 1 -- 172.21.15.114:0/1437619240 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x5569ac834f40 msgr2=0x5569ac839a20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T10:40:50.199 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:40:50.195+0000 7ff603fff640 1 --2- 172.21.15.114:0/1437619240 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x5569ac834f40 0x5569ac839a20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:40:50.199 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:40:50.195+0000 7ff603fff640 1 -- 172.21.15.114:0/1437619240 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x5569ac83f800 msgr2=0x5569ac839f80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T10:40:50.200 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:40:50.195+0000 7ff603fff640 1 --2- 172.21.15.114:0/1437619240 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x5569ac83f800 0x5569ac839f80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:40:50.200 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:40:50.195+0000 7ff603fff640 1 -- 172.21.15.114:0/1437619240 --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5569ac8bd830 con 0x5569ac834620 2024-04-23T10:40:50.200 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:40:50.195+0000 7ff608aa4640 1 --2- 172.21.15.114:0/1437619240 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x5569ac83f800 0x5569ac839f80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-23T10:40:50.200 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:40:50.195+0000 7ff603fff640 1 --2- 172.21.15.114:0/1437619240 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x5569ac834620 0x5569ac8394e0 secure :-1 s=READY pgs=86 cs=0 l=1 rev1=1 crypto rx=0x7ff5f4010810 tx=0x7ff5f4009700 comp rx=0 tx=0).ready entity=mon.1 client_cookie=7e67aaa3164ac7ab server_cookie=0 in_seq=0 out_seq=0 2024-04-23T10:40:50.200 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:40:50.195+0000 7ff5e37fe640 10 client.0 ms_handle_connect on v2:172.21.15.204:3300/0 2024-04-23T10:40:50.200 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:40:50.195+0000 7ff5e37fe640 1 -- 172.21.15.114:0/1437619240 <== mon.1 v2:172.21.15.204:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff5f40991c0 con 0x5569ac834620 2024-04-23T10:40:50.200 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:40:50.195+0000 7ff5e37fe640 1 -- 172.21.15.114:0/1437619240 <== mon.1 v2:172.21.15.204:3300/0 2 ==== osd_map(111..111 src has 1..111) ==== 8585+0+0 (secure 0 0 0) 0x7ff5f40ce4d0 con 0x5569ac834620 2024-04-23T10:40:50.200 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:40:50.195+0000 7ff60d21a740 1 -- 172.21.15.114:0/1437619240 --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7ff5f409ae50 con 0x5569ac834620 2024-04-23T10:40:50.200 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:40:50.195+0000 7ff5e37fe640 1 -- 172.21.15.114:0/1437619240 <== mon.1 v2:172.21.15.204:3300/0 3 ==== mon_get_version_reply(handle=1 version=49) ==== 24+0+0 (secure 0 0 0) 0x7ff5f4099360 con 0x5569ac834620 2024-04-23T10:40:50.200 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:40:50.195+0000 7ff60d21a740 10 client.26278 fetch_fsmap learned FSMap version 49 2024-04-23T10:40:50.200 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:40:50.195+0000 7ff60d21a740 1 -- 172.21.15.114:0/1437619240 --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_subscribe({config=0+,fsmap=49,monmap=4+}) -- 0x5569ac8beb40 con 0x5569ac834620 2024-04-23T10:40:50.201 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:40:50.196+0000 7ff5e37fe640 1 -- 172.21.15.114:0/1437619240 <== mon.1 v2:172.21.15.204:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7ff5f409ae50 con 0x5569ac834620 2024-04-23T10:40:50.201 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:40:50.196+0000 7ff5e37fe640 1 -- 172.21.15.114:0/1437619240 <== mon.1 v2:172.21.15.204:3300/0 5 ==== fsmap(e 49) ==== 8358+0+0 (secure 0 0 0) 0x7ff5f409bb30 con 0x5569ac834620 2024-04-23T10:40:50.201 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:40:50.196+0000 7ff60d21a740 10 client.26278 fetch_fsmap finished waiting for FSMap version 49 2024-04-23T10:40:50.201 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:40:50.196+0000 7ff60d21a740 10 client.26278 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-04-23T10:40:50.201 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:40:50.196+0000 7ff60d21a740 20 client.26278 populate_metadata read hostname 'smithi114' 2024-04-23T10:40:50.201 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:40:50.196+0000 7ff60d21a740 1 --2- 172.21.15.114:0/1437619240 >> [v2:172.21.15.119:6838/937297340,v1:172.21.15.119:6839/937297340] conn(0x5569ac8bf160 0x5569ac8c1550 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T10:40:50.201 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:40:50.196+0000 7ff60d21a740 4 client.26278 mds_command: new command op to 24545 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-04-23T10:40:50.201 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:40:50.196+0000 7ff60d21a740 1 -- 172.21.15.114:0/1437619240 --> [v2:172.21.15.119:6838/937297340,v1:172.21.15.119:6839/937297340] -- command(tid 0: {"prefix": "flush journal"}) -- 0x5569ac7a1200 con 0x5569ac8bf160 2024-04-23T10:40:50.201 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:40:50.196+0000 7ff6037fe640 1 --2- 172.21.15.114:0/1437619240 >> [v2:172.21.15.119:6838/937297340,v1:172.21.15.119:6839/937297340] conn(0x5569ac8bf160 0x5569ac8c1550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T10:40:50.201 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:40:50.197+0000 7ff5e37fe640 10 client.26278 ms_handle_connect on v2:172.21.15.119:6838/937297340 2024-04-23T10:40:50.202 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:40:50.197+0000 7ff6037fe640 1 --2- 172.21.15.114:0/1437619240 >> [v2:172.21.15.119:6838/937297340,v1:172.21.15.119:6839/937297340] conn(0x5569ac8bf160 0x5569ac8c1550 crc :-1 s=READY pgs=189 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=d542cf4e5568836f server_cookie=2ba619a11d217b9c in_seq=0 out_seq=0 2024-04-23T10:40:50.454 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:40:50.454+0000 7ff5e37fe640 1 -- 172.21.15.114:0/1437619240 <== mds.0 v2:172.21.15.119:6838/937297340 1 ==== command_reply(tid 0: 0 ) ==== 8+0+44 (crc 0 0 0) 0x5569ac7a1200 con 0x5569ac8bf160 2024-04-23T10:40:50.454 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:40:50.454+0000 7ff5e37fe640 10 client.26278 handle_command_reply: tid=0 2024-04-23T10:40:50.454 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:40:50.454+0000 7ff60d21a740 1 client.26278 shutdown 2024-04-23T10:40:50.455 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:40:50.454+0000 7ff60d21a740 1 -- 172.21.15.114:0/1437619240 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x5569ac834620 msgr2=0x5569ac8394e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T10:40:50.455 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:40:50.454+0000 7ff60d21a740 1 --2- 172.21.15.114:0/1437619240 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x5569ac834620 0x5569ac8394e0 secure :-1 s=READY pgs=86 cs=0 l=1 rev1=1 crypto rx=0x7ff5f4010810 tx=0x7ff5f4009700 comp rx=0 tx=0).stop 2024-04-23T10:40:50.455 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:40:50.454+0000 7ff60d21a740 1 -- 172.21.15.114:0/1437619240 shutdown_connections 2024-04-23T10:40:50.455 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:40:50.454+0000 7ff60d21a740 1 --2- 172.21.15.114:0/1437619240 >> [v2:172.21.15.119:6838/937297340,v1:172.21.15.119:6839/937297340] conn(0x5569ac8bf160 0x5569ac8c1550 crc :-1 s=READY pgs=189 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:40:50.455 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:40:50.454+0000 7ff60d21a740 1 --2- 172.21.15.114:0/1437619240 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x5569ac83f800 0x5569ac839f80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:40:50.455 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:40:50.454+0000 7ff60d21a740 1 --2- 172.21.15.114:0/1437619240 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x5569ac834f40 0x5569ac839a20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:40:50.455 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:40:50.454+0000 7ff5e37fe640 0 client.26278 ms_handle_reset on v2:172.21.15.119:6838/937297340 2024-04-23T10:40:50.455 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:40:50.454+0000 7ff60d21a740 1 --2- 172.21.15.114:0/1437619240 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x5569ac834620 0x5569ac8394e0 unknown :-1 s=CLOSED pgs=86 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:40:50.455 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:40:50.454+0000 7ff60d21a740 1 -- 172.21.15.114:0/1437619240 >> 172.21.15.114:0/1437619240 conn(0x5569ac830b30 msgr2=0x5569ac836860 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T10:40:50.455 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:40:50.455+0000 7ff60d21a740 1 -- 172.21.15.114:0/1437619240 shutdown_connections 2024-04-23T10:40:50.455 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:40:50.455+0000 7ff60d21a740 1 -- 172.21.15.114:0/1437619240 wait complete. 2024-04-23T10:40:50.455 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:40:50.455+0000 7ff60d21a740 20 client.26278 trim_cache size 0 max 16384 2024-04-23T10:40:50.456 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stdout:{ 2024-04-23T10:40:50.456 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stdout: "message": "", 2024-04-23T10:40:50.456 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stdout: "return_code": 0 2024-04-23T10:40:50.456 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stdout:} 2024-04-23T10:40:50.457 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:40:50.455+0000 7ff60a507640 1 -- 172.21.15.114:0/2563505110 >> [v2:172.21.15.114:6800/1395693466,v1:172.21.15.114:6801/1395693466] conn(0x7ff5ec0a3dc0 msgr2=0x7ff5ec0a6280 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T10:40:50.457 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:40:50.455+0000 7ff60a507640 1 --2- 172.21.15.114:0/2563505110 >> [v2:172.21.15.114:6800/1395693466,v1:172.21.15.114:6801/1395693466] conn(0x7ff5ec0a3dc0 0x7ff5ec0a6280 secure :-1 s=READY pgs=149 cs=0 l=1 rev1=1 crypto rx=0x7ff5fc031850 tx=0x7ff5fc03b000 comp rx=0 tx=0).stop 2024-04-23T10:40:50.457 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:40:50.455+0000 7ff60a507640 1 -- 172.21.15.114:0/2563505110 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7ff604101110 msgr2=0x7ff6041a05b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T10:40:50.457 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:40:50.455+0000 7ff60a507640 1 --2- 172.21.15.114:0/2563505110 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7ff604101110 0x7ff6041a05b0 secure :-1 s=READY pgs=84 cs=0 l=1 rev1=1 crypto rx=0x7ff5f400e9f0 tx=0x7ff5f400eeb0 comp rx=0 tx=0).stop 2024-04-23T10:40:50.457 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:40:50.455+0000 7ff60a507640 1 -- 172.21.15.114:0/2563505110 shutdown_connections 2024-04-23T10:40:50.457 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:40:50.455+0000 7ff60a507640 1 --2- 172.21.15.114:0/2563505110 >> [v2:172.21.15.114:6800/1395693466,v1:172.21.15.114:6801/1395693466] conn(0x7ff5ec0a3dc0 0x7ff5ec0a6280 unknown :-1 s=CLOSED pgs=149 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:40:50.457 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:40:50.455+0000 7ff60a507640 1 --2- 172.21.15.114:0/2563505110 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7ff60419a6c0 0x7ff60419ab40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:40:50.458 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:40:50.455+0000 7ff60a507640 1 --2- 172.21.15.114:0/2563505110 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7ff604106190 0x7ff6041a0af0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:40:50.458 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:40:50.455+0000 7ff60a507640 1 --2- 172.21.15.114:0/2563505110 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7ff604101110 0x7ff6041a05b0 unknown :-1 s=CLOSED pgs=84 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:40:50.458 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:40:50.455+0000 7ff60a507640 1 -- 172.21.15.114:0/2563505110 >> 172.21.15.114:0/2563505110 conn(0x7ff604077fd0 msgr2=0x7ff60410d3c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T10:40:50.458 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:40:50.455+0000 7ff60a507640 1 -- 172.21.15.114:0/2563505110 shutdown_connections 2024-04-23T10:40:50.458 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:40:50.456+0000 7ff60a507640 1 -- 172.21.15.114:0/2563505110 wait complete. 2024-04-23T10:40:50.491 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 23 10:40:50 smithi119 ceph-mon[32608]: pgmap v448: 97 pgs: 97 active+clean; 527 MiB data, 4.1 GiB used, 1.0 TiB / 1.0 TiB avail; 9.1 MiB/s rd, 8.2 MiB/s wr, 560 op/s 2024-04-23T10:40:50.497 INFO:tasks.workunit.client.0.smithi114.stdout: 10 27698 29.70 MB/sec warmup 56 sec latency 169.031 ms 2024-04-23T10:40:50.503 INFO:journalctl@ceph.mon.a.smithi114.stdout:Apr 23 10:40:50 smithi114 ceph-mon[29255]: pgmap v448: 97 pgs: 97 active+clean; 527 MiB data, 4.1 GiB used, 1.0 TiB / 1.0 TiB avail; 9.1 MiB/s rd, 8.2 MiB/s wr, 560 op/s 2024-04-23T10:40:50.578 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 23 10:40:50 smithi204 ceph-mon[32649]: pgmap v448: 97 pgs: 97 active+clean; 527 MiB data, 4.1 GiB used, 1.0 TiB / 1.0 TiB avail; 9.1 MiB/s rd, 8.2 MiB/s wr, 560 op/s 2024-04-23T10:40:51.497 INFO:tasks.workunit.client.0.smithi114.stdout: 10 28349 29.80 MB/sec warmup 57 sec latency 124.606 ms 2024-04-23T10:40:52.491 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 23 10:40:52 smithi119 ceph-mon[32608]: pgmap v449: 97 pgs: 97 active+clean; 527 MiB data, 4.1 GiB used, 1.0 TiB / 1.0 TiB avail; 9.1 MiB/s rd, 8.2 MiB/s wr, 560 op/s 2024-04-23T10:40:52.497 INFO:tasks.workunit.client.0.smithi114.stdout: 10 28991 29.93 MB/sec warmup 58 sec latency 52.259 ms 2024-04-23T10:40:52.503 INFO:journalctl@ceph.mon.a.smithi114.stdout:Apr 23 10:40:52 smithi114 ceph-mon[29255]: pgmap v449: 97 pgs: 97 active+clean; 527 MiB data, 4.1 GiB used, 1.0 TiB / 1.0 TiB avail; 9.1 MiB/s rd, 8.2 MiB/s wr, 560 op/s 2024-04-23T10:40:52.577 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 23 10:40:52 smithi204 ceph-mon[32649]: pgmap v449: 97 pgs: 97 active+clean; 527 MiB data, 4.1 GiB used, 1.0 TiB / 1.0 TiB avail; 9.1 MiB/s rd, 8.2 MiB/s wr, 560 op/s 2024-04-23T10:40:53.497 INFO:tasks.workunit.client.0.smithi114.stdout: 10 29714 30.11 MB/sec warmup 59 sec latency 32.900 ms 2024-04-23T10:40:54.491 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 23 10:40:54 smithi119 ceph-mon[32608]: pgmap v450: 97 pgs: 97 active+clean; 531 MiB data, 4.2 GiB used, 1.0 TiB / 1.0 TiB avail; 15 MiB/s rd, 11 MiB/s wr, 896 op/s 2024-04-23T10:40:54.497 INFO:tasks.workunit.client.0.smithi114.stdout: 10 30225 30.11 MB/sec warmup 60 sec latency 166.553 ms 2024-04-23T10:40:54.503 INFO:journalctl@ceph.mon.a.smithi114.stdout:Apr 23 10:40:54 smithi114 ceph-mon[29255]: pgmap v450: 97 pgs: 97 active+clean; 531 MiB data, 4.2 GiB used, 1.0 TiB / 1.0 TiB avail; 15 MiB/s rd, 11 MiB/s wr, 896 op/s 2024-04-23T10:40:54.577 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 23 10:40:54 smithi204 ceph-mon[32649]: pgmap v450: 97 pgs: 97 active+clean; 531 MiB data, 4.2 GiB used, 1.0 TiB / 1.0 TiB avail; 15 MiB/s rd, 11 MiB/s wr, 896 op/s 2024-04-23T10:40:55.498 INFO:tasks.workunit.client.0.smithi114.stdout: 10 30786 30.09 MB/sec warmup 61 sec latency 109.671 ms 2024-04-23T10:40:56.491 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 23 10:40:56 smithi119 ceph-mon[32608]: pgmap v451: 97 pgs: 97 active+clean; 531 MiB data, 4.2 GiB used, 1.0 TiB / 1.0 TiB avail; 9.9 MiB/s rd, 8.8 MiB/s wr, 565 op/s 2024-04-23T10:40:56.497 INFO:tasks.workunit.client.0.smithi114.stdout: 10 31380 30.07 MB/sec warmup 62 sec latency 92.427 ms 2024-04-23T10:40:56.504 INFO:journalctl@ceph.mon.a.smithi114.stdout:Apr 23 10:40:56 smithi114 ceph-mon[29255]: pgmap v451: 97 pgs: 97 active+clean; 531 MiB data, 4.2 GiB used, 1.0 TiB / 1.0 TiB avail; 9.9 MiB/s rd, 8.8 MiB/s wr, 565 op/s 2024-04-23T10:40:56.577 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 23 10:40:56 smithi204 ceph-mon[32649]: pgmap v451: 97 pgs: 97 active+clean; 531 MiB data, 4.2 GiB used, 1.0 TiB / 1.0 TiB avail; 9.9 MiB/s rd, 8.8 MiB/s wr, 565 op/s 2024-04-23T10:40:57.498 INFO:tasks.workunit.client.0.smithi114.stdout: 10 32015 30.17 MB/sec warmup 63 sec latency 49.456 ms 2024-04-23T10:40:58.491 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 23 10:40:58 smithi119 ceph-mon[32608]: pgmap v452: 97 pgs: 97 active+clean; 550 MiB data, 4.3 GiB used, 1.0 TiB / 1.0 TiB avail; 15 MiB/s rd, 13 MiB/s wr, 776 op/s 2024-04-23T10:40:58.497 INFO:tasks.workunit.client.0.smithi114.stdout: 10 32686 30.19 MB/sec warmup 64 sec latency 63.820 ms 2024-04-23T10:40:58.503 INFO:journalctl@ceph.mon.a.smithi114.stdout:Apr 23 10:40:58 smithi114 ceph-mon[29255]: pgmap v452: 97 pgs: 97 active+clean; 550 MiB data, 4.3 GiB used, 1.0 TiB / 1.0 TiB avail; 15 MiB/s rd, 13 MiB/s wr, 776 op/s 2024-04-23T10:40:58.577 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 23 10:40:58 smithi204 ceph-mon[32649]: pgmap v452: 97 pgs: 97 active+clean; 550 MiB data, 4.3 GiB used, 1.0 TiB / 1.0 TiB avail; 15 MiB/s rd, 13 MiB/s wr, 776 op/s 2024-04-23T10:40:59.498 INFO:tasks.workunit.client.0.smithi114.stdout: 10 33074 30.07 MB/sec warmup 65 sec latency 346.006 ms 2024-04-23T10:41:00.492 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 23 10:41:00 smithi119 ceph-mon[32608]: pgmap v453: 97 pgs: 97 active+clean; 550 MiB data, 4.3 GiB used, 1.0 TiB / 1.0 TiB avail; 10 MiB/s rd, 6.8 MiB/s wr, 547 op/s 2024-04-23T10:41:00.492 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 23 10:41:00 smithi119 ceph-mon[32608]: from='mgr.15291 172.21.15.114:0/1095789187' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-23T10:41:00.497 INFO:tasks.workunit.client.0.smithi114.stdout: 10 33517 30.06 MB/sec warmup 66 sec latency 228.895 ms 2024-04-23T10:41:00.503 INFO:journalctl@ceph.mon.a.smithi114.stdout:Apr 23 10:41:00 smithi114 ceph-mon[29255]: pgmap v453: 97 pgs: 97 active+clean; 550 MiB data, 4.3 GiB used, 1.0 TiB / 1.0 TiB avail; 10 MiB/s rd, 6.8 MiB/s wr, 547 op/s 2024-04-23T10:41:00.503 INFO:journalctl@ceph.mon.a.smithi114.stdout:Apr 23 10:41:00 smithi114 ceph-mon[29255]: from='mgr.15291 172.21.15.114:0/1095789187' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-23T10:41:00.577 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 23 10:41:00 smithi204 ceph-mon[32649]: pgmap v453: 97 pgs: 97 active+clean; 550 MiB data, 4.3 GiB used, 1.0 TiB / 1.0 TiB avail; 10 MiB/s rd, 6.8 MiB/s wr, 547 op/s 2024-04-23T10:41:00.577 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 23 10:41:00 smithi204 ceph-mon[32649]: from='mgr.15291 172.21.15.114:0/1095789187' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-23T10:41:01.498 INFO:tasks.workunit.client.0.smithi114.stdout: 10 33524 29.63 MB/sec warmup 67 sec latency 1024.108 ms 2024-04-23T10:41:02.492 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 23 10:41:02 smithi119 ceph-mon[32608]: pgmap v454: 97 pgs: 97 active+clean; 550 MiB data, 4.3 GiB used, 1.0 TiB / 1.0 TiB avail; 10 MiB/s rd, 6.8 MiB/s wr, 547 op/s 2024-04-23T10:41:02.498 INFO:tasks.workunit.client.0.smithi114.stdout: 10 33524 29.19 MB/sec warmup 68 sec latency 2024.214 ms 2024-04-23T10:41:02.504 INFO:journalctl@ceph.mon.a.smithi114.stdout:Apr 23 10:41:02 smithi114 ceph-mon[29255]: pgmap v454: 97 pgs: 97 active+clean; 550 MiB data, 4.3 GiB used, 1.0 TiB / 1.0 TiB avail; 10 MiB/s rd, 6.8 MiB/s wr, 547 op/s 2024-04-23T10:41:02.578 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 23 10:41:02 smithi204 ceph-mon[32649]: pgmap v454: 97 pgs: 97 active+clean; 550 MiB data, 4.3 GiB used, 1.0 TiB / 1.0 TiB avail; 10 MiB/s rd, 6.8 MiB/s wr, 547 op/s 2024-04-23T10:41:03.498 INFO:tasks.workunit.client.0.smithi114.stdout: 10 33524 28.77 MB/sec warmup 69 sec latency 3024.291 ms 2024-04-23T10:41:03.556 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:41:03.553+0000 7f13b3dcf640 1 -- 172.21.15.114:0/2934820283 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f13ac1021a0 msgr2=0x7f13ac102580 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T10:41:03.556 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:41:03.553+0000 7f13b3dcf640 1 --2- 172.21.15.114:0/2934820283 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f13ac1021a0 0x7f13ac102580 secure :-1 s=READY pgs=937 cs=0 l=1 rev1=1 crypto rx=0x7f13a8009900 tx=0x7f13a802a410 comp rx=0 tx=0).stop 2024-04-23T10:41:03.556 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:41:03.553+0000 7f13b3dcf640 1 -- 172.21.15.114:0/2934820283 shutdown_connections 2024-04-23T10:41:03.556 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:41:03.553+0000 7f13b3dcf640 1 --2- 172.21.15.114:0/2934820283 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f13ac107270 0x7f13ac10ee40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:41:03.556 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:41:03.553+0000 7f13b3dcf640 1 --2- 172.21.15.114:0/2934820283 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f13ac102b50 0x7f13ac106b90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:41:03.556 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:41:03.553+0000 7f13b3dcf640 1 --2- 172.21.15.114:0/2934820283 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f13ac1021a0 0x7f13ac102580 unknown :-1 s=CLOSED pgs=937 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:41:03.557 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:41:03.553+0000 7f13b3dcf640 1 -- 172.21.15.114:0/2934820283 >> 172.21.15.114:0/2934820283 conn(0x7f13ac0fda20 msgr2=0x7f13ac0ffe40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T10:41:03.557 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:41:03.553+0000 7f13b3dcf640 1 -- 172.21.15.114:0/2934820283 shutdown_connections 2024-04-23T10:41:03.557 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:41:03.553+0000 7f13b3dcf640 1 -- 172.21.15.114:0/2934820283 wait complete. 2024-04-23T10:41:03.557 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:41:03.554+0000 7f13b3dcf640 1 Processor -- start 2024-04-23T10:41:03.557 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:41:03.554+0000 7f13b3dcf640 1 -- start start 2024-04-23T10:41:03.557 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:41:03.554+0000 7f13b3dcf640 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f13ac1021a0 0x7f13ac198630 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T10:41:03.557 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:41:03.554+0000 7f13b3dcf640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f13ac107270 0x7f13ac198b70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T10:41:03.557 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:41:03.554+0000 7f13b3dcf640 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f13ac19cef0 0x7f13ac19d3a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T10:41:03.558 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:41:03.554+0000 7f13b3dcf640 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x7f13ac1121b0 con 0x7f13ac107270 2024-04-23T10:41:03.558 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:41:03.554+0000 7f13b3dcf640 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 -- 0x7f13ac112030 con 0x7f13ac1021a0 2024-04-23T10:41:03.558 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:41:03.554+0000 7f13b3dcf640 1 -- --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_getmap magic: 0 -- 0x7f13ac112330 con 0x7f13ac19cef0 2024-04-23T10:41:03.558 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:41:03.554+0000 7f13b136a640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f13ac107270 0x7f13ac198b70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T10:41:03.558 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:41:03.554+0000 7f13b136a640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f13ac107270 0x7f13ac198b70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.114:49490/0 (socket says 172.21.15.114:49490) 2024-04-23T10:41:03.558 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:41:03.554+0000 7f13b136a640 1 -- 172.21.15.114:0/400835908 learned_addr learned my addr 172.21.15.114:0/400835908 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-04-23T10:41:03.558 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:41:03.554+0000 7f13b1b6b640 1 --2- 172.21.15.114:0/400835908 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f13ac1021a0 0x7f13ac198630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T10:41:03.558 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:41:03.554+0000 7f13b236c640 1 --2- 172.21.15.114:0/400835908 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f13ac19cef0 0x7f13ac19d3a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T10:41:03.559 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:41:03.555+0000 7f13b136a640 1 -- 172.21.15.114:0/400835908 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f13ac19cef0 msgr2=0x7f13ac19d3a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T10:41:03.559 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:41:03.555+0000 7f13b136a640 1 --2- 172.21.15.114:0/400835908 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f13ac19cef0 0x7f13ac19d3a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:41:03.559 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:41:03.555+0000 7f13b136a640 1 -- 172.21.15.114:0/400835908 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f13ac1021a0 msgr2=0x7f13ac198630 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T10:41:03.559 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:41:03.555+0000 7f13b136a640 1 --2- 172.21.15.114:0/400835908 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f13ac1021a0 0x7f13ac198630 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:41:03.559 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:41:03.555+0000 7f13b136a640 1 -- 172.21.15.114:0/400835908 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f13ac19d9e0 con 0x7f13ac107270 2024-04-23T10:41:03.559 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:41:03.555+0000 7f13b136a640 1 --2- 172.21.15.114:0/400835908 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f13ac107270 0x7f13ac198b70 secure :-1 s=READY pgs=938 cs=0 l=1 rev1=1 crypto rx=0x7f139c00c970 tx=0x7f139c00ce30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T10:41:03.559 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:41:03.555+0000 7f13a2ffd640 1 -- 172.21.15.114:0/400835908 <== mon.0 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f139c014070 con 0x7f13ac107270 2024-04-23T10:41:03.559 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:41:03.555+0000 7f13b3dcf640 1 -- 172.21.15.114:0/400835908 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f13ac19dcd0 con 0x7f13ac107270 2024-04-23T10:41:03.560 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:41:03.555+0000 7f13a2ffd640 1 -- 172.21.15.114:0/400835908 <== mon.0 v2:172.21.15.114:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f139c004440 con 0x7f13ac107270 2024-04-23T10:41:03.560 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:41:03.555+0000 7f13a2ffd640 1 -- 172.21.15.114:0/400835908 <== mon.0 v2:172.21.15.114:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f139c003e50 con 0x7f13ac107270 2024-04-23T10:41:03.560 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:41:03.555+0000 7f13b3dcf640 1 -- 172.21.15.114:0/400835908 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f13ac1a5510 con 0x7f13ac107270 2024-04-23T10:41:03.570 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:41:03.558+0000 7f13a2ffd640 1 -- 172.21.15.114:0/400835908 <== mon.0 v2:172.21.15.114:3300/0 4 ==== mgrmap(e 26) ==== 138778+0+0 (secure 0 0 0) 0x7f139c017070 con 0x7f13ac107270 2024-04-23T10:41:03.571 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:41:03.558+0000 7f13b6bbd740 1 Processor -- start 2024-04-23T10:41:03.571 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:41:03.558+0000 7f13b6bbd740 1 -- start start 2024-04-23T10:41:03.571 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:41:03.558+0000 7f13b6bbd740 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55e77925f620 0x55e77925fa00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T10:41:03.571 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:41:03.558+0000 7f13b6bbd740 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x55e77925ff40 0x55e77926a2c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T10:41:03.571 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:41:03.558+0000 7f13b6bbd740 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x55e77926a800 0x55e77926cbf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T10:41:03.571 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:41:03.558+0000 7f13b6bbd740 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x55e7791c4970 con 0x55e77925f620 2024-04-23T10:41:03.571 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:41:03.558+0000 7f13b6bbd740 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 -- 0x55e7791614a0 con 0x55e77926a800 2024-04-23T10:41:03.571 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:41:03.558+0000 7f13b6bbd740 1 -- --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_getmap magic: 0 -- 0x55e7791c0930 con 0x55e77925ff40 2024-04-23T10:41:03.572 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:41:03.559+0000 7f13b236c640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55e77925f620 0x55e77925fa00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T10:41:03.572 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:41:03.559+0000 7f13b136a640 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x55e77926a800 0x55e77926cbf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T10:41:03.572 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:41:03.559+0000 7f13b236c640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55e77925f620 0x55e77925fa00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.114:49492/0 (socket says 172.21.15.114:49492) 2024-04-23T10:41:03.572 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:41:03.559+0000 7f13b236c640 1 -- 172.21.15.114:0/1070664594 learned_addr learned my addr 172.21.15.114:0/1070664594 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-04-23T10:41:03.572 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:41:03.559+0000 7f13b1b6b640 1 --2- 172.21.15.114:0/1070664594 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x55e77925ff40 0x55e77926a2c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T10:41:03.572 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:41:03.559+0000 7f13a2ffd640 1 --2- 172.21.15.114:0/400835908 >> [v2:172.21.15.114:6800/1395693466,v1:172.21.15.114:6801/1395693466] conn(0x7f13980a3f60 0x7f13980a6420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T10:41:03.572 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:41:03.559+0000 7f13b236c640 1 -- 172.21.15.114:0/1070664594 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x55e77925ff40 msgr2=0x55e77926a2c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T10:41:03.572 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:41:03.559+0000 7f13b236c640 1 --2- 172.21.15.114:0/1070664594 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x55e77925ff40 0x55e77926a2c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:41:03.573 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:41:03.559+0000 7f13b236c640 1 -- 172.21.15.114:0/1070664594 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x55e77926a800 msgr2=0x55e77926cbf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T10:41:03.573 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:41:03.559+0000 7f13b236c640 1 --2- 172.21.15.114:0/1070664594 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x55e77926a800 0x55e77926cbf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:41:03.573 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:41:03.559+0000 7f13b236c640 1 -- 172.21.15.114:0/1070664594 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55e778ed7210 con 0x55e77925f620 2024-04-23T10:41:03.573 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:41:03.559+0000 7f13b136a640 1 --2- 172.21.15.114:0/1070664594 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x55e77926a800 0x55e77926cbf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-23T10:41:03.573 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:41:03.559+0000 7f13b236c640 1 --2- 172.21.15.114:0/400835908 >> [v2:172.21.15.114:6800/1395693466,v1:172.21.15.114:6801/1395693466] conn(0x7f13980a3f60 0x7f13980a6420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T10:41:03.573 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:41:03.559+0000 7f13a2ffd640 1 -- 172.21.15.114:0/400835908 <== mon.0 v2:172.21.15.114:3300/0 5 ==== osd_map(111..111 src has 1..111) ==== 8585+0+0 (secure 0 0 0) 0x7f139c0d0610 con 0x7f13ac107270 2024-04-23T10:41:03.573 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:41:03.559+0000 7f13b236c640 1 --2- 172.21.15.114:0/1070664594 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55e77925f620 0x55e77925fa00 secure :-1 s=READY pgs=939 cs=0 l=1 rev1=1 crypto rx=0x7f13a4011590 tx=0x7f13a4011a50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1757fe1ac6779e67 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T10:41:03.573 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:41:03.559+0000 7f1397fff640 1 -- 172.21.15.114:0/1070664594 <== mon.0 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f13a4017050 con 0x55e77925f620 2024-04-23T10:41:03.574 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:41:03.559+0000 7f1397fff640 1 -- 172.21.15.114:0/1070664594 <== mon.0 v2:172.21.15.114:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f13a401bb90 con 0x55e77925f620 2024-04-23T10:41:03.574 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:41:03.560+0000 7f1397fff640 1 -- 172.21.15.114:0/1070664594 <== mon.0 v2:172.21.15.114:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f13a4025e30 con 0x55e77925f620 2024-04-23T10:41:03.574 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:41:03.560+0000 7f13b6bbd740 1 -- 172.21.15.114:0/1070664594 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55e77925f620 msgr2=0x55e77925fa00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T10:41:03.574 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:41:03.560+0000 7f13b236c640 1 --2- 172.21.15.114:0/400835908 >> [v2:172.21.15.114:6800/1395693466,v1:172.21.15.114:6801/1395693466] conn(0x7f13980a3f60 0x7f13980a6420 secure :-1 s=READY pgs=150 cs=0 l=1 rev1=1 crypto rx=0x7f13a4022000 tx=0x7f13a4023000 comp rx=0 tx=0).ready entity=mgr.15291 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T10:41:03.574 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:41:03.560+0000 7f13b6bbd740 1 --2- 172.21.15.114:0/1070664594 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55e77925f620 0x55e77925fa00 secure :-1 s=READY pgs=939 cs=0 l=1 rev1=1 crypto rx=0x7f13a4011590 tx=0x7f13a4011a50 comp rx=0 tx=0).stop 2024-04-23T10:41:03.574 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:41:03.560+0000 7f13b6bbd740 1 -- 172.21.15.114:0/1070664594 shutdown_connections 2024-04-23T10:41:03.574 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:41:03.560+0000 7f13b6bbd740 1 --2- 172.21.15.114:0/1070664594 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x55e77926a800 0x55e77926cbf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:41:03.574 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:41:03.560+0000 7f13b6bbd740 1 --2- 172.21.15.114:0/1070664594 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x55e77925ff40 0x55e77926a2c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:41:03.574 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:41:03.560+0000 7f13b6bbd740 1 --2- 172.21.15.114:0/1070664594 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55e77925f620 0x55e77925fa00 unknown :-1 s=CLOSED pgs=939 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:41:03.575 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:41:03.560+0000 7f13b6bbd740 1 -- 172.21.15.114:0/1070664594 >> 172.21.15.114:0/1070664594 conn(0x55e77925bb30 msgr2=0x55e77925bf10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T10:41:03.575 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:41:03.560+0000 7f13b6bbd740 1 -- 172.21.15.114:0/1070664594 shutdown_connections 2024-04-23T10:41:03.575 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:41:03.560+0000 7f13b6bbd740 1 -- 172.21.15.114:0/1070664594 wait complete. 2024-04-23T10:41:03.575 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:41:03.561+0000 7f13b6bbd740 1 -- start start 2024-04-23T10:41:03.575 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:41:03.561+0000 7f13b6bbd740 1 Processor -- start 2024-04-23T10:41:03.575 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:41:03.561+0000 7f13b6bbd740 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x55e77925f620 0x55e77926a170 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T10:41:03.575 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:41:03.561+0000 7f13b6bbd740 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x55e77925ff40 0x55e779267460 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T10:41:03.575 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:41:03.561+0000 7f13b6bbd740 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55e77926a800 0x55e7792679c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T10:41:03.575 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:41:03.561+0000 7f13b6bbd740 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x55e77926d430 con 0x55e77926a800 2024-04-23T10:41:03.576 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:41:03.561+0000 7f13b6bbd740 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 -- 0x55e77915f0b0 con 0x55e77925ff40 2024-04-23T10:41:03.576 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:41:03.561+0000 7f13b6bbd740 1 -- --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_getmap magic: 0 -- 0x55e77926d5b0 con 0x55e77925f620 2024-04-23T10:41:03.576 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:41:03.561+0000 7f13b6bbd740 10 client.0 _ll_register_callbacks cb 0x55e77914cd30 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-23T10:41:03.576 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:41:03.561+0000 7f13b236c640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55e77926a800 0x55e7792679c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T10:41:03.576 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:41:03.561+0000 7f13b236c640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55e77926a800 0x55e7792679c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.114:49500/0 (socket says 172.21.15.114:49500) 2024-04-23T10:41:03.576 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:41:03.561+0000 7f13b236c640 1 -- 172.21.15.114:0/3724075584 learned_addr learned my addr 172.21.15.114:0/3724075584 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-04-23T10:41:03.576 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:41:03.561+0000 7f13b1b6b640 1 --2- 172.21.15.114:0/3724075584 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x55e77925f620 0x55e77926a170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T10:41:03.576 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:41:03.562+0000 7f13b136a640 1 --2- 172.21.15.114:0/3724075584 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x55e77925ff40 0x55e779267460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T10:41:03.576 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:41:03.562+0000 7f13b236c640 1 -- 172.21.15.114:0/3724075584 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x55e77925f620 msgr2=0x55e77926a170 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T10:41:03.577 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:41:03.562+0000 7f13b236c640 1 --2- 172.21.15.114:0/3724075584 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x55e77925f620 0x55e77926a170 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:41:03.577 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:41:03.562+0000 7f13b236c640 1 -- 172.21.15.114:0/3724075584 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x55e77925ff40 msgr2=0x55e779267460 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T10:41:03.577 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:41:03.562+0000 7f13b236c640 1 --2- 172.21.15.114:0/3724075584 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x55e77925ff40 0x55e779267460 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:41:03.577 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:41:03.562+0000 7f13b236c640 1 -- 172.21.15.114:0/3724075584 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55e778fa3620 con 0x55e77926a800 2024-04-23T10:41:03.577 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:41:03.562+0000 7f13b136a640 1 --2- 172.21.15.114:0/3724075584 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x55e77925ff40 0x55e779267460 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-23T10:41:03.577 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:41:03.562+0000 7f13b236c640 1 --2- 172.21.15.114:0/3724075584 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55e77926a800 0x55e7792679c0 secure :-1 s=READY pgs=940 cs=0 l=1 rev1=1 crypto rx=0x7f13a4025fa0 tx=0x7f13a4009500 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9caf7abd968b2130 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T10:41:03.577 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:41:03.562+0000 7f13957fa640 10 client.0 ms_handle_connect on v2:172.21.15.114:3300/0 2024-04-23T10:41:03.577 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:41:03.562+0000 7f13957fa640 1 -- 172.21.15.114:0/3724075584 <== mon.0 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f13a402a030 con 0x55e77926a800 2024-04-23T10:41:03.577 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:41:03.562+0000 7f13b6bbd740 1 -- 172.21.15.114:0/3724075584 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f13a401bb90 con 0x55e77926a800 2024-04-23T10:41:03.578 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:41:03.562+0000 7f13957fa640 1 -- 172.21.15.114:0/3724075584 <== mon.0 v2:172.21.15.114:3300/0 2 ==== osd_map(111..111 src has 1..111) ==== 8585+0+0 (secure 0 0 0) 0x7f13a4025640 con 0x55e77926a800 2024-04-23T10:41:03.578 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:41:03.562+0000 7f13957fa640 1 -- 172.21.15.114:0/3724075584 <== mon.0 v2:172.21.15.114:3300/0 3 ==== mon_get_version_reply(handle=1 version=49) ==== 24+0+0 (secure 0 0 0) 0x7f13a4017020 con 0x55e77926a800 2024-04-23T10:41:03.578 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:41:03.562+0000 7f13b6bbd740 10 client.17127 fetch_fsmap learned FSMap version 49 2024-04-23T10:41:03.578 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:41:03.562+0000 7f13b6bbd740 1 -- 172.21.15.114:0/3724075584 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({config=0+,fsmap=49,monmap=4+}) -- 0x55e779279360 con 0x55e77926a800 2024-04-23T10:41:03.578 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:41:03.562+0000 7f13957fa640 1 -- 172.21.15.114:0/3724075584 <== mon.0 v2:172.21.15.114:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f13a401bb90 con 0x55e77926a800 2024-04-23T10:41:03.578 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:41:03.563+0000 7f13957fa640 1 -- 172.21.15.114:0/3724075584 <== mon.0 v2:172.21.15.114:3300/0 5 ==== fsmap(e 49) ==== 8358+0+0 (secure 0 0 0) 0x7f13a4037030 con 0x55e77926a800 2024-04-23T10:41:03.579 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:41:03.563+0000 7f13b6bbd740 10 client.17127 fetch_fsmap finished waiting for FSMap version 49 2024-04-23T10:41:03.579 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:41:03.563+0000 7f13b6bbd740 10 client.17127 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-04-23T10:41:03.579 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:41:03.563+0000 7f13b6bbd740 20 client.17127 populate_metadata read hostname 'smithi114' 2024-04-23T10:41:03.579 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:41:03.563+0000 7f13b6bbd740 1 --2- 172.21.15.114:0/3724075584 >> [v2:172.21.15.119:6838/937297340,v1:172.21.15.119:6839/937297340] conn(0x55e77927a820 0x55e77927cc10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T10:41:03.579 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:41:03.563+0000 7f13b6bbd740 4 client.17127 mds_command: new command op to 24545 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-23T10:41:03.579 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:41:03.563+0000 7f13b6bbd740 1 -- 172.21.15.114:0/3724075584 --> [v2:172.21.15.119:6838/937297340,v1:172.21.15.119:6839/937297340] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55e7791d2390 con 0x55e77927a820 2024-04-23T10:41:03.579 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:41:03.563+0000 7f13b1b6b640 1 --2- 172.21.15.114:0/3724075584 >> [v2:172.21.15.119:6838/937297340,v1:172.21.15.119:6839/937297340] conn(0x55e77927a820 0x55e77927cc10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T10:41:03.579 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:41:03.563+0000 7f13957fa640 10 client.17127 ms_handle_connect on v2:172.21.15.119:6838/937297340 2024-04-23T10:41:03.580 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:41:03.563+0000 7f13b1b6b640 1 --2- 172.21.15.114:0/3724075584 >> [v2:172.21.15.119:6838/937297340,v1:172.21.15.119:6839/937297340] conn(0x55e77927a820 0x55e77927cc10 crc :-1 s=READY pgs=190 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=53218756aff3be5f server_cookie=ccdc7f500ce7a9b in_seq=0 out_seq=0 2024-04-23T10:41:03.580 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:41:03.564+0000 7f13957fa640 1 -- 172.21.15.114:0/3724075584 <== mds.0 v2:172.21.15.119:6838/937297340 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x55e7791d2390 con 0x55e77927a820 2024-04-23T10:41:03.580 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:41:03.564+0000 7f13957fa640 10 client.17127 handle_command_reply: tid=0 2024-04-23T10:41:03.580 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:41:03.564+0000 7f13b6bbd740 1 client.17127 shutdown 2024-04-23T10:41:03.580 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:41:03.564+0000 7f13b6bbd740 1 -- 172.21.15.114:0/3724075584 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55e77926a800 msgr2=0x55e7792679c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T10:41:03.581 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:41:03.564+0000 7f13b6bbd740 1 --2- 172.21.15.114:0/3724075584 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55e77926a800 0x55e7792679c0 secure :-1 s=READY pgs=940 cs=0 l=1 rev1=1 crypto rx=0x7f13a4025fa0 tx=0x7f13a4009500 comp rx=0 tx=0).stop 2024-04-23T10:41:03.581 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:41:03.564+0000 7f13b6bbd740 1 -- 172.21.15.114:0/3724075584 shutdown_connections 2024-04-23T10:41:03.581 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:41:03.564+0000 7f13b6bbd740 1 --2- 172.21.15.114:0/3724075584 >> [v2:172.21.15.119:6838/937297340,v1:172.21.15.119:6839/937297340] conn(0x55e77927a820 0x55e77927cc10 crc :-1 s=READY pgs=190 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:41:03.581 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:41:03.564+0000 7f13b6bbd740 1 --2- 172.21.15.114:0/3724075584 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55e77926a800 0x55e7792679c0 unknown :-1 s=CLOSED pgs=940 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:41:03.581 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:41:03.564+0000 7f13b6bbd740 1 --2- 172.21.15.114:0/3724075584 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x55e77925ff40 0x55e779267460 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:41:03.581 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:41:03.564+0000 7f13957fa640 0 client.17127 ms_handle_reset on v2:172.21.15.119:6838/937297340 2024-04-23T10:41:03.581 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:41:03.564+0000 7f13b6bbd740 1 --2- 172.21.15.114:0/3724075584 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x55e77925f620 0x55e77926a170 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:41:03.581 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:41:03.564+0000 7f13b6bbd740 1 -- 172.21.15.114:0/3724075584 >> 172.21.15.114:0/3724075584 conn(0x55e77925bb30 msgr2=0x55e779261860 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T10:41:03.581 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:41:03.564+0000 7f13b6bbd740 1 -- 172.21.15.114:0/3724075584 shutdown_connections 2024-04-23T10:41:03.582 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:41:03.564+0000 7f13b6bbd740 1 -- 172.21.15.114:0/3724075584 wait complete. 2024-04-23T10:41:03.582 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:41:03.564+0000 7f13b6bbd740 20 client.17127 trim_cache size 0 max 16384 2024-04-23T10:41:03.589 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:41:03.579+0000 7f13b6bbd740 1 Processor -- start 2024-04-23T10:41:03.589 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:41:03.579+0000 7f13b6bbd740 1 -- start start 2024-04-23T10:41:03.589 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:41:03.579+0000 7f13b6bbd740 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x55e77925f620 0x55e7792762f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T10:41:03.589 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:41:03.579+0000 7f13b6bbd740 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55e77925ff40 0x55e77925ab30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T10:41:03.589 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:41:03.579+0000 7f13b6bbd740 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x55e77926a800 0x55e779269ca0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T10:41:03.589 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:41:03.579+0000 7f13b6bbd740 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x7f13ac1b0580 con 0x55e77925ff40 2024-04-23T10:41:03.589 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:41:03.579+0000 7f13b6bbd740 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 -- 0x7f13ac1addc0 con 0x55e77925f620 2024-04-23T10:41:03.589 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:41:03.579+0000 7f13b6bbd740 1 -- --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_getmap magic: 0 -- 0x7f13ac1abc10 con 0x55e77926a800 2024-04-23T10:41:03.590 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:41:03.579+0000 7f13b136a640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55e77925ff40 0x55e77925ab30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T10:41:03.590 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:41:03.579+0000 7f13b136a640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55e77925ff40 0x55e77925ab30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.114:49516/0 (socket says 172.21.15.114:49516) 2024-04-23T10:41:03.590 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:41:03.579+0000 7f13b136a640 1 -- 172.21.15.114:0/510541213 learned_addr learned my addr 172.21.15.114:0/510541213 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-04-23T10:41:03.590 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:41:03.579+0000 7f13b1b6b640 1 --2- 172.21.15.114:0/510541213 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x55e77925f620 0x55e7792762f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T10:41:03.590 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:41:03.580+0000 7f13b136a640 1 -- 172.21.15.114:0/510541213 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x55e77926a800 msgr2=0x55e779269ca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T10:41:03.590 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:41:03.580+0000 7f13b236c640 1 --2- 172.21.15.114:0/510541213 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x55e77926a800 0x55e779269ca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T10:41:03.590 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:41:03.580+0000 7f13b136a640 1 --2- 172.21.15.114:0/510541213 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x55e77926a800 0x55e779269ca0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:41:03.590 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:41:03.580+0000 7f13b136a640 1 -- 172.21.15.114:0/510541213 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x55e77925f620 msgr2=0x55e7792762f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T10:41:03.590 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:41:03.580+0000 7f13b136a640 1 --2- 172.21.15.114:0/510541213 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x55e77925f620 0x55e7792762f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:41:03.591 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:41:03.580+0000 7f13b136a640 1 -- 172.21.15.114:0/510541213 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55e779267d30 con 0x55e77925ff40 2024-04-23T10:41:03.591 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:41:03.580+0000 7f13b1b6b640 1 --2- 172.21.15.114:0/510541213 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x55e77925f620 0x55e7792762f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-23T10:41:03.591 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:41:03.580+0000 7f13b136a640 1 --2- 172.21.15.114:0/510541213 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55e77925ff40 0x55e77925ab30 secure :-1 s=READY pgs=941 cs=0 l=1 rev1=1 crypto rx=0x7f139c09c500 tx=0x7f139c007ec0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=641315e631bab561 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T10:41:03.591 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:41:03.580+0000 7f13977fe640 1 -- 172.21.15.114:0/510541213 <== mon.0 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f139c09c570 con 0x55e77925ff40 2024-04-23T10:41:03.591 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:41:03.580+0000 7f13977fe640 1 -- 172.21.15.114:0/510541213 <== mon.0 v2:172.21.15.114:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f139c0a0920 con 0x55e77925ff40 2024-04-23T10:41:03.591 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:41:03.580+0000 7f13977fe640 1 -- 172.21.15.114:0/510541213 <== mon.0 v2:172.21.15.114:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f139c098be0 con 0x55e77925ff40 2024-04-23T10:41:03.591 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:41:03.581+0000 7f13b6bbd740 1 -- 172.21.15.114:0/510541213 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55e77925ff40 msgr2=0x55e77925ab30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T10:41:03.591 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:41:03.581+0000 7f13b6bbd740 1 --2- 172.21.15.114:0/510541213 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55e77925ff40 0x55e77925ab30 secure :-1 s=READY pgs=941 cs=0 l=1 rev1=1 crypto rx=0x7f139c09c500 tx=0x7f139c007ec0 comp rx=0 tx=0).stop 2024-04-23T10:41:03.592 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:41:03.581+0000 7f13b6bbd740 1 -- 172.21.15.114:0/510541213 shutdown_connections 2024-04-23T10:41:03.592 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:41:03.581+0000 7f13b6bbd740 1 --2- 172.21.15.114:0/510541213 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x55e77926a800 0x55e779269ca0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:41:03.592 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:41:03.581+0000 7f13b6bbd740 1 --2- 172.21.15.114:0/510541213 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55e77925ff40 0x55e77925ab30 unknown :-1 s=CLOSED pgs=941 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:41:03.592 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:41:03.581+0000 7f13b6bbd740 1 --2- 172.21.15.114:0/510541213 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x55e77925f620 0x55e7792762f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:41:03.592 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:41:03.581+0000 7f13b6bbd740 1 -- 172.21.15.114:0/510541213 >> 172.21.15.114:0/510541213 conn(0x55e77925bb30 msgr2=0x55e77925a420 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T10:41:03.592 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:41:03.581+0000 7f13b6bbd740 1 -- 172.21.15.114:0/510541213 shutdown_connections 2024-04-23T10:41:03.592 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:41:03.581+0000 7f13b6bbd740 1 -- 172.21.15.114:0/510541213 wait complete. 2024-04-23T10:41:03.592 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:41:03.581+0000 7f13b6bbd740 1 -- start start 2024-04-23T10:41:03.592 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:41:03.581+0000 7f13b6bbd740 1 Processor -- start 2024-04-23T10:41:03.592 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:41:03.581+0000 7f13b6bbd740 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55e77925f620 0x55e7792729f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T10:41:03.593 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:41:03.581+0000 7f13b6bbd740 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x55e77925ff40 0x55e779263d30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T10:41:03.593 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:41:03.581+0000 7f13b6bbd740 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x55e77926a800 0x55e779264270 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T10:41:03.593 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:41:03.581+0000 7f13b6bbd740 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x7f13ac1ac280 con 0x55e77925f620 2024-04-23T10:41:03.593 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:41:03.581+0000 7f13b6bbd740 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 -- 0x7f13ac1ad6f0 con 0x55e77926a800 2024-04-23T10:41:03.593 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:41:03.581+0000 7f13b6bbd740 1 -- --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_getmap magic: 0 -- 0x7f13ac1ac950 con 0x55e77925ff40 2024-04-23T10:41:03.593 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:41:03.582+0000 7f13b6bbd740 10 client.0 _ll_register_callbacks cb 0x7f13a4011440 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-23T10:41:03.593 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:41:03.582+0000 7f13b1b6b640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55e77925f620 0x55e7792729f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T10:41:03.593 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:41:03.582+0000 7f13b1b6b640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55e77925f620 0x55e7792729f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.114:49530/0 (socket says 172.21.15.114:49530) 2024-04-23T10:41:03.593 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:41:03.582+0000 7f13b1b6b640 1 -- 172.21.15.114:0/2500172572 learned_addr learned my addr 172.21.15.114:0/2500172572 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-04-23T10:41:03.593 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:41:03.582+0000 7f13b236c640 1 --2- 172.21.15.114:0/2500172572 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x55e77926a800 0x55e779264270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T10:41:03.594 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:41:03.582+0000 7f13b136a640 1 --2- 172.21.15.114:0/2500172572 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x55e77925ff40 0x55e779263d30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T10:41:03.594 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:41:03.582+0000 7f13b1b6b640 1 -- 172.21.15.114:0/2500172572 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x55e77925ff40 msgr2=0x55e779263d30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T10:41:03.594 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:41:03.582+0000 7f13b1b6b640 1 --2- 172.21.15.114:0/2500172572 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x55e77925ff40 0x55e779263d30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:41:03.594 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:41:03.582+0000 7f13b1b6b640 1 -- 172.21.15.114:0/2500172572 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x55e77926a800 msgr2=0x55e779264270 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T10:41:03.594 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:41:03.582+0000 7f13b1b6b640 1 --2- 172.21.15.114:0/2500172572 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x55e77926a800 0x55e779264270 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:41:03.594 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:41:03.582+0000 7f13b1b6b640 1 -- 172.21.15.114:0/2500172572 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55e779264830 con 0x55e77925f620 2024-04-23T10:41:03.594 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:41:03.582+0000 7f13b1b6b640 1 --2- 172.21.15.114:0/2500172572 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55e77925f620 0x55e7792729f0 secure :-1 s=READY pgs=942 cs=0 l=1 rev1=1 crypto rx=0x7f13a8033000 tx=0x7f13a802abc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d619a2eb81cb06ad server_cookie=0 in_seq=0 out_seq=0 2024-04-23T10:41:03.594 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:41:03.582+0000 7f13957fa640 10 client.0 ms_handle_connect on v2:172.21.15.114:3300/0 2024-04-23T10:41:03.594 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:41:03.582+0000 7f13957fa640 1 -- 172.21.15.114:0/2500172572 <== mon.0 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f13a804b060 con 0x55e77925f620 2024-04-23T10:41:03.595 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:41:03.582+0000 7f13957fa640 1 -- 172.21.15.114:0/2500172572 <== mon.0 v2:172.21.15.114:3300/0 2 ==== osd_map(111..111 src has 1..111) ==== 8585+0+0 (secure 0 0 0) 0x7f13a80043e0 con 0x55e77925f620 2024-04-23T10:41:03.595 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:41:03.582+0000 7f13b6bbd740 1 -- 172.21.15.114:0/2500172572 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f139c0a0920 con 0x55e77925f620 2024-04-23T10:41:03.595 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:41:03.582+0000 7f13957fa640 1 -- 172.21.15.114:0/2500172572 <== mon.0 v2:172.21.15.114:3300/0 3 ==== mon_get_version_reply(handle=1 version=49) ==== 24+0+0 (secure 0 0 0) 0x7f13a8035040 con 0x55e77925f620 2024-04-23T10:41:03.595 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:41:03.582+0000 7f13b6bbd740 10 client.17133 fetch_fsmap learned FSMap version 49 2024-04-23T10:41:03.595 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:41:03.582+0000 7f13b6bbd740 1 -- 172.21.15.114:0/2500172572 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({config=0+,fsmap=49,monmap=4+}) -- 0x55e7792e0970 con 0x55e77925f620 2024-04-23T10:41:03.595 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:41:03.583+0000 7f13957fa640 1 -- 172.21.15.114:0/2500172572 <== mon.0 v2:172.21.15.114:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f139c0a0920 con 0x55e77925f620 2024-04-23T10:41:03.595 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:41:03.583+0000 7f13957fa640 1 -- 172.21.15.114:0/2500172572 <== mon.0 v2:172.21.15.114:3300/0 5 ==== fsmap(e 49) ==== 8358+0+0 (secure 0 0 0) 0x7f13a8004750 con 0x55e77925f620 2024-04-23T10:41:03.595 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:41:03.583+0000 7f13b6bbd740 10 client.17133 fetch_fsmap finished waiting for FSMap version 49 2024-04-23T10:41:03.595 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:41:03.583+0000 7f13b6bbd740 10 client.17133 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-04-23T10:41:03.595 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:41:03.583+0000 7f13b6bbd740 20 client.17133 populate_metadata read hostname 'smithi114' 2024-04-23T10:41:03.596 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:41:03.583+0000 7f13b6bbd740 1 --2- 172.21.15.114:0/2500172572 >> [v2:172.21.15.119:6838/937297340,v1:172.21.15.119:6839/937297340] conn(0x55e7792e10d0 0x55e7792e34f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T10:41:03.596 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:41:03.583+0000 7f13b6bbd740 4 client.17133 mds_command: new command op to 24545 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-04-23T10:41:03.596 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:41:03.583+0000 7f13b6bbd740 1 -- 172.21.15.114:0/2500172572 --> [v2:172.21.15.119:6838/937297340,v1:172.21.15.119:6839/937297340] -- command(tid 0: {"prefix": "flush journal"}) -- 0x55e7791cc200 con 0x55e7792e10d0 2024-04-23T10:41:03.596 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:41:03.583+0000 7f13b136a640 1 --2- 172.21.15.114:0/2500172572 >> [v2:172.21.15.119:6838/937297340,v1:172.21.15.119:6839/937297340] conn(0x55e7792e10d0 0x55e7792e34f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T10:41:03.596 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:41:03.583+0000 7f13b136a640 1 --2- 172.21.15.114:0/2500172572 >> [v2:172.21.15.119:6838/937297340,v1:172.21.15.119:6839/937297340] conn(0x55e7792e10d0 0x55e7792e34f0 crc :-1 s=READY pgs=191 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=b6eda35b7b4bd3c7 server_cookie=e6f2a6886d5fbbfd in_seq=0 out_seq=0 2024-04-23T10:41:03.596 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:41:03.583+0000 7f13957fa640 10 client.17133 ms_handle_connect on v2:172.21.15.119:6838/937297340 2024-04-23T10:41:03.737 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:41:03.737+0000 7f13957fa640 1 -- 172.21.15.114:0/2500172572 <== mds.0 v2:172.21.15.119:6838/937297340 1 ==== command_reply(tid 0: 0 ) ==== 8+0+44 (crc 0 0 0) 0x55e7791cc200 con 0x55e7792e10d0 2024-04-23T10:41:03.737 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:41:03.737+0000 7f13957fa640 10 client.17133 handle_command_reply: tid=0 2024-04-23T10:41:03.737 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:41:03.737+0000 7f13b6bbd740 1 client.17133 shutdown 2024-04-23T10:41:03.737 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:41:03.737+0000 7f13b6bbd740 1 -- 172.21.15.114:0/2500172572 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55e77925f620 msgr2=0x55e7792729f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T10:41:03.737 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:41:03.737+0000 7f13b6bbd740 1 --2- 172.21.15.114:0/2500172572 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55e77925f620 0x55e7792729f0 secure :-1 s=READY pgs=942 cs=0 l=1 rev1=1 crypto rx=0x7f13a8033000 tx=0x7f13a802abc0 comp rx=0 tx=0).stop 2024-04-23T10:41:03.738 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:41:03.737+0000 7f13b6bbd740 1 -- 172.21.15.114:0/2500172572 shutdown_connections 2024-04-23T10:41:03.738 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:41:03.737+0000 7f13b6bbd740 1 --2- 172.21.15.114:0/2500172572 >> [v2:172.21.15.119:6838/937297340,v1:172.21.15.119:6839/937297340] conn(0x55e7792e10d0 0x55e7792e34f0 crc :-1 s=READY pgs=191 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:41:03.738 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:41:03.737+0000 7f13b6bbd740 1 --2- 172.21.15.114:0/2500172572 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x55e77926a800 0x55e779264270 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:41:03.738 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:41:03.737+0000 7f13b6bbd740 1 --2- 172.21.15.114:0/2500172572 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x55e77925ff40 0x55e779263d30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:41:03.738 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:41:03.737+0000 7f13957fa640 0 client.17133 ms_handle_reset on v2:172.21.15.119:6838/937297340 2024-04-23T10:41:03.738 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:41:03.737+0000 7f13b6bbd740 1 --2- 172.21.15.114:0/2500172572 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55e77925f620 0x55e7792729f0 unknown :-1 s=CLOSED pgs=942 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:41:03.738 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:41:03.737+0000 7f13b6bbd740 1 -- 172.21.15.114:0/2500172572 >> 172.21.15.114:0/2500172572 conn(0x55e77925bb30 msgr2=0x55e7792629d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T10:41:03.739 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:41:03.737+0000 7f13b6bbd740 1 -- 172.21.15.114:0/2500172572 shutdown_connections 2024-04-23T10:41:03.739 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:41:03.737+0000 7f13b6bbd740 1 -- 172.21.15.114:0/2500172572 wait complete. 2024-04-23T10:41:03.739 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:41:03.737+0000 7f13b6bbd740 20 client.17133 trim_cache size 0 max 16384 2024-04-23T10:41:03.739 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stdout:{ 2024-04-23T10:41:03.739 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stdout: "message": "", 2024-04-23T10:41:03.739 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stdout: "return_code": 0 2024-04-23T10:41:03.740 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stdout:} 2024-04-23T10:41:03.741 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:41:03.738+0000 7f13b3dcf640 1 -- 172.21.15.114:0/400835908 >> [v2:172.21.15.114:6800/1395693466,v1:172.21.15.114:6801/1395693466] conn(0x7f13980a3f60 msgr2=0x7f13980a6420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T10:41:03.741 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:41:03.738+0000 7f13b3dcf640 1 --2- 172.21.15.114:0/400835908 >> [v2:172.21.15.114:6800/1395693466,v1:172.21.15.114:6801/1395693466] conn(0x7f13980a3f60 0x7f13980a6420 secure :-1 s=READY pgs=150 cs=0 l=1 rev1=1 crypto rx=0x7f13a4022000 tx=0x7f13a4023000 comp rx=0 tx=0).stop 2024-04-23T10:41:03.741 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:41:03.738+0000 7f13b3dcf640 1 -- 172.21.15.114:0/400835908 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f13ac107270 msgr2=0x7f13ac198b70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T10:41:03.741 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:41:03.738+0000 7f13b3dcf640 1 --2- 172.21.15.114:0/400835908 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f13ac107270 0x7f13ac198b70 secure :-1 s=READY pgs=938 cs=0 l=1 rev1=1 crypto rx=0x7f139c00c970 tx=0x7f139c00ce30 comp rx=0 tx=0).stop 2024-04-23T10:41:03.741 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:41:03.738+0000 7f13b3dcf640 1 -- 172.21.15.114:0/400835908 shutdown_connections 2024-04-23T10:41:03.741 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:41:03.738+0000 7f13b3dcf640 1 --2- 172.21.15.114:0/400835908 >> [v2:172.21.15.114:6800/1395693466,v1:172.21.15.114:6801/1395693466] conn(0x7f13980a3f60 0x7f13980a6420 unknown :-1 s=CLOSED pgs=150 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:41:03.742 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:41:03.738+0000 7f13b3dcf640 1 --2- 172.21.15.114:0/400835908 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f13ac19cef0 0x7f13ac19d3a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:41:03.742 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:41:03.738+0000 7f13b3dcf640 1 --2- 172.21.15.114:0/400835908 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f13ac107270 0x7f13ac198b70 unknown :-1 s=CLOSED pgs=938 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:41:03.742 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:41:03.738+0000 7f13b3dcf640 1 --2- 172.21.15.114:0/400835908 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f13ac1021a0 0x7f13ac198630 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:41:03.742 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:41:03.738+0000 7f13b3dcf640 1 -- 172.21.15.114:0/400835908 >> 172.21.15.114:0/400835908 conn(0x7f13ac0fda20 msgr2=0x7f13ac0ffd30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T10:41:03.742 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:41:03.738+0000 7f13b3dcf640 1 -- 172.21.15.114:0/400835908 shutdown_connections 2024-04-23T10:41:03.742 INFO:teuthology.task.background_exec.ubuntu@smithi114.front.sepia.ceph.com.smithi114.stderr:2024-04-23T10:41:03.738+0000 7f13b3dcf640 1 -- 172.21.15.114:0/400835908 wait complete. 2024-04-23T10:41:04.404 INFO:tasks.workunit.client.0.smithi114.stdout:[19010] write failed on handle 13946 (Resource temporarily unavailable) 2024-04-23T10:41:04.404 INFO:tasks.workunit.client.0.smithi114.stdout:Child failed with status 1 2024-04-23T10:41:04.404 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-23T10:41:04.426 INFO:tasks.workunit:Stopping ['suites/dbench.sh'] on client.0... 2024-04-23T10:41:04.426 DEBUG:teuthology.orchestra.run.smithi114:> sudo rm -rf -- /home/ubuntu/cephtest/workunits.list.client.0 /home/ubuntu/cephtest/clone.client.0 2024-04-23T10:41:04.490 INFO:journalctl@ceph.mon.a.smithi114.stdout:Apr 23 10:41:04 smithi114 ceph-mon[29255]: pgmap v455: 97 pgs: 97 active+clean; 560 MiB data, 4.3 GiB used, 1.0 TiB / 1.0 TiB avail; 15 MiB/s rd, 12 MiB/s wr, 821 op/s 2024-04-23T10:41:04.492 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 23 10:41:04 smithi119 ceph-mon[32608]: pgmap v455: 97 pgs: 97 active+clean; 560 MiB data, 4.3 GiB used, 1.0 TiB / 1.0 TiB avail; 15 MiB/s rd, 12 MiB/s wr, 821 op/s 2024-04-23T10:41:04.578 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 23 10:41:04 smithi204 ceph-mon[32649]: pgmap v455: 97 pgs: 97 active+clean; 560 MiB data, 4.3 GiB used, 1.0 TiB / 1.0 TiB avail; 15 MiB/s rd, 12 MiB/s wr, 821 op/s 2024-04-23T10:41:04.790 ERROR:teuthology.run_tasks:Saw exception from tasks. Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_9481b1d62f50e7d0a4f3dd83adf6945b08d5ff17/teuthology/run_tasks.py", line 105, in run_tasks manager = run_one_task(taskname, ctx=ctx, config=config) File "/home/teuthworker/src/git.ceph.com_teuthology_9481b1d62f50e7d0a4f3dd83adf6945b08d5ff17/teuthology/run_tasks.py", line 83, in run_one_task return task(**kwargs) File "/home/teuthworker/src/github.com_ceph_ceph_0a6c3ed699031b80a2b419e7e795368719871394/qa/tasks/workunit.py", line 145, in task _spawn_on_all_clients(ctx, refspec, all_tasks, config.get('env'), File "/home/teuthworker/src/github.com_ceph_ceph_0a6c3ed699031b80a2b419e7e795368719871394/qa/tasks/workunit.py", line 295, in _spawn_on_all_clients p.spawn(_run_tests, ctx, refspec, role, [unit], env, File "/home/teuthworker/src/git.ceph.com_teuthology_9481b1d62f50e7d0a4f3dd83adf6945b08d5ff17/teuthology/parallel.py", line 84, in __exit__ for result in self: File "/home/teuthworker/src/git.ceph.com_teuthology_9481b1d62f50e7d0a4f3dd83adf6945b08d5ff17/teuthology/parallel.py", line 98, in __next__ resurrect_traceback(result) File "/home/teuthworker/src/git.ceph.com_teuthology_9481b1d62f50e7d0a4f3dd83adf6945b08d5ff17/teuthology/parallel.py", line 30, in resurrect_traceback raise exc.exc_info[1] File "/home/teuthworker/src/git.ceph.com_teuthology_9481b1d62f50e7d0a4f3dd83adf6945b08d5ff17/teuthology/parallel.py", line 23, in capture_traceback return func(*args, **kwargs) File "/home/teuthworker/src/github.com_ceph_ceph_0a6c3ed699031b80a2b419e7e795368719871394/qa/tasks/workunit.py", line 434, in _run_tests remote.run( File "/home/teuthworker/src/git.ceph.com_teuthology_9481b1d62f50e7d0a4f3dd83adf6945b08d5ff17/teuthology/orchestra/remote.py", line 523, in run r = self._runner(client=self.ssh, name=self.shortname, **kwargs) File "/home/teuthworker/src/git.ceph.com_teuthology_9481b1d62f50e7d0a4f3dd83adf6945b08d5ff17/teuthology/orchestra/run.py", line 455, in run r.wait() File "/home/teuthworker/src/git.ceph.com_teuthology_9481b1d62f50e7d0a4f3dd83adf6945b08d5ff17/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_teuthology_9481b1d62f50e7d0a4f3dd83adf6945b08d5ff17/teuthology/orchestra/run.py", line 181, in _raise_for_status raise CommandFailedError( teuthology.exceptions.CommandFailedError: Command failed (workunit test suites/dbench.sh) on smithi114 with status 1: 'mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=0a6c3ed699031b80a2b419e7e795368719871394 TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 CEPH_MNT=/home/ubuntu/cephtest/mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clone.client.0/qa/workunits/suites/dbench.sh' 2024-04-23T10:41:05.042 ERROR:teuthology.util.sentry: Sentry event: https://sentry.ceph.com/organizations/ceph/?query=0ffdb51de7e0431ca064be5e59f5473a Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_9481b1d62f50e7d0a4f3dd83adf6945b08d5ff17/teuthology/run_tasks.py", line 105, in run_tasks manager = run_one_task(taskname, ctx=ctx, config=config) File "/home/teuthworker/src/git.ceph.com_teuthology_9481b1d62f50e7d0a4f3dd83adf6945b08d5ff17/teuthology/run_tasks.py", line 83, in run_one_task return task(**kwargs) File "/home/teuthworker/src/github.com_ceph_ceph_0a6c3ed699031b80a2b419e7e795368719871394/qa/tasks/workunit.py", line 145, in task _spawn_on_all_clients(ctx, refspec, all_tasks, config.get('env'), File "/home/teuthworker/src/github.com_ceph_ceph_0a6c3ed699031b80a2b419e7e795368719871394/qa/tasks/workunit.py", line 295, in _spawn_on_all_clients p.spawn(_run_tests, ctx, refspec, role, [unit], env, File "/home/teuthworker/src/git.ceph.com_teuthology_9481b1d62f50e7d0a4f3dd83adf6945b08d5ff17/teuthology/parallel.py", line 84, in __exit__ for result in self: File "/home/teuthworker/src/git.ceph.com_teuthology_9481b1d62f50e7d0a4f3dd83adf6945b08d5ff17/teuthology/parallel.py", line 98, in __next__ resurrect_traceback(result) File "/home/teuthworker/src/git.ceph.com_teuthology_9481b1d62f50e7d0a4f3dd83adf6945b08d5ff17/teuthology/parallel.py", line 30, in resurrect_traceback raise exc.exc_info[1] File "/home/teuthworker/src/git.ceph.com_teuthology_9481b1d62f50e7d0a4f3dd83adf6945b08d5ff17/teuthology/parallel.py", line 23, in capture_traceback return func(*args, **kwargs) File "/home/teuthworker/src/github.com_ceph_ceph_0a6c3ed699031b80a2b419e7e795368719871394/qa/tasks/workunit.py", line 434, in _run_tests remote.run( File "/home/teuthworker/src/git.ceph.com_teuthology_9481b1d62f50e7d0a4f3dd83adf6945b08d5ff17/teuthology/orchestra/remote.py", line 523, in run r = self._runner(client=self.ssh, name=self.shortname, **kwargs) File "/home/teuthworker/src/git.ceph.com_teuthology_9481b1d62f50e7d0a4f3dd83adf6945b08d5ff17/teuthology/orchestra/run.py", line 455, in run r.wait() File "/home/teuthworker/src/git.ceph.com_teuthology_9481b1d62f50e7d0a4f3dd83adf6945b08d5ff17/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_teuthology_9481b1d62f50e7d0a4f3dd83adf6945b08d5ff17/teuthology/orchestra/run.py", line 181, in _raise_for_status raise CommandFailedError( teuthology.exceptions.CommandFailedError: Command failed (workunit test suites/dbench.sh) on smithi114 with status 1: 'mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=0a6c3ed699031b80a2b419e7e795368719871394 TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 CEPH_MNT=/home/ubuntu/cephtest/mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clone.client.0/qa/workunits/suites/dbench.sh' 2024-04-23T10:41:05.045 DEBUG:teuthology.run_tasks:Unwinding manager quiescer 2024-04-23T10:41:05.056 INFO:tasks.quiescer:joining Quiescers 2024-04-23T10:41:05.056 WARNING:tasks.quiescer:The quiescer is requested to stop, running cancel all 2024-04-23T10:41:05.056 DEBUG:teuthology.orchestra.run.smithi114:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-23T10:41:05.175 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:41:05.174+0000 7fb9a1cfc640 1 -- 172.21.15.114:0/3476484611 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fb99c102190 msgr2=0x7fb99c102570 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T10:41:05.175 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:41:05.174+0000 7fb9a1cfc640 1 --2- 172.21.15.114:0/3476484611 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fb99c102190 0x7fb99c102570 secure :-1 s=READY pgs=943 cs=0 l=1 rev1=1 crypto rx=0x7fb98c009900 tx=0x7fb98c02a410 comp rx=0 tx=0).stop 2024-04-23T10:41:05.176 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:41:05.174+0000 7fb9a1cfc640 1 -- 172.21.15.114:0/3476484611 shutdown_connections 2024-04-23T10:41:05.176 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:41:05.174+0000 7fb9a1cfc640 1 --2- 172.21.15.114:0/3476484611 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fb99c107260 0x7fb99c10ee30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:41:05.176 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:41:05.174+0000 7fb9a1cfc640 1 --2- 172.21.15.114:0/3476484611 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fb99c102b40 0x7fb99c106b80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:41:05.176 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:41:05.174+0000 7fb9a1cfc640 1 --2- 172.21.15.114:0/3476484611 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fb99c102190 0x7fb99c102570 unknown :-1 s=CLOSED pgs=943 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:41:05.176 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:41:05.174+0000 7fb9a1cfc640 1 -- 172.21.15.114:0/3476484611 >> 172.21.15.114:0/3476484611 conn(0x7fb99c079310 msgr2=0x7fb99c079720 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T10:41:05.176 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:41:05.174+0000 7fb9a1cfc640 1 -- 172.21.15.114:0/3476484611 shutdown_connections 2024-04-23T10:41:05.176 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:41:05.174+0000 7fb9a1cfc640 1 -- 172.21.15.114:0/3476484611 wait complete. 2024-04-23T10:41:05.176 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:41:05.175+0000 7fb9a1cfc640 1 Processor -- start 2024-04-23T10:41:05.176 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:41:05.175+0000 7fb9a1cfc640 1 -- start start 2024-04-23T10:41:05.177 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:41:05.175+0000 7fb9a1cfc640 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fb99c102190 0x7fb99c19e400 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T10:41:05.177 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:41:05.175+0000 7fb9a1cfc640 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fb99c107260 0x7fb99c19e940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T10:41:05.177 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:41:05.175+0000 7fb9a1cfc640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fb99c1985b0 0x7fb99c198a60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T10:41:05.177 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:41:05.175+0000 7fb9a1cfc640 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x7fb99c1121d0 con 0x7fb99c1985b0 2024-04-23T10:41:05.177 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:41:05.175+0000 7fb9a1cfc640 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 -- 0x7fb99c112050 con 0x7fb99c107260 2024-04-23T10:41:05.177 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:41:05.175+0000 7fb9a1cfc640 1 -- --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_getmap magic: 0 -- 0x7fb99c112350 con 0x7fb99c102190 2024-04-23T10:41:05.177 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:41:05.175+0000 7fb99bfff640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fb99c1985b0 0x7fb99c198a60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T10:41:05.177 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:41:05.175+0000 7fb99affd640 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fb99c107260 0x7fb99c19e940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T10:41:05.178 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:41:05.175+0000 7fb99bfff640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fb99c1985b0 0x7fb99c198a60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.114:49544/0 (socket says 172.21.15.114:49544) 2024-04-23T10:41:05.178 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:41:05.175+0000 7fb99bfff640 1 -- 172.21.15.114:0/668925059 learned_addr learned my addr 172.21.15.114:0/668925059 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-04-23T10:41:05.178 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:41:05.175+0000 7fb99b7fe640 1 --2- 172.21.15.114:0/668925059 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fb99c102190 0x7fb99c19e400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T10:41:05.178 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:41:05.175+0000 7fb99affd640 1 -- 172.21.15.114:0/668925059 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fb99c102190 msgr2=0x7fb99c19e400 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T10:41:05.178 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:41:05.176+0000 7fb99affd640 1 --2- 172.21.15.114:0/668925059 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fb99c102190 0x7fb99c19e400 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:41:05.178 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:41:05.176+0000 7fb99affd640 1 -- 172.21.15.114:0/668925059 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fb99c1985b0 msgr2=0x7fb99c198a60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T10:41:05.178 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:41:05.176+0000 7fb99affd640 1 --2- 172.21.15.114:0/668925059 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fb99c1985b0 0x7fb99c198a60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:41:05.178 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:41:05.176+0000 7fb99affd640 1 -- 172.21.15.114:0/668925059 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb99c1990d0 con 0x7fb99c107260 2024-04-23T10:41:05.178 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:41:05.176+0000 7fb99affd640 1 --2- 172.21.15.114:0/668925059 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fb99c107260 0x7fb99c19e940 secure :-1 s=READY pgs=103 cs=0 l=1 rev1=1 crypto rx=0x7fb99000e9c0 tx=0x7fb99000ee80 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T10:41:05.179 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:41:05.176+0000 7fb998ff9640 1 -- 172.21.15.114:0/668925059 <== mon.2 v2:172.21.15.119:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb99000ccc0 con 0x7fb99c107260 2024-04-23T10:41:05.179 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:41:05.176+0000 7fb998ff9640 1 -- 172.21.15.114:0/668925059 <== mon.2 v2:172.21.15.119:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fb9900108e0 con 0x7fb99c107260 2024-04-23T10:41:05.179 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:41:05.176+0000 7fb9a1cfc640 1 -- 172.21.15.114:0/668925059 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fb99c1993c0 con 0x7fb99c107260 2024-04-23T10:41:05.179 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:41:05.176+0000 7fb998ff9640 1 -- 172.21.15.114:0/668925059 <== mon.2 v2:172.21.15.119:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb9900187c0 con 0x7fb99c107260 2024-04-23T10:41:05.179 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:41:05.176+0000 7fb9a1cfc640 1 -- 172.21.15.114:0/668925059 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fb99c1a5610 con 0x7fb99c107260 2024-04-23T10:41:05.180 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:41:05.177+0000 7fb9a1cfc640 1 -- 172.21.15.114:0/668925059 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fb99c0691c0 con 0x7fb99c107260 2024-04-23T10:41:05.180 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:41:05.178+0000 7fb998ff9640 1 -- 172.21.15.114:0/668925059 <== mon.2 v2:172.21.15.119:3300/0 4 ==== mgrmap(e 26) ==== 138778+0+0 (secure 0 0 0) 0x7fb990010430 con 0x7fb99c107260 2024-04-23T10:41:05.180 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:41:05.179+0000 7fb998ff9640 1 --2- 172.21.15.114:0/668925059 >> [v2:172.21.15.114:6800/1395693466,v1:172.21.15.114:6801/1395693466] conn(0x7fb9840a3da0 0x7fb9840a6260 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T10:41:05.180 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:41:05.179+0000 7fb99b7fe640 1 --2- 172.21.15.114:0/668925059 >> [v2:172.21.15.114:6800/1395693466,v1:172.21.15.114:6801/1395693466] conn(0x7fb9840a3da0 0x7fb9840a6260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T10:41:05.180 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:41:05.179+0000 7fb998ff9640 1 -- 172.21.15.114:0/668925059 <== mon.2 v2:172.21.15.119:3300/0 5 ==== osd_map(111..111 src has 1..111) ==== 8585+0+0 (secure 0 0 0) 0x7fb9900d1070 con 0x7fb99c107260 2024-04-23T10:41:05.180 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:41:05.179+0000 7fb99b7fe640 1 --2- 172.21.15.114:0/668925059 >> [v2:172.21.15.114:6800/1395693466,v1:172.21.15.114:6801/1395693466] conn(0x7fb9840a3da0 0x7fb9840a6260 secure :-1 s=READY pgs=151 cs=0 l=1 rev1=1 crypto rx=0x7fb98c031440 tx=0x7fb98c005b50 comp rx=0 tx=0).ready entity=mgr.15291 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T10:41:05.181 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:41:05.183+0000 7fb998ff9640 1 -- 172.21.15.114:0/668925059 <== mon.2 v2:172.21.15.119:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7fb99009d020 con 0x7fb99c107260 2024-04-23T10:41:05.402 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:41:05.402+0000 7fb9a1cfc640 1 -- 172.21.15.114:0/668925059 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fb99c199ae0 con 0x7fb99c107260 2024-04-23T10:41:05.402 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:41:05.403+0000 7fb998ff9640 1 -- 172.21.15.114:0/668925059 <== mon.2 v2:172.21.15.119:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 49 v49) ==== 94+0+11891 (secure 0 0 0) 0x7fb99c199ae0 con 0x7fb99c107260 2024-04-23T10:41:05.402 INFO:teuthology.orchestra.run.smithi114.stdout: 2024-04-23T10:41:05.403 INFO:teuthology.orchestra.run.smithi114.stdout:{"epoch":49,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14547,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.114:6835/663493580","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6834","nonce":663493580},{"type":"v1","addr":"172.21.15.114:6835","nonce":663493580}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14565,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.114:6837/1439252154","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6836","nonce":1439252154},{"type":"v1","addr":"172.21.15.114:6837","nonce":1439252154}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5},{"gid":14634,"name":"g","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.114:6839/3849796939","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6838","nonce":3849796939},{"type":"v1","addr":"172.21.15.114:6839","nonce":3849796939}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":11},{"gid":14646,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.114:6841/3544523891","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6840","nonce":3544523891},{"type":"v1","addr":"172.21.15.114:6841","nonce":3544523891}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":12},{"gid":24385,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.204:6833/3866561754","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6832","nonce":3866561754},{"type":"v1","addr":"172.21.15.204:6833","nonce":3866561754}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2},{"gid":24427,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.204:6835/2533793172","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6834","nonce":2533793172},{"type":"v1","addr":"172.21.15.204:6835","nonce":2533793172}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":7}],"filesystems":[{"mdsmap":{"epoch":49,"flags":306,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":true},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-04-23T10:24:43.383158+0000","modified":"2024-04-23T10:40:23.446880+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":3,"in":[0,1,2],"up":{"mds_0":24545,"mds_1":24509,"mds_2":24479},"failed":[],"damaged":[],"stopped":[],"info":{"gid_24439":{"gid":24439,"name":"c","rank":2,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.204:6837/513822179","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6836","nonce":513822179},{"type":"v1","addr":"172.21.15.204:6837","nonce":513822179}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24443":{"gid":24443,"name":"k","rank":1,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.119:6833/783244660","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6832","nonce":783244660},{"type":"v1","addr":"172.21.15.119:6833","nonce":783244660}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24479":{"gid":24479,"name":"e","rank":2,"incarnation":19,"state":"up:active","state_seq":9,"addr":"172.21.15.119:6835/2033916709","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6834","nonce":2033916709},{"type":"v1","addr":"172.21.15.119:6835","nonce":2033916709}]},"join_fscid":-1,"export_targets":[0,1],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24509":{"gid":24509,"name":"h","rank":1,"incarnation":17,"state":"up:active","state_seq":6,"addr":"172.21.15.119:6837/1872363128","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6836","nonce":1872363128},{"type":"v1","addr":"172.21.15.119:6837","nonce":1872363128}]},"join_fscid":-1,"export_targets":[0],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24545":{"gid":24545,"name":"b","rank":0,"incarnation":14,"state":"up:active","state_seq":3,"addr":"172.21.15.119:6839/937297340","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6838","nonce":937297340},{"type":"v1","addr":"172.21.15.119:6839","nonce":937297340}]},"join_fscid":-1,"export_targets":[1,2],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_26224":{"gid":26224,"name":"i","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.204:6839/3251736831","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6838","nonce":3251736831},{"type":"v1","addr":"172.21.15.204:6839","nonce":3251736831}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":24545,"qdb_cluster":[24545,24479,24509]},"id":1}]} 2024-04-23T10:41:05.403 INFO:teuthology.orchestra.run.smithi114.stderr:dumped fsmap epoch 49 2024-04-23T10:41:05.404 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:41:05.405+0000 7fb9a1cfc640 1 -- 172.21.15.114:0/668925059 >> [v2:172.21.15.114:6800/1395693466,v1:172.21.15.114:6801/1395693466] conn(0x7fb9840a3da0 msgr2=0x7fb9840a6260 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T10:41:05.404 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:41:05.405+0000 7fb9a1cfc640 1 --2- 172.21.15.114:0/668925059 >> [v2:172.21.15.114:6800/1395693466,v1:172.21.15.114:6801/1395693466] conn(0x7fb9840a3da0 0x7fb9840a6260 secure :-1 s=READY pgs=151 cs=0 l=1 rev1=1 crypto rx=0x7fb98c031440 tx=0x7fb98c005b50 comp rx=0 tx=0).stop 2024-04-23T10:41:05.404 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:41:05.405+0000 7fb9a1cfc640 1 -- 172.21.15.114:0/668925059 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fb99c107260 msgr2=0x7fb99c19e940 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T10:41:05.404 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:41:05.405+0000 7fb9a1cfc640 1 --2- 172.21.15.114:0/668925059 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fb99c107260 0x7fb99c19e940 secure :-1 s=READY pgs=103 cs=0 l=1 rev1=1 crypto rx=0x7fb99000e9c0 tx=0x7fb99000ee80 comp rx=0 tx=0).stop 2024-04-23T10:41:05.404 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:41:05.405+0000 7fb9a1cfc640 1 -- 172.21.15.114:0/668925059 shutdown_connections 2024-04-23T10:41:05.405 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:41:05.405+0000 7fb9a1cfc640 1 --2- 172.21.15.114:0/668925059 >> [v2:172.21.15.114:6800/1395693466,v1:172.21.15.114:6801/1395693466] conn(0x7fb9840a3da0 0x7fb9840a6260 unknown :-1 s=CLOSED pgs=151 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:41:05.405 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:41:05.405+0000 7fb9a1cfc640 1 --2- 172.21.15.114:0/668925059 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fb99c1985b0 0x7fb99c198a60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:41:05.405 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:41:05.405+0000 7fb9a1cfc640 1 --2- 172.21.15.114:0/668925059 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fb99c107260 0x7fb99c19e940 unknown :-1 s=CLOSED pgs=103 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:41:05.405 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:41:05.405+0000 7fb9a1cfc640 1 --2- 172.21.15.114:0/668925059 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fb99c102190 0x7fb99c19e400 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:41:05.405 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:41:05.405+0000 7fb9a1cfc640 1 -- 172.21.15.114:0/668925059 >> 172.21.15.114:0/668925059 conn(0x7fb99c079310 msgr2=0x7fb99c100280 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T10:41:05.405 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:41:05.405+0000 7fb9a1cfc640 1 -- 172.21.15.114:0/668925059 shutdown_connections 2024-04-23T10:41:05.405 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:41:05.405+0000 7fb9a1cfc640 1 -- 172.21.15.114:0/668925059 wait complete. 2024-04-23T10:41:05.416 DEBUG:tasks.quiescer.fs.[cephfs]:Running ceph command: 'tell mds.24545 quiesce db --cancel --all' 2024-04-23T10:41:05.416 DEBUG:teuthology.orchestra.run.smithi114:> timeout 75 sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.24545 quiesce db --cancel --all 2024-04-23T10:41:05.519 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:41:05.520+0000 7fe5c379b640 1 -- 172.21.15.114:0/1308592229 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fe5bc1031d0 msgr2=0x7fe5bc1035b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T10:41:05.519 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:41:05.520+0000 7fe5c379b640 1 --2- 172.21.15.114:0/1308592229 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fe5bc1031d0 0x7fe5bc1035b0 secure :-1 s=READY pgs=944 cs=0 l=1 rev1=1 crypto rx=0x7fe5b8009900 tx=0x7fe5b802a510 comp rx=0 tx=0).stop 2024-04-23T10:41:05.519 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:41:05.520+0000 7fe5c379b640 1 -- 172.21.15.114:0/1308592229 shutdown_connections 2024-04-23T10:41:05.519 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:41:05.520+0000 7fe5c379b640 1 --2- 172.21.15.114:0/1308592229 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fe5bc10c510 0x7fe5bc10e9b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:41:05.519 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:41:05.520+0000 7fe5c379b640 1 --2- 172.21.15.114:0/1308592229 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fe5bc103af0 0x7fe5bc10bfd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:41:05.519 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:41:05.520+0000 7fe5c379b640 1 --2- 172.21.15.114:0/1308592229 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fe5bc1031d0 0x7fe5bc1035b0 unknown :-1 s=CLOSED pgs=944 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:41:05.520 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:41:05.520+0000 7fe5c379b640 1 -- 172.21.15.114:0/1308592229 >> 172.21.15.114:0/1308592229 conn(0x7fe5bc077f50 msgr2=0x7fe5bc078360 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T10:41:05.520 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:41:05.520+0000 7fe5c379b640 1 -- 172.21.15.114:0/1308592229 shutdown_connections 2024-04-23T10:41:05.520 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:41:05.520+0000 7fe5c379b640 1 -- 172.21.15.114:0/1308592229 wait complete. 2024-04-23T10:41:05.522 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:41:05.521+0000 7fe5c379b640 1 Processor -- start 2024-04-23T10:41:05.522 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:41:05.521+0000 7fe5c379b640 1 -- start start 2024-04-23T10:41:05.522 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:41:05.521+0000 7fe5c379b640 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fe5bc1031d0 0x7fe5bc19e440 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T10:41:05.522 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:41:05.521+0000 7fe5c379b640 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fe5bc103af0 0x7fe5bc19e980 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T10:41:05.522 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:41:05.521+0000 7fe5c379b640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fe5bc10c510 0x7fe5bc1985f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T10:41:05.522 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:41:05.521+0000 7fe5c379b640 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x7fe5bc112180 con 0x7fe5bc10c510 2024-04-23T10:41:05.523 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:41:05.521+0000 7fe5c379b640 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 -- 0x7fe5bc112000 con 0x7fe5bc103af0 2024-04-23T10:41:05.523 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:41:05.521+0000 7fe5c379b640 1 -- --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_getmap magic: 0 -- 0x7fe5bc112300 con 0x7fe5bc1031d0 2024-04-23T10:41:05.523 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:41:05.521+0000 7fe5c1d38640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fe5bc10c510 0x7fe5bc1985f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T10:41:05.523 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:41:05.521+0000 7fe5c1537640 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fe5bc1031d0 0x7fe5bc19e440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T10:41:05.523 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:41:05.521+0000 7fe5c1d38640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fe5bc10c510 0x7fe5bc1985f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.114:44516/0 (socket says 172.21.15.114:44516) 2024-04-23T10:41:05.523 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:41:05.521+0000 7fe5c1537640 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fe5bc1031d0 0x7fe5bc19e440 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.204:3300/0 says I am v2:172.21.15.114:54226/0 (socket says 172.21.15.114:54226) 2024-04-23T10:41:05.523 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:41:05.521+0000 7fe5c1d38640 1 -- 172.21.15.114:0/2800857836 learned_addr learned my addr 172.21.15.114:0/2800857836 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-04-23T10:41:05.523 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:41:05.521+0000 7fe5c0d36640 1 --2- 172.21.15.114:0/2800857836 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fe5bc103af0 0x7fe5bc19e980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T10:41:05.523 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:41:05.521+0000 7fe5c1d38640 1 -- 172.21.15.114:0/2800857836 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fe5bc1031d0 msgr2=0x7fe5bc19e440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T10:41:05.524 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:41:05.521+0000 7fe5c1d38640 1 --2- 172.21.15.114:0/2800857836 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fe5bc1031d0 0x7fe5bc19e440 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:41:05.524 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:41:05.521+0000 7fe5c1d38640 1 -- 172.21.15.114:0/2800857836 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fe5bc103af0 msgr2=0x7fe5bc19e980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T10:41:05.524 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:41:05.521+0000 7fe5c1d38640 1 --2- 172.21.15.114:0/2800857836 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fe5bc103af0 0x7fe5bc19e980 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:41:05.524 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:41:05.521+0000 7fe5c1d38640 1 -- 172.21.15.114:0/2800857836 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe5bc198d70 con 0x7fe5bc10c510 2024-04-23T10:41:05.524 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:41:05.521+0000 7fe5c1d38640 1 --2- 172.21.15.114:0/2800857836 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fe5bc10c510 0x7fe5bc1985f0 secure :-1 s=READY pgs=945 cs=0 l=1 rev1=1 crypto rx=0x7fe5b400eaa0 tx=0x7fe5b400ef60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T10:41:05.524 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:41:05.522+0000 7fe5b27fc640 1 -- 172.21.15.114:0/2800857836 <== mon.0 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe5b400cdb0 con 0x7fe5bc10c510 2024-04-23T10:41:05.524 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:41:05.522+0000 7fe5b27fc640 1 -- 172.21.15.114:0/2800857836 <== mon.0 v2:172.21.15.114:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fe5b4005020 con 0x7fe5bc10c510 2024-04-23T10:41:05.524 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:41:05.522+0000 7fe5c379b640 1 -- 172.21.15.114:0/2800857836 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fe5bc199060 con 0x7fe5bc10c510 2024-04-23T10:41:05.524 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:41:05.522+0000 7fe5b27fc640 1 -- 172.21.15.114:0/2800857836 <== mon.0 v2:172.21.15.114:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe5b40116d0 con 0x7fe5bc10c510 2024-04-23T10:41:05.524 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:41:05.522+0000 7fe5c379b640 1 -- 172.21.15.114:0/2800857836 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fe5bc1a5300 con 0x7fe5bc10c510 2024-04-23T10:41:05.530 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:41:05.524+0000 7fe5b27fc640 1 -- 172.21.15.114:0/2800857836 <== mon.0 v2:172.21.15.114:3300/0 4 ==== mgrmap(e 26) ==== 138778+0+0 (secure 0 0 0) 0x7fe5b40051d0 con 0x7fe5bc10c510 2024-04-23T10:41:05.530 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:41:05.524+0000 7fe5c6590740 1 Processor -- start 2024-04-23T10:41:05.530 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:41:05.524+0000 7fe5c6590740 1 -- start start 2024-04-23T10:41:05.531 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:41:05.524+0000 7fe5c6590740 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55eb421f5360 0x55eb421f5740 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T10:41:05.531 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:41:05.524+0000 7fe5c6590740 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x55eb421f5d10 0x55eb421ffdb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T10:41:05.531 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:41:05.524+0000 7fe5c6590740 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x55eb42200380 0x55eb422027d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T10:41:05.531 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:41:05.524+0000 7fe5c6590740 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x55eb4215b130 con 0x55eb421f5360 2024-04-23T10:41:05.531 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:41:05.524+0000 7fe5c6590740 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 -- 0x55eb420f7c50 con 0x55eb42200380 2024-04-23T10:41:05.531 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:41:05.524+0000 7fe5c6590740 1 -- --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_getmap magic: 0 -- 0x55eb421570f0 con 0x55eb421f5d10 2024-04-23T10:41:05.531 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:41:05.524+0000 7fe5b27fc640 1 --2- 172.21.15.114:0/2800857836 >> [v2:172.21.15.114:6800/1395693466,v1:172.21.15.114:6801/1395693466] conn(0x7fe5a80a3d20 0x7fe5a80a61e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T10:41:05.531 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:41:05.524+0000 7fe5c0d36640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55eb421f5360 0x55eb421f5740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T10:41:05.531 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:41:05.524+0000 7fe5c0d36640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55eb421f5360 0x55eb421f5740 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.114:44526/0 (socket says 172.21.15.114:44526) 2024-04-23T10:41:05.532 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:41:05.525+0000 7fe5c0d36640 1 -- 172.21.15.114:0/3609505162 learned_addr learned my addr 172.21.15.114:0/3609505162 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-04-23T10:41:05.532 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:41:05.525+0000 7fe5c1d38640 1 --2- 172.21.15.114:0/2800857836 >> [v2:172.21.15.114:6800/1395693466,v1:172.21.15.114:6801/1395693466] conn(0x7fe5a80a3d20 0x7fe5a80a61e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T10:41:05.532 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:41:05.525+0000 7fe5c1537640 1 --2- 172.21.15.114:0/3609505162 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x55eb421f5d10 0x55eb421ffdb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T10:41:05.532 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:41:05.525+0000 7fe5b27fc640 1 -- 172.21.15.114:0/2800857836 <== mon.0 v2:172.21.15.114:3300/0 5 ==== osd_map(111..111 src has 1..111) ==== 8585+0+0 (secure 0 0 0) 0x7fe5b40d09a0 con 0x7fe5bc10c510 2024-04-23T10:41:05.532 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:41:05.525+0000 7fe5c0d36640 1 --2- 172.21.15.114:0/3609505162 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x55eb42200380 0x55eb422027d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T10:41:05.532 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:41:05.525+0000 7fe5c0d36640 1 -- 172.21.15.114:0/3609505162 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x55eb421f5d10 msgr2=0x55eb421ffdb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T10:41:05.532 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:41:05.525+0000 7fe5c0d36640 1 --2- 172.21.15.114:0/3609505162 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x55eb421f5d10 0x55eb421ffdb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:41:05.532 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:41:05.525+0000 7fe5c0d36640 1 -- 172.21.15.114:0/3609505162 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x55eb42200380 msgr2=0x55eb422027d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T10:41:05.532 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:41:05.525+0000 7fe5c0d36640 1 --2- 172.21.15.114:0/3609505162 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x55eb42200380 0x55eb422027d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:41:05.532 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:41:05.525+0000 7fe5c0d36640 1 -- 172.21.15.114:0/3609505162 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55eb42111fd0 con 0x55eb421f5360 2024-04-23T10:41:05.533 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:41:05.525+0000 7fe5c1537640 1 --2- 172.21.15.114:0/3609505162 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x55eb421f5d10 0x55eb421ffdb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-23T10:41:05.533 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:41:05.525+0000 7fe5c1d38640 1 --2- 172.21.15.114:0/2800857836 >> [v2:172.21.15.114:6800/1395693466,v1:172.21.15.114:6801/1395693466] conn(0x7fe5a80a3d20 0x7fe5a80a61e0 secure :-1 s=READY pgs=152 cs=0 l=1 rev1=1 crypto rx=0x7fe5b400efd0 tx=0x7fe5b4091d90 comp rx=0 tx=0).ready entity=mgr.15291 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T10:41:05.533 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:41:05.525+0000 7fe5c0d36640 1 --2- 172.21.15.114:0/3609505162 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55eb421f5360 0x55eb421f5740 secure :-1 s=READY pgs=946 cs=0 l=1 rev1=1 crypto rx=0x7fe5ac00bfd0 tx=0x7fe5ac00b650 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a5c55c23693583fe server_cookie=0 in_seq=0 out_seq=0 2024-04-23T10:41:05.533 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:41:05.525+0000 7fe5a77fe640 1 -- 172.21.15.114:0/3609505162 <== mon.0 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe5ac019030 con 0x55eb421f5360 2024-04-23T10:41:05.533 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:41:05.525+0000 7fe5a77fe640 1 -- 172.21.15.114:0/3609505162 <== mon.0 v2:172.21.15.114:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fe5ac011970 con 0x55eb421f5360 2024-04-23T10:41:05.533 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:41:05.525+0000 7fe5a77fe640 1 -- 172.21.15.114:0/3609505162 <== mon.0 v2:172.21.15.114:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe5ac00f740 con 0x55eb421f5360 2024-04-23T10:41:05.533 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:41:05.526+0000 7fe5c6590740 1 -- 172.21.15.114:0/3609505162 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55eb421f5360 msgr2=0x55eb421f5740 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T10:41:05.533 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:41:05.526+0000 7fe5c6590740 1 --2- 172.21.15.114:0/3609505162 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55eb421f5360 0x55eb421f5740 secure :-1 s=READY pgs=946 cs=0 l=1 rev1=1 crypto rx=0x7fe5ac00bfd0 tx=0x7fe5ac00b650 comp rx=0 tx=0).stop 2024-04-23T10:41:05.533 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:41:05.526+0000 7fe5c6590740 1 -- 172.21.15.114:0/3609505162 shutdown_connections 2024-04-23T10:41:05.533 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:41:05.526+0000 7fe5c6590740 1 --2- 172.21.15.114:0/3609505162 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x55eb42200380 0x55eb422027d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:41:05.534 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:41:05.526+0000 7fe5c6590740 1 --2- 172.21.15.114:0/3609505162 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x55eb421f5d10 0x55eb421ffdb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:41:05.534 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:41:05.526+0000 7fe5c6590740 1 --2- 172.21.15.114:0/3609505162 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55eb421f5360 0x55eb421f5740 unknown :-1 s=CLOSED pgs=946 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:41:05.534 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:41:05.526+0000 7fe5c6590740 1 -- 172.21.15.114:0/3609505162 >> 172.21.15.114:0/3609505162 conn(0x55eb421f1620 msgr2=0x55eb421f1a00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T10:41:05.534 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:41:05.526+0000 7fe5c6590740 1 -- 172.21.15.114:0/3609505162 shutdown_connections 2024-04-23T10:41:05.534 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:41:05.526+0000 7fe5c6590740 1 -- 172.21.15.114:0/3609505162 wait complete. 2024-04-23T10:41:05.534 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:41:05.526+0000 7fe5c6590740 1 -- start start 2024-04-23T10:41:05.534 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:41:05.526+0000 7fe5c6590740 1 Processor -- start 2024-04-23T10:41:05.534 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:41:05.527+0000 7fe5c6590740 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x55eb421f5360 0x55eb421fd1d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T10:41:05.534 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:41:05.527+0000 7fe5c6590740 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55eb421f5d10 0x55eb421fd710 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T10:41:05.534 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:41:05.527+0000 7fe5c6590740 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x55eb42200380 0x55eb421fdc50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T10:41:05.535 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:41:05.527+0000 7fe5c6590740 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x55eb422032f0 con 0x55eb421f5d10 2024-04-23T10:41:05.535 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:41:05.527+0000 7fe5c6590740 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 -- 0x55eb420f5640 con 0x55eb42200380 2024-04-23T10:41:05.535 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:41:05.527+0000 7fe5c6590740 1 -- --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_getmap magic: 0 -- 0x55eb42203470 con 0x55eb421f5360 2024-04-23T10:41:05.535 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:41:05.527+0000 7fe5c6590740 10 client.0 _ll_register_callbacks cb 0x55eb420f5030 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-23T10:41:05.535 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:41:05.527+0000 7fe5c1537640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55eb421f5d10 0x55eb421fd710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T10:41:05.535 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:41:05.527+0000 7fe5c1537640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55eb421f5d10 0x55eb421fd710 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.114:44540/0 (socket says 172.21.15.114:44540) 2024-04-23T10:41:05.535 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:41:05.527+0000 7fe5c1537640 1 -- 172.21.15.114:0/3978864209 learned_addr learned my addr 172.21.15.114:0/3978864209 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-04-23T10:41:05.535 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:41:05.527+0000 7fe5c0d36640 1 --2- 172.21.15.114:0/3978864209 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x55eb421f5360 0x55eb421fd1d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T10:41:05.535 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:41:05.527+0000 7fe5c1537640 1 -- 172.21.15.114:0/3978864209 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x55eb421f5360 msgr2=0x55eb421fd1d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T10:41:05.535 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:41:05.527+0000 7fe5c1537640 1 --2- 172.21.15.114:0/3978864209 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x55eb421f5360 0x55eb421fd1d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:41:05.536 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:41:05.527+0000 7fe5c1537640 1 -- 172.21.15.114:0/3978864209 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x55eb42200380 msgr2=0x55eb421fdc50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T10:41:05.536 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:41:05.527+0000 7fe5c1537640 1 --2- 172.21.15.114:0/3978864209 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x55eb42200380 0x55eb421fdc50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:41:05.536 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:41:05.527+0000 7fe5c1537640 1 -- 172.21.15.114:0/3978864209 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55eb41f39a20 con 0x55eb421f5d10 2024-04-23T10:41:05.536 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:41:05.527+0000 7fe5c1537640 1 --2- 172.21.15.114:0/3978864209 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55eb421f5d10 0x55eb421fd710 secure :-1 s=READY pgs=947 cs=0 l=1 rev1=1 crypto rx=0x7fe5b802aa30 tx=0x7fe5b8032e90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d1c6ed796f09394a server_cookie=0 in_seq=0 out_seq=0 2024-04-23T10:41:05.536 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:41:05.527+0000 7fe5a4ff9640 10 client.0 ms_handle_connect on v2:172.21.15.114:3300/0 2024-04-23T10:41:05.536 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:41:05.527+0000 7fe5a4ff9640 1 -- 172.21.15.114:0/3978864209 <== mon.0 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe5b8035040 con 0x55eb421f5d10 2024-04-23T10:41:05.536 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:41:05.527+0000 7fe5a4ff9640 1 -- 172.21.15.114:0/3978864209 <== mon.0 v2:172.21.15.114:3300/0 2 ==== osd_map(111..111 src has 1..111) ==== 8585+0+0 (secure 0 0 0) 0x7fe5b8031760 con 0x55eb421f5d10 2024-04-23T10:41:05.536 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:41:05.527+0000 7fe5c6590740 1 -- 172.21.15.114:0/3978864209 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fe5ac011970 con 0x55eb421f5d10 2024-04-23T10:41:05.536 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:41:05.527+0000 7fe5a4ff9640 1 -- 172.21.15.114:0/3978864209 <== mon.0 v2:172.21.15.114:3300/0 3 ==== mon_get_version_reply(handle=1 version=49) ==== 24+0+0 (secure 0 0 0) 0x7fe5b8004040 con 0x55eb421f5d10 2024-04-23T10:41:05.537 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:41:05.527+0000 7fe5c6590740 10 client.17151 fetch_fsmap learned FSMap version 49 2024-04-23T10:41:05.537 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:41:05.527+0000 7fe5c6590740 1 -- 172.21.15.114:0/3978864209 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({config=0+,fsmap=49,monmap=4+}) -- 0x55eb41f9a1b0 con 0x55eb421f5d10 2024-04-23T10:41:05.537 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:41:05.528+0000 7fe5a4ff9640 1 -- 172.21.15.114:0/3978864209 <== mon.0 v2:172.21.15.114:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fe5ac011970 con 0x55eb421f5d10 2024-04-23T10:41:05.537 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:41:05.528+0000 7fe5a4ff9640 1 -- 172.21.15.114:0/3978864209 <== mon.0 v2:172.21.15.114:3300/0 5 ==== fsmap(e 49) ==== 8358+0+0 (secure 0 0 0) 0x7fe5b8004750 con 0x55eb421f5d10 2024-04-23T10:41:05.537 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:41:05.528+0000 7fe5c6590740 10 client.17151 fetch_fsmap finished waiting for FSMap version 49 2024-04-23T10:41:05.537 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:41:05.528+0000 7fe5c6590740 10 client.17151 resolve_mds: validated gid 24545 aka daemon mds.b 2024-04-23T10:41:05.537 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:41:05.528+0000 7fe5c6590740 20 client.17151 populate_metadata read hostname 'smithi114' 2024-04-23T10:41:05.537 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:41:05.528+0000 7fe5c6590740 1 --2- 172.21.15.114:0/3978864209 >> [v2:172.21.15.119:6838/937297340,v1:172.21.15.119:6839/937297340] conn(0x55eb42210860 0x55eb42212c50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T10:41:05.537 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:41:05.528+0000 7fe5c6590740 4 client.17151 mds_command: new command op to 24545 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-23T10:41:05.537 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:41:05.528+0000 7fe5c6590740 1 -- 172.21.15.114:0/3978864209 --> [v2:172.21.15.119:6838/937297340,v1:172.21.15.119:6839/937297340] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55eb42168b50 con 0x55eb42210860 2024-04-23T10:41:05.538 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:41:05.528+0000 7fe5c0d36640 1 --2- 172.21.15.114:0/3978864209 >> [v2:172.21.15.119:6838/937297340,v1:172.21.15.119:6839/937297340] conn(0x55eb42210860 0x55eb42212c50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T10:41:05.538 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:41:05.528+0000 7fe5a4ff9640 10 client.17151 ms_handle_connect on v2:172.21.15.119:6838/937297340 2024-04-23T10:41:05.538 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:41:05.528+0000 7fe5c0d36640 1 --2- 172.21.15.114:0/3978864209 >> [v2:172.21.15.119:6838/937297340,v1:172.21.15.119:6839/937297340] conn(0x55eb42210860 0x55eb42212c50 crc :-1 s=READY pgs=192 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=8bb40c9984808bb7 server_cookie=8999fa0d11794772 in_seq=0 out_seq=0 2024-04-23T10:41:05.538 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:41:05.529+0000 7fe5a4ff9640 1 -- 172.21.15.114:0/3978864209 <== mds.0 v2:172.21.15.119:6838/937297340 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x55eb42168b50 con 0x55eb42210860 2024-04-23T10:41:05.538 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:41:05.529+0000 7fe5a4ff9640 10 client.17151 handle_command_reply: tid=0 2024-04-23T10:41:05.538 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:41:05.529+0000 7fe5c6590740 1 client.17151 shutdown 2024-04-23T10:41:05.538 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:41:05.529+0000 7fe5c6590740 1 -- 172.21.15.114:0/3978864209 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55eb421f5d10 msgr2=0x55eb421fd710 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T10:41:05.538 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:41:05.529+0000 7fe5c6590740 1 --2- 172.21.15.114:0/3978864209 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55eb421f5d10 0x55eb421fd710 secure :-1 s=READY pgs=947 cs=0 l=1 rev1=1 crypto rx=0x7fe5b802aa30 tx=0x7fe5b8032e90 comp rx=0 tx=0).stop 2024-04-23T10:41:05.539 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:41:05.529+0000 7fe5c6590740 1 -- 172.21.15.114:0/3978864209 shutdown_connections 2024-04-23T10:41:05.539 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:41:05.529+0000 7fe5c6590740 1 --2- 172.21.15.114:0/3978864209 >> [v2:172.21.15.119:6838/937297340,v1:172.21.15.119:6839/937297340] conn(0x55eb42210860 0x55eb42212c50 crc :-1 s=READY pgs=192 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:41:05.539 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:41:05.529+0000 7fe5c6590740 1 --2- 172.21.15.114:0/3978864209 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x55eb42200380 0x55eb421fdc50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:41:05.539 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:41:05.529+0000 7fe5a4ff9640 0 client.17151 ms_handle_reset on v2:172.21.15.119:6838/937297340 2024-04-23T10:41:05.539 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:41:05.529+0000 7fe5c6590740 1 --2- 172.21.15.114:0/3978864209 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55eb421f5d10 0x55eb421fd710 unknown :-1 s=CLOSED pgs=947 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:41:05.539 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:41:05.529+0000 7fe5c6590740 1 --2- 172.21.15.114:0/3978864209 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x55eb421f5360 0x55eb421fd1d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:41:05.539 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:41:05.529+0000 7fe5c6590740 1 -- 172.21.15.114:0/3978864209 >> 172.21.15.114:0/3978864209 conn(0x55eb421f1620 msgr2=0x55eb421f7490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T10:41:05.539 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:41:05.529+0000 7fe5c6590740 1 -- 172.21.15.114:0/3978864209 shutdown_connections 2024-04-23T10:41:05.539 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:41:05.530+0000 7fe5c6590740 1 -- 172.21.15.114:0/3978864209 wait complete. 2024-04-23T10:41:05.540 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:41:05.530+0000 7fe5c6590740 20 client.17151 trim_cache size 0 max 16384 2024-04-23T10:41:05.547 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:41:05.546+0000 7fe5c6590740 1 Processor -- start 2024-04-23T10:41:05.547 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:41:05.546+0000 7fe5c6590740 1 -- start start 2024-04-23T10:41:05.547 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:41:05.547+0000 7fe5c6590740 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55eb421f5360 0x55eb421f7960 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T10:41:05.547 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:41:05.547+0000 7fe5c6590740 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x55eb421f5d10 0x55eb421f07c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T10:41:05.547 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:41:05.547+0000 7fe5c6590740 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x55eb42200380 0x55eb42218560 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T10:41:05.548 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:41:05.547+0000 7fe5c6590740 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x7fe5bc1b0640 con 0x55eb421f5360 2024-04-23T10:41:05.548 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:41:05.547+0000 7fe5c6590740 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 -- 0x7fe5bc1ade80 con 0x55eb42200380 2024-04-23T10:41:05.548 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:41:05.547+0000 7fe5c6590740 1 -- --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_getmap magic: 0 -- 0x7fe5bc1abcd0 con 0x55eb421f5d10 2024-04-23T10:41:05.548 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:41:05.547+0000 7fe5c0d36640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55eb421f5360 0x55eb421f7960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T10:41:05.548 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:41:05.547+0000 7fe5c0d36640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55eb421f5360 0x55eb421f7960 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.114:44552/0 (socket says 172.21.15.114:44552) 2024-04-23T10:41:05.548 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:41:05.547+0000 7fe5c0d36640 1 -- 172.21.15.114:0/3364867328 learned_addr learned my addr 172.21.15.114:0/3364867328 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-04-23T10:41:05.548 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:41:05.547+0000 7fe5c0d36640 1 --2- 172.21.15.114:0/3364867328 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x55eb42200380 0x55eb42218560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T10:41:05.548 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:41:05.547+0000 7fe5c0d36640 1 -- 172.21.15.114:0/3364867328 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x55eb421f5d10 msgr2=0x55eb421f07c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T10:41:05.549 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:41:05.547+0000 7fe5c1537640 1 --2- 172.21.15.114:0/3364867328 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x55eb421f5d10 0x55eb421f07c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T10:41:05.549 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:41:05.547+0000 7fe5c0d36640 1 --2- 172.21.15.114:0/3364867328 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x55eb421f5d10 0x55eb421f07c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:41:05.549 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:41:05.547+0000 7fe5c0d36640 1 -- 172.21.15.114:0/3364867328 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x55eb42200380 msgr2=0x55eb42218560 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T10:41:05.549 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:41:05.547+0000 7fe5c0d36640 1 --2- 172.21.15.114:0/3364867328 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x55eb42200380 0x55eb42218560 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:41:05.549 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:41:05.547+0000 7fe5c0d36640 1 -- 172.21.15.114:0/3364867328 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55eb41e2d310 con 0x55eb421f5360 2024-04-23T10:41:05.549 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:41:05.547+0000 7fe5c0d36640 1 --2- 172.21.15.114:0/3364867328 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55eb421f5360 0x55eb421f7960 secure :-1 s=READY pgs=948 cs=0 l=1 rev1=1 crypto rx=0x7fe5ac013ec0 tx=0x7fe5ac005bf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d65dfc400f4316e0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T10:41:05.549 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:41:05.547+0000 7fe5a6ffd640 1 -- 172.21.15.114:0/3364867328 <== mon.0 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe5ac019040 con 0x55eb421f5360 2024-04-23T10:41:05.549 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:41:05.547+0000 7fe5a6ffd640 1 -- 172.21.15.114:0/3364867328 <== mon.0 v2:172.21.15.114:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fe5ac005030 con 0x55eb421f5360 2024-04-23T10:41:05.549 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:41:05.547+0000 7fe5a6ffd640 1 -- 172.21.15.114:0/3364867328 <== mon.0 v2:172.21.15.114:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe5ac0283d0 con 0x55eb421f5360 2024-04-23T10:41:05.549 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:41:05.548+0000 7fe5c6590740 1 -- 172.21.15.114:0/3364867328 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55eb421f5360 msgr2=0x55eb421f7960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T10:41:05.550 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:41:05.548+0000 7fe5c6590740 1 --2- 172.21.15.114:0/3364867328 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55eb421f5360 0x55eb421f7960 secure :-1 s=READY pgs=948 cs=0 l=1 rev1=1 crypto rx=0x7fe5ac013ec0 tx=0x7fe5ac005bf0 comp rx=0 tx=0).stop 2024-04-23T10:41:05.550 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:41:05.548+0000 7fe5c6590740 1 -- 172.21.15.114:0/3364867328 shutdown_connections 2024-04-23T10:41:05.550 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:41:05.548+0000 7fe5c6590740 1 --2- 172.21.15.114:0/3364867328 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x55eb42200380 0x55eb42218560 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:41:05.550 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:41:05.548+0000 7fe5c6590740 1 --2- 172.21.15.114:0/3364867328 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x55eb421f5d10 0x55eb421f07c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:41:05.550 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:41:05.548+0000 7fe5c6590740 1 --2- 172.21.15.114:0/3364867328 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55eb421f5360 0x55eb421f7960 unknown :-1 s=CLOSED pgs=948 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:41:05.550 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:41:05.548+0000 7fe5c6590740 1 -- 172.21.15.114:0/3364867328 >> 172.21.15.114:0/3364867328 conn(0x55eb421f1620 msgr2=0x55eb421f00b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T10:41:05.550 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:41:05.548+0000 7fe5c6590740 1 -- 172.21.15.114:0/3364867328 shutdown_connections 2024-04-23T10:41:05.550 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:41:05.548+0000 7fe5c6590740 1 -- 172.21.15.114:0/3364867328 wait complete. 2024-04-23T10:41:05.550 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:41:05.548+0000 7fe5c6590740 1 -- start start 2024-04-23T10:41:05.551 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:41:05.548+0000 7fe5c6590740 1 Processor -- start 2024-04-23T10:41:05.554 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:41:05.549+0000 7fe5c6590740 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x55eb421f5360 0x55eb421fa140 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T10:41:05.554 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:41:05.549+0000 7fe5c6590740 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55eb421f5d10 0x55eb421fa680 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T10:41:05.554 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:41:05.549+0000 7fe5c6590740 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x55eb42200380 0x55eb421fabc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T10:41:05.554 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:41:05.549+0000 7fe5c6590740 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x7fe5bc1ac340 con 0x55eb421f5d10 2024-04-23T10:41:05.555 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:41:05.549+0000 7fe5c6590740 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 -- 0x7fe5bc1ad7b0 con 0x55eb42200380 2024-04-23T10:41:05.555 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:41:05.549+0000 7fe5c6590740 1 -- --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_getmap magic: 0 -- 0x7fe5bc1aca10 con 0x55eb421f5360 2024-04-23T10:41:05.555 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:41:05.549+0000 7fe5c6590740 10 client.0 _ll_register_callbacks cb 0x7fe5b8032b20 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-23T10:41:05.555 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:41:05.549+0000 7fe5c1537640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55eb421f5d10 0x55eb421fa680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T10:41:05.555 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:41:05.549+0000 7fe5c0d36640 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x55eb421f5360 0x55eb421fa140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T10:41:05.555 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:41:05.549+0000 7fe5c0d36640 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x55eb421f5360 0x55eb421fa140 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.204:3300/0 says I am v2:172.21.15.114:54248/0 (socket says 172.21.15.114:54248) 2024-04-23T10:41:05.555 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:41:05.549+0000 7fe5c1537640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55eb421f5d10 0x55eb421fa680 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.114:44556/0 (socket says 172.21.15.114:44556) 2024-04-23T10:41:05.555 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:41:05.549+0000 7fe5c0d36640 1 -- 172.21.15.114:0/4211059381 learned_addr learned my addr 172.21.15.114:0/4211059381 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-04-23T10:41:05.555 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:41:05.549+0000 7fe5c0d36640 1 --2- 172.21.15.114:0/4211059381 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x55eb42200380 0x55eb421fabc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T10:41:05.556 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:41:05.549+0000 7fe5c0d36640 1 -- 172.21.15.114:0/4211059381 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x55eb42200380 msgr2=0x55eb421fabc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T10:41:05.556 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:41:05.549+0000 7fe5c0d36640 1 --2- 172.21.15.114:0/4211059381 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x55eb42200380 0x55eb421fabc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:41:05.556 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:41:05.549+0000 7fe5c0d36640 1 -- 172.21.15.114:0/4211059381 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55eb421f5d10 msgr2=0x55eb421fa680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-23T10:41:05.556 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:41:05.549+0000 7fe5c0d36640 1 --2- 172.21.15.114:0/4211059381 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55eb421f5d10 0x55eb421fa680 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:41:05.556 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:41:05.549+0000 7fe5c0d36640 1 -- 172.21.15.114:0/4211059381 --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55eb42276070 con 0x55eb421f5360 2024-04-23T10:41:05.556 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:41:05.549+0000 7fe5c1537640 1 --2- 172.21.15.114:0/4211059381 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55eb421f5d10 0x55eb421fa680 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-23T10:41:05.556 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:41:05.549+0000 7fe5c0d36640 1 --2- 172.21.15.114:0/4211059381 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x55eb421f5360 0x55eb421fa140 secure :-1 s=READY pgs=87 cs=0 l=1 rev1=1 crypto rx=0x55eb421f2460 tx=0x7fe5ac02c040 comp rx=0 tx=0).ready entity=mon.1 client_cookie=a6cde1e1121c2654 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T10:41:05.556 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:41:05.549+0000 7fe5a4ff9640 10 client.0 ms_handle_connect on v2:172.21.15.204:3300/0 2024-04-23T10:41:05.556 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:41:05.550+0000 7fe5a4ff9640 1 -- 172.21.15.114:0/4211059381 <== mon.1 v2:172.21.15.204:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe5ac019070 con 0x55eb421f5360 2024-04-23T10:41:05.556 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:41:05.550+0000 7fe5c6590740 1 -- 172.21.15.114:0/4211059381 --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fe5ac005030 con 0x55eb421f5360 2024-04-23T10:41:05.557 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:41:05.550+0000 7fe5a4ff9640 1 -- 172.21.15.114:0/4211059381 <== mon.1 v2:172.21.15.204:3300/0 2 ==== osd_map(111..111 src has 1..111) ==== 8585+0+0 (secure 0 0 0) 0x7fe5ac030040 con 0x55eb421f5360 2024-04-23T10:41:05.557 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:41:05.550+0000 7fe5a4ff9640 1 -- 172.21.15.114:0/4211059381 <== mon.1 v2:172.21.15.204:3300/0 3 ==== mon_get_version_reply(handle=1 version=49) ==== 24+0+0 (secure 0 0 0) 0x7fe5ac028350 con 0x55eb421f5360 2024-04-23T10:41:05.557 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:41:05.550+0000 7fe5c6590740 10 client.26311 fetch_fsmap learned FSMap version 49 2024-04-23T10:41:05.557 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:41:05.550+0000 7fe5c6590740 1 -- 172.21.15.114:0/4211059381 --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_subscribe({config=0+,fsmap=49,monmap=4+}) -- 0x55eb42277330 con 0x55eb421f5360 2024-04-23T10:41:05.557 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:41:05.550+0000 7fe5a4ff9640 1 -- 172.21.15.114:0/4211059381 <== mon.1 v2:172.21.15.204:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fe5ac005030 con 0x55eb421f5360 2024-04-23T10:41:05.557 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:41:05.550+0000 7fe5a4ff9640 1 -- 172.21.15.114:0/4211059381 <== mon.1 v2:172.21.15.204:3300/0 5 ==== fsmap(e 49) ==== 8358+0+0 (secure 0 0 0) 0x7fe5ac03d070 con 0x55eb421f5360 2024-04-23T10:41:05.557 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:41:05.550+0000 7fe5c6590740 10 client.26311 fetch_fsmap finished waiting for FSMap version 49 2024-04-23T10:41:05.557 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:41:05.550+0000 7fe5c6590740 10 client.26311 resolve_mds: validated gid 24545 aka daemon mds.b 2024-04-23T10:41:05.557 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:41:05.550+0000 7fe5c6590740 20 client.26311 populate_metadata read hostname 'smithi114' 2024-04-23T10:41:05.557 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:41:05.550+0000 7fe5c6590740 1 --2- 172.21.15.114:0/4211059381 >> [v2:172.21.15.119:6838/937297340,v1:172.21.15.119:6839/937297340] conn(0x55eb422778d0 0x55eb42279cf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T10:41:05.558 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:41:05.550+0000 7fe5c6590740 4 client.26311 mds_command: new command op to 24545 tid=0 multi_id=0 [{"prefix": "quiesce db", "cancel": true, "all": true}] 2024-04-23T10:41:05.558 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:41:05.550+0000 7fe5c6590740 1 -- 172.21.15.114:0/4211059381 --> [v2:172.21.15.119:6838/937297340,v1:172.21.15.119:6839/937297340] -- command(tid 0: {"prefix": "quiesce db", "cancel": true, "all": true}) -- 0x55eb421629c0 con 0x55eb422778d0 2024-04-23T10:41:05.558 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:41:05.550+0000 7fe5c1537640 1 --2- 172.21.15.114:0/4211059381 >> [v2:172.21.15.119:6838/937297340,v1:172.21.15.119:6839/937297340] conn(0x55eb422778d0 0x55eb42279cf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T10:41:05.558 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:41:05.551+0000 7fe5a4ff9640 10 client.26311 ms_handle_connect on v2:172.21.15.119:6838/937297340 2024-04-23T10:41:05.558 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:41:05.551+0000 7fe5c1537640 1 --2- 172.21.15.114:0/4211059381 >> [v2:172.21.15.119:6838/937297340,v1:172.21.15.119:6839/937297340] conn(0x55eb422778d0 0x55eb42279cf0 crc :-1 s=READY pgs=193 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=eb2927bf093ffd90 server_cookie=714c7d79e4536fba in_seq=0 out_seq=0 2024-04-23T10:41:05.558 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:41:05.551+0000 7fe5a4ff9640 1 -- 172.21.15.114:0/4211059381 <== mds.0 v2:172.21.15.119:6838/937297340 1 ==== command_reply(tid 0: 0 ) ==== 8+0+80 (crc 0 0 0) 0x55eb421629c0 con 0x55eb422778d0 2024-04-23T10:41:05.558 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:41:05.551+0000 7fe5a4ff9640 10 client.26311 handle_command_reply: tid=0 2024-04-23T10:41:05.558 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:41:05.551+0000 7fe5c6590740 1 client.26311 shutdown 2024-04-23T10:41:05.558 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:41:05.551+0000 7fe5c6590740 1 -- 172.21.15.114:0/4211059381 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x55eb421f5360 msgr2=0x55eb421fa140 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T10:41:05.559 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:41:05.551+0000 7fe5c6590740 1 --2- 172.21.15.114:0/4211059381 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x55eb421f5360 0x55eb421fa140 secure :-1 s=READY pgs=87 cs=0 l=1 rev1=1 crypto rx=0x55eb421f2460 tx=0x7fe5ac02c040 comp rx=0 tx=0).stop 2024-04-23T10:41:05.559 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:41:05.551+0000 7fe5c6590740 1 -- 172.21.15.114:0/4211059381 shutdown_connections 2024-04-23T10:41:05.559 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:41:05.551+0000 7fe5c6590740 1 --2- 172.21.15.114:0/4211059381 >> [v2:172.21.15.119:6838/937297340,v1:172.21.15.119:6839/937297340] conn(0x55eb422778d0 0x55eb42279cf0 crc :-1 s=READY pgs=193 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:41:05.559 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:41:05.551+0000 7fe5c6590740 1 --2- 172.21.15.114:0/4211059381 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x55eb42200380 0x55eb421fabc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:41:05.559 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:41:05.551+0000 7fe5a4ff9640 0 client.26311 ms_handle_reset on v2:172.21.15.119:6838/937297340 2024-04-23T10:41:05.559 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:41:05.551+0000 7fe5c6590740 1 --2- 172.21.15.114:0/4211059381 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55eb421f5d10 0x55eb421fa680 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:41:05.559 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:41:05.551+0000 7fe5c6590740 1 --2- 172.21.15.114:0/4211059381 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x55eb421f5360 0x55eb421fa140 unknown :-1 s=CLOSED pgs=87 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:41:05.559 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:41:05.551+0000 7fe5c6590740 1 -- 172.21.15.114:0/4211059381 >> 172.21.15.114:0/4211059381 conn(0x55eb421f1620 msgr2=0x55eb421f0940 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T10:41:05.559 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:41:05.551+0000 7fe5c6590740 1 -- 172.21.15.114:0/4211059381 shutdown_connections 2024-04-23T10:41:05.559 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:41:05.551+0000 7fe5c6590740 1 -- 172.21.15.114:0/4211059381 wait complete. 2024-04-23T10:41:05.560 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:41:05.551+0000 7fe5c6590740 20 client.26311 trim_cache size 0 max 16384 2024-04-23T10:41:05.560 INFO:teuthology.orchestra.run.smithi114.stdout:{ 2024-04-23T10:41:05.560 INFO:teuthology.orchestra.run.smithi114.stdout: "epoch": 49, 2024-04-23T10:41:05.560 INFO:teuthology.orchestra.run.smithi114.stdout: "leader": 24545, 2024-04-23T10:41:05.560 INFO:teuthology.orchestra.run.smithi114.stdout: "set_version": 18, 2024-04-23T10:41:05.560 INFO:teuthology.orchestra.run.smithi114.stdout: "sets": {} 2024-04-23T10:41:05.560 INFO:teuthology.orchestra.run.smithi114.stdout:} 2024-04-23T10:41:05.562 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:41:05.552+0000 7fe5c379b640 1 -- 172.21.15.114:0/2800857836 >> [v2:172.21.15.114:6800/1395693466,v1:172.21.15.114:6801/1395693466] conn(0x7fe5a80a3d20 msgr2=0x7fe5a80a61e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T10:41:05.562 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:41:05.552+0000 7fe5c379b640 1 --2- 172.21.15.114:0/2800857836 >> [v2:172.21.15.114:6800/1395693466,v1:172.21.15.114:6801/1395693466] conn(0x7fe5a80a3d20 0x7fe5a80a61e0 secure :-1 s=READY pgs=152 cs=0 l=1 rev1=1 crypto rx=0x7fe5b400efd0 tx=0x7fe5b4091d90 comp rx=0 tx=0).stop 2024-04-23T10:41:05.562 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:41:05.552+0000 7fe5c379b640 1 -- 172.21.15.114:0/2800857836 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fe5bc10c510 msgr2=0x7fe5bc1985f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T10:41:05.562 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:41:05.552+0000 7fe5c379b640 1 --2- 172.21.15.114:0/2800857836 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fe5bc10c510 0x7fe5bc1985f0 secure :-1 s=READY pgs=945 cs=0 l=1 rev1=1 crypto rx=0x7fe5b400eaa0 tx=0x7fe5b400ef60 comp rx=0 tx=0).stop 2024-04-23T10:41:05.562 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:41:05.552+0000 7fe5c379b640 1 -- 172.21.15.114:0/2800857836 shutdown_connections 2024-04-23T10:41:05.562 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:41:05.552+0000 7fe5c379b640 1 --2- 172.21.15.114:0/2800857836 >> [v2:172.21.15.114:6800/1395693466,v1:172.21.15.114:6801/1395693466] conn(0x7fe5a80a3d20 0x7fe5a80a61e0 unknown :-1 s=CLOSED pgs=152 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:41:05.562 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:41:05.552+0000 7fe5c379b640 1 --2- 172.21.15.114:0/2800857836 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fe5bc10c510 0x7fe5bc1985f0 unknown :-1 s=CLOSED pgs=945 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:41:05.563 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:41:05.552+0000 7fe5c379b640 1 --2- 172.21.15.114:0/2800857836 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fe5bc103af0 0x7fe5bc19e980 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:41:05.563 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:41:05.552+0000 7fe5c379b640 1 --2- 172.21.15.114:0/2800857836 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fe5bc1031d0 0x7fe5bc19e440 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:41:05.563 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:41:05.552+0000 7fe5c379b640 1 -- 172.21.15.114:0/2800857836 >> 172.21.15.114:0/2800857836 conn(0x7fe5bc077f50 msgr2=0x7fe5bc0ff380 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T10:41:05.563 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:41:05.552+0000 7fe5c379b640 1 -- 172.21.15.114:0/2800857836 shutdown_connections 2024-04-23T10:41:05.563 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:41:05.552+0000 7fe5c379b640 1 -- 172.21.15.114:0/2800857836 wait complete. 2024-04-23T10:41:05.563 ERROR:teuthology.run_tasks:Manager failed: quiescer Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_9481b1d62f50e7d0a4f3dd83adf6945b08d5ff17/teuthology/run_tasks.py", line 154, in run_tasks suppress = manager.__exit__(*exc_info) File "/usr/lib/python3.8/contextlib.py", line 120, in __exit__ next(self.gen) File "/home/teuthworker/src/github.com_ceph_ceph_0a6c3ed699031b80a2b419e7e795368719871394/qa/tasks/quiescer.py", line 371, in task stop_all_quiescers(ctx.ceph[config['cluster']].thrashers) File "/home/teuthworker/src/github.com_ceph_ceph_0a6c3ed699031b80a2b419e7e795368719871394/qa/tasks/quiescer.py", line 324, in stop_all_quiescers raise RuntimeError(f"error during quiesce thrashing: {thrasher.exception}") RuntimeError: error during quiesce thrashing: Error quiescing root: 110 (ETIMEDOUT) 2024-04-23T10:41:05.567 DEBUG:teuthology.run_tasks:Unwinding manager background_exec 2024-04-23T10:41:05.577 INFO:teuthology.task.background_exec:Stopping background command on ubuntu@smithi114.front.sepia.ceph.com 2024-04-23T10:41:05.585 DEBUG:teuthology.run_tasks:Unwinding manager full_sequential_finally 2024-04-23T10:41:05.634 INFO:teuthology.task.full_sequential_finally:In full_sequential_finally, running task exec... 2024-04-23T10:41:05.634 INFO:teuthology.task.exec:Executing custom commands... 2024-04-23T10:41:05.634 INFO:teuthology.task.exec:Running commands on role mon.a host ubuntu@smithi114.front.sepia.ceph.com 2024-04-23T10:41:05.634 DEBUG:teuthology.orchestra.run.smithi114:> sudo TESTDIR=/home/ubuntu/cephtest bash -c 'date +%s > END_TIME' 2024-04-23T10:41:05.659 DEBUG:teuthology.orchestra.run.smithi114:> sudo TESTDIR=/home/ubuntu/cephtest bash -c 'START_TIME=$(cat START_TIME); END_TIME=$(cat END_TIME); DIFF_TIME=$((600-(END_TIME-START_TIME))); if [ "$DIFF_TIME" -gt 0 ]; then sleep "$DIFF_TIME"; fi' 2024-04-23T10:41:05.725 DEBUG:teuthology.orchestra.run.smithi114:> sudo TESTDIR=/home/ubuntu/cephtest bash -c 'ceph fs snap-schedule status --fs=cephfs --path=/' 2024-04-23T10:41:05.868 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:41:05.866+0000 7f140b47c640 1 -- 172.21.15.114:0/2078072901 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f1404101910 msgr2=0x7f1404105dd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T10:41:05.868 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:41:05.866+0000 7f140b47c640 1 --2- 172.21.15.114:0/2078072901 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f1404101910 0x7f1404105dd0 secure :-1 s=READY pgs=949 cs=0 l=1 rev1=1 crypto rx=0x7f13f4009930 tx=0x7f13f402a4d0 comp rx=0 tx=0).stop 2024-04-23T10:41:05.868 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:41:05.866+0000 7f140b47c640 1 -- 172.21.15.114:0/2078072901 shutdown_connections 2024-04-23T10:41:05.868 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:41:05.866+0000 7f140b47c640 1 --2- 172.21.15.114:0/2078072901 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f14041063f0 0x7f14041068a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:41:05.868 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:41:05.866+0000 7f140b47c640 1 --2- 172.21.15.114:0/2078072901 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f1404101910 0x7f1404105dd0 unknown :-1 s=CLOSED pgs=949 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:41:05.868 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:41:05.866+0000 7f140b47c640 1 --2- 172.21.15.114:0/2078072901 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f1404100ff0 0x7f14041013d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:41:05.869 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:41:05.866+0000 7f140b47c640 1 -- 172.21.15.114:0/2078072901 >> 172.21.15.114:0/2078072901 conn(0x7f14040771d0 msgr2=0x7f14040775e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T10:41:05.869 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:41:05.866+0000 7f140b47c640 1 -- 172.21.15.114:0/2078072901 shutdown_connections 2024-04-23T10:41:05.869 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:41:05.866+0000 7f140b47c640 1 -- 172.21.15.114:0/2078072901 wait complete. 2024-04-23T10:41:05.869 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:41:05.867+0000 7f140b47c640 1 Processor -- start 2024-04-23T10:41:05.869 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:41:05.867+0000 7f140b47c640 1 -- start start 2024-04-23T10:41:05.869 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:41:05.867+0000 7f140b47c640 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f1404100ff0 0x7f14041a4950 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T10:41:05.869 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:41:05.867+0000 7f140b47c640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f1404101910 0x7f14041a4e90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T10:41:05.869 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:41:05.867+0000 7f140b47c640 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f14041063f0 0x7f140419eb50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T10:41:05.869 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:41:05.867+0000 7f140b47c640 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x7f14041187d0 con 0x7f1404101910 2024-04-23T10:41:05.870 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:41:05.867+0000 7f140b47c640 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 -- 0x7f1404118650 con 0x7f14041063f0 2024-04-23T10:41:05.870 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:41:05.867+0000 7f140b47c640 1 -- --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_getmap magic: 0 -- 0x7f1404118950 con 0x7f1404100ff0 2024-04-23T10:41:05.870 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:41:05.867+0000 7f1408a17640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f1404101910 0x7f14041a4e90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T10:41:05.870 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:41:05.867+0000 7f1408a17640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f1404101910 0x7f14041a4e90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.114:44568/0 (socket says 172.21.15.114:44568) 2024-04-23T10:41:05.870 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:41:05.867+0000 7f1408a17640 1 -- 172.21.15.114:0/3373251331 learned_addr learned my addr 172.21.15.114:0/3373251331 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-04-23T10:41:05.870 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:41:05.867+0000 7f1409218640 1 --2- 172.21.15.114:0/3373251331 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f1404100ff0 0x7f14041a4950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T10:41:05.870 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:41:05.867+0000 7f1409a19640 1 --2- 172.21.15.114:0/3373251331 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f14041063f0 0x7f140419eb50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T10:41:05.870 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:41:05.868+0000 7f1408a17640 1 -- 172.21.15.114:0/3373251331 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f1404100ff0 msgr2=0x7f14041a4950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T10:41:05.870 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:41:05.868+0000 7f1408a17640 1 --2- 172.21.15.114:0/3373251331 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f1404100ff0 0x7f14041a4950 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:41:05.871 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:41:05.868+0000 7f1408a17640 1 -- 172.21.15.114:0/3373251331 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f14041063f0 msgr2=0x7f140419eb50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T10:41:05.871 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:41:05.868+0000 7f1408a17640 1 --2- 172.21.15.114:0/3373251331 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f14041063f0 0x7f140419eb50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:41:05.871 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:41:05.868+0000 7f1408a17640 1 -- 172.21.15.114:0/3373251331 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f140419f2d0 con 0x7f1404101910 2024-04-23T10:41:05.871 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:41:05.868+0000 7f1408a17640 1 --2- 172.21.15.114:0/3373251331 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f1404101910 0x7f14041a4e90 secure :-1 s=READY pgs=950 cs=0 l=1 rev1=1 crypto rx=0x7f13f4005940 tx=0x7f13f40042f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T10:41:05.871 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:41:05.868+0000 7f13fa7fc640 1 -- 172.21.15.114:0/3373251331 <== mon.0 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f13f403f030 con 0x7f1404101910 2024-04-23T10:41:05.871 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:41:05.868+0000 7f13fa7fc640 1 -- 172.21.15.114:0/3373251331 <== mon.0 v2:172.21.15.114:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f13f4003cb0 con 0x7f1404101910 2024-04-23T10:41:05.871 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:41:05.868+0000 7f140b47c640 1 -- 172.21.15.114:0/3373251331 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f140419f560 con 0x7f1404101910 2024-04-23T10:41:05.871 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:41:05.868+0000 7f13fa7fc640 1 -- 172.21.15.114:0/3373251331 <== mon.0 v2:172.21.15.114:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f13f4035070 con 0x7f1404101910 2024-04-23T10:41:05.871 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:41:05.868+0000 7f140b47c640 1 -- 172.21.15.114:0/3373251331 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f14041ab810 con 0x7f1404101910 2024-04-23T10:41:05.872 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:41:05.869+0000 7f140b47c640 1 -- 172.21.15.114:0/3373251331 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f140404fce0 con 0x7f1404101910 2024-04-23T10:41:05.872 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:41:05.870+0000 7f13fa7fc640 1 -- 172.21.15.114:0/3373251331 <== mon.0 v2:172.21.15.114:3300/0 4 ==== mgrmap(e 26) ==== 138778+0+0 (secure 0 0 0) 0x7f13f4042030 con 0x7f1404101910 2024-04-23T10:41:05.872 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:41:05.871+0000 7f13fa7fc640 1 --2- 172.21.15.114:0/3373251331 >> [v2:172.21.15.114:6800/1395693466,v1:172.21.15.114:6801/1395693466] conn(0x7f13f00a3dc0 0x7f13f00a6280 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T10:41:05.873 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:41:05.871+0000 7f1409218640 1 --2- 172.21.15.114:0/3373251331 >> [v2:172.21.15.114:6800/1395693466,v1:172.21.15.114:6801/1395693466] conn(0x7f13f00a3dc0 0x7f13f00a6280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T10:41:05.873 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:41:05.871+0000 7f13fa7fc640 1 -- 172.21.15.114:0/3373251331 <== mon.0 v2:172.21.15.114:3300/0 5 ==== osd_map(111..111 src has 1..111) ==== 8585+0+0 (secure 0 0 0) 0x7f13f40f1190 con 0x7f1404101910 2024-04-23T10:41:05.873 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:41:05.871+0000 7f1409218640 1 --2- 172.21.15.114:0/3373251331 >> [v2:172.21.15.114:6800/1395693466,v1:172.21.15.114:6801/1395693466] conn(0x7f13f00a3dc0 0x7f13f00a6280 secure :-1 s=READY pgs=153 cs=0 l=1 rev1=1 crypto rx=0x7f1400005d00 tx=0x7f140000b410 comp rx=0 tx=0).ready entity=mgr.15291 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T10:41:05.873 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:41:05.874+0000 7f13fa7fc640 1 -- 172.21.15.114:0/3373251331 <== mon.0 v2:172.21.15.114:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f13f40bd1a0 con 0x7f1404101910 2024-04-23T10:41:06.071 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:41:06.072+0000 7f140b47c640 1 -- 172.21.15.114:0/3373251331 --> [v2:172.21.15.114:6800/1395693466,v1:172.21.15.114:6801/1395693466] -- mgr_command(tid 0: {"prefix": "fs snap-schedule status", "fs": "cephfs", "path": "/", "target": ["mon-mgr", ""]}) -- 0x7f14041054c0 con 0x7f13f00a3dc0 2024-04-23T10:41:06.079 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:41:06.080+0000 7f13fa7fc640 1 -- 172.21.15.114:0/3373251331 <== mgr.15291 v2:172.21.15.114:6800/1395693466 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+350 (secure 0 0 0) 0x7f14041054c0 con 0x7f13f00a3dc0 2024-04-23T10:41:06.080 INFO:teuthology.orchestra.run.smithi114.stdout:{"fs": "cephfs", "subvol": null, "group": null, "path": "/", "rel_path": "/", "schedule": "1m", "retention": {"m": 6, "h": 3}, "start": "2024-04-23T00:00:00", "created": "2024-04-23T10:26:41", "first": "2024-04-23T10:27:00", "last": "2024-04-23T10:41:00", "last_pruned": "2024-04-23T10:41:00", "created_count": 15, "pruned_count": 9, "active": true} 2024-04-23T10:41:06.082 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:41:06.082+0000 7f140b47c640 1 -- 172.21.15.114:0/3373251331 >> [v2:172.21.15.114:6800/1395693466,v1:172.21.15.114:6801/1395693466] conn(0x7f13f00a3dc0 msgr2=0x7f13f00a6280 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T10:41:06.082 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:41:06.082+0000 7f140b47c640 1 --2- 172.21.15.114:0/3373251331 >> [v2:172.21.15.114:6800/1395693466,v1:172.21.15.114:6801/1395693466] conn(0x7f13f00a3dc0 0x7f13f00a6280 secure :-1 s=READY pgs=153 cs=0 l=1 rev1=1 crypto rx=0x7f1400005d00 tx=0x7f140000b410 comp rx=0 tx=0).stop 2024-04-23T10:41:06.082 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:41:06.083+0000 7f140b47c640 1 -- 172.21.15.114:0/3373251331 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f1404101910 msgr2=0x7f14041a4e90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T10:41:06.083 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:41:06.083+0000 7f140b47c640 1 --2- 172.21.15.114:0/3373251331 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f1404101910 0x7f14041a4e90 secure :-1 s=READY pgs=950 cs=0 l=1 rev1=1 crypto rx=0x7f13f4005940 tx=0x7f13f40042f0 comp rx=0 tx=0).stop 2024-04-23T10:41:06.083 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:41:06.083+0000 7f140b47c640 1 -- 172.21.15.114:0/3373251331 shutdown_connections 2024-04-23T10:41:06.083 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:41:06.083+0000 7f140b47c640 1 --2- 172.21.15.114:0/3373251331 >> [v2:172.21.15.114:6800/1395693466,v1:172.21.15.114:6801/1395693466] conn(0x7f13f00a3dc0 0x7f13f00a6280 unknown :-1 s=CLOSED pgs=153 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:41:06.083 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:41:06.083+0000 7f140b47c640 1 --2- 172.21.15.114:0/3373251331 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f14041063f0 0x7f140419eb50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:41:06.083 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:41:06.083+0000 7f140b47c640 1 --2- 172.21.15.114:0/3373251331 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f1404101910 0x7f14041a4e90 unknown :-1 s=CLOSED pgs=950 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:41:06.083 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:41:06.083+0000 7f140b47c640 1 --2- 172.21.15.114:0/3373251331 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f1404100ff0 0x7f14041a4950 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:41:06.083 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:41:06.083+0000 7f140b47c640 1 -- 172.21.15.114:0/3373251331 >> 172.21.15.114:0/3373251331 conn(0x7f14040771d0 msgr2=0x7f1404103da0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T10:41:06.084 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:41:06.083+0000 7f140b47c640 1 -- 172.21.15.114:0/3373251331 shutdown_connections 2024-04-23T10:41:06.084 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:41:06.083+0000 7f140b47c640 1 -- 172.21.15.114:0/3373251331 wait complete. 2024-04-23T10:41:06.093 DEBUG:teuthology.orchestra.run.smithi114:> sudo TESTDIR=/home/ubuntu/cephtest bash -c 'ceph fs snap-schedule list --fs=cephfs --path=/ --recursive=true' 2024-04-23T10:41:06.196 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:41:06.194+0000 7f85699e1640 1 -- 172.21.15.114:0/4199704509 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f8564101aa0 msgr2=0x7f8564105a90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T10:41:06.197 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:41:06.194+0000 7f85699e1640 1 --2- 172.21.15.114:0/4199704509 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f8564101aa0 0x7f8564105a90 secure :-1 s=READY pgs=951 cs=0 l=1 rev1=1 crypto rx=0x7f854c009900 tx=0x7f854c02a410 comp rx=0 tx=0).stop 2024-04-23T10:41:06.197 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:41:06.194+0000 7f85699e1640 1 -- 172.21.15.114:0/4199704509 shutdown_connections 2024-04-23T10:41:06.197 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:41:06.194+0000 7f85699e1640 1 --2- 172.21.15.114:0/4199704509 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f8564106170 0x7f856410dd40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:41:06.197 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:41:06.194+0000 7f85699e1640 1 --2- 172.21.15.114:0/4199704509 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f8564101aa0 0x7f8564105a90 unknown :-1 s=CLOSED pgs=951 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:41:06.197 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:41:06.194+0000 7f85699e1640 1 --2- 172.21.15.114:0/4199704509 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f85641010f0 0x7f85641014d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:41:06.197 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:41:06.194+0000 7f85699e1640 1 -- 172.21.15.114:0/4199704509 >> 172.21.15.114:0/4199704509 conn(0x7f8564077fd0 msgr2=0x7f85640783e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T10:41:06.198 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:41:06.194+0000 7f85699e1640 1 -- 172.21.15.114:0/4199704509 shutdown_connections 2024-04-23T10:41:06.198 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:41:06.194+0000 7f85699e1640 1 -- 172.21.15.114:0/4199704509 wait complete. 2024-04-23T10:41:06.198 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:41:06.195+0000 7f85699e1640 1 Processor -- start 2024-04-23T10:41:06.198 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:41:06.195+0000 7f85699e1640 1 -- start start 2024-04-23T10:41:06.198 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:41:06.195+0000 7f85699e1640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f85641010f0 0x7f8564199510 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T10:41:06.199 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:41:06.195+0000 7f85699e1640 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f8564106170 0x7f8564199a50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T10:41:06.199 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:41:06.195+0000 7f85699e1640 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f8564199fe0 0x7f85641a4590 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T10:41:06.199 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:41:06.195+0000 7f85699e1640 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x7f8564111090 con 0x7f85641010f0 2024-04-23T10:41:06.199 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:41:06.195+0000 7f85699e1640 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 -- 0x7f8564110f10 con 0x7f8564106170 2024-04-23T10:41:06.199 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:41:06.195+0000 7f85699e1640 1 -- --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_getmap magic: 0 -- 0x7f8564111210 con 0x7f8564199fe0 2024-04-23T10:41:06.199 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:41:06.195+0000 7f8562ffd640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f85641010f0 0x7f8564199510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T10:41:06.200 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:41:06.195+0000 7f8562ffd640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f85641010f0 0x7f8564199510 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.114:44582/0 (socket says 172.21.15.114:44582) 2024-04-23T10:41:06.200 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:41:06.195+0000 7f8562ffd640 1 -- 172.21.15.114:0/3488495991 learned_addr learned my addr 172.21.15.114:0/3488495991 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-04-23T10:41:06.200 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:41:06.195+0000 7f855bfff640 1 --2- 172.21.15.114:0/3488495991 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f8564106170 0x7f8564199a50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T10:41:06.200 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:41:06.195+0000 7f85637fe640 1 --2- 172.21.15.114:0/3488495991 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f8564199fe0 0x7f85641a4590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T10:41:06.200 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:41:06.196+0000 7f8562ffd640 1 -- 172.21.15.114:0/3488495991 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f8564199fe0 msgr2=0x7f85641a4590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T10:41:06.200 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:41:06.196+0000 7f8562ffd640 1 --2- 172.21.15.114:0/3488495991 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f8564199fe0 0x7f85641a4590 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:41:06.201 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:41:06.196+0000 7f8562ffd640 1 -- 172.21.15.114:0/3488495991 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f8564106170 msgr2=0x7f8564199a50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T10:41:06.201 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:41:06.196+0000 7f8562ffd640 1 --2- 172.21.15.114:0/3488495991 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f8564106170 0x7f8564199a50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:41:06.201 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:41:06.196+0000 7f8562ffd640 1 -- 172.21.15.114:0/3488495991 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f85641a4d60 con 0x7f85641010f0 2024-04-23T10:41:06.201 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:41:06.196+0000 7f8562ffd640 1 --2- 172.21.15.114:0/3488495991 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f85641010f0 0x7f8564199510 secure :-1 s=READY pgs=952 cs=0 l=1 rev1=1 crypto rx=0x7f855400d950 tx=0x7f855400de10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T10:41:06.201 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:41:06.196+0000 7f85689df640 1 -- 172.21.15.114:0/3488495991 <== mon.0 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8554017030 con 0x7f85641010f0 2024-04-23T10:41:06.201 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:41:06.196+0000 7f85699e1640 1 -- 172.21.15.114:0/3488495991 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f85641a4ff0 con 0x7f85641010f0 2024-04-23T10:41:06.202 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:41:06.196+0000 7f85689df640 1 -- 172.21.15.114:0/3488495991 <== mon.0 v2:172.21.15.114:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f8554004020 con 0x7f85641010f0 2024-04-23T10:41:06.202 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:41:06.196+0000 7f85689df640 1 -- 172.21.15.114:0/3488495991 <== mon.0 v2:172.21.15.114:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f85540047a0 con 0x7f85641010f0 2024-04-23T10:41:06.202 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:41:06.196+0000 7f85699e1640 1 -- 172.21.15.114:0/3488495991 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f85641a5510 con 0x7f85641010f0 2024-04-23T10:41:06.202 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:41:06.197+0000 7f85699e1640 1 -- 172.21.15.114:0/3488495991 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f856404fce0 con 0x7f85641010f0 2024-04-23T10:41:06.203 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:41:06.198+0000 7f85689df640 1 -- 172.21.15.114:0/3488495991 <== mon.0 v2:172.21.15.114:3300/0 4 ==== mgrmap(e 26) ==== 138778+0+0 (secure 0 0 0) 0x7f8554009a30 con 0x7f85641010f0 2024-04-23T10:41:06.204 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:41:06.199+0000 7f85689df640 1 --2- 172.21.15.114:0/3488495991 >> [v2:172.21.15.114:6800/1395693466,v1:172.21.15.114:6801/1395693466] conn(0x7f85500a3dc0 0x7f85500a6280 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T10:41:06.204 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:41:06.199+0000 7f855bfff640 1 --2- 172.21.15.114:0/3488495991 >> [v2:172.21.15.114:6800/1395693466,v1:172.21.15.114:6801/1395693466] conn(0x7f85500a3dc0 0x7f85500a6280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T10:41:06.204 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:41:06.199+0000 7f85689df640 1 -- 172.21.15.114:0/3488495991 <== mon.0 v2:172.21.15.114:3300/0 5 ==== osd_map(111..111 src has 1..111) ==== 8585+0+0 (secure 0 0 0) 0x7f85540d0830 con 0x7f85641010f0 2024-04-23T10:41:06.204 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:41:06.199+0000 7f855bfff640 1 --2- 172.21.15.114:0/3488495991 >> [v2:172.21.15.114:6800/1395693466,v1:172.21.15.114:6801/1395693466] conn(0x7f85500a3dc0 0x7f85500a6280 secure :-1 s=READY pgs=154 cs=0 l=1 rev1=1 crypto rx=0x7f854c000c00 tx=0x7f854c031400 comp rx=0 tx=0).ready entity=mgr.15291 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T10:41:06.204 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:41:06.202+0000 7f85689df640 1 -- 172.21.15.114:0/3488495991 <== mon.0 v2:172.21.15.114:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f855409ce90 con 0x7f85641010f0 2024-04-23T10:41:06.411 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:41:06.412+0000 7f85699e1640 1 -- 172.21.15.114:0/3488495991 --> [v2:172.21.15.114:6800/1395693466,v1:172.21.15.114:6801/1395693466] -- mgr_command(tid 0: {"prefix": "fs snap-schedule list", "fs": "cephfs", "path": "/", "recursive": true, "target": ["mon-mgr", ""]}) -- 0x7f8564101aa0 con 0x7f85500a3dc0 2024-04-23T10:41:06.419 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:41:06.420+0000 7f85689df640 1 -- 172.21.15.114:0/3488495991 <== mgr.15291 v2:172.21.15.114:6800/1395693466 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+10 (secure 0 0 0) 0x7f8564101aa0 con 0x7f85500a3dc0 2024-04-23T10:41:06.420 INFO:teuthology.orchestra.run.smithi114.stdout:/ 1m 6m3h 2024-04-23T10:41:06.422 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:41:06.422+0000 7f85699e1640 1 -- 172.21.15.114:0/3488495991 >> [v2:172.21.15.114:6800/1395693466,v1:172.21.15.114:6801/1395693466] conn(0x7f85500a3dc0 msgr2=0x7f85500a6280 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T10:41:06.422 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:41:06.422+0000 7f85699e1640 1 --2- 172.21.15.114:0/3488495991 >> [v2:172.21.15.114:6800/1395693466,v1:172.21.15.114:6801/1395693466] conn(0x7f85500a3dc0 0x7f85500a6280 secure :-1 s=READY pgs=154 cs=0 l=1 rev1=1 crypto rx=0x7f854c000c00 tx=0x7f854c031400 comp rx=0 tx=0).stop 2024-04-23T10:41:06.422 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:41:06.422+0000 7f85699e1640 1 -- 172.21.15.114:0/3488495991 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f85641010f0 msgr2=0x7f8564199510 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T10:41:06.423 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:41:06.422+0000 7f85699e1640 1 --2- 172.21.15.114:0/3488495991 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f85641010f0 0x7f8564199510 secure :-1 s=READY pgs=952 cs=0 l=1 rev1=1 crypto rx=0x7f855400d950 tx=0x7f855400de10 comp rx=0 tx=0).stop 2024-04-23T10:41:06.423 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:41:06.422+0000 7f85699e1640 1 -- 172.21.15.114:0/3488495991 shutdown_connections 2024-04-23T10:41:06.423 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:41:06.422+0000 7f85699e1640 1 --2- 172.21.15.114:0/3488495991 >> [v2:172.21.15.114:6800/1395693466,v1:172.21.15.114:6801/1395693466] conn(0x7f85500a3dc0 0x7f85500a6280 unknown :-1 s=CLOSED pgs=154 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:41:06.423 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:41:06.422+0000 7f85699e1640 1 --2- 172.21.15.114:0/3488495991 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f8564199fe0 0x7f85641a4590 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:41:06.423 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:41:06.422+0000 7f85699e1640 1 --2- 172.21.15.114:0/3488495991 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f8564106170 0x7f8564199a50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:41:06.423 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:41:06.422+0000 7f85699e1640 1 --2- 172.21.15.114:0/3488495991 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f85641010f0 0x7f8564199510 unknown :-1 s=CLOSED pgs=952 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:41:06.424 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:41:06.422+0000 7f85699e1640 1 -- 172.21.15.114:0/3488495991 >> 172.21.15.114:0/3488495991 conn(0x7f8564077fd0 msgr2=0x7f85640ff110 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T10:41:06.424 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:41:06.423+0000 7f85699e1640 1 -- 172.21.15.114:0/3488495991 shutdown_connections 2024-04-23T10:41:06.424 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:41:06.423+0000 7f85699e1640 1 -- 172.21.15.114:0/3488495991 wait complete. 2024-04-23T10:41:06.431 INFO:journalctl@ceph.mon.a.smithi114.stdout:Apr 23 10:41:06 smithi114 ceph-mon[29255]: pgmap v456: 97 pgs: 97 active+clean; 560 MiB data, 4.3 GiB used, 1.0 TiB / 1.0 TiB avail; 9.3 MiB/s rd, 8.9 MiB/s wr, 485 op/s 2024-04-23T10:41:06.432 INFO:journalctl@ceph.mon.a.smithi114.stdout:Apr 23 10:41:06 smithi114 ceph-mon[29255]: from='client.? 172.21.15.114:0/668925059' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-23T10:41:06.433 DEBUG:teuthology.run_tasks:Unwinding manager check-counter 2024-04-23T10:41:06.445 DEBUG:teuthology.orchestra.run.smithi114:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-04-23T10:41:06.492 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 23 10:41:06 smithi119 ceph-mon[32608]: pgmap v456: 97 pgs: 97 active+clean; 560 MiB data, 4.3 GiB used, 1.0 TiB / 1.0 TiB avail; 9.3 MiB/s rd, 8.9 MiB/s wr, 485 op/s 2024-04-23T10:41:06.492 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 23 10:41:06 smithi119 ceph-mon[32608]: from='client.? 172.21.15.114:0/668925059' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-23T10:41:06.549 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:41:06.546+0000 7fd254922640 1 -- 172.21.15.114:0/3479762365 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fd250106170 msgr2=0x7fd25010dd40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T10:41:06.549 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:41:06.546+0000 7fd254922640 1 --2- 172.21.15.114:0/3479762365 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fd250106170 0x7fd25010dd40 secure :-1 s=READY pgs=953 cs=0 l=1 rev1=1 crypto rx=0x7fd248009ca0 tx=0x7fd24802a8c0 comp rx=0 tx=0).stop 2024-04-23T10:41:06.550 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:41:06.546+0000 7fd254922640 1 -- 172.21.15.114:0/3479762365 shutdown_connections 2024-04-23T10:41:06.550 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:41:06.546+0000 7fd254922640 1 --2- 172.21.15.114:0/3479762365 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fd250106170 0x7fd25010dd40 unknown :-1 s=CLOSED pgs=953 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:41:06.550 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:41:06.546+0000 7fd254922640 1 --2- 172.21.15.114:0/3479762365 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fd250101a50 0x7fd250105a90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:41:06.550 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:41:06.546+0000 7fd254922640 1 --2- 172.21.15.114:0/3479762365 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fd2501010a0 0x7fd250101480 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:41:06.550 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:41:06.546+0000 7fd254922640 1 -- 172.21.15.114:0/3479762365 >> 172.21.15.114:0/3479762365 conn(0x7fd250077f50 msgr2=0x7fd250078360 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T10:41:06.551 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:41:06.546+0000 7fd254922640 1 -- 172.21.15.114:0/3479762365 shutdown_connections 2024-04-23T10:41:06.551 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:41:06.546+0000 7fd254922640 1 -- 172.21.15.114:0/3479762365 wait complete. 2024-04-23T10:41:06.551 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:41:06.547+0000 7fd254922640 1 Processor -- start 2024-04-23T10:41:06.551 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:41:06.547+0000 7fd254922640 1 -- start start 2024-04-23T10:41:06.551 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:41:06.547+0000 7fd254922640 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fd2501010a0 0x7fd250199540 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T10:41:06.551 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:41:06.547+0000 7fd254922640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fd250106170 0x7fd250199a80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T10:41:06.552 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:41:06.547+0000 7fd254922640 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fd25019a060 0x7fd2501a45c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T10:41:06.552 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:41:06.547+0000 7fd254922640 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x7fd2501110e0 con 0x7fd250106170 2024-04-23T10:41:06.552 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:41:06.547+0000 7fd254922640 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 -- 0x7fd250110f60 con 0x7fd2501010a0 2024-04-23T10:41:06.552 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:41:06.547+0000 7fd254922640 1 -- --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_getmap magic: 0 -- 0x7fd250111260 con 0x7fd25019a060 2024-04-23T10:41:06.552 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:41:06.547+0000 7fd247fff640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fd250106170 0x7fd250199a80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T10:41:06.552 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:41:06.547+0000 7fd247fff640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fd250106170 0x7fd250199a80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.114:44602/0 (socket says 172.21.15.114:44602) 2024-04-23T10:41:06.553 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:41:06.547+0000 7fd247fff640 1 -- 172.21.15.114:0/1698357046 learned_addr learned my addr 172.21.15.114:0/1698357046 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-04-23T10:41:06.553 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:41:06.547+0000 7fd24e59c640 1 --2- 172.21.15.114:0/1698357046 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fd2501010a0 0x7fd250199540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T10:41:06.553 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:41:06.547+0000 7fd24ed9d640 1 --2- 172.21.15.114:0/1698357046 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fd25019a060 0x7fd2501a45c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T10:41:06.553 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:41:06.548+0000 7fd247fff640 1 -- 172.21.15.114:0/1698357046 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fd25019a060 msgr2=0x7fd2501a45c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T10:41:06.553 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:41:06.548+0000 7fd247fff640 1 --2- 172.21.15.114:0/1698357046 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fd25019a060 0x7fd2501a45c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:41:06.553 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:41:06.548+0000 7fd247fff640 1 -- 172.21.15.114:0/1698357046 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fd2501010a0 msgr2=0x7fd250199540 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T10:41:06.554 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:41:06.548+0000 7fd247fff640 1 --2- 172.21.15.114:0/1698357046 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fd2501010a0 0x7fd250199540 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:41:06.554 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:41:06.548+0000 7fd247fff640 1 -- 172.21.15.114:0/1698357046 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd2501a4de0 con 0x7fd250106170 2024-04-23T10:41:06.554 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:41:06.548+0000 7fd247fff640 1 --2- 172.21.15.114:0/1698357046 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fd250106170 0x7fd250199a80 secure :-1 s=READY pgs=954 cs=0 l=1 rev1=1 crypto rx=0x7fd23800d740 tx=0x7fd23800dc00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T10:41:06.554 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:41:06.548+0000 7fd2477fe640 1 -- 172.21.15.114:0/1698357046 <== mon.0 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd238017070 con 0x7fd250106170 2024-04-23T10:41:06.554 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:41:06.548+0000 7fd2477fe640 1 -- 172.21.15.114:0/1698357046 <== mon.0 v2:172.21.15.114:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fd23800de50 con 0x7fd250106170 2024-04-23T10:41:06.554 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:41:06.548+0000 7fd2477fe640 1 -- 172.21.15.114:0/1698357046 <== mon.0 v2:172.21.15.114:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd238004ce0 con 0x7fd250106170 2024-04-23T10:41:06.555 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:41:06.548+0000 7fd254922640 1 -- 172.21.15.114:0/1698357046 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fd2501a50d0 con 0x7fd250106170 2024-04-23T10:41:06.555 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:41:06.548+0000 7fd254922640 1 -- 172.21.15.114:0/1698357046 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fd2501a55d0 con 0x7fd250106170 2024-04-23T10:41:06.555 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:41:06.549+0000 7fd254922640 1 -- 172.21.15.114:0/1698357046 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fd250068d70 con 0x7fd250106170 2024-04-23T10:41:06.555 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:41:06.550+0000 7fd2477fe640 1 -- 172.21.15.114:0/1698357046 <== mon.0 v2:172.21.15.114:3300/0 4 ==== mgrmap(e 26) ==== 138778+0+0 (secure 0 0 0) 0x7fd238004e80 con 0x7fd250106170 2024-04-23T10:41:06.555 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:41:06.550+0000 7fd2477fe640 1 --2- 172.21.15.114:0/1698357046 >> [v2:172.21.15.114:6800/1395693466,v1:172.21.15.114:6801/1395693466] conn(0x7fd23c0a3d20 0x7fd23c0a61e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T10:41:06.555 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:41:06.551+0000 7fd24e59c640 1 --2- 172.21.15.114:0/1698357046 >> [v2:172.21.15.114:6800/1395693466,v1:172.21.15.114:6801/1395693466] conn(0x7fd23c0a3d20 0x7fd23c0a61e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T10:41:06.556 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:41:06.551+0000 7fd2477fe640 1 -- 172.21.15.114:0/1698357046 <== mon.0 v2:172.21.15.114:3300/0 5 ==== osd_map(111..111 src has 1..111) ==== 8585+0+0 (secure 0 0 0) 0x7fd2380d0170 con 0x7fd250106170 2024-04-23T10:41:06.556 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:41:06.551+0000 7fd24e59c640 1 --2- 172.21.15.114:0/1698357046 >> [v2:172.21.15.114:6800/1395693466,v1:172.21.15.114:6801/1395693466] conn(0x7fd23c0a3d20 0x7fd23c0a61e0 secure :-1 s=READY pgs=155 cs=0 l=1 rev1=1 crypto rx=0x7fd240009bd0 tx=0x7fd240005f90 comp rx=0 tx=0).ready entity=mgr.15291 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T10:41:06.557 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:41:06.555+0000 7fd2477fe640 1 -- 172.21.15.114:0/1698357046 <== mon.0 v2:172.21.15.114:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7fd23809c7d0 con 0x7fd250106170 2024-04-23T10:41:06.578 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 23 10:41:06 smithi204 ceph-mon[32649]: pgmap v456: 97 pgs: 97 active+clean; 560 MiB data, 4.3 GiB used, 1.0 TiB / 1.0 TiB avail; 9.3 MiB/s rd, 8.9 MiB/s wr, 485 op/s 2024-04-23T10:41:06.578 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 23 10:41:06 smithi204 ceph-mon[32649]: from='client.? 172.21.15.114:0/668925059' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-23T10:41:06.732 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:41:06.733+0000 7fd254922640 1 -- 172.21.15.114:0/1698357046 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7fd250109400 con 0x7fd250106170 2024-04-23T10:41:06.733 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:41:06.734+0000 7fd2477fe640 1 -- 172.21.15.114:0/1698357046 <== mon.0 v2:172.21.15.114:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v111) ==== 74+0+23318 (secure 0 0 0) 0x7fd23809c0d0 con 0x7fd250106170 2024-04-23T10:41:06.734 INFO:teuthology.orchestra.run.smithi114.stdout: 2024-04-23T10:41:06.734 INFO:teuthology.orchestra.run.smithi114.stdout:{"epoch":111,"fsid":"aa240572-0159-11ef-bc93-c7b262605968","created":"2024-04-23T10:16:53.351136+0000","modified":"2024-04-23T10:40:09.399152+0000","last_up_change":"2024-04-23T10:23:54.153269+0000","last_in_change":"2024-04-23T10:23:32.963085+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":26,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-04-23T10:20:11.970297+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":12,"score_stable":12,"optimal_score":0.25,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-04-23T10:24:41.218540+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"86","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.8799999952316284,"score_stable":1.8799999952316284,"optimal_score":1,"raw_score_acting":1.8799999952316284,"raw_score_stable":1.8799999952316284,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-04-23T10:24:42.341799+0000","flags":8193,"flags_names":"hashpspool,selfmanaged_snaps","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"109","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":23,"snap_epoch":109,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"fd64adde-9445-45a4-a171-ef656e2c5709","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":88,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6802","nonce":736366878},{"type":"v1","addr":"172.21.15.114:6803","nonce":736366878}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6804","nonce":736366878},{"type":"v1","addr":"172.21.15.114:6805","nonce":736366878}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6808","nonce":736366878},{"type":"v1","addr":"172.21.15.114:6809","nonce":736366878}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6806","nonce":736366878},{"type":"v1","addr":"172.21.15.114:6807","nonce":736366878}]},"public_addr":"172.21.15.114:6803/736366878","cluster_addr":"172.21.15.114:6805/736366878","heartbeat_back_addr":"172.21.15.114:6809/736366878","heartbeat_front_addr":"172.21.15.114:6807/736366878","state":["exists","up"]},{"osd":1,"uuid":"e279a1e6-2702-472a-930a-e08ece76a861","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":85,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6800","nonce":1133540779},{"type":"v1","addr":"172.21.15.119:6801","nonce":1133540779}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6802","nonce":1133540779},{"type":"v1","addr":"172.21.15.119:6803","nonce":1133540779}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6806","nonce":1133540779},{"type":"v1","addr":"172.21.15.119:6807","nonce":1133540779}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6804","nonce":1133540779},{"type":"v1","addr":"172.21.15.119:6805","nonce":1133540779}]},"public_addr":"172.21.15.119:6801/1133540779","cluster_addr":"172.21.15.119:6803/1133540779","heartbeat_back_addr":"172.21.15.119:6807/1133540779","heartbeat_front_addr":"172.21.15.119:6805/1133540779","state":["exists","up"]},{"osd":2,"uuid":"f4c9f2b4-ff49-4e14-89cd-b477feeb9382","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":85,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6800","nonce":799652511},{"type":"v1","addr":"172.21.15.204:6801","nonce":799652511}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6802","nonce":799652511},{"type":"v1","addr":"172.21.15.204:6803","nonce":799652511}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6806","nonce":799652511},{"type":"v1","addr":"172.21.15.204:6807","nonce":799652511}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6804","nonce":799652511},{"type":"v1","addr":"172.21.15.204:6805","nonce":799652511}]},"public_addr":"172.21.15.204:6801/799652511","cluster_addr":"172.21.15.204:6803/799652511","heartbeat_back_addr":"172.21.15.204:6807/799652511","heartbeat_front_addr":"172.21.15.204:6805/799652511","state":["exists","up"]},{"osd":3,"uuid":"52d2a566-e71c-4bb3-8d6c-5abdd43177c2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":25,"up_thru":88,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6810","nonce":4275595091},{"type":"v1","addr":"172.21.15.114:6811","nonce":4275595091}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6812","nonce":4275595091},{"type":"v1","addr":"172.21.15.114:6813","nonce":4275595091}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6816","nonce":4275595091},{"type":"v1","addr":"172.21.15.114:6817","nonce":4275595091}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6814","nonce":4275595091},{"type":"v1","addr":"172.21.15.114:6815","nonce":4275595091}]},"public_addr":"172.21.15.114:6811/4275595091","cluster_addr":"172.21.15.114:6813/4275595091","heartbeat_back_addr":"172.21.15.114:6817/4275595091","heartbeat_front_addr":"172.21.15.114:6815/4275595091","state":["exists","up"]},{"osd":4,"uuid":"d911a7c1-4bdc-4d28-b8a6-73896a6e7de6","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":33,"up_thru":85,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6808","nonce":1611007172},{"type":"v1","addr":"172.21.15.119:6809","nonce":1611007172}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6810","nonce":1611007172},{"type":"v1","addr":"172.21.15.119:6811","nonce":1611007172}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6814","nonce":1611007172},{"type":"v1","addr":"172.21.15.119:6815","nonce":1611007172}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6812","nonce":1611007172},{"type":"v1","addr":"172.21.15.119:6813","nonce":1611007172}]},"public_addr":"172.21.15.119:6809/1611007172","cluster_addr":"172.21.15.119:6811/1611007172","heartbeat_back_addr":"172.21.15.119:6815/1611007172","heartbeat_front_addr":"172.21.15.119:6813/1611007172","state":["exists","up"]},{"osd":5,"uuid":"f7590e1a-e93c-4c29-b15f-38ee52d1f921","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":39,"up_thru":85,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6808","nonce":1427563312},{"type":"v1","addr":"172.21.15.204:6809","nonce":1427563312}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6810","nonce":1427563312},{"type":"v1","addr":"172.21.15.204:6811","nonce":1427563312}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6814","nonce":1427563312},{"type":"v1","addr":"172.21.15.204:6815","nonce":1427563312}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6812","nonce":1427563312},{"type":"v1","addr":"172.21.15.204:6813","nonce":1427563312}]},"public_addr":"172.21.15.204:6809/1427563312","cluster_addr":"172.21.15.204:6811/1427563312","heartbeat_back_addr":"172.21.15.204:6815/1427563312","heartbeat_front_addr":"172.21.15.204:6813/1427563312","state":["exists","up"]},{"osd":6,"uuid":"534d20e8-3e54-4db8-a845-6119f49c7a78","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":47,"up_thru":88,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6818","nonce":4072912544},{"type":"v1","addr":"172.21.15.114:6819","nonce":4072912544}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6820","nonce":4072912544},{"type":"v1","addr":"172.21.15.114:6821","nonce":4072912544}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6824","nonce":4072912544},{"type":"v1","addr":"172.21.15.114:6825","nonce":4072912544}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6822","nonce":4072912544},{"type":"v1","addr":"172.21.15.114:6823","nonce":4072912544}]},"public_addr":"172.21.15.114:6819/4072912544","cluster_addr":"172.21.15.114:6821/4072912544","heartbeat_back_addr":"172.21.15.114:6825/4072912544","heartbeat_front_addr":"172.21.15.114:6823/4072912544","state":["exists","up"]},{"osd":7,"uuid":"f0ec22b8-1b48-4859-9d65-b00403aff3ad","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":53,"up_thru":88,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6816","nonce":2237458793},{"type":"v1","addr":"172.21.15.119:6817","nonce":2237458793}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6818","nonce":2237458793},{"type":"v1","addr":"172.21.15.119:6819","nonce":2237458793}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6822","nonce":2237458793},{"type":"v1","addr":"172.21.15.119:6823","nonce":2237458793}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6820","nonce":2237458793},{"type":"v1","addr":"172.21.15.119:6821","nonce":2237458793}]},"public_addr":"172.21.15.119:6817/2237458793","cluster_addr":"172.21.15.119:6819/2237458793","heartbeat_back_addr":"172.21.15.119:6823/2237458793","heartbeat_front_addr":"172.21.15.119:6821/2237458793","state":["exists","up"]},{"osd":8,"uuid":"3ac89686-6d76-4cef-9089-ac29e4b3bcac","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":62,"up_thru":91,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6816","nonce":1426318231},{"type":"v1","addr":"172.21.15.204:6817","nonce":1426318231}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6818","nonce":1426318231},{"type":"v1","addr":"172.21.15.204:6819","nonce":1426318231}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6822","nonce":1426318231},{"type":"v1","addr":"172.21.15.204:6823","nonce":1426318231}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6820","nonce":1426318231},{"type":"v1","addr":"172.21.15.204:6821","nonce":1426318231}]},"public_addr":"172.21.15.204:6817/1426318231","cluster_addr":"172.21.15.204:6819/1426318231","heartbeat_back_addr":"172.21.15.204:6823/1426318231","heartbeat_front_addr":"172.21.15.204:6821/1426318231","state":["exists","up"]},{"osd":9,"uuid":"a80ffe4d-1e98-460f-b046-c4754285cc5a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":69,"up_thru":88,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6826","nonce":2750635721},{"type":"v1","addr":"172.21.15.114:6827","nonce":2750635721}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6828","nonce":2750635721},{"type":"v1","addr":"172.21.15.114:6829","nonce":2750635721}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6832","nonce":2750635721},{"type":"v1","addr":"172.21.15.114:6833","nonce":2750635721}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6830","nonce":2750635721},{"type":"v1","addr":"172.21.15.114:6831","nonce":2750635721}]},"public_addr":"172.21.15.114:6827/2750635721","cluster_addr":"172.21.15.114:6829/2750635721","heartbeat_back_addr":"172.21.15.114:6833/2750635721","heartbeat_front_addr":"172.21.15.114:6831/2750635721","state":["exists","up"]},{"osd":10,"uuid":"cd75a579-bf12-4600-ac09-a8bfce772ced","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":74,"up_thru":85,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6824","nonce":1400187701},{"type":"v1","addr":"172.21.15.119:6825","nonce":1400187701}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6826","nonce":1400187701},{"type":"v1","addr":"172.21.15.119:6827","nonce":1400187701}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6830","nonce":1400187701},{"type":"v1","addr":"172.21.15.119:6831","nonce":1400187701}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6828","nonce":1400187701},{"type":"v1","addr":"172.21.15.119:6829","nonce":1400187701}]},"public_addr":"172.21.15.119:6825/1400187701","cluster_addr":"172.21.15.119:6827/1400187701","heartbeat_back_addr":"172.21.15.119:6831/1400187701","heartbeat_front_addr":"172.21.15.119:6829/1400187701","state":["exists","up"]},{"osd":11,"uuid":"53f037a5-3ec0-483a-80a1-dfee923a68e5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":82,"up_thru":91,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6824","nonce":2407298490},{"type":"v1","addr":"172.21.15.204:6825","nonce":2407298490}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6826","nonce":2407298490},{"type":"v1","addr":"172.21.15.204:6827","nonce":2407298490}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6830","nonce":2407298490},{"type":"v1","addr":"172.21.15.204:6831","nonce":2407298490}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6828","nonce":2407298490},{"type":"v1","addr":"172.21.15.204:6829","nonce":2407298490}]},"public_addr":"172.21.15.204:6825/2407298490","cluster_addr":"172.21.15.204:6827/2407298490","heartbeat_back_addr":"172.21.15.204:6831/2407298490","heartbeat_front_addr":"172.21.15.204:6829/2407298490","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-23T10:19:18.401147+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-23T10:19:42.611299+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-23T10:20:07.277367+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-23T10:20:31.208368+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-23T10:20:55.866225+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-23T10:21:19.961237+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-23T10:21:44.480783+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-23T10:22:09.173641+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-23T10:22:34.030328+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-23T10:22:59.652293+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-23T10:23:25.659673+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-23T10:23:51.168171+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"2.b","mappings":[{"from":8,"to":11}]},{"pgid":"2.e","mappings":[{"from":2,"to":4}]},{"pgid":"2.10","mappings":[{"from":2,"to":1}]},{"pgid":"2.1d","mappings":[{"from":8,"to":11}]},{"pgid":"3.e","mappings":[{"from":5,"to":9}]},{"pgid":"3.12","mappings":[{"from":3,"to":9}]},{"pgid":"3.17","mappings":[{"from":4,"to":6}]},{"pgid":"3.23","mappings":[{"from":4,"to":7}]},{"pgid":"3.27","mappings":[{"from":4,"to":6}]},{"pgid":"3.31","mappings":[{"from":4,"to":8}]},{"pgid":"3.38","mappings":[{"from":3,"to":9}]},{"pgid":"3.39","mappings":[{"from":5,"to":8}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.114:0/480720790":"2024-04-24T10:26:40.272662+0000","172.21.15.114:6801/465333808":"2024-04-24T10:26:40.272662+0000","172.21.15.114:6800/465333808":"2024-04-24T10:26:40.272662+0000","172.21.15.114:0/1787801099":"2024-04-24T10:26:40.272662+0000","172.21.15.114:6801/1604678557":"2024-04-24T10:17:45.874609+0000","172.21.15.114:0/4025385854":"2024-04-24T10:17:45.874609+0000","172.21.15.114:0/2228273469":"2024-04-24T10:17:19.910348+0000","172.21.15.114:0/2713201622":"2024-04-24T10:26:40.272662+0000","172.21.15.114:6800/2831530838":"2024-04-24T10:17:19.910348+0000","172.21.15.114:0/637004095":"2024-04-24T10:26:40.272662+0000","172.21.15.114:0/1333447450":"2024-04-24T10:17:45.874609+0000","172.21.15.114:0/2233026840":"2024-04-24T10:17:19.910348+0000","172.21.15.114:0/1902281545":"2024-04-24T10:17:19.910348+0000","172.21.15.114:6801/2831530838":"2024-04-24T10:17:19.910348+0000","172.21.15.114:6800/1604678557":"2024-04-24T10:17:45.874609+0000","172.21.15.114:0/1623553811":"2024-04-24T10:17:45.874609+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-23T10:41:06.736 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:41:06.736+0000 7fd254922640 1 -- 172.21.15.114:0/1698357046 >> [v2:172.21.15.114:6800/1395693466,v1:172.21.15.114:6801/1395693466] conn(0x7fd23c0a3d20 msgr2=0x7fd23c0a61e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T10:41:06.736 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:41:06.736+0000 7fd254922640 1 --2- 172.21.15.114:0/1698357046 >> [v2:172.21.15.114:6800/1395693466,v1:172.21.15.114:6801/1395693466] conn(0x7fd23c0a3d20 0x7fd23c0a61e0 secure :-1 s=READY pgs=155 cs=0 l=1 rev1=1 crypto rx=0x7fd240009bd0 tx=0x7fd240005f90 comp rx=0 tx=0).stop 2024-04-23T10:41:06.736 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:41:06.736+0000 7fd254922640 1 -- 172.21.15.114:0/1698357046 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fd250106170 msgr2=0x7fd250199a80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T10:41:06.736 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:41:06.736+0000 7fd254922640 1 --2- 172.21.15.114:0/1698357046 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fd250106170 0x7fd250199a80 secure :-1 s=READY pgs=954 cs=0 l=1 rev1=1 crypto rx=0x7fd23800d740 tx=0x7fd23800dc00 comp rx=0 tx=0).stop 2024-04-23T10:41:06.736 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:41:06.736+0000 7fd254922640 1 -- 172.21.15.114:0/1698357046 shutdown_connections 2024-04-23T10:41:06.736 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:41:06.736+0000 7fd254922640 1 --2- 172.21.15.114:0/1698357046 >> [v2:172.21.15.114:6800/1395693466,v1:172.21.15.114:6801/1395693466] conn(0x7fd23c0a3d20 0x7fd23c0a61e0 unknown :-1 s=CLOSED pgs=155 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:41:06.737 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:41:06.736+0000 7fd254922640 1 --2- 172.21.15.114:0/1698357046 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fd25019a060 0x7fd2501a45c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:41:06.737 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:41:06.736+0000 7fd254922640 1 --2- 172.21.15.114:0/1698357046 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fd250106170 0x7fd250199a80 unknown :-1 s=CLOSED pgs=954 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:41:06.737 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:41:06.736+0000 7fd254922640 1 --2- 172.21.15.114:0/1698357046 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fd2501010a0 0x7fd250199540 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:41:06.737 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:41:06.736+0000 7fd254922640 1 -- 172.21.15.114:0/1698357046 >> 172.21.15.114:0/1698357046 conn(0x7fd250077f50 msgr2=0x7fd2500fef30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T10:41:06.737 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:41:06.737+0000 7fd254922640 1 -- 172.21.15.114:0/1698357046 shutdown_connections 2024-04-23T10:41:06.737 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:41:06.737+0000 7fd254922640 1 -- 172.21.15.114:0/1698357046 wait complete. 2024-04-23T10:41:06.750 INFO:tasks.check_counter.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-04-23T10:20:11.970297+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '22', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 12, 'score_stable': 12, 'optimal_score': 0.25, 'raw_score_acting': 3, 'raw_score_stable': 3, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 2, 'pool_name': 'cephfs_metadata', 'create_time': '2024-04-23T10:24:41.218540+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '86', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.8799999952316284, 'score_stable': 1.8799999952316284, 'optimal_score': 1, 'raw_score_acting': 1.8799999952316284, 'raw_score_stable': 1.8799999952316284, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 3, 'pool_name': 'cephfs_data', 'create_time': '2024-04-23T10:24:42.341799+0000', 'flags': 8193, 'flags_names': 'hashpspool,selfmanaged_snaps', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '109', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 23, 'snap_epoch': 109, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-04-23T10:41:06.751 DEBUG:teuthology.orchestra.run.smithi114:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-04-23T10:41:06.856 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:41:06.854+0000 7fd54b9ad640 1 -- 172.21.15.114:0/1788508973 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fd544107260 msgr2=0x7fd54410ee30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T10:41:06.856 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:41:06.854+0000 7fd54b9ad640 1 --2- 172.21.15.114:0/1788508973 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fd544107260 0x7fd54410ee30 secure :-1 s=READY pgs=955 cs=0 l=1 rev1=1 crypto rx=0x7fd53c009ca0 tx=0x7fd53c02a8c0 comp rx=0 tx=0).stop 2024-04-23T10:41:06.856 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:41:06.854+0000 7fd54b9ad640 1 -- 172.21.15.114:0/1788508973 shutdown_connections 2024-04-23T10:41:06.857 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:41:06.854+0000 7fd54b9ad640 1 --2- 172.21.15.114:0/1788508973 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fd544107260 0x7fd54410ee30 unknown :-1 s=CLOSED pgs=955 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:41:06.857 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:41:06.854+0000 7fd54b9ad640 1 --2- 172.21.15.114:0/1788508973 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fd544102b40 0x7fd544106b80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:41:06.857 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:41:06.854+0000 7fd54b9ad640 1 --2- 172.21.15.114:0/1788508973 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fd544102190 0x7fd544102570 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:41:06.857 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:41:06.854+0000 7fd54b9ad640 1 -- 172.21.15.114:0/1788508973 >> 172.21.15.114:0/1788508973 conn(0x7fd5440793a0 msgr2=0x7fd5440797b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T10:41:06.857 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:41:06.854+0000 7fd54b9ad640 1 -- 172.21.15.114:0/1788508973 shutdown_connections 2024-04-23T10:41:06.857 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:41:06.854+0000 7fd54b9ad640 1 -- 172.21.15.114:0/1788508973 wait complete. 2024-04-23T10:41:06.858 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:41:06.854+0000 7fd54b9ad640 1 Processor -- start 2024-04-23T10:41:06.858 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:41:06.854+0000 7fd54b9ad640 1 -- start start 2024-04-23T10:41:06.858 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:41:06.855+0000 7fd54b9ad640 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fd544102190 0x7fd5441986b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T10:41:06.858 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:41:06.855+0000 7fd54b9ad640 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fd544107260 0x7fd544198bf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T10:41:06.858 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:41:06.855+0000 7fd54b9ad640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fd54419cf70 0x7fd54419d420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T10:41:06.858 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:41:06.855+0000 7fd54b9ad640 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x7fd5441121d0 con 0x7fd54419cf70 2024-04-23T10:41:06.858 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:41:06.855+0000 7fd54b9ad640 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 -- 0x7fd544112050 con 0x7fd544102190 2024-04-23T10:41:06.859 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:41:06.855+0000 7fd54b9ad640 1 -- --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_getmap magic: 0 -- 0x7fd544112350 con 0x7fd544107260 2024-04-23T10:41:06.859 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:41:06.855+0000 7fd549f4a640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fd54419cf70 0x7fd54419d420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T10:41:06.859 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:41:06.855+0000 7fd549f4a640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fd54419cf70 0x7fd54419d420 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.114:44608/0 (socket says 172.21.15.114:44608) 2024-04-23T10:41:06.859 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:41:06.855+0000 7fd549f4a640 1 -- 172.21.15.114:0/3276108747 learned_addr learned my addr 172.21.15.114:0/3276108747 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-04-23T10:41:06.859 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:41:06.855+0000 7fd549749640 1 --2- 172.21.15.114:0/3276108747 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fd544102190 0x7fd5441986b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T10:41:06.859 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:41:06.855+0000 7fd549f4a640 1 -- 172.21.15.114:0/3276108747 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fd544107260 msgr2=0x7fd544198bf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T10:41:06.860 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:41:06.855+0000 7fd549f4a640 1 --2- 172.21.15.114:0/3276108747 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fd544107260 0x7fd544198bf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:41:06.860 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:41:06.855+0000 7fd549f4a640 1 -- 172.21.15.114:0/3276108747 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fd544102190 msgr2=0x7fd5441986b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T10:41:06.860 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:41:06.855+0000 7fd549f4a640 1 --2- 172.21.15.114:0/3276108747 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fd544102190 0x7fd5441986b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:41:06.860 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:41:06.855+0000 7fd549f4a640 1 -- 172.21.15.114:0/3276108747 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd54419da60 con 0x7fd54419cf70 2024-04-23T10:41:06.860 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:41:06.855+0000 7fd549f4a640 1 --2- 172.21.15.114:0/3276108747 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fd54419cf70 0x7fd54419d420 secure :-1 s=READY pgs=956 cs=0 l=1 rev1=1 crypto rx=0x7fd53c005cc0 tx=0x7fd53c02f380 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T10:41:06.860 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:41:06.855+0000 7fd53a7fc640 1 -- 172.21.15.114:0/3276108747 <== mon.0 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd53c003bf0 con 0x7fd54419cf70 2024-04-23T10:41:06.861 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:41:06.855+0000 7fd53a7fc640 1 -- 172.21.15.114:0/3276108747 <== mon.0 v2:172.21.15.114:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fd53c003d90 con 0x7fd54419cf70 2024-04-23T10:41:06.861 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:41:06.855+0000 7fd54b9ad640 1 -- 172.21.15.114:0/3276108747 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fd54419dcf0 con 0x7fd54419cf70 2024-04-23T10:41:06.861 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:41:06.855+0000 7fd53a7fc640 1 -- 172.21.15.114:0/3276108747 <== mon.0 v2:172.21.15.114:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd53c02fbc0 con 0x7fd54419cf70 2024-04-23T10:41:06.861 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:41:06.855+0000 7fd54b9ad640 1 -- 172.21.15.114:0/3276108747 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fd5441a5510 con 0x7fd54419cf70 2024-04-23T10:41:06.861 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:41:06.856+0000 7fd54b9ad640 1 -- 172.21.15.114:0/3276108747 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fd5440691c0 con 0x7fd54419cf70 2024-04-23T10:41:06.861 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:41:06.857+0000 7fd53a7fc640 1 -- 172.21.15.114:0/3276108747 <== mon.0 v2:172.21.15.114:3300/0 4 ==== mgrmap(e 26) ==== 138778+0+0 (secure 0 0 0) 0x7fd53c02fd60 con 0x7fd54419cf70 2024-04-23T10:41:06.862 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:41:06.858+0000 7fd53a7fc640 1 --2- 172.21.15.114:0/3276108747 >> [v2:172.21.15.114:6800/1395693466,v1:172.21.15.114:6801/1395693466] conn(0x7fd5300a3d20 0x7fd5300a61e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T10:41:06.863 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:41:06.858+0000 7fd549749640 1 --2- 172.21.15.114:0/3276108747 >> [v2:172.21.15.114:6800/1395693466,v1:172.21.15.114:6801/1395693466] conn(0x7fd5300a3d20 0x7fd5300a61e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T10:41:06.863 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:41:06.858+0000 7fd53a7fc640 1 -- 172.21.15.114:0/3276108747 <== mon.0 v2:172.21.15.114:3300/0 5 ==== osd_map(111..111 src has 1..111) ==== 8585+0+0 (secure 0 0 0) 0x7fd53c0f1e50 con 0x7fd54419cf70 2024-04-23T10:41:06.863 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:41:06.858+0000 7fd549749640 1 --2- 172.21.15.114:0/3276108747 >> [v2:172.21.15.114:6800/1395693466,v1:172.21.15.114:6801/1395693466] conn(0x7fd5300a3d20 0x7fd5300a61e0 secure :-1 s=READY pgs=156 cs=0 l=1 rev1=1 crypto rx=0x7fd534005fd0 tx=0x7fd53400b390 comp rx=0 tx=0).ready entity=mgr.15291 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T10:41:06.863 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:41:06.862+0000 7fd53a7fc640 1 -- 172.21.15.114:0/3276108747 <== mon.0 v2:172.21.15.114:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7fd53c004610 con 0x7fd54419cf70 2024-04-23T10:41:07.039 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:41:07.040+0000 7fd54b9ad640 1 -- 172.21.15.114:0/3276108747 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) -- 0x7fd544102b40 con 0x7fd54419cf70 2024-04-23T10:41:07.039 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:41:07.040+0000 7fd53a7fc640 1 -- 172.21.15.114:0/3276108747 <== mon.0 v2:172.21.15.114:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v111) ==== 93+0+10 (secure 0 0 0) 0x7fd53c0098b0 con 0x7fd54419cf70 2024-04-23T10:41:07.040 INFO:teuthology.orchestra.run.smithi114.stdout:pg_num: 1 2024-04-23T10:41:07.042 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:41:07.043+0000 7fd54b9ad640 1 -- 172.21.15.114:0/3276108747 >> [v2:172.21.15.114:6800/1395693466,v1:172.21.15.114:6801/1395693466] conn(0x7fd5300a3d20 msgr2=0x7fd5300a61e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T10:41:07.042 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:41:07.043+0000 7fd54b9ad640 1 --2- 172.21.15.114:0/3276108747 >> [v2:172.21.15.114:6800/1395693466,v1:172.21.15.114:6801/1395693466] conn(0x7fd5300a3d20 0x7fd5300a61e0 secure :-1 s=READY pgs=156 cs=0 l=1 rev1=1 crypto rx=0x7fd534005fd0 tx=0x7fd53400b390 comp rx=0 tx=0).stop 2024-04-23T10:41:07.042 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:41:07.043+0000 7fd54b9ad640 1 -- 172.21.15.114:0/3276108747 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fd54419cf70 msgr2=0x7fd54419d420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T10:41:07.043 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:41:07.043+0000 7fd54b9ad640 1 --2- 172.21.15.114:0/3276108747 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fd54419cf70 0x7fd54419d420 secure :-1 s=READY pgs=956 cs=0 l=1 rev1=1 crypto rx=0x7fd53c005cc0 tx=0x7fd53c02f380 comp rx=0 tx=0).stop 2024-04-23T10:41:07.043 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:41:07.043+0000 7fd54b9ad640 1 -- 172.21.15.114:0/3276108747 shutdown_connections 2024-04-23T10:41:07.043 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:41:07.043+0000 7fd54b9ad640 1 --2- 172.21.15.114:0/3276108747 >> [v2:172.21.15.114:6800/1395693466,v1:172.21.15.114:6801/1395693466] conn(0x7fd5300a3d20 0x7fd5300a61e0 unknown :-1 s=CLOSED pgs=156 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:41:07.043 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:41:07.043+0000 7fd54b9ad640 1 --2- 172.21.15.114:0/3276108747 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fd54419cf70 0x7fd54419d420 unknown :-1 s=CLOSED pgs=956 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:41:07.043 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:41:07.043+0000 7fd54b9ad640 1 --2- 172.21.15.114:0/3276108747 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fd544107260 0x7fd544198bf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:41:07.043 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:41:07.043+0000 7fd54b9ad640 1 --2- 172.21.15.114:0/3276108747 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fd544102190 0x7fd5441986b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:41:07.044 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:41:07.043+0000 7fd54b9ad640 1 -- 172.21.15.114:0/3276108747 >> 172.21.15.114:0/3276108747 conn(0x7fd5440793a0 msgr2=0x7fd5440ffcf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T10:41:07.044 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:41:07.043+0000 7fd54b9ad640 1 -- 172.21.15.114:0/3276108747 shutdown_connections 2024-04-23T10:41:07.044 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:41:07.043+0000 7fd54b9ad640 1 -- 172.21.15.114:0/3276108747 wait complete. 2024-04-23T10:41:07.053 DEBUG:teuthology.orchestra.run.smithi114:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2024-04-23T10:41:07.159 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:41:07.157+0000 7f98de0e8640 1 -- 172.21.15.114:0/4036033421 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f98d8101080 msgr2=0x7f98d8101460 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T10:41:07.159 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:41:07.157+0000 7f98de0e8640 1 --2- 172.21.15.114:0/4036033421 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f98d8101080 0x7f98d8101460 secure :-1 s=READY pgs=957 cs=0 l=1 rev1=1 crypto rx=0x7f98c8009900 tx=0x7f98c802a410 comp rx=0 tx=0).stop 2024-04-23T10:41:07.159 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:41:07.157+0000 7f98de0e8640 1 -- 172.21.15.114:0/4036033421 shutdown_connections 2024-04-23T10:41:07.160 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:41:07.157+0000 7f98de0e8640 1 --2- 172.21.15.114:0/4036033421 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f98d8106150 0x7f98d810dd20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:41:07.160 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:41:07.157+0000 7f98de0e8640 1 --2- 172.21.15.114:0/4036033421 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f98d8101a30 0x7f98d8105a70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:41:07.160 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:41:07.157+0000 7f98de0e8640 1 --2- 172.21.15.114:0/4036033421 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f98d8101080 0x7f98d8101460 unknown :-1 s=CLOSED pgs=957 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:41:07.160 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:41:07.157+0000 7f98de0e8640 1 -- 172.21.15.114:0/4036033421 >> 172.21.15.114:0/4036033421 conn(0x7f98d8078000 msgr2=0x7f98d8078410 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T10:41:07.160 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:41:07.157+0000 7f98de0e8640 1 -- 172.21.15.114:0/4036033421 shutdown_connections 2024-04-23T10:41:07.160 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:41:07.157+0000 7f98de0e8640 1 -- 172.21.15.114:0/4036033421 wait complete. 2024-04-23T10:41:07.161 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:41:07.157+0000 7f98de0e8640 1 Processor -- start 2024-04-23T10:41:07.161 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:41:07.157+0000 7f98de0e8640 1 -- start start 2024-04-23T10:41:07.161 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:41:07.158+0000 7f98de0e8640 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f98d8101080 0x7f98d81994f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T10:41:07.161 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:41:07.158+0000 7f98de0e8640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f98d8106150 0x7f98d8199a30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T10:41:07.161 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:41:07.158+0000 7f98de0e8640 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f98d819a010 0x7f98d81a4570 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T10:41:07.161 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:41:07.158+0000 7f98de0e8640 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x7f98d8111090 con 0x7f98d8106150 2024-04-23T10:41:07.162 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:41:07.158+0000 7f98de0e8640 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 -- 0x7f98d8110f10 con 0x7f98d819a010 2024-04-23T10:41:07.162 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:41:07.158+0000 7f98de0e8640 1 -- --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_getmap magic: 0 -- 0x7f98d8111210 con 0x7f98d8101080 2024-04-23T10:41:07.162 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:41:07.158+0000 7f98cffff640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f98d8106150 0x7f98d8199a30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T10:41:07.162 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:41:07.158+0000 7f98cffff640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f98d8106150 0x7f98d8199a30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.114:44618/0 (socket says 172.21.15.114:44618) 2024-04-23T10:41:07.162 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:41:07.158+0000 7f98cffff640 1 -- 172.21.15.114:0/4194930089 learned_addr learned my addr 172.21.15.114:0/4194930089 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-04-23T10:41:07.162 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:41:07.158+0000 7f98d77fe640 1 --2- 172.21.15.114:0/4194930089 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f98d8101080 0x7f98d81994f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T10:41:07.163 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:41:07.158+0000 7f98cffff640 1 -- 172.21.15.114:0/4194930089 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f98d8101080 msgr2=0x7f98d81994f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T10:41:07.163 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:41:07.158+0000 7f98cffff640 1 --2- 172.21.15.114:0/4194930089 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f98d8101080 0x7f98d81994f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:41:07.163 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:41:07.158+0000 7f98cffff640 1 -- 172.21.15.114:0/4194930089 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f98d819a010 msgr2=0x7f98d81a4570 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T10:41:07.163 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:41:07.158+0000 7f98cffff640 1 --2- 172.21.15.114:0/4194930089 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f98d819a010 0x7f98d81a4570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:41:07.163 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:41:07.158+0000 7f98cffff640 1 -- 172.21.15.114:0/4194930089 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f98d81a4d90 con 0x7f98d8106150 2024-04-23T10:41:07.164 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:41:07.158+0000 7f98d77fe640 1 --2- 172.21.15.114:0/4194930089 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f98d8101080 0x7f98d81994f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-23T10:41:07.164 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:41:07.158+0000 7f98cffff640 1 --2- 172.21.15.114:0/4194930089 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f98d8106150 0x7f98d8199a30 secure :-1 s=READY pgs=958 cs=0 l=1 rev1=1 crypto rx=0x7f98c000d770 tx=0x7f98c000dc30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T10:41:07.164 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:41:07.158+0000 7f98d57fa640 1 -- 172.21.15.114:0/4194930089 <== mon.0 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f98c0017070 con 0x7f98d8106150 2024-04-23T10:41:07.164 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:41:07.158+0000 7f98de0e8640 1 -- 172.21.15.114:0/4194930089 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f98d81a5080 con 0x7f98d8106150 2024-04-23T10:41:07.164 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:41:07.158+0000 7f98d57fa640 1 -- 172.21.15.114:0/4194930089 <== mon.0 v2:172.21.15.114:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f98c0009c30 con 0x7f98d8106150 2024-04-23T10:41:07.164 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:41:07.158+0000 7f98d57fa640 1 -- 172.21.15.114:0/4194930089 <== mon.0 v2:172.21.15.114:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f98c000f620 con 0x7f98d8106150 2024-04-23T10:41:07.164 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:41:07.158+0000 7f98de0e8640 1 -- 172.21.15.114:0/4194930089 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f98d81a55d0 con 0x7f98d8106150 2024-04-23T10:41:07.165 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:41:07.159+0000 7f98de0e8640 1 -- 172.21.15.114:0/4194930089 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f98d8068d70 con 0x7f98d8106150 2024-04-23T10:41:07.165 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:41:07.160+0000 7f98d57fa640 1 -- 172.21.15.114:0/4194930089 <== mon.0 v2:172.21.15.114:3300/0 4 ==== mgrmap(e 26) ==== 138778+0+0 (secure 0 0 0) 0x7f98c001a070 con 0x7f98d8106150 2024-04-23T10:41:07.166 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:41:07.161+0000 7f98d57fa640 1 --2- 172.21.15.114:0/4194930089 >> [v2:172.21.15.114:6800/1395693466,v1:172.21.15.114:6801/1395693466] conn(0x7f98c40a3d20 0x7f98c40a61e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T10:41:07.166 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:41:07.161+0000 7f98d77fe640 1 --2- 172.21.15.114:0/4194930089 >> [v2:172.21.15.114:6800/1395693466,v1:172.21.15.114:6801/1395693466] conn(0x7f98c40a3d20 0x7f98c40a61e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T10:41:07.166 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:41:07.161+0000 7f98d57fa640 1 -- 172.21.15.114:0/4194930089 <== mon.0 v2:172.21.15.114:3300/0 5 ==== osd_map(111..111 src has 1..111) ==== 8585+0+0 (secure 0 0 0) 0x7f98c00d0460 con 0x7f98d8106150 2024-04-23T10:41:07.167 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:41:07.161+0000 7f98d77fe640 1 --2- 172.21.15.114:0/4194930089 >> [v2:172.21.15.114:6800/1395693466,v1:172.21.15.114:6801/1395693466] conn(0x7f98c40a3d20 0x7f98c40a61e0 secure :-1 s=READY pgs=157 cs=0 l=1 rev1=1 crypto rx=0x7f98c8031440 tx=0x7f98c8005a20 comp rx=0 tx=0).ready entity=mgr.15291 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T10:41:07.167 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:41:07.165+0000 7f98d57fa640 1 -- 172.21.15.114:0/4194930089 <== mon.0 v2:172.21.15.114:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f98c0009de0 con 0x7f98d8106150 2024-04-23T10:41:07.344 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:41:07.345+0000 7f98de0e8640 1 -- 172.21.15.114:0/4194930089 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) -- 0x7f98d8101a30 con 0x7f98d8106150 2024-04-23T10:41:07.344 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:41:07.345+0000 7f98d57fa640 1 -- 172.21.15.114:0/4194930089 <== mon.0 v2:172.21.15.114:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v111) ==== 104+0+11 (secure 0 0 0) 0x7f98c009cac0 con 0x7f98d8106150 2024-04-23T10:41:07.344 INFO:teuthology.orchestra.run.smithi114.stdout:pg_num: 32 2024-04-23T10:41:07.346 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:41:07.347+0000 7f98de0e8640 1 -- 172.21.15.114:0/4194930089 >> [v2:172.21.15.114:6800/1395693466,v1:172.21.15.114:6801/1395693466] conn(0x7f98c40a3d20 msgr2=0x7f98c40a61e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T10:41:07.347 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:41:07.347+0000 7f98de0e8640 1 --2- 172.21.15.114:0/4194930089 >> [v2:172.21.15.114:6800/1395693466,v1:172.21.15.114:6801/1395693466] conn(0x7f98c40a3d20 0x7f98c40a61e0 secure :-1 s=READY pgs=157 cs=0 l=1 rev1=1 crypto rx=0x7f98c8031440 tx=0x7f98c8005a20 comp rx=0 tx=0).stop 2024-04-23T10:41:07.347 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:41:07.347+0000 7f98de0e8640 1 -- 172.21.15.114:0/4194930089 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f98d8106150 msgr2=0x7f98d8199a30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T10:41:07.347 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:41:07.347+0000 7f98de0e8640 1 --2- 172.21.15.114:0/4194930089 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f98d8106150 0x7f98d8199a30 secure :-1 s=READY pgs=958 cs=0 l=1 rev1=1 crypto rx=0x7f98c000d770 tx=0x7f98c000dc30 comp rx=0 tx=0).stop 2024-04-23T10:41:07.347 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:41:07.347+0000 7f98de0e8640 1 -- 172.21.15.114:0/4194930089 shutdown_connections 2024-04-23T10:41:07.347 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:41:07.347+0000 7f98de0e8640 1 --2- 172.21.15.114:0/4194930089 >> [v2:172.21.15.114:6800/1395693466,v1:172.21.15.114:6801/1395693466] conn(0x7f98c40a3d20 0x7f98c40a61e0 unknown :-1 s=CLOSED pgs=157 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:41:07.347 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:41:07.347+0000 7f98de0e8640 1 --2- 172.21.15.114:0/4194930089 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f98d819a010 0x7f98d81a4570 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:41:07.348 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:41:07.347+0000 7f98de0e8640 1 --2- 172.21.15.114:0/4194930089 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f98d8106150 0x7f98d8199a30 unknown :-1 s=CLOSED pgs=958 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:41:07.348 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:41:07.347+0000 7f98de0e8640 1 --2- 172.21.15.114:0/4194930089 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f98d8101080 0x7f98d81994f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:41:07.348 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:41:07.347+0000 7f98de0e8640 1 -- 172.21.15.114:0/4194930089 >> 172.21.15.114:0/4194930089 conn(0x7f98d8078000 msgr2=0x7f98d80ff130 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T10:41:07.348 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:41:07.347+0000 7f98de0e8640 1 -- 172.21.15.114:0/4194930089 shutdown_connections 2024-04-23T10:41:07.348 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:41:07.347+0000 7f98de0e8640 1 -- 172.21.15.114:0/4194930089 wait complete. 2024-04-23T10:41:07.356 INFO:journalctl@ceph.mon.a.smithi114.stdout:Apr 23 10:41:07 smithi114 ceph-mon[29255]: from='client.17163 -' entity='client.admin' cmd=[{"prefix": "fs snap-schedule status", "fs": "cephfs", "path": "/", "target": ["mon-mgr", ""]}]: dispatch 2024-04-23T10:41:07.356 INFO:journalctl@ceph.mon.a.smithi114.stdout:Apr 23 10:41:07 smithi114 ceph-mon[29255]: from='mgr.15291 172.21.15.114:0/1095789187' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-23T10:41:07.357 INFO:journalctl@ceph.mon.a.smithi114.stdout:Apr 23 10:41:07 smithi114 ceph-mon[29255]: from='mgr.15291 172.21.15.114:0/1095789187' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-23T10:41:07.357 INFO:journalctl@ceph.mon.a.smithi114.stdout:Apr 23 10:41:07 smithi114 ceph-mon[29255]: from='mgr.15291 172.21.15.114:0/1095789187' entity='mgr.x' 2024-04-23T10:41:07.357 INFO:journalctl@ceph.mon.a.smithi114.stdout:Apr 23 10:41:07 smithi114 ceph-mon[29255]: from='client.? 172.21.15.114:0/1698357046' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-23T10:41:07.357 INFO:journalctl@ceph.mon.a.smithi114.stdout:Apr 23 10:41:07 smithi114 ceph-mon[29255]: from='client.? 172.21.15.114:0/3276108747' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-04-23T10:41:07.358 DEBUG:teuthology.orchestra.run.smithi114:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2024-04-23T10:41:07.465 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:41:07.464+0000 7fcc38f9b640 1 -- 172.21.15.114:0/1359640542 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fcc34102b40 msgr2=0x7fcc34106b80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T10:41:07.466 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:41:07.464+0000 7fcc38f9b640 1 --2- 172.21.15.114:0/1359640542 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fcc34102b40 0x7fcc34106b80 secure :-1 s=READY pgs=959 cs=0 l=1 rev1=1 crypto rx=0x7fcc1c009900 tx=0x7fcc1c02a410 comp rx=0 tx=0).stop 2024-04-23T10:41:07.466 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:41:07.464+0000 7fcc38f9b640 1 -- 172.21.15.114:0/1359640542 shutdown_connections 2024-04-23T10:41:07.466 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:41:07.464+0000 7fcc38f9b640 1 --2- 172.21.15.114:0/1359640542 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fcc34107260 0x7fcc3410ee30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:41:07.466 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:41:07.464+0000 7fcc38f9b640 1 --2- 172.21.15.114:0/1359640542 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fcc34102b40 0x7fcc34106b80 unknown :-1 s=CLOSED pgs=959 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:41:07.466 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:41:07.464+0000 7fcc38f9b640 1 --2- 172.21.15.114:0/1359640542 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fcc34102190 0x7fcc34102570 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:41:07.466 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:41:07.464+0000 7fcc38f9b640 1 -- 172.21.15.114:0/1359640542 >> 172.21.15.114:0/1359640542 conn(0x7fcc340fd990 msgr2=0x7fcc340ffdb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T10:41:07.467 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:41:07.464+0000 7fcc38f9b640 1 -- 172.21.15.114:0/1359640542 shutdown_connections 2024-04-23T10:41:07.467 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:41:07.464+0000 7fcc38f9b640 1 -- 172.21.15.114:0/1359640542 wait complete. 2024-04-23T10:41:07.467 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:41:07.464+0000 7fcc38f9b640 1 Processor -- start 2024-04-23T10:41:07.467 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:41:07.464+0000 7fcc38f9b640 1 -- start start 2024-04-23T10:41:07.467 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:41:07.465+0000 7fcc38f9b640 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fcc34102190 0x7fcc3419e3f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T10:41:07.467 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:41:07.465+0000 7fcc38f9b640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fcc34107260 0x7fcc3419e930 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T10:41:07.468 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:41:07.465+0000 7fcc38f9b640 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fcc341985a0 0x7fcc34198a50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T10:41:07.468 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:41:07.465+0000 7fcc38f9b640 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x7fcc341121a0 con 0x7fcc34107260 2024-04-23T10:41:07.468 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:41:07.465+0000 7fcc38f9b640 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 -- 0x7fcc34112020 con 0x7fcc341985a0 2024-04-23T10:41:07.468 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:41:07.465+0000 7fcc38f9b640 1 -- --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_getmap magic: 0 -- 0x7fcc34112320 con 0x7fcc34102190 2024-04-23T10:41:07.468 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:41:07.465+0000 7fcc31d9b640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fcc34107260 0x7fcc3419e930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T10:41:07.468 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:41:07.465+0000 7fcc31d9b640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fcc34107260 0x7fcc3419e930 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.114:44630/0 (socket says 172.21.15.114:44630) 2024-04-23T10:41:07.468 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:41:07.465+0000 7fcc31d9b640 1 -- 172.21.15.114:0/1160765680 learned_addr learned my addr 172.21.15.114:0/1160765680 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-04-23T10:41:07.469 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:41:07.465+0000 7fcc3259c640 1 --2- 172.21.15.114:0/1160765680 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fcc34102190 0x7fcc3419e3f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T10:41:07.469 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:41:07.465+0000 7fcc32d9d640 1 --2- 172.21.15.114:0/1160765680 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fcc341985a0 0x7fcc34198a50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T10:41:07.469 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:41:07.465+0000 7fcc31d9b640 1 -- 172.21.15.114:0/1160765680 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fcc34102190 msgr2=0x7fcc3419e3f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T10:41:07.469 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:41:07.465+0000 7fcc31d9b640 1 --2- 172.21.15.114:0/1160765680 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fcc34102190 0x7fcc3419e3f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:41:07.469 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:41:07.465+0000 7fcc31d9b640 1 -- 172.21.15.114:0/1160765680 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fcc341985a0 msgr2=0x7fcc34198a50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T10:41:07.469 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:41:07.465+0000 7fcc31d9b640 1 --2- 172.21.15.114:0/1160765680 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fcc341985a0 0x7fcc34198a50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:41:07.469 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:41:07.465+0000 7fcc31d9b640 1 -- 172.21.15.114:0/1160765680 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fcc341990c0 con 0x7fcc34107260 2024-04-23T10:41:07.470 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:41:07.465+0000 7fcc31d9b640 1 --2- 172.21.15.114:0/1160765680 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fcc34107260 0x7fcc3419e930 secure :-1 s=READY pgs=960 cs=0 l=1 rev1=1 crypto rx=0x7fcc1c032fd0 tx=0x7fcc1c004270 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T10:41:07.470 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:41:07.465+0000 7fcc2b7fe640 1 -- 172.21.15.114:0/1160765680 <== mon.0 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcc1c03f030 con 0x7fcc34107260 2024-04-23T10:41:07.470 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:41:07.465+0000 7fcc38f9b640 1 -- 172.21.15.114:0/1160765680 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fcc34199350 con 0x7fcc34107260 2024-04-23T10:41:07.470 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:41:07.465+0000 7fcc2b7fe640 1 -- 172.21.15.114:0/1160765680 <== mon.0 v2:172.21.15.114:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fcc1c004490 con 0x7fcc34107260 2024-04-23T10:41:07.470 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:41:07.465+0000 7fcc2b7fe640 1 -- 172.21.15.114:0/1160765680 <== mon.0 v2:172.21.15.114:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcc1c035070 con 0x7fcc34107260 2024-04-23T10:41:07.470 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:41:07.465+0000 7fcc38f9b640 1 -- 172.21.15.114:0/1160765680 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fcc341a55a0 con 0x7fcc34107260 2024-04-23T10:41:07.471 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:41:07.466+0000 7fcc38f9b640 1 -- 172.21.15.114:0/1160765680 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fcc340691c0 con 0x7fcc34107260 2024-04-23T10:41:07.472 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:41:07.467+0000 7fcc2b7fe640 1 -- 172.21.15.114:0/1160765680 <== mon.0 v2:172.21.15.114:3300/0 4 ==== mgrmap(e 26) ==== 138778+0+0 (secure 0 0 0) 0x7fcc1c042030 con 0x7fcc34107260 2024-04-23T10:41:07.472 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:41:07.468+0000 7fcc2b7fe640 1 --2- 172.21.15.114:0/1160765680 >> [v2:172.21.15.114:6800/1395693466,v1:172.21.15.114:6801/1395693466] conn(0x7fcc200a3d20 0x7fcc200a61e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T10:41:07.472 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:41:07.468+0000 7fcc3259c640 1 --2- 172.21.15.114:0/1160765680 >> [v2:172.21.15.114:6800/1395693466,v1:172.21.15.114:6801/1395693466] conn(0x7fcc200a3d20 0x7fcc200a61e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T10:41:07.472 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:41:07.468+0000 7fcc2b7fe640 1 -- 172.21.15.114:0/1160765680 <== mon.0 v2:172.21.15.114:3300/0 5 ==== osd_map(111..111 src has 1..111) ==== 8585+0+0 (secure 0 0 0) 0x7fcc1c0f0da0 con 0x7fcc34107260 2024-04-23T10:41:07.472 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:41:07.469+0000 7fcc3259c640 1 --2- 172.21.15.114:0/1160765680 >> [v2:172.21.15.114:6800/1395693466,v1:172.21.15.114:6801/1395693466] conn(0x7fcc200a3d20 0x7fcc200a61e0 secure :-1 s=READY pgs=158 cs=0 l=1 rev1=1 crypto rx=0x7fcc2c005fd0 tx=0x7fcc2c009870 comp rx=0 tx=0).ready entity=mgr.15291 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T10:41:07.473 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:41:07.473+0000 7fcc2b7fe640 1 -- 172.21.15.114:0/1160765680 <== mon.0 v2:172.21.15.114:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7fcc1c0bd400 con 0x7fcc34107260 2024-04-23T10:41:07.492 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 23 10:41:07 smithi119 ceph-mon[32608]: from='client.17163 -' entity='client.admin' cmd=[{"prefix": "fs snap-schedule status", "fs": "cephfs", "path": "/", "target": ["mon-mgr", ""]}]: dispatch 2024-04-23T10:41:07.492 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 23 10:41:07 smithi119 ceph-mon[32608]: from='mgr.15291 172.21.15.114:0/1095789187' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-23T10:41:07.492 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 23 10:41:07 smithi119 ceph-mon[32608]: from='mgr.15291 172.21.15.114:0/1095789187' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-23T10:41:07.492 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 23 10:41:07 smithi119 ceph-mon[32608]: from='mgr.15291 172.21.15.114:0/1095789187' entity='mgr.x' 2024-04-23T10:41:07.493 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 23 10:41:07 smithi119 ceph-mon[32608]: from='client.? 172.21.15.114:0/1698357046' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-23T10:41:07.493 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 23 10:41:07 smithi119 ceph-mon[32608]: from='client.? 172.21.15.114:0/3276108747' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-04-23T10:41:07.578 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 23 10:41:07 smithi204 ceph-mon[32649]: from='client.17163 -' entity='client.admin' cmd=[{"prefix": "fs snap-schedule status", "fs": "cephfs", "path": "/", "target": ["mon-mgr", ""]}]: dispatch 2024-04-23T10:41:07.578 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 23 10:41:07 smithi204 ceph-mon[32649]: from='mgr.15291 172.21.15.114:0/1095789187' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-23T10:41:07.578 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 23 10:41:07 smithi204 ceph-mon[32649]: from='mgr.15291 172.21.15.114:0/1095789187' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-23T10:41:07.578 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 23 10:41:07 smithi204 ceph-mon[32649]: from='mgr.15291 172.21.15.114:0/1095789187' entity='mgr.x' 2024-04-23T10:41:07.578 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 23 10:41:07 smithi204 ceph-mon[32649]: from='client.? 172.21.15.114:0/1698357046' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-23T10:41:07.578 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 23 10:41:07 smithi204 ceph-mon[32649]: from='client.? 172.21.15.114:0/3276108747' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-04-23T10:41:07.649 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:41:07.650+0000 7fcc38f9b640 1 -- 172.21.15.114:0/1160765680 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) -- 0x7fcc3410a4f0 con 0x7fcc34107260 2024-04-23T10:41:07.650 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:41:07.651+0000 7fcc2b7fe640 1 -- 172.21.15.114:0/1160765680 <== mon.0 v2:172.21.15.114:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v111) ==== 100+0+11 (secure 0 0 0) 0x7fcc1c0bcd00 con 0x7fcc34107260 2024-04-23T10:41:07.650 INFO:teuthology.orchestra.run.smithi114.stdout:pg_num: 64 2024-04-23T10:41:07.652 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:41:07.653+0000 7fcc38f9b640 1 -- 172.21.15.114:0/1160765680 >> [v2:172.21.15.114:6800/1395693466,v1:172.21.15.114:6801/1395693466] conn(0x7fcc200a3d20 msgr2=0x7fcc200a61e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T10:41:07.652 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:41:07.653+0000 7fcc38f9b640 1 --2- 172.21.15.114:0/1160765680 >> [v2:172.21.15.114:6800/1395693466,v1:172.21.15.114:6801/1395693466] conn(0x7fcc200a3d20 0x7fcc200a61e0 secure :-1 s=READY pgs=158 cs=0 l=1 rev1=1 crypto rx=0x7fcc2c005fd0 tx=0x7fcc2c009870 comp rx=0 tx=0).stop 2024-04-23T10:41:07.652 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:41:07.653+0000 7fcc38f9b640 1 -- 172.21.15.114:0/1160765680 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fcc34107260 msgr2=0x7fcc3419e930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T10:41:07.653 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:41:07.653+0000 7fcc38f9b640 1 --2- 172.21.15.114:0/1160765680 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fcc34107260 0x7fcc3419e930 secure :-1 s=READY pgs=960 cs=0 l=1 rev1=1 crypto rx=0x7fcc1c032fd0 tx=0x7fcc1c004270 comp rx=0 tx=0).stop 2024-04-23T10:41:07.653 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:41:07.653+0000 7fcc38f9b640 1 -- 172.21.15.114:0/1160765680 shutdown_connections 2024-04-23T10:41:07.653 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:41:07.653+0000 7fcc38f9b640 1 --2- 172.21.15.114:0/1160765680 >> [v2:172.21.15.114:6800/1395693466,v1:172.21.15.114:6801/1395693466] conn(0x7fcc200a3d20 0x7fcc200a61e0 unknown :-1 s=CLOSED pgs=158 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:41:07.653 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:41:07.653+0000 7fcc38f9b640 1 --2- 172.21.15.114:0/1160765680 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fcc341985a0 0x7fcc34198a50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:41:07.653 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:41:07.653+0000 7fcc38f9b640 1 --2- 172.21.15.114:0/1160765680 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fcc34107260 0x7fcc3419e930 unknown :-1 s=CLOSED pgs=960 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:41:07.653 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:41:07.653+0000 7fcc38f9b640 1 --2- 172.21.15.114:0/1160765680 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fcc34102190 0x7fcc3419e3f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:41:07.654 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:41:07.653+0000 7fcc38f9b640 1 -- 172.21.15.114:0/1160765680 >> 172.21.15.114:0/1160765680 conn(0x7fcc340fd990 msgr2=0x7fcc340fed60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T10:41:07.654 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:41:07.653+0000 7fcc38f9b640 1 -- 172.21.15.114:0/1160765680 shutdown_connections 2024-04-23T10:41:07.654 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:41:07.653+0000 7fcc38f9b640 1 -- 172.21.15.114:0/1160765680 wait complete. 2024-04-23T10:41:07.663 DEBUG:teuthology.orchestra.run.smithi114:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph mgr dump --format=json-pretty 2024-04-23T10:41:07.765 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:41:07.763+0000 7f2988079640 1 -- 172.21.15.114:0/3040656490 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f2980100b10 msgr2=0x7f2980100ef0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T10:41:07.765 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:41:07.763+0000 7f2988079640 1 --2- 172.21.15.114:0/3040656490 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f2980100b10 0x7f2980100ef0 secure :-1 s=READY pgs=961 cs=0 l=1 rev1=1 crypto rx=0x7f2970009900 tx=0x7f297002a410 comp rx=0 tx=0).stop 2024-04-23T10:41:07.765 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:41:07.763+0000 7f2988079640 1 -- 172.21.15.114:0/3040656490 shutdown_connections 2024-04-23T10:41:07.766 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:41:07.763+0000 7f2988079640 1 --2- 172.21.15.114:0/3040656490 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f29801072d0 0x7f298010eec0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:41:07.766 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:41:07.763+0000 7f2988079640 1 --2- 172.21.15.114:0/3040656490 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f29801014c0 0x7f2980106bf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:41:07.766 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:41:07.763+0000 7f2988079640 1 --2- 172.21.15.114:0/3040656490 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f2980100b10 0x7f2980100ef0 unknown :-1 s=CLOSED pgs=961 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:41:07.766 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:41:07.763+0000 7f2988079640 1 -- 172.21.15.114:0/3040656490 >> 172.21.15.114:0/3040656490 conn(0x7f2980076130 msgr2=0x7f2980076540 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T10:41:07.766 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:41:07.763+0000 7f2988079640 1 -- 172.21.15.114:0/3040656490 shutdown_connections 2024-04-23T10:41:07.766 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:41:07.763+0000 7f2988079640 1 -- 172.21.15.114:0/3040656490 wait complete. 2024-04-23T10:41:07.767 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:41:07.764+0000 7f2988079640 1 Processor -- start 2024-04-23T10:41:07.767 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:41:07.764+0000 7f2988079640 1 -- start start 2024-04-23T10:41:07.767 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:41:07.764+0000 7f2988079640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f2980100b10 0x7f298019e3f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T10:41:07.767 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:41:07.764+0000 7f2988079640 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f29801072d0 0x7f298019e930 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T10:41:07.767 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:41:07.764+0000 7f2988079640 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f29801985a0 0x7f2980198a50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T10:41:07.767 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:41:07.764+0000 7f2988079640 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x7f2980112260 con 0x7f2980100b10 2024-04-23T10:41:07.768 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:41:07.764+0000 7f2988079640 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 -- 0x7f29801120e0 con 0x7f29801985a0 2024-04-23T10:41:07.768 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:41:07.764+0000 7f2988079640 1 -- --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_getmap magic: 0 -- 0x7f29801123e0 con 0x7f29801072d0 2024-04-23T10:41:07.768 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:41:07.764+0000 7f2985e15640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f2980100b10 0x7f298019e3f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T10:41:07.768 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:41:07.764+0000 7f2985e15640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f2980100b10 0x7f298019e3f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.114:44642/0 (socket says 172.21.15.114:44642) 2024-04-23T10:41:07.768 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:41:07.764+0000 7f2985e15640 1 -- 172.21.15.114:0/595361704 learned_addr learned my addr 172.21.15.114:0/595361704 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-04-23T10:41:07.768 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:41:07.764+0000 7f2986616640 1 --2- 172.21.15.114:0/595361704 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f29801985a0 0x7f2980198a50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T10:41:07.769 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:41:07.764+0000 7f2985e15640 1 -- 172.21.15.114:0/595361704 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f29801072d0 msgr2=0x7f298019e930 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T10:41:07.769 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:41:07.764+0000 7f2985e15640 1 --2- 172.21.15.114:0/595361704 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f29801072d0 0x7f298019e930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:41:07.769 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:41:07.764+0000 7f2985e15640 1 -- 172.21.15.114:0/595361704 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f29801985a0 msgr2=0x7f2980198a50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T10:41:07.769 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:41:07.764+0000 7f2985e15640 1 --2- 172.21.15.114:0/595361704 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f29801985a0 0x7f2980198a50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:41:07.769 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:41:07.764+0000 7f2985e15640 1 -- 172.21.15.114:0/595361704 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f29801990c0 con 0x7f2980100b10 2024-04-23T10:41:07.769 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:41:07.764+0000 7f2986616640 1 --2- 172.21.15.114:0/595361704 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f29801985a0 0x7f2980198a50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-23T10:41:07.769 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:41:07.764+0000 7f2985e15640 1 --2- 172.21.15.114:0/595361704 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f2980100b10 0x7f298019e3f0 secure :-1 s=READY pgs=962 cs=0 l=1 rev1=1 crypto rx=0x7f2970005940 tx=0x7f29700042f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T10:41:07.770 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:41:07.764+0000 7f2976ffd640 1 -- 172.21.15.114:0/595361704 <== mon.0 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f297003f030 con 0x7f2980100b10 2024-04-23T10:41:07.770 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:41:07.764+0000 7f2976ffd640 1 -- 172.21.15.114:0/595361704 <== mon.0 v2:172.21.15.114:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f2970003cb0 con 0x7f2980100b10 2024-04-23T10:41:07.770 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:41:07.764+0000 7f2988079640 1 -- 172.21.15.114:0/595361704 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f2980199350 con 0x7f2980100b10 2024-04-23T10:41:07.770 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:41:07.764+0000 7f2976ffd640 1 -- 172.21.15.114:0/595361704 <== mon.0 v2:172.21.15.114:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2970035070 con 0x7f2980100b10 2024-04-23T10:41:07.770 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:41:07.764+0000 7f2988079640 1 -- 172.21.15.114:0/595361704 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f29801a5540 con 0x7f2980100b10 2024-04-23T10:41:07.770 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:41:07.765+0000 7f2988079640 1 -- 172.21.15.114:0/595361704 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f29800691c0 con 0x7f2980100b10 2024-04-23T10:41:07.771 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:41:07.766+0000 7f2976ffd640 1 -- 172.21.15.114:0/595361704 <== mon.0 v2:172.21.15.114:3300/0 4 ==== mgrmap(e 26) ==== 138778+0+0 (secure 0 0 0) 0x7f2970042030 con 0x7f2980100b10 2024-04-23T10:41:07.772 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:41:07.767+0000 7f2976ffd640 1 --2- 172.21.15.114:0/595361704 >> [v2:172.21.15.114:6800/1395693466,v1:172.21.15.114:6801/1395693466] conn(0x7f296c0a3d20 0x7f296c0a61e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T10:41:07.772 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:41:07.767+0000 7f2985614640 1 --2- 172.21.15.114:0/595361704 >> [v2:172.21.15.114:6800/1395693466,v1:172.21.15.114:6801/1395693466] conn(0x7f296c0a3d20 0x7f296c0a61e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T10:41:07.772 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:41:07.767+0000 7f2976ffd640 1 -- 172.21.15.114:0/595361704 <== mon.0 v2:172.21.15.114:3300/0 5 ==== osd_map(111..111 src has 1..111) ==== 8585+0+0 (secure 0 0 0) 0x7f29700f1170 con 0x7f2980100b10 2024-04-23T10:41:07.772 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:41:07.767+0000 7f2985614640 1 --2- 172.21.15.114:0/595361704 >> [v2:172.21.15.114:6800/1395693466,v1:172.21.15.114:6801/1395693466] conn(0x7f296c0a3d20 0x7f296c0a61e0 secure :-1 s=READY pgs=159 cs=0 l=1 rev1=1 crypto rx=0x7f29801004e0 tx=0x7f297c005c70 comp rx=0 tx=0).ready entity=mgr.15291 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T10:41:07.772 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:41:07.771+0000 7f2976ffd640 1 -- 172.21.15.114:0/595361704 <== mon.0 v2:172.21.15.114:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f29700bd7d0 con 0x7f2980100b10 2024-04-23T10:41:07.999 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:41:08.001+0000 7f2988079640 1 -- 172.21.15.114:0/595361704 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_command({"prefix": "mgr dump", "format": "json-pretty"} v 0) -- 0x7f29801014c0 con 0x7f2980100b10 2024-04-23T10:41:08.008 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:41:08.009+0000 7f2976ffd640 1 -- 172.21.15.114:0/595361704 <== mon.0 v2:172.21.15.114:3300/0 7 ==== mon_command_ack([{"prefix": "mgr dump", "format": "json-pretty"}]=0 v26) ==== 81+0+746071 (secure 0 0 0) 0x7f29700bd0d0 con 0x7f2980100b10 2024-04-23T10:41:08.021 INFO:teuthology.orchestra.run.smithi114.stdout: 2024-04-23T10:41:08.021 INFO:teuthology.orchestra.run.smithi114.stdout:{ 2024-04-23T10:41:08.021 INFO:teuthology.orchestra.run.smithi114.stdout: "epoch": 26, 2024-04-23T10:41:08.021 INFO:teuthology.orchestra.run.smithi114.stdout: "flags": 0, 2024-04-23T10:41:08.021 INFO:teuthology.orchestra.run.smithi114.stdout: "active_gid": 15291, 2024-04-23T10:41:08.022 INFO:teuthology.orchestra.run.smithi114.stdout: "active_name": "x", 2024-04-23T10:41:08.022 INFO:teuthology.orchestra.run.smithi114.stdout: "active_addrs": { 2024-04-23T10:41:08.022 INFO:teuthology.orchestra.run.smithi114.stdout: "addrvec": [ 2024-04-23T10:41:08.022 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-04-23T10:41:08.022 INFO:teuthology.orchestra.run.smithi114.stdout: "type": "v2", 2024-04-23T10:41:08.022 INFO:teuthology.orchestra.run.smithi114.stdout: "addr": "172.21.15.114:6800", 2024-04-23T10:41:08.023 INFO:teuthology.orchestra.run.smithi114.stdout: "nonce": 1395693466 2024-04-23T10:41:08.023 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-23T10:41:08.023 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-04-23T10:41:08.023 INFO:teuthology.orchestra.run.smithi114.stdout: "type": "v1", 2024-04-23T10:41:08.023 INFO:teuthology.orchestra.run.smithi114.stdout: "addr": "172.21.15.114:6801", 2024-04-23T10:41:08.023 INFO:teuthology.orchestra.run.smithi114.stdout: "nonce": 1395693466 2024-04-23T10:41:08.023 INFO:teuthology.orchestra.run.smithi114.stdout: } 2024-04-23T10:41:08.024 INFO:teuthology.orchestra.run.smithi114.stdout: ] 2024-04-23T10:41:08.024 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-23T10:41:08.024 INFO:teuthology.orchestra.run.smithi114.stdout: "active_addr": "172.21.15.114:6801/1395693466", 2024-04-23T10:41:08.024 INFO:teuthology.orchestra.run.smithi114.stdout: "active_change": "2024-04-23T10:26:40.272886+0000", 2024-04-23T10:41:08.024 INFO:teuthology.orchestra.run.smithi114.stdout: "active_mgr_features": 4540701547738038271, 2024-04-23T10:41:08.024 INFO:teuthology.orchestra.run.smithi114.stdout: "available": true, 2024-04-23T10:41:08.024 INFO:teuthology.orchestra.run.smithi114.stdout: "standbys": [ 2024-04-23T10:41:08.025 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-04-23T10:41:08.025 INFO:teuthology.orchestra.run.smithi114.stdout: "gid": 24844, 2024-04-23T10:41:08.025 INFO:teuthology.orchestra.run.smithi114.stdout: "name": "z", 2024-04-23T10:41:08.025 INFO:teuthology.orchestra.run.smithi114.stdout: "mgr_features": 4540701547738038271, 2024-04-23T10:41:08.025 INFO:teuthology.orchestra.run.smithi114.stdout: "available_modules": [ 2024-04-23T10:41:08.025 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-04-23T10:41:08.025 INFO:teuthology.orchestra.run.smithi114.stdout: "name": "alerts", 2024-04-23T10:41:08.026 INFO:teuthology.orchestra.run.smithi114.stdout: "can_run": true, 2024-04-23T10:41:08.026 INFO:teuthology.orchestra.run.smithi114.stdout: "error_string": "", 2024-04-23T10:41:08.026 INFO:teuthology.orchestra.run.smithi114.stdout: "module_options": { 2024-04-23T10:41:08.026 INFO:teuthology.orchestra.run.smithi114.stdout: "interval": { 2024-04-23T10:41:08.026 INFO:teuthology.orchestra.run.smithi114.stdout: "name": "interval", 2024-04-23T10:41:08.026 INFO:teuthology.orchestra.run.smithi114.stdout: "type": "secs", 2024-04-23T10:41:08.027 INFO:teuthology.orchestra.run.smithi114.stdout: "level": "advanced", 2024-04-23T10:41:08.027 INFO:teuthology.orchestra.run.smithi114.stdout: "flags": 1, 2024-04-23T10:41:08.027 INFO:teuthology.orchestra.run.smithi114.stdout: "default_value": "60", 2024-04-23T10:41:08.027 INFO:teuthology.orchestra.run.smithi114.stdout: "min": "", 2024-04-23T10:41:08.027 INFO:teuthology.orchestra.run.smithi114.stdout: "max": "", 2024-04-23T10:41:08.027 INFO:teuthology.orchestra.run.smithi114.stdout: "enum_allowed": [], 2024-04-23T10:41:08.027 INFO:teuthology.orchestra.run.smithi114.stdout: "desc": "How frequently to reexamine health status", 2024-04-23T10:41:08.028 INFO:teuthology.orchestra.run.smithi114.stdout: "long_desc": "", 2024-04-23T10:41:08.028 INFO:teuthology.orchestra.run.smithi114.stdout: "tags": [], 2024-04-23T10:41:08.028 INFO:teuthology.orchestra.run.smithi114.stdout: "see_also": [] 2024-04-23T10:41:08.028 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-23T10:41:08.028 INFO:teuthology.orchestra.run.smithi114.stdout: "log_level": { 2024-04-23T10:41:08.028 INFO:teuthology.orchestra.run.smithi114.stdout: "name": "log_level", 2024-04-23T10:41:08.029 INFO:teuthology.orchestra.run.smithi114.stdout: "type": "str", 2024-04-23T10:41:08.029 INFO:teuthology.orchestra.run.smithi114.stdout: "level": "advanced", 2024-04-23T10:41:08.029 INFO:teuthology.orchestra.run.smithi114.stdout: "flags": 1, 2024-04-23T10:41:08.029 INFO:teuthology.orchestra.run.smithi114.stdout: "default_value": "", 2024-04-23T10:41:08.029 INFO:teuthology.orchestra.run.smithi114.stdout: "min": "", 2024-04-23T10:41:08.029 INFO:teuthology.orchestra.run.smithi114.stdout: "max": "", 2024-04-23T10:41:08.029 INFO:teuthology.orchestra.run.smithi114.stdout: "enum_allowed": [ 2024-04-23T10:41:08.030 INFO:teuthology.orchestra.run.smithi114.stdout: "", 2024-04-23T10:41:08.030 INFO:teuthology.orchestra.run.smithi114.stdout: "critical", 2024-04-23T10:41:08.030 INFO:teuthology.orchestra.run.smithi114.stdout: "debug", 2024-04-23T10:41:08.030 INFO:teuthology.orchestra.run.smithi114.stdout: "error", 2024-04-23T10:41:08.030 INFO:teuthology.orchestra.run.smithi114.stdout: "info", 2024-04-23T10:41:08.030 INFO:teuthology.orchestra.run.smithi114.stdout: "warning" 2024-04-23T10:41:08.030 INFO:teuthology.orchestra.run.smithi114.stdout: ], 2024-04-23T10:41:08.031 INFO:teuthology.orchestra.run.smithi114.stdout: "desc": "", 2024-04-23T10:41:08.031 INFO:teuthology.orchestra.run.smithi114.stdout: "long_desc": "", 2024-04-23T10:41:08.031 INFO:teuthology.orchestra.run.smithi114.stdout: "tags": [], 2024-04-23T10:41:08.031 INFO:teuthology.orchestra.run.smithi114.stdout: "see_also": [] 2024-04-23T10:41:08.031 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-23T10:41:08.031 INFO:teuthology.orchestra.run.smithi114.stdout: "log_to_cluster": { 2024-04-23T10:41:08.032 INFO:teuthology.orchestra.run.smithi114.stdout: "name": "log_to_cluster", 2024-04-23T10:41:08.032 INFO:teuthology.orchestra.run.smithi114.stdout: "type": "bool", 2024-04-23T10:41:08.032 INFO:teuthology.orchestra.run.smithi114.stdout: "level": "advanced", 2024-04-23T10:41:08.032 INFO:teuthology.orchestra.run.smithi114.stdout: "flags": 1, 2024-04-23T10:41:08.032 INFO:teuthology.orchestra.run.smithi114.stdout: "default_value": "False", 2024-04-23T10:41:08.032 INFO:teuthology.orchestra.run.smithi114.stdout: "min": "", 2024-04-23T10:41:08.032 INFO:teuthology.orchestra.run.smithi114.stdout: "max": "", 2024-04-23T10:41:08.032 INFO:teuthology.orchestra.run.smithi114.stdout: "enum_allowed": [], 2024-04-23T10:41:08.033 INFO:teuthology.orchestra.run.smithi114.stdout: "desc": "", 2024-04-23T10:41:08.033 INFO:teuthology.orchestra.run.smithi114.stdout: "long_desc": "", 2024-04-23T10:41:08.033 INFO:teuthology.orchestra.run.smithi114.stdout: "tags": [], 2024-04-23T10:41:08.033 INFO:teuthology.orchestra.run.smithi114.stdout: "see_also": [] 2024-04-23T10:41:08.033 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-23T10:41:08.033 INFO:teuthology.orchestra.run.smithi114.stdout: "log_to_cluster_level": { 2024-04-23T10:41:08.034 INFO:teuthology.orchestra.run.smithi114.stdout: "name": "log_to_cluster_level", 2024-04-23T10:41:08.034 INFO:teuthology.orchestra.run.smithi114.stdout: "type": "str", 2024-04-23T10:41:08.034 INFO:teuthology.orchestra.run.smithi114.stdout: "level": "advanced", 2024-04-23T10:41:08.034 INFO:teuthology.orchestra.run.smithi114.stdout: "flags": 1, 2024-04-23T10:41:08.034 INFO:teuthology.orchestra.run.smithi114.stdout: "default_value": "info", 2024-04-23T10:41:08.034 INFO:teuthology.orchestra.run.smithi114.stdout: "min": "", 2024-04-23T10:41:08.034 INFO:teuthology.orchestra.run.smithi114.stdout: "max": "", 2024-04-23T10:41:08.035 INFO:teuthology.orchestra.run.smithi114.stdout: "enum_allowed": [ 2024-04-23T10:41:08.035 INFO:teuthology.orchestra.run.smithi114.stdout: "", 2024-04-23T10:41:08.035 INFO:teuthology.orchestra.run.smithi114.stdout: "critical", 2024-04-23T10:41:08.035 INFO:teuthology.orchestra.run.smithi114.stdout: "debug", 2024-04-23T10:41:08.035 INFO:teuthology.orchestra.run.smithi114.stdout: "error", 2024-04-23T10:41:08.035 INFO:teuthology.orchestra.run.smithi114.stdout: "info", 2024-04-23T10:41:08.035 INFO:teuthology.orchestra.run.smithi114.stdout: "warning" 2024-04-23T10:41:08.036 INFO:teuthology.orchestra.run.smithi114.stdout: ], 2024-04-23T10:41:08.036 INFO:teuthology.orchestra.run.smithi114.stdout: "desc": "", 2024-04-23T10:41:08.036 INFO:teuthology.orchestra.run.smithi114.stdout: "long_desc": "", 2024-04-23T10:41:08.036 INFO:teuthology.orchestra.run.smithi114.stdout: "tags": [], 2024-04-23T10:41:08.036 INFO:teuthology.orchestra.run.smithi114.stdout: "see_also": [] 2024-04-23T10:41:08.036 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-23T10:41:08.036 INFO:teuthology.orchestra.run.smithi114.stdout: "log_to_file": { 2024-04-23T10:41:08.037 INFO:teuthology.orchestra.run.smithi114.stdout: "name": "log_to_file", 2024-04-23T10:41:08.037 INFO:teuthology.orchestra.run.smithi114.stdout: "type": "bool", 2024-04-23T10:41:08.037 INFO:teuthology.orchestra.run.smithi114.stdout: "level": "advanced", 2024-04-23T10:41:08.037 INFO:teuthology.orchestra.run.smithi114.stdout: "flags": 1, 2024-04-23T10:41:08.037 INFO:teuthology.orchestra.run.smithi114.stdout: "default_value": "False", 2024-04-23T10:41:08.037 INFO:teuthology.orchestra.run.smithi114.stdout: "min": "", 2024-04-23T10:41:08.037 INFO:teuthology.orchestra.run.smithi114.stdout: "max": "", 2024-04-23T10:41:08.038 INFO:teuthology.orchestra.run.smithi114.stdout: "enum_allowed": [], 2024-04-23T10:41:08.038 INFO:teuthology.orchestra.run.smithi114.stdout: "desc": "", 2024-04-23T10:41:08.038 INFO:teuthology.orchestra.run.smithi114.stdout: "long_desc": "", 2024-04-23T10:41:08.038 INFO:teuthology.orchestra.run.smithi114.stdout: "tags": [], 2024-04-23T10:41:08.038 INFO:teuthology.orchestra.run.smithi114.stdout: "see_also": [] 2024-04-23T10:41:08.038 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-23T10:41:08.039 INFO:teuthology.orchestra.run.smithi114.stdout: "smtp_destination": { 2024-04-23T10:41:08.039 INFO:teuthology.orchestra.run.smithi114.stdout: "name": "smtp_destination", 2024-04-23T10:41:08.039 INFO:teuthology.orchestra.run.smithi114.stdout: "type": "str", 2024-04-23T10:41:08.039 INFO:teuthology.orchestra.run.smithi114.stdout: "level": "advanced", 2024-04-23T10:41:08.039 INFO:teuthology.orchestra.run.smithi114.stdout: "flags": 1, 2024-04-23T10:41:08.039 INFO:teuthology.orchestra.run.smithi114.stdout: "default_value": "", 2024-04-23T10:41:08.039 INFO:teuthology.orchestra.run.smithi114.stdout: "min": "", 2024-04-23T10:41:08.040 INFO:teuthology.orchestra.run.smithi114.stdout: "max": "", 2024-04-23T10:41:08.040 INFO:teuthology.orchestra.run.smithi114.stdout: "enum_allowed": [], 2024-04-23T10:41:08.040 INFO:teuthology.orchestra.run.smithi114.stdout: "desc": "Email address to send alerts to", 2024-04-23T10:41:08.040 INFO:teuthology.orchestra.run.smithi114.stdout: "long_desc": "", 2024-04-23T10:41:08.040 INFO:teuthology.orchestra.run.smithi114.stdout: "tags": [], 2024-04-23T10:41:08.040 INFO:teuthology.orchestra.run.smithi114.stdout: "see_also": [] 2024-04-23T10:41:08.040 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-23T10:41:08.041 INFO:teuthology.orchestra.run.smithi114.stdout: "smtp_from_name": { 2024-04-23T10:41:08.041 INFO:teuthology.orchestra.run.smithi114.stdout: "name": "smtp_from_name", 2024-04-23T10:41:08.041 INFO:teuthology.orchestra.run.smithi114.stdout: "type": "str", 2024-04-23T10:41:08.041 INFO:teuthology.orchestra.run.smithi114.stdout: "level": "advanced", 2024-04-23T10:41:08.041 INFO:teuthology.orchestra.run.smithi114.stdout: "flags": 1, 2024-04-23T10:41:08.041 INFO:teuthology.orchestra.run.smithi114.stdout: "default_value": "Ceph", 2024-04-23T10:41:08.041 INFO:teuthology.orchestra.run.smithi114.stdout: "min": "", 2024-04-23T10:41:08.042 INFO:teuthology.orchestra.run.smithi114.stdout: "max": "", 2024-04-23T10:41:08.042 INFO:teuthology.orchestra.run.smithi114.stdout: "enum_allowed": [], 2024-04-23T10:41:08.042 INFO:teuthology.orchestra.run.smithi114.stdout: "desc": "Email From: name", 2024-04-23T10:41:08.042 INFO:teuthology.orchestra.run.smithi114.stdout: "long_desc": "", 2024-04-23T10:41:08.042 INFO:teuthology.orchestra.run.smithi114.stdout: "tags": [], 2024-04-23T10:41:08.042 INFO:teuthology.orchestra.run.smithi114.stdout: "see_also": [] 2024-04-23T10:41:08.043 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-23T10:41:08.043 INFO:teuthology.orchestra.run.smithi114.stdout: "smtp_host": { 2024-04-23T10:41:08.043 INFO:teuthology.orchestra.run.smithi114.stdout: "name": "smtp_host", 2024-04-23T10:41:08.043 INFO:teuthology.orchestra.run.smithi114.stdout: "type": "str", 2024-04-23T10:41:08.043 INFO:teuthology.orchestra.run.smithi114.stdout: "level": "advanced", 2024-04-23T10:41:08.043 INFO:teuthology.orchestra.run.smithi114.stdout: "flags": 1, 2024-04-23T10:41:08.043 INFO:teuthology.orchestra.run.smithi114.stdout: "default_value": "", 2024-04-23T10:41:08.043 INFO:teuthology.orchestra.run.smithi114.stdout: "min": "", 2024-04-23T10:41:08.044 INFO:teuthology.orchestra.run.smithi114.stdout: "max": "", 2024-04-23T10:41:08.044 INFO:teuthology.orchestra.run.smithi114.stdout: "enum_allowed": [], 2024-04-23T10:41:08.044 INFO:teuthology.orchestra.run.smithi114.stdout: "desc": "SMTP server", 2024-04-23T10:41:08.044 INFO:teuthology.orchestra.run.smithi114.stdout: "long_desc": "", 2024-04-23T10:41:08.044 INFO:teuthology.orchestra.run.smithi114.stdout: "tags": [], 2024-04-23T10:41:08.044 INFO:teuthology.orchestra.run.smithi114.stdout: "see_also": [] 2024-04-23T10:41:08.045 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-23T10:41:08.045 INFO:teuthology.orchestra.run.smithi114.stdout: "smtp_password": { 2024-04-23T10:41:08.045 INFO:teuthology.orchestra.run.smithi114.stdout: "name": "smtp_password", 2024-04-23T10:41:08.045 INFO:teuthology.orchestra.run.smithi114.stdout: "type": "str", 2024-04-23T10:41:08.045 INFO:teuthology.orchestra.run.smithi114.stdout: "level": "advanced", 2024-04-23T10:41:08.045 INFO:teuthology.orchestra.run.smithi114.stdout: "flags": 1, 2024-04-23T10:41:08.045 INFO:teuthology.orchestra.run.smithi114.stdout: "default_value": "", 2024-04-23T10:41:08.045 INFO:teuthology.orchestra.run.smithi114.stdout: "min": "", 2024-04-23T10:41:08.046 INFO:teuthology.orchestra.run.smithi114.stdout: "max": "", 2024-04-23T10:41:08.046 INFO:teuthology.orchestra.run.smithi114.stdout: "enum_allowed": [], 2024-04-23T10:41:08.046 INFO:teuthology.orchestra.run.smithi114.stdout: "desc": "Password to authenticate with", 2024-04-23T10:41:08.046 INFO:teuthology.orchestra.run.smithi114.stdout: "long_desc": "", 2024-04-23T10:41:08.046 INFO:teuthology.orchestra.run.smithi114.stdout: "tags": [], 2024-04-23T10:41:08.046 INFO:teuthology.orchestra.run.smithi114.stdout: "see_also": [] 2024-04-23T10:41:08.047 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-23T10:41:08.047 INFO:teuthology.orchestra.run.smithi114.stdout: "smtp_port": { 2024-04-23T10:41:08.047 INFO:teuthology.orchestra.run.smithi114.stdout: "name": "smtp_port", 2024-04-23T10:41:08.047 INFO:teuthology.orchestra.run.smithi114.stdout: "type": "int", 2024-04-23T10:41:08.047 INFO:teuthology.orchestra.run.smithi114.stdout: "level": "advanced", 2024-04-23T10:41:08.047 INFO:teuthology.orchestra.run.smithi114.stdout: "flags": 1, 2024-04-23T10:41:08.047 INFO:teuthology.orchestra.run.smithi114.stdout: "default_value": "465", 2024-04-23T10:41:08.047 INFO:teuthology.orchestra.run.smithi114.stdout: "min": "", 2024-04-23T10:41:08.048 INFO:teuthology.orchestra.run.smithi114.stdout: "max": "", 2024-04-23T10:41:08.048 INFO:teuthology.orchestra.run.smithi114.stdout: "enum_allowed": [], 2024-04-23T10:41:08.048 INFO:teuthology.orchestra.run.smithi114.stdout: "desc": "SMTP port", 2024-04-23T10:41:08.048 INFO:teuthology.orchestra.run.smithi114.stdout: "long_desc": "", 2024-04-23T10:41:08.048 INFO:teuthology.orchestra.run.smithi114.stdout: "tags": [], 2024-04-23T10:41:08.048 INFO:teuthology.orchestra.run.smithi114.stdout: "see_also": [] 2024-04-23T10:41:08.048 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-23T10:41:08.049 INFO:teuthology.orchestra.run.smithi114.stdout: "smtp_sender": { 2024-04-23T10:41:08.049 INFO:teuthology.orchestra.run.smithi114.stdout: "name": "smtp_sender", 2024-04-23T10:41:08.049 INFO:teuthology.orchestra.run.smithi114.stdout: "type": "str", 2024-04-23T10:41:08.049 INFO:teuthology.orchestra.run.smithi114.stdout: "level": "advanced", 2024-04-23T10:41:08.049 INFO:teuthology.orchestra.run.smithi114.stdout: "flags": 1, 2024-04-23T10:41:08.049 INFO:teuthology.orchestra.run.smithi114.stdout: "default_value": "", 2024-04-23T10:41:08.049 INFO:teuthology.orchestra.run.smithi114.stdout: "min": "", 2024-04-23T10:41:08.049 INFO:teuthology.orchestra.run.smithi114.stdout: "max": "", 2024-04-23T10:41:08.050 INFO:teuthology.orchestra.run.smithi114.stdout: "enum_allowed": [], 2024-04-23T10:41:08.050 INFO:teuthology.orchestra.run.smithi114.stdout: "desc": "SMTP envelope sender", 2024-04-23T10:41:08.050 INFO:teuthology.orchestra.run.smithi114.stdout: "long_desc": "", 2024-04-23T10:41:08.050 INFO:teuthology.orchestra.run.smithi114.stdout: "tags": [], 2024-04-23T10:41:08.050 INFO:teuthology.orchestra.run.smithi114.stdout: "see_also": [] 2024-04-23T10:41:08.050 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-23T10:41:08.051 INFO:teuthology.orchestra.run.smithi114.stdout: "smtp_ssl": { 2024-04-23T10:41:08.051 INFO:teuthology.orchestra.run.smithi114.stdout: "name": "smtp_ssl", 2024-04-23T10:41:08.051 INFO:teuthology.orchestra.run.smithi114.stdout: "type": "bool", 2024-04-23T10:41:08.051 INFO:teuthology.orchestra.run.smithi114.stdout: "level": "advanced", 2024-04-23T10:41:08.051 INFO:teuthology.orchestra.run.smithi114.stdout: "flags": 1, 2024-04-23T10:41:08.051 INFO:teuthology.orchestra.run.smithi114.stdout: "default_value": "True", 2024-04-23T10:41:08.051 INFO:teuthology.orchestra.run.smithi114.stdout: "min": "", 2024-04-23T10:41:08.052 INFO:teuthology.orchestra.run.smithi114.stdout: "max": "", 2024-04-23T10:41:08.052 INFO:teuthology.orchestra.run.smithi114.stdout: "enum_allowed": [], 2024-04-23T10:41:08.052 INFO:teuthology.orchestra.run.smithi114.stdout: "desc": "Use SSL to connect to SMTP server", 2024-04-23T10:41:08.052 INFO:teuthology.orchestra.run.smithi114.stdout: "long_desc": "", 2024-04-23T10:41:08.052 INFO:teuthology.orchestra.run.smithi114.stdout: "tags": [], 2024-04-23T10:41:08.052 INFO:teuthology.orchestra.run.smithi114.stdout: "see_also": [] 2024-04-23T10:41:08.052 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-23T10:41:08.053 INFO:teuthology.orchestra.run.smithi114.stdout: "smtp_user": { 2024-04-23T10:41:08.053 INFO:teuthology.orchestra.run.smithi114.stdout: "name": "smtp_user", 2024-04-23T10:41:08.053 INFO:teuthology.orchestra.run.smithi114.stdout: "type": "str", 2024-04-23T10:41:08.053 INFO:teuthology.orchestra.run.smithi114.stdout: "level": "advanced", 2024-04-23T10:41:08.053 INFO:teuthology.orchestra.run.smithi114.stdout: "flags": 1, 2024-04-23T10:41:08.053 INFO:teuthology.orchestra.run.smithi114.stdout: "default_value": "", 2024-04-23T10:41:08.053 INFO:teuthology.orchestra.run.smithi114.stdout: "min": "", 2024-04-23T10:41:08.053 INFO:teuthology.orchestra.run.smithi114.stdout: "max": "", 2024-04-23T10:41:08.054 INFO:teuthology.orchestra.run.smithi114.stdout: "enum_allowed": [], 2024-04-23T10:41:08.054 INFO:teuthology.orchestra.run.smithi114.stdout: "desc": "User to authenticate as", 2024-04-23T10:41:08.054 INFO:teuthology.orchestra.run.smithi114.stdout: "long_desc": "", 2024-04-23T10:41:08.054 INFO:teuthology.orchestra.run.smithi114.stdout: "tags": [], 2024-04-23T10:41:08.054 INFO:teuthology.orchestra.run.smithi114.stdout: "see_also": [] 2024-04-23T10:41:08.054 INFO:teuthology.orchestra.run.smithi114.stdout: } 2024-04-23T10:41:08.054 INFO:teuthology.orchestra.run.smithi114.stdout: } 2024-04-23T10:41:08.055 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-23T10:41:08.055 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-04-23T10:41:08.055 INFO:teuthology.orchestra.run.smithi114.stdout: "name": "balancer", 2024-04-23T10:41:08.055 INFO:teuthology.orchestra.run.smithi114.stdout: "can_run": true, 2024-04-23T10:41:08.055 INFO:teuthology.orchestra.run.smithi114.stdout: "error_string": "", 2024-04-23T10:41:08.055 INFO:teuthology.orchestra.run.smithi114.stdout: "module_options": { 2024-04-23T10:41:08.055 INFO:teuthology.orchestra.run.smithi114.stdout: "active": { 2024-04-23T10:41:08.056 INFO:teuthology.orchestra.run.smithi114.stdout: "name": "active", 2024-04-23T10:41:08.056 INFO:teuthology.orchestra.run.smithi114.stdout: "type": "bool", 2024-04-23T10:41:08.056 INFO:teuthology.orchestra.run.smithi114.stdout: "level": "advanced", 2024-04-23T10:41:08.056 INFO:teuthology.orchestra.run.smithi114.stdout: "flags": 1, 2024-04-23T10:41:08.056 INFO:teuthology.orchestra.run.smithi114.stdout: "default_value": "True", 2024-04-23T10:41:08.056 INFO:teuthology.orchestra.run.smithi114.stdout: "min": "", 2024-04-23T10:41:08.056 INFO:teuthology.orchestra.run.smithi114.stdout: "max": "", 2024-04-23T10:41:08.057 INFO:teuthology.orchestra.run.smithi114.stdout: "enum_allowed": [], 2024-04-23T10:41:08.057 INFO:teuthology.orchestra.run.smithi114.stdout: "desc": "automatically balance PGs across cluster", 2024-04-23T10:41:08.057 INFO:teuthology.orchestra.run.smithi114.stdout: "long_desc": "", 2024-04-23T10:41:08.057 INFO:teuthology.orchestra.run.smithi114.stdout: "tags": [], 2024-04-23T10:41:08.057 INFO:teuthology.orchestra.run.smithi114.stdout: "see_also": [] 2024-04-23T10:41:08.057 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-23T10:41:08.057 INFO:teuthology.orchestra.run.smithi114.stdout: "begin_time": { 2024-04-23T10:41:08.057 INFO:teuthology.orchestra.run.smithi114.stdout: "name": "begin_time", 2024-04-23T10:41:08.058 INFO:teuthology.orchestra.run.smithi114.stdout: "type": "str", 2024-04-23T10:41:08.058 INFO:teuthology.orchestra.run.smithi114.stdout: "level": "advanced", 2024-04-23T10:41:08.058 INFO:teuthology.orchestra.run.smithi114.stdout: "flags": 1, 2024-04-23T10:41:08.058 INFO:teuthology.orchestra.run.smithi114.stdout: "default_value": "0000", 2024-04-23T10:41:08.058 INFO:teuthology.orchestra.run.smithi114.stdout: "min": "", 2024-04-23T10:41:08.058 INFO:teuthology.orchestra.run.smithi114.stdout: "max": "", 2024-04-23T10:41:08.058 INFO:teuthology.orchestra.run.smithi114.stdout: "enum_allowed": [], 2024-04-23T10:41:08.059 INFO:teuthology.orchestra.run.smithi114.stdout: "desc": "beginning time of day to automatically balance", 2024-04-23T10:41:08.059 INFO:teuthology.orchestra.run.smithi114.stdout: "long_desc": "This is a time of day in the format HHMM.", 2024-04-23T10:41:08.059 INFO:teuthology.orchestra.run.smithi114.stdout: "tags": [], 2024-04-23T10:41:08.059 INFO:teuthology.orchestra.run.smithi114.stdout: "see_also": [] 2024-04-23T10:41:08.059 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-23T10:41:08.059 INFO:teuthology.orchestra.run.smithi114.stdout: "begin_weekday": { 2024-04-23T10:41:08.059 INFO:teuthology.orchestra.run.smithi114.stdout: "name": "begin_weekday", 2024-04-23T10:41:08.060 INFO:teuthology.orchestra.run.smithi114.stdout: "type": "uint", 2024-04-23T10:41:08.060 INFO:teuthology.orchestra.run.smithi114.stdout: "level": "advanced", 2024-04-23T10:41:08.060 INFO:teuthology.orchestra.run.smithi114.stdout: "flags": 1, 2024-04-23T10:41:08.060 INFO:teuthology.orchestra.run.smithi114.stdout: "default_value": "0", 2024-04-23T10:41:08.060 INFO:teuthology.orchestra.run.smithi114.stdout: "min": "0", 2024-04-23T10:41:08.060 INFO:teuthology.orchestra.run.smithi114.stdout: "max": "6", 2024-04-23T10:41:08.060 INFO:teuthology.orchestra.run.smithi114.stdout: "enum_allowed": [], 2024-04-23T10:41:08.060 INFO:teuthology.orchestra.run.smithi114.stdout: "desc": "Restrict automatic balancing to this day of the week or later", 2024-04-23T10:41:08.061 INFO:teuthology.orchestra.run.smithi114.stdout: "long_desc": "0 = Sunday, 1 = Monday, etc.", 2024-04-23T10:41:08.061 INFO:teuthology.orchestra.run.smithi114.stdout: "tags": [], 2024-04-23T10:41:08.061 INFO:teuthology.orchestra.run.smithi114.stdout: "see_also": [] 2024-04-23T10:41:08.061 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-23T10:41:08.061 INFO:teuthology.orchestra.run.smithi114.stdout: "crush_compat_max_iterations": { 2024-04-23T10:41:08.061 INFO:teuthology.orchestra.run.smithi114.stdout: "name": "crush_compat_max_iterations", 2024-04-23T10:41:08.061 INFO:teuthology.orchestra.run.smithi114.stdout: "type": "uint", 2024-04-23T10:41:08.061 INFO:teuthology.orchestra.run.smithi114.stdout: "level": "advanced", 2024-04-23T10:41:08.062 INFO:teuthology.orchestra.run.smithi114.stdout: "flags": 1, 2024-04-23T10:41:08.062 INFO:teuthology.orchestra.run.smithi114.stdout: "default_value": "25", 2024-04-23T10:41:08.062 INFO:teuthology.orchestra.run.smithi114.stdout: "min": "1", 2024-04-23T10:41:08.062 INFO:teuthology.orchestra.run.smithi114.stdout: "max": "250", 2024-04-23T10:41:08.062 INFO:teuthology.orchestra.run.smithi114.stdout: "enum_allowed": [], 2024-04-23T10:41:08.062 INFO:teuthology.orchestra.run.smithi114.stdout: "desc": "maximum number of iterations to attempt optimization", 2024-04-23T10:41:08.062 INFO:teuthology.orchestra.run.smithi114.stdout: "long_desc": "", 2024-04-23T10:41:08.063 INFO:teuthology.orchestra.run.smithi114.stdout: "tags": [], 2024-04-23T10:41:08.063 INFO:teuthology.orchestra.run.smithi114.stdout: "see_also": [] 2024-04-23T10:41:08.063 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-23T10:41:08.063 INFO:teuthology.orchestra.run.smithi114.stdout: "crush_compat_metrics": { 2024-04-23T10:41:08.063 INFO:teuthology.orchestra.run.smithi114.stdout: "name": "crush_compat_metrics", 2024-04-23T10:41:08.063 INFO:teuthology.orchestra.run.smithi114.stdout: "type": "str", 2024-04-23T10:41:08.063 INFO:teuthology.orchestra.run.smithi114.stdout: "level": "advanced", 2024-04-23T10:41:08.063 INFO:teuthology.orchestra.run.smithi114.stdout: "flags": 1, 2024-04-23T10:41:08.064 INFO:teuthology.orchestra.run.smithi114.stdout: "default_value": "pgs,objects,bytes", 2024-04-23T10:41:08.064 INFO:teuthology.orchestra.run.smithi114.stdout: "min": "", 2024-04-23T10:41:08.064 INFO:teuthology.orchestra.run.smithi114.stdout: "max": "", 2024-04-23T10:41:08.064 INFO:teuthology.orchestra.run.smithi114.stdout: "enum_allowed": [], 2024-04-23T10:41:08.064 INFO:teuthology.orchestra.run.smithi114.stdout: "desc": "metrics with which to calculate OSD utilization", 2024-04-23T10:41:08.064 INFO:teuthology.orchestra.run.smithi114.stdout: "long_desc": "Value is a list of one or more of \"pgs\", \"objects\", or \"bytes\", and indicates which metrics to use to balance utilization.", 2024-04-23T10:41:08.064 INFO:teuthology.orchestra.run.smithi114.stdout: "tags": [], 2024-04-23T10:41:08.064 INFO:teuthology.orchestra.run.smithi114.stdout: "see_also": [] 2024-04-23T10:41:08.065 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-23T10:41:08.065 INFO:teuthology.orchestra.run.smithi114.stdout: "crush_compat_step": { 2024-04-23T10:41:08.065 INFO:teuthology.orchestra.run.smithi114.stdout: "name": "crush_compat_step", 2024-04-23T10:41:08.065 INFO:teuthology.orchestra.run.smithi114.stdout: "type": "float", 2024-04-23T10:41:08.065 INFO:teuthology.orchestra.run.smithi114.stdout: "level": "advanced", 2024-04-23T10:41:08.065 INFO:teuthology.orchestra.run.smithi114.stdout: "flags": 1, 2024-04-23T10:41:08.065 INFO:teuthology.orchestra.run.smithi114.stdout: "default_value": "0.5", 2024-04-23T10:41:08.066 INFO:teuthology.orchestra.run.smithi114.stdout: "min": "0.001", 2024-04-23T10:41:08.066 INFO:teuthology.orchestra.run.smithi114.stdout: "max": "0.999", 2024-04-23T10:41:08.066 INFO:teuthology.orchestra.run.smithi114.stdout: "enum_allowed": [], 2024-04-23T10:41:08.066 INFO:teuthology.orchestra.run.smithi114.stdout: "desc": "aggressiveness of optimization", 2024-04-23T10:41:08.066 INFO:teuthology.orchestra.run.smithi114.stdout: "long_desc": ".99 is very aggressive, .01 is less aggressive", 2024-04-23T10:41:08.066 INFO:teuthology.orchestra.run.smithi114.stdout: "tags": [], 2024-04-23T10:41:08.066 INFO:teuthology.orchestra.run.smithi114.stdout: "see_also": [] 2024-04-23T10:41:08.066 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-23T10:41:08.067 INFO:teuthology.orchestra.run.smithi114.stdout: "end_time": { 2024-04-23T10:41:08.067 INFO:teuthology.orchestra.run.smithi114.stdout: "name": "end_time", 2024-04-23T10:41:08.067 INFO:teuthology.orchestra.run.smithi114.stdout: "type": "str", 2024-04-23T10:41:08.067 INFO:teuthology.orchestra.run.smithi114.stdout: "level": "advanced", 2024-04-23T10:41:08.067 INFO:teuthology.orchestra.run.smithi114.stdout: "flags": 1, 2024-04-23T10:41:08.067 INFO:teuthology.orchestra.run.smithi114.stdout: "default_value": "2359", 2024-04-23T10:41:08.067 INFO:teuthology.orchestra.run.smithi114.stdout: "min": "", 2024-04-23T10:41:08.068 INFO:teuthology.orchestra.run.smithi114.stdout: "max": "", 2024-04-23T10:41:08.068 INFO:teuthology.orchestra.run.smithi114.stdout: "enum_allowed": [], 2024-04-23T10:41:08.068 INFO:teuthology.orchestra.run.smithi114.stdout: "desc": "ending time of day to automatically balance", 2024-04-23T10:41:08.068 INFO:teuthology.orchestra.run.smithi114.stdout: "long_desc": "This is a time of day in the format HHMM.", 2024-04-23T10:41:08.068 INFO:teuthology.orchestra.run.smithi114.stdout: "tags": [], 2024-04-23T10:41:08.068 INFO:teuthology.orchestra.run.smithi114.stdout: "see_also": [] 2024-04-23T10:41:08.068 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-23T10:41:08.068 INFO:teuthology.orchestra.run.smithi114.stdout: "end_weekday": { 2024-04-23T10:41:08.069 INFO:teuthology.orchestra.run.smithi114.stdout: "name": "end_weekday", 2024-04-23T10:41:08.069 INFO:teuthology.orchestra.run.smithi114.stdout: "type": "uint", 2024-04-23T10:41:08.069 INFO:teuthology.orchestra.run.smithi114.stdout: "level": "advanced", 2024-04-23T10:41:08.069 INFO:teuthology.orchestra.run.smithi114.stdout: "flags": 1, 2024-04-23T10:41:08.069 INFO:teuthology.orchestra.run.smithi114.stdout: "default_value": "0", 2024-04-23T10:41:08.069 INFO:teuthology.orchestra.run.smithi114.stdout: "min": "0", 2024-04-23T10:41:08.069 INFO:teuthology.orchestra.run.smithi114.stdout: "max": "6", 2024-04-23T10:41:08.069 INFO:teuthology.orchestra.run.smithi114.stdout: "enum_allowed": [], 2024-04-23T10:41:08.070 INFO:teuthology.orchestra.run.smithi114.stdout: "desc": "Restrict automatic balancing to days of the week earlier than this", 2024-04-23T10:41:08.070 INFO:teuthology.orchestra.run.smithi114.stdout: "long_desc": "0 = Sunday, 1 = Monday, etc.", 2024-04-23T10:41:08.070 INFO:teuthology.orchestra.run.smithi114.stdout: "tags": [], 2024-04-23T10:41:08.070 INFO:teuthology.orchestra.run.smithi114.stdout: "see_also": [] 2024-04-23T10:41:08.070 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-23T10:41:08.070 INFO:teuthology.orchestra.run.smithi114.stdout: "log_level": { 2024-04-23T10:41:08.070 INFO:teuthology.orchestra.run.smithi114.stdout: "name": "log_level", 2024-04-23T10:41:08.071 INFO:teuthology.orchestra.run.smithi114.stdout: "type": "str", 2024-04-23T10:41:08.071 INFO:teuthology.orchestra.run.smithi114.stdout: "level": "advanced", 2024-04-23T10:41:08.071 INFO:teuthology.orchestra.run.smithi114.stdout: "flags": 1, 2024-04-23T10:41:08.071 INFO:teuthology.orchestra.run.smithi114.stdout: "default_value": "", 2024-04-23T10:41:08.071 INFO:teuthology.orchestra.run.smithi114.stdout: "min": "", 2024-04-23T10:41:08.071 INFO:teuthology.orchestra.run.smithi114.stdout: "max": "", 2024-04-23T10:41:08.071 INFO:teuthology.orchestra.run.smithi114.stdout: "enum_allowed": [ 2024-04-23T10:41:08.071 INFO:teuthology.orchestra.run.smithi114.stdout: "", 2024-04-23T10:41:08.072 INFO:teuthology.orchestra.run.smithi114.stdout: "critical", 2024-04-23T10:41:08.072 INFO:teuthology.orchestra.run.smithi114.stdout: "debug", 2024-04-23T10:41:08.072 INFO:teuthology.orchestra.run.smithi114.stdout: "error", 2024-04-23T10:41:08.072 INFO:teuthology.orchestra.run.smithi114.stdout: "info", 2024-04-23T10:41:08.072 INFO:teuthology.orchestra.run.smithi114.stdout: "warning" 2024-04-23T10:41:08.072 INFO:teuthology.orchestra.run.smithi114.stdout: ], 2024-04-23T10:41:08.072 INFO:teuthology.orchestra.run.smithi114.stdout: "desc": "", 2024-04-23T10:41:08.072 INFO:teuthology.orchestra.run.smithi114.stdout: "long_desc": "", 2024-04-23T10:41:08.073 INFO:teuthology.orchestra.run.smithi114.stdout: "tags": [], 2024-04-23T10:41:08.073 INFO:teuthology.orchestra.run.smithi114.stdout: "see_also": [] 2024-04-23T10:41:08.073 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-23T10:41:08.073 INFO:teuthology.orchestra.run.smithi114.stdout: "log_to_cluster": { 2024-04-23T10:41:08.073 INFO:teuthology.orchestra.run.smithi114.stdout: "name": "log_to_cluster", 2024-04-23T10:41:08.073 INFO:teuthology.orchestra.run.smithi114.stdout: "type": "bool", 2024-04-23T10:41:08.073 INFO:teuthology.orchestra.run.smithi114.stdout: "level": "advanced", 2024-04-23T10:41:08.073 INFO:teuthology.orchestra.run.smithi114.stdout: "flags": 1, 2024-04-23T10:41:08.074 INFO:teuthology.orchestra.run.smithi114.stdout: "default_value": "False", 2024-04-23T10:41:08.074 INFO:teuthology.orchestra.run.smithi114.stdout: "min": "", 2024-04-23T10:41:08.074 INFO:teuthology.orchestra.run.smithi114.stdout: "max": "", 2024-04-23T10:41:08.074 INFO:teuthology.orchestra.run.smithi114.stdout: "enum_allowed": [], 2024-04-23T10:41:08.074 INFO:teuthology.orchestra.run.smithi114.stdout: "desc": "", 2024-04-23T10:41:08.074 INFO:teuthology.orchestra.run.smithi114.stdout: "long_desc": "", 2024-04-23T10:41:08.074 INFO:teuthology.orchestra.run.smithi114.stdout: "tags": [], 2024-04-23T10:41:08.075 INFO:teuthology.orchestra.run.smithi114.stdout: "see_also": [] 2024-04-23T10:41:08.075 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-23T10:41:08.075 INFO:teuthology.orchestra.run.smithi114.stdout: "log_to_cluster_level": { 2024-04-23T10:41:08.075 INFO:teuthology.orchestra.run.smithi114.stdout: "name": "log_to_cluster_level", 2024-04-23T10:41:08.075 INFO:teuthology.orchestra.run.smithi114.stdout: "type": "str", 2024-04-23T10:41:08.075 INFO:teuthology.orchestra.run.smithi114.stdout: "level": "advanced", 2024-04-23T10:41:08.075 INFO:teuthology.orchestra.run.smithi114.stdout: "flags": 1, 2024-04-23T10:41:08.075 INFO:teuthology.orchestra.run.smithi114.stdout: "default_value": "info", 2024-04-23T10:41:08.076 INFO:teuthology.orchestra.run.smithi114.stdout: "min": "", 2024-04-23T10:41:08.076 INFO:teuthology.orchestra.run.smithi114.stdout: "max": "", 2024-04-23T10:41:08.076 INFO:teuthology.orchestra.run.smithi114.stdout: "enum_allowed": [ 2024-04-23T10:41:08.076 INFO:teuthology.orchestra.run.smithi114.stdout: "", 2024-04-23T10:41:08.076 INFO:teuthology.orchestra.run.smithi114.stdout: "critical", 2024-04-23T10:41:08.076 INFO:teuthology.orchestra.run.smithi114.stdout: "debug", 2024-04-23T10:41:08.076 INFO:teuthology.orchestra.run.smithi114.stdout: "error", 2024-04-23T10:41:08.076 INFO:teuthology.orchestra.run.smithi114.stdout: "info", 2024-04-23T10:41:08.077 INFO:teuthology.orchestra.run.smithi114.stdout: "warning" 2024-04-23T10:41:08.077 INFO:teuthology.orchestra.run.smithi114.stdout: ], 2024-04-23T10:41:08.077 INFO:teuthology.orchestra.run.smithi114.stdout: "desc": "", 2024-04-23T10:41:08.077 INFO:teuthology.orchestra.run.smithi114.stdout: "long_desc": "", 2024-04-23T10:41:08.077 INFO:teuthology.orchestra.run.smithi114.stdout: "tags": [], 2024-04-23T10:41:08.077 INFO:teuthology.orchestra.run.smithi114.stdout: "see_also": [] 2024-04-23T10:41:08.077 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-23T10:41:08.077 INFO:teuthology.orchestra.run.smithi114.stdout: "log_to_file": { 2024-04-23T10:41:08.078 INFO:teuthology.orchestra.run.smithi114.stdout: "name": "log_to_file", 2024-04-23T10:41:08.078 INFO:teuthology.orchestra.run.smithi114.stdout: "type": "bool", 2024-04-23T10:41:08.078 INFO:teuthology.orchestra.run.smithi114.stdout: "level": "advanced", 2024-04-23T10:41:08.078 INFO:teuthology.orchestra.run.smithi114.stdout: "flags": 1, 2024-04-23T10:41:08.078 INFO:teuthology.orchestra.run.smithi114.stdout: "default_value": "False", 2024-04-23T10:41:08.078 INFO:teuthology.orchestra.run.smithi114.stdout: "min": "", 2024-04-23T10:41:08.078 INFO:teuthology.orchestra.run.smithi114.stdout: "max": "", 2024-04-23T10:41:08.079 INFO:teuthology.orchestra.run.smithi114.stdout: "enum_allowed": [], 2024-04-23T10:41:08.079 INFO:teuthology.orchestra.run.smithi114.stdout: "desc": "", 2024-04-23T10:41:08.079 INFO:teuthology.orchestra.run.smithi114.stdout: "long_desc": "", 2024-04-23T10:41:08.079 INFO:teuthology.orchestra.run.smithi114.stdout: "tags": [], 2024-04-23T10:41:08.079 INFO:teuthology.orchestra.run.smithi114.stdout: "see_also": [] 2024-04-23T10:41:08.079 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-23T10:41:08.079 INFO:teuthology.orchestra.run.smithi114.stdout: "min_score": { 2024-04-23T10:41:08.079 INFO:teuthology.orchestra.run.smithi114.stdout: "name": "min_score", 2024-04-23T10:41:08.080 INFO:teuthology.orchestra.run.smithi114.stdout: "type": "float", 2024-04-23T10:41:08.080 INFO:teuthology.orchestra.run.smithi114.stdout: "level": "advanced", 2024-04-23T10:41:08.080 INFO:teuthology.orchestra.run.smithi114.stdout: "flags": 1, 2024-04-23T10:41:08.080 INFO:teuthology.orchestra.run.smithi114.stdout: "default_value": "0", 2024-04-23T10:41:08.080 INFO:teuthology.orchestra.run.smithi114.stdout: "min": "", 2024-04-23T10:41:08.080 INFO:teuthology.orchestra.run.smithi114.stdout: "max": "", 2024-04-23T10:41:08.080 INFO:teuthology.orchestra.run.smithi114.stdout: "enum_allowed": [], 2024-04-23T10:41:08.080 INFO:teuthology.orchestra.run.smithi114.stdout: "desc": "minimum score, below which no optimization is attempted", 2024-04-23T10:41:08.081 INFO:teuthology.orchestra.run.smithi114.stdout: "long_desc": "", 2024-04-23T10:41:08.081 INFO:teuthology.orchestra.run.smithi114.stdout: "tags": [], 2024-04-23T10:41:08.081 INFO:teuthology.orchestra.run.smithi114.stdout: "see_also": [] 2024-04-23T10:41:08.081 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-23T10:41:08.081 INFO:teuthology.orchestra.run.smithi114.stdout: "mode": { 2024-04-23T10:41:08.081 INFO:teuthology.orchestra.run.smithi114.stdout: "name": "mode", 2024-04-23T10:41:08.081 INFO:teuthology.orchestra.run.smithi114.stdout: "type": "str", 2024-04-23T10:41:08.081 INFO:teuthology.orchestra.run.smithi114.stdout: "level": "advanced", 2024-04-23T10:41:08.082 INFO:teuthology.orchestra.run.smithi114.stdout: "flags": 1, 2024-04-23T10:41:08.082 INFO:teuthology.orchestra.run.smithi114.stdout: "default_value": "upmap", 2024-04-23T10:41:08.082 INFO:teuthology.orchestra.run.smithi114.stdout: "min": "", 2024-04-23T10:41:08.082 INFO:teuthology.orchestra.run.smithi114.stdout: "max": "", 2024-04-23T10:41:08.082 INFO:teuthology.orchestra.run.smithi114.stdout: "enum_allowed": [ 2024-04-23T10:41:08.082 INFO:teuthology.orchestra.run.smithi114.stdout: "crush-compat", 2024-04-23T10:41:08.082 INFO:teuthology.orchestra.run.smithi114.stdout: "none", 2024-04-23T10:41:08.082 INFO:teuthology.orchestra.run.smithi114.stdout: "read", 2024-04-23T10:41:08.083 INFO:teuthology.orchestra.run.smithi114.stdout: "upmap", 2024-04-23T10:41:08.083 INFO:teuthology.orchestra.run.smithi114.stdout: "upmap-read" 2024-04-23T10:41:08.083 INFO:teuthology.orchestra.run.smithi114.stdout: ], 2024-04-23T10:41:08.083 INFO:teuthology.orchestra.run.smithi114.stdout: "desc": "Balancer mode", 2024-04-23T10:41:08.083 INFO:teuthology.orchestra.run.smithi114.stdout: "long_desc": "", 2024-04-23T10:41:08.083 INFO:teuthology.orchestra.run.smithi114.stdout: "tags": [], 2024-04-23T10:41:08.083 INFO:teuthology.orchestra.run.smithi114.stdout: "see_also": [] 2024-04-23T10:41:08.083 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-23T10:41:08.084 INFO:teuthology.orchestra.run.smithi114.stdout: "pool_ids": { 2024-04-23T10:41:08.084 INFO:teuthology.orchestra.run.smithi114.stdout: "name": "pool_ids", 2024-04-23T10:41:08.084 INFO:teuthology.orchestra.run.smithi114.stdout: "type": "str", 2024-04-23T10:41:08.084 INFO:teuthology.orchestra.run.smithi114.stdout: "level": "advanced", 2024-04-23T10:41:08.084 INFO:teuthology.orchestra.run.smithi114.stdout: "flags": 1, 2024-04-23T10:41:08.084 INFO:teuthology.orchestra.run.smithi114.stdout: "default_value": "", 2024-04-23T10:41:08.084 INFO:teuthology.orchestra.run.smithi114.stdout: "min": "", 2024-04-23T10:41:08.084 INFO:teuthology.orchestra.run.smithi114.stdout: "max": "", 2024-04-23T10:41:08.085 INFO:teuthology.orchestra.run.smithi114.stdout: "enum_allowed": [], 2024-04-23T10:41:08.085 INFO:teuthology.orchestra.run.smithi114.stdout: "desc": "pools which the automatic balancing will be limited to", 2024-04-23T10:41:08.085 INFO:teuthology.orchestra.run.smithi114.stdout: "long_desc": "", 2024-04-23T10:41:08.085 INFO:teuthology.orchestra.run.smithi114.stdout: "tags": [], 2024-04-23T10:41:08.085 INFO:teuthology.orchestra.run.smithi114.stdout: "see_also": [] 2024-04-23T10:41:08.085 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-23T10:41:08.085 INFO:teuthology.orchestra.run.smithi114.stdout: "sleep_interval": { 2024-04-23T10:41:08.085 INFO:teuthology.orchestra.run.smithi114.stdout: "name": "sleep_interval", 2024-04-23T10:41:08.086 INFO:teuthology.orchestra.run.smithi114.stdout: "type": "secs", 2024-04-23T10:41:08.086 INFO:teuthology.orchestra.run.smithi114.stdout: "level": "advanced", 2024-04-23T10:41:08.086 INFO:teuthology.orchestra.run.smithi114.stdout: "flags": 1, 2024-04-23T10:41:08.086 INFO:teuthology.orchestra.run.smithi114.stdout: "default_value": "60", 2024-04-23T10:41:08.086 INFO:teuthology.orchestra.run.smithi114.stdout: "min": "", 2024-04-23T10:41:08.086 INFO:teuthology.orchestra.run.smithi114.stdout: "max": "", 2024-04-23T10:41:08.086 INFO:teuthology.orchestra.run.smithi114.stdout: "enum_allowed": [], 2024-04-23T10:41:08.086 INFO:teuthology.orchestra.run.smithi114.stdout: "desc": "how frequently to wake up and attempt optimization", 2024-04-23T10:41:08.087 INFO:teuthology.orchestra.run.smithi114.stdout: "long_desc": "", 2024-04-23T10:41:08.087 INFO:teuthology.orchestra.run.smithi114.stdout: "tags": [], 2024-04-23T10:41:08.087 INFO:teuthology.orchestra.run.smithi114.stdout: "see_also": [] 2024-04-23T10:41:08.087 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-23T10:41:08.087 INFO:teuthology.orchestra.run.smithi114.stdout: "upmap_max_deviation": { 2024-04-23T10:41:08.087 INFO:teuthology.orchestra.run.smithi114.stdout: "name": "upmap_max_deviation", 2024-04-23T10:41:08.087 INFO:teuthology.orchestra.run.smithi114.stdout: "type": "int", 2024-04-23T10:41:08.087 INFO:teuthology.orchestra.run.smithi114.stdout: "level": "advanced", 2024-04-23T10:41:08.087 INFO:teuthology.orchestra.run.smithi114.stdout: "flags": 1, 2024-04-23T10:41:08.088 INFO:teuthology.orchestra.run.smithi114.stdout: "default_value": "5", 2024-04-23T10:41:08.088 INFO:teuthology.orchestra.run.smithi114.stdout: "min": "1", 2024-04-23T10:41:08.088 INFO:teuthology.orchestra.run.smithi114.stdout: "max": "", 2024-04-23T10:41:08.088 INFO:teuthology.orchestra.run.smithi114.stdout: "enum_allowed": [], 2024-04-23T10:41:08.088 INFO:teuthology.orchestra.run.smithi114.stdout: "desc": "deviation below which no optimization is attempted", 2024-04-23T10:41:08.088 INFO:teuthology.orchestra.run.smithi114.stdout: "long_desc": "If the number of PGs are within this count then no optimization is attempted", 2024-04-23T10:41:08.088 INFO:teuthology.orchestra.run.smithi114.stdout: "tags": [], 2024-04-23T10:41:08.088 INFO:teuthology.orchestra.run.smithi114.stdout: "see_also": [] 2024-04-23T10:41:08.089 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-23T10:41:08.089 INFO:teuthology.orchestra.run.smithi114.stdout: "upmap_max_optimizations": { 2024-04-23T10:41:08.089 INFO:teuthology.orchestra.run.smithi114.stdout: "name": "upmap_max_optimizations", 2024-04-23T10:41:08.089 INFO:teuthology.orchestra.run.smithi114.stdout: "type": "uint", 2024-04-23T10:41:08.089 INFO:teuthology.orchestra.run.smithi114.stdout: "level": "advanced", 2024-04-23T10:41:08.089 INFO:teuthology.orchestra.run.smithi114.stdout: "flags": 1, 2024-04-23T10:41:08.089 INFO:teuthology.orchestra.run.smithi114.stdout: "default_value": "10", 2024-04-23T10:41:08.089 INFO:teuthology.orchestra.run.smithi114.stdout: "min": "", 2024-04-23T10:41:08.090 INFO:teuthology.orchestra.run.smithi114.stdout: "max": "", 2024-04-23T10:41:08.090 INFO:teuthology.orchestra.run.smithi114.stdout: "enum_allowed": [], 2024-04-23T10:41:08.090 INFO:teuthology.orchestra.run.smithi114.stdout: "desc": "maximum upmap optimizations to make per attempt", 2024-04-23T10:41:08.090 INFO:teuthology.orchestra.run.smithi114.stdout: "long_desc": "", 2024-04-23T10:41:08.090 INFO:teuthology.orchestra.run.smithi114.stdout: "tags": [], 2024-04-23T10:41:08.090 INFO:teuthology.orchestra.run.smithi114.stdout: "see_also": [] 2024-04-23T10:41:08.090 INFO:teuthology.orchestra.run.smithi114.stdout: } 2024-04-23T10:41:08.090 INFO:teuthology.orchestra.run.smithi114.stdout: } 2024-04-23T10:41:08.091 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-23T10:41:08.091 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-04-23T10:41:08.091 INFO:teuthology.orchestra.run.smithi114.stdout: "name": "cephadm", 2024-04-23T10:41:08.091 INFO:teuthology.orchestra.run.smithi114.stdout: "can_run": true, 2024-04-23T10:41:08.091 INFO:teuthology.orchestra.run.smithi114.stdout: "error_string": "", 2024-04-23T10:41:08.091 INFO:teuthology.orchestra.run.smithi114.stdout: "module_options": { 2024-04-23T10:41:08.091 INFO:teuthology.orchestra.run.smithi114.stdout: "agent_down_multiplier": { 2024-04-23T10:41:08.091 INFO:teuthology.orchestra.run.smithi114.stdout: "name": "agent_down_multiplier", 2024-04-23T10:41:08.092 INFO:teuthology.orchestra.run.smithi114.stdout: "type": "float", 2024-04-23T10:41:08.092 INFO:teuthology.orchestra.run.smithi114.stdout: "level": "advanced", 2024-04-23T10:41:08.092 INFO:teuthology.orchestra.run.smithi114.stdout: "flags": 0, 2024-04-23T10:41:08.092 INFO:teuthology.orchestra.run.smithi114.stdout: "default_value": "3.0", 2024-04-23T10:41:08.092 INFO:teuthology.orchestra.run.smithi114.stdout: "min": "", 2024-04-23T10:41:08.092 INFO:teuthology.orchestra.run.smithi114.stdout: "max": "", 2024-04-23T10:41:08.092 INFO:teuthology.orchestra.run.smithi114.stdout: "enum_allowed": [], 2024-04-23T10:41:08.092 INFO:teuthology.orchestra.run.smithi114.stdout: "desc": "Multiplied by agent refresh rate to calculate how long agent must not report before being marked down", 2024-04-23T10:41:08.093 INFO:teuthology.orchestra.run.smithi114.stdout: "long_desc": "", 2024-04-23T10:41:08.093 INFO:teuthology.orchestra.run.smithi114.stdout: "tags": [], 2024-04-23T10:41:08.093 INFO:teuthology.orchestra.run.smithi114.stdout: "see_also": [] 2024-04-23T10:41:08.093 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-23T10:41:08.093 INFO:teuthology.orchestra.run.smithi114.stdout: "agent_refresh_rate": { 2024-04-23T10:41:08.093 INFO:teuthology.orchestra.run.smithi114.stdout: "name": "agent_refresh_rate", 2024-04-23T10:41:08.093 INFO:teuthology.orchestra.run.smithi114.stdout: "type": "secs", 2024-04-23T10:41:08.093 INFO:teuthology.orchestra.run.smithi114.stdout: "level": "advanced", 2024-04-23T10:41:08.093 INFO:teuthology.orchestra.run.smithi114.stdout: "flags": 0, 2024-04-23T10:41:08.094 INFO:teuthology.orchestra.run.smithi114.stdout: "default_value": "20", 2024-04-23T10:41:08.094 INFO:teuthology.orchestra.run.smithi114.stdout: "min": "", 2024-04-23T10:41:08.094 INFO:teuthology.orchestra.run.smithi114.stdout: "max": "", 2024-04-23T10:41:08.094 INFO:teuthology.orchestra.run.smithi114.stdout: "enum_allowed": [], 2024-04-23T10:41:08.094 INFO:teuthology.orchestra.run.smithi114.stdout: "desc": "How often agent on each host will try to gather and send metadata", 2024-04-23T10:41:08.094 INFO:teuthology.orchestra.run.smithi114.stdout: "long_desc": "", 2024-04-23T10:41:08.094 INFO:teuthology.orchestra.run.smithi114.stdout: "tags": [], 2024-04-23T10:41:08.094 INFO:teuthology.orchestra.run.smithi114.stdout: "see_also": [] 2024-04-23T10:41:08.095 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-23T10:41:08.095 INFO:teuthology.orchestra.run.smithi114.stdout: "agent_starting_port": { 2024-04-23T10:41:08.095 INFO:teuthology.orchestra.run.smithi114.stdout: "name": "agent_starting_port", 2024-04-23T10:41:08.095 INFO:teuthology.orchestra.run.smithi114.stdout: "type": "int", 2024-04-23T10:41:08.095 INFO:teuthology.orchestra.run.smithi114.stdout: "level": "advanced", 2024-04-23T10:41:08.095 INFO:teuthology.orchestra.run.smithi114.stdout: "flags": 0, 2024-04-23T10:41:08.095 INFO:teuthology.orchestra.run.smithi114.stdout: "default_value": "4721", 2024-04-23T10:41:08.095 INFO:teuthology.orchestra.run.smithi114.stdout: "min": "", 2024-04-23T10:41:08.095 INFO:teuthology.orchestra.run.smithi114.stdout: "max": "", 2024-04-23T10:41:08.096 INFO:teuthology.orchestra.run.smithi114.stdout: "enum_allowed": [], 2024-04-23T10:41:08.096 INFO:teuthology.orchestra.run.smithi114.stdout: "desc": "First port agent will try to bind to (will also try up to next 1000 subsequent ports if blocked)", 2024-04-23T10:41:08.096 INFO:teuthology.orchestra.run.smithi114.stdout: "long_desc": "", 2024-04-23T10:41:08.096 INFO:teuthology.orchestra.run.smithi114.stdout: "tags": [], 2024-04-23T10:41:08.096 INFO:teuthology.orchestra.run.smithi114.stdout: "see_also": [] 2024-04-23T10:41:08.096 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-23T10:41:08.096 INFO:teuthology.orchestra.run.smithi114.stdout: "allow_ptrace": { 2024-04-23T10:41:08.096 INFO:teuthology.orchestra.run.smithi114.stdout: "name": "allow_ptrace", 2024-04-23T10:41:08.097 INFO:teuthology.orchestra.run.smithi114.stdout: "type": "bool", 2024-04-23T10:41:08.097 INFO:teuthology.orchestra.run.smithi114.stdout: "level": "advanced", 2024-04-23T10:41:08.097 INFO:teuthology.orchestra.run.smithi114.stdout: "flags": 0, 2024-04-23T10:41:08.097 INFO:teuthology.orchestra.run.smithi114.stdout: "default_value": "False", 2024-04-23T10:41:08.097 INFO:teuthology.orchestra.run.smithi114.stdout: "min": "", 2024-04-23T10:41:08.097 INFO:teuthology.orchestra.run.smithi114.stdout: "max": "", 2024-04-23T10:41:08.097 INFO:teuthology.orchestra.run.smithi114.stdout: "enum_allowed": [], 2024-04-23T10:41:08.097 INFO:teuthology.orchestra.run.smithi114.stdout: "desc": "allow SYS_PTRACE capability on ceph containers", 2024-04-23T10:41:08.098 INFO:teuthology.orchestra.run.smithi114.stdout: "long_desc": "The SYS_PTRACE capability is needed to attach to a process with gdb or strace. Enabling this options can allow debugging daemons that encounter problems at runtime.", 2024-04-23T10:41:08.098 INFO:teuthology.orchestra.run.smithi114.stdout: "tags": [], 2024-04-23T10:41:08.098 INFO:teuthology.orchestra.run.smithi114.stdout: "see_also": [] 2024-04-23T10:41:08.098 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-23T10:41:08.098 INFO:teuthology.orchestra.run.smithi114.stdout: "autotune_interval": { 2024-04-23T10:41:08.098 INFO:teuthology.orchestra.run.smithi114.stdout: "name": "autotune_interval", 2024-04-23T10:41:08.098 INFO:teuthology.orchestra.run.smithi114.stdout: "type": "secs", 2024-04-23T10:41:08.098 INFO:teuthology.orchestra.run.smithi114.stdout: "level": "advanced", 2024-04-23T10:41:08.099 INFO:teuthology.orchestra.run.smithi114.stdout: "flags": 0, 2024-04-23T10:41:08.099 INFO:teuthology.orchestra.run.smithi114.stdout: "default_value": "600", 2024-04-23T10:41:08.099 INFO:teuthology.orchestra.run.smithi114.stdout: "min": "", 2024-04-23T10:41:08.099 INFO:teuthology.orchestra.run.smithi114.stdout: "max": "", 2024-04-23T10:41:08.099 INFO:teuthology.orchestra.run.smithi114.stdout: "enum_allowed": [], 2024-04-23T10:41:08.099 INFO:teuthology.orchestra.run.smithi114.stdout: "desc": "how frequently to autotune daemon memory", 2024-04-23T10:41:08.099 INFO:teuthology.orchestra.run.smithi114.stdout: "long_desc": "", 2024-04-23T10:41:08.099 INFO:teuthology.orchestra.run.smithi114.stdout: "tags": [], 2024-04-23T10:41:08.099 INFO:teuthology.orchestra.run.smithi114.stdout: "see_also": [] 2024-04-23T10:41:08.100 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-23T10:41:08.100 INFO:teuthology.orchestra.run.smithi114.stdout: "autotune_memory_target_ratio": { 2024-04-23T10:41:08.100 INFO:teuthology.orchestra.run.smithi114.stdout: "name": "autotune_memory_target_ratio", 2024-04-23T10:41:08.100 INFO:teuthology.orchestra.run.smithi114.stdout: "type": "float", 2024-04-23T10:41:08.100 INFO:teuthology.orchestra.run.smithi114.stdout: "level": "advanced", 2024-04-23T10:41:08.100 INFO:teuthology.orchestra.run.smithi114.stdout: "flags": 0, 2024-04-23T10:41:08.100 INFO:teuthology.orchestra.run.smithi114.stdout: "default_value": "0.7", 2024-04-23T10:41:08.100 INFO:teuthology.orchestra.run.smithi114.stdout: "min": "", 2024-04-23T10:41:08.101 INFO:teuthology.orchestra.run.smithi114.stdout: "max": "", 2024-04-23T10:41:08.101 INFO:teuthology.orchestra.run.smithi114.stdout: "enum_allowed": [], 2024-04-23T10:41:08.101 INFO:teuthology.orchestra.run.smithi114.stdout: "desc": "ratio of total system memory to divide amongst autotuned daemons", 2024-04-23T10:41:08.101 INFO:teuthology.orchestra.run.smithi114.stdout: "long_desc": "", 2024-04-23T10:41:08.101 INFO:teuthology.orchestra.run.smithi114.stdout: "tags": [], 2024-04-23T10:41:08.101 INFO:teuthology.orchestra.run.smithi114.stdout: "see_also": [] 2024-04-23T10:41:08.101 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-23T10:41:08.101 INFO:teuthology.orchestra.run.smithi114.stdout: "cephadm_log_destination": { 2024-04-23T10:41:08.101 INFO:teuthology.orchestra.run.smithi114.stdout: "name": "cephadm_log_destination", 2024-04-23T10:41:08.102 INFO:teuthology.orchestra.run.smithi114.stdout: "type": "str", 2024-04-23T10:41:08.102 INFO:teuthology.orchestra.run.smithi114.stdout: "level": "advanced", 2024-04-23T10:41:08.102 INFO:teuthology.orchestra.run.smithi114.stdout: "flags": 0, 2024-04-23T10:41:08.102 INFO:teuthology.orchestra.run.smithi114.stdout: "default_value": "", 2024-04-23T10:41:08.102 INFO:teuthology.orchestra.run.smithi114.stdout: "min": "", 2024-04-23T10:41:08.102 INFO:teuthology.orchestra.run.smithi114.stdout: "max": "", 2024-04-23T10:41:08.102 INFO:teuthology.orchestra.run.smithi114.stdout: "enum_allowed": [ 2024-04-23T10:41:08.102 INFO:teuthology.orchestra.run.smithi114.stdout: "file", 2024-04-23T10:41:08.103 INFO:teuthology.orchestra.run.smithi114.stdout: "file,syslog", 2024-04-23T10:41:08.103 INFO:teuthology.orchestra.run.smithi114.stdout: "syslog" 2024-04-23T10:41:08.103 INFO:teuthology.orchestra.run.smithi114.stdout: ], 2024-04-23T10:41:08.103 INFO:teuthology.orchestra.run.smithi114.stdout: "desc": "Destination for cephadm command's persistent logging", 2024-04-23T10:41:08.103 INFO:teuthology.orchestra.run.smithi114.stdout: "long_desc": "", 2024-04-23T10:41:08.103 INFO:teuthology.orchestra.run.smithi114.stdout: "tags": [], 2024-04-23T10:41:08.103 INFO:teuthology.orchestra.run.smithi114.stdout: "see_also": [] 2024-04-23T10:41:08.103 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-23T10:41:08.103 INFO:teuthology.orchestra.run.smithi114.stdout: "cgroups_split": { 2024-04-23T10:41:08.104 INFO:teuthology.orchestra.run.smithi114.stdout: "name": "cgroups_split", 2024-04-23T10:41:08.104 INFO:teuthology.orchestra.run.smithi114.stdout: "type": "bool", 2024-04-23T10:41:08.104 INFO:teuthology.orchestra.run.smithi114.stdout: "level": "advanced", 2024-04-23T10:41:08.104 INFO:teuthology.orchestra.run.smithi114.stdout: "flags": 0, 2024-04-23T10:41:08.104 INFO:teuthology.orchestra.run.smithi114.stdout: "default_value": "True", 2024-04-23T10:41:08.104 INFO:teuthology.orchestra.run.smithi114.stdout: "min": "", 2024-04-23T10:41:08.104 INFO:teuthology.orchestra.run.smithi114.stdout: "max": "", 2024-04-23T10:41:08.104 INFO:teuthology.orchestra.run.smithi114.stdout: "enum_allowed": [], 2024-04-23T10:41:08.105 INFO:teuthology.orchestra.run.smithi114.stdout: "desc": "Pass --cgroups=split when cephadm creates containers (currently podman only)", 2024-04-23T10:41:08.105 INFO:teuthology.orchestra.run.smithi114.stdout: "long_desc": "", 2024-04-23T10:41:08.105 INFO:teuthology.orchestra.run.smithi114.stdout: "tags": [], 2024-04-23T10:41:08.105 INFO:teuthology.orchestra.run.smithi114.stdout: "see_also": [] 2024-04-23T10:41:08.105 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-23T10:41:08.105 INFO:teuthology.orchestra.run.smithi114.stdout: "config_checks_enabled": { 2024-04-23T10:41:08.105 INFO:teuthology.orchestra.run.smithi114.stdout: "name": "config_checks_enabled", 2024-04-23T10:41:08.105 INFO:teuthology.orchestra.run.smithi114.stdout: "type": "bool", 2024-04-23T10:41:08.105 INFO:teuthology.orchestra.run.smithi114.stdout: "level": "advanced", 2024-04-23T10:41:08.106 INFO:teuthology.orchestra.run.smithi114.stdout: "flags": 0, 2024-04-23T10:41:08.106 INFO:teuthology.orchestra.run.smithi114.stdout: "default_value": "False", 2024-04-23T10:41:08.106 INFO:teuthology.orchestra.run.smithi114.stdout: "min": "", 2024-04-23T10:41:08.106 INFO:teuthology.orchestra.run.smithi114.stdout: "max": "", 2024-04-23T10:41:08.106 INFO:teuthology.orchestra.run.smithi114.stdout: "enum_allowed": [], 2024-04-23T10:41:08.106 INFO:teuthology.orchestra.run.smithi114.stdout: "desc": "Enable or disable the cephadm configuration analysis", 2024-04-23T10:41:08.106 INFO:teuthology.orchestra.run.smithi114.stdout: "long_desc": "", 2024-04-23T10:41:08.106 INFO:teuthology.orchestra.run.smithi114.stdout: "tags": [], 2024-04-23T10:41:08.107 INFO:teuthology.orchestra.run.smithi114.stdout: "see_also": [] 2024-04-23T10:41:08.107 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-23T10:41:08.107 INFO:teuthology.orchestra.run.smithi114.stdout: "config_dashboard": { 2024-04-23T10:41:08.107 INFO:teuthology.orchestra.run.smithi114.stdout: "name": "config_dashboard", 2024-04-23T10:41:08.107 INFO:teuthology.orchestra.run.smithi114.stdout: "type": "bool", 2024-04-23T10:41:08.107 INFO:teuthology.orchestra.run.smithi114.stdout: "level": "advanced", 2024-04-23T10:41:08.107 INFO:teuthology.orchestra.run.smithi114.stdout: "flags": 0, 2024-04-23T10:41:08.107 INFO:teuthology.orchestra.run.smithi114.stdout: "default_value": "True", 2024-04-23T10:41:08.107 INFO:teuthology.orchestra.run.smithi114.stdout: "min": "", 2024-04-23T10:41:08.108 INFO:teuthology.orchestra.run.smithi114.stdout: "max": "", 2024-04-23T10:41:08.108 INFO:teuthology.orchestra.run.smithi114.stdout: "enum_allowed": [], 2024-04-23T10:41:08.108 INFO:teuthology.orchestra.run.smithi114.stdout: "desc": "manage configs like API endpoints in Dashboard.", 2024-04-23T10:41:08.108 INFO:teuthology.orchestra.run.smithi114.stdout: "long_desc": "", 2024-04-23T10:41:08.108 INFO:teuthology.orchestra.run.smithi114.stdout: "tags": [], 2024-04-23T10:41:08.108 INFO:teuthology.orchestra.run.smithi114.stdout: "see_also": [] 2024-04-23T10:41:08.108 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-23T10:41:08.108 INFO:teuthology.orchestra.run.smithi114.stdout: "container_image_alertmanager": { 2024-04-23T10:41:08.109 INFO:teuthology.orchestra.run.smithi114.stdout: "name": "container_image_alertmanager", 2024-04-23T10:41:08.109 INFO:teuthology.orchestra.run.smithi114.stdout: "type": "str", 2024-04-23T10:41:08.109 INFO:teuthology.orchestra.run.smithi114.stdout: "level": "advanced", 2024-04-23T10:41:08.109 INFO:teuthology.orchestra.run.smithi114.stdout: "flags": 0, 2024-04-23T10:41:08.109 INFO:teuthology.orchestra.run.smithi114.stdout: "default_value": "quay.io/prometheus/alertmanager:v0.25.0", 2024-04-23T10:41:08.109 INFO:teuthology.orchestra.run.smithi114.stdout: "min": "", 2024-04-23T10:41:08.109 INFO:teuthology.orchestra.run.smithi114.stdout: "max": "", 2024-04-23T10:41:08.109 INFO:teuthology.orchestra.run.smithi114.stdout: "enum_allowed": [], 2024-04-23T10:41:08.110 INFO:teuthology.orchestra.run.smithi114.stdout: "desc": "Prometheus container image", 2024-04-23T10:41:08.110 INFO:teuthology.orchestra.run.smithi114.stdout: "long_desc": "", 2024-04-23T10:41:08.110 INFO:teuthology.orchestra.run.smithi114.stdout: "tags": [], 2024-04-23T10:41:08.110 INFO:teuthology.orchestra.run.smithi114.stdout: "see_also": [] 2024-04-23T10:41:08.110 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-23T10:41:08.110 INFO:teuthology.orchestra.run.smithi114.stdout: "container_image_base": { 2024-04-23T10:41:08.110 INFO:teuthology.orchestra.run.smithi114.stdout: "name": "container_image_base", 2024-04-23T10:41:08.110 INFO:teuthology.orchestra.run.smithi114.stdout: "type": "str", 2024-04-23T10:41:08.110 INFO:teuthology.orchestra.run.smithi114.stdout: "level": "advanced", 2024-04-23T10:41:08.111 INFO:teuthology.orchestra.run.smithi114.stdout: "flags": 1, 2024-04-23T10:41:08.111 INFO:teuthology.orchestra.run.smithi114.stdout: "default_value": "quay.io/ceph/ceph", 2024-04-23T10:41:08.111 INFO:teuthology.orchestra.run.smithi114.stdout: "min": "", 2024-04-23T10:41:08.111 INFO:teuthology.orchestra.run.smithi114.stdout: "max": "", 2024-04-23T10:41:08.111 INFO:teuthology.orchestra.run.smithi114.stdout: "enum_allowed": [], 2024-04-23T10:41:08.111 INFO:teuthology.orchestra.run.smithi114.stdout: "desc": "Container image name, without the tag", 2024-04-23T10:41:08.111 INFO:teuthology.orchestra.run.smithi114.stdout: "long_desc": "", 2024-04-23T10:41:08.111 INFO:teuthology.orchestra.run.smithi114.stdout: "tags": [], 2024-04-23T10:41:08.111 INFO:teuthology.orchestra.run.smithi114.stdout: "see_also": [] 2024-04-23T10:41:08.112 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-23T10:41:08.112 INFO:teuthology.orchestra.run.smithi114.stdout: "container_image_elasticsearch": { 2024-04-23T10:41:08.112 INFO:teuthology.orchestra.run.smithi114.stdout: "name": "container_image_elasticsearch", 2024-04-23T10:41:08.112 INFO:teuthology.orchestra.run.smithi114.stdout: "type": "str", 2024-04-23T10:41:08.112 INFO:teuthology.orchestra.run.smithi114.stdout: "level": "advanced", 2024-04-23T10:41:08.112 INFO:teuthology.orchestra.run.smithi114.stdout: "flags": 0, 2024-04-23T10:41:08.112 INFO:teuthology.orchestra.run.smithi114.stdout: "default_value": "quay.io/omrizeneva/elasticsearch:6.8.23", 2024-04-23T10:41:08.112 INFO:teuthology.orchestra.run.smithi114.stdout: "min": "", 2024-04-23T10:41:08.113 INFO:teuthology.orchestra.run.smithi114.stdout: "max": "", 2024-04-23T10:41:08.113 INFO:teuthology.orchestra.run.smithi114.stdout: "enum_allowed": [], 2024-04-23T10:41:08.113 INFO:teuthology.orchestra.run.smithi114.stdout: "desc": "elasticsearch container image", 2024-04-23T10:41:08.113 INFO:teuthology.orchestra.run.smithi114.stdout: "long_desc": "", 2024-04-23T10:41:08.113 INFO:teuthology.orchestra.run.smithi114.stdout: "tags": [], 2024-04-23T10:41:08.113 INFO:teuthology.orchestra.run.smithi114.stdout: "see_also": [] 2024-04-23T10:41:08.113 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-23T10:41:08.113 INFO:teuthology.orchestra.run.smithi114.stdout: "container_image_grafana": { 2024-04-23T10:41:08.113 INFO:teuthology.orchestra.run.smithi114.stdout: "name": "container_image_grafana", 2024-04-23T10:41:08.114 INFO:teuthology.orchestra.run.smithi114.stdout: "type": "str", 2024-04-23T10:41:08.114 INFO:teuthology.orchestra.run.smithi114.stdout: "level": "advanced", 2024-04-23T10:41:08.114 INFO:teuthology.orchestra.run.smithi114.stdout: "flags": 0, 2024-04-23T10:41:08.114 INFO:teuthology.orchestra.run.smithi114.stdout: "default_value": "quay.io/ceph/grafana:9.4.12", 2024-04-23T10:41:08.114 INFO:teuthology.orchestra.run.smithi114.stdout: "min": "", 2024-04-23T10:41:08.114 INFO:teuthology.orchestra.run.smithi114.stdout: "max": "", 2024-04-23T10:41:08.114 INFO:teuthology.orchestra.run.smithi114.stdout: "enum_allowed": [], 2024-04-23T10:41:08.114 INFO:teuthology.orchestra.run.smithi114.stdout: "desc": "Prometheus container image", 2024-04-23T10:41:08.114 INFO:teuthology.orchestra.run.smithi114.stdout: "long_desc": "", 2024-04-23T10:41:08.115 INFO:teuthology.orchestra.run.smithi114.stdout: "tags": [], 2024-04-23T10:41:08.115 INFO:teuthology.orchestra.run.smithi114.stdout: "see_also": [] 2024-04-23T10:41:08.115 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-23T10:41:08.115 INFO:teuthology.orchestra.run.smithi114.stdout: "container_image_haproxy": { 2024-04-23T10:41:08.115 INFO:teuthology.orchestra.run.smithi114.stdout: "name": "container_image_haproxy", 2024-04-23T10:41:08.115 INFO:teuthology.orchestra.run.smithi114.stdout: "type": "str", 2024-04-23T10:41:08.115 INFO:teuthology.orchestra.run.smithi114.stdout: "level": "advanced", 2024-04-23T10:41:08.115 INFO:teuthology.orchestra.run.smithi114.stdout: "flags": 0, 2024-04-23T10:41:08.115 INFO:teuthology.orchestra.run.smithi114.stdout: "default_value": "quay.io/ceph/haproxy:2.3", 2024-04-23T10:41:08.116 INFO:teuthology.orchestra.run.smithi114.stdout: "min": "", 2024-04-23T10:41:08.116 INFO:teuthology.orchestra.run.smithi114.stdout: "max": "", 2024-04-23T10:41:08.116 INFO:teuthology.orchestra.run.smithi114.stdout: "enum_allowed": [], 2024-04-23T10:41:08.116 INFO:teuthology.orchestra.run.smithi114.stdout: "desc": "HAproxy container image", 2024-04-23T10:41:08.116 INFO:teuthology.orchestra.run.smithi114.stdout: "long_desc": "", 2024-04-23T10:41:08.116 INFO:teuthology.orchestra.run.smithi114.stdout: "tags": [], 2024-04-23T10:41:08.116 INFO:teuthology.orchestra.run.smithi114.stdout: "see_also": [] 2024-04-23T10:41:08.116 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-23T10:41:08.117 INFO:teuthology.orchestra.run.smithi114.stdout: "container_image_jaeger_agent": { 2024-04-23T10:41:08.117 INFO:teuthology.orchestra.run.smithi114.stdout: "name": "container_image_jaeger_agent", 2024-04-23T10:41:08.117 INFO:teuthology.orchestra.run.smithi114.stdout: "type": "str", 2024-04-23T10:41:08.117 INFO:teuthology.orchestra.run.smithi114.stdout: "level": "advanced", 2024-04-23T10:41:08.117 INFO:teuthology.orchestra.run.smithi114.stdout: "flags": 0, 2024-04-23T10:41:08.117 INFO:teuthology.orchestra.run.smithi114.stdout: "default_value": "quay.io/jaegertracing/jaeger-agent:1.29", 2024-04-23T10:41:08.117 INFO:teuthology.orchestra.run.smithi114.stdout: "min": "", 2024-04-23T10:41:08.117 INFO:teuthology.orchestra.run.smithi114.stdout: "max": "", 2024-04-23T10:41:08.118 INFO:teuthology.orchestra.run.smithi114.stdout: "enum_allowed": [], 2024-04-23T10:41:08.118 INFO:teuthology.orchestra.run.smithi114.stdout: "desc": "Jaeger agent container image", 2024-04-23T10:41:08.118 INFO:teuthology.orchestra.run.smithi114.stdout: "long_desc": "", 2024-04-23T10:41:08.118 INFO:teuthology.orchestra.run.smithi114.stdout: "tags": [], 2024-04-23T10:41:08.118 INFO:teuthology.orchestra.run.smithi114.stdout: "see_also": [] 2024-04-23T10:41:08.118 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-23T10:41:08.118 INFO:teuthology.orchestra.run.smithi114.stdout: "container_image_jaeger_collector": { 2024-04-23T10:41:08.118 INFO:teuthology.orchestra.run.smithi114.stdout: "name": "container_image_jaeger_collector", 2024-04-23T10:41:08.118 INFO:teuthology.orchestra.run.smithi114.stdout: "type": "str", 2024-04-23T10:41:08.119 INFO:teuthology.orchestra.run.smithi114.stdout: "level": "advanced", 2024-04-23T10:41:08.119 INFO:teuthology.orchestra.run.smithi114.stdout: "flags": 0, 2024-04-23T10:41:08.119 INFO:teuthology.orchestra.run.smithi114.stdout: "default_value": "quay.io/jaegertracing/jaeger-collector:1.29", 2024-04-23T10:41:08.119 INFO:teuthology.orchestra.run.smithi114.stdout: "min": "", 2024-04-23T10:41:08.119 INFO:teuthology.orchestra.run.smithi114.stdout: "max": "", 2024-04-23T10:41:08.119 INFO:teuthology.orchestra.run.smithi114.stdout: "enum_allowed": [], 2024-04-23T10:41:08.119 INFO:teuthology.orchestra.run.smithi114.stdout: "desc": "Jaeger collector container image", 2024-04-23T10:41:08.119 INFO:teuthology.orchestra.run.smithi114.stdout: "long_desc": "", 2024-04-23T10:41:08.120 INFO:teuthology.orchestra.run.smithi114.stdout: "tags": [], 2024-04-23T10:41:08.120 INFO:teuthology.orchestra.run.smithi114.stdout: "see_also": [] 2024-04-23T10:41:08.120 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-23T10:41:08.120 INFO:teuthology.orchestra.run.smithi114.stdout: "container_image_jaeger_query": { 2024-04-23T10:41:08.120 INFO:teuthology.orchestra.run.smithi114.stdout: "name": "container_image_jaeger_query", 2024-04-23T10:41:08.120 INFO:teuthology.orchestra.run.smithi114.stdout: "type": "str", 2024-04-23T10:41:08.120 INFO:teuthology.orchestra.run.smithi114.stdout: "level": "advanced", 2024-04-23T10:41:08.120 INFO:teuthology.orchestra.run.smithi114.stdout: "flags": 0, 2024-04-23T10:41:08.120 INFO:teuthology.orchestra.run.smithi114.stdout: "default_value": "quay.io/jaegertracing/jaeger-query:1.29", 2024-04-23T10:41:08.121 INFO:teuthology.orchestra.run.smithi114.stdout: "min": "", 2024-04-23T10:41:08.121 INFO:teuthology.orchestra.run.smithi114.stdout: "max": "", 2024-04-23T10:41:08.121 INFO:teuthology.orchestra.run.smithi114.stdout: "enum_allowed": [], 2024-04-23T10:41:08.121 INFO:teuthology.orchestra.run.smithi114.stdout: "desc": "Jaeger query container image", 2024-04-23T10:41:08.121 INFO:teuthology.orchestra.run.smithi114.stdout: "long_desc": "", 2024-04-23T10:41:08.121 INFO:teuthology.orchestra.run.smithi114.stdout: "tags": [], 2024-04-23T10:41:08.121 INFO:teuthology.orchestra.run.smithi114.stdout: "see_also": [] 2024-04-23T10:41:08.121 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-23T10:41:08.121 INFO:teuthology.orchestra.run.smithi114.stdout: "container_image_keepalived": { 2024-04-23T10:41:08.122 INFO:teuthology.orchestra.run.smithi114.stdout: "name": "container_image_keepalived", 2024-04-23T10:41:08.122 INFO:teuthology.orchestra.run.smithi114.stdout: "type": "str", 2024-04-23T10:41:08.122 INFO:teuthology.orchestra.run.smithi114.stdout: "level": "advanced", 2024-04-23T10:41:08.122 INFO:teuthology.orchestra.run.smithi114.stdout: "flags": 0, 2024-04-23T10:41:08.122 INFO:teuthology.orchestra.run.smithi114.stdout: "default_value": "quay.io/ceph/keepalived:2.2.4", 2024-04-23T10:41:08.122 INFO:teuthology.orchestra.run.smithi114.stdout: "min": "", 2024-04-23T10:41:08.122 INFO:teuthology.orchestra.run.smithi114.stdout: "max": "", 2024-04-23T10:41:08.122 INFO:teuthology.orchestra.run.smithi114.stdout: "enum_allowed": [], 2024-04-23T10:41:08.123 INFO:teuthology.orchestra.run.smithi114.stdout: "desc": "Keepalived container image", 2024-04-23T10:41:08.123 INFO:teuthology.orchestra.run.smithi114.stdout: "long_desc": "", 2024-04-23T10:41:08.123 INFO:teuthology.orchestra.run.smithi114.stdout: "tags": [], 2024-04-23T10:41:08.123 INFO:teuthology.orchestra.run.smithi114.stdout: "see_also": [] 2024-04-23T10:41:08.123 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-23T10:41:08.123 INFO:teuthology.orchestra.run.smithi114.stdout: "container_image_loki": { 2024-04-23T10:41:08.123 INFO:teuthology.orchestra.run.smithi114.stdout: "name": "container_image_loki", 2024-04-23T10:41:08.123 INFO:teuthology.orchestra.run.smithi114.stdout: "type": "str", 2024-04-23T10:41:08.123 INFO:teuthology.orchestra.run.smithi114.stdout: "level": "advanced", 2024-04-23T10:41:08.124 INFO:teuthology.orchestra.run.smithi114.stdout: "flags": 0, 2024-04-23T10:41:08.124 INFO:teuthology.orchestra.run.smithi114.stdout: "default_value": "docker.io/grafana/loki:2.4.0", 2024-04-23T10:41:08.124 INFO:teuthology.orchestra.run.smithi114.stdout: "min": "", 2024-04-23T10:41:08.124 INFO:teuthology.orchestra.run.smithi114.stdout: "max": "", 2024-04-23T10:41:08.124 INFO:teuthology.orchestra.run.smithi114.stdout: "enum_allowed": [], 2024-04-23T10:41:08.124 INFO:teuthology.orchestra.run.smithi114.stdout: "desc": "Loki container image", 2024-04-23T10:41:08.124 INFO:teuthology.orchestra.run.smithi114.stdout: "long_desc": "", 2024-04-23T10:41:08.124 INFO:teuthology.orchestra.run.smithi114.stdout: "tags": [], 2024-04-23T10:41:08.124 INFO:teuthology.orchestra.run.smithi114.stdout: "see_also": [] 2024-04-23T10:41:08.125 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-23T10:41:08.125 INFO:teuthology.orchestra.run.smithi114.stdout: "container_image_node_exporter": { 2024-04-23T10:41:08.125 INFO:teuthology.orchestra.run.smithi114.stdout: "name": "container_image_node_exporter", 2024-04-23T10:41:08.125 INFO:teuthology.orchestra.run.smithi114.stdout: "type": "str", 2024-04-23T10:41:08.125 INFO:teuthology.orchestra.run.smithi114.stdout: "level": "advanced", 2024-04-23T10:41:08.125 INFO:teuthology.orchestra.run.smithi114.stdout: "flags": 0, 2024-04-23T10:41:08.125 INFO:teuthology.orchestra.run.smithi114.stdout: "default_value": "quay.io/prometheus/node-exporter:v1.5.0", 2024-04-23T10:41:08.125 INFO:teuthology.orchestra.run.smithi114.stdout: "min": "", 2024-04-23T10:41:08.126 INFO:teuthology.orchestra.run.smithi114.stdout: "max": "", 2024-04-23T10:41:08.126 INFO:teuthology.orchestra.run.smithi114.stdout: "enum_allowed": [], 2024-04-23T10:41:08.126 INFO:teuthology.orchestra.run.smithi114.stdout: "desc": "Prometheus container image", 2024-04-23T10:41:08.126 INFO:teuthology.orchestra.run.smithi114.stdout: "long_desc": "", 2024-04-23T10:41:08.126 INFO:teuthology.orchestra.run.smithi114.stdout: "tags": [], 2024-04-23T10:41:08.126 INFO:teuthology.orchestra.run.smithi114.stdout: "see_also": [] 2024-04-23T10:41:08.126 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-23T10:41:08.126 INFO:teuthology.orchestra.run.smithi114.stdout: "container_image_nvmeof": { 2024-04-23T10:41:08.126 INFO:teuthology.orchestra.run.smithi114.stdout: "name": "container_image_nvmeof", 2024-04-23T10:41:08.127 INFO:teuthology.orchestra.run.smithi114.stdout: "type": "str", 2024-04-23T10:41:08.127 INFO:teuthology.orchestra.run.smithi114.stdout: "level": "advanced", 2024-04-23T10:41:08.127 INFO:teuthology.orchestra.run.smithi114.stdout: "flags": 0, 2024-04-23T10:41:08.127 INFO:teuthology.orchestra.run.smithi114.stdout: "default_value": "quay.io/ceph/nvmeof:1.2.1", 2024-04-23T10:41:08.127 INFO:teuthology.orchestra.run.smithi114.stdout: "min": "", 2024-04-23T10:41:08.127 INFO:teuthology.orchestra.run.smithi114.stdout: "max": "", 2024-04-23T10:41:08.127 INFO:teuthology.orchestra.run.smithi114.stdout: "enum_allowed": [], 2024-04-23T10:41:08.127 INFO:teuthology.orchestra.run.smithi114.stdout: "desc": "Nvme-of container image", 2024-04-23T10:41:08.127 INFO:teuthology.orchestra.run.smithi114.stdout: "long_desc": "", 2024-04-23T10:41:08.128 INFO:teuthology.orchestra.run.smithi114.stdout: "tags": [], 2024-04-23T10:41:08.128 INFO:teuthology.orchestra.run.smithi114.stdout: "see_also": [] 2024-04-23T10:41:08.128 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-23T10:41:08.128 INFO:teuthology.orchestra.run.smithi114.stdout: "container_image_prometheus": { 2024-04-23T10:41:08.128 INFO:teuthology.orchestra.run.smithi114.stdout: "name": "container_image_prometheus", 2024-04-23T10:41:08.128 INFO:teuthology.orchestra.run.smithi114.stdout: "type": "str", 2024-04-23T10:41:08.128 INFO:teuthology.orchestra.run.smithi114.stdout: "level": "advanced", 2024-04-23T10:41:08.128 INFO:teuthology.orchestra.run.smithi114.stdout: "flags": 0, 2024-04-23T10:41:08.128 INFO:teuthology.orchestra.run.smithi114.stdout: "default_value": "quay.io/prometheus/prometheus:v2.43.0", 2024-04-23T10:41:08.129 INFO:teuthology.orchestra.run.smithi114.stdout: "min": "", 2024-04-23T10:41:08.129 INFO:teuthology.orchestra.run.smithi114.stdout: "max": "", 2024-04-23T10:41:08.129 INFO:teuthology.orchestra.run.smithi114.stdout: "enum_allowed": [], 2024-04-23T10:41:08.129 INFO:teuthology.orchestra.run.smithi114.stdout: "desc": "Prometheus container image", 2024-04-23T10:41:08.129 INFO:teuthology.orchestra.run.smithi114.stdout: "long_desc": "", 2024-04-23T10:41:08.129 INFO:teuthology.orchestra.run.smithi114.stdout: "tags": [], 2024-04-23T10:41:08.129 INFO:teuthology.orchestra.run.smithi114.stdout: "see_also": [] 2024-04-23T10:41:08.129 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-23T10:41:08.129 INFO:teuthology.orchestra.run.smithi114.stdout: "container_image_promtail": { 2024-04-23T10:41:08.130 INFO:teuthology.orchestra.run.smithi114.stdout: "name": "container_image_promtail", 2024-04-23T10:41:08.130 INFO:teuthology.orchestra.run.smithi114.stdout: "type": "str", 2024-04-23T10:41:08.130 INFO:teuthology.orchestra.run.smithi114.stdout: "level": "advanced", 2024-04-23T10:41:08.130 INFO:teuthology.orchestra.run.smithi114.stdout: "flags": 0, 2024-04-23T10:41:08.130 INFO:teuthology.orchestra.run.smithi114.stdout: "default_value": "docker.io/grafana/promtail:2.4.0", 2024-04-23T10:41:08.130 INFO:teuthology.orchestra.run.smithi114.stdout: "min": "", 2024-04-23T10:41:08.130 INFO:teuthology.orchestra.run.smithi114.stdout: "max": "", 2024-04-23T10:41:08.130 INFO:teuthology.orchestra.run.smithi114.stdout: "enum_allowed": [], 2024-04-23T10:41:08.131 INFO:teuthology.orchestra.run.smithi114.stdout: "desc": "Promtail container image", 2024-04-23T10:41:08.131 INFO:teuthology.orchestra.run.smithi114.stdout: "long_desc": "", 2024-04-23T10:41:08.131 INFO:teuthology.orchestra.run.smithi114.stdout: "tags": [], 2024-04-23T10:41:08.131 INFO:teuthology.orchestra.run.smithi114.stdout: "see_also": [] 2024-04-23T10:41:08.131 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-23T10:41:08.131 INFO:teuthology.orchestra.run.smithi114.stdout: "container_image_samba": { 2024-04-23T10:41:08.131 INFO:teuthology.orchestra.run.smithi114.stdout: "name": "container_image_samba", 2024-04-23T10:41:08.131 INFO:teuthology.orchestra.run.smithi114.stdout: "type": "str", 2024-04-23T10:41:08.131 INFO:teuthology.orchestra.run.smithi114.stdout: "level": "advanced", 2024-04-23T10:41:08.132 INFO:teuthology.orchestra.run.smithi114.stdout: "flags": 0, 2024-04-23T10:41:08.132 INFO:teuthology.orchestra.run.smithi114.stdout: "default_value": "quay.io/samba.org/samba-server:devbuilds-centos-amd64", 2024-04-23T10:41:08.132 INFO:teuthology.orchestra.run.smithi114.stdout: "min": "", 2024-04-23T10:41:08.132 INFO:teuthology.orchestra.run.smithi114.stdout: "max": "", 2024-04-23T10:41:08.132 INFO:teuthology.orchestra.run.smithi114.stdout: "enum_allowed": [], 2024-04-23T10:41:08.132 INFO:teuthology.orchestra.run.smithi114.stdout: "desc": "Samba/SMB container image", 2024-04-23T10:41:08.132 INFO:teuthology.orchestra.run.smithi114.stdout: "long_desc": "", 2024-04-23T10:41:08.132 INFO:teuthology.orchestra.run.smithi114.stdout: "tags": [], 2024-04-23T10:41:08.132 INFO:teuthology.orchestra.run.smithi114.stdout: "see_also": [] 2024-04-23T10:41:08.133 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-23T10:41:08.133 INFO:teuthology.orchestra.run.smithi114.stdout: "container_image_snmp_gateway": { 2024-04-23T10:41:08.133 INFO:teuthology.orchestra.run.smithi114.stdout: "name": "container_image_snmp_gateway", 2024-04-23T10:41:08.133 INFO:teuthology.orchestra.run.smithi114.stdout: "type": "str", 2024-04-23T10:41:08.133 INFO:teuthology.orchestra.run.smithi114.stdout: "level": "advanced", 2024-04-23T10:41:08.133 INFO:teuthology.orchestra.run.smithi114.stdout: "flags": 0, 2024-04-23T10:41:08.133 INFO:teuthology.orchestra.run.smithi114.stdout: "default_value": "docker.io/maxwo/snmp-notifier:v1.2.1", 2024-04-23T10:41:08.133 INFO:teuthology.orchestra.run.smithi114.stdout: "min": "", 2024-04-23T10:41:08.134 INFO:teuthology.orchestra.run.smithi114.stdout: "max": "", 2024-04-23T10:41:08.134 INFO:teuthology.orchestra.run.smithi114.stdout: "enum_allowed": [], 2024-04-23T10:41:08.134 INFO:teuthology.orchestra.run.smithi114.stdout: "desc": "SNMP Gateway container image", 2024-04-23T10:41:08.134 INFO:teuthology.orchestra.run.smithi114.stdout: "long_desc": "", 2024-04-23T10:41:08.134 INFO:teuthology.orchestra.run.smithi114.stdout: "tags": [], 2024-04-23T10:41:08.134 INFO:teuthology.orchestra.run.smithi114.stdout: "see_also": [] 2024-04-23T10:41:08.134 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-23T10:41:08.134 INFO:teuthology.orchestra.run.smithi114.stdout: "container_init": { 2024-04-23T10:41:08.135 INFO:teuthology.orchestra.run.smithi114.stdout: "name": "container_init", 2024-04-23T10:41:08.135 INFO:teuthology.orchestra.run.smithi114.stdout: "type": "bool", 2024-04-23T10:41:08.135 INFO:teuthology.orchestra.run.smithi114.stdout: "level": "advanced", 2024-04-23T10:41:08.135 INFO:teuthology.orchestra.run.smithi114.stdout: "flags": 0, 2024-04-23T10:41:08.135 INFO:teuthology.orchestra.run.smithi114.stdout: "default_value": "True", 2024-04-23T10:41:08.135 INFO:teuthology.orchestra.run.smithi114.stdout: "min": "", 2024-04-23T10:41:08.135 INFO:teuthology.orchestra.run.smithi114.stdout: "max": "", 2024-04-23T10:41:08.135 INFO:teuthology.orchestra.run.smithi114.stdout: "enum_allowed": [], 2024-04-23T10:41:08.135 INFO:teuthology.orchestra.run.smithi114.stdout: "desc": "Run podman/docker with `--init`", 2024-04-23T10:41:08.136 INFO:teuthology.orchestra.run.smithi114.stdout: "long_desc": "", 2024-04-23T10:41:08.136 INFO:teuthology.orchestra.run.smithi114.stdout: "tags": [], 2024-04-23T10:41:08.136 INFO:teuthology.orchestra.run.smithi114.stdout: "see_also": [] 2024-04-23T10:41:08.136 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-23T10:41:08.136 INFO:teuthology.orchestra.run.smithi114.stdout: "daemon_cache_timeout": { 2024-04-23T10:41:08.136 INFO:teuthology.orchestra.run.smithi114.stdout: "name": "daemon_cache_timeout", 2024-04-23T10:41:08.136 INFO:teuthology.orchestra.run.smithi114.stdout: "type": "secs", 2024-04-23T10:41:08.136 INFO:teuthology.orchestra.run.smithi114.stdout: "level": "advanced", 2024-04-23T10:41:08.136 INFO:teuthology.orchestra.run.smithi114.stdout: "flags": 0, 2024-04-23T10:41:08.137 INFO:teuthology.orchestra.run.smithi114.stdout: "default_value": "600", 2024-04-23T10:41:08.137 INFO:teuthology.orchestra.run.smithi114.stdout: "min": "", 2024-04-23T10:41:08.137 INFO:teuthology.orchestra.run.smithi114.stdout: "max": "", 2024-04-23T10:41:08.137 INFO:teuthology.orchestra.run.smithi114.stdout: "enum_allowed": [], 2024-04-23T10:41:08.137 INFO:teuthology.orchestra.run.smithi114.stdout: "desc": "seconds to cache service (daemon) inventory", 2024-04-23T10:41:08.137 INFO:teuthology.orchestra.run.smithi114.stdout: "long_desc": "", 2024-04-23T10:41:08.137 INFO:teuthology.orchestra.run.smithi114.stdout: "tags": [], 2024-04-23T10:41:08.137 INFO:teuthology.orchestra.run.smithi114.stdout: "see_also": [] 2024-04-23T10:41:08.137 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-23T10:41:08.138 INFO:teuthology.orchestra.run.smithi114.stdout: "default_cephadm_command_timeout": { 2024-04-23T10:41:08.138 INFO:teuthology.orchestra.run.smithi114.stdout: "name": "default_cephadm_command_timeout", 2024-04-23T10:41:08.138 INFO:teuthology.orchestra.run.smithi114.stdout: "type": "int", 2024-04-23T10:41:08.138 INFO:teuthology.orchestra.run.smithi114.stdout: "level": "advanced", 2024-04-23T10:41:08.138 INFO:teuthology.orchestra.run.smithi114.stdout: "flags": 0, 2024-04-23T10:41:08.138 INFO:teuthology.orchestra.run.smithi114.stdout: "default_value": "900", 2024-04-23T10:41:08.138 INFO:teuthology.orchestra.run.smithi114.stdout: "min": "", 2024-04-23T10:41:08.138 INFO:teuthology.orchestra.run.smithi114.stdout: "max": "", 2024-04-23T10:41:08.139 INFO:teuthology.orchestra.run.smithi114.stdout: "enum_allowed": [], 2024-04-23T10:41:08.139 INFO:teuthology.orchestra.run.smithi114.stdout: "desc": "Default timeout applied to cephadm commands run directly on the host (in seconds)", 2024-04-23T10:41:08.139 INFO:teuthology.orchestra.run.smithi114.stdout: "long_desc": "", 2024-04-23T10:41:08.139 INFO:teuthology.orchestra.run.smithi114.stdout: "tags": [], 2024-04-23T10:41:08.139 INFO:teuthology.orchestra.run.smithi114.stdout: "see_also": [] 2024-04-23T10:41:08.139 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-23T10:41:08.139 INFO:teuthology.orchestra.run.smithi114.stdout: "default_registry": { 2024-04-23T10:41:08.139 INFO:teuthology.orchestra.run.smithi114.stdout: "name": "default_registry", 2024-04-23T10:41:08.139 INFO:teuthology.orchestra.run.smithi114.stdout: "type": "str", 2024-04-23T10:41:08.140 INFO:teuthology.orchestra.run.smithi114.stdout: "level": "advanced", 2024-04-23T10:41:08.140 INFO:teuthology.orchestra.run.smithi114.stdout: "flags": 0, 2024-04-23T10:41:08.140 INFO:teuthology.orchestra.run.smithi114.stdout: "default_value": "docker.io", 2024-04-23T10:41:08.140 INFO:teuthology.orchestra.run.smithi114.stdout: "min": "", 2024-04-23T10:41:08.140 INFO:teuthology.orchestra.run.smithi114.stdout: "max": "", 2024-04-23T10:41:08.140 INFO:teuthology.orchestra.run.smithi114.stdout: "enum_allowed": [], 2024-04-23T10:41:08.140 INFO:teuthology.orchestra.run.smithi114.stdout: "desc": "Search-registry to which we should normalize unqualified image names. This is not the default registry", 2024-04-23T10:41:08.140 INFO:teuthology.orchestra.run.smithi114.stdout: "long_desc": "", 2024-04-23T10:41:08.140 INFO:teuthology.orchestra.run.smithi114.stdout: "tags": [], 2024-04-23T10:41:08.141 INFO:teuthology.orchestra.run.smithi114.stdout: "see_also": [] 2024-04-23T10:41:08.141 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-23T10:41:08.141 INFO:teuthology.orchestra.run.smithi114.stdout: "device_cache_timeout": { 2024-04-23T10:41:08.141 INFO:teuthology.orchestra.run.smithi114.stdout: "name": "device_cache_timeout", 2024-04-23T10:41:08.141 INFO:teuthology.orchestra.run.smithi114.stdout: "type": "secs", 2024-04-23T10:41:08.141 INFO:teuthology.orchestra.run.smithi114.stdout: "level": "advanced", 2024-04-23T10:41:08.141 INFO:teuthology.orchestra.run.smithi114.stdout: "flags": 0, 2024-04-23T10:41:08.141 INFO:teuthology.orchestra.run.smithi114.stdout: "default_value": "1800", 2024-04-23T10:41:08.141 INFO:teuthology.orchestra.run.smithi114.stdout: "min": "", 2024-04-23T10:41:08.142 INFO:teuthology.orchestra.run.smithi114.stdout: "max": "", 2024-04-23T10:41:08.142 INFO:teuthology.orchestra.run.smithi114.stdout: "enum_allowed": [], 2024-04-23T10:41:08.142 INFO:teuthology.orchestra.run.smithi114.stdout: "desc": "seconds to cache device inventory", 2024-04-23T10:41:08.142 INFO:teuthology.orchestra.run.smithi114.stdout: "long_desc": "", 2024-04-23T10:41:08.142 INFO:teuthology.orchestra.run.smithi114.stdout: "tags": [], 2024-04-23T10:41:08.142 INFO:teuthology.orchestra.run.smithi114.stdout: "see_also": [] 2024-04-23T10:41:08.142 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-23T10:41:08.143 INFO:teuthology.orchestra.run.smithi114.stdout: "device_enhanced_scan": { 2024-04-23T10:41:08.143 INFO:teuthology.orchestra.run.smithi114.stdout: "name": "device_enhanced_scan", 2024-04-23T10:41:08.143 INFO:teuthology.orchestra.run.smithi114.stdout: "type": "bool", 2024-04-23T10:41:08.143 INFO:teuthology.orchestra.run.smithi114.stdout: "level": "advanced", 2024-04-23T10:41:08.143 INFO:teuthology.orchestra.run.smithi114.stdout: "flags": 0, 2024-04-23T10:41:08.143 INFO:teuthology.orchestra.run.smithi114.stdout: "default_value": "False", 2024-04-23T10:41:08.143 INFO:teuthology.orchestra.run.smithi114.stdout: "min": "", 2024-04-23T10:41:08.143 INFO:teuthology.orchestra.run.smithi114.stdout: "max": "", 2024-04-23T10:41:08.143 INFO:teuthology.orchestra.run.smithi114.stdout: "enum_allowed": [], 2024-04-23T10:41:08.144 INFO:teuthology.orchestra.run.smithi114.stdout: "desc": "Use libstoragemgmt during device scans", 2024-04-23T10:41:08.144 INFO:teuthology.orchestra.run.smithi114.stdout: "long_desc": "", 2024-04-23T10:41:08.144 INFO:teuthology.orchestra.run.smithi114.stdout: "tags": [], 2024-04-23T10:41:08.144 INFO:teuthology.orchestra.run.smithi114.stdout: "see_also": [] 2024-04-23T10:41:08.144 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-23T10:41:08.144 INFO:teuthology.orchestra.run.smithi114.stdout: "facts_cache_timeout": { 2024-04-23T10:41:08.144 INFO:teuthology.orchestra.run.smithi114.stdout: "name": "facts_cache_timeout", 2024-04-23T10:41:08.144 INFO:teuthology.orchestra.run.smithi114.stdout: "type": "secs", 2024-04-23T10:41:08.145 INFO:teuthology.orchestra.run.smithi114.stdout: "level": "advanced", 2024-04-23T10:41:08.145 INFO:teuthology.orchestra.run.smithi114.stdout: "flags": 0, 2024-04-23T10:41:08.145 INFO:teuthology.orchestra.run.smithi114.stdout: "default_value": "60", 2024-04-23T10:41:08.145 INFO:teuthology.orchestra.run.smithi114.stdout: "min": "", 2024-04-23T10:41:08.145 INFO:teuthology.orchestra.run.smithi114.stdout: "max": "", 2024-04-23T10:41:08.145 INFO:teuthology.orchestra.run.smithi114.stdout: "enum_allowed": [], 2024-04-23T10:41:08.145 INFO:teuthology.orchestra.run.smithi114.stdout: "desc": "seconds to cache host facts data", 2024-04-23T10:41:08.145 INFO:teuthology.orchestra.run.smithi114.stdout: "long_desc": "", 2024-04-23T10:41:08.145 INFO:teuthology.orchestra.run.smithi114.stdout: "tags": [], 2024-04-23T10:41:08.146 INFO:teuthology.orchestra.run.smithi114.stdout: "see_also": [] 2024-04-23T10:41:08.146 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-23T10:41:08.146 INFO:teuthology.orchestra.run.smithi114.stdout: "grafana_dashboards_path": { 2024-04-23T10:41:08.146 INFO:teuthology.orchestra.run.smithi114.stdout: "name": "grafana_dashboards_path", 2024-04-23T10:41:08.146 INFO:teuthology.orchestra.run.smithi114.stdout: "type": "str", 2024-04-23T10:41:08.146 INFO:teuthology.orchestra.run.smithi114.stdout: "level": "advanced", 2024-04-23T10:41:08.146 INFO:teuthology.orchestra.run.smithi114.stdout: "flags": 0, 2024-04-23T10:41:08.146 INFO:teuthology.orchestra.run.smithi114.stdout: "default_value": "/etc/grafana/dashboards/ceph-dashboard/", 2024-04-23T10:41:08.147 INFO:teuthology.orchestra.run.smithi114.stdout: "min": "", 2024-04-23T10:41:08.147 INFO:teuthology.orchestra.run.smithi114.stdout: "max": "", 2024-04-23T10:41:08.147 INFO:teuthology.orchestra.run.smithi114.stdout: "enum_allowed": [], 2024-04-23T10:41:08.147 INFO:teuthology.orchestra.run.smithi114.stdout: "desc": "location of dashboards to include in grafana deployments", 2024-04-23T10:41:08.147 INFO:teuthology.orchestra.run.smithi114.stdout: "long_desc": "", 2024-04-23T10:41:08.147 INFO:teuthology.orchestra.run.smithi114.stdout: "tags": [], 2024-04-23T10:41:08.147 INFO:teuthology.orchestra.run.smithi114.stdout: "see_also": [] 2024-04-23T10:41:08.147 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-23T10:41:08.147 INFO:teuthology.orchestra.run.smithi114.stdout: "host_check_interval": { 2024-04-23T10:41:08.147 INFO:teuthology.orchestra.run.smithi114.stdout: "name": "host_check_interval", 2024-04-23T10:41:08.148 INFO:teuthology.orchestra.run.smithi114.stdout: "type": "secs", 2024-04-23T10:41:08.148 INFO:teuthology.orchestra.run.smithi114.stdout: "level": "advanced", 2024-04-23T10:41:08.148 INFO:teuthology.orchestra.run.smithi114.stdout: "flags": 0, 2024-04-23T10:41:08.148 INFO:teuthology.orchestra.run.smithi114.stdout: "default_value": "600", 2024-04-23T10:41:08.148 INFO:teuthology.orchestra.run.smithi114.stdout: "min": "", 2024-04-23T10:41:08.148 INFO:teuthology.orchestra.run.smithi114.stdout: "max": "", 2024-04-23T10:41:08.148 INFO:teuthology.orchestra.run.smithi114.stdout: "enum_allowed": [], 2024-04-23T10:41:08.148 INFO:teuthology.orchestra.run.smithi114.stdout: "desc": "how frequently to perform a host check", 2024-04-23T10:41:08.148 INFO:teuthology.orchestra.run.smithi114.stdout: "long_desc": "", 2024-04-23T10:41:08.149 INFO:teuthology.orchestra.run.smithi114.stdout: "tags": [], 2024-04-23T10:41:08.149 INFO:teuthology.orchestra.run.smithi114.stdout: "see_also": [] 2024-04-23T10:41:08.149 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-23T10:41:08.149 INFO:teuthology.orchestra.run.smithi114.stdout: "hw_monitoring": { 2024-04-23T10:41:08.149 INFO:teuthology.orchestra.run.smithi114.stdout: "name": "hw_monitoring", 2024-04-23T10:41:08.149 INFO:teuthology.orchestra.run.smithi114.stdout: "type": "bool", 2024-04-23T10:41:08.149 INFO:teuthology.orchestra.run.smithi114.stdout: "level": "advanced", 2024-04-23T10:41:08.149 INFO:teuthology.orchestra.run.smithi114.stdout: "flags": 0, 2024-04-23T10:41:08.149 INFO:teuthology.orchestra.run.smithi114.stdout: "default_value": "False", 2024-04-23T10:41:08.150 INFO:teuthology.orchestra.run.smithi114.stdout: "min": "", 2024-04-23T10:41:08.150 INFO:teuthology.orchestra.run.smithi114.stdout: "max": "", 2024-04-23T10:41:08.150 INFO:teuthology.orchestra.run.smithi114.stdout: "enum_allowed": [], 2024-04-23T10:41:08.150 INFO:teuthology.orchestra.run.smithi114.stdout: "desc": "Deploy hw monitoring daemon on every host.", 2024-04-23T10:41:08.150 INFO:teuthology.orchestra.run.smithi114.stdout: "long_desc": "", 2024-04-23T10:41:08.150 INFO:teuthology.orchestra.run.smithi114.stdout: "tags": [], 2024-04-23T10:41:08.150 INFO:teuthology.orchestra.run.smithi114.stdout: "see_also": [] 2024-04-23T10:41:08.150 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-23T10:41:08.150 INFO:teuthology.orchestra.run.smithi114.stdout: "inventory_list_all": { 2024-04-23T10:41:08.151 INFO:teuthology.orchestra.run.smithi114.stdout: "name": "inventory_list_all", 2024-04-23T10:41:08.151 INFO:teuthology.orchestra.run.smithi114.stdout: "type": "bool", 2024-04-23T10:41:08.151 INFO:teuthology.orchestra.run.smithi114.stdout: "level": "advanced", 2024-04-23T10:41:08.151 INFO:teuthology.orchestra.run.smithi114.stdout: "flags": 0, 2024-04-23T10:41:08.151 INFO:teuthology.orchestra.run.smithi114.stdout: "default_value": "False", 2024-04-23T10:41:08.151 INFO:teuthology.orchestra.run.smithi114.stdout: "min": "", 2024-04-23T10:41:08.151 INFO:teuthology.orchestra.run.smithi114.stdout: "max": "", 2024-04-23T10:41:08.151 INFO:teuthology.orchestra.run.smithi114.stdout: "enum_allowed": [], 2024-04-23T10:41:08.151 INFO:teuthology.orchestra.run.smithi114.stdout: "desc": "Whether ceph-volume inventory should report more devices (mostly mappers (LVs / mpaths), partitions...)", 2024-04-23T10:41:08.152 INFO:teuthology.orchestra.run.smithi114.stdout: "long_desc": "", 2024-04-23T10:41:08.152 INFO:teuthology.orchestra.run.smithi114.stdout: "tags": [], 2024-04-23T10:41:08.152 INFO:teuthology.orchestra.run.smithi114.stdout: "see_also": [] 2024-04-23T10:41:08.152 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-23T10:41:08.152 INFO:teuthology.orchestra.run.smithi114.stdout: "log_level": { 2024-04-23T10:41:08.152 INFO:teuthology.orchestra.run.smithi114.stdout: "name": "log_level", 2024-04-23T10:41:08.152 INFO:teuthology.orchestra.run.smithi114.stdout: "type": "str", 2024-04-23T10:41:08.152 INFO:teuthology.orchestra.run.smithi114.stdout: "level": "advanced", 2024-04-23T10:41:08.152 INFO:teuthology.orchestra.run.smithi114.stdout: "flags": 1, 2024-04-23T10:41:08.153 INFO:teuthology.orchestra.run.smithi114.stdout: "default_value": "", 2024-04-23T10:41:08.153 INFO:teuthology.orchestra.run.smithi114.stdout: "min": "", 2024-04-23T10:41:08.153 INFO:teuthology.orchestra.run.smithi114.stdout: "max": "", 2024-04-23T10:41:08.153 INFO:teuthology.orchestra.run.smithi114.stdout: "enum_allowed": [ 2024-04-23T10:41:08.153 INFO:teuthology.orchestra.run.smithi114.stdout: "", 2024-04-23T10:41:08.153 INFO:teuthology.orchestra.run.smithi114.stdout: "critical", 2024-04-23T10:41:08.153 INFO:teuthology.orchestra.run.smithi114.stdout: "debug", 2024-04-23T10:41:08.153 INFO:teuthology.orchestra.run.smithi114.stdout: "error", 2024-04-23T10:41:08.153 INFO:teuthology.orchestra.run.smithi114.stdout: "info", 2024-04-23T10:41:08.153 INFO:teuthology.orchestra.run.smithi114.stdout: "warning" 2024-04-23T10:41:08.154 INFO:teuthology.orchestra.run.smithi114.stdout: ], 2024-04-23T10:41:08.154 INFO:teuthology.orchestra.run.smithi114.stdout: "desc": "", 2024-04-23T10:41:08.154 INFO:teuthology.orchestra.run.smithi114.stdout: "long_desc": "", 2024-04-23T10:41:08.154 INFO:teuthology.orchestra.run.smithi114.stdout: "tags": [], 2024-04-23T10:41:08.154 INFO:teuthology.orchestra.run.smithi114.stdout: "see_also": [] 2024-04-23T10:41:08.154 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-23T10:41:08.154 INFO:teuthology.orchestra.run.smithi114.stdout: "log_refresh_metadata": { 2024-04-23T10:41:08.154 INFO:teuthology.orchestra.run.smithi114.stdout: "name": "log_refresh_metadata", 2024-04-23T10:41:08.155 INFO:teuthology.orchestra.run.smithi114.stdout: "type": "bool", 2024-04-23T10:41:08.155 INFO:teuthology.orchestra.run.smithi114.stdout: "level": "advanced", 2024-04-23T10:41:08.155 INFO:teuthology.orchestra.run.smithi114.stdout: "flags": 0, 2024-04-23T10:41:08.155 INFO:teuthology.orchestra.run.smithi114.stdout: "default_value": "False", 2024-04-23T10:41:08.155 INFO:teuthology.orchestra.run.smithi114.stdout: "min": "", 2024-04-23T10:41:08.155 INFO:teuthology.orchestra.run.smithi114.stdout: "max": "", 2024-04-23T10:41:08.155 INFO:teuthology.orchestra.run.smithi114.stdout: "enum_allowed": [], 2024-04-23T10:41:08.155 INFO:teuthology.orchestra.run.smithi114.stdout: "desc": "Log all refresh metadata. Includes daemon, device, and host info collected regularly. Only has effect if logging at debug level", 2024-04-23T10:41:08.155 INFO:teuthology.orchestra.run.smithi114.stdout: "long_desc": "", 2024-04-23T10:41:08.156 INFO:teuthology.orchestra.run.smithi114.stdout: "tags": [], 2024-04-23T10:41:08.156 INFO:teuthology.orchestra.run.smithi114.stdout: "see_also": [] 2024-04-23T10:41:08.156 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-23T10:41:08.156 INFO:teuthology.orchestra.run.smithi114.stdout: "log_to_cluster": { 2024-04-23T10:41:08.156 INFO:teuthology.orchestra.run.smithi114.stdout: "name": "log_to_cluster", 2024-04-23T10:41:08.156 INFO:teuthology.orchestra.run.smithi114.stdout: "type": "bool", 2024-04-23T10:41:08.156 INFO:teuthology.orchestra.run.smithi114.stdout: "level": "advanced", 2024-04-23T10:41:08.156 INFO:teuthology.orchestra.run.smithi114.stdout: "flags": 0, 2024-04-23T10:41:08.156 INFO:teuthology.orchestra.run.smithi114.stdout: "default_value": "True", 2024-04-23T10:41:08.157 INFO:teuthology.orchestra.run.smithi114.stdout: "min": "", 2024-04-23T10:41:08.157 INFO:teuthology.orchestra.run.smithi114.stdout: "max": "", 2024-04-23T10:41:08.157 INFO:teuthology.orchestra.run.smithi114.stdout: "enum_allowed": [], 2024-04-23T10:41:08.157 INFO:teuthology.orchestra.run.smithi114.stdout: "desc": "log to the \"cephadm\" cluster log channel\"", 2024-04-23T10:41:08.157 INFO:teuthology.orchestra.run.smithi114.stdout: "long_desc": "", 2024-04-23T10:41:08.157 INFO:teuthology.orchestra.run.smithi114.stdout: "tags": [], 2024-04-23T10:41:08.157 INFO:teuthology.orchestra.run.smithi114.stdout: "see_also": [] 2024-04-23T10:41:08.157 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-23T10:41:08.157 INFO:teuthology.orchestra.run.smithi114.stdout: "log_to_cluster_level": { 2024-04-23T10:41:08.157 INFO:teuthology.orchestra.run.smithi114.stdout: "name": "log_to_cluster_level", 2024-04-23T10:41:08.158 INFO:teuthology.orchestra.run.smithi114.stdout: "type": "str", 2024-04-23T10:41:08.158 INFO:teuthology.orchestra.run.smithi114.stdout: "level": "advanced", 2024-04-23T10:41:08.158 INFO:teuthology.orchestra.run.smithi114.stdout: "flags": 1, 2024-04-23T10:41:08.158 INFO:teuthology.orchestra.run.smithi114.stdout: "default_value": "info", 2024-04-23T10:41:08.158 INFO:teuthology.orchestra.run.smithi114.stdout: "min": "", 2024-04-23T10:41:08.158 INFO:teuthology.orchestra.run.smithi114.stdout: "max": "", 2024-04-23T10:41:08.158 INFO:teuthology.orchestra.run.smithi114.stdout: "enum_allowed": [ 2024-04-23T10:41:08.158 INFO:teuthology.orchestra.run.smithi114.stdout: "", 2024-04-23T10:41:08.159 INFO:teuthology.orchestra.run.smithi114.stdout: "critical", 2024-04-23T10:41:08.159 INFO:teuthology.orchestra.run.smithi114.stdout: "debug", 2024-04-23T10:41:08.159 INFO:teuthology.orchestra.run.smithi114.stdout: "error", 2024-04-23T10:41:08.159 INFO:teuthology.orchestra.run.smithi114.stdout: "info", 2024-04-23T10:41:08.159 INFO:teuthology.orchestra.run.smithi114.stdout: "warning" 2024-04-23T10:41:08.159 INFO:teuthology.orchestra.run.smithi114.stdout: ], 2024-04-23T10:41:08.159 INFO:teuthology.orchestra.run.smithi114.stdout: "desc": "", 2024-04-23T10:41:08.159 INFO:teuthology.orchestra.run.smithi114.stdout: "long_desc": "", 2024-04-23T10:41:08.159 INFO:teuthology.orchestra.run.smithi114.stdout: "tags": [], 2024-04-23T10:41:08.159 INFO:teuthology.orchestra.run.smithi114.stdout: "see_also": [] 2024-04-23T10:41:08.160 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-23T10:41:08.160 INFO:teuthology.orchestra.run.smithi114.stdout: "log_to_file": { 2024-04-23T10:41:08.160 INFO:teuthology.orchestra.run.smithi114.stdout: "name": "log_to_file", 2024-04-23T10:41:08.160 INFO:teuthology.orchestra.run.smithi114.stdout: "type": "bool", 2024-04-23T10:41:08.160 INFO:teuthology.orchestra.run.smithi114.stdout: "level": "advanced", 2024-04-23T10:41:08.160 INFO:teuthology.orchestra.run.smithi114.stdout: "flags": 1, 2024-04-23T10:41:08.160 INFO:teuthology.orchestra.run.smithi114.stdout: "default_value": "False", 2024-04-23T10:41:08.160 INFO:teuthology.orchestra.run.smithi114.stdout: "min": "", 2024-04-23T10:41:08.160 INFO:teuthology.orchestra.run.smithi114.stdout: "max": "", 2024-04-23T10:41:08.161 INFO:teuthology.orchestra.run.smithi114.stdout: "enum_allowed": [], 2024-04-23T10:41:08.161 INFO:teuthology.orchestra.run.smithi114.stdout: "desc": "", 2024-04-23T10:41:08.161 INFO:teuthology.orchestra.run.smithi114.stdout: "long_desc": "", 2024-04-23T10:41:08.161 INFO:teuthology.orchestra.run.smithi114.stdout: "tags": [], 2024-04-23T10:41:08.161 INFO:teuthology.orchestra.run.smithi114.stdout: "see_also": [] 2024-04-23T10:41:08.161 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-23T10:41:08.161 INFO:teuthology.orchestra.run.smithi114.stdout: "manage_etc_ceph_ceph_conf": { 2024-04-23T10:41:08.161 INFO:teuthology.orchestra.run.smithi114.stdout: "name": "manage_etc_ceph_ceph_conf", 2024-04-23T10:41:08.161 INFO:teuthology.orchestra.run.smithi114.stdout: "type": "bool", 2024-04-23T10:41:08.161 INFO:teuthology.orchestra.run.smithi114.stdout: "level": "advanced", 2024-04-23T10:41:08.162 INFO:teuthology.orchestra.run.smithi114.stdout: "flags": 0, 2024-04-23T10:41:08.162 INFO:teuthology.orchestra.run.smithi114.stdout: "default_value": "False", 2024-04-23T10:41:08.162 INFO:teuthology.orchestra.run.smithi114.stdout: "min": "", 2024-04-23T10:41:08.162 INFO:teuthology.orchestra.run.smithi114.stdout: "max": "", 2024-04-23T10:41:08.162 INFO:teuthology.orchestra.run.smithi114.stdout: "enum_allowed": [], 2024-04-23T10:41:08.162 INFO:teuthology.orchestra.run.smithi114.stdout: "desc": "Manage and own /etc/ceph/ceph.conf on the hosts.", 2024-04-23T10:41:08.162 INFO:teuthology.orchestra.run.smithi114.stdout: "long_desc": "", 2024-04-23T10:41:08.162 INFO:teuthology.orchestra.run.smithi114.stdout: "tags": [], 2024-04-23T10:41:08.163 INFO:teuthology.orchestra.run.smithi114.stdout: "see_also": [] 2024-04-23T10:41:08.163 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-23T10:41:08.163 INFO:teuthology.orchestra.run.smithi114.stdout: "manage_etc_ceph_ceph_conf_hosts": { 2024-04-23T10:41:08.163 INFO:teuthology.orchestra.run.smithi114.stdout: "name": "manage_etc_ceph_ceph_conf_hosts", 2024-04-23T10:41:08.163 INFO:teuthology.orchestra.run.smithi114.stdout: "type": "str", 2024-04-23T10:41:08.163 INFO:teuthology.orchestra.run.smithi114.stdout: "level": "advanced", 2024-04-23T10:41:08.163 INFO:teuthology.orchestra.run.smithi114.stdout: "flags": 0, 2024-04-23T10:41:08.163 INFO:teuthology.orchestra.run.smithi114.stdout: "default_value": "*", 2024-04-23T10:41:08.163 INFO:teuthology.orchestra.run.smithi114.stdout: "min": "", 2024-04-23T10:41:08.164 INFO:teuthology.orchestra.run.smithi114.stdout: "max": "", 2024-04-23T10:41:08.164 INFO:teuthology.orchestra.run.smithi114.stdout: "enum_allowed": [], 2024-04-23T10:41:08.164 INFO:teuthology.orchestra.run.smithi114.stdout: "desc": "PlacementSpec describing on which hosts to manage /etc/ceph/ceph.conf", 2024-04-23T10:41:08.164 INFO:teuthology.orchestra.run.smithi114.stdout: "long_desc": "", 2024-04-23T10:41:08.164 INFO:teuthology.orchestra.run.smithi114.stdout: "tags": [], 2024-04-23T10:41:08.164 INFO:teuthology.orchestra.run.smithi114.stdout: "see_also": [] 2024-04-23T10:41:08.164 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-23T10:41:08.164 INFO:teuthology.orchestra.run.smithi114.stdout: "max_count_per_host": { 2024-04-23T10:41:08.164 INFO:teuthology.orchestra.run.smithi114.stdout: "name": "max_count_per_host", 2024-04-23T10:41:08.165 INFO:teuthology.orchestra.run.smithi114.stdout: "type": "int", 2024-04-23T10:41:08.165 INFO:teuthology.orchestra.run.smithi114.stdout: "level": "advanced", 2024-04-23T10:41:08.165 INFO:teuthology.orchestra.run.smithi114.stdout: "flags": 0, 2024-04-23T10:41:08.165 INFO:teuthology.orchestra.run.smithi114.stdout: "default_value": "10", 2024-04-23T10:41:08.165 INFO:teuthology.orchestra.run.smithi114.stdout: "min": "", 2024-04-23T10:41:08.165 INFO:teuthology.orchestra.run.smithi114.stdout: "max": "", 2024-04-23T10:41:08.165 INFO:teuthology.orchestra.run.smithi114.stdout: "enum_allowed": [], 2024-04-23T10:41:08.165 INFO:teuthology.orchestra.run.smithi114.stdout: "desc": "max number of daemons per service per host", 2024-04-23T10:41:08.165 INFO:teuthology.orchestra.run.smithi114.stdout: "long_desc": "", 2024-04-23T10:41:08.166 INFO:teuthology.orchestra.run.smithi114.stdout: "tags": [], 2024-04-23T10:41:08.166 INFO:teuthology.orchestra.run.smithi114.stdout: "see_also": [] 2024-04-23T10:41:08.166 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-23T10:41:08.166 INFO:teuthology.orchestra.run.smithi114.stdout: "max_osd_draining_count": { 2024-04-23T10:41:08.166 INFO:teuthology.orchestra.run.smithi114.stdout: "name": "max_osd_draining_count", 2024-04-23T10:41:08.166 INFO:teuthology.orchestra.run.smithi114.stdout: "type": "int", 2024-04-23T10:41:08.166 INFO:teuthology.orchestra.run.smithi114.stdout: "level": "advanced", 2024-04-23T10:41:08.166 INFO:teuthology.orchestra.run.smithi114.stdout: "flags": 0, 2024-04-23T10:41:08.166 INFO:teuthology.orchestra.run.smithi114.stdout: "default_value": "10", 2024-04-23T10:41:08.167 INFO:teuthology.orchestra.run.smithi114.stdout: "min": "", 2024-04-23T10:41:08.167 INFO:teuthology.orchestra.run.smithi114.stdout: "max": "", 2024-04-23T10:41:08.167 INFO:teuthology.orchestra.run.smithi114.stdout: "enum_allowed": [], 2024-04-23T10:41:08.167 INFO:teuthology.orchestra.run.smithi114.stdout: "desc": "max number of osds that will be drained simultaneously when osds are removed", 2024-04-23T10:41:08.167 INFO:teuthology.orchestra.run.smithi114.stdout: "long_desc": "", 2024-04-23T10:41:08.167 INFO:teuthology.orchestra.run.smithi114.stdout: "tags": [], 2024-04-23T10:41:08.167 INFO:teuthology.orchestra.run.smithi114.stdout: "see_also": [] 2024-04-23T10:41:08.167 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-23T10:41:08.167 INFO:teuthology.orchestra.run.smithi114.stdout: "migration_current": { 2024-04-23T10:41:08.168 INFO:teuthology.orchestra.run.smithi114.stdout: "name": "migration_current", 2024-04-23T10:41:08.168 INFO:teuthology.orchestra.run.smithi114.stdout: "type": "int", 2024-04-23T10:41:08.168 INFO:teuthology.orchestra.run.smithi114.stdout: "level": "advanced", 2024-04-23T10:41:08.168 INFO:teuthology.orchestra.run.smithi114.stdout: "flags": 0, 2024-04-23T10:41:08.168 INFO:teuthology.orchestra.run.smithi114.stdout: "default_value": "", 2024-04-23T10:41:08.168 INFO:teuthology.orchestra.run.smithi114.stdout: "min": "", 2024-04-23T10:41:08.168 INFO:teuthology.orchestra.run.smithi114.stdout: "max": "", 2024-04-23T10:41:08.168 INFO:teuthology.orchestra.run.smithi114.stdout: "enum_allowed": [], 2024-04-23T10:41:08.168 INFO:teuthology.orchestra.run.smithi114.stdout: "desc": "internal - do not modify", 2024-04-23T10:41:08.169 INFO:teuthology.orchestra.run.smithi114.stdout: "long_desc": "", 2024-04-23T10:41:08.169 INFO:teuthology.orchestra.run.smithi114.stdout: "tags": [], 2024-04-23T10:41:08.169 INFO:teuthology.orchestra.run.smithi114.stdout: "see_also": [] 2024-04-23T10:41:08.169 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-23T10:41:08.169 INFO:teuthology.orchestra.run.smithi114.stdout: "mode": { 2024-04-23T10:41:08.169 INFO:teuthology.orchestra.run.smithi114.stdout: "name": "mode", 2024-04-23T10:41:08.169 INFO:teuthology.orchestra.run.smithi114.stdout: "type": "str", 2024-04-23T10:41:08.169 INFO:teuthology.orchestra.run.smithi114.stdout: "level": "advanced", 2024-04-23T10:41:08.169 INFO:teuthology.orchestra.run.smithi114.stdout: "flags": 0, 2024-04-23T10:41:08.170 INFO:teuthology.orchestra.run.smithi114.stdout: "default_value": "root", 2024-04-23T10:41:08.170 INFO:teuthology.orchestra.run.smithi114.stdout: "min": "", 2024-04-23T10:41:08.170 INFO:teuthology.orchestra.run.smithi114.stdout: "max": "", 2024-04-23T10:41:08.170 INFO:teuthology.orchestra.run.smithi114.stdout: "enum_allowed": [ 2024-04-23T10:41:08.170 INFO:teuthology.orchestra.run.smithi114.stdout: "cephadm-package", 2024-04-23T10:41:08.170 INFO:teuthology.orchestra.run.smithi114.stdout: "root" 2024-04-23T10:41:08.170 INFO:teuthology.orchestra.run.smithi114.stdout: ], 2024-04-23T10:41:08.170 INFO:teuthology.orchestra.run.smithi114.stdout: "desc": "mode for remote execution of cephadm", 2024-04-23T10:41:08.171 INFO:teuthology.orchestra.run.smithi114.stdout: "long_desc": "", 2024-04-23T10:41:08.171 INFO:teuthology.orchestra.run.smithi114.stdout: "tags": [], 2024-04-23T10:41:08.171 INFO:teuthology.orchestra.run.smithi114.stdout: "see_also": [] 2024-04-23T10:41:08.171 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-23T10:41:08.171 INFO:teuthology.orchestra.run.smithi114.stdout: "oob_default_addr": { 2024-04-23T10:41:08.171 INFO:teuthology.orchestra.run.smithi114.stdout: "name": "oob_default_addr", 2024-04-23T10:41:08.171 INFO:teuthology.orchestra.run.smithi114.stdout: "type": "str", 2024-04-23T10:41:08.171 INFO:teuthology.orchestra.run.smithi114.stdout: "level": "advanced", 2024-04-23T10:41:08.171 INFO:teuthology.orchestra.run.smithi114.stdout: "flags": 0, 2024-04-23T10:41:08.172 INFO:teuthology.orchestra.run.smithi114.stdout: "default_value": "169.254.1.1", 2024-04-23T10:41:08.172 INFO:teuthology.orchestra.run.smithi114.stdout: "min": "", 2024-04-23T10:41:08.172 INFO:teuthology.orchestra.run.smithi114.stdout: "max": "", 2024-04-23T10:41:08.172 INFO:teuthology.orchestra.run.smithi114.stdout: "enum_allowed": [], 2024-04-23T10:41:08.172 INFO:teuthology.orchestra.run.smithi114.stdout: "desc": "Default address for RedFish API (oob management).", 2024-04-23T10:41:08.172 INFO:teuthology.orchestra.run.smithi114.stdout: "long_desc": "", 2024-04-23T10:41:08.172 INFO:teuthology.orchestra.run.smithi114.stdout: "tags": [], 2024-04-23T10:41:08.172 INFO:teuthology.orchestra.run.smithi114.stdout: "see_also": [] 2024-04-23T10:41:08.173 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-23T10:41:08.173 INFO:teuthology.orchestra.run.smithi114.stdout: "prometheus_alerts_path": { 2024-04-23T10:41:08.173 INFO:teuthology.orchestra.run.smithi114.stdout: "name": "prometheus_alerts_path", 2024-04-23T10:41:08.173 INFO:teuthology.orchestra.run.smithi114.stdout: "type": "str", 2024-04-23T10:41:08.173 INFO:teuthology.orchestra.run.smithi114.stdout: "level": "advanced", 2024-04-23T10:41:08.173 INFO:teuthology.orchestra.run.smithi114.stdout: "flags": 0, 2024-04-23T10:41:08.173 INFO:teuthology.orchestra.run.smithi114.stdout: "default_value": "/etc/prometheus/ceph/ceph_default_alerts.yml", 2024-04-23T10:41:08.173 INFO:teuthology.orchestra.run.smithi114.stdout: "min": "", 2024-04-23T10:41:08.173 INFO:teuthology.orchestra.run.smithi114.stdout: "max": "", 2024-04-23T10:41:08.173 INFO:teuthology.orchestra.run.smithi114.stdout: "enum_allowed": [], 2024-04-23T10:41:08.174 INFO:teuthology.orchestra.run.smithi114.stdout: "desc": "location of alerts to include in prometheus deployments", 2024-04-23T10:41:08.174 INFO:teuthology.orchestra.run.smithi114.stdout: "long_desc": "", 2024-04-23T10:41:08.174 INFO:teuthology.orchestra.run.smithi114.stdout: "tags": [], 2024-04-23T10:41:08.174 INFO:teuthology.orchestra.run.smithi114.stdout: "see_also": [] 2024-04-23T10:41:08.174 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-23T10:41:08.174 INFO:teuthology.orchestra.run.smithi114.stdout: "registry_insecure": { 2024-04-23T10:41:08.174 INFO:teuthology.orchestra.run.smithi114.stdout: "name": "registry_insecure", 2024-04-23T10:41:08.175 INFO:teuthology.orchestra.run.smithi114.stdout: "type": "bool", 2024-04-23T10:41:08.175 INFO:teuthology.orchestra.run.smithi114.stdout: "level": "advanced", 2024-04-23T10:41:08.175 INFO:teuthology.orchestra.run.smithi114.stdout: "flags": 0, 2024-04-23T10:41:08.175 INFO:teuthology.orchestra.run.smithi114.stdout: "default_value": "False", 2024-04-23T10:41:08.175 INFO:teuthology.orchestra.run.smithi114.stdout: "min": "", 2024-04-23T10:41:08.175 INFO:teuthology.orchestra.run.smithi114.stdout: "max": "", 2024-04-23T10:41:08.175 INFO:teuthology.orchestra.run.smithi114.stdout: "enum_allowed": [], 2024-04-23T10:41:08.175 INFO:teuthology.orchestra.run.smithi114.stdout: "desc": "Registry is to be considered insecure (no TLS available). Only for development purposes.", 2024-04-23T10:41:08.175 INFO:teuthology.orchestra.run.smithi114.stdout: "long_desc": "", 2024-04-23T10:41:08.176 INFO:teuthology.orchestra.run.smithi114.stdout: "tags": [], 2024-04-23T10:41:08.176 INFO:teuthology.orchestra.run.smithi114.stdout: "see_also": [] 2024-04-23T10:41:08.176 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-23T10:41:08.176 INFO:teuthology.orchestra.run.smithi114.stdout: "registry_password": { 2024-04-23T10:41:08.176 INFO:teuthology.orchestra.run.smithi114.stdout: "name": "registry_password", 2024-04-23T10:41:08.176 INFO:teuthology.orchestra.run.smithi114.stdout: "type": "str", 2024-04-23T10:41:08.176 INFO:teuthology.orchestra.run.smithi114.stdout: "level": "advanced", 2024-04-23T10:41:08.176 INFO:teuthology.orchestra.run.smithi114.stdout: "flags": 0, 2024-04-23T10:41:08.177 INFO:teuthology.orchestra.run.smithi114.stdout: "default_value": "", 2024-04-23T10:41:08.177 INFO:teuthology.orchestra.run.smithi114.stdout: "min": "", 2024-04-23T10:41:08.177 INFO:teuthology.orchestra.run.smithi114.stdout: "max": "", 2024-04-23T10:41:08.177 INFO:teuthology.orchestra.run.smithi114.stdout: "enum_allowed": [], 2024-04-23T10:41:08.177 INFO:teuthology.orchestra.run.smithi114.stdout: "desc": "Custom repository password. Only used for logging into a registry.", 2024-04-23T10:41:08.177 INFO:teuthology.orchestra.run.smithi114.stdout: "long_desc": "", 2024-04-23T10:41:08.177 INFO:teuthology.orchestra.run.smithi114.stdout: "tags": [], 2024-04-23T10:41:08.177 INFO:teuthology.orchestra.run.smithi114.stdout: "see_also": [] 2024-04-23T10:41:08.177 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-23T10:41:08.178 INFO:teuthology.orchestra.run.smithi114.stdout: "registry_url": { 2024-04-23T10:41:08.178 INFO:teuthology.orchestra.run.smithi114.stdout: "name": "registry_url", 2024-04-23T10:41:08.178 INFO:teuthology.orchestra.run.smithi114.stdout: "type": "str", 2024-04-23T10:41:08.178 INFO:teuthology.orchestra.run.smithi114.stdout: "level": "advanced", 2024-04-23T10:41:08.178 INFO:teuthology.orchestra.run.smithi114.stdout: "flags": 0, 2024-04-23T10:41:08.178 INFO:teuthology.orchestra.run.smithi114.stdout: "default_value": "", 2024-04-23T10:41:08.178 INFO:teuthology.orchestra.run.smithi114.stdout: "min": "", 2024-04-23T10:41:08.178 INFO:teuthology.orchestra.run.smithi114.stdout: "max": "", 2024-04-23T10:41:08.179 INFO:teuthology.orchestra.run.smithi114.stdout: "enum_allowed": [], 2024-04-23T10:41:08.179 INFO:teuthology.orchestra.run.smithi114.stdout: "desc": "Registry url for login purposes. This is not the default registry", 2024-04-23T10:41:08.179 INFO:teuthology.orchestra.run.smithi114.stdout: "long_desc": "", 2024-04-23T10:41:08.179 INFO:teuthology.orchestra.run.smithi114.stdout: "tags": [], 2024-04-23T10:41:08.179 INFO:teuthology.orchestra.run.smithi114.stdout: "see_also": [] 2024-04-23T10:41:08.179 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-23T10:41:08.179 INFO:teuthology.orchestra.run.smithi114.stdout: "registry_username": { 2024-04-23T10:41:08.179 INFO:teuthology.orchestra.run.smithi114.stdout: "name": "registry_username", 2024-04-23T10:41:08.179 INFO:teuthology.orchestra.run.smithi114.stdout: "type": "str", 2024-04-23T10:41:08.179 INFO:teuthology.orchestra.run.smithi114.stdout: "level": "advanced", 2024-04-23T10:41:08.180 INFO:teuthology.orchestra.run.smithi114.stdout: "flags": 0, 2024-04-23T10:41:08.180 INFO:teuthology.orchestra.run.smithi114.stdout: "default_value": "", 2024-04-23T10:41:08.180 INFO:teuthology.orchestra.run.smithi114.stdout: "min": "", 2024-04-23T10:41:08.180 INFO:teuthology.orchestra.run.smithi114.stdout: "max": "", 2024-04-23T10:41:08.180 INFO:teuthology.orchestra.run.smithi114.stdout: "enum_allowed": [], 2024-04-23T10:41:08.180 INFO:teuthology.orchestra.run.smithi114.stdout: "desc": "Custom repository username. Only used for logging into a registry.", 2024-04-23T10:41:08.180 INFO:teuthology.orchestra.run.smithi114.stdout: "long_desc": "", 2024-04-23T10:41:08.180 INFO:teuthology.orchestra.run.smithi114.stdout: "tags": [], 2024-04-23T10:41:08.181 INFO:teuthology.orchestra.run.smithi114.stdout: "see_also": [] 2024-04-23T10:41:08.181 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-23T10:41:08.181 INFO:teuthology.orchestra.run.smithi114.stdout: "secure_monitoring_stack": { 2024-04-23T10:41:08.181 INFO:teuthology.orchestra.run.smithi114.stdout: "name": "secure_monitoring_stack", 2024-04-23T10:41:08.181 INFO:teuthology.orchestra.run.smithi114.stdout: "type": "bool", 2024-04-23T10:41:08.181 INFO:teuthology.orchestra.run.smithi114.stdout: "level": "advanced", 2024-04-23T10:41:08.181 INFO:teuthology.orchestra.run.smithi114.stdout: "flags": 0, 2024-04-23T10:41:08.181 INFO:teuthology.orchestra.run.smithi114.stdout: "default_value": "False", 2024-04-23T10:41:08.181 INFO:teuthology.orchestra.run.smithi114.stdout: "min": "", 2024-04-23T10:41:08.181 INFO:teuthology.orchestra.run.smithi114.stdout: "max": "", 2024-04-23T10:41:08.182 INFO:teuthology.orchestra.run.smithi114.stdout: "enum_allowed": [], 2024-04-23T10:41:08.182 INFO:teuthology.orchestra.run.smithi114.stdout: "desc": "Enable TLS security for all the monitoring stack daemons", 2024-04-23T10:41:08.182 INFO:teuthology.orchestra.run.smithi114.stdout: "long_desc": "", 2024-04-23T10:41:08.182 INFO:teuthology.orchestra.run.smithi114.stdout: "tags": [], 2024-04-23T10:41:08.182 INFO:teuthology.orchestra.run.smithi114.stdout: "see_also": [] 2024-04-23T10:41:08.182 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-23T10:41:08.182 INFO:teuthology.orchestra.run.smithi114.stdout: "service_discovery_port": { 2024-04-23T10:41:08.182 INFO:teuthology.orchestra.run.smithi114.stdout: "name": "service_discovery_port", 2024-04-23T10:41:08.183 INFO:teuthology.orchestra.run.smithi114.stdout: "type": "int", 2024-04-23T10:41:08.183 INFO:teuthology.orchestra.run.smithi114.stdout: "level": "advanced", 2024-04-23T10:41:08.183 INFO:teuthology.orchestra.run.smithi114.stdout: "flags": 0, 2024-04-23T10:41:08.183 INFO:teuthology.orchestra.run.smithi114.stdout: "default_value": "8765", 2024-04-23T10:41:08.183 INFO:teuthology.orchestra.run.smithi114.stdout: "min": "", 2024-04-23T10:41:08.183 INFO:teuthology.orchestra.run.smithi114.stdout: "max": "", 2024-04-23T10:41:08.183 INFO:teuthology.orchestra.run.smithi114.stdout: "enum_allowed": [], 2024-04-23T10:41:08.183 INFO:teuthology.orchestra.run.smithi114.stdout: "desc": "cephadm service discovery port", 2024-04-23T10:41:08.183 INFO:teuthology.orchestra.run.smithi114.stdout: "long_desc": "", 2024-04-23T10:41:08.183 INFO:teuthology.orchestra.run.smithi114.stdout: "tags": [], 2024-04-23T10:41:08.184 INFO:teuthology.orchestra.run.smithi114.stdout: "see_also": [] 2024-04-23T10:41:08.184 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-23T10:41:08.184 INFO:teuthology.orchestra.run.smithi114.stdout: "ssh_config_file": { 2024-04-23T10:41:08.184 INFO:teuthology.orchestra.run.smithi114.stdout: "name": "ssh_config_file", 2024-04-23T10:41:08.184 INFO:teuthology.orchestra.run.smithi114.stdout: "type": "str", 2024-04-23T10:41:08.184 INFO:teuthology.orchestra.run.smithi114.stdout: "level": "advanced", 2024-04-23T10:41:08.184 INFO:teuthology.orchestra.run.smithi114.stdout: "flags": 0, 2024-04-23T10:41:08.184 INFO:teuthology.orchestra.run.smithi114.stdout: "default_value": "", 2024-04-23T10:41:08.184 INFO:teuthology.orchestra.run.smithi114.stdout: "min": "", 2024-04-23T10:41:08.184 INFO:teuthology.orchestra.run.smithi114.stdout: "max": "", 2024-04-23T10:41:08.185 INFO:teuthology.orchestra.run.smithi114.stdout: "enum_allowed": [], 2024-04-23T10:41:08.185 INFO:teuthology.orchestra.run.smithi114.stdout: "desc": "customized SSH config file to connect to managed hosts", 2024-04-23T10:41:08.185 INFO:teuthology.orchestra.run.smithi114.stdout: "long_desc": "", 2024-04-23T10:41:08.185 INFO:teuthology.orchestra.run.smithi114.stdout: "tags": [], 2024-04-23T10:41:08.185 INFO:teuthology.orchestra.run.smithi114.stdout: "see_also": [] 2024-04-23T10:41:08.185 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-23T10:41:08.185 INFO:teuthology.orchestra.run.smithi114.stdout: "use_agent": { 2024-04-23T10:41:08.185 INFO:teuthology.orchestra.run.smithi114.stdout: "name": "use_agent", 2024-04-23T10:41:08.185 INFO:teuthology.orchestra.run.smithi114.stdout: "type": "bool", 2024-04-23T10:41:08.186 INFO:teuthology.orchestra.run.smithi114.stdout: "level": "advanced", 2024-04-23T10:41:08.186 INFO:teuthology.orchestra.run.smithi114.stdout: "flags": 0, 2024-04-23T10:41:08.186 INFO:teuthology.orchestra.run.smithi114.stdout: "default_value": "False", 2024-04-23T10:41:08.186 INFO:teuthology.orchestra.run.smithi114.stdout: "min": "", 2024-04-23T10:41:08.186 INFO:teuthology.orchestra.run.smithi114.stdout: "max": "", 2024-04-23T10:41:08.186 INFO:teuthology.orchestra.run.smithi114.stdout: "enum_allowed": [], 2024-04-23T10:41:08.186 INFO:teuthology.orchestra.run.smithi114.stdout: "desc": "Use cephadm agent on each host to gather and send metadata", 2024-04-23T10:41:08.186 INFO:teuthology.orchestra.run.smithi114.stdout: "long_desc": "", 2024-04-23T10:41:08.186 INFO:teuthology.orchestra.run.smithi114.stdout: "tags": [], 2024-04-23T10:41:08.187 INFO:teuthology.orchestra.run.smithi114.stdout: "see_also": [] 2024-04-23T10:41:08.187 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-23T10:41:08.187 INFO:teuthology.orchestra.run.smithi114.stdout: "use_repo_digest": { 2024-04-23T10:41:08.187 INFO:teuthology.orchestra.run.smithi114.stdout: "name": "use_repo_digest", 2024-04-23T10:41:08.187 INFO:teuthology.orchestra.run.smithi114.stdout: "type": "bool", 2024-04-23T10:41:08.187 INFO:teuthology.orchestra.run.smithi114.stdout: "level": "advanced", 2024-04-23T10:41:08.187 INFO:teuthology.orchestra.run.smithi114.stdout: "flags": 0, 2024-04-23T10:41:08.187 INFO:teuthology.orchestra.run.smithi114.stdout: "default_value": "True", 2024-04-23T10:41:08.187 INFO:teuthology.orchestra.run.smithi114.stdout: "min": "", 2024-04-23T10:41:08.187 INFO:teuthology.orchestra.run.smithi114.stdout: "max": "", 2024-04-23T10:41:08.188 INFO:teuthology.orchestra.run.smithi114.stdout: "enum_allowed": [], 2024-04-23T10:41:08.188 INFO:teuthology.orchestra.run.smithi114.stdout: "desc": "Automatically convert image tags to image digest. Make sure all daemons use the same image", 2024-04-23T10:41:08.188 INFO:teuthology.orchestra.run.smithi114.stdout: "long_desc": "", 2024-04-23T10:41:08.188 INFO:teuthology.orchestra.run.smithi114.stdout: "tags": [], 2024-04-23T10:41:08.188 INFO:teuthology.orchestra.run.smithi114.stdout: "see_also": [] 2024-04-23T10:41:08.188 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-23T10:41:08.188 INFO:teuthology.orchestra.run.smithi114.stdout: "warn_on_failed_host_check": { 2024-04-23T10:41:08.188 INFO:teuthology.orchestra.run.smithi114.stdout: "name": "warn_on_failed_host_check", 2024-04-23T10:41:08.188 INFO:teuthology.orchestra.run.smithi114.stdout: "type": "bool", 2024-04-23T10:41:08.188 INFO:teuthology.orchestra.run.smithi114.stdout: "level": "advanced", 2024-04-23T10:41:08.189 INFO:teuthology.orchestra.run.smithi114.stdout: "flags": 0, 2024-04-23T10:41:08.189 INFO:teuthology.orchestra.run.smithi114.stdout: "default_value": "True", 2024-04-23T10:41:08.189 INFO:teuthology.orchestra.run.smithi114.stdout: "min": "", 2024-04-23T10:41:08.189 INFO:teuthology.orchestra.run.smithi114.stdout: "max": "", 2024-04-23T10:41:08.189 INFO:teuthology.orchestra.run.smithi114.stdout: "enum_allowed": [], 2024-04-23T10:41:08.189 INFO:teuthology.orchestra.run.smithi114.stdout: "desc": "raise a health warning if the host check fails", 2024-04-23T10:41:08.189 INFO:teuthology.orchestra.run.smithi114.stdout: "long_desc": "", 2024-04-23T10:41:08.189 INFO:teuthology.orchestra.run.smithi114.stdout: "tags": [], 2024-04-23T10:41:08.189 INFO:teuthology.orchestra.run.smithi114.stdout: "see_also": [] 2024-04-23T10:41:08.190 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-23T10:41:08.190 INFO:teuthology.orchestra.run.smithi114.stdout: "warn_on_stray_daemons": { 2024-04-23T10:41:08.190 INFO:teuthology.orchestra.run.smithi114.stdout: "name": "warn_on_stray_daemons", 2024-04-23T10:41:08.190 INFO:teuthology.orchestra.run.smithi114.stdout: "type": "bool", 2024-04-23T10:41:08.190 INFO:teuthology.orchestra.run.smithi114.stdout: "level": "advanced", 2024-04-23T10:41:08.190 INFO:teuthology.orchestra.run.smithi114.stdout: "flags": 0, 2024-04-23T10:41:08.190 INFO:teuthology.orchestra.run.smithi114.stdout: "default_value": "True", 2024-04-23T10:41:08.190 INFO:teuthology.orchestra.run.smithi114.stdout: "min": "", 2024-04-23T10:41:08.190 INFO:teuthology.orchestra.run.smithi114.stdout: "max": "", 2024-04-23T10:41:08.191 INFO:teuthology.orchestra.run.smithi114.stdout: "enum_allowed": [], 2024-04-23T10:41:08.191 INFO:teuthology.orchestra.run.smithi114.stdout: "desc": "raise a health warning if daemons are detected that are not managed by cephadm", 2024-04-23T10:41:08.191 INFO:teuthology.orchestra.run.smithi114.stdout: "long_desc": "", 2024-04-23T10:41:08.191 INFO:teuthology.orchestra.run.smithi114.stdout: "tags": [], 2024-04-23T10:41:08.191 INFO:teuthology.orchestra.run.smithi114.stdout: "see_also": [] 2024-04-23T10:41:08.191 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-23T10:41:08.191 INFO:teuthology.orchestra.run.smithi114.stdout: "warn_on_stray_hosts": { 2024-04-23T10:41:08.191 INFO:teuthology.orchestra.run.smithi114.stdout: "name": "warn_on_stray_hosts", 2024-04-23T10:41:08.191 INFO:teuthology.orchestra.run.smithi114.stdout: "type": "bool", 2024-04-23T10:41:08.191 INFO:teuthology.orchestra.run.smithi114.stdout: "level": "advanced", 2024-04-23T10:41:08.192 INFO:teuthology.orchestra.run.smithi114.stdout: "flags": 0, 2024-04-23T10:41:08.192 INFO:teuthology.orchestra.run.smithi114.stdout: "default_value": "True", 2024-04-23T10:41:08.192 INFO:teuthology.orchestra.run.smithi114.stdout: "min": "", 2024-04-23T10:41:08.192 INFO:teuthology.orchestra.run.smithi114.stdout: "max": "", 2024-04-23T10:41:08.192 INFO:teuthology.orchestra.run.smithi114.stdout: "enum_allowed": [], 2024-04-23T10:41:08.192 INFO:teuthology.orchestra.run.smithi114.stdout: "desc": "raise a health warning if daemons are detected on a host that is not managed by cephadm", 2024-04-23T10:41:08.192 INFO:teuthology.orchestra.run.smithi114.stdout: "long_desc": "", 2024-04-23T10:41:08.192 INFO:teuthology.orchestra.run.smithi114.stdout: "tags": [], 2024-04-23T10:41:08.192 INFO:teuthology.orchestra.run.smithi114.stdout: "see_also": [] 2024-04-23T10:41:08.192 INFO:teuthology.orchestra.run.smithi114.stdout: } 2024-04-23T10:41:08.193 INFO:teuthology.orchestra.run.smithi114.stdout: } 2024-04-23T10:41:08.193 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-23T10:41:08.193 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-04-23T10:41:08.193 INFO:teuthology.orchestra.run.smithi114.stdout: "name": "crash", 2024-04-23T10:41:08.193 INFO:teuthology.orchestra.run.smithi114.stdout: "can_run": true, 2024-04-23T10:41:08.193 INFO:teuthology.orchestra.run.smithi114.stdout: "error_string": "", 2024-04-23T10:41:08.193 INFO:teuthology.orchestra.run.smithi114.stdout: "module_options": { 2024-04-23T10:41:08.193 INFO:teuthology.orchestra.run.smithi114.stdout: "log_level": { 2024-04-23T10:41:08.193 INFO:teuthology.orchestra.run.smithi114.stdout: "name": "log_level", 2024-04-23T10:41:08.194 INFO:teuthology.orchestra.run.smithi114.stdout: "type": "str", 2024-04-23T10:41:08.194 INFO:teuthology.orchestra.run.smithi114.stdout: "level": "advanced", 2024-04-23T10:41:08.194 INFO:teuthology.orchestra.run.smithi114.stdout: "flags": 1, 2024-04-23T10:41:08.194 INFO:teuthology.orchestra.run.smithi114.stdout: "default_value": "", 2024-04-23T10:41:08.194 INFO:teuthology.orchestra.run.smithi114.stdout: "min": "", 2024-04-23T10:41:08.194 INFO:teuthology.orchestra.run.smithi114.stdout: "max": "", 2024-04-23T10:41:08.194 INFO:teuthology.orchestra.run.smithi114.stdout: "enum_allowed": [ 2024-04-23T10:41:08.194 INFO:teuthology.orchestra.run.smithi114.stdout: "", 2024-04-23T10:41:08.194 INFO:teuthology.orchestra.run.smithi114.stdout: "critical", 2024-04-23T10:41:08.195 INFO:teuthology.orchestra.run.smithi114.stdout: "debug", 2024-04-23T10:41:08.195 INFO:teuthology.orchestra.run.smithi114.stdout: "error", 2024-04-23T10:41:08.195 INFO:teuthology.orchestra.run.smithi114.stdout: "info", 2024-04-23T10:41:08.195 INFO:teuthology.orchestra.run.smithi114.stdout: "warning" 2024-04-23T10:41:08.195 INFO:teuthology.orchestra.run.smithi114.stdout: ], 2024-04-23T10:41:08.195 INFO:teuthology.orchestra.run.smithi114.stdout: "desc": "", 2024-04-23T10:41:08.195 INFO:teuthology.orchestra.run.smithi114.stdout: "long_desc": "", 2024-04-23T10:41:08.195 INFO:teuthology.orchestra.run.smithi114.stdout: "tags": [], 2024-04-23T10:41:08.195 INFO:teuthology.orchestra.run.smithi114.stdout: "see_also": [] 2024-04-23T10:41:08.196 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-23T10:41:08.196 INFO:teuthology.orchestra.run.smithi114.stdout: "log_to_cluster": { 2024-04-23T10:41:08.196 INFO:teuthology.orchestra.run.smithi114.stdout: "name": "log_to_cluster", 2024-04-23T10:41:08.196 INFO:teuthology.orchestra.run.smithi114.stdout: "type": "bool", 2024-04-23T10:41:08.196 INFO:teuthology.orchestra.run.smithi114.stdout: "level": "advanced", 2024-04-23T10:41:08.196 INFO:teuthology.orchestra.run.smithi114.stdout: "flags": 1, 2024-04-23T10:41:08.196 INFO:teuthology.orchestra.run.smithi114.stdout: "default_value": "False", 2024-04-23T10:41:08.196 INFO:teuthology.orchestra.run.smithi114.stdout: "min": "", 2024-04-23T10:41:08.196 INFO:teuthology.orchestra.run.smithi114.stdout: "max": "", 2024-04-23T10:41:08.196 INFO:teuthology.orchestra.run.smithi114.stdout: "enum_allowed": [], 2024-04-23T10:41:08.197 INFO:teuthology.orchestra.run.smithi114.stdout: "desc": "", 2024-04-23T10:41:08.197 INFO:teuthology.orchestra.run.smithi114.stdout: "long_desc": "", 2024-04-23T10:41:08.197 INFO:teuthology.orchestra.run.smithi114.stdout: "tags": [], 2024-04-23T10:41:08.197 INFO:teuthology.orchestra.run.smithi114.stdout: "see_also": [] 2024-04-23T10:41:08.197 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-23T10:41:08.197 INFO:teuthology.orchestra.run.smithi114.stdout: "log_to_cluster_level": { 2024-04-23T10:41:08.197 INFO:teuthology.orchestra.run.smithi114.stdout: "name": "log_to_cluster_level", 2024-04-23T10:41:08.197 INFO:teuthology.orchestra.run.smithi114.stdout: "type": "str", 2024-04-23T10:41:08.197 INFO:teuthology.orchestra.run.smithi114.stdout: "level": "advanced", 2024-04-23T10:41:08.198 INFO:teuthology.orchestra.run.smithi114.stdout: "flags": 1, 2024-04-23T10:41:08.198 INFO:teuthology.orchestra.run.smithi114.stdout: "default_value": "info", 2024-04-23T10:41:08.198 INFO:teuthology.orchestra.run.smithi114.stdout: "min": "", 2024-04-23T10:41:08.198 INFO:teuthology.orchestra.run.smithi114.stdout: "max": "", 2024-04-23T10:41:08.198 INFO:teuthology.orchestra.run.smithi114.stdout: "enum_allowed": [ 2024-04-23T10:41:08.198 INFO:teuthology.orchestra.run.smithi114.stdout: "", 2024-04-23T10:41:08.198 INFO:teuthology.orchestra.run.smithi114.stdout: "critical", 2024-04-23T10:41:08.198 INFO:teuthology.orchestra.run.smithi114.stdout: "debug", 2024-04-23T10:41:08.198 INFO:teuthology.orchestra.run.smithi114.stdout: "error", 2024-04-23T10:41:08.199 INFO:teuthology.orchestra.run.smithi114.stdout: "info", 2024-04-23T10:41:08.199 INFO:teuthology.orchestra.run.smithi114.stdout: "warning" 2024-04-23T10:41:08.199 INFO:teuthology.orchestra.run.smithi114.stdout: ], 2024-04-23T10:41:08.199 INFO:teuthology.orchestra.run.smithi114.stdout: "desc": "", 2024-04-23T10:41:08.199 INFO:teuthology.orchestra.run.smithi114.stdout: "long_desc": "", 2024-04-23T10:41:08.199 INFO:teuthology.orchestra.run.smithi114.stdout: "tags": [], 2024-04-23T10:41:08.199 INFO:teuthology.orchestra.run.smithi114.stdout: "see_also": [] 2024-04-23T10:41:08.199 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-23T10:41:08.199 INFO:teuthology.orchestra.run.smithi114.stdout: "log_to_file": { 2024-04-23T10:41:08.199 INFO:teuthology.orchestra.run.smithi114.stdout: "name": "log_to_file", 2024-04-23T10:41:08.200 INFO:teuthology.orchestra.run.smithi114.stdout: "type": "bool", 2024-04-23T10:41:08.200 INFO:teuthology.orchestra.run.smithi114.stdout: "level": "advanced", 2024-04-23T10:41:08.200 INFO:teuthology.orchestra.run.smithi114.stdout: "flags": 1, 2024-04-23T10:41:08.200 INFO:teuthology.orchestra.run.smithi114.stdout: "default_value": "False", 2024-04-23T10:41:08.200 INFO:teuthology.orchestra.run.smithi114.stdout: "min": "", 2024-04-23T10:41:08.200 INFO:teuthology.orchestra.run.smithi114.stdout: "max": "", 2024-04-23T10:41:08.200 INFO:teuthology.orchestra.run.smithi114.stdout: "enum_allowed": [], 2024-04-23T10:41:08.200 INFO:teuthology.orchestra.run.smithi114.stdout: "desc": "", 2024-04-23T10:41:08.200 INFO:teuthology.orchestra.run.smithi114.stdout: "long_desc": "", 2024-04-23T10:41:08.200 INFO:teuthology.orchestra.run.smithi114.stdout: "tags": [], 2024-04-23T10:41:08.201 INFO:teuthology.orchestra.run.smithi114.stdout: "see_also": [] 2024-04-23T10:41:08.201 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-23T10:41:08.201 INFO:teuthology.orchestra.run.smithi114.stdout: "retain_interval": { 2024-04-23T10:41:08.201 INFO:teuthology.orchestra.run.smithi114.stdout: "name": "retain_interval", 2024-04-23T10:41:08.201 INFO:teuthology.orchestra.run.smithi114.stdout: "type": "secs", 2024-04-23T10:41:08.201 INFO:teuthology.orchestra.run.smithi114.stdout: "level": "advanced", 2024-04-23T10:41:08.201 INFO:teuthology.orchestra.run.smithi114.stdout: "flags": 1, 2024-04-23T10:41:08.201 INFO:teuthology.orchestra.run.smithi114.stdout: "default_value": "31536000", 2024-04-23T10:41:08.201 INFO:teuthology.orchestra.run.smithi114.stdout: "min": "", 2024-04-23T10:41:08.202 INFO:teuthology.orchestra.run.smithi114.stdout: "max": "", 2024-04-23T10:41:08.202 INFO:teuthology.orchestra.run.smithi114.stdout: "enum_allowed": [], 2024-04-23T10:41:08.202 INFO:teuthology.orchestra.run.smithi114.stdout: "desc": "how long to retain crashes before pruning them", 2024-04-23T10:41:08.202 INFO:teuthology.orchestra.run.smithi114.stdout: "long_desc": "", 2024-04-23T10:41:08.202 INFO:teuthology.orchestra.run.smithi114.stdout: "tags": [], 2024-04-23T10:41:08.202 INFO:teuthology.orchestra.run.smithi114.stdout: "see_also": [] 2024-04-23T10:41:08.202 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-23T10:41:08.202 INFO:teuthology.orchestra.run.smithi114.stdout: "warn_recent_interval": { 2024-04-23T10:41:08.202 INFO:teuthology.orchestra.run.smithi114.stdout: "name": "warn_recent_interval", 2024-04-23T10:41:08.203 INFO:teuthology.orchestra.run.smithi114.stdout: "type": "secs", 2024-04-23T10:41:08.203 INFO:teuthology.orchestra.run.smithi114.stdout: "level": "advanced", 2024-04-23T10:41:08.203 INFO:teuthology.orchestra.run.smithi114.stdout: "flags": 1, 2024-04-23T10:41:08.203 INFO:teuthology.orchestra.run.smithi114.stdout: "default_value": "1209600", 2024-04-23T10:41:08.203 INFO:teuthology.orchestra.run.smithi114.stdout: "min": "", 2024-04-23T10:41:08.203 INFO:teuthology.orchestra.run.smithi114.stdout: "max": "", 2024-04-23T10:41:08.203 INFO:teuthology.orchestra.run.smithi114.stdout: "enum_allowed": [], 2024-04-23T10:41:08.203 INFO:teuthology.orchestra.run.smithi114.stdout: "desc": "time interval in which to warn about recent crashes", 2024-04-23T10:41:08.203 INFO:teuthology.orchestra.run.smithi114.stdout: "long_desc": "", 2024-04-23T10:41:08.204 INFO:teuthology.orchestra.run.smithi114.stdout: "tags": [], 2024-04-23T10:41:08.204 INFO:teuthology.orchestra.run.smithi114.stdout: "see_also": [] 2024-04-23T10:41:08.204 INFO:teuthology.orchestra.run.smithi114.stdout: } 2024-04-23T10:41:08.204 INFO:teuthology.orchestra.run.smithi114.stdout: } 2024-04-23T10:41:08.204 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-23T10:41:08.204 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-04-23T10:41:08.204 INFO:teuthology.orchestra.run.smithi114.stdout: "name": "dashboard", 2024-04-23T10:41:08.204 INFO:teuthology.orchestra.run.smithi114.stdout: "can_run": true, 2024-04-23T10:41:08.204 INFO:teuthology.orchestra.run.smithi114.stdout: "error_string": "", 2024-04-23T10:41:08.204 INFO:teuthology.orchestra.run.smithi114.stdout: "module_options": { 2024-04-23T10:41:08.205 INFO:teuthology.orchestra.run.smithi114.stdout: "ACCOUNT_LOCKOUT_ATTEMPTS": { 2024-04-23T10:41:08.205 INFO:teuthology.orchestra.run.smithi114.stdout: "name": "ACCOUNT_LOCKOUT_ATTEMPTS", 2024-04-23T10:41:08.205 INFO:teuthology.orchestra.run.smithi114.stdout: "type": "int", 2024-04-23T10:41:08.205 INFO:teuthology.orchestra.run.smithi114.stdout: "level": "advanced", 2024-04-23T10:41:08.205 INFO:teuthology.orchestra.run.smithi114.stdout: "flags": 0, 2024-04-23T10:41:08.205 INFO:teuthology.orchestra.run.smithi114.stdout: "default_value": "10", 2024-04-23T10:41:08.205 INFO:teuthology.orchestra.run.smithi114.stdout: "min": "", 2024-04-23T10:41:08.205 INFO:teuthology.orchestra.run.smithi114.stdout: "max": "", 2024-04-23T10:41:08.205 INFO:teuthology.orchestra.run.smithi114.stdout: "enum_allowed": [], 2024-04-23T10:41:08.205 INFO:teuthology.orchestra.run.smithi114.stdout: "desc": "", 2024-04-23T10:41:08.206 INFO:teuthology.orchestra.run.smithi114.stdout: "long_desc": "", 2024-04-23T10:41:08.206 INFO:teuthology.orchestra.run.smithi114.stdout: "tags": [], 2024-04-23T10:41:08.206 INFO:teuthology.orchestra.run.smithi114.stdout: "see_also": [] 2024-04-23T10:41:08.206 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-23T10:41:08.206 INFO:teuthology.orchestra.run.smithi114.stdout: "ALERTMANAGER_API_HOST": { 2024-04-23T10:41:08.206 INFO:teuthology.orchestra.run.smithi114.stdout: "name": "ALERTMANAGER_API_HOST", 2024-04-23T10:41:08.206 INFO:teuthology.orchestra.run.smithi114.stdout: "type": "str", 2024-04-23T10:41:08.206 INFO:teuthology.orchestra.run.smithi114.stdout: "level": "advanced", 2024-04-23T10:41:08.206 INFO:teuthology.orchestra.run.smithi114.stdout: "flags": 0, 2024-04-23T10:41:08.207 INFO:teuthology.orchestra.run.smithi114.stdout: "default_value": "", 2024-04-23T10:41:08.207 INFO:teuthology.orchestra.run.smithi114.stdout: "min": "", 2024-04-23T10:41:08.207 INFO:teuthology.orchestra.run.smithi114.stdout: "max": "", 2024-04-23T10:41:08.207 INFO:teuthology.orchestra.run.smithi114.stdout: "enum_allowed": [], 2024-04-23T10:41:08.207 INFO:teuthology.orchestra.run.smithi114.stdout: "desc": "", 2024-04-23T10:41:08.207 INFO:teuthology.orchestra.run.smithi114.stdout: "long_desc": "", 2024-04-23T10:41:08.207 INFO:teuthology.orchestra.run.smithi114.stdout: "tags": [], 2024-04-23T10:41:08.207 INFO:teuthology.orchestra.run.smithi114.stdout: "see_also": [] 2024-04-23T10:41:08.207 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-23T10:41:08.207 INFO:teuthology.orchestra.run.smithi114.stdout: "ALERTMANAGER_API_SSL_VERIFY": { 2024-04-23T10:41:08.208 INFO:teuthology.orchestra.run.smithi114.stdout: "name": "ALERTMANAGER_API_SSL_VERIFY", 2024-04-23T10:41:08.208 INFO:teuthology.orchestra.run.smithi114.stdout: "type": "bool", 2024-04-23T10:41:08.208 INFO:teuthology.orchestra.run.smithi114.stdout: "level": "advanced", 2024-04-23T10:41:08.208 INFO:teuthology.orchestra.run.smithi114.stdout: "flags": 0, 2024-04-23T10:41:08.208 INFO:teuthology.orchestra.run.smithi114.stdout: "default_value": "True", 2024-04-23T10:41:08.208 INFO:teuthology.orchestra.run.smithi114.stdout: "min": "", 2024-04-23T10:41:08.208 INFO:teuthology.orchestra.run.smithi114.stdout: "max": "", 2024-04-23T10:41:08.208 INFO:teuthology.orchestra.run.smithi114.stdout: "enum_allowed": [], 2024-04-23T10:41:08.208 INFO:teuthology.orchestra.run.smithi114.stdout: "desc": "", 2024-04-23T10:41:08.208 INFO:teuthology.orchestra.run.smithi114.stdout: "long_desc": "", 2024-04-23T10:41:08.209 INFO:teuthology.orchestra.run.smithi114.stdout: "tags": [], 2024-04-23T10:41:08.209 INFO:teuthology.orchestra.run.smithi114.stdout: "see_also": [] 2024-04-23T10:41:08.209 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-23T10:41:08.209 INFO:teuthology.orchestra.run.smithi114.stdout: "AUDIT_API_ENABLED": { 2024-04-23T10:41:08.209 INFO:teuthology.orchestra.run.smithi114.stdout: "name": "AUDIT_API_ENABLED", 2024-04-23T10:41:08.209 INFO:teuthology.orchestra.run.smithi114.stdout: "type": "bool", 2024-04-23T10:41:08.209 INFO:teuthology.orchestra.run.smithi114.stdout: "level": "advanced", 2024-04-23T10:41:08.209 INFO:teuthology.orchestra.run.smithi114.stdout: "flags": 0, 2024-04-23T10:41:08.209 INFO:teuthology.orchestra.run.smithi114.stdout: "default_value": "False", 2024-04-23T10:41:08.210 INFO:teuthology.orchestra.run.smithi114.stdout: "min": "", 2024-04-23T10:41:08.210 INFO:teuthology.orchestra.run.smithi114.stdout: "max": "", 2024-04-23T10:41:08.210 INFO:teuthology.orchestra.run.smithi114.stdout: "enum_allowed": [], 2024-04-23T10:41:08.210 INFO:teuthology.orchestra.run.smithi114.stdout: "desc": "", 2024-04-23T10:41:08.210 INFO:teuthology.orchestra.run.smithi114.stdout: "long_desc": "", 2024-04-23T10:41:08.210 INFO:teuthology.orchestra.run.smithi114.stdout: "tags": [], 2024-04-23T10:41:08.210 INFO:teuthology.orchestra.run.smithi114.stdout: "see_also": [] 2024-04-23T10:41:08.210 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-23T10:41:08.210 INFO:teuthology.orchestra.run.smithi114.stdout: "AUDIT_API_LOG_PAYLOAD": { 2024-04-23T10:41:08.211 INFO:teuthology.orchestra.run.smithi114.stdout: "name": "AUDIT_API_LOG_PAYLOAD", 2024-04-23T10:41:08.211 INFO:teuthology.orchestra.run.smithi114.stdout: "type": "bool", 2024-04-23T10:41:08.211 INFO:teuthology.orchestra.run.smithi114.stdout: "level": "advanced", 2024-04-23T10:41:08.211 INFO:teuthology.orchestra.run.smithi114.stdout: "flags": 0, 2024-04-23T10:41:08.211 INFO:teuthology.orchestra.run.smithi114.stdout: "default_value": "True", 2024-04-23T10:41:08.211 INFO:teuthology.orchestra.run.smithi114.stdout: "min": "", 2024-04-23T10:41:08.211 INFO:teuthology.orchestra.run.smithi114.stdout: "max": "", 2024-04-23T10:41:08.211 INFO:teuthology.orchestra.run.smithi114.stdout: "enum_allowed": [], 2024-04-23T10:41:08.211 INFO:teuthology.orchestra.run.smithi114.stdout: "desc": "", 2024-04-23T10:41:08.211 INFO:teuthology.orchestra.run.smithi114.stdout: "long_desc": "", 2024-04-23T10:41:08.212 INFO:teuthology.orchestra.run.smithi114.stdout: "tags": [], 2024-04-23T10:41:08.212 INFO:teuthology.orchestra.run.smithi114.stdout: "see_also": [] 2024-04-23T10:41:08.212 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-23T10:41:08.212 INFO:teuthology.orchestra.run.smithi114.stdout: "ENABLE_BROWSABLE_API": { 2024-04-23T10:41:08.212 INFO:teuthology.orchestra.run.smithi114.stdout: "name": "ENABLE_BROWSABLE_API", 2024-04-23T10:41:08.212 INFO:teuthology.orchestra.run.smithi114.stdout: "type": "bool", 2024-04-23T10:41:08.212 INFO:teuthology.orchestra.run.smithi114.stdout: "level": "advanced", 2024-04-23T10:41:08.212 INFO:teuthology.orchestra.run.smithi114.stdout: "flags": 0, 2024-04-23T10:41:08.212 INFO:teuthology.orchestra.run.smithi114.stdout: "default_value": "True", 2024-04-23T10:41:08.212 INFO:teuthology.orchestra.run.smithi114.stdout: "min": "", 2024-04-23T10:41:08.213 INFO:teuthology.orchestra.run.smithi114.stdout: "max": "", 2024-04-23T10:41:08.213 INFO:teuthology.orchestra.run.smithi114.stdout: "enum_allowed": [], 2024-04-23T10:41:08.213 INFO:teuthology.orchestra.run.smithi114.stdout: "desc": "", 2024-04-23T10:41:08.213 INFO:teuthology.orchestra.run.smithi114.stdout: "long_desc": "", 2024-04-23T10:41:08.213 INFO:teuthology.orchestra.run.smithi114.stdout: "tags": [], 2024-04-23T10:41:08.213 INFO:teuthology.orchestra.run.smithi114.stdout: "see_also": [] 2024-04-23T10:41:08.213 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-23T10:41:08.213 INFO:teuthology.orchestra.run.smithi114.stdout: "FEATURE_TOGGLE_CEPHFS": { 2024-04-23T10:41:08.213 INFO:teuthology.orchestra.run.smithi114.stdout: "name": "FEATURE_TOGGLE_CEPHFS", 2024-04-23T10:41:08.214 INFO:teuthology.orchestra.run.smithi114.stdout: "type": "bool", 2024-04-23T10:41:08.214 INFO:teuthology.orchestra.run.smithi114.stdout: "level": "advanced", 2024-04-23T10:41:08.214 INFO:teuthology.orchestra.run.smithi114.stdout: "flags": 0, 2024-04-23T10:41:08.214 INFO:teuthology.orchestra.run.smithi114.stdout: "default_value": "True", 2024-04-23T10:41:08.214 INFO:teuthology.orchestra.run.smithi114.stdout: "min": "", 2024-04-23T10:41:08.214 INFO:teuthology.orchestra.run.smithi114.stdout: "max": "", 2024-04-23T10:41:08.214 INFO:teuthology.orchestra.run.smithi114.stdout: "enum_allowed": [], 2024-04-23T10:41:08.214 INFO:teuthology.orchestra.run.smithi114.stdout: "desc": "", 2024-04-23T10:41:08.214 INFO:teuthology.orchestra.run.smithi114.stdout: "long_desc": "", 2024-04-23T10:41:08.215 INFO:teuthology.orchestra.run.smithi114.stdout: "tags": [], 2024-04-23T10:41:08.215 INFO:teuthology.orchestra.run.smithi114.stdout: "see_also": [] 2024-04-23T10:41:08.215 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-23T10:41:08.215 INFO:teuthology.orchestra.run.smithi114.stdout: "FEATURE_TOGGLE_DASHBOARD": { 2024-04-23T10:41:08.215 INFO:teuthology.orchestra.run.smithi114.stdout: "name": "FEATURE_TOGGLE_DASHBOARD", 2024-04-23T10:41:08.215 INFO:teuthology.orchestra.run.smithi114.stdout: "type": "bool", 2024-04-23T10:41:08.215 INFO:teuthology.orchestra.run.smithi114.stdout: "level": "advanced", 2024-04-23T10:41:08.215 INFO:teuthology.orchestra.run.smithi114.stdout: "flags": 0, 2024-04-23T10:41:08.215 INFO:teuthology.orchestra.run.smithi114.stdout: "default_value": "True", 2024-04-23T10:41:08.215 INFO:teuthology.orchestra.run.smithi114.stdout: "min": "", 2024-04-23T10:41:08.216 INFO:teuthology.orchestra.run.smithi114.stdout: "max": "", 2024-04-23T10:41:08.216 INFO:teuthology.orchestra.run.smithi114.stdout: "enum_allowed": [], 2024-04-23T10:41:08.216 INFO:teuthology.orchestra.run.smithi114.stdout: "desc": "", 2024-04-23T10:41:08.216 INFO:teuthology.orchestra.run.smithi114.stdout: "long_desc": "", 2024-04-23T10:41:08.216 INFO:teuthology.orchestra.run.smithi114.stdout: "tags": [], 2024-04-23T10:41:08.216 INFO:teuthology.orchestra.run.smithi114.stdout: "see_also": [] 2024-04-23T10:41:08.216 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-23T10:41:08.216 INFO:teuthology.orchestra.run.smithi114.stdout: "FEATURE_TOGGLE_ISCSI": { 2024-04-23T10:41:08.216 INFO:teuthology.orchestra.run.smithi114.stdout: "name": "FEATURE_TOGGLE_ISCSI", 2024-04-23T10:41:08.216 INFO:teuthology.orchestra.run.smithi114.stdout: "type": "bool", 2024-04-23T10:41:08.217 INFO:teuthology.orchestra.run.smithi114.stdout: "level": "advanced", 2024-04-23T10:41:08.217 INFO:teuthology.orchestra.run.smithi114.stdout: "flags": 0, 2024-04-23T10:41:08.217 INFO:teuthology.orchestra.run.smithi114.stdout: "default_value": "True", 2024-04-23T10:41:08.217 INFO:teuthology.orchestra.run.smithi114.stdout: "min": "", 2024-04-23T10:41:08.217 INFO:teuthology.orchestra.run.smithi114.stdout: "max": "", 2024-04-23T10:41:08.217 INFO:teuthology.orchestra.run.smithi114.stdout: "enum_allowed": [], 2024-04-23T10:41:08.217 INFO:teuthology.orchestra.run.smithi114.stdout: "desc": "", 2024-04-23T10:41:08.217 INFO:teuthology.orchestra.run.smithi114.stdout: "long_desc": "", 2024-04-23T10:41:08.217 INFO:teuthology.orchestra.run.smithi114.stdout: "tags": [], 2024-04-23T10:41:08.217 INFO:teuthology.orchestra.run.smithi114.stdout: "see_also": [] 2024-04-23T10:41:08.218 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-23T10:41:08.218 INFO:teuthology.orchestra.run.smithi114.stdout: "FEATURE_TOGGLE_MIRRORING": { 2024-04-23T10:41:08.218 INFO:teuthology.orchestra.run.smithi114.stdout: "name": "FEATURE_TOGGLE_MIRRORING", 2024-04-23T10:41:08.218 INFO:teuthology.orchestra.run.smithi114.stdout: "type": "bool", 2024-04-23T10:41:08.218 INFO:teuthology.orchestra.run.smithi114.stdout: "level": "advanced", 2024-04-23T10:41:08.218 INFO:teuthology.orchestra.run.smithi114.stdout: "flags": 0, 2024-04-23T10:41:08.218 INFO:teuthology.orchestra.run.smithi114.stdout: "default_value": "True", 2024-04-23T10:41:08.218 INFO:teuthology.orchestra.run.smithi114.stdout: "min": "", 2024-04-23T10:41:08.218 INFO:teuthology.orchestra.run.smithi114.stdout: "max": "", 2024-04-23T10:41:08.219 INFO:teuthology.orchestra.run.smithi114.stdout: "enum_allowed": [], 2024-04-23T10:41:08.219 INFO:teuthology.orchestra.run.smithi114.stdout: "desc": "", 2024-04-23T10:41:08.219 INFO:teuthology.orchestra.run.smithi114.stdout: "long_desc": "", 2024-04-23T10:41:08.219 INFO:teuthology.orchestra.run.smithi114.stdout: "tags": [], 2024-04-23T10:41:08.219 INFO:teuthology.orchestra.run.smithi114.stdout: "see_also": [] 2024-04-23T10:41:08.219 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-23T10:41:08.219 INFO:teuthology.orchestra.run.smithi114.stdout: "FEATURE_TOGGLE_NFS": { 2024-04-23T10:41:08.219 INFO:teuthology.orchestra.run.smithi114.stdout: "name": "FEATURE_TOGGLE_NFS", 2024-04-23T10:41:08.219 INFO:teuthology.orchestra.run.smithi114.stdout: "type": "bool", 2024-04-23T10:41:08.219 INFO:teuthology.orchestra.run.smithi114.stdout: "level": "advanced", 2024-04-23T10:41:08.220 INFO:teuthology.orchestra.run.smithi114.stdout: "flags": 0, 2024-04-23T10:41:08.220 INFO:teuthology.orchestra.run.smithi114.stdout: "default_value": "True", 2024-04-23T10:41:08.220 INFO:teuthology.orchestra.run.smithi114.stdout: "min": "", 2024-04-23T10:41:08.220 INFO:teuthology.orchestra.run.smithi114.stdout: "max": "", 2024-04-23T10:41:08.220 INFO:teuthology.orchestra.run.smithi114.stdout: "enum_allowed": [], 2024-04-23T10:41:08.220 INFO:teuthology.orchestra.run.smithi114.stdout: "desc": "", 2024-04-23T10:41:08.220 INFO:teuthology.orchestra.run.smithi114.stdout: "long_desc": "", 2024-04-23T10:41:08.220 INFO:teuthology.orchestra.run.smithi114.stdout: "tags": [], 2024-04-23T10:41:08.220 INFO:teuthology.orchestra.run.smithi114.stdout: "see_also": [] 2024-04-23T10:41:08.220 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-23T10:41:08.221 INFO:teuthology.orchestra.run.smithi114.stdout: "FEATURE_TOGGLE_RBD": { 2024-04-23T10:41:08.221 INFO:teuthology.orchestra.run.smithi114.stdout: "name": "FEATURE_TOGGLE_RBD", 2024-04-23T10:41:08.221 INFO:teuthology.orchestra.run.smithi114.stdout: "type": "bool", 2024-04-23T10:41:08.221 INFO:teuthology.orchestra.run.smithi114.stdout: "level": "advanced", 2024-04-23T10:41:08.221 INFO:teuthology.orchestra.run.smithi114.stdout: "flags": 0, 2024-04-23T10:41:08.221 INFO:teuthology.orchestra.run.smithi114.stdout: "default_value": "True", 2024-04-23T10:41:08.221 INFO:teuthology.orchestra.run.smithi114.stdout: "min": "", 2024-04-23T10:41:08.221 INFO:teuthology.orchestra.run.smithi114.stdout: "max": "", 2024-04-23T10:41:08.221 INFO:teuthology.orchestra.run.smithi114.stdout: "enum_allowed": [], 2024-04-23T10:41:08.221 INFO:teuthology.orchestra.run.smithi114.stdout: "desc": "", 2024-04-23T10:41:08.222 INFO:teuthology.orchestra.run.smithi114.stdout: "long_desc": "", 2024-04-23T10:41:08.222 INFO:teuthology.orchestra.run.smithi114.stdout: "tags": [], 2024-04-23T10:41:08.222 INFO:teuthology.orchestra.run.smithi114.stdout: "see_also": [] 2024-04-23T10:41:08.222 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-23T10:41:08.222 INFO:teuthology.orchestra.run.smithi114.stdout: "FEATURE_TOGGLE_RGW": { 2024-04-23T10:41:08.222 INFO:teuthology.orchestra.run.smithi114.stdout: "name": "FEATURE_TOGGLE_RGW", 2024-04-23T10:41:08.222 INFO:teuthology.orchestra.run.smithi114.stdout: "type": "bool", 2024-04-23T10:41:08.222 INFO:teuthology.orchestra.run.smithi114.stdout: "level": "advanced", 2024-04-23T10:41:08.222 INFO:teuthology.orchestra.run.smithi114.stdout: "flags": 0, 2024-04-23T10:41:08.223 INFO:teuthology.orchestra.run.smithi114.stdout: "default_value": "True", 2024-04-23T10:41:08.223 INFO:teuthology.orchestra.run.smithi114.stdout: "min": "", 2024-04-23T10:41:08.223 INFO:teuthology.orchestra.run.smithi114.stdout: "max": "", 2024-04-23T10:41:08.223 INFO:teuthology.orchestra.run.smithi114.stdout: "enum_allowed": [], 2024-04-23T10:41:08.223 INFO:teuthology.orchestra.run.smithi114.stdout: "desc": "", 2024-04-23T10:41:08.223 INFO:teuthology.orchestra.run.smithi114.stdout: "long_desc": "", 2024-04-23T10:41:08.223 INFO:teuthology.orchestra.run.smithi114.stdout: "tags": [], 2024-04-23T10:41:08.223 INFO:teuthology.orchestra.run.smithi114.stdout: "see_also": [] 2024-04-23T10:41:08.223 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-23T10:41:08.223 INFO:teuthology.orchestra.run.smithi114.stdout: "GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE": { 2024-04-23T10:41:08.224 INFO:teuthology.orchestra.run.smithi114.stdout: "name": "GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE", 2024-04-23T10:41:08.224 INFO:teuthology.orchestra.run.smithi114.stdout: "type": "str", 2024-04-23T10:41:08.224 INFO:teuthology.orchestra.run.smithi114.stdout: "level": "advanced", 2024-04-23T10:41:08.224 INFO:teuthology.orchestra.run.smithi114.stdout: "flags": 0, 2024-04-23T10:41:08.224 INFO:teuthology.orchestra.run.smithi114.stdout: "default_value": "", 2024-04-23T10:41:08.224 INFO:teuthology.orchestra.run.smithi114.stdout: "min": "", 2024-04-23T10:41:08.224 INFO:teuthology.orchestra.run.smithi114.stdout: "max": "", 2024-04-23T10:41:08.224 INFO:teuthology.orchestra.run.smithi114.stdout: "enum_allowed": [], 2024-04-23T10:41:08.224 INFO:teuthology.orchestra.run.smithi114.stdout: "desc": "", 2024-04-23T10:41:08.225 INFO:teuthology.orchestra.run.smithi114.stdout: "long_desc": "", 2024-04-23T10:41:08.225 INFO:teuthology.orchestra.run.smithi114.stdout: "tags": [], 2024-04-23T10:41:08.225 INFO:teuthology.orchestra.run.smithi114.stdout: "see_also": [] 2024-04-23T10:41:08.225 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-23T10:41:08.225 INFO:teuthology.orchestra.run.smithi114.stdout: "GRAFANA_API_PASSWORD": { 2024-04-23T10:41:08.225 INFO:teuthology.orchestra.run.smithi114.stdout: "name": "GRAFANA_API_PASSWORD", 2024-04-23T10:41:08.225 INFO:teuthology.orchestra.run.smithi114.stdout: "type": "str", 2024-04-23T10:41:08.225 INFO:teuthology.orchestra.run.smithi114.stdout: "level": "advanced", 2024-04-23T10:41:08.225 INFO:teuthology.orchestra.run.smithi114.stdout: "flags": 0, 2024-04-23T10:41:08.225 INFO:teuthology.orchestra.run.smithi114.stdout: "default_value": "admin", 2024-04-23T10:41:08.226 INFO:teuthology.orchestra.run.smithi114.stdout: "min": "", 2024-04-23T10:41:08.226 INFO:teuthology.orchestra.run.smithi114.stdout: "max": "", 2024-04-23T10:41:08.226 INFO:teuthology.orchestra.run.smithi114.stdout: "enum_allowed": [], 2024-04-23T10:41:08.226 INFO:teuthology.orchestra.run.smithi114.stdout: "desc": "", 2024-04-23T10:41:08.226 INFO:teuthology.orchestra.run.smithi114.stdout: "long_desc": "", 2024-04-23T10:41:08.226 INFO:teuthology.orchestra.run.smithi114.stdout: "tags": [], 2024-04-23T10:41:08.226 INFO:teuthology.orchestra.run.smithi114.stdout: "see_also": [] 2024-04-23T10:41:08.226 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-23T10:41:08.226 INFO:teuthology.orchestra.run.smithi114.stdout: "GRAFANA_API_SSL_VERIFY": { 2024-04-23T10:41:08.227 INFO:teuthology.orchestra.run.smithi114.stdout: "name": "GRAFANA_API_SSL_VERIFY", 2024-04-23T10:41:08.227 INFO:teuthology.orchestra.run.smithi114.stdout: "type": "bool", 2024-04-23T10:41:08.227 INFO:teuthology.orchestra.run.smithi114.stdout: "level": "advanced", 2024-04-23T10:41:08.227 INFO:teuthology.orchestra.run.smithi114.stdout: "flags": 0, 2024-04-23T10:41:08.227 INFO:teuthology.orchestra.run.smithi114.stdout: "default_value": "True", 2024-04-23T10:41:08.227 INFO:teuthology.orchestra.run.smithi114.stdout: "min": "", 2024-04-23T10:41:08.227 INFO:teuthology.orchestra.run.smithi114.stdout: "max": "", 2024-04-23T10:41:08.227 INFO:teuthology.orchestra.run.smithi114.stdout: "enum_allowed": [], 2024-04-23T10:41:08.227 INFO:teuthology.orchestra.run.smithi114.stdout: "desc": "", 2024-04-23T10:41:08.227 INFO:teuthology.orchestra.run.smithi114.stdout: "long_desc": "", 2024-04-23T10:41:08.228 INFO:teuthology.orchestra.run.smithi114.stdout: "tags": [], 2024-04-23T10:41:08.228 INFO:teuthology.orchestra.run.smithi114.stdout: "see_also": [] 2024-04-23T10:41:08.228 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-23T10:41:08.228 INFO:teuthology.orchestra.run.smithi114.stdout: "GRAFANA_API_URL": { 2024-04-23T10:41:08.228 INFO:teuthology.orchestra.run.smithi114.stdout: "name": "GRAFANA_API_URL", 2024-04-23T10:41:08.228 INFO:teuthology.orchestra.run.smithi114.stdout: "type": "str", 2024-04-23T10:41:08.228 INFO:teuthology.orchestra.run.smithi114.stdout: "level": "advanced", 2024-04-23T10:41:08.228 INFO:teuthology.orchestra.run.smithi114.stdout: "flags": 0, 2024-04-23T10:41:08.228 INFO:teuthology.orchestra.run.smithi114.stdout: "default_value": "", 2024-04-23T10:41:08.228 INFO:teuthology.orchestra.run.smithi114.stdout: "min": "", 2024-04-23T10:41:08.229 INFO:teuthology.orchestra.run.smithi114.stdout: "max": "", 2024-04-23T10:41:08.229 INFO:teuthology.orchestra.run.smithi114.stdout: "enum_allowed": [], 2024-04-23T10:41:08.229 INFO:teuthology.orchestra.run.smithi114.stdout: "desc": "", 2024-04-23T10:41:08.229 INFO:teuthology.orchestra.run.smithi114.stdout: "long_desc": "", 2024-04-23T10:41:08.229 INFO:teuthology.orchestra.run.smithi114.stdout: "tags": [], 2024-04-23T10:41:08.229 INFO:teuthology.orchestra.run.smithi114.stdout: "see_also": [] 2024-04-23T10:41:08.229 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-23T10:41:08.229 INFO:teuthology.orchestra.run.smithi114.stdout: "GRAFANA_API_USERNAME": { 2024-04-23T10:41:08.229 INFO:teuthology.orchestra.run.smithi114.stdout: "name": "GRAFANA_API_USERNAME", 2024-04-23T10:41:08.229 INFO:teuthology.orchestra.run.smithi114.stdout: "type": "str", 2024-04-23T10:41:08.230 INFO:teuthology.orchestra.run.smithi114.stdout: "level": "advanced", 2024-04-23T10:41:08.230 INFO:teuthology.orchestra.run.smithi114.stdout: "flags": 0, 2024-04-23T10:41:08.230 INFO:teuthology.orchestra.run.smithi114.stdout: "default_value": "admin", 2024-04-23T10:41:08.230 INFO:teuthology.orchestra.run.smithi114.stdout: "min": "", 2024-04-23T10:41:08.230 INFO:teuthology.orchestra.run.smithi114.stdout: "max": "", 2024-04-23T10:41:08.230 INFO:teuthology.orchestra.run.smithi114.stdout: "enum_allowed": [], 2024-04-23T10:41:08.230 INFO:teuthology.orchestra.run.smithi114.stdout: "desc": "", 2024-04-23T10:41:08.230 INFO:teuthology.orchestra.run.smithi114.stdout: "long_desc": "", 2024-04-23T10:41:08.230 INFO:teuthology.orchestra.run.smithi114.stdout: "tags": [], 2024-04-23T10:41:08.231 INFO:teuthology.orchestra.run.smithi114.stdout: "see_also": [] 2024-04-23T10:41:08.231 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-23T10:41:08.231 INFO:teuthology.orchestra.run.smithi114.stdout: "GRAFANA_FRONTEND_API_URL": { 2024-04-23T10:41:08.231 INFO:teuthology.orchestra.run.smithi114.stdout: "name": "GRAFANA_FRONTEND_API_URL", 2024-04-23T10:41:08.231 INFO:teuthology.orchestra.run.smithi114.stdout: "type": "str", 2024-04-23T10:41:08.231 INFO:teuthology.orchestra.run.smithi114.stdout: "level": "advanced", 2024-04-23T10:41:08.231 INFO:teuthology.orchestra.run.smithi114.stdout: "flags": 0, 2024-04-23T10:41:08.231 INFO:teuthology.orchestra.run.smithi114.stdout: "default_value": "", 2024-04-23T10:41:08.231 INFO:teuthology.orchestra.run.smithi114.stdout: "min": "", 2024-04-23T10:41:08.231 INFO:teuthology.orchestra.run.smithi114.stdout: "max": "", 2024-04-23T10:41:08.232 INFO:teuthology.orchestra.run.smithi114.stdout: "enum_allowed": [], 2024-04-23T10:41:08.232 INFO:teuthology.orchestra.run.smithi114.stdout: "desc": "", 2024-04-23T10:41:08.232 INFO:teuthology.orchestra.run.smithi114.stdout: "long_desc": "", 2024-04-23T10:41:08.232 INFO:teuthology.orchestra.run.smithi114.stdout: "tags": [], 2024-04-23T10:41:08.232 INFO:teuthology.orchestra.run.smithi114.stdout: "see_also": [] 2024-04-23T10:41:08.232 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-23T10:41:08.232 INFO:teuthology.orchestra.run.smithi114.stdout: "GRAFANA_UPDATE_DASHBOARDS": { 2024-04-23T10:41:08.232 INFO:teuthology.orchestra.run.smithi114.stdout: "name": "GRAFANA_UPDATE_DASHBOARDS", 2024-04-23T10:41:08.232 INFO:teuthology.orchestra.run.smithi114.stdout: "type": "bool", 2024-04-23T10:41:08.232 INFO:teuthology.orchestra.run.smithi114.stdout: "level": "advanced", 2024-04-23T10:41:08.233 INFO:teuthology.orchestra.run.smithi114.stdout: "flags": 0, 2024-04-23T10:41:08.233 INFO:teuthology.orchestra.run.smithi114.stdout: "default_value": "False", 2024-04-23T10:41:08.233 INFO:teuthology.orchestra.run.smithi114.stdout: "min": "", 2024-04-23T10:41:08.233 INFO:teuthology.orchestra.run.smithi114.stdout: "max": "", 2024-04-23T10:41:08.233 INFO:teuthology.orchestra.run.smithi114.stdout: "enum_allowed": [], 2024-04-23T10:41:08.233 INFO:teuthology.orchestra.run.smithi114.stdout: "desc": "", 2024-04-23T10:41:08.233 INFO:teuthology.orchestra.run.smithi114.stdout: "long_desc": "", 2024-04-23T10:41:08.233 INFO:teuthology.orchestra.run.smithi114.stdout: "tags": [], 2024-04-23T10:41:08.233 INFO:teuthology.orchestra.run.smithi114.stdout: "see_also": [] 2024-04-23T10:41:08.233 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-23T10:41:08.234 INFO:teuthology.orchestra.run.smithi114.stdout: "ISCSI_API_SSL_VERIFICATION": { 2024-04-23T10:41:08.234 INFO:teuthology.orchestra.run.smithi114.stdout: "name": "ISCSI_API_SSL_VERIFICATION", 2024-04-23T10:41:08.234 INFO:teuthology.orchestra.run.smithi114.stdout: "type": "bool", 2024-04-23T10:41:08.234 INFO:teuthology.orchestra.run.smithi114.stdout: "level": "advanced", 2024-04-23T10:41:08.234 INFO:teuthology.orchestra.run.smithi114.stdout: "flags": 0, 2024-04-23T10:41:08.234 INFO:teuthology.orchestra.run.smithi114.stdout: "default_value": "True", 2024-04-23T10:41:08.234 INFO:teuthology.orchestra.run.smithi114.stdout: "min": "", 2024-04-23T10:41:08.234 INFO:teuthology.orchestra.run.smithi114.stdout: "max": "", 2024-04-23T10:41:08.234 INFO:teuthology.orchestra.run.smithi114.stdout: "enum_allowed": [], 2024-04-23T10:41:08.235 INFO:teuthology.orchestra.run.smithi114.stdout: "desc": "", 2024-04-23T10:41:08.235 INFO:teuthology.orchestra.run.smithi114.stdout: "long_desc": "", 2024-04-23T10:41:08.235 INFO:teuthology.orchestra.run.smithi114.stdout: "tags": [], 2024-04-23T10:41:08.235 INFO:teuthology.orchestra.run.smithi114.stdout: "see_also": [] 2024-04-23T10:41:08.235 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-23T10:41:08.235 INFO:teuthology.orchestra.run.smithi114.stdout: "ISSUE_TRACKER_API_KEY": { 2024-04-23T10:41:08.235 INFO:teuthology.orchestra.run.smithi114.stdout: "name": "ISSUE_TRACKER_API_KEY", 2024-04-23T10:41:08.235 INFO:teuthology.orchestra.run.smithi114.stdout: "type": "str", 2024-04-23T10:41:08.235 INFO:teuthology.orchestra.run.smithi114.stdout: "level": "advanced", 2024-04-23T10:41:08.235 INFO:teuthology.orchestra.run.smithi114.stdout: "flags": 0, 2024-04-23T10:41:08.236 INFO:teuthology.orchestra.run.smithi114.stdout: "default_value": "", 2024-04-23T10:41:08.236 INFO:teuthology.orchestra.run.smithi114.stdout: "min": "", 2024-04-23T10:41:08.236 INFO:teuthology.orchestra.run.smithi114.stdout: "max": "", 2024-04-23T10:41:08.236 INFO:teuthology.orchestra.run.smithi114.stdout: "enum_allowed": [], 2024-04-23T10:41:08.236 INFO:teuthology.orchestra.run.smithi114.stdout: "desc": "", 2024-04-23T10:41:08.236 INFO:teuthology.orchestra.run.smithi114.stdout: "long_desc": "", 2024-04-23T10:41:08.236 INFO:teuthology.orchestra.run.smithi114.stdout: "tags": [], 2024-04-23T10:41:08.236 INFO:teuthology.orchestra.run.smithi114.stdout: "see_also": [] 2024-04-23T10:41:08.236 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-23T10:41:08.237 INFO:teuthology.orchestra.run.smithi114.stdout: "MANAGED_BY_CLUSTERS": { 2024-04-23T10:41:08.237 INFO:teuthology.orchestra.run.smithi114.stdout: "name": "MANAGED_BY_CLUSTERS", 2024-04-23T10:41:08.237 INFO:teuthology.orchestra.run.smithi114.stdout: "type": "str", 2024-04-23T10:41:08.237 INFO:teuthology.orchestra.run.smithi114.stdout: "level": "advanced", 2024-04-23T10:41:08.237 INFO:teuthology.orchestra.run.smithi114.stdout: "flags": 0, 2024-04-23T10:41:08.237 INFO:teuthology.orchestra.run.smithi114.stdout: "default_value": "[]", 2024-04-23T10:41:08.237 INFO:teuthology.orchestra.run.smithi114.stdout: "min": "", 2024-04-23T10:41:08.237 INFO:teuthology.orchestra.run.smithi114.stdout: "max": "", 2024-04-23T10:41:08.237 INFO:teuthology.orchestra.run.smithi114.stdout: "enum_allowed": [], 2024-04-23T10:41:08.237 INFO:teuthology.orchestra.run.smithi114.stdout: "desc": "", 2024-04-23T10:41:08.238 INFO:teuthology.orchestra.run.smithi114.stdout: "long_desc": "", 2024-04-23T10:41:08.238 INFO:teuthology.orchestra.run.smithi114.stdout: "tags": [], 2024-04-23T10:41:08.238 INFO:teuthology.orchestra.run.smithi114.stdout: "see_also": [] 2024-04-23T10:41:08.238 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-23T10:41:08.238 INFO:teuthology.orchestra.run.smithi114.stdout: "MULTICLUSTER_CONFIG": { 2024-04-23T10:41:08.238 INFO:teuthology.orchestra.run.smithi114.stdout: "name": "MULTICLUSTER_CONFIG", 2024-04-23T10:41:08.238 INFO:teuthology.orchestra.run.smithi114.stdout: "type": "str", 2024-04-23T10:41:08.238 INFO:teuthology.orchestra.run.smithi114.stdout: "level": "advanced", 2024-04-23T10:41:08.238 INFO:teuthology.orchestra.run.smithi114.stdout: "flags": 0, 2024-04-23T10:41:08.238 INFO:teuthology.orchestra.run.smithi114.stdout: "default_value": "{}", 2024-04-23T10:41:08.239 INFO:teuthology.orchestra.run.smithi114.stdout: "min": "", 2024-04-23T10:41:08.239 INFO:teuthology.orchestra.run.smithi114.stdout: "max": "", 2024-04-23T10:41:08.239 INFO:teuthology.orchestra.run.smithi114.stdout: "enum_allowed": [], 2024-04-23T10:41:08.239 INFO:teuthology.orchestra.run.smithi114.stdout: "desc": "", 2024-04-23T10:41:08.239 INFO:teuthology.orchestra.run.smithi114.stdout: "long_desc": "", 2024-04-23T10:41:08.239 INFO:teuthology.orchestra.run.smithi114.stdout: "tags": [], 2024-04-23T10:41:08.239 INFO:teuthology.orchestra.run.smithi114.stdout: "see_also": [] 2024-04-23T10:41:08.239 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-23T10:41:08.239 INFO:teuthology.orchestra.run.smithi114.stdout: "PROMETHEUS_API_HOST": { 2024-04-23T10:41:08.239 INFO:teuthology.orchestra.run.smithi114.stdout: "name": "PROMETHEUS_API_HOST", 2024-04-23T10:41:08.240 INFO:teuthology.orchestra.run.smithi114.stdout: "type": "str", 2024-04-23T10:41:08.240 INFO:teuthology.orchestra.run.smithi114.stdout: "level": "advanced", 2024-04-23T10:41:08.240 INFO:teuthology.orchestra.run.smithi114.stdout: "flags": 0, 2024-04-23T10:41:08.240 INFO:teuthology.orchestra.run.smithi114.stdout: "default_value": "", 2024-04-23T10:41:08.240 INFO:teuthology.orchestra.run.smithi114.stdout: "min": "", 2024-04-23T10:41:08.240 INFO:teuthology.orchestra.run.smithi114.stdout: "max": "", 2024-04-23T10:41:08.240 INFO:teuthology.orchestra.run.smithi114.stdout: "enum_allowed": [], 2024-04-23T10:41:08.240 INFO:teuthology.orchestra.run.smithi114.stdout: "desc": "", 2024-04-23T10:41:08.240 INFO:teuthology.orchestra.run.smithi114.stdout: "long_desc": "", 2024-04-23T10:41:08.241 INFO:teuthology.orchestra.run.smithi114.stdout: "tags": [], 2024-04-23T10:41:08.241 INFO:teuthology.orchestra.run.smithi114.stdout: "see_also": [] 2024-04-23T10:41:08.241 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-23T10:41:08.241 INFO:teuthology.orchestra.run.smithi114.stdout: "PROMETHEUS_API_SSL_VERIFY": { 2024-04-23T10:41:08.241 INFO:teuthology.orchestra.run.smithi114.stdout: "name": "PROMETHEUS_API_SSL_VERIFY", 2024-04-23T10:41:08.241 INFO:teuthology.orchestra.run.smithi114.stdout: "type": "bool", 2024-04-23T10:41:08.241 INFO:teuthology.orchestra.run.smithi114.stdout: "level": "advanced", 2024-04-23T10:41:08.241 INFO:teuthology.orchestra.run.smithi114.stdout: "flags": 0, 2024-04-23T10:41:08.241 INFO:teuthology.orchestra.run.smithi114.stdout: "default_value": "True", 2024-04-23T10:41:08.242 INFO:teuthology.orchestra.run.smithi114.stdout: "min": "", 2024-04-23T10:41:08.242 INFO:teuthology.orchestra.run.smithi114.stdout: "max": "", 2024-04-23T10:41:08.242 INFO:teuthology.orchestra.run.smithi114.stdout: "enum_allowed": [], 2024-04-23T10:41:08.242 INFO:teuthology.orchestra.run.smithi114.stdout: "desc": "", 2024-04-23T10:41:08.242 INFO:teuthology.orchestra.run.smithi114.stdout: "long_desc": "", 2024-04-23T10:41:08.242 INFO:teuthology.orchestra.run.smithi114.stdout: "tags": [], 2024-04-23T10:41:08.242 INFO:teuthology.orchestra.run.smithi114.stdout: "see_also": [] 2024-04-23T10:41:08.242 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-23T10:41:08.242 INFO:teuthology.orchestra.run.smithi114.stdout: "PWD_POLICY_CHECK_COMPLEXITY_ENABLED": { 2024-04-23T10:41:08.243 INFO:teuthology.orchestra.run.smithi114.stdout: "name": "PWD_POLICY_CHECK_COMPLEXITY_ENABLED", 2024-04-23T10:41:08.243 INFO:teuthology.orchestra.run.smithi114.stdout: "type": "bool", 2024-04-23T10:41:08.243 INFO:teuthology.orchestra.run.smithi114.stdout: "level": "advanced", 2024-04-23T10:41:08.243 INFO:teuthology.orchestra.run.smithi114.stdout: "flags": 0, 2024-04-23T10:41:08.243 INFO:teuthology.orchestra.run.smithi114.stdout: "default_value": "False", 2024-04-23T10:41:08.243 INFO:teuthology.orchestra.run.smithi114.stdout: "min": "", 2024-04-23T10:41:08.243 INFO:teuthology.orchestra.run.smithi114.stdout: "max": "", 2024-04-23T10:41:08.243 INFO:teuthology.orchestra.run.smithi114.stdout: "enum_allowed": [], 2024-04-23T10:41:08.243 INFO:teuthology.orchestra.run.smithi114.stdout: "desc": "", 2024-04-23T10:41:08.243 INFO:teuthology.orchestra.run.smithi114.stdout: "long_desc": "", 2024-04-23T10:41:08.244 INFO:teuthology.orchestra.run.smithi114.stdout: "tags": [], 2024-04-23T10:41:08.244 INFO:teuthology.orchestra.run.smithi114.stdout: "see_also": [] 2024-04-23T10:41:08.244 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-23T10:41:08.244 INFO:teuthology.orchestra.run.smithi114.stdout: "PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED": { 2024-04-23T10:41:08.244 INFO:teuthology.orchestra.run.smithi114.stdout: "name": "PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED", 2024-04-23T10:41:08.244 INFO:teuthology.orchestra.run.smithi114.stdout: "type": "bool", 2024-04-23T10:41:08.244 INFO:teuthology.orchestra.run.smithi114.stdout: "level": "advanced", 2024-04-23T10:41:08.244 INFO:teuthology.orchestra.run.smithi114.stdout: "flags": 0, 2024-04-23T10:41:08.244 INFO:teuthology.orchestra.run.smithi114.stdout: "default_value": "False", 2024-04-23T10:41:08.244 INFO:teuthology.orchestra.run.smithi114.stdout: "min": "", 2024-04-23T10:41:08.245 INFO:teuthology.orchestra.run.smithi114.stdout: "max": "", 2024-04-23T10:41:08.245 INFO:teuthology.orchestra.run.smithi114.stdout: "enum_allowed": [], 2024-04-23T10:41:08.245 INFO:teuthology.orchestra.run.smithi114.stdout: "desc": "", 2024-04-23T10:41:08.245 INFO:teuthology.orchestra.run.smithi114.stdout: "long_desc": "", 2024-04-23T10:41:08.245 INFO:teuthology.orchestra.run.smithi114.stdout: "tags": [], 2024-04-23T10:41:08.245 INFO:teuthology.orchestra.run.smithi114.stdout: "see_also": [] 2024-04-23T10:41:08.245 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-23T10:41:08.245 INFO:teuthology.orchestra.run.smithi114.stdout: "PWD_POLICY_CHECK_LENGTH_ENABLED": { 2024-04-23T10:41:08.245 INFO:teuthology.orchestra.run.smithi114.stdout: "name": "PWD_POLICY_CHECK_LENGTH_ENABLED", 2024-04-23T10:41:08.245 INFO:teuthology.orchestra.run.smithi114.stdout: "type": "bool", 2024-04-23T10:41:08.246 INFO:teuthology.orchestra.run.smithi114.stdout: "level": "advanced", 2024-04-23T10:41:08.246 INFO:teuthology.orchestra.run.smithi114.stdout: "flags": 0, 2024-04-23T10:41:08.246 INFO:teuthology.orchestra.run.smithi114.stdout: "default_value": "True", 2024-04-23T10:41:08.246 INFO:teuthology.orchestra.run.smithi114.stdout: "min": "", 2024-04-23T10:41:08.246 INFO:teuthology.orchestra.run.smithi114.stdout: "max": "", 2024-04-23T10:41:08.246 INFO:teuthology.orchestra.run.smithi114.stdout: "enum_allowed": [], 2024-04-23T10:41:08.246 INFO:teuthology.orchestra.run.smithi114.stdout: "desc": "", 2024-04-23T10:41:08.246 INFO:teuthology.orchestra.run.smithi114.stdout: "long_desc": "", 2024-04-23T10:41:08.246 INFO:teuthology.orchestra.run.smithi114.stdout: "tags": [], 2024-04-23T10:41:08.247 INFO:teuthology.orchestra.run.smithi114.stdout: "see_also": [] 2024-04-23T10:41:08.247 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-23T10:41:08.247 INFO:teuthology.orchestra.run.smithi114.stdout: "PWD_POLICY_CHECK_OLDPWD_ENABLED": { 2024-04-23T10:41:08.247 INFO:teuthology.orchestra.run.smithi114.stdout: "name": "PWD_POLICY_CHECK_OLDPWD_ENABLED", 2024-04-23T10:41:08.247 INFO:teuthology.orchestra.run.smithi114.stdout: "type": "bool", 2024-04-23T10:41:08.247 INFO:teuthology.orchestra.run.smithi114.stdout: "level": "advanced", 2024-04-23T10:41:08.247 INFO:teuthology.orchestra.run.smithi114.stdout: "flags": 0, 2024-04-23T10:41:08.247 INFO:teuthology.orchestra.run.smithi114.stdout: "default_value": "True", 2024-04-23T10:41:08.247 INFO:teuthology.orchestra.run.smithi114.stdout: "min": "", 2024-04-23T10:41:08.247 INFO:teuthology.orchestra.run.smithi114.stdout: "max": "", 2024-04-23T10:41:08.248 INFO:teuthology.orchestra.run.smithi114.stdout: "enum_allowed": [], 2024-04-23T10:41:08.248 INFO:teuthology.orchestra.run.smithi114.stdout: "desc": "", 2024-04-23T10:41:08.248 INFO:teuthology.orchestra.run.smithi114.stdout: "long_desc": "", 2024-04-23T10:41:08.248 INFO:teuthology.orchestra.run.smithi114.stdout: "tags": [], 2024-04-23T10:41:08.248 INFO:teuthology.orchestra.run.smithi114.stdout: "see_also": [] 2024-04-23T10:41:08.248 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-23T10:41:08.248 INFO:teuthology.orchestra.run.smithi114.stdout: "PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED": { 2024-04-23T10:41:08.248 INFO:teuthology.orchestra.run.smithi114.stdout: "name": "PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED", 2024-04-23T10:41:08.248 INFO:teuthology.orchestra.run.smithi114.stdout: "type": "bool", 2024-04-23T10:41:08.248 INFO:teuthology.orchestra.run.smithi114.stdout: "level": "advanced", 2024-04-23T10:41:08.249 INFO:teuthology.orchestra.run.smithi114.stdout: "flags": 0, 2024-04-23T10:41:08.249 INFO:teuthology.orchestra.run.smithi114.stdout: "default_value": "False", 2024-04-23T10:41:08.249 INFO:teuthology.orchestra.run.smithi114.stdout: "min": "", 2024-04-23T10:41:08.249 INFO:teuthology.orchestra.run.smithi114.stdout: "max": "", 2024-04-23T10:41:08.249 INFO:teuthology.orchestra.run.smithi114.stdout: "enum_allowed": [], 2024-04-23T10:41:08.249 INFO:teuthology.orchestra.run.smithi114.stdout: "desc": "", 2024-04-23T10:41:08.249 INFO:teuthology.orchestra.run.smithi114.stdout: "long_desc": "", 2024-04-23T10:41:08.249 INFO:teuthology.orchestra.run.smithi114.stdout: "tags": [], 2024-04-23T10:41:08.249 INFO:teuthology.orchestra.run.smithi114.stdout: "see_also": [] 2024-04-23T10:41:08.249 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-23T10:41:08.250 INFO:teuthology.orchestra.run.smithi114.stdout: "PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED": { 2024-04-23T10:41:08.250 INFO:teuthology.orchestra.run.smithi114.stdout: "name": "PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED", 2024-04-23T10:41:08.250 INFO:teuthology.orchestra.run.smithi114.stdout: "type": "bool", 2024-04-23T10:41:08.250 INFO:teuthology.orchestra.run.smithi114.stdout: "level": "advanced", 2024-04-23T10:41:08.250 INFO:teuthology.orchestra.run.smithi114.stdout: "flags": 0, 2024-04-23T10:41:08.250 INFO:teuthology.orchestra.run.smithi114.stdout: "default_value": "False", 2024-04-23T10:41:08.250 INFO:teuthology.orchestra.run.smithi114.stdout: "min": "", 2024-04-23T10:41:08.250 INFO:teuthology.orchestra.run.smithi114.stdout: "max": "", 2024-04-23T10:41:08.250 INFO:teuthology.orchestra.run.smithi114.stdout: "enum_allowed": [], 2024-04-23T10:41:08.251 INFO:teuthology.orchestra.run.smithi114.stdout: "desc": "", 2024-04-23T10:41:08.251 INFO:teuthology.orchestra.run.smithi114.stdout: "long_desc": "", 2024-04-23T10:41:08.251 INFO:teuthology.orchestra.run.smithi114.stdout: "tags": [], 2024-04-23T10:41:08.251 INFO:teuthology.orchestra.run.smithi114.stdout: "see_also": [] 2024-04-23T10:41:08.251 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-23T10:41:08.251 INFO:teuthology.orchestra.run.smithi114.stdout: "PWD_POLICY_CHECK_USERNAME_ENABLED": { 2024-04-23T10:41:08.251 INFO:teuthology.orchestra.run.smithi114.stdout: "name": "PWD_POLICY_CHECK_USERNAME_ENABLED", 2024-04-23T10:41:08.251 INFO:teuthology.orchestra.run.smithi114.stdout: "type": "bool", 2024-04-23T10:41:08.251 INFO:teuthology.orchestra.run.smithi114.stdout: "level": "advanced", 2024-04-23T10:41:08.251 INFO:teuthology.orchestra.run.smithi114.stdout: "flags": 0, 2024-04-23T10:41:08.252 INFO:teuthology.orchestra.run.smithi114.stdout: "default_value": "False", 2024-04-23T10:41:08.252 INFO:teuthology.orchestra.run.smithi114.stdout: "min": "", 2024-04-23T10:41:08.252 INFO:teuthology.orchestra.run.smithi114.stdout: "max": "", 2024-04-23T10:41:08.252 INFO:teuthology.orchestra.run.smithi114.stdout: "enum_allowed": [], 2024-04-23T10:41:08.252 INFO:teuthology.orchestra.run.smithi114.stdout: "desc": "", 2024-04-23T10:41:08.252 INFO:teuthology.orchestra.run.smithi114.stdout: "long_desc": "", 2024-04-23T10:41:08.252 INFO:teuthology.orchestra.run.smithi114.stdout: "tags": [], 2024-04-23T10:41:08.252 INFO:teuthology.orchestra.run.smithi114.stdout: "see_also": [] 2024-04-23T10:41:08.252 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-23T10:41:08.252 INFO:teuthology.orchestra.run.smithi114.stdout: "PWD_POLICY_ENABLED": { 2024-04-23T10:41:08.253 INFO:teuthology.orchestra.run.smithi114.stdout: "name": "PWD_POLICY_ENABLED", 2024-04-23T10:41:08.253 INFO:teuthology.orchestra.run.smithi114.stdout: "type": "bool", 2024-04-23T10:41:08.253 INFO:teuthology.orchestra.run.smithi114.stdout: "level": "advanced", 2024-04-23T10:41:08.253 INFO:teuthology.orchestra.run.smithi114.stdout: "flags": 0, 2024-04-23T10:41:08.253 INFO:teuthology.orchestra.run.smithi114.stdout: "default_value": "True", 2024-04-23T10:41:08.253 INFO:teuthology.orchestra.run.smithi114.stdout: "min": "", 2024-04-23T10:41:08.253 INFO:teuthology.orchestra.run.smithi114.stdout: "max": "", 2024-04-23T10:41:08.253 INFO:teuthology.orchestra.run.smithi114.stdout: "enum_allowed": [], 2024-04-23T10:41:08.253 INFO:teuthology.orchestra.run.smithi114.stdout: "desc": "", 2024-04-23T10:41:08.254 INFO:teuthology.orchestra.run.smithi114.stdout: "long_desc": "", 2024-04-23T10:41:08.254 INFO:teuthology.orchestra.run.smithi114.stdout: "tags": [], 2024-04-23T10:41:08.254 INFO:teuthology.orchestra.run.smithi114.stdout: "see_also": [] 2024-04-23T10:41:08.254 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-23T10:41:08.254 INFO:teuthology.orchestra.run.smithi114.stdout: "PWD_POLICY_EXCLUSION_LIST": { 2024-04-23T10:41:08.254 INFO:teuthology.orchestra.run.smithi114.stdout: "name": "PWD_POLICY_EXCLUSION_LIST", 2024-04-23T10:41:08.254 INFO:teuthology.orchestra.run.smithi114.stdout: "type": "str", 2024-04-23T10:41:08.254 INFO:teuthology.orchestra.run.smithi114.stdout: "level": "advanced", 2024-04-23T10:41:08.254 INFO:teuthology.orchestra.run.smithi114.stdout: "flags": 0, 2024-04-23T10:41:08.254 INFO:teuthology.orchestra.run.smithi114.stdout: "default_value": "osd,host,dashboard,pool,block,nfs,ceph,monitors,gateway,logs,crush,maps", 2024-04-23T10:41:08.255 INFO:teuthology.orchestra.run.smithi114.stdout: "min": "", 2024-04-23T10:41:08.255 INFO:teuthology.orchestra.run.smithi114.stdout: "max": "", 2024-04-23T10:41:08.255 INFO:teuthology.orchestra.run.smithi114.stdout: "enum_allowed": [], 2024-04-23T10:41:08.255 INFO:teuthology.orchestra.run.smithi114.stdout: "desc": "", 2024-04-23T10:41:08.255 INFO:teuthology.orchestra.run.smithi114.stdout: "long_desc": "", 2024-04-23T10:41:08.255 INFO:teuthology.orchestra.run.smithi114.stdout: "tags": [], 2024-04-23T10:41:08.255 INFO:teuthology.orchestra.run.smithi114.stdout: "see_also": [] 2024-04-23T10:41:08.255 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-23T10:41:08.255 INFO:teuthology.orchestra.run.smithi114.stdout: "PWD_POLICY_MIN_COMPLEXITY": { 2024-04-23T10:41:08.255 INFO:teuthology.orchestra.run.smithi114.stdout: "name": "PWD_POLICY_MIN_COMPLEXITY", 2024-04-23T10:41:08.256 INFO:teuthology.orchestra.run.smithi114.stdout: "type": "int", 2024-04-23T10:41:08.256 INFO:teuthology.orchestra.run.smithi114.stdout: "level": "advanced", 2024-04-23T10:41:08.256 INFO:teuthology.orchestra.run.smithi114.stdout: "flags": 0, 2024-04-23T10:41:08.256 INFO:teuthology.orchestra.run.smithi114.stdout: "default_value": "10", 2024-04-23T10:41:08.256 INFO:teuthology.orchestra.run.smithi114.stdout: "min": "", 2024-04-23T10:41:08.256 INFO:teuthology.orchestra.run.smithi114.stdout: "max": "", 2024-04-23T10:41:08.256 INFO:teuthology.orchestra.run.smithi114.stdout: "enum_allowed": [], 2024-04-23T10:41:08.256 INFO:teuthology.orchestra.run.smithi114.stdout: "desc": "", 2024-04-23T10:41:08.256 INFO:teuthology.orchestra.run.smithi114.stdout: "long_desc": "", 2024-04-23T10:41:08.256 INFO:teuthology.orchestra.run.smithi114.stdout: "tags": [], 2024-04-23T10:41:08.257 INFO:teuthology.orchestra.run.smithi114.stdout: "see_also": [] 2024-04-23T10:41:08.257 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-23T10:41:08.257 INFO:teuthology.orchestra.run.smithi114.stdout: "PWD_POLICY_MIN_LENGTH": { 2024-04-23T10:41:08.257 INFO:teuthology.orchestra.run.smithi114.stdout: "name": "PWD_POLICY_MIN_LENGTH", 2024-04-23T10:41:08.257 INFO:teuthology.orchestra.run.smithi114.stdout: "type": "int", 2024-04-23T10:41:08.257 INFO:teuthology.orchestra.run.smithi114.stdout: "level": "advanced", 2024-04-23T10:41:08.257 INFO:teuthology.orchestra.run.smithi114.stdout: "flags": 0, 2024-04-23T10:41:08.257 INFO:teuthology.orchestra.run.smithi114.stdout: "default_value": "8", 2024-04-23T10:41:08.257 INFO:teuthology.orchestra.run.smithi114.stdout: "min": "", 2024-04-23T10:41:08.258 INFO:teuthology.orchestra.run.smithi114.stdout: "max": "", 2024-04-23T10:41:08.258 INFO:teuthology.orchestra.run.smithi114.stdout: "enum_allowed": [], 2024-04-23T10:41:08.258 INFO:teuthology.orchestra.run.smithi114.stdout: "desc": "", 2024-04-23T10:41:08.258 INFO:teuthology.orchestra.run.smithi114.stdout: "long_desc": "", 2024-04-23T10:41:08.258 INFO:teuthology.orchestra.run.smithi114.stdout: "tags": [], 2024-04-23T10:41:08.258 INFO:teuthology.orchestra.run.smithi114.stdout: "see_also": [] 2024-04-23T10:41:08.258 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-23T10:41:08.258 INFO:teuthology.orchestra.run.smithi114.stdout: "REST_REQUESTS_TIMEOUT": { 2024-04-23T10:41:08.258 INFO:teuthology.orchestra.run.smithi114.stdout: "name": "REST_REQUESTS_TIMEOUT", 2024-04-23T10:41:08.259 INFO:teuthology.orchestra.run.smithi114.stdout: "type": "int", 2024-04-23T10:41:08.259 INFO:teuthology.orchestra.run.smithi114.stdout: "level": "advanced", 2024-04-23T10:41:08.259 INFO:teuthology.orchestra.run.smithi114.stdout: "flags": 0, 2024-04-23T10:41:08.259 INFO:teuthology.orchestra.run.smithi114.stdout: "default_value": "45", 2024-04-23T10:41:08.259 INFO:teuthology.orchestra.run.smithi114.stdout: "min": "", 2024-04-23T10:41:08.259 INFO:teuthology.orchestra.run.smithi114.stdout: "max": "", 2024-04-23T10:41:08.259 INFO:teuthology.orchestra.run.smithi114.stdout: "enum_allowed": [], 2024-04-23T10:41:08.259 INFO:teuthology.orchestra.run.smithi114.stdout: "desc": "", 2024-04-23T10:41:08.259 INFO:teuthology.orchestra.run.smithi114.stdout: "long_desc": "", 2024-04-23T10:41:08.259 INFO:teuthology.orchestra.run.smithi114.stdout: "tags": [], 2024-04-23T10:41:08.260 INFO:teuthology.orchestra.run.smithi114.stdout: "see_also": [] 2024-04-23T10:41:08.260 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-23T10:41:08.260 INFO:teuthology.orchestra.run.smithi114.stdout: "RGW_API_ACCESS_KEY": { 2024-04-23T10:41:08.260 INFO:teuthology.orchestra.run.smithi114.stdout: "name": "RGW_API_ACCESS_KEY", 2024-04-23T10:41:08.260 INFO:teuthology.orchestra.run.smithi114.stdout: "type": "str", 2024-04-23T10:41:08.260 INFO:teuthology.orchestra.run.smithi114.stdout: "level": "advanced", 2024-04-23T10:41:08.260 INFO:teuthology.orchestra.run.smithi114.stdout: "flags": 0, 2024-04-23T10:41:08.260 INFO:teuthology.orchestra.run.smithi114.stdout: "default_value": "", 2024-04-23T10:41:08.260 INFO:teuthology.orchestra.run.smithi114.stdout: "min": "", 2024-04-23T10:41:08.261 INFO:teuthology.orchestra.run.smithi114.stdout: "max": "", 2024-04-23T10:41:08.261 INFO:teuthology.orchestra.run.smithi114.stdout: "enum_allowed": [], 2024-04-23T10:41:08.261 INFO:teuthology.orchestra.run.smithi114.stdout: "desc": "", 2024-04-23T10:41:08.261 INFO:teuthology.orchestra.run.smithi114.stdout: "long_desc": "", 2024-04-23T10:41:08.261 INFO:teuthology.orchestra.run.smithi114.stdout: "tags": [], 2024-04-23T10:41:08.261 INFO:teuthology.orchestra.run.smithi114.stdout: "see_also": [] 2024-04-23T10:41:08.261 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-23T10:41:08.261 INFO:teuthology.orchestra.run.smithi114.stdout: "RGW_API_ADMIN_RESOURCE": { 2024-04-23T10:41:08.261 INFO:teuthology.orchestra.run.smithi114.stdout: "name": "RGW_API_ADMIN_RESOURCE", 2024-04-23T10:41:08.261 INFO:teuthology.orchestra.run.smithi114.stdout: "type": "str", 2024-04-23T10:41:08.262 INFO:teuthology.orchestra.run.smithi114.stdout: "level": "advanced", 2024-04-23T10:41:08.262 INFO:teuthology.orchestra.run.smithi114.stdout: "flags": 0, 2024-04-23T10:41:08.262 INFO:teuthology.orchestra.run.smithi114.stdout: "default_value": "admin", 2024-04-23T10:41:08.262 INFO:teuthology.orchestra.run.smithi114.stdout: "min": "", 2024-04-23T10:41:08.262 INFO:teuthology.orchestra.run.smithi114.stdout: "max": "", 2024-04-23T10:41:08.262 INFO:teuthology.orchestra.run.smithi114.stdout: "enum_allowed": [], 2024-04-23T10:41:08.262 INFO:teuthology.orchestra.run.smithi114.stdout: "desc": "", 2024-04-23T10:41:08.262 INFO:teuthology.orchestra.run.smithi114.stdout: "long_desc": "", 2024-04-23T10:41:08.262 INFO:teuthology.orchestra.run.smithi114.stdout: "tags": [], 2024-04-23T10:41:08.262 INFO:teuthology.orchestra.run.smithi114.stdout: "see_also": [] 2024-04-23T10:41:08.263 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-23T10:41:08.263 INFO:teuthology.orchestra.run.smithi114.stdout: "RGW_API_SECRET_KEY": { 2024-04-23T10:41:08.263 INFO:teuthology.orchestra.run.smithi114.stdout: "name": "RGW_API_SECRET_KEY", 2024-04-23T10:41:08.263 INFO:teuthology.orchestra.run.smithi114.stdout: "type": "str", 2024-04-23T10:41:08.263 INFO:teuthology.orchestra.run.smithi114.stdout: "level": "advanced", 2024-04-23T10:41:08.263 INFO:teuthology.orchestra.run.smithi114.stdout: "flags": 0, 2024-04-23T10:41:08.263 INFO:teuthology.orchestra.run.smithi114.stdout: "default_value": "", 2024-04-23T10:41:08.263 INFO:teuthology.orchestra.run.smithi114.stdout: "min": "", 2024-04-23T10:41:08.263 INFO:teuthology.orchestra.run.smithi114.stdout: "max": "", 2024-04-23T10:41:08.264 INFO:teuthology.orchestra.run.smithi114.stdout: "enum_allowed": [], 2024-04-23T10:41:08.264 INFO:teuthology.orchestra.run.smithi114.stdout: "desc": "", 2024-04-23T10:41:08.264 INFO:teuthology.orchestra.run.smithi114.stdout: "long_desc": "", 2024-04-23T10:41:08.264 INFO:teuthology.orchestra.run.smithi114.stdout: "tags": [], 2024-04-23T10:41:08.264 INFO:teuthology.orchestra.run.smithi114.stdout: "see_also": [] 2024-04-23T10:41:08.264 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-23T10:41:08.264 INFO:teuthology.orchestra.run.smithi114.stdout: "RGW_API_SSL_VERIFY": { 2024-04-23T10:41:08.264 INFO:teuthology.orchestra.run.smithi114.stdout: "name": "RGW_API_SSL_VERIFY", 2024-04-23T10:41:08.264 INFO:teuthology.orchestra.run.smithi114.stdout: "type": "bool", 2024-04-23T10:41:08.264 INFO:teuthology.orchestra.run.smithi114.stdout: "level": "advanced", 2024-04-23T10:41:08.265 INFO:teuthology.orchestra.run.smithi114.stdout: "flags": 0, 2024-04-23T10:41:08.265 INFO:teuthology.orchestra.run.smithi114.stdout: "default_value": "True", 2024-04-23T10:41:08.265 INFO:teuthology.orchestra.run.smithi114.stdout: "min": "", 2024-04-23T10:41:08.265 INFO:teuthology.orchestra.run.smithi114.stdout: "max": "", 2024-04-23T10:41:08.265 INFO:teuthology.orchestra.run.smithi114.stdout: "enum_allowed": [], 2024-04-23T10:41:08.265 INFO:teuthology.orchestra.run.smithi114.stdout: "desc": "", 2024-04-23T10:41:08.265 INFO:teuthology.orchestra.run.smithi114.stdout: "long_desc": "", 2024-04-23T10:41:08.265 INFO:teuthology.orchestra.run.smithi114.stdout: "tags": [], 2024-04-23T10:41:08.265 INFO:teuthology.orchestra.run.smithi114.stdout: "see_also": [] 2024-04-23T10:41:08.265 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-23T10:41:08.266 INFO:teuthology.orchestra.run.smithi114.stdout: "UNSAFE_TLS_v1_2": { 2024-04-23T10:41:08.266 INFO:teuthology.orchestra.run.smithi114.stdout: "name": "UNSAFE_TLS_v1_2", 2024-04-23T10:41:08.266 INFO:teuthology.orchestra.run.smithi114.stdout: "type": "bool", 2024-04-23T10:41:08.266 INFO:teuthology.orchestra.run.smithi114.stdout: "level": "advanced", 2024-04-23T10:41:08.266 INFO:teuthology.orchestra.run.smithi114.stdout: "flags": 0, 2024-04-23T10:41:08.266 INFO:teuthology.orchestra.run.smithi114.stdout: "default_value": "False", 2024-04-23T10:41:08.266 INFO:teuthology.orchestra.run.smithi114.stdout: "min": "", 2024-04-23T10:41:08.266 INFO:teuthology.orchestra.run.smithi114.stdout: "max": "", 2024-04-23T10:41:08.266 INFO:teuthology.orchestra.run.smithi114.stdout: "enum_allowed": [], 2024-04-23T10:41:08.266 INFO:teuthology.orchestra.run.smithi114.stdout: "desc": "", 2024-04-23T10:41:08.267 INFO:teuthology.orchestra.run.smithi114.stdout: "long_desc": "", 2024-04-23T10:41:08.267 INFO:teuthology.orchestra.run.smithi114.stdout: "tags": [], 2024-04-23T10:41:08.267 INFO:teuthology.orchestra.run.smithi114.stdout: "see_also": [] 2024-04-23T10:41:08.267 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-23T10:41:08.267 INFO:teuthology.orchestra.run.smithi114.stdout: "USER_PWD_EXPIRATION_SPAN": { 2024-04-23T10:41:08.267 INFO:teuthology.orchestra.run.smithi114.stdout: "name": "USER_PWD_EXPIRATION_SPAN", 2024-04-23T10:41:08.267 INFO:teuthology.orchestra.run.smithi114.stdout: "type": "int", 2024-04-23T10:41:08.267 INFO:teuthology.orchestra.run.smithi114.stdout: "level": "advanced", 2024-04-23T10:41:08.267 INFO:teuthology.orchestra.run.smithi114.stdout: "flags": 0, 2024-04-23T10:41:08.267 INFO:teuthology.orchestra.run.smithi114.stdout: "default_value": "0", 2024-04-23T10:41:08.268 INFO:teuthology.orchestra.run.smithi114.stdout: "min": "", 2024-04-23T10:41:08.268 INFO:teuthology.orchestra.run.smithi114.stdout: "max": "", 2024-04-23T10:41:08.268 INFO:teuthology.orchestra.run.smithi114.stdout: "enum_allowed": [], 2024-04-23T10:41:08.268 INFO:teuthology.orchestra.run.smithi114.stdout: "desc": "", 2024-04-23T10:41:08.268 INFO:teuthology.orchestra.run.smithi114.stdout: "long_desc": "", 2024-04-23T10:41:08.268 INFO:teuthology.orchestra.run.smithi114.stdout: "tags": [], 2024-04-23T10:41:08.268 INFO:teuthology.orchestra.run.smithi114.stdout: "see_also": [] 2024-04-23T10:41:08.268 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-23T10:41:08.268 INFO:teuthology.orchestra.run.smithi114.stdout: "USER_PWD_EXPIRATION_WARNING_1": { 2024-04-23T10:41:08.268 INFO:teuthology.orchestra.run.smithi114.stdout: "name": "USER_PWD_EXPIRATION_WARNING_1", 2024-04-23T10:41:08.269 INFO:teuthology.orchestra.run.smithi114.stdout: "type": "int", 2024-04-23T10:41:08.269 INFO:teuthology.orchestra.run.smithi114.stdout: "level": "advanced", 2024-04-23T10:41:08.269 INFO:teuthology.orchestra.run.smithi114.stdout: "flags": 0, 2024-04-23T10:41:08.269 INFO:teuthology.orchestra.run.smithi114.stdout: "default_value": "10", 2024-04-23T10:41:08.269 INFO:teuthology.orchestra.run.smithi114.stdout: "min": "", 2024-04-23T10:41:08.269 INFO:teuthology.orchestra.run.smithi114.stdout: "max": "", 2024-04-23T10:41:08.269 INFO:teuthology.orchestra.run.smithi114.stdout: "enum_allowed": [], 2024-04-23T10:41:08.269 INFO:teuthology.orchestra.run.smithi114.stdout: "desc": "", 2024-04-23T10:41:08.269 INFO:teuthology.orchestra.run.smithi114.stdout: "long_desc": "", 2024-04-23T10:41:08.269 INFO:teuthology.orchestra.run.smithi114.stdout: "tags": [], 2024-04-23T10:41:08.270 INFO:teuthology.orchestra.run.smithi114.stdout: "see_also": [] 2024-04-23T10:41:08.270 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-23T10:41:08.270 INFO:teuthology.orchestra.run.smithi114.stdout: "USER_PWD_EXPIRATION_WARNING_2": { 2024-04-23T10:41:08.270 INFO:teuthology.orchestra.run.smithi114.stdout: "name": "USER_PWD_EXPIRATION_WARNING_2", 2024-04-23T10:41:08.270 INFO:teuthology.orchestra.run.smithi114.stdout: "type": "int", 2024-04-23T10:41:08.270 INFO:teuthology.orchestra.run.smithi114.stdout: "level": "advanced", 2024-04-23T10:41:08.270 INFO:teuthology.orchestra.run.smithi114.stdout: "flags": 0, 2024-04-23T10:41:08.270 INFO:teuthology.orchestra.run.smithi114.stdout: "default_value": "5", 2024-04-23T10:41:08.270 INFO:teuthology.orchestra.run.smithi114.stdout: "min": "", 2024-04-23T10:41:08.271 INFO:teuthology.orchestra.run.smithi114.stdout: "max": "", 2024-04-23T10:41:08.271 INFO:teuthology.orchestra.run.smithi114.stdout: "enum_allowed": [], 2024-04-23T10:41:08.271 INFO:teuthology.orchestra.run.smithi114.stdout: "desc": "", 2024-04-23T10:41:08.271 INFO:teuthology.orchestra.run.smithi114.stdout: "long_desc": "", 2024-04-23T10:41:08.271 INFO:teuthology.orchestra.run.smithi114.stdout: "tags": [], 2024-04-23T10:41:08.271 INFO:teuthology.orchestra.run.smithi114.stdout: "see_also": [] 2024-04-23T10:41:08.271 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-23T10:41:08.271 INFO:teuthology.orchestra.run.smithi114.stdout: "cross_origin_url": { 2024-04-23T10:41:08.271 INFO:teuthology.orchestra.run.smithi114.stdout: "name": "cross_origin_url", 2024-04-23T10:41:08.272 INFO:teuthology.orchestra.run.smithi114.stdout: "type": "str", 2024-04-23T10:41:08.272 INFO:teuthology.orchestra.run.smithi114.stdout: "level": "advanced", 2024-04-23T10:41:08.272 INFO:teuthology.orchestra.run.smithi114.stdout: "flags": 0, 2024-04-23T10:41:08.272 INFO:teuthology.orchestra.run.smithi114.stdout: "default_value": "", 2024-04-23T10:41:08.272 INFO:teuthology.orchestra.run.smithi114.stdout: "min": "", 2024-04-23T10:41:08.272 INFO:teuthology.orchestra.run.smithi114.stdout: "max": "", 2024-04-23T10:41:08.272 INFO:teuthology.orchestra.run.smithi114.stdout: "enum_allowed": [], 2024-04-23T10:41:08.272 INFO:teuthology.orchestra.run.smithi114.stdout: "desc": "", 2024-04-23T10:41:08.272 INFO:teuthology.orchestra.run.smithi114.stdout: "long_desc": "", 2024-04-23T10:41:08.272 INFO:teuthology.orchestra.run.smithi114.stdout: "tags": [], 2024-04-23T10:41:08.273 INFO:teuthology.orchestra.run.smithi114.stdout: "see_also": [] 2024-04-23T10:41:08.273 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-23T10:41:08.273 INFO:teuthology.orchestra.run.smithi114.stdout: "crt_file": { 2024-04-23T10:41:08.273 INFO:teuthology.orchestra.run.smithi114.stdout: "name": "crt_file", 2024-04-23T10:41:08.273 INFO:teuthology.orchestra.run.smithi114.stdout: "type": "str", 2024-04-23T10:41:08.273 INFO:teuthology.orchestra.run.smithi114.stdout: "level": "advanced", 2024-04-23T10:41:08.273 INFO:teuthology.orchestra.run.smithi114.stdout: "flags": 0, 2024-04-23T10:41:08.273 INFO:teuthology.orchestra.run.smithi114.stdout: "default_value": "", 2024-04-23T10:41:08.273 INFO:teuthology.orchestra.run.smithi114.stdout: "min": "", 2024-04-23T10:41:08.273 INFO:teuthology.orchestra.run.smithi114.stdout: "max": "", 2024-04-23T10:41:08.274 INFO:teuthology.orchestra.run.smithi114.stdout: "enum_allowed": [], 2024-04-23T10:41:08.274 INFO:teuthology.orchestra.run.smithi114.stdout: "desc": "", 2024-04-23T10:41:08.274 INFO:teuthology.orchestra.run.smithi114.stdout: "long_desc": "", 2024-04-23T10:41:08.274 INFO:teuthology.orchestra.run.smithi114.stdout: "tags": [], 2024-04-23T10:41:08.274 INFO:teuthology.orchestra.run.smithi114.stdout: "see_also": [] 2024-04-23T10:41:08.274 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-23T10:41:08.274 INFO:teuthology.orchestra.run.smithi114.stdout: "debug": { 2024-04-23T10:41:08.274 INFO:teuthology.orchestra.run.smithi114.stdout: "name": "debug", 2024-04-23T10:41:08.274 INFO:teuthology.orchestra.run.smithi114.stdout: "type": "bool", 2024-04-23T10:41:08.274 INFO:teuthology.orchestra.run.smithi114.stdout: "level": "advanced", 2024-04-23T10:41:08.275 INFO:teuthology.orchestra.run.smithi114.stdout: "flags": 0, 2024-04-23T10:41:08.275 INFO:teuthology.orchestra.run.smithi114.stdout: "default_value": "False", 2024-04-23T10:41:08.275 INFO:teuthology.orchestra.run.smithi114.stdout: "min": "", 2024-04-23T10:41:08.275 INFO:teuthology.orchestra.run.smithi114.stdout: "max": "", 2024-04-23T10:41:08.275 INFO:teuthology.orchestra.run.smithi114.stdout: "enum_allowed": [], 2024-04-23T10:41:08.275 INFO:teuthology.orchestra.run.smithi114.stdout: "desc": "Enable/disable debug options", 2024-04-23T10:41:08.275 INFO:teuthology.orchestra.run.smithi114.stdout: "long_desc": "", 2024-04-23T10:41:08.275 INFO:teuthology.orchestra.run.smithi114.stdout: "tags": [], 2024-04-23T10:41:08.275 INFO:teuthology.orchestra.run.smithi114.stdout: "see_also": [] 2024-04-23T10:41:08.275 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-23T10:41:08.276 INFO:teuthology.orchestra.run.smithi114.stdout: "jwt_token_ttl": { 2024-04-23T10:41:08.276 INFO:teuthology.orchestra.run.smithi114.stdout: "name": "jwt_token_ttl", 2024-04-23T10:41:08.276 INFO:teuthology.orchestra.run.smithi114.stdout: "type": "int", 2024-04-23T10:41:08.276 INFO:teuthology.orchestra.run.smithi114.stdout: "level": "advanced", 2024-04-23T10:41:08.276 INFO:teuthology.orchestra.run.smithi114.stdout: "flags": 0, 2024-04-23T10:41:08.276 INFO:teuthology.orchestra.run.smithi114.stdout: "default_value": "28800", 2024-04-23T10:41:08.276 INFO:teuthology.orchestra.run.smithi114.stdout: "min": "", 2024-04-23T10:41:08.276 INFO:teuthology.orchestra.run.smithi114.stdout: "max": "", 2024-04-23T10:41:08.276 INFO:teuthology.orchestra.run.smithi114.stdout: "enum_allowed": [], 2024-04-23T10:41:08.276 INFO:teuthology.orchestra.run.smithi114.stdout: "desc": "", 2024-04-23T10:41:08.277 INFO:teuthology.orchestra.run.smithi114.stdout: "long_desc": "", 2024-04-23T10:41:08.277 INFO:teuthology.orchestra.run.smithi114.stdout: "tags": [], 2024-04-23T10:41:08.277 INFO:teuthology.orchestra.run.smithi114.stdout: "see_also": [] 2024-04-23T10:41:08.277 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-23T10:41:08.277 INFO:teuthology.orchestra.run.smithi114.stdout: "key_file": { 2024-04-23T10:41:08.277 INFO:teuthology.orchestra.run.smithi114.stdout: "name": "key_file", 2024-04-23T10:41:08.277 INFO:teuthology.orchestra.run.smithi114.stdout: "type": "str", 2024-04-23T10:41:08.277 INFO:teuthology.orchestra.run.smithi114.stdout: "level": "advanced", 2024-04-23T10:41:08.277 INFO:teuthology.orchestra.run.smithi114.stdout: "flags": 0, 2024-04-23T10:41:08.277 INFO:teuthology.orchestra.run.smithi114.stdout: "default_value": "", 2024-04-23T10:41:08.278 INFO:teuthology.orchestra.run.smithi114.stdout: "min": "", 2024-04-23T10:41:08.278 INFO:teuthology.orchestra.run.smithi114.stdout: "max": "", 2024-04-23T10:41:08.278 INFO:teuthology.orchestra.run.smithi114.stdout: "enum_allowed": [], 2024-04-23T10:41:08.278 INFO:teuthology.orchestra.run.smithi114.stdout: "desc": "", 2024-04-23T10:41:08.278 INFO:teuthology.orchestra.run.smithi114.stdout: "long_desc": "", 2024-04-23T10:41:08.278 INFO:teuthology.orchestra.run.smithi114.stdout: "tags": [], 2024-04-23T10:41:08.278 INFO:teuthology.orchestra.run.smithi114.stdout: "see_also": [] 2024-04-23T10:41:08.278 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-23T10:41:08.278 INFO:teuthology.orchestra.run.smithi114.stdout: "log_level": { 2024-04-23T10:41:08.279 INFO:teuthology.orchestra.run.smithi114.stdout: "name": "log_level", 2024-04-23T10:41:08.279 INFO:teuthology.orchestra.run.smithi114.stdout: "type": "str", 2024-04-23T10:41:08.279 INFO:teuthology.orchestra.run.smithi114.stdout: "level": "advanced", 2024-04-23T10:41:08.279 INFO:teuthology.orchestra.run.smithi114.stdout: "flags": 1, 2024-04-23T10:41:08.279 INFO:teuthology.orchestra.run.smithi114.stdout: "default_value": "", 2024-04-23T10:41:08.279 INFO:teuthology.orchestra.run.smithi114.stdout: "min": "", 2024-04-23T10:41:08.279 INFO:teuthology.orchestra.run.smithi114.stdout: "max": "", 2024-04-23T10:41:08.279 INFO:teuthology.orchestra.run.smithi114.stdout: "enum_allowed": [ 2024-04-23T10:41:08.279 INFO:teuthology.orchestra.run.smithi114.stdout: "", 2024-04-23T10:41:08.279 INFO:teuthology.orchestra.run.smithi114.stdout: "critical", 2024-04-23T10:41:08.280 INFO:teuthology.orchestra.run.smithi114.stdout: "debug", 2024-04-23T10:41:08.280 INFO:teuthology.orchestra.run.smithi114.stdout: "error", 2024-04-23T10:41:08.280 INFO:teuthology.orchestra.run.smithi114.stdout: "info", 2024-04-23T10:41:08.280 INFO:teuthology.orchestra.run.smithi114.stdout: "warning" 2024-04-23T10:41:08.280 INFO:teuthology.orchestra.run.smithi114.stdout: ], 2024-04-23T10:41:08.280 INFO:teuthology.orchestra.run.smithi114.stdout: "desc": "", 2024-04-23T10:41:08.280 INFO:teuthology.orchestra.run.smithi114.stdout: "long_desc": "", 2024-04-23T10:41:08.280 INFO:teuthology.orchestra.run.smithi114.stdout: "tags": [], 2024-04-23T10:41:08.280 INFO:teuthology.orchestra.run.smithi114.stdout: "see_also": [] 2024-04-23T10:41:08.280 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-23T10:41:08.281 INFO:teuthology.orchestra.run.smithi114.stdout: "log_to_cluster": { 2024-04-23T10:41:08.281 INFO:teuthology.orchestra.run.smithi114.stdout: "name": "log_to_cluster", 2024-04-23T10:41:08.281 INFO:teuthology.orchestra.run.smithi114.stdout: "type": "bool", 2024-04-23T10:41:08.281 INFO:teuthology.orchestra.run.smithi114.stdout: "level": "advanced", 2024-04-23T10:41:08.281 INFO:teuthology.orchestra.run.smithi114.stdout: "flags": 1, 2024-04-23T10:41:08.281 INFO:teuthology.orchestra.run.smithi114.stdout: "default_value": "False", 2024-04-23T10:41:08.281 INFO:teuthology.orchestra.run.smithi114.stdout: "min": "", 2024-04-23T10:41:08.281 INFO:teuthology.orchestra.run.smithi114.stdout: "max": "", 2024-04-23T10:41:08.281 INFO:teuthology.orchestra.run.smithi114.stdout: "enum_allowed": [], 2024-04-23T10:41:08.281 INFO:teuthology.orchestra.run.smithi114.stdout: "desc": "", 2024-04-23T10:41:08.282 INFO:teuthology.orchestra.run.smithi114.stdout: "long_desc": "", 2024-04-23T10:41:08.282 INFO:teuthology.orchestra.run.smithi114.stdout: "tags": [], 2024-04-23T10:41:08.282 INFO:teuthology.orchestra.run.smithi114.stdout: "see_also": [] 2024-04-23T10:41:08.282 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-23T10:41:08.282 INFO:teuthology.orchestra.run.smithi114.stdout: "log_to_cluster_level": { 2024-04-23T10:41:08.282 INFO:teuthology.orchestra.run.smithi114.stdout: "name": "log_to_cluster_level", 2024-04-23T10:41:08.282 INFO:teuthology.orchestra.run.smithi114.stdout: "type": "str", 2024-04-23T10:41:08.282 INFO:teuthology.orchestra.run.smithi114.stdout: "level": "advanced", 2024-04-23T10:41:08.282 INFO:teuthology.orchestra.run.smithi114.stdout: "flags": 1, 2024-04-23T10:41:08.283 INFO:teuthology.orchestra.run.smithi114.stdout: "default_value": "info", 2024-04-23T10:41:08.283 INFO:teuthology.orchestra.run.smithi114.stdout: "min": "", 2024-04-23T10:41:08.283 INFO:teuthology.orchestra.run.smithi114.stdout: "max": "", 2024-04-23T10:41:08.283 INFO:teuthology.orchestra.run.smithi114.stdout: "enum_allowed": [ 2024-04-23T10:41:08.283 INFO:teuthology.orchestra.run.smithi114.stdout: "", 2024-04-23T10:41:08.283 INFO:teuthology.orchestra.run.smithi114.stdout: "critical", 2024-04-23T10:41:08.283 INFO:teuthology.orchestra.run.smithi114.stdout: "debug", 2024-04-23T10:41:08.283 INFO:teuthology.orchestra.run.smithi114.stdout: "error", 2024-04-23T10:41:08.283 INFO:teuthology.orchestra.run.smithi114.stdout: "info", 2024-04-23T10:41:08.283 INFO:teuthology.orchestra.run.smithi114.stdout: "warning" 2024-04-23T10:41:08.284 INFO:teuthology.orchestra.run.smithi114.stdout: ], 2024-04-23T10:41:08.284 INFO:teuthology.orchestra.run.smithi114.stdout: "desc": "", 2024-04-23T10:41:08.284 INFO:teuthology.orchestra.run.smithi114.stdout: "long_desc": "", 2024-04-23T10:41:08.284 INFO:teuthology.orchestra.run.smithi114.stdout: "tags": [], 2024-04-23T10:41:08.284 INFO:teuthology.orchestra.run.smithi114.stdout: "see_also": [] 2024-04-23T10:41:08.284 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-23T10:41:08.284 INFO:teuthology.orchestra.run.smithi114.stdout: "log_to_file": { 2024-04-23T10:41:08.284 INFO:teuthology.orchestra.run.smithi114.stdout: "name": "log_to_file", 2024-04-23T10:41:08.284 INFO:teuthology.orchestra.run.smithi114.stdout: "type": "bool", 2024-04-23T10:41:08.284 INFO:teuthology.orchestra.run.smithi114.stdout: "level": "advanced", 2024-04-23T10:41:08.285 INFO:teuthology.orchestra.run.smithi114.stdout: "flags": 1, 2024-04-23T10:41:08.285 INFO:teuthology.orchestra.run.smithi114.stdout: "default_value": "False", 2024-04-23T10:41:08.285 INFO:teuthology.orchestra.run.smithi114.stdout: "min": "", 2024-04-23T10:41:08.285 INFO:teuthology.orchestra.run.smithi114.stdout: "max": "", 2024-04-23T10:41:08.285 INFO:teuthology.orchestra.run.smithi114.stdout: "enum_allowed": [], 2024-04-23T10:41:08.285 INFO:teuthology.orchestra.run.smithi114.stdout: "desc": "", 2024-04-23T10:41:08.285 INFO:teuthology.orchestra.run.smithi114.stdout: "long_desc": "", 2024-04-23T10:41:08.285 INFO:teuthology.orchestra.run.smithi114.stdout: "tags": [], 2024-04-23T10:41:08.285 INFO:teuthology.orchestra.run.smithi114.stdout: "see_also": [] 2024-04-23T10:41:08.285 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-23T10:41:08.286 INFO:teuthology.orchestra.run.smithi114.stdout: "motd": { 2024-04-23T10:41:08.286 INFO:teuthology.orchestra.run.smithi114.stdout: "name": "motd", 2024-04-23T10:41:08.286 INFO:teuthology.orchestra.run.smithi114.stdout: "type": "str", 2024-04-23T10:41:08.286 INFO:teuthology.orchestra.run.smithi114.stdout: "level": "advanced", 2024-04-23T10:41:08.286 INFO:teuthology.orchestra.run.smithi114.stdout: "flags": 0, 2024-04-23T10:41:08.286 INFO:teuthology.orchestra.run.smithi114.stdout: "default_value": "", 2024-04-23T10:41:08.286 INFO:teuthology.orchestra.run.smithi114.stdout: "min": "", 2024-04-23T10:41:08.286 INFO:teuthology.orchestra.run.smithi114.stdout: "max": "", 2024-04-23T10:41:08.286 INFO:teuthology.orchestra.run.smithi114.stdout: "enum_allowed": [], 2024-04-23T10:41:08.286 INFO:teuthology.orchestra.run.smithi114.stdout: "desc": "The message of the day", 2024-04-23T10:41:08.287 INFO:teuthology.orchestra.run.smithi114.stdout: "long_desc": "", 2024-04-23T10:41:08.287 INFO:teuthology.orchestra.run.smithi114.stdout: "tags": [], 2024-04-23T10:41:08.287 INFO:teuthology.orchestra.run.smithi114.stdout: "see_also": [] 2024-04-23T10:41:08.287 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-23T10:41:08.287 INFO:teuthology.orchestra.run.smithi114.stdout: "redirect_resolve_ip_addr": { 2024-04-23T10:41:08.287 INFO:teuthology.orchestra.run.smithi114.stdout: "name": "redirect_resolve_ip_addr", 2024-04-23T10:41:08.287 INFO:teuthology.orchestra.run.smithi114.stdout: "type": "bool", 2024-04-23T10:41:08.287 INFO:teuthology.orchestra.run.smithi114.stdout: "level": "advanced", 2024-04-23T10:41:08.287 INFO:teuthology.orchestra.run.smithi114.stdout: "flags": 0, 2024-04-23T10:41:08.287 INFO:teuthology.orchestra.run.smithi114.stdout: "default_value": "False", 2024-04-23T10:41:08.288 INFO:teuthology.orchestra.run.smithi114.stdout: "min": "", 2024-04-23T10:41:08.288 INFO:teuthology.orchestra.run.smithi114.stdout: "max": "", 2024-04-23T10:41:08.288 INFO:teuthology.orchestra.run.smithi114.stdout: "enum_allowed": [], 2024-04-23T10:41:08.288 INFO:teuthology.orchestra.run.smithi114.stdout: "desc": "", 2024-04-23T10:41:08.288 INFO:teuthology.orchestra.run.smithi114.stdout: "long_desc": "", 2024-04-23T10:41:08.288 INFO:teuthology.orchestra.run.smithi114.stdout: "tags": [], 2024-04-23T10:41:08.288 INFO:teuthology.orchestra.run.smithi114.stdout: "see_also": [] 2024-04-23T10:41:08.288 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-23T10:41:08.288 INFO:teuthology.orchestra.run.smithi114.stdout: "server_addr": { 2024-04-23T10:41:08.289 INFO:teuthology.orchestra.run.smithi114.stdout: "name": "server_addr", 2024-04-23T10:41:08.289 INFO:teuthology.orchestra.run.smithi114.stdout: "type": "str", 2024-04-23T10:41:08.289 INFO:teuthology.orchestra.run.smithi114.stdout: "level": "advanced", 2024-04-23T10:41:08.289 INFO:teuthology.orchestra.run.smithi114.stdout: "flags": 0, 2024-04-23T10:41:08.289 INFO:teuthology.orchestra.run.smithi114.stdout: "default_value": "::", 2024-04-23T10:41:08.289 INFO:teuthology.orchestra.run.smithi114.stdout: "min": "", 2024-04-23T10:41:08.289 INFO:teuthology.orchestra.run.smithi114.stdout: "max": "", 2024-04-23T10:41:08.289 INFO:teuthology.orchestra.run.smithi114.stdout: "enum_allowed": [], 2024-04-23T10:41:08.289 INFO:teuthology.orchestra.run.smithi114.stdout: "desc": "", 2024-04-23T10:41:08.289 INFO:teuthology.orchestra.run.smithi114.stdout: "long_desc": "", 2024-04-23T10:41:08.290 INFO:teuthology.orchestra.run.smithi114.stdout: "tags": [], 2024-04-23T10:41:08.290 INFO:teuthology.orchestra.run.smithi114.stdout: "see_also": [] 2024-04-23T10:41:08.290 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-23T10:41:08.290 INFO:teuthology.orchestra.run.smithi114.stdout: "server_port": { 2024-04-23T10:41:08.290 INFO:teuthology.orchestra.run.smithi114.stdout: "name": "server_port", 2024-04-23T10:41:08.290 INFO:teuthology.orchestra.run.smithi114.stdout: "type": "int", 2024-04-23T10:41:08.290 INFO:teuthology.orchestra.run.smithi114.stdout: "level": "advanced", 2024-04-23T10:41:08.290 INFO:teuthology.orchestra.run.smithi114.stdout: "flags": 0, 2024-04-23T10:41:08.290 INFO:teuthology.orchestra.run.smithi114.stdout: "default_value": "8080", 2024-04-23T10:41:08.291 INFO:teuthology.orchestra.run.smithi114.stdout: "min": "", 2024-04-23T10:41:08.291 INFO:teuthology.orchestra.run.smithi114.stdout: "max": "", 2024-04-23T10:41:08.291 INFO:teuthology.orchestra.run.smithi114.stdout: "enum_allowed": [], 2024-04-23T10:41:08.291 INFO:teuthology.orchestra.run.smithi114.stdout: "desc": "", 2024-04-23T10:41:08.291 INFO:teuthology.orchestra.run.smithi114.stdout: "long_desc": "", 2024-04-23T10:41:08.291 INFO:teuthology.orchestra.run.smithi114.stdout: "tags": [], 2024-04-23T10:41:08.291 INFO:teuthology.orchestra.run.smithi114.stdout: "see_also": [] 2024-04-23T10:41:08.291 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-23T10:41:08.291 INFO:teuthology.orchestra.run.smithi114.stdout: "ssl": { 2024-04-23T10:41:08.291 INFO:teuthology.orchestra.run.smithi114.stdout: "name": "ssl", 2024-04-23T10:41:08.292 INFO:teuthology.orchestra.run.smithi114.stdout: "type": "bool", 2024-04-23T10:41:08.292 INFO:teuthology.orchestra.run.smithi114.stdout: "level": "advanced", 2024-04-23T10:41:08.292 INFO:teuthology.orchestra.run.smithi114.stdout: "flags": 0, 2024-04-23T10:41:08.292 INFO:teuthology.orchestra.run.smithi114.stdout: "default_value": "True", 2024-04-23T10:41:08.292 INFO:teuthology.orchestra.run.smithi114.stdout: "min": "", 2024-04-23T10:41:08.292 INFO:teuthology.orchestra.run.smithi114.stdout: "max": "", 2024-04-23T10:41:08.292 INFO:teuthology.orchestra.run.smithi114.stdout: "enum_allowed": [], 2024-04-23T10:41:08.292 INFO:teuthology.orchestra.run.smithi114.stdout: "desc": "", 2024-04-23T10:41:08.292 INFO:teuthology.orchestra.run.smithi114.stdout: "long_desc": "", 2024-04-23T10:41:08.292 INFO:teuthology.orchestra.run.smithi114.stdout: "tags": [], 2024-04-23T10:41:08.293 INFO:teuthology.orchestra.run.smithi114.stdout: "see_also": [] 2024-04-23T10:41:08.293 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-23T10:41:08.293 INFO:teuthology.orchestra.run.smithi114.stdout: "ssl_server_port": { 2024-04-23T10:41:08.293 INFO:teuthology.orchestra.run.smithi114.stdout: "name": "ssl_server_port", 2024-04-23T10:41:08.293 INFO:teuthology.orchestra.run.smithi114.stdout: "type": "int", 2024-04-23T10:41:08.293 INFO:teuthology.orchestra.run.smithi114.stdout: "level": "advanced", 2024-04-23T10:41:08.293 INFO:teuthology.orchestra.run.smithi114.stdout: "flags": 0, 2024-04-23T10:41:08.293 INFO:teuthology.orchestra.run.smithi114.stdout: "default_value": "8443", 2024-04-23T10:41:08.294 INFO:teuthology.orchestra.run.smithi114.stdout: "min": "", 2024-04-23T10:41:08.294 INFO:teuthology.orchestra.run.smithi114.stdout: "max": "", 2024-04-23T10:41:08.294 INFO:teuthology.orchestra.run.smithi114.stdout: "enum_allowed": [], 2024-04-23T10:41:08.294 INFO:teuthology.orchestra.run.smithi114.stdout: "desc": "", 2024-04-23T10:41:08.294 INFO:teuthology.orchestra.run.smithi114.stdout: "long_desc": "", 2024-04-23T10:41:08.294 INFO:teuthology.orchestra.run.smithi114.stdout: "tags": [], 2024-04-23T10:41:08.294 INFO:teuthology.orchestra.run.smithi114.stdout: "see_also": [] 2024-04-23T10:41:08.294 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-23T10:41:08.294 INFO:teuthology.orchestra.run.smithi114.stdout: "standby_behaviour": { 2024-04-23T10:41:08.295 INFO:teuthology.orchestra.run.smithi114.stdout: "name": "standby_behaviour", 2024-04-23T10:41:08.295 INFO:teuthology.orchestra.run.smithi114.stdout: "type": "str", 2024-04-23T10:41:08.295 INFO:teuthology.orchestra.run.smithi114.stdout: "level": "advanced", 2024-04-23T10:41:08.295 INFO:teuthology.orchestra.run.smithi114.stdout: "flags": 0, 2024-04-23T10:41:08.295 INFO:teuthology.orchestra.run.smithi114.stdout: "default_value": "redirect", 2024-04-23T10:41:08.295 INFO:teuthology.orchestra.run.smithi114.stdout: "min": "", 2024-04-23T10:41:08.295 INFO:teuthology.orchestra.run.smithi114.stdout: "max": "", 2024-04-23T10:41:08.295 INFO:teuthology.orchestra.run.smithi114.stdout: "enum_allowed": [ 2024-04-23T10:41:08.296 INFO:teuthology.orchestra.run.smithi114.stdout: "error", 2024-04-23T10:41:08.296 INFO:teuthology.orchestra.run.smithi114.stdout: "redirect" 2024-04-23T10:41:08.296 INFO:teuthology.orchestra.run.smithi114.stdout: ], 2024-04-23T10:41:08.296 INFO:teuthology.orchestra.run.smithi114.stdout: "desc": "", 2024-04-23T10:41:08.296 INFO:teuthology.orchestra.run.smithi114.stdout: "long_desc": "", 2024-04-23T10:41:08.296 INFO:teuthology.orchestra.run.smithi114.stdout: "tags": [], 2024-04-23T10:41:08.296 INFO:teuthology.orchestra.run.smithi114.stdout: "see_also": [] 2024-04-23T10:41:08.296 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-23T10:41:08.296 INFO:teuthology.orchestra.run.smithi114.stdout: "standby_error_status_code": { 2024-04-23T10:41:08.297 INFO:teuthology.orchestra.run.smithi114.stdout: "name": "standby_error_status_code", 2024-04-23T10:41:08.297 INFO:teuthology.orchestra.run.smithi114.stdout: "type": "int", 2024-04-23T10:41:08.297 INFO:teuthology.orchestra.run.smithi114.stdout: "level": "advanced", 2024-04-23T10:41:08.297 INFO:teuthology.orchestra.run.smithi114.stdout: "flags": 0, 2024-04-23T10:41:08.297 INFO:teuthology.orchestra.run.smithi114.stdout: "default_value": "500", 2024-04-23T10:41:08.297 INFO:teuthology.orchestra.run.smithi114.stdout: "min": "400", 2024-04-23T10:41:08.297 INFO:teuthology.orchestra.run.smithi114.stdout: "max": "599", 2024-04-23T10:41:08.297 INFO:teuthology.orchestra.run.smithi114.stdout: "enum_allowed": [], 2024-04-23T10:41:08.297 INFO:teuthology.orchestra.run.smithi114.stdout: "desc": "", 2024-04-23T10:41:08.297 INFO:teuthology.orchestra.run.smithi114.stdout: "long_desc": "", 2024-04-23T10:41:08.298 INFO:teuthology.orchestra.run.smithi114.stdout: "tags": [], 2024-04-23T10:41:08.298 INFO:teuthology.orchestra.run.smithi114.stdout: "see_also": [] 2024-04-23T10:41:08.298 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-23T10:41:08.298 INFO:teuthology.orchestra.run.smithi114.stdout: "url_prefix": { 2024-04-23T10:41:08.298 INFO:teuthology.orchestra.run.smithi114.stdout: "name": "url_prefix", 2024-04-23T10:41:08.298 INFO:teuthology.orchestra.run.smithi114.stdout: "type": "str", 2024-04-23T10:41:08.298 INFO:teuthology.orchestra.run.smithi114.stdout: "level": "advanced", 2024-04-23T10:41:08.298 INFO:teuthology.orchestra.run.smithi114.stdout: "flags": 0, 2024-04-23T10:41:08.298 INFO:teuthology.orchestra.run.smithi114.stdout: "default_value": "", 2024-04-23T10:41:08.299 INFO:teuthology.orchestra.run.smithi114.stdout: "min": "", 2024-04-23T10:41:08.299 INFO:teuthology.orchestra.run.smithi114.stdout: "max": "", 2024-04-23T10:41:08.299 INFO:teuthology.orchestra.run.smithi114.stdout: "enum_allowed": [], 2024-04-23T10:41:08.299 INFO:teuthology.orchestra.run.smithi114.stdout: "desc": "", 2024-04-23T10:41:08.299 INFO:teuthology.orchestra.run.smithi114.stdout: "long_desc": "", 2024-04-23T10:41:08.299 INFO:teuthology.orchestra.run.smithi114.stdout: "tags": [], 2024-04-23T10:41:08.299 INFO:teuthology.orchestra.run.smithi114.stdout: "see_also": [] 2024-04-23T10:41:08.299 INFO:teuthology.orchestra.run.smithi114.stdout: } 2024-04-23T10:41:08.299 INFO:teuthology.orchestra.run.smithi114.stdout: } 2024-04-23T10:41:08.299 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-23T10:41:08.300 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-04-23T10:41:08.300 INFO:teuthology.orchestra.run.smithi114.stdout: "name": "devicehealth", 2024-04-23T10:41:08.300 INFO:teuthology.orchestra.run.smithi114.stdout: "can_run": true, 2024-04-23T10:41:08.300 INFO:teuthology.orchestra.run.smithi114.stdout: "error_string": "", 2024-04-23T10:41:08.300 INFO:teuthology.orchestra.run.smithi114.stdout: "module_options": { 2024-04-23T10:41:08.300 INFO:teuthology.orchestra.run.smithi114.stdout: "enable_monitoring": { 2024-04-23T10:41:08.300 INFO:teuthology.orchestra.run.smithi114.stdout: "name": "enable_monitoring", 2024-04-23T10:41:08.300 INFO:teuthology.orchestra.run.smithi114.stdout: "type": "bool", 2024-04-23T10:41:08.300 INFO:teuthology.orchestra.run.smithi114.stdout: "level": "advanced", 2024-04-23T10:41:08.300 INFO:teuthology.orchestra.run.smithi114.stdout: "flags": 1, 2024-04-23T10:41:08.301 INFO:teuthology.orchestra.run.smithi114.stdout: "default_value": "True", 2024-04-23T10:41:08.301 INFO:teuthology.orchestra.run.smithi114.stdout: "min": "", 2024-04-23T10:41:08.301 INFO:teuthology.orchestra.run.smithi114.stdout: "max": "", 2024-04-23T10:41:08.301 INFO:teuthology.orchestra.run.smithi114.stdout: "enum_allowed": [], 2024-04-23T10:41:08.301 INFO:teuthology.orchestra.run.smithi114.stdout: "desc": "monitor device health metrics", 2024-04-23T10:41:08.301 INFO:teuthology.orchestra.run.smithi114.stdout: "long_desc": "", 2024-04-23T10:41:08.301 INFO:teuthology.orchestra.run.smithi114.stdout: "tags": [], 2024-04-23T10:41:08.301 INFO:teuthology.orchestra.run.smithi114.stdout: "see_also": [] 2024-04-23T10:41:08.301 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-23T10:41:08.302 INFO:teuthology.orchestra.run.smithi114.stdout: "log_level": { 2024-04-23T10:41:08.302 INFO:teuthology.orchestra.run.smithi114.stdout: "name": "log_level", 2024-04-23T10:41:08.302 INFO:teuthology.orchestra.run.smithi114.stdout: "type": "str", 2024-04-23T10:41:08.302 INFO:teuthology.orchestra.run.smithi114.stdout: "level": "advanced", 2024-04-23T10:41:08.302 INFO:teuthology.orchestra.run.smithi114.stdout: "flags": 1, 2024-04-23T10:41:08.302 INFO:teuthology.orchestra.run.smithi114.stdout: "default_value": "", 2024-04-23T10:41:08.302 INFO:teuthology.orchestra.run.smithi114.stdout: "min": "", 2024-04-23T10:41:08.302 INFO:teuthology.orchestra.run.smithi114.stdout: "max": "", 2024-04-23T10:41:08.302 INFO:teuthology.orchestra.run.smithi114.stdout: "enum_allowed": [ 2024-04-23T10:41:08.303 INFO:teuthology.orchestra.run.smithi114.stdout: "", 2024-04-23T10:41:08.303 INFO:teuthology.orchestra.run.smithi114.stdout: "critical", 2024-04-23T10:41:08.303 INFO:teuthology.orchestra.run.smithi114.stdout: "debug", 2024-04-23T10:41:08.303 INFO:teuthology.orchestra.run.smithi114.stdout: "error", 2024-04-23T10:41:08.303 INFO:teuthology.orchestra.run.smithi114.stdout: "info", 2024-04-23T10:41:08.303 INFO:teuthology.orchestra.run.smithi114.stdout: "warning" 2024-04-23T10:41:08.303 INFO:teuthology.orchestra.run.smithi114.stdout: ], 2024-04-23T10:41:08.303 INFO:teuthology.orchestra.run.smithi114.stdout: "desc": "", 2024-04-23T10:41:08.303 INFO:teuthology.orchestra.run.smithi114.stdout: "long_desc": "", 2024-04-23T10:41:08.303 INFO:teuthology.orchestra.run.smithi114.stdout: "tags": [], 2024-04-23T10:41:08.304 INFO:teuthology.orchestra.run.smithi114.stdout: "see_also": [] 2024-04-23T10:41:08.304 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-23T10:41:08.304 INFO:teuthology.orchestra.run.smithi114.stdout: "log_to_cluster": { 2024-04-23T10:41:08.304 INFO:teuthology.orchestra.run.smithi114.stdout: "name": "log_to_cluster", 2024-04-23T10:41:08.304 INFO:teuthology.orchestra.run.smithi114.stdout: "type": "bool", 2024-04-23T10:41:08.304 INFO:teuthology.orchestra.run.smithi114.stdout: "level": "advanced", 2024-04-23T10:41:08.304 INFO:teuthology.orchestra.run.smithi114.stdout: "flags": 1, 2024-04-23T10:41:08.304 INFO:teuthology.orchestra.run.smithi114.stdout: "default_value": "False", 2024-04-23T10:41:08.304 INFO:teuthology.orchestra.run.smithi114.stdout: "min": "", 2024-04-23T10:41:08.304 INFO:teuthology.orchestra.run.smithi114.stdout: "max": "", 2024-04-23T10:41:08.305 INFO:teuthology.orchestra.run.smithi114.stdout: "enum_allowed": [], 2024-04-23T10:41:08.305 INFO:teuthology.orchestra.run.smithi114.stdout: "desc": "", 2024-04-23T10:41:08.305 INFO:teuthology.orchestra.run.smithi114.stdout: "long_desc": "", 2024-04-23T10:41:08.305 INFO:teuthology.orchestra.run.smithi114.stdout: "tags": [], 2024-04-23T10:41:08.305 INFO:teuthology.orchestra.run.smithi114.stdout: "see_also": [] 2024-04-23T10:41:08.305 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-23T10:41:08.305 INFO:teuthology.orchestra.run.smithi114.stdout: "log_to_cluster_level": { 2024-04-23T10:41:08.305 INFO:teuthology.orchestra.run.smithi114.stdout: "name": "log_to_cluster_level", 2024-04-23T10:41:08.305 INFO:teuthology.orchestra.run.smithi114.stdout: "type": "str", 2024-04-23T10:41:08.305 INFO:teuthology.orchestra.run.smithi114.stdout: "level": "advanced", 2024-04-23T10:41:08.306 INFO:teuthology.orchestra.run.smithi114.stdout: "flags": 1, 2024-04-23T10:41:08.306 INFO:teuthology.orchestra.run.smithi114.stdout: "default_value": "info", 2024-04-23T10:41:08.306 INFO:teuthology.orchestra.run.smithi114.stdout: "min": "", 2024-04-23T10:41:08.306 INFO:teuthology.orchestra.run.smithi114.stdout: "max": "", 2024-04-23T10:41:08.306 INFO:teuthology.orchestra.run.smithi114.stdout: "enum_allowed": [ 2024-04-23T10:41:08.306 INFO:teuthology.orchestra.run.smithi114.stdout: "", 2024-04-23T10:41:08.306 INFO:teuthology.orchestra.run.smithi114.stdout: "critical", 2024-04-23T10:41:08.306 INFO:teuthology.orchestra.run.smithi114.stdout: "debug", 2024-04-23T10:41:08.306 INFO:teuthology.orchestra.run.smithi114.stdout: "error", 2024-04-23T10:41:08.307 INFO:teuthology.orchestra.run.smithi114.stdout: "info", 2024-04-23T10:41:08.307 INFO:teuthology.orchestra.run.smithi114.stdout: "warning" 2024-04-23T10:41:08.307 INFO:teuthology.orchestra.run.smithi114.stdout: ], 2024-04-23T10:41:08.307 INFO:teuthology.orchestra.run.smithi114.stdout: "desc": "", 2024-04-23T10:41:08.307 INFO:teuthology.orchestra.run.smithi114.stdout: "long_desc": "", 2024-04-23T10:41:08.307 INFO:teuthology.orchestra.run.smithi114.stdout: "tags": [], 2024-04-23T10:41:08.307 INFO:teuthology.orchestra.run.smithi114.stdout: "see_also": [] 2024-04-23T10:41:08.307 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-23T10:41:08.307 INFO:teuthology.orchestra.run.smithi114.stdout: "log_to_file": { 2024-04-23T10:41:08.307 INFO:teuthology.orchestra.run.smithi114.stdout: "name": "log_to_file", 2024-04-23T10:41:08.308 INFO:teuthology.orchestra.run.smithi114.stdout: "type": "bool", 2024-04-23T10:41:08.308 INFO:teuthology.orchestra.run.smithi114.stdout: "level": "advanced", 2024-04-23T10:41:08.308 INFO:teuthology.orchestra.run.smithi114.stdout: "flags": 1, 2024-04-23T10:41:08.308 INFO:teuthology.orchestra.run.smithi114.stdout: "default_value": "False", 2024-04-23T10:41:08.308 INFO:teuthology.orchestra.run.smithi114.stdout: "min": "", 2024-04-23T10:41:08.308 INFO:teuthology.orchestra.run.smithi114.stdout: "max": "", 2024-04-23T10:41:08.308 INFO:teuthology.orchestra.run.smithi114.stdout: "enum_allowed": [], 2024-04-23T10:41:08.308 INFO:teuthology.orchestra.run.smithi114.stdout: "desc": "", 2024-04-23T10:41:08.308 INFO:teuthology.orchestra.run.smithi114.stdout: "long_desc": "", 2024-04-23T10:41:08.308 INFO:teuthology.orchestra.run.smithi114.stdout: "tags": [], 2024-04-23T10:41:08.309 INFO:teuthology.orchestra.run.smithi114.stdout: "see_also": [] 2024-04-23T10:41:08.309 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-23T10:41:08.309 INFO:teuthology.orchestra.run.smithi114.stdout: "mark_out_threshold": { 2024-04-23T10:41:08.309 INFO:teuthology.orchestra.run.smithi114.stdout: "name": "mark_out_threshold", 2024-04-23T10:41:08.309 INFO:teuthology.orchestra.run.smithi114.stdout: "type": "secs", 2024-04-23T10:41:08.309 INFO:teuthology.orchestra.run.smithi114.stdout: "level": "advanced", 2024-04-23T10:41:08.309 INFO:teuthology.orchestra.run.smithi114.stdout: "flags": 1, 2024-04-23T10:41:08.309 INFO:teuthology.orchestra.run.smithi114.stdout: "default_value": "2419200", 2024-04-23T10:41:08.309 INFO:teuthology.orchestra.run.smithi114.stdout: "min": "", 2024-04-23T10:41:08.309 INFO:teuthology.orchestra.run.smithi114.stdout: "max": "", 2024-04-23T10:41:08.310 INFO:teuthology.orchestra.run.smithi114.stdout: "enum_allowed": [], 2024-04-23T10:41:08.310 INFO:teuthology.orchestra.run.smithi114.stdout: "desc": "automatically mark OSD if it may fail before this long", 2024-04-23T10:41:08.310 INFO:teuthology.orchestra.run.smithi114.stdout: "long_desc": "", 2024-04-23T10:41:08.310 INFO:teuthology.orchestra.run.smithi114.stdout: "tags": [], 2024-04-23T10:41:08.310 INFO:teuthology.orchestra.run.smithi114.stdout: "see_also": [] 2024-04-23T10:41:08.310 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-23T10:41:08.310 INFO:teuthology.orchestra.run.smithi114.stdout: "pool_name": { 2024-04-23T10:41:08.310 INFO:teuthology.orchestra.run.smithi114.stdout: "name": "pool_name", 2024-04-23T10:41:08.310 INFO:teuthology.orchestra.run.smithi114.stdout: "type": "str", 2024-04-23T10:41:08.311 INFO:teuthology.orchestra.run.smithi114.stdout: "level": "advanced", 2024-04-23T10:41:08.311 INFO:teuthology.orchestra.run.smithi114.stdout: "flags": 1, 2024-04-23T10:41:08.311 INFO:teuthology.orchestra.run.smithi114.stdout: "default_value": "device_health_metrics", 2024-04-23T10:41:08.311 INFO:teuthology.orchestra.run.smithi114.stdout: "min": "", 2024-04-23T10:41:08.311 INFO:teuthology.orchestra.run.smithi114.stdout: "max": "", 2024-04-23T10:41:08.311 INFO:teuthology.orchestra.run.smithi114.stdout: "enum_allowed": [], 2024-04-23T10:41:08.311 INFO:teuthology.orchestra.run.smithi114.stdout: "desc": "name of pool in which to store device health metrics", 2024-04-23T10:41:08.311 INFO:teuthology.orchestra.run.smithi114.stdout: "long_desc": "", 2024-04-23T10:41:08.311 INFO:teuthology.orchestra.run.smithi114.stdout: "tags": [], 2024-04-23T10:41:08.311 INFO:teuthology.orchestra.run.smithi114.stdout: "see_also": [] 2024-04-23T10:41:08.312 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-23T10:41:08.312 INFO:teuthology.orchestra.run.smithi114.stdout: "retention_period": { 2024-04-23T10:41:08.312 INFO:teuthology.orchestra.run.smithi114.stdout: "name": "retention_period", 2024-04-23T10:41:08.312 INFO:teuthology.orchestra.run.smithi114.stdout: "type": "secs", 2024-04-23T10:41:08.312 INFO:teuthology.orchestra.run.smithi114.stdout: "level": "advanced", 2024-04-23T10:41:08.312 INFO:teuthology.orchestra.run.smithi114.stdout: "flags": 1, 2024-04-23T10:41:08.312 INFO:teuthology.orchestra.run.smithi114.stdout: "default_value": "15552000", 2024-04-23T10:41:08.312 INFO:teuthology.orchestra.run.smithi114.stdout: "min": "", 2024-04-23T10:41:08.312 INFO:teuthology.orchestra.run.smithi114.stdout: "max": "", 2024-04-23T10:41:08.312 INFO:teuthology.orchestra.run.smithi114.stdout: "enum_allowed": [], 2024-04-23T10:41:08.313 INFO:teuthology.orchestra.run.smithi114.stdout: "desc": "how long to retain device health metrics", 2024-04-23T10:41:08.313 INFO:teuthology.orchestra.run.smithi114.stdout: "long_desc": "", 2024-04-23T10:41:08.313 INFO:teuthology.orchestra.run.smithi114.stdout: "tags": [], 2024-04-23T10:41:08.313 INFO:teuthology.orchestra.run.smithi114.stdout: "see_also": [] 2024-04-23T10:41:08.313 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-23T10:41:08.313 INFO:teuthology.orchestra.run.smithi114.stdout: "scrape_frequency": { 2024-04-23T10:41:08.313 INFO:teuthology.orchestra.run.smithi114.stdout: "name": "scrape_frequency", 2024-04-23T10:41:08.313 INFO:teuthology.orchestra.run.smithi114.stdout: "type": "secs", 2024-04-23T10:41:08.313 INFO:teuthology.orchestra.run.smithi114.stdout: "level": "advanced", 2024-04-23T10:41:08.313 INFO:teuthology.orchestra.run.smithi114.stdout: "flags": 1, 2024-04-23T10:41:08.314 INFO:teuthology.orchestra.run.smithi114.stdout: "default_value": "86400", 2024-04-23T10:41:08.314 INFO:teuthology.orchestra.run.smithi114.stdout: "min": "", 2024-04-23T10:41:08.314 INFO:teuthology.orchestra.run.smithi114.stdout: "max": "", 2024-04-23T10:41:08.314 INFO:teuthology.orchestra.run.smithi114.stdout: "enum_allowed": [], 2024-04-23T10:41:08.314 INFO:teuthology.orchestra.run.smithi114.stdout: "desc": "how frequently to scrape device health metrics", 2024-04-23T10:41:08.314 INFO:teuthology.orchestra.run.smithi114.stdout: "long_desc": "", 2024-04-23T10:41:08.314 INFO:teuthology.orchestra.run.smithi114.stdout: "tags": [], 2024-04-23T10:41:08.314 INFO:teuthology.orchestra.run.smithi114.stdout: "see_also": [] 2024-04-23T10:41:08.314 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-23T10:41:08.315 INFO:teuthology.orchestra.run.smithi114.stdout: "self_heal": { 2024-04-23T10:41:08.315 INFO:teuthology.orchestra.run.smithi114.stdout: "name": "self_heal", 2024-04-23T10:41:08.315 INFO:teuthology.orchestra.run.smithi114.stdout: "type": "bool", 2024-04-23T10:41:08.315 INFO:teuthology.orchestra.run.smithi114.stdout: "level": "advanced", 2024-04-23T10:41:08.315 INFO:teuthology.orchestra.run.smithi114.stdout: "flags": 1, 2024-04-23T10:41:08.315 INFO:teuthology.orchestra.run.smithi114.stdout: "default_value": "True", 2024-04-23T10:41:08.315 INFO:teuthology.orchestra.run.smithi114.stdout: "min": "", 2024-04-23T10:41:08.315 INFO:teuthology.orchestra.run.smithi114.stdout: "max": "", 2024-04-23T10:41:08.315 INFO:teuthology.orchestra.run.smithi114.stdout: "enum_allowed": [], 2024-04-23T10:41:08.315 INFO:teuthology.orchestra.run.smithi114.stdout: "desc": "preemptively heal cluster around devices that may fail", 2024-04-23T10:41:08.316 INFO:teuthology.orchestra.run.smithi114.stdout: "long_desc": "", 2024-04-23T10:41:08.316 INFO:teuthology.orchestra.run.smithi114.stdout: "tags": [], 2024-04-23T10:41:08.316 INFO:teuthology.orchestra.run.smithi114.stdout: "see_also": [] 2024-04-23T10:41:08.316 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-23T10:41:08.316 INFO:teuthology.orchestra.run.smithi114.stdout: "sleep_interval": { 2024-04-23T10:41:08.316 INFO:teuthology.orchestra.run.smithi114.stdout: "name": "sleep_interval", 2024-04-23T10:41:08.316 INFO:teuthology.orchestra.run.smithi114.stdout: "type": "secs", 2024-04-23T10:41:08.316 INFO:teuthology.orchestra.run.smithi114.stdout: "level": "advanced", 2024-04-23T10:41:08.316 INFO:teuthology.orchestra.run.smithi114.stdout: "flags": 1, 2024-04-23T10:41:08.316 INFO:teuthology.orchestra.run.smithi114.stdout: "default_value": "600", 2024-04-23T10:41:08.317 INFO:teuthology.orchestra.run.smithi114.stdout: "min": "", 2024-04-23T10:41:08.317 INFO:teuthology.orchestra.run.smithi114.stdout: "max": "", 2024-04-23T10:41:08.317 INFO:teuthology.orchestra.run.smithi114.stdout: "enum_allowed": [], 2024-04-23T10:41:08.317 INFO:teuthology.orchestra.run.smithi114.stdout: "desc": "how frequently to wake up and check device health", 2024-04-23T10:41:08.317 INFO:teuthology.orchestra.run.smithi114.stdout: "long_desc": "", 2024-04-23T10:41:08.317 INFO:teuthology.orchestra.run.smithi114.stdout: "tags": [], 2024-04-23T10:41:08.317 INFO:teuthology.orchestra.run.smithi114.stdout: "see_also": [] 2024-04-23T10:41:08.317 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-23T10:41:08.317 INFO:teuthology.orchestra.run.smithi114.stdout: "warn_threshold": { 2024-04-23T10:41:08.317 INFO:teuthology.orchestra.run.smithi114.stdout: "name": "warn_threshold", 2024-04-23T10:41:08.318 INFO:teuthology.orchestra.run.smithi114.stdout: "type": "secs", 2024-04-23T10:41:08.318 INFO:teuthology.orchestra.run.smithi114.stdout: "level": "advanced", 2024-04-23T10:41:08.318 INFO:teuthology.orchestra.run.smithi114.stdout: "flags": 1, 2024-04-23T10:41:08.318 INFO:teuthology.orchestra.run.smithi114.stdout: "default_value": "7257600", 2024-04-23T10:41:08.318 INFO:teuthology.orchestra.run.smithi114.stdout: "min": "", 2024-04-23T10:41:08.318 INFO:teuthology.orchestra.run.smithi114.stdout: "max": "", 2024-04-23T10:41:08.318 INFO:teuthology.orchestra.run.smithi114.stdout: "enum_allowed": [], 2024-04-23T10:41:08.318 INFO:teuthology.orchestra.run.smithi114.stdout: "desc": "raise health warning if OSD may fail before this long", 2024-04-23T10:41:08.318 INFO:teuthology.orchestra.run.smithi114.stdout: "long_desc": "", 2024-04-23T10:41:08.319 INFO:teuthology.orchestra.run.smithi114.stdout: "tags": [], 2024-04-23T10:41:08.319 INFO:teuthology.orchestra.run.smithi114.stdout: "see_also": [] 2024-04-23T10:41:08.319 INFO:teuthology.orchestra.run.smithi114.stdout: } 2024-04-23T10:41:08.319 INFO:teuthology.orchestra.run.smithi114.stdout: } 2024-04-23T10:41:08.319 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-23T10:41:08.319 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-04-23T10:41:08.319 INFO:teuthology.orchestra.run.smithi114.stdout: "name": "diskprediction_local", 2024-04-23T10:41:08.319 INFO:teuthology.orchestra.run.smithi114.stdout: "can_run": true, 2024-04-23T10:41:08.319 INFO:teuthology.orchestra.run.smithi114.stdout: "error_string": "", 2024-04-23T10:41:08.319 INFO:teuthology.orchestra.run.smithi114.stdout: "module_options": { 2024-04-23T10:41:08.320 INFO:teuthology.orchestra.run.smithi114.stdout: "log_level": { 2024-04-23T10:41:08.320 INFO:teuthology.orchestra.run.smithi114.stdout: "name": "log_level", 2024-04-23T10:41:08.320 INFO:teuthology.orchestra.run.smithi114.stdout: "type": "str", 2024-04-23T10:41:08.320 INFO:teuthology.orchestra.run.smithi114.stdout: "level": "advanced", 2024-04-23T10:41:08.320 INFO:teuthology.orchestra.run.smithi114.stdout: "flags": 1, 2024-04-23T10:41:08.320 INFO:teuthology.orchestra.run.smithi114.stdout: "default_value": "", 2024-04-23T10:41:08.320 INFO:teuthology.orchestra.run.smithi114.stdout: "min": "", 2024-04-23T10:41:08.320 INFO:teuthology.orchestra.run.smithi114.stdout: "max": "", 2024-04-23T10:41:08.320 INFO:teuthology.orchestra.run.smithi114.stdout: "enum_allowed": [ 2024-04-23T10:41:08.320 INFO:teuthology.orchestra.run.smithi114.stdout: "", 2024-04-23T10:41:08.321 INFO:teuthology.orchestra.run.smithi114.stdout: "critical", 2024-04-23T10:41:08.321 INFO:teuthology.orchestra.run.smithi114.stdout: "debug", 2024-04-23T10:41:08.321 INFO:teuthology.orchestra.run.smithi114.stdout: "error", 2024-04-23T10:41:08.321 INFO:teuthology.orchestra.run.smithi114.stdout: "info", 2024-04-23T10:41:08.321 INFO:teuthology.orchestra.run.smithi114.stdout: "warning" 2024-04-23T10:41:08.321 INFO:teuthology.orchestra.run.smithi114.stdout: ], 2024-04-23T10:41:08.321 INFO:teuthology.orchestra.run.smithi114.stdout: "desc": "", 2024-04-23T10:41:08.321 INFO:teuthology.orchestra.run.smithi114.stdout: "long_desc": "", 2024-04-23T10:41:08.321 INFO:teuthology.orchestra.run.smithi114.stdout: "tags": [], 2024-04-23T10:41:08.321 INFO:teuthology.orchestra.run.smithi114.stdout: "see_also": [] 2024-04-23T10:41:08.322 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-23T10:41:08.322 INFO:teuthology.orchestra.run.smithi114.stdout: "log_to_cluster": { 2024-04-23T10:41:08.322 INFO:teuthology.orchestra.run.smithi114.stdout: "name": "log_to_cluster", 2024-04-23T10:41:08.322 INFO:teuthology.orchestra.run.smithi114.stdout: "type": "bool", 2024-04-23T10:41:08.322 INFO:teuthology.orchestra.run.smithi114.stdout: "level": "advanced", 2024-04-23T10:41:08.322 INFO:teuthology.orchestra.run.smithi114.stdout: "flags": 1, 2024-04-23T10:41:08.322 INFO:teuthology.orchestra.run.smithi114.stdout: "default_value": "False", 2024-04-23T10:41:08.322 INFO:teuthology.orchestra.run.smithi114.stdout: "min": "", 2024-04-23T10:41:08.322 INFO:teuthology.orchestra.run.smithi114.stdout: "max": "", 2024-04-23T10:41:08.322 INFO:teuthology.orchestra.run.smithi114.stdout: "enum_allowed": [], 2024-04-23T10:41:08.323 INFO:teuthology.orchestra.run.smithi114.stdout: "desc": "", 2024-04-23T10:41:08.323 INFO:teuthology.orchestra.run.smithi114.stdout: "long_desc": "", 2024-04-23T10:41:08.323 INFO:teuthology.orchestra.run.smithi114.stdout: "tags": [], 2024-04-23T10:41:08.323 INFO:teuthology.orchestra.run.smithi114.stdout: "see_also": [] 2024-04-23T10:41:08.323 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-23T10:41:08.323 INFO:teuthology.orchestra.run.smithi114.stdout: "log_to_cluster_level": { 2024-04-23T10:41:08.323 INFO:teuthology.orchestra.run.smithi114.stdout: "name": "log_to_cluster_level", 2024-04-23T10:41:08.323 INFO:teuthology.orchestra.run.smithi114.stdout: "type": "str", 2024-04-23T10:41:08.323 INFO:teuthology.orchestra.run.smithi114.stdout: "level": "advanced", 2024-04-23T10:41:08.323 INFO:teuthology.orchestra.run.smithi114.stdout: "flags": 1, 2024-04-23T10:41:08.324 INFO:teuthology.orchestra.run.smithi114.stdout: "default_value": "info", 2024-04-23T10:41:08.324 INFO:teuthology.orchestra.run.smithi114.stdout: "min": "", 2024-04-23T10:41:08.324 INFO:teuthology.orchestra.run.smithi114.stdout: "max": "", 2024-04-23T10:41:08.324 INFO:teuthology.orchestra.run.smithi114.stdout: "enum_allowed": [ 2024-04-23T10:41:08.324 INFO:teuthology.orchestra.run.smithi114.stdout: "", 2024-04-23T10:41:08.324 INFO:teuthology.orchestra.run.smithi114.stdout: "critical", 2024-04-23T10:41:08.324 INFO:teuthology.orchestra.run.smithi114.stdout: "debug", 2024-04-23T10:41:08.324 INFO:teuthology.orchestra.run.smithi114.stdout: "error", 2024-04-23T10:41:08.324 INFO:teuthology.orchestra.run.smithi114.stdout: "info", 2024-04-23T10:41:08.324 INFO:teuthology.orchestra.run.smithi114.stdout: "warning" 2024-04-23T10:41:08.325 INFO:teuthology.orchestra.run.smithi114.stdout: ], 2024-04-23T10:41:08.325 INFO:teuthology.orchestra.run.smithi114.stdout: "desc": "", 2024-04-23T10:41:08.325 INFO:teuthology.orchestra.run.smithi114.stdout: "long_desc": "", 2024-04-23T10:41:08.325 INFO:teuthology.orchestra.run.smithi114.stdout: "tags": [], 2024-04-23T10:41:08.325 INFO:teuthology.orchestra.run.smithi114.stdout: "see_also": [] 2024-04-23T10:41:08.325 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-23T10:41:08.325 INFO:teuthology.orchestra.run.smithi114.stdout: "log_to_file": { 2024-04-23T10:41:08.325 INFO:teuthology.orchestra.run.smithi114.stdout: "name": "log_to_file", 2024-04-23T10:41:08.325 INFO:teuthology.orchestra.run.smithi114.stdout: "type": "bool", 2024-04-23T10:41:08.326 INFO:teuthology.orchestra.run.smithi114.stdout: "level": "advanced", 2024-04-23T10:41:08.326 INFO:teuthology.orchestra.run.smithi114.stdout: "flags": 1, 2024-04-23T10:41:08.326 INFO:teuthology.orchestra.run.smithi114.stdout: "default_value": "False", 2024-04-23T10:41:08.326 INFO:teuthology.orchestra.run.smithi114.stdout: "min": "", 2024-04-23T10:41:08.326 INFO:teuthology.orchestra.run.smithi114.stdout: "max": "", 2024-04-23T10:41:08.326 INFO:teuthology.orchestra.run.smithi114.stdout: "enum_allowed": [], 2024-04-23T10:41:08.326 INFO:teuthology.orchestra.run.smithi114.stdout: "desc": "", 2024-04-23T10:41:08.326 INFO:teuthology.orchestra.run.smithi114.stdout: "long_desc": "", 2024-04-23T10:41:08.326 INFO:teuthology.orchestra.run.smithi114.stdout: "tags": [], 2024-04-23T10:41:08.326 INFO:teuthology.orchestra.run.smithi114.stdout: "see_also": [] 2024-04-23T10:41:08.327 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-23T10:41:08.327 INFO:teuthology.orchestra.run.smithi114.stdout: "predict_interval": { 2024-04-23T10:41:08.327 INFO:teuthology.orchestra.run.smithi114.stdout: "name": "predict_interval", 2024-04-23T10:41:08.327 INFO:teuthology.orchestra.run.smithi114.stdout: "type": "str", 2024-04-23T10:41:08.327 INFO:teuthology.orchestra.run.smithi114.stdout: "level": "advanced", 2024-04-23T10:41:08.327 INFO:teuthology.orchestra.run.smithi114.stdout: "flags": 0, 2024-04-23T10:41:08.327 INFO:teuthology.orchestra.run.smithi114.stdout: "default_value": "86400", 2024-04-23T10:41:08.327 INFO:teuthology.orchestra.run.smithi114.stdout: "min": "", 2024-04-23T10:41:08.327 INFO:teuthology.orchestra.run.smithi114.stdout: "max": "", 2024-04-23T10:41:08.327 INFO:teuthology.orchestra.run.smithi114.stdout: "enum_allowed": [], 2024-04-23T10:41:08.328 INFO:teuthology.orchestra.run.smithi114.stdout: "desc": "", 2024-04-23T10:41:08.328 INFO:teuthology.orchestra.run.smithi114.stdout: "long_desc": "", 2024-04-23T10:41:08.328 INFO:teuthology.orchestra.run.smithi114.stdout: "tags": [], 2024-04-23T10:41:08.328 INFO:teuthology.orchestra.run.smithi114.stdout: "see_also": [] 2024-04-23T10:41:08.328 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-23T10:41:08.328 INFO:teuthology.orchestra.run.smithi114.stdout: "predictor_model": { 2024-04-23T10:41:08.328 INFO:teuthology.orchestra.run.smithi114.stdout: "name": "predictor_model", 2024-04-23T10:41:08.328 INFO:teuthology.orchestra.run.smithi114.stdout: "type": "str", 2024-04-23T10:41:08.328 INFO:teuthology.orchestra.run.smithi114.stdout: "level": "advanced", 2024-04-23T10:41:08.328 INFO:teuthology.orchestra.run.smithi114.stdout: "flags": 0, 2024-04-23T10:41:08.329 INFO:teuthology.orchestra.run.smithi114.stdout: "default_value": "prophetstor", 2024-04-23T10:41:08.329 INFO:teuthology.orchestra.run.smithi114.stdout: "min": "", 2024-04-23T10:41:08.329 INFO:teuthology.orchestra.run.smithi114.stdout: "max": "", 2024-04-23T10:41:08.329 INFO:teuthology.orchestra.run.smithi114.stdout: "enum_allowed": [], 2024-04-23T10:41:08.329 INFO:teuthology.orchestra.run.smithi114.stdout: "desc": "", 2024-04-23T10:41:08.329 INFO:teuthology.orchestra.run.smithi114.stdout: "long_desc": "", 2024-04-23T10:41:08.329 INFO:teuthology.orchestra.run.smithi114.stdout: "tags": [], 2024-04-23T10:41:08.329 INFO:teuthology.orchestra.run.smithi114.stdout: "see_also": [] 2024-04-23T10:41:08.329 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-23T10:41:08.330 INFO:teuthology.orchestra.run.smithi114.stdout: "sleep_interval": { 2024-04-23T10:41:08.330 INFO:teuthology.orchestra.run.smithi114.stdout: "name": "sleep_interval", 2024-04-23T10:41:08.330 INFO:teuthology.orchestra.run.smithi114.stdout: "type": "str", 2024-04-23T10:41:08.330 INFO:teuthology.orchestra.run.smithi114.stdout: "level": "advanced", 2024-04-23T10:41:08.330 INFO:teuthology.orchestra.run.smithi114.stdout: "flags": 0, 2024-04-23T10:41:08.330 INFO:teuthology.orchestra.run.smithi114.stdout: "default_value": "600", 2024-04-23T10:41:08.330 INFO:teuthology.orchestra.run.smithi114.stdout: "min": "", 2024-04-23T10:41:08.330 INFO:teuthology.orchestra.run.smithi114.stdout: "max": "", 2024-04-23T10:41:08.330 INFO:teuthology.orchestra.run.smithi114.stdout: "enum_allowed": [], 2024-04-23T10:41:08.330 INFO:teuthology.orchestra.run.smithi114.stdout: "desc": "", 2024-04-23T10:41:08.331 INFO:teuthology.orchestra.run.smithi114.stdout: "long_desc": "", 2024-04-23T10:41:08.331 INFO:teuthology.orchestra.run.smithi114.stdout: "tags": [], 2024-04-23T10:41:08.331 INFO:teuthology.orchestra.run.smithi114.stdout: "see_also": [] 2024-04-23T10:41:08.331 INFO:teuthology.orchestra.run.smithi114.stdout: } 2024-04-23T10:41:08.331 INFO:teuthology.orchestra.run.smithi114.stdout: } 2024-04-23T10:41:08.331 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-23T10:41:08.331 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-04-23T10:41:08.331 INFO:teuthology.orchestra.run.smithi114.stdout: "name": "influx", 2024-04-23T10:41:08.331 INFO:teuthology.orchestra.run.smithi114.stdout: "can_run": false, 2024-04-23T10:41:08.331 INFO:teuthology.orchestra.run.smithi114.stdout: "error_string": "influxdb python module not found", 2024-04-23T10:41:08.332 INFO:teuthology.orchestra.run.smithi114.stdout: "module_options": { 2024-04-23T10:41:08.332 INFO:teuthology.orchestra.run.smithi114.stdout: "batch_size": { 2024-04-23T10:41:08.332 INFO:teuthology.orchestra.run.smithi114.stdout: "name": "batch_size", 2024-04-23T10:41:08.332 INFO:teuthology.orchestra.run.smithi114.stdout: "type": "int", 2024-04-23T10:41:08.332 INFO:teuthology.orchestra.run.smithi114.stdout: "level": "advanced", 2024-04-23T10:41:08.332 INFO:teuthology.orchestra.run.smithi114.stdout: "flags": 0, 2024-04-23T10:41:08.332 INFO:teuthology.orchestra.run.smithi114.stdout: "default_value": "5000", 2024-04-23T10:41:08.332 INFO:teuthology.orchestra.run.smithi114.stdout: "min": "", 2024-04-23T10:41:08.332 INFO:teuthology.orchestra.run.smithi114.stdout: "max": "", 2024-04-23T10:41:08.332 INFO:teuthology.orchestra.run.smithi114.stdout: "enum_allowed": [], 2024-04-23T10:41:08.333 INFO:teuthology.orchestra.run.smithi114.stdout: "desc": "How big batches of data points should be when sending to InfluxDB.", 2024-04-23T10:41:08.333 INFO:teuthology.orchestra.run.smithi114.stdout: "long_desc": "", 2024-04-23T10:41:08.333 INFO:teuthology.orchestra.run.smithi114.stdout: "tags": [], 2024-04-23T10:41:08.333 INFO:teuthology.orchestra.run.smithi114.stdout: "see_also": [] 2024-04-23T10:41:08.333 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-23T10:41:08.333 INFO:teuthology.orchestra.run.smithi114.stdout: "database": { 2024-04-23T10:41:08.333 INFO:teuthology.orchestra.run.smithi114.stdout: "name": "database", 2024-04-23T10:41:08.333 INFO:teuthology.orchestra.run.smithi114.stdout: "type": "str", 2024-04-23T10:41:08.333 INFO:teuthology.orchestra.run.smithi114.stdout: "level": "advanced", 2024-04-23T10:41:08.333 INFO:teuthology.orchestra.run.smithi114.stdout: "flags": 0, 2024-04-23T10:41:08.334 INFO:teuthology.orchestra.run.smithi114.stdout: "default_value": "ceph", 2024-04-23T10:41:08.334 INFO:teuthology.orchestra.run.smithi114.stdout: "min": "", 2024-04-23T10:41:08.334 INFO:teuthology.orchestra.run.smithi114.stdout: "max": "", 2024-04-23T10:41:08.334 INFO:teuthology.orchestra.run.smithi114.stdout: "enum_allowed": [], 2024-04-23T10:41:08.334 INFO:teuthology.orchestra.run.smithi114.stdout: "desc": "InfluxDB database name. You will need to create this database and grant write privileges to the configured username or the username must have admin privileges to create it.", 2024-04-23T10:41:08.334 INFO:teuthology.orchestra.run.smithi114.stdout: "long_desc": "", 2024-04-23T10:41:08.334 INFO:teuthology.orchestra.run.smithi114.stdout: "tags": [], 2024-04-23T10:41:08.334 INFO:teuthology.orchestra.run.smithi114.stdout: "see_also": [] 2024-04-23T10:41:08.334 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-23T10:41:08.335 INFO:teuthology.orchestra.run.smithi114.stdout: "hostname": { 2024-04-23T10:41:08.335 INFO:teuthology.orchestra.run.smithi114.stdout: "name": "hostname", 2024-04-23T10:41:08.335 INFO:teuthology.orchestra.run.smithi114.stdout: "type": "str", 2024-04-23T10:41:08.335 INFO:teuthology.orchestra.run.smithi114.stdout: "level": "advanced", 2024-04-23T10:41:08.335 INFO:teuthology.orchestra.run.smithi114.stdout: "flags": 0, 2024-04-23T10:41:08.335 INFO:teuthology.orchestra.run.smithi114.stdout: "default_value": "", 2024-04-23T10:41:08.335 INFO:teuthology.orchestra.run.smithi114.stdout: "min": "", 2024-04-23T10:41:08.335 INFO:teuthology.orchestra.run.smithi114.stdout: "max": "", 2024-04-23T10:41:08.335 INFO:teuthology.orchestra.run.smithi114.stdout: "enum_allowed": [], 2024-04-23T10:41:08.335 INFO:teuthology.orchestra.run.smithi114.stdout: "desc": "InfluxDB server hostname", 2024-04-23T10:41:08.336 INFO:teuthology.orchestra.run.smithi114.stdout: "long_desc": "", 2024-04-23T10:41:08.336 INFO:teuthology.orchestra.run.smithi114.stdout: "tags": [], 2024-04-23T10:41:08.336 INFO:teuthology.orchestra.run.smithi114.stdout: "see_also": [] 2024-04-23T10:41:08.336 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-23T10:41:08.336 INFO:teuthology.orchestra.run.smithi114.stdout: "interval": { 2024-04-23T10:41:08.336 INFO:teuthology.orchestra.run.smithi114.stdout: "name": "interval", 2024-04-23T10:41:08.336 INFO:teuthology.orchestra.run.smithi114.stdout: "type": "secs", 2024-04-23T10:41:08.336 INFO:teuthology.orchestra.run.smithi114.stdout: "level": "advanced", 2024-04-23T10:41:08.336 INFO:teuthology.orchestra.run.smithi114.stdout: "flags": 0, 2024-04-23T10:41:08.336 INFO:teuthology.orchestra.run.smithi114.stdout: "default_value": "30", 2024-04-23T10:41:08.337 INFO:teuthology.orchestra.run.smithi114.stdout: "min": "5", 2024-04-23T10:41:08.337 INFO:teuthology.orchestra.run.smithi114.stdout: "max": "", 2024-04-23T10:41:08.337 INFO:teuthology.orchestra.run.smithi114.stdout: "enum_allowed": [], 2024-04-23T10:41:08.337 INFO:teuthology.orchestra.run.smithi114.stdout: "desc": "Time between reports to InfluxDB. Default 30 seconds.", 2024-04-23T10:41:08.337 INFO:teuthology.orchestra.run.smithi114.stdout: "long_desc": "", 2024-04-23T10:41:08.337 INFO:teuthology.orchestra.run.smithi114.stdout: "tags": [], 2024-04-23T10:41:08.337 INFO:teuthology.orchestra.run.smithi114.stdout: "see_also": [] 2024-04-23T10:41:08.337 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-23T10:41:08.337 INFO:teuthology.orchestra.run.smithi114.stdout: "log_level": { 2024-04-23T10:41:08.337 INFO:teuthology.orchestra.run.smithi114.stdout: "name": "log_level", 2024-04-23T10:41:08.338 INFO:teuthology.orchestra.run.smithi114.stdout: "type": "str", 2024-04-23T10:41:08.338 INFO:teuthology.orchestra.run.smithi114.stdout: "level": "advanced", 2024-04-23T10:41:08.338 INFO:teuthology.orchestra.run.smithi114.stdout: "flags": 1, 2024-04-23T10:41:08.338 INFO:teuthology.orchestra.run.smithi114.stdout: "default_value": "", 2024-04-23T10:41:08.338 INFO:teuthology.orchestra.run.smithi114.stdout: "min": "", 2024-04-23T10:41:08.338 INFO:teuthology.orchestra.run.smithi114.stdout: "max": "", 2024-04-23T10:41:08.338 INFO:teuthology.orchestra.run.smithi114.stdout: "enum_allowed": [ 2024-04-23T10:41:08.338 INFO:teuthology.orchestra.run.smithi114.stdout: "", 2024-04-23T10:41:08.338 INFO:teuthology.orchestra.run.smithi114.stdout: "critical", 2024-04-23T10:41:08.338 INFO:teuthology.orchestra.run.smithi114.stdout: "debug", 2024-04-23T10:41:08.339 INFO:teuthology.orchestra.run.smithi114.stdout: "error", 2024-04-23T10:41:08.339 INFO:teuthology.orchestra.run.smithi114.stdout: "info", 2024-04-23T10:41:08.339 INFO:teuthology.orchestra.run.smithi114.stdout: "warning" 2024-04-23T10:41:08.339 INFO:teuthology.orchestra.run.smithi114.stdout: ], 2024-04-23T10:41:08.339 INFO:teuthology.orchestra.run.smithi114.stdout: "desc": "", 2024-04-23T10:41:08.339 INFO:teuthology.orchestra.run.smithi114.stdout: "long_desc": "", 2024-04-23T10:41:08.339 INFO:teuthology.orchestra.run.smithi114.stdout: "tags": [], 2024-04-23T10:41:08.339 INFO:teuthology.orchestra.run.smithi114.stdout: "see_also": [] 2024-04-23T10:41:08.339 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-23T10:41:08.340 INFO:teuthology.orchestra.run.smithi114.stdout: "log_to_cluster": { 2024-04-23T10:41:08.340 INFO:teuthology.orchestra.run.smithi114.stdout: "name": "log_to_cluster", 2024-04-23T10:41:08.340 INFO:teuthology.orchestra.run.smithi114.stdout: "type": "bool", 2024-04-23T10:41:08.340 INFO:teuthology.orchestra.run.smithi114.stdout: "level": "advanced", 2024-04-23T10:41:08.340 INFO:teuthology.orchestra.run.smithi114.stdout: "flags": 1, 2024-04-23T10:41:08.340 INFO:teuthology.orchestra.run.smithi114.stdout: "default_value": "False", 2024-04-23T10:41:08.340 INFO:teuthology.orchestra.run.smithi114.stdout: "min": "", 2024-04-23T10:41:08.340 INFO:teuthology.orchestra.run.smithi114.stdout: "max": "", 2024-04-23T10:41:08.340 INFO:teuthology.orchestra.run.smithi114.stdout: "enum_allowed": [], 2024-04-23T10:41:08.340 INFO:teuthology.orchestra.run.smithi114.stdout: "desc": "", 2024-04-23T10:41:08.341 INFO:teuthology.orchestra.run.smithi114.stdout: "long_desc": "", 2024-04-23T10:41:08.341 INFO:teuthology.orchestra.run.smithi114.stdout: "tags": [], 2024-04-23T10:41:08.341 INFO:teuthology.orchestra.run.smithi114.stdout: "see_also": [] 2024-04-23T10:41:08.341 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-23T10:41:08.341 INFO:teuthology.orchestra.run.smithi114.stdout: "log_to_cluster_level": { 2024-04-23T10:41:08.341 INFO:teuthology.orchestra.run.smithi114.stdout: "name": "log_to_cluster_level", 2024-04-23T10:41:08.341 INFO:teuthology.orchestra.run.smithi114.stdout: "type": "str", 2024-04-23T10:41:08.341 INFO:teuthology.orchestra.run.smithi114.stdout: "level": "advanced", 2024-04-23T10:41:08.341 INFO:teuthology.orchestra.run.smithi114.stdout: "flags": 1, 2024-04-23T10:41:08.341 INFO:teuthology.orchestra.run.smithi114.stdout: "default_value": "info", 2024-04-23T10:41:08.342 INFO:teuthology.orchestra.run.smithi114.stdout: "min": "", 2024-04-23T10:41:08.342 INFO:teuthology.orchestra.run.smithi114.stdout: "max": "", 2024-04-23T10:41:08.342 INFO:teuthology.orchestra.run.smithi114.stdout: "enum_allowed": [ 2024-04-23T10:41:08.342 INFO:teuthology.orchestra.run.smithi114.stdout: "", 2024-04-23T10:41:08.342 INFO:teuthology.orchestra.run.smithi114.stdout: "critical", 2024-04-23T10:41:08.342 INFO:teuthology.orchestra.run.smithi114.stdout: "debug", 2024-04-23T10:41:08.342 INFO:teuthology.orchestra.run.smithi114.stdout: "error", 2024-04-23T10:41:08.342 INFO:teuthology.orchestra.run.smithi114.stdout: "info", 2024-04-23T10:41:08.342 INFO:teuthology.orchestra.run.smithi114.stdout: "warning" 2024-04-23T10:41:08.343 INFO:teuthology.orchestra.run.smithi114.stdout: ], 2024-04-23T10:41:08.343 INFO:teuthology.orchestra.run.smithi114.stdout: "desc": "", 2024-04-23T10:41:08.343 INFO:teuthology.orchestra.run.smithi114.stdout: "long_desc": "", 2024-04-23T10:41:08.343 INFO:teuthology.orchestra.run.smithi114.stdout: "tags": [], 2024-04-23T10:41:08.343 INFO:teuthology.orchestra.run.smithi114.stdout: "see_also": [] 2024-04-23T10:41:08.343 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-23T10:41:08.343 INFO:teuthology.orchestra.run.smithi114.stdout: "log_to_file": { 2024-04-23T10:41:08.343 INFO:teuthology.orchestra.run.smithi114.stdout: "name": "log_to_file", 2024-04-23T10:41:08.343 INFO:teuthology.orchestra.run.smithi114.stdout: "type": "bool", 2024-04-23T10:41:08.344 INFO:teuthology.orchestra.run.smithi114.stdout: "level": "advanced", 2024-04-23T10:41:08.344 INFO:teuthology.orchestra.run.smithi114.stdout: "flags": 1, 2024-04-23T10:41:08.344 INFO:teuthology.orchestra.run.smithi114.stdout: "default_value": "False", 2024-04-23T10:41:08.344 INFO:teuthology.orchestra.run.smithi114.stdout: "min": "", 2024-04-23T10:41:08.344 INFO:teuthology.orchestra.run.smithi114.stdout: "max": "", 2024-04-23T10:41:08.344 INFO:teuthology.orchestra.run.smithi114.stdout: "enum_allowed": [], 2024-04-23T10:41:08.344 INFO:teuthology.orchestra.run.smithi114.stdout: "desc": "", 2024-04-23T10:41:08.344 INFO:teuthology.orchestra.run.smithi114.stdout: "long_desc": "", 2024-04-23T10:41:08.344 INFO:teuthology.orchestra.run.smithi114.stdout: "tags": [], 2024-04-23T10:41:08.344 INFO:teuthology.orchestra.run.smithi114.stdout: "see_also": [] 2024-04-23T10:41:08.345 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-23T10:41:08.345 INFO:teuthology.orchestra.run.smithi114.stdout: "password": { 2024-04-23T10:41:08.345 INFO:teuthology.orchestra.run.smithi114.stdout: "name": "password", 2024-04-23T10:41:08.345 INFO:teuthology.orchestra.run.smithi114.stdout: "type": "str", 2024-04-23T10:41:08.345 INFO:teuthology.orchestra.run.smithi114.stdout: "level": "advanced", 2024-04-23T10:41:08.345 INFO:teuthology.orchestra.run.smithi114.stdout: "flags": 0, 2024-04-23T10:41:08.345 INFO:teuthology.orchestra.run.smithi114.stdout: "default_value": "", 2024-04-23T10:41:08.345 INFO:teuthology.orchestra.run.smithi114.stdout: "min": "", 2024-04-23T10:41:08.345 INFO:teuthology.orchestra.run.smithi114.stdout: "max": "", 2024-04-23T10:41:08.346 INFO:teuthology.orchestra.run.smithi114.stdout: "enum_allowed": [], 2024-04-23T10:41:08.346 INFO:teuthology.orchestra.run.smithi114.stdout: "desc": "password of InfluxDB server user", 2024-04-23T10:41:08.346 INFO:teuthology.orchestra.run.smithi114.stdout: "long_desc": "", 2024-04-23T10:41:08.346 INFO:teuthology.orchestra.run.smithi114.stdout: "tags": [], 2024-04-23T10:41:08.346 INFO:teuthology.orchestra.run.smithi114.stdout: "see_also": [] 2024-04-23T10:41:08.346 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-23T10:41:08.346 INFO:teuthology.orchestra.run.smithi114.stdout: "port": { 2024-04-23T10:41:08.346 INFO:teuthology.orchestra.run.smithi114.stdout: "name": "port", 2024-04-23T10:41:08.346 INFO:teuthology.orchestra.run.smithi114.stdout: "type": "int", 2024-04-23T10:41:08.346 INFO:teuthology.orchestra.run.smithi114.stdout: "level": "advanced", 2024-04-23T10:41:08.347 INFO:teuthology.orchestra.run.smithi114.stdout: "flags": 0, 2024-04-23T10:41:08.347 INFO:teuthology.orchestra.run.smithi114.stdout: "default_value": "8086", 2024-04-23T10:41:08.347 INFO:teuthology.orchestra.run.smithi114.stdout: "min": "", 2024-04-23T10:41:08.347 INFO:teuthology.orchestra.run.smithi114.stdout: "max": "", 2024-04-23T10:41:08.347 INFO:teuthology.orchestra.run.smithi114.stdout: "enum_allowed": [], 2024-04-23T10:41:08.347 INFO:teuthology.orchestra.run.smithi114.stdout: "desc": "InfluxDB server port", 2024-04-23T10:41:08.347 INFO:teuthology.orchestra.run.smithi114.stdout: "long_desc": "", 2024-04-23T10:41:08.347 INFO:teuthology.orchestra.run.smithi114.stdout: "tags": [], 2024-04-23T10:41:08.347 INFO:teuthology.orchestra.run.smithi114.stdout: "see_also": [] 2024-04-23T10:41:08.347 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-23T10:41:08.348 INFO:teuthology.orchestra.run.smithi114.stdout: "ssl": { 2024-04-23T10:41:08.348 INFO:teuthology.orchestra.run.smithi114.stdout: "name": "ssl", 2024-04-23T10:41:08.348 INFO:teuthology.orchestra.run.smithi114.stdout: "type": "str", 2024-04-23T10:41:08.348 INFO:teuthology.orchestra.run.smithi114.stdout: "level": "advanced", 2024-04-23T10:41:08.348 INFO:teuthology.orchestra.run.smithi114.stdout: "flags": 0, 2024-04-23T10:41:08.348 INFO:teuthology.orchestra.run.smithi114.stdout: "default_value": "false", 2024-04-23T10:41:08.348 INFO:teuthology.orchestra.run.smithi114.stdout: "min": "", 2024-04-23T10:41:08.348 INFO:teuthology.orchestra.run.smithi114.stdout: "max": "", 2024-04-23T10:41:08.348 INFO:teuthology.orchestra.run.smithi114.stdout: "enum_allowed": [], 2024-04-23T10:41:08.349 INFO:teuthology.orchestra.run.smithi114.stdout: "desc": "Use https connection for InfluxDB server. Use \"true\" or \"false\".", 2024-04-23T10:41:08.349 INFO:teuthology.orchestra.run.smithi114.stdout: "long_desc": "", 2024-04-23T10:41:08.349 INFO:teuthology.orchestra.run.smithi114.stdout: "tags": [], 2024-04-23T10:41:08.349 INFO:teuthology.orchestra.run.smithi114.stdout: "see_also": [] 2024-04-23T10:41:08.349 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-23T10:41:08.349 INFO:teuthology.orchestra.run.smithi114.stdout: "threads": { 2024-04-23T10:41:08.349 INFO:teuthology.orchestra.run.smithi114.stdout: "name": "threads", 2024-04-23T10:41:08.349 INFO:teuthology.orchestra.run.smithi114.stdout: "type": "int", 2024-04-23T10:41:08.349 INFO:teuthology.orchestra.run.smithi114.stdout: "level": "advanced", 2024-04-23T10:41:08.349 INFO:teuthology.orchestra.run.smithi114.stdout: "flags": 0, 2024-04-23T10:41:08.350 INFO:teuthology.orchestra.run.smithi114.stdout: "default_value": "5", 2024-04-23T10:41:08.350 INFO:teuthology.orchestra.run.smithi114.stdout: "min": "1", 2024-04-23T10:41:08.350 INFO:teuthology.orchestra.run.smithi114.stdout: "max": "32", 2024-04-23T10:41:08.350 INFO:teuthology.orchestra.run.smithi114.stdout: "enum_allowed": [], 2024-04-23T10:41:08.350 INFO:teuthology.orchestra.run.smithi114.stdout: "desc": "How many worker threads should be spawned for sending data to InfluxDB.", 2024-04-23T10:41:08.350 INFO:teuthology.orchestra.run.smithi114.stdout: "long_desc": "", 2024-04-23T10:41:08.350 INFO:teuthology.orchestra.run.smithi114.stdout: "tags": [], 2024-04-23T10:41:08.350 INFO:teuthology.orchestra.run.smithi114.stdout: "see_also": [] 2024-04-23T10:41:08.350 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-23T10:41:08.351 INFO:teuthology.orchestra.run.smithi114.stdout: "username": { 2024-04-23T10:41:08.351 INFO:teuthology.orchestra.run.smithi114.stdout: "name": "username", 2024-04-23T10:41:08.351 INFO:teuthology.orchestra.run.smithi114.stdout: "type": "str", 2024-04-23T10:41:08.351 INFO:teuthology.orchestra.run.smithi114.stdout: "level": "advanced", 2024-04-23T10:41:08.351 INFO:teuthology.orchestra.run.smithi114.stdout: "flags": 0, 2024-04-23T10:41:08.351 INFO:teuthology.orchestra.run.smithi114.stdout: "default_value": "", 2024-04-23T10:41:08.351 INFO:teuthology.orchestra.run.smithi114.stdout: "min": "", 2024-04-23T10:41:08.351 INFO:teuthology.orchestra.run.smithi114.stdout: "max": "", 2024-04-23T10:41:08.351 INFO:teuthology.orchestra.run.smithi114.stdout: "enum_allowed": [], 2024-04-23T10:41:08.351 INFO:teuthology.orchestra.run.smithi114.stdout: "desc": "username of InfluxDB server user", 2024-04-23T10:41:08.352 INFO:teuthology.orchestra.run.smithi114.stdout: "long_desc": "", 2024-04-23T10:41:08.352 INFO:teuthology.orchestra.run.smithi114.stdout: "tags": [], 2024-04-23T10:41:08.352 INFO:teuthology.orchestra.run.smithi114.stdout: "see_also": [] 2024-04-23T10:41:08.352 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-23T10:41:08.352 INFO:teuthology.orchestra.run.smithi114.stdout: "verify_ssl": { 2024-04-23T10:41:08.352 INFO:teuthology.orchestra.run.smithi114.stdout: "name": "verify_ssl", 2024-04-23T10:41:08.352 INFO:teuthology.orchestra.run.smithi114.stdout: "type": "str", 2024-04-23T10:41:08.352 INFO:teuthology.orchestra.run.smithi114.stdout: "level": "advanced", 2024-04-23T10:41:08.352 INFO:teuthology.orchestra.run.smithi114.stdout: "flags": 0, 2024-04-23T10:41:08.352 INFO:teuthology.orchestra.run.smithi114.stdout: "default_value": "true", 2024-04-23T10:41:08.353 INFO:teuthology.orchestra.run.smithi114.stdout: "min": "", 2024-04-23T10:41:08.353 INFO:teuthology.orchestra.run.smithi114.stdout: "max": "", 2024-04-23T10:41:08.353 INFO:teuthology.orchestra.run.smithi114.stdout: "enum_allowed": [], 2024-04-23T10:41:08.353 INFO:teuthology.orchestra.run.smithi114.stdout: "desc": "Verify https cert for InfluxDB server. Use \"true\" or \"false\".", 2024-04-23T10:41:08.353 INFO:teuthology.orchestra.run.smithi114.stdout: "long_desc": "", 2024-04-23T10:41:08.353 INFO:teuthology.orchestra.run.smithi114.stdout: "tags": [], 2024-04-23T10:41:08.353 INFO:teuthology.orchestra.run.smithi114.stdout: "see_also": [] 2024-04-23T10:41:08.353 INFO:teuthology.orchestra.run.smithi114.stdout: } 2024-04-23T10:41:08.353 INFO:teuthology.orchestra.run.smithi114.stdout: } 2024-04-23T10:41:08.354 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-23T10:41:08.354 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-04-23T10:41:08.354 INFO:teuthology.orchestra.run.smithi114.stdout: "name": "insights", 2024-04-23T10:41:08.354 INFO:teuthology.orchestra.run.smithi114.stdout: "can_run": true, 2024-04-23T10:41:08.354 INFO:teuthology.orchestra.run.smithi114.stdout: "error_string": "", 2024-04-23T10:41:08.354 INFO:teuthology.orchestra.run.smithi114.stdout: "module_options": { 2024-04-23T10:41:08.354 INFO:teuthology.orchestra.run.smithi114.stdout: "log_level": { 2024-04-23T10:41:08.354 INFO:teuthology.orchestra.run.smithi114.stdout: "name": "log_level", 2024-04-23T10:41:08.354 INFO:teuthology.orchestra.run.smithi114.stdout: "type": "str", 2024-04-23T10:41:08.354 INFO:teuthology.orchestra.run.smithi114.stdout: "level": "advanced", 2024-04-23T10:41:08.355 INFO:teuthology.orchestra.run.smithi114.stdout: "flags": 1, 2024-04-23T10:41:08.355 INFO:teuthology.orchestra.run.smithi114.stdout: "default_value": "", 2024-04-23T10:41:08.355 INFO:teuthology.orchestra.run.smithi114.stdout: "min": "", 2024-04-23T10:41:08.355 INFO:teuthology.orchestra.run.smithi114.stdout: "max": "", 2024-04-23T10:41:08.355 INFO:teuthology.orchestra.run.smithi114.stdout: "enum_allowed": [ 2024-04-23T10:41:08.355 INFO:teuthology.orchestra.run.smithi114.stdout: "", 2024-04-23T10:41:08.355 INFO:teuthology.orchestra.run.smithi114.stdout: "critical", 2024-04-23T10:41:08.355 INFO:teuthology.orchestra.run.smithi114.stdout: "debug", 2024-04-23T10:41:08.355 INFO:teuthology.orchestra.run.smithi114.stdout: "error", 2024-04-23T10:41:08.355 INFO:teuthology.orchestra.run.smithi114.stdout: "info", 2024-04-23T10:41:08.356 INFO:teuthology.orchestra.run.smithi114.stdout: "warning" 2024-04-23T10:41:08.356 INFO:teuthology.orchestra.run.smithi114.stdout: ], 2024-04-23T10:41:08.356 INFO:teuthology.orchestra.run.smithi114.stdout: "desc": "", 2024-04-23T10:41:08.356 INFO:teuthology.orchestra.run.smithi114.stdout: "long_desc": "", 2024-04-23T10:41:08.356 INFO:teuthology.orchestra.run.smithi114.stdout: "tags": [], 2024-04-23T10:41:08.356 INFO:teuthology.orchestra.run.smithi114.stdout: "see_also": [] 2024-04-23T10:41:08.356 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-23T10:41:08.356 INFO:teuthology.orchestra.run.smithi114.stdout: "log_to_cluster": { 2024-04-23T10:41:08.356 INFO:teuthology.orchestra.run.smithi114.stdout: "name": "log_to_cluster", 2024-04-23T10:41:08.356 INFO:teuthology.orchestra.run.smithi114.stdout: "type": "bool", 2024-04-23T10:41:08.357 INFO:teuthology.orchestra.run.smithi114.stdout: "level": "advanced", 2024-04-23T10:41:08.357 INFO:teuthology.orchestra.run.smithi114.stdout: "flags": 1, 2024-04-23T10:41:08.357 INFO:teuthology.orchestra.run.smithi114.stdout: "default_value": "False", 2024-04-23T10:41:08.357 INFO:teuthology.orchestra.run.smithi114.stdout: "min": "", 2024-04-23T10:41:08.357 INFO:teuthology.orchestra.run.smithi114.stdout: "max": "", 2024-04-23T10:41:08.357 INFO:teuthology.orchestra.run.smithi114.stdout: "enum_allowed": [], 2024-04-23T10:41:08.357 INFO:teuthology.orchestra.run.smithi114.stdout: "desc": "", 2024-04-23T10:41:08.357 INFO:teuthology.orchestra.run.smithi114.stdout: "long_desc": "", 2024-04-23T10:41:08.357 INFO:teuthology.orchestra.run.smithi114.stdout: "tags": [], 2024-04-23T10:41:08.357 INFO:teuthology.orchestra.run.smithi114.stdout: "see_also": [] 2024-04-23T10:41:08.358 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-23T10:41:08.358 INFO:teuthology.orchestra.run.smithi114.stdout: "log_to_cluster_level": { 2024-04-23T10:41:08.358 INFO:teuthology.orchestra.run.smithi114.stdout: "name": "log_to_cluster_level", 2024-04-23T10:41:08.358 INFO:teuthology.orchestra.run.smithi114.stdout: "type": "str", 2024-04-23T10:41:08.358 INFO:teuthology.orchestra.run.smithi114.stdout: "level": "advanced", 2024-04-23T10:41:08.358 INFO:teuthology.orchestra.run.smithi114.stdout: "flags": 1, 2024-04-23T10:41:08.358 INFO:teuthology.orchestra.run.smithi114.stdout: "default_value": "info", 2024-04-23T10:41:08.358 INFO:teuthology.orchestra.run.smithi114.stdout: "min": "", 2024-04-23T10:41:08.358 INFO:teuthology.orchestra.run.smithi114.stdout: "max": "", 2024-04-23T10:41:08.359 INFO:teuthology.orchestra.run.smithi114.stdout: "enum_allowed": [ 2024-04-23T10:41:08.359 INFO:teuthology.orchestra.run.smithi114.stdout: "", 2024-04-23T10:41:08.359 INFO:teuthology.orchestra.run.smithi114.stdout: "critical", 2024-04-23T10:41:08.359 INFO:teuthology.orchestra.run.smithi114.stdout: "debug", 2024-04-23T10:41:08.359 INFO:teuthology.orchestra.run.smithi114.stdout: "error", 2024-04-23T10:41:08.359 INFO:teuthology.orchestra.run.smithi114.stdout: "info", 2024-04-23T10:41:08.359 INFO:teuthology.orchestra.run.smithi114.stdout: "warning" 2024-04-23T10:41:08.359 INFO:teuthology.orchestra.run.smithi114.stdout: ], 2024-04-23T10:41:08.359 INFO:teuthology.orchestra.run.smithi114.stdout: "desc": "", 2024-04-23T10:41:08.359 INFO:teuthology.orchestra.run.smithi114.stdout: "long_desc": "", 2024-04-23T10:41:08.360 INFO:teuthology.orchestra.run.smithi114.stdout: "tags": [], 2024-04-23T10:41:08.360 INFO:teuthology.orchestra.run.smithi114.stdout: "see_also": [] 2024-04-23T10:41:08.360 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-23T10:41:08.360 INFO:teuthology.orchestra.run.smithi114.stdout: "log_to_file": { 2024-04-23T10:41:08.360 INFO:teuthology.orchestra.run.smithi114.stdout: "name": "log_to_file", 2024-04-23T10:41:08.360 INFO:teuthology.orchestra.run.smithi114.stdout: "type": "bool", 2024-04-23T10:41:08.360 INFO:teuthology.orchestra.run.smithi114.stdout: "level": "advanced", 2024-04-23T10:41:08.360 INFO:teuthology.orchestra.run.smithi114.stdout: "flags": 1, 2024-04-23T10:41:08.360 INFO:teuthology.orchestra.run.smithi114.stdout: "default_value": "False", 2024-04-23T10:41:08.361 INFO:teuthology.orchestra.run.smithi114.stdout: "min": "", 2024-04-23T10:41:08.361 INFO:teuthology.orchestra.run.smithi114.stdout: "max": "", 2024-04-23T10:41:08.361 INFO:teuthology.orchestra.run.smithi114.stdout: "enum_allowed": [], 2024-04-23T10:41:08.361 INFO:teuthology.orchestra.run.smithi114.stdout: "desc": "", 2024-04-23T10:41:08.361 INFO:teuthology.orchestra.run.smithi114.stdout: "long_desc": "", 2024-04-23T10:41:08.361 INFO:teuthology.orchestra.run.smithi114.stdout: "tags": [], 2024-04-23T10:41:08.361 INFO:teuthology.orchestra.run.smithi114.stdout: "see_also": [] 2024-04-23T10:41:08.361 INFO:teuthology.orchestra.run.smithi114.stdout: } 2024-04-23T10:41:08.361 INFO:teuthology.orchestra.run.smithi114.stdout: } 2024-04-23T10:41:08.361 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-23T10:41:08.362 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-04-23T10:41:08.362 INFO:teuthology.orchestra.run.smithi114.stdout: "name": "iostat", 2024-04-23T10:41:08.362 INFO:teuthology.orchestra.run.smithi114.stdout: "can_run": true, 2024-04-23T10:41:08.362 INFO:teuthology.orchestra.run.smithi114.stdout: "error_string": "", 2024-04-23T10:41:08.362 INFO:teuthology.orchestra.run.smithi114.stdout: "module_options": { 2024-04-23T10:41:08.362 INFO:teuthology.orchestra.run.smithi114.stdout: "log_level": { 2024-04-23T10:41:08.362 INFO:teuthology.orchestra.run.smithi114.stdout: "name": "log_level", 2024-04-23T10:41:08.362 INFO:teuthology.orchestra.run.smithi114.stdout: "type": "str", 2024-04-23T10:41:08.362 INFO:teuthology.orchestra.run.smithi114.stdout: "level": "advanced", 2024-04-23T10:41:08.362 INFO:teuthology.orchestra.run.smithi114.stdout: "flags": 1, 2024-04-23T10:41:08.363 INFO:teuthology.orchestra.run.smithi114.stdout: "default_value": "", 2024-04-23T10:41:08.363 INFO:teuthology.orchestra.run.smithi114.stdout: "min": "", 2024-04-23T10:41:08.363 INFO:teuthology.orchestra.run.smithi114.stdout: "max": "", 2024-04-23T10:41:08.363 INFO:teuthology.orchestra.run.smithi114.stdout: "enum_allowed": [ 2024-04-23T10:41:08.363 INFO:teuthology.orchestra.run.smithi114.stdout: "", 2024-04-23T10:41:08.363 INFO:teuthology.orchestra.run.smithi114.stdout: "critical", 2024-04-23T10:41:08.363 INFO:teuthology.orchestra.run.smithi114.stdout: "debug", 2024-04-23T10:41:08.363 INFO:teuthology.orchestra.run.smithi114.stdout: "error", 2024-04-23T10:41:08.363 INFO:teuthology.orchestra.run.smithi114.stdout: "info", 2024-04-23T10:41:08.363 INFO:teuthology.orchestra.run.smithi114.stdout: "warning" 2024-04-23T10:41:08.364 INFO:teuthology.orchestra.run.smithi114.stdout: ], 2024-04-23T10:41:08.364 INFO:teuthology.orchestra.run.smithi114.stdout: "desc": "", 2024-04-23T10:41:08.364 INFO:teuthology.orchestra.run.smithi114.stdout: "long_desc": "", 2024-04-23T10:41:08.364 INFO:teuthology.orchestra.run.smithi114.stdout: "tags": [], 2024-04-23T10:41:08.364 INFO:teuthology.orchestra.run.smithi114.stdout: "see_also": [] 2024-04-23T10:41:08.364 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-23T10:41:08.364 INFO:teuthology.orchestra.run.smithi114.stdout: "log_to_cluster": { 2024-04-23T10:41:08.364 INFO:teuthology.orchestra.run.smithi114.stdout: "name": "log_to_cluster", 2024-04-23T10:41:08.364 INFO:teuthology.orchestra.run.smithi114.stdout: "type": "bool", 2024-04-23T10:41:08.364 INFO:teuthology.orchestra.run.smithi114.stdout: "level": "advanced", 2024-04-23T10:41:08.365 INFO:teuthology.orchestra.run.smithi114.stdout: "flags": 1, 2024-04-23T10:41:08.365 INFO:teuthology.orchestra.run.smithi114.stdout: "default_value": "False", 2024-04-23T10:41:08.365 INFO:teuthology.orchestra.run.smithi114.stdout: "min": "", 2024-04-23T10:41:08.365 INFO:teuthology.orchestra.run.smithi114.stdout: "max": "", 2024-04-23T10:41:08.365 INFO:teuthology.orchestra.run.smithi114.stdout: "enum_allowed": [], 2024-04-23T10:41:08.365 INFO:teuthology.orchestra.run.smithi114.stdout: "desc": "", 2024-04-23T10:41:08.365 INFO:teuthology.orchestra.run.smithi114.stdout: "long_desc": "", 2024-04-23T10:41:08.365 INFO:teuthology.orchestra.run.smithi114.stdout: "tags": [], 2024-04-23T10:41:08.365 INFO:teuthology.orchestra.run.smithi114.stdout: "see_also": [] 2024-04-23T10:41:08.366 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-23T10:41:08.366 INFO:teuthology.orchestra.run.smithi114.stdout: "log_to_cluster_level": { 2024-04-23T10:41:08.366 INFO:teuthology.orchestra.run.smithi114.stdout: "name": "log_to_cluster_level", 2024-04-23T10:41:08.366 INFO:teuthology.orchestra.run.smithi114.stdout: "type": "str", 2024-04-23T10:41:08.366 INFO:teuthology.orchestra.run.smithi114.stdout: "level": "advanced", 2024-04-23T10:41:08.366 INFO:teuthology.orchestra.run.smithi114.stdout: "flags": 1, 2024-04-23T10:41:08.366 INFO:teuthology.orchestra.run.smithi114.stdout: "default_value": "info", 2024-04-23T10:41:08.366 INFO:teuthology.orchestra.run.smithi114.stdout: "min": "", 2024-04-23T10:41:08.366 INFO:teuthology.orchestra.run.smithi114.stdout: "max": "", 2024-04-23T10:41:08.366 INFO:teuthology.orchestra.run.smithi114.stdout: "enum_allowed": [ 2024-04-23T10:41:08.367 INFO:teuthology.orchestra.run.smithi114.stdout: "", 2024-04-23T10:41:08.367 INFO:teuthology.orchestra.run.smithi114.stdout: "critical", 2024-04-23T10:41:08.367 INFO:teuthology.orchestra.run.smithi114.stdout: "debug", 2024-04-23T10:41:08.367 INFO:teuthology.orchestra.run.smithi114.stdout: "error", 2024-04-23T10:41:08.367 INFO:teuthology.orchestra.run.smithi114.stdout: "info", 2024-04-23T10:41:08.367 INFO:teuthology.orchestra.run.smithi114.stdout: "warning" 2024-04-23T10:41:08.367 INFO:teuthology.orchestra.run.smithi114.stdout: ], 2024-04-23T10:41:08.367 INFO:teuthology.orchestra.run.smithi114.stdout: "desc": "", 2024-04-23T10:41:08.367 INFO:teuthology.orchestra.run.smithi114.stdout: "long_desc": "", 2024-04-23T10:41:08.367 INFO:teuthology.orchestra.run.smithi114.stdout: "tags": [], 2024-04-23T10:41:08.368 INFO:teuthology.orchestra.run.smithi114.stdout: "see_also": [] 2024-04-23T10:41:08.368 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-23T10:41:08.368 INFO:teuthology.orchestra.run.smithi114.stdout: "log_to_file": { 2024-04-23T10:41:08.368 INFO:teuthology.orchestra.run.smithi114.stdout: "name": "log_to_file", 2024-04-23T10:41:08.368 INFO:teuthology.orchestra.run.smithi114.stdout: "type": "bool", 2024-04-23T10:41:08.368 INFO:teuthology.orchestra.run.smithi114.stdout: "level": "advanced", 2024-04-23T10:41:08.368 INFO:teuthology.orchestra.run.smithi114.stdout: "flags": 1, 2024-04-23T10:41:08.368 INFO:teuthology.orchestra.run.smithi114.stdout: "default_value": "False", 2024-04-23T10:41:08.368 INFO:teuthology.orchestra.run.smithi114.stdout: "min": "", 2024-04-23T10:41:08.369 INFO:teuthology.orchestra.run.smithi114.stdout: "max": "", 2024-04-23T10:41:08.369 INFO:teuthology.orchestra.run.smithi114.stdout: "enum_allowed": [], 2024-04-23T10:41:08.369 INFO:teuthology.orchestra.run.smithi114.stdout: "desc": "", 2024-04-23T10:41:08.369 INFO:teuthology.orchestra.run.smithi114.stdout: "long_desc": "", 2024-04-23T10:41:08.369 INFO:teuthology.orchestra.run.smithi114.stdout: "tags": [], 2024-04-23T10:41:08.369 INFO:teuthology.orchestra.run.smithi114.stdout: "see_also": [] 2024-04-23T10:41:08.369 INFO:teuthology.orchestra.run.smithi114.stdout: } 2024-04-23T10:41:08.369 INFO:teuthology.orchestra.run.smithi114.stdout: } 2024-04-23T10:41:08.369 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-23T10:41:08.369 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-04-23T10:41:08.370 INFO:teuthology.orchestra.run.smithi114.stdout: "name": "k8sevents", 2024-04-23T10:41:08.370 INFO:teuthology.orchestra.run.smithi114.stdout: "can_run": true, 2024-04-23T10:41:08.370 INFO:teuthology.orchestra.run.smithi114.stdout: "error_string": "", 2024-04-23T10:41:08.370 INFO:teuthology.orchestra.run.smithi114.stdout: "module_options": { 2024-04-23T10:41:08.370 INFO:teuthology.orchestra.run.smithi114.stdout: "ceph_event_retention_days": { 2024-04-23T10:41:08.370 INFO:teuthology.orchestra.run.smithi114.stdout: "name": "ceph_event_retention_days", 2024-04-23T10:41:08.370 INFO:teuthology.orchestra.run.smithi114.stdout: "type": "int", 2024-04-23T10:41:08.370 INFO:teuthology.orchestra.run.smithi114.stdout: "level": "advanced", 2024-04-23T10:41:08.370 INFO:teuthology.orchestra.run.smithi114.stdout: "flags": 0, 2024-04-23T10:41:08.370 INFO:teuthology.orchestra.run.smithi114.stdout: "default_value": "7", 2024-04-23T10:41:08.371 INFO:teuthology.orchestra.run.smithi114.stdout: "min": "", 2024-04-23T10:41:08.371 INFO:teuthology.orchestra.run.smithi114.stdout: "max": "", 2024-04-23T10:41:08.371 INFO:teuthology.orchestra.run.smithi114.stdout: "enum_allowed": [], 2024-04-23T10:41:08.371 INFO:teuthology.orchestra.run.smithi114.stdout: "desc": "Days to hold ceph event information within local cache", 2024-04-23T10:41:08.371 INFO:teuthology.orchestra.run.smithi114.stdout: "long_desc": "", 2024-04-23T10:41:08.371 INFO:teuthology.orchestra.run.smithi114.stdout: "tags": [], 2024-04-23T10:41:08.371 INFO:teuthology.orchestra.run.smithi114.stdout: "see_also": [] 2024-04-23T10:41:08.371 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-23T10:41:08.371 INFO:teuthology.orchestra.run.smithi114.stdout: "config_check_secs": { 2024-04-23T10:41:08.372 INFO:teuthology.orchestra.run.smithi114.stdout: "name": "config_check_secs", 2024-04-23T10:41:08.372 INFO:teuthology.orchestra.run.smithi114.stdout: "type": "int", 2024-04-23T10:41:08.372 INFO:teuthology.orchestra.run.smithi114.stdout: "level": "advanced", 2024-04-23T10:41:08.372 INFO:teuthology.orchestra.run.smithi114.stdout: "flags": 0, 2024-04-23T10:41:08.372 INFO:teuthology.orchestra.run.smithi114.stdout: "default_value": "10", 2024-04-23T10:41:08.372 INFO:teuthology.orchestra.run.smithi114.stdout: "min": "10", 2024-04-23T10:41:08.372 INFO:teuthology.orchestra.run.smithi114.stdout: "max": "", 2024-04-23T10:41:08.372 INFO:teuthology.orchestra.run.smithi114.stdout: "enum_allowed": [], 2024-04-23T10:41:08.372 INFO:teuthology.orchestra.run.smithi114.stdout: "desc": "interval (secs) to check for cluster configuration changes", 2024-04-23T10:41:08.372 INFO:teuthology.orchestra.run.smithi114.stdout: "long_desc": "", 2024-04-23T10:41:08.373 INFO:teuthology.orchestra.run.smithi114.stdout: "tags": [], 2024-04-23T10:41:08.373 INFO:teuthology.orchestra.run.smithi114.stdout: "see_also": [] 2024-04-23T10:41:08.373 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-23T10:41:08.373 INFO:teuthology.orchestra.run.smithi114.stdout: "log_level": { 2024-04-23T10:41:08.373 INFO:teuthology.orchestra.run.smithi114.stdout: "name": "log_level", 2024-04-23T10:41:08.373 INFO:teuthology.orchestra.run.smithi114.stdout: "type": "str", 2024-04-23T10:41:08.373 INFO:teuthology.orchestra.run.smithi114.stdout: "level": "advanced", 2024-04-23T10:41:08.373 INFO:teuthology.orchestra.run.smithi114.stdout: "flags": 1, 2024-04-23T10:41:08.373 INFO:teuthology.orchestra.run.smithi114.stdout: "default_value": "", 2024-04-23T10:41:08.374 INFO:teuthology.orchestra.run.smithi114.stdout: "min": "", 2024-04-23T10:41:08.374 INFO:teuthology.orchestra.run.smithi114.stdout: "max": "", 2024-04-23T10:41:08.374 INFO:teuthology.orchestra.run.smithi114.stdout: "enum_allowed": [ 2024-04-23T10:41:08.374 INFO:teuthology.orchestra.run.smithi114.stdout: "", 2024-04-23T10:41:08.374 INFO:teuthology.orchestra.run.smithi114.stdout: "critical", 2024-04-23T10:41:08.374 INFO:teuthology.orchestra.run.smithi114.stdout: "debug", 2024-04-23T10:41:08.374 INFO:teuthology.orchestra.run.smithi114.stdout: "error", 2024-04-23T10:41:08.374 INFO:teuthology.orchestra.run.smithi114.stdout: "info", 2024-04-23T10:41:08.374 INFO:teuthology.orchestra.run.smithi114.stdout: "warning" 2024-04-23T10:41:08.374 INFO:teuthology.orchestra.run.smithi114.stdout: ], 2024-04-23T10:41:08.375 INFO:teuthology.orchestra.run.smithi114.stdout: "desc": "", 2024-04-23T10:41:08.375 INFO:teuthology.orchestra.run.smithi114.stdout: "long_desc": "", 2024-04-23T10:41:08.375 INFO:teuthology.orchestra.run.smithi114.stdout: "tags": [], 2024-04-23T10:41:08.375 INFO:teuthology.orchestra.run.smithi114.stdout: "see_also": [] 2024-04-23T10:41:08.375 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-23T10:41:08.375 INFO:teuthology.orchestra.run.smithi114.stdout: "log_to_cluster": { 2024-04-23T10:41:08.375 INFO:teuthology.orchestra.run.smithi114.stdout: "name": "log_to_cluster", 2024-04-23T10:41:08.375 INFO:teuthology.orchestra.run.smithi114.stdout: "type": "bool", 2024-04-23T10:41:08.375 INFO:teuthology.orchestra.run.smithi114.stdout: "level": "advanced", 2024-04-23T10:41:08.375 INFO:teuthology.orchestra.run.smithi114.stdout: "flags": 1, 2024-04-23T10:41:08.376 INFO:teuthology.orchestra.run.smithi114.stdout: "default_value": "False", 2024-04-23T10:41:08.376 INFO:teuthology.orchestra.run.smithi114.stdout: "min": "", 2024-04-23T10:41:08.376 INFO:teuthology.orchestra.run.smithi114.stdout: "max": "", 2024-04-23T10:41:08.376 INFO:teuthology.orchestra.run.smithi114.stdout: "enum_allowed": [], 2024-04-23T10:41:08.376 INFO:teuthology.orchestra.run.smithi114.stdout: "desc": "", 2024-04-23T10:41:08.376 INFO:teuthology.orchestra.run.smithi114.stdout: "long_desc": "", 2024-04-23T10:41:08.376 INFO:teuthology.orchestra.run.smithi114.stdout: "tags": [], 2024-04-23T10:41:08.376 INFO:teuthology.orchestra.run.smithi114.stdout: "see_also": [] 2024-04-23T10:41:08.376 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-23T10:41:08.376 INFO:teuthology.orchestra.run.smithi114.stdout: "log_to_cluster_level": { 2024-04-23T10:41:08.377 INFO:teuthology.orchestra.run.smithi114.stdout: "name": "log_to_cluster_level", 2024-04-23T10:41:08.377 INFO:teuthology.orchestra.run.smithi114.stdout: "type": "str", 2024-04-23T10:41:08.377 INFO:teuthology.orchestra.run.smithi114.stdout: "level": "advanced", 2024-04-23T10:41:08.377 INFO:teuthology.orchestra.run.smithi114.stdout: "flags": 1, 2024-04-23T10:41:08.377 INFO:teuthology.orchestra.run.smithi114.stdout: "default_value": "info", 2024-04-23T10:41:08.377 INFO:teuthology.orchestra.run.smithi114.stdout: "min": "", 2024-04-23T10:41:08.377 INFO:teuthology.orchestra.run.smithi114.stdout: "max": "", 2024-04-23T10:41:08.377 INFO:teuthology.orchestra.run.smithi114.stdout: "enum_allowed": [ 2024-04-23T10:41:08.377 INFO:teuthology.orchestra.run.smithi114.stdout: "", 2024-04-23T10:41:08.377 INFO:teuthology.orchestra.run.smithi114.stdout: "critical", 2024-04-23T10:41:08.378 INFO:teuthology.orchestra.run.smithi114.stdout: "debug", 2024-04-23T10:41:08.378 INFO:teuthology.orchestra.run.smithi114.stdout: "error", 2024-04-23T10:41:08.378 INFO:teuthology.orchestra.run.smithi114.stdout: "info", 2024-04-23T10:41:08.378 INFO:teuthology.orchestra.run.smithi114.stdout: "warning" 2024-04-23T10:41:08.378 INFO:teuthology.orchestra.run.smithi114.stdout: ], 2024-04-23T10:41:08.378 INFO:teuthology.orchestra.run.smithi114.stdout: "desc": "", 2024-04-23T10:41:08.378 INFO:teuthology.orchestra.run.smithi114.stdout: "long_desc": "", 2024-04-23T10:41:08.378 INFO:teuthology.orchestra.run.smithi114.stdout: "tags": [], 2024-04-23T10:41:08.378 INFO:teuthology.orchestra.run.smithi114.stdout: "see_also": [] 2024-04-23T10:41:08.379 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-23T10:41:08.379 INFO:teuthology.orchestra.run.smithi114.stdout: "log_to_file": { 2024-04-23T10:41:08.379 INFO:teuthology.orchestra.run.smithi114.stdout: "name": "log_to_file", 2024-04-23T10:41:08.379 INFO:teuthology.orchestra.run.smithi114.stdout: "type": "bool", 2024-04-23T10:41:08.379 INFO:teuthology.orchestra.run.smithi114.stdout: "level": "advanced", 2024-04-23T10:41:08.379 INFO:teuthology.orchestra.run.smithi114.stdout: "flags": 1, 2024-04-23T10:41:08.379 INFO:teuthology.orchestra.run.smithi114.stdout: "default_value": "False", 2024-04-23T10:41:08.379 INFO:teuthology.orchestra.run.smithi114.stdout: "min": "", 2024-04-23T10:41:08.379 INFO:teuthology.orchestra.run.smithi114.stdout: "max": "", 2024-04-23T10:41:08.379 INFO:teuthology.orchestra.run.smithi114.stdout: "enum_allowed": [], 2024-04-23T10:41:08.380 INFO:teuthology.orchestra.run.smithi114.stdout: "desc": "", 2024-04-23T10:41:08.380 INFO:teuthology.orchestra.run.smithi114.stdout: "long_desc": "", 2024-04-23T10:41:08.380 INFO:teuthology.orchestra.run.smithi114.stdout: "tags": [], 2024-04-23T10:41:08.380 INFO:teuthology.orchestra.run.smithi114.stdout: "see_also": [] 2024-04-23T10:41:08.380 INFO:teuthology.orchestra.run.smithi114.stdout: } 2024-04-23T10:41:08.380 INFO:teuthology.orchestra.run.smithi114.stdout: } 2024-04-23T10:41:08.380 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-23T10:41:08.380 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-04-23T10:41:08.380 INFO:teuthology.orchestra.run.smithi114.stdout: "name": "localpool", 2024-04-23T10:41:08.380 INFO:teuthology.orchestra.run.smithi114.stdout: "can_run": true, 2024-04-23T10:41:08.381 INFO:teuthology.orchestra.run.smithi114.stdout: "error_string": "", 2024-04-23T10:41:08.381 INFO:teuthology.orchestra.run.smithi114.stdout: "module_options": { 2024-04-23T10:41:08.381 INFO:teuthology.orchestra.run.smithi114.stdout: "failure_domain": { 2024-04-23T10:41:08.381 INFO:teuthology.orchestra.run.smithi114.stdout: "name": "failure_domain", 2024-04-23T10:41:08.381 INFO:teuthology.orchestra.run.smithi114.stdout: "type": "str", 2024-04-23T10:41:08.381 INFO:teuthology.orchestra.run.smithi114.stdout: "level": "advanced", 2024-04-23T10:41:08.381 INFO:teuthology.orchestra.run.smithi114.stdout: "flags": 1, 2024-04-23T10:41:08.381 INFO:teuthology.orchestra.run.smithi114.stdout: "default_value": "host", 2024-04-23T10:41:08.381 INFO:teuthology.orchestra.run.smithi114.stdout: "min": "", 2024-04-23T10:41:08.381 INFO:teuthology.orchestra.run.smithi114.stdout: "max": "", 2024-04-23T10:41:08.382 INFO:teuthology.orchestra.run.smithi114.stdout: "enum_allowed": [], 2024-04-23T10:41:08.382 INFO:teuthology.orchestra.run.smithi114.stdout: "desc": "failure domain for any created local pool", 2024-04-23T10:41:08.382 INFO:teuthology.orchestra.run.smithi114.stdout: "long_desc": "what failure domain we should separate data replicas across.", 2024-04-23T10:41:08.382 INFO:teuthology.orchestra.run.smithi114.stdout: "tags": [], 2024-04-23T10:41:08.382 INFO:teuthology.orchestra.run.smithi114.stdout: "see_also": [] 2024-04-23T10:41:08.382 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-23T10:41:08.382 INFO:teuthology.orchestra.run.smithi114.stdout: "log_level": { 2024-04-23T10:41:08.382 INFO:teuthology.orchestra.run.smithi114.stdout: "name": "log_level", 2024-04-23T10:41:08.382 INFO:teuthology.orchestra.run.smithi114.stdout: "type": "str", 2024-04-23T10:41:08.383 INFO:teuthology.orchestra.run.smithi114.stdout: "level": "advanced", 2024-04-23T10:41:08.383 INFO:teuthology.orchestra.run.smithi114.stdout: "flags": 1, 2024-04-23T10:41:08.383 INFO:teuthology.orchestra.run.smithi114.stdout: "default_value": "", 2024-04-23T10:41:08.383 INFO:teuthology.orchestra.run.smithi114.stdout: "min": "", 2024-04-23T10:41:08.383 INFO:teuthology.orchestra.run.smithi114.stdout: "max": "", 2024-04-23T10:41:08.383 INFO:teuthology.orchestra.run.smithi114.stdout: "enum_allowed": [ 2024-04-23T10:41:08.383 INFO:teuthology.orchestra.run.smithi114.stdout: "", 2024-04-23T10:41:08.383 INFO:teuthology.orchestra.run.smithi114.stdout: "critical", 2024-04-23T10:41:08.383 INFO:teuthology.orchestra.run.smithi114.stdout: "debug", 2024-04-23T10:41:08.383 INFO:teuthology.orchestra.run.smithi114.stdout: "error", 2024-04-23T10:41:08.384 INFO:teuthology.orchestra.run.smithi114.stdout: "info", 2024-04-23T10:41:08.384 INFO:teuthology.orchestra.run.smithi114.stdout: "warning" 2024-04-23T10:41:08.384 INFO:teuthology.orchestra.run.smithi114.stdout: ], 2024-04-23T10:41:08.384 INFO:teuthology.orchestra.run.smithi114.stdout: "desc": "", 2024-04-23T10:41:08.384 INFO:teuthology.orchestra.run.smithi114.stdout: "long_desc": "", 2024-04-23T10:41:08.384 INFO:teuthology.orchestra.run.smithi114.stdout: "tags": [], 2024-04-23T10:41:08.384 INFO:teuthology.orchestra.run.smithi114.stdout: "see_also": [] 2024-04-23T10:41:08.384 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-23T10:41:08.384 INFO:teuthology.orchestra.run.smithi114.stdout: "log_to_cluster": { 2024-04-23T10:41:08.384 INFO:teuthology.orchestra.run.smithi114.stdout: "name": "log_to_cluster", 2024-04-23T10:41:08.385 INFO:teuthology.orchestra.run.smithi114.stdout: "type": "bool", 2024-04-23T10:41:08.385 INFO:teuthology.orchestra.run.smithi114.stdout: "level": "advanced", 2024-04-23T10:41:08.385 INFO:teuthology.orchestra.run.smithi114.stdout: "flags": 1, 2024-04-23T10:41:08.385 INFO:teuthology.orchestra.run.smithi114.stdout: "default_value": "False", 2024-04-23T10:41:08.385 INFO:teuthology.orchestra.run.smithi114.stdout: "min": "", 2024-04-23T10:41:08.385 INFO:teuthology.orchestra.run.smithi114.stdout: "max": "", 2024-04-23T10:41:08.385 INFO:teuthology.orchestra.run.smithi114.stdout: "enum_allowed": [], 2024-04-23T10:41:08.385 INFO:teuthology.orchestra.run.smithi114.stdout: "desc": "", 2024-04-23T10:41:08.385 INFO:teuthology.orchestra.run.smithi114.stdout: "long_desc": "", 2024-04-23T10:41:08.385 INFO:teuthology.orchestra.run.smithi114.stdout: "tags": [], 2024-04-23T10:41:08.386 INFO:teuthology.orchestra.run.smithi114.stdout: "see_also": [] 2024-04-23T10:41:08.386 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-23T10:41:08.386 INFO:teuthology.orchestra.run.smithi114.stdout: "log_to_cluster_level": { 2024-04-23T10:41:08.386 INFO:teuthology.orchestra.run.smithi114.stdout: "name": "log_to_cluster_level", 2024-04-23T10:41:08.386 INFO:teuthology.orchestra.run.smithi114.stdout: "type": "str", 2024-04-23T10:41:08.386 INFO:teuthology.orchestra.run.smithi114.stdout: "level": "advanced", 2024-04-23T10:41:08.386 INFO:teuthology.orchestra.run.smithi114.stdout: "flags": 1, 2024-04-23T10:41:08.386 INFO:teuthology.orchestra.run.smithi114.stdout: "default_value": "info", 2024-04-23T10:41:08.386 INFO:teuthology.orchestra.run.smithi114.stdout: "min": "", 2024-04-23T10:41:08.386 INFO:teuthology.orchestra.run.smithi114.stdout: "max": "", 2024-04-23T10:41:08.387 INFO:teuthology.orchestra.run.smithi114.stdout: "enum_allowed": [ 2024-04-23T10:41:08.387 INFO:teuthology.orchestra.run.smithi114.stdout: "", 2024-04-23T10:41:08.387 INFO:teuthology.orchestra.run.smithi114.stdout: "critical", 2024-04-23T10:41:08.387 INFO:teuthology.orchestra.run.smithi114.stdout: "debug", 2024-04-23T10:41:08.387 INFO:teuthology.orchestra.run.smithi114.stdout: "error", 2024-04-23T10:41:08.387 INFO:teuthology.orchestra.run.smithi114.stdout: "info", 2024-04-23T10:41:08.387 INFO:teuthology.orchestra.run.smithi114.stdout: "warning" 2024-04-23T10:41:08.387 INFO:teuthology.orchestra.run.smithi114.stdout: ], 2024-04-23T10:41:08.387 INFO:teuthology.orchestra.run.smithi114.stdout: "desc": "", 2024-04-23T10:41:08.388 INFO:teuthology.orchestra.run.smithi114.stdout: "long_desc": "", 2024-04-23T10:41:08.388 INFO:teuthology.orchestra.run.smithi114.stdout: "tags": [], 2024-04-23T10:41:08.388 INFO:teuthology.orchestra.run.smithi114.stdout: "see_also": [] 2024-04-23T10:41:08.388 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-23T10:41:08.388 INFO:teuthology.orchestra.run.smithi114.stdout: "log_to_file": { 2024-04-23T10:41:08.388 INFO:teuthology.orchestra.run.smithi114.stdout: "name": "log_to_file", 2024-04-23T10:41:08.388 INFO:teuthology.orchestra.run.smithi114.stdout: "type": "bool", 2024-04-23T10:41:08.388 INFO:teuthology.orchestra.run.smithi114.stdout: "level": "advanced", 2024-04-23T10:41:08.388 INFO:teuthology.orchestra.run.smithi114.stdout: "flags": 1, 2024-04-23T10:41:08.389 INFO:teuthology.orchestra.run.smithi114.stdout: "default_value": "False", 2024-04-23T10:41:08.389 INFO:teuthology.orchestra.run.smithi114.stdout: "min": "", 2024-04-23T10:41:08.389 INFO:teuthology.orchestra.run.smithi114.stdout: "max": "", 2024-04-23T10:41:08.389 INFO:teuthology.orchestra.run.smithi114.stdout: "enum_allowed": [], 2024-04-23T10:41:08.389 INFO:teuthology.orchestra.run.smithi114.stdout: "desc": "", 2024-04-23T10:41:08.389 INFO:teuthology.orchestra.run.smithi114.stdout: "long_desc": "", 2024-04-23T10:41:08.389 INFO:teuthology.orchestra.run.smithi114.stdout: "tags": [], 2024-04-23T10:41:08.389 INFO:teuthology.orchestra.run.smithi114.stdout: "see_also": [] 2024-04-23T10:41:08.389 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-23T10:41:08.389 INFO:teuthology.orchestra.run.smithi114.stdout: "min_size": { 2024-04-23T10:41:08.390 INFO:teuthology.orchestra.run.smithi114.stdout: "name": "min_size", 2024-04-23T10:41:08.390 INFO:teuthology.orchestra.run.smithi114.stdout: "type": "int", 2024-04-23T10:41:08.390 INFO:teuthology.orchestra.run.smithi114.stdout: "level": "advanced", 2024-04-23T10:41:08.390 INFO:teuthology.orchestra.run.smithi114.stdout: "flags": 1, 2024-04-23T10:41:08.390 INFO:teuthology.orchestra.run.smithi114.stdout: "default_value": "", 2024-04-23T10:41:08.390 INFO:teuthology.orchestra.run.smithi114.stdout: "min": "", 2024-04-23T10:41:08.390 INFO:teuthology.orchestra.run.smithi114.stdout: "max": "", 2024-04-23T10:41:08.390 INFO:teuthology.orchestra.run.smithi114.stdout: "enum_allowed": [], 2024-04-23T10:41:08.390 INFO:teuthology.orchestra.run.smithi114.stdout: "desc": "default min_size for any created local pool", 2024-04-23T10:41:08.390 INFO:teuthology.orchestra.run.smithi114.stdout: "long_desc": "value to set min_size to (unchanged from Ceph's default if this option is not set)", 2024-04-23T10:41:08.391 INFO:teuthology.orchestra.run.smithi114.stdout: "tags": [], 2024-04-23T10:41:08.391 INFO:teuthology.orchestra.run.smithi114.stdout: "see_also": [] 2024-04-23T10:41:08.391 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-23T10:41:08.391 INFO:teuthology.orchestra.run.smithi114.stdout: "num_rep": { 2024-04-23T10:41:08.391 INFO:teuthology.orchestra.run.smithi114.stdout: "name": "num_rep", 2024-04-23T10:41:08.391 INFO:teuthology.orchestra.run.smithi114.stdout: "type": "int", 2024-04-23T10:41:08.391 INFO:teuthology.orchestra.run.smithi114.stdout: "level": "advanced", 2024-04-23T10:41:08.391 INFO:teuthology.orchestra.run.smithi114.stdout: "flags": 1, 2024-04-23T10:41:08.391 INFO:teuthology.orchestra.run.smithi114.stdout: "default_value": "3", 2024-04-23T10:41:08.391 INFO:teuthology.orchestra.run.smithi114.stdout: "min": "", 2024-04-23T10:41:08.392 INFO:teuthology.orchestra.run.smithi114.stdout: "max": "", 2024-04-23T10:41:08.392 INFO:teuthology.orchestra.run.smithi114.stdout: "enum_allowed": [], 2024-04-23T10:41:08.392 INFO:teuthology.orchestra.run.smithi114.stdout: "desc": "default replica count for any created local pool", 2024-04-23T10:41:08.392 INFO:teuthology.orchestra.run.smithi114.stdout: "long_desc": "", 2024-04-23T10:41:08.392 INFO:teuthology.orchestra.run.smithi114.stdout: "tags": [], 2024-04-23T10:41:08.392 INFO:teuthology.orchestra.run.smithi114.stdout: "see_also": [] 2024-04-23T10:41:08.392 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-23T10:41:08.392 INFO:teuthology.orchestra.run.smithi114.stdout: "pg_num": { 2024-04-23T10:41:08.392 INFO:teuthology.orchestra.run.smithi114.stdout: "name": "pg_num", 2024-04-23T10:41:08.392 INFO:teuthology.orchestra.run.smithi114.stdout: "type": "int", 2024-04-23T10:41:08.393 INFO:teuthology.orchestra.run.smithi114.stdout: "level": "advanced", 2024-04-23T10:41:08.393 INFO:teuthology.orchestra.run.smithi114.stdout: "flags": 1, 2024-04-23T10:41:08.393 INFO:teuthology.orchestra.run.smithi114.stdout: "default_value": "128", 2024-04-23T10:41:08.393 INFO:teuthology.orchestra.run.smithi114.stdout: "min": "", 2024-04-23T10:41:08.393 INFO:teuthology.orchestra.run.smithi114.stdout: "max": "", 2024-04-23T10:41:08.393 INFO:teuthology.orchestra.run.smithi114.stdout: "enum_allowed": [], 2024-04-23T10:41:08.393 INFO:teuthology.orchestra.run.smithi114.stdout: "desc": "default pg_num for any created local pool", 2024-04-23T10:41:08.393 INFO:teuthology.orchestra.run.smithi114.stdout: "long_desc": "", 2024-04-23T10:41:08.393 INFO:teuthology.orchestra.run.smithi114.stdout: "tags": [], 2024-04-23T10:41:08.394 INFO:teuthology.orchestra.run.smithi114.stdout: "see_also": [] 2024-04-23T10:41:08.394 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-23T10:41:08.394 INFO:teuthology.orchestra.run.smithi114.stdout: "prefix": { 2024-04-23T10:41:08.394 INFO:teuthology.orchestra.run.smithi114.stdout: "name": "prefix", 2024-04-23T10:41:08.394 INFO:teuthology.orchestra.run.smithi114.stdout: "type": "str", 2024-04-23T10:41:08.394 INFO:teuthology.orchestra.run.smithi114.stdout: "level": "advanced", 2024-04-23T10:41:08.394 INFO:teuthology.orchestra.run.smithi114.stdout: "flags": 1, 2024-04-23T10:41:08.394 INFO:teuthology.orchestra.run.smithi114.stdout: "default_value": "", 2024-04-23T10:41:08.394 INFO:teuthology.orchestra.run.smithi114.stdout: "min": "", 2024-04-23T10:41:08.395 INFO:teuthology.orchestra.run.smithi114.stdout: "max": "", 2024-04-23T10:41:08.395 INFO:teuthology.orchestra.run.smithi114.stdout: "enum_allowed": [], 2024-04-23T10:41:08.395 INFO:teuthology.orchestra.run.smithi114.stdout: "desc": "name prefix for any created local pool", 2024-04-23T10:41:08.395 INFO:teuthology.orchestra.run.smithi114.stdout: "long_desc": "", 2024-04-23T10:41:08.395 INFO:teuthology.orchestra.run.smithi114.stdout: "tags": [], 2024-04-23T10:41:08.395 INFO:teuthology.orchestra.run.smithi114.stdout: "see_also": [] 2024-04-23T10:41:08.395 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-23T10:41:08.395 INFO:teuthology.orchestra.run.smithi114.stdout: "subtree": { 2024-04-23T10:41:08.395 INFO:teuthology.orchestra.run.smithi114.stdout: "name": "subtree", 2024-04-23T10:41:08.395 INFO:teuthology.orchestra.run.smithi114.stdout: "type": "str", 2024-04-23T10:41:08.396 INFO:teuthology.orchestra.run.smithi114.stdout: "level": "advanced", 2024-04-23T10:41:08.396 INFO:teuthology.orchestra.run.smithi114.stdout: "flags": 1, 2024-04-23T10:41:08.396 INFO:teuthology.orchestra.run.smithi114.stdout: "default_value": "rack", 2024-04-23T10:41:08.396 INFO:teuthology.orchestra.run.smithi114.stdout: "min": "", 2024-04-23T10:41:08.396 INFO:teuthology.orchestra.run.smithi114.stdout: "max": "", 2024-04-23T10:41:08.396 INFO:teuthology.orchestra.run.smithi114.stdout: "enum_allowed": [], 2024-04-23T10:41:08.396 INFO:teuthology.orchestra.run.smithi114.stdout: "desc": "CRUSH level for which to create a local pool", 2024-04-23T10:41:08.396 INFO:teuthology.orchestra.run.smithi114.stdout: "long_desc": "which CRUSH subtree type the module should create a pool for.", 2024-04-23T10:41:08.396 INFO:teuthology.orchestra.run.smithi114.stdout: "tags": [], 2024-04-23T10:41:08.396 INFO:teuthology.orchestra.run.smithi114.stdout: "see_also": [] 2024-04-23T10:41:08.397 INFO:teuthology.orchestra.run.smithi114.stdout: } 2024-04-23T10:41:08.397 INFO:teuthology.orchestra.run.smithi114.stdout: } 2024-04-23T10:41:08.397 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-23T10:41:08.397 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-04-23T10:41:08.397 INFO:teuthology.orchestra.run.smithi114.stdout: "name": "mds_autoscaler", 2024-04-23T10:41:08.397 INFO:teuthology.orchestra.run.smithi114.stdout: "can_run": true, 2024-04-23T10:41:08.397 INFO:teuthology.orchestra.run.smithi114.stdout: "error_string": "", 2024-04-23T10:41:08.397 INFO:teuthology.orchestra.run.smithi114.stdout: "module_options": { 2024-04-23T10:41:08.397 INFO:teuthology.orchestra.run.smithi114.stdout: "log_level": { 2024-04-23T10:41:08.398 INFO:teuthology.orchestra.run.smithi114.stdout: "name": "log_level", 2024-04-23T10:41:08.398 INFO:teuthology.orchestra.run.smithi114.stdout: "type": "str", 2024-04-23T10:41:08.398 INFO:teuthology.orchestra.run.smithi114.stdout: "level": "advanced", 2024-04-23T10:41:08.398 INFO:teuthology.orchestra.run.smithi114.stdout: "flags": 1, 2024-04-23T10:41:08.398 INFO:teuthology.orchestra.run.smithi114.stdout: "default_value": "", 2024-04-23T10:41:08.398 INFO:teuthology.orchestra.run.smithi114.stdout: "min": "", 2024-04-23T10:41:08.398 INFO:teuthology.orchestra.run.smithi114.stdout: "max": "", 2024-04-23T10:41:08.398 INFO:teuthology.orchestra.run.smithi114.stdout: "enum_allowed": [ 2024-04-23T10:41:08.398 INFO:teuthology.orchestra.run.smithi114.stdout: "", 2024-04-23T10:41:08.398 INFO:teuthology.orchestra.run.smithi114.stdout: "critical", 2024-04-23T10:41:08.399 INFO:teuthology.orchestra.run.smithi114.stdout: "debug", 2024-04-23T10:41:08.399 INFO:teuthology.orchestra.run.smithi114.stdout: "error", 2024-04-23T10:41:08.399 INFO:teuthology.orchestra.run.smithi114.stdout: "info", 2024-04-23T10:41:08.399 INFO:teuthology.orchestra.run.smithi114.stdout: "warning" 2024-04-23T10:41:08.399 INFO:teuthology.orchestra.run.smithi114.stdout: ], 2024-04-23T10:41:08.399 INFO:teuthology.orchestra.run.smithi114.stdout: "desc": "", 2024-04-23T10:41:08.399 INFO:teuthology.orchestra.run.smithi114.stdout: "long_desc": "", 2024-04-23T10:41:08.399 INFO:teuthology.orchestra.run.smithi114.stdout: "tags": [], 2024-04-23T10:41:08.399 INFO:teuthology.orchestra.run.smithi114.stdout: "see_also": [] 2024-04-23T10:41:08.399 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-23T10:41:08.400 INFO:teuthology.orchestra.run.smithi114.stdout: "log_to_cluster": { 2024-04-23T10:41:08.400 INFO:teuthology.orchestra.run.smithi114.stdout: "name": "log_to_cluster", 2024-04-23T10:41:08.400 INFO:teuthology.orchestra.run.smithi114.stdout: "type": "bool", 2024-04-23T10:41:08.400 INFO:teuthology.orchestra.run.smithi114.stdout: "level": "advanced", 2024-04-23T10:41:08.400 INFO:teuthology.orchestra.run.smithi114.stdout: "flags": 1, 2024-04-23T10:41:08.400 INFO:teuthology.orchestra.run.smithi114.stdout: "default_value": "False", 2024-04-23T10:41:08.400 INFO:teuthology.orchestra.run.smithi114.stdout: "min": "", 2024-04-23T10:41:08.400 INFO:teuthology.orchestra.run.smithi114.stdout: "max": "", 2024-04-23T10:41:08.400 INFO:teuthology.orchestra.run.smithi114.stdout: "enum_allowed": [], 2024-04-23T10:41:08.400 INFO:teuthology.orchestra.run.smithi114.stdout: "desc": "", 2024-04-23T10:41:08.401 INFO:teuthology.orchestra.run.smithi114.stdout: "long_desc": "", 2024-04-23T10:41:08.401 INFO:teuthology.orchestra.run.smithi114.stdout: "tags": [], 2024-04-23T10:41:08.401 INFO:teuthology.orchestra.run.smithi114.stdout: "see_also": [] 2024-04-23T10:41:08.401 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-23T10:41:08.401 INFO:teuthology.orchestra.run.smithi114.stdout: "log_to_cluster_level": { 2024-04-23T10:41:08.401 INFO:teuthology.orchestra.run.smithi114.stdout: "name": "log_to_cluster_level", 2024-04-23T10:41:08.401 INFO:teuthology.orchestra.run.smithi114.stdout: "type": "str", 2024-04-23T10:41:08.401 INFO:teuthology.orchestra.run.smithi114.stdout: "level": "advanced", 2024-04-23T10:41:08.401 INFO:teuthology.orchestra.run.smithi114.stdout: "flags": 1, 2024-04-23T10:41:08.401 INFO:teuthology.orchestra.run.smithi114.stdout: "default_value": "info", 2024-04-23T10:41:08.402 INFO:teuthology.orchestra.run.smithi114.stdout: "min": "", 2024-04-23T10:41:08.402 INFO:teuthology.orchestra.run.smithi114.stdout: "max": "", 2024-04-23T10:41:08.402 INFO:teuthology.orchestra.run.smithi114.stdout: "enum_allowed": [ 2024-04-23T10:41:08.402 INFO:teuthology.orchestra.run.smithi114.stdout: "", 2024-04-23T10:41:08.402 INFO:teuthology.orchestra.run.smithi114.stdout: "critical", 2024-04-23T10:41:08.402 INFO:teuthology.orchestra.run.smithi114.stdout: "debug", 2024-04-23T10:41:08.402 INFO:teuthology.orchestra.run.smithi114.stdout: "error", 2024-04-23T10:41:08.402 INFO:teuthology.orchestra.run.smithi114.stdout: "info", 2024-04-23T10:41:08.402 INFO:teuthology.orchestra.run.smithi114.stdout: "warning" 2024-04-23T10:41:08.403 INFO:teuthology.orchestra.run.smithi114.stdout: ], 2024-04-23T10:41:08.403 INFO:teuthology.orchestra.run.smithi114.stdout: "desc": "", 2024-04-23T10:41:08.403 INFO:teuthology.orchestra.run.smithi114.stdout: "long_desc": "", 2024-04-23T10:41:08.403 INFO:teuthology.orchestra.run.smithi114.stdout: "tags": [], 2024-04-23T10:41:08.403 INFO:teuthology.orchestra.run.smithi114.stdout: "see_also": [] 2024-04-23T10:41:08.403 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-23T10:41:08.403 INFO:teuthology.orchestra.run.smithi114.stdout: "log_to_file": { 2024-04-23T10:41:08.403 INFO:teuthology.orchestra.run.smithi114.stdout: "name": "log_to_file", 2024-04-23T10:41:08.403 INFO:teuthology.orchestra.run.smithi114.stdout: "type": "bool", 2024-04-23T10:41:08.404 INFO:teuthology.orchestra.run.smithi114.stdout: "level": "advanced", 2024-04-23T10:41:08.404 INFO:teuthology.orchestra.run.smithi114.stdout: "flags": 1, 2024-04-23T10:41:08.404 INFO:teuthology.orchestra.run.smithi114.stdout: "default_value": "False", 2024-04-23T10:41:08.404 INFO:teuthology.orchestra.run.smithi114.stdout: "min": "", 2024-04-23T10:41:08.404 INFO:teuthology.orchestra.run.smithi114.stdout: "max": "", 2024-04-23T10:41:08.404 INFO:teuthology.orchestra.run.smithi114.stdout: "enum_allowed": [], 2024-04-23T10:41:08.404 INFO:teuthology.orchestra.run.smithi114.stdout: "desc": "", 2024-04-23T10:41:08.404 INFO:teuthology.orchestra.run.smithi114.stdout: "long_desc": "", 2024-04-23T10:41:08.404 INFO:teuthology.orchestra.run.smithi114.stdout: "tags": [], 2024-04-23T10:41:08.404 INFO:teuthology.orchestra.run.smithi114.stdout: "see_also": [] 2024-04-23T10:41:08.405 INFO:teuthology.orchestra.run.smithi114.stdout: } 2024-04-23T10:41:08.405 INFO:teuthology.orchestra.run.smithi114.stdout: } 2024-04-23T10:41:08.405 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-23T10:41:08.405 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-04-23T10:41:08.405 INFO:teuthology.orchestra.run.smithi114.stdout: "name": "mirroring", 2024-04-23T10:41:08.405 INFO:teuthology.orchestra.run.smithi114.stdout: "can_run": true, 2024-04-23T10:41:08.405 INFO:teuthology.orchestra.run.smithi114.stdout: "error_string": "", 2024-04-23T10:41:08.405 INFO:teuthology.orchestra.run.smithi114.stdout: "module_options": { 2024-04-23T10:41:08.405 INFO:teuthology.orchestra.run.smithi114.stdout: "log_level": { 2024-04-23T10:41:08.405 INFO:teuthology.orchestra.run.smithi114.stdout: "name": "log_level", 2024-04-23T10:41:08.406 INFO:teuthology.orchestra.run.smithi114.stdout: "type": "str", 2024-04-23T10:41:08.406 INFO:teuthology.orchestra.run.smithi114.stdout: "level": "advanced", 2024-04-23T10:41:08.406 INFO:teuthology.orchestra.run.smithi114.stdout: "flags": 1, 2024-04-23T10:41:08.406 INFO:teuthology.orchestra.run.smithi114.stdout: "default_value": "", 2024-04-23T10:41:08.406 INFO:teuthology.orchestra.run.smithi114.stdout: "min": "", 2024-04-23T10:41:08.406 INFO:teuthology.orchestra.run.smithi114.stdout: "max": "", 2024-04-23T10:41:08.406 INFO:teuthology.orchestra.run.smithi114.stdout: "enum_allowed": [ 2024-04-23T10:41:08.406 INFO:teuthology.orchestra.run.smithi114.stdout: "", 2024-04-23T10:41:08.406 INFO:teuthology.orchestra.run.smithi114.stdout: "critical", 2024-04-23T10:41:08.406 INFO:teuthology.orchestra.run.smithi114.stdout: "debug", 2024-04-23T10:41:08.407 INFO:teuthology.orchestra.run.smithi114.stdout: "error", 2024-04-23T10:41:08.407 INFO:teuthology.orchestra.run.smithi114.stdout: "info", 2024-04-23T10:41:08.407 INFO:teuthology.orchestra.run.smithi114.stdout: "warning" 2024-04-23T10:41:08.407 INFO:teuthology.orchestra.run.smithi114.stdout: ], 2024-04-23T10:41:08.407 INFO:teuthology.orchestra.run.smithi114.stdout: "desc": "", 2024-04-23T10:41:08.407 INFO:teuthology.orchestra.run.smithi114.stdout: "long_desc": "", 2024-04-23T10:41:08.407 INFO:teuthology.orchestra.run.smithi114.stdout: "tags": [], 2024-04-23T10:41:08.407 INFO:teuthology.orchestra.run.smithi114.stdout: "see_also": [] 2024-04-23T10:41:08.407 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-23T10:41:08.408 INFO:teuthology.orchestra.run.smithi114.stdout: "log_to_cluster": { 2024-04-23T10:41:08.408 INFO:teuthology.orchestra.run.smithi114.stdout: "name": "log_to_cluster", 2024-04-23T10:41:08.408 INFO:teuthology.orchestra.run.smithi114.stdout: "type": "bool", 2024-04-23T10:41:08.408 INFO:teuthology.orchestra.run.smithi114.stdout: "level": "advanced", 2024-04-23T10:41:08.408 INFO:teuthology.orchestra.run.smithi114.stdout: "flags": 1, 2024-04-23T10:41:08.408 INFO:teuthology.orchestra.run.smithi114.stdout: "default_value": "False", 2024-04-23T10:41:08.408 INFO:teuthology.orchestra.run.smithi114.stdout: "min": "", 2024-04-23T10:41:08.408 INFO:teuthology.orchestra.run.smithi114.stdout: "max": "", 2024-04-23T10:41:08.408 INFO:teuthology.orchestra.run.smithi114.stdout: "enum_allowed": [], 2024-04-23T10:41:08.408 INFO:teuthology.orchestra.run.smithi114.stdout: "desc": "", 2024-04-23T10:41:08.409 INFO:teuthology.orchestra.run.smithi114.stdout: "long_desc": "", 2024-04-23T10:41:08.409 INFO:teuthology.orchestra.run.smithi114.stdout: "tags": [], 2024-04-23T10:41:08.409 INFO:teuthology.orchestra.run.smithi114.stdout: "see_also": [] 2024-04-23T10:41:08.409 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-23T10:41:08.409 INFO:teuthology.orchestra.run.smithi114.stdout: "log_to_cluster_level": { 2024-04-23T10:41:08.409 INFO:teuthology.orchestra.run.smithi114.stdout: "name": "log_to_cluster_level", 2024-04-23T10:41:08.409 INFO:teuthology.orchestra.run.smithi114.stdout: "type": "str", 2024-04-23T10:41:08.409 INFO:teuthology.orchestra.run.smithi114.stdout: "level": "advanced", 2024-04-23T10:41:08.409 INFO:teuthology.orchestra.run.smithi114.stdout: "flags": 1, 2024-04-23T10:41:08.409 INFO:teuthology.orchestra.run.smithi114.stdout: "default_value": "info", 2024-04-23T10:41:08.410 INFO:teuthology.orchestra.run.smithi114.stdout: "min": "", 2024-04-23T10:41:08.410 INFO:teuthology.orchestra.run.smithi114.stdout: "max": "", 2024-04-23T10:41:08.410 INFO:teuthology.orchestra.run.smithi114.stdout: "enum_allowed": [ 2024-04-23T10:41:08.410 INFO:teuthology.orchestra.run.smithi114.stdout: "", 2024-04-23T10:41:08.410 INFO:teuthology.orchestra.run.smithi114.stdout: "critical", 2024-04-23T10:41:08.410 INFO:teuthology.orchestra.run.smithi114.stdout: "debug", 2024-04-23T10:41:08.410 INFO:teuthology.orchestra.run.smithi114.stdout: "error", 2024-04-23T10:41:08.410 INFO:teuthology.orchestra.run.smithi114.stdout: "info", 2024-04-23T10:41:08.410 INFO:teuthology.orchestra.run.smithi114.stdout: "warning" 2024-04-23T10:41:08.411 INFO:teuthology.orchestra.run.smithi114.stdout: ], 2024-04-23T10:41:08.411 INFO:teuthology.orchestra.run.smithi114.stdout: "desc": "", 2024-04-23T10:41:08.411 INFO:teuthology.orchestra.run.smithi114.stdout: "long_desc": "", 2024-04-23T10:41:08.411 INFO:teuthology.orchestra.run.smithi114.stdout: "tags": [], 2024-04-23T10:41:08.411 INFO:teuthology.orchestra.run.smithi114.stdout: "see_also": [] 2024-04-23T10:41:08.411 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-23T10:41:08.411 INFO:teuthology.orchestra.run.smithi114.stdout: "log_to_file": { 2024-04-23T10:41:08.411 INFO:teuthology.orchestra.run.smithi114.stdout: "name": "log_to_file", 2024-04-23T10:41:08.411 INFO:teuthology.orchestra.run.smithi114.stdout: "type": "bool", 2024-04-23T10:41:08.411 INFO:teuthology.orchestra.run.smithi114.stdout: "level": "advanced", 2024-04-23T10:41:08.412 INFO:teuthology.orchestra.run.smithi114.stdout: "flags": 1, 2024-04-23T10:41:08.412 INFO:teuthology.orchestra.run.smithi114.stdout: "default_value": "False", 2024-04-23T10:41:08.412 INFO:teuthology.orchestra.run.smithi114.stdout: "min": "", 2024-04-23T10:41:08.412 INFO:teuthology.orchestra.run.smithi114.stdout: "max": "", 2024-04-23T10:41:08.412 INFO:teuthology.orchestra.run.smithi114.stdout: "enum_allowed": [], 2024-04-23T10:41:08.412 INFO:teuthology.orchestra.run.smithi114.stdout: "desc": "", 2024-04-23T10:41:08.412 INFO:teuthology.orchestra.run.smithi114.stdout: "long_desc": "", 2024-04-23T10:41:08.412 INFO:teuthology.orchestra.run.smithi114.stdout: "tags": [], 2024-04-23T10:41:08.412 INFO:teuthology.orchestra.run.smithi114.stdout: "see_also": [] 2024-04-23T10:41:08.412 INFO:teuthology.orchestra.run.smithi114.stdout: } 2024-04-23T10:41:08.413 INFO:teuthology.orchestra.run.smithi114.stdout: } 2024-04-23T10:41:08.413 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-23T10:41:08.413 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-04-23T10:41:08.413 INFO:teuthology.orchestra.run.smithi114.stdout: "name": "nfs", 2024-04-23T10:41:08.413 INFO:teuthology.orchestra.run.smithi114.stdout: "can_run": true, 2024-04-23T10:41:08.413 INFO:teuthology.orchestra.run.smithi114.stdout: "error_string": "", 2024-04-23T10:41:08.413 INFO:teuthology.orchestra.run.smithi114.stdout: "module_options": { 2024-04-23T10:41:08.413 INFO:teuthology.orchestra.run.smithi114.stdout: "log_level": { 2024-04-23T10:41:08.413 INFO:teuthology.orchestra.run.smithi114.stdout: "name": "log_level", 2024-04-23T10:41:08.413 INFO:teuthology.orchestra.run.smithi114.stdout: "type": "str", 2024-04-23T10:41:08.414 INFO:teuthology.orchestra.run.smithi114.stdout: "level": "advanced", 2024-04-23T10:41:08.414 INFO:teuthology.orchestra.run.smithi114.stdout: "flags": 1, 2024-04-23T10:41:08.414 INFO:teuthology.orchestra.run.smithi114.stdout: "default_value": "", 2024-04-23T10:41:08.414 INFO:teuthology.orchestra.run.smithi114.stdout: "min": "", 2024-04-23T10:41:08.414 INFO:teuthology.orchestra.run.smithi114.stdout: "max": "", 2024-04-23T10:41:08.414 INFO:teuthology.orchestra.run.smithi114.stdout: "enum_allowed": [ 2024-04-23T10:41:08.414 INFO:teuthology.orchestra.run.smithi114.stdout: "", 2024-04-23T10:41:08.414 INFO:teuthology.orchestra.run.smithi114.stdout: "critical", 2024-04-23T10:41:08.414 INFO:teuthology.orchestra.run.smithi114.stdout: "debug", 2024-04-23T10:41:08.415 INFO:teuthology.orchestra.run.smithi114.stdout: "error", 2024-04-23T10:41:08.415 INFO:teuthology.orchestra.run.smithi114.stdout: "info", 2024-04-23T10:41:08.415 INFO:teuthology.orchestra.run.smithi114.stdout: "warning" 2024-04-23T10:41:08.415 INFO:teuthology.orchestra.run.smithi114.stdout: ], 2024-04-23T10:41:08.415 INFO:teuthology.orchestra.run.smithi114.stdout: "desc": "", 2024-04-23T10:41:08.415 INFO:teuthology.orchestra.run.smithi114.stdout: "long_desc": "", 2024-04-23T10:41:08.415 INFO:teuthology.orchestra.run.smithi114.stdout: "tags": [], 2024-04-23T10:41:08.415 INFO:teuthology.orchestra.run.smithi114.stdout: "see_also": [] 2024-04-23T10:41:08.415 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-23T10:41:08.415 INFO:teuthology.orchestra.run.smithi114.stdout: "log_to_cluster": { 2024-04-23T10:41:08.416 INFO:teuthology.orchestra.run.smithi114.stdout: "name": "log_to_cluster", 2024-04-23T10:41:08.416 INFO:teuthology.orchestra.run.smithi114.stdout: "type": "bool", 2024-04-23T10:41:08.416 INFO:teuthology.orchestra.run.smithi114.stdout: "level": "advanced", 2024-04-23T10:41:08.416 INFO:teuthology.orchestra.run.smithi114.stdout: "flags": 1, 2024-04-23T10:41:08.416 INFO:teuthology.orchestra.run.smithi114.stdout: "default_value": "False", 2024-04-23T10:41:08.416 INFO:teuthology.orchestra.run.smithi114.stdout: "min": "", 2024-04-23T10:41:08.416 INFO:teuthology.orchestra.run.smithi114.stdout: "max": "", 2024-04-23T10:41:08.416 INFO:teuthology.orchestra.run.smithi114.stdout: "enum_allowed": [], 2024-04-23T10:41:08.416 INFO:teuthology.orchestra.run.smithi114.stdout: "desc": "", 2024-04-23T10:41:08.416 INFO:teuthology.orchestra.run.smithi114.stdout: "long_desc": "", 2024-04-23T10:41:08.417 INFO:teuthology.orchestra.run.smithi114.stdout: "tags": [], 2024-04-23T10:41:08.417 INFO:teuthology.orchestra.run.smithi114.stdout: "see_also": [] 2024-04-23T10:41:08.417 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-23T10:41:08.417 INFO:teuthology.orchestra.run.smithi114.stdout: "log_to_cluster_level": { 2024-04-23T10:41:08.417 INFO:teuthology.orchestra.run.smithi114.stdout: "name": "log_to_cluster_level", 2024-04-23T10:41:08.417 INFO:teuthology.orchestra.run.smithi114.stdout: "type": "str", 2024-04-23T10:41:08.417 INFO:teuthology.orchestra.run.smithi114.stdout: "level": "advanced", 2024-04-23T10:41:08.417 INFO:teuthology.orchestra.run.smithi114.stdout: "flags": 1, 2024-04-23T10:41:08.417 INFO:teuthology.orchestra.run.smithi114.stdout: "default_value": "info", 2024-04-23T10:41:08.417 INFO:teuthology.orchestra.run.smithi114.stdout: "min": "", 2024-04-23T10:41:08.418 INFO:teuthology.orchestra.run.smithi114.stdout: "max": "", 2024-04-23T10:41:08.418 INFO:teuthology.orchestra.run.smithi114.stdout: "enum_allowed": [ 2024-04-23T10:41:08.418 INFO:teuthology.orchestra.run.smithi114.stdout: "", 2024-04-23T10:41:08.418 INFO:teuthology.orchestra.run.smithi114.stdout: "critical", 2024-04-23T10:41:08.418 INFO:teuthology.orchestra.run.smithi114.stdout: "debug", 2024-04-23T10:41:08.418 INFO:teuthology.orchestra.run.smithi114.stdout: "error", 2024-04-23T10:41:08.418 INFO:teuthology.orchestra.run.smithi114.stdout: "info", 2024-04-23T10:41:08.418 INFO:teuthology.orchestra.run.smithi114.stdout: "warning" 2024-04-23T10:41:08.418 INFO:teuthology.orchestra.run.smithi114.stdout: ], 2024-04-23T10:41:08.419 INFO:teuthology.orchestra.run.smithi114.stdout: "desc": "", 2024-04-23T10:41:08.419 INFO:teuthology.orchestra.run.smithi114.stdout: "long_desc": "", 2024-04-23T10:41:08.419 INFO:teuthology.orchestra.run.smithi114.stdout: "tags": [], 2024-04-23T10:41:08.419 INFO:teuthology.orchestra.run.smithi114.stdout: "see_also": [] 2024-04-23T10:41:08.419 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-23T10:41:08.419 INFO:teuthology.orchestra.run.smithi114.stdout: "log_to_file": { 2024-04-23T10:41:08.419 INFO:teuthology.orchestra.run.smithi114.stdout: "name": "log_to_file", 2024-04-23T10:41:08.419 INFO:teuthology.orchestra.run.smithi114.stdout: "type": "bool", 2024-04-23T10:41:08.419 INFO:teuthology.orchestra.run.smithi114.stdout: "level": "advanced", 2024-04-23T10:41:08.419 INFO:teuthology.orchestra.run.smithi114.stdout: "flags": 1, 2024-04-23T10:41:08.420 INFO:teuthology.orchestra.run.smithi114.stdout: "default_value": "False", 2024-04-23T10:41:08.420 INFO:teuthology.orchestra.run.smithi114.stdout: "min": "", 2024-04-23T10:41:08.420 INFO:teuthology.orchestra.run.smithi114.stdout: "max": "", 2024-04-23T10:41:08.420 INFO:teuthology.orchestra.run.smithi114.stdout: "enum_allowed": [], 2024-04-23T10:41:08.420 INFO:teuthology.orchestra.run.smithi114.stdout: "desc": "", 2024-04-23T10:41:08.420 INFO:teuthology.orchestra.run.smithi114.stdout: "long_desc": "", 2024-04-23T10:41:08.420 INFO:teuthology.orchestra.run.smithi114.stdout: "tags": [], 2024-04-23T10:41:08.420 INFO:teuthology.orchestra.run.smithi114.stdout: "see_also": [] 2024-04-23T10:41:08.420 INFO:teuthology.orchestra.run.smithi114.stdout: } 2024-04-23T10:41:08.420 INFO:teuthology.orchestra.run.smithi114.stdout: } 2024-04-23T10:41:08.421 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-23T10:41:08.421 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-04-23T10:41:08.421 INFO:teuthology.orchestra.run.smithi114.stdout: "name": "orchestrator", 2024-04-23T10:41:08.421 INFO:teuthology.orchestra.run.smithi114.stdout: "can_run": true, 2024-04-23T10:41:08.421 INFO:teuthology.orchestra.run.smithi114.stdout: "error_string": "", 2024-04-23T10:41:08.421 INFO:teuthology.orchestra.run.smithi114.stdout: "module_options": { 2024-04-23T10:41:08.421 INFO:teuthology.orchestra.run.smithi114.stdout: "fail_fs": { 2024-04-23T10:41:08.421 INFO:teuthology.orchestra.run.smithi114.stdout: "name": "fail_fs", 2024-04-23T10:41:08.421 INFO:teuthology.orchestra.run.smithi114.stdout: "type": "bool", 2024-04-23T10:41:08.421 INFO:teuthology.orchestra.run.smithi114.stdout: "level": "advanced", 2024-04-23T10:41:08.422 INFO:teuthology.orchestra.run.smithi114.stdout: "flags": 0, 2024-04-23T10:41:08.422 INFO:teuthology.orchestra.run.smithi114.stdout: "default_value": "False", 2024-04-23T10:41:08.422 INFO:teuthology.orchestra.run.smithi114.stdout: "min": "", 2024-04-23T10:41:08.422 INFO:teuthology.orchestra.run.smithi114.stdout: "max": "", 2024-04-23T10:41:08.422 INFO:teuthology.orchestra.run.smithi114.stdout: "enum_allowed": [], 2024-04-23T10:41:08.422 INFO:teuthology.orchestra.run.smithi114.stdout: "desc": "Fail filesystem for rapid multi-rank mds upgrade", 2024-04-23T10:41:08.422 INFO:teuthology.orchestra.run.smithi114.stdout: "long_desc": "", 2024-04-23T10:41:08.422 INFO:teuthology.orchestra.run.smithi114.stdout: "tags": [], 2024-04-23T10:41:08.422 INFO:teuthology.orchestra.run.smithi114.stdout: "see_also": [] 2024-04-23T10:41:08.422 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-23T10:41:08.423 INFO:teuthology.orchestra.run.smithi114.stdout: "log_level": { 2024-04-23T10:41:08.423 INFO:teuthology.orchestra.run.smithi114.stdout: "name": "log_level", 2024-04-23T10:41:08.423 INFO:teuthology.orchestra.run.smithi114.stdout: "type": "str", 2024-04-23T10:41:08.423 INFO:teuthology.orchestra.run.smithi114.stdout: "level": "advanced", 2024-04-23T10:41:08.423 INFO:teuthology.orchestra.run.smithi114.stdout: "flags": 1, 2024-04-23T10:41:08.423 INFO:teuthology.orchestra.run.smithi114.stdout: "default_value": "", 2024-04-23T10:41:08.423 INFO:teuthology.orchestra.run.smithi114.stdout: "min": "", 2024-04-23T10:41:08.423 INFO:teuthology.orchestra.run.smithi114.stdout: "max": "", 2024-04-23T10:41:08.423 INFO:teuthology.orchestra.run.smithi114.stdout: "enum_allowed": [ 2024-04-23T10:41:08.423 INFO:teuthology.orchestra.run.smithi114.stdout: "", 2024-04-23T10:41:08.424 INFO:teuthology.orchestra.run.smithi114.stdout: "critical", 2024-04-23T10:41:08.424 INFO:teuthology.orchestra.run.smithi114.stdout: "debug", 2024-04-23T10:41:08.424 INFO:teuthology.orchestra.run.smithi114.stdout: "error", 2024-04-23T10:41:08.424 INFO:teuthology.orchestra.run.smithi114.stdout: "info", 2024-04-23T10:41:08.424 INFO:teuthology.orchestra.run.smithi114.stdout: "warning" 2024-04-23T10:41:08.424 INFO:teuthology.orchestra.run.smithi114.stdout: ], 2024-04-23T10:41:08.424 INFO:teuthology.orchestra.run.smithi114.stdout: "desc": "", 2024-04-23T10:41:08.424 INFO:teuthology.orchestra.run.smithi114.stdout: "long_desc": "", 2024-04-23T10:41:08.424 INFO:teuthology.orchestra.run.smithi114.stdout: "tags": [], 2024-04-23T10:41:08.425 INFO:teuthology.orchestra.run.smithi114.stdout: "see_also": [] 2024-04-23T10:41:08.425 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-23T10:41:08.425 INFO:teuthology.orchestra.run.smithi114.stdout: "log_to_cluster": { 2024-04-23T10:41:08.425 INFO:teuthology.orchestra.run.smithi114.stdout: "name": "log_to_cluster", 2024-04-23T10:41:08.425 INFO:teuthology.orchestra.run.smithi114.stdout: "type": "bool", 2024-04-23T10:41:08.425 INFO:teuthology.orchestra.run.smithi114.stdout: "level": "advanced", 2024-04-23T10:41:08.425 INFO:teuthology.orchestra.run.smithi114.stdout: "flags": 1, 2024-04-23T10:41:08.425 INFO:teuthology.orchestra.run.smithi114.stdout: "default_value": "False", 2024-04-23T10:41:08.425 INFO:teuthology.orchestra.run.smithi114.stdout: "min": "", 2024-04-23T10:41:08.425 INFO:teuthology.orchestra.run.smithi114.stdout: "max": "", 2024-04-23T10:41:08.426 INFO:teuthology.orchestra.run.smithi114.stdout: "enum_allowed": [], 2024-04-23T10:41:08.426 INFO:teuthology.orchestra.run.smithi114.stdout: "desc": "", 2024-04-23T10:41:08.426 INFO:teuthology.orchestra.run.smithi114.stdout: "long_desc": "", 2024-04-23T10:41:08.426 INFO:teuthology.orchestra.run.smithi114.stdout: "tags": [], 2024-04-23T10:41:08.426 INFO:teuthology.orchestra.run.smithi114.stdout: "see_also": [] 2024-04-23T10:41:08.426 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-23T10:41:08.426 INFO:teuthology.orchestra.run.smithi114.stdout: "log_to_cluster_level": { 2024-04-23T10:41:08.426 INFO:teuthology.orchestra.run.smithi114.stdout: "name": "log_to_cluster_level", 2024-04-23T10:41:08.426 INFO:teuthology.orchestra.run.smithi114.stdout: "type": "str", 2024-04-23T10:41:08.426 INFO:teuthology.orchestra.run.smithi114.stdout: "level": "advanced", 2024-04-23T10:41:08.427 INFO:teuthology.orchestra.run.smithi114.stdout: "flags": 1, 2024-04-23T10:41:08.427 INFO:teuthology.orchestra.run.smithi114.stdout: "default_value": "info", 2024-04-23T10:41:08.427 INFO:teuthology.orchestra.run.smithi114.stdout: "min": "", 2024-04-23T10:41:08.427 INFO:teuthology.orchestra.run.smithi114.stdout: "max": "", 2024-04-23T10:41:08.427 INFO:teuthology.orchestra.run.smithi114.stdout: "enum_allowed": [ 2024-04-23T10:41:08.427 INFO:teuthology.orchestra.run.smithi114.stdout: "", 2024-04-23T10:41:08.427 INFO:teuthology.orchestra.run.smithi114.stdout: "critical", 2024-04-23T10:41:08.427 INFO:teuthology.orchestra.run.smithi114.stdout: "debug", 2024-04-23T10:41:08.427 INFO:teuthology.orchestra.run.smithi114.stdout: "error", 2024-04-23T10:41:08.428 INFO:teuthology.orchestra.run.smithi114.stdout: "info", 2024-04-23T10:41:08.428 INFO:teuthology.orchestra.run.smithi114.stdout: "warning" 2024-04-23T10:41:08.428 INFO:teuthology.orchestra.run.smithi114.stdout: ], 2024-04-23T10:41:08.428 INFO:teuthology.orchestra.run.smithi114.stdout: "desc": "", 2024-04-23T10:41:08.428 INFO:teuthology.orchestra.run.smithi114.stdout: "long_desc": "", 2024-04-23T10:41:08.428 INFO:teuthology.orchestra.run.smithi114.stdout: "tags": [], 2024-04-23T10:41:08.428 INFO:teuthology.orchestra.run.smithi114.stdout: "see_also": [] 2024-04-23T10:41:08.428 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-23T10:41:08.428 INFO:teuthology.orchestra.run.smithi114.stdout: "log_to_file": { 2024-04-23T10:41:08.428 INFO:teuthology.orchestra.run.smithi114.stdout: "name": "log_to_file", 2024-04-23T10:41:08.429 INFO:teuthology.orchestra.run.smithi114.stdout: "type": "bool", 2024-04-23T10:41:08.429 INFO:teuthology.orchestra.run.smithi114.stdout: "level": "advanced", 2024-04-23T10:41:08.429 INFO:teuthology.orchestra.run.smithi114.stdout: "flags": 1, 2024-04-23T10:41:08.429 INFO:teuthology.orchestra.run.smithi114.stdout: "default_value": "False", 2024-04-23T10:41:08.429 INFO:teuthology.orchestra.run.smithi114.stdout: "min": "", 2024-04-23T10:41:08.429 INFO:teuthology.orchestra.run.smithi114.stdout: "max": "", 2024-04-23T10:41:08.429 INFO:teuthology.orchestra.run.smithi114.stdout: "enum_allowed": [], 2024-04-23T10:41:08.429 INFO:teuthology.orchestra.run.smithi114.stdout: "desc": "", 2024-04-23T10:41:08.429 INFO:teuthology.orchestra.run.smithi114.stdout: "long_desc": "", 2024-04-23T10:41:08.429 INFO:teuthology.orchestra.run.smithi114.stdout: "tags": [], 2024-04-23T10:41:08.430 INFO:teuthology.orchestra.run.smithi114.stdout: "see_also": [] 2024-04-23T10:41:08.430 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-23T10:41:08.430 INFO:teuthology.orchestra.run.smithi114.stdout: "orchestrator": { 2024-04-23T10:41:08.430 INFO:teuthology.orchestra.run.smithi114.stdout: "name": "orchestrator", 2024-04-23T10:41:08.430 INFO:teuthology.orchestra.run.smithi114.stdout: "type": "str", 2024-04-23T10:41:08.430 INFO:teuthology.orchestra.run.smithi114.stdout: "level": "advanced", 2024-04-23T10:41:08.430 INFO:teuthology.orchestra.run.smithi114.stdout: "flags": 1, 2024-04-23T10:41:08.430 INFO:teuthology.orchestra.run.smithi114.stdout: "default_value": "", 2024-04-23T10:41:08.430 INFO:teuthology.orchestra.run.smithi114.stdout: "min": "", 2024-04-23T10:41:08.430 INFO:teuthology.orchestra.run.smithi114.stdout: "max": "", 2024-04-23T10:41:08.431 INFO:teuthology.orchestra.run.smithi114.stdout: "enum_allowed": [ 2024-04-23T10:41:08.431 INFO:teuthology.orchestra.run.smithi114.stdout: "cephadm", 2024-04-23T10:41:08.431 INFO:teuthology.orchestra.run.smithi114.stdout: "rook", 2024-04-23T10:41:08.431 INFO:teuthology.orchestra.run.smithi114.stdout: "test_orchestrator" 2024-04-23T10:41:08.431 INFO:teuthology.orchestra.run.smithi114.stdout: ], 2024-04-23T10:41:08.431 INFO:teuthology.orchestra.run.smithi114.stdout: "desc": "Orchestrator backend", 2024-04-23T10:41:08.431 INFO:teuthology.orchestra.run.smithi114.stdout: "long_desc": "", 2024-04-23T10:41:08.431 INFO:teuthology.orchestra.run.smithi114.stdout: "tags": [], 2024-04-23T10:41:08.431 INFO:teuthology.orchestra.run.smithi114.stdout: "see_also": [] 2024-04-23T10:41:08.431 INFO:teuthology.orchestra.run.smithi114.stdout: } 2024-04-23T10:41:08.432 INFO:teuthology.orchestra.run.smithi114.stdout: } 2024-04-23T10:41:08.432 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-23T10:41:08.432 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-04-23T10:41:08.432 INFO:teuthology.orchestra.run.smithi114.stdout: "name": "osd_perf_query", 2024-04-23T10:41:08.432 INFO:teuthology.orchestra.run.smithi114.stdout: "can_run": true, 2024-04-23T10:41:08.432 INFO:teuthology.orchestra.run.smithi114.stdout: "error_string": "", 2024-04-23T10:41:08.432 INFO:teuthology.orchestra.run.smithi114.stdout: "module_options": { 2024-04-23T10:41:08.432 INFO:teuthology.orchestra.run.smithi114.stdout: "log_level": { 2024-04-23T10:41:08.432 INFO:teuthology.orchestra.run.smithi114.stdout: "name": "log_level", 2024-04-23T10:41:08.432 INFO:teuthology.orchestra.run.smithi114.stdout: "type": "str", 2024-04-23T10:41:08.433 INFO:teuthology.orchestra.run.smithi114.stdout: "level": "advanced", 2024-04-23T10:41:08.433 INFO:teuthology.orchestra.run.smithi114.stdout: "flags": 1, 2024-04-23T10:41:08.433 INFO:teuthology.orchestra.run.smithi114.stdout: "default_value": "", 2024-04-23T10:41:08.433 INFO:teuthology.orchestra.run.smithi114.stdout: "min": "", 2024-04-23T10:41:08.433 INFO:teuthology.orchestra.run.smithi114.stdout: "max": "", 2024-04-23T10:41:08.433 INFO:teuthology.orchestra.run.smithi114.stdout: "enum_allowed": [ 2024-04-23T10:41:08.433 INFO:teuthology.orchestra.run.smithi114.stdout: "", 2024-04-23T10:41:08.433 INFO:teuthology.orchestra.run.smithi114.stdout: "critical", 2024-04-23T10:41:08.433 INFO:teuthology.orchestra.run.smithi114.stdout: "debug", 2024-04-23T10:41:08.433 INFO:teuthology.orchestra.run.smithi114.stdout: "error", 2024-04-23T10:41:08.434 INFO:teuthology.orchestra.run.smithi114.stdout: "info", 2024-04-23T10:41:08.434 INFO:teuthology.orchestra.run.smithi114.stdout: "warning" 2024-04-23T10:41:08.434 INFO:teuthology.orchestra.run.smithi114.stdout: ], 2024-04-23T10:41:08.434 INFO:teuthology.orchestra.run.smithi114.stdout: "desc": "", 2024-04-23T10:41:08.434 INFO:teuthology.orchestra.run.smithi114.stdout: "long_desc": "", 2024-04-23T10:41:08.434 INFO:teuthology.orchestra.run.smithi114.stdout: "tags": [], 2024-04-23T10:41:08.434 INFO:teuthology.orchestra.run.smithi114.stdout: "see_also": [] 2024-04-23T10:41:08.434 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-23T10:41:08.434 INFO:teuthology.orchestra.run.smithi114.stdout: "log_to_cluster": { 2024-04-23T10:41:08.435 INFO:teuthology.orchestra.run.smithi114.stdout: "name": "log_to_cluster", 2024-04-23T10:41:08.435 INFO:teuthology.orchestra.run.smithi114.stdout: "type": "bool", 2024-04-23T10:41:08.435 INFO:teuthology.orchestra.run.smithi114.stdout: "level": "advanced", 2024-04-23T10:41:08.435 INFO:teuthology.orchestra.run.smithi114.stdout: "flags": 1, 2024-04-23T10:41:08.435 INFO:teuthology.orchestra.run.smithi114.stdout: "default_value": "False", 2024-04-23T10:41:08.435 INFO:teuthology.orchestra.run.smithi114.stdout: "min": "", 2024-04-23T10:41:08.435 INFO:teuthology.orchestra.run.smithi114.stdout: "max": "", 2024-04-23T10:41:08.435 INFO:teuthology.orchestra.run.smithi114.stdout: "enum_allowed": [], 2024-04-23T10:41:08.435 INFO:teuthology.orchestra.run.smithi114.stdout: "desc": "", 2024-04-23T10:41:08.435 INFO:teuthology.orchestra.run.smithi114.stdout: "long_desc": "", 2024-04-23T10:41:08.436 INFO:teuthology.orchestra.run.smithi114.stdout: "tags": [], 2024-04-23T10:41:08.436 INFO:teuthology.orchestra.run.smithi114.stdout: "see_also": [] 2024-04-23T10:41:08.436 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-23T10:41:08.436 INFO:teuthology.orchestra.run.smithi114.stdout: "log_to_cluster_level": { 2024-04-23T10:41:08.436 INFO:teuthology.orchestra.run.smithi114.stdout: "name": "log_to_cluster_level", 2024-04-23T10:41:08.436 INFO:teuthology.orchestra.run.smithi114.stdout: "type": "str", 2024-04-23T10:41:08.436 INFO:teuthology.orchestra.run.smithi114.stdout: "level": "advanced", 2024-04-23T10:41:08.436 INFO:teuthology.orchestra.run.smithi114.stdout: "flags": 1, 2024-04-23T10:41:08.436 INFO:teuthology.orchestra.run.smithi114.stdout: "default_value": "info", 2024-04-23T10:41:08.436 INFO:teuthology.orchestra.run.smithi114.stdout: "min": "", 2024-04-23T10:41:08.437 INFO:teuthology.orchestra.run.smithi114.stdout: "max": "", 2024-04-23T10:41:08.437 INFO:teuthology.orchestra.run.smithi114.stdout: "enum_allowed": [ 2024-04-23T10:41:08.437 INFO:teuthology.orchestra.run.smithi114.stdout: "", 2024-04-23T10:41:08.437 INFO:teuthology.orchestra.run.smithi114.stdout: "critical", 2024-04-23T10:41:08.437 INFO:teuthology.orchestra.run.smithi114.stdout: "debug", 2024-04-23T10:41:08.437 INFO:teuthology.orchestra.run.smithi114.stdout: "error", 2024-04-23T10:41:08.437 INFO:teuthology.orchestra.run.smithi114.stdout: "info", 2024-04-23T10:41:08.437 INFO:teuthology.orchestra.run.smithi114.stdout: "warning" 2024-04-23T10:41:08.437 INFO:teuthology.orchestra.run.smithi114.stdout: ], 2024-04-23T10:41:08.437 INFO:teuthology.orchestra.run.smithi114.stdout: "desc": "", 2024-04-23T10:41:08.438 INFO:teuthology.orchestra.run.smithi114.stdout: "long_desc": "", 2024-04-23T10:41:08.438 INFO:teuthology.orchestra.run.smithi114.stdout: "tags": [], 2024-04-23T10:41:08.438 INFO:teuthology.orchestra.run.smithi114.stdout: "see_also": [] 2024-04-23T10:41:08.438 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-23T10:41:08.438 INFO:teuthology.orchestra.run.smithi114.stdout: "log_to_file": { 2024-04-23T10:41:08.438 INFO:teuthology.orchestra.run.smithi114.stdout: "name": "log_to_file", 2024-04-23T10:41:08.438 INFO:teuthology.orchestra.run.smithi114.stdout: "type": "bool", 2024-04-23T10:41:08.438 INFO:teuthology.orchestra.run.smithi114.stdout: "level": "advanced", 2024-04-23T10:41:08.438 INFO:teuthology.orchestra.run.smithi114.stdout: "flags": 1, 2024-04-23T10:41:08.438 INFO:teuthology.orchestra.run.smithi114.stdout: "default_value": "False", 2024-04-23T10:41:08.439 INFO:teuthology.orchestra.run.smithi114.stdout: "min": "", 2024-04-23T10:41:08.439 INFO:teuthology.orchestra.run.smithi114.stdout: "max": "", 2024-04-23T10:41:08.439 INFO:teuthology.orchestra.run.smithi114.stdout: "enum_allowed": [], 2024-04-23T10:41:08.439 INFO:teuthology.orchestra.run.smithi114.stdout: "desc": "", 2024-04-23T10:41:08.439 INFO:teuthology.orchestra.run.smithi114.stdout: "long_desc": "", 2024-04-23T10:41:08.439 INFO:teuthology.orchestra.run.smithi114.stdout: "tags": [], 2024-04-23T10:41:08.439 INFO:teuthology.orchestra.run.smithi114.stdout: "see_also": [] 2024-04-23T10:41:08.439 INFO:teuthology.orchestra.run.smithi114.stdout: } 2024-04-23T10:41:08.439 INFO:teuthology.orchestra.run.smithi114.stdout: } 2024-04-23T10:41:08.439 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-23T10:41:08.440 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-04-23T10:41:08.440 INFO:teuthology.orchestra.run.smithi114.stdout: "name": "osd_support", 2024-04-23T10:41:08.440 INFO:teuthology.orchestra.run.smithi114.stdout: "can_run": true, 2024-04-23T10:41:08.440 INFO:teuthology.orchestra.run.smithi114.stdout: "error_string": "", 2024-04-23T10:41:08.440 INFO:teuthology.orchestra.run.smithi114.stdout: "module_options": { 2024-04-23T10:41:08.440 INFO:teuthology.orchestra.run.smithi114.stdout: "log_level": { 2024-04-23T10:41:08.440 INFO:teuthology.orchestra.run.smithi114.stdout: "name": "log_level", 2024-04-23T10:41:08.440 INFO:teuthology.orchestra.run.smithi114.stdout: "type": "str", 2024-04-23T10:41:08.440 INFO:teuthology.orchestra.run.smithi114.stdout: "level": "advanced", 2024-04-23T10:41:08.440 INFO:teuthology.orchestra.run.smithi114.stdout: "flags": 1, 2024-04-23T10:41:08.441 INFO:teuthology.orchestra.run.smithi114.stdout: "default_value": "", 2024-04-23T10:41:08.441 INFO:teuthology.orchestra.run.smithi114.stdout: "min": "", 2024-04-23T10:41:08.441 INFO:teuthology.orchestra.run.smithi114.stdout: "max": "", 2024-04-23T10:41:08.441 INFO:teuthology.orchestra.run.smithi114.stdout: "enum_allowed": [ 2024-04-23T10:41:08.441 INFO:teuthology.orchestra.run.smithi114.stdout: "", 2024-04-23T10:41:08.441 INFO:teuthology.orchestra.run.smithi114.stdout: "critical", 2024-04-23T10:41:08.441 INFO:teuthology.orchestra.run.smithi114.stdout: "debug", 2024-04-23T10:41:08.441 INFO:teuthology.orchestra.run.smithi114.stdout: "error", 2024-04-23T10:41:08.441 INFO:teuthology.orchestra.run.smithi114.stdout: "info", 2024-04-23T10:41:08.441 INFO:teuthology.orchestra.run.smithi114.stdout: "warning" 2024-04-23T10:41:08.442 INFO:teuthology.orchestra.run.smithi114.stdout: ], 2024-04-23T10:41:08.442 INFO:teuthology.orchestra.run.smithi114.stdout: "desc": "", 2024-04-23T10:41:08.442 INFO:teuthology.orchestra.run.smithi114.stdout: "long_desc": "", 2024-04-23T10:41:08.442 INFO:teuthology.orchestra.run.smithi114.stdout: "tags": [], 2024-04-23T10:41:08.442 INFO:teuthology.orchestra.run.smithi114.stdout: "see_also": [] 2024-04-23T10:41:08.442 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-23T10:41:08.442 INFO:teuthology.orchestra.run.smithi114.stdout: "log_to_cluster": { 2024-04-23T10:41:08.442 INFO:teuthology.orchestra.run.smithi114.stdout: "name": "log_to_cluster", 2024-04-23T10:41:08.443 INFO:teuthology.orchestra.run.smithi114.stdout: "type": "bool", 2024-04-23T10:41:08.443 INFO:teuthology.orchestra.run.smithi114.stdout: "level": "advanced", 2024-04-23T10:41:08.443 INFO:teuthology.orchestra.run.smithi114.stdout: "flags": 1, 2024-04-23T10:41:08.443 INFO:teuthology.orchestra.run.smithi114.stdout: "default_value": "False", 2024-04-23T10:41:08.443 INFO:teuthology.orchestra.run.smithi114.stdout: "min": "", 2024-04-23T10:41:08.443 INFO:teuthology.orchestra.run.smithi114.stdout: "max": "", 2024-04-23T10:41:08.443 INFO:teuthology.orchestra.run.smithi114.stdout: "enum_allowed": [], 2024-04-23T10:41:08.443 INFO:teuthology.orchestra.run.smithi114.stdout: "desc": "", 2024-04-23T10:41:08.443 INFO:teuthology.orchestra.run.smithi114.stdout: "long_desc": "", 2024-04-23T10:41:08.444 INFO:teuthology.orchestra.run.smithi114.stdout: "tags": [], 2024-04-23T10:41:08.444 INFO:teuthology.orchestra.run.smithi114.stdout: "see_also": [] 2024-04-23T10:41:08.444 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-23T10:41:08.444 INFO:teuthology.orchestra.run.smithi114.stdout: "log_to_cluster_level": { 2024-04-23T10:41:08.444 INFO:teuthology.orchestra.run.smithi114.stdout: "name": "log_to_cluster_level", 2024-04-23T10:41:08.444 INFO:teuthology.orchestra.run.smithi114.stdout: "type": "str", 2024-04-23T10:41:08.444 INFO:teuthology.orchestra.run.smithi114.stdout: "level": "advanced", 2024-04-23T10:41:08.444 INFO:teuthology.orchestra.run.smithi114.stdout: "flags": 1, 2024-04-23T10:41:08.444 INFO:teuthology.orchestra.run.smithi114.stdout: "default_value": "info", 2024-04-23T10:41:08.444 INFO:teuthology.orchestra.run.smithi114.stdout: "min": "", 2024-04-23T10:41:08.445 INFO:teuthology.orchestra.run.smithi114.stdout: "max": "", 2024-04-23T10:41:08.445 INFO:teuthology.orchestra.run.smithi114.stdout: "enum_allowed": [ 2024-04-23T10:41:08.445 INFO:teuthology.orchestra.run.smithi114.stdout: "", 2024-04-23T10:41:08.445 INFO:teuthology.orchestra.run.smithi114.stdout: "critical", 2024-04-23T10:41:08.445 INFO:teuthology.orchestra.run.smithi114.stdout: "debug", 2024-04-23T10:41:08.445 INFO:teuthology.orchestra.run.smithi114.stdout: "error", 2024-04-23T10:41:08.445 INFO:teuthology.orchestra.run.smithi114.stdout: "info", 2024-04-23T10:41:08.445 INFO:teuthology.orchestra.run.smithi114.stdout: "warning" 2024-04-23T10:41:08.445 INFO:teuthology.orchestra.run.smithi114.stdout: ], 2024-04-23T10:41:08.445 INFO:teuthology.orchestra.run.smithi114.stdout: "desc": "", 2024-04-23T10:41:08.446 INFO:teuthology.orchestra.run.smithi114.stdout: "long_desc": "", 2024-04-23T10:41:08.446 INFO:teuthology.orchestra.run.smithi114.stdout: "tags": [], 2024-04-23T10:41:08.446 INFO:teuthology.orchestra.run.smithi114.stdout: "see_also": [] 2024-04-23T10:41:08.446 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-23T10:41:08.446 INFO:teuthology.orchestra.run.smithi114.stdout: "log_to_file": { 2024-04-23T10:41:08.446 INFO:teuthology.orchestra.run.smithi114.stdout: "name": "log_to_file", 2024-04-23T10:41:08.446 INFO:teuthology.orchestra.run.smithi114.stdout: "type": "bool", 2024-04-23T10:41:08.446 INFO:teuthology.orchestra.run.smithi114.stdout: "level": "advanced", 2024-04-23T10:41:08.446 INFO:teuthology.orchestra.run.smithi114.stdout: "flags": 1, 2024-04-23T10:41:08.447 INFO:teuthology.orchestra.run.smithi114.stdout: "default_value": "False", 2024-04-23T10:41:08.447 INFO:teuthology.orchestra.run.smithi114.stdout: "min": "", 2024-04-23T10:41:08.447 INFO:teuthology.orchestra.run.smithi114.stdout: "max": "", 2024-04-23T10:41:08.447 INFO:teuthology.orchestra.run.smithi114.stdout: "enum_allowed": [], 2024-04-23T10:41:08.447 INFO:teuthology.orchestra.run.smithi114.stdout: "desc": "", 2024-04-23T10:41:08.447 INFO:teuthology.orchestra.run.smithi114.stdout: "long_desc": "", 2024-04-23T10:41:08.447 INFO:teuthology.orchestra.run.smithi114.stdout: "tags": [], 2024-04-23T10:41:08.447 INFO:teuthology.orchestra.run.smithi114.stdout: "see_also": [] 2024-04-23T10:41:08.447 INFO:teuthology.orchestra.run.smithi114.stdout: } 2024-04-23T10:41:08.447 INFO:teuthology.orchestra.run.smithi114.stdout: } 2024-04-23T10:41:08.448 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-23T10:41:08.448 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-04-23T10:41:08.448 INFO:teuthology.orchestra.run.smithi114.stdout: "name": "pg_autoscaler", 2024-04-23T10:41:08.448 INFO:teuthology.orchestra.run.smithi114.stdout: "can_run": true, 2024-04-23T10:41:08.448 INFO:teuthology.orchestra.run.smithi114.stdout: "error_string": "", 2024-04-23T10:41:08.448 INFO:teuthology.orchestra.run.smithi114.stdout: "module_options": { 2024-04-23T10:41:08.448 INFO:teuthology.orchestra.run.smithi114.stdout: "log_level": { 2024-04-23T10:41:08.448 INFO:teuthology.orchestra.run.smithi114.stdout: "name": "log_level", 2024-04-23T10:41:08.448 INFO:teuthology.orchestra.run.smithi114.stdout: "type": "str", 2024-04-23T10:41:08.448 INFO:teuthology.orchestra.run.smithi114.stdout: "level": "advanced", 2024-04-23T10:41:08.449 INFO:teuthology.orchestra.run.smithi114.stdout: "flags": 1, 2024-04-23T10:41:08.449 INFO:teuthology.orchestra.run.smithi114.stdout: "default_value": "", 2024-04-23T10:41:08.449 INFO:teuthology.orchestra.run.smithi114.stdout: "min": "", 2024-04-23T10:41:08.449 INFO:teuthology.orchestra.run.smithi114.stdout: "max": "", 2024-04-23T10:41:08.449 INFO:teuthology.orchestra.run.smithi114.stdout: "enum_allowed": [ 2024-04-23T10:41:08.449 INFO:teuthology.orchestra.run.smithi114.stdout: "", 2024-04-23T10:41:08.449 INFO:teuthology.orchestra.run.smithi114.stdout: "critical", 2024-04-23T10:41:08.449 INFO:teuthology.orchestra.run.smithi114.stdout: "debug", 2024-04-23T10:41:08.449 INFO:teuthology.orchestra.run.smithi114.stdout: "error", 2024-04-23T10:41:08.450 INFO:teuthology.orchestra.run.smithi114.stdout: "info", 2024-04-23T10:41:08.450 INFO:teuthology.orchestra.run.smithi114.stdout: "warning" 2024-04-23T10:41:08.450 INFO:teuthology.orchestra.run.smithi114.stdout: ], 2024-04-23T10:41:08.450 INFO:teuthology.orchestra.run.smithi114.stdout: "desc": "", 2024-04-23T10:41:08.450 INFO:teuthology.orchestra.run.smithi114.stdout: "long_desc": "", 2024-04-23T10:41:08.450 INFO:teuthology.orchestra.run.smithi114.stdout: "tags": [], 2024-04-23T10:41:08.450 INFO:teuthology.orchestra.run.smithi114.stdout: "see_also": [] 2024-04-23T10:41:08.450 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-23T10:41:08.450 INFO:teuthology.orchestra.run.smithi114.stdout: "log_to_cluster": { 2024-04-23T10:41:08.450 INFO:teuthology.orchestra.run.smithi114.stdout: "name": "log_to_cluster", 2024-04-23T10:41:08.451 INFO:teuthology.orchestra.run.smithi114.stdout: "type": "bool", 2024-04-23T10:41:08.451 INFO:teuthology.orchestra.run.smithi114.stdout: "level": "advanced", 2024-04-23T10:41:08.451 INFO:teuthology.orchestra.run.smithi114.stdout: "flags": 1, 2024-04-23T10:41:08.451 INFO:teuthology.orchestra.run.smithi114.stdout: "default_value": "False", 2024-04-23T10:41:08.451 INFO:teuthology.orchestra.run.smithi114.stdout: "min": "", 2024-04-23T10:41:08.451 INFO:teuthology.orchestra.run.smithi114.stdout: "max": "", 2024-04-23T10:41:08.451 INFO:teuthology.orchestra.run.smithi114.stdout: "enum_allowed": [], 2024-04-23T10:41:08.451 INFO:teuthology.orchestra.run.smithi114.stdout: "desc": "", 2024-04-23T10:41:08.451 INFO:teuthology.orchestra.run.smithi114.stdout: "long_desc": "", 2024-04-23T10:41:08.451 INFO:teuthology.orchestra.run.smithi114.stdout: "tags": [], 2024-04-23T10:41:08.452 INFO:teuthology.orchestra.run.smithi114.stdout: "see_also": [] 2024-04-23T10:41:08.452 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-23T10:41:08.452 INFO:teuthology.orchestra.run.smithi114.stdout: "log_to_cluster_level": { 2024-04-23T10:41:08.452 INFO:teuthology.orchestra.run.smithi114.stdout: "name": "log_to_cluster_level", 2024-04-23T10:41:08.452 INFO:teuthology.orchestra.run.smithi114.stdout: "type": "str", 2024-04-23T10:41:08.452 INFO:teuthology.orchestra.run.smithi114.stdout: "level": "advanced", 2024-04-23T10:41:08.452 INFO:teuthology.orchestra.run.smithi114.stdout: "flags": 1, 2024-04-23T10:41:08.452 INFO:teuthology.orchestra.run.smithi114.stdout: "default_value": "info", 2024-04-23T10:41:08.452 INFO:teuthology.orchestra.run.smithi114.stdout: "min": "", 2024-04-23T10:41:08.452 INFO:teuthology.orchestra.run.smithi114.stdout: "max": "", 2024-04-23T10:41:08.453 INFO:teuthology.orchestra.run.smithi114.stdout: "enum_allowed": [ 2024-04-23T10:41:08.453 INFO:teuthology.orchestra.run.smithi114.stdout: "", 2024-04-23T10:41:08.453 INFO:teuthology.orchestra.run.smithi114.stdout: "critical", 2024-04-23T10:41:08.453 INFO:teuthology.orchestra.run.smithi114.stdout: "debug", 2024-04-23T10:41:08.453 INFO:teuthology.orchestra.run.smithi114.stdout: "error", 2024-04-23T10:41:08.453 INFO:teuthology.orchestra.run.smithi114.stdout: "info", 2024-04-23T10:41:08.453 INFO:teuthology.orchestra.run.smithi114.stdout: "warning" 2024-04-23T10:41:08.453 INFO:teuthology.orchestra.run.smithi114.stdout: ], 2024-04-23T10:41:08.453 INFO:teuthology.orchestra.run.smithi114.stdout: "desc": "", 2024-04-23T10:41:08.454 INFO:teuthology.orchestra.run.smithi114.stdout: "long_desc": "", 2024-04-23T10:41:08.454 INFO:teuthology.orchestra.run.smithi114.stdout: "tags": [], 2024-04-23T10:41:08.454 INFO:teuthology.orchestra.run.smithi114.stdout: "see_also": [] 2024-04-23T10:41:08.454 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-23T10:41:08.454 INFO:teuthology.orchestra.run.smithi114.stdout: "log_to_file": { 2024-04-23T10:41:08.454 INFO:teuthology.orchestra.run.smithi114.stdout: "name": "log_to_file", 2024-04-23T10:41:08.454 INFO:teuthology.orchestra.run.smithi114.stdout: "type": "bool", 2024-04-23T10:41:08.454 INFO:teuthology.orchestra.run.smithi114.stdout: "level": "advanced", 2024-04-23T10:41:08.454 INFO:teuthology.orchestra.run.smithi114.stdout: "flags": 1, 2024-04-23T10:41:08.455 INFO:teuthology.orchestra.run.smithi114.stdout: "default_value": "False", 2024-04-23T10:41:08.455 INFO:teuthology.orchestra.run.smithi114.stdout: "min": "", 2024-04-23T10:41:08.455 INFO:teuthology.orchestra.run.smithi114.stdout: "max": "", 2024-04-23T10:41:08.455 INFO:teuthology.orchestra.run.smithi114.stdout: "enum_allowed": [], 2024-04-23T10:41:08.455 INFO:teuthology.orchestra.run.smithi114.stdout: "desc": "", 2024-04-23T10:41:08.455 INFO:teuthology.orchestra.run.smithi114.stdout: "long_desc": "", 2024-04-23T10:41:08.455 INFO:teuthology.orchestra.run.smithi114.stdout: "tags": [], 2024-04-23T10:41:08.455 INFO:teuthology.orchestra.run.smithi114.stdout: "see_also": [] 2024-04-23T10:41:08.455 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-23T10:41:08.455 INFO:teuthology.orchestra.run.smithi114.stdout: "sleep_interval": { 2024-04-23T10:41:08.456 INFO:teuthology.orchestra.run.smithi114.stdout: "name": "sleep_interval", 2024-04-23T10:41:08.456 INFO:teuthology.orchestra.run.smithi114.stdout: "type": "secs", 2024-04-23T10:41:08.456 INFO:teuthology.orchestra.run.smithi114.stdout: "level": "advanced", 2024-04-23T10:41:08.456 INFO:teuthology.orchestra.run.smithi114.stdout: "flags": 0, 2024-04-23T10:41:08.456 INFO:teuthology.orchestra.run.smithi114.stdout: "default_value": "60", 2024-04-23T10:41:08.456 INFO:teuthology.orchestra.run.smithi114.stdout: "min": "", 2024-04-23T10:41:08.456 INFO:teuthology.orchestra.run.smithi114.stdout: "max": "", 2024-04-23T10:41:08.456 INFO:teuthology.orchestra.run.smithi114.stdout: "enum_allowed": [], 2024-04-23T10:41:08.456 INFO:teuthology.orchestra.run.smithi114.stdout: "desc": "", 2024-04-23T10:41:08.456 INFO:teuthology.orchestra.run.smithi114.stdout: "long_desc": "", 2024-04-23T10:41:08.457 INFO:teuthology.orchestra.run.smithi114.stdout: "tags": [], 2024-04-23T10:41:08.457 INFO:teuthology.orchestra.run.smithi114.stdout: "see_also": [] 2024-04-23T10:41:08.457 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-23T10:41:08.457 INFO:teuthology.orchestra.run.smithi114.stdout: "threshold": { 2024-04-23T10:41:08.457 INFO:teuthology.orchestra.run.smithi114.stdout: "name": "threshold", 2024-04-23T10:41:08.457 INFO:teuthology.orchestra.run.smithi114.stdout: "type": "float", 2024-04-23T10:41:08.457 INFO:teuthology.orchestra.run.smithi114.stdout: "level": "advanced", 2024-04-23T10:41:08.457 INFO:teuthology.orchestra.run.smithi114.stdout: "flags": 0, 2024-04-23T10:41:08.457 INFO:teuthology.orchestra.run.smithi114.stdout: "default_value": "3.0", 2024-04-23T10:41:08.457 INFO:teuthology.orchestra.run.smithi114.stdout: "min": "1.0", 2024-04-23T10:41:08.458 INFO:teuthology.orchestra.run.smithi114.stdout: "max": "", 2024-04-23T10:41:08.458 INFO:teuthology.orchestra.run.smithi114.stdout: "enum_allowed": [], 2024-04-23T10:41:08.458 INFO:teuthology.orchestra.run.smithi114.stdout: "desc": "scaling threshold", 2024-04-23T10:41:08.458 INFO:teuthology.orchestra.run.smithi114.stdout: "long_desc": "The factor by which the `NEW PG_NUM` must vary from the current`PG_NUM` before being accepted. Cannot be less than 1.0", 2024-04-23T10:41:08.458 INFO:teuthology.orchestra.run.smithi114.stdout: "tags": [], 2024-04-23T10:41:08.458 INFO:teuthology.orchestra.run.smithi114.stdout: "see_also": [] 2024-04-23T10:41:08.458 INFO:teuthology.orchestra.run.smithi114.stdout: } 2024-04-23T10:41:08.458 INFO:teuthology.orchestra.run.smithi114.stdout: } 2024-04-23T10:41:08.458 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-23T10:41:08.458 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-04-23T10:41:08.459 INFO:teuthology.orchestra.run.smithi114.stdout: "name": "progress", 2024-04-23T10:41:08.459 INFO:teuthology.orchestra.run.smithi114.stdout: "can_run": true, 2024-04-23T10:41:08.459 INFO:teuthology.orchestra.run.smithi114.stdout: "error_string": "", 2024-04-23T10:41:08.459 INFO:teuthology.orchestra.run.smithi114.stdout: "module_options": { 2024-04-23T10:41:08.459 INFO:teuthology.orchestra.run.smithi114.stdout: "allow_pg_recovery_event": { 2024-04-23T10:41:08.459 INFO:teuthology.orchestra.run.smithi114.stdout: "name": "allow_pg_recovery_event", 2024-04-23T10:41:08.459 INFO:teuthology.orchestra.run.smithi114.stdout: "type": "bool", 2024-04-23T10:41:08.459 INFO:teuthology.orchestra.run.smithi114.stdout: "level": "advanced", 2024-04-23T10:41:08.459 INFO:teuthology.orchestra.run.smithi114.stdout: "flags": 1, 2024-04-23T10:41:08.459 INFO:teuthology.orchestra.run.smithi114.stdout: "default_value": "False", 2024-04-23T10:41:08.460 INFO:teuthology.orchestra.run.smithi114.stdout: "min": "", 2024-04-23T10:41:08.460 INFO:teuthology.orchestra.run.smithi114.stdout: "max": "", 2024-04-23T10:41:08.460 INFO:teuthology.orchestra.run.smithi114.stdout: "enum_allowed": [], 2024-04-23T10:41:08.460 INFO:teuthology.orchestra.run.smithi114.stdout: "desc": "allow the module to show pg recovery progress", 2024-04-23T10:41:08.460 INFO:teuthology.orchestra.run.smithi114.stdout: "long_desc": "", 2024-04-23T10:41:08.460 INFO:teuthology.orchestra.run.smithi114.stdout: "tags": [], 2024-04-23T10:41:08.460 INFO:teuthology.orchestra.run.smithi114.stdout: "see_also": [] 2024-04-23T10:41:08.460 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-23T10:41:08.460 INFO:teuthology.orchestra.run.smithi114.stdout: "enabled": { 2024-04-23T10:41:08.461 INFO:teuthology.orchestra.run.smithi114.stdout: "name": "enabled", 2024-04-23T10:41:08.461 INFO:teuthology.orchestra.run.smithi114.stdout: "type": "bool", 2024-04-23T10:41:08.461 INFO:teuthology.orchestra.run.smithi114.stdout: "level": "advanced", 2024-04-23T10:41:08.461 INFO:teuthology.orchestra.run.smithi114.stdout: "flags": 0, 2024-04-23T10:41:08.461 INFO:teuthology.orchestra.run.smithi114.stdout: "default_value": "True", 2024-04-23T10:41:08.461 INFO:teuthology.orchestra.run.smithi114.stdout: "min": "", 2024-04-23T10:41:08.461 INFO:teuthology.orchestra.run.smithi114.stdout: "max": "", 2024-04-23T10:41:08.461 INFO:teuthology.orchestra.run.smithi114.stdout: "enum_allowed": [], 2024-04-23T10:41:08.461 INFO:teuthology.orchestra.run.smithi114.stdout: "desc": "", 2024-04-23T10:41:08.461 INFO:teuthology.orchestra.run.smithi114.stdout: "long_desc": "", 2024-04-23T10:41:08.462 INFO:teuthology.orchestra.run.smithi114.stdout: "tags": [], 2024-04-23T10:41:08.462 INFO:teuthology.orchestra.run.smithi114.stdout: "see_also": [] 2024-04-23T10:41:08.462 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-23T10:41:08.462 INFO:teuthology.orchestra.run.smithi114.stdout: "log_level": { 2024-04-23T10:41:08.462 INFO:teuthology.orchestra.run.smithi114.stdout: "name": "log_level", 2024-04-23T10:41:08.462 INFO:teuthology.orchestra.run.smithi114.stdout: "type": "str", 2024-04-23T10:41:08.462 INFO:teuthology.orchestra.run.smithi114.stdout: "level": "advanced", 2024-04-23T10:41:08.462 INFO:teuthology.orchestra.run.smithi114.stdout: "flags": 1, 2024-04-23T10:41:08.462 INFO:teuthology.orchestra.run.smithi114.stdout: "default_value": "", 2024-04-23T10:41:08.462 INFO:teuthology.orchestra.run.smithi114.stdout: "min": "", 2024-04-23T10:41:08.463 INFO:teuthology.orchestra.run.smithi114.stdout: "max": "", 2024-04-23T10:41:08.463 INFO:teuthology.orchestra.run.smithi114.stdout: "enum_allowed": [ 2024-04-23T10:41:08.463 INFO:teuthology.orchestra.run.smithi114.stdout: "", 2024-04-23T10:41:08.463 INFO:teuthology.orchestra.run.smithi114.stdout: "critical", 2024-04-23T10:41:08.463 INFO:teuthology.orchestra.run.smithi114.stdout: "debug", 2024-04-23T10:41:08.463 INFO:teuthology.orchestra.run.smithi114.stdout: "error", 2024-04-23T10:41:08.463 INFO:teuthology.orchestra.run.smithi114.stdout: "info", 2024-04-23T10:41:08.463 INFO:teuthology.orchestra.run.smithi114.stdout: "warning" 2024-04-23T10:41:08.463 INFO:teuthology.orchestra.run.smithi114.stdout: ], 2024-04-23T10:41:08.463 INFO:teuthology.orchestra.run.smithi114.stdout: "desc": "", 2024-04-23T10:41:08.464 INFO:teuthology.orchestra.run.smithi114.stdout: "long_desc": "", 2024-04-23T10:41:08.464 INFO:teuthology.orchestra.run.smithi114.stdout: "tags": [], 2024-04-23T10:41:08.464 INFO:teuthology.orchestra.run.smithi114.stdout: "see_also": [] 2024-04-23T10:41:08.464 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-23T10:41:08.464 INFO:teuthology.orchestra.run.smithi114.stdout: "log_to_cluster": { 2024-04-23T10:41:08.464 INFO:teuthology.orchestra.run.smithi114.stdout: "name": "log_to_cluster", 2024-04-23T10:41:08.464 INFO:teuthology.orchestra.run.smithi114.stdout: "type": "bool", 2024-04-23T10:41:08.464 INFO:teuthology.orchestra.run.smithi114.stdout: "level": "advanced", 2024-04-23T10:41:08.464 INFO:teuthology.orchestra.run.smithi114.stdout: "flags": 1, 2024-04-23T10:41:08.464 INFO:teuthology.orchestra.run.smithi114.stdout: "default_value": "False", 2024-04-23T10:41:08.465 INFO:teuthology.orchestra.run.smithi114.stdout: "min": "", 2024-04-23T10:41:08.465 INFO:teuthology.orchestra.run.smithi114.stdout: "max": "", 2024-04-23T10:41:08.465 INFO:teuthology.orchestra.run.smithi114.stdout: "enum_allowed": [], 2024-04-23T10:41:08.465 INFO:teuthology.orchestra.run.smithi114.stdout: "desc": "", 2024-04-23T10:41:08.465 INFO:teuthology.orchestra.run.smithi114.stdout: "long_desc": "", 2024-04-23T10:41:08.465 INFO:teuthology.orchestra.run.smithi114.stdout: "tags": [], 2024-04-23T10:41:08.465 INFO:teuthology.orchestra.run.smithi114.stdout: "see_also": [] 2024-04-23T10:41:08.465 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-23T10:41:08.465 INFO:teuthology.orchestra.run.smithi114.stdout: "log_to_cluster_level": { 2024-04-23T10:41:08.465 INFO:teuthology.orchestra.run.smithi114.stdout: "name": "log_to_cluster_level", 2024-04-23T10:41:08.466 INFO:teuthology.orchestra.run.smithi114.stdout: "type": "str", 2024-04-23T10:41:08.466 INFO:teuthology.orchestra.run.smithi114.stdout: "level": "advanced", 2024-04-23T10:41:08.466 INFO:teuthology.orchestra.run.smithi114.stdout: "flags": 1, 2024-04-23T10:41:08.466 INFO:teuthology.orchestra.run.smithi114.stdout: "default_value": "info", 2024-04-23T10:41:08.466 INFO:teuthology.orchestra.run.smithi114.stdout: "min": "", 2024-04-23T10:41:08.466 INFO:teuthology.orchestra.run.smithi114.stdout: "max": "", 2024-04-23T10:41:08.466 INFO:teuthology.orchestra.run.smithi114.stdout: "enum_allowed": [ 2024-04-23T10:41:08.466 INFO:teuthology.orchestra.run.smithi114.stdout: "", 2024-04-23T10:41:08.466 INFO:teuthology.orchestra.run.smithi114.stdout: "critical", 2024-04-23T10:41:08.466 INFO:teuthology.orchestra.run.smithi114.stdout: "debug", 2024-04-23T10:41:08.467 INFO:teuthology.orchestra.run.smithi114.stdout: "error", 2024-04-23T10:41:08.467 INFO:teuthology.orchestra.run.smithi114.stdout: "info", 2024-04-23T10:41:08.467 INFO:teuthology.orchestra.run.smithi114.stdout: "warning" 2024-04-23T10:41:08.467 INFO:teuthology.orchestra.run.smithi114.stdout: ], 2024-04-23T10:41:08.467 INFO:teuthology.orchestra.run.smithi114.stdout: "desc": "", 2024-04-23T10:41:08.467 INFO:teuthology.orchestra.run.smithi114.stdout: "long_desc": "", 2024-04-23T10:41:08.467 INFO:teuthology.orchestra.run.smithi114.stdout: "tags": [], 2024-04-23T10:41:08.467 INFO:teuthology.orchestra.run.smithi114.stdout: "see_also": [] 2024-04-23T10:41:08.467 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-23T10:41:08.468 INFO:teuthology.orchestra.run.smithi114.stdout: "log_to_file": { 2024-04-23T10:41:08.468 INFO:teuthology.orchestra.run.smithi114.stdout: "name": "log_to_file", 2024-04-23T10:41:08.468 INFO:teuthology.orchestra.run.smithi114.stdout: "type": "bool", 2024-04-23T10:41:08.468 INFO:teuthology.orchestra.run.smithi114.stdout: "level": "advanced", 2024-04-23T10:41:08.468 INFO:teuthology.orchestra.run.smithi114.stdout: "flags": 1, 2024-04-23T10:41:08.468 INFO:teuthology.orchestra.run.smithi114.stdout: "default_value": "False", 2024-04-23T10:41:08.468 INFO:teuthology.orchestra.run.smithi114.stdout: "min": "", 2024-04-23T10:41:08.468 INFO:teuthology.orchestra.run.smithi114.stdout: "max": "", 2024-04-23T10:41:08.468 INFO:teuthology.orchestra.run.smithi114.stdout: "enum_allowed": [], 2024-04-23T10:41:08.468 INFO:teuthology.orchestra.run.smithi114.stdout: "desc": "", 2024-04-23T10:41:08.469 INFO:teuthology.orchestra.run.smithi114.stdout: "long_desc": "", 2024-04-23T10:41:08.469 INFO:teuthology.orchestra.run.smithi114.stdout: "tags": [], 2024-04-23T10:41:08.469 INFO:teuthology.orchestra.run.smithi114.stdout: "see_also": [] 2024-04-23T10:41:08.469 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-23T10:41:08.469 INFO:teuthology.orchestra.run.smithi114.stdout: "max_completed_events": { 2024-04-23T10:41:08.469 INFO:teuthology.orchestra.run.smithi114.stdout: "name": "max_completed_events", 2024-04-23T10:41:08.469 INFO:teuthology.orchestra.run.smithi114.stdout: "type": "int", 2024-04-23T10:41:08.469 INFO:teuthology.orchestra.run.smithi114.stdout: "level": "advanced", 2024-04-23T10:41:08.469 INFO:teuthology.orchestra.run.smithi114.stdout: "flags": 1, 2024-04-23T10:41:08.469 INFO:teuthology.orchestra.run.smithi114.stdout: "default_value": "50", 2024-04-23T10:41:08.470 INFO:teuthology.orchestra.run.smithi114.stdout: "min": "", 2024-04-23T10:41:08.470 INFO:teuthology.orchestra.run.smithi114.stdout: "max": "", 2024-04-23T10:41:08.470 INFO:teuthology.orchestra.run.smithi114.stdout: "enum_allowed": [], 2024-04-23T10:41:08.470 INFO:teuthology.orchestra.run.smithi114.stdout: "desc": "number of past completed events to remember", 2024-04-23T10:41:08.470 INFO:teuthology.orchestra.run.smithi114.stdout: "long_desc": "", 2024-04-23T10:41:08.470 INFO:teuthology.orchestra.run.smithi114.stdout: "tags": [], 2024-04-23T10:41:08.470 INFO:teuthology.orchestra.run.smithi114.stdout: "see_also": [] 2024-04-23T10:41:08.470 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-23T10:41:08.470 INFO:teuthology.orchestra.run.smithi114.stdout: "sleep_interval": { 2024-04-23T10:41:08.471 INFO:teuthology.orchestra.run.smithi114.stdout: "name": "sleep_interval", 2024-04-23T10:41:08.471 INFO:teuthology.orchestra.run.smithi114.stdout: "type": "secs", 2024-04-23T10:41:08.471 INFO:teuthology.orchestra.run.smithi114.stdout: "level": "advanced", 2024-04-23T10:41:08.471 INFO:teuthology.orchestra.run.smithi114.stdout: "flags": 1, 2024-04-23T10:41:08.471 INFO:teuthology.orchestra.run.smithi114.stdout: "default_value": "5", 2024-04-23T10:41:08.471 INFO:teuthology.orchestra.run.smithi114.stdout: "min": "", 2024-04-23T10:41:08.471 INFO:teuthology.orchestra.run.smithi114.stdout: "max": "", 2024-04-23T10:41:08.471 INFO:teuthology.orchestra.run.smithi114.stdout: "enum_allowed": [], 2024-04-23T10:41:08.471 INFO:teuthology.orchestra.run.smithi114.stdout: "desc": "how long the module is going to sleep", 2024-04-23T10:41:08.471 INFO:teuthology.orchestra.run.smithi114.stdout: "long_desc": "", 2024-04-23T10:41:08.472 INFO:teuthology.orchestra.run.smithi114.stdout: "tags": [], 2024-04-23T10:41:08.472 INFO:teuthology.orchestra.run.smithi114.stdout: "see_also": [] 2024-04-23T10:41:08.472 INFO:teuthology.orchestra.run.smithi114.stdout: } 2024-04-23T10:41:08.472 INFO:teuthology.orchestra.run.smithi114.stdout: } 2024-04-23T10:41:08.472 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-23T10:41:08.472 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-04-23T10:41:08.472 INFO:teuthology.orchestra.run.smithi114.stdout: "name": "prometheus", 2024-04-23T10:41:08.472 INFO:teuthology.orchestra.run.smithi114.stdout: "can_run": true, 2024-04-23T10:41:08.472 INFO:teuthology.orchestra.run.smithi114.stdout: "error_string": "", 2024-04-23T10:41:08.473 INFO:teuthology.orchestra.run.smithi114.stdout: "module_options": { 2024-04-23T10:41:08.473 INFO:teuthology.orchestra.run.smithi114.stdout: "cache": { 2024-04-23T10:41:08.473 INFO:teuthology.orchestra.run.smithi114.stdout: "name": "cache", 2024-04-23T10:41:08.473 INFO:teuthology.orchestra.run.smithi114.stdout: "type": "bool", 2024-04-23T10:41:08.473 INFO:teuthology.orchestra.run.smithi114.stdout: "level": "advanced", 2024-04-23T10:41:08.473 INFO:teuthology.orchestra.run.smithi114.stdout: "flags": 0, 2024-04-23T10:41:08.473 INFO:teuthology.orchestra.run.smithi114.stdout: "default_value": "True", 2024-04-23T10:41:08.473 INFO:teuthology.orchestra.run.smithi114.stdout: "min": "", 2024-04-23T10:41:08.473 INFO:teuthology.orchestra.run.smithi114.stdout: "max": "", 2024-04-23T10:41:08.473 INFO:teuthology.orchestra.run.smithi114.stdout: "enum_allowed": [], 2024-04-23T10:41:08.474 INFO:teuthology.orchestra.run.smithi114.stdout: "desc": "", 2024-04-23T10:41:08.474 INFO:teuthology.orchestra.run.smithi114.stdout: "long_desc": "", 2024-04-23T10:41:08.474 INFO:teuthology.orchestra.run.smithi114.stdout: "tags": [], 2024-04-23T10:41:08.474 INFO:teuthology.orchestra.run.smithi114.stdout: "see_also": [] 2024-04-23T10:41:08.474 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-23T10:41:08.474 INFO:teuthology.orchestra.run.smithi114.stdout: "exclude_perf_counters": { 2024-04-23T10:41:08.474 INFO:teuthology.orchestra.run.smithi114.stdout: "name": "exclude_perf_counters", 2024-04-23T10:41:08.474 INFO:teuthology.orchestra.run.smithi114.stdout: "type": "bool", 2024-04-23T10:41:08.474 INFO:teuthology.orchestra.run.smithi114.stdout: "level": "advanced", 2024-04-23T10:41:08.475 INFO:teuthology.orchestra.run.smithi114.stdout: "flags": 1, 2024-04-23T10:41:08.475 INFO:teuthology.orchestra.run.smithi114.stdout: "default_value": "True", 2024-04-23T10:41:08.475 INFO:teuthology.orchestra.run.smithi114.stdout: "min": "", 2024-04-23T10:41:08.475 INFO:teuthology.orchestra.run.smithi114.stdout: "max": "", 2024-04-23T10:41:08.475 INFO:teuthology.orchestra.run.smithi114.stdout: "enum_allowed": [], 2024-04-23T10:41:08.475 INFO:teuthology.orchestra.run.smithi114.stdout: "desc": "Do not include perf-counters in the metrics output", 2024-04-23T10:41:08.475 INFO:teuthology.orchestra.run.smithi114.stdout: "long_desc": "Gathering perf-counters from a single Prometheus exporter can degrade ceph-mgr performance, especially in large clusters. Instead, Ceph-exporter daemons are now used by default for perf-counter gathering. This should only be disabled when no ceph-exporters are deployed.", 2024-04-23T10:41:08.475 INFO:teuthology.orchestra.run.smithi114.stdout: "tags": [], 2024-04-23T10:41:08.475 INFO:teuthology.orchestra.run.smithi114.stdout: "see_also": [] 2024-04-23T10:41:08.475 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-23T10:41:08.476 INFO:teuthology.orchestra.run.smithi114.stdout: "log_level": { 2024-04-23T10:41:08.476 INFO:teuthology.orchestra.run.smithi114.stdout: "name": "log_level", 2024-04-23T10:41:08.476 INFO:teuthology.orchestra.run.smithi114.stdout: "type": "str", 2024-04-23T10:41:08.476 INFO:teuthology.orchestra.run.smithi114.stdout: "level": "advanced", 2024-04-23T10:41:08.476 INFO:teuthology.orchestra.run.smithi114.stdout: "flags": 1, 2024-04-23T10:41:08.476 INFO:teuthology.orchestra.run.smithi114.stdout: "default_value": "", 2024-04-23T10:41:08.476 INFO:teuthology.orchestra.run.smithi114.stdout: "min": "", 2024-04-23T10:41:08.476 INFO:teuthology.orchestra.run.smithi114.stdout: "max": "", 2024-04-23T10:41:08.476 INFO:teuthology.orchestra.run.smithi114.stdout: "enum_allowed": [ 2024-04-23T10:41:08.477 INFO:teuthology.orchestra.run.smithi114.stdout: "", 2024-04-23T10:41:08.477 INFO:teuthology.orchestra.run.smithi114.stdout: "critical", 2024-04-23T10:41:08.477 INFO:teuthology.orchestra.run.smithi114.stdout: "debug", 2024-04-23T10:41:08.477 INFO:teuthology.orchestra.run.smithi114.stdout: "error", 2024-04-23T10:41:08.477 INFO:teuthology.orchestra.run.smithi114.stdout: "info", 2024-04-23T10:41:08.477 INFO:teuthology.orchestra.run.smithi114.stdout: "warning" 2024-04-23T10:41:08.477 INFO:teuthology.orchestra.run.smithi114.stdout: ], 2024-04-23T10:41:08.477 INFO:teuthology.orchestra.run.smithi114.stdout: "desc": "", 2024-04-23T10:41:08.477 INFO:teuthology.orchestra.run.smithi114.stdout: "long_desc": "", 2024-04-23T10:41:08.477 INFO:teuthology.orchestra.run.smithi114.stdout: "tags": [], 2024-04-23T10:41:08.478 INFO:teuthology.orchestra.run.smithi114.stdout: "see_also": [] 2024-04-23T10:41:08.478 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-23T10:41:08.478 INFO:teuthology.orchestra.run.smithi114.stdout: "log_to_cluster": { 2024-04-23T10:41:08.478 INFO:teuthology.orchestra.run.smithi114.stdout: "name": "log_to_cluster", 2024-04-23T10:41:08.478 INFO:teuthology.orchestra.run.smithi114.stdout: "type": "bool", 2024-04-23T10:41:08.478 INFO:teuthology.orchestra.run.smithi114.stdout: "level": "advanced", 2024-04-23T10:41:08.478 INFO:teuthology.orchestra.run.smithi114.stdout: "flags": 1, 2024-04-23T10:41:08.478 INFO:teuthology.orchestra.run.smithi114.stdout: "default_value": "False", 2024-04-23T10:41:08.478 INFO:teuthology.orchestra.run.smithi114.stdout: "min": "", 2024-04-23T10:41:08.478 INFO:teuthology.orchestra.run.smithi114.stdout: "max": "", 2024-04-23T10:41:08.479 INFO:teuthology.orchestra.run.smithi114.stdout: "enum_allowed": [], 2024-04-23T10:41:08.479 INFO:teuthology.orchestra.run.smithi114.stdout: "desc": "", 2024-04-23T10:41:08.479 INFO:teuthology.orchestra.run.smithi114.stdout: "long_desc": "", 2024-04-23T10:41:08.479 INFO:teuthology.orchestra.run.smithi114.stdout: "tags": [], 2024-04-23T10:41:08.479 INFO:teuthology.orchestra.run.smithi114.stdout: "see_also": [] 2024-04-23T10:41:08.479 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-23T10:41:08.479 INFO:teuthology.orchestra.run.smithi114.stdout: "log_to_cluster_level": { 2024-04-23T10:41:08.479 INFO:teuthology.orchestra.run.smithi114.stdout: "name": "log_to_cluster_level", 2024-04-23T10:41:08.479 INFO:teuthology.orchestra.run.smithi114.stdout: "type": "str", 2024-04-23T10:41:08.479 INFO:teuthology.orchestra.run.smithi114.stdout: "level": "advanced", 2024-04-23T10:41:08.480 INFO:teuthology.orchestra.run.smithi114.stdout: "flags": 1, 2024-04-23T10:41:08.480 INFO:teuthology.orchestra.run.smithi114.stdout: "default_value": "info", 2024-04-23T10:41:08.480 INFO:teuthology.orchestra.run.smithi114.stdout: "min": "", 2024-04-23T10:41:08.480 INFO:teuthology.orchestra.run.smithi114.stdout: "max": "", 2024-04-23T10:41:08.480 INFO:teuthology.orchestra.run.smithi114.stdout: "enum_allowed": [ 2024-04-23T10:41:08.480 INFO:teuthology.orchestra.run.smithi114.stdout: "", 2024-04-23T10:41:08.480 INFO:teuthology.orchestra.run.smithi114.stdout: "critical", 2024-04-23T10:41:08.480 INFO:teuthology.orchestra.run.smithi114.stdout: "debug", 2024-04-23T10:41:08.480 INFO:teuthology.orchestra.run.smithi114.stdout: "error", 2024-04-23T10:41:08.480 INFO:teuthology.orchestra.run.smithi114.stdout: "info", 2024-04-23T10:41:08.481 INFO:teuthology.orchestra.run.smithi114.stdout: "warning" 2024-04-23T10:41:08.481 INFO:teuthology.orchestra.run.smithi114.stdout: ], 2024-04-23T10:41:08.481 INFO:teuthology.orchestra.run.smithi114.stdout: "desc": "", 2024-04-23T10:41:08.481 INFO:teuthology.orchestra.run.smithi114.stdout: "long_desc": "", 2024-04-23T10:41:08.481 INFO:teuthology.orchestra.run.smithi114.stdout: "tags": [], 2024-04-23T10:41:08.481 INFO:teuthology.orchestra.run.smithi114.stdout: "see_also": [] 2024-04-23T10:41:08.481 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-23T10:41:08.481 INFO:teuthology.orchestra.run.smithi114.stdout: "log_to_file": { 2024-04-23T10:41:08.481 INFO:teuthology.orchestra.run.smithi114.stdout: "name": "log_to_file", 2024-04-23T10:41:08.481 INFO:teuthology.orchestra.run.smithi114.stdout: "type": "bool", 2024-04-23T10:41:08.482 INFO:teuthology.orchestra.run.smithi114.stdout: "level": "advanced", 2024-04-23T10:41:08.482 INFO:teuthology.orchestra.run.smithi114.stdout: "flags": 1, 2024-04-23T10:41:08.482 INFO:teuthology.orchestra.run.smithi114.stdout: "default_value": "False", 2024-04-23T10:41:08.482 INFO:teuthology.orchestra.run.smithi114.stdout: "min": "", 2024-04-23T10:41:08.482 INFO:teuthology.orchestra.run.smithi114.stdout: "max": "", 2024-04-23T10:41:08.482 INFO:teuthology.orchestra.run.smithi114.stdout: "enum_allowed": [], 2024-04-23T10:41:08.482 INFO:teuthology.orchestra.run.smithi114.stdout: "desc": "", 2024-04-23T10:41:08.482 INFO:teuthology.orchestra.run.smithi114.stdout: "long_desc": "", 2024-04-23T10:41:08.482 INFO:teuthology.orchestra.run.smithi114.stdout: "tags": [], 2024-04-23T10:41:08.482 INFO:teuthology.orchestra.run.smithi114.stdout: "see_also": [] 2024-04-23T10:41:08.483 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-23T10:41:08.483 INFO:teuthology.orchestra.run.smithi114.stdout: "rbd_stats_pools": { 2024-04-23T10:41:08.483 INFO:teuthology.orchestra.run.smithi114.stdout: "name": "rbd_stats_pools", 2024-04-23T10:41:08.483 INFO:teuthology.orchestra.run.smithi114.stdout: "type": "str", 2024-04-23T10:41:08.483 INFO:teuthology.orchestra.run.smithi114.stdout: "level": "advanced", 2024-04-23T10:41:08.483 INFO:teuthology.orchestra.run.smithi114.stdout: "flags": 0, 2024-04-23T10:41:08.483 INFO:teuthology.orchestra.run.smithi114.stdout: "default_value": "", 2024-04-23T10:41:08.483 INFO:teuthology.orchestra.run.smithi114.stdout: "min": "", 2024-04-23T10:41:08.483 INFO:teuthology.orchestra.run.smithi114.stdout: "max": "", 2024-04-23T10:41:08.483 INFO:teuthology.orchestra.run.smithi114.stdout: "enum_allowed": [], 2024-04-23T10:41:08.484 INFO:teuthology.orchestra.run.smithi114.stdout: "desc": "", 2024-04-23T10:41:08.484 INFO:teuthology.orchestra.run.smithi114.stdout: "long_desc": "", 2024-04-23T10:41:08.484 INFO:teuthology.orchestra.run.smithi114.stdout: "tags": [], 2024-04-23T10:41:08.484 INFO:teuthology.orchestra.run.smithi114.stdout: "see_also": [] 2024-04-23T10:41:08.484 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-23T10:41:08.484 INFO:teuthology.orchestra.run.smithi114.stdout: "rbd_stats_pools_refresh_interval": { 2024-04-23T10:41:08.484 INFO:teuthology.orchestra.run.smithi114.stdout: "name": "rbd_stats_pools_refresh_interval", 2024-04-23T10:41:08.484 INFO:teuthology.orchestra.run.smithi114.stdout: "type": "int", 2024-04-23T10:41:08.484 INFO:teuthology.orchestra.run.smithi114.stdout: "level": "advanced", 2024-04-23T10:41:08.484 INFO:teuthology.orchestra.run.smithi114.stdout: "flags": 0, 2024-04-23T10:41:08.485 INFO:teuthology.orchestra.run.smithi114.stdout: "default_value": "300", 2024-04-23T10:41:08.485 INFO:teuthology.orchestra.run.smithi114.stdout: "min": "", 2024-04-23T10:41:08.485 INFO:teuthology.orchestra.run.smithi114.stdout: "max": "", 2024-04-23T10:41:08.485 INFO:teuthology.orchestra.run.smithi114.stdout: "enum_allowed": [], 2024-04-23T10:41:08.485 INFO:teuthology.orchestra.run.smithi114.stdout: "desc": "", 2024-04-23T10:41:08.485 INFO:teuthology.orchestra.run.smithi114.stdout: "long_desc": "", 2024-04-23T10:41:08.485 INFO:teuthology.orchestra.run.smithi114.stdout: "tags": [], 2024-04-23T10:41:08.485 INFO:teuthology.orchestra.run.smithi114.stdout: "see_also": [] 2024-04-23T10:41:08.485 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-23T10:41:08.485 INFO:teuthology.orchestra.run.smithi114.stdout: "scrape_interval": { 2024-04-23T10:41:08.486 INFO:teuthology.orchestra.run.smithi114.stdout: "name": "scrape_interval", 2024-04-23T10:41:08.486 INFO:teuthology.orchestra.run.smithi114.stdout: "type": "float", 2024-04-23T10:41:08.486 INFO:teuthology.orchestra.run.smithi114.stdout: "level": "advanced", 2024-04-23T10:41:08.486 INFO:teuthology.orchestra.run.smithi114.stdout: "flags": 0, 2024-04-23T10:41:08.486 INFO:teuthology.orchestra.run.smithi114.stdout: "default_value": "15.0", 2024-04-23T10:41:08.486 INFO:teuthology.orchestra.run.smithi114.stdout: "min": "", 2024-04-23T10:41:08.486 INFO:teuthology.orchestra.run.smithi114.stdout: "max": "", 2024-04-23T10:41:08.486 INFO:teuthology.orchestra.run.smithi114.stdout: "enum_allowed": [], 2024-04-23T10:41:08.486 INFO:teuthology.orchestra.run.smithi114.stdout: "desc": "", 2024-04-23T10:41:08.487 INFO:teuthology.orchestra.run.smithi114.stdout: "long_desc": "", 2024-04-23T10:41:08.487 INFO:teuthology.orchestra.run.smithi114.stdout: "tags": [], 2024-04-23T10:41:08.487 INFO:teuthology.orchestra.run.smithi114.stdout: "see_also": [] 2024-04-23T10:41:08.487 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-23T10:41:08.487 INFO:teuthology.orchestra.run.smithi114.stdout: "server_addr": { 2024-04-23T10:41:08.487 INFO:teuthology.orchestra.run.smithi114.stdout: "name": "server_addr", 2024-04-23T10:41:08.487 INFO:teuthology.orchestra.run.smithi114.stdout: "type": "str", 2024-04-23T10:41:08.487 INFO:teuthology.orchestra.run.smithi114.stdout: "level": "advanced", 2024-04-23T10:41:08.487 INFO:teuthology.orchestra.run.smithi114.stdout: "flags": 0, 2024-04-23T10:41:08.487 INFO:teuthology.orchestra.run.smithi114.stdout: "default_value": "::", 2024-04-23T10:41:08.488 INFO:teuthology.orchestra.run.smithi114.stdout: "min": "", 2024-04-23T10:41:08.488 INFO:teuthology.orchestra.run.smithi114.stdout: "max": "", 2024-04-23T10:41:08.488 INFO:teuthology.orchestra.run.smithi114.stdout: "enum_allowed": [], 2024-04-23T10:41:08.488 INFO:teuthology.orchestra.run.smithi114.stdout: "desc": "the IPv4 or IPv6 address on which the module listens for HTTP requests", 2024-04-23T10:41:08.488 INFO:teuthology.orchestra.run.smithi114.stdout: "long_desc": "", 2024-04-23T10:41:08.488 INFO:teuthology.orchestra.run.smithi114.stdout: "tags": [], 2024-04-23T10:41:08.488 INFO:teuthology.orchestra.run.smithi114.stdout: "see_also": [] 2024-04-23T10:41:08.488 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-23T10:41:08.488 INFO:teuthology.orchestra.run.smithi114.stdout: "server_port": { 2024-04-23T10:41:08.488 INFO:teuthology.orchestra.run.smithi114.stdout: "name": "server_port", 2024-04-23T10:41:08.489 INFO:teuthology.orchestra.run.smithi114.stdout: "type": "int", 2024-04-23T10:41:08.489 INFO:teuthology.orchestra.run.smithi114.stdout: "level": "advanced", 2024-04-23T10:41:08.489 INFO:teuthology.orchestra.run.smithi114.stdout: "flags": 1, 2024-04-23T10:41:08.489 INFO:teuthology.orchestra.run.smithi114.stdout: "default_value": "9283", 2024-04-23T10:41:08.489 INFO:teuthology.orchestra.run.smithi114.stdout: "min": "", 2024-04-23T10:41:08.489 INFO:teuthology.orchestra.run.smithi114.stdout: "max": "", 2024-04-23T10:41:08.489 INFO:teuthology.orchestra.run.smithi114.stdout: "enum_allowed": [], 2024-04-23T10:41:08.489 INFO:teuthology.orchestra.run.smithi114.stdout: "desc": "the port on which the module listens for HTTP requests", 2024-04-23T10:41:08.489 INFO:teuthology.orchestra.run.smithi114.stdout: "long_desc": "", 2024-04-23T10:41:08.489 INFO:teuthology.orchestra.run.smithi114.stdout: "tags": [], 2024-04-23T10:41:08.490 INFO:teuthology.orchestra.run.smithi114.stdout: "see_also": [] 2024-04-23T10:41:08.490 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-23T10:41:08.490 INFO:teuthology.orchestra.run.smithi114.stdout: "stale_cache_strategy": { 2024-04-23T10:41:08.490 INFO:teuthology.orchestra.run.smithi114.stdout: "name": "stale_cache_strategy", 2024-04-23T10:41:08.490 INFO:teuthology.orchestra.run.smithi114.stdout: "type": "str", 2024-04-23T10:41:08.490 INFO:teuthology.orchestra.run.smithi114.stdout: "level": "advanced", 2024-04-23T10:41:08.490 INFO:teuthology.orchestra.run.smithi114.stdout: "flags": 0, 2024-04-23T10:41:08.490 INFO:teuthology.orchestra.run.smithi114.stdout: "default_value": "log", 2024-04-23T10:41:08.490 INFO:teuthology.orchestra.run.smithi114.stdout: "min": "", 2024-04-23T10:41:08.491 INFO:teuthology.orchestra.run.smithi114.stdout: "max": "", 2024-04-23T10:41:08.491 INFO:teuthology.orchestra.run.smithi114.stdout: "enum_allowed": [], 2024-04-23T10:41:08.491 INFO:teuthology.orchestra.run.smithi114.stdout: "desc": "", 2024-04-23T10:41:08.491 INFO:teuthology.orchestra.run.smithi114.stdout: "long_desc": "", 2024-04-23T10:41:08.491 INFO:teuthology.orchestra.run.smithi114.stdout: "tags": [], 2024-04-23T10:41:08.491 INFO:teuthology.orchestra.run.smithi114.stdout: "see_also": [] 2024-04-23T10:41:08.491 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-23T10:41:08.491 INFO:teuthology.orchestra.run.smithi114.stdout: "standby_behaviour": { 2024-04-23T10:41:08.491 INFO:teuthology.orchestra.run.smithi114.stdout: "name": "standby_behaviour", 2024-04-23T10:41:08.491 INFO:teuthology.orchestra.run.smithi114.stdout: "type": "str", 2024-04-23T10:41:08.492 INFO:teuthology.orchestra.run.smithi114.stdout: "level": "advanced", 2024-04-23T10:41:08.492 INFO:teuthology.orchestra.run.smithi114.stdout: "flags": 1, 2024-04-23T10:41:08.492 INFO:teuthology.orchestra.run.smithi114.stdout: "default_value": "default", 2024-04-23T10:41:08.492 INFO:teuthology.orchestra.run.smithi114.stdout: "min": "", 2024-04-23T10:41:08.492 INFO:teuthology.orchestra.run.smithi114.stdout: "max": "", 2024-04-23T10:41:08.492 INFO:teuthology.orchestra.run.smithi114.stdout: "enum_allowed": [ 2024-04-23T10:41:08.492 INFO:teuthology.orchestra.run.smithi114.stdout: "default", 2024-04-23T10:41:08.492 INFO:teuthology.orchestra.run.smithi114.stdout: "error" 2024-04-23T10:41:08.492 INFO:teuthology.orchestra.run.smithi114.stdout: ], 2024-04-23T10:41:08.492 INFO:teuthology.orchestra.run.smithi114.stdout: "desc": "", 2024-04-23T10:41:08.493 INFO:teuthology.orchestra.run.smithi114.stdout: "long_desc": "", 2024-04-23T10:41:08.493 INFO:teuthology.orchestra.run.smithi114.stdout: "tags": [], 2024-04-23T10:41:08.493 INFO:teuthology.orchestra.run.smithi114.stdout: "see_also": [] 2024-04-23T10:41:08.493 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-23T10:41:08.493 INFO:teuthology.orchestra.run.smithi114.stdout: "standby_error_status_code": { 2024-04-23T10:41:08.493 INFO:teuthology.orchestra.run.smithi114.stdout: "name": "standby_error_status_code", 2024-04-23T10:41:08.493 INFO:teuthology.orchestra.run.smithi114.stdout: "type": "int", 2024-04-23T10:41:08.493 INFO:teuthology.orchestra.run.smithi114.stdout: "level": "advanced", 2024-04-23T10:41:08.493 INFO:teuthology.orchestra.run.smithi114.stdout: "flags": 1, 2024-04-23T10:41:08.494 INFO:teuthology.orchestra.run.smithi114.stdout: "default_value": "500", 2024-04-23T10:41:08.494 INFO:teuthology.orchestra.run.smithi114.stdout: "min": "400", 2024-04-23T10:41:08.494 INFO:teuthology.orchestra.run.smithi114.stdout: "max": "599", 2024-04-23T10:41:08.494 INFO:teuthology.orchestra.run.smithi114.stdout: "enum_allowed": [], 2024-04-23T10:41:08.494 INFO:teuthology.orchestra.run.smithi114.stdout: "desc": "", 2024-04-23T10:41:08.494 INFO:teuthology.orchestra.run.smithi114.stdout: "long_desc": "", 2024-04-23T10:41:08.494 INFO:teuthology.orchestra.run.smithi114.stdout: "tags": [], 2024-04-23T10:41:08.494 INFO:teuthology.orchestra.run.smithi114.stdout: "see_also": [] 2024-04-23T10:41:08.494 INFO:teuthology.orchestra.run.smithi114.stdout: } 2024-04-23T10:41:08.495 INFO:teuthology.orchestra.run.smithi114.stdout: } 2024-04-23T10:41:08.495 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-23T10:41:08.495 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-04-23T10:41:08.495 INFO:teuthology.orchestra.run.smithi114.stdout: "name": "rbd_support", 2024-04-23T10:41:08.495 INFO:teuthology.orchestra.run.smithi114.stdout: "can_run": true, 2024-04-23T10:41:08.495 INFO:teuthology.orchestra.run.smithi114.stdout: "error_string": "", 2024-04-23T10:41:08.495 INFO:teuthology.orchestra.run.smithi114.stdout: "module_options": { 2024-04-23T10:41:08.495 INFO:teuthology.orchestra.run.smithi114.stdout: "log_level": { 2024-04-23T10:41:08.495 INFO:teuthology.orchestra.run.smithi114.stdout: "name": "log_level", 2024-04-23T10:41:08.495 INFO:teuthology.orchestra.run.smithi114.stdout: "type": "str", 2024-04-23T10:41:08.496 INFO:teuthology.orchestra.run.smithi114.stdout: "level": "advanced", 2024-04-23T10:41:08.496 INFO:teuthology.orchestra.run.smithi114.stdout: "flags": 1, 2024-04-23T10:41:08.496 INFO:teuthology.orchestra.run.smithi114.stdout: "default_value": "", 2024-04-23T10:41:08.496 INFO:teuthology.orchestra.run.smithi114.stdout: "min": "", 2024-04-23T10:41:08.496 INFO:teuthology.orchestra.run.smithi114.stdout: "max": "", 2024-04-23T10:41:08.496 INFO:teuthology.orchestra.run.smithi114.stdout: "enum_allowed": [ 2024-04-23T10:41:08.496 INFO:teuthology.orchestra.run.smithi114.stdout: "", 2024-04-23T10:41:08.496 INFO:teuthology.orchestra.run.smithi114.stdout: "critical", 2024-04-23T10:41:08.496 INFO:teuthology.orchestra.run.smithi114.stdout: "debug", 2024-04-23T10:41:08.496 INFO:teuthology.orchestra.run.smithi114.stdout: "error", 2024-04-23T10:41:08.497 INFO:teuthology.orchestra.run.smithi114.stdout: "info", 2024-04-23T10:41:08.497 INFO:teuthology.orchestra.run.smithi114.stdout: "warning" 2024-04-23T10:41:08.497 INFO:teuthology.orchestra.run.smithi114.stdout: ], 2024-04-23T10:41:08.497 INFO:teuthology.orchestra.run.smithi114.stdout: "desc": "", 2024-04-23T10:41:08.497 INFO:teuthology.orchestra.run.smithi114.stdout: "long_desc": "", 2024-04-23T10:41:08.497 INFO:teuthology.orchestra.run.smithi114.stdout: "tags": [], 2024-04-23T10:41:08.497 INFO:teuthology.orchestra.run.smithi114.stdout: "see_also": [] 2024-04-23T10:41:08.497 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-23T10:41:08.497 INFO:teuthology.orchestra.run.smithi114.stdout: "log_to_cluster": { 2024-04-23T10:41:08.497 INFO:teuthology.orchestra.run.smithi114.stdout: "name": "log_to_cluster", 2024-04-23T10:41:08.498 INFO:teuthology.orchestra.run.smithi114.stdout: "type": "bool", 2024-04-23T10:41:08.498 INFO:teuthology.orchestra.run.smithi114.stdout: "level": "advanced", 2024-04-23T10:41:08.498 INFO:teuthology.orchestra.run.smithi114.stdout: "flags": 1, 2024-04-23T10:41:08.498 INFO:teuthology.orchestra.run.smithi114.stdout: "default_value": "False", 2024-04-23T10:41:08.498 INFO:teuthology.orchestra.run.smithi114.stdout: "min": "", 2024-04-23T10:41:08.498 INFO:teuthology.orchestra.run.smithi114.stdout: "max": "", 2024-04-23T10:41:08.498 INFO:teuthology.orchestra.run.smithi114.stdout: "enum_allowed": [], 2024-04-23T10:41:08.498 INFO:teuthology.orchestra.run.smithi114.stdout: "desc": "", 2024-04-23T10:41:08.498 INFO:teuthology.orchestra.run.smithi114.stdout: "long_desc": "", 2024-04-23T10:41:08.499 INFO:teuthology.orchestra.run.smithi114.stdout: "tags": [], 2024-04-23T10:41:08.499 INFO:teuthology.orchestra.run.smithi114.stdout: "see_also": [] 2024-04-23T10:41:08.499 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-23T10:41:08.499 INFO:teuthology.orchestra.run.smithi114.stdout: "log_to_cluster_level": { 2024-04-23T10:41:08.499 INFO:teuthology.orchestra.run.smithi114.stdout: "name": "log_to_cluster_level", 2024-04-23T10:41:08.499 INFO:teuthology.orchestra.run.smithi114.stdout: "type": "str", 2024-04-23T10:41:08.499 INFO:teuthology.orchestra.run.smithi114.stdout: "level": "advanced", 2024-04-23T10:41:08.499 INFO:teuthology.orchestra.run.smithi114.stdout: "flags": 1, 2024-04-23T10:41:08.499 INFO:teuthology.orchestra.run.smithi114.stdout: "default_value": "info", 2024-04-23T10:41:08.499 INFO:teuthology.orchestra.run.smithi114.stdout: "min": "", 2024-04-23T10:41:08.500 INFO:teuthology.orchestra.run.smithi114.stdout: "max": "", 2024-04-23T10:41:08.500 INFO:teuthology.orchestra.run.smithi114.stdout: "enum_allowed": [ 2024-04-23T10:41:08.500 INFO:teuthology.orchestra.run.smithi114.stdout: "", 2024-04-23T10:41:08.500 INFO:teuthology.orchestra.run.smithi114.stdout: "critical", 2024-04-23T10:41:08.500 INFO:teuthology.orchestra.run.smithi114.stdout: "debug", 2024-04-23T10:41:08.500 INFO:teuthology.orchestra.run.smithi114.stdout: "error", 2024-04-23T10:41:08.500 INFO:teuthology.orchestra.run.smithi114.stdout: "info", 2024-04-23T10:41:08.500 INFO:teuthology.orchestra.run.smithi114.stdout: "warning" 2024-04-23T10:41:08.500 INFO:teuthology.orchestra.run.smithi114.stdout: ], 2024-04-23T10:41:08.500 INFO:teuthology.orchestra.run.smithi114.stdout: "desc": "", 2024-04-23T10:41:08.501 INFO:teuthology.orchestra.run.smithi114.stdout: "long_desc": "", 2024-04-23T10:41:08.501 INFO:teuthology.orchestra.run.smithi114.stdout: "tags": [], 2024-04-23T10:41:08.501 INFO:teuthology.orchestra.run.smithi114.stdout: "see_also": [] 2024-04-23T10:41:08.501 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-23T10:41:08.501 INFO:teuthology.orchestra.run.smithi114.stdout: "log_to_file": { 2024-04-23T10:41:08.501 INFO:teuthology.orchestra.run.smithi114.stdout: "name": "log_to_file", 2024-04-23T10:41:08.501 INFO:teuthology.orchestra.run.smithi114.stdout: "type": "bool", 2024-04-23T10:41:08.501 INFO:teuthology.orchestra.run.smithi114.stdout: "level": "advanced", 2024-04-23T10:41:08.501 INFO:teuthology.orchestra.run.smithi114.stdout: "flags": 1, 2024-04-23T10:41:08.501 INFO:teuthology.orchestra.run.smithi114.stdout: "default_value": "False", 2024-04-23T10:41:08.502 INFO:teuthology.orchestra.run.smithi114.stdout: "min": "", 2024-04-23T10:41:08.502 INFO:teuthology.orchestra.run.smithi114.stdout: "max": "", 2024-04-23T10:41:08.502 INFO:teuthology.orchestra.run.smithi114.stdout: "enum_allowed": [], 2024-04-23T10:41:08.502 INFO:teuthology.orchestra.run.smithi114.stdout: "desc": "", 2024-04-23T10:41:08.502 INFO:teuthology.orchestra.run.smithi114.stdout: "long_desc": "", 2024-04-23T10:41:08.502 INFO:teuthology.orchestra.run.smithi114.stdout: "tags": [], 2024-04-23T10:41:08.502 INFO:teuthology.orchestra.run.smithi114.stdout: "see_also": [] 2024-04-23T10:41:08.502 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-23T10:41:08.502 INFO:teuthology.orchestra.run.smithi114.stdout: "max_concurrent_snap_create": { 2024-04-23T10:41:08.503 INFO:teuthology.orchestra.run.smithi114.stdout: "name": "max_concurrent_snap_create", 2024-04-23T10:41:08.503 INFO:teuthology.orchestra.run.smithi114.stdout: "type": "int", 2024-04-23T10:41:08.503 INFO:teuthology.orchestra.run.smithi114.stdout: "level": "advanced", 2024-04-23T10:41:08.503 INFO:teuthology.orchestra.run.smithi114.stdout: "flags": 0, 2024-04-23T10:41:08.503 INFO:teuthology.orchestra.run.smithi114.stdout: "default_value": "10", 2024-04-23T10:41:08.503 INFO:teuthology.orchestra.run.smithi114.stdout: "min": "", 2024-04-23T10:41:08.503 INFO:teuthology.orchestra.run.smithi114.stdout: "max": "", 2024-04-23T10:41:08.503 INFO:teuthology.orchestra.run.smithi114.stdout: "enum_allowed": [], 2024-04-23T10:41:08.503 INFO:teuthology.orchestra.run.smithi114.stdout: "desc": "", 2024-04-23T10:41:08.503 INFO:teuthology.orchestra.run.smithi114.stdout: "long_desc": "", 2024-04-23T10:41:08.504 INFO:teuthology.orchestra.run.smithi114.stdout: "tags": [], 2024-04-23T10:41:08.504 INFO:teuthology.orchestra.run.smithi114.stdout: "see_also": [] 2024-04-23T10:41:08.504 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-23T10:41:08.504 INFO:teuthology.orchestra.run.smithi114.stdout: "mirror_snapshot_schedule": { 2024-04-23T10:41:08.504 INFO:teuthology.orchestra.run.smithi114.stdout: "name": "mirror_snapshot_schedule", 2024-04-23T10:41:08.504 INFO:teuthology.orchestra.run.smithi114.stdout: "type": "str", 2024-04-23T10:41:08.504 INFO:teuthology.orchestra.run.smithi114.stdout: "level": "advanced", 2024-04-23T10:41:08.504 INFO:teuthology.orchestra.run.smithi114.stdout: "flags": 0, 2024-04-23T10:41:08.504 INFO:teuthology.orchestra.run.smithi114.stdout: "default_value": "", 2024-04-23T10:41:08.505 INFO:teuthology.orchestra.run.smithi114.stdout: "min": "", 2024-04-23T10:41:08.505 INFO:teuthology.orchestra.run.smithi114.stdout: "max": "", 2024-04-23T10:41:08.505 INFO:teuthology.orchestra.run.smithi114.stdout: "enum_allowed": [], 2024-04-23T10:41:08.505 INFO:teuthology.orchestra.run.smithi114.stdout: "desc": "", 2024-04-23T10:41:08.505 INFO:teuthology.orchestra.run.smithi114.stdout: "long_desc": "", 2024-04-23T10:41:08.505 INFO:teuthology.orchestra.run.smithi114.stdout: "tags": [], 2024-04-23T10:41:08.505 INFO:teuthology.orchestra.run.smithi114.stdout: "see_also": [] 2024-04-23T10:41:08.505 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-23T10:41:08.505 INFO:teuthology.orchestra.run.smithi114.stdout: "trash_purge_schedule": { 2024-04-23T10:41:08.506 INFO:teuthology.orchestra.run.smithi114.stdout: "name": "trash_purge_schedule", 2024-04-23T10:41:08.506 INFO:teuthology.orchestra.run.smithi114.stdout: "type": "str", 2024-04-23T10:41:08.506 INFO:teuthology.orchestra.run.smithi114.stdout: "level": "advanced", 2024-04-23T10:41:08.506 INFO:teuthology.orchestra.run.smithi114.stdout: "flags": 0, 2024-04-23T10:41:08.506 INFO:teuthology.orchestra.run.smithi114.stdout: "default_value": "", 2024-04-23T10:41:08.506 INFO:teuthology.orchestra.run.smithi114.stdout: "min": "", 2024-04-23T10:41:08.506 INFO:teuthology.orchestra.run.smithi114.stdout: "max": "", 2024-04-23T10:41:08.506 INFO:teuthology.orchestra.run.smithi114.stdout: "enum_allowed": [], 2024-04-23T10:41:08.507 INFO:teuthology.orchestra.run.smithi114.stdout: "desc": "", 2024-04-23T10:41:08.507 INFO:teuthology.orchestra.run.smithi114.stdout: "long_desc": "", 2024-04-23T10:41:08.507 INFO:teuthology.orchestra.run.smithi114.stdout: "tags": [], 2024-04-23T10:41:08.507 INFO:teuthology.orchestra.run.smithi114.stdout: "see_also": [] 2024-04-23T10:41:08.507 INFO:teuthology.orchestra.run.smithi114.stdout: } 2024-04-23T10:41:08.507 INFO:teuthology.orchestra.run.smithi114.stdout: } 2024-04-23T10:41:08.507 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-23T10:41:08.507 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-04-23T10:41:08.507 INFO:teuthology.orchestra.run.smithi114.stdout: "name": "restful", 2024-04-23T10:41:08.507 INFO:teuthology.orchestra.run.smithi114.stdout: "can_run": true, 2024-04-23T10:41:08.508 INFO:teuthology.orchestra.run.smithi114.stdout: "error_string": "", 2024-04-23T10:41:08.508 INFO:teuthology.orchestra.run.smithi114.stdout: "module_options": { 2024-04-23T10:41:08.508 INFO:teuthology.orchestra.run.smithi114.stdout: "enable_auth": { 2024-04-23T10:41:08.508 INFO:teuthology.orchestra.run.smithi114.stdout: "name": "enable_auth", 2024-04-23T10:41:08.508 INFO:teuthology.orchestra.run.smithi114.stdout: "type": "bool", 2024-04-23T10:41:08.508 INFO:teuthology.orchestra.run.smithi114.stdout: "level": "advanced", 2024-04-23T10:41:08.508 INFO:teuthology.orchestra.run.smithi114.stdout: "flags": 0, 2024-04-23T10:41:08.508 INFO:teuthology.orchestra.run.smithi114.stdout: "default_value": "True", 2024-04-23T10:41:08.508 INFO:teuthology.orchestra.run.smithi114.stdout: "min": "", 2024-04-23T10:41:08.508 INFO:teuthology.orchestra.run.smithi114.stdout: "max": "", 2024-04-23T10:41:08.509 INFO:teuthology.orchestra.run.smithi114.stdout: "enum_allowed": [], 2024-04-23T10:41:08.509 INFO:teuthology.orchestra.run.smithi114.stdout: "desc": "", 2024-04-23T10:41:08.509 INFO:teuthology.orchestra.run.smithi114.stdout: "long_desc": "", 2024-04-23T10:41:08.509 INFO:teuthology.orchestra.run.smithi114.stdout: "tags": [], 2024-04-23T10:41:08.509 INFO:teuthology.orchestra.run.smithi114.stdout: "see_also": [] 2024-04-23T10:41:08.509 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-23T10:41:08.509 INFO:teuthology.orchestra.run.smithi114.stdout: "key_file": { 2024-04-23T10:41:08.509 INFO:teuthology.orchestra.run.smithi114.stdout: "name": "key_file", 2024-04-23T10:41:08.509 INFO:teuthology.orchestra.run.smithi114.stdout: "type": "str", 2024-04-23T10:41:08.509 INFO:teuthology.orchestra.run.smithi114.stdout: "level": "advanced", 2024-04-23T10:41:08.510 INFO:teuthology.orchestra.run.smithi114.stdout: "flags": 0, 2024-04-23T10:41:08.510 INFO:teuthology.orchestra.run.smithi114.stdout: "default_value": "", 2024-04-23T10:41:08.510 INFO:teuthology.orchestra.run.smithi114.stdout: "min": "", 2024-04-23T10:41:08.510 INFO:teuthology.orchestra.run.smithi114.stdout: "max": "", 2024-04-23T10:41:08.510 INFO:teuthology.orchestra.run.smithi114.stdout: "enum_allowed": [], 2024-04-23T10:41:08.510 INFO:teuthology.orchestra.run.smithi114.stdout: "desc": "", 2024-04-23T10:41:08.510 INFO:teuthology.orchestra.run.smithi114.stdout: "long_desc": "", 2024-04-23T10:41:08.510 INFO:teuthology.orchestra.run.smithi114.stdout: "tags": [], 2024-04-23T10:41:08.510 INFO:teuthology.orchestra.run.smithi114.stdout: "see_also": [] 2024-04-23T10:41:08.511 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-23T10:41:08.511 INFO:teuthology.orchestra.run.smithi114.stdout: "log_level": { 2024-04-23T10:41:08.511 INFO:teuthology.orchestra.run.smithi114.stdout: "name": "log_level", 2024-04-23T10:41:08.511 INFO:teuthology.orchestra.run.smithi114.stdout: "type": "str", 2024-04-23T10:41:08.511 INFO:teuthology.orchestra.run.smithi114.stdout: "level": "advanced", 2024-04-23T10:41:08.511 INFO:teuthology.orchestra.run.smithi114.stdout: "flags": 1, 2024-04-23T10:41:08.511 INFO:teuthology.orchestra.run.smithi114.stdout: "default_value": "", 2024-04-23T10:41:08.511 INFO:teuthology.orchestra.run.smithi114.stdout: "min": "", 2024-04-23T10:41:08.511 INFO:teuthology.orchestra.run.smithi114.stdout: "max": "", 2024-04-23T10:41:08.511 INFO:teuthology.orchestra.run.smithi114.stdout: "enum_allowed": [ 2024-04-23T10:41:08.512 INFO:teuthology.orchestra.run.smithi114.stdout: "", 2024-04-23T10:41:08.512 INFO:teuthology.orchestra.run.smithi114.stdout: "critical", 2024-04-23T10:41:08.512 INFO:teuthology.orchestra.run.smithi114.stdout: "debug", 2024-04-23T10:41:08.512 INFO:teuthology.orchestra.run.smithi114.stdout: "error", 2024-04-23T10:41:08.512 INFO:teuthology.orchestra.run.smithi114.stdout: "info", 2024-04-23T10:41:08.512 INFO:teuthology.orchestra.run.smithi114.stdout: "warning" 2024-04-23T10:41:08.512 INFO:teuthology.orchestra.run.smithi114.stdout: ], 2024-04-23T10:41:08.512 INFO:teuthology.orchestra.run.smithi114.stdout: "desc": "", 2024-04-23T10:41:08.512 INFO:teuthology.orchestra.run.smithi114.stdout: "long_desc": "", 2024-04-23T10:41:08.513 INFO:teuthology.orchestra.run.smithi114.stdout: "tags": [], 2024-04-23T10:41:08.513 INFO:teuthology.orchestra.run.smithi114.stdout: "see_also": [] 2024-04-23T10:41:08.513 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-23T10:41:08.513 INFO:teuthology.orchestra.run.smithi114.stdout: "log_to_cluster": { 2024-04-23T10:41:08.513 INFO:teuthology.orchestra.run.smithi114.stdout: "name": "log_to_cluster", 2024-04-23T10:41:08.513 INFO:teuthology.orchestra.run.smithi114.stdout: "type": "bool", 2024-04-23T10:41:08.513 INFO:teuthology.orchestra.run.smithi114.stdout: "level": "advanced", 2024-04-23T10:41:08.513 INFO:teuthology.orchestra.run.smithi114.stdout: "flags": 1, 2024-04-23T10:41:08.513 INFO:teuthology.orchestra.run.smithi114.stdout: "default_value": "False", 2024-04-23T10:41:08.513 INFO:teuthology.orchestra.run.smithi114.stdout: "min": "", 2024-04-23T10:41:08.514 INFO:teuthology.orchestra.run.smithi114.stdout: "max": "", 2024-04-23T10:41:08.514 INFO:teuthology.orchestra.run.smithi114.stdout: "enum_allowed": [], 2024-04-23T10:41:08.514 INFO:teuthology.orchestra.run.smithi114.stdout: "desc": "", 2024-04-23T10:41:08.514 INFO:teuthology.orchestra.run.smithi114.stdout: "long_desc": "", 2024-04-23T10:41:08.514 INFO:teuthology.orchestra.run.smithi114.stdout: "tags": [], 2024-04-23T10:41:08.514 INFO:teuthology.orchestra.run.smithi114.stdout: "see_also": [] 2024-04-23T10:41:08.514 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-23T10:41:08.514 INFO:teuthology.orchestra.run.smithi114.stdout: "log_to_cluster_level": { 2024-04-23T10:41:08.514 INFO:teuthology.orchestra.run.smithi114.stdout: "name": "log_to_cluster_level", 2024-04-23T10:41:08.514 INFO:teuthology.orchestra.run.smithi114.stdout: "type": "str", 2024-04-23T10:41:08.515 INFO:teuthology.orchestra.run.smithi114.stdout: "level": "advanced", 2024-04-23T10:41:08.515 INFO:teuthology.orchestra.run.smithi114.stdout: "flags": 1, 2024-04-23T10:41:08.515 INFO:teuthology.orchestra.run.smithi114.stdout: "default_value": "info", 2024-04-23T10:41:08.515 INFO:teuthology.orchestra.run.smithi114.stdout: "min": "", 2024-04-23T10:41:08.515 INFO:teuthology.orchestra.run.smithi114.stdout: "max": "", 2024-04-23T10:41:08.515 INFO:teuthology.orchestra.run.smithi114.stdout: "enum_allowed": [ 2024-04-23T10:41:08.515 INFO:teuthology.orchestra.run.smithi114.stdout: "", 2024-04-23T10:41:08.515 INFO:teuthology.orchestra.run.smithi114.stdout: "critical", 2024-04-23T10:41:08.515 INFO:teuthology.orchestra.run.smithi114.stdout: "debug", 2024-04-23T10:41:08.515 INFO:teuthology.orchestra.run.smithi114.stdout: "error", 2024-04-23T10:41:08.516 INFO:teuthology.orchestra.run.smithi114.stdout: "info", 2024-04-23T10:41:08.516 INFO:teuthology.orchestra.run.smithi114.stdout: "warning" 2024-04-23T10:41:08.516 INFO:teuthology.orchestra.run.smithi114.stdout: ], 2024-04-23T10:41:08.516 INFO:teuthology.orchestra.run.smithi114.stdout: "desc": "", 2024-04-23T10:41:08.516 INFO:teuthology.orchestra.run.smithi114.stdout: "long_desc": "", 2024-04-23T10:41:08.516 INFO:teuthology.orchestra.run.smithi114.stdout: "tags": [], 2024-04-23T10:41:08.516 INFO:teuthology.orchestra.run.smithi114.stdout: "see_also": [] 2024-04-23T10:41:08.516 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-23T10:41:08.516 INFO:teuthology.orchestra.run.smithi114.stdout: "log_to_file": { 2024-04-23T10:41:08.516 INFO:teuthology.orchestra.run.smithi114.stdout: "name": "log_to_file", 2024-04-23T10:41:08.517 INFO:teuthology.orchestra.run.smithi114.stdout: "type": "bool", 2024-04-23T10:41:08.517 INFO:teuthology.orchestra.run.smithi114.stdout: "level": "advanced", 2024-04-23T10:41:08.517 INFO:teuthology.orchestra.run.smithi114.stdout: "flags": 1, 2024-04-23T10:41:08.517 INFO:teuthology.orchestra.run.smithi114.stdout: "default_value": "False", 2024-04-23T10:41:08.517 INFO:teuthology.orchestra.run.smithi114.stdout: "min": "", 2024-04-23T10:41:08.517 INFO:teuthology.orchestra.run.smithi114.stdout: "max": "", 2024-04-23T10:41:08.517 INFO:teuthology.orchestra.run.smithi114.stdout: "enum_allowed": [], 2024-04-23T10:41:08.517 INFO:teuthology.orchestra.run.smithi114.stdout: "desc": "", 2024-04-23T10:41:08.517 INFO:teuthology.orchestra.run.smithi114.stdout: "long_desc": "", 2024-04-23T10:41:08.518 INFO:teuthology.orchestra.run.smithi114.stdout: "tags": [], 2024-04-23T10:41:08.518 INFO:teuthology.orchestra.run.smithi114.stdout: "see_also": [] 2024-04-23T10:41:08.518 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-23T10:41:08.518 INFO:teuthology.orchestra.run.smithi114.stdout: "server_addr": { 2024-04-23T10:41:08.518 INFO:teuthology.orchestra.run.smithi114.stdout: "name": "server_addr", 2024-04-23T10:41:08.518 INFO:teuthology.orchestra.run.smithi114.stdout: "type": "str", 2024-04-23T10:41:08.518 INFO:teuthology.orchestra.run.smithi114.stdout: "level": "advanced", 2024-04-23T10:41:08.518 INFO:teuthology.orchestra.run.smithi114.stdout: "flags": 0, 2024-04-23T10:41:08.518 INFO:teuthology.orchestra.run.smithi114.stdout: "default_value": "", 2024-04-23T10:41:08.518 INFO:teuthology.orchestra.run.smithi114.stdout: "min": "", 2024-04-23T10:41:08.519 INFO:teuthology.orchestra.run.smithi114.stdout: "max": "", 2024-04-23T10:41:08.519 INFO:teuthology.orchestra.run.smithi114.stdout: "enum_allowed": [], 2024-04-23T10:41:08.519 INFO:teuthology.orchestra.run.smithi114.stdout: "desc": "", 2024-04-23T10:41:08.519 INFO:teuthology.orchestra.run.smithi114.stdout: "long_desc": "", 2024-04-23T10:41:08.519 INFO:teuthology.orchestra.run.smithi114.stdout: "tags": [], 2024-04-23T10:41:08.519 INFO:teuthology.orchestra.run.smithi114.stdout: "see_also": [] 2024-04-23T10:41:08.519 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-23T10:41:08.519 INFO:teuthology.orchestra.run.smithi114.stdout: "server_port": { 2024-04-23T10:41:08.519 INFO:teuthology.orchestra.run.smithi114.stdout: "name": "server_port", 2024-04-23T10:41:08.519 INFO:teuthology.orchestra.run.smithi114.stdout: "type": "str", 2024-04-23T10:41:08.520 INFO:teuthology.orchestra.run.smithi114.stdout: "level": "advanced", 2024-04-23T10:41:08.520 INFO:teuthology.orchestra.run.smithi114.stdout: "flags": 0, 2024-04-23T10:41:08.520 INFO:teuthology.orchestra.run.smithi114.stdout: "default_value": "", 2024-04-23T10:41:08.520 INFO:teuthology.orchestra.run.smithi114.stdout: "min": "", 2024-04-23T10:41:08.520 INFO:teuthology.orchestra.run.smithi114.stdout: "max": "", 2024-04-23T10:41:08.520 INFO:teuthology.orchestra.run.smithi114.stdout: "enum_allowed": [], 2024-04-23T10:41:08.520 INFO:teuthology.orchestra.run.smithi114.stdout: "desc": "", 2024-04-23T10:41:08.520 INFO:teuthology.orchestra.run.smithi114.stdout: "long_desc": "", 2024-04-23T10:41:08.520 INFO:teuthology.orchestra.run.smithi114.stdout: "tags": [], 2024-04-23T10:41:08.520 INFO:teuthology.orchestra.run.smithi114.stdout: "see_also": [] 2024-04-23T10:41:08.521 INFO:teuthology.orchestra.run.smithi114.stdout: } 2024-04-23T10:41:08.521 INFO:teuthology.orchestra.run.smithi114.stdout: } 2024-04-23T10:41:08.521 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-23T10:41:08.521 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-04-23T10:41:08.521 INFO:teuthology.orchestra.run.smithi114.stdout: "name": "rgw", 2024-04-23T10:41:08.521 INFO:teuthology.orchestra.run.smithi114.stdout: "can_run": true, 2024-04-23T10:41:08.521 INFO:teuthology.orchestra.run.smithi114.stdout: "error_string": "", 2024-04-23T10:41:08.521 INFO:teuthology.orchestra.run.smithi114.stdout: "module_options": { 2024-04-23T10:41:08.521 INFO:teuthology.orchestra.run.smithi114.stdout: "log_level": { 2024-04-23T10:41:08.521 INFO:teuthology.orchestra.run.smithi114.stdout: "name": "log_level", 2024-04-23T10:41:08.522 INFO:teuthology.orchestra.run.smithi114.stdout: "type": "str", 2024-04-23T10:41:08.522 INFO:teuthology.orchestra.run.smithi114.stdout: "level": "advanced", 2024-04-23T10:41:08.522 INFO:teuthology.orchestra.run.smithi114.stdout: "flags": 1, 2024-04-23T10:41:08.522 INFO:teuthology.orchestra.run.smithi114.stdout: "default_value": "", 2024-04-23T10:41:08.522 INFO:teuthology.orchestra.run.smithi114.stdout: "min": "", 2024-04-23T10:41:08.522 INFO:teuthology.orchestra.run.smithi114.stdout: "max": "", 2024-04-23T10:41:08.522 INFO:teuthology.orchestra.run.smithi114.stdout: "enum_allowed": [ 2024-04-23T10:41:08.522 INFO:teuthology.orchestra.run.smithi114.stdout: "", 2024-04-23T10:41:08.522 INFO:teuthology.orchestra.run.smithi114.stdout: "critical", 2024-04-23T10:41:08.523 INFO:teuthology.orchestra.run.smithi114.stdout: "debug", 2024-04-23T10:41:08.523 INFO:teuthology.orchestra.run.smithi114.stdout: "error", 2024-04-23T10:41:08.523 INFO:teuthology.orchestra.run.smithi114.stdout: "info", 2024-04-23T10:41:08.523 INFO:teuthology.orchestra.run.smithi114.stdout: "warning" 2024-04-23T10:41:08.523 INFO:teuthology.orchestra.run.smithi114.stdout: ], 2024-04-23T10:41:08.523 INFO:teuthology.orchestra.run.smithi114.stdout: "desc": "", 2024-04-23T10:41:08.523 INFO:teuthology.orchestra.run.smithi114.stdout: "long_desc": "", 2024-04-23T10:41:08.523 INFO:teuthology.orchestra.run.smithi114.stdout: "tags": [], 2024-04-23T10:41:08.523 INFO:teuthology.orchestra.run.smithi114.stdout: "see_also": [] 2024-04-23T10:41:08.523 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-23T10:41:08.524 INFO:teuthology.orchestra.run.smithi114.stdout: "log_to_cluster": { 2024-04-23T10:41:08.524 INFO:teuthology.orchestra.run.smithi114.stdout: "name": "log_to_cluster", 2024-04-23T10:41:08.524 INFO:teuthology.orchestra.run.smithi114.stdout: "type": "bool", 2024-04-23T10:41:08.524 INFO:teuthology.orchestra.run.smithi114.stdout: "level": "advanced", 2024-04-23T10:41:08.524 INFO:teuthology.orchestra.run.smithi114.stdout: "flags": 1, 2024-04-23T10:41:08.524 INFO:teuthology.orchestra.run.smithi114.stdout: "default_value": "False", 2024-04-23T10:41:08.524 INFO:teuthology.orchestra.run.smithi114.stdout: "min": "", 2024-04-23T10:41:08.524 INFO:teuthology.orchestra.run.smithi114.stdout: "max": "", 2024-04-23T10:41:08.524 INFO:teuthology.orchestra.run.smithi114.stdout: "enum_allowed": [], 2024-04-23T10:41:08.524 INFO:teuthology.orchestra.run.smithi114.stdout: "desc": "", 2024-04-23T10:41:08.525 INFO:teuthology.orchestra.run.smithi114.stdout: "long_desc": "", 2024-04-23T10:41:08.525 INFO:teuthology.orchestra.run.smithi114.stdout: "tags": [], 2024-04-23T10:41:08.525 INFO:teuthology.orchestra.run.smithi114.stdout: "see_also": [] 2024-04-23T10:41:08.525 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-23T10:41:08.525 INFO:teuthology.orchestra.run.smithi114.stdout: "log_to_cluster_level": { 2024-04-23T10:41:08.525 INFO:teuthology.orchestra.run.smithi114.stdout: "name": "log_to_cluster_level", 2024-04-23T10:41:08.525 INFO:teuthology.orchestra.run.smithi114.stdout: "type": "str", 2024-04-23T10:41:08.525 INFO:teuthology.orchestra.run.smithi114.stdout: "level": "advanced", 2024-04-23T10:41:08.525 INFO:teuthology.orchestra.run.smithi114.stdout: "flags": 1, 2024-04-23T10:41:08.525 INFO:teuthology.orchestra.run.smithi114.stdout: "default_value": "info", 2024-04-23T10:41:08.526 INFO:teuthology.orchestra.run.smithi114.stdout: "min": "", 2024-04-23T10:41:08.526 INFO:teuthology.orchestra.run.smithi114.stdout: "max": "", 2024-04-23T10:41:08.526 INFO:teuthology.orchestra.run.smithi114.stdout: "enum_allowed": [ 2024-04-23T10:41:08.526 INFO:teuthology.orchestra.run.smithi114.stdout: "", 2024-04-23T10:41:08.526 INFO:teuthology.orchestra.run.smithi114.stdout: "critical", 2024-04-23T10:41:08.526 INFO:teuthology.orchestra.run.smithi114.stdout: "debug", 2024-04-23T10:41:08.526 INFO:teuthology.orchestra.run.smithi114.stdout: "error", 2024-04-23T10:41:08.526 INFO:teuthology.orchestra.run.smithi114.stdout: "info", 2024-04-23T10:41:08.526 INFO:teuthology.orchestra.run.smithi114.stdout: "warning" 2024-04-23T10:41:08.527 INFO:teuthology.orchestra.run.smithi114.stdout: ], 2024-04-23T10:41:08.527 INFO:teuthology.orchestra.run.smithi114.stdout: "desc": "", 2024-04-23T10:41:08.527 INFO:teuthology.orchestra.run.smithi114.stdout: "long_desc": "", 2024-04-23T10:41:08.527 INFO:teuthology.orchestra.run.smithi114.stdout: "tags": [], 2024-04-23T10:41:08.527 INFO:teuthology.orchestra.run.smithi114.stdout: "see_also": [] 2024-04-23T10:41:08.527 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-23T10:41:08.527 INFO:teuthology.orchestra.run.smithi114.stdout: "log_to_file": { 2024-04-23T10:41:08.527 INFO:teuthology.orchestra.run.smithi114.stdout: "name": "log_to_file", 2024-04-23T10:41:08.527 INFO:teuthology.orchestra.run.smithi114.stdout: "type": "bool", 2024-04-23T10:41:08.527 INFO:teuthology.orchestra.run.smithi114.stdout: "level": "advanced", 2024-04-23T10:41:08.528 INFO:teuthology.orchestra.run.smithi114.stdout: "flags": 1, 2024-04-23T10:41:08.528 INFO:teuthology.orchestra.run.smithi114.stdout: "default_value": "False", 2024-04-23T10:41:08.528 INFO:teuthology.orchestra.run.smithi114.stdout: "min": "", 2024-04-23T10:41:08.528 INFO:teuthology.orchestra.run.smithi114.stdout: "max": "", 2024-04-23T10:41:08.528 INFO:teuthology.orchestra.run.smithi114.stdout: "enum_allowed": [], 2024-04-23T10:41:08.528 INFO:teuthology.orchestra.run.smithi114.stdout: "desc": "", 2024-04-23T10:41:08.528 INFO:teuthology.orchestra.run.smithi114.stdout: "long_desc": "", 2024-04-23T10:41:08.528 INFO:teuthology.orchestra.run.smithi114.stdout: "tags": [], 2024-04-23T10:41:08.528 INFO:teuthology.orchestra.run.smithi114.stdout: "see_also": [] 2024-04-23T10:41:08.528 INFO:teuthology.orchestra.run.smithi114.stdout: } 2024-04-23T10:41:08.529 INFO:teuthology.orchestra.run.smithi114.stdout: } 2024-04-23T10:41:08.529 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-23T10:41:08.529 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-04-23T10:41:08.529 INFO:teuthology.orchestra.run.smithi114.stdout: "name": "rook", 2024-04-23T10:41:08.529 INFO:teuthology.orchestra.run.smithi114.stdout: "can_run": true, 2024-04-23T10:41:08.529 INFO:teuthology.orchestra.run.smithi114.stdout: "error_string": "", 2024-04-23T10:41:08.529 INFO:teuthology.orchestra.run.smithi114.stdout: "module_options": { 2024-04-23T10:41:08.529 INFO:teuthology.orchestra.run.smithi114.stdout: "log_level": { 2024-04-23T10:41:08.529 INFO:teuthology.orchestra.run.smithi114.stdout: "name": "log_level", 2024-04-23T10:41:08.529 INFO:teuthology.orchestra.run.smithi114.stdout: "type": "str", 2024-04-23T10:41:08.530 INFO:teuthology.orchestra.run.smithi114.stdout: "level": "advanced", 2024-04-23T10:41:08.530 INFO:teuthology.orchestra.run.smithi114.stdout: "flags": 1, 2024-04-23T10:41:08.530 INFO:teuthology.orchestra.run.smithi114.stdout: "default_value": "", 2024-04-23T10:41:08.530 INFO:teuthology.orchestra.run.smithi114.stdout: "min": "", 2024-04-23T10:41:08.530 INFO:teuthology.orchestra.run.smithi114.stdout: "max": "", 2024-04-23T10:41:08.530 INFO:teuthology.orchestra.run.smithi114.stdout: "enum_allowed": [ 2024-04-23T10:41:08.530 INFO:teuthology.orchestra.run.smithi114.stdout: "", 2024-04-23T10:41:08.530 INFO:teuthology.orchestra.run.smithi114.stdout: "critical", 2024-04-23T10:41:08.530 INFO:teuthology.orchestra.run.smithi114.stdout: "debug", 2024-04-23T10:41:08.530 INFO:teuthology.orchestra.run.smithi114.stdout: "error", 2024-04-23T10:41:08.531 INFO:teuthology.orchestra.run.smithi114.stdout: "info", 2024-04-23T10:41:08.531 INFO:teuthology.orchestra.run.smithi114.stdout: "warning" 2024-04-23T10:41:08.531 INFO:teuthology.orchestra.run.smithi114.stdout: ], 2024-04-23T10:41:08.531 INFO:teuthology.orchestra.run.smithi114.stdout: "desc": "", 2024-04-23T10:41:08.531 INFO:teuthology.orchestra.run.smithi114.stdout: "long_desc": "", 2024-04-23T10:41:08.531 INFO:teuthology.orchestra.run.smithi114.stdout: "tags": [], 2024-04-23T10:41:08.531 INFO:teuthology.orchestra.run.smithi114.stdout: "see_also": [] 2024-04-23T10:41:08.531 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-23T10:41:08.531 INFO:teuthology.orchestra.run.smithi114.stdout: "log_to_cluster": { 2024-04-23T10:41:08.532 INFO:teuthology.orchestra.run.smithi114.stdout: "name": "log_to_cluster", 2024-04-23T10:41:08.532 INFO:teuthology.orchestra.run.smithi114.stdout: "type": "bool", 2024-04-23T10:41:08.532 INFO:teuthology.orchestra.run.smithi114.stdout: "level": "advanced", 2024-04-23T10:41:08.532 INFO:teuthology.orchestra.run.smithi114.stdout: "flags": 1, 2024-04-23T10:41:08.532 INFO:teuthology.orchestra.run.smithi114.stdout: "default_value": "False", 2024-04-23T10:41:08.532 INFO:teuthology.orchestra.run.smithi114.stdout: "min": "", 2024-04-23T10:41:08.532 INFO:teuthology.orchestra.run.smithi114.stdout: "max": "", 2024-04-23T10:41:08.532 INFO:teuthology.orchestra.run.smithi114.stdout: "enum_allowed": [], 2024-04-23T10:41:08.532 INFO:teuthology.orchestra.run.smithi114.stdout: "desc": "", 2024-04-23T10:41:08.532 INFO:teuthology.orchestra.run.smithi114.stdout: "long_desc": "", 2024-04-23T10:41:08.533 INFO:teuthology.orchestra.run.smithi114.stdout: "tags": [], 2024-04-23T10:41:08.533 INFO:teuthology.orchestra.run.smithi114.stdout: "see_also": [] 2024-04-23T10:41:08.533 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-23T10:41:08.533 INFO:teuthology.orchestra.run.smithi114.stdout: "log_to_cluster_level": { 2024-04-23T10:41:08.533 INFO:teuthology.orchestra.run.smithi114.stdout: "name": "log_to_cluster_level", 2024-04-23T10:41:08.533 INFO:teuthology.orchestra.run.smithi114.stdout: "type": "str", 2024-04-23T10:41:08.533 INFO:teuthology.orchestra.run.smithi114.stdout: "level": "advanced", 2024-04-23T10:41:08.533 INFO:teuthology.orchestra.run.smithi114.stdout: "flags": 1, 2024-04-23T10:41:08.533 INFO:teuthology.orchestra.run.smithi114.stdout: "default_value": "info", 2024-04-23T10:41:08.533 INFO:teuthology.orchestra.run.smithi114.stdout: "min": "", 2024-04-23T10:41:08.534 INFO:teuthology.orchestra.run.smithi114.stdout: "max": "", 2024-04-23T10:41:08.534 INFO:teuthology.orchestra.run.smithi114.stdout: "enum_allowed": [ 2024-04-23T10:41:08.534 INFO:teuthology.orchestra.run.smithi114.stdout: "", 2024-04-23T10:41:08.534 INFO:teuthology.orchestra.run.smithi114.stdout: "critical", 2024-04-23T10:41:08.534 INFO:teuthology.orchestra.run.smithi114.stdout: "debug", 2024-04-23T10:41:08.534 INFO:teuthology.orchestra.run.smithi114.stdout: "error", 2024-04-23T10:41:08.534 INFO:teuthology.orchestra.run.smithi114.stdout: "info", 2024-04-23T10:41:08.534 INFO:teuthology.orchestra.run.smithi114.stdout: "warning" 2024-04-23T10:41:08.534 INFO:teuthology.orchestra.run.smithi114.stdout: ], 2024-04-23T10:41:08.535 INFO:teuthology.orchestra.run.smithi114.stdout: "desc": "", 2024-04-23T10:41:08.535 INFO:teuthology.orchestra.run.smithi114.stdout: "long_desc": "", 2024-04-23T10:41:08.535 INFO:teuthology.orchestra.run.smithi114.stdout: "tags": [], 2024-04-23T10:41:08.535 INFO:teuthology.orchestra.run.smithi114.stdout: "see_also": [] 2024-04-23T10:41:08.535 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-23T10:41:08.535 INFO:teuthology.orchestra.run.smithi114.stdout: "log_to_file": { 2024-04-23T10:41:08.535 INFO:teuthology.orchestra.run.smithi114.stdout: "name": "log_to_file", 2024-04-23T10:41:08.535 INFO:teuthology.orchestra.run.smithi114.stdout: "type": "bool", 2024-04-23T10:41:08.535 INFO:teuthology.orchestra.run.smithi114.stdout: "level": "advanced", 2024-04-23T10:41:08.535 INFO:teuthology.orchestra.run.smithi114.stdout: "flags": 1, 2024-04-23T10:41:08.536 INFO:teuthology.orchestra.run.smithi114.stdout: "default_value": "False", 2024-04-23T10:41:08.536 INFO:teuthology.orchestra.run.smithi114.stdout: "min": "", 2024-04-23T10:41:08.536 INFO:teuthology.orchestra.run.smithi114.stdout: "max": "", 2024-04-23T10:41:08.536 INFO:teuthology.orchestra.run.smithi114.stdout: "enum_allowed": [], 2024-04-23T10:41:08.536 INFO:teuthology.orchestra.run.smithi114.stdout: "desc": "", 2024-04-23T10:41:08.536 INFO:teuthology.orchestra.run.smithi114.stdout: "long_desc": "", 2024-04-23T10:41:08.536 INFO:teuthology.orchestra.run.smithi114.stdout: "tags": [], 2024-04-23T10:41:08.536 INFO:teuthology.orchestra.run.smithi114.stdout: "see_also": [] 2024-04-23T10:41:08.536 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-23T10:41:08.536 INFO:teuthology.orchestra.run.smithi114.stdout: "storage_class": { 2024-04-23T10:41:08.537 INFO:teuthology.orchestra.run.smithi114.stdout: "name": "storage_class", 2024-04-23T10:41:08.537 INFO:teuthology.orchestra.run.smithi114.stdout: "type": "str", 2024-04-23T10:41:08.537 INFO:teuthology.orchestra.run.smithi114.stdout: "level": "advanced", 2024-04-23T10:41:08.537 INFO:teuthology.orchestra.run.smithi114.stdout: "flags": 0, 2024-04-23T10:41:08.537 INFO:teuthology.orchestra.run.smithi114.stdout: "default_value": "local", 2024-04-23T10:41:08.537 INFO:teuthology.orchestra.run.smithi114.stdout: "min": "", 2024-04-23T10:41:08.537 INFO:teuthology.orchestra.run.smithi114.stdout: "max": "", 2024-04-23T10:41:08.537 INFO:teuthology.orchestra.run.smithi114.stdout: "enum_allowed": [], 2024-04-23T10:41:08.537 INFO:teuthology.orchestra.run.smithi114.stdout: "desc": "storage class name for LSO-discovered PVs", 2024-04-23T10:41:08.537 INFO:teuthology.orchestra.run.smithi114.stdout: "long_desc": "", 2024-04-23T10:41:08.538 INFO:teuthology.orchestra.run.smithi114.stdout: "tags": [], 2024-04-23T10:41:08.538 INFO:teuthology.orchestra.run.smithi114.stdout: "see_also": [] 2024-04-23T10:41:08.538 INFO:teuthology.orchestra.run.smithi114.stdout: } 2024-04-23T10:41:08.538 INFO:teuthology.orchestra.run.smithi114.stdout: } 2024-04-23T10:41:08.538 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-23T10:41:08.538 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-04-23T10:41:08.538 INFO:teuthology.orchestra.run.smithi114.stdout: "name": "selftest", 2024-04-23T10:41:08.538 INFO:teuthology.orchestra.run.smithi114.stdout: "can_run": true, 2024-04-23T10:41:08.538 INFO:teuthology.orchestra.run.smithi114.stdout: "error_string": "", 2024-04-23T10:41:08.539 INFO:teuthology.orchestra.run.smithi114.stdout: "module_options": { 2024-04-23T10:41:08.539 INFO:teuthology.orchestra.run.smithi114.stdout: "log_level": { 2024-04-23T10:41:08.539 INFO:teuthology.orchestra.run.smithi114.stdout: "name": "log_level", 2024-04-23T10:41:08.539 INFO:teuthology.orchestra.run.smithi114.stdout: "type": "str", 2024-04-23T10:41:08.539 INFO:teuthology.orchestra.run.smithi114.stdout: "level": "advanced", 2024-04-23T10:41:08.539 INFO:teuthology.orchestra.run.smithi114.stdout: "flags": 1, 2024-04-23T10:41:08.539 INFO:teuthology.orchestra.run.smithi114.stdout: "default_value": "", 2024-04-23T10:41:08.539 INFO:teuthology.orchestra.run.smithi114.stdout: "min": "", 2024-04-23T10:41:08.539 INFO:teuthology.orchestra.run.smithi114.stdout: "max": "", 2024-04-23T10:41:08.539 INFO:teuthology.orchestra.run.smithi114.stdout: "enum_allowed": [ 2024-04-23T10:41:08.540 INFO:teuthology.orchestra.run.smithi114.stdout: "", 2024-04-23T10:41:08.540 INFO:teuthology.orchestra.run.smithi114.stdout: "critical", 2024-04-23T10:41:08.540 INFO:teuthology.orchestra.run.smithi114.stdout: "debug", 2024-04-23T10:41:08.540 INFO:teuthology.orchestra.run.smithi114.stdout: "error", 2024-04-23T10:41:08.540 INFO:teuthology.orchestra.run.smithi114.stdout: "info", 2024-04-23T10:41:08.540 INFO:teuthology.orchestra.run.smithi114.stdout: "warning" 2024-04-23T10:41:08.540 INFO:teuthology.orchestra.run.smithi114.stdout: ], 2024-04-23T10:41:08.540 INFO:teuthology.orchestra.run.smithi114.stdout: "desc": "", 2024-04-23T10:41:08.540 INFO:teuthology.orchestra.run.smithi114.stdout: "long_desc": "", 2024-04-23T10:41:08.540 INFO:teuthology.orchestra.run.smithi114.stdout: "tags": [], 2024-04-23T10:41:08.541 INFO:teuthology.orchestra.run.smithi114.stdout: "see_also": [] 2024-04-23T10:41:08.541 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-23T10:41:08.541 INFO:teuthology.orchestra.run.smithi114.stdout: "log_to_cluster": { 2024-04-23T10:41:08.541 INFO:teuthology.orchestra.run.smithi114.stdout: "name": "log_to_cluster", 2024-04-23T10:41:08.541 INFO:teuthology.orchestra.run.smithi114.stdout: "type": "bool", 2024-04-23T10:41:08.541 INFO:teuthology.orchestra.run.smithi114.stdout: "level": "advanced", 2024-04-23T10:41:08.541 INFO:teuthology.orchestra.run.smithi114.stdout: "flags": 1, 2024-04-23T10:41:08.541 INFO:teuthology.orchestra.run.smithi114.stdout: "default_value": "False", 2024-04-23T10:41:08.541 INFO:teuthology.orchestra.run.smithi114.stdout: "min": "", 2024-04-23T10:41:08.542 INFO:teuthology.orchestra.run.smithi114.stdout: "max": "", 2024-04-23T10:41:08.542 INFO:teuthology.orchestra.run.smithi114.stdout: "enum_allowed": [], 2024-04-23T10:41:08.542 INFO:teuthology.orchestra.run.smithi114.stdout: "desc": "", 2024-04-23T10:41:08.542 INFO:teuthology.orchestra.run.smithi114.stdout: "long_desc": "", 2024-04-23T10:41:08.542 INFO:teuthology.orchestra.run.smithi114.stdout: "tags": [], 2024-04-23T10:41:08.542 INFO:teuthology.orchestra.run.smithi114.stdout: "see_also": [] 2024-04-23T10:41:08.542 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-23T10:41:08.542 INFO:teuthology.orchestra.run.smithi114.stdout: "log_to_cluster_level": { 2024-04-23T10:41:08.542 INFO:teuthology.orchestra.run.smithi114.stdout: "name": "log_to_cluster_level", 2024-04-23T10:41:08.543 INFO:teuthology.orchestra.run.smithi114.stdout: "type": "str", 2024-04-23T10:41:08.543 INFO:teuthology.orchestra.run.smithi114.stdout: "level": "advanced", 2024-04-23T10:41:08.543 INFO:teuthology.orchestra.run.smithi114.stdout: "flags": 1, 2024-04-23T10:41:08.543 INFO:teuthology.orchestra.run.smithi114.stdout: "default_value": "info", 2024-04-23T10:41:08.543 INFO:teuthology.orchestra.run.smithi114.stdout: "min": "", 2024-04-23T10:41:08.543 INFO:teuthology.orchestra.run.smithi114.stdout: "max": "", 2024-04-23T10:41:08.543 INFO:teuthology.orchestra.run.smithi114.stdout: "enum_allowed": [ 2024-04-23T10:41:08.543 INFO:teuthology.orchestra.run.smithi114.stdout: "", 2024-04-23T10:41:08.544 INFO:teuthology.orchestra.run.smithi114.stdout: "critical", 2024-04-23T10:41:08.544 INFO:teuthology.orchestra.run.smithi114.stdout: "debug", 2024-04-23T10:41:08.544 INFO:teuthology.orchestra.run.smithi114.stdout: "error", 2024-04-23T10:41:08.544 INFO:teuthology.orchestra.run.smithi114.stdout: "info", 2024-04-23T10:41:08.544 INFO:teuthology.orchestra.run.smithi114.stdout: "warning" 2024-04-23T10:41:08.544 INFO:teuthology.orchestra.run.smithi114.stdout: ], 2024-04-23T10:41:08.544 INFO:teuthology.orchestra.run.smithi114.stdout: "desc": "", 2024-04-23T10:41:08.544 INFO:teuthology.orchestra.run.smithi114.stdout: "long_desc": "", 2024-04-23T10:41:08.544 INFO:teuthology.orchestra.run.smithi114.stdout: "tags": [], 2024-04-23T10:41:08.544 INFO:teuthology.orchestra.run.smithi114.stdout: "see_also": [] 2024-04-23T10:41:08.545 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-23T10:41:08.545 INFO:teuthology.orchestra.run.smithi114.stdout: "log_to_file": { 2024-04-23T10:41:08.545 INFO:teuthology.orchestra.run.smithi114.stdout: "name": "log_to_file", 2024-04-23T10:41:08.545 INFO:teuthology.orchestra.run.smithi114.stdout: "type": "bool", 2024-04-23T10:41:08.545 INFO:teuthology.orchestra.run.smithi114.stdout: "level": "advanced", 2024-04-23T10:41:08.545 INFO:teuthology.orchestra.run.smithi114.stdout: "flags": 1, 2024-04-23T10:41:08.545 INFO:teuthology.orchestra.run.smithi114.stdout: "default_value": "False", 2024-04-23T10:41:08.545 INFO:teuthology.orchestra.run.smithi114.stdout: "min": "", 2024-04-23T10:41:08.545 INFO:teuthology.orchestra.run.smithi114.stdout: "max": "", 2024-04-23T10:41:08.545 INFO:teuthology.orchestra.run.smithi114.stdout: "enum_allowed": [], 2024-04-23T10:41:08.546 INFO:teuthology.orchestra.run.smithi114.stdout: "desc": "", 2024-04-23T10:41:08.546 INFO:teuthology.orchestra.run.smithi114.stdout: "long_desc": "", 2024-04-23T10:41:08.546 INFO:teuthology.orchestra.run.smithi114.stdout: "tags": [], 2024-04-23T10:41:08.546 INFO:teuthology.orchestra.run.smithi114.stdout: "see_also": [] 2024-04-23T10:41:08.546 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-23T10:41:08.546 INFO:teuthology.orchestra.run.smithi114.stdout: "roption1": { 2024-04-23T10:41:08.546 INFO:teuthology.orchestra.run.smithi114.stdout: "name": "roption1", 2024-04-23T10:41:08.546 INFO:teuthology.orchestra.run.smithi114.stdout: "type": "str", 2024-04-23T10:41:08.546 INFO:teuthology.orchestra.run.smithi114.stdout: "level": "advanced", 2024-04-23T10:41:08.547 INFO:teuthology.orchestra.run.smithi114.stdout: "flags": 0, 2024-04-23T10:41:08.547 INFO:teuthology.orchestra.run.smithi114.stdout: "default_value": "", 2024-04-23T10:41:08.547 INFO:teuthology.orchestra.run.smithi114.stdout: "min": "", 2024-04-23T10:41:08.547 INFO:teuthology.orchestra.run.smithi114.stdout: "max": "", 2024-04-23T10:41:08.547 INFO:teuthology.orchestra.run.smithi114.stdout: "enum_allowed": [], 2024-04-23T10:41:08.547 INFO:teuthology.orchestra.run.smithi114.stdout: "desc": "", 2024-04-23T10:41:08.547 INFO:teuthology.orchestra.run.smithi114.stdout: "long_desc": "", 2024-04-23T10:41:08.547 INFO:teuthology.orchestra.run.smithi114.stdout: "tags": [], 2024-04-23T10:41:08.547 INFO:teuthology.orchestra.run.smithi114.stdout: "see_also": [] 2024-04-23T10:41:08.548 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-23T10:41:08.548 INFO:teuthology.orchestra.run.smithi114.stdout: "roption2": { 2024-04-23T10:41:08.548 INFO:teuthology.orchestra.run.smithi114.stdout: "name": "roption2", 2024-04-23T10:41:08.548 INFO:teuthology.orchestra.run.smithi114.stdout: "type": "str", 2024-04-23T10:41:08.548 INFO:teuthology.orchestra.run.smithi114.stdout: "level": "advanced", 2024-04-23T10:41:08.548 INFO:teuthology.orchestra.run.smithi114.stdout: "flags": 0, 2024-04-23T10:41:08.548 INFO:teuthology.orchestra.run.smithi114.stdout: "default_value": "xyz", 2024-04-23T10:41:08.548 INFO:teuthology.orchestra.run.smithi114.stdout: "min": "", 2024-04-23T10:41:08.548 INFO:teuthology.orchestra.run.smithi114.stdout: "max": "", 2024-04-23T10:41:08.548 INFO:teuthology.orchestra.run.smithi114.stdout: "enum_allowed": [], 2024-04-23T10:41:08.549 INFO:teuthology.orchestra.run.smithi114.stdout: "desc": "", 2024-04-23T10:41:08.549 INFO:teuthology.orchestra.run.smithi114.stdout: "long_desc": "", 2024-04-23T10:41:08.549 INFO:teuthology.orchestra.run.smithi114.stdout: "tags": [], 2024-04-23T10:41:08.549 INFO:teuthology.orchestra.run.smithi114.stdout: "see_also": [] 2024-04-23T10:41:08.549 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-23T10:41:08.549 INFO:teuthology.orchestra.run.smithi114.stdout: "rwoption1": { 2024-04-23T10:41:08.549 INFO:teuthology.orchestra.run.smithi114.stdout: "name": "rwoption1", 2024-04-23T10:41:08.549 INFO:teuthology.orchestra.run.smithi114.stdout: "type": "str", 2024-04-23T10:41:08.549 INFO:teuthology.orchestra.run.smithi114.stdout: "level": "advanced", 2024-04-23T10:41:08.549 INFO:teuthology.orchestra.run.smithi114.stdout: "flags": 0, 2024-04-23T10:41:08.550 INFO:teuthology.orchestra.run.smithi114.stdout: "default_value": "", 2024-04-23T10:41:08.550 INFO:teuthology.orchestra.run.smithi114.stdout: "min": "", 2024-04-23T10:41:08.550 INFO:teuthology.orchestra.run.smithi114.stdout: "max": "", 2024-04-23T10:41:08.550 INFO:teuthology.orchestra.run.smithi114.stdout: "enum_allowed": [], 2024-04-23T10:41:08.550 INFO:teuthology.orchestra.run.smithi114.stdout: "desc": "", 2024-04-23T10:41:08.550 INFO:teuthology.orchestra.run.smithi114.stdout: "long_desc": "", 2024-04-23T10:41:08.550 INFO:teuthology.orchestra.run.smithi114.stdout: "tags": [], 2024-04-23T10:41:08.550 INFO:teuthology.orchestra.run.smithi114.stdout: "see_also": [] 2024-04-23T10:41:08.550 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-23T10:41:08.550 INFO:teuthology.orchestra.run.smithi114.stdout: "rwoption2": { 2024-04-23T10:41:08.551 INFO:teuthology.orchestra.run.smithi114.stdout: "name": "rwoption2", 2024-04-23T10:41:08.551 INFO:teuthology.orchestra.run.smithi114.stdout: "type": "int", 2024-04-23T10:41:08.551 INFO:teuthology.orchestra.run.smithi114.stdout: "level": "advanced", 2024-04-23T10:41:08.551 INFO:teuthology.orchestra.run.smithi114.stdout: "flags": 0, 2024-04-23T10:41:08.551 INFO:teuthology.orchestra.run.smithi114.stdout: "default_value": "", 2024-04-23T10:41:08.551 INFO:teuthology.orchestra.run.smithi114.stdout: "min": "", 2024-04-23T10:41:08.551 INFO:teuthology.orchestra.run.smithi114.stdout: "max": "", 2024-04-23T10:41:08.551 INFO:teuthology.orchestra.run.smithi114.stdout: "enum_allowed": [], 2024-04-23T10:41:08.551 INFO:teuthology.orchestra.run.smithi114.stdout: "desc": "", 2024-04-23T10:41:08.551 INFO:teuthology.orchestra.run.smithi114.stdout: "long_desc": "", 2024-04-23T10:41:08.552 INFO:teuthology.orchestra.run.smithi114.stdout: "tags": [], 2024-04-23T10:41:08.552 INFO:teuthology.orchestra.run.smithi114.stdout: "see_also": [] 2024-04-23T10:41:08.552 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-23T10:41:08.552 INFO:teuthology.orchestra.run.smithi114.stdout: "rwoption3": { 2024-04-23T10:41:08.552 INFO:teuthology.orchestra.run.smithi114.stdout: "name": "rwoption3", 2024-04-23T10:41:08.552 INFO:teuthology.orchestra.run.smithi114.stdout: "type": "float", 2024-04-23T10:41:08.552 INFO:teuthology.orchestra.run.smithi114.stdout: "level": "advanced", 2024-04-23T10:41:08.552 INFO:teuthology.orchestra.run.smithi114.stdout: "flags": 0, 2024-04-23T10:41:08.552 INFO:teuthology.orchestra.run.smithi114.stdout: "default_value": "", 2024-04-23T10:41:08.552 INFO:teuthology.orchestra.run.smithi114.stdout: "min": "", 2024-04-23T10:41:08.553 INFO:teuthology.orchestra.run.smithi114.stdout: "max": "", 2024-04-23T10:41:08.553 INFO:teuthology.orchestra.run.smithi114.stdout: "enum_allowed": [], 2024-04-23T10:41:08.553 INFO:teuthology.orchestra.run.smithi114.stdout: "desc": "", 2024-04-23T10:41:08.553 INFO:teuthology.orchestra.run.smithi114.stdout: "long_desc": "", 2024-04-23T10:41:08.553 INFO:teuthology.orchestra.run.smithi114.stdout: "tags": [], 2024-04-23T10:41:08.553 INFO:teuthology.orchestra.run.smithi114.stdout: "see_also": [] 2024-04-23T10:41:08.553 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-23T10:41:08.553 INFO:teuthology.orchestra.run.smithi114.stdout: "rwoption4": { 2024-04-23T10:41:08.553 INFO:teuthology.orchestra.run.smithi114.stdout: "name": "rwoption4", 2024-04-23T10:41:08.554 INFO:teuthology.orchestra.run.smithi114.stdout: "type": "str", 2024-04-23T10:41:08.554 INFO:teuthology.orchestra.run.smithi114.stdout: "level": "advanced", 2024-04-23T10:41:08.554 INFO:teuthology.orchestra.run.smithi114.stdout: "flags": 0, 2024-04-23T10:41:08.554 INFO:teuthology.orchestra.run.smithi114.stdout: "default_value": "", 2024-04-23T10:41:08.554 INFO:teuthology.orchestra.run.smithi114.stdout: "min": "", 2024-04-23T10:41:08.554 INFO:teuthology.orchestra.run.smithi114.stdout: "max": "", 2024-04-23T10:41:08.554 INFO:teuthology.orchestra.run.smithi114.stdout: "enum_allowed": [], 2024-04-23T10:41:08.554 INFO:teuthology.orchestra.run.smithi114.stdout: "desc": "", 2024-04-23T10:41:08.554 INFO:teuthology.orchestra.run.smithi114.stdout: "long_desc": "", 2024-04-23T10:41:08.554 INFO:teuthology.orchestra.run.smithi114.stdout: "tags": [], 2024-04-23T10:41:08.555 INFO:teuthology.orchestra.run.smithi114.stdout: "see_also": [] 2024-04-23T10:41:08.555 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-23T10:41:08.555 INFO:teuthology.orchestra.run.smithi114.stdout: "rwoption5": { 2024-04-23T10:41:08.555 INFO:teuthology.orchestra.run.smithi114.stdout: "name": "rwoption5", 2024-04-23T10:41:08.555 INFO:teuthology.orchestra.run.smithi114.stdout: "type": "bool", 2024-04-23T10:41:08.555 INFO:teuthology.orchestra.run.smithi114.stdout: "level": "advanced", 2024-04-23T10:41:08.555 INFO:teuthology.orchestra.run.smithi114.stdout: "flags": 0, 2024-04-23T10:41:08.555 INFO:teuthology.orchestra.run.smithi114.stdout: "default_value": "", 2024-04-23T10:41:08.555 INFO:teuthology.orchestra.run.smithi114.stdout: "min": "", 2024-04-23T10:41:08.556 INFO:teuthology.orchestra.run.smithi114.stdout: "max": "", 2024-04-23T10:41:08.556 INFO:teuthology.orchestra.run.smithi114.stdout: "enum_allowed": [], 2024-04-23T10:41:08.556 INFO:teuthology.orchestra.run.smithi114.stdout: "desc": "", 2024-04-23T10:41:08.556 INFO:teuthology.orchestra.run.smithi114.stdout: "long_desc": "", 2024-04-23T10:41:08.556 INFO:teuthology.orchestra.run.smithi114.stdout: "tags": [], 2024-04-23T10:41:08.556 INFO:teuthology.orchestra.run.smithi114.stdout: "see_also": [] 2024-04-23T10:41:08.556 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-23T10:41:08.556 INFO:teuthology.orchestra.run.smithi114.stdout: "rwoption6": { 2024-04-23T10:41:08.556 INFO:teuthology.orchestra.run.smithi114.stdout: "name": "rwoption6", 2024-04-23T10:41:08.556 INFO:teuthology.orchestra.run.smithi114.stdout: "type": "bool", 2024-04-23T10:41:08.557 INFO:teuthology.orchestra.run.smithi114.stdout: "level": "advanced", 2024-04-23T10:41:08.557 INFO:teuthology.orchestra.run.smithi114.stdout: "flags": 0, 2024-04-23T10:41:08.557 INFO:teuthology.orchestra.run.smithi114.stdout: "default_value": "True", 2024-04-23T10:41:08.557 INFO:teuthology.orchestra.run.smithi114.stdout: "min": "", 2024-04-23T10:41:08.557 INFO:teuthology.orchestra.run.smithi114.stdout: "max": "", 2024-04-23T10:41:08.557 INFO:teuthology.orchestra.run.smithi114.stdout: "enum_allowed": [], 2024-04-23T10:41:08.557 INFO:teuthology.orchestra.run.smithi114.stdout: "desc": "", 2024-04-23T10:41:08.557 INFO:teuthology.orchestra.run.smithi114.stdout: "long_desc": "", 2024-04-23T10:41:08.557 INFO:teuthology.orchestra.run.smithi114.stdout: "tags": [], 2024-04-23T10:41:08.557 INFO:teuthology.orchestra.run.smithi114.stdout: "see_also": [] 2024-04-23T10:41:08.558 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-23T10:41:08.558 INFO:teuthology.orchestra.run.smithi114.stdout: "rwoption7": { 2024-04-23T10:41:08.558 INFO:teuthology.orchestra.run.smithi114.stdout: "name": "rwoption7", 2024-04-23T10:41:08.558 INFO:teuthology.orchestra.run.smithi114.stdout: "type": "int", 2024-04-23T10:41:08.558 INFO:teuthology.orchestra.run.smithi114.stdout: "level": "advanced", 2024-04-23T10:41:08.558 INFO:teuthology.orchestra.run.smithi114.stdout: "flags": 0, 2024-04-23T10:41:08.558 INFO:teuthology.orchestra.run.smithi114.stdout: "default_value": "", 2024-04-23T10:41:08.558 INFO:teuthology.orchestra.run.smithi114.stdout: "min": "1", 2024-04-23T10:41:08.558 INFO:teuthology.orchestra.run.smithi114.stdout: "max": "42", 2024-04-23T10:41:08.558 INFO:teuthology.orchestra.run.smithi114.stdout: "enum_allowed": [], 2024-04-23T10:41:08.559 INFO:teuthology.orchestra.run.smithi114.stdout: "desc": "", 2024-04-23T10:41:08.559 INFO:teuthology.orchestra.run.smithi114.stdout: "long_desc": "", 2024-04-23T10:41:08.559 INFO:teuthology.orchestra.run.smithi114.stdout: "tags": [], 2024-04-23T10:41:08.559 INFO:teuthology.orchestra.run.smithi114.stdout: "see_also": [] 2024-04-23T10:41:08.559 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-23T10:41:08.559 INFO:teuthology.orchestra.run.smithi114.stdout: "testkey": { 2024-04-23T10:41:08.559 INFO:teuthology.orchestra.run.smithi114.stdout: "name": "testkey", 2024-04-23T10:41:08.559 INFO:teuthology.orchestra.run.smithi114.stdout: "type": "str", 2024-04-23T10:41:08.559 INFO:teuthology.orchestra.run.smithi114.stdout: "level": "advanced", 2024-04-23T10:41:08.559 INFO:teuthology.orchestra.run.smithi114.stdout: "flags": 0, 2024-04-23T10:41:08.560 INFO:teuthology.orchestra.run.smithi114.stdout: "default_value": "", 2024-04-23T10:41:08.560 INFO:teuthology.orchestra.run.smithi114.stdout: "min": "", 2024-04-23T10:41:08.560 INFO:teuthology.orchestra.run.smithi114.stdout: "max": "", 2024-04-23T10:41:08.560 INFO:teuthology.orchestra.run.smithi114.stdout: "enum_allowed": [], 2024-04-23T10:41:08.560 INFO:teuthology.orchestra.run.smithi114.stdout: "desc": "", 2024-04-23T10:41:08.560 INFO:teuthology.orchestra.run.smithi114.stdout: "long_desc": "", 2024-04-23T10:41:08.560 INFO:teuthology.orchestra.run.smithi114.stdout: "tags": [], 2024-04-23T10:41:08.560 INFO:teuthology.orchestra.run.smithi114.stdout: "see_also": [] 2024-04-23T10:41:08.560 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-23T10:41:08.560 INFO:teuthology.orchestra.run.smithi114.stdout: "testlkey": { 2024-04-23T10:41:08.561 INFO:teuthology.orchestra.run.smithi114.stdout: "name": "testlkey", 2024-04-23T10:41:08.561 INFO:teuthology.orchestra.run.smithi114.stdout: "type": "str", 2024-04-23T10:41:08.561 INFO:teuthology.orchestra.run.smithi114.stdout: "level": "advanced", 2024-04-23T10:41:08.561 INFO:teuthology.orchestra.run.smithi114.stdout: "flags": 0, 2024-04-23T10:41:08.561 INFO:teuthology.orchestra.run.smithi114.stdout: "default_value": "", 2024-04-23T10:41:08.561 INFO:teuthology.orchestra.run.smithi114.stdout: "min": "", 2024-04-23T10:41:08.561 INFO:teuthology.orchestra.run.smithi114.stdout: "max": "", 2024-04-23T10:41:08.561 INFO:teuthology.orchestra.run.smithi114.stdout: "enum_allowed": [], 2024-04-23T10:41:08.561 INFO:teuthology.orchestra.run.smithi114.stdout: "desc": "", 2024-04-23T10:41:08.562 INFO:teuthology.orchestra.run.smithi114.stdout: "long_desc": "", 2024-04-23T10:41:08.562 INFO:teuthology.orchestra.run.smithi114.stdout: "tags": [], 2024-04-23T10:41:08.562 INFO:teuthology.orchestra.run.smithi114.stdout: "see_also": [] 2024-04-23T10:41:08.562 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-23T10:41:08.562 INFO:teuthology.orchestra.run.smithi114.stdout: "testnewline": { 2024-04-23T10:41:08.562 INFO:teuthology.orchestra.run.smithi114.stdout: "name": "testnewline", 2024-04-23T10:41:08.562 INFO:teuthology.orchestra.run.smithi114.stdout: "type": "str", 2024-04-23T10:41:08.562 INFO:teuthology.orchestra.run.smithi114.stdout: "level": "advanced", 2024-04-23T10:41:08.562 INFO:teuthology.orchestra.run.smithi114.stdout: "flags": 0, 2024-04-23T10:41:08.562 INFO:teuthology.orchestra.run.smithi114.stdout: "default_value": "", 2024-04-23T10:41:08.563 INFO:teuthology.orchestra.run.smithi114.stdout: "min": "", 2024-04-23T10:41:08.563 INFO:teuthology.orchestra.run.smithi114.stdout: "max": "", 2024-04-23T10:41:08.563 INFO:teuthology.orchestra.run.smithi114.stdout: "enum_allowed": [], 2024-04-23T10:41:08.563 INFO:teuthology.orchestra.run.smithi114.stdout: "desc": "", 2024-04-23T10:41:08.563 INFO:teuthology.orchestra.run.smithi114.stdout: "long_desc": "", 2024-04-23T10:41:08.563 INFO:teuthology.orchestra.run.smithi114.stdout: "tags": [], 2024-04-23T10:41:08.563 INFO:teuthology.orchestra.run.smithi114.stdout: "see_also": [] 2024-04-23T10:41:08.563 INFO:teuthology.orchestra.run.smithi114.stdout: } 2024-04-23T10:41:08.563 INFO:teuthology.orchestra.run.smithi114.stdout: } 2024-04-23T10:41:08.563 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-23T10:41:08.564 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-04-23T10:41:08.564 INFO:teuthology.orchestra.run.smithi114.stdout: "name": "snap_schedule", 2024-04-23T10:41:08.564 INFO:teuthology.orchestra.run.smithi114.stdout: "can_run": true, 2024-04-23T10:41:08.564 INFO:teuthology.orchestra.run.smithi114.stdout: "error_string": "", 2024-04-23T10:41:08.564 INFO:teuthology.orchestra.run.smithi114.stdout: "module_options": { 2024-04-23T10:41:08.564 INFO:teuthology.orchestra.run.smithi114.stdout: "allow_m_granularity": { 2024-04-23T10:41:08.564 INFO:teuthology.orchestra.run.smithi114.stdout: "name": "allow_m_granularity", 2024-04-23T10:41:08.564 INFO:teuthology.orchestra.run.smithi114.stdout: "type": "bool", 2024-04-23T10:41:08.564 INFO:teuthology.orchestra.run.smithi114.stdout: "level": "advanced", 2024-04-23T10:41:08.564 INFO:teuthology.orchestra.run.smithi114.stdout: "flags": 1, 2024-04-23T10:41:08.565 INFO:teuthology.orchestra.run.smithi114.stdout: "default_value": "False", 2024-04-23T10:41:08.565 INFO:teuthology.orchestra.run.smithi114.stdout: "min": "", 2024-04-23T10:41:08.565 INFO:teuthology.orchestra.run.smithi114.stdout: "max": "", 2024-04-23T10:41:08.565 INFO:teuthology.orchestra.run.smithi114.stdout: "enum_allowed": [], 2024-04-23T10:41:08.565 INFO:teuthology.orchestra.run.smithi114.stdout: "desc": "allow minute scheduled snapshots", 2024-04-23T10:41:08.565 INFO:teuthology.orchestra.run.smithi114.stdout: "long_desc": "", 2024-04-23T10:41:08.565 INFO:teuthology.orchestra.run.smithi114.stdout: "tags": [], 2024-04-23T10:41:08.565 INFO:teuthology.orchestra.run.smithi114.stdout: "see_also": [] 2024-04-23T10:41:08.565 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-23T10:41:08.565 INFO:teuthology.orchestra.run.smithi114.stdout: "dump_on_update": { 2024-04-23T10:41:08.566 INFO:teuthology.orchestra.run.smithi114.stdout: "name": "dump_on_update", 2024-04-23T10:41:08.566 INFO:teuthology.orchestra.run.smithi114.stdout: "type": "bool", 2024-04-23T10:41:08.566 INFO:teuthology.orchestra.run.smithi114.stdout: "level": "advanced", 2024-04-23T10:41:08.566 INFO:teuthology.orchestra.run.smithi114.stdout: "flags": 1, 2024-04-23T10:41:08.566 INFO:teuthology.orchestra.run.smithi114.stdout: "default_value": "False", 2024-04-23T10:41:08.566 INFO:teuthology.orchestra.run.smithi114.stdout: "min": "", 2024-04-23T10:41:08.566 INFO:teuthology.orchestra.run.smithi114.stdout: "max": "", 2024-04-23T10:41:08.566 INFO:teuthology.orchestra.run.smithi114.stdout: "enum_allowed": [], 2024-04-23T10:41:08.566 INFO:teuthology.orchestra.run.smithi114.stdout: "desc": "dump database to debug log on update", 2024-04-23T10:41:08.567 INFO:teuthology.orchestra.run.smithi114.stdout: "long_desc": "", 2024-04-23T10:41:08.567 INFO:teuthology.orchestra.run.smithi114.stdout: "tags": [], 2024-04-23T10:41:08.567 INFO:teuthology.orchestra.run.smithi114.stdout: "see_also": [] 2024-04-23T10:41:08.567 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-23T10:41:08.567 INFO:teuthology.orchestra.run.smithi114.stdout: "log_level": { 2024-04-23T10:41:08.567 INFO:teuthology.orchestra.run.smithi114.stdout: "name": "log_level", 2024-04-23T10:41:08.567 INFO:teuthology.orchestra.run.smithi114.stdout: "type": "str", 2024-04-23T10:41:08.567 INFO:teuthology.orchestra.run.smithi114.stdout: "level": "advanced", 2024-04-23T10:41:08.567 INFO:teuthology.orchestra.run.smithi114.stdout: "flags": 1, 2024-04-23T10:41:08.568 INFO:teuthology.orchestra.run.smithi114.stdout: "default_value": "", 2024-04-23T10:41:08.568 INFO:teuthology.orchestra.run.smithi114.stdout: "min": "", 2024-04-23T10:41:08.568 INFO:teuthology.orchestra.run.smithi114.stdout: "max": "", 2024-04-23T10:41:08.568 INFO:teuthology.orchestra.run.smithi114.stdout: "enum_allowed": [ 2024-04-23T10:41:08.568 INFO:teuthology.orchestra.run.smithi114.stdout: "", 2024-04-23T10:41:08.568 INFO:teuthology.orchestra.run.smithi114.stdout: "critical", 2024-04-23T10:41:08.568 INFO:teuthology.orchestra.run.smithi114.stdout: "debug", 2024-04-23T10:41:08.568 INFO:teuthology.orchestra.run.smithi114.stdout: "error", 2024-04-23T10:41:08.568 INFO:teuthology.orchestra.run.smithi114.stdout: "info", 2024-04-23T10:41:08.568 INFO:teuthology.orchestra.run.smithi114.stdout: "warning" 2024-04-23T10:41:08.569 INFO:teuthology.orchestra.run.smithi114.stdout: ], 2024-04-23T10:41:08.569 INFO:teuthology.orchestra.run.smithi114.stdout: "desc": "", 2024-04-23T10:41:08.569 INFO:teuthology.orchestra.run.smithi114.stdout: "long_desc": "", 2024-04-23T10:41:08.569 INFO:teuthology.orchestra.run.smithi114.stdout: "tags": [], 2024-04-23T10:41:08.569 INFO:teuthology.orchestra.run.smithi114.stdout: "see_also": [] 2024-04-23T10:41:08.569 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-23T10:41:08.569 INFO:teuthology.orchestra.run.smithi114.stdout: "log_to_cluster": { 2024-04-23T10:41:08.569 INFO:teuthology.orchestra.run.smithi114.stdout: "name": "log_to_cluster", 2024-04-23T10:41:08.569 INFO:teuthology.orchestra.run.smithi114.stdout: "type": "bool", 2024-04-23T10:41:08.569 INFO:teuthology.orchestra.run.smithi114.stdout: "level": "advanced", 2024-04-23T10:41:08.570 INFO:teuthology.orchestra.run.smithi114.stdout: "flags": 1, 2024-04-23T10:41:08.570 INFO:teuthology.orchestra.run.smithi114.stdout: "default_value": "False", 2024-04-23T10:41:08.570 INFO:teuthology.orchestra.run.smithi114.stdout: "min": "", 2024-04-23T10:41:08.570 INFO:teuthology.orchestra.run.smithi114.stdout: "max": "", 2024-04-23T10:41:08.570 INFO:teuthology.orchestra.run.smithi114.stdout: "enum_allowed": [], 2024-04-23T10:41:08.570 INFO:teuthology.orchestra.run.smithi114.stdout: "desc": "", 2024-04-23T10:41:08.570 INFO:teuthology.orchestra.run.smithi114.stdout: "long_desc": "", 2024-04-23T10:41:08.570 INFO:teuthology.orchestra.run.smithi114.stdout: "tags": [], 2024-04-23T10:41:08.570 INFO:teuthology.orchestra.run.smithi114.stdout: "see_also": [] 2024-04-23T10:41:08.571 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-23T10:41:08.571 INFO:teuthology.orchestra.run.smithi114.stdout: "log_to_cluster_level": { 2024-04-23T10:41:08.571 INFO:teuthology.orchestra.run.smithi114.stdout: "name": "log_to_cluster_level", 2024-04-23T10:41:08.571 INFO:teuthology.orchestra.run.smithi114.stdout: "type": "str", 2024-04-23T10:41:08.571 INFO:teuthology.orchestra.run.smithi114.stdout: "level": "advanced", 2024-04-23T10:41:08.571 INFO:teuthology.orchestra.run.smithi114.stdout: "flags": 1, 2024-04-23T10:41:08.571 INFO:teuthology.orchestra.run.smithi114.stdout: "default_value": "info", 2024-04-23T10:41:08.571 INFO:teuthology.orchestra.run.smithi114.stdout: "min": "", 2024-04-23T10:41:08.571 INFO:teuthology.orchestra.run.smithi114.stdout: "max": "", 2024-04-23T10:41:08.571 INFO:teuthology.orchestra.run.smithi114.stdout: "enum_allowed": [ 2024-04-23T10:41:08.572 INFO:teuthology.orchestra.run.smithi114.stdout: "", 2024-04-23T10:41:08.572 INFO:teuthology.orchestra.run.smithi114.stdout: "critical", 2024-04-23T10:41:08.572 INFO:teuthology.orchestra.run.smithi114.stdout: "debug", 2024-04-23T10:41:08.572 INFO:teuthology.orchestra.run.smithi114.stdout: "error", 2024-04-23T10:41:08.572 INFO:teuthology.orchestra.run.smithi114.stdout: "info", 2024-04-23T10:41:08.572 INFO:teuthology.orchestra.run.smithi114.stdout: "warning" 2024-04-23T10:41:08.572 INFO:teuthology.orchestra.run.smithi114.stdout: ], 2024-04-23T10:41:08.572 INFO:teuthology.orchestra.run.smithi114.stdout: "desc": "", 2024-04-23T10:41:08.572 INFO:teuthology.orchestra.run.smithi114.stdout: "long_desc": "", 2024-04-23T10:41:08.572 INFO:teuthology.orchestra.run.smithi114.stdout: "tags": [], 2024-04-23T10:41:08.573 INFO:teuthology.orchestra.run.smithi114.stdout: "see_also": [] 2024-04-23T10:41:08.573 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-23T10:41:08.573 INFO:teuthology.orchestra.run.smithi114.stdout: "log_to_file": { 2024-04-23T10:41:08.573 INFO:teuthology.orchestra.run.smithi114.stdout: "name": "log_to_file", 2024-04-23T10:41:08.573 INFO:teuthology.orchestra.run.smithi114.stdout: "type": "bool", 2024-04-23T10:41:08.573 INFO:teuthology.orchestra.run.smithi114.stdout: "level": "advanced", 2024-04-23T10:41:08.573 INFO:teuthology.orchestra.run.smithi114.stdout: "flags": 1, 2024-04-23T10:41:08.573 INFO:teuthology.orchestra.run.smithi114.stdout: "default_value": "False", 2024-04-23T10:41:08.573 INFO:teuthology.orchestra.run.smithi114.stdout: "min": "", 2024-04-23T10:41:08.573 INFO:teuthology.orchestra.run.smithi114.stdout: "max": "", 2024-04-23T10:41:08.574 INFO:teuthology.orchestra.run.smithi114.stdout: "enum_allowed": [], 2024-04-23T10:41:08.574 INFO:teuthology.orchestra.run.smithi114.stdout: "desc": "", 2024-04-23T10:41:08.574 INFO:teuthology.orchestra.run.smithi114.stdout: "long_desc": "", 2024-04-23T10:41:08.574 INFO:teuthology.orchestra.run.smithi114.stdout: "tags": [], 2024-04-23T10:41:08.574 INFO:teuthology.orchestra.run.smithi114.stdout: "see_also": [] 2024-04-23T10:41:08.574 INFO:teuthology.orchestra.run.smithi114.stdout: } 2024-04-23T10:41:08.574 INFO:teuthology.orchestra.run.smithi114.stdout: } 2024-04-23T10:41:08.574 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-23T10:41:08.574 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-04-23T10:41:08.575 INFO:teuthology.orchestra.run.smithi114.stdout: "name": "stats", 2024-04-23T10:41:08.575 INFO:teuthology.orchestra.run.smithi114.stdout: "can_run": true, 2024-04-23T10:41:08.575 INFO:teuthology.orchestra.run.smithi114.stdout: "error_string": "", 2024-04-23T10:41:08.575 INFO:teuthology.orchestra.run.smithi114.stdout: "module_options": { 2024-04-23T10:41:08.575 INFO:teuthology.orchestra.run.smithi114.stdout: "log_level": { 2024-04-23T10:41:08.575 INFO:teuthology.orchestra.run.smithi114.stdout: "name": "log_level", 2024-04-23T10:41:08.575 INFO:teuthology.orchestra.run.smithi114.stdout: "type": "str", 2024-04-23T10:41:08.575 INFO:teuthology.orchestra.run.smithi114.stdout: "level": "advanced", 2024-04-23T10:41:08.575 INFO:teuthology.orchestra.run.smithi114.stdout: "flags": 1, 2024-04-23T10:41:08.575 INFO:teuthology.orchestra.run.smithi114.stdout: "default_value": "", 2024-04-23T10:41:08.576 INFO:teuthology.orchestra.run.smithi114.stdout: "min": "", 2024-04-23T10:41:08.576 INFO:teuthology.orchestra.run.smithi114.stdout: "max": "", 2024-04-23T10:41:08.576 INFO:teuthology.orchestra.run.smithi114.stdout: "enum_allowed": [ 2024-04-23T10:41:08.576 INFO:teuthology.orchestra.run.smithi114.stdout: "", 2024-04-23T10:41:08.576 INFO:teuthology.orchestra.run.smithi114.stdout: "critical", 2024-04-23T10:41:08.576 INFO:teuthology.orchestra.run.smithi114.stdout: "debug", 2024-04-23T10:41:08.576 INFO:teuthology.orchestra.run.smithi114.stdout: "error", 2024-04-23T10:41:08.576 INFO:teuthology.orchestra.run.smithi114.stdout: "info", 2024-04-23T10:41:08.576 INFO:teuthology.orchestra.run.smithi114.stdout: "warning" 2024-04-23T10:41:08.576 INFO:teuthology.orchestra.run.smithi114.stdout: ], 2024-04-23T10:41:08.577 INFO:teuthology.orchestra.run.smithi114.stdout: "desc": "", 2024-04-23T10:41:08.577 INFO:teuthology.orchestra.run.smithi114.stdout: "long_desc": "", 2024-04-23T10:41:08.577 INFO:teuthology.orchestra.run.smithi114.stdout: "tags": [], 2024-04-23T10:41:08.577 INFO:teuthology.orchestra.run.smithi114.stdout: "see_also": [] 2024-04-23T10:41:08.577 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-23T10:41:08.577 INFO:teuthology.orchestra.run.smithi114.stdout: "log_to_cluster": { 2024-04-23T10:41:08.577 INFO:teuthology.orchestra.run.smithi114.stdout: "name": "log_to_cluster", 2024-04-23T10:41:08.577 INFO:teuthology.orchestra.run.smithi114.stdout: "type": "bool", 2024-04-23T10:41:08.577 INFO:teuthology.orchestra.run.smithi114.stdout: "level": "advanced", 2024-04-23T10:41:08.578 INFO:teuthology.orchestra.run.smithi114.stdout: "flags": 1, 2024-04-23T10:41:08.578 INFO:teuthology.orchestra.run.smithi114.stdout: "default_value": "False", 2024-04-23T10:41:08.578 INFO:teuthology.orchestra.run.smithi114.stdout: "min": "", 2024-04-23T10:41:08.578 INFO:teuthology.orchestra.run.smithi114.stdout: "max": "", 2024-04-23T10:41:08.578 INFO:teuthology.orchestra.run.smithi114.stdout: "enum_allowed": [], 2024-04-23T10:41:08.578 INFO:teuthology.orchestra.run.smithi114.stdout: "desc": "", 2024-04-23T10:41:08.578 INFO:teuthology.orchestra.run.smithi114.stdout: "long_desc": "", 2024-04-23T10:41:08.578 INFO:teuthology.orchestra.run.smithi114.stdout: "tags": [], 2024-04-23T10:41:08.578 INFO:teuthology.orchestra.run.smithi114.stdout: "see_also": [] 2024-04-23T10:41:08.578 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-23T10:41:08.579 INFO:teuthology.orchestra.run.smithi114.stdout: "log_to_cluster_level": { 2024-04-23T10:41:08.579 INFO:teuthology.orchestra.run.smithi114.stdout: "name": "log_to_cluster_level", 2024-04-23T10:41:08.579 INFO:teuthology.orchestra.run.smithi114.stdout: "type": "str", 2024-04-23T10:41:08.579 INFO:teuthology.orchestra.run.smithi114.stdout: "level": "advanced", 2024-04-23T10:41:08.579 INFO:teuthology.orchestra.run.smithi114.stdout: "flags": 1, 2024-04-23T10:41:08.579 INFO:teuthology.orchestra.run.smithi114.stdout: "default_value": "info", 2024-04-23T10:41:08.579 INFO:teuthology.orchestra.run.smithi114.stdout: "min": "", 2024-04-23T10:41:08.579 INFO:teuthology.orchestra.run.smithi114.stdout: "max": "", 2024-04-23T10:41:08.579 INFO:teuthology.orchestra.run.smithi114.stdout: "enum_allowed": [ 2024-04-23T10:41:08.579 INFO:teuthology.orchestra.run.smithi114.stdout: "", 2024-04-23T10:41:08.580 INFO:teuthology.orchestra.run.smithi114.stdout: "critical", 2024-04-23T10:41:08.580 INFO:teuthology.orchestra.run.smithi114.stdout: "debug", 2024-04-23T10:41:08.580 INFO:teuthology.orchestra.run.smithi114.stdout: "error", 2024-04-23T10:41:08.580 INFO:teuthology.orchestra.run.smithi114.stdout: "info", 2024-04-23T10:41:08.580 INFO:teuthology.orchestra.run.smithi114.stdout: "warning" 2024-04-23T10:41:08.580 INFO:teuthology.orchestra.run.smithi114.stdout: ], 2024-04-23T10:41:08.580 INFO:teuthology.orchestra.run.smithi114.stdout: "desc": "", 2024-04-23T10:41:08.580 INFO:teuthology.orchestra.run.smithi114.stdout: "long_desc": "", 2024-04-23T10:41:08.580 INFO:teuthology.orchestra.run.smithi114.stdout: "tags": [], 2024-04-23T10:41:08.580 INFO:teuthology.orchestra.run.smithi114.stdout: "see_also": [] 2024-04-23T10:41:08.581 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-23T10:41:08.581 INFO:teuthology.orchestra.run.smithi114.stdout: "log_to_file": { 2024-04-23T10:41:08.581 INFO:teuthology.orchestra.run.smithi114.stdout: "name": "log_to_file", 2024-04-23T10:41:08.581 INFO:teuthology.orchestra.run.smithi114.stdout: "type": "bool", 2024-04-23T10:41:08.581 INFO:teuthology.orchestra.run.smithi114.stdout: "level": "advanced", 2024-04-23T10:41:08.581 INFO:teuthology.orchestra.run.smithi114.stdout: "flags": 1, 2024-04-23T10:41:08.581 INFO:teuthology.orchestra.run.smithi114.stdout: "default_value": "False", 2024-04-23T10:41:08.581 INFO:teuthology.orchestra.run.smithi114.stdout: "min": "", 2024-04-23T10:41:08.581 INFO:teuthology.orchestra.run.smithi114.stdout: "max": "", 2024-04-23T10:41:08.582 INFO:teuthology.orchestra.run.smithi114.stdout: "enum_allowed": [], 2024-04-23T10:41:08.582 INFO:teuthology.orchestra.run.smithi114.stdout: "desc": "", 2024-04-23T10:41:08.582 INFO:teuthology.orchestra.run.smithi114.stdout: "long_desc": "", 2024-04-23T10:41:08.582 INFO:teuthology.orchestra.run.smithi114.stdout: "tags": [], 2024-04-23T10:41:08.582 INFO:teuthology.orchestra.run.smithi114.stdout: "see_also": [] 2024-04-23T10:41:08.582 INFO:teuthology.orchestra.run.smithi114.stdout: } 2024-04-23T10:41:08.582 INFO:teuthology.orchestra.run.smithi114.stdout: } 2024-04-23T10:41:08.582 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-23T10:41:08.582 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-04-23T10:41:08.582 INFO:teuthology.orchestra.run.smithi114.stdout: "name": "status", 2024-04-23T10:41:08.583 INFO:teuthology.orchestra.run.smithi114.stdout: "can_run": true, 2024-04-23T10:41:08.583 INFO:teuthology.orchestra.run.smithi114.stdout: "error_string": "", 2024-04-23T10:41:08.583 INFO:teuthology.orchestra.run.smithi114.stdout: "module_options": { 2024-04-23T10:41:08.583 INFO:teuthology.orchestra.run.smithi114.stdout: "log_level": { 2024-04-23T10:41:08.583 INFO:teuthology.orchestra.run.smithi114.stdout: "name": "log_level", 2024-04-23T10:41:08.583 INFO:teuthology.orchestra.run.smithi114.stdout: "type": "str", 2024-04-23T10:41:08.583 INFO:teuthology.orchestra.run.smithi114.stdout: "level": "advanced", 2024-04-23T10:41:08.583 INFO:teuthology.orchestra.run.smithi114.stdout: "flags": 1, 2024-04-23T10:41:08.583 INFO:teuthology.orchestra.run.smithi114.stdout: "default_value": "", 2024-04-23T10:41:08.583 INFO:teuthology.orchestra.run.smithi114.stdout: "min": "", 2024-04-23T10:41:08.584 INFO:teuthology.orchestra.run.smithi114.stdout: "max": "", 2024-04-23T10:41:08.584 INFO:teuthology.orchestra.run.smithi114.stdout: "enum_allowed": [ 2024-04-23T10:41:08.584 INFO:teuthology.orchestra.run.smithi114.stdout: "", 2024-04-23T10:41:08.584 INFO:teuthology.orchestra.run.smithi114.stdout: "critical", 2024-04-23T10:41:08.584 INFO:teuthology.orchestra.run.smithi114.stdout: "debug", 2024-04-23T10:41:08.584 INFO:teuthology.orchestra.run.smithi114.stdout: "error", 2024-04-23T10:41:08.584 INFO:teuthology.orchestra.run.smithi114.stdout: "info", 2024-04-23T10:41:08.584 INFO:teuthology.orchestra.run.smithi114.stdout: "warning" 2024-04-23T10:41:08.584 INFO:teuthology.orchestra.run.smithi114.stdout: ], 2024-04-23T10:41:08.584 INFO:teuthology.orchestra.run.smithi114.stdout: "desc": "", 2024-04-23T10:41:08.585 INFO:teuthology.orchestra.run.smithi114.stdout: "long_desc": "", 2024-04-23T10:41:08.585 INFO:teuthology.orchestra.run.smithi114.stdout: "tags": [], 2024-04-23T10:41:08.585 INFO:teuthology.orchestra.run.smithi114.stdout: "see_also": [] 2024-04-23T10:41:08.585 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-23T10:41:08.585 INFO:teuthology.orchestra.run.smithi114.stdout: "log_to_cluster": { 2024-04-23T10:41:08.585 INFO:teuthology.orchestra.run.smithi114.stdout: "name": "log_to_cluster", 2024-04-23T10:41:08.585 INFO:teuthology.orchestra.run.smithi114.stdout: "type": "bool", 2024-04-23T10:41:08.585 INFO:teuthology.orchestra.run.smithi114.stdout: "level": "advanced", 2024-04-23T10:41:08.585 INFO:teuthology.orchestra.run.smithi114.stdout: "flags": 1, 2024-04-23T10:41:08.585 INFO:teuthology.orchestra.run.smithi114.stdout: "default_value": "False", 2024-04-23T10:41:08.586 INFO:teuthology.orchestra.run.smithi114.stdout: "min": "", 2024-04-23T10:41:08.586 INFO:teuthology.orchestra.run.smithi114.stdout: "max": "", 2024-04-23T10:41:08.586 INFO:teuthology.orchestra.run.smithi114.stdout: "enum_allowed": [], 2024-04-23T10:41:08.586 INFO:teuthology.orchestra.run.smithi114.stdout: "desc": "", 2024-04-23T10:41:08.586 INFO:teuthology.orchestra.run.smithi114.stdout: "long_desc": "", 2024-04-23T10:41:08.586 INFO:teuthology.orchestra.run.smithi114.stdout: "tags": [], 2024-04-23T10:41:08.586 INFO:teuthology.orchestra.run.smithi114.stdout: "see_also": [] 2024-04-23T10:41:08.586 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-23T10:41:08.586 INFO:teuthology.orchestra.run.smithi114.stdout: "log_to_cluster_level": { 2024-04-23T10:41:08.586 INFO:teuthology.orchestra.run.smithi114.stdout: "name": "log_to_cluster_level", 2024-04-23T10:41:08.587 INFO:teuthology.orchestra.run.smithi114.stdout: "type": "str", 2024-04-23T10:41:08.587 INFO:teuthology.orchestra.run.smithi114.stdout: "level": "advanced", 2024-04-23T10:41:08.587 INFO:teuthology.orchestra.run.smithi114.stdout: "flags": 1, 2024-04-23T10:41:08.587 INFO:teuthology.orchestra.run.smithi114.stdout: "default_value": "info", 2024-04-23T10:41:08.587 INFO:teuthology.orchestra.run.smithi114.stdout: "min": "", 2024-04-23T10:41:08.587 INFO:teuthology.orchestra.run.smithi114.stdout: "max": "", 2024-04-23T10:41:08.587 INFO:teuthology.orchestra.run.smithi114.stdout: "enum_allowed": [ 2024-04-23T10:41:08.587 INFO:teuthology.orchestra.run.smithi114.stdout: "", 2024-04-23T10:41:08.587 INFO:teuthology.orchestra.run.smithi114.stdout: "critical", 2024-04-23T10:41:08.587 INFO:teuthology.orchestra.run.smithi114.stdout: "debug", 2024-04-23T10:41:08.588 INFO:teuthology.orchestra.run.smithi114.stdout: "error", 2024-04-23T10:41:08.588 INFO:teuthology.orchestra.run.smithi114.stdout: "info", 2024-04-23T10:41:08.588 INFO:teuthology.orchestra.run.smithi114.stdout: "warning" 2024-04-23T10:41:08.588 INFO:teuthology.orchestra.run.smithi114.stdout: ], 2024-04-23T10:41:08.588 INFO:teuthology.orchestra.run.smithi114.stdout: "desc": "", 2024-04-23T10:41:08.588 INFO:teuthology.orchestra.run.smithi114.stdout: "long_desc": "", 2024-04-23T10:41:08.588 INFO:teuthology.orchestra.run.smithi114.stdout: "tags": [], 2024-04-23T10:41:08.588 INFO:teuthology.orchestra.run.smithi114.stdout: "see_also": [] 2024-04-23T10:41:08.588 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-23T10:41:08.589 INFO:teuthology.orchestra.run.smithi114.stdout: "log_to_file": { 2024-04-23T10:41:08.589 INFO:teuthology.orchestra.run.smithi114.stdout: "name": "log_to_file", 2024-04-23T10:41:08.589 INFO:teuthology.orchestra.run.smithi114.stdout: "type": "bool", 2024-04-23T10:41:08.589 INFO:teuthology.orchestra.run.smithi114.stdout: "level": "advanced", 2024-04-23T10:41:08.589 INFO:teuthology.orchestra.run.smithi114.stdout: "flags": 1, 2024-04-23T10:41:08.589 INFO:teuthology.orchestra.run.smithi114.stdout: "default_value": "False", 2024-04-23T10:41:08.589 INFO:teuthology.orchestra.run.smithi114.stdout: "min": "", 2024-04-23T10:41:08.589 INFO:teuthology.orchestra.run.smithi114.stdout: "max": "", 2024-04-23T10:41:08.589 INFO:teuthology.orchestra.run.smithi114.stdout: "enum_allowed": [], 2024-04-23T10:41:08.589 INFO:teuthology.orchestra.run.smithi114.stdout: "desc": "", 2024-04-23T10:41:08.590 INFO:teuthology.orchestra.run.smithi114.stdout: "long_desc": "", 2024-04-23T10:41:08.590 INFO:teuthology.orchestra.run.smithi114.stdout: "tags": [], 2024-04-23T10:41:08.590 INFO:teuthology.orchestra.run.smithi114.stdout: "see_also": [] 2024-04-23T10:41:08.590 INFO:teuthology.orchestra.run.smithi114.stdout: } 2024-04-23T10:41:08.590 INFO:teuthology.orchestra.run.smithi114.stdout: } 2024-04-23T10:41:08.590 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-23T10:41:08.590 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-04-23T10:41:08.590 INFO:teuthology.orchestra.run.smithi114.stdout: "name": "telegraf", 2024-04-23T10:41:08.590 INFO:teuthology.orchestra.run.smithi114.stdout: "can_run": true, 2024-04-23T10:41:08.591 INFO:teuthology.orchestra.run.smithi114.stdout: "error_string": "", 2024-04-23T10:41:08.591 INFO:teuthology.orchestra.run.smithi114.stdout: "module_options": { 2024-04-23T10:41:08.591 INFO:teuthology.orchestra.run.smithi114.stdout: "address": { 2024-04-23T10:41:08.591 INFO:teuthology.orchestra.run.smithi114.stdout: "name": "address", 2024-04-23T10:41:08.591 INFO:teuthology.orchestra.run.smithi114.stdout: "type": "str", 2024-04-23T10:41:08.591 INFO:teuthology.orchestra.run.smithi114.stdout: "level": "advanced", 2024-04-23T10:41:08.591 INFO:teuthology.orchestra.run.smithi114.stdout: "flags": 0, 2024-04-23T10:41:08.591 INFO:teuthology.orchestra.run.smithi114.stdout: "default_value": "unixgram:///tmp/telegraf.sock", 2024-04-23T10:41:08.591 INFO:teuthology.orchestra.run.smithi114.stdout: "min": "", 2024-04-23T10:41:08.591 INFO:teuthology.orchestra.run.smithi114.stdout: "max": "", 2024-04-23T10:41:08.592 INFO:teuthology.orchestra.run.smithi114.stdout: "enum_allowed": [], 2024-04-23T10:41:08.592 INFO:teuthology.orchestra.run.smithi114.stdout: "desc": "", 2024-04-23T10:41:08.592 INFO:teuthology.orchestra.run.smithi114.stdout: "long_desc": "", 2024-04-23T10:41:08.592 INFO:teuthology.orchestra.run.smithi114.stdout: "tags": [], 2024-04-23T10:41:08.592 INFO:teuthology.orchestra.run.smithi114.stdout: "see_also": [] 2024-04-23T10:41:08.592 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-23T10:41:08.592 INFO:teuthology.orchestra.run.smithi114.stdout: "interval": { 2024-04-23T10:41:08.592 INFO:teuthology.orchestra.run.smithi114.stdout: "name": "interval", 2024-04-23T10:41:08.592 INFO:teuthology.orchestra.run.smithi114.stdout: "type": "secs", 2024-04-23T10:41:08.592 INFO:teuthology.orchestra.run.smithi114.stdout: "level": "advanced", 2024-04-23T10:41:08.593 INFO:teuthology.orchestra.run.smithi114.stdout: "flags": 0, 2024-04-23T10:41:08.593 INFO:teuthology.orchestra.run.smithi114.stdout: "default_value": "15", 2024-04-23T10:41:08.593 INFO:teuthology.orchestra.run.smithi114.stdout: "min": "", 2024-04-23T10:41:08.593 INFO:teuthology.orchestra.run.smithi114.stdout: "max": "", 2024-04-23T10:41:08.593 INFO:teuthology.orchestra.run.smithi114.stdout: "enum_allowed": [], 2024-04-23T10:41:08.593 INFO:teuthology.orchestra.run.smithi114.stdout: "desc": "", 2024-04-23T10:41:08.593 INFO:teuthology.orchestra.run.smithi114.stdout: "long_desc": "", 2024-04-23T10:41:08.593 INFO:teuthology.orchestra.run.smithi114.stdout: "tags": [], 2024-04-23T10:41:08.593 INFO:teuthology.orchestra.run.smithi114.stdout: "see_also": [] 2024-04-23T10:41:08.594 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-23T10:41:08.594 INFO:teuthology.orchestra.run.smithi114.stdout: "log_level": { 2024-04-23T10:41:08.594 INFO:teuthology.orchestra.run.smithi114.stdout: "name": "log_level", 2024-04-23T10:41:08.594 INFO:teuthology.orchestra.run.smithi114.stdout: "type": "str", 2024-04-23T10:41:08.594 INFO:teuthology.orchestra.run.smithi114.stdout: "level": "advanced", 2024-04-23T10:41:08.594 INFO:teuthology.orchestra.run.smithi114.stdout: "flags": 1, 2024-04-23T10:41:08.594 INFO:teuthology.orchestra.run.smithi114.stdout: "default_value": "", 2024-04-23T10:41:08.594 INFO:teuthology.orchestra.run.smithi114.stdout: "min": "", 2024-04-23T10:41:08.594 INFO:teuthology.orchestra.run.smithi114.stdout: "max": "", 2024-04-23T10:41:08.594 INFO:teuthology.orchestra.run.smithi114.stdout: "enum_allowed": [ 2024-04-23T10:41:08.595 INFO:teuthology.orchestra.run.smithi114.stdout: "", 2024-04-23T10:41:08.595 INFO:teuthology.orchestra.run.smithi114.stdout: "critical", 2024-04-23T10:41:08.595 INFO:teuthology.orchestra.run.smithi114.stdout: "debug", 2024-04-23T10:41:08.595 INFO:teuthology.orchestra.run.smithi114.stdout: "error", 2024-04-23T10:41:08.595 INFO:teuthology.orchestra.run.smithi114.stdout: "info", 2024-04-23T10:41:08.595 INFO:teuthology.orchestra.run.smithi114.stdout: "warning" 2024-04-23T10:41:08.595 INFO:teuthology.orchestra.run.smithi114.stdout: ], 2024-04-23T10:41:08.595 INFO:teuthology.orchestra.run.smithi114.stdout: "desc": "", 2024-04-23T10:41:08.595 INFO:teuthology.orchestra.run.smithi114.stdout: "long_desc": "", 2024-04-23T10:41:08.596 INFO:teuthology.orchestra.run.smithi114.stdout: "tags": [], 2024-04-23T10:41:08.596 INFO:teuthology.orchestra.run.smithi114.stdout: "see_also": [] 2024-04-23T10:41:08.596 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-23T10:41:08.596 INFO:teuthology.orchestra.run.smithi114.stdout: "log_to_cluster": { 2024-04-23T10:41:08.596 INFO:teuthology.orchestra.run.smithi114.stdout: "name": "log_to_cluster", 2024-04-23T10:41:08.596 INFO:teuthology.orchestra.run.smithi114.stdout: "type": "bool", 2024-04-23T10:41:08.596 INFO:teuthology.orchestra.run.smithi114.stdout: "level": "advanced", 2024-04-23T10:41:08.596 INFO:teuthology.orchestra.run.smithi114.stdout: "flags": 1, 2024-04-23T10:41:08.596 INFO:teuthology.orchestra.run.smithi114.stdout: "default_value": "False", 2024-04-23T10:41:08.596 INFO:teuthology.orchestra.run.smithi114.stdout: "min": "", 2024-04-23T10:41:08.597 INFO:teuthology.orchestra.run.smithi114.stdout: "max": "", 2024-04-23T10:41:08.597 INFO:teuthology.orchestra.run.smithi114.stdout: "enum_allowed": [], 2024-04-23T10:41:08.597 INFO:teuthology.orchestra.run.smithi114.stdout: "desc": "", 2024-04-23T10:41:08.597 INFO:teuthology.orchestra.run.smithi114.stdout: "long_desc": "", 2024-04-23T10:41:08.597 INFO:teuthology.orchestra.run.smithi114.stdout: "tags": [], 2024-04-23T10:41:08.597 INFO:teuthology.orchestra.run.smithi114.stdout: "see_also": [] 2024-04-23T10:41:08.597 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-23T10:41:08.597 INFO:teuthology.orchestra.run.smithi114.stdout: "log_to_cluster_level": { 2024-04-23T10:41:08.597 INFO:teuthology.orchestra.run.smithi114.stdout: "name": "log_to_cluster_level", 2024-04-23T10:41:08.597 INFO:teuthology.orchestra.run.smithi114.stdout: "type": "str", 2024-04-23T10:41:08.598 INFO:teuthology.orchestra.run.smithi114.stdout: "level": "advanced", 2024-04-23T10:41:08.598 INFO:teuthology.orchestra.run.smithi114.stdout: "flags": 1, 2024-04-23T10:41:08.598 INFO:teuthology.orchestra.run.smithi114.stdout: "default_value": "info", 2024-04-23T10:41:08.598 INFO:teuthology.orchestra.run.smithi114.stdout: "min": "", 2024-04-23T10:41:08.598 INFO:teuthology.orchestra.run.smithi114.stdout: "max": "", 2024-04-23T10:41:08.598 INFO:teuthology.orchestra.run.smithi114.stdout: "enum_allowed": [ 2024-04-23T10:41:08.598 INFO:teuthology.orchestra.run.smithi114.stdout: "", 2024-04-23T10:41:08.598 INFO:teuthology.orchestra.run.smithi114.stdout: "critical", 2024-04-23T10:41:08.598 INFO:teuthology.orchestra.run.smithi114.stdout: "debug", 2024-04-23T10:41:08.598 INFO:teuthology.orchestra.run.smithi114.stdout: "error", 2024-04-23T10:41:08.599 INFO:teuthology.orchestra.run.smithi114.stdout: "info", 2024-04-23T10:41:08.599 INFO:teuthology.orchestra.run.smithi114.stdout: "warning" 2024-04-23T10:41:08.599 INFO:teuthology.orchestra.run.smithi114.stdout: ], 2024-04-23T10:41:08.599 INFO:teuthology.orchestra.run.smithi114.stdout: "desc": "", 2024-04-23T10:41:08.599 INFO:teuthology.orchestra.run.smithi114.stdout: "long_desc": "", 2024-04-23T10:41:08.599 INFO:teuthology.orchestra.run.smithi114.stdout: "tags": [], 2024-04-23T10:41:08.599 INFO:teuthology.orchestra.run.smithi114.stdout: "see_also": [] 2024-04-23T10:41:08.599 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-23T10:41:08.599 INFO:teuthology.orchestra.run.smithi114.stdout: "log_to_file": { 2024-04-23T10:41:08.600 INFO:teuthology.orchestra.run.smithi114.stdout: "name": "log_to_file", 2024-04-23T10:41:08.600 INFO:teuthology.orchestra.run.smithi114.stdout: "type": "bool", 2024-04-23T10:41:08.600 INFO:teuthology.orchestra.run.smithi114.stdout: "level": "advanced", 2024-04-23T10:41:08.600 INFO:teuthology.orchestra.run.smithi114.stdout: "flags": 1, 2024-04-23T10:41:08.600 INFO:teuthology.orchestra.run.smithi114.stdout: "default_value": "False", 2024-04-23T10:41:08.600 INFO:teuthology.orchestra.run.smithi114.stdout: "min": "", 2024-04-23T10:41:08.600 INFO:teuthology.orchestra.run.smithi114.stdout: "max": "", 2024-04-23T10:41:08.600 INFO:teuthology.orchestra.run.smithi114.stdout: "enum_allowed": [], 2024-04-23T10:41:08.600 INFO:teuthology.orchestra.run.smithi114.stdout: "desc": "", 2024-04-23T10:41:08.600 INFO:teuthology.orchestra.run.smithi114.stdout: "long_desc": "", 2024-04-23T10:41:08.601 INFO:teuthology.orchestra.run.smithi114.stdout: "tags": [], 2024-04-23T10:41:08.601 INFO:teuthology.orchestra.run.smithi114.stdout: "see_also": [] 2024-04-23T10:41:08.601 INFO:teuthology.orchestra.run.smithi114.stdout: } 2024-04-23T10:41:08.601 INFO:teuthology.orchestra.run.smithi114.stdout: } 2024-04-23T10:41:08.601 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-23T10:41:08.601 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-04-23T10:41:08.601 INFO:teuthology.orchestra.run.smithi114.stdout: "name": "telemetry", 2024-04-23T10:41:08.601 INFO:teuthology.orchestra.run.smithi114.stdout: "can_run": true, 2024-04-23T10:41:08.601 INFO:teuthology.orchestra.run.smithi114.stdout: "error_string": "", 2024-04-23T10:41:08.601 INFO:teuthology.orchestra.run.smithi114.stdout: "module_options": { 2024-04-23T10:41:08.602 INFO:teuthology.orchestra.run.smithi114.stdout: "channel_basic": { 2024-04-23T10:41:08.602 INFO:teuthology.orchestra.run.smithi114.stdout: "name": "channel_basic", 2024-04-23T10:41:08.602 INFO:teuthology.orchestra.run.smithi114.stdout: "type": "bool", 2024-04-23T10:41:08.602 INFO:teuthology.orchestra.run.smithi114.stdout: "level": "advanced", 2024-04-23T10:41:08.602 INFO:teuthology.orchestra.run.smithi114.stdout: "flags": 0, 2024-04-23T10:41:08.602 INFO:teuthology.orchestra.run.smithi114.stdout: "default_value": "True", 2024-04-23T10:41:08.602 INFO:teuthology.orchestra.run.smithi114.stdout: "min": "", 2024-04-23T10:41:08.602 INFO:teuthology.orchestra.run.smithi114.stdout: "max": "", 2024-04-23T10:41:08.602 INFO:teuthology.orchestra.run.smithi114.stdout: "enum_allowed": [], 2024-04-23T10:41:08.602 INFO:teuthology.orchestra.run.smithi114.stdout: "desc": "Share basic cluster information (size, version)", 2024-04-23T10:41:08.603 INFO:teuthology.orchestra.run.smithi114.stdout: "long_desc": "", 2024-04-23T10:41:08.603 INFO:teuthology.orchestra.run.smithi114.stdout: "tags": [], 2024-04-23T10:41:08.603 INFO:teuthology.orchestra.run.smithi114.stdout: "see_also": [] 2024-04-23T10:41:08.603 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-23T10:41:08.603 INFO:teuthology.orchestra.run.smithi114.stdout: "channel_crash": { 2024-04-23T10:41:08.603 INFO:teuthology.orchestra.run.smithi114.stdout: "name": "channel_crash", 2024-04-23T10:41:08.603 INFO:teuthology.orchestra.run.smithi114.stdout: "type": "bool", 2024-04-23T10:41:08.603 INFO:teuthology.orchestra.run.smithi114.stdout: "level": "advanced", 2024-04-23T10:41:08.603 INFO:teuthology.orchestra.run.smithi114.stdout: "flags": 0, 2024-04-23T10:41:08.604 INFO:teuthology.orchestra.run.smithi114.stdout: "default_value": "True", 2024-04-23T10:41:08.604 INFO:teuthology.orchestra.run.smithi114.stdout: "min": "", 2024-04-23T10:41:08.604 INFO:teuthology.orchestra.run.smithi114.stdout: "max": "", 2024-04-23T10:41:08.604 INFO:teuthology.orchestra.run.smithi114.stdout: "enum_allowed": [], 2024-04-23T10:41:08.604 INFO:teuthology.orchestra.run.smithi114.stdout: "desc": "Share metadata about Ceph daemon crashes (version, stack straces, etc)", 2024-04-23T10:41:08.604 INFO:teuthology.orchestra.run.smithi114.stdout: "long_desc": "", 2024-04-23T10:41:08.604 INFO:teuthology.orchestra.run.smithi114.stdout: "tags": [], 2024-04-23T10:41:08.604 INFO:teuthology.orchestra.run.smithi114.stdout: "see_also": [] 2024-04-23T10:41:08.604 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-23T10:41:08.604 INFO:teuthology.orchestra.run.smithi114.stdout: "channel_device": { 2024-04-23T10:41:08.605 INFO:teuthology.orchestra.run.smithi114.stdout: "name": "channel_device", 2024-04-23T10:41:08.605 INFO:teuthology.orchestra.run.smithi114.stdout: "type": "bool", 2024-04-23T10:41:08.605 INFO:teuthology.orchestra.run.smithi114.stdout: "level": "advanced", 2024-04-23T10:41:08.605 INFO:teuthology.orchestra.run.smithi114.stdout: "flags": 0, 2024-04-23T10:41:08.605 INFO:teuthology.orchestra.run.smithi114.stdout: "default_value": "True", 2024-04-23T10:41:08.605 INFO:teuthology.orchestra.run.smithi114.stdout: "min": "", 2024-04-23T10:41:08.605 INFO:teuthology.orchestra.run.smithi114.stdout: "max": "", 2024-04-23T10:41:08.605 INFO:teuthology.orchestra.run.smithi114.stdout: "enum_allowed": [], 2024-04-23T10:41:08.605 INFO:teuthology.orchestra.run.smithi114.stdout: "desc": "Share device health metrics (e.g., SMART data, minus potentially identifying info like serial numbers)", 2024-04-23T10:41:08.605 INFO:teuthology.orchestra.run.smithi114.stdout: "long_desc": "", 2024-04-23T10:41:08.606 INFO:teuthology.orchestra.run.smithi114.stdout: "tags": [], 2024-04-23T10:41:08.606 INFO:teuthology.orchestra.run.smithi114.stdout: "see_also": [] 2024-04-23T10:41:08.606 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-23T10:41:08.606 INFO:teuthology.orchestra.run.smithi114.stdout: "channel_ident": { 2024-04-23T10:41:08.606 INFO:teuthology.orchestra.run.smithi114.stdout: "name": "channel_ident", 2024-04-23T10:41:08.606 INFO:teuthology.orchestra.run.smithi114.stdout: "type": "bool", 2024-04-23T10:41:08.606 INFO:teuthology.orchestra.run.smithi114.stdout: "level": "advanced", 2024-04-23T10:41:08.606 INFO:teuthology.orchestra.run.smithi114.stdout: "flags": 0, 2024-04-23T10:41:08.606 INFO:teuthology.orchestra.run.smithi114.stdout: "default_value": "False", 2024-04-23T10:41:08.607 INFO:teuthology.orchestra.run.smithi114.stdout: "min": "", 2024-04-23T10:41:08.607 INFO:teuthology.orchestra.run.smithi114.stdout: "max": "", 2024-04-23T10:41:08.607 INFO:teuthology.orchestra.run.smithi114.stdout: "enum_allowed": [], 2024-04-23T10:41:08.607 INFO:teuthology.orchestra.run.smithi114.stdout: "desc": "Share a user-provided description and/or contact email for the cluster", 2024-04-23T10:41:08.607 INFO:teuthology.orchestra.run.smithi114.stdout: "long_desc": "", 2024-04-23T10:41:08.607 INFO:teuthology.orchestra.run.smithi114.stdout: "tags": [], 2024-04-23T10:41:08.607 INFO:teuthology.orchestra.run.smithi114.stdout: "see_also": [] 2024-04-23T10:41:08.607 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-23T10:41:08.607 INFO:teuthology.orchestra.run.smithi114.stdout: "channel_perf": { 2024-04-23T10:41:08.607 INFO:teuthology.orchestra.run.smithi114.stdout: "name": "channel_perf", 2024-04-23T10:41:08.608 INFO:teuthology.orchestra.run.smithi114.stdout: "type": "bool", 2024-04-23T10:41:08.608 INFO:teuthology.orchestra.run.smithi114.stdout: "level": "advanced", 2024-04-23T10:41:08.608 INFO:teuthology.orchestra.run.smithi114.stdout: "flags": 0, 2024-04-23T10:41:08.608 INFO:teuthology.orchestra.run.smithi114.stdout: "default_value": "False", 2024-04-23T10:41:08.608 INFO:teuthology.orchestra.run.smithi114.stdout: "min": "", 2024-04-23T10:41:08.608 INFO:teuthology.orchestra.run.smithi114.stdout: "max": "", 2024-04-23T10:41:08.608 INFO:teuthology.orchestra.run.smithi114.stdout: "enum_allowed": [], 2024-04-23T10:41:08.608 INFO:teuthology.orchestra.run.smithi114.stdout: "desc": "Share various performance metrics of a cluster", 2024-04-23T10:41:08.608 INFO:teuthology.orchestra.run.smithi114.stdout: "long_desc": "", 2024-04-23T10:41:08.608 INFO:teuthology.orchestra.run.smithi114.stdout: "tags": [], 2024-04-23T10:41:08.609 INFO:teuthology.orchestra.run.smithi114.stdout: "see_also": [] 2024-04-23T10:41:08.609 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-23T10:41:08.609 INFO:teuthology.orchestra.run.smithi114.stdout: "contact": { 2024-04-23T10:41:08.609 INFO:teuthology.orchestra.run.smithi114.stdout: "name": "contact", 2024-04-23T10:41:08.609 INFO:teuthology.orchestra.run.smithi114.stdout: "type": "str", 2024-04-23T10:41:08.609 INFO:teuthology.orchestra.run.smithi114.stdout: "level": "advanced", 2024-04-23T10:41:08.609 INFO:teuthology.orchestra.run.smithi114.stdout: "flags": 0, 2024-04-23T10:41:08.609 INFO:teuthology.orchestra.run.smithi114.stdout: "default_value": "", 2024-04-23T10:41:08.609 INFO:teuthology.orchestra.run.smithi114.stdout: "min": "", 2024-04-23T10:41:08.609 INFO:teuthology.orchestra.run.smithi114.stdout: "max": "", 2024-04-23T10:41:08.610 INFO:teuthology.orchestra.run.smithi114.stdout: "enum_allowed": [], 2024-04-23T10:41:08.610 INFO:teuthology.orchestra.run.smithi114.stdout: "desc": "", 2024-04-23T10:41:08.610 INFO:teuthology.orchestra.run.smithi114.stdout: "long_desc": "", 2024-04-23T10:41:08.610 INFO:teuthology.orchestra.run.smithi114.stdout: "tags": [], 2024-04-23T10:41:08.610 INFO:teuthology.orchestra.run.smithi114.stdout: "see_also": [] 2024-04-23T10:41:08.610 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-23T10:41:08.610 INFO:teuthology.orchestra.run.smithi114.stdout: "description": { 2024-04-23T10:41:08.610 INFO:teuthology.orchestra.run.smithi114.stdout: "name": "description", 2024-04-23T10:41:08.610 INFO:teuthology.orchestra.run.smithi114.stdout: "type": "str", 2024-04-23T10:41:08.610 INFO:teuthology.orchestra.run.smithi114.stdout: "level": "advanced", 2024-04-23T10:41:08.611 INFO:teuthology.orchestra.run.smithi114.stdout: "flags": 0, 2024-04-23T10:41:08.611 INFO:teuthology.orchestra.run.smithi114.stdout: "default_value": "", 2024-04-23T10:41:08.611 INFO:teuthology.orchestra.run.smithi114.stdout: "min": "", 2024-04-23T10:41:08.611 INFO:teuthology.orchestra.run.smithi114.stdout: "max": "", 2024-04-23T10:41:08.611 INFO:teuthology.orchestra.run.smithi114.stdout: "enum_allowed": [], 2024-04-23T10:41:08.611 INFO:teuthology.orchestra.run.smithi114.stdout: "desc": "", 2024-04-23T10:41:08.611 INFO:teuthology.orchestra.run.smithi114.stdout: "long_desc": "", 2024-04-23T10:41:08.611 INFO:teuthology.orchestra.run.smithi114.stdout: "tags": [], 2024-04-23T10:41:08.611 INFO:teuthology.orchestra.run.smithi114.stdout: "see_also": [] 2024-04-23T10:41:08.611 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-23T10:41:08.612 INFO:teuthology.orchestra.run.smithi114.stdout: "device_url": { 2024-04-23T10:41:08.612 INFO:teuthology.orchestra.run.smithi114.stdout: "name": "device_url", 2024-04-23T10:41:08.612 INFO:teuthology.orchestra.run.smithi114.stdout: "type": "str", 2024-04-23T10:41:08.612 INFO:teuthology.orchestra.run.smithi114.stdout: "level": "advanced", 2024-04-23T10:41:08.612 INFO:teuthology.orchestra.run.smithi114.stdout: "flags": 0, 2024-04-23T10:41:08.612 INFO:teuthology.orchestra.run.smithi114.stdout: "default_value": "https://telemetry.ceph.com/device", 2024-04-23T10:41:08.612 INFO:teuthology.orchestra.run.smithi114.stdout: "min": "", 2024-04-23T10:41:08.612 INFO:teuthology.orchestra.run.smithi114.stdout: "max": "", 2024-04-23T10:41:08.612 INFO:teuthology.orchestra.run.smithi114.stdout: "enum_allowed": [], 2024-04-23T10:41:08.612 INFO:teuthology.orchestra.run.smithi114.stdout: "desc": "", 2024-04-23T10:41:08.613 INFO:teuthology.orchestra.run.smithi114.stdout: "long_desc": "", 2024-04-23T10:41:08.613 INFO:teuthology.orchestra.run.smithi114.stdout: "tags": [], 2024-04-23T10:41:08.613 INFO:teuthology.orchestra.run.smithi114.stdout: "see_also": [] 2024-04-23T10:41:08.613 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-23T10:41:08.613 INFO:teuthology.orchestra.run.smithi114.stdout: "enabled": { 2024-04-23T10:41:08.613 INFO:teuthology.orchestra.run.smithi114.stdout: "name": "enabled", 2024-04-23T10:41:08.613 INFO:teuthology.orchestra.run.smithi114.stdout: "type": "bool", 2024-04-23T10:41:08.613 INFO:teuthology.orchestra.run.smithi114.stdout: "level": "advanced", 2024-04-23T10:41:08.613 INFO:teuthology.orchestra.run.smithi114.stdout: "flags": 0, 2024-04-23T10:41:08.614 INFO:teuthology.orchestra.run.smithi114.stdout: "default_value": "False", 2024-04-23T10:41:08.614 INFO:teuthology.orchestra.run.smithi114.stdout: "min": "", 2024-04-23T10:41:08.614 INFO:teuthology.orchestra.run.smithi114.stdout: "max": "", 2024-04-23T10:41:08.614 INFO:teuthology.orchestra.run.smithi114.stdout: "enum_allowed": [], 2024-04-23T10:41:08.614 INFO:teuthology.orchestra.run.smithi114.stdout: "desc": "", 2024-04-23T10:41:08.614 INFO:teuthology.orchestra.run.smithi114.stdout: "long_desc": "", 2024-04-23T10:41:08.614 INFO:teuthology.orchestra.run.smithi114.stdout: "tags": [], 2024-04-23T10:41:08.614 INFO:teuthology.orchestra.run.smithi114.stdout: "see_also": [] 2024-04-23T10:41:08.614 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-23T10:41:08.614 INFO:teuthology.orchestra.run.smithi114.stdout: "interval": { 2024-04-23T10:41:08.615 INFO:teuthology.orchestra.run.smithi114.stdout: "name": "interval", 2024-04-23T10:41:08.615 INFO:teuthology.orchestra.run.smithi114.stdout: "type": "int", 2024-04-23T10:41:08.615 INFO:teuthology.orchestra.run.smithi114.stdout: "level": "advanced", 2024-04-23T10:41:08.615 INFO:teuthology.orchestra.run.smithi114.stdout: "flags": 0, 2024-04-23T10:41:08.615 INFO:teuthology.orchestra.run.smithi114.stdout: "default_value": "24", 2024-04-23T10:41:08.615 INFO:teuthology.orchestra.run.smithi114.stdout: "min": "8", 2024-04-23T10:41:08.615 INFO:teuthology.orchestra.run.smithi114.stdout: "max": "", 2024-04-23T10:41:08.615 INFO:teuthology.orchestra.run.smithi114.stdout: "enum_allowed": [], 2024-04-23T10:41:08.615 INFO:teuthology.orchestra.run.smithi114.stdout: "desc": "", 2024-04-23T10:41:08.615 INFO:teuthology.orchestra.run.smithi114.stdout: "long_desc": "", 2024-04-23T10:41:08.616 INFO:teuthology.orchestra.run.smithi114.stdout: "tags": [], 2024-04-23T10:41:08.616 INFO:teuthology.orchestra.run.smithi114.stdout: "see_also": [] 2024-04-23T10:41:08.616 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-23T10:41:08.616 INFO:teuthology.orchestra.run.smithi114.stdout: "last_opt_revision": { 2024-04-23T10:41:08.616 INFO:teuthology.orchestra.run.smithi114.stdout: "name": "last_opt_revision", 2024-04-23T10:41:08.616 INFO:teuthology.orchestra.run.smithi114.stdout: "type": "int", 2024-04-23T10:41:08.616 INFO:teuthology.orchestra.run.smithi114.stdout: "level": "advanced", 2024-04-23T10:41:08.616 INFO:teuthology.orchestra.run.smithi114.stdout: "flags": 0, 2024-04-23T10:41:08.616 INFO:teuthology.orchestra.run.smithi114.stdout: "default_value": "1", 2024-04-23T10:41:08.616 INFO:teuthology.orchestra.run.smithi114.stdout: "min": "", 2024-04-23T10:41:08.617 INFO:teuthology.orchestra.run.smithi114.stdout: "max": "", 2024-04-23T10:41:08.617 INFO:teuthology.orchestra.run.smithi114.stdout: "enum_allowed": [], 2024-04-23T10:41:08.617 INFO:teuthology.orchestra.run.smithi114.stdout: "desc": "", 2024-04-23T10:41:08.617 INFO:teuthology.orchestra.run.smithi114.stdout: "long_desc": "", 2024-04-23T10:41:08.617 INFO:teuthology.orchestra.run.smithi114.stdout: "tags": [], 2024-04-23T10:41:08.617 INFO:teuthology.orchestra.run.smithi114.stdout: "see_also": [] 2024-04-23T10:41:08.617 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-23T10:41:08.617 INFO:teuthology.orchestra.run.smithi114.stdout: "leaderboard": { 2024-04-23T10:41:08.617 INFO:teuthology.orchestra.run.smithi114.stdout: "name": "leaderboard", 2024-04-23T10:41:08.617 INFO:teuthology.orchestra.run.smithi114.stdout: "type": "bool", 2024-04-23T10:41:08.618 INFO:teuthology.orchestra.run.smithi114.stdout: "level": "advanced", 2024-04-23T10:41:08.618 INFO:teuthology.orchestra.run.smithi114.stdout: "flags": 0, 2024-04-23T10:41:08.618 INFO:teuthology.orchestra.run.smithi114.stdout: "default_value": "False", 2024-04-23T10:41:08.618 INFO:teuthology.orchestra.run.smithi114.stdout: "min": "", 2024-04-23T10:41:08.618 INFO:teuthology.orchestra.run.smithi114.stdout: "max": "", 2024-04-23T10:41:08.618 INFO:teuthology.orchestra.run.smithi114.stdout: "enum_allowed": [], 2024-04-23T10:41:08.618 INFO:teuthology.orchestra.run.smithi114.stdout: "desc": "", 2024-04-23T10:41:08.618 INFO:teuthology.orchestra.run.smithi114.stdout: "long_desc": "", 2024-04-23T10:41:08.618 INFO:teuthology.orchestra.run.smithi114.stdout: "tags": [], 2024-04-23T10:41:08.618 INFO:teuthology.orchestra.run.smithi114.stdout: "see_also": [] 2024-04-23T10:41:08.619 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-23T10:41:08.619 INFO:teuthology.orchestra.run.smithi114.stdout: "leaderboard_description": { 2024-04-23T10:41:08.619 INFO:teuthology.orchestra.run.smithi114.stdout: "name": "leaderboard_description", 2024-04-23T10:41:08.619 INFO:teuthology.orchestra.run.smithi114.stdout: "type": "str", 2024-04-23T10:41:08.619 INFO:teuthology.orchestra.run.smithi114.stdout: "level": "advanced", 2024-04-23T10:41:08.619 INFO:teuthology.orchestra.run.smithi114.stdout: "flags": 0, 2024-04-23T10:41:08.619 INFO:teuthology.orchestra.run.smithi114.stdout: "default_value": "", 2024-04-23T10:41:08.619 INFO:teuthology.orchestra.run.smithi114.stdout: "min": "", 2024-04-23T10:41:08.619 INFO:teuthology.orchestra.run.smithi114.stdout: "max": "", 2024-04-23T10:41:08.619 INFO:teuthology.orchestra.run.smithi114.stdout: "enum_allowed": [], 2024-04-23T10:41:08.620 INFO:teuthology.orchestra.run.smithi114.stdout: "desc": "", 2024-04-23T10:41:08.620 INFO:teuthology.orchestra.run.smithi114.stdout: "long_desc": "", 2024-04-23T10:41:08.620 INFO:teuthology.orchestra.run.smithi114.stdout: "tags": [], 2024-04-23T10:41:08.620 INFO:teuthology.orchestra.run.smithi114.stdout: "see_also": [] 2024-04-23T10:41:08.620 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-23T10:41:08.620 INFO:teuthology.orchestra.run.smithi114.stdout: "log_level": { 2024-04-23T10:41:08.620 INFO:teuthology.orchestra.run.smithi114.stdout: "name": "log_level", 2024-04-23T10:41:08.620 INFO:teuthology.orchestra.run.smithi114.stdout: "type": "str", 2024-04-23T10:41:08.620 INFO:teuthology.orchestra.run.smithi114.stdout: "level": "advanced", 2024-04-23T10:41:08.621 INFO:teuthology.orchestra.run.smithi114.stdout: "flags": 1, 2024-04-23T10:41:08.621 INFO:teuthology.orchestra.run.smithi114.stdout: "default_value": "", 2024-04-23T10:41:08.621 INFO:teuthology.orchestra.run.smithi114.stdout: "min": "", 2024-04-23T10:41:08.621 INFO:teuthology.orchestra.run.smithi114.stdout: "max": "", 2024-04-23T10:41:08.621 INFO:teuthology.orchestra.run.smithi114.stdout: "enum_allowed": [ 2024-04-23T10:41:08.621 INFO:teuthology.orchestra.run.smithi114.stdout: "", 2024-04-23T10:41:08.621 INFO:teuthology.orchestra.run.smithi114.stdout: "critical", 2024-04-23T10:41:08.621 INFO:teuthology.orchestra.run.smithi114.stdout: "debug", 2024-04-23T10:41:08.621 INFO:teuthology.orchestra.run.smithi114.stdout: "error", 2024-04-23T10:41:08.621 INFO:teuthology.orchestra.run.smithi114.stdout: "info", 2024-04-23T10:41:08.622 INFO:teuthology.orchestra.run.smithi114.stdout: "warning" 2024-04-23T10:41:08.622 INFO:teuthology.orchestra.run.smithi114.stdout: ], 2024-04-23T10:41:08.622 INFO:teuthology.orchestra.run.smithi114.stdout: "desc": "", 2024-04-23T10:41:08.622 INFO:teuthology.orchestra.run.smithi114.stdout: "long_desc": "", 2024-04-23T10:41:08.622 INFO:teuthology.orchestra.run.smithi114.stdout: "tags": [], 2024-04-23T10:41:08.622 INFO:teuthology.orchestra.run.smithi114.stdout: "see_also": [] 2024-04-23T10:41:08.622 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-23T10:41:08.622 INFO:teuthology.orchestra.run.smithi114.stdout: "log_to_cluster": { 2024-04-23T10:41:08.622 INFO:teuthology.orchestra.run.smithi114.stdout: "name": "log_to_cluster", 2024-04-23T10:41:08.622 INFO:teuthology.orchestra.run.smithi114.stdout: "type": "bool", 2024-04-23T10:41:08.623 INFO:teuthology.orchestra.run.smithi114.stdout: "level": "advanced", 2024-04-23T10:41:08.623 INFO:teuthology.orchestra.run.smithi114.stdout: "flags": 1, 2024-04-23T10:41:08.623 INFO:teuthology.orchestra.run.smithi114.stdout: "default_value": "False", 2024-04-23T10:41:08.623 INFO:teuthology.orchestra.run.smithi114.stdout: "min": "", 2024-04-23T10:41:08.623 INFO:teuthology.orchestra.run.smithi114.stdout: "max": "", 2024-04-23T10:41:08.623 INFO:teuthology.orchestra.run.smithi114.stdout: "enum_allowed": [], 2024-04-23T10:41:08.623 INFO:teuthology.orchestra.run.smithi114.stdout: "desc": "", 2024-04-23T10:41:08.623 INFO:teuthology.orchestra.run.smithi114.stdout: "long_desc": "", 2024-04-23T10:41:08.623 INFO:teuthology.orchestra.run.smithi114.stdout: "tags": [], 2024-04-23T10:41:08.623 INFO:teuthology.orchestra.run.smithi114.stdout: "see_also": [] 2024-04-23T10:41:08.624 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-23T10:41:08.624 INFO:teuthology.orchestra.run.smithi114.stdout: "log_to_cluster_level": { 2024-04-23T10:41:08.624 INFO:teuthology.orchestra.run.smithi114.stdout: "name": "log_to_cluster_level", 2024-04-23T10:41:08.624 INFO:teuthology.orchestra.run.smithi114.stdout: "type": "str", 2024-04-23T10:41:08.624 INFO:teuthology.orchestra.run.smithi114.stdout: "level": "advanced", 2024-04-23T10:41:08.624 INFO:teuthology.orchestra.run.smithi114.stdout: "flags": 1, 2024-04-23T10:41:08.624 INFO:teuthology.orchestra.run.smithi114.stdout: "default_value": "info", 2024-04-23T10:41:08.624 INFO:teuthology.orchestra.run.smithi114.stdout: "min": "", 2024-04-23T10:41:08.624 INFO:teuthology.orchestra.run.smithi114.stdout: "max": "", 2024-04-23T10:41:08.624 INFO:teuthology.orchestra.run.smithi114.stdout: "enum_allowed": [ 2024-04-23T10:41:08.625 INFO:teuthology.orchestra.run.smithi114.stdout: "", 2024-04-23T10:41:08.625 INFO:teuthology.orchestra.run.smithi114.stdout: "critical", 2024-04-23T10:41:08.625 INFO:teuthology.orchestra.run.smithi114.stdout: "debug", 2024-04-23T10:41:08.625 INFO:teuthology.orchestra.run.smithi114.stdout: "error", 2024-04-23T10:41:08.625 INFO:teuthology.orchestra.run.smithi114.stdout: "info", 2024-04-23T10:41:08.625 INFO:teuthology.orchestra.run.smithi114.stdout: "warning" 2024-04-23T10:41:08.625 INFO:teuthology.orchestra.run.smithi114.stdout: ], 2024-04-23T10:41:08.625 INFO:teuthology.orchestra.run.smithi114.stdout: "desc": "", 2024-04-23T10:41:08.625 INFO:teuthology.orchestra.run.smithi114.stdout: "long_desc": "", 2024-04-23T10:41:08.625 INFO:teuthology.orchestra.run.smithi114.stdout: "tags": [], 2024-04-23T10:41:08.626 INFO:teuthology.orchestra.run.smithi114.stdout: "see_also": [] 2024-04-23T10:41:08.626 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-23T10:41:08.626 INFO:teuthology.orchestra.run.smithi114.stdout: "log_to_file": { 2024-04-23T10:41:08.626 INFO:teuthology.orchestra.run.smithi114.stdout: "name": "log_to_file", 2024-04-23T10:41:08.626 INFO:teuthology.orchestra.run.smithi114.stdout: "type": "bool", 2024-04-23T10:41:08.626 INFO:teuthology.orchestra.run.smithi114.stdout: "level": "advanced", 2024-04-23T10:41:08.626 INFO:teuthology.orchestra.run.smithi114.stdout: "flags": 1, 2024-04-23T10:41:08.626 INFO:teuthology.orchestra.run.smithi114.stdout: "default_value": "False", 2024-04-23T10:41:08.626 INFO:teuthology.orchestra.run.smithi114.stdout: "min": "", 2024-04-23T10:41:08.627 INFO:teuthology.orchestra.run.smithi114.stdout: "max": "", 2024-04-23T10:41:08.627 INFO:teuthology.orchestra.run.smithi114.stdout: "enum_allowed": [], 2024-04-23T10:41:08.627 INFO:teuthology.orchestra.run.smithi114.stdout: "desc": "", 2024-04-23T10:41:08.627 INFO:teuthology.orchestra.run.smithi114.stdout: "long_desc": "", 2024-04-23T10:41:08.627 INFO:teuthology.orchestra.run.smithi114.stdout: "tags": [], 2024-04-23T10:41:08.627 INFO:teuthology.orchestra.run.smithi114.stdout: "see_also": [] 2024-04-23T10:41:08.627 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-23T10:41:08.627 INFO:teuthology.orchestra.run.smithi114.stdout: "organization": { 2024-04-23T10:41:08.627 INFO:teuthology.orchestra.run.smithi114.stdout: "name": "organization", 2024-04-23T10:41:08.627 INFO:teuthology.orchestra.run.smithi114.stdout: "type": "str", 2024-04-23T10:41:08.628 INFO:teuthology.orchestra.run.smithi114.stdout: "level": "advanced", 2024-04-23T10:41:08.628 INFO:teuthology.orchestra.run.smithi114.stdout: "flags": 0, 2024-04-23T10:41:08.628 INFO:teuthology.orchestra.run.smithi114.stdout: "default_value": "", 2024-04-23T10:41:08.628 INFO:teuthology.orchestra.run.smithi114.stdout: "min": "", 2024-04-23T10:41:08.628 INFO:teuthology.orchestra.run.smithi114.stdout: "max": "", 2024-04-23T10:41:08.628 INFO:teuthology.orchestra.run.smithi114.stdout: "enum_allowed": [], 2024-04-23T10:41:08.628 INFO:teuthology.orchestra.run.smithi114.stdout: "desc": "", 2024-04-23T10:41:08.628 INFO:teuthology.orchestra.run.smithi114.stdout: "long_desc": "", 2024-04-23T10:41:08.628 INFO:teuthology.orchestra.run.smithi114.stdout: "tags": [], 2024-04-23T10:41:08.628 INFO:teuthology.orchestra.run.smithi114.stdout: "see_also": [] 2024-04-23T10:41:08.629 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-23T10:41:08.629 INFO:teuthology.orchestra.run.smithi114.stdout: "proxy": { 2024-04-23T10:41:08.629 INFO:teuthology.orchestra.run.smithi114.stdout: "name": "proxy", 2024-04-23T10:41:08.629 INFO:teuthology.orchestra.run.smithi114.stdout: "type": "str", 2024-04-23T10:41:08.629 INFO:teuthology.orchestra.run.smithi114.stdout: "level": "advanced", 2024-04-23T10:41:08.629 INFO:teuthology.orchestra.run.smithi114.stdout: "flags": 0, 2024-04-23T10:41:08.629 INFO:teuthology.orchestra.run.smithi114.stdout: "default_value": "", 2024-04-23T10:41:08.629 INFO:teuthology.orchestra.run.smithi114.stdout: "min": "", 2024-04-23T10:41:08.629 INFO:teuthology.orchestra.run.smithi114.stdout: "max": "", 2024-04-23T10:41:08.629 INFO:teuthology.orchestra.run.smithi114.stdout: "enum_allowed": [], 2024-04-23T10:41:08.630 INFO:teuthology.orchestra.run.smithi114.stdout: "desc": "", 2024-04-23T10:41:08.630 INFO:teuthology.orchestra.run.smithi114.stdout: "long_desc": "", 2024-04-23T10:41:08.630 INFO:teuthology.orchestra.run.smithi114.stdout: "tags": [], 2024-04-23T10:41:08.630 INFO:teuthology.orchestra.run.smithi114.stdout: "see_also": [] 2024-04-23T10:41:08.630 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-23T10:41:08.630 INFO:teuthology.orchestra.run.smithi114.stdout: "url": { 2024-04-23T10:41:08.630 INFO:teuthology.orchestra.run.smithi114.stdout: "name": "url", 2024-04-23T10:41:08.630 INFO:teuthology.orchestra.run.smithi114.stdout: "type": "str", 2024-04-23T10:41:08.630 INFO:teuthology.orchestra.run.smithi114.stdout: "level": "advanced", 2024-04-23T10:41:08.630 INFO:teuthology.orchestra.run.smithi114.stdout: "flags": 0, 2024-04-23T10:41:08.631 INFO:teuthology.orchestra.run.smithi114.stdout: "default_value": "https://telemetry.ceph.com/report", 2024-04-23T10:41:08.631 INFO:teuthology.orchestra.run.smithi114.stdout: "min": "", 2024-04-23T10:41:08.631 INFO:teuthology.orchestra.run.smithi114.stdout: "max": "", 2024-04-23T10:41:08.631 INFO:teuthology.orchestra.run.smithi114.stdout: "enum_allowed": [], 2024-04-23T10:41:08.631 INFO:teuthology.orchestra.run.smithi114.stdout: "desc": "", 2024-04-23T10:41:08.631 INFO:teuthology.orchestra.run.smithi114.stdout: "long_desc": "", 2024-04-23T10:41:08.631 INFO:teuthology.orchestra.run.smithi114.stdout: "tags": [], 2024-04-23T10:41:08.631 INFO:teuthology.orchestra.run.smithi114.stdout: "see_also": [] 2024-04-23T10:41:08.631 INFO:teuthology.orchestra.run.smithi114.stdout: } 2024-04-23T10:41:08.632 INFO:teuthology.orchestra.run.smithi114.stdout: } 2024-04-23T10:41:08.632 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-23T10:41:08.632 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-04-23T10:41:08.632 INFO:teuthology.orchestra.run.smithi114.stdout: "name": "test_orchestrator", 2024-04-23T10:41:08.632 INFO:teuthology.orchestra.run.smithi114.stdout: "can_run": true, 2024-04-23T10:41:08.632 INFO:teuthology.orchestra.run.smithi114.stdout: "error_string": "", 2024-04-23T10:41:08.632 INFO:teuthology.orchestra.run.smithi114.stdout: "module_options": { 2024-04-23T10:41:08.632 INFO:teuthology.orchestra.run.smithi114.stdout: "log_level": { 2024-04-23T10:41:08.632 INFO:teuthology.orchestra.run.smithi114.stdout: "name": "log_level", 2024-04-23T10:41:08.632 INFO:teuthology.orchestra.run.smithi114.stdout: "type": "str", 2024-04-23T10:41:08.633 INFO:teuthology.orchestra.run.smithi114.stdout: "level": "advanced", 2024-04-23T10:41:08.633 INFO:teuthology.orchestra.run.smithi114.stdout: "flags": 1, 2024-04-23T10:41:08.633 INFO:teuthology.orchestra.run.smithi114.stdout: "default_value": "", 2024-04-23T10:41:08.633 INFO:teuthology.orchestra.run.smithi114.stdout: "min": "", 2024-04-23T10:41:08.633 INFO:teuthology.orchestra.run.smithi114.stdout: "max": "", 2024-04-23T10:41:08.633 INFO:teuthology.orchestra.run.smithi114.stdout: "enum_allowed": [ 2024-04-23T10:41:08.633 INFO:teuthology.orchestra.run.smithi114.stdout: "", 2024-04-23T10:41:08.633 INFO:teuthology.orchestra.run.smithi114.stdout: "critical", 2024-04-23T10:41:08.633 INFO:teuthology.orchestra.run.smithi114.stdout: "debug", 2024-04-23T10:41:08.633 INFO:teuthology.orchestra.run.smithi114.stdout: "error", 2024-04-23T10:41:08.634 INFO:teuthology.orchestra.run.smithi114.stdout: "info", 2024-04-23T10:41:08.634 INFO:teuthology.orchestra.run.smithi114.stdout: "warning" 2024-04-23T10:41:08.634 INFO:teuthology.orchestra.run.smithi114.stdout: ], 2024-04-23T10:41:08.634 INFO:teuthology.orchestra.run.smithi114.stdout: "desc": "", 2024-04-23T10:41:08.634 INFO:teuthology.orchestra.run.smithi114.stdout: "long_desc": "", 2024-04-23T10:41:08.634 INFO:teuthology.orchestra.run.smithi114.stdout: "tags": [], 2024-04-23T10:41:08.634 INFO:teuthology.orchestra.run.smithi114.stdout: "see_also": [] 2024-04-23T10:41:08.634 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-23T10:41:08.634 INFO:teuthology.orchestra.run.smithi114.stdout: "log_to_cluster": { 2024-04-23T10:41:08.635 INFO:teuthology.orchestra.run.smithi114.stdout: "name": "log_to_cluster", 2024-04-23T10:41:08.635 INFO:teuthology.orchestra.run.smithi114.stdout: "type": "bool", 2024-04-23T10:41:08.635 INFO:teuthology.orchestra.run.smithi114.stdout: "level": "advanced", 2024-04-23T10:41:08.635 INFO:teuthology.orchestra.run.smithi114.stdout: "flags": 1, 2024-04-23T10:41:08.635 INFO:teuthology.orchestra.run.smithi114.stdout: "default_value": "False", 2024-04-23T10:41:08.635 INFO:teuthology.orchestra.run.smithi114.stdout: "min": "", 2024-04-23T10:41:08.635 INFO:teuthology.orchestra.run.smithi114.stdout: "max": "", 2024-04-23T10:41:08.635 INFO:teuthology.orchestra.run.smithi114.stdout: "enum_allowed": [], 2024-04-23T10:41:08.635 INFO:teuthology.orchestra.run.smithi114.stdout: "desc": "", 2024-04-23T10:41:08.636 INFO:teuthology.orchestra.run.smithi114.stdout: "long_desc": "", 2024-04-23T10:41:08.636 INFO:teuthology.orchestra.run.smithi114.stdout: "tags": [], 2024-04-23T10:41:08.636 INFO:teuthology.orchestra.run.smithi114.stdout: "see_also": [] 2024-04-23T10:41:08.636 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-23T10:41:08.636 INFO:teuthology.orchestra.run.smithi114.stdout: "log_to_cluster_level": { 2024-04-23T10:41:08.636 INFO:teuthology.orchestra.run.smithi114.stdout: "name": "log_to_cluster_level", 2024-04-23T10:41:08.636 INFO:teuthology.orchestra.run.smithi114.stdout: "type": "str", 2024-04-23T10:41:08.636 INFO:teuthology.orchestra.run.smithi114.stdout: "level": "advanced", 2024-04-23T10:41:08.636 INFO:teuthology.orchestra.run.smithi114.stdout: "flags": 1, 2024-04-23T10:41:08.636 INFO:teuthology.orchestra.run.smithi114.stdout: "default_value": "info", 2024-04-23T10:41:08.637 INFO:teuthology.orchestra.run.smithi114.stdout: "min": "", 2024-04-23T10:41:08.637 INFO:teuthology.orchestra.run.smithi114.stdout: "max": "", 2024-04-23T10:41:08.637 INFO:teuthology.orchestra.run.smithi114.stdout: "enum_allowed": [ 2024-04-23T10:41:08.637 INFO:teuthology.orchestra.run.smithi114.stdout: "", 2024-04-23T10:41:08.637 INFO:teuthology.orchestra.run.smithi114.stdout: "critical", 2024-04-23T10:41:08.637 INFO:teuthology.orchestra.run.smithi114.stdout: "debug", 2024-04-23T10:41:08.637 INFO:teuthology.orchestra.run.smithi114.stdout: "error", 2024-04-23T10:41:08.637 INFO:teuthology.orchestra.run.smithi114.stdout: "info", 2024-04-23T10:41:08.637 INFO:teuthology.orchestra.run.smithi114.stdout: "warning" 2024-04-23T10:41:08.637 INFO:teuthology.orchestra.run.smithi114.stdout: ], 2024-04-23T10:41:08.638 INFO:teuthology.orchestra.run.smithi114.stdout: "desc": "", 2024-04-23T10:41:08.638 INFO:teuthology.orchestra.run.smithi114.stdout: "long_desc": "", 2024-04-23T10:41:08.638 INFO:teuthology.orchestra.run.smithi114.stdout: "tags": [], 2024-04-23T10:41:08.638 INFO:teuthology.orchestra.run.smithi114.stdout: "see_also": [] 2024-04-23T10:41:08.638 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-23T10:41:08.638 INFO:teuthology.orchestra.run.smithi114.stdout: "log_to_file": { 2024-04-23T10:41:08.638 INFO:teuthology.orchestra.run.smithi114.stdout: "name": "log_to_file", 2024-04-23T10:41:08.638 INFO:teuthology.orchestra.run.smithi114.stdout: "type": "bool", 2024-04-23T10:41:08.638 INFO:teuthology.orchestra.run.smithi114.stdout: "level": "advanced", 2024-04-23T10:41:08.639 INFO:teuthology.orchestra.run.smithi114.stdout: "flags": 1, 2024-04-23T10:41:08.639 INFO:teuthology.orchestra.run.smithi114.stdout: "default_value": "False", 2024-04-23T10:41:08.639 INFO:teuthology.orchestra.run.smithi114.stdout: "min": "", 2024-04-23T10:41:08.639 INFO:teuthology.orchestra.run.smithi114.stdout: "max": "", 2024-04-23T10:41:08.639 INFO:teuthology.orchestra.run.smithi114.stdout: "enum_allowed": [], 2024-04-23T10:41:08.639 INFO:teuthology.orchestra.run.smithi114.stdout: "desc": "", 2024-04-23T10:41:08.639 INFO:teuthology.orchestra.run.smithi114.stdout: "long_desc": "", 2024-04-23T10:41:08.639 INFO:teuthology.orchestra.run.smithi114.stdout: "tags": [], 2024-04-23T10:41:08.639 INFO:teuthology.orchestra.run.smithi114.stdout: "see_also": [] 2024-04-23T10:41:08.639 INFO:teuthology.orchestra.run.smithi114.stdout: } 2024-04-23T10:41:08.640 INFO:teuthology.orchestra.run.smithi114.stdout: } 2024-04-23T10:41:08.640 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-23T10:41:08.640 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-04-23T10:41:08.640 INFO:teuthology.orchestra.run.smithi114.stdout: "name": "volumes", 2024-04-23T10:41:08.640 INFO:teuthology.orchestra.run.smithi114.stdout: "can_run": true, 2024-04-23T10:41:08.640 INFO:teuthology.orchestra.run.smithi114.stdout: "error_string": "", 2024-04-23T10:41:08.640 INFO:teuthology.orchestra.run.smithi114.stdout: "module_options": { 2024-04-23T10:41:08.640 INFO:teuthology.orchestra.run.smithi114.stdout: "log_level": { 2024-04-23T10:41:08.640 INFO:teuthology.orchestra.run.smithi114.stdout: "name": "log_level", 2024-04-23T10:41:08.640 INFO:teuthology.orchestra.run.smithi114.stdout: "type": "str", 2024-04-23T10:41:08.641 INFO:teuthology.orchestra.run.smithi114.stdout: "level": "advanced", 2024-04-23T10:41:08.641 INFO:teuthology.orchestra.run.smithi114.stdout: "flags": 1, 2024-04-23T10:41:08.641 INFO:teuthology.orchestra.run.smithi114.stdout: "default_value": "", 2024-04-23T10:41:08.641 INFO:teuthology.orchestra.run.smithi114.stdout: "min": "", 2024-04-23T10:41:08.641 INFO:teuthology.orchestra.run.smithi114.stdout: "max": "", 2024-04-23T10:41:08.641 INFO:teuthology.orchestra.run.smithi114.stdout: "enum_allowed": [ 2024-04-23T10:41:08.641 INFO:teuthology.orchestra.run.smithi114.stdout: "", 2024-04-23T10:41:08.641 INFO:teuthology.orchestra.run.smithi114.stdout: "critical", 2024-04-23T10:41:08.641 INFO:teuthology.orchestra.run.smithi114.stdout: "debug", 2024-04-23T10:41:08.641 INFO:teuthology.orchestra.run.smithi114.stdout: "error", 2024-04-23T10:41:08.642 INFO:teuthology.orchestra.run.smithi114.stdout: "info", 2024-04-23T10:41:08.642 INFO:teuthology.orchestra.run.smithi114.stdout: "warning" 2024-04-23T10:41:08.642 INFO:teuthology.orchestra.run.smithi114.stdout: ], 2024-04-23T10:41:08.642 INFO:teuthology.orchestra.run.smithi114.stdout: "desc": "", 2024-04-23T10:41:08.642 INFO:teuthology.orchestra.run.smithi114.stdout: "long_desc": "", 2024-04-23T10:41:08.642 INFO:teuthology.orchestra.run.smithi114.stdout: "tags": [], 2024-04-23T10:41:08.642 INFO:teuthology.orchestra.run.smithi114.stdout: "see_also": [] 2024-04-23T10:41:08.642 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-23T10:41:08.642 INFO:teuthology.orchestra.run.smithi114.stdout: "log_to_cluster": { 2024-04-23T10:41:08.643 INFO:teuthology.orchestra.run.smithi114.stdout: "name": "log_to_cluster", 2024-04-23T10:41:08.643 INFO:teuthology.orchestra.run.smithi114.stdout: "type": "bool", 2024-04-23T10:41:08.643 INFO:teuthology.orchestra.run.smithi114.stdout: "level": "advanced", 2024-04-23T10:41:08.643 INFO:teuthology.orchestra.run.smithi114.stdout: "flags": 1, 2024-04-23T10:41:08.643 INFO:teuthology.orchestra.run.smithi114.stdout: "default_value": "False", 2024-04-23T10:41:08.643 INFO:teuthology.orchestra.run.smithi114.stdout: "min": "", 2024-04-23T10:41:08.643 INFO:teuthology.orchestra.run.smithi114.stdout: "max": "", 2024-04-23T10:41:08.643 INFO:teuthology.orchestra.run.smithi114.stdout: "enum_allowed": [], 2024-04-23T10:41:08.643 INFO:teuthology.orchestra.run.smithi114.stdout: "desc": "", 2024-04-23T10:41:08.644 INFO:teuthology.orchestra.run.smithi114.stdout: "long_desc": "", 2024-04-23T10:41:08.644 INFO:teuthology.orchestra.run.smithi114.stdout: "tags": [], 2024-04-23T10:41:08.644 INFO:teuthology.orchestra.run.smithi114.stdout: "see_also": [] 2024-04-23T10:41:08.644 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-23T10:41:08.644 INFO:teuthology.orchestra.run.smithi114.stdout: "log_to_cluster_level": { 2024-04-23T10:41:08.644 INFO:teuthology.orchestra.run.smithi114.stdout: "name": "log_to_cluster_level", 2024-04-23T10:41:08.644 INFO:teuthology.orchestra.run.smithi114.stdout: "type": "str", 2024-04-23T10:41:08.644 INFO:teuthology.orchestra.run.smithi114.stdout: "level": "advanced", 2024-04-23T10:41:08.644 INFO:teuthology.orchestra.run.smithi114.stdout: "flags": 1, 2024-04-23T10:41:08.645 INFO:teuthology.orchestra.run.smithi114.stdout: "default_value": "info", 2024-04-23T10:41:08.645 INFO:teuthology.orchestra.run.smithi114.stdout: "min": "", 2024-04-23T10:41:08.645 INFO:teuthology.orchestra.run.smithi114.stdout: "max": "", 2024-04-23T10:41:08.645 INFO:teuthology.orchestra.run.smithi114.stdout: "enum_allowed": [ 2024-04-23T10:41:08.645 INFO:teuthology.orchestra.run.smithi114.stdout: "", 2024-04-23T10:41:08.645 INFO:teuthology.orchestra.run.smithi114.stdout: "critical", 2024-04-23T10:41:08.645 INFO:teuthology.orchestra.run.smithi114.stdout: "debug", 2024-04-23T10:41:08.645 INFO:teuthology.orchestra.run.smithi114.stdout: "error", 2024-04-23T10:41:08.645 INFO:teuthology.orchestra.run.smithi114.stdout: "info", 2024-04-23T10:41:08.645 INFO:teuthology.orchestra.run.smithi114.stdout: "warning" 2024-04-23T10:41:08.646 INFO:teuthology.orchestra.run.smithi114.stdout: ], 2024-04-23T10:41:08.646 INFO:teuthology.orchestra.run.smithi114.stdout: "desc": "", 2024-04-23T10:41:08.646 INFO:teuthology.orchestra.run.smithi114.stdout: "long_desc": "", 2024-04-23T10:41:08.646 INFO:teuthology.orchestra.run.smithi114.stdout: "tags": [], 2024-04-23T10:41:08.646 INFO:teuthology.orchestra.run.smithi114.stdout: "see_also": [] 2024-04-23T10:41:08.646 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-23T10:41:08.646 INFO:teuthology.orchestra.run.smithi114.stdout: "log_to_file": { 2024-04-23T10:41:08.646 INFO:teuthology.orchestra.run.smithi114.stdout: "name": "log_to_file", 2024-04-23T10:41:08.646 INFO:teuthology.orchestra.run.smithi114.stdout: "type": "bool", 2024-04-23T10:41:08.647 INFO:teuthology.orchestra.run.smithi114.stdout: "level": "advanced", 2024-04-23T10:41:08.647 INFO:teuthology.orchestra.run.smithi114.stdout: "flags": 1, 2024-04-23T10:41:08.647 INFO:teuthology.orchestra.run.smithi114.stdout: "default_value": "False", 2024-04-23T10:41:08.647 INFO:teuthology.orchestra.run.smithi114.stdout: "min": "", 2024-04-23T10:41:08.647 INFO:teuthology.orchestra.run.smithi114.stdout: "max": "", 2024-04-23T10:41:08.647 INFO:teuthology.orchestra.run.smithi114.stdout: "enum_allowed": [], 2024-04-23T10:41:08.647 INFO:teuthology.orchestra.run.smithi114.stdout: "desc": "", 2024-04-23T10:41:08.647 INFO:teuthology.orchestra.run.smithi114.stdout: "long_desc": "", 2024-04-23T10:41:08.647 INFO:teuthology.orchestra.run.smithi114.stdout: "tags": [], 2024-04-23T10:41:08.647 INFO:teuthology.orchestra.run.smithi114.stdout: "see_also": [] 2024-04-23T10:41:08.648 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-23T10:41:08.648 INFO:teuthology.orchestra.run.smithi114.stdout: "max_concurrent_clones": { 2024-04-23T10:41:08.648 INFO:teuthology.orchestra.run.smithi114.stdout: "name": "max_concurrent_clones", 2024-04-23T10:41:08.648 INFO:teuthology.orchestra.run.smithi114.stdout: "type": "int", 2024-04-23T10:41:08.648 INFO:teuthology.orchestra.run.smithi114.stdout: "level": "advanced", 2024-04-23T10:41:08.648 INFO:teuthology.orchestra.run.smithi114.stdout: "flags": 0, 2024-04-23T10:41:08.648 INFO:teuthology.orchestra.run.smithi114.stdout: "default_value": "4", 2024-04-23T10:41:08.648 INFO:teuthology.orchestra.run.smithi114.stdout: "min": "", 2024-04-23T10:41:08.648 INFO:teuthology.orchestra.run.smithi114.stdout: "max": "", 2024-04-23T10:41:08.649 INFO:teuthology.orchestra.run.smithi114.stdout: "enum_allowed": [], 2024-04-23T10:41:08.649 INFO:teuthology.orchestra.run.smithi114.stdout: "desc": "Number of asynchronous cloner threads", 2024-04-23T10:41:08.649 INFO:teuthology.orchestra.run.smithi114.stdout: "long_desc": "", 2024-04-23T10:41:08.649 INFO:teuthology.orchestra.run.smithi114.stdout: "tags": [], 2024-04-23T10:41:08.649 INFO:teuthology.orchestra.run.smithi114.stdout: "see_also": [] 2024-04-23T10:41:08.649 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-23T10:41:08.649 INFO:teuthology.orchestra.run.smithi114.stdout: "periodic_async_work": { 2024-04-23T10:41:08.649 INFO:teuthology.orchestra.run.smithi114.stdout: "name": "periodic_async_work", 2024-04-23T10:41:08.649 INFO:teuthology.orchestra.run.smithi114.stdout: "type": "bool", 2024-04-23T10:41:08.649 INFO:teuthology.orchestra.run.smithi114.stdout: "level": "advanced", 2024-04-23T10:41:08.650 INFO:teuthology.orchestra.run.smithi114.stdout: "flags": 0, 2024-04-23T10:41:08.650 INFO:teuthology.orchestra.run.smithi114.stdout: "default_value": "False", 2024-04-23T10:41:08.650 INFO:teuthology.orchestra.run.smithi114.stdout: "min": "", 2024-04-23T10:41:08.650 INFO:teuthology.orchestra.run.smithi114.stdout: "max": "", 2024-04-23T10:41:08.650 INFO:teuthology.orchestra.run.smithi114.stdout: "enum_allowed": [], 2024-04-23T10:41:08.650 INFO:teuthology.orchestra.run.smithi114.stdout: "desc": "Periodically check for async work", 2024-04-23T10:41:08.650 INFO:teuthology.orchestra.run.smithi114.stdout: "long_desc": "", 2024-04-23T10:41:08.650 INFO:teuthology.orchestra.run.smithi114.stdout: "tags": [], 2024-04-23T10:41:08.650 INFO:teuthology.orchestra.run.smithi114.stdout: "see_also": [] 2024-04-23T10:41:08.650 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-23T10:41:08.651 INFO:teuthology.orchestra.run.smithi114.stdout: "snapshot_clone_delay": { 2024-04-23T10:41:08.651 INFO:teuthology.orchestra.run.smithi114.stdout: "name": "snapshot_clone_delay", 2024-04-23T10:41:08.651 INFO:teuthology.orchestra.run.smithi114.stdout: "type": "int", 2024-04-23T10:41:08.651 INFO:teuthology.orchestra.run.smithi114.stdout: "level": "advanced", 2024-04-23T10:41:08.651 INFO:teuthology.orchestra.run.smithi114.stdout: "flags": 0, 2024-04-23T10:41:08.651 INFO:teuthology.orchestra.run.smithi114.stdout: "default_value": "0", 2024-04-23T10:41:08.651 INFO:teuthology.orchestra.run.smithi114.stdout: "min": "", 2024-04-23T10:41:08.651 INFO:teuthology.orchestra.run.smithi114.stdout: "max": "", 2024-04-23T10:41:08.651 INFO:teuthology.orchestra.run.smithi114.stdout: "enum_allowed": [], 2024-04-23T10:41:08.651 INFO:teuthology.orchestra.run.smithi114.stdout: "desc": "Delay clone begin operation by snapshot_clone_delay seconds", 2024-04-23T10:41:08.652 INFO:teuthology.orchestra.run.smithi114.stdout: "long_desc": "", 2024-04-23T10:41:08.652 INFO:teuthology.orchestra.run.smithi114.stdout: "tags": [], 2024-04-23T10:41:08.652 INFO:teuthology.orchestra.run.smithi114.stdout: "see_also": [] 2024-04-23T10:41:08.652 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-23T10:41:08.652 INFO:teuthology.orchestra.run.smithi114.stdout: "snapshot_clone_no_wait": { 2024-04-23T10:41:08.652 INFO:teuthology.orchestra.run.smithi114.stdout: "name": "snapshot_clone_no_wait", 2024-04-23T10:41:08.652 INFO:teuthology.orchestra.run.smithi114.stdout: "type": "bool", 2024-04-23T10:41:08.652 INFO:teuthology.orchestra.run.smithi114.stdout: "level": "advanced", 2024-04-23T10:41:08.652 INFO:teuthology.orchestra.run.smithi114.stdout: "flags": 0, 2024-04-23T10:41:08.653 INFO:teuthology.orchestra.run.smithi114.stdout: "default_value": "True", 2024-04-23T10:41:08.653 INFO:teuthology.orchestra.run.smithi114.stdout: "min": "", 2024-04-23T10:41:08.653 INFO:teuthology.orchestra.run.smithi114.stdout: "max": "", 2024-04-23T10:41:08.653 INFO:teuthology.orchestra.run.smithi114.stdout: "enum_allowed": [], 2024-04-23T10:41:08.653 INFO:teuthology.orchestra.run.smithi114.stdout: "desc": "Reject subvolume clone request when cloner threads are busy", 2024-04-23T10:41:08.653 INFO:teuthology.orchestra.run.smithi114.stdout: "long_desc": "", 2024-04-23T10:41:08.653 INFO:teuthology.orchestra.run.smithi114.stdout: "tags": [], 2024-04-23T10:41:08.653 INFO:teuthology.orchestra.run.smithi114.stdout: "see_also": [] 2024-04-23T10:41:08.653 INFO:teuthology.orchestra.run.smithi114.stdout: } 2024-04-23T10:41:08.653 INFO:teuthology.orchestra.run.smithi114.stdout: } 2024-04-23T10:41:08.654 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-23T10:41:08.654 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-04-23T10:41:08.654 INFO:teuthology.orchestra.run.smithi114.stdout: "name": "zabbix", 2024-04-23T10:41:08.654 INFO:teuthology.orchestra.run.smithi114.stdout: "can_run": true, 2024-04-23T10:41:08.654 INFO:teuthology.orchestra.run.smithi114.stdout: "error_string": "", 2024-04-23T10:41:08.654 INFO:teuthology.orchestra.run.smithi114.stdout: "module_options": { 2024-04-23T10:41:08.654 INFO:teuthology.orchestra.run.smithi114.stdout: "discovery_interval": { 2024-04-23T10:41:08.654 INFO:teuthology.orchestra.run.smithi114.stdout: "name": "discovery_interval", 2024-04-23T10:41:08.654 INFO:teuthology.orchestra.run.smithi114.stdout: "type": "uint", 2024-04-23T10:41:08.655 INFO:teuthology.orchestra.run.smithi114.stdout: "level": "advanced", 2024-04-23T10:41:08.655 INFO:teuthology.orchestra.run.smithi114.stdout: "flags": 0, 2024-04-23T10:41:08.655 INFO:teuthology.orchestra.run.smithi114.stdout: "default_value": "100", 2024-04-23T10:41:08.655 INFO:teuthology.orchestra.run.smithi114.stdout: "min": "", 2024-04-23T10:41:08.655 INFO:teuthology.orchestra.run.smithi114.stdout: "max": "", 2024-04-23T10:41:08.655 INFO:teuthology.orchestra.run.smithi114.stdout: "enum_allowed": [], 2024-04-23T10:41:08.655 INFO:teuthology.orchestra.run.smithi114.stdout: "desc": "", 2024-04-23T10:41:08.655 INFO:teuthology.orchestra.run.smithi114.stdout: "long_desc": "", 2024-04-23T10:41:08.655 INFO:teuthology.orchestra.run.smithi114.stdout: "tags": [], 2024-04-23T10:41:08.655 INFO:teuthology.orchestra.run.smithi114.stdout: "see_also": [] 2024-04-23T10:41:08.656 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-23T10:41:08.656 INFO:teuthology.orchestra.run.smithi114.stdout: "identifier": { 2024-04-23T10:41:08.656 INFO:teuthology.orchestra.run.smithi114.stdout: "name": "identifier", 2024-04-23T10:41:08.656 INFO:teuthology.orchestra.run.smithi114.stdout: "type": "str", 2024-04-23T10:41:08.656 INFO:teuthology.orchestra.run.smithi114.stdout: "level": "advanced", 2024-04-23T10:41:08.656 INFO:teuthology.orchestra.run.smithi114.stdout: "flags": 0, 2024-04-23T10:41:08.656 INFO:teuthology.orchestra.run.smithi114.stdout: "default_value": "", 2024-04-23T10:41:08.656 INFO:teuthology.orchestra.run.smithi114.stdout: "min": "", 2024-04-23T10:41:08.656 INFO:teuthology.orchestra.run.smithi114.stdout: "max": "", 2024-04-23T10:41:08.656 INFO:teuthology.orchestra.run.smithi114.stdout: "enum_allowed": [], 2024-04-23T10:41:08.657 INFO:teuthology.orchestra.run.smithi114.stdout: "desc": "", 2024-04-23T10:41:08.657 INFO:teuthology.orchestra.run.smithi114.stdout: "long_desc": "", 2024-04-23T10:41:08.657 INFO:teuthology.orchestra.run.smithi114.stdout: "tags": [], 2024-04-23T10:41:08.657 INFO:teuthology.orchestra.run.smithi114.stdout: "see_also": [] 2024-04-23T10:41:08.657 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-23T10:41:08.657 INFO:teuthology.orchestra.run.smithi114.stdout: "interval": { 2024-04-23T10:41:08.657 INFO:teuthology.orchestra.run.smithi114.stdout: "name": "interval", 2024-04-23T10:41:08.657 INFO:teuthology.orchestra.run.smithi114.stdout: "type": "secs", 2024-04-23T10:41:08.657 INFO:teuthology.orchestra.run.smithi114.stdout: "level": "advanced", 2024-04-23T10:41:08.658 INFO:teuthology.orchestra.run.smithi114.stdout: "flags": 0, 2024-04-23T10:41:08.658 INFO:teuthology.orchestra.run.smithi114.stdout: "default_value": "60", 2024-04-23T10:41:08.658 INFO:teuthology.orchestra.run.smithi114.stdout: "min": "", 2024-04-23T10:41:08.658 INFO:teuthology.orchestra.run.smithi114.stdout: "max": "", 2024-04-23T10:41:08.658 INFO:teuthology.orchestra.run.smithi114.stdout: "enum_allowed": [], 2024-04-23T10:41:08.658 INFO:teuthology.orchestra.run.smithi114.stdout: "desc": "", 2024-04-23T10:41:08.658 INFO:teuthology.orchestra.run.smithi114.stdout: "long_desc": "", 2024-04-23T10:41:08.658 INFO:teuthology.orchestra.run.smithi114.stdout: "tags": [], 2024-04-23T10:41:08.658 INFO:teuthology.orchestra.run.smithi114.stdout: "see_also": [] 2024-04-23T10:41:08.658 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-23T10:41:08.659 INFO:teuthology.orchestra.run.smithi114.stdout: "log_level": { 2024-04-23T10:41:08.659 INFO:teuthology.orchestra.run.smithi114.stdout: "name": "log_level", 2024-04-23T10:41:08.659 INFO:teuthology.orchestra.run.smithi114.stdout: "type": "str", 2024-04-23T10:41:08.659 INFO:teuthology.orchestra.run.smithi114.stdout: "level": "advanced", 2024-04-23T10:41:08.659 INFO:teuthology.orchestra.run.smithi114.stdout: "flags": 1, 2024-04-23T10:41:08.659 INFO:teuthology.orchestra.run.smithi114.stdout: "default_value": "", 2024-04-23T10:41:08.659 INFO:teuthology.orchestra.run.smithi114.stdout: "min": "", 2024-04-23T10:41:08.659 INFO:teuthology.orchestra.run.smithi114.stdout: "max": "", 2024-04-23T10:41:08.659 INFO:teuthology.orchestra.run.smithi114.stdout: "enum_allowed": [ 2024-04-23T10:41:08.659 INFO:teuthology.orchestra.run.smithi114.stdout: "", 2024-04-23T10:41:08.660 INFO:teuthology.orchestra.run.smithi114.stdout: "critical", 2024-04-23T10:41:08.660 INFO:teuthology.orchestra.run.smithi114.stdout: "debug", 2024-04-23T10:41:08.660 INFO:teuthology.orchestra.run.smithi114.stdout: "error", 2024-04-23T10:41:08.660 INFO:teuthology.orchestra.run.smithi114.stdout: "info", 2024-04-23T10:41:08.660 INFO:teuthology.orchestra.run.smithi114.stdout: "warning" 2024-04-23T10:41:08.660 INFO:teuthology.orchestra.run.smithi114.stdout: ], 2024-04-23T10:41:08.660 INFO:teuthology.orchestra.run.smithi114.stdout: "desc": "", 2024-04-23T10:41:08.660 INFO:teuthology.orchestra.run.smithi114.stdout: "long_desc": "", 2024-04-23T10:41:08.660 INFO:teuthology.orchestra.run.smithi114.stdout: "tags": [], 2024-04-23T10:41:08.661 INFO:teuthology.orchestra.run.smithi114.stdout: "see_also": [] 2024-04-23T10:41:08.661 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-23T10:41:08.661 INFO:teuthology.orchestra.run.smithi114.stdout: "log_to_cluster": { 2024-04-23T10:41:08.661 INFO:teuthology.orchestra.run.smithi114.stdout: "name": "log_to_cluster", 2024-04-23T10:41:08.661 INFO:teuthology.orchestra.run.smithi114.stdout: "type": "bool", 2024-04-23T10:41:08.661 INFO:teuthology.orchestra.run.smithi114.stdout: "level": "advanced", 2024-04-23T10:41:08.661 INFO:teuthology.orchestra.run.smithi114.stdout: "flags": 1, 2024-04-23T10:41:08.661 INFO:teuthology.orchestra.run.smithi114.stdout: "default_value": "False", 2024-04-23T10:41:08.661 INFO:teuthology.orchestra.run.smithi114.stdout: "min": "", 2024-04-23T10:41:08.661 INFO:teuthology.orchestra.run.smithi114.stdout: "max": "", 2024-04-23T10:41:08.662 INFO:teuthology.orchestra.run.smithi114.stdout: "enum_allowed": [], 2024-04-23T10:41:08.662 INFO:teuthology.orchestra.run.smithi114.stdout: "desc": "", 2024-04-23T10:41:08.662 INFO:teuthology.orchestra.run.smithi114.stdout: "long_desc": "", 2024-04-23T10:41:08.662 INFO:teuthology.orchestra.run.smithi114.stdout: "tags": [], 2024-04-23T10:41:08.662 INFO:teuthology.orchestra.run.smithi114.stdout: "see_also": [] 2024-04-23T10:41:08.662 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-23T10:41:08.662 INFO:teuthology.orchestra.run.smithi114.stdout: "log_to_cluster_level": { 2024-04-23T10:41:08.662 INFO:teuthology.orchestra.run.smithi114.stdout: "name": "log_to_cluster_level", 2024-04-23T10:41:08.662 INFO:teuthology.orchestra.run.smithi114.stdout: "type": "str", 2024-04-23T10:41:08.663 INFO:teuthology.orchestra.run.smithi114.stdout: "level": "advanced", 2024-04-23T10:41:08.663 INFO:teuthology.orchestra.run.smithi114.stdout: "flags": 1, 2024-04-23T10:41:08.663 INFO:teuthology.orchestra.run.smithi114.stdout: "default_value": "info", 2024-04-23T10:41:08.663 INFO:teuthology.orchestra.run.smithi114.stdout: "min": "", 2024-04-23T10:41:08.663 INFO:teuthology.orchestra.run.smithi114.stdout: "max": "", 2024-04-23T10:41:08.663 INFO:teuthology.orchestra.run.smithi114.stdout: "enum_allowed": [ 2024-04-23T10:41:08.663 INFO:teuthology.orchestra.run.smithi114.stdout: "", 2024-04-23T10:41:08.663 INFO:teuthology.orchestra.run.smithi114.stdout: "critical", 2024-04-23T10:41:08.663 INFO:teuthology.orchestra.run.smithi114.stdout: "debug", 2024-04-23T10:41:08.663 INFO:teuthology.orchestra.run.smithi114.stdout: "error", 2024-04-23T10:41:08.664 INFO:teuthology.orchestra.run.smithi114.stdout: "info", 2024-04-23T10:41:08.664 INFO:teuthology.orchestra.run.smithi114.stdout: "warning" 2024-04-23T10:41:08.664 INFO:teuthology.orchestra.run.smithi114.stdout: ], 2024-04-23T10:41:08.664 INFO:teuthology.orchestra.run.smithi114.stdout: "desc": "", 2024-04-23T10:41:08.664 INFO:teuthology.orchestra.run.smithi114.stdout: "long_desc": "", 2024-04-23T10:41:08.664 INFO:teuthology.orchestra.run.smithi114.stdout: "tags": [], 2024-04-23T10:41:08.664 INFO:teuthology.orchestra.run.smithi114.stdout: "see_also": [] 2024-04-23T10:41:08.664 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-23T10:41:08.664 INFO:teuthology.orchestra.run.smithi114.stdout: "log_to_file": { 2024-04-23T10:41:08.664 INFO:teuthology.orchestra.run.smithi114.stdout: "name": "log_to_file", 2024-04-23T10:41:08.664 INFO:teuthology.orchestra.run.smithi114.stdout: "type": "bool", 2024-04-23T10:41:08.665 INFO:teuthology.orchestra.run.smithi114.stdout: "level": "advanced", 2024-04-23T10:41:08.665 INFO:teuthology.orchestra.run.smithi114.stdout: "flags": 1, 2024-04-23T10:41:08.665 INFO:teuthology.orchestra.run.smithi114.stdout: "default_value": "False", 2024-04-23T10:41:08.665 INFO:teuthology.orchestra.run.smithi114.stdout: "min": "", 2024-04-23T10:41:08.665 INFO:teuthology.orchestra.run.smithi114.stdout: "max": "", 2024-04-23T10:41:08.665 INFO:teuthology.orchestra.run.smithi114.stdout: "enum_allowed": [], 2024-04-23T10:41:08.665 INFO:teuthology.orchestra.run.smithi114.stdout: "desc": "", 2024-04-23T10:41:08.665 INFO:teuthology.orchestra.run.smithi114.stdout: "long_desc": "", 2024-04-23T10:41:08.665 INFO:teuthology.orchestra.run.smithi114.stdout: "tags": [], 2024-04-23T10:41:08.666 INFO:teuthology.orchestra.run.smithi114.stdout: "see_also": [] 2024-04-23T10:41:08.666 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-23T10:41:08.666 INFO:teuthology.orchestra.run.smithi114.stdout: "zabbix_host": { 2024-04-23T10:41:08.666 INFO:teuthology.orchestra.run.smithi114.stdout: "name": "zabbix_host", 2024-04-23T10:41:08.666 INFO:teuthology.orchestra.run.smithi114.stdout: "type": "str", 2024-04-23T10:41:08.666 INFO:teuthology.orchestra.run.smithi114.stdout: "level": "advanced", 2024-04-23T10:41:08.666 INFO:teuthology.orchestra.run.smithi114.stdout: "flags": 0, 2024-04-23T10:41:08.666 INFO:teuthology.orchestra.run.smithi114.stdout: "default_value": "", 2024-04-23T10:41:08.666 INFO:teuthology.orchestra.run.smithi114.stdout: "min": "", 2024-04-23T10:41:08.666 INFO:teuthology.orchestra.run.smithi114.stdout: "max": "", 2024-04-23T10:41:08.667 INFO:teuthology.orchestra.run.smithi114.stdout: "enum_allowed": [], 2024-04-23T10:41:08.667 INFO:teuthology.orchestra.run.smithi114.stdout: "desc": "", 2024-04-23T10:41:08.667 INFO:teuthology.orchestra.run.smithi114.stdout: "long_desc": "", 2024-04-23T10:41:08.667 INFO:teuthology.orchestra.run.smithi114.stdout: "tags": [], 2024-04-23T10:41:08.667 INFO:teuthology.orchestra.run.smithi114.stdout: "see_also": [] 2024-04-23T10:41:08.667 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-23T10:41:08.667 INFO:teuthology.orchestra.run.smithi114.stdout: "zabbix_port": { 2024-04-23T10:41:08.667 INFO:teuthology.orchestra.run.smithi114.stdout: "name": "zabbix_port", 2024-04-23T10:41:08.667 INFO:teuthology.orchestra.run.smithi114.stdout: "type": "int", 2024-04-23T10:41:08.667 INFO:teuthology.orchestra.run.smithi114.stdout: "level": "advanced", 2024-04-23T10:41:08.668 INFO:teuthology.orchestra.run.smithi114.stdout: "flags": 0, 2024-04-23T10:41:08.668 INFO:teuthology.orchestra.run.smithi114.stdout: "default_value": "10051", 2024-04-23T10:41:08.668 INFO:teuthology.orchestra.run.smithi114.stdout: "min": "", 2024-04-23T10:41:08.668 INFO:teuthology.orchestra.run.smithi114.stdout: "max": "", 2024-04-23T10:41:08.668 INFO:teuthology.orchestra.run.smithi114.stdout: "enum_allowed": [], 2024-04-23T10:41:08.668 INFO:teuthology.orchestra.run.smithi114.stdout: "desc": "", 2024-04-23T10:41:08.668 INFO:teuthology.orchestra.run.smithi114.stdout: "long_desc": "", 2024-04-23T10:41:08.668 INFO:teuthology.orchestra.run.smithi114.stdout: "tags": [], 2024-04-23T10:41:08.668 INFO:teuthology.orchestra.run.smithi114.stdout: "see_also": [] 2024-04-23T10:41:08.668 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-23T10:41:08.669 INFO:teuthology.orchestra.run.smithi114.stdout: "zabbix_sender": { 2024-04-23T10:41:08.669 INFO:teuthology.orchestra.run.smithi114.stdout: "name": "zabbix_sender", 2024-04-23T10:41:08.669 INFO:teuthology.orchestra.run.smithi114.stdout: "type": "str", 2024-04-23T10:41:08.669 INFO:teuthology.orchestra.run.smithi114.stdout: "level": "advanced", 2024-04-23T10:41:08.669 INFO:teuthology.orchestra.run.smithi114.stdout: "flags": 0, 2024-04-23T10:41:08.669 INFO:teuthology.orchestra.run.smithi114.stdout: "default_value": "/usr/bin/zabbix_sender", 2024-04-23T10:41:08.669 INFO:teuthology.orchestra.run.smithi114.stdout: "min": "", 2024-04-23T10:41:08.669 INFO:teuthology.orchestra.run.smithi114.stdout: "max": "", 2024-04-23T10:41:08.669 INFO:teuthology.orchestra.run.smithi114.stdout: "enum_allowed": [], 2024-04-23T10:41:08.669 INFO:teuthology.orchestra.run.smithi114.stdout: "desc": "", 2024-04-23T10:41:08.670 INFO:teuthology.orchestra.run.smithi114.stdout: "long_desc": "", 2024-04-23T10:41:08.670 INFO:teuthology.orchestra.run.smithi114.stdout: "tags": [], 2024-04-23T10:41:08.670 INFO:teuthology.orchestra.run.smithi114.stdout: "see_also": [] 2024-04-23T10:41:08.670 INFO:teuthology.orchestra.run.smithi114.stdout: } 2024-04-23T10:41:08.670 INFO:teuthology.orchestra.run.smithi114.stdout: } 2024-04-23T10:41:08.670 INFO:teuthology.orchestra.run.smithi114.stdout: } 2024-04-23T10:41:08.670 INFO:teuthology.orchestra.run.smithi114.stdout: ] 2024-04-23T10:41:08.670 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-23T10:41:08.670 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-04-23T10:41:08.671 INFO:teuthology.orchestra.run.smithi114.stdout: "gid": 24902, 2024-04-23T10:41:08.671 INFO:teuthology.orchestra.run.smithi114.stdout: "name": "y", 2024-04-23T10:41:08.671 INFO:teuthology.orchestra.run.smithi114.stdout: "mgr_features": 4540701547738038271, 2024-04-23T10:41:08.671 INFO:teuthology.orchestra.run.smithi114.stdout: "available_modules": [ 2024-04-23T10:41:08.671 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-04-23T10:41:08.671 INFO:teuthology.orchestra.run.smithi114.stdout: "name": "alerts", 2024-04-23T10:41:08.671 INFO:teuthology.orchestra.run.smithi114.stdout: "can_run": true, 2024-04-23T10:41:08.671 INFO:teuthology.orchestra.run.smithi114.stdout: "error_string": "", 2024-04-23T10:41:08.671 INFO:teuthology.orchestra.run.smithi114.stdout: "module_options": { 2024-04-23T10:41:08.671 INFO:teuthology.orchestra.run.smithi114.stdout: "interval": { 2024-04-23T10:41:08.672 INFO:teuthology.orchestra.run.smithi114.stdout: "name": "interval", 2024-04-23T10:41:08.672 INFO:teuthology.orchestra.run.smithi114.stdout: "type": "secs", 2024-04-23T10:41:08.672 INFO:teuthology.orchestra.run.smithi114.stdout: "level": "advanced", 2024-04-23T10:41:08.672 INFO:teuthology.orchestra.run.smithi114.stdout: "flags": 1, 2024-04-23T10:41:08.672 INFO:teuthology.orchestra.run.smithi114.stdout: "default_value": "60", 2024-04-23T10:41:08.672 INFO:teuthology.orchestra.run.smithi114.stdout: "min": "", 2024-04-23T10:41:08.672 INFO:teuthology.orchestra.run.smithi114.stdout: "max": "", 2024-04-23T10:41:08.672 INFO:teuthology.orchestra.run.smithi114.stdout: "enum_allowed": [], 2024-04-23T10:41:08.672 INFO:teuthology.orchestra.run.smithi114.stdout: "desc": "How frequently to reexamine health status", 2024-04-23T10:41:08.672 INFO:teuthology.orchestra.run.smithi114.stdout: "long_desc": "", 2024-04-23T10:41:08.673 INFO:teuthology.orchestra.run.smithi114.stdout: "tags": [], 2024-04-23T10:41:08.673 INFO:teuthology.orchestra.run.smithi114.stdout: "see_also": [] 2024-04-23T10:41:08.673 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-23T10:41:08.673 INFO:teuthology.orchestra.run.smithi114.stdout: "log_level": { 2024-04-23T10:41:08.673 INFO:teuthology.orchestra.run.smithi114.stdout: "name": "log_level", 2024-04-23T10:41:08.673 INFO:teuthology.orchestra.run.smithi114.stdout: "type": "str", 2024-04-23T10:41:08.673 INFO:teuthology.orchestra.run.smithi114.stdout: "level": "advanced", 2024-04-23T10:41:08.673 INFO:teuthology.orchestra.run.smithi114.stdout: "flags": 1, 2024-04-23T10:41:08.673 INFO:teuthology.orchestra.run.smithi114.stdout: "default_value": "", 2024-04-23T10:41:08.673 INFO:teuthology.orchestra.run.smithi114.stdout: "min": "", 2024-04-23T10:41:08.674 INFO:teuthology.orchestra.run.smithi114.stdout: "max": "", 2024-04-23T10:41:08.674 INFO:teuthology.orchestra.run.smithi114.stdout: "enum_allowed": [ 2024-04-23T10:41:08.674 INFO:teuthology.orchestra.run.smithi114.stdout: "", 2024-04-23T10:41:08.674 INFO:teuthology.orchestra.run.smithi114.stdout: "critical", 2024-04-23T10:41:08.674 INFO:teuthology.orchestra.run.smithi114.stdout: "debug", 2024-04-23T10:41:08.674 INFO:teuthology.orchestra.run.smithi114.stdout: "error", 2024-04-23T10:41:08.674 INFO:teuthology.orchestra.run.smithi114.stdout: "info", 2024-04-23T10:41:08.674 INFO:teuthology.orchestra.run.smithi114.stdout: "warning" 2024-04-23T10:41:08.674 INFO:teuthology.orchestra.run.smithi114.stdout: ], 2024-04-23T10:41:08.675 INFO:teuthology.orchestra.run.smithi114.stdout: "desc": "", 2024-04-23T10:41:08.675 INFO:teuthology.orchestra.run.smithi114.stdout: "long_desc": "", 2024-04-23T10:41:08.675 INFO:teuthology.orchestra.run.smithi114.stdout: "tags": [], 2024-04-23T10:41:08.675 INFO:teuthology.orchestra.run.smithi114.stdout: "see_also": [] 2024-04-23T10:41:08.675 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-23T10:41:08.675 INFO:teuthology.orchestra.run.smithi114.stdout: "log_to_cluster": { 2024-04-23T10:41:08.675 INFO:teuthology.orchestra.run.smithi114.stdout: "name": "log_to_cluster", 2024-04-23T10:41:08.675 INFO:teuthology.orchestra.run.smithi114.stdout: "type": "bool", 2024-04-23T10:41:08.675 INFO:teuthology.orchestra.run.smithi114.stdout: "level": "advanced", 2024-04-23T10:41:08.675 INFO:teuthology.orchestra.run.smithi114.stdout: "flags": 1, 2024-04-23T10:41:08.676 INFO:teuthology.orchestra.run.smithi114.stdout: "default_value": "False", 2024-04-23T10:41:08.676 INFO:teuthology.orchestra.run.smithi114.stdout: "min": "", 2024-04-23T10:41:08.676 INFO:teuthology.orchestra.run.smithi114.stdout: "max": "", 2024-04-23T10:41:08.676 INFO:teuthology.orchestra.run.smithi114.stdout: "enum_allowed": [], 2024-04-23T10:41:08.676 INFO:teuthology.orchestra.run.smithi114.stdout: "desc": "", 2024-04-23T10:41:08.676 INFO:teuthology.orchestra.run.smithi114.stdout: "long_desc": "", 2024-04-23T10:41:08.676 INFO:teuthology.orchestra.run.smithi114.stdout: "tags": [], 2024-04-23T10:41:08.676 INFO:teuthology.orchestra.run.smithi114.stdout: "see_also": [] 2024-04-23T10:41:08.676 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-23T10:41:08.676 INFO:teuthology.orchestra.run.smithi114.stdout: "log_to_cluster_level": { 2024-04-23T10:41:08.677 INFO:teuthology.orchestra.run.smithi114.stdout: "name": "log_to_cluster_level", 2024-04-23T10:41:08.677 INFO:teuthology.orchestra.run.smithi114.stdout: "type": "str", 2024-04-23T10:41:08.677 INFO:teuthology.orchestra.run.smithi114.stdout: "level": "advanced", 2024-04-23T10:41:08.677 INFO:teuthology.orchestra.run.smithi114.stdout: "flags": 1, 2024-04-23T10:41:08.677 INFO:teuthology.orchestra.run.smithi114.stdout: "default_value": "info", 2024-04-23T10:41:08.677 INFO:teuthology.orchestra.run.smithi114.stdout: "min": "", 2024-04-23T10:41:08.677 INFO:teuthology.orchestra.run.smithi114.stdout: "max": "", 2024-04-23T10:41:08.677 INFO:teuthology.orchestra.run.smithi114.stdout: "enum_allowed": [ 2024-04-23T10:41:08.677 INFO:teuthology.orchestra.run.smithi114.stdout: "", 2024-04-23T10:41:08.677 INFO:teuthology.orchestra.run.smithi114.stdout: "critical", 2024-04-23T10:41:08.678 INFO:teuthology.orchestra.run.smithi114.stdout: "debug", 2024-04-23T10:41:08.678 INFO:teuthology.orchestra.run.smithi114.stdout: "error", 2024-04-23T10:41:08.678 INFO:teuthology.orchestra.run.smithi114.stdout: "info", 2024-04-23T10:41:08.678 INFO:teuthology.orchestra.run.smithi114.stdout: "warning" 2024-04-23T10:41:08.678 INFO:teuthology.orchestra.run.smithi114.stdout: ], 2024-04-23T10:41:08.678 INFO:teuthology.orchestra.run.smithi114.stdout: "desc": "", 2024-04-23T10:41:08.678 INFO:teuthology.orchestra.run.smithi114.stdout: "long_desc": "", 2024-04-23T10:41:08.678 INFO:teuthology.orchestra.run.smithi114.stdout: "tags": [], 2024-04-23T10:41:08.678 INFO:teuthology.orchestra.run.smithi114.stdout: "see_also": [] 2024-04-23T10:41:08.679 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-23T10:41:08.679 INFO:teuthology.orchestra.run.smithi114.stdout: "log_to_file": { 2024-04-23T10:41:08.679 INFO:teuthology.orchestra.run.smithi114.stdout: "name": "log_to_file", 2024-04-23T10:41:08.679 INFO:teuthology.orchestra.run.smithi114.stdout: "type": "bool", 2024-04-23T10:41:08.679 INFO:teuthology.orchestra.run.smithi114.stdout: "level": "advanced", 2024-04-23T10:41:08.679 INFO:teuthology.orchestra.run.smithi114.stdout: "flags": 1, 2024-04-23T10:41:08.679 INFO:teuthology.orchestra.run.smithi114.stdout: "default_value": "False", 2024-04-23T10:41:08.679 INFO:teuthology.orchestra.run.smithi114.stdout: "min": "", 2024-04-23T10:41:08.679 INFO:teuthology.orchestra.run.smithi114.stdout: "max": "", 2024-04-23T10:41:08.679 INFO:teuthology.orchestra.run.smithi114.stdout: "enum_allowed": [], 2024-04-23T10:41:08.680 INFO:teuthology.orchestra.run.smithi114.stdout: "desc": "", 2024-04-23T10:41:08.680 INFO:teuthology.orchestra.run.smithi114.stdout: "long_desc": "", 2024-04-23T10:41:08.680 INFO:teuthology.orchestra.run.smithi114.stdout: "tags": [], 2024-04-23T10:41:08.680 INFO:teuthology.orchestra.run.smithi114.stdout: "see_also": [] 2024-04-23T10:41:08.680 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-23T10:41:08.680 INFO:teuthology.orchestra.run.smithi114.stdout: "smtp_destination": { 2024-04-23T10:41:08.680 INFO:teuthology.orchestra.run.smithi114.stdout: "name": "smtp_destination", 2024-04-23T10:41:08.680 INFO:teuthology.orchestra.run.smithi114.stdout: "type": "str", 2024-04-23T10:41:08.680 INFO:teuthology.orchestra.run.smithi114.stdout: "level": "advanced", 2024-04-23T10:41:08.680 INFO:teuthology.orchestra.run.smithi114.stdout: "flags": 1, 2024-04-23T10:41:08.681 INFO:teuthology.orchestra.run.smithi114.stdout: "default_value": "", 2024-04-23T10:41:08.681 INFO:teuthology.orchestra.run.smithi114.stdout: "min": "", 2024-04-23T10:41:08.681 INFO:teuthology.orchestra.run.smithi114.stdout: "max": "", 2024-04-23T10:41:08.681 INFO:teuthology.orchestra.run.smithi114.stdout: "enum_allowed": [], 2024-04-23T10:41:08.681 INFO:teuthology.orchestra.run.smithi114.stdout: "desc": "Email address to send alerts to", 2024-04-23T10:41:08.681 INFO:teuthology.orchestra.run.smithi114.stdout: "long_desc": "", 2024-04-23T10:41:08.681 INFO:teuthology.orchestra.run.smithi114.stdout: "tags": [], 2024-04-23T10:41:08.681 INFO:teuthology.orchestra.run.smithi114.stdout: "see_also": [] 2024-04-23T10:41:08.681 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-23T10:41:08.681 INFO:teuthology.orchestra.run.smithi114.stdout: "smtp_from_name": { 2024-04-23T10:41:08.682 INFO:teuthology.orchestra.run.smithi114.stdout: "name": "smtp_from_name", 2024-04-23T10:41:08.682 INFO:teuthology.orchestra.run.smithi114.stdout: "type": "str", 2024-04-23T10:41:08.682 INFO:teuthology.orchestra.run.smithi114.stdout: "level": "advanced", 2024-04-23T10:41:08.682 INFO:teuthology.orchestra.run.smithi114.stdout: "flags": 1, 2024-04-23T10:41:08.682 INFO:teuthology.orchestra.run.smithi114.stdout: "default_value": "Ceph", 2024-04-23T10:41:08.682 INFO:teuthology.orchestra.run.smithi114.stdout: "min": "", 2024-04-23T10:41:08.682 INFO:teuthology.orchestra.run.smithi114.stdout: "max": "", 2024-04-23T10:41:08.682 INFO:teuthology.orchestra.run.smithi114.stdout: "enum_allowed": [], 2024-04-23T10:41:08.682 INFO:teuthology.orchestra.run.smithi114.stdout: "desc": "Email From: name", 2024-04-23T10:41:08.682 INFO:teuthology.orchestra.run.smithi114.stdout: "long_desc": "", 2024-04-23T10:41:08.683 INFO:teuthology.orchestra.run.smithi114.stdout: "tags": [], 2024-04-23T10:41:08.683 INFO:teuthology.orchestra.run.smithi114.stdout: "see_also": [] 2024-04-23T10:41:08.683 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-23T10:41:08.683 INFO:teuthology.orchestra.run.smithi114.stdout: "smtp_host": { 2024-04-23T10:41:08.683 INFO:teuthology.orchestra.run.smithi114.stdout: "name": "smtp_host", 2024-04-23T10:41:08.683 INFO:teuthology.orchestra.run.smithi114.stdout: "type": "str", 2024-04-23T10:41:08.683 INFO:teuthology.orchestra.run.smithi114.stdout: "level": "advanced", 2024-04-23T10:41:08.683 INFO:teuthology.orchestra.run.smithi114.stdout: "flags": 1, 2024-04-23T10:41:08.683 INFO:teuthology.orchestra.run.smithi114.stdout: "default_value": "", 2024-04-23T10:41:08.683 INFO:teuthology.orchestra.run.smithi114.stdout: "min": "", 2024-04-23T10:41:08.684 INFO:teuthology.orchestra.run.smithi114.stdout: "max": "", 2024-04-23T10:41:08.684 INFO:teuthology.orchestra.run.smithi114.stdout: "enum_allowed": [], 2024-04-23T10:41:08.684 INFO:teuthology.orchestra.run.smithi114.stdout: "desc": "SMTP server", 2024-04-23T10:41:08.684 INFO:teuthology.orchestra.run.smithi114.stdout: "long_desc": "", 2024-04-23T10:41:08.684 INFO:teuthology.orchestra.run.smithi114.stdout: "tags": [], 2024-04-23T10:41:08.684 INFO:teuthology.orchestra.run.smithi114.stdout: "see_also": [] 2024-04-23T10:41:08.684 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-23T10:41:08.684 INFO:teuthology.orchestra.run.smithi114.stdout: "smtp_password": { 2024-04-23T10:41:08.684 INFO:teuthology.orchestra.run.smithi114.stdout: "name": "smtp_password", 2024-04-23T10:41:08.685 INFO:teuthology.orchestra.run.smithi114.stdout: "type": "str", 2024-04-23T10:41:08.685 INFO:teuthology.orchestra.run.smithi114.stdout: "level": "advanced", 2024-04-23T10:41:08.685 INFO:teuthology.orchestra.run.smithi114.stdout: "flags": 1, 2024-04-23T10:41:08.685 INFO:teuthology.orchestra.run.smithi114.stdout: "default_value": "", 2024-04-23T10:41:08.685 INFO:teuthology.orchestra.run.smithi114.stdout: "min": "", 2024-04-23T10:41:08.685 INFO:teuthology.orchestra.run.smithi114.stdout: "max": "", 2024-04-23T10:41:08.685 INFO:teuthology.orchestra.run.smithi114.stdout: "enum_allowed": [], 2024-04-23T10:41:08.685 INFO:teuthology.orchestra.run.smithi114.stdout: "desc": "Password to authenticate with", 2024-04-23T10:41:08.685 INFO:teuthology.orchestra.run.smithi114.stdout: "long_desc": "", 2024-04-23T10:41:08.685 INFO:teuthology.orchestra.run.smithi114.stdout: "tags": [], 2024-04-23T10:41:08.686 INFO:teuthology.orchestra.run.smithi114.stdout: "see_also": [] 2024-04-23T10:41:08.686 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-23T10:41:08.686 INFO:teuthology.orchestra.run.smithi114.stdout: "smtp_port": { 2024-04-23T10:41:08.686 INFO:teuthology.orchestra.run.smithi114.stdout: "name": "smtp_port", 2024-04-23T10:41:08.686 INFO:teuthology.orchestra.run.smithi114.stdout: "type": "int", 2024-04-23T10:41:08.686 INFO:teuthology.orchestra.run.smithi114.stdout: "level": "advanced", 2024-04-23T10:41:08.686 INFO:teuthology.orchestra.run.smithi114.stdout: "flags": 1, 2024-04-23T10:41:08.686 INFO:teuthology.orchestra.run.smithi114.stdout: "default_value": "465", 2024-04-23T10:41:08.686 INFO:teuthology.orchestra.run.smithi114.stdout: "min": "", 2024-04-23T10:41:08.687 INFO:teuthology.orchestra.run.smithi114.stdout: "max": "", 2024-04-23T10:41:08.687 INFO:teuthology.orchestra.run.smithi114.stdout: "enum_allowed": [], 2024-04-23T10:41:08.687 INFO:teuthology.orchestra.run.smithi114.stdout: "desc": "SMTP port", 2024-04-23T10:41:08.687 INFO:teuthology.orchestra.run.smithi114.stdout: "long_desc": "", 2024-04-23T10:41:08.687 INFO:teuthology.orchestra.run.smithi114.stdout: "tags": [], 2024-04-23T10:41:08.687 INFO:teuthology.orchestra.run.smithi114.stdout: "see_also": [] 2024-04-23T10:41:08.687 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-23T10:41:08.687 INFO:teuthology.orchestra.run.smithi114.stdout: "smtp_sender": { 2024-04-23T10:41:08.687 INFO:teuthology.orchestra.run.smithi114.stdout: "name": "smtp_sender", 2024-04-23T10:41:08.687 INFO:teuthology.orchestra.run.smithi114.stdout: "type": "str", 2024-04-23T10:41:08.688 INFO:teuthology.orchestra.run.smithi114.stdout: "level": "advanced", 2024-04-23T10:41:08.688 INFO:teuthology.orchestra.run.smithi114.stdout: "flags": 1, 2024-04-23T10:41:08.688 INFO:teuthology.orchestra.run.smithi114.stdout: "default_value": "", 2024-04-23T10:41:08.688 INFO:teuthology.orchestra.run.smithi114.stdout: "min": "", 2024-04-23T10:41:08.688 INFO:teuthology.orchestra.run.smithi114.stdout: "max": "", 2024-04-23T10:41:08.688 INFO:teuthology.orchestra.run.smithi114.stdout: "enum_allowed": [], 2024-04-23T10:41:08.688 INFO:teuthology.orchestra.run.smithi114.stdout: "desc": "SMTP envelope sender", 2024-04-23T10:41:08.688 INFO:teuthology.orchestra.run.smithi114.stdout: "long_desc": "", 2024-04-23T10:41:08.688 INFO:teuthology.orchestra.run.smithi114.stdout: "tags": [], 2024-04-23T10:41:08.688 INFO:teuthology.orchestra.run.smithi114.stdout: "see_also": [] 2024-04-23T10:41:08.689 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-23T10:41:08.689 INFO:teuthology.orchestra.run.smithi114.stdout: "smtp_ssl": { 2024-04-23T10:41:08.689 INFO:teuthology.orchestra.run.smithi114.stdout: "name": "smtp_ssl", 2024-04-23T10:41:08.689 INFO:teuthology.orchestra.run.smithi114.stdout: "type": "bool", 2024-04-23T10:41:08.689 INFO:teuthology.orchestra.run.smithi114.stdout: "level": "advanced", 2024-04-23T10:41:08.689 INFO:teuthology.orchestra.run.smithi114.stdout: "flags": 1, 2024-04-23T10:41:08.689 INFO:teuthology.orchestra.run.smithi114.stdout: "default_value": "True", 2024-04-23T10:41:08.689 INFO:teuthology.orchestra.run.smithi114.stdout: "min": "", 2024-04-23T10:41:08.689 INFO:teuthology.orchestra.run.smithi114.stdout: "max": "", 2024-04-23T10:41:08.689 INFO:teuthology.orchestra.run.smithi114.stdout: "enum_allowed": [], 2024-04-23T10:41:08.690 INFO:teuthology.orchestra.run.smithi114.stdout: "desc": "Use SSL to connect to SMTP server", 2024-04-23T10:41:08.690 INFO:teuthology.orchestra.run.smithi114.stdout: "long_desc": "", 2024-04-23T10:41:08.690 INFO:teuthology.orchestra.run.smithi114.stdout: "tags": [], 2024-04-23T10:41:08.690 INFO:teuthology.orchestra.run.smithi114.stdout: "see_also": [] 2024-04-23T10:41:08.690 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-23T10:41:08.690 INFO:teuthology.orchestra.run.smithi114.stdout: "smtp_user": { 2024-04-23T10:41:08.690 INFO:teuthology.orchestra.run.smithi114.stdout: "name": "smtp_user", 2024-04-23T10:41:08.690 INFO:teuthology.orchestra.run.smithi114.stdout: "type": "str", 2024-04-23T10:41:08.690 INFO:teuthology.orchestra.run.smithi114.stdout: "level": "advanced", 2024-04-23T10:41:08.690 INFO:teuthology.orchestra.run.smithi114.stdout: "flags": 1, 2024-04-23T10:41:08.691 INFO:teuthology.orchestra.run.smithi114.stdout: "default_value": "", 2024-04-23T10:41:08.691 INFO:teuthology.orchestra.run.smithi114.stdout: "min": "", 2024-04-23T10:41:08.691 INFO:teuthology.orchestra.run.smithi114.stdout: "max": "", 2024-04-23T10:41:08.691 INFO:teuthology.orchestra.run.smithi114.stdout: "enum_allowed": [], 2024-04-23T10:41:08.691 INFO:teuthology.orchestra.run.smithi114.stdout: "desc": "User to authenticate as", 2024-04-23T10:41:08.691 INFO:teuthology.orchestra.run.smithi114.stdout: "long_desc": "", 2024-04-23T10:41:08.691 INFO:teuthology.orchestra.run.smithi114.stdout: "tags": [], 2024-04-23T10:41:08.691 INFO:teuthology.orchestra.run.smithi114.stdout: "see_also": [] 2024-04-23T10:41:08.691 INFO:teuthology.orchestra.run.smithi114.stdout: } 2024-04-23T10:41:08.691 INFO:teuthology.orchestra.run.smithi114.stdout: } 2024-04-23T10:41:08.692 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-23T10:41:08.692 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-04-23T10:41:08.692 INFO:teuthology.orchestra.run.smithi114.stdout: "name": "balancer", 2024-04-23T10:41:08.692 INFO:teuthology.orchestra.run.smithi114.stdout: "can_run": true, 2024-04-23T10:41:08.692 INFO:teuthology.orchestra.run.smithi114.stdout: "error_string": "", 2024-04-23T10:41:08.692 INFO:teuthology.orchestra.run.smithi114.stdout: "module_options": { 2024-04-23T10:41:08.692 INFO:teuthology.orchestra.run.smithi114.stdout: "active": { 2024-04-23T10:41:08.692 INFO:teuthology.orchestra.run.smithi114.stdout: "name": "active", 2024-04-23T10:41:08.692 INFO:teuthology.orchestra.run.smithi114.stdout: "type": "bool", 2024-04-23T10:41:08.692 INFO:teuthology.orchestra.run.smithi114.stdout: "level": "advanced", 2024-04-23T10:41:08.693 INFO:teuthology.orchestra.run.smithi114.stdout: "flags": 1, 2024-04-23T10:41:08.693 INFO:teuthology.orchestra.run.smithi114.stdout: "default_value": "True", 2024-04-23T10:41:08.693 INFO:teuthology.orchestra.run.smithi114.stdout: "min": "", 2024-04-23T10:41:08.693 INFO:teuthology.orchestra.run.smithi114.stdout: "max": "", 2024-04-23T10:41:08.693 INFO:teuthology.orchestra.run.smithi114.stdout: "enum_allowed": [], 2024-04-23T10:41:08.693 INFO:teuthology.orchestra.run.smithi114.stdout: "desc": "automatically balance PGs across cluster", 2024-04-23T10:41:08.693 INFO:teuthology.orchestra.run.smithi114.stdout: "long_desc": "", 2024-04-23T10:41:08.693 INFO:teuthology.orchestra.run.smithi114.stdout: "tags": [], 2024-04-23T10:41:08.693 INFO:teuthology.orchestra.run.smithi114.stdout: "see_also": [] 2024-04-23T10:41:08.693 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-23T10:41:08.694 INFO:teuthology.orchestra.run.smithi114.stdout: "begin_time": { 2024-04-23T10:41:08.694 INFO:teuthology.orchestra.run.smithi114.stdout: "name": "begin_time", 2024-04-23T10:41:08.694 INFO:teuthology.orchestra.run.smithi114.stdout: "type": "str", 2024-04-23T10:41:08.694 INFO:teuthology.orchestra.run.smithi114.stdout: "level": "advanced", 2024-04-23T10:41:08.694 INFO:teuthology.orchestra.run.smithi114.stdout: "flags": 1, 2024-04-23T10:41:08.694 INFO:teuthology.orchestra.run.smithi114.stdout: "default_value": "0000", 2024-04-23T10:41:08.694 INFO:teuthology.orchestra.run.smithi114.stdout: "min": "", 2024-04-23T10:41:08.694 INFO:teuthology.orchestra.run.smithi114.stdout: "max": "", 2024-04-23T10:41:08.694 INFO:teuthology.orchestra.run.smithi114.stdout: "enum_allowed": [], 2024-04-23T10:41:08.695 INFO:teuthology.orchestra.run.smithi114.stdout: "desc": "beginning time of day to automatically balance", 2024-04-23T10:41:08.695 INFO:teuthology.orchestra.run.smithi114.stdout: "long_desc": "This is a time of day in the format HHMM.", 2024-04-23T10:41:08.695 INFO:teuthology.orchestra.run.smithi114.stdout: "tags": [], 2024-04-23T10:41:08.695 INFO:teuthology.orchestra.run.smithi114.stdout: "see_also": [] 2024-04-23T10:41:08.695 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-23T10:41:08.695 INFO:teuthology.orchestra.run.smithi114.stdout: "begin_weekday": { 2024-04-23T10:41:08.695 INFO:teuthology.orchestra.run.smithi114.stdout: "name": "begin_weekday", 2024-04-23T10:41:08.695 INFO:teuthology.orchestra.run.smithi114.stdout: "type": "uint", 2024-04-23T10:41:08.695 INFO:teuthology.orchestra.run.smithi114.stdout: "level": "advanced", 2024-04-23T10:41:08.695 INFO:teuthology.orchestra.run.smithi114.stdout: "flags": 1, 2024-04-23T10:41:08.696 INFO:teuthology.orchestra.run.smithi114.stdout: "default_value": "0", 2024-04-23T10:41:08.696 INFO:teuthology.orchestra.run.smithi114.stdout: "min": "0", 2024-04-23T10:41:08.696 INFO:teuthology.orchestra.run.smithi114.stdout: "max": "6", 2024-04-23T10:41:08.696 INFO:teuthology.orchestra.run.smithi114.stdout: "enum_allowed": [], 2024-04-23T10:41:08.696 INFO:teuthology.orchestra.run.smithi114.stdout: "desc": "Restrict automatic balancing to this day of the week or later", 2024-04-23T10:41:08.696 INFO:teuthology.orchestra.run.smithi114.stdout: "long_desc": "0 = Sunday, 1 = Monday, etc.", 2024-04-23T10:41:08.696 INFO:teuthology.orchestra.run.smithi114.stdout: "tags": [], 2024-04-23T10:41:08.696 INFO:teuthology.orchestra.run.smithi114.stdout: "see_also": [] 2024-04-23T10:41:08.696 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-23T10:41:08.697 INFO:teuthology.orchestra.run.smithi114.stdout: "crush_compat_max_iterations": { 2024-04-23T10:41:08.697 INFO:teuthology.orchestra.run.smithi114.stdout: "name": "crush_compat_max_iterations", 2024-04-23T10:41:08.697 INFO:teuthology.orchestra.run.smithi114.stdout: "type": "uint", 2024-04-23T10:41:08.697 INFO:teuthology.orchestra.run.smithi114.stdout: "level": "advanced", 2024-04-23T10:41:08.697 INFO:teuthology.orchestra.run.smithi114.stdout: "flags": 1, 2024-04-23T10:41:08.697 INFO:teuthology.orchestra.run.smithi114.stdout: "default_value": "25", 2024-04-23T10:41:08.697 INFO:teuthology.orchestra.run.smithi114.stdout: "min": "1", 2024-04-23T10:41:08.697 INFO:teuthology.orchestra.run.smithi114.stdout: "max": "250", 2024-04-23T10:41:08.697 INFO:teuthology.orchestra.run.smithi114.stdout: "enum_allowed": [], 2024-04-23T10:41:08.697 INFO:teuthology.orchestra.run.smithi114.stdout: "desc": "maximum number of iterations to attempt optimization", 2024-04-23T10:41:08.698 INFO:teuthology.orchestra.run.smithi114.stdout: "long_desc": "", 2024-04-23T10:41:08.698 INFO:teuthology.orchestra.run.smithi114.stdout: "tags": [], 2024-04-23T10:41:08.698 INFO:teuthology.orchestra.run.smithi114.stdout: "see_also": [] 2024-04-23T10:41:08.698 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-23T10:41:08.698 INFO:teuthology.orchestra.run.smithi114.stdout: "crush_compat_metrics": { 2024-04-23T10:41:08.698 INFO:teuthology.orchestra.run.smithi114.stdout: "name": "crush_compat_metrics", 2024-04-23T10:41:08.698 INFO:teuthology.orchestra.run.smithi114.stdout: "type": "str", 2024-04-23T10:41:08.698 INFO:teuthology.orchestra.run.smithi114.stdout: "level": "advanced", 2024-04-23T10:41:08.698 INFO:teuthology.orchestra.run.smithi114.stdout: "flags": 1, 2024-04-23T10:41:08.698 INFO:teuthology.orchestra.run.smithi114.stdout: "default_value": "pgs,objects,bytes", 2024-04-23T10:41:08.699 INFO:teuthology.orchestra.run.smithi114.stdout: "min": "", 2024-04-23T10:41:08.699 INFO:teuthology.orchestra.run.smithi114.stdout: "max": "", 2024-04-23T10:41:08.699 INFO:teuthology.orchestra.run.smithi114.stdout: "enum_allowed": [], 2024-04-23T10:41:08.699 INFO:teuthology.orchestra.run.smithi114.stdout: "desc": "metrics with which to calculate OSD utilization", 2024-04-23T10:41:08.699 INFO:teuthology.orchestra.run.smithi114.stdout: "long_desc": "Value is a list of one or more of \"pgs\", \"objects\", or \"bytes\", and indicates which metrics to use to balance utilization.", 2024-04-23T10:41:08.699 INFO:teuthology.orchestra.run.smithi114.stdout: "tags": [], 2024-04-23T10:41:08.699 INFO:teuthology.orchestra.run.smithi114.stdout: "see_also": [] 2024-04-23T10:41:08.699 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-23T10:41:08.699 INFO:teuthology.orchestra.run.smithi114.stdout: "crush_compat_step": { 2024-04-23T10:41:08.699 INFO:teuthology.orchestra.run.smithi114.stdout: "name": "crush_compat_step", 2024-04-23T10:41:08.700 INFO:teuthology.orchestra.run.smithi114.stdout: "type": "float", 2024-04-23T10:41:08.700 INFO:teuthology.orchestra.run.smithi114.stdout: "level": "advanced", 2024-04-23T10:41:08.700 INFO:teuthology.orchestra.run.smithi114.stdout: "flags": 1, 2024-04-23T10:41:08.700 INFO:teuthology.orchestra.run.smithi114.stdout: "default_value": "0.5", 2024-04-23T10:41:08.700 INFO:teuthology.orchestra.run.smithi114.stdout: "min": "0.001", 2024-04-23T10:41:08.700 INFO:teuthology.orchestra.run.smithi114.stdout: "max": "0.999", 2024-04-23T10:41:08.700 INFO:teuthology.orchestra.run.smithi114.stdout: "enum_allowed": [], 2024-04-23T10:41:08.700 INFO:teuthology.orchestra.run.smithi114.stdout: "desc": "aggressiveness of optimization", 2024-04-23T10:41:08.700 INFO:teuthology.orchestra.run.smithi114.stdout: "long_desc": ".99 is very aggressive, .01 is less aggressive", 2024-04-23T10:41:08.700 INFO:teuthology.orchestra.run.smithi114.stdout: "tags": [], 2024-04-23T10:41:08.701 INFO:teuthology.orchestra.run.smithi114.stdout: "see_also": [] 2024-04-23T10:41:08.701 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-23T10:41:08.701 INFO:teuthology.orchestra.run.smithi114.stdout: "end_time": { 2024-04-23T10:41:08.701 INFO:teuthology.orchestra.run.smithi114.stdout: "name": "end_time", 2024-04-23T10:41:08.701 INFO:teuthology.orchestra.run.smithi114.stdout: "type": "str", 2024-04-23T10:41:08.701 INFO:teuthology.orchestra.run.smithi114.stdout: "level": "advanced", 2024-04-23T10:41:08.701 INFO:teuthology.orchestra.run.smithi114.stdout: "flags": 1, 2024-04-23T10:41:08.701 INFO:teuthology.orchestra.run.smithi114.stdout: "default_value": "2359", 2024-04-23T10:41:08.701 INFO:teuthology.orchestra.run.smithi114.stdout: "min": "", 2024-04-23T10:41:08.701 INFO:teuthology.orchestra.run.smithi114.stdout: "max": "", 2024-04-23T10:41:08.702 INFO:teuthology.orchestra.run.smithi114.stdout: "enum_allowed": [], 2024-04-23T10:41:08.702 INFO:teuthology.orchestra.run.smithi114.stdout: "desc": "ending time of day to automatically balance", 2024-04-23T10:41:08.702 INFO:teuthology.orchestra.run.smithi114.stdout: "long_desc": "This is a time of day in the format HHMM.", 2024-04-23T10:41:08.702 INFO:teuthology.orchestra.run.smithi114.stdout: "tags": [], 2024-04-23T10:41:08.702 INFO:teuthology.orchestra.run.smithi114.stdout: "see_also": [] 2024-04-23T10:41:08.702 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-23T10:41:08.702 INFO:teuthology.orchestra.run.smithi114.stdout: "end_weekday": { 2024-04-23T10:41:08.702 INFO:teuthology.orchestra.run.smithi114.stdout: "name": "end_weekday", 2024-04-23T10:41:08.702 INFO:teuthology.orchestra.run.smithi114.stdout: "type": "uint", 2024-04-23T10:41:08.703 INFO:teuthology.orchestra.run.smithi114.stdout: "level": "advanced", 2024-04-23T10:41:08.703 INFO:teuthology.orchestra.run.smithi114.stdout: "flags": 1, 2024-04-23T10:41:08.703 INFO:teuthology.orchestra.run.smithi114.stdout: "default_value": "0", 2024-04-23T10:41:08.703 INFO:teuthology.orchestra.run.smithi114.stdout: "min": "0", 2024-04-23T10:41:08.703 INFO:teuthology.orchestra.run.smithi114.stdout: "max": "6", 2024-04-23T10:41:08.703 INFO:teuthology.orchestra.run.smithi114.stdout: "enum_allowed": [], 2024-04-23T10:41:08.703 INFO:teuthology.orchestra.run.smithi114.stdout: "desc": "Restrict automatic balancing to days of the week earlier than this", 2024-04-23T10:41:08.703 INFO:teuthology.orchestra.run.smithi114.stdout: "long_desc": "0 = Sunday, 1 = Monday, etc.", 2024-04-23T10:41:08.703 INFO:teuthology.orchestra.run.smithi114.stdout: "tags": [], 2024-04-23T10:41:08.703 INFO:teuthology.orchestra.run.smithi114.stdout: "see_also": [] 2024-04-23T10:41:08.704 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-23T10:41:08.704 INFO:teuthology.orchestra.run.smithi114.stdout: "log_level": { 2024-04-23T10:41:08.704 INFO:teuthology.orchestra.run.smithi114.stdout: "name": "log_level", 2024-04-23T10:41:08.704 INFO:teuthology.orchestra.run.smithi114.stdout: "type": "str", 2024-04-23T10:41:08.704 INFO:teuthology.orchestra.run.smithi114.stdout: "level": "advanced", 2024-04-23T10:41:08.704 INFO:teuthology.orchestra.run.smithi114.stdout: "flags": 1, 2024-04-23T10:41:08.704 INFO:teuthology.orchestra.run.smithi114.stdout: "default_value": "", 2024-04-23T10:41:08.704 INFO:teuthology.orchestra.run.smithi114.stdout: "min": "", 2024-04-23T10:41:08.704 INFO:teuthology.orchestra.run.smithi114.stdout: "max": "", 2024-04-23T10:41:08.704 INFO:teuthology.orchestra.run.smithi114.stdout: "enum_allowed": [ 2024-04-23T10:41:08.705 INFO:teuthology.orchestra.run.smithi114.stdout: "", 2024-04-23T10:41:08.705 INFO:teuthology.orchestra.run.smithi114.stdout: "critical", 2024-04-23T10:41:08.705 INFO:teuthology.orchestra.run.smithi114.stdout: "debug", 2024-04-23T10:41:08.705 INFO:teuthology.orchestra.run.smithi114.stdout: "error", 2024-04-23T10:41:08.705 INFO:teuthology.orchestra.run.smithi114.stdout: "info", 2024-04-23T10:41:08.705 INFO:teuthology.orchestra.run.smithi114.stdout: "warning" 2024-04-23T10:41:08.705 INFO:teuthology.orchestra.run.smithi114.stdout: ], 2024-04-23T10:41:08.705 INFO:teuthology.orchestra.run.smithi114.stdout: "desc": "", 2024-04-23T10:41:08.705 INFO:teuthology.orchestra.run.smithi114.stdout: "long_desc": "", 2024-04-23T10:41:08.705 INFO:teuthology.orchestra.run.smithi114.stdout: "tags": [], 2024-04-23T10:41:08.706 INFO:teuthology.orchestra.run.smithi114.stdout: "see_also": [] 2024-04-23T10:41:08.706 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-23T10:41:08.706 INFO:teuthology.orchestra.run.smithi114.stdout: "log_to_cluster": { 2024-04-23T10:41:08.706 INFO:teuthology.orchestra.run.smithi114.stdout: "name": "log_to_cluster", 2024-04-23T10:41:08.706 INFO:teuthology.orchestra.run.smithi114.stdout: "type": "bool", 2024-04-23T10:41:08.706 INFO:teuthology.orchestra.run.smithi114.stdout: "level": "advanced", 2024-04-23T10:41:08.706 INFO:teuthology.orchestra.run.smithi114.stdout: "flags": 1, 2024-04-23T10:41:08.706 INFO:teuthology.orchestra.run.smithi114.stdout: "default_value": "False", 2024-04-23T10:41:08.706 INFO:teuthology.orchestra.run.smithi114.stdout: "min": "", 2024-04-23T10:41:08.707 INFO:teuthology.orchestra.run.smithi114.stdout: "max": "", 2024-04-23T10:41:08.707 INFO:teuthology.orchestra.run.smithi114.stdout: "enum_allowed": [], 2024-04-23T10:41:08.707 INFO:teuthology.orchestra.run.smithi114.stdout: "desc": "", 2024-04-23T10:41:08.707 INFO:teuthology.orchestra.run.smithi114.stdout: "long_desc": "", 2024-04-23T10:41:08.707 INFO:teuthology.orchestra.run.smithi114.stdout: "tags": [], 2024-04-23T10:41:08.707 INFO:teuthology.orchestra.run.smithi114.stdout: "see_also": [] 2024-04-23T10:41:08.707 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-23T10:41:08.707 INFO:teuthology.orchestra.run.smithi114.stdout: "log_to_cluster_level": { 2024-04-23T10:41:08.707 INFO:teuthology.orchestra.run.smithi114.stdout: "name": "log_to_cluster_level", 2024-04-23T10:41:08.707 INFO:teuthology.orchestra.run.smithi114.stdout: "type": "str", 2024-04-23T10:41:08.707 INFO:teuthology.orchestra.run.smithi114.stdout: "level": "advanced", 2024-04-23T10:41:08.708 INFO:teuthology.orchestra.run.smithi114.stdout: "flags": 1, 2024-04-23T10:41:08.708 INFO:teuthology.orchestra.run.smithi114.stdout: "default_value": "info", 2024-04-23T10:41:08.708 INFO:teuthology.orchestra.run.smithi114.stdout: "min": "", 2024-04-23T10:41:08.708 INFO:teuthology.orchestra.run.smithi114.stdout: "max": "", 2024-04-23T10:41:08.708 INFO:teuthology.orchestra.run.smithi114.stdout: "enum_allowed": [ 2024-04-23T10:41:08.708 INFO:teuthology.orchestra.run.smithi114.stdout: "", 2024-04-23T10:41:08.708 INFO:teuthology.orchestra.run.smithi114.stdout: "critical", 2024-04-23T10:41:08.708 INFO:teuthology.orchestra.run.smithi114.stdout: "debug", 2024-04-23T10:41:08.708 INFO:teuthology.orchestra.run.smithi114.stdout: "error", 2024-04-23T10:41:08.708 INFO:teuthology.orchestra.run.smithi114.stdout: "info", 2024-04-23T10:41:08.709 INFO:teuthology.orchestra.run.smithi114.stdout: "warning" 2024-04-23T10:41:08.709 INFO:teuthology.orchestra.run.smithi114.stdout: ], 2024-04-23T10:41:08.709 INFO:teuthology.orchestra.run.smithi114.stdout: "desc": "", 2024-04-23T10:41:08.709 INFO:teuthology.orchestra.run.smithi114.stdout: "long_desc": "", 2024-04-23T10:41:08.709 INFO:teuthology.orchestra.run.smithi114.stdout: "tags": [], 2024-04-23T10:41:08.709 INFO:teuthology.orchestra.run.smithi114.stdout: "see_also": [] 2024-04-23T10:41:08.709 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-23T10:41:08.709 INFO:teuthology.orchestra.run.smithi114.stdout: "log_to_file": { 2024-04-23T10:41:08.709 INFO:teuthology.orchestra.run.smithi114.stdout: "name": "log_to_file", 2024-04-23T10:41:08.710 INFO:teuthology.orchestra.run.smithi114.stdout: "type": "bool", 2024-04-23T10:41:08.710 INFO:teuthology.orchestra.run.smithi114.stdout: "level": "advanced", 2024-04-23T10:41:08.710 INFO:teuthology.orchestra.run.smithi114.stdout: "flags": 1, 2024-04-23T10:41:08.710 INFO:teuthology.orchestra.run.smithi114.stdout: "default_value": "False", 2024-04-23T10:41:08.710 INFO:teuthology.orchestra.run.smithi114.stdout: "min": "", 2024-04-23T10:41:08.710 INFO:teuthology.orchestra.run.smithi114.stdout: "max": "", 2024-04-23T10:41:08.710 INFO:teuthology.orchestra.run.smithi114.stdout: "enum_allowed": [], 2024-04-23T10:41:08.710 INFO:teuthology.orchestra.run.smithi114.stdout: "desc": "", 2024-04-23T10:41:08.710 INFO:teuthology.orchestra.run.smithi114.stdout: "long_desc": "", 2024-04-23T10:41:08.710 INFO:teuthology.orchestra.run.smithi114.stdout: "tags": [], 2024-04-23T10:41:08.711 INFO:teuthology.orchestra.run.smithi114.stdout: "see_also": [] 2024-04-23T10:41:08.711 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-23T10:41:08.711 INFO:teuthology.orchestra.run.smithi114.stdout: "min_score": { 2024-04-23T10:41:08.711 INFO:teuthology.orchestra.run.smithi114.stdout: "name": "min_score", 2024-04-23T10:41:08.711 INFO:teuthology.orchestra.run.smithi114.stdout: "type": "float", 2024-04-23T10:41:08.711 INFO:teuthology.orchestra.run.smithi114.stdout: "level": "advanced", 2024-04-23T10:41:08.711 INFO:teuthology.orchestra.run.smithi114.stdout: "flags": 1, 2024-04-23T10:41:08.711 INFO:teuthology.orchestra.run.smithi114.stdout: "default_value": "0", 2024-04-23T10:41:08.711 INFO:teuthology.orchestra.run.smithi114.stdout: "min": "", 2024-04-23T10:41:08.711 INFO:teuthology.orchestra.run.smithi114.stdout: "max": "", 2024-04-23T10:41:08.712 INFO:teuthology.orchestra.run.smithi114.stdout: "enum_allowed": [], 2024-04-23T10:41:08.712 INFO:teuthology.orchestra.run.smithi114.stdout: "desc": "minimum score, below which no optimization is attempted", 2024-04-23T10:41:08.712 INFO:teuthology.orchestra.run.smithi114.stdout: "long_desc": "", 2024-04-23T10:41:08.712 INFO:teuthology.orchestra.run.smithi114.stdout: "tags": [], 2024-04-23T10:41:08.712 INFO:teuthology.orchestra.run.smithi114.stdout: "see_also": [] 2024-04-23T10:41:08.712 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-23T10:41:08.712 INFO:teuthology.orchestra.run.smithi114.stdout: "mode": { 2024-04-23T10:41:08.712 INFO:teuthology.orchestra.run.smithi114.stdout: "name": "mode", 2024-04-23T10:41:08.712 INFO:teuthology.orchestra.run.smithi114.stdout: "type": "str", 2024-04-23T10:41:08.712 INFO:teuthology.orchestra.run.smithi114.stdout: "level": "advanced", 2024-04-23T10:41:08.713 INFO:teuthology.orchestra.run.smithi114.stdout: "flags": 1, 2024-04-23T10:41:08.713 INFO:teuthology.orchestra.run.smithi114.stdout: "default_value": "upmap", 2024-04-23T10:41:08.713 INFO:teuthology.orchestra.run.smithi114.stdout: "min": "", 2024-04-23T10:41:08.713 INFO:teuthology.orchestra.run.smithi114.stdout: "max": "", 2024-04-23T10:41:08.713 INFO:teuthology.orchestra.run.smithi114.stdout: "enum_allowed": [ 2024-04-23T10:41:08.713 INFO:teuthology.orchestra.run.smithi114.stdout: "crush-compat", 2024-04-23T10:41:08.713 INFO:teuthology.orchestra.run.smithi114.stdout: "none", 2024-04-23T10:41:08.713 INFO:teuthology.orchestra.run.smithi114.stdout: "read", 2024-04-23T10:41:08.713 INFO:teuthology.orchestra.run.smithi114.stdout: "upmap", 2024-04-23T10:41:08.713 INFO:teuthology.orchestra.run.smithi114.stdout: "upmap-read" 2024-04-23T10:41:08.714 INFO:teuthology.orchestra.run.smithi114.stdout: ], 2024-04-23T10:41:08.714 INFO:teuthology.orchestra.run.smithi114.stdout: "desc": "Balancer mode", 2024-04-23T10:41:08.714 INFO:teuthology.orchestra.run.smithi114.stdout: "long_desc": "", 2024-04-23T10:41:08.714 INFO:teuthology.orchestra.run.smithi114.stdout: "tags": [], 2024-04-23T10:41:08.714 INFO:teuthology.orchestra.run.smithi114.stdout: "see_also": [] 2024-04-23T10:41:08.714 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-23T10:41:08.714 INFO:teuthology.orchestra.run.smithi114.stdout: "pool_ids": { 2024-04-23T10:41:08.714 INFO:teuthology.orchestra.run.smithi114.stdout: "name": "pool_ids", 2024-04-23T10:41:08.714 INFO:teuthology.orchestra.run.smithi114.stdout: "type": "str", 2024-04-23T10:41:08.714 INFO:teuthology.orchestra.run.smithi114.stdout: "level": "advanced", 2024-04-23T10:41:08.715 INFO:teuthology.orchestra.run.smithi114.stdout: "flags": 1, 2024-04-23T10:41:08.715 INFO:teuthology.orchestra.run.smithi114.stdout: "default_value": "", 2024-04-23T10:41:08.715 INFO:teuthology.orchestra.run.smithi114.stdout: "min": "", 2024-04-23T10:41:08.715 INFO:teuthology.orchestra.run.smithi114.stdout: "max": "", 2024-04-23T10:41:08.715 INFO:teuthology.orchestra.run.smithi114.stdout: "enum_allowed": [], 2024-04-23T10:41:08.715 INFO:teuthology.orchestra.run.smithi114.stdout: "desc": "pools which the automatic balancing will be limited to", 2024-04-23T10:41:08.715 INFO:teuthology.orchestra.run.smithi114.stdout: "long_desc": "", 2024-04-23T10:41:08.715 INFO:teuthology.orchestra.run.smithi114.stdout: "tags": [], 2024-04-23T10:41:08.715 INFO:teuthology.orchestra.run.smithi114.stdout: "see_also": [] 2024-04-23T10:41:08.716 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-23T10:41:08.716 INFO:teuthology.orchestra.run.smithi114.stdout: "sleep_interval": { 2024-04-23T10:41:08.716 INFO:teuthology.orchestra.run.smithi114.stdout: "name": "sleep_interval", 2024-04-23T10:41:08.716 INFO:teuthology.orchestra.run.smithi114.stdout: "type": "secs", 2024-04-23T10:41:08.716 INFO:teuthology.orchestra.run.smithi114.stdout: "level": "advanced", 2024-04-23T10:41:08.716 INFO:teuthology.orchestra.run.smithi114.stdout: "flags": 1, 2024-04-23T10:41:08.716 INFO:teuthology.orchestra.run.smithi114.stdout: "default_value": "60", 2024-04-23T10:41:08.716 INFO:teuthology.orchestra.run.smithi114.stdout: "min": "", 2024-04-23T10:41:08.716 INFO:teuthology.orchestra.run.smithi114.stdout: "max": "", 2024-04-23T10:41:08.716 INFO:teuthology.orchestra.run.smithi114.stdout: "enum_allowed": [], 2024-04-23T10:41:08.717 INFO:teuthology.orchestra.run.smithi114.stdout: "desc": "how frequently to wake up and attempt optimization", 2024-04-23T10:41:08.717 INFO:teuthology.orchestra.run.smithi114.stdout: "long_desc": "", 2024-04-23T10:41:08.717 INFO:teuthology.orchestra.run.smithi114.stdout: "tags": [], 2024-04-23T10:41:08.717 INFO:teuthology.orchestra.run.smithi114.stdout: "see_also": [] 2024-04-23T10:41:08.717 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-23T10:41:08.717 INFO:teuthology.orchestra.run.smithi114.stdout: "upmap_max_deviation": { 2024-04-23T10:41:08.717 INFO:teuthology.orchestra.run.smithi114.stdout: "name": "upmap_max_deviation", 2024-04-23T10:41:08.717 INFO:teuthology.orchestra.run.smithi114.stdout: "type": "int", 2024-04-23T10:41:08.717 INFO:teuthology.orchestra.run.smithi114.stdout: "level": "advanced", 2024-04-23T10:41:08.717 INFO:teuthology.orchestra.run.smithi114.stdout: "flags": 1, 2024-04-23T10:41:08.718 INFO:teuthology.orchestra.run.smithi114.stdout: "default_value": "5", 2024-04-23T10:41:08.718 INFO:teuthology.orchestra.run.smithi114.stdout: "min": "1", 2024-04-23T10:41:08.718 INFO:teuthology.orchestra.run.smithi114.stdout: "max": "", 2024-04-23T10:41:08.718 INFO:teuthology.orchestra.run.smithi114.stdout: "enum_allowed": [], 2024-04-23T10:41:08.718 INFO:teuthology.orchestra.run.smithi114.stdout: "desc": "deviation below which no optimization is attempted", 2024-04-23T10:41:08.718 INFO:teuthology.orchestra.run.smithi114.stdout: "long_desc": "If the number of PGs are within this count then no optimization is attempted", 2024-04-23T10:41:08.718 INFO:teuthology.orchestra.run.smithi114.stdout: "tags": [], 2024-04-23T10:41:08.718 INFO:teuthology.orchestra.run.smithi114.stdout: "see_also": [] 2024-04-23T10:41:08.718 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-23T10:41:08.718 INFO:teuthology.orchestra.run.smithi114.stdout: "upmap_max_optimizations": { 2024-04-23T10:41:08.719 INFO:teuthology.orchestra.run.smithi114.stdout: "name": "upmap_max_optimizations", 2024-04-23T10:41:08.719 INFO:teuthology.orchestra.run.smithi114.stdout: "type": "uint", 2024-04-23T10:41:08.719 INFO:teuthology.orchestra.run.smithi114.stdout: "level": "advanced", 2024-04-23T10:41:08.719 INFO:teuthology.orchestra.run.smithi114.stdout: "flags": 1, 2024-04-23T10:41:08.719 INFO:teuthology.orchestra.run.smithi114.stdout: "default_value": "10", 2024-04-23T10:41:08.719 INFO:teuthology.orchestra.run.smithi114.stdout: "min": "", 2024-04-23T10:41:08.719 INFO:teuthology.orchestra.run.smithi114.stdout: "max": "", 2024-04-23T10:41:08.719 INFO:teuthology.orchestra.run.smithi114.stdout: "enum_allowed": [], 2024-04-23T10:41:08.719 INFO:teuthology.orchestra.run.smithi114.stdout: "desc": "maximum upmap optimizations to make per attempt", 2024-04-23T10:41:08.719 INFO:teuthology.orchestra.run.smithi114.stdout: "long_desc": "", 2024-04-23T10:41:08.720 INFO:teuthology.orchestra.run.smithi114.stdout: "tags": [], 2024-04-23T10:41:08.720 INFO:teuthology.orchestra.run.smithi114.stdout: "see_also": [] 2024-04-23T10:41:08.720 INFO:teuthology.orchestra.run.smithi114.stdout: } 2024-04-23T10:41:08.720 INFO:teuthology.orchestra.run.smithi114.stdout: } 2024-04-23T10:41:08.720 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-23T10:41:08.720 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-04-23T10:41:08.720 INFO:teuthology.orchestra.run.smithi114.stdout: "name": "cephadm", 2024-04-23T10:41:08.720 INFO:teuthology.orchestra.run.smithi114.stdout: "can_run": true, 2024-04-23T10:41:08.720 INFO:teuthology.orchestra.run.smithi114.stdout: "error_string": "", 2024-04-23T10:41:08.720 INFO:teuthology.orchestra.run.smithi114.stdout: "module_options": { 2024-04-23T10:41:08.721 INFO:teuthology.orchestra.run.smithi114.stdout: "agent_down_multiplier": { 2024-04-23T10:41:08.721 INFO:teuthology.orchestra.run.smithi114.stdout: "name": "agent_down_multiplier", 2024-04-23T10:41:08.721 INFO:teuthology.orchestra.run.smithi114.stdout: "type": "float", 2024-04-23T10:41:08.721 INFO:teuthology.orchestra.run.smithi114.stdout: "level": "advanced", 2024-04-23T10:41:08.721 INFO:teuthology.orchestra.run.smithi114.stdout: "flags": 0, 2024-04-23T10:41:08.721 INFO:teuthology.orchestra.run.smithi114.stdout: "default_value": "3.0", 2024-04-23T10:41:08.721 INFO:teuthology.orchestra.run.smithi114.stdout: "min": "", 2024-04-23T10:41:08.721 INFO:teuthology.orchestra.run.smithi114.stdout: "max": "", 2024-04-23T10:41:08.721 INFO:teuthology.orchestra.run.smithi114.stdout: "enum_allowed": [], 2024-04-23T10:41:08.721 INFO:teuthology.orchestra.run.smithi114.stdout: "desc": "Multiplied by agent refresh rate to calculate how long agent must not report before being marked down", 2024-04-23T10:41:08.722 INFO:teuthology.orchestra.run.smithi114.stdout: "long_desc": "", 2024-04-23T10:41:08.722 INFO:teuthology.orchestra.run.smithi114.stdout: "tags": [], 2024-04-23T10:41:08.722 INFO:teuthology.orchestra.run.smithi114.stdout: "see_also": [] 2024-04-23T10:41:08.722 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-23T10:41:08.722 INFO:teuthology.orchestra.run.smithi114.stdout: "agent_refresh_rate": { 2024-04-23T10:41:08.722 INFO:teuthology.orchestra.run.smithi114.stdout: "name": "agent_refresh_rate", 2024-04-23T10:41:08.722 INFO:teuthology.orchestra.run.smithi114.stdout: "type": "secs", 2024-04-23T10:41:08.722 INFO:teuthology.orchestra.run.smithi114.stdout: "level": "advanced", 2024-04-23T10:41:08.722 INFO:teuthology.orchestra.run.smithi114.stdout: "flags": 0, 2024-04-23T10:41:08.723 INFO:teuthology.orchestra.run.smithi114.stdout: "default_value": "20", 2024-04-23T10:41:08.723 INFO:teuthology.orchestra.run.smithi114.stdout: "min": "", 2024-04-23T10:41:08.723 INFO:teuthology.orchestra.run.smithi114.stdout: "max": "", 2024-04-23T10:41:08.723 INFO:teuthology.orchestra.run.smithi114.stdout: "enum_allowed": [], 2024-04-23T10:41:08.723 INFO:teuthology.orchestra.run.smithi114.stdout: "desc": "How often agent on each host will try to gather and send metadata", 2024-04-23T10:41:08.723 INFO:teuthology.orchestra.run.smithi114.stdout: "long_desc": "", 2024-04-23T10:41:08.723 INFO:teuthology.orchestra.run.smithi114.stdout: "tags": [], 2024-04-23T10:41:08.723 INFO:teuthology.orchestra.run.smithi114.stdout: "see_also": [] 2024-04-23T10:41:08.723 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-23T10:41:08.723 INFO:teuthology.orchestra.run.smithi114.stdout: "agent_starting_port": { 2024-04-23T10:41:08.724 INFO:teuthology.orchestra.run.smithi114.stdout: "name": "agent_starting_port", 2024-04-23T10:41:08.724 INFO:teuthology.orchestra.run.smithi114.stdout: "type": "int", 2024-04-23T10:41:08.724 INFO:teuthology.orchestra.run.smithi114.stdout: "level": "advanced", 2024-04-23T10:41:08.724 INFO:teuthology.orchestra.run.smithi114.stdout: "flags": 0, 2024-04-23T10:41:08.724 INFO:teuthology.orchestra.run.smithi114.stdout: "default_value": "4721", 2024-04-23T10:41:08.724 INFO:teuthology.orchestra.run.smithi114.stdout: "min": "", 2024-04-23T10:41:08.724 INFO:teuthology.orchestra.run.smithi114.stdout: "max": "", 2024-04-23T10:41:08.724 INFO:teuthology.orchestra.run.smithi114.stdout: "enum_allowed": [], 2024-04-23T10:41:08.724 INFO:teuthology.orchestra.run.smithi114.stdout: "desc": "First port agent will try to bind to (will also try up to next 1000 subsequent ports if blocked)", 2024-04-23T10:41:08.725 INFO:teuthology.orchestra.run.smithi114.stdout: "long_desc": "", 2024-04-23T10:41:08.725 INFO:teuthology.orchestra.run.smithi114.stdout: "tags": [], 2024-04-23T10:41:08.725 INFO:teuthology.orchestra.run.smithi114.stdout: "see_also": [] 2024-04-23T10:41:08.725 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-23T10:41:08.725 INFO:teuthology.orchestra.run.smithi114.stdout: "allow_ptrace": { 2024-04-23T10:41:08.725 INFO:teuthology.orchestra.run.smithi114.stdout: "name": "allow_ptrace", 2024-04-23T10:41:08.725 INFO:teuthology.orchestra.run.smithi114.stdout: "type": "bool", 2024-04-23T10:41:08.725 INFO:teuthology.orchestra.run.smithi114.stdout: "level": "advanced", 2024-04-23T10:41:08.725 INFO:teuthology.orchestra.run.smithi114.stdout: "flags": 0, 2024-04-23T10:41:08.725 INFO:teuthology.orchestra.run.smithi114.stdout: "default_value": "False", 2024-04-23T10:41:08.726 INFO:teuthology.orchestra.run.smithi114.stdout: "min": "", 2024-04-23T10:41:08.726 INFO:teuthology.orchestra.run.smithi114.stdout: "max": "", 2024-04-23T10:41:08.726 INFO:teuthology.orchestra.run.smithi114.stdout: "enum_allowed": [], 2024-04-23T10:41:08.726 INFO:teuthology.orchestra.run.smithi114.stdout: "desc": "allow SYS_PTRACE capability on ceph containers", 2024-04-23T10:41:08.726 INFO:teuthology.orchestra.run.smithi114.stdout: "long_desc": "The SYS_PTRACE capability is needed to attach to a process with gdb or strace. Enabling this options can allow debugging daemons that encounter problems at runtime.", 2024-04-23T10:41:08.726 INFO:teuthology.orchestra.run.smithi114.stdout: "tags": [], 2024-04-23T10:41:08.726 INFO:teuthology.orchestra.run.smithi114.stdout: "see_also": [] 2024-04-23T10:41:08.726 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-23T10:41:08.726 INFO:teuthology.orchestra.run.smithi114.stdout: "autotune_interval": { 2024-04-23T10:41:08.726 INFO:teuthology.orchestra.run.smithi114.stdout: "name": "autotune_interval", 2024-04-23T10:41:08.727 INFO:teuthology.orchestra.run.smithi114.stdout: "type": "secs", 2024-04-23T10:41:08.727 INFO:teuthology.orchestra.run.smithi114.stdout: "level": "advanced", 2024-04-23T10:41:08.727 INFO:teuthology.orchestra.run.smithi114.stdout: "flags": 0, 2024-04-23T10:41:08.727 INFO:teuthology.orchestra.run.smithi114.stdout: "default_value": "600", 2024-04-23T10:41:08.727 INFO:teuthology.orchestra.run.smithi114.stdout: "min": "", 2024-04-23T10:41:08.727 INFO:teuthology.orchestra.run.smithi114.stdout: "max": "", 2024-04-23T10:41:08.727 INFO:teuthology.orchestra.run.smithi114.stdout: "enum_allowed": [], 2024-04-23T10:41:08.727 INFO:teuthology.orchestra.run.smithi114.stdout: "desc": "how frequently to autotune daemon memory", 2024-04-23T10:41:08.727 INFO:teuthology.orchestra.run.smithi114.stdout: "long_desc": "", 2024-04-23T10:41:08.727 INFO:teuthology.orchestra.run.smithi114.stdout: "tags": [], 2024-04-23T10:41:08.728 INFO:teuthology.orchestra.run.smithi114.stdout: "see_also": [] 2024-04-23T10:41:08.728 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-23T10:41:08.728 INFO:teuthology.orchestra.run.smithi114.stdout: "autotune_memory_target_ratio": { 2024-04-23T10:41:08.728 INFO:teuthology.orchestra.run.smithi114.stdout: "name": "autotune_memory_target_ratio", 2024-04-23T10:41:08.728 INFO:teuthology.orchestra.run.smithi114.stdout: "type": "float", 2024-04-23T10:41:08.728 INFO:teuthology.orchestra.run.smithi114.stdout: "level": "advanced", 2024-04-23T10:41:08.728 INFO:teuthology.orchestra.run.smithi114.stdout: "flags": 0, 2024-04-23T10:41:08.728 INFO:teuthology.orchestra.run.smithi114.stdout: "default_value": "0.7", 2024-04-23T10:41:08.728 INFO:teuthology.orchestra.run.smithi114.stdout: "min": "", 2024-04-23T10:41:08.729 INFO:teuthology.orchestra.run.smithi114.stdout: "max": "", 2024-04-23T10:41:08.729 INFO:teuthology.orchestra.run.smithi114.stdout: "enum_allowed": [], 2024-04-23T10:41:08.729 INFO:teuthology.orchestra.run.smithi114.stdout: "desc": "ratio of total system memory to divide amongst autotuned daemons", 2024-04-23T10:41:08.729 INFO:teuthology.orchestra.run.smithi114.stdout: "long_desc": "", 2024-04-23T10:41:08.729 INFO:teuthology.orchestra.run.smithi114.stdout: "tags": [], 2024-04-23T10:41:08.729 INFO:teuthology.orchestra.run.smithi114.stdout: "see_also": [] 2024-04-23T10:41:08.729 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-23T10:41:08.729 INFO:teuthology.orchestra.run.smithi114.stdout: "cephadm_log_destination": { 2024-04-23T10:41:08.729 INFO:teuthology.orchestra.run.smithi114.stdout: "name": "cephadm_log_destination", 2024-04-23T10:41:08.729 INFO:teuthology.orchestra.run.smithi114.stdout: "type": "str", 2024-04-23T10:41:08.730 INFO:teuthology.orchestra.run.smithi114.stdout: "level": "advanced", 2024-04-23T10:41:08.730 INFO:teuthology.orchestra.run.smithi114.stdout: "flags": 0, 2024-04-23T10:41:08.730 INFO:teuthology.orchestra.run.smithi114.stdout: "default_value": "", 2024-04-23T10:41:08.730 INFO:teuthology.orchestra.run.smithi114.stdout: "min": "", 2024-04-23T10:41:08.730 INFO:teuthology.orchestra.run.smithi114.stdout: "max": "", 2024-04-23T10:41:08.730 INFO:teuthology.orchestra.run.smithi114.stdout: "enum_allowed": [ 2024-04-23T10:41:08.730 INFO:teuthology.orchestra.run.smithi114.stdout: "file", 2024-04-23T10:41:08.730 INFO:teuthology.orchestra.run.smithi114.stdout: "file,syslog", 2024-04-23T10:41:08.730 INFO:teuthology.orchestra.run.smithi114.stdout: "syslog" 2024-04-23T10:41:08.730 INFO:teuthology.orchestra.run.smithi114.stdout: ], 2024-04-23T10:41:08.731 INFO:teuthology.orchestra.run.smithi114.stdout: "desc": "Destination for cephadm command's persistent logging", 2024-04-23T10:41:08.731 INFO:teuthology.orchestra.run.smithi114.stdout: "long_desc": "", 2024-04-23T10:41:08.731 INFO:teuthology.orchestra.run.smithi114.stdout: "tags": [], 2024-04-23T10:41:08.731 INFO:teuthology.orchestra.run.smithi114.stdout: "see_also": [] 2024-04-23T10:41:08.731 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-23T10:41:08.731 INFO:teuthology.orchestra.run.smithi114.stdout: "cgroups_split": { 2024-04-23T10:41:08.731 INFO:teuthology.orchestra.run.smithi114.stdout: "name": "cgroups_split", 2024-04-23T10:41:08.731 INFO:teuthology.orchestra.run.smithi114.stdout: "type": "bool", 2024-04-23T10:41:08.731 INFO:teuthology.orchestra.run.smithi114.stdout: "level": "advanced", 2024-04-23T10:41:08.731 INFO:teuthology.orchestra.run.smithi114.stdout: "flags": 0, 2024-04-23T10:41:08.732 INFO:teuthology.orchestra.run.smithi114.stdout: "default_value": "True", 2024-04-23T10:41:08.732 INFO:teuthology.orchestra.run.smithi114.stdout: "min": "", 2024-04-23T10:41:08.732 INFO:teuthology.orchestra.run.smithi114.stdout: "max": "", 2024-04-23T10:41:08.732 INFO:teuthology.orchestra.run.smithi114.stdout: "enum_allowed": [], 2024-04-23T10:41:08.732 INFO:teuthology.orchestra.run.smithi114.stdout: "desc": "Pass --cgroups=split when cephadm creates containers (currently podman only)", 2024-04-23T10:41:08.732 INFO:teuthology.orchestra.run.smithi114.stdout: "long_desc": "", 2024-04-23T10:41:08.732 INFO:teuthology.orchestra.run.smithi114.stdout: "tags": [], 2024-04-23T10:41:08.732 INFO:teuthology.orchestra.run.smithi114.stdout: "see_also": [] 2024-04-23T10:41:08.732 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-23T10:41:08.732 INFO:teuthology.orchestra.run.smithi114.stdout: "config_checks_enabled": { 2024-04-23T10:41:08.733 INFO:teuthology.orchestra.run.smithi114.stdout: "name": "config_checks_enabled", 2024-04-23T10:41:08.733 INFO:teuthology.orchestra.run.smithi114.stdout: "type": "bool", 2024-04-23T10:41:08.733 INFO:teuthology.orchestra.run.smithi114.stdout: "level": "advanced", 2024-04-23T10:41:08.733 INFO:teuthology.orchestra.run.smithi114.stdout: "flags": 0, 2024-04-23T10:41:08.733 INFO:teuthology.orchestra.run.smithi114.stdout: "default_value": "False", 2024-04-23T10:41:08.733 INFO:teuthology.orchestra.run.smithi114.stdout: "min": "", 2024-04-23T10:41:08.733 INFO:teuthology.orchestra.run.smithi114.stdout: "max": "", 2024-04-23T10:41:08.733 INFO:teuthology.orchestra.run.smithi114.stdout: "enum_allowed": [], 2024-04-23T10:41:08.733 INFO:teuthology.orchestra.run.smithi114.stdout: "desc": "Enable or disable the cephadm configuration analysis", 2024-04-23T10:41:08.733 INFO:teuthology.orchestra.run.smithi114.stdout: "long_desc": "", 2024-04-23T10:41:08.734 INFO:teuthology.orchestra.run.smithi114.stdout: "tags": [], 2024-04-23T10:41:08.734 INFO:teuthology.orchestra.run.smithi114.stdout: "see_also": [] 2024-04-23T10:41:08.734 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-23T10:41:08.734 INFO:teuthology.orchestra.run.smithi114.stdout: "config_dashboard": { 2024-04-23T10:41:08.734 INFO:teuthology.orchestra.run.smithi114.stdout: "name": "config_dashboard", 2024-04-23T10:41:08.734 INFO:teuthology.orchestra.run.smithi114.stdout: "type": "bool", 2024-04-23T10:41:08.734 INFO:teuthology.orchestra.run.smithi114.stdout: "level": "advanced", 2024-04-23T10:41:08.734 INFO:teuthology.orchestra.run.smithi114.stdout: "flags": 0, 2024-04-23T10:41:08.734 INFO:teuthology.orchestra.run.smithi114.stdout: "default_value": "True", 2024-04-23T10:41:08.735 INFO:teuthology.orchestra.run.smithi114.stdout: "min": "", 2024-04-23T10:41:08.735 INFO:teuthology.orchestra.run.smithi114.stdout: "max": "", 2024-04-23T10:41:08.735 INFO:teuthology.orchestra.run.smithi114.stdout: "enum_allowed": [], 2024-04-23T10:41:08.735 INFO:teuthology.orchestra.run.smithi114.stdout: "desc": "manage configs like API endpoints in Dashboard.", 2024-04-23T10:41:08.735 INFO:teuthology.orchestra.run.smithi114.stdout: "long_desc": "", 2024-04-23T10:41:08.735 INFO:teuthology.orchestra.run.smithi114.stdout: "tags": [], 2024-04-23T10:41:08.735 INFO:teuthology.orchestra.run.smithi114.stdout: "see_also": [] 2024-04-23T10:41:08.735 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-23T10:41:08.735 INFO:teuthology.orchestra.run.smithi114.stdout: "container_image_alertmanager": { 2024-04-23T10:41:08.735 INFO:teuthology.orchestra.run.smithi114.stdout: "name": "container_image_alertmanager", 2024-04-23T10:41:08.736 INFO:teuthology.orchestra.run.smithi114.stdout: "type": "str", 2024-04-23T10:41:08.736 INFO:teuthology.orchestra.run.smithi114.stdout: "level": "advanced", 2024-04-23T10:41:08.736 INFO:teuthology.orchestra.run.smithi114.stdout: "flags": 0, 2024-04-23T10:41:08.736 INFO:teuthology.orchestra.run.smithi114.stdout: "default_value": "quay.io/prometheus/alertmanager:v0.25.0", 2024-04-23T10:41:08.736 INFO:teuthology.orchestra.run.smithi114.stdout: "min": "", 2024-04-23T10:41:08.736 INFO:teuthology.orchestra.run.smithi114.stdout: "max": "", 2024-04-23T10:41:08.736 INFO:teuthology.orchestra.run.smithi114.stdout: "enum_allowed": [], 2024-04-23T10:41:08.736 INFO:teuthology.orchestra.run.smithi114.stdout: "desc": "Prometheus container image", 2024-04-23T10:41:08.736 INFO:teuthology.orchestra.run.smithi114.stdout: "long_desc": "", 2024-04-23T10:41:08.736 INFO:teuthology.orchestra.run.smithi114.stdout: "tags": [], 2024-04-23T10:41:08.737 INFO:teuthology.orchestra.run.smithi114.stdout: "see_also": [] 2024-04-23T10:41:08.737 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-23T10:41:08.737 INFO:teuthology.orchestra.run.smithi114.stdout: "container_image_base": { 2024-04-23T10:41:08.737 INFO:teuthology.orchestra.run.smithi114.stdout: "name": "container_image_base", 2024-04-23T10:41:08.737 INFO:teuthology.orchestra.run.smithi114.stdout: "type": "str", 2024-04-23T10:41:08.737 INFO:teuthology.orchestra.run.smithi114.stdout: "level": "advanced", 2024-04-23T10:41:08.737 INFO:teuthology.orchestra.run.smithi114.stdout: "flags": 1, 2024-04-23T10:41:08.737 INFO:teuthology.orchestra.run.smithi114.stdout: "default_value": "quay.io/ceph/ceph", 2024-04-23T10:41:08.737 INFO:teuthology.orchestra.run.smithi114.stdout: "min": "", 2024-04-23T10:41:08.737 INFO:teuthology.orchestra.run.smithi114.stdout: "max": "", 2024-04-23T10:41:08.738 INFO:teuthology.orchestra.run.smithi114.stdout: "enum_allowed": [], 2024-04-23T10:41:08.738 INFO:teuthology.orchestra.run.smithi114.stdout: "desc": "Container image name, without the tag", 2024-04-23T10:41:08.738 INFO:teuthology.orchestra.run.smithi114.stdout: "long_desc": "", 2024-04-23T10:41:08.738 INFO:teuthology.orchestra.run.smithi114.stdout: "tags": [], 2024-04-23T10:41:08.738 INFO:teuthology.orchestra.run.smithi114.stdout: "see_also": [] 2024-04-23T10:41:08.738 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-23T10:41:08.738 INFO:teuthology.orchestra.run.smithi114.stdout: "container_image_elasticsearch": { 2024-04-23T10:41:08.738 INFO:teuthology.orchestra.run.smithi114.stdout: "name": "container_image_elasticsearch", 2024-04-23T10:41:08.738 INFO:teuthology.orchestra.run.smithi114.stdout: "type": "str", 2024-04-23T10:41:08.738 INFO:teuthology.orchestra.run.smithi114.stdout: "level": "advanced", 2024-04-23T10:41:08.739 INFO:teuthology.orchestra.run.smithi114.stdout: "flags": 0, 2024-04-23T10:41:08.739 INFO:teuthology.orchestra.run.smithi114.stdout: "default_value": "quay.io/omrizeneva/elasticsearch:6.8.23", 2024-04-23T10:41:08.739 INFO:teuthology.orchestra.run.smithi114.stdout: "min": "", 2024-04-23T10:41:08.739 INFO:teuthology.orchestra.run.smithi114.stdout: "max": "", 2024-04-23T10:41:08.739 INFO:teuthology.orchestra.run.smithi114.stdout: "enum_allowed": [], 2024-04-23T10:41:08.739 INFO:teuthology.orchestra.run.smithi114.stdout: "desc": "elasticsearch container image", 2024-04-23T10:41:08.739 INFO:teuthology.orchestra.run.smithi114.stdout: "long_desc": "", 2024-04-23T10:41:08.739 INFO:teuthology.orchestra.run.smithi114.stdout: "tags": [], 2024-04-23T10:41:08.739 INFO:teuthology.orchestra.run.smithi114.stdout: "see_also": [] 2024-04-23T10:41:08.740 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-23T10:41:08.740 INFO:teuthology.orchestra.run.smithi114.stdout: "container_image_grafana": { 2024-04-23T10:41:08.740 INFO:teuthology.orchestra.run.smithi114.stdout: "name": "container_image_grafana", 2024-04-23T10:41:08.740 INFO:teuthology.orchestra.run.smithi114.stdout: "type": "str", 2024-04-23T10:41:08.740 INFO:teuthology.orchestra.run.smithi114.stdout: "level": "advanced", 2024-04-23T10:41:08.740 INFO:teuthology.orchestra.run.smithi114.stdout: "flags": 0, 2024-04-23T10:41:08.740 INFO:teuthology.orchestra.run.smithi114.stdout: "default_value": "quay.io/ceph/grafana:9.4.12", 2024-04-23T10:41:08.740 INFO:teuthology.orchestra.run.smithi114.stdout: "min": "", 2024-04-23T10:41:08.740 INFO:teuthology.orchestra.run.smithi114.stdout: "max": "", 2024-04-23T10:41:08.740 INFO:teuthology.orchestra.run.smithi114.stdout: "enum_allowed": [], 2024-04-23T10:41:08.741 INFO:teuthology.orchestra.run.smithi114.stdout: "desc": "Prometheus container image", 2024-04-23T10:41:08.741 INFO:teuthology.orchestra.run.smithi114.stdout: "long_desc": "", 2024-04-23T10:41:08.741 INFO:teuthology.orchestra.run.smithi114.stdout: "tags": [], 2024-04-23T10:41:08.741 INFO:teuthology.orchestra.run.smithi114.stdout: "see_also": [] 2024-04-23T10:41:08.741 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-23T10:41:08.741 INFO:teuthology.orchestra.run.smithi114.stdout: "container_image_haproxy": { 2024-04-23T10:41:08.741 INFO:teuthology.orchestra.run.smithi114.stdout: "name": "container_image_haproxy", 2024-04-23T10:41:08.741 INFO:teuthology.orchestra.run.smithi114.stdout: "type": "str", 2024-04-23T10:41:08.741 INFO:teuthology.orchestra.run.smithi114.stdout: "level": "advanced", 2024-04-23T10:41:08.742 INFO:teuthology.orchestra.run.smithi114.stdout: "flags": 0, 2024-04-23T10:41:08.742 INFO:teuthology.orchestra.run.smithi114.stdout: "default_value": "quay.io/ceph/haproxy:2.3", 2024-04-23T10:41:08.742 INFO:teuthology.orchestra.run.smithi114.stdout: "min": "", 2024-04-23T10:41:08.742 INFO:teuthology.orchestra.run.smithi114.stdout: "max": "", 2024-04-23T10:41:08.742 INFO:teuthology.orchestra.run.smithi114.stdout: "enum_allowed": [], 2024-04-23T10:41:08.742 INFO:teuthology.orchestra.run.smithi114.stdout: "desc": "HAproxy container image", 2024-04-23T10:41:08.742 INFO:teuthology.orchestra.run.smithi114.stdout: "long_desc": "", 2024-04-23T10:41:08.742 INFO:teuthology.orchestra.run.smithi114.stdout: "tags": [], 2024-04-23T10:41:08.742 INFO:teuthology.orchestra.run.smithi114.stdout: "see_also": [] 2024-04-23T10:41:08.742 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-23T10:41:08.743 INFO:teuthology.orchestra.run.smithi114.stdout: "container_image_jaeger_agent": { 2024-04-23T10:41:08.743 INFO:teuthology.orchestra.run.smithi114.stdout: "name": "container_image_jaeger_agent", 2024-04-23T10:41:08.743 INFO:teuthology.orchestra.run.smithi114.stdout: "type": "str", 2024-04-23T10:41:08.743 INFO:teuthology.orchestra.run.smithi114.stdout: "level": "advanced", 2024-04-23T10:41:08.743 INFO:teuthology.orchestra.run.smithi114.stdout: "flags": 0, 2024-04-23T10:41:08.743 INFO:teuthology.orchestra.run.smithi114.stdout: "default_value": "quay.io/jaegertracing/jaeger-agent:1.29", 2024-04-23T10:41:08.743 INFO:teuthology.orchestra.run.smithi114.stdout: "min": "", 2024-04-23T10:41:08.743 INFO:teuthology.orchestra.run.smithi114.stdout: "max": "", 2024-04-23T10:41:08.743 INFO:teuthology.orchestra.run.smithi114.stdout: "enum_allowed": [], 2024-04-23T10:41:08.744 INFO:teuthology.orchestra.run.smithi114.stdout: "desc": "Jaeger agent container image", 2024-04-23T10:41:08.744 INFO:teuthology.orchestra.run.smithi114.stdout: "long_desc": "", 2024-04-23T10:41:08.744 INFO:teuthology.orchestra.run.smithi114.stdout: "tags": [], 2024-04-23T10:41:08.744 INFO:teuthology.orchestra.run.smithi114.stdout: "see_also": [] 2024-04-23T10:41:08.744 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-23T10:41:08.744 INFO:teuthology.orchestra.run.smithi114.stdout: "container_image_jaeger_collector": { 2024-04-23T10:41:08.744 INFO:teuthology.orchestra.run.smithi114.stdout: "name": "container_image_jaeger_collector", 2024-04-23T10:41:08.744 INFO:teuthology.orchestra.run.smithi114.stdout: "type": "str", 2024-04-23T10:41:08.744 INFO:teuthology.orchestra.run.smithi114.stdout: "level": "advanced", 2024-04-23T10:41:08.745 INFO:teuthology.orchestra.run.smithi114.stdout: "flags": 0, 2024-04-23T10:41:08.745 INFO:teuthology.orchestra.run.smithi114.stdout: "default_value": "quay.io/jaegertracing/jaeger-collector:1.29", 2024-04-23T10:41:08.745 INFO:teuthology.orchestra.run.smithi114.stdout: "min": "", 2024-04-23T10:41:08.745 INFO:teuthology.orchestra.run.smithi114.stdout: "max": "", 2024-04-23T10:41:08.745 INFO:teuthology.orchestra.run.smithi114.stdout: "enum_allowed": [], 2024-04-23T10:41:08.745 INFO:teuthology.orchestra.run.smithi114.stdout: "desc": "Jaeger collector container image", 2024-04-23T10:41:08.745 INFO:teuthology.orchestra.run.smithi114.stdout: "long_desc": "", 2024-04-23T10:41:08.745 INFO:teuthology.orchestra.run.smithi114.stdout: "tags": [], 2024-04-23T10:41:08.745 INFO:teuthology.orchestra.run.smithi114.stdout: "see_also": [] 2024-04-23T10:41:08.746 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-23T10:41:08.746 INFO:teuthology.orchestra.run.smithi114.stdout: "container_image_jaeger_query": { 2024-04-23T10:41:08.746 INFO:teuthology.orchestra.run.smithi114.stdout: "name": "container_image_jaeger_query", 2024-04-23T10:41:08.746 INFO:teuthology.orchestra.run.smithi114.stdout: "type": "str", 2024-04-23T10:41:08.746 INFO:teuthology.orchestra.run.smithi114.stdout: "level": "advanced", 2024-04-23T10:41:08.746 INFO:teuthology.orchestra.run.smithi114.stdout: "flags": 0, 2024-04-23T10:41:08.746 INFO:teuthology.orchestra.run.smithi114.stdout: "default_value": "quay.io/jaegertracing/jaeger-query:1.29", 2024-04-23T10:41:08.746 INFO:teuthology.orchestra.run.smithi114.stdout: "min": "", 2024-04-23T10:41:08.746 INFO:teuthology.orchestra.run.smithi114.stdout: "max": "", 2024-04-23T10:41:08.746 INFO:teuthology.orchestra.run.smithi114.stdout: "enum_allowed": [], 2024-04-23T10:41:08.747 INFO:teuthology.orchestra.run.smithi114.stdout: "desc": "Jaeger query container image", 2024-04-23T10:41:08.747 INFO:teuthology.orchestra.run.smithi114.stdout: "long_desc": "", 2024-04-23T10:41:08.747 INFO:teuthology.orchestra.run.smithi114.stdout: "tags": [], 2024-04-23T10:41:08.747 INFO:teuthology.orchestra.run.smithi114.stdout: "see_also": [] 2024-04-23T10:41:08.747 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-23T10:41:08.747 INFO:teuthology.orchestra.run.smithi114.stdout: "container_image_keepalived": { 2024-04-23T10:41:08.747 INFO:teuthology.orchestra.run.smithi114.stdout: "name": "container_image_keepalived", 2024-04-23T10:41:08.747 INFO:teuthology.orchestra.run.smithi114.stdout: "type": "str", 2024-04-23T10:41:08.747 INFO:teuthology.orchestra.run.smithi114.stdout: "level": "advanced", 2024-04-23T10:41:08.747 INFO:teuthology.orchestra.run.smithi114.stdout: "flags": 0, 2024-04-23T10:41:08.748 INFO:teuthology.orchestra.run.smithi114.stdout: "default_value": "quay.io/ceph/keepalived:2.2.4", 2024-04-23T10:41:08.748 INFO:teuthology.orchestra.run.smithi114.stdout: "min": "", 2024-04-23T10:41:08.748 INFO:teuthology.orchestra.run.smithi114.stdout: "max": "", 2024-04-23T10:41:08.748 INFO:teuthology.orchestra.run.smithi114.stdout: "enum_allowed": [], 2024-04-23T10:41:08.748 INFO:teuthology.orchestra.run.smithi114.stdout: "desc": "Keepalived container image", 2024-04-23T10:41:08.748 INFO:teuthology.orchestra.run.smithi114.stdout: "long_desc": "", 2024-04-23T10:41:08.748 INFO:teuthology.orchestra.run.smithi114.stdout: "tags": [], 2024-04-23T10:41:08.748 INFO:teuthology.orchestra.run.smithi114.stdout: "see_also": [] 2024-04-23T10:41:08.748 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-23T10:41:08.748 INFO:teuthology.orchestra.run.smithi114.stdout: "container_image_loki": { 2024-04-23T10:41:08.749 INFO:teuthology.orchestra.run.smithi114.stdout: "name": "container_image_loki", 2024-04-23T10:41:08.749 INFO:teuthology.orchestra.run.smithi114.stdout: "type": "str", 2024-04-23T10:41:08.749 INFO:teuthology.orchestra.run.smithi114.stdout: "level": "advanced", 2024-04-23T10:41:08.749 INFO:teuthology.orchestra.run.smithi114.stdout: "flags": 0, 2024-04-23T10:41:08.749 INFO:teuthology.orchestra.run.smithi114.stdout: "default_value": "docker.io/grafana/loki:2.4.0", 2024-04-23T10:41:08.749 INFO:teuthology.orchestra.run.smithi114.stdout: "min": "", 2024-04-23T10:41:08.749 INFO:teuthology.orchestra.run.smithi114.stdout: "max": "", 2024-04-23T10:41:08.749 INFO:teuthology.orchestra.run.smithi114.stdout: "enum_allowed": [], 2024-04-23T10:41:08.749 INFO:teuthology.orchestra.run.smithi114.stdout: "desc": "Loki container image", 2024-04-23T10:41:08.749 INFO:teuthology.orchestra.run.smithi114.stdout: "long_desc": "", 2024-04-23T10:41:08.750 INFO:teuthology.orchestra.run.smithi114.stdout: "tags": [], 2024-04-23T10:41:08.750 INFO:teuthology.orchestra.run.smithi114.stdout: "see_also": [] 2024-04-23T10:41:08.750 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-23T10:41:08.750 INFO:teuthology.orchestra.run.smithi114.stdout: "container_image_node_exporter": { 2024-04-23T10:41:08.750 INFO:teuthology.orchestra.run.smithi114.stdout: "name": "container_image_node_exporter", 2024-04-23T10:41:08.750 INFO:teuthology.orchestra.run.smithi114.stdout: "type": "str", 2024-04-23T10:41:08.750 INFO:teuthology.orchestra.run.smithi114.stdout: "level": "advanced", 2024-04-23T10:41:08.750 INFO:teuthology.orchestra.run.smithi114.stdout: "flags": 0, 2024-04-23T10:41:08.750 INFO:teuthology.orchestra.run.smithi114.stdout: "default_value": "quay.io/prometheus/node-exporter:v1.5.0", 2024-04-23T10:41:08.751 INFO:teuthology.orchestra.run.smithi114.stdout: "min": "", 2024-04-23T10:41:08.751 INFO:teuthology.orchestra.run.smithi114.stdout: "max": "", 2024-04-23T10:41:08.751 INFO:teuthology.orchestra.run.smithi114.stdout: "enum_allowed": [], 2024-04-23T10:41:08.751 INFO:teuthology.orchestra.run.smithi114.stdout: "desc": "Prometheus container image", 2024-04-23T10:41:08.751 INFO:teuthology.orchestra.run.smithi114.stdout: "long_desc": "", 2024-04-23T10:41:08.751 INFO:teuthology.orchestra.run.smithi114.stdout: "tags": [], 2024-04-23T10:41:08.751 INFO:teuthology.orchestra.run.smithi114.stdout: "see_also": [] 2024-04-23T10:41:08.751 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-23T10:41:08.751 INFO:teuthology.orchestra.run.smithi114.stdout: "container_image_nvmeof": { 2024-04-23T10:41:08.751 INFO:teuthology.orchestra.run.smithi114.stdout: "name": "container_image_nvmeof", 2024-04-23T10:41:08.752 INFO:teuthology.orchestra.run.smithi114.stdout: "type": "str", 2024-04-23T10:41:08.752 INFO:teuthology.orchestra.run.smithi114.stdout: "level": "advanced", 2024-04-23T10:41:08.752 INFO:teuthology.orchestra.run.smithi114.stdout: "flags": 0, 2024-04-23T10:41:08.752 INFO:teuthology.orchestra.run.smithi114.stdout: "default_value": "quay.io/ceph/nvmeof:1.2.1", 2024-04-23T10:41:08.752 INFO:teuthology.orchestra.run.smithi114.stdout: "min": "", 2024-04-23T10:41:08.752 INFO:teuthology.orchestra.run.smithi114.stdout: "max": "", 2024-04-23T10:41:08.752 INFO:teuthology.orchestra.run.smithi114.stdout: "enum_allowed": [], 2024-04-23T10:41:08.752 INFO:teuthology.orchestra.run.smithi114.stdout: "desc": "Nvme-of container image", 2024-04-23T10:41:08.752 INFO:teuthology.orchestra.run.smithi114.stdout: "long_desc": "", 2024-04-23T10:41:08.752 INFO:teuthology.orchestra.run.smithi114.stdout: "tags": [], 2024-04-23T10:41:08.753 INFO:teuthology.orchestra.run.smithi114.stdout: "see_also": [] 2024-04-23T10:41:08.753 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-23T10:41:08.753 INFO:teuthology.orchestra.run.smithi114.stdout: "container_image_prometheus": { 2024-04-23T10:41:08.753 INFO:teuthology.orchestra.run.smithi114.stdout: "name": "container_image_prometheus", 2024-04-23T10:41:08.753 INFO:teuthology.orchestra.run.smithi114.stdout: "type": "str", 2024-04-23T10:41:08.753 INFO:teuthology.orchestra.run.smithi114.stdout: "level": "advanced", 2024-04-23T10:41:08.753 INFO:teuthology.orchestra.run.smithi114.stdout: "flags": 0, 2024-04-23T10:41:08.753 INFO:teuthology.orchestra.run.smithi114.stdout: "default_value": "quay.io/prometheus/prometheus:v2.43.0", 2024-04-23T10:41:08.753 INFO:teuthology.orchestra.run.smithi114.stdout: "min": "", 2024-04-23T10:41:08.753 INFO:teuthology.orchestra.run.smithi114.stdout: "max": "", 2024-04-23T10:41:08.754 INFO:teuthology.orchestra.run.smithi114.stdout: "enum_allowed": [], 2024-04-23T10:41:08.754 INFO:teuthology.orchestra.run.smithi114.stdout: "desc": "Prometheus container image", 2024-04-23T10:41:08.754 INFO:teuthology.orchestra.run.smithi114.stdout: "long_desc": "", 2024-04-23T10:41:08.754 INFO:teuthology.orchestra.run.smithi114.stdout: "tags": [], 2024-04-23T10:41:08.754 INFO:teuthology.orchestra.run.smithi114.stdout: "see_also": [] 2024-04-23T10:41:08.754 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-23T10:41:08.754 INFO:teuthology.orchestra.run.smithi114.stdout: "container_image_promtail": { 2024-04-23T10:41:08.754 INFO:teuthology.orchestra.run.smithi114.stdout: "name": "container_image_promtail", 2024-04-23T10:41:08.754 INFO:teuthology.orchestra.run.smithi114.stdout: "type": "str", 2024-04-23T10:41:08.755 INFO:teuthology.orchestra.run.smithi114.stdout: "level": "advanced", 2024-04-23T10:41:08.755 INFO:teuthology.orchestra.run.smithi114.stdout: "flags": 0, 2024-04-23T10:41:08.755 INFO:teuthology.orchestra.run.smithi114.stdout: "default_value": "docker.io/grafana/promtail:2.4.0", 2024-04-23T10:41:08.755 INFO:teuthology.orchestra.run.smithi114.stdout: "min": "", 2024-04-23T10:41:08.755 INFO:teuthology.orchestra.run.smithi114.stdout: "max": "", 2024-04-23T10:41:08.755 INFO:teuthology.orchestra.run.smithi114.stdout: "enum_allowed": [], 2024-04-23T10:41:08.755 INFO:teuthology.orchestra.run.smithi114.stdout: "desc": "Promtail container image", 2024-04-23T10:41:08.755 INFO:teuthology.orchestra.run.smithi114.stdout: "long_desc": "", 2024-04-23T10:41:08.755 INFO:teuthology.orchestra.run.smithi114.stdout: "tags": [], 2024-04-23T10:41:08.755 INFO:teuthology.orchestra.run.smithi114.stdout: "see_also": [] 2024-04-23T10:41:08.756 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-23T10:41:08.756 INFO:teuthology.orchestra.run.smithi114.stdout: "container_image_samba": { 2024-04-23T10:41:08.756 INFO:teuthology.orchestra.run.smithi114.stdout: "name": "container_image_samba", 2024-04-23T10:41:08.756 INFO:teuthology.orchestra.run.smithi114.stdout: "type": "str", 2024-04-23T10:41:08.756 INFO:teuthology.orchestra.run.smithi114.stdout: "level": "advanced", 2024-04-23T10:41:08.756 INFO:teuthology.orchestra.run.smithi114.stdout: "flags": 0, 2024-04-23T10:41:08.756 INFO:teuthology.orchestra.run.smithi114.stdout: "default_value": "quay.io/samba.org/samba-server:devbuilds-centos-amd64", 2024-04-23T10:41:08.756 INFO:teuthology.orchestra.run.smithi114.stdout: "min": "", 2024-04-23T10:41:08.756 INFO:teuthology.orchestra.run.smithi114.stdout: "max": "", 2024-04-23T10:41:08.757 INFO:teuthology.orchestra.run.smithi114.stdout: "enum_allowed": [], 2024-04-23T10:41:08.757 INFO:teuthology.orchestra.run.smithi114.stdout: "desc": "Samba/SMB container image", 2024-04-23T10:41:08.757 INFO:teuthology.orchestra.run.smithi114.stdout: "long_desc": "", 2024-04-23T10:41:08.757 INFO:teuthology.orchestra.run.smithi114.stdout: "tags": [], 2024-04-23T10:41:08.757 INFO:teuthology.orchestra.run.smithi114.stdout: "see_also": [] 2024-04-23T10:41:08.757 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-23T10:41:08.757 INFO:teuthology.orchestra.run.smithi114.stdout: "container_image_snmp_gateway": { 2024-04-23T10:41:08.757 INFO:teuthology.orchestra.run.smithi114.stdout: "name": "container_image_snmp_gateway", 2024-04-23T10:41:08.757 INFO:teuthology.orchestra.run.smithi114.stdout: "type": "str", 2024-04-23T10:41:08.757 INFO:teuthology.orchestra.run.smithi114.stdout: "level": "advanced", 2024-04-23T10:41:08.758 INFO:teuthology.orchestra.run.smithi114.stdout: "flags": 0, 2024-04-23T10:41:08.758 INFO:teuthology.orchestra.run.smithi114.stdout: "default_value": "docker.io/maxwo/snmp-notifier:v1.2.1", 2024-04-23T10:41:08.758 INFO:teuthology.orchestra.run.smithi114.stdout: "min": "", 2024-04-23T10:41:08.758 INFO:teuthology.orchestra.run.smithi114.stdout: "max": "", 2024-04-23T10:41:08.758 INFO:teuthology.orchestra.run.smithi114.stdout: "enum_allowed": [], 2024-04-23T10:41:08.758 INFO:teuthology.orchestra.run.smithi114.stdout: "desc": "SNMP Gateway container image", 2024-04-23T10:41:08.758 INFO:teuthology.orchestra.run.smithi114.stdout: "long_desc": "", 2024-04-23T10:41:08.758 INFO:teuthology.orchestra.run.smithi114.stdout: "tags": [], 2024-04-23T10:41:08.758 INFO:teuthology.orchestra.run.smithi114.stdout: "see_also": [] 2024-04-23T10:41:08.758 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-23T10:41:08.759 INFO:teuthology.orchestra.run.smithi114.stdout: "container_init": { 2024-04-23T10:41:08.759 INFO:teuthology.orchestra.run.smithi114.stdout: "name": "container_init", 2024-04-23T10:41:08.759 INFO:teuthology.orchestra.run.smithi114.stdout: "type": "bool", 2024-04-23T10:41:08.759 INFO:teuthology.orchestra.run.smithi114.stdout: "level": "advanced", 2024-04-23T10:41:08.759 INFO:teuthology.orchestra.run.smithi114.stdout: "flags": 0, 2024-04-23T10:41:08.759 INFO:teuthology.orchestra.run.smithi114.stdout: "default_value": "True", 2024-04-23T10:41:08.759 INFO:teuthology.orchestra.run.smithi114.stdout: "min": "", 2024-04-23T10:41:08.759 INFO:teuthology.orchestra.run.smithi114.stdout: "max": "", 2024-04-23T10:41:08.759 INFO:teuthology.orchestra.run.smithi114.stdout: "enum_allowed": [], 2024-04-23T10:41:08.760 INFO:teuthology.orchestra.run.smithi114.stdout: "desc": "Run podman/docker with `--init`", 2024-04-23T10:41:08.760 INFO:teuthology.orchestra.run.smithi114.stdout: "long_desc": "", 2024-04-23T10:41:08.760 INFO:teuthology.orchestra.run.smithi114.stdout: "tags": [], 2024-04-23T10:41:08.760 INFO:teuthology.orchestra.run.smithi114.stdout: "see_also": [] 2024-04-23T10:41:08.760 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-23T10:41:08.760 INFO:teuthology.orchestra.run.smithi114.stdout: "daemon_cache_timeout": { 2024-04-23T10:41:08.760 INFO:teuthology.orchestra.run.smithi114.stdout: "name": "daemon_cache_timeout", 2024-04-23T10:41:08.760 INFO:teuthology.orchestra.run.smithi114.stdout: "type": "secs", 2024-04-23T10:41:08.760 INFO:teuthology.orchestra.run.smithi114.stdout: "level": "advanced", 2024-04-23T10:41:08.760 INFO:teuthology.orchestra.run.smithi114.stdout: "flags": 0, 2024-04-23T10:41:08.761 INFO:teuthology.orchestra.run.smithi114.stdout: "default_value": "600", 2024-04-23T10:41:08.761 INFO:teuthology.orchestra.run.smithi114.stdout: "min": "", 2024-04-23T10:41:08.761 INFO:teuthology.orchestra.run.smithi114.stdout: "max": "", 2024-04-23T10:41:08.761 INFO:teuthology.orchestra.run.smithi114.stdout: "enum_allowed": [], 2024-04-23T10:41:08.761 INFO:teuthology.orchestra.run.smithi114.stdout: "desc": "seconds to cache service (daemon) inventory", 2024-04-23T10:41:08.761 INFO:teuthology.orchestra.run.smithi114.stdout: "long_desc": "", 2024-04-23T10:41:08.761 INFO:teuthology.orchestra.run.smithi114.stdout: "tags": [], 2024-04-23T10:41:08.761 INFO:teuthology.orchestra.run.smithi114.stdout: "see_also": [] 2024-04-23T10:41:08.761 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-23T10:41:08.761 INFO:teuthology.orchestra.run.smithi114.stdout: "default_cephadm_command_timeout": { 2024-04-23T10:41:08.762 INFO:teuthology.orchestra.run.smithi114.stdout: "name": "default_cephadm_command_timeout", 2024-04-23T10:41:08.762 INFO:teuthology.orchestra.run.smithi114.stdout: "type": "int", 2024-04-23T10:41:08.762 INFO:teuthology.orchestra.run.smithi114.stdout: "level": "advanced", 2024-04-23T10:41:08.762 INFO:teuthology.orchestra.run.smithi114.stdout: "flags": 0, 2024-04-23T10:41:08.762 INFO:teuthology.orchestra.run.smithi114.stdout: "default_value": "900", 2024-04-23T10:41:08.762 INFO:teuthology.orchestra.run.smithi114.stdout: "min": "", 2024-04-23T10:41:08.762 INFO:teuthology.orchestra.run.smithi114.stdout: "max": "", 2024-04-23T10:41:08.762 INFO:teuthology.orchestra.run.smithi114.stdout: "enum_allowed": [], 2024-04-23T10:41:08.762 INFO:teuthology.orchestra.run.smithi114.stdout: "desc": "Default timeout applied to cephadm commands run directly on the host (in seconds)", 2024-04-23T10:41:08.763 INFO:teuthology.orchestra.run.smithi114.stdout: "long_desc": "", 2024-04-23T10:41:08.763 INFO:teuthology.orchestra.run.smithi114.stdout: "tags": [], 2024-04-23T10:41:08.763 INFO:teuthology.orchestra.run.smithi114.stdout: "see_also": [] 2024-04-23T10:41:08.763 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-23T10:41:08.763 INFO:teuthology.orchestra.run.smithi114.stdout: "default_registry": { 2024-04-23T10:41:08.763 INFO:teuthology.orchestra.run.smithi114.stdout: "name": "default_registry", 2024-04-23T10:41:08.763 INFO:teuthology.orchestra.run.smithi114.stdout: "type": "str", 2024-04-23T10:41:08.763 INFO:teuthology.orchestra.run.smithi114.stdout: "level": "advanced", 2024-04-23T10:41:08.763 INFO:teuthology.orchestra.run.smithi114.stdout: "flags": 0, 2024-04-23T10:41:08.763 INFO:teuthology.orchestra.run.smithi114.stdout: "default_value": "docker.io", 2024-04-23T10:41:08.764 INFO:teuthology.orchestra.run.smithi114.stdout: "min": "", 2024-04-23T10:41:08.764 INFO:teuthology.orchestra.run.smithi114.stdout: "max": "", 2024-04-23T10:41:08.764 INFO:teuthology.orchestra.run.smithi114.stdout: "enum_allowed": [], 2024-04-23T10:41:08.764 INFO:teuthology.orchestra.run.smithi114.stdout: "desc": "Search-registry to which we should normalize unqualified image names. This is not the default registry", 2024-04-23T10:41:08.764 INFO:teuthology.orchestra.run.smithi114.stdout: "long_desc": "", 2024-04-23T10:41:08.764 INFO:teuthology.orchestra.run.smithi114.stdout: "tags": [], 2024-04-23T10:41:08.764 INFO:teuthology.orchestra.run.smithi114.stdout: "see_also": [] 2024-04-23T10:41:08.764 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-23T10:41:08.764 INFO:teuthology.orchestra.run.smithi114.stdout: "device_cache_timeout": { 2024-04-23T10:41:08.764 INFO:teuthology.orchestra.run.smithi114.stdout: "name": "device_cache_timeout", 2024-04-23T10:41:08.765 INFO:teuthology.orchestra.run.smithi114.stdout: "type": "secs", 2024-04-23T10:41:08.765 INFO:teuthology.orchestra.run.smithi114.stdout: "level": "advanced", 2024-04-23T10:41:08.765 INFO:teuthology.orchestra.run.smithi114.stdout: "flags": 0, 2024-04-23T10:41:08.765 INFO:teuthology.orchestra.run.smithi114.stdout: "default_value": "1800", 2024-04-23T10:41:08.765 INFO:teuthology.orchestra.run.smithi114.stdout: "min": "", 2024-04-23T10:41:08.765 INFO:teuthology.orchestra.run.smithi114.stdout: "max": "", 2024-04-23T10:41:08.765 INFO:teuthology.orchestra.run.smithi114.stdout: "enum_allowed": [], 2024-04-23T10:41:08.765 INFO:teuthology.orchestra.run.smithi114.stdout: "desc": "seconds to cache device inventory", 2024-04-23T10:41:08.765 INFO:teuthology.orchestra.run.smithi114.stdout: "long_desc": "", 2024-04-23T10:41:08.765 INFO:teuthology.orchestra.run.smithi114.stdout: "tags": [], 2024-04-23T10:41:08.766 INFO:teuthology.orchestra.run.smithi114.stdout: "see_also": [] 2024-04-23T10:41:08.766 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-23T10:41:08.766 INFO:teuthology.orchestra.run.smithi114.stdout: "device_enhanced_scan": { 2024-04-23T10:41:08.766 INFO:teuthology.orchestra.run.smithi114.stdout: "name": "device_enhanced_scan", 2024-04-23T10:41:08.766 INFO:teuthology.orchestra.run.smithi114.stdout: "type": "bool", 2024-04-23T10:41:08.766 INFO:teuthology.orchestra.run.smithi114.stdout: "level": "advanced", 2024-04-23T10:41:08.766 INFO:teuthology.orchestra.run.smithi114.stdout: "flags": 0, 2024-04-23T10:41:08.766 INFO:teuthology.orchestra.run.smithi114.stdout: "default_value": "False", 2024-04-23T10:41:08.766 INFO:teuthology.orchestra.run.smithi114.stdout: "min": "", 2024-04-23T10:41:08.767 INFO:teuthology.orchestra.run.smithi114.stdout: "max": "", 2024-04-23T10:41:08.767 INFO:teuthology.orchestra.run.smithi114.stdout: "enum_allowed": [], 2024-04-23T10:41:08.767 INFO:teuthology.orchestra.run.smithi114.stdout: "desc": "Use libstoragemgmt during device scans", 2024-04-23T10:41:08.767 INFO:teuthology.orchestra.run.smithi114.stdout: "long_desc": "", 2024-04-23T10:41:08.767 INFO:teuthology.orchestra.run.smithi114.stdout: "tags": [], 2024-04-23T10:41:08.767 INFO:teuthology.orchestra.run.smithi114.stdout: "see_also": [] 2024-04-23T10:41:08.767 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-23T10:41:08.767 INFO:teuthology.orchestra.run.smithi114.stdout: "facts_cache_timeout": { 2024-04-23T10:41:08.767 INFO:teuthology.orchestra.run.smithi114.stdout: "name": "facts_cache_timeout", 2024-04-23T10:41:08.767 INFO:teuthology.orchestra.run.smithi114.stdout: "type": "secs", 2024-04-23T10:41:08.768 INFO:teuthology.orchestra.run.smithi114.stdout: "level": "advanced", 2024-04-23T10:41:08.768 INFO:teuthology.orchestra.run.smithi114.stdout: "flags": 0, 2024-04-23T10:41:08.768 INFO:teuthology.orchestra.run.smithi114.stdout: "default_value": "60", 2024-04-23T10:41:08.768 INFO:teuthology.orchestra.run.smithi114.stdout: "min": "", 2024-04-23T10:41:08.768 INFO:teuthology.orchestra.run.smithi114.stdout: "max": "", 2024-04-23T10:41:08.768 INFO:teuthology.orchestra.run.smithi114.stdout: "enum_allowed": [], 2024-04-23T10:41:08.768 INFO:teuthology.orchestra.run.smithi114.stdout: "desc": "seconds to cache host facts data", 2024-04-23T10:41:08.768 INFO:teuthology.orchestra.run.smithi114.stdout: "long_desc": "", 2024-04-23T10:41:08.768 INFO:teuthology.orchestra.run.smithi114.stdout: "tags": [], 2024-04-23T10:41:08.768 INFO:teuthology.orchestra.run.smithi114.stdout: "see_also": [] 2024-04-23T10:41:08.769 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-23T10:41:08.769 INFO:teuthology.orchestra.run.smithi114.stdout: "grafana_dashboards_path": { 2024-04-23T10:41:08.769 INFO:teuthology.orchestra.run.smithi114.stdout: "name": "grafana_dashboards_path", 2024-04-23T10:41:08.769 INFO:teuthology.orchestra.run.smithi114.stdout: "type": "str", 2024-04-23T10:41:08.769 INFO:teuthology.orchestra.run.smithi114.stdout: "level": "advanced", 2024-04-23T10:41:08.769 INFO:teuthology.orchestra.run.smithi114.stdout: "flags": 0, 2024-04-23T10:41:08.769 INFO:teuthology.orchestra.run.smithi114.stdout: "default_value": "/etc/grafana/dashboards/ceph-dashboard/", 2024-04-23T10:41:08.769 INFO:teuthology.orchestra.run.smithi114.stdout: "min": "", 2024-04-23T10:41:08.769 INFO:teuthology.orchestra.run.smithi114.stdout: "max": "", 2024-04-23T10:41:08.769 INFO:teuthology.orchestra.run.smithi114.stdout: "enum_allowed": [], 2024-04-23T10:41:08.770 INFO:teuthology.orchestra.run.smithi114.stdout: "desc": "location of dashboards to include in grafana deployments", 2024-04-23T10:41:08.770 INFO:teuthology.orchestra.run.smithi114.stdout: "long_desc": "", 2024-04-23T10:41:08.770 INFO:teuthology.orchestra.run.smithi114.stdout: "tags": [], 2024-04-23T10:41:08.770 INFO:teuthology.orchestra.run.smithi114.stdout: "see_also": [] 2024-04-23T10:41:08.770 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-23T10:41:08.770 INFO:teuthology.orchestra.run.smithi114.stdout: "host_check_interval": { 2024-04-23T10:41:08.770 INFO:teuthology.orchestra.run.smithi114.stdout: "name": "host_check_interval", 2024-04-23T10:41:08.770 INFO:teuthology.orchestra.run.smithi114.stdout: "type": "secs", 2024-04-23T10:41:08.770 INFO:teuthology.orchestra.run.smithi114.stdout: "level": "advanced", 2024-04-23T10:41:08.771 INFO:teuthology.orchestra.run.smithi114.stdout: "flags": 0, 2024-04-23T10:41:08.771 INFO:teuthology.orchestra.run.smithi114.stdout: "default_value": "600", 2024-04-23T10:41:08.771 INFO:teuthology.orchestra.run.smithi114.stdout: "min": "", 2024-04-23T10:41:08.771 INFO:teuthology.orchestra.run.smithi114.stdout: "max": "", 2024-04-23T10:41:08.771 INFO:teuthology.orchestra.run.smithi114.stdout: "enum_allowed": [], 2024-04-23T10:41:08.771 INFO:teuthology.orchestra.run.smithi114.stdout: "desc": "how frequently to perform a host check", 2024-04-23T10:41:08.771 INFO:teuthology.orchestra.run.smithi114.stdout: "long_desc": "", 2024-04-23T10:41:08.771 INFO:teuthology.orchestra.run.smithi114.stdout: "tags": [], 2024-04-23T10:41:08.771 INFO:teuthology.orchestra.run.smithi114.stdout: "see_also": [] 2024-04-23T10:41:08.771 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-23T10:41:08.772 INFO:teuthology.orchestra.run.smithi114.stdout: "hw_monitoring": { 2024-04-23T10:41:08.772 INFO:teuthology.orchestra.run.smithi114.stdout: "name": "hw_monitoring", 2024-04-23T10:41:08.772 INFO:teuthology.orchestra.run.smithi114.stdout: "type": "bool", 2024-04-23T10:41:08.772 INFO:teuthology.orchestra.run.smithi114.stdout: "level": "advanced", 2024-04-23T10:41:08.772 INFO:teuthology.orchestra.run.smithi114.stdout: "flags": 0, 2024-04-23T10:41:08.772 INFO:teuthology.orchestra.run.smithi114.stdout: "default_value": "False", 2024-04-23T10:41:08.772 INFO:teuthology.orchestra.run.smithi114.stdout: "min": "", 2024-04-23T10:41:08.772 INFO:teuthology.orchestra.run.smithi114.stdout: "max": "", 2024-04-23T10:41:08.772 INFO:teuthology.orchestra.run.smithi114.stdout: "enum_allowed": [], 2024-04-23T10:41:08.772 INFO:teuthology.orchestra.run.smithi114.stdout: "desc": "Deploy hw monitoring daemon on every host.", 2024-04-23T10:41:08.773 INFO:teuthology.orchestra.run.smithi114.stdout: "long_desc": "", 2024-04-23T10:41:08.773 INFO:teuthology.orchestra.run.smithi114.stdout: "tags": [], 2024-04-23T10:41:08.773 INFO:teuthology.orchestra.run.smithi114.stdout: "see_also": [] 2024-04-23T10:41:08.773 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-23T10:41:08.773 INFO:teuthology.orchestra.run.smithi114.stdout: "inventory_list_all": { 2024-04-23T10:41:08.773 INFO:teuthology.orchestra.run.smithi114.stdout: "name": "inventory_list_all", 2024-04-23T10:41:08.773 INFO:teuthology.orchestra.run.smithi114.stdout: "type": "bool", 2024-04-23T10:41:08.773 INFO:teuthology.orchestra.run.smithi114.stdout: "level": "advanced", 2024-04-23T10:41:08.773 INFO:teuthology.orchestra.run.smithi114.stdout: "flags": 0, 2024-04-23T10:41:08.773 INFO:teuthology.orchestra.run.smithi114.stdout: "default_value": "False", 2024-04-23T10:41:08.774 INFO:teuthology.orchestra.run.smithi114.stdout: "min": "", 2024-04-23T10:41:08.774 INFO:teuthology.orchestra.run.smithi114.stdout: "max": "", 2024-04-23T10:41:08.774 INFO:teuthology.orchestra.run.smithi114.stdout: "enum_allowed": [], 2024-04-23T10:41:08.774 INFO:teuthology.orchestra.run.smithi114.stdout: "desc": "Whether ceph-volume inventory should report more devices (mostly mappers (LVs / mpaths), partitions...)", 2024-04-23T10:41:08.774 INFO:teuthology.orchestra.run.smithi114.stdout: "long_desc": "", 2024-04-23T10:41:08.774 INFO:teuthology.orchestra.run.smithi114.stdout: "tags": [], 2024-04-23T10:41:08.774 INFO:teuthology.orchestra.run.smithi114.stdout: "see_also": [] 2024-04-23T10:41:08.774 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-23T10:41:08.774 INFO:teuthology.orchestra.run.smithi114.stdout: "log_level": { 2024-04-23T10:41:08.775 INFO:teuthology.orchestra.run.smithi114.stdout: "name": "log_level", 2024-04-23T10:41:08.775 INFO:teuthology.orchestra.run.smithi114.stdout: "type": "str", 2024-04-23T10:41:08.775 INFO:teuthology.orchestra.run.smithi114.stdout: "level": "advanced", 2024-04-23T10:41:08.775 INFO:teuthology.orchestra.run.smithi114.stdout: "flags": 1, 2024-04-23T10:41:08.775 INFO:teuthology.orchestra.run.smithi114.stdout: "default_value": "", 2024-04-23T10:41:08.775 INFO:teuthology.orchestra.run.smithi114.stdout: "min": "", 2024-04-23T10:41:08.775 INFO:teuthology.orchestra.run.smithi114.stdout: "max": "", 2024-04-23T10:41:08.775 INFO:teuthology.orchestra.run.smithi114.stdout: "enum_allowed": [ 2024-04-23T10:41:08.775 INFO:teuthology.orchestra.run.smithi114.stdout: "", 2024-04-23T10:41:08.775 INFO:teuthology.orchestra.run.smithi114.stdout: "critical", 2024-04-23T10:41:08.776 INFO:teuthology.orchestra.run.smithi114.stdout: "debug", 2024-04-23T10:41:08.776 INFO:teuthology.orchestra.run.smithi114.stdout: "error", 2024-04-23T10:41:08.776 INFO:teuthology.orchestra.run.smithi114.stdout: "info", 2024-04-23T10:41:08.776 INFO:teuthology.orchestra.run.smithi114.stdout: "warning" 2024-04-23T10:41:08.776 INFO:teuthology.orchestra.run.smithi114.stdout: ], 2024-04-23T10:41:08.776 INFO:teuthology.orchestra.run.smithi114.stdout: "desc": "", 2024-04-23T10:41:08.776 INFO:teuthology.orchestra.run.smithi114.stdout: "long_desc": "", 2024-04-23T10:41:08.776 INFO:teuthology.orchestra.run.smithi114.stdout: "tags": [], 2024-04-23T10:41:08.776 INFO:teuthology.orchestra.run.smithi114.stdout: "see_also": [] 2024-04-23T10:41:08.776 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-23T10:41:08.777 INFO:teuthology.orchestra.run.smithi114.stdout: "log_refresh_metadata": { 2024-04-23T10:41:08.777 INFO:teuthology.orchestra.run.smithi114.stdout: "name": "log_refresh_metadata", 2024-04-23T10:41:08.777 INFO:teuthology.orchestra.run.smithi114.stdout: "type": "bool", 2024-04-23T10:41:08.777 INFO:teuthology.orchestra.run.smithi114.stdout: "level": "advanced", 2024-04-23T10:41:08.777 INFO:teuthology.orchestra.run.smithi114.stdout: "flags": 0, 2024-04-23T10:41:08.777 INFO:teuthology.orchestra.run.smithi114.stdout: "default_value": "False", 2024-04-23T10:41:08.777 INFO:teuthology.orchestra.run.smithi114.stdout: "min": "", 2024-04-23T10:41:08.777 INFO:teuthology.orchestra.run.smithi114.stdout: "max": "", 2024-04-23T10:41:08.777 INFO:teuthology.orchestra.run.smithi114.stdout: "enum_allowed": [], 2024-04-23T10:41:08.777 INFO:teuthology.orchestra.run.smithi114.stdout: "desc": "Log all refresh metadata. Includes daemon, device, and host info collected regularly. Only has effect if logging at debug level", 2024-04-23T10:41:08.778 INFO:teuthology.orchestra.run.smithi114.stdout: "long_desc": "", 2024-04-23T10:41:08.778 INFO:teuthology.orchestra.run.smithi114.stdout: "tags": [], 2024-04-23T10:41:08.778 INFO:teuthology.orchestra.run.smithi114.stdout: "see_also": [] 2024-04-23T10:41:08.778 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-23T10:41:08.778 INFO:teuthology.orchestra.run.smithi114.stdout: "log_to_cluster": { 2024-04-23T10:41:08.778 INFO:teuthology.orchestra.run.smithi114.stdout: "name": "log_to_cluster", 2024-04-23T10:41:08.778 INFO:teuthology.orchestra.run.smithi114.stdout: "type": "bool", 2024-04-23T10:41:08.778 INFO:teuthology.orchestra.run.smithi114.stdout: "level": "advanced", 2024-04-23T10:41:08.778 INFO:teuthology.orchestra.run.smithi114.stdout: "flags": 0, 2024-04-23T10:41:08.779 INFO:teuthology.orchestra.run.smithi114.stdout: "default_value": "True", 2024-04-23T10:41:08.779 INFO:teuthology.orchestra.run.smithi114.stdout: "min": "", 2024-04-23T10:41:08.779 INFO:teuthology.orchestra.run.smithi114.stdout: "max": "", 2024-04-23T10:41:08.779 INFO:teuthology.orchestra.run.smithi114.stdout: "enum_allowed": [], 2024-04-23T10:41:08.779 INFO:teuthology.orchestra.run.smithi114.stdout: "desc": "log to the \"cephadm\" cluster log channel\"", 2024-04-23T10:41:08.779 INFO:teuthology.orchestra.run.smithi114.stdout: "long_desc": "", 2024-04-23T10:41:08.779 INFO:teuthology.orchestra.run.smithi114.stdout: "tags": [], 2024-04-23T10:41:08.779 INFO:teuthology.orchestra.run.smithi114.stdout: "see_also": [] 2024-04-23T10:41:08.779 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-23T10:41:08.779 INFO:teuthology.orchestra.run.smithi114.stdout: "log_to_cluster_level": { 2024-04-23T10:41:08.780 INFO:teuthology.orchestra.run.smithi114.stdout: "name": "log_to_cluster_level", 2024-04-23T10:41:08.780 INFO:teuthology.orchestra.run.smithi114.stdout: "type": "str", 2024-04-23T10:41:08.780 INFO:teuthology.orchestra.run.smithi114.stdout: "level": "advanced", 2024-04-23T10:41:08.780 INFO:teuthology.orchestra.run.smithi114.stdout: "flags": 1, 2024-04-23T10:41:08.780 INFO:teuthology.orchestra.run.smithi114.stdout: "default_value": "info", 2024-04-23T10:41:08.780 INFO:teuthology.orchestra.run.smithi114.stdout: "min": "", 2024-04-23T10:41:08.780 INFO:teuthology.orchestra.run.smithi114.stdout: "max": "", 2024-04-23T10:41:08.780 INFO:teuthology.orchestra.run.smithi114.stdout: "enum_allowed": [ 2024-04-23T10:41:08.780 INFO:teuthology.orchestra.run.smithi114.stdout: "", 2024-04-23T10:41:08.780 INFO:teuthology.orchestra.run.smithi114.stdout: "critical", 2024-04-23T10:41:08.781 INFO:teuthology.orchestra.run.smithi114.stdout: "debug", 2024-04-23T10:41:08.781 INFO:teuthology.orchestra.run.smithi114.stdout: "error", 2024-04-23T10:41:08.781 INFO:teuthology.orchestra.run.smithi114.stdout: "info", 2024-04-23T10:41:08.781 INFO:teuthology.orchestra.run.smithi114.stdout: "warning" 2024-04-23T10:41:08.781 INFO:teuthology.orchestra.run.smithi114.stdout: ], 2024-04-23T10:41:08.781 INFO:teuthology.orchestra.run.smithi114.stdout: "desc": "", 2024-04-23T10:41:08.781 INFO:teuthology.orchestra.run.smithi114.stdout: "long_desc": "", 2024-04-23T10:41:08.781 INFO:teuthology.orchestra.run.smithi114.stdout: "tags": [], 2024-04-23T10:41:08.781 INFO:teuthology.orchestra.run.smithi114.stdout: "see_also": [] 2024-04-23T10:41:08.781 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-23T10:41:08.782 INFO:teuthology.orchestra.run.smithi114.stdout: "log_to_file": { 2024-04-23T10:41:08.782 INFO:teuthology.orchestra.run.smithi114.stdout: "name": "log_to_file", 2024-04-23T10:41:08.782 INFO:teuthology.orchestra.run.smithi114.stdout: "type": "bool", 2024-04-23T10:41:08.782 INFO:teuthology.orchestra.run.smithi114.stdout: "level": "advanced", 2024-04-23T10:41:08.782 INFO:teuthology.orchestra.run.smithi114.stdout: "flags": 1, 2024-04-23T10:41:08.782 INFO:teuthology.orchestra.run.smithi114.stdout: "default_value": "False", 2024-04-23T10:41:08.782 INFO:teuthology.orchestra.run.smithi114.stdout: "min": "", 2024-04-23T10:41:08.782 INFO:teuthology.orchestra.run.smithi114.stdout: "max": "", 2024-04-23T10:41:08.782 INFO:teuthology.orchestra.run.smithi114.stdout: "enum_allowed": [], 2024-04-23T10:41:08.782 INFO:teuthology.orchestra.run.smithi114.stdout: "desc": "", 2024-04-23T10:41:08.783 INFO:teuthology.orchestra.run.smithi114.stdout: "long_desc": "", 2024-04-23T10:41:08.783 INFO:teuthology.orchestra.run.smithi114.stdout: "tags": [], 2024-04-23T10:41:08.783 INFO:teuthology.orchestra.run.smithi114.stdout: "see_also": [] 2024-04-23T10:41:08.783 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-23T10:41:08.783 INFO:teuthology.orchestra.run.smithi114.stdout: "manage_etc_ceph_ceph_conf": { 2024-04-23T10:41:08.783 INFO:teuthology.orchestra.run.smithi114.stdout: "name": "manage_etc_ceph_ceph_conf", 2024-04-23T10:41:08.783 INFO:teuthology.orchestra.run.smithi114.stdout: "type": "bool", 2024-04-23T10:41:08.783 INFO:teuthology.orchestra.run.smithi114.stdout: "level": "advanced", 2024-04-23T10:41:08.783 INFO:teuthology.orchestra.run.smithi114.stdout: "flags": 0, 2024-04-23T10:41:08.783 INFO:teuthology.orchestra.run.smithi114.stdout: "default_value": "False", 2024-04-23T10:41:08.784 INFO:teuthology.orchestra.run.smithi114.stdout: "min": "", 2024-04-23T10:41:08.784 INFO:teuthology.orchestra.run.smithi114.stdout: "max": "", 2024-04-23T10:41:08.784 INFO:teuthology.orchestra.run.smithi114.stdout: "enum_allowed": [], 2024-04-23T10:41:08.784 INFO:teuthology.orchestra.run.smithi114.stdout: "desc": "Manage and own /etc/ceph/ceph.conf on the hosts.", 2024-04-23T10:41:08.784 INFO:teuthology.orchestra.run.smithi114.stdout: "long_desc": "", 2024-04-23T10:41:08.784 INFO:teuthology.orchestra.run.smithi114.stdout: "tags": [], 2024-04-23T10:41:08.784 INFO:teuthology.orchestra.run.smithi114.stdout: "see_also": [] 2024-04-23T10:41:08.784 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-23T10:41:08.784 INFO:teuthology.orchestra.run.smithi114.stdout: "manage_etc_ceph_ceph_conf_hosts": { 2024-04-23T10:41:08.785 INFO:teuthology.orchestra.run.smithi114.stdout: "name": "manage_etc_ceph_ceph_conf_hosts", 2024-04-23T10:41:08.785 INFO:teuthology.orchestra.run.smithi114.stdout: "type": "str", 2024-04-23T10:41:08.785 INFO:teuthology.orchestra.run.smithi114.stdout: "level": "advanced", 2024-04-23T10:41:08.785 INFO:teuthology.orchestra.run.smithi114.stdout: "flags": 0, 2024-04-23T10:41:08.785 INFO:teuthology.orchestra.run.smithi114.stdout: "default_value": "*", 2024-04-23T10:41:08.785 INFO:teuthology.orchestra.run.smithi114.stdout: "min": "", 2024-04-23T10:41:08.785 INFO:teuthology.orchestra.run.smithi114.stdout: "max": "", 2024-04-23T10:41:08.785 INFO:teuthology.orchestra.run.smithi114.stdout: "enum_allowed": [], 2024-04-23T10:41:08.785 INFO:teuthology.orchestra.run.smithi114.stdout: "desc": "PlacementSpec describing on which hosts to manage /etc/ceph/ceph.conf", 2024-04-23T10:41:08.785 INFO:teuthology.orchestra.run.smithi114.stdout: "long_desc": "", 2024-04-23T10:41:08.786 INFO:teuthology.orchestra.run.smithi114.stdout: "tags": [], 2024-04-23T10:41:08.786 INFO:teuthology.orchestra.run.smithi114.stdout: "see_also": [] 2024-04-23T10:41:08.786 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-23T10:41:08.786 INFO:teuthology.orchestra.run.smithi114.stdout: "max_count_per_host": { 2024-04-23T10:41:08.786 INFO:teuthology.orchestra.run.smithi114.stdout: "name": "max_count_per_host", 2024-04-23T10:41:08.786 INFO:teuthology.orchestra.run.smithi114.stdout: "type": "int", 2024-04-23T10:41:08.786 INFO:teuthology.orchestra.run.smithi114.stdout: "level": "advanced", 2024-04-23T10:41:08.786 INFO:teuthology.orchestra.run.smithi114.stdout: "flags": 0, 2024-04-23T10:41:08.786 INFO:teuthology.orchestra.run.smithi114.stdout: "default_value": "10", 2024-04-23T10:41:08.786 INFO:teuthology.orchestra.run.smithi114.stdout: "min": "", 2024-04-23T10:41:08.787 INFO:teuthology.orchestra.run.smithi114.stdout: "max": "", 2024-04-23T10:41:08.787 INFO:teuthology.orchestra.run.smithi114.stdout: "enum_allowed": [], 2024-04-23T10:41:08.787 INFO:teuthology.orchestra.run.smithi114.stdout: "desc": "max number of daemons per service per host", 2024-04-23T10:41:08.787 INFO:teuthology.orchestra.run.smithi114.stdout: "long_desc": "", 2024-04-23T10:41:08.787 INFO:teuthology.orchestra.run.smithi114.stdout: "tags": [], 2024-04-23T10:41:08.787 INFO:teuthology.orchestra.run.smithi114.stdout: "see_also": [] 2024-04-23T10:41:08.787 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-23T10:41:08.787 INFO:teuthology.orchestra.run.smithi114.stdout: "max_osd_draining_count": { 2024-04-23T10:41:08.787 INFO:teuthology.orchestra.run.smithi114.stdout: "name": "max_osd_draining_count", 2024-04-23T10:41:08.787 INFO:teuthology.orchestra.run.smithi114.stdout: "type": "int", 2024-04-23T10:41:08.788 INFO:teuthology.orchestra.run.smithi114.stdout: "level": "advanced", 2024-04-23T10:41:08.788 INFO:teuthology.orchestra.run.smithi114.stdout: "flags": 0, 2024-04-23T10:41:08.788 INFO:teuthology.orchestra.run.smithi114.stdout: "default_value": "10", 2024-04-23T10:41:08.788 INFO:teuthology.orchestra.run.smithi114.stdout: "min": "", 2024-04-23T10:41:08.788 INFO:teuthology.orchestra.run.smithi114.stdout: "max": "", 2024-04-23T10:41:08.788 INFO:teuthology.orchestra.run.smithi114.stdout: "enum_allowed": [], 2024-04-23T10:41:08.788 INFO:teuthology.orchestra.run.smithi114.stdout: "desc": "max number of osds that will be drained simultaneously when osds are removed", 2024-04-23T10:41:08.788 INFO:teuthology.orchestra.run.smithi114.stdout: "long_desc": "", 2024-04-23T10:41:08.788 INFO:teuthology.orchestra.run.smithi114.stdout: "tags": [], 2024-04-23T10:41:08.788 INFO:teuthology.orchestra.run.smithi114.stdout: "see_also": [] 2024-04-23T10:41:08.789 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-23T10:41:08.789 INFO:teuthology.orchestra.run.smithi114.stdout: "migration_current": { 2024-04-23T10:41:08.789 INFO:teuthology.orchestra.run.smithi114.stdout: "name": "migration_current", 2024-04-23T10:41:08.789 INFO:teuthology.orchestra.run.smithi114.stdout: "type": "int", 2024-04-23T10:41:08.789 INFO:teuthology.orchestra.run.smithi114.stdout: "level": "advanced", 2024-04-23T10:41:08.789 INFO:teuthology.orchestra.run.smithi114.stdout: "flags": 0, 2024-04-23T10:41:08.789 INFO:teuthology.orchestra.run.smithi114.stdout: "default_value": "", 2024-04-23T10:41:08.789 INFO:teuthology.orchestra.run.smithi114.stdout: "min": "", 2024-04-23T10:41:08.789 INFO:teuthology.orchestra.run.smithi114.stdout: "max": "", 2024-04-23T10:41:08.789 INFO:teuthology.orchestra.run.smithi114.stdout: "enum_allowed": [], 2024-04-23T10:41:08.790 INFO:teuthology.orchestra.run.smithi114.stdout: "desc": "internal - do not modify", 2024-04-23T10:41:08.790 INFO:teuthology.orchestra.run.smithi114.stdout: "long_desc": "", 2024-04-23T10:41:08.790 INFO:teuthology.orchestra.run.smithi114.stdout: "tags": [], 2024-04-23T10:41:08.790 INFO:teuthology.orchestra.run.smithi114.stdout: "see_also": [] 2024-04-23T10:41:08.790 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-23T10:41:08.790 INFO:teuthology.orchestra.run.smithi114.stdout: "mode": { 2024-04-23T10:41:08.790 INFO:teuthology.orchestra.run.smithi114.stdout: "name": "mode", 2024-04-23T10:41:08.790 INFO:teuthology.orchestra.run.smithi114.stdout: "type": "str", 2024-04-23T10:41:08.790 INFO:teuthology.orchestra.run.smithi114.stdout: "level": "advanced", 2024-04-23T10:41:08.790 INFO:teuthology.orchestra.run.smithi114.stdout: "flags": 0, 2024-04-23T10:41:08.791 INFO:teuthology.orchestra.run.smithi114.stdout: "default_value": "root", 2024-04-23T10:41:08.791 INFO:teuthology.orchestra.run.smithi114.stdout: "min": "", 2024-04-23T10:41:08.791 INFO:teuthology.orchestra.run.smithi114.stdout: "max": "", 2024-04-23T10:41:08.791 INFO:teuthology.orchestra.run.smithi114.stdout: "enum_allowed": [ 2024-04-23T10:41:08.791 INFO:teuthology.orchestra.run.smithi114.stdout: "cephadm-package", 2024-04-23T10:41:08.791 INFO:teuthology.orchestra.run.smithi114.stdout: "root" 2024-04-23T10:41:08.791 INFO:teuthology.orchestra.run.smithi114.stdout: ], 2024-04-23T10:41:08.791 INFO:teuthology.orchestra.run.smithi114.stdout: "desc": "mode for remote execution of cephadm", 2024-04-23T10:41:08.791 INFO:teuthology.orchestra.run.smithi114.stdout: "long_desc": "", 2024-04-23T10:41:08.792 INFO:teuthology.orchestra.run.smithi114.stdout: "tags": [], 2024-04-23T10:41:08.792 INFO:teuthology.orchestra.run.smithi114.stdout: "see_also": [] 2024-04-23T10:41:08.792 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-23T10:41:08.792 INFO:teuthology.orchestra.run.smithi114.stdout: "oob_default_addr": { 2024-04-23T10:41:08.792 INFO:teuthology.orchestra.run.smithi114.stdout: "name": "oob_default_addr", 2024-04-23T10:41:08.792 INFO:teuthology.orchestra.run.smithi114.stdout: "type": "str", 2024-04-23T10:41:08.792 INFO:teuthology.orchestra.run.smithi114.stdout: "level": "advanced", 2024-04-23T10:41:08.792 INFO:teuthology.orchestra.run.smithi114.stdout: "flags": 0, 2024-04-23T10:41:08.792 INFO:teuthology.orchestra.run.smithi114.stdout: "default_value": "169.254.1.1", 2024-04-23T10:41:08.792 INFO:teuthology.orchestra.run.smithi114.stdout: "min": "", 2024-04-23T10:41:08.793 INFO:teuthology.orchestra.run.smithi114.stdout: "max": "", 2024-04-23T10:41:08.793 INFO:teuthology.orchestra.run.smithi114.stdout: "enum_allowed": [], 2024-04-23T10:41:08.793 INFO:teuthology.orchestra.run.smithi114.stdout: "desc": "Default address for RedFish API (oob management).", 2024-04-23T10:41:08.793 INFO:teuthology.orchestra.run.smithi114.stdout: "long_desc": "", 2024-04-23T10:41:08.793 INFO:teuthology.orchestra.run.smithi114.stdout: "tags": [], 2024-04-23T10:41:08.793 INFO:teuthology.orchestra.run.smithi114.stdout: "see_also": [] 2024-04-23T10:41:08.793 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-23T10:41:08.793 INFO:teuthology.orchestra.run.smithi114.stdout: "prometheus_alerts_path": { 2024-04-23T10:41:08.793 INFO:teuthology.orchestra.run.smithi114.stdout: "name": "prometheus_alerts_path", 2024-04-23T10:41:08.794 INFO:teuthology.orchestra.run.smithi114.stdout: "type": "str", 2024-04-23T10:41:08.794 INFO:teuthology.orchestra.run.smithi114.stdout: "level": "advanced", 2024-04-23T10:41:08.794 INFO:teuthology.orchestra.run.smithi114.stdout: "flags": 0, 2024-04-23T10:41:08.794 INFO:teuthology.orchestra.run.smithi114.stdout: "default_value": "/etc/prometheus/ceph/ceph_default_alerts.yml", 2024-04-23T10:41:08.794 INFO:teuthology.orchestra.run.smithi114.stdout: "min": "", 2024-04-23T10:41:08.794 INFO:teuthology.orchestra.run.smithi114.stdout: "max": "", 2024-04-23T10:41:08.794 INFO:teuthology.orchestra.run.smithi114.stdout: "enum_allowed": [], 2024-04-23T10:41:08.794 INFO:teuthology.orchestra.run.smithi114.stdout: "desc": "location of alerts to include in prometheus deployments", 2024-04-23T10:41:08.794 INFO:teuthology.orchestra.run.smithi114.stdout: "long_desc": "", 2024-04-23T10:41:08.794 INFO:teuthology.orchestra.run.smithi114.stdout: "tags": [], 2024-04-23T10:41:08.795 INFO:teuthology.orchestra.run.smithi114.stdout: "see_also": [] 2024-04-23T10:41:08.795 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-23T10:41:08.795 INFO:teuthology.orchestra.run.smithi114.stdout: "registry_insecure": { 2024-04-23T10:41:08.795 INFO:teuthology.orchestra.run.smithi114.stdout: "name": "registry_insecure", 2024-04-23T10:41:08.795 INFO:teuthology.orchestra.run.smithi114.stdout: "type": "bool", 2024-04-23T10:41:08.795 INFO:teuthology.orchestra.run.smithi114.stdout: "level": "advanced", 2024-04-23T10:41:08.795 INFO:teuthology.orchestra.run.smithi114.stdout: "flags": 0, 2024-04-23T10:41:08.795 INFO:teuthology.orchestra.run.smithi114.stdout: "default_value": "False", 2024-04-23T10:41:08.795 INFO:teuthology.orchestra.run.smithi114.stdout: "min": "", 2024-04-23T10:41:08.796 INFO:teuthology.orchestra.run.smithi114.stdout: "max": "", 2024-04-23T10:41:08.796 INFO:teuthology.orchestra.run.smithi114.stdout: "enum_allowed": [], 2024-04-23T10:41:08.796 INFO:teuthology.orchestra.run.smithi114.stdout: "desc": "Registry is to be considered insecure (no TLS available). Only for development purposes.", 2024-04-23T10:41:08.796 INFO:teuthology.orchestra.run.smithi114.stdout: "long_desc": "", 2024-04-23T10:41:08.796 INFO:teuthology.orchestra.run.smithi114.stdout: "tags": [], 2024-04-23T10:41:08.796 INFO:teuthology.orchestra.run.smithi114.stdout: "see_also": [] 2024-04-23T10:41:08.796 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-23T10:41:08.796 INFO:teuthology.orchestra.run.smithi114.stdout: "registry_password": { 2024-04-23T10:41:08.796 INFO:teuthology.orchestra.run.smithi114.stdout: "name": "registry_password", 2024-04-23T10:41:08.796 INFO:teuthology.orchestra.run.smithi114.stdout: "type": "str", 2024-04-23T10:41:08.797 INFO:teuthology.orchestra.run.smithi114.stdout: "level": "advanced", 2024-04-23T10:41:08.797 INFO:teuthology.orchestra.run.smithi114.stdout: "flags": 0, 2024-04-23T10:41:08.797 INFO:teuthology.orchestra.run.smithi114.stdout: "default_value": "", 2024-04-23T10:41:08.797 INFO:teuthology.orchestra.run.smithi114.stdout: "min": "", 2024-04-23T10:41:08.797 INFO:teuthology.orchestra.run.smithi114.stdout: "max": "", 2024-04-23T10:41:08.797 INFO:teuthology.orchestra.run.smithi114.stdout: "enum_allowed": [], 2024-04-23T10:41:08.797 INFO:teuthology.orchestra.run.smithi114.stdout: "desc": "Custom repository password. Only used for logging into a registry.", 2024-04-23T10:41:08.797 INFO:teuthology.orchestra.run.smithi114.stdout: "long_desc": "", 2024-04-23T10:41:08.797 INFO:teuthology.orchestra.run.smithi114.stdout: "tags": [], 2024-04-23T10:41:08.797 INFO:teuthology.orchestra.run.smithi114.stdout: "see_also": [] 2024-04-23T10:41:08.798 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-23T10:41:08.798 INFO:teuthology.orchestra.run.smithi114.stdout: "registry_url": { 2024-04-23T10:41:08.798 INFO:teuthology.orchestra.run.smithi114.stdout: "name": "registry_url", 2024-04-23T10:41:08.798 INFO:teuthology.orchestra.run.smithi114.stdout: "type": "str", 2024-04-23T10:41:08.798 INFO:teuthology.orchestra.run.smithi114.stdout: "level": "advanced", 2024-04-23T10:41:08.798 INFO:teuthology.orchestra.run.smithi114.stdout: "flags": 0, 2024-04-23T10:41:08.798 INFO:teuthology.orchestra.run.smithi114.stdout: "default_value": "", 2024-04-23T10:41:08.798 INFO:teuthology.orchestra.run.smithi114.stdout: "min": "", 2024-04-23T10:41:08.798 INFO:teuthology.orchestra.run.smithi114.stdout: "max": "", 2024-04-23T10:41:08.798 INFO:teuthology.orchestra.run.smithi114.stdout: "enum_allowed": [], 2024-04-23T10:41:08.799 INFO:teuthology.orchestra.run.smithi114.stdout: "desc": "Registry url for login purposes. This is not the default registry", 2024-04-23T10:41:08.799 INFO:teuthology.orchestra.run.smithi114.stdout: "long_desc": "", 2024-04-23T10:41:08.799 INFO:teuthology.orchestra.run.smithi114.stdout: "tags": [], 2024-04-23T10:41:08.799 INFO:teuthology.orchestra.run.smithi114.stdout: "see_also": [] 2024-04-23T10:41:08.799 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-23T10:41:08.799 INFO:teuthology.orchestra.run.smithi114.stdout: "registry_username": { 2024-04-23T10:41:08.799 INFO:teuthology.orchestra.run.smithi114.stdout: "name": "registry_username", 2024-04-23T10:41:08.799 INFO:teuthology.orchestra.run.smithi114.stdout: "type": "str", 2024-04-23T10:41:08.799 INFO:teuthology.orchestra.run.smithi114.stdout: "level": "advanced", 2024-04-23T10:41:08.799 INFO:teuthology.orchestra.run.smithi114.stdout: "flags": 0, 2024-04-23T10:41:08.800 INFO:teuthology.orchestra.run.smithi114.stdout: "default_value": "", 2024-04-23T10:41:08.800 INFO:teuthology.orchestra.run.smithi114.stdout: "min": "", 2024-04-23T10:41:08.800 INFO:teuthology.orchestra.run.smithi114.stdout: "max": "", 2024-04-23T10:41:08.800 INFO:teuthology.orchestra.run.smithi114.stdout: "enum_allowed": [], 2024-04-23T10:41:08.800 INFO:teuthology.orchestra.run.smithi114.stdout: "desc": "Custom repository username. Only used for logging into a registry.", 2024-04-23T10:41:08.800 INFO:teuthology.orchestra.run.smithi114.stdout: "long_desc": "", 2024-04-23T10:41:08.800 INFO:teuthology.orchestra.run.smithi114.stdout: "tags": [], 2024-04-23T10:41:08.800 INFO:teuthology.orchestra.run.smithi114.stdout: "see_also": [] 2024-04-23T10:41:08.800 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-23T10:41:08.800 INFO:teuthology.orchestra.run.smithi114.stdout: "secure_monitoring_stack": { 2024-04-23T10:41:08.801 INFO:teuthology.orchestra.run.smithi114.stdout: "name": "secure_monitoring_stack", 2024-04-23T10:41:08.801 INFO:teuthology.orchestra.run.smithi114.stdout: "type": "bool", 2024-04-23T10:41:08.801 INFO:teuthology.orchestra.run.smithi114.stdout: "level": "advanced", 2024-04-23T10:41:08.801 INFO:teuthology.orchestra.run.smithi114.stdout: "flags": 0, 2024-04-23T10:41:08.801 INFO:teuthology.orchestra.run.smithi114.stdout: "default_value": "False", 2024-04-23T10:41:08.801 INFO:teuthology.orchestra.run.smithi114.stdout: "min": "", 2024-04-23T10:41:08.801 INFO:teuthology.orchestra.run.smithi114.stdout: "max": "", 2024-04-23T10:41:08.801 INFO:teuthology.orchestra.run.smithi114.stdout: "enum_allowed": [], 2024-04-23T10:41:08.801 INFO:teuthology.orchestra.run.smithi114.stdout: "desc": "Enable TLS security for all the monitoring stack daemons", 2024-04-23T10:41:08.802 INFO:teuthology.orchestra.run.smithi114.stdout: "long_desc": "", 2024-04-23T10:41:08.802 INFO:teuthology.orchestra.run.smithi114.stdout: "tags": [], 2024-04-23T10:41:08.802 INFO:teuthology.orchestra.run.smithi114.stdout: "see_also": [] 2024-04-23T10:41:08.802 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-23T10:41:08.802 INFO:teuthology.orchestra.run.smithi114.stdout: "service_discovery_port": { 2024-04-23T10:41:08.802 INFO:teuthology.orchestra.run.smithi114.stdout: "name": "service_discovery_port", 2024-04-23T10:41:08.802 INFO:teuthology.orchestra.run.smithi114.stdout: "type": "int", 2024-04-23T10:41:08.802 INFO:teuthology.orchestra.run.smithi114.stdout: "level": "advanced", 2024-04-23T10:41:08.802 INFO:teuthology.orchestra.run.smithi114.stdout: "flags": 0, 2024-04-23T10:41:08.802 INFO:teuthology.orchestra.run.smithi114.stdout: "default_value": "8765", 2024-04-23T10:41:08.803 INFO:teuthology.orchestra.run.smithi114.stdout: "min": "", 2024-04-23T10:41:08.803 INFO:teuthology.orchestra.run.smithi114.stdout: "max": "", 2024-04-23T10:41:08.803 INFO:teuthology.orchestra.run.smithi114.stdout: "enum_allowed": [], 2024-04-23T10:41:08.803 INFO:teuthology.orchestra.run.smithi114.stdout: "desc": "cephadm service discovery port", 2024-04-23T10:41:08.803 INFO:teuthology.orchestra.run.smithi114.stdout: "long_desc": "", 2024-04-23T10:41:08.803 INFO:teuthology.orchestra.run.smithi114.stdout: "tags": [], 2024-04-23T10:41:08.803 INFO:teuthology.orchestra.run.smithi114.stdout: "see_also": [] 2024-04-23T10:41:08.803 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-23T10:41:08.803 INFO:teuthology.orchestra.run.smithi114.stdout: "ssh_config_file": { 2024-04-23T10:41:08.803 INFO:teuthology.orchestra.run.smithi114.stdout: "name": "ssh_config_file", 2024-04-23T10:41:08.804 INFO:teuthology.orchestra.run.smithi114.stdout: "type": "str", 2024-04-23T10:41:08.804 INFO:teuthology.orchestra.run.smithi114.stdout: "level": "advanced", 2024-04-23T10:41:08.804 INFO:teuthology.orchestra.run.smithi114.stdout: "flags": 0, 2024-04-23T10:41:08.804 INFO:teuthology.orchestra.run.smithi114.stdout: "default_value": "", 2024-04-23T10:41:08.804 INFO:teuthology.orchestra.run.smithi114.stdout: "min": "", 2024-04-23T10:41:08.804 INFO:teuthology.orchestra.run.smithi114.stdout: "max": "", 2024-04-23T10:41:08.804 INFO:teuthology.orchestra.run.smithi114.stdout: "enum_allowed": [], 2024-04-23T10:41:08.804 INFO:teuthology.orchestra.run.smithi114.stdout: "desc": "customized SSH config file to connect to managed hosts", 2024-04-23T10:41:08.804 INFO:teuthology.orchestra.run.smithi114.stdout: "long_desc": "", 2024-04-23T10:41:08.805 INFO:teuthology.orchestra.run.smithi114.stdout: "tags": [], 2024-04-23T10:41:08.805 INFO:teuthology.orchestra.run.smithi114.stdout: "see_also": [] 2024-04-23T10:41:08.805 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-23T10:41:08.805 INFO:teuthology.orchestra.run.smithi114.stdout: "use_agent": { 2024-04-23T10:41:08.805 INFO:teuthology.orchestra.run.smithi114.stdout: "name": "use_agent", 2024-04-23T10:41:08.805 INFO:teuthology.orchestra.run.smithi114.stdout: "type": "bool", 2024-04-23T10:41:08.805 INFO:teuthology.orchestra.run.smithi114.stdout: "level": "advanced", 2024-04-23T10:41:08.805 INFO:teuthology.orchestra.run.smithi114.stdout: "flags": 0, 2024-04-23T10:41:08.805 INFO:teuthology.orchestra.run.smithi114.stdout: "default_value": "False", 2024-04-23T10:41:08.805 INFO:teuthology.orchestra.run.smithi114.stdout: "min": "", 2024-04-23T10:41:08.806 INFO:teuthology.orchestra.run.smithi114.stdout: "max": "", 2024-04-23T10:41:08.806 INFO:teuthology.orchestra.run.smithi114.stdout: "enum_allowed": [], 2024-04-23T10:41:08.806 INFO:teuthology.orchestra.run.smithi114.stdout: "desc": "Use cephadm agent on each host to gather and send metadata", 2024-04-23T10:41:08.806 INFO:teuthology.orchestra.run.smithi114.stdout: "long_desc": "", 2024-04-23T10:41:08.806 INFO:teuthology.orchestra.run.smithi114.stdout: "tags": [], 2024-04-23T10:41:08.806 INFO:teuthology.orchestra.run.smithi114.stdout: "see_also": [] 2024-04-23T10:41:08.806 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-23T10:41:08.806 INFO:teuthology.orchestra.run.smithi114.stdout: "use_repo_digest": { 2024-04-23T10:41:08.806 INFO:teuthology.orchestra.run.smithi114.stdout: "name": "use_repo_digest", 2024-04-23T10:41:08.806 INFO:teuthology.orchestra.run.smithi114.stdout: "type": "bool", 2024-04-23T10:41:08.807 INFO:teuthology.orchestra.run.smithi114.stdout: "level": "advanced", 2024-04-23T10:41:08.807 INFO:teuthology.orchestra.run.smithi114.stdout: "flags": 0, 2024-04-23T10:41:08.807 INFO:teuthology.orchestra.run.smithi114.stdout: "default_value": "True", 2024-04-23T10:41:08.807 INFO:teuthology.orchestra.run.smithi114.stdout: "min": "", 2024-04-23T10:41:08.807 INFO:teuthology.orchestra.run.smithi114.stdout: "max": "", 2024-04-23T10:41:08.807 INFO:teuthology.orchestra.run.smithi114.stdout: "enum_allowed": [], 2024-04-23T10:41:08.807 INFO:teuthology.orchestra.run.smithi114.stdout: "desc": "Automatically convert image tags to image digest. Make sure all daemons use the same image", 2024-04-23T10:41:08.807 INFO:teuthology.orchestra.run.smithi114.stdout: "long_desc": "", 2024-04-23T10:41:08.807 INFO:teuthology.orchestra.run.smithi114.stdout: "tags": [], 2024-04-23T10:41:08.808 INFO:teuthology.orchestra.run.smithi114.stdout: "see_also": [] 2024-04-23T10:41:08.808 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-23T10:41:08.808 INFO:teuthology.orchestra.run.smithi114.stdout: "warn_on_failed_host_check": { 2024-04-23T10:41:08.808 INFO:teuthology.orchestra.run.smithi114.stdout: "name": "warn_on_failed_host_check", 2024-04-23T10:41:08.808 INFO:teuthology.orchestra.run.smithi114.stdout: "type": "bool", 2024-04-23T10:41:08.808 INFO:teuthology.orchestra.run.smithi114.stdout: "level": "advanced", 2024-04-23T10:41:08.808 INFO:teuthology.orchestra.run.smithi114.stdout: "flags": 0, 2024-04-23T10:41:08.808 INFO:teuthology.orchestra.run.smithi114.stdout: "default_value": "True", 2024-04-23T10:41:08.808 INFO:teuthology.orchestra.run.smithi114.stdout: "min": "", 2024-04-23T10:41:08.808 INFO:teuthology.orchestra.run.smithi114.stdout: "max": "", 2024-04-23T10:41:08.809 INFO:teuthology.orchestra.run.smithi114.stdout: "enum_allowed": [], 2024-04-23T10:41:08.809 INFO:teuthology.orchestra.run.smithi114.stdout: "desc": "raise a health warning if the host check fails", 2024-04-23T10:41:08.809 INFO:teuthology.orchestra.run.smithi114.stdout: "long_desc": "", 2024-04-23T10:41:08.809 INFO:teuthology.orchestra.run.smithi114.stdout: "tags": [], 2024-04-23T10:41:08.809 INFO:teuthology.orchestra.run.smithi114.stdout: "see_also": [] 2024-04-23T10:41:08.809 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-23T10:41:08.809 INFO:teuthology.orchestra.run.smithi114.stdout: "warn_on_stray_daemons": { 2024-04-23T10:41:08.809 INFO:teuthology.orchestra.run.smithi114.stdout: "name": "warn_on_stray_daemons", 2024-04-23T10:41:08.809 INFO:teuthology.orchestra.run.smithi114.stdout: "type": "bool", 2024-04-23T10:41:08.809 INFO:teuthology.orchestra.run.smithi114.stdout: "level": "advanced", 2024-04-23T10:41:08.810 INFO:teuthology.orchestra.run.smithi114.stdout: "flags": 0, 2024-04-23T10:41:08.810 INFO:teuthology.orchestra.run.smithi114.stdout: "default_value": "True", 2024-04-23T10:41:08.810 INFO:teuthology.orchestra.run.smithi114.stdout: "min": "", 2024-04-23T10:41:08.810 INFO:teuthology.orchestra.run.smithi114.stdout: "max": "", 2024-04-23T10:41:08.810 INFO:teuthology.orchestra.run.smithi114.stdout: "enum_allowed": [], 2024-04-23T10:41:08.810 INFO:teuthology.orchestra.run.smithi114.stdout: "desc": "raise a health warning if daemons are detected that are not managed by cephadm", 2024-04-23T10:41:08.810 INFO:teuthology.orchestra.run.smithi114.stdout: "long_desc": "", 2024-04-23T10:41:08.810 INFO:teuthology.orchestra.run.smithi114.stdout: "tags": [], 2024-04-23T10:41:08.810 INFO:teuthology.orchestra.run.smithi114.stdout: "see_also": [] 2024-04-23T10:41:08.811 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-23T10:41:08.811 INFO:teuthology.orchestra.run.smithi114.stdout: "warn_on_stray_hosts": { 2024-04-23T10:41:08.811 INFO:teuthology.orchestra.run.smithi114.stdout: "name": "warn_on_stray_hosts", 2024-04-23T10:41:08.811 INFO:teuthology.orchestra.run.smithi114.stdout: "type": "bool", 2024-04-23T10:41:08.811 INFO:teuthology.orchestra.run.smithi114.stdout: "level": "advanced", 2024-04-23T10:41:08.811 INFO:teuthology.orchestra.run.smithi114.stdout: "flags": 0, 2024-04-23T10:41:08.811 INFO:teuthology.orchestra.run.smithi114.stdout: "default_value": "True", 2024-04-23T10:41:08.811 INFO:teuthology.orchestra.run.smithi114.stdout: "min": "", 2024-04-23T10:41:08.811 INFO:teuthology.orchestra.run.smithi114.stdout: "max": "", 2024-04-23T10:41:08.811 INFO:teuthology.orchestra.run.smithi114.stdout: "enum_allowed": [], 2024-04-23T10:41:08.812 INFO:teuthology.orchestra.run.smithi114.stdout: "desc": "raise a health warning if daemons are detected on a host that is not managed by cephadm", 2024-04-23T10:41:08.812 INFO:teuthology.orchestra.run.smithi114.stdout: "long_desc": "", 2024-04-23T10:41:08.812 INFO:teuthology.orchestra.run.smithi114.stdout: "tags": [], 2024-04-23T10:41:08.812 INFO:teuthology.orchestra.run.smithi114.stdout: "see_also": [] 2024-04-23T10:41:08.812 INFO:teuthology.orchestra.run.smithi114.stdout: } 2024-04-23T10:41:08.812 INFO:teuthology.orchestra.run.smithi114.stdout: } 2024-04-23T10:41:08.812 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-23T10:41:08.812 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-04-23T10:41:08.812 INFO:teuthology.orchestra.run.smithi114.stdout: "name": "crash", 2024-04-23T10:41:08.813 INFO:teuthology.orchestra.run.smithi114.stdout: "can_run": true, 2024-04-23T10:41:08.813 INFO:teuthology.orchestra.run.smithi114.stdout: "error_string": "", 2024-04-23T10:41:08.813 INFO:teuthology.orchestra.run.smithi114.stdout: "module_options": { 2024-04-23T10:41:08.813 INFO:teuthology.orchestra.run.smithi114.stdout: "log_level": { 2024-04-23T10:41:08.813 INFO:teuthology.orchestra.run.smithi114.stdout: "name": "log_level", 2024-04-23T10:41:08.813 INFO:teuthology.orchestra.run.smithi114.stdout: "type": "str", 2024-04-23T10:41:08.813 INFO:teuthology.orchestra.run.smithi114.stdout: "level": "advanced", 2024-04-23T10:41:08.813 INFO:teuthology.orchestra.run.smithi114.stdout: "flags": 1, 2024-04-23T10:41:08.813 INFO:teuthology.orchestra.run.smithi114.stdout: "default_value": "", 2024-04-23T10:41:08.813 INFO:teuthology.orchestra.run.smithi114.stdout: "min": "", 2024-04-23T10:41:08.814 INFO:teuthology.orchestra.run.smithi114.stdout: "max": "", 2024-04-23T10:41:08.814 INFO:teuthology.orchestra.run.smithi114.stdout: "enum_allowed": [ 2024-04-23T10:41:08.814 INFO:teuthology.orchestra.run.smithi114.stdout: "", 2024-04-23T10:41:08.814 INFO:teuthology.orchestra.run.smithi114.stdout: "critical", 2024-04-23T10:41:08.814 INFO:teuthology.orchestra.run.smithi114.stdout: "debug", 2024-04-23T10:41:08.814 INFO:teuthology.orchestra.run.smithi114.stdout: "error", 2024-04-23T10:41:08.814 INFO:teuthology.orchestra.run.smithi114.stdout: "info", 2024-04-23T10:41:08.814 INFO:teuthology.orchestra.run.smithi114.stdout: "warning" 2024-04-23T10:41:08.814 INFO:teuthology.orchestra.run.smithi114.stdout: ], 2024-04-23T10:41:08.814 INFO:teuthology.orchestra.run.smithi114.stdout: "desc": "", 2024-04-23T10:41:08.815 INFO:teuthology.orchestra.run.smithi114.stdout: "long_desc": "", 2024-04-23T10:41:08.815 INFO:teuthology.orchestra.run.smithi114.stdout: "tags": [], 2024-04-23T10:41:08.815 INFO:teuthology.orchestra.run.smithi114.stdout: "see_also": [] 2024-04-23T10:41:08.815 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-23T10:41:08.815 INFO:teuthology.orchestra.run.smithi114.stdout: "log_to_cluster": { 2024-04-23T10:41:08.815 INFO:teuthology.orchestra.run.smithi114.stdout: "name": "log_to_cluster", 2024-04-23T10:41:08.815 INFO:teuthology.orchestra.run.smithi114.stdout: "type": "bool", 2024-04-23T10:41:08.815 INFO:teuthology.orchestra.run.smithi114.stdout: "level": "advanced", 2024-04-23T10:41:08.815 INFO:teuthology.orchestra.run.smithi114.stdout: "flags": 1, 2024-04-23T10:41:08.815 INFO:teuthology.orchestra.run.smithi114.stdout: "default_value": "False", 2024-04-23T10:41:08.816 INFO:teuthology.orchestra.run.smithi114.stdout: "min": "", 2024-04-23T10:41:08.816 INFO:teuthology.orchestra.run.smithi114.stdout: "max": "", 2024-04-23T10:41:08.816 INFO:teuthology.orchestra.run.smithi114.stdout: "enum_allowed": [], 2024-04-23T10:41:08.816 INFO:teuthology.orchestra.run.smithi114.stdout: "desc": "", 2024-04-23T10:41:08.816 INFO:teuthology.orchestra.run.smithi114.stdout: "long_desc": "", 2024-04-23T10:41:08.816 INFO:teuthology.orchestra.run.smithi114.stdout: "tags": [], 2024-04-23T10:41:08.816 INFO:teuthology.orchestra.run.smithi114.stdout: "see_also": [] 2024-04-23T10:41:08.816 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-23T10:41:08.816 INFO:teuthology.orchestra.run.smithi114.stdout: "log_to_cluster_level": { 2024-04-23T10:41:08.816 INFO:teuthology.orchestra.run.smithi114.stdout: "name": "log_to_cluster_level", 2024-04-23T10:41:08.817 INFO:teuthology.orchestra.run.smithi114.stdout: "type": "str", 2024-04-23T10:41:08.817 INFO:teuthology.orchestra.run.smithi114.stdout: "level": "advanced", 2024-04-23T10:41:08.817 INFO:teuthology.orchestra.run.smithi114.stdout: "flags": 1, 2024-04-23T10:41:08.817 INFO:teuthology.orchestra.run.smithi114.stdout: "default_value": "info", 2024-04-23T10:41:08.817 INFO:teuthology.orchestra.run.smithi114.stdout: "min": "", 2024-04-23T10:41:08.817 INFO:teuthology.orchestra.run.smithi114.stdout: "max": "", 2024-04-23T10:41:08.817 INFO:teuthology.orchestra.run.smithi114.stdout: "enum_allowed": [ 2024-04-23T10:41:08.817 INFO:teuthology.orchestra.run.smithi114.stdout: "", 2024-04-23T10:41:08.817 INFO:teuthology.orchestra.run.smithi114.stdout: "critical", 2024-04-23T10:41:08.817 INFO:teuthology.orchestra.run.smithi114.stdout: "debug", 2024-04-23T10:41:08.818 INFO:teuthology.orchestra.run.smithi114.stdout: "error", 2024-04-23T10:41:08.818 INFO:teuthology.orchestra.run.smithi114.stdout: "info", 2024-04-23T10:41:08.818 INFO:teuthology.orchestra.run.smithi114.stdout: "warning" 2024-04-23T10:41:08.818 INFO:teuthology.orchestra.run.smithi114.stdout: ], 2024-04-23T10:41:08.818 INFO:teuthology.orchestra.run.smithi114.stdout: "desc": "", 2024-04-23T10:41:08.818 INFO:teuthology.orchestra.run.smithi114.stdout: "long_desc": "", 2024-04-23T10:41:08.818 INFO:teuthology.orchestra.run.smithi114.stdout: "tags": [], 2024-04-23T10:41:08.818 INFO:teuthology.orchestra.run.smithi114.stdout: "see_also": [] 2024-04-23T10:41:08.818 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-23T10:41:08.818 INFO:teuthology.orchestra.run.smithi114.stdout: "log_to_file": { 2024-04-23T10:41:08.819 INFO:teuthology.orchestra.run.smithi114.stdout: "name": "log_to_file", 2024-04-23T10:41:08.819 INFO:teuthology.orchestra.run.smithi114.stdout: "type": "bool", 2024-04-23T10:41:08.819 INFO:teuthology.orchestra.run.smithi114.stdout: "level": "advanced", 2024-04-23T10:41:08.819 INFO:teuthology.orchestra.run.smithi114.stdout: "flags": 1, 2024-04-23T10:41:08.819 INFO:teuthology.orchestra.run.smithi114.stdout: "default_value": "False", 2024-04-23T10:41:08.819 INFO:teuthology.orchestra.run.smithi114.stdout: "min": "", 2024-04-23T10:41:08.819 INFO:teuthology.orchestra.run.smithi114.stdout: "max": "", 2024-04-23T10:41:08.819 INFO:teuthology.orchestra.run.smithi114.stdout: "enum_allowed": [], 2024-04-23T10:41:08.819 INFO:teuthology.orchestra.run.smithi114.stdout: "desc": "", 2024-04-23T10:41:08.820 INFO:teuthology.orchestra.run.smithi114.stdout: "long_desc": "", 2024-04-23T10:41:08.820 INFO:teuthology.orchestra.run.smithi114.stdout: "tags": [], 2024-04-23T10:41:08.820 INFO:teuthology.orchestra.run.smithi114.stdout: "see_also": [] 2024-04-23T10:41:08.820 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-23T10:41:08.820 INFO:teuthology.orchestra.run.smithi114.stdout: "retain_interval": { 2024-04-23T10:41:08.820 INFO:teuthology.orchestra.run.smithi114.stdout: "name": "retain_interval", 2024-04-23T10:41:08.820 INFO:teuthology.orchestra.run.smithi114.stdout: "type": "secs", 2024-04-23T10:41:08.820 INFO:teuthology.orchestra.run.smithi114.stdout: "level": "advanced", 2024-04-23T10:41:08.820 INFO:teuthology.orchestra.run.smithi114.stdout: "flags": 1, 2024-04-23T10:41:08.820 INFO:teuthology.orchestra.run.smithi114.stdout: "default_value": "31536000", 2024-04-23T10:41:08.821 INFO:teuthology.orchestra.run.smithi114.stdout: "min": "", 2024-04-23T10:41:08.821 INFO:teuthology.orchestra.run.smithi114.stdout: "max": "", 2024-04-23T10:41:08.821 INFO:teuthology.orchestra.run.smithi114.stdout: "enum_allowed": [], 2024-04-23T10:41:08.821 INFO:teuthology.orchestra.run.smithi114.stdout: "desc": "how long to retain crashes before pruning them", 2024-04-23T10:41:08.821 INFO:teuthology.orchestra.run.smithi114.stdout: "long_desc": "", 2024-04-23T10:41:08.821 INFO:teuthology.orchestra.run.smithi114.stdout: "tags": [], 2024-04-23T10:41:08.821 INFO:teuthology.orchestra.run.smithi114.stdout: "see_also": [] 2024-04-23T10:41:08.821 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-23T10:41:08.821 INFO:teuthology.orchestra.run.smithi114.stdout: "warn_recent_interval": { 2024-04-23T10:41:08.821 INFO:teuthology.orchestra.run.smithi114.stdout: "name": "warn_recent_interval", 2024-04-23T10:41:08.822 INFO:teuthology.orchestra.run.smithi114.stdout: "type": "secs", 2024-04-23T10:41:08.822 INFO:teuthology.orchestra.run.smithi114.stdout: "level": "advanced", 2024-04-23T10:41:08.822 INFO:teuthology.orchestra.run.smithi114.stdout: "flags": 1, 2024-04-23T10:41:08.822 INFO:teuthology.orchestra.run.smithi114.stdout: "default_value": "1209600", 2024-04-23T10:41:08.822 INFO:teuthology.orchestra.run.smithi114.stdout: "min": "", 2024-04-23T10:41:08.822 INFO:teuthology.orchestra.run.smithi114.stdout: "max": "", 2024-04-23T10:41:08.822 INFO:teuthology.orchestra.run.smithi114.stdout: "enum_allowed": [], 2024-04-23T10:41:08.822 INFO:teuthology.orchestra.run.smithi114.stdout: "desc": "time interval in which to warn about recent crashes", 2024-04-23T10:41:08.822 INFO:teuthology.orchestra.run.smithi114.stdout: "long_desc": "", 2024-04-23T10:41:08.822 INFO:teuthology.orchestra.run.smithi114.stdout: "tags": [], 2024-04-23T10:41:08.823 INFO:teuthology.orchestra.run.smithi114.stdout: "see_also": [] 2024-04-23T10:41:08.823 INFO:teuthology.orchestra.run.smithi114.stdout: } 2024-04-23T10:41:08.823 INFO:teuthology.orchestra.run.smithi114.stdout: } 2024-04-23T10:41:08.823 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-23T10:41:08.823 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-04-23T10:41:08.823 INFO:teuthology.orchestra.run.smithi114.stdout: "name": "dashboard", 2024-04-23T10:41:08.823 INFO:teuthology.orchestra.run.smithi114.stdout: "can_run": true, 2024-04-23T10:41:08.823 INFO:teuthology.orchestra.run.smithi114.stdout: "error_string": "", 2024-04-23T10:41:08.823 INFO:teuthology.orchestra.run.smithi114.stdout: "module_options": { 2024-04-23T10:41:08.824 INFO:teuthology.orchestra.run.smithi114.stdout: "ACCOUNT_LOCKOUT_ATTEMPTS": { 2024-04-23T10:41:08.824 INFO:teuthology.orchestra.run.smithi114.stdout: "name": "ACCOUNT_LOCKOUT_ATTEMPTS", 2024-04-23T10:41:08.824 INFO:teuthology.orchestra.run.smithi114.stdout: "type": "int", 2024-04-23T10:41:08.824 INFO:teuthology.orchestra.run.smithi114.stdout: "level": "advanced", 2024-04-23T10:41:08.824 INFO:teuthology.orchestra.run.smithi114.stdout: "flags": 0, 2024-04-23T10:41:08.824 INFO:teuthology.orchestra.run.smithi114.stdout: "default_value": "10", 2024-04-23T10:41:08.824 INFO:teuthology.orchestra.run.smithi114.stdout: "min": "", 2024-04-23T10:41:08.824 INFO:teuthology.orchestra.run.smithi114.stdout: "max": "", 2024-04-23T10:41:08.824 INFO:teuthology.orchestra.run.smithi114.stdout: "enum_allowed": [], 2024-04-23T10:41:08.824 INFO:teuthology.orchestra.run.smithi114.stdout: "desc": "", 2024-04-23T10:41:08.825 INFO:teuthology.orchestra.run.smithi114.stdout: "long_desc": "", 2024-04-23T10:41:08.825 INFO:teuthology.orchestra.run.smithi114.stdout: "tags": [], 2024-04-23T10:41:08.825 INFO:teuthology.orchestra.run.smithi114.stdout: "see_also": [] 2024-04-23T10:41:08.825 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-23T10:41:08.825 INFO:teuthology.orchestra.run.smithi114.stdout: "ALERTMANAGER_API_HOST": { 2024-04-23T10:41:08.825 INFO:teuthology.orchestra.run.smithi114.stdout: "name": "ALERTMANAGER_API_HOST", 2024-04-23T10:41:08.825 INFO:teuthology.orchestra.run.smithi114.stdout: "type": "str", 2024-04-23T10:41:08.825 INFO:teuthology.orchestra.run.smithi114.stdout: "level": "advanced", 2024-04-23T10:41:08.825 INFO:teuthology.orchestra.run.smithi114.stdout: "flags": 0, 2024-04-23T10:41:08.825 INFO:teuthology.orchestra.run.smithi114.stdout: "default_value": "", 2024-04-23T10:41:08.826 INFO:teuthology.orchestra.run.smithi114.stdout: "min": "", 2024-04-23T10:41:08.826 INFO:teuthology.orchestra.run.smithi114.stdout: "max": "", 2024-04-23T10:41:08.826 INFO:teuthology.orchestra.run.smithi114.stdout: "enum_allowed": [], 2024-04-23T10:41:08.826 INFO:teuthology.orchestra.run.smithi114.stdout: "desc": "", 2024-04-23T10:41:08.826 INFO:teuthology.orchestra.run.smithi114.stdout: "long_desc": "", 2024-04-23T10:41:08.826 INFO:teuthology.orchestra.run.smithi114.stdout: "tags": [], 2024-04-23T10:41:08.826 INFO:teuthology.orchestra.run.smithi114.stdout: "see_also": [] 2024-04-23T10:41:08.826 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-23T10:41:08.826 INFO:teuthology.orchestra.run.smithi114.stdout: "ALERTMANAGER_API_SSL_VERIFY": { 2024-04-23T10:41:08.826 INFO:teuthology.orchestra.run.smithi114.stdout: "name": "ALERTMANAGER_API_SSL_VERIFY", 2024-04-23T10:41:08.827 INFO:teuthology.orchestra.run.smithi114.stdout: "type": "bool", 2024-04-23T10:41:08.827 INFO:teuthology.orchestra.run.smithi114.stdout: "level": "advanced", 2024-04-23T10:41:08.827 INFO:teuthology.orchestra.run.smithi114.stdout: "flags": 0, 2024-04-23T10:41:08.827 INFO:teuthology.orchestra.run.smithi114.stdout: "default_value": "True", 2024-04-23T10:41:08.827 INFO:teuthology.orchestra.run.smithi114.stdout: "min": "", 2024-04-23T10:41:08.827 INFO:teuthology.orchestra.run.smithi114.stdout: "max": "", 2024-04-23T10:41:08.827 INFO:teuthology.orchestra.run.smithi114.stdout: "enum_allowed": [], 2024-04-23T10:41:08.827 INFO:teuthology.orchestra.run.smithi114.stdout: "desc": "", 2024-04-23T10:41:08.827 INFO:teuthology.orchestra.run.smithi114.stdout: "long_desc": "", 2024-04-23T10:41:08.828 INFO:teuthology.orchestra.run.smithi114.stdout: "tags": [], 2024-04-23T10:41:08.828 INFO:teuthology.orchestra.run.smithi114.stdout: "see_also": [] 2024-04-23T10:41:08.828 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-23T10:41:08.828 INFO:teuthology.orchestra.run.smithi114.stdout: "AUDIT_API_ENABLED": { 2024-04-23T10:41:08.828 INFO:teuthology.orchestra.run.smithi114.stdout: "name": "AUDIT_API_ENABLED", 2024-04-23T10:41:08.828 INFO:teuthology.orchestra.run.smithi114.stdout: "type": "bool", 2024-04-23T10:41:08.828 INFO:teuthology.orchestra.run.smithi114.stdout: "level": "advanced", 2024-04-23T10:41:08.828 INFO:teuthology.orchestra.run.smithi114.stdout: "flags": 0, 2024-04-23T10:41:08.828 INFO:teuthology.orchestra.run.smithi114.stdout: "default_value": "False", 2024-04-23T10:41:08.828 INFO:teuthology.orchestra.run.smithi114.stdout: "min": "", 2024-04-23T10:41:08.829 INFO:teuthology.orchestra.run.smithi114.stdout: "max": "", 2024-04-23T10:41:08.829 INFO:teuthology.orchestra.run.smithi114.stdout: "enum_allowed": [], 2024-04-23T10:41:08.829 INFO:teuthology.orchestra.run.smithi114.stdout: "desc": "", 2024-04-23T10:41:08.829 INFO:teuthology.orchestra.run.smithi114.stdout: "long_desc": "", 2024-04-23T10:41:08.829 INFO:teuthology.orchestra.run.smithi114.stdout: "tags": [], 2024-04-23T10:41:08.829 INFO:teuthology.orchestra.run.smithi114.stdout: "see_also": [] 2024-04-23T10:41:08.829 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-23T10:41:08.829 INFO:teuthology.orchestra.run.smithi114.stdout: "AUDIT_API_LOG_PAYLOAD": { 2024-04-23T10:41:08.829 INFO:teuthology.orchestra.run.smithi114.stdout: "name": "AUDIT_API_LOG_PAYLOAD", 2024-04-23T10:41:08.829 INFO:teuthology.orchestra.run.smithi114.stdout: "type": "bool", 2024-04-23T10:41:08.830 INFO:teuthology.orchestra.run.smithi114.stdout: "level": "advanced", 2024-04-23T10:41:08.830 INFO:teuthology.orchestra.run.smithi114.stdout: "flags": 0, 2024-04-23T10:41:08.830 INFO:teuthology.orchestra.run.smithi114.stdout: "default_value": "True", 2024-04-23T10:41:08.830 INFO:teuthology.orchestra.run.smithi114.stdout: "min": "", 2024-04-23T10:41:08.830 INFO:teuthology.orchestra.run.smithi114.stdout: "max": "", 2024-04-23T10:41:08.830 INFO:teuthology.orchestra.run.smithi114.stdout: "enum_allowed": [], 2024-04-23T10:41:08.830 INFO:teuthology.orchestra.run.smithi114.stdout: "desc": "", 2024-04-23T10:41:08.830 INFO:teuthology.orchestra.run.smithi114.stdout: "long_desc": "", 2024-04-23T10:41:08.830 INFO:teuthology.orchestra.run.smithi114.stdout: "tags": [], 2024-04-23T10:41:08.831 INFO:teuthology.orchestra.run.smithi114.stdout: "see_also": [] 2024-04-23T10:41:08.831 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-23T10:41:08.831 INFO:teuthology.orchestra.run.smithi114.stdout: "ENABLE_BROWSABLE_API": { 2024-04-23T10:41:08.831 INFO:teuthology.orchestra.run.smithi114.stdout: "name": "ENABLE_BROWSABLE_API", 2024-04-23T10:41:08.831 INFO:teuthology.orchestra.run.smithi114.stdout: "type": "bool", 2024-04-23T10:41:08.831 INFO:teuthology.orchestra.run.smithi114.stdout: "level": "advanced", 2024-04-23T10:41:08.831 INFO:teuthology.orchestra.run.smithi114.stdout: "flags": 0, 2024-04-23T10:41:08.831 INFO:teuthology.orchestra.run.smithi114.stdout: "default_value": "True", 2024-04-23T10:41:08.831 INFO:teuthology.orchestra.run.smithi114.stdout: "min": "", 2024-04-23T10:41:08.831 INFO:teuthology.orchestra.run.smithi114.stdout: "max": "", 2024-04-23T10:41:08.832 INFO:teuthology.orchestra.run.smithi114.stdout: "enum_allowed": [], 2024-04-23T10:41:08.832 INFO:teuthology.orchestra.run.smithi114.stdout: "desc": "", 2024-04-23T10:41:08.832 INFO:teuthology.orchestra.run.smithi114.stdout: "long_desc": "", 2024-04-23T10:41:08.832 INFO:teuthology.orchestra.run.smithi114.stdout: "tags": [], 2024-04-23T10:41:08.832 INFO:teuthology.orchestra.run.smithi114.stdout: "see_also": [] 2024-04-23T10:41:08.832 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-23T10:41:08.832 INFO:teuthology.orchestra.run.smithi114.stdout: "FEATURE_TOGGLE_CEPHFS": { 2024-04-23T10:41:08.832 INFO:teuthology.orchestra.run.smithi114.stdout: "name": "FEATURE_TOGGLE_CEPHFS", 2024-04-23T10:41:08.832 INFO:teuthology.orchestra.run.smithi114.stdout: "type": "bool", 2024-04-23T10:41:08.832 INFO:teuthology.orchestra.run.smithi114.stdout: "level": "advanced", 2024-04-23T10:41:08.833 INFO:teuthology.orchestra.run.smithi114.stdout: "flags": 0, 2024-04-23T10:41:08.833 INFO:teuthology.orchestra.run.smithi114.stdout: "default_value": "True", 2024-04-23T10:41:08.833 INFO:teuthology.orchestra.run.smithi114.stdout: "min": "", 2024-04-23T10:41:08.833 INFO:teuthology.orchestra.run.smithi114.stdout: "max": "", 2024-04-23T10:41:08.833 INFO:teuthology.orchestra.run.smithi114.stdout: "enum_allowed": [], 2024-04-23T10:41:08.833 INFO:teuthology.orchestra.run.smithi114.stdout: "desc": "", 2024-04-23T10:41:08.833 INFO:teuthology.orchestra.run.smithi114.stdout: "long_desc": "", 2024-04-23T10:41:08.833 INFO:teuthology.orchestra.run.smithi114.stdout: "tags": [], 2024-04-23T10:41:08.833 INFO:teuthology.orchestra.run.smithi114.stdout: "see_also": [] 2024-04-23T10:41:08.833 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-23T10:41:08.834 INFO:teuthology.orchestra.run.smithi114.stdout: "FEATURE_TOGGLE_DASHBOARD": { 2024-04-23T10:41:08.834 INFO:teuthology.orchestra.run.smithi114.stdout: "name": "FEATURE_TOGGLE_DASHBOARD", 2024-04-23T10:41:08.834 INFO:teuthology.orchestra.run.smithi114.stdout: "type": "bool", 2024-04-23T10:41:08.834 INFO:teuthology.orchestra.run.smithi114.stdout: "level": "advanced", 2024-04-23T10:41:08.834 INFO:teuthology.orchestra.run.smithi114.stdout: "flags": 0, 2024-04-23T10:41:08.834 INFO:teuthology.orchestra.run.smithi114.stdout: "default_value": "True", 2024-04-23T10:41:08.834 INFO:teuthology.orchestra.run.smithi114.stdout: "min": "", 2024-04-23T10:41:08.834 INFO:teuthology.orchestra.run.smithi114.stdout: "max": "", 2024-04-23T10:41:08.834 INFO:teuthology.orchestra.run.smithi114.stdout: "enum_allowed": [], 2024-04-23T10:41:08.835 INFO:teuthology.orchestra.run.smithi114.stdout: "desc": "", 2024-04-23T10:41:08.835 INFO:teuthology.orchestra.run.smithi114.stdout: "long_desc": "", 2024-04-23T10:41:08.835 INFO:teuthology.orchestra.run.smithi114.stdout: "tags": [], 2024-04-23T10:41:08.835 INFO:teuthology.orchestra.run.smithi114.stdout: "see_also": [] 2024-04-23T10:41:08.835 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-23T10:41:08.835 INFO:teuthology.orchestra.run.smithi114.stdout: "FEATURE_TOGGLE_ISCSI": { 2024-04-23T10:41:08.835 INFO:teuthology.orchestra.run.smithi114.stdout: "name": "FEATURE_TOGGLE_ISCSI", 2024-04-23T10:41:08.835 INFO:teuthology.orchestra.run.smithi114.stdout: "type": "bool", 2024-04-23T10:41:08.835 INFO:teuthology.orchestra.run.smithi114.stdout: "level": "advanced", 2024-04-23T10:41:08.835 INFO:teuthology.orchestra.run.smithi114.stdout: "flags": 0, 2024-04-23T10:41:08.836 INFO:teuthology.orchestra.run.smithi114.stdout: "default_value": "True", 2024-04-23T10:41:08.836 INFO:teuthology.orchestra.run.smithi114.stdout: "min": "", 2024-04-23T10:41:08.836 INFO:teuthology.orchestra.run.smithi114.stdout: "max": "", 2024-04-23T10:41:08.836 INFO:teuthology.orchestra.run.smithi114.stdout: "enum_allowed": [], 2024-04-23T10:41:08.836 INFO:teuthology.orchestra.run.smithi114.stdout: "desc": "", 2024-04-23T10:41:08.836 INFO:teuthology.orchestra.run.smithi114.stdout: "long_desc": "", 2024-04-23T10:41:08.836 INFO:teuthology.orchestra.run.smithi114.stdout: "tags": [], 2024-04-23T10:41:08.836 INFO:teuthology.orchestra.run.smithi114.stdout: "see_also": [] 2024-04-23T10:41:08.836 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-23T10:41:08.836 INFO:teuthology.orchestra.run.smithi114.stdout: "FEATURE_TOGGLE_MIRRORING": { 2024-04-23T10:41:08.837 INFO:teuthology.orchestra.run.smithi114.stdout: "name": "FEATURE_TOGGLE_MIRRORING", 2024-04-23T10:41:08.837 INFO:teuthology.orchestra.run.smithi114.stdout: "type": "bool", 2024-04-23T10:41:08.837 INFO:teuthology.orchestra.run.smithi114.stdout: "level": "advanced", 2024-04-23T10:41:08.837 INFO:teuthology.orchestra.run.smithi114.stdout: "flags": 0, 2024-04-23T10:41:08.837 INFO:teuthology.orchestra.run.smithi114.stdout: "default_value": "True", 2024-04-23T10:41:08.837 INFO:teuthology.orchestra.run.smithi114.stdout: "min": "", 2024-04-23T10:41:08.837 INFO:teuthology.orchestra.run.smithi114.stdout: "max": "", 2024-04-23T10:41:08.837 INFO:teuthology.orchestra.run.smithi114.stdout: "enum_allowed": [], 2024-04-23T10:41:08.837 INFO:teuthology.orchestra.run.smithi114.stdout: "desc": "", 2024-04-23T10:41:08.837 INFO:teuthology.orchestra.run.smithi114.stdout: "long_desc": "", 2024-04-23T10:41:08.838 INFO:teuthology.orchestra.run.smithi114.stdout: "tags": [], 2024-04-23T10:41:08.838 INFO:teuthology.orchestra.run.smithi114.stdout: "see_also": [] 2024-04-23T10:41:08.838 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-23T10:41:08.838 INFO:teuthology.orchestra.run.smithi114.stdout: "FEATURE_TOGGLE_NFS": { 2024-04-23T10:41:08.838 INFO:teuthology.orchestra.run.smithi114.stdout: "name": "FEATURE_TOGGLE_NFS", 2024-04-23T10:41:08.838 INFO:teuthology.orchestra.run.smithi114.stdout: "type": "bool", 2024-04-23T10:41:08.838 INFO:teuthology.orchestra.run.smithi114.stdout: "level": "advanced", 2024-04-23T10:41:08.838 INFO:teuthology.orchestra.run.smithi114.stdout: "flags": 0, 2024-04-23T10:41:08.838 INFO:teuthology.orchestra.run.smithi114.stdout: "default_value": "True", 2024-04-23T10:41:08.838 INFO:teuthology.orchestra.run.smithi114.stdout: "min": "", 2024-04-23T10:41:08.839 INFO:teuthology.orchestra.run.smithi114.stdout: "max": "", 2024-04-23T10:41:08.839 INFO:teuthology.orchestra.run.smithi114.stdout: "enum_allowed": [], 2024-04-23T10:41:08.839 INFO:teuthology.orchestra.run.smithi114.stdout: "desc": "", 2024-04-23T10:41:08.839 INFO:teuthology.orchestra.run.smithi114.stdout: "long_desc": "", 2024-04-23T10:41:08.839 INFO:teuthology.orchestra.run.smithi114.stdout: "tags": [], 2024-04-23T10:41:08.839 INFO:teuthology.orchestra.run.smithi114.stdout: "see_also": [] 2024-04-23T10:41:08.839 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-23T10:41:08.839 INFO:teuthology.orchestra.run.smithi114.stdout: "FEATURE_TOGGLE_RBD": { 2024-04-23T10:41:08.839 INFO:teuthology.orchestra.run.smithi114.stdout: "name": "FEATURE_TOGGLE_RBD", 2024-04-23T10:41:08.839 INFO:teuthology.orchestra.run.smithi114.stdout: "type": "bool", 2024-04-23T10:41:08.840 INFO:teuthology.orchestra.run.smithi114.stdout: "level": "advanced", 2024-04-23T10:41:08.840 INFO:teuthology.orchestra.run.smithi114.stdout: "flags": 0, 2024-04-23T10:41:08.840 INFO:teuthology.orchestra.run.smithi114.stdout: "default_value": "True", 2024-04-23T10:41:08.840 INFO:teuthology.orchestra.run.smithi114.stdout: "min": "", 2024-04-23T10:41:08.840 INFO:teuthology.orchestra.run.smithi114.stdout: "max": "", 2024-04-23T10:41:08.840 INFO:teuthology.orchestra.run.smithi114.stdout: "enum_allowed": [], 2024-04-23T10:41:08.840 INFO:teuthology.orchestra.run.smithi114.stdout: "desc": "", 2024-04-23T10:41:08.840 INFO:teuthology.orchestra.run.smithi114.stdout: "long_desc": "", 2024-04-23T10:41:08.840 INFO:teuthology.orchestra.run.smithi114.stdout: "tags": [], 2024-04-23T10:41:08.840 INFO:teuthology.orchestra.run.smithi114.stdout: "see_also": [] 2024-04-23T10:41:08.841 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-23T10:41:08.841 INFO:teuthology.orchestra.run.smithi114.stdout: "FEATURE_TOGGLE_RGW": { 2024-04-23T10:41:08.841 INFO:teuthology.orchestra.run.smithi114.stdout: "name": "FEATURE_TOGGLE_RGW", 2024-04-23T10:41:08.841 INFO:teuthology.orchestra.run.smithi114.stdout: "type": "bool", 2024-04-23T10:41:08.841 INFO:teuthology.orchestra.run.smithi114.stdout: "level": "advanced", 2024-04-23T10:41:08.841 INFO:teuthology.orchestra.run.smithi114.stdout: "flags": 0, 2024-04-23T10:41:08.841 INFO:teuthology.orchestra.run.smithi114.stdout: "default_value": "True", 2024-04-23T10:41:08.841 INFO:teuthology.orchestra.run.smithi114.stdout: "min": "", 2024-04-23T10:41:08.841 INFO:teuthology.orchestra.run.smithi114.stdout: "max": "", 2024-04-23T10:41:08.842 INFO:teuthology.orchestra.run.smithi114.stdout: "enum_allowed": [], 2024-04-23T10:41:08.842 INFO:teuthology.orchestra.run.smithi114.stdout: "desc": "", 2024-04-23T10:41:08.842 INFO:teuthology.orchestra.run.smithi114.stdout: "long_desc": "", 2024-04-23T10:41:08.842 INFO:teuthology.orchestra.run.smithi114.stdout: "tags": [], 2024-04-23T10:41:08.842 INFO:teuthology.orchestra.run.smithi114.stdout: "see_also": [] 2024-04-23T10:41:08.842 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-23T10:41:08.842 INFO:teuthology.orchestra.run.smithi114.stdout: "GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE": { 2024-04-23T10:41:08.842 INFO:teuthology.orchestra.run.smithi114.stdout: "name": "GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE", 2024-04-23T10:41:08.842 INFO:teuthology.orchestra.run.smithi114.stdout: "type": "str", 2024-04-23T10:41:08.842 INFO:teuthology.orchestra.run.smithi114.stdout: "level": "advanced", 2024-04-23T10:41:08.843 INFO:teuthology.orchestra.run.smithi114.stdout: "flags": 0, 2024-04-23T10:41:08.843 INFO:teuthology.orchestra.run.smithi114.stdout: "default_value": "", 2024-04-23T10:41:08.843 INFO:teuthology.orchestra.run.smithi114.stdout: "min": "", 2024-04-23T10:41:08.843 INFO:teuthology.orchestra.run.smithi114.stdout: "max": "", 2024-04-23T10:41:08.843 INFO:teuthology.orchestra.run.smithi114.stdout: "enum_allowed": [], 2024-04-23T10:41:08.843 INFO:teuthology.orchestra.run.smithi114.stdout: "desc": "", 2024-04-23T10:41:08.843 INFO:teuthology.orchestra.run.smithi114.stdout: "long_desc": "", 2024-04-23T10:41:08.843 INFO:teuthology.orchestra.run.smithi114.stdout: "tags": [], 2024-04-23T10:41:08.843 INFO:teuthology.orchestra.run.smithi114.stdout: "see_also": [] 2024-04-23T10:41:08.843 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-23T10:41:08.844 INFO:teuthology.orchestra.run.smithi114.stdout: "GRAFANA_API_PASSWORD": { 2024-04-23T10:41:08.844 INFO:teuthology.orchestra.run.smithi114.stdout: "name": "GRAFANA_API_PASSWORD", 2024-04-23T10:41:08.844 INFO:teuthology.orchestra.run.smithi114.stdout: "type": "str", 2024-04-23T10:41:08.844 INFO:teuthology.orchestra.run.smithi114.stdout: "level": "advanced", 2024-04-23T10:41:08.844 INFO:teuthology.orchestra.run.smithi114.stdout: "flags": 0, 2024-04-23T10:41:08.844 INFO:teuthology.orchestra.run.smithi114.stdout: "default_value": "admin", 2024-04-23T10:41:08.844 INFO:teuthology.orchestra.run.smithi114.stdout: "min": "", 2024-04-23T10:41:08.844 INFO:teuthology.orchestra.run.smithi114.stdout: "max": "", 2024-04-23T10:41:08.844 INFO:teuthology.orchestra.run.smithi114.stdout: "enum_allowed": [], 2024-04-23T10:41:08.845 INFO:teuthology.orchestra.run.smithi114.stdout: "desc": "", 2024-04-23T10:41:08.845 INFO:teuthology.orchestra.run.smithi114.stdout: "long_desc": "", 2024-04-23T10:41:08.845 INFO:teuthology.orchestra.run.smithi114.stdout: "tags": [], 2024-04-23T10:41:08.845 INFO:teuthology.orchestra.run.smithi114.stdout: "see_also": [] 2024-04-23T10:41:08.845 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-23T10:41:08.845 INFO:teuthology.orchestra.run.smithi114.stdout: "GRAFANA_API_SSL_VERIFY": { 2024-04-23T10:41:08.845 INFO:teuthology.orchestra.run.smithi114.stdout: "name": "GRAFANA_API_SSL_VERIFY", 2024-04-23T10:41:08.845 INFO:teuthology.orchestra.run.smithi114.stdout: "type": "bool", 2024-04-23T10:41:08.845 INFO:teuthology.orchestra.run.smithi114.stdout: "level": "advanced", 2024-04-23T10:41:08.845 INFO:teuthology.orchestra.run.smithi114.stdout: "flags": 0, 2024-04-23T10:41:08.846 INFO:teuthology.orchestra.run.smithi114.stdout: "default_value": "True", 2024-04-23T10:41:08.846 INFO:teuthology.orchestra.run.smithi114.stdout: "min": "", 2024-04-23T10:41:08.846 INFO:teuthology.orchestra.run.smithi114.stdout: "max": "", 2024-04-23T10:41:08.846 INFO:teuthology.orchestra.run.smithi114.stdout: "enum_allowed": [], 2024-04-23T10:41:08.846 INFO:teuthology.orchestra.run.smithi114.stdout: "desc": "", 2024-04-23T10:41:08.846 INFO:teuthology.orchestra.run.smithi114.stdout: "long_desc": "", 2024-04-23T10:41:08.846 INFO:teuthology.orchestra.run.smithi114.stdout: "tags": [], 2024-04-23T10:41:08.846 INFO:teuthology.orchestra.run.smithi114.stdout: "see_also": [] 2024-04-23T10:41:08.846 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-23T10:41:08.847 INFO:teuthology.orchestra.run.smithi114.stdout: "GRAFANA_API_URL": { 2024-04-23T10:41:08.847 INFO:teuthology.orchestra.run.smithi114.stdout: "name": "GRAFANA_API_URL", 2024-04-23T10:41:08.847 INFO:teuthology.orchestra.run.smithi114.stdout: "type": "str", 2024-04-23T10:41:08.847 INFO:teuthology.orchestra.run.smithi114.stdout: "level": "advanced", 2024-04-23T10:41:08.847 INFO:teuthology.orchestra.run.smithi114.stdout: "flags": 0, 2024-04-23T10:41:08.847 INFO:teuthology.orchestra.run.smithi114.stdout: "default_value": "", 2024-04-23T10:41:08.847 INFO:teuthology.orchestra.run.smithi114.stdout: "min": "", 2024-04-23T10:41:08.847 INFO:teuthology.orchestra.run.smithi114.stdout: "max": "", 2024-04-23T10:41:08.847 INFO:teuthology.orchestra.run.smithi114.stdout: "enum_allowed": [], 2024-04-23T10:41:08.847 INFO:teuthology.orchestra.run.smithi114.stdout: "desc": "", 2024-04-23T10:41:08.848 INFO:teuthology.orchestra.run.smithi114.stdout: "long_desc": "", 2024-04-23T10:41:08.848 INFO:teuthology.orchestra.run.smithi114.stdout: "tags": [], 2024-04-23T10:41:08.848 INFO:teuthology.orchestra.run.smithi114.stdout: "see_also": [] 2024-04-23T10:41:08.848 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-23T10:41:08.848 INFO:teuthology.orchestra.run.smithi114.stdout: "GRAFANA_API_USERNAME": { 2024-04-23T10:41:08.848 INFO:teuthology.orchestra.run.smithi114.stdout: "name": "GRAFANA_API_USERNAME", 2024-04-23T10:41:08.848 INFO:teuthology.orchestra.run.smithi114.stdout: "type": "str", 2024-04-23T10:41:08.848 INFO:teuthology.orchestra.run.smithi114.stdout: "level": "advanced", 2024-04-23T10:41:08.848 INFO:teuthology.orchestra.run.smithi114.stdout: "flags": 0, 2024-04-23T10:41:08.848 INFO:teuthology.orchestra.run.smithi114.stdout: "default_value": "admin", 2024-04-23T10:41:08.849 INFO:teuthology.orchestra.run.smithi114.stdout: "min": "", 2024-04-23T10:41:08.849 INFO:teuthology.orchestra.run.smithi114.stdout: "max": "", 2024-04-23T10:41:08.849 INFO:teuthology.orchestra.run.smithi114.stdout: "enum_allowed": [], 2024-04-23T10:41:08.849 INFO:teuthology.orchestra.run.smithi114.stdout: "desc": "", 2024-04-23T10:41:08.849 INFO:teuthology.orchestra.run.smithi114.stdout: "long_desc": "", 2024-04-23T10:41:08.849 INFO:teuthology.orchestra.run.smithi114.stdout: "tags": [], 2024-04-23T10:41:08.849 INFO:teuthology.orchestra.run.smithi114.stdout: "see_also": [] 2024-04-23T10:41:08.849 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-23T10:41:08.849 INFO:teuthology.orchestra.run.smithi114.stdout: "GRAFANA_FRONTEND_API_URL": { 2024-04-23T10:41:08.849 INFO:teuthology.orchestra.run.smithi114.stdout: "name": "GRAFANA_FRONTEND_API_URL", 2024-04-23T10:41:08.850 INFO:teuthology.orchestra.run.smithi114.stdout: "type": "str", 2024-04-23T10:41:08.850 INFO:teuthology.orchestra.run.smithi114.stdout: "level": "advanced", 2024-04-23T10:41:08.850 INFO:teuthology.orchestra.run.smithi114.stdout: "flags": 0, 2024-04-23T10:41:08.850 INFO:teuthology.orchestra.run.smithi114.stdout: "default_value": "", 2024-04-23T10:41:08.850 INFO:teuthology.orchestra.run.smithi114.stdout: "min": "", 2024-04-23T10:41:08.850 INFO:teuthology.orchestra.run.smithi114.stdout: "max": "", 2024-04-23T10:41:08.850 INFO:teuthology.orchestra.run.smithi114.stdout: "enum_allowed": [], 2024-04-23T10:41:08.850 INFO:teuthology.orchestra.run.smithi114.stdout: "desc": "", 2024-04-23T10:41:08.850 INFO:teuthology.orchestra.run.smithi114.stdout: "long_desc": "", 2024-04-23T10:41:08.851 INFO:teuthology.orchestra.run.smithi114.stdout: "tags": [], 2024-04-23T10:41:08.851 INFO:teuthology.orchestra.run.smithi114.stdout: "see_also": [] 2024-04-23T10:41:08.851 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-23T10:41:08.851 INFO:teuthology.orchestra.run.smithi114.stdout: "GRAFANA_UPDATE_DASHBOARDS": { 2024-04-23T10:41:08.851 INFO:teuthology.orchestra.run.smithi114.stdout: "name": "GRAFANA_UPDATE_DASHBOARDS", 2024-04-23T10:41:08.851 INFO:teuthology.orchestra.run.smithi114.stdout: "type": "bool", 2024-04-23T10:41:08.851 INFO:teuthology.orchestra.run.smithi114.stdout: "level": "advanced", 2024-04-23T10:41:08.851 INFO:teuthology.orchestra.run.smithi114.stdout: "flags": 0, 2024-04-23T10:41:08.851 INFO:teuthology.orchestra.run.smithi114.stdout: "default_value": "False", 2024-04-23T10:41:08.851 INFO:teuthology.orchestra.run.smithi114.stdout: "min": "", 2024-04-23T10:41:08.852 INFO:teuthology.orchestra.run.smithi114.stdout: "max": "", 2024-04-23T10:41:08.852 INFO:teuthology.orchestra.run.smithi114.stdout: "enum_allowed": [], 2024-04-23T10:41:08.852 INFO:teuthology.orchestra.run.smithi114.stdout: "desc": "", 2024-04-23T10:41:08.852 INFO:teuthology.orchestra.run.smithi114.stdout: "long_desc": "", 2024-04-23T10:41:08.852 INFO:teuthology.orchestra.run.smithi114.stdout: "tags": [], 2024-04-23T10:41:08.852 INFO:teuthology.orchestra.run.smithi114.stdout: "see_also": [] 2024-04-23T10:41:08.852 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-23T10:41:08.852 INFO:teuthology.orchestra.run.smithi114.stdout: "ISCSI_API_SSL_VERIFICATION": { 2024-04-23T10:41:08.852 INFO:teuthology.orchestra.run.smithi114.stdout: "name": "ISCSI_API_SSL_VERIFICATION", 2024-04-23T10:41:08.852 INFO:teuthology.orchestra.run.smithi114.stdout: "type": "bool", 2024-04-23T10:41:08.853 INFO:teuthology.orchestra.run.smithi114.stdout: "level": "advanced", 2024-04-23T10:41:08.853 INFO:teuthology.orchestra.run.smithi114.stdout: "flags": 0, 2024-04-23T10:41:08.853 INFO:teuthology.orchestra.run.smithi114.stdout: "default_value": "True", 2024-04-23T10:41:08.853 INFO:teuthology.orchestra.run.smithi114.stdout: "min": "", 2024-04-23T10:41:08.853 INFO:teuthology.orchestra.run.smithi114.stdout: "max": "", 2024-04-23T10:41:08.853 INFO:teuthology.orchestra.run.smithi114.stdout: "enum_allowed": [], 2024-04-23T10:41:08.853 INFO:teuthology.orchestra.run.smithi114.stdout: "desc": "", 2024-04-23T10:41:08.853 INFO:teuthology.orchestra.run.smithi114.stdout: "long_desc": "", 2024-04-23T10:41:08.853 INFO:teuthology.orchestra.run.smithi114.stdout: "tags": [], 2024-04-23T10:41:08.853 INFO:teuthology.orchestra.run.smithi114.stdout: "see_also": [] 2024-04-23T10:41:08.854 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-23T10:41:08.854 INFO:teuthology.orchestra.run.smithi114.stdout: "ISSUE_TRACKER_API_KEY": { 2024-04-23T10:41:08.854 INFO:teuthology.orchestra.run.smithi114.stdout: "name": "ISSUE_TRACKER_API_KEY", 2024-04-23T10:41:08.854 INFO:teuthology.orchestra.run.smithi114.stdout: "type": "str", 2024-04-23T10:41:08.854 INFO:teuthology.orchestra.run.smithi114.stdout: "level": "advanced", 2024-04-23T10:41:08.854 INFO:teuthology.orchestra.run.smithi114.stdout: "flags": 0, 2024-04-23T10:41:08.854 INFO:teuthology.orchestra.run.smithi114.stdout: "default_value": "", 2024-04-23T10:41:08.854 INFO:teuthology.orchestra.run.smithi114.stdout: "min": "", 2024-04-23T10:41:08.854 INFO:teuthology.orchestra.run.smithi114.stdout: "max": "", 2024-04-23T10:41:08.855 INFO:teuthology.orchestra.run.smithi114.stdout: "enum_allowed": [], 2024-04-23T10:41:08.855 INFO:teuthology.orchestra.run.smithi114.stdout: "desc": "", 2024-04-23T10:41:08.855 INFO:teuthology.orchestra.run.smithi114.stdout: "long_desc": "", 2024-04-23T10:41:08.855 INFO:teuthology.orchestra.run.smithi114.stdout: "tags": [], 2024-04-23T10:41:08.855 INFO:teuthology.orchestra.run.smithi114.stdout: "see_also": [] 2024-04-23T10:41:08.855 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-23T10:41:08.855 INFO:teuthology.orchestra.run.smithi114.stdout: "MANAGED_BY_CLUSTERS": { 2024-04-23T10:41:08.855 INFO:teuthology.orchestra.run.smithi114.stdout: "name": "MANAGED_BY_CLUSTERS", 2024-04-23T10:41:08.855 INFO:teuthology.orchestra.run.smithi114.stdout: "type": "str", 2024-04-23T10:41:08.855 INFO:teuthology.orchestra.run.smithi114.stdout: "level": "advanced", 2024-04-23T10:41:08.856 INFO:teuthology.orchestra.run.smithi114.stdout: "flags": 0, 2024-04-23T10:41:08.856 INFO:teuthology.orchestra.run.smithi114.stdout: "default_value": "[]", 2024-04-23T10:41:08.856 INFO:teuthology.orchestra.run.smithi114.stdout: "min": "", 2024-04-23T10:41:08.856 INFO:teuthology.orchestra.run.smithi114.stdout: "max": "", 2024-04-23T10:41:08.856 INFO:teuthology.orchestra.run.smithi114.stdout: "enum_allowed": [], 2024-04-23T10:41:08.856 INFO:teuthology.orchestra.run.smithi114.stdout: "desc": "", 2024-04-23T10:41:08.856 INFO:teuthology.orchestra.run.smithi114.stdout: "long_desc": "", 2024-04-23T10:41:08.856 INFO:teuthology.orchestra.run.smithi114.stdout: "tags": [], 2024-04-23T10:41:08.856 INFO:teuthology.orchestra.run.smithi114.stdout: "see_also": [] 2024-04-23T10:41:08.856 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-23T10:41:08.857 INFO:teuthology.orchestra.run.smithi114.stdout: "MULTICLUSTER_CONFIG": { 2024-04-23T10:41:08.857 INFO:teuthology.orchestra.run.smithi114.stdout: "name": "MULTICLUSTER_CONFIG", 2024-04-23T10:41:08.857 INFO:teuthology.orchestra.run.smithi114.stdout: "type": "str", 2024-04-23T10:41:08.857 INFO:teuthology.orchestra.run.smithi114.stdout: "level": "advanced", 2024-04-23T10:41:08.857 INFO:teuthology.orchestra.run.smithi114.stdout: "flags": 0, 2024-04-23T10:41:08.857 INFO:teuthology.orchestra.run.smithi114.stdout: "default_value": "{}", 2024-04-23T10:41:08.857 INFO:teuthology.orchestra.run.smithi114.stdout: "min": "", 2024-04-23T10:41:08.857 INFO:teuthology.orchestra.run.smithi114.stdout: "max": "", 2024-04-23T10:41:08.857 INFO:teuthology.orchestra.run.smithi114.stdout: "enum_allowed": [], 2024-04-23T10:41:08.857 INFO:teuthology.orchestra.run.smithi114.stdout: "desc": "", 2024-04-23T10:41:08.858 INFO:teuthology.orchestra.run.smithi114.stdout: "long_desc": "", 2024-04-23T10:41:08.858 INFO:teuthology.orchestra.run.smithi114.stdout: "tags": [], 2024-04-23T10:41:08.858 INFO:teuthology.orchestra.run.smithi114.stdout: "see_also": [] 2024-04-23T10:41:08.858 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-23T10:41:08.858 INFO:teuthology.orchestra.run.smithi114.stdout: "PROMETHEUS_API_HOST": { 2024-04-23T10:41:08.858 INFO:teuthology.orchestra.run.smithi114.stdout: "name": "PROMETHEUS_API_HOST", 2024-04-23T10:41:08.858 INFO:teuthology.orchestra.run.smithi114.stdout: "type": "str", 2024-04-23T10:41:08.858 INFO:teuthology.orchestra.run.smithi114.stdout: "level": "advanced", 2024-04-23T10:41:08.858 INFO:teuthology.orchestra.run.smithi114.stdout: "flags": 0, 2024-04-23T10:41:08.859 INFO:teuthology.orchestra.run.smithi114.stdout: "default_value": "", 2024-04-23T10:41:08.859 INFO:teuthology.orchestra.run.smithi114.stdout: "min": "", 2024-04-23T10:41:08.859 INFO:teuthology.orchestra.run.smithi114.stdout: "max": "", 2024-04-23T10:41:08.859 INFO:teuthology.orchestra.run.smithi114.stdout: "enum_allowed": [], 2024-04-23T10:41:08.859 INFO:teuthology.orchestra.run.smithi114.stdout: "desc": "", 2024-04-23T10:41:08.859 INFO:teuthology.orchestra.run.smithi114.stdout: "long_desc": "", 2024-04-23T10:41:08.859 INFO:teuthology.orchestra.run.smithi114.stdout: "tags": [], 2024-04-23T10:41:08.859 INFO:teuthology.orchestra.run.smithi114.stdout: "see_also": [] 2024-04-23T10:41:08.859 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-23T10:41:08.859 INFO:teuthology.orchestra.run.smithi114.stdout: "PROMETHEUS_API_SSL_VERIFY": { 2024-04-23T10:41:08.860 INFO:teuthology.orchestra.run.smithi114.stdout: "name": "PROMETHEUS_API_SSL_VERIFY", 2024-04-23T10:41:08.860 INFO:teuthology.orchestra.run.smithi114.stdout: "type": "bool", 2024-04-23T10:41:08.860 INFO:teuthology.orchestra.run.smithi114.stdout: "level": "advanced", 2024-04-23T10:41:08.860 INFO:teuthology.orchestra.run.smithi114.stdout: "flags": 0, 2024-04-23T10:41:08.860 INFO:teuthology.orchestra.run.smithi114.stdout: "default_value": "True", 2024-04-23T10:41:08.860 INFO:teuthology.orchestra.run.smithi114.stdout: "min": "", 2024-04-23T10:41:08.860 INFO:teuthology.orchestra.run.smithi114.stdout: "max": "", 2024-04-23T10:41:08.860 INFO:teuthology.orchestra.run.smithi114.stdout: "enum_allowed": [], 2024-04-23T10:41:08.860 INFO:teuthology.orchestra.run.smithi114.stdout: "desc": "", 2024-04-23T10:41:08.860 INFO:teuthology.orchestra.run.smithi114.stdout: "long_desc": "", 2024-04-23T10:41:08.861 INFO:teuthology.orchestra.run.smithi114.stdout: "tags": [], 2024-04-23T10:41:08.861 INFO:teuthology.orchestra.run.smithi114.stdout: "see_also": [] 2024-04-23T10:41:08.861 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-23T10:41:08.861 INFO:teuthology.orchestra.run.smithi114.stdout: "PWD_POLICY_CHECK_COMPLEXITY_ENABLED": { 2024-04-23T10:41:08.861 INFO:teuthology.orchestra.run.smithi114.stdout: "name": "PWD_POLICY_CHECK_COMPLEXITY_ENABLED", 2024-04-23T10:41:08.861 INFO:teuthology.orchestra.run.smithi114.stdout: "type": "bool", 2024-04-23T10:41:08.861 INFO:teuthology.orchestra.run.smithi114.stdout: "level": "advanced", 2024-04-23T10:41:08.861 INFO:teuthology.orchestra.run.smithi114.stdout: "flags": 0, 2024-04-23T10:41:08.861 INFO:teuthology.orchestra.run.smithi114.stdout: "default_value": "False", 2024-04-23T10:41:08.861 INFO:teuthology.orchestra.run.smithi114.stdout: "min": "", 2024-04-23T10:41:08.862 INFO:teuthology.orchestra.run.smithi114.stdout: "max": "", 2024-04-23T10:41:08.862 INFO:teuthology.orchestra.run.smithi114.stdout: "enum_allowed": [], 2024-04-23T10:41:08.862 INFO:teuthology.orchestra.run.smithi114.stdout: "desc": "", 2024-04-23T10:41:08.862 INFO:teuthology.orchestra.run.smithi114.stdout: "long_desc": "", 2024-04-23T10:41:08.862 INFO:teuthology.orchestra.run.smithi114.stdout: "tags": [], 2024-04-23T10:41:08.862 INFO:teuthology.orchestra.run.smithi114.stdout: "see_also": [] 2024-04-23T10:41:08.862 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-23T10:41:08.862 INFO:teuthology.orchestra.run.smithi114.stdout: "PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED": { 2024-04-23T10:41:08.862 INFO:teuthology.orchestra.run.smithi114.stdout: "name": "PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED", 2024-04-23T10:41:08.862 INFO:teuthology.orchestra.run.smithi114.stdout: "type": "bool", 2024-04-23T10:41:08.863 INFO:teuthology.orchestra.run.smithi114.stdout: "level": "advanced", 2024-04-23T10:41:08.863 INFO:teuthology.orchestra.run.smithi114.stdout: "flags": 0, 2024-04-23T10:41:08.863 INFO:teuthology.orchestra.run.smithi114.stdout: "default_value": "False", 2024-04-23T10:41:08.863 INFO:teuthology.orchestra.run.smithi114.stdout: "min": "", 2024-04-23T10:41:08.863 INFO:teuthology.orchestra.run.smithi114.stdout: "max": "", 2024-04-23T10:41:08.863 INFO:teuthology.orchestra.run.smithi114.stdout: "enum_allowed": [], 2024-04-23T10:41:08.863 INFO:teuthology.orchestra.run.smithi114.stdout: "desc": "", 2024-04-23T10:41:08.863 INFO:teuthology.orchestra.run.smithi114.stdout: "long_desc": "", 2024-04-23T10:41:08.863 INFO:teuthology.orchestra.run.smithi114.stdout: "tags": [], 2024-04-23T10:41:08.863 INFO:teuthology.orchestra.run.smithi114.stdout: "see_also": [] 2024-04-23T10:41:08.864 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-23T10:41:08.864 INFO:teuthology.orchestra.run.smithi114.stdout: "PWD_POLICY_CHECK_LENGTH_ENABLED": { 2024-04-23T10:41:08.864 INFO:teuthology.orchestra.run.smithi114.stdout: "name": "PWD_POLICY_CHECK_LENGTH_ENABLED", 2024-04-23T10:41:08.864 INFO:teuthology.orchestra.run.smithi114.stdout: "type": "bool", 2024-04-23T10:41:08.864 INFO:teuthology.orchestra.run.smithi114.stdout: "level": "advanced", 2024-04-23T10:41:08.864 INFO:teuthology.orchestra.run.smithi114.stdout: "flags": 0, 2024-04-23T10:41:08.864 INFO:teuthology.orchestra.run.smithi114.stdout: "default_value": "True", 2024-04-23T10:41:08.864 INFO:teuthology.orchestra.run.smithi114.stdout: "min": "", 2024-04-23T10:41:08.864 INFO:teuthology.orchestra.run.smithi114.stdout: "max": "", 2024-04-23T10:41:08.864 INFO:teuthology.orchestra.run.smithi114.stdout: "enum_allowed": [], 2024-04-23T10:41:08.865 INFO:teuthology.orchestra.run.smithi114.stdout: "desc": "", 2024-04-23T10:41:08.865 INFO:teuthology.orchestra.run.smithi114.stdout: "long_desc": "", 2024-04-23T10:41:08.865 INFO:teuthology.orchestra.run.smithi114.stdout: "tags": [], 2024-04-23T10:41:08.865 INFO:teuthology.orchestra.run.smithi114.stdout: "see_also": [] 2024-04-23T10:41:08.865 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-23T10:41:08.865 INFO:teuthology.orchestra.run.smithi114.stdout: "PWD_POLICY_CHECK_OLDPWD_ENABLED": { 2024-04-23T10:41:08.865 INFO:teuthology.orchestra.run.smithi114.stdout: "name": "PWD_POLICY_CHECK_OLDPWD_ENABLED", 2024-04-23T10:41:08.865 INFO:teuthology.orchestra.run.smithi114.stdout: "type": "bool", 2024-04-23T10:41:08.865 INFO:teuthology.orchestra.run.smithi114.stdout: "level": "advanced", 2024-04-23T10:41:08.865 INFO:teuthology.orchestra.run.smithi114.stdout: "flags": 0, 2024-04-23T10:41:08.866 INFO:teuthology.orchestra.run.smithi114.stdout: "default_value": "True", 2024-04-23T10:41:08.866 INFO:teuthology.orchestra.run.smithi114.stdout: "min": "", 2024-04-23T10:41:08.866 INFO:teuthology.orchestra.run.smithi114.stdout: "max": "", 2024-04-23T10:41:08.866 INFO:teuthology.orchestra.run.smithi114.stdout: "enum_allowed": [], 2024-04-23T10:41:08.866 INFO:teuthology.orchestra.run.smithi114.stdout: "desc": "", 2024-04-23T10:41:08.866 INFO:teuthology.orchestra.run.smithi114.stdout: "long_desc": "", 2024-04-23T10:41:08.866 INFO:teuthology.orchestra.run.smithi114.stdout: "tags": [], 2024-04-23T10:41:08.866 INFO:teuthology.orchestra.run.smithi114.stdout: "see_also": [] 2024-04-23T10:41:08.866 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-23T10:41:08.867 INFO:teuthology.orchestra.run.smithi114.stdout: "PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED": { 2024-04-23T10:41:08.867 INFO:teuthology.orchestra.run.smithi114.stdout: "name": "PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED", 2024-04-23T10:41:08.867 INFO:teuthology.orchestra.run.smithi114.stdout: "type": "bool", 2024-04-23T10:41:08.867 INFO:teuthology.orchestra.run.smithi114.stdout: "level": "advanced", 2024-04-23T10:41:08.867 INFO:teuthology.orchestra.run.smithi114.stdout: "flags": 0, 2024-04-23T10:41:08.867 INFO:teuthology.orchestra.run.smithi114.stdout: "default_value": "False", 2024-04-23T10:41:08.867 INFO:teuthology.orchestra.run.smithi114.stdout: "min": "", 2024-04-23T10:41:08.867 INFO:teuthology.orchestra.run.smithi114.stdout: "max": "", 2024-04-23T10:41:08.867 INFO:teuthology.orchestra.run.smithi114.stdout: "enum_allowed": [], 2024-04-23T10:41:08.867 INFO:teuthology.orchestra.run.smithi114.stdout: "desc": "", 2024-04-23T10:41:08.868 INFO:teuthology.orchestra.run.smithi114.stdout: "long_desc": "", 2024-04-23T10:41:08.868 INFO:teuthology.orchestra.run.smithi114.stdout: "tags": [], 2024-04-23T10:41:08.868 INFO:teuthology.orchestra.run.smithi114.stdout: "see_also": [] 2024-04-23T10:41:08.868 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-23T10:41:08.868 INFO:teuthology.orchestra.run.smithi114.stdout: "PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED": { 2024-04-23T10:41:08.868 INFO:teuthology.orchestra.run.smithi114.stdout: "name": "PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED", 2024-04-23T10:41:08.868 INFO:teuthology.orchestra.run.smithi114.stdout: "type": "bool", 2024-04-23T10:41:08.868 INFO:teuthology.orchestra.run.smithi114.stdout: "level": "advanced", 2024-04-23T10:41:08.868 INFO:teuthology.orchestra.run.smithi114.stdout: "flags": 0, 2024-04-23T10:41:08.869 INFO:teuthology.orchestra.run.smithi114.stdout: "default_value": "False", 2024-04-23T10:41:08.869 INFO:teuthology.orchestra.run.smithi114.stdout: "min": "", 2024-04-23T10:41:08.869 INFO:teuthology.orchestra.run.smithi114.stdout: "max": "", 2024-04-23T10:41:08.869 INFO:teuthology.orchestra.run.smithi114.stdout: "enum_allowed": [], 2024-04-23T10:41:08.869 INFO:teuthology.orchestra.run.smithi114.stdout: "desc": "", 2024-04-23T10:41:08.869 INFO:teuthology.orchestra.run.smithi114.stdout: "long_desc": "", 2024-04-23T10:41:08.869 INFO:teuthology.orchestra.run.smithi114.stdout: "tags": [], 2024-04-23T10:41:08.869 INFO:teuthology.orchestra.run.smithi114.stdout: "see_also": [] 2024-04-23T10:41:08.869 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-23T10:41:08.869 INFO:teuthology.orchestra.run.smithi114.stdout: "PWD_POLICY_CHECK_USERNAME_ENABLED": { 2024-04-23T10:41:08.870 INFO:teuthology.orchestra.run.smithi114.stdout: "name": "PWD_POLICY_CHECK_USERNAME_ENABLED", 2024-04-23T10:41:08.870 INFO:teuthology.orchestra.run.smithi114.stdout: "type": "bool", 2024-04-23T10:41:08.870 INFO:teuthology.orchestra.run.smithi114.stdout: "level": "advanced", 2024-04-23T10:41:08.870 INFO:teuthology.orchestra.run.smithi114.stdout: "flags": 0, 2024-04-23T10:41:08.870 INFO:teuthology.orchestra.run.smithi114.stdout: "default_value": "False", 2024-04-23T10:41:08.870 INFO:teuthology.orchestra.run.smithi114.stdout: "min": "", 2024-04-23T10:41:08.870 INFO:teuthology.orchestra.run.smithi114.stdout: "max": "", 2024-04-23T10:41:08.870 INFO:teuthology.orchestra.run.smithi114.stdout: "enum_allowed": [], 2024-04-23T10:41:08.870 INFO:teuthology.orchestra.run.smithi114.stdout: "desc": "", 2024-04-23T10:41:08.871 INFO:teuthology.orchestra.run.smithi114.stdout: "long_desc": "", 2024-04-23T10:41:08.871 INFO:teuthology.orchestra.run.smithi114.stdout: "tags": [], 2024-04-23T10:41:08.871 INFO:teuthology.orchestra.run.smithi114.stdout: "see_also": [] 2024-04-23T10:41:08.871 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-23T10:41:08.871 INFO:teuthology.orchestra.run.smithi114.stdout: "PWD_POLICY_ENABLED": { 2024-04-23T10:41:08.871 INFO:teuthology.orchestra.run.smithi114.stdout: "name": "PWD_POLICY_ENABLED", 2024-04-23T10:41:08.871 INFO:teuthology.orchestra.run.smithi114.stdout: "type": "bool", 2024-04-23T10:41:08.871 INFO:teuthology.orchestra.run.smithi114.stdout: "level": "advanced", 2024-04-23T10:41:08.871 INFO:teuthology.orchestra.run.smithi114.stdout: "flags": 0, 2024-04-23T10:41:08.871 INFO:teuthology.orchestra.run.smithi114.stdout: "default_value": "True", 2024-04-23T10:41:08.872 INFO:teuthology.orchestra.run.smithi114.stdout: "min": "", 2024-04-23T10:41:08.872 INFO:teuthology.orchestra.run.smithi114.stdout: "max": "", 2024-04-23T10:41:08.872 INFO:teuthology.orchestra.run.smithi114.stdout: "enum_allowed": [], 2024-04-23T10:41:08.872 INFO:teuthology.orchestra.run.smithi114.stdout: "desc": "", 2024-04-23T10:41:08.872 INFO:teuthology.orchestra.run.smithi114.stdout: "long_desc": "", 2024-04-23T10:41:08.872 INFO:teuthology.orchestra.run.smithi114.stdout: "tags": [], 2024-04-23T10:41:08.872 INFO:teuthology.orchestra.run.smithi114.stdout: "see_also": [] 2024-04-23T10:41:08.872 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-23T10:41:08.872 INFO:teuthology.orchestra.run.smithi114.stdout: "PWD_POLICY_EXCLUSION_LIST": { 2024-04-23T10:41:08.873 INFO:teuthology.orchestra.run.smithi114.stdout: "name": "PWD_POLICY_EXCLUSION_LIST", 2024-04-23T10:41:08.873 INFO:teuthology.orchestra.run.smithi114.stdout: "type": "str", 2024-04-23T10:41:08.873 INFO:teuthology.orchestra.run.smithi114.stdout: "level": "advanced", 2024-04-23T10:41:08.873 INFO:teuthology.orchestra.run.smithi114.stdout: "flags": 0, 2024-04-23T10:41:08.873 INFO:teuthology.orchestra.run.smithi114.stdout: "default_value": "osd,host,dashboard,pool,block,nfs,ceph,monitors,gateway,logs,crush,maps", 2024-04-23T10:41:08.873 INFO:teuthology.orchestra.run.smithi114.stdout: "min": "", 2024-04-23T10:41:08.873 INFO:teuthology.orchestra.run.smithi114.stdout: "max": "", 2024-04-23T10:41:08.873 INFO:teuthology.orchestra.run.smithi114.stdout: "enum_allowed": [], 2024-04-23T10:41:08.873 INFO:teuthology.orchestra.run.smithi114.stdout: "desc": "", 2024-04-23T10:41:08.873 INFO:teuthology.orchestra.run.smithi114.stdout: "long_desc": "", 2024-04-23T10:41:08.874 INFO:teuthology.orchestra.run.smithi114.stdout: "tags": [], 2024-04-23T10:41:08.874 INFO:teuthology.orchestra.run.smithi114.stdout: "see_also": [] 2024-04-23T10:41:08.874 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-23T10:41:08.874 INFO:teuthology.orchestra.run.smithi114.stdout: "PWD_POLICY_MIN_COMPLEXITY": { 2024-04-23T10:41:08.874 INFO:teuthology.orchestra.run.smithi114.stdout: "name": "PWD_POLICY_MIN_COMPLEXITY", 2024-04-23T10:41:08.874 INFO:teuthology.orchestra.run.smithi114.stdout: "type": "int", 2024-04-23T10:41:08.874 INFO:teuthology.orchestra.run.smithi114.stdout: "level": "advanced", 2024-04-23T10:41:08.874 INFO:teuthology.orchestra.run.smithi114.stdout: "flags": 0, 2024-04-23T10:41:08.874 INFO:teuthology.orchestra.run.smithi114.stdout: "default_value": "10", 2024-04-23T10:41:08.874 INFO:teuthology.orchestra.run.smithi114.stdout: "min": "", 2024-04-23T10:41:08.875 INFO:teuthology.orchestra.run.smithi114.stdout: "max": "", 2024-04-23T10:41:08.875 INFO:teuthology.orchestra.run.smithi114.stdout: "enum_allowed": [], 2024-04-23T10:41:08.875 INFO:teuthology.orchestra.run.smithi114.stdout: "desc": "", 2024-04-23T10:41:08.875 INFO:teuthology.orchestra.run.smithi114.stdout: "long_desc": "", 2024-04-23T10:41:08.875 INFO:teuthology.orchestra.run.smithi114.stdout: "tags": [], 2024-04-23T10:41:08.875 INFO:teuthology.orchestra.run.smithi114.stdout: "see_also": [] 2024-04-23T10:41:08.875 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-23T10:41:08.875 INFO:teuthology.orchestra.run.smithi114.stdout: "PWD_POLICY_MIN_LENGTH": { 2024-04-23T10:41:08.875 INFO:teuthology.orchestra.run.smithi114.stdout: "name": "PWD_POLICY_MIN_LENGTH", 2024-04-23T10:41:08.875 INFO:teuthology.orchestra.run.smithi114.stdout: "type": "int", 2024-04-23T10:41:08.876 INFO:teuthology.orchestra.run.smithi114.stdout: "level": "advanced", 2024-04-23T10:41:08.876 INFO:teuthology.orchestra.run.smithi114.stdout: "flags": 0, 2024-04-23T10:41:08.876 INFO:teuthology.orchestra.run.smithi114.stdout: "default_value": "8", 2024-04-23T10:41:08.876 INFO:teuthology.orchestra.run.smithi114.stdout: "min": "", 2024-04-23T10:41:08.876 INFO:teuthology.orchestra.run.smithi114.stdout: "max": "", 2024-04-23T10:41:08.876 INFO:teuthology.orchestra.run.smithi114.stdout: "enum_allowed": [], 2024-04-23T10:41:08.876 INFO:teuthology.orchestra.run.smithi114.stdout: "desc": "", 2024-04-23T10:41:08.876 INFO:teuthology.orchestra.run.smithi114.stdout: "long_desc": "", 2024-04-23T10:41:08.876 INFO:teuthology.orchestra.run.smithi114.stdout: "tags": [], 2024-04-23T10:41:08.876 INFO:teuthology.orchestra.run.smithi114.stdout: "see_also": [] 2024-04-23T10:41:08.877 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-23T10:41:08.877 INFO:teuthology.orchestra.run.smithi114.stdout: "REST_REQUESTS_TIMEOUT": { 2024-04-23T10:41:08.877 INFO:teuthology.orchestra.run.smithi114.stdout: "name": "REST_REQUESTS_TIMEOUT", 2024-04-23T10:41:08.877 INFO:teuthology.orchestra.run.smithi114.stdout: "type": "int", 2024-04-23T10:41:08.877 INFO:teuthology.orchestra.run.smithi114.stdout: "level": "advanced", 2024-04-23T10:41:08.877 INFO:teuthology.orchestra.run.smithi114.stdout: "flags": 0, 2024-04-23T10:41:08.877 INFO:teuthology.orchestra.run.smithi114.stdout: "default_value": "45", 2024-04-23T10:41:08.877 INFO:teuthology.orchestra.run.smithi114.stdout: "min": "", 2024-04-23T10:41:08.877 INFO:teuthology.orchestra.run.smithi114.stdout: "max": "", 2024-04-23T10:41:08.877 INFO:teuthology.orchestra.run.smithi114.stdout: "enum_allowed": [], 2024-04-23T10:41:08.878 INFO:teuthology.orchestra.run.smithi114.stdout: "desc": "", 2024-04-23T10:41:08.878 INFO:teuthology.orchestra.run.smithi114.stdout: "long_desc": "", 2024-04-23T10:41:08.878 INFO:teuthology.orchestra.run.smithi114.stdout: "tags": [], 2024-04-23T10:41:08.878 INFO:teuthology.orchestra.run.smithi114.stdout: "see_also": [] 2024-04-23T10:41:08.878 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-23T10:41:08.878 INFO:teuthology.orchestra.run.smithi114.stdout: "RGW_API_ACCESS_KEY": { 2024-04-23T10:41:08.878 INFO:teuthology.orchestra.run.smithi114.stdout: "name": "RGW_API_ACCESS_KEY", 2024-04-23T10:41:08.878 INFO:teuthology.orchestra.run.smithi114.stdout: "type": "str", 2024-04-23T10:41:08.878 INFO:teuthology.orchestra.run.smithi114.stdout: "level": "advanced", 2024-04-23T10:41:08.879 INFO:teuthology.orchestra.run.smithi114.stdout: "flags": 0, 2024-04-23T10:41:08.879 INFO:teuthology.orchestra.run.smithi114.stdout: "default_value": "", 2024-04-23T10:41:08.879 INFO:teuthology.orchestra.run.smithi114.stdout: "min": "", 2024-04-23T10:41:08.879 INFO:teuthology.orchestra.run.smithi114.stdout: "max": "", 2024-04-23T10:41:08.879 INFO:teuthology.orchestra.run.smithi114.stdout: "enum_allowed": [], 2024-04-23T10:41:08.879 INFO:teuthology.orchestra.run.smithi114.stdout: "desc": "", 2024-04-23T10:41:08.879 INFO:teuthology.orchestra.run.smithi114.stdout: "long_desc": "", 2024-04-23T10:41:08.879 INFO:teuthology.orchestra.run.smithi114.stdout: "tags": [], 2024-04-23T10:41:08.879 INFO:teuthology.orchestra.run.smithi114.stdout: "see_also": [] 2024-04-23T10:41:08.879 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-23T10:41:08.880 INFO:teuthology.orchestra.run.smithi114.stdout: "RGW_API_ADMIN_RESOURCE": { 2024-04-23T10:41:08.880 INFO:teuthology.orchestra.run.smithi114.stdout: "name": "RGW_API_ADMIN_RESOURCE", 2024-04-23T10:41:08.880 INFO:teuthology.orchestra.run.smithi114.stdout: "type": "str", 2024-04-23T10:41:08.880 INFO:teuthology.orchestra.run.smithi114.stdout: "level": "advanced", 2024-04-23T10:41:08.880 INFO:teuthology.orchestra.run.smithi114.stdout: "flags": 0, 2024-04-23T10:41:08.880 INFO:teuthology.orchestra.run.smithi114.stdout: "default_value": "admin", 2024-04-23T10:41:08.880 INFO:teuthology.orchestra.run.smithi114.stdout: "min": "", 2024-04-23T10:41:08.880 INFO:teuthology.orchestra.run.smithi114.stdout: "max": "", 2024-04-23T10:41:08.880 INFO:teuthology.orchestra.run.smithi114.stdout: "enum_allowed": [], 2024-04-23T10:41:08.880 INFO:teuthology.orchestra.run.smithi114.stdout: "desc": "", 2024-04-23T10:41:08.881 INFO:teuthology.orchestra.run.smithi114.stdout: "long_desc": "", 2024-04-23T10:41:08.881 INFO:teuthology.orchestra.run.smithi114.stdout: "tags": [], 2024-04-23T10:41:08.881 INFO:teuthology.orchestra.run.smithi114.stdout: "see_also": [] 2024-04-23T10:41:08.881 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-23T10:41:08.881 INFO:teuthology.orchestra.run.smithi114.stdout: "RGW_API_SECRET_KEY": { 2024-04-23T10:41:08.881 INFO:teuthology.orchestra.run.smithi114.stdout: "name": "RGW_API_SECRET_KEY", 2024-04-23T10:41:08.881 INFO:teuthology.orchestra.run.smithi114.stdout: "type": "str", 2024-04-23T10:41:08.881 INFO:teuthology.orchestra.run.smithi114.stdout: "level": "advanced", 2024-04-23T10:41:08.881 INFO:teuthology.orchestra.run.smithi114.stdout: "flags": 0, 2024-04-23T10:41:08.881 INFO:teuthology.orchestra.run.smithi114.stdout: "default_value": "", 2024-04-23T10:41:08.882 INFO:teuthology.orchestra.run.smithi114.stdout: "min": "", 2024-04-23T10:41:08.882 INFO:teuthology.orchestra.run.smithi114.stdout: "max": "", 2024-04-23T10:41:08.882 INFO:teuthology.orchestra.run.smithi114.stdout: "enum_allowed": [], 2024-04-23T10:41:08.882 INFO:teuthology.orchestra.run.smithi114.stdout: "desc": "", 2024-04-23T10:41:08.882 INFO:teuthology.orchestra.run.smithi114.stdout: "long_desc": "", 2024-04-23T10:41:08.882 INFO:teuthology.orchestra.run.smithi114.stdout: "tags": [], 2024-04-23T10:41:08.882 INFO:teuthology.orchestra.run.smithi114.stdout: "see_also": [] 2024-04-23T10:41:08.882 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-23T10:41:08.882 INFO:teuthology.orchestra.run.smithi114.stdout: "RGW_API_SSL_VERIFY": { 2024-04-23T10:41:08.882 INFO:teuthology.orchestra.run.smithi114.stdout: "name": "RGW_API_SSL_VERIFY", 2024-04-23T10:41:08.883 INFO:teuthology.orchestra.run.smithi114.stdout: "type": "bool", 2024-04-23T10:41:08.883 INFO:teuthology.orchestra.run.smithi114.stdout: "level": "advanced", 2024-04-23T10:41:08.883 INFO:teuthology.orchestra.run.smithi114.stdout: "flags": 0, 2024-04-23T10:41:08.883 INFO:teuthology.orchestra.run.smithi114.stdout: "default_value": "True", 2024-04-23T10:41:08.883 INFO:teuthology.orchestra.run.smithi114.stdout: "min": "", 2024-04-23T10:41:08.883 INFO:teuthology.orchestra.run.smithi114.stdout: "max": "", 2024-04-23T10:41:08.883 INFO:teuthology.orchestra.run.smithi114.stdout: "enum_allowed": [], 2024-04-23T10:41:08.883 INFO:teuthology.orchestra.run.smithi114.stdout: "desc": "", 2024-04-23T10:41:08.883 INFO:teuthology.orchestra.run.smithi114.stdout: "long_desc": "", 2024-04-23T10:41:08.883 INFO:teuthology.orchestra.run.smithi114.stdout: "tags": [], 2024-04-23T10:41:08.884 INFO:teuthology.orchestra.run.smithi114.stdout: "see_also": [] 2024-04-23T10:41:08.884 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-23T10:41:08.884 INFO:teuthology.orchestra.run.smithi114.stdout: "UNSAFE_TLS_v1_2": { 2024-04-23T10:41:08.884 INFO:teuthology.orchestra.run.smithi114.stdout: "name": "UNSAFE_TLS_v1_2", 2024-04-23T10:41:08.884 INFO:teuthology.orchestra.run.smithi114.stdout: "type": "bool", 2024-04-23T10:41:08.884 INFO:teuthology.orchestra.run.smithi114.stdout: "level": "advanced", 2024-04-23T10:41:08.884 INFO:teuthology.orchestra.run.smithi114.stdout: "flags": 0, 2024-04-23T10:41:08.884 INFO:teuthology.orchestra.run.smithi114.stdout: "default_value": "False", 2024-04-23T10:41:08.884 INFO:teuthology.orchestra.run.smithi114.stdout: "min": "", 2024-04-23T10:41:08.884 INFO:teuthology.orchestra.run.smithi114.stdout: "max": "", 2024-04-23T10:41:08.885 INFO:teuthology.orchestra.run.smithi114.stdout: "enum_allowed": [], 2024-04-23T10:41:08.885 INFO:teuthology.orchestra.run.smithi114.stdout: "desc": "", 2024-04-23T10:41:08.885 INFO:teuthology.orchestra.run.smithi114.stdout: "long_desc": "", 2024-04-23T10:41:08.885 INFO:teuthology.orchestra.run.smithi114.stdout: "tags": [], 2024-04-23T10:41:08.885 INFO:teuthology.orchestra.run.smithi114.stdout: "see_also": [] 2024-04-23T10:41:08.885 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-23T10:41:08.885 INFO:teuthology.orchestra.run.smithi114.stdout: "USER_PWD_EXPIRATION_SPAN": { 2024-04-23T10:41:08.885 INFO:teuthology.orchestra.run.smithi114.stdout: "name": "USER_PWD_EXPIRATION_SPAN", 2024-04-23T10:41:08.885 INFO:teuthology.orchestra.run.smithi114.stdout: "type": "int", 2024-04-23T10:41:08.886 INFO:teuthology.orchestra.run.smithi114.stdout: "level": "advanced", 2024-04-23T10:41:08.886 INFO:teuthology.orchestra.run.smithi114.stdout: "flags": 0, 2024-04-23T10:41:08.886 INFO:teuthology.orchestra.run.smithi114.stdout: "default_value": "0", 2024-04-23T10:41:08.886 INFO:teuthology.orchestra.run.smithi114.stdout: "min": "", 2024-04-23T10:41:08.886 INFO:teuthology.orchestra.run.smithi114.stdout: "max": "", 2024-04-23T10:41:08.886 INFO:teuthology.orchestra.run.smithi114.stdout: "enum_allowed": [], 2024-04-23T10:41:08.886 INFO:teuthology.orchestra.run.smithi114.stdout: "desc": "", 2024-04-23T10:41:08.886 INFO:teuthology.orchestra.run.smithi114.stdout: "long_desc": "", 2024-04-23T10:41:08.886 INFO:teuthology.orchestra.run.smithi114.stdout: "tags": [], 2024-04-23T10:41:08.886 INFO:teuthology.orchestra.run.smithi114.stdout: "see_also": [] 2024-04-23T10:41:08.887 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-23T10:41:08.887 INFO:teuthology.orchestra.run.smithi114.stdout: "USER_PWD_EXPIRATION_WARNING_1": { 2024-04-23T10:41:08.887 INFO:teuthology.orchestra.run.smithi114.stdout: "name": "USER_PWD_EXPIRATION_WARNING_1", 2024-04-23T10:41:08.887 INFO:teuthology.orchestra.run.smithi114.stdout: "type": "int", 2024-04-23T10:41:08.887 INFO:teuthology.orchestra.run.smithi114.stdout: "level": "advanced", 2024-04-23T10:41:08.887 INFO:teuthology.orchestra.run.smithi114.stdout: "flags": 0, 2024-04-23T10:41:08.887 INFO:teuthology.orchestra.run.smithi114.stdout: "default_value": "10", 2024-04-23T10:41:08.887 INFO:teuthology.orchestra.run.smithi114.stdout: "min": "", 2024-04-23T10:41:08.887 INFO:teuthology.orchestra.run.smithi114.stdout: "max": "", 2024-04-23T10:41:08.887 INFO:teuthology.orchestra.run.smithi114.stdout: "enum_allowed": [], 2024-04-23T10:41:08.888 INFO:teuthology.orchestra.run.smithi114.stdout: "desc": "", 2024-04-23T10:41:08.888 INFO:teuthology.orchestra.run.smithi114.stdout: "long_desc": "", 2024-04-23T10:41:08.888 INFO:teuthology.orchestra.run.smithi114.stdout: "tags": [], 2024-04-23T10:41:08.888 INFO:teuthology.orchestra.run.smithi114.stdout: "see_also": [] 2024-04-23T10:41:08.888 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-23T10:41:08.888 INFO:teuthology.orchestra.run.smithi114.stdout: "USER_PWD_EXPIRATION_WARNING_2": { 2024-04-23T10:41:08.888 INFO:teuthology.orchestra.run.smithi114.stdout: "name": "USER_PWD_EXPIRATION_WARNING_2", 2024-04-23T10:41:08.888 INFO:teuthology.orchestra.run.smithi114.stdout: "type": "int", 2024-04-23T10:41:08.888 INFO:teuthology.orchestra.run.smithi114.stdout: "level": "advanced", 2024-04-23T10:41:08.888 INFO:teuthology.orchestra.run.smithi114.stdout: "flags": 0, 2024-04-23T10:41:08.889 INFO:teuthology.orchestra.run.smithi114.stdout: "default_value": "5", 2024-04-23T10:41:08.889 INFO:teuthology.orchestra.run.smithi114.stdout: "min": "", 2024-04-23T10:41:08.889 INFO:teuthology.orchestra.run.smithi114.stdout: "max": "", 2024-04-23T10:41:08.889 INFO:teuthology.orchestra.run.smithi114.stdout: "enum_allowed": [], 2024-04-23T10:41:08.889 INFO:teuthology.orchestra.run.smithi114.stdout: "desc": "", 2024-04-23T10:41:08.889 INFO:teuthology.orchestra.run.smithi114.stdout: "long_desc": "", 2024-04-23T10:41:08.889 INFO:teuthology.orchestra.run.smithi114.stdout: "tags": [], 2024-04-23T10:41:08.889 INFO:teuthology.orchestra.run.smithi114.stdout: "see_also": [] 2024-04-23T10:41:08.889 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-23T10:41:08.889 INFO:teuthology.orchestra.run.smithi114.stdout: "cross_origin_url": { 2024-04-23T10:41:08.890 INFO:teuthology.orchestra.run.smithi114.stdout: "name": "cross_origin_url", 2024-04-23T10:41:08.890 INFO:teuthology.orchestra.run.smithi114.stdout: "type": "str", 2024-04-23T10:41:08.890 INFO:teuthology.orchestra.run.smithi114.stdout: "level": "advanced", 2024-04-23T10:41:08.890 INFO:teuthology.orchestra.run.smithi114.stdout: "flags": 0, 2024-04-23T10:41:08.890 INFO:teuthology.orchestra.run.smithi114.stdout: "default_value": "", 2024-04-23T10:41:08.890 INFO:teuthology.orchestra.run.smithi114.stdout: "min": "", 2024-04-23T10:41:08.890 INFO:teuthology.orchestra.run.smithi114.stdout: "max": "", 2024-04-23T10:41:08.890 INFO:teuthology.orchestra.run.smithi114.stdout: "enum_allowed": [], 2024-04-23T10:41:08.890 INFO:teuthology.orchestra.run.smithi114.stdout: "desc": "", 2024-04-23T10:41:08.891 INFO:teuthology.orchestra.run.smithi114.stdout: "long_desc": "", 2024-04-23T10:41:08.891 INFO:teuthology.orchestra.run.smithi114.stdout: "tags": [], 2024-04-23T10:41:08.891 INFO:teuthology.orchestra.run.smithi114.stdout: "see_also": [] 2024-04-23T10:41:08.891 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-23T10:41:08.891 INFO:teuthology.orchestra.run.smithi114.stdout: "crt_file": { 2024-04-23T10:41:08.891 INFO:teuthology.orchestra.run.smithi114.stdout: "name": "crt_file", 2024-04-23T10:41:08.891 INFO:teuthology.orchestra.run.smithi114.stdout: "type": "str", 2024-04-23T10:41:08.891 INFO:teuthology.orchestra.run.smithi114.stdout: "level": "advanced", 2024-04-23T10:41:08.891 INFO:teuthology.orchestra.run.smithi114.stdout: "flags": 0, 2024-04-23T10:41:08.891 INFO:teuthology.orchestra.run.smithi114.stdout: "default_value": "", 2024-04-23T10:41:08.892 INFO:teuthology.orchestra.run.smithi114.stdout: "min": "", 2024-04-23T10:41:08.892 INFO:teuthology.orchestra.run.smithi114.stdout: "max": "", 2024-04-23T10:41:08.892 INFO:teuthology.orchestra.run.smithi114.stdout: "enum_allowed": [], 2024-04-23T10:41:08.892 INFO:teuthology.orchestra.run.smithi114.stdout: "desc": "", 2024-04-23T10:41:08.892 INFO:teuthology.orchestra.run.smithi114.stdout: "long_desc": "", 2024-04-23T10:41:08.892 INFO:teuthology.orchestra.run.smithi114.stdout: "tags": [], 2024-04-23T10:41:08.892 INFO:teuthology.orchestra.run.smithi114.stdout: "see_also": [] 2024-04-23T10:41:08.892 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-23T10:41:08.892 INFO:teuthology.orchestra.run.smithi114.stdout: "debug": { 2024-04-23T10:41:08.892 INFO:teuthology.orchestra.run.smithi114.stdout: "name": "debug", 2024-04-23T10:41:08.893 INFO:teuthology.orchestra.run.smithi114.stdout: "type": "bool", 2024-04-23T10:41:08.893 INFO:teuthology.orchestra.run.smithi114.stdout: "level": "advanced", 2024-04-23T10:41:08.893 INFO:teuthology.orchestra.run.smithi114.stdout: "flags": 0, 2024-04-23T10:41:08.893 INFO:teuthology.orchestra.run.smithi114.stdout: "default_value": "False", 2024-04-23T10:41:08.893 INFO:teuthology.orchestra.run.smithi114.stdout: "min": "", 2024-04-23T10:41:08.893 INFO:teuthology.orchestra.run.smithi114.stdout: "max": "", 2024-04-23T10:41:08.893 INFO:teuthology.orchestra.run.smithi114.stdout: "enum_allowed": [], 2024-04-23T10:41:08.893 INFO:teuthology.orchestra.run.smithi114.stdout: "desc": "Enable/disable debug options", 2024-04-23T10:41:08.893 INFO:teuthology.orchestra.run.smithi114.stdout: "long_desc": "", 2024-04-23T10:41:08.893 INFO:teuthology.orchestra.run.smithi114.stdout: "tags": [], 2024-04-23T10:41:08.894 INFO:teuthology.orchestra.run.smithi114.stdout: "see_also": [] 2024-04-23T10:41:08.894 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-23T10:41:08.894 INFO:teuthology.orchestra.run.smithi114.stdout: "jwt_token_ttl": { 2024-04-23T10:41:08.894 INFO:teuthology.orchestra.run.smithi114.stdout: "name": "jwt_token_ttl", 2024-04-23T10:41:08.894 INFO:teuthology.orchestra.run.smithi114.stdout: "type": "int", 2024-04-23T10:41:08.894 INFO:teuthology.orchestra.run.smithi114.stdout: "level": "advanced", 2024-04-23T10:41:08.983 INFO:teuthology.orchestra.run.smithi114.stdout: "flags": 0, 2024-04-23T10:41:08.984 INFO:teuthology.orchestra.run.smithi114.stdout: "default_value": "28800", 2024-04-23T10:41:08.984 INFO:teuthology.orchestra.run.smithi114.stdout: "min": "", 2024-04-23T10:41:08.984 INFO:teuthology.orchestra.run.smithi114.stdout: "max": "", 2024-04-23T10:41:08.984 INFO:teuthology.orchestra.run.smithi114.stdout: "enum_allowed": [], 2024-04-23T10:41:08.984 INFO:teuthology.orchestra.run.smithi114.stdout: "desc": "", 2024-04-23T10:41:08.984 INFO:teuthology.orchestra.run.smithi114.stdout: "long_desc": "", 2024-04-23T10:41:08.984 INFO:teuthology.orchestra.run.smithi114.stdout: "tags": [], 2024-04-23T10:41:08.984 INFO:teuthology.orchestra.run.smithi114.stdout: "see_also": [] 2024-04-23T10:41:08.984 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-23T10:41:08.985 INFO:teuthology.orchestra.run.smithi114.stdout: "key_file": { 2024-04-23T10:41:08.985 INFO:teuthology.orchestra.run.smithi114.stdout: "name": "key_file", 2024-04-23T10:41:08.985 INFO:teuthology.orchestra.run.smithi114.stdout: "type": "str", 2024-04-23T10:41:08.985 INFO:teuthology.orchestra.run.smithi114.stdout: "level": "advanced", 2024-04-23T10:41:08.985 INFO:teuthology.orchestra.run.smithi114.stdout: "flags": 0, 2024-04-23T10:41:08.985 INFO:teuthology.orchestra.run.smithi114.stdout: "default_value": "", 2024-04-23T10:41:08.985 INFO:teuthology.orchestra.run.smithi114.stdout: "min": "", 2024-04-23T10:41:08.985 INFO:teuthology.orchestra.run.smithi114.stdout: "max": "", 2024-04-23T10:41:08.986 INFO:teuthology.orchestra.run.smithi114.stdout: "enum_allowed": [], 2024-04-23T10:41:08.986 INFO:teuthology.orchestra.run.smithi114.stdout: "desc": "", 2024-04-23T10:41:08.986 INFO:teuthology.orchestra.run.smithi114.stdout: "long_desc": "", 2024-04-23T10:41:08.986 INFO:teuthology.orchestra.run.smithi114.stdout: "tags": [], 2024-04-23T10:41:08.986 INFO:teuthology.orchestra.run.smithi114.stdout: "see_also": [] 2024-04-23T10:41:08.986 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-23T10:41:08.986 INFO:teuthology.orchestra.run.smithi114.stdout: "log_level": { 2024-04-23T10:41:08.986 INFO:teuthology.orchestra.run.smithi114.stdout: "name": "log_level", 2024-04-23T10:41:08.986 INFO:teuthology.orchestra.run.smithi114.stdout: "type": "str", 2024-04-23T10:41:08.987 INFO:teuthology.orchestra.run.smithi114.stdout: "level": "advanced", 2024-04-23T10:41:08.987 INFO:teuthology.orchestra.run.smithi114.stdout: "flags": 1, 2024-04-23T10:41:08.987 INFO:teuthology.orchestra.run.smithi114.stdout: "default_value": "", 2024-04-23T10:41:08.987 INFO:teuthology.orchestra.run.smithi114.stdout: "min": "", 2024-04-23T10:41:08.987 INFO:teuthology.orchestra.run.smithi114.stdout: "max": "", 2024-04-23T10:41:08.987 INFO:teuthology.orchestra.run.smithi114.stdout: "enum_allowed": [ 2024-04-23T10:41:08.987 INFO:teuthology.orchestra.run.smithi114.stdout: "", 2024-04-23T10:41:08.987 INFO:teuthology.orchestra.run.smithi114.stdout: "critical", 2024-04-23T10:41:08.987 INFO:teuthology.orchestra.run.smithi114.stdout: "debug", 2024-04-23T10:41:08.988 INFO:teuthology.orchestra.run.smithi114.stdout: "error", 2024-04-23T10:41:08.988 INFO:teuthology.orchestra.run.smithi114.stdout: "info", 2024-04-23T10:41:08.988 INFO:teuthology.orchestra.run.smithi114.stdout: "warning" 2024-04-23T10:41:08.988 INFO:teuthology.orchestra.run.smithi114.stdout: ], 2024-04-23T10:41:08.988 INFO:teuthology.orchestra.run.smithi114.stdout: "desc": "", 2024-04-23T10:41:08.988 INFO:teuthology.orchestra.run.smithi114.stdout: "long_desc": "", 2024-04-23T10:41:08.988 INFO:teuthology.orchestra.run.smithi114.stdout: "tags": [], 2024-04-23T10:41:08.988 INFO:teuthology.orchestra.run.smithi114.stdout: "see_also": [] 2024-04-23T10:41:08.989 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-23T10:41:08.989 INFO:teuthology.orchestra.run.smithi114.stdout: "log_to_cluster": { 2024-04-23T10:41:08.989 INFO:teuthology.orchestra.run.smithi114.stdout: "name": "log_to_cluster", 2024-04-23T10:41:08.989 INFO:teuthology.orchestra.run.smithi114.stdout: "type": "bool", 2024-04-23T10:41:08.989 INFO:teuthology.orchestra.run.smithi114.stdout: "level": "advanced", 2024-04-23T10:41:08.989 INFO:teuthology.orchestra.run.smithi114.stdout: "flags": 1, 2024-04-23T10:41:08.989 INFO:teuthology.orchestra.run.smithi114.stdout: "default_value": "False", 2024-04-23T10:41:08.989 INFO:teuthology.orchestra.run.smithi114.stdout: "min": "", 2024-04-23T10:41:08.990 INFO:teuthology.orchestra.run.smithi114.stdout: "max": "", 2024-04-23T10:41:08.990 INFO:teuthology.orchestra.run.smithi114.stdout: "enum_allowed": [], 2024-04-23T10:41:08.990 INFO:teuthology.orchestra.run.smithi114.stdout: "desc": "", 2024-04-23T10:41:08.990 INFO:teuthology.orchestra.run.smithi114.stdout: "long_desc": "", 2024-04-23T10:41:08.990 INFO:teuthology.orchestra.run.smithi114.stdout: "tags": [], 2024-04-23T10:41:08.990 INFO:teuthology.orchestra.run.smithi114.stdout: "see_also": [] 2024-04-23T10:41:08.990 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-23T10:41:08.990 INFO:teuthology.orchestra.run.smithi114.stdout: "log_to_cluster_level": { 2024-04-23T10:41:08.991 INFO:teuthology.orchestra.run.smithi114.stdout: "name": "log_to_cluster_level", 2024-04-23T10:41:08.991 INFO:teuthology.orchestra.run.smithi114.stdout: "type": "str", 2024-04-23T10:41:08.991 INFO:teuthology.orchestra.run.smithi114.stdout: "level": "advanced", 2024-04-23T10:41:08.991 INFO:teuthology.orchestra.run.smithi114.stdout: "flags": 1, 2024-04-23T10:41:08.991 INFO:teuthology.orchestra.run.smithi114.stdout: "default_value": "info", 2024-04-23T10:41:08.991 INFO:teuthology.orchestra.run.smithi114.stdout: "min": "", 2024-04-23T10:41:08.991 INFO:teuthology.orchestra.run.smithi114.stdout: "max": "", 2024-04-23T10:41:08.991 INFO:teuthology.orchestra.run.smithi114.stdout: "enum_allowed": [ 2024-04-23T10:41:08.992 INFO:teuthology.orchestra.run.smithi114.stdout: "", 2024-04-23T10:41:08.992 INFO:teuthology.orchestra.run.smithi114.stdout: "critical", 2024-04-23T10:41:08.992 INFO:teuthology.orchestra.run.smithi114.stdout: "debug", 2024-04-23T10:41:08.992 INFO:teuthology.orchestra.run.smithi114.stdout: "error", 2024-04-23T10:41:08.992 INFO:teuthology.orchestra.run.smithi114.stdout: "info", 2024-04-23T10:41:08.992 INFO:teuthology.orchestra.run.smithi114.stdout: "warning" 2024-04-23T10:41:08.992 INFO:teuthology.orchestra.run.smithi114.stdout: ], 2024-04-23T10:41:08.992 INFO:teuthology.orchestra.run.smithi114.stdout: "desc": "", 2024-04-23T10:41:08.992 INFO:teuthology.orchestra.run.smithi114.stdout: "long_desc": "", 2024-04-23T10:41:08.993 INFO:teuthology.orchestra.run.smithi114.stdout: "tags": [], 2024-04-23T10:41:08.993 INFO:teuthology.orchestra.run.smithi114.stdout: "see_also": [] 2024-04-23T10:41:08.993 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-23T10:41:08.993 INFO:teuthology.orchestra.run.smithi114.stdout: "log_to_file": { 2024-04-23T10:41:08.993 INFO:teuthology.orchestra.run.smithi114.stdout: "name": "log_to_file", 2024-04-23T10:41:08.993 INFO:teuthology.orchestra.run.smithi114.stdout: "type": "bool", 2024-04-23T10:41:08.993 INFO:teuthology.orchestra.run.smithi114.stdout: "level": "advanced", 2024-04-23T10:41:08.993 INFO:teuthology.orchestra.run.smithi114.stdout: "flags": 1, 2024-04-23T10:41:08.994 INFO:teuthology.orchestra.run.smithi114.stdout: "default_value": "False", 2024-04-23T10:41:08.994 INFO:teuthology.orchestra.run.smithi114.stdout: "min": "", 2024-04-23T10:41:08.994 INFO:teuthology.orchestra.run.smithi114.stdout: "max": "", 2024-04-23T10:41:08.994 INFO:teuthology.orchestra.run.smithi114.stdout: "enum_allowed": [], 2024-04-23T10:41:08.994 INFO:teuthology.orchestra.run.smithi114.stdout: "desc": "", 2024-04-23T10:41:08.994 INFO:teuthology.orchestra.run.smithi114.stdout: "long_desc": "", 2024-04-23T10:41:08.994 INFO:teuthology.orchestra.run.smithi114.stdout: "tags": [], 2024-04-23T10:41:08.994 INFO:teuthology.orchestra.run.smithi114.stdout: "see_also": [] 2024-04-23T10:41:08.994 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-23T10:41:08.995 INFO:teuthology.orchestra.run.smithi114.stdout: "motd": { 2024-04-23T10:41:08.995 INFO:teuthology.orchestra.run.smithi114.stdout: "name": "motd", 2024-04-23T10:41:08.995 INFO:teuthology.orchestra.run.smithi114.stdout: "type": "str", 2024-04-23T10:41:08.995 INFO:teuthology.orchestra.run.smithi114.stdout: "level": "advanced", 2024-04-23T10:41:08.995 INFO:teuthology.orchestra.run.smithi114.stdout: "flags": 0, 2024-04-23T10:41:08.995 INFO:teuthology.orchestra.run.smithi114.stdout: "default_value": "", 2024-04-23T10:41:08.995 INFO:teuthology.orchestra.run.smithi114.stdout: "min": "", 2024-04-23T10:41:08.995 INFO:teuthology.orchestra.run.smithi114.stdout: "max": "", 2024-04-23T10:41:08.996 INFO:teuthology.orchestra.run.smithi114.stdout: "enum_allowed": [], 2024-04-23T10:41:08.996 INFO:teuthology.orchestra.run.smithi114.stdout: "desc": "The message of the day", 2024-04-23T10:41:08.996 INFO:teuthology.orchestra.run.smithi114.stdout: "long_desc": "", 2024-04-23T10:41:08.996 INFO:teuthology.orchestra.run.smithi114.stdout: "tags": [], 2024-04-23T10:41:08.996 INFO:teuthology.orchestra.run.smithi114.stdout: "see_also": [] 2024-04-23T10:41:08.996 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-23T10:41:08.996 INFO:teuthology.orchestra.run.smithi114.stdout: "redirect_resolve_ip_addr": { 2024-04-23T10:41:08.996 INFO:teuthology.orchestra.run.smithi114.stdout: "name": "redirect_resolve_ip_addr", 2024-04-23T10:41:08.997 INFO:teuthology.orchestra.run.smithi114.stdout: "type": "bool", 2024-04-23T10:41:08.997 INFO:teuthology.orchestra.run.smithi114.stdout: "level": "advanced", 2024-04-23T10:41:08.997 INFO:teuthology.orchestra.run.smithi114.stdout: "flags": 0, 2024-04-23T10:41:08.997 INFO:teuthology.orchestra.run.smithi114.stdout: "default_value": "False", 2024-04-23T10:41:08.997 INFO:teuthology.orchestra.run.smithi114.stdout: "min": "", 2024-04-23T10:41:08.997 INFO:teuthology.orchestra.run.smithi114.stdout: "max": "", 2024-04-23T10:41:08.997 INFO:teuthology.orchestra.run.smithi114.stdout: "enum_allowed": [], 2024-04-23T10:41:08.997 INFO:teuthology.orchestra.run.smithi114.stdout: "desc": "", 2024-04-23T10:41:08.998 INFO:teuthology.orchestra.run.smithi114.stdout: "long_desc": "", 2024-04-23T10:41:08.998 INFO:teuthology.orchestra.run.smithi114.stdout: "tags": [], 2024-04-23T10:41:08.998 INFO:teuthology.orchestra.run.smithi114.stdout: "see_also": [] 2024-04-23T10:41:08.998 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-23T10:41:08.998 INFO:teuthology.orchestra.run.smithi114.stdout: "server_addr": { 2024-04-23T10:41:08.998 INFO:teuthology.orchestra.run.smithi114.stdout: "name": "server_addr", 2024-04-23T10:41:08.998 INFO:teuthology.orchestra.run.smithi114.stdout: "type": "str", 2024-04-23T10:41:08.998 INFO:teuthology.orchestra.run.smithi114.stdout: "level": "advanced", 2024-04-23T10:41:08.998 INFO:teuthology.orchestra.run.smithi114.stdout: "flags": 0, 2024-04-23T10:41:08.999 INFO:teuthology.orchestra.run.smithi114.stdout: "default_value": "::", 2024-04-23T10:41:08.999 INFO:teuthology.orchestra.run.smithi114.stdout: "min": "", 2024-04-23T10:41:08.999 INFO:teuthology.orchestra.run.smithi114.stdout: "max": "", 2024-04-23T10:41:08.999 INFO:teuthology.orchestra.run.smithi114.stdout: "enum_allowed": [], 2024-04-23T10:41:08.999 INFO:teuthology.orchestra.run.smithi114.stdout: "desc": "", 2024-04-23T10:41:08.999 INFO:teuthology.orchestra.run.smithi114.stdout: "long_desc": "", 2024-04-23T10:41:08.999 INFO:teuthology.orchestra.run.smithi114.stdout: "tags": [], 2024-04-23T10:41:08.999 INFO:teuthology.orchestra.run.smithi114.stdout: "see_also": [] 2024-04-23T10:41:09.000 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-23T10:41:09.000 INFO:teuthology.orchestra.run.smithi114.stdout: "server_port": { 2024-04-23T10:41:09.000 INFO:teuthology.orchestra.run.smithi114.stdout: "name": "server_port", 2024-04-23T10:41:09.000 INFO:teuthology.orchestra.run.smithi114.stdout: "type": "int", 2024-04-23T10:41:09.000 INFO:teuthology.orchestra.run.smithi114.stdout: "level": "advanced", 2024-04-23T10:41:09.000 INFO:teuthology.orchestra.run.smithi114.stdout: "flags": 0, 2024-04-23T10:41:09.000 INFO:teuthology.orchestra.run.smithi114.stdout: "default_value": "8080", 2024-04-23T10:41:09.001 INFO:teuthology.orchestra.run.smithi114.stdout: "min": "", 2024-04-23T10:41:09.001 INFO:teuthology.orchestra.run.smithi114.stdout: "max": "", 2024-04-23T10:41:09.001 INFO:teuthology.orchestra.run.smithi114.stdout: "enum_allowed": [], 2024-04-23T10:41:09.001 INFO:teuthology.orchestra.run.smithi114.stdout: "desc": "", 2024-04-23T10:41:09.001 INFO:teuthology.orchestra.run.smithi114.stdout: "long_desc": "", 2024-04-23T10:41:09.001 INFO:teuthology.orchestra.run.smithi114.stdout: "tags": [], 2024-04-23T10:41:09.001 INFO:teuthology.orchestra.run.smithi114.stdout: "see_also": [] 2024-04-23T10:41:09.001 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-23T10:41:09.001 INFO:teuthology.orchestra.run.smithi114.stdout: "ssl": { 2024-04-23T10:41:09.002 INFO:teuthology.orchestra.run.smithi114.stdout: "name": "ssl", 2024-04-23T10:41:09.002 INFO:teuthology.orchestra.run.smithi114.stdout: "type": "bool", 2024-04-23T10:41:09.002 INFO:teuthology.orchestra.run.smithi114.stdout: "level": "advanced", 2024-04-23T10:41:09.002 INFO:teuthology.orchestra.run.smithi114.stdout: "flags": 0, 2024-04-23T10:41:09.002 INFO:teuthology.orchestra.run.smithi114.stdout: "default_value": "True", 2024-04-23T10:41:09.002 INFO:teuthology.orchestra.run.smithi114.stdout: "min": "", 2024-04-23T10:41:09.002 INFO:teuthology.orchestra.run.smithi114.stdout: "max": "", 2024-04-23T10:41:09.002 INFO:teuthology.orchestra.run.smithi114.stdout: "enum_allowed": [], 2024-04-23T10:41:09.003 INFO:teuthology.orchestra.run.smithi114.stdout: "desc": "", 2024-04-23T10:41:09.003 INFO:teuthology.orchestra.run.smithi114.stdout: "long_desc": "", 2024-04-23T10:41:09.003 INFO:teuthology.orchestra.run.smithi114.stdout: "tags": [], 2024-04-23T10:41:09.003 INFO:teuthology.orchestra.run.smithi114.stdout: "see_also": [] 2024-04-23T10:41:09.003 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-23T10:41:09.003 INFO:teuthology.orchestra.run.smithi114.stdout: "ssl_server_port": { 2024-04-23T10:41:09.003 INFO:teuthology.orchestra.run.smithi114.stdout: "name": "ssl_server_port", 2024-04-23T10:41:09.003 INFO:teuthology.orchestra.run.smithi114.stdout: "type": "int", 2024-04-23T10:41:09.003 INFO:teuthology.orchestra.run.smithi114.stdout: "level": "advanced", 2024-04-23T10:41:09.004 INFO:teuthology.orchestra.run.smithi114.stdout: "flags": 0, 2024-04-23T10:41:09.004 INFO:teuthology.orchestra.run.smithi114.stdout: "default_value": "8443", 2024-04-23T10:41:09.004 INFO:teuthology.orchestra.run.smithi114.stdout: "min": "", 2024-04-23T10:41:09.004 INFO:teuthology.orchestra.run.smithi114.stdout: "max": "", 2024-04-23T10:41:09.004 INFO:teuthology.orchestra.run.smithi114.stdout: "enum_allowed": [], 2024-04-23T10:41:09.004 INFO:teuthology.orchestra.run.smithi114.stdout: "desc": "", 2024-04-23T10:41:09.004 INFO:teuthology.orchestra.run.smithi114.stdout: "long_desc": "", 2024-04-23T10:41:09.004 INFO:teuthology.orchestra.run.smithi114.stdout: "tags": [], 2024-04-23T10:41:09.004 INFO:teuthology.orchestra.run.smithi114.stdout: "see_also": [] 2024-04-23T10:41:09.005 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-23T10:41:09.005 INFO:teuthology.orchestra.run.smithi114.stdout: "standby_behaviour": { 2024-04-23T10:41:09.005 INFO:teuthology.orchestra.run.smithi114.stdout: "name": "standby_behaviour", 2024-04-23T10:41:09.005 INFO:teuthology.orchestra.run.smithi114.stdout: "type": "str", 2024-04-23T10:41:09.005 INFO:teuthology.orchestra.run.smithi114.stdout: "level": "advanced", 2024-04-23T10:41:09.005 INFO:teuthology.orchestra.run.smithi114.stdout: "flags": 0, 2024-04-23T10:41:09.005 INFO:teuthology.orchestra.run.smithi114.stdout: "default_value": "redirect", 2024-04-23T10:41:09.005 INFO:teuthology.orchestra.run.smithi114.stdout: "min": "", 2024-04-23T10:41:09.005 INFO:teuthology.orchestra.run.smithi114.stdout: "max": "", 2024-04-23T10:41:09.006 INFO:teuthology.orchestra.run.smithi114.stdout: "enum_allowed": [ 2024-04-23T10:41:09.006 INFO:teuthology.orchestra.run.smithi114.stdout: "error", 2024-04-23T10:41:09.006 INFO:teuthology.orchestra.run.smithi114.stdout: "redirect" 2024-04-23T10:41:09.006 INFO:teuthology.orchestra.run.smithi114.stdout: ], 2024-04-23T10:41:09.006 INFO:teuthology.orchestra.run.smithi114.stdout: "desc": "", 2024-04-23T10:41:09.006 INFO:teuthology.orchestra.run.smithi114.stdout: "long_desc": "", 2024-04-23T10:41:09.006 INFO:teuthology.orchestra.run.smithi114.stdout: "tags": [], 2024-04-23T10:41:09.006 INFO:teuthology.orchestra.run.smithi114.stdout: "see_also": [] 2024-04-23T10:41:09.007 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-23T10:41:09.007 INFO:teuthology.orchestra.run.smithi114.stdout: "standby_error_status_code": { 2024-04-23T10:41:09.007 INFO:teuthology.orchestra.run.smithi114.stdout: "name": "standby_error_status_code", 2024-04-23T10:41:09.007 INFO:teuthology.orchestra.run.smithi114.stdout: "type": "int", 2024-04-23T10:41:09.007 INFO:teuthology.orchestra.run.smithi114.stdout: "level": "advanced", 2024-04-23T10:41:09.007 INFO:teuthology.orchestra.run.smithi114.stdout: "flags": 0, 2024-04-23T10:41:09.007 INFO:teuthology.orchestra.run.smithi114.stdout: "default_value": "500", 2024-04-23T10:41:09.007 INFO:teuthology.orchestra.run.smithi114.stdout: "min": "400", 2024-04-23T10:41:09.007 INFO:teuthology.orchestra.run.smithi114.stdout: "max": "599", 2024-04-23T10:41:09.008 INFO:teuthology.orchestra.run.smithi114.stdout: "enum_allowed": [], 2024-04-23T10:41:09.008 INFO:teuthology.orchestra.run.smithi114.stdout: "desc": "", 2024-04-23T10:41:09.008 INFO:teuthology.orchestra.run.smithi114.stdout: "long_desc": "", 2024-04-23T10:41:09.008 INFO:teuthology.orchestra.run.smithi114.stdout: "tags": [], 2024-04-23T10:41:09.008 INFO:teuthology.orchestra.run.smithi114.stdout: "see_also": [] 2024-04-23T10:41:09.008 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-23T10:41:09.008 INFO:teuthology.orchestra.run.smithi114.stdout: "url_prefix": { 2024-04-23T10:41:09.008 INFO:teuthology.orchestra.run.smithi114.stdout: "name": "url_prefix", 2024-04-23T10:41:09.009 INFO:teuthology.orchestra.run.smithi114.stdout: "type": "str", 2024-04-23T10:41:09.009 INFO:teuthology.orchestra.run.smithi114.stdout: "level": "advanced", 2024-04-23T10:41:09.009 INFO:teuthology.orchestra.run.smithi114.stdout: "flags": 0, 2024-04-23T10:41:09.009 INFO:teuthology.orchestra.run.smithi114.stdout: "default_value": "", 2024-04-23T10:41:09.009 INFO:teuthology.orchestra.run.smithi114.stdout: "min": "", 2024-04-23T10:41:09.009 INFO:teuthology.orchestra.run.smithi114.stdout: "max": "", 2024-04-23T10:41:09.009 INFO:teuthology.orchestra.run.smithi114.stdout: "enum_allowed": [], 2024-04-23T10:41:09.010 INFO:teuthology.orchestra.run.smithi114.stdout: "desc": "", 2024-04-23T10:41:09.010 INFO:teuthology.orchestra.run.smithi114.stdout: "long_desc": "", 2024-04-23T10:41:09.010 INFO:teuthology.orchestra.run.smithi114.stdout: "tags": [], 2024-04-23T10:41:09.010 INFO:teuthology.orchestra.run.smithi114.stdout: "see_also": [] 2024-04-23T10:41:09.010 INFO:teuthology.orchestra.run.smithi114.stdout: } 2024-04-23T10:41:09.010 INFO:teuthology.orchestra.run.smithi114.stdout: } 2024-04-23T10:41:09.010 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-23T10:41:09.010 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-04-23T10:41:09.011 INFO:teuthology.orchestra.run.smithi114.stdout: "name": "devicehealth", 2024-04-23T10:41:09.011 INFO:teuthology.orchestra.run.smithi114.stdout: "can_run": true, 2024-04-23T10:41:09.011 INFO:teuthology.orchestra.run.smithi114.stdout: "error_string": "", 2024-04-23T10:41:09.011 INFO:teuthology.orchestra.run.smithi114.stdout: "module_options": { 2024-04-23T10:41:09.011 INFO:teuthology.orchestra.run.smithi114.stdout: "enable_monitoring": { 2024-04-23T10:41:09.011 INFO:teuthology.orchestra.run.smithi114.stdout: "name": "enable_monitoring", 2024-04-23T10:41:09.011 INFO:teuthology.orchestra.run.smithi114.stdout: "type": "bool", 2024-04-23T10:41:09.011 INFO:teuthology.orchestra.run.smithi114.stdout: "level": "advanced", 2024-04-23T10:41:09.012 INFO:teuthology.orchestra.run.smithi114.stdout: "flags": 1, 2024-04-23T10:41:09.012 INFO:teuthology.orchestra.run.smithi114.stdout: "default_value": "True", 2024-04-23T10:41:09.012 INFO:teuthology.orchestra.run.smithi114.stdout: "min": "", 2024-04-23T10:41:09.012 INFO:teuthology.orchestra.run.smithi114.stdout: "max": "", 2024-04-23T10:41:09.012 INFO:teuthology.orchestra.run.smithi114.stdout: "enum_allowed": [], 2024-04-23T10:41:09.012 INFO:teuthology.orchestra.run.smithi114.stdout: "desc": "monitor device health metrics", 2024-04-23T10:41:09.012 INFO:teuthology.orchestra.run.smithi114.stdout: "long_desc": "", 2024-04-23T10:41:09.012 INFO:teuthology.orchestra.run.smithi114.stdout: "tags": [], 2024-04-23T10:41:09.012 INFO:teuthology.orchestra.run.smithi114.stdout: "see_also": [] 2024-04-23T10:41:09.012 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-23T10:41:09.013 INFO:teuthology.orchestra.run.smithi114.stdout: "log_level": { 2024-04-23T10:41:09.013 INFO:teuthology.orchestra.run.smithi114.stdout: "name": "log_level", 2024-04-23T10:41:09.013 INFO:teuthology.orchestra.run.smithi114.stdout: "type": "str", 2024-04-23T10:41:09.013 INFO:teuthology.orchestra.run.smithi114.stdout: "level": "advanced", 2024-04-23T10:41:09.013 INFO:teuthology.orchestra.run.smithi114.stdout: "flags": 1, 2024-04-23T10:41:09.013 INFO:teuthology.orchestra.run.smithi114.stdout: "default_value": "", 2024-04-23T10:41:09.013 INFO:teuthology.orchestra.run.smithi114.stdout: "min": "", 2024-04-23T10:41:09.013 INFO:teuthology.orchestra.run.smithi114.stdout: "max": "", 2024-04-23T10:41:09.013 INFO:teuthology.orchestra.run.smithi114.stdout: "enum_allowed": [ 2024-04-23T10:41:09.014 INFO:teuthology.orchestra.run.smithi114.stdout: "", 2024-04-23T10:41:09.014 INFO:teuthology.orchestra.run.smithi114.stdout: "critical", 2024-04-23T10:41:09.014 INFO:teuthology.orchestra.run.smithi114.stdout: "debug", 2024-04-23T10:41:09.014 INFO:teuthology.orchestra.run.smithi114.stdout: "error", 2024-04-23T10:41:09.014 INFO:teuthology.orchestra.run.smithi114.stdout: "info", 2024-04-23T10:41:09.014 INFO:teuthology.orchestra.run.smithi114.stdout: "warning" 2024-04-23T10:41:09.014 INFO:teuthology.orchestra.run.smithi114.stdout: ], 2024-04-23T10:41:09.014 INFO:teuthology.orchestra.run.smithi114.stdout: "desc": "", 2024-04-23T10:41:09.015 INFO:teuthology.orchestra.run.smithi114.stdout: "long_desc": "", 2024-04-23T10:41:09.015 INFO:teuthology.orchestra.run.smithi114.stdout: "tags": [], 2024-04-23T10:41:09.015 INFO:teuthology.orchestra.run.smithi114.stdout: "see_also": [] 2024-04-23T10:41:09.015 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-23T10:41:09.015 INFO:teuthology.orchestra.run.smithi114.stdout: "log_to_cluster": { 2024-04-23T10:41:09.015 INFO:teuthology.orchestra.run.smithi114.stdout: "name": "log_to_cluster", 2024-04-23T10:41:09.015 INFO:teuthology.orchestra.run.smithi114.stdout: "type": "bool", 2024-04-23T10:41:09.015 INFO:teuthology.orchestra.run.smithi114.stdout: "level": "advanced", 2024-04-23T10:41:09.015 INFO:teuthology.orchestra.run.smithi114.stdout: "flags": 1, 2024-04-23T10:41:09.016 INFO:teuthology.orchestra.run.smithi114.stdout: "default_value": "False", 2024-04-23T10:41:09.016 INFO:teuthology.orchestra.run.smithi114.stdout: "min": "", 2024-04-23T10:41:09.016 INFO:teuthology.orchestra.run.smithi114.stdout: "max": "", 2024-04-23T10:41:09.016 INFO:teuthology.orchestra.run.smithi114.stdout: "enum_allowed": [], 2024-04-23T10:41:09.016 INFO:teuthology.orchestra.run.smithi114.stdout: "desc": "", 2024-04-23T10:41:09.016 INFO:teuthology.orchestra.run.smithi114.stdout: "long_desc": "", 2024-04-23T10:41:09.016 INFO:teuthology.orchestra.run.smithi114.stdout: "tags": [], 2024-04-23T10:41:09.016 INFO:teuthology.orchestra.run.smithi114.stdout: "see_also": [] 2024-04-23T10:41:09.016 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-23T10:41:09.017 INFO:teuthology.orchestra.run.smithi114.stdout: "log_to_cluster_level": { 2024-04-23T10:41:09.017 INFO:teuthology.orchestra.run.smithi114.stdout: "name": "log_to_cluster_level", 2024-04-23T10:41:09.017 INFO:teuthology.orchestra.run.smithi114.stdout: "type": "str", 2024-04-23T10:41:09.017 INFO:teuthology.orchestra.run.smithi114.stdout: "level": "advanced", 2024-04-23T10:41:09.017 INFO:teuthology.orchestra.run.smithi114.stdout: "flags": 1, 2024-04-23T10:41:09.017 INFO:teuthology.orchestra.run.smithi114.stdout: "default_value": "info", 2024-04-23T10:41:09.017 INFO:teuthology.orchestra.run.smithi114.stdout: "min": "", 2024-04-23T10:41:09.017 INFO:teuthology.orchestra.run.smithi114.stdout: "max": "", 2024-04-23T10:41:09.017 INFO:teuthology.orchestra.run.smithi114.stdout: "enum_allowed": [ 2024-04-23T10:41:09.018 INFO:teuthology.orchestra.run.smithi114.stdout: "", 2024-04-23T10:41:09.018 INFO:teuthology.orchestra.run.smithi114.stdout: "critical", 2024-04-23T10:41:09.018 INFO:teuthology.orchestra.run.smithi114.stdout: "debug", 2024-04-23T10:41:09.018 INFO:teuthology.orchestra.run.smithi114.stdout: "error", 2024-04-23T10:41:09.018 INFO:teuthology.orchestra.run.smithi114.stdout: "info", 2024-04-23T10:41:09.018 INFO:teuthology.orchestra.run.smithi114.stdout: "warning" 2024-04-23T10:41:09.018 INFO:teuthology.orchestra.run.smithi114.stdout: ], 2024-04-23T10:41:09.018 INFO:teuthology.orchestra.run.smithi114.stdout: "desc": "", 2024-04-23T10:41:09.019 INFO:teuthology.orchestra.run.smithi114.stdout: "long_desc": "", 2024-04-23T10:41:09.019 INFO:teuthology.orchestra.run.smithi114.stdout: "tags": [], 2024-04-23T10:41:09.019 INFO:teuthology.orchestra.run.smithi114.stdout: "see_also": [] 2024-04-23T10:41:09.019 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-23T10:41:09.019 INFO:teuthology.orchestra.run.smithi114.stdout: "log_to_file": { 2024-04-23T10:41:09.019 INFO:teuthology.orchestra.run.smithi114.stdout: "name": "log_to_file", 2024-04-23T10:41:09.019 INFO:teuthology.orchestra.run.smithi114.stdout: "type": "bool", 2024-04-23T10:41:09.019 INFO:teuthology.orchestra.run.smithi114.stdout: "level": "advanced", 2024-04-23T10:41:09.020 INFO:teuthology.orchestra.run.smithi114.stdout: "flags": 1, 2024-04-23T10:41:09.020 INFO:teuthology.orchestra.run.smithi114.stdout: "default_value": "False", 2024-04-23T10:41:09.020 INFO:teuthology.orchestra.run.smithi114.stdout: "min": "", 2024-04-23T10:41:09.020 INFO:teuthology.orchestra.run.smithi114.stdout: "max": "", 2024-04-23T10:41:09.020 INFO:teuthology.orchestra.run.smithi114.stdout: "enum_allowed": [], 2024-04-23T10:41:09.020 INFO:teuthology.orchestra.run.smithi114.stdout: "desc": "", 2024-04-23T10:41:09.020 INFO:teuthology.orchestra.run.smithi114.stdout: "long_desc": "", 2024-04-23T10:41:09.020 INFO:teuthology.orchestra.run.smithi114.stdout: "tags": [], 2024-04-23T10:41:09.020 INFO:teuthology.orchestra.run.smithi114.stdout: "see_also": [] 2024-04-23T10:41:09.021 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-23T10:41:09.021 INFO:teuthology.orchestra.run.smithi114.stdout: "mark_out_threshold": { 2024-04-23T10:41:09.021 INFO:teuthology.orchestra.run.smithi114.stdout: "name": "mark_out_threshold", 2024-04-23T10:41:09.021 INFO:teuthology.orchestra.run.smithi114.stdout: "type": "secs", 2024-04-23T10:41:09.021 INFO:teuthology.orchestra.run.smithi114.stdout: "level": "advanced", 2024-04-23T10:41:09.021 INFO:teuthology.orchestra.run.smithi114.stdout: "flags": 1, 2024-04-23T10:41:09.021 INFO:teuthology.orchestra.run.smithi114.stdout: "default_value": "2419200", 2024-04-23T10:41:09.021 INFO:teuthology.orchestra.run.smithi114.stdout: "min": "", 2024-04-23T10:41:09.022 INFO:teuthology.orchestra.run.smithi114.stdout: "max": "", 2024-04-23T10:41:09.022 INFO:teuthology.orchestra.run.smithi114.stdout: "enum_allowed": [], 2024-04-23T10:41:09.022 INFO:teuthology.orchestra.run.smithi114.stdout: "desc": "automatically mark OSD if it may fail before this long", 2024-04-23T10:41:09.022 INFO:teuthology.orchestra.run.smithi114.stdout: "long_desc": "", 2024-04-23T10:41:09.022 INFO:teuthology.orchestra.run.smithi114.stdout: "tags": [], 2024-04-23T10:41:09.022 INFO:teuthology.orchestra.run.smithi114.stdout: "see_also": [] 2024-04-23T10:41:09.022 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-23T10:41:09.022 INFO:teuthology.orchestra.run.smithi114.stdout: "pool_name": { 2024-04-23T10:41:09.022 INFO:teuthology.orchestra.run.smithi114.stdout: "name": "pool_name", 2024-04-23T10:41:09.023 INFO:teuthology.orchestra.run.smithi114.stdout: "type": "str", 2024-04-23T10:41:09.023 INFO:teuthology.orchestra.run.smithi114.stdout: "level": "advanced", 2024-04-23T10:41:09.023 INFO:teuthology.orchestra.run.smithi114.stdout: "flags": 1, 2024-04-23T10:41:09.023 INFO:teuthology.orchestra.run.smithi114.stdout: "default_value": "device_health_metrics", 2024-04-23T10:41:09.023 INFO:teuthology.orchestra.run.smithi114.stdout: "min": "", 2024-04-23T10:41:09.023 INFO:teuthology.orchestra.run.smithi114.stdout: "max": "", 2024-04-23T10:41:09.023 INFO:teuthology.orchestra.run.smithi114.stdout: "enum_allowed": [], 2024-04-23T10:41:09.023 INFO:teuthology.orchestra.run.smithi114.stdout: "desc": "name of pool in which to store device health metrics", 2024-04-23T10:41:09.023 INFO:teuthology.orchestra.run.smithi114.stdout: "long_desc": "", 2024-04-23T10:41:09.024 INFO:teuthology.orchestra.run.smithi114.stdout: "tags": [], 2024-04-23T10:41:09.024 INFO:teuthology.orchestra.run.smithi114.stdout: "see_also": [] 2024-04-23T10:41:09.024 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-23T10:41:09.024 INFO:teuthology.orchestra.run.smithi114.stdout: "retention_period": { 2024-04-23T10:41:09.024 INFO:teuthology.orchestra.run.smithi114.stdout: "name": "retention_period", 2024-04-23T10:41:09.024 INFO:teuthology.orchestra.run.smithi114.stdout: "type": "secs", 2024-04-23T10:41:09.024 INFO:teuthology.orchestra.run.smithi114.stdout: "level": "advanced", 2024-04-23T10:41:09.024 INFO:teuthology.orchestra.run.smithi114.stdout: "flags": 1, 2024-04-23T10:41:09.024 INFO:teuthology.orchestra.run.smithi114.stdout: "default_value": "15552000", 2024-04-23T10:41:09.025 INFO:teuthology.orchestra.run.smithi114.stdout: "min": "", 2024-04-23T10:41:09.025 INFO:teuthology.orchestra.run.smithi114.stdout: "max": "", 2024-04-23T10:41:09.025 INFO:teuthology.orchestra.run.smithi114.stdout: "enum_allowed": [], 2024-04-23T10:41:09.025 INFO:teuthology.orchestra.run.smithi114.stdout: "desc": "how long to retain device health metrics", 2024-04-23T10:41:09.025 INFO:teuthology.orchestra.run.smithi114.stdout: "long_desc": "", 2024-04-23T10:41:09.025 INFO:teuthology.orchestra.run.smithi114.stdout: "tags": [], 2024-04-23T10:41:09.025 INFO:teuthology.orchestra.run.smithi114.stdout: "see_also": [] 2024-04-23T10:41:09.025 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-23T10:41:09.025 INFO:teuthology.orchestra.run.smithi114.stdout: "scrape_frequency": { 2024-04-23T10:41:09.026 INFO:teuthology.orchestra.run.smithi114.stdout: "name": "scrape_frequency", 2024-04-23T10:41:09.026 INFO:teuthology.orchestra.run.smithi114.stdout: "type": "secs", 2024-04-23T10:41:09.026 INFO:teuthology.orchestra.run.smithi114.stdout: "level": "advanced", 2024-04-23T10:41:09.026 INFO:teuthology.orchestra.run.smithi114.stdout: "flags": 1, 2024-04-23T10:41:09.026 INFO:teuthology.orchestra.run.smithi114.stdout: "default_value": "86400", 2024-04-23T10:41:09.026 INFO:teuthology.orchestra.run.smithi114.stdout: "min": "", 2024-04-23T10:41:09.026 INFO:teuthology.orchestra.run.smithi114.stdout: "max": "", 2024-04-23T10:41:09.026 INFO:teuthology.orchestra.run.smithi114.stdout: "enum_allowed": [], 2024-04-23T10:41:09.026 INFO:teuthology.orchestra.run.smithi114.stdout: "desc": "how frequently to scrape device health metrics", 2024-04-23T10:41:09.027 INFO:teuthology.orchestra.run.smithi114.stdout: "long_desc": "", 2024-04-23T10:41:09.027 INFO:teuthology.orchestra.run.smithi114.stdout: "tags": [], 2024-04-23T10:41:09.027 INFO:teuthology.orchestra.run.smithi114.stdout: "see_also": [] 2024-04-23T10:41:09.027 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-23T10:41:09.027 INFO:teuthology.orchestra.run.smithi114.stdout: "self_heal": { 2024-04-23T10:41:09.027 INFO:teuthology.orchestra.run.smithi114.stdout: "name": "self_heal", 2024-04-23T10:41:09.027 INFO:teuthology.orchestra.run.smithi114.stdout: "type": "bool", 2024-04-23T10:41:09.027 INFO:teuthology.orchestra.run.smithi114.stdout: "level": "advanced", 2024-04-23T10:41:09.027 INFO:teuthology.orchestra.run.smithi114.stdout: "flags": 1, 2024-04-23T10:41:09.028 INFO:teuthology.orchestra.run.smithi114.stdout: "default_value": "True", 2024-04-23T10:41:09.028 INFO:teuthology.orchestra.run.smithi114.stdout: "min": "", 2024-04-23T10:41:09.028 INFO:teuthology.orchestra.run.smithi114.stdout: "max": "", 2024-04-23T10:41:09.028 INFO:teuthology.orchestra.run.smithi114.stdout: "enum_allowed": [], 2024-04-23T10:41:09.028 INFO:teuthology.orchestra.run.smithi114.stdout: "desc": "preemptively heal cluster around devices that may fail", 2024-04-23T10:41:09.028 INFO:teuthology.orchestra.run.smithi114.stdout: "long_desc": "", 2024-04-23T10:41:09.028 INFO:teuthology.orchestra.run.smithi114.stdout: "tags": [], 2024-04-23T10:41:09.028 INFO:teuthology.orchestra.run.smithi114.stdout: "see_also": [] 2024-04-23T10:41:09.029 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-23T10:41:09.029 INFO:teuthology.orchestra.run.smithi114.stdout: "sleep_interval": { 2024-04-23T10:41:09.029 INFO:teuthology.orchestra.run.smithi114.stdout: "name": "sleep_interval", 2024-04-23T10:41:09.029 INFO:teuthology.orchestra.run.smithi114.stdout: "type": "secs", 2024-04-23T10:41:09.029 INFO:teuthology.orchestra.run.smithi114.stdout: "level": "advanced", 2024-04-23T10:41:09.029 INFO:teuthology.orchestra.run.smithi114.stdout: "flags": 1, 2024-04-23T10:41:09.029 INFO:teuthology.orchestra.run.smithi114.stdout: "default_value": "600", 2024-04-23T10:41:09.029 INFO:teuthology.orchestra.run.smithi114.stdout: "min": "", 2024-04-23T10:41:09.029 INFO:teuthology.orchestra.run.smithi114.stdout: "max": "", 2024-04-23T10:41:09.029 INFO:teuthology.orchestra.run.smithi114.stdout: "enum_allowed": [], 2024-04-23T10:41:09.030 INFO:teuthology.orchestra.run.smithi114.stdout: "desc": "how frequently to wake up and check device health", 2024-04-23T10:41:09.030 INFO:teuthology.orchestra.run.smithi114.stdout: "long_desc": "", 2024-04-23T10:41:09.030 INFO:teuthology.orchestra.run.smithi114.stdout: "tags": [], 2024-04-23T10:41:09.030 INFO:teuthology.orchestra.run.smithi114.stdout: "see_also": [] 2024-04-23T10:41:09.030 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-23T10:41:09.030 INFO:teuthology.orchestra.run.smithi114.stdout: "warn_threshold": { 2024-04-23T10:41:09.030 INFO:teuthology.orchestra.run.smithi114.stdout: "name": "warn_threshold", 2024-04-23T10:41:09.030 INFO:teuthology.orchestra.run.smithi114.stdout: "type": "secs", 2024-04-23T10:41:09.031 INFO:teuthology.orchestra.run.smithi114.stdout: "level": "advanced", 2024-04-23T10:41:09.031 INFO:teuthology.orchestra.run.smithi114.stdout: "flags": 1, 2024-04-23T10:41:09.031 INFO:teuthology.orchestra.run.smithi114.stdout: "default_value": "7257600", 2024-04-23T10:41:09.031 INFO:teuthology.orchestra.run.smithi114.stdout: "min": "", 2024-04-23T10:41:09.031 INFO:teuthology.orchestra.run.smithi114.stdout: "max": "", 2024-04-23T10:41:09.031 INFO:teuthology.orchestra.run.smithi114.stdout: "enum_allowed": [], 2024-04-23T10:41:09.031 INFO:teuthology.orchestra.run.smithi114.stdout: "desc": "raise health warning if OSD may fail before this long", 2024-04-23T10:41:09.031 INFO:teuthology.orchestra.run.smithi114.stdout: "long_desc": "", 2024-04-23T10:41:09.031 INFO:teuthology.orchestra.run.smithi114.stdout: "tags": [], 2024-04-23T10:41:09.032 INFO:teuthology.orchestra.run.smithi114.stdout: "see_also": [] 2024-04-23T10:41:09.032 INFO:teuthology.orchestra.run.smithi114.stdout: } 2024-04-23T10:41:09.032 INFO:teuthology.orchestra.run.smithi114.stdout: } 2024-04-23T10:41:09.032 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-23T10:41:09.032 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-04-23T10:41:09.032 INFO:teuthology.orchestra.run.smithi114.stdout: "name": "diskprediction_local", 2024-04-23T10:41:09.032 INFO:teuthology.orchestra.run.smithi114.stdout: "can_run": true, 2024-04-23T10:41:09.032 INFO:teuthology.orchestra.run.smithi114.stdout: "error_string": "", 2024-04-23T10:41:09.032 INFO:teuthology.orchestra.run.smithi114.stdout: "module_options": { 2024-04-23T10:41:09.033 INFO:teuthology.orchestra.run.smithi114.stdout: "log_level": { 2024-04-23T10:41:09.033 INFO:teuthology.orchestra.run.smithi114.stdout: "name": "log_level", 2024-04-23T10:41:09.033 INFO:teuthology.orchestra.run.smithi114.stdout: "type": "str", 2024-04-23T10:41:09.033 INFO:teuthology.orchestra.run.smithi114.stdout: "level": "advanced", 2024-04-23T10:41:09.033 INFO:teuthology.orchestra.run.smithi114.stdout: "flags": 1, 2024-04-23T10:41:09.033 INFO:teuthology.orchestra.run.smithi114.stdout: "default_value": "", 2024-04-23T10:41:09.033 INFO:teuthology.orchestra.run.smithi114.stdout: "min": "", 2024-04-23T10:41:09.033 INFO:teuthology.orchestra.run.smithi114.stdout: "max": "", 2024-04-23T10:41:09.033 INFO:teuthology.orchestra.run.smithi114.stdout: "enum_allowed": [ 2024-04-23T10:41:09.034 INFO:teuthology.orchestra.run.smithi114.stdout: "", 2024-04-23T10:41:09.034 INFO:teuthology.orchestra.run.smithi114.stdout: "critical", 2024-04-23T10:41:09.034 INFO:teuthology.orchestra.run.smithi114.stdout: "debug", 2024-04-23T10:41:09.034 INFO:teuthology.orchestra.run.smithi114.stdout: "error", 2024-04-23T10:41:09.034 INFO:teuthology.orchestra.run.smithi114.stdout: "info", 2024-04-23T10:41:09.034 INFO:teuthology.orchestra.run.smithi114.stdout: "warning" 2024-04-23T10:41:09.034 INFO:teuthology.orchestra.run.smithi114.stdout: ], 2024-04-23T10:41:09.034 INFO:teuthology.orchestra.run.smithi114.stdout: "desc": "", 2024-04-23T10:41:09.034 INFO:teuthology.orchestra.run.smithi114.stdout: "long_desc": "", 2024-04-23T10:41:09.035 INFO:teuthology.orchestra.run.smithi114.stdout: "tags": [], 2024-04-23T10:41:09.035 INFO:teuthology.orchestra.run.smithi114.stdout: "see_also": [] 2024-04-23T10:41:09.035 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-23T10:41:09.035 INFO:teuthology.orchestra.run.smithi114.stdout: "log_to_cluster": { 2024-04-23T10:41:09.035 INFO:teuthology.orchestra.run.smithi114.stdout: "name": "log_to_cluster", 2024-04-23T10:41:09.035 INFO:teuthology.orchestra.run.smithi114.stdout: "type": "bool", 2024-04-23T10:41:09.035 INFO:teuthology.orchestra.run.smithi114.stdout: "level": "advanced", 2024-04-23T10:41:09.035 INFO:teuthology.orchestra.run.smithi114.stdout: "flags": 1, 2024-04-23T10:41:09.035 INFO:teuthology.orchestra.run.smithi114.stdout: "default_value": "False", 2024-04-23T10:41:09.036 INFO:teuthology.orchestra.run.smithi114.stdout: "min": "", 2024-04-23T10:41:09.036 INFO:teuthology.orchestra.run.smithi114.stdout: "max": "", 2024-04-23T10:41:09.036 INFO:teuthology.orchestra.run.smithi114.stdout: "enum_allowed": [], 2024-04-23T10:41:09.036 INFO:teuthology.orchestra.run.smithi114.stdout: "desc": "", 2024-04-23T10:41:09.036 INFO:teuthology.orchestra.run.smithi114.stdout: "long_desc": "", 2024-04-23T10:41:09.036 INFO:teuthology.orchestra.run.smithi114.stdout: "tags": [], 2024-04-23T10:41:09.036 INFO:teuthology.orchestra.run.smithi114.stdout: "see_also": [] 2024-04-23T10:41:09.036 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-23T10:41:09.037 INFO:teuthology.orchestra.run.smithi114.stdout: "log_to_cluster_level": { 2024-04-23T10:41:09.037 INFO:teuthology.orchestra.run.smithi114.stdout: "name": "log_to_cluster_level", 2024-04-23T10:41:09.037 INFO:teuthology.orchestra.run.smithi114.stdout: "type": "str", 2024-04-23T10:41:09.037 INFO:teuthology.orchestra.run.smithi114.stdout: "level": "advanced", 2024-04-23T10:41:09.037 INFO:teuthology.orchestra.run.smithi114.stdout: "flags": 1, 2024-04-23T10:41:09.037 INFO:teuthology.orchestra.run.smithi114.stdout: "default_value": "info", 2024-04-23T10:41:09.037 INFO:teuthology.orchestra.run.smithi114.stdout: "min": "", 2024-04-23T10:41:09.037 INFO:teuthology.orchestra.run.smithi114.stdout: "max": "", 2024-04-23T10:41:09.037 INFO:teuthology.orchestra.run.smithi114.stdout: "enum_allowed": [ 2024-04-23T10:41:09.037 INFO:teuthology.orchestra.run.smithi114.stdout: "", 2024-04-23T10:41:09.038 INFO:teuthology.orchestra.run.smithi114.stdout: "critical", 2024-04-23T10:41:09.038 INFO:teuthology.orchestra.run.smithi114.stdout: "debug", 2024-04-23T10:41:09.038 INFO:teuthology.orchestra.run.smithi114.stdout: "error", 2024-04-23T10:41:09.038 INFO:teuthology.orchestra.run.smithi114.stdout: "info", 2024-04-23T10:41:09.038 INFO:teuthology.orchestra.run.smithi114.stdout: "warning" 2024-04-23T10:41:09.038 INFO:teuthology.orchestra.run.smithi114.stdout: ], 2024-04-23T10:41:09.038 INFO:teuthology.orchestra.run.smithi114.stdout: "desc": "", 2024-04-23T10:41:09.038 INFO:teuthology.orchestra.run.smithi114.stdout: "long_desc": "", 2024-04-23T10:41:09.039 INFO:teuthology.orchestra.run.smithi114.stdout: "tags": [], 2024-04-23T10:41:09.039 INFO:teuthology.orchestra.run.smithi114.stdout: "see_also": [] 2024-04-23T10:41:09.039 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-23T10:41:09.039 INFO:teuthology.orchestra.run.smithi114.stdout: "log_to_file": { 2024-04-23T10:41:09.039 INFO:teuthology.orchestra.run.smithi114.stdout: "name": "log_to_file", 2024-04-23T10:41:09.039 INFO:teuthology.orchestra.run.smithi114.stdout: "type": "bool", 2024-04-23T10:41:09.039 INFO:teuthology.orchestra.run.smithi114.stdout: "level": "advanced", 2024-04-23T10:41:09.039 INFO:teuthology.orchestra.run.smithi114.stdout: "flags": 1, 2024-04-23T10:41:09.039 INFO:teuthology.orchestra.run.smithi114.stdout: "default_value": "False", 2024-04-23T10:41:09.040 INFO:teuthology.orchestra.run.smithi114.stdout: "min": "", 2024-04-23T10:41:09.040 INFO:teuthology.orchestra.run.smithi114.stdout: "max": "", 2024-04-23T10:41:09.040 INFO:teuthology.orchestra.run.smithi114.stdout: "enum_allowed": [], 2024-04-23T10:41:09.040 INFO:teuthology.orchestra.run.smithi114.stdout: "desc": "", 2024-04-23T10:41:09.040 INFO:teuthology.orchestra.run.smithi114.stdout: "long_desc": "", 2024-04-23T10:41:09.040 INFO:teuthology.orchestra.run.smithi114.stdout: "tags": [], 2024-04-23T10:41:09.040 INFO:teuthology.orchestra.run.smithi114.stdout: "see_also": [] 2024-04-23T10:41:09.040 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-23T10:41:09.041 INFO:teuthology.orchestra.run.smithi114.stdout: "predict_interval": { 2024-04-23T10:41:09.041 INFO:teuthology.orchestra.run.smithi114.stdout: "name": "predict_interval", 2024-04-23T10:41:09.041 INFO:teuthology.orchestra.run.smithi114.stdout: "type": "str", 2024-04-23T10:41:09.041 INFO:teuthology.orchestra.run.smithi114.stdout: "level": "advanced", 2024-04-23T10:41:09.041 INFO:teuthology.orchestra.run.smithi114.stdout: "flags": 0, 2024-04-23T10:41:09.041 INFO:teuthology.orchestra.run.smithi114.stdout: "default_value": "86400", 2024-04-23T10:41:09.041 INFO:teuthology.orchestra.run.smithi114.stdout: "min": "", 2024-04-23T10:41:09.041 INFO:teuthology.orchestra.run.smithi114.stdout: "max": "", 2024-04-23T10:41:09.041 INFO:teuthology.orchestra.run.smithi114.stdout: "enum_allowed": [], 2024-04-23T10:41:09.042 INFO:teuthology.orchestra.run.smithi114.stdout: "desc": "", 2024-04-23T10:41:09.042 INFO:teuthology.orchestra.run.smithi114.stdout: "long_desc": "", 2024-04-23T10:41:09.042 INFO:teuthology.orchestra.run.smithi114.stdout: "tags": [], 2024-04-23T10:41:09.042 INFO:teuthology.orchestra.run.smithi114.stdout: "see_also": [] 2024-04-23T10:41:09.042 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-23T10:41:09.042 INFO:teuthology.orchestra.run.smithi114.stdout: "predictor_model": { 2024-04-23T10:41:09.042 INFO:teuthology.orchestra.run.smithi114.stdout: "name": "predictor_model", 2024-04-23T10:41:09.042 INFO:teuthology.orchestra.run.smithi114.stdout: "type": "str", 2024-04-23T10:41:09.042 INFO:teuthology.orchestra.run.smithi114.stdout: "level": "advanced", 2024-04-23T10:41:09.043 INFO:teuthology.orchestra.run.smithi114.stdout: "flags": 0, 2024-04-23T10:41:09.043 INFO:teuthology.orchestra.run.smithi114.stdout: "default_value": "prophetstor", 2024-04-23T10:41:09.043 INFO:teuthology.orchestra.run.smithi114.stdout: "min": "", 2024-04-23T10:41:09.043 INFO:teuthology.orchestra.run.smithi114.stdout: "max": "", 2024-04-23T10:41:09.043 INFO:teuthology.orchestra.run.smithi114.stdout: "enum_allowed": [], 2024-04-23T10:41:09.043 INFO:teuthology.orchestra.run.smithi114.stdout: "desc": "", 2024-04-23T10:41:09.043 INFO:teuthology.orchestra.run.smithi114.stdout: "long_desc": "", 2024-04-23T10:41:09.043 INFO:teuthology.orchestra.run.smithi114.stdout: "tags": [], 2024-04-23T10:41:09.044 INFO:teuthology.orchestra.run.smithi114.stdout: "see_also": [] 2024-04-23T10:41:09.044 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-23T10:41:09.044 INFO:teuthology.orchestra.run.smithi114.stdout: "sleep_interval": { 2024-04-23T10:41:09.044 INFO:teuthology.orchestra.run.smithi114.stdout: "name": "sleep_interval", 2024-04-23T10:41:09.044 INFO:teuthology.orchestra.run.smithi114.stdout: "type": "str", 2024-04-23T10:41:09.044 INFO:teuthology.orchestra.run.smithi114.stdout: "level": "advanced", 2024-04-23T10:41:09.044 INFO:teuthology.orchestra.run.smithi114.stdout: "flags": 0, 2024-04-23T10:41:09.044 INFO:teuthology.orchestra.run.smithi114.stdout: "default_value": "600", 2024-04-23T10:41:09.044 INFO:teuthology.orchestra.run.smithi114.stdout: "min": "", 2024-04-23T10:41:09.045 INFO:teuthology.orchestra.run.smithi114.stdout: "max": "", 2024-04-23T10:41:09.045 INFO:teuthology.orchestra.run.smithi114.stdout: "enum_allowed": [], 2024-04-23T10:41:09.045 INFO:teuthology.orchestra.run.smithi114.stdout: "desc": "", 2024-04-23T10:41:09.045 INFO:teuthology.orchestra.run.smithi114.stdout: "long_desc": "", 2024-04-23T10:41:09.045 INFO:teuthology.orchestra.run.smithi114.stdout: "tags": [], 2024-04-23T10:41:09.045 INFO:teuthology.orchestra.run.smithi114.stdout: "see_also": [] 2024-04-23T10:41:09.045 INFO:teuthology.orchestra.run.smithi114.stdout: } 2024-04-23T10:41:09.045 INFO:teuthology.orchestra.run.smithi114.stdout: } 2024-04-23T10:41:09.045 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-23T10:41:09.046 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-04-23T10:41:09.046 INFO:teuthology.orchestra.run.smithi114.stdout: "name": "influx", 2024-04-23T10:41:09.046 INFO:teuthology.orchestra.run.smithi114.stdout: "can_run": false, 2024-04-23T10:41:09.046 INFO:teuthology.orchestra.run.smithi114.stdout: "error_string": "influxdb python module not found", 2024-04-23T10:41:09.046 INFO:teuthology.orchestra.run.smithi114.stdout: "module_options": { 2024-04-23T10:41:09.046 INFO:teuthology.orchestra.run.smithi114.stdout: "batch_size": { 2024-04-23T10:41:09.046 INFO:teuthology.orchestra.run.smithi114.stdout: "name": "batch_size", 2024-04-23T10:41:09.046 INFO:teuthology.orchestra.run.smithi114.stdout: "type": "int", 2024-04-23T10:41:09.047 INFO:teuthology.orchestra.run.smithi114.stdout: "level": "advanced", 2024-04-23T10:41:09.047 INFO:teuthology.orchestra.run.smithi114.stdout: "flags": 0, 2024-04-23T10:41:09.047 INFO:teuthology.orchestra.run.smithi114.stdout: "default_value": "5000", 2024-04-23T10:41:09.047 INFO:teuthology.orchestra.run.smithi114.stdout: "min": "", 2024-04-23T10:41:09.047 INFO:teuthology.orchestra.run.smithi114.stdout: "max": "", 2024-04-23T10:41:09.047 INFO:teuthology.orchestra.run.smithi114.stdout: "enum_allowed": [], 2024-04-23T10:41:09.047 INFO:teuthology.orchestra.run.smithi114.stdout: "desc": "How big batches of data points should be when sending to InfluxDB.", 2024-04-23T10:41:09.047 INFO:teuthology.orchestra.run.smithi114.stdout: "long_desc": "", 2024-04-23T10:41:09.047 INFO:teuthology.orchestra.run.smithi114.stdout: "tags": [], 2024-04-23T10:41:09.048 INFO:teuthology.orchestra.run.smithi114.stdout: "see_also": [] 2024-04-23T10:41:09.048 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-23T10:41:09.048 INFO:teuthology.orchestra.run.smithi114.stdout: "database": { 2024-04-23T10:41:09.048 INFO:teuthology.orchestra.run.smithi114.stdout: "name": "database", 2024-04-23T10:41:09.048 INFO:teuthology.orchestra.run.smithi114.stdout: "type": "str", 2024-04-23T10:41:09.048 INFO:teuthology.orchestra.run.smithi114.stdout: "level": "advanced", 2024-04-23T10:41:09.048 INFO:teuthology.orchestra.run.smithi114.stdout: "flags": 0, 2024-04-23T10:41:09.048 INFO:teuthology.orchestra.run.smithi114.stdout: "default_value": "ceph", 2024-04-23T10:41:09.049 INFO:teuthology.orchestra.run.smithi114.stdout: "min": "", 2024-04-23T10:41:09.049 INFO:teuthology.orchestra.run.smithi114.stdout: "max": "", 2024-04-23T10:41:09.049 INFO:teuthology.orchestra.run.smithi114.stdout: "enum_allowed": [], 2024-04-23T10:41:09.049 INFO:teuthology.orchestra.run.smithi114.stdout: "desc": "InfluxDB database name. You will need to create this database and grant write privileges to the configured username or the username must have admin privileges to create it.", 2024-04-23T10:41:09.049 INFO:teuthology.orchestra.run.smithi114.stdout: "long_desc": "", 2024-04-23T10:41:09.049 INFO:teuthology.orchestra.run.smithi114.stdout: "tags": [], 2024-04-23T10:41:09.049 INFO:teuthology.orchestra.run.smithi114.stdout: "see_also": [] 2024-04-23T10:41:09.049 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-23T10:41:09.049 INFO:teuthology.orchestra.run.smithi114.stdout: "hostname": { 2024-04-23T10:41:09.050 INFO:teuthology.orchestra.run.smithi114.stdout: "name": "hostname", 2024-04-23T10:41:09.050 INFO:teuthology.orchestra.run.smithi114.stdout: "type": "str", 2024-04-23T10:41:09.050 INFO:teuthology.orchestra.run.smithi114.stdout: "level": "advanced", 2024-04-23T10:41:09.050 INFO:teuthology.orchestra.run.smithi114.stdout: "flags": 0, 2024-04-23T10:41:09.050 INFO:teuthology.orchestra.run.smithi114.stdout: "default_value": "", 2024-04-23T10:41:09.050 INFO:teuthology.orchestra.run.smithi114.stdout: "min": "", 2024-04-23T10:41:09.050 INFO:teuthology.orchestra.run.smithi114.stdout: "max": "", 2024-04-23T10:41:09.050 INFO:teuthology.orchestra.run.smithi114.stdout: "enum_allowed": [], 2024-04-23T10:41:09.050 INFO:teuthology.orchestra.run.smithi114.stdout: "desc": "InfluxDB server hostname", 2024-04-23T10:41:09.051 INFO:teuthology.orchestra.run.smithi114.stdout: "long_desc": "", 2024-04-23T10:41:09.051 INFO:teuthology.orchestra.run.smithi114.stdout: "tags": [], 2024-04-23T10:41:09.051 INFO:teuthology.orchestra.run.smithi114.stdout: "see_also": [] 2024-04-23T10:41:09.051 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-23T10:41:09.051 INFO:teuthology.orchestra.run.smithi114.stdout: "interval": { 2024-04-23T10:41:09.051 INFO:teuthology.orchestra.run.smithi114.stdout: "name": "interval", 2024-04-23T10:41:09.051 INFO:teuthology.orchestra.run.smithi114.stdout: "type": "secs", 2024-04-23T10:41:09.051 INFO:teuthology.orchestra.run.smithi114.stdout: "level": "advanced", 2024-04-23T10:41:09.051 INFO:teuthology.orchestra.run.smithi114.stdout: "flags": 0, 2024-04-23T10:41:09.052 INFO:teuthology.orchestra.run.smithi114.stdout: "default_value": "30", 2024-04-23T10:41:09.052 INFO:teuthology.orchestra.run.smithi114.stdout: "min": "5", 2024-04-23T10:41:09.052 INFO:teuthology.orchestra.run.smithi114.stdout: "max": "", 2024-04-23T10:41:09.052 INFO:teuthology.orchestra.run.smithi114.stdout: "enum_allowed": [], 2024-04-23T10:41:09.052 INFO:teuthology.orchestra.run.smithi114.stdout: "desc": "Time between reports to InfluxDB. Default 30 seconds.", 2024-04-23T10:41:09.052 INFO:teuthology.orchestra.run.smithi114.stdout: "long_desc": "", 2024-04-23T10:41:09.052 INFO:teuthology.orchestra.run.smithi114.stdout: "tags": [], 2024-04-23T10:41:09.052 INFO:teuthology.orchestra.run.smithi114.stdout: "see_also": [] 2024-04-23T10:41:09.052 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-23T10:41:09.053 INFO:teuthology.orchestra.run.smithi114.stdout: "log_level": { 2024-04-23T10:41:09.053 INFO:teuthology.orchestra.run.smithi114.stdout: "name": "log_level", 2024-04-23T10:41:09.053 INFO:teuthology.orchestra.run.smithi114.stdout: "type": "str", 2024-04-23T10:41:09.053 INFO:teuthology.orchestra.run.smithi114.stdout: "level": "advanced", 2024-04-23T10:41:09.053 INFO:teuthology.orchestra.run.smithi114.stdout: "flags": 1, 2024-04-23T10:41:09.053 INFO:teuthology.orchestra.run.smithi114.stdout: "default_value": "", 2024-04-23T10:41:09.053 INFO:teuthology.orchestra.run.smithi114.stdout: "min": "", 2024-04-23T10:41:09.053 INFO:teuthology.orchestra.run.smithi114.stdout: "max": "", 2024-04-23T10:41:09.054 INFO:teuthology.orchestra.run.smithi114.stdout: "enum_allowed": [ 2024-04-23T10:41:09.054 INFO:teuthology.orchestra.run.smithi114.stdout: "", 2024-04-23T10:41:09.054 INFO:teuthology.orchestra.run.smithi114.stdout: "critical", 2024-04-23T10:41:09.054 INFO:teuthology.orchestra.run.smithi114.stdout: "debug", 2024-04-23T10:41:09.054 INFO:teuthology.orchestra.run.smithi114.stdout: "error", 2024-04-23T10:41:09.054 INFO:teuthology.orchestra.run.smithi114.stdout: "info", 2024-04-23T10:41:09.054 INFO:teuthology.orchestra.run.smithi114.stdout: "warning" 2024-04-23T10:41:09.054 INFO:teuthology.orchestra.run.smithi114.stdout: ], 2024-04-23T10:41:09.054 INFO:teuthology.orchestra.run.smithi114.stdout: "desc": "", 2024-04-23T10:41:09.055 INFO:teuthology.orchestra.run.smithi114.stdout: "long_desc": "", 2024-04-23T10:41:09.055 INFO:teuthology.orchestra.run.smithi114.stdout: "tags": [], 2024-04-23T10:41:09.055 INFO:teuthology.orchestra.run.smithi114.stdout: "see_also": [] 2024-04-23T10:41:09.055 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-23T10:41:09.055 INFO:teuthology.orchestra.run.smithi114.stdout: "log_to_cluster": { 2024-04-23T10:41:09.055 INFO:teuthology.orchestra.run.smithi114.stdout: "name": "log_to_cluster", 2024-04-23T10:41:09.055 INFO:teuthology.orchestra.run.smithi114.stdout: "type": "bool", 2024-04-23T10:41:09.055 INFO:teuthology.orchestra.run.smithi114.stdout: "level": "advanced", 2024-04-23T10:41:09.056 INFO:teuthology.orchestra.run.smithi114.stdout: "flags": 1, 2024-04-23T10:41:09.056 INFO:teuthology.orchestra.run.smithi114.stdout: "default_value": "False", 2024-04-23T10:41:09.056 INFO:teuthology.orchestra.run.smithi114.stdout: "min": "", 2024-04-23T10:41:09.056 INFO:teuthology.orchestra.run.smithi114.stdout: "max": "", 2024-04-23T10:41:09.056 INFO:teuthology.orchestra.run.smithi114.stdout: "enum_allowed": [], 2024-04-23T10:41:09.056 INFO:teuthology.orchestra.run.smithi114.stdout: "desc": "", 2024-04-23T10:41:09.056 INFO:teuthology.orchestra.run.smithi114.stdout: "long_desc": "", 2024-04-23T10:41:09.056 INFO:teuthology.orchestra.run.smithi114.stdout: "tags": [], 2024-04-23T10:41:09.056 INFO:teuthology.orchestra.run.smithi114.stdout: "see_also": [] 2024-04-23T10:41:09.057 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-23T10:41:09.057 INFO:teuthology.orchestra.run.smithi114.stdout: "log_to_cluster_level": { 2024-04-23T10:41:09.057 INFO:teuthology.orchestra.run.smithi114.stdout: "name": "log_to_cluster_level", 2024-04-23T10:41:09.057 INFO:teuthology.orchestra.run.smithi114.stdout: "type": "str", 2024-04-23T10:41:09.057 INFO:teuthology.orchestra.run.smithi114.stdout: "level": "advanced", 2024-04-23T10:41:09.057 INFO:teuthology.orchestra.run.smithi114.stdout: "flags": 1, 2024-04-23T10:41:09.057 INFO:teuthology.orchestra.run.smithi114.stdout: "default_value": "info", 2024-04-23T10:41:09.057 INFO:teuthology.orchestra.run.smithi114.stdout: "min": "", 2024-04-23T10:41:09.057 INFO:teuthology.orchestra.run.smithi114.stdout: "max": "", 2024-04-23T10:41:09.058 INFO:teuthology.orchestra.run.smithi114.stdout: "enum_allowed": [ 2024-04-23T10:41:09.058 INFO:teuthology.orchestra.run.smithi114.stdout: "", 2024-04-23T10:41:09.058 INFO:teuthology.orchestra.run.smithi114.stdout: "critical", 2024-04-23T10:41:09.058 INFO:teuthology.orchestra.run.smithi114.stdout: "debug", 2024-04-23T10:41:09.058 INFO:teuthology.orchestra.run.smithi114.stdout: "error", 2024-04-23T10:41:09.058 INFO:teuthology.orchestra.run.smithi114.stdout: "info", 2024-04-23T10:41:09.058 INFO:teuthology.orchestra.run.smithi114.stdout: "warning" 2024-04-23T10:41:09.058 INFO:teuthology.orchestra.run.smithi114.stdout: ], 2024-04-23T10:41:09.058 INFO:teuthology.orchestra.run.smithi114.stdout: "desc": "", 2024-04-23T10:41:09.059 INFO:teuthology.orchestra.run.smithi114.stdout: "long_desc": "", 2024-04-23T10:41:09.059 INFO:teuthology.orchestra.run.smithi114.stdout: "tags": [], 2024-04-23T10:41:09.059 INFO:teuthology.orchestra.run.smithi114.stdout: "see_also": [] 2024-04-23T10:41:09.059 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-23T10:41:09.059 INFO:teuthology.orchestra.run.smithi114.stdout: "log_to_file": { 2024-04-23T10:41:09.059 INFO:teuthology.orchestra.run.smithi114.stdout: "name": "log_to_file", 2024-04-23T10:41:09.059 INFO:teuthology.orchestra.run.smithi114.stdout: "type": "bool", 2024-04-23T10:41:09.059 INFO:teuthology.orchestra.run.smithi114.stdout: "level": "advanced", 2024-04-23T10:41:09.059 INFO:teuthology.orchestra.run.smithi114.stdout: "flags": 1, 2024-04-23T10:41:09.060 INFO:teuthology.orchestra.run.smithi114.stdout: "default_value": "False", 2024-04-23T10:41:09.060 INFO:teuthology.orchestra.run.smithi114.stdout: "min": "", 2024-04-23T10:41:09.060 INFO:teuthology.orchestra.run.smithi114.stdout: "max": "", 2024-04-23T10:41:09.060 INFO:teuthology.orchestra.run.smithi114.stdout: "enum_allowed": [], 2024-04-23T10:41:09.060 INFO:teuthology.orchestra.run.smithi114.stdout: "desc": "", 2024-04-23T10:41:09.060 INFO:teuthology.orchestra.run.smithi114.stdout: "long_desc": "", 2024-04-23T10:41:09.060 INFO:teuthology.orchestra.run.smithi114.stdout: "tags": [], 2024-04-23T10:41:09.060 INFO:teuthology.orchestra.run.smithi114.stdout: "see_also": [] 2024-04-23T10:41:09.061 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-23T10:41:09.061 INFO:teuthology.orchestra.run.smithi114.stdout: "password": { 2024-04-23T10:41:09.061 INFO:teuthology.orchestra.run.smithi114.stdout: "name": "password", 2024-04-23T10:41:09.061 INFO:teuthology.orchestra.run.smithi114.stdout: "type": "str", 2024-04-23T10:41:09.061 INFO:teuthology.orchestra.run.smithi114.stdout: "level": "advanced", 2024-04-23T10:41:09.061 INFO:teuthology.orchestra.run.smithi114.stdout: "flags": 0, 2024-04-23T10:41:09.061 INFO:teuthology.orchestra.run.smithi114.stdout: "default_value": "", 2024-04-23T10:41:09.061 INFO:teuthology.orchestra.run.smithi114.stdout: "min": "", 2024-04-23T10:41:09.062 INFO:teuthology.orchestra.run.smithi114.stdout: "max": "", 2024-04-23T10:41:09.062 INFO:teuthology.orchestra.run.smithi114.stdout: "enum_allowed": [], 2024-04-23T10:41:09.062 INFO:teuthology.orchestra.run.smithi114.stdout: "desc": "password of InfluxDB server user", 2024-04-23T10:41:09.062 INFO:teuthology.orchestra.run.smithi114.stdout: "long_desc": "", 2024-04-23T10:41:09.062 INFO:teuthology.orchestra.run.smithi114.stdout: "tags": [], 2024-04-23T10:41:09.062 INFO:teuthology.orchestra.run.smithi114.stdout: "see_also": [] 2024-04-23T10:41:09.062 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-23T10:41:09.062 INFO:teuthology.orchestra.run.smithi114.stdout: "port": { 2024-04-23T10:41:09.062 INFO:teuthology.orchestra.run.smithi114.stdout: "name": "port", 2024-04-23T10:41:09.063 INFO:teuthology.orchestra.run.smithi114.stdout: "type": "int", 2024-04-23T10:41:09.063 INFO:teuthology.orchestra.run.smithi114.stdout: "level": "advanced", 2024-04-23T10:41:09.063 INFO:teuthology.orchestra.run.smithi114.stdout: "flags": 0, 2024-04-23T10:41:09.063 INFO:teuthology.orchestra.run.smithi114.stdout: "default_value": "8086", 2024-04-23T10:41:09.063 INFO:teuthology.orchestra.run.smithi114.stdout: "min": "", 2024-04-23T10:41:09.063 INFO:teuthology.orchestra.run.smithi114.stdout: "max": "", 2024-04-23T10:41:09.063 INFO:teuthology.orchestra.run.smithi114.stdout: "enum_allowed": [], 2024-04-23T10:41:09.063 INFO:teuthology.orchestra.run.smithi114.stdout: "desc": "InfluxDB server port", 2024-04-23T10:41:09.064 INFO:teuthology.orchestra.run.smithi114.stdout: "long_desc": "", 2024-04-23T10:41:09.064 INFO:teuthology.orchestra.run.smithi114.stdout: "tags": [], 2024-04-23T10:41:09.064 INFO:teuthology.orchestra.run.smithi114.stdout: "see_also": [] 2024-04-23T10:41:09.064 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-23T10:41:09.064 INFO:teuthology.orchestra.run.smithi114.stdout: "ssl": { 2024-04-23T10:41:09.064 INFO:teuthology.orchestra.run.smithi114.stdout: "name": "ssl", 2024-04-23T10:41:09.064 INFO:teuthology.orchestra.run.smithi114.stdout: "type": "str", 2024-04-23T10:41:09.064 INFO:teuthology.orchestra.run.smithi114.stdout: "level": "advanced", 2024-04-23T10:41:09.064 INFO:teuthology.orchestra.run.smithi114.stdout: "flags": 0, 2024-04-23T10:41:09.065 INFO:teuthology.orchestra.run.smithi114.stdout: "default_value": "false", 2024-04-23T10:41:09.065 INFO:teuthology.orchestra.run.smithi114.stdout: "min": "", 2024-04-23T10:41:09.065 INFO:teuthology.orchestra.run.smithi114.stdout: "max": "", 2024-04-23T10:41:09.065 INFO:teuthology.orchestra.run.smithi114.stdout: "enum_allowed": [], 2024-04-23T10:41:09.065 INFO:teuthology.orchestra.run.smithi114.stdout: "desc": "Use https connection for InfluxDB server. Use \"true\" or \"false\".", 2024-04-23T10:41:09.065 INFO:teuthology.orchestra.run.smithi114.stdout: "long_desc": "", 2024-04-23T10:41:09.065 INFO:teuthology.orchestra.run.smithi114.stdout: "tags": [], 2024-04-23T10:41:09.065 INFO:teuthology.orchestra.run.smithi114.stdout: "see_also": [] 2024-04-23T10:41:09.065 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-23T10:41:09.066 INFO:teuthology.orchestra.run.smithi114.stdout: "threads": { 2024-04-23T10:41:09.066 INFO:teuthology.orchestra.run.smithi114.stdout: "name": "threads", 2024-04-23T10:41:09.066 INFO:teuthology.orchestra.run.smithi114.stdout: "type": "int", 2024-04-23T10:41:09.066 INFO:teuthology.orchestra.run.smithi114.stdout: "level": "advanced", 2024-04-23T10:41:09.066 INFO:teuthology.orchestra.run.smithi114.stdout: "flags": 0, 2024-04-23T10:41:09.066 INFO:teuthology.orchestra.run.smithi114.stdout: "default_value": "5", 2024-04-23T10:41:09.066 INFO:teuthology.orchestra.run.smithi114.stdout: "min": "1", 2024-04-23T10:41:09.066 INFO:teuthology.orchestra.run.smithi114.stdout: "max": "32", 2024-04-23T10:41:09.067 INFO:teuthology.orchestra.run.smithi114.stdout: "enum_allowed": [], 2024-04-23T10:41:09.067 INFO:teuthology.orchestra.run.smithi114.stdout: "desc": "How many worker threads should be spawned for sending data to InfluxDB.", 2024-04-23T10:41:09.067 INFO:teuthology.orchestra.run.smithi114.stdout: "long_desc": "", 2024-04-23T10:41:09.067 INFO:teuthology.orchestra.run.smithi114.stdout: "tags": [], 2024-04-23T10:41:09.067 INFO:teuthology.orchestra.run.smithi114.stdout: "see_also": [] 2024-04-23T10:41:09.067 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-23T10:41:09.067 INFO:teuthology.orchestra.run.smithi114.stdout: "username": { 2024-04-23T10:41:09.067 INFO:teuthology.orchestra.run.smithi114.stdout: "name": "username", 2024-04-23T10:41:09.067 INFO:teuthology.orchestra.run.smithi114.stdout: "type": "str", 2024-04-23T10:41:09.068 INFO:teuthology.orchestra.run.smithi114.stdout: "level": "advanced", 2024-04-23T10:41:09.068 INFO:teuthology.orchestra.run.smithi114.stdout: "flags": 0, 2024-04-23T10:41:09.068 INFO:teuthology.orchestra.run.smithi114.stdout: "default_value": "", 2024-04-23T10:41:09.068 INFO:teuthology.orchestra.run.smithi114.stdout: "min": "", 2024-04-23T10:41:09.068 INFO:teuthology.orchestra.run.smithi114.stdout: "max": "", 2024-04-23T10:41:09.068 INFO:teuthology.orchestra.run.smithi114.stdout: "enum_allowed": [], 2024-04-23T10:41:09.068 INFO:teuthology.orchestra.run.smithi114.stdout: "desc": "username of InfluxDB server user", 2024-04-23T10:41:09.068 INFO:teuthology.orchestra.run.smithi114.stdout: "long_desc": "", 2024-04-23T10:41:09.069 INFO:teuthology.orchestra.run.smithi114.stdout: "tags": [], 2024-04-23T10:41:09.069 INFO:teuthology.orchestra.run.smithi114.stdout: "see_also": [] 2024-04-23T10:41:09.069 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-23T10:41:09.069 INFO:teuthology.orchestra.run.smithi114.stdout: "verify_ssl": { 2024-04-23T10:41:09.069 INFO:teuthology.orchestra.run.smithi114.stdout: "name": "verify_ssl", 2024-04-23T10:41:09.069 INFO:teuthology.orchestra.run.smithi114.stdout: "type": "str", 2024-04-23T10:41:09.069 INFO:teuthology.orchestra.run.smithi114.stdout: "level": "advanced", 2024-04-23T10:41:09.069 INFO:teuthology.orchestra.run.smithi114.stdout: "flags": 0, 2024-04-23T10:41:09.069 INFO:teuthology.orchestra.run.smithi114.stdout: "default_value": "true", 2024-04-23T10:41:09.070 INFO:teuthology.orchestra.run.smithi114.stdout: "min": "", 2024-04-23T10:41:09.070 INFO:teuthology.orchestra.run.smithi114.stdout: "max": "", 2024-04-23T10:41:09.070 INFO:teuthology.orchestra.run.smithi114.stdout: "enum_allowed": [], 2024-04-23T10:41:09.070 INFO:teuthology.orchestra.run.smithi114.stdout: "desc": "Verify https cert for InfluxDB server. Use \"true\" or \"false\".", 2024-04-23T10:41:09.070 INFO:teuthology.orchestra.run.smithi114.stdout: "long_desc": "", 2024-04-23T10:41:09.070 INFO:teuthology.orchestra.run.smithi114.stdout: "tags": [], 2024-04-23T10:41:09.070 INFO:teuthology.orchestra.run.smithi114.stdout: "see_also": [] 2024-04-23T10:41:09.070 INFO:teuthology.orchestra.run.smithi114.stdout: } 2024-04-23T10:41:09.070 INFO:teuthology.orchestra.run.smithi114.stdout: } 2024-04-23T10:41:09.071 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-23T10:41:09.071 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-04-23T10:41:09.071 INFO:teuthology.orchestra.run.smithi114.stdout: "name": "insights", 2024-04-23T10:41:09.071 INFO:teuthology.orchestra.run.smithi114.stdout: "can_run": true, 2024-04-23T10:41:09.071 INFO:teuthology.orchestra.run.smithi114.stdout: "error_string": "", 2024-04-23T10:41:09.071 INFO:teuthology.orchestra.run.smithi114.stdout: "module_options": { 2024-04-23T10:41:09.071 INFO:teuthology.orchestra.run.smithi114.stdout: "log_level": { 2024-04-23T10:41:09.071 INFO:teuthology.orchestra.run.smithi114.stdout: "name": "log_level", 2024-04-23T10:41:09.072 INFO:teuthology.orchestra.run.smithi114.stdout: "type": "str", 2024-04-23T10:41:09.072 INFO:teuthology.orchestra.run.smithi114.stdout: "level": "advanced", 2024-04-23T10:41:09.072 INFO:teuthology.orchestra.run.smithi114.stdout: "flags": 1, 2024-04-23T10:41:09.072 INFO:teuthology.orchestra.run.smithi114.stdout: "default_value": "", 2024-04-23T10:41:09.072 INFO:teuthology.orchestra.run.smithi114.stdout: "min": "", 2024-04-23T10:41:09.072 INFO:teuthology.orchestra.run.smithi114.stdout: "max": "", 2024-04-23T10:41:09.072 INFO:teuthology.orchestra.run.smithi114.stdout: "enum_allowed": [ 2024-04-23T10:41:09.072 INFO:teuthology.orchestra.run.smithi114.stdout: "", 2024-04-23T10:41:09.072 INFO:teuthology.orchestra.run.smithi114.stdout: "critical", 2024-04-23T10:41:09.073 INFO:teuthology.orchestra.run.smithi114.stdout: "debug", 2024-04-23T10:41:09.073 INFO:teuthology.orchestra.run.smithi114.stdout: "error", 2024-04-23T10:41:09.073 INFO:teuthology.orchestra.run.smithi114.stdout: "info", 2024-04-23T10:41:09.073 INFO:teuthology.orchestra.run.smithi114.stdout: "warning" 2024-04-23T10:41:09.073 INFO:teuthology.orchestra.run.smithi114.stdout: ], 2024-04-23T10:41:09.073 INFO:teuthology.orchestra.run.smithi114.stdout: "desc": "", 2024-04-23T10:41:09.073 INFO:teuthology.orchestra.run.smithi114.stdout: "long_desc": "", 2024-04-23T10:41:09.073 INFO:teuthology.orchestra.run.smithi114.stdout: "tags": [], 2024-04-23T10:41:09.073 INFO:teuthology.orchestra.run.smithi114.stdout: "see_also": [] 2024-04-23T10:41:09.074 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-23T10:41:09.074 INFO:teuthology.orchestra.run.smithi114.stdout: "log_to_cluster": { 2024-04-23T10:41:09.074 INFO:teuthology.orchestra.run.smithi114.stdout: "name": "log_to_cluster", 2024-04-23T10:41:09.074 INFO:teuthology.orchestra.run.smithi114.stdout: "type": "bool", 2024-04-23T10:41:09.074 INFO:teuthology.orchestra.run.smithi114.stdout: "level": "advanced", 2024-04-23T10:41:09.074 INFO:teuthology.orchestra.run.smithi114.stdout: "flags": 1, 2024-04-23T10:41:09.074 INFO:teuthology.orchestra.run.smithi114.stdout: "default_value": "False", 2024-04-23T10:41:09.074 INFO:teuthology.orchestra.run.smithi114.stdout: "min": "", 2024-04-23T10:41:09.075 INFO:teuthology.orchestra.run.smithi114.stdout: "max": "", 2024-04-23T10:41:09.075 INFO:teuthology.orchestra.run.smithi114.stdout: "enum_allowed": [], 2024-04-23T10:41:09.075 INFO:teuthology.orchestra.run.smithi114.stdout: "desc": "", 2024-04-23T10:41:09.075 INFO:teuthology.orchestra.run.smithi114.stdout: "long_desc": "", 2024-04-23T10:41:09.075 INFO:teuthology.orchestra.run.smithi114.stdout: "tags": [], 2024-04-23T10:41:09.075 INFO:teuthology.orchestra.run.smithi114.stdout: "see_also": [] 2024-04-23T10:41:09.075 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-23T10:41:09.075 INFO:teuthology.orchestra.run.smithi114.stdout: "log_to_cluster_level": { 2024-04-23T10:41:09.076 INFO:teuthology.orchestra.run.smithi114.stdout: "name": "log_to_cluster_level", 2024-04-23T10:41:09.076 INFO:teuthology.orchestra.run.smithi114.stdout: "type": "str", 2024-04-23T10:41:09.076 INFO:teuthology.orchestra.run.smithi114.stdout: "level": "advanced", 2024-04-23T10:41:09.076 INFO:teuthology.orchestra.run.smithi114.stdout: "flags": 1, 2024-04-23T10:41:09.076 INFO:teuthology.orchestra.run.smithi114.stdout: "default_value": "info", 2024-04-23T10:41:09.076 INFO:teuthology.orchestra.run.smithi114.stdout: "min": "", 2024-04-23T10:41:09.076 INFO:teuthology.orchestra.run.smithi114.stdout: "max": "", 2024-04-23T10:41:09.076 INFO:teuthology.orchestra.run.smithi114.stdout: "enum_allowed": [ 2024-04-23T10:41:09.076 INFO:teuthology.orchestra.run.smithi114.stdout: "", 2024-04-23T10:41:09.077 INFO:teuthology.orchestra.run.smithi114.stdout: "critical", 2024-04-23T10:41:09.077 INFO:teuthology.orchestra.run.smithi114.stdout: "debug", 2024-04-23T10:41:09.077 INFO:teuthology.orchestra.run.smithi114.stdout: "error", 2024-04-23T10:41:09.077 INFO:teuthology.orchestra.run.smithi114.stdout: "info", 2024-04-23T10:41:09.077 INFO:teuthology.orchestra.run.smithi114.stdout: "warning" 2024-04-23T10:41:09.077 INFO:teuthology.orchestra.run.smithi114.stdout: ], 2024-04-23T10:41:09.077 INFO:teuthology.orchestra.run.smithi114.stdout: "desc": "", 2024-04-23T10:41:09.077 INFO:teuthology.orchestra.run.smithi114.stdout: "long_desc": "", 2024-04-23T10:41:09.078 INFO:teuthology.orchestra.run.smithi114.stdout: "tags": [], 2024-04-23T10:41:09.078 INFO:teuthology.orchestra.run.smithi114.stdout: "see_also": [] 2024-04-23T10:41:09.078 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-23T10:41:09.078 INFO:teuthology.orchestra.run.smithi114.stdout: "log_to_file": { 2024-04-23T10:41:09.078 INFO:teuthology.orchestra.run.smithi114.stdout: "name": "log_to_file", 2024-04-23T10:41:09.078 INFO:teuthology.orchestra.run.smithi114.stdout: "type": "bool", 2024-04-23T10:41:09.078 INFO:teuthology.orchestra.run.smithi114.stdout: "level": "advanced", 2024-04-23T10:41:09.078 INFO:teuthology.orchestra.run.smithi114.stdout: "flags": 1, 2024-04-23T10:41:09.078 INFO:teuthology.orchestra.run.smithi114.stdout: "default_value": "False", 2024-04-23T10:41:09.079 INFO:teuthology.orchestra.run.smithi114.stdout: "min": "", 2024-04-23T10:41:09.079 INFO:teuthology.orchestra.run.smithi114.stdout: "max": "", 2024-04-23T10:41:09.079 INFO:teuthology.orchestra.run.smithi114.stdout: "enum_allowed": [], 2024-04-23T10:41:09.079 INFO:teuthology.orchestra.run.smithi114.stdout: "desc": "", 2024-04-23T10:41:09.079 INFO:teuthology.orchestra.run.smithi114.stdout: "long_desc": "", 2024-04-23T10:41:09.079 INFO:teuthology.orchestra.run.smithi114.stdout: "tags": [], 2024-04-23T10:41:09.079 INFO:teuthology.orchestra.run.smithi114.stdout: "see_also": [] 2024-04-23T10:41:09.079 INFO:teuthology.orchestra.run.smithi114.stdout: } 2024-04-23T10:41:09.079 INFO:teuthology.orchestra.run.smithi114.stdout: } 2024-04-23T10:41:09.080 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-23T10:41:09.080 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-04-23T10:41:09.080 INFO:teuthology.orchestra.run.smithi114.stdout: "name": "iostat", 2024-04-23T10:41:09.080 INFO:teuthology.orchestra.run.smithi114.stdout: "can_run": true, 2024-04-23T10:41:09.080 INFO:teuthology.orchestra.run.smithi114.stdout: "error_string": "", 2024-04-23T10:41:09.080 INFO:teuthology.orchestra.run.smithi114.stdout: "module_options": { 2024-04-23T10:41:09.080 INFO:teuthology.orchestra.run.smithi114.stdout: "log_level": { 2024-04-23T10:41:09.080 INFO:teuthology.orchestra.run.smithi114.stdout: "name": "log_level", 2024-04-23T10:41:09.081 INFO:teuthology.orchestra.run.smithi114.stdout: "type": "str", 2024-04-23T10:41:09.081 INFO:teuthology.orchestra.run.smithi114.stdout: "level": "advanced", 2024-04-23T10:41:09.081 INFO:teuthology.orchestra.run.smithi114.stdout: "flags": 1, 2024-04-23T10:41:09.081 INFO:teuthology.orchestra.run.smithi114.stdout: "default_value": "", 2024-04-23T10:41:09.081 INFO:teuthology.orchestra.run.smithi114.stdout: "min": "", 2024-04-23T10:41:09.081 INFO:teuthology.orchestra.run.smithi114.stdout: "max": "", 2024-04-23T10:41:09.081 INFO:teuthology.orchestra.run.smithi114.stdout: "enum_allowed": [ 2024-04-23T10:41:09.081 INFO:teuthology.orchestra.run.smithi114.stdout: "", 2024-04-23T10:41:09.081 INFO:teuthology.orchestra.run.smithi114.stdout: "critical", 2024-04-23T10:41:09.082 INFO:teuthology.orchestra.run.smithi114.stdout: "debug", 2024-04-23T10:41:09.082 INFO:teuthology.orchestra.run.smithi114.stdout: "error", 2024-04-23T10:41:09.082 INFO:teuthology.orchestra.run.smithi114.stdout: "info", 2024-04-23T10:41:09.082 INFO:teuthology.orchestra.run.smithi114.stdout: "warning" 2024-04-23T10:41:09.082 INFO:teuthology.orchestra.run.smithi114.stdout: ], 2024-04-23T10:41:09.082 INFO:teuthology.orchestra.run.smithi114.stdout: "desc": "", 2024-04-23T10:41:09.082 INFO:teuthology.orchestra.run.smithi114.stdout: "long_desc": "", 2024-04-23T10:41:09.082 INFO:teuthology.orchestra.run.smithi114.stdout: "tags": [], 2024-04-23T10:41:09.083 INFO:teuthology.orchestra.run.smithi114.stdout: "see_also": [] 2024-04-23T10:41:09.083 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-23T10:41:09.083 INFO:teuthology.orchestra.run.smithi114.stdout: "log_to_cluster": { 2024-04-23T10:41:09.083 INFO:teuthology.orchestra.run.smithi114.stdout: "name": "log_to_cluster", 2024-04-23T10:41:09.083 INFO:teuthology.orchestra.run.smithi114.stdout: "type": "bool", 2024-04-23T10:41:09.083 INFO:teuthology.orchestra.run.smithi114.stdout: "level": "advanced", 2024-04-23T10:41:09.083 INFO:teuthology.orchestra.run.smithi114.stdout: "flags": 1, 2024-04-23T10:41:09.083 INFO:teuthology.orchestra.run.smithi114.stdout: "default_value": "False", 2024-04-23T10:41:09.083 INFO:teuthology.orchestra.run.smithi114.stdout: "min": "", 2024-04-23T10:41:09.084 INFO:teuthology.orchestra.run.smithi114.stdout: "max": "", 2024-04-23T10:41:09.084 INFO:teuthology.orchestra.run.smithi114.stdout: "enum_allowed": [], 2024-04-23T10:41:09.084 INFO:teuthology.orchestra.run.smithi114.stdout: "desc": "", 2024-04-23T10:41:09.084 INFO:teuthology.orchestra.run.smithi114.stdout: "long_desc": "", 2024-04-23T10:41:09.084 INFO:teuthology.orchestra.run.smithi114.stdout: "tags": [], 2024-04-23T10:41:09.084 INFO:teuthology.orchestra.run.smithi114.stdout: "see_also": [] 2024-04-23T10:41:09.084 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-23T10:41:09.084 INFO:teuthology.orchestra.run.smithi114.stdout: "log_to_cluster_level": { 2024-04-23T10:41:09.084 INFO:teuthology.orchestra.run.smithi114.stdout: "name": "log_to_cluster_level", 2024-04-23T10:41:09.085 INFO:teuthology.orchestra.run.smithi114.stdout: "type": "str", 2024-04-23T10:41:09.085 INFO:teuthology.orchestra.run.smithi114.stdout: "level": "advanced", 2024-04-23T10:41:09.085 INFO:teuthology.orchestra.run.smithi114.stdout: "flags": 1, 2024-04-23T10:41:09.085 INFO:teuthology.orchestra.run.smithi114.stdout: "default_value": "info", 2024-04-23T10:41:09.085 INFO:teuthology.orchestra.run.smithi114.stdout: "min": "", 2024-04-23T10:41:09.085 INFO:teuthology.orchestra.run.smithi114.stdout: "max": "", 2024-04-23T10:41:09.085 INFO:teuthology.orchestra.run.smithi114.stdout: "enum_allowed": [ 2024-04-23T10:41:09.085 INFO:teuthology.orchestra.run.smithi114.stdout: "", 2024-04-23T10:41:09.086 INFO:teuthology.orchestra.run.smithi114.stdout: "critical", 2024-04-23T10:41:09.086 INFO:teuthology.orchestra.run.smithi114.stdout: "debug", 2024-04-23T10:41:09.086 INFO:teuthology.orchestra.run.smithi114.stdout: "error", 2024-04-23T10:41:09.086 INFO:teuthology.orchestra.run.smithi114.stdout: "info", 2024-04-23T10:41:09.086 INFO:teuthology.orchestra.run.smithi114.stdout: "warning" 2024-04-23T10:41:09.086 INFO:teuthology.orchestra.run.smithi114.stdout: ], 2024-04-23T10:41:09.086 INFO:teuthology.orchestra.run.smithi114.stdout: "desc": "", 2024-04-23T10:41:09.086 INFO:teuthology.orchestra.run.smithi114.stdout: "long_desc": "", 2024-04-23T10:41:09.086 INFO:teuthology.orchestra.run.smithi114.stdout: "tags": [], 2024-04-23T10:41:09.087 INFO:teuthology.orchestra.run.smithi114.stdout: "see_also": [] 2024-04-23T10:41:09.087 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-23T10:41:09.087 INFO:teuthology.orchestra.run.smithi114.stdout: "log_to_file": { 2024-04-23T10:41:09.087 INFO:teuthology.orchestra.run.smithi114.stdout: "name": "log_to_file", 2024-04-23T10:41:09.087 INFO:teuthology.orchestra.run.smithi114.stdout: "type": "bool", 2024-04-23T10:41:09.087 INFO:teuthology.orchestra.run.smithi114.stdout: "level": "advanced", 2024-04-23T10:41:09.087 INFO:teuthology.orchestra.run.smithi114.stdout: "flags": 1, 2024-04-23T10:41:09.087 INFO:teuthology.orchestra.run.smithi114.stdout: "default_value": "False", 2024-04-23T10:41:09.087 INFO:teuthology.orchestra.run.smithi114.stdout: "min": "", 2024-04-23T10:41:09.088 INFO:teuthology.orchestra.run.smithi114.stdout: "max": "", 2024-04-23T10:41:09.088 INFO:teuthology.orchestra.run.smithi114.stdout: "enum_allowed": [], 2024-04-23T10:41:09.088 INFO:teuthology.orchestra.run.smithi114.stdout: "desc": "", 2024-04-23T10:41:09.088 INFO:teuthology.orchestra.run.smithi114.stdout: "long_desc": "", 2024-04-23T10:41:09.088 INFO:teuthology.orchestra.run.smithi114.stdout: "tags": [], 2024-04-23T10:41:09.088 INFO:teuthology.orchestra.run.smithi114.stdout: "see_also": [] 2024-04-23T10:41:09.088 INFO:teuthology.orchestra.run.smithi114.stdout: } 2024-04-23T10:41:09.088 INFO:teuthology.orchestra.run.smithi114.stdout: } 2024-04-23T10:41:09.089 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-23T10:41:09.089 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-04-23T10:41:09.089 INFO:teuthology.orchestra.run.smithi114.stdout: "name": "k8sevents", 2024-04-23T10:41:09.089 INFO:teuthology.orchestra.run.smithi114.stdout: "can_run": true, 2024-04-23T10:41:09.089 INFO:teuthology.orchestra.run.smithi114.stdout: "error_string": "", 2024-04-23T10:41:09.089 INFO:teuthology.orchestra.run.smithi114.stdout: "module_options": { 2024-04-23T10:41:09.089 INFO:teuthology.orchestra.run.smithi114.stdout: "ceph_event_retention_days": { 2024-04-23T10:41:09.089 INFO:teuthology.orchestra.run.smithi114.stdout: "name": "ceph_event_retention_days", 2024-04-23T10:41:09.089 INFO:teuthology.orchestra.run.smithi114.stdout: "type": "int", 2024-04-23T10:41:09.090 INFO:teuthology.orchestra.run.smithi114.stdout: "level": "advanced", 2024-04-23T10:41:09.090 INFO:teuthology.orchestra.run.smithi114.stdout: "flags": 0, 2024-04-23T10:41:09.090 INFO:teuthology.orchestra.run.smithi114.stdout: "default_value": "7", 2024-04-23T10:41:09.090 INFO:teuthology.orchestra.run.smithi114.stdout: "min": "", 2024-04-23T10:41:09.090 INFO:teuthology.orchestra.run.smithi114.stdout: "max": "", 2024-04-23T10:41:09.090 INFO:teuthology.orchestra.run.smithi114.stdout: "enum_allowed": [], 2024-04-23T10:41:09.090 INFO:teuthology.orchestra.run.smithi114.stdout: "desc": "Days to hold ceph event information within local cache", 2024-04-23T10:41:09.090 INFO:teuthology.orchestra.run.smithi114.stdout: "long_desc": "", 2024-04-23T10:41:09.091 INFO:teuthology.orchestra.run.smithi114.stdout: "tags": [], 2024-04-23T10:41:09.091 INFO:teuthology.orchestra.run.smithi114.stdout: "see_also": [] 2024-04-23T10:41:09.091 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-23T10:41:09.091 INFO:teuthology.orchestra.run.smithi114.stdout: "config_check_secs": { 2024-04-23T10:41:09.091 INFO:teuthology.orchestra.run.smithi114.stdout: "name": "config_check_secs", 2024-04-23T10:41:09.091 INFO:teuthology.orchestra.run.smithi114.stdout: "type": "int", 2024-04-23T10:41:09.091 INFO:teuthology.orchestra.run.smithi114.stdout: "level": "advanced", 2024-04-23T10:41:09.091 INFO:teuthology.orchestra.run.smithi114.stdout: "flags": 0, 2024-04-23T10:41:09.091 INFO:teuthology.orchestra.run.smithi114.stdout: "default_value": "10", 2024-04-23T10:41:09.092 INFO:teuthology.orchestra.run.smithi114.stdout: "min": "10", 2024-04-23T10:41:09.092 INFO:teuthology.orchestra.run.smithi114.stdout: "max": "", 2024-04-23T10:41:09.092 INFO:teuthology.orchestra.run.smithi114.stdout: "enum_allowed": [], 2024-04-23T10:41:09.092 INFO:teuthology.orchestra.run.smithi114.stdout: "desc": "interval (secs) to check for cluster configuration changes", 2024-04-23T10:41:09.092 INFO:teuthology.orchestra.run.smithi114.stdout: "long_desc": "", 2024-04-23T10:41:09.092 INFO:teuthology.orchestra.run.smithi114.stdout: "tags": [], 2024-04-23T10:41:09.092 INFO:teuthology.orchestra.run.smithi114.stdout: "see_also": [] 2024-04-23T10:41:09.092 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-23T10:41:09.092 INFO:teuthology.orchestra.run.smithi114.stdout: "log_level": { 2024-04-23T10:41:09.093 INFO:teuthology.orchestra.run.smithi114.stdout: "name": "log_level", 2024-04-23T10:41:09.093 INFO:teuthology.orchestra.run.smithi114.stdout: "type": "str", 2024-04-23T10:41:09.093 INFO:teuthology.orchestra.run.smithi114.stdout: "level": "advanced", 2024-04-23T10:41:09.093 INFO:teuthology.orchestra.run.smithi114.stdout: "flags": 1, 2024-04-23T10:41:09.093 INFO:teuthology.orchestra.run.smithi114.stdout: "default_value": "", 2024-04-23T10:41:09.093 INFO:teuthology.orchestra.run.smithi114.stdout: "min": "", 2024-04-23T10:41:09.093 INFO:teuthology.orchestra.run.smithi114.stdout: "max": "", 2024-04-23T10:41:09.093 INFO:teuthology.orchestra.run.smithi114.stdout: "enum_allowed": [ 2024-04-23T10:41:09.093 INFO:teuthology.orchestra.run.smithi114.stdout: "", 2024-04-23T10:41:09.094 INFO:teuthology.orchestra.run.smithi114.stdout: "critical", 2024-04-23T10:41:09.094 INFO:teuthology.orchestra.run.smithi114.stdout: "debug", 2024-04-23T10:41:09.094 INFO:teuthology.orchestra.run.smithi114.stdout: "error", 2024-04-23T10:41:09.094 INFO:teuthology.orchestra.run.smithi114.stdout: "info", 2024-04-23T10:41:09.094 INFO:teuthology.orchestra.run.smithi114.stdout: "warning" 2024-04-23T10:41:09.094 INFO:teuthology.orchestra.run.smithi114.stdout: ], 2024-04-23T10:41:09.094 INFO:teuthology.orchestra.run.smithi114.stdout: "desc": "", 2024-04-23T10:41:09.094 INFO:teuthology.orchestra.run.smithi114.stdout: "long_desc": "", 2024-04-23T10:41:09.095 INFO:teuthology.orchestra.run.smithi114.stdout: "tags": [], 2024-04-23T10:41:09.095 INFO:teuthology.orchestra.run.smithi114.stdout: "see_also": [] 2024-04-23T10:41:09.095 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-23T10:41:09.095 INFO:teuthology.orchestra.run.smithi114.stdout: "log_to_cluster": { 2024-04-23T10:41:09.095 INFO:teuthology.orchestra.run.smithi114.stdout: "name": "log_to_cluster", 2024-04-23T10:41:09.095 INFO:teuthology.orchestra.run.smithi114.stdout: "type": "bool", 2024-04-23T10:41:09.095 INFO:teuthology.orchestra.run.smithi114.stdout: "level": "advanced", 2024-04-23T10:41:09.095 INFO:teuthology.orchestra.run.smithi114.stdout: "flags": 1, 2024-04-23T10:41:09.095 INFO:teuthology.orchestra.run.smithi114.stdout: "default_value": "False", 2024-04-23T10:41:09.096 INFO:teuthology.orchestra.run.smithi114.stdout: "min": "", 2024-04-23T10:41:09.096 INFO:teuthology.orchestra.run.smithi114.stdout: "max": "", 2024-04-23T10:41:09.096 INFO:teuthology.orchestra.run.smithi114.stdout: "enum_allowed": [], 2024-04-23T10:41:09.096 INFO:teuthology.orchestra.run.smithi114.stdout: "desc": "", 2024-04-23T10:41:09.096 INFO:teuthology.orchestra.run.smithi114.stdout: "long_desc": "", 2024-04-23T10:41:09.096 INFO:teuthology.orchestra.run.smithi114.stdout: "tags": [], 2024-04-23T10:41:09.096 INFO:teuthology.orchestra.run.smithi114.stdout: "see_also": [] 2024-04-23T10:41:09.096 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-23T10:41:09.096 INFO:teuthology.orchestra.run.smithi114.stdout: "log_to_cluster_level": { 2024-04-23T10:41:09.097 INFO:teuthology.orchestra.run.smithi114.stdout: "name": "log_to_cluster_level", 2024-04-23T10:41:09.097 INFO:teuthology.orchestra.run.smithi114.stdout: "type": "str", 2024-04-23T10:41:09.097 INFO:teuthology.orchestra.run.smithi114.stdout: "level": "advanced", 2024-04-23T10:41:09.097 INFO:teuthology.orchestra.run.smithi114.stdout: "flags": 1, 2024-04-23T10:41:09.097 INFO:teuthology.orchestra.run.smithi114.stdout: "default_value": "info", 2024-04-23T10:41:09.097 INFO:teuthology.orchestra.run.smithi114.stdout: "min": "", 2024-04-23T10:41:09.097 INFO:teuthology.orchestra.run.smithi114.stdout: "max": "", 2024-04-23T10:41:09.097 INFO:teuthology.orchestra.run.smithi114.stdout: "enum_allowed": [ 2024-04-23T10:41:09.097 INFO:teuthology.orchestra.run.smithi114.stdout: "", 2024-04-23T10:41:09.098 INFO:teuthology.orchestra.run.smithi114.stdout: "critical", 2024-04-23T10:41:09.098 INFO:teuthology.orchestra.run.smithi114.stdout: "debug", 2024-04-23T10:41:09.098 INFO:teuthology.orchestra.run.smithi114.stdout: "error", 2024-04-23T10:41:09.098 INFO:teuthology.orchestra.run.smithi114.stdout: "info", 2024-04-23T10:41:09.098 INFO:teuthology.orchestra.run.smithi114.stdout: "warning" 2024-04-23T10:41:09.098 INFO:teuthology.orchestra.run.smithi114.stdout: ], 2024-04-23T10:41:09.098 INFO:teuthology.orchestra.run.smithi114.stdout: "desc": "", 2024-04-23T10:41:09.098 INFO:teuthology.orchestra.run.smithi114.stdout: "long_desc": "", 2024-04-23T10:41:09.098 INFO:teuthology.orchestra.run.smithi114.stdout: "tags": [], 2024-04-23T10:41:09.099 INFO:teuthology.orchestra.run.smithi114.stdout: "see_also": [] 2024-04-23T10:41:09.099 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-23T10:41:09.099 INFO:teuthology.orchestra.run.smithi114.stdout: "log_to_file": { 2024-04-23T10:41:09.099 INFO:teuthology.orchestra.run.smithi114.stdout: "name": "log_to_file", 2024-04-23T10:41:09.099 INFO:teuthology.orchestra.run.smithi114.stdout: "type": "bool", 2024-04-23T10:41:09.099 INFO:teuthology.orchestra.run.smithi114.stdout: "level": "advanced", 2024-04-23T10:41:09.099 INFO:teuthology.orchestra.run.smithi114.stdout: "flags": 1, 2024-04-23T10:41:09.099 INFO:teuthology.orchestra.run.smithi114.stdout: "default_value": "False", 2024-04-23T10:41:09.100 INFO:teuthology.orchestra.run.smithi114.stdout: "min": "", 2024-04-23T10:41:09.100 INFO:teuthology.orchestra.run.smithi114.stdout: "max": "", 2024-04-23T10:41:09.100 INFO:teuthology.orchestra.run.smithi114.stdout: "enum_allowed": [], 2024-04-23T10:41:09.100 INFO:teuthology.orchestra.run.smithi114.stdout: "desc": "", 2024-04-23T10:41:09.100 INFO:teuthology.orchestra.run.smithi114.stdout: "long_desc": "", 2024-04-23T10:41:09.100 INFO:teuthology.orchestra.run.smithi114.stdout: "tags": [], 2024-04-23T10:41:09.100 INFO:teuthology.orchestra.run.smithi114.stdout: "see_also": [] 2024-04-23T10:41:09.100 INFO:teuthology.orchestra.run.smithi114.stdout: } 2024-04-23T10:41:09.100 INFO:teuthology.orchestra.run.smithi114.stdout: } 2024-04-23T10:41:09.101 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-23T10:41:09.101 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-04-23T10:41:09.101 INFO:teuthology.orchestra.run.smithi114.stdout: "name": "localpool", 2024-04-23T10:41:09.101 INFO:teuthology.orchestra.run.smithi114.stdout: "can_run": true, 2024-04-23T10:41:09.101 INFO:teuthology.orchestra.run.smithi114.stdout: "error_string": "", 2024-04-23T10:41:09.101 INFO:teuthology.orchestra.run.smithi114.stdout: "module_options": { 2024-04-23T10:41:09.101 INFO:teuthology.orchestra.run.smithi114.stdout: "failure_domain": { 2024-04-23T10:41:09.101 INFO:teuthology.orchestra.run.smithi114.stdout: "name": "failure_domain", 2024-04-23T10:41:09.101 INFO:teuthology.orchestra.run.smithi114.stdout: "type": "str", 2024-04-23T10:41:09.102 INFO:teuthology.orchestra.run.smithi114.stdout: "level": "advanced", 2024-04-23T10:41:09.102 INFO:teuthology.orchestra.run.smithi114.stdout: "flags": 1, 2024-04-23T10:41:09.102 INFO:teuthology.orchestra.run.smithi114.stdout: "default_value": "host", 2024-04-23T10:41:09.102 INFO:teuthology.orchestra.run.smithi114.stdout: "min": "", 2024-04-23T10:41:09.102 INFO:teuthology.orchestra.run.smithi114.stdout: "max": "", 2024-04-23T10:41:09.102 INFO:teuthology.orchestra.run.smithi114.stdout: "enum_allowed": [], 2024-04-23T10:41:09.102 INFO:teuthology.orchestra.run.smithi114.stdout: "desc": "failure domain for any created local pool", 2024-04-23T10:41:09.102 INFO:teuthology.orchestra.run.smithi114.stdout: "long_desc": "what failure domain we should separate data replicas across.", 2024-04-23T10:41:09.103 INFO:teuthology.orchestra.run.smithi114.stdout: "tags": [], 2024-04-23T10:41:09.103 INFO:teuthology.orchestra.run.smithi114.stdout: "see_also": [] 2024-04-23T10:41:09.103 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-23T10:41:09.103 INFO:teuthology.orchestra.run.smithi114.stdout: "log_level": { 2024-04-23T10:41:09.103 INFO:teuthology.orchestra.run.smithi114.stdout: "name": "log_level", 2024-04-23T10:41:09.103 INFO:teuthology.orchestra.run.smithi114.stdout: "type": "str", 2024-04-23T10:41:09.103 INFO:teuthology.orchestra.run.smithi114.stdout: "level": "advanced", 2024-04-23T10:41:09.103 INFO:teuthology.orchestra.run.smithi114.stdout: "flags": 1, 2024-04-23T10:41:09.103 INFO:teuthology.orchestra.run.smithi114.stdout: "default_value": "", 2024-04-23T10:41:09.104 INFO:teuthology.orchestra.run.smithi114.stdout: "min": "", 2024-04-23T10:41:09.104 INFO:teuthology.orchestra.run.smithi114.stdout: "max": "", 2024-04-23T10:41:09.104 INFO:teuthology.orchestra.run.smithi114.stdout: "enum_allowed": [ 2024-04-23T10:41:09.104 INFO:teuthology.orchestra.run.smithi114.stdout: "", 2024-04-23T10:41:09.104 INFO:teuthology.orchestra.run.smithi114.stdout: "critical", 2024-04-23T10:41:09.104 INFO:teuthology.orchestra.run.smithi114.stdout: "debug", 2024-04-23T10:41:09.104 INFO:teuthology.orchestra.run.smithi114.stdout: "error", 2024-04-23T10:41:09.104 INFO:teuthology.orchestra.run.smithi114.stdout: "info", 2024-04-23T10:41:09.105 INFO:teuthology.orchestra.run.smithi114.stdout: "warning" 2024-04-23T10:41:09.105 INFO:teuthology.orchestra.run.smithi114.stdout: ], 2024-04-23T10:41:09.105 INFO:teuthology.orchestra.run.smithi114.stdout: "desc": "", 2024-04-23T10:41:09.105 INFO:teuthology.orchestra.run.smithi114.stdout: "long_desc": "", 2024-04-23T10:41:09.105 INFO:teuthology.orchestra.run.smithi114.stdout: "tags": [], 2024-04-23T10:41:09.105 INFO:teuthology.orchestra.run.smithi114.stdout: "see_also": [] 2024-04-23T10:41:09.105 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-23T10:41:09.106 INFO:teuthology.orchestra.run.smithi114.stdout: "log_to_cluster": { 2024-04-23T10:41:09.106 INFO:teuthology.orchestra.run.smithi114.stdout: "name": "log_to_cluster", 2024-04-23T10:41:09.106 INFO:teuthology.orchestra.run.smithi114.stdout: "type": "bool", 2024-04-23T10:41:09.106 INFO:teuthology.orchestra.run.smithi114.stdout: "level": "advanced", 2024-04-23T10:41:09.106 INFO:teuthology.orchestra.run.smithi114.stdout: "flags": 1, 2024-04-23T10:41:09.106 INFO:teuthology.orchestra.run.smithi114.stdout: "default_value": "False", 2024-04-23T10:41:09.106 INFO:teuthology.orchestra.run.smithi114.stdout: "min": "", 2024-04-23T10:41:09.106 INFO:teuthology.orchestra.run.smithi114.stdout: "max": "", 2024-04-23T10:41:09.107 INFO:teuthology.orchestra.run.smithi114.stdout: "enum_allowed": [], 2024-04-23T10:41:09.107 INFO:teuthology.orchestra.run.smithi114.stdout: "desc": "", 2024-04-23T10:41:09.107 INFO:teuthology.orchestra.run.smithi114.stdout: "long_desc": "", 2024-04-23T10:41:09.107 INFO:teuthology.orchestra.run.smithi114.stdout: "tags": [], 2024-04-23T10:41:09.107 INFO:teuthology.orchestra.run.smithi114.stdout: "see_also": [] 2024-04-23T10:41:09.107 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-23T10:41:09.107 INFO:teuthology.orchestra.run.smithi114.stdout: "log_to_cluster_level": { 2024-04-23T10:41:09.107 INFO:teuthology.orchestra.run.smithi114.stdout: "name": "log_to_cluster_level", 2024-04-23T10:41:09.107 INFO:teuthology.orchestra.run.smithi114.stdout: "type": "str", 2024-04-23T10:41:09.108 INFO:teuthology.orchestra.run.smithi114.stdout: "level": "advanced", 2024-04-23T10:41:09.108 INFO:teuthology.orchestra.run.smithi114.stdout: "flags": 1, 2024-04-23T10:41:09.108 INFO:teuthology.orchestra.run.smithi114.stdout: "default_value": "info", 2024-04-23T10:41:09.108 INFO:teuthology.orchestra.run.smithi114.stdout: "min": "", 2024-04-23T10:41:09.108 INFO:teuthology.orchestra.run.smithi114.stdout: "max": "", 2024-04-23T10:41:09.108 INFO:teuthology.orchestra.run.smithi114.stdout: "enum_allowed": [ 2024-04-23T10:41:09.108 INFO:teuthology.orchestra.run.smithi114.stdout: "", 2024-04-23T10:41:09.108 INFO:teuthology.orchestra.run.smithi114.stdout: "critical", 2024-04-23T10:41:09.108 INFO:teuthology.orchestra.run.smithi114.stdout: "debug", 2024-04-23T10:41:09.109 INFO:teuthology.orchestra.run.smithi114.stdout: "error", 2024-04-23T10:41:09.109 INFO:teuthology.orchestra.run.smithi114.stdout: "info", 2024-04-23T10:41:09.109 INFO:teuthology.orchestra.run.smithi114.stdout: "warning" 2024-04-23T10:41:09.109 INFO:teuthology.orchestra.run.smithi114.stdout: ], 2024-04-23T10:41:09.109 INFO:teuthology.orchestra.run.smithi114.stdout: "desc": "", 2024-04-23T10:41:09.109 INFO:teuthology.orchestra.run.smithi114.stdout: "long_desc": "", 2024-04-23T10:41:09.109 INFO:teuthology.orchestra.run.smithi114.stdout: "tags": [], 2024-04-23T10:41:09.109 INFO:teuthology.orchestra.run.smithi114.stdout: "see_also": [] 2024-04-23T10:41:09.110 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-23T10:41:09.110 INFO:teuthology.orchestra.run.smithi114.stdout: "log_to_file": { 2024-04-23T10:41:09.110 INFO:teuthology.orchestra.run.smithi114.stdout: "name": "log_to_file", 2024-04-23T10:41:09.110 INFO:teuthology.orchestra.run.smithi114.stdout: "type": "bool", 2024-04-23T10:41:09.110 INFO:teuthology.orchestra.run.smithi114.stdout: "level": "advanced", 2024-04-23T10:41:09.110 INFO:teuthology.orchestra.run.smithi114.stdout: "flags": 1, 2024-04-23T10:41:09.110 INFO:teuthology.orchestra.run.smithi114.stdout: "default_value": "False", 2024-04-23T10:41:09.110 INFO:teuthology.orchestra.run.smithi114.stdout: "min": "", 2024-04-23T10:41:09.110 INFO:teuthology.orchestra.run.smithi114.stdout: "max": "", 2024-04-23T10:41:09.111 INFO:teuthology.orchestra.run.smithi114.stdout: "enum_allowed": [], 2024-04-23T10:41:09.111 INFO:teuthology.orchestra.run.smithi114.stdout: "desc": "", 2024-04-23T10:41:09.111 INFO:teuthology.orchestra.run.smithi114.stdout: "long_desc": "", 2024-04-23T10:41:09.111 INFO:teuthology.orchestra.run.smithi114.stdout: "tags": [], 2024-04-23T10:41:09.111 INFO:teuthology.orchestra.run.smithi114.stdout: "see_also": [] 2024-04-23T10:41:09.111 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-23T10:41:09.111 INFO:teuthology.orchestra.run.smithi114.stdout: "min_size": { 2024-04-23T10:41:09.111 INFO:teuthology.orchestra.run.smithi114.stdout: "name": "min_size", 2024-04-23T10:41:09.112 INFO:teuthology.orchestra.run.smithi114.stdout: "type": "int", 2024-04-23T10:41:09.112 INFO:teuthology.orchestra.run.smithi114.stdout: "level": "advanced", 2024-04-23T10:41:09.112 INFO:teuthology.orchestra.run.smithi114.stdout: "flags": 1, 2024-04-23T10:41:09.112 INFO:teuthology.orchestra.run.smithi114.stdout: "default_value": "", 2024-04-23T10:41:09.112 INFO:teuthology.orchestra.run.smithi114.stdout: "min": "", 2024-04-23T10:41:09.112 INFO:teuthology.orchestra.run.smithi114.stdout: "max": "", 2024-04-23T10:41:09.112 INFO:teuthology.orchestra.run.smithi114.stdout: "enum_allowed": [], 2024-04-23T10:41:09.112 INFO:teuthology.orchestra.run.smithi114.stdout: "desc": "default min_size for any created local pool", 2024-04-23T10:41:09.112 INFO:teuthology.orchestra.run.smithi114.stdout: "long_desc": "value to set min_size to (unchanged from Ceph's default if this option is not set)", 2024-04-23T10:41:09.113 INFO:teuthology.orchestra.run.smithi114.stdout: "tags": [], 2024-04-23T10:41:09.113 INFO:teuthology.orchestra.run.smithi114.stdout: "see_also": [] 2024-04-23T10:41:09.113 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-23T10:41:09.113 INFO:teuthology.orchestra.run.smithi114.stdout: "num_rep": { 2024-04-23T10:41:09.113 INFO:teuthology.orchestra.run.smithi114.stdout: "name": "num_rep", 2024-04-23T10:41:09.113 INFO:teuthology.orchestra.run.smithi114.stdout: "type": "int", 2024-04-23T10:41:09.113 INFO:teuthology.orchestra.run.smithi114.stdout: "level": "advanced", 2024-04-23T10:41:09.113 INFO:teuthology.orchestra.run.smithi114.stdout: "flags": 1, 2024-04-23T10:41:09.113 INFO:teuthology.orchestra.run.smithi114.stdout: "default_value": "3", 2024-04-23T10:41:09.114 INFO:teuthology.orchestra.run.smithi114.stdout: "min": "", 2024-04-23T10:41:09.114 INFO:teuthology.orchestra.run.smithi114.stdout: "max": "", 2024-04-23T10:41:09.114 INFO:teuthology.orchestra.run.smithi114.stdout: "enum_allowed": [], 2024-04-23T10:41:09.114 INFO:teuthology.orchestra.run.smithi114.stdout: "desc": "default replica count for any created local pool", 2024-04-23T10:41:09.114 INFO:teuthology.orchestra.run.smithi114.stdout: "long_desc": "", 2024-04-23T10:41:09.114 INFO:teuthology.orchestra.run.smithi114.stdout: "tags": [], 2024-04-23T10:41:09.114 INFO:teuthology.orchestra.run.smithi114.stdout: "see_also": [] 2024-04-23T10:41:09.114 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-23T10:41:09.115 INFO:teuthology.orchestra.run.smithi114.stdout: "pg_num": { 2024-04-23T10:41:09.115 INFO:teuthology.orchestra.run.smithi114.stdout: "name": "pg_num", 2024-04-23T10:41:09.115 INFO:teuthology.orchestra.run.smithi114.stdout: "type": "int", 2024-04-23T10:41:09.115 INFO:teuthology.orchestra.run.smithi114.stdout: "level": "advanced", 2024-04-23T10:41:09.115 INFO:teuthology.orchestra.run.smithi114.stdout: "flags": 1, 2024-04-23T10:41:09.115 INFO:teuthology.orchestra.run.smithi114.stdout: "default_value": "128", 2024-04-23T10:41:09.115 INFO:teuthology.orchestra.run.smithi114.stdout: "min": "", 2024-04-23T10:41:09.115 INFO:teuthology.orchestra.run.smithi114.stdout: "max": "", 2024-04-23T10:41:09.115 INFO:teuthology.orchestra.run.smithi114.stdout: "enum_allowed": [], 2024-04-23T10:41:09.116 INFO:teuthology.orchestra.run.smithi114.stdout: "desc": "default pg_num for any created local pool", 2024-04-23T10:41:09.116 INFO:teuthology.orchestra.run.smithi114.stdout: "long_desc": "", 2024-04-23T10:41:09.116 INFO:teuthology.orchestra.run.smithi114.stdout: "tags": [], 2024-04-23T10:41:09.116 INFO:teuthology.orchestra.run.smithi114.stdout: "see_also": [] 2024-04-23T10:41:09.116 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-23T10:41:09.116 INFO:teuthology.orchestra.run.smithi114.stdout: "prefix": { 2024-04-23T10:41:09.116 INFO:teuthology.orchestra.run.smithi114.stdout: "name": "prefix", 2024-04-23T10:41:09.116 INFO:teuthology.orchestra.run.smithi114.stdout: "type": "str", 2024-04-23T10:41:09.116 INFO:teuthology.orchestra.run.smithi114.stdout: "level": "advanced", 2024-04-23T10:41:09.117 INFO:teuthology.orchestra.run.smithi114.stdout: "flags": 1, 2024-04-23T10:41:09.117 INFO:teuthology.orchestra.run.smithi114.stdout: "default_value": "", 2024-04-23T10:41:09.117 INFO:teuthology.orchestra.run.smithi114.stdout: "min": "", 2024-04-23T10:41:09.117 INFO:teuthology.orchestra.run.smithi114.stdout: "max": "", 2024-04-23T10:41:09.117 INFO:teuthology.orchestra.run.smithi114.stdout: "enum_allowed": [], 2024-04-23T10:41:09.117 INFO:teuthology.orchestra.run.smithi114.stdout: "desc": "name prefix for any created local pool", 2024-04-23T10:41:09.117 INFO:teuthology.orchestra.run.smithi114.stdout: "long_desc": "", 2024-04-23T10:41:09.117 INFO:teuthology.orchestra.run.smithi114.stdout: "tags": [], 2024-04-23T10:41:09.117 INFO:teuthology.orchestra.run.smithi114.stdout: "see_also": [] 2024-04-23T10:41:09.118 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-23T10:41:09.118 INFO:teuthology.orchestra.run.smithi114.stdout: "subtree": { 2024-04-23T10:41:09.118 INFO:teuthology.orchestra.run.smithi114.stdout: "name": "subtree", 2024-04-23T10:41:09.118 INFO:teuthology.orchestra.run.smithi114.stdout: "type": "str", 2024-04-23T10:41:09.118 INFO:teuthology.orchestra.run.smithi114.stdout: "level": "advanced", 2024-04-23T10:41:09.118 INFO:teuthology.orchestra.run.smithi114.stdout: "flags": 1, 2024-04-23T10:41:09.118 INFO:teuthology.orchestra.run.smithi114.stdout: "default_value": "rack", 2024-04-23T10:41:09.118 INFO:teuthology.orchestra.run.smithi114.stdout: "min": "", 2024-04-23T10:41:09.119 INFO:teuthology.orchestra.run.smithi114.stdout: "max": "", 2024-04-23T10:41:09.119 INFO:teuthology.orchestra.run.smithi114.stdout: "enum_allowed": [], 2024-04-23T10:41:09.119 INFO:teuthology.orchestra.run.smithi114.stdout: "desc": "CRUSH level for which to create a local pool", 2024-04-23T10:41:09.119 INFO:teuthology.orchestra.run.smithi114.stdout: "long_desc": "which CRUSH subtree type the module should create a pool for.", 2024-04-23T10:41:09.119 INFO:teuthology.orchestra.run.smithi114.stdout: "tags": [], 2024-04-23T10:41:09.119 INFO:teuthology.orchestra.run.smithi114.stdout: "see_also": [] 2024-04-23T10:41:09.119 INFO:teuthology.orchestra.run.smithi114.stdout: } 2024-04-23T10:41:09.119 INFO:teuthology.orchestra.run.smithi114.stdout: } 2024-04-23T10:41:09.120 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-23T10:41:09.120 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-04-23T10:41:09.120 INFO:teuthology.orchestra.run.smithi114.stdout: "name": "mds_autoscaler", 2024-04-23T10:41:09.120 INFO:teuthology.orchestra.run.smithi114.stdout: "can_run": true, 2024-04-23T10:41:09.120 INFO:teuthology.orchestra.run.smithi114.stdout: "error_string": "", 2024-04-23T10:41:09.120 INFO:teuthology.orchestra.run.smithi114.stdout: "module_options": { 2024-04-23T10:41:09.120 INFO:teuthology.orchestra.run.smithi114.stdout: "log_level": { 2024-04-23T10:41:09.120 INFO:teuthology.orchestra.run.smithi114.stdout: "name": "log_level", 2024-04-23T10:41:09.120 INFO:teuthology.orchestra.run.smithi114.stdout: "type": "str", 2024-04-23T10:41:09.121 INFO:teuthology.orchestra.run.smithi114.stdout: "level": "advanced", 2024-04-23T10:41:09.121 INFO:teuthology.orchestra.run.smithi114.stdout: "flags": 1, 2024-04-23T10:41:09.121 INFO:teuthology.orchestra.run.smithi114.stdout: "default_value": "", 2024-04-23T10:41:09.121 INFO:teuthology.orchestra.run.smithi114.stdout: "min": "", 2024-04-23T10:41:09.121 INFO:teuthology.orchestra.run.smithi114.stdout: "max": "", 2024-04-23T10:41:09.121 INFO:teuthology.orchestra.run.smithi114.stdout: "enum_allowed": [ 2024-04-23T10:41:09.121 INFO:teuthology.orchestra.run.smithi114.stdout: "", 2024-04-23T10:41:09.121 INFO:teuthology.orchestra.run.smithi114.stdout: "critical", 2024-04-23T10:41:09.121 INFO:teuthology.orchestra.run.smithi114.stdout: "debug", 2024-04-23T10:41:09.122 INFO:teuthology.orchestra.run.smithi114.stdout: "error", 2024-04-23T10:41:09.122 INFO:teuthology.orchestra.run.smithi114.stdout: "info", 2024-04-23T10:41:09.122 INFO:teuthology.orchestra.run.smithi114.stdout: "warning" 2024-04-23T10:41:09.122 INFO:teuthology.orchestra.run.smithi114.stdout: ], 2024-04-23T10:41:09.122 INFO:teuthology.orchestra.run.smithi114.stdout: "desc": "", 2024-04-23T10:41:09.122 INFO:teuthology.orchestra.run.smithi114.stdout: "long_desc": "", 2024-04-23T10:41:09.122 INFO:teuthology.orchestra.run.smithi114.stdout: "tags": [], 2024-04-23T10:41:09.122 INFO:teuthology.orchestra.run.smithi114.stdout: "see_also": [] 2024-04-23T10:41:09.122 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-23T10:41:09.123 INFO:teuthology.orchestra.run.smithi114.stdout: "log_to_cluster": { 2024-04-23T10:41:09.123 INFO:teuthology.orchestra.run.smithi114.stdout: "name": "log_to_cluster", 2024-04-23T10:41:09.123 INFO:teuthology.orchestra.run.smithi114.stdout: "type": "bool", 2024-04-23T10:41:09.123 INFO:teuthology.orchestra.run.smithi114.stdout: "level": "advanced", 2024-04-23T10:41:09.123 INFO:teuthology.orchestra.run.smithi114.stdout: "flags": 1, 2024-04-23T10:41:09.123 INFO:teuthology.orchestra.run.smithi114.stdout: "default_value": "False", 2024-04-23T10:41:09.123 INFO:teuthology.orchestra.run.smithi114.stdout: "min": "", 2024-04-23T10:41:09.123 INFO:teuthology.orchestra.run.smithi114.stdout: "max": "", 2024-04-23T10:41:09.124 INFO:teuthology.orchestra.run.smithi114.stdout: "enum_allowed": [], 2024-04-23T10:41:09.124 INFO:teuthology.orchestra.run.smithi114.stdout: "desc": "", 2024-04-23T10:41:09.124 INFO:teuthology.orchestra.run.smithi114.stdout: "long_desc": "", 2024-04-23T10:41:09.124 INFO:teuthology.orchestra.run.smithi114.stdout: "tags": [], 2024-04-23T10:41:09.124 INFO:teuthology.orchestra.run.smithi114.stdout: "see_also": [] 2024-04-23T10:41:09.124 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-23T10:41:09.124 INFO:teuthology.orchestra.run.smithi114.stdout: "log_to_cluster_level": { 2024-04-23T10:41:09.124 INFO:teuthology.orchestra.run.smithi114.stdout: "name": "log_to_cluster_level", 2024-04-23T10:41:09.124 INFO:teuthology.orchestra.run.smithi114.stdout: "type": "str", 2024-04-23T10:41:09.125 INFO:teuthology.orchestra.run.smithi114.stdout: "level": "advanced", 2024-04-23T10:41:09.125 INFO:teuthology.orchestra.run.smithi114.stdout: "flags": 1, 2024-04-23T10:41:09.125 INFO:teuthology.orchestra.run.smithi114.stdout: "default_value": "info", 2024-04-23T10:41:09.125 INFO:teuthology.orchestra.run.smithi114.stdout: "min": "", 2024-04-23T10:41:09.125 INFO:teuthology.orchestra.run.smithi114.stdout: "max": "", 2024-04-23T10:41:09.125 INFO:teuthology.orchestra.run.smithi114.stdout: "enum_allowed": [ 2024-04-23T10:41:09.125 INFO:teuthology.orchestra.run.smithi114.stdout: "", 2024-04-23T10:41:09.125 INFO:teuthology.orchestra.run.smithi114.stdout: "critical", 2024-04-23T10:41:09.126 INFO:teuthology.orchestra.run.smithi114.stdout: "debug", 2024-04-23T10:41:09.126 INFO:teuthology.orchestra.run.smithi114.stdout: "error", 2024-04-23T10:41:09.126 INFO:teuthology.orchestra.run.smithi114.stdout: "info", 2024-04-23T10:41:09.126 INFO:teuthology.orchestra.run.smithi114.stdout: "warning" 2024-04-23T10:41:09.126 INFO:teuthology.orchestra.run.smithi114.stdout: ], 2024-04-23T10:41:09.126 INFO:teuthology.orchestra.run.smithi114.stdout: "desc": "", 2024-04-23T10:41:09.126 INFO:teuthology.orchestra.run.smithi114.stdout: "long_desc": "", 2024-04-23T10:41:09.126 INFO:teuthology.orchestra.run.smithi114.stdout: "tags": [], 2024-04-23T10:41:09.126 INFO:teuthology.orchestra.run.smithi114.stdout: "see_also": [] 2024-04-23T10:41:09.127 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-23T10:41:09.127 INFO:teuthology.orchestra.run.smithi114.stdout: "log_to_file": { 2024-04-23T10:41:09.127 INFO:teuthology.orchestra.run.smithi114.stdout: "name": "log_to_file", 2024-04-23T10:41:09.127 INFO:teuthology.orchestra.run.smithi114.stdout: "type": "bool", 2024-04-23T10:41:09.127 INFO:teuthology.orchestra.run.smithi114.stdout: "level": "advanced", 2024-04-23T10:41:09.127 INFO:teuthology.orchestra.run.smithi114.stdout: "flags": 1, 2024-04-23T10:41:09.127 INFO:teuthology.orchestra.run.smithi114.stdout: "default_value": "False", 2024-04-23T10:41:09.128 INFO:teuthology.orchestra.run.smithi114.stdout: "min": "", 2024-04-23T10:41:09.128 INFO:teuthology.orchestra.run.smithi114.stdout: "max": "", 2024-04-23T10:41:09.128 INFO:teuthology.orchestra.run.smithi114.stdout: "enum_allowed": [], 2024-04-23T10:41:09.128 INFO:teuthology.orchestra.run.smithi114.stdout: "desc": "", 2024-04-23T10:41:09.128 INFO:teuthology.orchestra.run.smithi114.stdout: "long_desc": "", 2024-04-23T10:41:09.128 INFO:teuthology.orchestra.run.smithi114.stdout: "tags": [], 2024-04-23T10:41:09.128 INFO:teuthology.orchestra.run.smithi114.stdout: "see_also": [] 2024-04-23T10:41:09.128 INFO:teuthology.orchestra.run.smithi114.stdout: } 2024-04-23T10:41:09.128 INFO:teuthology.orchestra.run.smithi114.stdout: } 2024-04-23T10:41:09.129 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-23T10:41:09.129 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-04-23T10:41:09.129 INFO:teuthology.orchestra.run.smithi114.stdout: "name": "mirroring", 2024-04-23T10:41:09.129 INFO:teuthology.orchestra.run.smithi114.stdout: "can_run": true, 2024-04-23T10:41:09.129 INFO:teuthology.orchestra.run.smithi114.stdout: "error_string": "", 2024-04-23T10:41:09.129 INFO:teuthology.orchestra.run.smithi114.stdout: "module_options": { 2024-04-23T10:41:09.129 INFO:teuthology.orchestra.run.smithi114.stdout: "log_level": { 2024-04-23T10:41:09.129 INFO:teuthology.orchestra.run.smithi114.stdout: "name": "log_level", 2024-04-23T10:41:09.129 INFO:teuthology.orchestra.run.smithi114.stdout: "type": "str", 2024-04-23T10:41:09.130 INFO:teuthology.orchestra.run.smithi114.stdout: "level": "advanced", 2024-04-23T10:41:09.130 INFO:teuthology.orchestra.run.smithi114.stdout: "flags": 1, 2024-04-23T10:41:09.130 INFO:teuthology.orchestra.run.smithi114.stdout: "default_value": "", 2024-04-23T10:41:09.130 INFO:teuthology.orchestra.run.smithi114.stdout: "min": "", 2024-04-23T10:41:09.130 INFO:teuthology.orchestra.run.smithi114.stdout: "max": "", 2024-04-23T10:41:09.130 INFO:teuthology.orchestra.run.smithi114.stdout: "enum_allowed": [ 2024-04-23T10:41:09.130 INFO:teuthology.orchestra.run.smithi114.stdout: "", 2024-04-23T10:41:09.130 INFO:teuthology.orchestra.run.smithi114.stdout: "critical", 2024-04-23T10:41:09.131 INFO:teuthology.orchestra.run.smithi114.stdout: "debug", 2024-04-23T10:41:09.131 INFO:teuthology.orchestra.run.smithi114.stdout: "error", 2024-04-23T10:41:09.131 INFO:teuthology.orchestra.run.smithi114.stdout: "info", 2024-04-23T10:41:09.131 INFO:teuthology.orchestra.run.smithi114.stdout: "warning" 2024-04-23T10:41:09.131 INFO:teuthology.orchestra.run.smithi114.stdout: ], 2024-04-23T10:41:09.131 INFO:teuthology.orchestra.run.smithi114.stdout: "desc": "", 2024-04-23T10:41:09.131 INFO:teuthology.orchestra.run.smithi114.stdout: "long_desc": "", 2024-04-23T10:41:09.131 INFO:teuthology.orchestra.run.smithi114.stdout: "tags": [], 2024-04-23T10:41:09.131 INFO:teuthology.orchestra.run.smithi114.stdout: "see_also": [] 2024-04-23T10:41:09.132 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-23T10:41:09.132 INFO:teuthology.orchestra.run.smithi114.stdout: "log_to_cluster": { 2024-04-23T10:41:09.132 INFO:teuthology.orchestra.run.smithi114.stdout: "name": "log_to_cluster", 2024-04-23T10:41:09.132 INFO:teuthology.orchestra.run.smithi114.stdout: "type": "bool", 2024-04-23T10:41:09.132 INFO:teuthology.orchestra.run.smithi114.stdout: "level": "advanced", 2024-04-23T10:41:09.132 INFO:teuthology.orchestra.run.smithi114.stdout: "flags": 1, 2024-04-23T10:41:09.132 INFO:teuthology.orchestra.run.smithi114.stdout: "default_value": "False", 2024-04-23T10:41:09.132 INFO:teuthology.orchestra.run.smithi114.stdout: "min": "", 2024-04-23T10:41:09.133 INFO:teuthology.orchestra.run.smithi114.stdout: "max": "", 2024-04-23T10:41:09.133 INFO:teuthology.orchestra.run.smithi114.stdout: "enum_allowed": [], 2024-04-23T10:41:09.133 INFO:teuthology.orchestra.run.smithi114.stdout: "desc": "", 2024-04-23T10:41:09.133 INFO:teuthology.orchestra.run.smithi114.stdout: "long_desc": "", 2024-04-23T10:41:09.133 INFO:teuthology.orchestra.run.smithi114.stdout: "tags": [], 2024-04-23T10:41:09.133 INFO:teuthology.orchestra.run.smithi114.stdout: "see_also": [] 2024-04-23T10:41:09.133 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-23T10:41:09.133 INFO:teuthology.orchestra.run.smithi114.stdout: "log_to_cluster_level": { 2024-04-23T10:41:09.133 INFO:teuthology.orchestra.run.smithi114.stdout: "name": "log_to_cluster_level", 2024-04-23T10:41:09.134 INFO:teuthology.orchestra.run.smithi114.stdout: "type": "str", 2024-04-23T10:41:09.134 INFO:teuthology.orchestra.run.smithi114.stdout: "level": "advanced", 2024-04-23T10:41:09.134 INFO:teuthology.orchestra.run.smithi114.stdout: "flags": 1, 2024-04-23T10:41:09.134 INFO:teuthology.orchestra.run.smithi114.stdout: "default_value": "info", 2024-04-23T10:41:09.134 INFO:teuthology.orchestra.run.smithi114.stdout: "min": "", 2024-04-23T10:41:09.134 INFO:teuthology.orchestra.run.smithi114.stdout: "max": "", 2024-04-23T10:41:09.134 INFO:teuthology.orchestra.run.smithi114.stdout: "enum_allowed": [ 2024-04-23T10:41:09.134 INFO:teuthology.orchestra.run.smithi114.stdout: "", 2024-04-23T10:41:09.135 INFO:teuthology.orchestra.run.smithi114.stdout: "critical", 2024-04-23T10:41:09.135 INFO:teuthology.orchestra.run.smithi114.stdout: "debug", 2024-04-23T10:41:09.135 INFO:teuthology.orchestra.run.smithi114.stdout: "error", 2024-04-23T10:41:09.135 INFO:teuthology.orchestra.run.smithi114.stdout: "info", 2024-04-23T10:41:09.135 INFO:teuthology.orchestra.run.smithi114.stdout: "warning" 2024-04-23T10:41:09.135 INFO:teuthology.orchestra.run.smithi114.stdout: ], 2024-04-23T10:41:09.135 INFO:teuthology.orchestra.run.smithi114.stdout: "desc": "", 2024-04-23T10:41:09.135 INFO:teuthology.orchestra.run.smithi114.stdout: "long_desc": "", 2024-04-23T10:41:09.135 INFO:teuthology.orchestra.run.smithi114.stdout: "tags": [], 2024-04-23T10:41:09.136 INFO:teuthology.orchestra.run.smithi114.stdout: "see_also": [] 2024-04-23T10:41:09.136 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-23T10:41:09.136 INFO:teuthology.orchestra.run.smithi114.stdout: "log_to_file": { 2024-04-23T10:41:09.136 INFO:teuthology.orchestra.run.smithi114.stdout: "name": "log_to_file", 2024-04-23T10:41:09.136 INFO:teuthology.orchestra.run.smithi114.stdout: "type": "bool", 2024-04-23T10:41:09.136 INFO:teuthology.orchestra.run.smithi114.stdout: "level": "advanced", 2024-04-23T10:41:09.136 INFO:teuthology.orchestra.run.smithi114.stdout: "flags": 1, 2024-04-23T10:41:09.136 INFO:teuthology.orchestra.run.smithi114.stdout: "default_value": "False", 2024-04-23T10:41:09.137 INFO:teuthology.orchestra.run.smithi114.stdout: "min": "", 2024-04-23T10:41:09.137 INFO:teuthology.orchestra.run.smithi114.stdout: "max": "", 2024-04-23T10:41:09.137 INFO:teuthology.orchestra.run.smithi114.stdout: "enum_allowed": [], 2024-04-23T10:41:09.137 INFO:teuthology.orchestra.run.smithi114.stdout: "desc": "", 2024-04-23T10:41:09.137 INFO:teuthology.orchestra.run.smithi114.stdout: "long_desc": "", 2024-04-23T10:41:09.137 INFO:teuthology.orchestra.run.smithi114.stdout: "tags": [], 2024-04-23T10:41:09.137 INFO:teuthology.orchestra.run.smithi114.stdout: "see_also": [] 2024-04-23T10:41:09.137 INFO:teuthology.orchestra.run.smithi114.stdout: } 2024-04-23T10:41:09.137 INFO:teuthology.orchestra.run.smithi114.stdout: } 2024-04-23T10:41:09.137 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-23T10:41:09.138 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-04-23T10:41:09.138 INFO:teuthology.orchestra.run.smithi114.stdout: "name": "nfs", 2024-04-23T10:41:09.138 INFO:teuthology.orchestra.run.smithi114.stdout: "can_run": true, 2024-04-23T10:41:09.138 INFO:teuthology.orchestra.run.smithi114.stdout: "error_string": "", 2024-04-23T10:41:09.138 INFO:teuthology.orchestra.run.smithi114.stdout: "module_options": { 2024-04-23T10:41:09.138 INFO:teuthology.orchestra.run.smithi114.stdout: "log_level": { 2024-04-23T10:41:09.138 INFO:teuthology.orchestra.run.smithi114.stdout: "name": "log_level", 2024-04-23T10:41:09.138 INFO:teuthology.orchestra.run.smithi114.stdout: "type": "str", 2024-04-23T10:41:09.139 INFO:teuthology.orchestra.run.smithi114.stdout: "level": "advanced", 2024-04-23T10:41:09.139 INFO:teuthology.orchestra.run.smithi114.stdout: "flags": 1, 2024-04-23T10:41:09.139 INFO:teuthology.orchestra.run.smithi114.stdout: "default_value": "", 2024-04-23T10:41:09.139 INFO:teuthology.orchestra.run.smithi114.stdout: "min": "", 2024-04-23T10:41:09.139 INFO:teuthology.orchestra.run.smithi114.stdout: "max": "", 2024-04-23T10:41:09.139 INFO:teuthology.orchestra.run.smithi114.stdout: "enum_allowed": [ 2024-04-23T10:41:09.139 INFO:teuthology.orchestra.run.smithi114.stdout: "", 2024-04-23T10:41:09.139 INFO:teuthology.orchestra.run.smithi114.stdout: "critical", 2024-04-23T10:41:09.139 INFO:teuthology.orchestra.run.smithi114.stdout: "debug", 2024-04-23T10:41:09.140 INFO:teuthology.orchestra.run.smithi114.stdout: "error", 2024-04-23T10:41:09.140 INFO:teuthology.orchestra.run.smithi114.stdout: "info", 2024-04-23T10:41:09.140 INFO:teuthology.orchestra.run.smithi114.stdout: "warning" 2024-04-23T10:41:09.140 INFO:teuthology.orchestra.run.smithi114.stdout: ], 2024-04-23T10:41:09.140 INFO:teuthology.orchestra.run.smithi114.stdout: "desc": "", 2024-04-23T10:41:09.140 INFO:teuthology.orchestra.run.smithi114.stdout: "long_desc": "", 2024-04-23T10:41:09.140 INFO:teuthology.orchestra.run.smithi114.stdout: "tags": [], 2024-04-23T10:41:09.140 INFO:teuthology.orchestra.run.smithi114.stdout: "see_also": [] 2024-04-23T10:41:09.140 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-23T10:41:09.141 INFO:teuthology.orchestra.run.smithi114.stdout: "log_to_cluster": { 2024-04-23T10:41:09.141 INFO:teuthology.orchestra.run.smithi114.stdout: "name": "log_to_cluster", 2024-04-23T10:41:09.141 INFO:teuthology.orchestra.run.smithi114.stdout: "type": "bool", 2024-04-23T10:41:09.141 INFO:teuthology.orchestra.run.smithi114.stdout: "level": "advanced", 2024-04-23T10:41:09.141 INFO:teuthology.orchestra.run.smithi114.stdout: "flags": 1, 2024-04-23T10:41:09.141 INFO:teuthology.orchestra.run.smithi114.stdout: "default_value": "False", 2024-04-23T10:41:09.141 INFO:teuthology.orchestra.run.smithi114.stdout: "min": "", 2024-04-23T10:41:09.141 INFO:teuthology.orchestra.run.smithi114.stdout: "max": "", 2024-04-23T10:41:09.142 INFO:teuthology.orchestra.run.smithi114.stdout: "enum_allowed": [], 2024-04-23T10:41:09.142 INFO:teuthology.orchestra.run.smithi114.stdout: "desc": "", 2024-04-23T10:41:09.142 INFO:teuthology.orchestra.run.smithi114.stdout: "long_desc": "", 2024-04-23T10:41:09.142 INFO:teuthology.orchestra.run.smithi114.stdout: "tags": [], 2024-04-23T10:41:09.142 INFO:teuthology.orchestra.run.smithi114.stdout: "see_also": [] 2024-04-23T10:41:09.142 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-23T10:41:09.142 INFO:teuthology.orchestra.run.smithi114.stdout: "log_to_cluster_level": { 2024-04-23T10:41:09.142 INFO:teuthology.orchestra.run.smithi114.stdout: "name": "log_to_cluster_level", 2024-04-23T10:41:09.142 INFO:teuthology.orchestra.run.smithi114.stdout: "type": "str", 2024-04-23T10:41:09.143 INFO:teuthology.orchestra.run.smithi114.stdout: "level": "advanced", 2024-04-23T10:41:09.143 INFO:teuthology.orchestra.run.smithi114.stdout: "flags": 1, 2024-04-23T10:41:09.143 INFO:teuthology.orchestra.run.smithi114.stdout: "default_value": "info", 2024-04-23T10:41:09.143 INFO:teuthology.orchestra.run.smithi114.stdout: "min": "", 2024-04-23T10:41:09.143 INFO:teuthology.orchestra.run.smithi114.stdout: "max": "", 2024-04-23T10:41:09.143 INFO:teuthology.orchestra.run.smithi114.stdout: "enum_allowed": [ 2024-04-23T10:41:09.143 INFO:teuthology.orchestra.run.smithi114.stdout: "", 2024-04-23T10:41:09.143 INFO:teuthology.orchestra.run.smithi114.stdout: "critical", 2024-04-23T10:41:09.143 INFO:teuthology.orchestra.run.smithi114.stdout: "debug", 2024-04-23T10:41:09.144 INFO:teuthology.orchestra.run.smithi114.stdout: "error", 2024-04-23T10:41:09.144 INFO:teuthology.orchestra.run.smithi114.stdout: "info", 2024-04-23T10:41:09.144 INFO:teuthology.orchestra.run.smithi114.stdout: "warning" 2024-04-23T10:41:09.144 INFO:teuthology.orchestra.run.smithi114.stdout: ], 2024-04-23T10:41:09.144 INFO:teuthology.orchestra.run.smithi114.stdout: "desc": "", 2024-04-23T10:41:09.144 INFO:teuthology.orchestra.run.smithi114.stdout: "long_desc": "", 2024-04-23T10:41:09.144 INFO:teuthology.orchestra.run.smithi114.stdout: "tags": [], 2024-04-23T10:41:09.144 INFO:teuthology.orchestra.run.smithi114.stdout: "see_also": [] 2024-04-23T10:41:09.144 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-23T10:41:09.145 INFO:teuthology.orchestra.run.smithi114.stdout: "log_to_file": { 2024-04-23T10:41:09.145 INFO:teuthology.orchestra.run.smithi114.stdout: "name": "log_to_file", 2024-04-23T10:41:09.145 INFO:teuthology.orchestra.run.smithi114.stdout: "type": "bool", 2024-04-23T10:41:09.145 INFO:teuthology.orchestra.run.smithi114.stdout: "level": "advanced", 2024-04-23T10:41:09.145 INFO:teuthology.orchestra.run.smithi114.stdout: "flags": 1, 2024-04-23T10:41:09.145 INFO:teuthology.orchestra.run.smithi114.stdout: "default_value": "False", 2024-04-23T10:41:09.145 INFO:teuthology.orchestra.run.smithi114.stdout: "min": "", 2024-04-23T10:41:09.145 INFO:teuthology.orchestra.run.smithi114.stdout: "max": "", 2024-04-23T10:41:09.146 INFO:teuthology.orchestra.run.smithi114.stdout: "enum_allowed": [], 2024-04-23T10:41:09.146 INFO:teuthology.orchestra.run.smithi114.stdout: "desc": "", 2024-04-23T10:41:09.146 INFO:teuthology.orchestra.run.smithi114.stdout: "long_desc": "", 2024-04-23T10:41:09.146 INFO:teuthology.orchestra.run.smithi114.stdout: "tags": [], 2024-04-23T10:41:09.146 INFO:teuthology.orchestra.run.smithi114.stdout: "see_also": [] 2024-04-23T10:41:09.146 INFO:teuthology.orchestra.run.smithi114.stdout: } 2024-04-23T10:41:09.146 INFO:teuthology.orchestra.run.smithi114.stdout: } 2024-04-23T10:41:09.146 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-23T10:41:09.146 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-04-23T10:41:09.147 INFO:teuthology.orchestra.run.smithi114.stdout: "name": "orchestrator", 2024-04-23T10:41:09.147 INFO:teuthology.orchestra.run.smithi114.stdout: "can_run": true, 2024-04-23T10:41:09.147 INFO:teuthology.orchestra.run.smithi114.stdout: "error_string": "", 2024-04-23T10:41:09.147 INFO:teuthology.orchestra.run.smithi114.stdout: "module_options": { 2024-04-23T10:41:09.147 INFO:teuthology.orchestra.run.smithi114.stdout: "fail_fs": { 2024-04-23T10:41:09.147 INFO:teuthology.orchestra.run.smithi114.stdout: "name": "fail_fs", 2024-04-23T10:41:09.147 INFO:teuthology.orchestra.run.smithi114.stdout: "type": "bool", 2024-04-23T10:41:09.147 INFO:teuthology.orchestra.run.smithi114.stdout: "level": "advanced", 2024-04-23T10:41:09.148 INFO:teuthology.orchestra.run.smithi114.stdout: "flags": 0, 2024-04-23T10:41:09.148 INFO:teuthology.orchestra.run.smithi114.stdout: "default_value": "False", 2024-04-23T10:41:09.148 INFO:teuthology.orchestra.run.smithi114.stdout: "min": "", 2024-04-23T10:41:09.148 INFO:teuthology.orchestra.run.smithi114.stdout: "max": "", 2024-04-23T10:41:09.148 INFO:teuthology.orchestra.run.smithi114.stdout: "enum_allowed": [], 2024-04-23T10:41:09.148 INFO:teuthology.orchestra.run.smithi114.stdout: "desc": "Fail filesystem for rapid multi-rank mds upgrade", 2024-04-23T10:41:09.148 INFO:teuthology.orchestra.run.smithi114.stdout: "long_desc": "", 2024-04-23T10:41:09.148 INFO:teuthology.orchestra.run.smithi114.stdout: "tags": [], 2024-04-23T10:41:09.148 INFO:teuthology.orchestra.run.smithi114.stdout: "see_also": [] 2024-04-23T10:41:09.149 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-23T10:41:09.149 INFO:teuthology.orchestra.run.smithi114.stdout: "log_level": { 2024-04-23T10:41:09.149 INFO:teuthology.orchestra.run.smithi114.stdout: "name": "log_level", 2024-04-23T10:41:09.149 INFO:teuthology.orchestra.run.smithi114.stdout: "type": "str", 2024-04-23T10:41:09.149 INFO:teuthology.orchestra.run.smithi114.stdout: "level": "advanced", 2024-04-23T10:41:09.149 INFO:teuthology.orchestra.run.smithi114.stdout: "flags": 1, 2024-04-23T10:41:09.149 INFO:teuthology.orchestra.run.smithi114.stdout: "default_value": "", 2024-04-23T10:41:09.149 INFO:teuthology.orchestra.run.smithi114.stdout: "min": "", 2024-04-23T10:41:09.149 INFO:teuthology.orchestra.run.smithi114.stdout: "max": "", 2024-04-23T10:41:09.150 INFO:teuthology.orchestra.run.smithi114.stdout: "enum_allowed": [ 2024-04-23T10:41:09.150 INFO:teuthology.orchestra.run.smithi114.stdout: "", 2024-04-23T10:41:09.150 INFO:teuthology.orchestra.run.smithi114.stdout: "critical", 2024-04-23T10:41:09.150 INFO:teuthology.orchestra.run.smithi114.stdout: "debug", 2024-04-23T10:41:09.150 INFO:teuthology.orchestra.run.smithi114.stdout: "error", 2024-04-23T10:41:09.150 INFO:teuthology.orchestra.run.smithi114.stdout: "info", 2024-04-23T10:41:09.150 INFO:teuthology.orchestra.run.smithi114.stdout: "warning" 2024-04-23T10:41:09.150 INFO:teuthology.orchestra.run.smithi114.stdout: ], 2024-04-23T10:41:09.151 INFO:teuthology.orchestra.run.smithi114.stdout: "desc": "", 2024-04-23T10:41:09.151 INFO:teuthology.orchestra.run.smithi114.stdout: "long_desc": "", 2024-04-23T10:41:09.151 INFO:teuthology.orchestra.run.smithi114.stdout: "tags": [], 2024-04-23T10:41:09.151 INFO:teuthology.orchestra.run.smithi114.stdout: "see_also": [] 2024-04-23T10:41:09.151 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-23T10:41:09.151 INFO:teuthology.orchestra.run.smithi114.stdout: "log_to_cluster": { 2024-04-23T10:41:09.151 INFO:teuthology.orchestra.run.smithi114.stdout: "name": "log_to_cluster", 2024-04-23T10:41:09.151 INFO:teuthology.orchestra.run.smithi114.stdout: "type": "bool", 2024-04-23T10:41:09.152 INFO:teuthology.orchestra.run.smithi114.stdout: "level": "advanced", 2024-04-23T10:41:09.152 INFO:teuthology.orchestra.run.smithi114.stdout: "flags": 1, 2024-04-23T10:41:09.152 INFO:teuthology.orchestra.run.smithi114.stdout: "default_value": "False", 2024-04-23T10:41:09.152 INFO:teuthology.orchestra.run.smithi114.stdout: "min": "", 2024-04-23T10:41:09.152 INFO:teuthology.orchestra.run.smithi114.stdout: "max": "", 2024-04-23T10:41:09.152 INFO:teuthology.orchestra.run.smithi114.stdout: "enum_allowed": [], 2024-04-23T10:41:09.152 INFO:teuthology.orchestra.run.smithi114.stdout: "desc": "", 2024-04-23T10:41:09.152 INFO:teuthology.orchestra.run.smithi114.stdout: "long_desc": "", 2024-04-23T10:41:09.152 INFO:teuthology.orchestra.run.smithi114.stdout: "tags": [], 2024-04-23T10:41:09.153 INFO:teuthology.orchestra.run.smithi114.stdout: "see_also": [] 2024-04-23T10:41:09.153 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-23T10:41:09.153 INFO:teuthology.orchestra.run.smithi114.stdout: "log_to_cluster_level": { 2024-04-23T10:41:09.153 INFO:teuthology.orchestra.run.smithi114.stdout: "name": "log_to_cluster_level", 2024-04-23T10:41:09.153 INFO:teuthology.orchestra.run.smithi114.stdout: "type": "str", 2024-04-23T10:41:09.153 INFO:teuthology.orchestra.run.smithi114.stdout: "level": "advanced", 2024-04-23T10:41:09.153 INFO:teuthology.orchestra.run.smithi114.stdout: "flags": 1, 2024-04-23T10:41:09.153 INFO:teuthology.orchestra.run.smithi114.stdout: "default_value": "info", 2024-04-23T10:41:09.153 INFO:teuthology.orchestra.run.smithi114.stdout: "min": "", 2024-04-23T10:41:09.154 INFO:teuthology.orchestra.run.smithi114.stdout: "max": "", 2024-04-23T10:41:09.154 INFO:teuthology.orchestra.run.smithi114.stdout: "enum_allowed": [ 2024-04-23T10:41:09.154 INFO:teuthology.orchestra.run.smithi114.stdout: "", 2024-04-23T10:41:09.154 INFO:teuthology.orchestra.run.smithi114.stdout: "critical", 2024-04-23T10:41:09.154 INFO:teuthology.orchestra.run.smithi114.stdout: "debug", 2024-04-23T10:41:09.154 INFO:teuthology.orchestra.run.smithi114.stdout: "error", 2024-04-23T10:41:09.154 INFO:teuthology.orchestra.run.smithi114.stdout: "info", 2024-04-23T10:41:09.154 INFO:teuthology.orchestra.run.smithi114.stdout: "warning" 2024-04-23T10:41:09.155 INFO:teuthology.orchestra.run.smithi114.stdout: ], 2024-04-23T10:41:09.155 INFO:teuthology.orchestra.run.smithi114.stdout: "desc": "", 2024-04-23T10:41:09.155 INFO:teuthology.orchestra.run.smithi114.stdout: "long_desc": "", 2024-04-23T10:41:09.155 INFO:teuthology.orchestra.run.smithi114.stdout: "tags": [], 2024-04-23T10:41:09.155 INFO:teuthology.orchestra.run.smithi114.stdout: "see_also": [] 2024-04-23T10:41:09.155 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-23T10:41:09.155 INFO:teuthology.orchestra.run.smithi114.stdout: "log_to_file": { 2024-04-23T10:41:09.155 INFO:teuthology.orchestra.run.smithi114.stdout: "name": "log_to_file", 2024-04-23T10:41:09.155 INFO:teuthology.orchestra.run.smithi114.stdout: "type": "bool", 2024-04-23T10:41:09.156 INFO:teuthology.orchestra.run.smithi114.stdout: "level": "advanced", 2024-04-23T10:41:09.156 INFO:teuthology.orchestra.run.smithi114.stdout: "flags": 1, 2024-04-23T10:41:09.156 INFO:teuthology.orchestra.run.smithi114.stdout: "default_value": "False", 2024-04-23T10:41:09.156 INFO:teuthology.orchestra.run.smithi114.stdout: "min": "", 2024-04-23T10:41:09.156 INFO:teuthology.orchestra.run.smithi114.stdout: "max": "", 2024-04-23T10:41:09.156 INFO:teuthology.orchestra.run.smithi114.stdout: "enum_allowed": [], 2024-04-23T10:41:09.156 INFO:teuthology.orchestra.run.smithi114.stdout: "desc": "", 2024-04-23T10:41:09.156 INFO:teuthology.orchestra.run.smithi114.stdout: "long_desc": "", 2024-04-23T10:41:09.156 INFO:teuthology.orchestra.run.smithi114.stdout: "tags": [], 2024-04-23T10:41:09.157 INFO:teuthology.orchestra.run.smithi114.stdout: "see_also": [] 2024-04-23T10:41:09.157 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-23T10:41:09.157 INFO:teuthology.orchestra.run.smithi114.stdout: "orchestrator": { 2024-04-23T10:41:09.157 INFO:teuthology.orchestra.run.smithi114.stdout: "name": "orchestrator", 2024-04-23T10:41:09.157 INFO:teuthology.orchestra.run.smithi114.stdout: "type": "str", 2024-04-23T10:41:09.157 INFO:teuthology.orchestra.run.smithi114.stdout: "level": "advanced", 2024-04-23T10:41:09.157 INFO:teuthology.orchestra.run.smithi114.stdout: "flags": 1, 2024-04-23T10:41:09.157 INFO:teuthology.orchestra.run.smithi114.stdout: "default_value": "", 2024-04-23T10:41:09.157 INFO:teuthology.orchestra.run.smithi114.stdout: "min": "", 2024-04-23T10:41:09.158 INFO:teuthology.orchestra.run.smithi114.stdout: "max": "", 2024-04-23T10:41:09.158 INFO:teuthology.orchestra.run.smithi114.stdout: "enum_allowed": [ 2024-04-23T10:41:09.158 INFO:teuthology.orchestra.run.smithi114.stdout: "cephadm", 2024-04-23T10:41:09.158 INFO:teuthology.orchestra.run.smithi114.stdout: "rook", 2024-04-23T10:41:09.158 INFO:teuthology.orchestra.run.smithi114.stdout: "test_orchestrator" 2024-04-23T10:41:09.158 INFO:teuthology.orchestra.run.smithi114.stdout: ], 2024-04-23T10:41:09.158 INFO:teuthology.orchestra.run.smithi114.stdout: "desc": "Orchestrator backend", 2024-04-23T10:41:09.158 INFO:teuthology.orchestra.run.smithi114.stdout: "long_desc": "", 2024-04-23T10:41:09.159 INFO:teuthology.orchestra.run.smithi114.stdout: "tags": [], 2024-04-23T10:41:09.159 INFO:teuthology.orchestra.run.smithi114.stdout: "see_also": [] 2024-04-23T10:41:09.159 INFO:teuthology.orchestra.run.smithi114.stdout: } 2024-04-23T10:41:09.159 INFO:teuthology.orchestra.run.smithi114.stdout: } 2024-04-23T10:41:09.159 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-23T10:41:09.159 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-04-23T10:41:09.159 INFO:teuthology.orchestra.run.smithi114.stdout: "name": "osd_perf_query", 2024-04-23T10:41:09.159 INFO:teuthology.orchestra.run.smithi114.stdout: "can_run": true, 2024-04-23T10:41:09.159 INFO:teuthology.orchestra.run.smithi114.stdout: "error_string": "", 2024-04-23T10:41:09.160 INFO:teuthology.orchestra.run.smithi114.stdout: "module_options": { 2024-04-23T10:41:09.160 INFO:teuthology.orchestra.run.smithi114.stdout: "log_level": { 2024-04-23T10:41:09.160 INFO:teuthology.orchestra.run.smithi114.stdout: "name": "log_level", 2024-04-23T10:41:09.160 INFO:teuthology.orchestra.run.smithi114.stdout: "type": "str", 2024-04-23T10:41:09.160 INFO:teuthology.orchestra.run.smithi114.stdout: "level": "advanced", 2024-04-23T10:41:09.160 INFO:teuthology.orchestra.run.smithi114.stdout: "flags": 1, 2024-04-23T10:41:09.160 INFO:teuthology.orchestra.run.smithi114.stdout: "default_value": "", 2024-04-23T10:41:09.160 INFO:teuthology.orchestra.run.smithi114.stdout: "min": "", 2024-04-23T10:41:09.160 INFO:teuthology.orchestra.run.smithi114.stdout: "max": "", 2024-04-23T10:41:09.161 INFO:teuthology.orchestra.run.smithi114.stdout: "enum_allowed": [ 2024-04-23T10:41:09.161 INFO:teuthology.orchestra.run.smithi114.stdout: "", 2024-04-23T10:41:09.161 INFO:teuthology.orchestra.run.smithi114.stdout: "critical", 2024-04-23T10:41:09.161 INFO:teuthology.orchestra.run.smithi114.stdout: "debug", 2024-04-23T10:41:09.161 INFO:teuthology.orchestra.run.smithi114.stdout: "error", 2024-04-23T10:41:09.161 INFO:teuthology.orchestra.run.smithi114.stdout: "info", 2024-04-23T10:41:09.161 INFO:teuthology.orchestra.run.smithi114.stdout: "warning" 2024-04-23T10:41:09.161 INFO:teuthology.orchestra.run.smithi114.stdout: ], 2024-04-23T10:41:09.161 INFO:teuthology.orchestra.run.smithi114.stdout: "desc": "", 2024-04-23T10:41:09.162 INFO:teuthology.orchestra.run.smithi114.stdout: "long_desc": "", 2024-04-23T10:41:09.162 INFO:teuthology.orchestra.run.smithi114.stdout: "tags": [], 2024-04-23T10:41:09.162 INFO:teuthology.orchestra.run.smithi114.stdout: "see_also": [] 2024-04-23T10:41:09.162 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-23T10:41:09.162 INFO:teuthology.orchestra.run.smithi114.stdout: "log_to_cluster": { 2024-04-23T10:41:09.162 INFO:teuthology.orchestra.run.smithi114.stdout: "name": "log_to_cluster", 2024-04-23T10:41:09.162 INFO:teuthology.orchestra.run.smithi114.stdout: "type": "bool", 2024-04-23T10:41:09.162 INFO:teuthology.orchestra.run.smithi114.stdout: "level": "advanced", 2024-04-23T10:41:09.163 INFO:teuthology.orchestra.run.smithi114.stdout: "flags": 1, 2024-04-23T10:41:09.163 INFO:teuthology.orchestra.run.smithi114.stdout: "default_value": "False", 2024-04-23T10:41:09.163 INFO:teuthology.orchestra.run.smithi114.stdout: "min": "", 2024-04-23T10:41:09.163 INFO:teuthology.orchestra.run.smithi114.stdout: "max": "", 2024-04-23T10:41:09.163 INFO:teuthology.orchestra.run.smithi114.stdout: "enum_allowed": [], 2024-04-23T10:41:09.163 INFO:teuthology.orchestra.run.smithi114.stdout: "desc": "", 2024-04-23T10:41:09.163 INFO:teuthology.orchestra.run.smithi114.stdout: "long_desc": "", 2024-04-23T10:41:09.163 INFO:teuthology.orchestra.run.smithi114.stdout: "tags": [], 2024-04-23T10:41:09.163 INFO:teuthology.orchestra.run.smithi114.stdout: "see_also": [] 2024-04-23T10:41:09.164 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-23T10:41:09.164 INFO:teuthology.orchestra.run.smithi114.stdout: "log_to_cluster_level": { 2024-04-23T10:41:09.164 INFO:teuthology.orchestra.run.smithi114.stdout: "name": "log_to_cluster_level", 2024-04-23T10:41:09.164 INFO:teuthology.orchestra.run.smithi114.stdout: "type": "str", 2024-04-23T10:41:09.164 INFO:teuthology.orchestra.run.smithi114.stdout: "level": "advanced", 2024-04-23T10:41:09.164 INFO:teuthology.orchestra.run.smithi114.stdout: "flags": 1, 2024-04-23T10:41:09.164 INFO:teuthology.orchestra.run.smithi114.stdout: "default_value": "info", 2024-04-23T10:41:09.164 INFO:teuthology.orchestra.run.smithi114.stdout: "min": "", 2024-04-23T10:41:09.165 INFO:teuthology.orchestra.run.smithi114.stdout: "max": "", 2024-04-23T10:41:09.165 INFO:teuthology.orchestra.run.smithi114.stdout: "enum_allowed": [ 2024-04-23T10:41:09.165 INFO:teuthology.orchestra.run.smithi114.stdout: "", 2024-04-23T10:41:09.165 INFO:teuthology.orchestra.run.smithi114.stdout: "critical", 2024-04-23T10:41:09.165 INFO:teuthology.orchestra.run.smithi114.stdout: "debug", 2024-04-23T10:41:09.165 INFO:teuthology.orchestra.run.smithi114.stdout: "error", 2024-04-23T10:41:09.165 INFO:teuthology.orchestra.run.smithi114.stdout: "info", 2024-04-23T10:41:09.165 INFO:teuthology.orchestra.run.smithi114.stdout: "warning" 2024-04-23T10:41:09.165 INFO:teuthology.orchestra.run.smithi114.stdout: ], 2024-04-23T10:41:09.166 INFO:teuthology.orchestra.run.smithi114.stdout: "desc": "", 2024-04-23T10:41:09.166 INFO:teuthology.orchestra.run.smithi114.stdout: "long_desc": "", 2024-04-23T10:41:09.166 INFO:teuthology.orchestra.run.smithi114.stdout: "tags": [], 2024-04-23T10:41:09.166 INFO:teuthology.orchestra.run.smithi114.stdout: "see_also": [] 2024-04-23T10:41:09.166 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-23T10:41:09.166 INFO:teuthology.orchestra.run.smithi114.stdout: "log_to_file": { 2024-04-23T10:41:09.166 INFO:teuthology.orchestra.run.smithi114.stdout: "name": "log_to_file", 2024-04-23T10:41:09.166 INFO:teuthology.orchestra.run.smithi114.stdout: "type": "bool", 2024-04-23T10:41:09.167 INFO:teuthology.orchestra.run.smithi114.stdout: "level": "advanced", 2024-04-23T10:41:09.167 INFO:teuthology.orchestra.run.smithi114.stdout: "flags": 1, 2024-04-23T10:41:09.167 INFO:teuthology.orchestra.run.smithi114.stdout: "default_value": "False", 2024-04-23T10:41:09.167 INFO:teuthology.orchestra.run.smithi114.stdout: "min": "", 2024-04-23T10:41:09.167 INFO:teuthology.orchestra.run.smithi114.stdout: "max": "", 2024-04-23T10:41:09.167 INFO:teuthology.orchestra.run.smithi114.stdout: "enum_allowed": [], 2024-04-23T10:41:09.167 INFO:teuthology.orchestra.run.smithi114.stdout: "desc": "", 2024-04-23T10:41:09.167 INFO:teuthology.orchestra.run.smithi114.stdout: "long_desc": "", 2024-04-23T10:41:09.167 INFO:teuthology.orchestra.run.smithi114.stdout: "tags": [], 2024-04-23T10:41:09.168 INFO:teuthology.orchestra.run.smithi114.stdout: "see_also": [] 2024-04-23T10:41:09.168 INFO:teuthology.orchestra.run.smithi114.stdout: } 2024-04-23T10:41:09.168 INFO:teuthology.orchestra.run.smithi114.stdout: } 2024-04-23T10:41:09.168 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-23T10:41:09.168 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-04-23T10:41:09.168 INFO:teuthology.orchestra.run.smithi114.stdout: "name": "osd_support", 2024-04-23T10:41:09.168 INFO:teuthology.orchestra.run.smithi114.stdout: "can_run": true, 2024-04-23T10:41:09.168 INFO:teuthology.orchestra.run.smithi114.stdout: "error_string": "", 2024-04-23T10:41:09.169 INFO:teuthology.orchestra.run.smithi114.stdout: "module_options": { 2024-04-23T10:41:09.169 INFO:teuthology.orchestra.run.smithi114.stdout: "log_level": { 2024-04-23T10:41:09.169 INFO:teuthology.orchestra.run.smithi114.stdout: "name": "log_level", 2024-04-23T10:41:09.169 INFO:teuthology.orchestra.run.smithi114.stdout: "type": "str", 2024-04-23T10:41:09.169 INFO:teuthology.orchestra.run.smithi114.stdout: "level": "advanced", 2024-04-23T10:41:09.169 INFO:teuthology.orchestra.run.smithi114.stdout: "flags": 1, 2024-04-23T10:41:09.169 INFO:teuthology.orchestra.run.smithi114.stdout: "default_value": "", 2024-04-23T10:41:09.169 INFO:teuthology.orchestra.run.smithi114.stdout: "min": "", 2024-04-23T10:41:09.169 INFO:teuthology.orchestra.run.smithi114.stdout: "max": "", 2024-04-23T10:41:09.170 INFO:teuthology.orchestra.run.smithi114.stdout: "enum_allowed": [ 2024-04-23T10:41:09.170 INFO:teuthology.orchestra.run.smithi114.stdout: "", 2024-04-23T10:41:09.170 INFO:teuthology.orchestra.run.smithi114.stdout: "critical", 2024-04-23T10:41:09.170 INFO:teuthology.orchestra.run.smithi114.stdout: "debug", 2024-04-23T10:41:09.170 INFO:teuthology.orchestra.run.smithi114.stdout: "error", 2024-04-23T10:41:09.170 INFO:teuthology.orchestra.run.smithi114.stdout: "info", 2024-04-23T10:41:09.170 INFO:teuthology.orchestra.run.smithi114.stdout: "warning" 2024-04-23T10:41:09.170 INFO:teuthology.orchestra.run.smithi114.stdout: ], 2024-04-23T10:41:09.170 INFO:teuthology.orchestra.run.smithi114.stdout: "desc": "", 2024-04-23T10:41:09.171 INFO:teuthology.orchestra.run.smithi114.stdout: "long_desc": "", 2024-04-23T10:41:09.171 INFO:teuthology.orchestra.run.smithi114.stdout: "tags": [], 2024-04-23T10:41:09.171 INFO:teuthology.orchestra.run.smithi114.stdout: "see_also": [] 2024-04-23T10:41:09.171 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-23T10:41:09.171 INFO:teuthology.orchestra.run.smithi114.stdout: "log_to_cluster": { 2024-04-23T10:41:09.171 INFO:teuthology.orchestra.run.smithi114.stdout: "name": "log_to_cluster", 2024-04-23T10:41:09.171 INFO:teuthology.orchestra.run.smithi114.stdout: "type": "bool", 2024-04-23T10:41:09.171 INFO:teuthology.orchestra.run.smithi114.stdout: "level": "advanced", 2024-04-23T10:41:09.172 INFO:teuthology.orchestra.run.smithi114.stdout: "flags": 1, 2024-04-23T10:41:09.172 INFO:teuthology.orchestra.run.smithi114.stdout: "default_value": "False", 2024-04-23T10:41:09.172 INFO:teuthology.orchestra.run.smithi114.stdout: "min": "", 2024-04-23T10:41:09.172 INFO:teuthology.orchestra.run.smithi114.stdout: "max": "", 2024-04-23T10:41:09.172 INFO:teuthology.orchestra.run.smithi114.stdout: "enum_allowed": [], 2024-04-23T10:41:09.172 INFO:teuthology.orchestra.run.smithi114.stdout: "desc": "", 2024-04-23T10:41:09.172 INFO:teuthology.orchestra.run.smithi114.stdout: "long_desc": "", 2024-04-23T10:41:09.172 INFO:teuthology.orchestra.run.smithi114.stdout: "tags": [], 2024-04-23T10:41:09.172 INFO:teuthology.orchestra.run.smithi114.stdout: "see_also": [] 2024-04-23T10:41:09.173 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-23T10:41:09.173 INFO:teuthology.orchestra.run.smithi114.stdout: "log_to_cluster_level": { 2024-04-23T10:41:09.173 INFO:teuthology.orchestra.run.smithi114.stdout: "name": "log_to_cluster_level", 2024-04-23T10:41:09.173 INFO:teuthology.orchestra.run.smithi114.stdout: "type": "str", 2024-04-23T10:41:09.173 INFO:teuthology.orchestra.run.smithi114.stdout: "level": "advanced", 2024-04-23T10:41:09.173 INFO:teuthology.orchestra.run.smithi114.stdout: "flags": 1, 2024-04-23T10:41:09.173 INFO:teuthology.orchestra.run.smithi114.stdout: "default_value": "info", 2024-04-23T10:41:09.173 INFO:teuthology.orchestra.run.smithi114.stdout: "min": "", 2024-04-23T10:41:09.173 INFO:teuthology.orchestra.run.smithi114.stdout: "max": "", 2024-04-23T10:41:09.174 INFO:teuthology.orchestra.run.smithi114.stdout: "enum_allowed": [ 2024-04-23T10:41:09.174 INFO:teuthology.orchestra.run.smithi114.stdout: "", 2024-04-23T10:41:09.174 INFO:teuthology.orchestra.run.smithi114.stdout: "critical", 2024-04-23T10:41:09.174 INFO:teuthology.orchestra.run.smithi114.stdout: "debug", 2024-04-23T10:41:09.174 INFO:teuthology.orchestra.run.smithi114.stdout: "error", 2024-04-23T10:41:09.174 INFO:teuthology.orchestra.run.smithi114.stdout: "info", 2024-04-23T10:41:09.174 INFO:teuthology.orchestra.run.smithi114.stdout: "warning" 2024-04-23T10:41:09.174 INFO:teuthology.orchestra.run.smithi114.stdout: ], 2024-04-23T10:41:09.175 INFO:teuthology.orchestra.run.smithi114.stdout: "desc": "", 2024-04-23T10:41:09.175 INFO:teuthology.orchestra.run.smithi114.stdout: "long_desc": "", 2024-04-23T10:41:09.175 INFO:teuthology.orchestra.run.smithi114.stdout: "tags": [], 2024-04-23T10:41:09.175 INFO:teuthology.orchestra.run.smithi114.stdout: "see_also": [] 2024-04-23T10:41:09.175 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-23T10:41:09.175 INFO:teuthology.orchestra.run.smithi114.stdout: "log_to_file": { 2024-04-23T10:41:09.175 INFO:teuthology.orchestra.run.smithi114.stdout: "name": "log_to_file", 2024-04-23T10:41:09.175 INFO:teuthology.orchestra.run.smithi114.stdout: "type": "bool", 2024-04-23T10:41:09.175 INFO:teuthology.orchestra.run.smithi114.stdout: "level": "advanced", 2024-04-23T10:41:09.176 INFO:teuthology.orchestra.run.smithi114.stdout: "flags": 1, 2024-04-23T10:41:09.176 INFO:teuthology.orchestra.run.smithi114.stdout: "default_value": "False", 2024-04-23T10:41:09.176 INFO:teuthology.orchestra.run.smithi114.stdout: "min": "", 2024-04-23T10:41:09.176 INFO:teuthology.orchestra.run.smithi114.stdout: "max": "", 2024-04-23T10:41:09.176 INFO:teuthology.orchestra.run.smithi114.stdout: "enum_allowed": [], 2024-04-23T10:41:09.176 INFO:teuthology.orchestra.run.smithi114.stdout: "desc": "", 2024-04-23T10:41:09.176 INFO:teuthology.orchestra.run.smithi114.stdout: "long_desc": "", 2024-04-23T10:41:09.176 INFO:teuthology.orchestra.run.smithi114.stdout: "tags": [], 2024-04-23T10:41:09.176 INFO:teuthology.orchestra.run.smithi114.stdout: "see_also": [] 2024-04-23T10:41:09.177 INFO:teuthology.orchestra.run.smithi114.stdout: } 2024-04-23T10:41:09.177 INFO:teuthology.orchestra.run.smithi114.stdout: } 2024-04-23T10:41:09.177 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-23T10:41:09.177 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-04-23T10:41:09.177 INFO:teuthology.orchestra.run.smithi114.stdout: "name": "pg_autoscaler", 2024-04-23T10:41:09.177 INFO:teuthology.orchestra.run.smithi114.stdout: "can_run": true, 2024-04-23T10:41:09.177 INFO:teuthology.orchestra.run.smithi114.stdout: "error_string": "", 2024-04-23T10:41:09.177 INFO:teuthology.orchestra.run.smithi114.stdout: "module_options": { 2024-04-23T10:41:09.177 INFO:teuthology.orchestra.run.smithi114.stdout: "log_level": { 2024-04-23T10:41:09.178 INFO:teuthology.orchestra.run.smithi114.stdout: "name": "log_level", 2024-04-23T10:41:09.178 INFO:teuthology.orchestra.run.smithi114.stdout: "type": "str", 2024-04-23T10:41:09.178 INFO:teuthology.orchestra.run.smithi114.stdout: "level": "advanced", 2024-04-23T10:41:09.178 INFO:teuthology.orchestra.run.smithi114.stdout: "flags": 1, 2024-04-23T10:41:09.178 INFO:teuthology.orchestra.run.smithi114.stdout: "default_value": "", 2024-04-23T10:41:09.178 INFO:teuthology.orchestra.run.smithi114.stdout: "min": "", 2024-04-23T10:41:09.178 INFO:teuthology.orchestra.run.smithi114.stdout: "max": "", 2024-04-23T10:41:09.178 INFO:teuthology.orchestra.run.smithi114.stdout: "enum_allowed": [ 2024-04-23T10:41:09.179 INFO:teuthology.orchestra.run.smithi114.stdout: "", 2024-04-23T10:41:09.179 INFO:teuthology.orchestra.run.smithi114.stdout: "critical", 2024-04-23T10:41:09.179 INFO:teuthology.orchestra.run.smithi114.stdout: "debug", 2024-04-23T10:41:09.179 INFO:teuthology.orchestra.run.smithi114.stdout: "error", 2024-04-23T10:41:09.179 INFO:teuthology.orchestra.run.smithi114.stdout: "info", 2024-04-23T10:41:09.179 INFO:teuthology.orchestra.run.smithi114.stdout: "warning" 2024-04-23T10:41:09.179 INFO:teuthology.orchestra.run.smithi114.stdout: ], 2024-04-23T10:41:09.179 INFO:teuthology.orchestra.run.smithi114.stdout: "desc": "", 2024-04-23T10:41:09.179 INFO:teuthology.orchestra.run.smithi114.stdout: "long_desc": "", 2024-04-23T10:41:09.180 INFO:teuthology.orchestra.run.smithi114.stdout: "tags": [], 2024-04-23T10:41:09.180 INFO:teuthology.orchestra.run.smithi114.stdout: "see_also": [] 2024-04-23T10:41:09.180 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-23T10:41:09.180 INFO:teuthology.orchestra.run.smithi114.stdout: "log_to_cluster": { 2024-04-23T10:41:09.180 INFO:teuthology.orchestra.run.smithi114.stdout: "name": "log_to_cluster", 2024-04-23T10:41:09.180 INFO:teuthology.orchestra.run.smithi114.stdout: "type": "bool", 2024-04-23T10:41:09.180 INFO:teuthology.orchestra.run.smithi114.stdout: "level": "advanced", 2024-04-23T10:41:09.180 INFO:teuthology.orchestra.run.smithi114.stdout: "flags": 1, 2024-04-23T10:41:09.180 INFO:teuthology.orchestra.run.smithi114.stdout: "default_value": "False", 2024-04-23T10:41:09.180 INFO:teuthology.orchestra.run.smithi114.stdout: "min": "", 2024-04-23T10:41:09.181 INFO:teuthology.orchestra.run.smithi114.stdout: "max": "", 2024-04-23T10:41:09.181 INFO:teuthology.orchestra.run.smithi114.stdout: "enum_allowed": [], 2024-04-23T10:41:09.181 INFO:teuthology.orchestra.run.smithi114.stdout: "desc": "", 2024-04-23T10:41:09.181 INFO:teuthology.orchestra.run.smithi114.stdout: "long_desc": "", 2024-04-23T10:41:09.181 INFO:teuthology.orchestra.run.smithi114.stdout: "tags": [], 2024-04-23T10:41:09.181 INFO:teuthology.orchestra.run.smithi114.stdout: "see_also": [] 2024-04-23T10:41:09.181 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-23T10:41:09.181 INFO:teuthology.orchestra.run.smithi114.stdout: "log_to_cluster_level": { 2024-04-23T10:41:09.182 INFO:teuthology.orchestra.run.smithi114.stdout: "name": "log_to_cluster_level", 2024-04-23T10:41:09.182 INFO:teuthology.orchestra.run.smithi114.stdout: "type": "str", 2024-04-23T10:41:09.182 INFO:teuthology.orchestra.run.smithi114.stdout: "level": "advanced", 2024-04-23T10:41:09.182 INFO:teuthology.orchestra.run.smithi114.stdout: "flags": 1, 2024-04-23T10:41:09.182 INFO:teuthology.orchestra.run.smithi114.stdout: "default_value": "info", 2024-04-23T10:41:09.182 INFO:teuthology.orchestra.run.smithi114.stdout: "min": "", 2024-04-23T10:41:09.182 INFO:teuthology.orchestra.run.smithi114.stdout: "max": "", 2024-04-23T10:41:09.182 INFO:teuthology.orchestra.run.smithi114.stdout: "enum_allowed": [ 2024-04-23T10:41:09.182 INFO:teuthology.orchestra.run.smithi114.stdout: "", 2024-04-23T10:41:09.183 INFO:teuthology.orchestra.run.smithi114.stdout: "critical", 2024-04-23T10:41:09.183 INFO:teuthology.orchestra.run.smithi114.stdout: "debug", 2024-04-23T10:41:09.183 INFO:teuthology.orchestra.run.smithi114.stdout: "error", 2024-04-23T10:41:09.183 INFO:teuthology.orchestra.run.smithi114.stdout: "info", 2024-04-23T10:41:09.183 INFO:teuthology.orchestra.run.smithi114.stdout: "warning" 2024-04-23T10:41:09.183 INFO:teuthology.orchestra.run.smithi114.stdout: ], 2024-04-23T10:41:09.183 INFO:teuthology.orchestra.run.smithi114.stdout: "desc": "", 2024-04-23T10:41:09.183 INFO:teuthology.orchestra.run.smithi114.stdout: "long_desc": "", 2024-04-23T10:41:09.183 INFO:teuthology.orchestra.run.smithi114.stdout: "tags": [], 2024-04-23T10:41:09.184 INFO:teuthology.orchestra.run.smithi114.stdout: "see_also": [] 2024-04-23T10:41:09.184 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-23T10:41:09.184 INFO:teuthology.orchestra.run.smithi114.stdout: "log_to_file": { 2024-04-23T10:41:09.184 INFO:teuthology.orchestra.run.smithi114.stdout: "name": "log_to_file", 2024-04-23T10:41:09.184 INFO:teuthology.orchestra.run.smithi114.stdout: "type": "bool", 2024-04-23T10:41:09.184 INFO:teuthology.orchestra.run.smithi114.stdout: "level": "advanced", 2024-04-23T10:41:09.184 INFO:teuthology.orchestra.run.smithi114.stdout: "flags": 1, 2024-04-23T10:41:09.184 INFO:teuthology.orchestra.run.smithi114.stdout: "default_value": "False", 2024-04-23T10:41:09.185 INFO:teuthology.orchestra.run.smithi114.stdout: "min": "", 2024-04-23T10:41:09.185 INFO:teuthology.orchestra.run.smithi114.stdout: "max": "", 2024-04-23T10:41:09.185 INFO:teuthology.orchestra.run.smithi114.stdout: "enum_allowed": [], 2024-04-23T10:41:09.185 INFO:teuthology.orchestra.run.smithi114.stdout: "desc": "", 2024-04-23T10:41:09.185 INFO:teuthology.orchestra.run.smithi114.stdout: "long_desc": "", 2024-04-23T10:41:09.185 INFO:teuthology.orchestra.run.smithi114.stdout: "tags": [], 2024-04-23T10:41:09.185 INFO:teuthology.orchestra.run.smithi114.stdout: "see_also": [] 2024-04-23T10:41:09.185 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-23T10:41:09.185 INFO:teuthology.orchestra.run.smithi114.stdout: "sleep_interval": { 2024-04-23T10:41:09.186 INFO:teuthology.orchestra.run.smithi114.stdout: "name": "sleep_interval", 2024-04-23T10:41:09.186 INFO:teuthology.orchestra.run.smithi114.stdout: "type": "secs", 2024-04-23T10:41:09.186 INFO:teuthology.orchestra.run.smithi114.stdout: "level": "advanced", 2024-04-23T10:41:09.186 INFO:teuthology.orchestra.run.smithi114.stdout: "flags": 0, 2024-04-23T10:41:09.186 INFO:teuthology.orchestra.run.smithi114.stdout: "default_value": "60", 2024-04-23T10:41:09.186 INFO:teuthology.orchestra.run.smithi114.stdout: "min": "", 2024-04-23T10:41:09.186 INFO:teuthology.orchestra.run.smithi114.stdout: "max": "", 2024-04-23T10:41:09.186 INFO:teuthology.orchestra.run.smithi114.stdout: "enum_allowed": [], 2024-04-23T10:41:09.186 INFO:teuthology.orchestra.run.smithi114.stdout: "desc": "", 2024-04-23T10:41:09.187 INFO:teuthology.orchestra.run.smithi114.stdout: "long_desc": "", 2024-04-23T10:41:09.187 INFO:teuthology.orchestra.run.smithi114.stdout: "tags": [], 2024-04-23T10:41:09.187 INFO:teuthology.orchestra.run.smithi114.stdout: "see_also": [] 2024-04-23T10:41:09.187 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-23T10:41:09.187 INFO:teuthology.orchestra.run.smithi114.stdout: "threshold": { 2024-04-23T10:41:09.187 INFO:teuthology.orchestra.run.smithi114.stdout: "name": "threshold", 2024-04-23T10:41:09.187 INFO:teuthology.orchestra.run.smithi114.stdout: "type": "float", 2024-04-23T10:41:09.187 INFO:teuthology.orchestra.run.smithi114.stdout: "level": "advanced", 2024-04-23T10:41:09.188 INFO:teuthology.orchestra.run.smithi114.stdout: "flags": 0, 2024-04-23T10:41:09.188 INFO:teuthology.orchestra.run.smithi114.stdout: "default_value": "3.0", 2024-04-23T10:41:09.188 INFO:teuthology.orchestra.run.smithi114.stdout: "min": "1.0", 2024-04-23T10:41:09.188 INFO:teuthology.orchestra.run.smithi114.stdout: "max": "", 2024-04-23T10:41:09.188 INFO:teuthology.orchestra.run.smithi114.stdout: "enum_allowed": [], 2024-04-23T10:41:09.188 INFO:teuthology.orchestra.run.smithi114.stdout: "desc": "scaling threshold", 2024-04-23T10:41:09.188 INFO:teuthology.orchestra.run.smithi114.stdout: "long_desc": "The factor by which the `NEW PG_NUM` must vary from the current`PG_NUM` before being accepted. Cannot be less than 1.0", 2024-04-23T10:41:09.188 INFO:teuthology.orchestra.run.smithi114.stdout: "tags": [], 2024-04-23T10:41:09.188 INFO:teuthology.orchestra.run.smithi114.stdout: "see_also": [] 2024-04-23T10:41:09.189 INFO:teuthology.orchestra.run.smithi114.stdout: } 2024-04-23T10:41:09.189 INFO:teuthology.orchestra.run.smithi114.stdout: } 2024-04-23T10:41:09.189 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-23T10:41:09.189 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-04-23T10:41:09.189 INFO:teuthology.orchestra.run.smithi114.stdout: "name": "progress", 2024-04-23T10:41:09.189 INFO:teuthology.orchestra.run.smithi114.stdout: "can_run": true, 2024-04-23T10:41:09.189 INFO:teuthology.orchestra.run.smithi114.stdout: "error_string": "", 2024-04-23T10:41:09.189 INFO:teuthology.orchestra.run.smithi114.stdout: "module_options": { 2024-04-23T10:41:09.189 INFO:teuthology.orchestra.run.smithi114.stdout: "allow_pg_recovery_event": { 2024-04-23T10:41:09.190 INFO:teuthology.orchestra.run.smithi114.stdout: "name": "allow_pg_recovery_event", 2024-04-23T10:41:09.190 INFO:teuthology.orchestra.run.smithi114.stdout: "type": "bool", 2024-04-23T10:41:09.190 INFO:teuthology.orchestra.run.smithi114.stdout: "level": "advanced", 2024-04-23T10:41:09.190 INFO:teuthology.orchestra.run.smithi114.stdout: "flags": 1, 2024-04-23T10:41:09.190 INFO:teuthology.orchestra.run.smithi114.stdout: "default_value": "False", 2024-04-23T10:41:09.190 INFO:teuthology.orchestra.run.smithi114.stdout: "min": "", 2024-04-23T10:41:09.190 INFO:teuthology.orchestra.run.smithi114.stdout: "max": "", 2024-04-23T10:41:09.190 INFO:teuthology.orchestra.run.smithi114.stdout: "enum_allowed": [], 2024-04-23T10:41:09.191 INFO:teuthology.orchestra.run.smithi114.stdout: "desc": "allow the module to show pg recovery progress", 2024-04-23T10:41:09.191 INFO:teuthology.orchestra.run.smithi114.stdout: "long_desc": "", 2024-04-23T10:41:09.191 INFO:teuthology.orchestra.run.smithi114.stdout: "tags": [], 2024-04-23T10:41:09.191 INFO:teuthology.orchestra.run.smithi114.stdout: "see_also": [] 2024-04-23T10:41:09.191 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-23T10:41:09.191 INFO:teuthology.orchestra.run.smithi114.stdout: "enabled": { 2024-04-23T10:41:09.191 INFO:teuthology.orchestra.run.smithi114.stdout: "name": "enabled", 2024-04-23T10:41:09.191 INFO:teuthology.orchestra.run.smithi114.stdout: "type": "bool", 2024-04-23T10:41:09.191 INFO:teuthology.orchestra.run.smithi114.stdout: "level": "advanced", 2024-04-23T10:41:09.192 INFO:teuthology.orchestra.run.smithi114.stdout: "flags": 0, 2024-04-23T10:41:09.192 INFO:teuthology.orchestra.run.smithi114.stdout: "default_value": "True", 2024-04-23T10:41:09.192 INFO:teuthology.orchestra.run.smithi114.stdout: "min": "", 2024-04-23T10:41:09.192 INFO:teuthology.orchestra.run.smithi114.stdout: "max": "", 2024-04-23T10:41:09.192 INFO:teuthology.orchestra.run.smithi114.stdout: "enum_allowed": [], 2024-04-23T10:41:09.192 INFO:teuthology.orchestra.run.smithi114.stdout: "desc": "", 2024-04-23T10:41:09.192 INFO:teuthology.orchestra.run.smithi114.stdout: "long_desc": "", 2024-04-23T10:41:09.192 INFO:teuthology.orchestra.run.smithi114.stdout: "tags": [], 2024-04-23T10:41:09.192 INFO:teuthology.orchestra.run.smithi114.stdout: "see_also": [] 2024-04-23T10:41:09.193 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-23T10:41:09.193 INFO:teuthology.orchestra.run.smithi114.stdout: "log_level": { 2024-04-23T10:41:09.193 INFO:teuthology.orchestra.run.smithi114.stdout: "name": "log_level", 2024-04-23T10:41:09.193 INFO:teuthology.orchestra.run.smithi114.stdout: "type": "str", 2024-04-23T10:41:09.193 INFO:teuthology.orchestra.run.smithi114.stdout: "level": "advanced", 2024-04-23T10:41:09.193 INFO:teuthology.orchestra.run.smithi114.stdout: "flags": 1, 2024-04-23T10:41:09.193 INFO:teuthology.orchestra.run.smithi114.stdout: "default_value": "", 2024-04-23T10:41:09.193 INFO:teuthology.orchestra.run.smithi114.stdout: "min": "", 2024-04-23T10:41:09.193 INFO:teuthology.orchestra.run.smithi114.stdout: "max": "", 2024-04-23T10:41:09.194 INFO:teuthology.orchestra.run.smithi114.stdout: "enum_allowed": [ 2024-04-23T10:41:09.194 INFO:teuthology.orchestra.run.smithi114.stdout: "", 2024-04-23T10:41:09.194 INFO:teuthology.orchestra.run.smithi114.stdout: "critical", 2024-04-23T10:41:09.194 INFO:teuthology.orchestra.run.smithi114.stdout: "debug", 2024-04-23T10:41:09.194 INFO:teuthology.orchestra.run.smithi114.stdout: "error", 2024-04-23T10:41:09.194 INFO:teuthology.orchestra.run.smithi114.stdout: "info", 2024-04-23T10:41:09.194 INFO:teuthology.orchestra.run.smithi114.stdout: "warning" 2024-04-23T10:41:09.194 INFO:teuthology.orchestra.run.smithi114.stdout: ], 2024-04-23T10:41:09.195 INFO:teuthology.orchestra.run.smithi114.stdout: "desc": "", 2024-04-23T10:41:09.195 INFO:teuthology.orchestra.run.smithi114.stdout: "long_desc": "", 2024-04-23T10:41:09.195 INFO:teuthology.orchestra.run.smithi114.stdout: "tags": [], 2024-04-23T10:41:09.195 INFO:teuthology.orchestra.run.smithi114.stdout: "see_also": [] 2024-04-23T10:41:09.195 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-23T10:41:09.195 INFO:teuthology.orchestra.run.smithi114.stdout: "log_to_cluster": { 2024-04-23T10:41:09.195 INFO:teuthology.orchestra.run.smithi114.stdout: "name": "log_to_cluster", 2024-04-23T10:41:09.195 INFO:teuthology.orchestra.run.smithi114.stdout: "type": "bool", 2024-04-23T10:41:09.196 INFO:teuthology.orchestra.run.smithi114.stdout: "level": "advanced", 2024-04-23T10:41:09.196 INFO:teuthology.orchestra.run.smithi114.stdout: "flags": 1, 2024-04-23T10:41:09.196 INFO:teuthology.orchestra.run.smithi114.stdout: "default_value": "False", 2024-04-23T10:41:09.196 INFO:teuthology.orchestra.run.smithi114.stdout: "min": "", 2024-04-23T10:41:09.196 INFO:teuthology.orchestra.run.smithi114.stdout: "max": "", 2024-04-23T10:41:09.196 INFO:teuthology.orchestra.run.smithi114.stdout: "enum_allowed": [], 2024-04-23T10:41:09.196 INFO:teuthology.orchestra.run.smithi114.stdout: "desc": "", 2024-04-23T10:41:09.196 INFO:teuthology.orchestra.run.smithi114.stdout: "long_desc": "", 2024-04-23T10:41:09.196 INFO:teuthology.orchestra.run.smithi114.stdout: "tags": [], 2024-04-23T10:41:09.197 INFO:teuthology.orchestra.run.smithi114.stdout: "see_also": [] 2024-04-23T10:41:09.197 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-23T10:41:09.197 INFO:teuthology.orchestra.run.smithi114.stdout: "log_to_cluster_level": { 2024-04-23T10:41:09.197 INFO:teuthology.orchestra.run.smithi114.stdout: "name": "log_to_cluster_level", 2024-04-23T10:41:09.197 INFO:teuthology.orchestra.run.smithi114.stdout: "type": "str", 2024-04-23T10:41:09.197 INFO:teuthology.orchestra.run.smithi114.stdout: "level": "advanced", 2024-04-23T10:41:09.197 INFO:teuthology.orchestra.run.smithi114.stdout: "flags": 1, 2024-04-23T10:41:09.197 INFO:teuthology.orchestra.run.smithi114.stdout: "default_value": "info", 2024-04-23T10:41:09.197 INFO:teuthology.orchestra.run.smithi114.stdout: "min": "", 2024-04-23T10:41:09.198 INFO:teuthology.orchestra.run.smithi114.stdout: "max": "", 2024-04-23T10:41:09.198 INFO:teuthology.orchestra.run.smithi114.stdout: "enum_allowed": [ 2024-04-23T10:41:09.198 INFO:teuthology.orchestra.run.smithi114.stdout: "", 2024-04-23T10:41:09.198 INFO:teuthology.orchestra.run.smithi114.stdout: "critical", 2024-04-23T10:41:09.198 INFO:teuthology.orchestra.run.smithi114.stdout: "debug", 2024-04-23T10:41:09.198 INFO:teuthology.orchestra.run.smithi114.stdout: "error", 2024-04-23T10:41:09.201 INFO:teuthology.orchestra.run.smithi114.stdout: "info", 2024-04-23T10:41:09.202 INFO:teuthology.orchestra.run.smithi114.stdout: "warning" 2024-04-23T10:41:09.202 INFO:teuthology.orchestra.run.smithi114.stdout: ], 2024-04-23T10:41:09.202 INFO:teuthology.orchestra.run.smithi114.stdout: "desc": "", 2024-04-23T10:41:09.202 INFO:teuthology.orchestra.run.smithi114.stdout: "long_desc": "", 2024-04-23T10:41:09.202 INFO:teuthology.orchestra.run.smithi114.stdout: "tags": [], 2024-04-23T10:41:09.202 INFO:teuthology.orchestra.run.smithi114.stdout: "see_also": [] 2024-04-23T10:41:09.202 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-23T10:41:09.202 INFO:teuthology.orchestra.run.smithi114.stdout: "log_to_file": { 2024-04-23T10:41:09.202 INFO:teuthology.orchestra.run.smithi114.stdout: "name": "log_to_file", 2024-04-23T10:41:09.203 INFO:teuthology.orchestra.run.smithi114.stdout: "type": "bool", 2024-04-23T10:41:09.203 INFO:teuthology.orchestra.run.smithi114.stdout: "level": "advanced", 2024-04-23T10:41:09.203 INFO:teuthology.orchestra.run.smithi114.stdout: "flags": 1, 2024-04-23T10:41:09.203 INFO:teuthology.orchestra.run.smithi114.stdout: "default_value": "False", 2024-04-23T10:41:09.203 INFO:teuthology.orchestra.run.smithi114.stdout: "min": "", 2024-04-23T10:41:09.203 INFO:teuthology.orchestra.run.smithi114.stdout: "max": "", 2024-04-23T10:41:09.203 INFO:teuthology.orchestra.run.smithi114.stdout: "enum_allowed": [], 2024-04-23T10:41:09.203 INFO:teuthology.orchestra.run.smithi114.stdout: "desc": "", 2024-04-23T10:41:09.203 INFO:teuthology.orchestra.run.smithi114.stdout: "long_desc": "", 2024-04-23T10:41:09.204 INFO:teuthology.orchestra.run.smithi114.stdout: "tags": [], 2024-04-23T10:41:09.204 INFO:teuthology.orchestra.run.smithi114.stdout: "see_also": [] 2024-04-23T10:41:09.204 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-23T10:41:09.204 INFO:teuthology.orchestra.run.smithi114.stdout: "max_completed_events": { 2024-04-23T10:41:09.204 INFO:teuthology.orchestra.run.smithi114.stdout: "name": "max_completed_events", 2024-04-23T10:41:09.204 INFO:teuthology.orchestra.run.smithi114.stdout: "type": "int", 2024-04-23T10:41:09.204 INFO:teuthology.orchestra.run.smithi114.stdout: "level": "advanced", 2024-04-23T10:41:09.204 INFO:teuthology.orchestra.run.smithi114.stdout: "flags": 1, 2024-04-23T10:41:09.205 INFO:teuthology.orchestra.run.smithi114.stdout: "default_value": "50", 2024-04-23T10:41:09.205 INFO:teuthology.orchestra.run.smithi114.stdout: "min": "", 2024-04-23T10:41:09.205 INFO:teuthology.orchestra.run.smithi114.stdout: "max": "", 2024-04-23T10:41:09.205 INFO:teuthology.orchestra.run.smithi114.stdout: "enum_allowed": [], 2024-04-23T10:41:09.205 INFO:teuthology.orchestra.run.smithi114.stdout: "desc": "number of past completed events to remember", 2024-04-23T10:41:09.205 INFO:teuthology.orchestra.run.smithi114.stdout: "long_desc": "", 2024-04-23T10:41:09.205 INFO:teuthology.orchestra.run.smithi114.stdout: "tags": [], 2024-04-23T10:41:09.205 INFO:teuthology.orchestra.run.smithi114.stdout: "see_also": [] 2024-04-23T10:41:09.205 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-23T10:41:09.206 INFO:teuthology.orchestra.run.smithi114.stdout: "sleep_interval": { 2024-04-23T10:41:09.206 INFO:teuthology.orchestra.run.smithi114.stdout: "name": "sleep_interval", 2024-04-23T10:41:09.206 INFO:teuthology.orchestra.run.smithi114.stdout: "type": "secs", 2024-04-23T10:41:09.206 INFO:teuthology.orchestra.run.smithi114.stdout: "level": "advanced", 2024-04-23T10:41:09.206 INFO:teuthology.orchestra.run.smithi114.stdout: "flags": 1, 2024-04-23T10:41:09.206 INFO:teuthology.orchestra.run.smithi114.stdout: "default_value": "5", 2024-04-23T10:41:09.206 INFO:teuthology.orchestra.run.smithi114.stdout: "min": "", 2024-04-23T10:41:09.206 INFO:teuthology.orchestra.run.smithi114.stdout: "max": "", 2024-04-23T10:41:09.206 INFO:teuthology.orchestra.run.smithi114.stdout: "enum_allowed": [], 2024-04-23T10:41:09.207 INFO:teuthology.orchestra.run.smithi114.stdout: "desc": "how long the module is going to sleep", 2024-04-23T10:41:09.207 INFO:teuthology.orchestra.run.smithi114.stdout: "long_desc": "", 2024-04-23T10:41:09.207 INFO:teuthology.orchestra.run.smithi114.stdout: "tags": [], 2024-04-23T10:41:09.207 INFO:teuthology.orchestra.run.smithi114.stdout: "see_also": [] 2024-04-23T10:41:09.207 INFO:teuthology.orchestra.run.smithi114.stdout: } 2024-04-23T10:41:09.207 INFO:teuthology.orchestra.run.smithi114.stdout: } 2024-04-23T10:41:09.207 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-23T10:41:09.207 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-04-23T10:41:09.208 INFO:teuthology.orchestra.run.smithi114.stdout: "name": "prometheus", 2024-04-23T10:41:09.208 INFO:teuthology.orchestra.run.smithi114.stdout: "can_run": true, 2024-04-23T10:41:09.208 INFO:teuthology.orchestra.run.smithi114.stdout: "error_string": "", 2024-04-23T10:41:09.208 INFO:teuthology.orchestra.run.smithi114.stdout: "module_options": { 2024-04-23T10:41:09.208 INFO:teuthology.orchestra.run.smithi114.stdout: "cache": { 2024-04-23T10:41:09.208 INFO:teuthology.orchestra.run.smithi114.stdout: "name": "cache", 2024-04-23T10:41:09.208 INFO:teuthology.orchestra.run.smithi114.stdout: "type": "bool", 2024-04-23T10:41:09.208 INFO:teuthology.orchestra.run.smithi114.stdout: "level": "advanced", 2024-04-23T10:41:09.208 INFO:teuthology.orchestra.run.smithi114.stdout: "flags": 0, 2024-04-23T10:41:09.209 INFO:teuthology.orchestra.run.smithi114.stdout: "default_value": "True", 2024-04-23T10:41:09.209 INFO:teuthology.orchestra.run.smithi114.stdout: "min": "", 2024-04-23T10:41:09.209 INFO:teuthology.orchestra.run.smithi114.stdout: "max": "", 2024-04-23T10:41:09.209 INFO:teuthology.orchestra.run.smithi114.stdout: "enum_allowed": [], 2024-04-23T10:41:09.209 INFO:teuthology.orchestra.run.smithi114.stdout: "desc": "", 2024-04-23T10:41:09.209 INFO:teuthology.orchestra.run.smithi114.stdout: "long_desc": "", 2024-04-23T10:41:09.209 INFO:teuthology.orchestra.run.smithi114.stdout: "tags": [], 2024-04-23T10:41:09.209 INFO:teuthology.orchestra.run.smithi114.stdout: "see_also": [] 2024-04-23T10:41:09.209 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-23T10:41:09.210 INFO:teuthology.orchestra.run.smithi114.stdout: "exclude_perf_counters": { 2024-04-23T10:41:09.210 INFO:teuthology.orchestra.run.smithi114.stdout: "name": "exclude_perf_counters", 2024-04-23T10:41:09.210 INFO:teuthology.orchestra.run.smithi114.stdout: "type": "bool", 2024-04-23T10:41:09.210 INFO:teuthology.orchestra.run.smithi114.stdout: "level": "advanced", 2024-04-23T10:41:09.210 INFO:teuthology.orchestra.run.smithi114.stdout: "flags": 1, 2024-04-23T10:41:09.210 INFO:teuthology.orchestra.run.smithi114.stdout: "default_value": "True", 2024-04-23T10:41:09.210 INFO:teuthology.orchestra.run.smithi114.stdout: "min": "", 2024-04-23T10:41:09.210 INFO:teuthology.orchestra.run.smithi114.stdout: "max": "", 2024-04-23T10:41:09.211 INFO:teuthology.orchestra.run.smithi114.stdout: "enum_allowed": [], 2024-04-23T10:41:09.211 INFO:teuthology.orchestra.run.smithi114.stdout: "desc": "Do not include perf-counters in the metrics output", 2024-04-23T10:41:09.211 INFO:teuthology.orchestra.run.smithi114.stdout: "long_desc": "Gathering perf-counters from a single Prometheus exporter can degrade ceph-mgr performance, especially in large clusters. Instead, Ceph-exporter daemons are now used by default for perf-counter gathering. This should only be disabled when no ceph-exporters are deployed.", 2024-04-23T10:41:09.211 INFO:teuthology.orchestra.run.smithi114.stdout: "tags": [], 2024-04-23T10:41:09.211 INFO:teuthology.orchestra.run.smithi114.stdout: "see_also": [] 2024-04-23T10:41:09.211 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-23T10:41:09.211 INFO:teuthology.orchestra.run.smithi114.stdout: "log_level": { 2024-04-23T10:41:09.211 INFO:teuthology.orchestra.run.smithi114.stdout: "name": "log_level", 2024-04-23T10:41:09.211 INFO:teuthology.orchestra.run.smithi114.stdout: "type": "str", 2024-04-23T10:41:09.212 INFO:teuthology.orchestra.run.smithi114.stdout: "level": "advanced", 2024-04-23T10:41:09.212 INFO:teuthology.orchestra.run.smithi114.stdout: "flags": 1, 2024-04-23T10:41:09.212 INFO:teuthology.orchestra.run.smithi114.stdout: "default_value": "", 2024-04-23T10:41:09.212 INFO:teuthology.orchestra.run.smithi114.stdout: "min": "", 2024-04-23T10:41:09.212 INFO:teuthology.orchestra.run.smithi114.stdout: "max": "", 2024-04-23T10:41:09.212 INFO:teuthology.orchestra.run.smithi114.stdout: "enum_allowed": [ 2024-04-23T10:41:09.212 INFO:teuthology.orchestra.run.smithi114.stdout: "", 2024-04-23T10:41:09.212 INFO:teuthology.orchestra.run.smithi114.stdout: "critical", 2024-04-23T10:41:09.213 INFO:teuthology.orchestra.run.smithi114.stdout: "debug", 2024-04-23T10:41:09.213 INFO:teuthology.orchestra.run.smithi114.stdout: "error", 2024-04-23T10:41:09.213 INFO:teuthology.orchestra.run.smithi114.stdout: "info", 2024-04-23T10:41:09.213 INFO:teuthology.orchestra.run.smithi114.stdout: "warning" 2024-04-23T10:41:09.213 INFO:teuthology.orchestra.run.smithi114.stdout: ], 2024-04-23T10:41:09.213 INFO:teuthology.orchestra.run.smithi114.stdout: "desc": "", 2024-04-23T10:41:09.213 INFO:teuthology.orchestra.run.smithi114.stdout: "long_desc": "", 2024-04-23T10:41:09.213 INFO:teuthology.orchestra.run.smithi114.stdout: "tags": [], 2024-04-23T10:41:09.213 INFO:teuthology.orchestra.run.smithi114.stdout: "see_also": [] 2024-04-23T10:41:09.214 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-23T10:41:09.214 INFO:teuthology.orchestra.run.smithi114.stdout: "log_to_cluster": { 2024-04-23T10:41:09.214 INFO:teuthology.orchestra.run.smithi114.stdout: "name": "log_to_cluster", 2024-04-23T10:41:09.214 INFO:teuthology.orchestra.run.smithi114.stdout: "type": "bool", 2024-04-23T10:41:09.214 INFO:teuthology.orchestra.run.smithi114.stdout: "level": "advanced", 2024-04-23T10:41:09.214 INFO:teuthology.orchestra.run.smithi114.stdout: "flags": 1, 2024-04-23T10:41:09.214 INFO:teuthology.orchestra.run.smithi114.stdout: "default_value": "False", 2024-04-23T10:41:09.214 INFO:teuthology.orchestra.run.smithi114.stdout: "min": "", 2024-04-23T10:41:09.215 INFO:teuthology.orchestra.run.smithi114.stdout: "max": "", 2024-04-23T10:41:09.215 INFO:teuthology.orchestra.run.smithi114.stdout: "enum_allowed": [], 2024-04-23T10:41:09.215 INFO:teuthology.orchestra.run.smithi114.stdout: "desc": "", 2024-04-23T10:41:09.215 INFO:teuthology.orchestra.run.smithi114.stdout: "long_desc": "", 2024-04-23T10:41:09.215 INFO:teuthology.orchestra.run.smithi114.stdout: "tags": [], 2024-04-23T10:41:09.215 INFO:teuthology.orchestra.run.smithi114.stdout: "see_also": [] 2024-04-23T10:41:09.215 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-23T10:41:09.215 INFO:teuthology.orchestra.run.smithi114.stdout: "log_to_cluster_level": { 2024-04-23T10:41:09.215 INFO:teuthology.orchestra.run.smithi114.stdout: "name": "log_to_cluster_level", 2024-04-23T10:41:09.216 INFO:teuthology.orchestra.run.smithi114.stdout: "type": "str", 2024-04-23T10:41:09.216 INFO:teuthology.orchestra.run.smithi114.stdout: "level": "advanced", 2024-04-23T10:41:09.216 INFO:teuthology.orchestra.run.smithi114.stdout: "flags": 1, 2024-04-23T10:41:09.216 INFO:teuthology.orchestra.run.smithi114.stdout: "default_value": "info", 2024-04-23T10:41:09.216 INFO:teuthology.orchestra.run.smithi114.stdout: "min": "", 2024-04-23T10:41:09.216 INFO:teuthology.orchestra.run.smithi114.stdout: "max": "", 2024-04-23T10:41:09.216 INFO:teuthology.orchestra.run.smithi114.stdout: "enum_allowed": [ 2024-04-23T10:41:09.216 INFO:teuthology.orchestra.run.smithi114.stdout: "", 2024-04-23T10:41:09.216 INFO:teuthology.orchestra.run.smithi114.stdout: "critical", 2024-04-23T10:41:09.217 INFO:teuthology.orchestra.run.smithi114.stdout: "debug", 2024-04-23T10:41:09.217 INFO:teuthology.orchestra.run.smithi114.stdout: "error", 2024-04-23T10:41:09.217 INFO:teuthology.orchestra.run.smithi114.stdout: "info", 2024-04-23T10:41:09.217 INFO:teuthology.orchestra.run.smithi114.stdout: "warning" 2024-04-23T10:41:09.217 INFO:teuthology.orchestra.run.smithi114.stdout: ], 2024-04-23T10:41:09.217 INFO:teuthology.orchestra.run.smithi114.stdout: "desc": "", 2024-04-23T10:41:09.217 INFO:teuthology.orchestra.run.smithi114.stdout: "long_desc": "", 2024-04-23T10:41:09.217 INFO:teuthology.orchestra.run.smithi114.stdout: "tags": [], 2024-04-23T10:41:09.217 INFO:teuthology.orchestra.run.smithi114.stdout: "see_also": [] 2024-04-23T10:41:09.218 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-23T10:41:09.218 INFO:teuthology.orchestra.run.smithi114.stdout: "log_to_file": { 2024-04-23T10:41:09.218 INFO:teuthology.orchestra.run.smithi114.stdout: "name": "log_to_file", 2024-04-23T10:41:09.218 INFO:teuthology.orchestra.run.smithi114.stdout: "type": "bool", 2024-04-23T10:41:09.218 INFO:teuthology.orchestra.run.smithi114.stdout: "level": "advanced", 2024-04-23T10:41:09.218 INFO:teuthology.orchestra.run.smithi114.stdout: "flags": 1, 2024-04-23T10:41:09.218 INFO:teuthology.orchestra.run.smithi114.stdout: "default_value": "False", 2024-04-23T10:41:09.218 INFO:teuthology.orchestra.run.smithi114.stdout: "min": "", 2024-04-23T10:41:09.219 INFO:teuthology.orchestra.run.smithi114.stdout: "max": "", 2024-04-23T10:41:09.219 INFO:teuthology.orchestra.run.smithi114.stdout: "enum_allowed": [], 2024-04-23T10:41:09.219 INFO:teuthology.orchestra.run.smithi114.stdout: "desc": "", 2024-04-23T10:41:09.219 INFO:teuthology.orchestra.run.smithi114.stdout: "long_desc": "", 2024-04-23T10:41:09.219 INFO:teuthology.orchestra.run.smithi114.stdout: "tags": [], 2024-04-23T10:41:09.219 INFO:teuthology.orchestra.run.smithi114.stdout: "see_also": [] 2024-04-23T10:41:09.219 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-23T10:41:09.219 INFO:teuthology.orchestra.run.smithi114.stdout: "rbd_stats_pools": { 2024-04-23T10:41:09.219 INFO:teuthology.orchestra.run.smithi114.stdout: "name": "rbd_stats_pools", 2024-04-23T10:41:09.220 INFO:teuthology.orchestra.run.smithi114.stdout: "type": "str", 2024-04-23T10:41:09.220 INFO:teuthology.orchestra.run.smithi114.stdout: "level": "advanced", 2024-04-23T10:41:09.220 INFO:teuthology.orchestra.run.smithi114.stdout: "flags": 0, 2024-04-23T10:41:09.220 INFO:teuthology.orchestra.run.smithi114.stdout: "default_value": "", 2024-04-23T10:41:09.220 INFO:teuthology.orchestra.run.smithi114.stdout: "min": "", 2024-04-23T10:41:09.220 INFO:teuthology.orchestra.run.smithi114.stdout: "max": "", 2024-04-23T10:41:09.220 INFO:teuthology.orchestra.run.smithi114.stdout: "enum_allowed": [], 2024-04-23T10:41:09.220 INFO:teuthology.orchestra.run.smithi114.stdout: "desc": "", 2024-04-23T10:41:09.220 INFO:teuthology.orchestra.run.smithi114.stdout: "long_desc": "", 2024-04-23T10:41:09.221 INFO:teuthology.orchestra.run.smithi114.stdout: "tags": [], 2024-04-23T10:41:09.221 INFO:teuthology.orchestra.run.smithi114.stdout: "see_also": [] 2024-04-23T10:41:09.221 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-23T10:41:09.221 INFO:teuthology.orchestra.run.smithi114.stdout: "rbd_stats_pools_refresh_interval": { 2024-04-23T10:41:09.221 INFO:teuthology.orchestra.run.smithi114.stdout: "name": "rbd_stats_pools_refresh_interval", 2024-04-23T10:41:09.221 INFO:teuthology.orchestra.run.smithi114.stdout: "type": "int", 2024-04-23T10:41:09.221 INFO:teuthology.orchestra.run.smithi114.stdout: "level": "advanced", 2024-04-23T10:41:09.221 INFO:teuthology.orchestra.run.smithi114.stdout: "flags": 0, 2024-04-23T10:41:09.222 INFO:teuthology.orchestra.run.smithi114.stdout: "default_value": "300", 2024-04-23T10:41:09.222 INFO:teuthology.orchestra.run.smithi114.stdout: "min": "", 2024-04-23T10:41:09.222 INFO:teuthology.orchestra.run.smithi114.stdout: "max": "", 2024-04-23T10:41:09.222 INFO:teuthology.orchestra.run.smithi114.stdout: "enum_allowed": [], 2024-04-23T10:41:09.222 INFO:teuthology.orchestra.run.smithi114.stdout: "desc": "", 2024-04-23T10:41:09.222 INFO:teuthology.orchestra.run.smithi114.stdout: "long_desc": "", 2024-04-23T10:41:09.222 INFO:teuthology.orchestra.run.smithi114.stdout: "tags": [], 2024-04-23T10:41:09.222 INFO:teuthology.orchestra.run.smithi114.stdout: "see_also": [] 2024-04-23T10:41:09.222 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-23T10:41:09.223 INFO:teuthology.orchestra.run.smithi114.stdout: "scrape_interval": { 2024-04-23T10:41:09.223 INFO:teuthology.orchestra.run.smithi114.stdout: "name": "scrape_interval", 2024-04-23T10:41:09.223 INFO:teuthology.orchestra.run.smithi114.stdout: "type": "float", 2024-04-23T10:41:09.223 INFO:teuthology.orchestra.run.smithi114.stdout: "level": "advanced", 2024-04-23T10:41:09.223 INFO:teuthology.orchestra.run.smithi114.stdout: "flags": 0, 2024-04-23T10:41:09.223 INFO:teuthology.orchestra.run.smithi114.stdout: "default_value": "15.0", 2024-04-23T10:41:09.223 INFO:teuthology.orchestra.run.smithi114.stdout: "min": "", 2024-04-23T10:41:09.223 INFO:teuthology.orchestra.run.smithi114.stdout: "max": "", 2024-04-23T10:41:09.224 INFO:teuthology.orchestra.run.smithi114.stdout: "enum_allowed": [], 2024-04-23T10:41:09.224 INFO:teuthology.orchestra.run.smithi114.stdout: "desc": "", 2024-04-23T10:41:09.224 INFO:teuthology.orchestra.run.smithi114.stdout: "long_desc": "", 2024-04-23T10:41:09.224 INFO:teuthology.orchestra.run.smithi114.stdout: "tags": [], 2024-04-23T10:41:09.224 INFO:teuthology.orchestra.run.smithi114.stdout: "see_also": [] 2024-04-23T10:41:09.224 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-23T10:41:09.224 INFO:teuthology.orchestra.run.smithi114.stdout: "server_addr": { 2024-04-23T10:41:09.224 INFO:teuthology.orchestra.run.smithi114.stdout: "name": "server_addr", 2024-04-23T10:41:09.224 INFO:teuthology.orchestra.run.smithi114.stdout: "type": "str", 2024-04-23T10:41:09.225 INFO:teuthology.orchestra.run.smithi114.stdout: "level": "advanced", 2024-04-23T10:41:09.225 INFO:teuthology.orchestra.run.smithi114.stdout: "flags": 0, 2024-04-23T10:41:09.225 INFO:teuthology.orchestra.run.smithi114.stdout: "default_value": "::", 2024-04-23T10:41:09.225 INFO:teuthology.orchestra.run.smithi114.stdout: "min": "", 2024-04-23T10:41:09.225 INFO:teuthology.orchestra.run.smithi114.stdout: "max": "", 2024-04-23T10:41:09.225 INFO:teuthology.orchestra.run.smithi114.stdout: "enum_allowed": [], 2024-04-23T10:41:09.225 INFO:teuthology.orchestra.run.smithi114.stdout: "desc": "the IPv4 or IPv6 address on which the module listens for HTTP requests", 2024-04-23T10:41:09.225 INFO:teuthology.orchestra.run.smithi114.stdout: "long_desc": "", 2024-04-23T10:41:09.226 INFO:teuthology.orchestra.run.smithi114.stdout: "tags": [], 2024-04-23T10:41:09.226 INFO:teuthology.orchestra.run.smithi114.stdout: "see_also": [] 2024-04-23T10:41:09.226 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-23T10:41:09.226 INFO:teuthology.orchestra.run.smithi114.stdout: "server_port": { 2024-04-23T10:41:09.226 INFO:teuthology.orchestra.run.smithi114.stdout: "name": "server_port", 2024-04-23T10:41:09.226 INFO:teuthology.orchestra.run.smithi114.stdout: "type": "int", 2024-04-23T10:41:09.226 INFO:teuthology.orchestra.run.smithi114.stdout: "level": "advanced", 2024-04-23T10:41:09.226 INFO:teuthology.orchestra.run.smithi114.stdout: "flags": 1, 2024-04-23T10:41:09.226 INFO:teuthology.orchestra.run.smithi114.stdout: "default_value": "9283", 2024-04-23T10:41:09.227 INFO:teuthology.orchestra.run.smithi114.stdout: "min": "", 2024-04-23T10:41:09.227 INFO:teuthology.orchestra.run.smithi114.stdout: "max": "", 2024-04-23T10:41:09.227 INFO:teuthology.orchestra.run.smithi114.stdout: "enum_allowed": [], 2024-04-23T10:41:09.227 INFO:teuthology.orchestra.run.smithi114.stdout: "desc": "the port on which the module listens for HTTP requests", 2024-04-23T10:41:09.227 INFO:teuthology.orchestra.run.smithi114.stdout: "long_desc": "", 2024-04-23T10:41:09.227 INFO:teuthology.orchestra.run.smithi114.stdout: "tags": [], 2024-04-23T10:41:09.227 INFO:teuthology.orchestra.run.smithi114.stdout: "see_also": [] 2024-04-23T10:41:09.227 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-23T10:41:09.227 INFO:teuthology.orchestra.run.smithi114.stdout: "stale_cache_strategy": { 2024-04-23T10:41:09.228 INFO:teuthology.orchestra.run.smithi114.stdout: "name": "stale_cache_strategy", 2024-04-23T10:41:09.228 INFO:teuthology.orchestra.run.smithi114.stdout: "type": "str", 2024-04-23T10:41:09.228 INFO:teuthology.orchestra.run.smithi114.stdout: "level": "advanced", 2024-04-23T10:41:09.228 INFO:teuthology.orchestra.run.smithi114.stdout: "flags": 0, 2024-04-23T10:41:09.228 INFO:teuthology.orchestra.run.smithi114.stdout: "default_value": "log", 2024-04-23T10:41:09.228 INFO:teuthology.orchestra.run.smithi114.stdout: "min": "", 2024-04-23T10:41:09.228 INFO:teuthology.orchestra.run.smithi114.stdout: "max": "", 2024-04-23T10:41:09.228 INFO:teuthology.orchestra.run.smithi114.stdout: "enum_allowed": [], 2024-04-23T10:41:09.228 INFO:teuthology.orchestra.run.smithi114.stdout: "desc": "", 2024-04-23T10:41:09.229 INFO:teuthology.orchestra.run.smithi114.stdout: "long_desc": "", 2024-04-23T10:41:09.229 INFO:teuthology.orchestra.run.smithi114.stdout: "tags": [], 2024-04-23T10:41:09.229 INFO:teuthology.orchestra.run.smithi114.stdout: "see_also": [] 2024-04-23T10:41:09.229 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-23T10:41:09.229 INFO:teuthology.orchestra.run.smithi114.stdout: "standby_behaviour": { 2024-04-23T10:41:09.229 INFO:teuthology.orchestra.run.smithi114.stdout: "name": "standby_behaviour", 2024-04-23T10:41:09.229 INFO:teuthology.orchestra.run.smithi114.stdout: "type": "str", 2024-04-23T10:41:09.229 INFO:teuthology.orchestra.run.smithi114.stdout: "level": "advanced", 2024-04-23T10:41:09.230 INFO:teuthology.orchestra.run.smithi114.stdout: "flags": 1, 2024-04-23T10:41:09.230 INFO:teuthology.orchestra.run.smithi114.stdout: "default_value": "default", 2024-04-23T10:41:09.230 INFO:teuthology.orchestra.run.smithi114.stdout: "min": "", 2024-04-23T10:41:09.230 INFO:teuthology.orchestra.run.smithi114.stdout: "max": "", 2024-04-23T10:41:09.230 INFO:teuthology.orchestra.run.smithi114.stdout: "enum_allowed": [ 2024-04-23T10:41:09.230 INFO:teuthology.orchestra.run.smithi114.stdout: "default", 2024-04-23T10:41:09.230 INFO:teuthology.orchestra.run.smithi114.stdout: "error" 2024-04-23T10:41:09.230 INFO:teuthology.orchestra.run.smithi114.stdout: ], 2024-04-23T10:41:09.230 INFO:teuthology.orchestra.run.smithi114.stdout: "desc": "", 2024-04-23T10:41:09.231 INFO:teuthology.orchestra.run.smithi114.stdout: "long_desc": "", 2024-04-23T10:41:09.231 INFO:teuthology.orchestra.run.smithi114.stdout: "tags": [], 2024-04-23T10:41:09.231 INFO:teuthology.orchestra.run.smithi114.stdout: "see_also": [] 2024-04-23T10:41:09.231 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-23T10:41:09.231 INFO:teuthology.orchestra.run.smithi114.stdout: "standby_error_status_code": { 2024-04-23T10:41:09.231 INFO:teuthology.orchestra.run.smithi114.stdout: "name": "standby_error_status_code", 2024-04-23T10:41:09.231 INFO:teuthology.orchestra.run.smithi114.stdout: "type": "int", 2024-04-23T10:41:09.231 INFO:teuthology.orchestra.run.smithi114.stdout: "level": "advanced", 2024-04-23T10:41:09.231 INFO:teuthology.orchestra.run.smithi114.stdout: "flags": 1, 2024-04-23T10:41:09.232 INFO:teuthology.orchestra.run.smithi114.stdout: "default_value": "500", 2024-04-23T10:41:09.232 INFO:teuthology.orchestra.run.smithi114.stdout: "min": "400", 2024-04-23T10:41:09.232 INFO:teuthology.orchestra.run.smithi114.stdout: "max": "599", 2024-04-23T10:41:09.232 INFO:teuthology.orchestra.run.smithi114.stdout: "enum_allowed": [], 2024-04-23T10:41:09.232 INFO:teuthology.orchestra.run.smithi114.stdout: "desc": "", 2024-04-23T10:41:09.232 INFO:teuthology.orchestra.run.smithi114.stdout: "long_desc": "", 2024-04-23T10:41:09.232 INFO:teuthology.orchestra.run.smithi114.stdout: "tags": [], 2024-04-23T10:41:09.232 INFO:teuthology.orchestra.run.smithi114.stdout: "see_also": [] 2024-04-23T10:41:09.232 INFO:teuthology.orchestra.run.smithi114.stdout: } 2024-04-23T10:41:09.233 INFO:teuthology.orchestra.run.smithi114.stdout: } 2024-04-23T10:41:09.233 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-23T10:41:09.233 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-04-23T10:41:09.233 INFO:teuthology.orchestra.run.smithi114.stdout: "name": "rbd_support", 2024-04-23T10:41:09.233 INFO:teuthology.orchestra.run.smithi114.stdout: "can_run": true, 2024-04-23T10:41:09.233 INFO:teuthology.orchestra.run.smithi114.stdout: "error_string": "", 2024-04-23T10:41:09.233 INFO:teuthology.orchestra.run.smithi114.stdout: "module_options": { 2024-04-23T10:41:09.233 INFO:teuthology.orchestra.run.smithi114.stdout: "log_level": { 2024-04-23T10:41:09.234 INFO:teuthology.orchestra.run.smithi114.stdout: "name": "log_level", 2024-04-23T10:41:09.234 INFO:teuthology.orchestra.run.smithi114.stdout: "type": "str", 2024-04-23T10:41:09.234 INFO:teuthology.orchestra.run.smithi114.stdout: "level": "advanced", 2024-04-23T10:41:09.234 INFO:teuthology.orchestra.run.smithi114.stdout: "flags": 1, 2024-04-23T10:41:09.234 INFO:teuthology.orchestra.run.smithi114.stdout: "default_value": "", 2024-04-23T10:41:09.234 INFO:teuthology.orchestra.run.smithi114.stdout: "min": "", 2024-04-23T10:41:09.234 INFO:teuthology.orchestra.run.smithi114.stdout: "max": "", 2024-04-23T10:41:09.234 INFO:teuthology.orchestra.run.smithi114.stdout: "enum_allowed": [ 2024-04-23T10:41:09.234 INFO:teuthology.orchestra.run.smithi114.stdout: "", 2024-04-23T10:41:09.235 INFO:teuthology.orchestra.run.smithi114.stdout: "critical", 2024-04-23T10:41:09.235 INFO:teuthology.orchestra.run.smithi114.stdout: "debug", 2024-04-23T10:41:09.235 INFO:teuthology.orchestra.run.smithi114.stdout: "error", 2024-04-23T10:41:09.235 INFO:teuthology.orchestra.run.smithi114.stdout: "info", 2024-04-23T10:41:09.235 INFO:teuthology.orchestra.run.smithi114.stdout: "warning" 2024-04-23T10:41:09.235 INFO:teuthology.orchestra.run.smithi114.stdout: ], 2024-04-23T10:41:09.235 INFO:teuthology.orchestra.run.smithi114.stdout: "desc": "", 2024-04-23T10:41:09.235 INFO:teuthology.orchestra.run.smithi114.stdout: "long_desc": "", 2024-04-23T10:41:09.236 INFO:teuthology.orchestra.run.smithi114.stdout: "tags": [], 2024-04-23T10:41:09.236 INFO:teuthology.orchestra.run.smithi114.stdout: "see_also": [] 2024-04-23T10:41:09.236 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-23T10:41:09.236 INFO:teuthology.orchestra.run.smithi114.stdout: "log_to_cluster": { 2024-04-23T10:41:09.236 INFO:teuthology.orchestra.run.smithi114.stdout: "name": "log_to_cluster", 2024-04-23T10:41:09.236 INFO:teuthology.orchestra.run.smithi114.stdout: "type": "bool", 2024-04-23T10:41:09.236 INFO:teuthology.orchestra.run.smithi114.stdout: "level": "advanced", 2024-04-23T10:41:09.236 INFO:teuthology.orchestra.run.smithi114.stdout: "flags": 1, 2024-04-23T10:41:09.236 INFO:teuthology.orchestra.run.smithi114.stdout: "default_value": "False", 2024-04-23T10:41:09.236 INFO:teuthology.orchestra.run.smithi114.stdout: "min": "", 2024-04-23T10:41:09.237 INFO:teuthology.orchestra.run.smithi114.stdout: "max": "", 2024-04-23T10:41:09.237 INFO:teuthology.orchestra.run.smithi114.stdout: "enum_allowed": [], 2024-04-23T10:41:09.237 INFO:teuthology.orchestra.run.smithi114.stdout: "desc": "", 2024-04-23T10:41:09.237 INFO:teuthology.orchestra.run.smithi114.stdout: "long_desc": "", 2024-04-23T10:41:09.237 INFO:teuthology.orchestra.run.smithi114.stdout: "tags": [], 2024-04-23T10:41:09.237 INFO:teuthology.orchestra.run.smithi114.stdout: "see_also": [] 2024-04-23T10:41:09.237 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-23T10:41:09.237 INFO:teuthology.orchestra.run.smithi114.stdout: "log_to_cluster_level": { 2024-04-23T10:41:09.238 INFO:teuthology.orchestra.run.smithi114.stdout: "name": "log_to_cluster_level", 2024-04-23T10:41:09.238 INFO:teuthology.orchestra.run.smithi114.stdout: "type": "str", 2024-04-23T10:41:09.238 INFO:teuthology.orchestra.run.smithi114.stdout: "level": "advanced", 2024-04-23T10:41:09.238 INFO:teuthology.orchestra.run.smithi114.stdout: "flags": 1, 2024-04-23T10:41:09.238 INFO:teuthology.orchestra.run.smithi114.stdout: "default_value": "info", 2024-04-23T10:41:09.238 INFO:teuthology.orchestra.run.smithi114.stdout: "min": "", 2024-04-23T10:41:09.238 INFO:teuthology.orchestra.run.smithi114.stdout: "max": "", 2024-04-23T10:41:09.238 INFO:teuthology.orchestra.run.smithi114.stdout: "enum_allowed": [ 2024-04-23T10:41:09.238 INFO:teuthology.orchestra.run.smithi114.stdout: "", 2024-04-23T10:41:09.239 INFO:teuthology.orchestra.run.smithi114.stdout: "critical", 2024-04-23T10:41:09.239 INFO:teuthology.orchestra.run.smithi114.stdout: "debug", 2024-04-23T10:41:09.239 INFO:teuthology.orchestra.run.smithi114.stdout: "error", 2024-04-23T10:41:09.239 INFO:teuthology.orchestra.run.smithi114.stdout: "info", 2024-04-23T10:41:09.239 INFO:teuthology.orchestra.run.smithi114.stdout: "warning" 2024-04-23T10:41:09.239 INFO:teuthology.orchestra.run.smithi114.stdout: ], 2024-04-23T10:41:09.239 INFO:teuthology.orchestra.run.smithi114.stdout: "desc": "", 2024-04-23T10:41:09.239 INFO:teuthology.orchestra.run.smithi114.stdout: "long_desc": "", 2024-04-23T10:41:09.240 INFO:teuthology.orchestra.run.smithi114.stdout: "tags": [], 2024-04-23T10:41:09.240 INFO:teuthology.orchestra.run.smithi114.stdout: "see_also": [] 2024-04-23T10:41:09.240 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-23T10:41:09.240 INFO:teuthology.orchestra.run.smithi114.stdout: "log_to_file": { 2024-04-23T10:41:09.240 INFO:teuthology.orchestra.run.smithi114.stdout: "name": "log_to_file", 2024-04-23T10:41:09.240 INFO:teuthology.orchestra.run.smithi114.stdout: "type": "bool", 2024-04-23T10:41:09.240 INFO:teuthology.orchestra.run.smithi114.stdout: "level": "advanced", 2024-04-23T10:41:09.240 INFO:teuthology.orchestra.run.smithi114.stdout: "flags": 1, 2024-04-23T10:41:09.240 INFO:teuthology.orchestra.run.smithi114.stdout: "default_value": "False", 2024-04-23T10:41:09.241 INFO:teuthology.orchestra.run.smithi114.stdout: "min": "", 2024-04-23T10:41:09.241 INFO:teuthology.orchestra.run.smithi114.stdout: "max": "", 2024-04-23T10:41:09.241 INFO:teuthology.orchestra.run.smithi114.stdout: "enum_allowed": [], 2024-04-23T10:41:09.241 INFO:teuthology.orchestra.run.smithi114.stdout: "desc": "", 2024-04-23T10:41:09.241 INFO:teuthology.orchestra.run.smithi114.stdout: "long_desc": "", 2024-04-23T10:41:09.241 INFO:teuthology.orchestra.run.smithi114.stdout: "tags": [], 2024-04-23T10:41:09.241 INFO:teuthology.orchestra.run.smithi114.stdout: "see_also": [] 2024-04-23T10:41:09.241 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-23T10:41:09.241 INFO:teuthology.orchestra.run.smithi114.stdout: "max_concurrent_snap_create": { 2024-04-23T10:41:09.242 INFO:teuthology.orchestra.run.smithi114.stdout: "name": "max_concurrent_snap_create", 2024-04-23T10:41:09.242 INFO:teuthology.orchestra.run.smithi114.stdout: "type": "int", 2024-04-23T10:41:09.242 INFO:teuthology.orchestra.run.smithi114.stdout: "level": "advanced", 2024-04-23T10:41:09.242 INFO:teuthology.orchestra.run.smithi114.stdout: "flags": 0, 2024-04-23T10:41:09.242 INFO:teuthology.orchestra.run.smithi114.stdout: "default_value": "10", 2024-04-23T10:41:09.242 INFO:teuthology.orchestra.run.smithi114.stdout: "min": "", 2024-04-23T10:41:09.242 INFO:teuthology.orchestra.run.smithi114.stdout: "max": "", 2024-04-23T10:41:09.242 INFO:teuthology.orchestra.run.smithi114.stdout: "enum_allowed": [], 2024-04-23T10:41:09.243 INFO:teuthology.orchestra.run.smithi114.stdout: "desc": "", 2024-04-23T10:41:09.243 INFO:teuthology.orchestra.run.smithi114.stdout: "long_desc": "", 2024-04-23T10:41:09.243 INFO:teuthology.orchestra.run.smithi114.stdout: "tags": [], 2024-04-23T10:41:09.243 INFO:teuthology.orchestra.run.smithi114.stdout: "see_also": [] 2024-04-23T10:41:09.243 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-23T10:41:09.243 INFO:teuthology.orchestra.run.smithi114.stdout: "mirror_snapshot_schedule": { 2024-04-23T10:41:09.243 INFO:teuthology.orchestra.run.smithi114.stdout: "name": "mirror_snapshot_schedule", 2024-04-23T10:41:09.243 INFO:teuthology.orchestra.run.smithi114.stdout: "type": "str", 2024-04-23T10:41:09.243 INFO:teuthology.orchestra.run.smithi114.stdout: "level": "advanced", 2024-04-23T10:41:09.244 INFO:teuthology.orchestra.run.smithi114.stdout: "flags": 0, 2024-04-23T10:41:09.244 INFO:teuthology.orchestra.run.smithi114.stdout: "default_value": "", 2024-04-23T10:41:09.244 INFO:teuthology.orchestra.run.smithi114.stdout: "min": "", 2024-04-23T10:41:09.244 INFO:teuthology.orchestra.run.smithi114.stdout: "max": "", 2024-04-23T10:41:09.244 INFO:teuthology.orchestra.run.smithi114.stdout: "enum_allowed": [], 2024-04-23T10:41:09.244 INFO:teuthology.orchestra.run.smithi114.stdout: "desc": "", 2024-04-23T10:41:09.244 INFO:teuthology.orchestra.run.smithi114.stdout: "long_desc": "", 2024-04-23T10:41:09.244 INFO:teuthology.orchestra.run.smithi114.stdout: "tags": [], 2024-04-23T10:41:09.244 INFO:teuthology.orchestra.run.smithi114.stdout: "see_also": [] 2024-04-23T10:41:09.245 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-23T10:41:09.245 INFO:teuthology.orchestra.run.smithi114.stdout: "trash_purge_schedule": { 2024-04-23T10:41:09.245 INFO:teuthology.orchestra.run.smithi114.stdout: "name": "trash_purge_schedule", 2024-04-23T10:41:09.245 INFO:teuthology.orchestra.run.smithi114.stdout: "type": "str", 2024-04-23T10:41:09.245 INFO:teuthology.orchestra.run.smithi114.stdout: "level": "advanced", 2024-04-23T10:41:09.245 INFO:teuthology.orchestra.run.smithi114.stdout: "flags": 0, 2024-04-23T10:41:09.245 INFO:teuthology.orchestra.run.smithi114.stdout: "default_value": "", 2024-04-23T10:41:09.245 INFO:teuthology.orchestra.run.smithi114.stdout: "min": "", 2024-04-23T10:41:09.245 INFO:teuthology.orchestra.run.smithi114.stdout: "max": "", 2024-04-23T10:41:09.246 INFO:teuthology.orchestra.run.smithi114.stdout: "enum_allowed": [], 2024-04-23T10:41:09.246 INFO:teuthology.orchestra.run.smithi114.stdout: "desc": "", 2024-04-23T10:41:09.246 INFO:teuthology.orchestra.run.smithi114.stdout: "long_desc": "", 2024-04-23T10:41:09.246 INFO:teuthology.orchestra.run.smithi114.stdout: "tags": [], 2024-04-23T10:41:09.246 INFO:teuthology.orchestra.run.smithi114.stdout: "see_also": [] 2024-04-23T10:41:09.246 INFO:teuthology.orchestra.run.smithi114.stdout: } 2024-04-23T10:41:09.246 INFO:teuthology.orchestra.run.smithi114.stdout: } 2024-04-23T10:41:09.246 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-23T10:41:09.246 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-04-23T10:41:09.247 INFO:teuthology.orchestra.run.smithi114.stdout: "name": "restful", 2024-04-23T10:41:09.247 INFO:teuthology.orchestra.run.smithi114.stdout: "can_run": true, 2024-04-23T10:41:09.247 INFO:teuthology.orchestra.run.smithi114.stdout: "error_string": "", 2024-04-23T10:41:09.247 INFO:teuthology.orchestra.run.smithi114.stdout: "module_options": { 2024-04-23T10:41:09.247 INFO:teuthology.orchestra.run.smithi114.stdout: "enable_auth": { 2024-04-23T10:41:09.247 INFO:teuthology.orchestra.run.smithi114.stdout: "name": "enable_auth", 2024-04-23T10:41:09.247 INFO:teuthology.orchestra.run.smithi114.stdout: "type": "bool", 2024-04-23T10:41:09.247 INFO:teuthology.orchestra.run.smithi114.stdout: "level": "advanced", 2024-04-23T10:41:09.247 INFO:teuthology.orchestra.run.smithi114.stdout: "flags": 0, 2024-04-23T10:41:09.248 INFO:teuthology.orchestra.run.smithi114.stdout: "default_value": "True", 2024-04-23T10:41:09.248 INFO:teuthology.orchestra.run.smithi114.stdout: "min": "", 2024-04-23T10:41:09.248 INFO:teuthology.orchestra.run.smithi114.stdout: "max": "", 2024-04-23T10:41:09.248 INFO:teuthology.orchestra.run.smithi114.stdout: "enum_allowed": [], 2024-04-23T10:41:09.248 INFO:teuthology.orchestra.run.smithi114.stdout: "desc": "", 2024-04-23T10:41:09.248 INFO:teuthology.orchestra.run.smithi114.stdout: "long_desc": "", 2024-04-23T10:41:09.248 INFO:teuthology.orchestra.run.smithi114.stdout: "tags": [], 2024-04-23T10:41:09.248 INFO:teuthology.orchestra.run.smithi114.stdout: "see_also": [] 2024-04-23T10:41:09.249 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-23T10:41:09.249 INFO:teuthology.orchestra.run.smithi114.stdout: "key_file": { 2024-04-23T10:41:09.249 INFO:teuthology.orchestra.run.smithi114.stdout: "name": "key_file", 2024-04-23T10:41:09.249 INFO:teuthology.orchestra.run.smithi114.stdout: "type": "str", 2024-04-23T10:41:09.249 INFO:teuthology.orchestra.run.smithi114.stdout: "level": "advanced", 2024-04-23T10:41:09.249 INFO:teuthology.orchestra.run.smithi114.stdout: "flags": 0, 2024-04-23T10:41:09.249 INFO:teuthology.orchestra.run.smithi114.stdout: "default_value": "", 2024-04-23T10:41:09.249 INFO:teuthology.orchestra.run.smithi114.stdout: "min": "", 2024-04-23T10:41:09.249 INFO:teuthology.orchestra.run.smithi114.stdout: "max": "", 2024-04-23T10:41:09.250 INFO:teuthology.orchestra.run.smithi114.stdout: "enum_allowed": [], 2024-04-23T10:41:09.250 INFO:teuthology.orchestra.run.smithi114.stdout: "desc": "", 2024-04-23T10:41:09.250 INFO:teuthology.orchestra.run.smithi114.stdout: "long_desc": "", 2024-04-23T10:41:09.250 INFO:teuthology.orchestra.run.smithi114.stdout: "tags": [], 2024-04-23T10:41:09.250 INFO:teuthology.orchestra.run.smithi114.stdout: "see_also": [] 2024-04-23T10:41:09.250 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-23T10:41:09.250 INFO:teuthology.orchestra.run.smithi114.stdout: "log_level": { 2024-04-23T10:41:09.250 INFO:teuthology.orchestra.run.smithi114.stdout: "name": "log_level", 2024-04-23T10:41:09.251 INFO:teuthology.orchestra.run.smithi114.stdout: "type": "str", 2024-04-23T10:41:09.251 INFO:teuthology.orchestra.run.smithi114.stdout: "level": "advanced", 2024-04-23T10:41:09.251 INFO:teuthology.orchestra.run.smithi114.stdout: "flags": 1, 2024-04-23T10:41:09.251 INFO:teuthology.orchestra.run.smithi114.stdout: "default_value": "", 2024-04-23T10:41:09.251 INFO:teuthology.orchestra.run.smithi114.stdout: "min": "", 2024-04-23T10:41:09.251 INFO:teuthology.orchestra.run.smithi114.stdout: "max": "", 2024-04-23T10:41:09.251 INFO:teuthology.orchestra.run.smithi114.stdout: "enum_allowed": [ 2024-04-23T10:41:09.251 INFO:teuthology.orchestra.run.smithi114.stdout: "", 2024-04-23T10:41:09.252 INFO:teuthology.orchestra.run.smithi114.stdout: "critical", 2024-04-23T10:41:09.252 INFO:teuthology.orchestra.run.smithi114.stdout: "debug", 2024-04-23T10:41:09.252 INFO:teuthology.orchestra.run.smithi114.stdout: "error", 2024-04-23T10:41:09.252 INFO:teuthology.orchestra.run.smithi114.stdout: "info", 2024-04-23T10:41:09.252 INFO:teuthology.orchestra.run.smithi114.stdout: "warning" 2024-04-23T10:41:09.252 INFO:teuthology.orchestra.run.smithi114.stdout: ], 2024-04-23T10:41:09.252 INFO:teuthology.orchestra.run.smithi114.stdout: "desc": "", 2024-04-23T10:41:09.252 INFO:teuthology.orchestra.run.smithi114.stdout: "long_desc": "", 2024-04-23T10:41:09.252 INFO:teuthology.orchestra.run.smithi114.stdout: "tags": [], 2024-04-23T10:41:09.253 INFO:teuthology.orchestra.run.smithi114.stdout: "see_also": [] 2024-04-23T10:41:09.253 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-23T10:41:09.253 INFO:teuthology.orchestra.run.smithi114.stdout: "log_to_cluster": { 2024-04-23T10:41:09.253 INFO:teuthology.orchestra.run.smithi114.stdout: "name": "log_to_cluster", 2024-04-23T10:41:09.253 INFO:teuthology.orchestra.run.smithi114.stdout: "type": "bool", 2024-04-23T10:41:09.253 INFO:teuthology.orchestra.run.smithi114.stdout: "level": "advanced", 2024-04-23T10:41:09.253 INFO:teuthology.orchestra.run.smithi114.stdout: "flags": 1, 2024-04-23T10:41:09.253 INFO:teuthology.orchestra.run.smithi114.stdout: "default_value": "False", 2024-04-23T10:41:09.254 INFO:teuthology.orchestra.run.smithi114.stdout: "min": "", 2024-04-23T10:41:09.254 INFO:teuthology.orchestra.run.smithi114.stdout: "max": "", 2024-04-23T10:41:09.254 INFO:teuthology.orchestra.run.smithi114.stdout: "enum_allowed": [], 2024-04-23T10:41:09.254 INFO:teuthology.orchestra.run.smithi114.stdout: "desc": "", 2024-04-23T10:41:09.254 INFO:teuthology.orchestra.run.smithi114.stdout: "long_desc": "", 2024-04-23T10:41:09.254 INFO:teuthology.orchestra.run.smithi114.stdout: "tags": [], 2024-04-23T10:41:09.254 INFO:teuthology.orchestra.run.smithi114.stdout: "see_also": [] 2024-04-23T10:41:09.254 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-23T10:41:09.255 INFO:teuthology.orchestra.run.smithi114.stdout: "log_to_cluster_level": { 2024-04-23T10:41:09.255 INFO:teuthology.orchestra.run.smithi114.stdout: "name": "log_to_cluster_level", 2024-04-23T10:41:09.255 INFO:teuthology.orchestra.run.smithi114.stdout: "type": "str", 2024-04-23T10:41:09.255 INFO:teuthology.orchestra.run.smithi114.stdout: "level": "advanced", 2024-04-23T10:41:09.255 INFO:teuthology.orchestra.run.smithi114.stdout: "flags": 1, 2024-04-23T10:41:09.255 INFO:teuthology.orchestra.run.smithi114.stdout: "default_value": "info", 2024-04-23T10:41:09.255 INFO:teuthology.orchestra.run.smithi114.stdout: "min": "", 2024-04-23T10:41:09.255 INFO:teuthology.orchestra.run.smithi114.stdout: "max": "", 2024-04-23T10:41:09.255 INFO:teuthology.orchestra.run.smithi114.stdout: "enum_allowed": [ 2024-04-23T10:41:09.256 INFO:teuthology.orchestra.run.smithi114.stdout: "", 2024-04-23T10:41:09.256 INFO:teuthology.orchestra.run.smithi114.stdout: "critical", 2024-04-23T10:41:09.256 INFO:teuthology.orchestra.run.smithi114.stdout: "debug", 2024-04-23T10:41:09.256 INFO:teuthology.orchestra.run.smithi114.stdout: "error", 2024-04-23T10:41:09.256 INFO:teuthology.orchestra.run.smithi114.stdout: "info", 2024-04-23T10:41:09.256 INFO:teuthology.orchestra.run.smithi114.stdout: "warning" 2024-04-23T10:41:09.256 INFO:teuthology.orchestra.run.smithi114.stdout: ], 2024-04-23T10:41:09.256 INFO:teuthology.orchestra.run.smithi114.stdout: "desc": "", 2024-04-23T10:41:09.256 INFO:teuthology.orchestra.run.smithi114.stdout: "long_desc": "", 2024-04-23T10:41:09.257 INFO:teuthology.orchestra.run.smithi114.stdout: "tags": [], 2024-04-23T10:41:09.257 INFO:teuthology.orchestra.run.smithi114.stdout: "see_also": [] 2024-04-23T10:41:09.257 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-23T10:41:09.257 INFO:teuthology.orchestra.run.smithi114.stdout: "log_to_file": { 2024-04-23T10:41:09.257 INFO:teuthology.orchestra.run.smithi114.stdout: "name": "log_to_file", 2024-04-23T10:41:09.257 INFO:teuthology.orchestra.run.smithi114.stdout: "type": "bool", 2024-04-23T10:41:09.257 INFO:teuthology.orchestra.run.smithi114.stdout: "level": "advanced", 2024-04-23T10:41:09.257 INFO:teuthology.orchestra.run.smithi114.stdout: "flags": 1, 2024-04-23T10:41:09.257 INFO:teuthology.orchestra.run.smithi114.stdout: "default_value": "False", 2024-04-23T10:41:09.258 INFO:teuthology.orchestra.run.smithi114.stdout: "min": "", 2024-04-23T10:41:09.258 INFO:teuthology.orchestra.run.smithi114.stdout: "max": "", 2024-04-23T10:41:09.258 INFO:teuthology.orchestra.run.smithi114.stdout: "enum_allowed": [], 2024-04-23T10:41:09.258 INFO:teuthology.orchestra.run.smithi114.stdout: "desc": "", 2024-04-23T10:41:09.258 INFO:teuthology.orchestra.run.smithi114.stdout: "long_desc": "", 2024-04-23T10:41:09.258 INFO:teuthology.orchestra.run.smithi114.stdout: "tags": [], 2024-04-23T10:41:09.258 INFO:teuthology.orchestra.run.smithi114.stdout: "see_also": [] 2024-04-23T10:41:09.258 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-23T10:41:09.259 INFO:teuthology.orchestra.run.smithi114.stdout: "server_addr": { 2024-04-23T10:41:09.259 INFO:teuthology.orchestra.run.smithi114.stdout: "name": "server_addr", 2024-04-23T10:41:09.259 INFO:teuthology.orchestra.run.smithi114.stdout: "type": "str", 2024-04-23T10:41:09.259 INFO:teuthology.orchestra.run.smithi114.stdout: "level": "advanced", 2024-04-23T10:41:09.259 INFO:teuthology.orchestra.run.smithi114.stdout: "flags": 0, 2024-04-23T10:41:09.259 INFO:teuthology.orchestra.run.smithi114.stdout: "default_value": "", 2024-04-23T10:41:09.259 INFO:teuthology.orchestra.run.smithi114.stdout: "min": "", 2024-04-23T10:41:09.259 INFO:teuthology.orchestra.run.smithi114.stdout: "max": "", 2024-04-23T10:41:09.259 INFO:teuthology.orchestra.run.smithi114.stdout: "enum_allowed": [], 2024-04-23T10:41:09.260 INFO:teuthology.orchestra.run.smithi114.stdout: "desc": "", 2024-04-23T10:41:09.260 INFO:teuthology.orchestra.run.smithi114.stdout: "long_desc": "", 2024-04-23T10:41:09.260 INFO:teuthology.orchestra.run.smithi114.stdout: "tags": [], 2024-04-23T10:41:09.260 INFO:teuthology.orchestra.run.smithi114.stdout: "see_also": [] 2024-04-23T10:41:09.260 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-23T10:41:09.260 INFO:teuthology.orchestra.run.smithi114.stdout: "server_port": { 2024-04-23T10:41:09.260 INFO:teuthology.orchestra.run.smithi114.stdout: "name": "server_port", 2024-04-23T10:41:09.260 INFO:teuthology.orchestra.run.smithi114.stdout: "type": "str", 2024-04-23T10:41:09.261 INFO:teuthology.orchestra.run.smithi114.stdout: "level": "advanced", 2024-04-23T10:41:09.261 INFO:teuthology.orchestra.run.smithi114.stdout: "flags": 0, 2024-04-23T10:41:09.261 INFO:teuthology.orchestra.run.smithi114.stdout: "default_value": "", 2024-04-23T10:41:09.261 INFO:teuthology.orchestra.run.smithi114.stdout: "min": "", 2024-04-23T10:41:09.261 INFO:teuthology.orchestra.run.smithi114.stdout: "max": "", 2024-04-23T10:41:09.261 INFO:teuthology.orchestra.run.smithi114.stdout: "enum_allowed": [], 2024-04-23T10:41:09.261 INFO:teuthology.orchestra.run.smithi114.stdout: "desc": "", 2024-04-23T10:41:09.261 INFO:teuthology.orchestra.run.smithi114.stdout: "long_desc": "", 2024-04-23T10:41:09.262 INFO:teuthology.orchestra.run.smithi114.stdout: "tags": [], 2024-04-23T10:41:09.262 INFO:teuthology.orchestra.run.smithi114.stdout: "see_also": [] 2024-04-23T10:41:09.262 INFO:teuthology.orchestra.run.smithi114.stdout: } 2024-04-23T10:41:09.262 INFO:teuthology.orchestra.run.smithi114.stdout: } 2024-04-23T10:41:09.262 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-23T10:41:09.262 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-04-23T10:41:09.262 INFO:teuthology.orchestra.run.smithi114.stdout: "name": "rgw", 2024-04-23T10:41:09.262 INFO:teuthology.orchestra.run.smithi114.stdout: "can_run": true, 2024-04-23T10:41:09.262 INFO:teuthology.orchestra.run.smithi114.stdout: "error_string": "", 2024-04-23T10:41:09.263 INFO:teuthology.orchestra.run.smithi114.stdout: "module_options": { 2024-04-23T10:41:09.263 INFO:teuthology.orchestra.run.smithi114.stdout: "log_level": { 2024-04-23T10:41:09.263 INFO:teuthology.orchestra.run.smithi114.stdout: "name": "log_level", 2024-04-23T10:41:09.263 INFO:teuthology.orchestra.run.smithi114.stdout: "type": "str", 2024-04-23T10:41:09.263 INFO:teuthology.orchestra.run.smithi114.stdout: "level": "advanced", 2024-04-23T10:41:09.263 INFO:teuthology.orchestra.run.smithi114.stdout: "flags": 1, 2024-04-23T10:41:09.263 INFO:teuthology.orchestra.run.smithi114.stdout: "default_value": "", 2024-04-23T10:41:09.263 INFO:teuthology.orchestra.run.smithi114.stdout: "min": "", 2024-04-23T10:41:09.264 INFO:teuthology.orchestra.run.smithi114.stdout: "max": "", 2024-04-23T10:41:09.264 INFO:teuthology.orchestra.run.smithi114.stdout: "enum_allowed": [ 2024-04-23T10:41:09.264 INFO:teuthology.orchestra.run.smithi114.stdout: "", 2024-04-23T10:41:09.264 INFO:teuthology.orchestra.run.smithi114.stdout: "critical", 2024-04-23T10:41:09.264 INFO:teuthology.orchestra.run.smithi114.stdout: "debug", 2024-04-23T10:41:09.264 INFO:teuthology.orchestra.run.smithi114.stdout: "error", 2024-04-23T10:41:09.264 INFO:teuthology.orchestra.run.smithi114.stdout: "info", 2024-04-23T10:41:09.264 INFO:teuthology.orchestra.run.smithi114.stdout: "warning" 2024-04-23T10:41:09.264 INFO:teuthology.orchestra.run.smithi114.stdout: ], 2024-04-23T10:41:09.265 INFO:teuthology.orchestra.run.smithi114.stdout: "desc": "", 2024-04-23T10:41:09.265 INFO:teuthology.orchestra.run.smithi114.stdout: "long_desc": "", 2024-04-23T10:41:09.265 INFO:teuthology.orchestra.run.smithi114.stdout: "tags": [], 2024-04-23T10:41:09.265 INFO:teuthology.orchestra.run.smithi114.stdout: "see_also": [] 2024-04-23T10:41:09.265 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-23T10:41:09.265 INFO:teuthology.orchestra.run.smithi114.stdout: "log_to_cluster": { 2024-04-23T10:41:09.265 INFO:teuthology.orchestra.run.smithi114.stdout: "name": "log_to_cluster", 2024-04-23T10:41:09.265 INFO:teuthology.orchestra.run.smithi114.stdout: "type": "bool", 2024-04-23T10:41:09.265 INFO:teuthology.orchestra.run.smithi114.stdout: "level": "advanced", 2024-04-23T10:41:09.266 INFO:teuthology.orchestra.run.smithi114.stdout: "flags": 1, 2024-04-23T10:41:09.266 INFO:teuthology.orchestra.run.smithi114.stdout: "default_value": "False", 2024-04-23T10:41:09.266 INFO:teuthology.orchestra.run.smithi114.stdout: "min": "", 2024-04-23T10:41:09.266 INFO:teuthology.orchestra.run.smithi114.stdout: "max": "", 2024-04-23T10:41:09.266 INFO:teuthology.orchestra.run.smithi114.stdout: "enum_allowed": [], 2024-04-23T10:41:09.266 INFO:teuthology.orchestra.run.smithi114.stdout: "desc": "", 2024-04-23T10:41:09.266 INFO:teuthology.orchestra.run.smithi114.stdout: "long_desc": "", 2024-04-23T10:41:09.266 INFO:teuthology.orchestra.run.smithi114.stdout: "tags": [], 2024-04-23T10:41:09.267 INFO:teuthology.orchestra.run.smithi114.stdout: "see_also": [] 2024-04-23T10:41:09.267 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-23T10:41:09.267 INFO:teuthology.orchestra.run.smithi114.stdout: "log_to_cluster_level": { 2024-04-23T10:41:09.267 INFO:teuthology.orchestra.run.smithi114.stdout: "name": "log_to_cluster_level", 2024-04-23T10:41:09.267 INFO:teuthology.orchestra.run.smithi114.stdout: "type": "str", 2024-04-23T10:41:09.267 INFO:teuthology.orchestra.run.smithi114.stdout: "level": "advanced", 2024-04-23T10:41:09.267 INFO:teuthology.orchestra.run.smithi114.stdout: "flags": 1, 2024-04-23T10:41:09.267 INFO:teuthology.orchestra.run.smithi114.stdout: "default_value": "info", 2024-04-23T10:41:09.268 INFO:teuthology.orchestra.run.smithi114.stdout: "min": "", 2024-04-23T10:41:09.268 INFO:teuthology.orchestra.run.smithi114.stdout: "max": "", 2024-04-23T10:41:09.268 INFO:teuthology.orchestra.run.smithi114.stdout: "enum_allowed": [ 2024-04-23T10:41:09.268 INFO:teuthology.orchestra.run.smithi114.stdout: "", 2024-04-23T10:41:09.268 INFO:teuthology.orchestra.run.smithi114.stdout: "critical", 2024-04-23T10:41:09.268 INFO:teuthology.orchestra.run.smithi114.stdout: "debug", 2024-04-23T10:41:09.268 INFO:teuthology.orchestra.run.smithi114.stdout: "error", 2024-04-23T10:41:09.268 INFO:teuthology.orchestra.run.smithi114.stdout: "info", 2024-04-23T10:41:09.269 INFO:teuthology.orchestra.run.smithi114.stdout: "warning" 2024-04-23T10:41:09.269 INFO:teuthology.orchestra.run.smithi114.stdout: ], 2024-04-23T10:41:09.269 INFO:teuthology.orchestra.run.smithi114.stdout: "desc": "", 2024-04-23T10:41:09.269 INFO:teuthology.orchestra.run.smithi114.stdout: "long_desc": "", 2024-04-23T10:41:09.269 INFO:teuthology.orchestra.run.smithi114.stdout: "tags": [], 2024-04-23T10:41:09.269 INFO:teuthology.orchestra.run.smithi114.stdout: "see_also": [] 2024-04-23T10:41:09.269 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-23T10:41:09.270 INFO:teuthology.orchestra.run.smithi114.stdout: "log_to_file": { 2024-04-23T10:41:09.270 INFO:teuthology.orchestra.run.smithi114.stdout: "name": "log_to_file", 2024-04-23T10:41:09.270 INFO:teuthology.orchestra.run.smithi114.stdout: "type": "bool", 2024-04-23T10:41:09.270 INFO:teuthology.orchestra.run.smithi114.stdout: "level": "advanced", 2024-04-23T10:41:09.270 INFO:teuthology.orchestra.run.smithi114.stdout: "flags": 1, 2024-04-23T10:41:09.270 INFO:teuthology.orchestra.run.smithi114.stdout: "default_value": "False", 2024-04-23T10:41:09.270 INFO:teuthology.orchestra.run.smithi114.stdout: "min": "", 2024-04-23T10:41:09.270 INFO:teuthology.orchestra.run.smithi114.stdout: "max": "", 2024-04-23T10:41:09.270 INFO:teuthology.orchestra.run.smithi114.stdout: "enum_allowed": [], 2024-04-23T10:41:09.271 INFO:teuthology.orchestra.run.smithi114.stdout: "desc": "", 2024-04-23T10:41:09.271 INFO:teuthology.orchestra.run.smithi114.stdout: "long_desc": "", 2024-04-23T10:41:09.271 INFO:teuthology.orchestra.run.smithi114.stdout: "tags": [], 2024-04-23T10:41:09.271 INFO:teuthology.orchestra.run.smithi114.stdout: "see_also": [] 2024-04-23T10:41:09.271 INFO:teuthology.orchestra.run.smithi114.stdout: } 2024-04-23T10:41:09.271 INFO:teuthology.orchestra.run.smithi114.stdout: } 2024-04-23T10:41:09.271 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-23T10:41:09.271 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-04-23T10:41:09.272 INFO:teuthology.orchestra.run.smithi114.stdout: "name": "rook", 2024-04-23T10:41:09.272 INFO:teuthology.orchestra.run.smithi114.stdout: "can_run": true, 2024-04-23T10:41:09.272 INFO:teuthology.orchestra.run.smithi114.stdout: "error_string": "", 2024-04-23T10:41:09.272 INFO:teuthology.orchestra.run.smithi114.stdout: "module_options": { 2024-04-23T10:41:09.272 INFO:teuthology.orchestra.run.smithi114.stdout: "log_level": { 2024-04-23T10:41:09.272 INFO:teuthology.orchestra.run.smithi114.stdout: "name": "log_level", 2024-04-23T10:41:09.272 INFO:teuthology.orchestra.run.smithi114.stdout: "type": "str", 2024-04-23T10:41:09.272 INFO:teuthology.orchestra.run.smithi114.stdout: "level": "advanced", 2024-04-23T10:41:09.272 INFO:teuthology.orchestra.run.smithi114.stdout: "flags": 1, 2024-04-23T10:41:09.273 INFO:teuthology.orchestra.run.smithi114.stdout: "default_value": "", 2024-04-23T10:41:09.273 INFO:teuthology.orchestra.run.smithi114.stdout: "min": "", 2024-04-23T10:41:09.273 INFO:teuthology.orchestra.run.smithi114.stdout: "max": "", 2024-04-23T10:41:09.273 INFO:teuthology.orchestra.run.smithi114.stdout: "enum_allowed": [ 2024-04-23T10:41:09.273 INFO:teuthology.orchestra.run.smithi114.stdout: "", 2024-04-23T10:41:09.273 INFO:teuthology.orchestra.run.smithi114.stdout: "critical", 2024-04-23T10:41:09.273 INFO:teuthology.orchestra.run.smithi114.stdout: "debug", 2024-04-23T10:41:09.273 INFO:teuthology.orchestra.run.smithi114.stdout: "error", 2024-04-23T10:41:09.273 INFO:teuthology.orchestra.run.smithi114.stdout: "info", 2024-04-23T10:41:09.274 INFO:teuthology.orchestra.run.smithi114.stdout: "warning" 2024-04-23T10:41:09.274 INFO:teuthology.orchestra.run.smithi114.stdout: ], 2024-04-23T10:41:09.274 INFO:teuthology.orchestra.run.smithi114.stdout: "desc": "", 2024-04-23T10:41:09.274 INFO:teuthology.orchestra.run.smithi114.stdout: "long_desc": "", 2024-04-23T10:41:09.274 INFO:teuthology.orchestra.run.smithi114.stdout: "tags": [], 2024-04-23T10:41:09.274 INFO:teuthology.orchestra.run.smithi114.stdout: "see_also": [] 2024-04-23T10:41:09.274 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-23T10:41:09.274 INFO:teuthology.orchestra.run.smithi114.stdout: "log_to_cluster": { 2024-04-23T10:41:09.275 INFO:teuthology.orchestra.run.smithi114.stdout: "name": "log_to_cluster", 2024-04-23T10:41:09.275 INFO:teuthology.orchestra.run.smithi114.stdout: "type": "bool", 2024-04-23T10:41:09.275 INFO:teuthology.orchestra.run.smithi114.stdout: "level": "advanced", 2024-04-23T10:41:09.275 INFO:teuthology.orchestra.run.smithi114.stdout: "flags": 1, 2024-04-23T10:41:09.275 INFO:teuthology.orchestra.run.smithi114.stdout: "default_value": "False", 2024-04-23T10:41:09.275 INFO:teuthology.orchestra.run.smithi114.stdout: "min": "", 2024-04-23T10:41:09.275 INFO:teuthology.orchestra.run.smithi114.stdout: "max": "", 2024-04-23T10:41:09.275 INFO:teuthology.orchestra.run.smithi114.stdout: "enum_allowed": [], 2024-04-23T10:41:09.275 INFO:teuthology.orchestra.run.smithi114.stdout: "desc": "", 2024-04-23T10:41:09.276 INFO:teuthology.orchestra.run.smithi114.stdout: "long_desc": "", 2024-04-23T10:41:09.276 INFO:teuthology.orchestra.run.smithi114.stdout: "tags": [], 2024-04-23T10:41:09.276 INFO:teuthology.orchestra.run.smithi114.stdout: "see_also": [] 2024-04-23T10:41:09.276 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-23T10:41:09.276 INFO:teuthology.orchestra.run.smithi114.stdout: "log_to_cluster_level": { 2024-04-23T10:41:09.276 INFO:teuthology.orchestra.run.smithi114.stdout: "name": "log_to_cluster_level", 2024-04-23T10:41:09.276 INFO:teuthology.orchestra.run.smithi114.stdout: "type": "str", 2024-04-23T10:41:09.276 INFO:teuthology.orchestra.run.smithi114.stdout: "level": "advanced", 2024-04-23T10:41:09.277 INFO:teuthology.orchestra.run.smithi114.stdout: "flags": 1, 2024-04-23T10:41:09.277 INFO:teuthology.orchestra.run.smithi114.stdout: "default_value": "info", 2024-04-23T10:41:09.277 INFO:teuthology.orchestra.run.smithi114.stdout: "min": "", 2024-04-23T10:41:09.277 INFO:teuthology.orchestra.run.smithi114.stdout: "max": "", 2024-04-23T10:41:09.277 INFO:teuthology.orchestra.run.smithi114.stdout: "enum_allowed": [ 2024-04-23T10:41:09.277 INFO:teuthology.orchestra.run.smithi114.stdout: "", 2024-04-23T10:41:09.277 INFO:teuthology.orchestra.run.smithi114.stdout: "critical", 2024-04-23T10:41:09.277 INFO:teuthology.orchestra.run.smithi114.stdout: "debug", 2024-04-23T10:41:09.278 INFO:teuthology.orchestra.run.smithi114.stdout: "error", 2024-04-23T10:41:09.278 INFO:teuthology.orchestra.run.smithi114.stdout: "info", 2024-04-23T10:41:09.278 INFO:teuthology.orchestra.run.smithi114.stdout: "warning" 2024-04-23T10:41:09.278 INFO:teuthology.orchestra.run.smithi114.stdout: ], 2024-04-23T10:41:09.278 INFO:teuthology.orchestra.run.smithi114.stdout: "desc": "", 2024-04-23T10:41:09.278 INFO:teuthology.orchestra.run.smithi114.stdout: "long_desc": "", 2024-04-23T10:41:09.278 INFO:teuthology.orchestra.run.smithi114.stdout: "tags": [], 2024-04-23T10:41:09.278 INFO:teuthology.orchestra.run.smithi114.stdout: "see_also": [] 2024-04-23T10:41:09.279 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-23T10:41:09.279 INFO:teuthology.orchestra.run.smithi114.stdout: "log_to_file": { 2024-04-23T10:41:09.279 INFO:teuthology.orchestra.run.smithi114.stdout: "name": "log_to_file", 2024-04-23T10:41:09.279 INFO:teuthology.orchestra.run.smithi114.stdout: "type": "bool", 2024-04-23T10:41:09.279 INFO:teuthology.orchestra.run.smithi114.stdout: "level": "advanced", 2024-04-23T10:41:09.279 INFO:teuthology.orchestra.run.smithi114.stdout: "flags": 1, 2024-04-23T10:41:09.279 INFO:teuthology.orchestra.run.smithi114.stdout: "default_value": "False", 2024-04-23T10:41:09.279 INFO:teuthology.orchestra.run.smithi114.stdout: "min": "", 2024-04-23T10:41:09.280 INFO:teuthology.orchestra.run.smithi114.stdout: "max": "", 2024-04-23T10:41:09.280 INFO:teuthology.orchestra.run.smithi114.stdout: "enum_allowed": [], 2024-04-23T10:41:09.280 INFO:teuthology.orchestra.run.smithi114.stdout: "desc": "", 2024-04-23T10:41:09.280 INFO:teuthology.orchestra.run.smithi114.stdout: "long_desc": "", 2024-04-23T10:41:09.280 INFO:teuthology.orchestra.run.smithi114.stdout: "tags": [], 2024-04-23T10:41:09.280 INFO:teuthology.orchestra.run.smithi114.stdout: "see_also": [] 2024-04-23T10:41:09.280 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-23T10:41:09.280 INFO:teuthology.orchestra.run.smithi114.stdout: "storage_class": { 2024-04-23T10:41:09.280 INFO:teuthology.orchestra.run.smithi114.stdout: "name": "storage_class", 2024-04-23T10:41:09.281 INFO:teuthology.orchestra.run.smithi114.stdout: "type": "str", 2024-04-23T10:41:09.281 INFO:teuthology.orchestra.run.smithi114.stdout: "level": "advanced", 2024-04-23T10:41:09.281 INFO:teuthology.orchestra.run.smithi114.stdout: "flags": 0, 2024-04-23T10:41:09.281 INFO:teuthology.orchestra.run.smithi114.stdout: "default_value": "local", 2024-04-23T10:41:09.281 INFO:teuthology.orchestra.run.smithi114.stdout: "min": "", 2024-04-23T10:41:09.281 INFO:teuthology.orchestra.run.smithi114.stdout: "max": "", 2024-04-23T10:41:09.281 INFO:teuthology.orchestra.run.smithi114.stdout: "enum_allowed": [], 2024-04-23T10:41:09.282 INFO:teuthology.orchestra.run.smithi114.stdout: "desc": "storage class name for LSO-discovered PVs", 2024-04-23T10:41:09.282 INFO:teuthology.orchestra.run.smithi114.stdout: "long_desc": "", 2024-04-23T10:41:09.282 INFO:teuthology.orchestra.run.smithi114.stdout: "tags": [], 2024-04-23T10:41:09.282 INFO:teuthology.orchestra.run.smithi114.stdout: "see_also": [] 2024-04-23T10:41:09.282 INFO:teuthology.orchestra.run.smithi114.stdout: } 2024-04-23T10:41:09.282 INFO:teuthology.orchestra.run.smithi114.stdout: } 2024-04-23T10:41:09.282 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-23T10:41:09.282 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-04-23T10:41:09.282 INFO:teuthology.orchestra.run.smithi114.stdout: "name": "selftest", 2024-04-23T10:41:09.283 INFO:teuthology.orchestra.run.smithi114.stdout: "can_run": true, 2024-04-23T10:41:09.283 INFO:teuthology.orchestra.run.smithi114.stdout: "error_string": "", 2024-04-23T10:41:09.283 INFO:teuthology.orchestra.run.smithi114.stdout: "module_options": { 2024-04-23T10:41:09.283 INFO:teuthology.orchestra.run.smithi114.stdout: "log_level": { 2024-04-23T10:41:09.283 INFO:teuthology.orchestra.run.smithi114.stdout: "name": "log_level", 2024-04-23T10:41:09.283 INFO:teuthology.orchestra.run.smithi114.stdout: "type": "str", 2024-04-23T10:41:09.283 INFO:teuthology.orchestra.run.smithi114.stdout: "level": "advanced", 2024-04-23T10:41:09.283 INFO:teuthology.orchestra.run.smithi114.stdout: "flags": 1, 2024-04-23T10:41:09.284 INFO:teuthology.orchestra.run.smithi114.stdout: "default_value": "", 2024-04-23T10:41:09.284 INFO:teuthology.orchestra.run.smithi114.stdout: "min": "", 2024-04-23T10:41:09.284 INFO:teuthology.orchestra.run.smithi114.stdout: "max": "", 2024-04-23T10:41:09.284 INFO:teuthology.orchestra.run.smithi114.stdout: "enum_allowed": [ 2024-04-23T10:41:09.284 INFO:teuthology.orchestra.run.smithi114.stdout: "", 2024-04-23T10:41:09.284 INFO:teuthology.orchestra.run.smithi114.stdout: "critical", 2024-04-23T10:41:09.284 INFO:teuthology.orchestra.run.smithi114.stdout: "debug", 2024-04-23T10:41:09.284 INFO:teuthology.orchestra.run.smithi114.stdout: "error", 2024-04-23T10:41:09.284 INFO:teuthology.orchestra.run.smithi114.stdout: "info", 2024-04-23T10:41:09.285 INFO:teuthology.orchestra.run.smithi114.stdout: "warning" 2024-04-23T10:41:09.285 INFO:teuthology.orchestra.run.smithi114.stdout: ], 2024-04-23T10:41:09.285 INFO:teuthology.orchestra.run.smithi114.stdout: "desc": "", 2024-04-23T10:41:09.285 INFO:teuthology.orchestra.run.smithi114.stdout: "long_desc": "", 2024-04-23T10:41:09.285 INFO:teuthology.orchestra.run.smithi114.stdout: "tags": [], 2024-04-23T10:41:09.285 INFO:teuthology.orchestra.run.smithi114.stdout: "see_also": [] 2024-04-23T10:41:09.285 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-23T10:41:09.285 INFO:teuthology.orchestra.run.smithi114.stdout: "log_to_cluster": { 2024-04-23T10:41:09.286 INFO:teuthology.orchestra.run.smithi114.stdout: "name": "log_to_cluster", 2024-04-23T10:41:09.286 INFO:teuthology.orchestra.run.smithi114.stdout: "type": "bool", 2024-04-23T10:41:09.286 INFO:teuthology.orchestra.run.smithi114.stdout: "level": "advanced", 2024-04-23T10:41:09.286 INFO:teuthology.orchestra.run.smithi114.stdout: "flags": 1, 2024-04-23T10:41:09.286 INFO:teuthology.orchestra.run.smithi114.stdout: "default_value": "False", 2024-04-23T10:41:09.286 INFO:teuthology.orchestra.run.smithi114.stdout: "min": "", 2024-04-23T10:41:09.286 INFO:teuthology.orchestra.run.smithi114.stdout: "max": "", 2024-04-23T10:41:09.286 INFO:teuthology.orchestra.run.smithi114.stdout: "enum_allowed": [], 2024-04-23T10:41:09.286 INFO:teuthology.orchestra.run.smithi114.stdout: "desc": "", 2024-04-23T10:41:09.287 INFO:teuthology.orchestra.run.smithi114.stdout: "long_desc": "", 2024-04-23T10:41:09.287 INFO:teuthology.orchestra.run.smithi114.stdout: "tags": [], 2024-04-23T10:41:09.287 INFO:teuthology.orchestra.run.smithi114.stdout: "see_also": [] 2024-04-23T10:41:09.287 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-23T10:41:09.287 INFO:teuthology.orchestra.run.smithi114.stdout: "log_to_cluster_level": { 2024-04-23T10:41:09.287 INFO:teuthology.orchestra.run.smithi114.stdout: "name": "log_to_cluster_level", 2024-04-23T10:41:09.287 INFO:teuthology.orchestra.run.smithi114.stdout: "type": "str", 2024-04-23T10:41:09.287 INFO:teuthology.orchestra.run.smithi114.stdout: "level": "advanced", 2024-04-23T10:41:09.287 INFO:teuthology.orchestra.run.smithi114.stdout: "flags": 1, 2024-04-23T10:41:09.288 INFO:teuthology.orchestra.run.smithi114.stdout: "default_value": "info", 2024-04-23T10:41:09.288 INFO:teuthology.orchestra.run.smithi114.stdout: "min": "", 2024-04-23T10:41:09.288 INFO:teuthology.orchestra.run.smithi114.stdout: "max": "", 2024-04-23T10:41:09.288 INFO:teuthology.orchestra.run.smithi114.stdout: "enum_allowed": [ 2024-04-23T10:41:09.288 INFO:teuthology.orchestra.run.smithi114.stdout: "", 2024-04-23T10:41:09.288 INFO:teuthology.orchestra.run.smithi114.stdout: "critical", 2024-04-23T10:41:09.288 INFO:teuthology.orchestra.run.smithi114.stdout: "debug", 2024-04-23T10:41:09.288 INFO:teuthology.orchestra.run.smithi114.stdout: "error", 2024-04-23T10:41:09.289 INFO:teuthology.orchestra.run.smithi114.stdout: "info", 2024-04-23T10:41:09.289 INFO:teuthology.orchestra.run.smithi114.stdout: "warning" 2024-04-23T10:41:09.289 INFO:teuthology.orchestra.run.smithi114.stdout: ], 2024-04-23T10:41:09.289 INFO:teuthology.orchestra.run.smithi114.stdout: "desc": "", 2024-04-23T10:41:09.289 INFO:teuthology.orchestra.run.smithi114.stdout: "long_desc": "", 2024-04-23T10:41:09.289 INFO:teuthology.orchestra.run.smithi114.stdout: "tags": [], 2024-04-23T10:41:09.289 INFO:teuthology.orchestra.run.smithi114.stdout: "see_also": [] 2024-04-23T10:41:09.289 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-23T10:41:09.289 INFO:teuthology.orchestra.run.smithi114.stdout: "log_to_file": { 2024-04-23T10:41:09.290 INFO:teuthology.orchestra.run.smithi114.stdout: "name": "log_to_file", 2024-04-23T10:41:09.290 INFO:teuthology.orchestra.run.smithi114.stdout: "type": "bool", 2024-04-23T10:41:09.290 INFO:teuthology.orchestra.run.smithi114.stdout: "level": "advanced", 2024-04-23T10:41:09.290 INFO:teuthology.orchestra.run.smithi114.stdout: "flags": 1, 2024-04-23T10:41:09.290 INFO:teuthology.orchestra.run.smithi114.stdout: "default_value": "False", 2024-04-23T10:41:09.290 INFO:teuthology.orchestra.run.smithi114.stdout: "min": "", 2024-04-23T10:41:09.290 INFO:teuthology.orchestra.run.smithi114.stdout: "max": "", 2024-04-23T10:41:09.291 INFO:teuthology.orchestra.run.smithi114.stdout: "enum_allowed": [], 2024-04-23T10:41:09.291 INFO:teuthology.orchestra.run.smithi114.stdout: "desc": "", 2024-04-23T10:41:09.291 INFO:teuthology.orchestra.run.smithi114.stdout: "long_desc": "", 2024-04-23T10:41:09.291 INFO:teuthology.orchestra.run.smithi114.stdout: "tags": [], 2024-04-23T10:41:09.291 INFO:teuthology.orchestra.run.smithi114.stdout: "see_also": [] 2024-04-23T10:41:09.291 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-23T10:41:09.291 INFO:teuthology.orchestra.run.smithi114.stdout: "roption1": { 2024-04-23T10:41:09.291 INFO:teuthology.orchestra.run.smithi114.stdout: "name": "roption1", 2024-04-23T10:41:09.292 INFO:teuthology.orchestra.run.smithi114.stdout: "type": "str", 2024-04-23T10:41:09.292 INFO:teuthology.orchestra.run.smithi114.stdout: "level": "advanced", 2024-04-23T10:41:09.292 INFO:teuthology.orchestra.run.smithi114.stdout: "flags": 0, 2024-04-23T10:41:09.292 INFO:teuthology.orchestra.run.smithi114.stdout: "default_value": "", 2024-04-23T10:41:09.292 INFO:teuthology.orchestra.run.smithi114.stdout: "min": "", 2024-04-23T10:41:09.292 INFO:teuthology.orchestra.run.smithi114.stdout: "max": "", 2024-04-23T10:41:09.292 INFO:teuthology.orchestra.run.smithi114.stdout: "enum_allowed": [], 2024-04-23T10:41:09.292 INFO:teuthology.orchestra.run.smithi114.stdout: "desc": "", 2024-04-23T10:41:09.293 INFO:teuthology.orchestra.run.smithi114.stdout: "long_desc": "", 2024-04-23T10:41:09.293 INFO:teuthology.orchestra.run.smithi114.stdout: "tags": [], 2024-04-23T10:41:09.293 INFO:teuthology.orchestra.run.smithi114.stdout: "see_also": [] 2024-04-23T10:41:09.293 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-23T10:41:09.293 INFO:teuthology.orchestra.run.smithi114.stdout: "roption2": { 2024-04-23T10:41:09.293 INFO:teuthology.orchestra.run.smithi114.stdout: "name": "roption2", 2024-04-23T10:41:09.293 INFO:teuthology.orchestra.run.smithi114.stdout: "type": "str", 2024-04-23T10:41:09.293 INFO:teuthology.orchestra.run.smithi114.stdout: "level": "advanced", 2024-04-23T10:41:09.294 INFO:teuthology.orchestra.run.smithi114.stdout: "flags": 0, 2024-04-23T10:41:09.294 INFO:teuthology.orchestra.run.smithi114.stdout: "default_value": "xyz", 2024-04-23T10:41:09.294 INFO:teuthology.orchestra.run.smithi114.stdout: "min": "", 2024-04-23T10:41:09.294 INFO:teuthology.orchestra.run.smithi114.stdout: "max": "", 2024-04-23T10:41:09.294 INFO:teuthology.orchestra.run.smithi114.stdout: "enum_allowed": [], 2024-04-23T10:41:09.294 INFO:teuthology.orchestra.run.smithi114.stdout: "desc": "", 2024-04-23T10:41:09.294 INFO:teuthology.orchestra.run.smithi114.stdout: "long_desc": "", 2024-04-23T10:41:09.294 INFO:teuthology.orchestra.run.smithi114.stdout: "tags": [], 2024-04-23T10:41:09.295 INFO:teuthology.orchestra.run.smithi114.stdout: "see_also": [] 2024-04-23T10:41:09.295 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-23T10:41:09.295 INFO:teuthology.orchestra.run.smithi114.stdout: "rwoption1": { 2024-04-23T10:41:09.295 INFO:teuthology.orchestra.run.smithi114.stdout: "name": "rwoption1", 2024-04-23T10:41:09.295 INFO:teuthology.orchestra.run.smithi114.stdout: "type": "str", 2024-04-23T10:41:09.295 INFO:teuthology.orchestra.run.smithi114.stdout: "level": "advanced", 2024-04-23T10:41:09.295 INFO:teuthology.orchestra.run.smithi114.stdout: "flags": 0, 2024-04-23T10:41:09.295 INFO:teuthology.orchestra.run.smithi114.stdout: "default_value": "", 2024-04-23T10:41:09.296 INFO:teuthology.orchestra.run.smithi114.stdout: "min": "", 2024-04-23T10:41:09.296 INFO:teuthology.orchestra.run.smithi114.stdout: "max": "", 2024-04-23T10:41:09.296 INFO:teuthology.orchestra.run.smithi114.stdout: "enum_allowed": [], 2024-04-23T10:41:09.296 INFO:teuthology.orchestra.run.smithi114.stdout: "desc": "", 2024-04-23T10:41:09.296 INFO:teuthology.orchestra.run.smithi114.stdout: "long_desc": "", 2024-04-23T10:41:09.296 INFO:teuthology.orchestra.run.smithi114.stdout: "tags": [], 2024-04-23T10:41:09.296 INFO:teuthology.orchestra.run.smithi114.stdout: "see_also": [] 2024-04-23T10:41:09.296 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-23T10:41:09.296 INFO:teuthology.orchestra.run.smithi114.stdout: "rwoption2": { 2024-04-23T10:41:09.297 INFO:teuthology.orchestra.run.smithi114.stdout: "name": "rwoption2", 2024-04-23T10:41:09.297 INFO:teuthology.orchestra.run.smithi114.stdout: "type": "int", 2024-04-23T10:41:09.297 INFO:teuthology.orchestra.run.smithi114.stdout: "level": "advanced", 2024-04-23T10:41:09.297 INFO:teuthology.orchestra.run.smithi114.stdout: "flags": 0, 2024-04-23T10:41:09.297 INFO:teuthology.orchestra.run.smithi114.stdout: "default_value": "", 2024-04-23T10:41:09.297 INFO:teuthology.orchestra.run.smithi114.stdout: "min": "", 2024-04-23T10:41:09.297 INFO:teuthology.orchestra.run.smithi114.stdout: "max": "", 2024-04-23T10:41:09.297 INFO:teuthology.orchestra.run.smithi114.stdout: "enum_allowed": [], 2024-04-23T10:41:09.298 INFO:teuthology.orchestra.run.smithi114.stdout: "desc": "", 2024-04-23T10:41:09.298 INFO:teuthology.orchestra.run.smithi114.stdout: "long_desc": "", 2024-04-23T10:41:09.298 INFO:teuthology.orchestra.run.smithi114.stdout: "tags": [], 2024-04-23T10:41:09.298 INFO:teuthology.orchestra.run.smithi114.stdout: "see_also": [] 2024-04-23T10:41:09.298 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-23T10:41:09.298 INFO:teuthology.orchestra.run.smithi114.stdout: "rwoption3": { 2024-04-23T10:41:09.298 INFO:teuthology.orchestra.run.smithi114.stdout: "name": "rwoption3", 2024-04-23T10:41:09.298 INFO:teuthology.orchestra.run.smithi114.stdout: "type": "float", 2024-04-23T10:41:09.298 INFO:teuthology.orchestra.run.smithi114.stdout: "level": "advanced", 2024-04-23T10:41:09.299 INFO:teuthology.orchestra.run.smithi114.stdout: "flags": 0, 2024-04-23T10:41:09.299 INFO:teuthology.orchestra.run.smithi114.stdout: "default_value": "", 2024-04-23T10:41:09.299 INFO:teuthology.orchestra.run.smithi114.stdout: "min": "", 2024-04-23T10:41:09.299 INFO:teuthology.orchestra.run.smithi114.stdout: "max": "", 2024-04-23T10:41:09.299 INFO:teuthology.orchestra.run.smithi114.stdout: "enum_allowed": [], 2024-04-23T10:41:09.299 INFO:teuthology.orchestra.run.smithi114.stdout: "desc": "", 2024-04-23T10:41:09.299 INFO:teuthology.orchestra.run.smithi114.stdout: "long_desc": "", 2024-04-23T10:41:09.299 INFO:teuthology.orchestra.run.smithi114.stdout: "tags": [], 2024-04-23T10:41:09.300 INFO:teuthology.orchestra.run.smithi114.stdout: "see_also": [] 2024-04-23T10:41:09.300 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-23T10:41:09.300 INFO:teuthology.orchestra.run.smithi114.stdout: "rwoption4": { 2024-04-23T10:41:09.300 INFO:teuthology.orchestra.run.smithi114.stdout: "name": "rwoption4", 2024-04-23T10:41:09.300 INFO:teuthology.orchestra.run.smithi114.stdout: "type": "str", 2024-04-23T10:41:09.300 INFO:teuthology.orchestra.run.smithi114.stdout: "level": "advanced", 2024-04-23T10:41:09.300 INFO:teuthology.orchestra.run.smithi114.stdout: "flags": 0, 2024-04-23T10:41:09.300 INFO:teuthology.orchestra.run.smithi114.stdout: "default_value": "", 2024-04-23T10:41:09.301 INFO:teuthology.orchestra.run.smithi114.stdout: "min": "", 2024-04-23T10:41:09.301 INFO:teuthology.orchestra.run.smithi114.stdout: "max": "", 2024-04-23T10:41:09.301 INFO:teuthology.orchestra.run.smithi114.stdout: "enum_allowed": [], 2024-04-23T10:41:09.301 INFO:teuthology.orchestra.run.smithi114.stdout: "desc": "", 2024-04-23T10:41:09.301 INFO:teuthology.orchestra.run.smithi114.stdout: "long_desc": "", 2024-04-23T10:41:09.301 INFO:teuthology.orchestra.run.smithi114.stdout: "tags": [], 2024-04-23T10:41:09.301 INFO:teuthology.orchestra.run.smithi114.stdout: "see_also": [] 2024-04-23T10:41:09.301 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-23T10:41:09.302 INFO:teuthology.orchestra.run.smithi114.stdout: "rwoption5": { 2024-04-23T10:41:09.302 INFO:teuthology.orchestra.run.smithi114.stdout: "name": "rwoption5", 2024-04-23T10:41:09.302 INFO:teuthology.orchestra.run.smithi114.stdout: "type": "bool", 2024-04-23T10:41:09.302 INFO:teuthology.orchestra.run.smithi114.stdout: "level": "advanced", 2024-04-23T10:41:09.302 INFO:teuthology.orchestra.run.smithi114.stdout: "flags": 0, 2024-04-23T10:41:09.302 INFO:teuthology.orchestra.run.smithi114.stdout: "default_value": "", 2024-04-23T10:41:09.302 INFO:teuthology.orchestra.run.smithi114.stdout: "min": "", 2024-04-23T10:41:09.302 INFO:teuthology.orchestra.run.smithi114.stdout: "max": "", 2024-04-23T10:41:09.303 INFO:teuthology.orchestra.run.smithi114.stdout: "enum_allowed": [], 2024-04-23T10:41:09.303 INFO:teuthology.orchestra.run.smithi114.stdout: "desc": "", 2024-04-23T10:41:09.303 INFO:teuthology.orchestra.run.smithi114.stdout: "long_desc": "", 2024-04-23T10:41:09.303 INFO:teuthology.orchestra.run.smithi114.stdout: "tags": [], 2024-04-23T10:41:09.303 INFO:teuthology.orchestra.run.smithi114.stdout: "see_also": [] 2024-04-23T10:41:09.303 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-23T10:41:09.303 INFO:teuthology.orchestra.run.smithi114.stdout: "rwoption6": { 2024-04-23T10:41:09.303 INFO:teuthology.orchestra.run.smithi114.stdout: "name": "rwoption6", 2024-04-23T10:41:09.303 INFO:teuthology.orchestra.run.smithi114.stdout: "type": "bool", 2024-04-23T10:41:09.304 INFO:teuthology.orchestra.run.smithi114.stdout: "level": "advanced", 2024-04-23T10:41:09.304 INFO:teuthology.orchestra.run.smithi114.stdout: "flags": 0, 2024-04-23T10:41:09.304 INFO:teuthology.orchestra.run.smithi114.stdout: "default_value": "True", 2024-04-23T10:41:09.304 INFO:teuthology.orchestra.run.smithi114.stdout: "min": "", 2024-04-23T10:41:09.304 INFO:teuthology.orchestra.run.smithi114.stdout: "max": "", 2024-04-23T10:41:09.304 INFO:teuthology.orchestra.run.smithi114.stdout: "enum_allowed": [], 2024-04-23T10:41:09.304 INFO:teuthology.orchestra.run.smithi114.stdout: "desc": "", 2024-04-23T10:41:09.304 INFO:teuthology.orchestra.run.smithi114.stdout: "long_desc": "", 2024-04-23T10:41:09.305 INFO:teuthology.orchestra.run.smithi114.stdout: "tags": [], 2024-04-23T10:41:09.305 INFO:teuthology.orchestra.run.smithi114.stdout: "see_also": [] 2024-04-23T10:41:09.305 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-23T10:41:09.305 INFO:teuthology.orchestra.run.smithi114.stdout: "rwoption7": { 2024-04-23T10:41:09.305 INFO:teuthology.orchestra.run.smithi114.stdout: "name": "rwoption7", 2024-04-23T10:41:09.305 INFO:teuthology.orchestra.run.smithi114.stdout: "type": "int", 2024-04-23T10:41:09.305 INFO:teuthology.orchestra.run.smithi114.stdout: "level": "advanced", 2024-04-23T10:41:09.305 INFO:teuthology.orchestra.run.smithi114.stdout: "flags": 0, 2024-04-23T10:41:09.305 INFO:teuthology.orchestra.run.smithi114.stdout: "default_value": "", 2024-04-23T10:41:09.306 INFO:teuthology.orchestra.run.smithi114.stdout: "min": "1", 2024-04-23T10:41:09.306 INFO:teuthology.orchestra.run.smithi114.stdout: "max": "42", 2024-04-23T10:41:09.306 INFO:teuthology.orchestra.run.smithi114.stdout: "enum_allowed": [], 2024-04-23T10:41:09.306 INFO:teuthology.orchestra.run.smithi114.stdout: "desc": "", 2024-04-23T10:41:09.306 INFO:teuthology.orchestra.run.smithi114.stdout: "long_desc": "", 2024-04-23T10:41:09.306 INFO:teuthology.orchestra.run.smithi114.stdout: "tags": [], 2024-04-23T10:41:09.306 INFO:teuthology.orchestra.run.smithi114.stdout: "see_also": [] 2024-04-23T10:41:09.306 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-23T10:41:09.306 INFO:teuthology.orchestra.run.smithi114.stdout: "testkey": { 2024-04-23T10:41:09.307 INFO:teuthology.orchestra.run.smithi114.stdout: "name": "testkey", 2024-04-23T10:41:09.307 INFO:teuthology.orchestra.run.smithi114.stdout: "type": "str", 2024-04-23T10:41:09.307 INFO:teuthology.orchestra.run.smithi114.stdout: "level": "advanced", 2024-04-23T10:41:09.307 INFO:teuthology.orchestra.run.smithi114.stdout: "flags": 0, 2024-04-23T10:41:09.307 INFO:teuthology.orchestra.run.smithi114.stdout: "default_value": "", 2024-04-23T10:41:09.307 INFO:teuthology.orchestra.run.smithi114.stdout: "min": "", 2024-04-23T10:41:09.307 INFO:teuthology.orchestra.run.smithi114.stdout: "max": "", 2024-04-23T10:41:09.307 INFO:teuthology.orchestra.run.smithi114.stdout: "enum_allowed": [], 2024-04-23T10:41:09.307 INFO:teuthology.orchestra.run.smithi114.stdout: "desc": "", 2024-04-23T10:41:09.308 INFO:teuthology.orchestra.run.smithi114.stdout: "long_desc": "", 2024-04-23T10:41:09.308 INFO:teuthology.orchestra.run.smithi114.stdout: "tags": [], 2024-04-23T10:41:09.308 INFO:teuthology.orchestra.run.smithi114.stdout: "see_also": [] 2024-04-23T10:41:09.308 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-23T10:41:09.308 INFO:teuthology.orchestra.run.smithi114.stdout: "testlkey": { 2024-04-23T10:41:09.308 INFO:teuthology.orchestra.run.smithi114.stdout: "name": "testlkey", 2024-04-23T10:41:09.308 INFO:teuthology.orchestra.run.smithi114.stdout: "type": "str", 2024-04-23T10:41:09.308 INFO:teuthology.orchestra.run.smithi114.stdout: "level": "advanced", 2024-04-23T10:41:09.309 INFO:teuthology.orchestra.run.smithi114.stdout: "flags": 0, 2024-04-23T10:41:09.309 INFO:teuthology.orchestra.run.smithi114.stdout: "default_value": "", 2024-04-23T10:41:09.309 INFO:teuthology.orchestra.run.smithi114.stdout: "min": "", 2024-04-23T10:41:09.309 INFO:teuthology.orchestra.run.smithi114.stdout: "max": "", 2024-04-23T10:41:09.309 INFO:teuthology.orchestra.run.smithi114.stdout: "enum_allowed": [], 2024-04-23T10:41:09.309 INFO:teuthology.orchestra.run.smithi114.stdout: "desc": "", 2024-04-23T10:41:09.309 INFO:teuthology.orchestra.run.smithi114.stdout: "long_desc": "", 2024-04-23T10:41:09.309 INFO:teuthology.orchestra.run.smithi114.stdout: "tags": [], 2024-04-23T10:41:09.309 INFO:teuthology.orchestra.run.smithi114.stdout: "see_also": [] 2024-04-23T10:41:09.310 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-23T10:41:09.310 INFO:teuthology.orchestra.run.smithi114.stdout: "testnewline": { 2024-04-23T10:41:09.310 INFO:teuthology.orchestra.run.smithi114.stdout: "name": "testnewline", 2024-04-23T10:41:09.310 INFO:teuthology.orchestra.run.smithi114.stdout: "type": "str", 2024-04-23T10:41:09.310 INFO:teuthology.orchestra.run.smithi114.stdout: "level": "advanced", 2024-04-23T10:41:09.310 INFO:teuthology.orchestra.run.smithi114.stdout: "flags": 0, 2024-04-23T10:41:09.310 INFO:teuthology.orchestra.run.smithi114.stdout: "default_value": "", 2024-04-23T10:41:09.310 INFO:teuthology.orchestra.run.smithi114.stdout: "min": "", 2024-04-23T10:41:09.311 INFO:teuthology.orchestra.run.smithi114.stdout: "max": "", 2024-04-23T10:41:09.311 INFO:teuthology.orchestra.run.smithi114.stdout: "enum_allowed": [], 2024-04-23T10:41:09.311 INFO:teuthology.orchestra.run.smithi114.stdout: "desc": "", 2024-04-23T10:41:09.311 INFO:teuthology.orchestra.run.smithi114.stdout: "long_desc": "", 2024-04-23T10:41:09.311 INFO:teuthology.orchestra.run.smithi114.stdout: "tags": [], 2024-04-23T10:41:09.311 INFO:teuthology.orchestra.run.smithi114.stdout: "see_also": [] 2024-04-23T10:41:09.311 INFO:teuthology.orchestra.run.smithi114.stdout: } 2024-04-23T10:41:09.311 INFO:teuthology.orchestra.run.smithi114.stdout: } 2024-04-23T10:41:09.311 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-23T10:41:09.312 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-04-23T10:41:09.312 INFO:teuthology.orchestra.run.smithi114.stdout: "name": "snap_schedule", 2024-04-23T10:41:09.312 INFO:teuthology.orchestra.run.smithi114.stdout: "can_run": true, 2024-04-23T10:41:09.312 INFO:teuthology.orchestra.run.smithi114.stdout: "error_string": "", 2024-04-23T10:41:09.312 INFO:teuthology.orchestra.run.smithi114.stdout: "module_options": { 2024-04-23T10:41:09.312 INFO:teuthology.orchestra.run.smithi114.stdout: "allow_m_granularity": { 2024-04-23T10:41:09.312 INFO:teuthology.orchestra.run.smithi114.stdout: "name": "allow_m_granularity", 2024-04-23T10:41:09.312 INFO:teuthology.orchestra.run.smithi114.stdout: "type": "bool", 2024-04-23T10:41:09.313 INFO:teuthology.orchestra.run.smithi114.stdout: "level": "advanced", 2024-04-23T10:41:09.313 INFO:teuthology.orchestra.run.smithi114.stdout: "flags": 1, 2024-04-23T10:41:09.313 INFO:teuthology.orchestra.run.smithi114.stdout: "default_value": "False", 2024-04-23T10:41:09.313 INFO:teuthology.orchestra.run.smithi114.stdout: "min": "", 2024-04-23T10:41:09.313 INFO:teuthology.orchestra.run.smithi114.stdout: "max": "", 2024-04-23T10:41:09.313 INFO:teuthology.orchestra.run.smithi114.stdout: "enum_allowed": [], 2024-04-23T10:41:09.313 INFO:teuthology.orchestra.run.smithi114.stdout: "desc": "allow minute scheduled snapshots", 2024-04-23T10:41:09.313 INFO:teuthology.orchestra.run.smithi114.stdout: "long_desc": "", 2024-04-23T10:41:09.314 INFO:teuthology.orchestra.run.smithi114.stdout: "tags": [], 2024-04-23T10:41:09.314 INFO:teuthology.orchestra.run.smithi114.stdout: "see_also": [] 2024-04-23T10:41:09.314 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-23T10:41:09.314 INFO:teuthology.orchestra.run.smithi114.stdout: "dump_on_update": { 2024-04-23T10:41:09.314 INFO:teuthology.orchestra.run.smithi114.stdout: "name": "dump_on_update", 2024-04-23T10:41:09.314 INFO:teuthology.orchestra.run.smithi114.stdout: "type": "bool", 2024-04-23T10:41:09.314 INFO:teuthology.orchestra.run.smithi114.stdout: "level": "advanced", 2024-04-23T10:41:09.314 INFO:teuthology.orchestra.run.smithi114.stdout: "flags": 1, 2024-04-23T10:41:09.314 INFO:teuthology.orchestra.run.smithi114.stdout: "default_value": "False", 2024-04-23T10:41:09.315 INFO:teuthology.orchestra.run.smithi114.stdout: "min": "", 2024-04-23T10:41:09.315 INFO:teuthology.orchestra.run.smithi114.stdout: "max": "", 2024-04-23T10:41:09.315 INFO:teuthology.orchestra.run.smithi114.stdout: "enum_allowed": [], 2024-04-23T10:41:09.315 INFO:teuthology.orchestra.run.smithi114.stdout: "desc": "dump database to debug log on update", 2024-04-23T10:41:09.315 INFO:teuthology.orchestra.run.smithi114.stdout: "long_desc": "", 2024-04-23T10:41:09.315 INFO:teuthology.orchestra.run.smithi114.stdout: "tags": [], 2024-04-23T10:41:09.315 INFO:teuthology.orchestra.run.smithi114.stdout: "see_also": [] 2024-04-23T10:41:09.315 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-23T10:41:09.315 INFO:teuthology.orchestra.run.smithi114.stdout: "log_level": { 2024-04-23T10:41:09.316 INFO:teuthology.orchestra.run.smithi114.stdout: "name": "log_level", 2024-04-23T10:41:09.316 INFO:teuthology.orchestra.run.smithi114.stdout: "type": "str", 2024-04-23T10:41:09.316 INFO:teuthology.orchestra.run.smithi114.stdout: "level": "advanced", 2024-04-23T10:41:09.316 INFO:teuthology.orchestra.run.smithi114.stdout: "flags": 1, 2024-04-23T10:41:09.316 INFO:teuthology.orchestra.run.smithi114.stdout: "default_value": "", 2024-04-23T10:41:09.316 INFO:teuthology.orchestra.run.smithi114.stdout: "min": "", 2024-04-23T10:41:09.316 INFO:teuthology.orchestra.run.smithi114.stdout: "max": "", 2024-04-23T10:41:09.316 INFO:teuthology.orchestra.run.smithi114.stdout: "enum_allowed": [ 2024-04-23T10:41:09.317 INFO:teuthology.orchestra.run.smithi114.stdout: "", 2024-04-23T10:41:09.317 INFO:teuthology.orchestra.run.smithi114.stdout: "critical", 2024-04-23T10:41:09.317 INFO:teuthology.orchestra.run.smithi114.stdout: "debug", 2024-04-23T10:41:09.317 INFO:teuthology.orchestra.run.smithi114.stdout: "error", 2024-04-23T10:41:09.317 INFO:teuthology.orchestra.run.smithi114.stdout: "info", 2024-04-23T10:41:09.317 INFO:teuthology.orchestra.run.smithi114.stdout: "warning" 2024-04-23T10:41:09.317 INFO:teuthology.orchestra.run.smithi114.stdout: ], 2024-04-23T10:41:09.317 INFO:teuthology.orchestra.run.smithi114.stdout: "desc": "", 2024-04-23T10:41:09.318 INFO:teuthology.orchestra.run.smithi114.stdout: "long_desc": "", 2024-04-23T10:41:09.318 INFO:teuthology.orchestra.run.smithi114.stdout: "tags": [], 2024-04-23T10:41:09.318 INFO:teuthology.orchestra.run.smithi114.stdout: "see_also": [] 2024-04-23T10:41:09.318 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-23T10:41:09.318 INFO:teuthology.orchestra.run.smithi114.stdout: "log_to_cluster": { 2024-04-23T10:41:09.318 INFO:teuthology.orchestra.run.smithi114.stdout: "name": "log_to_cluster", 2024-04-23T10:41:09.318 INFO:teuthology.orchestra.run.smithi114.stdout: "type": "bool", 2024-04-23T10:41:09.318 INFO:teuthology.orchestra.run.smithi114.stdout: "level": "advanced", 2024-04-23T10:41:09.318 INFO:teuthology.orchestra.run.smithi114.stdout: "flags": 1, 2024-04-23T10:41:09.319 INFO:teuthology.orchestra.run.smithi114.stdout: "default_value": "False", 2024-04-23T10:41:09.319 INFO:teuthology.orchestra.run.smithi114.stdout: "min": "", 2024-04-23T10:41:09.319 INFO:teuthology.orchestra.run.smithi114.stdout: "max": "", 2024-04-23T10:41:09.319 INFO:teuthology.orchestra.run.smithi114.stdout: "enum_allowed": [], 2024-04-23T10:41:09.319 INFO:teuthology.orchestra.run.smithi114.stdout: "desc": "", 2024-04-23T10:41:09.319 INFO:teuthology.orchestra.run.smithi114.stdout: "long_desc": "", 2024-04-23T10:41:09.319 INFO:teuthology.orchestra.run.smithi114.stdout: "tags": [], 2024-04-23T10:41:09.319 INFO:teuthology.orchestra.run.smithi114.stdout: "see_also": [] 2024-04-23T10:41:09.319 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-23T10:41:09.320 INFO:teuthology.orchestra.run.smithi114.stdout: "log_to_cluster_level": { 2024-04-23T10:41:09.320 INFO:teuthology.orchestra.run.smithi114.stdout: "name": "log_to_cluster_level", 2024-04-23T10:41:09.320 INFO:teuthology.orchestra.run.smithi114.stdout: "type": "str", 2024-04-23T10:41:09.320 INFO:teuthology.orchestra.run.smithi114.stdout: "level": "advanced", 2024-04-23T10:41:09.320 INFO:teuthology.orchestra.run.smithi114.stdout: "flags": 1, 2024-04-23T10:41:09.320 INFO:teuthology.orchestra.run.smithi114.stdout: "default_value": "info", 2024-04-23T10:41:09.320 INFO:teuthology.orchestra.run.smithi114.stdout: "min": "", 2024-04-23T10:41:09.320 INFO:teuthology.orchestra.run.smithi114.stdout: "max": "", 2024-04-23T10:41:09.320 INFO:teuthology.orchestra.run.smithi114.stdout: "enum_allowed": [ 2024-04-23T10:41:09.321 INFO:teuthology.orchestra.run.smithi114.stdout: "", 2024-04-23T10:41:09.321 INFO:teuthology.orchestra.run.smithi114.stdout: "critical", 2024-04-23T10:41:09.321 INFO:teuthology.orchestra.run.smithi114.stdout: "debug", 2024-04-23T10:41:09.321 INFO:teuthology.orchestra.run.smithi114.stdout: "error", 2024-04-23T10:41:09.321 INFO:teuthology.orchestra.run.smithi114.stdout: "info", 2024-04-23T10:41:09.321 INFO:teuthology.orchestra.run.smithi114.stdout: "warning" 2024-04-23T10:41:09.321 INFO:teuthology.orchestra.run.smithi114.stdout: ], 2024-04-23T10:41:09.321 INFO:teuthology.orchestra.run.smithi114.stdout: "desc": "", 2024-04-23T10:41:09.321 INFO:teuthology.orchestra.run.smithi114.stdout: "long_desc": "", 2024-04-23T10:41:09.322 INFO:teuthology.orchestra.run.smithi114.stdout: "tags": [], 2024-04-23T10:41:09.322 INFO:teuthology.orchestra.run.smithi114.stdout: "see_also": [] 2024-04-23T10:41:09.322 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-23T10:41:09.322 INFO:teuthology.orchestra.run.smithi114.stdout: "log_to_file": { 2024-04-23T10:41:09.322 INFO:teuthology.orchestra.run.smithi114.stdout: "name": "log_to_file", 2024-04-23T10:41:09.322 INFO:teuthology.orchestra.run.smithi114.stdout: "type": "bool", 2024-04-23T10:41:09.322 INFO:teuthology.orchestra.run.smithi114.stdout: "level": "advanced", 2024-04-23T10:41:09.322 INFO:teuthology.orchestra.run.smithi114.stdout: "flags": 1, 2024-04-23T10:41:09.323 INFO:teuthology.orchestra.run.smithi114.stdout: "default_value": "False", 2024-04-23T10:41:09.323 INFO:teuthology.orchestra.run.smithi114.stdout: "min": "", 2024-04-23T10:41:09.323 INFO:teuthology.orchestra.run.smithi114.stdout: "max": "", 2024-04-23T10:41:09.323 INFO:teuthology.orchestra.run.smithi114.stdout: "enum_allowed": [], 2024-04-23T10:41:09.323 INFO:teuthology.orchestra.run.smithi114.stdout: "desc": "", 2024-04-23T10:41:09.323 INFO:teuthology.orchestra.run.smithi114.stdout: "long_desc": "", 2024-04-23T10:41:09.323 INFO:teuthology.orchestra.run.smithi114.stdout: "tags": [], 2024-04-23T10:41:09.323 INFO:teuthology.orchestra.run.smithi114.stdout: "see_also": [] 2024-04-23T10:41:09.323 INFO:teuthology.orchestra.run.smithi114.stdout: } 2024-04-23T10:41:09.324 INFO:teuthology.orchestra.run.smithi114.stdout: } 2024-04-23T10:41:09.324 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-23T10:41:09.324 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-04-23T10:41:09.324 INFO:teuthology.orchestra.run.smithi114.stdout: "name": "stats", 2024-04-23T10:41:09.324 INFO:teuthology.orchestra.run.smithi114.stdout: "can_run": true, 2024-04-23T10:41:09.324 INFO:teuthology.orchestra.run.smithi114.stdout: "error_string": "", 2024-04-23T10:41:09.324 INFO:teuthology.orchestra.run.smithi114.stdout: "module_options": { 2024-04-23T10:41:09.324 INFO:teuthology.orchestra.run.smithi114.stdout: "log_level": { 2024-04-23T10:41:09.325 INFO:teuthology.orchestra.run.smithi114.stdout: "name": "log_level", 2024-04-23T10:41:09.325 INFO:teuthology.orchestra.run.smithi114.stdout: "type": "str", 2024-04-23T10:41:09.325 INFO:teuthology.orchestra.run.smithi114.stdout: "level": "advanced", 2024-04-23T10:41:09.325 INFO:teuthology.orchestra.run.smithi114.stdout: "flags": 1, 2024-04-23T10:41:09.325 INFO:teuthology.orchestra.run.smithi114.stdout: "default_value": "", 2024-04-23T10:41:09.325 INFO:teuthology.orchestra.run.smithi114.stdout: "min": "", 2024-04-23T10:41:09.325 INFO:teuthology.orchestra.run.smithi114.stdout: "max": "", 2024-04-23T10:41:09.325 INFO:teuthology.orchestra.run.smithi114.stdout: "enum_allowed": [ 2024-04-23T10:41:09.325 INFO:teuthology.orchestra.run.smithi114.stdout: "", 2024-04-23T10:41:09.326 INFO:teuthology.orchestra.run.smithi114.stdout: "critical", 2024-04-23T10:41:09.326 INFO:teuthology.orchestra.run.smithi114.stdout: "debug", 2024-04-23T10:41:09.326 INFO:teuthology.orchestra.run.smithi114.stdout: "error", 2024-04-23T10:41:09.326 INFO:teuthology.orchestra.run.smithi114.stdout: "info", 2024-04-23T10:41:09.326 INFO:teuthology.orchestra.run.smithi114.stdout: "warning" 2024-04-23T10:41:09.326 INFO:teuthology.orchestra.run.smithi114.stdout: ], 2024-04-23T10:41:09.326 INFO:teuthology.orchestra.run.smithi114.stdout: "desc": "", 2024-04-23T10:41:09.326 INFO:teuthology.orchestra.run.smithi114.stdout: "long_desc": "", 2024-04-23T10:41:09.327 INFO:teuthology.orchestra.run.smithi114.stdout: "tags": [], 2024-04-23T10:41:09.327 INFO:teuthology.orchestra.run.smithi114.stdout: "see_also": [] 2024-04-23T10:41:09.327 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-23T10:41:09.327 INFO:teuthology.orchestra.run.smithi114.stdout: "log_to_cluster": { 2024-04-23T10:41:09.327 INFO:teuthology.orchestra.run.smithi114.stdout: "name": "log_to_cluster", 2024-04-23T10:41:09.327 INFO:teuthology.orchestra.run.smithi114.stdout: "type": "bool", 2024-04-23T10:41:09.327 INFO:teuthology.orchestra.run.smithi114.stdout: "level": "advanced", 2024-04-23T10:41:09.327 INFO:teuthology.orchestra.run.smithi114.stdout: "flags": 1, 2024-04-23T10:41:09.327 INFO:teuthology.orchestra.run.smithi114.stdout: "default_value": "False", 2024-04-23T10:41:09.328 INFO:teuthology.orchestra.run.smithi114.stdout: "min": "", 2024-04-23T10:41:09.328 INFO:teuthology.orchestra.run.smithi114.stdout: "max": "", 2024-04-23T10:41:09.328 INFO:teuthology.orchestra.run.smithi114.stdout: "enum_allowed": [], 2024-04-23T10:41:09.328 INFO:teuthology.orchestra.run.smithi114.stdout: "desc": "", 2024-04-23T10:41:09.328 INFO:teuthology.orchestra.run.smithi114.stdout: "long_desc": "", 2024-04-23T10:41:09.328 INFO:teuthology.orchestra.run.smithi114.stdout: "tags": [], 2024-04-23T10:41:09.328 INFO:teuthology.orchestra.run.smithi114.stdout: "see_also": [] 2024-04-23T10:41:09.328 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-23T10:41:09.328 INFO:teuthology.orchestra.run.smithi114.stdout: "log_to_cluster_level": { 2024-04-23T10:41:09.329 INFO:teuthology.orchestra.run.smithi114.stdout: "name": "log_to_cluster_level", 2024-04-23T10:41:09.329 INFO:teuthology.orchestra.run.smithi114.stdout: "type": "str", 2024-04-23T10:41:09.329 INFO:teuthology.orchestra.run.smithi114.stdout: "level": "advanced", 2024-04-23T10:41:09.329 INFO:teuthology.orchestra.run.smithi114.stdout: "flags": 1, 2024-04-23T10:41:09.329 INFO:teuthology.orchestra.run.smithi114.stdout: "default_value": "info", 2024-04-23T10:41:09.329 INFO:teuthology.orchestra.run.smithi114.stdout: "min": "", 2024-04-23T10:41:09.329 INFO:teuthology.orchestra.run.smithi114.stdout: "max": "", 2024-04-23T10:41:09.329 INFO:teuthology.orchestra.run.smithi114.stdout: "enum_allowed": [ 2024-04-23T10:41:09.329 INFO:teuthology.orchestra.run.smithi114.stdout: "", 2024-04-23T10:41:09.330 INFO:teuthology.orchestra.run.smithi114.stdout: "critical", 2024-04-23T10:41:09.330 INFO:teuthology.orchestra.run.smithi114.stdout: "debug", 2024-04-23T10:41:09.330 INFO:teuthology.orchestra.run.smithi114.stdout: "error", 2024-04-23T10:41:09.330 INFO:teuthology.orchestra.run.smithi114.stdout: "info", 2024-04-23T10:41:09.330 INFO:teuthology.orchestra.run.smithi114.stdout: "warning" 2024-04-23T10:41:09.330 INFO:teuthology.orchestra.run.smithi114.stdout: ], 2024-04-23T10:41:09.330 INFO:teuthology.orchestra.run.smithi114.stdout: "desc": "", 2024-04-23T10:41:09.330 INFO:teuthology.orchestra.run.smithi114.stdout: "long_desc": "", 2024-04-23T10:41:09.331 INFO:teuthology.orchestra.run.smithi114.stdout: "tags": [], 2024-04-23T10:41:09.331 INFO:teuthology.orchestra.run.smithi114.stdout: "see_also": [] 2024-04-23T10:41:09.331 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-23T10:41:09.331 INFO:teuthology.orchestra.run.smithi114.stdout: "log_to_file": { 2024-04-23T10:41:09.331 INFO:teuthology.orchestra.run.smithi114.stdout: "name": "log_to_file", 2024-04-23T10:41:09.331 INFO:teuthology.orchestra.run.smithi114.stdout: "type": "bool", 2024-04-23T10:41:09.331 INFO:teuthology.orchestra.run.smithi114.stdout: "level": "advanced", 2024-04-23T10:41:09.331 INFO:teuthology.orchestra.run.smithi114.stdout: "flags": 1, 2024-04-23T10:41:09.331 INFO:teuthology.orchestra.run.smithi114.stdout: "default_value": "False", 2024-04-23T10:41:09.332 INFO:teuthology.orchestra.run.smithi114.stdout: "min": "", 2024-04-23T10:41:09.332 INFO:teuthology.orchestra.run.smithi114.stdout: "max": "", 2024-04-23T10:41:09.332 INFO:teuthology.orchestra.run.smithi114.stdout: "enum_allowed": [], 2024-04-23T10:41:09.332 INFO:teuthology.orchestra.run.smithi114.stdout: "desc": "", 2024-04-23T10:41:09.332 INFO:teuthology.orchestra.run.smithi114.stdout: "long_desc": "", 2024-04-23T10:41:09.332 INFO:teuthology.orchestra.run.smithi114.stdout: "tags": [], 2024-04-23T10:41:09.332 INFO:teuthology.orchestra.run.smithi114.stdout: "see_also": [] 2024-04-23T10:41:09.332 INFO:teuthology.orchestra.run.smithi114.stdout: } 2024-04-23T10:41:09.332 INFO:teuthology.orchestra.run.smithi114.stdout: } 2024-04-23T10:41:09.333 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-23T10:41:09.333 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-04-23T10:41:09.333 INFO:teuthology.orchestra.run.smithi114.stdout: "name": "status", 2024-04-23T10:41:09.333 INFO:teuthology.orchestra.run.smithi114.stdout: "can_run": true, 2024-04-23T10:41:09.333 INFO:teuthology.orchestra.run.smithi114.stdout: "error_string": "", 2024-04-23T10:41:09.333 INFO:teuthology.orchestra.run.smithi114.stdout: "module_options": { 2024-04-23T10:41:09.333 INFO:teuthology.orchestra.run.smithi114.stdout: "log_level": { 2024-04-23T10:41:09.333 INFO:teuthology.orchestra.run.smithi114.stdout: "name": "log_level", 2024-04-23T10:41:09.333 INFO:teuthology.orchestra.run.smithi114.stdout: "type": "str", 2024-04-23T10:41:09.334 INFO:teuthology.orchestra.run.smithi114.stdout: "level": "advanced", 2024-04-23T10:41:09.334 INFO:teuthology.orchestra.run.smithi114.stdout: "flags": 1, 2024-04-23T10:41:09.334 INFO:teuthology.orchestra.run.smithi114.stdout: "default_value": "", 2024-04-23T10:41:09.334 INFO:teuthology.orchestra.run.smithi114.stdout: "min": "", 2024-04-23T10:41:09.334 INFO:teuthology.orchestra.run.smithi114.stdout: "max": "", 2024-04-23T10:41:09.334 INFO:teuthology.orchestra.run.smithi114.stdout: "enum_allowed": [ 2024-04-23T10:41:09.334 INFO:teuthology.orchestra.run.smithi114.stdout: "", 2024-04-23T10:41:09.334 INFO:teuthology.orchestra.run.smithi114.stdout: "critical", 2024-04-23T10:41:09.335 INFO:teuthology.orchestra.run.smithi114.stdout: "debug", 2024-04-23T10:41:09.335 INFO:teuthology.orchestra.run.smithi114.stdout: "error", 2024-04-23T10:41:09.335 INFO:teuthology.orchestra.run.smithi114.stdout: "info", 2024-04-23T10:41:09.335 INFO:teuthology.orchestra.run.smithi114.stdout: "warning" 2024-04-23T10:41:09.335 INFO:teuthology.orchestra.run.smithi114.stdout: ], 2024-04-23T10:41:09.335 INFO:teuthology.orchestra.run.smithi114.stdout: "desc": "", 2024-04-23T10:41:09.335 INFO:teuthology.orchestra.run.smithi114.stdout: "long_desc": "", 2024-04-23T10:41:09.335 INFO:teuthology.orchestra.run.smithi114.stdout: "tags": [], 2024-04-23T10:41:09.335 INFO:teuthology.orchestra.run.smithi114.stdout: "see_also": [] 2024-04-23T10:41:09.336 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-23T10:41:09.336 INFO:teuthology.orchestra.run.smithi114.stdout: "log_to_cluster": { 2024-04-23T10:41:09.336 INFO:teuthology.orchestra.run.smithi114.stdout: "name": "log_to_cluster", 2024-04-23T10:41:09.336 INFO:teuthology.orchestra.run.smithi114.stdout: "type": "bool", 2024-04-23T10:41:09.336 INFO:teuthology.orchestra.run.smithi114.stdout: "level": "advanced", 2024-04-23T10:41:09.336 INFO:teuthology.orchestra.run.smithi114.stdout: "flags": 1, 2024-04-23T10:41:09.336 INFO:teuthology.orchestra.run.smithi114.stdout: "default_value": "False", 2024-04-23T10:41:09.336 INFO:teuthology.orchestra.run.smithi114.stdout: "min": "", 2024-04-23T10:41:09.337 INFO:teuthology.orchestra.run.smithi114.stdout: "max": "", 2024-04-23T10:41:09.337 INFO:teuthology.orchestra.run.smithi114.stdout: "enum_allowed": [], 2024-04-23T10:41:09.337 INFO:teuthology.orchestra.run.smithi114.stdout: "desc": "", 2024-04-23T10:41:09.337 INFO:teuthology.orchestra.run.smithi114.stdout: "long_desc": "", 2024-04-23T10:41:09.337 INFO:teuthology.orchestra.run.smithi114.stdout: "tags": [], 2024-04-23T10:41:09.337 INFO:teuthology.orchestra.run.smithi114.stdout: "see_also": [] 2024-04-23T10:41:09.337 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-23T10:41:09.337 INFO:teuthology.orchestra.run.smithi114.stdout: "log_to_cluster_level": { 2024-04-23T10:41:09.337 INFO:teuthology.orchestra.run.smithi114.stdout: "name": "log_to_cluster_level", 2024-04-23T10:41:09.338 INFO:teuthology.orchestra.run.smithi114.stdout: "type": "str", 2024-04-23T10:41:09.338 INFO:teuthology.orchestra.run.smithi114.stdout: "level": "advanced", 2024-04-23T10:41:09.338 INFO:teuthology.orchestra.run.smithi114.stdout: "flags": 1, 2024-04-23T10:41:09.338 INFO:teuthology.orchestra.run.smithi114.stdout: "default_value": "info", 2024-04-23T10:41:09.338 INFO:teuthology.orchestra.run.smithi114.stdout: "min": "", 2024-04-23T10:41:09.338 INFO:teuthology.orchestra.run.smithi114.stdout: "max": "", 2024-04-23T10:41:09.338 INFO:teuthology.orchestra.run.smithi114.stdout: "enum_allowed": [ 2024-04-23T10:41:09.338 INFO:teuthology.orchestra.run.smithi114.stdout: "", 2024-04-23T10:41:09.339 INFO:teuthology.orchestra.run.smithi114.stdout: "critical", 2024-04-23T10:41:09.339 INFO:teuthology.orchestra.run.smithi114.stdout: "debug", 2024-04-23T10:41:09.339 INFO:teuthology.orchestra.run.smithi114.stdout: "error", 2024-04-23T10:41:09.339 INFO:teuthology.orchestra.run.smithi114.stdout: "info", 2024-04-23T10:41:09.339 INFO:teuthology.orchestra.run.smithi114.stdout: "warning" 2024-04-23T10:41:09.339 INFO:teuthology.orchestra.run.smithi114.stdout: ], 2024-04-23T10:41:09.339 INFO:teuthology.orchestra.run.smithi114.stdout: "desc": "", 2024-04-23T10:41:09.339 INFO:teuthology.orchestra.run.smithi114.stdout: "long_desc": "", 2024-04-23T10:41:09.339 INFO:teuthology.orchestra.run.smithi114.stdout: "tags": [], 2024-04-23T10:41:09.340 INFO:teuthology.orchestra.run.smithi114.stdout: "see_also": [] 2024-04-23T10:41:09.340 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-23T10:41:09.340 INFO:teuthology.orchestra.run.smithi114.stdout: "log_to_file": { 2024-04-23T10:41:09.340 INFO:teuthology.orchestra.run.smithi114.stdout: "name": "log_to_file", 2024-04-23T10:41:09.340 INFO:teuthology.orchestra.run.smithi114.stdout: "type": "bool", 2024-04-23T10:41:09.340 INFO:teuthology.orchestra.run.smithi114.stdout: "level": "advanced", 2024-04-23T10:41:09.340 INFO:teuthology.orchestra.run.smithi114.stdout: "flags": 1, 2024-04-23T10:41:09.340 INFO:teuthology.orchestra.run.smithi114.stdout: "default_value": "False", 2024-04-23T10:41:09.341 INFO:teuthology.orchestra.run.smithi114.stdout: "min": "", 2024-04-23T10:41:09.341 INFO:teuthology.orchestra.run.smithi114.stdout: "max": "", 2024-04-23T10:41:09.341 INFO:teuthology.orchestra.run.smithi114.stdout: "enum_allowed": [], 2024-04-23T10:41:09.341 INFO:teuthology.orchestra.run.smithi114.stdout: "desc": "", 2024-04-23T10:41:09.341 INFO:teuthology.orchestra.run.smithi114.stdout: "long_desc": "", 2024-04-23T10:41:09.341 INFO:teuthology.orchestra.run.smithi114.stdout: "tags": [], 2024-04-23T10:41:09.341 INFO:teuthology.orchestra.run.smithi114.stdout: "see_also": [] 2024-04-23T10:41:09.341 INFO:teuthology.orchestra.run.smithi114.stdout: } 2024-04-23T10:41:09.341 INFO:teuthology.orchestra.run.smithi114.stdout: } 2024-04-23T10:41:09.342 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-23T10:41:09.342 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-04-23T10:41:09.342 INFO:teuthology.orchestra.run.smithi114.stdout: "name": "telegraf", 2024-04-23T10:41:09.342 INFO:teuthology.orchestra.run.smithi114.stdout: "can_run": true, 2024-04-23T10:41:09.342 INFO:teuthology.orchestra.run.smithi114.stdout: "error_string": "", 2024-04-23T10:41:09.342 INFO:teuthology.orchestra.run.smithi114.stdout: "module_options": { 2024-04-23T10:41:09.342 INFO:teuthology.orchestra.run.smithi114.stdout: "address": { 2024-04-23T10:41:09.342 INFO:teuthology.orchestra.run.smithi114.stdout: "name": "address", 2024-04-23T10:41:09.342 INFO:teuthology.orchestra.run.smithi114.stdout: "type": "str", 2024-04-23T10:41:09.343 INFO:teuthology.orchestra.run.smithi114.stdout: "level": "advanced", 2024-04-23T10:41:09.343 INFO:teuthology.orchestra.run.smithi114.stdout: "flags": 0, 2024-04-23T10:41:09.343 INFO:teuthology.orchestra.run.smithi114.stdout: "default_value": "unixgram:///tmp/telegraf.sock", 2024-04-23T10:41:09.343 INFO:teuthology.orchestra.run.smithi114.stdout: "min": "", 2024-04-23T10:41:09.343 INFO:teuthology.orchestra.run.smithi114.stdout: "max": "", 2024-04-23T10:41:09.343 INFO:teuthology.orchestra.run.smithi114.stdout: "enum_allowed": [], 2024-04-23T10:41:09.343 INFO:teuthology.orchestra.run.smithi114.stdout: "desc": "", 2024-04-23T10:41:09.343 INFO:teuthology.orchestra.run.smithi114.stdout: "long_desc": "", 2024-04-23T10:41:09.344 INFO:teuthology.orchestra.run.smithi114.stdout: "tags": [], 2024-04-23T10:41:09.344 INFO:teuthology.orchestra.run.smithi114.stdout: "see_also": [] 2024-04-23T10:41:09.344 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-23T10:41:09.344 INFO:teuthology.orchestra.run.smithi114.stdout: "interval": { 2024-04-23T10:41:09.344 INFO:teuthology.orchestra.run.smithi114.stdout: "name": "interval", 2024-04-23T10:41:09.344 INFO:teuthology.orchestra.run.smithi114.stdout: "type": "secs", 2024-04-23T10:41:09.344 INFO:teuthology.orchestra.run.smithi114.stdout: "level": "advanced", 2024-04-23T10:41:09.344 INFO:teuthology.orchestra.run.smithi114.stdout: "flags": 0, 2024-04-23T10:41:09.344 INFO:teuthology.orchestra.run.smithi114.stdout: "default_value": "15", 2024-04-23T10:41:09.345 INFO:teuthology.orchestra.run.smithi114.stdout: "min": "", 2024-04-23T10:41:09.345 INFO:teuthology.orchestra.run.smithi114.stdout: "max": "", 2024-04-23T10:41:09.345 INFO:teuthology.orchestra.run.smithi114.stdout: "enum_allowed": [], 2024-04-23T10:41:09.345 INFO:teuthology.orchestra.run.smithi114.stdout: "desc": "", 2024-04-23T10:41:09.345 INFO:teuthology.orchestra.run.smithi114.stdout: "long_desc": "", 2024-04-23T10:41:09.345 INFO:teuthology.orchestra.run.smithi114.stdout: "tags": [], 2024-04-23T10:41:09.345 INFO:teuthology.orchestra.run.smithi114.stdout: "see_also": [] 2024-04-23T10:41:09.345 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-23T10:41:09.345 INFO:teuthology.orchestra.run.smithi114.stdout: "log_level": { 2024-04-23T10:41:09.346 INFO:teuthology.orchestra.run.smithi114.stdout: "name": "log_level", 2024-04-23T10:41:09.346 INFO:teuthology.orchestra.run.smithi114.stdout: "type": "str", 2024-04-23T10:41:09.346 INFO:teuthology.orchestra.run.smithi114.stdout: "level": "advanced", 2024-04-23T10:41:09.346 INFO:teuthology.orchestra.run.smithi114.stdout: "flags": 1, 2024-04-23T10:41:09.346 INFO:teuthology.orchestra.run.smithi114.stdout: "default_value": "", 2024-04-23T10:41:09.346 INFO:teuthology.orchestra.run.smithi114.stdout: "min": "", 2024-04-23T10:41:09.346 INFO:teuthology.orchestra.run.smithi114.stdout: "max": "", 2024-04-23T10:41:09.346 INFO:teuthology.orchestra.run.smithi114.stdout: "enum_allowed": [ 2024-04-23T10:41:09.347 INFO:teuthology.orchestra.run.smithi114.stdout: "", 2024-04-23T10:41:09.347 INFO:teuthology.orchestra.run.smithi114.stdout: "critical", 2024-04-23T10:41:09.347 INFO:teuthology.orchestra.run.smithi114.stdout: "debug", 2024-04-23T10:41:09.347 INFO:teuthology.orchestra.run.smithi114.stdout: "error", 2024-04-23T10:41:09.347 INFO:teuthology.orchestra.run.smithi114.stdout: "info", 2024-04-23T10:41:09.347 INFO:teuthology.orchestra.run.smithi114.stdout: "warning" 2024-04-23T10:41:09.347 INFO:teuthology.orchestra.run.smithi114.stdout: ], 2024-04-23T10:41:09.347 INFO:teuthology.orchestra.run.smithi114.stdout: "desc": "", 2024-04-23T10:41:09.347 INFO:teuthology.orchestra.run.smithi114.stdout: "long_desc": "", 2024-04-23T10:41:09.348 INFO:teuthology.orchestra.run.smithi114.stdout: "tags": [], 2024-04-23T10:41:09.348 INFO:teuthology.orchestra.run.smithi114.stdout: "see_also": [] 2024-04-23T10:41:09.348 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-23T10:41:09.348 INFO:teuthology.orchestra.run.smithi114.stdout: "log_to_cluster": { 2024-04-23T10:41:09.348 INFO:teuthology.orchestra.run.smithi114.stdout: "name": "log_to_cluster", 2024-04-23T10:41:09.348 INFO:teuthology.orchestra.run.smithi114.stdout: "type": "bool", 2024-04-23T10:41:09.348 INFO:teuthology.orchestra.run.smithi114.stdout: "level": "advanced", 2024-04-23T10:41:09.348 INFO:teuthology.orchestra.run.smithi114.stdout: "flags": 1, 2024-04-23T10:41:09.348 INFO:teuthology.orchestra.run.smithi114.stdout: "default_value": "False", 2024-04-23T10:41:09.349 INFO:teuthology.orchestra.run.smithi114.stdout: "min": "", 2024-04-23T10:41:09.349 INFO:teuthology.orchestra.run.smithi114.stdout: "max": "", 2024-04-23T10:41:09.349 INFO:teuthology.orchestra.run.smithi114.stdout: "enum_allowed": [], 2024-04-23T10:41:09.349 INFO:teuthology.orchestra.run.smithi114.stdout: "desc": "", 2024-04-23T10:41:09.349 INFO:teuthology.orchestra.run.smithi114.stdout: "long_desc": "", 2024-04-23T10:41:09.349 INFO:teuthology.orchestra.run.smithi114.stdout: "tags": [], 2024-04-23T10:41:09.349 INFO:teuthology.orchestra.run.smithi114.stdout: "see_also": [] 2024-04-23T10:41:09.349 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-23T10:41:09.349 INFO:teuthology.orchestra.run.smithi114.stdout: "log_to_cluster_level": { 2024-04-23T10:41:09.350 INFO:teuthology.orchestra.run.smithi114.stdout: "name": "log_to_cluster_level", 2024-04-23T10:41:09.350 INFO:teuthology.orchestra.run.smithi114.stdout: "type": "str", 2024-04-23T10:41:09.350 INFO:teuthology.orchestra.run.smithi114.stdout: "level": "advanced", 2024-04-23T10:41:09.350 INFO:teuthology.orchestra.run.smithi114.stdout: "flags": 1, 2024-04-23T10:41:09.350 INFO:teuthology.orchestra.run.smithi114.stdout: "default_value": "info", 2024-04-23T10:41:09.350 INFO:teuthology.orchestra.run.smithi114.stdout: "min": "", 2024-04-23T10:41:09.350 INFO:teuthology.orchestra.run.smithi114.stdout: "max": "", 2024-04-23T10:41:09.350 INFO:teuthology.orchestra.run.smithi114.stdout: "enum_allowed": [ 2024-04-23T10:41:09.350 INFO:teuthology.orchestra.run.smithi114.stdout: "", 2024-04-23T10:41:09.351 INFO:teuthology.orchestra.run.smithi114.stdout: "critical", 2024-04-23T10:41:09.351 INFO:teuthology.orchestra.run.smithi114.stdout: "debug", 2024-04-23T10:41:09.351 INFO:teuthology.orchestra.run.smithi114.stdout: "error", 2024-04-23T10:41:09.351 INFO:teuthology.orchestra.run.smithi114.stdout: "info", 2024-04-23T10:41:09.351 INFO:teuthology.orchestra.run.smithi114.stdout: "warning" 2024-04-23T10:41:09.351 INFO:teuthology.orchestra.run.smithi114.stdout: ], 2024-04-23T10:41:09.351 INFO:teuthology.orchestra.run.smithi114.stdout: "desc": "", 2024-04-23T10:41:09.351 INFO:teuthology.orchestra.run.smithi114.stdout: "long_desc": "", 2024-04-23T10:41:09.352 INFO:teuthology.orchestra.run.smithi114.stdout: "tags": [], 2024-04-23T10:41:09.352 INFO:teuthology.orchestra.run.smithi114.stdout: "see_also": [] 2024-04-23T10:41:09.352 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-23T10:41:09.352 INFO:teuthology.orchestra.run.smithi114.stdout: "log_to_file": { 2024-04-23T10:41:09.352 INFO:teuthology.orchestra.run.smithi114.stdout: "name": "log_to_file", 2024-04-23T10:41:09.352 INFO:teuthology.orchestra.run.smithi114.stdout: "type": "bool", 2024-04-23T10:41:09.352 INFO:teuthology.orchestra.run.smithi114.stdout: "level": "advanced", 2024-04-23T10:41:09.352 INFO:teuthology.orchestra.run.smithi114.stdout: "flags": 1, 2024-04-23T10:41:09.352 INFO:teuthology.orchestra.run.smithi114.stdout: "default_value": "False", 2024-04-23T10:41:09.353 INFO:teuthology.orchestra.run.smithi114.stdout: "min": "", 2024-04-23T10:41:09.353 INFO:teuthology.orchestra.run.smithi114.stdout: "max": "", 2024-04-23T10:41:09.353 INFO:teuthology.orchestra.run.smithi114.stdout: "enum_allowed": [], 2024-04-23T10:41:09.353 INFO:teuthology.orchestra.run.smithi114.stdout: "desc": "", 2024-04-23T10:41:09.353 INFO:teuthology.orchestra.run.smithi114.stdout: "long_desc": "", 2024-04-23T10:41:09.353 INFO:teuthology.orchestra.run.smithi114.stdout: "tags": [], 2024-04-23T10:41:09.353 INFO:teuthology.orchestra.run.smithi114.stdout: "see_also": [] 2024-04-23T10:41:09.353 INFO:teuthology.orchestra.run.smithi114.stdout: } 2024-04-23T10:41:09.353 INFO:teuthology.orchestra.run.smithi114.stdout: } 2024-04-23T10:41:09.354 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-23T10:41:09.354 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-04-23T10:41:09.354 INFO:teuthology.orchestra.run.smithi114.stdout: "name": "telemetry", 2024-04-23T10:41:09.354 INFO:teuthology.orchestra.run.smithi114.stdout: "can_run": true, 2024-04-23T10:41:09.354 INFO:teuthology.orchestra.run.smithi114.stdout: "error_string": "", 2024-04-23T10:41:09.354 INFO:teuthology.orchestra.run.smithi114.stdout: "module_options": { 2024-04-23T10:41:09.354 INFO:teuthology.orchestra.run.smithi114.stdout: "channel_basic": { 2024-04-23T10:41:09.354 INFO:teuthology.orchestra.run.smithi114.stdout: "name": "channel_basic", 2024-04-23T10:41:09.354 INFO:teuthology.orchestra.run.smithi114.stdout: "type": "bool", 2024-04-23T10:41:09.355 INFO:teuthology.orchestra.run.smithi114.stdout: "level": "advanced", 2024-04-23T10:41:09.355 INFO:teuthology.orchestra.run.smithi114.stdout: "flags": 0, 2024-04-23T10:41:09.355 INFO:teuthology.orchestra.run.smithi114.stdout: "default_value": "True", 2024-04-23T10:41:09.355 INFO:teuthology.orchestra.run.smithi114.stdout: "min": "", 2024-04-23T10:41:09.355 INFO:teuthology.orchestra.run.smithi114.stdout: "max": "", 2024-04-23T10:41:09.355 INFO:teuthology.orchestra.run.smithi114.stdout: "enum_allowed": [], 2024-04-23T10:41:09.355 INFO:teuthology.orchestra.run.smithi114.stdout: "desc": "Share basic cluster information (size, version)", 2024-04-23T10:41:09.355 INFO:teuthology.orchestra.run.smithi114.stdout: "long_desc": "", 2024-04-23T10:41:09.356 INFO:teuthology.orchestra.run.smithi114.stdout: "tags": [], 2024-04-23T10:41:09.356 INFO:teuthology.orchestra.run.smithi114.stdout: "see_also": [] 2024-04-23T10:41:09.356 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-23T10:41:09.356 INFO:teuthology.orchestra.run.smithi114.stdout: "channel_crash": { 2024-04-23T10:41:09.356 INFO:teuthology.orchestra.run.smithi114.stdout: "name": "channel_crash", 2024-04-23T10:41:09.356 INFO:teuthology.orchestra.run.smithi114.stdout: "type": "bool", 2024-04-23T10:41:09.356 INFO:teuthology.orchestra.run.smithi114.stdout: "level": "advanced", 2024-04-23T10:41:09.356 INFO:teuthology.orchestra.run.smithi114.stdout: "flags": 0, 2024-04-23T10:41:09.356 INFO:teuthology.orchestra.run.smithi114.stdout: "default_value": "True", 2024-04-23T10:41:09.357 INFO:teuthology.orchestra.run.smithi114.stdout: "min": "", 2024-04-23T10:41:09.357 INFO:teuthology.orchestra.run.smithi114.stdout: "max": "", 2024-04-23T10:41:09.357 INFO:teuthology.orchestra.run.smithi114.stdout: "enum_allowed": [], 2024-04-23T10:41:09.357 INFO:teuthology.orchestra.run.smithi114.stdout: "desc": "Share metadata about Ceph daemon crashes (version, stack straces, etc)", 2024-04-23T10:41:09.357 INFO:teuthology.orchestra.run.smithi114.stdout: "long_desc": "", 2024-04-23T10:41:09.357 INFO:teuthology.orchestra.run.smithi114.stdout: "tags": [], 2024-04-23T10:41:09.357 INFO:teuthology.orchestra.run.smithi114.stdout: "see_also": [] 2024-04-23T10:41:09.357 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-23T10:41:09.358 INFO:teuthology.orchestra.run.smithi114.stdout: "channel_device": { 2024-04-23T10:41:09.358 INFO:teuthology.orchestra.run.smithi114.stdout: "name": "channel_device", 2024-04-23T10:41:09.358 INFO:teuthology.orchestra.run.smithi114.stdout: "type": "bool", 2024-04-23T10:41:09.358 INFO:teuthology.orchestra.run.smithi114.stdout: "level": "advanced", 2024-04-23T10:41:09.358 INFO:teuthology.orchestra.run.smithi114.stdout: "flags": 0, 2024-04-23T10:41:09.358 INFO:teuthology.orchestra.run.smithi114.stdout: "default_value": "True", 2024-04-23T10:41:09.358 INFO:teuthology.orchestra.run.smithi114.stdout: "min": "", 2024-04-23T10:41:09.358 INFO:teuthology.orchestra.run.smithi114.stdout: "max": "", 2024-04-23T10:41:09.359 INFO:teuthology.orchestra.run.smithi114.stdout: "enum_allowed": [], 2024-04-23T10:41:09.359 INFO:teuthology.orchestra.run.smithi114.stdout: "desc": "Share device health metrics (e.g., SMART data, minus potentially identifying info like serial numbers)", 2024-04-23T10:41:09.359 INFO:teuthology.orchestra.run.smithi114.stdout: "long_desc": "", 2024-04-23T10:41:09.359 INFO:teuthology.orchestra.run.smithi114.stdout: "tags": [], 2024-04-23T10:41:09.359 INFO:teuthology.orchestra.run.smithi114.stdout: "see_also": [] 2024-04-23T10:41:09.359 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-23T10:41:09.359 INFO:teuthology.orchestra.run.smithi114.stdout: "channel_ident": { 2024-04-23T10:41:09.359 INFO:teuthology.orchestra.run.smithi114.stdout: "name": "channel_ident", 2024-04-23T10:41:09.359 INFO:teuthology.orchestra.run.smithi114.stdout: "type": "bool", 2024-04-23T10:41:09.360 INFO:teuthology.orchestra.run.smithi114.stdout: "level": "advanced", 2024-04-23T10:41:09.360 INFO:teuthology.orchestra.run.smithi114.stdout: "flags": 0, 2024-04-23T10:41:09.360 INFO:teuthology.orchestra.run.smithi114.stdout: "default_value": "False", 2024-04-23T10:41:09.360 INFO:teuthology.orchestra.run.smithi114.stdout: "min": "", 2024-04-23T10:41:09.360 INFO:teuthology.orchestra.run.smithi114.stdout: "max": "", 2024-04-23T10:41:09.360 INFO:teuthology.orchestra.run.smithi114.stdout: "enum_allowed": [], 2024-04-23T10:41:09.360 INFO:teuthology.orchestra.run.smithi114.stdout: "desc": "Share a user-provided description and/or contact email for the cluster", 2024-04-23T10:41:09.360 INFO:teuthology.orchestra.run.smithi114.stdout: "long_desc": "", 2024-04-23T10:41:09.360 INFO:teuthology.orchestra.run.smithi114.stdout: "tags": [], 2024-04-23T10:41:09.361 INFO:teuthology.orchestra.run.smithi114.stdout: "see_also": [] 2024-04-23T10:41:09.361 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-23T10:41:09.361 INFO:teuthology.orchestra.run.smithi114.stdout: "channel_perf": { 2024-04-23T10:41:09.361 INFO:teuthology.orchestra.run.smithi114.stdout: "name": "channel_perf", 2024-04-23T10:41:09.361 INFO:teuthology.orchestra.run.smithi114.stdout: "type": "bool", 2024-04-23T10:41:09.361 INFO:teuthology.orchestra.run.smithi114.stdout: "level": "advanced", 2024-04-23T10:41:09.361 INFO:teuthology.orchestra.run.smithi114.stdout: "flags": 0, 2024-04-23T10:41:09.361 INFO:teuthology.orchestra.run.smithi114.stdout: "default_value": "False", 2024-04-23T10:41:09.361 INFO:teuthology.orchestra.run.smithi114.stdout: "min": "", 2024-04-23T10:41:09.362 INFO:teuthology.orchestra.run.smithi114.stdout: "max": "", 2024-04-23T10:41:09.362 INFO:teuthology.orchestra.run.smithi114.stdout: "enum_allowed": [], 2024-04-23T10:41:09.362 INFO:teuthology.orchestra.run.smithi114.stdout: "desc": "Share various performance metrics of a cluster", 2024-04-23T10:41:09.362 INFO:teuthology.orchestra.run.smithi114.stdout: "long_desc": "", 2024-04-23T10:41:09.362 INFO:teuthology.orchestra.run.smithi114.stdout: "tags": [], 2024-04-23T10:41:09.362 INFO:teuthology.orchestra.run.smithi114.stdout: "see_also": [] 2024-04-23T10:41:09.362 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-23T10:41:09.362 INFO:teuthology.orchestra.run.smithi114.stdout: "contact": { 2024-04-23T10:41:09.363 INFO:teuthology.orchestra.run.smithi114.stdout: "name": "contact", 2024-04-23T10:41:09.363 INFO:teuthology.orchestra.run.smithi114.stdout: "type": "str", 2024-04-23T10:41:09.363 INFO:teuthology.orchestra.run.smithi114.stdout: "level": "advanced", 2024-04-23T10:41:09.363 INFO:teuthology.orchestra.run.smithi114.stdout: "flags": 0, 2024-04-23T10:41:09.363 INFO:teuthology.orchestra.run.smithi114.stdout: "default_value": "", 2024-04-23T10:41:09.363 INFO:teuthology.orchestra.run.smithi114.stdout: "min": "", 2024-04-23T10:41:09.363 INFO:teuthology.orchestra.run.smithi114.stdout: "max": "", 2024-04-23T10:41:09.363 INFO:teuthology.orchestra.run.smithi114.stdout: "enum_allowed": [], 2024-04-23T10:41:09.363 INFO:teuthology.orchestra.run.smithi114.stdout: "desc": "", 2024-04-23T10:41:09.363 INFO:teuthology.orchestra.run.smithi114.stdout: "long_desc": "", 2024-04-23T10:41:09.364 INFO:teuthology.orchestra.run.smithi114.stdout: "tags": [], 2024-04-23T10:41:09.364 INFO:teuthology.orchestra.run.smithi114.stdout: "see_also": [] 2024-04-23T10:41:09.364 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-23T10:41:09.364 INFO:teuthology.orchestra.run.smithi114.stdout: "description": { 2024-04-23T10:41:09.364 INFO:teuthology.orchestra.run.smithi114.stdout: "name": "description", 2024-04-23T10:41:09.364 INFO:teuthology.orchestra.run.smithi114.stdout: "type": "str", 2024-04-23T10:41:09.364 INFO:teuthology.orchestra.run.smithi114.stdout: "level": "advanced", 2024-04-23T10:41:09.364 INFO:teuthology.orchestra.run.smithi114.stdout: "flags": 0, 2024-04-23T10:41:09.364 INFO:teuthology.orchestra.run.smithi114.stdout: "default_value": "", 2024-04-23T10:41:09.365 INFO:teuthology.orchestra.run.smithi114.stdout: "min": "", 2024-04-23T10:41:09.365 INFO:teuthology.orchestra.run.smithi114.stdout: "max": "", 2024-04-23T10:41:09.365 INFO:teuthology.orchestra.run.smithi114.stdout: "enum_allowed": [], 2024-04-23T10:41:09.365 INFO:teuthology.orchestra.run.smithi114.stdout: "desc": "", 2024-04-23T10:41:09.365 INFO:teuthology.orchestra.run.smithi114.stdout: "long_desc": "", 2024-04-23T10:41:09.365 INFO:teuthology.orchestra.run.smithi114.stdout: "tags": [], 2024-04-23T10:41:09.365 INFO:teuthology.orchestra.run.smithi114.stdout: "see_also": [] 2024-04-23T10:41:09.365 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-23T10:41:09.366 INFO:teuthology.orchestra.run.smithi114.stdout: "device_url": { 2024-04-23T10:41:09.366 INFO:teuthology.orchestra.run.smithi114.stdout: "name": "device_url", 2024-04-23T10:41:09.366 INFO:teuthology.orchestra.run.smithi114.stdout: "type": "str", 2024-04-23T10:41:09.366 INFO:teuthology.orchestra.run.smithi114.stdout: "level": "advanced", 2024-04-23T10:41:09.366 INFO:teuthology.orchestra.run.smithi114.stdout: "flags": 0, 2024-04-23T10:41:09.366 INFO:teuthology.orchestra.run.smithi114.stdout: "default_value": "https://telemetry.ceph.com/device", 2024-04-23T10:41:09.366 INFO:teuthology.orchestra.run.smithi114.stdout: "min": "", 2024-04-23T10:41:09.366 INFO:teuthology.orchestra.run.smithi114.stdout: "max": "", 2024-04-23T10:41:09.366 INFO:teuthology.orchestra.run.smithi114.stdout: "enum_allowed": [], 2024-04-23T10:41:09.367 INFO:teuthology.orchestra.run.smithi114.stdout: "desc": "", 2024-04-23T10:41:09.367 INFO:teuthology.orchestra.run.smithi114.stdout: "long_desc": "", 2024-04-23T10:41:09.367 INFO:teuthology.orchestra.run.smithi114.stdout: "tags": [], 2024-04-23T10:41:09.367 INFO:teuthology.orchestra.run.smithi114.stdout: "see_also": [] 2024-04-23T10:41:09.367 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-23T10:41:09.367 INFO:teuthology.orchestra.run.smithi114.stdout: "enabled": { 2024-04-23T10:41:09.367 INFO:teuthology.orchestra.run.smithi114.stdout: "name": "enabled", 2024-04-23T10:41:09.367 INFO:teuthology.orchestra.run.smithi114.stdout: "type": "bool", 2024-04-23T10:41:09.367 INFO:teuthology.orchestra.run.smithi114.stdout: "level": "advanced", 2024-04-23T10:41:09.368 INFO:teuthology.orchestra.run.smithi114.stdout: "flags": 0, 2024-04-23T10:41:09.368 INFO:teuthology.orchestra.run.smithi114.stdout: "default_value": "False", 2024-04-23T10:41:09.368 INFO:teuthology.orchestra.run.smithi114.stdout: "min": "", 2024-04-23T10:41:09.368 INFO:teuthology.orchestra.run.smithi114.stdout: "max": "", 2024-04-23T10:41:09.368 INFO:teuthology.orchestra.run.smithi114.stdout: "enum_allowed": [], 2024-04-23T10:41:09.368 INFO:teuthology.orchestra.run.smithi114.stdout: "desc": "", 2024-04-23T10:41:09.368 INFO:teuthology.orchestra.run.smithi114.stdout: "long_desc": "", 2024-04-23T10:41:09.368 INFO:teuthology.orchestra.run.smithi114.stdout: "tags": [], 2024-04-23T10:41:09.368 INFO:teuthology.orchestra.run.smithi114.stdout: "see_also": [] 2024-04-23T10:41:09.368 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-23T10:41:09.369 INFO:teuthology.orchestra.run.smithi114.stdout: "interval": { 2024-04-23T10:41:09.369 INFO:teuthology.orchestra.run.smithi114.stdout: "name": "interval", 2024-04-23T10:41:09.369 INFO:teuthology.orchestra.run.smithi114.stdout: "type": "int", 2024-04-23T10:41:09.369 INFO:teuthology.orchestra.run.smithi114.stdout: "level": "advanced", 2024-04-23T10:41:09.369 INFO:teuthology.orchestra.run.smithi114.stdout: "flags": 0, 2024-04-23T10:41:09.369 INFO:teuthology.orchestra.run.smithi114.stdout: "default_value": "24", 2024-04-23T10:41:09.369 INFO:teuthology.orchestra.run.smithi114.stdout: "min": "8", 2024-04-23T10:41:09.369 INFO:teuthology.orchestra.run.smithi114.stdout: "max": "", 2024-04-23T10:41:09.369 INFO:teuthology.orchestra.run.smithi114.stdout: "enum_allowed": [], 2024-04-23T10:41:09.370 INFO:teuthology.orchestra.run.smithi114.stdout: "desc": "", 2024-04-23T10:41:09.370 INFO:teuthology.orchestra.run.smithi114.stdout: "long_desc": "", 2024-04-23T10:41:09.370 INFO:teuthology.orchestra.run.smithi114.stdout: "tags": [], 2024-04-23T10:41:09.370 INFO:teuthology.orchestra.run.smithi114.stdout: "see_also": [] 2024-04-23T10:41:09.370 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-23T10:41:09.370 INFO:teuthology.orchestra.run.smithi114.stdout: "last_opt_revision": { 2024-04-23T10:41:09.370 INFO:teuthology.orchestra.run.smithi114.stdout: "name": "last_opt_revision", 2024-04-23T10:41:09.370 INFO:teuthology.orchestra.run.smithi114.stdout: "type": "int", 2024-04-23T10:41:09.370 INFO:teuthology.orchestra.run.smithi114.stdout: "level": "advanced", 2024-04-23T10:41:09.371 INFO:teuthology.orchestra.run.smithi114.stdout: "flags": 0, 2024-04-23T10:41:09.371 INFO:teuthology.orchestra.run.smithi114.stdout: "default_value": "1", 2024-04-23T10:41:09.371 INFO:teuthology.orchestra.run.smithi114.stdout: "min": "", 2024-04-23T10:41:09.371 INFO:teuthology.orchestra.run.smithi114.stdout: "max": "", 2024-04-23T10:41:09.371 INFO:teuthology.orchestra.run.smithi114.stdout: "enum_allowed": [], 2024-04-23T10:41:09.371 INFO:teuthology.orchestra.run.smithi114.stdout: "desc": "", 2024-04-23T10:41:09.371 INFO:teuthology.orchestra.run.smithi114.stdout: "long_desc": "", 2024-04-23T10:41:09.371 INFO:teuthology.orchestra.run.smithi114.stdout: "tags": [], 2024-04-23T10:41:09.371 INFO:teuthology.orchestra.run.smithi114.stdout: "see_also": [] 2024-04-23T10:41:09.372 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-23T10:41:09.372 INFO:teuthology.orchestra.run.smithi114.stdout: "leaderboard": { 2024-04-23T10:41:09.372 INFO:teuthology.orchestra.run.smithi114.stdout: "name": "leaderboard", 2024-04-23T10:41:09.372 INFO:teuthology.orchestra.run.smithi114.stdout: "type": "bool", 2024-04-23T10:41:09.372 INFO:teuthology.orchestra.run.smithi114.stdout: "level": "advanced", 2024-04-23T10:41:09.372 INFO:teuthology.orchestra.run.smithi114.stdout: "flags": 0, 2024-04-23T10:41:09.372 INFO:teuthology.orchestra.run.smithi114.stdout: "default_value": "False", 2024-04-23T10:41:09.372 INFO:teuthology.orchestra.run.smithi114.stdout: "min": "", 2024-04-23T10:41:09.373 INFO:teuthology.orchestra.run.smithi114.stdout: "max": "", 2024-04-23T10:41:09.373 INFO:teuthology.orchestra.run.smithi114.stdout: "enum_allowed": [], 2024-04-23T10:41:09.373 INFO:teuthology.orchestra.run.smithi114.stdout: "desc": "", 2024-04-23T10:41:09.373 INFO:teuthology.orchestra.run.smithi114.stdout: "long_desc": "", 2024-04-23T10:41:09.373 INFO:teuthology.orchestra.run.smithi114.stdout: "tags": [], 2024-04-23T10:41:09.373 INFO:teuthology.orchestra.run.smithi114.stdout: "see_also": [] 2024-04-23T10:41:09.373 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-23T10:41:09.373 INFO:teuthology.orchestra.run.smithi114.stdout: "leaderboard_description": { 2024-04-23T10:41:09.373 INFO:teuthology.orchestra.run.smithi114.stdout: "name": "leaderboard_description", 2024-04-23T10:41:09.374 INFO:teuthology.orchestra.run.smithi114.stdout: "type": "str", 2024-04-23T10:41:09.374 INFO:teuthology.orchestra.run.smithi114.stdout: "level": "advanced", 2024-04-23T10:41:09.374 INFO:teuthology.orchestra.run.smithi114.stdout: "flags": 0, 2024-04-23T10:41:09.374 INFO:teuthology.orchestra.run.smithi114.stdout: "default_value": "", 2024-04-23T10:41:09.374 INFO:teuthology.orchestra.run.smithi114.stdout: "min": "", 2024-04-23T10:41:09.374 INFO:teuthology.orchestra.run.smithi114.stdout: "max": "", 2024-04-23T10:41:09.374 INFO:teuthology.orchestra.run.smithi114.stdout: "enum_allowed": [], 2024-04-23T10:41:09.374 INFO:teuthology.orchestra.run.smithi114.stdout: "desc": "", 2024-04-23T10:41:09.374 INFO:teuthology.orchestra.run.smithi114.stdout: "long_desc": "", 2024-04-23T10:41:09.375 INFO:teuthology.orchestra.run.smithi114.stdout: "tags": [], 2024-04-23T10:41:09.375 INFO:teuthology.orchestra.run.smithi114.stdout: "see_also": [] 2024-04-23T10:41:09.375 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-23T10:41:09.375 INFO:teuthology.orchestra.run.smithi114.stdout: "log_level": { 2024-04-23T10:41:09.375 INFO:teuthology.orchestra.run.smithi114.stdout: "name": "log_level", 2024-04-23T10:41:09.375 INFO:teuthology.orchestra.run.smithi114.stdout: "type": "str", 2024-04-23T10:41:09.375 INFO:teuthology.orchestra.run.smithi114.stdout: "level": "advanced", 2024-04-23T10:41:09.375 INFO:teuthology.orchestra.run.smithi114.stdout: "flags": 1, 2024-04-23T10:41:09.375 INFO:teuthology.orchestra.run.smithi114.stdout: "default_value": "", 2024-04-23T10:41:09.376 INFO:teuthology.orchestra.run.smithi114.stdout: "min": "", 2024-04-23T10:41:09.376 INFO:teuthology.orchestra.run.smithi114.stdout: "max": "", 2024-04-23T10:41:09.376 INFO:teuthology.orchestra.run.smithi114.stdout: "enum_allowed": [ 2024-04-23T10:41:09.376 INFO:teuthology.orchestra.run.smithi114.stdout: "", 2024-04-23T10:41:09.376 INFO:teuthology.orchestra.run.smithi114.stdout: "critical", 2024-04-23T10:41:09.376 INFO:teuthology.orchestra.run.smithi114.stdout: "debug", 2024-04-23T10:41:09.376 INFO:teuthology.orchestra.run.smithi114.stdout: "error", 2024-04-23T10:41:09.376 INFO:teuthology.orchestra.run.smithi114.stdout: "info", 2024-04-23T10:41:09.376 INFO:teuthology.orchestra.run.smithi114.stdout: "warning" 2024-04-23T10:41:09.377 INFO:teuthology.orchestra.run.smithi114.stdout: ], 2024-04-23T10:41:09.377 INFO:teuthology.orchestra.run.smithi114.stdout: "desc": "", 2024-04-23T10:41:09.377 INFO:teuthology.orchestra.run.smithi114.stdout: "long_desc": "", 2024-04-23T10:41:09.377 INFO:teuthology.orchestra.run.smithi114.stdout: "tags": [], 2024-04-23T10:41:09.377 INFO:teuthology.orchestra.run.smithi114.stdout: "see_also": [] 2024-04-23T10:41:09.377 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-23T10:41:09.377 INFO:teuthology.orchestra.run.smithi114.stdout: "log_to_cluster": { 2024-04-23T10:41:09.377 INFO:teuthology.orchestra.run.smithi114.stdout: "name": "log_to_cluster", 2024-04-23T10:41:09.378 INFO:teuthology.orchestra.run.smithi114.stdout: "type": "bool", 2024-04-23T10:41:09.378 INFO:teuthology.orchestra.run.smithi114.stdout: "level": "advanced", 2024-04-23T10:41:09.378 INFO:teuthology.orchestra.run.smithi114.stdout: "flags": 1, 2024-04-23T10:41:09.378 INFO:teuthology.orchestra.run.smithi114.stdout: "default_value": "False", 2024-04-23T10:41:09.378 INFO:teuthology.orchestra.run.smithi114.stdout: "min": "", 2024-04-23T10:41:09.378 INFO:teuthology.orchestra.run.smithi114.stdout: "max": "", 2024-04-23T10:41:09.378 INFO:teuthology.orchestra.run.smithi114.stdout: "enum_allowed": [], 2024-04-23T10:41:09.378 INFO:teuthology.orchestra.run.smithi114.stdout: "desc": "", 2024-04-23T10:41:09.379 INFO:teuthology.orchestra.run.smithi114.stdout: "long_desc": "", 2024-04-23T10:41:09.379 INFO:teuthology.orchestra.run.smithi114.stdout: "tags": [], 2024-04-23T10:41:09.379 INFO:teuthology.orchestra.run.smithi114.stdout: "see_also": [] 2024-04-23T10:41:09.379 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-23T10:41:09.379 INFO:teuthology.orchestra.run.smithi114.stdout: "log_to_cluster_level": { 2024-04-23T10:41:09.379 INFO:teuthology.orchestra.run.smithi114.stdout: "name": "log_to_cluster_level", 2024-04-23T10:41:09.379 INFO:teuthology.orchestra.run.smithi114.stdout: "type": "str", 2024-04-23T10:41:09.379 INFO:teuthology.orchestra.run.smithi114.stdout: "level": "advanced", 2024-04-23T10:41:09.379 INFO:teuthology.orchestra.run.smithi114.stdout: "flags": 1, 2024-04-23T10:41:09.380 INFO:teuthology.orchestra.run.smithi114.stdout: "default_value": "info", 2024-04-23T10:41:09.380 INFO:teuthology.orchestra.run.smithi114.stdout: "min": "", 2024-04-23T10:41:09.380 INFO:teuthology.orchestra.run.smithi114.stdout: "max": "", 2024-04-23T10:41:09.380 INFO:teuthology.orchestra.run.smithi114.stdout: "enum_allowed": [ 2024-04-23T10:41:09.380 INFO:teuthology.orchestra.run.smithi114.stdout: "", 2024-04-23T10:41:09.380 INFO:teuthology.orchestra.run.smithi114.stdout: "critical", 2024-04-23T10:41:09.380 INFO:teuthology.orchestra.run.smithi114.stdout: "debug", 2024-04-23T10:41:09.380 INFO:teuthology.orchestra.run.smithi114.stdout: "error", 2024-04-23T10:41:09.380 INFO:teuthology.orchestra.run.smithi114.stdout: "info", 2024-04-23T10:41:09.381 INFO:teuthology.orchestra.run.smithi114.stdout: "warning" 2024-04-23T10:41:09.381 INFO:teuthology.orchestra.run.smithi114.stdout: ], 2024-04-23T10:41:09.381 INFO:teuthology.orchestra.run.smithi114.stdout: "desc": "", 2024-04-23T10:41:09.381 INFO:teuthology.orchestra.run.smithi114.stdout: "long_desc": "", 2024-04-23T10:41:09.381 INFO:teuthology.orchestra.run.smithi114.stdout: "tags": [], 2024-04-23T10:41:09.381 INFO:teuthology.orchestra.run.smithi114.stdout: "see_also": [] 2024-04-23T10:41:09.381 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-23T10:41:09.381 INFO:teuthology.orchestra.run.smithi114.stdout: "log_to_file": { 2024-04-23T10:41:09.381 INFO:teuthology.orchestra.run.smithi114.stdout: "name": "log_to_file", 2024-04-23T10:41:09.381 INFO:teuthology.orchestra.run.smithi114.stdout: "type": "bool", 2024-04-23T10:41:09.382 INFO:teuthology.orchestra.run.smithi114.stdout: "level": "advanced", 2024-04-23T10:41:09.382 INFO:teuthology.orchestra.run.smithi114.stdout: "flags": 1, 2024-04-23T10:41:09.382 INFO:teuthology.orchestra.run.smithi114.stdout: "default_value": "False", 2024-04-23T10:41:09.382 INFO:teuthology.orchestra.run.smithi114.stdout: "min": "", 2024-04-23T10:41:09.382 INFO:teuthology.orchestra.run.smithi114.stdout: "max": "", 2024-04-23T10:41:09.382 INFO:teuthology.orchestra.run.smithi114.stdout: "enum_allowed": [], 2024-04-23T10:41:09.382 INFO:teuthology.orchestra.run.smithi114.stdout: "desc": "", 2024-04-23T10:41:09.382 INFO:teuthology.orchestra.run.smithi114.stdout: "long_desc": "", 2024-04-23T10:41:09.383 INFO:teuthology.orchestra.run.smithi114.stdout: "tags": [], 2024-04-23T10:41:09.383 INFO:teuthology.orchestra.run.smithi114.stdout: "see_also": [] 2024-04-23T10:41:09.383 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-23T10:41:09.383 INFO:teuthology.orchestra.run.smithi114.stdout: "organization": { 2024-04-23T10:41:09.383 INFO:teuthology.orchestra.run.smithi114.stdout: "name": "organization", 2024-04-23T10:41:09.383 INFO:teuthology.orchestra.run.smithi114.stdout: "type": "str", 2024-04-23T10:41:09.383 INFO:teuthology.orchestra.run.smithi114.stdout: "level": "advanced", 2024-04-23T10:41:09.383 INFO:teuthology.orchestra.run.smithi114.stdout: "flags": 0, 2024-04-23T10:41:09.383 INFO:teuthology.orchestra.run.smithi114.stdout: "default_value": "", 2024-04-23T10:41:09.384 INFO:teuthology.orchestra.run.smithi114.stdout: "min": "", 2024-04-23T10:41:09.384 INFO:teuthology.orchestra.run.smithi114.stdout: "max": "", 2024-04-23T10:41:09.384 INFO:teuthology.orchestra.run.smithi114.stdout: "enum_allowed": [], 2024-04-23T10:41:09.384 INFO:teuthology.orchestra.run.smithi114.stdout: "desc": "", 2024-04-23T10:41:09.384 INFO:teuthology.orchestra.run.smithi114.stdout: "long_desc": "", 2024-04-23T10:41:09.384 INFO:teuthology.orchestra.run.smithi114.stdout: "tags": [], 2024-04-23T10:41:09.384 INFO:teuthology.orchestra.run.smithi114.stdout: "see_also": [] 2024-04-23T10:41:09.384 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-23T10:41:09.384 INFO:teuthology.orchestra.run.smithi114.stdout: "proxy": { 2024-04-23T10:41:09.384 INFO:teuthology.orchestra.run.smithi114.stdout: "name": "proxy", 2024-04-23T10:41:09.385 INFO:teuthology.orchestra.run.smithi114.stdout: "type": "str", 2024-04-23T10:41:09.385 INFO:teuthology.orchestra.run.smithi114.stdout: "level": "advanced", 2024-04-23T10:41:09.385 INFO:teuthology.orchestra.run.smithi114.stdout: "flags": 0, 2024-04-23T10:41:09.385 INFO:teuthology.orchestra.run.smithi114.stdout: "default_value": "", 2024-04-23T10:41:09.385 INFO:teuthology.orchestra.run.smithi114.stdout: "min": "", 2024-04-23T10:41:09.385 INFO:teuthology.orchestra.run.smithi114.stdout: "max": "", 2024-04-23T10:41:09.385 INFO:teuthology.orchestra.run.smithi114.stdout: "enum_allowed": [], 2024-04-23T10:41:09.385 INFO:teuthology.orchestra.run.smithi114.stdout: "desc": "", 2024-04-23T10:41:09.385 INFO:teuthology.orchestra.run.smithi114.stdout: "long_desc": "", 2024-04-23T10:41:09.386 INFO:teuthology.orchestra.run.smithi114.stdout: "tags": [], 2024-04-23T10:41:09.386 INFO:teuthology.orchestra.run.smithi114.stdout: "see_also": [] 2024-04-23T10:41:09.386 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-23T10:41:09.386 INFO:teuthology.orchestra.run.smithi114.stdout: "url": { 2024-04-23T10:41:09.386 INFO:teuthology.orchestra.run.smithi114.stdout: "name": "url", 2024-04-23T10:41:09.386 INFO:teuthology.orchestra.run.smithi114.stdout: "type": "str", 2024-04-23T10:41:09.386 INFO:teuthology.orchestra.run.smithi114.stdout: "level": "advanced", 2024-04-23T10:41:09.386 INFO:teuthology.orchestra.run.smithi114.stdout: "flags": 0, 2024-04-23T10:41:09.387 INFO:teuthology.orchestra.run.smithi114.stdout: "default_value": "https://telemetry.ceph.com/report", 2024-04-23T10:41:09.387 INFO:teuthology.orchestra.run.smithi114.stdout: "min": "", 2024-04-23T10:41:09.387 INFO:teuthology.orchestra.run.smithi114.stdout: "max": "", 2024-04-23T10:41:09.387 INFO:teuthology.orchestra.run.smithi114.stdout: "enum_allowed": [], 2024-04-23T10:41:09.387 INFO:teuthology.orchestra.run.smithi114.stdout: "desc": "", 2024-04-23T10:41:09.387 INFO:teuthology.orchestra.run.smithi114.stdout: "long_desc": "", 2024-04-23T10:41:09.387 INFO:teuthology.orchestra.run.smithi114.stdout: "tags": [], 2024-04-23T10:41:09.387 INFO:teuthology.orchestra.run.smithi114.stdout: "see_also": [] 2024-04-23T10:41:09.388 INFO:teuthology.orchestra.run.smithi114.stdout: } 2024-04-23T10:41:09.388 INFO:teuthology.orchestra.run.smithi114.stdout: } 2024-04-23T10:41:09.388 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-23T10:41:09.388 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-04-23T10:41:09.388 INFO:teuthology.orchestra.run.smithi114.stdout: "name": "test_orchestrator", 2024-04-23T10:41:09.388 INFO:teuthology.orchestra.run.smithi114.stdout: "can_run": true, 2024-04-23T10:41:09.388 INFO:teuthology.orchestra.run.smithi114.stdout: "error_string": "", 2024-04-23T10:41:09.388 INFO:teuthology.orchestra.run.smithi114.stdout: "module_options": { 2024-04-23T10:41:09.388 INFO:teuthology.orchestra.run.smithi114.stdout: "log_level": { 2024-04-23T10:41:09.389 INFO:teuthology.orchestra.run.smithi114.stdout: "name": "log_level", 2024-04-23T10:41:09.389 INFO:teuthology.orchestra.run.smithi114.stdout: "type": "str", 2024-04-23T10:41:09.389 INFO:teuthology.orchestra.run.smithi114.stdout: "level": "advanced", 2024-04-23T10:41:09.389 INFO:teuthology.orchestra.run.smithi114.stdout: "flags": 1, 2024-04-23T10:41:09.389 INFO:teuthology.orchestra.run.smithi114.stdout: "default_value": "", 2024-04-23T10:41:09.389 INFO:teuthology.orchestra.run.smithi114.stdout: "min": "", 2024-04-23T10:41:09.389 INFO:teuthology.orchestra.run.smithi114.stdout: "max": "", 2024-04-23T10:41:09.389 INFO:teuthology.orchestra.run.smithi114.stdout: "enum_allowed": [ 2024-04-23T10:41:09.389 INFO:teuthology.orchestra.run.smithi114.stdout: "", 2024-04-23T10:41:09.390 INFO:teuthology.orchestra.run.smithi114.stdout: "critical", 2024-04-23T10:41:09.390 INFO:teuthology.orchestra.run.smithi114.stdout: "debug", 2024-04-23T10:41:09.390 INFO:teuthology.orchestra.run.smithi114.stdout: "error", 2024-04-23T10:41:09.390 INFO:teuthology.orchestra.run.smithi114.stdout: "info", 2024-04-23T10:41:09.390 INFO:teuthology.orchestra.run.smithi114.stdout: "warning" 2024-04-23T10:41:09.390 INFO:teuthology.orchestra.run.smithi114.stdout: ], 2024-04-23T10:41:09.390 INFO:teuthology.orchestra.run.smithi114.stdout: "desc": "", 2024-04-23T10:41:09.390 INFO:teuthology.orchestra.run.smithi114.stdout: "long_desc": "", 2024-04-23T10:41:09.390 INFO:teuthology.orchestra.run.smithi114.stdout: "tags": [], 2024-04-23T10:41:09.391 INFO:teuthology.orchestra.run.smithi114.stdout: "see_also": [] 2024-04-23T10:41:09.391 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-23T10:41:09.391 INFO:teuthology.orchestra.run.smithi114.stdout: "log_to_cluster": { 2024-04-23T10:41:09.391 INFO:teuthology.orchestra.run.smithi114.stdout: "name": "log_to_cluster", 2024-04-23T10:41:09.391 INFO:teuthology.orchestra.run.smithi114.stdout: "type": "bool", 2024-04-23T10:41:09.391 INFO:teuthology.orchestra.run.smithi114.stdout: "level": "advanced", 2024-04-23T10:41:09.391 INFO:teuthology.orchestra.run.smithi114.stdout: "flags": 1, 2024-04-23T10:41:09.391 INFO:teuthology.orchestra.run.smithi114.stdout: "default_value": "False", 2024-04-23T10:41:09.392 INFO:teuthology.orchestra.run.smithi114.stdout: "min": "", 2024-04-23T10:41:09.392 INFO:teuthology.orchestra.run.smithi114.stdout: "max": "", 2024-04-23T10:41:09.392 INFO:teuthology.orchestra.run.smithi114.stdout: "enum_allowed": [], 2024-04-23T10:41:09.392 INFO:teuthology.orchestra.run.smithi114.stdout: "desc": "", 2024-04-23T10:41:09.392 INFO:teuthology.orchestra.run.smithi114.stdout: "long_desc": "", 2024-04-23T10:41:09.392 INFO:teuthology.orchestra.run.smithi114.stdout: "tags": [], 2024-04-23T10:41:09.392 INFO:teuthology.orchestra.run.smithi114.stdout: "see_also": [] 2024-04-23T10:41:09.392 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-23T10:41:09.392 INFO:teuthology.orchestra.run.smithi114.stdout: "log_to_cluster_level": { 2024-04-23T10:41:09.393 INFO:teuthology.orchestra.run.smithi114.stdout: "name": "log_to_cluster_level", 2024-04-23T10:41:09.393 INFO:teuthology.orchestra.run.smithi114.stdout: "type": "str", 2024-04-23T10:41:09.393 INFO:teuthology.orchestra.run.smithi114.stdout: "level": "advanced", 2024-04-23T10:41:09.393 INFO:teuthology.orchestra.run.smithi114.stdout: "flags": 1, 2024-04-23T10:41:09.393 INFO:teuthology.orchestra.run.smithi114.stdout: "default_value": "info", 2024-04-23T10:41:09.393 INFO:teuthology.orchestra.run.smithi114.stdout: "min": "", 2024-04-23T10:41:09.393 INFO:teuthology.orchestra.run.smithi114.stdout: "max": "", 2024-04-23T10:41:09.393 INFO:teuthology.orchestra.run.smithi114.stdout: "enum_allowed": [ 2024-04-23T10:41:09.393 INFO:teuthology.orchestra.run.smithi114.stdout: "", 2024-04-23T10:41:09.394 INFO:teuthology.orchestra.run.smithi114.stdout: "critical", 2024-04-23T10:41:09.394 INFO:teuthology.orchestra.run.smithi114.stdout: "debug", 2024-04-23T10:41:09.394 INFO:teuthology.orchestra.run.smithi114.stdout: "error", 2024-04-23T10:41:09.394 INFO:teuthology.orchestra.run.smithi114.stdout: "info", 2024-04-23T10:41:09.394 INFO:teuthology.orchestra.run.smithi114.stdout: "warning" 2024-04-23T10:41:09.394 INFO:teuthology.orchestra.run.smithi114.stdout: ], 2024-04-23T10:41:09.394 INFO:teuthology.orchestra.run.smithi114.stdout: "desc": "", 2024-04-23T10:41:09.394 INFO:teuthology.orchestra.run.smithi114.stdout: "long_desc": "", 2024-04-23T10:41:09.395 INFO:teuthology.orchestra.run.smithi114.stdout: "tags": [], 2024-04-23T10:41:09.395 INFO:teuthology.orchestra.run.smithi114.stdout: "see_also": [] 2024-04-23T10:41:09.395 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-23T10:41:09.395 INFO:teuthology.orchestra.run.smithi114.stdout: "log_to_file": { 2024-04-23T10:41:09.395 INFO:teuthology.orchestra.run.smithi114.stdout: "name": "log_to_file", 2024-04-23T10:41:09.395 INFO:teuthology.orchestra.run.smithi114.stdout: "type": "bool", 2024-04-23T10:41:09.395 INFO:teuthology.orchestra.run.smithi114.stdout: "level": "advanced", 2024-04-23T10:41:09.395 INFO:teuthology.orchestra.run.smithi114.stdout: "flags": 1, 2024-04-23T10:41:09.395 INFO:teuthology.orchestra.run.smithi114.stdout: "default_value": "False", 2024-04-23T10:41:09.396 INFO:teuthology.orchestra.run.smithi114.stdout: "min": "", 2024-04-23T10:41:09.396 INFO:teuthology.orchestra.run.smithi114.stdout: "max": "", 2024-04-23T10:41:09.396 INFO:teuthology.orchestra.run.smithi114.stdout: "enum_allowed": [], 2024-04-23T10:41:09.396 INFO:teuthology.orchestra.run.smithi114.stdout: "desc": "", 2024-04-23T10:41:09.396 INFO:teuthology.orchestra.run.smithi114.stdout: "long_desc": "", 2024-04-23T10:41:09.396 INFO:teuthology.orchestra.run.smithi114.stdout: "tags": [], 2024-04-23T10:41:09.396 INFO:teuthology.orchestra.run.smithi114.stdout: "see_also": [] 2024-04-23T10:41:09.396 INFO:teuthology.orchestra.run.smithi114.stdout: } 2024-04-23T10:41:09.396 INFO:teuthology.orchestra.run.smithi114.stdout: } 2024-04-23T10:41:09.396 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-23T10:41:09.397 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-04-23T10:41:09.397 INFO:teuthology.orchestra.run.smithi114.stdout: "name": "volumes", 2024-04-23T10:41:09.397 INFO:teuthology.orchestra.run.smithi114.stdout: "can_run": true, 2024-04-23T10:41:09.397 INFO:teuthology.orchestra.run.smithi114.stdout: "error_string": "", 2024-04-23T10:41:09.397 INFO:teuthology.orchestra.run.smithi114.stdout: "module_options": { 2024-04-23T10:41:09.397 INFO:teuthology.orchestra.run.smithi114.stdout: "log_level": { 2024-04-23T10:41:09.397 INFO:teuthology.orchestra.run.smithi114.stdout: "name": "log_level", 2024-04-23T10:41:09.397 INFO:teuthology.orchestra.run.smithi114.stdout: "type": "str", 2024-04-23T10:41:09.397 INFO:teuthology.orchestra.run.smithi114.stdout: "level": "advanced", 2024-04-23T10:41:09.398 INFO:teuthology.orchestra.run.smithi114.stdout: "flags": 1, 2024-04-23T10:41:09.398 INFO:teuthology.orchestra.run.smithi114.stdout: "default_value": "", 2024-04-23T10:41:09.398 INFO:teuthology.orchestra.run.smithi114.stdout: "min": "", 2024-04-23T10:41:09.398 INFO:teuthology.orchestra.run.smithi114.stdout: "max": "", 2024-04-23T10:41:09.398 INFO:teuthology.orchestra.run.smithi114.stdout: "enum_allowed": [ 2024-04-23T10:41:09.398 INFO:teuthology.orchestra.run.smithi114.stdout: "", 2024-04-23T10:41:09.398 INFO:teuthology.orchestra.run.smithi114.stdout: "critical", 2024-04-23T10:41:09.398 INFO:teuthology.orchestra.run.smithi114.stdout: "debug", 2024-04-23T10:41:09.398 INFO:teuthology.orchestra.run.smithi114.stdout: "error", 2024-04-23T10:41:09.399 INFO:teuthology.orchestra.run.smithi114.stdout: "info", 2024-04-23T10:41:09.399 INFO:teuthology.orchestra.run.smithi114.stdout: "warning" 2024-04-23T10:41:09.399 INFO:teuthology.orchestra.run.smithi114.stdout: ], 2024-04-23T10:41:09.399 INFO:teuthology.orchestra.run.smithi114.stdout: "desc": "", 2024-04-23T10:41:09.399 INFO:teuthology.orchestra.run.smithi114.stdout: "long_desc": "", 2024-04-23T10:41:09.399 INFO:teuthology.orchestra.run.smithi114.stdout: "tags": [], 2024-04-23T10:41:09.399 INFO:teuthology.orchestra.run.smithi114.stdout: "see_also": [] 2024-04-23T10:41:09.399 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-23T10:41:09.399 INFO:teuthology.orchestra.run.smithi114.stdout: "log_to_cluster": { 2024-04-23T10:41:09.400 INFO:teuthology.orchestra.run.smithi114.stdout: "name": "log_to_cluster", 2024-04-23T10:41:09.400 INFO:teuthology.orchestra.run.smithi114.stdout: "type": "bool", 2024-04-23T10:41:09.400 INFO:teuthology.orchestra.run.smithi114.stdout: "level": "advanced", 2024-04-23T10:41:09.400 INFO:teuthology.orchestra.run.smithi114.stdout: "flags": 1, 2024-04-23T10:41:09.400 INFO:teuthology.orchestra.run.smithi114.stdout: "default_value": "False", 2024-04-23T10:41:09.400 INFO:teuthology.orchestra.run.smithi114.stdout: "min": "", 2024-04-23T10:41:09.400 INFO:teuthology.orchestra.run.smithi114.stdout: "max": "", 2024-04-23T10:41:09.400 INFO:teuthology.orchestra.run.smithi114.stdout: "enum_allowed": [], 2024-04-23T10:41:09.400 INFO:teuthology.orchestra.run.smithi114.stdout: "desc": "", 2024-04-23T10:41:09.401 INFO:teuthology.orchestra.run.smithi114.stdout: "long_desc": "", 2024-04-23T10:41:09.401 INFO:teuthology.orchestra.run.smithi114.stdout: "tags": [], 2024-04-23T10:41:09.401 INFO:teuthology.orchestra.run.smithi114.stdout: "see_also": [] 2024-04-23T10:41:09.401 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-23T10:41:09.401 INFO:teuthology.orchestra.run.smithi114.stdout: "log_to_cluster_level": { 2024-04-23T10:41:09.401 INFO:teuthology.orchestra.run.smithi114.stdout: "name": "log_to_cluster_level", 2024-04-23T10:41:09.401 INFO:teuthology.orchestra.run.smithi114.stdout: "type": "str", 2024-04-23T10:41:09.401 INFO:teuthology.orchestra.run.smithi114.stdout: "level": "advanced", 2024-04-23T10:41:09.401 INFO:teuthology.orchestra.run.smithi114.stdout: "flags": 1, 2024-04-23T10:41:09.402 INFO:teuthology.orchestra.run.smithi114.stdout: "default_value": "info", 2024-04-23T10:41:09.402 INFO:teuthology.orchestra.run.smithi114.stdout: "min": "", 2024-04-23T10:41:09.402 INFO:teuthology.orchestra.run.smithi114.stdout: "max": "", 2024-04-23T10:41:09.402 INFO:teuthology.orchestra.run.smithi114.stdout: "enum_allowed": [ 2024-04-23T10:41:09.402 INFO:teuthology.orchestra.run.smithi114.stdout: "", 2024-04-23T10:41:09.406 INFO:teuthology.orchestra.run.smithi114.stdout: "critical", 2024-04-23T10:41:09.406 INFO:teuthology.orchestra.run.smithi114.stdout: "debug", 2024-04-23T10:41:09.406 INFO:teuthology.orchestra.run.smithi114.stdout: "error", 2024-04-23T10:41:09.406 INFO:teuthology.orchestra.run.smithi114.stdout: "info", 2024-04-23T10:41:09.406 INFO:teuthology.orchestra.run.smithi114.stdout: "warning" 2024-04-23T10:41:09.406 INFO:teuthology.orchestra.run.smithi114.stdout: ], 2024-04-23T10:41:09.407 INFO:teuthology.orchestra.run.smithi114.stdout: "desc": "", 2024-04-23T10:41:09.407 INFO:teuthology.orchestra.run.smithi114.stdout: "long_desc": "", 2024-04-23T10:41:09.407 INFO:teuthology.orchestra.run.smithi114.stdout: "tags": [], 2024-04-23T10:41:09.407 INFO:teuthology.orchestra.run.smithi114.stdout: "see_also": [] 2024-04-23T10:41:09.407 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-23T10:41:09.407 INFO:teuthology.orchestra.run.smithi114.stdout: "log_to_file": { 2024-04-23T10:41:09.407 INFO:teuthology.orchestra.run.smithi114.stdout: "name": "log_to_file", 2024-04-23T10:41:09.407 INFO:teuthology.orchestra.run.smithi114.stdout: "type": "bool", 2024-04-23T10:41:09.407 INFO:teuthology.orchestra.run.smithi114.stdout: "level": "advanced", 2024-04-23T10:41:09.408 INFO:teuthology.orchestra.run.smithi114.stdout: "flags": 1, 2024-04-23T10:41:09.408 INFO:teuthology.orchestra.run.smithi114.stdout: "default_value": "False", 2024-04-23T10:41:09.408 INFO:teuthology.orchestra.run.smithi114.stdout: "min": "", 2024-04-23T10:41:09.408 INFO:teuthology.orchestra.run.smithi114.stdout: "max": "", 2024-04-23T10:41:09.408 INFO:teuthology.orchestra.run.smithi114.stdout: "enum_allowed": [], 2024-04-23T10:41:09.408 INFO:teuthology.orchestra.run.smithi114.stdout: "desc": "", 2024-04-23T10:41:09.408 INFO:teuthology.orchestra.run.smithi114.stdout: "long_desc": "", 2024-04-23T10:41:09.408 INFO:teuthology.orchestra.run.smithi114.stdout: "tags": [], 2024-04-23T10:41:09.408 INFO:teuthology.orchestra.run.smithi114.stdout: "see_also": [] 2024-04-23T10:41:09.409 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-23T10:41:09.409 INFO:teuthology.orchestra.run.smithi114.stdout: "max_concurrent_clones": { 2024-04-23T10:41:09.409 INFO:teuthology.orchestra.run.smithi114.stdout: "name": "max_concurrent_clones", 2024-04-23T10:41:09.409 INFO:teuthology.orchestra.run.smithi114.stdout: "type": "int", 2024-04-23T10:41:09.409 INFO:teuthology.orchestra.run.smithi114.stdout: "level": "advanced", 2024-04-23T10:41:09.409 INFO:teuthology.orchestra.run.smithi114.stdout: "flags": 0, 2024-04-23T10:41:09.409 INFO:teuthology.orchestra.run.smithi114.stdout: "default_value": "4", 2024-04-23T10:41:09.409 INFO:teuthology.orchestra.run.smithi114.stdout: "min": "", 2024-04-23T10:41:09.409 INFO:teuthology.orchestra.run.smithi114.stdout: "max": "", 2024-04-23T10:41:09.410 INFO:teuthology.orchestra.run.smithi114.stdout: "enum_allowed": [], 2024-04-23T10:41:09.410 INFO:teuthology.orchestra.run.smithi114.stdout: "desc": "Number of asynchronous cloner threads", 2024-04-23T10:41:09.410 INFO:teuthology.orchestra.run.smithi114.stdout: "long_desc": "", 2024-04-23T10:41:09.410 INFO:teuthology.orchestra.run.smithi114.stdout: "tags": [], 2024-04-23T10:41:09.410 INFO:teuthology.orchestra.run.smithi114.stdout: "see_also": [] 2024-04-23T10:41:09.410 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-23T10:41:09.410 INFO:teuthology.orchestra.run.smithi114.stdout: "periodic_async_work": { 2024-04-23T10:41:09.410 INFO:teuthology.orchestra.run.smithi114.stdout: "name": "periodic_async_work", 2024-04-23T10:41:09.411 INFO:teuthology.orchestra.run.smithi114.stdout: "type": "bool", 2024-04-23T10:41:09.411 INFO:teuthology.orchestra.run.smithi114.stdout: "level": "advanced", 2024-04-23T10:41:09.411 INFO:teuthology.orchestra.run.smithi114.stdout: "flags": 0, 2024-04-23T10:41:09.411 INFO:teuthology.orchestra.run.smithi114.stdout: "default_value": "False", 2024-04-23T10:41:09.411 INFO:teuthology.orchestra.run.smithi114.stdout: "min": "", 2024-04-23T10:41:09.411 INFO:teuthology.orchestra.run.smithi114.stdout: "max": "", 2024-04-23T10:41:09.411 INFO:teuthology.orchestra.run.smithi114.stdout: "enum_allowed": [], 2024-04-23T10:41:09.411 INFO:teuthology.orchestra.run.smithi114.stdout: "desc": "Periodically check for async work", 2024-04-23T10:41:09.411 INFO:teuthology.orchestra.run.smithi114.stdout: "long_desc": "", 2024-04-23T10:41:09.412 INFO:teuthology.orchestra.run.smithi114.stdout: "tags": [], 2024-04-23T10:41:09.412 INFO:teuthology.orchestra.run.smithi114.stdout: "see_also": [] 2024-04-23T10:41:09.412 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-23T10:41:09.412 INFO:teuthology.orchestra.run.smithi114.stdout: "snapshot_clone_delay": { 2024-04-23T10:41:09.412 INFO:teuthology.orchestra.run.smithi114.stdout: "name": "snapshot_clone_delay", 2024-04-23T10:41:09.412 INFO:teuthology.orchestra.run.smithi114.stdout: "type": "int", 2024-04-23T10:41:09.412 INFO:teuthology.orchestra.run.smithi114.stdout: "level": "advanced", 2024-04-23T10:41:09.412 INFO:teuthology.orchestra.run.smithi114.stdout: "flags": 0, 2024-04-23T10:41:09.412 INFO:teuthology.orchestra.run.smithi114.stdout: "default_value": "0", 2024-04-23T10:41:09.413 INFO:teuthology.orchestra.run.smithi114.stdout: "min": "", 2024-04-23T10:41:09.413 INFO:teuthology.orchestra.run.smithi114.stdout: "max": "", 2024-04-23T10:41:09.413 INFO:teuthology.orchestra.run.smithi114.stdout: "enum_allowed": [], 2024-04-23T10:41:09.413 INFO:teuthology.orchestra.run.smithi114.stdout: "desc": "Delay clone begin operation by snapshot_clone_delay seconds", 2024-04-23T10:41:09.413 INFO:teuthology.orchestra.run.smithi114.stdout: "long_desc": "", 2024-04-23T10:41:09.413 INFO:teuthology.orchestra.run.smithi114.stdout: "tags": [], 2024-04-23T10:41:09.413 INFO:teuthology.orchestra.run.smithi114.stdout: "see_also": [] 2024-04-23T10:41:09.413 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-23T10:41:09.413 INFO:teuthology.orchestra.run.smithi114.stdout: "snapshot_clone_no_wait": { 2024-04-23T10:41:09.413 INFO:teuthology.orchestra.run.smithi114.stdout: "name": "snapshot_clone_no_wait", 2024-04-23T10:41:09.414 INFO:teuthology.orchestra.run.smithi114.stdout: "type": "bool", 2024-04-23T10:41:09.414 INFO:teuthology.orchestra.run.smithi114.stdout: "level": "advanced", 2024-04-23T10:41:09.414 INFO:teuthology.orchestra.run.smithi114.stdout: "flags": 0, 2024-04-23T10:41:09.414 INFO:teuthology.orchestra.run.smithi114.stdout: "default_value": "True", 2024-04-23T10:41:09.414 INFO:teuthology.orchestra.run.smithi114.stdout: "min": "", 2024-04-23T10:41:09.414 INFO:teuthology.orchestra.run.smithi114.stdout: "max": "", 2024-04-23T10:41:09.414 INFO:teuthology.orchestra.run.smithi114.stdout: "enum_allowed": [], 2024-04-23T10:41:09.414 INFO:teuthology.orchestra.run.smithi114.stdout: "desc": "Reject subvolume clone request when cloner threads are busy", 2024-04-23T10:41:09.415 INFO:teuthology.orchestra.run.smithi114.stdout: "long_desc": "", 2024-04-23T10:41:09.415 INFO:teuthology.orchestra.run.smithi114.stdout: "tags": [], 2024-04-23T10:41:09.415 INFO:teuthology.orchestra.run.smithi114.stdout: "see_also": [] 2024-04-23T10:41:09.415 INFO:teuthology.orchestra.run.smithi114.stdout: } 2024-04-23T10:41:09.415 INFO:teuthology.orchestra.run.smithi114.stdout: } 2024-04-23T10:41:09.415 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-23T10:41:09.415 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-04-23T10:41:09.415 INFO:teuthology.orchestra.run.smithi114.stdout: "name": "zabbix", 2024-04-23T10:41:09.416 INFO:teuthology.orchestra.run.smithi114.stdout: "can_run": true, 2024-04-23T10:41:09.416 INFO:teuthology.orchestra.run.smithi114.stdout: "error_string": "", 2024-04-23T10:41:09.416 INFO:teuthology.orchestra.run.smithi114.stdout: "module_options": { 2024-04-23T10:41:09.416 INFO:teuthology.orchestra.run.smithi114.stdout: "discovery_interval": { 2024-04-23T10:41:09.416 INFO:teuthology.orchestra.run.smithi114.stdout: "name": "discovery_interval", 2024-04-23T10:41:09.416 INFO:teuthology.orchestra.run.smithi114.stdout: "type": "uint", 2024-04-23T10:41:09.416 INFO:teuthology.orchestra.run.smithi114.stdout: "level": "advanced", 2024-04-23T10:41:09.416 INFO:teuthology.orchestra.run.smithi114.stdout: "flags": 0, 2024-04-23T10:41:09.416 INFO:teuthology.orchestra.run.smithi114.stdout: "default_value": "100", 2024-04-23T10:41:09.417 INFO:teuthology.orchestra.run.smithi114.stdout: "min": "", 2024-04-23T10:41:09.417 INFO:teuthology.orchestra.run.smithi114.stdout: "max": "", 2024-04-23T10:41:09.417 INFO:teuthology.orchestra.run.smithi114.stdout: "enum_allowed": [], 2024-04-23T10:41:09.417 INFO:teuthology.orchestra.run.smithi114.stdout: "desc": "", 2024-04-23T10:41:09.417 INFO:teuthology.orchestra.run.smithi114.stdout: "long_desc": "", 2024-04-23T10:41:09.417 INFO:teuthology.orchestra.run.smithi114.stdout: "tags": [], 2024-04-23T10:41:09.417 INFO:teuthology.orchestra.run.smithi114.stdout: "see_also": [] 2024-04-23T10:41:09.417 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-23T10:41:09.417 INFO:teuthology.orchestra.run.smithi114.stdout: "identifier": { 2024-04-23T10:41:09.417 INFO:teuthology.orchestra.run.smithi114.stdout: "name": "identifier", 2024-04-23T10:41:09.418 INFO:teuthology.orchestra.run.smithi114.stdout: "type": "str", 2024-04-23T10:41:09.418 INFO:teuthology.orchestra.run.smithi114.stdout: "level": "advanced", 2024-04-23T10:41:09.418 INFO:teuthology.orchestra.run.smithi114.stdout: "flags": 0, 2024-04-23T10:41:09.418 INFO:teuthology.orchestra.run.smithi114.stdout: "default_value": "", 2024-04-23T10:41:09.418 INFO:teuthology.orchestra.run.smithi114.stdout: "min": "", 2024-04-23T10:41:09.418 INFO:teuthology.orchestra.run.smithi114.stdout: "max": "", 2024-04-23T10:41:09.418 INFO:teuthology.orchestra.run.smithi114.stdout: "enum_allowed": [], 2024-04-23T10:41:09.418 INFO:teuthology.orchestra.run.smithi114.stdout: "desc": "", 2024-04-23T10:41:09.419 INFO:teuthology.orchestra.run.smithi114.stdout: "long_desc": "", 2024-04-23T10:41:09.419 INFO:teuthology.orchestra.run.smithi114.stdout: "tags": [], 2024-04-23T10:41:09.419 INFO:teuthology.orchestra.run.smithi114.stdout: "see_also": [] 2024-04-23T10:41:09.419 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-23T10:41:09.419 INFO:teuthology.orchestra.run.smithi114.stdout: "interval": { 2024-04-23T10:41:09.419 INFO:teuthology.orchestra.run.smithi114.stdout: "name": "interval", 2024-04-23T10:41:09.419 INFO:teuthology.orchestra.run.smithi114.stdout: "type": "secs", 2024-04-23T10:41:09.419 INFO:teuthology.orchestra.run.smithi114.stdout: "level": "advanced", 2024-04-23T10:41:09.419 INFO:teuthology.orchestra.run.smithi114.stdout: "flags": 0, 2024-04-23T10:41:09.419 INFO:teuthology.orchestra.run.smithi114.stdout: "default_value": "60", 2024-04-23T10:41:09.420 INFO:teuthology.orchestra.run.smithi114.stdout: "min": "", 2024-04-23T10:41:09.420 INFO:teuthology.orchestra.run.smithi114.stdout: "max": "", 2024-04-23T10:41:09.420 INFO:teuthology.orchestra.run.smithi114.stdout: "enum_allowed": [], 2024-04-23T10:41:09.420 INFO:teuthology.orchestra.run.smithi114.stdout: "desc": "", 2024-04-23T10:41:09.420 INFO:teuthology.orchestra.run.smithi114.stdout: "long_desc": "", 2024-04-23T10:41:09.420 INFO:teuthology.orchestra.run.smithi114.stdout: "tags": [], 2024-04-23T10:41:09.420 INFO:teuthology.orchestra.run.smithi114.stdout: "see_also": [] 2024-04-23T10:41:09.420 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-23T10:41:09.420 INFO:teuthology.orchestra.run.smithi114.stdout: "log_level": { 2024-04-23T10:41:09.421 INFO:teuthology.orchestra.run.smithi114.stdout: "name": "log_level", 2024-04-23T10:41:09.421 INFO:teuthology.orchestra.run.smithi114.stdout: "type": "str", 2024-04-23T10:41:09.421 INFO:teuthology.orchestra.run.smithi114.stdout: "level": "advanced", 2024-04-23T10:41:09.421 INFO:teuthology.orchestra.run.smithi114.stdout: "flags": 1, 2024-04-23T10:41:09.421 INFO:teuthology.orchestra.run.smithi114.stdout: "default_value": "", 2024-04-23T10:41:09.421 INFO:teuthology.orchestra.run.smithi114.stdout: "min": "", 2024-04-23T10:41:09.421 INFO:teuthology.orchestra.run.smithi114.stdout: "max": "", 2024-04-23T10:41:09.421 INFO:teuthology.orchestra.run.smithi114.stdout: "enum_allowed": [ 2024-04-23T10:41:09.421 INFO:teuthology.orchestra.run.smithi114.stdout: "", 2024-04-23T10:41:09.421 INFO:teuthology.orchestra.run.smithi114.stdout: "critical", 2024-04-23T10:41:09.422 INFO:teuthology.orchestra.run.smithi114.stdout: "debug", 2024-04-23T10:41:09.422 INFO:teuthology.orchestra.run.smithi114.stdout: "error", 2024-04-23T10:41:09.422 INFO:teuthology.orchestra.run.smithi114.stdout: "info", 2024-04-23T10:41:09.422 INFO:teuthology.orchestra.run.smithi114.stdout: "warning" 2024-04-23T10:41:09.422 INFO:teuthology.orchestra.run.smithi114.stdout: ], 2024-04-23T10:41:09.422 INFO:teuthology.orchestra.run.smithi114.stdout: "desc": "", 2024-04-23T10:41:09.422 INFO:teuthology.orchestra.run.smithi114.stdout: "long_desc": "", 2024-04-23T10:41:09.422 INFO:teuthology.orchestra.run.smithi114.stdout: "tags": [], 2024-04-23T10:41:09.423 INFO:teuthology.orchestra.run.smithi114.stdout: "see_also": [] 2024-04-23T10:41:09.423 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-23T10:41:09.423 INFO:teuthology.orchestra.run.smithi114.stdout: "log_to_cluster": { 2024-04-23T10:41:09.423 INFO:teuthology.orchestra.run.smithi114.stdout: "name": "log_to_cluster", 2024-04-23T10:41:09.423 INFO:teuthology.orchestra.run.smithi114.stdout: "type": "bool", 2024-04-23T10:41:09.423 INFO:teuthology.orchestra.run.smithi114.stdout: "level": "advanced", 2024-04-23T10:41:09.423 INFO:teuthology.orchestra.run.smithi114.stdout: "flags": 1, 2024-04-23T10:41:09.423 INFO:teuthology.orchestra.run.smithi114.stdout: "default_value": "False", 2024-04-23T10:41:09.423 INFO:teuthology.orchestra.run.smithi114.stdout: "min": "", 2024-04-23T10:41:09.423 INFO:teuthology.orchestra.run.smithi114.stdout: "max": "", 2024-04-23T10:41:09.424 INFO:teuthology.orchestra.run.smithi114.stdout: "enum_allowed": [], 2024-04-23T10:41:09.424 INFO:teuthology.orchestra.run.smithi114.stdout: "desc": "", 2024-04-23T10:41:09.424 INFO:teuthology.orchestra.run.smithi114.stdout: "long_desc": "", 2024-04-23T10:41:09.424 INFO:teuthology.orchestra.run.smithi114.stdout: "tags": [], 2024-04-23T10:41:09.424 INFO:teuthology.orchestra.run.smithi114.stdout: "see_also": [] 2024-04-23T10:41:09.424 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-23T10:41:09.424 INFO:teuthology.orchestra.run.smithi114.stdout: "log_to_cluster_level": { 2024-04-23T10:41:09.424 INFO:teuthology.orchestra.run.smithi114.stdout: "name": "log_to_cluster_level", 2024-04-23T10:41:09.424 INFO:teuthology.orchestra.run.smithi114.stdout: "type": "str", 2024-04-23T10:41:09.425 INFO:teuthology.orchestra.run.smithi114.stdout: "level": "advanced", 2024-04-23T10:41:09.425 INFO:teuthology.orchestra.run.smithi114.stdout: "flags": 1, 2024-04-23T10:41:09.425 INFO:teuthology.orchestra.run.smithi114.stdout: "default_value": "info", 2024-04-23T10:41:09.425 INFO:teuthology.orchestra.run.smithi114.stdout: "min": "", 2024-04-23T10:41:09.425 INFO:teuthology.orchestra.run.smithi114.stdout: "max": "", 2024-04-23T10:41:09.425 INFO:teuthology.orchestra.run.smithi114.stdout: "enum_allowed": [ 2024-04-23T10:41:09.425 INFO:teuthology.orchestra.run.smithi114.stdout: "", 2024-04-23T10:41:09.425 INFO:teuthology.orchestra.run.smithi114.stdout: "critical", 2024-04-23T10:41:09.425 INFO:teuthology.orchestra.run.smithi114.stdout: "debug", 2024-04-23T10:41:09.426 INFO:teuthology.orchestra.run.smithi114.stdout: "error", 2024-04-23T10:41:09.426 INFO:teuthology.orchestra.run.smithi114.stdout: "info", 2024-04-23T10:41:09.426 INFO:teuthology.orchestra.run.smithi114.stdout: "warning" 2024-04-23T10:41:09.426 INFO:teuthology.orchestra.run.smithi114.stdout: ], 2024-04-23T10:41:09.426 INFO:teuthology.orchestra.run.smithi114.stdout: "desc": "", 2024-04-23T10:41:09.426 INFO:teuthology.orchestra.run.smithi114.stdout: "long_desc": "", 2024-04-23T10:41:09.426 INFO:teuthology.orchestra.run.smithi114.stdout: "tags": [], 2024-04-23T10:41:09.426 INFO:teuthology.orchestra.run.smithi114.stdout: "see_also": [] 2024-04-23T10:41:09.427 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-23T10:41:09.427 INFO:teuthology.orchestra.run.smithi114.stdout: "log_to_file": { 2024-04-23T10:41:09.427 INFO:teuthology.orchestra.run.smithi114.stdout: "name": "log_to_file", 2024-04-23T10:41:09.427 INFO:teuthology.orchestra.run.smithi114.stdout: "type": "bool", 2024-04-23T10:41:09.427 INFO:teuthology.orchestra.run.smithi114.stdout: "level": "advanced", 2024-04-23T10:41:09.427 INFO:teuthology.orchestra.run.smithi114.stdout: "flags": 1, 2024-04-23T10:41:09.427 INFO:teuthology.orchestra.run.smithi114.stdout: "default_value": "False", 2024-04-23T10:41:09.427 INFO:teuthology.orchestra.run.smithi114.stdout: "min": "", 2024-04-23T10:41:09.427 INFO:teuthology.orchestra.run.smithi114.stdout: "max": "", 2024-04-23T10:41:09.428 INFO:teuthology.orchestra.run.smithi114.stdout: "enum_allowed": [], 2024-04-23T10:41:09.428 INFO:teuthology.orchestra.run.smithi114.stdout: "desc": "", 2024-04-23T10:41:09.428 INFO:teuthology.orchestra.run.smithi114.stdout: "long_desc": "", 2024-04-23T10:41:09.428 INFO:teuthology.orchestra.run.smithi114.stdout: "tags": [], 2024-04-23T10:41:09.428 INFO:teuthology.orchestra.run.smithi114.stdout: "see_also": [] 2024-04-23T10:41:09.428 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-23T10:41:09.428 INFO:teuthology.orchestra.run.smithi114.stdout: "zabbix_host": { 2024-04-23T10:41:09.428 INFO:teuthology.orchestra.run.smithi114.stdout: "name": "zabbix_host", 2024-04-23T10:41:09.428 INFO:teuthology.orchestra.run.smithi114.stdout: "type": "str", 2024-04-23T10:41:09.429 INFO:teuthology.orchestra.run.smithi114.stdout: "level": "advanced", 2024-04-23T10:41:09.429 INFO:teuthology.orchestra.run.smithi114.stdout: "flags": 0, 2024-04-23T10:41:09.429 INFO:teuthology.orchestra.run.smithi114.stdout: "default_value": "", 2024-04-23T10:41:09.429 INFO:teuthology.orchestra.run.smithi114.stdout: "min": "", 2024-04-23T10:41:09.429 INFO:teuthology.orchestra.run.smithi114.stdout: "max": "", 2024-04-23T10:41:09.429 INFO:teuthology.orchestra.run.smithi114.stdout: "enum_allowed": [], 2024-04-23T10:41:09.429 INFO:teuthology.orchestra.run.smithi114.stdout: "desc": "", 2024-04-23T10:41:09.429 INFO:teuthology.orchestra.run.smithi114.stdout: "long_desc": "", 2024-04-23T10:41:09.429 INFO:teuthology.orchestra.run.smithi114.stdout: "tags": [], 2024-04-23T10:41:09.430 INFO:teuthology.orchestra.run.smithi114.stdout: "see_also": [] 2024-04-23T10:41:09.430 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-23T10:41:09.430 INFO:teuthology.orchestra.run.smithi114.stdout: "zabbix_port": { 2024-04-23T10:41:09.430 INFO:teuthology.orchestra.run.smithi114.stdout: "name": "zabbix_port", 2024-04-23T10:41:09.430 INFO:teuthology.orchestra.run.smithi114.stdout: "type": "int", 2024-04-23T10:41:09.430 INFO:teuthology.orchestra.run.smithi114.stdout: "level": "advanced", 2024-04-23T10:41:09.430 INFO:teuthology.orchestra.run.smithi114.stdout: "flags": 0, 2024-04-23T10:41:09.430 INFO:teuthology.orchestra.run.smithi114.stdout: "default_value": "10051", 2024-04-23T10:41:09.431 INFO:teuthology.orchestra.run.smithi114.stdout: "min": "", 2024-04-23T10:41:09.431 INFO:teuthology.orchestra.run.smithi114.stdout: "max": "", 2024-04-23T10:41:09.431 INFO:teuthology.orchestra.run.smithi114.stdout: "enum_allowed": [], 2024-04-23T10:41:09.431 INFO:teuthology.orchestra.run.smithi114.stdout: "desc": "", 2024-04-23T10:41:09.431 INFO:teuthology.orchestra.run.smithi114.stdout: "long_desc": "", 2024-04-23T10:41:09.431 INFO:teuthology.orchestra.run.smithi114.stdout: "tags": [], 2024-04-23T10:41:09.431 INFO:teuthology.orchestra.run.smithi114.stdout: "see_also": [] 2024-04-23T10:41:09.431 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-23T10:41:09.431 INFO:teuthology.orchestra.run.smithi114.stdout: "zabbix_sender": { 2024-04-23T10:41:09.432 INFO:teuthology.orchestra.run.smithi114.stdout: "name": "zabbix_sender", 2024-04-23T10:41:09.432 INFO:teuthology.orchestra.run.smithi114.stdout: "type": "str", 2024-04-23T10:41:09.432 INFO:teuthology.orchestra.run.smithi114.stdout: "level": "advanced", 2024-04-23T10:41:09.432 INFO:teuthology.orchestra.run.smithi114.stdout: "flags": 0, 2024-04-23T10:41:09.432 INFO:teuthology.orchestra.run.smithi114.stdout: "default_value": "/usr/bin/zabbix_sender", 2024-04-23T10:41:09.432 INFO:teuthology.orchestra.run.smithi114.stdout: "min": "", 2024-04-23T10:41:09.432 INFO:teuthology.orchestra.run.smithi114.stdout: "max": "", 2024-04-23T10:41:09.432 INFO:teuthology.orchestra.run.smithi114.stdout: "enum_allowed": [], 2024-04-23T10:41:09.433 INFO:teuthology.orchestra.run.smithi114.stdout: "desc": "", 2024-04-23T10:41:09.433 INFO:teuthology.orchestra.run.smithi114.stdout: "long_desc": "", 2024-04-23T10:41:09.433 INFO:teuthology.orchestra.run.smithi114.stdout: "tags": [], 2024-04-23T10:41:09.433 INFO:teuthology.orchestra.run.smithi114.stdout: "see_also": [] 2024-04-23T10:41:09.433 INFO:teuthology.orchestra.run.smithi114.stdout: } 2024-04-23T10:41:09.433 INFO:teuthology.orchestra.run.smithi114.stdout: } 2024-04-23T10:41:09.433 INFO:teuthology.orchestra.run.smithi114.stdout: } 2024-04-23T10:41:09.433 INFO:teuthology.orchestra.run.smithi114.stdout: ] 2024-04-23T10:41:09.433 INFO:teuthology.orchestra.run.smithi114.stdout: } 2024-04-23T10:41:09.434 INFO:teuthology.orchestra.run.smithi114.stdout: ], 2024-04-23T10:41:09.434 INFO:teuthology.orchestra.run.smithi114.stdout: "modules": [ 2024-04-23T10:41:09.434 INFO:teuthology.orchestra.run.smithi114.stdout: "cephadm", 2024-04-23T10:41:09.434 INFO:teuthology.orchestra.run.smithi114.stdout: "dashboard", 2024-04-23T10:41:09.434 INFO:teuthology.orchestra.run.smithi114.stdout: "iostat", 2024-04-23T10:41:09.434 INFO:teuthology.orchestra.run.smithi114.stdout: "nfs", 2024-04-23T10:41:09.434 INFO:teuthology.orchestra.run.smithi114.stdout: "restful", 2024-04-23T10:41:09.434 INFO:teuthology.orchestra.run.smithi114.stdout: "snap_schedule" 2024-04-23T10:41:09.435 INFO:teuthology.orchestra.run.smithi114.stdout: ], 2024-04-23T10:41:09.435 INFO:teuthology.orchestra.run.smithi114.stdout: "available_modules": [ 2024-04-23T10:41:09.435 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-04-23T10:41:09.435 INFO:teuthology.orchestra.run.smithi114.stdout: "name": "alerts", 2024-04-23T10:41:09.435 INFO:teuthology.orchestra.run.smithi114.stdout: "can_run": true, 2024-04-23T10:41:09.435 INFO:teuthology.orchestra.run.smithi114.stdout: "error_string": "", 2024-04-23T10:41:09.435 INFO:teuthology.orchestra.run.smithi114.stdout: "module_options": { 2024-04-23T10:41:09.435 INFO:teuthology.orchestra.run.smithi114.stdout: "interval": { 2024-04-23T10:41:09.435 INFO:teuthology.orchestra.run.smithi114.stdout: "name": "interval", 2024-04-23T10:41:09.435 INFO:teuthology.orchestra.run.smithi114.stdout: "type": "secs", 2024-04-23T10:41:09.436 INFO:teuthology.orchestra.run.smithi114.stdout: "level": "advanced", 2024-04-23T10:41:09.436 INFO:teuthology.orchestra.run.smithi114.stdout: "flags": 1, 2024-04-23T10:41:09.436 INFO:teuthology.orchestra.run.smithi114.stdout: "default_value": "60", 2024-04-23T10:41:09.436 INFO:teuthology.orchestra.run.smithi114.stdout: "min": "", 2024-04-23T10:41:09.436 INFO:teuthology.orchestra.run.smithi114.stdout: "max": "", 2024-04-23T10:41:09.436 INFO:teuthology.orchestra.run.smithi114.stdout: "enum_allowed": [], 2024-04-23T10:41:09.436 INFO:teuthology.orchestra.run.smithi114.stdout: "desc": "How frequently to reexamine health status", 2024-04-23T10:41:09.436 INFO:teuthology.orchestra.run.smithi114.stdout: "long_desc": "", 2024-04-23T10:41:09.436 INFO:teuthology.orchestra.run.smithi114.stdout: "tags": [], 2024-04-23T10:41:09.437 INFO:teuthology.orchestra.run.smithi114.stdout: "see_also": [] 2024-04-23T10:41:09.437 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-23T10:41:09.437 INFO:teuthology.orchestra.run.smithi114.stdout: "log_level": { 2024-04-23T10:41:09.437 INFO:teuthology.orchestra.run.smithi114.stdout: "name": "log_level", 2024-04-23T10:41:09.437 INFO:teuthology.orchestra.run.smithi114.stdout: "type": "str", 2024-04-23T10:41:09.437 INFO:teuthology.orchestra.run.smithi114.stdout: "level": "advanced", 2024-04-23T10:41:09.437 INFO:teuthology.orchestra.run.smithi114.stdout: "flags": 1, 2024-04-23T10:41:09.437 INFO:teuthology.orchestra.run.smithi114.stdout: "default_value": "", 2024-04-23T10:41:09.438 INFO:teuthology.orchestra.run.smithi114.stdout: "min": "", 2024-04-23T10:41:09.438 INFO:teuthology.orchestra.run.smithi114.stdout: "max": "", 2024-04-23T10:41:09.438 INFO:teuthology.orchestra.run.smithi114.stdout: "enum_allowed": [ 2024-04-23T10:41:09.438 INFO:teuthology.orchestra.run.smithi114.stdout: "", 2024-04-23T10:41:09.438 INFO:teuthology.orchestra.run.smithi114.stdout: "critical", 2024-04-23T10:41:09.438 INFO:teuthology.orchestra.run.smithi114.stdout: "debug", 2024-04-23T10:41:09.438 INFO:teuthology.orchestra.run.smithi114.stdout: "error", 2024-04-23T10:41:09.438 INFO:teuthology.orchestra.run.smithi114.stdout: "info", 2024-04-23T10:41:09.438 INFO:teuthology.orchestra.run.smithi114.stdout: "warning" 2024-04-23T10:41:09.439 INFO:teuthology.orchestra.run.smithi114.stdout: ], 2024-04-23T10:41:09.439 INFO:teuthology.orchestra.run.smithi114.stdout: "desc": "", 2024-04-23T10:41:09.439 INFO:teuthology.orchestra.run.smithi114.stdout: "long_desc": "", 2024-04-23T10:41:09.439 INFO:teuthology.orchestra.run.smithi114.stdout: "tags": [], 2024-04-23T10:41:09.439 INFO:teuthology.orchestra.run.smithi114.stdout: "see_also": [] 2024-04-23T10:41:09.439 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-23T10:41:09.439 INFO:teuthology.orchestra.run.smithi114.stdout: "log_to_cluster": { 2024-04-23T10:41:09.439 INFO:teuthology.orchestra.run.smithi114.stdout: "name": "log_to_cluster", 2024-04-23T10:41:09.439 INFO:teuthology.orchestra.run.smithi114.stdout: "type": "bool", 2024-04-23T10:41:09.439 INFO:teuthology.orchestra.run.smithi114.stdout: "level": "advanced", 2024-04-23T10:41:09.440 INFO:teuthology.orchestra.run.smithi114.stdout: "flags": 1, 2024-04-23T10:41:09.440 INFO:teuthology.orchestra.run.smithi114.stdout: "default_value": "False", 2024-04-23T10:41:09.440 INFO:teuthology.orchestra.run.smithi114.stdout: "min": "", 2024-04-23T10:41:09.440 INFO:teuthology.orchestra.run.smithi114.stdout: "max": "", 2024-04-23T10:41:09.440 INFO:teuthology.orchestra.run.smithi114.stdout: "enum_allowed": [], 2024-04-23T10:41:09.440 INFO:teuthology.orchestra.run.smithi114.stdout: "desc": "", 2024-04-23T10:41:09.440 INFO:teuthology.orchestra.run.smithi114.stdout: "long_desc": "", 2024-04-23T10:41:09.440 INFO:teuthology.orchestra.run.smithi114.stdout: "tags": [], 2024-04-23T10:41:09.441 INFO:teuthology.orchestra.run.smithi114.stdout: "see_also": [] 2024-04-23T10:41:09.441 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-23T10:41:09.441 INFO:teuthology.orchestra.run.smithi114.stdout: "log_to_cluster_level": { 2024-04-23T10:41:09.441 INFO:teuthology.orchestra.run.smithi114.stdout: "name": "log_to_cluster_level", 2024-04-23T10:41:09.441 INFO:teuthology.orchestra.run.smithi114.stdout: "type": "str", 2024-04-23T10:41:09.441 INFO:teuthology.orchestra.run.smithi114.stdout: "level": "advanced", 2024-04-23T10:41:09.441 INFO:teuthology.orchestra.run.smithi114.stdout: "flags": 1, 2024-04-23T10:41:09.441 INFO:teuthology.orchestra.run.smithi114.stdout: "default_value": "info", 2024-04-23T10:41:09.441 INFO:teuthology.orchestra.run.smithi114.stdout: "min": "", 2024-04-23T10:41:09.441 INFO:teuthology.orchestra.run.smithi114.stdout: "max": "", 2024-04-23T10:41:09.442 INFO:teuthology.orchestra.run.smithi114.stdout: "enum_allowed": [ 2024-04-23T10:41:09.442 INFO:teuthology.orchestra.run.smithi114.stdout: "", 2024-04-23T10:41:09.442 INFO:teuthology.orchestra.run.smithi114.stdout: "critical", 2024-04-23T10:41:09.442 INFO:teuthology.orchestra.run.smithi114.stdout: "debug", 2024-04-23T10:41:09.442 INFO:teuthology.orchestra.run.smithi114.stdout: "error", 2024-04-23T10:41:09.442 INFO:teuthology.orchestra.run.smithi114.stdout: "info", 2024-04-23T10:41:09.442 INFO:teuthology.orchestra.run.smithi114.stdout: "warning" 2024-04-23T10:41:09.442 INFO:teuthology.orchestra.run.smithi114.stdout: ], 2024-04-23T10:41:09.443 INFO:teuthology.orchestra.run.smithi114.stdout: "desc": "", 2024-04-23T10:41:09.443 INFO:teuthology.orchestra.run.smithi114.stdout: "long_desc": "", 2024-04-23T10:41:09.443 INFO:teuthology.orchestra.run.smithi114.stdout: "tags": [], 2024-04-23T10:41:09.443 INFO:teuthology.orchestra.run.smithi114.stdout: "see_also": [] 2024-04-23T10:41:09.443 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-23T10:41:09.443 INFO:teuthology.orchestra.run.smithi114.stdout: "log_to_file": { 2024-04-23T10:41:09.443 INFO:teuthology.orchestra.run.smithi114.stdout: "name": "log_to_file", 2024-04-23T10:41:09.443 INFO:teuthology.orchestra.run.smithi114.stdout: "type": "bool", 2024-04-23T10:41:09.443 INFO:teuthology.orchestra.run.smithi114.stdout: "level": "advanced", 2024-04-23T10:41:09.444 INFO:teuthology.orchestra.run.smithi114.stdout: "flags": 1, 2024-04-23T10:41:09.444 INFO:teuthology.orchestra.run.smithi114.stdout: "default_value": "False", 2024-04-23T10:41:09.444 INFO:teuthology.orchestra.run.smithi114.stdout: "min": "", 2024-04-23T10:41:09.444 INFO:teuthology.orchestra.run.smithi114.stdout: "max": "", 2024-04-23T10:41:09.444 INFO:teuthology.orchestra.run.smithi114.stdout: "enum_allowed": [], 2024-04-23T10:41:09.444 INFO:teuthology.orchestra.run.smithi114.stdout: "desc": "", 2024-04-23T10:41:09.444 INFO:teuthology.orchestra.run.smithi114.stdout: "long_desc": "", 2024-04-23T10:41:09.444 INFO:teuthology.orchestra.run.smithi114.stdout: "tags": [], 2024-04-23T10:41:09.444 INFO:teuthology.orchestra.run.smithi114.stdout: "see_also": [] 2024-04-23T10:41:09.445 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-23T10:41:09.445 INFO:teuthology.orchestra.run.smithi114.stdout: "smtp_destination": { 2024-04-23T10:41:09.445 INFO:teuthology.orchestra.run.smithi114.stdout: "name": "smtp_destination", 2024-04-23T10:41:09.445 INFO:teuthology.orchestra.run.smithi114.stdout: "type": "str", 2024-04-23T10:41:09.445 INFO:teuthology.orchestra.run.smithi114.stdout: "level": "advanced", 2024-04-23T10:41:09.445 INFO:teuthology.orchestra.run.smithi114.stdout: "flags": 1, 2024-04-23T10:41:09.445 INFO:teuthology.orchestra.run.smithi114.stdout: "default_value": "", 2024-04-23T10:41:09.445 INFO:teuthology.orchestra.run.smithi114.stdout: "min": "", 2024-04-23T10:41:09.445 INFO:teuthology.orchestra.run.smithi114.stdout: "max": "", 2024-04-23T10:41:09.446 INFO:teuthology.orchestra.run.smithi114.stdout: "enum_allowed": [], 2024-04-23T10:41:09.446 INFO:teuthology.orchestra.run.smithi114.stdout: "desc": "Email address to send alerts to", 2024-04-23T10:41:09.446 INFO:teuthology.orchestra.run.smithi114.stdout: "long_desc": "", 2024-04-23T10:41:09.446 INFO:teuthology.orchestra.run.smithi114.stdout: "tags": [], 2024-04-23T10:41:09.446 INFO:teuthology.orchestra.run.smithi114.stdout: "see_also": [] 2024-04-23T10:41:09.446 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-23T10:41:09.446 INFO:teuthology.orchestra.run.smithi114.stdout: "smtp_from_name": { 2024-04-23T10:41:09.446 INFO:teuthology.orchestra.run.smithi114.stdout: "name": "smtp_from_name", 2024-04-23T10:41:09.447 INFO:teuthology.orchestra.run.smithi114.stdout: "type": "str", 2024-04-23T10:41:09.447 INFO:teuthology.orchestra.run.smithi114.stdout: "level": "advanced", 2024-04-23T10:41:09.447 INFO:teuthology.orchestra.run.smithi114.stdout: "flags": 1, 2024-04-23T10:41:09.447 INFO:teuthology.orchestra.run.smithi114.stdout: "default_value": "Ceph", 2024-04-23T10:41:09.447 INFO:teuthology.orchestra.run.smithi114.stdout: "min": "", 2024-04-23T10:41:09.447 INFO:teuthology.orchestra.run.smithi114.stdout: "max": "", 2024-04-23T10:41:09.447 INFO:teuthology.orchestra.run.smithi114.stdout: "enum_allowed": [], 2024-04-23T10:41:09.447 INFO:teuthology.orchestra.run.smithi114.stdout: "desc": "Email From: name", 2024-04-23T10:41:09.447 INFO:teuthology.orchestra.run.smithi114.stdout: "long_desc": "", 2024-04-23T10:41:09.448 INFO:teuthology.orchestra.run.smithi114.stdout: "tags": [], 2024-04-23T10:41:09.448 INFO:teuthology.orchestra.run.smithi114.stdout: "see_also": [] 2024-04-23T10:41:09.448 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-23T10:41:09.448 INFO:teuthology.orchestra.run.smithi114.stdout: "smtp_host": { 2024-04-23T10:41:09.448 INFO:teuthology.orchestra.run.smithi114.stdout: "name": "smtp_host", 2024-04-23T10:41:09.448 INFO:teuthology.orchestra.run.smithi114.stdout: "type": "str", 2024-04-23T10:41:09.448 INFO:teuthology.orchestra.run.smithi114.stdout: "level": "advanced", 2024-04-23T10:41:09.448 INFO:teuthology.orchestra.run.smithi114.stdout: "flags": 1, 2024-04-23T10:41:09.448 INFO:teuthology.orchestra.run.smithi114.stdout: "default_value": "", 2024-04-23T10:41:09.449 INFO:teuthology.orchestra.run.smithi114.stdout: "min": "", 2024-04-23T10:41:09.449 INFO:teuthology.orchestra.run.smithi114.stdout: "max": "", 2024-04-23T10:41:09.449 INFO:teuthology.orchestra.run.smithi114.stdout: "enum_allowed": [], 2024-04-23T10:41:09.449 INFO:teuthology.orchestra.run.smithi114.stdout: "desc": "SMTP server", 2024-04-23T10:41:09.449 INFO:teuthology.orchestra.run.smithi114.stdout: "long_desc": "", 2024-04-23T10:41:09.449 INFO:teuthology.orchestra.run.smithi114.stdout: "tags": [], 2024-04-23T10:41:09.449 INFO:teuthology.orchestra.run.smithi114.stdout: "see_also": [] 2024-04-23T10:41:09.449 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-23T10:41:09.449 INFO:teuthology.orchestra.run.smithi114.stdout: "smtp_password": { 2024-04-23T10:41:09.450 INFO:teuthology.orchestra.run.smithi114.stdout: "name": "smtp_password", 2024-04-23T10:41:09.450 INFO:teuthology.orchestra.run.smithi114.stdout: "type": "str", 2024-04-23T10:41:09.450 INFO:teuthology.orchestra.run.smithi114.stdout: "level": "advanced", 2024-04-23T10:41:09.450 INFO:teuthology.orchestra.run.smithi114.stdout: "flags": 1, 2024-04-23T10:41:09.450 INFO:teuthology.orchestra.run.smithi114.stdout: "default_value": "", 2024-04-23T10:41:09.450 INFO:teuthology.orchestra.run.smithi114.stdout: "min": "", 2024-04-23T10:41:09.450 INFO:teuthology.orchestra.run.smithi114.stdout: "max": "", 2024-04-23T10:41:09.450 INFO:teuthology.orchestra.run.smithi114.stdout: "enum_allowed": [], 2024-04-23T10:41:09.451 INFO:teuthology.orchestra.run.smithi114.stdout: "desc": "Password to authenticate with", 2024-04-23T10:41:09.451 INFO:teuthology.orchestra.run.smithi114.stdout: "long_desc": "", 2024-04-23T10:41:09.451 INFO:teuthology.orchestra.run.smithi114.stdout: "tags": [], 2024-04-23T10:41:09.451 INFO:teuthology.orchestra.run.smithi114.stdout: "see_also": [] 2024-04-23T10:41:09.451 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-23T10:41:09.451 INFO:teuthology.orchestra.run.smithi114.stdout: "smtp_port": { 2024-04-23T10:41:09.451 INFO:teuthology.orchestra.run.smithi114.stdout: "name": "smtp_port", 2024-04-23T10:41:09.451 INFO:teuthology.orchestra.run.smithi114.stdout: "type": "int", 2024-04-23T10:41:09.451 INFO:teuthology.orchestra.run.smithi114.stdout: "level": "advanced", 2024-04-23T10:41:09.451 INFO:teuthology.orchestra.run.smithi114.stdout: "flags": 1, 2024-04-23T10:41:09.452 INFO:teuthology.orchestra.run.smithi114.stdout: "default_value": "465", 2024-04-23T10:41:09.452 INFO:teuthology.orchestra.run.smithi114.stdout: "min": "", 2024-04-23T10:41:09.452 INFO:teuthology.orchestra.run.smithi114.stdout: "max": "", 2024-04-23T10:41:09.452 INFO:teuthology.orchestra.run.smithi114.stdout: "enum_allowed": [], 2024-04-23T10:41:09.452 INFO:teuthology.orchestra.run.smithi114.stdout: "desc": "SMTP port", 2024-04-23T10:41:09.452 INFO:teuthology.orchestra.run.smithi114.stdout: "long_desc": "", 2024-04-23T10:41:09.452 INFO:teuthology.orchestra.run.smithi114.stdout: "tags": [], 2024-04-23T10:41:09.452 INFO:teuthology.orchestra.run.smithi114.stdout: "see_also": [] 2024-04-23T10:41:09.452 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-23T10:41:09.453 INFO:teuthology.orchestra.run.smithi114.stdout: "smtp_sender": { 2024-04-23T10:41:09.453 INFO:teuthology.orchestra.run.smithi114.stdout: "name": "smtp_sender", 2024-04-23T10:41:09.453 INFO:teuthology.orchestra.run.smithi114.stdout: "type": "str", 2024-04-23T10:41:09.453 INFO:teuthology.orchestra.run.smithi114.stdout: "level": "advanced", 2024-04-23T10:41:09.453 INFO:teuthology.orchestra.run.smithi114.stdout: "flags": 1, 2024-04-23T10:41:09.453 INFO:teuthology.orchestra.run.smithi114.stdout: "default_value": "", 2024-04-23T10:41:09.453 INFO:teuthology.orchestra.run.smithi114.stdout: "min": "", 2024-04-23T10:41:09.453 INFO:teuthology.orchestra.run.smithi114.stdout: "max": "", 2024-04-23T10:41:09.453 INFO:teuthology.orchestra.run.smithi114.stdout: "enum_allowed": [], 2024-04-23T10:41:09.454 INFO:teuthology.orchestra.run.smithi114.stdout: "desc": "SMTP envelope sender", 2024-04-23T10:41:09.454 INFO:teuthology.orchestra.run.smithi114.stdout: "long_desc": "", 2024-04-23T10:41:09.454 INFO:teuthology.orchestra.run.smithi114.stdout: "tags": [], 2024-04-23T10:41:09.454 INFO:teuthology.orchestra.run.smithi114.stdout: "see_also": [] 2024-04-23T10:41:09.454 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-23T10:41:09.454 INFO:teuthology.orchestra.run.smithi114.stdout: "smtp_ssl": { 2024-04-23T10:41:09.454 INFO:teuthology.orchestra.run.smithi114.stdout: "name": "smtp_ssl", 2024-04-23T10:41:09.454 INFO:teuthology.orchestra.run.smithi114.stdout: "type": "bool", 2024-04-23T10:41:09.455 INFO:teuthology.orchestra.run.smithi114.stdout: "level": "advanced", 2024-04-23T10:41:09.455 INFO:teuthology.orchestra.run.smithi114.stdout: "flags": 1, 2024-04-23T10:41:09.455 INFO:teuthology.orchestra.run.smithi114.stdout: "default_value": "True", 2024-04-23T10:41:09.455 INFO:teuthology.orchestra.run.smithi114.stdout: "min": "", 2024-04-23T10:41:09.455 INFO:teuthology.orchestra.run.smithi114.stdout: "max": "", 2024-04-23T10:41:09.455 INFO:teuthology.orchestra.run.smithi114.stdout: "enum_allowed": [], 2024-04-23T10:41:09.455 INFO:teuthology.orchestra.run.smithi114.stdout: "desc": "Use SSL to connect to SMTP server", 2024-04-23T10:41:09.455 INFO:teuthology.orchestra.run.smithi114.stdout: "long_desc": "", 2024-04-23T10:41:09.455 INFO:teuthology.orchestra.run.smithi114.stdout: "tags": [], 2024-04-23T10:41:09.456 INFO:teuthology.orchestra.run.smithi114.stdout: "see_also": [] 2024-04-23T10:41:09.456 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-23T10:41:09.456 INFO:teuthology.orchestra.run.smithi114.stdout: "smtp_user": { 2024-04-23T10:41:09.456 INFO:teuthology.orchestra.run.smithi114.stdout: "name": "smtp_user", 2024-04-23T10:41:09.456 INFO:teuthology.orchestra.run.smithi114.stdout: "type": "str", 2024-04-23T10:41:09.456 INFO:teuthology.orchestra.run.smithi114.stdout: "level": "advanced", 2024-04-23T10:41:09.456 INFO:teuthology.orchestra.run.smithi114.stdout: "flags": 1, 2024-04-23T10:41:09.456 INFO:teuthology.orchestra.run.smithi114.stdout: "default_value": "", 2024-04-23T10:41:09.456 INFO:teuthology.orchestra.run.smithi114.stdout: "min": "", 2024-04-23T10:41:09.457 INFO:teuthology.orchestra.run.smithi114.stdout: "max": "", 2024-04-23T10:41:09.457 INFO:teuthology.orchestra.run.smithi114.stdout: "enum_allowed": [], 2024-04-23T10:41:09.457 INFO:teuthology.orchestra.run.smithi114.stdout: "desc": "User to authenticate as", 2024-04-23T10:41:09.457 INFO:teuthology.orchestra.run.smithi114.stdout: "long_desc": "", 2024-04-23T10:41:09.457 INFO:teuthology.orchestra.run.smithi114.stdout: "tags": [], 2024-04-23T10:41:09.457 INFO:teuthology.orchestra.run.smithi114.stdout: "see_also": [] 2024-04-23T10:41:09.457 INFO:teuthology.orchestra.run.smithi114.stdout: } 2024-04-23T10:41:09.457 INFO:teuthology.orchestra.run.smithi114.stdout: } 2024-04-23T10:41:09.457 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-23T10:41:09.458 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-04-23T10:41:09.458 INFO:teuthology.orchestra.run.smithi114.stdout: "name": "balancer", 2024-04-23T10:41:09.458 INFO:teuthology.orchestra.run.smithi114.stdout: "can_run": true, 2024-04-23T10:41:09.458 INFO:teuthology.orchestra.run.smithi114.stdout: "error_string": "", 2024-04-23T10:41:09.458 INFO:teuthology.orchestra.run.smithi114.stdout: "module_options": { 2024-04-23T10:41:09.458 INFO:teuthology.orchestra.run.smithi114.stdout: "active": { 2024-04-23T10:41:09.458 INFO:teuthology.orchestra.run.smithi114.stdout: "name": "active", 2024-04-23T10:41:09.458 INFO:teuthology.orchestra.run.smithi114.stdout: "type": "bool", 2024-04-23T10:41:09.459 INFO:teuthology.orchestra.run.smithi114.stdout: "level": "advanced", 2024-04-23T10:41:09.459 INFO:teuthology.orchestra.run.smithi114.stdout: "flags": 1, 2024-04-23T10:41:09.459 INFO:teuthology.orchestra.run.smithi114.stdout: "default_value": "True", 2024-04-23T10:41:09.459 INFO:teuthology.orchestra.run.smithi114.stdout: "min": "", 2024-04-23T10:41:09.459 INFO:teuthology.orchestra.run.smithi114.stdout: "max": "", 2024-04-23T10:41:09.459 INFO:teuthology.orchestra.run.smithi114.stdout: "enum_allowed": [], 2024-04-23T10:41:09.459 INFO:teuthology.orchestra.run.smithi114.stdout: "desc": "automatically balance PGs across cluster", 2024-04-23T10:41:09.459 INFO:teuthology.orchestra.run.smithi114.stdout: "long_desc": "", 2024-04-23T10:41:09.459 INFO:teuthology.orchestra.run.smithi114.stdout: "tags": [], 2024-04-23T10:41:09.459 INFO:teuthology.orchestra.run.smithi114.stdout: "see_also": [] 2024-04-23T10:41:09.460 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-23T10:41:09.460 INFO:teuthology.orchestra.run.smithi114.stdout: "begin_time": { 2024-04-23T10:41:09.460 INFO:teuthology.orchestra.run.smithi114.stdout: "name": "begin_time", 2024-04-23T10:41:09.460 INFO:teuthology.orchestra.run.smithi114.stdout: "type": "str", 2024-04-23T10:41:09.460 INFO:teuthology.orchestra.run.smithi114.stdout: "level": "advanced", 2024-04-23T10:41:09.460 INFO:teuthology.orchestra.run.smithi114.stdout: "flags": 1, 2024-04-23T10:41:09.460 INFO:teuthology.orchestra.run.smithi114.stdout: "default_value": "0000", 2024-04-23T10:41:09.460 INFO:teuthology.orchestra.run.smithi114.stdout: "min": "", 2024-04-23T10:41:09.460 INFO:teuthology.orchestra.run.smithi114.stdout: "max": "", 2024-04-23T10:41:09.461 INFO:teuthology.orchestra.run.smithi114.stdout: "enum_allowed": [], 2024-04-23T10:41:09.461 INFO:teuthology.orchestra.run.smithi114.stdout: "desc": "beginning time of day to automatically balance", 2024-04-23T10:41:09.461 INFO:teuthology.orchestra.run.smithi114.stdout: "long_desc": "This is a time of day in the format HHMM.", 2024-04-23T10:41:09.461 INFO:teuthology.orchestra.run.smithi114.stdout: "tags": [], 2024-04-23T10:41:09.461 INFO:teuthology.orchestra.run.smithi114.stdout: "see_also": [] 2024-04-23T10:41:09.461 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-23T10:41:09.461 INFO:teuthology.orchestra.run.smithi114.stdout: "begin_weekday": { 2024-04-23T10:41:09.461 INFO:teuthology.orchestra.run.smithi114.stdout: "name": "begin_weekday", 2024-04-23T10:41:09.461 INFO:teuthology.orchestra.run.smithi114.stdout: "type": "uint", 2024-04-23T10:41:09.462 INFO:teuthology.orchestra.run.smithi114.stdout: "level": "advanced", 2024-04-23T10:41:09.462 INFO:teuthology.orchestra.run.smithi114.stdout: "flags": 1, 2024-04-23T10:41:09.462 INFO:teuthology.orchestra.run.smithi114.stdout: "default_value": "0", 2024-04-23T10:41:09.462 INFO:teuthology.orchestra.run.smithi114.stdout: "min": "0", 2024-04-23T10:41:09.462 INFO:teuthology.orchestra.run.smithi114.stdout: "max": "6", 2024-04-23T10:41:09.462 INFO:teuthology.orchestra.run.smithi114.stdout: "enum_allowed": [], 2024-04-23T10:41:09.462 INFO:teuthology.orchestra.run.smithi114.stdout: "desc": "Restrict automatic balancing to this day of the week or later", 2024-04-23T10:41:09.462 INFO:teuthology.orchestra.run.smithi114.stdout: "long_desc": "0 = Sunday, 1 = Monday, etc.", 2024-04-23T10:41:09.463 INFO:teuthology.orchestra.run.smithi114.stdout: "tags": [], 2024-04-23T10:41:09.463 INFO:teuthology.orchestra.run.smithi114.stdout: "see_also": [] 2024-04-23T10:41:09.463 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-23T10:41:09.463 INFO:teuthology.orchestra.run.smithi114.stdout: "crush_compat_max_iterations": { 2024-04-23T10:41:09.463 INFO:teuthology.orchestra.run.smithi114.stdout: "name": "crush_compat_max_iterations", 2024-04-23T10:41:09.463 INFO:teuthology.orchestra.run.smithi114.stdout: "type": "uint", 2024-04-23T10:41:09.463 INFO:teuthology.orchestra.run.smithi114.stdout: "level": "advanced", 2024-04-23T10:41:09.463 INFO:teuthology.orchestra.run.smithi114.stdout: "flags": 1, 2024-04-23T10:41:09.463 INFO:teuthology.orchestra.run.smithi114.stdout: "default_value": "25", 2024-04-23T10:41:09.464 INFO:teuthology.orchestra.run.smithi114.stdout: "min": "1", 2024-04-23T10:41:09.464 INFO:teuthology.orchestra.run.smithi114.stdout: "max": "250", 2024-04-23T10:41:09.464 INFO:teuthology.orchestra.run.smithi114.stdout: "enum_allowed": [], 2024-04-23T10:41:09.464 INFO:teuthology.orchestra.run.smithi114.stdout: "desc": "maximum number of iterations to attempt optimization", 2024-04-23T10:41:09.464 INFO:teuthology.orchestra.run.smithi114.stdout: "long_desc": "", 2024-04-23T10:41:09.464 INFO:teuthology.orchestra.run.smithi114.stdout: "tags": [], 2024-04-23T10:41:09.464 INFO:teuthology.orchestra.run.smithi114.stdout: "see_also": [] 2024-04-23T10:41:09.464 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-23T10:41:09.464 INFO:teuthology.orchestra.run.smithi114.stdout: "crush_compat_metrics": { 2024-04-23T10:41:09.464 INFO:teuthology.orchestra.run.smithi114.stdout: "name": "crush_compat_metrics", 2024-04-23T10:41:09.465 INFO:teuthology.orchestra.run.smithi114.stdout: "type": "str", 2024-04-23T10:41:09.465 INFO:teuthology.orchestra.run.smithi114.stdout: "level": "advanced", 2024-04-23T10:41:09.465 INFO:teuthology.orchestra.run.smithi114.stdout: "flags": 1, 2024-04-23T10:41:09.465 INFO:teuthology.orchestra.run.smithi114.stdout: "default_value": "pgs,objects,bytes", 2024-04-23T10:41:09.465 INFO:teuthology.orchestra.run.smithi114.stdout: "min": "", 2024-04-23T10:41:09.465 INFO:teuthology.orchestra.run.smithi114.stdout: "max": "", 2024-04-23T10:41:09.465 INFO:teuthology.orchestra.run.smithi114.stdout: "enum_allowed": [], 2024-04-23T10:41:09.465 INFO:teuthology.orchestra.run.smithi114.stdout: "desc": "metrics with which to calculate OSD utilization", 2024-04-23T10:41:09.465 INFO:teuthology.orchestra.run.smithi114.stdout: "long_desc": "Value is a list of one or more of \"pgs\", \"objects\", or \"bytes\", and indicates which metrics to use to balance utilization.", 2024-04-23T10:41:09.466 INFO:teuthology.orchestra.run.smithi114.stdout: "tags": [], 2024-04-23T10:41:09.466 INFO:teuthology.orchestra.run.smithi114.stdout: "see_also": [] 2024-04-23T10:41:09.466 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-23T10:41:09.466 INFO:teuthology.orchestra.run.smithi114.stdout: "crush_compat_step": { 2024-04-23T10:41:09.466 INFO:teuthology.orchestra.run.smithi114.stdout: "name": "crush_compat_step", 2024-04-23T10:41:09.466 INFO:teuthology.orchestra.run.smithi114.stdout: "type": "float", 2024-04-23T10:41:09.466 INFO:teuthology.orchestra.run.smithi114.stdout: "level": "advanced", 2024-04-23T10:41:09.466 INFO:teuthology.orchestra.run.smithi114.stdout: "flags": 1, 2024-04-23T10:41:09.467 INFO:teuthology.orchestra.run.smithi114.stdout: "default_value": "0.5", 2024-04-23T10:41:09.467 INFO:teuthology.orchestra.run.smithi114.stdout: "min": "0.001", 2024-04-23T10:41:09.467 INFO:teuthology.orchestra.run.smithi114.stdout: "max": "0.999", 2024-04-23T10:41:09.467 INFO:teuthology.orchestra.run.smithi114.stdout: "enum_allowed": [], 2024-04-23T10:41:09.467 INFO:teuthology.orchestra.run.smithi114.stdout: "desc": "aggressiveness of optimization", 2024-04-23T10:41:09.467 INFO:teuthology.orchestra.run.smithi114.stdout: "long_desc": ".99 is very aggressive, .01 is less aggressive", 2024-04-23T10:41:09.467 INFO:teuthology.orchestra.run.smithi114.stdout: "tags": [], 2024-04-23T10:41:09.467 INFO:teuthology.orchestra.run.smithi114.stdout: "see_also": [] 2024-04-23T10:41:09.467 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-23T10:41:09.467 INFO:teuthology.orchestra.run.smithi114.stdout: "end_time": { 2024-04-23T10:41:09.468 INFO:teuthology.orchestra.run.smithi114.stdout: "name": "end_time", 2024-04-23T10:41:09.468 INFO:teuthology.orchestra.run.smithi114.stdout: "type": "str", 2024-04-23T10:41:09.468 INFO:teuthology.orchestra.run.smithi114.stdout: "level": "advanced", 2024-04-23T10:41:09.468 INFO:teuthology.orchestra.run.smithi114.stdout: "flags": 1, 2024-04-23T10:41:09.468 INFO:teuthology.orchestra.run.smithi114.stdout: "default_value": "2359", 2024-04-23T10:41:09.468 INFO:teuthology.orchestra.run.smithi114.stdout: "min": "", 2024-04-23T10:41:09.468 INFO:teuthology.orchestra.run.smithi114.stdout: "max": "", 2024-04-23T10:41:09.468 INFO:teuthology.orchestra.run.smithi114.stdout: "enum_allowed": [], 2024-04-23T10:41:09.468 INFO:teuthology.orchestra.run.smithi114.stdout: "desc": "ending time of day to automatically balance", 2024-04-23T10:41:09.468 INFO:teuthology.orchestra.run.smithi114.stdout: "long_desc": "This is a time of day in the format HHMM.", 2024-04-23T10:41:09.469 INFO:teuthology.orchestra.run.smithi114.stdout: "tags": [], 2024-04-23T10:41:09.469 INFO:teuthology.orchestra.run.smithi114.stdout: "see_also": [] 2024-04-23T10:41:09.469 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-23T10:41:09.469 INFO:teuthology.orchestra.run.smithi114.stdout: "end_weekday": { 2024-04-23T10:41:09.469 INFO:teuthology.orchestra.run.smithi114.stdout: "name": "end_weekday", 2024-04-23T10:41:09.469 INFO:teuthology.orchestra.run.smithi114.stdout: "type": "uint", 2024-04-23T10:41:09.469 INFO:teuthology.orchestra.run.smithi114.stdout: "level": "advanced", 2024-04-23T10:41:09.469 INFO:teuthology.orchestra.run.smithi114.stdout: "flags": 1, 2024-04-23T10:41:09.469 INFO:teuthology.orchestra.run.smithi114.stdout: "default_value": "0", 2024-04-23T10:41:09.470 INFO:teuthology.orchestra.run.smithi114.stdout: "min": "0", 2024-04-23T10:41:09.470 INFO:teuthology.orchestra.run.smithi114.stdout: "max": "6", 2024-04-23T10:41:09.470 INFO:teuthology.orchestra.run.smithi114.stdout: "enum_allowed": [], 2024-04-23T10:41:09.470 INFO:teuthology.orchestra.run.smithi114.stdout: "desc": "Restrict automatic balancing to days of the week earlier than this", 2024-04-23T10:41:09.470 INFO:teuthology.orchestra.run.smithi114.stdout: "long_desc": "0 = Sunday, 1 = Monday, etc.", 2024-04-23T10:41:09.470 INFO:teuthology.orchestra.run.smithi114.stdout: "tags": [], 2024-04-23T10:41:09.470 INFO:teuthology.orchestra.run.smithi114.stdout: "see_also": [] 2024-04-23T10:41:09.470 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-23T10:41:09.470 INFO:teuthology.orchestra.run.smithi114.stdout: "log_level": { 2024-04-23T10:41:09.471 INFO:teuthology.orchestra.run.smithi114.stdout: "name": "log_level", 2024-04-23T10:41:09.471 INFO:teuthology.orchestra.run.smithi114.stdout: "type": "str", 2024-04-23T10:41:09.471 INFO:teuthology.orchestra.run.smithi114.stdout: "level": "advanced", 2024-04-23T10:41:09.471 INFO:teuthology.orchestra.run.smithi114.stdout: "flags": 1, 2024-04-23T10:41:09.471 INFO:teuthology.orchestra.run.smithi114.stdout: "default_value": "", 2024-04-23T10:41:09.471 INFO:teuthology.orchestra.run.smithi114.stdout: "min": "", 2024-04-23T10:41:09.471 INFO:teuthology.orchestra.run.smithi114.stdout: "max": "", 2024-04-23T10:41:09.471 INFO:teuthology.orchestra.run.smithi114.stdout: "enum_allowed": [ 2024-04-23T10:41:09.471 INFO:teuthology.orchestra.run.smithi114.stdout: "", 2024-04-23T10:41:09.472 INFO:teuthology.orchestra.run.smithi114.stdout: "critical", 2024-04-23T10:41:09.472 INFO:teuthology.orchestra.run.smithi114.stdout: "debug", 2024-04-23T10:41:09.472 INFO:teuthology.orchestra.run.smithi114.stdout: "error", 2024-04-23T10:41:09.472 INFO:teuthology.orchestra.run.smithi114.stdout: "info", 2024-04-23T10:41:09.472 INFO:teuthology.orchestra.run.smithi114.stdout: "warning" 2024-04-23T10:41:09.472 INFO:teuthology.orchestra.run.smithi114.stdout: ], 2024-04-23T10:41:09.472 INFO:teuthology.orchestra.run.smithi114.stdout: "desc": "", 2024-04-23T10:41:09.472 INFO:teuthology.orchestra.run.smithi114.stdout: "long_desc": "", 2024-04-23T10:41:09.472 INFO:teuthology.orchestra.run.smithi114.stdout: "tags": [], 2024-04-23T10:41:09.473 INFO:teuthology.orchestra.run.smithi114.stdout: "see_also": [] 2024-04-23T10:41:09.473 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-23T10:41:09.473 INFO:teuthology.orchestra.run.smithi114.stdout: "log_to_cluster": { 2024-04-23T10:41:09.473 INFO:teuthology.orchestra.run.smithi114.stdout: "name": "log_to_cluster", 2024-04-23T10:41:09.473 INFO:teuthology.orchestra.run.smithi114.stdout: "type": "bool", 2024-04-23T10:41:09.473 INFO:teuthology.orchestra.run.smithi114.stdout: "level": "advanced", 2024-04-23T10:41:09.473 INFO:teuthology.orchestra.run.smithi114.stdout: "flags": 1, 2024-04-23T10:41:09.473 INFO:teuthology.orchestra.run.smithi114.stdout: "default_value": "False", 2024-04-23T10:41:09.473 INFO:teuthology.orchestra.run.smithi114.stdout: "min": "", 2024-04-23T10:41:09.474 INFO:teuthology.orchestra.run.smithi114.stdout: "max": "", 2024-04-23T10:41:09.474 INFO:teuthology.orchestra.run.smithi114.stdout: "enum_allowed": [], 2024-04-23T10:41:09.474 INFO:teuthology.orchestra.run.smithi114.stdout: "desc": "", 2024-04-23T10:41:09.474 INFO:teuthology.orchestra.run.smithi114.stdout: "long_desc": "", 2024-04-23T10:41:09.474 INFO:teuthology.orchestra.run.smithi114.stdout: "tags": [], 2024-04-23T10:41:09.474 INFO:teuthology.orchestra.run.smithi114.stdout: "see_also": [] 2024-04-23T10:41:09.474 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-23T10:41:09.474 INFO:teuthology.orchestra.run.smithi114.stdout: "log_to_cluster_level": { 2024-04-23T10:41:09.474 INFO:teuthology.orchestra.run.smithi114.stdout: "name": "log_to_cluster_level", 2024-04-23T10:41:09.475 INFO:teuthology.orchestra.run.smithi114.stdout: "type": "str", 2024-04-23T10:41:09.475 INFO:teuthology.orchestra.run.smithi114.stdout: "level": "advanced", 2024-04-23T10:41:09.475 INFO:teuthology.orchestra.run.smithi114.stdout: "flags": 1, 2024-04-23T10:41:09.475 INFO:teuthology.orchestra.run.smithi114.stdout: "default_value": "info", 2024-04-23T10:41:09.475 INFO:teuthology.orchestra.run.smithi114.stdout: "min": "", 2024-04-23T10:41:09.475 INFO:teuthology.orchestra.run.smithi114.stdout: "max": "", 2024-04-23T10:41:09.475 INFO:teuthology.orchestra.run.smithi114.stdout: "enum_allowed": [ 2024-04-23T10:41:09.475 INFO:teuthology.orchestra.run.smithi114.stdout: "", 2024-04-23T10:41:09.475 INFO:teuthology.orchestra.run.smithi114.stdout: "critical", 2024-04-23T10:41:09.476 INFO:teuthology.orchestra.run.smithi114.stdout: "debug", 2024-04-23T10:41:09.476 INFO:teuthology.orchestra.run.smithi114.stdout: "error", 2024-04-23T10:41:09.476 INFO:teuthology.orchestra.run.smithi114.stdout: "info", 2024-04-23T10:41:09.476 INFO:teuthology.orchestra.run.smithi114.stdout: "warning" 2024-04-23T10:41:09.476 INFO:teuthology.orchestra.run.smithi114.stdout: ], 2024-04-23T10:41:09.476 INFO:teuthology.orchestra.run.smithi114.stdout: "desc": "", 2024-04-23T10:41:09.476 INFO:teuthology.orchestra.run.smithi114.stdout: "long_desc": "", 2024-04-23T10:41:09.476 INFO:teuthology.orchestra.run.smithi114.stdout: "tags": [], 2024-04-23T10:41:09.476 INFO:teuthology.orchestra.run.smithi114.stdout: "see_also": [] 2024-04-23T10:41:09.476 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-23T10:41:09.477 INFO:teuthology.orchestra.run.smithi114.stdout: "log_to_file": { 2024-04-23T10:41:09.477 INFO:teuthology.orchestra.run.smithi114.stdout: "name": "log_to_file", 2024-04-23T10:41:09.477 INFO:teuthology.orchestra.run.smithi114.stdout: "type": "bool", 2024-04-23T10:41:09.477 INFO:teuthology.orchestra.run.smithi114.stdout: "level": "advanced", 2024-04-23T10:41:09.477 INFO:teuthology.orchestra.run.smithi114.stdout: "flags": 1, 2024-04-23T10:41:09.477 INFO:teuthology.orchestra.run.smithi114.stdout: "default_value": "False", 2024-04-23T10:41:09.477 INFO:teuthology.orchestra.run.smithi114.stdout: "min": "", 2024-04-23T10:41:09.477 INFO:teuthology.orchestra.run.smithi114.stdout: "max": "", 2024-04-23T10:41:09.477 INFO:teuthology.orchestra.run.smithi114.stdout: "enum_allowed": [], 2024-04-23T10:41:09.478 INFO:teuthology.orchestra.run.smithi114.stdout: "desc": "", 2024-04-23T10:41:09.478 INFO:teuthology.orchestra.run.smithi114.stdout: "long_desc": "", 2024-04-23T10:41:09.478 INFO:teuthology.orchestra.run.smithi114.stdout: "tags": [], 2024-04-23T10:41:09.478 INFO:teuthology.orchestra.run.smithi114.stdout: "see_also": [] 2024-04-23T10:41:09.478 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-23T10:41:09.478 INFO:teuthology.orchestra.run.smithi114.stdout: "min_score": { 2024-04-23T10:41:09.478 INFO:teuthology.orchestra.run.smithi114.stdout: "name": "min_score", 2024-04-23T10:41:09.478 INFO:teuthology.orchestra.run.smithi114.stdout: "type": "float", 2024-04-23T10:41:09.479 INFO:teuthology.orchestra.run.smithi114.stdout: "level": "advanced", 2024-04-23T10:41:09.479 INFO:teuthology.orchestra.run.smithi114.stdout: "flags": 1, 2024-04-23T10:41:09.479 INFO:teuthology.orchestra.run.smithi114.stdout: "default_value": "0", 2024-04-23T10:41:09.479 INFO:teuthology.orchestra.run.smithi114.stdout: "min": "", 2024-04-23T10:41:09.479 INFO:teuthology.orchestra.run.smithi114.stdout: "max": "", 2024-04-23T10:41:09.479 INFO:teuthology.orchestra.run.smithi114.stdout: "enum_allowed": [], 2024-04-23T10:41:09.479 INFO:teuthology.orchestra.run.smithi114.stdout: "desc": "minimum score, below which no optimization is attempted", 2024-04-23T10:41:09.479 INFO:teuthology.orchestra.run.smithi114.stdout: "long_desc": "", 2024-04-23T10:41:09.479 INFO:teuthology.orchestra.run.smithi114.stdout: "tags": [], 2024-04-23T10:41:09.479 INFO:teuthology.orchestra.run.smithi114.stdout: "see_also": [] 2024-04-23T10:41:09.480 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-23T10:41:09.480 INFO:teuthology.orchestra.run.smithi114.stdout: "mode": { 2024-04-23T10:41:09.480 INFO:teuthology.orchestra.run.smithi114.stdout: "name": "mode", 2024-04-23T10:41:09.480 INFO:teuthology.orchestra.run.smithi114.stdout: "type": "str", 2024-04-23T10:41:09.480 INFO:teuthology.orchestra.run.smithi114.stdout: "level": "advanced", 2024-04-23T10:41:09.480 INFO:teuthology.orchestra.run.smithi114.stdout: "flags": 1, 2024-04-23T10:41:09.480 INFO:teuthology.orchestra.run.smithi114.stdout: "default_value": "upmap", 2024-04-23T10:41:09.480 INFO:teuthology.orchestra.run.smithi114.stdout: "min": "", 2024-04-23T10:41:09.480 INFO:teuthology.orchestra.run.smithi114.stdout: "max": "", 2024-04-23T10:41:09.481 INFO:teuthology.orchestra.run.smithi114.stdout: "enum_allowed": [ 2024-04-23T10:41:09.481 INFO:teuthology.orchestra.run.smithi114.stdout: "crush-compat", 2024-04-23T10:41:09.481 INFO:teuthology.orchestra.run.smithi114.stdout: "none", 2024-04-23T10:41:09.481 INFO:teuthology.orchestra.run.smithi114.stdout: "read", 2024-04-23T10:41:09.481 INFO:teuthology.orchestra.run.smithi114.stdout: "upmap", 2024-04-23T10:41:09.481 INFO:teuthology.orchestra.run.smithi114.stdout: "upmap-read" 2024-04-23T10:41:09.481 INFO:teuthology.orchestra.run.smithi114.stdout: ], 2024-04-23T10:41:09.481 INFO:teuthology.orchestra.run.smithi114.stdout: "desc": "Balancer mode", 2024-04-23T10:41:09.481 INFO:teuthology.orchestra.run.smithi114.stdout: "long_desc": "", 2024-04-23T10:41:09.481 INFO:teuthology.orchestra.run.smithi114.stdout: "tags": [], 2024-04-23T10:41:09.482 INFO:teuthology.orchestra.run.smithi114.stdout: "see_also": [] 2024-04-23T10:41:09.482 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-23T10:41:09.482 INFO:teuthology.orchestra.run.smithi114.stdout: "pool_ids": { 2024-04-23T10:41:09.482 INFO:teuthology.orchestra.run.smithi114.stdout: "name": "pool_ids", 2024-04-23T10:41:09.482 INFO:teuthology.orchestra.run.smithi114.stdout: "type": "str", 2024-04-23T10:41:09.482 INFO:teuthology.orchestra.run.smithi114.stdout: "level": "advanced", 2024-04-23T10:41:09.482 INFO:teuthology.orchestra.run.smithi114.stdout: "flags": 1, 2024-04-23T10:41:09.482 INFO:teuthology.orchestra.run.smithi114.stdout: "default_value": "", 2024-04-23T10:41:09.483 INFO:teuthology.orchestra.run.smithi114.stdout: "min": "", 2024-04-23T10:41:09.483 INFO:teuthology.orchestra.run.smithi114.stdout: "max": "", 2024-04-23T10:41:09.483 INFO:teuthology.orchestra.run.smithi114.stdout: "enum_allowed": [], 2024-04-23T10:41:09.483 INFO:teuthology.orchestra.run.smithi114.stdout: "desc": "pools which the automatic balancing will be limited to", 2024-04-23T10:41:09.483 INFO:teuthology.orchestra.run.smithi114.stdout: "long_desc": "", 2024-04-23T10:41:09.483 INFO:teuthology.orchestra.run.smithi114.stdout: "tags": [], 2024-04-23T10:41:09.483 INFO:teuthology.orchestra.run.smithi114.stdout: "see_also": [] 2024-04-23T10:41:09.483 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-23T10:41:09.483 INFO:teuthology.orchestra.run.smithi114.stdout: "sleep_interval": { 2024-04-23T10:41:09.483 INFO:teuthology.orchestra.run.smithi114.stdout: "name": "sleep_interval", 2024-04-23T10:41:09.484 INFO:teuthology.orchestra.run.smithi114.stdout: "type": "secs", 2024-04-23T10:41:09.484 INFO:teuthology.orchestra.run.smithi114.stdout: "level": "advanced", 2024-04-23T10:41:09.484 INFO:teuthology.orchestra.run.smithi114.stdout: "flags": 1, 2024-04-23T10:41:09.484 INFO:teuthology.orchestra.run.smithi114.stdout: "default_value": "60", 2024-04-23T10:41:09.484 INFO:teuthology.orchestra.run.smithi114.stdout: "min": "", 2024-04-23T10:41:09.484 INFO:teuthology.orchestra.run.smithi114.stdout: "max": "", 2024-04-23T10:41:09.484 INFO:teuthology.orchestra.run.smithi114.stdout: "enum_allowed": [], 2024-04-23T10:41:09.484 INFO:teuthology.orchestra.run.smithi114.stdout: "desc": "how frequently to wake up and attempt optimization", 2024-04-23T10:41:09.484 INFO:teuthology.orchestra.run.smithi114.stdout: "long_desc": "", 2024-04-23T10:41:09.485 INFO:teuthology.orchestra.run.smithi114.stdout: "tags": [], 2024-04-23T10:41:09.485 INFO:teuthology.orchestra.run.smithi114.stdout: "see_also": [] 2024-04-23T10:41:09.485 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-23T10:41:09.485 INFO:teuthology.orchestra.run.smithi114.stdout: "upmap_max_deviation": { 2024-04-23T10:41:09.485 INFO:teuthology.orchestra.run.smithi114.stdout: "name": "upmap_max_deviation", 2024-04-23T10:41:09.485 INFO:teuthology.orchestra.run.smithi114.stdout: "type": "int", 2024-04-23T10:41:09.485 INFO:teuthology.orchestra.run.smithi114.stdout: "level": "advanced", 2024-04-23T10:41:09.485 INFO:teuthology.orchestra.run.smithi114.stdout: "flags": 1, 2024-04-23T10:41:09.485 INFO:teuthology.orchestra.run.smithi114.stdout: "default_value": "5", 2024-04-23T10:41:09.485 INFO:teuthology.orchestra.run.smithi114.stdout: "min": "1", 2024-04-23T10:41:09.486 INFO:teuthology.orchestra.run.smithi114.stdout: "max": "", 2024-04-23T10:41:09.486 INFO:teuthology.orchestra.run.smithi114.stdout: "enum_allowed": [], 2024-04-23T10:41:09.486 INFO:teuthology.orchestra.run.smithi114.stdout: "desc": "deviation below which no optimization is attempted", 2024-04-23T10:41:09.486 INFO:teuthology.orchestra.run.smithi114.stdout: "long_desc": "If the number of PGs are within this count then no optimization is attempted", 2024-04-23T10:41:09.486 INFO:teuthology.orchestra.run.smithi114.stdout: "tags": [], 2024-04-23T10:41:09.486 INFO:teuthology.orchestra.run.smithi114.stdout: "see_also": [] 2024-04-23T10:41:09.486 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-23T10:41:09.486 INFO:teuthology.orchestra.run.smithi114.stdout: "upmap_max_optimizations": { 2024-04-23T10:41:09.487 INFO:teuthology.orchestra.run.smithi114.stdout: "name": "upmap_max_optimizations", 2024-04-23T10:41:09.487 INFO:teuthology.orchestra.run.smithi114.stdout: "type": "uint", 2024-04-23T10:41:09.487 INFO:teuthology.orchestra.run.smithi114.stdout: "level": "advanced", 2024-04-23T10:41:09.487 INFO:teuthology.orchestra.run.smithi114.stdout: "flags": 1, 2024-04-23T10:41:09.487 INFO:teuthology.orchestra.run.smithi114.stdout: "default_value": "10", 2024-04-23T10:41:09.487 INFO:teuthology.orchestra.run.smithi114.stdout: "min": "", 2024-04-23T10:41:09.487 INFO:teuthology.orchestra.run.smithi114.stdout: "max": "", 2024-04-23T10:41:09.487 INFO:teuthology.orchestra.run.smithi114.stdout: "enum_allowed": [], 2024-04-23T10:41:09.487 INFO:teuthology.orchestra.run.smithi114.stdout: "desc": "maximum upmap optimizations to make per attempt", 2024-04-23T10:41:09.487 INFO:teuthology.orchestra.run.smithi114.stdout: "long_desc": "", 2024-04-23T10:41:09.488 INFO:teuthology.orchestra.run.smithi114.stdout: "tags": [], 2024-04-23T10:41:09.488 INFO:teuthology.orchestra.run.smithi114.stdout: "see_also": [] 2024-04-23T10:41:09.488 INFO:teuthology.orchestra.run.smithi114.stdout: } 2024-04-23T10:41:09.488 INFO:teuthology.orchestra.run.smithi114.stdout: } 2024-04-23T10:41:09.488 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-23T10:41:09.488 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-04-23T10:41:09.488 INFO:teuthology.orchestra.run.smithi114.stdout: "name": "cephadm", 2024-04-23T10:41:09.488 INFO:teuthology.orchestra.run.smithi114.stdout: "can_run": true, 2024-04-23T10:41:09.488 INFO:teuthology.orchestra.run.smithi114.stdout: "error_string": "", 2024-04-23T10:41:09.489 INFO:teuthology.orchestra.run.smithi114.stdout: "module_options": { 2024-04-23T10:41:09.489 INFO:teuthology.orchestra.run.smithi114.stdout: "agent_down_multiplier": { 2024-04-23T10:41:09.489 INFO:teuthology.orchestra.run.smithi114.stdout: "name": "agent_down_multiplier", 2024-04-23T10:41:09.489 INFO:teuthology.orchestra.run.smithi114.stdout: "type": "float", 2024-04-23T10:41:09.489 INFO:teuthology.orchestra.run.smithi114.stdout: "level": "advanced", 2024-04-23T10:41:09.489 INFO:teuthology.orchestra.run.smithi114.stdout: "flags": 0, 2024-04-23T10:41:09.489 INFO:teuthology.orchestra.run.smithi114.stdout: "default_value": "3.0", 2024-04-23T10:41:09.489 INFO:teuthology.orchestra.run.smithi114.stdout: "min": "", 2024-04-23T10:41:09.489 INFO:teuthology.orchestra.run.smithi114.stdout: "max": "", 2024-04-23T10:41:09.489 INFO:teuthology.orchestra.run.smithi114.stdout: "enum_allowed": [], 2024-04-23T10:41:09.490 INFO:teuthology.orchestra.run.smithi114.stdout: "desc": "Multiplied by agent refresh rate to calculate how long agent must not report before being marked down", 2024-04-23T10:41:09.490 INFO:teuthology.orchestra.run.smithi114.stdout: "long_desc": "", 2024-04-23T10:41:09.490 INFO:teuthology.orchestra.run.smithi114.stdout: "tags": [], 2024-04-23T10:41:09.490 INFO:teuthology.orchestra.run.smithi114.stdout: "see_also": [] 2024-04-23T10:41:09.490 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-23T10:41:09.490 INFO:teuthology.orchestra.run.smithi114.stdout: "agent_refresh_rate": { 2024-04-23T10:41:09.490 INFO:teuthology.orchestra.run.smithi114.stdout: "name": "agent_refresh_rate", 2024-04-23T10:41:09.490 INFO:teuthology.orchestra.run.smithi114.stdout: "type": "secs", 2024-04-23T10:41:09.491 INFO:teuthology.orchestra.run.smithi114.stdout: "level": "advanced", 2024-04-23T10:41:09.491 INFO:teuthology.orchestra.run.smithi114.stdout: "flags": 0, 2024-04-23T10:41:09.491 INFO:teuthology.orchestra.run.smithi114.stdout: "default_value": "20", 2024-04-23T10:41:09.491 INFO:teuthology.orchestra.run.smithi114.stdout: "min": "", 2024-04-23T10:41:09.491 INFO:teuthology.orchestra.run.smithi114.stdout: "max": "", 2024-04-23T10:41:09.491 INFO:teuthology.orchestra.run.smithi114.stdout: "enum_allowed": [], 2024-04-23T10:41:09.491 INFO:teuthology.orchestra.run.smithi114.stdout: "desc": "How often agent on each host will try to gather and send metadata", 2024-04-23T10:41:09.491 INFO:teuthology.orchestra.run.smithi114.stdout: "long_desc": "", 2024-04-23T10:41:09.491 INFO:teuthology.orchestra.run.smithi114.stdout: "tags": [], 2024-04-23T10:41:09.492 INFO:teuthology.orchestra.run.smithi114.stdout: "see_also": [] 2024-04-23T10:41:09.492 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-23T10:41:09.492 INFO:teuthology.orchestra.run.smithi114.stdout: "agent_starting_port": { 2024-04-23T10:41:09.492 INFO:teuthology.orchestra.run.smithi114.stdout: "name": "agent_starting_port", 2024-04-23T10:41:09.492 INFO:teuthology.orchestra.run.smithi114.stdout: "type": "int", 2024-04-23T10:41:09.492 INFO:teuthology.orchestra.run.smithi114.stdout: "level": "advanced", 2024-04-23T10:41:09.492 INFO:teuthology.orchestra.run.smithi114.stdout: "flags": 0, 2024-04-23T10:41:09.492 INFO:teuthology.orchestra.run.smithi114.stdout: "default_value": "4721", 2024-04-23T10:41:09.492 INFO:teuthology.orchestra.run.smithi114.stdout: "min": "", 2024-04-23T10:41:09.492 INFO:teuthology.orchestra.run.smithi114.stdout: "max": "", 2024-04-23T10:41:09.493 INFO:teuthology.orchestra.run.smithi114.stdout: "enum_allowed": [], 2024-04-23T10:41:09.493 INFO:teuthology.orchestra.run.smithi114.stdout: "desc": "First port agent will try to bind to (will also try up to next 1000 subsequent ports if blocked)", 2024-04-23T10:41:09.493 INFO:teuthology.orchestra.run.smithi114.stdout: "long_desc": "", 2024-04-23T10:41:09.493 INFO:teuthology.orchestra.run.smithi114.stdout: "tags": [], 2024-04-23T10:41:09.493 INFO:teuthology.orchestra.run.smithi114.stdout: "see_also": [] 2024-04-23T10:41:09.493 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-23T10:41:09.493 INFO:teuthology.orchestra.run.smithi114.stdout: "allow_ptrace": { 2024-04-23T10:41:09.493 INFO:teuthology.orchestra.run.smithi114.stdout: "name": "allow_ptrace", 2024-04-23T10:41:09.493 INFO:teuthology.orchestra.run.smithi114.stdout: "type": "bool", 2024-04-23T10:41:09.494 INFO:teuthology.orchestra.run.smithi114.stdout: "level": "advanced", 2024-04-23T10:41:09.494 INFO:teuthology.orchestra.run.smithi114.stdout: "flags": 0, 2024-04-23T10:41:09.494 INFO:teuthology.orchestra.run.smithi114.stdout: "default_value": "False", 2024-04-23T10:41:09.494 INFO:teuthology.orchestra.run.smithi114.stdout: "min": "", 2024-04-23T10:41:09.494 INFO:teuthology.orchestra.run.smithi114.stdout: "max": "", 2024-04-23T10:41:09.494 INFO:teuthology.orchestra.run.smithi114.stdout: "enum_allowed": [], 2024-04-23T10:41:09.494 INFO:teuthology.orchestra.run.smithi114.stdout: "desc": "allow SYS_PTRACE capability on ceph containers", 2024-04-23T10:41:09.494 INFO:teuthology.orchestra.run.smithi114.stdout: "long_desc": "The SYS_PTRACE capability is needed to attach to a process with gdb or strace. Enabling this options can allow debugging daemons that encounter problems at runtime.", 2024-04-23T10:41:09.495 INFO:teuthology.orchestra.run.smithi114.stdout: "tags": [], 2024-04-23T10:41:09.495 INFO:teuthology.orchestra.run.smithi114.stdout: "see_also": [] 2024-04-23T10:41:09.495 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-23T10:41:09.495 INFO:teuthology.orchestra.run.smithi114.stdout: "autotune_interval": { 2024-04-23T10:41:09.495 INFO:teuthology.orchestra.run.smithi114.stdout: "name": "autotune_interval", 2024-04-23T10:41:09.495 INFO:teuthology.orchestra.run.smithi114.stdout: "type": "secs", 2024-04-23T10:41:09.495 INFO:teuthology.orchestra.run.smithi114.stdout: "level": "advanced", 2024-04-23T10:41:09.495 INFO:teuthology.orchestra.run.smithi114.stdout: "flags": 0, 2024-04-23T10:41:09.495 INFO:teuthology.orchestra.run.smithi114.stdout: "default_value": "600", 2024-04-23T10:41:09.495 INFO:teuthology.orchestra.run.smithi114.stdout: "min": "", 2024-04-23T10:41:09.496 INFO:teuthology.orchestra.run.smithi114.stdout: "max": "", 2024-04-23T10:41:09.496 INFO:teuthology.orchestra.run.smithi114.stdout: "enum_allowed": [], 2024-04-23T10:41:09.496 INFO:teuthology.orchestra.run.smithi114.stdout: "desc": "how frequently to autotune daemon memory", 2024-04-23T10:41:09.496 INFO:teuthology.orchestra.run.smithi114.stdout: "long_desc": "", 2024-04-23T10:41:09.496 INFO:teuthology.orchestra.run.smithi114.stdout: "tags": [], 2024-04-23T10:41:09.496 INFO:teuthology.orchestra.run.smithi114.stdout: "see_also": [] 2024-04-23T10:41:09.496 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-23T10:41:09.496 INFO:teuthology.orchestra.run.smithi114.stdout: "autotune_memory_target_ratio": { 2024-04-23T10:41:09.497 INFO:teuthology.orchestra.run.smithi114.stdout: "name": "autotune_memory_target_ratio", 2024-04-23T10:41:09.497 INFO:teuthology.orchestra.run.smithi114.stdout: "type": "float", 2024-04-23T10:41:09.497 INFO:teuthology.orchestra.run.smithi114.stdout: "level": "advanced", 2024-04-23T10:41:09.497 INFO:teuthology.orchestra.run.smithi114.stdout: "flags": 0, 2024-04-23T10:41:09.497 INFO:teuthology.orchestra.run.smithi114.stdout: "default_value": "0.7", 2024-04-23T10:41:09.497 INFO:teuthology.orchestra.run.smithi114.stdout: "min": "", 2024-04-23T10:41:09.497 INFO:teuthology.orchestra.run.smithi114.stdout: "max": "", 2024-04-23T10:41:09.497 INFO:teuthology.orchestra.run.smithi114.stdout: "enum_allowed": [], 2024-04-23T10:41:09.497 INFO:teuthology.orchestra.run.smithi114.stdout: "desc": "ratio of total system memory to divide amongst autotuned daemons", 2024-04-23T10:41:09.498 INFO:teuthology.orchestra.run.smithi114.stdout: "long_desc": "", 2024-04-23T10:41:09.498 INFO:teuthology.orchestra.run.smithi114.stdout: "tags": [], 2024-04-23T10:41:09.498 INFO:teuthology.orchestra.run.smithi114.stdout: "see_also": [] 2024-04-23T10:41:09.498 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-23T10:41:09.498 INFO:teuthology.orchestra.run.smithi114.stdout: "cephadm_log_destination": { 2024-04-23T10:41:09.498 INFO:teuthology.orchestra.run.smithi114.stdout: "name": "cephadm_log_destination", 2024-04-23T10:41:09.498 INFO:teuthology.orchestra.run.smithi114.stdout: "type": "str", 2024-04-23T10:41:09.498 INFO:teuthology.orchestra.run.smithi114.stdout: "level": "advanced", 2024-04-23T10:41:09.499 INFO:teuthology.orchestra.run.smithi114.stdout: "flags": 0, 2024-04-23T10:41:09.499 INFO:teuthology.orchestra.run.smithi114.stdout: "default_value": "", 2024-04-23T10:41:09.499 INFO:teuthology.orchestra.run.smithi114.stdout: "min": "", 2024-04-23T10:41:09.499 INFO:teuthology.orchestra.run.smithi114.stdout: "max": "", 2024-04-23T10:41:09.499 INFO:teuthology.orchestra.run.smithi114.stdout: "enum_allowed": [ 2024-04-23T10:41:09.499 INFO:teuthology.orchestra.run.smithi114.stdout: "file", 2024-04-23T10:41:09.499 INFO:teuthology.orchestra.run.smithi114.stdout: "file,syslog", 2024-04-23T10:41:09.499 INFO:teuthology.orchestra.run.smithi114.stdout: "syslog" 2024-04-23T10:41:09.499 INFO:teuthology.orchestra.run.smithi114.stdout: ], 2024-04-23T10:41:09.499 INFO:teuthology.orchestra.run.smithi114.stdout: "desc": "Destination for cephadm command's persistent logging", 2024-04-23T10:41:09.500 INFO:teuthology.orchestra.run.smithi114.stdout: "long_desc": "", 2024-04-23T10:41:09.500 INFO:teuthology.orchestra.run.smithi114.stdout: "tags": [], 2024-04-23T10:41:09.500 INFO:teuthology.orchestra.run.smithi114.stdout: "see_also": [] 2024-04-23T10:41:09.500 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-23T10:41:09.500 INFO:teuthology.orchestra.run.smithi114.stdout: "cgroups_split": { 2024-04-23T10:41:09.500 INFO:teuthology.orchestra.run.smithi114.stdout: "name": "cgroups_split", 2024-04-23T10:41:09.500 INFO:teuthology.orchestra.run.smithi114.stdout: "type": "bool", 2024-04-23T10:41:09.500 INFO:teuthology.orchestra.run.smithi114.stdout: "level": "advanced", 2024-04-23T10:41:09.500 INFO:teuthology.orchestra.run.smithi114.stdout: "flags": 0, 2024-04-23T10:41:09.501 INFO:teuthology.orchestra.run.smithi114.stdout: "default_value": "True", 2024-04-23T10:41:09.501 INFO:teuthology.orchestra.run.smithi114.stdout: "min": "", 2024-04-23T10:41:09.501 INFO:teuthology.orchestra.run.smithi114.stdout: "max": "", 2024-04-23T10:41:09.501 INFO:teuthology.orchestra.run.smithi114.stdout: "enum_allowed": [], 2024-04-23T10:41:09.501 INFO:teuthology.orchestra.run.smithi114.stdout: "desc": "Pass --cgroups=split when cephadm creates containers (currently podman only)", 2024-04-23T10:41:09.501 INFO:teuthology.orchestra.run.smithi114.stdout: "long_desc": "", 2024-04-23T10:41:09.501 INFO:teuthology.orchestra.run.smithi114.stdout: "tags": [], 2024-04-23T10:41:09.501 INFO:teuthology.orchestra.run.smithi114.stdout: "see_also": [] 2024-04-23T10:41:09.501 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-23T10:41:09.501 INFO:teuthology.orchestra.run.smithi114.stdout: "config_checks_enabled": { 2024-04-23T10:41:09.502 INFO:teuthology.orchestra.run.smithi114.stdout: "name": "config_checks_enabled", 2024-04-23T10:41:09.502 INFO:teuthology.orchestra.run.smithi114.stdout: "type": "bool", 2024-04-23T10:41:09.502 INFO:teuthology.orchestra.run.smithi114.stdout: "level": "advanced", 2024-04-23T10:41:09.502 INFO:teuthology.orchestra.run.smithi114.stdout: "flags": 0, 2024-04-23T10:41:09.502 INFO:teuthology.orchestra.run.smithi114.stdout: "default_value": "False", 2024-04-23T10:41:09.502 INFO:teuthology.orchestra.run.smithi114.stdout: "min": "", 2024-04-23T10:41:09.502 INFO:teuthology.orchestra.run.smithi114.stdout: "max": "", 2024-04-23T10:41:09.503 INFO:teuthology.orchestra.run.smithi114.stdout: "enum_allowed": [], 2024-04-23T10:41:09.503 INFO:teuthology.orchestra.run.smithi114.stdout: "desc": "Enable or disable the cephadm configuration analysis", 2024-04-23T10:41:09.503 INFO:teuthology.orchestra.run.smithi114.stdout: "long_desc": "", 2024-04-23T10:41:09.503 INFO:teuthology.orchestra.run.smithi114.stdout: "tags": [], 2024-04-23T10:41:09.503 INFO:teuthology.orchestra.run.smithi114.stdout: "see_also": [] 2024-04-23T10:41:09.503 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-23T10:41:09.503 INFO:teuthology.orchestra.run.smithi114.stdout: "config_dashboard": { 2024-04-23T10:41:09.503 INFO:teuthology.orchestra.run.smithi114.stdout: "name": "config_dashboard", 2024-04-23T10:41:09.503 INFO:teuthology.orchestra.run.smithi114.stdout: "type": "bool", 2024-04-23T10:41:09.504 INFO:teuthology.orchestra.run.smithi114.stdout: "level": "advanced", 2024-04-23T10:41:09.504 INFO:teuthology.orchestra.run.smithi114.stdout: "flags": 0, 2024-04-23T10:41:09.504 INFO:teuthology.orchestra.run.smithi114.stdout: "default_value": "True", 2024-04-23T10:41:09.504 INFO:teuthology.orchestra.run.smithi114.stdout: "min": "", 2024-04-23T10:41:09.504 INFO:teuthology.orchestra.run.smithi114.stdout: "max": "", 2024-04-23T10:41:09.504 INFO:teuthology.orchestra.run.smithi114.stdout: "enum_allowed": [], 2024-04-23T10:41:09.504 INFO:teuthology.orchestra.run.smithi114.stdout: "desc": "manage configs like API endpoints in Dashboard.", 2024-04-23T10:41:09.504 INFO:teuthology.orchestra.run.smithi114.stdout: "long_desc": "", 2024-04-23T10:41:09.504 INFO:teuthology.orchestra.run.smithi114.stdout: "tags": [], 2024-04-23T10:41:09.504 INFO:teuthology.orchestra.run.smithi114.stdout: "see_also": [] 2024-04-23T10:41:09.505 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-23T10:41:09.505 INFO:teuthology.orchestra.run.smithi114.stdout: "container_image_alertmanager": { 2024-04-23T10:41:09.505 INFO:teuthology.orchestra.run.smithi114.stdout: "name": "container_image_alertmanager", 2024-04-23T10:41:09.505 INFO:teuthology.orchestra.run.smithi114.stdout: "type": "str", 2024-04-23T10:41:09.505 INFO:teuthology.orchestra.run.smithi114.stdout: "level": "advanced", 2024-04-23T10:41:09.505 INFO:teuthology.orchestra.run.smithi114.stdout: "flags": 0, 2024-04-23T10:41:09.505 INFO:teuthology.orchestra.run.smithi114.stdout: "default_value": "quay.io/prometheus/alertmanager:v0.25.0", 2024-04-23T10:41:09.505 INFO:teuthology.orchestra.run.smithi114.stdout: "min": "", 2024-04-23T10:41:09.506 INFO:teuthology.orchestra.run.smithi114.stdout: "max": "", 2024-04-23T10:41:09.506 INFO:teuthology.orchestra.run.smithi114.stdout: "enum_allowed": [], 2024-04-23T10:41:09.506 INFO:teuthology.orchestra.run.smithi114.stdout: "desc": "Prometheus container image", 2024-04-23T10:41:09.506 INFO:teuthology.orchestra.run.smithi114.stdout: "long_desc": "", 2024-04-23T10:41:09.506 INFO:teuthology.orchestra.run.smithi114.stdout: "tags": [], 2024-04-23T10:41:09.506 INFO:teuthology.orchestra.run.smithi114.stdout: "see_also": [] 2024-04-23T10:41:09.506 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-23T10:41:09.506 INFO:teuthology.orchestra.run.smithi114.stdout: "container_image_base": { 2024-04-23T10:41:09.506 INFO:teuthology.orchestra.run.smithi114.stdout: "name": "container_image_base", 2024-04-23T10:41:09.507 INFO:teuthology.orchestra.run.smithi114.stdout: "type": "str", 2024-04-23T10:41:09.507 INFO:teuthology.orchestra.run.smithi114.stdout: "level": "advanced", 2024-04-23T10:41:09.507 INFO:teuthology.orchestra.run.smithi114.stdout: "flags": 1, 2024-04-23T10:41:09.507 INFO:teuthology.orchestra.run.smithi114.stdout: "default_value": "quay.io/ceph/ceph", 2024-04-23T10:41:09.507 INFO:teuthology.orchestra.run.smithi114.stdout: "min": "", 2024-04-23T10:41:09.507 INFO:teuthology.orchestra.run.smithi114.stdout: "max": "", 2024-04-23T10:41:09.507 INFO:teuthology.orchestra.run.smithi114.stdout: "enum_allowed": [], 2024-04-23T10:41:09.507 INFO:teuthology.orchestra.run.smithi114.stdout: "desc": "Container image name, without the tag", 2024-04-23T10:41:09.507 INFO:teuthology.orchestra.run.smithi114.stdout: "long_desc": "", 2024-04-23T10:41:09.508 INFO:teuthology.orchestra.run.smithi114.stdout: "tags": [], 2024-04-23T10:41:09.508 INFO:teuthology.orchestra.run.smithi114.stdout: "see_also": [] 2024-04-23T10:41:09.508 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-23T10:41:09.508 INFO:teuthology.orchestra.run.smithi114.stdout: "container_image_elasticsearch": { 2024-04-23T10:41:09.508 INFO:teuthology.orchestra.run.smithi114.stdout: "name": "container_image_elasticsearch", 2024-04-23T10:41:09.508 INFO:teuthology.orchestra.run.smithi114.stdout: "type": "str", 2024-04-23T10:41:09.508 INFO:teuthology.orchestra.run.smithi114.stdout: "level": "advanced", 2024-04-23T10:41:09.508 INFO:teuthology.orchestra.run.smithi114.stdout: "flags": 0, 2024-04-23T10:41:09.509 INFO:teuthology.orchestra.run.smithi114.stdout: "default_value": "quay.io/omrizeneva/elasticsearch:6.8.23", 2024-04-23T10:41:09.509 INFO:teuthology.orchestra.run.smithi114.stdout: "min": "", 2024-04-23T10:41:09.509 INFO:teuthology.orchestra.run.smithi114.stdout: "max": "", 2024-04-23T10:41:09.509 INFO:teuthology.orchestra.run.smithi114.stdout: "enum_allowed": [], 2024-04-23T10:41:09.509 INFO:teuthology.orchestra.run.smithi114.stdout: "desc": "elasticsearch container image", 2024-04-23T10:41:09.509 INFO:teuthology.orchestra.run.smithi114.stdout: "long_desc": "", 2024-04-23T10:41:09.509 INFO:teuthology.orchestra.run.smithi114.stdout: "tags": [], 2024-04-23T10:41:09.509 INFO:teuthology.orchestra.run.smithi114.stdout: "see_also": [] 2024-04-23T10:41:09.510 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-23T10:41:09.510 INFO:teuthology.orchestra.run.smithi114.stdout: "container_image_grafana": { 2024-04-23T10:41:09.510 INFO:teuthology.orchestra.run.smithi114.stdout: "name": "container_image_grafana", 2024-04-23T10:41:09.510 INFO:teuthology.orchestra.run.smithi114.stdout: "type": "str", 2024-04-23T10:41:09.510 INFO:teuthology.orchestra.run.smithi114.stdout: "level": "advanced", 2024-04-23T10:41:09.510 INFO:teuthology.orchestra.run.smithi114.stdout: "flags": 0, 2024-04-23T10:41:09.510 INFO:teuthology.orchestra.run.smithi114.stdout: "default_value": "quay.io/ceph/grafana:9.4.12", 2024-04-23T10:41:09.510 INFO:teuthology.orchestra.run.smithi114.stdout: "min": "", 2024-04-23T10:41:09.511 INFO:teuthology.orchestra.run.smithi114.stdout: "max": "", 2024-04-23T10:41:09.511 INFO:teuthology.orchestra.run.smithi114.stdout: "enum_allowed": [], 2024-04-23T10:41:09.511 INFO:teuthology.orchestra.run.smithi114.stdout: "desc": "Prometheus container image", 2024-04-23T10:41:09.511 INFO:teuthology.orchestra.run.smithi114.stdout: "long_desc": "", 2024-04-23T10:41:09.511 INFO:teuthology.orchestra.run.smithi114.stdout: "tags": [], 2024-04-23T10:41:09.511 INFO:teuthology.orchestra.run.smithi114.stdout: "see_also": [] 2024-04-23T10:41:09.511 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-23T10:41:09.511 INFO:teuthology.orchestra.run.smithi114.stdout: "container_image_haproxy": { 2024-04-23T10:41:09.511 INFO:teuthology.orchestra.run.smithi114.stdout: "name": "container_image_haproxy", 2024-04-23T10:41:09.512 INFO:teuthology.orchestra.run.smithi114.stdout: "type": "str", 2024-04-23T10:41:09.512 INFO:teuthology.orchestra.run.smithi114.stdout: "level": "advanced", 2024-04-23T10:41:09.512 INFO:teuthology.orchestra.run.smithi114.stdout: "flags": 0, 2024-04-23T10:41:09.512 INFO:teuthology.orchestra.run.smithi114.stdout: "default_value": "quay.io/ceph/haproxy:2.3", 2024-04-23T10:41:09.512 INFO:teuthology.orchestra.run.smithi114.stdout: "min": "", 2024-04-23T10:41:09.512 INFO:teuthology.orchestra.run.smithi114.stdout: "max": "", 2024-04-23T10:41:09.512 INFO:teuthology.orchestra.run.smithi114.stdout: "enum_allowed": [], 2024-04-23T10:41:09.512 INFO:teuthology.orchestra.run.smithi114.stdout: "desc": "HAproxy container image", 2024-04-23T10:41:09.512 INFO:teuthology.orchestra.run.smithi114.stdout: "long_desc": "", 2024-04-23T10:41:09.513 INFO:teuthology.orchestra.run.smithi114.stdout: "tags": [], 2024-04-23T10:41:09.513 INFO:teuthology.orchestra.run.smithi114.stdout: "see_also": [] 2024-04-23T10:41:09.513 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-23T10:41:09.513 INFO:teuthology.orchestra.run.smithi114.stdout: "container_image_jaeger_agent": { 2024-04-23T10:41:09.513 INFO:teuthology.orchestra.run.smithi114.stdout: "name": "container_image_jaeger_agent", 2024-04-23T10:41:09.513 INFO:teuthology.orchestra.run.smithi114.stdout: "type": "str", 2024-04-23T10:41:09.513 INFO:teuthology.orchestra.run.smithi114.stdout: "level": "advanced", 2024-04-23T10:41:09.513 INFO:teuthology.orchestra.run.smithi114.stdout: "flags": 0, 2024-04-23T10:41:09.513 INFO:teuthology.orchestra.run.smithi114.stdout: "default_value": "quay.io/jaegertracing/jaeger-agent:1.29", 2024-04-23T10:41:09.513 INFO:teuthology.orchestra.run.smithi114.stdout: "min": "", 2024-04-23T10:41:09.514 INFO:teuthology.orchestra.run.smithi114.stdout: "max": "", 2024-04-23T10:41:09.514 INFO:teuthology.orchestra.run.smithi114.stdout: "enum_allowed": [], 2024-04-23T10:41:09.514 INFO:teuthology.orchestra.run.smithi114.stdout: "desc": "Jaeger agent container image", 2024-04-23T10:41:09.514 INFO:teuthology.orchestra.run.smithi114.stdout: "long_desc": "", 2024-04-23T10:41:09.514 INFO:teuthology.orchestra.run.smithi114.stdout: "tags": [], 2024-04-23T10:41:09.514 INFO:teuthology.orchestra.run.smithi114.stdout: "see_also": [] 2024-04-23T10:41:09.514 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-23T10:41:09.514 INFO:teuthology.orchestra.run.smithi114.stdout: "container_image_jaeger_collector": { 2024-04-23T10:41:09.515 INFO:teuthology.orchestra.run.smithi114.stdout: "name": "container_image_jaeger_collector", 2024-04-23T10:41:09.515 INFO:teuthology.orchestra.run.smithi114.stdout: "type": "str", 2024-04-23T10:41:09.515 INFO:teuthology.orchestra.run.smithi114.stdout: "level": "advanced", 2024-04-23T10:41:09.515 INFO:teuthology.orchestra.run.smithi114.stdout: "flags": 0, 2024-04-23T10:41:09.515 INFO:teuthology.orchestra.run.smithi114.stdout: "default_value": "quay.io/jaegertracing/jaeger-collector:1.29", 2024-04-23T10:41:09.515 INFO:teuthology.orchestra.run.smithi114.stdout: "min": "", 2024-04-23T10:41:09.515 INFO:teuthology.orchestra.run.smithi114.stdout: "max": "", 2024-04-23T10:41:09.515 INFO:teuthology.orchestra.run.smithi114.stdout: "enum_allowed": [], 2024-04-23T10:41:09.515 INFO:teuthology.orchestra.run.smithi114.stdout: "desc": "Jaeger collector container image", 2024-04-23T10:41:09.515 INFO:teuthology.orchestra.run.smithi114.stdout: "long_desc": "", 2024-04-23T10:41:09.516 INFO:teuthology.orchestra.run.smithi114.stdout: "tags": [], 2024-04-23T10:41:09.516 INFO:teuthology.orchestra.run.smithi114.stdout: "see_also": [] 2024-04-23T10:41:09.516 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-23T10:41:09.516 INFO:teuthology.orchestra.run.smithi114.stdout: "container_image_jaeger_query": { 2024-04-23T10:41:09.516 INFO:teuthology.orchestra.run.smithi114.stdout: "name": "container_image_jaeger_query", 2024-04-23T10:41:09.516 INFO:teuthology.orchestra.run.smithi114.stdout: "type": "str", 2024-04-23T10:41:09.516 INFO:teuthology.orchestra.run.smithi114.stdout: "level": "advanced", 2024-04-23T10:41:09.516 INFO:teuthology.orchestra.run.smithi114.stdout: "flags": 0, 2024-04-23T10:41:09.516 INFO:teuthology.orchestra.run.smithi114.stdout: "default_value": "quay.io/jaegertracing/jaeger-query:1.29", 2024-04-23T10:41:09.517 INFO:teuthology.orchestra.run.smithi114.stdout: "min": "", 2024-04-23T10:41:09.517 INFO:teuthology.orchestra.run.smithi114.stdout: "max": "", 2024-04-23T10:41:09.517 INFO:teuthology.orchestra.run.smithi114.stdout: "enum_allowed": [], 2024-04-23T10:41:09.517 INFO:teuthology.orchestra.run.smithi114.stdout: "desc": "Jaeger query container image", 2024-04-23T10:41:09.517 INFO:teuthology.orchestra.run.smithi114.stdout: "long_desc": "", 2024-04-23T10:41:09.517 INFO:teuthology.orchestra.run.smithi114.stdout: "tags": [], 2024-04-23T10:41:09.517 INFO:teuthology.orchestra.run.smithi114.stdout: "see_also": [] 2024-04-23T10:41:09.517 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-23T10:41:09.517 INFO:teuthology.orchestra.run.smithi114.stdout: "container_image_keepalived": { 2024-04-23T10:41:09.517 INFO:teuthology.orchestra.run.smithi114.stdout: "name": "container_image_keepalived", 2024-04-23T10:41:09.518 INFO:teuthology.orchestra.run.smithi114.stdout: "type": "str", 2024-04-23T10:41:09.518 INFO:teuthology.orchestra.run.smithi114.stdout: "level": "advanced", 2024-04-23T10:41:09.518 INFO:teuthology.orchestra.run.smithi114.stdout: "flags": 0, 2024-04-23T10:41:09.518 INFO:teuthology.orchestra.run.smithi114.stdout: "default_value": "quay.io/ceph/keepalived:2.2.4", 2024-04-23T10:41:09.518 INFO:teuthology.orchestra.run.smithi114.stdout: "min": "", 2024-04-23T10:41:09.518 INFO:teuthology.orchestra.run.smithi114.stdout: "max": "", 2024-04-23T10:41:09.518 INFO:teuthology.orchestra.run.smithi114.stdout: "enum_allowed": [], 2024-04-23T10:41:09.518 INFO:teuthology.orchestra.run.smithi114.stdout: "desc": "Keepalived container image", 2024-04-23T10:41:09.519 INFO:teuthology.orchestra.run.smithi114.stdout: "long_desc": "", 2024-04-23T10:41:09.519 INFO:teuthology.orchestra.run.smithi114.stdout: "tags": [], 2024-04-23T10:41:09.519 INFO:teuthology.orchestra.run.smithi114.stdout: "see_also": [] 2024-04-23T10:41:09.519 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-23T10:41:09.519 INFO:teuthology.orchestra.run.smithi114.stdout: "container_image_loki": { 2024-04-23T10:41:09.519 INFO:teuthology.orchestra.run.smithi114.stdout: "name": "container_image_loki", 2024-04-23T10:41:09.519 INFO:teuthology.orchestra.run.smithi114.stdout: "type": "str", 2024-04-23T10:41:09.519 INFO:teuthology.orchestra.run.smithi114.stdout: "level": "advanced", 2024-04-23T10:41:09.519 INFO:teuthology.orchestra.run.smithi114.stdout: "flags": 0, 2024-04-23T10:41:09.519 INFO:teuthology.orchestra.run.smithi114.stdout: "default_value": "docker.io/grafana/loki:2.4.0", 2024-04-23T10:41:09.520 INFO:teuthology.orchestra.run.smithi114.stdout: "min": "", 2024-04-23T10:41:09.520 INFO:teuthology.orchestra.run.smithi114.stdout: "max": "", 2024-04-23T10:41:09.520 INFO:teuthology.orchestra.run.smithi114.stdout: "enum_allowed": [], 2024-04-23T10:41:09.520 INFO:teuthology.orchestra.run.smithi114.stdout: "desc": "Loki container image", 2024-04-23T10:41:09.520 INFO:teuthology.orchestra.run.smithi114.stdout: "long_desc": "", 2024-04-23T10:41:09.520 INFO:teuthology.orchestra.run.smithi114.stdout: "tags": [], 2024-04-23T10:41:09.520 INFO:teuthology.orchestra.run.smithi114.stdout: "see_also": [] 2024-04-23T10:41:09.520 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-23T10:41:09.520 INFO:teuthology.orchestra.run.smithi114.stdout: "container_image_node_exporter": { 2024-04-23T10:41:09.521 INFO:teuthology.orchestra.run.smithi114.stdout: "name": "container_image_node_exporter", 2024-04-23T10:41:09.521 INFO:teuthology.orchestra.run.smithi114.stdout: "type": "str", 2024-04-23T10:41:09.521 INFO:teuthology.orchestra.run.smithi114.stdout: "level": "advanced", 2024-04-23T10:41:09.521 INFO:teuthology.orchestra.run.smithi114.stdout: "flags": 0, 2024-04-23T10:41:09.521 INFO:teuthology.orchestra.run.smithi114.stdout: "default_value": "quay.io/prometheus/node-exporter:v1.5.0", 2024-04-23T10:41:09.521 INFO:teuthology.orchestra.run.smithi114.stdout: "min": "", 2024-04-23T10:41:09.521 INFO:teuthology.orchestra.run.smithi114.stdout: "max": "", 2024-04-23T10:41:09.521 INFO:teuthology.orchestra.run.smithi114.stdout: "enum_allowed": [], 2024-04-23T10:41:09.521 INFO:teuthology.orchestra.run.smithi114.stdout: "desc": "Prometheus container image", 2024-04-23T10:41:09.521 INFO:teuthology.orchestra.run.smithi114.stdout: "long_desc": "", 2024-04-23T10:41:09.522 INFO:teuthology.orchestra.run.smithi114.stdout: "tags": [], 2024-04-23T10:41:09.522 INFO:teuthology.orchestra.run.smithi114.stdout: "see_also": [] 2024-04-23T10:41:09.522 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-23T10:41:09.522 INFO:teuthology.orchestra.run.smithi114.stdout: "container_image_nvmeof": { 2024-04-23T10:41:09.522 INFO:teuthology.orchestra.run.smithi114.stdout: "name": "container_image_nvmeof", 2024-04-23T10:41:09.522 INFO:teuthology.orchestra.run.smithi114.stdout: "type": "str", 2024-04-23T10:41:09.522 INFO:teuthology.orchestra.run.smithi114.stdout: "level": "advanced", 2024-04-23T10:41:09.522 INFO:teuthology.orchestra.run.smithi114.stdout: "flags": 0, 2024-04-23T10:41:09.523 INFO:teuthology.orchestra.run.smithi114.stdout: "default_value": "quay.io/ceph/nvmeof:1.2.1", 2024-04-23T10:41:09.523 INFO:teuthology.orchestra.run.smithi114.stdout: "min": "", 2024-04-23T10:41:09.523 INFO:teuthology.orchestra.run.smithi114.stdout: "max": "", 2024-04-23T10:41:09.523 INFO:teuthology.orchestra.run.smithi114.stdout: "enum_allowed": [], 2024-04-23T10:41:09.523 INFO:teuthology.orchestra.run.smithi114.stdout: "desc": "Nvme-of container image", 2024-04-23T10:41:09.523 INFO:teuthology.orchestra.run.smithi114.stdout: "long_desc": "", 2024-04-23T10:41:09.523 INFO:teuthology.orchestra.run.smithi114.stdout: "tags": [], 2024-04-23T10:41:09.523 INFO:teuthology.orchestra.run.smithi114.stdout: "see_also": [] 2024-04-23T10:41:09.523 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-23T10:41:09.524 INFO:teuthology.orchestra.run.smithi114.stdout: "container_image_prometheus": { 2024-04-23T10:41:09.524 INFO:teuthology.orchestra.run.smithi114.stdout: "name": "container_image_prometheus", 2024-04-23T10:41:09.524 INFO:teuthology.orchestra.run.smithi114.stdout: "type": "str", 2024-04-23T10:41:09.524 INFO:teuthology.orchestra.run.smithi114.stdout: "level": "advanced", 2024-04-23T10:41:09.524 INFO:teuthology.orchestra.run.smithi114.stdout: "flags": 0, 2024-04-23T10:41:09.524 INFO:teuthology.orchestra.run.smithi114.stdout: "default_value": "quay.io/prometheus/prometheus:v2.43.0", 2024-04-23T10:41:09.524 INFO:teuthology.orchestra.run.smithi114.stdout: "min": "", 2024-04-23T10:41:09.524 INFO:teuthology.orchestra.run.smithi114.stdout: "max": "", 2024-04-23T10:41:09.524 INFO:teuthology.orchestra.run.smithi114.stdout: "enum_allowed": [], 2024-04-23T10:41:09.525 INFO:teuthology.orchestra.run.smithi114.stdout: "desc": "Prometheus container image", 2024-04-23T10:41:09.525 INFO:teuthology.orchestra.run.smithi114.stdout: "long_desc": "", 2024-04-23T10:41:09.525 INFO:teuthology.orchestra.run.smithi114.stdout: "tags": [], 2024-04-23T10:41:09.525 INFO:teuthology.orchestra.run.smithi114.stdout: "see_also": [] 2024-04-23T10:41:09.525 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-23T10:41:09.525 INFO:teuthology.orchestra.run.smithi114.stdout: "container_image_promtail": { 2024-04-23T10:41:09.525 INFO:teuthology.orchestra.run.smithi114.stdout: "name": "container_image_promtail", 2024-04-23T10:41:09.525 INFO:teuthology.orchestra.run.smithi114.stdout: "type": "str", 2024-04-23T10:41:09.525 INFO:teuthology.orchestra.run.smithi114.stdout: "level": "advanced", 2024-04-23T10:41:09.525 INFO:teuthology.orchestra.run.smithi114.stdout: "flags": 0, 2024-04-23T10:41:09.526 INFO:teuthology.orchestra.run.smithi114.stdout: "default_value": "docker.io/grafana/promtail:2.4.0", 2024-04-23T10:41:09.526 INFO:teuthology.orchestra.run.smithi114.stdout: "min": "", 2024-04-23T10:41:09.526 INFO:teuthology.orchestra.run.smithi114.stdout: "max": "", 2024-04-23T10:41:09.526 INFO:teuthology.orchestra.run.smithi114.stdout: "enum_allowed": [], 2024-04-23T10:41:09.526 INFO:teuthology.orchestra.run.smithi114.stdout: "desc": "Promtail container image", 2024-04-23T10:41:09.526 INFO:teuthology.orchestra.run.smithi114.stdout: "long_desc": "", 2024-04-23T10:41:09.526 INFO:teuthology.orchestra.run.smithi114.stdout: "tags": [], 2024-04-23T10:41:09.526 INFO:teuthology.orchestra.run.smithi114.stdout: "see_also": [] 2024-04-23T10:41:09.527 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-23T10:41:09.527 INFO:teuthology.orchestra.run.smithi114.stdout: "container_image_samba": { 2024-04-23T10:41:09.527 INFO:teuthology.orchestra.run.smithi114.stdout: "name": "container_image_samba", 2024-04-23T10:41:09.527 INFO:teuthology.orchestra.run.smithi114.stdout: "type": "str", 2024-04-23T10:41:09.527 INFO:teuthology.orchestra.run.smithi114.stdout: "level": "advanced", 2024-04-23T10:41:09.527 INFO:teuthology.orchestra.run.smithi114.stdout: "flags": 0, 2024-04-23T10:41:09.527 INFO:teuthology.orchestra.run.smithi114.stdout: "default_value": "quay.io/samba.org/samba-server:devbuilds-centos-amd64", 2024-04-23T10:41:09.527 INFO:teuthology.orchestra.run.smithi114.stdout: "min": "", 2024-04-23T10:41:09.527 INFO:teuthology.orchestra.run.smithi114.stdout: "max": "", 2024-04-23T10:41:09.527 INFO:teuthology.orchestra.run.smithi114.stdout: "enum_allowed": [], 2024-04-23T10:41:09.528 INFO:teuthology.orchestra.run.smithi114.stdout: "desc": "Samba/SMB container image", 2024-04-23T10:41:09.528 INFO:teuthology.orchestra.run.smithi114.stdout: "long_desc": "", 2024-04-23T10:41:09.528 INFO:teuthology.orchestra.run.smithi114.stdout: "tags": [], 2024-04-23T10:41:09.528 INFO:teuthology.orchestra.run.smithi114.stdout: "see_also": [] 2024-04-23T10:41:09.528 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-23T10:41:09.528 INFO:teuthology.orchestra.run.smithi114.stdout: "container_image_snmp_gateway": { 2024-04-23T10:41:09.528 INFO:teuthology.orchestra.run.smithi114.stdout: "name": "container_image_snmp_gateway", 2024-04-23T10:41:09.528 INFO:teuthology.orchestra.run.smithi114.stdout: "type": "str", 2024-04-23T10:41:09.528 INFO:teuthology.orchestra.run.smithi114.stdout: "level": "advanced", 2024-04-23T10:41:09.529 INFO:teuthology.orchestra.run.smithi114.stdout: "flags": 0, 2024-04-23T10:41:09.529 INFO:teuthology.orchestra.run.smithi114.stdout: "default_value": "docker.io/maxwo/snmp-notifier:v1.2.1", 2024-04-23T10:41:09.529 INFO:teuthology.orchestra.run.smithi114.stdout: "min": "", 2024-04-23T10:41:09.529 INFO:teuthology.orchestra.run.smithi114.stdout: "max": "", 2024-04-23T10:41:09.529 INFO:teuthology.orchestra.run.smithi114.stdout: "enum_allowed": [], 2024-04-23T10:41:09.529 INFO:teuthology.orchestra.run.smithi114.stdout: "desc": "SNMP Gateway container image", 2024-04-23T10:41:09.529 INFO:teuthology.orchestra.run.smithi114.stdout: "long_desc": "", 2024-04-23T10:41:09.529 INFO:teuthology.orchestra.run.smithi114.stdout: "tags": [], 2024-04-23T10:41:09.529 INFO:teuthology.orchestra.run.smithi114.stdout: "see_also": [] 2024-04-23T10:41:09.529 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-23T10:41:09.530 INFO:teuthology.orchestra.run.smithi114.stdout: "container_init": { 2024-04-23T10:41:09.530 INFO:teuthology.orchestra.run.smithi114.stdout: "name": "container_init", 2024-04-23T10:41:09.530 INFO:teuthology.orchestra.run.smithi114.stdout: "type": "bool", 2024-04-23T10:41:09.530 INFO:teuthology.orchestra.run.smithi114.stdout: "level": "advanced", 2024-04-23T10:41:09.530 INFO:teuthology.orchestra.run.smithi114.stdout: "flags": 0, 2024-04-23T10:41:09.530 INFO:teuthology.orchestra.run.smithi114.stdout: "default_value": "True", 2024-04-23T10:41:09.530 INFO:teuthology.orchestra.run.smithi114.stdout: "min": "", 2024-04-23T10:41:09.530 INFO:teuthology.orchestra.run.smithi114.stdout: "max": "", 2024-04-23T10:41:09.531 INFO:teuthology.orchestra.run.smithi114.stdout: "enum_allowed": [], 2024-04-23T10:41:09.531 INFO:teuthology.orchestra.run.smithi114.stdout: "desc": "Run podman/docker with `--init`", 2024-04-23T10:41:09.531 INFO:teuthology.orchestra.run.smithi114.stdout: "long_desc": "", 2024-04-23T10:41:09.531 INFO:teuthology.orchestra.run.smithi114.stdout: "tags": [], 2024-04-23T10:41:09.531 INFO:teuthology.orchestra.run.smithi114.stdout: "see_also": [] 2024-04-23T10:41:09.531 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-23T10:41:09.531 INFO:teuthology.orchestra.run.smithi114.stdout: "daemon_cache_timeout": { 2024-04-23T10:41:09.531 INFO:teuthology.orchestra.run.smithi114.stdout: "name": "daemon_cache_timeout", 2024-04-23T10:41:09.531 INFO:teuthology.orchestra.run.smithi114.stdout: "type": "secs", 2024-04-23T10:41:09.532 INFO:teuthology.orchestra.run.smithi114.stdout: "level": "advanced", 2024-04-23T10:41:09.532 INFO:teuthology.orchestra.run.smithi114.stdout: "flags": 0, 2024-04-23T10:41:09.532 INFO:teuthology.orchestra.run.smithi114.stdout: "default_value": "600", 2024-04-23T10:41:09.532 INFO:teuthology.orchestra.run.smithi114.stdout: "min": "", 2024-04-23T10:41:09.532 INFO:teuthology.orchestra.run.smithi114.stdout: "max": "", 2024-04-23T10:41:09.532 INFO:teuthology.orchestra.run.smithi114.stdout: "enum_allowed": [], 2024-04-23T10:41:09.532 INFO:teuthology.orchestra.run.smithi114.stdout: "desc": "seconds to cache service (daemon) inventory", 2024-04-23T10:41:09.532 INFO:teuthology.orchestra.run.smithi114.stdout: "long_desc": "", 2024-04-23T10:41:09.532 INFO:teuthology.orchestra.run.smithi114.stdout: "tags": [], 2024-04-23T10:41:09.532 INFO:teuthology.orchestra.run.smithi114.stdout: "see_also": [] 2024-04-23T10:41:09.533 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-23T10:41:09.533 INFO:teuthology.orchestra.run.smithi114.stdout: "default_cephadm_command_timeout": { 2024-04-23T10:41:09.533 INFO:teuthology.orchestra.run.smithi114.stdout: "name": "default_cephadm_command_timeout", 2024-04-23T10:41:09.533 INFO:teuthology.orchestra.run.smithi114.stdout: "type": "int", 2024-04-23T10:41:09.533 INFO:teuthology.orchestra.run.smithi114.stdout: "level": "advanced", 2024-04-23T10:41:09.533 INFO:teuthology.orchestra.run.smithi114.stdout: "flags": 0, 2024-04-23T10:41:09.533 INFO:teuthology.orchestra.run.smithi114.stdout: "default_value": "900", 2024-04-23T10:41:09.533 INFO:teuthology.orchestra.run.smithi114.stdout: "min": "", 2024-04-23T10:41:09.533 INFO:teuthology.orchestra.run.smithi114.stdout: "max": "", 2024-04-23T10:41:09.534 INFO:teuthology.orchestra.run.smithi114.stdout: "enum_allowed": [], 2024-04-23T10:41:09.534 INFO:teuthology.orchestra.run.smithi114.stdout: "desc": "Default timeout applied to cephadm commands run directly on the host (in seconds)", 2024-04-23T10:41:09.534 INFO:teuthology.orchestra.run.smithi114.stdout: "long_desc": "", 2024-04-23T10:41:09.534 INFO:teuthology.orchestra.run.smithi114.stdout: "tags": [], 2024-04-23T10:41:09.534 INFO:teuthology.orchestra.run.smithi114.stdout: "see_also": [] 2024-04-23T10:41:09.534 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-23T10:41:09.534 INFO:teuthology.orchestra.run.smithi114.stdout: "default_registry": { 2024-04-23T10:41:09.534 INFO:teuthology.orchestra.run.smithi114.stdout: "name": "default_registry", 2024-04-23T10:41:09.534 INFO:teuthology.orchestra.run.smithi114.stdout: "type": "str", 2024-04-23T10:41:09.535 INFO:teuthology.orchestra.run.smithi114.stdout: "level": "advanced", 2024-04-23T10:41:09.535 INFO:teuthology.orchestra.run.smithi114.stdout: "flags": 0, 2024-04-23T10:41:09.535 INFO:teuthology.orchestra.run.smithi114.stdout: "default_value": "docker.io", 2024-04-23T10:41:09.535 INFO:teuthology.orchestra.run.smithi114.stdout: "min": "", 2024-04-23T10:41:09.535 INFO:teuthology.orchestra.run.smithi114.stdout: "max": "", 2024-04-23T10:41:09.535 INFO:teuthology.orchestra.run.smithi114.stdout: "enum_allowed": [], 2024-04-23T10:41:09.535 INFO:teuthology.orchestra.run.smithi114.stdout: "desc": "Search-registry to which we should normalize unqualified image names. This is not the default registry", 2024-04-23T10:41:09.535 INFO:teuthology.orchestra.run.smithi114.stdout: "long_desc": "", 2024-04-23T10:41:09.535 INFO:teuthology.orchestra.run.smithi114.stdout: "tags": [], 2024-04-23T10:41:09.535 INFO:teuthology.orchestra.run.smithi114.stdout: "see_also": [] 2024-04-23T10:41:09.536 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-23T10:41:09.536 INFO:teuthology.orchestra.run.smithi114.stdout: "device_cache_timeout": { 2024-04-23T10:41:09.536 INFO:teuthology.orchestra.run.smithi114.stdout: "name": "device_cache_timeout", 2024-04-23T10:41:09.536 INFO:teuthology.orchestra.run.smithi114.stdout: "type": "secs", 2024-04-23T10:41:09.536 INFO:teuthology.orchestra.run.smithi114.stdout: "level": "advanced", 2024-04-23T10:41:09.536 INFO:teuthology.orchestra.run.smithi114.stdout: "flags": 0, 2024-04-23T10:41:09.536 INFO:teuthology.orchestra.run.smithi114.stdout: "default_value": "1800", 2024-04-23T10:41:09.536 INFO:teuthology.orchestra.run.smithi114.stdout: "min": "", 2024-04-23T10:41:09.536 INFO:teuthology.orchestra.run.smithi114.stdout: "max": "", 2024-04-23T10:41:09.537 INFO:teuthology.orchestra.run.smithi114.stdout: "enum_allowed": [], 2024-04-23T10:41:09.537 INFO:teuthology.orchestra.run.smithi114.stdout: "desc": "seconds to cache device inventory", 2024-04-23T10:41:09.537 INFO:teuthology.orchestra.run.smithi114.stdout: "long_desc": "", 2024-04-23T10:41:09.537 INFO:teuthology.orchestra.run.smithi114.stdout: "tags": [], 2024-04-23T10:41:09.537 INFO:teuthology.orchestra.run.smithi114.stdout: "see_also": [] 2024-04-23T10:41:09.537 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-23T10:41:09.537 INFO:teuthology.orchestra.run.smithi114.stdout: "device_enhanced_scan": { 2024-04-23T10:41:09.537 INFO:teuthology.orchestra.run.smithi114.stdout: "name": "device_enhanced_scan", 2024-04-23T10:41:09.537 INFO:teuthology.orchestra.run.smithi114.stdout: "type": "bool", 2024-04-23T10:41:09.537 INFO:teuthology.orchestra.run.smithi114.stdout: "level": "advanced", 2024-04-23T10:41:09.538 INFO:teuthology.orchestra.run.smithi114.stdout: "flags": 0, 2024-04-23T10:41:09.538 INFO:teuthology.orchestra.run.smithi114.stdout: "default_value": "False", 2024-04-23T10:41:09.538 INFO:teuthology.orchestra.run.smithi114.stdout: "min": "", 2024-04-23T10:41:09.538 INFO:teuthology.orchestra.run.smithi114.stdout: "max": "", 2024-04-23T10:41:09.538 INFO:teuthology.orchestra.run.smithi114.stdout: "enum_allowed": [], 2024-04-23T10:41:09.538 INFO:teuthology.orchestra.run.smithi114.stdout: "desc": "Use libstoragemgmt during device scans", 2024-04-23T10:41:09.538 INFO:teuthology.orchestra.run.smithi114.stdout: "long_desc": "", 2024-04-23T10:41:09.538 INFO:teuthology.orchestra.run.smithi114.stdout: "tags": [], 2024-04-23T10:41:09.538 INFO:teuthology.orchestra.run.smithi114.stdout: "see_also": [] 2024-04-23T10:41:09.539 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-23T10:41:09.539 INFO:teuthology.orchestra.run.smithi114.stdout: "facts_cache_timeout": { 2024-04-23T10:41:09.539 INFO:teuthology.orchestra.run.smithi114.stdout: "name": "facts_cache_timeout", 2024-04-23T10:41:09.539 INFO:teuthology.orchestra.run.smithi114.stdout: "type": "secs", 2024-04-23T10:41:09.539 INFO:teuthology.orchestra.run.smithi114.stdout: "level": "advanced", 2024-04-23T10:41:09.539 INFO:teuthology.orchestra.run.smithi114.stdout: "flags": 0, 2024-04-23T10:41:09.539 INFO:teuthology.orchestra.run.smithi114.stdout: "default_value": "60", 2024-04-23T10:41:09.539 INFO:teuthology.orchestra.run.smithi114.stdout: "min": "", 2024-04-23T10:41:09.539 INFO:teuthology.orchestra.run.smithi114.stdout: "max": "", 2024-04-23T10:41:09.539 INFO:teuthology.orchestra.run.smithi114.stdout: "enum_allowed": [], 2024-04-23T10:41:09.540 INFO:teuthology.orchestra.run.smithi114.stdout: "desc": "seconds to cache host facts data", 2024-04-23T10:41:09.540 INFO:teuthology.orchestra.run.smithi114.stdout: "long_desc": "", 2024-04-23T10:41:09.540 INFO:teuthology.orchestra.run.smithi114.stdout: "tags": [], 2024-04-23T10:41:09.540 INFO:teuthology.orchestra.run.smithi114.stdout: "see_also": [] 2024-04-23T10:41:09.540 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-23T10:41:09.540 INFO:teuthology.orchestra.run.smithi114.stdout: "grafana_dashboards_path": { 2024-04-23T10:41:09.540 INFO:teuthology.orchestra.run.smithi114.stdout: "name": "grafana_dashboards_path", 2024-04-23T10:41:09.540 INFO:teuthology.orchestra.run.smithi114.stdout: "type": "str", 2024-04-23T10:41:09.540 INFO:teuthology.orchestra.run.smithi114.stdout: "level": "advanced", 2024-04-23T10:41:09.541 INFO:teuthology.orchestra.run.smithi114.stdout: "flags": 0, 2024-04-23T10:41:09.541 INFO:teuthology.orchestra.run.smithi114.stdout: "default_value": "/etc/grafana/dashboards/ceph-dashboard/", 2024-04-23T10:41:09.541 INFO:teuthology.orchestra.run.smithi114.stdout: "min": "", 2024-04-23T10:41:09.541 INFO:teuthology.orchestra.run.smithi114.stdout: "max": "", 2024-04-23T10:41:09.541 INFO:teuthology.orchestra.run.smithi114.stdout: "enum_allowed": [], 2024-04-23T10:41:09.541 INFO:teuthology.orchestra.run.smithi114.stdout: "desc": "location of dashboards to include in grafana deployments", 2024-04-23T10:41:09.541 INFO:teuthology.orchestra.run.smithi114.stdout: "long_desc": "", 2024-04-23T10:41:09.541 INFO:teuthology.orchestra.run.smithi114.stdout: "tags": [], 2024-04-23T10:41:09.541 INFO:teuthology.orchestra.run.smithi114.stdout: "see_also": [] 2024-04-23T10:41:09.542 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-23T10:41:09.542 INFO:teuthology.orchestra.run.smithi114.stdout: "host_check_interval": { 2024-04-23T10:41:09.542 INFO:teuthology.orchestra.run.smithi114.stdout: "name": "host_check_interval", 2024-04-23T10:41:09.542 INFO:teuthology.orchestra.run.smithi114.stdout: "type": "secs", 2024-04-23T10:41:09.542 INFO:teuthology.orchestra.run.smithi114.stdout: "level": "advanced", 2024-04-23T10:41:09.542 INFO:teuthology.orchestra.run.smithi114.stdout: "flags": 0, 2024-04-23T10:41:09.542 INFO:teuthology.orchestra.run.smithi114.stdout: "default_value": "600", 2024-04-23T10:41:09.542 INFO:teuthology.orchestra.run.smithi114.stdout: "min": "", 2024-04-23T10:41:09.543 INFO:teuthology.orchestra.run.smithi114.stdout: "max": "", 2024-04-23T10:41:09.543 INFO:teuthology.orchestra.run.smithi114.stdout: "enum_allowed": [], 2024-04-23T10:41:09.543 INFO:teuthology.orchestra.run.smithi114.stdout: "desc": "how frequently to perform a host check", 2024-04-23T10:41:09.543 INFO:teuthology.orchestra.run.smithi114.stdout: "long_desc": "", 2024-04-23T10:41:09.543 INFO:teuthology.orchestra.run.smithi114.stdout: "tags": [], 2024-04-23T10:41:09.543 INFO:teuthology.orchestra.run.smithi114.stdout: "see_also": [] 2024-04-23T10:41:09.543 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-23T10:41:09.543 INFO:teuthology.orchestra.run.smithi114.stdout: "hw_monitoring": { 2024-04-23T10:41:09.543 INFO:teuthology.orchestra.run.smithi114.stdout: "name": "hw_monitoring", 2024-04-23T10:41:09.543 INFO:teuthology.orchestra.run.smithi114.stdout: "type": "bool", 2024-04-23T10:41:09.544 INFO:teuthology.orchestra.run.smithi114.stdout: "level": "advanced", 2024-04-23T10:41:09.544 INFO:teuthology.orchestra.run.smithi114.stdout: "flags": 0, 2024-04-23T10:41:09.544 INFO:teuthology.orchestra.run.smithi114.stdout: "default_value": "False", 2024-04-23T10:41:09.544 INFO:teuthology.orchestra.run.smithi114.stdout: "min": "", 2024-04-23T10:41:09.544 INFO:teuthology.orchestra.run.smithi114.stdout: "max": "", 2024-04-23T10:41:09.544 INFO:teuthology.orchestra.run.smithi114.stdout: "enum_allowed": [], 2024-04-23T10:41:09.544 INFO:teuthology.orchestra.run.smithi114.stdout: "desc": "Deploy hw monitoring daemon on every host.", 2024-04-23T10:41:09.544 INFO:teuthology.orchestra.run.smithi114.stdout: "long_desc": "", 2024-04-23T10:41:09.544 INFO:teuthology.orchestra.run.smithi114.stdout: "tags": [], 2024-04-23T10:41:09.545 INFO:teuthology.orchestra.run.smithi114.stdout: "see_also": [] 2024-04-23T10:41:09.545 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-23T10:41:09.545 INFO:teuthology.orchestra.run.smithi114.stdout: "inventory_list_all": { 2024-04-23T10:41:09.545 INFO:teuthology.orchestra.run.smithi114.stdout: "name": "inventory_list_all", 2024-04-23T10:41:09.545 INFO:teuthology.orchestra.run.smithi114.stdout: "type": "bool", 2024-04-23T10:41:09.545 INFO:teuthology.orchestra.run.smithi114.stdout: "level": "advanced", 2024-04-23T10:41:09.545 INFO:teuthology.orchestra.run.smithi114.stdout: "flags": 0, 2024-04-23T10:41:09.545 INFO:teuthology.orchestra.run.smithi114.stdout: "default_value": "False", 2024-04-23T10:41:09.545 INFO:teuthology.orchestra.run.smithi114.stdout: "min": "", 2024-04-23T10:41:09.546 INFO:teuthology.orchestra.run.smithi114.stdout: "max": "", 2024-04-23T10:41:09.546 INFO:teuthology.orchestra.run.smithi114.stdout: "enum_allowed": [], 2024-04-23T10:41:09.546 INFO:teuthology.orchestra.run.smithi114.stdout: "desc": "Whether ceph-volume inventory should report more devices (mostly mappers (LVs / mpaths), partitions...)", 2024-04-23T10:41:09.546 INFO:teuthology.orchestra.run.smithi114.stdout: "long_desc": "", 2024-04-23T10:41:09.546 INFO:teuthology.orchestra.run.smithi114.stdout: "tags": [], 2024-04-23T10:41:09.546 INFO:teuthology.orchestra.run.smithi114.stdout: "see_also": [] 2024-04-23T10:41:09.546 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-23T10:41:09.546 INFO:teuthology.orchestra.run.smithi114.stdout: "log_level": { 2024-04-23T10:41:09.546 INFO:teuthology.orchestra.run.smithi114.stdout: "name": "log_level", 2024-04-23T10:41:09.547 INFO:teuthology.orchestra.run.smithi114.stdout: "type": "str", 2024-04-23T10:41:09.547 INFO:teuthology.orchestra.run.smithi114.stdout: "level": "advanced", 2024-04-23T10:41:09.547 INFO:teuthology.orchestra.run.smithi114.stdout: "flags": 1, 2024-04-23T10:41:09.547 INFO:teuthology.orchestra.run.smithi114.stdout: "default_value": "", 2024-04-23T10:41:09.547 INFO:teuthology.orchestra.run.smithi114.stdout: "min": "", 2024-04-23T10:41:09.547 INFO:teuthology.orchestra.run.smithi114.stdout: "max": "", 2024-04-23T10:41:09.547 INFO:teuthology.orchestra.run.smithi114.stdout: "enum_allowed": [ 2024-04-23T10:41:09.547 INFO:teuthology.orchestra.run.smithi114.stdout: "", 2024-04-23T10:41:09.547 INFO:teuthology.orchestra.run.smithi114.stdout: "critical", 2024-04-23T10:41:09.548 INFO:teuthology.orchestra.run.smithi114.stdout: "debug", 2024-04-23T10:41:09.548 INFO:teuthology.orchestra.run.smithi114.stdout: "error", 2024-04-23T10:41:09.548 INFO:teuthology.orchestra.run.smithi114.stdout: "info", 2024-04-23T10:41:09.548 INFO:teuthology.orchestra.run.smithi114.stdout: "warning" 2024-04-23T10:41:09.548 INFO:teuthology.orchestra.run.smithi114.stdout: ], 2024-04-23T10:41:09.548 INFO:teuthology.orchestra.run.smithi114.stdout: "desc": "", 2024-04-23T10:41:09.548 INFO:teuthology.orchestra.run.smithi114.stdout: "long_desc": "", 2024-04-23T10:41:09.548 INFO:teuthology.orchestra.run.smithi114.stdout: "tags": [], 2024-04-23T10:41:09.548 INFO:teuthology.orchestra.run.smithi114.stdout: "see_also": [] 2024-04-23T10:41:09.549 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-23T10:41:09.549 INFO:teuthology.orchestra.run.smithi114.stdout: "log_refresh_metadata": { 2024-04-23T10:41:09.549 INFO:teuthology.orchestra.run.smithi114.stdout: "name": "log_refresh_metadata", 2024-04-23T10:41:09.549 INFO:teuthology.orchestra.run.smithi114.stdout: "type": "bool", 2024-04-23T10:41:09.549 INFO:teuthology.orchestra.run.smithi114.stdout: "level": "advanced", 2024-04-23T10:41:09.549 INFO:teuthology.orchestra.run.smithi114.stdout: "flags": 0, 2024-04-23T10:41:09.549 INFO:teuthology.orchestra.run.smithi114.stdout: "default_value": "False", 2024-04-23T10:41:09.549 INFO:teuthology.orchestra.run.smithi114.stdout: "min": "", 2024-04-23T10:41:09.549 INFO:teuthology.orchestra.run.smithi114.stdout: "max": "", 2024-04-23T10:41:09.550 INFO:teuthology.orchestra.run.smithi114.stdout: "enum_allowed": [], 2024-04-23T10:41:09.550 INFO:teuthology.orchestra.run.smithi114.stdout: "desc": "Log all refresh metadata. Includes daemon, device, and host info collected regularly. Only has effect if logging at debug level", 2024-04-23T10:41:09.550 INFO:teuthology.orchestra.run.smithi114.stdout: "long_desc": "", 2024-04-23T10:41:09.550 INFO:teuthology.orchestra.run.smithi114.stdout: "tags": [], 2024-04-23T10:41:09.550 INFO:teuthology.orchestra.run.smithi114.stdout: "see_also": [] 2024-04-23T10:41:09.550 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-23T10:41:09.550 INFO:teuthology.orchestra.run.smithi114.stdout: "log_to_cluster": { 2024-04-23T10:41:09.550 INFO:teuthology.orchestra.run.smithi114.stdout: "name": "log_to_cluster", 2024-04-23T10:41:09.550 INFO:teuthology.orchestra.run.smithi114.stdout: "type": "bool", 2024-04-23T10:41:09.551 INFO:teuthology.orchestra.run.smithi114.stdout: "level": "advanced", 2024-04-23T10:41:09.551 INFO:teuthology.orchestra.run.smithi114.stdout: "flags": 0, 2024-04-23T10:41:09.551 INFO:teuthology.orchestra.run.smithi114.stdout: "default_value": "True", 2024-04-23T10:41:09.551 INFO:teuthology.orchestra.run.smithi114.stdout: "min": "", 2024-04-23T10:41:09.551 INFO:teuthology.orchestra.run.smithi114.stdout: "max": "", 2024-04-23T10:41:09.551 INFO:teuthology.orchestra.run.smithi114.stdout: "enum_allowed": [], 2024-04-23T10:41:09.551 INFO:teuthology.orchestra.run.smithi114.stdout: "desc": "log to the \"cephadm\" cluster log channel\"", 2024-04-23T10:41:09.551 INFO:teuthology.orchestra.run.smithi114.stdout: "long_desc": "", 2024-04-23T10:41:09.551 INFO:teuthology.orchestra.run.smithi114.stdout: "tags": [], 2024-04-23T10:41:09.552 INFO:teuthology.orchestra.run.smithi114.stdout: "see_also": [] 2024-04-23T10:41:09.552 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-23T10:41:09.552 INFO:teuthology.orchestra.run.smithi114.stdout: "log_to_cluster_level": { 2024-04-23T10:41:09.552 INFO:teuthology.orchestra.run.smithi114.stdout: "name": "log_to_cluster_level", 2024-04-23T10:41:09.552 INFO:teuthology.orchestra.run.smithi114.stdout: "type": "str", 2024-04-23T10:41:09.552 INFO:teuthology.orchestra.run.smithi114.stdout: "level": "advanced", 2024-04-23T10:41:09.552 INFO:teuthology.orchestra.run.smithi114.stdout: "flags": 1, 2024-04-23T10:41:09.552 INFO:teuthology.orchestra.run.smithi114.stdout: "default_value": "info", 2024-04-23T10:41:09.552 INFO:teuthology.orchestra.run.smithi114.stdout: "min": "", 2024-04-23T10:41:09.552 INFO:teuthology.orchestra.run.smithi114.stdout: "max": "", 2024-04-23T10:41:09.553 INFO:teuthology.orchestra.run.smithi114.stdout: "enum_allowed": [ 2024-04-23T10:41:09.553 INFO:teuthology.orchestra.run.smithi114.stdout: "", 2024-04-23T10:41:09.553 INFO:teuthology.orchestra.run.smithi114.stdout: "critical", 2024-04-23T10:41:09.553 INFO:teuthology.orchestra.run.smithi114.stdout: "debug", 2024-04-23T10:41:09.553 INFO:teuthology.orchestra.run.smithi114.stdout: "error", 2024-04-23T10:41:09.553 INFO:teuthology.orchestra.run.smithi114.stdout: "info", 2024-04-23T10:41:09.553 INFO:teuthology.orchestra.run.smithi114.stdout: "warning" 2024-04-23T10:41:09.553 INFO:teuthology.orchestra.run.smithi114.stdout: ], 2024-04-23T10:41:09.553 INFO:teuthology.orchestra.run.smithi114.stdout: "desc": "", 2024-04-23T10:41:09.554 INFO:teuthology.orchestra.run.smithi114.stdout: "long_desc": "", 2024-04-23T10:41:09.554 INFO:teuthology.orchestra.run.smithi114.stdout: "tags": [], 2024-04-23T10:41:09.554 INFO:teuthology.orchestra.run.smithi114.stdout: "see_also": [] 2024-04-23T10:41:09.554 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-23T10:41:09.554 INFO:teuthology.orchestra.run.smithi114.stdout: "log_to_file": { 2024-04-23T10:41:09.554 INFO:teuthology.orchestra.run.smithi114.stdout: "name": "log_to_file", 2024-04-23T10:41:09.554 INFO:teuthology.orchestra.run.smithi114.stdout: "type": "bool", 2024-04-23T10:41:09.554 INFO:teuthology.orchestra.run.smithi114.stdout: "level": "advanced", 2024-04-23T10:41:09.554 INFO:teuthology.orchestra.run.smithi114.stdout: "flags": 1, 2024-04-23T10:41:09.555 INFO:teuthology.orchestra.run.smithi114.stdout: "default_value": "False", 2024-04-23T10:41:09.555 INFO:teuthology.orchestra.run.smithi114.stdout: "min": "", 2024-04-23T10:41:09.555 INFO:teuthology.orchestra.run.smithi114.stdout: "max": "", 2024-04-23T10:41:09.555 INFO:teuthology.orchestra.run.smithi114.stdout: "enum_allowed": [], 2024-04-23T10:41:09.555 INFO:teuthology.orchestra.run.smithi114.stdout: "desc": "", 2024-04-23T10:41:09.555 INFO:teuthology.orchestra.run.smithi114.stdout: "long_desc": "", 2024-04-23T10:41:09.555 INFO:teuthology.orchestra.run.smithi114.stdout: "tags": [], 2024-04-23T10:41:09.555 INFO:teuthology.orchestra.run.smithi114.stdout: "see_also": [] 2024-04-23T10:41:09.555 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-23T10:41:09.556 INFO:teuthology.orchestra.run.smithi114.stdout: "manage_etc_ceph_ceph_conf": { 2024-04-23T10:41:09.556 INFO:teuthology.orchestra.run.smithi114.stdout: "name": "manage_etc_ceph_ceph_conf", 2024-04-23T10:41:09.556 INFO:teuthology.orchestra.run.smithi114.stdout: "type": "bool", 2024-04-23T10:41:09.556 INFO:teuthology.orchestra.run.smithi114.stdout: "level": "advanced", 2024-04-23T10:41:09.556 INFO:teuthology.orchestra.run.smithi114.stdout: "flags": 0, 2024-04-23T10:41:09.556 INFO:teuthology.orchestra.run.smithi114.stdout: "default_value": "False", 2024-04-23T10:41:09.556 INFO:teuthology.orchestra.run.smithi114.stdout: "min": "", 2024-04-23T10:41:09.556 INFO:teuthology.orchestra.run.smithi114.stdout: "max": "", 2024-04-23T10:41:09.556 INFO:teuthology.orchestra.run.smithi114.stdout: "enum_allowed": [], 2024-04-23T10:41:09.557 INFO:teuthology.orchestra.run.smithi114.stdout: "desc": "Manage and own /etc/ceph/ceph.conf on the hosts.", 2024-04-23T10:41:09.557 INFO:teuthology.orchestra.run.smithi114.stdout: "long_desc": "", 2024-04-23T10:41:09.557 INFO:teuthology.orchestra.run.smithi114.stdout: "tags": [], 2024-04-23T10:41:09.557 INFO:teuthology.orchestra.run.smithi114.stdout: "see_also": [] 2024-04-23T10:41:09.557 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-23T10:41:09.557 INFO:teuthology.orchestra.run.smithi114.stdout: "manage_etc_ceph_ceph_conf_hosts": { 2024-04-23T10:41:09.557 INFO:teuthology.orchestra.run.smithi114.stdout: "name": "manage_etc_ceph_ceph_conf_hosts", 2024-04-23T10:41:09.557 INFO:teuthology.orchestra.run.smithi114.stdout: "type": "str", 2024-04-23T10:41:09.557 INFO:teuthology.orchestra.run.smithi114.stdout: "level": "advanced", 2024-04-23T10:41:09.558 INFO:teuthology.orchestra.run.smithi114.stdout: "flags": 0, 2024-04-23T10:41:09.558 INFO:teuthology.orchestra.run.smithi114.stdout: "default_value": "*", 2024-04-23T10:41:09.558 INFO:teuthology.orchestra.run.smithi114.stdout: "min": "", 2024-04-23T10:41:09.558 INFO:teuthology.orchestra.run.smithi114.stdout: "max": "", 2024-04-23T10:41:09.558 INFO:teuthology.orchestra.run.smithi114.stdout: "enum_allowed": [], 2024-04-23T10:41:09.558 INFO:teuthology.orchestra.run.smithi114.stdout: "desc": "PlacementSpec describing on which hosts to manage /etc/ceph/ceph.conf", 2024-04-23T10:41:09.558 INFO:teuthology.orchestra.run.smithi114.stdout: "long_desc": "", 2024-04-23T10:41:09.558 INFO:teuthology.orchestra.run.smithi114.stdout: "tags": [], 2024-04-23T10:41:09.558 INFO:teuthology.orchestra.run.smithi114.stdout: "see_also": [] 2024-04-23T10:41:09.558 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-23T10:41:09.559 INFO:teuthology.orchestra.run.smithi114.stdout: "max_count_per_host": { 2024-04-23T10:41:09.559 INFO:teuthology.orchestra.run.smithi114.stdout: "name": "max_count_per_host", 2024-04-23T10:41:09.559 INFO:teuthology.orchestra.run.smithi114.stdout: "type": "int", 2024-04-23T10:41:09.559 INFO:teuthology.orchestra.run.smithi114.stdout: "level": "advanced", 2024-04-23T10:41:09.559 INFO:teuthology.orchestra.run.smithi114.stdout: "flags": 0, 2024-04-23T10:41:09.559 INFO:teuthology.orchestra.run.smithi114.stdout: "default_value": "10", 2024-04-23T10:41:09.559 INFO:teuthology.orchestra.run.smithi114.stdout: "min": "", 2024-04-23T10:41:09.559 INFO:teuthology.orchestra.run.smithi114.stdout: "max": "", 2024-04-23T10:41:09.559 INFO:teuthology.orchestra.run.smithi114.stdout: "enum_allowed": [], 2024-04-23T10:41:09.560 INFO:teuthology.orchestra.run.smithi114.stdout: "desc": "max number of daemons per service per host", 2024-04-23T10:41:09.560 INFO:teuthology.orchestra.run.smithi114.stdout: "long_desc": "", 2024-04-23T10:41:09.560 INFO:teuthology.orchestra.run.smithi114.stdout: "tags": [], 2024-04-23T10:41:09.560 INFO:teuthology.orchestra.run.smithi114.stdout: "see_also": [] 2024-04-23T10:41:09.560 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-23T10:41:09.560 INFO:teuthology.orchestra.run.smithi114.stdout: "max_osd_draining_count": { 2024-04-23T10:41:09.560 INFO:teuthology.orchestra.run.smithi114.stdout: "name": "max_osd_draining_count", 2024-04-23T10:41:09.560 INFO:teuthology.orchestra.run.smithi114.stdout: "type": "int", 2024-04-23T10:41:09.560 INFO:teuthology.orchestra.run.smithi114.stdout: "level": "advanced", 2024-04-23T10:41:09.560 INFO:teuthology.orchestra.run.smithi114.stdout: "flags": 0, 2024-04-23T10:41:09.561 INFO:teuthology.orchestra.run.smithi114.stdout: "default_value": "10", 2024-04-23T10:41:09.561 INFO:teuthology.orchestra.run.smithi114.stdout: "min": "", 2024-04-23T10:41:09.561 INFO:teuthology.orchestra.run.smithi114.stdout: "max": "", 2024-04-23T10:41:09.561 INFO:teuthology.orchestra.run.smithi114.stdout: "enum_allowed": [], 2024-04-23T10:41:09.561 INFO:teuthology.orchestra.run.smithi114.stdout: "desc": "max number of osds that will be drained simultaneously when osds are removed", 2024-04-23T10:41:09.561 INFO:teuthology.orchestra.run.smithi114.stdout: "long_desc": "", 2024-04-23T10:41:09.561 INFO:teuthology.orchestra.run.smithi114.stdout: "tags": [], 2024-04-23T10:41:09.561 INFO:teuthology.orchestra.run.smithi114.stdout: "see_also": [] 2024-04-23T10:41:09.561 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-23T10:41:09.561 INFO:teuthology.orchestra.run.smithi114.stdout: "migration_current": { 2024-04-23T10:41:09.562 INFO:teuthology.orchestra.run.smithi114.stdout: "name": "migration_current", 2024-04-23T10:41:09.562 INFO:teuthology.orchestra.run.smithi114.stdout: "type": "int", 2024-04-23T10:41:09.562 INFO:teuthology.orchestra.run.smithi114.stdout: "level": "advanced", 2024-04-23T10:41:09.562 INFO:teuthology.orchestra.run.smithi114.stdout: "flags": 0, 2024-04-23T10:41:09.562 INFO:teuthology.orchestra.run.smithi114.stdout: "default_value": "", 2024-04-23T10:41:09.562 INFO:teuthology.orchestra.run.smithi114.stdout: "min": "", 2024-04-23T10:41:09.562 INFO:teuthology.orchestra.run.smithi114.stdout: "max": "", 2024-04-23T10:41:09.562 INFO:teuthology.orchestra.run.smithi114.stdout: "enum_allowed": [], 2024-04-23T10:41:09.563 INFO:teuthology.orchestra.run.smithi114.stdout: "desc": "internal - do not modify", 2024-04-23T10:41:09.563 INFO:teuthology.orchestra.run.smithi114.stdout: "long_desc": "", 2024-04-23T10:41:09.563 INFO:teuthology.orchestra.run.smithi114.stdout: "tags": [], 2024-04-23T10:41:09.563 INFO:teuthology.orchestra.run.smithi114.stdout: "see_also": [] 2024-04-23T10:41:09.563 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-23T10:41:09.563 INFO:teuthology.orchestra.run.smithi114.stdout: "mode": { 2024-04-23T10:41:09.563 INFO:teuthology.orchestra.run.smithi114.stdout: "name": "mode", 2024-04-23T10:41:09.563 INFO:teuthology.orchestra.run.smithi114.stdout: "type": "str", 2024-04-23T10:41:09.563 INFO:teuthology.orchestra.run.smithi114.stdout: "level": "advanced", 2024-04-23T10:41:09.564 INFO:teuthology.orchestra.run.smithi114.stdout: "flags": 0, 2024-04-23T10:41:09.564 INFO:teuthology.orchestra.run.smithi114.stdout: "default_value": "root", 2024-04-23T10:41:09.564 INFO:teuthology.orchestra.run.smithi114.stdout: "min": "", 2024-04-23T10:41:09.564 INFO:teuthology.orchestra.run.smithi114.stdout: "max": "", 2024-04-23T10:41:09.564 INFO:teuthology.orchestra.run.smithi114.stdout: "enum_allowed": [ 2024-04-23T10:41:09.564 INFO:teuthology.orchestra.run.smithi114.stdout: "cephadm-package", 2024-04-23T10:41:09.564 INFO:teuthology.orchestra.run.smithi114.stdout: "root" 2024-04-23T10:41:09.564 INFO:teuthology.orchestra.run.smithi114.stdout: ], 2024-04-23T10:41:09.564 INFO:teuthology.orchestra.run.smithi114.stdout: "desc": "mode for remote execution of cephadm", 2024-04-23T10:41:09.564 INFO:teuthology.orchestra.run.smithi114.stdout: "long_desc": "", 2024-04-23T10:41:09.565 INFO:teuthology.orchestra.run.smithi114.stdout: "tags": [], 2024-04-23T10:41:09.565 INFO:teuthology.orchestra.run.smithi114.stdout: "see_also": [] 2024-04-23T10:41:09.565 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-23T10:41:09.565 INFO:teuthology.orchestra.run.smithi114.stdout: "oob_default_addr": { 2024-04-23T10:41:09.565 INFO:teuthology.orchestra.run.smithi114.stdout: "name": "oob_default_addr", 2024-04-23T10:41:09.565 INFO:teuthology.orchestra.run.smithi114.stdout: "type": "str", 2024-04-23T10:41:09.565 INFO:teuthology.orchestra.run.smithi114.stdout: "level": "advanced", 2024-04-23T10:41:09.565 INFO:teuthology.orchestra.run.smithi114.stdout: "flags": 0, 2024-04-23T10:41:09.565 INFO:teuthology.orchestra.run.smithi114.stdout: "default_value": "169.254.1.1", 2024-04-23T10:41:09.566 INFO:teuthology.orchestra.run.smithi114.stdout: "min": "", 2024-04-23T10:41:09.566 INFO:teuthology.orchestra.run.smithi114.stdout: "max": "", 2024-04-23T10:41:09.566 INFO:teuthology.orchestra.run.smithi114.stdout: "enum_allowed": [], 2024-04-23T10:41:09.566 INFO:teuthology.orchestra.run.smithi114.stdout: "desc": "Default address for RedFish API (oob management).", 2024-04-23T10:41:09.566 INFO:teuthology.orchestra.run.smithi114.stdout: "long_desc": "", 2024-04-23T10:41:09.566 INFO:teuthology.orchestra.run.smithi114.stdout: "tags": [], 2024-04-23T10:41:09.566 INFO:teuthology.orchestra.run.smithi114.stdout: "see_also": [] 2024-04-23T10:41:09.566 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-23T10:41:09.566 INFO:teuthology.orchestra.run.smithi114.stdout: "prometheus_alerts_path": { 2024-04-23T10:41:09.567 INFO:teuthology.orchestra.run.smithi114.stdout: "name": "prometheus_alerts_path", 2024-04-23T10:41:09.567 INFO:teuthology.orchestra.run.smithi114.stdout: "type": "str", 2024-04-23T10:41:09.567 INFO:teuthology.orchestra.run.smithi114.stdout: "level": "advanced", 2024-04-23T10:41:09.567 INFO:teuthology.orchestra.run.smithi114.stdout: "flags": 0, 2024-04-23T10:41:09.567 INFO:teuthology.orchestra.run.smithi114.stdout: "default_value": "/etc/prometheus/ceph/ceph_default_alerts.yml", 2024-04-23T10:41:09.567 INFO:teuthology.orchestra.run.smithi114.stdout: "min": "", 2024-04-23T10:41:09.567 INFO:teuthology.orchestra.run.smithi114.stdout: "max": "", 2024-04-23T10:41:09.567 INFO:teuthology.orchestra.run.smithi114.stdout: "enum_allowed": [], 2024-04-23T10:41:09.567 INFO:teuthology.orchestra.run.smithi114.stdout: "desc": "location of alerts to include in prometheus deployments", 2024-04-23T10:41:09.568 INFO:teuthology.orchestra.run.smithi114.stdout: "long_desc": "", 2024-04-23T10:41:09.568 INFO:teuthology.orchestra.run.smithi114.stdout: "tags": [], 2024-04-23T10:41:09.568 INFO:teuthology.orchestra.run.smithi114.stdout: "see_also": [] 2024-04-23T10:41:09.568 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-23T10:41:09.568 INFO:teuthology.orchestra.run.smithi114.stdout: "registry_insecure": { 2024-04-23T10:41:09.568 INFO:teuthology.orchestra.run.smithi114.stdout: "name": "registry_insecure", 2024-04-23T10:41:09.568 INFO:teuthology.orchestra.run.smithi114.stdout: "type": "bool", 2024-04-23T10:41:09.568 INFO:teuthology.orchestra.run.smithi114.stdout: "level": "advanced", 2024-04-23T10:41:09.569 INFO:teuthology.orchestra.run.smithi114.stdout: "flags": 0, 2024-04-23T10:41:09.569 INFO:teuthology.orchestra.run.smithi114.stdout: "default_value": "False", 2024-04-23T10:41:09.569 INFO:teuthology.orchestra.run.smithi114.stdout: "min": "", 2024-04-23T10:41:09.569 INFO:teuthology.orchestra.run.smithi114.stdout: "max": "", 2024-04-23T10:41:09.569 INFO:teuthology.orchestra.run.smithi114.stdout: "enum_allowed": [], 2024-04-23T10:41:09.569 INFO:teuthology.orchestra.run.smithi114.stdout: "desc": "Registry is to be considered insecure (no TLS available). Only for development purposes.", 2024-04-23T10:41:09.569 INFO:teuthology.orchestra.run.smithi114.stdout: "long_desc": "", 2024-04-23T10:41:09.569 INFO:teuthology.orchestra.run.smithi114.stdout: "tags": [], 2024-04-23T10:41:09.569 INFO:teuthology.orchestra.run.smithi114.stdout: "see_also": [] 2024-04-23T10:41:09.569 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-23T10:41:09.570 INFO:teuthology.orchestra.run.smithi114.stdout: "registry_password": { 2024-04-23T10:41:09.570 INFO:teuthology.orchestra.run.smithi114.stdout: "name": "registry_password", 2024-04-23T10:41:09.570 INFO:teuthology.orchestra.run.smithi114.stdout: "type": "str", 2024-04-23T10:41:09.570 INFO:teuthology.orchestra.run.smithi114.stdout: "level": "advanced", 2024-04-23T10:41:09.570 INFO:teuthology.orchestra.run.smithi114.stdout: "flags": 0, 2024-04-23T10:41:09.570 INFO:teuthology.orchestra.run.smithi114.stdout: "default_value": "", 2024-04-23T10:41:09.570 INFO:teuthology.orchestra.run.smithi114.stdout: "min": "", 2024-04-23T10:41:09.570 INFO:teuthology.orchestra.run.smithi114.stdout: "max": "", 2024-04-23T10:41:09.570 INFO:teuthology.orchestra.run.smithi114.stdout: "enum_allowed": [], 2024-04-23T10:41:09.570 INFO:teuthology.orchestra.run.smithi114.stdout: "desc": "Custom repository password. Only used for logging into a registry.", 2024-04-23T10:41:09.571 INFO:teuthology.orchestra.run.smithi114.stdout: "long_desc": "", 2024-04-23T10:41:09.571 INFO:teuthology.orchestra.run.smithi114.stdout: "tags": [], 2024-04-23T10:41:09.571 INFO:teuthology.orchestra.run.smithi114.stdout: "see_also": [] 2024-04-23T10:41:09.571 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-23T10:41:09.571 INFO:teuthology.orchestra.run.smithi114.stdout: "registry_url": { 2024-04-23T10:41:09.571 INFO:teuthology.orchestra.run.smithi114.stdout: "name": "registry_url", 2024-04-23T10:41:09.571 INFO:teuthology.orchestra.run.smithi114.stdout: "type": "str", 2024-04-23T10:41:09.571 INFO:teuthology.orchestra.run.smithi114.stdout: "level": "advanced", 2024-04-23T10:41:09.571 INFO:teuthology.orchestra.run.smithi114.stdout: "flags": 0, 2024-04-23T10:41:09.572 INFO:teuthology.orchestra.run.smithi114.stdout: "default_value": "", 2024-04-23T10:41:09.572 INFO:teuthology.orchestra.run.smithi114.stdout: "min": "", 2024-04-23T10:41:09.572 INFO:teuthology.orchestra.run.smithi114.stdout: "max": "", 2024-04-23T10:41:09.572 INFO:teuthology.orchestra.run.smithi114.stdout: "enum_allowed": [], 2024-04-23T10:41:09.572 INFO:teuthology.orchestra.run.smithi114.stdout: "desc": "Registry url for login purposes. This is not the default registry", 2024-04-23T10:41:09.572 INFO:teuthology.orchestra.run.smithi114.stdout: "long_desc": "", 2024-04-23T10:41:09.572 INFO:teuthology.orchestra.run.smithi114.stdout: "tags": [], 2024-04-23T10:41:09.572 INFO:teuthology.orchestra.run.smithi114.stdout: "see_also": [] 2024-04-23T10:41:09.572 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-23T10:41:09.572 INFO:teuthology.orchestra.run.smithi114.stdout: "registry_username": { 2024-04-23T10:41:09.573 INFO:teuthology.orchestra.run.smithi114.stdout: "name": "registry_username", 2024-04-23T10:41:09.573 INFO:teuthology.orchestra.run.smithi114.stdout: "type": "str", 2024-04-23T10:41:09.573 INFO:teuthology.orchestra.run.smithi114.stdout: "level": "advanced", 2024-04-23T10:41:09.573 INFO:teuthology.orchestra.run.smithi114.stdout: "flags": 0, 2024-04-23T10:41:09.573 INFO:teuthology.orchestra.run.smithi114.stdout: "default_value": "", 2024-04-23T10:41:09.573 INFO:teuthology.orchestra.run.smithi114.stdout: "min": "", 2024-04-23T10:41:09.573 INFO:teuthology.orchestra.run.smithi114.stdout: "max": "", 2024-04-23T10:41:09.573 INFO:teuthology.orchestra.run.smithi114.stdout: "enum_allowed": [], 2024-04-23T10:41:09.574 INFO:teuthology.orchestra.run.smithi114.stdout: "desc": "Custom repository username. Only used for logging into a registry.", 2024-04-23T10:41:09.574 INFO:teuthology.orchestra.run.smithi114.stdout: "long_desc": "", 2024-04-23T10:41:09.574 INFO:teuthology.orchestra.run.smithi114.stdout: "tags": [], 2024-04-23T10:41:09.574 INFO:teuthology.orchestra.run.smithi114.stdout: "see_also": [] 2024-04-23T10:41:09.574 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-23T10:41:09.574 INFO:teuthology.orchestra.run.smithi114.stdout: "secure_monitoring_stack": { 2024-04-23T10:41:09.574 INFO:teuthology.orchestra.run.smithi114.stdout: "name": "secure_monitoring_stack", 2024-04-23T10:41:09.574 INFO:teuthology.orchestra.run.smithi114.stdout: "type": "bool", 2024-04-23T10:41:09.574 INFO:teuthology.orchestra.run.smithi114.stdout: "level": "advanced", 2024-04-23T10:41:09.575 INFO:teuthology.orchestra.run.smithi114.stdout: "flags": 0, 2024-04-23T10:41:09.575 INFO:teuthology.orchestra.run.smithi114.stdout: "default_value": "False", 2024-04-23T10:41:09.575 INFO:teuthology.orchestra.run.smithi114.stdout: "min": "", 2024-04-23T10:41:09.575 INFO:teuthology.orchestra.run.smithi114.stdout: "max": "", 2024-04-23T10:41:09.575 INFO:teuthology.orchestra.run.smithi114.stdout: "enum_allowed": [], 2024-04-23T10:41:09.575 INFO:teuthology.orchestra.run.smithi114.stdout: "desc": "Enable TLS security for all the monitoring stack daemons", 2024-04-23T10:41:09.575 INFO:teuthology.orchestra.run.smithi114.stdout: "long_desc": "", 2024-04-23T10:41:09.575 INFO:teuthology.orchestra.run.smithi114.stdout: "tags": [], 2024-04-23T10:41:09.575 INFO:teuthology.orchestra.run.smithi114.stdout: "see_also": [] 2024-04-23T10:41:09.576 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-23T10:41:09.576 INFO:teuthology.orchestra.run.smithi114.stdout: "service_discovery_port": { 2024-04-23T10:41:09.576 INFO:teuthology.orchestra.run.smithi114.stdout: "name": "service_discovery_port", 2024-04-23T10:41:09.576 INFO:teuthology.orchestra.run.smithi114.stdout: "type": "int", 2024-04-23T10:41:09.576 INFO:teuthology.orchestra.run.smithi114.stdout: "level": "advanced", 2024-04-23T10:41:09.576 INFO:teuthology.orchestra.run.smithi114.stdout: "flags": 0, 2024-04-23T10:41:09.576 INFO:teuthology.orchestra.run.smithi114.stdout: "default_value": "8765", 2024-04-23T10:41:09.576 INFO:teuthology.orchestra.run.smithi114.stdout: "min": "", 2024-04-23T10:41:09.576 INFO:teuthology.orchestra.run.smithi114.stdout: "max": "", 2024-04-23T10:41:09.576 INFO:teuthology.orchestra.run.smithi114.stdout: "enum_allowed": [], 2024-04-23T10:41:09.577 INFO:teuthology.orchestra.run.smithi114.stdout: "desc": "cephadm service discovery port", 2024-04-23T10:41:09.577 INFO:teuthology.orchestra.run.smithi114.stdout: "long_desc": "", 2024-04-23T10:41:09.577 INFO:teuthology.orchestra.run.smithi114.stdout: "tags": [], 2024-04-23T10:41:09.577 INFO:teuthology.orchestra.run.smithi114.stdout: "see_also": [] 2024-04-23T10:41:09.577 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-23T10:41:09.577 INFO:teuthology.orchestra.run.smithi114.stdout: "ssh_config_file": { 2024-04-23T10:41:09.577 INFO:teuthology.orchestra.run.smithi114.stdout: "name": "ssh_config_file", 2024-04-23T10:41:09.577 INFO:teuthology.orchestra.run.smithi114.stdout: "type": "str", 2024-04-23T10:41:09.577 INFO:teuthology.orchestra.run.smithi114.stdout: "level": "advanced", 2024-04-23T10:41:09.578 INFO:teuthology.orchestra.run.smithi114.stdout: "flags": 0, 2024-04-23T10:41:09.578 INFO:teuthology.orchestra.run.smithi114.stdout: "default_value": "", 2024-04-23T10:41:09.578 INFO:teuthology.orchestra.run.smithi114.stdout: "min": "", 2024-04-23T10:41:09.578 INFO:teuthology.orchestra.run.smithi114.stdout: "max": "", 2024-04-23T10:41:09.578 INFO:teuthology.orchestra.run.smithi114.stdout: "enum_allowed": [], 2024-04-23T10:41:09.578 INFO:teuthology.orchestra.run.smithi114.stdout: "desc": "customized SSH config file to connect to managed hosts", 2024-04-23T10:41:09.578 INFO:teuthology.orchestra.run.smithi114.stdout: "long_desc": "", 2024-04-23T10:41:09.578 INFO:teuthology.orchestra.run.smithi114.stdout: "tags": [], 2024-04-23T10:41:09.578 INFO:teuthology.orchestra.run.smithi114.stdout: "see_also": [] 2024-04-23T10:41:09.579 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-23T10:41:09.579 INFO:teuthology.orchestra.run.smithi114.stdout: "use_agent": { 2024-04-23T10:41:09.579 INFO:teuthology.orchestra.run.smithi114.stdout: "name": "use_agent", 2024-04-23T10:41:09.579 INFO:teuthology.orchestra.run.smithi114.stdout: "type": "bool", 2024-04-23T10:41:09.579 INFO:teuthology.orchestra.run.smithi114.stdout: "level": "advanced", 2024-04-23T10:41:09.579 INFO:teuthology.orchestra.run.smithi114.stdout: "flags": 0, 2024-04-23T10:41:09.579 INFO:teuthology.orchestra.run.smithi114.stdout: "default_value": "False", 2024-04-23T10:41:09.579 INFO:teuthology.orchestra.run.smithi114.stdout: "min": "", 2024-04-23T10:41:09.579 INFO:teuthology.orchestra.run.smithi114.stdout: "max": "", 2024-04-23T10:41:09.580 INFO:teuthology.orchestra.run.smithi114.stdout: "enum_allowed": [], 2024-04-23T10:41:09.580 INFO:teuthology.orchestra.run.smithi114.stdout: "desc": "Use cephadm agent on each host to gather and send metadata", 2024-04-23T10:41:09.580 INFO:teuthology.orchestra.run.smithi114.stdout: "long_desc": "", 2024-04-23T10:41:09.580 INFO:teuthology.orchestra.run.smithi114.stdout: "tags": [], 2024-04-23T10:41:09.580 INFO:teuthology.orchestra.run.smithi114.stdout: "see_also": [] 2024-04-23T10:41:09.580 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-23T10:41:09.580 INFO:teuthology.orchestra.run.smithi114.stdout: "use_repo_digest": { 2024-04-23T10:41:09.580 INFO:teuthology.orchestra.run.smithi114.stdout: "name": "use_repo_digest", 2024-04-23T10:41:09.580 INFO:teuthology.orchestra.run.smithi114.stdout: "type": "bool", 2024-04-23T10:41:09.580 INFO:teuthology.orchestra.run.smithi114.stdout: "level": "advanced", 2024-04-23T10:41:09.581 INFO:teuthology.orchestra.run.smithi114.stdout: "flags": 0, 2024-04-23T10:41:09.581 INFO:teuthology.orchestra.run.smithi114.stdout: "default_value": "True", 2024-04-23T10:41:09.581 INFO:teuthology.orchestra.run.smithi114.stdout: "min": "", 2024-04-23T10:41:09.581 INFO:teuthology.orchestra.run.smithi114.stdout: "max": "", 2024-04-23T10:41:09.581 INFO:teuthology.orchestra.run.smithi114.stdout: "enum_allowed": [], 2024-04-23T10:41:09.581 INFO:teuthology.orchestra.run.smithi114.stdout: "desc": "Automatically convert image tags to image digest. Make sure all daemons use the same image", 2024-04-23T10:41:09.581 INFO:teuthology.orchestra.run.smithi114.stdout: "long_desc": "", 2024-04-23T10:41:09.581 INFO:teuthology.orchestra.run.smithi114.stdout: "tags": [], 2024-04-23T10:41:09.581 INFO:teuthology.orchestra.run.smithi114.stdout: "see_also": [] 2024-04-23T10:41:09.582 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-23T10:41:09.582 INFO:teuthology.orchestra.run.smithi114.stdout: "warn_on_failed_host_check": { 2024-04-23T10:41:09.582 INFO:teuthology.orchestra.run.smithi114.stdout: "name": "warn_on_failed_host_check", 2024-04-23T10:41:09.582 INFO:teuthology.orchestra.run.smithi114.stdout: "type": "bool", 2024-04-23T10:41:09.582 INFO:teuthology.orchestra.run.smithi114.stdout: "level": "advanced", 2024-04-23T10:41:09.582 INFO:teuthology.orchestra.run.smithi114.stdout: "flags": 0, 2024-04-23T10:41:09.582 INFO:teuthology.orchestra.run.smithi114.stdout: "default_value": "True", 2024-04-23T10:41:09.582 INFO:teuthology.orchestra.run.smithi114.stdout: "min": "", 2024-04-23T10:41:09.582 INFO:teuthology.orchestra.run.smithi114.stdout: "max": "", 2024-04-23T10:41:09.583 INFO:teuthology.orchestra.run.smithi114.stdout: "enum_allowed": [], 2024-04-23T10:41:09.583 INFO:teuthology.orchestra.run.smithi114.stdout: "desc": "raise a health warning if the host check fails", 2024-04-23T10:41:09.583 INFO:teuthology.orchestra.run.smithi114.stdout: "long_desc": "", 2024-04-23T10:41:09.583 INFO:teuthology.orchestra.run.smithi114.stdout: "tags": [], 2024-04-23T10:41:09.583 INFO:teuthology.orchestra.run.smithi114.stdout: "see_also": [] 2024-04-23T10:41:09.583 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-23T10:41:09.583 INFO:teuthology.orchestra.run.smithi114.stdout: "warn_on_stray_daemons": { 2024-04-23T10:41:09.583 INFO:teuthology.orchestra.run.smithi114.stdout: "name": "warn_on_stray_daemons", 2024-04-23T10:41:09.583 INFO:teuthology.orchestra.run.smithi114.stdout: "type": "bool", 2024-04-23T10:41:09.584 INFO:teuthology.orchestra.run.smithi114.stdout: "level": "advanced", 2024-04-23T10:41:09.584 INFO:teuthology.orchestra.run.smithi114.stdout: "flags": 0, 2024-04-23T10:41:09.584 INFO:teuthology.orchestra.run.smithi114.stdout: "default_value": "True", 2024-04-23T10:41:09.584 INFO:teuthology.orchestra.run.smithi114.stdout: "min": "", 2024-04-23T10:41:09.584 INFO:teuthology.orchestra.run.smithi114.stdout: "max": "", 2024-04-23T10:41:09.584 INFO:teuthology.orchestra.run.smithi114.stdout: "enum_allowed": [], 2024-04-23T10:41:09.584 INFO:teuthology.orchestra.run.smithi114.stdout: "desc": "raise a health warning if daemons are detected that are not managed by cephadm", 2024-04-23T10:41:09.584 INFO:teuthology.orchestra.run.smithi114.stdout: "long_desc": "", 2024-04-23T10:41:09.584 INFO:teuthology.orchestra.run.smithi114.stdout: "tags": [], 2024-04-23T10:41:09.584 INFO:teuthology.orchestra.run.smithi114.stdout: "see_also": [] 2024-04-23T10:41:09.585 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-23T10:41:09.585 INFO:teuthology.orchestra.run.smithi114.stdout: "warn_on_stray_hosts": { 2024-04-23T10:41:09.585 INFO:teuthology.orchestra.run.smithi114.stdout: "name": "warn_on_stray_hosts", 2024-04-23T10:41:09.585 INFO:teuthology.orchestra.run.smithi114.stdout: "type": "bool", 2024-04-23T10:41:09.585 INFO:teuthology.orchestra.run.smithi114.stdout: "level": "advanced", 2024-04-23T10:41:09.585 INFO:teuthology.orchestra.run.smithi114.stdout: "flags": 0, 2024-04-23T10:41:09.585 INFO:teuthology.orchestra.run.smithi114.stdout: "default_value": "True", 2024-04-23T10:41:09.585 INFO:teuthology.orchestra.run.smithi114.stdout: "min": "", 2024-04-23T10:41:09.585 INFO:teuthology.orchestra.run.smithi114.stdout: "max": "", 2024-04-23T10:41:09.585 INFO:teuthology.orchestra.run.smithi114.stdout: "enum_allowed": [], 2024-04-23T10:41:09.586 INFO:teuthology.orchestra.run.smithi114.stdout: "desc": "raise a health warning if daemons are detected on a host that is not managed by cephadm", 2024-04-23T10:41:09.586 INFO:teuthology.orchestra.run.smithi114.stdout: "long_desc": "", 2024-04-23T10:41:09.586 INFO:teuthology.orchestra.run.smithi114.stdout: "tags": [], 2024-04-23T10:41:09.586 INFO:teuthology.orchestra.run.smithi114.stdout: "see_also": [] 2024-04-23T10:41:09.586 INFO:teuthology.orchestra.run.smithi114.stdout: } 2024-04-23T10:41:09.586 INFO:teuthology.orchestra.run.smithi114.stdout: } 2024-04-23T10:41:09.586 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-23T10:41:09.586 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-04-23T10:41:09.586 INFO:teuthology.orchestra.run.smithi114.stdout: "name": "crash", 2024-04-23T10:41:09.587 INFO:teuthology.orchestra.run.smithi114.stdout: "can_run": true, 2024-04-23T10:41:09.587 INFO:teuthology.orchestra.run.smithi114.stdout: "error_string": "", 2024-04-23T10:41:09.587 INFO:teuthology.orchestra.run.smithi114.stdout: "module_options": { 2024-04-23T10:41:09.587 INFO:teuthology.orchestra.run.smithi114.stdout: "log_level": { 2024-04-23T10:41:09.587 INFO:teuthology.orchestra.run.smithi114.stdout: "name": "log_level", 2024-04-23T10:41:09.587 INFO:teuthology.orchestra.run.smithi114.stdout: "type": "str", 2024-04-23T10:41:09.587 INFO:teuthology.orchestra.run.smithi114.stdout: "level": "advanced", 2024-04-23T10:41:09.587 INFO:teuthology.orchestra.run.smithi114.stdout: "flags": 1, 2024-04-23T10:41:09.587 INFO:teuthology.orchestra.run.smithi114.stdout: "default_value": "", 2024-04-23T10:41:09.588 INFO:teuthology.orchestra.run.smithi114.stdout: "min": "", 2024-04-23T10:41:09.588 INFO:teuthology.orchestra.run.smithi114.stdout: "max": "", 2024-04-23T10:41:09.588 INFO:teuthology.orchestra.run.smithi114.stdout: "enum_allowed": [ 2024-04-23T10:41:09.588 INFO:teuthology.orchestra.run.smithi114.stdout: "", 2024-04-23T10:41:09.588 INFO:teuthology.orchestra.run.smithi114.stdout: "critical", 2024-04-23T10:41:09.588 INFO:teuthology.orchestra.run.smithi114.stdout: "debug", 2024-04-23T10:41:09.588 INFO:teuthology.orchestra.run.smithi114.stdout: "error", 2024-04-23T10:41:09.588 INFO:teuthology.orchestra.run.smithi114.stdout: "info", 2024-04-23T10:41:09.588 INFO:teuthology.orchestra.run.smithi114.stdout: "warning" 2024-04-23T10:41:09.588 INFO:teuthology.orchestra.run.smithi114.stdout: ], 2024-04-23T10:41:09.589 INFO:teuthology.orchestra.run.smithi114.stdout: "desc": "", 2024-04-23T10:41:09.589 INFO:teuthology.orchestra.run.smithi114.stdout: "long_desc": "", 2024-04-23T10:41:09.589 INFO:teuthology.orchestra.run.smithi114.stdout: "tags": [], 2024-04-23T10:41:09.589 INFO:teuthology.orchestra.run.smithi114.stdout: "see_also": [] 2024-04-23T10:41:09.589 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-23T10:41:09.589 INFO:teuthology.orchestra.run.smithi114.stdout: "log_to_cluster": { 2024-04-23T10:41:09.589 INFO:teuthology.orchestra.run.smithi114.stdout: "name": "log_to_cluster", 2024-04-23T10:41:09.589 INFO:teuthology.orchestra.run.smithi114.stdout: "type": "bool", 2024-04-23T10:41:09.589 INFO:teuthology.orchestra.run.smithi114.stdout: "level": "advanced", 2024-04-23T10:41:09.589 INFO:teuthology.orchestra.run.smithi114.stdout: "flags": 1, 2024-04-23T10:41:09.590 INFO:teuthology.orchestra.run.smithi114.stdout: "default_value": "False", 2024-04-23T10:41:09.590 INFO:teuthology.orchestra.run.smithi114.stdout: "min": "", 2024-04-23T10:41:09.590 INFO:teuthology.orchestra.run.smithi114.stdout: "max": "", 2024-04-23T10:41:09.590 INFO:teuthology.orchestra.run.smithi114.stdout: "enum_allowed": [], 2024-04-23T10:41:09.590 INFO:teuthology.orchestra.run.smithi114.stdout: "desc": "", 2024-04-23T10:41:09.590 INFO:teuthology.orchestra.run.smithi114.stdout: "long_desc": "", 2024-04-23T10:41:09.590 INFO:teuthology.orchestra.run.smithi114.stdout: "tags": [], 2024-04-23T10:41:09.590 INFO:teuthology.orchestra.run.smithi114.stdout: "see_also": [] 2024-04-23T10:41:09.591 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-23T10:41:09.591 INFO:teuthology.orchestra.run.smithi114.stdout: "log_to_cluster_level": { 2024-04-23T10:41:09.591 INFO:teuthology.orchestra.run.smithi114.stdout: "name": "log_to_cluster_level", 2024-04-23T10:41:09.591 INFO:teuthology.orchestra.run.smithi114.stdout: "type": "str", 2024-04-23T10:41:09.591 INFO:teuthology.orchestra.run.smithi114.stdout: "level": "advanced", 2024-04-23T10:41:09.591 INFO:teuthology.orchestra.run.smithi114.stdout: "flags": 1, 2024-04-23T10:41:09.591 INFO:teuthology.orchestra.run.smithi114.stdout: "default_value": "info", 2024-04-23T10:41:09.591 INFO:teuthology.orchestra.run.smithi114.stdout: "min": "", 2024-04-23T10:41:09.591 INFO:teuthology.orchestra.run.smithi114.stdout: "max": "", 2024-04-23T10:41:09.592 INFO:teuthology.orchestra.run.smithi114.stdout: "enum_allowed": [ 2024-04-23T10:41:09.592 INFO:teuthology.orchestra.run.smithi114.stdout: "", 2024-04-23T10:41:09.592 INFO:teuthology.orchestra.run.smithi114.stdout: "critical", 2024-04-23T10:41:09.592 INFO:teuthology.orchestra.run.smithi114.stdout: "debug", 2024-04-23T10:41:09.592 INFO:teuthology.orchestra.run.smithi114.stdout: "error", 2024-04-23T10:41:09.592 INFO:teuthology.orchestra.run.smithi114.stdout: "info", 2024-04-23T10:41:09.592 INFO:teuthology.orchestra.run.smithi114.stdout: "warning" 2024-04-23T10:41:09.592 INFO:teuthology.orchestra.run.smithi114.stdout: ], 2024-04-23T10:41:09.592 INFO:teuthology.orchestra.run.smithi114.stdout: "desc": "", 2024-04-23T10:41:09.592 INFO:teuthology.orchestra.run.smithi114.stdout: "long_desc": "", 2024-04-23T10:41:09.593 INFO:teuthology.orchestra.run.smithi114.stdout: "tags": [], 2024-04-23T10:41:09.593 INFO:teuthology.orchestra.run.smithi114.stdout: "see_also": [] 2024-04-23T10:41:09.593 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-23T10:41:09.593 INFO:teuthology.orchestra.run.smithi114.stdout: "log_to_file": { 2024-04-23T10:41:09.593 INFO:teuthology.orchestra.run.smithi114.stdout: "name": "log_to_file", 2024-04-23T10:41:09.593 INFO:teuthology.orchestra.run.smithi114.stdout: "type": "bool", 2024-04-23T10:41:09.593 INFO:teuthology.orchestra.run.smithi114.stdout: "level": "advanced", 2024-04-23T10:41:09.593 INFO:teuthology.orchestra.run.smithi114.stdout: "flags": 1, 2024-04-23T10:41:09.593 INFO:teuthology.orchestra.run.smithi114.stdout: "default_value": "False", 2024-04-23T10:41:09.594 INFO:teuthology.orchestra.run.smithi114.stdout: "min": "", 2024-04-23T10:41:09.594 INFO:teuthology.orchestra.run.smithi114.stdout: "max": "", 2024-04-23T10:41:09.594 INFO:teuthology.orchestra.run.smithi114.stdout: "enum_allowed": [], 2024-04-23T10:41:09.594 INFO:teuthology.orchestra.run.smithi114.stdout: "desc": "", 2024-04-23T10:41:09.594 INFO:teuthology.orchestra.run.smithi114.stdout: "long_desc": "", 2024-04-23T10:41:09.594 INFO:teuthology.orchestra.run.smithi114.stdout: "tags": [], 2024-04-23T10:41:09.594 INFO:teuthology.orchestra.run.smithi114.stdout: "see_also": [] 2024-04-23T10:41:09.594 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-23T10:41:09.594 INFO:teuthology.orchestra.run.smithi114.stdout: "retain_interval": { 2024-04-23T10:41:09.594 INFO:teuthology.orchestra.run.smithi114.stdout: "name": "retain_interval", 2024-04-23T10:41:09.595 INFO:teuthology.orchestra.run.smithi114.stdout: "type": "secs", 2024-04-23T10:41:09.595 INFO:teuthology.orchestra.run.smithi114.stdout: "level": "advanced", 2024-04-23T10:41:09.595 INFO:teuthology.orchestra.run.smithi114.stdout: "flags": 1, 2024-04-23T10:41:09.595 INFO:teuthology.orchestra.run.smithi114.stdout: "default_value": "31536000", 2024-04-23T10:41:09.595 INFO:teuthology.orchestra.run.smithi114.stdout: "min": "", 2024-04-23T10:41:09.595 INFO:teuthology.orchestra.run.smithi114.stdout: "max": "", 2024-04-23T10:41:09.595 INFO:teuthology.orchestra.run.smithi114.stdout: "enum_allowed": [], 2024-04-23T10:41:09.595 INFO:teuthology.orchestra.run.smithi114.stdout: "desc": "how long to retain crashes before pruning them", 2024-04-23T10:41:09.595 INFO:teuthology.orchestra.run.smithi114.stdout: "long_desc": "", 2024-04-23T10:41:09.596 INFO:teuthology.orchestra.run.smithi114.stdout: "tags": [], 2024-04-23T10:41:09.596 INFO:teuthology.orchestra.run.smithi114.stdout: "see_also": [] 2024-04-23T10:41:09.596 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-23T10:41:09.596 INFO:teuthology.orchestra.run.smithi114.stdout: "warn_recent_interval": { 2024-04-23T10:41:09.596 INFO:teuthology.orchestra.run.smithi114.stdout: "name": "warn_recent_interval", 2024-04-23T10:41:09.596 INFO:teuthology.orchestra.run.smithi114.stdout: "type": "secs", 2024-04-23T10:41:09.596 INFO:teuthology.orchestra.run.smithi114.stdout: "level": "advanced", 2024-04-23T10:41:09.596 INFO:teuthology.orchestra.run.smithi114.stdout: "flags": 1, 2024-04-23T10:41:09.597 INFO:teuthology.orchestra.run.smithi114.stdout: "default_value": "1209600", 2024-04-23T10:41:09.597 INFO:teuthology.orchestra.run.smithi114.stdout: "min": "", 2024-04-23T10:41:09.597 INFO:teuthology.orchestra.run.smithi114.stdout: "max": "", 2024-04-23T10:41:09.597 INFO:teuthology.orchestra.run.smithi114.stdout: "enum_allowed": [], 2024-04-23T10:41:09.597 INFO:teuthology.orchestra.run.smithi114.stdout: "desc": "time interval in which to warn about recent crashes", 2024-04-23T10:41:09.597 INFO:teuthology.orchestra.run.smithi114.stdout: "long_desc": "", 2024-04-23T10:41:09.597 INFO:teuthology.orchestra.run.smithi114.stdout: "tags": [], 2024-04-23T10:41:09.597 INFO:teuthology.orchestra.run.smithi114.stdout: "see_also": [] 2024-04-23T10:41:09.597 INFO:teuthology.orchestra.run.smithi114.stdout: } 2024-04-23T10:41:09.598 INFO:teuthology.orchestra.run.smithi114.stdout: } 2024-04-23T10:41:09.598 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-23T10:41:09.598 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-04-23T10:41:09.598 INFO:teuthology.orchestra.run.smithi114.stdout: "name": "dashboard", 2024-04-23T10:41:09.598 INFO:teuthology.orchestra.run.smithi114.stdout: "can_run": true, 2024-04-23T10:41:09.598 INFO:teuthology.orchestra.run.smithi114.stdout: "error_string": "", 2024-04-23T10:41:09.598 INFO:teuthology.orchestra.run.smithi114.stdout: "module_options": { 2024-04-23T10:41:09.598 INFO:teuthology.orchestra.run.smithi114.stdout: "ACCOUNT_LOCKOUT_ATTEMPTS": { 2024-04-23T10:41:09.598 INFO:teuthology.orchestra.run.smithi114.stdout: "name": "ACCOUNT_LOCKOUT_ATTEMPTS", 2024-04-23T10:41:09.599 INFO:teuthology.orchestra.run.smithi114.stdout: "type": "int", 2024-04-23T10:41:09.599 INFO:teuthology.orchestra.run.smithi114.stdout: "level": "advanced", 2024-04-23T10:41:09.599 INFO:teuthology.orchestra.run.smithi114.stdout: "flags": 0, 2024-04-23T10:41:09.599 INFO:teuthology.orchestra.run.smithi114.stdout: "default_value": "10", 2024-04-23T10:41:09.599 INFO:teuthology.orchestra.run.smithi114.stdout: "min": "", 2024-04-23T10:41:09.599 INFO:teuthology.orchestra.run.smithi114.stdout: "max": "", 2024-04-23T10:41:09.599 INFO:teuthology.orchestra.run.smithi114.stdout: "enum_allowed": [], 2024-04-23T10:41:09.599 INFO:teuthology.orchestra.run.smithi114.stdout: "desc": "", 2024-04-23T10:41:09.599 INFO:teuthology.orchestra.run.smithi114.stdout: "long_desc": "", 2024-04-23T10:41:09.600 INFO:teuthology.orchestra.run.smithi114.stdout: "tags": [], 2024-04-23T10:41:09.600 INFO:teuthology.orchestra.run.smithi114.stdout: "see_also": [] 2024-04-23T10:41:09.600 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-23T10:41:09.600 INFO:teuthology.orchestra.run.smithi114.stdout: "ALERTMANAGER_API_HOST": { 2024-04-23T10:41:09.600 INFO:teuthology.orchestra.run.smithi114.stdout: "name": "ALERTMANAGER_API_HOST", 2024-04-23T10:41:09.600 INFO:teuthology.orchestra.run.smithi114.stdout: "type": "str", 2024-04-23T10:41:09.600 INFO:teuthology.orchestra.run.smithi114.stdout: "level": "advanced", 2024-04-23T10:41:09.600 INFO:teuthology.orchestra.run.smithi114.stdout: "flags": 0, 2024-04-23T10:41:09.600 INFO:teuthology.orchestra.run.smithi114.stdout: "default_value": "", 2024-04-23T10:41:09.601 INFO:teuthology.orchestra.run.smithi114.stdout: "min": "", 2024-04-23T10:41:09.601 INFO:teuthology.orchestra.run.smithi114.stdout: "max": "", 2024-04-23T10:41:09.601 INFO:teuthology.orchestra.run.smithi114.stdout: "enum_allowed": [], 2024-04-23T10:41:09.601 INFO:teuthology.orchestra.run.smithi114.stdout: "desc": "", 2024-04-23T10:41:09.601 INFO:teuthology.orchestra.run.smithi114.stdout: "long_desc": "", 2024-04-23T10:41:09.601 INFO:teuthology.orchestra.run.smithi114.stdout: "tags": [], 2024-04-23T10:41:09.601 INFO:teuthology.orchestra.run.smithi114.stdout: "see_also": [] 2024-04-23T10:41:09.601 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-23T10:41:09.601 INFO:teuthology.orchestra.run.smithi114.stdout: "ALERTMANAGER_API_SSL_VERIFY": { 2024-04-23T10:41:09.602 INFO:teuthology.orchestra.run.smithi114.stdout: "name": "ALERTMANAGER_API_SSL_VERIFY", 2024-04-23T10:41:09.602 INFO:teuthology.orchestra.run.smithi114.stdout: "type": "bool", 2024-04-23T10:41:09.602 INFO:teuthology.orchestra.run.smithi114.stdout: "level": "advanced", 2024-04-23T10:41:09.602 INFO:teuthology.orchestra.run.smithi114.stdout: "flags": 0, 2024-04-23T10:41:09.602 INFO:teuthology.orchestra.run.smithi114.stdout: "default_value": "True", 2024-04-23T10:41:09.602 INFO:teuthology.orchestra.run.smithi114.stdout: "min": "", 2024-04-23T10:41:09.602 INFO:teuthology.orchestra.run.smithi114.stdout: "max": "", 2024-04-23T10:41:09.602 INFO:teuthology.orchestra.run.smithi114.stdout: "enum_allowed": [], 2024-04-23T10:41:09.602 INFO:teuthology.orchestra.run.smithi114.stdout: "desc": "", 2024-04-23T10:41:09.603 INFO:teuthology.orchestra.run.smithi114.stdout: "long_desc": "", 2024-04-23T10:41:09.603 INFO:teuthology.orchestra.run.smithi114.stdout: "tags": [], 2024-04-23T10:41:09.603 INFO:teuthology.orchestra.run.smithi114.stdout: "see_also": [] 2024-04-23T10:41:09.603 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-23T10:41:09.603 INFO:teuthology.orchestra.run.smithi114.stdout: "AUDIT_API_ENABLED": { 2024-04-23T10:41:09.603 INFO:teuthology.orchestra.run.smithi114.stdout: "name": "AUDIT_API_ENABLED", 2024-04-23T10:41:09.603 INFO:teuthology.orchestra.run.smithi114.stdout: "type": "bool", 2024-04-23T10:41:09.603 INFO:teuthology.orchestra.run.smithi114.stdout: "level": "advanced", 2024-04-23T10:41:09.604 INFO:teuthology.orchestra.run.smithi114.stdout: "flags": 0, 2024-04-23T10:41:09.604 INFO:teuthology.orchestra.run.smithi114.stdout: "default_value": "False", 2024-04-23T10:41:09.604 INFO:teuthology.orchestra.run.smithi114.stdout: "min": "", 2024-04-23T10:41:09.604 INFO:teuthology.orchestra.run.smithi114.stdout: "max": "", 2024-04-23T10:41:09.604 INFO:teuthology.orchestra.run.smithi114.stdout: "enum_allowed": [], 2024-04-23T10:41:09.604 INFO:teuthology.orchestra.run.smithi114.stdout: "desc": "", 2024-04-23T10:41:09.604 INFO:teuthology.orchestra.run.smithi114.stdout: "long_desc": "", 2024-04-23T10:41:09.604 INFO:teuthology.orchestra.run.smithi114.stdout: "tags": [], 2024-04-23T10:41:09.604 INFO:teuthology.orchestra.run.smithi114.stdout: "see_also": [] 2024-04-23T10:41:09.604 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-23T10:41:09.605 INFO:teuthology.orchestra.run.smithi114.stdout: "AUDIT_API_LOG_PAYLOAD": { 2024-04-23T10:41:09.605 INFO:teuthology.orchestra.run.smithi114.stdout: "name": "AUDIT_API_LOG_PAYLOAD", 2024-04-23T10:41:09.605 INFO:teuthology.orchestra.run.smithi114.stdout: "type": "bool", 2024-04-23T10:41:09.605 INFO:teuthology.orchestra.run.smithi114.stdout: "level": "advanced", 2024-04-23T10:41:09.605 INFO:teuthology.orchestra.run.smithi114.stdout: "flags": 0, 2024-04-23T10:41:09.605 INFO:teuthology.orchestra.run.smithi114.stdout: "default_value": "True", 2024-04-23T10:41:09.605 INFO:teuthology.orchestra.run.smithi114.stdout: "min": "", 2024-04-23T10:41:09.605 INFO:teuthology.orchestra.run.smithi114.stdout: "max": "", 2024-04-23T10:41:09.605 INFO:teuthology.orchestra.run.smithi114.stdout: "enum_allowed": [], 2024-04-23T10:41:09.606 INFO:teuthology.orchestra.run.smithi114.stdout: "desc": "", 2024-04-23T10:41:09.606 INFO:teuthology.orchestra.run.smithi114.stdout: "long_desc": "", 2024-04-23T10:41:09.606 INFO:teuthology.orchestra.run.smithi114.stdout: "tags": [], 2024-04-23T10:41:09.606 INFO:teuthology.orchestra.run.smithi114.stdout: "see_also": [] 2024-04-23T10:41:09.606 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-23T10:41:09.609 INFO:teuthology.orchestra.run.smithi114.stdout: "ENABLE_BROWSABLE_API": { 2024-04-23T10:41:09.609 INFO:teuthology.orchestra.run.smithi114.stdout: "name": "ENABLE_BROWSABLE_API", 2024-04-23T10:41:09.609 INFO:teuthology.orchestra.run.smithi114.stdout: "type": "bool", 2024-04-23T10:41:09.610 INFO:teuthology.orchestra.run.smithi114.stdout: "level": "advanced", 2024-04-23T10:41:09.610 INFO:teuthology.orchestra.run.smithi114.stdout: "flags": 0, 2024-04-23T10:41:09.610 INFO:teuthology.orchestra.run.smithi114.stdout: "default_value": "True", 2024-04-23T10:41:09.610 INFO:teuthology.orchestra.run.smithi114.stdout: "min": "", 2024-04-23T10:41:09.610 INFO:teuthology.orchestra.run.smithi114.stdout: "max": "", 2024-04-23T10:41:09.610 INFO:teuthology.orchestra.run.smithi114.stdout: "enum_allowed": [], 2024-04-23T10:41:09.610 INFO:teuthology.orchestra.run.smithi114.stdout: "desc": "", 2024-04-23T10:41:09.610 INFO:teuthology.orchestra.run.smithi114.stdout: "long_desc": "", 2024-04-23T10:41:09.611 INFO:teuthology.orchestra.run.smithi114.stdout: "tags": [], 2024-04-23T10:41:09.611 INFO:teuthology.orchestra.run.smithi114.stdout: "see_also": [] 2024-04-23T10:41:09.611 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-23T10:41:09.611 INFO:teuthology.orchestra.run.smithi114.stdout: "FEATURE_TOGGLE_CEPHFS": { 2024-04-23T10:41:09.611 INFO:teuthology.orchestra.run.smithi114.stdout: "name": "FEATURE_TOGGLE_CEPHFS", 2024-04-23T10:41:09.611 INFO:teuthology.orchestra.run.smithi114.stdout: "type": "bool", 2024-04-23T10:41:09.611 INFO:teuthology.orchestra.run.smithi114.stdout: "level": "advanced", 2024-04-23T10:41:09.611 INFO:teuthology.orchestra.run.smithi114.stdout: "flags": 0, 2024-04-23T10:41:09.611 INFO:teuthology.orchestra.run.smithi114.stdout: "default_value": "True", 2024-04-23T10:41:09.612 INFO:teuthology.orchestra.run.smithi114.stdout: "min": "", 2024-04-23T10:41:09.612 INFO:teuthology.orchestra.run.smithi114.stdout: "max": "", 2024-04-23T10:41:09.612 INFO:teuthology.orchestra.run.smithi114.stdout: "enum_allowed": [], 2024-04-23T10:41:09.612 INFO:teuthology.orchestra.run.smithi114.stdout: "desc": "", 2024-04-23T10:41:09.612 INFO:teuthology.orchestra.run.smithi114.stdout: "long_desc": "", 2024-04-23T10:41:09.612 INFO:teuthology.orchestra.run.smithi114.stdout: "tags": [], 2024-04-23T10:41:09.612 INFO:teuthology.orchestra.run.smithi114.stdout: "see_also": [] 2024-04-23T10:41:09.612 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-23T10:41:09.612 INFO:teuthology.orchestra.run.smithi114.stdout: "FEATURE_TOGGLE_DASHBOARD": { 2024-04-23T10:41:09.613 INFO:teuthology.orchestra.run.smithi114.stdout: "name": "FEATURE_TOGGLE_DASHBOARD", 2024-04-23T10:41:09.613 INFO:teuthology.orchestra.run.smithi114.stdout: "type": "bool", 2024-04-23T10:41:09.613 INFO:teuthology.orchestra.run.smithi114.stdout: "level": "advanced", 2024-04-23T10:41:09.613 INFO:teuthology.orchestra.run.smithi114.stdout: "flags": 0, 2024-04-23T10:41:09.613 INFO:teuthology.orchestra.run.smithi114.stdout: "default_value": "True", 2024-04-23T10:41:09.613 INFO:teuthology.orchestra.run.smithi114.stdout: "min": "", 2024-04-23T10:41:09.613 INFO:teuthology.orchestra.run.smithi114.stdout: "max": "", 2024-04-23T10:41:09.613 INFO:teuthology.orchestra.run.smithi114.stdout: "enum_allowed": [], 2024-04-23T10:41:09.613 INFO:teuthology.orchestra.run.smithi114.stdout: "desc": "", 2024-04-23T10:41:09.614 INFO:teuthology.orchestra.run.smithi114.stdout: "long_desc": "", 2024-04-23T10:41:09.614 INFO:teuthology.orchestra.run.smithi114.stdout: "tags": [], 2024-04-23T10:41:09.614 INFO:teuthology.orchestra.run.smithi114.stdout: "see_also": [] 2024-04-23T10:41:09.614 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-23T10:41:09.614 INFO:teuthology.orchestra.run.smithi114.stdout: "FEATURE_TOGGLE_ISCSI": { 2024-04-23T10:41:09.614 INFO:teuthology.orchestra.run.smithi114.stdout: "name": "FEATURE_TOGGLE_ISCSI", 2024-04-23T10:41:09.614 INFO:teuthology.orchestra.run.smithi114.stdout: "type": "bool", 2024-04-23T10:41:09.614 INFO:teuthology.orchestra.run.smithi114.stdout: "level": "advanced", 2024-04-23T10:41:09.614 INFO:teuthology.orchestra.run.smithi114.stdout: "flags": 0, 2024-04-23T10:41:09.615 INFO:teuthology.orchestra.run.smithi114.stdout: "default_value": "True", 2024-04-23T10:41:09.615 INFO:teuthology.orchestra.run.smithi114.stdout: "min": "", 2024-04-23T10:41:09.615 INFO:teuthology.orchestra.run.smithi114.stdout: "max": "", 2024-04-23T10:41:09.615 INFO:teuthology.orchestra.run.smithi114.stdout: "enum_allowed": [], 2024-04-23T10:41:09.615 INFO:teuthology.orchestra.run.smithi114.stdout: "desc": "", 2024-04-23T10:41:09.615 INFO:teuthology.orchestra.run.smithi114.stdout: "long_desc": "", 2024-04-23T10:41:09.615 INFO:teuthology.orchestra.run.smithi114.stdout: "tags": [], 2024-04-23T10:41:09.615 INFO:teuthology.orchestra.run.smithi114.stdout: "see_also": [] 2024-04-23T10:41:09.615 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-23T10:41:09.616 INFO:teuthology.orchestra.run.smithi114.stdout: "FEATURE_TOGGLE_MIRRORING": { 2024-04-23T10:41:09.616 INFO:teuthology.orchestra.run.smithi114.stdout: "name": "FEATURE_TOGGLE_MIRRORING", 2024-04-23T10:41:09.616 INFO:teuthology.orchestra.run.smithi114.stdout: "type": "bool", 2024-04-23T10:41:09.616 INFO:teuthology.orchestra.run.smithi114.stdout: "level": "advanced", 2024-04-23T10:41:09.616 INFO:teuthology.orchestra.run.smithi114.stdout: "flags": 0, 2024-04-23T10:41:09.616 INFO:teuthology.orchestra.run.smithi114.stdout: "default_value": "True", 2024-04-23T10:41:09.616 INFO:teuthology.orchestra.run.smithi114.stdout: "min": "", 2024-04-23T10:41:09.616 INFO:teuthology.orchestra.run.smithi114.stdout: "max": "", 2024-04-23T10:41:09.616 INFO:teuthology.orchestra.run.smithi114.stdout: "enum_allowed": [], 2024-04-23T10:41:09.617 INFO:teuthology.orchestra.run.smithi114.stdout: "desc": "", 2024-04-23T10:41:09.617 INFO:teuthology.orchestra.run.smithi114.stdout: "long_desc": "", 2024-04-23T10:41:09.617 INFO:teuthology.orchestra.run.smithi114.stdout: "tags": [], 2024-04-23T10:41:09.617 INFO:teuthology.orchestra.run.smithi114.stdout: "see_also": [] 2024-04-23T10:41:09.617 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-23T10:41:09.617 INFO:teuthology.orchestra.run.smithi114.stdout: "FEATURE_TOGGLE_NFS": { 2024-04-23T10:41:09.617 INFO:teuthology.orchestra.run.smithi114.stdout: "name": "FEATURE_TOGGLE_NFS", 2024-04-23T10:41:09.617 INFO:teuthology.orchestra.run.smithi114.stdout: "type": "bool", 2024-04-23T10:41:09.617 INFO:teuthology.orchestra.run.smithi114.stdout: "level": "advanced", 2024-04-23T10:41:09.618 INFO:teuthology.orchestra.run.smithi114.stdout: "flags": 0, 2024-04-23T10:41:09.618 INFO:teuthology.orchestra.run.smithi114.stdout: "default_value": "True", 2024-04-23T10:41:09.618 INFO:teuthology.orchestra.run.smithi114.stdout: "min": "", 2024-04-23T10:41:09.618 INFO:teuthology.orchestra.run.smithi114.stdout: "max": "", 2024-04-23T10:41:09.618 INFO:teuthology.orchestra.run.smithi114.stdout: "enum_allowed": [], 2024-04-23T10:41:09.618 INFO:teuthology.orchestra.run.smithi114.stdout: "desc": "", 2024-04-23T10:41:09.618 INFO:teuthology.orchestra.run.smithi114.stdout: "long_desc": "", 2024-04-23T10:41:09.618 INFO:teuthology.orchestra.run.smithi114.stdout: "tags": [], 2024-04-23T10:41:09.618 INFO:teuthology.orchestra.run.smithi114.stdout: "see_also": [] 2024-04-23T10:41:09.619 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-23T10:41:09.619 INFO:teuthology.orchestra.run.smithi114.stdout: "FEATURE_TOGGLE_RBD": { 2024-04-23T10:41:09.619 INFO:teuthology.orchestra.run.smithi114.stdout: "name": "FEATURE_TOGGLE_RBD", 2024-04-23T10:41:09.619 INFO:teuthology.orchestra.run.smithi114.stdout: "type": "bool", 2024-04-23T10:41:09.619 INFO:teuthology.orchestra.run.smithi114.stdout: "level": "advanced", 2024-04-23T10:41:09.619 INFO:teuthology.orchestra.run.smithi114.stdout: "flags": 0, 2024-04-23T10:41:09.619 INFO:teuthology.orchestra.run.smithi114.stdout: "default_value": "True", 2024-04-23T10:41:09.619 INFO:teuthology.orchestra.run.smithi114.stdout: "min": "", 2024-04-23T10:41:09.620 INFO:teuthology.orchestra.run.smithi114.stdout: "max": "", 2024-04-23T10:41:09.620 INFO:teuthology.orchestra.run.smithi114.stdout: "enum_allowed": [], 2024-04-23T10:41:09.620 INFO:teuthology.orchestra.run.smithi114.stdout: "desc": "", 2024-04-23T10:41:09.620 INFO:teuthology.orchestra.run.smithi114.stdout: "long_desc": "", 2024-04-23T10:41:09.620 INFO:teuthology.orchestra.run.smithi114.stdout: "tags": [], 2024-04-23T10:41:09.620 INFO:teuthology.orchestra.run.smithi114.stdout: "see_also": [] 2024-04-23T10:41:09.620 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-23T10:41:09.620 INFO:teuthology.orchestra.run.smithi114.stdout: "FEATURE_TOGGLE_RGW": { 2024-04-23T10:41:09.620 INFO:teuthology.orchestra.run.smithi114.stdout: "name": "FEATURE_TOGGLE_RGW", 2024-04-23T10:41:09.620 INFO:teuthology.orchestra.run.smithi114.stdout: "type": "bool", 2024-04-23T10:41:09.621 INFO:teuthology.orchestra.run.smithi114.stdout: "level": "advanced", 2024-04-23T10:41:09.621 INFO:teuthology.orchestra.run.smithi114.stdout: "flags": 0, 2024-04-23T10:41:09.621 INFO:teuthology.orchestra.run.smithi114.stdout: "default_value": "True", 2024-04-23T10:41:09.621 INFO:teuthology.orchestra.run.smithi114.stdout: "min": "", 2024-04-23T10:41:09.621 INFO:teuthology.orchestra.run.smithi114.stdout: "max": "", 2024-04-23T10:41:09.621 INFO:teuthology.orchestra.run.smithi114.stdout: "enum_allowed": [], 2024-04-23T10:41:09.621 INFO:teuthology.orchestra.run.smithi114.stdout: "desc": "", 2024-04-23T10:41:09.621 INFO:teuthology.orchestra.run.smithi114.stdout: "long_desc": "", 2024-04-23T10:41:09.621 INFO:teuthology.orchestra.run.smithi114.stdout: "tags": [], 2024-04-23T10:41:09.622 INFO:teuthology.orchestra.run.smithi114.stdout: "see_also": [] 2024-04-23T10:41:09.622 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-23T10:41:09.622 INFO:teuthology.orchestra.run.smithi114.stdout: "GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE": { 2024-04-23T10:41:09.622 INFO:teuthology.orchestra.run.smithi114.stdout: "name": "GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE", 2024-04-23T10:41:09.622 INFO:teuthology.orchestra.run.smithi114.stdout: "type": "str", 2024-04-23T10:41:09.622 INFO:teuthology.orchestra.run.smithi114.stdout: "level": "advanced", 2024-04-23T10:41:09.622 INFO:teuthology.orchestra.run.smithi114.stdout: "flags": 0, 2024-04-23T10:41:09.622 INFO:teuthology.orchestra.run.smithi114.stdout: "default_value": "", 2024-04-23T10:41:09.622 INFO:teuthology.orchestra.run.smithi114.stdout: "min": "", 2024-04-23T10:41:09.623 INFO:teuthology.orchestra.run.smithi114.stdout: "max": "", 2024-04-23T10:41:09.623 INFO:teuthology.orchestra.run.smithi114.stdout: "enum_allowed": [], 2024-04-23T10:41:09.623 INFO:teuthology.orchestra.run.smithi114.stdout: "desc": "", 2024-04-23T10:41:09.623 INFO:teuthology.orchestra.run.smithi114.stdout: "long_desc": "", 2024-04-23T10:41:09.623 INFO:teuthology.orchestra.run.smithi114.stdout: "tags": [], 2024-04-23T10:41:09.623 INFO:teuthology.orchestra.run.smithi114.stdout: "see_also": [] 2024-04-23T10:41:09.623 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-23T10:41:09.623 INFO:teuthology.orchestra.run.smithi114.stdout: "GRAFANA_API_PASSWORD": { 2024-04-23T10:41:09.623 INFO:teuthology.orchestra.run.smithi114.stdout: "name": "GRAFANA_API_PASSWORD", 2024-04-23T10:41:09.624 INFO:teuthology.orchestra.run.smithi114.stdout: "type": "str", 2024-04-23T10:41:09.624 INFO:teuthology.orchestra.run.smithi114.stdout: "level": "advanced", 2024-04-23T10:41:09.624 INFO:teuthology.orchestra.run.smithi114.stdout: "flags": 0, 2024-04-23T10:41:09.624 INFO:teuthology.orchestra.run.smithi114.stdout: "default_value": "admin", 2024-04-23T10:41:09.624 INFO:teuthology.orchestra.run.smithi114.stdout: "min": "", 2024-04-23T10:41:09.624 INFO:teuthology.orchestra.run.smithi114.stdout: "max": "", 2024-04-23T10:41:09.624 INFO:teuthology.orchestra.run.smithi114.stdout: "enum_allowed": [], 2024-04-23T10:41:09.624 INFO:teuthology.orchestra.run.smithi114.stdout: "desc": "", 2024-04-23T10:41:09.624 INFO:teuthology.orchestra.run.smithi114.stdout: "long_desc": "", 2024-04-23T10:41:09.625 INFO:teuthology.orchestra.run.smithi114.stdout: "tags": [], 2024-04-23T10:41:09.625 INFO:teuthology.orchestra.run.smithi114.stdout: "see_also": [] 2024-04-23T10:41:09.625 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-23T10:41:09.625 INFO:teuthology.orchestra.run.smithi114.stdout: "GRAFANA_API_SSL_VERIFY": { 2024-04-23T10:41:09.625 INFO:teuthology.orchestra.run.smithi114.stdout: "name": "GRAFANA_API_SSL_VERIFY", 2024-04-23T10:41:09.625 INFO:teuthology.orchestra.run.smithi114.stdout: "type": "bool", 2024-04-23T10:41:09.625 INFO:teuthology.orchestra.run.smithi114.stdout: "level": "advanced", 2024-04-23T10:41:09.625 INFO:teuthology.orchestra.run.smithi114.stdout: "flags": 0, 2024-04-23T10:41:09.625 INFO:teuthology.orchestra.run.smithi114.stdout: "default_value": "True", 2024-04-23T10:41:09.626 INFO:teuthology.orchestra.run.smithi114.stdout: "min": "", 2024-04-23T10:41:09.626 INFO:teuthology.orchestra.run.smithi114.stdout: "max": "", 2024-04-23T10:41:09.626 INFO:teuthology.orchestra.run.smithi114.stdout: "enum_allowed": [], 2024-04-23T10:41:09.626 INFO:teuthology.orchestra.run.smithi114.stdout: "desc": "", 2024-04-23T10:41:09.626 INFO:teuthology.orchestra.run.smithi114.stdout: "long_desc": "", 2024-04-23T10:41:09.626 INFO:teuthology.orchestra.run.smithi114.stdout: "tags": [], 2024-04-23T10:41:09.626 INFO:teuthology.orchestra.run.smithi114.stdout: "see_also": [] 2024-04-23T10:41:09.626 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-23T10:41:09.626 INFO:teuthology.orchestra.run.smithi114.stdout: "GRAFANA_API_URL": { 2024-04-23T10:41:09.627 INFO:teuthology.orchestra.run.smithi114.stdout: "name": "GRAFANA_API_URL", 2024-04-23T10:41:09.627 INFO:teuthology.orchestra.run.smithi114.stdout: "type": "str", 2024-04-23T10:41:09.627 INFO:teuthology.orchestra.run.smithi114.stdout: "level": "advanced", 2024-04-23T10:41:09.627 INFO:teuthology.orchestra.run.smithi114.stdout: "flags": 0, 2024-04-23T10:41:09.627 INFO:teuthology.orchestra.run.smithi114.stdout: "default_value": "", 2024-04-23T10:41:09.627 INFO:teuthology.orchestra.run.smithi114.stdout: "min": "", 2024-04-23T10:41:09.627 INFO:teuthology.orchestra.run.smithi114.stdout: "max": "", 2024-04-23T10:41:09.627 INFO:teuthology.orchestra.run.smithi114.stdout: "enum_allowed": [], 2024-04-23T10:41:09.627 INFO:teuthology.orchestra.run.smithi114.stdout: "desc": "", 2024-04-23T10:41:09.627 INFO:teuthology.orchestra.run.smithi114.stdout: "long_desc": "", 2024-04-23T10:41:09.628 INFO:teuthology.orchestra.run.smithi114.stdout: "tags": [], 2024-04-23T10:41:09.628 INFO:teuthology.orchestra.run.smithi114.stdout: "see_also": [] 2024-04-23T10:41:09.628 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-23T10:41:09.628 INFO:teuthology.orchestra.run.smithi114.stdout: "GRAFANA_API_USERNAME": { 2024-04-23T10:41:09.628 INFO:teuthology.orchestra.run.smithi114.stdout: "name": "GRAFANA_API_USERNAME", 2024-04-23T10:41:09.628 INFO:teuthology.orchestra.run.smithi114.stdout: "type": "str", 2024-04-23T10:41:09.628 INFO:teuthology.orchestra.run.smithi114.stdout: "level": "advanced", 2024-04-23T10:41:09.628 INFO:teuthology.orchestra.run.smithi114.stdout: "flags": 0, 2024-04-23T10:41:09.629 INFO:teuthology.orchestra.run.smithi114.stdout: "default_value": "admin", 2024-04-23T10:41:09.629 INFO:teuthology.orchestra.run.smithi114.stdout: "min": "", 2024-04-23T10:41:09.629 INFO:teuthology.orchestra.run.smithi114.stdout: "max": "", 2024-04-23T10:41:09.629 INFO:teuthology.orchestra.run.smithi114.stdout: "enum_allowed": [], 2024-04-23T10:41:09.629 INFO:teuthology.orchestra.run.smithi114.stdout: "desc": "", 2024-04-23T10:41:09.629 INFO:teuthology.orchestra.run.smithi114.stdout: "long_desc": "", 2024-04-23T10:41:09.629 INFO:teuthology.orchestra.run.smithi114.stdout: "tags": [], 2024-04-23T10:41:09.629 INFO:teuthology.orchestra.run.smithi114.stdout: "see_also": [] 2024-04-23T10:41:09.629 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-23T10:41:09.629 INFO:teuthology.orchestra.run.smithi114.stdout: "GRAFANA_FRONTEND_API_URL": { 2024-04-23T10:41:09.630 INFO:teuthology.orchestra.run.smithi114.stdout: "name": "GRAFANA_FRONTEND_API_URL", 2024-04-23T10:41:09.630 INFO:teuthology.orchestra.run.smithi114.stdout: "type": "str", 2024-04-23T10:41:09.630 INFO:teuthology.orchestra.run.smithi114.stdout: "level": "advanced", 2024-04-23T10:41:09.630 INFO:teuthology.orchestra.run.smithi114.stdout: "flags": 0, 2024-04-23T10:41:09.630 INFO:teuthology.orchestra.run.smithi114.stdout: "default_value": "", 2024-04-23T10:41:09.630 INFO:teuthology.orchestra.run.smithi114.stdout: "min": "", 2024-04-23T10:41:09.630 INFO:teuthology.orchestra.run.smithi114.stdout: "max": "", 2024-04-23T10:41:09.630 INFO:teuthology.orchestra.run.smithi114.stdout: "enum_allowed": [], 2024-04-23T10:41:09.630 INFO:teuthology.orchestra.run.smithi114.stdout: "desc": "", 2024-04-23T10:41:09.631 INFO:teuthology.orchestra.run.smithi114.stdout: "long_desc": "", 2024-04-23T10:41:09.631 INFO:teuthology.orchestra.run.smithi114.stdout: "tags": [], 2024-04-23T10:41:09.631 INFO:teuthology.orchestra.run.smithi114.stdout: "see_also": [] 2024-04-23T10:41:09.631 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-23T10:41:09.631 INFO:teuthology.orchestra.run.smithi114.stdout: "GRAFANA_UPDATE_DASHBOARDS": { 2024-04-23T10:41:09.631 INFO:teuthology.orchestra.run.smithi114.stdout: "name": "GRAFANA_UPDATE_DASHBOARDS", 2024-04-23T10:41:09.631 INFO:teuthology.orchestra.run.smithi114.stdout: "type": "bool", 2024-04-23T10:41:09.631 INFO:teuthology.orchestra.run.smithi114.stdout: "level": "advanced", 2024-04-23T10:41:09.632 INFO:teuthology.orchestra.run.smithi114.stdout: "flags": 0, 2024-04-23T10:41:09.632 INFO:teuthology.orchestra.run.smithi114.stdout: "default_value": "False", 2024-04-23T10:41:09.632 INFO:teuthology.orchestra.run.smithi114.stdout: "min": "", 2024-04-23T10:41:09.632 INFO:teuthology.orchestra.run.smithi114.stdout: "max": "", 2024-04-23T10:41:09.632 INFO:teuthology.orchestra.run.smithi114.stdout: "enum_allowed": [], 2024-04-23T10:41:09.632 INFO:teuthology.orchestra.run.smithi114.stdout: "desc": "", 2024-04-23T10:41:09.632 INFO:teuthology.orchestra.run.smithi114.stdout: "long_desc": "", 2024-04-23T10:41:09.632 INFO:teuthology.orchestra.run.smithi114.stdout: "tags": [], 2024-04-23T10:41:09.632 INFO:teuthology.orchestra.run.smithi114.stdout: "see_also": [] 2024-04-23T10:41:09.633 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-23T10:41:09.633 INFO:teuthology.orchestra.run.smithi114.stdout: "ISCSI_API_SSL_VERIFICATION": { 2024-04-23T10:41:09.633 INFO:teuthology.orchestra.run.smithi114.stdout: "name": "ISCSI_API_SSL_VERIFICATION", 2024-04-23T10:41:09.633 INFO:teuthology.orchestra.run.smithi114.stdout: "type": "bool", 2024-04-23T10:41:09.633 INFO:teuthology.orchestra.run.smithi114.stdout: "level": "advanced", 2024-04-23T10:41:09.633 INFO:teuthology.orchestra.run.smithi114.stdout: "flags": 0, 2024-04-23T10:41:09.633 INFO:teuthology.orchestra.run.smithi114.stdout: "default_value": "True", 2024-04-23T10:41:09.633 INFO:teuthology.orchestra.run.smithi114.stdout: "min": "", 2024-04-23T10:41:09.633 INFO:teuthology.orchestra.run.smithi114.stdout: "max": "", 2024-04-23T10:41:09.633 INFO:teuthology.orchestra.run.smithi114.stdout: "enum_allowed": [], 2024-04-23T10:41:09.634 INFO:teuthology.orchestra.run.smithi114.stdout: "desc": "", 2024-04-23T10:41:09.634 INFO:teuthology.orchestra.run.smithi114.stdout: "long_desc": "", 2024-04-23T10:41:09.634 INFO:teuthology.orchestra.run.smithi114.stdout: "tags": [], 2024-04-23T10:41:09.634 INFO:teuthology.orchestra.run.smithi114.stdout: "see_also": [] 2024-04-23T10:41:09.634 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-23T10:41:09.634 INFO:teuthology.orchestra.run.smithi114.stdout: "ISSUE_TRACKER_API_KEY": { 2024-04-23T10:41:09.634 INFO:teuthology.orchestra.run.smithi114.stdout: "name": "ISSUE_TRACKER_API_KEY", 2024-04-23T10:41:09.634 INFO:teuthology.orchestra.run.smithi114.stdout: "type": "str", 2024-04-23T10:41:09.635 INFO:teuthology.orchestra.run.smithi114.stdout: "level": "advanced", 2024-04-23T10:41:09.635 INFO:teuthology.orchestra.run.smithi114.stdout: "flags": 0, 2024-04-23T10:41:09.635 INFO:teuthology.orchestra.run.smithi114.stdout: "default_value": "", 2024-04-23T10:41:09.635 INFO:teuthology.orchestra.run.smithi114.stdout: "min": "", 2024-04-23T10:41:09.635 INFO:teuthology.orchestra.run.smithi114.stdout: "max": "", 2024-04-23T10:41:09.635 INFO:teuthology.orchestra.run.smithi114.stdout: "enum_allowed": [], 2024-04-23T10:41:09.635 INFO:teuthology.orchestra.run.smithi114.stdout: "desc": "", 2024-04-23T10:41:09.635 INFO:teuthology.orchestra.run.smithi114.stdout: "long_desc": "", 2024-04-23T10:41:09.635 INFO:teuthology.orchestra.run.smithi114.stdout: "tags": [], 2024-04-23T10:41:09.635 INFO:teuthology.orchestra.run.smithi114.stdout: "see_also": [] 2024-04-23T10:41:09.636 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-23T10:41:09.636 INFO:teuthology.orchestra.run.smithi114.stdout: "MANAGED_BY_CLUSTERS": { 2024-04-23T10:41:09.636 INFO:teuthology.orchestra.run.smithi114.stdout: "name": "MANAGED_BY_CLUSTERS", 2024-04-23T10:41:09.636 INFO:teuthology.orchestra.run.smithi114.stdout: "type": "str", 2024-04-23T10:41:09.636 INFO:teuthology.orchestra.run.smithi114.stdout: "level": "advanced", 2024-04-23T10:41:09.636 INFO:teuthology.orchestra.run.smithi114.stdout: "flags": 0, 2024-04-23T10:41:09.636 INFO:teuthology.orchestra.run.smithi114.stdout: "default_value": "[]", 2024-04-23T10:41:09.636 INFO:teuthology.orchestra.run.smithi114.stdout: "min": "", 2024-04-23T10:41:09.636 INFO:teuthology.orchestra.run.smithi114.stdout: "max": "", 2024-04-23T10:41:09.637 INFO:teuthology.orchestra.run.smithi114.stdout: "enum_allowed": [], 2024-04-23T10:41:09.637 INFO:teuthology.orchestra.run.smithi114.stdout: "desc": "", 2024-04-23T10:41:09.637 INFO:teuthology.orchestra.run.smithi114.stdout: "long_desc": "", 2024-04-23T10:41:09.637 INFO:teuthology.orchestra.run.smithi114.stdout: "tags": [], 2024-04-23T10:41:09.637 INFO:teuthology.orchestra.run.smithi114.stdout: "see_also": [] 2024-04-23T10:41:09.637 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-23T10:41:09.637 INFO:teuthology.orchestra.run.smithi114.stdout: "MULTICLUSTER_CONFIG": { 2024-04-23T10:41:09.637 INFO:teuthology.orchestra.run.smithi114.stdout: "name": "MULTICLUSTER_CONFIG", 2024-04-23T10:41:09.637 INFO:teuthology.orchestra.run.smithi114.stdout: "type": "str", 2024-04-23T10:41:09.638 INFO:teuthology.orchestra.run.smithi114.stdout: "level": "advanced", 2024-04-23T10:41:09.638 INFO:teuthology.orchestra.run.smithi114.stdout: "flags": 0, 2024-04-23T10:41:09.638 INFO:teuthology.orchestra.run.smithi114.stdout: "default_value": "{}", 2024-04-23T10:41:09.638 INFO:teuthology.orchestra.run.smithi114.stdout: "min": "", 2024-04-23T10:41:09.638 INFO:teuthology.orchestra.run.smithi114.stdout: "max": "", 2024-04-23T10:41:09.638 INFO:teuthology.orchestra.run.smithi114.stdout: "enum_allowed": [], 2024-04-23T10:41:09.638 INFO:teuthology.orchestra.run.smithi114.stdout: "desc": "", 2024-04-23T10:41:09.638 INFO:teuthology.orchestra.run.smithi114.stdout: "long_desc": "", 2024-04-23T10:41:09.638 INFO:teuthology.orchestra.run.smithi114.stdout: "tags": [], 2024-04-23T10:41:09.639 INFO:teuthology.orchestra.run.smithi114.stdout: "see_also": [] 2024-04-23T10:41:09.639 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-23T10:41:09.639 INFO:teuthology.orchestra.run.smithi114.stdout: "PROMETHEUS_API_HOST": { 2024-04-23T10:41:09.639 INFO:teuthology.orchestra.run.smithi114.stdout: "name": "PROMETHEUS_API_HOST", 2024-04-23T10:41:09.639 INFO:teuthology.orchestra.run.smithi114.stdout: "type": "str", 2024-04-23T10:41:09.639 INFO:teuthology.orchestra.run.smithi114.stdout: "level": "advanced", 2024-04-23T10:41:09.639 INFO:teuthology.orchestra.run.smithi114.stdout: "flags": 0, 2024-04-23T10:41:09.639 INFO:teuthology.orchestra.run.smithi114.stdout: "default_value": "", 2024-04-23T10:41:09.639 INFO:teuthology.orchestra.run.smithi114.stdout: "min": "", 2024-04-23T10:41:09.639 INFO:teuthology.orchestra.run.smithi114.stdout: "max": "", 2024-04-23T10:41:09.640 INFO:teuthology.orchestra.run.smithi114.stdout: "enum_allowed": [], 2024-04-23T10:41:09.640 INFO:teuthology.orchestra.run.smithi114.stdout: "desc": "", 2024-04-23T10:41:09.640 INFO:teuthology.orchestra.run.smithi114.stdout: "long_desc": "", 2024-04-23T10:41:09.640 INFO:teuthology.orchestra.run.smithi114.stdout: "tags": [], 2024-04-23T10:41:09.640 INFO:teuthology.orchestra.run.smithi114.stdout: "see_also": [] 2024-04-23T10:41:09.640 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-23T10:41:09.640 INFO:teuthology.orchestra.run.smithi114.stdout: "PROMETHEUS_API_SSL_VERIFY": { 2024-04-23T10:41:09.640 INFO:teuthology.orchestra.run.smithi114.stdout: "name": "PROMETHEUS_API_SSL_VERIFY", 2024-04-23T10:41:09.640 INFO:teuthology.orchestra.run.smithi114.stdout: "type": "bool", 2024-04-23T10:41:09.641 INFO:teuthology.orchestra.run.smithi114.stdout: "level": "advanced", 2024-04-23T10:41:09.641 INFO:teuthology.orchestra.run.smithi114.stdout: "flags": 0, 2024-04-23T10:41:09.641 INFO:teuthology.orchestra.run.smithi114.stdout: "default_value": "True", 2024-04-23T10:41:09.641 INFO:teuthology.orchestra.run.smithi114.stdout: "min": "", 2024-04-23T10:41:09.641 INFO:teuthology.orchestra.run.smithi114.stdout: "max": "", 2024-04-23T10:41:09.641 INFO:teuthology.orchestra.run.smithi114.stdout: "enum_allowed": [], 2024-04-23T10:41:09.641 INFO:teuthology.orchestra.run.smithi114.stdout: "desc": "", 2024-04-23T10:41:09.641 INFO:teuthology.orchestra.run.smithi114.stdout: "long_desc": "", 2024-04-23T10:41:09.642 INFO:teuthology.orchestra.run.smithi114.stdout: "tags": [], 2024-04-23T10:41:09.642 INFO:teuthology.orchestra.run.smithi114.stdout: "see_also": [] 2024-04-23T10:41:09.642 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-23T10:41:09.642 INFO:teuthology.orchestra.run.smithi114.stdout: "PWD_POLICY_CHECK_COMPLEXITY_ENABLED": { 2024-04-23T10:41:09.642 INFO:teuthology.orchestra.run.smithi114.stdout: "name": "PWD_POLICY_CHECK_COMPLEXITY_ENABLED", 2024-04-23T10:41:09.642 INFO:teuthology.orchestra.run.smithi114.stdout: "type": "bool", 2024-04-23T10:41:09.642 INFO:teuthology.orchestra.run.smithi114.stdout: "level": "advanced", 2024-04-23T10:41:09.642 INFO:teuthology.orchestra.run.smithi114.stdout: "flags": 0, 2024-04-23T10:41:09.642 INFO:teuthology.orchestra.run.smithi114.stdout: "default_value": "False", 2024-04-23T10:41:09.643 INFO:teuthology.orchestra.run.smithi114.stdout: "min": "", 2024-04-23T10:41:09.643 INFO:teuthology.orchestra.run.smithi114.stdout: "max": "", 2024-04-23T10:41:09.643 INFO:teuthology.orchestra.run.smithi114.stdout: "enum_allowed": [], 2024-04-23T10:41:09.643 INFO:teuthology.orchestra.run.smithi114.stdout: "desc": "", 2024-04-23T10:41:09.643 INFO:teuthology.orchestra.run.smithi114.stdout: "long_desc": "", 2024-04-23T10:41:09.643 INFO:teuthology.orchestra.run.smithi114.stdout: "tags": [], 2024-04-23T10:41:09.643 INFO:teuthology.orchestra.run.smithi114.stdout: "see_also": [] 2024-04-23T10:41:09.643 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-23T10:41:09.643 INFO:teuthology.orchestra.run.smithi114.stdout: "PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED": { 2024-04-23T10:41:09.644 INFO:teuthology.orchestra.run.smithi114.stdout: "name": "PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED", 2024-04-23T10:41:09.644 INFO:teuthology.orchestra.run.smithi114.stdout: "type": "bool", 2024-04-23T10:41:09.644 INFO:teuthology.orchestra.run.smithi114.stdout: "level": "advanced", 2024-04-23T10:41:09.644 INFO:teuthology.orchestra.run.smithi114.stdout: "flags": 0, 2024-04-23T10:41:09.644 INFO:teuthology.orchestra.run.smithi114.stdout: "default_value": "False", 2024-04-23T10:41:09.644 INFO:teuthology.orchestra.run.smithi114.stdout: "min": "", 2024-04-23T10:41:09.644 INFO:teuthology.orchestra.run.smithi114.stdout: "max": "", 2024-04-23T10:41:09.644 INFO:teuthology.orchestra.run.smithi114.stdout: "enum_allowed": [], 2024-04-23T10:41:09.644 INFO:teuthology.orchestra.run.smithi114.stdout: "desc": "", 2024-04-23T10:41:09.645 INFO:teuthology.orchestra.run.smithi114.stdout: "long_desc": "", 2024-04-23T10:41:09.645 INFO:teuthology.orchestra.run.smithi114.stdout: "tags": [], 2024-04-23T10:41:09.645 INFO:teuthology.orchestra.run.smithi114.stdout: "see_also": [] 2024-04-23T10:41:09.645 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-23T10:41:09.645 INFO:teuthology.orchestra.run.smithi114.stdout: "PWD_POLICY_CHECK_LENGTH_ENABLED": { 2024-04-23T10:41:09.645 INFO:teuthology.orchestra.run.smithi114.stdout: "name": "PWD_POLICY_CHECK_LENGTH_ENABLED", 2024-04-23T10:41:09.645 INFO:teuthology.orchestra.run.smithi114.stdout: "type": "bool", 2024-04-23T10:41:09.645 INFO:teuthology.orchestra.run.smithi114.stdout: "level": "advanced", 2024-04-23T10:41:09.645 INFO:teuthology.orchestra.run.smithi114.stdout: "flags": 0, 2024-04-23T10:41:09.646 INFO:teuthology.orchestra.run.smithi114.stdout: "default_value": "True", 2024-04-23T10:41:09.646 INFO:teuthology.orchestra.run.smithi114.stdout: "min": "", 2024-04-23T10:41:09.646 INFO:teuthology.orchestra.run.smithi114.stdout: "max": "", 2024-04-23T10:41:09.646 INFO:teuthology.orchestra.run.smithi114.stdout: "enum_allowed": [], 2024-04-23T10:41:09.646 INFO:teuthology.orchestra.run.smithi114.stdout: "desc": "", 2024-04-23T10:41:09.646 INFO:teuthology.orchestra.run.smithi114.stdout: "long_desc": "", 2024-04-23T10:41:09.646 INFO:teuthology.orchestra.run.smithi114.stdout: "tags": [], 2024-04-23T10:41:09.646 INFO:teuthology.orchestra.run.smithi114.stdout: "see_also": [] 2024-04-23T10:41:09.646 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-23T10:41:09.647 INFO:teuthology.orchestra.run.smithi114.stdout: "PWD_POLICY_CHECK_OLDPWD_ENABLED": { 2024-04-23T10:41:09.647 INFO:teuthology.orchestra.run.smithi114.stdout: "name": "PWD_POLICY_CHECK_OLDPWD_ENABLED", 2024-04-23T10:41:09.647 INFO:teuthology.orchestra.run.smithi114.stdout: "type": "bool", 2024-04-23T10:41:09.647 INFO:teuthology.orchestra.run.smithi114.stdout: "level": "advanced", 2024-04-23T10:41:09.647 INFO:teuthology.orchestra.run.smithi114.stdout: "flags": 0, 2024-04-23T10:41:09.647 INFO:teuthology.orchestra.run.smithi114.stdout: "default_value": "True", 2024-04-23T10:41:09.647 INFO:teuthology.orchestra.run.smithi114.stdout: "min": "", 2024-04-23T10:41:09.647 INFO:teuthology.orchestra.run.smithi114.stdout: "max": "", 2024-04-23T10:41:09.647 INFO:teuthology.orchestra.run.smithi114.stdout: "enum_allowed": [], 2024-04-23T10:41:09.648 INFO:teuthology.orchestra.run.smithi114.stdout: "desc": "", 2024-04-23T10:41:09.648 INFO:teuthology.orchestra.run.smithi114.stdout: "long_desc": "", 2024-04-23T10:41:09.648 INFO:teuthology.orchestra.run.smithi114.stdout: "tags": [], 2024-04-23T10:41:09.648 INFO:teuthology.orchestra.run.smithi114.stdout: "see_also": [] 2024-04-23T10:41:09.648 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-23T10:41:09.648 INFO:teuthology.orchestra.run.smithi114.stdout: "PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED": { 2024-04-23T10:41:09.648 INFO:teuthology.orchestra.run.smithi114.stdout: "name": "PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED", 2024-04-23T10:41:09.648 INFO:teuthology.orchestra.run.smithi114.stdout: "type": "bool", 2024-04-23T10:41:09.648 INFO:teuthology.orchestra.run.smithi114.stdout: "level": "advanced", 2024-04-23T10:41:09.649 INFO:teuthology.orchestra.run.smithi114.stdout: "flags": 0, 2024-04-23T10:41:09.649 INFO:teuthology.orchestra.run.smithi114.stdout: "default_value": "False", 2024-04-23T10:41:09.649 INFO:teuthology.orchestra.run.smithi114.stdout: "min": "", 2024-04-23T10:41:09.649 INFO:teuthology.orchestra.run.smithi114.stdout: "max": "", 2024-04-23T10:41:09.649 INFO:teuthology.orchestra.run.smithi114.stdout: "enum_allowed": [], 2024-04-23T10:41:09.649 INFO:teuthology.orchestra.run.smithi114.stdout: "desc": "", 2024-04-23T10:41:09.649 INFO:teuthology.orchestra.run.smithi114.stdout: "long_desc": "", 2024-04-23T10:41:09.649 INFO:teuthology.orchestra.run.smithi114.stdout: "tags": [], 2024-04-23T10:41:09.650 INFO:teuthology.orchestra.run.smithi114.stdout: "see_also": [] 2024-04-23T10:41:09.650 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-23T10:41:09.650 INFO:teuthology.orchestra.run.smithi114.stdout: "PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED": { 2024-04-23T10:41:09.650 INFO:teuthology.orchestra.run.smithi114.stdout: "name": "PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED", 2024-04-23T10:41:09.650 INFO:teuthology.orchestra.run.smithi114.stdout: "type": "bool", 2024-04-23T10:41:09.650 INFO:teuthology.orchestra.run.smithi114.stdout: "level": "advanced", 2024-04-23T10:41:09.650 INFO:teuthology.orchestra.run.smithi114.stdout: "flags": 0, 2024-04-23T10:41:09.650 INFO:teuthology.orchestra.run.smithi114.stdout: "default_value": "False", 2024-04-23T10:41:09.650 INFO:teuthology.orchestra.run.smithi114.stdout: "min": "", 2024-04-23T10:41:09.651 INFO:teuthology.orchestra.run.smithi114.stdout: "max": "", 2024-04-23T10:41:09.651 INFO:teuthology.orchestra.run.smithi114.stdout: "enum_allowed": [], 2024-04-23T10:41:09.651 INFO:teuthology.orchestra.run.smithi114.stdout: "desc": "", 2024-04-23T10:41:09.651 INFO:teuthology.orchestra.run.smithi114.stdout: "long_desc": "", 2024-04-23T10:41:09.651 INFO:teuthology.orchestra.run.smithi114.stdout: "tags": [], 2024-04-23T10:41:09.651 INFO:teuthology.orchestra.run.smithi114.stdout: "see_also": [] 2024-04-23T10:41:09.651 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-23T10:41:09.651 INFO:teuthology.orchestra.run.smithi114.stdout: "PWD_POLICY_CHECK_USERNAME_ENABLED": { 2024-04-23T10:41:09.651 INFO:teuthology.orchestra.run.smithi114.stdout: "name": "PWD_POLICY_CHECK_USERNAME_ENABLED", 2024-04-23T10:41:09.652 INFO:teuthology.orchestra.run.smithi114.stdout: "type": "bool", 2024-04-23T10:41:09.652 INFO:teuthology.orchestra.run.smithi114.stdout: "level": "advanced", 2024-04-23T10:41:09.652 INFO:teuthology.orchestra.run.smithi114.stdout: "flags": 0, 2024-04-23T10:41:09.652 INFO:teuthology.orchestra.run.smithi114.stdout: "default_value": "False", 2024-04-23T10:41:09.652 INFO:teuthology.orchestra.run.smithi114.stdout: "min": "", 2024-04-23T10:41:09.652 INFO:teuthology.orchestra.run.smithi114.stdout: "max": "", 2024-04-23T10:41:09.652 INFO:teuthology.orchestra.run.smithi114.stdout: "enum_allowed": [], 2024-04-23T10:41:09.652 INFO:teuthology.orchestra.run.smithi114.stdout: "desc": "", 2024-04-23T10:41:09.652 INFO:teuthology.orchestra.run.smithi114.stdout: "long_desc": "", 2024-04-23T10:41:09.652 INFO:teuthology.orchestra.run.smithi114.stdout: "tags": [], 2024-04-23T10:41:09.653 INFO:teuthology.orchestra.run.smithi114.stdout: "see_also": [] 2024-04-23T10:41:09.653 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-23T10:41:09.653 INFO:teuthology.orchestra.run.smithi114.stdout: "PWD_POLICY_ENABLED": { 2024-04-23T10:41:09.653 INFO:teuthology.orchestra.run.smithi114.stdout: "name": "PWD_POLICY_ENABLED", 2024-04-23T10:41:09.653 INFO:teuthology.orchestra.run.smithi114.stdout: "type": "bool", 2024-04-23T10:41:09.653 INFO:teuthology.orchestra.run.smithi114.stdout: "level": "advanced", 2024-04-23T10:41:09.653 INFO:teuthology.orchestra.run.smithi114.stdout: "flags": 0, 2024-04-23T10:41:09.653 INFO:teuthology.orchestra.run.smithi114.stdout: "default_value": "True", 2024-04-23T10:41:09.653 INFO:teuthology.orchestra.run.smithi114.stdout: "min": "", 2024-04-23T10:41:09.654 INFO:teuthology.orchestra.run.smithi114.stdout: "max": "", 2024-04-23T10:41:09.654 INFO:teuthology.orchestra.run.smithi114.stdout: "enum_allowed": [], 2024-04-23T10:41:09.654 INFO:teuthology.orchestra.run.smithi114.stdout: "desc": "", 2024-04-23T10:41:09.654 INFO:teuthology.orchestra.run.smithi114.stdout: "long_desc": "", 2024-04-23T10:41:09.654 INFO:teuthology.orchestra.run.smithi114.stdout: "tags": [], 2024-04-23T10:41:09.654 INFO:teuthology.orchestra.run.smithi114.stdout: "see_also": [] 2024-04-23T10:41:09.654 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-23T10:41:09.654 INFO:teuthology.orchestra.run.smithi114.stdout: "PWD_POLICY_EXCLUSION_LIST": { 2024-04-23T10:41:09.654 INFO:teuthology.orchestra.run.smithi114.stdout: "name": "PWD_POLICY_EXCLUSION_LIST", 2024-04-23T10:41:09.655 INFO:teuthology.orchestra.run.smithi114.stdout: "type": "str", 2024-04-23T10:41:09.655 INFO:teuthology.orchestra.run.smithi114.stdout: "level": "advanced", 2024-04-23T10:41:09.655 INFO:teuthology.orchestra.run.smithi114.stdout: "flags": 0, 2024-04-23T10:41:09.655 INFO:teuthology.orchestra.run.smithi114.stdout: "default_value": "osd,host,dashboard,pool,block,nfs,ceph,monitors,gateway,logs,crush,maps", 2024-04-23T10:41:09.655 INFO:teuthology.orchestra.run.smithi114.stdout: "min": "", 2024-04-23T10:41:09.655 INFO:teuthology.orchestra.run.smithi114.stdout: "max": "", 2024-04-23T10:41:09.655 INFO:teuthology.orchestra.run.smithi114.stdout: "enum_allowed": [], 2024-04-23T10:41:09.655 INFO:teuthology.orchestra.run.smithi114.stdout: "desc": "", 2024-04-23T10:41:09.656 INFO:teuthology.orchestra.run.smithi114.stdout: "long_desc": "", 2024-04-23T10:41:09.656 INFO:teuthology.orchestra.run.smithi114.stdout: "tags": [], 2024-04-23T10:41:09.656 INFO:teuthology.orchestra.run.smithi114.stdout: "see_also": [] 2024-04-23T10:41:09.656 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-23T10:41:09.656 INFO:teuthology.orchestra.run.smithi114.stdout: "PWD_POLICY_MIN_COMPLEXITY": { 2024-04-23T10:41:09.656 INFO:teuthology.orchestra.run.smithi114.stdout: "name": "PWD_POLICY_MIN_COMPLEXITY", 2024-04-23T10:41:09.656 INFO:teuthology.orchestra.run.smithi114.stdout: "type": "int", 2024-04-23T10:41:09.656 INFO:teuthology.orchestra.run.smithi114.stdout: "level": "advanced", 2024-04-23T10:41:09.656 INFO:teuthology.orchestra.run.smithi114.stdout: "flags": 0, 2024-04-23T10:41:09.657 INFO:teuthology.orchestra.run.smithi114.stdout: "default_value": "10", 2024-04-23T10:41:09.657 INFO:teuthology.orchestra.run.smithi114.stdout: "min": "", 2024-04-23T10:41:09.657 INFO:teuthology.orchestra.run.smithi114.stdout: "max": "", 2024-04-23T10:41:09.657 INFO:teuthology.orchestra.run.smithi114.stdout: "enum_allowed": [], 2024-04-23T10:41:09.657 INFO:teuthology.orchestra.run.smithi114.stdout: "desc": "", 2024-04-23T10:41:09.657 INFO:teuthology.orchestra.run.smithi114.stdout: "long_desc": "", 2024-04-23T10:41:09.657 INFO:teuthology.orchestra.run.smithi114.stdout: "tags": [], 2024-04-23T10:41:09.657 INFO:teuthology.orchestra.run.smithi114.stdout: "see_also": [] 2024-04-23T10:41:09.658 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-23T10:41:09.658 INFO:teuthology.orchestra.run.smithi114.stdout: "PWD_POLICY_MIN_LENGTH": { 2024-04-23T10:41:09.658 INFO:teuthology.orchestra.run.smithi114.stdout: "name": "PWD_POLICY_MIN_LENGTH", 2024-04-23T10:41:09.658 INFO:teuthology.orchestra.run.smithi114.stdout: "type": "int", 2024-04-23T10:41:09.658 INFO:teuthology.orchestra.run.smithi114.stdout: "level": "advanced", 2024-04-23T10:41:09.658 INFO:teuthology.orchestra.run.smithi114.stdout: "flags": 0, 2024-04-23T10:41:09.658 INFO:teuthology.orchestra.run.smithi114.stdout: "default_value": "8", 2024-04-23T10:41:09.658 INFO:teuthology.orchestra.run.smithi114.stdout: "min": "", 2024-04-23T10:41:09.658 INFO:teuthology.orchestra.run.smithi114.stdout: "max": "", 2024-04-23T10:41:09.659 INFO:teuthology.orchestra.run.smithi114.stdout: "enum_allowed": [], 2024-04-23T10:41:09.659 INFO:teuthology.orchestra.run.smithi114.stdout: "desc": "", 2024-04-23T10:41:09.659 INFO:teuthology.orchestra.run.smithi114.stdout: "long_desc": "", 2024-04-23T10:41:09.659 INFO:teuthology.orchestra.run.smithi114.stdout: "tags": [], 2024-04-23T10:41:09.659 INFO:teuthology.orchestra.run.smithi114.stdout: "see_also": [] 2024-04-23T10:41:09.659 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-23T10:41:09.659 INFO:teuthology.orchestra.run.smithi114.stdout: "REST_REQUESTS_TIMEOUT": { 2024-04-23T10:41:09.659 INFO:teuthology.orchestra.run.smithi114.stdout: "name": "REST_REQUESTS_TIMEOUT", 2024-04-23T10:41:09.659 INFO:teuthology.orchestra.run.smithi114.stdout: "type": "int", 2024-04-23T10:41:09.660 INFO:teuthology.orchestra.run.smithi114.stdout: "level": "advanced", 2024-04-23T10:41:09.660 INFO:teuthology.orchestra.run.smithi114.stdout: "flags": 0, 2024-04-23T10:41:09.660 INFO:teuthology.orchestra.run.smithi114.stdout: "default_value": "45", 2024-04-23T10:41:09.660 INFO:teuthology.orchestra.run.smithi114.stdout: "min": "", 2024-04-23T10:41:09.660 INFO:teuthology.orchestra.run.smithi114.stdout: "max": "", 2024-04-23T10:41:09.660 INFO:teuthology.orchestra.run.smithi114.stdout: "enum_allowed": [], 2024-04-23T10:41:09.660 INFO:teuthology.orchestra.run.smithi114.stdout: "desc": "", 2024-04-23T10:41:09.660 INFO:teuthology.orchestra.run.smithi114.stdout: "long_desc": "", 2024-04-23T10:41:09.660 INFO:teuthology.orchestra.run.smithi114.stdout: "tags": [], 2024-04-23T10:41:09.661 INFO:teuthology.orchestra.run.smithi114.stdout: "see_also": [] 2024-04-23T10:41:09.661 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-23T10:41:09.661 INFO:teuthology.orchestra.run.smithi114.stdout: "RGW_API_ACCESS_KEY": { 2024-04-23T10:41:09.661 INFO:teuthology.orchestra.run.smithi114.stdout: "name": "RGW_API_ACCESS_KEY", 2024-04-23T10:41:09.661 INFO:teuthology.orchestra.run.smithi114.stdout: "type": "str", 2024-04-23T10:41:09.661 INFO:teuthology.orchestra.run.smithi114.stdout: "level": "advanced", 2024-04-23T10:41:09.661 INFO:teuthology.orchestra.run.smithi114.stdout: "flags": 0, 2024-04-23T10:41:09.661 INFO:teuthology.orchestra.run.smithi114.stdout: "default_value": "", 2024-04-23T10:41:09.661 INFO:teuthology.orchestra.run.smithi114.stdout: "min": "", 2024-04-23T10:41:09.662 INFO:teuthology.orchestra.run.smithi114.stdout: "max": "", 2024-04-23T10:41:09.662 INFO:teuthology.orchestra.run.smithi114.stdout: "enum_allowed": [], 2024-04-23T10:41:09.662 INFO:teuthology.orchestra.run.smithi114.stdout: "desc": "", 2024-04-23T10:41:09.662 INFO:teuthology.orchestra.run.smithi114.stdout: "long_desc": "", 2024-04-23T10:41:09.662 INFO:teuthology.orchestra.run.smithi114.stdout: "tags": [], 2024-04-23T10:41:09.662 INFO:teuthology.orchestra.run.smithi114.stdout: "see_also": [] 2024-04-23T10:41:09.662 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-23T10:41:09.662 INFO:teuthology.orchestra.run.smithi114.stdout: "RGW_API_ADMIN_RESOURCE": { 2024-04-23T10:41:09.663 INFO:teuthology.orchestra.run.smithi114.stdout: "name": "RGW_API_ADMIN_RESOURCE", 2024-04-23T10:41:09.663 INFO:teuthology.orchestra.run.smithi114.stdout: "type": "str", 2024-04-23T10:41:09.663 INFO:teuthology.orchestra.run.smithi114.stdout: "level": "advanced", 2024-04-23T10:41:09.663 INFO:teuthology.orchestra.run.smithi114.stdout: "flags": 0, 2024-04-23T10:41:09.663 INFO:teuthology.orchestra.run.smithi114.stdout: "default_value": "admin", 2024-04-23T10:41:09.663 INFO:teuthology.orchestra.run.smithi114.stdout: "min": "", 2024-04-23T10:41:09.663 INFO:teuthology.orchestra.run.smithi114.stdout: "max": "", 2024-04-23T10:41:09.663 INFO:teuthology.orchestra.run.smithi114.stdout: "enum_allowed": [], 2024-04-23T10:41:09.664 INFO:teuthology.orchestra.run.smithi114.stdout: "desc": "", 2024-04-23T10:41:09.664 INFO:teuthology.orchestra.run.smithi114.stdout: "long_desc": "", 2024-04-23T10:41:09.664 INFO:teuthology.orchestra.run.smithi114.stdout: "tags": [], 2024-04-23T10:41:09.664 INFO:teuthology.orchestra.run.smithi114.stdout: "see_also": [] 2024-04-23T10:41:09.664 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-23T10:41:09.664 INFO:teuthology.orchestra.run.smithi114.stdout: "RGW_API_SECRET_KEY": { 2024-04-23T10:41:09.664 INFO:teuthology.orchestra.run.smithi114.stdout: "name": "RGW_API_SECRET_KEY", 2024-04-23T10:41:09.664 INFO:teuthology.orchestra.run.smithi114.stdout: "type": "str", 2024-04-23T10:41:09.664 INFO:teuthology.orchestra.run.smithi114.stdout: "level": "advanced", 2024-04-23T10:41:09.665 INFO:teuthology.orchestra.run.smithi114.stdout: "flags": 0, 2024-04-23T10:41:09.665 INFO:teuthology.orchestra.run.smithi114.stdout: "default_value": "", 2024-04-23T10:41:09.665 INFO:teuthology.orchestra.run.smithi114.stdout: "min": "", 2024-04-23T10:41:09.665 INFO:teuthology.orchestra.run.smithi114.stdout: "max": "", 2024-04-23T10:41:09.665 INFO:teuthology.orchestra.run.smithi114.stdout: "enum_allowed": [], 2024-04-23T10:41:09.665 INFO:teuthology.orchestra.run.smithi114.stdout: "desc": "", 2024-04-23T10:41:09.665 INFO:teuthology.orchestra.run.smithi114.stdout: "long_desc": "", 2024-04-23T10:41:09.666 INFO:teuthology.orchestra.run.smithi114.stdout: "tags": [], 2024-04-23T10:41:09.666 INFO:teuthology.orchestra.run.smithi114.stdout: "see_also": [] 2024-04-23T10:41:09.666 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-23T10:41:09.666 INFO:teuthology.orchestra.run.smithi114.stdout: "RGW_API_SSL_VERIFY": { 2024-04-23T10:41:09.666 INFO:teuthology.orchestra.run.smithi114.stdout: "name": "RGW_API_SSL_VERIFY", 2024-04-23T10:41:09.666 INFO:teuthology.orchestra.run.smithi114.stdout: "type": "bool", 2024-04-23T10:41:09.666 INFO:teuthology.orchestra.run.smithi114.stdout: "level": "advanced", 2024-04-23T10:41:09.666 INFO:teuthology.orchestra.run.smithi114.stdout: "flags": 0, 2024-04-23T10:41:09.667 INFO:teuthology.orchestra.run.smithi114.stdout: "default_value": "True", 2024-04-23T10:41:09.667 INFO:teuthology.orchestra.run.smithi114.stdout: "min": "", 2024-04-23T10:41:09.667 INFO:teuthology.orchestra.run.smithi114.stdout: "max": "", 2024-04-23T10:41:09.667 INFO:teuthology.orchestra.run.smithi114.stdout: "enum_allowed": [], 2024-04-23T10:41:09.667 INFO:teuthology.orchestra.run.smithi114.stdout: "desc": "", 2024-04-23T10:41:09.667 INFO:teuthology.orchestra.run.smithi114.stdout: "long_desc": "", 2024-04-23T10:41:09.667 INFO:teuthology.orchestra.run.smithi114.stdout: "tags": [], 2024-04-23T10:41:09.667 INFO:teuthology.orchestra.run.smithi114.stdout: "see_also": [] 2024-04-23T10:41:09.668 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-23T10:41:09.668 INFO:teuthology.orchestra.run.smithi114.stdout: "UNSAFE_TLS_v1_2": { 2024-04-23T10:41:09.668 INFO:teuthology.orchestra.run.smithi114.stdout: "name": "UNSAFE_TLS_v1_2", 2024-04-23T10:41:09.668 INFO:teuthology.orchestra.run.smithi114.stdout: "type": "bool", 2024-04-23T10:41:09.668 INFO:teuthology.orchestra.run.smithi114.stdout: "level": "advanced", 2024-04-23T10:41:09.668 INFO:teuthology.orchestra.run.smithi114.stdout: "flags": 0, 2024-04-23T10:41:09.668 INFO:teuthology.orchestra.run.smithi114.stdout: "default_value": "False", 2024-04-23T10:41:09.668 INFO:teuthology.orchestra.run.smithi114.stdout: "min": "", 2024-04-23T10:41:09.668 INFO:teuthology.orchestra.run.smithi114.stdout: "max": "", 2024-04-23T10:41:09.669 INFO:teuthology.orchestra.run.smithi114.stdout: "enum_allowed": [], 2024-04-23T10:41:09.669 INFO:teuthology.orchestra.run.smithi114.stdout: "desc": "", 2024-04-23T10:41:09.669 INFO:teuthology.orchestra.run.smithi114.stdout: "long_desc": "", 2024-04-23T10:41:09.669 INFO:teuthology.orchestra.run.smithi114.stdout: "tags": [], 2024-04-23T10:41:09.669 INFO:teuthology.orchestra.run.smithi114.stdout: "see_also": [] 2024-04-23T10:41:09.669 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-23T10:41:09.669 INFO:teuthology.orchestra.run.smithi114.stdout: "USER_PWD_EXPIRATION_SPAN": { 2024-04-23T10:41:09.669 INFO:teuthology.orchestra.run.smithi114.stdout: "name": "USER_PWD_EXPIRATION_SPAN", 2024-04-23T10:41:09.670 INFO:teuthology.orchestra.run.smithi114.stdout: "type": "int", 2024-04-23T10:41:09.670 INFO:teuthology.orchestra.run.smithi114.stdout: "level": "advanced", 2024-04-23T10:41:09.670 INFO:teuthology.orchestra.run.smithi114.stdout: "flags": 0, 2024-04-23T10:41:09.670 INFO:teuthology.orchestra.run.smithi114.stdout: "default_value": "0", 2024-04-23T10:41:09.670 INFO:teuthology.orchestra.run.smithi114.stdout: "min": "", 2024-04-23T10:41:09.670 INFO:teuthology.orchestra.run.smithi114.stdout: "max": "", 2024-04-23T10:41:09.670 INFO:teuthology.orchestra.run.smithi114.stdout: "enum_allowed": [], 2024-04-23T10:41:09.670 INFO:teuthology.orchestra.run.smithi114.stdout: "desc": "", 2024-04-23T10:41:09.671 INFO:teuthology.orchestra.run.smithi114.stdout: "long_desc": "", 2024-04-23T10:41:09.671 INFO:teuthology.orchestra.run.smithi114.stdout: "tags": [], 2024-04-23T10:41:09.671 INFO:teuthology.orchestra.run.smithi114.stdout: "see_also": [] 2024-04-23T10:41:09.671 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-23T10:41:09.671 INFO:teuthology.orchestra.run.smithi114.stdout: "USER_PWD_EXPIRATION_WARNING_1": { 2024-04-23T10:41:09.671 INFO:teuthology.orchestra.run.smithi114.stdout: "name": "USER_PWD_EXPIRATION_WARNING_1", 2024-04-23T10:41:09.671 INFO:teuthology.orchestra.run.smithi114.stdout: "type": "int", 2024-04-23T10:41:09.671 INFO:teuthology.orchestra.run.smithi114.stdout: "level": "advanced", 2024-04-23T10:41:09.671 INFO:teuthology.orchestra.run.smithi114.stdout: "flags": 0, 2024-04-23T10:41:09.672 INFO:teuthology.orchestra.run.smithi114.stdout: "default_value": "10", 2024-04-23T10:41:09.672 INFO:teuthology.orchestra.run.smithi114.stdout: "min": "", 2024-04-23T10:41:09.672 INFO:teuthology.orchestra.run.smithi114.stdout: "max": "", 2024-04-23T10:41:09.672 INFO:teuthology.orchestra.run.smithi114.stdout: "enum_allowed": [], 2024-04-23T10:41:09.672 INFO:teuthology.orchestra.run.smithi114.stdout: "desc": "", 2024-04-23T10:41:09.672 INFO:teuthology.orchestra.run.smithi114.stdout: "long_desc": "", 2024-04-23T10:41:09.672 INFO:teuthology.orchestra.run.smithi114.stdout: "tags": [], 2024-04-23T10:41:09.672 INFO:teuthology.orchestra.run.smithi114.stdout: "see_also": [] 2024-04-23T10:41:09.672 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-23T10:41:09.672 INFO:teuthology.orchestra.run.smithi114.stdout: "USER_PWD_EXPIRATION_WARNING_2": { 2024-04-23T10:41:09.673 INFO:teuthology.orchestra.run.smithi114.stdout: "name": "USER_PWD_EXPIRATION_WARNING_2", 2024-04-23T10:41:09.673 INFO:teuthology.orchestra.run.smithi114.stdout: "type": "int", 2024-04-23T10:41:09.673 INFO:teuthology.orchestra.run.smithi114.stdout: "level": "advanced", 2024-04-23T10:41:09.673 INFO:teuthology.orchestra.run.smithi114.stdout: "flags": 0, 2024-04-23T10:41:09.673 INFO:teuthology.orchestra.run.smithi114.stdout: "default_value": "5", 2024-04-23T10:41:09.673 INFO:teuthology.orchestra.run.smithi114.stdout: "min": "", 2024-04-23T10:41:09.673 INFO:teuthology.orchestra.run.smithi114.stdout: "max": "", 2024-04-23T10:41:09.673 INFO:teuthology.orchestra.run.smithi114.stdout: "enum_allowed": [], 2024-04-23T10:41:09.673 INFO:teuthology.orchestra.run.smithi114.stdout: "desc": "", 2024-04-23T10:41:09.674 INFO:teuthology.orchestra.run.smithi114.stdout: "long_desc": "", 2024-04-23T10:41:09.674 INFO:teuthology.orchestra.run.smithi114.stdout: "tags": [], 2024-04-23T10:41:09.674 INFO:teuthology.orchestra.run.smithi114.stdout: "see_also": [] 2024-04-23T10:41:09.674 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-23T10:41:09.674 INFO:teuthology.orchestra.run.smithi114.stdout: "cross_origin_url": { 2024-04-23T10:41:09.674 INFO:teuthology.orchestra.run.smithi114.stdout: "name": "cross_origin_url", 2024-04-23T10:41:09.674 INFO:teuthology.orchestra.run.smithi114.stdout: "type": "str", 2024-04-23T10:41:09.674 INFO:teuthology.orchestra.run.smithi114.stdout: "level": "advanced", 2024-04-23T10:41:09.674 INFO:teuthology.orchestra.run.smithi114.stdout: "flags": 0, 2024-04-23T10:41:09.675 INFO:teuthology.orchestra.run.smithi114.stdout: "default_value": "", 2024-04-23T10:41:09.675 INFO:teuthology.orchestra.run.smithi114.stdout: "min": "", 2024-04-23T10:41:09.675 INFO:teuthology.orchestra.run.smithi114.stdout: "max": "", 2024-04-23T10:41:09.675 INFO:teuthology.orchestra.run.smithi114.stdout: "enum_allowed": [], 2024-04-23T10:41:09.675 INFO:teuthology.orchestra.run.smithi114.stdout: "desc": "", 2024-04-23T10:41:09.675 INFO:teuthology.orchestra.run.smithi114.stdout: "long_desc": "", 2024-04-23T10:41:09.675 INFO:teuthology.orchestra.run.smithi114.stdout: "tags": [], 2024-04-23T10:41:09.675 INFO:teuthology.orchestra.run.smithi114.stdout: "see_also": [] 2024-04-23T10:41:09.675 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-23T10:41:09.675 INFO:teuthology.orchestra.run.smithi114.stdout: "crt_file": { 2024-04-23T10:41:09.676 INFO:teuthology.orchestra.run.smithi114.stdout: "name": "crt_file", 2024-04-23T10:41:09.676 INFO:teuthology.orchestra.run.smithi114.stdout: "type": "str", 2024-04-23T10:41:09.676 INFO:teuthology.orchestra.run.smithi114.stdout: "level": "advanced", 2024-04-23T10:41:09.676 INFO:teuthology.orchestra.run.smithi114.stdout: "flags": 0, 2024-04-23T10:41:09.676 INFO:teuthology.orchestra.run.smithi114.stdout: "default_value": "", 2024-04-23T10:41:09.676 INFO:teuthology.orchestra.run.smithi114.stdout: "min": "", 2024-04-23T10:41:09.676 INFO:teuthology.orchestra.run.smithi114.stdout: "max": "", 2024-04-23T10:41:09.676 INFO:teuthology.orchestra.run.smithi114.stdout: "enum_allowed": [], 2024-04-23T10:41:09.676 INFO:teuthology.orchestra.run.smithi114.stdout: "desc": "", 2024-04-23T10:41:09.676 INFO:teuthology.orchestra.run.smithi114.stdout: "long_desc": "", 2024-04-23T10:41:09.677 INFO:teuthology.orchestra.run.smithi114.stdout: "tags": [], 2024-04-23T10:41:09.677 INFO:teuthology.orchestra.run.smithi114.stdout: "see_also": [] 2024-04-23T10:41:09.677 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-23T10:41:09.677 INFO:teuthology.orchestra.run.smithi114.stdout: "debug": { 2024-04-23T10:41:09.677 INFO:teuthology.orchestra.run.smithi114.stdout: "name": "debug", 2024-04-23T10:41:09.677 INFO:teuthology.orchestra.run.smithi114.stdout: "type": "bool", 2024-04-23T10:41:09.677 INFO:teuthology.orchestra.run.smithi114.stdout: "level": "advanced", 2024-04-23T10:41:09.677 INFO:teuthology.orchestra.run.smithi114.stdout: "flags": 0, 2024-04-23T10:41:09.677 INFO:teuthology.orchestra.run.smithi114.stdout: "default_value": "False", 2024-04-23T10:41:09.677 INFO:teuthology.orchestra.run.smithi114.stdout: "min": "", 2024-04-23T10:41:09.678 INFO:teuthology.orchestra.run.smithi114.stdout: "max": "", 2024-04-23T10:41:09.678 INFO:teuthology.orchestra.run.smithi114.stdout: "enum_allowed": [], 2024-04-23T10:41:09.678 INFO:teuthology.orchestra.run.smithi114.stdout: "desc": "Enable/disable debug options", 2024-04-23T10:41:09.678 INFO:teuthology.orchestra.run.smithi114.stdout: "long_desc": "", 2024-04-23T10:41:09.678 INFO:teuthology.orchestra.run.smithi114.stdout: "tags": [], 2024-04-23T10:41:09.678 INFO:teuthology.orchestra.run.smithi114.stdout: "see_also": [] 2024-04-23T10:41:09.678 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-23T10:41:09.678 INFO:teuthology.orchestra.run.smithi114.stdout: "jwt_token_ttl": { 2024-04-23T10:41:09.679 INFO:teuthology.orchestra.run.smithi114.stdout: "name": "jwt_token_ttl", 2024-04-23T10:41:09.679 INFO:teuthology.orchestra.run.smithi114.stdout: "type": "int", 2024-04-23T10:41:09.679 INFO:teuthology.orchestra.run.smithi114.stdout: "level": "advanced", 2024-04-23T10:41:09.679 INFO:teuthology.orchestra.run.smithi114.stdout: "flags": 0, 2024-04-23T10:41:09.679 INFO:teuthology.orchestra.run.smithi114.stdout: "default_value": "28800", 2024-04-23T10:41:09.679 INFO:teuthology.orchestra.run.smithi114.stdout: "min": "", 2024-04-23T10:41:09.679 INFO:teuthology.orchestra.run.smithi114.stdout: "max": "", 2024-04-23T10:41:09.679 INFO:teuthology.orchestra.run.smithi114.stdout: "enum_allowed": [], 2024-04-23T10:41:09.679 INFO:teuthology.orchestra.run.smithi114.stdout: "desc": "", 2024-04-23T10:41:09.679 INFO:teuthology.orchestra.run.smithi114.stdout: "long_desc": "", 2024-04-23T10:41:09.680 INFO:teuthology.orchestra.run.smithi114.stdout: "tags": [], 2024-04-23T10:41:09.680 INFO:teuthology.orchestra.run.smithi114.stdout: "see_also": [] 2024-04-23T10:41:09.680 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-23T10:41:09.680 INFO:teuthology.orchestra.run.smithi114.stdout: "key_file": { 2024-04-23T10:41:09.680 INFO:teuthology.orchestra.run.smithi114.stdout: "name": "key_file", 2024-04-23T10:41:09.680 INFO:teuthology.orchestra.run.smithi114.stdout: "type": "str", 2024-04-23T10:41:09.680 INFO:teuthology.orchestra.run.smithi114.stdout: "level": "advanced", 2024-04-23T10:41:09.680 INFO:teuthology.orchestra.run.smithi114.stdout: "flags": 0, 2024-04-23T10:41:09.680 INFO:teuthology.orchestra.run.smithi114.stdout: "default_value": "", 2024-04-23T10:41:09.680 INFO:teuthology.orchestra.run.smithi114.stdout: "min": "", 2024-04-23T10:41:09.681 INFO:teuthology.orchestra.run.smithi114.stdout: "max": "", 2024-04-23T10:41:09.681 INFO:teuthology.orchestra.run.smithi114.stdout: "enum_allowed": [], 2024-04-23T10:41:09.681 INFO:teuthology.orchestra.run.smithi114.stdout: "desc": "", 2024-04-23T10:41:09.681 INFO:teuthology.orchestra.run.smithi114.stdout: "long_desc": "", 2024-04-23T10:41:09.681 INFO:teuthology.orchestra.run.smithi114.stdout: "tags": [], 2024-04-23T10:41:09.681 INFO:teuthology.orchestra.run.smithi114.stdout: "see_also": [] 2024-04-23T10:41:09.681 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-23T10:41:09.681 INFO:teuthology.orchestra.run.smithi114.stdout: "log_level": { 2024-04-23T10:41:09.681 INFO:teuthology.orchestra.run.smithi114.stdout: "name": "log_level", 2024-04-23T10:41:09.681 INFO:teuthology.orchestra.run.smithi114.stdout: "type": "str", 2024-04-23T10:41:09.682 INFO:teuthology.orchestra.run.smithi114.stdout: "level": "advanced", 2024-04-23T10:41:09.682 INFO:teuthology.orchestra.run.smithi114.stdout: "flags": 1, 2024-04-23T10:41:09.682 INFO:teuthology.orchestra.run.smithi114.stdout: "default_value": "", 2024-04-23T10:41:09.682 INFO:teuthology.orchestra.run.smithi114.stdout: "min": "", 2024-04-23T10:41:09.682 INFO:teuthology.orchestra.run.smithi114.stdout: "max": "", 2024-04-23T10:41:09.682 INFO:teuthology.orchestra.run.smithi114.stdout: "enum_allowed": [ 2024-04-23T10:41:09.682 INFO:teuthology.orchestra.run.smithi114.stdout: "", 2024-04-23T10:41:09.682 INFO:teuthology.orchestra.run.smithi114.stdout: "critical", 2024-04-23T10:41:09.683 INFO:teuthology.orchestra.run.smithi114.stdout: "debug", 2024-04-23T10:41:09.683 INFO:teuthology.orchestra.run.smithi114.stdout: "error", 2024-04-23T10:41:09.683 INFO:teuthology.orchestra.run.smithi114.stdout: "info", 2024-04-23T10:41:09.683 INFO:teuthology.orchestra.run.smithi114.stdout: "warning" 2024-04-23T10:41:09.683 INFO:teuthology.orchestra.run.smithi114.stdout: ], 2024-04-23T10:41:09.683 INFO:teuthology.orchestra.run.smithi114.stdout: "desc": "", 2024-04-23T10:41:09.683 INFO:teuthology.orchestra.run.smithi114.stdout: "long_desc": "", 2024-04-23T10:41:09.683 INFO:teuthology.orchestra.run.smithi114.stdout: "tags": [], 2024-04-23T10:41:09.683 INFO:teuthology.orchestra.run.smithi114.stdout: "see_also": [] 2024-04-23T10:41:09.683 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-23T10:41:09.684 INFO:teuthology.orchestra.run.smithi114.stdout: "log_to_cluster": { 2024-04-23T10:41:09.684 INFO:teuthology.orchestra.run.smithi114.stdout: "name": "log_to_cluster", 2024-04-23T10:41:09.684 INFO:teuthology.orchestra.run.smithi114.stdout: "type": "bool", 2024-04-23T10:41:09.684 INFO:teuthology.orchestra.run.smithi114.stdout: "level": "advanced", 2024-04-23T10:41:09.684 INFO:teuthology.orchestra.run.smithi114.stdout: "flags": 1, 2024-04-23T10:41:09.684 INFO:teuthology.orchestra.run.smithi114.stdout: "default_value": "False", 2024-04-23T10:41:09.684 INFO:teuthology.orchestra.run.smithi114.stdout: "min": "", 2024-04-23T10:41:09.684 INFO:teuthology.orchestra.run.smithi114.stdout: "max": "", 2024-04-23T10:41:09.684 INFO:teuthology.orchestra.run.smithi114.stdout: "enum_allowed": [], 2024-04-23T10:41:09.684 INFO:teuthology.orchestra.run.smithi114.stdout: "desc": "", 2024-04-23T10:41:09.685 INFO:teuthology.orchestra.run.smithi114.stdout: "long_desc": "", 2024-04-23T10:41:09.685 INFO:teuthology.orchestra.run.smithi114.stdout: "tags": [], 2024-04-23T10:41:09.685 INFO:teuthology.orchestra.run.smithi114.stdout: "see_also": [] 2024-04-23T10:41:09.685 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-23T10:41:09.685 INFO:teuthology.orchestra.run.smithi114.stdout: "log_to_cluster_level": { 2024-04-23T10:41:09.685 INFO:teuthology.orchestra.run.smithi114.stdout: "name": "log_to_cluster_level", 2024-04-23T10:41:09.685 INFO:teuthology.orchestra.run.smithi114.stdout: "type": "str", 2024-04-23T10:41:09.685 INFO:teuthology.orchestra.run.smithi114.stdout: "level": "advanced", 2024-04-23T10:41:09.685 INFO:teuthology.orchestra.run.smithi114.stdout: "flags": 1, 2024-04-23T10:41:09.685 INFO:teuthology.orchestra.run.smithi114.stdout: "default_value": "info", 2024-04-23T10:41:09.686 INFO:teuthology.orchestra.run.smithi114.stdout: "min": "", 2024-04-23T10:41:09.686 INFO:teuthology.orchestra.run.smithi114.stdout: "max": "", 2024-04-23T10:41:09.686 INFO:teuthology.orchestra.run.smithi114.stdout: "enum_allowed": [ 2024-04-23T10:41:09.686 INFO:teuthology.orchestra.run.smithi114.stdout: "", 2024-04-23T10:41:09.686 INFO:teuthology.orchestra.run.smithi114.stdout: "critical", 2024-04-23T10:41:09.686 INFO:teuthology.orchestra.run.smithi114.stdout: "debug", 2024-04-23T10:41:09.686 INFO:teuthology.orchestra.run.smithi114.stdout: "error", 2024-04-23T10:41:09.686 INFO:teuthology.orchestra.run.smithi114.stdout: "info", 2024-04-23T10:41:09.687 INFO:teuthology.orchestra.run.smithi114.stdout: "warning" 2024-04-23T10:41:09.687 INFO:teuthology.orchestra.run.smithi114.stdout: ], 2024-04-23T10:41:09.687 INFO:teuthology.orchestra.run.smithi114.stdout: "desc": "", 2024-04-23T10:41:09.687 INFO:teuthology.orchestra.run.smithi114.stdout: "long_desc": "", 2024-04-23T10:41:09.687 INFO:teuthology.orchestra.run.smithi114.stdout: "tags": [], 2024-04-23T10:41:09.687 INFO:teuthology.orchestra.run.smithi114.stdout: "see_also": [] 2024-04-23T10:41:09.687 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-23T10:41:09.687 INFO:teuthology.orchestra.run.smithi114.stdout: "log_to_file": { 2024-04-23T10:41:09.687 INFO:teuthology.orchestra.run.smithi114.stdout: "name": "log_to_file", 2024-04-23T10:41:09.687 INFO:teuthology.orchestra.run.smithi114.stdout: "type": "bool", 2024-04-23T10:41:09.688 INFO:teuthology.orchestra.run.smithi114.stdout: "level": "advanced", 2024-04-23T10:41:09.688 INFO:teuthology.orchestra.run.smithi114.stdout: "flags": 1, 2024-04-23T10:41:09.688 INFO:teuthology.orchestra.run.smithi114.stdout: "default_value": "False", 2024-04-23T10:41:09.688 INFO:teuthology.orchestra.run.smithi114.stdout: "min": "", 2024-04-23T10:41:09.688 INFO:teuthology.orchestra.run.smithi114.stdout: "max": "", 2024-04-23T10:41:09.688 INFO:teuthology.orchestra.run.smithi114.stdout: "enum_allowed": [], 2024-04-23T10:41:09.688 INFO:teuthology.orchestra.run.smithi114.stdout: "desc": "", 2024-04-23T10:41:09.688 INFO:teuthology.orchestra.run.smithi114.stdout: "long_desc": "", 2024-04-23T10:41:09.688 INFO:teuthology.orchestra.run.smithi114.stdout: "tags": [], 2024-04-23T10:41:09.689 INFO:teuthology.orchestra.run.smithi114.stdout: "see_also": [] 2024-04-23T10:41:09.689 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-23T10:41:09.689 INFO:teuthology.orchestra.run.smithi114.stdout: "motd": { 2024-04-23T10:41:09.689 INFO:teuthology.orchestra.run.smithi114.stdout: "name": "motd", 2024-04-23T10:41:09.689 INFO:teuthology.orchestra.run.smithi114.stdout: "type": "str", 2024-04-23T10:41:09.689 INFO:teuthology.orchestra.run.smithi114.stdout: "level": "advanced", 2024-04-23T10:41:09.689 INFO:teuthology.orchestra.run.smithi114.stdout: "flags": 0, 2024-04-23T10:41:09.689 INFO:teuthology.orchestra.run.smithi114.stdout: "default_value": "", 2024-04-23T10:41:09.689 INFO:teuthology.orchestra.run.smithi114.stdout: "min": "", 2024-04-23T10:41:09.690 INFO:teuthology.orchestra.run.smithi114.stdout: "max": "", 2024-04-23T10:41:09.690 INFO:teuthology.orchestra.run.smithi114.stdout: "enum_allowed": [], 2024-04-23T10:41:09.690 INFO:teuthology.orchestra.run.smithi114.stdout: "desc": "The message of the day", 2024-04-23T10:41:09.690 INFO:teuthology.orchestra.run.smithi114.stdout: "long_desc": "", 2024-04-23T10:41:09.690 INFO:teuthology.orchestra.run.smithi114.stdout: "tags": [], 2024-04-23T10:41:09.690 INFO:teuthology.orchestra.run.smithi114.stdout: "see_also": [] 2024-04-23T10:41:09.690 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-23T10:41:09.690 INFO:teuthology.orchestra.run.smithi114.stdout: "redirect_resolve_ip_addr": { 2024-04-23T10:41:09.690 INFO:teuthology.orchestra.run.smithi114.stdout: "name": "redirect_resolve_ip_addr", 2024-04-23T10:41:09.691 INFO:teuthology.orchestra.run.smithi114.stdout: "type": "bool", 2024-04-23T10:41:09.691 INFO:teuthology.orchestra.run.smithi114.stdout: "level": "advanced", 2024-04-23T10:41:09.691 INFO:teuthology.orchestra.run.smithi114.stdout: "flags": 0, 2024-04-23T10:41:09.691 INFO:teuthology.orchestra.run.smithi114.stdout: "default_value": "False", 2024-04-23T10:41:09.691 INFO:teuthology.orchestra.run.smithi114.stdout: "min": "", 2024-04-23T10:41:09.691 INFO:teuthology.orchestra.run.smithi114.stdout: "max": "", 2024-04-23T10:41:09.691 INFO:teuthology.orchestra.run.smithi114.stdout: "enum_allowed": [], 2024-04-23T10:41:09.691 INFO:teuthology.orchestra.run.smithi114.stdout: "desc": "", 2024-04-23T10:41:09.691 INFO:teuthology.orchestra.run.smithi114.stdout: "long_desc": "", 2024-04-23T10:41:09.692 INFO:teuthology.orchestra.run.smithi114.stdout: "tags": [], 2024-04-23T10:41:09.692 INFO:teuthology.orchestra.run.smithi114.stdout: "see_also": [] 2024-04-23T10:41:09.692 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-23T10:41:09.692 INFO:teuthology.orchestra.run.smithi114.stdout: "server_addr": { 2024-04-23T10:41:09.692 INFO:teuthology.orchestra.run.smithi114.stdout: "name": "server_addr", 2024-04-23T10:41:09.692 INFO:teuthology.orchestra.run.smithi114.stdout: "type": "str", 2024-04-23T10:41:09.692 INFO:teuthology.orchestra.run.smithi114.stdout: "level": "advanced", 2024-04-23T10:41:09.692 INFO:teuthology.orchestra.run.smithi114.stdout: "flags": 0, 2024-04-23T10:41:09.692 INFO:teuthology.orchestra.run.smithi114.stdout: "default_value": "::", 2024-04-23T10:41:09.692 INFO:teuthology.orchestra.run.smithi114.stdout: "min": "", 2024-04-23T10:41:09.693 INFO:teuthology.orchestra.run.smithi114.stdout: "max": "", 2024-04-23T10:41:09.693 INFO:teuthology.orchestra.run.smithi114.stdout: "enum_allowed": [], 2024-04-23T10:41:09.693 INFO:teuthology.orchestra.run.smithi114.stdout: "desc": "", 2024-04-23T10:41:09.693 INFO:teuthology.orchestra.run.smithi114.stdout: "long_desc": "", 2024-04-23T10:41:09.693 INFO:teuthology.orchestra.run.smithi114.stdout: "tags": [], 2024-04-23T10:41:09.693 INFO:teuthology.orchestra.run.smithi114.stdout: "see_also": [] 2024-04-23T10:41:09.693 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-23T10:41:09.693 INFO:teuthology.orchestra.run.smithi114.stdout: "server_port": { 2024-04-23T10:41:09.693 INFO:teuthology.orchestra.run.smithi114.stdout: "name": "server_port", 2024-04-23T10:41:09.694 INFO:teuthology.orchestra.run.smithi114.stdout: "type": "int", 2024-04-23T10:41:09.694 INFO:teuthology.orchestra.run.smithi114.stdout: "level": "advanced", 2024-04-23T10:41:09.694 INFO:teuthology.orchestra.run.smithi114.stdout: "flags": 0, 2024-04-23T10:41:09.694 INFO:teuthology.orchestra.run.smithi114.stdout: "default_value": "8080", 2024-04-23T10:41:09.694 INFO:teuthology.orchestra.run.smithi114.stdout: "min": "", 2024-04-23T10:41:09.694 INFO:teuthology.orchestra.run.smithi114.stdout: "max": "", 2024-04-23T10:41:09.694 INFO:teuthology.orchestra.run.smithi114.stdout: "enum_allowed": [], 2024-04-23T10:41:09.694 INFO:teuthology.orchestra.run.smithi114.stdout: "desc": "", 2024-04-23T10:41:09.694 INFO:teuthology.orchestra.run.smithi114.stdout: "long_desc": "", 2024-04-23T10:41:09.695 INFO:teuthology.orchestra.run.smithi114.stdout: "tags": [], 2024-04-23T10:41:09.695 INFO:teuthology.orchestra.run.smithi114.stdout: "see_also": [] 2024-04-23T10:41:09.695 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-23T10:41:09.695 INFO:teuthology.orchestra.run.smithi114.stdout: "ssl": { 2024-04-23T10:41:09.695 INFO:teuthology.orchestra.run.smithi114.stdout: "name": "ssl", 2024-04-23T10:41:09.695 INFO:teuthology.orchestra.run.smithi114.stdout: "type": "bool", 2024-04-23T10:41:09.695 INFO:teuthology.orchestra.run.smithi114.stdout: "level": "advanced", 2024-04-23T10:41:09.695 INFO:teuthology.orchestra.run.smithi114.stdout: "flags": 0, 2024-04-23T10:41:09.695 INFO:teuthology.orchestra.run.smithi114.stdout: "default_value": "True", 2024-04-23T10:41:09.696 INFO:teuthology.orchestra.run.smithi114.stdout: "min": "", 2024-04-23T10:41:09.696 INFO:teuthology.orchestra.run.smithi114.stdout: "max": "", 2024-04-23T10:41:09.696 INFO:teuthology.orchestra.run.smithi114.stdout: "enum_allowed": [], 2024-04-23T10:41:09.696 INFO:teuthology.orchestra.run.smithi114.stdout: "desc": "", 2024-04-23T10:41:09.696 INFO:teuthology.orchestra.run.smithi114.stdout: "long_desc": "", 2024-04-23T10:41:09.696 INFO:teuthology.orchestra.run.smithi114.stdout: "tags": [], 2024-04-23T10:41:09.696 INFO:teuthology.orchestra.run.smithi114.stdout: "see_also": [] 2024-04-23T10:41:09.696 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-23T10:41:09.696 INFO:teuthology.orchestra.run.smithi114.stdout: "ssl_server_port": { 2024-04-23T10:41:09.697 INFO:teuthology.orchestra.run.smithi114.stdout: "name": "ssl_server_port", 2024-04-23T10:41:09.697 INFO:teuthology.orchestra.run.smithi114.stdout: "type": "int", 2024-04-23T10:41:09.697 INFO:teuthology.orchestra.run.smithi114.stdout: "level": "advanced", 2024-04-23T10:41:09.697 INFO:teuthology.orchestra.run.smithi114.stdout: "flags": 0, 2024-04-23T10:41:09.697 INFO:teuthology.orchestra.run.smithi114.stdout: "default_value": "8443", 2024-04-23T10:41:09.697 INFO:teuthology.orchestra.run.smithi114.stdout: "min": "", 2024-04-23T10:41:09.697 INFO:teuthology.orchestra.run.smithi114.stdout: "max": "", 2024-04-23T10:41:09.697 INFO:teuthology.orchestra.run.smithi114.stdout: "enum_allowed": [], 2024-04-23T10:41:09.697 INFO:teuthology.orchestra.run.smithi114.stdout: "desc": "", 2024-04-23T10:41:09.698 INFO:teuthology.orchestra.run.smithi114.stdout: "long_desc": "", 2024-04-23T10:41:09.698 INFO:teuthology.orchestra.run.smithi114.stdout: "tags": [], 2024-04-23T10:41:09.698 INFO:teuthology.orchestra.run.smithi114.stdout: "see_also": [] 2024-04-23T10:41:09.698 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-23T10:41:09.698 INFO:teuthology.orchestra.run.smithi114.stdout: "standby_behaviour": { 2024-04-23T10:41:09.698 INFO:teuthology.orchestra.run.smithi114.stdout: "name": "standby_behaviour", 2024-04-23T10:41:09.698 INFO:teuthology.orchestra.run.smithi114.stdout: "type": "str", 2024-04-23T10:41:09.698 INFO:teuthology.orchestra.run.smithi114.stdout: "level": "advanced", 2024-04-23T10:41:09.698 INFO:teuthology.orchestra.run.smithi114.stdout: "flags": 0, 2024-04-23T10:41:09.698 INFO:teuthology.orchestra.run.smithi114.stdout: "default_value": "redirect", 2024-04-23T10:41:09.699 INFO:teuthology.orchestra.run.smithi114.stdout: "min": "", 2024-04-23T10:41:09.699 INFO:teuthology.orchestra.run.smithi114.stdout: "max": "", 2024-04-23T10:41:09.699 INFO:teuthology.orchestra.run.smithi114.stdout: "enum_allowed": [ 2024-04-23T10:41:09.699 INFO:teuthology.orchestra.run.smithi114.stdout: "error", 2024-04-23T10:41:09.699 INFO:teuthology.orchestra.run.smithi114.stdout: "redirect" 2024-04-23T10:41:09.699 INFO:teuthology.orchestra.run.smithi114.stdout: ], 2024-04-23T10:41:09.699 INFO:teuthology.orchestra.run.smithi114.stdout: "desc": "", 2024-04-23T10:41:09.699 INFO:teuthology.orchestra.run.smithi114.stdout: "long_desc": "", 2024-04-23T10:41:09.699 INFO:teuthology.orchestra.run.smithi114.stdout: "tags": [], 2024-04-23T10:41:09.699 INFO:teuthology.orchestra.run.smithi114.stdout: "see_also": [] 2024-04-23T10:41:09.700 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-23T10:41:09.700 INFO:teuthology.orchestra.run.smithi114.stdout: "standby_error_status_code": { 2024-04-23T10:41:09.700 INFO:teuthology.orchestra.run.smithi114.stdout: "name": "standby_error_status_code", 2024-04-23T10:41:09.700 INFO:teuthology.orchestra.run.smithi114.stdout: "type": "int", 2024-04-23T10:41:09.700 INFO:teuthology.orchestra.run.smithi114.stdout: "level": "advanced", 2024-04-23T10:41:09.700 INFO:teuthology.orchestra.run.smithi114.stdout: "flags": 0, 2024-04-23T10:41:09.700 INFO:teuthology.orchestra.run.smithi114.stdout: "default_value": "500", 2024-04-23T10:41:09.700 INFO:teuthology.orchestra.run.smithi114.stdout: "min": "400", 2024-04-23T10:41:09.700 INFO:teuthology.orchestra.run.smithi114.stdout: "max": "599", 2024-04-23T10:41:09.701 INFO:teuthology.orchestra.run.smithi114.stdout: "enum_allowed": [], 2024-04-23T10:41:09.701 INFO:teuthology.orchestra.run.smithi114.stdout: "desc": "", 2024-04-23T10:41:09.701 INFO:teuthology.orchestra.run.smithi114.stdout: "long_desc": "", 2024-04-23T10:41:09.701 INFO:teuthology.orchestra.run.smithi114.stdout: "tags": [], 2024-04-23T10:41:09.701 INFO:teuthology.orchestra.run.smithi114.stdout: "see_also": [] 2024-04-23T10:41:09.701 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-23T10:41:09.701 INFO:teuthology.orchestra.run.smithi114.stdout: "url_prefix": { 2024-04-23T10:41:09.701 INFO:teuthology.orchestra.run.smithi114.stdout: "name": "url_prefix", 2024-04-23T10:41:09.701 INFO:teuthology.orchestra.run.smithi114.stdout: "type": "str", 2024-04-23T10:41:09.701 INFO:teuthology.orchestra.run.smithi114.stdout: "level": "advanced", 2024-04-23T10:41:09.702 INFO:teuthology.orchestra.run.smithi114.stdout: "flags": 0, 2024-04-23T10:41:09.702 INFO:teuthology.orchestra.run.smithi114.stdout: "default_value": "", 2024-04-23T10:41:09.702 INFO:teuthology.orchestra.run.smithi114.stdout: "min": "", 2024-04-23T10:41:09.702 INFO:teuthology.orchestra.run.smithi114.stdout: "max": "", 2024-04-23T10:41:09.702 INFO:teuthology.orchestra.run.smithi114.stdout: "enum_allowed": [], 2024-04-23T10:41:09.702 INFO:teuthology.orchestra.run.smithi114.stdout: "desc": "", 2024-04-23T10:41:09.702 INFO:teuthology.orchestra.run.smithi114.stdout: "long_desc": "", 2024-04-23T10:41:09.702 INFO:teuthology.orchestra.run.smithi114.stdout: "tags": [], 2024-04-23T10:41:09.702 INFO:teuthology.orchestra.run.smithi114.stdout: "see_also": [] 2024-04-23T10:41:09.702 INFO:teuthology.orchestra.run.smithi114.stdout: } 2024-04-23T10:41:09.703 INFO:teuthology.orchestra.run.smithi114.stdout: } 2024-04-23T10:41:09.703 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-23T10:41:09.703 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-04-23T10:41:09.703 INFO:teuthology.orchestra.run.smithi114.stdout: "name": "devicehealth", 2024-04-23T10:41:09.703 INFO:teuthology.orchestra.run.smithi114.stdout: "can_run": true, 2024-04-23T10:41:09.703 INFO:teuthology.orchestra.run.smithi114.stdout: "error_string": "", 2024-04-23T10:41:09.703 INFO:teuthology.orchestra.run.smithi114.stdout: "module_options": { 2024-04-23T10:41:09.703 INFO:teuthology.orchestra.run.smithi114.stdout: "enable_monitoring": { 2024-04-23T10:41:09.703 INFO:teuthology.orchestra.run.smithi114.stdout: "name": "enable_monitoring", 2024-04-23T10:41:09.703 INFO:teuthology.orchestra.run.smithi114.stdout: "type": "bool", 2024-04-23T10:41:09.704 INFO:teuthology.orchestra.run.smithi114.stdout: "level": "advanced", 2024-04-23T10:41:09.704 INFO:teuthology.orchestra.run.smithi114.stdout: "flags": 1, 2024-04-23T10:41:09.704 INFO:teuthology.orchestra.run.smithi114.stdout: "default_value": "True", 2024-04-23T10:41:09.704 INFO:teuthology.orchestra.run.smithi114.stdout: "min": "", 2024-04-23T10:41:09.704 INFO:teuthology.orchestra.run.smithi114.stdout: "max": "", 2024-04-23T10:41:09.704 INFO:teuthology.orchestra.run.smithi114.stdout: "enum_allowed": [], 2024-04-23T10:41:09.704 INFO:teuthology.orchestra.run.smithi114.stdout: "desc": "monitor device health metrics", 2024-04-23T10:41:09.704 INFO:teuthology.orchestra.run.smithi114.stdout: "long_desc": "", 2024-04-23T10:41:09.704 INFO:teuthology.orchestra.run.smithi114.stdout: "tags": [], 2024-04-23T10:41:09.704 INFO:teuthology.orchestra.run.smithi114.stdout: "see_also": [] 2024-04-23T10:41:09.705 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-23T10:41:09.705 INFO:teuthology.orchestra.run.smithi114.stdout: "log_level": { 2024-04-23T10:41:09.705 INFO:teuthology.orchestra.run.smithi114.stdout: "name": "log_level", 2024-04-23T10:41:09.705 INFO:teuthology.orchestra.run.smithi114.stdout: "type": "str", 2024-04-23T10:41:09.705 INFO:teuthology.orchestra.run.smithi114.stdout: "level": "advanced", 2024-04-23T10:41:09.705 INFO:teuthology.orchestra.run.smithi114.stdout: "flags": 1, 2024-04-23T10:41:09.705 INFO:teuthology.orchestra.run.smithi114.stdout: "default_value": "", 2024-04-23T10:41:09.705 INFO:teuthology.orchestra.run.smithi114.stdout: "min": "", 2024-04-23T10:41:09.705 INFO:teuthology.orchestra.run.smithi114.stdout: "max": "", 2024-04-23T10:41:09.705 INFO:teuthology.orchestra.run.smithi114.stdout: "enum_allowed": [ 2024-04-23T10:41:09.706 INFO:teuthology.orchestra.run.smithi114.stdout: "", 2024-04-23T10:41:09.706 INFO:teuthology.orchestra.run.smithi114.stdout: "critical", 2024-04-23T10:41:09.706 INFO:teuthology.orchestra.run.smithi114.stdout: "debug", 2024-04-23T10:41:09.706 INFO:teuthology.orchestra.run.smithi114.stdout: "error", 2024-04-23T10:41:09.706 INFO:teuthology.orchestra.run.smithi114.stdout: "info", 2024-04-23T10:41:09.706 INFO:teuthology.orchestra.run.smithi114.stdout: "warning" 2024-04-23T10:41:09.706 INFO:teuthology.orchestra.run.smithi114.stdout: ], 2024-04-23T10:41:09.706 INFO:teuthology.orchestra.run.smithi114.stdout: "desc": "", 2024-04-23T10:41:09.706 INFO:teuthology.orchestra.run.smithi114.stdout: "long_desc": "", 2024-04-23T10:41:09.707 INFO:teuthology.orchestra.run.smithi114.stdout: "tags": [], 2024-04-23T10:41:09.707 INFO:teuthology.orchestra.run.smithi114.stdout: "see_also": [] 2024-04-23T10:41:09.707 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-23T10:41:09.707 INFO:teuthology.orchestra.run.smithi114.stdout: "log_to_cluster": { 2024-04-23T10:41:09.707 INFO:teuthology.orchestra.run.smithi114.stdout: "name": "log_to_cluster", 2024-04-23T10:41:09.707 INFO:teuthology.orchestra.run.smithi114.stdout: "type": "bool", 2024-04-23T10:41:09.707 INFO:teuthology.orchestra.run.smithi114.stdout: "level": "advanced", 2024-04-23T10:41:09.707 INFO:teuthology.orchestra.run.smithi114.stdout: "flags": 1, 2024-04-23T10:41:09.707 INFO:teuthology.orchestra.run.smithi114.stdout: "default_value": "False", 2024-04-23T10:41:09.708 INFO:teuthology.orchestra.run.smithi114.stdout: "min": "", 2024-04-23T10:41:09.708 INFO:teuthology.orchestra.run.smithi114.stdout: "max": "", 2024-04-23T10:41:09.708 INFO:teuthology.orchestra.run.smithi114.stdout: "enum_allowed": [], 2024-04-23T10:41:09.708 INFO:teuthology.orchestra.run.smithi114.stdout: "desc": "", 2024-04-23T10:41:09.708 INFO:teuthology.orchestra.run.smithi114.stdout: "long_desc": "", 2024-04-23T10:41:09.708 INFO:teuthology.orchestra.run.smithi114.stdout: "tags": [], 2024-04-23T10:41:09.708 INFO:teuthology.orchestra.run.smithi114.stdout: "see_also": [] 2024-04-23T10:41:09.708 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-23T10:41:09.708 INFO:teuthology.orchestra.run.smithi114.stdout: "log_to_cluster_level": { 2024-04-23T10:41:09.708 INFO:teuthology.orchestra.run.smithi114.stdout: "name": "log_to_cluster_level", 2024-04-23T10:41:09.709 INFO:teuthology.orchestra.run.smithi114.stdout: "type": "str", 2024-04-23T10:41:09.709 INFO:teuthology.orchestra.run.smithi114.stdout: "level": "advanced", 2024-04-23T10:41:09.709 INFO:teuthology.orchestra.run.smithi114.stdout: "flags": 1, 2024-04-23T10:41:09.709 INFO:teuthology.orchestra.run.smithi114.stdout: "default_value": "info", 2024-04-23T10:41:09.709 INFO:teuthology.orchestra.run.smithi114.stdout: "min": "", 2024-04-23T10:41:09.709 INFO:teuthology.orchestra.run.smithi114.stdout: "max": "", 2024-04-23T10:41:09.709 INFO:teuthology.orchestra.run.smithi114.stdout: "enum_allowed": [ 2024-04-23T10:41:09.709 INFO:teuthology.orchestra.run.smithi114.stdout: "", 2024-04-23T10:41:09.709 INFO:teuthology.orchestra.run.smithi114.stdout: "critical", 2024-04-23T10:41:09.709 INFO:teuthology.orchestra.run.smithi114.stdout: "debug", 2024-04-23T10:41:09.710 INFO:teuthology.orchestra.run.smithi114.stdout: "error", 2024-04-23T10:41:09.710 INFO:teuthology.orchestra.run.smithi114.stdout: "info", 2024-04-23T10:41:09.710 INFO:teuthology.orchestra.run.smithi114.stdout: "warning" 2024-04-23T10:41:09.710 INFO:teuthology.orchestra.run.smithi114.stdout: ], 2024-04-23T10:41:09.710 INFO:teuthology.orchestra.run.smithi114.stdout: "desc": "", 2024-04-23T10:41:09.710 INFO:teuthology.orchestra.run.smithi114.stdout: "long_desc": "", 2024-04-23T10:41:09.710 INFO:teuthology.orchestra.run.smithi114.stdout: "tags": [], 2024-04-23T10:41:09.710 INFO:teuthology.orchestra.run.smithi114.stdout: "see_also": [] 2024-04-23T10:41:09.710 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-23T10:41:09.711 INFO:teuthology.orchestra.run.smithi114.stdout: "log_to_file": { 2024-04-23T10:41:09.711 INFO:teuthology.orchestra.run.smithi114.stdout: "name": "log_to_file", 2024-04-23T10:41:09.711 INFO:teuthology.orchestra.run.smithi114.stdout: "type": "bool", 2024-04-23T10:41:09.711 INFO:teuthology.orchestra.run.smithi114.stdout: "level": "advanced", 2024-04-23T10:41:09.711 INFO:teuthology.orchestra.run.smithi114.stdout: "flags": 1, 2024-04-23T10:41:09.711 INFO:teuthology.orchestra.run.smithi114.stdout: "default_value": "False", 2024-04-23T10:41:09.711 INFO:teuthology.orchestra.run.smithi114.stdout: "min": "", 2024-04-23T10:41:09.711 INFO:teuthology.orchestra.run.smithi114.stdout: "max": "", 2024-04-23T10:41:09.711 INFO:teuthology.orchestra.run.smithi114.stdout: "enum_allowed": [], 2024-04-23T10:41:09.711 INFO:teuthology.orchestra.run.smithi114.stdout: "desc": "", 2024-04-23T10:41:09.712 INFO:teuthology.orchestra.run.smithi114.stdout: "long_desc": "", 2024-04-23T10:41:09.712 INFO:teuthology.orchestra.run.smithi114.stdout: "tags": [], 2024-04-23T10:41:09.712 INFO:teuthology.orchestra.run.smithi114.stdout: "see_also": [] 2024-04-23T10:41:09.712 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-23T10:41:09.712 INFO:teuthology.orchestra.run.smithi114.stdout: "mark_out_threshold": { 2024-04-23T10:41:09.712 INFO:teuthology.orchestra.run.smithi114.stdout: "name": "mark_out_threshold", 2024-04-23T10:41:09.712 INFO:teuthology.orchestra.run.smithi114.stdout: "type": "secs", 2024-04-23T10:41:09.712 INFO:teuthology.orchestra.run.smithi114.stdout: "level": "advanced", 2024-04-23T10:41:09.712 INFO:teuthology.orchestra.run.smithi114.stdout: "flags": 1, 2024-04-23T10:41:09.712 INFO:teuthology.orchestra.run.smithi114.stdout: "default_value": "2419200", 2024-04-23T10:41:09.713 INFO:teuthology.orchestra.run.smithi114.stdout: "min": "", 2024-04-23T10:41:09.713 INFO:teuthology.orchestra.run.smithi114.stdout: "max": "", 2024-04-23T10:41:09.713 INFO:teuthology.orchestra.run.smithi114.stdout: "enum_allowed": [], 2024-04-23T10:41:09.713 INFO:teuthology.orchestra.run.smithi114.stdout: "desc": "automatically mark OSD if it may fail before this long", 2024-04-23T10:41:09.713 INFO:teuthology.orchestra.run.smithi114.stdout: "long_desc": "", 2024-04-23T10:41:09.713 INFO:teuthology.orchestra.run.smithi114.stdout: "tags": [], 2024-04-23T10:41:09.713 INFO:teuthology.orchestra.run.smithi114.stdout: "see_also": [] 2024-04-23T10:41:09.713 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-23T10:41:09.713 INFO:teuthology.orchestra.run.smithi114.stdout: "pool_name": { 2024-04-23T10:41:09.713 INFO:teuthology.orchestra.run.smithi114.stdout: "name": "pool_name", 2024-04-23T10:41:09.714 INFO:teuthology.orchestra.run.smithi114.stdout: "type": "str", 2024-04-23T10:41:09.714 INFO:teuthology.orchestra.run.smithi114.stdout: "level": "advanced", 2024-04-23T10:41:09.714 INFO:teuthology.orchestra.run.smithi114.stdout: "flags": 1, 2024-04-23T10:41:09.714 INFO:teuthology.orchestra.run.smithi114.stdout: "default_value": "device_health_metrics", 2024-04-23T10:41:09.714 INFO:teuthology.orchestra.run.smithi114.stdout: "min": "", 2024-04-23T10:41:09.714 INFO:teuthology.orchestra.run.smithi114.stdout: "max": "", 2024-04-23T10:41:09.714 INFO:teuthology.orchestra.run.smithi114.stdout: "enum_allowed": [], 2024-04-23T10:41:09.714 INFO:teuthology.orchestra.run.smithi114.stdout: "desc": "name of pool in which to store device health metrics", 2024-04-23T10:41:09.714 INFO:teuthology.orchestra.run.smithi114.stdout: "long_desc": "", 2024-04-23T10:41:09.714 INFO:teuthology.orchestra.run.smithi114.stdout: "tags": [], 2024-04-23T10:41:09.715 INFO:teuthology.orchestra.run.smithi114.stdout: "see_also": [] 2024-04-23T10:41:09.715 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-23T10:41:09.715 INFO:teuthology.orchestra.run.smithi114.stdout: "retention_period": { 2024-04-23T10:41:09.715 INFO:teuthology.orchestra.run.smithi114.stdout: "name": "retention_period", 2024-04-23T10:41:09.715 INFO:teuthology.orchestra.run.smithi114.stdout: "type": "secs", 2024-04-23T10:41:09.715 INFO:teuthology.orchestra.run.smithi114.stdout: "level": "advanced", 2024-04-23T10:41:09.715 INFO:teuthology.orchestra.run.smithi114.stdout: "flags": 1, 2024-04-23T10:41:09.715 INFO:teuthology.orchestra.run.smithi114.stdout: "default_value": "15552000", 2024-04-23T10:41:09.715 INFO:teuthology.orchestra.run.smithi114.stdout: "min": "", 2024-04-23T10:41:09.716 INFO:teuthology.orchestra.run.smithi114.stdout: "max": "", 2024-04-23T10:41:09.716 INFO:teuthology.orchestra.run.smithi114.stdout: "enum_allowed": [], 2024-04-23T10:41:09.716 INFO:teuthology.orchestra.run.smithi114.stdout: "desc": "how long to retain device health metrics", 2024-04-23T10:41:09.716 INFO:teuthology.orchestra.run.smithi114.stdout: "long_desc": "", 2024-04-23T10:41:09.716 INFO:teuthology.orchestra.run.smithi114.stdout: "tags": [], 2024-04-23T10:41:09.716 INFO:teuthology.orchestra.run.smithi114.stdout: "see_also": [] 2024-04-23T10:41:09.716 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-23T10:41:09.717 INFO:teuthology.orchestra.run.smithi114.stdout: "scrape_frequency": { 2024-04-23T10:41:09.717 INFO:teuthology.orchestra.run.smithi114.stdout: "name": "scrape_frequency", 2024-04-23T10:41:09.717 INFO:teuthology.orchestra.run.smithi114.stdout: "type": "secs", 2024-04-23T10:41:09.717 INFO:teuthology.orchestra.run.smithi114.stdout: "level": "advanced", 2024-04-23T10:41:09.717 INFO:teuthology.orchestra.run.smithi114.stdout: "flags": 1, 2024-04-23T10:41:09.717 INFO:teuthology.orchestra.run.smithi114.stdout: "default_value": "86400", 2024-04-23T10:41:09.717 INFO:teuthology.orchestra.run.smithi114.stdout: "min": "", 2024-04-23T10:41:09.717 INFO:teuthology.orchestra.run.smithi114.stdout: "max": "", 2024-04-23T10:41:09.717 INFO:teuthology.orchestra.run.smithi114.stdout: "enum_allowed": [], 2024-04-23T10:41:09.717 INFO:teuthology.orchestra.run.smithi114.stdout: "desc": "how frequently to scrape device health metrics", 2024-04-23T10:41:09.718 INFO:teuthology.orchestra.run.smithi114.stdout: "long_desc": "", 2024-04-23T10:41:09.718 INFO:teuthology.orchestra.run.smithi114.stdout: "tags": [], 2024-04-23T10:41:09.718 INFO:teuthology.orchestra.run.smithi114.stdout: "see_also": [] 2024-04-23T10:41:09.718 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-23T10:41:09.718 INFO:teuthology.orchestra.run.smithi114.stdout: "self_heal": { 2024-04-23T10:41:09.718 INFO:teuthology.orchestra.run.smithi114.stdout: "name": "self_heal", 2024-04-23T10:41:09.718 INFO:teuthology.orchestra.run.smithi114.stdout: "type": "bool", 2024-04-23T10:41:09.718 INFO:teuthology.orchestra.run.smithi114.stdout: "level": "advanced", 2024-04-23T10:41:09.718 INFO:teuthology.orchestra.run.smithi114.stdout: "flags": 1, 2024-04-23T10:41:09.719 INFO:teuthology.orchestra.run.smithi114.stdout: "default_value": "True", 2024-04-23T10:41:09.719 INFO:teuthology.orchestra.run.smithi114.stdout: "min": "", 2024-04-23T10:41:09.719 INFO:teuthology.orchestra.run.smithi114.stdout: "max": "", 2024-04-23T10:41:09.719 INFO:teuthology.orchestra.run.smithi114.stdout: "enum_allowed": [], 2024-04-23T10:41:09.719 INFO:teuthology.orchestra.run.smithi114.stdout: "desc": "preemptively heal cluster around devices that may fail", 2024-04-23T10:41:09.719 INFO:teuthology.orchestra.run.smithi114.stdout: "long_desc": "", 2024-04-23T10:41:09.719 INFO:teuthology.orchestra.run.smithi114.stdout: "tags": [], 2024-04-23T10:41:09.719 INFO:teuthology.orchestra.run.smithi114.stdout: "see_also": [] 2024-04-23T10:41:09.719 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-23T10:41:09.719 INFO:teuthology.orchestra.run.smithi114.stdout: "sleep_interval": { 2024-04-23T10:41:09.720 INFO:teuthology.orchestra.run.smithi114.stdout: "name": "sleep_interval", 2024-04-23T10:41:09.720 INFO:teuthology.orchestra.run.smithi114.stdout: "type": "secs", 2024-04-23T10:41:09.720 INFO:teuthology.orchestra.run.smithi114.stdout: "level": "advanced", 2024-04-23T10:41:09.720 INFO:teuthology.orchestra.run.smithi114.stdout: "flags": 1, 2024-04-23T10:41:09.720 INFO:teuthology.orchestra.run.smithi114.stdout: "default_value": "600", 2024-04-23T10:41:09.720 INFO:teuthology.orchestra.run.smithi114.stdout: "min": "", 2024-04-23T10:41:09.720 INFO:teuthology.orchestra.run.smithi114.stdout: "max": "", 2024-04-23T10:41:09.720 INFO:teuthology.orchestra.run.smithi114.stdout: "enum_allowed": [], 2024-04-23T10:41:09.720 INFO:teuthology.orchestra.run.smithi114.stdout: "desc": "how frequently to wake up and check device health", 2024-04-23T10:41:09.720 INFO:teuthology.orchestra.run.smithi114.stdout: "long_desc": "", 2024-04-23T10:41:09.721 INFO:teuthology.orchestra.run.smithi114.stdout: "tags": [], 2024-04-23T10:41:09.721 INFO:teuthology.orchestra.run.smithi114.stdout: "see_also": [] 2024-04-23T10:41:09.721 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-23T10:41:09.721 INFO:teuthology.orchestra.run.smithi114.stdout: "warn_threshold": { 2024-04-23T10:41:09.721 INFO:teuthology.orchestra.run.smithi114.stdout: "name": "warn_threshold", 2024-04-23T10:41:09.721 INFO:teuthology.orchestra.run.smithi114.stdout: "type": "secs", 2024-04-23T10:41:09.721 INFO:teuthology.orchestra.run.smithi114.stdout: "level": "advanced", 2024-04-23T10:41:09.721 INFO:teuthology.orchestra.run.smithi114.stdout: "flags": 1, 2024-04-23T10:41:09.721 INFO:teuthology.orchestra.run.smithi114.stdout: "default_value": "7257600", 2024-04-23T10:41:09.722 INFO:teuthology.orchestra.run.smithi114.stdout: "min": "", 2024-04-23T10:41:09.722 INFO:teuthology.orchestra.run.smithi114.stdout: "max": "", 2024-04-23T10:41:09.722 INFO:teuthology.orchestra.run.smithi114.stdout: "enum_allowed": [], 2024-04-23T10:41:09.722 INFO:teuthology.orchestra.run.smithi114.stdout: "desc": "raise health warning if OSD may fail before this long", 2024-04-23T10:41:09.722 INFO:teuthology.orchestra.run.smithi114.stdout: "long_desc": "", 2024-04-23T10:41:09.722 INFO:teuthology.orchestra.run.smithi114.stdout: "tags": [], 2024-04-23T10:41:09.722 INFO:teuthology.orchestra.run.smithi114.stdout: "see_also": [] 2024-04-23T10:41:09.722 INFO:teuthology.orchestra.run.smithi114.stdout: } 2024-04-23T10:41:09.722 INFO:teuthology.orchestra.run.smithi114.stdout: } 2024-04-23T10:41:09.723 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-23T10:41:09.723 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-04-23T10:41:09.723 INFO:teuthology.orchestra.run.smithi114.stdout: "name": "diskprediction_local", 2024-04-23T10:41:09.723 INFO:teuthology.orchestra.run.smithi114.stdout: "can_run": true, 2024-04-23T10:41:09.723 INFO:teuthology.orchestra.run.smithi114.stdout: "error_string": "", 2024-04-23T10:41:09.723 INFO:teuthology.orchestra.run.smithi114.stdout: "module_options": { 2024-04-23T10:41:09.723 INFO:teuthology.orchestra.run.smithi114.stdout: "log_level": { 2024-04-23T10:41:09.723 INFO:teuthology.orchestra.run.smithi114.stdout: "name": "log_level", 2024-04-23T10:41:09.723 INFO:teuthology.orchestra.run.smithi114.stdout: "type": "str", 2024-04-23T10:41:09.723 INFO:teuthology.orchestra.run.smithi114.stdout: "level": "advanced", 2024-04-23T10:41:09.724 INFO:teuthology.orchestra.run.smithi114.stdout: "flags": 1, 2024-04-23T10:41:09.724 INFO:teuthology.orchestra.run.smithi114.stdout: "default_value": "", 2024-04-23T10:41:09.724 INFO:teuthology.orchestra.run.smithi114.stdout: "min": "", 2024-04-23T10:41:09.724 INFO:teuthology.orchestra.run.smithi114.stdout: "max": "", 2024-04-23T10:41:09.724 INFO:teuthology.orchestra.run.smithi114.stdout: "enum_allowed": [ 2024-04-23T10:41:09.724 INFO:teuthology.orchestra.run.smithi114.stdout: "", 2024-04-23T10:41:09.724 INFO:teuthology.orchestra.run.smithi114.stdout: "critical", 2024-04-23T10:41:09.724 INFO:teuthology.orchestra.run.smithi114.stdout: "debug", 2024-04-23T10:41:09.724 INFO:teuthology.orchestra.run.smithi114.stdout: "error", 2024-04-23T10:41:09.724 INFO:teuthology.orchestra.run.smithi114.stdout: "info", 2024-04-23T10:41:09.725 INFO:teuthology.orchestra.run.smithi114.stdout: "warning" 2024-04-23T10:41:09.725 INFO:teuthology.orchestra.run.smithi114.stdout: ], 2024-04-23T10:41:09.725 INFO:teuthology.orchestra.run.smithi114.stdout: "desc": "", 2024-04-23T10:41:09.725 INFO:teuthology.orchestra.run.smithi114.stdout: "long_desc": "", 2024-04-23T10:41:09.725 INFO:teuthology.orchestra.run.smithi114.stdout: "tags": [], 2024-04-23T10:41:09.725 INFO:teuthology.orchestra.run.smithi114.stdout: "see_also": [] 2024-04-23T10:41:09.725 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-23T10:41:09.725 INFO:teuthology.orchestra.run.smithi114.stdout: "log_to_cluster": { 2024-04-23T10:41:09.725 INFO:teuthology.orchestra.run.smithi114.stdout: "name": "log_to_cluster", 2024-04-23T10:41:09.725 INFO:teuthology.orchestra.run.smithi114.stdout: "type": "bool", 2024-04-23T10:41:09.726 INFO:teuthology.orchestra.run.smithi114.stdout: "level": "advanced", 2024-04-23T10:41:09.726 INFO:teuthology.orchestra.run.smithi114.stdout: "flags": 1, 2024-04-23T10:41:09.726 INFO:teuthology.orchestra.run.smithi114.stdout: "default_value": "False", 2024-04-23T10:41:09.726 INFO:teuthology.orchestra.run.smithi114.stdout: "min": "", 2024-04-23T10:41:09.726 INFO:teuthology.orchestra.run.smithi114.stdout: "max": "", 2024-04-23T10:41:09.726 INFO:teuthology.orchestra.run.smithi114.stdout: "enum_allowed": [], 2024-04-23T10:41:09.726 INFO:teuthology.orchestra.run.smithi114.stdout: "desc": "", 2024-04-23T10:41:09.726 INFO:teuthology.orchestra.run.smithi114.stdout: "long_desc": "", 2024-04-23T10:41:09.726 INFO:teuthology.orchestra.run.smithi114.stdout: "tags": [], 2024-04-23T10:41:09.727 INFO:teuthology.orchestra.run.smithi114.stdout: "see_also": [] 2024-04-23T10:41:09.727 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-23T10:41:09.727 INFO:teuthology.orchestra.run.smithi114.stdout: "log_to_cluster_level": { 2024-04-23T10:41:09.727 INFO:teuthology.orchestra.run.smithi114.stdout: "name": "log_to_cluster_level", 2024-04-23T10:41:09.727 INFO:teuthology.orchestra.run.smithi114.stdout: "type": "str", 2024-04-23T10:41:09.727 INFO:teuthology.orchestra.run.smithi114.stdout: "level": "advanced", 2024-04-23T10:41:09.727 INFO:teuthology.orchestra.run.smithi114.stdout: "flags": 1, 2024-04-23T10:41:09.727 INFO:teuthology.orchestra.run.smithi114.stdout: "default_value": "info", 2024-04-23T10:41:09.727 INFO:teuthology.orchestra.run.smithi114.stdout: "min": "", 2024-04-23T10:41:09.727 INFO:teuthology.orchestra.run.smithi114.stdout: "max": "", 2024-04-23T10:41:09.728 INFO:teuthology.orchestra.run.smithi114.stdout: "enum_allowed": [ 2024-04-23T10:41:09.728 INFO:teuthology.orchestra.run.smithi114.stdout: "", 2024-04-23T10:41:09.728 INFO:teuthology.orchestra.run.smithi114.stdout: "critical", 2024-04-23T10:41:09.728 INFO:teuthology.orchestra.run.smithi114.stdout: "debug", 2024-04-23T10:41:09.728 INFO:teuthology.orchestra.run.smithi114.stdout: "error", 2024-04-23T10:41:09.728 INFO:teuthology.orchestra.run.smithi114.stdout: "info", 2024-04-23T10:41:09.728 INFO:teuthology.orchestra.run.smithi114.stdout: "warning" 2024-04-23T10:41:09.728 INFO:teuthology.orchestra.run.smithi114.stdout: ], 2024-04-23T10:41:09.728 INFO:teuthology.orchestra.run.smithi114.stdout: "desc": "", 2024-04-23T10:41:09.728 INFO:teuthology.orchestra.run.smithi114.stdout: "long_desc": "", 2024-04-23T10:41:09.729 INFO:teuthology.orchestra.run.smithi114.stdout: "tags": [], 2024-04-23T10:41:09.729 INFO:teuthology.orchestra.run.smithi114.stdout: "see_also": [] 2024-04-23T10:41:09.729 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-23T10:41:09.729 INFO:teuthology.orchestra.run.smithi114.stdout: "log_to_file": { 2024-04-23T10:41:09.729 INFO:teuthology.orchestra.run.smithi114.stdout: "name": "log_to_file", 2024-04-23T10:41:09.729 INFO:teuthology.orchestra.run.smithi114.stdout: "type": "bool", 2024-04-23T10:41:09.729 INFO:teuthology.orchestra.run.smithi114.stdout: "level": "advanced", 2024-04-23T10:41:09.729 INFO:teuthology.orchestra.run.smithi114.stdout: "flags": 1, 2024-04-23T10:41:09.729 INFO:teuthology.orchestra.run.smithi114.stdout: "default_value": "False", 2024-04-23T10:41:09.729 INFO:teuthology.orchestra.run.smithi114.stdout: "min": "", 2024-04-23T10:41:09.730 INFO:teuthology.orchestra.run.smithi114.stdout: "max": "", 2024-04-23T10:41:09.730 INFO:teuthology.orchestra.run.smithi114.stdout: "enum_allowed": [], 2024-04-23T10:41:09.730 INFO:teuthology.orchestra.run.smithi114.stdout: "desc": "", 2024-04-23T10:41:09.730 INFO:teuthology.orchestra.run.smithi114.stdout: "long_desc": "", 2024-04-23T10:41:09.730 INFO:teuthology.orchestra.run.smithi114.stdout: "tags": [], 2024-04-23T10:41:09.730 INFO:teuthology.orchestra.run.smithi114.stdout: "see_also": [] 2024-04-23T10:41:09.730 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-23T10:41:09.730 INFO:teuthology.orchestra.run.smithi114.stdout: "predict_interval": { 2024-04-23T10:41:09.730 INFO:teuthology.orchestra.run.smithi114.stdout: "name": "predict_interval", 2024-04-23T10:41:09.731 INFO:teuthology.orchestra.run.smithi114.stdout: "type": "str", 2024-04-23T10:41:09.731 INFO:teuthology.orchestra.run.smithi114.stdout: "level": "advanced", 2024-04-23T10:41:09.731 INFO:teuthology.orchestra.run.smithi114.stdout: "flags": 0, 2024-04-23T10:41:09.731 INFO:teuthology.orchestra.run.smithi114.stdout: "default_value": "86400", 2024-04-23T10:41:09.731 INFO:teuthology.orchestra.run.smithi114.stdout: "min": "", 2024-04-23T10:41:09.731 INFO:teuthology.orchestra.run.smithi114.stdout: "max": "", 2024-04-23T10:41:09.731 INFO:teuthology.orchestra.run.smithi114.stdout: "enum_allowed": [], 2024-04-23T10:41:09.731 INFO:teuthology.orchestra.run.smithi114.stdout: "desc": "", 2024-04-23T10:41:09.731 INFO:teuthology.orchestra.run.smithi114.stdout: "long_desc": "", 2024-04-23T10:41:09.732 INFO:teuthology.orchestra.run.smithi114.stdout: "tags": [], 2024-04-23T10:41:09.732 INFO:teuthology.orchestra.run.smithi114.stdout: "see_also": [] 2024-04-23T10:41:09.732 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-23T10:41:09.732 INFO:teuthology.orchestra.run.smithi114.stdout: "predictor_model": { 2024-04-23T10:41:09.732 INFO:teuthology.orchestra.run.smithi114.stdout: "name": "predictor_model", 2024-04-23T10:41:09.732 INFO:teuthology.orchestra.run.smithi114.stdout: "type": "str", 2024-04-23T10:41:09.732 INFO:teuthology.orchestra.run.smithi114.stdout: "level": "advanced", 2024-04-23T10:41:09.732 INFO:teuthology.orchestra.run.smithi114.stdout: "flags": 0, 2024-04-23T10:41:09.732 INFO:teuthology.orchestra.run.smithi114.stdout: "default_value": "prophetstor", 2024-04-23T10:41:09.732 INFO:teuthology.orchestra.run.smithi114.stdout: "min": "", 2024-04-23T10:41:09.733 INFO:teuthology.orchestra.run.smithi114.stdout: "max": "", 2024-04-23T10:41:09.733 INFO:teuthology.orchestra.run.smithi114.stdout: "enum_allowed": [], 2024-04-23T10:41:09.733 INFO:teuthology.orchestra.run.smithi114.stdout: "desc": "", 2024-04-23T10:41:09.733 INFO:teuthology.orchestra.run.smithi114.stdout: "long_desc": "", 2024-04-23T10:41:09.733 INFO:teuthology.orchestra.run.smithi114.stdout: "tags": [], 2024-04-23T10:41:09.733 INFO:teuthology.orchestra.run.smithi114.stdout: "see_also": [] 2024-04-23T10:41:09.733 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-23T10:41:09.733 INFO:teuthology.orchestra.run.smithi114.stdout: "sleep_interval": { 2024-04-23T10:41:09.733 INFO:teuthology.orchestra.run.smithi114.stdout: "name": "sleep_interval", 2024-04-23T10:41:09.733 INFO:teuthology.orchestra.run.smithi114.stdout: "type": "str", 2024-04-23T10:41:09.734 INFO:teuthology.orchestra.run.smithi114.stdout: "level": "advanced", 2024-04-23T10:41:09.734 INFO:teuthology.orchestra.run.smithi114.stdout: "flags": 0, 2024-04-23T10:41:09.734 INFO:teuthology.orchestra.run.smithi114.stdout: "default_value": "600", 2024-04-23T10:41:09.734 INFO:teuthology.orchestra.run.smithi114.stdout: "min": "", 2024-04-23T10:41:09.734 INFO:teuthology.orchestra.run.smithi114.stdout: "max": "", 2024-04-23T10:41:09.734 INFO:teuthology.orchestra.run.smithi114.stdout: "enum_allowed": [], 2024-04-23T10:41:09.734 INFO:teuthology.orchestra.run.smithi114.stdout: "desc": "", 2024-04-23T10:41:09.734 INFO:teuthology.orchestra.run.smithi114.stdout: "long_desc": "", 2024-04-23T10:41:09.734 INFO:teuthology.orchestra.run.smithi114.stdout: "tags": [], 2024-04-23T10:41:09.735 INFO:teuthology.orchestra.run.smithi114.stdout: "see_also": [] 2024-04-23T10:41:09.735 INFO:teuthology.orchestra.run.smithi114.stdout: } 2024-04-23T10:41:09.735 INFO:teuthology.orchestra.run.smithi114.stdout: } 2024-04-23T10:41:09.735 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-23T10:41:09.735 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-04-23T10:41:09.735 INFO:teuthology.orchestra.run.smithi114.stdout: "name": "influx", 2024-04-23T10:41:09.735 INFO:teuthology.orchestra.run.smithi114.stdout: "can_run": false, 2024-04-23T10:41:09.735 INFO:teuthology.orchestra.run.smithi114.stdout: "error_string": "influxdb python module not found", 2024-04-23T10:41:09.735 INFO:teuthology.orchestra.run.smithi114.stdout: "module_options": { 2024-04-23T10:41:09.735 INFO:teuthology.orchestra.run.smithi114.stdout: "batch_size": { 2024-04-23T10:41:09.736 INFO:teuthology.orchestra.run.smithi114.stdout: "name": "batch_size", 2024-04-23T10:41:09.736 INFO:teuthology.orchestra.run.smithi114.stdout: "type": "int", 2024-04-23T10:41:09.736 INFO:teuthology.orchestra.run.smithi114.stdout: "level": "advanced", 2024-04-23T10:41:09.736 INFO:teuthology.orchestra.run.smithi114.stdout: "flags": 0, 2024-04-23T10:41:09.736 INFO:teuthology.orchestra.run.smithi114.stdout: "default_value": "5000", 2024-04-23T10:41:09.736 INFO:teuthology.orchestra.run.smithi114.stdout: "min": "", 2024-04-23T10:41:09.736 INFO:teuthology.orchestra.run.smithi114.stdout: "max": "", 2024-04-23T10:41:09.736 INFO:teuthology.orchestra.run.smithi114.stdout: "enum_allowed": [], 2024-04-23T10:41:09.736 INFO:teuthology.orchestra.run.smithi114.stdout: "desc": "How big batches of data points should be when sending to InfluxDB.", 2024-04-23T10:41:09.736 INFO:teuthology.orchestra.run.smithi114.stdout: "long_desc": "", 2024-04-23T10:41:09.737 INFO:teuthology.orchestra.run.smithi114.stdout: "tags": [], 2024-04-23T10:41:09.737 INFO:teuthology.orchestra.run.smithi114.stdout: "see_also": [] 2024-04-23T10:41:09.737 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-23T10:41:09.737 INFO:teuthology.orchestra.run.smithi114.stdout: "database": { 2024-04-23T10:41:09.737 INFO:teuthology.orchestra.run.smithi114.stdout: "name": "database", 2024-04-23T10:41:09.737 INFO:teuthology.orchestra.run.smithi114.stdout: "type": "str", 2024-04-23T10:41:09.737 INFO:teuthology.orchestra.run.smithi114.stdout: "level": "advanced", 2024-04-23T10:41:09.737 INFO:teuthology.orchestra.run.smithi114.stdout: "flags": 0, 2024-04-23T10:41:09.737 INFO:teuthology.orchestra.run.smithi114.stdout: "default_value": "ceph", 2024-04-23T10:41:09.738 INFO:teuthology.orchestra.run.smithi114.stdout: "min": "", 2024-04-23T10:41:09.738 INFO:teuthology.orchestra.run.smithi114.stdout: "max": "", 2024-04-23T10:41:09.738 INFO:teuthology.orchestra.run.smithi114.stdout: "enum_allowed": [], 2024-04-23T10:41:09.738 INFO:teuthology.orchestra.run.smithi114.stdout: "desc": "InfluxDB database name. You will need to create this database and grant write privileges to the configured username or the username must have admin privileges to create it.", 2024-04-23T10:41:09.738 INFO:teuthology.orchestra.run.smithi114.stdout: "long_desc": "", 2024-04-23T10:41:09.738 INFO:teuthology.orchestra.run.smithi114.stdout: "tags": [], 2024-04-23T10:41:09.738 INFO:teuthology.orchestra.run.smithi114.stdout: "see_also": [] 2024-04-23T10:41:09.738 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-23T10:41:09.739 INFO:teuthology.orchestra.run.smithi114.stdout: "hostname": { 2024-04-23T10:41:09.739 INFO:teuthology.orchestra.run.smithi114.stdout: "name": "hostname", 2024-04-23T10:41:09.739 INFO:teuthology.orchestra.run.smithi114.stdout: "type": "str", 2024-04-23T10:41:09.739 INFO:teuthology.orchestra.run.smithi114.stdout: "level": "advanced", 2024-04-23T10:41:09.739 INFO:teuthology.orchestra.run.smithi114.stdout: "flags": 0, 2024-04-23T10:41:09.739 INFO:teuthology.orchestra.run.smithi114.stdout: "default_value": "", 2024-04-23T10:41:09.739 INFO:teuthology.orchestra.run.smithi114.stdout: "min": "", 2024-04-23T10:41:09.739 INFO:teuthology.orchestra.run.smithi114.stdout: "max": "", 2024-04-23T10:41:09.739 INFO:teuthology.orchestra.run.smithi114.stdout: "enum_allowed": [], 2024-04-23T10:41:09.739 INFO:teuthology.orchestra.run.smithi114.stdout: "desc": "InfluxDB server hostname", 2024-04-23T10:41:09.740 INFO:teuthology.orchestra.run.smithi114.stdout: "long_desc": "", 2024-04-23T10:41:09.740 INFO:teuthology.orchestra.run.smithi114.stdout: "tags": [], 2024-04-23T10:41:09.740 INFO:teuthology.orchestra.run.smithi114.stdout: "see_also": [] 2024-04-23T10:41:09.740 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-23T10:41:09.740 INFO:teuthology.orchestra.run.smithi114.stdout: "interval": { 2024-04-23T10:41:09.740 INFO:teuthology.orchestra.run.smithi114.stdout: "name": "interval", 2024-04-23T10:41:09.740 INFO:teuthology.orchestra.run.smithi114.stdout: "type": "secs", 2024-04-23T10:41:09.740 INFO:teuthology.orchestra.run.smithi114.stdout: "level": "advanced", 2024-04-23T10:41:09.740 INFO:teuthology.orchestra.run.smithi114.stdout: "flags": 0, 2024-04-23T10:41:09.741 INFO:teuthology.orchestra.run.smithi114.stdout: "default_value": "30", 2024-04-23T10:41:09.741 INFO:teuthology.orchestra.run.smithi114.stdout: "min": "5", 2024-04-23T10:41:09.741 INFO:teuthology.orchestra.run.smithi114.stdout: "max": "", 2024-04-23T10:41:09.741 INFO:teuthology.orchestra.run.smithi114.stdout: "enum_allowed": [], 2024-04-23T10:41:09.741 INFO:teuthology.orchestra.run.smithi114.stdout: "desc": "Time between reports to InfluxDB. Default 30 seconds.", 2024-04-23T10:41:09.741 INFO:teuthology.orchestra.run.smithi114.stdout: "long_desc": "", 2024-04-23T10:41:09.741 INFO:teuthology.orchestra.run.smithi114.stdout: "tags": [], 2024-04-23T10:41:09.741 INFO:teuthology.orchestra.run.smithi114.stdout: "see_also": [] 2024-04-23T10:41:09.741 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-23T10:41:09.741 INFO:teuthology.orchestra.run.smithi114.stdout: "log_level": { 2024-04-23T10:41:09.742 INFO:teuthology.orchestra.run.smithi114.stdout: "name": "log_level", 2024-04-23T10:41:09.742 INFO:teuthology.orchestra.run.smithi114.stdout: "type": "str", 2024-04-23T10:41:09.742 INFO:teuthology.orchestra.run.smithi114.stdout: "level": "advanced", 2024-04-23T10:41:09.742 INFO:teuthology.orchestra.run.smithi114.stdout: "flags": 1, 2024-04-23T10:41:09.742 INFO:teuthology.orchestra.run.smithi114.stdout: "default_value": "", 2024-04-23T10:41:09.742 INFO:teuthology.orchestra.run.smithi114.stdout: "min": "", 2024-04-23T10:41:09.742 INFO:teuthology.orchestra.run.smithi114.stdout: "max": "", 2024-04-23T10:41:09.742 INFO:teuthology.orchestra.run.smithi114.stdout: "enum_allowed": [ 2024-04-23T10:41:09.742 INFO:teuthology.orchestra.run.smithi114.stdout: "", 2024-04-23T10:41:09.743 INFO:teuthology.orchestra.run.smithi114.stdout: "critical", 2024-04-23T10:41:09.743 INFO:teuthology.orchestra.run.smithi114.stdout: "debug", 2024-04-23T10:41:09.743 INFO:teuthology.orchestra.run.smithi114.stdout: "error", 2024-04-23T10:41:09.743 INFO:teuthology.orchestra.run.smithi114.stdout: "info", 2024-04-23T10:41:09.743 INFO:teuthology.orchestra.run.smithi114.stdout: "warning" 2024-04-23T10:41:09.743 INFO:teuthology.orchestra.run.smithi114.stdout: ], 2024-04-23T10:41:09.743 INFO:teuthology.orchestra.run.smithi114.stdout: "desc": "", 2024-04-23T10:41:09.743 INFO:teuthology.orchestra.run.smithi114.stdout: "long_desc": "", 2024-04-23T10:41:09.743 INFO:teuthology.orchestra.run.smithi114.stdout: "tags": [], 2024-04-23T10:41:09.743 INFO:teuthology.orchestra.run.smithi114.stdout: "see_also": [] 2024-04-23T10:41:09.744 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-23T10:41:09.744 INFO:teuthology.orchestra.run.smithi114.stdout: "log_to_cluster": { 2024-04-23T10:41:09.744 INFO:teuthology.orchestra.run.smithi114.stdout: "name": "log_to_cluster", 2024-04-23T10:41:09.744 INFO:teuthology.orchestra.run.smithi114.stdout: "type": "bool", 2024-04-23T10:41:09.744 INFO:teuthology.orchestra.run.smithi114.stdout: "level": "advanced", 2024-04-23T10:41:09.744 INFO:teuthology.orchestra.run.smithi114.stdout: "flags": 1, 2024-04-23T10:41:09.744 INFO:teuthology.orchestra.run.smithi114.stdout: "default_value": "False", 2024-04-23T10:41:09.744 INFO:teuthology.orchestra.run.smithi114.stdout: "min": "", 2024-04-23T10:41:09.744 INFO:teuthology.orchestra.run.smithi114.stdout: "max": "", 2024-04-23T10:41:09.744 INFO:teuthology.orchestra.run.smithi114.stdout: "enum_allowed": [], 2024-04-23T10:41:09.745 INFO:teuthology.orchestra.run.smithi114.stdout: "desc": "", 2024-04-23T10:41:09.745 INFO:teuthology.orchestra.run.smithi114.stdout: "long_desc": "", 2024-04-23T10:41:09.745 INFO:teuthology.orchestra.run.smithi114.stdout: "tags": [], 2024-04-23T10:41:09.745 INFO:teuthology.orchestra.run.smithi114.stdout: "see_also": [] 2024-04-23T10:41:09.745 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-23T10:41:09.745 INFO:teuthology.orchestra.run.smithi114.stdout: "log_to_cluster_level": { 2024-04-23T10:41:09.745 INFO:teuthology.orchestra.run.smithi114.stdout: "name": "log_to_cluster_level", 2024-04-23T10:41:09.745 INFO:teuthology.orchestra.run.smithi114.stdout: "type": "str", 2024-04-23T10:41:09.745 INFO:teuthology.orchestra.run.smithi114.stdout: "level": "advanced", 2024-04-23T10:41:09.745 INFO:teuthology.orchestra.run.smithi114.stdout: "flags": 1, 2024-04-23T10:41:09.746 INFO:teuthology.orchestra.run.smithi114.stdout: "default_value": "info", 2024-04-23T10:41:09.746 INFO:teuthology.orchestra.run.smithi114.stdout: "min": "", 2024-04-23T10:41:09.746 INFO:teuthology.orchestra.run.smithi114.stdout: "max": "", 2024-04-23T10:41:09.746 INFO:teuthology.orchestra.run.smithi114.stdout: "enum_allowed": [ 2024-04-23T10:41:09.746 INFO:teuthology.orchestra.run.smithi114.stdout: "", 2024-04-23T10:41:09.746 INFO:teuthology.orchestra.run.smithi114.stdout: "critical", 2024-04-23T10:41:09.746 INFO:teuthology.orchestra.run.smithi114.stdout: "debug", 2024-04-23T10:41:09.746 INFO:teuthology.orchestra.run.smithi114.stdout: "error", 2024-04-23T10:41:09.746 INFO:teuthology.orchestra.run.smithi114.stdout: "info", 2024-04-23T10:41:09.747 INFO:teuthology.orchestra.run.smithi114.stdout: "warning" 2024-04-23T10:41:09.747 INFO:teuthology.orchestra.run.smithi114.stdout: ], 2024-04-23T10:41:09.747 INFO:teuthology.orchestra.run.smithi114.stdout: "desc": "", 2024-04-23T10:41:09.747 INFO:teuthology.orchestra.run.smithi114.stdout: "long_desc": "", 2024-04-23T10:41:09.747 INFO:teuthology.orchestra.run.smithi114.stdout: "tags": [], 2024-04-23T10:41:09.747 INFO:teuthology.orchestra.run.smithi114.stdout: "see_also": [] 2024-04-23T10:41:09.747 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-23T10:41:09.747 INFO:teuthology.orchestra.run.smithi114.stdout: "log_to_file": { 2024-04-23T10:41:09.747 INFO:teuthology.orchestra.run.smithi114.stdout: "name": "log_to_file", 2024-04-23T10:41:09.747 INFO:teuthology.orchestra.run.smithi114.stdout: "type": "bool", 2024-04-23T10:41:09.748 INFO:teuthology.orchestra.run.smithi114.stdout: "level": "advanced", 2024-04-23T10:41:09.748 INFO:teuthology.orchestra.run.smithi114.stdout: "flags": 1, 2024-04-23T10:41:09.748 INFO:teuthology.orchestra.run.smithi114.stdout: "default_value": "False", 2024-04-23T10:41:09.748 INFO:teuthology.orchestra.run.smithi114.stdout: "min": "", 2024-04-23T10:41:09.748 INFO:teuthology.orchestra.run.smithi114.stdout: "max": "", 2024-04-23T10:41:09.748 INFO:teuthology.orchestra.run.smithi114.stdout: "enum_allowed": [], 2024-04-23T10:41:09.748 INFO:teuthology.orchestra.run.smithi114.stdout: "desc": "", 2024-04-23T10:41:09.748 INFO:teuthology.orchestra.run.smithi114.stdout: "long_desc": "", 2024-04-23T10:41:09.748 INFO:teuthology.orchestra.run.smithi114.stdout: "tags": [], 2024-04-23T10:41:09.749 INFO:teuthology.orchestra.run.smithi114.stdout: "see_also": [] 2024-04-23T10:41:09.749 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-23T10:41:09.749 INFO:teuthology.orchestra.run.smithi114.stdout: "password": { 2024-04-23T10:41:09.749 INFO:teuthology.orchestra.run.smithi114.stdout: "name": "password", 2024-04-23T10:41:09.749 INFO:teuthology.orchestra.run.smithi114.stdout: "type": "str", 2024-04-23T10:41:09.749 INFO:teuthology.orchestra.run.smithi114.stdout: "level": "advanced", 2024-04-23T10:41:09.749 INFO:teuthology.orchestra.run.smithi114.stdout: "flags": 0, 2024-04-23T10:41:09.749 INFO:teuthology.orchestra.run.smithi114.stdout: "default_value": "", 2024-04-23T10:41:09.749 INFO:teuthology.orchestra.run.smithi114.stdout: "min": "", 2024-04-23T10:41:09.749 INFO:teuthology.orchestra.run.smithi114.stdout: "max": "", 2024-04-23T10:41:09.750 INFO:teuthology.orchestra.run.smithi114.stdout: "enum_allowed": [], 2024-04-23T10:41:09.750 INFO:teuthology.orchestra.run.smithi114.stdout: "desc": "password of InfluxDB server user", 2024-04-23T10:41:09.750 INFO:teuthology.orchestra.run.smithi114.stdout: "long_desc": "", 2024-04-23T10:41:09.750 INFO:teuthology.orchestra.run.smithi114.stdout: "tags": [], 2024-04-23T10:41:09.750 INFO:teuthology.orchestra.run.smithi114.stdout: "see_also": [] 2024-04-23T10:41:09.750 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-23T10:41:09.750 INFO:teuthology.orchestra.run.smithi114.stdout: "port": { 2024-04-23T10:41:09.750 INFO:teuthology.orchestra.run.smithi114.stdout: "name": "port", 2024-04-23T10:41:09.751 INFO:teuthology.orchestra.run.smithi114.stdout: "type": "int", 2024-04-23T10:41:09.751 INFO:teuthology.orchestra.run.smithi114.stdout: "level": "advanced", 2024-04-23T10:41:09.751 INFO:teuthology.orchestra.run.smithi114.stdout: "flags": 0, 2024-04-23T10:41:09.751 INFO:teuthology.orchestra.run.smithi114.stdout: "default_value": "8086", 2024-04-23T10:41:09.751 INFO:teuthology.orchestra.run.smithi114.stdout: "min": "", 2024-04-23T10:41:09.751 INFO:teuthology.orchestra.run.smithi114.stdout: "max": "", 2024-04-23T10:41:09.751 INFO:teuthology.orchestra.run.smithi114.stdout: "enum_allowed": [], 2024-04-23T10:41:09.751 INFO:teuthology.orchestra.run.smithi114.stdout: "desc": "InfluxDB server port", 2024-04-23T10:41:09.751 INFO:teuthology.orchestra.run.smithi114.stdout: "long_desc": "", 2024-04-23T10:41:09.751 INFO:teuthology.orchestra.run.smithi114.stdout: "tags": [], 2024-04-23T10:41:09.752 INFO:teuthology.orchestra.run.smithi114.stdout: "see_also": [] 2024-04-23T10:41:09.752 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-23T10:41:09.752 INFO:teuthology.orchestra.run.smithi114.stdout: "ssl": { 2024-04-23T10:41:09.752 INFO:teuthology.orchestra.run.smithi114.stdout: "name": "ssl", 2024-04-23T10:41:09.752 INFO:teuthology.orchestra.run.smithi114.stdout: "type": "str", 2024-04-23T10:41:09.752 INFO:teuthology.orchestra.run.smithi114.stdout: "level": "advanced", 2024-04-23T10:41:09.752 INFO:teuthology.orchestra.run.smithi114.stdout: "flags": 0, 2024-04-23T10:41:09.752 INFO:teuthology.orchestra.run.smithi114.stdout: "default_value": "false", 2024-04-23T10:41:09.752 INFO:teuthology.orchestra.run.smithi114.stdout: "min": "", 2024-04-23T10:41:09.752 INFO:teuthology.orchestra.run.smithi114.stdout: "max": "", 2024-04-23T10:41:09.753 INFO:teuthology.orchestra.run.smithi114.stdout: "enum_allowed": [], 2024-04-23T10:41:09.753 INFO:teuthology.orchestra.run.smithi114.stdout: "desc": "Use https connection for InfluxDB server. Use \"true\" or \"false\".", 2024-04-23T10:41:09.753 INFO:teuthology.orchestra.run.smithi114.stdout: "long_desc": "", 2024-04-23T10:41:09.753 INFO:teuthology.orchestra.run.smithi114.stdout: "tags": [], 2024-04-23T10:41:09.753 INFO:teuthology.orchestra.run.smithi114.stdout: "see_also": [] 2024-04-23T10:41:09.753 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-23T10:41:09.753 INFO:teuthology.orchestra.run.smithi114.stdout: "threads": { 2024-04-23T10:41:09.753 INFO:teuthology.orchestra.run.smithi114.stdout: "name": "threads", 2024-04-23T10:41:09.753 INFO:teuthology.orchestra.run.smithi114.stdout: "type": "int", 2024-04-23T10:41:09.754 INFO:teuthology.orchestra.run.smithi114.stdout: "level": "advanced", 2024-04-23T10:41:09.754 INFO:teuthology.orchestra.run.smithi114.stdout: "flags": 0, 2024-04-23T10:41:09.754 INFO:teuthology.orchestra.run.smithi114.stdout: "default_value": "5", 2024-04-23T10:41:09.754 INFO:teuthology.orchestra.run.smithi114.stdout: "min": "1", 2024-04-23T10:41:09.754 INFO:teuthology.orchestra.run.smithi114.stdout: "max": "32", 2024-04-23T10:41:09.754 INFO:teuthology.orchestra.run.smithi114.stdout: "enum_allowed": [], 2024-04-23T10:41:09.754 INFO:teuthology.orchestra.run.smithi114.stdout: "desc": "How many worker threads should be spawned for sending data to InfluxDB.", 2024-04-23T10:41:09.754 INFO:teuthology.orchestra.run.smithi114.stdout: "long_desc": "", 2024-04-23T10:41:09.754 INFO:teuthology.orchestra.run.smithi114.stdout: "tags": [], 2024-04-23T10:41:09.755 INFO:teuthology.orchestra.run.smithi114.stdout: "see_also": [] 2024-04-23T10:41:09.755 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-23T10:41:09.755 INFO:teuthology.orchestra.run.smithi114.stdout: "username": { 2024-04-23T10:41:09.755 INFO:teuthology.orchestra.run.smithi114.stdout: "name": "username", 2024-04-23T10:41:09.755 INFO:teuthology.orchestra.run.smithi114.stdout: "type": "str", 2024-04-23T10:41:09.755 INFO:teuthology.orchestra.run.smithi114.stdout: "level": "advanced", 2024-04-23T10:41:09.755 INFO:teuthology.orchestra.run.smithi114.stdout: "flags": 0, 2024-04-23T10:41:09.755 INFO:teuthology.orchestra.run.smithi114.stdout: "default_value": "", 2024-04-23T10:41:09.755 INFO:teuthology.orchestra.run.smithi114.stdout: "min": "", 2024-04-23T10:41:09.755 INFO:teuthology.orchestra.run.smithi114.stdout: "max": "", 2024-04-23T10:41:09.756 INFO:teuthology.orchestra.run.smithi114.stdout: "enum_allowed": [], 2024-04-23T10:41:09.756 INFO:teuthology.orchestra.run.smithi114.stdout: "desc": "username of InfluxDB server user", 2024-04-23T10:41:09.756 INFO:teuthology.orchestra.run.smithi114.stdout: "long_desc": "", 2024-04-23T10:41:09.756 INFO:teuthology.orchestra.run.smithi114.stdout: "tags": [], 2024-04-23T10:41:09.756 INFO:teuthology.orchestra.run.smithi114.stdout: "see_also": [] 2024-04-23T10:41:09.756 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-23T10:41:09.756 INFO:teuthology.orchestra.run.smithi114.stdout: "verify_ssl": { 2024-04-23T10:41:09.756 INFO:teuthology.orchestra.run.smithi114.stdout: "name": "verify_ssl", 2024-04-23T10:41:09.756 INFO:teuthology.orchestra.run.smithi114.stdout: "type": "str", 2024-04-23T10:41:09.756 INFO:teuthology.orchestra.run.smithi114.stdout: "level": "advanced", 2024-04-23T10:41:09.757 INFO:teuthology.orchestra.run.smithi114.stdout: "flags": 0, 2024-04-23T10:41:09.757 INFO:teuthology.orchestra.run.smithi114.stdout: "default_value": "true", 2024-04-23T10:41:09.757 INFO:teuthology.orchestra.run.smithi114.stdout: "min": "", 2024-04-23T10:41:09.757 INFO:teuthology.orchestra.run.smithi114.stdout: "max": "", 2024-04-23T10:41:09.757 INFO:teuthology.orchestra.run.smithi114.stdout: "enum_allowed": [], 2024-04-23T10:41:09.757 INFO:teuthology.orchestra.run.smithi114.stdout: "desc": "Verify https cert for InfluxDB server. Use \"true\" or \"false\".", 2024-04-23T10:41:09.757 INFO:teuthology.orchestra.run.smithi114.stdout: "long_desc": "", 2024-04-23T10:41:09.757 INFO:teuthology.orchestra.run.smithi114.stdout: "tags": [], 2024-04-23T10:41:09.757 INFO:teuthology.orchestra.run.smithi114.stdout: "see_also": [] 2024-04-23T10:41:09.758 INFO:teuthology.orchestra.run.smithi114.stdout: } 2024-04-23T10:41:09.758 INFO:teuthology.orchestra.run.smithi114.stdout: } 2024-04-23T10:41:09.758 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-23T10:41:09.758 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-04-23T10:41:09.758 INFO:teuthology.orchestra.run.smithi114.stdout: "name": "insights", 2024-04-23T10:41:09.758 INFO:teuthology.orchestra.run.smithi114.stdout: "can_run": true, 2024-04-23T10:41:09.758 INFO:teuthology.orchestra.run.smithi114.stdout: "error_string": "", 2024-04-23T10:41:09.758 INFO:teuthology.orchestra.run.smithi114.stdout: "module_options": { 2024-04-23T10:41:09.758 INFO:teuthology.orchestra.run.smithi114.stdout: "log_level": { 2024-04-23T10:41:09.759 INFO:teuthology.orchestra.run.smithi114.stdout: "name": "log_level", 2024-04-23T10:41:09.759 INFO:teuthology.orchestra.run.smithi114.stdout: "type": "str", 2024-04-23T10:41:09.759 INFO:teuthology.orchestra.run.smithi114.stdout: "level": "advanced", 2024-04-23T10:41:09.759 INFO:teuthology.orchestra.run.smithi114.stdout: "flags": 1, 2024-04-23T10:41:09.759 INFO:teuthology.orchestra.run.smithi114.stdout: "default_value": "", 2024-04-23T10:41:09.759 INFO:teuthology.orchestra.run.smithi114.stdout: "min": "", 2024-04-23T10:41:09.759 INFO:teuthology.orchestra.run.smithi114.stdout: "max": "", 2024-04-23T10:41:09.759 INFO:teuthology.orchestra.run.smithi114.stdout: "enum_allowed": [ 2024-04-23T10:41:09.759 INFO:teuthology.orchestra.run.smithi114.stdout: "", 2024-04-23T10:41:09.759 INFO:teuthology.orchestra.run.smithi114.stdout: "critical", 2024-04-23T10:41:09.760 INFO:teuthology.orchestra.run.smithi114.stdout: "debug", 2024-04-23T10:41:09.760 INFO:teuthology.orchestra.run.smithi114.stdout: "error", 2024-04-23T10:41:09.760 INFO:teuthology.orchestra.run.smithi114.stdout: "info", 2024-04-23T10:41:09.760 INFO:teuthology.orchestra.run.smithi114.stdout: "warning" 2024-04-23T10:41:09.760 INFO:teuthology.orchestra.run.smithi114.stdout: ], 2024-04-23T10:41:09.760 INFO:teuthology.orchestra.run.smithi114.stdout: "desc": "", 2024-04-23T10:41:09.760 INFO:teuthology.orchestra.run.smithi114.stdout: "long_desc": "", 2024-04-23T10:41:09.760 INFO:teuthology.orchestra.run.smithi114.stdout: "tags": [], 2024-04-23T10:41:09.760 INFO:teuthology.orchestra.run.smithi114.stdout: "see_also": [] 2024-04-23T10:41:09.761 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-23T10:41:09.761 INFO:teuthology.orchestra.run.smithi114.stdout: "log_to_cluster": { 2024-04-23T10:41:09.761 INFO:teuthology.orchestra.run.smithi114.stdout: "name": "log_to_cluster", 2024-04-23T10:41:09.761 INFO:teuthology.orchestra.run.smithi114.stdout: "type": "bool", 2024-04-23T10:41:09.761 INFO:teuthology.orchestra.run.smithi114.stdout: "level": "advanced", 2024-04-23T10:41:09.761 INFO:teuthology.orchestra.run.smithi114.stdout: "flags": 1, 2024-04-23T10:41:09.761 INFO:teuthology.orchestra.run.smithi114.stdout: "default_value": "False", 2024-04-23T10:41:09.761 INFO:teuthology.orchestra.run.smithi114.stdout: "min": "", 2024-04-23T10:41:09.761 INFO:teuthology.orchestra.run.smithi114.stdout: "max": "", 2024-04-23T10:41:09.762 INFO:teuthology.orchestra.run.smithi114.stdout: "enum_allowed": [], 2024-04-23T10:41:09.762 INFO:teuthology.orchestra.run.smithi114.stdout: "desc": "", 2024-04-23T10:41:09.762 INFO:teuthology.orchestra.run.smithi114.stdout: "long_desc": "", 2024-04-23T10:41:09.762 INFO:teuthology.orchestra.run.smithi114.stdout: "tags": [], 2024-04-23T10:41:09.762 INFO:teuthology.orchestra.run.smithi114.stdout: "see_also": [] 2024-04-23T10:41:09.762 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-23T10:41:09.762 INFO:teuthology.orchestra.run.smithi114.stdout: "log_to_cluster_level": { 2024-04-23T10:41:09.762 INFO:teuthology.orchestra.run.smithi114.stdout: "name": "log_to_cluster_level", 2024-04-23T10:41:09.762 INFO:teuthology.orchestra.run.smithi114.stdout: "type": "str", 2024-04-23T10:41:09.763 INFO:teuthology.orchestra.run.smithi114.stdout: "level": "advanced", 2024-04-23T10:41:09.763 INFO:teuthology.orchestra.run.smithi114.stdout: "flags": 1, 2024-04-23T10:41:09.763 INFO:teuthology.orchestra.run.smithi114.stdout: "default_value": "info", 2024-04-23T10:41:09.763 INFO:teuthology.orchestra.run.smithi114.stdout: "min": "", 2024-04-23T10:41:09.763 INFO:teuthology.orchestra.run.smithi114.stdout: "max": "", 2024-04-23T10:41:09.763 INFO:teuthology.orchestra.run.smithi114.stdout: "enum_allowed": [ 2024-04-23T10:41:09.763 INFO:teuthology.orchestra.run.smithi114.stdout: "", 2024-04-23T10:41:09.763 INFO:teuthology.orchestra.run.smithi114.stdout: "critical", 2024-04-23T10:41:09.764 INFO:teuthology.orchestra.run.smithi114.stdout: "debug", 2024-04-23T10:41:09.764 INFO:teuthology.orchestra.run.smithi114.stdout: "error", 2024-04-23T10:41:09.764 INFO:teuthology.orchestra.run.smithi114.stdout: "info", 2024-04-23T10:41:09.764 INFO:teuthology.orchestra.run.smithi114.stdout: "warning" 2024-04-23T10:41:09.764 INFO:teuthology.orchestra.run.smithi114.stdout: ], 2024-04-23T10:41:09.764 INFO:teuthology.orchestra.run.smithi114.stdout: "desc": "", 2024-04-23T10:41:09.764 INFO:teuthology.orchestra.run.smithi114.stdout: "long_desc": "", 2024-04-23T10:41:09.764 INFO:teuthology.orchestra.run.smithi114.stdout: "tags": [], 2024-04-23T10:41:09.764 INFO:teuthology.orchestra.run.smithi114.stdout: "see_also": [] 2024-04-23T10:41:09.765 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-23T10:41:09.765 INFO:teuthology.orchestra.run.smithi114.stdout: "log_to_file": { 2024-04-23T10:41:09.765 INFO:teuthology.orchestra.run.smithi114.stdout: "name": "log_to_file", 2024-04-23T10:41:09.765 INFO:teuthology.orchestra.run.smithi114.stdout: "type": "bool", 2024-04-23T10:41:09.765 INFO:teuthology.orchestra.run.smithi114.stdout: "level": "advanced", 2024-04-23T10:41:09.765 INFO:teuthology.orchestra.run.smithi114.stdout: "flags": 1, 2024-04-23T10:41:09.765 INFO:teuthology.orchestra.run.smithi114.stdout: "default_value": "False", 2024-04-23T10:41:09.765 INFO:teuthology.orchestra.run.smithi114.stdout: "min": "", 2024-04-23T10:41:09.765 INFO:teuthology.orchestra.run.smithi114.stdout: "max": "", 2024-04-23T10:41:09.766 INFO:teuthology.orchestra.run.smithi114.stdout: "enum_allowed": [], 2024-04-23T10:41:09.766 INFO:teuthology.orchestra.run.smithi114.stdout: "desc": "", 2024-04-23T10:41:09.766 INFO:teuthology.orchestra.run.smithi114.stdout: "long_desc": "", 2024-04-23T10:41:09.766 INFO:teuthology.orchestra.run.smithi114.stdout: "tags": [], 2024-04-23T10:41:09.766 INFO:teuthology.orchestra.run.smithi114.stdout: "see_also": [] 2024-04-23T10:41:09.766 INFO:teuthology.orchestra.run.smithi114.stdout: } 2024-04-23T10:41:09.766 INFO:teuthology.orchestra.run.smithi114.stdout: } 2024-04-23T10:41:09.766 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-23T10:41:09.766 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-04-23T10:41:09.767 INFO:teuthology.orchestra.run.smithi114.stdout: "name": "iostat", 2024-04-23T10:41:09.767 INFO:teuthology.orchestra.run.smithi114.stdout: "can_run": true, 2024-04-23T10:41:09.767 INFO:teuthology.orchestra.run.smithi114.stdout: "error_string": "", 2024-04-23T10:41:09.767 INFO:teuthology.orchestra.run.smithi114.stdout: "module_options": { 2024-04-23T10:41:09.767 INFO:teuthology.orchestra.run.smithi114.stdout: "log_level": { 2024-04-23T10:41:09.767 INFO:teuthology.orchestra.run.smithi114.stdout: "name": "log_level", 2024-04-23T10:41:09.767 INFO:teuthology.orchestra.run.smithi114.stdout: "type": "str", 2024-04-23T10:41:09.767 INFO:teuthology.orchestra.run.smithi114.stdout: "level": "advanced", 2024-04-23T10:41:09.767 INFO:teuthology.orchestra.run.smithi114.stdout: "flags": 1, 2024-04-23T10:41:09.768 INFO:teuthology.orchestra.run.smithi114.stdout: "default_value": "", 2024-04-23T10:41:09.768 INFO:teuthology.orchestra.run.smithi114.stdout: "min": "", 2024-04-23T10:41:09.768 INFO:teuthology.orchestra.run.smithi114.stdout: "max": "", 2024-04-23T10:41:09.768 INFO:teuthology.orchestra.run.smithi114.stdout: "enum_allowed": [ 2024-04-23T10:41:09.768 INFO:teuthology.orchestra.run.smithi114.stdout: "", 2024-04-23T10:41:09.768 INFO:teuthology.orchestra.run.smithi114.stdout: "critical", 2024-04-23T10:41:09.768 INFO:teuthology.orchestra.run.smithi114.stdout: "debug", 2024-04-23T10:41:09.768 INFO:teuthology.orchestra.run.smithi114.stdout: "error", 2024-04-23T10:41:09.769 INFO:teuthology.orchestra.run.smithi114.stdout: "info", 2024-04-23T10:41:09.769 INFO:teuthology.orchestra.run.smithi114.stdout: "warning" 2024-04-23T10:41:09.769 INFO:teuthology.orchestra.run.smithi114.stdout: ], 2024-04-23T10:41:09.769 INFO:teuthology.orchestra.run.smithi114.stdout: "desc": "", 2024-04-23T10:41:09.769 INFO:teuthology.orchestra.run.smithi114.stdout: "long_desc": "", 2024-04-23T10:41:09.769 INFO:teuthology.orchestra.run.smithi114.stdout: "tags": [], 2024-04-23T10:41:09.769 INFO:teuthology.orchestra.run.smithi114.stdout: "see_also": [] 2024-04-23T10:41:09.769 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-23T10:41:09.769 INFO:teuthology.orchestra.run.smithi114.stdout: "log_to_cluster": { 2024-04-23T10:41:09.769 INFO:teuthology.orchestra.run.smithi114.stdout: "name": "log_to_cluster", 2024-04-23T10:41:09.770 INFO:teuthology.orchestra.run.smithi114.stdout: "type": "bool", 2024-04-23T10:41:09.770 INFO:teuthology.orchestra.run.smithi114.stdout: "level": "advanced", 2024-04-23T10:41:09.770 INFO:teuthology.orchestra.run.smithi114.stdout: "flags": 1, 2024-04-23T10:41:09.770 INFO:teuthology.orchestra.run.smithi114.stdout: "default_value": "False", 2024-04-23T10:41:09.770 INFO:teuthology.orchestra.run.smithi114.stdout: "min": "", 2024-04-23T10:41:09.770 INFO:teuthology.orchestra.run.smithi114.stdout: "max": "", 2024-04-23T10:41:09.770 INFO:teuthology.orchestra.run.smithi114.stdout: "enum_allowed": [], 2024-04-23T10:41:09.770 INFO:teuthology.orchestra.run.smithi114.stdout: "desc": "", 2024-04-23T10:41:09.770 INFO:teuthology.orchestra.run.smithi114.stdout: "long_desc": "", 2024-04-23T10:41:09.770 INFO:teuthology.orchestra.run.smithi114.stdout: "tags": [], 2024-04-23T10:41:09.771 INFO:teuthology.orchestra.run.smithi114.stdout: "see_also": [] 2024-04-23T10:41:09.771 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-23T10:41:09.771 INFO:teuthology.orchestra.run.smithi114.stdout: "log_to_cluster_level": { 2024-04-23T10:41:09.771 INFO:teuthology.orchestra.run.smithi114.stdout: "name": "log_to_cluster_level", 2024-04-23T10:41:09.771 INFO:teuthology.orchestra.run.smithi114.stdout: "type": "str", 2024-04-23T10:41:09.771 INFO:teuthology.orchestra.run.smithi114.stdout: "level": "advanced", 2024-04-23T10:41:09.771 INFO:teuthology.orchestra.run.smithi114.stdout: "flags": 1, 2024-04-23T10:41:09.771 INFO:teuthology.orchestra.run.smithi114.stdout: "default_value": "info", 2024-04-23T10:41:09.771 INFO:teuthology.orchestra.run.smithi114.stdout: "min": "", 2024-04-23T10:41:09.772 INFO:teuthology.orchestra.run.smithi114.stdout: "max": "", 2024-04-23T10:41:09.772 INFO:teuthology.orchestra.run.smithi114.stdout: "enum_allowed": [ 2024-04-23T10:41:09.772 INFO:teuthology.orchestra.run.smithi114.stdout: "", 2024-04-23T10:41:09.772 INFO:teuthology.orchestra.run.smithi114.stdout: "critical", 2024-04-23T10:41:09.772 INFO:teuthology.orchestra.run.smithi114.stdout: "debug", 2024-04-23T10:41:09.772 INFO:teuthology.orchestra.run.smithi114.stdout: "error", 2024-04-23T10:41:09.772 INFO:teuthology.orchestra.run.smithi114.stdout: "info", 2024-04-23T10:41:09.772 INFO:teuthology.orchestra.run.smithi114.stdout: "warning" 2024-04-23T10:41:09.772 INFO:teuthology.orchestra.run.smithi114.stdout: ], 2024-04-23T10:41:09.772 INFO:teuthology.orchestra.run.smithi114.stdout: "desc": "", 2024-04-23T10:41:09.773 INFO:teuthology.orchestra.run.smithi114.stdout: "long_desc": "", 2024-04-23T10:41:09.773 INFO:teuthology.orchestra.run.smithi114.stdout: "tags": [], 2024-04-23T10:41:09.773 INFO:teuthology.orchestra.run.smithi114.stdout: "see_also": [] 2024-04-23T10:41:09.773 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-23T10:41:09.773 INFO:teuthology.orchestra.run.smithi114.stdout: "log_to_file": { 2024-04-23T10:41:09.773 INFO:teuthology.orchestra.run.smithi114.stdout: "name": "log_to_file", 2024-04-23T10:41:09.773 INFO:teuthology.orchestra.run.smithi114.stdout: "type": "bool", 2024-04-23T10:41:09.773 INFO:teuthology.orchestra.run.smithi114.stdout: "level": "advanced", 2024-04-23T10:41:09.773 INFO:teuthology.orchestra.run.smithi114.stdout: "flags": 1, 2024-04-23T10:41:09.773 INFO:teuthology.orchestra.run.smithi114.stdout: "default_value": "False", 2024-04-23T10:41:09.774 INFO:teuthology.orchestra.run.smithi114.stdout: "min": "", 2024-04-23T10:41:09.774 INFO:teuthology.orchestra.run.smithi114.stdout: "max": "", 2024-04-23T10:41:09.774 INFO:teuthology.orchestra.run.smithi114.stdout: "enum_allowed": [], 2024-04-23T10:41:09.774 INFO:teuthology.orchestra.run.smithi114.stdout: "desc": "", 2024-04-23T10:41:09.774 INFO:teuthology.orchestra.run.smithi114.stdout: "long_desc": "", 2024-04-23T10:41:09.774 INFO:teuthology.orchestra.run.smithi114.stdout: "tags": [], 2024-04-23T10:41:09.774 INFO:teuthology.orchestra.run.smithi114.stdout: "see_also": [] 2024-04-23T10:41:09.774 INFO:teuthology.orchestra.run.smithi114.stdout: } 2024-04-23T10:41:09.774 INFO:teuthology.orchestra.run.smithi114.stdout: } 2024-04-23T10:41:09.775 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-23T10:41:09.775 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-04-23T10:41:09.775 INFO:teuthology.orchestra.run.smithi114.stdout: "name": "k8sevents", 2024-04-23T10:41:09.775 INFO:teuthology.orchestra.run.smithi114.stdout: "can_run": true, 2024-04-23T10:41:09.775 INFO:teuthology.orchestra.run.smithi114.stdout: "error_string": "", 2024-04-23T10:41:09.775 INFO:teuthology.orchestra.run.smithi114.stdout: "module_options": { 2024-04-23T10:41:09.775 INFO:teuthology.orchestra.run.smithi114.stdout: "ceph_event_retention_days": { 2024-04-23T10:41:09.775 INFO:teuthology.orchestra.run.smithi114.stdout: "name": "ceph_event_retention_days", 2024-04-23T10:41:09.775 INFO:teuthology.orchestra.run.smithi114.stdout: "type": "int", 2024-04-23T10:41:09.775 INFO:teuthology.orchestra.run.smithi114.stdout: "level": "advanced", 2024-04-23T10:41:09.776 INFO:teuthology.orchestra.run.smithi114.stdout: "flags": 0, 2024-04-23T10:41:09.776 INFO:teuthology.orchestra.run.smithi114.stdout: "default_value": "7", 2024-04-23T10:41:09.776 INFO:teuthology.orchestra.run.smithi114.stdout: "min": "", 2024-04-23T10:41:09.776 INFO:teuthology.orchestra.run.smithi114.stdout: "max": "", 2024-04-23T10:41:09.776 INFO:teuthology.orchestra.run.smithi114.stdout: "enum_allowed": [], 2024-04-23T10:41:09.776 INFO:teuthology.orchestra.run.smithi114.stdout: "desc": "Days to hold ceph event information within local cache", 2024-04-23T10:41:09.776 INFO:teuthology.orchestra.run.smithi114.stdout: "long_desc": "", 2024-04-23T10:41:09.776 INFO:teuthology.orchestra.run.smithi114.stdout: "tags": [], 2024-04-23T10:41:09.776 INFO:teuthology.orchestra.run.smithi114.stdout: "see_also": [] 2024-04-23T10:41:09.776 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-23T10:41:09.777 INFO:teuthology.orchestra.run.smithi114.stdout: "config_check_secs": { 2024-04-23T10:41:09.777 INFO:teuthology.orchestra.run.smithi114.stdout: "name": "config_check_secs", 2024-04-23T10:41:09.777 INFO:teuthology.orchestra.run.smithi114.stdout: "type": "int", 2024-04-23T10:41:09.777 INFO:teuthology.orchestra.run.smithi114.stdout: "level": "advanced", 2024-04-23T10:41:09.777 INFO:teuthology.orchestra.run.smithi114.stdout: "flags": 0, 2024-04-23T10:41:09.777 INFO:teuthology.orchestra.run.smithi114.stdout: "default_value": "10", 2024-04-23T10:41:09.777 INFO:teuthology.orchestra.run.smithi114.stdout: "min": "10", 2024-04-23T10:41:09.777 INFO:teuthology.orchestra.run.smithi114.stdout: "max": "", 2024-04-23T10:41:09.777 INFO:teuthology.orchestra.run.smithi114.stdout: "enum_allowed": [], 2024-04-23T10:41:09.778 INFO:teuthology.orchestra.run.smithi114.stdout: "desc": "interval (secs) to check for cluster configuration changes", 2024-04-23T10:41:09.778 INFO:teuthology.orchestra.run.smithi114.stdout: "long_desc": "", 2024-04-23T10:41:09.778 INFO:teuthology.orchestra.run.smithi114.stdout: "tags": [], 2024-04-23T10:41:09.778 INFO:teuthology.orchestra.run.smithi114.stdout: "see_also": [] 2024-04-23T10:41:09.778 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-23T10:41:09.778 INFO:teuthology.orchestra.run.smithi114.stdout: "log_level": { 2024-04-23T10:41:09.778 INFO:teuthology.orchestra.run.smithi114.stdout: "name": "log_level", 2024-04-23T10:41:09.778 INFO:teuthology.orchestra.run.smithi114.stdout: "type": "str", 2024-04-23T10:41:09.778 INFO:teuthology.orchestra.run.smithi114.stdout: "level": "advanced", 2024-04-23T10:41:09.778 INFO:teuthology.orchestra.run.smithi114.stdout: "flags": 1, 2024-04-23T10:41:09.779 INFO:teuthology.orchestra.run.smithi114.stdout: "default_value": "", 2024-04-23T10:41:09.779 INFO:teuthology.orchestra.run.smithi114.stdout: "min": "", 2024-04-23T10:41:09.779 INFO:teuthology.orchestra.run.smithi114.stdout: "max": "", 2024-04-23T10:41:09.779 INFO:teuthology.orchestra.run.smithi114.stdout: "enum_allowed": [ 2024-04-23T10:41:09.779 INFO:teuthology.orchestra.run.smithi114.stdout: "", 2024-04-23T10:41:09.779 INFO:teuthology.orchestra.run.smithi114.stdout: "critical", 2024-04-23T10:41:09.779 INFO:teuthology.orchestra.run.smithi114.stdout: "debug", 2024-04-23T10:41:09.779 INFO:teuthology.orchestra.run.smithi114.stdout: "error", 2024-04-23T10:41:09.779 INFO:teuthology.orchestra.run.smithi114.stdout: "info", 2024-04-23T10:41:09.779 INFO:teuthology.orchestra.run.smithi114.stdout: "warning" 2024-04-23T10:41:09.780 INFO:teuthology.orchestra.run.smithi114.stdout: ], 2024-04-23T10:41:09.780 INFO:teuthology.orchestra.run.smithi114.stdout: "desc": "", 2024-04-23T10:41:09.780 INFO:teuthology.orchestra.run.smithi114.stdout: "long_desc": "", 2024-04-23T10:41:09.780 INFO:teuthology.orchestra.run.smithi114.stdout: "tags": [], 2024-04-23T10:41:09.780 INFO:teuthology.orchestra.run.smithi114.stdout: "see_also": [] 2024-04-23T10:41:09.780 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-23T10:41:09.780 INFO:teuthology.orchestra.run.smithi114.stdout: "log_to_cluster": { 2024-04-23T10:41:09.780 INFO:teuthology.orchestra.run.smithi114.stdout: "name": "log_to_cluster", 2024-04-23T10:41:09.780 INFO:teuthology.orchestra.run.smithi114.stdout: "type": "bool", 2024-04-23T10:41:09.780 INFO:teuthology.orchestra.run.smithi114.stdout: "level": "advanced", 2024-04-23T10:41:09.781 INFO:teuthology.orchestra.run.smithi114.stdout: "flags": 1, 2024-04-23T10:41:09.781 INFO:teuthology.orchestra.run.smithi114.stdout: "default_value": "False", 2024-04-23T10:41:09.781 INFO:teuthology.orchestra.run.smithi114.stdout: "min": "", 2024-04-23T10:41:09.781 INFO:teuthology.orchestra.run.smithi114.stdout: "max": "", 2024-04-23T10:41:09.781 INFO:teuthology.orchestra.run.smithi114.stdout: "enum_allowed": [], 2024-04-23T10:41:09.781 INFO:teuthology.orchestra.run.smithi114.stdout: "desc": "", 2024-04-23T10:41:09.781 INFO:teuthology.orchestra.run.smithi114.stdout: "long_desc": "", 2024-04-23T10:41:09.781 INFO:teuthology.orchestra.run.smithi114.stdout: "tags": [], 2024-04-23T10:41:09.781 INFO:teuthology.orchestra.run.smithi114.stdout: "see_also": [] 2024-04-23T10:41:09.782 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-23T10:41:09.782 INFO:teuthology.orchestra.run.smithi114.stdout: "log_to_cluster_level": { 2024-04-23T10:41:09.782 INFO:teuthology.orchestra.run.smithi114.stdout: "name": "log_to_cluster_level", 2024-04-23T10:41:09.782 INFO:teuthology.orchestra.run.smithi114.stdout: "type": "str", 2024-04-23T10:41:09.782 INFO:teuthology.orchestra.run.smithi114.stdout: "level": "advanced", 2024-04-23T10:41:09.782 INFO:teuthology.orchestra.run.smithi114.stdout: "flags": 1, 2024-04-23T10:41:09.782 INFO:teuthology.orchestra.run.smithi114.stdout: "default_value": "info", 2024-04-23T10:41:09.782 INFO:teuthology.orchestra.run.smithi114.stdout: "min": "", 2024-04-23T10:41:09.782 INFO:teuthology.orchestra.run.smithi114.stdout: "max": "", 2024-04-23T10:41:09.782 INFO:teuthology.orchestra.run.smithi114.stdout: "enum_allowed": [ 2024-04-23T10:41:09.783 INFO:teuthology.orchestra.run.smithi114.stdout: "", 2024-04-23T10:41:09.783 INFO:teuthology.orchestra.run.smithi114.stdout: "critical", 2024-04-23T10:41:09.783 INFO:teuthology.orchestra.run.smithi114.stdout: "debug", 2024-04-23T10:41:09.783 INFO:teuthology.orchestra.run.smithi114.stdout: "error", 2024-04-23T10:41:09.783 INFO:teuthology.orchestra.run.smithi114.stdout: "info", 2024-04-23T10:41:09.783 INFO:teuthology.orchestra.run.smithi114.stdout: "warning" 2024-04-23T10:41:09.783 INFO:teuthology.orchestra.run.smithi114.stdout: ], 2024-04-23T10:41:09.783 INFO:teuthology.orchestra.run.smithi114.stdout: "desc": "", 2024-04-23T10:41:09.783 INFO:teuthology.orchestra.run.smithi114.stdout: "long_desc": "", 2024-04-23T10:41:09.783 INFO:teuthology.orchestra.run.smithi114.stdout: "tags": [], 2024-04-23T10:41:09.784 INFO:teuthology.orchestra.run.smithi114.stdout: "see_also": [] 2024-04-23T10:41:09.784 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-23T10:41:09.784 INFO:teuthology.orchestra.run.smithi114.stdout: "log_to_file": { 2024-04-23T10:41:09.784 INFO:teuthology.orchestra.run.smithi114.stdout: "name": "log_to_file", 2024-04-23T10:41:09.784 INFO:teuthology.orchestra.run.smithi114.stdout: "type": "bool", 2024-04-23T10:41:09.784 INFO:teuthology.orchestra.run.smithi114.stdout: "level": "advanced", 2024-04-23T10:41:09.784 INFO:teuthology.orchestra.run.smithi114.stdout: "flags": 1, 2024-04-23T10:41:09.784 INFO:teuthology.orchestra.run.smithi114.stdout: "default_value": "False", 2024-04-23T10:41:09.784 INFO:teuthology.orchestra.run.smithi114.stdout: "min": "", 2024-04-23T10:41:09.784 INFO:teuthology.orchestra.run.smithi114.stdout: "max": "", 2024-04-23T10:41:09.785 INFO:teuthology.orchestra.run.smithi114.stdout: "enum_allowed": [], 2024-04-23T10:41:09.785 INFO:teuthology.orchestra.run.smithi114.stdout: "desc": "", 2024-04-23T10:41:09.785 INFO:teuthology.orchestra.run.smithi114.stdout: "long_desc": "", 2024-04-23T10:41:09.785 INFO:teuthology.orchestra.run.smithi114.stdout: "tags": [], 2024-04-23T10:41:09.785 INFO:teuthology.orchestra.run.smithi114.stdout: "see_also": [] 2024-04-23T10:41:09.785 INFO:teuthology.orchestra.run.smithi114.stdout: } 2024-04-23T10:41:09.785 INFO:teuthology.orchestra.run.smithi114.stdout: } 2024-04-23T10:41:09.785 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-23T10:41:09.785 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-04-23T10:41:09.786 INFO:teuthology.orchestra.run.smithi114.stdout: "name": "localpool", 2024-04-23T10:41:09.786 INFO:teuthology.orchestra.run.smithi114.stdout: "can_run": true, 2024-04-23T10:41:09.786 INFO:teuthology.orchestra.run.smithi114.stdout: "error_string": "", 2024-04-23T10:41:09.786 INFO:teuthology.orchestra.run.smithi114.stdout: "module_options": { 2024-04-23T10:41:09.786 INFO:teuthology.orchestra.run.smithi114.stdout: "failure_domain": { 2024-04-23T10:41:09.786 INFO:teuthology.orchestra.run.smithi114.stdout: "name": "failure_domain", 2024-04-23T10:41:09.786 INFO:teuthology.orchestra.run.smithi114.stdout: "type": "str", 2024-04-23T10:41:09.786 INFO:teuthology.orchestra.run.smithi114.stdout: "level": "advanced", 2024-04-23T10:41:09.786 INFO:teuthology.orchestra.run.smithi114.stdout: "flags": 1, 2024-04-23T10:41:09.786 INFO:teuthology.orchestra.run.smithi114.stdout: "default_value": "host", 2024-04-23T10:41:09.787 INFO:teuthology.orchestra.run.smithi114.stdout: "min": "", 2024-04-23T10:41:09.787 INFO:teuthology.orchestra.run.smithi114.stdout: "max": "", 2024-04-23T10:41:09.787 INFO:teuthology.orchestra.run.smithi114.stdout: "enum_allowed": [], 2024-04-23T10:41:09.787 INFO:teuthology.orchestra.run.smithi114.stdout: "desc": "failure domain for any created local pool", 2024-04-23T10:41:09.787 INFO:teuthology.orchestra.run.smithi114.stdout: "long_desc": "what failure domain we should separate data replicas across.", 2024-04-23T10:41:09.787 INFO:teuthology.orchestra.run.smithi114.stdout: "tags": [], 2024-04-23T10:41:09.787 INFO:teuthology.orchestra.run.smithi114.stdout: "see_also": [] 2024-04-23T10:41:09.787 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-23T10:41:09.787 INFO:teuthology.orchestra.run.smithi114.stdout: "log_level": { 2024-04-23T10:41:09.787 INFO:teuthology.orchestra.run.smithi114.stdout: "name": "log_level", 2024-04-23T10:41:09.788 INFO:teuthology.orchestra.run.smithi114.stdout: "type": "str", 2024-04-23T10:41:09.788 INFO:teuthology.orchestra.run.smithi114.stdout: "level": "advanced", 2024-04-23T10:41:09.788 INFO:teuthology.orchestra.run.smithi114.stdout: "flags": 1, 2024-04-23T10:41:09.788 INFO:teuthology.orchestra.run.smithi114.stdout: "default_value": "", 2024-04-23T10:41:09.788 INFO:teuthology.orchestra.run.smithi114.stdout: "min": "", 2024-04-23T10:41:09.788 INFO:teuthology.orchestra.run.smithi114.stdout: "max": "", 2024-04-23T10:41:09.788 INFO:teuthology.orchestra.run.smithi114.stdout: "enum_allowed": [ 2024-04-23T10:41:09.788 INFO:teuthology.orchestra.run.smithi114.stdout: "", 2024-04-23T10:41:09.788 INFO:teuthology.orchestra.run.smithi114.stdout: "critical", 2024-04-23T10:41:09.788 INFO:teuthology.orchestra.run.smithi114.stdout: "debug", 2024-04-23T10:41:09.789 INFO:teuthology.orchestra.run.smithi114.stdout: "error", 2024-04-23T10:41:09.789 INFO:teuthology.orchestra.run.smithi114.stdout: "info", 2024-04-23T10:41:09.789 INFO:teuthology.orchestra.run.smithi114.stdout: "warning" 2024-04-23T10:41:09.789 INFO:teuthology.orchestra.run.smithi114.stdout: ], 2024-04-23T10:41:09.789 INFO:teuthology.orchestra.run.smithi114.stdout: "desc": "", 2024-04-23T10:41:09.789 INFO:teuthology.orchestra.run.smithi114.stdout: "long_desc": "", 2024-04-23T10:41:09.789 INFO:teuthology.orchestra.run.smithi114.stdout: "tags": [], 2024-04-23T10:41:09.789 INFO:teuthology.orchestra.run.smithi114.stdout: "see_also": [] 2024-04-23T10:41:09.789 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-23T10:41:09.790 INFO:teuthology.orchestra.run.smithi114.stdout: "log_to_cluster": { 2024-04-23T10:41:09.790 INFO:teuthology.orchestra.run.smithi114.stdout: "name": "log_to_cluster", 2024-04-23T10:41:09.790 INFO:teuthology.orchestra.run.smithi114.stdout: "type": "bool", 2024-04-23T10:41:09.790 INFO:teuthology.orchestra.run.smithi114.stdout: "level": "advanced", 2024-04-23T10:41:09.790 INFO:teuthology.orchestra.run.smithi114.stdout: "flags": 1, 2024-04-23T10:41:09.790 INFO:teuthology.orchestra.run.smithi114.stdout: "default_value": "False", 2024-04-23T10:41:09.790 INFO:teuthology.orchestra.run.smithi114.stdout: "min": "", 2024-04-23T10:41:09.790 INFO:teuthology.orchestra.run.smithi114.stdout: "max": "", 2024-04-23T10:41:09.790 INFO:teuthology.orchestra.run.smithi114.stdout: "enum_allowed": [], 2024-04-23T10:41:09.791 INFO:teuthology.orchestra.run.smithi114.stdout: "desc": "", 2024-04-23T10:41:09.791 INFO:teuthology.orchestra.run.smithi114.stdout: "long_desc": "", 2024-04-23T10:41:09.791 INFO:teuthology.orchestra.run.smithi114.stdout: "tags": [], 2024-04-23T10:41:09.791 INFO:teuthology.orchestra.run.smithi114.stdout: "see_also": [] 2024-04-23T10:41:09.791 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-23T10:41:09.791 INFO:teuthology.orchestra.run.smithi114.stdout: "log_to_cluster_level": { 2024-04-23T10:41:09.791 INFO:teuthology.orchestra.run.smithi114.stdout: "name": "log_to_cluster_level", 2024-04-23T10:41:09.791 INFO:teuthology.orchestra.run.smithi114.stdout: "type": "str", 2024-04-23T10:41:09.791 INFO:teuthology.orchestra.run.smithi114.stdout: "level": "advanced", 2024-04-23T10:41:09.791 INFO:teuthology.orchestra.run.smithi114.stdout: "flags": 1, 2024-04-23T10:41:09.792 INFO:teuthology.orchestra.run.smithi114.stdout: "default_value": "info", 2024-04-23T10:41:09.792 INFO:teuthology.orchestra.run.smithi114.stdout: "min": "", 2024-04-23T10:41:09.792 INFO:teuthology.orchestra.run.smithi114.stdout: "max": "", 2024-04-23T10:41:09.792 INFO:teuthology.orchestra.run.smithi114.stdout: "enum_allowed": [ 2024-04-23T10:41:09.792 INFO:teuthology.orchestra.run.smithi114.stdout: "", 2024-04-23T10:41:09.792 INFO:teuthology.orchestra.run.smithi114.stdout: "critical", 2024-04-23T10:41:09.792 INFO:teuthology.orchestra.run.smithi114.stdout: "debug", 2024-04-23T10:41:09.792 INFO:teuthology.orchestra.run.smithi114.stdout: "error", 2024-04-23T10:41:09.792 INFO:teuthology.orchestra.run.smithi114.stdout: "info", 2024-04-23T10:41:09.792 INFO:teuthology.orchestra.run.smithi114.stdout: "warning" 2024-04-23T10:41:09.793 INFO:teuthology.orchestra.run.smithi114.stdout: ], 2024-04-23T10:41:09.793 INFO:teuthology.orchestra.run.smithi114.stdout: "desc": "", 2024-04-23T10:41:09.793 INFO:teuthology.orchestra.run.smithi114.stdout: "long_desc": "", 2024-04-23T10:41:09.793 INFO:teuthology.orchestra.run.smithi114.stdout: "tags": [], 2024-04-23T10:41:09.793 INFO:teuthology.orchestra.run.smithi114.stdout: "see_also": [] 2024-04-23T10:41:09.793 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-23T10:41:09.793 INFO:teuthology.orchestra.run.smithi114.stdout: "log_to_file": { 2024-04-23T10:41:09.793 INFO:teuthology.orchestra.run.smithi114.stdout: "name": "log_to_file", 2024-04-23T10:41:09.793 INFO:teuthology.orchestra.run.smithi114.stdout: "type": "bool", 2024-04-23T10:41:09.794 INFO:teuthology.orchestra.run.smithi114.stdout: "level": "advanced", 2024-04-23T10:41:09.794 INFO:teuthology.orchestra.run.smithi114.stdout: "flags": 1, 2024-04-23T10:41:09.794 INFO:teuthology.orchestra.run.smithi114.stdout: "default_value": "False", 2024-04-23T10:41:09.794 INFO:teuthology.orchestra.run.smithi114.stdout: "min": "", 2024-04-23T10:41:09.794 INFO:teuthology.orchestra.run.smithi114.stdout: "max": "", 2024-04-23T10:41:09.794 INFO:teuthology.orchestra.run.smithi114.stdout: "enum_allowed": [], 2024-04-23T10:41:09.794 INFO:teuthology.orchestra.run.smithi114.stdout: "desc": "", 2024-04-23T10:41:09.794 INFO:teuthology.orchestra.run.smithi114.stdout: "long_desc": "", 2024-04-23T10:41:09.794 INFO:teuthology.orchestra.run.smithi114.stdout: "tags": [], 2024-04-23T10:41:09.794 INFO:teuthology.orchestra.run.smithi114.stdout: "see_also": [] 2024-04-23T10:41:09.795 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-23T10:41:09.795 INFO:teuthology.orchestra.run.smithi114.stdout: "min_size": { 2024-04-23T10:41:09.795 INFO:teuthology.orchestra.run.smithi114.stdout: "name": "min_size", 2024-04-23T10:41:09.795 INFO:teuthology.orchestra.run.smithi114.stdout: "type": "int", 2024-04-23T10:41:09.795 INFO:teuthology.orchestra.run.smithi114.stdout: "level": "advanced", 2024-04-23T10:41:09.795 INFO:teuthology.orchestra.run.smithi114.stdout: "flags": 1, 2024-04-23T10:41:09.795 INFO:teuthology.orchestra.run.smithi114.stdout: "default_value": "", 2024-04-23T10:41:09.795 INFO:teuthology.orchestra.run.smithi114.stdout: "min": "", 2024-04-23T10:41:09.795 INFO:teuthology.orchestra.run.smithi114.stdout: "max": "", 2024-04-23T10:41:09.796 INFO:teuthology.orchestra.run.smithi114.stdout: "enum_allowed": [], 2024-04-23T10:41:09.796 INFO:teuthology.orchestra.run.smithi114.stdout: "desc": "default min_size for any created local pool", 2024-04-23T10:41:09.796 INFO:teuthology.orchestra.run.smithi114.stdout: "long_desc": "value to set min_size to (unchanged from Ceph's default if this option is not set)", 2024-04-23T10:41:09.796 INFO:teuthology.orchestra.run.smithi114.stdout: "tags": [], 2024-04-23T10:41:09.796 INFO:teuthology.orchestra.run.smithi114.stdout: "see_also": [] 2024-04-23T10:41:09.796 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-23T10:41:09.796 INFO:teuthology.orchestra.run.smithi114.stdout: "num_rep": { 2024-04-23T10:41:09.796 INFO:teuthology.orchestra.run.smithi114.stdout: "name": "num_rep", 2024-04-23T10:41:09.796 INFO:teuthology.orchestra.run.smithi114.stdout: "type": "int", 2024-04-23T10:41:09.796 INFO:teuthology.orchestra.run.smithi114.stdout: "level": "advanced", 2024-04-23T10:41:09.797 INFO:teuthology.orchestra.run.smithi114.stdout: "flags": 1, 2024-04-23T10:41:09.797 INFO:teuthology.orchestra.run.smithi114.stdout: "default_value": "3", 2024-04-23T10:41:09.797 INFO:teuthology.orchestra.run.smithi114.stdout: "min": "", 2024-04-23T10:41:09.797 INFO:teuthology.orchestra.run.smithi114.stdout: "max": "", 2024-04-23T10:41:09.797 INFO:teuthology.orchestra.run.smithi114.stdout: "enum_allowed": [], 2024-04-23T10:41:09.797 INFO:teuthology.orchestra.run.smithi114.stdout: "desc": "default replica count for any created local pool", 2024-04-23T10:41:09.797 INFO:teuthology.orchestra.run.smithi114.stdout: "long_desc": "", 2024-04-23T10:41:09.797 INFO:teuthology.orchestra.run.smithi114.stdout: "tags": [], 2024-04-23T10:41:09.797 INFO:teuthology.orchestra.run.smithi114.stdout: "see_also": [] 2024-04-23T10:41:09.798 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-23T10:41:09.798 INFO:teuthology.orchestra.run.smithi114.stdout: "pg_num": { 2024-04-23T10:41:09.798 INFO:teuthology.orchestra.run.smithi114.stdout: "name": "pg_num", 2024-04-23T10:41:09.798 INFO:teuthology.orchestra.run.smithi114.stdout: "type": "int", 2024-04-23T10:41:09.798 INFO:teuthology.orchestra.run.smithi114.stdout: "level": "advanced", 2024-04-23T10:41:09.798 INFO:teuthology.orchestra.run.smithi114.stdout: "flags": 1, 2024-04-23T10:41:09.798 INFO:teuthology.orchestra.run.smithi114.stdout: "default_value": "128", 2024-04-23T10:41:09.798 INFO:teuthology.orchestra.run.smithi114.stdout: "min": "", 2024-04-23T10:41:09.798 INFO:teuthology.orchestra.run.smithi114.stdout: "max": "", 2024-04-23T10:41:09.798 INFO:teuthology.orchestra.run.smithi114.stdout: "enum_allowed": [], 2024-04-23T10:41:09.799 INFO:teuthology.orchestra.run.smithi114.stdout: "desc": "default pg_num for any created local pool", 2024-04-23T10:41:09.799 INFO:teuthology.orchestra.run.smithi114.stdout: "long_desc": "", 2024-04-23T10:41:09.799 INFO:teuthology.orchestra.run.smithi114.stdout: "tags": [], 2024-04-23T10:41:09.799 INFO:teuthology.orchestra.run.smithi114.stdout: "see_also": [] 2024-04-23T10:41:09.799 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-23T10:41:09.799 INFO:teuthology.orchestra.run.smithi114.stdout: "prefix": { 2024-04-23T10:41:09.799 INFO:teuthology.orchestra.run.smithi114.stdout: "name": "prefix", 2024-04-23T10:41:09.799 INFO:teuthology.orchestra.run.smithi114.stdout: "type": "str", 2024-04-23T10:41:09.799 INFO:teuthology.orchestra.run.smithi114.stdout: "level": "advanced", 2024-04-23T10:41:09.799 INFO:teuthology.orchestra.run.smithi114.stdout: "flags": 1, 2024-04-23T10:41:09.800 INFO:teuthology.orchestra.run.smithi114.stdout: "default_value": "", 2024-04-23T10:41:09.800 INFO:teuthology.orchestra.run.smithi114.stdout: "min": "", 2024-04-23T10:41:09.800 INFO:teuthology.orchestra.run.smithi114.stdout: "max": "", 2024-04-23T10:41:09.800 INFO:teuthology.orchestra.run.smithi114.stdout: "enum_allowed": [], 2024-04-23T10:41:09.800 INFO:teuthology.orchestra.run.smithi114.stdout: "desc": "name prefix for any created local pool", 2024-04-23T10:41:09.800 INFO:teuthology.orchestra.run.smithi114.stdout: "long_desc": "", 2024-04-23T10:41:09.800 INFO:teuthology.orchestra.run.smithi114.stdout: "tags": [], 2024-04-23T10:41:09.800 INFO:teuthology.orchestra.run.smithi114.stdout: "see_also": [] 2024-04-23T10:41:09.800 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-23T10:41:09.800 INFO:teuthology.orchestra.run.smithi114.stdout: "subtree": { 2024-04-23T10:41:09.801 INFO:teuthology.orchestra.run.smithi114.stdout: "name": "subtree", 2024-04-23T10:41:09.801 INFO:teuthology.orchestra.run.smithi114.stdout: "type": "str", 2024-04-23T10:41:09.801 INFO:teuthology.orchestra.run.smithi114.stdout: "level": "advanced", 2024-04-23T10:41:09.801 INFO:teuthology.orchestra.run.smithi114.stdout: "flags": 1, 2024-04-23T10:41:09.801 INFO:teuthology.orchestra.run.smithi114.stdout: "default_value": "rack", 2024-04-23T10:41:09.801 INFO:teuthology.orchestra.run.smithi114.stdout: "min": "", 2024-04-23T10:41:09.801 INFO:teuthology.orchestra.run.smithi114.stdout: "max": "", 2024-04-23T10:41:09.801 INFO:teuthology.orchestra.run.smithi114.stdout: "enum_allowed": [], 2024-04-23T10:41:09.801 INFO:teuthology.orchestra.run.smithi114.stdout: "desc": "CRUSH level for which to create a local pool", 2024-04-23T10:41:09.801 INFO:teuthology.orchestra.run.smithi114.stdout: "long_desc": "which CRUSH subtree type the module should create a pool for.", 2024-04-23T10:41:09.802 INFO:teuthology.orchestra.run.smithi114.stdout: "tags": [], 2024-04-23T10:41:09.802 INFO:teuthology.orchestra.run.smithi114.stdout: "see_also": [] 2024-04-23T10:41:09.802 INFO:teuthology.orchestra.run.smithi114.stdout: } 2024-04-23T10:41:09.802 INFO:teuthology.orchestra.run.smithi114.stdout: } 2024-04-23T10:41:09.802 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-23T10:41:09.802 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-04-23T10:41:09.802 INFO:teuthology.orchestra.run.smithi114.stdout: "name": "mds_autoscaler", 2024-04-23T10:41:09.802 INFO:teuthology.orchestra.run.smithi114.stdout: "can_run": true, 2024-04-23T10:41:09.802 INFO:teuthology.orchestra.run.smithi114.stdout: "error_string": "", 2024-04-23T10:41:09.803 INFO:teuthology.orchestra.run.smithi114.stdout: "module_options": { 2024-04-23T10:41:09.803 INFO:teuthology.orchestra.run.smithi114.stdout: "log_level": { 2024-04-23T10:41:09.803 INFO:teuthology.orchestra.run.smithi114.stdout: "name": "log_level", 2024-04-23T10:41:09.803 INFO:teuthology.orchestra.run.smithi114.stdout: "type": "str", 2024-04-23T10:41:09.803 INFO:teuthology.orchestra.run.smithi114.stdout: "level": "advanced", 2024-04-23T10:41:09.803 INFO:teuthology.orchestra.run.smithi114.stdout: "flags": 1, 2024-04-23T10:41:09.803 INFO:teuthology.orchestra.run.smithi114.stdout: "default_value": "", 2024-04-23T10:41:09.803 INFO:teuthology.orchestra.run.smithi114.stdout: "min": "", 2024-04-23T10:41:09.803 INFO:teuthology.orchestra.run.smithi114.stdout: "max": "", 2024-04-23T10:41:09.803 INFO:teuthology.orchestra.run.smithi114.stdout: "enum_allowed": [ 2024-04-23T10:41:09.804 INFO:teuthology.orchestra.run.smithi114.stdout: "", 2024-04-23T10:41:09.804 INFO:teuthology.orchestra.run.smithi114.stdout: "critical", 2024-04-23T10:41:09.804 INFO:teuthology.orchestra.run.smithi114.stdout: "debug", 2024-04-23T10:41:09.804 INFO:teuthology.orchestra.run.smithi114.stdout: "error", 2024-04-23T10:41:09.804 INFO:teuthology.orchestra.run.smithi114.stdout: "info", 2024-04-23T10:41:09.804 INFO:teuthology.orchestra.run.smithi114.stdout: "warning" 2024-04-23T10:41:09.804 INFO:teuthology.orchestra.run.smithi114.stdout: ], 2024-04-23T10:41:09.804 INFO:teuthology.orchestra.run.smithi114.stdout: "desc": "", 2024-04-23T10:41:09.804 INFO:teuthology.orchestra.run.smithi114.stdout: "long_desc": "", 2024-04-23T10:41:09.804 INFO:teuthology.orchestra.run.smithi114.stdout: "tags": [], 2024-04-23T10:41:09.805 INFO:teuthology.orchestra.run.smithi114.stdout: "see_also": [] 2024-04-23T10:41:09.805 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-23T10:41:09.805 INFO:teuthology.orchestra.run.smithi114.stdout: "log_to_cluster": { 2024-04-23T10:41:09.805 INFO:teuthology.orchestra.run.smithi114.stdout: "name": "log_to_cluster", 2024-04-23T10:41:09.805 INFO:teuthology.orchestra.run.smithi114.stdout: "type": "bool", 2024-04-23T10:41:09.805 INFO:teuthology.orchestra.run.smithi114.stdout: "level": "advanced", 2024-04-23T10:41:09.805 INFO:teuthology.orchestra.run.smithi114.stdout: "flags": 1, 2024-04-23T10:41:09.805 INFO:teuthology.orchestra.run.smithi114.stdout: "default_value": "False", 2024-04-23T10:41:09.805 INFO:teuthology.orchestra.run.smithi114.stdout: "min": "", 2024-04-23T10:41:09.805 INFO:teuthology.orchestra.run.smithi114.stdout: "max": "", 2024-04-23T10:41:09.806 INFO:teuthology.orchestra.run.smithi114.stdout: "enum_allowed": [], 2024-04-23T10:41:09.806 INFO:teuthology.orchestra.run.smithi114.stdout: "desc": "", 2024-04-23T10:41:09.806 INFO:teuthology.orchestra.run.smithi114.stdout: "long_desc": "", 2024-04-23T10:41:09.806 INFO:teuthology.orchestra.run.smithi114.stdout: "tags": [], 2024-04-23T10:41:09.806 INFO:teuthology.orchestra.run.smithi114.stdout: "see_also": [] 2024-04-23T10:41:09.806 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-23T10:41:09.806 INFO:teuthology.orchestra.run.smithi114.stdout: "log_to_cluster_level": { 2024-04-23T10:41:09.806 INFO:teuthology.orchestra.run.smithi114.stdout: "name": "log_to_cluster_level", 2024-04-23T10:41:09.806 INFO:teuthology.orchestra.run.smithi114.stdout: "type": "str", 2024-04-23T10:41:09.806 INFO:teuthology.orchestra.run.smithi114.stdout: "level": "advanced", 2024-04-23T10:41:09.807 INFO:teuthology.orchestra.run.smithi114.stdout: "flags": 1, 2024-04-23T10:41:09.807 INFO:teuthology.orchestra.run.smithi114.stdout: "default_value": "info", 2024-04-23T10:41:09.807 INFO:teuthology.orchestra.run.smithi114.stdout: "min": "", 2024-04-23T10:41:09.807 INFO:teuthology.orchestra.run.smithi114.stdout: "max": "", 2024-04-23T10:41:09.807 INFO:teuthology.orchestra.run.smithi114.stdout: "enum_allowed": [ 2024-04-23T10:41:09.807 INFO:teuthology.orchestra.run.smithi114.stdout: "", 2024-04-23T10:41:09.807 INFO:teuthology.orchestra.run.smithi114.stdout: "critical", 2024-04-23T10:41:09.807 INFO:teuthology.orchestra.run.smithi114.stdout: "debug", 2024-04-23T10:41:09.807 INFO:teuthology.orchestra.run.smithi114.stdout: "error", 2024-04-23T10:41:09.807 INFO:teuthology.orchestra.run.smithi114.stdout: "info", 2024-04-23T10:41:09.808 INFO:teuthology.orchestra.run.smithi114.stdout: "warning" 2024-04-23T10:41:09.808 INFO:teuthology.orchestra.run.smithi114.stdout: ], 2024-04-23T10:41:09.808 INFO:teuthology.orchestra.run.smithi114.stdout: "desc": "", 2024-04-23T10:41:09.808 INFO:teuthology.orchestra.run.smithi114.stdout: "long_desc": "", 2024-04-23T10:41:09.808 INFO:teuthology.orchestra.run.smithi114.stdout: "tags": [], 2024-04-23T10:41:09.808 INFO:teuthology.orchestra.run.smithi114.stdout: "see_also": [] 2024-04-23T10:41:09.808 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-23T10:41:09.808 INFO:teuthology.orchestra.run.smithi114.stdout: "log_to_file": { 2024-04-23T10:41:09.808 INFO:teuthology.orchestra.run.smithi114.stdout: "name": "log_to_file", 2024-04-23T10:41:09.808 INFO:teuthology.orchestra.run.smithi114.stdout: "type": "bool", 2024-04-23T10:41:09.809 INFO:teuthology.orchestra.run.smithi114.stdout: "level": "advanced", 2024-04-23T10:41:09.809 INFO:teuthology.orchestra.run.smithi114.stdout: "flags": 1, 2024-04-23T10:41:09.809 INFO:teuthology.orchestra.run.smithi114.stdout: "default_value": "False", 2024-04-23T10:41:09.809 INFO:teuthology.orchestra.run.smithi114.stdout: "min": "", 2024-04-23T10:41:09.809 INFO:teuthology.orchestra.run.smithi114.stdout: "max": "", 2024-04-23T10:41:09.809 INFO:teuthology.orchestra.run.smithi114.stdout: "enum_allowed": [], 2024-04-23T10:41:09.809 INFO:teuthology.orchestra.run.smithi114.stdout: "desc": "", 2024-04-23T10:41:09.809 INFO:teuthology.orchestra.run.smithi114.stdout: "long_desc": "", 2024-04-23T10:41:09.809 INFO:teuthology.orchestra.run.smithi114.stdout: "tags": [], 2024-04-23T10:41:09.809 INFO:teuthology.orchestra.run.smithi114.stdout: "see_also": [] 2024-04-23T10:41:09.810 INFO:teuthology.orchestra.run.smithi114.stdout: } 2024-04-23T10:41:09.810 INFO:teuthology.orchestra.run.smithi114.stdout: } 2024-04-23T10:41:09.810 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-23T10:41:09.810 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-04-23T10:41:09.810 INFO:teuthology.orchestra.run.smithi114.stdout: "name": "mirroring", 2024-04-23T10:41:09.810 INFO:teuthology.orchestra.run.smithi114.stdout: "can_run": true, 2024-04-23T10:41:09.810 INFO:teuthology.orchestra.run.smithi114.stdout: "error_string": "", 2024-04-23T10:41:09.810 INFO:teuthology.orchestra.run.smithi114.stdout: "module_options": { 2024-04-23T10:41:09.811 INFO:teuthology.orchestra.run.smithi114.stdout: "log_level": { 2024-04-23T10:41:09.811 INFO:teuthology.orchestra.run.smithi114.stdout: "name": "log_level", 2024-04-23T10:41:09.811 INFO:teuthology.orchestra.run.smithi114.stdout: "type": "str", 2024-04-23T10:41:09.811 INFO:teuthology.orchestra.run.smithi114.stdout: "level": "advanced", 2024-04-23T10:41:09.811 INFO:teuthology.orchestra.run.smithi114.stdout: "flags": 1, 2024-04-23T10:41:09.811 INFO:teuthology.orchestra.run.smithi114.stdout: "default_value": "", 2024-04-23T10:41:09.811 INFO:teuthology.orchestra.run.smithi114.stdout: "min": "", 2024-04-23T10:41:09.811 INFO:teuthology.orchestra.run.smithi114.stdout: "max": "", 2024-04-23T10:41:09.811 INFO:teuthology.orchestra.run.smithi114.stdout: "enum_allowed": [ 2024-04-23T10:41:09.811 INFO:teuthology.orchestra.run.smithi114.stdout: "", 2024-04-23T10:41:09.812 INFO:teuthology.orchestra.run.smithi114.stdout: "critical", 2024-04-23T10:41:09.812 INFO:teuthology.orchestra.run.smithi114.stdout: "debug", 2024-04-23T10:41:09.812 INFO:teuthology.orchestra.run.smithi114.stdout: "error", 2024-04-23T10:41:09.812 INFO:teuthology.orchestra.run.smithi114.stdout: "info", 2024-04-23T10:41:09.812 INFO:teuthology.orchestra.run.smithi114.stdout: "warning" 2024-04-23T10:41:09.812 INFO:teuthology.orchestra.run.smithi114.stdout: ], 2024-04-23T10:41:09.812 INFO:teuthology.orchestra.run.smithi114.stdout: "desc": "", 2024-04-23T10:41:09.812 INFO:teuthology.orchestra.run.smithi114.stdout: "long_desc": "", 2024-04-23T10:41:09.812 INFO:teuthology.orchestra.run.smithi114.stdout: "tags": [], 2024-04-23T10:41:09.812 INFO:teuthology.orchestra.run.smithi114.stdout: "see_also": [] 2024-04-23T10:41:09.813 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-23T10:41:09.813 INFO:teuthology.orchestra.run.smithi114.stdout: "log_to_cluster": { 2024-04-23T10:41:09.813 INFO:teuthology.orchestra.run.smithi114.stdout: "name": "log_to_cluster", 2024-04-23T10:41:09.813 INFO:teuthology.orchestra.run.smithi114.stdout: "type": "bool", 2024-04-23T10:41:09.813 INFO:teuthology.orchestra.run.smithi114.stdout: "level": "advanced", 2024-04-23T10:41:09.813 INFO:teuthology.orchestra.run.smithi114.stdout: "flags": 1, 2024-04-23T10:41:09.813 INFO:teuthology.orchestra.run.smithi114.stdout: "default_value": "False", 2024-04-23T10:41:09.813 INFO:teuthology.orchestra.run.smithi114.stdout: "min": "", 2024-04-23T10:41:09.813 INFO:teuthology.orchestra.run.smithi114.stdout: "max": "", 2024-04-23T10:41:09.814 INFO:teuthology.orchestra.run.smithi114.stdout: "enum_allowed": [], 2024-04-23T10:41:09.814 INFO:teuthology.orchestra.run.smithi114.stdout: "desc": "", 2024-04-23T10:41:09.814 INFO:teuthology.orchestra.run.smithi114.stdout: "long_desc": "", 2024-04-23T10:41:09.814 INFO:teuthology.orchestra.run.smithi114.stdout: "tags": [], 2024-04-23T10:41:09.814 INFO:teuthology.orchestra.run.smithi114.stdout: "see_also": [] 2024-04-23T10:41:09.814 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-23T10:41:09.814 INFO:teuthology.orchestra.run.smithi114.stdout: "log_to_cluster_level": { 2024-04-23T10:41:09.814 INFO:teuthology.orchestra.run.smithi114.stdout: "name": "log_to_cluster_level", 2024-04-23T10:41:09.814 INFO:teuthology.orchestra.run.smithi114.stdout: "type": "str", 2024-04-23T10:41:09.814 INFO:teuthology.orchestra.run.smithi114.stdout: "level": "advanced", 2024-04-23T10:41:09.815 INFO:teuthology.orchestra.run.smithi114.stdout: "flags": 1, 2024-04-23T10:41:09.815 INFO:teuthology.orchestra.run.smithi114.stdout: "default_value": "info", 2024-04-23T10:41:09.815 INFO:teuthology.orchestra.run.smithi114.stdout: "min": "", 2024-04-23T10:41:09.815 INFO:teuthology.orchestra.run.smithi114.stdout: "max": "", 2024-04-23T10:41:09.815 INFO:teuthology.orchestra.run.smithi114.stdout: "enum_allowed": [ 2024-04-23T10:41:09.815 INFO:teuthology.orchestra.run.smithi114.stdout: "", 2024-04-23T10:41:09.815 INFO:teuthology.orchestra.run.smithi114.stdout: "critical", 2024-04-23T10:41:09.815 INFO:teuthology.orchestra.run.smithi114.stdout: "debug", 2024-04-23T10:41:09.815 INFO:teuthology.orchestra.run.smithi114.stdout: "error", 2024-04-23T10:41:09.816 INFO:teuthology.orchestra.run.smithi114.stdout: "info", 2024-04-23T10:41:09.816 INFO:teuthology.orchestra.run.smithi114.stdout: "warning" 2024-04-23T10:41:09.816 INFO:teuthology.orchestra.run.smithi114.stdout: ], 2024-04-23T10:41:09.816 INFO:teuthology.orchestra.run.smithi114.stdout: "desc": "", 2024-04-23T10:41:09.816 INFO:teuthology.orchestra.run.smithi114.stdout: "long_desc": "", 2024-04-23T10:41:09.816 INFO:teuthology.orchestra.run.smithi114.stdout: "tags": [], 2024-04-23T10:41:09.816 INFO:teuthology.orchestra.run.smithi114.stdout: "see_also": [] 2024-04-23T10:41:09.816 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-23T10:41:09.816 INFO:teuthology.orchestra.run.smithi114.stdout: "log_to_file": { 2024-04-23T10:41:09.816 INFO:teuthology.orchestra.run.smithi114.stdout: "name": "log_to_file", 2024-04-23T10:41:09.817 INFO:teuthology.orchestra.run.smithi114.stdout: "type": "bool", 2024-04-23T10:41:09.817 INFO:teuthology.orchestra.run.smithi114.stdout: "level": "advanced", 2024-04-23T10:41:09.817 INFO:teuthology.orchestra.run.smithi114.stdout: "flags": 1, 2024-04-23T10:41:09.817 INFO:teuthology.orchestra.run.smithi114.stdout: "default_value": "False", 2024-04-23T10:41:09.817 INFO:teuthology.orchestra.run.smithi114.stdout: "min": "", 2024-04-23T10:41:09.817 INFO:teuthology.orchestra.run.smithi114.stdout: "max": "", 2024-04-23T10:41:09.817 INFO:teuthology.orchestra.run.smithi114.stdout: "enum_allowed": [], 2024-04-23T10:41:09.817 INFO:teuthology.orchestra.run.smithi114.stdout: "desc": "", 2024-04-23T10:41:09.817 INFO:teuthology.orchestra.run.smithi114.stdout: "long_desc": "", 2024-04-23T10:41:09.817 INFO:teuthology.orchestra.run.smithi114.stdout: "tags": [], 2024-04-23T10:41:09.818 INFO:teuthology.orchestra.run.smithi114.stdout: "see_also": [] 2024-04-23T10:41:09.818 INFO:teuthology.orchestra.run.smithi114.stdout: } 2024-04-23T10:41:09.818 INFO:teuthology.orchestra.run.smithi114.stdout: } 2024-04-23T10:41:09.818 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-23T10:41:09.818 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-04-23T10:41:09.818 INFO:teuthology.orchestra.run.smithi114.stdout: "name": "nfs", 2024-04-23T10:41:09.818 INFO:teuthology.orchestra.run.smithi114.stdout: "can_run": true, 2024-04-23T10:41:09.818 INFO:teuthology.orchestra.run.smithi114.stdout: "error_string": "", 2024-04-23T10:41:09.818 INFO:teuthology.orchestra.run.smithi114.stdout: "module_options": { 2024-04-23T10:41:09.819 INFO:teuthology.orchestra.run.smithi114.stdout: "log_level": { 2024-04-23T10:41:09.819 INFO:teuthology.orchestra.run.smithi114.stdout: "name": "log_level", 2024-04-23T10:41:09.819 INFO:teuthology.orchestra.run.smithi114.stdout: "type": "str", 2024-04-23T10:41:09.819 INFO:teuthology.orchestra.run.smithi114.stdout: "level": "advanced", 2024-04-23T10:41:09.819 INFO:teuthology.orchestra.run.smithi114.stdout: "flags": 1, 2024-04-23T10:41:09.819 INFO:teuthology.orchestra.run.smithi114.stdout: "default_value": "", 2024-04-23T10:41:09.819 INFO:teuthology.orchestra.run.smithi114.stdout: "min": "", 2024-04-23T10:41:09.819 INFO:teuthology.orchestra.run.smithi114.stdout: "max": "", 2024-04-23T10:41:09.819 INFO:teuthology.orchestra.run.smithi114.stdout: "enum_allowed": [ 2024-04-23T10:41:09.819 INFO:teuthology.orchestra.run.smithi114.stdout: "", 2024-04-23T10:41:09.820 INFO:teuthology.orchestra.run.smithi114.stdout: "critical", 2024-04-23T10:41:09.820 INFO:teuthology.orchestra.run.smithi114.stdout: "debug", 2024-04-23T10:41:09.820 INFO:teuthology.orchestra.run.smithi114.stdout: "error", 2024-04-23T10:41:09.820 INFO:teuthology.orchestra.run.smithi114.stdout: "info", 2024-04-23T10:41:09.820 INFO:teuthology.orchestra.run.smithi114.stdout: "warning" 2024-04-23T10:41:09.820 INFO:teuthology.orchestra.run.smithi114.stdout: ], 2024-04-23T10:41:09.820 INFO:teuthology.orchestra.run.smithi114.stdout: "desc": "", 2024-04-23T10:41:09.820 INFO:teuthology.orchestra.run.smithi114.stdout: "long_desc": "", 2024-04-23T10:41:09.820 INFO:teuthology.orchestra.run.smithi114.stdout: "tags": [], 2024-04-23T10:41:09.820 INFO:teuthology.orchestra.run.smithi114.stdout: "see_also": [] 2024-04-23T10:41:09.821 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-23T10:41:09.821 INFO:teuthology.orchestra.run.smithi114.stdout: "log_to_cluster": { 2024-04-23T10:41:09.821 INFO:teuthology.orchestra.run.smithi114.stdout: "name": "log_to_cluster", 2024-04-23T10:41:09.821 INFO:teuthology.orchestra.run.smithi114.stdout: "type": "bool", 2024-04-23T10:41:09.821 INFO:teuthology.orchestra.run.smithi114.stdout: "level": "advanced", 2024-04-23T10:41:09.821 INFO:teuthology.orchestra.run.smithi114.stdout: "flags": 1, 2024-04-23T10:41:09.821 INFO:teuthology.orchestra.run.smithi114.stdout: "default_value": "False", 2024-04-23T10:41:09.821 INFO:teuthology.orchestra.run.smithi114.stdout: "min": "", 2024-04-23T10:41:09.821 INFO:teuthology.orchestra.run.smithi114.stdout: "max": "", 2024-04-23T10:41:09.822 INFO:teuthology.orchestra.run.smithi114.stdout: "enum_allowed": [], 2024-04-23T10:41:09.822 INFO:teuthology.orchestra.run.smithi114.stdout: "desc": "", 2024-04-23T10:41:09.822 INFO:teuthology.orchestra.run.smithi114.stdout: "long_desc": "", 2024-04-23T10:41:09.822 INFO:teuthology.orchestra.run.smithi114.stdout: "tags": [], 2024-04-23T10:41:09.822 INFO:teuthology.orchestra.run.smithi114.stdout: "see_also": [] 2024-04-23T10:41:09.822 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-23T10:41:09.822 INFO:teuthology.orchestra.run.smithi114.stdout: "log_to_cluster_level": { 2024-04-23T10:41:09.822 INFO:teuthology.orchestra.run.smithi114.stdout: "name": "log_to_cluster_level", 2024-04-23T10:41:09.822 INFO:teuthology.orchestra.run.smithi114.stdout: "type": "str", 2024-04-23T10:41:09.822 INFO:teuthology.orchestra.run.smithi114.stdout: "level": "advanced", 2024-04-23T10:41:09.823 INFO:teuthology.orchestra.run.smithi114.stdout: "flags": 1, 2024-04-23T10:41:09.823 INFO:teuthology.orchestra.run.smithi114.stdout: "default_value": "info", 2024-04-23T10:41:09.823 INFO:teuthology.orchestra.run.smithi114.stdout: "min": "", 2024-04-23T10:41:09.823 INFO:teuthology.orchestra.run.smithi114.stdout: "max": "", 2024-04-23T10:41:09.823 INFO:teuthology.orchestra.run.smithi114.stdout: "enum_allowed": [ 2024-04-23T10:41:09.823 INFO:teuthology.orchestra.run.smithi114.stdout: "", 2024-04-23T10:41:09.823 INFO:teuthology.orchestra.run.smithi114.stdout: "critical", 2024-04-23T10:41:09.823 INFO:teuthology.orchestra.run.smithi114.stdout: "debug", 2024-04-23T10:41:09.823 INFO:teuthology.orchestra.run.smithi114.stdout: "error", 2024-04-23T10:41:09.823 INFO:teuthology.orchestra.run.smithi114.stdout: "info", 2024-04-23T10:41:09.824 INFO:teuthology.orchestra.run.smithi114.stdout: "warning" 2024-04-23T10:41:09.824 INFO:teuthology.orchestra.run.smithi114.stdout: ], 2024-04-23T10:41:09.824 INFO:teuthology.orchestra.run.smithi114.stdout: "desc": "", 2024-04-23T10:41:09.824 INFO:teuthology.orchestra.run.smithi114.stdout: "long_desc": "", 2024-04-23T10:41:09.824 INFO:teuthology.orchestra.run.smithi114.stdout: "tags": [], 2024-04-23T10:41:09.824 INFO:teuthology.orchestra.run.smithi114.stdout: "see_also": [] 2024-04-23T10:41:09.824 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-23T10:41:09.824 INFO:teuthology.orchestra.run.smithi114.stdout: "log_to_file": { 2024-04-23T10:41:09.824 INFO:teuthology.orchestra.run.smithi114.stdout: "name": "log_to_file", 2024-04-23T10:41:09.825 INFO:teuthology.orchestra.run.smithi114.stdout: "type": "bool", 2024-04-23T10:41:09.825 INFO:teuthology.orchestra.run.smithi114.stdout: "level": "advanced", 2024-04-23T10:41:09.825 INFO:teuthology.orchestra.run.smithi114.stdout: "flags": 1, 2024-04-23T10:41:09.825 INFO:teuthology.orchestra.run.smithi114.stdout: "default_value": "False", 2024-04-23T10:41:09.825 INFO:teuthology.orchestra.run.smithi114.stdout: "min": "", 2024-04-23T10:41:09.825 INFO:teuthology.orchestra.run.smithi114.stdout: "max": "", 2024-04-23T10:41:09.825 INFO:teuthology.orchestra.run.smithi114.stdout: "enum_allowed": [], 2024-04-23T10:41:09.825 INFO:teuthology.orchestra.run.smithi114.stdout: "desc": "", 2024-04-23T10:41:09.825 INFO:teuthology.orchestra.run.smithi114.stdout: "long_desc": "", 2024-04-23T10:41:09.825 INFO:teuthology.orchestra.run.smithi114.stdout: "tags": [], 2024-04-23T10:41:09.826 INFO:teuthology.orchestra.run.smithi114.stdout: "see_also": [] 2024-04-23T10:41:09.826 INFO:teuthology.orchestra.run.smithi114.stdout: } 2024-04-23T10:41:09.826 INFO:teuthology.orchestra.run.smithi114.stdout: } 2024-04-23T10:41:09.826 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-23T10:41:09.826 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-04-23T10:41:09.826 INFO:teuthology.orchestra.run.smithi114.stdout: "name": "orchestrator", 2024-04-23T10:41:09.826 INFO:teuthology.orchestra.run.smithi114.stdout: "can_run": true, 2024-04-23T10:41:09.826 INFO:teuthology.orchestra.run.smithi114.stdout: "error_string": "", 2024-04-23T10:41:09.826 INFO:teuthology.orchestra.run.smithi114.stdout: "module_options": { 2024-04-23T10:41:09.826 INFO:teuthology.orchestra.run.smithi114.stdout: "fail_fs": { 2024-04-23T10:41:09.827 INFO:teuthology.orchestra.run.smithi114.stdout: "name": "fail_fs", 2024-04-23T10:41:09.827 INFO:teuthology.orchestra.run.smithi114.stdout: "type": "bool", 2024-04-23T10:41:09.827 INFO:teuthology.orchestra.run.smithi114.stdout: "level": "advanced", 2024-04-23T10:41:09.827 INFO:teuthology.orchestra.run.smithi114.stdout: "flags": 0, 2024-04-23T10:41:09.827 INFO:teuthology.orchestra.run.smithi114.stdout: "default_value": "False", 2024-04-23T10:41:09.827 INFO:teuthology.orchestra.run.smithi114.stdout: "min": "", 2024-04-23T10:41:09.827 INFO:teuthology.orchestra.run.smithi114.stdout: "max": "", 2024-04-23T10:41:09.827 INFO:teuthology.orchestra.run.smithi114.stdout: "enum_allowed": [], 2024-04-23T10:41:09.827 INFO:teuthology.orchestra.run.smithi114.stdout: "desc": "Fail filesystem for rapid multi-rank mds upgrade", 2024-04-23T10:41:09.828 INFO:teuthology.orchestra.run.smithi114.stdout: "long_desc": "", 2024-04-23T10:41:09.828 INFO:teuthology.orchestra.run.smithi114.stdout: "tags": [], 2024-04-23T10:41:09.828 INFO:teuthology.orchestra.run.smithi114.stdout: "see_also": [] 2024-04-23T10:41:09.828 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-23T10:41:09.828 INFO:teuthology.orchestra.run.smithi114.stdout: "log_level": { 2024-04-23T10:41:09.828 INFO:teuthology.orchestra.run.smithi114.stdout: "name": "log_level", 2024-04-23T10:41:09.828 INFO:teuthology.orchestra.run.smithi114.stdout: "type": "str", 2024-04-23T10:41:09.828 INFO:teuthology.orchestra.run.smithi114.stdout: "level": "advanced", 2024-04-23T10:41:09.828 INFO:teuthology.orchestra.run.smithi114.stdout: "flags": 1, 2024-04-23T10:41:09.828 INFO:teuthology.orchestra.run.smithi114.stdout: "default_value": "", 2024-04-23T10:41:09.829 INFO:teuthology.orchestra.run.smithi114.stdout: "min": "", 2024-04-23T10:41:09.829 INFO:teuthology.orchestra.run.smithi114.stdout: "max": "", 2024-04-23T10:41:09.829 INFO:teuthology.orchestra.run.smithi114.stdout: "enum_allowed": [ 2024-04-23T10:41:09.829 INFO:teuthology.orchestra.run.smithi114.stdout: "", 2024-04-23T10:41:09.829 INFO:teuthology.orchestra.run.smithi114.stdout: "critical", 2024-04-23T10:41:09.829 INFO:teuthology.orchestra.run.smithi114.stdout: "debug", 2024-04-23T10:41:09.829 INFO:teuthology.orchestra.run.smithi114.stdout: "error", 2024-04-23T10:41:09.829 INFO:teuthology.orchestra.run.smithi114.stdout: "info", 2024-04-23T10:41:09.829 INFO:teuthology.orchestra.run.smithi114.stdout: "warning" 2024-04-23T10:41:09.829 INFO:teuthology.orchestra.run.smithi114.stdout: ], 2024-04-23T10:41:09.830 INFO:teuthology.orchestra.run.smithi114.stdout: "desc": "", 2024-04-23T10:41:09.830 INFO:teuthology.orchestra.run.smithi114.stdout: "long_desc": "", 2024-04-23T10:41:09.830 INFO:teuthology.orchestra.run.smithi114.stdout: "tags": [], 2024-04-23T10:41:09.830 INFO:teuthology.orchestra.run.smithi114.stdout: "see_also": [] 2024-04-23T10:41:09.830 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-23T10:41:09.830 INFO:teuthology.orchestra.run.smithi114.stdout: "log_to_cluster": { 2024-04-23T10:41:09.830 INFO:teuthology.orchestra.run.smithi114.stdout: "name": "log_to_cluster", 2024-04-23T10:41:09.830 INFO:teuthology.orchestra.run.smithi114.stdout: "type": "bool", 2024-04-23T10:41:09.830 INFO:teuthology.orchestra.run.smithi114.stdout: "level": "advanced", 2024-04-23T10:41:09.831 INFO:teuthology.orchestra.run.smithi114.stdout: "flags": 1, 2024-04-23T10:41:09.831 INFO:teuthology.orchestra.run.smithi114.stdout: "default_value": "False", 2024-04-23T10:41:09.831 INFO:teuthology.orchestra.run.smithi114.stdout: "min": "", 2024-04-23T10:41:09.831 INFO:teuthology.orchestra.run.smithi114.stdout: "max": "", 2024-04-23T10:41:09.831 INFO:teuthology.orchestra.run.smithi114.stdout: "enum_allowed": [], 2024-04-23T10:41:09.831 INFO:teuthology.orchestra.run.smithi114.stdout: "desc": "", 2024-04-23T10:41:09.831 INFO:teuthology.orchestra.run.smithi114.stdout: "long_desc": "", 2024-04-23T10:41:09.831 INFO:teuthology.orchestra.run.smithi114.stdout: "tags": [], 2024-04-23T10:41:09.831 INFO:teuthology.orchestra.run.smithi114.stdout: "see_also": [] 2024-04-23T10:41:09.831 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-23T10:41:09.832 INFO:teuthology.orchestra.run.smithi114.stdout: "log_to_cluster_level": { 2024-04-23T10:41:09.832 INFO:teuthology.orchestra.run.smithi114.stdout: "name": "log_to_cluster_level", 2024-04-23T10:41:09.832 INFO:teuthology.orchestra.run.smithi114.stdout: "type": "str", 2024-04-23T10:41:09.832 INFO:teuthology.orchestra.run.smithi114.stdout: "level": "advanced", 2024-04-23T10:41:09.832 INFO:teuthology.orchestra.run.smithi114.stdout: "flags": 1, 2024-04-23T10:41:09.832 INFO:teuthology.orchestra.run.smithi114.stdout: "default_value": "info", 2024-04-23T10:41:09.832 INFO:teuthology.orchestra.run.smithi114.stdout: "min": "", 2024-04-23T10:41:09.832 INFO:teuthology.orchestra.run.smithi114.stdout: "max": "", 2024-04-23T10:41:09.832 INFO:teuthology.orchestra.run.smithi114.stdout: "enum_allowed": [ 2024-04-23T10:41:09.832 INFO:teuthology.orchestra.run.smithi114.stdout: "", 2024-04-23T10:41:09.833 INFO:teuthology.orchestra.run.smithi114.stdout: "critical", 2024-04-23T10:41:09.833 INFO:teuthology.orchestra.run.smithi114.stdout: "debug", 2024-04-23T10:41:09.833 INFO:teuthology.orchestra.run.smithi114.stdout: "error", 2024-04-23T10:41:09.833 INFO:teuthology.orchestra.run.smithi114.stdout: "info", 2024-04-23T10:41:09.833 INFO:teuthology.orchestra.run.smithi114.stdout: "warning" 2024-04-23T10:41:09.833 INFO:teuthology.orchestra.run.smithi114.stdout: ], 2024-04-23T10:41:09.833 INFO:teuthology.orchestra.run.smithi114.stdout: "desc": "", 2024-04-23T10:41:09.833 INFO:teuthology.orchestra.run.smithi114.stdout: "long_desc": "", 2024-04-23T10:41:09.833 INFO:teuthology.orchestra.run.smithi114.stdout: "tags": [], 2024-04-23T10:41:09.834 INFO:teuthology.orchestra.run.smithi114.stdout: "see_also": [] 2024-04-23T10:41:09.834 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-23T10:41:09.834 INFO:teuthology.orchestra.run.smithi114.stdout: "log_to_file": { 2024-04-23T10:41:09.834 INFO:teuthology.orchestra.run.smithi114.stdout: "name": "log_to_file", 2024-04-23T10:41:09.834 INFO:teuthology.orchestra.run.smithi114.stdout: "type": "bool", 2024-04-23T10:41:09.834 INFO:teuthology.orchestra.run.smithi114.stdout: "level": "advanced", 2024-04-23T10:41:09.834 INFO:teuthology.orchestra.run.smithi114.stdout: "flags": 1, 2024-04-23T10:41:09.834 INFO:teuthology.orchestra.run.smithi114.stdout: "default_value": "False", 2024-04-23T10:41:09.834 INFO:teuthology.orchestra.run.smithi114.stdout: "min": "", 2024-04-23T10:41:09.834 INFO:teuthology.orchestra.run.smithi114.stdout: "max": "", 2024-04-23T10:41:09.835 INFO:teuthology.orchestra.run.smithi114.stdout: "enum_allowed": [], 2024-04-23T10:41:09.835 INFO:teuthology.orchestra.run.smithi114.stdout: "desc": "", 2024-04-23T10:41:09.835 INFO:teuthology.orchestra.run.smithi114.stdout: "long_desc": "", 2024-04-23T10:41:09.835 INFO:teuthology.orchestra.run.smithi114.stdout: "tags": [], 2024-04-23T10:41:09.835 INFO:teuthology.orchestra.run.smithi114.stdout: "see_also": [] 2024-04-23T10:41:09.835 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-23T10:41:09.835 INFO:teuthology.orchestra.run.smithi114.stdout: "orchestrator": { 2024-04-23T10:41:09.835 INFO:teuthology.orchestra.run.smithi114.stdout: "name": "orchestrator", 2024-04-23T10:41:09.835 INFO:teuthology.orchestra.run.smithi114.stdout: "type": "str", 2024-04-23T10:41:09.835 INFO:teuthology.orchestra.run.smithi114.stdout: "level": "advanced", 2024-04-23T10:41:09.836 INFO:teuthology.orchestra.run.smithi114.stdout: "flags": 1, 2024-04-23T10:41:09.836 INFO:teuthology.orchestra.run.smithi114.stdout: "default_value": "", 2024-04-23T10:41:09.836 INFO:teuthology.orchestra.run.smithi114.stdout: "min": "", 2024-04-23T10:41:09.836 INFO:teuthology.orchestra.run.smithi114.stdout: "max": "", 2024-04-23T10:41:09.836 INFO:teuthology.orchestra.run.smithi114.stdout: "enum_allowed": [ 2024-04-23T10:41:09.836 INFO:teuthology.orchestra.run.smithi114.stdout: "cephadm", 2024-04-23T10:41:09.836 INFO:teuthology.orchestra.run.smithi114.stdout: "rook", 2024-04-23T10:41:09.836 INFO:teuthology.orchestra.run.smithi114.stdout: "test_orchestrator" 2024-04-23T10:41:09.836 INFO:teuthology.orchestra.run.smithi114.stdout: ], 2024-04-23T10:41:09.836 INFO:teuthology.orchestra.run.smithi114.stdout: "desc": "Orchestrator backend", 2024-04-23T10:41:09.837 INFO:teuthology.orchestra.run.smithi114.stdout: "long_desc": "", 2024-04-23T10:41:09.837 INFO:teuthology.orchestra.run.smithi114.stdout: "tags": [], 2024-04-23T10:41:09.837 INFO:teuthology.orchestra.run.smithi114.stdout: "see_also": [] 2024-04-23T10:41:09.837 INFO:teuthology.orchestra.run.smithi114.stdout: } 2024-04-23T10:41:09.837 INFO:teuthology.orchestra.run.smithi114.stdout: } 2024-04-23T10:41:09.837 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-23T10:41:09.837 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-04-23T10:41:09.837 INFO:teuthology.orchestra.run.smithi114.stdout: "name": "osd_perf_query", 2024-04-23T10:41:09.837 INFO:teuthology.orchestra.run.smithi114.stdout: "can_run": true, 2024-04-23T10:41:09.837 INFO:teuthology.orchestra.run.smithi114.stdout: "error_string": "", 2024-04-23T10:41:09.838 INFO:teuthology.orchestra.run.smithi114.stdout: "module_options": { 2024-04-23T10:41:09.838 INFO:teuthology.orchestra.run.smithi114.stdout: "log_level": { 2024-04-23T10:41:09.838 INFO:teuthology.orchestra.run.smithi114.stdout: "name": "log_level", 2024-04-23T10:41:09.838 INFO:teuthology.orchestra.run.smithi114.stdout: "type": "str", 2024-04-23T10:41:09.838 INFO:teuthology.orchestra.run.smithi114.stdout: "level": "advanced", 2024-04-23T10:41:09.838 INFO:teuthology.orchestra.run.smithi114.stdout: "flags": 1, 2024-04-23T10:41:09.838 INFO:teuthology.orchestra.run.smithi114.stdout: "default_value": "", 2024-04-23T10:41:09.838 INFO:teuthology.orchestra.run.smithi114.stdout: "min": "", 2024-04-23T10:41:09.839 INFO:teuthology.orchestra.run.smithi114.stdout: "max": "", 2024-04-23T10:41:09.839 INFO:teuthology.orchestra.run.smithi114.stdout: "enum_allowed": [ 2024-04-23T10:41:09.839 INFO:teuthology.orchestra.run.smithi114.stdout: "", 2024-04-23T10:41:09.839 INFO:teuthology.orchestra.run.smithi114.stdout: "critical", 2024-04-23T10:41:09.839 INFO:teuthology.orchestra.run.smithi114.stdout: "debug", 2024-04-23T10:41:09.839 INFO:teuthology.orchestra.run.smithi114.stdout: "error", 2024-04-23T10:41:09.839 INFO:teuthology.orchestra.run.smithi114.stdout: "info", 2024-04-23T10:41:09.839 INFO:teuthology.orchestra.run.smithi114.stdout: "warning" 2024-04-23T10:41:09.839 INFO:teuthology.orchestra.run.smithi114.stdout: ], 2024-04-23T10:41:09.839 INFO:teuthology.orchestra.run.smithi114.stdout: "desc": "", 2024-04-23T10:41:09.840 INFO:teuthology.orchestra.run.smithi114.stdout: "long_desc": "", 2024-04-23T10:41:09.840 INFO:teuthology.orchestra.run.smithi114.stdout: "tags": [], 2024-04-23T10:41:09.840 INFO:teuthology.orchestra.run.smithi114.stdout: "see_also": [] 2024-04-23T10:41:09.840 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-23T10:41:09.840 INFO:teuthology.orchestra.run.smithi114.stdout: "log_to_cluster": { 2024-04-23T10:41:09.840 INFO:teuthology.orchestra.run.smithi114.stdout: "name": "log_to_cluster", 2024-04-23T10:41:09.840 INFO:teuthology.orchestra.run.smithi114.stdout: "type": "bool", 2024-04-23T10:41:09.840 INFO:teuthology.orchestra.run.smithi114.stdout: "level": "advanced", 2024-04-23T10:41:09.840 INFO:teuthology.orchestra.run.smithi114.stdout: "flags": 1, 2024-04-23T10:41:09.840 INFO:teuthology.orchestra.run.smithi114.stdout: "default_value": "False", 2024-04-23T10:41:09.841 INFO:teuthology.orchestra.run.smithi114.stdout: "min": "", 2024-04-23T10:41:09.841 INFO:teuthology.orchestra.run.smithi114.stdout: "max": "", 2024-04-23T10:41:09.841 INFO:teuthology.orchestra.run.smithi114.stdout: "enum_allowed": [], 2024-04-23T10:41:09.841 INFO:teuthology.orchestra.run.smithi114.stdout: "desc": "", 2024-04-23T10:41:09.841 INFO:teuthology.orchestra.run.smithi114.stdout: "long_desc": "", 2024-04-23T10:41:09.841 INFO:teuthology.orchestra.run.smithi114.stdout: "tags": [], 2024-04-23T10:41:09.841 INFO:teuthology.orchestra.run.smithi114.stdout: "see_also": [] 2024-04-23T10:41:09.841 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-23T10:41:09.841 INFO:teuthology.orchestra.run.smithi114.stdout: "log_to_cluster_level": { 2024-04-23T10:41:09.841 INFO:teuthology.orchestra.run.smithi114.stdout: "name": "log_to_cluster_level", 2024-04-23T10:41:09.842 INFO:teuthology.orchestra.run.smithi114.stdout: "type": "str", 2024-04-23T10:41:09.842 INFO:teuthology.orchestra.run.smithi114.stdout: "level": "advanced", 2024-04-23T10:41:09.842 INFO:teuthology.orchestra.run.smithi114.stdout: "flags": 1, 2024-04-23T10:41:09.842 INFO:teuthology.orchestra.run.smithi114.stdout: "default_value": "info", 2024-04-23T10:41:09.842 INFO:teuthology.orchestra.run.smithi114.stdout: "min": "", 2024-04-23T10:41:09.842 INFO:teuthology.orchestra.run.smithi114.stdout: "max": "", 2024-04-23T10:41:09.842 INFO:teuthology.orchestra.run.smithi114.stdout: "enum_allowed": [ 2024-04-23T10:41:09.842 INFO:teuthology.orchestra.run.smithi114.stdout: "", 2024-04-23T10:41:09.842 INFO:teuthology.orchestra.run.smithi114.stdout: "critical", 2024-04-23T10:41:09.843 INFO:teuthology.orchestra.run.smithi114.stdout: "debug", 2024-04-23T10:41:09.843 INFO:teuthology.orchestra.run.smithi114.stdout: "error", 2024-04-23T10:41:09.843 INFO:teuthology.orchestra.run.smithi114.stdout: "info", 2024-04-23T10:41:09.843 INFO:teuthology.orchestra.run.smithi114.stdout: "warning" 2024-04-23T10:41:09.843 INFO:teuthology.orchestra.run.smithi114.stdout: ], 2024-04-23T10:41:09.843 INFO:teuthology.orchestra.run.smithi114.stdout: "desc": "", 2024-04-23T10:41:09.843 INFO:teuthology.orchestra.run.smithi114.stdout: "long_desc": "", 2024-04-23T10:41:09.843 INFO:teuthology.orchestra.run.smithi114.stdout: "tags": [], 2024-04-23T10:41:09.843 INFO:teuthology.orchestra.run.smithi114.stdout: "see_also": [] 2024-04-23T10:41:09.843 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-23T10:41:09.844 INFO:teuthology.orchestra.run.smithi114.stdout: "log_to_file": { 2024-04-23T10:41:09.844 INFO:teuthology.orchestra.run.smithi114.stdout: "name": "log_to_file", 2024-04-23T10:41:09.844 INFO:teuthology.orchestra.run.smithi114.stdout: "type": "bool", 2024-04-23T10:41:09.844 INFO:teuthology.orchestra.run.smithi114.stdout: "level": "advanced", 2024-04-23T10:41:09.844 INFO:teuthology.orchestra.run.smithi114.stdout: "flags": 1, 2024-04-23T10:41:09.844 INFO:teuthology.orchestra.run.smithi114.stdout: "default_value": "False", 2024-04-23T10:41:09.844 INFO:teuthology.orchestra.run.smithi114.stdout: "min": "", 2024-04-23T10:41:09.844 INFO:teuthology.orchestra.run.smithi114.stdout: "max": "", 2024-04-23T10:41:09.844 INFO:teuthology.orchestra.run.smithi114.stdout: "enum_allowed": [], 2024-04-23T10:41:09.845 INFO:teuthology.orchestra.run.smithi114.stdout: "desc": "", 2024-04-23T10:41:09.845 INFO:teuthology.orchestra.run.smithi114.stdout: "long_desc": "", 2024-04-23T10:41:09.845 INFO:teuthology.orchestra.run.smithi114.stdout: "tags": [], 2024-04-23T10:41:09.845 INFO:teuthology.orchestra.run.smithi114.stdout: "see_also": [] 2024-04-23T10:41:09.845 INFO:teuthology.orchestra.run.smithi114.stdout: } 2024-04-23T10:41:09.845 INFO:teuthology.orchestra.run.smithi114.stdout: } 2024-04-23T10:41:09.845 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-23T10:41:09.845 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-04-23T10:41:09.845 INFO:teuthology.orchestra.run.smithi114.stdout: "name": "osd_support", 2024-04-23T10:41:09.845 INFO:teuthology.orchestra.run.smithi114.stdout: "can_run": true, 2024-04-23T10:41:09.846 INFO:teuthology.orchestra.run.smithi114.stdout: "error_string": "", 2024-04-23T10:41:09.846 INFO:teuthology.orchestra.run.smithi114.stdout: "module_options": { 2024-04-23T10:41:09.846 INFO:teuthology.orchestra.run.smithi114.stdout: "log_level": { 2024-04-23T10:41:09.846 INFO:teuthology.orchestra.run.smithi114.stdout: "name": "log_level", 2024-04-23T10:41:09.846 INFO:teuthology.orchestra.run.smithi114.stdout: "type": "str", 2024-04-23T10:41:09.846 INFO:teuthology.orchestra.run.smithi114.stdout: "level": "advanced", 2024-04-23T10:41:09.846 INFO:teuthology.orchestra.run.smithi114.stdout: "flags": 1, 2024-04-23T10:41:09.846 INFO:teuthology.orchestra.run.smithi114.stdout: "default_value": "", 2024-04-23T10:41:09.846 INFO:teuthology.orchestra.run.smithi114.stdout: "min": "", 2024-04-23T10:41:09.847 INFO:teuthology.orchestra.run.smithi114.stdout: "max": "", 2024-04-23T10:41:09.847 INFO:teuthology.orchestra.run.smithi114.stdout: "enum_allowed": [ 2024-04-23T10:41:09.847 INFO:teuthology.orchestra.run.smithi114.stdout: "", 2024-04-23T10:41:09.847 INFO:teuthology.orchestra.run.smithi114.stdout: "critical", 2024-04-23T10:41:09.847 INFO:teuthology.orchestra.run.smithi114.stdout: "debug", 2024-04-23T10:41:09.847 INFO:teuthology.orchestra.run.smithi114.stdout: "error", 2024-04-23T10:41:09.847 INFO:teuthology.orchestra.run.smithi114.stdout: "info", 2024-04-23T10:41:09.847 INFO:teuthology.orchestra.run.smithi114.stdout: "warning" 2024-04-23T10:41:09.847 INFO:teuthology.orchestra.run.smithi114.stdout: ], 2024-04-23T10:41:09.847 INFO:teuthology.orchestra.run.smithi114.stdout: "desc": "", 2024-04-23T10:41:09.848 INFO:teuthology.orchestra.run.smithi114.stdout: "long_desc": "", 2024-04-23T10:41:09.848 INFO:teuthology.orchestra.run.smithi114.stdout: "tags": [], 2024-04-23T10:41:09.848 INFO:teuthology.orchestra.run.smithi114.stdout: "see_also": [] 2024-04-23T10:41:09.848 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-23T10:41:09.848 INFO:teuthology.orchestra.run.smithi114.stdout: "log_to_cluster": { 2024-04-23T10:41:09.848 INFO:teuthology.orchestra.run.smithi114.stdout: "name": "log_to_cluster", 2024-04-23T10:41:09.848 INFO:teuthology.orchestra.run.smithi114.stdout: "type": "bool", 2024-04-23T10:41:09.848 INFO:teuthology.orchestra.run.smithi114.stdout: "level": "advanced", 2024-04-23T10:41:09.848 INFO:teuthology.orchestra.run.smithi114.stdout: "flags": 1, 2024-04-23T10:41:09.849 INFO:teuthology.orchestra.run.smithi114.stdout: "default_value": "False", 2024-04-23T10:41:09.849 INFO:teuthology.orchestra.run.smithi114.stdout: "min": "", 2024-04-23T10:41:09.849 INFO:teuthology.orchestra.run.smithi114.stdout: "max": "", 2024-04-23T10:41:09.849 INFO:teuthology.orchestra.run.smithi114.stdout: "enum_allowed": [], 2024-04-23T10:41:09.849 INFO:teuthology.orchestra.run.smithi114.stdout: "desc": "", 2024-04-23T10:41:09.849 INFO:teuthology.orchestra.run.smithi114.stdout: "long_desc": "", 2024-04-23T10:41:09.849 INFO:teuthology.orchestra.run.smithi114.stdout: "tags": [], 2024-04-23T10:41:09.849 INFO:teuthology.orchestra.run.smithi114.stdout: "see_also": [] 2024-04-23T10:41:09.849 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-23T10:41:09.849 INFO:teuthology.orchestra.run.smithi114.stdout: "log_to_cluster_level": { 2024-04-23T10:41:09.850 INFO:teuthology.orchestra.run.smithi114.stdout: "name": "log_to_cluster_level", 2024-04-23T10:41:09.850 INFO:teuthology.orchestra.run.smithi114.stdout: "type": "str", 2024-04-23T10:41:09.850 INFO:teuthology.orchestra.run.smithi114.stdout: "level": "advanced", 2024-04-23T10:41:09.850 INFO:teuthology.orchestra.run.smithi114.stdout: "flags": 1, 2024-04-23T10:41:09.850 INFO:teuthology.orchestra.run.smithi114.stdout: "default_value": "info", 2024-04-23T10:41:09.850 INFO:teuthology.orchestra.run.smithi114.stdout: "min": "", 2024-04-23T10:41:09.850 INFO:teuthology.orchestra.run.smithi114.stdout: "max": "", 2024-04-23T10:41:09.850 INFO:teuthology.orchestra.run.smithi114.stdout: "enum_allowed": [ 2024-04-23T10:41:09.851 INFO:teuthology.orchestra.run.smithi114.stdout: "", 2024-04-23T10:41:09.851 INFO:teuthology.orchestra.run.smithi114.stdout: "critical", 2024-04-23T10:41:09.851 INFO:teuthology.orchestra.run.smithi114.stdout: "debug", 2024-04-23T10:41:09.851 INFO:teuthology.orchestra.run.smithi114.stdout: "error", 2024-04-23T10:41:09.851 INFO:teuthology.orchestra.run.smithi114.stdout: "info", 2024-04-23T10:41:09.851 INFO:teuthology.orchestra.run.smithi114.stdout: "warning" 2024-04-23T10:41:09.851 INFO:teuthology.orchestra.run.smithi114.stdout: ], 2024-04-23T10:41:09.851 INFO:teuthology.orchestra.run.smithi114.stdout: "desc": "", 2024-04-23T10:41:09.851 INFO:teuthology.orchestra.run.smithi114.stdout: "long_desc": "", 2024-04-23T10:41:09.851 INFO:teuthology.orchestra.run.smithi114.stdout: "tags": [], 2024-04-23T10:41:09.852 INFO:teuthology.orchestra.run.smithi114.stdout: "see_also": [] 2024-04-23T10:41:09.852 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-23T10:41:09.852 INFO:teuthology.orchestra.run.smithi114.stdout: "log_to_file": { 2024-04-23T10:41:09.852 INFO:teuthology.orchestra.run.smithi114.stdout: "name": "log_to_file", 2024-04-23T10:41:09.852 INFO:teuthology.orchestra.run.smithi114.stdout: "type": "bool", 2024-04-23T10:41:09.852 INFO:teuthology.orchestra.run.smithi114.stdout: "level": "advanced", 2024-04-23T10:41:09.852 INFO:teuthology.orchestra.run.smithi114.stdout: "flags": 1, 2024-04-23T10:41:09.852 INFO:teuthology.orchestra.run.smithi114.stdout: "default_value": "False", 2024-04-23T10:41:09.852 INFO:teuthology.orchestra.run.smithi114.stdout: "min": "", 2024-04-23T10:41:09.852 INFO:teuthology.orchestra.run.smithi114.stdout: "max": "", 2024-04-23T10:41:09.853 INFO:teuthology.orchestra.run.smithi114.stdout: "enum_allowed": [], 2024-04-23T10:41:09.853 INFO:teuthology.orchestra.run.smithi114.stdout: "desc": "", 2024-04-23T10:41:09.853 INFO:teuthology.orchestra.run.smithi114.stdout: "long_desc": "", 2024-04-23T10:41:09.853 INFO:teuthology.orchestra.run.smithi114.stdout: "tags": [], 2024-04-23T10:41:09.853 INFO:teuthology.orchestra.run.smithi114.stdout: "see_also": [] 2024-04-23T10:41:09.853 INFO:teuthology.orchestra.run.smithi114.stdout: } 2024-04-23T10:41:09.853 INFO:teuthology.orchestra.run.smithi114.stdout: } 2024-04-23T10:41:09.853 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-23T10:41:09.853 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-04-23T10:41:09.853 INFO:teuthology.orchestra.run.smithi114.stdout: "name": "pg_autoscaler", 2024-04-23T10:41:09.854 INFO:teuthology.orchestra.run.smithi114.stdout: "can_run": true, 2024-04-23T10:41:09.854 INFO:teuthology.orchestra.run.smithi114.stdout: "error_string": "", 2024-04-23T10:41:09.854 INFO:teuthology.orchestra.run.smithi114.stdout: "module_options": { 2024-04-23T10:41:09.854 INFO:teuthology.orchestra.run.smithi114.stdout: "log_level": { 2024-04-23T10:41:09.854 INFO:teuthology.orchestra.run.smithi114.stdout: "name": "log_level", 2024-04-23T10:41:09.854 INFO:teuthology.orchestra.run.smithi114.stdout: "type": "str", 2024-04-23T10:41:09.854 INFO:teuthology.orchestra.run.smithi114.stdout: "level": "advanced", 2024-04-23T10:41:09.854 INFO:teuthology.orchestra.run.smithi114.stdout: "flags": 1, 2024-04-23T10:41:09.854 INFO:teuthology.orchestra.run.smithi114.stdout: "default_value": "", 2024-04-23T10:41:09.855 INFO:teuthology.orchestra.run.smithi114.stdout: "min": "", 2024-04-23T10:41:09.855 INFO:teuthology.orchestra.run.smithi114.stdout: "max": "", 2024-04-23T10:41:09.855 INFO:teuthology.orchestra.run.smithi114.stdout: "enum_allowed": [ 2024-04-23T10:41:09.855 INFO:teuthology.orchestra.run.smithi114.stdout: "", 2024-04-23T10:41:09.855 INFO:teuthology.orchestra.run.smithi114.stdout: "critical", 2024-04-23T10:41:09.855 INFO:teuthology.orchestra.run.smithi114.stdout: "debug", 2024-04-23T10:41:09.855 INFO:teuthology.orchestra.run.smithi114.stdout: "error", 2024-04-23T10:41:09.855 INFO:teuthology.orchestra.run.smithi114.stdout: "info", 2024-04-23T10:41:09.855 INFO:teuthology.orchestra.run.smithi114.stdout: "warning" 2024-04-23T10:41:09.855 INFO:teuthology.orchestra.run.smithi114.stdout: ], 2024-04-23T10:41:09.856 INFO:teuthology.orchestra.run.smithi114.stdout: "desc": "", 2024-04-23T10:41:09.856 INFO:teuthology.orchestra.run.smithi114.stdout: "long_desc": "", 2024-04-23T10:41:09.856 INFO:teuthology.orchestra.run.smithi114.stdout: "tags": [], 2024-04-23T10:41:09.856 INFO:teuthology.orchestra.run.smithi114.stdout: "see_also": [] 2024-04-23T10:41:09.856 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-23T10:41:09.856 INFO:teuthology.orchestra.run.smithi114.stdout: "log_to_cluster": { 2024-04-23T10:41:09.856 INFO:teuthology.orchestra.run.smithi114.stdout: "name": "log_to_cluster", 2024-04-23T10:41:09.856 INFO:teuthology.orchestra.run.smithi114.stdout: "type": "bool", 2024-04-23T10:41:09.856 INFO:teuthology.orchestra.run.smithi114.stdout: "level": "advanced", 2024-04-23T10:41:09.856 INFO:teuthology.orchestra.run.smithi114.stdout: "flags": 1, 2024-04-23T10:41:09.857 INFO:teuthology.orchestra.run.smithi114.stdout: "default_value": "False", 2024-04-23T10:41:09.857 INFO:teuthology.orchestra.run.smithi114.stdout: "min": "", 2024-04-23T10:41:09.857 INFO:teuthology.orchestra.run.smithi114.stdout: "max": "", 2024-04-23T10:41:09.857 INFO:teuthology.orchestra.run.smithi114.stdout: "enum_allowed": [], 2024-04-23T10:41:09.857 INFO:teuthology.orchestra.run.smithi114.stdout: "desc": "", 2024-04-23T10:41:09.857 INFO:teuthology.orchestra.run.smithi114.stdout: "long_desc": "", 2024-04-23T10:41:09.857 INFO:teuthology.orchestra.run.smithi114.stdout: "tags": [], 2024-04-23T10:41:09.857 INFO:teuthology.orchestra.run.smithi114.stdout: "see_also": [] 2024-04-23T10:41:09.857 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-23T10:41:09.857 INFO:teuthology.orchestra.run.smithi114.stdout: "log_to_cluster_level": { 2024-04-23T10:41:09.858 INFO:teuthology.orchestra.run.smithi114.stdout: "name": "log_to_cluster_level", 2024-04-23T10:41:09.858 INFO:teuthology.orchestra.run.smithi114.stdout: "type": "str", 2024-04-23T10:41:09.858 INFO:teuthology.orchestra.run.smithi114.stdout: "level": "advanced", 2024-04-23T10:41:09.858 INFO:teuthology.orchestra.run.smithi114.stdout: "flags": 1, 2024-04-23T10:41:09.858 INFO:teuthology.orchestra.run.smithi114.stdout: "default_value": "info", 2024-04-23T10:41:09.858 INFO:teuthology.orchestra.run.smithi114.stdout: "min": "", 2024-04-23T10:41:09.858 INFO:teuthology.orchestra.run.smithi114.stdout: "max": "", 2024-04-23T10:41:09.858 INFO:teuthology.orchestra.run.smithi114.stdout: "enum_allowed": [ 2024-04-23T10:41:09.859 INFO:teuthology.orchestra.run.smithi114.stdout: "", 2024-04-23T10:41:09.859 INFO:teuthology.orchestra.run.smithi114.stdout: "critical", 2024-04-23T10:41:09.859 INFO:teuthology.orchestra.run.smithi114.stdout: "debug", 2024-04-23T10:41:09.859 INFO:teuthology.orchestra.run.smithi114.stdout: "error", 2024-04-23T10:41:09.859 INFO:teuthology.orchestra.run.smithi114.stdout: "info", 2024-04-23T10:41:09.859 INFO:teuthology.orchestra.run.smithi114.stdout: "warning" 2024-04-23T10:41:09.859 INFO:teuthology.orchestra.run.smithi114.stdout: ], 2024-04-23T10:41:09.859 INFO:teuthology.orchestra.run.smithi114.stdout: "desc": "", 2024-04-23T10:41:09.859 INFO:teuthology.orchestra.run.smithi114.stdout: "long_desc": "", 2024-04-23T10:41:09.859 INFO:teuthology.orchestra.run.smithi114.stdout: "tags": [], 2024-04-23T10:41:09.860 INFO:teuthology.orchestra.run.smithi114.stdout: "see_also": [] 2024-04-23T10:41:09.860 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-23T10:41:09.860 INFO:teuthology.orchestra.run.smithi114.stdout: "log_to_file": { 2024-04-23T10:41:09.860 INFO:teuthology.orchestra.run.smithi114.stdout: "name": "log_to_file", 2024-04-23T10:41:09.860 INFO:teuthology.orchestra.run.smithi114.stdout: "type": "bool", 2024-04-23T10:41:09.860 INFO:teuthology.orchestra.run.smithi114.stdout: "level": "advanced", 2024-04-23T10:41:09.860 INFO:teuthology.orchestra.run.smithi114.stdout: "flags": 1, 2024-04-23T10:41:09.860 INFO:teuthology.orchestra.run.smithi114.stdout: "default_value": "False", 2024-04-23T10:41:09.860 INFO:teuthology.orchestra.run.smithi114.stdout: "min": "", 2024-04-23T10:41:09.860 INFO:teuthology.orchestra.run.smithi114.stdout: "max": "", 2024-04-23T10:41:09.861 INFO:teuthology.orchestra.run.smithi114.stdout: "enum_allowed": [], 2024-04-23T10:41:09.861 INFO:teuthology.orchestra.run.smithi114.stdout: "desc": "", 2024-04-23T10:41:09.861 INFO:teuthology.orchestra.run.smithi114.stdout: "long_desc": "", 2024-04-23T10:41:09.861 INFO:teuthology.orchestra.run.smithi114.stdout: "tags": [], 2024-04-23T10:41:09.861 INFO:teuthology.orchestra.run.smithi114.stdout: "see_also": [] 2024-04-23T10:41:09.861 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-23T10:41:09.861 INFO:teuthology.orchestra.run.smithi114.stdout: "sleep_interval": { 2024-04-23T10:41:09.861 INFO:teuthology.orchestra.run.smithi114.stdout: "name": "sleep_interval", 2024-04-23T10:41:09.861 INFO:teuthology.orchestra.run.smithi114.stdout: "type": "secs", 2024-04-23T10:41:09.861 INFO:teuthology.orchestra.run.smithi114.stdout: "level": "advanced", 2024-04-23T10:41:09.862 INFO:teuthology.orchestra.run.smithi114.stdout: "flags": 0, 2024-04-23T10:41:09.862 INFO:teuthology.orchestra.run.smithi114.stdout: "default_value": "60", 2024-04-23T10:41:09.862 INFO:teuthology.orchestra.run.smithi114.stdout: "min": "", 2024-04-23T10:41:09.862 INFO:teuthology.orchestra.run.smithi114.stdout: "max": "", 2024-04-23T10:41:09.862 INFO:teuthology.orchestra.run.smithi114.stdout: "enum_allowed": [], 2024-04-23T10:41:09.862 INFO:teuthology.orchestra.run.smithi114.stdout: "desc": "", 2024-04-23T10:41:09.862 INFO:teuthology.orchestra.run.smithi114.stdout: "long_desc": "", 2024-04-23T10:41:09.862 INFO:teuthology.orchestra.run.smithi114.stdout: "tags": [], 2024-04-23T10:41:09.862 INFO:teuthology.orchestra.run.smithi114.stdout: "see_also": [] 2024-04-23T10:41:09.862 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-23T10:41:09.863 INFO:teuthology.orchestra.run.smithi114.stdout: "threshold": { 2024-04-23T10:41:09.863 INFO:teuthology.orchestra.run.smithi114.stdout: "name": "threshold", 2024-04-23T10:41:09.863 INFO:teuthology.orchestra.run.smithi114.stdout: "type": "float", 2024-04-23T10:41:09.863 INFO:teuthology.orchestra.run.smithi114.stdout: "level": "advanced", 2024-04-23T10:41:09.863 INFO:teuthology.orchestra.run.smithi114.stdout: "flags": 0, 2024-04-23T10:41:09.863 INFO:teuthology.orchestra.run.smithi114.stdout: "default_value": "3.0", 2024-04-23T10:41:09.863 INFO:teuthology.orchestra.run.smithi114.stdout: "min": "1.0", 2024-04-23T10:41:09.863 INFO:teuthology.orchestra.run.smithi114.stdout: "max": "", 2024-04-23T10:41:09.863 INFO:teuthology.orchestra.run.smithi114.stdout: "enum_allowed": [], 2024-04-23T10:41:09.863 INFO:teuthology.orchestra.run.smithi114.stdout: "desc": "scaling threshold", 2024-04-23T10:41:09.864 INFO:teuthology.orchestra.run.smithi114.stdout: "long_desc": "The factor by which the `NEW PG_NUM` must vary from the current`PG_NUM` before being accepted. Cannot be less than 1.0", 2024-04-23T10:41:09.864 INFO:teuthology.orchestra.run.smithi114.stdout: "tags": [], 2024-04-23T10:41:09.864 INFO:teuthology.orchestra.run.smithi114.stdout: "see_also": [] 2024-04-23T10:41:09.864 INFO:teuthology.orchestra.run.smithi114.stdout: } 2024-04-23T10:41:09.864 INFO:teuthology.orchestra.run.smithi114.stdout: } 2024-04-23T10:41:09.864 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-23T10:41:09.864 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-04-23T10:41:09.864 INFO:teuthology.orchestra.run.smithi114.stdout: "name": "progress", 2024-04-23T10:41:09.864 INFO:teuthology.orchestra.run.smithi114.stdout: "can_run": true, 2024-04-23T10:41:09.865 INFO:teuthology.orchestra.run.smithi114.stdout: "error_string": "", 2024-04-23T10:41:09.865 INFO:teuthology.orchestra.run.smithi114.stdout: "module_options": { 2024-04-23T10:41:09.865 INFO:teuthology.orchestra.run.smithi114.stdout: "allow_pg_recovery_event": { 2024-04-23T10:41:09.865 INFO:teuthology.orchestra.run.smithi114.stdout: "name": "allow_pg_recovery_event", 2024-04-23T10:41:09.865 INFO:teuthology.orchestra.run.smithi114.stdout: "type": "bool", 2024-04-23T10:41:09.865 INFO:teuthology.orchestra.run.smithi114.stdout: "level": "advanced", 2024-04-23T10:41:09.865 INFO:teuthology.orchestra.run.smithi114.stdout: "flags": 1, 2024-04-23T10:41:09.865 INFO:teuthology.orchestra.run.smithi114.stdout: "default_value": "False", 2024-04-23T10:41:09.865 INFO:teuthology.orchestra.run.smithi114.stdout: "min": "", 2024-04-23T10:41:09.865 INFO:teuthology.orchestra.run.smithi114.stdout: "max": "", 2024-04-23T10:41:09.866 INFO:teuthology.orchestra.run.smithi114.stdout: "enum_allowed": [], 2024-04-23T10:41:09.866 INFO:teuthology.orchestra.run.smithi114.stdout: "desc": "allow the module to show pg recovery progress", 2024-04-23T10:41:09.866 INFO:teuthology.orchestra.run.smithi114.stdout: "long_desc": "", 2024-04-23T10:41:09.866 INFO:teuthology.orchestra.run.smithi114.stdout: "tags": [], 2024-04-23T10:41:09.866 INFO:teuthology.orchestra.run.smithi114.stdout: "see_also": [] 2024-04-23T10:41:09.866 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-23T10:41:09.866 INFO:teuthology.orchestra.run.smithi114.stdout: "enabled": { 2024-04-23T10:41:09.866 INFO:teuthology.orchestra.run.smithi114.stdout: "name": "enabled", 2024-04-23T10:41:09.866 INFO:teuthology.orchestra.run.smithi114.stdout: "type": "bool", 2024-04-23T10:41:09.866 INFO:teuthology.orchestra.run.smithi114.stdout: "level": "advanced", 2024-04-23T10:41:09.867 INFO:teuthology.orchestra.run.smithi114.stdout: "flags": 0, 2024-04-23T10:41:09.867 INFO:teuthology.orchestra.run.smithi114.stdout: "default_value": "True", 2024-04-23T10:41:09.867 INFO:teuthology.orchestra.run.smithi114.stdout: "min": "", 2024-04-23T10:41:09.867 INFO:teuthology.orchestra.run.smithi114.stdout: "max": "", 2024-04-23T10:41:09.867 INFO:teuthology.orchestra.run.smithi114.stdout: "enum_allowed": [], 2024-04-23T10:41:09.867 INFO:teuthology.orchestra.run.smithi114.stdout: "desc": "", 2024-04-23T10:41:09.867 INFO:teuthology.orchestra.run.smithi114.stdout: "long_desc": "", 2024-04-23T10:41:09.867 INFO:teuthology.orchestra.run.smithi114.stdout: "tags": [], 2024-04-23T10:41:09.867 INFO:teuthology.orchestra.run.smithi114.stdout: "see_also": [] 2024-04-23T10:41:09.867 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-23T10:41:09.868 INFO:teuthology.orchestra.run.smithi114.stdout: "log_level": { 2024-04-23T10:41:09.868 INFO:teuthology.orchestra.run.smithi114.stdout: "name": "log_level", 2024-04-23T10:41:09.868 INFO:teuthology.orchestra.run.smithi114.stdout: "type": "str", 2024-04-23T10:41:09.868 INFO:teuthology.orchestra.run.smithi114.stdout: "level": "advanced", 2024-04-23T10:41:09.868 INFO:teuthology.orchestra.run.smithi114.stdout: "flags": 1, 2024-04-23T10:41:09.868 INFO:teuthology.orchestra.run.smithi114.stdout: "default_value": "", 2024-04-23T10:41:09.868 INFO:teuthology.orchestra.run.smithi114.stdout: "min": "", 2024-04-23T10:41:09.868 INFO:teuthology.orchestra.run.smithi114.stdout: "max": "", 2024-04-23T10:41:09.868 INFO:teuthology.orchestra.run.smithi114.stdout: "enum_allowed": [ 2024-04-23T10:41:09.869 INFO:teuthology.orchestra.run.smithi114.stdout: "", 2024-04-23T10:41:09.869 INFO:teuthology.orchestra.run.smithi114.stdout: "critical", 2024-04-23T10:41:09.869 INFO:teuthology.orchestra.run.smithi114.stdout: "debug", 2024-04-23T10:41:09.869 INFO:teuthology.orchestra.run.smithi114.stdout: "error", 2024-04-23T10:41:09.869 INFO:teuthology.orchestra.run.smithi114.stdout: "info", 2024-04-23T10:41:09.869 INFO:teuthology.orchestra.run.smithi114.stdout: "warning" 2024-04-23T10:41:09.869 INFO:teuthology.orchestra.run.smithi114.stdout: ], 2024-04-23T10:41:09.869 INFO:teuthology.orchestra.run.smithi114.stdout: "desc": "", 2024-04-23T10:41:09.869 INFO:teuthology.orchestra.run.smithi114.stdout: "long_desc": "", 2024-04-23T10:41:09.869 INFO:teuthology.orchestra.run.smithi114.stdout: "tags": [], 2024-04-23T10:41:09.870 INFO:teuthology.orchestra.run.smithi114.stdout: "see_also": [] 2024-04-23T10:41:09.870 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-23T10:41:09.870 INFO:teuthology.orchestra.run.smithi114.stdout: "log_to_cluster": { 2024-04-23T10:41:09.870 INFO:teuthology.orchestra.run.smithi114.stdout: "name": "log_to_cluster", 2024-04-23T10:41:09.870 INFO:teuthology.orchestra.run.smithi114.stdout: "type": "bool", 2024-04-23T10:41:09.870 INFO:teuthology.orchestra.run.smithi114.stdout: "level": "advanced", 2024-04-23T10:41:09.870 INFO:teuthology.orchestra.run.smithi114.stdout: "flags": 1, 2024-04-23T10:41:09.870 INFO:teuthology.orchestra.run.smithi114.stdout: "default_value": "False", 2024-04-23T10:41:09.870 INFO:teuthology.orchestra.run.smithi114.stdout: "min": "", 2024-04-23T10:41:09.871 INFO:teuthology.orchestra.run.smithi114.stdout: "max": "", 2024-04-23T10:41:09.871 INFO:teuthology.orchestra.run.smithi114.stdout: "enum_allowed": [], 2024-04-23T10:41:09.871 INFO:teuthology.orchestra.run.smithi114.stdout: "desc": "", 2024-04-23T10:41:09.871 INFO:teuthology.orchestra.run.smithi114.stdout: "long_desc": "", 2024-04-23T10:41:09.871 INFO:teuthology.orchestra.run.smithi114.stdout: "tags": [], 2024-04-23T10:41:09.871 INFO:teuthology.orchestra.run.smithi114.stdout: "see_also": [] 2024-04-23T10:41:09.871 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-23T10:41:09.871 INFO:teuthology.orchestra.run.smithi114.stdout: "log_to_cluster_level": { 2024-04-23T10:41:09.871 INFO:teuthology.orchestra.run.smithi114.stdout: "name": "log_to_cluster_level", 2024-04-23T10:41:09.871 INFO:teuthology.orchestra.run.smithi114.stdout: "type": "str", 2024-04-23T10:41:09.872 INFO:teuthology.orchestra.run.smithi114.stdout: "level": "advanced", 2024-04-23T10:41:09.872 INFO:teuthology.orchestra.run.smithi114.stdout: "flags": 1, 2024-04-23T10:41:09.872 INFO:teuthology.orchestra.run.smithi114.stdout: "default_value": "info", 2024-04-23T10:41:09.872 INFO:teuthology.orchestra.run.smithi114.stdout: "min": "", 2024-04-23T10:41:09.872 INFO:teuthology.orchestra.run.smithi114.stdout: "max": "", 2024-04-23T10:41:09.872 INFO:teuthology.orchestra.run.smithi114.stdout: "enum_allowed": [ 2024-04-23T10:41:09.872 INFO:teuthology.orchestra.run.smithi114.stdout: "", 2024-04-23T10:41:09.872 INFO:teuthology.orchestra.run.smithi114.stdout: "critical", 2024-04-23T10:41:09.872 INFO:teuthology.orchestra.run.smithi114.stdout: "debug", 2024-04-23T10:41:09.872 INFO:teuthology.orchestra.run.smithi114.stdout: "error", 2024-04-23T10:41:09.873 INFO:teuthology.orchestra.run.smithi114.stdout: "info", 2024-04-23T10:41:09.873 INFO:teuthology.orchestra.run.smithi114.stdout: "warning" 2024-04-23T10:41:09.873 INFO:teuthology.orchestra.run.smithi114.stdout: ], 2024-04-23T10:41:09.873 INFO:teuthology.orchestra.run.smithi114.stdout: "desc": "", 2024-04-23T10:41:09.873 INFO:teuthology.orchestra.run.smithi114.stdout: "long_desc": "", 2024-04-23T10:41:09.873 INFO:teuthology.orchestra.run.smithi114.stdout: "tags": [], 2024-04-23T10:41:09.873 INFO:teuthology.orchestra.run.smithi114.stdout: "see_also": [] 2024-04-23T10:41:09.873 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-23T10:41:09.873 INFO:teuthology.orchestra.run.smithi114.stdout: "log_to_file": { 2024-04-23T10:41:09.873 INFO:teuthology.orchestra.run.smithi114.stdout: "name": "log_to_file", 2024-04-23T10:41:09.874 INFO:teuthology.orchestra.run.smithi114.stdout: "type": "bool", 2024-04-23T10:41:09.874 INFO:teuthology.orchestra.run.smithi114.stdout: "level": "advanced", 2024-04-23T10:41:09.874 INFO:teuthology.orchestra.run.smithi114.stdout: "flags": 1, 2024-04-23T10:41:09.874 INFO:teuthology.orchestra.run.smithi114.stdout: "default_value": "False", 2024-04-23T10:41:09.874 INFO:teuthology.orchestra.run.smithi114.stdout: "min": "", 2024-04-23T10:41:09.874 INFO:teuthology.orchestra.run.smithi114.stdout: "max": "", 2024-04-23T10:41:09.874 INFO:teuthology.orchestra.run.smithi114.stdout: "enum_allowed": [], 2024-04-23T10:41:09.874 INFO:teuthology.orchestra.run.smithi114.stdout: "desc": "", 2024-04-23T10:41:09.874 INFO:teuthology.orchestra.run.smithi114.stdout: "long_desc": "", 2024-04-23T10:41:09.875 INFO:teuthology.orchestra.run.smithi114.stdout: "tags": [], 2024-04-23T10:41:09.875 INFO:teuthology.orchestra.run.smithi114.stdout: "see_also": [] 2024-04-23T10:41:09.875 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-23T10:41:09.875 INFO:teuthology.orchestra.run.smithi114.stdout: "max_completed_events": { 2024-04-23T10:41:09.875 INFO:teuthology.orchestra.run.smithi114.stdout: "name": "max_completed_events", 2024-04-23T10:41:09.875 INFO:teuthology.orchestra.run.smithi114.stdout: "type": "int", 2024-04-23T10:41:09.875 INFO:teuthology.orchestra.run.smithi114.stdout: "level": "advanced", 2024-04-23T10:41:09.875 INFO:teuthology.orchestra.run.smithi114.stdout: "flags": 1, 2024-04-23T10:41:09.875 INFO:teuthology.orchestra.run.smithi114.stdout: "default_value": "50", 2024-04-23T10:41:09.875 INFO:teuthology.orchestra.run.smithi114.stdout: "min": "", 2024-04-23T10:41:09.876 INFO:teuthology.orchestra.run.smithi114.stdout: "max": "", 2024-04-23T10:41:09.876 INFO:teuthology.orchestra.run.smithi114.stdout: "enum_allowed": [], 2024-04-23T10:41:09.876 INFO:teuthology.orchestra.run.smithi114.stdout: "desc": "number of past completed events to remember", 2024-04-23T10:41:09.876 INFO:teuthology.orchestra.run.smithi114.stdout: "long_desc": "", 2024-04-23T10:41:09.876 INFO:teuthology.orchestra.run.smithi114.stdout: "tags": [], 2024-04-23T10:41:09.876 INFO:teuthology.orchestra.run.smithi114.stdout: "see_also": [] 2024-04-23T10:41:09.876 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-23T10:41:09.876 INFO:teuthology.orchestra.run.smithi114.stdout: "sleep_interval": { 2024-04-23T10:41:09.876 INFO:teuthology.orchestra.run.smithi114.stdout: "name": "sleep_interval", 2024-04-23T10:41:09.876 INFO:teuthology.orchestra.run.smithi114.stdout: "type": "secs", 2024-04-23T10:41:09.877 INFO:teuthology.orchestra.run.smithi114.stdout: "level": "advanced", 2024-04-23T10:41:09.877 INFO:teuthology.orchestra.run.smithi114.stdout: "flags": 1, 2024-04-23T10:41:09.877 INFO:teuthology.orchestra.run.smithi114.stdout: "default_value": "5", 2024-04-23T10:41:09.877 INFO:teuthology.orchestra.run.smithi114.stdout: "min": "", 2024-04-23T10:41:09.877 INFO:teuthology.orchestra.run.smithi114.stdout: "max": "", 2024-04-23T10:41:09.877 INFO:teuthology.orchestra.run.smithi114.stdout: "enum_allowed": [], 2024-04-23T10:41:09.877 INFO:teuthology.orchestra.run.smithi114.stdout: "desc": "how long the module is going to sleep", 2024-04-23T10:41:09.877 INFO:teuthology.orchestra.run.smithi114.stdout: "long_desc": "", 2024-04-23T10:41:09.877 INFO:teuthology.orchestra.run.smithi114.stdout: "tags": [], 2024-04-23T10:41:09.877 INFO:teuthology.orchestra.run.smithi114.stdout: "see_also": [] 2024-04-23T10:41:09.878 INFO:teuthology.orchestra.run.smithi114.stdout: } 2024-04-23T10:41:09.878 INFO:teuthology.orchestra.run.smithi114.stdout: } 2024-04-23T10:41:09.878 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-23T10:41:09.878 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-04-23T10:41:09.878 INFO:teuthology.orchestra.run.smithi114.stdout: "name": "prometheus", 2024-04-23T10:41:09.878 INFO:teuthology.orchestra.run.smithi114.stdout: "can_run": true, 2024-04-23T10:41:09.878 INFO:teuthology.orchestra.run.smithi114.stdout: "error_string": "", 2024-04-23T10:41:09.878 INFO:teuthology.orchestra.run.smithi114.stdout: "module_options": { 2024-04-23T10:41:09.878 INFO:teuthology.orchestra.run.smithi114.stdout: "cache": { 2024-04-23T10:41:09.879 INFO:teuthology.orchestra.run.smithi114.stdout: "name": "cache", 2024-04-23T10:41:09.879 INFO:teuthology.orchestra.run.smithi114.stdout: "type": "bool", 2024-04-23T10:41:09.879 INFO:teuthology.orchestra.run.smithi114.stdout: "level": "advanced", 2024-04-23T10:41:09.879 INFO:teuthology.orchestra.run.smithi114.stdout: "flags": 0, 2024-04-23T10:41:09.879 INFO:teuthology.orchestra.run.smithi114.stdout: "default_value": "True", 2024-04-23T10:41:09.879 INFO:teuthology.orchestra.run.smithi114.stdout: "min": "", 2024-04-23T10:41:09.879 INFO:teuthology.orchestra.run.smithi114.stdout: "max": "", 2024-04-23T10:41:09.879 INFO:teuthology.orchestra.run.smithi114.stdout: "enum_allowed": [], 2024-04-23T10:41:09.879 INFO:teuthology.orchestra.run.smithi114.stdout: "desc": "", 2024-04-23T10:41:09.879 INFO:teuthology.orchestra.run.smithi114.stdout: "long_desc": "", 2024-04-23T10:41:09.880 INFO:teuthology.orchestra.run.smithi114.stdout: "tags": [], 2024-04-23T10:41:09.880 INFO:teuthology.orchestra.run.smithi114.stdout: "see_also": [] 2024-04-23T10:41:09.880 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-23T10:41:09.880 INFO:teuthology.orchestra.run.smithi114.stdout: "exclude_perf_counters": { 2024-04-23T10:41:09.880 INFO:teuthology.orchestra.run.smithi114.stdout: "name": "exclude_perf_counters", 2024-04-23T10:41:09.880 INFO:teuthology.orchestra.run.smithi114.stdout: "type": "bool", 2024-04-23T10:41:09.880 INFO:teuthology.orchestra.run.smithi114.stdout: "level": "advanced", 2024-04-23T10:41:09.880 INFO:teuthology.orchestra.run.smithi114.stdout: "flags": 1, 2024-04-23T10:41:09.880 INFO:teuthology.orchestra.run.smithi114.stdout: "default_value": "True", 2024-04-23T10:41:09.880 INFO:teuthology.orchestra.run.smithi114.stdout: "min": "", 2024-04-23T10:41:09.881 INFO:teuthology.orchestra.run.smithi114.stdout: "max": "", 2024-04-23T10:41:09.881 INFO:teuthology.orchestra.run.smithi114.stdout: "enum_allowed": [], 2024-04-23T10:41:09.881 INFO:teuthology.orchestra.run.smithi114.stdout: "desc": "Do not include perf-counters in the metrics output", 2024-04-23T10:41:09.881 INFO:teuthology.orchestra.run.smithi114.stdout: "long_desc": "Gathering perf-counters from a single Prometheus exporter can degrade ceph-mgr performance, especially in large clusters. Instead, Ceph-exporter daemons are now used by default for perf-counter gathering. This should only be disabled when no ceph-exporters are deployed.", 2024-04-23T10:41:09.881 INFO:teuthology.orchestra.run.smithi114.stdout: "tags": [], 2024-04-23T10:41:09.881 INFO:teuthology.orchestra.run.smithi114.stdout: "see_also": [] 2024-04-23T10:41:09.881 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-23T10:41:09.881 INFO:teuthology.orchestra.run.smithi114.stdout: "log_level": { 2024-04-23T10:41:09.881 INFO:teuthology.orchestra.run.smithi114.stdout: "name": "log_level", 2024-04-23T10:41:09.881 INFO:teuthology.orchestra.run.smithi114.stdout: "type": "str", 2024-04-23T10:41:09.882 INFO:teuthology.orchestra.run.smithi114.stdout: "level": "advanced", 2024-04-23T10:41:09.882 INFO:teuthology.orchestra.run.smithi114.stdout: "flags": 1, 2024-04-23T10:41:09.882 INFO:teuthology.orchestra.run.smithi114.stdout: "default_value": "", 2024-04-23T10:41:09.882 INFO:teuthology.orchestra.run.smithi114.stdout: "min": "", 2024-04-23T10:41:09.882 INFO:teuthology.orchestra.run.smithi114.stdout: "max": "", 2024-04-23T10:41:09.882 INFO:teuthology.orchestra.run.smithi114.stdout: "enum_allowed": [ 2024-04-23T10:41:09.882 INFO:teuthology.orchestra.run.smithi114.stdout: "", 2024-04-23T10:41:09.882 INFO:teuthology.orchestra.run.smithi114.stdout: "critical", 2024-04-23T10:41:09.882 INFO:teuthology.orchestra.run.smithi114.stdout: "debug", 2024-04-23T10:41:09.882 INFO:teuthology.orchestra.run.smithi114.stdout: "error", 2024-04-23T10:41:09.883 INFO:teuthology.orchestra.run.smithi114.stdout: "info", 2024-04-23T10:41:09.883 INFO:teuthology.orchestra.run.smithi114.stdout: "warning" 2024-04-23T10:41:09.883 INFO:teuthology.orchestra.run.smithi114.stdout: ], 2024-04-23T10:41:09.883 INFO:teuthology.orchestra.run.smithi114.stdout: "desc": "", 2024-04-23T10:41:09.883 INFO:teuthology.orchestra.run.smithi114.stdout: "long_desc": "", 2024-04-23T10:41:09.883 INFO:teuthology.orchestra.run.smithi114.stdout: "tags": [], 2024-04-23T10:41:09.883 INFO:teuthology.orchestra.run.smithi114.stdout: "see_also": [] 2024-04-23T10:41:09.883 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-23T10:41:09.883 INFO:teuthology.orchestra.run.smithi114.stdout: "log_to_cluster": { 2024-04-23T10:41:09.884 INFO:teuthology.orchestra.run.smithi114.stdout: "name": "log_to_cluster", 2024-04-23T10:41:09.884 INFO:teuthology.orchestra.run.smithi114.stdout: "type": "bool", 2024-04-23T10:41:09.884 INFO:teuthology.orchestra.run.smithi114.stdout: "level": "advanced", 2024-04-23T10:41:09.884 INFO:teuthology.orchestra.run.smithi114.stdout: "flags": 1, 2024-04-23T10:41:09.884 INFO:teuthology.orchestra.run.smithi114.stdout: "default_value": "False", 2024-04-23T10:41:09.884 INFO:teuthology.orchestra.run.smithi114.stdout: "min": "", 2024-04-23T10:41:09.884 INFO:teuthology.orchestra.run.smithi114.stdout: "max": "", 2024-04-23T10:41:09.884 INFO:teuthology.orchestra.run.smithi114.stdout: "enum_allowed": [], 2024-04-23T10:41:09.884 INFO:teuthology.orchestra.run.smithi114.stdout: "desc": "", 2024-04-23T10:41:09.884 INFO:teuthology.orchestra.run.smithi114.stdout: "long_desc": "", 2024-04-23T10:41:09.885 INFO:teuthology.orchestra.run.smithi114.stdout: "tags": [], 2024-04-23T10:41:09.885 INFO:teuthology.orchestra.run.smithi114.stdout: "see_also": [] 2024-04-23T10:41:09.885 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-23T10:41:09.885 INFO:teuthology.orchestra.run.smithi114.stdout: "log_to_cluster_level": { 2024-04-23T10:41:09.885 INFO:teuthology.orchestra.run.smithi114.stdout: "name": "log_to_cluster_level", 2024-04-23T10:41:09.885 INFO:teuthology.orchestra.run.smithi114.stdout: "type": "str", 2024-04-23T10:41:09.885 INFO:teuthology.orchestra.run.smithi114.stdout: "level": "advanced", 2024-04-23T10:41:09.885 INFO:teuthology.orchestra.run.smithi114.stdout: "flags": 1, 2024-04-23T10:41:09.885 INFO:teuthology.orchestra.run.smithi114.stdout: "default_value": "info", 2024-04-23T10:41:09.885 INFO:teuthology.orchestra.run.smithi114.stdout: "min": "", 2024-04-23T10:41:09.885 INFO:teuthology.orchestra.run.smithi114.stdout: "max": "", 2024-04-23T10:41:09.886 INFO:teuthology.orchestra.run.smithi114.stdout: "enum_allowed": [ 2024-04-23T10:41:09.886 INFO:teuthology.orchestra.run.smithi114.stdout: "", 2024-04-23T10:41:09.886 INFO:teuthology.orchestra.run.smithi114.stdout: "critical", 2024-04-23T10:41:09.886 INFO:teuthology.orchestra.run.smithi114.stdout: "debug", 2024-04-23T10:41:09.886 INFO:teuthology.orchestra.run.smithi114.stdout: "error", 2024-04-23T10:41:09.886 INFO:teuthology.orchestra.run.smithi114.stdout: "info", 2024-04-23T10:41:09.886 INFO:teuthology.orchestra.run.smithi114.stdout: "warning" 2024-04-23T10:41:09.886 INFO:teuthology.orchestra.run.smithi114.stdout: ], 2024-04-23T10:41:09.886 INFO:teuthology.orchestra.run.smithi114.stdout: "desc": "", 2024-04-23T10:41:09.887 INFO:teuthology.orchestra.run.smithi114.stdout: "long_desc": "", 2024-04-23T10:41:09.887 INFO:teuthology.orchestra.run.smithi114.stdout: "tags": [], 2024-04-23T10:41:09.887 INFO:teuthology.orchestra.run.smithi114.stdout: "see_also": [] 2024-04-23T10:41:09.887 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-23T10:41:09.887 INFO:teuthology.orchestra.run.smithi114.stdout: "log_to_file": { 2024-04-23T10:41:09.887 INFO:teuthology.orchestra.run.smithi114.stdout: "name": "log_to_file", 2024-04-23T10:41:09.887 INFO:teuthology.orchestra.run.smithi114.stdout: "type": "bool", 2024-04-23T10:41:09.887 INFO:teuthology.orchestra.run.smithi114.stdout: "level": "advanced", 2024-04-23T10:41:09.887 INFO:teuthology.orchestra.run.smithi114.stdout: "flags": 1, 2024-04-23T10:41:09.887 INFO:teuthology.orchestra.run.smithi114.stdout: "default_value": "False", 2024-04-23T10:41:09.888 INFO:teuthology.orchestra.run.smithi114.stdout: "min": "", 2024-04-23T10:41:09.888 INFO:teuthology.orchestra.run.smithi114.stdout: "max": "", 2024-04-23T10:41:09.888 INFO:teuthology.orchestra.run.smithi114.stdout: "enum_allowed": [], 2024-04-23T10:41:09.888 INFO:teuthology.orchestra.run.smithi114.stdout: "desc": "", 2024-04-23T10:41:09.888 INFO:teuthology.orchestra.run.smithi114.stdout: "long_desc": "", 2024-04-23T10:41:09.888 INFO:teuthology.orchestra.run.smithi114.stdout: "tags": [], 2024-04-23T10:41:09.888 INFO:teuthology.orchestra.run.smithi114.stdout: "see_also": [] 2024-04-23T10:41:09.888 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-23T10:41:09.888 INFO:teuthology.orchestra.run.smithi114.stdout: "rbd_stats_pools": { 2024-04-23T10:41:09.888 INFO:teuthology.orchestra.run.smithi114.stdout: "name": "rbd_stats_pools", 2024-04-23T10:41:09.889 INFO:teuthology.orchestra.run.smithi114.stdout: "type": "str", 2024-04-23T10:41:09.889 INFO:teuthology.orchestra.run.smithi114.stdout: "level": "advanced", 2024-04-23T10:41:09.889 INFO:teuthology.orchestra.run.smithi114.stdout: "flags": 0, 2024-04-23T10:41:09.889 INFO:teuthology.orchestra.run.smithi114.stdout: "default_value": "", 2024-04-23T10:41:09.889 INFO:teuthology.orchestra.run.smithi114.stdout: "min": "", 2024-04-23T10:41:09.889 INFO:teuthology.orchestra.run.smithi114.stdout: "max": "", 2024-04-23T10:41:09.889 INFO:teuthology.orchestra.run.smithi114.stdout: "enum_allowed": [], 2024-04-23T10:41:09.889 INFO:teuthology.orchestra.run.smithi114.stdout: "desc": "", 2024-04-23T10:41:09.889 INFO:teuthology.orchestra.run.smithi114.stdout: "long_desc": "", 2024-04-23T10:41:09.889 INFO:teuthology.orchestra.run.smithi114.stdout: "tags": [], 2024-04-23T10:41:09.890 INFO:teuthology.orchestra.run.smithi114.stdout: "see_also": [] 2024-04-23T10:41:09.890 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-23T10:41:09.890 INFO:teuthology.orchestra.run.smithi114.stdout: "rbd_stats_pools_refresh_interval": { 2024-04-23T10:41:09.890 INFO:teuthology.orchestra.run.smithi114.stdout: "name": "rbd_stats_pools_refresh_interval", 2024-04-23T10:41:09.890 INFO:teuthology.orchestra.run.smithi114.stdout: "type": "int", 2024-04-23T10:41:09.890 INFO:teuthology.orchestra.run.smithi114.stdout: "level": "advanced", 2024-04-23T10:41:09.890 INFO:teuthology.orchestra.run.smithi114.stdout: "flags": 0, 2024-04-23T10:41:09.890 INFO:teuthology.orchestra.run.smithi114.stdout: "default_value": "300", 2024-04-23T10:41:09.890 INFO:teuthology.orchestra.run.smithi114.stdout: "min": "", 2024-04-23T10:41:09.891 INFO:teuthology.orchestra.run.smithi114.stdout: "max": "", 2024-04-23T10:41:09.891 INFO:teuthology.orchestra.run.smithi114.stdout: "enum_allowed": [], 2024-04-23T10:41:09.891 INFO:teuthology.orchestra.run.smithi114.stdout: "desc": "", 2024-04-23T10:41:09.891 INFO:teuthology.orchestra.run.smithi114.stdout: "long_desc": "", 2024-04-23T10:41:09.891 INFO:teuthology.orchestra.run.smithi114.stdout: "tags": [], 2024-04-23T10:41:09.891 INFO:teuthology.orchestra.run.smithi114.stdout: "see_also": [] 2024-04-23T10:41:09.891 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-23T10:41:09.891 INFO:teuthology.orchestra.run.smithi114.stdout: "scrape_interval": { 2024-04-23T10:41:09.891 INFO:teuthology.orchestra.run.smithi114.stdout: "name": "scrape_interval", 2024-04-23T10:41:09.891 INFO:teuthology.orchestra.run.smithi114.stdout: "type": "float", 2024-04-23T10:41:09.892 INFO:teuthology.orchestra.run.smithi114.stdout: "level": "advanced", 2024-04-23T10:41:09.892 INFO:teuthology.orchestra.run.smithi114.stdout: "flags": 0, 2024-04-23T10:41:09.892 INFO:teuthology.orchestra.run.smithi114.stdout: "default_value": "15.0", 2024-04-23T10:41:09.892 INFO:teuthology.orchestra.run.smithi114.stdout: "min": "", 2024-04-23T10:41:09.892 INFO:teuthology.orchestra.run.smithi114.stdout: "max": "", 2024-04-23T10:41:09.892 INFO:teuthology.orchestra.run.smithi114.stdout: "enum_allowed": [], 2024-04-23T10:41:09.892 INFO:teuthology.orchestra.run.smithi114.stdout: "desc": "", 2024-04-23T10:41:09.892 INFO:teuthology.orchestra.run.smithi114.stdout: "long_desc": "", 2024-04-23T10:41:09.892 INFO:teuthology.orchestra.run.smithi114.stdout: "tags": [], 2024-04-23T10:41:09.893 INFO:teuthology.orchestra.run.smithi114.stdout: "see_also": [] 2024-04-23T10:41:09.893 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-23T10:41:09.893 INFO:teuthology.orchestra.run.smithi114.stdout: "server_addr": { 2024-04-23T10:41:09.893 INFO:teuthology.orchestra.run.smithi114.stdout: "name": "server_addr", 2024-04-23T10:41:09.893 INFO:teuthology.orchestra.run.smithi114.stdout: "type": "str", 2024-04-23T10:41:09.893 INFO:teuthology.orchestra.run.smithi114.stdout: "level": "advanced", 2024-04-23T10:41:09.893 INFO:teuthology.orchestra.run.smithi114.stdout: "flags": 0, 2024-04-23T10:41:09.893 INFO:teuthology.orchestra.run.smithi114.stdout: "default_value": "::", 2024-04-23T10:41:09.893 INFO:teuthology.orchestra.run.smithi114.stdout: "min": "", 2024-04-23T10:41:09.893 INFO:teuthology.orchestra.run.smithi114.stdout: "max": "", 2024-04-23T10:41:09.894 INFO:teuthology.orchestra.run.smithi114.stdout: "enum_allowed": [], 2024-04-23T10:41:09.894 INFO:teuthology.orchestra.run.smithi114.stdout: "desc": "the IPv4 or IPv6 address on which the module listens for HTTP requests", 2024-04-23T10:41:09.894 INFO:teuthology.orchestra.run.smithi114.stdout: "long_desc": "", 2024-04-23T10:41:09.894 INFO:teuthology.orchestra.run.smithi114.stdout: "tags": [], 2024-04-23T10:41:09.894 INFO:teuthology.orchestra.run.smithi114.stdout: "see_also": [] 2024-04-23T10:41:09.894 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-23T10:41:09.894 INFO:teuthology.orchestra.run.smithi114.stdout: "server_port": { 2024-04-23T10:41:09.894 INFO:teuthology.orchestra.run.smithi114.stdout: "name": "server_port", 2024-04-23T10:41:09.894 INFO:teuthology.orchestra.run.smithi114.stdout: "type": "int", 2024-04-23T10:41:09.895 INFO:teuthology.orchestra.run.smithi114.stdout: "level": "advanced", 2024-04-23T10:41:09.895 INFO:teuthology.orchestra.run.smithi114.stdout: "flags": 1, 2024-04-23T10:41:09.895 INFO:teuthology.orchestra.run.smithi114.stdout: "default_value": "9283", 2024-04-23T10:41:09.895 INFO:teuthology.orchestra.run.smithi114.stdout: "min": "", 2024-04-23T10:41:09.895 INFO:teuthology.orchestra.run.smithi114.stdout: "max": "", 2024-04-23T10:41:09.895 INFO:teuthology.orchestra.run.smithi114.stdout: "enum_allowed": [], 2024-04-23T10:41:09.895 INFO:teuthology.orchestra.run.smithi114.stdout: "desc": "the port on which the module listens for HTTP requests", 2024-04-23T10:41:09.895 INFO:teuthology.orchestra.run.smithi114.stdout: "long_desc": "", 2024-04-23T10:41:09.895 INFO:teuthology.orchestra.run.smithi114.stdout: "tags": [], 2024-04-23T10:41:09.895 INFO:teuthology.orchestra.run.smithi114.stdout: "see_also": [] 2024-04-23T10:41:09.896 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-23T10:41:09.896 INFO:teuthology.orchestra.run.smithi114.stdout: "stale_cache_strategy": { 2024-04-23T10:41:09.896 INFO:teuthology.orchestra.run.smithi114.stdout: "name": "stale_cache_strategy", 2024-04-23T10:41:09.896 INFO:teuthology.orchestra.run.smithi114.stdout: "type": "str", 2024-04-23T10:41:09.896 INFO:teuthology.orchestra.run.smithi114.stdout: "level": "advanced", 2024-04-23T10:41:09.896 INFO:teuthology.orchestra.run.smithi114.stdout: "flags": 0, 2024-04-23T10:41:09.896 INFO:teuthology.orchestra.run.smithi114.stdout: "default_value": "log", 2024-04-23T10:41:09.896 INFO:teuthology.orchestra.run.smithi114.stdout: "min": "", 2024-04-23T10:41:09.896 INFO:teuthology.orchestra.run.smithi114.stdout: "max": "", 2024-04-23T10:41:09.896 INFO:teuthology.orchestra.run.smithi114.stdout: "enum_allowed": [], 2024-04-23T10:41:09.897 INFO:teuthology.orchestra.run.smithi114.stdout: "desc": "", 2024-04-23T10:41:09.897 INFO:teuthology.orchestra.run.smithi114.stdout: "long_desc": "", 2024-04-23T10:41:09.897 INFO:teuthology.orchestra.run.smithi114.stdout: "tags": [], 2024-04-23T10:41:09.897 INFO:teuthology.orchestra.run.smithi114.stdout: "see_also": [] 2024-04-23T10:41:09.897 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-23T10:41:09.897 INFO:teuthology.orchestra.run.smithi114.stdout: "standby_behaviour": { 2024-04-23T10:41:09.897 INFO:teuthology.orchestra.run.smithi114.stdout: "name": "standby_behaviour", 2024-04-23T10:41:09.897 INFO:teuthology.orchestra.run.smithi114.stdout: "type": "str", 2024-04-23T10:41:09.897 INFO:teuthology.orchestra.run.smithi114.stdout: "level": "advanced", 2024-04-23T10:41:09.898 INFO:teuthology.orchestra.run.smithi114.stdout: "flags": 1, 2024-04-23T10:41:09.898 INFO:teuthology.orchestra.run.smithi114.stdout: "default_value": "default", 2024-04-23T10:41:09.898 INFO:teuthology.orchestra.run.smithi114.stdout: "min": "", 2024-04-23T10:41:09.898 INFO:teuthology.orchestra.run.smithi114.stdout: "max": "", 2024-04-23T10:41:09.898 INFO:teuthology.orchestra.run.smithi114.stdout: "enum_allowed": [ 2024-04-23T10:41:09.898 INFO:teuthology.orchestra.run.smithi114.stdout: "default", 2024-04-23T10:41:09.898 INFO:teuthology.orchestra.run.smithi114.stdout: "error" 2024-04-23T10:41:09.898 INFO:teuthology.orchestra.run.smithi114.stdout: ], 2024-04-23T10:41:09.898 INFO:teuthology.orchestra.run.smithi114.stdout: "desc": "", 2024-04-23T10:41:09.898 INFO:teuthology.orchestra.run.smithi114.stdout: "long_desc": "", 2024-04-23T10:41:09.899 INFO:teuthology.orchestra.run.smithi114.stdout: "tags": [], 2024-04-23T10:41:09.899 INFO:teuthology.orchestra.run.smithi114.stdout: "see_also": [] 2024-04-23T10:41:09.899 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-23T10:41:09.899 INFO:teuthology.orchestra.run.smithi114.stdout: "standby_error_status_code": { 2024-04-23T10:41:09.899 INFO:teuthology.orchestra.run.smithi114.stdout: "name": "standby_error_status_code", 2024-04-23T10:41:09.899 INFO:teuthology.orchestra.run.smithi114.stdout: "type": "int", 2024-04-23T10:41:09.899 INFO:teuthology.orchestra.run.smithi114.stdout: "level": "advanced", 2024-04-23T10:41:09.899 INFO:teuthology.orchestra.run.smithi114.stdout: "flags": 1, 2024-04-23T10:41:09.899 INFO:teuthology.orchestra.run.smithi114.stdout: "default_value": "500", 2024-04-23T10:41:09.899 INFO:teuthology.orchestra.run.smithi114.stdout: "min": "400", 2024-04-23T10:41:09.900 INFO:teuthology.orchestra.run.smithi114.stdout: "max": "599", 2024-04-23T10:41:09.900 INFO:teuthology.orchestra.run.smithi114.stdout: "enum_allowed": [], 2024-04-23T10:41:09.900 INFO:teuthology.orchestra.run.smithi114.stdout: "desc": "", 2024-04-23T10:41:09.900 INFO:teuthology.orchestra.run.smithi114.stdout: "long_desc": "", 2024-04-23T10:41:09.900 INFO:teuthology.orchestra.run.smithi114.stdout: "tags": [], 2024-04-23T10:41:09.900 INFO:teuthology.orchestra.run.smithi114.stdout: "see_also": [] 2024-04-23T10:41:09.900 INFO:teuthology.orchestra.run.smithi114.stdout: } 2024-04-23T10:41:09.900 INFO:teuthology.orchestra.run.smithi114.stdout: } 2024-04-23T10:41:09.900 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-23T10:41:09.901 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-04-23T10:41:09.901 INFO:teuthology.orchestra.run.smithi114.stdout: "name": "rbd_support", 2024-04-23T10:41:09.901 INFO:teuthology.orchestra.run.smithi114.stdout: "can_run": true, 2024-04-23T10:41:09.901 INFO:teuthology.orchestra.run.smithi114.stdout: "error_string": "", 2024-04-23T10:41:09.901 INFO:teuthology.orchestra.run.smithi114.stdout: "module_options": { 2024-04-23T10:41:09.901 INFO:teuthology.orchestra.run.smithi114.stdout: "log_level": { 2024-04-23T10:41:09.901 INFO:teuthology.orchestra.run.smithi114.stdout: "name": "log_level", 2024-04-23T10:41:09.901 INFO:teuthology.orchestra.run.smithi114.stdout: "type": "str", 2024-04-23T10:41:09.901 INFO:teuthology.orchestra.run.smithi114.stdout: "level": "advanced", 2024-04-23T10:41:09.901 INFO:teuthology.orchestra.run.smithi114.stdout: "flags": 1, 2024-04-23T10:41:09.902 INFO:teuthology.orchestra.run.smithi114.stdout: "default_value": "", 2024-04-23T10:41:09.902 INFO:teuthology.orchestra.run.smithi114.stdout: "min": "", 2024-04-23T10:41:09.902 INFO:teuthology.orchestra.run.smithi114.stdout: "max": "", 2024-04-23T10:41:09.902 INFO:teuthology.orchestra.run.smithi114.stdout: "enum_allowed": [ 2024-04-23T10:41:09.902 INFO:teuthology.orchestra.run.smithi114.stdout: "", 2024-04-23T10:41:09.902 INFO:teuthology.orchestra.run.smithi114.stdout: "critical", 2024-04-23T10:41:09.902 INFO:teuthology.orchestra.run.smithi114.stdout: "debug", 2024-04-23T10:41:09.902 INFO:teuthology.orchestra.run.smithi114.stdout: "error", 2024-04-23T10:41:09.902 INFO:teuthology.orchestra.run.smithi114.stdout: "info", 2024-04-23T10:41:09.902 INFO:teuthology.orchestra.run.smithi114.stdout: "warning" 2024-04-23T10:41:09.903 INFO:teuthology.orchestra.run.smithi114.stdout: ], 2024-04-23T10:41:09.903 INFO:teuthology.orchestra.run.smithi114.stdout: "desc": "", 2024-04-23T10:41:09.903 INFO:teuthology.orchestra.run.smithi114.stdout: "long_desc": "", 2024-04-23T10:41:09.903 INFO:teuthology.orchestra.run.smithi114.stdout: "tags": [], 2024-04-23T10:41:09.903 INFO:teuthology.orchestra.run.smithi114.stdout: "see_also": [] 2024-04-23T10:41:09.903 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-23T10:41:09.903 INFO:teuthology.orchestra.run.smithi114.stdout: "log_to_cluster": { 2024-04-23T10:41:09.903 INFO:teuthology.orchestra.run.smithi114.stdout: "name": "log_to_cluster", 2024-04-23T10:41:09.903 INFO:teuthology.orchestra.run.smithi114.stdout: "type": "bool", 2024-04-23T10:41:09.903 INFO:teuthology.orchestra.run.smithi114.stdout: "level": "advanced", 2024-04-23T10:41:09.904 INFO:teuthology.orchestra.run.smithi114.stdout: "flags": 1, 2024-04-23T10:41:09.904 INFO:teuthology.orchestra.run.smithi114.stdout: "default_value": "False", 2024-04-23T10:41:09.904 INFO:teuthology.orchestra.run.smithi114.stdout: "min": "", 2024-04-23T10:41:09.904 INFO:teuthology.orchestra.run.smithi114.stdout: "max": "", 2024-04-23T10:41:09.904 INFO:teuthology.orchestra.run.smithi114.stdout: "enum_allowed": [], 2024-04-23T10:41:09.904 INFO:teuthology.orchestra.run.smithi114.stdout: "desc": "", 2024-04-23T10:41:09.904 INFO:teuthology.orchestra.run.smithi114.stdout: "long_desc": "", 2024-04-23T10:41:09.904 INFO:teuthology.orchestra.run.smithi114.stdout: "tags": [], 2024-04-23T10:41:09.904 INFO:teuthology.orchestra.run.smithi114.stdout: "see_also": [] 2024-04-23T10:41:09.904 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-23T10:41:09.905 INFO:teuthology.orchestra.run.smithi114.stdout: "log_to_cluster_level": { 2024-04-23T10:41:09.905 INFO:teuthology.orchestra.run.smithi114.stdout: "name": "log_to_cluster_level", 2024-04-23T10:41:09.905 INFO:teuthology.orchestra.run.smithi114.stdout: "type": "str", 2024-04-23T10:41:09.905 INFO:teuthology.orchestra.run.smithi114.stdout: "level": "advanced", 2024-04-23T10:41:09.905 INFO:teuthology.orchestra.run.smithi114.stdout: "flags": 1, 2024-04-23T10:41:09.905 INFO:teuthology.orchestra.run.smithi114.stdout: "default_value": "info", 2024-04-23T10:41:09.905 INFO:teuthology.orchestra.run.smithi114.stdout: "min": "", 2024-04-23T10:41:09.905 INFO:teuthology.orchestra.run.smithi114.stdout: "max": "", 2024-04-23T10:41:09.905 INFO:teuthology.orchestra.run.smithi114.stdout: "enum_allowed": [ 2024-04-23T10:41:09.905 INFO:teuthology.orchestra.run.smithi114.stdout: "", 2024-04-23T10:41:09.906 INFO:teuthology.orchestra.run.smithi114.stdout: "critical", 2024-04-23T10:41:09.906 INFO:teuthology.orchestra.run.smithi114.stdout: "debug", 2024-04-23T10:41:09.906 INFO:teuthology.orchestra.run.smithi114.stdout: "error", 2024-04-23T10:41:09.906 INFO:teuthology.orchestra.run.smithi114.stdout: "info", 2024-04-23T10:41:09.906 INFO:teuthology.orchestra.run.smithi114.stdout: "warning" 2024-04-23T10:41:09.906 INFO:teuthology.orchestra.run.smithi114.stdout: ], 2024-04-23T10:41:09.906 INFO:teuthology.orchestra.run.smithi114.stdout: "desc": "", 2024-04-23T10:41:09.906 INFO:teuthology.orchestra.run.smithi114.stdout: "long_desc": "", 2024-04-23T10:41:09.906 INFO:teuthology.orchestra.run.smithi114.stdout: "tags": [], 2024-04-23T10:41:09.907 INFO:teuthology.orchestra.run.smithi114.stdout: "see_also": [] 2024-04-23T10:41:09.907 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-23T10:41:09.907 INFO:teuthology.orchestra.run.smithi114.stdout: "log_to_file": { 2024-04-23T10:41:09.907 INFO:teuthology.orchestra.run.smithi114.stdout: "name": "log_to_file", 2024-04-23T10:41:09.907 INFO:teuthology.orchestra.run.smithi114.stdout: "type": "bool", 2024-04-23T10:41:09.907 INFO:teuthology.orchestra.run.smithi114.stdout: "level": "advanced", 2024-04-23T10:41:09.907 INFO:teuthology.orchestra.run.smithi114.stdout: "flags": 1, 2024-04-23T10:41:09.907 INFO:teuthology.orchestra.run.smithi114.stdout: "default_value": "False", 2024-04-23T10:41:09.907 INFO:teuthology.orchestra.run.smithi114.stdout: "min": "", 2024-04-23T10:41:09.907 INFO:teuthology.orchestra.run.smithi114.stdout: "max": "", 2024-04-23T10:41:09.908 INFO:teuthology.orchestra.run.smithi114.stdout: "enum_allowed": [], 2024-04-23T10:41:09.908 INFO:teuthology.orchestra.run.smithi114.stdout: "desc": "", 2024-04-23T10:41:09.908 INFO:teuthology.orchestra.run.smithi114.stdout: "long_desc": "", 2024-04-23T10:41:09.908 INFO:teuthology.orchestra.run.smithi114.stdout: "tags": [], 2024-04-23T10:41:09.908 INFO:teuthology.orchestra.run.smithi114.stdout: "see_also": [] 2024-04-23T10:41:09.908 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-23T10:41:09.908 INFO:teuthology.orchestra.run.smithi114.stdout: "max_concurrent_snap_create": { 2024-04-23T10:41:09.908 INFO:teuthology.orchestra.run.smithi114.stdout: "name": "max_concurrent_snap_create", 2024-04-23T10:41:09.908 INFO:teuthology.orchestra.run.smithi114.stdout: "type": "int", 2024-04-23T10:41:09.908 INFO:teuthology.orchestra.run.smithi114.stdout: "level": "advanced", 2024-04-23T10:41:09.909 INFO:teuthology.orchestra.run.smithi114.stdout: "flags": 0, 2024-04-23T10:41:09.909 INFO:teuthology.orchestra.run.smithi114.stdout: "default_value": "10", 2024-04-23T10:41:09.909 INFO:teuthology.orchestra.run.smithi114.stdout: "min": "", 2024-04-23T10:41:09.909 INFO:teuthology.orchestra.run.smithi114.stdout: "max": "", 2024-04-23T10:41:09.909 INFO:teuthology.orchestra.run.smithi114.stdout: "enum_allowed": [], 2024-04-23T10:41:09.909 INFO:teuthology.orchestra.run.smithi114.stdout: "desc": "", 2024-04-23T10:41:09.909 INFO:teuthology.orchestra.run.smithi114.stdout: "long_desc": "", 2024-04-23T10:41:09.909 INFO:teuthology.orchestra.run.smithi114.stdout: "tags": [], 2024-04-23T10:41:09.909 INFO:teuthology.orchestra.run.smithi114.stdout: "see_also": [] 2024-04-23T10:41:09.909 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-23T10:41:09.910 INFO:teuthology.orchestra.run.smithi114.stdout: "mirror_snapshot_schedule": { 2024-04-23T10:41:09.910 INFO:teuthology.orchestra.run.smithi114.stdout: "name": "mirror_snapshot_schedule", 2024-04-23T10:41:09.910 INFO:teuthology.orchestra.run.smithi114.stdout: "type": "str", 2024-04-23T10:41:09.910 INFO:teuthology.orchestra.run.smithi114.stdout: "level": "advanced", 2024-04-23T10:41:09.910 INFO:teuthology.orchestra.run.smithi114.stdout: "flags": 0, 2024-04-23T10:41:09.910 INFO:teuthology.orchestra.run.smithi114.stdout: "default_value": "", 2024-04-23T10:41:09.910 INFO:teuthology.orchestra.run.smithi114.stdout: "min": "", 2024-04-23T10:41:09.910 INFO:teuthology.orchestra.run.smithi114.stdout: "max": "", 2024-04-23T10:41:09.910 INFO:teuthology.orchestra.run.smithi114.stdout: "enum_allowed": [], 2024-04-23T10:41:09.911 INFO:teuthology.orchestra.run.smithi114.stdout: "desc": "", 2024-04-23T10:41:09.911 INFO:teuthology.orchestra.run.smithi114.stdout: "long_desc": "", 2024-04-23T10:41:09.911 INFO:teuthology.orchestra.run.smithi114.stdout: "tags": [], 2024-04-23T10:41:09.911 INFO:teuthology.orchestra.run.smithi114.stdout: "see_also": [] 2024-04-23T10:41:09.911 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-23T10:41:09.911 INFO:teuthology.orchestra.run.smithi114.stdout: "trash_purge_schedule": { 2024-04-23T10:41:09.911 INFO:teuthology.orchestra.run.smithi114.stdout: "name": "trash_purge_schedule", 2024-04-23T10:41:09.911 INFO:teuthology.orchestra.run.smithi114.stdout: "type": "str", 2024-04-23T10:41:09.911 INFO:teuthology.orchestra.run.smithi114.stdout: "level": "advanced", 2024-04-23T10:41:09.911 INFO:teuthology.orchestra.run.smithi114.stdout: "flags": 0, 2024-04-23T10:41:09.912 INFO:teuthology.orchestra.run.smithi114.stdout: "default_value": "", 2024-04-23T10:41:09.912 INFO:teuthology.orchestra.run.smithi114.stdout: "min": "", 2024-04-23T10:41:09.912 INFO:teuthology.orchestra.run.smithi114.stdout: "max": "", 2024-04-23T10:41:09.912 INFO:teuthology.orchestra.run.smithi114.stdout: "enum_allowed": [], 2024-04-23T10:41:09.912 INFO:teuthology.orchestra.run.smithi114.stdout: "desc": "", 2024-04-23T10:41:09.912 INFO:teuthology.orchestra.run.smithi114.stdout: "long_desc": "", 2024-04-23T10:41:09.912 INFO:teuthology.orchestra.run.smithi114.stdout: "tags": [], 2024-04-23T10:41:09.912 INFO:teuthology.orchestra.run.smithi114.stdout: "see_also": [] 2024-04-23T10:41:09.912 INFO:teuthology.orchestra.run.smithi114.stdout: } 2024-04-23T10:41:09.913 INFO:teuthology.orchestra.run.smithi114.stdout: } 2024-04-23T10:41:09.913 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-23T10:41:09.913 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-04-23T10:41:09.913 INFO:teuthology.orchestra.run.smithi114.stdout: "name": "restful", 2024-04-23T10:41:09.913 INFO:teuthology.orchestra.run.smithi114.stdout: "can_run": true, 2024-04-23T10:41:09.913 INFO:teuthology.orchestra.run.smithi114.stdout: "error_string": "", 2024-04-23T10:41:09.913 INFO:teuthology.orchestra.run.smithi114.stdout: "module_options": { 2024-04-23T10:41:09.913 INFO:teuthology.orchestra.run.smithi114.stdout: "enable_auth": { 2024-04-23T10:41:09.913 INFO:teuthology.orchestra.run.smithi114.stdout: "name": "enable_auth", 2024-04-23T10:41:09.913 INFO:teuthology.orchestra.run.smithi114.stdout: "type": "bool", 2024-04-23T10:41:09.914 INFO:teuthology.orchestra.run.smithi114.stdout: "level": "advanced", 2024-04-23T10:41:09.914 INFO:teuthology.orchestra.run.smithi114.stdout: "flags": 0, 2024-04-23T10:41:09.914 INFO:teuthology.orchestra.run.smithi114.stdout: "default_value": "True", 2024-04-23T10:41:09.914 INFO:teuthology.orchestra.run.smithi114.stdout: "min": "", 2024-04-23T10:41:09.914 INFO:teuthology.orchestra.run.smithi114.stdout: "max": "", 2024-04-23T10:41:09.914 INFO:teuthology.orchestra.run.smithi114.stdout: "enum_allowed": [], 2024-04-23T10:41:09.914 INFO:teuthology.orchestra.run.smithi114.stdout: "desc": "", 2024-04-23T10:41:09.914 INFO:teuthology.orchestra.run.smithi114.stdout: "long_desc": "", 2024-04-23T10:41:09.914 INFO:teuthology.orchestra.run.smithi114.stdout: "tags": [], 2024-04-23T10:41:09.914 INFO:teuthology.orchestra.run.smithi114.stdout: "see_also": [] 2024-04-23T10:41:09.915 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-23T10:41:09.915 INFO:teuthology.orchestra.run.smithi114.stdout: "key_file": { 2024-04-23T10:41:09.915 INFO:teuthology.orchestra.run.smithi114.stdout: "name": "key_file", 2024-04-23T10:41:09.915 INFO:teuthology.orchestra.run.smithi114.stdout: "type": "str", 2024-04-23T10:41:09.915 INFO:teuthology.orchestra.run.smithi114.stdout: "level": "advanced", 2024-04-23T10:41:09.915 INFO:teuthology.orchestra.run.smithi114.stdout: "flags": 0, 2024-04-23T10:41:09.915 INFO:teuthology.orchestra.run.smithi114.stdout: "default_value": "", 2024-04-23T10:41:09.915 INFO:teuthology.orchestra.run.smithi114.stdout: "min": "", 2024-04-23T10:41:09.915 INFO:teuthology.orchestra.run.smithi114.stdout: "max": "", 2024-04-23T10:41:09.915 INFO:teuthology.orchestra.run.smithi114.stdout: "enum_allowed": [], 2024-04-23T10:41:09.916 INFO:teuthology.orchestra.run.smithi114.stdout: "desc": "", 2024-04-23T10:41:09.916 INFO:teuthology.orchestra.run.smithi114.stdout: "long_desc": "", 2024-04-23T10:41:09.916 INFO:teuthology.orchestra.run.smithi114.stdout: "tags": [], 2024-04-23T10:41:09.916 INFO:teuthology.orchestra.run.smithi114.stdout: "see_also": [] 2024-04-23T10:41:09.916 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-23T10:41:09.916 INFO:teuthology.orchestra.run.smithi114.stdout: "log_level": { 2024-04-23T10:41:09.916 INFO:teuthology.orchestra.run.smithi114.stdout: "name": "log_level", 2024-04-23T10:41:09.916 INFO:teuthology.orchestra.run.smithi114.stdout: "type": "str", 2024-04-23T10:41:09.916 INFO:teuthology.orchestra.run.smithi114.stdout: "level": "advanced", 2024-04-23T10:41:09.916 INFO:teuthology.orchestra.run.smithi114.stdout: "flags": 1, 2024-04-23T10:41:09.917 INFO:teuthology.orchestra.run.smithi114.stdout: "default_value": "", 2024-04-23T10:41:09.917 INFO:teuthology.orchestra.run.smithi114.stdout: "min": "", 2024-04-23T10:41:09.917 INFO:teuthology.orchestra.run.smithi114.stdout: "max": "", 2024-04-23T10:41:09.917 INFO:teuthology.orchestra.run.smithi114.stdout: "enum_allowed": [ 2024-04-23T10:41:09.917 INFO:teuthology.orchestra.run.smithi114.stdout: "", 2024-04-23T10:41:09.917 INFO:teuthology.orchestra.run.smithi114.stdout: "critical", 2024-04-23T10:41:09.917 INFO:teuthology.orchestra.run.smithi114.stdout: "debug", 2024-04-23T10:41:09.917 INFO:teuthology.orchestra.run.smithi114.stdout: "error", 2024-04-23T10:41:09.917 INFO:teuthology.orchestra.run.smithi114.stdout: "info", 2024-04-23T10:41:09.917 INFO:teuthology.orchestra.run.smithi114.stdout: "warning" 2024-04-23T10:41:09.918 INFO:teuthology.orchestra.run.smithi114.stdout: ], 2024-04-23T10:41:09.918 INFO:teuthology.orchestra.run.smithi114.stdout: "desc": "", 2024-04-23T10:41:09.918 INFO:teuthology.orchestra.run.smithi114.stdout: "long_desc": "", 2024-04-23T10:41:09.918 INFO:teuthology.orchestra.run.smithi114.stdout: "tags": [], 2024-04-23T10:41:09.918 INFO:teuthology.orchestra.run.smithi114.stdout: "see_also": [] 2024-04-23T10:41:09.918 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-23T10:41:09.918 INFO:teuthology.orchestra.run.smithi114.stdout: "log_to_cluster": { 2024-04-23T10:41:09.918 INFO:teuthology.orchestra.run.smithi114.stdout: "name": "log_to_cluster", 2024-04-23T10:41:09.918 INFO:teuthology.orchestra.run.smithi114.stdout: "type": "bool", 2024-04-23T10:41:09.919 INFO:teuthology.orchestra.run.smithi114.stdout: "level": "advanced", 2024-04-23T10:41:09.919 INFO:teuthology.orchestra.run.smithi114.stdout: "flags": 1, 2024-04-23T10:41:09.919 INFO:teuthology.orchestra.run.smithi114.stdout: "default_value": "False", 2024-04-23T10:41:09.919 INFO:teuthology.orchestra.run.smithi114.stdout: "min": "", 2024-04-23T10:41:09.919 INFO:teuthology.orchestra.run.smithi114.stdout: "max": "", 2024-04-23T10:41:09.919 INFO:teuthology.orchestra.run.smithi114.stdout: "enum_allowed": [], 2024-04-23T10:41:09.919 INFO:teuthology.orchestra.run.smithi114.stdout: "desc": "", 2024-04-23T10:41:09.919 INFO:teuthology.orchestra.run.smithi114.stdout: "long_desc": "", 2024-04-23T10:41:09.919 INFO:teuthology.orchestra.run.smithi114.stdout: "tags": [], 2024-04-23T10:41:09.919 INFO:teuthology.orchestra.run.smithi114.stdout: "see_also": [] 2024-04-23T10:41:09.920 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-23T10:41:09.920 INFO:teuthology.orchestra.run.smithi114.stdout: "log_to_cluster_level": { 2024-04-23T10:41:09.920 INFO:teuthology.orchestra.run.smithi114.stdout: "name": "log_to_cluster_level", 2024-04-23T10:41:09.920 INFO:teuthology.orchestra.run.smithi114.stdout: "type": "str", 2024-04-23T10:41:09.920 INFO:teuthology.orchestra.run.smithi114.stdout: "level": "advanced", 2024-04-23T10:41:09.920 INFO:teuthology.orchestra.run.smithi114.stdout: "flags": 1, 2024-04-23T10:41:09.920 INFO:teuthology.orchestra.run.smithi114.stdout: "default_value": "info", 2024-04-23T10:41:09.920 INFO:teuthology.orchestra.run.smithi114.stdout: "min": "", 2024-04-23T10:41:09.920 INFO:teuthology.orchestra.run.smithi114.stdout: "max": "", 2024-04-23T10:41:09.920 INFO:teuthology.orchestra.run.smithi114.stdout: "enum_allowed": [ 2024-04-23T10:41:09.921 INFO:teuthology.orchestra.run.smithi114.stdout: "", 2024-04-23T10:41:09.921 INFO:teuthology.orchestra.run.smithi114.stdout: "critical", 2024-04-23T10:41:09.921 INFO:teuthology.orchestra.run.smithi114.stdout: "debug", 2024-04-23T10:41:09.921 INFO:teuthology.orchestra.run.smithi114.stdout: "error", 2024-04-23T10:41:09.921 INFO:teuthology.orchestra.run.smithi114.stdout: "info", 2024-04-23T10:41:09.921 INFO:teuthology.orchestra.run.smithi114.stdout: "warning" 2024-04-23T10:41:09.921 INFO:teuthology.orchestra.run.smithi114.stdout: ], 2024-04-23T10:41:09.921 INFO:teuthology.orchestra.run.smithi114.stdout: "desc": "", 2024-04-23T10:41:09.921 INFO:teuthology.orchestra.run.smithi114.stdout: "long_desc": "", 2024-04-23T10:41:09.921 INFO:teuthology.orchestra.run.smithi114.stdout: "tags": [], 2024-04-23T10:41:09.922 INFO:teuthology.orchestra.run.smithi114.stdout: "see_also": [] 2024-04-23T10:41:09.922 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-23T10:41:09.922 INFO:teuthology.orchestra.run.smithi114.stdout: "log_to_file": { 2024-04-23T10:41:09.922 INFO:teuthology.orchestra.run.smithi114.stdout: "name": "log_to_file", 2024-04-23T10:41:09.922 INFO:teuthology.orchestra.run.smithi114.stdout: "type": "bool", 2024-04-23T10:41:09.922 INFO:teuthology.orchestra.run.smithi114.stdout: "level": "advanced", 2024-04-23T10:41:09.922 INFO:teuthology.orchestra.run.smithi114.stdout: "flags": 1, 2024-04-23T10:41:09.922 INFO:teuthology.orchestra.run.smithi114.stdout: "default_value": "False", 2024-04-23T10:41:09.922 INFO:teuthology.orchestra.run.smithi114.stdout: "min": "", 2024-04-23T10:41:09.923 INFO:teuthology.orchestra.run.smithi114.stdout: "max": "", 2024-04-23T10:41:09.923 INFO:teuthology.orchestra.run.smithi114.stdout: "enum_allowed": [], 2024-04-23T10:41:09.923 INFO:teuthology.orchestra.run.smithi114.stdout: "desc": "", 2024-04-23T10:41:09.923 INFO:teuthology.orchestra.run.smithi114.stdout: "long_desc": "", 2024-04-23T10:41:09.923 INFO:teuthology.orchestra.run.smithi114.stdout: "tags": [], 2024-04-23T10:41:09.923 INFO:teuthology.orchestra.run.smithi114.stdout: "see_also": [] 2024-04-23T10:41:09.923 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-23T10:41:09.923 INFO:teuthology.orchestra.run.smithi114.stdout: "server_addr": { 2024-04-23T10:41:09.923 INFO:teuthology.orchestra.run.smithi114.stdout: "name": "server_addr", 2024-04-23T10:41:09.923 INFO:teuthology.orchestra.run.smithi114.stdout: "type": "str", 2024-04-23T10:41:09.924 INFO:teuthology.orchestra.run.smithi114.stdout: "level": "advanced", 2024-04-23T10:41:09.924 INFO:teuthology.orchestra.run.smithi114.stdout: "flags": 0, 2024-04-23T10:41:09.924 INFO:teuthology.orchestra.run.smithi114.stdout: "default_value": "", 2024-04-23T10:41:09.924 INFO:teuthology.orchestra.run.smithi114.stdout: "min": "", 2024-04-23T10:41:09.924 INFO:teuthology.orchestra.run.smithi114.stdout: "max": "", 2024-04-23T10:41:09.924 INFO:teuthology.orchestra.run.smithi114.stdout: "enum_allowed": [], 2024-04-23T10:41:09.924 INFO:teuthology.orchestra.run.smithi114.stdout: "desc": "", 2024-04-23T10:41:09.924 INFO:teuthology.orchestra.run.smithi114.stdout: "long_desc": "", 2024-04-23T10:41:09.924 INFO:teuthology.orchestra.run.smithi114.stdout: "tags": [], 2024-04-23T10:41:09.924 INFO:teuthology.orchestra.run.smithi114.stdout: "see_also": [] 2024-04-23T10:41:09.925 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-23T10:41:09.925 INFO:teuthology.orchestra.run.smithi114.stdout: "server_port": { 2024-04-23T10:41:09.925 INFO:teuthology.orchestra.run.smithi114.stdout: "name": "server_port", 2024-04-23T10:41:09.925 INFO:teuthology.orchestra.run.smithi114.stdout: "type": "str", 2024-04-23T10:41:09.925 INFO:teuthology.orchestra.run.smithi114.stdout: "level": "advanced", 2024-04-23T10:41:09.925 INFO:teuthology.orchestra.run.smithi114.stdout: "flags": 0, 2024-04-23T10:41:09.925 INFO:teuthology.orchestra.run.smithi114.stdout: "default_value": "", 2024-04-23T10:41:09.925 INFO:teuthology.orchestra.run.smithi114.stdout: "min": "", 2024-04-23T10:41:09.925 INFO:teuthology.orchestra.run.smithi114.stdout: "max": "", 2024-04-23T10:41:09.925 INFO:teuthology.orchestra.run.smithi114.stdout: "enum_allowed": [], 2024-04-23T10:41:09.926 INFO:teuthology.orchestra.run.smithi114.stdout: "desc": "", 2024-04-23T10:41:09.926 INFO:teuthology.orchestra.run.smithi114.stdout: "long_desc": "", 2024-04-23T10:41:09.926 INFO:teuthology.orchestra.run.smithi114.stdout: "tags": [], 2024-04-23T10:41:09.926 INFO:teuthology.orchestra.run.smithi114.stdout: "see_also": [] 2024-04-23T10:41:09.926 INFO:teuthology.orchestra.run.smithi114.stdout: } 2024-04-23T10:41:09.926 INFO:teuthology.orchestra.run.smithi114.stdout: } 2024-04-23T10:41:09.926 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-23T10:41:09.926 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-04-23T10:41:09.926 INFO:teuthology.orchestra.run.smithi114.stdout: "name": "rgw", 2024-04-23T10:41:09.927 INFO:teuthology.orchestra.run.smithi114.stdout: "can_run": true, 2024-04-23T10:41:09.927 INFO:teuthology.orchestra.run.smithi114.stdout: "error_string": "", 2024-04-23T10:41:09.927 INFO:teuthology.orchestra.run.smithi114.stdout: "module_options": { 2024-04-23T10:41:09.927 INFO:teuthology.orchestra.run.smithi114.stdout: "log_level": { 2024-04-23T10:41:09.927 INFO:teuthology.orchestra.run.smithi114.stdout: "name": "log_level", 2024-04-23T10:41:09.927 INFO:teuthology.orchestra.run.smithi114.stdout: "type": "str", 2024-04-23T10:41:09.927 INFO:teuthology.orchestra.run.smithi114.stdout: "level": "advanced", 2024-04-23T10:41:09.927 INFO:teuthology.orchestra.run.smithi114.stdout: "flags": 1, 2024-04-23T10:41:09.927 INFO:teuthology.orchestra.run.smithi114.stdout: "default_value": "", 2024-04-23T10:41:09.927 INFO:teuthology.orchestra.run.smithi114.stdout: "min": "", 2024-04-23T10:41:09.928 INFO:teuthology.orchestra.run.smithi114.stdout: "max": "", 2024-04-23T10:41:09.928 INFO:teuthology.orchestra.run.smithi114.stdout: "enum_allowed": [ 2024-04-23T10:41:09.928 INFO:teuthology.orchestra.run.smithi114.stdout: "", 2024-04-23T10:41:09.928 INFO:teuthology.orchestra.run.smithi114.stdout: "critical", 2024-04-23T10:41:09.928 INFO:teuthology.orchestra.run.smithi114.stdout: "debug", 2024-04-23T10:41:09.928 INFO:teuthology.orchestra.run.smithi114.stdout: "error", 2024-04-23T10:41:09.928 INFO:teuthology.orchestra.run.smithi114.stdout: "info", 2024-04-23T10:41:09.928 INFO:teuthology.orchestra.run.smithi114.stdout: "warning" 2024-04-23T10:41:09.928 INFO:teuthology.orchestra.run.smithi114.stdout: ], 2024-04-23T10:41:09.928 INFO:teuthology.orchestra.run.smithi114.stdout: "desc": "", 2024-04-23T10:41:09.929 INFO:teuthology.orchestra.run.smithi114.stdout: "long_desc": "", 2024-04-23T10:41:09.929 INFO:teuthology.orchestra.run.smithi114.stdout: "tags": [], 2024-04-23T10:41:09.929 INFO:teuthology.orchestra.run.smithi114.stdout: "see_also": [] 2024-04-23T10:41:09.929 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-23T10:41:09.929 INFO:teuthology.orchestra.run.smithi114.stdout: "log_to_cluster": { 2024-04-23T10:41:09.929 INFO:teuthology.orchestra.run.smithi114.stdout: "name": "log_to_cluster", 2024-04-23T10:41:09.929 INFO:teuthology.orchestra.run.smithi114.stdout: "type": "bool", 2024-04-23T10:41:09.929 INFO:teuthology.orchestra.run.smithi114.stdout: "level": "advanced", 2024-04-23T10:41:09.929 INFO:teuthology.orchestra.run.smithi114.stdout: "flags": 1, 2024-04-23T10:41:09.929 INFO:teuthology.orchestra.run.smithi114.stdout: "default_value": "False", 2024-04-23T10:41:09.930 INFO:teuthology.orchestra.run.smithi114.stdout: "min": "", 2024-04-23T10:41:09.930 INFO:teuthology.orchestra.run.smithi114.stdout: "max": "", 2024-04-23T10:41:09.930 INFO:teuthology.orchestra.run.smithi114.stdout: "enum_allowed": [], 2024-04-23T10:41:09.930 INFO:teuthology.orchestra.run.smithi114.stdout: "desc": "", 2024-04-23T10:41:09.930 INFO:teuthology.orchestra.run.smithi114.stdout: "long_desc": "", 2024-04-23T10:41:09.930 INFO:teuthology.orchestra.run.smithi114.stdout: "tags": [], 2024-04-23T10:41:09.930 INFO:teuthology.orchestra.run.smithi114.stdout: "see_also": [] 2024-04-23T10:41:09.930 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-23T10:41:09.930 INFO:teuthology.orchestra.run.smithi114.stdout: "log_to_cluster_level": { 2024-04-23T10:41:09.930 INFO:teuthology.orchestra.run.smithi114.stdout: "name": "log_to_cluster_level", 2024-04-23T10:41:09.931 INFO:teuthology.orchestra.run.smithi114.stdout: "type": "str", 2024-04-23T10:41:09.931 INFO:teuthology.orchestra.run.smithi114.stdout: "level": "advanced", 2024-04-23T10:41:09.931 INFO:teuthology.orchestra.run.smithi114.stdout: "flags": 1, 2024-04-23T10:41:09.931 INFO:teuthology.orchestra.run.smithi114.stdout: "default_value": "info", 2024-04-23T10:41:09.931 INFO:teuthology.orchestra.run.smithi114.stdout: "min": "", 2024-04-23T10:41:09.931 INFO:teuthology.orchestra.run.smithi114.stdout: "max": "", 2024-04-23T10:41:09.931 INFO:teuthology.orchestra.run.smithi114.stdout: "enum_allowed": [ 2024-04-23T10:41:09.931 INFO:teuthology.orchestra.run.smithi114.stdout: "", 2024-04-23T10:41:09.931 INFO:teuthology.orchestra.run.smithi114.stdout: "critical", 2024-04-23T10:41:09.931 INFO:teuthology.orchestra.run.smithi114.stdout: "debug", 2024-04-23T10:41:09.932 INFO:teuthology.orchestra.run.smithi114.stdout: "error", 2024-04-23T10:41:09.932 INFO:teuthology.orchestra.run.smithi114.stdout: "info", 2024-04-23T10:41:09.932 INFO:teuthology.orchestra.run.smithi114.stdout: "warning" 2024-04-23T10:41:09.932 INFO:teuthology.orchestra.run.smithi114.stdout: ], 2024-04-23T10:41:09.932 INFO:teuthology.orchestra.run.smithi114.stdout: "desc": "", 2024-04-23T10:41:09.932 INFO:teuthology.orchestra.run.smithi114.stdout: "long_desc": "", 2024-04-23T10:41:09.932 INFO:teuthology.orchestra.run.smithi114.stdout: "tags": [], 2024-04-23T10:41:09.932 INFO:teuthology.orchestra.run.smithi114.stdout: "see_also": [] 2024-04-23T10:41:09.932 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-23T10:41:09.932 INFO:teuthology.orchestra.run.smithi114.stdout: "log_to_file": { 2024-04-23T10:41:09.933 INFO:teuthology.orchestra.run.smithi114.stdout: "name": "log_to_file", 2024-04-23T10:41:09.933 INFO:teuthology.orchestra.run.smithi114.stdout: "type": "bool", 2024-04-23T10:41:09.933 INFO:teuthology.orchestra.run.smithi114.stdout: "level": "advanced", 2024-04-23T10:41:09.933 INFO:teuthology.orchestra.run.smithi114.stdout: "flags": 1, 2024-04-23T10:41:09.933 INFO:teuthology.orchestra.run.smithi114.stdout: "default_value": "False", 2024-04-23T10:41:09.933 INFO:teuthology.orchestra.run.smithi114.stdout: "min": "", 2024-04-23T10:41:09.933 INFO:teuthology.orchestra.run.smithi114.stdout: "max": "", 2024-04-23T10:41:09.933 INFO:teuthology.orchestra.run.smithi114.stdout: "enum_allowed": [], 2024-04-23T10:41:09.933 INFO:teuthology.orchestra.run.smithi114.stdout: "desc": "", 2024-04-23T10:41:09.933 INFO:teuthology.orchestra.run.smithi114.stdout: "long_desc": "", 2024-04-23T10:41:09.934 INFO:teuthology.orchestra.run.smithi114.stdout: "tags": [], 2024-04-23T10:41:09.934 INFO:teuthology.orchestra.run.smithi114.stdout: "see_also": [] 2024-04-23T10:41:09.934 INFO:teuthology.orchestra.run.smithi114.stdout: } 2024-04-23T10:41:09.934 INFO:teuthology.orchestra.run.smithi114.stdout: } 2024-04-23T10:41:09.934 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-23T10:41:09.934 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-04-23T10:41:09.934 INFO:teuthology.orchestra.run.smithi114.stdout: "name": "rook", 2024-04-23T10:41:09.934 INFO:teuthology.orchestra.run.smithi114.stdout: "can_run": true, 2024-04-23T10:41:09.934 INFO:teuthology.orchestra.run.smithi114.stdout: "error_string": "", 2024-04-23T10:41:09.935 INFO:teuthology.orchestra.run.smithi114.stdout: "module_options": { 2024-04-23T10:41:09.935 INFO:teuthology.orchestra.run.smithi114.stdout: "log_level": { 2024-04-23T10:41:09.935 INFO:teuthology.orchestra.run.smithi114.stdout: "name": "log_level", 2024-04-23T10:41:09.935 INFO:teuthology.orchestra.run.smithi114.stdout: "type": "str", 2024-04-23T10:41:09.935 INFO:teuthology.orchestra.run.smithi114.stdout: "level": "advanced", 2024-04-23T10:41:09.935 INFO:teuthology.orchestra.run.smithi114.stdout: "flags": 1, 2024-04-23T10:41:09.935 INFO:teuthology.orchestra.run.smithi114.stdout: "default_value": "", 2024-04-23T10:41:09.935 INFO:teuthology.orchestra.run.smithi114.stdout: "min": "", 2024-04-23T10:41:09.935 INFO:teuthology.orchestra.run.smithi114.stdout: "max": "", 2024-04-23T10:41:09.935 INFO:teuthology.orchestra.run.smithi114.stdout: "enum_allowed": [ 2024-04-23T10:41:09.936 INFO:teuthology.orchestra.run.smithi114.stdout: "", 2024-04-23T10:41:09.936 INFO:teuthology.orchestra.run.smithi114.stdout: "critical", 2024-04-23T10:41:09.936 INFO:teuthology.orchestra.run.smithi114.stdout: "debug", 2024-04-23T10:41:09.936 INFO:teuthology.orchestra.run.smithi114.stdout: "error", 2024-04-23T10:41:09.936 INFO:teuthology.orchestra.run.smithi114.stdout: "info", 2024-04-23T10:41:09.936 INFO:teuthology.orchestra.run.smithi114.stdout: "warning" 2024-04-23T10:41:09.936 INFO:teuthology.orchestra.run.smithi114.stdout: ], 2024-04-23T10:41:09.936 INFO:teuthology.orchestra.run.smithi114.stdout: "desc": "", 2024-04-23T10:41:09.936 INFO:teuthology.orchestra.run.smithi114.stdout: "long_desc": "", 2024-04-23T10:41:09.936 INFO:teuthology.orchestra.run.smithi114.stdout: "tags": [], 2024-04-23T10:41:09.937 INFO:teuthology.orchestra.run.smithi114.stdout: "see_also": [] 2024-04-23T10:41:09.937 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-23T10:41:09.937 INFO:teuthology.orchestra.run.smithi114.stdout: "log_to_cluster": { 2024-04-23T10:41:09.937 INFO:teuthology.orchestra.run.smithi114.stdout: "name": "log_to_cluster", 2024-04-23T10:41:09.937 INFO:teuthology.orchestra.run.smithi114.stdout: "type": "bool", 2024-04-23T10:41:09.937 INFO:teuthology.orchestra.run.smithi114.stdout: "level": "advanced", 2024-04-23T10:41:09.937 INFO:teuthology.orchestra.run.smithi114.stdout: "flags": 1, 2024-04-23T10:41:09.937 INFO:teuthology.orchestra.run.smithi114.stdout: "default_value": "False", 2024-04-23T10:41:09.937 INFO:teuthology.orchestra.run.smithi114.stdout: "min": "", 2024-04-23T10:41:09.937 INFO:teuthology.orchestra.run.smithi114.stdout: "max": "", 2024-04-23T10:41:09.938 INFO:teuthology.orchestra.run.smithi114.stdout: "enum_allowed": [], 2024-04-23T10:41:09.938 INFO:teuthology.orchestra.run.smithi114.stdout: "desc": "", 2024-04-23T10:41:09.938 INFO:teuthology.orchestra.run.smithi114.stdout: "long_desc": "", 2024-04-23T10:41:09.938 INFO:teuthology.orchestra.run.smithi114.stdout: "tags": [], 2024-04-23T10:41:09.938 INFO:teuthology.orchestra.run.smithi114.stdout: "see_also": [] 2024-04-23T10:41:09.938 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-23T10:41:09.938 INFO:teuthology.orchestra.run.smithi114.stdout: "log_to_cluster_level": { 2024-04-23T10:41:09.938 INFO:teuthology.orchestra.run.smithi114.stdout: "name": "log_to_cluster_level", 2024-04-23T10:41:09.938 INFO:teuthology.orchestra.run.smithi114.stdout: "type": "str", 2024-04-23T10:41:09.939 INFO:teuthology.orchestra.run.smithi114.stdout: "level": "advanced", 2024-04-23T10:41:09.939 INFO:teuthology.orchestra.run.smithi114.stdout: "flags": 1, 2024-04-23T10:41:09.939 INFO:teuthology.orchestra.run.smithi114.stdout: "default_value": "info", 2024-04-23T10:41:09.939 INFO:teuthology.orchestra.run.smithi114.stdout: "min": "", 2024-04-23T10:41:09.939 INFO:teuthology.orchestra.run.smithi114.stdout: "max": "", 2024-04-23T10:41:09.939 INFO:teuthology.orchestra.run.smithi114.stdout: "enum_allowed": [ 2024-04-23T10:41:09.939 INFO:teuthology.orchestra.run.smithi114.stdout: "", 2024-04-23T10:41:09.939 INFO:teuthology.orchestra.run.smithi114.stdout: "critical", 2024-04-23T10:41:09.939 INFO:teuthology.orchestra.run.smithi114.stdout: "debug", 2024-04-23T10:41:09.939 INFO:teuthology.orchestra.run.smithi114.stdout: "error", 2024-04-23T10:41:09.940 INFO:teuthology.orchestra.run.smithi114.stdout: "info", 2024-04-23T10:41:09.940 INFO:teuthology.orchestra.run.smithi114.stdout: "warning" 2024-04-23T10:41:09.940 INFO:teuthology.orchestra.run.smithi114.stdout: ], 2024-04-23T10:41:09.940 INFO:teuthology.orchestra.run.smithi114.stdout: "desc": "", 2024-04-23T10:41:09.940 INFO:teuthology.orchestra.run.smithi114.stdout: "long_desc": "", 2024-04-23T10:41:09.940 INFO:teuthology.orchestra.run.smithi114.stdout: "tags": [], 2024-04-23T10:41:09.940 INFO:teuthology.orchestra.run.smithi114.stdout: "see_also": [] 2024-04-23T10:41:09.940 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-23T10:41:09.940 INFO:teuthology.orchestra.run.smithi114.stdout: "log_to_file": { 2024-04-23T10:41:09.940 INFO:teuthology.orchestra.run.smithi114.stdout: "name": "log_to_file", 2024-04-23T10:41:09.941 INFO:teuthology.orchestra.run.smithi114.stdout: "type": "bool", 2024-04-23T10:41:09.941 INFO:teuthology.orchestra.run.smithi114.stdout: "level": "advanced", 2024-04-23T10:41:09.941 INFO:teuthology.orchestra.run.smithi114.stdout: "flags": 1, 2024-04-23T10:41:09.941 INFO:teuthology.orchestra.run.smithi114.stdout: "default_value": "False", 2024-04-23T10:41:09.941 INFO:teuthology.orchestra.run.smithi114.stdout: "min": "", 2024-04-23T10:41:09.941 INFO:teuthology.orchestra.run.smithi114.stdout: "max": "", 2024-04-23T10:41:09.941 INFO:teuthology.orchestra.run.smithi114.stdout: "enum_allowed": [], 2024-04-23T10:41:09.941 INFO:teuthology.orchestra.run.smithi114.stdout: "desc": "", 2024-04-23T10:41:09.941 INFO:teuthology.orchestra.run.smithi114.stdout: "long_desc": "", 2024-04-23T10:41:09.942 INFO:teuthology.orchestra.run.smithi114.stdout: "tags": [], 2024-04-23T10:41:09.942 INFO:teuthology.orchestra.run.smithi114.stdout: "see_also": [] 2024-04-23T10:41:09.942 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-23T10:41:09.942 INFO:teuthology.orchestra.run.smithi114.stdout: "storage_class": { 2024-04-23T10:41:09.942 INFO:teuthology.orchestra.run.smithi114.stdout: "name": "storage_class", 2024-04-23T10:41:09.942 INFO:teuthology.orchestra.run.smithi114.stdout: "type": "str", 2024-04-23T10:41:09.942 INFO:teuthology.orchestra.run.smithi114.stdout: "level": "advanced", 2024-04-23T10:41:09.942 INFO:teuthology.orchestra.run.smithi114.stdout: "flags": 0, 2024-04-23T10:41:09.942 INFO:teuthology.orchestra.run.smithi114.stdout: "default_value": "local", 2024-04-23T10:41:09.942 INFO:teuthology.orchestra.run.smithi114.stdout: "min": "", 2024-04-23T10:41:09.943 INFO:teuthology.orchestra.run.smithi114.stdout: "max": "", 2024-04-23T10:41:09.943 INFO:teuthology.orchestra.run.smithi114.stdout: "enum_allowed": [], 2024-04-23T10:41:09.943 INFO:teuthology.orchestra.run.smithi114.stdout: "desc": "storage class name for LSO-discovered PVs", 2024-04-23T10:41:09.943 INFO:teuthology.orchestra.run.smithi114.stdout: "long_desc": "", 2024-04-23T10:41:09.943 INFO:teuthology.orchestra.run.smithi114.stdout: "tags": [], 2024-04-23T10:41:09.943 INFO:teuthology.orchestra.run.smithi114.stdout: "see_also": [] 2024-04-23T10:41:09.943 INFO:teuthology.orchestra.run.smithi114.stdout: } 2024-04-23T10:41:09.943 INFO:teuthology.orchestra.run.smithi114.stdout: } 2024-04-23T10:41:09.943 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-23T10:41:09.943 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-04-23T10:41:09.944 INFO:teuthology.orchestra.run.smithi114.stdout: "name": "selftest", 2024-04-23T10:41:09.944 INFO:teuthology.orchestra.run.smithi114.stdout: "can_run": true, 2024-04-23T10:41:09.944 INFO:teuthology.orchestra.run.smithi114.stdout: "error_string": "", 2024-04-23T10:41:09.944 INFO:teuthology.orchestra.run.smithi114.stdout: "module_options": { 2024-04-23T10:41:09.944 INFO:teuthology.orchestra.run.smithi114.stdout: "log_level": { 2024-04-23T10:41:09.944 INFO:teuthology.orchestra.run.smithi114.stdout: "name": "log_level", 2024-04-23T10:41:09.944 INFO:teuthology.orchestra.run.smithi114.stdout: "type": "str", 2024-04-23T10:41:09.944 INFO:teuthology.orchestra.run.smithi114.stdout: "level": "advanced", 2024-04-23T10:41:09.944 INFO:teuthology.orchestra.run.smithi114.stdout: "flags": 1, 2024-04-23T10:41:09.944 INFO:teuthology.orchestra.run.smithi114.stdout: "default_value": "", 2024-04-23T10:41:09.945 INFO:teuthology.orchestra.run.smithi114.stdout: "min": "", 2024-04-23T10:41:09.945 INFO:teuthology.orchestra.run.smithi114.stdout: "max": "", 2024-04-23T10:41:09.945 INFO:teuthology.orchestra.run.smithi114.stdout: "enum_allowed": [ 2024-04-23T10:41:09.945 INFO:teuthology.orchestra.run.smithi114.stdout: "", 2024-04-23T10:41:09.945 INFO:teuthology.orchestra.run.smithi114.stdout: "critical", 2024-04-23T10:41:09.945 INFO:teuthology.orchestra.run.smithi114.stdout: "debug", 2024-04-23T10:41:09.945 INFO:teuthology.orchestra.run.smithi114.stdout: "error", 2024-04-23T10:41:09.945 INFO:teuthology.orchestra.run.smithi114.stdout: "info", 2024-04-23T10:41:09.945 INFO:teuthology.orchestra.run.smithi114.stdout: "warning" 2024-04-23T10:41:09.946 INFO:teuthology.orchestra.run.smithi114.stdout: ], 2024-04-23T10:41:09.946 INFO:teuthology.orchestra.run.smithi114.stdout: "desc": "", 2024-04-23T10:41:09.946 INFO:teuthology.orchestra.run.smithi114.stdout: "long_desc": "", 2024-04-23T10:41:09.946 INFO:teuthology.orchestra.run.smithi114.stdout: "tags": [], 2024-04-23T10:41:09.946 INFO:teuthology.orchestra.run.smithi114.stdout: "see_also": [] 2024-04-23T10:41:09.946 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-23T10:41:09.946 INFO:teuthology.orchestra.run.smithi114.stdout: "log_to_cluster": { 2024-04-23T10:41:09.946 INFO:teuthology.orchestra.run.smithi114.stdout: "name": "log_to_cluster", 2024-04-23T10:41:09.946 INFO:teuthology.orchestra.run.smithi114.stdout: "type": "bool", 2024-04-23T10:41:09.947 INFO:teuthology.orchestra.run.smithi114.stdout: "level": "advanced", 2024-04-23T10:41:09.947 INFO:teuthology.orchestra.run.smithi114.stdout: "flags": 1, 2024-04-23T10:41:09.947 INFO:teuthology.orchestra.run.smithi114.stdout: "default_value": "False", 2024-04-23T10:41:09.947 INFO:teuthology.orchestra.run.smithi114.stdout: "min": "", 2024-04-23T10:41:09.947 INFO:teuthology.orchestra.run.smithi114.stdout: "max": "", 2024-04-23T10:41:09.947 INFO:teuthology.orchestra.run.smithi114.stdout: "enum_allowed": [], 2024-04-23T10:41:09.947 INFO:teuthology.orchestra.run.smithi114.stdout: "desc": "", 2024-04-23T10:41:09.947 INFO:teuthology.orchestra.run.smithi114.stdout: "long_desc": "", 2024-04-23T10:41:09.947 INFO:teuthology.orchestra.run.smithi114.stdout: "tags": [], 2024-04-23T10:41:09.947 INFO:teuthology.orchestra.run.smithi114.stdout: "see_also": [] 2024-04-23T10:41:09.948 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-23T10:41:09.948 INFO:teuthology.orchestra.run.smithi114.stdout: "log_to_cluster_level": { 2024-04-23T10:41:09.948 INFO:teuthology.orchestra.run.smithi114.stdout: "name": "log_to_cluster_level", 2024-04-23T10:41:09.948 INFO:teuthology.orchestra.run.smithi114.stdout: "type": "str", 2024-04-23T10:41:09.948 INFO:teuthology.orchestra.run.smithi114.stdout: "level": "advanced", 2024-04-23T10:41:09.948 INFO:teuthology.orchestra.run.smithi114.stdout: "flags": 1, 2024-04-23T10:41:09.948 INFO:teuthology.orchestra.run.smithi114.stdout: "default_value": "info", 2024-04-23T10:41:09.948 INFO:teuthology.orchestra.run.smithi114.stdout: "min": "", 2024-04-23T10:41:09.948 INFO:teuthology.orchestra.run.smithi114.stdout: "max": "", 2024-04-23T10:41:09.948 INFO:teuthology.orchestra.run.smithi114.stdout: "enum_allowed": [ 2024-04-23T10:41:09.949 INFO:teuthology.orchestra.run.smithi114.stdout: "", 2024-04-23T10:41:09.949 INFO:teuthology.orchestra.run.smithi114.stdout: "critical", 2024-04-23T10:41:09.949 INFO:teuthology.orchestra.run.smithi114.stdout: "debug", 2024-04-23T10:41:09.949 INFO:teuthology.orchestra.run.smithi114.stdout: "error", 2024-04-23T10:41:09.949 INFO:teuthology.orchestra.run.smithi114.stdout: "info", 2024-04-23T10:41:09.949 INFO:teuthology.orchestra.run.smithi114.stdout: "warning" 2024-04-23T10:41:09.949 INFO:teuthology.orchestra.run.smithi114.stdout: ], 2024-04-23T10:41:09.949 INFO:teuthology.orchestra.run.smithi114.stdout: "desc": "", 2024-04-23T10:41:09.949 INFO:teuthology.orchestra.run.smithi114.stdout: "long_desc": "", 2024-04-23T10:41:09.950 INFO:teuthology.orchestra.run.smithi114.stdout: "tags": [], 2024-04-23T10:41:09.950 INFO:teuthology.orchestra.run.smithi114.stdout: "see_also": [] 2024-04-23T10:41:09.950 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-23T10:41:09.950 INFO:teuthology.orchestra.run.smithi114.stdout: "log_to_file": { 2024-04-23T10:41:09.950 INFO:teuthology.orchestra.run.smithi114.stdout: "name": "log_to_file", 2024-04-23T10:41:09.950 INFO:teuthology.orchestra.run.smithi114.stdout: "type": "bool", 2024-04-23T10:41:09.950 INFO:teuthology.orchestra.run.smithi114.stdout: "level": "advanced", 2024-04-23T10:41:09.950 INFO:teuthology.orchestra.run.smithi114.stdout: "flags": 1, 2024-04-23T10:41:09.950 INFO:teuthology.orchestra.run.smithi114.stdout: "default_value": "False", 2024-04-23T10:41:09.951 INFO:teuthology.orchestra.run.smithi114.stdout: "min": "", 2024-04-23T10:41:09.951 INFO:teuthology.orchestra.run.smithi114.stdout: "max": "", 2024-04-23T10:41:09.951 INFO:teuthology.orchestra.run.smithi114.stdout: "enum_allowed": [], 2024-04-23T10:41:09.951 INFO:teuthology.orchestra.run.smithi114.stdout: "desc": "", 2024-04-23T10:41:09.951 INFO:teuthology.orchestra.run.smithi114.stdout: "long_desc": "", 2024-04-23T10:41:09.951 INFO:teuthology.orchestra.run.smithi114.stdout: "tags": [], 2024-04-23T10:41:09.951 INFO:teuthology.orchestra.run.smithi114.stdout: "see_also": [] 2024-04-23T10:41:09.951 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-23T10:41:09.951 INFO:teuthology.orchestra.run.smithi114.stdout: "roption1": { 2024-04-23T10:41:09.952 INFO:teuthology.orchestra.run.smithi114.stdout: "name": "roption1", 2024-04-23T10:41:09.952 INFO:teuthology.orchestra.run.smithi114.stdout: "type": "str", 2024-04-23T10:41:09.952 INFO:teuthology.orchestra.run.smithi114.stdout: "level": "advanced", 2024-04-23T10:41:09.952 INFO:teuthology.orchestra.run.smithi114.stdout: "flags": 0, 2024-04-23T10:41:09.952 INFO:teuthology.orchestra.run.smithi114.stdout: "default_value": "", 2024-04-23T10:41:09.952 INFO:teuthology.orchestra.run.smithi114.stdout: "min": "", 2024-04-23T10:41:09.952 INFO:teuthology.orchestra.run.smithi114.stdout: "max": "", 2024-04-23T10:41:09.952 INFO:teuthology.orchestra.run.smithi114.stdout: "enum_allowed": [], 2024-04-23T10:41:09.952 INFO:teuthology.orchestra.run.smithi114.stdout: "desc": "", 2024-04-23T10:41:09.952 INFO:teuthology.orchestra.run.smithi114.stdout: "long_desc": "", 2024-04-23T10:41:09.953 INFO:teuthology.orchestra.run.smithi114.stdout: "tags": [], 2024-04-23T10:41:09.953 INFO:teuthology.orchestra.run.smithi114.stdout: "see_also": [] 2024-04-23T10:41:09.953 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-23T10:41:09.953 INFO:teuthology.orchestra.run.smithi114.stdout: "roption2": { 2024-04-23T10:41:09.953 INFO:teuthology.orchestra.run.smithi114.stdout: "name": "roption2", 2024-04-23T10:41:09.953 INFO:teuthology.orchestra.run.smithi114.stdout: "type": "str", 2024-04-23T10:41:09.953 INFO:teuthology.orchestra.run.smithi114.stdout: "level": "advanced", 2024-04-23T10:41:09.953 INFO:teuthology.orchestra.run.smithi114.stdout: "flags": 0, 2024-04-23T10:41:09.953 INFO:teuthology.orchestra.run.smithi114.stdout: "default_value": "xyz", 2024-04-23T10:41:09.953 INFO:teuthology.orchestra.run.smithi114.stdout: "min": "", 2024-04-23T10:41:09.954 INFO:teuthology.orchestra.run.smithi114.stdout: "max": "", 2024-04-23T10:41:09.954 INFO:teuthology.orchestra.run.smithi114.stdout: "enum_allowed": [], 2024-04-23T10:41:09.954 INFO:teuthology.orchestra.run.smithi114.stdout: "desc": "", 2024-04-23T10:41:09.954 INFO:teuthology.orchestra.run.smithi114.stdout: "long_desc": "", 2024-04-23T10:41:09.954 INFO:teuthology.orchestra.run.smithi114.stdout: "tags": [], 2024-04-23T10:41:09.954 INFO:teuthology.orchestra.run.smithi114.stdout: "see_also": [] 2024-04-23T10:41:09.954 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-23T10:41:09.954 INFO:teuthology.orchestra.run.smithi114.stdout: "rwoption1": { 2024-04-23T10:41:09.954 INFO:teuthology.orchestra.run.smithi114.stdout: "name": "rwoption1", 2024-04-23T10:41:09.955 INFO:teuthology.orchestra.run.smithi114.stdout: "type": "str", 2024-04-23T10:41:09.955 INFO:teuthology.orchestra.run.smithi114.stdout: "level": "advanced", 2024-04-23T10:41:09.955 INFO:teuthology.orchestra.run.smithi114.stdout: "flags": 0, 2024-04-23T10:41:09.955 INFO:teuthology.orchestra.run.smithi114.stdout: "default_value": "", 2024-04-23T10:41:09.955 INFO:teuthology.orchestra.run.smithi114.stdout: "min": "", 2024-04-23T10:41:09.955 INFO:teuthology.orchestra.run.smithi114.stdout: "max": "", 2024-04-23T10:41:09.955 INFO:teuthology.orchestra.run.smithi114.stdout: "enum_allowed": [], 2024-04-23T10:41:09.955 INFO:teuthology.orchestra.run.smithi114.stdout: "desc": "", 2024-04-23T10:41:09.955 INFO:teuthology.orchestra.run.smithi114.stdout: "long_desc": "", 2024-04-23T10:41:09.955 INFO:teuthology.orchestra.run.smithi114.stdout: "tags": [], 2024-04-23T10:41:09.956 INFO:teuthology.orchestra.run.smithi114.stdout: "see_also": [] 2024-04-23T10:41:09.956 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-23T10:41:09.956 INFO:teuthology.orchestra.run.smithi114.stdout: "rwoption2": { 2024-04-23T10:41:09.956 INFO:teuthology.orchestra.run.smithi114.stdout: "name": "rwoption2", 2024-04-23T10:41:09.956 INFO:teuthology.orchestra.run.smithi114.stdout: "type": "int", 2024-04-23T10:41:09.956 INFO:teuthology.orchestra.run.smithi114.stdout: "level": "advanced", 2024-04-23T10:41:09.956 INFO:teuthology.orchestra.run.smithi114.stdout: "flags": 0, 2024-04-23T10:41:09.956 INFO:teuthology.orchestra.run.smithi114.stdout: "default_value": "", 2024-04-23T10:41:09.956 INFO:teuthology.orchestra.run.smithi114.stdout: "min": "", 2024-04-23T10:41:09.956 INFO:teuthology.orchestra.run.smithi114.stdout: "max": "", 2024-04-23T10:41:09.957 INFO:teuthology.orchestra.run.smithi114.stdout: "enum_allowed": [], 2024-04-23T10:41:09.957 INFO:teuthology.orchestra.run.smithi114.stdout: "desc": "", 2024-04-23T10:41:09.957 INFO:teuthology.orchestra.run.smithi114.stdout: "long_desc": "", 2024-04-23T10:41:09.957 INFO:teuthology.orchestra.run.smithi114.stdout: "tags": [], 2024-04-23T10:41:09.957 INFO:teuthology.orchestra.run.smithi114.stdout: "see_also": [] 2024-04-23T10:41:09.957 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-23T10:41:09.957 INFO:teuthology.orchestra.run.smithi114.stdout: "rwoption3": { 2024-04-23T10:41:09.957 INFO:teuthology.orchestra.run.smithi114.stdout: "name": "rwoption3", 2024-04-23T10:41:09.957 INFO:teuthology.orchestra.run.smithi114.stdout: "type": "float", 2024-04-23T10:41:09.957 INFO:teuthology.orchestra.run.smithi114.stdout: "level": "advanced", 2024-04-23T10:41:09.958 INFO:teuthology.orchestra.run.smithi114.stdout: "flags": 0, 2024-04-23T10:41:09.958 INFO:teuthology.orchestra.run.smithi114.stdout: "default_value": "", 2024-04-23T10:41:09.958 INFO:teuthology.orchestra.run.smithi114.stdout: "min": "", 2024-04-23T10:41:09.958 INFO:teuthology.orchestra.run.smithi114.stdout: "max": "", 2024-04-23T10:41:09.958 INFO:teuthology.orchestra.run.smithi114.stdout: "enum_allowed": [], 2024-04-23T10:41:09.958 INFO:teuthology.orchestra.run.smithi114.stdout: "desc": "", 2024-04-23T10:41:09.958 INFO:teuthology.orchestra.run.smithi114.stdout: "long_desc": "", 2024-04-23T10:41:09.958 INFO:teuthology.orchestra.run.smithi114.stdout: "tags": [], 2024-04-23T10:41:09.958 INFO:teuthology.orchestra.run.smithi114.stdout: "see_also": [] 2024-04-23T10:41:09.959 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-23T10:41:09.959 INFO:teuthology.orchestra.run.smithi114.stdout: "rwoption4": { 2024-04-23T10:41:09.959 INFO:teuthology.orchestra.run.smithi114.stdout: "name": "rwoption4", 2024-04-23T10:41:09.959 INFO:teuthology.orchestra.run.smithi114.stdout: "type": "str", 2024-04-23T10:41:09.959 INFO:teuthology.orchestra.run.smithi114.stdout: "level": "advanced", 2024-04-23T10:41:09.959 INFO:teuthology.orchestra.run.smithi114.stdout: "flags": 0, 2024-04-23T10:41:09.959 INFO:teuthology.orchestra.run.smithi114.stdout: "default_value": "", 2024-04-23T10:41:09.959 INFO:teuthology.orchestra.run.smithi114.stdout: "min": "", 2024-04-23T10:41:09.959 INFO:teuthology.orchestra.run.smithi114.stdout: "max": "", 2024-04-23T10:41:09.959 INFO:teuthology.orchestra.run.smithi114.stdout: "enum_allowed": [], 2024-04-23T10:41:09.960 INFO:teuthology.orchestra.run.smithi114.stdout: "desc": "", 2024-04-23T10:41:09.960 INFO:teuthology.orchestra.run.smithi114.stdout: "long_desc": "", 2024-04-23T10:41:09.960 INFO:teuthology.orchestra.run.smithi114.stdout: "tags": [], 2024-04-23T10:41:09.960 INFO:teuthology.orchestra.run.smithi114.stdout: "see_also": [] 2024-04-23T10:41:09.960 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-23T10:41:09.960 INFO:teuthology.orchestra.run.smithi114.stdout: "rwoption5": { 2024-04-23T10:41:09.960 INFO:teuthology.orchestra.run.smithi114.stdout: "name": "rwoption5", 2024-04-23T10:41:09.960 INFO:teuthology.orchestra.run.smithi114.stdout: "type": "bool", 2024-04-23T10:41:09.960 INFO:teuthology.orchestra.run.smithi114.stdout: "level": "advanced", 2024-04-23T10:41:09.960 INFO:teuthology.orchestra.run.smithi114.stdout: "flags": 0, 2024-04-23T10:41:09.961 INFO:teuthology.orchestra.run.smithi114.stdout: "default_value": "", 2024-04-23T10:41:09.961 INFO:teuthology.orchestra.run.smithi114.stdout: "min": "", 2024-04-23T10:41:09.961 INFO:teuthology.orchestra.run.smithi114.stdout: "max": "", 2024-04-23T10:41:09.961 INFO:teuthology.orchestra.run.smithi114.stdout: "enum_allowed": [], 2024-04-23T10:41:09.961 INFO:teuthology.orchestra.run.smithi114.stdout: "desc": "", 2024-04-23T10:41:09.961 INFO:teuthology.orchestra.run.smithi114.stdout: "long_desc": "", 2024-04-23T10:41:09.961 INFO:teuthology.orchestra.run.smithi114.stdout: "tags": [], 2024-04-23T10:41:09.961 INFO:teuthology.orchestra.run.smithi114.stdout: "see_also": [] 2024-04-23T10:41:09.961 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-23T10:41:09.961 INFO:teuthology.orchestra.run.smithi114.stdout: "rwoption6": { 2024-04-23T10:41:09.962 INFO:teuthology.orchestra.run.smithi114.stdout: "name": "rwoption6", 2024-04-23T10:41:09.962 INFO:teuthology.orchestra.run.smithi114.stdout: "type": "bool", 2024-04-23T10:41:09.962 INFO:teuthology.orchestra.run.smithi114.stdout: "level": "advanced", 2024-04-23T10:41:09.962 INFO:teuthology.orchestra.run.smithi114.stdout: "flags": 0, 2024-04-23T10:41:09.962 INFO:teuthology.orchestra.run.smithi114.stdout: "default_value": "True", 2024-04-23T10:41:09.962 INFO:teuthology.orchestra.run.smithi114.stdout: "min": "", 2024-04-23T10:41:09.962 INFO:teuthology.orchestra.run.smithi114.stdout: "max": "", 2024-04-23T10:41:09.962 INFO:teuthology.orchestra.run.smithi114.stdout: "enum_allowed": [], 2024-04-23T10:41:09.962 INFO:teuthology.orchestra.run.smithi114.stdout: "desc": "", 2024-04-23T10:41:09.962 INFO:teuthology.orchestra.run.smithi114.stdout: "long_desc": "", 2024-04-23T10:41:09.963 INFO:teuthology.orchestra.run.smithi114.stdout: "tags": [], 2024-04-23T10:41:09.963 INFO:teuthology.orchestra.run.smithi114.stdout: "see_also": [] 2024-04-23T10:41:09.963 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-23T10:41:09.963 INFO:teuthology.orchestra.run.smithi114.stdout: "rwoption7": { 2024-04-23T10:41:09.963 INFO:teuthology.orchestra.run.smithi114.stdout: "name": "rwoption7", 2024-04-23T10:41:09.963 INFO:teuthology.orchestra.run.smithi114.stdout: "type": "int", 2024-04-23T10:41:09.963 INFO:teuthology.orchestra.run.smithi114.stdout: "level": "advanced", 2024-04-23T10:41:09.963 INFO:teuthology.orchestra.run.smithi114.stdout: "flags": 0, 2024-04-23T10:41:09.963 INFO:teuthology.orchestra.run.smithi114.stdout: "default_value": "", 2024-04-23T10:41:09.963 INFO:teuthology.orchestra.run.smithi114.stdout: "min": "1", 2024-04-23T10:41:09.964 INFO:teuthology.orchestra.run.smithi114.stdout: "max": "42", 2024-04-23T10:41:09.964 INFO:teuthology.orchestra.run.smithi114.stdout: "enum_allowed": [], 2024-04-23T10:41:09.964 INFO:teuthology.orchestra.run.smithi114.stdout: "desc": "", 2024-04-23T10:41:09.964 INFO:teuthology.orchestra.run.smithi114.stdout: "long_desc": "", 2024-04-23T10:41:09.964 INFO:teuthology.orchestra.run.smithi114.stdout: "tags": [], 2024-04-23T10:41:09.964 INFO:teuthology.orchestra.run.smithi114.stdout: "see_also": [] 2024-04-23T10:41:09.964 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-23T10:41:09.964 INFO:teuthology.orchestra.run.smithi114.stdout: "testkey": { 2024-04-23T10:41:09.964 INFO:teuthology.orchestra.run.smithi114.stdout: "name": "testkey", 2024-04-23T10:41:09.964 INFO:teuthology.orchestra.run.smithi114.stdout: "type": "str", 2024-04-23T10:41:09.965 INFO:teuthology.orchestra.run.smithi114.stdout: "level": "advanced", 2024-04-23T10:41:09.965 INFO:teuthology.orchestra.run.smithi114.stdout: "flags": 0, 2024-04-23T10:41:09.965 INFO:teuthology.orchestra.run.smithi114.stdout: "default_value": "", 2024-04-23T10:41:09.965 INFO:teuthology.orchestra.run.smithi114.stdout: "min": "", 2024-04-23T10:41:09.965 INFO:teuthology.orchestra.run.smithi114.stdout: "max": "", 2024-04-23T10:41:09.965 INFO:teuthology.orchestra.run.smithi114.stdout: "enum_allowed": [], 2024-04-23T10:41:09.965 INFO:teuthology.orchestra.run.smithi114.stdout: "desc": "", 2024-04-23T10:41:09.965 INFO:teuthology.orchestra.run.smithi114.stdout: "long_desc": "", 2024-04-23T10:41:09.965 INFO:teuthology.orchestra.run.smithi114.stdout: "tags": [], 2024-04-23T10:41:09.965 INFO:teuthology.orchestra.run.smithi114.stdout: "see_also": [] 2024-04-23T10:41:09.966 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-23T10:41:09.966 INFO:teuthology.orchestra.run.smithi114.stdout: "testlkey": { 2024-04-23T10:41:09.966 INFO:teuthology.orchestra.run.smithi114.stdout: "name": "testlkey", 2024-04-23T10:41:09.966 INFO:teuthology.orchestra.run.smithi114.stdout: "type": "str", 2024-04-23T10:41:09.966 INFO:teuthology.orchestra.run.smithi114.stdout: "level": "advanced", 2024-04-23T10:41:09.966 INFO:teuthology.orchestra.run.smithi114.stdout: "flags": 0, 2024-04-23T10:41:09.966 INFO:teuthology.orchestra.run.smithi114.stdout: "default_value": "", 2024-04-23T10:41:09.966 INFO:teuthology.orchestra.run.smithi114.stdout: "min": "", 2024-04-23T10:41:09.966 INFO:teuthology.orchestra.run.smithi114.stdout: "max": "", 2024-04-23T10:41:09.967 INFO:teuthology.orchestra.run.smithi114.stdout: "enum_allowed": [], 2024-04-23T10:41:09.967 INFO:teuthology.orchestra.run.smithi114.stdout: "desc": "", 2024-04-23T10:41:09.967 INFO:teuthology.orchestra.run.smithi114.stdout: "long_desc": "", 2024-04-23T10:41:09.967 INFO:teuthology.orchestra.run.smithi114.stdout: "tags": [], 2024-04-23T10:41:09.967 INFO:teuthology.orchestra.run.smithi114.stdout: "see_also": [] 2024-04-23T10:41:09.967 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-23T10:41:09.967 INFO:teuthology.orchestra.run.smithi114.stdout: "testnewline": { 2024-04-23T10:41:09.967 INFO:teuthology.orchestra.run.smithi114.stdout: "name": "testnewline", 2024-04-23T10:41:09.967 INFO:teuthology.orchestra.run.smithi114.stdout: "type": "str", 2024-04-23T10:41:09.967 INFO:teuthology.orchestra.run.smithi114.stdout: "level": "advanced", 2024-04-23T10:41:09.968 INFO:teuthology.orchestra.run.smithi114.stdout: "flags": 0, 2024-04-23T10:41:09.968 INFO:teuthology.orchestra.run.smithi114.stdout: "default_value": "", 2024-04-23T10:41:09.968 INFO:teuthology.orchestra.run.smithi114.stdout: "min": "", 2024-04-23T10:41:09.968 INFO:teuthology.orchestra.run.smithi114.stdout: "max": "", 2024-04-23T10:41:09.968 INFO:teuthology.orchestra.run.smithi114.stdout: "enum_allowed": [], 2024-04-23T10:41:09.968 INFO:teuthology.orchestra.run.smithi114.stdout: "desc": "", 2024-04-23T10:41:09.968 INFO:teuthology.orchestra.run.smithi114.stdout: "long_desc": "", 2024-04-23T10:41:09.968 INFO:teuthology.orchestra.run.smithi114.stdout: "tags": [], 2024-04-23T10:41:09.968 INFO:teuthology.orchestra.run.smithi114.stdout: "see_also": [] 2024-04-23T10:41:09.968 INFO:teuthology.orchestra.run.smithi114.stdout: } 2024-04-23T10:41:09.969 INFO:teuthology.orchestra.run.smithi114.stdout: } 2024-04-23T10:41:09.969 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-23T10:41:09.969 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-04-23T10:41:09.969 INFO:teuthology.orchestra.run.smithi114.stdout: "name": "snap_schedule", 2024-04-23T10:41:09.969 INFO:teuthology.orchestra.run.smithi114.stdout: "can_run": true, 2024-04-23T10:41:09.969 INFO:teuthology.orchestra.run.smithi114.stdout: "error_string": "", 2024-04-23T10:41:09.969 INFO:teuthology.orchestra.run.smithi114.stdout: "module_options": { 2024-04-23T10:41:09.969 INFO:teuthology.orchestra.run.smithi114.stdout: "allow_m_granularity": { 2024-04-23T10:41:09.969 INFO:teuthology.orchestra.run.smithi114.stdout: "name": "allow_m_granularity", 2024-04-23T10:41:09.969 INFO:teuthology.orchestra.run.smithi114.stdout: "type": "bool", 2024-04-23T10:41:09.970 INFO:teuthology.orchestra.run.smithi114.stdout: "level": "advanced", 2024-04-23T10:41:09.970 INFO:teuthology.orchestra.run.smithi114.stdout: "flags": 1, 2024-04-23T10:41:09.970 INFO:teuthology.orchestra.run.smithi114.stdout: "default_value": "False", 2024-04-23T10:41:09.970 INFO:teuthology.orchestra.run.smithi114.stdout: "min": "", 2024-04-23T10:41:09.970 INFO:teuthology.orchestra.run.smithi114.stdout: "max": "", 2024-04-23T10:41:09.970 INFO:teuthology.orchestra.run.smithi114.stdout: "enum_allowed": [], 2024-04-23T10:41:09.970 INFO:teuthology.orchestra.run.smithi114.stdout: "desc": "allow minute scheduled snapshots", 2024-04-23T10:41:09.970 INFO:teuthology.orchestra.run.smithi114.stdout: "long_desc": "", 2024-04-23T10:41:09.970 INFO:teuthology.orchestra.run.smithi114.stdout: "tags": [], 2024-04-23T10:41:09.971 INFO:teuthology.orchestra.run.smithi114.stdout: "see_also": [] 2024-04-23T10:41:09.971 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-23T10:41:09.971 INFO:teuthology.orchestra.run.smithi114.stdout: "dump_on_update": { 2024-04-23T10:41:09.971 INFO:teuthology.orchestra.run.smithi114.stdout: "name": "dump_on_update", 2024-04-23T10:41:09.971 INFO:teuthology.orchestra.run.smithi114.stdout: "type": "bool", 2024-04-23T10:41:09.971 INFO:teuthology.orchestra.run.smithi114.stdout: "level": "advanced", 2024-04-23T10:41:09.971 INFO:teuthology.orchestra.run.smithi114.stdout: "flags": 1, 2024-04-23T10:41:09.971 INFO:teuthology.orchestra.run.smithi114.stdout: "default_value": "False", 2024-04-23T10:41:09.971 INFO:teuthology.orchestra.run.smithi114.stdout: "min": "", 2024-04-23T10:41:09.971 INFO:teuthology.orchestra.run.smithi114.stdout: "max": "", 2024-04-23T10:41:09.972 INFO:teuthology.orchestra.run.smithi114.stdout: "enum_allowed": [], 2024-04-23T10:41:09.972 INFO:teuthology.orchestra.run.smithi114.stdout: "desc": "dump database to debug log on update", 2024-04-23T10:41:09.972 INFO:teuthology.orchestra.run.smithi114.stdout: "long_desc": "", 2024-04-23T10:41:09.972 INFO:teuthology.orchestra.run.smithi114.stdout: "tags": [], 2024-04-23T10:41:09.972 INFO:teuthology.orchestra.run.smithi114.stdout: "see_also": [] 2024-04-23T10:41:09.972 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-23T10:41:09.972 INFO:teuthology.orchestra.run.smithi114.stdout: "log_level": { 2024-04-23T10:41:09.972 INFO:teuthology.orchestra.run.smithi114.stdout: "name": "log_level", 2024-04-23T10:41:09.972 INFO:teuthology.orchestra.run.smithi114.stdout: "type": "str", 2024-04-23T10:41:09.972 INFO:teuthology.orchestra.run.smithi114.stdout: "level": "advanced", 2024-04-23T10:41:09.973 INFO:teuthology.orchestra.run.smithi114.stdout: "flags": 1, 2024-04-23T10:41:09.973 INFO:teuthology.orchestra.run.smithi114.stdout: "default_value": "", 2024-04-23T10:41:09.973 INFO:teuthology.orchestra.run.smithi114.stdout: "min": "", 2024-04-23T10:41:09.973 INFO:teuthology.orchestra.run.smithi114.stdout: "max": "", 2024-04-23T10:41:09.973 INFO:teuthology.orchestra.run.smithi114.stdout: "enum_allowed": [ 2024-04-23T10:41:09.973 INFO:teuthology.orchestra.run.smithi114.stdout: "", 2024-04-23T10:41:09.973 INFO:teuthology.orchestra.run.smithi114.stdout: "critical", 2024-04-23T10:41:09.973 INFO:teuthology.orchestra.run.smithi114.stdout: "debug", 2024-04-23T10:41:09.973 INFO:teuthology.orchestra.run.smithi114.stdout: "error", 2024-04-23T10:41:09.974 INFO:teuthology.orchestra.run.smithi114.stdout: "info", 2024-04-23T10:41:09.974 INFO:teuthology.orchestra.run.smithi114.stdout: "warning" 2024-04-23T10:41:09.974 INFO:teuthology.orchestra.run.smithi114.stdout: ], 2024-04-23T10:41:09.974 INFO:teuthology.orchestra.run.smithi114.stdout: "desc": "", 2024-04-23T10:41:09.974 INFO:teuthology.orchestra.run.smithi114.stdout: "long_desc": "", 2024-04-23T10:41:09.974 INFO:teuthology.orchestra.run.smithi114.stdout: "tags": [], 2024-04-23T10:41:09.974 INFO:teuthology.orchestra.run.smithi114.stdout: "see_also": [] 2024-04-23T10:41:09.974 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-23T10:41:09.974 INFO:teuthology.orchestra.run.smithi114.stdout: "log_to_cluster": { 2024-04-23T10:41:09.974 INFO:teuthology.orchestra.run.smithi114.stdout: "name": "log_to_cluster", 2024-04-23T10:41:09.975 INFO:teuthology.orchestra.run.smithi114.stdout: "type": "bool", 2024-04-23T10:41:09.975 INFO:teuthology.orchestra.run.smithi114.stdout: "level": "advanced", 2024-04-23T10:41:09.975 INFO:teuthology.orchestra.run.smithi114.stdout: "flags": 1, 2024-04-23T10:41:09.975 INFO:teuthology.orchestra.run.smithi114.stdout: "default_value": "False", 2024-04-23T10:41:09.975 INFO:teuthology.orchestra.run.smithi114.stdout: "min": "", 2024-04-23T10:41:09.975 INFO:teuthology.orchestra.run.smithi114.stdout: "max": "", 2024-04-23T10:41:09.975 INFO:teuthology.orchestra.run.smithi114.stdout: "enum_allowed": [], 2024-04-23T10:41:09.975 INFO:teuthology.orchestra.run.smithi114.stdout: "desc": "", 2024-04-23T10:41:09.975 INFO:teuthology.orchestra.run.smithi114.stdout: "long_desc": "", 2024-04-23T10:41:09.975 INFO:teuthology.orchestra.run.smithi114.stdout: "tags": [], 2024-04-23T10:41:09.976 INFO:teuthology.orchestra.run.smithi114.stdout: "see_also": [] 2024-04-23T10:41:09.976 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-23T10:41:09.976 INFO:teuthology.orchestra.run.smithi114.stdout: "log_to_cluster_level": { 2024-04-23T10:41:09.976 INFO:teuthology.orchestra.run.smithi114.stdout: "name": "log_to_cluster_level", 2024-04-23T10:41:09.976 INFO:teuthology.orchestra.run.smithi114.stdout: "type": "str", 2024-04-23T10:41:09.976 INFO:teuthology.orchestra.run.smithi114.stdout: "level": "advanced", 2024-04-23T10:41:09.976 INFO:teuthology.orchestra.run.smithi114.stdout: "flags": 1, 2024-04-23T10:41:09.976 INFO:teuthology.orchestra.run.smithi114.stdout: "default_value": "info", 2024-04-23T10:41:09.976 INFO:teuthology.orchestra.run.smithi114.stdout: "min": "", 2024-04-23T10:41:09.976 INFO:teuthology.orchestra.run.smithi114.stdout: "max": "", 2024-04-23T10:41:09.977 INFO:teuthology.orchestra.run.smithi114.stdout: "enum_allowed": [ 2024-04-23T10:41:09.977 INFO:teuthology.orchestra.run.smithi114.stdout: "", 2024-04-23T10:41:09.977 INFO:teuthology.orchestra.run.smithi114.stdout: "critical", 2024-04-23T10:41:09.977 INFO:teuthology.orchestra.run.smithi114.stdout: "debug", 2024-04-23T10:41:09.977 INFO:teuthology.orchestra.run.smithi114.stdout: "error", 2024-04-23T10:41:09.977 INFO:teuthology.orchestra.run.smithi114.stdout: "info", 2024-04-23T10:41:09.977 INFO:teuthology.orchestra.run.smithi114.stdout: "warning" 2024-04-23T10:41:09.977 INFO:teuthology.orchestra.run.smithi114.stdout: ], 2024-04-23T10:41:09.977 INFO:teuthology.orchestra.run.smithi114.stdout: "desc": "", 2024-04-23T10:41:09.977 INFO:teuthology.orchestra.run.smithi114.stdout: "long_desc": "", 2024-04-23T10:41:09.978 INFO:teuthology.orchestra.run.smithi114.stdout: "tags": [], 2024-04-23T10:41:09.978 INFO:teuthology.orchestra.run.smithi114.stdout: "see_also": [] 2024-04-23T10:41:09.978 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-23T10:41:09.978 INFO:teuthology.orchestra.run.smithi114.stdout: "log_to_file": { 2024-04-23T10:41:09.978 INFO:teuthology.orchestra.run.smithi114.stdout: "name": "log_to_file", 2024-04-23T10:41:09.978 INFO:teuthology.orchestra.run.smithi114.stdout: "type": "bool", 2024-04-23T10:41:09.978 INFO:teuthology.orchestra.run.smithi114.stdout: "level": "advanced", 2024-04-23T10:41:09.978 INFO:teuthology.orchestra.run.smithi114.stdout: "flags": 1, 2024-04-23T10:41:09.978 INFO:teuthology.orchestra.run.smithi114.stdout: "default_value": "False", 2024-04-23T10:41:09.979 INFO:teuthology.orchestra.run.smithi114.stdout: "min": "", 2024-04-23T10:41:09.979 INFO:teuthology.orchestra.run.smithi114.stdout: "max": "", 2024-04-23T10:41:09.979 INFO:teuthology.orchestra.run.smithi114.stdout: "enum_allowed": [], 2024-04-23T10:41:09.979 INFO:teuthology.orchestra.run.smithi114.stdout: "desc": "", 2024-04-23T10:41:09.979 INFO:teuthology.orchestra.run.smithi114.stdout: "long_desc": "", 2024-04-23T10:41:09.979 INFO:teuthology.orchestra.run.smithi114.stdout: "tags": [], 2024-04-23T10:41:09.979 INFO:teuthology.orchestra.run.smithi114.stdout: "see_also": [] 2024-04-23T10:41:09.979 INFO:teuthology.orchestra.run.smithi114.stdout: } 2024-04-23T10:41:09.979 INFO:teuthology.orchestra.run.smithi114.stdout: } 2024-04-23T10:41:09.979 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-23T10:41:09.980 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-04-23T10:41:09.980 INFO:teuthology.orchestra.run.smithi114.stdout: "name": "stats", 2024-04-23T10:41:09.980 INFO:teuthology.orchestra.run.smithi114.stdout: "can_run": true, 2024-04-23T10:41:09.980 INFO:teuthology.orchestra.run.smithi114.stdout: "error_string": "", 2024-04-23T10:41:09.980 INFO:teuthology.orchestra.run.smithi114.stdout: "module_options": { 2024-04-23T10:41:09.980 INFO:teuthology.orchestra.run.smithi114.stdout: "log_level": { 2024-04-23T10:41:09.980 INFO:teuthology.orchestra.run.smithi114.stdout: "name": "log_level", 2024-04-23T10:41:09.980 INFO:teuthology.orchestra.run.smithi114.stdout: "type": "str", 2024-04-23T10:41:09.980 INFO:teuthology.orchestra.run.smithi114.stdout: "level": "advanced", 2024-04-23T10:41:09.980 INFO:teuthology.orchestra.run.smithi114.stdout: "flags": 1, 2024-04-23T10:41:09.981 INFO:teuthology.orchestra.run.smithi114.stdout: "default_value": "", 2024-04-23T10:41:09.981 INFO:teuthology.orchestra.run.smithi114.stdout: "min": "", 2024-04-23T10:41:09.981 INFO:teuthology.orchestra.run.smithi114.stdout: "max": "", 2024-04-23T10:41:09.981 INFO:teuthology.orchestra.run.smithi114.stdout: "enum_allowed": [ 2024-04-23T10:41:09.981 INFO:teuthology.orchestra.run.smithi114.stdout: "", 2024-04-23T10:41:09.981 INFO:teuthology.orchestra.run.smithi114.stdout: "critical", 2024-04-23T10:41:09.981 INFO:teuthology.orchestra.run.smithi114.stdout: "debug", 2024-04-23T10:41:09.981 INFO:teuthology.orchestra.run.smithi114.stdout: "error", 2024-04-23T10:41:09.981 INFO:teuthology.orchestra.run.smithi114.stdout: "info", 2024-04-23T10:41:09.981 INFO:teuthology.orchestra.run.smithi114.stdout: "warning" 2024-04-23T10:41:09.982 INFO:teuthology.orchestra.run.smithi114.stdout: ], 2024-04-23T10:41:09.982 INFO:teuthology.orchestra.run.smithi114.stdout: "desc": "", 2024-04-23T10:41:09.982 INFO:teuthology.orchestra.run.smithi114.stdout: "long_desc": "", 2024-04-23T10:41:09.982 INFO:teuthology.orchestra.run.smithi114.stdout: "tags": [], 2024-04-23T10:41:09.982 INFO:teuthology.orchestra.run.smithi114.stdout: "see_also": [] 2024-04-23T10:41:09.982 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-23T10:41:09.982 INFO:teuthology.orchestra.run.smithi114.stdout: "log_to_cluster": { 2024-04-23T10:41:09.982 INFO:teuthology.orchestra.run.smithi114.stdout: "name": "log_to_cluster", 2024-04-23T10:41:09.982 INFO:teuthology.orchestra.run.smithi114.stdout: "type": "bool", 2024-04-23T10:41:09.983 INFO:teuthology.orchestra.run.smithi114.stdout: "level": "advanced", 2024-04-23T10:41:09.983 INFO:teuthology.orchestra.run.smithi114.stdout: "flags": 1, 2024-04-23T10:41:09.983 INFO:teuthology.orchestra.run.smithi114.stdout: "default_value": "False", 2024-04-23T10:41:09.983 INFO:teuthology.orchestra.run.smithi114.stdout: "min": "", 2024-04-23T10:41:09.983 INFO:teuthology.orchestra.run.smithi114.stdout: "max": "", 2024-04-23T10:41:09.983 INFO:teuthology.orchestra.run.smithi114.stdout: "enum_allowed": [], 2024-04-23T10:41:09.983 INFO:teuthology.orchestra.run.smithi114.stdout: "desc": "", 2024-04-23T10:41:09.983 INFO:teuthology.orchestra.run.smithi114.stdout: "long_desc": "", 2024-04-23T10:41:09.983 INFO:teuthology.orchestra.run.smithi114.stdout: "tags": [], 2024-04-23T10:41:09.983 INFO:teuthology.orchestra.run.smithi114.stdout: "see_also": [] 2024-04-23T10:41:09.984 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-23T10:41:09.984 INFO:teuthology.orchestra.run.smithi114.stdout: "log_to_cluster_level": { 2024-04-23T10:41:09.984 INFO:teuthology.orchestra.run.smithi114.stdout: "name": "log_to_cluster_level", 2024-04-23T10:41:09.984 INFO:teuthology.orchestra.run.smithi114.stdout: "type": "str", 2024-04-23T10:41:09.984 INFO:teuthology.orchestra.run.smithi114.stdout: "level": "advanced", 2024-04-23T10:41:09.984 INFO:teuthology.orchestra.run.smithi114.stdout: "flags": 1, 2024-04-23T10:41:09.984 INFO:teuthology.orchestra.run.smithi114.stdout: "default_value": "info", 2024-04-23T10:41:09.984 INFO:teuthology.orchestra.run.smithi114.stdout: "min": "", 2024-04-23T10:41:09.984 INFO:teuthology.orchestra.run.smithi114.stdout: "max": "", 2024-04-23T10:41:09.984 INFO:teuthology.orchestra.run.smithi114.stdout: "enum_allowed": [ 2024-04-23T10:41:09.985 INFO:teuthology.orchestra.run.smithi114.stdout: "", 2024-04-23T10:41:09.985 INFO:teuthology.orchestra.run.smithi114.stdout: "critical", 2024-04-23T10:41:09.985 INFO:teuthology.orchestra.run.smithi114.stdout: "debug", 2024-04-23T10:41:09.985 INFO:teuthology.orchestra.run.smithi114.stdout: "error", 2024-04-23T10:41:09.985 INFO:teuthology.orchestra.run.smithi114.stdout: "info", 2024-04-23T10:41:09.985 INFO:teuthology.orchestra.run.smithi114.stdout: "warning" 2024-04-23T10:41:09.985 INFO:teuthology.orchestra.run.smithi114.stdout: ], 2024-04-23T10:41:09.985 INFO:teuthology.orchestra.run.smithi114.stdout: "desc": "", 2024-04-23T10:41:09.985 INFO:teuthology.orchestra.run.smithi114.stdout: "long_desc": "", 2024-04-23T10:41:09.985 INFO:teuthology.orchestra.run.smithi114.stdout: "tags": [], 2024-04-23T10:41:09.986 INFO:teuthology.orchestra.run.smithi114.stdout: "see_also": [] 2024-04-23T10:41:09.986 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-23T10:41:09.986 INFO:teuthology.orchestra.run.smithi114.stdout: "log_to_file": { 2024-04-23T10:41:09.986 INFO:teuthology.orchestra.run.smithi114.stdout: "name": "log_to_file", 2024-04-23T10:41:09.986 INFO:teuthology.orchestra.run.smithi114.stdout: "type": "bool", 2024-04-23T10:41:09.986 INFO:teuthology.orchestra.run.smithi114.stdout: "level": "advanced", 2024-04-23T10:41:09.986 INFO:teuthology.orchestra.run.smithi114.stdout: "flags": 1, 2024-04-23T10:41:09.986 INFO:teuthology.orchestra.run.smithi114.stdout: "default_value": "False", 2024-04-23T10:41:09.986 INFO:teuthology.orchestra.run.smithi114.stdout: "min": "", 2024-04-23T10:41:09.987 INFO:teuthology.orchestra.run.smithi114.stdout: "max": "", 2024-04-23T10:41:09.987 INFO:teuthology.orchestra.run.smithi114.stdout: "enum_allowed": [], 2024-04-23T10:41:09.987 INFO:teuthology.orchestra.run.smithi114.stdout: "desc": "", 2024-04-23T10:41:09.987 INFO:teuthology.orchestra.run.smithi114.stdout: "long_desc": "", 2024-04-23T10:41:09.987 INFO:teuthology.orchestra.run.smithi114.stdout: "tags": [], 2024-04-23T10:41:09.987 INFO:teuthology.orchestra.run.smithi114.stdout: "see_also": [] 2024-04-23T10:41:09.987 INFO:teuthology.orchestra.run.smithi114.stdout: } 2024-04-23T10:41:09.987 INFO:teuthology.orchestra.run.smithi114.stdout: } 2024-04-23T10:41:09.987 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-23T10:41:09.987 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-04-23T10:41:09.988 INFO:teuthology.orchestra.run.smithi114.stdout: "name": "status", 2024-04-23T10:41:09.988 INFO:teuthology.orchestra.run.smithi114.stdout: "can_run": true, 2024-04-23T10:41:09.988 INFO:teuthology.orchestra.run.smithi114.stdout: "error_string": "", 2024-04-23T10:41:09.988 INFO:teuthology.orchestra.run.smithi114.stdout: "module_options": { 2024-04-23T10:41:09.988 INFO:teuthology.orchestra.run.smithi114.stdout: "log_level": { 2024-04-23T10:41:09.988 INFO:teuthology.orchestra.run.smithi114.stdout: "name": "log_level", 2024-04-23T10:41:09.988 INFO:teuthology.orchestra.run.smithi114.stdout: "type": "str", 2024-04-23T10:41:09.988 INFO:teuthology.orchestra.run.smithi114.stdout: "level": "advanced", 2024-04-23T10:41:09.988 INFO:teuthology.orchestra.run.smithi114.stdout: "flags": 1, 2024-04-23T10:41:09.988 INFO:teuthology.orchestra.run.smithi114.stdout: "default_value": "", 2024-04-23T10:41:09.989 INFO:teuthology.orchestra.run.smithi114.stdout: "min": "", 2024-04-23T10:41:09.989 INFO:teuthology.orchestra.run.smithi114.stdout: "max": "", 2024-04-23T10:41:09.989 INFO:teuthology.orchestra.run.smithi114.stdout: "enum_allowed": [ 2024-04-23T10:41:09.989 INFO:teuthology.orchestra.run.smithi114.stdout: "", 2024-04-23T10:41:09.989 INFO:teuthology.orchestra.run.smithi114.stdout: "critical", 2024-04-23T10:41:09.989 INFO:teuthology.orchestra.run.smithi114.stdout: "debug", 2024-04-23T10:41:09.989 INFO:teuthology.orchestra.run.smithi114.stdout: "error", 2024-04-23T10:41:09.989 INFO:teuthology.orchestra.run.smithi114.stdout: "info", 2024-04-23T10:41:09.989 INFO:teuthology.orchestra.run.smithi114.stdout: "warning" 2024-04-23T10:41:09.989 INFO:teuthology.orchestra.run.smithi114.stdout: ], 2024-04-23T10:41:09.990 INFO:teuthology.orchestra.run.smithi114.stdout: "desc": "", 2024-04-23T10:41:09.990 INFO:teuthology.orchestra.run.smithi114.stdout: "long_desc": "", 2024-04-23T10:41:09.990 INFO:teuthology.orchestra.run.smithi114.stdout: "tags": [], 2024-04-23T10:41:09.990 INFO:teuthology.orchestra.run.smithi114.stdout: "see_also": [] 2024-04-23T10:41:09.990 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-23T10:41:09.990 INFO:teuthology.orchestra.run.smithi114.stdout: "log_to_cluster": { 2024-04-23T10:41:09.990 INFO:teuthology.orchestra.run.smithi114.stdout: "name": "log_to_cluster", 2024-04-23T10:41:09.990 INFO:teuthology.orchestra.run.smithi114.stdout: "type": "bool", 2024-04-23T10:41:09.990 INFO:teuthology.orchestra.run.smithi114.stdout: "level": "advanced", 2024-04-23T10:41:09.990 INFO:teuthology.orchestra.run.smithi114.stdout: "flags": 1, 2024-04-23T10:41:09.991 INFO:teuthology.orchestra.run.smithi114.stdout: "default_value": "False", 2024-04-23T10:41:09.991 INFO:teuthology.orchestra.run.smithi114.stdout: "min": "", 2024-04-23T10:41:09.991 INFO:teuthology.orchestra.run.smithi114.stdout: "max": "", 2024-04-23T10:41:09.991 INFO:teuthology.orchestra.run.smithi114.stdout: "enum_allowed": [], 2024-04-23T10:41:09.991 INFO:teuthology.orchestra.run.smithi114.stdout: "desc": "", 2024-04-23T10:41:09.991 INFO:teuthology.orchestra.run.smithi114.stdout: "long_desc": "", 2024-04-23T10:41:09.991 INFO:teuthology.orchestra.run.smithi114.stdout: "tags": [], 2024-04-23T10:41:09.991 INFO:teuthology.orchestra.run.smithi114.stdout: "see_also": [] 2024-04-23T10:41:09.991 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-23T10:41:09.991 INFO:teuthology.orchestra.run.smithi114.stdout: "log_to_cluster_level": { 2024-04-23T10:41:09.992 INFO:teuthology.orchestra.run.smithi114.stdout: "name": "log_to_cluster_level", 2024-04-23T10:41:09.992 INFO:teuthology.orchestra.run.smithi114.stdout: "type": "str", 2024-04-23T10:41:09.992 INFO:teuthology.orchestra.run.smithi114.stdout: "level": "advanced", 2024-04-23T10:41:09.992 INFO:teuthology.orchestra.run.smithi114.stdout: "flags": 1, 2024-04-23T10:41:09.992 INFO:teuthology.orchestra.run.smithi114.stdout: "default_value": "info", 2024-04-23T10:41:09.992 INFO:teuthology.orchestra.run.smithi114.stdout: "min": "", 2024-04-23T10:41:09.992 INFO:teuthology.orchestra.run.smithi114.stdout: "max": "", 2024-04-23T10:41:09.992 INFO:teuthology.orchestra.run.smithi114.stdout: "enum_allowed": [ 2024-04-23T10:41:09.992 INFO:teuthology.orchestra.run.smithi114.stdout: "", 2024-04-23T10:41:09.992 INFO:teuthology.orchestra.run.smithi114.stdout: "critical", 2024-04-23T10:41:09.993 INFO:teuthology.orchestra.run.smithi114.stdout: "debug", 2024-04-23T10:41:09.993 INFO:teuthology.orchestra.run.smithi114.stdout: "error", 2024-04-23T10:41:09.993 INFO:teuthology.orchestra.run.smithi114.stdout: "info", 2024-04-23T10:41:09.993 INFO:teuthology.orchestra.run.smithi114.stdout: "warning" 2024-04-23T10:41:09.993 INFO:teuthology.orchestra.run.smithi114.stdout: ], 2024-04-23T10:41:09.993 INFO:teuthology.orchestra.run.smithi114.stdout: "desc": "", 2024-04-23T10:41:09.993 INFO:teuthology.orchestra.run.smithi114.stdout: "long_desc": "", 2024-04-23T10:41:09.993 INFO:teuthology.orchestra.run.smithi114.stdout: "tags": [], 2024-04-23T10:41:09.993 INFO:teuthology.orchestra.run.smithi114.stdout: "see_also": [] 2024-04-23T10:41:09.993 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-23T10:41:09.994 INFO:teuthology.orchestra.run.smithi114.stdout: "log_to_file": { 2024-04-23T10:41:09.994 INFO:teuthology.orchestra.run.smithi114.stdout: "name": "log_to_file", 2024-04-23T10:41:09.994 INFO:teuthology.orchestra.run.smithi114.stdout: "type": "bool", 2024-04-23T10:41:09.994 INFO:teuthology.orchestra.run.smithi114.stdout: "level": "advanced", 2024-04-23T10:41:09.994 INFO:teuthology.orchestra.run.smithi114.stdout: "flags": 1, 2024-04-23T10:41:09.994 INFO:teuthology.orchestra.run.smithi114.stdout: "default_value": "False", 2024-04-23T10:41:09.994 INFO:teuthology.orchestra.run.smithi114.stdout: "min": "", 2024-04-23T10:41:09.994 INFO:teuthology.orchestra.run.smithi114.stdout: "max": "", 2024-04-23T10:41:09.994 INFO:teuthology.orchestra.run.smithi114.stdout: "enum_allowed": [], 2024-04-23T10:41:09.994 INFO:teuthology.orchestra.run.smithi114.stdout: "desc": "", 2024-04-23T10:41:09.995 INFO:teuthology.orchestra.run.smithi114.stdout: "long_desc": "", 2024-04-23T10:41:09.995 INFO:teuthology.orchestra.run.smithi114.stdout: "tags": [], 2024-04-23T10:41:09.995 INFO:teuthology.orchestra.run.smithi114.stdout: "see_also": [] 2024-04-23T10:41:09.995 INFO:teuthology.orchestra.run.smithi114.stdout: } 2024-04-23T10:41:09.995 INFO:teuthology.orchestra.run.smithi114.stdout: } 2024-04-23T10:41:09.995 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-23T10:41:09.995 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-04-23T10:41:09.995 INFO:teuthology.orchestra.run.smithi114.stdout: "name": "telegraf", 2024-04-23T10:41:09.995 INFO:teuthology.orchestra.run.smithi114.stdout: "can_run": true, 2024-04-23T10:41:09.995 INFO:teuthology.orchestra.run.smithi114.stdout: "error_string": "", 2024-04-23T10:41:09.996 INFO:teuthology.orchestra.run.smithi114.stdout: "module_options": { 2024-04-23T10:41:09.996 INFO:teuthology.orchestra.run.smithi114.stdout: "address": { 2024-04-23T10:41:09.996 INFO:teuthology.orchestra.run.smithi114.stdout: "name": "address", 2024-04-23T10:41:09.996 INFO:teuthology.orchestra.run.smithi114.stdout: "type": "str", 2024-04-23T10:41:09.996 INFO:teuthology.orchestra.run.smithi114.stdout: "level": "advanced", 2024-04-23T10:41:09.996 INFO:teuthology.orchestra.run.smithi114.stdout: "flags": 0, 2024-04-23T10:41:09.996 INFO:teuthology.orchestra.run.smithi114.stdout: "default_value": "unixgram:///tmp/telegraf.sock", 2024-04-23T10:41:09.996 INFO:teuthology.orchestra.run.smithi114.stdout: "min": "", 2024-04-23T10:41:09.996 INFO:teuthology.orchestra.run.smithi114.stdout: "max": "", 2024-04-23T10:41:09.997 INFO:teuthology.orchestra.run.smithi114.stdout: "enum_allowed": [], 2024-04-23T10:41:09.997 INFO:teuthology.orchestra.run.smithi114.stdout: "desc": "", 2024-04-23T10:41:09.997 INFO:teuthology.orchestra.run.smithi114.stdout: "long_desc": "", 2024-04-23T10:41:09.997 INFO:teuthology.orchestra.run.smithi114.stdout: "tags": [], 2024-04-23T10:41:09.997 INFO:teuthology.orchestra.run.smithi114.stdout: "see_also": [] 2024-04-23T10:41:09.997 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-23T10:41:09.997 INFO:teuthology.orchestra.run.smithi114.stdout: "interval": { 2024-04-23T10:41:09.997 INFO:teuthology.orchestra.run.smithi114.stdout: "name": "interval", 2024-04-23T10:41:09.997 INFO:teuthology.orchestra.run.smithi114.stdout: "type": "secs", 2024-04-23T10:41:09.997 INFO:teuthology.orchestra.run.smithi114.stdout: "level": "advanced", 2024-04-23T10:41:09.998 INFO:teuthology.orchestra.run.smithi114.stdout: "flags": 0, 2024-04-23T10:41:09.998 INFO:teuthology.orchestra.run.smithi114.stdout: "default_value": "15", 2024-04-23T10:41:09.998 INFO:teuthology.orchestra.run.smithi114.stdout: "min": "", 2024-04-23T10:41:09.998 INFO:teuthology.orchestra.run.smithi114.stdout: "max": "", 2024-04-23T10:41:09.998 INFO:teuthology.orchestra.run.smithi114.stdout: "enum_allowed": [], 2024-04-23T10:41:09.998 INFO:teuthology.orchestra.run.smithi114.stdout: "desc": "", 2024-04-23T10:41:09.998 INFO:teuthology.orchestra.run.smithi114.stdout: "long_desc": "", 2024-04-23T10:41:09.998 INFO:teuthology.orchestra.run.smithi114.stdout: "tags": [], 2024-04-23T10:41:09.998 INFO:teuthology.orchestra.run.smithi114.stdout: "see_also": [] 2024-04-23T10:41:09.999 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-23T10:41:09.999 INFO:teuthology.orchestra.run.smithi114.stdout: "log_level": { 2024-04-23T10:41:09.999 INFO:teuthology.orchestra.run.smithi114.stdout: "name": "log_level", 2024-04-23T10:41:09.999 INFO:teuthology.orchestra.run.smithi114.stdout: "type": "str", 2024-04-23T10:41:09.999 INFO:teuthology.orchestra.run.smithi114.stdout: "level": "advanced", 2024-04-23T10:41:09.999 INFO:teuthology.orchestra.run.smithi114.stdout: "flags": 1, 2024-04-23T10:41:09.999 INFO:teuthology.orchestra.run.smithi114.stdout: "default_value": "", 2024-04-23T10:41:09.999 INFO:teuthology.orchestra.run.smithi114.stdout: "min": "", 2024-04-23T10:41:09.999 INFO:teuthology.orchestra.run.smithi114.stdout: "max": "", 2024-04-23T10:41:09.999 INFO:teuthology.orchestra.run.smithi114.stdout: "enum_allowed": [ 2024-04-23T10:41:10.000 INFO:teuthology.orchestra.run.smithi114.stdout: "", 2024-04-23T10:41:10.000 INFO:teuthology.orchestra.run.smithi114.stdout: "critical", 2024-04-23T10:41:10.000 INFO:teuthology.orchestra.run.smithi114.stdout: "debug", 2024-04-23T10:41:10.000 INFO:teuthology.orchestra.run.smithi114.stdout: "error", 2024-04-23T10:41:10.000 INFO:teuthology.orchestra.run.smithi114.stdout: "info", 2024-04-23T10:41:10.000 INFO:teuthology.orchestra.run.smithi114.stdout: "warning" 2024-04-23T10:41:10.000 INFO:teuthology.orchestra.run.smithi114.stdout: ], 2024-04-23T10:41:10.000 INFO:teuthology.orchestra.run.smithi114.stdout: "desc": "", 2024-04-23T10:41:10.000 INFO:teuthology.orchestra.run.smithi114.stdout: "long_desc": "", 2024-04-23T10:41:10.000 INFO:teuthology.orchestra.run.smithi114.stdout: "tags": [], 2024-04-23T10:41:10.001 INFO:teuthology.orchestra.run.smithi114.stdout: "see_also": [] 2024-04-23T10:41:10.001 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-23T10:41:10.001 INFO:teuthology.orchestra.run.smithi114.stdout: "log_to_cluster": { 2024-04-23T10:41:10.001 INFO:teuthology.orchestra.run.smithi114.stdout: "name": "log_to_cluster", 2024-04-23T10:41:10.001 INFO:teuthology.orchestra.run.smithi114.stdout: "type": "bool", 2024-04-23T10:41:10.001 INFO:teuthology.orchestra.run.smithi114.stdout: "level": "advanced", 2024-04-23T10:41:10.001 INFO:teuthology.orchestra.run.smithi114.stdout: "flags": 1, 2024-04-23T10:41:10.001 INFO:teuthology.orchestra.run.smithi114.stdout: "default_value": "False", 2024-04-23T10:41:10.002 INFO:teuthology.orchestra.run.smithi114.stdout: "min": "", 2024-04-23T10:41:10.002 INFO:teuthology.orchestra.run.smithi114.stdout: "max": "", 2024-04-23T10:41:10.002 INFO:teuthology.orchestra.run.smithi114.stdout: "enum_allowed": [], 2024-04-23T10:41:10.002 INFO:teuthology.orchestra.run.smithi114.stdout: "desc": "", 2024-04-23T10:41:10.002 INFO:teuthology.orchestra.run.smithi114.stdout: "long_desc": "", 2024-04-23T10:41:10.002 INFO:teuthology.orchestra.run.smithi114.stdout: "tags": [], 2024-04-23T10:41:10.002 INFO:teuthology.orchestra.run.smithi114.stdout: "see_also": [] 2024-04-23T10:41:10.002 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-23T10:41:10.002 INFO:teuthology.orchestra.run.smithi114.stdout: "log_to_cluster_level": { 2024-04-23T10:41:10.002 INFO:teuthology.orchestra.run.smithi114.stdout: "name": "log_to_cluster_level", 2024-04-23T10:41:10.003 INFO:teuthology.orchestra.run.smithi114.stdout: "type": "str", 2024-04-23T10:41:10.003 INFO:teuthology.orchestra.run.smithi114.stdout: "level": "advanced", 2024-04-23T10:41:10.003 INFO:teuthology.orchestra.run.smithi114.stdout: "flags": 1, 2024-04-23T10:41:10.003 INFO:teuthology.orchestra.run.smithi114.stdout: "default_value": "info", 2024-04-23T10:41:10.003 INFO:teuthology.orchestra.run.smithi114.stdout: "min": "", 2024-04-23T10:41:10.003 INFO:teuthology.orchestra.run.smithi114.stdout: "max": "", 2024-04-23T10:41:10.003 INFO:teuthology.orchestra.run.smithi114.stdout: "enum_allowed": [ 2024-04-23T10:41:10.003 INFO:teuthology.orchestra.run.smithi114.stdout: "", 2024-04-23T10:41:10.003 INFO:teuthology.orchestra.run.smithi114.stdout: "critical", 2024-04-23T10:41:10.004 INFO:teuthology.orchestra.run.smithi114.stdout: "debug", 2024-04-23T10:41:10.004 INFO:teuthology.orchestra.run.smithi114.stdout: "error", 2024-04-23T10:41:10.004 INFO:teuthology.orchestra.run.smithi114.stdout: "info", 2024-04-23T10:41:10.004 INFO:teuthology.orchestra.run.smithi114.stdout: "warning" 2024-04-23T10:41:10.004 INFO:teuthology.orchestra.run.smithi114.stdout: ], 2024-04-23T10:41:10.004 INFO:teuthology.orchestra.run.smithi114.stdout: "desc": "", 2024-04-23T10:41:10.004 INFO:teuthology.orchestra.run.smithi114.stdout: "long_desc": "", 2024-04-23T10:41:10.004 INFO:teuthology.orchestra.run.smithi114.stdout: "tags": [], 2024-04-23T10:41:10.004 INFO:teuthology.orchestra.run.smithi114.stdout: "see_also": [] 2024-04-23T10:41:10.005 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-23T10:41:10.005 INFO:teuthology.orchestra.run.smithi114.stdout: "log_to_file": { 2024-04-23T10:41:10.005 INFO:teuthology.orchestra.run.smithi114.stdout: "name": "log_to_file", 2024-04-23T10:41:10.005 INFO:teuthology.orchestra.run.smithi114.stdout: "type": "bool", 2024-04-23T10:41:10.005 INFO:teuthology.orchestra.run.smithi114.stdout: "level": "advanced", 2024-04-23T10:41:10.005 INFO:teuthology.orchestra.run.smithi114.stdout: "flags": 1, 2024-04-23T10:41:10.005 INFO:teuthology.orchestra.run.smithi114.stdout: "default_value": "False", 2024-04-23T10:41:10.005 INFO:teuthology.orchestra.run.smithi114.stdout: "min": "", 2024-04-23T10:41:10.005 INFO:teuthology.orchestra.run.smithi114.stdout: "max": "", 2024-04-23T10:41:10.005 INFO:teuthology.orchestra.run.smithi114.stdout: "enum_allowed": [], 2024-04-23T10:41:10.006 INFO:teuthology.orchestra.run.smithi114.stdout: "desc": "", 2024-04-23T10:41:10.006 INFO:teuthology.orchestra.run.smithi114.stdout: "long_desc": "", 2024-04-23T10:41:10.006 INFO:teuthology.orchestra.run.smithi114.stdout: "tags": [], 2024-04-23T10:41:10.006 INFO:teuthology.orchestra.run.smithi114.stdout: "see_also": [] 2024-04-23T10:41:10.006 INFO:teuthology.orchestra.run.smithi114.stdout: } 2024-04-23T10:41:10.006 INFO:teuthology.orchestra.run.smithi114.stdout: } 2024-04-23T10:41:10.006 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-23T10:41:10.006 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-04-23T10:41:10.006 INFO:teuthology.orchestra.run.smithi114.stdout: "name": "telemetry", 2024-04-23T10:41:10.006 INFO:teuthology.orchestra.run.smithi114.stdout: "can_run": true, 2024-04-23T10:41:10.007 INFO:teuthology.orchestra.run.smithi114.stdout: "error_string": "", 2024-04-23T10:41:10.007 INFO:teuthology.orchestra.run.smithi114.stdout: "module_options": { 2024-04-23T10:41:10.007 INFO:teuthology.orchestra.run.smithi114.stdout: "channel_basic": { 2024-04-23T10:41:10.007 INFO:teuthology.orchestra.run.smithi114.stdout: "name": "channel_basic", 2024-04-23T10:41:10.007 INFO:teuthology.orchestra.run.smithi114.stdout: "type": "bool", 2024-04-23T10:41:10.007 INFO:teuthology.orchestra.run.smithi114.stdout: "level": "advanced", 2024-04-23T10:41:10.007 INFO:teuthology.orchestra.run.smithi114.stdout: "flags": 0, 2024-04-23T10:41:10.007 INFO:teuthology.orchestra.run.smithi114.stdout: "default_value": "True", 2024-04-23T10:41:10.007 INFO:teuthology.orchestra.run.smithi114.stdout: "min": "", 2024-04-23T10:41:10.007 INFO:teuthology.orchestra.run.smithi114.stdout: "max": "", 2024-04-23T10:41:10.008 INFO:teuthology.orchestra.run.smithi114.stdout: "enum_allowed": [], 2024-04-23T10:41:10.008 INFO:teuthology.orchestra.run.smithi114.stdout: "desc": "Share basic cluster information (size, version)", 2024-04-23T10:41:10.008 INFO:teuthology.orchestra.run.smithi114.stdout: "long_desc": "", 2024-04-23T10:41:10.008 INFO:teuthology.orchestra.run.smithi114.stdout: "tags": [], 2024-04-23T10:41:10.008 INFO:teuthology.orchestra.run.smithi114.stdout: "see_also": [] 2024-04-23T10:41:10.008 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-23T10:41:10.008 INFO:teuthology.orchestra.run.smithi114.stdout: "channel_crash": { 2024-04-23T10:41:10.008 INFO:teuthology.orchestra.run.smithi114.stdout: "name": "channel_crash", 2024-04-23T10:41:10.008 INFO:teuthology.orchestra.run.smithi114.stdout: "type": "bool", 2024-04-23T10:41:10.008 INFO:teuthology.orchestra.run.smithi114.stdout: "level": "advanced", 2024-04-23T10:41:10.009 INFO:teuthology.orchestra.run.smithi114.stdout: "flags": 0, 2024-04-23T10:41:10.009 INFO:teuthology.orchestra.run.smithi114.stdout: "default_value": "True", 2024-04-23T10:41:10.009 INFO:teuthology.orchestra.run.smithi114.stdout: "min": "", 2024-04-23T10:41:10.009 INFO:teuthology.orchestra.run.smithi114.stdout: "max": "", 2024-04-23T10:41:10.009 INFO:teuthology.orchestra.run.smithi114.stdout: "enum_allowed": [], 2024-04-23T10:41:10.009 INFO:teuthology.orchestra.run.smithi114.stdout: "desc": "Share metadata about Ceph daemon crashes (version, stack straces, etc)", 2024-04-23T10:41:10.009 INFO:teuthology.orchestra.run.smithi114.stdout: "long_desc": "", 2024-04-23T10:41:10.009 INFO:teuthology.orchestra.run.smithi114.stdout: "tags": [], 2024-04-23T10:41:10.009 INFO:teuthology.orchestra.run.smithi114.stdout: "see_also": [] 2024-04-23T10:41:10.010 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-23T10:41:10.010 INFO:teuthology.orchestra.run.smithi114.stdout: "channel_device": { 2024-04-23T10:41:10.010 INFO:teuthology.orchestra.run.smithi114.stdout: "name": "channel_device", 2024-04-23T10:41:10.010 INFO:teuthology.orchestra.run.smithi114.stdout: "type": "bool", 2024-04-23T10:41:10.010 INFO:teuthology.orchestra.run.smithi114.stdout: "level": "advanced", 2024-04-23T10:41:10.010 INFO:teuthology.orchestra.run.smithi114.stdout: "flags": 0, 2024-04-23T10:41:10.010 INFO:teuthology.orchestra.run.smithi114.stdout: "default_value": "True", 2024-04-23T10:41:10.010 INFO:teuthology.orchestra.run.smithi114.stdout: "min": "", 2024-04-23T10:41:10.010 INFO:teuthology.orchestra.run.smithi114.stdout: "max": "", 2024-04-23T10:41:10.010 INFO:teuthology.orchestra.run.smithi114.stdout: "enum_allowed": [], 2024-04-23T10:41:10.011 INFO:teuthology.orchestra.run.smithi114.stdout: "desc": "Share device health metrics (e.g., SMART data, minus potentially identifying info like serial numbers)", 2024-04-23T10:41:10.011 INFO:teuthology.orchestra.run.smithi114.stdout: "long_desc": "", 2024-04-23T10:41:10.011 INFO:teuthology.orchestra.run.smithi114.stdout: "tags": [], 2024-04-23T10:41:10.011 INFO:teuthology.orchestra.run.smithi114.stdout: "see_also": [] 2024-04-23T10:41:10.011 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-23T10:41:10.011 INFO:teuthology.orchestra.run.smithi114.stdout: "channel_ident": { 2024-04-23T10:41:10.011 INFO:teuthology.orchestra.run.smithi114.stdout: "name": "channel_ident", 2024-04-23T10:41:10.011 INFO:teuthology.orchestra.run.smithi114.stdout: "type": "bool", 2024-04-23T10:41:10.012 INFO:teuthology.orchestra.run.smithi114.stdout: "level": "advanced", 2024-04-23T10:41:10.012 INFO:teuthology.orchestra.run.smithi114.stdout: "flags": 0, 2024-04-23T10:41:10.012 INFO:teuthology.orchestra.run.smithi114.stdout: "default_value": "False", 2024-04-23T10:41:10.012 INFO:teuthology.orchestra.run.smithi114.stdout: "min": "", 2024-04-23T10:41:10.012 INFO:teuthology.orchestra.run.smithi114.stdout: "max": "", 2024-04-23T10:41:10.012 INFO:teuthology.orchestra.run.smithi114.stdout: "enum_allowed": [], 2024-04-23T10:41:10.012 INFO:teuthology.orchestra.run.smithi114.stdout: "desc": "Share a user-provided description and/or contact email for the cluster", 2024-04-23T10:41:10.012 INFO:teuthology.orchestra.run.smithi114.stdout: "long_desc": "", 2024-04-23T10:41:10.012 INFO:teuthology.orchestra.run.smithi114.stdout: "tags": [], 2024-04-23T10:41:10.013 INFO:teuthology.orchestra.run.smithi114.stdout: "see_also": [] 2024-04-23T10:41:10.013 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-23T10:41:10.013 INFO:teuthology.orchestra.run.smithi114.stdout: "channel_perf": { 2024-04-23T10:41:10.013 INFO:teuthology.orchestra.run.smithi114.stdout: "name": "channel_perf", 2024-04-23T10:41:10.013 INFO:teuthology.orchestra.run.smithi114.stdout: "type": "bool", 2024-04-23T10:41:10.013 INFO:teuthology.orchestra.run.smithi114.stdout: "level": "advanced", 2024-04-23T10:41:10.013 INFO:teuthology.orchestra.run.smithi114.stdout: "flags": 0, 2024-04-23T10:41:10.013 INFO:teuthology.orchestra.run.smithi114.stdout: "default_value": "False", 2024-04-23T10:41:10.013 INFO:teuthology.orchestra.run.smithi114.stdout: "min": "", 2024-04-23T10:41:10.013 INFO:teuthology.orchestra.run.smithi114.stdout: "max": "", 2024-04-23T10:41:10.014 INFO:teuthology.orchestra.run.smithi114.stdout: "enum_allowed": [], 2024-04-23T10:41:10.014 INFO:teuthology.orchestra.run.smithi114.stdout: "desc": "Share various performance metrics of a cluster", 2024-04-23T10:41:10.014 INFO:teuthology.orchestra.run.smithi114.stdout: "long_desc": "", 2024-04-23T10:41:10.014 INFO:teuthology.orchestra.run.smithi114.stdout: "tags": [], 2024-04-23T10:41:10.014 INFO:teuthology.orchestra.run.smithi114.stdout: "see_also": [] 2024-04-23T10:41:10.014 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-23T10:41:10.014 INFO:teuthology.orchestra.run.smithi114.stdout: "contact": { 2024-04-23T10:41:10.014 INFO:teuthology.orchestra.run.smithi114.stdout: "name": "contact", 2024-04-23T10:41:10.014 INFO:teuthology.orchestra.run.smithi114.stdout: "type": "str", 2024-04-23T10:41:10.015 INFO:teuthology.orchestra.run.smithi114.stdout: "level": "advanced", 2024-04-23T10:41:10.015 INFO:teuthology.orchestra.run.smithi114.stdout: "flags": 0, 2024-04-23T10:41:10.015 INFO:teuthology.orchestra.run.smithi114.stdout: "default_value": "", 2024-04-23T10:41:10.015 INFO:teuthology.orchestra.run.smithi114.stdout: "min": "", 2024-04-23T10:41:10.015 INFO:teuthology.orchestra.run.smithi114.stdout: "max": "", 2024-04-23T10:41:10.015 INFO:teuthology.orchestra.run.smithi114.stdout: "enum_allowed": [], 2024-04-23T10:41:10.015 INFO:teuthology.orchestra.run.smithi114.stdout: "desc": "", 2024-04-23T10:41:10.015 INFO:teuthology.orchestra.run.smithi114.stdout: "long_desc": "", 2024-04-23T10:41:10.015 INFO:teuthology.orchestra.run.smithi114.stdout: "tags": [], 2024-04-23T10:41:10.016 INFO:teuthology.orchestra.run.smithi114.stdout: "see_also": [] 2024-04-23T10:41:10.016 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-23T10:41:10.016 INFO:teuthology.orchestra.run.smithi114.stdout: "description": { 2024-04-23T10:41:10.016 INFO:teuthology.orchestra.run.smithi114.stdout: "name": "description", 2024-04-23T10:41:10.016 INFO:teuthology.orchestra.run.smithi114.stdout: "type": "str", 2024-04-23T10:41:10.016 INFO:teuthology.orchestra.run.smithi114.stdout: "level": "advanced", 2024-04-23T10:41:10.016 INFO:teuthology.orchestra.run.smithi114.stdout: "flags": 0, 2024-04-23T10:41:10.016 INFO:teuthology.orchestra.run.smithi114.stdout: "default_value": "", 2024-04-23T10:41:10.016 INFO:teuthology.orchestra.run.smithi114.stdout: "min": "", 2024-04-23T10:41:10.017 INFO:teuthology.orchestra.run.smithi114.stdout: "max": "", 2024-04-23T10:41:10.017 INFO:teuthology.orchestra.run.smithi114.stdout: "enum_allowed": [], 2024-04-23T10:41:10.017 INFO:teuthology.orchestra.run.smithi114.stdout: "desc": "", 2024-04-23T10:41:10.017 INFO:teuthology.orchestra.run.smithi114.stdout: "long_desc": "", 2024-04-23T10:41:10.017 INFO:teuthology.orchestra.run.smithi114.stdout: "tags": [], 2024-04-23T10:41:10.017 INFO:teuthology.orchestra.run.smithi114.stdout: "see_also": [] 2024-04-23T10:41:10.017 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-23T10:41:10.017 INFO:teuthology.orchestra.run.smithi114.stdout: "device_url": { 2024-04-23T10:41:10.017 INFO:teuthology.orchestra.run.smithi114.stdout: "name": "device_url", 2024-04-23T10:41:10.018 INFO:teuthology.orchestra.run.smithi114.stdout: "type": "str", 2024-04-23T10:41:10.018 INFO:teuthology.orchestra.run.smithi114.stdout: "level": "advanced", 2024-04-23T10:41:10.018 INFO:teuthology.orchestra.run.smithi114.stdout: "flags": 0, 2024-04-23T10:41:10.018 INFO:teuthology.orchestra.run.smithi114.stdout: "default_value": "https://telemetry.ceph.com/device", 2024-04-23T10:41:10.018 INFO:teuthology.orchestra.run.smithi114.stdout: "min": "", 2024-04-23T10:41:10.018 INFO:teuthology.orchestra.run.smithi114.stdout: "max": "", 2024-04-23T10:41:10.018 INFO:teuthology.orchestra.run.smithi114.stdout: "enum_allowed": [], 2024-04-23T10:41:10.018 INFO:teuthology.orchestra.run.smithi114.stdout: "desc": "", 2024-04-23T10:41:10.019 INFO:teuthology.orchestra.run.smithi114.stdout: "long_desc": "", 2024-04-23T10:41:10.019 INFO:teuthology.orchestra.run.smithi114.stdout: "tags": [], 2024-04-23T10:41:10.019 INFO:teuthology.orchestra.run.smithi114.stdout: "see_also": [] 2024-04-23T10:41:10.019 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-23T10:41:10.019 INFO:teuthology.orchestra.run.smithi114.stdout: "enabled": { 2024-04-23T10:41:10.019 INFO:teuthology.orchestra.run.smithi114.stdout: "name": "enabled", 2024-04-23T10:41:10.019 INFO:teuthology.orchestra.run.smithi114.stdout: "type": "bool", 2024-04-23T10:41:10.019 INFO:teuthology.orchestra.run.smithi114.stdout: "level": "advanced", 2024-04-23T10:41:10.019 INFO:teuthology.orchestra.run.smithi114.stdout: "flags": 0, 2024-04-23T10:41:10.020 INFO:teuthology.orchestra.run.smithi114.stdout: "default_value": "False", 2024-04-23T10:41:10.020 INFO:teuthology.orchestra.run.smithi114.stdout: "min": "", 2024-04-23T10:41:10.020 INFO:teuthology.orchestra.run.smithi114.stdout: "max": "", 2024-04-23T10:41:10.020 INFO:teuthology.orchestra.run.smithi114.stdout: "enum_allowed": [], 2024-04-23T10:41:10.020 INFO:teuthology.orchestra.run.smithi114.stdout: "desc": "", 2024-04-23T10:41:10.020 INFO:teuthology.orchestra.run.smithi114.stdout: "long_desc": "", 2024-04-23T10:41:10.020 INFO:teuthology.orchestra.run.smithi114.stdout: "tags": [], 2024-04-23T10:41:10.020 INFO:teuthology.orchestra.run.smithi114.stdout: "see_also": [] 2024-04-23T10:41:10.020 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-23T10:41:10.020 INFO:teuthology.orchestra.run.smithi114.stdout: "interval": { 2024-04-23T10:41:10.021 INFO:teuthology.orchestra.run.smithi114.stdout: "name": "interval", 2024-04-23T10:41:10.021 INFO:teuthology.orchestra.run.smithi114.stdout: "type": "int", 2024-04-23T10:41:10.021 INFO:teuthology.orchestra.run.smithi114.stdout: "level": "advanced", 2024-04-23T10:41:10.021 INFO:teuthology.orchestra.run.smithi114.stdout: "flags": 0, 2024-04-23T10:41:10.021 INFO:teuthology.orchestra.run.smithi114.stdout: "default_value": "24", 2024-04-23T10:41:10.021 INFO:teuthology.orchestra.run.smithi114.stdout: "min": "8", 2024-04-23T10:41:10.021 INFO:teuthology.orchestra.run.smithi114.stdout: "max": "", 2024-04-23T10:41:10.021 INFO:teuthology.orchestra.run.smithi114.stdout: "enum_allowed": [], 2024-04-23T10:41:10.021 INFO:teuthology.orchestra.run.smithi114.stdout: "desc": "", 2024-04-23T10:41:10.022 INFO:teuthology.orchestra.run.smithi114.stdout: "long_desc": "", 2024-04-23T10:41:10.022 INFO:teuthology.orchestra.run.smithi114.stdout: "tags": [], 2024-04-23T10:41:10.022 INFO:teuthology.orchestra.run.smithi114.stdout: "see_also": [] 2024-04-23T10:41:10.022 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-23T10:41:10.022 INFO:teuthology.orchestra.run.smithi114.stdout: "last_opt_revision": { 2024-04-23T10:41:10.022 INFO:teuthology.orchestra.run.smithi114.stdout: "name": "last_opt_revision", 2024-04-23T10:41:10.022 INFO:teuthology.orchestra.run.smithi114.stdout: "type": "int", 2024-04-23T10:41:10.022 INFO:teuthology.orchestra.run.smithi114.stdout: "level": "advanced", 2024-04-23T10:41:10.022 INFO:teuthology.orchestra.run.smithi114.stdout: "flags": 0, 2024-04-23T10:41:10.023 INFO:teuthology.orchestra.run.smithi114.stdout: "default_value": "1", 2024-04-23T10:41:10.023 INFO:teuthology.orchestra.run.smithi114.stdout: "min": "", 2024-04-23T10:41:10.023 INFO:teuthology.orchestra.run.smithi114.stdout: "max": "", 2024-04-23T10:41:10.023 INFO:teuthology.orchestra.run.smithi114.stdout: "enum_allowed": [], 2024-04-23T10:41:10.023 INFO:teuthology.orchestra.run.smithi114.stdout: "desc": "", 2024-04-23T10:41:10.023 INFO:teuthology.orchestra.run.smithi114.stdout: "long_desc": "", 2024-04-23T10:41:10.023 INFO:teuthology.orchestra.run.smithi114.stdout: "tags": [], 2024-04-23T10:41:10.023 INFO:teuthology.orchestra.run.smithi114.stdout: "see_also": [] 2024-04-23T10:41:10.023 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-23T10:41:10.023 INFO:teuthology.orchestra.run.smithi114.stdout: "leaderboard": { 2024-04-23T10:41:10.024 INFO:teuthology.orchestra.run.smithi114.stdout: "name": "leaderboard", 2024-04-23T10:41:10.024 INFO:teuthology.orchestra.run.smithi114.stdout: "type": "bool", 2024-04-23T10:41:10.024 INFO:teuthology.orchestra.run.smithi114.stdout: "level": "advanced", 2024-04-23T10:41:10.024 INFO:teuthology.orchestra.run.smithi114.stdout: "flags": 0, 2024-04-23T10:41:10.024 INFO:teuthology.orchestra.run.smithi114.stdout: "default_value": "False", 2024-04-23T10:41:10.024 INFO:teuthology.orchestra.run.smithi114.stdout: "min": "", 2024-04-23T10:41:10.024 INFO:teuthology.orchestra.run.smithi114.stdout: "max": "", 2024-04-23T10:41:10.024 INFO:teuthology.orchestra.run.smithi114.stdout: "enum_allowed": [], 2024-04-23T10:41:10.024 INFO:teuthology.orchestra.run.smithi114.stdout: "desc": "", 2024-04-23T10:41:10.024 INFO:teuthology.orchestra.run.smithi114.stdout: "long_desc": "", 2024-04-23T10:41:10.025 INFO:teuthology.orchestra.run.smithi114.stdout: "tags": [], 2024-04-23T10:41:10.025 INFO:teuthology.orchestra.run.smithi114.stdout: "see_also": [] 2024-04-23T10:41:10.025 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-23T10:41:10.025 INFO:teuthology.orchestra.run.smithi114.stdout: "leaderboard_description": { 2024-04-23T10:41:10.025 INFO:teuthology.orchestra.run.smithi114.stdout: "name": "leaderboard_description", 2024-04-23T10:41:10.025 INFO:teuthology.orchestra.run.smithi114.stdout: "type": "str", 2024-04-23T10:41:10.025 INFO:teuthology.orchestra.run.smithi114.stdout: "level": "advanced", 2024-04-23T10:41:10.025 INFO:teuthology.orchestra.run.smithi114.stdout: "flags": 0, 2024-04-23T10:41:10.025 INFO:teuthology.orchestra.run.smithi114.stdout: "default_value": "", 2024-04-23T10:41:10.025 INFO:teuthology.orchestra.run.smithi114.stdout: "min": "", 2024-04-23T10:41:10.026 INFO:teuthology.orchestra.run.smithi114.stdout: "max": "", 2024-04-23T10:41:10.026 INFO:teuthology.orchestra.run.smithi114.stdout: "enum_allowed": [], 2024-04-23T10:41:10.026 INFO:teuthology.orchestra.run.smithi114.stdout: "desc": "", 2024-04-23T10:41:10.026 INFO:teuthology.orchestra.run.smithi114.stdout: "long_desc": "", 2024-04-23T10:41:10.026 INFO:teuthology.orchestra.run.smithi114.stdout: "tags": [], 2024-04-23T10:41:10.026 INFO:teuthology.orchestra.run.smithi114.stdout: "see_also": [] 2024-04-23T10:41:10.026 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-23T10:41:10.026 INFO:teuthology.orchestra.run.smithi114.stdout: "log_level": { 2024-04-23T10:41:10.026 INFO:teuthology.orchestra.run.smithi114.stdout: "name": "log_level", 2024-04-23T10:41:10.026 INFO:teuthology.orchestra.run.smithi114.stdout: "type": "str", 2024-04-23T10:41:10.027 INFO:teuthology.orchestra.run.smithi114.stdout: "level": "advanced", 2024-04-23T10:41:10.027 INFO:teuthology.orchestra.run.smithi114.stdout: "flags": 1, 2024-04-23T10:41:10.027 INFO:teuthology.orchestra.run.smithi114.stdout: "default_value": "", 2024-04-23T10:41:10.027 INFO:teuthology.orchestra.run.smithi114.stdout: "min": "", 2024-04-23T10:41:10.027 INFO:teuthology.orchestra.run.smithi114.stdout: "max": "", 2024-04-23T10:41:10.027 INFO:teuthology.orchestra.run.smithi114.stdout: "enum_allowed": [ 2024-04-23T10:41:10.027 INFO:teuthology.orchestra.run.smithi114.stdout: "", 2024-04-23T10:41:10.027 INFO:teuthology.orchestra.run.smithi114.stdout: "critical", 2024-04-23T10:41:10.027 INFO:teuthology.orchestra.run.smithi114.stdout: "debug", 2024-04-23T10:41:10.027 INFO:teuthology.orchestra.run.smithi114.stdout: "error", 2024-04-23T10:41:10.028 INFO:teuthology.orchestra.run.smithi114.stdout: "info", 2024-04-23T10:41:10.028 INFO:teuthology.orchestra.run.smithi114.stdout: "warning" 2024-04-23T10:41:10.028 INFO:teuthology.orchestra.run.smithi114.stdout: ], 2024-04-23T10:41:10.028 INFO:teuthology.orchestra.run.smithi114.stdout: "desc": "", 2024-04-23T10:41:10.028 INFO:teuthology.orchestra.run.smithi114.stdout: "long_desc": "", 2024-04-23T10:41:10.028 INFO:teuthology.orchestra.run.smithi114.stdout: "tags": [], 2024-04-23T10:41:10.028 INFO:teuthology.orchestra.run.smithi114.stdout: "see_also": [] 2024-04-23T10:41:10.028 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-23T10:41:10.028 INFO:teuthology.orchestra.run.smithi114.stdout: "log_to_cluster": { 2024-04-23T10:41:10.028 INFO:teuthology.orchestra.run.smithi114.stdout: "name": "log_to_cluster", 2024-04-23T10:41:10.029 INFO:teuthology.orchestra.run.smithi114.stdout: "type": "bool", 2024-04-23T10:41:10.029 INFO:teuthology.orchestra.run.smithi114.stdout: "level": "advanced", 2024-04-23T10:41:10.029 INFO:teuthology.orchestra.run.smithi114.stdout: "flags": 1, 2024-04-23T10:41:10.029 INFO:teuthology.orchestra.run.smithi114.stdout: "default_value": "False", 2024-04-23T10:41:10.029 INFO:teuthology.orchestra.run.smithi114.stdout: "min": "", 2024-04-23T10:41:10.029 INFO:teuthology.orchestra.run.smithi114.stdout: "max": "", 2024-04-23T10:41:10.029 INFO:teuthology.orchestra.run.smithi114.stdout: "enum_allowed": [], 2024-04-23T10:41:10.029 INFO:teuthology.orchestra.run.smithi114.stdout: "desc": "", 2024-04-23T10:41:10.029 INFO:teuthology.orchestra.run.smithi114.stdout: "long_desc": "", 2024-04-23T10:41:10.029 INFO:teuthology.orchestra.run.smithi114.stdout: "tags": [], 2024-04-23T10:41:10.030 INFO:teuthology.orchestra.run.smithi114.stdout: "see_also": [] 2024-04-23T10:41:10.030 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-23T10:41:10.030 INFO:teuthology.orchestra.run.smithi114.stdout: "log_to_cluster_level": { 2024-04-23T10:41:10.030 INFO:teuthology.orchestra.run.smithi114.stdout: "name": "log_to_cluster_level", 2024-04-23T10:41:10.030 INFO:teuthology.orchestra.run.smithi114.stdout: "type": "str", 2024-04-23T10:41:10.030 INFO:teuthology.orchestra.run.smithi114.stdout: "level": "advanced", 2024-04-23T10:41:10.030 INFO:teuthology.orchestra.run.smithi114.stdout: "flags": 1, 2024-04-23T10:41:10.030 INFO:teuthology.orchestra.run.smithi114.stdout: "default_value": "info", 2024-04-23T10:41:10.030 INFO:teuthology.orchestra.run.smithi114.stdout: "min": "", 2024-04-23T10:41:10.031 INFO:teuthology.orchestra.run.smithi114.stdout: "max": "", 2024-04-23T10:41:10.031 INFO:teuthology.orchestra.run.smithi114.stdout: "enum_allowed": [ 2024-04-23T10:41:10.031 INFO:teuthology.orchestra.run.smithi114.stdout: "", 2024-04-23T10:41:10.031 INFO:teuthology.orchestra.run.smithi114.stdout: "critical", 2024-04-23T10:41:10.031 INFO:teuthology.orchestra.run.smithi114.stdout: "debug", 2024-04-23T10:41:10.031 INFO:teuthology.orchestra.run.smithi114.stdout: "error", 2024-04-23T10:41:10.031 INFO:teuthology.orchestra.run.smithi114.stdout: "info", 2024-04-23T10:41:10.031 INFO:teuthology.orchestra.run.smithi114.stdout: "warning" 2024-04-23T10:41:10.031 INFO:teuthology.orchestra.run.smithi114.stdout: ], 2024-04-23T10:41:10.031 INFO:teuthology.orchestra.run.smithi114.stdout: "desc": "", 2024-04-23T10:41:10.032 INFO:teuthology.orchestra.run.smithi114.stdout: "long_desc": "", 2024-04-23T10:41:10.032 INFO:teuthology.orchestra.run.smithi114.stdout: "tags": [], 2024-04-23T10:41:10.032 INFO:teuthology.orchestra.run.smithi114.stdout: "see_also": [] 2024-04-23T10:41:10.032 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-23T10:41:10.032 INFO:teuthology.orchestra.run.smithi114.stdout: "log_to_file": { 2024-04-23T10:41:10.032 INFO:teuthology.orchestra.run.smithi114.stdout: "name": "log_to_file", 2024-04-23T10:41:10.032 INFO:teuthology.orchestra.run.smithi114.stdout: "type": "bool", 2024-04-23T10:41:10.032 INFO:teuthology.orchestra.run.smithi114.stdout: "level": "advanced", 2024-04-23T10:41:10.032 INFO:teuthology.orchestra.run.smithi114.stdout: "flags": 1, 2024-04-23T10:41:10.032 INFO:teuthology.orchestra.run.smithi114.stdout: "default_value": "False", 2024-04-23T10:41:10.033 INFO:teuthology.orchestra.run.smithi114.stdout: "min": "", 2024-04-23T10:41:10.033 INFO:teuthology.orchestra.run.smithi114.stdout: "max": "", 2024-04-23T10:41:10.033 INFO:teuthology.orchestra.run.smithi114.stdout: "enum_allowed": [], 2024-04-23T10:41:10.033 INFO:teuthology.orchestra.run.smithi114.stdout: "desc": "", 2024-04-23T10:41:10.033 INFO:teuthology.orchestra.run.smithi114.stdout: "long_desc": "", 2024-04-23T10:41:10.033 INFO:teuthology.orchestra.run.smithi114.stdout: "tags": [], 2024-04-23T10:41:10.033 INFO:teuthology.orchestra.run.smithi114.stdout: "see_also": [] 2024-04-23T10:41:10.033 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-23T10:41:10.033 INFO:teuthology.orchestra.run.smithi114.stdout: "organization": { 2024-04-23T10:41:10.033 INFO:teuthology.orchestra.run.smithi114.stdout: "name": "organization", 2024-04-23T10:41:10.034 INFO:teuthology.orchestra.run.smithi114.stdout: "type": "str", 2024-04-23T10:41:10.034 INFO:teuthology.orchestra.run.smithi114.stdout: "level": "advanced", 2024-04-23T10:41:10.034 INFO:teuthology.orchestra.run.smithi114.stdout: "flags": 0, 2024-04-23T10:41:10.034 INFO:teuthology.orchestra.run.smithi114.stdout: "default_value": "", 2024-04-23T10:41:10.034 INFO:teuthology.orchestra.run.smithi114.stdout: "min": "", 2024-04-23T10:41:10.034 INFO:teuthology.orchestra.run.smithi114.stdout: "max": "", 2024-04-23T10:41:10.034 INFO:teuthology.orchestra.run.smithi114.stdout: "enum_allowed": [], 2024-04-23T10:41:10.034 INFO:teuthology.orchestra.run.smithi114.stdout: "desc": "", 2024-04-23T10:41:10.034 INFO:teuthology.orchestra.run.smithi114.stdout: "long_desc": "", 2024-04-23T10:41:10.035 INFO:teuthology.orchestra.run.smithi114.stdout: "tags": [], 2024-04-23T10:41:10.035 INFO:teuthology.orchestra.run.smithi114.stdout: "see_also": [] 2024-04-23T10:41:10.035 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-23T10:41:10.035 INFO:teuthology.orchestra.run.smithi114.stdout: "proxy": { 2024-04-23T10:41:10.035 INFO:teuthology.orchestra.run.smithi114.stdout: "name": "proxy", 2024-04-23T10:41:10.035 INFO:teuthology.orchestra.run.smithi114.stdout: "type": "str", 2024-04-23T10:41:10.035 INFO:teuthology.orchestra.run.smithi114.stdout: "level": "advanced", 2024-04-23T10:41:10.035 INFO:teuthology.orchestra.run.smithi114.stdout: "flags": 0, 2024-04-23T10:41:10.035 INFO:teuthology.orchestra.run.smithi114.stdout: "default_value": "", 2024-04-23T10:41:10.035 INFO:teuthology.orchestra.run.smithi114.stdout: "min": "", 2024-04-23T10:41:10.036 INFO:teuthology.orchestra.run.smithi114.stdout: "max": "", 2024-04-23T10:41:10.036 INFO:teuthology.orchestra.run.smithi114.stdout: "enum_allowed": [], 2024-04-23T10:41:10.036 INFO:teuthology.orchestra.run.smithi114.stdout: "desc": "", 2024-04-23T10:41:10.036 INFO:teuthology.orchestra.run.smithi114.stdout: "long_desc": "", 2024-04-23T10:41:10.036 INFO:teuthology.orchestra.run.smithi114.stdout: "tags": [], 2024-04-23T10:41:10.036 INFO:teuthology.orchestra.run.smithi114.stdout: "see_also": [] 2024-04-23T10:41:10.036 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-23T10:41:10.036 INFO:teuthology.orchestra.run.smithi114.stdout: "url": { 2024-04-23T10:41:10.036 INFO:teuthology.orchestra.run.smithi114.stdout: "name": "url", 2024-04-23T10:41:10.036 INFO:teuthology.orchestra.run.smithi114.stdout: "type": "str", 2024-04-23T10:41:10.037 INFO:teuthology.orchestra.run.smithi114.stdout: "level": "advanced", 2024-04-23T10:41:10.037 INFO:teuthology.orchestra.run.smithi114.stdout: "flags": 0, 2024-04-23T10:41:10.037 INFO:teuthology.orchestra.run.smithi114.stdout: "default_value": "https://telemetry.ceph.com/report", 2024-04-23T10:41:10.037 INFO:teuthology.orchestra.run.smithi114.stdout: "min": "", 2024-04-23T10:41:10.037 INFO:teuthology.orchestra.run.smithi114.stdout: "max": "", 2024-04-23T10:41:10.037 INFO:teuthology.orchestra.run.smithi114.stdout: "enum_allowed": [], 2024-04-23T10:41:10.037 INFO:teuthology.orchestra.run.smithi114.stdout: "desc": "", 2024-04-23T10:41:10.037 INFO:teuthology.orchestra.run.smithi114.stdout: "long_desc": "", 2024-04-23T10:41:10.037 INFO:teuthology.orchestra.run.smithi114.stdout: "tags": [], 2024-04-23T10:41:10.037 INFO:teuthology.orchestra.run.smithi114.stdout: "see_also": [] 2024-04-23T10:41:10.038 INFO:teuthology.orchestra.run.smithi114.stdout: } 2024-04-23T10:41:10.038 INFO:teuthology.orchestra.run.smithi114.stdout: } 2024-04-23T10:41:10.038 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-23T10:41:10.038 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-04-23T10:41:10.038 INFO:teuthology.orchestra.run.smithi114.stdout: "name": "test_orchestrator", 2024-04-23T10:41:10.038 INFO:teuthology.orchestra.run.smithi114.stdout: "can_run": true, 2024-04-23T10:41:10.038 INFO:teuthology.orchestra.run.smithi114.stdout: "error_string": "", 2024-04-23T10:41:10.038 INFO:teuthology.orchestra.run.smithi114.stdout: "module_options": { 2024-04-23T10:41:10.038 INFO:teuthology.orchestra.run.smithi114.stdout: "log_level": { 2024-04-23T10:41:10.039 INFO:teuthology.orchestra.run.smithi114.stdout: "name": "log_level", 2024-04-23T10:41:10.039 INFO:teuthology.orchestra.run.smithi114.stdout: "type": "str", 2024-04-23T10:41:10.039 INFO:teuthology.orchestra.run.smithi114.stdout: "level": "advanced", 2024-04-23T10:41:10.039 INFO:teuthology.orchestra.run.smithi114.stdout: "flags": 1, 2024-04-23T10:41:10.039 INFO:teuthology.orchestra.run.smithi114.stdout: "default_value": "", 2024-04-23T10:41:10.039 INFO:teuthology.orchestra.run.smithi114.stdout: "min": "", 2024-04-23T10:41:10.039 INFO:teuthology.orchestra.run.smithi114.stdout: "max": "", 2024-04-23T10:41:10.039 INFO:teuthology.orchestra.run.smithi114.stdout: "enum_allowed": [ 2024-04-23T10:41:10.039 INFO:teuthology.orchestra.run.smithi114.stdout: "", 2024-04-23T10:41:10.039 INFO:teuthology.orchestra.run.smithi114.stdout: "critical", 2024-04-23T10:41:10.040 INFO:teuthology.orchestra.run.smithi114.stdout: "debug", 2024-04-23T10:41:10.040 INFO:teuthology.orchestra.run.smithi114.stdout: "error", 2024-04-23T10:41:10.040 INFO:teuthology.orchestra.run.smithi114.stdout: "info", 2024-04-23T10:41:10.040 INFO:teuthology.orchestra.run.smithi114.stdout: "warning" 2024-04-23T10:41:10.040 INFO:teuthology.orchestra.run.smithi114.stdout: ], 2024-04-23T10:41:10.040 INFO:teuthology.orchestra.run.smithi114.stdout: "desc": "", 2024-04-23T10:41:10.040 INFO:teuthology.orchestra.run.smithi114.stdout: "long_desc": "", 2024-04-23T10:41:10.040 INFO:teuthology.orchestra.run.smithi114.stdout: "tags": [], 2024-04-23T10:41:10.040 INFO:teuthology.orchestra.run.smithi114.stdout: "see_also": [] 2024-04-23T10:41:10.040 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-23T10:41:10.041 INFO:teuthology.orchestra.run.smithi114.stdout: "log_to_cluster": { 2024-04-23T10:41:10.041 INFO:teuthology.orchestra.run.smithi114.stdout: "name": "log_to_cluster", 2024-04-23T10:41:10.041 INFO:teuthology.orchestra.run.smithi114.stdout: "type": "bool", 2024-04-23T10:41:10.041 INFO:teuthology.orchestra.run.smithi114.stdout: "level": "advanced", 2024-04-23T10:41:10.041 INFO:teuthology.orchestra.run.smithi114.stdout: "flags": 1, 2024-04-23T10:41:10.041 INFO:teuthology.orchestra.run.smithi114.stdout: "default_value": "False", 2024-04-23T10:41:10.041 INFO:teuthology.orchestra.run.smithi114.stdout: "min": "", 2024-04-23T10:41:10.041 INFO:teuthology.orchestra.run.smithi114.stdout: "max": "", 2024-04-23T10:41:10.041 INFO:teuthology.orchestra.run.smithi114.stdout: "enum_allowed": [], 2024-04-23T10:41:10.041 INFO:teuthology.orchestra.run.smithi114.stdout: "desc": "", 2024-04-23T10:41:10.042 INFO:teuthology.orchestra.run.smithi114.stdout: "long_desc": "", 2024-04-23T10:41:10.042 INFO:teuthology.orchestra.run.smithi114.stdout: "tags": [], 2024-04-23T10:41:10.042 INFO:teuthology.orchestra.run.smithi114.stdout: "see_also": [] 2024-04-23T10:41:10.042 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-23T10:41:10.042 INFO:teuthology.orchestra.run.smithi114.stdout: "log_to_cluster_level": { 2024-04-23T10:41:10.042 INFO:teuthology.orchestra.run.smithi114.stdout: "name": "log_to_cluster_level", 2024-04-23T10:41:10.042 INFO:teuthology.orchestra.run.smithi114.stdout: "type": "str", 2024-04-23T10:41:10.042 INFO:teuthology.orchestra.run.smithi114.stdout: "level": "advanced", 2024-04-23T10:41:10.042 INFO:teuthology.orchestra.run.smithi114.stdout: "flags": 1, 2024-04-23T10:41:10.043 INFO:teuthology.orchestra.run.smithi114.stdout: "default_value": "info", 2024-04-23T10:41:10.043 INFO:teuthology.orchestra.run.smithi114.stdout: "min": "", 2024-04-23T10:41:10.043 INFO:teuthology.orchestra.run.smithi114.stdout: "max": "", 2024-04-23T10:41:10.043 INFO:teuthology.orchestra.run.smithi114.stdout: "enum_allowed": [ 2024-04-23T10:41:10.043 INFO:teuthology.orchestra.run.smithi114.stdout: "", 2024-04-23T10:41:10.043 INFO:teuthology.orchestra.run.smithi114.stdout: "critical", 2024-04-23T10:41:10.043 INFO:teuthology.orchestra.run.smithi114.stdout: "debug", 2024-04-23T10:41:10.043 INFO:teuthology.orchestra.run.smithi114.stdout: "error", 2024-04-23T10:41:10.043 INFO:teuthology.orchestra.run.smithi114.stdout: "info", 2024-04-23T10:41:10.043 INFO:teuthology.orchestra.run.smithi114.stdout: "warning" 2024-04-23T10:41:10.044 INFO:teuthology.orchestra.run.smithi114.stdout: ], 2024-04-23T10:41:10.044 INFO:teuthology.orchestra.run.smithi114.stdout: "desc": "", 2024-04-23T10:41:10.044 INFO:teuthology.orchestra.run.smithi114.stdout: "long_desc": "", 2024-04-23T10:41:10.044 INFO:teuthology.orchestra.run.smithi114.stdout: "tags": [], 2024-04-23T10:41:10.044 INFO:teuthology.orchestra.run.smithi114.stdout: "see_also": [] 2024-04-23T10:41:10.044 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-23T10:41:10.044 INFO:teuthology.orchestra.run.smithi114.stdout: "log_to_file": { 2024-04-23T10:41:10.044 INFO:teuthology.orchestra.run.smithi114.stdout: "name": "log_to_file", 2024-04-23T10:41:10.044 INFO:teuthology.orchestra.run.smithi114.stdout: "type": "bool", 2024-04-23T10:41:10.044 INFO:teuthology.orchestra.run.smithi114.stdout: "level": "advanced", 2024-04-23T10:41:10.045 INFO:teuthology.orchestra.run.smithi114.stdout: "flags": 1, 2024-04-23T10:41:10.045 INFO:teuthology.orchestra.run.smithi114.stdout: "default_value": "False", 2024-04-23T10:41:10.045 INFO:teuthology.orchestra.run.smithi114.stdout: "min": "", 2024-04-23T10:41:10.045 INFO:teuthology.orchestra.run.smithi114.stdout: "max": "", 2024-04-23T10:41:10.045 INFO:teuthology.orchestra.run.smithi114.stdout: "enum_allowed": [], 2024-04-23T10:41:10.045 INFO:teuthology.orchestra.run.smithi114.stdout: "desc": "", 2024-04-23T10:41:10.045 INFO:teuthology.orchestra.run.smithi114.stdout: "long_desc": "", 2024-04-23T10:41:10.045 INFO:teuthology.orchestra.run.smithi114.stdout: "tags": [], 2024-04-23T10:41:10.045 INFO:teuthology.orchestra.run.smithi114.stdout: "see_also": [] 2024-04-23T10:41:10.045 INFO:teuthology.orchestra.run.smithi114.stdout: } 2024-04-23T10:41:10.046 INFO:teuthology.orchestra.run.smithi114.stdout: } 2024-04-23T10:41:10.046 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-23T10:41:10.046 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-04-23T10:41:10.046 INFO:teuthology.orchestra.run.smithi114.stdout: "name": "volumes", 2024-04-23T10:41:10.046 INFO:teuthology.orchestra.run.smithi114.stdout: "can_run": true, 2024-04-23T10:41:10.046 INFO:teuthology.orchestra.run.smithi114.stdout: "error_string": "", 2024-04-23T10:41:10.046 INFO:teuthology.orchestra.run.smithi114.stdout: "module_options": { 2024-04-23T10:41:10.046 INFO:teuthology.orchestra.run.smithi114.stdout: "log_level": { 2024-04-23T10:41:10.046 INFO:teuthology.orchestra.run.smithi114.stdout: "name": "log_level", 2024-04-23T10:41:10.047 INFO:teuthology.orchestra.run.smithi114.stdout: "type": "str", 2024-04-23T10:41:10.047 INFO:teuthology.orchestra.run.smithi114.stdout: "level": "advanced", 2024-04-23T10:41:10.047 INFO:teuthology.orchestra.run.smithi114.stdout: "flags": 1, 2024-04-23T10:41:10.047 INFO:teuthology.orchestra.run.smithi114.stdout: "default_value": "", 2024-04-23T10:41:10.047 INFO:teuthology.orchestra.run.smithi114.stdout: "min": "", 2024-04-23T10:41:10.047 INFO:teuthology.orchestra.run.smithi114.stdout: "max": "", 2024-04-23T10:41:10.047 INFO:teuthology.orchestra.run.smithi114.stdout: "enum_allowed": [ 2024-04-23T10:41:10.047 INFO:teuthology.orchestra.run.smithi114.stdout: "", 2024-04-23T10:41:10.047 INFO:teuthology.orchestra.run.smithi114.stdout: "critical", 2024-04-23T10:41:10.047 INFO:teuthology.orchestra.run.smithi114.stdout: "debug", 2024-04-23T10:41:10.048 INFO:teuthology.orchestra.run.smithi114.stdout: "error", 2024-04-23T10:41:10.048 INFO:teuthology.orchestra.run.smithi114.stdout: "info", 2024-04-23T10:41:10.048 INFO:teuthology.orchestra.run.smithi114.stdout: "warning" 2024-04-23T10:41:10.048 INFO:teuthology.orchestra.run.smithi114.stdout: ], 2024-04-23T10:41:10.048 INFO:teuthology.orchestra.run.smithi114.stdout: "desc": "", 2024-04-23T10:41:10.048 INFO:teuthology.orchestra.run.smithi114.stdout: "long_desc": "", 2024-04-23T10:41:10.048 INFO:teuthology.orchestra.run.smithi114.stdout: "tags": [], 2024-04-23T10:41:10.048 INFO:teuthology.orchestra.run.smithi114.stdout: "see_also": [] 2024-04-23T10:41:10.048 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-23T10:41:10.049 INFO:teuthology.orchestra.run.smithi114.stdout: "log_to_cluster": { 2024-04-23T10:41:10.049 INFO:teuthology.orchestra.run.smithi114.stdout: "name": "log_to_cluster", 2024-04-23T10:41:10.049 INFO:teuthology.orchestra.run.smithi114.stdout: "type": "bool", 2024-04-23T10:41:10.049 INFO:teuthology.orchestra.run.smithi114.stdout: "level": "advanced", 2024-04-23T10:41:10.049 INFO:teuthology.orchestra.run.smithi114.stdout: "flags": 1, 2024-04-23T10:41:10.049 INFO:teuthology.orchestra.run.smithi114.stdout: "default_value": "False", 2024-04-23T10:41:10.049 INFO:teuthology.orchestra.run.smithi114.stdout: "min": "", 2024-04-23T10:41:10.049 INFO:teuthology.orchestra.run.smithi114.stdout: "max": "", 2024-04-23T10:41:10.049 INFO:teuthology.orchestra.run.smithi114.stdout: "enum_allowed": [], 2024-04-23T10:41:10.049 INFO:teuthology.orchestra.run.smithi114.stdout: "desc": "", 2024-04-23T10:41:10.050 INFO:teuthology.orchestra.run.smithi114.stdout: "long_desc": "", 2024-04-23T10:41:10.050 INFO:teuthology.orchestra.run.smithi114.stdout: "tags": [], 2024-04-23T10:41:10.050 INFO:teuthology.orchestra.run.smithi114.stdout: "see_also": [] 2024-04-23T10:41:10.050 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-23T10:41:10.050 INFO:teuthology.orchestra.run.smithi114.stdout: "log_to_cluster_level": { 2024-04-23T10:41:10.050 INFO:teuthology.orchestra.run.smithi114.stdout: "name": "log_to_cluster_level", 2024-04-23T10:41:10.050 INFO:teuthology.orchestra.run.smithi114.stdout: "type": "str", 2024-04-23T10:41:10.050 INFO:teuthology.orchestra.run.smithi114.stdout: "level": "advanced", 2024-04-23T10:41:10.051 INFO:teuthology.orchestra.run.smithi114.stdout: "flags": 1, 2024-04-23T10:41:10.051 INFO:teuthology.orchestra.run.smithi114.stdout: "default_value": "info", 2024-04-23T10:41:10.051 INFO:teuthology.orchestra.run.smithi114.stdout: "min": "", 2024-04-23T10:41:10.051 INFO:teuthology.orchestra.run.smithi114.stdout: "max": "", 2024-04-23T10:41:10.051 INFO:teuthology.orchestra.run.smithi114.stdout: "enum_allowed": [ 2024-04-23T10:41:10.051 INFO:teuthology.orchestra.run.smithi114.stdout: "", 2024-04-23T10:41:10.051 INFO:teuthology.orchestra.run.smithi114.stdout: "critical", 2024-04-23T10:41:10.051 INFO:teuthology.orchestra.run.smithi114.stdout: "debug", 2024-04-23T10:41:10.051 INFO:teuthology.orchestra.run.smithi114.stdout: "error", 2024-04-23T10:41:10.051 INFO:teuthology.orchestra.run.smithi114.stdout: "info", 2024-04-23T10:41:10.052 INFO:teuthology.orchestra.run.smithi114.stdout: "warning" 2024-04-23T10:41:10.052 INFO:teuthology.orchestra.run.smithi114.stdout: ], 2024-04-23T10:41:10.052 INFO:teuthology.orchestra.run.smithi114.stdout: "desc": "", 2024-04-23T10:41:10.052 INFO:teuthology.orchestra.run.smithi114.stdout: "long_desc": "", 2024-04-23T10:41:10.052 INFO:teuthology.orchestra.run.smithi114.stdout: "tags": [], 2024-04-23T10:41:10.052 INFO:teuthology.orchestra.run.smithi114.stdout: "see_also": [] 2024-04-23T10:41:10.052 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-23T10:41:10.052 INFO:teuthology.orchestra.run.smithi114.stdout: "log_to_file": { 2024-04-23T10:41:10.052 INFO:teuthology.orchestra.run.smithi114.stdout: "name": "log_to_file", 2024-04-23T10:41:10.052 INFO:teuthology.orchestra.run.smithi114.stdout: "type": "bool", 2024-04-23T10:41:10.053 INFO:teuthology.orchestra.run.smithi114.stdout: "level": "advanced", 2024-04-23T10:41:10.053 INFO:teuthology.orchestra.run.smithi114.stdout: "flags": 1, 2024-04-23T10:41:10.053 INFO:teuthology.orchestra.run.smithi114.stdout: "default_value": "False", 2024-04-23T10:41:10.053 INFO:teuthology.orchestra.run.smithi114.stdout: "min": "", 2024-04-23T10:41:10.053 INFO:teuthology.orchestra.run.smithi114.stdout: "max": "", 2024-04-23T10:41:10.053 INFO:teuthology.orchestra.run.smithi114.stdout: "enum_allowed": [], 2024-04-23T10:41:10.053 INFO:teuthology.orchestra.run.smithi114.stdout: "desc": "", 2024-04-23T10:41:10.053 INFO:teuthology.orchestra.run.smithi114.stdout: "long_desc": "", 2024-04-23T10:41:10.053 INFO:teuthology.orchestra.run.smithi114.stdout: "tags": [], 2024-04-23T10:41:10.053 INFO:teuthology.orchestra.run.smithi114.stdout: "see_also": [] 2024-04-23T10:41:10.054 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-23T10:41:10.054 INFO:teuthology.orchestra.run.smithi114.stdout: "max_concurrent_clones": { 2024-04-23T10:41:10.054 INFO:teuthology.orchestra.run.smithi114.stdout: "name": "max_concurrent_clones", 2024-04-23T10:41:10.054 INFO:teuthology.orchestra.run.smithi114.stdout: "type": "int", 2024-04-23T10:41:10.054 INFO:teuthology.orchestra.run.smithi114.stdout: "level": "advanced", 2024-04-23T10:41:10.054 INFO:teuthology.orchestra.run.smithi114.stdout: "flags": 0, 2024-04-23T10:41:10.054 INFO:teuthology.orchestra.run.smithi114.stdout: "default_value": "4", 2024-04-23T10:41:10.054 INFO:teuthology.orchestra.run.smithi114.stdout: "min": "", 2024-04-23T10:41:10.054 INFO:teuthology.orchestra.run.smithi114.stdout: "max": "", 2024-04-23T10:41:10.055 INFO:teuthology.orchestra.run.smithi114.stdout: "enum_allowed": [], 2024-04-23T10:41:10.055 INFO:teuthology.orchestra.run.smithi114.stdout: "desc": "Number of asynchronous cloner threads", 2024-04-23T10:41:10.055 INFO:teuthology.orchestra.run.smithi114.stdout: "long_desc": "", 2024-04-23T10:41:10.055 INFO:teuthology.orchestra.run.smithi114.stdout: "tags": [], 2024-04-23T10:41:10.055 INFO:teuthology.orchestra.run.smithi114.stdout: "see_also": [] 2024-04-23T10:41:10.055 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-23T10:41:10.055 INFO:teuthology.orchestra.run.smithi114.stdout: "periodic_async_work": { 2024-04-23T10:41:10.055 INFO:teuthology.orchestra.run.smithi114.stdout: "name": "periodic_async_work", 2024-04-23T10:41:10.055 INFO:teuthology.orchestra.run.smithi114.stdout: "type": "bool", 2024-04-23T10:41:10.055 INFO:teuthology.orchestra.run.smithi114.stdout: "level": "advanced", 2024-04-23T10:41:10.056 INFO:teuthology.orchestra.run.smithi114.stdout: "flags": 0, 2024-04-23T10:41:10.056 INFO:teuthology.orchestra.run.smithi114.stdout: "default_value": "False", 2024-04-23T10:41:10.056 INFO:teuthology.orchestra.run.smithi114.stdout: "min": "", 2024-04-23T10:41:10.056 INFO:teuthology.orchestra.run.smithi114.stdout: "max": "", 2024-04-23T10:41:10.056 INFO:teuthology.orchestra.run.smithi114.stdout: "enum_allowed": [], 2024-04-23T10:41:10.056 INFO:teuthology.orchestra.run.smithi114.stdout: "desc": "Periodically check for async work", 2024-04-23T10:41:10.056 INFO:teuthology.orchestra.run.smithi114.stdout: "long_desc": "", 2024-04-23T10:41:10.056 INFO:teuthology.orchestra.run.smithi114.stdout: "tags": [], 2024-04-23T10:41:10.056 INFO:teuthology.orchestra.run.smithi114.stdout: "see_also": [] 2024-04-23T10:41:10.056 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-23T10:41:10.057 INFO:teuthology.orchestra.run.smithi114.stdout: "snapshot_clone_delay": { 2024-04-23T10:41:10.057 INFO:teuthology.orchestra.run.smithi114.stdout: "name": "snapshot_clone_delay", 2024-04-23T10:41:10.057 INFO:teuthology.orchestra.run.smithi114.stdout: "type": "int", 2024-04-23T10:41:10.057 INFO:teuthology.orchestra.run.smithi114.stdout: "level": "advanced", 2024-04-23T10:41:10.057 INFO:teuthology.orchestra.run.smithi114.stdout: "flags": 0, 2024-04-23T10:41:10.057 INFO:teuthology.orchestra.run.smithi114.stdout: "default_value": "0", 2024-04-23T10:41:10.057 INFO:teuthology.orchestra.run.smithi114.stdout: "min": "", 2024-04-23T10:41:10.057 INFO:teuthology.orchestra.run.smithi114.stdout: "max": "", 2024-04-23T10:41:10.057 INFO:teuthology.orchestra.run.smithi114.stdout: "enum_allowed": [], 2024-04-23T10:41:10.057 INFO:teuthology.orchestra.run.smithi114.stdout: "desc": "Delay clone begin operation by snapshot_clone_delay seconds", 2024-04-23T10:41:10.058 INFO:teuthology.orchestra.run.smithi114.stdout: "long_desc": "", 2024-04-23T10:41:10.058 INFO:teuthology.orchestra.run.smithi114.stdout: "tags": [], 2024-04-23T10:41:10.058 INFO:teuthology.orchestra.run.smithi114.stdout: "see_also": [] 2024-04-23T10:41:10.058 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-23T10:41:10.058 INFO:teuthology.orchestra.run.smithi114.stdout: "snapshot_clone_no_wait": { 2024-04-23T10:41:10.058 INFO:teuthology.orchestra.run.smithi114.stdout: "name": "snapshot_clone_no_wait", 2024-04-23T10:41:10.058 INFO:teuthology.orchestra.run.smithi114.stdout: "type": "bool", 2024-04-23T10:41:10.058 INFO:teuthology.orchestra.run.smithi114.stdout: "level": "advanced", 2024-04-23T10:41:10.058 INFO:teuthology.orchestra.run.smithi114.stdout: "flags": 0, 2024-04-23T10:41:10.059 INFO:teuthology.orchestra.run.smithi114.stdout: "default_value": "True", 2024-04-23T10:41:10.059 INFO:teuthology.orchestra.run.smithi114.stdout: "min": "", 2024-04-23T10:41:10.059 INFO:teuthology.orchestra.run.smithi114.stdout: "max": "", 2024-04-23T10:41:10.059 INFO:teuthology.orchestra.run.smithi114.stdout: "enum_allowed": [], 2024-04-23T10:41:10.059 INFO:teuthology.orchestra.run.smithi114.stdout: "desc": "Reject subvolume clone request when cloner threads are busy", 2024-04-23T10:41:10.059 INFO:teuthology.orchestra.run.smithi114.stdout: "long_desc": "", 2024-04-23T10:41:10.059 INFO:teuthology.orchestra.run.smithi114.stdout: "tags": [], 2024-04-23T10:41:10.059 INFO:teuthology.orchestra.run.smithi114.stdout: "see_also": [] 2024-04-23T10:41:10.059 INFO:teuthology.orchestra.run.smithi114.stdout: } 2024-04-23T10:41:10.059 INFO:teuthology.orchestra.run.smithi114.stdout: } 2024-04-23T10:41:10.060 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-23T10:41:10.060 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-04-23T10:41:10.060 INFO:teuthology.orchestra.run.smithi114.stdout: "name": "zabbix", 2024-04-23T10:41:10.060 INFO:teuthology.orchestra.run.smithi114.stdout: "can_run": true, 2024-04-23T10:41:10.060 INFO:teuthology.orchestra.run.smithi114.stdout: "error_string": "", 2024-04-23T10:41:10.060 INFO:teuthology.orchestra.run.smithi114.stdout: "module_options": { 2024-04-23T10:41:10.060 INFO:teuthology.orchestra.run.smithi114.stdout: "discovery_interval": { 2024-04-23T10:41:10.060 INFO:teuthology.orchestra.run.smithi114.stdout: "name": "discovery_interval", 2024-04-23T10:41:10.060 INFO:teuthology.orchestra.run.smithi114.stdout: "type": "uint", 2024-04-23T10:41:10.060 INFO:teuthology.orchestra.run.smithi114.stdout: "level": "advanced", 2024-04-23T10:41:10.061 INFO:teuthology.orchestra.run.smithi114.stdout: "flags": 0, 2024-04-23T10:41:10.061 INFO:teuthology.orchestra.run.smithi114.stdout: "default_value": "100", 2024-04-23T10:41:10.061 INFO:teuthology.orchestra.run.smithi114.stdout: "min": "", 2024-04-23T10:41:10.061 INFO:teuthology.orchestra.run.smithi114.stdout: "max": "", 2024-04-23T10:41:10.061 INFO:teuthology.orchestra.run.smithi114.stdout: "enum_allowed": [], 2024-04-23T10:41:10.061 INFO:teuthology.orchestra.run.smithi114.stdout: "desc": "", 2024-04-23T10:41:10.061 INFO:teuthology.orchestra.run.smithi114.stdout: "long_desc": "", 2024-04-23T10:41:10.061 INFO:teuthology.orchestra.run.smithi114.stdout: "tags": [], 2024-04-23T10:41:10.061 INFO:teuthology.orchestra.run.smithi114.stdout: "see_also": [] 2024-04-23T10:41:10.062 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-23T10:41:10.062 INFO:teuthology.orchestra.run.smithi114.stdout: "identifier": { 2024-04-23T10:41:10.062 INFO:teuthology.orchestra.run.smithi114.stdout: "name": "identifier", 2024-04-23T10:41:10.062 INFO:teuthology.orchestra.run.smithi114.stdout: "type": "str", 2024-04-23T10:41:10.062 INFO:teuthology.orchestra.run.smithi114.stdout: "level": "advanced", 2024-04-23T10:41:10.062 INFO:teuthology.orchestra.run.smithi114.stdout: "flags": 0, 2024-04-23T10:41:10.062 INFO:teuthology.orchestra.run.smithi114.stdout: "default_value": "", 2024-04-23T10:41:10.062 INFO:teuthology.orchestra.run.smithi114.stdout: "min": "", 2024-04-23T10:41:10.062 INFO:teuthology.orchestra.run.smithi114.stdout: "max": "", 2024-04-23T10:41:10.063 INFO:teuthology.orchestra.run.smithi114.stdout: "enum_allowed": [], 2024-04-23T10:41:10.063 INFO:teuthology.orchestra.run.smithi114.stdout: "desc": "", 2024-04-23T10:41:10.063 INFO:teuthology.orchestra.run.smithi114.stdout: "long_desc": "", 2024-04-23T10:41:10.063 INFO:teuthology.orchestra.run.smithi114.stdout: "tags": [], 2024-04-23T10:41:10.063 INFO:teuthology.orchestra.run.smithi114.stdout: "see_also": [] 2024-04-23T10:41:10.063 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-23T10:41:10.063 INFO:teuthology.orchestra.run.smithi114.stdout: "interval": { 2024-04-23T10:41:10.063 INFO:teuthology.orchestra.run.smithi114.stdout: "name": "interval", 2024-04-23T10:41:10.063 INFO:teuthology.orchestra.run.smithi114.stdout: "type": "secs", 2024-04-23T10:41:10.063 INFO:teuthology.orchestra.run.smithi114.stdout: "level": "advanced", 2024-04-23T10:41:10.064 INFO:teuthology.orchestra.run.smithi114.stdout: "flags": 0, 2024-04-23T10:41:10.064 INFO:teuthology.orchestra.run.smithi114.stdout: "default_value": "60", 2024-04-23T10:41:10.064 INFO:teuthology.orchestra.run.smithi114.stdout: "min": "", 2024-04-23T10:41:10.064 INFO:teuthology.orchestra.run.smithi114.stdout: "max": "", 2024-04-23T10:41:10.064 INFO:teuthology.orchestra.run.smithi114.stdout: "enum_allowed": [], 2024-04-23T10:41:10.064 INFO:teuthology.orchestra.run.smithi114.stdout: "desc": "", 2024-04-23T10:41:10.064 INFO:teuthology.orchestra.run.smithi114.stdout: "long_desc": "", 2024-04-23T10:41:10.064 INFO:teuthology.orchestra.run.smithi114.stdout: "tags": [], 2024-04-23T10:41:10.064 INFO:teuthology.orchestra.run.smithi114.stdout: "see_also": [] 2024-04-23T10:41:10.064 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-23T10:41:10.065 INFO:teuthology.orchestra.run.smithi114.stdout: "log_level": { 2024-04-23T10:41:10.065 INFO:teuthology.orchestra.run.smithi114.stdout: "name": "log_level", 2024-04-23T10:41:10.065 INFO:teuthology.orchestra.run.smithi114.stdout: "type": "str", 2024-04-23T10:41:10.065 INFO:teuthology.orchestra.run.smithi114.stdout: "level": "advanced", 2024-04-23T10:41:10.065 INFO:teuthology.orchestra.run.smithi114.stdout: "flags": 1, 2024-04-23T10:41:10.065 INFO:teuthology.orchestra.run.smithi114.stdout: "default_value": "", 2024-04-23T10:41:10.065 INFO:teuthology.orchestra.run.smithi114.stdout: "min": "", 2024-04-23T10:41:10.065 INFO:teuthology.orchestra.run.smithi114.stdout: "max": "", 2024-04-23T10:41:10.065 INFO:teuthology.orchestra.run.smithi114.stdout: "enum_allowed": [ 2024-04-23T10:41:10.065 INFO:teuthology.orchestra.run.smithi114.stdout: "", 2024-04-23T10:41:10.066 INFO:teuthology.orchestra.run.smithi114.stdout: "critical", 2024-04-23T10:41:10.066 INFO:teuthology.orchestra.run.smithi114.stdout: "debug", 2024-04-23T10:41:10.066 INFO:teuthology.orchestra.run.smithi114.stdout: "error", 2024-04-23T10:41:10.066 INFO:teuthology.orchestra.run.smithi114.stdout: "info", 2024-04-23T10:41:10.066 INFO:teuthology.orchestra.run.smithi114.stdout: "warning" 2024-04-23T10:41:10.066 INFO:teuthology.orchestra.run.smithi114.stdout: ], 2024-04-23T10:41:10.066 INFO:teuthology.orchestra.run.smithi114.stdout: "desc": "", 2024-04-23T10:41:10.066 INFO:teuthology.orchestra.run.smithi114.stdout: "long_desc": "", 2024-04-23T10:41:10.066 INFO:teuthology.orchestra.run.smithi114.stdout: "tags": [], 2024-04-23T10:41:10.066 INFO:teuthology.orchestra.run.smithi114.stdout: "see_also": [] 2024-04-23T10:41:10.067 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-23T10:41:10.067 INFO:teuthology.orchestra.run.smithi114.stdout: "log_to_cluster": { 2024-04-23T10:41:10.067 INFO:teuthology.orchestra.run.smithi114.stdout: "name": "log_to_cluster", 2024-04-23T10:41:10.067 INFO:teuthology.orchestra.run.smithi114.stdout: "type": "bool", 2024-04-23T10:41:10.067 INFO:teuthology.orchestra.run.smithi114.stdout: "level": "advanced", 2024-04-23T10:41:10.067 INFO:teuthology.orchestra.run.smithi114.stdout: "flags": 1, 2024-04-23T10:41:10.067 INFO:teuthology.orchestra.run.smithi114.stdout: "default_value": "False", 2024-04-23T10:41:10.067 INFO:teuthology.orchestra.run.smithi114.stdout: "min": "", 2024-04-23T10:41:10.067 INFO:teuthology.orchestra.run.smithi114.stdout: "max": "", 2024-04-23T10:41:10.067 INFO:teuthology.orchestra.run.smithi114.stdout: "enum_allowed": [], 2024-04-23T10:41:10.068 INFO:teuthology.orchestra.run.smithi114.stdout: "desc": "", 2024-04-23T10:41:10.068 INFO:teuthology.orchestra.run.smithi114.stdout: "long_desc": "", 2024-04-23T10:41:10.068 INFO:teuthology.orchestra.run.smithi114.stdout: "tags": [], 2024-04-23T10:41:10.068 INFO:teuthology.orchestra.run.smithi114.stdout: "see_also": [] 2024-04-23T10:41:10.068 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-23T10:41:10.068 INFO:teuthology.orchestra.run.smithi114.stdout: "log_to_cluster_level": { 2024-04-23T10:41:10.068 INFO:teuthology.orchestra.run.smithi114.stdout: "name": "log_to_cluster_level", 2024-04-23T10:41:10.068 INFO:teuthology.orchestra.run.smithi114.stdout: "type": "str", 2024-04-23T10:41:10.068 INFO:teuthology.orchestra.run.smithi114.stdout: "level": "advanced", 2024-04-23T10:41:10.068 INFO:teuthology.orchestra.run.smithi114.stdout: "flags": 1, 2024-04-23T10:41:10.069 INFO:teuthology.orchestra.run.smithi114.stdout: "default_value": "info", 2024-04-23T10:41:10.069 INFO:teuthology.orchestra.run.smithi114.stdout: "min": "", 2024-04-23T10:41:10.069 INFO:teuthology.orchestra.run.smithi114.stdout: "max": "", 2024-04-23T10:41:10.069 INFO:teuthology.orchestra.run.smithi114.stdout: "enum_allowed": [ 2024-04-23T10:41:10.069 INFO:teuthology.orchestra.run.smithi114.stdout: "", 2024-04-23T10:41:10.069 INFO:teuthology.orchestra.run.smithi114.stdout: "critical", 2024-04-23T10:41:10.069 INFO:teuthology.orchestra.run.smithi114.stdout: "debug", 2024-04-23T10:41:10.069 INFO:teuthology.orchestra.run.smithi114.stdout: "error", 2024-04-23T10:41:10.069 INFO:teuthology.orchestra.run.smithi114.stdout: "info", 2024-04-23T10:41:10.069 INFO:teuthology.orchestra.run.smithi114.stdout: "warning" 2024-04-23T10:41:10.070 INFO:teuthology.orchestra.run.smithi114.stdout: ], 2024-04-23T10:41:10.070 INFO:teuthology.orchestra.run.smithi114.stdout: "desc": "", 2024-04-23T10:41:10.070 INFO:teuthology.orchestra.run.smithi114.stdout: "long_desc": "", 2024-04-23T10:41:10.070 INFO:teuthology.orchestra.run.smithi114.stdout: "tags": [], 2024-04-23T10:41:10.070 INFO:teuthology.orchestra.run.smithi114.stdout: "see_also": [] 2024-04-23T10:41:10.070 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-23T10:41:10.070 INFO:teuthology.orchestra.run.smithi114.stdout: "log_to_file": { 2024-04-23T10:41:10.070 INFO:teuthology.orchestra.run.smithi114.stdout: "name": "log_to_file", 2024-04-23T10:41:10.070 INFO:teuthology.orchestra.run.smithi114.stdout: "type": "bool", 2024-04-23T10:41:10.071 INFO:teuthology.orchestra.run.smithi114.stdout: "level": "advanced", 2024-04-23T10:41:10.071 INFO:teuthology.orchestra.run.smithi114.stdout: "flags": 1, 2024-04-23T10:41:10.071 INFO:teuthology.orchestra.run.smithi114.stdout: "default_value": "False", 2024-04-23T10:41:10.071 INFO:teuthology.orchestra.run.smithi114.stdout: "min": "", 2024-04-23T10:41:10.071 INFO:teuthology.orchestra.run.smithi114.stdout: "max": "", 2024-04-23T10:41:10.071 INFO:teuthology.orchestra.run.smithi114.stdout: "enum_allowed": [], 2024-04-23T10:41:10.071 INFO:teuthology.orchestra.run.smithi114.stdout: "desc": "", 2024-04-23T10:41:10.071 INFO:teuthology.orchestra.run.smithi114.stdout: "long_desc": "", 2024-04-23T10:41:10.071 INFO:teuthology.orchestra.run.smithi114.stdout: "tags": [], 2024-04-23T10:41:10.071 INFO:teuthology.orchestra.run.smithi114.stdout: "see_also": [] 2024-04-23T10:41:10.072 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-23T10:41:10.072 INFO:teuthology.orchestra.run.smithi114.stdout: "zabbix_host": { 2024-04-23T10:41:10.072 INFO:teuthology.orchestra.run.smithi114.stdout: "name": "zabbix_host", 2024-04-23T10:41:10.072 INFO:teuthology.orchestra.run.smithi114.stdout: "type": "str", 2024-04-23T10:41:10.072 INFO:teuthology.orchestra.run.smithi114.stdout: "level": "advanced", 2024-04-23T10:41:10.072 INFO:teuthology.orchestra.run.smithi114.stdout: "flags": 0, 2024-04-23T10:41:10.072 INFO:teuthology.orchestra.run.smithi114.stdout: "default_value": "", 2024-04-23T10:41:10.072 INFO:teuthology.orchestra.run.smithi114.stdout: "min": "", 2024-04-23T10:41:10.072 INFO:teuthology.orchestra.run.smithi114.stdout: "max": "", 2024-04-23T10:41:10.072 INFO:teuthology.orchestra.run.smithi114.stdout: "enum_allowed": [], 2024-04-23T10:41:10.073 INFO:teuthology.orchestra.run.smithi114.stdout: "desc": "", 2024-04-23T10:41:10.073 INFO:teuthology.orchestra.run.smithi114.stdout: "long_desc": "", 2024-04-23T10:41:10.073 INFO:teuthology.orchestra.run.smithi114.stdout: "tags": [], 2024-04-23T10:41:10.073 INFO:teuthology.orchestra.run.smithi114.stdout: "see_also": [] 2024-04-23T10:41:10.073 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-23T10:41:10.073 INFO:teuthology.orchestra.run.smithi114.stdout: "zabbix_port": { 2024-04-23T10:41:10.073 INFO:teuthology.orchestra.run.smithi114.stdout: "name": "zabbix_port", 2024-04-23T10:41:10.073 INFO:teuthology.orchestra.run.smithi114.stdout: "type": "int", 2024-04-23T10:41:10.073 INFO:teuthology.orchestra.run.smithi114.stdout: "level": "advanced", 2024-04-23T10:41:10.073 INFO:teuthology.orchestra.run.smithi114.stdout: "flags": 0, 2024-04-23T10:41:10.074 INFO:teuthology.orchestra.run.smithi114.stdout: "default_value": "10051", 2024-04-23T10:41:10.074 INFO:teuthology.orchestra.run.smithi114.stdout: "min": "", 2024-04-23T10:41:10.074 INFO:teuthology.orchestra.run.smithi114.stdout: "max": "", 2024-04-23T10:41:10.074 INFO:teuthology.orchestra.run.smithi114.stdout: "enum_allowed": [], 2024-04-23T10:41:10.074 INFO:teuthology.orchestra.run.smithi114.stdout: "desc": "", 2024-04-23T10:41:10.074 INFO:teuthology.orchestra.run.smithi114.stdout: "long_desc": "", 2024-04-23T10:41:10.074 INFO:teuthology.orchestra.run.smithi114.stdout: "tags": [], 2024-04-23T10:41:10.074 INFO:teuthology.orchestra.run.smithi114.stdout: "see_also": [] 2024-04-23T10:41:10.074 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-23T10:41:10.075 INFO:teuthology.orchestra.run.smithi114.stdout: "zabbix_sender": { 2024-04-23T10:41:10.075 INFO:teuthology.orchestra.run.smithi114.stdout: "name": "zabbix_sender", 2024-04-23T10:41:10.075 INFO:teuthology.orchestra.run.smithi114.stdout: "type": "str", 2024-04-23T10:41:10.075 INFO:teuthology.orchestra.run.smithi114.stdout: "level": "advanced", 2024-04-23T10:41:10.075 INFO:teuthology.orchestra.run.smithi114.stdout: "flags": 0, 2024-04-23T10:41:10.075 INFO:teuthology.orchestra.run.smithi114.stdout: "default_value": "/usr/bin/zabbix_sender", 2024-04-23T10:41:10.075 INFO:teuthology.orchestra.run.smithi114.stdout: "min": "", 2024-04-23T10:41:10.075 INFO:teuthology.orchestra.run.smithi114.stdout: "max": "", 2024-04-23T10:41:10.075 INFO:teuthology.orchestra.run.smithi114.stdout: "enum_allowed": [], 2024-04-23T10:41:10.075 INFO:teuthology.orchestra.run.smithi114.stdout: "desc": "", 2024-04-23T10:41:10.076 INFO:teuthology.orchestra.run.smithi114.stdout: "long_desc": "", 2024-04-23T10:41:10.076 INFO:teuthology.orchestra.run.smithi114.stdout: "tags": [], 2024-04-23T10:41:10.076 INFO:teuthology.orchestra.run.smithi114.stdout: "see_also": [] 2024-04-23T10:41:10.076 INFO:teuthology.orchestra.run.smithi114.stdout: } 2024-04-23T10:41:10.076 INFO:teuthology.orchestra.run.smithi114.stdout: } 2024-04-23T10:41:10.076 INFO:teuthology.orchestra.run.smithi114.stdout: } 2024-04-23T10:41:10.076 INFO:teuthology.orchestra.run.smithi114.stdout: ], 2024-04-23T10:41:10.076 INFO:teuthology.orchestra.run.smithi114.stdout: "services": { 2024-04-23T10:41:10.076 INFO:teuthology.orchestra.run.smithi114.stdout: "dashboard": "https://172.21.15.114:8443/" 2024-04-23T10:41:10.076 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-23T10:41:10.077 INFO:teuthology.orchestra.run.smithi114.stdout: "always_on_modules": { 2024-04-23T10:41:10.077 INFO:teuthology.orchestra.run.smithi114.stdout: "octopus": [ 2024-04-23T10:41:10.077 INFO:teuthology.orchestra.run.smithi114.stdout: "balancer", 2024-04-23T10:41:10.077 INFO:teuthology.orchestra.run.smithi114.stdout: "crash", 2024-04-23T10:41:10.077 INFO:teuthology.orchestra.run.smithi114.stdout: "devicehealth", 2024-04-23T10:41:10.077 INFO:teuthology.orchestra.run.smithi114.stdout: "orchestrator", 2024-04-23T10:41:10.077 INFO:teuthology.orchestra.run.smithi114.stdout: "pg_autoscaler", 2024-04-23T10:41:10.077 INFO:teuthology.orchestra.run.smithi114.stdout: "progress", 2024-04-23T10:41:10.077 INFO:teuthology.orchestra.run.smithi114.stdout: "rbd_support", 2024-04-23T10:41:10.077 INFO:teuthology.orchestra.run.smithi114.stdout: "status", 2024-04-23T10:41:10.078 INFO:teuthology.orchestra.run.smithi114.stdout: "telemetry", 2024-04-23T10:41:10.078 INFO:teuthology.orchestra.run.smithi114.stdout: "volumes" 2024-04-23T10:41:10.078 INFO:teuthology.orchestra.run.smithi114.stdout: ], 2024-04-23T10:41:10.078 INFO:teuthology.orchestra.run.smithi114.stdout: "pacific": [ 2024-04-23T10:41:10.078 INFO:teuthology.orchestra.run.smithi114.stdout: "balancer", 2024-04-23T10:41:10.078 INFO:teuthology.orchestra.run.smithi114.stdout: "crash", 2024-04-23T10:41:10.078 INFO:teuthology.orchestra.run.smithi114.stdout: "devicehealth", 2024-04-23T10:41:10.078 INFO:teuthology.orchestra.run.smithi114.stdout: "orchestrator", 2024-04-23T10:41:10.078 INFO:teuthology.orchestra.run.smithi114.stdout: "pg_autoscaler", 2024-04-23T10:41:10.079 INFO:teuthology.orchestra.run.smithi114.stdout: "progress", 2024-04-23T10:41:10.079 INFO:teuthology.orchestra.run.smithi114.stdout: "rbd_support", 2024-04-23T10:41:10.079 INFO:teuthology.orchestra.run.smithi114.stdout: "status", 2024-04-23T10:41:10.079 INFO:teuthology.orchestra.run.smithi114.stdout: "telemetry", 2024-04-23T10:41:10.079 INFO:teuthology.orchestra.run.smithi114.stdout: "volumes" 2024-04-23T10:41:10.079 INFO:teuthology.orchestra.run.smithi114.stdout: ], 2024-04-23T10:41:10.079 INFO:teuthology.orchestra.run.smithi114.stdout: "quincy": [ 2024-04-23T10:41:10.079 INFO:teuthology.orchestra.run.smithi114.stdout: "balancer", 2024-04-23T10:41:10.079 INFO:teuthology.orchestra.run.smithi114.stdout: "crash", 2024-04-23T10:41:10.079 INFO:teuthology.orchestra.run.smithi114.stdout: "devicehealth", 2024-04-23T10:41:10.080 INFO:teuthology.orchestra.run.smithi114.stdout: "orchestrator", 2024-04-23T10:41:10.080 INFO:teuthology.orchestra.run.smithi114.stdout: "pg_autoscaler", 2024-04-23T10:41:10.080 INFO:teuthology.orchestra.run.smithi114.stdout: "progress", 2024-04-23T10:41:10.080 INFO:teuthology.orchestra.run.smithi114.stdout: "rbd_support", 2024-04-23T10:41:10.080 INFO:teuthology.orchestra.run.smithi114.stdout: "status", 2024-04-23T10:41:10.080 INFO:teuthology.orchestra.run.smithi114.stdout: "telemetry", 2024-04-23T10:41:10.080 INFO:teuthology.orchestra.run.smithi114.stdout: "volumes" 2024-04-23T10:41:10.080 INFO:teuthology.orchestra.run.smithi114.stdout: ], 2024-04-23T10:41:10.080 INFO:teuthology.orchestra.run.smithi114.stdout: "reef": [ 2024-04-23T10:41:10.080 INFO:teuthology.orchestra.run.smithi114.stdout: "balancer", 2024-04-23T10:41:10.081 INFO:teuthology.orchestra.run.smithi114.stdout: "crash", 2024-04-23T10:41:10.081 INFO:teuthology.orchestra.run.smithi114.stdout: "devicehealth", 2024-04-23T10:41:10.081 INFO:teuthology.orchestra.run.smithi114.stdout: "orchestrator", 2024-04-23T10:41:10.081 INFO:teuthology.orchestra.run.smithi114.stdout: "pg_autoscaler", 2024-04-23T10:41:10.081 INFO:teuthology.orchestra.run.smithi114.stdout: "progress", 2024-04-23T10:41:10.081 INFO:teuthology.orchestra.run.smithi114.stdout: "rbd_support", 2024-04-23T10:41:10.081 INFO:teuthology.orchestra.run.smithi114.stdout: "status", 2024-04-23T10:41:10.081 INFO:teuthology.orchestra.run.smithi114.stdout: "telemetry", 2024-04-23T10:41:10.081 INFO:teuthology.orchestra.run.smithi114.stdout: "volumes" 2024-04-23T10:41:10.081 INFO:teuthology.orchestra.run.smithi114.stdout: ], 2024-04-23T10:41:10.082 INFO:teuthology.orchestra.run.smithi114.stdout: "squid": [ 2024-04-23T10:41:10.082 INFO:teuthology.orchestra.run.smithi114.stdout: "balancer", 2024-04-23T10:41:10.082 INFO:teuthology.orchestra.run.smithi114.stdout: "crash", 2024-04-23T10:41:10.082 INFO:teuthology.orchestra.run.smithi114.stdout: "devicehealth", 2024-04-23T10:41:10.082 INFO:teuthology.orchestra.run.smithi114.stdout: "orchestrator", 2024-04-23T10:41:10.082 INFO:teuthology.orchestra.run.smithi114.stdout: "pg_autoscaler", 2024-04-23T10:41:10.082 INFO:teuthology.orchestra.run.smithi114.stdout: "progress", 2024-04-23T10:41:10.082 INFO:teuthology.orchestra.run.smithi114.stdout: "rbd_support", 2024-04-23T10:41:10.082 INFO:teuthology.orchestra.run.smithi114.stdout: "status", 2024-04-23T10:41:10.082 INFO:teuthology.orchestra.run.smithi114.stdout: "telemetry", 2024-04-23T10:41:10.083 INFO:teuthology.orchestra.run.smithi114.stdout: "volumes" 2024-04-23T10:41:10.083 INFO:teuthology.orchestra.run.smithi114.stdout: ] 2024-04-23T10:41:10.083 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-23T10:41:10.083 INFO:teuthology.orchestra.run.smithi114.stdout: "last_failure_osd_epoch": 90, 2024-04-23T10:41:10.083 INFO:teuthology.orchestra.run.smithi114.stdout: "active_clients": [ 2024-04-23T10:41:10.083 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-04-23T10:41:10.083 INFO:teuthology.orchestra.run.smithi114.stdout: "name": "devicehealth", 2024-04-23T10:41:10.083 INFO:teuthology.orchestra.run.smithi114.stdout: "addrvec": [ 2024-04-23T10:41:10.083 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-04-23T10:41:10.083 INFO:teuthology.orchestra.run.smithi114.stdout: "type": "v2", 2024-04-23T10:41:10.084 INFO:teuthology.orchestra.run.smithi114.stdout: "addr": "172.21.15.114:0", 2024-04-23T10:41:10.084 INFO:teuthology.orchestra.run.smithi114.stdout: "nonce": 2371252335 2024-04-23T10:41:10.084 INFO:teuthology.orchestra.run.smithi114.stdout: } 2024-04-23T10:41:10.084 INFO:teuthology.orchestra.run.smithi114.stdout: ] 2024-04-23T10:41:10.084 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-23T10:41:10.084 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-04-23T10:41:10.084 INFO:teuthology.orchestra.run.smithi114.stdout: "name": "libcephsqlite", 2024-04-23T10:41:10.084 INFO:teuthology.orchestra.run.smithi114.stdout: "addrvec": [ 2024-04-23T10:41:10.084 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-04-23T10:41:10.085 INFO:teuthology.orchestra.run.smithi114.stdout: "type": "v2", 2024-04-23T10:41:10.085 INFO:teuthology.orchestra.run.smithi114.stdout: "addr": "172.21.15.114:0", 2024-04-23T10:41:10.085 INFO:teuthology.orchestra.run.smithi114.stdout: "nonce": 313699975 2024-04-23T10:41:10.085 INFO:teuthology.orchestra.run.smithi114.stdout: } 2024-04-23T10:41:10.085 INFO:teuthology.orchestra.run.smithi114.stdout: ] 2024-04-23T10:41:10.085 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-23T10:41:10.085 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-04-23T10:41:10.085 INFO:teuthology.orchestra.run.smithi114.stdout: "name": "rbd_support", 2024-04-23T10:41:10.085 INFO:teuthology.orchestra.run.smithi114.stdout: "addrvec": [ 2024-04-23T10:41:10.085 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-04-23T10:41:10.086 INFO:teuthology.orchestra.run.smithi114.stdout: "type": "v2", 2024-04-23T10:41:10.086 INFO:teuthology.orchestra.run.smithi114.stdout: "addr": "172.21.15.114:0", 2024-04-23T10:41:10.086 INFO:teuthology.orchestra.run.smithi114.stdout: "nonce": 2131924038 2024-04-23T10:41:10.086 INFO:teuthology.orchestra.run.smithi114.stdout: } 2024-04-23T10:41:10.086 INFO:teuthology.orchestra.run.smithi114.stdout: ] 2024-04-23T10:41:10.086 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-23T10:41:10.086 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-04-23T10:41:10.086 INFO:teuthology.orchestra.run.smithi114.stdout: "name": "snap_schedule", 2024-04-23T10:41:10.086 INFO:teuthology.orchestra.run.smithi114.stdout: "addrvec": [ 2024-04-23T10:41:10.086 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-04-23T10:41:10.087 INFO:teuthology.orchestra.run.smithi114.stdout: "type": "v2", 2024-04-23T10:41:10.087 INFO:teuthology.orchestra.run.smithi114.stdout: "addr": "172.21.15.114:0", 2024-04-23T10:41:10.087 INFO:teuthology.orchestra.run.smithi114.stdout: "nonce": 302929340 2024-04-23T10:41:10.087 INFO:teuthology.orchestra.run.smithi114.stdout: } 2024-04-23T10:41:10.087 INFO:teuthology.orchestra.run.smithi114.stdout: ] 2024-04-23T10:41:10.087 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-23T10:41:10.087 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-04-23T10:41:10.087 INFO:teuthology.orchestra.run.smithi114.stdout: "name": "snap_schedule", 2024-04-23T10:41:10.087 INFO:teuthology.orchestra.run.smithi114.stdout: "addrvec": [ 2024-04-23T10:41:10.087 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-04-23T10:41:10.088 INFO:teuthology.orchestra.run.smithi114.stdout: "type": "v2", 2024-04-23T10:41:10.088 INFO:teuthology.orchestra.run.smithi114.stdout: "addr": "172.21.15.114:0", 2024-04-23T10:41:10.088 INFO:teuthology.orchestra.run.smithi114.stdout: "nonce": 3840117861 2024-04-23T10:41:10.088 INFO:teuthology.orchestra.run.smithi114.stdout: } 2024-04-23T10:41:10.088 INFO:teuthology.orchestra.run.smithi114.stdout: ] 2024-04-23T10:41:10.088 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-23T10:41:10.088 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-04-23T10:41:10.088 INFO:teuthology.orchestra.run.smithi114.stdout: "name": "volumes", 2024-04-23T10:41:10.088 INFO:teuthology.orchestra.run.smithi114.stdout: "addrvec": [ 2024-04-23T10:41:10.088 INFO:teuthology.orchestra.run.smithi114.stdout: { 2024-04-23T10:41:10.089 INFO:teuthology.orchestra.run.smithi114.stdout: "type": "v2", 2024-04-23T10:41:10.089 INFO:teuthology.orchestra.run.smithi114.stdout: "addr": "172.21.15.114:0", 2024-04-23T10:41:10.089 INFO:teuthology.orchestra.run.smithi114.stdout: "nonce": 1517217611 2024-04-23T10:41:10.089 INFO:teuthology.orchestra.run.smithi114.stdout: } 2024-04-23T10:41:10.089 INFO:teuthology.orchestra.run.smithi114.stdout: ] 2024-04-23T10:41:10.089 INFO:teuthology.orchestra.run.smithi114.stdout: } 2024-04-23T10:41:10.089 INFO:teuthology.orchestra.run.smithi114.stdout: ] 2024-04-23T10:41:10.089 INFO:teuthology.orchestra.run.smithi114.stdout:} 2024-04-23T10:41:10.090 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 23 10:41:08 smithi119 ceph-mon[32608]: from='client.17169 -' entity='client.admin' cmd=[{"prefix": "fs snap-schedule list", "fs": "cephfs", "path": "/", "recursive": true, "target": ["mon-mgr", ""]}]: dispatch 2024-04-23T10:41:10.091 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 23 10:41:08 smithi119 ceph-mon[32608]: pgmap v457: 97 pgs: 97 active+clean; 547 MiB data, 4.3 GiB used, 1.0 TiB / 1.0 TiB avail; 9.7 MiB/s rd, 11 MiB/s wr, 669 op/s 2024-04-23T10:41:10.091 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 23 10:41:08 smithi119 ceph-mon[32608]: from='client.? 172.21.15.114:0/4194930089' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2024-04-23T10:41:10.091 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 23 10:41:08 smithi119 ceph-mon[32608]: from='client.? 172.21.15.114:0/1160765680' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2024-04-23T10:41:10.091 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 23 10:41:08 smithi119 ceph-mon[32608]: from='client.? 172.21.15.114:0/595361704' entity='client.admin' cmd=[{"prefix": "mgr dump", "format": "json-pretty"}]: dispatch 2024-04-23T10:41:10.091 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 23 10:41:08 smithi204 ceph-mon[32649]: from='client.17169 -' entity='client.admin' cmd=[{"prefix": "fs snap-schedule list", "fs": "cephfs", "path": "/", "recursive": true, "target": ["mon-mgr", ""]}]: dispatch 2024-04-23T10:41:10.091 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 23 10:41:08 smithi204 ceph-mon[32649]: pgmap v457: 97 pgs: 97 active+clean; 547 MiB data, 4.3 GiB used, 1.0 TiB / 1.0 TiB avail; 9.7 MiB/s rd, 11 MiB/s wr, 669 op/s 2024-04-23T10:41:10.091 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 23 10:41:08 smithi204 ceph-mon[32649]: from='client.? 172.21.15.114:0/4194930089' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2024-04-23T10:41:10.091 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 23 10:41:08 smithi204 ceph-mon[32649]: from='client.? 172.21.15.114:0/1160765680' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2024-04-23T10:41:10.092 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 23 10:41:08 smithi204 ceph-mon[32649]: from='client.? 172.21.15.114:0/595361704' entity='client.admin' cmd=[{"prefix": "mgr dump", "format": "json-pretty"}]: dispatch 2024-04-23T10:41:10.092 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:41:08.014+0000 7f2988079640 1 -- 172.21.15.114:0/595361704 >> [v2:172.21.15.114:6800/1395693466,v1:172.21.15.114:6801/1395693466] conn(0x7f296c0a3d20 msgr2=0x7f296c0a61e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T10:41:10.093 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:41:08.014+0000 7f2988079640 1 --2- 172.21.15.114:0/595361704 >> [v2:172.21.15.114:6800/1395693466,v1:172.21.15.114:6801/1395693466] conn(0x7f296c0a3d20 0x7f296c0a61e0 secure :-1 s=READY pgs=159 cs=0 l=1 rev1=1 crypto rx=0x7f29801004e0 tx=0x7f297c005c70 comp rx=0 tx=0).stop 2024-04-23T10:41:10.093 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:41:08.014+0000 7f2988079640 1 -- 172.21.15.114:0/595361704 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f2980100b10 msgr2=0x7f298019e3f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T10:41:10.093 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:41:08.014+0000 7f2988079640 1 --2- 172.21.15.114:0/595361704 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f2980100b10 0x7f298019e3f0 secure :-1 s=READY pgs=962 cs=0 l=1 rev1=1 crypto rx=0x7f2970005940 tx=0x7f29700042f0 comp rx=0 tx=0).stop 2024-04-23T10:41:10.093 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:41:08.014+0000 7f2988079640 1 -- 172.21.15.114:0/595361704 shutdown_connections 2024-04-23T10:41:10.093 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:41:08.014+0000 7f2988079640 1 --2- 172.21.15.114:0/595361704 >> [v2:172.21.15.114:6800/1395693466,v1:172.21.15.114:6801/1395693466] conn(0x7f296c0a3d20 0x7f296c0a61e0 unknown :-1 s=CLOSED pgs=159 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:41:10.093 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:41:08.014+0000 7f2988079640 1 --2- 172.21.15.114:0/595361704 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f29801985a0 0x7f2980198a50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:41:10.093 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:41:08.014+0000 7f2988079640 1 --2- 172.21.15.114:0/595361704 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f29801072d0 0x7f298019e930 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:41:10.093 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:41:08.014+0000 7f2988079640 1 --2- 172.21.15.114:0/595361704 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f2980100b10 0x7f298019e3f0 unknown :-1 s=CLOSED pgs=962 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:41:10.094 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:41:08.014+0000 7f2988079640 1 -- 172.21.15.114:0/595361704 >> 172.21.15.114:0/595361704 conn(0x7f2980076130 msgr2=0x7f29800fec80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T10:41:10.094 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:41:08.014+0000 7f2988079640 1 -- 172.21.15.114:0/595361704 shutdown_connections 2024-04-23T10:41:10.094 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:41:08.014+0000 7f2988079640 1 -- 172.21.15.114:0/595361704 wait complete. 2024-04-23T10:41:10.099 DEBUG:tasks.check_counter:Getting stats from a 2024-04-23T10:41:10.099 DEBUG:teuthology.orchestra.run.smithi114:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d69f13d827952bb07bb9b7973bca14f97d764d6f shell --fsid aa240572-0159-11ef-bc93-c7b262605968 -- ceph daemon mds.a perf dump 2024-04-23T10:41:10.100 INFO:journalctl@ceph.mon.a.smithi114.stdout:Apr 23 10:41:08 smithi114 ceph-mon[29255]: from='client.17169 -' entity='client.admin' cmd=[{"prefix": "fs snap-schedule list", "fs": "cephfs", "path": "/", "recursive": true, "target": ["mon-mgr", ""]}]: dispatch 2024-04-23T10:41:10.100 INFO:journalctl@ceph.mon.a.smithi114.stdout:Apr 23 10:41:08 smithi114 ceph-mon[29255]: pgmap v457: 97 pgs: 97 active+clean; 547 MiB data, 4.3 GiB used, 1.0 TiB / 1.0 TiB avail; 9.7 MiB/s rd, 11 MiB/s wr, 669 op/s 2024-04-23T10:41:10.100 INFO:journalctl@ceph.mon.a.smithi114.stdout:Apr 23 10:41:08 smithi114 ceph-mon[29255]: from='client.? 172.21.15.114:0/4194930089' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2024-04-23T10:41:10.100 INFO:journalctl@ceph.mon.a.smithi114.stdout:Apr 23 10:41:08 smithi114 ceph-mon[29255]: from='client.? 172.21.15.114:0/1160765680' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2024-04-23T10:41:10.100 INFO:journalctl@ceph.mon.a.smithi114.stdout:Apr 23 10:41:08 smithi114 ceph-mon[29255]: from='client.? 172.21.15.114:0/595361704' entity='client.admin' cmd=[{"prefix": "mgr dump", "format": "json-pretty"}]: dispatch 2024-04-23T10:41:10.335 INFO:teuthology.orchestra.run.smithi114.stderr:Inferring config /var/lib/ceph/aa240572-0159-11ef-bc93-c7b262605968/mon.a/config 2024-04-23T10:41:10.459 INFO:journalctl@ceph.mon.a.smithi114.stdout:Apr 23 10:41:10 smithi114 ceph-mon[29255]: pgmap v458: 97 pgs: 97 active+clean; 547 MiB data, 4.3 GiB used, 1.0 TiB / 1.0 TiB avail; 5.0 MiB/s rd, 7.4 MiB/s wr, 457 op/s 2024-04-23T10:41:10.459 INFO:journalctl@ceph.mon.a.smithi114.stdout:Apr 23 10:41:10 smithi114 ceph-mon[29255]: osdmap e112: 12 total, 12 up, 12 in 2024-04-23T10:41:10.578 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 23 10:41:10 smithi204 ceph-mon[32649]: pgmap v458: 97 pgs: 97 active+clean; 547 MiB data, 4.3 GiB used, 1.0 TiB / 1.0 TiB avail; 5.0 MiB/s rd, 7.4 MiB/s wr, 457 op/s 2024-04-23T10:41:10.578 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 23 10:41:10 smithi204 ceph-mon[32649]: osdmap e112: 12 total, 12 up, 12 in 2024-04-23T10:41:10.742 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 23 10:41:10 smithi119 ceph-mon[32608]: pgmap v458: 97 pgs: 97 active+clean; 547 MiB data, 4.3 GiB used, 1.0 TiB / 1.0 TiB avail; 5.0 MiB/s rd, 7.4 MiB/s wr, 457 op/s 2024-04-23T10:41:10.742 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 23 10:41:10 smithi119 ceph-mon[32608]: osdmap e112: 12 total, 12 up, 12 in 2024-04-23T10:41:12.179 INFO:teuthology.orchestra.run.smithi114.stdout:{ 2024-04-23T10:41:12.180 INFO:teuthology.orchestra.run.smithi114.stdout: "AsyncMessenger::Worker-0": { 2024-04-23T10:41:12.180 INFO:teuthology.orchestra.run.smithi114.stdout: "msgr_recv_messages": 2, 2024-04-23T10:41:12.180 INFO:teuthology.orchestra.run.smithi114.stdout: "msgr_send_messages": 4, 2024-04-23T10:41:12.181 INFO:teuthology.orchestra.run.smithi114.stdout: "msgr_recv_bytes": 303, 2024-04-23T10:41:12.181 INFO:teuthology.orchestra.run.smithi114.stdout: "msgr_send_bytes": 23311, 2024-04-23T10:41:12.181 INFO:teuthology.orchestra.run.smithi114.stdout: "msgr_created_connections": 6, 2024-04-23T10:41:12.181 INFO:teuthology.orchestra.run.smithi114.stdout: "msgr_active_connections": 18446744073709551615, 2024-04-23T10:41:12.181 INFO:teuthology.orchestra.run.smithi114.stdout: "msgr_running_total_time": 0.001929772, 2024-04-23T10:41:12.181 INFO:teuthology.orchestra.run.smithi114.stdout: "msgr_running_send_time": 0.000052350, 2024-04-23T10:41:12.182 INFO:teuthology.orchestra.run.smithi114.stdout: "msgr_running_recv_time": 0.000811722, 2024-04-23T10:41:12.182 INFO:teuthology.orchestra.run.smithi114.stdout: "msgr_running_fast_dispatch_time": 0.000000000, 2024-04-23T10:41:12.182 INFO:teuthology.orchestra.run.smithi114.stdout: "msgr_send_messages_queue_lat": { 2024-04-23T10:41:12.182 INFO:teuthology.orchestra.run.smithi114.stdout: "avgcount": 2, 2024-04-23T10:41:12.182 INFO:teuthology.orchestra.run.smithi114.stdout: "sum": 0.000018571, 2024-04-23T10:41:12.183 INFO:teuthology.orchestra.run.smithi114.stdout: "avgtime": 0.000009285 2024-04-23T10:41:12.183 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-23T10:41:12.183 INFO:teuthology.orchestra.run.smithi114.stdout: "msgr_handle_ack_lat": { 2024-04-23T10:41:12.183 INFO:teuthology.orchestra.run.smithi114.stdout: "avgcount": 2, 2024-04-23T10:41:12.183 INFO:teuthology.orchestra.run.smithi114.stdout: "sum": 0.000000139, 2024-04-23T10:41:12.183 INFO:teuthology.orchestra.run.smithi114.stdout: "avgtime": 0.000000069 2024-04-23T10:41:12.184 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-23T10:41:12.184 INFO:teuthology.orchestra.run.smithi114.stdout: "msgr_recv_encrypted_bytes": 0, 2024-04-23T10:41:12.184 INFO:teuthology.orchestra.run.smithi114.stdout: "msgr_send_encrypted_bytes": 0 2024-04-23T10:41:12.184 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-23T10:41:12.184 INFO:teuthology.orchestra.run.smithi114.stdout: "AsyncMessenger::Worker-1": { 2024-04-23T10:41:12.184 INFO:teuthology.orchestra.run.smithi114.stdout: "msgr_recv_messages": 278, 2024-04-23T10:41:12.185 INFO:teuthology.orchestra.run.smithi114.stdout: "msgr_send_messages": 267, 2024-04-23T10:41:12.185 INFO:teuthology.orchestra.run.smithi114.stdout: "msgr_recv_bytes": 1605712, 2024-04-23T10:41:12.185 INFO:teuthology.orchestra.run.smithi114.stdout: "msgr_send_bytes": 164544, 2024-04-23T10:41:12.185 INFO:teuthology.orchestra.run.smithi114.stdout: "msgr_created_connections": 2, 2024-04-23T10:41:12.185 INFO:teuthology.orchestra.run.smithi114.stdout: "msgr_active_connections": 1, 2024-04-23T10:41:12.186 INFO:teuthology.orchestra.run.smithi114.stdout: "msgr_running_total_time": 0.047848246, 2024-04-23T10:41:12.186 INFO:teuthology.orchestra.run.smithi114.stdout: "msgr_running_send_time": 0.016715926, 2024-04-23T10:41:12.186 INFO:teuthology.orchestra.run.smithi114.stdout: "msgr_running_recv_time": 0.035005859, 2024-04-23T10:41:12.186 INFO:teuthology.orchestra.run.smithi114.stdout: "msgr_running_fast_dispatch_time": 0.005120655, 2024-04-23T10:41:12.186 INFO:teuthology.orchestra.run.smithi114.stdout: "msgr_send_messages_queue_lat": { 2024-04-23T10:41:12.186 INFO:teuthology.orchestra.run.smithi114.stdout: "avgcount": 266, 2024-04-23T10:41:12.187 INFO:teuthology.orchestra.run.smithi114.stdout: "sum": 0.011161473, 2024-04-23T10:41:12.187 INFO:teuthology.orchestra.run.smithi114.stdout: "avgtime": 0.000041960 2024-04-23T10:41:12.187 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-23T10:41:12.187 INFO:teuthology.orchestra.run.smithi114.stdout: "msgr_handle_ack_lat": { 2024-04-23T10:41:12.187 INFO:teuthology.orchestra.run.smithi114.stdout: "avgcount": 0, 2024-04-23T10:41:12.187 INFO:teuthology.orchestra.run.smithi114.stdout: "sum": 0.000000000, 2024-04-23T10:41:12.188 INFO:teuthology.orchestra.run.smithi114.stdout: "avgtime": 0.000000000 2024-04-23T10:41:12.188 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-23T10:41:12.188 INFO:teuthology.orchestra.run.smithi114.stdout: "msgr_recv_encrypted_bytes": 1605712, 2024-04-23T10:41:12.188 INFO:teuthology.orchestra.run.smithi114.stdout: "msgr_send_encrypted_bytes": 164544 2024-04-23T10:41:12.188 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-23T10:41:12.189 INFO:teuthology.orchestra.run.smithi114.stdout: "AsyncMessenger::Worker-2": { 2024-04-23T10:41:12.189 INFO:teuthology.orchestra.run.smithi114.stdout: "msgr_recv_messages": 6, 2024-04-23T10:41:12.189 INFO:teuthology.orchestra.run.smithi114.stdout: "msgr_send_messages": 206, 2024-04-23T10:41:12.189 INFO:teuthology.orchestra.run.smithi114.stdout: "msgr_recv_bytes": 3312, 2024-04-23T10:41:12.189 INFO:teuthology.orchestra.run.smithi114.stdout: "msgr_send_bytes": 192432, 2024-04-23T10:41:12.189 INFO:teuthology.orchestra.run.smithi114.stdout: "msgr_created_connections": 5, 2024-04-23T10:41:12.190 INFO:teuthology.orchestra.run.smithi114.stdout: "msgr_active_connections": 1, 2024-04-23T10:41:12.190 INFO:teuthology.orchestra.run.smithi114.stdout: "msgr_running_total_time": 0.015899210, 2024-04-23T10:41:12.190 INFO:teuthology.orchestra.run.smithi114.stdout: "msgr_running_send_time": 0.010728778, 2024-04-23T10:41:12.190 INFO:teuthology.orchestra.run.smithi114.stdout: "msgr_running_recv_time": 0.002614948, 2024-04-23T10:41:12.190 INFO:teuthology.orchestra.run.smithi114.stdout: "msgr_running_fast_dispatch_time": 0.000000000, 2024-04-23T10:41:12.191 INFO:teuthology.orchestra.run.smithi114.stdout: "msgr_send_messages_queue_lat": { 2024-04-23T10:41:12.191 INFO:teuthology.orchestra.run.smithi114.stdout: "avgcount": 204, 2024-04-23T10:41:12.191 INFO:teuthology.orchestra.run.smithi114.stdout: "sum": 0.017471071, 2024-04-23T10:41:12.191 INFO:teuthology.orchestra.run.smithi114.stdout: "avgtime": 0.000085642 2024-04-23T10:41:12.191 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-23T10:41:12.192 INFO:teuthology.orchestra.run.smithi114.stdout: "msgr_handle_ack_lat": { 2024-04-23T10:41:12.192 INFO:teuthology.orchestra.run.smithi114.stdout: "avgcount": 0, 2024-04-23T10:41:12.192 INFO:teuthology.orchestra.run.smithi114.stdout: "sum": 0.000000000, 2024-04-23T10:41:12.192 INFO:teuthology.orchestra.run.smithi114.stdout: "avgtime": 0.000000000 2024-04-23T10:41:12.192 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-23T10:41:12.192 INFO:teuthology.orchestra.run.smithi114.stdout: "msgr_recv_encrypted_bytes": 3312, 2024-04-23T10:41:12.193 INFO:teuthology.orchestra.run.smithi114.stdout: "msgr_send_encrypted_bytes": 192432 2024-04-23T10:41:12.193 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-23T10:41:12.193 INFO:teuthology.orchestra.run.smithi114.stdout: "cct": { 2024-04-23T10:41:12.193 INFO:teuthology.orchestra.run.smithi114.stdout: "total_workers": 0, 2024-04-23T10:41:12.193 INFO:teuthology.orchestra.run.smithi114.stdout: "unhealthy_workers": 0 2024-04-23T10:41:12.193 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-23T10:41:12.194 INFO:teuthology.orchestra.run.smithi114.stdout: "mempool": { 2024-04-23T10:41:12.194 INFO:teuthology.orchestra.run.smithi114.stdout: "bloom_filter_bytes": 0, 2024-04-23T10:41:12.194 INFO:teuthology.orchestra.run.smithi114.stdout: "bloom_filter_items": 0, 2024-04-23T10:41:12.194 INFO:teuthology.orchestra.run.smithi114.stdout: "bluestore_alloc_bytes": 0, 2024-04-23T10:41:12.194 INFO:teuthology.orchestra.run.smithi114.stdout: "bluestore_alloc_items": 0, 2024-04-23T10:41:12.194 INFO:teuthology.orchestra.run.smithi114.stdout: "bluestore_cache_data_bytes": 0, 2024-04-23T10:41:12.195 INFO:teuthology.orchestra.run.smithi114.stdout: "bluestore_cache_data_items": 0, 2024-04-23T10:41:12.195 INFO:teuthology.orchestra.run.smithi114.stdout: "bluestore_cache_onode_bytes": 0, 2024-04-23T10:41:12.195 INFO:teuthology.orchestra.run.smithi114.stdout: "bluestore_cache_onode_items": 0, 2024-04-23T10:41:12.195 INFO:teuthology.orchestra.run.smithi114.stdout: "bluestore_cache_meta_bytes": 0, 2024-04-23T10:41:12.195 INFO:teuthology.orchestra.run.smithi114.stdout: "bluestore_cache_meta_items": 0, 2024-04-23T10:41:12.195 INFO:teuthology.orchestra.run.smithi114.stdout: "bluestore_cache_other_bytes": 0, 2024-04-23T10:41:12.196 INFO:teuthology.orchestra.run.smithi114.stdout: "bluestore_cache_other_items": 0, 2024-04-23T10:41:12.196 INFO:teuthology.orchestra.run.smithi114.stdout: "bluestore_cache_buffer_bytes": 0, 2024-04-23T10:41:12.196 INFO:teuthology.orchestra.run.smithi114.stdout: "bluestore_cache_buffer_items": 0, 2024-04-23T10:41:12.196 INFO:teuthology.orchestra.run.smithi114.stdout: "bluestore_extent_bytes": 0, 2024-04-23T10:41:12.196 INFO:teuthology.orchestra.run.smithi114.stdout: "bluestore_extent_items": 0, 2024-04-23T10:41:12.197 INFO:teuthology.orchestra.run.smithi114.stdout: "bluestore_blob_bytes": 0, 2024-04-23T10:41:12.197 INFO:teuthology.orchestra.run.smithi114.stdout: "bluestore_blob_items": 0, 2024-04-23T10:41:12.197 INFO:teuthology.orchestra.run.smithi114.stdout: "bluestore_shared_blob_bytes": 0, 2024-04-23T10:41:12.197 INFO:teuthology.orchestra.run.smithi114.stdout: "bluestore_shared_blob_items": 0, 2024-04-23T10:41:12.197 INFO:teuthology.orchestra.run.smithi114.stdout: "bluestore_inline_bl_bytes": 0, 2024-04-23T10:41:12.197 INFO:teuthology.orchestra.run.smithi114.stdout: "bluestore_inline_bl_items": 0, 2024-04-23T10:41:12.198 INFO:teuthology.orchestra.run.smithi114.stdout: "bluestore_fsck_bytes": 0, 2024-04-23T10:41:12.198 INFO:teuthology.orchestra.run.smithi114.stdout: "bluestore_fsck_items": 0, 2024-04-23T10:41:12.198 INFO:teuthology.orchestra.run.smithi114.stdout: "bluestore_txc_bytes": 0, 2024-04-23T10:41:12.198 INFO:teuthology.orchestra.run.smithi114.stdout: "bluestore_txc_items": 0, 2024-04-23T10:41:12.198 INFO:teuthology.orchestra.run.smithi114.stdout: "bluestore_writing_deferred_bytes": 0, 2024-04-23T10:41:12.199 INFO:teuthology.orchestra.run.smithi114.stdout: "bluestore_writing_deferred_items": 0, 2024-04-23T10:41:12.199 INFO:teuthology.orchestra.run.smithi114.stdout: "bluestore_writing_bytes": 0, 2024-04-23T10:41:12.199 INFO:teuthology.orchestra.run.smithi114.stdout: "bluestore_writing_items": 0, 2024-04-23T10:41:12.199 INFO:teuthology.orchestra.run.smithi114.stdout: "bluefs_bytes": 0, 2024-04-23T10:41:12.199 INFO:teuthology.orchestra.run.smithi114.stdout: "bluefs_items": 0, 2024-04-23T10:41:12.199 INFO:teuthology.orchestra.run.smithi114.stdout: "bluefs_file_reader_bytes": 0, 2024-04-23T10:41:12.200 INFO:teuthology.orchestra.run.smithi114.stdout: "bluefs_file_reader_items": 0, 2024-04-23T10:41:12.200 INFO:teuthology.orchestra.run.smithi114.stdout: "bluefs_file_writer_bytes": 0, 2024-04-23T10:41:12.200 INFO:teuthology.orchestra.run.smithi114.stdout: "bluefs_file_writer_items": 0, 2024-04-23T10:41:12.200 INFO:teuthology.orchestra.run.smithi114.stdout: "buffer_anon_bytes": 148903, 2024-04-23T10:41:12.200 INFO:teuthology.orchestra.run.smithi114.stdout: "buffer_anon_items": 27, 2024-04-23T10:41:12.200 INFO:teuthology.orchestra.run.smithi114.stdout: "buffer_meta_bytes": 0, 2024-04-23T10:41:12.201 INFO:teuthology.orchestra.run.smithi114.stdout: "buffer_meta_items": 0, 2024-04-23T10:41:12.201 INFO:teuthology.orchestra.run.smithi114.stdout: "osd_bytes": 0, 2024-04-23T10:41:12.201 INFO:teuthology.orchestra.run.smithi114.stdout: "osd_items": 0, 2024-04-23T10:41:12.201 INFO:teuthology.orchestra.run.smithi114.stdout: "osd_mapbl_bytes": 0, 2024-04-23T10:41:12.201 INFO:teuthology.orchestra.run.smithi114.stdout: "osd_mapbl_items": 0, 2024-04-23T10:41:12.201 INFO:teuthology.orchestra.run.smithi114.stdout: "osd_pglog_bytes": 0, 2024-04-23T10:41:12.202 INFO:teuthology.orchestra.run.smithi114.stdout: "osd_pglog_items": 0, 2024-04-23T10:41:12.202 INFO:teuthology.orchestra.run.smithi114.stdout: "osdmap_bytes": 0, 2024-04-23T10:41:12.202 INFO:teuthology.orchestra.run.smithi114.stdout: "osdmap_items": 0, 2024-04-23T10:41:12.202 INFO:teuthology.orchestra.run.smithi114.stdout: "osdmap_mapping_bytes": 0, 2024-04-23T10:41:12.202 INFO:teuthology.orchestra.run.smithi114.stdout: "osdmap_mapping_items": 0, 2024-04-23T10:41:12.202 INFO:teuthology.orchestra.run.smithi114.stdout: "pgmap_bytes": 0, 2024-04-23T10:41:12.203 INFO:teuthology.orchestra.run.smithi114.stdout: "pgmap_items": 0, 2024-04-23T10:41:12.203 INFO:teuthology.orchestra.run.smithi114.stdout: "mds_co_bytes": 928, 2024-04-23T10:41:12.203 INFO:teuthology.orchestra.run.smithi114.stdout: "mds_co_items": 2, 2024-04-23T10:41:12.203 INFO:teuthology.orchestra.run.smithi114.stdout: "unittest_1_bytes": 0, 2024-04-23T10:41:12.203 INFO:teuthology.orchestra.run.smithi114.stdout: "unittest_1_items": 0, 2024-04-23T10:41:12.203 INFO:teuthology.orchestra.run.smithi114.stdout: "unittest_2_bytes": 0, 2024-04-23T10:41:12.204 INFO:teuthology.orchestra.run.smithi114.stdout: "unittest_2_items": 0 2024-04-23T10:41:12.204 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-23T10:41:12.204 INFO:teuthology.orchestra.run.smithi114.stdout: "throttle-msgr_dispatch_throttler-mds": { 2024-04-23T10:41:12.204 INFO:teuthology.orchestra.run.smithi114.stdout: "val": 0, 2024-04-23T10:41:12.204 INFO:teuthology.orchestra.run.smithi114.stdout: "max": 104857600, 2024-04-23T10:41:12.204 INFO:teuthology.orchestra.run.smithi114.stdout: "get_started": 0, 2024-04-23T10:41:12.205 INFO:teuthology.orchestra.run.smithi114.stdout: "get": 282, 2024-04-23T10:41:12.205 INFO:teuthology.orchestra.run.smithi114.stdout: "get_sum": 1566619, 2024-04-23T10:41:12.205 INFO:teuthology.orchestra.run.smithi114.stdout: "get_or_fail_fail": 0, 2024-04-23T10:41:12.205 INFO:teuthology.orchestra.run.smithi114.stdout: "get_or_fail_success": 282, 2024-04-23T10:41:12.205 INFO:teuthology.orchestra.run.smithi114.stdout: "take": 0, 2024-04-23T10:41:12.205 INFO:teuthology.orchestra.run.smithi114.stdout: "take_sum": 0, 2024-04-23T10:41:12.206 INFO:teuthology.orchestra.run.smithi114.stdout: "put": 282, 2024-04-23T10:41:12.206 INFO:teuthology.orchestra.run.smithi114.stdout: "put_sum": 1566619, 2024-04-23T10:41:12.206 INFO:teuthology.orchestra.run.smithi114.stdout: "wait": { 2024-04-23T10:41:12.206 INFO:teuthology.orchestra.run.smithi114.stdout: "avgcount": 0, 2024-04-23T10:41:12.206 INFO:teuthology.orchestra.run.smithi114.stdout: "sum": 0.000000000, 2024-04-23T10:41:12.207 INFO:teuthology.orchestra.run.smithi114.stdout: "avgtime": 0.000000000 2024-04-23T10:41:12.207 INFO:teuthology.orchestra.run.smithi114.stdout: } 2024-04-23T10:41:12.207 INFO:teuthology.orchestra.run.smithi114.stdout: } 2024-04-23T10:41:12.207 INFO:teuthology.orchestra.run.smithi114.stdout:} 2024-04-23T10:41:12.464 INFO:journalctl@ceph.mon.a.smithi114.stdout:Apr 23 10:41:12 smithi114 ceph-mon[29255]: pgmap v460: 97 pgs: 97 active+clean; 547 MiB data, 4.3 GiB used, 1.0 TiB / 1.0 TiB avail; 6.0 MiB/s rd, 8.9 MiB/s wr, 549 op/s 2024-04-23T10:41:12.578 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 23 10:41:12 smithi204 ceph-mon[32649]: pgmap v460: 97 pgs: 97 active+clean; 547 MiB data, 4.3 GiB used, 1.0 TiB / 1.0 TiB avail; 6.0 MiB/s rd, 8.9 MiB/s wr, 549 op/s 2024-04-23T10:41:12.742 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 23 10:41:12 smithi119 ceph-mon[32608]: pgmap v460: 97 pgs: 97 active+clean; 547 MiB data, 4.3 GiB used, 1.0 TiB / 1.0 TiB avail; 6.0 MiB/s rd, 8.9 MiB/s wr, 549 op/s 2024-04-23T10:41:12.893 WARNING:tasks.check_counter:Counter 'mds.exported' not found on daemon mds.a 2024-04-23T10:41:12.893 WARNING:tasks.check_counter:Counter 'mds.imported' not found on daemon mds.a 2024-04-23T10:41:12.894 WARNING:tasks.check_counter:Counter 'mds_cache.dir_update' not found on daemon mds.a 2024-04-23T10:41:12.894 WARNING:tasks.check_counter:Counter 'mds_cache.dir_update_receipt' not found on daemon mds.a 2024-04-23T10:41:12.894 WARNING:tasks.check_counter:Counter 'mds.root_rsnaps' not found on daemon mds.a 2024-04-23T10:41:12.894 WARNING:tasks.check_counter:Counter 'mds_server.req_mksnap_latency.avgcount' not found on daemon mds.a 2024-04-23T10:41:12.894 WARNING:tasks.check_counter:Counter 'mds_server.req_rmsnap_latency.avgcount' not found on daemon mds.a 2024-04-23T10:41:12.894 DEBUG:tasks.check_counter:Getting stats from d 2024-04-23T10:41:12.895 DEBUG:teuthology.orchestra.run.smithi114:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d69f13d827952bb07bb9b7973bca14f97d764d6f shell --fsid aa240572-0159-11ef-bc93-c7b262605968 -- ceph daemon mds.d perf dump 2024-04-23T10:41:13.151 INFO:teuthology.orchestra.run.smithi114.stderr:Inferring config /var/lib/ceph/aa240572-0159-11ef-bc93-c7b262605968/mon.a/config 2024-04-23T10:41:14.504 INFO:journalctl@ceph.mon.a.smithi114.stdout:Apr 23 10:41:14 smithi114 ceph-mon[29255]: pgmap v461: 97 pgs: 97 active+clean; 558 MiB data, 4.3 GiB used, 1.0 TiB / 1.0 TiB avail; 690 KiB/s rd, 5.1 MiB/s wr, 288 op/s 2024-04-23T10:41:14.505 INFO:journalctl@ceph.mon.a.smithi114.stdout:Apr 23 10:41:14 smithi114 ceph-mon[29255]: osdmap e113: 12 total, 12 up, 12 in 2024-04-23T10:41:14.577 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 23 10:41:14 smithi204 ceph-mon[32649]: pgmap v461: 97 pgs: 97 active+clean; 558 MiB data, 4.3 GiB used, 1.0 TiB / 1.0 TiB avail; 690 KiB/s rd, 5.1 MiB/s wr, 288 op/s 2024-04-23T10:41:14.578 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 23 10:41:14 smithi204 ceph-mon[32649]: osdmap e113: 12 total, 12 up, 12 in 2024-04-23T10:41:14.741 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 23 10:41:14 smithi119 ceph-mon[32608]: pgmap v461: 97 pgs: 97 active+clean; 558 MiB data, 4.3 GiB used, 1.0 TiB / 1.0 TiB avail; 690 KiB/s rd, 5.1 MiB/s wr, 288 op/s 2024-04-23T10:41:14.742 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 23 10:41:14 smithi119 ceph-mon[32608]: osdmap e113: 12 total, 12 up, 12 in 2024-04-23T10:41:14.819 INFO:teuthology.orchestra.run.smithi114.stdout:{ 2024-04-23T10:41:14.819 INFO:teuthology.orchestra.run.smithi114.stdout: "AsyncMessenger::Worker-0": { 2024-04-23T10:41:14.819 INFO:teuthology.orchestra.run.smithi114.stdout: "msgr_recv_messages": 2, 2024-04-23T10:41:14.819 INFO:teuthology.orchestra.run.smithi114.stdout: "msgr_send_messages": 4, 2024-04-23T10:41:14.819 INFO:teuthology.orchestra.run.smithi114.stdout: "msgr_recv_bytes": 303, 2024-04-23T10:41:14.820 INFO:teuthology.orchestra.run.smithi114.stdout: "msgr_send_bytes": 23311, 2024-04-23T10:41:14.820 INFO:teuthology.orchestra.run.smithi114.stdout: "msgr_created_connections": 6, 2024-04-23T10:41:14.820 INFO:teuthology.orchestra.run.smithi114.stdout: "msgr_active_connections": 18446744073709551615, 2024-04-23T10:41:14.820 INFO:teuthology.orchestra.run.smithi114.stdout: "msgr_running_total_time": 0.001974672, 2024-04-23T10:41:14.820 INFO:teuthology.orchestra.run.smithi114.stdout: "msgr_running_send_time": 0.000068163, 2024-04-23T10:41:14.820 INFO:teuthology.orchestra.run.smithi114.stdout: "msgr_running_recv_time": 0.000799489, 2024-04-23T10:41:14.820 INFO:teuthology.orchestra.run.smithi114.stdout: "msgr_running_fast_dispatch_time": 0.000000000, 2024-04-23T10:41:14.821 INFO:teuthology.orchestra.run.smithi114.stdout: "msgr_send_messages_queue_lat": { 2024-04-23T10:41:14.821 INFO:teuthology.orchestra.run.smithi114.stdout: "avgcount": 2, 2024-04-23T10:41:14.821 INFO:teuthology.orchestra.run.smithi114.stdout: "sum": 0.000022317, 2024-04-23T10:41:14.821 INFO:teuthology.orchestra.run.smithi114.stdout: "avgtime": 0.000011158 2024-04-23T10:41:14.821 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-23T10:41:14.821 INFO:teuthology.orchestra.run.smithi114.stdout: "msgr_handle_ack_lat": { 2024-04-23T10:41:14.821 INFO:teuthology.orchestra.run.smithi114.stdout: "avgcount": 2, 2024-04-23T10:41:14.821 INFO:teuthology.orchestra.run.smithi114.stdout: "sum": 0.000000236, 2024-04-23T10:41:14.821 INFO:teuthology.orchestra.run.smithi114.stdout: "avgtime": 0.000000118 2024-04-23T10:41:14.822 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-23T10:41:14.822 INFO:teuthology.orchestra.run.smithi114.stdout: "msgr_recv_encrypted_bytes": 0, 2024-04-23T10:41:14.822 INFO:teuthology.orchestra.run.smithi114.stdout: "msgr_send_encrypted_bytes": 0 2024-04-23T10:41:14.822 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-23T10:41:14.822 INFO:teuthology.orchestra.run.smithi114.stdout: "AsyncMessenger::Worker-1": { 2024-04-23T10:41:14.822 INFO:teuthology.orchestra.run.smithi114.stdout: "msgr_recv_messages": 2, 2024-04-23T10:41:14.822 INFO:teuthology.orchestra.run.smithi114.stdout: "msgr_send_messages": 208, 2024-04-23T10:41:14.822 INFO:teuthology.orchestra.run.smithi114.stdout: "msgr_recv_bytes": 320, 2024-04-23T10:41:14.823 INFO:teuthology.orchestra.run.smithi114.stdout: "msgr_send_bytes": 192688, 2024-04-23T10:41:14.823 INFO:teuthology.orchestra.run.smithi114.stdout: "msgr_created_connections": 5, 2024-04-23T10:41:14.823 INFO:teuthology.orchestra.run.smithi114.stdout: "msgr_active_connections": 1, 2024-04-23T10:41:14.823 INFO:teuthology.orchestra.run.smithi114.stdout: "msgr_running_total_time": 0.015336371, 2024-04-23T10:41:14.823 INFO:teuthology.orchestra.run.smithi114.stdout: "msgr_running_send_time": 0.010944643, 2024-04-23T10:41:14.823 INFO:teuthology.orchestra.run.smithi114.stdout: "msgr_running_recv_time": 0.001869610, 2024-04-23T10:41:14.823 INFO:teuthology.orchestra.run.smithi114.stdout: "msgr_running_fast_dispatch_time": 0.000000000, 2024-04-23T10:41:14.823 INFO:teuthology.orchestra.run.smithi114.stdout: "msgr_send_messages_queue_lat": { 2024-04-23T10:41:14.823 INFO:teuthology.orchestra.run.smithi114.stdout: "avgcount": 205, 2024-04-23T10:41:14.823 INFO:teuthology.orchestra.run.smithi114.stdout: "sum": 0.011024334, 2024-04-23T10:41:14.824 INFO:teuthology.orchestra.run.smithi114.stdout: "avgtime": 0.000053777 2024-04-23T10:41:14.824 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-23T10:41:14.824 INFO:teuthology.orchestra.run.smithi114.stdout: "msgr_handle_ack_lat": { 2024-04-23T10:41:14.824 INFO:teuthology.orchestra.run.smithi114.stdout: "avgcount": 0, 2024-04-23T10:41:14.824 INFO:teuthology.orchestra.run.smithi114.stdout: "sum": 0.000000000, 2024-04-23T10:41:14.824 INFO:teuthology.orchestra.run.smithi114.stdout: "avgtime": 0.000000000 2024-04-23T10:41:14.824 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-23T10:41:14.824 INFO:teuthology.orchestra.run.smithi114.stdout: "msgr_recv_encrypted_bytes": 320, 2024-04-23T10:41:14.825 INFO:teuthology.orchestra.run.smithi114.stdout: "msgr_send_encrypted_bytes": 192688 2024-04-23T10:41:14.825 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-23T10:41:14.825 INFO:teuthology.orchestra.run.smithi114.stdout: "AsyncMessenger::Worker-2": { 2024-04-23T10:41:14.825 INFO:teuthology.orchestra.run.smithi114.stdout: "msgr_recv_messages": 287, 2024-04-23T10:41:14.825 INFO:teuthology.orchestra.run.smithi114.stdout: "msgr_send_messages": 274, 2024-04-23T10:41:14.825 INFO:teuthology.orchestra.run.smithi114.stdout: "msgr_recv_bytes": 1610064, 2024-04-23T10:41:14.825 INFO:teuthology.orchestra.run.smithi114.stdout: "msgr_send_bytes": 168192, 2024-04-23T10:41:14.825 INFO:teuthology.orchestra.run.smithi114.stdout: "msgr_created_connections": 2, 2024-04-23T10:41:14.825 INFO:teuthology.orchestra.run.smithi114.stdout: "msgr_active_connections": 1, 2024-04-23T10:41:14.826 INFO:teuthology.orchestra.run.smithi114.stdout: "msgr_running_total_time": 0.046024945, 2024-04-23T10:41:14.826 INFO:teuthology.orchestra.run.smithi114.stdout: "msgr_running_send_time": 0.015582006, 2024-04-23T10:41:14.826 INFO:teuthology.orchestra.run.smithi114.stdout: "msgr_running_recv_time": 0.033910673, 2024-04-23T10:41:14.826 INFO:teuthology.orchestra.run.smithi114.stdout: "msgr_running_fast_dispatch_time": 0.004940068, 2024-04-23T10:41:14.826 INFO:teuthology.orchestra.run.smithi114.stdout: "msgr_send_messages_queue_lat": { 2024-04-23T10:41:14.826 INFO:teuthology.orchestra.run.smithi114.stdout: "avgcount": 274, 2024-04-23T10:41:14.826 INFO:teuthology.orchestra.run.smithi114.stdout: "sum": 0.012086859, 2024-04-23T10:41:14.826 INFO:teuthology.orchestra.run.smithi114.stdout: "avgtime": 0.000044112 2024-04-23T10:41:14.826 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-23T10:41:14.826 INFO:teuthology.orchestra.run.smithi114.stdout: "msgr_handle_ack_lat": { 2024-04-23T10:41:14.827 INFO:teuthology.orchestra.run.smithi114.stdout: "avgcount": 0, 2024-04-23T10:41:14.827 INFO:teuthology.orchestra.run.smithi114.stdout: "sum": 0.000000000, 2024-04-23T10:41:14.827 INFO:teuthology.orchestra.run.smithi114.stdout: "avgtime": 0.000000000 2024-04-23T10:41:14.827 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-23T10:41:14.827 INFO:teuthology.orchestra.run.smithi114.stdout: "msgr_recv_encrypted_bytes": 1610064, 2024-04-23T10:41:14.827 INFO:teuthology.orchestra.run.smithi114.stdout: "msgr_send_encrypted_bytes": 168192 2024-04-23T10:41:14.827 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-23T10:41:14.827 INFO:teuthology.orchestra.run.smithi114.stdout: "cct": { 2024-04-23T10:41:14.827 INFO:teuthology.orchestra.run.smithi114.stdout: "total_workers": 0, 2024-04-23T10:41:14.827 INFO:teuthology.orchestra.run.smithi114.stdout: "unhealthy_workers": 0 2024-04-23T10:41:14.828 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-23T10:41:14.828 INFO:teuthology.orchestra.run.smithi114.stdout: "mempool": { 2024-04-23T10:41:14.828 INFO:teuthology.orchestra.run.smithi114.stdout: "bloom_filter_bytes": 0, 2024-04-23T10:41:14.828 INFO:teuthology.orchestra.run.smithi114.stdout: "bloom_filter_items": 0, 2024-04-23T10:41:14.828 INFO:teuthology.orchestra.run.smithi114.stdout: "bluestore_alloc_bytes": 0, 2024-04-23T10:41:14.828 INFO:teuthology.orchestra.run.smithi114.stdout: "bluestore_alloc_items": 0, 2024-04-23T10:41:14.828 INFO:teuthology.orchestra.run.smithi114.stdout: "bluestore_cache_data_bytes": 0, 2024-04-23T10:41:14.828 INFO:teuthology.orchestra.run.smithi114.stdout: "bluestore_cache_data_items": 0, 2024-04-23T10:41:14.828 INFO:teuthology.orchestra.run.smithi114.stdout: "bluestore_cache_onode_bytes": 0, 2024-04-23T10:41:14.829 INFO:teuthology.orchestra.run.smithi114.stdout: "bluestore_cache_onode_items": 0, 2024-04-23T10:41:14.829 INFO:teuthology.orchestra.run.smithi114.stdout: "bluestore_cache_meta_bytes": 0, 2024-04-23T10:41:14.829 INFO:teuthology.orchestra.run.smithi114.stdout: "bluestore_cache_meta_items": 0, 2024-04-23T10:41:14.829 INFO:teuthology.orchestra.run.smithi114.stdout: "bluestore_cache_other_bytes": 0, 2024-04-23T10:41:14.829 INFO:teuthology.orchestra.run.smithi114.stdout: "bluestore_cache_other_items": 0, 2024-04-23T10:41:14.829 INFO:teuthology.orchestra.run.smithi114.stdout: "bluestore_cache_buffer_bytes": 0, 2024-04-23T10:41:14.829 INFO:teuthology.orchestra.run.smithi114.stdout: "bluestore_cache_buffer_items": 0, 2024-04-23T10:41:14.829 INFO:teuthology.orchestra.run.smithi114.stdout: "bluestore_extent_bytes": 0, 2024-04-23T10:41:14.830 INFO:teuthology.orchestra.run.smithi114.stdout: "bluestore_extent_items": 0, 2024-04-23T10:41:14.830 INFO:teuthology.orchestra.run.smithi114.stdout: "bluestore_blob_bytes": 0, 2024-04-23T10:41:14.830 INFO:teuthology.orchestra.run.smithi114.stdout: "bluestore_blob_items": 0, 2024-04-23T10:41:14.830 INFO:teuthology.orchestra.run.smithi114.stdout: "bluestore_shared_blob_bytes": 0, 2024-04-23T10:41:14.830 INFO:teuthology.orchestra.run.smithi114.stdout: "bluestore_shared_blob_items": 0, 2024-04-23T10:41:14.830 INFO:teuthology.orchestra.run.smithi114.stdout: "bluestore_inline_bl_bytes": 0, 2024-04-23T10:41:14.830 INFO:teuthology.orchestra.run.smithi114.stdout: "bluestore_inline_bl_items": 0, 2024-04-23T10:41:14.831 INFO:teuthology.orchestra.run.smithi114.stdout: "bluestore_fsck_bytes": 0, 2024-04-23T10:41:14.831 INFO:teuthology.orchestra.run.smithi114.stdout: "bluestore_fsck_items": 0, 2024-04-23T10:41:14.831 INFO:teuthology.orchestra.run.smithi114.stdout: "bluestore_txc_bytes": 0, 2024-04-23T10:41:14.831 INFO:teuthology.orchestra.run.smithi114.stdout: "bluestore_txc_items": 0, 2024-04-23T10:41:14.831 INFO:teuthology.orchestra.run.smithi114.stdout: "bluestore_writing_deferred_bytes": 0, 2024-04-23T10:41:14.831 INFO:teuthology.orchestra.run.smithi114.stdout: "bluestore_writing_deferred_items": 0, 2024-04-23T10:41:14.831 INFO:teuthology.orchestra.run.smithi114.stdout: "bluestore_writing_bytes": 0, 2024-04-23T10:41:14.831 INFO:teuthology.orchestra.run.smithi114.stdout: "bluestore_writing_items": 0, 2024-04-23T10:41:14.831 INFO:teuthology.orchestra.run.smithi114.stdout: "bluefs_bytes": 0, 2024-04-23T10:41:14.832 INFO:teuthology.orchestra.run.smithi114.stdout: "bluefs_items": 0, 2024-04-23T10:41:14.832 INFO:teuthology.orchestra.run.smithi114.stdout: "bluefs_file_reader_bytes": 0, 2024-04-23T10:41:14.832 INFO:teuthology.orchestra.run.smithi114.stdout: "bluefs_file_reader_items": 0, 2024-04-23T10:41:14.832 INFO:teuthology.orchestra.run.smithi114.stdout: "bluefs_file_writer_bytes": 0, 2024-04-23T10:41:14.832 INFO:teuthology.orchestra.run.smithi114.stdout: "bluefs_file_writer_items": 0, 2024-04-23T10:41:14.832 INFO:teuthology.orchestra.run.smithi114.stdout: "buffer_anon_bytes": 148903, 2024-04-23T10:41:14.832 INFO:teuthology.orchestra.run.smithi114.stdout: "buffer_anon_items": 27, 2024-04-23T10:41:14.833 INFO:teuthology.orchestra.run.smithi114.stdout: "buffer_meta_bytes": 0, 2024-04-23T10:41:14.833 INFO:teuthology.orchestra.run.smithi114.stdout: "buffer_meta_items": 0, 2024-04-23T10:41:14.833 INFO:teuthology.orchestra.run.smithi114.stdout: "osd_bytes": 0, 2024-04-23T10:41:14.833 INFO:teuthology.orchestra.run.smithi114.stdout: "osd_items": 0, 2024-04-23T10:41:14.833 INFO:teuthology.orchestra.run.smithi114.stdout: "osd_mapbl_bytes": 0, 2024-04-23T10:41:14.833 INFO:teuthology.orchestra.run.smithi114.stdout: "osd_mapbl_items": 0, 2024-04-23T10:41:14.833 INFO:teuthology.orchestra.run.smithi114.stdout: "osd_pglog_bytes": 0, 2024-04-23T10:41:14.833 INFO:teuthology.orchestra.run.smithi114.stdout: "osd_pglog_items": 0, 2024-04-23T10:41:14.833 INFO:teuthology.orchestra.run.smithi114.stdout: "osdmap_bytes": 0, 2024-04-23T10:41:14.834 INFO:teuthology.orchestra.run.smithi114.stdout: "osdmap_items": 0, 2024-04-23T10:41:14.834 INFO:teuthology.orchestra.run.smithi114.stdout: "osdmap_mapping_bytes": 0, 2024-04-23T10:41:14.834 INFO:teuthology.orchestra.run.smithi114.stdout: "osdmap_mapping_items": 0, 2024-04-23T10:41:14.834 INFO:teuthology.orchestra.run.smithi114.stdout: "pgmap_bytes": 0, 2024-04-23T10:41:14.834 INFO:teuthology.orchestra.run.smithi114.stdout: "pgmap_items": 0, 2024-04-23T10:41:14.834 INFO:teuthology.orchestra.run.smithi114.stdout: "mds_co_bytes": 928, 2024-04-23T10:41:14.834 INFO:teuthology.orchestra.run.smithi114.stdout: "mds_co_items": 2, 2024-04-23T10:41:14.834 INFO:teuthology.orchestra.run.smithi114.stdout: "unittest_1_bytes": 0, 2024-04-23T10:41:14.834 INFO:teuthology.orchestra.run.smithi114.stdout: "unittest_1_items": 0, 2024-04-23T10:41:14.835 INFO:teuthology.orchestra.run.smithi114.stdout: "unittest_2_bytes": 0, 2024-04-23T10:41:14.835 INFO:teuthology.orchestra.run.smithi114.stdout: "unittest_2_items": 0 2024-04-23T10:41:14.835 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-23T10:41:14.835 INFO:teuthology.orchestra.run.smithi114.stdout: "throttle-msgr_dispatch_throttler-mds": { 2024-04-23T10:41:14.835 INFO:teuthology.orchestra.run.smithi114.stdout: "val": 0, 2024-04-23T10:41:14.835 INFO:teuthology.orchestra.run.smithi114.stdout: "max": 104857600, 2024-04-23T10:41:14.835 INFO:teuthology.orchestra.run.smithi114.stdout: "get_started": 0, 2024-04-23T10:41:14.835 INFO:teuthology.orchestra.run.smithi114.stdout: "get": 287, 2024-04-23T10:41:14.835 INFO:teuthology.orchestra.run.smithi114.stdout: "get_sum": 1567269, 2024-04-23T10:41:14.836 INFO:teuthology.orchestra.run.smithi114.stdout: "get_or_fail_fail": 0, 2024-04-23T10:41:14.836 INFO:teuthology.orchestra.run.smithi114.stdout: "get_or_fail_success": 287, 2024-04-23T10:41:14.836 INFO:teuthology.orchestra.run.smithi114.stdout: "take": 0, 2024-04-23T10:41:14.836 INFO:teuthology.orchestra.run.smithi114.stdout: "take_sum": 0, 2024-04-23T10:41:14.836 INFO:teuthology.orchestra.run.smithi114.stdout: "put": 287, 2024-04-23T10:41:14.836 INFO:teuthology.orchestra.run.smithi114.stdout: "put_sum": 1567269, 2024-04-23T10:41:14.836 INFO:teuthology.orchestra.run.smithi114.stdout: "wait": { 2024-04-23T10:41:14.836 INFO:teuthology.orchestra.run.smithi114.stdout: "avgcount": 0, 2024-04-23T10:41:14.836 INFO:teuthology.orchestra.run.smithi114.stdout: "sum": 0.000000000, 2024-04-23T10:41:14.836 INFO:teuthology.orchestra.run.smithi114.stdout: "avgtime": 0.000000000 2024-04-23T10:41:14.837 INFO:teuthology.orchestra.run.smithi114.stdout: } 2024-04-23T10:41:14.837 INFO:teuthology.orchestra.run.smithi114.stdout: } 2024-04-23T10:41:14.837 INFO:teuthology.orchestra.run.smithi114.stdout:} 2024-04-23T10:41:15.450 WARNING:tasks.check_counter:Counter 'mds.exported' not found on daemon mds.d 2024-04-23T10:41:15.450 WARNING:tasks.check_counter:Counter 'mds.imported' not found on daemon mds.d 2024-04-23T10:41:15.450 WARNING:tasks.check_counter:Counter 'mds_cache.dir_update' not found on daemon mds.d 2024-04-23T10:41:15.451 WARNING:tasks.check_counter:Counter 'mds_cache.dir_update_receipt' not found on daemon mds.d 2024-04-23T10:41:15.451 WARNING:tasks.check_counter:Counter 'mds.root_rsnaps' not found on daemon mds.d 2024-04-23T10:41:15.451 WARNING:tasks.check_counter:Counter 'mds_server.req_mksnap_latency.avgcount' not found on daemon mds.d 2024-04-23T10:41:15.451 WARNING:tasks.check_counter:Counter 'mds_server.req_rmsnap_latency.avgcount' not found on daemon mds.d 2024-04-23T10:41:15.451 DEBUG:tasks.check_counter:Getting stats from g 2024-04-23T10:41:15.452 DEBUG:teuthology.orchestra.run.smithi114:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d69f13d827952bb07bb9b7973bca14f97d764d6f shell --fsid aa240572-0159-11ef-bc93-c7b262605968 -- ceph daemon mds.g perf dump 2024-04-23T10:41:15.693 INFO:teuthology.orchestra.run.smithi114.stderr:Inferring config /var/lib/ceph/aa240572-0159-11ef-bc93-c7b262605968/mon.a/config 2024-04-23T10:41:16.504 INFO:journalctl@ceph.mon.a.smithi114.stdout:Apr 23 10:41:16 smithi114 ceph-mon[29255]: pgmap v463: 97 pgs: 97 active+clean; 558 MiB data, 4.3 GiB used, 1.0 TiB / 1.0 TiB avail; 152 KiB/s rd, 2.8 MiB/s wr, 84 op/s 2024-04-23T10:41:16.578 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 23 10:41:16 smithi204 ceph-mon[32649]: pgmap v463: 97 pgs: 97 active+clean; 558 MiB data, 4.3 GiB used, 1.0 TiB / 1.0 TiB avail; 152 KiB/s rd, 2.8 MiB/s wr, 84 op/s 2024-04-23T10:41:16.742 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 23 10:41:16 smithi119 ceph-mon[32608]: pgmap v463: 97 pgs: 97 active+clean; 558 MiB data, 4.3 GiB used, 1.0 TiB / 1.0 TiB avail; 152 KiB/s rd, 2.8 MiB/s wr, 84 op/s 2024-04-23T10:41:17.316 INFO:teuthology.orchestra.run.smithi114.stdout:{ 2024-04-23T10:41:17.316 INFO:teuthology.orchestra.run.smithi114.stdout: "AsyncMessenger::Worker-0": { 2024-04-23T10:41:17.317 INFO:teuthology.orchestra.run.smithi114.stdout: "msgr_recv_messages": 280, 2024-04-23T10:41:17.317 INFO:teuthology.orchestra.run.smithi114.stdout: "msgr_send_messages": 269, 2024-04-23T10:41:17.317 INFO:teuthology.orchestra.run.smithi114.stdout: "msgr_recv_bytes": 1606256, 2024-04-23T10:41:17.317 INFO:teuthology.orchestra.run.smithi114.stdout: "msgr_send_bytes": 165824, 2024-04-23T10:41:17.317 INFO:teuthology.orchestra.run.smithi114.stdout: "msgr_created_connections": 4, 2024-04-23T10:41:17.317 INFO:teuthology.orchestra.run.smithi114.stdout: "msgr_active_connections": 0, 2024-04-23T10:41:17.317 INFO:teuthology.orchestra.run.smithi114.stdout: "msgr_running_total_time": 0.049374355, 2024-04-23T10:41:17.318 INFO:teuthology.orchestra.run.smithi114.stdout: "msgr_running_send_time": 0.015763859, 2024-04-23T10:41:17.318 INFO:teuthology.orchestra.run.smithi114.stdout: "msgr_running_recv_time": 0.036229790, 2024-04-23T10:41:17.318 INFO:teuthology.orchestra.run.smithi114.stdout: "msgr_running_fast_dispatch_time": 0.005075914, 2024-04-23T10:41:17.318 INFO:teuthology.orchestra.run.smithi114.stdout: "msgr_send_messages_queue_lat": { 2024-04-23T10:41:17.318 INFO:teuthology.orchestra.run.smithi114.stdout: "avgcount": 268, 2024-04-23T10:41:17.319 INFO:teuthology.orchestra.run.smithi114.stdout: "sum": 0.012768908, 2024-04-23T10:41:17.319 INFO:teuthology.orchestra.run.smithi114.stdout: "avgtime": 0.000047645 2024-04-23T10:41:17.319 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-23T10:41:17.319 INFO:teuthology.orchestra.run.smithi114.stdout: "msgr_handle_ack_lat": { 2024-04-23T10:41:17.319 INFO:teuthology.orchestra.run.smithi114.stdout: "avgcount": 0, 2024-04-23T10:41:17.319 INFO:teuthology.orchestra.run.smithi114.stdout: "sum": 0.000000000, 2024-04-23T10:41:17.319 INFO:teuthology.orchestra.run.smithi114.stdout: "avgtime": 0.000000000 2024-04-23T10:41:17.320 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-23T10:41:17.320 INFO:teuthology.orchestra.run.smithi114.stdout: "msgr_recv_encrypted_bytes": 1606256, 2024-04-23T10:41:17.320 INFO:teuthology.orchestra.run.smithi114.stdout: "msgr_send_encrypted_bytes": 165824 2024-04-23T10:41:17.320 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-23T10:41:17.320 INFO:teuthology.orchestra.run.smithi114.stdout: "AsyncMessenger::Worker-1": { 2024-04-23T10:41:17.320 INFO:teuthology.orchestra.run.smithi114.stdout: "msgr_recv_messages": 6, 2024-04-23T10:41:17.321 INFO:teuthology.orchestra.run.smithi114.stdout: "msgr_send_messages": 208, 2024-04-23T10:41:17.321 INFO:teuthology.orchestra.run.smithi114.stdout: "msgr_recv_bytes": 3312, 2024-04-23T10:41:17.321 INFO:teuthology.orchestra.run.smithi114.stdout: "msgr_send_bytes": 192784, 2024-04-23T10:41:17.321 INFO:teuthology.orchestra.run.smithi114.stdout: "msgr_created_connections": 5, 2024-04-23T10:41:17.321 INFO:teuthology.orchestra.run.smithi114.stdout: "msgr_active_connections": 1, 2024-04-23T10:41:17.321 INFO:teuthology.orchestra.run.smithi114.stdout: "msgr_running_total_time": 0.015240017, 2024-04-23T10:41:17.321 INFO:teuthology.orchestra.run.smithi114.stdout: "msgr_running_send_time": 0.010632039, 2024-04-23T10:41:17.322 INFO:teuthology.orchestra.run.smithi114.stdout: "msgr_running_recv_time": 0.002103532, 2024-04-23T10:41:17.322 INFO:teuthology.orchestra.run.smithi114.stdout: "msgr_running_fast_dispatch_time": 0.000000000, 2024-04-23T10:41:17.322 INFO:teuthology.orchestra.run.smithi114.stdout: "msgr_send_messages_queue_lat": { 2024-04-23T10:41:17.322 INFO:teuthology.orchestra.run.smithi114.stdout: "avgcount": 206, 2024-04-23T10:41:17.322 INFO:teuthology.orchestra.run.smithi114.stdout: "sum": 0.016539711, 2024-04-23T10:41:17.323 INFO:teuthology.orchestra.run.smithi114.stdout: "avgtime": 0.000080289 2024-04-23T10:41:17.323 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-23T10:41:17.323 INFO:teuthology.orchestra.run.smithi114.stdout: "msgr_handle_ack_lat": { 2024-04-23T10:41:17.323 INFO:teuthology.orchestra.run.smithi114.stdout: "avgcount": 0, 2024-04-23T10:41:17.323 INFO:teuthology.orchestra.run.smithi114.stdout: "sum": 0.000000000, 2024-04-23T10:41:17.323 INFO:teuthology.orchestra.run.smithi114.stdout: "avgtime": 0.000000000 2024-04-23T10:41:17.323 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-23T10:41:17.324 INFO:teuthology.orchestra.run.smithi114.stdout: "msgr_recv_encrypted_bytes": 3312, 2024-04-23T10:41:17.324 INFO:teuthology.orchestra.run.smithi114.stdout: "msgr_send_encrypted_bytes": 192784 2024-04-23T10:41:17.324 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-23T10:41:17.324 INFO:teuthology.orchestra.run.smithi114.stdout: "AsyncMessenger::Worker-2": { 2024-04-23T10:41:17.324 INFO:teuthology.orchestra.run.smithi114.stdout: "msgr_recv_messages": 2, 2024-04-23T10:41:17.324 INFO:teuthology.orchestra.run.smithi114.stdout: "msgr_send_messages": 4, 2024-04-23T10:41:17.325 INFO:teuthology.orchestra.run.smithi114.stdout: "msgr_recv_bytes": 303, 2024-04-23T10:41:17.325 INFO:teuthology.orchestra.run.smithi114.stdout: "msgr_send_bytes": 23311, 2024-04-23T10:41:17.325 INFO:teuthology.orchestra.run.smithi114.stdout: "msgr_created_connections": 4, 2024-04-23T10:41:17.325 INFO:teuthology.orchestra.run.smithi114.stdout: "msgr_active_connections": 0, 2024-04-23T10:41:17.325 INFO:teuthology.orchestra.run.smithi114.stdout: "msgr_running_total_time": 0.001154476, 2024-04-23T10:41:17.325 INFO:teuthology.orchestra.run.smithi114.stdout: "msgr_running_send_time": 0.000062483, 2024-04-23T10:41:17.325 INFO:teuthology.orchestra.run.smithi114.stdout: "msgr_running_recv_time": 0.000966295, 2024-04-23T10:41:17.326 INFO:teuthology.orchestra.run.smithi114.stdout: "msgr_running_fast_dispatch_time": 0.000000000, 2024-04-23T10:41:17.326 INFO:teuthology.orchestra.run.smithi114.stdout: "msgr_send_messages_queue_lat": { 2024-04-23T10:41:17.326 INFO:teuthology.orchestra.run.smithi114.stdout: "avgcount": 2, 2024-04-23T10:41:17.326 INFO:teuthology.orchestra.run.smithi114.stdout: "sum": 0.000018570, 2024-04-23T10:41:17.326 INFO:teuthology.orchestra.run.smithi114.stdout: "avgtime": 0.000009285 2024-04-23T10:41:17.326 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-23T10:41:17.327 INFO:teuthology.orchestra.run.smithi114.stdout: "msgr_handle_ack_lat": { 2024-04-23T10:41:17.327 INFO:teuthology.orchestra.run.smithi114.stdout: "avgcount": 2, 2024-04-23T10:41:17.327 INFO:teuthology.orchestra.run.smithi114.stdout: "sum": 0.000000193, 2024-04-23T10:41:17.327 INFO:teuthology.orchestra.run.smithi114.stdout: "avgtime": 0.000000096 2024-04-23T10:41:17.327 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-23T10:41:17.327 INFO:teuthology.orchestra.run.smithi114.stdout: "msgr_recv_encrypted_bytes": 0, 2024-04-23T10:41:17.327 INFO:teuthology.orchestra.run.smithi114.stdout: "msgr_send_encrypted_bytes": 0 2024-04-23T10:41:17.328 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-23T10:41:17.328 INFO:teuthology.orchestra.run.smithi114.stdout: "cct": { 2024-04-23T10:41:17.328 INFO:teuthology.orchestra.run.smithi114.stdout: "total_workers": 0, 2024-04-23T10:41:17.328 INFO:teuthology.orchestra.run.smithi114.stdout: "unhealthy_workers": 0 2024-04-23T10:41:17.328 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-23T10:41:17.328 INFO:teuthology.orchestra.run.smithi114.stdout: "mempool": { 2024-04-23T10:41:17.328 INFO:teuthology.orchestra.run.smithi114.stdout: "bloom_filter_bytes": 0, 2024-04-23T10:41:17.329 INFO:teuthology.orchestra.run.smithi114.stdout: "bloom_filter_items": 0, 2024-04-23T10:41:17.329 INFO:teuthology.orchestra.run.smithi114.stdout: "bluestore_alloc_bytes": 0, 2024-04-23T10:41:17.329 INFO:teuthology.orchestra.run.smithi114.stdout: "bluestore_alloc_items": 0, 2024-04-23T10:41:17.329 INFO:teuthology.orchestra.run.smithi114.stdout: "bluestore_cache_data_bytes": 0, 2024-04-23T10:41:17.329 INFO:teuthology.orchestra.run.smithi114.stdout: "bluestore_cache_data_items": 0, 2024-04-23T10:41:17.329 INFO:teuthology.orchestra.run.smithi114.stdout: "bluestore_cache_onode_bytes": 0, 2024-04-23T10:41:17.329 INFO:teuthology.orchestra.run.smithi114.stdout: "bluestore_cache_onode_items": 0, 2024-04-23T10:41:17.330 INFO:teuthology.orchestra.run.smithi114.stdout: "bluestore_cache_meta_bytes": 0, 2024-04-23T10:41:17.330 INFO:teuthology.orchestra.run.smithi114.stdout: "bluestore_cache_meta_items": 0, 2024-04-23T10:41:17.330 INFO:teuthology.orchestra.run.smithi114.stdout: "bluestore_cache_other_bytes": 0, 2024-04-23T10:41:17.330 INFO:teuthology.orchestra.run.smithi114.stdout: "bluestore_cache_other_items": 0, 2024-04-23T10:41:17.330 INFO:teuthology.orchestra.run.smithi114.stdout: "bluestore_cache_buffer_bytes": 0, 2024-04-23T10:41:17.330 INFO:teuthology.orchestra.run.smithi114.stdout: "bluestore_cache_buffer_items": 0, 2024-04-23T10:41:17.331 INFO:teuthology.orchestra.run.smithi114.stdout: "bluestore_extent_bytes": 0, 2024-04-23T10:41:17.331 INFO:teuthology.orchestra.run.smithi114.stdout: "bluestore_extent_items": 0, 2024-04-23T10:41:17.331 INFO:teuthology.orchestra.run.smithi114.stdout: "bluestore_blob_bytes": 0, 2024-04-23T10:41:17.331 INFO:teuthology.orchestra.run.smithi114.stdout: "bluestore_blob_items": 0, 2024-04-23T10:41:17.331 INFO:teuthology.orchestra.run.smithi114.stdout: "bluestore_shared_blob_bytes": 0, 2024-04-23T10:41:17.331 INFO:teuthology.orchestra.run.smithi114.stdout: "bluestore_shared_blob_items": 0, 2024-04-23T10:41:17.332 INFO:teuthology.orchestra.run.smithi114.stdout: "bluestore_inline_bl_bytes": 0, 2024-04-23T10:41:17.332 INFO:teuthology.orchestra.run.smithi114.stdout: "bluestore_inline_bl_items": 0, 2024-04-23T10:41:17.332 INFO:teuthology.orchestra.run.smithi114.stdout: "bluestore_fsck_bytes": 0, 2024-04-23T10:41:17.332 INFO:teuthology.orchestra.run.smithi114.stdout: "bluestore_fsck_items": 0, 2024-04-23T10:41:17.332 INFO:teuthology.orchestra.run.smithi114.stdout: "bluestore_txc_bytes": 0, 2024-04-23T10:41:17.332 INFO:teuthology.orchestra.run.smithi114.stdout: "bluestore_txc_items": 0, 2024-04-23T10:41:17.333 INFO:teuthology.orchestra.run.smithi114.stdout: "bluestore_writing_deferred_bytes": 0, 2024-04-23T10:41:17.333 INFO:teuthology.orchestra.run.smithi114.stdout: "bluestore_writing_deferred_items": 0, 2024-04-23T10:41:17.333 INFO:teuthology.orchestra.run.smithi114.stdout: "bluestore_writing_bytes": 0, 2024-04-23T10:41:17.333 INFO:teuthology.orchestra.run.smithi114.stdout: "bluestore_writing_items": 0, 2024-04-23T10:41:17.333 INFO:teuthology.orchestra.run.smithi114.stdout: "bluefs_bytes": 0, 2024-04-23T10:41:17.333 INFO:teuthology.orchestra.run.smithi114.stdout: "bluefs_items": 0, 2024-04-23T10:41:17.333 INFO:teuthology.orchestra.run.smithi114.stdout: "bluefs_file_reader_bytes": 0, 2024-04-23T10:41:17.334 INFO:teuthology.orchestra.run.smithi114.stdout: "bluefs_file_reader_items": 0, 2024-04-23T10:41:17.334 INFO:teuthology.orchestra.run.smithi114.stdout: "bluefs_file_writer_bytes": 0, 2024-04-23T10:41:17.334 INFO:teuthology.orchestra.run.smithi114.stdout: "bluefs_file_writer_items": 0, 2024-04-23T10:41:17.334 INFO:teuthology.orchestra.run.smithi114.stdout: "buffer_anon_bytes": 148903, 2024-04-23T10:41:17.334 INFO:teuthology.orchestra.run.smithi114.stdout: "buffer_anon_items": 27, 2024-04-23T10:41:17.334 INFO:teuthology.orchestra.run.smithi114.stdout: "buffer_meta_bytes": 0, 2024-04-23T10:41:17.335 INFO:teuthology.orchestra.run.smithi114.stdout: "buffer_meta_items": 0, 2024-04-23T10:41:17.335 INFO:teuthology.orchestra.run.smithi114.stdout: "osd_bytes": 0, 2024-04-23T10:41:17.335 INFO:teuthology.orchestra.run.smithi114.stdout: "osd_items": 0, 2024-04-23T10:41:17.335 INFO:teuthology.orchestra.run.smithi114.stdout: "osd_mapbl_bytes": 0, 2024-04-23T10:41:17.335 INFO:teuthology.orchestra.run.smithi114.stdout: "osd_mapbl_items": 0, 2024-04-23T10:41:17.335 INFO:teuthology.orchestra.run.smithi114.stdout: "osd_pglog_bytes": 0, 2024-04-23T10:41:17.335 INFO:teuthology.orchestra.run.smithi114.stdout: "osd_pglog_items": 0, 2024-04-23T10:41:17.336 INFO:teuthology.orchestra.run.smithi114.stdout: "osdmap_bytes": 0, 2024-04-23T10:41:17.336 INFO:teuthology.orchestra.run.smithi114.stdout: "osdmap_items": 0, 2024-04-23T10:41:17.336 INFO:teuthology.orchestra.run.smithi114.stdout: "osdmap_mapping_bytes": 0, 2024-04-23T10:41:17.336 INFO:teuthology.orchestra.run.smithi114.stdout: "osdmap_mapping_items": 0, 2024-04-23T10:41:17.336 INFO:teuthology.orchestra.run.smithi114.stdout: "pgmap_bytes": 0, 2024-04-23T10:41:17.336 INFO:teuthology.orchestra.run.smithi114.stdout: "pgmap_items": 0, 2024-04-23T10:41:17.336 INFO:teuthology.orchestra.run.smithi114.stdout: "mds_co_bytes": 928, 2024-04-23T10:41:17.337 INFO:teuthology.orchestra.run.smithi114.stdout: "mds_co_items": 2, 2024-04-23T10:41:17.337 INFO:teuthology.orchestra.run.smithi114.stdout: "unittest_1_bytes": 0, 2024-04-23T10:41:17.337 INFO:teuthology.orchestra.run.smithi114.stdout: "unittest_1_items": 0, 2024-04-23T10:41:17.337 INFO:teuthology.orchestra.run.smithi114.stdout: "unittest_2_bytes": 0, 2024-04-23T10:41:17.337 INFO:teuthology.orchestra.run.smithi114.stdout: "unittest_2_items": 0 2024-04-23T10:41:17.337 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-23T10:41:17.337 INFO:teuthology.orchestra.run.smithi114.stdout: "throttle-msgr_dispatch_throttler-mds": { 2024-04-23T10:41:17.338 INFO:teuthology.orchestra.run.smithi114.stdout: "val": 0, 2024-04-23T10:41:17.338 INFO:teuthology.orchestra.run.smithi114.stdout: "max": 104857600, 2024-04-23T10:41:17.338 INFO:teuthology.orchestra.run.smithi114.stdout: "get_started": 0, 2024-04-23T10:41:17.338 INFO:teuthology.orchestra.run.smithi114.stdout: "get": 284, 2024-04-23T10:41:17.338 INFO:teuthology.orchestra.run.smithi114.stdout: "get_sum": 1566879, 2024-04-23T10:41:17.338 INFO:teuthology.orchestra.run.smithi114.stdout: "get_or_fail_fail": 0, 2024-04-23T10:41:17.338 INFO:teuthology.orchestra.run.smithi114.stdout: "get_or_fail_success": 284, 2024-04-23T10:41:17.339 INFO:teuthology.orchestra.run.smithi114.stdout: "take": 0, 2024-04-23T10:41:17.339 INFO:teuthology.orchestra.run.smithi114.stdout: "take_sum": 0, 2024-04-23T10:41:17.339 INFO:teuthology.orchestra.run.smithi114.stdout: "put": 284, 2024-04-23T10:41:17.339 INFO:teuthology.orchestra.run.smithi114.stdout: "put_sum": 1566879, 2024-04-23T10:41:17.339 INFO:teuthology.orchestra.run.smithi114.stdout: "wait": { 2024-04-23T10:41:17.339 INFO:teuthology.orchestra.run.smithi114.stdout: "avgcount": 0, 2024-04-23T10:41:17.339 INFO:teuthology.orchestra.run.smithi114.stdout: "sum": 0.000000000, 2024-04-23T10:41:17.340 INFO:teuthology.orchestra.run.smithi114.stdout: "avgtime": 0.000000000 2024-04-23T10:41:17.340 INFO:teuthology.orchestra.run.smithi114.stdout: } 2024-04-23T10:41:17.340 INFO:teuthology.orchestra.run.smithi114.stdout: } 2024-04-23T10:41:17.340 INFO:teuthology.orchestra.run.smithi114.stdout:} 2024-04-23T10:41:17.940 WARNING:tasks.check_counter:Counter 'mds.exported' not found on daemon mds.g 2024-04-23T10:41:17.940 WARNING:tasks.check_counter:Counter 'mds.imported' not found on daemon mds.g 2024-04-23T10:41:17.941 WARNING:tasks.check_counter:Counter 'mds_cache.dir_update' not found on daemon mds.g 2024-04-23T10:41:17.941 WARNING:tasks.check_counter:Counter 'mds_cache.dir_update_receipt' not found on daemon mds.g 2024-04-23T10:41:17.941 WARNING:tasks.check_counter:Counter 'mds.root_rsnaps' not found on daemon mds.g 2024-04-23T10:41:17.941 WARNING:tasks.check_counter:Counter 'mds_server.req_mksnap_latency.avgcount' not found on daemon mds.g 2024-04-23T10:41:17.941 WARNING:tasks.check_counter:Counter 'mds_server.req_rmsnap_latency.avgcount' not found on daemon mds.g 2024-04-23T10:41:17.941 DEBUG:tasks.check_counter:Getting stats from j 2024-04-23T10:41:17.942 DEBUG:teuthology.orchestra.run.smithi114:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d69f13d827952bb07bb9b7973bca14f97d764d6f shell --fsid aa240572-0159-11ef-bc93-c7b262605968 -- ceph daemon mds.j perf dump 2024-04-23T10:41:18.199 INFO:teuthology.orchestra.run.smithi114.stderr:Inferring config /var/lib/ceph/aa240572-0159-11ef-bc93-c7b262605968/mon.a/config 2024-04-23T10:41:18.535 INFO:journalctl@ceph.mon.a.smithi114.stdout:Apr 23 10:41:18 smithi114 ceph-mon[29255]: pgmap v464: 97 pgs: 97 active+clean; 564 MiB data, 4.4 GiB used, 1.0 TiB / 1.0 TiB avail; 447 KiB/s rd, 5.9 MiB/s wr, 189 op/s 2024-04-23T10:41:18.578 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 23 10:41:18 smithi204 ceph-mon[32649]: pgmap v464: 97 pgs: 97 active+clean; 564 MiB data, 4.4 GiB used, 1.0 TiB / 1.0 TiB avail; 447 KiB/s rd, 5.9 MiB/s wr, 189 op/s 2024-04-23T10:41:18.742 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 23 10:41:18 smithi119 ceph-mon[32608]: pgmap v464: 97 pgs: 97 active+clean; 564 MiB data, 4.4 GiB used, 1.0 TiB / 1.0 TiB avail; 447 KiB/s rd, 5.9 MiB/s wr, 189 op/s 2024-04-23T10:41:19.966 INFO:teuthology.orchestra.run.smithi114.stdout:{ 2024-04-23T10:41:19.966 INFO:teuthology.orchestra.run.smithi114.stdout: "AsyncMessenger::Worker-0": { 2024-04-23T10:41:19.967 INFO:teuthology.orchestra.run.smithi114.stdout: "msgr_recv_messages": 285, 2024-04-23T10:41:19.967 INFO:teuthology.orchestra.run.smithi114.stdout: "msgr_send_messages": 274, 2024-04-23T10:41:19.967 INFO:teuthology.orchestra.run.smithi114.stdout: "msgr_recv_bytes": 1607616, 2024-04-23T10:41:19.967 INFO:teuthology.orchestra.run.smithi114.stdout: "msgr_send_bytes": 169024, 2024-04-23T10:41:19.967 INFO:teuthology.orchestra.run.smithi114.stdout: "msgr_created_connections": 4, 2024-04-23T10:41:19.967 INFO:teuthology.orchestra.run.smithi114.stdout: "msgr_active_connections": 0, 2024-04-23T10:41:19.967 INFO:teuthology.orchestra.run.smithi114.stdout: "msgr_running_total_time": 0.051164880, 2024-04-23T10:41:19.967 INFO:teuthology.orchestra.run.smithi114.stdout: "msgr_running_send_time": 0.016220899, 2024-04-23T10:41:19.967 INFO:teuthology.orchestra.run.smithi114.stdout: "msgr_running_recv_time": 0.037307681, 2024-04-23T10:41:19.968 INFO:teuthology.orchestra.run.smithi114.stdout: "msgr_running_fast_dispatch_time": 0.005239142, 2024-04-23T10:41:19.968 INFO:teuthology.orchestra.run.smithi114.stdout: "msgr_send_messages_queue_lat": { 2024-04-23T10:41:19.968 INFO:teuthology.orchestra.run.smithi114.stdout: "avgcount": 273, 2024-04-23T10:41:19.968 INFO:teuthology.orchestra.run.smithi114.stdout: "sum": 0.011185773, 2024-04-23T10:41:19.968 INFO:teuthology.orchestra.run.smithi114.stdout: "avgtime": 0.000040973 2024-04-23T10:41:19.968 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-23T10:41:19.968 INFO:teuthology.orchestra.run.smithi114.stdout: "msgr_handle_ack_lat": { 2024-04-23T10:41:19.968 INFO:teuthology.orchestra.run.smithi114.stdout: "avgcount": 0, 2024-04-23T10:41:19.968 INFO:teuthology.orchestra.run.smithi114.stdout: "sum": 0.000000000, 2024-04-23T10:41:19.968 INFO:teuthology.orchestra.run.smithi114.stdout: "avgtime": 0.000000000 2024-04-23T10:41:19.969 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-23T10:41:19.969 INFO:teuthology.orchestra.run.smithi114.stdout: "msgr_recv_encrypted_bytes": 1607616, 2024-04-23T10:41:19.969 INFO:teuthology.orchestra.run.smithi114.stdout: "msgr_send_encrypted_bytes": 169024 2024-04-23T10:41:19.969 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-23T10:41:19.969 INFO:teuthology.orchestra.run.smithi114.stdout: "AsyncMessenger::Worker-1": { 2024-04-23T10:41:19.969 INFO:teuthology.orchestra.run.smithi114.stdout: "msgr_recv_messages": 6, 2024-04-23T10:41:19.969 INFO:teuthology.orchestra.run.smithi114.stdout: "msgr_send_messages": 211, 2024-04-23T10:41:19.969 INFO:teuthology.orchestra.run.smithi114.stdout: "msgr_recv_bytes": 3312, 2024-04-23T10:41:19.969 INFO:teuthology.orchestra.run.smithi114.stdout: "msgr_send_bytes": 193312, 2024-04-23T10:41:19.970 INFO:teuthology.orchestra.run.smithi114.stdout: "msgr_created_connections": 5, 2024-04-23T10:41:19.970 INFO:teuthology.orchestra.run.smithi114.stdout: "msgr_active_connections": 1, 2024-04-23T10:41:19.970 INFO:teuthology.orchestra.run.smithi114.stdout: "msgr_running_total_time": 0.015933690, 2024-04-23T10:41:19.970 INFO:teuthology.orchestra.run.smithi114.stdout: "msgr_running_send_time": 0.010696379, 2024-04-23T10:41:19.970 INFO:teuthology.orchestra.run.smithi114.stdout: "msgr_running_recv_time": 0.002659843, 2024-04-23T10:41:19.970 INFO:teuthology.orchestra.run.smithi114.stdout: "msgr_running_fast_dispatch_time": 0.000000000, 2024-04-23T10:41:19.970 INFO:teuthology.orchestra.run.smithi114.stdout: "msgr_send_messages_queue_lat": { 2024-04-23T10:41:19.970 INFO:teuthology.orchestra.run.smithi114.stdout: "avgcount": 209, 2024-04-23T10:41:19.970 INFO:teuthology.orchestra.run.smithi114.stdout: "sum": 0.015136235, 2024-04-23T10:41:19.971 INFO:teuthology.orchestra.run.smithi114.stdout: "avgtime": 0.000072422 2024-04-23T10:41:19.971 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-23T10:41:19.971 INFO:teuthology.orchestra.run.smithi114.stdout: "msgr_handle_ack_lat": { 2024-04-23T10:41:19.971 INFO:teuthology.orchestra.run.smithi114.stdout: "avgcount": 0, 2024-04-23T10:41:19.971 INFO:teuthology.orchestra.run.smithi114.stdout: "sum": 0.000000000, 2024-04-23T10:41:19.971 INFO:teuthology.orchestra.run.smithi114.stdout: "avgtime": 0.000000000 2024-04-23T10:41:19.971 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-23T10:41:19.971 INFO:teuthology.orchestra.run.smithi114.stdout: "msgr_recv_encrypted_bytes": 3312, 2024-04-23T10:41:19.971 INFO:teuthology.orchestra.run.smithi114.stdout: "msgr_send_encrypted_bytes": 193312 2024-04-23T10:41:19.971 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-23T10:41:19.972 INFO:teuthology.orchestra.run.smithi114.stdout: "AsyncMessenger::Worker-2": { 2024-04-23T10:41:19.972 INFO:teuthology.orchestra.run.smithi114.stdout: "msgr_recv_messages": 2, 2024-04-23T10:41:19.972 INFO:teuthology.orchestra.run.smithi114.stdout: "msgr_send_messages": 4, 2024-04-23T10:41:19.972 INFO:teuthology.orchestra.run.smithi114.stdout: "msgr_recv_bytes": 303, 2024-04-23T10:41:19.972 INFO:teuthology.orchestra.run.smithi114.stdout: "msgr_send_bytes": 23311, 2024-04-23T10:41:19.972 INFO:teuthology.orchestra.run.smithi114.stdout: "msgr_created_connections": 4, 2024-04-23T10:41:19.972 INFO:teuthology.orchestra.run.smithi114.stdout: "msgr_active_connections": 0, 2024-04-23T10:41:19.972 INFO:teuthology.orchestra.run.smithi114.stdout: "msgr_running_total_time": 0.001129481, 2024-04-23T10:41:19.972 INFO:teuthology.orchestra.run.smithi114.stdout: "msgr_running_send_time": 0.000066600, 2024-04-23T10:41:19.973 INFO:teuthology.orchestra.run.smithi114.stdout: "msgr_running_recv_time": 0.000920287, 2024-04-23T10:41:19.973 INFO:teuthology.orchestra.run.smithi114.stdout: "msgr_running_fast_dispatch_time": 0.000000000, 2024-04-23T10:41:19.973 INFO:teuthology.orchestra.run.smithi114.stdout: "msgr_send_messages_queue_lat": { 2024-04-23T10:41:19.973 INFO:teuthology.orchestra.run.smithi114.stdout: "avgcount": 2, 2024-04-23T10:41:19.973 INFO:teuthology.orchestra.run.smithi114.stdout: "sum": 0.000021632, 2024-04-23T10:41:19.973 INFO:teuthology.orchestra.run.smithi114.stdout: "avgtime": 0.000010816 2024-04-23T10:41:19.973 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-23T10:41:19.973 INFO:teuthology.orchestra.run.smithi114.stdout: "msgr_handle_ack_lat": { 2024-04-23T10:41:19.973 INFO:teuthology.orchestra.run.smithi114.stdout: "avgcount": 2, 2024-04-23T10:41:19.973 INFO:teuthology.orchestra.run.smithi114.stdout: "sum": 0.000000234, 2024-04-23T10:41:19.974 INFO:teuthology.orchestra.run.smithi114.stdout: "avgtime": 0.000000117 2024-04-23T10:41:19.974 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-23T10:41:19.974 INFO:teuthology.orchestra.run.smithi114.stdout: "msgr_recv_encrypted_bytes": 0, 2024-04-23T10:41:19.974 INFO:teuthology.orchestra.run.smithi114.stdout: "msgr_send_encrypted_bytes": 0 2024-04-23T10:41:19.974 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-23T10:41:19.974 INFO:teuthology.orchestra.run.smithi114.stdout: "cct": { 2024-04-23T10:41:19.974 INFO:teuthology.orchestra.run.smithi114.stdout: "total_workers": 0, 2024-04-23T10:41:19.974 INFO:teuthology.orchestra.run.smithi114.stdout: "unhealthy_workers": 0 2024-04-23T10:41:19.974 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-23T10:41:19.975 INFO:teuthology.orchestra.run.smithi114.stdout: "mempool": { 2024-04-23T10:41:19.975 INFO:teuthology.orchestra.run.smithi114.stdout: "bloom_filter_bytes": 0, 2024-04-23T10:41:19.975 INFO:teuthology.orchestra.run.smithi114.stdout: "bloom_filter_items": 0, 2024-04-23T10:41:19.975 INFO:teuthology.orchestra.run.smithi114.stdout: "bluestore_alloc_bytes": 0, 2024-04-23T10:41:19.975 INFO:teuthology.orchestra.run.smithi114.stdout: "bluestore_alloc_items": 0, 2024-04-23T10:41:19.975 INFO:teuthology.orchestra.run.smithi114.stdout: "bluestore_cache_data_bytes": 0, 2024-04-23T10:41:19.975 INFO:teuthology.orchestra.run.smithi114.stdout: "bluestore_cache_data_items": 0, 2024-04-23T10:41:19.975 INFO:teuthology.orchestra.run.smithi114.stdout: "bluestore_cache_onode_bytes": 0, 2024-04-23T10:41:19.975 INFO:teuthology.orchestra.run.smithi114.stdout: "bluestore_cache_onode_items": 0, 2024-04-23T10:41:19.975 INFO:teuthology.orchestra.run.smithi114.stdout: "bluestore_cache_meta_bytes": 0, 2024-04-23T10:41:19.976 INFO:teuthology.orchestra.run.smithi114.stdout: "bluestore_cache_meta_items": 0, 2024-04-23T10:41:19.976 INFO:teuthology.orchestra.run.smithi114.stdout: "bluestore_cache_other_bytes": 0, 2024-04-23T10:41:19.976 INFO:teuthology.orchestra.run.smithi114.stdout: "bluestore_cache_other_items": 0, 2024-04-23T10:41:19.976 INFO:teuthology.orchestra.run.smithi114.stdout: "bluestore_cache_buffer_bytes": 0, 2024-04-23T10:41:19.976 INFO:teuthology.orchestra.run.smithi114.stdout: "bluestore_cache_buffer_items": 0, 2024-04-23T10:41:19.976 INFO:teuthology.orchestra.run.smithi114.stdout: "bluestore_extent_bytes": 0, 2024-04-23T10:41:19.976 INFO:teuthology.orchestra.run.smithi114.stdout: "bluestore_extent_items": 0, 2024-04-23T10:41:19.976 INFO:teuthology.orchestra.run.smithi114.stdout: "bluestore_blob_bytes": 0, 2024-04-23T10:41:19.976 INFO:teuthology.orchestra.run.smithi114.stdout: "bluestore_blob_items": 0, 2024-04-23T10:41:19.976 INFO:teuthology.orchestra.run.smithi114.stdout: "bluestore_shared_blob_bytes": 0, 2024-04-23T10:41:19.977 INFO:teuthology.orchestra.run.smithi114.stdout: "bluestore_shared_blob_items": 0, 2024-04-23T10:41:19.977 INFO:teuthology.orchestra.run.smithi114.stdout: "bluestore_inline_bl_bytes": 0, 2024-04-23T10:41:19.977 INFO:teuthology.orchestra.run.smithi114.stdout: "bluestore_inline_bl_items": 0, 2024-04-23T10:41:19.977 INFO:teuthology.orchestra.run.smithi114.stdout: "bluestore_fsck_bytes": 0, 2024-04-23T10:41:19.977 INFO:teuthology.orchestra.run.smithi114.stdout: "bluestore_fsck_items": 0, 2024-04-23T10:41:19.977 INFO:teuthology.orchestra.run.smithi114.stdout: "bluestore_txc_bytes": 0, 2024-04-23T10:41:19.977 INFO:teuthology.orchestra.run.smithi114.stdout: "bluestore_txc_items": 0, 2024-04-23T10:41:19.977 INFO:teuthology.orchestra.run.smithi114.stdout: "bluestore_writing_deferred_bytes": 0, 2024-04-23T10:41:19.977 INFO:teuthology.orchestra.run.smithi114.stdout: "bluestore_writing_deferred_items": 0, 2024-04-23T10:41:19.978 INFO:teuthology.orchestra.run.smithi114.stdout: "bluestore_writing_bytes": 0, 2024-04-23T10:41:19.978 INFO:teuthology.orchestra.run.smithi114.stdout: "bluestore_writing_items": 0, 2024-04-23T10:41:19.978 INFO:teuthology.orchestra.run.smithi114.stdout: "bluefs_bytes": 0, 2024-04-23T10:41:19.978 INFO:teuthology.orchestra.run.smithi114.stdout: "bluefs_items": 0, 2024-04-23T10:41:19.978 INFO:teuthology.orchestra.run.smithi114.stdout: "bluefs_file_reader_bytes": 0, 2024-04-23T10:41:19.978 INFO:teuthology.orchestra.run.smithi114.stdout: "bluefs_file_reader_items": 0, 2024-04-23T10:41:19.978 INFO:teuthology.orchestra.run.smithi114.stdout: "bluefs_file_writer_bytes": 0, 2024-04-23T10:41:19.979 INFO:teuthology.orchestra.run.smithi114.stdout: "bluefs_file_writer_items": 0, 2024-04-23T10:41:19.979 INFO:teuthology.orchestra.run.smithi114.stdout: "buffer_anon_bytes": 148903, 2024-04-23T10:41:19.979 INFO:teuthology.orchestra.run.smithi114.stdout: "buffer_anon_items": 27, 2024-04-23T10:41:19.979 INFO:teuthology.orchestra.run.smithi114.stdout: "buffer_meta_bytes": 0, 2024-04-23T10:41:19.979 INFO:teuthology.orchestra.run.smithi114.stdout: "buffer_meta_items": 0, 2024-04-23T10:41:19.979 INFO:teuthology.orchestra.run.smithi114.stdout: "osd_bytes": 0, 2024-04-23T10:41:19.980 INFO:teuthology.orchestra.run.smithi114.stdout: "osd_items": 0, 2024-04-23T10:41:19.980 INFO:teuthology.orchestra.run.smithi114.stdout: "osd_mapbl_bytes": 0, 2024-04-23T10:41:19.980 INFO:teuthology.orchestra.run.smithi114.stdout: "osd_mapbl_items": 0, 2024-04-23T10:41:19.980 INFO:teuthology.orchestra.run.smithi114.stdout: "osd_pglog_bytes": 0, 2024-04-23T10:41:19.980 INFO:teuthology.orchestra.run.smithi114.stdout: "osd_pglog_items": 0, 2024-04-23T10:41:19.980 INFO:teuthology.orchestra.run.smithi114.stdout: "osdmap_bytes": 0, 2024-04-23T10:41:19.980 INFO:teuthology.orchestra.run.smithi114.stdout: "osdmap_items": 0, 2024-04-23T10:41:19.980 INFO:teuthology.orchestra.run.smithi114.stdout: "osdmap_mapping_bytes": 0, 2024-04-23T10:41:19.980 INFO:teuthology.orchestra.run.smithi114.stdout: "osdmap_mapping_items": 0, 2024-04-23T10:41:19.981 INFO:teuthology.orchestra.run.smithi114.stdout: "pgmap_bytes": 0, 2024-04-23T10:41:19.981 INFO:teuthology.orchestra.run.smithi114.stdout: "pgmap_items": 0, 2024-04-23T10:41:19.981 INFO:teuthology.orchestra.run.smithi114.stdout: "mds_co_bytes": 928, 2024-04-23T10:41:19.981 INFO:teuthology.orchestra.run.smithi114.stdout: "mds_co_items": 2, 2024-04-23T10:41:19.981 INFO:teuthology.orchestra.run.smithi114.stdout: "unittest_1_bytes": 0, 2024-04-23T10:41:19.981 INFO:teuthology.orchestra.run.smithi114.stdout: "unittest_1_items": 0, 2024-04-23T10:41:19.981 INFO:teuthology.orchestra.run.smithi114.stdout: "unittest_2_bytes": 0, 2024-04-23T10:41:19.981 INFO:teuthology.orchestra.run.smithi114.stdout: "unittest_2_items": 0 2024-04-23T10:41:19.981 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-23T10:41:19.981 INFO:teuthology.orchestra.run.smithi114.stdout: "throttle-msgr_dispatch_throttler-mds": { 2024-04-23T10:41:19.982 INFO:teuthology.orchestra.run.smithi114.stdout: "val": 0, 2024-04-23T10:41:19.982 INFO:teuthology.orchestra.run.smithi114.stdout: "max": 104857600, 2024-04-23T10:41:19.982 INFO:teuthology.orchestra.run.smithi114.stdout: "get_started": 0, 2024-04-23T10:41:19.982 INFO:teuthology.orchestra.run.smithi114.stdout: "get": 289, 2024-04-23T10:41:19.982 INFO:teuthology.orchestra.run.smithi114.stdout: "get_sum": 1567529, 2024-04-23T10:41:19.982 INFO:teuthology.orchestra.run.smithi114.stdout: "get_or_fail_fail": 0, 2024-04-23T10:41:19.982 INFO:teuthology.orchestra.run.smithi114.stdout: "get_or_fail_success": 289, 2024-04-23T10:41:19.982 INFO:teuthology.orchestra.run.smithi114.stdout: "take": 0, 2024-04-23T10:41:19.982 INFO:teuthology.orchestra.run.smithi114.stdout: "take_sum": 0, 2024-04-23T10:41:19.983 INFO:teuthology.orchestra.run.smithi114.stdout: "put": 289, 2024-04-23T10:41:19.983 INFO:teuthology.orchestra.run.smithi114.stdout: "put_sum": 1567529, 2024-04-23T10:41:19.983 INFO:teuthology.orchestra.run.smithi114.stdout: "wait": { 2024-04-23T10:41:19.983 INFO:teuthology.orchestra.run.smithi114.stdout: "avgcount": 0, 2024-04-23T10:41:19.983 INFO:teuthology.orchestra.run.smithi114.stdout: "sum": 0.000000000, 2024-04-23T10:41:19.983 INFO:teuthology.orchestra.run.smithi114.stdout: "avgtime": 0.000000000 2024-04-23T10:41:19.983 INFO:teuthology.orchestra.run.smithi114.stdout: } 2024-04-23T10:41:19.983 INFO:teuthology.orchestra.run.smithi114.stdout: } 2024-04-23T10:41:19.983 INFO:teuthology.orchestra.run.smithi114.stdout:} 2024-04-23T10:41:20.578 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 23 10:41:20 smithi204 ceph-mon[32649]: pgmap v465: 97 pgs: 97 active+clean; 564 MiB data, 4.4 GiB used, 1.0 TiB / 1.0 TiB avail; 388 KiB/s rd, 5.1 MiB/s wr, 164 op/s 2024-04-23T10:41:20.628 INFO:journalctl@ceph.mon.a.smithi114.stdout:Apr 23 10:41:20 smithi114 ceph-mon[29255]: pgmap v465: 97 pgs: 97 active+clean; 564 MiB data, 4.4 GiB used, 1.0 TiB / 1.0 TiB avail; 388 KiB/s rd, 5.1 MiB/s wr, 164 op/s 2024-04-23T10:41:20.654 WARNING:tasks.check_counter:Counter 'mds.exported' not found on daemon mds.j 2024-04-23T10:41:20.655 WARNING:tasks.check_counter:Counter 'mds.imported' not found on daemon mds.j 2024-04-23T10:41:20.655 WARNING:tasks.check_counter:Counter 'mds_cache.dir_update' not found on daemon mds.j 2024-04-23T10:41:20.655 WARNING:tasks.check_counter:Counter 'mds_cache.dir_update_receipt' not found on daemon mds.j 2024-04-23T10:41:20.655 WARNING:tasks.check_counter:Counter 'mds.root_rsnaps' not found on daemon mds.j 2024-04-23T10:41:20.655 WARNING:tasks.check_counter:Counter 'mds_server.req_mksnap_latency.avgcount' not found on daemon mds.j 2024-04-23T10:41:20.655 WARNING:tasks.check_counter:Counter 'mds_server.req_rmsnap_latency.avgcount' not found on daemon mds.j 2024-04-23T10:41:20.655 DEBUG:tasks.check_counter:Getting stats from b 2024-04-23T10:41:20.656 DEBUG:teuthology.orchestra.run.smithi119:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d69f13d827952bb07bb9b7973bca14f97d764d6f shell --fsid aa240572-0159-11ef-bc93-c7b262605968 -- ceph daemon mds.b perf dump 2024-04-23T10:41:20.720 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 23 10:41:20 smithi119 ceph-mon[32608]: pgmap v465: 97 pgs: 97 active+clean; 564 MiB data, 4.4 GiB used, 1.0 TiB / 1.0 TiB avail; 388 KiB/s rd, 5.1 MiB/s wr, 164 op/s 2024-04-23T10:41:20.966 INFO:teuthology.orchestra.run.smithi119.stderr:Inferring config /var/lib/ceph/aa240572-0159-11ef-bc93-c7b262605968/mon.b/config 2024-04-23T10:41:22.386 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 23 10:41:22 smithi119 ceph-mon[32608]: pgmap v466: 97 pgs: 97 active+clean; 564 MiB data, 4.4 GiB used, 1.0 TiB / 1.0 TiB avail; 358 KiB/s rd, 4.7 MiB/s wr, 151 op/s 2024-04-23T10:41:22.577 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 23 10:41:22 smithi204 ceph-mon[32649]: pgmap v466: 97 pgs: 97 active+clean; 564 MiB data, 4.4 GiB used, 1.0 TiB / 1.0 TiB avail; 358 KiB/s rd, 4.7 MiB/s wr, 151 op/s 2024-04-23T10:41:22.753 INFO:journalctl@ceph.mon.a.smithi114.stdout:Apr 23 10:41:22 smithi114 ceph-mon[29255]: pgmap v466: 97 pgs: 97 active+clean; 564 MiB data, 4.4 GiB used, 1.0 TiB / 1.0 TiB avail; 358 KiB/s rd, 4.7 MiB/s wr, 151 op/s 2024-04-23T10:41:22.867 INFO:teuthology.orchestra.run.smithi119.stdout:{ 2024-04-23T10:41:22.867 INFO:teuthology.orchestra.run.smithi119.stdout: "AsyncMessenger::Worker-0": { 2024-04-23T10:41:22.867 INFO:teuthology.orchestra.run.smithi119.stdout: "msgr_recv_messages": 133907, 2024-04-23T10:41:22.867 INFO:teuthology.orchestra.run.smithi119.stdout: "msgr_send_messages": 184205, 2024-04-23T10:41:22.867 INFO:teuthology.orchestra.run.smithi119.stdout: "msgr_recv_bytes": 33055054, 2024-04-23T10:41:22.867 INFO:teuthology.orchestra.run.smithi119.stdout: "msgr_send_bytes": 294972672, 2024-04-23T10:41:22.867 INFO:teuthology.orchestra.run.smithi119.stdout: "msgr_created_connections": 154, 2024-04-23T10:41:22.868 INFO:teuthology.orchestra.run.smithi119.stdout: "msgr_active_connections": 7, 2024-04-23T10:41:22.868 INFO:teuthology.orchestra.run.smithi119.stdout: "msgr_running_total_time": 10.909264531, 2024-04-23T10:41:22.868 INFO:teuthology.orchestra.run.smithi119.stdout: "msgr_running_send_time": 4.976921467, 2024-04-23T10:41:22.868 INFO:teuthology.orchestra.run.smithi119.stdout: "msgr_running_recv_time": 5.219103390, 2024-04-23T10:41:22.868 INFO:teuthology.orchestra.run.smithi119.stdout: "msgr_running_fast_dispatch_time": 1.353015012, 2024-04-23T10:41:22.868 INFO:teuthology.orchestra.run.smithi119.stdout: "msgr_send_messages_queue_lat": { 2024-04-23T10:41:22.868 INFO:teuthology.orchestra.run.smithi119.stdout: "avgcount": 184206, 2024-04-23T10:41:22.868 INFO:teuthology.orchestra.run.smithi119.stdout: "sum": 12.184186385, 2024-04-23T10:41:22.868 INFO:teuthology.orchestra.run.smithi119.stdout: "avgtime": 0.000066144 2024-04-23T10:41:22.869 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-04-23T10:41:22.869 INFO:teuthology.orchestra.run.smithi119.stdout: "msgr_handle_ack_lat": { 2024-04-23T10:41:22.869 INFO:teuthology.orchestra.run.smithi119.stdout: "avgcount": 249565, 2024-04-23T10:41:22.869 INFO:teuthology.orchestra.run.smithi119.stdout: "sum": 0.047379805, 2024-04-23T10:41:22.869 INFO:teuthology.orchestra.run.smithi119.stdout: "avgtime": 0.000000189 2024-04-23T10:41:22.869 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-04-23T10:41:22.869 INFO:teuthology.orchestra.run.smithi119.stdout: "msgr_recv_encrypted_bytes": 42544, 2024-04-23T10:41:22.869 INFO:teuthology.orchestra.run.smithi119.stdout: "msgr_send_encrypted_bytes": 1376 2024-04-23T10:41:22.869 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-04-23T10:41:22.869 INFO:teuthology.orchestra.run.smithi119.stdout: "AsyncMessenger::Worker-1": { 2024-04-23T10:41:22.870 INFO:teuthology.orchestra.run.smithi119.stdout: "msgr_recv_messages": 176008, 2024-04-23T10:41:22.870 INFO:teuthology.orchestra.run.smithi119.stdout: "msgr_send_messages": 200035, 2024-04-23T10:41:22.870 INFO:teuthology.orchestra.run.smithi119.stdout: "msgr_recv_bytes": 44256226, 2024-04-23T10:41:22.870 INFO:teuthology.orchestra.run.smithi119.stdout: "msgr_send_bytes": 169268991, 2024-04-23T10:41:22.870 INFO:teuthology.orchestra.run.smithi119.stdout: "msgr_created_connections": 19, 2024-04-23T10:41:22.870 INFO:teuthology.orchestra.run.smithi119.stdout: "msgr_active_connections": 5, 2024-04-23T10:41:22.870 INFO:teuthology.orchestra.run.smithi119.stdout: "msgr_running_total_time": 11.347609597, 2024-04-23T10:41:22.870 INFO:teuthology.orchestra.run.smithi119.stdout: "msgr_running_send_time": 4.975584621, 2024-04-23T10:41:22.870 INFO:teuthology.orchestra.run.smithi119.stdout: "msgr_running_recv_time": 6.722049937, 2024-04-23T10:41:22.871 INFO:teuthology.orchestra.run.smithi119.stdout: "msgr_running_fast_dispatch_time": 0.887686084, 2024-04-23T10:41:22.871 INFO:teuthology.orchestra.run.smithi119.stdout: "msgr_send_messages_queue_lat": { 2024-04-23T10:41:22.871 INFO:teuthology.orchestra.run.smithi119.stdout: "avgcount": 200035, 2024-04-23T10:41:22.871 INFO:teuthology.orchestra.run.smithi119.stdout: "sum": 14.625170402, 2024-04-23T10:41:22.871 INFO:teuthology.orchestra.run.smithi119.stdout: "avgtime": 0.000073113 2024-04-23T10:41:22.871 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-04-23T10:41:22.871 INFO:teuthology.orchestra.run.smithi119.stdout: "msgr_handle_ack_lat": { 2024-04-23T10:41:22.871 INFO:teuthology.orchestra.run.smithi119.stdout: "avgcount": 329848, 2024-04-23T10:41:22.871 INFO:teuthology.orchestra.run.smithi119.stdout: "sum": 0.055188327, 2024-04-23T10:41:22.872 INFO:teuthology.orchestra.run.smithi119.stdout: "avgtime": 0.000000167 2024-04-23T10:41:22.872 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-04-23T10:41:22.872 INFO:teuthology.orchestra.run.smithi119.stdout: "msgr_recv_encrypted_bytes": 44256, 2024-04-23T10:41:22.872 INFO:teuthology.orchestra.run.smithi119.stdout: "msgr_send_encrypted_bytes": 538064 2024-04-23T10:41:22.872 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-04-23T10:41:22.872 INFO:teuthology.orchestra.run.smithi119.stdout: "AsyncMessenger::Worker-2": { 2024-04-23T10:41:22.872 INFO:teuthology.orchestra.run.smithi119.stdout: "msgr_recv_messages": 211835, 2024-04-23T10:41:22.872 INFO:teuthology.orchestra.run.smithi119.stdout: "msgr_send_messages": 281504, 2024-04-23T10:41:22.873 INFO:teuthology.orchestra.run.smithi119.stdout: "msgr_recv_bytes": 64181275, 2024-04-23T10:41:22.873 INFO:teuthology.orchestra.run.smithi119.stdout: "msgr_send_bytes": 280540171, 2024-04-23T10:41:22.873 INFO:teuthology.orchestra.run.smithi119.stdout: "msgr_created_connections": 19, 2024-04-23T10:41:22.873 INFO:teuthology.orchestra.run.smithi119.stdout: "msgr_active_connections": 9, 2024-04-23T10:41:22.873 INFO:teuthology.orchestra.run.smithi119.stdout: "msgr_running_total_time": 11.266165685, 2024-04-23T10:41:22.873 INFO:teuthology.orchestra.run.smithi119.stdout: "msgr_running_send_time": 4.909628022, 2024-04-23T10:41:22.873 INFO:teuthology.orchestra.run.smithi119.stdout: "msgr_running_recv_time": 7.355470577, 2024-04-23T10:41:22.873 INFO:teuthology.orchestra.run.smithi119.stdout: "msgr_running_fast_dispatch_time": 0.797339014, 2024-04-23T10:41:22.873 INFO:teuthology.orchestra.run.smithi119.stdout: "msgr_send_messages_queue_lat": { 2024-04-23T10:41:22.873 INFO:teuthology.orchestra.run.smithi119.stdout: "avgcount": 281498, 2024-04-23T10:41:22.874 INFO:teuthology.orchestra.run.smithi119.stdout: "sum": 14.856025787, 2024-04-23T10:41:22.874 INFO:teuthology.orchestra.run.smithi119.stdout: "avgtime": 0.000052774 2024-04-23T10:41:22.874 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-04-23T10:41:22.874 INFO:teuthology.orchestra.run.smithi119.stdout: "msgr_handle_ack_lat": { 2024-04-23T10:41:22.874 INFO:teuthology.orchestra.run.smithi119.stdout: "avgcount": 419355, 2024-04-23T10:41:22.874 INFO:teuthology.orchestra.run.smithi119.stdout: "sum": 0.063862170, 2024-04-23T10:41:22.874 INFO:teuthology.orchestra.run.smithi119.stdout: "avgtime": 0.000000152 2024-04-23T10:41:22.874 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-04-23T10:41:22.875 INFO:teuthology.orchestra.run.smithi119.stdout: "msgr_recv_encrypted_bytes": 2297152, 2024-04-23T10:41:22.875 INFO:teuthology.orchestra.run.smithi119.stdout: "msgr_send_encrypted_bytes": 356768 2024-04-23T10:41:22.875 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-04-23T10:41:22.875 INFO:teuthology.orchestra.run.smithi119.stdout: "cct": { 2024-04-23T10:41:22.875 INFO:teuthology.orchestra.run.smithi119.stdout: "total_workers": 1, 2024-04-23T10:41:22.875 INFO:teuthology.orchestra.run.smithi119.stdout: "unhealthy_workers": 0 2024-04-23T10:41:22.875 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-04-23T10:41:22.875 INFO:teuthology.orchestra.run.smithi119.stdout: "finisher-MDSRank": { 2024-04-23T10:41:22.875 INFO:teuthology.orchestra.run.smithi119.stdout: "queue_len": 0, 2024-04-23T10:41:22.875 INFO:teuthology.orchestra.run.smithi119.stdout: "complete_latency": { 2024-04-23T10:41:22.876 INFO:teuthology.orchestra.run.smithi119.stdout: "avgcount": 83046, 2024-04-23T10:41:22.876 INFO:teuthology.orchestra.run.smithi119.stdout: "sum": 78.498507475, 2024-04-23T10:41:22.876 INFO:teuthology.orchestra.run.smithi119.stdout: "avgtime": 0.000945241 2024-04-23T10:41:22.876 INFO:teuthology.orchestra.run.smithi119.stdout: } 2024-04-23T10:41:22.876 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-04-23T10:41:22.876 INFO:teuthology.orchestra.run.smithi119.stdout: "finisher-PurgeQueue": { 2024-04-23T10:41:22.876 INFO:teuthology.orchestra.run.smithi119.stdout: "queue_len": 0, 2024-04-23T10:41:22.876 INFO:teuthology.orchestra.run.smithi119.stdout: "complete_latency": { 2024-04-23T10:41:22.876 INFO:teuthology.orchestra.run.smithi119.stdout: "avgcount": 25999, 2024-04-23T10:41:22.877 INFO:teuthology.orchestra.run.smithi119.stdout: "sum": 5.197094984, 2024-04-23T10:41:22.877 INFO:teuthology.orchestra.run.smithi119.stdout: "avgtime": 0.000199895 2024-04-23T10:41:22.877 INFO:teuthology.orchestra.run.smithi119.stdout: } 2024-04-23T10:41:22.877 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-04-23T10:41:22.877 INFO:teuthology.orchestra.run.smithi119.stdout: "mds": { 2024-04-23T10:41:22.877 INFO:teuthology.orchestra.run.smithi119.stdout: "request": 136805, 2024-04-23T10:41:22.877 INFO:teuthology.orchestra.run.smithi119.stdout: "reply": 124383, 2024-04-23T10:41:22.877 INFO:teuthology.orchestra.run.smithi119.stdout: "reply_latency": { 2024-04-23T10:41:22.877 INFO:teuthology.orchestra.run.smithi119.stdout: "avgcount": 124383, 2024-04-23T10:41:22.877 INFO:teuthology.orchestra.run.smithi119.stdout: "sum": 815.307906360, 2024-04-23T10:41:22.878 INFO:teuthology.orchestra.run.smithi119.stdout: "avgtime": 0.006554817 2024-04-23T10:41:22.878 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-04-23T10:41:22.878 INFO:teuthology.orchestra.run.smithi119.stdout: "slow_reply": 0, 2024-04-23T10:41:22.878 INFO:teuthology.orchestra.run.smithi119.stdout: "forward": 12422, 2024-04-23T10:41:22.878 INFO:teuthology.orchestra.run.smithi119.stdout: "dir_fetch_complete": 18, 2024-04-23T10:41:22.878 INFO:teuthology.orchestra.run.smithi119.stdout: "dir_fetch_keys": 0, 2024-04-23T10:41:22.878 INFO:teuthology.orchestra.run.smithi119.stdout: "dir_commit": 1899, 2024-04-23T10:41:22.878 INFO:teuthology.orchestra.run.smithi119.stdout: "dir_split": 9, 2024-04-23T10:41:22.878 INFO:teuthology.orchestra.run.smithi119.stdout: "dir_merge": 3, 2024-04-23T10:41:22.879 INFO:teuthology.orchestra.run.smithi119.stdout: "inodes": 2097, 2024-04-23T10:41:22.879 INFO:teuthology.orchestra.run.smithi119.stdout: "inodes_top": 615, 2024-04-23T10:41:22.879 INFO:teuthology.orchestra.run.smithi119.stdout: "inodes_bottom": 1397, 2024-04-23T10:41:22.879 INFO:teuthology.orchestra.run.smithi119.stdout: "inodes_pin_tail": 85, 2024-04-23T10:41:22.879 INFO:teuthology.orchestra.run.smithi119.stdout: "inodes_pinned": 1218, 2024-04-23T10:41:22.879 INFO:teuthology.orchestra.run.smithi119.stdout: "inodes_expired": 1972, 2024-04-23T10:41:22.879 INFO:teuthology.orchestra.run.smithi119.stdout: "inodes_with_caps": 642, 2024-04-23T10:41:22.879 INFO:teuthology.orchestra.run.smithi119.stdout: "caps": 643, 2024-04-23T10:41:22.879 INFO:teuthology.orchestra.run.smithi119.stdout: "subtrees": 14, 2024-04-23T10:41:22.880 INFO:teuthology.orchestra.run.smithi119.stdout: "traverse": 222695, 2024-04-23T10:41:22.880 INFO:teuthology.orchestra.run.smithi119.stdout: "traverse_hit": 140951, 2024-04-23T10:41:22.880 INFO:teuthology.orchestra.run.smithi119.stdout: "traverse_forward": 12392, 2024-04-23T10:41:22.880 INFO:teuthology.orchestra.run.smithi119.stdout: "traverse_discover": 183, 2024-04-23T10:41:22.880 INFO:teuthology.orchestra.run.smithi119.stdout: "traverse_dir_fetch": 5, 2024-04-23T10:41:22.880 INFO:teuthology.orchestra.run.smithi119.stdout: "traverse_remote_ino": 0, 2024-04-23T10:41:22.880 INFO:teuthology.orchestra.run.smithi119.stdout: "traverse_lock": 1395, 2024-04-23T10:41:22.880 INFO:teuthology.orchestra.run.smithi119.stdout: "load_cent": 206498, 2024-04-23T10:41:22.880 INFO:teuthology.orchestra.run.smithi119.stdout: "q": 34, 2024-04-23T10:41:22.881 INFO:teuthology.orchestra.run.smithi119.stdout: "exported": 15, 2024-04-23T10:41:22.881 INFO:teuthology.orchestra.run.smithi119.stdout: "exported_inodes": 1171, 2024-04-23T10:41:22.881 INFO:teuthology.orchestra.run.smithi119.stdout: "imported": 19, 2024-04-23T10:41:22.881 INFO:teuthology.orchestra.run.smithi119.stdout: "imported_inodes": 507, 2024-04-23T10:41:22.881 INFO:teuthology.orchestra.run.smithi119.stdout: "openino_dir_fetch": 0, 2024-04-23T10:41:22.881 INFO:teuthology.orchestra.run.smithi119.stdout: "openino_backtrace_fetch": 0, 2024-04-23T10:41:22.881 INFO:teuthology.orchestra.run.smithi119.stdout: "openino_peer_discover": 0, 2024-04-23T10:41:22.881 INFO:teuthology.orchestra.run.smithi119.stdout: "root_rfiles": 731, 2024-04-23T10:41:22.881 INFO:teuthology.orchestra.run.smithi119.stdout: "root_rbytes": 212014623, 2024-04-23T10:41:22.882 INFO:teuthology.orchestra.run.smithi119.stdout: "root_rsnaps": 6, 2024-04-23T10:41:22.882 INFO:teuthology.orchestra.run.smithi119.stdout: "scrub_backtrace_fetch": 0, 2024-04-23T10:41:22.882 INFO:teuthology.orchestra.run.smithi119.stdout: "scrub_set_tag": 0, 2024-04-23T10:41:22.882 INFO:teuthology.orchestra.run.smithi119.stdout: "scrub_backtrace_repaired": 0, 2024-04-23T10:41:22.882 INFO:teuthology.orchestra.run.smithi119.stdout: "scrub_inotable_repaired": 0, 2024-04-23T10:41:22.882 INFO:teuthology.orchestra.run.smithi119.stdout: "scrub_dir_inodes": 0, 2024-04-23T10:41:22.882 INFO:teuthology.orchestra.run.smithi119.stdout: "scrub_dir_base_inodes": 0, 2024-04-23T10:41:22.882 INFO:teuthology.orchestra.run.smithi119.stdout: "scrub_dirfrag_rstats": 0, 2024-04-23T10:41:22.883 INFO:teuthology.orchestra.run.smithi119.stdout: "scrub_file_inodes": 0, 2024-04-23T10:41:22.883 INFO:teuthology.orchestra.run.smithi119.stdout: "handle_inode_file_caps": 23, 2024-04-23T10:41:22.883 INFO:teuthology.orchestra.run.smithi119.stdout: "ceph_cap_op_revoke": 22491, 2024-04-23T10:41:22.883 INFO:teuthology.orchestra.run.smithi119.stdout: "ceph_cap_op_grant": 51434, 2024-04-23T10:41:22.883 INFO:teuthology.orchestra.run.smithi119.stdout: "ceph_cap_op_trunc": 0, 2024-04-23T10:41:22.883 INFO:teuthology.orchestra.run.smithi119.stdout: "ceph_cap_op_flushsnap_ack": 0, 2024-04-23T10:41:22.883 INFO:teuthology.orchestra.run.smithi119.stdout: "ceph_cap_op_flush_ack": 0, 2024-04-23T10:41:22.883 INFO:teuthology.orchestra.run.smithi119.stdout: "handle_client_caps": 44564, 2024-04-23T10:41:22.883 INFO:teuthology.orchestra.run.smithi119.stdout: "handle_client_caps_dirty": 13536, 2024-04-23T10:41:22.883 INFO:teuthology.orchestra.run.smithi119.stdout: "handle_client_cap_release": 6499, 2024-04-23T10:41:22.884 INFO:teuthology.orchestra.run.smithi119.stdout: "process_request_cap_release": 70327 2024-04-23T10:41:22.884 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-04-23T10:41:22.884 INFO:teuthology.orchestra.run.smithi119.stdout: "mds_cache": { 2024-04-23T10:41:22.884 INFO:teuthology.orchestra.run.smithi119.stdout: "dir_update": 263, 2024-04-23T10:41:22.884 INFO:teuthology.orchestra.run.smithi119.stdout: "dir_update_receipt": 99, 2024-04-23T10:41:22.884 INFO:teuthology.orchestra.run.smithi119.stdout: "dir_try_discover": 63, 2024-04-23T10:41:22.884 INFO:teuthology.orchestra.run.smithi119.stdout: "dir_send_discover": 219, 2024-04-23T10:41:22.884 INFO:teuthology.orchestra.run.smithi119.stdout: "dir_handle_discover": 1537, 2024-04-23T10:41:22.884 INFO:teuthology.orchestra.run.smithi119.stdout: "num_strays": 252, 2024-04-23T10:41:22.885 INFO:teuthology.orchestra.run.smithi119.stdout: "num_strays_delayed": 0, 2024-04-23T10:41:22.885 INFO:teuthology.orchestra.run.smithi119.stdout: "num_strays_enqueuing": 0, 2024-04-23T10:41:22.885 INFO:teuthology.orchestra.run.smithi119.stdout: "strays_created": 15836, 2024-04-23T10:41:22.885 INFO:teuthology.orchestra.run.smithi119.stdout: "strays_enqueued": 15584, 2024-04-23T10:41:22.885 INFO:teuthology.orchestra.run.smithi119.stdout: "strays_reintegrated": 0, 2024-04-23T10:41:22.885 INFO:teuthology.orchestra.run.smithi119.stdout: "strays_migrated": 0, 2024-04-23T10:41:22.885 INFO:teuthology.orchestra.run.smithi119.stdout: "num_recovering_processing": 0, 2024-04-23T10:41:22.885 INFO:teuthology.orchestra.run.smithi119.stdout: "num_recovering_enqueued": 0, 2024-04-23T10:41:22.885 INFO:teuthology.orchestra.run.smithi119.stdout: "num_recovering_prioritized": 0, 2024-04-23T10:41:22.885 INFO:teuthology.orchestra.run.smithi119.stdout: "recovery_started": 0, 2024-04-23T10:41:22.886 INFO:teuthology.orchestra.run.smithi119.stdout: "recovery_completed": 0, 2024-04-23T10:41:22.886 INFO:teuthology.orchestra.run.smithi119.stdout: "ireq_quiesce_path": 5, 2024-04-23T10:41:22.886 INFO:teuthology.orchestra.run.smithi119.stdout: "ireq_quiesce_inode": 188, 2024-04-23T10:41:22.886 INFO:teuthology.orchestra.run.smithi119.stdout: "ireq_enqueue_scrub": 0, 2024-04-23T10:41:22.886 INFO:teuthology.orchestra.run.smithi119.stdout: "ireq_exportdir": 15, 2024-04-23T10:41:22.886 INFO:teuthology.orchestra.run.smithi119.stdout: "ireq_flush": 0, 2024-04-23T10:41:22.886 INFO:teuthology.orchestra.run.smithi119.stdout: "ireq_fragmentdir": 12, 2024-04-23T10:41:22.886 INFO:teuthology.orchestra.run.smithi119.stdout: "ireq_fragstats": 0, 2024-04-23T10:41:22.886 INFO:teuthology.orchestra.run.smithi119.stdout: "ireq_inodestats": 0 2024-04-23T10:41:22.887 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-04-23T10:41:22.887 INFO:teuthology.orchestra.run.smithi119.stdout: "mds_log": { 2024-04-23T10:41:22.887 INFO:teuthology.orchestra.run.smithi119.stdout: "evlrg": 0, 2024-04-23T10:41:22.887 INFO:teuthology.orchestra.run.smithi119.stdout: "evadd": 96318, 2024-04-23T10:41:22.887 INFO:teuthology.orchestra.run.smithi119.stdout: "evex": 95055, 2024-04-23T10:41:22.887 INFO:teuthology.orchestra.run.smithi119.stdout: "evtrm": 95055, 2024-04-23T10:41:22.887 INFO:teuthology.orchestra.run.smithi119.stdout: "ev": 1263, 2024-04-23T10:41:22.887 INFO:teuthology.orchestra.run.smithi119.stdout: "evexg": 0, 2024-04-23T10:41:22.887 INFO:teuthology.orchestra.run.smithi119.stdout: "evexd": 3544, 2024-04-23T10:41:22.888 INFO:teuthology.orchestra.run.smithi119.stdout: "segadd": 193, 2024-04-23T10:41:22.888 INFO:teuthology.orchestra.run.smithi119.stdout: "segex": 191, 2024-04-23T10:41:22.888 INFO:teuthology.orchestra.run.smithi119.stdout: "segtrm": 191, 2024-04-23T10:41:22.888 INFO:teuthology.orchestra.run.smithi119.stdout: "seg": 2, 2024-04-23T10:41:22.888 INFO:teuthology.orchestra.run.smithi119.stdout: "segmjr": 1, 2024-04-23T10:41:22.888 INFO:teuthology.orchestra.run.smithi119.stdout: "segexg": 0, 2024-04-23T10:41:22.888 INFO:teuthology.orchestra.run.smithi119.stdout: "segexd": 6, 2024-04-23T10:41:22.888 INFO:teuthology.orchestra.run.smithi119.stdout: "expos": 412041178, 2024-04-23T10:41:22.888 INFO:teuthology.orchestra.run.smithi119.stdout: "wrpos": 414851926, 2024-04-23T10:41:22.888 INFO:teuthology.orchestra.run.smithi119.stdout: "rdpos": 0, 2024-04-23T10:41:22.889 INFO:teuthology.orchestra.run.smithi119.stdout: "jlat": { 2024-04-23T10:41:22.889 INFO:teuthology.orchestra.run.smithi119.stdout: "avgcount": 42532, 2024-04-23T10:41:22.889 INFO:teuthology.orchestra.run.smithi119.stdout: "sum": 135.866837410, 2024-04-23T10:41:22.889 INFO:teuthology.orchestra.run.smithi119.stdout: "avgtime": 0.003194461 2024-04-23T10:41:22.889 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-04-23T10:41:22.889 INFO:teuthology.orchestra.run.smithi119.stdout: "replayed": 0 2024-04-23T10:41:22.889 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-04-23T10:41:22.889 INFO:teuthology.orchestra.run.smithi119.stdout: "mds_mem": { 2024-04-23T10:41:22.889 INFO:teuthology.orchestra.run.smithi119.stdout: "ino": 1313, 2024-04-23T10:41:22.890 INFO:teuthology.orchestra.run.smithi119.stdout: "ino+": 18079, 2024-04-23T10:41:22.890 INFO:teuthology.orchestra.run.smithi119.stdout: "ino-": 16766, 2024-04-23T10:41:22.890 INFO:teuthology.orchestra.run.smithi119.stdout: "dir": 212, 2024-04-23T10:41:22.890 INFO:teuthology.orchestra.run.smithi119.stdout: "dir+": 310, 2024-04-23T10:41:22.890 INFO:teuthology.orchestra.run.smithi119.stdout: "dir-": 98, 2024-04-23T10:41:22.890 INFO:teuthology.orchestra.run.smithi119.stdout: "dn": 2110, 2024-04-23T10:41:22.890 INFO:teuthology.orchestra.run.smithi119.stdout: "dn+": 35923, 2024-04-23T10:41:22.890 INFO:teuthology.orchestra.run.smithi119.stdout: "dn-": 33813, 2024-04-23T10:41:22.890 INFO:teuthology.orchestra.run.smithi119.stdout: "cap": 530, 2024-04-23T10:41:22.891 INFO:teuthology.orchestra.run.smithi119.stdout: "cap+": 17162, 2024-04-23T10:41:22.891 INFO:teuthology.orchestra.run.smithi119.stdout: "cap-": 16632, 2024-04-23T10:41:22.891 INFO:teuthology.orchestra.run.smithi119.stdout: "rss": 96972, 2024-04-23T10:41:22.891 INFO:teuthology.orchestra.run.smithi119.stdout: "heap": 231800 2024-04-23T10:41:22.891 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-04-23T10:41:22.891 INFO:teuthology.orchestra.run.smithi119.stdout: "mds_server": { 2024-04-23T10:41:22.891 INFO:teuthology.orchestra.run.smithi119.stdout: "dispatch_client_request": 195133, 2024-04-23T10:41:22.891 INFO:teuthology.orchestra.run.smithi119.stdout: "dispatch_server_request": 33418, 2024-04-23T10:41:22.891 INFO:teuthology.orchestra.run.smithi119.stdout: "handle_client_request": 136805, 2024-04-23T10:41:22.892 INFO:teuthology.orchestra.run.smithi119.stdout: "handle_client_session": 2877, 2024-04-23T10:41:22.892 INFO:teuthology.orchestra.run.smithi119.stdout: "handle_peer_request": 66364, 2024-04-23T10:41:22.892 INFO:teuthology.orchestra.run.smithi119.stdout: "req_create_latency": { 2024-04-23T10:41:22.892 INFO:teuthology.orchestra.run.smithi119.stdout: "avgcount": 16525, 2024-04-23T10:41:22.892 INFO:teuthology.orchestra.run.smithi119.stdout: "sum": 79.749245995, 2024-04-23T10:41:22.892 INFO:teuthology.orchestra.run.smithi119.stdout: "avgtime": 0.004825975 2024-04-23T10:41:22.892 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-04-23T10:41:22.892 INFO:teuthology.orchestra.run.smithi119.stdout: "req_getattr_latency": { 2024-04-23T10:41:22.892 INFO:teuthology.orchestra.run.smithi119.stdout: "avgcount": 22306, 2024-04-23T10:41:22.893 INFO:teuthology.orchestra.run.smithi119.stdout: "sum": 132.414765315, 2024-04-23T10:41:22.893 INFO:teuthology.orchestra.run.smithi119.stdout: "avgtime": 0.005936284 2024-04-23T10:41:22.893 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-04-23T10:41:22.893 INFO:teuthology.orchestra.run.smithi119.stdout: "req_getfilelock_latency": { 2024-04-23T10:41:22.893 INFO:teuthology.orchestra.run.smithi119.stdout: "avgcount": 0, 2024-04-23T10:41:22.893 INFO:teuthology.orchestra.run.smithi119.stdout: "sum": 0.000000000, 2024-04-23T10:41:22.893 INFO:teuthology.orchestra.run.smithi119.stdout: "avgtime": 0.000000000 2024-04-23T10:41:22.893 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-04-23T10:41:22.893 INFO:teuthology.orchestra.run.smithi119.stdout: "req_link_latency": { 2024-04-23T10:41:22.893 INFO:teuthology.orchestra.run.smithi119.stdout: "avgcount": 0, 2024-04-23T10:41:22.894 INFO:teuthology.orchestra.run.smithi119.stdout: "sum": 0.000000000, 2024-04-23T10:41:22.894 INFO:teuthology.orchestra.run.smithi119.stdout: "avgtime": 0.000000000 2024-04-23T10:41:22.894 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-04-23T10:41:22.894 INFO:teuthology.orchestra.run.smithi119.stdout: "req_lookup_latency": { 2024-04-23T10:41:22.894 INFO:teuthology.orchestra.run.smithi119.stdout: "avgcount": 37137, 2024-04-23T10:41:22.894 INFO:teuthology.orchestra.run.smithi119.stdout: "sum": 116.256544957, 2024-04-23T10:41:22.894 INFO:teuthology.orchestra.run.smithi119.stdout: "avgtime": 0.003130477 2024-04-23T10:41:22.894 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-04-23T10:41:22.894 INFO:teuthology.orchestra.run.smithi119.stdout: "req_lookuphash_latency": { 2024-04-23T10:41:22.895 INFO:teuthology.orchestra.run.smithi119.stdout: "avgcount": 0, 2024-04-23T10:41:22.895 INFO:teuthology.orchestra.run.smithi119.stdout: "sum": 0.000000000, 2024-04-23T10:41:22.895 INFO:teuthology.orchestra.run.smithi119.stdout: "avgtime": 0.000000000 2024-04-23T10:41:22.895 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-04-23T10:41:22.895 INFO:teuthology.orchestra.run.smithi119.stdout: "req_lookupino_latency": { 2024-04-23T10:41:22.895 INFO:teuthology.orchestra.run.smithi119.stdout: "avgcount": 1, 2024-04-23T10:41:22.895 INFO:teuthology.orchestra.run.smithi119.stdout: "sum": 0.000194406, 2024-04-23T10:41:22.895 INFO:teuthology.orchestra.run.smithi119.stdout: "avgtime": 0.000194406 2024-04-23T10:41:22.895 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-04-23T10:41:22.895 INFO:teuthology.orchestra.run.smithi119.stdout: "req_lookupname_latency": { 2024-04-23T10:41:22.896 INFO:teuthology.orchestra.run.smithi119.stdout: "avgcount": 0, 2024-04-23T10:41:22.896 INFO:teuthology.orchestra.run.smithi119.stdout: "sum": 0.000000000, 2024-04-23T10:41:22.896 INFO:teuthology.orchestra.run.smithi119.stdout: "avgtime": 0.000000000 2024-04-23T10:41:22.896 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-04-23T10:41:22.896 INFO:teuthology.orchestra.run.smithi119.stdout: "req_lookupparent_latency": { 2024-04-23T10:41:22.896 INFO:teuthology.orchestra.run.smithi119.stdout: "avgcount": 0, 2024-04-23T10:41:22.896 INFO:teuthology.orchestra.run.smithi119.stdout: "sum": 0.000000000, 2024-04-23T10:41:22.896 INFO:teuthology.orchestra.run.smithi119.stdout: "avgtime": 0.000000000 2024-04-23T10:41:22.896 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-04-23T10:41:22.896 INFO:teuthology.orchestra.run.smithi119.stdout: "req_lookupsnap_latency": { 2024-04-23T10:41:22.897 INFO:teuthology.orchestra.run.smithi119.stdout: "avgcount": 0, 2024-04-23T10:41:22.897 INFO:teuthology.orchestra.run.smithi119.stdout: "sum": 0.000000000, 2024-04-23T10:41:22.897 INFO:teuthology.orchestra.run.smithi119.stdout: "avgtime": 0.000000000 2024-04-23T10:41:22.898 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-04-23T10:41:22.898 INFO:teuthology.orchestra.run.smithi119.stdout: "req_lssnap_latency": { 2024-04-23T10:41:22.898 INFO:teuthology.orchestra.run.smithi119.stdout: "avgcount": 15, 2024-04-23T10:41:22.898 INFO:teuthology.orchestra.run.smithi119.stdout: "sum": 0.004146318, 2024-04-23T10:41:22.898 INFO:teuthology.orchestra.run.smithi119.stdout: "avgtime": 0.000276421 2024-04-23T10:41:22.898 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-04-23T10:41:22.898 INFO:teuthology.orchestra.run.smithi119.stdout: "req_mkdir_latency": { 2024-04-23T10:41:22.898 INFO:teuthology.orchestra.run.smithi119.stdout: "avgcount": 119, 2024-04-23T10:41:22.898 INFO:teuthology.orchestra.run.smithi119.stdout: "sum": 10.369957343, 2024-04-23T10:41:22.899 INFO:teuthology.orchestra.run.smithi119.stdout: "avgtime": 0.087142498 2024-04-23T10:41:22.899 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-04-23T10:41:22.899 INFO:teuthology.orchestra.run.smithi119.stdout: "req_mknod_latency": { 2024-04-23T10:41:22.899 INFO:teuthology.orchestra.run.smithi119.stdout: "avgcount": 0, 2024-04-23T10:41:22.899 INFO:teuthology.orchestra.run.smithi119.stdout: "sum": 0.000000000, 2024-04-23T10:41:22.899 INFO:teuthology.orchestra.run.smithi119.stdout: "avgtime": 0.000000000 2024-04-23T10:41:22.899 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-04-23T10:41:22.899 INFO:teuthology.orchestra.run.smithi119.stdout: "req_mksnap_latency": { 2024-04-23T10:41:22.899 INFO:teuthology.orchestra.run.smithi119.stdout: "avgcount": 15, 2024-04-23T10:41:22.899 INFO:teuthology.orchestra.run.smithi119.stdout: "sum": 17.639233635, 2024-04-23T10:41:22.900 INFO:teuthology.orchestra.run.smithi119.stdout: "avgtime": 1.175948909 2024-04-23T10:41:22.900 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-04-23T10:41:22.900 INFO:teuthology.orchestra.run.smithi119.stdout: "req_open_latency": { 2024-04-23T10:41:22.900 INFO:teuthology.orchestra.run.smithi119.stdout: "avgcount": 12, 2024-04-23T10:41:22.900 INFO:teuthology.orchestra.run.smithi119.stdout: "sum": 0.050859527, 2024-04-23T10:41:22.900 INFO:teuthology.orchestra.run.smithi119.stdout: "avgtime": 0.004238293 2024-04-23T10:41:22.900 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-04-23T10:41:22.900 INFO:teuthology.orchestra.run.smithi119.stdout: "req_readdir_latency": { 2024-04-23T10:41:22.900 INFO:teuthology.orchestra.run.smithi119.stdout: "avgcount": 26671, 2024-04-23T10:41:22.901 INFO:teuthology.orchestra.run.smithi119.stdout: "sum": 153.974202797, 2024-04-23T10:41:22.901 INFO:teuthology.orchestra.run.smithi119.stdout: "avgtime": 0.005773094 2024-04-23T10:41:22.901 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-04-23T10:41:22.901 INFO:teuthology.orchestra.run.smithi119.stdout: "req_rename_latency": { 2024-04-23T10:41:22.901 INFO:teuthology.orchestra.run.smithi119.stdout: "avgcount": 4706, 2024-04-23T10:41:22.901 INFO:teuthology.orchestra.run.smithi119.stdout: "sum": 221.555477795, 2024-04-23T10:41:22.901 INFO:teuthology.orchestra.run.smithi119.stdout: "avgtime": 0.047079362 2024-04-23T10:41:22.901 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-04-23T10:41:22.901 INFO:teuthology.orchestra.run.smithi119.stdout: "req_renamesnap_latency": { 2024-04-23T10:41:22.902 INFO:teuthology.orchestra.run.smithi119.stdout: "avgcount": 0, 2024-04-23T10:41:22.902 INFO:teuthology.orchestra.run.smithi119.stdout: "sum": 0.000000000, 2024-04-23T10:41:22.902 INFO:teuthology.orchestra.run.smithi119.stdout: "avgtime": 0.000000000 2024-04-23T10:41:22.902 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-04-23T10:41:22.902 INFO:teuthology.orchestra.run.smithi119.stdout: "req_snapdiff_latency": { 2024-04-23T10:41:22.902 INFO:teuthology.orchestra.run.smithi119.stdout: "avgcount": 0, 2024-04-23T10:41:22.902 INFO:teuthology.orchestra.run.smithi119.stdout: "sum": 0.000000000, 2024-04-23T10:41:22.902 INFO:teuthology.orchestra.run.smithi119.stdout: "avgtime": 0.000000000 2024-04-23T10:41:22.902 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-04-23T10:41:22.902 INFO:teuthology.orchestra.run.smithi119.stdout: "req_rmdir_latency": { 2024-04-23T10:41:22.903 INFO:teuthology.orchestra.run.smithi119.stdout: "avgcount": 1, 2024-04-23T10:41:22.903 INFO:teuthology.orchestra.run.smithi119.stdout: "sum": 0.004394346, 2024-04-23T10:41:22.903 INFO:teuthology.orchestra.run.smithi119.stdout: "avgtime": 0.004394346 2024-04-23T10:41:22.903 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-04-23T10:41:22.903 INFO:teuthology.orchestra.run.smithi119.stdout: "req_rmsnap_latency": { 2024-04-23T10:41:22.903 INFO:teuthology.orchestra.run.smithi119.stdout: "avgcount": 9, 2024-04-23T10:41:22.903 INFO:teuthology.orchestra.run.smithi119.stdout: "sum": 8.776098504, 2024-04-23T10:41:22.903 INFO:teuthology.orchestra.run.smithi119.stdout: "avgtime": 0.975122056 2024-04-23T10:41:22.903 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-04-23T10:41:22.904 INFO:teuthology.orchestra.run.smithi119.stdout: "req_rmxattr_latency": { 2024-04-23T10:41:22.904 INFO:teuthology.orchestra.run.smithi119.stdout: "avgcount": 0, 2024-04-23T10:41:22.904 INFO:teuthology.orchestra.run.smithi119.stdout: "sum": 0.000000000, 2024-04-23T10:41:22.904 INFO:teuthology.orchestra.run.smithi119.stdout: "avgtime": 0.000000000 2024-04-23T10:41:22.904 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-04-23T10:41:22.904 INFO:teuthology.orchestra.run.smithi119.stdout: "req_setattr_latency": { 2024-04-23T10:41:22.904 INFO:teuthology.orchestra.run.smithi119.stdout: "avgcount": 15, 2024-04-23T10:41:22.904 INFO:teuthology.orchestra.run.smithi119.stdout: "sum": 0.113395535, 2024-04-23T10:41:22.904 INFO:teuthology.orchestra.run.smithi119.stdout: "avgtime": 0.007559702 2024-04-23T10:41:22.904 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-04-23T10:41:22.905 INFO:teuthology.orchestra.run.smithi119.stdout: "req_setdirlayout_latency": { 2024-04-23T10:41:22.905 INFO:teuthology.orchestra.run.smithi119.stdout: "avgcount": 0, 2024-04-23T10:41:22.905 INFO:teuthology.orchestra.run.smithi119.stdout: "sum": 0.000000000, 2024-04-23T10:41:22.905 INFO:teuthology.orchestra.run.smithi119.stdout: "avgtime": 0.000000000 2024-04-23T10:41:22.905 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-04-23T10:41:22.905 INFO:teuthology.orchestra.run.smithi119.stdout: "req_setfilelock_latency": { 2024-04-23T10:41:22.905 INFO:teuthology.orchestra.run.smithi119.stdout: "avgcount": 1008, 2024-04-23T10:41:22.905 INFO:teuthology.orchestra.run.smithi119.stdout: "sum": 1.829499668, 2024-04-23T10:41:22.905 INFO:teuthology.orchestra.run.smithi119.stdout: "avgtime": 0.001814979 2024-04-23T10:41:22.906 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-04-23T10:41:22.906 INFO:teuthology.orchestra.run.smithi119.stdout: "req_setlayout_latency": { 2024-04-23T10:41:22.906 INFO:teuthology.orchestra.run.smithi119.stdout: "avgcount": 0, 2024-04-23T10:41:22.906 INFO:teuthology.orchestra.run.smithi119.stdout: "sum": 0.000000000, 2024-04-23T10:41:22.906 INFO:teuthology.orchestra.run.smithi119.stdout: "avgtime": 0.000000000 2024-04-23T10:41:22.906 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-04-23T10:41:22.906 INFO:teuthology.orchestra.run.smithi119.stdout: "req_setxattr_latency": { 2024-04-23T10:41:22.906 INFO:teuthology.orchestra.run.smithi119.stdout: "avgcount": 7, 2024-04-23T10:41:22.906 INFO:teuthology.orchestra.run.smithi119.stdout: "sum": 0.020300673, 2024-04-23T10:41:22.907 INFO:teuthology.orchestra.run.smithi119.stdout: "avgtime": 0.002900096 2024-04-23T10:41:22.907 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-04-23T10:41:22.907 INFO:teuthology.orchestra.run.smithi119.stdout: "req_symlink_latency": { 2024-04-23T10:41:22.907 INFO:teuthology.orchestra.run.smithi119.stdout: "avgcount": 0, 2024-04-23T10:41:22.907 INFO:teuthology.orchestra.run.smithi119.stdout: "sum": 0.000000000, 2024-04-23T10:41:22.907 INFO:teuthology.orchestra.run.smithi119.stdout: "avgtime": 0.000000000 2024-04-23T10:41:22.907 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-04-23T10:41:22.907 INFO:teuthology.orchestra.run.smithi119.stdout: "req_unlink_latency": { 2024-04-23T10:41:22.907 INFO:teuthology.orchestra.run.smithi119.stdout: "avgcount": 15836, 2024-04-23T10:41:22.907 INFO:teuthology.orchestra.run.smithi119.stdout: "sum": 72.549589546, 2024-04-23T10:41:22.908 INFO:teuthology.orchestra.run.smithi119.stdout: "avgtime": 0.004581307 2024-04-23T10:41:22.908 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-04-23T10:41:22.908 INFO:teuthology.orchestra.run.smithi119.stdout: "cap_revoke_eviction": 0, 2024-04-23T10:41:22.908 INFO:teuthology.orchestra.run.smithi119.stdout: "cap_acquisition_throttle": 0, 2024-04-23T10:41:22.908 INFO:teuthology.orchestra.run.smithi119.stdout: "req_getvxattr_latency": { 2024-04-23T10:41:22.908 INFO:teuthology.orchestra.run.smithi119.stdout: "avgcount": 0, 2024-04-23T10:41:22.908 INFO:teuthology.orchestra.run.smithi119.stdout: "sum": 0.000000000, 2024-04-23T10:41:22.908 INFO:teuthology.orchestra.run.smithi119.stdout: "avgtime": 0.000000000 2024-04-23T10:41:22.909 INFO:teuthology.orchestra.run.smithi119.stdout: } 2024-04-23T10:41:22.909 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-04-23T10:41:22.909 INFO:teuthology.orchestra.run.smithi119.stdout: "mds_sessions": { 2024-04-23T10:41:22.909 INFO:teuthology.orchestra.run.smithi119.stdout: "session_count": 2, 2024-04-23T10:41:22.909 INFO:teuthology.orchestra.run.smithi119.stdout: "session_add": 5, 2024-04-23T10:41:22.909 INFO:teuthology.orchestra.run.smithi119.stdout: "session_remove": 3, 2024-04-23T10:41:22.909 INFO:teuthology.orchestra.run.smithi119.stdout: "sessions_open": 2, 2024-04-23T10:41:22.909 INFO:teuthology.orchestra.run.smithi119.stdout: "sessions_stale": 0, 2024-04-23T10:41:22.909 INFO:teuthology.orchestra.run.smithi119.stdout: "total_load": 25950, 2024-04-23T10:41:22.909 INFO:teuthology.orchestra.run.smithi119.stdout: "average_load": 12975, 2024-04-23T10:41:22.910 INFO:teuthology.orchestra.run.smithi119.stdout: "avg_session_uptime": 868, 2024-04-23T10:41:22.910 INFO:teuthology.orchestra.run.smithi119.stdout: "mdthresh_evicted": 0 2024-04-23T10:41:22.910 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-04-23T10:41:22.910 INFO:teuthology.orchestra.run.smithi119.stdout: "mempool": { 2024-04-23T10:41:22.910 INFO:teuthology.orchestra.run.smithi119.stdout: "bloom_filter_bytes": 360, 2024-04-23T10:41:22.910 INFO:teuthology.orchestra.run.smithi119.stdout: "bloom_filter_items": 360, 2024-04-23T10:41:22.910 INFO:teuthology.orchestra.run.smithi119.stdout: "bluestore_alloc_bytes": 0, 2024-04-23T10:41:22.910 INFO:teuthology.orchestra.run.smithi119.stdout: "bluestore_alloc_items": 0, 2024-04-23T10:41:22.910 INFO:teuthology.orchestra.run.smithi119.stdout: "bluestore_cache_data_bytes": 0, 2024-04-23T10:41:22.911 INFO:teuthology.orchestra.run.smithi119.stdout: "bluestore_cache_data_items": 0, 2024-04-23T10:41:22.911 INFO:teuthology.orchestra.run.smithi119.stdout: "bluestore_cache_onode_bytes": 0, 2024-04-23T10:41:22.911 INFO:teuthology.orchestra.run.smithi119.stdout: "bluestore_cache_onode_items": 0, 2024-04-23T10:41:22.911 INFO:teuthology.orchestra.run.smithi119.stdout: "bluestore_cache_meta_bytes": 0, 2024-04-23T10:41:22.911 INFO:teuthology.orchestra.run.smithi119.stdout: "bluestore_cache_meta_items": 0, 2024-04-23T10:41:22.911 INFO:teuthology.orchestra.run.smithi119.stdout: "bluestore_cache_other_bytes": 0, 2024-04-23T10:41:22.911 INFO:teuthology.orchestra.run.smithi119.stdout: "bluestore_cache_other_items": 0, 2024-04-23T10:41:22.911 INFO:teuthology.orchestra.run.smithi119.stdout: "bluestore_cache_buffer_bytes": 0, 2024-04-23T10:41:22.911 INFO:teuthology.orchestra.run.smithi119.stdout: "bluestore_cache_buffer_items": 0, 2024-04-23T10:41:22.912 INFO:teuthology.orchestra.run.smithi119.stdout: "bluestore_extent_bytes": 0, 2024-04-23T10:41:22.912 INFO:teuthology.orchestra.run.smithi119.stdout: "bluestore_extent_items": 0, 2024-04-23T10:41:22.912 INFO:teuthology.orchestra.run.smithi119.stdout: "bluestore_blob_bytes": 0, 2024-04-23T10:41:22.912 INFO:teuthology.orchestra.run.smithi119.stdout: "bluestore_blob_items": 0, 2024-04-23T10:41:22.912 INFO:teuthology.orchestra.run.smithi119.stdout: "bluestore_shared_blob_bytes": 0, 2024-04-23T10:41:22.912 INFO:teuthology.orchestra.run.smithi119.stdout: "bluestore_shared_blob_items": 0, 2024-04-23T10:41:22.912 INFO:teuthology.orchestra.run.smithi119.stdout: "bluestore_inline_bl_bytes": 0, 2024-04-23T10:41:22.912 INFO:teuthology.orchestra.run.smithi119.stdout: "bluestore_inline_bl_items": 0, 2024-04-23T10:41:22.912 INFO:teuthology.orchestra.run.smithi119.stdout: "bluestore_fsck_bytes": 0, 2024-04-23T10:41:22.913 INFO:teuthology.orchestra.run.smithi119.stdout: "bluestore_fsck_items": 0, 2024-04-23T10:41:22.913 INFO:teuthology.orchestra.run.smithi119.stdout: "bluestore_txc_bytes": 0, 2024-04-23T10:41:22.913 INFO:teuthology.orchestra.run.smithi119.stdout: "bluestore_txc_items": 0, 2024-04-23T10:41:22.913 INFO:teuthology.orchestra.run.smithi119.stdout: "bluestore_writing_deferred_bytes": 0, 2024-04-23T10:41:22.913 INFO:teuthology.orchestra.run.smithi119.stdout: "bluestore_writing_deferred_items": 0, 2024-04-23T10:41:22.913 INFO:teuthology.orchestra.run.smithi119.stdout: "bluestore_writing_bytes": 0, 2024-04-23T10:41:22.913 INFO:teuthology.orchestra.run.smithi119.stdout: "bluestore_writing_items": 0, 2024-04-23T10:41:22.913 INFO:teuthology.orchestra.run.smithi119.stdout: "bluefs_bytes": 0, 2024-04-23T10:41:22.913 INFO:teuthology.orchestra.run.smithi119.stdout: "bluefs_items": 0, 2024-04-23T10:41:22.914 INFO:teuthology.orchestra.run.smithi119.stdout: "bluefs_file_reader_bytes": 0, 2024-04-23T10:41:22.914 INFO:teuthology.orchestra.run.smithi119.stdout: "bluefs_file_reader_items": 0, 2024-04-23T10:41:22.914 INFO:teuthology.orchestra.run.smithi119.stdout: "bluefs_file_writer_bytes": 0, 2024-04-23T10:41:22.914 INFO:teuthology.orchestra.run.smithi119.stdout: "bluefs_file_writer_items": 0, 2024-04-23T10:41:22.914 INFO:teuthology.orchestra.run.smithi119.stdout: "buffer_anon_bytes": 2084359, 2024-04-23T10:41:22.914 INFO:teuthology.orchestra.run.smithi119.stdout: "buffer_anon_items": 1779, 2024-04-23T10:41:22.914 INFO:teuthology.orchestra.run.smithi119.stdout: "buffer_meta_bytes": 0, 2024-04-23T10:41:22.914 INFO:teuthology.orchestra.run.smithi119.stdout: "buffer_meta_items": 0, 2024-04-23T10:41:22.914 INFO:teuthology.orchestra.run.smithi119.stdout: "osd_bytes": 0, 2024-04-23T10:41:22.915 INFO:teuthology.orchestra.run.smithi119.stdout: "osd_items": 0, 2024-04-23T10:41:22.915 INFO:teuthology.orchestra.run.smithi119.stdout: "osd_mapbl_bytes": 0, 2024-04-23T10:41:22.915 INFO:teuthology.orchestra.run.smithi119.stdout: "osd_mapbl_items": 0, 2024-04-23T10:41:22.915 INFO:teuthology.orchestra.run.smithi119.stdout: "osd_pglog_bytes": 0, 2024-04-23T10:41:22.915 INFO:teuthology.orchestra.run.smithi119.stdout: "osd_pglog_items": 0, 2024-04-23T10:41:22.915 INFO:teuthology.orchestra.run.smithi119.stdout: "osdmap_bytes": 8416, 2024-04-23T10:41:22.915 INFO:teuthology.orchestra.run.smithi119.stdout: "osdmap_items": 175, 2024-04-23T10:41:22.915 INFO:teuthology.orchestra.run.smithi119.stdout: "osdmap_mapping_bytes": 0, 2024-04-23T10:41:22.915 INFO:teuthology.orchestra.run.smithi119.stdout: "osdmap_mapping_items": 0, 2024-04-23T10:41:22.915 INFO:teuthology.orchestra.run.smithi119.stdout: "pgmap_bytes": 0, 2024-04-23T10:41:22.916 INFO:teuthology.orchestra.run.smithi119.stdout: "pgmap_items": 0, 2024-04-23T10:41:22.916 INFO:teuthology.orchestra.run.smithi119.stdout: "mds_co_bytes": 6141813, 2024-04-23T10:41:22.916 INFO:teuthology.orchestra.run.smithi119.stdout: "mds_co_items": 128336, 2024-04-23T10:41:22.916 INFO:teuthology.orchestra.run.smithi119.stdout: "unittest_1_bytes": 0, 2024-04-23T10:41:22.916 INFO:teuthology.orchestra.run.smithi119.stdout: "unittest_1_items": 0, 2024-04-23T10:41:22.916 INFO:teuthology.orchestra.run.smithi119.stdout: "unittest_2_bytes": 0, 2024-04-23T10:41:22.916 INFO:teuthology.orchestra.run.smithi119.stdout: "unittest_2_items": 0 2024-04-23T10:41:22.916 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-04-23T10:41:22.916 INFO:teuthology.orchestra.run.smithi119.stdout: "objecter": { 2024-04-23T10:41:22.917 INFO:teuthology.orchestra.run.smithi119.stdout: "op_active": 0, 2024-04-23T10:41:22.917 INFO:teuthology.orchestra.run.smithi119.stdout: "op_laggy": 0, 2024-04-23T10:41:22.917 INFO:teuthology.orchestra.run.smithi119.stdout: "op_send": 73033, 2024-04-23T10:41:22.917 INFO:teuthology.orchestra.run.smithi119.stdout: "op_send_bytes": 429392983, 2024-04-23T10:41:22.917 INFO:teuthology.orchestra.run.smithi119.stdout: "op_resend": 37, 2024-04-23T10:41:22.917 INFO:teuthology.orchestra.run.smithi119.stdout: "op_reply": 73033, 2024-04-23T10:41:22.917 INFO:teuthology.orchestra.run.smithi119.stdout: "op_latency": { 2024-04-23T10:41:22.917 INFO:teuthology.orchestra.run.smithi119.stdout: "avgcount": 73033, 2024-04-23T10:41:22.917 INFO:teuthology.orchestra.run.smithi119.stdout: "sum": 745.488011975, 2024-04-23T10:41:22.917 INFO:teuthology.orchestra.run.smithi119.stdout: "avgtime": 0.010207550 2024-04-23T10:41:22.918 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-04-23T10:41:22.918 INFO:teuthology.orchestra.run.smithi119.stdout: "op_inflight": 0, 2024-04-23T10:41:22.918 INFO:teuthology.orchestra.run.smithi119.stdout: "oplen_avg": { 2024-04-23T10:41:22.918 INFO:teuthology.orchestra.run.smithi119.stdout: "avgcount": 73033, 2024-04-23T10:41:22.918 INFO:teuthology.orchestra.run.smithi119.stdout: "sum": 97524 2024-04-23T10:41:22.918 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-04-23T10:41:22.918 INFO:teuthology.orchestra.run.smithi119.stdout: "op": 73033, 2024-04-23T10:41:22.918 INFO:teuthology.orchestra.run.smithi119.stdout: "op_r": 626, 2024-04-23T10:41:22.919 INFO:teuthology.orchestra.run.smithi119.stdout: "op_w": 72407, 2024-04-23T10:41:22.919 INFO:teuthology.orchestra.run.smithi119.stdout: "op_rmw": 0, 2024-04-23T10:41:22.919 INFO:teuthology.orchestra.run.smithi119.stdout: "op_pg": 0, 2024-04-23T10:41:22.919 INFO:teuthology.orchestra.run.smithi119.stdout: "osdop_stat": 1693, 2024-04-23T10:41:22.919 INFO:teuthology.orchestra.run.smithi119.stdout: "osdop_create": 9719, 2024-04-23T10:41:22.919 INFO:teuthology.orchestra.run.smithi119.stdout: "osdop_read": 608, 2024-04-23T10:41:22.919 INFO:teuthology.orchestra.run.smithi119.stdout: "osdop_write": 43140, 2024-04-23T10:41:22.919 INFO:teuthology.orchestra.run.smithi119.stdout: "osdop_writefull": 596, 2024-04-23T10:41:22.919 INFO:teuthology.orchestra.run.smithi119.stdout: "osdop_writesame": 0, 2024-04-23T10:41:22.920 INFO:teuthology.orchestra.run.smithi119.stdout: "osdop_append": 0, 2024-04-23T10:41:22.920 INFO:teuthology.orchestra.run.smithi119.stdout: "osdop_zero": 0, 2024-04-23T10:41:22.920 INFO:teuthology.orchestra.run.smithi119.stdout: "osdop_truncate": 9, 2024-04-23T10:41:22.920 INFO:teuthology.orchestra.run.smithi119.stdout: "osdop_delete": 16304, 2024-04-23T10:41:22.920 INFO:teuthology.orchestra.run.smithi119.stdout: "osdop_mapext": 0, 2024-04-23T10:41:22.920 INFO:teuthology.orchestra.run.smithi119.stdout: "osdop_sparse_read": 0, 2024-04-23T10:41:22.920 INFO:teuthology.orchestra.run.smithi119.stdout: "osdop_clonerange": 0, 2024-04-23T10:41:22.920 INFO:teuthology.orchestra.run.smithi119.stdout: "osdop_getxattr": 9, 2024-04-23T10:41:22.920 INFO:teuthology.orchestra.run.smithi119.stdout: "osdop_setxattr": 19438, 2024-04-23T10:41:22.920 INFO:teuthology.orchestra.run.smithi119.stdout: "osdop_cmpxattr": 0, 2024-04-23T10:41:22.921 INFO:teuthology.orchestra.run.smithi119.stdout: "osdop_rmxattr": 0, 2024-04-23T10:41:22.921 INFO:teuthology.orchestra.run.smithi119.stdout: "osdop_resetxattrs": 0, 2024-04-23T10:41:22.921 INFO:teuthology.orchestra.run.smithi119.stdout: "osdop_call": 0, 2024-04-23T10:41:22.921 INFO:teuthology.orchestra.run.smithi119.stdout: "osdop_watch": 0, 2024-04-23T10:41:22.921 INFO:teuthology.orchestra.run.smithi119.stdout: "osdop_notify": 0, 2024-04-23T10:41:22.921 INFO:teuthology.orchestra.run.smithi119.stdout: "osdop_src_cmpxattr": 0, 2024-04-23T10:41:22.921 INFO:teuthology.orchestra.run.smithi119.stdout: "osdop_pgls": 0, 2024-04-23T10:41:22.921 INFO:teuthology.orchestra.run.smithi119.stdout: "osdop_pgls_filter": 0, 2024-04-23T10:41:22.921 INFO:teuthology.orchestra.run.smithi119.stdout: "osdop_other": 0, 2024-04-23T10:41:22.922 INFO:teuthology.orchestra.run.smithi119.stdout: "linger_active": 0, 2024-04-23T10:41:22.922 INFO:teuthology.orchestra.run.smithi119.stdout: "linger_send": 0, 2024-04-23T10:41:22.922 INFO:teuthology.orchestra.run.smithi119.stdout: "linger_resend": 0, 2024-04-23T10:41:22.922 INFO:teuthology.orchestra.run.smithi119.stdout: "linger_ping": 0, 2024-04-23T10:41:22.922 INFO:teuthology.orchestra.run.smithi119.stdout: "poolop_active": 0, 2024-04-23T10:41:22.922 INFO:teuthology.orchestra.run.smithi119.stdout: "poolop_send": 0, 2024-04-23T10:41:22.922 INFO:teuthology.orchestra.run.smithi119.stdout: "poolop_resend": 0, 2024-04-23T10:41:22.922 INFO:teuthology.orchestra.run.smithi119.stdout: "poolstat_active": 0, 2024-04-23T10:41:22.923 INFO:teuthology.orchestra.run.smithi119.stdout: "poolstat_send": 0, 2024-04-23T10:41:22.923 INFO:teuthology.orchestra.run.smithi119.stdout: "poolstat_resend": 0, 2024-04-23T10:41:22.923 INFO:teuthology.orchestra.run.smithi119.stdout: "statfs_active": 0, 2024-04-23T10:41:22.923 INFO:teuthology.orchestra.run.smithi119.stdout: "statfs_send": 0, 2024-04-23T10:41:22.923 INFO:teuthology.orchestra.run.smithi119.stdout: "statfs_resend": 0, 2024-04-23T10:41:22.923 INFO:teuthology.orchestra.run.smithi119.stdout: "command_active": 0, 2024-04-23T10:41:22.923 INFO:teuthology.orchestra.run.smithi119.stdout: "command_send": 0, 2024-04-23T10:41:22.923 INFO:teuthology.orchestra.run.smithi119.stdout: "command_resend": 0, 2024-04-23T10:41:22.923 INFO:teuthology.orchestra.run.smithi119.stdout: "map_epoch": 113, 2024-04-23T10:41:22.923 INFO:teuthology.orchestra.run.smithi119.stdout: "map_full": 0, 2024-04-23T10:41:22.924 INFO:teuthology.orchestra.run.smithi119.stdout: "map_inc": 27, 2024-04-23T10:41:22.924 INFO:teuthology.orchestra.run.smithi119.stdout: "osd_sessions": 12, 2024-04-23T10:41:22.924 INFO:teuthology.orchestra.run.smithi119.stdout: "osd_session_open": 12, 2024-04-23T10:41:22.924 INFO:teuthology.orchestra.run.smithi119.stdout: "osd_session_close": 0, 2024-04-23T10:41:22.924 INFO:teuthology.orchestra.run.smithi119.stdout: "osd_laggy": 0, 2024-04-23T10:41:22.924 INFO:teuthology.orchestra.run.smithi119.stdout: "omap_wr": 4824, 2024-04-23T10:41:22.924 INFO:teuthology.orchestra.run.smithi119.stdout: "omap_rd": 36, 2024-04-23T10:41:22.924 INFO:teuthology.orchestra.run.smithi119.stdout: "omap_del": 1148 2024-04-23T10:41:22.924 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-04-23T10:41:22.924 INFO:teuthology.orchestra.run.smithi119.stdout: "oft": { 2024-04-23T10:41:22.925 INFO:teuthology.orchestra.run.smithi119.stdout: "omap_total_objs": 1, 2024-04-23T10:41:22.925 INFO:teuthology.orchestra.run.smithi119.stdout: "omap_total_kv_pairs": 0, 2024-04-23T10:41:22.925 INFO:teuthology.orchestra.run.smithi119.stdout: "omap_total_updates": 0, 2024-04-23T10:41:22.925 INFO:teuthology.orchestra.run.smithi119.stdout: "omap_total_removes": 0 2024-04-23T10:41:22.925 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-04-23T10:41:22.925 INFO:teuthology.orchestra.run.smithi119.stdout: "osd-slow-ops": { 2024-04-23T10:41:22.925 INFO:teuthology.orchestra.run.smithi119.stdout: "slow_ops_count": 341 2024-04-23T10:41:22.925 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-04-23T10:41:22.925 INFO:teuthology.orchestra.run.smithi119.stdout: "purge_queue": { 2024-04-23T10:41:22.926 INFO:teuthology.orchestra.run.smithi119.stdout: "pq_executing_ops": 0, 2024-04-23T10:41:22.926 INFO:teuthology.orchestra.run.smithi119.stdout: "pq_executing_ops_high_water": 10, 2024-04-23T10:41:22.926 INFO:teuthology.orchestra.run.smithi119.stdout: "pq_executing": 0, 2024-04-23T10:41:22.926 INFO:teuthology.orchestra.run.smithi119.stdout: "pq_executing_high_water": 10, 2024-04-23T10:41:22.926 INFO:teuthology.orchestra.run.smithi119.stdout: "pq_executed_ops": 15584, 2024-04-23T10:41:22.926 INFO:teuthology.orchestra.run.smithi119.stdout: "pq_executed": 15584, 2024-04-23T10:41:22.926 INFO:teuthology.orchestra.run.smithi119.stdout: "pq_item_in_journal": 0 2024-04-23T10:41:22.926 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-04-23T10:41:22.926 INFO:teuthology.orchestra.run.smithi119.stdout: "throttle-msgr_dispatch_throttler-mds": { 2024-04-23T10:41:22.927 INFO:teuthology.orchestra.run.smithi119.stdout: "val": 0, 2024-04-23T10:41:22.927 INFO:teuthology.orchestra.run.smithi119.stdout: "max": 104857600, 2024-04-23T10:41:22.927 INFO:teuthology.orchestra.run.smithi119.stdout: "get_started": 0, 2024-04-23T10:41:22.927 INFO:teuthology.orchestra.run.smithi119.stdout: "get": 521746, 2024-04-23T10:41:22.927 INFO:teuthology.orchestra.run.smithi119.stdout: "get_sum": 94449677, 2024-04-23T10:41:22.927 INFO:teuthology.orchestra.run.smithi119.stdout: "get_or_fail_fail": 0, 2024-04-23T10:41:22.927 INFO:teuthology.orchestra.run.smithi119.stdout: "get_or_fail_success": 521746, 2024-04-23T10:41:22.927 INFO:teuthology.orchestra.run.smithi119.stdout: "take": 0, 2024-04-23T10:41:22.927 INFO:teuthology.orchestra.run.smithi119.stdout: "take_sum": 0, 2024-04-23T10:41:22.928 INFO:teuthology.orchestra.run.smithi119.stdout: "put": 521746, 2024-04-23T10:41:22.928 INFO:teuthology.orchestra.run.smithi119.stdout: "put_sum": 94449677, 2024-04-23T10:41:22.928 INFO:teuthology.orchestra.run.smithi119.stdout: "wait": { 2024-04-23T10:41:22.928 INFO:teuthology.orchestra.run.smithi119.stdout: "avgcount": 0, 2024-04-23T10:41:22.928 INFO:teuthology.orchestra.run.smithi119.stdout: "sum": 0.000000000, 2024-04-23T10:41:22.928 INFO:teuthology.orchestra.run.smithi119.stdout: "avgtime": 0.000000000 2024-04-23T10:41:22.928 INFO:teuthology.orchestra.run.smithi119.stdout: } 2024-04-23T10:41:22.928 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-04-23T10:41:22.928 INFO:teuthology.orchestra.run.smithi119.stdout: "throttle-objecter_bytes": { 2024-04-23T10:41:22.929 INFO:teuthology.orchestra.run.smithi119.stdout: "val": 0, 2024-04-23T10:41:22.929 INFO:teuthology.orchestra.run.smithi119.stdout: "max": 104857600, 2024-04-23T10:41:22.929 INFO:teuthology.orchestra.run.smithi119.stdout: "get_started": 0, 2024-04-23T10:41:22.929 INFO:teuthology.orchestra.run.smithi119.stdout: "get": 0, 2024-04-23T10:41:22.929 INFO:teuthology.orchestra.run.smithi119.stdout: "get_sum": 0, 2024-04-23T10:41:22.929 INFO:teuthology.orchestra.run.smithi119.stdout: "get_or_fail_fail": 0, 2024-04-23T10:41:22.929 INFO:teuthology.orchestra.run.smithi119.stdout: "get_or_fail_success": 0, 2024-04-23T10:41:22.929 INFO:teuthology.orchestra.run.smithi119.stdout: "take": 73033, 2024-04-23T10:41:22.929 INFO:teuthology.orchestra.run.smithi119.stdout: "take_sum": 430969951, 2024-04-23T10:41:22.929 INFO:teuthology.orchestra.run.smithi119.stdout: "put": 73033, 2024-04-23T10:41:22.930 INFO:teuthology.orchestra.run.smithi119.stdout: "put_sum": 430969951, 2024-04-23T10:41:22.930 INFO:teuthology.orchestra.run.smithi119.stdout: "wait": { 2024-04-23T10:41:22.930 INFO:teuthology.orchestra.run.smithi119.stdout: "avgcount": 0, 2024-04-23T10:41:22.930 INFO:teuthology.orchestra.run.smithi119.stdout: "sum": 0.000000000, 2024-04-23T10:41:22.930 INFO:teuthology.orchestra.run.smithi119.stdout: "avgtime": 0.000000000 2024-04-23T10:41:22.930 INFO:teuthology.orchestra.run.smithi119.stdout: } 2024-04-23T10:41:22.930 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-04-23T10:41:22.930 INFO:teuthology.orchestra.run.smithi119.stdout: "throttle-objecter_ops": { 2024-04-23T10:41:22.930 INFO:teuthology.orchestra.run.smithi119.stdout: "val": 0, 2024-04-23T10:41:22.931 INFO:teuthology.orchestra.run.smithi119.stdout: "max": 1024, 2024-04-23T10:41:22.931 INFO:teuthology.orchestra.run.smithi119.stdout: "get_started": 0, 2024-04-23T10:41:22.931 INFO:teuthology.orchestra.run.smithi119.stdout: "get": 0, 2024-04-23T10:41:22.931 INFO:teuthology.orchestra.run.smithi119.stdout: "get_sum": 0, 2024-04-23T10:41:22.931 INFO:teuthology.orchestra.run.smithi119.stdout: "get_or_fail_fail": 0, 2024-04-23T10:41:22.931 INFO:teuthology.orchestra.run.smithi119.stdout: "get_or_fail_success": 0, 2024-04-23T10:41:22.931 INFO:teuthology.orchestra.run.smithi119.stdout: "take": 73033, 2024-04-23T10:41:22.931 INFO:teuthology.orchestra.run.smithi119.stdout: "take_sum": 73033, 2024-04-23T10:41:22.931 INFO:teuthology.orchestra.run.smithi119.stdout: "put": 73033, 2024-04-23T10:41:22.931 INFO:teuthology.orchestra.run.smithi119.stdout: "put_sum": 73033, 2024-04-23T10:41:22.932 INFO:teuthology.orchestra.run.smithi119.stdout: "wait": { 2024-04-23T10:41:22.932 INFO:teuthology.orchestra.run.smithi119.stdout: "avgcount": 0, 2024-04-23T10:41:22.932 INFO:teuthology.orchestra.run.smithi119.stdout: "sum": 0.000000000, 2024-04-23T10:41:22.932 INFO:teuthology.orchestra.run.smithi119.stdout: "avgtime": 0.000000000 2024-04-23T10:41:22.932 INFO:teuthology.orchestra.run.smithi119.stdout: } 2024-04-23T10:41:22.932 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-04-23T10:41:22.932 INFO:teuthology.orchestra.run.smithi119.stdout: "throttle-write_buf_throttle": { 2024-04-23T10:41:22.932 INFO:teuthology.orchestra.run.smithi119.stdout: "val": 0, 2024-04-23T10:41:22.933 INFO:teuthology.orchestra.run.smithi119.stdout: "max": 3758096384, 2024-04-23T10:41:22.933 INFO:teuthology.orchestra.run.smithi119.stdout: "get_started": 0, 2024-04-23T10:41:22.933 INFO:teuthology.orchestra.run.smithi119.stdout: "get": 15584, 2024-04-23T10:41:22.933 INFO:teuthology.orchestra.run.smithi119.stdout: "get_sum": 1577256, 2024-04-23T10:41:22.933 INFO:teuthology.orchestra.run.smithi119.stdout: "get_or_fail_fail": 0, 2024-04-23T10:41:22.933 INFO:teuthology.orchestra.run.smithi119.stdout: "get_or_fail_success": 15584, 2024-04-23T10:41:22.933 INFO:teuthology.orchestra.run.smithi119.stdout: "take": 0, 2024-04-23T10:41:22.933 INFO:teuthology.orchestra.run.smithi119.stdout: "take_sum": 0, 2024-04-23T10:41:22.933 INFO:teuthology.orchestra.run.smithi119.stdout: "put": 608, 2024-04-23T10:41:22.934 INFO:teuthology.orchestra.run.smithi119.stdout: "put_sum": 1577256, 2024-04-23T10:41:22.934 INFO:teuthology.orchestra.run.smithi119.stdout: "wait": { 2024-04-23T10:41:22.934 INFO:teuthology.orchestra.run.smithi119.stdout: "avgcount": 0, 2024-04-23T10:41:22.934 INFO:teuthology.orchestra.run.smithi119.stdout: "sum": 0.000000000, 2024-04-23T10:41:22.934 INFO:teuthology.orchestra.run.smithi119.stdout: "avgtime": 0.000000000 2024-04-23T10:41:22.934 INFO:teuthology.orchestra.run.smithi119.stdout: } 2024-04-23T10:41:22.934 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-04-23T10:41:22.934 INFO:teuthology.orchestra.run.smithi119.stdout: "throttle-write_buf_throttle-0x5571cf0ca500": { 2024-04-23T10:41:22.934 INFO:teuthology.orchestra.run.smithi119.stdout: "val": 67088, 2024-04-23T10:41:22.935 INFO:teuthology.orchestra.run.smithi119.stdout: "max": 3758096384, 2024-04-23T10:41:22.935 INFO:teuthology.orchestra.run.smithi119.stdout: "get_started": 0, 2024-04-23T10:41:22.935 INFO:teuthology.orchestra.run.smithi119.stdout: "get": 96318, 2024-04-23T10:41:22.935 INFO:teuthology.orchestra.run.smithi119.stdout: "get_sum": 410657622, 2024-04-23T10:41:22.935 INFO:teuthology.orchestra.run.smithi119.stdout: "get_or_fail_fail": 0, 2024-04-23T10:41:22.935 INFO:teuthology.orchestra.run.smithi119.stdout: "get_or_fail_success": 96318, 2024-04-23T10:41:22.936 INFO:teuthology.orchestra.run.smithi119.stdout: "take": 0, 2024-04-23T10:41:22.936 INFO:teuthology.orchestra.run.smithi119.stdout: "take_sum": 0, 2024-04-23T10:41:22.936 INFO:teuthology.orchestra.run.smithi119.stdout: "put": 42532, 2024-04-23T10:41:22.936 INFO:teuthology.orchestra.run.smithi119.stdout: "put_sum": 410590534, 2024-04-23T10:41:22.936 INFO:teuthology.orchestra.run.smithi119.stdout: "wait": { 2024-04-23T10:41:22.936 INFO:teuthology.orchestra.run.smithi119.stdout: "avgcount": 0, 2024-04-23T10:41:22.936 INFO:teuthology.orchestra.run.smithi119.stdout: "sum": 0.000000000, 2024-04-23T10:41:22.936 INFO:teuthology.orchestra.run.smithi119.stdout: "avgtime": 0.000000000 2024-04-23T10:41:22.936 INFO:teuthology.orchestra.run.smithi119.stdout: } 2024-04-23T10:41:22.937 INFO:teuthology.orchestra.run.smithi119.stdout: } 2024-04-23T10:41:22.937 INFO:teuthology.orchestra.run.smithi119.stdout:} 2024-04-23T10:41:23.641 INFO:tasks.check_counter:Daemon mds.b mds.exported=15 2024-04-23T10:41:23.641 INFO:tasks.check_counter:Daemon mds.b mds.imported=19 2024-04-23T10:41:23.641 INFO:tasks.check_counter:Daemon mds.b mds_cache.dir_update=263 2024-04-23T10:41:23.641 INFO:tasks.check_counter:Daemon mds.b mds_cache.dir_update_receipt=99 2024-04-23T10:41:23.641 INFO:tasks.check_counter:Daemon mds.b mds.root_rsnaps=6 2024-04-23T10:41:23.642 INFO:tasks.check_counter:Daemon mds.b mds_server.req_mksnap_latency.avgcount=15 2024-04-23T10:41:23.642 INFO:tasks.check_counter:Daemon mds.b mds_server.req_rmsnap_latency.avgcount=9 2024-04-23T10:41:23.642 DEBUG:tasks.check_counter:Getting stats from e 2024-04-23T10:41:23.642 DEBUG:teuthology.orchestra.run.smithi119:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d69f13d827952bb07bb9b7973bca14f97d764d6f shell --fsid aa240572-0159-11ef-bc93-c7b262605968 -- ceph daemon mds.e perf dump 2024-04-23T10:41:23.889 INFO:teuthology.orchestra.run.smithi119.stderr:Inferring config /var/lib/ceph/aa240572-0159-11ef-bc93-c7b262605968/mon.b/config 2024-04-23T10:41:24.578 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 23 10:41:24 smithi204 ceph-mon[32649]: pgmap v467: 97 pgs: 97 active+clean; 559 MiB data, 4.3 GiB used, 1.0 TiB / 1.0 TiB avail; 604 KiB/s rd, 5.2 MiB/s wr, 188 op/s 2024-04-23T10:41:24.742 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 23 10:41:24 smithi119 ceph-mon[32608]: pgmap v467: 97 pgs: 97 active+clean; 559 MiB data, 4.3 GiB used, 1.0 TiB / 1.0 TiB avail; 604 KiB/s rd, 5.2 MiB/s wr, 188 op/s 2024-04-23T10:41:24.754 INFO:journalctl@ceph.mon.a.smithi114.stdout:Apr 23 10:41:24 smithi114 ceph-mon[29255]: pgmap v467: 97 pgs: 97 active+clean; 559 MiB data, 4.3 GiB used, 1.0 TiB / 1.0 TiB avail; 604 KiB/s rd, 5.2 MiB/s wr, 188 op/s 2024-04-23T10:41:25.786 INFO:teuthology.orchestra.run.smithi119.stdout:{ 2024-04-23T10:41:25.786 INFO:teuthology.orchestra.run.smithi119.stdout: "AsyncMessenger::Worker-0": { 2024-04-23T10:41:25.786 INFO:teuthology.orchestra.run.smithi119.stdout: "msgr_recv_messages": 13516, 2024-04-23T10:41:25.786 INFO:teuthology.orchestra.run.smithi119.stdout: "msgr_send_messages": 13695, 2024-04-23T10:41:25.787 INFO:teuthology.orchestra.run.smithi119.stdout: "msgr_recv_bytes": 3373300, 2024-04-23T10:41:25.787 INFO:teuthology.orchestra.run.smithi119.stdout: "msgr_send_bytes": 66667600, 2024-04-23T10:41:25.787 INFO:teuthology.orchestra.run.smithi119.stdout: "msgr_created_connections": 17, 2024-04-23T10:41:25.787 INFO:teuthology.orchestra.run.smithi119.stdout: "msgr_active_connections": 3, 2024-04-23T10:41:25.787 INFO:teuthology.orchestra.run.smithi119.stdout: "msgr_running_total_time": 0.966889631, 2024-04-23T10:41:25.787 INFO:teuthology.orchestra.run.smithi119.stdout: "msgr_running_send_time": 0.352329411, 2024-04-23T10:41:25.788 INFO:teuthology.orchestra.run.smithi119.stdout: "msgr_running_recv_time": 0.210135246, 2024-04-23T10:41:25.788 INFO:teuthology.orchestra.run.smithi119.stdout: "msgr_running_fast_dispatch_time": 0.332041440, 2024-04-23T10:41:25.788 INFO:teuthology.orchestra.run.smithi119.stdout: "msgr_send_messages_queue_lat": { 2024-04-23T10:41:25.788 INFO:teuthology.orchestra.run.smithi119.stdout: "avgcount": 13693, 2024-04-23T10:41:25.788 INFO:teuthology.orchestra.run.smithi119.stdout: "sum": 0.502135412, 2024-04-23T10:41:25.789 INFO:teuthology.orchestra.run.smithi119.stdout: "avgtime": 0.000036670 2024-04-23T10:41:25.789 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-04-23T10:41:25.789 INFO:teuthology.orchestra.run.smithi119.stdout: "msgr_handle_ack_lat": { 2024-04-23T10:41:25.789 INFO:teuthology.orchestra.run.smithi119.stdout: "avgcount": 19, 2024-04-23T10:41:25.789 INFO:teuthology.orchestra.run.smithi119.stdout: "sum": 0.000002232, 2024-04-23T10:41:25.790 INFO:teuthology.orchestra.run.smithi119.stdout: "avgtime": 0.000000117 2024-04-23T10:41:25.790 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-04-23T10:41:25.790 INFO:teuthology.orchestra.run.smithi119.stdout: "msgr_recv_encrypted_bytes": 2096, 2024-04-23T10:41:25.790 INFO:teuthology.orchestra.run.smithi119.stdout: "msgr_send_encrypted_bytes": 406528 2024-04-23T10:41:25.790 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-04-23T10:41:25.790 INFO:teuthology.orchestra.run.smithi119.stdout: "AsyncMessenger::Worker-1": { 2024-04-23T10:41:25.791 INFO:teuthology.orchestra.run.smithi119.stdout: "msgr_recv_messages": 32492, 2024-04-23T10:41:25.791 INFO:teuthology.orchestra.run.smithi119.stdout: "msgr_send_messages": 31670, 2024-04-23T10:41:25.791 INFO:teuthology.orchestra.run.smithi119.stdout: "msgr_recv_bytes": 8151516, 2024-04-23T10:41:25.791 INFO:teuthology.orchestra.run.smithi119.stdout: "msgr_send_bytes": 98012459, 2024-04-23T10:41:25.791 INFO:teuthology.orchestra.run.smithi119.stdout: "msgr_created_connections": 13, 2024-04-23T10:41:25.791 INFO:teuthology.orchestra.run.smithi119.stdout: "msgr_active_connections": 6, 2024-04-23T10:41:25.792 INFO:teuthology.orchestra.run.smithi119.stdout: "msgr_running_total_time": 2.544551507, 2024-04-23T10:41:25.792 INFO:teuthology.orchestra.run.smithi119.stdout: "msgr_running_send_time": 0.931789353, 2024-04-23T10:41:25.792 INFO:teuthology.orchestra.run.smithi119.stdout: "msgr_running_recv_time": 0.929041754, 2024-04-23T10:41:25.792 INFO:teuthology.orchestra.run.smithi119.stdout: "msgr_running_fast_dispatch_time": 0.722266323, 2024-04-23T10:41:25.792 INFO:teuthology.orchestra.run.smithi119.stdout: "msgr_send_messages_queue_lat": { 2024-04-23T10:41:25.792 INFO:teuthology.orchestra.run.smithi119.stdout: "avgcount": 31667, 2024-04-23T10:41:25.793 INFO:teuthology.orchestra.run.smithi119.stdout: "sum": 1.560362189, 2024-04-23T10:41:25.793 INFO:teuthology.orchestra.run.smithi119.stdout: "avgtime": 0.000049274 2024-04-23T10:41:25.793 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-04-23T10:41:25.793 INFO:teuthology.orchestra.run.smithi119.stdout: "msgr_handle_ack_lat": { 2024-04-23T10:41:25.793 INFO:teuthology.orchestra.run.smithi119.stdout: "avgcount": 29617, 2024-04-23T10:41:25.794 INFO:teuthology.orchestra.run.smithi119.stdout: "sum": 0.005910487, 2024-04-23T10:41:25.794 INFO:teuthology.orchestra.run.smithi119.stdout: "avgtime": 0.000000199 2024-04-23T10:41:25.794 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-04-23T10:41:25.794 INFO:teuthology.orchestra.run.smithi119.stdout: "msgr_recv_encrypted_bytes": 1088, 2024-04-23T10:41:25.794 INFO:teuthology.orchestra.run.smithi119.stdout: "msgr_send_encrypted_bytes": 127664 2024-04-23T10:41:25.795 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-04-23T10:41:25.795 INFO:teuthology.orchestra.run.smithi119.stdout: "AsyncMessenger::Worker-2": { 2024-04-23T10:41:25.795 INFO:teuthology.orchestra.run.smithi119.stdout: "msgr_recv_messages": 310024, 2024-04-23T10:41:25.795 INFO:teuthology.orchestra.run.smithi119.stdout: "msgr_send_messages": 359902, 2024-04-23T10:41:25.795 INFO:teuthology.orchestra.run.smithi119.stdout: "msgr_recv_bytes": 96571347, 2024-04-23T10:41:25.795 INFO:teuthology.orchestra.run.smithi119.stdout: "msgr_send_bytes": 198298294, 2024-04-23T10:41:25.796 INFO:teuthology.orchestra.run.smithi119.stdout: "msgr_created_connections": 8, 2024-04-23T10:41:25.796 INFO:teuthology.orchestra.run.smithi119.stdout: "msgr_active_connections": 5, 2024-04-23T10:41:25.796 INFO:teuthology.orchestra.run.smithi119.stdout: "msgr_running_total_time": 16.569683368, 2024-04-23T10:41:25.796 INFO:teuthology.orchestra.run.smithi119.stdout: "msgr_running_send_time": 7.625553664, 2024-04-23T10:41:25.796 INFO:teuthology.orchestra.run.smithi119.stdout: "msgr_running_recv_time": 10.910794213, 2024-04-23T10:41:25.797 INFO:teuthology.orchestra.run.smithi119.stdout: "msgr_running_fast_dispatch_time": 0.331972032, 2024-04-23T10:41:25.797 INFO:teuthology.orchestra.run.smithi119.stdout: "msgr_send_messages_queue_lat": { 2024-04-23T10:41:25.797 INFO:teuthology.orchestra.run.smithi119.stdout: "avgcount": 359902, 2024-04-23T10:41:25.797 INFO:teuthology.orchestra.run.smithi119.stdout: "sum": 8.362898354, 2024-04-23T10:41:25.797 INFO:teuthology.orchestra.run.smithi119.stdout: "avgtime": 0.000023236 2024-04-23T10:41:25.797 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-04-23T10:41:25.798 INFO:teuthology.orchestra.run.smithi119.stdout: "msgr_handle_ack_lat": { 2024-04-23T10:41:25.798 INFO:teuthology.orchestra.run.smithi119.stdout: "avgcount": 626891, 2024-04-23T10:41:25.798 INFO:teuthology.orchestra.run.smithi119.stdout: "sum": 0.089415122, 2024-04-23T10:41:25.798 INFO:teuthology.orchestra.run.smithi119.stdout: "avgtime": 0.000000142 2024-04-23T10:41:25.798 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-04-23T10:41:25.798 INFO:teuthology.orchestra.run.smithi119.stdout: "msgr_recv_encrypted_bytes": 1771728, 2024-04-23T10:41:25.799 INFO:teuthology.orchestra.run.smithi119.stdout: "msgr_send_encrypted_bytes": 181344 2024-04-23T10:41:25.799 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-04-23T10:41:25.799 INFO:teuthology.orchestra.run.smithi119.stdout: "cct": { 2024-04-23T10:41:25.799 INFO:teuthology.orchestra.run.smithi119.stdout: "total_workers": 1, 2024-04-23T10:41:25.799 INFO:teuthology.orchestra.run.smithi119.stdout: "unhealthy_workers": 0 2024-04-23T10:41:25.799 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-04-23T10:41:25.800 INFO:teuthology.orchestra.run.smithi119.stdout: "finisher-MDSRank": { 2024-04-23T10:41:25.800 INFO:teuthology.orchestra.run.smithi119.stdout: "queue_len": 0, 2024-04-23T10:41:25.800 INFO:teuthology.orchestra.run.smithi119.stdout: "complete_latency": { 2024-04-23T10:41:25.800 INFO:teuthology.orchestra.run.smithi119.stdout: "avgcount": 41306, 2024-04-23T10:41:25.800 INFO:teuthology.orchestra.run.smithi119.stdout: "sum": 33.308029125, 2024-04-23T10:41:25.800 INFO:teuthology.orchestra.run.smithi119.stdout: "avgtime": 0.000806372 2024-04-23T10:41:25.801 INFO:teuthology.orchestra.run.smithi119.stdout: } 2024-04-23T10:41:25.801 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-04-23T10:41:25.801 INFO:teuthology.orchestra.run.smithi119.stdout: "finisher-PurgeQueue": { 2024-04-23T10:41:25.801 INFO:teuthology.orchestra.run.smithi119.stdout: "queue_len": 0, 2024-04-23T10:41:25.801 INFO:teuthology.orchestra.run.smithi119.stdout: "complete_latency": { 2024-04-23T10:41:25.801 INFO:teuthology.orchestra.run.smithi119.stdout: "avgcount": 22828, 2024-04-23T10:41:25.802 INFO:teuthology.orchestra.run.smithi119.stdout: "sum": 2.457499211, 2024-04-23T10:41:25.802 INFO:teuthology.orchestra.run.smithi119.stdout: "avgtime": 0.000107652 2024-04-23T10:41:25.802 INFO:teuthology.orchestra.run.smithi119.stdout: } 2024-04-23T10:41:25.802 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-04-23T10:41:25.802 INFO:teuthology.orchestra.run.smithi119.stdout: "mds": { 2024-04-23T10:41:25.803 INFO:teuthology.orchestra.run.smithi119.stdout: "request": 93267, 2024-04-23T10:41:25.803 INFO:teuthology.orchestra.run.smithi119.stdout: "reply": 80762, 2024-04-23T10:41:25.803 INFO:teuthology.orchestra.run.smithi119.stdout: "reply_latency": { 2024-04-23T10:41:25.803 INFO:teuthology.orchestra.run.smithi119.stdout: "avgcount": 80762, 2024-04-23T10:41:25.803 INFO:teuthology.orchestra.run.smithi119.stdout: "sum": 235.815350058, 2024-04-23T10:41:25.803 INFO:teuthology.orchestra.run.smithi119.stdout: "avgtime": 0.002919880 2024-04-23T10:41:25.803 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-04-23T10:41:25.804 INFO:teuthology.orchestra.run.smithi119.stdout: "slow_reply": 0, 2024-04-23T10:41:25.804 INFO:teuthology.orchestra.run.smithi119.stdout: "forward": 12503, 2024-04-23T10:41:25.804 INFO:teuthology.orchestra.run.smithi119.stdout: "dir_fetch_complete": 4, 2024-04-23T10:41:25.804 INFO:teuthology.orchestra.run.smithi119.stdout: "dir_fetch_keys": 0, 2024-04-23T10:41:25.804 INFO:teuthology.orchestra.run.smithi119.stdout: "dir_commit": 35, 2024-04-23T10:41:25.804 INFO:teuthology.orchestra.run.smithi119.stdout: "dir_split": 3, 2024-04-23T10:41:25.805 INFO:teuthology.orchestra.run.smithi119.stdout: "dir_merge": 0, 2024-04-23T10:41:25.805 INFO:teuthology.orchestra.run.smithi119.stdout: "inodes": 1578, 2024-04-23T10:41:25.805 INFO:teuthology.orchestra.run.smithi119.stdout: "inodes_top": 121, 2024-04-23T10:41:25.805 INFO:teuthology.orchestra.run.smithi119.stdout: "inodes_bottom": 1348, 2024-04-23T10:41:25.805 INFO:teuthology.orchestra.run.smithi119.stdout: "inodes_pin_tail": 109, 2024-04-23T10:41:25.805 INFO:teuthology.orchestra.run.smithi119.stdout: "inodes_pinned": 1404, 2024-04-23T10:41:25.805 INFO:teuthology.orchestra.run.smithi119.stdout: "inodes_expired": 369, 2024-04-23T10:41:25.806 INFO:teuthology.orchestra.run.smithi119.stdout: "inodes_with_caps": 487, 2024-04-23T10:41:25.806 INFO:teuthology.orchestra.run.smithi119.stdout: "caps": 487, 2024-04-23T10:41:25.806 INFO:teuthology.orchestra.run.smithi119.stdout: "subtrees": 13, 2024-04-23T10:41:25.806 INFO:teuthology.orchestra.run.smithi119.stdout: "traverse": 187792, 2024-04-23T10:41:25.806 INFO:teuthology.orchestra.run.smithi119.stdout: "traverse_hit": 80262, 2024-04-23T10:41:25.806 INFO:teuthology.orchestra.run.smithi119.stdout: "traverse_forward": 12492, 2024-04-23T10:41:25.807 INFO:teuthology.orchestra.run.smithi119.stdout: "traverse_discover": 155, 2024-04-23T10:41:25.807 INFO:teuthology.orchestra.run.smithi119.stdout: "traverse_dir_fetch": 0, 2024-04-23T10:41:25.807 INFO:teuthology.orchestra.run.smithi119.stdout: "traverse_remote_ino": 0, 2024-04-23T10:41:25.807 INFO:teuthology.orchestra.run.smithi119.stdout: "traverse_lock": 1843, 2024-04-23T10:41:25.807 INFO:teuthology.orchestra.run.smithi119.stdout: "load_cent": 183586, 2024-04-23T10:41:25.807 INFO:teuthology.orchestra.run.smithi119.stdout: "q": 0, 2024-04-23T10:41:25.808 INFO:teuthology.orchestra.run.smithi119.stdout: "exported": 9, 2024-04-23T10:41:25.808 INFO:teuthology.orchestra.run.smithi119.stdout: "exported_inodes": 199, 2024-04-23T10:41:25.808 INFO:teuthology.orchestra.run.smithi119.stdout: "imported": 9, 2024-04-23T10:41:25.808 INFO:teuthology.orchestra.run.smithi119.stdout: "imported_inodes": 551, 2024-04-23T10:41:25.808 INFO:teuthology.orchestra.run.smithi119.stdout: "openino_dir_fetch": 0, 2024-04-23T10:41:25.808 INFO:teuthology.orchestra.run.smithi119.stdout: "openino_backtrace_fetch": 0, 2024-04-23T10:41:25.809 INFO:teuthology.orchestra.run.smithi119.stdout: "openino_peer_discover": 0, 2024-04-23T10:41:25.809 INFO:teuthology.orchestra.run.smithi119.stdout: "root_rfiles": 733, 2024-04-23T10:41:25.809 INFO:teuthology.orchestra.run.smithi119.stdout: "root_rbytes": 212014623, 2024-04-23T10:41:25.809 INFO:teuthology.orchestra.run.smithi119.stdout: "root_rsnaps": 6, 2024-04-23T10:41:25.809 INFO:teuthology.orchestra.run.smithi119.stdout: "scrub_backtrace_fetch": 0, 2024-04-23T10:41:25.809 INFO:teuthology.orchestra.run.smithi119.stdout: "scrub_set_tag": 0, 2024-04-23T10:41:25.810 INFO:teuthology.orchestra.run.smithi119.stdout: "scrub_backtrace_repaired": 0, 2024-04-23T10:41:25.810 INFO:teuthology.orchestra.run.smithi119.stdout: "scrub_inotable_repaired": 0, 2024-04-23T10:41:25.810 INFO:teuthology.orchestra.run.smithi119.stdout: "scrub_dir_inodes": 0, 2024-04-23T10:41:25.810 INFO:teuthology.orchestra.run.smithi119.stdout: "scrub_dir_base_inodes": 0, 2024-04-23T10:41:25.810 INFO:teuthology.orchestra.run.smithi119.stdout: "scrub_dirfrag_rstats": 0, 2024-04-23T10:41:25.811 INFO:teuthology.orchestra.run.smithi119.stdout: "scrub_file_inodes": 0, 2024-04-23T10:41:25.811 INFO:teuthology.orchestra.run.smithi119.stdout: "handle_inode_file_caps": 2, 2024-04-23T10:41:25.811 INFO:teuthology.orchestra.run.smithi119.stdout: "ceph_cap_op_revoke": 4766, 2024-04-23T10:41:25.811 INFO:teuthology.orchestra.run.smithi119.stdout: "ceph_cap_op_grant": 51904, 2024-04-23T10:41:25.811 INFO:teuthology.orchestra.run.smithi119.stdout: "ceph_cap_op_trunc": 0, 2024-04-23T10:41:25.811 INFO:teuthology.orchestra.run.smithi119.stdout: "ceph_cap_op_flushsnap_ack": 0, 2024-04-23T10:41:25.811 INFO:teuthology.orchestra.run.smithi119.stdout: "ceph_cap_op_flush_ack": 0, 2024-04-23T10:41:25.812 INFO:teuthology.orchestra.run.smithi119.stdout: "handle_client_caps": 19346, 2024-04-23T10:41:25.812 INFO:teuthology.orchestra.run.smithi119.stdout: "handle_client_caps_dirty": 9743, 2024-04-23T10:41:25.812 INFO:teuthology.orchestra.run.smithi119.stdout: "handle_client_cap_release": 2739, 2024-04-23T10:41:25.812 INFO:teuthology.orchestra.run.smithi119.stdout: "process_request_cap_release": 74892 2024-04-23T10:41:25.812 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-04-23T10:41:25.812 INFO:teuthology.orchestra.run.smithi119.stdout: "mds_cache": { 2024-04-23T10:41:25.813 INFO:teuthology.orchestra.run.smithi119.stdout: "dir_update": 350, 2024-04-23T10:41:25.813 INFO:teuthology.orchestra.run.smithi119.stdout: "dir_update_receipt": 1494, 2024-04-23T10:41:25.813 INFO:teuthology.orchestra.run.smithi119.stdout: "dir_try_discover": 808, 2024-04-23T10:41:25.813 INFO:teuthology.orchestra.run.smithi119.stdout: "dir_send_discover": 1421, 2024-04-23T10:41:25.813 INFO:teuthology.orchestra.run.smithi119.stdout: "dir_handle_discover": 534, 2024-04-23T10:41:25.813 INFO:teuthology.orchestra.run.smithi119.stdout: "num_strays": 247, 2024-04-23T10:41:25.813 INFO:teuthology.orchestra.run.smithi119.stdout: "num_strays_delayed": 0, 2024-04-23T10:41:25.814 INFO:teuthology.orchestra.run.smithi119.stdout: "num_strays_enqueuing": 0, 2024-04-23T10:41:25.814 INFO:teuthology.orchestra.run.smithi119.stdout: "strays_created": 14213, 2024-04-23T10:41:25.814 INFO:teuthology.orchestra.run.smithi119.stdout: "strays_enqueued": 13970, 2024-04-23T10:41:25.814 INFO:teuthology.orchestra.run.smithi119.stdout: "strays_reintegrated": 0, 2024-04-23T10:41:25.814 INFO:teuthology.orchestra.run.smithi119.stdout: "strays_migrated": 0, 2024-04-23T10:41:25.814 INFO:teuthology.orchestra.run.smithi119.stdout: "num_recovering_processing": 0, 2024-04-23T10:41:25.815 INFO:teuthology.orchestra.run.smithi119.stdout: "num_recovering_enqueued": 0, 2024-04-23T10:41:25.815 INFO:teuthology.orchestra.run.smithi119.stdout: "num_recovering_prioritized": 0, 2024-04-23T10:41:25.815 INFO:teuthology.orchestra.run.smithi119.stdout: "recovery_started": 0, 2024-04-23T10:41:25.815 INFO:teuthology.orchestra.run.smithi119.stdout: "recovery_completed": 0, 2024-04-23T10:41:25.815 INFO:teuthology.orchestra.run.smithi119.stdout: "ireq_quiesce_path": 5, 2024-04-23T10:41:25.815 INFO:teuthology.orchestra.run.smithi119.stdout: "ireq_quiesce_inode": 141, 2024-04-23T10:41:25.816 INFO:teuthology.orchestra.run.smithi119.stdout: "ireq_enqueue_scrub": 0, 2024-04-23T10:41:25.816 INFO:teuthology.orchestra.run.smithi119.stdout: "ireq_exportdir": 10, 2024-04-23T10:41:25.816 INFO:teuthology.orchestra.run.smithi119.stdout: "ireq_flush": 0, 2024-04-23T10:41:25.816 INFO:teuthology.orchestra.run.smithi119.stdout: "ireq_fragmentdir": 3, 2024-04-23T10:41:25.816 INFO:teuthology.orchestra.run.smithi119.stdout: "ireq_fragstats": 0, 2024-04-23T10:41:25.816 INFO:teuthology.orchestra.run.smithi119.stdout: "ireq_inodestats": 0 2024-04-23T10:41:25.817 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-04-23T10:41:25.817 INFO:teuthology.orchestra.run.smithi119.stdout: "mds_log": { 2024-04-23T10:41:25.817 INFO:teuthology.orchestra.run.smithi119.stdout: "evlrg": 0, 2024-04-23T10:41:25.817 INFO:teuthology.orchestra.run.smithi119.stdout: "evadd": 76412, 2024-04-23T10:41:25.817 INFO:teuthology.orchestra.run.smithi119.stdout: "evex": 0, 2024-04-23T10:41:25.817 INFO:teuthology.orchestra.run.smithi119.stdout: "evtrm": 0, 2024-04-23T10:41:25.817 INFO:teuthology.orchestra.run.smithi119.stdout: "ev": 76412, 2024-04-23T10:41:25.818 INFO:teuthology.orchestra.run.smithi119.stdout: "evexg": 0, 2024-04-23T10:41:25.818 INFO:teuthology.orchestra.run.smithi119.stdout: "evexd": 0, 2024-04-23T10:41:25.818 INFO:teuthology.orchestra.run.smithi119.stdout: "segadd": 102, 2024-04-23T10:41:25.818 INFO:teuthology.orchestra.run.smithi119.stdout: "segex": 0, 2024-04-23T10:41:25.818 INFO:teuthology.orchestra.run.smithi119.stdout: "segtrm": 0, 2024-04-23T10:41:25.818 INFO:teuthology.orchestra.run.smithi119.stdout: "seg": 102, 2024-04-23T10:41:25.818 INFO:teuthology.orchestra.run.smithi119.stdout: "segmjr": 8, 2024-04-23T10:41:25.819 INFO:teuthology.orchestra.run.smithi119.stdout: "segexg": 0, 2024-04-23T10:41:25.819 INFO:teuthology.orchestra.run.smithi119.stdout: "segexd": 0, 2024-04-23T10:41:25.819 INFO:teuthology.orchestra.run.smithi119.stdout: "expos": 4194304, 2024-04-23T10:41:25.819 INFO:teuthology.orchestra.run.smithi119.stdout: "wrpos": 169481669, 2024-04-23T10:41:25.819 INFO:teuthology.orchestra.run.smithi119.stdout: "rdpos": 0, 2024-04-23T10:41:25.819 INFO:teuthology.orchestra.run.smithi119.stdout: "jlat": { 2024-04-23T10:41:25.820 INFO:teuthology.orchestra.run.smithi119.stdout: "avgcount": 21201, 2024-04-23T10:41:25.820 INFO:teuthology.orchestra.run.smithi119.stdout: "sum": 55.403232009, 2024-04-23T10:41:25.820 INFO:teuthology.orchestra.run.smithi119.stdout: "avgtime": 0.002613236 2024-04-23T10:41:25.820 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-04-23T10:41:25.820 INFO:teuthology.orchestra.run.smithi119.stdout: "replayed": 0 2024-04-23T10:41:25.820 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-04-23T10:41:25.820 INFO:teuthology.orchestra.run.smithi119.stdout: "mds_mem": { 2024-04-23T10:41:25.821 INFO:teuthology.orchestra.run.smithi119.stdout: "ino": 1063, 2024-04-23T10:41:25.821 INFO:teuthology.orchestra.run.smithi119.stdout: "ino+": 15291, 2024-04-23T10:41:25.821 INFO:teuthology.orchestra.run.smithi119.stdout: "ino-": 14228, 2024-04-23T10:41:25.821 INFO:teuthology.orchestra.run.smithi119.stdout: "dir": 186, 2024-04-23T10:41:25.821 INFO:teuthology.orchestra.run.smithi119.stdout: "dir+": 785, 2024-04-23T10:41:25.821 INFO:teuthology.orchestra.run.smithi119.stdout: "dir-": 599, 2024-04-23T10:41:25.821 INFO:teuthology.orchestra.run.smithi119.stdout: "dn": 1552, 2024-04-23T10:41:25.822 INFO:teuthology.orchestra.run.smithi119.stdout: "dn+": 22464, 2024-04-23T10:41:25.822 INFO:teuthology.orchestra.run.smithi119.stdout: "dn-": 20912, 2024-04-23T10:41:25.822 INFO:teuthology.orchestra.run.smithi119.stdout: "cap": 339, 2024-04-23T10:41:25.822 INFO:teuthology.orchestra.run.smithi119.stdout: "cap+": 15176, 2024-04-23T10:41:25.822 INFO:teuthology.orchestra.run.smithi119.stdout: "cap-": 14837, 2024-04-23T10:41:25.822 INFO:teuthology.orchestra.run.smithi119.stdout: "rss": 85612, 2024-04-23T10:41:25.823 INFO:teuthology.orchestra.run.smithi119.stdout: "heap": 223608 2024-04-23T10:41:25.823 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-04-23T10:41:25.823 INFO:teuthology.orchestra.run.smithi119.stdout: "mds_server": { 2024-04-23T10:41:25.823 INFO:teuthology.orchestra.run.smithi119.stdout: "dispatch_client_request": 163816, 2024-04-23T10:41:25.823 INFO:teuthology.orchestra.run.smithi119.stdout: "dispatch_server_request": 916, 2024-04-23T10:41:25.823 INFO:teuthology.orchestra.run.smithi119.stdout: "handle_client_request": 93267, 2024-04-23T10:41:25.823 INFO:teuthology.orchestra.run.smithi119.stdout: "handle_client_session": 1637, 2024-04-23T10:41:25.824 INFO:teuthology.orchestra.run.smithi119.stdout: "handle_peer_request": 26433, 2024-04-23T10:41:25.824 INFO:teuthology.orchestra.run.smithi119.stdout: "req_create_latency": { 2024-04-23T10:41:25.824 INFO:teuthology.orchestra.run.smithi119.stdout: "avgcount": 14289, 2024-04-23T10:41:25.824 INFO:teuthology.orchestra.run.smithi119.stdout: "sum": 71.459156953, 2024-04-23T10:41:25.824 INFO:teuthology.orchestra.run.smithi119.stdout: "avgtime": 0.005000990 2024-04-23T10:41:25.824 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-04-23T10:41:25.824 INFO:teuthology.orchestra.run.smithi119.stdout: "req_getattr_latency": { 2024-04-23T10:41:25.825 INFO:teuthology.orchestra.run.smithi119.stdout: "avgcount": 243, 2024-04-23T10:41:25.825 INFO:teuthology.orchestra.run.smithi119.stdout: "sum": 0.486859510, 2024-04-23T10:41:25.825 INFO:teuthology.orchestra.run.smithi119.stdout: "avgtime": 0.002003537 2024-04-23T10:41:25.825 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-04-23T10:41:25.825 INFO:teuthology.orchestra.run.smithi119.stdout: "req_getfilelock_latency": { 2024-04-23T10:41:25.825 INFO:teuthology.orchestra.run.smithi119.stdout: "avgcount": 0, 2024-04-23T10:41:25.825 INFO:teuthology.orchestra.run.smithi119.stdout: "sum": 0.000000000, 2024-04-23T10:41:25.826 INFO:teuthology.orchestra.run.smithi119.stdout: "avgtime": 0.000000000 2024-04-23T10:41:25.826 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-04-23T10:41:25.826 INFO:teuthology.orchestra.run.smithi119.stdout: "req_link_latency": { 2024-04-23T10:41:25.826 INFO:teuthology.orchestra.run.smithi119.stdout: "avgcount": 0, 2024-04-23T10:41:25.826 INFO:teuthology.orchestra.run.smithi119.stdout: "sum": 0.000000000, 2024-04-23T10:41:25.826 INFO:teuthology.orchestra.run.smithi119.stdout: "avgtime": 0.000000000 2024-04-23T10:41:25.826 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-04-23T10:41:25.827 INFO:teuthology.orchestra.run.smithi119.stdout: "req_lookup_latency": { 2024-04-23T10:41:25.827 INFO:teuthology.orchestra.run.smithi119.stdout: "avgcount": 20749, 2024-04-23T10:41:25.827 INFO:teuthology.orchestra.run.smithi119.stdout: "sum": 49.730377048, 2024-04-23T10:41:25.827 INFO:teuthology.orchestra.run.smithi119.stdout: "avgtime": 0.002396760 2024-04-23T10:41:25.827 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-04-23T10:41:25.827 INFO:teuthology.orchestra.run.smithi119.stdout: "req_lookuphash_latency": { 2024-04-23T10:41:25.828 INFO:teuthology.orchestra.run.smithi119.stdout: "avgcount": 0, 2024-04-23T10:41:25.828 INFO:teuthology.orchestra.run.smithi119.stdout: "sum": 0.000000000, 2024-04-23T10:41:25.828 INFO:teuthology.orchestra.run.smithi119.stdout: "avgtime": 0.000000000 2024-04-23T10:41:25.828 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-04-23T10:41:25.828 INFO:teuthology.orchestra.run.smithi119.stdout: "req_lookupino_latency": { 2024-04-23T10:41:25.828 INFO:teuthology.orchestra.run.smithi119.stdout: "avgcount": 1, 2024-04-23T10:41:25.828 INFO:teuthology.orchestra.run.smithi119.stdout: "sum": 0.000125508, 2024-04-23T10:41:25.829 INFO:teuthology.orchestra.run.smithi119.stdout: "avgtime": 0.000125508 2024-04-23T10:41:25.829 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-04-23T10:41:25.829 INFO:teuthology.orchestra.run.smithi119.stdout: "req_lookupname_latency": { 2024-04-23T10:41:25.829 INFO:teuthology.orchestra.run.smithi119.stdout: "avgcount": 0, 2024-04-23T10:41:25.829 INFO:teuthology.orchestra.run.smithi119.stdout: "sum": 0.000000000, 2024-04-23T10:41:25.829 INFO:teuthology.orchestra.run.smithi119.stdout: "avgtime": 0.000000000 2024-04-23T10:41:25.829 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-04-23T10:41:25.830 INFO:teuthology.orchestra.run.smithi119.stdout: "req_lookupparent_latency": { 2024-04-23T10:41:25.830 INFO:teuthology.orchestra.run.smithi119.stdout: "avgcount": 0, 2024-04-23T10:41:25.830 INFO:teuthology.orchestra.run.smithi119.stdout: "sum": 0.000000000, 2024-04-23T10:41:25.830 INFO:teuthology.orchestra.run.smithi119.stdout: "avgtime": 0.000000000 2024-04-23T10:41:25.830 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-04-23T10:41:25.830 INFO:teuthology.orchestra.run.smithi119.stdout: "req_lookupsnap_latency": { 2024-04-23T10:41:25.830 INFO:teuthology.orchestra.run.smithi119.stdout: "avgcount": 0, 2024-04-23T10:41:25.831 INFO:teuthology.orchestra.run.smithi119.stdout: "sum": 0.000000000, 2024-04-23T10:41:25.831 INFO:teuthology.orchestra.run.smithi119.stdout: "avgtime": 0.000000000 2024-04-23T10:41:25.832 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-04-23T10:41:25.832 INFO:teuthology.orchestra.run.smithi119.stdout: "req_lssnap_latency": { 2024-04-23T10:41:25.832 INFO:teuthology.orchestra.run.smithi119.stdout: "avgcount": 0, 2024-04-23T10:41:25.832 INFO:teuthology.orchestra.run.smithi119.stdout: "sum": 0.000000000, 2024-04-23T10:41:25.832 INFO:teuthology.orchestra.run.smithi119.stdout: "avgtime": 0.000000000 2024-04-23T10:41:25.832 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-04-23T10:41:25.833 INFO:teuthology.orchestra.run.smithi119.stdout: "req_mkdir_latency": { 2024-04-23T10:41:25.833 INFO:teuthology.orchestra.run.smithi119.stdout: "avgcount": 0, 2024-04-23T10:41:25.833 INFO:teuthology.orchestra.run.smithi119.stdout: "sum": 0.000000000, 2024-04-23T10:41:25.833 INFO:teuthology.orchestra.run.smithi119.stdout: "avgtime": 0.000000000 2024-04-23T10:41:25.833 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-04-23T10:41:25.833 INFO:teuthology.orchestra.run.smithi119.stdout: "req_mknod_latency": { 2024-04-23T10:41:25.833 INFO:teuthology.orchestra.run.smithi119.stdout: "avgcount": 0, 2024-04-23T10:41:25.834 INFO:teuthology.orchestra.run.smithi119.stdout: "sum": 0.000000000, 2024-04-23T10:41:25.834 INFO:teuthology.orchestra.run.smithi119.stdout: "avgtime": 0.000000000 2024-04-23T10:41:25.834 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-04-23T10:41:25.834 INFO:teuthology.orchestra.run.smithi119.stdout: "req_mksnap_latency": { 2024-04-23T10:41:25.834 INFO:teuthology.orchestra.run.smithi119.stdout: "avgcount": 0, 2024-04-23T10:41:25.834 INFO:teuthology.orchestra.run.smithi119.stdout: "sum": 0.000000000, 2024-04-23T10:41:25.834 INFO:teuthology.orchestra.run.smithi119.stdout: "avgtime": 0.000000000 2024-04-23T10:41:25.835 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-04-23T10:41:25.835 INFO:teuthology.orchestra.run.smithi119.stdout: "req_open_latency": { 2024-04-23T10:41:25.835 INFO:teuthology.orchestra.run.smithi119.stdout: "avgcount": 0, 2024-04-23T10:41:25.835 INFO:teuthology.orchestra.run.smithi119.stdout: "sum": 0.000000000, 2024-04-23T10:41:25.835 INFO:teuthology.orchestra.run.smithi119.stdout: "avgtime": 0.000000000 2024-04-23T10:41:25.835 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-04-23T10:41:25.835 INFO:teuthology.orchestra.run.smithi119.stdout: "req_readdir_latency": { 2024-04-23T10:41:25.836 INFO:teuthology.orchestra.run.smithi119.stdout: "avgcount": 28965, 2024-04-23T10:41:25.836 INFO:teuthology.orchestra.run.smithi119.stdout: "sum": 48.927843664, 2024-04-23T10:41:25.836 INFO:teuthology.orchestra.run.smithi119.stdout: "avgtime": 0.001689205 2024-04-23T10:41:25.836 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-04-23T10:41:25.836 INFO:teuthology.orchestra.run.smithi119.stdout: "req_rename_latency": { 2024-04-23T10:41:25.836 INFO:teuthology.orchestra.run.smithi119.stdout: "avgcount": 2209, 2024-04-23T10:41:25.836 INFO:teuthology.orchestra.run.smithi119.stdout: "sum": 9.632739961, 2024-04-23T10:41:25.837 INFO:teuthology.orchestra.run.smithi119.stdout: "avgtime": 0.004360679 2024-04-23T10:41:25.837 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-04-23T10:41:25.837 INFO:teuthology.orchestra.run.smithi119.stdout: "req_renamesnap_latency": { 2024-04-23T10:41:25.837 INFO:teuthology.orchestra.run.smithi119.stdout: "avgcount": 0, 2024-04-23T10:41:25.837 INFO:teuthology.orchestra.run.smithi119.stdout: "sum": 0.000000000, 2024-04-23T10:41:25.837 INFO:teuthology.orchestra.run.smithi119.stdout: "avgtime": 0.000000000 2024-04-23T10:41:25.837 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-04-23T10:41:25.837 INFO:teuthology.orchestra.run.smithi119.stdout: "req_snapdiff_latency": { 2024-04-23T10:41:25.838 INFO:teuthology.orchestra.run.smithi119.stdout: "avgcount": 0, 2024-04-23T10:41:25.838 INFO:teuthology.orchestra.run.smithi119.stdout: "sum": 0.000000000, 2024-04-23T10:41:25.838 INFO:teuthology.orchestra.run.smithi119.stdout: "avgtime": 0.000000000 2024-04-23T10:41:25.838 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-04-23T10:41:25.838 INFO:teuthology.orchestra.run.smithi119.stdout: "req_rmdir_latency": { 2024-04-23T10:41:25.838 INFO:teuthology.orchestra.run.smithi119.stdout: "avgcount": 0, 2024-04-23T10:41:25.839 INFO:teuthology.orchestra.run.smithi119.stdout: "sum": 0.000000000, 2024-04-23T10:41:25.839 INFO:teuthology.orchestra.run.smithi119.stdout: "avgtime": 0.000000000 2024-04-23T10:41:25.839 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-04-23T10:41:25.839 INFO:teuthology.orchestra.run.smithi119.stdout: "req_rmsnap_latency": { 2024-04-23T10:41:25.839 INFO:teuthology.orchestra.run.smithi119.stdout: "avgcount": 0, 2024-04-23T10:41:25.839 INFO:teuthology.orchestra.run.smithi119.stdout: "sum": 0.000000000, 2024-04-23T10:41:25.839 INFO:teuthology.orchestra.run.smithi119.stdout: "avgtime": 0.000000000 2024-04-23T10:41:25.839 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-04-23T10:41:25.840 INFO:teuthology.orchestra.run.smithi119.stdout: "req_rmxattr_latency": { 2024-04-23T10:41:25.840 INFO:teuthology.orchestra.run.smithi119.stdout: "avgcount": 0, 2024-04-23T10:41:25.840 INFO:teuthology.orchestra.run.smithi119.stdout: "sum": 0.000000000, 2024-04-23T10:41:25.840 INFO:teuthology.orchestra.run.smithi119.stdout: "avgtime": 0.000000000 2024-04-23T10:41:25.840 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-04-23T10:41:25.840 INFO:teuthology.orchestra.run.smithi119.stdout: "req_setattr_latency": { 2024-04-23T10:41:25.840 INFO:teuthology.orchestra.run.smithi119.stdout: "avgcount": 9, 2024-04-23T10:41:25.841 INFO:teuthology.orchestra.run.smithi119.stdout: "sum": 22.156507600, 2024-04-23T10:41:25.841 INFO:teuthology.orchestra.run.smithi119.stdout: "avgtime": 2.461834177 2024-04-23T10:41:25.841 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-04-23T10:41:25.841 INFO:teuthology.orchestra.run.smithi119.stdout: "req_setdirlayout_latency": { 2024-04-23T10:41:25.841 INFO:teuthology.orchestra.run.smithi119.stdout: "avgcount": 0, 2024-04-23T10:41:25.841 INFO:teuthology.orchestra.run.smithi119.stdout: "sum": 0.000000000, 2024-04-23T10:41:25.841 INFO:teuthology.orchestra.run.smithi119.stdout: "avgtime": 0.000000000 2024-04-23T10:41:25.842 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-04-23T10:41:25.842 INFO:teuthology.orchestra.run.smithi119.stdout: "req_setfilelock_latency": { 2024-04-23T10:41:25.842 INFO:teuthology.orchestra.run.smithi119.stdout: "avgcount": 84, 2024-04-23T10:41:25.842 INFO:teuthology.orchestra.run.smithi119.stdout: "sum": 0.244827340, 2024-04-23T10:41:25.842 INFO:teuthology.orchestra.run.smithi119.stdout: "avgtime": 0.002914611 2024-04-23T10:41:25.842 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-04-23T10:41:25.842 INFO:teuthology.orchestra.run.smithi119.stdout: "req_setlayout_latency": { 2024-04-23T10:41:25.843 INFO:teuthology.orchestra.run.smithi119.stdout: "avgcount": 0, 2024-04-23T10:41:25.843 INFO:teuthology.orchestra.run.smithi119.stdout: "sum": 0.000000000, 2024-04-23T10:41:25.843 INFO:teuthology.orchestra.run.smithi119.stdout: "avgtime": 0.000000000 2024-04-23T10:41:25.843 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-04-23T10:41:25.843 INFO:teuthology.orchestra.run.smithi119.stdout: "req_setxattr_latency": { 2024-04-23T10:41:25.843 INFO:teuthology.orchestra.run.smithi119.stdout: "avgcount": 0, 2024-04-23T10:41:25.843 INFO:teuthology.orchestra.run.smithi119.stdout: "sum": 0.000000000, 2024-04-23T10:41:25.844 INFO:teuthology.orchestra.run.smithi119.stdout: "avgtime": 0.000000000 2024-04-23T10:41:25.844 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-04-23T10:41:25.844 INFO:teuthology.orchestra.run.smithi119.stdout: "req_symlink_latency": { 2024-04-23T10:41:25.844 INFO:teuthology.orchestra.run.smithi119.stdout: "avgcount": 0, 2024-04-23T10:41:25.844 INFO:teuthology.orchestra.run.smithi119.stdout: "sum": 0.000000000, 2024-04-23T10:41:25.844 INFO:teuthology.orchestra.run.smithi119.stdout: "avgtime": 0.000000000 2024-04-23T10:41:25.844 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-04-23T10:41:25.844 INFO:teuthology.orchestra.run.smithi119.stdout: "req_unlink_latency": { 2024-04-23T10:41:25.845 INFO:teuthology.orchestra.run.smithi119.stdout: "avgcount": 14213, 2024-04-23T10:41:25.845 INFO:teuthology.orchestra.run.smithi119.stdout: "sum": 33.176912474, 2024-04-23T10:41:25.845 INFO:teuthology.orchestra.run.smithi119.stdout: "avgtime": 0.002334265 2024-04-23T10:41:25.845 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-04-23T10:41:25.845 INFO:teuthology.orchestra.run.smithi119.stdout: "cap_revoke_eviction": 0, 2024-04-23T10:41:25.845 INFO:teuthology.orchestra.run.smithi119.stdout: "cap_acquisition_throttle": 0, 2024-04-23T10:41:25.845 INFO:teuthology.orchestra.run.smithi119.stdout: "req_getvxattr_latency": { 2024-04-23T10:41:25.846 INFO:teuthology.orchestra.run.smithi119.stdout: "avgcount": 0, 2024-04-23T10:41:25.846 INFO:teuthology.orchestra.run.smithi119.stdout: "sum": 0.000000000, 2024-04-23T10:41:25.846 INFO:teuthology.orchestra.run.smithi119.stdout: "avgtime": 0.000000000 2024-04-23T10:41:25.846 INFO:teuthology.orchestra.run.smithi119.stdout: } 2024-04-23T10:41:25.846 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-04-23T10:41:25.846 INFO:teuthology.orchestra.run.smithi119.stdout: "mds_sessions": { 2024-04-23T10:41:25.846 INFO:teuthology.orchestra.run.smithi119.stdout: "session_count": 1, 2024-04-23T10:41:25.847 INFO:teuthology.orchestra.run.smithi119.stdout: "session_add": 3, 2024-04-23T10:41:25.847 INFO:teuthology.orchestra.run.smithi119.stdout: "session_remove": 2, 2024-04-23T10:41:25.847 INFO:teuthology.orchestra.run.smithi119.stdout: "sessions_open": 1, 2024-04-23T10:41:25.847 INFO:teuthology.orchestra.run.smithi119.stdout: "sessions_stale": 0, 2024-04-23T10:41:25.847 INFO:teuthology.orchestra.run.smithi119.stdout: "total_load": 14273, 2024-04-23T10:41:25.847 INFO:teuthology.orchestra.run.smithi119.stdout: "average_load": 14273, 2024-04-23T10:41:25.847 INFO:teuthology.orchestra.run.smithi119.stdout: "avg_session_uptime": 875, 2024-04-23T10:41:25.847 INFO:teuthology.orchestra.run.smithi119.stdout: "mdthresh_evicted": 0 2024-04-23T10:41:25.848 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-04-23T10:41:25.848 INFO:teuthology.orchestra.run.smithi119.stdout: "mempool": { 2024-04-23T10:41:25.848 INFO:teuthology.orchestra.run.smithi119.stdout: "bloom_filter_bytes": 0, 2024-04-23T10:41:25.848 INFO:teuthology.orchestra.run.smithi119.stdout: "bloom_filter_items": 0, 2024-04-23T10:41:25.848 INFO:teuthology.orchestra.run.smithi119.stdout: "bluestore_alloc_bytes": 0, 2024-04-23T10:41:25.848 INFO:teuthology.orchestra.run.smithi119.stdout: "bluestore_alloc_items": 0, 2024-04-23T10:41:25.848 INFO:teuthology.orchestra.run.smithi119.stdout: "bluestore_cache_data_bytes": 0, 2024-04-23T10:41:25.849 INFO:teuthology.orchestra.run.smithi119.stdout: "bluestore_cache_data_items": 0, 2024-04-23T10:41:25.849 INFO:teuthology.orchestra.run.smithi119.stdout: "bluestore_cache_onode_bytes": 0, 2024-04-23T10:41:25.849 INFO:teuthology.orchestra.run.smithi119.stdout: "bluestore_cache_onode_items": 0, 2024-04-23T10:41:25.849 INFO:teuthology.orchestra.run.smithi119.stdout: "bluestore_cache_meta_bytes": 0, 2024-04-23T10:41:25.849 INFO:teuthology.orchestra.run.smithi119.stdout: "bluestore_cache_meta_items": 0, 2024-04-23T10:41:25.849 INFO:teuthology.orchestra.run.smithi119.stdout: "bluestore_cache_other_bytes": 0, 2024-04-23T10:41:25.849 INFO:teuthology.orchestra.run.smithi119.stdout: "bluestore_cache_other_items": 0, 2024-04-23T10:41:25.850 INFO:teuthology.orchestra.run.smithi119.stdout: "bluestore_cache_buffer_bytes": 0, 2024-04-23T10:41:25.850 INFO:teuthology.orchestra.run.smithi119.stdout: "bluestore_cache_buffer_items": 0, 2024-04-23T10:41:25.850 INFO:teuthology.orchestra.run.smithi119.stdout: "bluestore_extent_bytes": 0, 2024-04-23T10:41:25.850 INFO:teuthology.orchestra.run.smithi119.stdout: "bluestore_extent_items": 0, 2024-04-23T10:41:25.850 INFO:teuthology.orchestra.run.smithi119.stdout: "bluestore_blob_bytes": 0, 2024-04-23T10:41:25.850 INFO:teuthology.orchestra.run.smithi119.stdout: "bluestore_blob_items": 0, 2024-04-23T10:41:25.850 INFO:teuthology.orchestra.run.smithi119.stdout: "bluestore_shared_blob_bytes": 0, 2024-04-23T10:41:25.850 INFO:teuthology.orchestra.run.smithi119.stdout: "bluestore_shared_blob_items": 0, 2024-04-23T10:41:25.851 INFO:teuthology.orchestra.run.smithi119.stdout: "bluestore_inline_bl_bytes": 0, 2024-04-23T10:41:25.851 INFO:teuthology.orchestra.run.smithi119.stdout: "bluestore_inline_bl_items": 0, 2024-04-23T10:41:25.851 INFO:teuthology.orchestra.run.smithi119.stdout: "bluestore_fsck_bytes": 0, 2024-04-23T10:41:25.851 INFO:teuthology.orchestra.run.smithi119.stdout: "bluestore_fsck_items": 0, 2024-04-23T10:41:25.851 INFO:teuthology.orchestra.run.smithi119.stdout: "bluestore_txc_bytes": 0, 2024-04-23T10:41:25.851 INFO:teuthology.orchestra.run.smithi119.stdout: "bluestore_txc_items": 0, 2024-04-23T10:41:25.851 INFO:teuthology.orchestra.run.smithi119.stdout: "bluestore_writing_deferred_bytes": 0, 2024-04-23T10:41:25.852 INFO:teuthology.orchestra.run.smithi119.stdout: "bluestore_writing_deferred_items": 0, 2024-04-23T10:41:25.852 INFO:teuthology.orchestra.run.smithi119.stdout: "bluestore_writing_bytes": 0, 2024-04-23T10:41:25.852 INFO:teuthology.orchestra.run.smithi119.stdout: "bluestore_writing_items": 0, 2024-04-23T10:41:25.852 INFO:teuthology.orchestra.run.smithi119.stdout: "bluefs_bytes": 0, 2024-04-23T10:41:25.852 INFO:teuthology.orchestra.run.smithi119.stdout: "bluefs_items": 0, 2024-04-23T10:41:25.852 INFO:teuthology.orchestra.run.smithi119.stdout: "bluefs_file_reader_bytes": 0, 2024-04-23T10:41:25.852 INFO:teuthology.orchestra.run.smithi119.stdout: "bluefs_file_reader_items": 0, 2024-04-23T10:41:25.852 INFO:teuthology.orchestra.run.smithi119.stdout: "bluefs_file_writer_bytes": 0, 2024-04-23T10:41:25.853 INFO:teuthology.orchestra.run.smithi119.stdout: "bluefs_file_writer_items": 0, 2024-04-23T10:41:25.853 INFO:teuthology.orchestra.run.smithi119.stdout: "buffer_anon_bytes": 2289641, 2024-04-23T10:41:25.853 INFO:teuthology.orchestra.run.smithi119.stdout: "buffer_anon_items": 1472, 2024-04-23T10:41:25.853 INFO:teuthology.orchestra.run.smithi119.stdout: "buffer_meta_bytes": 0, 2024-04-23T10:41:25.853 INFO:teuthology.orchestra.run.smithi119.stdout: "buffer_meta_items": 0, 2024-04-23T10:41:25.853 INFO:teuthology.orchestra.run.smithi119.stdout: "osd_bytes": 0, 2024-04-23T10:41:25.853 INFO:teuthology.orchestra.run.smithi119.stdout: "osd_items": 0, 2024-04-23T10:41:25.854 INFO:teuthology.orchestra.run.smithi119.stdout: "osd_mapbl_bytes": 0, 2024-04-23T10:41:25.854 INFO:teuthology.orchestra.run.smithi119.stdout: "osd_mapbl_items": 0, 2024-04-23T10:41:25.854 INFO:teuthology.orchestra.run.smithi119.stdout: "osd_pglog_bytes": 0, 2024-04-23T10:41:25.854 INFO:teuthology.orchestra.run.smithi119.stdout: "osd_pglog_items": 0, 2024-04-23T10:41:25.854 INFO:teuthology.orchestra.run.smithi119.stdout: "osdmap_bytes": 8416, 2024-04-23T10:41:25.854 INFO:teuthology.orchestra.run.smithi119.stdout: "osdmap_items": 175, 2024-04-23T10:41:25.854 INFO:teuthology.orchestra.run.smithi119.stdout: "osdmap_mapping_bytes": 0, 2024-04-23T10:41:25.855 INFO:teuthology.orchestra.run.smithi119.stdout: "osdmap_mapping_items": 0, 2024-04-23T10:41:25.855 INFO:teuthology.orchestra.run.smithi119.stdout: "pgmap_bytes": 0, 2024-04-23T10:41:25.855 INFO:teuthology.orchestra.run.smithi119.stdout: "pgmap_items": 0, 2024-04-23T10:41:25.855 INFO:teuthology.orchestra.run.smithi119.stdout: "mds_co_bytes": 4694776, 2024-04-23T10:41:25.855 INFO:teuthology.orchestra.run.smithi119.stdout: "mds_co_items": 92545, 2024-04-23T10:41:25.855 INFO:teuthology.orchestra.run.smithi119.stdout: "unittest_1_bytes": 0, 2024-04-23T10:41:25.855 INFO:teuthology.orchestra.run.smithi119.stdout: "unittest_1_items": 0, 2024-04-23T10:41:25.856 INFO:teuthology.orchestra.run.smithi119.stdout: "unittest_2_bytes": 0, 2024-04-23T10:41:25.856 INFO:teuthology.orchestra.run.smithi119.stdout: "unittest_2_items": 0 2024-04-23T10:41:25.856 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-04-23T10:41:25.856 INFO:teuthology.orchestra.run.smithi119.stdout: "objecter": { 2024-04-23T10:41:25.856 INFO:teuthology.orchestra.run.smithi119.stdout: "op_active": 0, 2024-04-23T10:41:25.856 INFO:teuthology.orchestra.run.smithi119.stdout: "op_laggy": 0, 2024-04-23T10:41:25.856 INFO:teuthology.orchestra.run.smithi119.stdout: "op_send": 37606, 2024-04-23T10:41:25.857 INFO:teuthology.orchestra.run.smithi119.stdout: "op_send_bytes": 167075131, 2024-04-23T10:41:25.857 INFO:teuthology.orchestra.run.smithi119.stdout: "op_resend": 33, 2024-04-23T10:41:25.857 INFO:teuthology.orchestra.run.smithi119.stdout: "op_reply": 37606, 2024-04-23T10:41:25.857 INFO:teuthology.orchestra.run.smithi119.stdout: "op_latency": { 2024-04-23T10:41:25.857 INFO:teuthology.orchestra.run.smithi119.stdout: "avgcount": 37606, 2024-04-23T10:41:25.857 INFO:teuthology.orchestra.run.smithi119.stdout: "sum": 106.056423928, 2024-04-23T10:41:25.857 INFO:teuthology.orchestra.run.smithi119.stdout: "avgtime": 0.002820199 2024-04-23T10:41:25.858 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-04-23T10:41:25.858 INFO:teuthology.orchestra.run.smithi119.stdout: "op_inflight": 0, 2024-04-23T10:41:25.858 INFO:teuthology.orchestra.run.smithi119.stdout: "oplen_avg": { 2024-04-23T10:41:25.858 INFO:teuthology.orchestra.run.smithi119.stdout: "avgcount": 37606, 2024-04-23T10:41:25.858 INFO:teuthology.orchestra.run.smithi119.stdout: "sum": 38522 2024-04-23T10:41:25.858 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-04-23T10:41:25.858 INFO:teuthology.orchestra.run.smithi119.stdout: "op": 37606, 2024-04-23T10:41:25.859 INFO:teuthology.orchestra.run.smithi119.stdout: "op_r": 706, 2024-04-23T10:41:25.859 INFO:teuthology.orchestra.run.smithi119.stdout: "op_w": 36900, 2024-04-23T10:41:25.859 INFO:teuthology.orchestra.run.smithi119.stdout: "op_rmw": 0, 2024-04-23T10:41:25.859 INFO:teuthology.orchestra.run.smithi119.stdout: "op_pg": 0, 2024-04-23T10:41:25.859 INFO:teuthology.orchestra.run.smithi119.stdout: "osdop_stat": 0, 2024-04-23T10:41:25.859 INFO:teuthology.orchestra.run.smithi119.stdout: "osdop_create": 10, 2024-04-23T10:41:25.859 INFO:teuthology.orchestra.run.smithi119.stdout: "osdop_read": 702, 2024-04-23T10:41:25.859 INFO:teuthology.orchestra.run.smithi119.stdout: "osdop_write": 21903, 2024-04-23T10:41:25.860 INFO:teuthology.orchestra.run.smithi119.stdout: "osdop_writefull": 383, 2024-04-23T10:41:25.860 INFO:teuthology.orchestra.run.smithi119.stdout: "osdop_writesame": 0, 2024-04-23T10:41:25.860 INFO:teuthology.orchestra.run.smithi119.stdout: "osdop_append": 0, 2024-04-23T10:41:25.860 INFO:teuthology.orchestra.run.smithi119.stdout: "osdop_zero": 4, 2024-04-23T10:41:25.860 INFO:teuthology.orchestra.run.smithi119.stdout: "osdop_truncate": 3, 2024-04-23T10:41:25.860 INFO:teuthology.orchestra.run.smithi119.stdout: "osdop_delete": 14017, 2024-04-23T10:41:25.860 INFO:teuthology.orchestra.run.smithi119.stdout: "osdop_mapext": 0, 2024-04-23T10:41:25.861 INFO:teuthology.orchestra.run.smithi119.stdout: "osdop_sparse_read": 0, 2024-04-23T10:41:25.861 INFO:teuthology.orchestra.run.smithi119.stdout: "osdop_clonerange": 0, 2024-04-23T10:41:25.861 INFO:teuthology.orchestra.run.smithi119.stdout: "osdop_getxattr": 1, 2024-04-23T10:41:25.861 INFO:teuthology.orchestra.run.smithi119.stdout: "osdop_setxattr": 20, 2024-04-23T10:41:25.861 INFO:teuthology.orchestra.run.smithi119.stdout: "osdop_cmpxattr": 0, 2024-04-23T10:41:25.861 INFO:teuthology.orchestra.run.smithi119.stdout: "osdop_rmxattr": 0, 2024-04-23T10:41:25.861 INFO:teuthology.orchestra.run.smithi119.stdout: "osdop_resetxattrs": 0, 2024-04-23T10:41:25.861 INFO:teuthology.orchestra.run.smithi119.stdout: "osdop_call": 0, 2024-04-23T10:41:25.862 INFO:teuthology.orchestra.run.smithi119.stdout: "osdop_watch": 0, 2024-04-23T10:41:25.862 INFO:teuthology.orchestra.run.smithi119.stdout: "osdop_notify": 0, 2024-04-23T10:41:25.862 INFO:teuthology.orchestra.run.smithi119.stdout: "osdop_src_cmpxattr": 0, 2024-04-23T10:41:25.862 INFO:teuthology.orchestra.run.smithi119.stdout: "osdop_pgls": 0, 2024-04-23T10:41:25.862 INFO:teuthology.orchestra.run.smithi119.stdout: "osdop_pgls_filter": 0, 2024-04-23T10:41:25.862 INFO:teuthology.orchestra.run.smithi119.stdout: "osdop_other": 0, 2024-04-23T10:41:25.862 INFO:teuthology.orchestra.run.smithi119.stdout: "linger_active": 0, 2024-04-23T10:41:25.863 INFO:teuthology.orchestra.run.smithi119.stdout: "linger_send": 0, 2024-04-23T10:41:25.863 INFO:teuthology.orchestra.run.smithi119.stdout: "linger_resend": 0, 2024-04-23T10:41:25.863 INFO:teuthology.orchestra.run.smithi119.stdout: "linger_ping": 0, 2024-04-23T10:41:25.863 INFO:teuthology.orchestra.run.smithi119.stdout: "poolop_active": 0, 2024-04-23T10:41:25.863 INFO:teuthology.orchestra.run.smithi119.stdout: "poolop_send": 0, 2024-04-23T10:41:25.863 INFO:teuthology.orchestra.run.smithi119.stdout: "poolop_resend": 0, 2024-04-23T10:41:25.863 INFO:teuthology.orchestra.run.smithi119.stdout: "poolstat_active": 0, 2024-04-23T10:41:25.863 INFO:teuthology.orchestra.run.smithi119.stdout: "poolstat_send": 0, 2024-04-23T10:41:25.864 INFO:teuthology.orchestra.run.smithi119.stdout: "poolstat_resend": 0, 2024-04-23T10:41:25.864 INFO:teuthology.orchestra.run.smithi119.stdout: "statfs_active": 0, 2024-04-23T10:41:25.864 INFO:teuthology.orchestra.run.smithi119.stdout: "statfs_send": 0, 2024-04-23T10:41:25.864 INFO:teuthology.orchestra.run.smithi119.stdout: "statfs_resend": 0, 2024-04-23T10:41:25.864 INFO:teuthology.orchestra.run.smithi119.stdout: "command_active": 0, 2024-04-23T10:41:25.864 INFO:teuthology.orchestra.run.smithi119.stdout: "command_send": 0, 2024-04-23T10:41:25.864 INFO:teuthology.orchestra.run.smithi119.stdout: "command_resend": 0, 2024-04-23T10:41:25.864 INFO:teuthology.orchestra.run.smithi119.stdout: "map_epoch": 113, 2024-04-23T10:41:25.865 INFO:teuthology.orchestra.run.smithi119.stdout: "map_full": 0, 2024-04-23T10:41:25.865 INFO:teuthology.orchestra.run.smithi119.stdout: "map_inc": 26, 2024-04-23T10:41:25.865 INFO:teuthology.orchestra.run.smithi119.stdout: "osd_sessions": 12, 2024-04-23T10:41:25.865 INFO:teuthology.orchestra.run.smithi119.stdout: "osd_session_open": 12, 2024-04-23T10:41:25.865 INFO:teuthology.orchestra.run.smithi119.stdout: "osd_session_close": 0, 2024-04-23T10:41:25.865 INFO:teuthology.orchestra.run.smithi119.stdout: "osd_laggy": 0, 2024-04-23T10:41:25.865 INFO:teuthology.orchestra.run.smithi119.stdout: "omap_wr": 1087, 2024-04-23T10:41:25.865 INFO:teuthology.orchestra.run.smithi119.stdout: "omap_rd": 8, 2024-04-23T10:41:25.866 INFO:teuthology.orchestra.run.smithi119.stdout: "omap_del": 384 2024-04-23T10:41:25.866 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-04-23T10:41:25.866 INFO:teuthology.orchestra.run.smithi119.stdout: "oft": { 2024-04-23T10:41:25.866 INFO:teuthology.orchestra.run.smithi119.stdout: "omap_total_objs": 1, 2024-04-23T10:41:25.866 INFO:teuthology.orchestra.run.smithi119.stdout: "omap_total_kv_pairs": 0, 2024-04-23T10:41:25.866 INFO:teuthology.orchestra.run.smithi119.stdout: "omap_total_updates": 0, 2024-04-23T10:41:25.866 INFO:teuthology.orchestra.run.smithi119.stdout: "omap_total_removes": 0 2024-04-23T10:41:25.866 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-04-23T10:41:25.867 INFO:teuthology.orchestra.run.smithi119.stdout: "osd-slow-ops": { 2024-04-23T10:41:25.867 INFO:teuthology.orchestra.run.smithi119.stdout: "slow_ops_count": 182 2024-04-23T10:41:25.867 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-04-23T10:41:25.867 INFO:teuthology.orchestra.run.smithi119.stdout: "purge_queue": { 2024-04-23T10:41:25.867 INFO:teuthology.orchestra.run.smithi119.stdout: "pq_executing_ops": 0, 2024-04-23T10:41:25.867 INFO:teuthology.orchestra.run.smithi119.stdout: "pq_executing_ops_high_water": 10, 2024-04-23T10:41:25.867 INFO:teuthology.orchestra.run.smithi119.stdout: "pq_executing": 0, 2024-04-23T10:41:25.868 INFO:teuthology.orchestra.run.smithi119.stdout: "pq_executing_high_water": 10, 2024-04-23T10:41:25.868 INFO:teuthology.orchestra.run.smithi119.stdout: "pq_executed_ops": 13970, 2024-04-23T10:41:25.868 INFO:teuthology.orchestra.run.smithi119.stdout: "pq_executed": 13970, 2024-04-23T10:41:25.868 INFO:teuthology.orchestra.run.smithi119.stdout: "pq_item_in_journal": 0 2024-04-23T10:41:25.868 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-04-23T10:41:25.868 INFO:teuthology.orchestra.run.smithi119.stdout: "throttle-msgr_dispatch_throttler-mds": { 2024-04-23T10:41:25.868 INFO:teuthology.orchestra.run.smithi119.stdout: "val": 0, 2024-04-23T10:41:25.868 INFO:teuthology.orchestra.run.smithi119.stdout: "max": 104857600, 2024-04-23T10:41:25.869 INFO:teuthology.orchestra.run.smithi119.stdout: "get_started": 0, 2024-04-23T10:41:25.869 INFO:teuthology.orchestra.run.smithi119.stdout: "get": 356028, 2024-04-23T10:41:25.869 INFO:teuthology.orchestra.run.smithi119.stdout: "get_sum": 76031210, 2024-04-23T10:41:25.869 INFO:teuthology.orchestra.run.smithi119.stdout: "get_or_fail_fail": 0, 2024-04-23T10:41:25.869 INFO:teuthology.orchestra.run.smithi119.stdout: "get_or_fail_success": 356028, 2024-04-23T10:41:25.869 INFO:teuthology.orchestra.run.smithi119.stdout: "take": 0, 2024-04-23T10:41:25.869 INFO:teuthology.orchestra.run.smithi119.stdout: "take_sum": 0, 2024-04-23T10:41:25.869 INFO:teuthology.orchestra.run.smithi119.stdout: "put": 356028, 2024-04-23T10:41:25.870 INFO:teuthology.orchestra.run.smithi119.stdout: "put_sum": 76031210, 2024-04-23T10:41:25.870 INFO:teuthology.orchestra.run.smithi119.stdout: "wait": { 2024-04-23T10:41:25.870 INFO:teuthology.orchestra.run.smithi119.stdout: "avgcount": 0, 2024-04-23T10:41:25.870 INFO:teuthology.orchestra.run.smithi119.stdout: "sum": 0.000000000, 2024-04-23T10:41:25.870 INFO:teuthology.orchestra.run.smithi119.stdout: "avgtime": 0.000000000 2024-04-23T10:41:25.870 INFO:teuthology.orchestra.run.smithi119.stdout: } 2024-04-23T10:41:25.870 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-04-23T10:41:25.870 INFO:teuthology.orchestra.run.smithi119.stdout: "throttle-objecter_bytes": { 2024-04-23T10:41:25.871 INFO:teuthology.orchestra.run.smithi119.stdout: "val": 0, 2024-04-23T10:41:25.871 INFO:teuthology.orchestra.run.smithi119.stdout: "max": 104857600, 2024-04-23T10:41:25.871 INFO:teuthology.orchestra.run.smithi119.stdout: "get_started": 0, 2024-04-23T10:41:25.871 INFO:teuthology.orchestra.run.smithi119.stdout: "get": 0, 2024-04-23T10:41:25.871 INFO:teuthology.orchestra.run.smithi119.stdout: "get_sum": 0, 2024-04-23T10:41:25.871 INFO:teuthology.orchestra.run.smithi119.stdout: "get_or_fail_fail": 0, 2024-04-23T10:41:25.871 INFO:teuthology.orchestra.run.smithi119.stdout: "get_or_fail_success": 0, 2024-04-23T10:41:25.871 INFO:teuthology.orchestra.run.smithi119.stdout: "take": 37606, 2024-04-23T10:41:25.872 INFO:teuthology.orchestra.run.smithi119.stdout: "take_sum": 168488485, 2024-04-23T10:41:25.872 INFO:teuthology.orchestra.run.smithi119.stdout: "put": 37606, 2024-04-23T10:41:25.872 INFO:teuthology.orchestra.run.smithi119.stdout: "put_sum": 168488485, 2024-04-23T10:41:25.872 INFO:teuthology.orchestra.run.smithi119.stdout: "wait": { 2024-04-23T10:41:25.872 INFO:teuthology.orchestra.run.smithi119.stdout: "avgcount": 0, 2024-04-23T10:41:25.872 INFO:teuthology.orchestra.run.smithi119.stdout: "sum": 0.000000000, 2024-04-23T10:41:25.872 INFO:teuthology.orchestra.run.smithi119.stdout: "avgtime": 0.000000000 2024-04-23T10:41:25.872 INFO:teuthology.orchestra.run.smithi119.stdout: } 2024-04-23T10:41:25.873 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-04-23T10:41:25.873 INFO:teuthology.orchestra.run.smithi119.stdout: "throttle-objecter_ops": { 2024-04-23T10:41:25.873 INFO:teuthology.orchestra.run.smithi119.stdout: "val": 0, 2024-04-23T10:41:25.873 INFO:teuthology.orchestra.run.smithi119.stdout: "max": 1024, 2024-04-23T10:41:25.873 INFO:teuthology.orchestra.run.smithi119.stdout: "get_started": 0, 2024-04-23T10:41:25.873 INFO:teuthology.orchestra.run.smithi119.stdout: "get": 0, 2024-04-23T10:41:25.873 INFO:teuthology.orchestra.run.smithi119.stdout: "get_sum": 0, 2024-04-23T10:41:25.873 INFO:teuthology.orchestra.run.smithi119.stdout: "get_or_fail_fail": 0, 2024-04-23T10:41:25.874 INFO:teuthology.orchestra.run.smithi119.stdout: "get_or_fail_success": 0, 2024-04-23T10:41:25.874 INFO:teuthology.orchestra.run.smithi119.stdout: "take": 37606, 2024-04-23T10:41:25.874 INFO:teuthology.orchestra.run.smithi119.stdout: "take_sum": 37606, 2024-04-23T10:41:25.874 INFO:teuthology.orchestra.run.smithi119.stdout: "put": 37606, 2024-04-23T10:41:25.874 INFO:teuthology.orchestra.run.smithi119.stdout: "put_sum": 37606, 2024-04-23T10:41:25.874 INFO:teuthology.orchestra.run.smithi119.stdout: "wait": { 2024-04-23T10:41:25.874 INFO:teuthology.orchestra.run.smithi119.stdout: "avgcount": 0, 2024-04-23T10:41:25.875 INFO:teuthology.orchestra.run.smithi119.stdout: "sum": 0.000000000, 2024-04-23T10:41:25.875 INFO:teuthology.orchestra.run.smithi119.stdout: "avgtime": 0.000000000 2024-04-23T10:41:25.875 INFO:teuthology.orchestra.run.smithi119.stdout: } 2024-04-23T10:41:25.875 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-04-23T10:41:25.875 INFO:teuthology.orchestra.run.smithi119.stdout: "throttle-write_buf_throttle": { 2024-04-23T10:41:25.875 INFO:teuthology.orchestra.run.smithi119.stdout: "val": 0, 2024-04-23T10:41:25.875 INFO:teuthology.orchestra.run.smithi119.stdout: "max": 3758096384, 2024-04-23T10:41:25.875 INFO:teuthology.orchestra.run.smithi119.stdout: "get_started": 0, 2024-04-23T10:41:25.876 INFO:teuthology.orchestra.run.smithi119.stdout: "get": 13970, 2024-04-23T10:41:25.876 INFO:teuthology.orchestra.run.smithi119.stdout: "get_sum": 1413418, 2024-04-23T10:41:25.876 INFO:teuthology.orchestra.run.smithi119.stdout: "get_or_fail_fail": 0, 2024-04-23T10:41:25.876 INFO:teuthology.orchestra.run.smithi119.stdout: "get_or_fail_success": 13970, 2024-04-23T10:41:25.876 INFO:teuthology.orchestra.run.smithi119.stdout: "take": 0, 2024-04-23T10:41:25.876 INFO:teuthology.orchestra.run.smithi119.stdout: "take_sum": 0, 2024-04-23T10:41:25.876 INFO:teuthology.orchestra.run.smithi119.stdout: "put": 702, 2024-04-23T10:41:25.876 INFO:teuthology.orchestra.run.smithi119.stdout: "put_sum": 1413418, 2024-04-23T10:41:25.876 INFO:teuthology.orchestra.run.smithi119.stdout: "wait": { 2024-04-23T10:41:25.877 INFO:teuthology.orchestra.run.smithi119.stdout: "avgcount": 0, 2024-04-23T10:41:25.877 INFO:teuthology.orchestra.run.smithi119.stdout: "sum": 0.000000000, 2024-04-23T10:41:25.877 INFO:teuthology.orchestra.run.smithi119.stdout: "avgtime": 0.000000000 2024-04-23T10:41:25.877 INFO:teuthology.orchestra.run.smithi119.stdout: } 2024-04-23T10:41:25.877 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-04-23T10:41:25.877 INFO:teuthology.orchestra.run.smithi119.stdout: "throttle-write_buf_throttle-0x55c68b952500": { 2024-04-23T10:41:25.877 INFO:teuthology.orchestra.run.smithi119.stdout: "val": 1220335, 2024-04-23T10:41:25.877 INFO:teuthology.orchestra.run.smithi119.stdout: "max": 3758096384, 2024-04-23T10:41:25.878 INFO:teuthology.orchestra.run.smithi119.stdout: "get_started": 0, 2024-04-23T10:41:25.878 INFO:teuthology.orchestra.run.smithi119.stdout: "get": 76412, 2024-04-23T10:41:25.878 INFO:teuthology.orchestra.run.smithi119.stdout: "get_sum": 165287365, 2024-04-23T10:41:25.878 INFO:teuthology.orchestra.run.smithi119.stdout: "get_or_fail_fail": 0, 2024-04-23T10:41:25.878 INFO:teuthology.orchestra.run.smithi119.stdout: "get_or_fail_success": 76412, 2024-04-23T10:41:25.878 INFO:teuthology.orchestra.run.smithi119.stdout: "take": 0, 2024-04-23T10:41:25.878 INFO:teuthology.orchestra.run.smithi119.stdout: "take_sum": 0, 2024-04-23T10:41:25.879 INFO:teuthology.orchestra.run.smithi119.stdout: "put": 21201, 2024-04-23T10:41:25.879 INFO:teuthology.orchestra.run.smithi119.stdout: "put_sum": 164067030, 2024-04-23T10:41:25.879 INFO:teuthology.orchestra.run.smithi119.stdout: "wait": { 2024-04-23T10:41:25.879 INFO:teuthology.orchestra.run.smithi119.stdout: "avgcount": 0, 2024-04-23T10:41:25.879 INFO:teuthology.orchestra.run.smithi119.stdout: "sum": 0.000000000, 2024-04-23T10:41:25.879 INFO:teuthology.orchestra.run.smithi119.stdout: "avgtime": 0.000000000 2024-04-23T10:41:25.879 INFO:teuthology.orchestra.run.smithi119.stdout: } 2024-04-23T10:41:25.879 INFO:teuthology.orchestra.run.smithi119.stdout: } 2024-04-23T10:41:25.880 INFO:teuthology.orchestra.run.smithi119.stdout:} 2024-04-23T10:41:26.565 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 23 10:41:26 smithi119 ceph-mon[32608]: pgmap v468: 97 pgs: 97 active+clean; 559 MiB data, 4.3 GiB used, 1.0 TiB / 1.0 TiB avail; 550 KiB/s rd, 4.8 MiB/s wr, 171 op/s 2024-04-23T10:41:26.578 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 23 10:41:26 smithi204 ceph-mon[32649]: pgmap v468: 97 pgs: 97 active+clean; 559 MiB data, 4.3 GiB used, 1.0 TiB / 1.0 TiB avail; 550 KiB/s rd, 4.8 MiB/s wr, 171 op/s 2024-04-23T10:41:26.754 INFO:journalctl@ceph.mon.a.smithi114.stdout:Apr 23 10:41:26 smithi114 ceph-mon[29255]: pgmap v468: 97 pgs: 97 active+clean; 559 MiB data, 4.3 GiB used, 1.0 TiB / 1.0 TiB avail; 550 KiB/s rd, 4.8 MiB/s wr, 171 op/s 2024-04-23T10:41:26.804 INFO:tasks.check_counter:Daemon mds.e mds.exported=9 2024-04-23T10:41:26.804 INFO:tasks.check_counter:Daemon mds.e mds.imported=9 2024-04-23T10:41:26.804 INFO:tasks.check_counter:Daemon mds.e mds_cache.dir_update=350 2024-04-23T10:41:26.804 INFO:tasks.check_counter:Daemon mds.e mds_cache.dir_update_receipt=1494 2024-04-23T10:41:26.804 INFO:tasks.check_counter:Daemon mds.e mds.root_rsnaps=6 2024-04-23T10:41:26.805 INFO:tasks.check_counter:Daemon mds.e mds_server.req_mksnap_latency.avgcount=0 2024-04-23T10:41:26.805 INFO:tasks.check_counter:Daemon mds.e mds_server.req_rmsnap_latency.avgcount=0 2024-04-23T10:41:26.805 DEBUG:tasks.check_counter:Getting stats from h 2024-04-23T10:41:26.805 DEBUG:teuthology.orchestra.run.smithi119:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d69f13d827952bb07bb9b7973bca14f97d764d6f shell --fsid aa240572-0159-11ef-bc93-c7b262605968 -- ceph daemon mds.h perf dump 2024-04-23T10:41:27.047 INFO:teuthology.orchestra.run.smithi119.stderr:Inferring config /var/lib/ceph/aa240572-0159-11ef-bc93-c7b262605968/mon.b/config 2024-04-23T10:41:28.427 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 23 10:41:28 smithi119 ceph-mon[32608]: pgmap v469: 97 pgs: 97 active+clean; 560 MiB data, 4.3 GiB used, 1.0 TiB / 1.0 TiB avail; 719 KiB/s rd, 6.8 MiB/s wr, 238 op/s 2024-04-23T10:41:28.578 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 23 10:41:28 smithi204 ceph-mon[32649]: pgmap v469: 97 pgs: 97 active+clean; 560 MiB data, 4.3 GiB used, 1.0 TiB / 1.0 TiB avail; 719 KiB/s rd, 6.8 MiB/s wr, 238 op/s 2024-04-23T10:41:28.716 INFO:teuthology.orchestra.run.smithi119.stdout:{ 2024-04-23T10:41:28.716 INFO:teuthology.orchestra.run.smithi119.stdout: "AsyncMessenger::Worker-0": { 2024-04-23T10:41:28.716 INFO:teuthology.orchestra.run.smithi119.stdout: "msgr_recv_messages": 172334, 2024-04-23T10:41:28.717 INFO:teuthology.orchestra.run.smithi119.stdout: "msgr_send_messages": 122777, 2024-04-23T10:41:28.717 INFO:teuthology.orchestra.run.smithi119.stdout: "msgr_recv_bytes": 51649619, 2024-04-23T10:41:28.717 INFO:teuthology.orchestra.run.smithi119.stdout: "msgr_send_bytes": 63373285, 2024-04-23T10:41:28.717 INFO:teuthology.orchestra.run.smithi119.stdout: "msgr_created_connections": 10, 2024-04-23T10:41:28.717 INFO:teuthology.orchestra.run.smithi119.stdout: "msgr_active_connections": 4, 2024-04-23T10:41:28.717 INFO:teuthology.orchestra.run.smithi119.stdout: "msgr_running_total_time": 9.213584253, 2024-04-23T10:41:28.718 INFO:teuthology.orchestra.run.smithi119.stdout: "msgr_running_send_time": 4.261411339, 2024-04-23T10:41:28.718 INFO:teuthology.orchestra.run.smithi119.stdout: "msgr_running_recv_time": 6.352449152, 2024-04-23T10:41:28.718 INFO:teuthology.orchestra.run.smithi119.stdout: "msgr_running_fast_dispatch_time": 0.182259315, 2024-04-23T10:41:28.718 INFO:teuthology.orchestra.run.smithi119.stdout: "msgr_send_messages_queue_lat": { 2024-04-23T10:41:28.718 INFO:teuthology.orchestra.run.smithi119.stdout: "avgcount": 122777, 2024-04-23T10:41:28.719 INFO:teuthology.orchestra.run.smithi119.stdout: "sum": 3.128790854, 2024-04-23T10:41:28.719 INFO:teuthology.orchestra.run.smithi119.stdout: "avgtime": 0.000025483 2024-04-23T10:41:28.719 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-04-23T10:41:28.719 INFO:teuthology.orchestra.run.smithi119.stdout: "msgr_handle_ack_lat": { 2024-04-23T10:41:28.719 INFO:teuthology.orchestra.run.smithi119.stdout: "avgcount": 278803, 2024-04-23T10:41:28.719 INFO:teuthology.orchestra.run.smithi119.stdout: "sum": 0.043526087, 2024-04-23T10:41:28.720 INFO:teuthology.orchestra.run.smithi119.stdout: "avgtime": 0.000000156 2024-04-23T10:41:28.720 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-04-23T10:41:28.720 INFO:teuthology.orchestra.run.smithi119.stdout: "msgr_recv_encrypted_bytes": 0, 2024-04-23T10:41:28.720 INFO:teuthology.orchestra.run.smithi119.stdout: "msgr_send_encrypted_bytes": 0 2024-04-23T10:41:28.720 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-04-23T10:41:28.721 INFO:teuthology.orchestra.run.smithi119.stdout: "AsyncMessenger::Worker-1": { 2024-04-23T10:41:28.721 INFO:teuthology.orchestra.run.smithi119.stdout: "msgr_recv_messages": 111475, 2024-04-23T10:41:28.721 INFO:teuthology.orchestra.run.smithi119.stdout: "msgr_send_messages": 141911, 2024-04-23T10:41:28.721 INFO:teuthology.orchestra.run.smithi119.stdout: "msgr_recv_bytes": 33233988, 2024-04-23T10:41:28.721 INFO:teuthology.orchestra.run.smithi119.stdout: "msgr_send_bytes": 178629846, 2024-04-23T10:41:28.721 INFO:teuthology.orchestra.run.smithi119.stdout: "msgr_created_connections": 17, 2024-04-23T10:41:28.722 INFO:teuthology.orchestra.run.smithi119.stdout: "msgr_active_connections": 5, 2024-04-23T10:41:28.722 INFO:teuthology.orchestra.run.smithi119.stdout: "msgr_running_total_time": 5.701308373, 2024-04-23T10:41:28.722 INFO:teuthology.orchestra.run.smithi119.stdout: "msgr_running_send_time": 2.485889221, 2024-04-23T10:41:28.722 INFO:teuthology.orchestra.run.smithi119.stdout: "msgr_running_recv_time": 3.747739323, 2024-04-23T10:41:28.722 INFO:teuthology.orchestra.run.smithi119.stdout: "msgr_running_fast_dispatch_time": 0.230322393, 2024-04-23T10:41:28.722 INFO:teuthology.orchestra.run.smithi119.stdout: "msgr_send_messages_queue_lat": { 2024-04-23T10:41:28.723 INFO:teuthology.orchestra.run.smithi119.stdout: "avgcount": 141909, 2024-04-23T10:41:28.723 INFO:teuthology.orchestra.run.smithi119.stdout: "sum": 4.760218561, 2024-04-23T10:41:28.723 INFO:teuthology.orchestra.run.smithi119.stdout: "avgtime": 0.000033544 2024-04-23T10:41:28.723 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-04-23T10:41:28.723 INFO:teuthology.orchestra.run.smithi119.stdout: "msgr_handle_ack_lat": { 2024-04-23T10:41:28.724 INFO:teuthology.orchestra.run.smithi119.stdout: "avgcount": 229908, 2024-04-23T10:41:28.724 INFO:teuthology.orchestra.run.smithi119.stdout: "sum": 0.033034794, 2024-04-23T10:41:28.724 INFO:teuthology.orchestra.run.smithi119.stdout: "avgtime": 0.000000143 2024-04-23T10:41:28.724 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-04-23T10:41:28.724 INFO:teuthology.orchestra.run.smithi119.stdout: "msgr_recv_encrypted_bytes": 6896, 2024-04-23T10:41:28.724 INFO:teuthology.orchestra.run.smithi119.stdout: "msgr_send_encrypted_bytes": 539952 2024-04-23T10:41:28.725 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-04-23T10:41:28.725 INFO:teuthology.orchestra.run.smithi119.stdout: "AsyncMessenger::Worker-2": { 2024-04-23T10:41:28.725 INFO:teuthology.orchestra.run.smithi119.stdout: "msgr_recv_messages": 6487, 2024-04-23T10:41:28.725 INFO:teuthology.orchestra.run.smithi119.stdout: "msgr_send_messages": 6501, 2024-04-23T10:41:28.725 INFO:teuthology.orchestra.run.smithi119.stdout: "msgr_recv_bytes": 3953687, 2024-04-23T10:41:28.725 INFO:teuthology.orchestra.run.smithi119.stdout: "msgr_send_bytes": 47949110, 2024-04-23T10:41:28.726 INFO:teuthology.orchestra.run.smithi119.stdout: "msgr_created_connections": 12, 2024-04-23T10:41:28.726 INFO:teuthology.orchestra.run.smithi119.stdout: "msgr_active_connections": 7, 2024-04-23T10:41:28.726 INFO:teuthology.orchestra.run.smithi119.stdout: "msgr_running_total_time": 0.630720991, 2024-04-23T10:41:28.726 INFO:teuthology.orchestra.run.smithi119.stdout: "msgr_running_send_time": 0.234672073, 2024-04-23T10:41:28.726 INFO:teuthology.orchestra.run.smithi119.stdout: "msgr_running_recv_time": 0.144855107, 2024-04-23T10:41:28.726 INFO:teuthology.orchestra.run.smithi119.stdout: "msgr_running_fast_dispatch_time": 0.227338439, 2024-04-23T10:41:28.727 INFO:teuthology.orchestra.run.smithi119.stdout: "msgr_send_messages_queue_lat": { 2024-04-23T10:41:28.727 INFO:teuthology.orchestra.run.smithi119.stdout: "avgcount": 6498, 2024-04-23T10:41:28.727 INFO:teuthology.orchestra.run.smithi119.stdout: "sum": 0.365898982, 2024-04-23T10:41:28.727 INFO:teuthology.orchestra.run.smithi119.stdout: "avgtime": 0.000056309 2024-04-23T10:41:28.727 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-04-23T10:41:28.728 INFO:teuthology.orchestra.run.smithi119.stdout: "msgr_handle_ack_lat": { 2024-04-23T10:41:28.728 INFO:teuthology.orchestra.run.smithi119.stdout: "avgcount": 88, 2024-04-23T10:41:28.728 INFO:teuthology.orchestra.run.smithi119.stdout: "sum": 0.000017149, 2024-04-23T10:41:28.728 INFO:teuthology.orchestra.run.smithi119.stdout: "avgtime": 0.000000194 2024-04-23T10:41:28.728 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-04-23T10:41:28.728 INFO:teuthology.orchestra.run.smithi119.stdout: "msgr_recv_encrypted_bytes": 1776864, 2024-04-23T10:41:28.728 INFO:teuthology.orchestra.run.smithi119.stdout: "msgr_send_encrypted_bytes": 197488 2024-04-23T10:41:28.729 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-04-23T10:41:28.729 INFO:teuthology.orchestra.run.smithi119.stdout: "cct": { 2024-04-23T10:41:28.729 INFO:teuthology.orchestra.run.smithi119.stdout: "total_workers": 1, 2024-04-23T10:41:28.729 INFO:teuthology.orchestra.run.smithi119.stdout: "unhealthy_workers": 0 2024-04-23T10:41:28.729 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-04-23T10:41:28.730 INFO:teuthology.orchestra.run.smithi119.stdout: "finisher-MDSRank": { 2024-04-23T10:41:28.730 INFO:teuthology.orchestra.run.smithi119.stdout: "queue_len": 0, 2024-04-23T10:41:28.730 INFO:teuthology.orchestra.run.smithi119.stdout: "complete_latency": { 2024-04-23T10:41:28.730 INFO:teuthology.orchestra.run.smithi119.stdout: "avgcount": 20044, 2024-04-23T10:41:28.730 INFO:teuthology.orchestra.run.smithi119.stdout: "sum": 23.289732886, 2024-04-23T10:41:28.731 INFO:teuthology.orchestra.run.smithi119.stdout: "avgtime": 0.001161930 2024-04-23T10:41:28.731 INFO:teuthology.orchestra.run.smithi119.stdout: } 2024-04-23T10:41:28.731 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-04-23T10:41:28.731 INFO:teuthology.orchestra.run.smithi119.stdout: "finisher-PurgeQueue": { 2024-04-23T10:41:28.731 INFO:teuthology.orchestra.run.smithi119.stdout: "queue_len": 0, 2024-04-23T10:41:28.732 INFO:teuthology.orchestra.run.smithi119.stdout: "complete_latency": { 2024-04-23T10:41:28.732 INFO:teuthology.orchestra.run.smithi119.stdout: "avgcount": 10502, 2024-04-23T10:41:28.732 INFO:teuthology.orchestra.run.smithi119.stdout: "sum": 1.731653629, 2024-04-23T10:41:28.732 INFO:teuthology.orchestra.run.smithi119.stdout: "avgtime": 0.000164887 2024-04-23T10:41:28.732 INFO:teuthology.orchestra.run.smithi119.stdout: } 2024-04-23T10:41:28.732 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-04-23T10:41:28.733 INFO:teuthology.orchestra.run.smithi119.stdout: "mds": { 2024-04-23T10:41:28.733 INFO:teuthology.orchestra.run.smithi119.stdout: "request": 79897, 2024-04-23T10:41:28.733 INFO:teuthology.orchestra.run.smithi119.stdout: "reply": 66521, 2024-04-23T10:41:28.733 INFO:teuthology.orchestra.run.smithi119.stdout: "reply_latency": { 2024-04-23T10:41:28.733 INFO:teuthology.orchestra.run.smithi119.stdout: "avgcount": 66521, 2024-04-23T10:41:28.733 INFO:teuthology.orchestra.run.smithi119.stdout: "sum": 208.842160119, 2024-04-23T10:41:28.733 INFO:teuthology.orchestra.run.smithi119.stdout: "avgtime": 0.003139492 2024-04-23T10:41:28.734 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-04-23T10:41:28.734 INFO:teuthology.orchestra.run.smithi119.stdout: "slow_reply": 0, 2024-04-23T10:41:28.734 INFO:teuthology.orchestra.run.smithi119.stdout: "forward": 13375, 2024-04-23T10:41:28.734 INFO:teuthology.orchestra.run.smithi119.stdout: "dir_fetch_complete": 0, 2024-04-23T10:41:28.734 INFO:teuthology.orchestra.run.smithi119.stdout: "dir_fetch_keys": 0, 2024-04-23T10:41:28.734 INFO:teuthology.orchestra.run.smithi119.stdout: "dir_commit": 44, 2024-04-23T10:41:28.735 INFO:teuthology.orchestra.run.smithi119.stdout: "dir_split": 4, 2024-04-23T10:41:28.735 INFO:teuthology.orchestra.run.smithi119.stdout: "dir_merge": 1, 2024-04-23T10:41:28.735 INFO:teuthology.orchestra.run.smithi119.stdout: "inodes": 1665, 2024-04-23T10:41:28.735 INFO:teuthology.orchestra.run.smithi119.stdout: "inodes_top": 193, 2024-04-23T10:41:28.735 INFO:teuthology.orchestra.run.smithi119.stdout: "inodes_bottom": 1460, 2024-04-23T10:41:28.735 INFO:teuthology.orchestra.run.smithi119.stdout: "inodes_pin_tail": 12, 2024-04-23T10:41:28.736 INFO:teuthology.orchestra.run.smithi119.stdout: "inodes_pinned": 1390, 2024-04-23T10:41:28.736 INFO:teuthology.orchestra.run.smithi119.stdout: "inodes_expired": 414, 2024-04-23T10:41:28.736 INFO:teuthology.orchestra.run.smithi119.stdout: "inodes_with_caps": 328, 2024-04-23T10:41:28.736 INFO:teuthology.orchestra.run.smithi119.stdout: "caps": 328, 2024-04-23T10:41:28.736 INFO:teuthology.orchestra.run.smithi119.stdout: "subtrees": 11, 2024-04-23T10:41:28.736 INFO:teuthology.orchestra.run.smithi119.stdout: "traverse": 119635, 2024-04-23T10:41:28.737 INFO:teuthology.orchestra.run.smithi119.stdout: "traverse_hit": 60728, 2024-04-23T10:41:28.737 INFO:teuthology.orchestra.run.smithi119.stdout: "traverse_forward": 13363, 2024-04-23T10:41:28.737 INFO:teuthology.orchestra.run.smithi119.stdout: "traverse_discover": 217, 2024-04-23T10:41:28.737 INFO:teuthology.orchestra.run.smithi119.stdout: "traverse_dir_fetch": 0, 2024-04-23T10:41:28.737 INFO:teuthology.orchestra.run.smithi119.stdout: "traverse_remote_ino": 0, 2024-04-23T10:41:28.737 INFO:teuthology.orchestra.run.smithi119.stdout: "traverse_lock": 404, 2024-04-23T10:41:28.737 INFO:teuthology.orchestra.run.smithi119.stdout: "load_cent": 23956, 2024-04-23T10:41:28.738 INFO:teuthology.orchestra.run.smithi119.stdout: "q": 0, 2024-04-23T10:41:28.738 INFO:teuthology.orchestra.run.smithi119.stdout: "exported": 10, 2024-04-23T10:41:28.738 INFO:teuthology.orchestra.run.smithi119.stdout: "exported_inodes": 308, 2024-04-23T10:41:28.738 INFO:teuthology.orchestra.run.smithi119.stdout: "imported": 6, 2024-04-23T10:41:28.738 INFO:teuthology.orchestra.run.smithi119.stdout: "imported_inodes": 620, 2024-04-23T10:41:28.738 INFO:teuthology.orchestra.run.smithi119.stdout: "openino_dir_fetch": 0, 2024-04-23T10:41:28.739 INFO:teuthology.orchestra.run.smithi119.stdout: "openino_backtrace_fetch": 0, 2024-04-23T10:41:28.739 INFO:teuthology.orchestra.run.smithi119.stdout: "openino_peer_discover": 0, 2024-04-23T10:41:28.739 INFO:teuthology.orchestra.run.smithi119.stdout: "root_rfiles": 733, 2024-04-23T10:41:28.739 INFO:teuthology.orchestra.run.smithi119.stdout: "root_rbytes": 212014623, 2024-04-23T10:41:28.739 INFO:teuthology.orchestra.run.smithi119.stdout: "root_rsnaps": 6, 2024-04-23T10:41:28.739 INFO:teuthology.orchestra.run.smithi119.stdout: "scrub_backtrace_fetch": 0, 2024-04-23T10:41:28.740 INFO:teuthology.orchestra.run.smithi119.stdout: "scrub_set_tag": 0, 2024-04-23T10:41:28.740 INFO:teuthology.orchestra.run.smithi119.stdout: "scrub_backtrace_repaired": 0, 2024-04-23T10:41:28.740 INFO:teuthology.orchestra.run.smithi119.stdout: "scrub_inotable_repaired": 0, 2024-04-23T10:41:28.740 INFO:teuthology.orchestra.run.smithi119.stdout: "scrub_dir_inodes": 0, 2024-04-23T10:41:28.740 INFO:teuthology.orchestra.run.smithi119.stdout: "scrub_dir_base_inodes": 0, 2024-04-23T10:41:28.740 INFO:teuthology.orchestra.run.smithi119.stdout: "scrub_dirfrag_rstats": 0, 2024-04-23T10:41:28.740 INFO:teuthology.orchestra.run.smithi119.stdout: "scrub_file_inodes": 0, 2024-04-23T10:41:28.741 INFO:teuthology.orchestra.run.smithi119.stdout: "handle_inode_file_caps": 0, 2024-04-23T10:41:28.741 INFO:teuthology.orchestra.run.smithi119.stdout: "ceph_cap_op_revoke": 3268, 2024-04-23T10:41:28.741 INFO:teuthology.orchestra.run.smithi119.stdout: "ceph_cap_op_grant": 28368, 2024-04-23T10:41:28.741 INFO:teuthology.orchestra.run.smithi119.stdout: "ceph_cap_op_trunc": 0, 2024-04-23T10:41:28.741 INFO:teuthology.orchestra.run.smithi119.stdout: "ceph_cap_op_flushsnap_ack": 0, 2024-04-23T10:41:28.741 INFO:teuthology.orchestra.run.smithi119.stdout: "ceph_cap_op_flush_ack": 0, 2024-04-23T10:41:28.741 INFO:teuthology.orchestra.run.smithi119.stdout: "handle_client_caps": 17862, 2024-04-23T10:41:28.742 INFO:teuthology.orchestra.run.smithi119.stdout: "handle_client_caps_dirty": 7693, 2024-04-23T10:41:28.742 INFO:teuthology.orchestra.run.smithi119.stdout: "handle_client_cap_release": 4310, 2024-04-23T10:41:28.742 INFO:teuthology.orchestra.run.smithi119.stdout: "process_request_cap_release": 57423 2024-04-23T10:41:28.742 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-04-23T10:41:28.742 INFO:teuthology.orchestra.run.smithi119.stdout: "mds_cache": { 2024-04-23T10:41:28.742 INFO:teuthology.orchestra.run.smithi119.stdout: "dir_update": 585, 2024-04-23T10:41:28.743 INFO:teuthology.orchestra.run.smithi119.stdout: "dir_update_receipt": 1093, 2024-04-23T10:41:28.743 INFO:teuthology.orchestra.run.smithi119.stdout: "dir_try_discover": 617, 2024-04-23T10:41:28.743 INFO:teuthology.orchestra.run.smithi119.stdout: "dir_send_discover": 1058, 2024-04-23T10:41:28.743 INFO:teuthology.orchestra.run.smithi119.stdout: "dir_handle_discover": 627, 2024-04-23T10:41:28.743 INFO:teuthology.orchestra.run.smithi119.stdout: "num_strays": 138, 2024-04-23T10:41:28.743 INFO:teuthology.orchestra.run.smithi119.stdout: "num_strays_delayed": 0, 2024-04-23T10:41:28.744 INFO:teuthology.orchestra.run.smithi119.stdout: "num_strays_enqueuing": 0, 2024-04-23T10:41:28.744 INFO:teuthology.orchestra.run.smithi119.stdout: "strays_created": 6422, 2024-04-23T10:41:28.744 INFO:teuthology.orchestra.run.smithi119.stdout: "strays_enqueued": 6284, 2024-04-23T10:41:28.744 INFO:teuthology.orchestra.run.smithi119.stdout: "strays_reintegrated": 0, 2024-04-23T10:41:28.744 INFO:teuthology.orchestra.run.smithi119.stdout: "strays_migrated": 0, 2024-04-23T10:41:28.744 INFO:teuthology.orchestra.run.smithi119.stdout: "num_recovering_processing": 0, 2024-04-23T10:41:28.744 INFO:teuthology.orchestra.run.smithi119.stdout: "num_recovering_enqueued": 0, 2024-04-23T10:41:28.745 INFO:teuthology.orchestra.run.smithi119.stdout: "num_recovering_prioritized": 0, 2024-04-23T10:41:28.745 INFO:teuthology.orchestra.run.smithi119.stdout: "recovery_started": 0, 2024-04-23T10:41:28.745 INFO:teuthology.orchestra.run.smithi119.stdout: "recovery_completed": 0, 2024-04-23T10:41:28.745 INFO:teuthology.orchestra.run.smithi119.stdout: "ireq_quiesce_path": 5, 2024-04-23T10:41:28.745 INFO:teuthology.orchestra.run.smithi119.stdout: "ireq_quiesce_inode": 139, 2024-04-23T10:41:28.745 INFO:teuthology.orchestra.run.smithi119.stdout: "ireq_enqueue_scrub": 0, 2024-04-23T10:41:28.745 INFO:teuthology.orchestra.run.smithi119.stdout: "ireq_exportdir": 18, 2024-04-23T10:41:28.746 INFO:teuthology.orchestra.run.smithi119.stdout: "ireq_flush": 0, 2024-04-23T10:41:28.746 INFO:teuthology.orchestra.run.smithi119.stdout: "ireq_fragmentdir": 5, 2024-04-23T10:41:28.746 INFO:teuthology.orchestra.run.smithi119.stdout: "ireq_fragstats": 0, 2024-04-23T10:41:28.746 INFO:teuthology.orchestra.run.smithi119.stdout: "ireq_inodestats": 0 2024-04-23T10:41:28.746 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-04-23T10:41:28.746 INFO:teuthology.orchestra.run.smithi119.stdout: "mds_log": { 2024-04-23T10:41:28.747 INFO:teuthology.orchestra.run.smithi119.stdout: "evlrg": 0, 2024-04-23T10:41:28.747 INFO:teuthology.orchestra.run.smithi119.stdout: "evadd": 62846, 2024-04-23T10:41:28.747 INFO:teuthology.orchestra.run.smithi119.stdout: "evex": 0, 2024-04-23T10:41:28.747 INFO:teuthology.orchestra.run.smithi119.stdout: "evtrm": 0, 2024-04-23T10:41:28.747 INFO:teuthology.orchestra.run.smithi119.stdout: "ev": 62846, 2024-04-23T10:41:28.747 INFO:teuthology.orchestra.run.smithi119.stdout: "evexg": 0, 2024-04-23T10:41:28.747 INFO:teuthology.orchestra.run.smithi119.stdout: "evexd": 0, 2024-04-23T10:41:28.748 INFO:teuthology.orchestra.run.smithi119.stdout: "segadd": 82, 2024-04-23T10:41:28.748 INFO:teuthology.orchestra.run.smithi119.stdout: "segex": 0, 2024-04-23T10:41:28.748 INFO:teuthology.orchestra.run.smithi119.stdout: "segtrm": 0, 2024-04-23T10:41:28.748 INFO:teuthology.orchestra.run.smithi119.stdout: "seg": 82, 2024-04-23T10:41:28.748 INFO:teuthology.orchestra.run.smithi119.stdout: "segmjr": 7, 2024-04-23T10:41:28.748 INFO:teuthology.orchestra.run.smithi119.stdout: "segexg": 0, 2024-04-23T10:41:28.749 INFO:teuthology.orchestra.run.smithi119.stdout: "segexd": 0, 2024-04-23T10:41:28.749 INFO:teuthology.orchestra.run.smithi119.stdout: "expos": 4194304, 2024-04-23T10:41:28.749 INFO:teuthology.orchestra.run.smithi119.stdout: "wrpos": 121395543, 2024-04-23T10:41:28.749 INFO:teuthology.orchestra.run.smithi119.stdout: "rdpos": 0, 2024-04-23T10:41:28.749 INFO:teuthology.orchestra.run.smithi119.stdout: "jlat": { 2024-04-23T10:41:28.749 INFO:teuthology.orchestra.run.smithi119.stdout: "avgcount": 10523, 2024-04-23T10:41:28.749 INFO:teuthology.orchestra.run.smithi119.stdout: "sum": 33.628639559, 2024-04-23T10:41:28.750 INFO:teuthology.orchestra.run.smithi119.stdout: "avgtime": 0.003195727 2024-04-23T10:41:28.750 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-04-23T10:41:28.750 INFO:teuthology.orchestra.run.smithi119.stdout: "replayed": 0 2024-04-23T10:41:28.750 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-04-23T10:41:28.750 INFO:teuthology.orchestra.run.smithi119.stdout: "mds_mem": { 2024-04-23T10:41:28.750 INFO:teuthology.orchestra.run.smithi119.stdout: "ino": 871, 2024-04-23T10:41:28.751 INFO:teuthology.orchestra.run.smithi119.stdout: "ino+": 7367, 2024-04-23T10:41:28.751 INFO:teuthology.orchestra.run.smithi119.stdout: "ino-": 6496, 2024-04-23T10:41:28.751 INFO:teuthology.orchestra.run.smithi119.stdout: "dir": 185, 2024-04-23T10:41:28.751 INFO:teuthology.orchestra.run.smithi119.stdout: "dir+": 629, 2024-04-23T10:41:28.751 INFO:teuthology.orchestra.run.smithi119.stdout: "dir-": 444, 2024-04-23T10:41:28.751 INFO:teuthology.orchestra.run.smithi119.stdout: "dn": 1525, 2024-04-23T10:41:28.751 INFO:teuthology.orchestra.run.smithi119.stdout: "dn+": 15119, 2024-04-23T10:41:28.752 INFO:teuthology.orchestra.run.smithi119.stdout: "dn-": 13594, 2024-04-23T10:41:28.752 INFO:teuthology.orchestra.run.smithi119.stdout: "cap": 328, 2024-04-23T10:41:28.752 INFO:teuthology.orchestra.run.smithi119.stdout: "cap+": 8091, 2024-04-23T10:41:28.752 INFO:teuthology.orchestra.run.smithi119.stdout: "cap-": 7763, 2024-04-23T10:41:28.752 INFO:teuthology.orchestra.run.smithi119.stdout: "rss": 77968, 2024-04-23T10:41:28.752 INFO:teuthology.orchestra.run.smithi119.stdout: "heap": 223608 2024-04-23T10:41:28.752 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-04-23T10:41:28.753 INFO:teuthology.orchestra.run.smithi119.stdout: "mds_server": { 2024-04-23T10:41:28.753 INFO:teuthology.orchestra.run.smithi119.stdout: "dispatch_client_request": 107734, 2024-04-23T10:41:28.753 INFO:teuthology.orchestra.run.smithi119.stdout: "dispatch_server_request": 65, 2024-04-23T10:41:28.753 INFO:teuthology.orchestra.run.smithi119.stdout: "handle_client_request": 79897, 2024-04-23T10:41:28.753 INFO:teuthology.orchestra.run.smithi119.stdout: "handle_client_session": 960, 2024-04-23T10:41:28.753 INFO:teuthology.orchestra.run.smithi119.stdout: "handle_peer_request": 7376, 2024-04-23T10:41:28.754 INFO:teuthology.orchestra.run.smithi119.stdout: "req_create_latency": { 2024-04-23T10:41:28.754 INFO:teuthology.orchestra.run.smithi119.stdout: "avgcount": 6390, 2024-04-23T10:41:28.754 INFO:teuthology.orchestra.run.smithi119.stdout: "sum": 28.652967298, 2024-04-23T10:41:28.754 INFO:teuthology.orchestra.run.smithi119.stdout: "avgtime": 0.004484032 2024-04-23T10:41:28.754 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-04-23T10:41:28.754 INFO:teuthology.orchestra.run.smithi119.stdout: "req_getattr_latency": { 2024-04-23T10:41:28.754 INFO:teuthology.orchestra.run.smithi119.stdout: "avgcount": 14, 2024-04-23T10:41:28.755 INFO:teuthology.orchestra.run.smithi119.stdout: "sum": 0.074474692, 2024-04-23T10:41:28.755 INFO:teuthology.orchestra.run.smithi119.stdout: "avgtime": 0.005319620 2024-04-23T10:41:28.755 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-04-23T10:41:28.755 INFO:teuthology.orchestra.run.smithi119.stdout: "req_getfilelock_latency": { 2024-04-23T10:41:28.755 INFO:teuthology.orchestra.run.smithi119.stdout: "avgcount": 0, 2024-04-23T10:41:28.755 INFO:teuthology.orchestra.run.smithi119.stdout: "sum": 0.000000000, 2024-04-23T10:41:28.755 INFO:teuthology.orchestra.run.smithi119.stdout: "avgtime": 0.000000000 2024-04-23T10:41:28.756 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-04-23T10:41:28.756 INFO:teuthology.orchestra.run.smithi119.stdout: "req_link_latency": { 2024-04-23T10:41:28.756 INFO:teuthology.orchestra.run.smithi119.stdout: "avgcount": 0, 2024-04-23T10:41:28.756 INFO:teuthology.orchestra.run.smithi119.stdout: "sum": 0.000000000, 2024-04-23T10:41:28.756 INFO:teuthology.orchestra.run.smithi119.stdout: "avgtime": 0.000000000 2024-04-23T10:41:28.756 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-04-23T10:41:28.757 INFO:teuthology.orchestra.run.smithi119.stdout: "req_lookup_latency": { 2024-04-23T10:41:28.757 INFO:teuthology.orchestra.run.smithi119.stdout: "avgcount": 13406, 2024-04-23T10:41:28.757 INFO:teuthology.orchestra.run.smithi119.stdout: "sum": 31.661085013, 2024-04-23T10:41:28.757 INFO:teuthology.orchestra.run.smithi119.stdout: "avgtime": 0.002361710 2024-04-23T10:41:28.757 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-04-23T10:41:28.757 INFO:teuthology.orchestra.run.smithi119.stdout: "req_lookuphash_latency": { 2024-04-23T10:41:28.758 INFO:teuthology.orchestra.run.smithi119.stdout: "avgcount": 0, 2024-04-23T10:41:28.758 INFO:teuthology.orchestra.run.smithi119.stdout: "sum": 0.000000000, 2024-04-23T10:41:28.758 INFO:teuthology.orchestra.run.smithi119.stdout: "avgtime": 0.000000000 2024-04-23T10:41:28.758 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-04-23T10:41:28.758 INFO:teuthology.orchestra.run.smithi119.stdout: "req_lookupino_latency": { 2024-04-23T10:41:28.758 INFO:teuthology.orchestra.run.smithi119.stdout: "avgcount": 0, 2024-04-23T10:41:28.758 INFO:teuthology.orchestra.run.smithi119.stdout: "sum": 0.000000000, 2024-04-23T10:41:28.759 INFO:teuthology.orchestra.run.smithi119.stdout: "avgtime": 0.000000000 2024-04-23T10:41:28.759 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-04-23T10:41:28.759 INFO:teuthology.orchestra.run.smithi119.stdout: "req_lookupname_latency": { 2024-04-23T10:41:28.759 INFO:teuthology.orchestra.run.smithi119.stdout: "avgcount": 0, 2024-04-23T10:41:28.759 INFO:teuthology.orchestra.run.smithi119.stdout: "sum": 0.000000000, 2024-04-23T10:41:28.759 INFO:teuthology.orchestra.run.smithi119.stdout: "avgtime": 0.000000000 2024-04-23T10:41:28.759 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-04-23T10:41:28.760 INFO:teuthology.orchestra.run.smithi119.stdout: "req_lookupparent_latency": { 2024-04-23T10:41:28.760 INFO:teuthology.orchestra.run.smithi119.stdout: "avgcount": 0, 2024-04-23T10:41:28.760 INFO:teuthology.orchestra.run.smithi119.stdout: "sum": 0.000000000, 2024-04-23T10:41:28.760 INFO:teuthology.orchestra.run.smithi119.stdout: "avgtime": 0.000000000 2024-04-23T10:41:28.760 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-04-23T10:41:28.760 INFO:teuthology.orchestra.run.smithi119.stdout: "req_lookupsnap_latency": { 2024-04-23T10:41:28.760 INFO:teuthology.orchestra.run.smithi119.stdout: "avgcount": 0, 2024-04-23T10:41:28.761 INFO:teuthology.orchestra.run.smithi119.stdout: "sum": 0.000000000, 2024-04-23T10:41:28.761 INFO:journalctl@ceph.mon.a.smithi114.stdout:Apr 23 10:41:28 smithi114 ceph-mon[29255]: pgmap v469: 97 pgs: 97 active+clean; 560 MiB data, 4.3 GiB used, 1.0 TiB / 1.0 TiB avail; 719 KiB/s rd, 6.8 MiB/s wr, 238 op/s 2024-04-23T10:41:28.762 INFO:teuthology.orchestra.run.smithi119.stdout: "avgtime": 0.000000000 2024-04-23T10:41:28.762 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-04-23T10:41:28.762 INFO:teuthology.orchestra.run.smithi119.stdout: "req_lssnap_latency": { 2024-04-23T10:41:28.762 INFO:teuthology.orchestra.run.smithi119.stdout: "avgcount": 0, 2024-04-23T10:41:28.762 INFO:teuthology.orchestra.run.smithi119.stdout: "sum": 0.000000000, 2024-04-23T10:41:28.763 INFO:teuthology.orchestra.run.smithi119.stdout: "avgtime": 0.000000000 2024-04-23T10:41:28.763 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-04-23T10:41:28.763 INFO:teuthology.orchestra.run.smithi119.stdout: "req_mkdir_latency": { 2024-04-23T10:41:28.763 INFO:teuthology.orchestra.run.smithi119.stdout: "avgcount": 0, 2024-04-23T10:41:28.763 INFO:teuthology.orchestra.run.smithi119.stdout: "sum": 0.000000000, 2024-04-23T10:41:28.763 INFO:teuthology.orchestra.run.smithi119.stdout: "avgtime": 0.000000000 2024-04-23T10:41:28.763 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-04-23T10:41:28.763 INFO:teuthology.orchestra.run.smithi119.stdout: "req_mknod_latency": { 2024-04-23T10:41:28.764 INFO:teuthology.orchestra.run.smithi119.stdout: "avgcount": 0, 2024-04-23T10:41:28.764 INFO:teuthology.orchestra.run.smithi119.stdout: "sum": 0.000000000, 2024-04-23T10:41:28.764 INFO:teuthology.orchestra.run.smithi119.stdout: "avgtime": 0.000000000 2024-04-23T10:41:28.764 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-04-23T10:41:28.764 INFO:teuthology.orchestra.run.smithi119.stdout: "req_mksnap_latency": { 2024-04-23T10:41:28.764 INFO:teuthology.orchestra.run.smithi119.stdout: "avgcount": 0, 2024-04-23T10:41:28.764 INFO:teuthology.orchestra.run.smithi119.stdout: "sum": 0.000000000, 2024-04-23T10:41:28.765 INFO:teuthology.orchestra.run.smithi119.stdout: "avgtime": 0.000000000 2024-04-23T10:41:28.765 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-04-23T10:41:28.765 INFO:teuthology.orchestra.run.smithi119.stdout: "req_open_latency": { 2024-04-23T10:41:28.765 INFO:teuthology.orchestra.run.smithi119.stdout: "avgcount": 0, 2024-04-23T10:41:28.765 INFO:teuthology.orchestra.run.smithi119.stdout: "sum": 0.000000000, 2024-04-23T10:41:28.765 INFO:teuthology.orchestra.run.smithi119.stdout: "avgtime": 0.000000000 2024-04-23T10:41:28.765 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-04-23T10:41:28.766 INFO:teuthology.orchestra.run.smithi119.stdout: "req_readdir_latency": { 2024-04-23T10:41:28.766 INFO:teuthology.orchestra.run.smithi119.stdout: "avgcount": 39395, 2024-04-23T10:41:28.766 INFO:teuthology.orchestra.run.smithi119.stdout: "sum": 48.434053083, 2024-04-23T10:41:28.766 INFO:teuthology.orchestra.run.smithi119.stdout: "avgtime": 0.001229446 2024-04-23T10:41:28.766 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-04-23T10:41:28.766 INFO:teuthology.orchestra.run.smithi119.stdout: "req_rename_latency": { 2024-04-23T10:41:28.766 INFO:teuthology.orchestra.run.smithi119.stdout: "avgcount": 800, 2024-04-23T10:41:28.767 INFO:teuthology.orchestra.run.smithi119.stdout: "sum": 12.817565970, 2024-04-23T10:41:28.767 INFO:teuthology.orchestra.run.smithi119.stdout: "avgtime": 0.016021957 2024-04-23T10:41:28.767 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-04-23T10:41:28.767 INFO:teuthology.orchestra.run.smithi119.stdout: "req_renamesnap_latency": { 2024-04-23T10:41:28.767 INFO:teuthology.orchestra.run.smithi119.stdout: "avgcount": 0, 2024-04-23T10:41:28.767 INFO:teuthology.orchestra.run.smithi119.stdout: "sum": 0.000000000, 2024-04-23T10:41:28.767 INFO:teuthology.orchestra.run.smithi119.stdout: "avgtime": 0.000000000 2024-04-23T10:41:28.768 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-04-23T10:41:28.768 INFO:teuthology.orchestra.run.smithi119.stdout: "req_snapdiff_latency": { 2024-04-23T10:41:28.768 INFO:teuthology.orchestra.run.smithi119.stdout: "avgcount": 0, 2024-04-23T10:41:28.768 INFO:teuthology.orchestra.run.smithi119.stdout: "sum": 0.000000000, 2024-04-23T10:41:28.768 INFO:teuthology.orchestra.run.smithi119.stdout: "avgtime": 0.000000000 2024-04-23T10:41:28.768 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-04-23T10:41:28.768 INFO:teuthology.orchestra.run.smithi119.stdout: "req_rmdir_latency": { 2024-04-23T10:41:28.769 INFO:teuthology.orchestra.run.smithi119.stdout: "avgcount": 0, 2024-04-23T10:41:28.769 INFO:teuthology.orchestra.run.smithi119.stdout: "sum": 0.000000000, 2024-04-23T10:41:28.769 INFO:teuthology.orchestra.run.smithi119.stdout: "avgtime": 0.000000000 2024-04-23T10:41:28.769 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-04-23T10:41:28.769 INFO:teuthology.orchestra.run.smithi119.stdout: "req_rmsnap_latency": { 2024-04-23T10:41:28.769 INFO:teuthology.orchestra.run.smithi119.stdout: "avgcount": 0, 2024-04-23T10:41:28.769 INFO:teuthology.orchestra.run.smithi119.stdout: "sum": 0.000000000, 2024-04-23T10:41:28.769 INFO:teuthology.orchestra.run.smithi119.stdout: "avgtime": 0.000000000 2024-04-23T10:41:28.770 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-04-23T10:41:28.770 INFO:teuthology.orchestra.run.smithi119.stdout: "req_rmxattr_latency": { 2024-04-23T10:41:28.770 INFO:teuthology.orchestra.run.smithi119.stdout: "avgcount": 0, 2024-04-23T10:41:28.770 INFO:teuthology.orchestra.run.smithi119.stdout: "sum": 0.000000000, 2024-04-23T10:41:28.770 INFO:teuthology.orchestra.run.smithi119.stdout: "avgtime": 0.000000000 2024-04-23T10:41:28.770 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-04-23T10:41:28.770 INFO:teuthology.orchestra.run.smithi119.stdout: "req_setattr_latency": { 2024-04-23T10:41:28.771 INFO:teuthology.orchestra.run.smithi119.stdout: "avgcount": 2, 2024-04-23T10:41:28.771 INFO:teuthology.orchestra.run.smithi119.stdout: "sum": 0.023778894, 2024-04-23T10:41:28.771 INFO:teuthology.orchestra.run.smithi119.stdout: "avgtime": 0.011889447 2024-04-23T10:41:28.771 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-04-23T10:41:28.771 INFO:teuthology.orchestra.run.smithi119.stdout: "req_setdirlayout_latency": { 2024-04-23T10:41:28.771 INFO:teuthology.orchestra.run.smithi119.stdout: "avgcount": 0, 2024-04-23T10:41:28.771 INFO:teuthology.orchestra.run.smithi119.stdout: "sum": 0.000000000, 2024-04-23T10:41:28.772 INFO:teuthology.orchestra.run.smithi119.stdout: "avgtime": 0.000000000 2024-04-23T10:41:28.772 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-04-23T10:41:28.772 INFO:teuthology.orchestra.run.smithi119.stdout: "req_setfilelock_latency": { 2024-04-23T10:41:28.772 INFO:teuthology.orchestra.run.smithi119.stdout: "avgcount": 92, 2024-04-23T10:41:28.772 INFO:teuthology.orchestra.run.smithi119.stdout: "sum": 0.238290413, 2024-04-23T10:41:28.772 INFO:teuthology.orchestra.run.smithi119.stdout: "avgtime": 0.002590113 2024-04-23T10:41:28.772 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-04-23T10:41:28.773 INFO:teuthology.orchestra.run.smithi119.stdout: "req_setlayout_latency": { 2024-04-23T10:41:28.773 INFO:teuthology.orchestra.run.smithi119.stdout: "avgcount": 0, 2024-04-23T10:41:28.773 INFO:teuthology.orchestra.run.smithi119.stdout: "sum": 0.000000000, 2024-04-23T10:41:28.773 INFO:teuthology.orchestra.run.smithi119.stdout: "avgtime": 0.000000000 2024-04-23T10:41:28.773 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-04-23T10:41:28.773 INFO:teuthology.orchestra.run.smithi119.stdout: "req_setxattr_latency": { 2024-04-23T10:41:28.773 INFO:teuthology.orchestra.run.smithi119.stdout: "avgcount": 0, 2024-04-23T10:41:28.774 INFO:teuthology.orchestra.run.smithi119.stdout: "sum": 0.000000000, 2024-04-23T10:41:28.774 INFO:teuthology.orchestra.run.smithi119.stdout: "avgtime": 0.000000000 2024-04-23T10:41:28.774 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-04-23T10:41:28.774 INFO:teuthology.orchestra.run.smithi119.stdout: "req_symlink_latency": { 2024-04-23T10:41:28.774 INFO:teuthology.orchestra.run.smithi119.stdout: "avgcount": 0, 2024-04-23T10:41:28.774 INFO:teuthology.orchestra.run.smithi119.stdout: "sum": 0.000000000, 2024-04-23T10:41:28.774 INFO:teuthology.orchestra.run.smithi119.stdout: "avgtime": 0.000000000 2024-04-23T10:41:28.775 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-04-23T10:41:28.775 INFO:teuthology.orchestra.run.smithi119.stdout: "req_unlink_latency": { 2024-04-23T10:41:28.775 INFO:teuthology.orchestra.run.smithi119.stdout: "avgcount": 6422, 2024-04-23T10:41:28.775 INFO:teuthology.orchestra.run.smithi119.stdout: "sum": 86.939944756, 2024-04-23T10:41:28.775 INFO:teuthology.orchestra.run.smithi119.stdout: "avgtime": 0.013537830 2024-04-23T10:41:28.775 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-04-23T10:41:28.775 INFO:teuthology.orchestra.run.smithi119.stdout: "cap_revoke_eviction": 0, 2024-04-23T10:41:28.776 INFO:teuthology.orchestra.run.smithi119.stdout: "cap_acquisition_throttle": 0, 2024-04-23T10:41:28.776 INFO:teuthology.orchestra.run.smithi119.stdout: "req_getvxattr_latency": { 2024-04-23T10:41:28.776 INFO:teuthology.orchestra.run.smithi119.stdout: "avgcount": 0, 2024-04-23T10:41:28.776 INFO:teuthology.orchestra.run.smithi119.stdout: "sum": 0.000000000, 2024-04-23T10:41:28.776 INFO:teuthology.orchestra.run.smithi119.stdout: "avgtime": 0.000000000 2024-04-23T10:41:28.776 INFO:teuthology.orchestra.run.smithi119.stdout: } 2024-04-23T10:41:28.776 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-04-23T10:41:28.777 INFO:teuthology.orchestra.run.smithi119.stdout: "mds_sessions": { 2024-04-23T10:41:28.777 INFO:teuthology.orchestra.run.smithi119.stdout: "session_count": 2, 2024-04-23T10:41:28.777 INFO:teuthology.orchestra.run.smithi119.stdout: "session_add": 4, 2024-04-23T10:41:28.777 INFO:teuthology.orchestra.run.smithi119.stdout: "session_remove": 2, 2024-04-23T10:41:28.777 INFO:teuthology.orchestra.run.smithi119.stdout: "sessions_open": 2, 2024-04-23T10:41:28.777 INFO:teuthology.orchestra.run.smithi119.stdout: "sessions_stale": 0, 2024-04-23T10:41:28.777 INFO:teuthology.orchestra.run.smithi119.stdout: "total_load": 15161, 2024-04-23T10:41:28.777 INFO:teuthology.orchestra.run.smithi119.stdout: "average_load": 7580, 2024-04-23T10:41:28.778 INFO:teuthology.orchestra.run.smithi119.stdout: "avg_session_uptime": 841, 2024-04-23T10:41:28.778 INFO:teuthology.orchestra.run.smithi119.stdout: "mdthresh_evicted": 0 2024-04-23T10:41:28.778 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-04-23T10:41:28.778 INFO:teuthology.orchestra.run.smithi119.stdout: "mempool": { 2024-04-23T10:41:28.778 INFO:teuthology.orchestra.run.smithi119.stdout: "bloom_filter_bytes": 0, 2024-04-23T10:41:28.778 INFO:teuthology.orchestra.run.smithi119.stdout: "bloom_filter_items": 0, 2024-04-23T10:41:28.779 INFO:teuthology.orchestra.run.smithi119.stdout: "bluestore_alloc_bytes": 0, 2024-04-23T10:41:28.779 INFO:teuthology.orchestra.run.smithi119.stdout: "bluestore_alloc_items": 0, 2024-04-23T10:41:28.779 INFO:teuthology.orchestra.run.smithi119.stdout: "bluestore_cache_data_bytes": 0, 2024-04-23T10:41:28.779 INFO:teuthology.orchestra.run.smithi119.stdout: "bluestore_cache_data_items": 0, 2024-04-23T10:41:28.779 INFO:teuthology.orchestra.run.smithi119.stdout: "bluestore_cache_onode_bytes": 0, 2024-04-23T10:41:28.779 INFO:teuthology.orchestra.run.smithi119.stdout: "bluestore_cache_onode_items": 0, 2024-04-23T10:41:28.779 INFO:teuthology.orchestra.run.smithi119.stdout: "bluestore_cache_meta_bytes": 0, 2024-04-23T10:41:28.779 INFO:teuthology.orchestra.run.smithi119.stdout: "bluestore_cache_meta_items": 0, 2024-04-23T10:41:28.780 INFO:teuthology.orchestra.run.smithi119.stdout: "bluestore_cache_other_bytes": 0, 2024-04-23T10:41:28.780 INFO:teuthology.orchestra.run.smithi119.stdout: "bluestore_cache_other_items": 0, 2024-04-23T10:41:28.780 INFO:teuthology.orchestra.run.smithi119.stdout: "bluestore_cache_buffer_bytes": 0, 2024-04-23T10:41:28.780 INFO:teuthology.orchestra.run.smithi119.stdout: "bluestore_cache_buffer_items": 0, 2024-04-23T10:41:28.780 INFO:teuthology.orchestra.run.smithi119.stdout: "bluestore_extent_bytes": 0, 2024-04-23T10:41:28.780 INFO:teuthology.orchestra.run.smithi119.stdout: "bluestore_extent_items": 0, 2024-04-23T10:41:28.780 INFO:teuthology.orchestra.run.smithi119.stdout: "bluestore_blob_bytes": 0, 2024-04-23T10:41:28.781 INFO:teuthology.orchestra.run.smithi119.stdout: "bluestore_blob_items": 0, 2024-04-23T10:41:28.781 INFO:teuthology.orchestra.run.smithi119.stdout: "bluestore_shared_blob_bytes": 0, 2024-04-23T10:41:28.781 INFO:teuthology.orchestra.run.smithi119.stdout: "bluestore_shared_blob_items": 0, 2024-04-23T10:41:28.781 INFO:teuthology.orchestra.run.smithi119.stdout: "bluestore_inline_bl_bytes": 0, 2024-04-23T10:41:28.781 INFO:teuthology.orchestra.run.smithi119.stdout: "bluestore_inline_bl_items": 0, 2024-04-23T10:41:28.781 INFO:teuthology.orchestra.run.smithi119.stdout: "bluestore_fsck_bytes": 0, 2024-04-23T10:41:28.781 INFO:teuthology.orchestra.run.smithi119.stdout: "bluestore_fsck_items": 0, 2024-04-23T10:41:28.781 INFO:teuthology.orchestra.run.smithi119.stdout: "bluestore_txc_bytes": 0, 2024-04-23T10:41:28.782 INFO:teuthology.orchestra.run.smithi119.stdout: "bluestore_txc_items": 0, 2024-04-23T10:41:28.782 INFO:teuthology.orchestra.run.smithi119.stdout: "bluestore_writing_deferred_bytes": 0, 2024-04-23T10:41:28.782 INFO:teuthology.orchestra.run.smithi119.stdout: "bluestore_writing_deferred_items": 0, 2024-04-23T10:41:28.782 INFO:teuthology.orchestra.run.smithi119.stdout: "bluestore_writing_bytes": 0, 2024-04-23T10:41:28.782 INFO:teuthology.orchestra.run.smithi119.stdout: "bluestore_writing_items": 0, 2024-04-23T10:41:28.782 INFO:teuthology.orchestra.run.smithi119.stdout: "bluefs_bytes": 0, 2024-04-23T10:41:28.782 INFO:teuthology.orchestra.run.smithi119.stdout: "bluefs_items": 0, 2024-04-23T10:41:28.783 INFO:teuthology.orchestra.run.smithi119.stdout: "bluefs_file_reader_bytes": 0, 2024-04-23T10:41:28.783 INFO:teuthology.orchestra.run.smithi119.stdout: "bluefs_file_reader_items": 0, 2024-04-23T10:41:28.783 INFO:teuthology.orchestra.run.smithi119.stdout: "bluefs_file_writer_bytes": 0, 2024-04-23T10:41:28.783 INFO:teuthology.orchestra.run.smithi119.stdout: "bluefs_file_writer_items": 0, 2024-04-23T10:41:28.783 INFO:teuthology.orchestra.run.smithi119.stdout: "buffer_anon_bytes": 4431587, 2024-04-23T10:41:28.783 INFO:teuthology.orchestra.run.smithi119.stdout: "buffer_anon_items": 1825, 2024-04-23T10:41:28.783 INFO:teuthology.orchestra.run.smithi119.stdout: "buffer_meta_bytes": 0, 2024-04-23T10:41:28.784 INFO:teuthology.orchestra.run.smithi119.stdout: "buffer_meta_items": 0, 2024-04-23T10:41:28.784 INFO:teuthology.orchestra.run.smithi119.stdout: "osd_bytes": 0, 2024-04-23T10:41:28.784 INFO:teuthology.orchestra.run.smithi119.stdout: "osd_items": 0, 2024-04-23T10:41:28.784 INFO:teuthology.orchestra.run.smithi119.stdout: "osd_mapbl_bytes": 0, 2024-04-23T10:41:28.784 INFO:teuthology.orchestra.run.smithi119.stdout: "osd_mapbl_items": 0, 2024-04-23T10:41:28.784 INFO:teuthology.orchestra.run.smithi119.stdout: "osd_pglog_bytes": 0, 2024-04-23T10:41:28.784 INFO:teuthology.orchestra.run.smithi119.stdout: "osd_pglog_items": 0, 2024-04-23T10:41:28.784 INFO:teuthology.orchestra.run.smithi119.stdout: "osdmap_bytes": 8416, 2024-04-23T10:41:28.785 INFO:teuthology.orchestra.run.smithi119.stdout: "osdmap_items": 175, 2024-04-23T10:41:28.785 INFO:teuthology.orchestra.run.smithi119.stdout: "osdmap_mapping_bytes": 0, 2024-04-23T10:41:28.785 INFO:teuthology.orchestra.run.smithi119.stdout: "osdmap_mapping_items": 0, 2024-04-23T10:41:28.785 INFO:teuthology.orchestra.run.smithi119.stdout: "pgmap_bytes": 0, 2024-04-23T10:41:28.785 INFO:teuthology.orchestra.run.smithi119.stdout: "pgmap_items": 0, 2024-04-23T10:41:28.785 INFO:teuthology.orchestra.run.smithi119.stdout: "mds_co_bytes": 4540728, 2024-04-23T10:41:28.785 INFO:teuthology.orchestra.run.smithi119.stdout: "mds_co_items": 103802, 2024-04-23T10:41:28.785 INFO:teuthology.orchestra.run.smithi119.stdout: "unittest_1_bytes": 0, 2024-04-23T10:41:28.786 INFO:teuthology.orchestra.run.smithi119.stdout: "unittest_1_items": 0, 2024-04-23T10:41:28.786 INFO:teuthology.orchestra.run.smithi119.stdout: "unittest_2_bytes": 0, 2024-04-23T10:41:28.786 INFO:teuthology.orchestra.run.smithi119.stdout: "unittest_2_items": 0 2024-04-23T10:41:28.786 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-04-23T10:41:28.786 INFO:teuthology.orchestra.run.smithi119.stdout: "objecter": { 2024-04-23T10:41:28.786 INFO:teuthology.orchestra.run.smithi119.stdout: "op_active": 0, 2024-04-23T10:41:28.786 INFO:teuthology.orchestra.run.smithi119.stdout: "op_laggy": 0, 2024-04-23T10:41:28.787 INFO:teuthology.orchestra.run.smithi119.stdout: "op_send": 18398, 2024-04-23T10:41:28.787 INFO:teuthology.orchestra.run.smithi119.stdout: "op_send_bytes": 118928494, 2024-04-23T10:41:28.787 INFO:teuthology.orchestra.run.smithi119.stdout: "op_resend": 33, 2024-04-23T10:41:28.787 INFO:teuthology.orchestra.run.smithi119.stdout: "op_reply": 18398, 2024-04-23T10:41:28.787 INFO:teuthology.orchestra.run.smithi119.stdout: "op_latency": { 2024-04-23T10:41:28.787 INFO:teuthology.orchestra.run.smithi119.stdout: "avgcount": 18398, 2024-04-23T10:41:28.787 INFO:teuthology.orchestra.run.smithi119.stdout: "sum": 59.374563483, 2024-04-23T10:41:28.788 INFO:teuthology.orchestra.run.smithi119.stdout: "avgtime": 0.003227229 2024-04-23T10:41:28.788 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-04-23T10:41:28.788 INFO:teuthology.orchestra.run.smithi119.stdout: "op_inflight": 0, 2024-04-23T10:41:28.788 INFO:teuthology.orchestra.run.smithi119.stdout: "oplen_avg": { 2024-04-23T10:41:28.788 INFO:teuthology.orchestra.run.smithi119.stdout: "avgcount": 18398, 2024-04-23T10:41:28.788 INFO:teuthology.orchestra.run.smithi119.stdout: "sum": 19111 2024-04-23T10:41:28.788 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-04-23T10:41:28.788 INFO:teuthology.orchestra.run.smithi119.stdout: "op": 18398, 2024-04-23T10:41:28.789 INFO:teuthology.orchestra.run.smithi119.stdout: "op_r": 372, 2024-04-23T10:41:28.789 INFO:teuthology.orchestra.run.smithi119.stdout: "op_w": 18026, 2024-04-23T10:41:28.789 INFO:teuthology.orchestra.run.smithi119.stdout: "op_rmw": 0, 2024-04-23T10:41:28.789 INFO:teuthology.orchestra.run.smithi119.stdout: "op_pg": 0, 2024-04-23T10:41:28.789 INFO:teuthology.orchestra.run.smithi119.stdout: "osdop_stat": 0, 2024-04-23T10:41:28.789 INFO:teuthology.orchestra.run.smithi119.stdout: "osdop_create": 10, 2024-04-23T10:41:28.789 INFO:teuthology.orchestra.run.smithi119.stdout: "osdop_read": 372, 2024-04-23T10:41:28.789 INFO:teuthology.orchestra.run.smithi119.stdout: "osdop_write": 10895, 2024-04-23T10:41:28.790 INFO:teuthology.orchestra.run.smithi119.stdout: "osdop_writefull": 232, 2024-04-23T10:41:28.790 INFO:teuthology.orchestra.run.smithi119.stdout: "osdop_writesame": 0, 2024-04-23T10:41:28.790 INFO:teuthology.orchestra.run.smithi119.stdout: "osdop_append": 0, 2024-04-23T10:41:28.790 INFO:teuthology.orchestra.run.smithi119.stdout: "osdop_zero": 0, 2024-04-23T10:41:28.790 INFO:teuthology.orchestra.run.smithi119.stdout: "osdop_truncate": 4, 2024-04-23T10:41:28.790 INFO:teuthology.orchestra.run.smithi119.stdout: "osdop_delete": 6331, 2024-04-23T10:41:28.790 INFO:teuthology.orchestra.run.smithi119.stdout: "osdop_mapext": 0, 2024-04-23T10:41:28.790 INFO:teuthology.orchestra.run.smithi119.stdout: "osdop_sparse_read": 0, 2024-04-23T10:41:28.791 INFO:teuthology.orchestra.run.smithi119.stdout: "osdop_clonerange": 0, 2024-04-23T10:41:28.791 INFO:teuthology.orchestra.run.smithi119.stdout: "osdop_getxattr": 0, 2024-04-23T10:41:28.791 INFO:teuthology.orchestra.run.smithi119.stdout: "osdop_setxattr": 20, 2024-04-23T10:41:28.791 INFO:teuthology.orchestra.run.smithi119.stdout: "osdop_cmpxattr": 0, 2024-04-23T10:41:28.791 INFO:teuthology.orchestra.run.smithi119.stdout: "osdop_rmxattr": 0, 2024-04-23T10:41:28.791 INFO:teuthology.orchestra.run.smithi119.stdout: "osdop_resetxattrs": 0, 2024-04-23T10:41:28.791 INFO:teuthology.orchestra.run.smithi119.stdout: "osdop_call": 0, 2024-04-23T10:41:28.791 INFO:teuthology.orchestra.run.smithi119.stdout: "osdop_watch": 0, 2024-04-23T10:41:28.792 INFO:teuthology.orchestra.run.smithi119.stdout: "osdop_notify": 0, 2024-04-23T10:41:28.792 INFO:teuthology.orchestra.run.smithi119.stdout: "osdop_src_cmpxattr": 0, 2024-04-23T10:41:28.792 INFO:teuthology.orchestra.run.smithi119.stdout: "osdop_pgls": 0, 2024-04-23T10:41:28.792 INFO:teuthology.orchestra.run.smithi119.stdout: "osdop_pgls_filter": 0, 2024-04-23T10:41:28.792 INFO:teuthology.orchestra.run.smithi119.stdout: "osdop_other": 0, 2024-04-23T10:41:28.792 INFO:teuthology.orchestra.run.smithi119.stdout: "linger_active": 0, 2024-04-23T10:41:28.793 INFO:teuthology.orchestra.run.smithi119.stdout: "linger_send": 0, 2024-04-23T10:41:28.793 INFO:teuthology.orchestra.run.smithi119.stdout: "linger_resend": 0, 2024-04-23T10:41:28.793 INFO:teuthology.orchestra.run.smithi119.stdout: "linger_ping": 0, 2024-04-23T10:41:28.793 INFO:teuthology.orchestra.run.smithi119.stdout: "poolop_active": 0, 2024-04-23T10:41:28.793 INFO:teuthology.orchestra.run.smithi119.stdout: "poolop_send": 0, 2024-04-23T10:41:28.793 INFO:teuthology.orchestra.run.smithi119.stdout: "poolop_resend": 0, 2024-04-23T10:41:28.793 INFO:teuthology.orchestra.run.smithi119.stdout: "poolstat_active": 0, 2024-04-23T10:41:28.793 INFO:teuthology.orchestra.run.smithi119.stdout: "poolstat_send": 0, 2024-04-23T10:41:28.794 INFO:teuthology.orchestra.run.smithi119.stdout: "poolstat_resend": 0, 2024-04-23T10:41:28.794 INFO:teuthology.orchestra.run.smithi119.stdout: "statfs_active": 0, 2024-04-23T10:41:28.794 INFO:teuthology.orchestra.run.smithi119.stdout: "statfs_send": 0, 2024-04-23T10:41:28.794 INFO:teuthology.orchestra.run.smithi119.stdout: "statfs_resend": 0, 2024-04-23T10:41:28.794 INFO:teuthology.orchestra.run.smithi119.stdout: "command_active": 0, 2024-04-23T10:41:28.794 INFO:teuthology.orchestra.run.smithi119.stdout: "command_send": 0, 2024-04-23T10:41:28.794 INFO:teuthology.orchestra.run.smithi119.stdout: "command_resend": 0, 2024-04-23T10:41:28.795 INFO:teuthology.orchestra.run.smithi119.stdout: "map_epoch": 113, 2024-04-23T10:41:28.795 INFO:teuthology.orchestra.run.smithi119.stdout: "map_full": 0, 2024-04-23T10:41:28.795 INFO:teuthology.orchestra.run.smithi119.stdout: "map_inc": 26, 2024-04-23T10:41:28.795 INFO:teuthology.orchestra.run.smithi119.stdout: "osd_sessions": 12, 2024-04-23T10:41:28.795 INFO:teuthology.orchestra.run.smithi119.stdout: "osd_session_open": 12, 2024-04-23T10:41:28.795 INFO:teuthology.orchestra.run.smithi119.stdout: "osd_session_close": 0, 2024-04-23T10:41:28.795 INFO:teuthology.orchestra.run.smithi119.stdout: "osd_laggy": 0, 2024-04-23T10:41:28.796 INFO:teuthology.orchestra.run.smithi119.stdout: "omap_wr": 1023, 2024-04-23T10:41:28.796 INFO:teuthology.orchestra.run.smithi119.stdout: "omap_rd": 0, 2024-04-23T10:41:28.796 INFO:teuthology.orchestra.run.smithi119.stdout: "omap_del": 224 2024-04-23T10:41:28.796 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-04-23T10:41:28.796 INFO:teuthology.orchestra.run.smithi119.stdout: "oft": { 2024-04-23T10:41:28.796 INFO:teuthology.orchestra.run.smithi119.stdout: "omap_total_objs": 1, 2024-04-23T10:41:28.796 INFO:teuthology.orchestra.run.smithi119.stdout: "omap_total_kv_pairs": 0, 2024-04-23T10:41:28.796 INFO:teuthology.orchestra.run.smithi119.stdout: "omap_total_updates": 0, 2024-04-23T10:41:28.797 INFO:teuthology.orchestra.run.smithi119.stdout: "omap_total_removes": 0 2024-04-23T10:41:28.797 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-04-23T10:41:28.797 INFO:teuthology.orchestra.run.smithi119.stdout: "osd-slow-ops": { 2024-04-23T10:41:28.797 INFO:teuthology.orchestra.run.smithi119.stdout: "slow_ops_count": 145 2024-04-23T10:41:28.797 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-04-23T10:41:28.797 INFO:teuthology.orchestra.run.smithi119.stdout: "purge_queue": { 2024-04-23T10:41:28.797 INFO:teuthology.orchestra.run.smithi119.stdout: "pq_executing_ops": 0, 2024-04-23T10:41:28.797 INFO:teuthology.orchestra.run.smithi119.stdout: "pq_executing_ops_high_water": 10, 2024-04-23T10:41:28.798 INFO:teuthology.orchestra.run.smithi119.stdout: "pq_executing": 0, 2024-04-23T10:41:28.798 INFO:teuthology.orchestra.run.smithi119.stdout: "pq_executing_high_water": 10, 2024-04-23T10:41:28.798 INFO:teuthology.orchestra.run.smithi119.stdout: "pq_executed_ops": 6284, 2024-04-23T10:41:28.798 INFO:teuthology.orchestra.run.smithi119.stdout: "pq_executed": 6284, 2024-04-23T10:41:28.798 INFO:teuthology.orchestra.run.smithi119.stdout: "pq_item_in_journal": 0 2024-04-23T10:41:28.798 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-04-23T10:41:28.798 INFO:teuthology.orchestra.run.smithi119.stdout: "throttle-msgr_dispatch_throttler-mds": { 2024-04-23T10:41:28.799 INFO:teuthology.orchestra.run.smithi119.stdout: "val": 0, 2024-04-23T10:41:28.799 INFO:teuthology.orchestra.run.smithi119.stdout: "max": 104857600, 2024-04-23T10:41:28.799 INFO:teuthology.orchestra.run.smithi119.stdout: "get_started": 0, 2024-04-23T10:41:28.799 INFO:teuthology.orchestra.run.smithi119.stdout: "get": 290292, 2024-04-23T10:41:28.799 INFO:teuthology.orchestra.run.smithi119.stdout: "get_sum": 62687637, 2024-04-23T10:41:28.799 INFO:teuthology.orchestra.run.smithi119.stdout: "get_or_fail_fail": 0, 2024-04-23T10:41:28.799 INFO:teuthology.orchestra.run.smithi119.stdout: "get_or_fail_success": 290292, 2024-04-23T10:41:28.799 INFO:teuthology.orchestra.run.smithi119.stdout: "take": 0, 2024-04-23T10:41:28.800 INFO:teuthology.orchestra.run.smithi119.stdout: "take_sum": 0, 2024-04-23T10:41:28.800 INFO:teuthology.orchestra.run.smithi119.stdout: "put": 290292, 2024-04-23T10:41:28.800 INFO:teuthology.orchestra.run.smithi119.stdout: "put_sum": 62687637, 2024-04-23T10:41:28.800 INFO:teuthology.orchestra.run.smithi119.stdout: "wait": { 2024-04-23T10:41:28.800 INFO:teuthology.orchestra.run.smithi119.stdout: "avgcount": 0, 2024-04-23T10:41:28.800 INFO:teuthology.orchestra.run.smithi119.stdout: "sum": 0.000000000, 2024-04-23T10:41:28.800 INFO:teuthology.orchestra.run.smithi119.stdout: "avgtime": 0.000000000 2024-04-23T10:41:28.800 INFO:teuthology.orchestra.run.smithi119.stdout: } 2024-04-23T10:41:28.801 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-04-23T10:41:28.801 INFO:teuthology.orchestra.run.smithi119.stdout: "throttle-objecter_bytes": { 2024-04-23T10:41:28.801 INFO:teuthology.orchestra.run.smithi119.stdout: "val": 0, 2024-04-23T10:41:28.801 INFO:teuthology.orchestra.run.smithi119.stdout: "max": 104857600, 2024-04-23T10:41:28.801 INFO:teuthology.orchestra.run.smithi119.stdout: "get_started": 0, 2024-04-23T10:41:28.801 INFO:teuthology.orchestra.run.smithi119.stdout: "get": 0, 2024-04-23T10:41:28.801 INFO:teuthology.orchestra.run.smithi119.stdout: "get_sum": 0, 2024-04-23T10:41:28.801 INFO:teuthology.orchestra.run.smithi119.stdout: "get_or_fail_fail": 0, 2024-04-23T10:41:28.802 INFO:teuthology.orchestra.run.smithi119.stdout: "get_or_fail_success": 0, 2024-04-23T10:41:28.802 INFO:teuthology.orchestra.run.smithi119.stdout: "take": 18398, 2024-04-23T10:41:28.802 INFO:teuthology.orchestra.run.smithi119.stdout: "take_sum": 119565586, 2024-04-23T10:41:28.802 INFO:teuthology.orchestra.run.smithi119.stdout: "put": 18398, 2024-04-23T10:41:28.802 INFO:teuthology.orchestra.run.smithi119.stdout: "put_sum": 119565586, 2024-04-23T10:41:28.802 INFO:teuthology.orchestra.run.smithi119.stdout: "wait": { 2024-04-23T10:41:28.802 INFO:teuthology.orchestra.run.smithi119.stdout: "avgcount": 0, 2024-04-23T10:41:28.803 INFO:teuthology.orchestra.run.smithi119.stdout: "sum": 0.000000000, 2024-04-23T10:41:28.803 INFO:teuthology.orchestra.run.smithi119.stdout: "avgtime": 0.000000000 2024-04-23T10:41:28.803 INFO:teuthology.orchestra.run.smithi119.stdout: } 2024-04-23T10:41:28.803 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-04-23T10:41:28.803 INFO:teuthology.orchestra.run.smithi119.stdout: "throttle-objecter_ops": { 2024-04-23T10:41:28.803 INFO:teuthology.orchestra.run.smithi119.stdout: "val": 0, 2024-04-23T10:41:28.803 INFO:teuthology.orchestra.run.smithi119.stdout: "max": 1024, 2024-04-23T10:41:28.803 INFO:teuthology.orchestra.run.smithi119.stdout: "get_started": 0, 2024-04-23T10:41:28.804 INFO:teuthology.orchestra.run.smithi119.stdout: "get": 0, 2024-04-23T10:41:28.804 INFO:teuthology.orchestra.run.smithi119.stdout: "get_sum": 0, 2024-04-23T10:41:28.804 INFO:teuthology.orchestra.run.smithi119.stdout: "get_or_fail_fail": 0, 2024-04-23T10:41:28.804 INFO:teuthology.orchestra.run.smithi119.stdout: "get_or_fail_success": 0, 2024-04-23T10:41:28.804 INFO:teuthology.orchestra.run.smithi119.stdout: "take": 18398, 2024-04-23T10:41:28.804 INFO:teuthology.orchestra.run.smithi119.stdout: "take_sum": 18398, 2024-04-23T10:41:28.804 INFO:teuthology.orchestra.run.smithi119.stdout: "put": 18398, 2024-04-23T10:41:28.804 INFO:teuthology.orchestra.run.smithi119.stdout: "put_sum": 18398, 2024-04-23T10:41:28.805 INFO:teuthology.orchestra.run.smithi119.stdout: "wait": { 2024-04-23T10:41:28.805 INFO:teuthology.orchestra.run.smithi119.stdout: "avgcount": 0, 2024-04-23T10:41:28.805 INFO:teuthology.orchestra.run.smithi119.stdout: "sum": 0.000000000, 2024-04-23T10:41:28.805 INFO:teuthology.orchestra.run.smithi119.stdout: "avgtime": 0.000000000 2024-04-23T10:41:28.805 INFO:teuthology.orchestra.run.smithi119.stdout: } 2024-04-23T10:41:28.805 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-04-23T10:41:28.805 INFO:teuthology.orchestra.run.smithi119.stdout: "throttle-write_buf_throttle": { 2024-04-23T10:41:28.805 INFO:teuthology.orchestra.run.smithi119.stdout: "val": 0, 2024-04-23T10:41:28.806 INFO:teuthology.orchestra.run.smithi119.stdout: "max": 3758096384, 2024-04-23T10:41:28.806 INFO:teuthology.orchestra.run.smithi119.stdout: "get_started": 0, 2024-04-23T10:41:28.806 INFO:teuthology.orchestra.run.smithi119.stdout: "get": 6284, 2024-04-23T10:41:28.806 INFO:teuthology.orchestra.run.smithi119.stdout: "get_sum": 637092, 2024-04-23T10:41:28.806 INFO:teuthology.orchestra.run.smithi119.stdout: "get_or_fail_fail": 0, 2024-04-23T10:41:28.806 INFO:teuthology.orchestra.run.smithi119.stdout: "get_or_fail_success": 6284, 2024-04-23T10:41:28.806 INFO:teuthology.orchestra.run.smithi119.stdout: "take": 0, 2024-04-23T10:41:28.807 INFO:teuthology.orchestra.run.smithi119.stdout: "take_sum": 0, 2024-04-23T10:41:28.807 INFO:teuthology.orchestra.run.smithi119.stdout: "put": 372, 2024-04-23T10:41:28.807 INFO:teuthology.orchestra.run.smithi119.stdout: "put_sum": 637092, 2024-04-23T10:41:28.807 INFO:teuthology.orchestra.run.smithi119.stdout: "wait": { 2024-04-23T10:41:28.807 INFO:teuthology.orchestra.run.smithi119.stdout: "avgcount": 0, 2024-04-23T10:41:28.807 INFO:teuthology.orchestra.run.smithi119.stdout: "sum": 0.000000000, 2024-04-23T10:41:28.807 INFO:teuthology.orchestra.run.smithi119.stdout: "avgtime": 0.000000000 2024-04-23T10:41:28.807 INFO:teuthology.orchestra.run.smithi119.stdout: } 2024-04-23T10:41:28.808 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-04-23T10:41:28.808 INFO:teuthology.orchestra.run.smithi119.stdout: "throttle-write_buf_throttle-0x55e71889a500": { 2024-04-23T10:41:28.808 INFO:teuthology.orchestra.run.smithi119.stdout: "val": 84507, 2024-04-23T10:41:28.808 INFO:teuthology.orchestra.run.smithi119.stdout: "max": 3758096384, 2024-04-23T10:41:28.808 INFO:teuthology.orchestra.run.smithi119.stdout: "get_started": 0, 2024-04-23T10:41:28.808 INFO:teuthology.orchestra.run.smithi119.stdout: "get": 62846, 2024-04-23T10:41:28.808 INFO:teuthology.orchestra.run.smithi119.stdout: "get_sum": 117201239, 2024-04-23T10:41:28.808 INFO:teuthology.orchestra.run.smithi119.stdout: "get_or_fail_fail": 0, 2024-04-23T10:41:28.809 INFO:teuthology.orchestra.run.smithi119.stdout: "get_or_fail_success": 62846, 2024-04-23T10:41:28.809 INFO:teuthology.orchestra.run.smithi119.stdout: "take": 0, 2024-04-23T10:41:28.809 INFO:teuthology.orchestra.run.smithi119.stdout: "take_sum": 0, 2024-04-23T10:41:28.809 INFO:teuthology.orchestra.run.smithi119.stdout: "put": 10523, 2024-04-23T10:41:28.809 INFO:teuthology.orchestra.run.smithi119.stdout: "put_sum": 117116732, 2024-04-23T10:41:28.809 INFO:teuthology.orchestra.run.smithi119.stdout: "wait": { 2024-04-23T10:41:28.809 INFO:teuthology.orchestra.run.smithi119.stdout: "avgcount": 0, 2024-04-23T10:41:28.809 INFO:teuthology.orchestra.run.smithi119.stdout: "sum": 0.000000000, 2024-04-23T10:41:28.810 INFO:teuthology.orchestra.run.smithi119.stdout: "avgtime": 0.000000000 2024-04-23T10:41:28.810 INFO:teuthology.orchestra.run.smithi119.stdout: } 2024-04-23T10:41:28.810 INFO:teuthology.orchestra.run.smithi119.stdout: } 2024-04-23T10:41:28.810 INFO:teuthology.orchestra.run.smithi119.stdout:} 2024-04-23T10:41:29.526 INFO:tasks.check_counter:Daemon mds.h mds.exported=10 2024-04-23T10:41:29.526 INFO:tasks.check_counter:Daemon mds.h mds.imported=6 2024-04-23T10:41:29.526 INFO:tasks.check_counter:Daemon mds.h mds_cache.dir_update=585 2024-04-23T10:41:29.526 INFO:tasks.check_counter:Daemon mds.h mds_cache.dir_update_receipt=1093 2024-04-23T10:41:29.526 INFO:tasks.check_counter:Daemon mds.h mds.root_rsnaps=6 2024-04-23T10:41:29.527 INFO:tasks.check_counter:Daemon mds.h mds_server.req_mksnap_latency.avgcount=0 2024-04-23T10:41:29.527 INFO:tasks.check_counter:Daemon mds.h mds_server.req_rmsnap_latency.avgcount=0 2024-04-23T10:41:29.527 DEBUG:tasks.check_counter:Getting stats from k 2024-04-23T10:41:29.527 DEBUG:teuthology.orchestra.run.smithi119:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d69f13d827952bb07bb9b7973bca14f97d764d6f shell --fsid aa240572-0159-11ef-bc93-c7b262605968 -- ceph daemon mds.k perf dump 2024-04-23T10:41:29.884 INFO:teuthology.orchestra.run.smithi119.stderr:Inferring config /var/lib/ceph/aa240572-0159-11ef-bc93-c7b262605968/mon.b/config 2024-04-23T10:41:30.578 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 23 10:41:30 smithi204 ceph-mon[32649]: pgmap v470: 97 pgs: 97 active+clean; 560 MiB data, 4.3 GiB used, 1.0 TiB / 1.0 TiB avail; 523 KiB/s rd, 4.7 MiB/s wr, 167 op/s 2024-04-23T10:41:30.711 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 23 10:41:30 smithi119 ceph-mon[32608]: pgmap v470: 97 pgs: 97 active+clean; 560 MiB data, 4.3 GiB used, 1.0 TiB / 1.0 TiB avail; 523 KiB/s rd, 4.7 MiB/s wr, 167 op/s 2024-04-23T10:41:30.754 INFO:journalctl@ceph.mon.a.smithi114.stdout:Apr 23 10:41:30 smithi114 ceph-mon[29255]: pgmap v470: 97 pgs: 97 active+clean; 560 MiB data, 4.3 GiB used, 1.0 TiB / 1.0 TiB avail; 523 KiB/s rd, 4.7 MiB/s wr, 167 op/s 2024-04-23T10:41:31.548 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 23 10:41:31 smithi119 ceph-mon[32608]: pgmap v471: 97 pgs: 97 active+clean; 560 MiB data, 4.3 GiB used, 1.0 TiB / 1.0 TiB avail; 523 KiB/s rd, 4.7 MiB/s wr, 167 op/s 2024-04-23T10:41:31.549 INFO:teuthology.orchestra.run.smithi119.stdout:{ 2024-04-23T10:41:31.549 INFO:teuthology.orchestra.run.smithi119.stdout: "AsyncMessenger::Worker-0": { 2024-04-23T10:41:31.550 INFO:teuthology.orchestra.run.smithi119.stdout: "msgr_recv_messages": 2203, 2024-04-23T10:41:31.550 INFO:teuthology.orchestra.run.smithi119.stdout: "msgr_send_messages": 2205, 2024-04-23T10:41:31.550 INFO:teuthology.orchestra.run.smithi119.stdout: "msgr_recv_bytes": 38392037, 2024-04-23T10:41:31.550 INFO:teuthology.orchestra.run.smithi119.stdout: "msgr_send_bytes": 700561, 2024-04-23T10:41:31.550 INFO:teuthology.orchestra.run.smithi119.stdout: "msgr_created_connections": 8, 2024-04-23T10:41:31.551 INFO:teuthology.orchestra.run.smithi119.stdout: "msgr_active_connections": 3, 2024-04-23T10:41:31.551 INFO:teuthology.orchestra.run.smithi119.stdout: "msgr_running_total_time": 0.185284711, 2024-04-23T10:41:31.551 INFO:teuthology.orchestra.run.smithi119.stdout: "msgr_running_send_time": 0.054713580, 2024-04-23T10:41:31.551 INFO:teuthology.orchestra.run.smithi119.stdout: "msgr_running_recv_time": 0.062598263, 2024-04-23T10:41:31.551 INFO:teuthology.orchestra.run.smithi119.stdout: "msgr_running_fast_dispatch_time": 0.053253572, 2024-04-23T10:41:31.551 INFO:teuthology.orchestra.run.smithi119.stdout: "msgr_send_messages_queue_lat": { 2024-04-23T10:41:31.552 INFO:teuthology.orchestra.run.smithi119.stdout: "avgcount": 2203, 2024-04-23T10:41:31.552 INFO:teuthology.orchestra.run.smithi119.stdout: "sum": 0.062454113, 2024-04-23T10:41:31.552 INFO:teuthology.orchestra.run.smithi119.stdout: "avgtime": 0.000028349 2024-04-23T10:41:31.552 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-04-23T10:41:31.552 INFO:teuthology.orchestra.run.smithi119.stdout: "msgr_handle_ack_lat": { 2024-04-23T10:41:31.552 INFO:teuthology.orchestra.run.smithi119.stdout: "avgcount": 0, 2024-04-23T10:41:31.553 INFO:teuthology.orchestra.run.smithi119.stdout: "sum": 0.000000000, 2024-04-23T10:41:31.553 INFO:teuthology.orchestra.run.smithi119.stdout: "avgtime": 0.000000000 2024-04-23T10:41:31.553 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-04-23T10:41:31.553 INFO:teuthology.orchestra.run.smithi119.stdout: "msgr_recv_encrypted_bytes": 0, 2024-04-23T10:41:31.553 INFO:teuthology.orchestra.run.smithi119.stdout: "msgr_send_encrypted_bytes": 0 2024-04-23T10:41:31.553 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-04-23T10:41:31.553 INFO:teuthology.orchestra.run.smithi119.stdout: "AsyncMessenger::Worker-1": { 2024-04-23T10:41:31.554 INFO:teuthology.orchestra.run.smithi119.stdout: "msgr_recv_messages": 773, 2024-04-23T10:41:31.554 INFO:teuthology.orchestra.run.smithi119.stdout: "msgr_send_messages": 984, 2024-04-23T10:41:31.554 INFO:teuthology.orchestra.run.smithi119.stdout: "msgr_recv_bytes": 46336032, 2024-04-23T10:41:31.554 INFO:teuthology.orchestra.run.smithi119.stdout: "msgr_send_bytes": 780568, 2024-04-23T10:41:31.554 INFO:teuthology.orchestra.run.smithi119.stdout: "msgr_created_connections": 9, 2024-04-23T10:41:31.554 INFO:teuthology.orchestra.run.smithi119.stdout: "msgr_active_connections": 5, 2024-04-23T10:41:31.555 INFO:teuthology.orchestra.run.smithi119.stdout: "msgr_running_total_time": 0.090096160, 2024-04-23T10:41:31.555 INFO:teuthology.orchestra.run.smithi119.stdout: "msgr_running_send_time": 0.023809668, 2024-04-23T10:41:31.555 INFO:teuthology.orchestra.run.smithi119.stdout: "msgr_running_recv_time": 0.041099167, 2024-04-23T10:41:31.555 INFO:teuthology.orchestra.run.smithi119.stdout: "msgr_running_fast_dispatch_time": 0.018688361, 2024-04-23T10:41:31.555 INFO:teuthology.orchestra.run.smithi119.stdout: "msgr_send_messages_queue_lat": { 2024-04-23T10:41:31.555 INFO:teuthology.orchestra.run.smithi119.stdout: "avgcount": 981, 2024-04-23T10:41:31.556 INFO:teuthology.orchestra.run.smithi119.stdout: "sum": 0.037912439, 2024-04-23T10:41:31.556 INFO:teuthology.orchestra.run.smithi119.stdout: "avgtime": 0.000038646 2024-04-23T10:41:31.556 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-04-23T10:41:31.556 INFO:teuthology.orchestra.run.smithi119.stdout: "msgr_handle_ack_lat": { 2024-04-23T10:41:31.556 INFO:teuthology.orchestra.run.smithi119.stdout: "avgcount": 0, 2024-04-23T10:41:31.556 INFO:teuthology.orchestra.run.smithi119.stdout: "sum": 0.000000000, 2024-04-23T10:41:31.556 INFO:teuthology.orchestra.run.smithi119.stdout: "avgtime": 0.000000000 2024-04-23T10:41:31.557 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-04-23T10:41:31.557 INFO:teuthology.orchestra.run.smithi119.stdout: "msgr_recv_encrypted_bytes": 320, 2024-04-23T10:41:31.557 INFO:teuthology.orchestra.run.smithi119.stdout: "msgr_send_encrypted_bytes": 535328 2024-04-23T10:41:31.557 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-04-23T10:41:31.557 INFO:teuthology.orchestra.run.smithi119.stdout: "AsyncMessenger::Worker-2": { 2024-04-23T10:41:31.557 INFO:teuthology.orchestra.run.smithi119.stdout: "msgr_recv_messages": 844, 2024-04-23T10:41:31.558 INFO:teuthology.orchestra.run.smithi119.stdout: "msgr_send_messages": 802, 2024-04-23T10:41:31.558 INFO:teuthology.orchestra.run.smithi119.stdout: "msgr_recv_bytes": 35456001, 2024-04-23T10:41:31.558 INFO:teuthology.orchestra.run.smithi119.stdout: "msgr_send_bytes": 356262, 2024-04-23T10:41:31.558 INFO:teuthology.orchestra.run.smithi119.stdout: "msgr_created_connections": 8, 2024-04-23T10:41:31.558 INFO:teuthology.orchestra.run.smithi119.stdout: "msgr_active_connections": 4, 2024-04-23T10:41:31.558 INFO:teuthology.orchestra.run.smithi119.stdout: "msgr_running_total_time": 0.104593582, 2024-04-23T10:41:31.559 INFO:teuthology.orchestra.run.smithi119.stdout: "msgr_running_send_time": 0.026249384, 2024-04-23T10:41:31.559 INFO:teuthology.orchestra.run.smithi119.stdout: "msgr_running_recv_time": 0.067943502, 2024-04-23T10:41:31.559 INFO:teuthology.orchestra.run.smithi119.stdout: "msgr_running_fast_dispatch_time": 0.017835701, 2024-04-23T10:41:31.559 INFO:teuthology.orchestra.run.smithi119.stdout: "msgr_send_messages_queue_lat": { 2024-04-23T10:41:31.559 INFO:teuthology.orchestra.run.smithi119.stdout: "avgcount": 802, 2024-04-23T10:41:31.559 INFO:teuthology.orchestra.run.smithi119.stdout: "sum": 0.026002841, 2024-04-23T10:41:31.560 INFO:teuthology.orchestra.run.smithi119.stdout: "avgtime": 0.000032422 2024-04-23T10:41:31.560 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-04-23T10:41:31.560 INFO:teuthology.orchestra.run.smithi119.stdout: "msgr_handle_ack_lat": { 2024-04-23T10:41:31.560 INFO:teuthology.orchestra.run.smithi119.stdout: "avgcount": 2, 2024-04-23T10:41:31.560 INFO:teuthology.orchestra.run.smithi119.stdout: "sum": 0.000000172, 2024-04-23T10:41:31.560 INFO:teuthology.orchestra.run.smithi119.stdout: "avgtime": 0.000000086 2024-04-23T10:41:31.560 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-04-23T10:41:31.561 INFO:teuthology.orchestra.run.smithi119.stdout: "msgr_recv_encrypted_bytes": 1765488, 2024-04-23T10:41:31.561 INFO:teuthology.orchestra.run.smithi119.stdout: "msgr_send_encrypted_bytes": 177600 2024-04-23T10:41:31.561 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-04-23T10:41:31.561 INFO:teuthology.orchestra.run.smithi119.stdout: "cct": { 2024-04-23T10:41:31.561 INFO:teuthology.orchestra.run.smithi119.stdout: "total_workers": 1, 2024-04-23T10:41:31.561 INFO:teuthology.orchestra.run.smithi119.stdout: "unhealthy_workers": 0 2024-04-23T10:41:31.562 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-04-23T10:41:31.562 INFO:teuthology.orchestra.run.smithi119.stdout: "finisher-MDSRank": { 2024-04-23T10:41:31.562 INFO:teuthology.orchestra.run.smithi119.stdout: "queue_len": 0, 2024-04-23T10:41:31.562 INFO:teuthology.orchestra.run.smithi119.stdout: "complete_latency": { 2024-04-23T10:41:31.562 INFO:teuthology.orchestra.run.smithi119.stdout: "avgcount": 5881, 2024-04-23T10:41:31.562 INFO:teuthology.orchestra.run.smithi119.stdout: "sum": 0.123117047, 2024-04-23T10:41:31.563 INFO:teuthology.orchestra.run.smithi119.stdout: "avgtime": 0.000020934 2024-04-23T10:41:31.563 INFO:teuthology.orchestra.run.smithi119.stdout: } 2024-04-23T10:41:31.563 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-04-23T10:41:31.563 INFO:teuthology.orchestra.run.smithi119.stdout: "finisher-PurgeQueue": { 2024-04-23T10:41:31.563 INFO:teuthology.orchestra.run.smithi119.stdout: "queue_len": 0, 2024-04-23T10:41:31.563 INFO:teuthology.orchestra.run.smithi119.stdout: "complete_latency": { 2024-04-23T10:41:31.563 INFO:teuthology.orchestra.run.smithi119.stdout: "avgcount": 0, 2024-04-23T10:41:31.564 INFO:teuthology.orchestra.run.smithi119.stdout: "sum": 0.000000000, 2024-04-23T10:41:31.564 INFO:teuthology.orchestra.run.smithi119.stdout: "avgtime": 0.000000000 2024-04-23T10:41:31.564 INFO:teuthology.orchestra.run.smithi119.stdout: } 2024-04-23T10:41:31.564 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-04-23T10:41:31.564 INFO:teuthology.orchestra.run.smithi119.stdout: "mds": { 2024-04-23T10:41:31.564 INFO:teuthology.orchestra.run.smithi119.stdout: "request": 0, 2024-04-23T10:41:31.564 INFO:teuthology.orchestra.run.smithi119.stdout: "reply": 0, 2024-04-23T10:41:31.565 INFO:teuthology.orchestra.run.smithi119.stdout: "reply_latency": { 2024-04-23T10:41:31.565 INFO:teuthology.orchestra.run.smithi119.stdout: "avgcount": 0, 2024-04-23T10:41:31.565 INFO:teuthology.orchestra.run.smithi119.stdout: "sum": 0.000000000, 2024-04-23T10:41:31.565 INFO:teuthology.orchestra.run.smithi119.stdout: "avgtime": 0.000000000 2024-04-23T10:41:31.565 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-04-23T10:41:31.565 INFO:teuthology.orchestra.run.smithi119.stdout: "slow_reply": 0, 2024-04-23T10:41:31.566 INFO:teuthology.orchestra.run.smithi119.stdout: "forward": 0, 2024-04-23T10:41:31.566 INFO:teuthology.orchestra.run.smithi119.stdout: "dir_fetch_complete": 0, 2024-04-23T10:41:31.566 INFO:teuthology.orchestra.run.smithi119.stdout: "dir_fetch_keys": 0, 2024-04-23T10:41:31.566 INFO:teuthology.orchestra.run.smithi119.stdout: "dir_commit": 0, 2024-04-23T10:41:31.566 INFO:teuthology.orchestra.run.smithi119.stdout: "dir_split": 0, 2024-04-23T10:41:31.566 INFO:teuthology.orchestra.run.smithi119.stdout: "dir_merge": 0, 2024-04-23T10:41:31.566 INFO:teuthology.orchestra.run.smithi119.stdout: "inodes": 7176, 2024-04-23T10:41:31.567 INFO:teuthology.orchestra.run.smithi119.stdout: "inodes_top": 47, 2024-04-23T10:41:31.567 INFO:teuthology.orchestra.run.smithi119.stdout: "inodes_bottom": 7129, 2024-04-23T10:41:31.567 INFO:teuthology.orchestra.run.smithi119.stdout: "inodes_pin_tail": 0, 2024-04-23T10:41:31.567 INFO:teuthology.orchestra.run.smithi119.stdout: "inodes_pinned": 7109, 2024-04-23T10:41:31.567 INFO:teuthology.orchestra.run.smithi119.stdout: "inodes_expired": 0, 2024-04-23T10:41:31.567 INFO:teuthology.orchestra.run.smithi119.stdout: "inodes_with_caps": 0, 2024-04-23T10:41:31.568 INFO:teuthology.orchestra.run.smithi119.stdout: "caps": 0, 2024-04-23T10:41:31.568 INFO:teuthology.orchestra.run.smithi119.stdout: "subtrees": 5, 2024-04-23T10:41:31.568 INFO:teuthology.orchestra.run.smithi119.stdout: "traverse": 0, 2024-04-23T10:41:31.568 INFO:teuthology.orchestra.run.smithi119.stdout: "traverse_hit": 0, 2024-04-23T10:41:31.568 INFO:teuthology.orchestra.run.smithi119.stdout: "traverse_forward": 0, 2024-04-23T10:41:31.568 INFO:teuthology.orchestra.run.smithi119.stdout: "traverse_discover": 0, 2024-04-23T10:41:31.568 INFO:teuthology.orchestra.run.smithi119.stdout: "traverse_dir_fetch": 0, 2024-04-23T10:41:31.569 INFO:teuthology.orchestra.run.smithi119.stdout: "traverse_remote_ino": 0, 2024-04-23T10:41:31.569 INFO:teuthology.orchestra.run.smithi119.stdout: "traverse_lock": 0, 2024-04-23T10:41:31.569 INFO:teuthology.orchestra.run.smithi119.stdout: "load_cent": 0, 2024-04-23T10:41:31.569 INFO:teuthology.orchestra.run.smithi119.stdout: "q": 0, 2024-04-23T10:41:31.569 INFO:teuthology.orchestra.run.smithi119.stdout: "exported": 0, 2024-04-23T10:41:31.569 INFO:teuthology.orchestra.run.smithi119.stdout: "exported_inodes": 0, 2024-04-23T10:41:31.569 INFO:teuthology.orchestra.run.smithi119.stdout: "imported": 0, 2024-04-23T10:41:31.570 INFO:teuthology.orchestra.run.smithi119.stdout: "imported_inodes": 0, 2024-04-23T10:41:31.570 INFO:teuthology.orchestra.run.smithi119.stdout: "openino_dir_fetch": 0, 2024-04-23T10:41:31.570 INFO:teuthology.orchestra.run.smithi119.stdout: "openino_backtrace_fetch": 0, 2024-04-23T10:41:31.570 INFO:teuthology.orchestra.run.smithi119.stdout: "openino_peer_discover": 0, 2024-04-23T10:41:31.570 INFO:teuthology.orchestra.run.smithi119.stdout: "root_rfiles": 0, 2024-04-23T10:41:31.570 INFO:teuthology.orchestra.run.smithi119.stdout: "root_rbytes": 0, 2024-04-23T10:41:31.571 INFO:teuthology.orchestra.run.smithi119.stdout: "root_rsnaps": 0, 2024-04-23T10:41:31.571 INFO:teuthology.orchestra.run.smithi119.stdout: "scrub_backtrace_fetch": 0, 2024-04-23T10:41:31.571 INFO:teuthology.orchestra.run.smithi119.stdout: "scrub_set_tag": 0, 2024-04-23T10:41:31.571 INFO:teuthology.orchestra.run.smithi119.stdout: "scrub_backtrace_repaired": 0, 2024-04-23T10:41:31.571 INFO:teuthology.orchestra.run.smithi119.stdout: "scrub_inotable_repaired": 0, 2024-04-23T10:41:31.571 INFO:teuthology.orchestra.run.smithi119.stdout: "scrub_dir_inodes": 0, 2024-04-23T10:41:31.571 INFO:teuthology.orchestra.run.smithi119.stdout: "scrub_dir_base_inodes": 0, 2024-04-23T10:41:31.572 INFO:teuthology.orchestra.run.smithi119.stdout: "scrub_dirfrag_rstats": 0, 2024-04-23T10:41:31.572 INFO:teuthology.orchestra.run.smithi119.stdout: "scrub_file_inodes": 0, 2024-04-23T10:41:31.572 INFO:teuthology.orchestra.run.smithi119.stdout: "handle_inode_file_caps": 0, 2024-04-23T10:41:31.572 INFO:teuthology.orchestra.run.smithi119.stdout: "ceph_cap_op_revoke": 0, 2024-04-23T10:41:31.572 INFO:teuthology.orchestra.run.smithi119.stdout: "ceph_cap_op_grant": 0, 2024-04-23T10:41:31.572 INFO:teuthology.orchestra.run.smithi119.stdout: "ceph_cap_op_trunc": 0, 2024-04-23T10:41:31.573 INFO:teuthology.orchestra.run.smithi119.stdout: "ceph_cap_op_flushsnap_ack": 0, 2024-04-23T10:41:31.573 INFO:teuthology.orchestra.run.smithi119.stdout: "ceph_cap_op_flush_ack": 0, 2024-04-23T10:41:31.573 INFO:teuthology.orchestra.run.smithi119.stdout: "handle_client_caps": 0, 2024-04-23T10:41:31.573 INFO:teuthology.orchestra.run.smithi119.stdout: "handle_client_caps_dirty": 0, 2024-04-23T10:41:31.573 INFO:teuthology.orchestra.run.smithi119.stdout: "handle_client_cap_release": 0, 2024-04-23T10:41:31.573 INFO:teuthology.orchestra.run.smithi119.stdout: "process_request_cap_release": 0 2024-04-23T10:41:31.573 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-04-23T10:41:31.573 INFO:teuthology.orchestra.run.smithi119.stdout: "mds_cache": { 2024-04-23T10:41:31.574 INFO:teuthology.orchestra.run.smithi119.stdout: "dir_update": 0, 2024-04-23T10:41:31.574 INFO:teuthology.orchestra.run.smithi119.stdout: "dir_update_receipt": 0, 2024-04-23T10:41:31.574 INFO:teuthology.orchestra.run.smithi119.stdout: "dir_try_discover": 0, 2024-04-23T10:41:31.574 INFO:teuthology.orchestra.run.smithi119.stdout: "dir_send_discover": 0, 2024-04-23T10:41:31.574 INFO:teuthology.orchestra.run.smithi119.stdout: "dir_handle_discover": 0, 2024-04-23T10:41:31.574 INFO:teuthology.orchestra.run.smithi119.stdout: "num_strays": 0, 2024-04-23T10:41:31.575 INFO:teuthology.orchestra.run.smithi119.stdout: "num_strays_delayed": 0, 2024-04-23T10:41:31.575 INFO:teuthology.orchestra.run.smithi119.stdout: "num_strays_enqueuing": 0, 2024-04-23T10:41:31.575 INFO:teuthology.orchestra.run.smithi119.stdout: "strays_created": 0, 2024-04-23T10:41:31.575 INFO:teuthology.orchestra.run.smithi119.stdout: "strays_enqueued": 0, 2024-04-23T10:41:31.575 INFO:teuthology.orchestra.run.smithi119.stdout: "strays_reintegrated": 0, 2024-04-23T10:41:31.575 INFO:teuthology.orchestra.run.smithi119.stdout: "strays_migrated": 0, 2024-04-23T10:41:31.575 INFO:teuthology.orchestra.run.smithi119.stdout: "num_recovering_processing": 0, 2024-04-23T10:41:31.576 INFO:teuthology.orchestra.run.smithi119.stdout: "num_recovering_enqueued": 0, 2024-04-23T10:41:31.576 INFO:teuthology.orchestra.run.smithi119.stdout: "num_recovering_prioritized": 0, 2024-04-23T10:41:31.576 INFO:teuthology.orchestra.run.smithi119.stdout: "recovery_started": 0, 2024-04-23T10:41:31.576 INFO:teuthology.orchestra.run.smithi119.stdout: "recovery_completed": 0, 2024-04-23T10:41:31.576 INFO:teuthology.orchestra.run.smithi119.stdout: "ireq_quiesce_path": 0, 2024-04-23T10:41:31.576 INFO:teuthology.orchestra.run.smithi119.stdout: "ireq_quiesce_inode": 0, 2024-04-23T10:41:31.576 INFO:teuthology.orchestra.run.smithi119.stdout: "ireq_enqueue_scrub": 0, 2024-04-23T10:41:31.577 INFO:teuthology.orchestra.run.smithi119.stdout: "ireq_exportdir": 0, 2024-04-23T10:41:31.577 INFO:teuthology.orchestra.run.smithi119.stdout: "ireq_flush": 0, 2024-04-23T10:41:31.577 INFO:teuthology.orchestra.run.smithi119.stdout: "ireq_fragmentdir": 0, 2024-04-23T10:41:31.577 INFO:teuthology.orchestra.run.smithi119.stdout: "ireq_fragstats": 0, 2024-04-23T10:41:31.577 INFO:teuthology.orchestra.run.smithi119.stdout: "ireq_inodestats": 0 2024-04-23T10:41:31.577 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-04-23T10:41:31.577 INFO:teuthology.orchestra.run.smithi119.stdout: "mds_log": { 2024-04-23T10:41:31.577 INFO:teuthology.orchestra.run.smithi119.stdout: "evlrg": 0, 2024-04-23T10:41:31.578 INFO:teuthology.orchestra.run.smithi119.stdout: "evadd": 0, 2024-04-23T10:41:31.578 INFO:teuthology.orchestra.run.smithi119.stdout: "evex": 0, 2024-04-23T10:41:31.578 INFO:teuthology.orchestra.run.smithi119.stdout: "evtrm": 0, 2024-04-23T10:41:31.578 INFO:teuthology.orchestra.run.smithi119.stdout: "ev": 62881, 2024-04-23T10:41:31.578 INFO:teuthology.orchestra.run.smithi119.stdout: "evexg": 0, 2024-04-23T10:41:31.578 INFO:teuthology.orchestra.run.smithi119.stdout: "evexd": 0, 2024-04-23T10:41:31.578 INFO:teuthology.orchestra.run.smithi119.stdout: "segadd": 0, 2024-04-23T10:41:31.579 INFO:teuthology.orchestra.run.smithi119.stdout: "segex": 0, 2024-04-23T10:41:31.579 INFO:teuthology.orchestra.run.smithi119.stdout: "segtrm": 0, 2024-04-23T10:41:31.579 INFO:teuthology.orchestra.run.smithi119.stdout: "seg": 82, 2024-04-23T10:41:31.579 INFO:teuthology.orchestra.run.smithi119.stdout: "segmjr": 7, 2024-04-23T10:41:31.579 INFO:teuthology.orchestra.run.smithi119.stdout: "segexg": 0, 2024-04-23T10:41:31.579 INFO:teuthology.orchestra.run.smithi119.stdout: "segexd": 0, 2024-04-23T10:41:31.579 INFO:teuthology.orchestra.run.smithi119.stdout: "expos": 4194304, 2024-04-23T10:41:31.580 INFO:teuthology.orchestra.run.smithi119.stdout: "wrpos": 121634498, 2024-04-23T10:41:31.580 INFO:teuthology.orchestra.run.smithi119.stdout: "rdpos": 121627207, 2024-04-23T10:41:31.580 INFO:teuthology.orchestra.run.smithi119.stdout: "jlat": { 2024-04-23T10:41:31.580 INFO:teuthology.orchestra.run.smithi119.stdout: "avgcount": 0, 2024-04-23T10:41:31.580 INFO:teuthology.orchestra.run.smithi119.stdout: "sum": 0.000000000, 2024-04-23T10:41:31.580 INFO:teuthology.orchestra.run.smithi119.stdout: "avgtime": 0.000000000 2024-04-23T10:41:31.580 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-04-23T10:41:31.581 INFO:teuthology.orchestra.run.smithi119.stdout: "replayed": 62881 2024-04-23T10:41:31.581 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-04-23T10:41:31.581 INFO:teuthology.orchestra.run.smithi119.stdout: "mds_mem": { 2024-04-23T10:41:31.581 INFO:teuthology.orchestra.run.smithi119.stdout: "ino": 625, 2024-04-23T10:41:31.581 INFO:teuthology.orchestra.run.smithi119.stdout: "ino+": 7194, 2024-04-23T10:41:31.581 INFO:teuthology.orchestra.run.smithi119.stdout: "ino-": 6569, 2024-04-23T10:41:31.581 INFO:teuthology.orchestra.run.smithi119.stdout: "dir": 66, 2024-04-23T10:41:31.582 INFO:teuthology.orchestra.run.smithi119.stdout: "dir+": 98, 2024-04-23T10:41:31.582 INFO:teuthology.orchestra.run.smithi119.stdout: "dir-": 32, 2024-04-23T10:41:31.582 INFO:teuthology.orchestra.run.smithi119.stdout: "dn": 7176, 2024-04-23T10:41:31.582 INFO:teuthology.orchestra.run.smithi119.stdout: "dn+": 7495, 2024-04-23T10:41:31.582 INFO:teuthology.orchestra.run.smithi119.stdout: "dn-": 319, 2024-04-23T10:41:31.582 INFO:teuthology.orchestra.run.smithi119.stdout: "cap": 0, 2024-04-23T10:41:31.583 INFO:teuthology.orchestra.run.smithi119.stdout: "cap+": 0, 2024-04-23T10:41:31.583 INFO:teuthology.orchestra.run.smithi119.stdout: "cap-": 0, 2024-04-23T10:41:31.583 INFO:teuthology.orchestra.run.smithi119.stdout: "rss": 63228, 2024-04-23T10:41:31.583 INFO:teuthology.orchestra.run.smithi119.stdout: "heap": 215416 2024-04-23T10:41:31.583 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-04-23T10:41:31.583 INFO:teuthology.orchestra.run.smithi119.stdout: "mds_server": { 2024-04-23T10:41:31.583 INFO:teuthology.orchestra.run.smithi119.stdout: "dispatch_client_request": 0, 2024-04-23T10:41:31.583 INFO:teuthology.orchestra.run.smithi119.stdout: "dispatch_server_request": 0, 2024-04-23T10:41:31.584 INFO:teuthology.orchestra.run.smithi119.stdout: "handle_client_request": 0, 2024-04-23T10:41:31.584 INFO:teuthology.orchestra.run.smithi119.stdout: "handle_client_session": 0, 2024-04-23T10:41:31.584 INFO:teuthology.orchestra.run.smithi119.stdout: "handle_peer_request": 0, 2024-04-23T10:41:31.584 INFO:teuthology.orchestra.run.smithi119.stdout: "req_create_latency": { 2024-04-23T10:41:31.584 INFO:teuthology.orchestra.run.smithi119.stdout: "avgcount": 0, 2024-04-23T10:41:31.584 INFO:teuthology.orchestra.run.smithi119.stdout: "sum": 0.000000000, 2024-04-23T10:41:31.584 INFO:teuthology.orchestra.run.smithi119.stdout: "avgtime": 0.000000000 2024-04-23T10:41:31.585 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-04-23T10:41:31.585 INFO:teuthology.orchestra.run.smithi119.stdout: "req_getattr_latency": { 2024-04-23T10:41:31.585 INFO:teuthology.orchestra.run.smithi119.stdout: "avgcount": 0, 2024-04-23T10:41:31.585 INFO:teuthology.orchestra.run.smithi119.stdout: "sum": 0.000000000, 2024-04-23T10:41:31.585 INFO:teuthology.orchestra.run.smithi119.stdout: "avgtime": 0.000000000 2024-04-23T10:41:31.585 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-04-23T10:41:31.585 INFO:teuthology.orchestra.run.smithi119.stdout: "req_getfilelock_latency": { 2024-04-23T10:41:31.585 INFO:teuthology.orchestra.run.smithi119.stdout: "avgcount": 0, 2024-04-23T10:41:31.586 INFO:teuthology.orchestra.run.smithi119.stdout: "sum": 0.000000000, 2024-04-23T10:41:31.586 INFO:teuthology.orchestra.run.smithi119.stdout: "avgtime": 0.000000000 2024-04-23T10:41:31.586 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-04-23T10:41:31.586 INFO:teuthology.orchestra.run.smithi119.stdout: "req_link_latency": { 2024-04-23T10:41:31.586 INFO:teuthology.orchestra.run.smithi119.stdout: "avgcount": 0, 2024-04-23T10:41:31.586 INFO:teuthology.orchestra.run.smithi119.stdout: "sum": 0.000000000, 2024-04-23T10:41:31.586 INFO:teuthology.orchestra.run.smithi119.stdout: "avgtime": 0.000000000 2024-04-23T10:41:31.587 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-04-23T10:41:31.587 INFO:teuthology.orchestra.run.smithi119.stdout: "req_lookup_latency": { 2024-04-23T10:41:31.587 INFO:teuthology.orchestra.run.smithi119.stdout: "avgcount": 0, 2024-04-23T10:41:31.587 INFO:teuthology.orchestra.run.smithi119.stdout: "sum": 0.000000000, 2024-04-23T10:41:31.587 INFO:teuthology.orchestra.run.smithi119.stdout: "avgtime": 0.000000000 2024-04-23T10:41:31.587 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-04-23T10:41:31.587 INFO:teuthology.orchestra.run.smithi119.stdout: "req_lookuphash_latency": { 2024-04-23T10:41:31.588 INFO:teuthology.orchestra.run.smithi119.stdout: "avgcount": 0, 2024-04-23T10:41:31.588 INFO:teuthology.orchestra.run.smithi119.stdout: "sum": 0.000000000, 2024-04-23T10:41:31.588 INFO:teuthology.orchestra.run.smithi119.stdout: "avgtime": 0.000000000 2024-04-23T10:41:31.588 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-04-23T10:41:31.588 INFO:teuthology.orchestra.run.smithi119.stdout: "req_lookupino_latency": { 2024-04-23T10:41:31.588 INFO:teuthology.orchestra.run.smithi119.stdout: "avgcount": 0, 2024-04-23T10:41:31.588 INFO:teuthology.orchestra.run.smithi119.stdout: "sum": 0.000000000, 2024-04-23T10:41:31.589 INFO:teuthology.orchestra.run.smithi119.stdout: "avgtime": 0.000000000 2024-04-23T10:41:31.589 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-04-23T10:41:31.589 INFO:teuthology.orchestra.run.smithi119.stdout: "req_lookupname_latency": { 2024-04-23T10:41:31.589 INFO:teuthology.orchestra.run.smithi119.stdout: "avgcount": 0, 2024-04-23T10:41:31.589 INFO:teuthology.orchestra.run.smithi119.stdout: "sum": 0.000000000, 2024-04-23T10:41:31.589 INFO:teuthology.orchestra.run.smithi119.stdout: "avgtime": 0.000000000 2024-04-23T10:41:31.589 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-04-23T10:41:31.589 INFO:teuthology.orchestra.run.smithi119.stdout: "req_lookupparent_latency": { 2024-04-23T10:41:31.590 INFO:teuthology.orchestra.run.smithi119.stdout: "avgcount": 0, 2024-04-23T10:41:31.590 INFO:teuthology.orchestra.run.smithi119.stdout: "sum": 0.000000000, 2024-04-23T10:41:31.590 INFO:teuthology.orchestra.run.smithi119.stdout: "avgtime": 0.000000000 2024-04-23T10:41:31.590 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-04-23T10:41:31.590 INFO:teuthology.orchestra.run.smithi119.stdout: "req_lookupsnap_latency": { 2024-04-23T10:41:31.590 INFO:teuthology.orchestra.run.smithi119.stdout: "avgcount": 0, 2024-04-23T10:41:31.590 INFO:teuthology.orchestra.run.smithi119.stdout: "sum": 0.000000000, 2024-04-23T10:41:31.591 INFO:teuthology.orchestra.run.smithi119.stdout: "avgtime": 0.000000000 2024-04-23T10:41:31.591 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-04-23T10:41:31.591 INFO:teuthology.orchestra.run.smithi119.stdout: "req_lssnap_latency": { 2024-04-23T10:41:31.591 INFO:teuthology.orchestra.run.smithi119.stdout: "avgcount": 0, 2024-04-23T10:41:31.591 INFO:teuthology.orchestra.run.smithi119.stdout: "sum": 0.000000000, 2024-04-23T10:41:31.591 INFO:teuthology.orchestra.run.smithi119.stdout: "avgtime": 0.000000000 2024-04-23T10:41:31.591 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-04-23T10:41:31.592 INFO:teuthology.orchestra.run.smithi119.stdout: "req_mkdir_latency": { 2024-04-23T10:41:31.592 INFO:teuthology.orchestra.run.smithi119.stdout: "avgcount": 0, 2024-04-23T10:41:31.593 INFO:teuthology.orchestra.run.smithi119.stdout: "sum": 0.000000000, 2024-04-23T10:41:31.593 INFO:teuthology.orchestra.run.smithi119.stdout: "avgtime": 0.000000000 2024-04-23T10:41:31.593 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-04-23T10:41:31.593 INFO:teuthology.orchestra.run.smithi119.stdout: "req_mknod_latency": { 2024-04-23T10:41:31.593 INFO:teuthology.orchestra.run.smithi119.stdout: "avgcount": 0, 2024-04-23T10:41:31.593 INFO:teuthology.orchestra.run.smithi119.stdout: "sum": 0.000000000, 2024-04-23T10:41:31.593 INFO:teuthology.orchestra.run.smithi119.stdout: "avgtime": 0.000000000 2024-04-23T10:41:31.594 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-04-23T10:41:31.594 INFO:teuthology.orchestra.run.smithi119.stdout: "req_mksnap_latency": { 2024-04-23T10:41:31.594 INFO:teuthology.orchestra.run.smithi119.stdout: "avgcount": 0, 2024-04-23T10:41:31.594 INFO:teuthology.orchestra.run.smithi119.stdout: "sum": 0.000000000, 2024-04-23T10:41:31.594 INFO:teuthology.orchestra.run.smithi119.stdout: "avgtime": 0.000000000 2024-04-23T10:41:31.594 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-04-23T10:41:31.594 INFO:teuthology.orchestra.run.smithi119.stdout: "req_open_latency": { 2024-04-23T10:41:31.595 INFO:teuthology.orchestra.run.smithi119.stdout: "avgcount": 0, 2024-04-23T10:41:31.595 INFO:teuthology.orchestra.run.smithi119.stdout: "sum": 0.000000000, 2024-04-23T10:41:31.595 INFO:teuthology.orchestra.run.smithi119.stdout: "avgtime": 0.000000000 2024-04-23T10:41:31.595 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-04-23T10:41:31.595 INFO:teuthology.orchestra.run.smithi119.stdout: "req_readdir_latency": { 2024-04-23T10:41:31.595 INFO:teuthology.orchestra.run.smithi119.stdout: "avgcount": 0, 2024-04-23T10:41:31.595 INFO:teuthology.orchestra.run.smithi119.stdout: "sum": 0.000000000, 2024-04-23T10:41:31.595 INFO:teuthology.orchestra.run.smithi119.stdout: "avgtime": 0.000000000 2024-04-23T10:41:31.596 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-04-23T10:41:31.596 INFO:teuthology.orchestra.run.smithi119.stdout: "req_rename_latency": { 2024-04-23T10:41:31.596 INFO:teuthology.orchestra.run.smithi119.stdout: "avgcount": 0, 2024-04-23T10:41:31.596 INFO:teuthology.orchestra.run.smithi119.stdout: "sum": 0.000000000, 2024-04-23T10:41:31.596 INFO:teuthology.orchestra.run.smithi119.stdout: "avgtime": 0.000000000 2024-04-23T10:41:31.596 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-04-23T10:41:31.596 INFO:teuthology.orchestra.run.smithi119.stdout: "req_renamesnap_latency": { 2024-04-23T10:41:31.597 INFO:teuthology.orchestra.run.smithi119.stdout: "avgcount": 0, 2024-04-23T10:41:31.597 INFO:teuthology.orchestra.run.smithi119.stdout: "sum": 0.000000000, 2024-04-23T10:41:31.597 INFO:teuthology.orchestra.run.smithi119.stdout: "avgtime": 0.000000000 2024-04-23T10:41:31.597 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-04-23T10:41:31.597 INFO:teuthology.orchestra.run.smithi119.stdout: "req_snapdiff_latency": { 2024-04-23T10:41:31.597 INFO:teuthology.orchestra.run.smithi119.stdout: "avgcount": 0, 2024-04-23T10:41:31.597 INFO:teuthology.orchestra.run.smithi119.stdout: "sum": 0.000000000, 2024-04-23T10:41:31.598 INFO:teuthology.orchestra.run.smithi119.stdout: "avgtime": 0.000000000 2024-04-23T10:41:31.598 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-04-23T10:41:31.598 INFO:teuthology.orchestra.run.smithi119.stdout: "req_rmdir_latency": { 2024-04-23T10:41:31.598 INFO:teuthology.orchestra.run.smithi119.stdout: "avgcount": 0, 2024-04-23T10:41:31.598 INFO:teuthology.orchestra.run.smithi119.stdout: "sum": 0.000000000, 2024-04-23T10:41:31.598 INFO:teuthology.orchestra.run.smithi119.stdout: "avgtime": 0.000000000 2024-04-23T10:41:31.598 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-04-23T10:41:31.598 INFO:teuthology.orchestra.run.smithi119.stdout: "req_rmsnap_latency": { 2024-04-23T10:41:31.599 INFO:teuthology.orchestra.run.smithi119.stdout: "avgcount": 0, 2024-04-23T10:41:31.599 INFO:teuthology.orchestra.run.smithi119.stdout: "sum": 0.000000000, 2024-04-23T10:41:31.599 INFO:teuthology.orchestra.run.smithi119.stdout: "avgtime": 0.000000000 2024-04-23T10:41:31.599 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-04-23T10:41:31.599 INFO:teuthology.orchestra.run.smithi119.stdout: "req_rmxattr_latency": { 2024-04-23T10:41:31.599 INFO:teuthology.orchestra.run.smithi119.stdout: "avgcount": 0, 2024-04-23T10:41:31.599 INFO:teuthology.orchestra.run.smithi119.stdout: "sum": 0.000000000, 2024-04-23T10:41:31.600 INFO:teuthology.orchestra.run.smithi119.stdout: "avgtime": 0.000000000 2024-04-23T10:41:31.600 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-04-23T10:41:31.600 INFO:teuthology.orchestra.run.smithi119.stdout: "req_setattr_latency": { 2024-04-23T10:41:31.600 INFO:teuthology.orchestra.run.smithi119.stdout: "avgcount": 0, 2024-04-23T10:41:31.600 INFO:teuthology.orchestra.run.smithi119.stdout: "sum": 0.000000000, 2024-04-23T10:41:31.600 INFO:teuthology.orchestra.run.smithi119.stdout: "avgtime": 0.000000000 2024-04-23T10:41:31.600 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-04-23T10:41:31.601 INFO:teuthology.orchestra.run.smithi119.stdout: "req_setdirlayout_latency": { 2024-04-23T10:41:31.601 INFO:teuthology.orchestra.run.smithi119.stdout: "avgcount": 0, 2024-04-23T10:41:31.601 INFO:teuthology.orchestra.run.smithi119.stdout: "sum": 0.000000000, 2024-04-23T10:41:31.601 INFO:teuthology.orchestra.run.smithi119.stdout: "avgtime": 0.000000000 2024-04-23T10:41:31.601 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-04-23T10:41:31.601 INFO:teuthology.orchestra.run.smithi119.stdout: "req_setfilelock_latency": { 2024-04-23T10:41:31.601 INFO:teuthology.orchestra.run.smithi119.stdout: "avgcount": 0, 2024-04-23T10:41:31.602 INFO:teuthology.orchestra.run.smithi119.stdout: "sum": 0.000000000, 2024-04-23T10:41:31.602 INFO:teuthology.orchestra.run.smithi119.stdout: "avgtime": 0.000000000 2024-04-23T10:41:31.602 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-04-23T10:41:31.602 INFO:teuthology.orchestra.run.smithi119.stdout: "req_setlayout_latency": { 2024-04-23T10:41:31.602 INFO:teuthology.orchestra.run.smithi119.stdout: "avgcount": 0, 2024-04-23T10:41:31.602 INFO:teuthology.orchestra.run.smithi119.stdout: "sum": 0.000000000, 2024-04-23T10:41:31.602 INFO:teuthology.orchestra.run.smithi119.stdout: "avgtime": 0.000000000 2024-04-23T10:41:31.602 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-04-23T10:41:31.603 INFO:teuthology.orchestra.run.smithi119.stdout: "req_setxattr_latency": { 2024-04-23T10:41:31.603 INFO:teuthology.orchestra.run.smithi119.stdout: "avgcount": 0, 2024-04-23T10:41:31.603 INFO:teuthology.orchestra.run.smithi119.stdout: "sum": 0.000000000, 2024-04-23T10:41:31.603 INFO:teuthology.orchestra.run.smithi119.stdout: "avgtime": 0.000000000 2024-04-23T10:41:31.603 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-04-23T10:41:31.603 INFO:teuthology.orchestra.run.smithi119.stdout: "req_symlink_latency": { 2024-04-23T10:41:31.603 INFO:teuthology.orchestra.run.smithi119.stdout: "avgcount": 0, 2024-04-23T10:41:31.604 INFO:teuthology.orchestra.run.smithi119.stdout: "sum": 0.000000000, 2024-04-23T10:41:31.604 INFO:teuthology.orchestra.run.smithi119.stdout: "avgtime": 0.000000000 2024-04-23T10:41:31.604 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-04-23T10:41:31.604 INFO:teuthology.orchestra.run.smithi119.stdout: "req_unlink_latency": { 2024-04-23T10:41:31.604 INFO:teuthology.orchestra.run.smithi119.stdout: "avgcount": 0, 2024-04-23T10:41:31.604 INFO:teuthology.orchestra.run.smithi119.stdout: "sum": 0.000000000, 2024-04-23T10:41:31.604 INFO:teuthology.orchestra.run.smithi119.stdout: "avgtime": 0.000000000 2024-04-23T10:41:31.604 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-04-23T10:41:31.605 INFO:teuthology.orchestra.run.smithi119.stdout: "cap_revoke_eviction": 0, 2024-04-23T10:41:31.605 INFO:teuthology.orchestra.run.smithi119.stdout: "cap_acquisition_throttle": 0, 2024-04-23T10:41:31.605 INFO:teuthology.orchestra.run.smithi119.stdout: "req_getvxattr_latency": { 2024-04-23T10:41:31.605 INFO:teuthology.orchestra.run.smithi119.stdout: "avgcount": 0, 2024-04-23T10:41:31.605 INFO:teuthology.orchestra.run.smithi119.stdout: "sum": 0.000000000, 2024-04-23T10:41:31.605 INFO:teuthology.orchestra.run.smithi119.stdout: "avgtime": 0.000000000 2024-04-23T10:41:31.605 INFO:teuthology.orchestra.run.smithi119.stdout: } 2024-04-23T10:41:31.605 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-04-23T10:41:31.606 INFO:teuthology.orchestra.run.smithi119.stdout: "mds_sessions": { 2024-04-23T10:41:31.606 INFO:teuthology.orchestra.run.smithi119.stdout: "session_count": 2, 2024-04-23T10:41:31.606 INFO:teuthology.orchestra.run.smithi119.stdout: "session_add": 4, 2024-04-23T10:41:31.606 INFO:teuthology.orchestra.run.smithi119.stdout: "session_remove": 2, 2024-04-23T10:41:31.606 INFO:teuthology.orchestra.run.smithi119.stdout: "sessions_open": 3, 2024-04-23T10:41:31.606 INFO:teuthology.orchestra.run.smithi119.stdout: "sessions_stale": 0, 2024-04-23T10:41:31.606 INFO:teuthology.orchestra.run.smithi119.stdout: "total_load": 0, 2024-04-23T10:41:31.607 INFO:teuthology.orchestra.run.smithi119.stdout: "average_load": 0, 2024-04-23T10:41:31.607 INFO:teuthology.orchestra.run.smithi119.stdout: "avg_session_uptime": 3670, 2024-04-23T10:41:31.607 INFO:teuthology.orchestra.run.smithi119.stdout: "mdthresh_evicted": 0 2024-04-23T10:41:31.607 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-04-23T10:41:31.607 INFO:teuthology.orchestra.run.smithi119.stdout: "mempool": { 2024-04-23T10:41:31.607 INFO:teuthology.orchestra.run.smithi119.stdout: "bloom_filter_bytes": 0, 2024-04-23T10:41:31.607 INFO:teuthology.orchestra.run.smithi119.stdout: "bloom_filter_items": 0, 2024-04-23T10:41:31.608 INFO:teuthology.orchestra.run.smithi119.stdout: "bluestore_alloc_bytes": 0, 2024-04-23T10:41:31.608 INFO:teuthology.orchestra.run.smithi119.stdout: "bluestore_alloc_items": 0, 2024-04-23T10:41:31.608 INFO:teuthology.orchestra.run.smithi119.stdout: "bluestore_cache_data_bytes": 0, 2024-04-23T10:41:31.608 INFO:teuthology.orchestra.run.smithi119.stdout: "bluestore_cache_data_items": 0, 2024-04-23T10:41:31.608 INFO:teuthology.orchestra.run.smithi119.stdout: "bluestore_cache_onode_bytes": 0, 2024-04-23T10:41:31.608 INFO:teuthology.orchestra.run.smithi119.stdout: "bluestore_cache_onode_items": 0, 2024-04-23T10:41:31.608 INFO:teuthology.orchestra.run.smithi119.stdout: "bluestore_cache_meta_bytes": 0, 2024-04-23T10:41:31.608 INFO:teuthology.orchestra.run.smithi119.stdout: "bluestore_cache_meta_items": 0, 2024-04-23T10:41:31.609 INFO:teuthology.orchestra.run.smithi119.stdout: "bluestore_cache_other_bytes": 0, 2024-04-23T10:41:31.609 INFO:teuthology.orchestra.run.smithi119.stdout: "bluestore_cache_other_items": 0, 2024-04-23T10:41:31.609 INFO:teuthology.orchestra.run.smithi119.stdout: "bluestore_cache_buffer_bytes": 0, 2024-04-23T10:41:31.609 INFO:teuthology.orchestra.run.smithi119.stdout: "bluestore_cache_buffer_items": 0, 2024-04-23T10:41:31.609 INFO:teuthology.orchestra.run.smithi119.stdout: "bluestore_extent_bytes": 0, 2024-04-23T10:41:31.609 INFO:teuthology.orchestra.run.smithi119.stdout: "bluestore_extent_items": 0, 2024-04-23T10:41:31.609 INFO:teuthology.orchestra.run.smithi119.stdout: "bluestore_blob_bytes": 0, 2024-04-23T10:41:31.609 INFO:teuthology.orchestra.run.smithi119.stdout: "bluestore_blob_items": 0, 2024-04-23T10:41:31.610 INFO:teuthology.orchestra.run.smithi119.stdout: "bluestore_shared_blob_bytes": 0, 2024-04-23T10:41:31.610 INFO:teuthology.orchestra.run.smithi119.stdout: "bluestore_shared_blob_items": 0, 2024-04-23T10:41:31.610 INFO:teuthology.orchestra.run.smithi119.stdout: "bluestore_inline_bl_bytes": 0, 2024-04-23T10:41:31.610 INFO:teuthology.orchestra.run.smithi119.stdout: "bluestore_inline_bl_items": 0, 2024-04-23T10:41:31.610 INFO:teuthology.orchestra.run.smithi119.stdout: "bluestore_fsck_bytes": 0, 2024-04-23T10:41:31.610 INFO:teuthology.orchestra.run.smithi119.stdout: "bluestore_fsck_items": 0, 2024-04-23T10:41:31.610 INFO:teuthology.orchestra.run.smithi119.stdout: "bluestore_txc_bytes": 0, 2024-04-23T10:41:31.611 INFO:teuthology.orchestra.run.smithi119.stdout: "bluestore_txc_items": 0, 2024-04-23T10:41:31.611 INFO:teuthology.orchestra.run.smithi119.stdout: "bluestore_writing_deferred_bytes": 0, 2024-04-23T10:41:31.611 INFO:teuthology.orchestra.run.smithi119.stdout: "bluestore_writing_deferred_items": 0, 2024-04-23T10:41:31.611 INFO:teuthology.orchestra.run.smithi119.stdout: "bluestore_writing_bytes": 0, 2024-04-23T10:41:31.611 INFO:teuthology.orchestra.run.smithi119.stdout: "bluestore_writing_items": 0, 2024-04-23T10:41:31.611 INFO:teuthology.orchestra.run.smithi119.stdout: "bluefs_bytes": 0, 2024-04-23T10:41:31.611 INFO:teuthology.orchestra.run.smithi119.stdout: "bluefs_items": 0, 2024-04-23T10:41:31.611 INFO:teuthology.orchestra.run.smithi119.stdout: "bluefs_file_reader_bytes": 0, 2024-04-23T10:41:31.612 INFO:teuthology.orchestra.run.smithi119.stdout: "bluefs_file_reader_items": 0, 2024-04-23T10:41:31.612 INFO:teuthology.orchestra.run.smithi119.stdout: "bluefs_file_writer_bytes": 0, 2024-04-23T10:41:31.612 INFO:teuthology.orchestra.run.smithi119.stdout: "bluefs_file_writer_items": 0, 2024-04-23T10:41:31.612 INFO:teuthology.orchestra.run.smithi119.stdout: "buffer_anon_bytes": 265355, 2024-04-23T10:41:31.612 INFO:teuthology.orchestra.run.smithi119.stdout: "buffer_anon_items": 740, 2024-04-23T10:41:31.612 INFO:teuthology.orchestra.run.smithi119.stdout: "buffer_meta_bytes": 0, 2024-04-23T10:41:31.612 INFO:teuthology.orchestra.run.smithi119.stdout: "buffer_meta_items": 0, 2024-04-23T10:41:31.613 INFO:teuthology.orchestra.run.smithi119.stdout: "osd_bytes": 0, 2024-04-23T10:41:31.613 INFO:teuthology.orchestra.run.smithi119.stdout: "osd_items": 0, 2024-04-23T10:41:31.613 INFO:teuthology.orchestra.run.smithi119.stdout: "osd_mapbl_bytes": 0, 2024-04-23T10:41:31.613 INFO:teuthology.orchestra.run.smithi119.stdout: "osd_mapbl_items": 0, 2024-04-23T10:41:31.613 INFO:teuthology.orchestra.run.smithi119.stdout: "osd_pglog_bytes": 0, 2024-04-23T10:41:31.613 INFO:teuthology.orchestra.run.smithi119.stdout: "osd_pglog_items": 0, 2024-04-23T10:41:31.613 INFO:teuthology.orchestra.run.smithi119.stdout: "osdmap_bytes": 8416, 2024-04-23T10:41:31.613 INFO:teuthology.orchestra.run.smithi119.stdout: "osdmap_items": 175, 2024-04-23T10:41:31.614 INFO:teuthology.orchestra.run.smithi119.stdout: "osdmap_mapping_bytes": 0, 2024-04-23T10:41:31.614 INFO:teuthology.orchestra.run.smithi119.stdout: "osdmap_mapping_items": 0, 2024-04-23T10:41:31.614 INFO:teuthology.orchestra.run.smithi119.stdout: "pgmap_bytes": 0, 2024-04-23T10:41:31.614 INFO:teuthology.orchestra.run.smithi119.stdout: "pgmap_items": 0, 2024-04-23T10:41:31.614 INFO:teuthology.orchestra.run.smithi119.stdout: "mds_co_bytes": 6742419, 2024-04-23T10:41:31.614 INFO:teuthology.orchestra.run.smithi119.stdout: "mds_co_items": 59113, 2024-04-23T10:41:31.614 INFO:teuthology.orchestra.run.smithi119.stdout: "unittest_1_bytes": 0, 2024-04-23T10:41:31.614 INFO:teuthology.orchestra.run.smithi119.stdout: "unittest_1_items": 0, 2024-04-23T10:41:31.615 INFO:teuthology.orchestra.run.smithi119.stdout: "unittest_2_bytes": 0, 2024-04-23T10:41:31.615 INFO:teuthology.orchestra.run.smithi119.stdout: "unittest_2_items": 0 2024-04-23T10:41:31.615 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-04-23T10:41:31.615 INFO:teuthology.orchestra.run.smithi119.stdout: "objecter": { 2024-04-23T10:41:31.615 INFO:teuthology.orchestra.run.smithi119.stdout: "op_active": 0, 2024-04-23T10:41:31.615 INFO:teuthology.orchestra.run.smithi119.stdout: "op_laggy": 0, 2024-04-23T10:41:31.615 INFO:teuthology.orchestra.run.smithi119.stdout: "op_send": 3462, 2024-04-23T10:41:31.615 INFO:teuthology.orchestra.run.smithi119.stdout: "op_send_bytes": 21, 2024-04-23T10:41:31.616 INFO:teuthology.orchestra.run.smithi119.stdout: "op_resend": 3, 2024-04-23T10:41:31.616 INFO:teuthology.orchestra.run.smithi119.stdout: "op_reply": 3462, 2024-04-23T10:41:31.616 INFO:teuthology.orchestra.run.smithi119.stdout: "op_latency": { 2024-04-23T10:41:31.616 INFO:teuthology.orchestra.run.smithi119.stdout: "avgcount": 3462, 2024-04-23T10:41:31.616 INFO:teuthology.orchestra.run.smithi119.stdout: "sum": 2.008951157, 2024-04-23T10:41:31.616 INFO:teuthology.orchestra.run.smithi119.stdout: "avgtime": 0.000580286 2024-04-23T10:41:31.616 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-04-23T10:41:31.616 INFO:teuthology.orchestra.run.smithi119.stdout: "op_inflight": 0, 2024-04-23T10:41:31.617 INFO:teuthology.orchestra.run.smithi119.stdout: "oplen_avg": { 2024-04-23T10:41:31.617 INFO:teuthology.orchestra.run.smithi119.stdout: "avgcount": 3462, 2024-04-23T10:41:31.617 INFO:teuthology.orchestra.run.smithi119.stdout: "sum": 3463 2024-04-23T10:41:31.617 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-04-23T10:41:31.617 INFO:teuthology.orchestra.run.smithi119.stdout: "op": 3462, 2024-04-23T10:41:31.617 INFO:teuthology.orchestra.run.smithi119.stdout: "op_r": 3462, 2024-04-23T10:41:31.617 INFO:teuthology.orchestra.run.smithi119.stdout: "op_w": 0, 2024-04-23T10:41:31.617 INFO:teuthology.orchestra.run.smithi119.stdout: "op_rmw": 0, 2024-04-23T10:41:31.618 INFO:teuthology.orchestra.run.smithi119.stdout: "op_pg": 0, 2024-04-23T10:41:31.618 INFO:teuthology.orchestra.run.smithi119.stdout: "osdop_stat": 1942, 2024-04-23T10:41:31.618 INFO:teuthology.orchestra.run.smithi119.stdout: "osdop_create": 0, 2024-04-23T10:41:31.618 INFO:teuthology.orchestra.run.smithi119.stdout: "osdop_read": 1518, 2024-04-23T10:41:31.618 INFO:teuthology.orchestra.run.smithi119.stdout: "osdop_write": 0, 2024-04-23T10:41:31.618 INFO:teuthology.orchestra.run.smithi119.stdout: "osdop_writefull": 0, 2024-04-23T10:41:31.618 INFO:teuthology.orchestra.run.smithi119.stdout: "osdop_writesame": 0, 2024-04-23T10:41:31.618 INFO:teuthology.orchestra.run.smithi119.stdout: "osdop_append": 0, 2024-04-23T10:41:31.619 INFO:teuthology.orchestra.run.smithi119.stdout: "osdop_zero": 0, 2024-04-23T10:41:31.619 INFO:teuthology.orchestra.run.smithi119.stdout: "osdop_truncate": 0, 2024-04-23T10:41:31.619 INFO:teuthology.orchestra.run.smithi119.stdout: "osdop_delete": 0, 2024-04-23T10:41:31.619 INFO:teuthology.orchestra.run.smithi119.stdout: "osdop_mapext": 0, 2024-04-23T10:41:31.619 INFO:teuthology.orchestra.run.smithi119.stdout: "osdop_sparse_read": 0, 2024-04-23T10:41:31.619 INFO:teuthology.orchestra.run.smithi119.stdout: "osdop_clonerange": 0, 2024-04-23T10:41:31.619 INFO:teuthology.orchestra.run.smithi119.stdout: "osdop_getxattr": 1, 2024-04-23T10:41:31.619 INFO:teuthology.orchestra.run.smithi119.stdout: "osdop_setxattr": 0, 2024-04-23T10:41:31.620 INFO:teuthology.orchestra.run.smithi119.stdout: "osdop_cmpxattr": 0, 2024-04-23T10:41:31.620 INFO:teuthology.orchestra.run.smithi119.stdout: "osdop_rmxattr": 0, 2024-04-23T10:41:31.620 INFO:teuthology.orchestra.run.smithi119.stdout: "osdop_resetxattrs": 0, 2024-04-23T10:41:31.620 INFO:teuthology.orchestra.run.smithi119.stdout: "osdop_call": 0, 2024-04-23T10:41:31.620 INFO:teuthology.orchestra.run.smithi119.stdout: "osdop_watch": 0, 2024-04-23T10:41:31.620 INFO:teuthology.orchestra.run.smithi119.stdout: "osdop_notify": 0, 2024-04-23T10:41:31.620 INFO:teuthology.orchestra.run.smithi119.stdout: "osdop_src_cmpxattr": 0, 2024-04-23T10:41:31.621 INFO:teuthology.orchestra.run.smithi119.stdout: "osdop_pgls": 0, 2024-04-23T10:41:31.621 INFO:teuthology.orchestra.run.smithi119.stdout: "osdop_pgls_filter": 0, 2024-04-23T10:41:31.621 INFO:teuthology.orchestra.run.smithi119.stdout: "osdop_other": 0, 2024-04-23T10:41:31.621 INFO:teuthology.orchestra.run.smithi119.stdout: "linger_active": 0, 2024-04-23T10:41:31.621 INFO:teuthology.orchestra.run.smithi119.stdout: "linger_send": 0, 2024-04-23T10:41:31.621 INFO:teuthology.orchestra.run.smithi119.stdout: "linger_resend": 0, 2024-04-23T10:41:31.621 INFO:teuthology.orchestra.run.smithi119.stdout: "linger_ping": 0, 2024-04-23T10:41:31.621 INFO:teuthology.orchestra.run.smithi119.stdout: "poolop_active": 0, 2024-04-23T10:41:31.622 INFO:teuthology.orchestra.run.smithi119.stdout: "poolop_send": 0, 2024-04-23T10:41:31.622 INFO:teuthology.orchestra.run.smithi119.stdout: "poolop_resend": 0, 2024-04-23T10:41:31.622 INFO:teuthology.orchestra.run.smithi119.stdout: "poolstat_active": 0, 2024-04-23T10:41:31.622 INFO:teuthology.orchestra.run.smithi119.stdout: "poolstat_send": 0, 2024-04-23T10:41:31.622 INFO:teuthology.orchestra.run.smithi119.stdout: "poolstat_resend": 0, 2024-04-23T10:41:31.622 INFO:teuthology.orchestra.run.smithi119.stdout: "statfs_active": 0, 2024-04-23T10:41:31.622 INFO:teuthology.orchestra.run.smithi119.stdout: "statfs_send": 0, 2024-04-23T10:41:31.622 INFO:teuthology.orchestra.run.smithi119.stdout: "statfs_resend": 0, 2024-04-23T10:41:31.623 INFO:teuthology.orchestra.run.smithi119.stdout: "command_active": 0, 2024-04-23T10:41:31.623 INFO:teuthology.orchestra.run.smithi119.stdout: "command_send": 0, 2024-04-23T10:41:31.623 INFO:teuthology.orchestra.run.smithi119.stdout: "command_resend": 0, 2024-04-23T10:41:31.623 INFO:teuthology.orchestra.run.smithi119.stdout: "map_epoch": 113, 2024-04-23T10:41:31.623 INFO:teuthology.orchestra.run.smithi119.stdout: "map_full": 0, 2024-04-23T10:41:31.623 INFO:teuthology.orchestra.run.smithi119.stdout: "map_inc": 26, 2024-04-23T10:41:31.623 INFO:teuthology.orchestra.run.smithi119.stdout: "osd_sessions": 11, 2024-04-23T10:41:31.623 INFO:teuthology.orchestra.run.smithi119.stdout: "osd_session_open": 12, 2024-04-23T10:41:31.624 INFO:teuthology.orchestra.run.smithi119.stdout: "osd_session_close": 1, 2024-04-23T10:41:31.624 INFO:teuthology.orchestra.run.smithi119.stdout: "osd_laggy": 0, 2024-04-23T10:41:31.624 INFO:teuthology.orchestra.run.smithi119.stdout: "omap_wr": 0, 2024-04-23T10:41:31.624 INFO:teuthology.orchestra.run.smithi119.stdout: "omap_rd": 2, 2024-04-23T10:41:31.624 INFO:teuthology.orchestra.run.smithi119.stdout: "omap_del": 0 2024-04-23T10:41:31.624 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-04-23T10:41:31.624 INFO:teuthology.orchestra.run.smithi119.stdout: "oft": { 2024-04-23T10:41:31.624 INFO:teuthology.orchestra.run.smithi119.stdout: "omap_total_objs": 0, 2024-04-23T10:41:31.625 INFO:teuthology.orchestra.run.smithi119.stdout: "omap_total_kv_pairs": 0, 2024-04-23T10:41:31.625 INFO:teuthology.orchestra.run.smithi119.stdout: "omap_total_updates": 0, 2024-04-23T10:41:31.625 INFO:teuthology.orchestra.run.smithi119.stdout: "omap_total_removes": 0 2024-04-23T10:41:31.625 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-04-23T10:41:31.625 INFO:teuthology.orchestra.run.smithi119.stdout: "osd-slow-ops": { 2024-04-23T10:41:31.625 INFO:teuthology.orchestra.run.smithi119.stdout: "slow_ops_count": 0 2024-04-23T10:41:31.625 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-04-23T10:41:31.625 INFO:teuthology.orchestra.run.smithi119.stdout: "purge_queue": { 2024-04-23T10:41:31.625 INFO:teuthology.orchestra.run.smithi119.stdout: "pq_executing_ops": 0, 2024-04-23T10:41:31.626 INFO:teuthology.orchestra.run.smithi119.stdout: "pq_executing_ops_high_water": 0, 2024-04-23T10:41:31.626 INFO:teuthology.orchestra.run.smithi119.stdout: "pq_executing": 0, 2024-04-23T10:41:31.626 INFO:teuthology.orchestra.run.smithi119.stdout: "pq_executing_high_water": 0, 2024-04-23T10:41:31.626 INFO:teuthology.orchestra.run.smithi119.stdout: "pq_executed_ops": 0, 2024-04-23T10:41:31.626 INFO:teuthology.orchestra.run.smithi119.stdout: "pq_executed": 0, 2024-04-23T10:41:31.626 INFO:teuthology.orchestra.run.smithi119.stdout: "pq_item_in_journal": 0 2024-04-23T10:41:31.626 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-04-23T10:41:31.627 INFO:teuthology.orchestra.run.smithi119.stdout: "throttle-msgr_dispatch_throttler-mds": { 2024-04-23T10:41:31.627 INFO:teuthology.orchestra.run.smithi119.stdout: "val": 0, 2024-04-23T10:41:31.627 INFO:teuthology.orchestra.run.smithi119.stdout: "max": 104857600, 2024-04-23T10:41:31.627 INFO:teuthology.orchestra.run.smithi119.stdout: "get_started": 0, 2024-04-23T10:41:31.627 INFO:teuthology.orchestra.run.smithi119.stdout: "get": 3816, 2024-04-23T10:41:31.627 INFO:teuthology.orchestra.run.smithi119.stdout: "get_sum": 119819889, 2024-04-23T10:41:31.627 INFO:teuthology.orchestra.run.smithi119.stdout: "get_or_fail_fail": 0, 2024-04-23T10:41:31.627 INFO:teuthology.orchestra.run.smithi119.stdout: "get_or_fail_success": 3816, 2024-04-23T10:41:31.628 INFO:teuthology.orchestra.run.smithi119.stdout: "take": 0, 2024-04-23T10:41:31.628 INFO:teuthology.orchestra.run.smithi119.stdout: "take_sum": 0, 2024-04-23T10:41:31.628 INFO:teuthology.orchestra.run.smithi119.stdout: "put": 3816, 2024-04-23T10:41:31.628 INFO:teuthology.orchestra.run.smithi119.stdout: "put_sum": 119819889, 2024-04-23T10:41:31.628 INFO:teuthology.orchestra.run.smithi119.stdout: "wait": { 2024-04-23T10:41:31.628 INFO:teuthology.orchestra.run.smithi119.stdout: "avgcount": 0, 2024-04-23T10:41:31.628 INFO:teuthology.orchestra.run.smithi119.stdout: "sum": 0.000000000, 2024-04-23T10:41:31.628 INFO:teuthology.orchestra.run.smithi119.stdout: "avgtime": 0.000000000 2024-04-23T10:41:31.628 INFO:teuthology.orchestra.run.smithi119.stdout: } 2024-04-23T10:41:31.629 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-04-23T10:41:31.629 INFO:teuthology.orchestra.run.smithi119.stdout: "throttle-objecter_bytes": { 2024-04-23T10:41:31.629 INFO:teuthology.orchestra.run.smithi119.stdout: "val": 0, 2024-04-23T10:41:31.629 INFO:teuthology.orchestra.run.smithi119.stdout: "max": 104857600, 2024-04-23T10:41:31.629 INFO:teuthology.orchestra.run.smithi119.stdout: "get_started": 0, 2024-04-23T10:41:31.629 INFO:teuthology.orchestra.run.smithi119.stdout: "get": 0, 2024-04-23T10:41:31.629 INFO:teuthology.orchestra.run.smithi119.stdout: "get_sum": 0, 2024-04-23T10:41:31.629 INFO:teuthology.orchestra.run.smithi119.stdout: "get_or_fail_fail": 0, 2024-04-23T10:41:31.630 INFO:teuthology.orchestra.run.smithi119.stdout: "get_or_fail_success": 0, 2024-04-23T10:41:31.630 INFO:teuthology.orchestra.run.smithi119.stdout: "take": 3462, 2024-04-23T10:41:31.630 INFO:teuthology.orchestra.run.smithi119.stdout: "take_sum": 117457099, 2024-04-23T10:41:31.630 INFO:teuthology.orchestra.run.smithi119.stdout: "put": 3462, 2024-04-23T10:41:31.630 INFO:teuthology.orchestra.run.smithi119.stdout: "put_sum": 117457099, 2024-04-23T10:41:31.630 INFO:teuthology.orchestra.run.smithi119.stdout: "wait": { 2024-04-23T10:41:31.630 INFO:teuthology.orchestra.run.smithi119.stdout: "avgcount": 0, 2024-04-23T10:41:31.631 INFO:teuthology.orchestra.run.smithi119.stdout: "sum": 0.000000000, 2024-04-23T10:41:31.631 INFO:teuthology.orchestra.run.smithi119.stdout: "avgtime": 0.000000000 2024-04-23T10:41:31.631 INFO:teuthology.orchestra.run.smithi119.stdout: } 2024-04-23T10:41:31.631 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-04-23T10:41:31.631 INFO:teuthology.orchestra.run.smithi119.stdout: "throttle-objecter_ops": { 2024-04-23T10:41:31.631 INFO:teuthology.orchestra.run.smithi119.stdout: "val": 0, 2024-04-23T10:41:31.631 INFO:teuthology.orchestra.run.smithi119.stdout: "max": 1024, 2024-04-23T10:41:31.631 INFO:teuthology.orchestra.run.smithi119.stdout: "get_started": 0, 2024-04-23T10:41:31.632 INFO:teuthology.orchestra.run.smithi119.stdout: "get": 0, 2024-04-23T10:41:31.632 INFO:teuthology.orchestra.run.smithi119.stdout: "get_sum": 0, 2024-04-23T10:41:31.632 INFO:teuthology.orchestra.run.smithi119.stdout: "get_or_fail_fail": 0, 2024-04-23T10:41:31.632 INFO:teuthology.orchestra.run.smithi119.stdout: "get_or_fail_success": 0, 2024-04-23T10:41:31.632 INFO:teuthology.orchestra.run.smithi119.stdout: "take": 3462, 2024-04-23T10:41:31.632 INFO:teuthology.orchestra.run.smithi119.stdout: "take_sum": 3462, 2024-04-23T10:41:31.632 INFO:teuthology.orchestra.run.smithi119.stdout: "put": 3462, 2024-04-23T10:41:31.632 INFO:teuthology.orchestra.run.smithi119.stdout: "put_sum": 3462, 2024-04-23T10:41:31.633 INFO:teuthology.orchestra.run.smithi119.stdout: "wait": { 2024-04-23T10:41:31.633 INFO:teuthology.orchestra.run.smithi119.stdout: "avgcount": 0, 2024-04-23T10:41:31.633 INFO:teuthology.orchestra.run.smithi119.stdout: "sum": 0.000000000, 2024-04-23T10:41:31.633 INFO:teuthology.orchestra.run.smithi119.stdout: "avgtime": 0.000000000 2024-04-23T10:41:31.633 INFO:teuthology.orchestra.run.smithi119.stdout: } 2024-04-23T10:41:31.633 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-04-23T10:41:31.633 INFO:teuthology.orchestra.run.smithi119.stdout: "throttle-write_buf_throttle": { 2024-04-23T10:41:31.633 INFO:teuthology.orchestra.run.smithi119.stdout: "val": 0, 2024-04-23T10:41:31.633 INFO:teuthology.orchestra.run.smithi119.stdout: "max": 3758096384, 2024-04-23T10:41:31.634 INFO:teuthology.orchestra.run.smithi119.stdout: "get_started": 0, 2024-04-23T10:41:31.634 INFO:teuthology.orchestra.run.smithi119.stdout: "get": 0, 2024-04-23T10:41:31.634 INFO:teuthology.orchestra.run.smithi119.stdout: "get_sum": 0, 2024-04-23T10:41:31.634 INFO:teuthology.orchestra.run.smithi119.stdout: "get_or_fail_fail": 0, 2024-04-23T10:41:31.634 INFO:teuthology.orchestra.run.smithi119.stdout: "get_or_fail_success": 0, 2024-04-23T10:41:31.634 INFO:teuthology.orchestra.run.smithi119.stdout: "take": 0, 2024-04-23T10:41:31.634 INFO:teuthology.orchestra.run.smithi119.stdout: "take_sum": 0, 2024-04-23T10:41:31.634 INFO:teuthology.orchestra.run.smithi119.stdout: "put": 0, 2024-04-23T10:41:31.635 INFO:teuthology.orchestra.run.smithi119.stdout: "put_sum": 0, 2024-04-23T10:41:31.635 INFO:teuthology.orchestra.run.smithi119.stdout: "wait": { 2024-04-23T10:41:31.635 INFO:teuthology.orchestra.run.smithi119.stdout: "avgcount": 0, 2024-04-23T10:41:31.635 INFO:teuthology.orchestra.run.smithi119.stdout: "sum": 0.000000000, 2024-04-23T10:41:31.635 INFO:teuthology.orchestra.run.smithi119.stdout: "avgtime": 0.000000000 2024-04-23T10:41:31.635 INFO:teuthology.orchestra.run.smithi119.stdout: } 2024-04-23T10:41:31.635 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-04-23T10:41:31.635 INFO:teuthology.orchestra.run.smithi119.stdout: "throttle-write_buf_throttle-0x5611774146e0": { 2024-04-23T10:41:31.636 INFO:teuthology.orchestra.run.smithi119.stdout: "val": 0, 2024-04-23T10:41:31.636 INFO:teuthology.orchestra.run.smithi119.stdout: "max": 3758096384, 2024-04-23T10:41:31.636 INFO:teuthology.orchestra.run.smithi119.stdout: "get_started": 0, 2024-04-23T10:41:31.636 INFO:teuthology.orchestra.run.smithi119.stdout: "get": 0, 2024-04-23T10:41:31.636 INFO:teuthology.orchestra.run.smithi119.stdout: "get_sum": 0, 2024-04-23T10:41:31.636 INFO:teuthology.orchestra.run.smithi119.stdout: "get_or_fail_fail": 0, 2024-04-23T10:41:31.636 INFO:teuthology.orchestra.run.smithi119.stdout: "get_or_fail_success": 0, 2024-04-23T10:41:31.636 INFO:teuthology.orchestra.run.smithi119.stdout: "take": 0, 2024-04-23T10:41:31.636 INFO:teuthology.orchestra.run.smithi119.stdout: "take_sum": 0, 2024-04-23T10:41:31.637 INFO:teuthology.orchestra.run.smithi119.stdout: "put": 0, 2024-04-23T10:41:31.637 INFO:teuthology.orchestra.run.smithi119.stdout: "put_sum": 0, 2024-04-23T10:41:31.637 INFO:teuthology.orchestra.run.smithi119.stdout: "wait": { 2024-04-23T10:41:31.637 INFO:teuthology.orchestra.run.smithi119.stdout: "avgcount": 0, 2024-04-23T10:41:31.637 INFO:teuthology.orchestra.run.smithi119.stdout: "sum": 0.000000000, 2024-04-23T10:41:31.637 INFO:teuthology.orchestra.run.smithi119.stdout: "avgtime": 0.000000000 2024-04-23T10:41:31.637 INFO:teuthology.orchestra.run.smithi119.stdout: } 2024-04-23T10:41:31.637 INFO:teuthology.orchestra.run.smithi119.stdout: } 2024-04-23T10:41:31.638 INFO:teuthology.orchestra.run.smithi119.stdout:} 2024-04-23T10:41:31.754 INFO:journalctl@ceph.mon.a.smithi114.stdout:Apr 23 10:41:31 smithi114 ceph-mon[29255]: pgmap v471: 97 pgs: 97 active+clean; 560 MiB data, 4.3 GiB used, 1.0 TiB / 1.0 TiB avail; 523 KiB/s rd, 4.7 MiB/s wr, 167 op/s 2024-04-23T10:41:31.828 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 23 10:41:31 smithi204 ceph-mon[32649]: pgmap v471: 97 pgs: 97 active+clean; 560 MiB data, 4.3 GiB used, 1.0 TiB / 1.0 TiB avail; 523 KiB/s rd, 4.7 MiB/s wr, 167 op/s 2024-04-23T10:41:32.449 INFO:tasks.check_counter:Daemon mds.k mds.exported=0 2024-04-23T10:41:32.450 INFO:tasks.check_counter:Daemon mds.k mds.imported=0 2024-04-23T10:41:32.451 INFO:tasks.check_counter:Daemon mds.k mds_cache.dir_update=0 2024-04-23T10:41:32.451 INFO:tasks.check_counter:Daemon mds.k mds_cache.dir_update_receipt=0 2024-04-23T10:41:32.451 INFO:tasks.check_counter:Daemon mds.k mds.root_rsnaps=0 2024-04-23T10:41:32.451 INFO:tasks.check_counter:Daemon mds.k mds_server.req_mksnap_latency.avgcount=0 2024-04-23T10:41:32.451 INFO:tasks.check_counter:Daemon mds.k mds_server.req_rmsnap_latency.avgcount=0 2024-04-23T10:41:32.451 DEBUG:tasks.check_counter:Getting stats from c 2024-04-23T10:41:32.452 DEBUG:teuthology.orchestra.run.smithi204:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d69f13d827952bb07bb9b7973bca14f97d764d6f shell --fsid aa240572-0159-11ef-bc93-c7b262605968 -- ceph daemon mds.c perf dump 2024-04-23T10:41:32.714 INFO:teuthology.orchestra.run.smithi204.stderr:Inferring config /var/lib/ceph/aa240572-0159-11ef-bc93-c7b262605968/mon.c/config 2024-04-23T10:41:33.578 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 23 10:41:33 smithi204 ceph-mon[32649]: pgmap v472: 97 pgs: 97 active+clean; 558 MiB data, 4.3 GiB used, 1.0 TiB / 1.0 TiB avail; 775 KiB/s rd, 7.1 MiB/s wr, 243 op/s 2024-04-23T10:41:33.991 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 23 10:41:33 smithi119 ceph-mon[32608]: pgmap v472: 97 pgs: 97 active+clean; 558 MiB data, 4.3 GiB used, 1.0 TiB / 1.0 TiB avail; 775 KiB/s rd, 7.1 MiB/s wr, 243 op/s 2024-04-23T10:41:34.003 INFO:journalctl@ceph.mon.a.smithi114.stdout:Apr 23 10:41:33 smithi114 ceph-mon[29255]: pgmap v472: 97 pgs: 97 active+clean; 558 MiB data, 4.3 GiB used, 1.0 TiB / 1.0 TiB avail; 775 KiB/s rd, 7.1 MiB/s wr, 243 op/s 2024-04-23T10:41:35.222 INFO:teuthology.orchestra.run.smithi204.stdout:{ 2024-04-23T10:41:35.223 INFO:teuthology.orchestra.run.smithi204.stdout: "AsyncMessenger::Worker-0": { 2024-04-23T10:41:35.223 INFO:teuthology.orchestra.run.smithi204.stdout: "msgr_recv_messages": 1056, 2024-04-23T10:41:35.223 INFO:teuthology.orchestra.run.smithi204.stdout: "msgr_send_messages": 1014, 2024-04-23T10:41:35.223 INFO:teuthology.orchestra.run.smithi204.stdout: "msgr_recv_bytes": 46762980, 2024-04-23T10:41:35.223 INFO:teuthology.orchestra.run.smithi204.stdout: "msgr_send_bytes": 422718, 2024-04-23T10:41:35.223 INFO:teuthology.orchestra.run.smithi204.stdout: "msgr_created_connections": 9, 2024-04-23T10:41:35.224 INFO:teuthology.orchestra.run.smithi204.stdout: "msgr_active_connections": 3, 2024-04-23T10:41:35.224 INFO:teuthology.orchestra.run.smithi204.stdout: "msgr_running_total_time": 0.129470044, 2024-04-23T10:41:35.224 INFO:teuthology.orchestra.run.smithi204.stdout: "msgr_running_send_time": 0.033080104, 2024-04-23T10:41:35.224 INFO:teuthology.orchestra.run.smithi204.stdout: "msgr_running_recv_time": 0.077863919, 2024-04-23T10:41:35.224 INFO:teuthology.orchestra.run.smithi204.stdout: "msgr_running_fast_dispatch_time": 0.023022114, 2024-04-23T10:41:35.224 INFO:teuthology.orchestra.run.smithi204.stdout: "msgr_send_messages_queue_lat": { 2024-04-23T10:41:35.224 INFO:teuthology.orchestra.run.smithi204.stdout: "avgcount": 1014, 2024-04-23T10:41:35.225 INFO:teuthology.orchestra.run.smithi204.stdout: "sum": 0.038528912, 2024-04-23T10:41:35.225 INFO:teuthology.orchestra.run.smithi204.stdout: "avgtime": 0.000037996 2024-04-23T10:41:35.225 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-04-23T10:41:35.225 INFO:teuthology.orchestra.run.smithi204.stdout: "msgr_handle_ack_lat": { 2024-04-23T10:41:35.225 INFO:teuthology.orchestra.run.smithi204.stdout: "avgcount": 2, 2024-04-23T10:41:35.225 INFO:teuthology.orchestra.run.smithi204.stdout: "sum": 0.000000171, 2024-04-23T10:41:35.225 INFO:teuthology.orchestra.run.smithi204.stdout: "avgtime": 0.000000085 2024-04-23T10:41:35.226 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-04-23T10:41:35.226 INFO:teuthology.orchestra.run.smithi204.stdout: "msgr_recv_encrypted_bytes": 1764672, 2024-04-23T10:41:35.226 INFO:teuthology.orchestra.run.smithi204.stdout: "msgr_send_encrypted_bytes": 175680 2024-04-23T10:41:35.226 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-04-23T10:41:35.226 INFO:teuthology.orchestra.run.smithi204.stdout: "AsyncMessenger::Worker-1": { 2024-04-23T10:41:35.226 INFO:teuthology.orchestra.run.smithi204.stdout: "msgr_recv_messages": 807, 2024-04-23T10:41:35.226 INFO:teuthology.orchestra.run.smithi204.stdout: "msgr_send_messages": 1015, 2024-04-23T10:41:35.227 INFO:teuthology.orchestra.run.smithi204.stdout: "msgr_recv_bytes": 46348324, 2024-04-23T10:41:35.227 INFO:teuthology.orchestra.run.smithi204.stdout: "msgr_send_bytes": 790795, 2024-04-23T10:41:35.227 INFO:teuthology.orchestra.run.smithi204.stdout: "msgr_created_connections": 8, 2024-04-23T10:41:35.227 INFO:teuthology.orchestra.run.smithi204.stdout: "msgr_active_connections": 4, 2024-04-23T10:41:35.227 INFO:teuthology.orchestra.run.smithi204.stdout: "msgr_running_total_time": 0.090938768, 2024-04-23T10:41:35.227 INFO:teuthology.orchestra.run.smithi204.stdout: "msgr_running_send_time": 0.027291306, 2024-04-23T10:41:35.227 INFO:teuthology.orchestra.run.smithi204.stdout: "msgr_running_recv_time": 0.037109735, 2024-04-23T10:41:35.227 INFO:teuthology.orchestra.run.smithi204.stdout: "msgr_running_fast_dispatch_time": 0.019055470, 2024-04-23T10:41:35.228 INFO:teuthology.orchestra.run.smithi204.stdout: "msgr_send_messages_queue_lat": { 2024-04-23T10:41:35.228 INFO:teuthology.orchestra.run.smithi204.stdout: "avgcount": 1012, 2024-04-23T10:41:35.228 INFO:teuthology.orchestra.run.smithi204.stdout: "sum": 0.038109665, 2024-04-23T10:41:35.228 INFO:teuthology.orchestra.run.smithi204.stdout: "avgtime": 0.000037657 2024-04-23T10:41:35.228 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-04-23T10:41:35.228 INFO:teuthology.orchestra.run.smithi204.stdout: "msgr_handle_ack_lat": { 2024-04-23T10:41:35.228 INFO:teuthology.orchestra.run.smithi204.stdout: "avgcount": 0, 2024-04-23T10:41:35.229 INFO:teuthology.orchestra.run.smithi204.stdout: "sum": 0.000000000, 2024-04-23T10:41:35.229 INFO:teuthology.orchestra.run.smithi204.stdout: "avgtime": 0.000000000 2024-04-23T10:41:35.229 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-04-23T10:41:35.229 INFO:teuthology.orchestra.run.smithi204.stdout: "msgr_recv_encrypted_bytes": 320, 2024-04-23T10:41:35.229 INFO:teuthology.orchestra.run.smithi204.stdout: "msgr_send_encrypted_bytes": 534800 2024-04-23T10:41:35.229 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-04-23T10:41:35.229 INFO:teuthology.orchestra.run.smithi204.stdout: "AsyncMessenger::Worker-2": { 2024-04-23T10:41:35.230 INFO:teuthology.orchestra.run.smithi204.stdout: "msgr_recv_messages": 2018, 2024-04-23T10:41:35.230 INFO:teuthology.orchestra.run.smithi204.stdout: "msgr_send_messages": 2020, 2024-04-23T10:41:35.230 INFO:teuthology.orchestra.run.smithi204.stdout: "msgr_recv_bytes": 76095049, 2024-04-23T10:41:35.230 INFO:teuthology.orchestra.run.smithi204.stdout: "msgr_send_bytes": 641782, 2024-04-23T10:41:35.230 INFO:teuthology.orchestra.run.smithi204.stdout: "msgr_created_connections": 6, 2024-04-23T10:41:35.230 INFO:teuthology.orchestra.run.smithi204.stdout: "msgr_active_connections": 4, 2024-04-23T10:41:35.230 INFO:teuthology.orchestra.run.smithi204.stdout: "msgr_running_total_time": 0.214827698, 2024-04-23T10:41:35.231 INFO:teuthology.orchestra.run.smithi204.stdout: "msgr_running_send_time": 0.056114226, 2024-04-23T10:41:35.231 INFO:teuthology.orchestra.run.smithi204.stdout: "msgr_running_recv_time": 0.084874060, 2024-04-23T10:41:35.231 INFO:teuthology.orchestra.run.smithi204.stdout: "msgr_running_fast_dispatch_time": 0.057915299, 2024-04-23T10:41:35.231 INFO:teuthology.orchestra.run.smithi204.stdout: "msgr_send_messages_queue_lat": { 2024-04-23T10:41:35.231 INFO:teuthology.orchestra.run.smithi204.stdout: "avgcount": 2018, 2024-04-23T10:41:35.231 INFO:teuthology.orchestra.run.smithi204.stdout: "sum": 0.075453499, 2024-04-23T10:41:35.231 INFO:teuthology.orchestra.run.smithi204.stdout: "avgtime": 0.000037390 2024-04-23T10:41:35.232 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-04-23T10:41:35.232 INFO:teuthology.orchestra.run.smithi204.stdout: "msgr_handle_ack_lat": { 2024-04-23T10:41:35.232 INFO:teuthology.orchestra.run.smithi204.stdout: "avgcount": 0, 2024-04-23T10:41:35.232 INFO:teuthology.orchestra.run.smithi204.stdout: "sum": 0.000000000, 2024-04-23T10:41:35.232 INFO:teuthology.orchestra.run.smithi204.stdout: "avgtime": 0.000000000 2024-04-23T10:41:35.232 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-04-23T10:41:35.232 INFO:teuthology.orchestra.run.smithi204.stdout: "msgr_recv_encrypted_bytes": 0, 2024-04-23T10:41:35.233 INFO:teuthology.orchestra.run.smithi204.stdout: "msgr_send_encrypted_bytes": 0 2024-04-23T10:41:35.233 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-04-23T10:41:35.233 INFO:teuthology.orchestra.run.smithi204.stdout: "cct": { 2024-04-23T10:41:35.233 INFO:teuthology.orchestra.run.smithi204.stdout: "total_workers": 1, 2024-04-23T10:41:35.233 INFO:teuthology.orchestra.run.smithi204.stdout: "unhealthy_workers": 0 2024-04-23T10:41:35.233 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-04-23T10:41:35.233 INFO:teuthology.orchestra.run.smithi204.stdout: "finisher-MDSRank": { 2024-04-23T10:41:35.234 INFO:teuthology.orchestra.run.smithi204.stdout: "queue_len": 0, 2024-04-23T10:41:35.234 INFO:teuthology.orchestra.run.smithi204.stdout: "complete_latency": { 2024-04-23T10:41:35.234 INFO:teuthology.orchestra.run.smithi204.stdout: "avgcount": 5965, 2024-04-23T10:41:35.234 INFO:teuthology.orchestra.run.smithi204.stdout: "sum": 0.140460970, 2024-04-23T10:41:35.234 INFO:teuthology.orchestra.run.smithi204.stdout: "avgtime": 0.000023547 2024-04-23T10:41:35.234 INFO:teuthology.orchestra.run.smithi204.stdout: } 2024-04-23T10:41:35.234 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-04-23T10:41:35.235 INFO:teuthology.orchestra.run.smithi204.stdout: "finisher-PurgeQueue": { 2024-04-23T10:41:35.235 INFO:teuthology.orchestra.run.smithi204.stdout: "queue_len": 0, 2024-04-23T10:41:35.235 INFO:teuthology.orchestra.run.smithi204.stdout: "complete_latency": { 2024-04-23T10:41:35.235 INFO:teuthology.orchestra.run.smithi204.stdout: "avgcount": 0, 2024-04-23T10:41:35.235 INFO:teuthology.orchestra.run.smithi204.stdout: "sum": 0.000000000, 2024-04-23T10:41:35.235 INFO:teuthology.orchestra.run.smithi204.stdout: "avgtime": 0.000000000 2024-04-23T10:41:35.235 INFO:teuthology.orchestra.run.smithi204.stdout: } 2024-04-23T10:41:35.236 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-04-23T10:41:35.236 INFO:teuthology.orchestra.run.smithi204.stdout: "mds": { 2024-04-23T10:41:35.236 INFO:teuthology.orchestra.run.smithi204.stdout: "request": 0, 2024-04-23T10:41:35.236 INFO:teuthology.orchestra.run.smithi204.stdout: "reply": 0, 2024-04-23T10:41:35.236 INFO:teuthology.orchestra.run.smithi204.stdout: "reply_latency": { 2024-04-23T10:41:35.236 INFO:teuthology.orchestra.run.smithi204.stdout: "avgcount": 0, 2024-04-23T10:41:35.236 INFO:teuthology.orchestra.run.smithi204.stdout: "sum": 0.000000000, 2024-04-23T10:41:35.236 INFO:teuthology.orchestra.run.smithi204.stdout: "avgtime": 0.000000000 2024-04-23T10:41:35.237 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-04-23T10:41:35.237 INFO:teuthology.orchestra.run.smithi204.stdout: "slow_reply": 0, 2024-04-23T10:41:35.237 INFO:teuthology.orchestra.run.smithi204.stdout: "forward": 0, 2024-04-23T10:41:35.237 INFO:teuthology.orchestra.run.smithi204.stdout: "dir_fetch_complete": 0, 2024-04-23T10:41:35.237 INFO:teuthology.orchestra.run.smithi204.stdout: "dir_fetch_keys": 0, 2024-04-23T10:41:35.237 INFO:teuthology.orchestra.run.smithi204.stdout: "dir_commit": 0, 2024-04-23T10:41:35.237 INFO:teuthology.orchestra.run.smithi204.stdout: "dir_split": 0, 2024-04-23T10:41:35.238 INFO:teuthology.orchestra.run.smithi204.stdout: "dir_merge": 0, 2024-04-23T10:41:35.238 INFO:teuthology.orchestra.run.smithi204.stdout: "inodes": 15167, 2024-04-23T10:41:35.238 INFO:teuthology.orchestra.run.smithi204.stdout: "inodes_top": 91, 2024-04-23T10:41:35.238 INFO:teuthology.orchestra.run.smithi204.stdout: "inodes_bottom": 15076, 2024-04-23T10:41:35.238 INFO:teuthology.orchestra.run.smithi204.stdout: "inodes_pin_tail": 0, 2024-04-23T10:41:35.238 INFO:teuthology.orchestra.run.smithi204.stdout: "inodes_pinned": 15038, 2024-04-23T10:41:35.238 INFO:teuthology.orchestra.run.smithi204.stdout: "inodes_expired": 0, 2024-04-23T10:41:35.239 INFO:teuthology.orchestra.run.smithi204.stdout: "inodes_with_caps": 0, 2024-04-23T10:41:35.239 INFO:teuthology.orchestra.run.smithi204.stdout: "caps": 0, 2024-04-23T10:41:35.239 INFO:teuthology.orchestra.run.smithi204.stdout: "subtrees": 9, 2024-04-23T10:41:35.239 INFO:teuthology.orchestra.run.smithi204.stdout: "traverse": 0, 2024-04-23T10:41:35.239 INFO:teuthology.orchestra.run.smithi204.stdout: "traverse_hit": 0, 2024-04-23T10:41:35.239 INFO:teuthology.orchestra.run.smithi204.stdout: "traverse_forward": 0, 2024-04-23T10:41:35.239 INFO:teuthology.orchestra.run.smithi204.stdout: "traverse_discover": 0, 2024-04-23T10:41:35.239 INFO:teuthology.orchestra.run.smithi204.stdout: "traverse_dir_fetch": 0, 2024-04-23T10:41:35.240 INFO:teuthology.orchestra.run.smithi204.stdout: "traverse_remote_ino": 0, 2024-04-23T10:41:35.240 INFO:teuthology.orchestra.run.smithi204.stdout: "traverse_lock": 0, 2024-04-23T10:41:35.240 INFO:teuthology.orchestra.run.smithi204.stdout: "load_cent": 0, 2024-04-23T10:41:35.240 INFO:teuthology.orchestra.run.smithi204.stdout: "q": 0, 2024-04-23T10:41:35.240 INFO:teuthology.orchestra.run.smithi204.stdout: "exported": 0, 2024-04-23T10:41:35.240 INFO:teuthology.orchestra.run.smithi204.stdout: "exported_inodes": 0, 2024-04-23T10:41:35.240 INFO:teuthology.orchestra.run.smithi204.stdout: "imported": 0, 2024-04-23T10:41:35.241 INFO:teuthology.orchestra.run.smithi204.stdout: "imported_inodes": 0, 2024-04-23T10:41:35.241 INFO:teuthology.orchestra.run.smithi204.stdout: "openino_dir_fetch": 0, 2024-04-23T10:41:35.241 INFO:teuthology.orchestra.run.smithi204.stdout: "openino_backtrace_fetch": 0, 2024-04-23T10:41:35.241 INFO:teuthology.orchestra.run.smithi204.stdout: "openino_peer_discover": 0, 2024-04-23T10:41:35.241 INFO:teuthology.orchestra.run.smithi204.stdout: "root_rfiles": 0, 2024-04-23T10:41:35.241 INFO:teuthology.orchestra.run.smithi204.stdout: "root_rbytes": 0, 2024-04-23T10:41:35.242 INFO:teuthology.orchestra.run.smithi204.stdout: "root_rsnaps": 0, 2024-04-23T10:41:35.242 INFO:teuthology.orchestra.run.smithi204.stdout: "scrub_backtrace_fetch": 0, 2024-04-23T10:41:35.242 INFO:teuthology.orchestra.run.smithi204.stdout: "scrub_set_tag": 0, 2024-04-23T10:41:35.242 INFO:teuthology.orchestra.run.smithi204.stdout: "scrub_backtrace_repaired": 0, 2024-04-23T10:41:35.242 INFO:teuthology.orchestra.run.smithi204.stdout: "scrub_inotable_repaired": 0, 2024-04-23T10:41:35.242 INFO:teuthology.orchestra.run.smithi204.stdout: "scrub_dir_inodes": 0, 2024-04-23T10:41:35.242 INFO:teuthology.orchestra.run.smithi204.stdout: "scrub_dir_base_inodes": 0, 2024-04-23T10:41:35.243 INFO:teuthology.orchestra.run.smithi204.stdout: "scrub_dirfrag_rstats": 0, 2024-04-23T10:41:35.243 INFO:teuthology.orchestra.run.smithi204.stdout: "scrub_file_inodes": 0, 2024-04-23T10:41:35.243 INFO:teuthology.orchestra.run.smithi204.stdout: "handle_inode_file_caps": 0, 2024-04-23T10:41:35.243 INFO:teuthology.orchestra.run.smithi204.stdout: "ceph_cap_op_revoke": 0, 2024-04-23T10:41:35.243 INFO:teuthology.orchestra.run.smithi204.stdout: "ceph_cap_op_grant": 0, 2024-04-23T10:41:35.243 INFO:teuthology.orchestra.run.smithi204.stdout: "ceph_cap_op_trunc": 0, 2024-04-23T10:41:35.243 INFO:teuthology.orchestra.run.smithi204.stdout: "ceph_cap_op_flushsnap_ack": 0, 2024-04-23T10:41:35.244 INFO:teuthology.orchestra.run.smithi204.stdout: "ceph_cap_op_flush_ack": 0, 2024-04-23T10:41:35.244 INFO:teuthology.orchestra.run.smithi204.stdout: "handle_client_caps": 0, 2024-04-23T10:41:35.244 INFO:teuthology.orchestra.run.smithi204.stdout: "handle_client_caps_dirty": 0, 2024-04-23T10:41:35.244 INFO:teuthology.orchestra.run.smithi204.stdout: "handle_client_cap_release": 0, 2024-04-23T10:41:35.244 INFO:teuthology.orchestra.run.smithi204.stdout: "process_request_cap_release": 0 2024-04-23T10:41:35.244 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-04-23T10:41:35.244 INFO:teuthology.orchestra.run.smithi204.stdout: "mds_cache": { 2024-04-23T10:41:35.244 INFO:teuthology.orchestra.run.smithi204.stdout: "dir_update": 0, 2024-04-23T10:41:35.245 INFO:teuthology.orchestra.run.smithi204.stdout: "dir_update_receipt": 0, 2024-04-23T10:41:35.245 INFO:teuthology.orchestra.run.smithi204.stdout: "dir_try_discover": 0, 2024-04-23T10:41:35.245 INFO:teuthology.orchestra.run.smithi204.stdout: "dir_send_discover": 0, 2024-04-23T10:41:35.245 INFO:teuthology.orchestra.run.smithi204.stdout: "dir_handle_discover": 0, 2024-04-23T10:41:35.245 INFO:teuthology.orchestra.run.smithi204.stdout: "num_strays": 0, 2024-04-23T10:41:35.245 INFO:teuthology.orchestra.run.smithi204.stdout: "num_strays_delayed": 0, 2024-04-23T10:41:35.245 INFO:teuthology.orchestra.run.smithi204.stdout: "num_strays_enqueuing": 0, 2024-04-23T10:41:35.245 INFO:teuthology.orchestra.run.smithi204.stdout: "strays_created": 0, 2024-04-23T10:41:35.246 INFO:teuthology.orchestra.run.smithi204.stdout: "strays_enqueued": 0, 2024-04-23T10:41:35.246 INFO:teuthology.orchestra.run.smithi204.stdout: "strays_reintegrated": 0, 2024-04-23T10:41:35.246 INFO:teuthology.orchestra.run.smithi204.stdout: "strays_migrated": 0, 2024-04-23T10:41:35.246 INFO:teuthology.orchestra.run.smithi204.stdout: "num_recovering_processing": 0, 2024-04-23T10:41:35.246 INFO:teuthology.orchestra.run.smithi204.stdout: "num_recovering_enqueued": 0, 2024-04-23T10:41:35.246 INFO:teuthology.orchestra.run.smithi204.stdout: "num_recovering_prioritized": 0, 2024-04-23T10:41:35.246 INFO:teuthology.orchestra.run.smithi204.stdout: "recovery_started": 0, 2024-04-23T10:41:35.247 INFO:teuthology.orchestra.run.smithi204.stdout: "recovery_completed": 0, 2024-04-23T10:41:35.247 INFO:teuthology.orchestra.run.smithi204.stdout: "ireq_quiesce_path": 0, 2024-04-23T10:41:35.247 INFO:teuthology.orchestra.run.smithi204.stdout: "ireq_quiesce_inode": 0, 2024-04-23T10:41:35.247 INFO:teuthology.orchestra.run.smithi204.stdout: "ireq_enqueue_scrub": 0, 2024-04-23T10:41:35.247 INFO:teuthology.orchestra.run.smithi204.stdout: "ireq_exportdir": 0, 2024-04-23T10:41:35.247 INFO:teuthology.orchestra.run.smithi204.stdout: "ireq_flush": 0, 2024-04-23T10:41:35.247 INFO:teuthology.orchestra.run.smithi204.stdout: "ireq_fragmentdir": 0, 2024-04-23T10:41:35.247 INFO:teuthology.orchestra.run.smithi204.stdout: "ireq_fragstats": 0, 2024-04-23T10:41:35.248 INFO:teuthology.orchestra.run.smithi204.stdout: "ireq_inodestats": 0 2024-04-23T10:41:35.248 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-04-23T10:41:35.248 INFO:teuthology.orchestra.run.smithi204.stdout: "mds_log": { 2024-04-23T10:41:35.248 INFO:teuthology.orchestra.run.smithi204.stdout: "evlrg": 0, 2024-04-23T10:41:35.248 INFO:teuthology.orchestra.run.smithi204.stdout: "evadd": 0, 2024-04-23T10:41:35.248 INFO:teuthology.orchestra.run.smithi204.stdout: "evex": 0, 2024-04-23T10:41:35.248 INFO:teuthology.orchestra.run.smithi204.stdout: "evtrm": 0, 2024-04-23T10:41:35.248 INFO:teuthology.orchestra.run.smithi204.stdout: "ev": 76741, 2024-04-23T10:41:35.249 INFO:teuthology.orchestra.run.smithi204.stdout: "evexg": 0, 2024-04-23T10:41:35.249 INFO:teuthology.orchestra.run.smithi204.stdout: "evexd": 0, 2024-04-23T10:41:35.249 INFO:teuthology.orchestra.run.smithi204.stdout: "segadd": 0, 2024-04-23T10:41:35.249 INFO:teuthology.orchestra.run.smithi204.stdout: "segex": 0, 2024-04-23T10:41:35.249 INFO:teuthology.orchestra.run.smithi204.stdout: "segtrm": 0, 2024-04-23T10:41:35.249 INFO:teuthology.orchestra.run.smithi204.stdout: "seg": 102, 2024-04-23T10:41:35.249 INFO:teuthology.orchestra.run.smithi204.stdout: "segmjr": 8, 2024-04-23T10:41:35.250 INFO:teuthology.orchestra.run.smithi204.stdout: "segexg": 0, 2024-04-23T10:41:35.250 INFO:teuthology.orchestra.run.smithi204.stdout: "segexd": 0, 2024-04-23T10:41:35.250 INFO:teuthology.orchestra.run.smithi204.stdout: "expos": 4194304, 2024-04-23T10:41:35.250 INFO:teuthology.orchestra.run.smithi204.stdout: "wrpos": 170647903, 2024-04-23T10:41:35.250 INFO:teuthology.orchestra.run.smithi204.stdout: "rdpos": 170645368, 2024-04-23T10:41:35.250 INFO:teuthology.orchestra.run.smithi204.stdout: "jlat": { 2024-04-23T10:41:35.250 INFO:teuthology.orchestra.run.smithi204.stdout: "avgcount": 0, 2024-04-23T10:41:35.250 INFO:teuthology.orchestra.run.smithi204.stdout: "sum": 0.000000000, 2024-04-23T10:41:35.251 INFO:teuthology.orchestra.run.smithi204.stdout: "avgtime": 0.000000000 2024-04-23T10:41:35.251 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-04-23T10:41:35.251 INFO:teuthology.orchestra.run.smithi204.stdout: "replayed": 76741 2024-04-23T10:41:35.251 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-04-23T10:41:35.251 INFO:teuthology.orchestra.run.smithi204.stdout: "mds_mem": { 2024-04-23T10:41:35.251 INFO:teuthology.orchestra.run.smithi204.stdout: "ino": 859, 2024-04-23T10:41:35.251 INFO:teuthology.orchestra.run.smithi204.stdout: "ino+": 15187, 2024-04-23T10:41:35.251 INFO:teuthology.orchestra.run.smithi204.stdout: "ino-": 14328, 2024-04-23T10:41:35.252 INFO:teuthology.orchestra.run.smithi204.stdout: "dir": 72, 2024-04-23T10:41:35.252 INFO:teuthology.orchestra.run.smithi204.stdout: "dir+": 84, 2024-04-23T10:41:35.252 INFO:teuthology.orchestra.run.smithi204.stdout: "dir-": 12, 2024-04-23T10:41:35.252 INFO:teuthology.orchestra.run.smithi204.stdout: "dn": 15210, 2024-04-23T10:41:35.252 INFO:teuthology.orchestra.run.smithi204.stdout: "dn+": 15391, 2024-04-23T10:41:35.252 INFO:teuthology.orchestra.run.smithi204.stdout: "dn-": 181, 2024-04-23T10:41:35.252 INFO:teuthology.orchestra.run.smithi204.stdout: "cap": 0, 2024-04-23T10:41:35.253 INFO:teuthology.orchestra.run.smithi204.stdout: "cap+": 0, 2024-04-23T10:41:35.253 INFO:teuthology.orchestra.run.smithi204.stdout: "cap-": 0, 2024-04-23T10:41:35.253 INFO:teuthology.orchestra.run.smithi204.stdout: "rss": 70668, 2024-04-23T10:41:35.253 INFO:teuthology.orchestra.run.smithi204.stdout: "heap": 215416 2024-04-23T10:41:35.253 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-04-23T10:41:35.253 INFO:teuthology.orchestra.run.smithi204.stdout: "mds_server": { 2024-04-23T10:41:35.253 INFO:teuthology.orchestra.run.smithi204.stdout: "dispatch_client_request": 0, 2024-04-23T10:41:35.253 INFO:teuthology.orchestra.run.smithi204.stdout: "dispatch_server_request": 0, 2024-04-23T10:41:35.254 INFO:teuthology.orchestra.run.smithi204.stdout: "handle_client_request": 0, 2024-04-23T10:41:35.254 INFO:teuthology.orchestra.run.smithi204.stdout: "handle_client_session": 0, 2024-04-23T10:41:35.254 INFO:teuthology.orchestra.run.smithi204.stdout: "handle_peer_request": 0, 2024-04-23T10:41:35.254 INFO:teuthology.orchestra.run.smithi204.stdout: "req_create_latency": { 2024-04-23T10:41:35.254 INFO:teuthology.orchestra.run.smithi204.stdout: "avgcount": 0, 2024-04-23T10:41:35.254 INFO:teuthology.orchestra.run.smithi204.stdout: "sum": 0.000000000, 2024-04-23T10:41:35.254 INFO:teuthology.orchestra.run.smithi204.stdout: "avgtime": 0.000000000 2024-04-23T10:41:35.255 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-04-23T10:41:35.255 INFO:teuthology.orchestra.run.smithi204.stdout: "req_getattr_latency": { 2024-04-23T10:41:35.255 INFO:teuthology.orchestra.run.smithi204.stdout: "avgcount": 0, 2024-04-23T10:41:35.255 INFO:teuthology.orchestra.run.smithi204.stdout: "sum": 0.000000000, 2024-04-23T10:41:35.255 INFO:teuthology.orchestra.run.smithi204.stdout: "avgtime": 0.000000000 2024-04-23T10:41:35.255 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-04-23T10:41:35.255 INFO:teuthology.orchestra.run.smithi204.stdout: "req_getfilelock_latency": { 2024-04-23T10:41:35.255 INFO:teuthology.orchestra.run.smithi204.stdout: "avgcount": 0, 2024-04-23T10:41:35.256 INFO:teuthology.orchestra.run.smithi204.stdout: "sum": 0.000000000, 2024-04-23T10:41:35.256 INFO:teuthology.orchestra.run.smithi204.stdout: "avgtime": 0.000000000 2024-04-23T10:41:35.256 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-04-23T10:41:35.256 INFO:teuthology.orchestra.run.smithi204.stdout: "req_link_latency": { 2024-04-23T10:41:35.256 INFO:teuthology.orchestra.run.smithi204.stdout: "avgcount": 0, 2024-04-23T10:41:35.256 INFO:teuthology.orchestra.run.smithi204.stdout: "sum": 0.000000000, 2024-04-23T10:41:35.256 INFO:teuthology.orchestra.run.smithi204.stdout: "avgtime": 0.000000000 2024-04-23T10:41:35.256 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-04-23T10:41:35.257 INFO:teuthology.orchestra.run.smithi204.stdout: "req_lookup_latency": { 2024-04-23T10:41:35.257 INFO:teuthology.orchestra.run.smithi204.stdout: "avgcount": 0, 2024-04-23T10:41:35.257 INFO:teuthology.orchestra.run.smithi204.stdout: "sum": 0.000000000, 2024-04-23T10:41:35.257 INFO:teuthology.orchestra.run.smithi204.stdout: "avgtime": 0.000000000 2024-04-23T10:41:35.257 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-04-23T10:41:35.257 INFO:teuthology.orchestra.run.smithi204.stdout: "req_lookuphash_latency": { 2024-04-23T10:41:35.257 INFO:teuthology.orchestra.run.smithi204.stdout: "avgcount": 0, 2024-04-23T10:41:35.257 INFO:teuthology.orchestra.run.smithi204.stdout: "sum": 0.000000000, 2024-04-23T10:41:35.258 INFO:teuthology.orchestra.run.smithi204.stdout: "avgtime": 0.000000000 2024-04-23T10:41:35.258 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-04-23T10:41:35.258 INFO:teuthology.orchestra.run.smithi204.stdout: "req_lookupino_latency": { 2024-04-23T10:41:35.258 INFO:teuthology.orchestra.run.smithi204.stdout: "avgcount": 0, 2024-04-23T10:41:35.258 INFO:teuthology.orchestra.run.smithi204.stdout: "sum": 0.000000000, 2024-04-23T10:41:35.258 INFO:teuthology.orchestra.run.smithi204.stdout: "avgtime": 0.000000000 2024-04-23T10:41:35.258 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-04-23T10:41:35.258 INFO:teuthology.orchestra.run.smithi204.stdout: "req_lookupname_latency": { 2024-04-23T10:41:35.259 INFO:teuthology.orchestra.run.smithi204.stdout: "avgcount": 0, 2024-04-23T10:41:35.259 INFO:teuthology.orchestra.run.smithi204.stdout: "sum": 0.000000000, 2024-04-23T10:41:35.259 INFO:teuthology.orchestra.run.smithi204.stdout: "avgtime": 0.000000000 2024-04-23T10:41:35.259 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-04-23T10:41:35.259 INFO:teuthology.orchestra.run.smithi204.stdout: "req_lookupparent_latency": { 2024-04-23T10:41:35.259 INFO:teuthology.orchestra.run.smithi204.stdout: "avgcount": 0, 2024-04-23T10:41:35.259 INFO:teuthology.orchestra.run.smithi204.stdout: "sum": 0.000000000, 2024-04-23T10:41:35.259 INFO:teuthology.orchestra.run.smithi204.stdout: "avgtime": 0.000000000 2024-04-23T10:41:35.260 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-04-23T10:41:35.260 INFO:teuthology.orchestra.run.smithi204.stdout: "req_lookupsnap_latency": { 2024-04-23T10:41:35.260 INFO:teuthology.orchestra.run.smithi204.stdout: "avgcount": 0, 2024-04-23T10:41:35.260 INFO:teuthology.orchestra.run.smithi204.stdout: "sum": 0.000000000, 2024-04-23T10:41:35.260 INFO:teuthology.orchestra.run.smithi204.stdout: "avgtime": 0.000000000 2024-04-23T10:41:35.260 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-04-23T10:41:35.260 INFO:teuthology.orchestra.run.smithi204.stdout: "req_lssnap_latency": { 2024-04-23T10:41:35.260 INFO:teuthology.orchestra.run.smithi204.stdout: "avgcount": 0, 2024-04-23T10:41:35.261 INFO:teuthology.orchestra.run.smithi204.stdout: "sum": 0.000000000, 2024-04-23T10:41:35.261 INFO:teuthology.orchestra.run.smithi204.stdout: "avgtime": 0.000000000 2024-04-23T10:41:35.261 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-04-23T10:41:35.261 INFO:teuthology.orchestra.run.smithi204.stdout: "req_mkdir_latency": { 2024-04-23T10:41:35.262 INFO:teuthology.orchestra.run.smithi204.stdout: "avgcount": 0, 2024-04-23T10:41:35.262 INFO:teuthology.orchestra.run.smithi204.stdout: "sum": 0.000000000, 2024-04-23T10:41:35.262 INFO:teuthology.orchestra.run.smithi204.stdout: "avgtime": 0.000000000 2024-04-23T10:41:35.262 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-04-23T10:41:35.262 INFO:teuthology.orchestra.run.smithi204.stdout: "req_mknod_latency": { 2024-04-23T10:41:35.262 INFO:teuthology.orchestra.run.smithi204.stdout: "avgcount": 0, 2024-04-23T10:41:35.262 INFO:teuthology.orchestra.run.smithi204.stdout: "sum": 0.000000000, 2024-04-23T10:41:35.263 INFO:teuthology.orchestra.run.smithi204.stdout: "avgtime": 0.000000000 2024-04-23T10:41:35.263 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-04-23T10:41:35.263 INFO:teuthology.orchestra.run.smithi204.stdout: "req_mksnap_latency": { 2024-04-23T10:41:35.263 INFO:teuthology.orchestra.run.smithi204.stdout: "avgcount": 0, 2024-04-23T10:41:35.263 INFO:teuthology.orchestra.run.smithi204.stdout: "sum": 0.000000000, 2024-04-23T10:41:35.263 INFO:teuthology.orchestra.run.smithi204.stdout: "avgtime": 0.000000000 2024-04-23T10:41:35.263 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-04-23T10:41:35.263 INFO:teuthology.orchestra.run.smithi204.stdout: "req_open_latency": { 2024-04-23T10:41:35.264 INFO:teuthology.orchestra.run.smithi204.stdout: "avgcount": 0, 2024-04-23T10:41:35.264 INFO:teuthology.orchestra.run.smithi204.stdout: "sum": 0.000000000, 2024-04-23T10:41:35.264 INFO:teuthology.orchestra.run.smithi204.stdout: "avgtime": 0.000000000 2024-04-23T10:41:35.264 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-04-23T10:41:35.264 INFO:teuthology.orchestra.run.smithi204.stdout: "req_readdir_latency": { 2024-04-23T10:41:35.264 INFO:teuthology.orchestra.run.smithi204.stdout: "avgcount": 0, 2024-04-23T10:41:35.264 INFO:teuthology.orchestra.run.smithi204.stdout: "sum": 0.000000000, 2024-04-23T10:41:35.264 INFO:teuthology.orchestra.run.smithi204.stdout: "avgtime": 0.000000000 2024-04-23T10:41:35.265 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-04-23T10:41:35.265 INFO:teuthology.orchestra.run.smithi204.stdout: "req_rename_latency": { 2024-04-23T10:41:35.265 INFO:teuthology.orchestra.run.smithi204.stdout: "avgcount": 0, 2024-04-23T10:41:35.265 INFO:teuthology.orchestra.run.smithi204.stdout: "sum": 0.000000000, 2024-04-23T10:41:35.265 INFO:teuthology.orchestra.run.smithi204.stdout: "avgtime": 0.000000000 2024-04-23T10:41:35.265 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-04-23T10:41:35.265 INFO:teuthology.orchestra.run.smithi204.stdout: "req_renamesnap_latency": { 2024-04-23T10:41:35.265 INFO:teuthology.orchestra.run.smithi204.stdout: "avgcount": 0, 2024-04-23T10:41:35.266 INFO:teuthology.orchestra.run.smithi204.stdout: "sum": 0.000000000, 2024-04-23T10:41:35.266 INFO:teuthology.orchestra.run.smithi204.stdout: "avgtime": 0.000000000 2024-04-23T10:41:35.266 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-04-23T10:41:35.266 INFO:teuthology.orchestra.run.smithi204.stdout: "req_snapdiff_latency": { 2024-04-23T10:41:35.266 INFO:teuthology.orchestra.run.smithi204.stdout: "avgcount": 0, 2024-04-23T10:41:35.266 INFO:teuthology.orchestra.run.smithi204.stdout: "sum": 0.000000000, 2024-04-23T10:41:35.266 INFO:teuthology.orchestra.run.smithi204.stdout: "avgtime": 0.000000000 2024-04-23T10:41:35.266 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-04-23T10:41:35.267 INFO:teuthology.orchestra.run.smithi204.stdout: "req_rmdir_latency": { 2024-04-23T10:41:35.267 INFO:teuthology.orchestra.run.smithi204.stdout: "avgcount": 0, 2024-04-23T10:41:35.267 INFO:teuthology.orchestra.run.smithi204.stdout: "sum": 0.000000000, 2024-04-23T10:41:35.267 INFO:teuthology.orchestra.run.smithi204.stdout: "avgtime": 0.000000000 2024-04-23T10:41:35.267 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-04-23T10:41:35.267 INFO:teuthology.orchestra.run.smithi204.stdout: "req_rmsnap_latency": { 2024-04-23T10:41:35.267 INFO:teuthology.orchestra.run.smithi204.stdout: "avgcount": 0, 2024-04-23T10:41:35.267 INFO:teuthology.orchestra.run.smithi204.stdout: "sum": 0.000000000, 2024-04-23T10:41:35.268 INFO:teuthology.orchestra.run.smithi204.stdout: "avgtime": 0.000000000 2024-04-23T10:41:35.268 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-04-23T10:41:35.268 INFO:teuthology.orchestra.run.smithi204.stdout: "req_rmxattr_latency": { 2024-04-23T10:41:35.268 INFO:teuthology.orchestra.run.smithi204.stdout: "avgcount": 0, 2024-04-23T10:41:35.268 INFO:teuthology.orchestra.run.smithi204.stdout: "sum": 0.000000000, 2024-04-23T10:41:35.268 INFO:teuthology.orchestra.run.smithi204.stdout: "avgtime": 0.000000000 2024-04-23T10:41:35.268 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-04-23T10:41:35.268 INFO:teuthology.orchestra.run.smithi204.stdout: "req_setattr_latency": { 2024-04-23T10:41:35.269 INFO:teuthology.orchestra.run.smithi204.stdout: "avgcount": 0, 2024-04-23T10:41:35.269 INFO:teuthology.orchestra.run.smithi204.stdout: "sum": 0.000000000, 2024-04-23T10:41:35.269 INFO:teuthology.orchestra.run.smithi204.stdout: "avgtime": 0.000000000 2024-04-23T10:41:35.269 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-04-23T10:41:35.269 INFO:teuthology.orchestra.run.smithi204.stdout: "req_setdirlayout_latency": { 2024-04-23T10:41:35.269 INFO:teuthology.orchestra.run.smithi204.stdout: "avgcount": 0, 2024-04-23T10:41:35.269 INFO:teuthology.orchestra.run.smithi204.stdout: "sum": 0.000000000, 2024-04-23T10:41:35.269 INFO:teuthology.orchestra.run.smithi204.stdout: "avgtime": 0.000000000 2024-04-23T10:41:35.270 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-04-23T10:41:35.270 INFO:teuthology.orchestra.run.smithi204.stdout: "req_setfilelock_latency": { 2024-04-23T10:41:35.270 INFO:teuthology.orchestra.run.smithi204.stdout: "avgcount": 0, 2024-04-23T10:41:35.270 INFO:teuthology.orchestra.run.smithi204.stdout: "sum": 0.000000000, 2024-04-23T10:41:35.270 INFO:teuthology.orchestra.run.smithi204.stdout: "avgtime": 0.000000000 2024-04-23T10:41:35.270 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-04-23T10:41:35.270 INFO:teuthology.orchestra.run.smithi204.stdout: "req_setlayout_latency": { 2024-04-23T10:41:35.271 INFO:teuthology.orchestra.run.smithi204.stdout: "avgcount": 0, 2024-04-23T10:41:35.271 INFO:teuthology.orchestra.run.smithi204.stdout: "sum": 0.000000000, 2024-04-23T10:41:35.271 INFO:teuthology.orchestra.run.smithi204.stdout: "avgtime": 0.000000000 2024-04-23T10:41:35.271 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-04-23T10:41:35.271 INFO:teuthology.orchestra.run.smithi204.stdout: "req_setxattr_latency": { 2024-04-23T10:41:35.271 INFO:teuthology.orchestra.run.smithi204.stdout: "avgcount": 0, 2024-04-23T10:41:35.271 INFO:teuthology.orchestra.run.smithi204.stdout: "sum": 0.000000000, 2024-04-23T10:41:35.271 INFO:teuthology.orchestra.run.smithi204.stdout: "avgtime": 0.000000000 2024-04-23T10:41:35.272 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-04-23T10:41:35.272 INFO:teuthology.orchestra.run.smithi204.stdout: "req_symlink_latency": { 2024-04-23T10:41:35.272 INFO:teuthology.orchestra.run.smithi204.stdout: "avgcount": 0, 2024-04-23T10:41:35.272 INFO:teuthology.orchestra.run.smithi204.stdout: "sum": 0.000000000, 2024-04-23T10:41:35.272 INFO:teuthology.orchestra.run.smithi204.stdout: "avgtime": 0.000000000 2024-04-23T10:41:35.272 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-04-23T10:41:35.272 INFO:teuthology.orchestra.run.smithi204.stdout: "req_unlink_latency": { 2024-04-23T10:41:35.272 INFO:teuthology.orchestra.run.smithi204.stdout: "avgcount": 0, 2024-04-23T10:41:35.273 INFO:teuthology.orchestra.run.smithi204.stdout: "sum": 0.000000000, 2024-04-23T10:41:35.273 INFO:teuthology.orchestra.run.smithi204.stdout: "avgtime": 0.000000000 2024-04-23T10:41:35.273 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-04-23T10:41:35.273 INFO:teuthology.orchestra.run.smithi204.stdout: "cap_revoke_eviction": 0, 2024-04-23T10:41:35.273 INFO:teuthology.orchestra.run.smithi204.stdout: "cap_acquisition_throttle": 0, 2024-04-23T10:41:35.273 INFO:teuthology.orchestra.run.smithi204.stdout: "req_getvxattr_latency": { 2024-04-23T10:41:35.273 INFO:teuthology.orchestra.run.smithi204.stdout: "avgcount": 0, 2024-04-23T10:41:35.273 INFO:teuthology.orchestra.run.smithi204.stdout: "sum": 0.000000000, 2024-04-23T10:41:35.274 INFO:teuthology.orchestra.run.smithi204.stdout: "avgtime": 0.000000000 2024-04-23T10:41:35.274 INFO:teuthology.orchestra.run.smithi204.stdout: } 2024-04-23T10:41:35.274 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-04-23T10:41:35.274 INFO:teuthology.orchestra.run.smithi204.stdout: "mds_sessions": { 2024-04-23T10:41:35.274 INFO:teuthology.orchestra.run.smithi204.stdout: "session_count": 1, 2024-04-23T10:41:35.274 INFO:teuthology.orchestra.run.smithi204.stdout: "session_add": 3, 2024-04-23T10:41:35.274 INFO:teuthology.orchestra.run.smithi204.stdout: "session_remove": 2, 2024-04-23T10:41:35.274 INFO:teuthology.orchestra.run.smithi204.stdout: "sessions_open": 2, 2024-04-23T10:41:35.275 INFO:teuthology.orchestra.run.smithi204.stdout: "sessions_stale": 0, 2024-04-23T10:41:35.275 INFO:teuthology.orchestra.run.smithi204.stdout: "total_load": 0, 2024-04-23T10:41:35.275 INFO:teuthology.orchestra.run.smithi204.stdout: "average_load": 0, 2024-04-23T10:41:35.275 INFO:teuthology.orchestra.run.smithi204.stdout: "avg_session_uptime": 4254, 2024-04-23T10:41:35.275 INFO:teuthology.orchestra.run.smithi204.stdout: "mdthresh_evicted": 0 2024-04-23T10:41:35.275 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-04-23T10:41:35.275 INFO:teuthology.orchestra.run.smithi204.stdout: "mempool": { 2024-04-23T10:41:35.275 INFO:teuthology.orchestra.run.smithi204.stdout: "bloom_filter_bytes": 0, 2024-04-23T10:41:35.276 INFO:teuthology.orchestra.run.smithi204.stdout: "bloom_filter_items": 0, 2024-04-23T10:41:35.276 INFO:teuthology.orchestra.run.smithi204.stdout: "bluestore_alloc_bytes": 0, 2024-04-23T10:41:35.276 INFO:teuthology.orchestra.run.smithi204.stdout: "bluestore_alloc_items": 0, 2024-04-23T10:41:35.276 INFO:teuthology.orchestra.run.smithi204.stdout: "bluestore_cache_data_bytes": 0, 2024-04-23T10:41:35.276 INFO:teuthology.orchestra.run.smithi204.stdout: "bluestore_cache_data_items": 0, 2024-04-23T10:41:35.276 INFO:teuthology.orchestra.run.smithi204.stdout: "bluestore_cache_onode_bytes": 0, 2024-04-23T10:41:35.276 INFO:teuthology.orchestra.run.smithi204.stdout: "bluestore_cache_onode_items": 0, 2024-04-23T10:41:35.276 INFO:teuthology.orchestra.run.smithi204.stdout: "bluestore_cache_meta_bytes": 0, 2024-04-23T10:41:35.277 INFO:teuthology.orchestra.run.smithi204.stdout: "bluestore_cache_meta_items": 0, 2024-04-23T10:41:35.277 INFO:teuthology.orchestra.run.smithi204.stdout: "bluestore_cache_other_bytes": 0, 2024-04-23T10:41:35.277 INFO:teuthology.orchestra.run.smithi204.stdout: "bluestore_cache_other_items": 0, 2024-04-23T10:41:35.277 INFO:teuthology.orchestra.run.smithi204.stdout: "bluestore_cache_buffer_bytes": 0, 2024-04-23T10:41:35.277 INFO:teuthology.orchestra.run.smithi204.stdout: "bluestore_cache_buffer_items": 0, 2024-04-23T10:41:35.277 INFO:teuthology.orchestra.run.smithi204.stdout: "bluestore_extent_bytes": 0, 2024-04-23T10:41:35.277 INFO:teuthology.orchestra.run.smithi204.stdout: "bluestore_extent_items": 0, 2024-04-23T10:41:35.277 INFO:teuthology.orchestra.run.smithi204.stdout: "bluestore_blob_bytes": 0, 2024-04-23T10:41:35.278 INFO:teuthology.orchestra.run.smithi204.stdout: "bluestore_blob_items": 0, 2024-04-23T10:41:35.278 INFO:teuthology.orchestra.run.smithi204.stdout: "bluestore_shared_blob_bytes": 0, 2024-04-23T10:41:35.278 INFO:teuthology.orchestra.run.smithi204.stdout: "bluestore_shared_blob_items": 0, 2024-04-23T10:41:35.278 INFO:teuthology.orchestra.run.smithi204.stdout: "bluestore_inline_bl_bytes": 0, 2024-04-23T10:41:35.278 INFO:teuthology.orchestra.run.smithi204.stdout: "bluestore_inline_bl_items": 0, 2024-04-23T10:41:35.278 INFO:teuthology.orchestra.run.smithi204.stdout: "bluestore_fsck_bytes": 0, 2024-04-23T10:41:35.278 INFO:teuthology.orchestra.run.smithi204.stdout: "bluestore_fsck_items": 0, 2024-04-23T10:41:35.278 INFO:teuthology.orchestra.run.smithi204.stdout: "bluestore_txc_bytes": 0, 2024-04-23T10:41:35.279 INFO:teuthology.orchestra.run.smithi204.stdout: "bluestore_txc_items": 0, 2024-04-23T10:41:35.279 INFO:teuthology.orchestra.run.smithi204.stdout: "bluestore_writing_deferred_bytes": 0, 2024-04-23T10:41:35.279 INFO:teuthology.orchestra.run.smithi204.stdout: "bluestore_writing_deferred_items": 0, 2024-04-23T10:41:35.279 INFO:teuthology.orchestra.run.smithi204.stdout: "bluestore_writing_bytes": 0, 2024-04-23T10:41:35.279 INFO:teuthology.orchestra.run.smithi204.stdout: "bluestore_writing_items": 0, 2024-04-23T10:41:35.279 INFO:teuthology.orchestra.run.smithi204.stdout: "bluefs_bytes": 0, 2024-04-23T10:41:35.279 INFO:teuthology.orchestra.run.smithi204.stdout: "bluefs_items": 0, 2024-04-23T10:41:35.279 INFO:teuthology.orchestra.run.smithi204.stdout: "bluefs_file_reader_bytes": 0, 2024-04-23T10:41:35.280 INFO:teuthology.orchestra.run.smithi204.stdout: "bluefs_file_reader_items": 0, 2024-04-23T10:41:35.280 INFO:teuthology.orchestra.run.smithi204.stdout: "bluefs_file_writer_bytes": 0, 2024-04-23T10:41:35.280 INFO:teuthology.orchestra.run.smithi204.stdout: "bluefs_file_writer_items": 0, 2024-04-23T10:41:35.280 INFO:teuthology.orchestra.run.smithi204.stdout: "buffer_anon_bytes": 268509, 2024-04-23T10:41:35.280 INFO:teuthology.orchestra.run.smithi204.stdout: "buffer_anon_items": 1078, 2024-04-23T10:41:35.280 INFO:teuthology.orchestra.run.smithi204.stdout: "buffer_meta_bytes": 0, 2024-04-23T10:41:35.280 INFO:teuthology.orchestra.run.smithi204.stdout: "buffer_meta_items": 0, 2024-04-23T10:41:35.280 INFO:teuthology.orchestra.run.smithi204.stdout: "osd_bytes": 0, 2024-04-23T10:41:35.280 INFO:teuthology.orchestra.run.smithi204.stdout: "osd_items": 0, 2024-04-23T10:41:35.281 INFO:teuthology.orchestra.run.smithi204.stdout: "osd_mapbl_bytes": 0, 2024-04-23T10:41:35.281 INFO:teuthology.orchestra.run.smithi204.stdout: "osd_mapbl_items": 0, 2024-04-23T10:41:35.281 INFO:teuthology.orchestra.run.smithi204.stdout: "osd_pglog_bytes": 0, 2024-04-23T10:41:35.281 INFO:teuthology.orchestra.run.smithi204.stdout: "osd_pglog_items": 0, 2024-04-23T10:41:35.281 INFO:teuthology.orchestra.run.smithi204.stdout: "osdmap_bytes": 8416, 2024-04-23T10:41:35.281 INFO:teuthology.orchestra.run.smithi204.stdout: "osdmap_items": 175, 2024-04-23T10:41:35.281 INFO:teuthology.orchestra.run.smithi204.stdout: "osdmap_mapping_bytes": 0, 2024-04-23T10:41:35.281 INFO:teuthology.orchestra.run.smithi204.stdout: "osdmap_mapping_items": 0, 2024-04-23T10:41:35.282 INFO:teuthology.orchestra.run.smithi204.stdout: "pgmap_bytes": 0, 2024-04-23T10:41:35.282 INFO:teuthology.orchestra.run.smithi204.stdout: "pgmap_items": 0, 2024-04-23T10:41:35.282 INFO:teuthology.orchestra.run.smithi204.stdout: "mds_co_bytes": 13191805, 2024-04-23T10:41:35.282 INFO:teuthology.orchestra.run.smithi204.stdout: "mds_co_items": 110670, 2024-04-23T10:41:35.282 INFO:teuthology.orchestra.run.smithi204.stdout: "unittest_1_bytes": 0, 2024-04-23T10:41:35.282 INFO:teuthology.orchestra.run.smithi204.stdout: "unittest_1_items": 0, 2024-04-23T10:41:35.282 INFO:teuthology.orchestra.run.smithi204.stdout: "unittest_2_bytes": 0, 2024-04-23T10:41:35.282 INFO:teuthology.orchestra.run.smithi204.stdout: "unittest_2_items": 0 2024-04-23T10:41:35.283 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-04-23T10:41:35.283 INFO:teuthology.orchestra.run.smithi204.stdout: "objecter": { 2024-04-23T10:41:35.283 INFO:teuthology.orchestra.run.smithi204.stdout: "op_active": 0, 2024-04-23T10:41:35.283 INFO:teuthology.orchestra.run.smithi204.stdout: "op_laggy": 0, 2024-04-23T10:41:35.283 INFO:teuthology.orchestra.run.smithi204.stdout: "op_send": 3526, 2024-04-23T10:41:35.283 INFO:teuthology.orchestra.run.smithi204.stdout: "op_send_bytes": 21, 2024-04-23T10:41:35.283 INFO:teuthology.orchestra.run.smithi204.stdout: "op_resend": 3, 2024-04-23T10:41:35.283 INFO:teuthology.orchestra.run.smithi204.stdout: "op_reply": 3526, 2024-04-23T10:41:35.284 INFO:teuthology.orchestra.run.smithi204.stdout: "op_latency": { 2024-04-23T10:41:35.284 INFO:teuthology.orchestra.run.smithi204.stdout: "avgcount": 3526, 2024-04-23T10:41:35.284 INFO:teuthology.orchestra.run.smithi204.stdout: "sum": 2.424966806, 2024-04-23T10:41:35.284 INFO:teuthology.orchestra.run.smithi204.stdout: "avgtime": 0.000687738 2024-04-23T10:41:35.284 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-04-23T10:41:35.284 INFO:teuthology.orchestra.run.smithi204.stdout: "op_inflight": 0, 2024-04-23T10:41:35.284 INFO:teuthology.orchestra.run.smithi204.stdout: "oplen_avg": { 2024-04-23T10:41:35.284 INFO:teuthology.orchestra.run.smithi204.stdout: "avgcount": 3526, 2024-04-23T10:41:35.285 INFO:teuthology.orchestra.run.smithi204.stdout: "sum": 3527 2024-04-23T10:41:35.285 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-04-23T10:41:35.285 INFO:teuthology.orchestra.run.smithi204.stdout: "op": 3526, 2024-04-23T10:41:35.285 INFO:teuthology.orchestra.run.smithi204.stdout: "op_r": 3526, 2024-04-23T10:41:35.285 INFO:teuthology.orchestra.run.smithi204.stdout: "op_w": 0, 2024-04-23T10:41:35.285 INFO:teuthology.orchestra.run.smithi204.stdout: "op_rmw": 0, 2024-04-23T10:41:35.285 INFO:teuthology.orchestra.run.smithi204.stdout: "op_pg": 0, 2024-04-23T10:41:35.285 INFO:teuthology.orchestra.run.smithi204.stdout: "osdop_stat": 1976, 2024-04-23T10:41:35.285 INFO:teuthology.orchestra.run.smithi204.stdout: "osdop_create": 0, 2024-04-23T10:41:35.286 INFO:teuthology.orchestra.run.smithi204.stdout: "osdop_read": 1548, 2024-04-23T10:41:35.286 INFO:teuthology.orchestra.run.smithi204.stdout: "osdop_write": 0, 2024-04-23T10:41:35.286 INFO:teuthology.orchestra.run.smithi204.stdout: "osdop_writefull": 0, 2024-04-23T10:41:35.286 INFO:teuthology.orchestra.run.smithi204.stdout: "osdop_writesame": 0, 2024-04-23T10:41:35.286 INFO:teuthology.orchestra.run.smithi204.stdout: "osdop_append": 0, 2024-04-23T10:41:35.286 INFO:teuthology.orchestra.run.smithi204.stdout: "osdop_zero": 0, 2024-04-23T10:41:35.286 INFO:teuthology.orchestra.run.smithi204.stdout: "osdop_truncate": 0, 2024-04-23T10:41:35.287 INFO:teuthology.orchestra.run.smithi204.stdout: "osdop_delete": 0, 2024-04-23T10:41:35.287 INFO:teuthology.orchestra.run.smithi204.stdout: "osdop_mapext": 0, 2024-04-23T10:41:35.287 INFO:teuthology.orchestra.run.smithi204.stdout: "osdop_sparse_read": 0, 2024-04-23T10:41:35.287 INFO:teuthology.orchestra.run.smithi204.stdout: "osdop_clonerange": 0, 2024-04-23T10:41:35.287 INFO:teuthology.orchestra.run.smithi204.stdout: "osdop_getxattr": 1, 2024-04-23T10:41:35.287 INFO:teuthology.orchestra.run.smithi204.stdout: "osdop_setxattr": 0, 2024-04-23T10:41:35.287 INFO:teuthology.orchestra.run.smithi204.stdout: "osdop_cmpxattr": 0, 2024-04-23T10:41:35.287 INFO:teuthology.orchestra.run.smithi204.stdout: "osdop_rmxattr": 0, 2024-04-23T10:41:35.287 INFO:teuthology.orchestra.run.smithi204.stdout: "osdop_resetxattrs": 0, 2024-04-23T10:41:35.288 INFO:teuthology.orchestra.run.smithi204.stdout: "osdop_call": 0, 2024-04-23T10:41:35.288 INFO:teuthology.orchestra.run.smithi204.stdout: "osdop_watch": 0, 2024-04-23T10:41:35.288 INFO:teuthology.orchestra.run.smithi204.stdout: "osdop_notify": 0, 2024-04-23T10:41:35.288 INFO:teuthology.orchestra.run.smithi204.stdout: "osdop_src_cmpxattr": 0, 2024-04-23T10:41:35.288 INFO:teuthology.orchestra.run.smithi204.stdout: "osdop_pgls": 0, 2024-04-23T10:41:35.288 INFO:teuthology.orchestra.run.smithi204.stdout: "osdop_pgls_filter": 0, 2024-04-23T10:41:35.288 INFO:teuthology.orchestra.run.smithi204.stdout: "osdop_other": 0, 2024-04-23T10:41:35.288 INFO:teuthology.orchestra.run.smithi204.stdout: "linger_active": 0, 2024-04-23T10:41:35.289 INFO:teuthology.orchestra.run.smithi204.stdout: "linger_send": 0, 2024-04-23T10:41:35.289 INFO:teuthology.orchestra.run.smithi204.stdout: "linger_resend": 0, 2024-04-23T10:41:35.289 INFO:teuthology.orchestra.run.smithi204.stdout: "linger_ping": 0, 2024-04-23T10:41:35.289 INFO:teuthology.orchestra.run.smithi204.stdout: "poolop_active": 0, 2024-04-23T10:41:35.289 INFO:teuthology.orchestra.run.smithi204.stdout: "poolop_send": 0, 2024-04-23T10:41:35.289 INFO:teuthology.orchestra.run.smithi204.stdout: "poolop_resend": 0, 2024-04-23T10:41:35.289 INFO:teuthology.orchestra.run.smithi204.stdout: "poolstat_active": 0, 2024-04-23T10:41:35.290 INFO:teuthology.orchestra.run.smithi204.stdout: "poolstat_send": 0, 2024-04-23T10:41:35.290 INFO:teuthology.orchestra.run.smithi204.stdout: "poolstat_resend": 0, 2024-04-23T10:41:35.290 INFO:teuthology.orchestra.run.smithi204.stdout: "statfs_active": 0, 2024-04-23T10:41:35.290 INFO:teuthology.orchestra.run.smithi204.stdout: "statfs_send": 0, 2024-04-23T10:41:35.290 INFO:teuthology.orchestra.run.smithi204.stdout: "statfs_resend": 0, 2024-04-23T10:41:35.290 INFO:teuthology.orchestra.run.smithi204.stdout: "command_active": 0, 2024-04-23T10:41:35.290 INFO:teuthology.orchestra.run.smithi204.stdout: "command_send": 0, 2024-04-23T10:41:35.291 INFO:teuthology.orchestra.run.smithi204.stdout: "command_resend": 0, 2024-04-23T10:41:35.291 INFO:teuthology.orchestra.run.smithi204.stdout: "map_epoch": 113, 2024-04-23T10:41:35.291 INFO:teuthology.orchestra.run.smithi204.stdout: "map_full": 0, 2024-04-23T10:41:35.291 INFO:teuthology.orchestra.run.smithi204.stdout: "map_inc": 26, 2024-04-23T10:41:35.291 INFO:teuthology.orchestra.run.smithi204.stdout: "osd_sessions": 10, 2024-04-23T10:41:35.291 INFO:teuthology.orchestra.run.smithi204.stdout: "osd_session_open": 10, 2024-04-23T10:41:35.291 INFO:teuthology.orchestra.run.smithi204.stdout: "osd_session_close": 0, 2024-04-23T10:41:35.291 INFO:teuthology.orchestra.run.smithi204.stdout: "osd_laggy": 0, 2024-04-23T10:41:35.292 INFO:teuthology.orchestra.run.smithi204.stdout: "omap_wr": 0, 2024-04-23T10:41:35.292 INFO:teuthology.orchestra.run.smithi204.stdout: "omap_rd": 2, 2024-04-23T10:41:35.292 INFO:teuthology.orchestra.run.smithi204.stdout: "omap_del": 0 2024-04-23T10:41:35.292 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-04-23T10:41:35.292 INFO:teuthology.orchestra.run.smithi204.stdout: "oft": { 2024-04-23T10:41:35.292 INFO:teuthology.orchestra.run.smithi204.stdout: "omap_total_objs": 0, 2024-04-23T10:41:35.292 INFO:teuthology.orchestra.run.smithi204.stdout: "omap_total_kv_pairs": 0, 2024-04-23T10:41:35.293 INFO:teuthology.orchestra.run.smithi204.stdout: "omap_total_updates": 0, 2024-04-23T10:41:35.293 INFO:teuthology.orchestra.run.smithi204.stdout: "omap_total_removes": 0 2024-04-23T10:41:35.293 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-04-23T10:41:35.293 INFO:teuthology.orchestra.run.smithi204.stdout: "osd-slow-ops": { 2024-04-23T10:41:35.293 INFO:teuthology.orchestra.run.smithi204.stdout: "slow_ops_count": 0 2024-04-23T10:41:35.293 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-04-23T10:41:35.293 INFO:teuthology.orchestra.run.smithi204.stdout: "purge_queue": { 2024-04-23T10:41:35.293 INFO:teuthology.orchestra.run.smithi204.stdout: "pq_executing_ops": 0, 2024-04-23T10:41:35.294 INFO:teuthology.orchestra.run.smithi204.stdout: "pq_executing_ops_high_water": 0, 2024-04-23T10:41:35.294 INFO:teuthology.orchestra.run.smithi204.stdout: "pq_executing": 0, 2024-04-23T10:41:35.294 INFO:teuthology.orchestra.run.smithi204.stdout: "pq_executing_high_water": 0, 2024-04-23T10:41:35.294 INFO:teuthology.orchestra.run.smithi204.stdout: "pq_executed_ops": 0, 2024-04-23T10:41:35.294 INFO:teuthology.orchestra.run.smithi204.stdout: "pq_executed": 0, 2024-04-23T10:41:35.294 INFO:teuthology.orchestra.run.smithi204.stdout: "pq_item_in_journal": 0 2024-04-23T10:41:35.294 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-04-23T10:41:35.295 INFO:teuthology.orchestra.run.smithi204.stdout: "throttle-msgr_dispatch_throttler-mds": { 2024-04-23T10:41:35.295 INFO:teuthology.orchestra.run.smithi204.stdout: "val": 0, 2024-04-23T10:41:35.295 INFO:teuthology.orchestra.run.smithi204.stdout: "max": 104857600, 2024-04-23T10:41:35.295 INFO:teuthology.orchestra.run.smithi204.stdout: "get_started": 0, 2024-04-23T10:41:35.295 INFO:teuthology.orchestra.run.smithi204.stdout: "get": 3877, 2024-04-23T10:41:35.295 INFO:teuthology.orchestra.run.smithi204.stdout: "get_sum": 168836838, 2024-04-23T10:41:35.295 INFO:teuthology.orchestra.run.smithi204.stdout: "get_or_fail_fail": 0, 2024-04-23T10:41:35.295 INFO:teuthology.orchestra.run.smithi204.stdout: "get_or_fail_success": 3877, 2024-04-23T10:41:35.295 INFO:teuthology.orchestra.run.smithi204.stdout: "take": 0, 2024-04-23T10:41:35.296 INFO:teuthology.orchestra.run.smithi204.stdout: "take_sum": 0, 2024-04-23T10:41:35.296 INFO:teuthology.orchestra.run.smithi204.stdout: "put": 3877, 2024-04-23T10:41:35.296 INFO:teuthology.orchestra.run.smithi204.stdout: "put_sum": 168836838, 2024-04-23T10:41:35.296 INFO:teuthology.orchestra.run.smithi204.stdout: "wait": { 2024-04-23T10:41:35.296 INFO:teuthology.orchestra.run.smithi204.stdout: "avgcount": 0, 2024-04-23T10:41:35.296 INFO:teuthology.orchestra.run.smithi204.stdout: "sum": 0.000000000, 2024-04-23T10:41:35.296 INFO:teuthology.orchestra.run.smithi204.stdout: "avgtime": 0.000000000 2024-04-23T10:41:35.296 INFO:teuthology.orchestra.run.smithi204.stdout: } 2024-04-23T10:41:35.297 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-04-23T10:41:35.297 INFO:teuthology.orchestra.run.smithi204.stdout: "throttle-objecter_bytes": { 2024-04-23T10:41:35.297 INFO:teuthology.orchestra.run.smithi204.stdout: "val": 0, 2024-04-23T10:41:35.297 INFO:teuthology.orchestra.run.smithi204.stdout: "max": 104857600, 2024-04-23T10:41:35.297 INFO:teuthology.orchestra.run.smithi204.stdout: "get_started": 0, 2024-04-23T10:41:35.297 INFO:teuthology.orchestra.run.smithi204.stdout: "get": 0, 2024-04-23T10:41:35.297 INFO:teuthology.orchestra.run.smithi204.stdout: "get_sum": 0, 2024-04-23T10:41:35.297 INFO:teuthology.orchestra.run.smithi204.stdout: "get_or_fail_fail": 0, 2024-04-23T10:41:35.297 INFO:teuthology.orchestra.run.smithi204.stdout: "get_or_fail_success": 0, 2024-04-23T10:41:35.298 INFO:teuthology.orchestra.run.smithi204.stdout: "take": 3526, 2024-04-23T10:41:35.298 INFO:teuthology.orchestra.run.smithi204.stdout: "take_sum": 166462376, 2024-04-23T10:41:35.298 INFO:teuthology.orchestra.run.smithi204.stdout: "put": 3526, 2024-04-23T10:41:35.298 INFO:teuthology.orchestra.run.smithi204.stdout: "put_sum": 166462376, 2024-04-23T10:41:35.298 INFO:teuthology.orchestra.run.smithi204.stdout: "wait": { 2024-04-23T10:41:35.298 INFO:teuthology.orchestra.run.smithi204.stdout: "avgcount": 0, 2024-04-23T10:41:35.298 INFO:teuthology.orchestra.run.smithi204.stdout: "sum": 0.000000000, 2024-04-23T10:41:35.299 INFO:teuthology.orchestra.run.smithi204.stdout: "avgtime": 0.000000000 2024-04-23T10:41:35.299 INFO:teuthology.orchestra.run.smithi204.stdout: } 2024-04-23T10:41:35.299 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-04-23T10:41:35.299 INFO:teuthology.orchestra.run.smithi204.stdout: "throttle-objecter_ops": { 2024-04-23T10:41:35.299 INFO:teuthology.orchestra.run.smithi204.stdout: "val": 0, 2024-04-23T10:41:35.299 INFO:teuthology.orchestra.run.smithi204.stdout: "max": 1024, 2024-04-23T10:41:35.299 INFO:teuthology.orchestra.run.smithi204.stdout: "get_started": 0, 2024-04-23T10:41:35.299 INFO:teuthology.orchestra.run.smithi204.stdout: "get": 0, 2024-04-23T10:41:35.299 INFO:teuthology.orchestra.run.smithi204.stdout: "get_sum": 0, 2024-04-23T10:41:35.300 INFO:teuthology.orchestra.run.smithi204.stdout: "get_or_fail_fail": 0, 2024-04-23T10:41:35.300 INFO:teuthology.orchestra.run.smithi204.stdout: "get_or_fail_success": 0, 2024-04-23T10:41:35.300 INFO:teuthology.orchestra.run.smithi204.stdout: "take": 3526, 2024-04-23T10:41:35.300 INFO:teuthology.orchestra.run.smithi204.stdout: "take_sum": 3526, 2024-04-23T10:41:35.300 INFO:teuthology.orchestra.run.smithi204.stdout: "put": 3526, 2024-04-23T10:41:35.300 INFO:teuthology.orchestra.run.smithi204.stdout: "put_sum": 3526, 2024-04-23T10:41:35.300 INFO:teuthology.orchestra.run.smithi204.stdout: "wait": { 2024-04-23T10:41:35.300 INFO:teuthology.orchestra.run.smithi204.stdout: "avgcount": 0, 2024-04-23T10:41:35.301 INFO:teuthology.orchestra.run.smithi204.stdout: "sum": 0.000000000, 2024-04-23T10:41:35.301 INFO:teuthology.orchestra.run.smithi204.stdout: "avgtime": 0.000000000 2024-04-23T10:41:35.301 INFO:teuthology.orchestra.run.smithi204.stdout: } 2024-04-23T10:41:35.301 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-04-23T10:41:35.301 INFO:teuthology.orchestra.run.smithi204.stdout: "throttle-write_buf_throttle": { 2024-04-23T10:41:35.301 INFO:teuthology.orchestra.run.smithi204.stdout: "val": 0, 2024-04-23T10:41:35.301 INFO:teuthology.orchestra.run.smithi204.stdout: "max": 3758096384, 2024-04-23T10:41:35.302 INFO:teuthology.orchestra.run.smithi204.stdout: "get_started": 0, 2024-04-23T10:41:35.302 INFO:teuthology.orchestra.run.smithi204.stdout: "get": 0, 2024-04-23T10:41:35.302 INFO:teuthology.orchestra.run.smithi204.stdout: "get_sum": 0, 2024-04-23T10:41:35.302 INFO:teuthology.orchestra.run.smithi204.stdout: "get_or_fail_fail": 0, 2024-04-23T10:41:35.302 INFO:teuthology.orchestra.run.smithi204.stdout: "get_or_fail_success": 0, 2024-04-23T10:41:35.302 INFO:teuthology.orchestra.run.smithi204.stdout: "take": 0, 2024-04-23T10:41:35.302 INFO:teuthology.orchestra.run.smithi204.stdout: "take_sum": 0, 2024-04-23T10:41:35.302 INFO:teuthology.orchestra.run.smithi204.stdout: "put": 0, 2024-04-23T10:41:35.303 INFO:teuthology.orchestra.run.smithi204.stdout: "put_sum": 0, 2024-04-23T10:41:35.303 INFO:teuthology.orchestra.run.smithi204.stdout: "wait": { 2024-04-23T10:41:35.303 INFO:teuthology.orchestra.run.smithi204.stdout: "avgcount": 0, 2024-04-23T10:41:35.303 INFO:teuthology.orchestra.run.smithi204.stdout: "sum": 0.000000000, 2024-04-23T10:41:35.303 INFO:teuthology.orchestra.run.smithi204.stdout: "avgtime": 0.000000000 2024-04-23T10:41:35.303 INFO:teuthology.orchestra.run.smithi204.stdout: } 2024-04-23T10:41:35.303 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-04-23T10:41:35.304 INFO:teuthology.orchestra.run.smithi204.stdout: "throttle-write_buf_throttle-0x5593196a06e0": { 2024-04-23T10:41:35.304 INFO:teuthology.orchestra.run.smithi204.stdout: "val": 0, 2024-04-23T10:41:35.304 INFO:teuthology.orchestra.run.smithi204.stdout: "max": 3758096384, 2024-04-23T10:41:35.304 INFO:teuthology.orchestra.run.smithi204.stdout: "get_started": 0, 2024-04-23T10:41:35.304 INFO:teuthology.orchestra.run.smithi204.stdout: "get": 0, 2024-04-23T10:41:35.304 INFO:teuthology.orchestra.run.smithi204.stdout: "get_sum": 0, 2024-04-23T10:41:35.304 INFO:teuthology.orchestra.run.smithi204.stdout: "get_or_fail_fail": 0, 2024-04-23T10:41:35.304 INFO:teuthology.orchestra.run.smithi204.stdout: "get_or_fail_success": 0, 2024-04-23T10:41:35.305 INFO:teuthology.orchestra.run.smithi204.stdout: "take": 0, 2024-04-23T10:41:35.305 INFO:teuthology.orchestra.run.smithi204.stdout: "take_sum": 0, 2024-04-23T10:41:35.305 INFO:teuthology.orchestra.run.smithi204.stdout: "put": 0, 2024-04-23T10:41:35.305 INFO:teuthology.orchestra.run.smithi204.stdout: "put_sum": 0, 2024-04-23T10:41:35.305 INFO:teuthology.orchestra.run.smithi204.stdout: "wait": { 2024-04-23T10:41:35.305 INFO:teuthology.orchestra.run.smithi204.stdout: "avgcount": 0, 2024-04-23T10:41:35.305 INFO:teuthology.orchestra.run.smithi204.stdout: "sum": 0.000000000, 2024-04-23T10:41:35.306 INFO:teuthology.orchestra.run.smithi204.stdout: "avgtime": 0.000000000 2024-04-23T10:41:35.306 INFO:teuthology.orchestra.run.smithi204.stdout: } 2024-04-23T10:41:35.306 INFO:teuthology.orchestra.run.smithi204.stdout: } 2024-04-23T10:41:35.306 INFO:teuthology.orchestra.run.smithi204.stdout:} 2024-04-23T10:41:35.828 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 23 10:41:35 smithi204 ceph-mon[32649]: pgmap v473: 97 pgs: 97 active+clean; 558 MiB data, 4.3 GiB used, 1.0 TiB / 1.0 TiB avail; 468 KiB/s rd, 4.7 MiB/s wr, 156 op/s 2024-04-23T10:41:35.829 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 23 10:41:35 smithi119 ceph-mon[32608]: pgmap v473: 97 pgs: 97 active+clean; 558 MiB data, 4.3 GiB used, 1.0 TiB / 1.0 TiB avail; 468 KiB/s rd, 4.7 MiB/s wr, 156 op/s 2024-04-23T10:41:35.988 INFO:tasks.check_counter:Daemon mds.c mds.exported=0 2024-04-23T10:41:35.988 INFO:tasks.check_counter:Daemon mds.c mds.imported=0 2024-04-23T10:41:35.988 INFO:tasks.check_counter:Daemon mds.c mds_cache.dir_update=0 2024-04-23T10:41:35.988 INFO:tasks.check_counter:Daemon mds.c mds_cache.dir_update_receipt=0 2024-04-23T10:41:35.988 INFO:tasks.check_counter:Daemon mds.c mds.root_rsnaps=0 2024-04-23T10:41:35.989 INFO:tasks.check_counter:Daemon mds.c mds_server.req_mksnap_latency.avgcount=0 2024-04-23T10:41:35.989 INFO:tasks.check_counter:Daemon mds.c mds_server.req_rmsnap_latency.avgcount=0 2024-04-23T10:41:35.989 DEBUG:tasks.check_counter:Getting stats from f 2024-04-23T10:41:35.989 DEBUG:teuthology.orchestra.run.smithi204:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d69f13d827952bb07bb9b7973bca14f97d764d6f shell --fsid aa240572-0159-11ef-bc93-c7b262605968 -- ceph daemon mds.f perf dump 2024-04-23T10:41:36.004 INFO:journalctl@ceph.mon.a.smithi114.stdout:Apr 23 10:41:35 smithi114 ceph-mon[29255]: pgmap v473: 97 pgs: 97 active+clean; 558 MiB data, 4.3 GiB used, 1.0 TiB / 1.0 TiB avail; 468 KiB/s rd, 4.7 MiB/s wr, 156 op/s 2024-04-23T10:41:36.234 INFO:teuthology.orchestra.run.smithi204.stderr:Inferring config /var/lib/ceph/aa240572-0159-11ef-bc93-c7b262605968/mon.c/config 2024-04-23T10:41:37.809 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 23 10:41:37 smithi204 ceph-mon[32649]: pgmap v474: 97 pgs: 97 active+clean; 551 MiB data, 4.3 GiB used, 1.0 TiB / 1.0 TiB avail; 703 KiB/s rd, 5.5 MiB/s wr, 194 op/s 2024-04-23T10:41:37.899 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 23 10:41:37 smithi119 ceph-mon[32608]: pgmap v474: 97 pgs: 97 active+clean; 551 MiB data, 4.3 GiB used, 1.0 TiB / 1.0 TiB avail; 703 KiB/s rd, 5.5 MiB/s wr, 194 op/s 2024-04-23T10:41:38.004 INFO:journalctl@ceph.mon.a.smithi114.stdout:Apr 23 10:41:37 smithi114 ceph-mon[29255]: pgmap v474: 97 pgs: 97 active+clean; 551 MiB data, 4.3 GiB used, 1.0 TiB / 1.0 TiB avail; 703 KiB/s rd, 5.5 MiB/s wr, 194 op/s 2024-04-23T10:41:38.082 INFO:teuthology.orchestra.run.smithi204.stdout:{ 2024-04-23T10:41:38.082 INFO:teuthology.orchestra.run.smithi204.stdout: "AsyncMessenger::Worker-0": { 2024-04-23T10:41:38.082 INFO:teuthology.orchestra.run.smithi204.stdout: "msgr_recv_messages": 293, 2024-04-23T10:41:38.083 INFO:teuthology.orchestra.run.smithi204.stdout: "msgr_send_messages": 280, 2024-04-23T10:41:38.083 INFO:teuthology.orchestra.run.smithi204.stdout: "msgr_recv_bytes": 1611760, 2024-04-23T10:41:38.083 INFO:teuthology.orchestra.run.smithi204.stdout: "msgr_send_bytes": 171552, 2024-04-23T10:41:38.083 INFO:teuthology.orchestra.run.smithi204.stdout: "msgr_created_connections": 4, 2024-04-23T10:41:38.083 INFO:teuthology.orchestra.run.smithi204.stdout: "msgr_active_connections": 0, 2024-04-23T10:41:38.083 INFO:teuthology.orchestra.run.smithi204.stdout: "msgr_running_total_time": 0.052349277, 2024-04-23T10:41:38.083 INFO:teuthology.orchestra.run.smithi204.stdout: "msgr_running_send_time": 0.016500812, 2024-04-23T10:41:38.084 INFO:teuthology.orchestra.run.smithi204.stdout: "msgr_running_recv_time": 0.035846637, 2024-04-23T10:41:38.084 INFO:teuthology.orchestra.run.smithi204.stdout: "msgr_running_fast_dispatch_time": 0.006792922, 2024-04-23T10:41:38.084 INFO:teuthology.orchestra.run.smithi204.stdout: "msgr_send_messages_queue_lat": { 2024-04-23T10:41:38.084 INFO:teuthology.orchestra.run.smithi204.stdout: "avgcount": 280, 2024-04-23T10:41:38.084 INFO:teuthology.orchestra.run.smithi204.stdout: "sum": 0.014546961, 2024-04-23T10:41:38.084 INFO:teuthology.orchestra.run.smithi204.stdout: "avgtime": 0.000051953 2024-04-23T10:41:38.085 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-04-23T10:41:38.085 INFO:teuthology.orchestra.run.smithi204.stdout: "msgr_handle_ack_lat": { 2024-04-23T10:41:38.085 INFO:teuthology.orchestra.run.smithi204.stdout: "avgcount": 0, 2024-04-23T10:41:38.085 INFO:teuthology.orchestra.run.smithi204.stdout: "sum": 0.000000000, 2024-04-23T10:41:38.085 INFO:teuthology.orchestra.run.smithi204.stdout: "avgtime": 0.000000000 2024-04-23T10:41:38.085 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-04-23T10:41:38.086 INFO:teuthology.orchestra.run.smithi204.stdout: "msgr_recv_encrypted_bytes": 1611760, 2024-04-23T10:41:38.086 INFO:teuthology.orchestra.run.smithi204.stdout: "msgr_send_encrypted_bytes": 171552 2024-04-23T10:41:38.086 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-04-23T10:41:38.086 INFO:teuthology.orchestra.run.smithi204.stdout: "AsyncMessenger::Worker-1": { 2024-04-23T10:41:38.086 INFO:teuthology.orchestra.run.smithi204.stdout: "msgr_recv_messages": 2, 2024-04-23T10:41:38.086 INFO:teuthology.orchestra.run.smithi204.stdout: "msgr_send_messages": 212, 2024-04-23T10:41:38.087 INFO:teuthology.orchestra.run.smithi204.stdout: "msgr_recv_bytes": 320, 2024-04-23T10:41:38.087 INFO:teuthology.orchestra.run.smithi204.stdout: "msgr_send_bytes": 193392, 2024-04-23T10:41:38.087 INFO:teuthology.orchestra.run.smithi204.stdout: "msgr_created_connections": 5, 2024-04-23T10:41:38.087 INFO:teuthology.orchestra.run.smithi204.stdout: "msgr_active_connections": 1, 2024-04-23T10:41:38.087 INFO:teuthology.orchestra.run.smithi204.stdout: "msgr_running_total_time": 0.013086279, 2024-04-23T10:41:38.087 INFO:teuthology.orchestra.run.smithi204.stdout: "msgr_running_send_time": 0.008387352, 2024-04-23T10:41:38.088 INFO:teuthology.orchestra.run.smithi204.stdout: "msgr_running_recv_time": 0.002194696, 2024-04-23T10:41:38.088 INFO:teuthology.orchestra.run.smithi204.stdout: "msgr_running_fast_dispatch_time": 0.000000000, 2024-04-23T10:41:38.088 INFO:teuthology.orchestra.run.smithi204.stdout: "msgr_send_messages_queue_lat": { 2024-04-23T10:41:38.088 INFO:teuthology.orchestra.run.smithi204.stdout: "avgcount": 209, 2024-04-23T10:41:38.088 INFO:teuthology.orchestra.run.smithi204.stdout: "sum": 0.011574512, 2024-04-23T10:41:38.088 INFO:teuthology.orchestra.run.smithi204.stdout: "avgtime": 0.000055380 2024-04-23T10:41:38.089 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-04-23T10:41:38.089 INFO:teuthology.orchestra.run.smithi204.stdout: "msgr_handle_ack_lat": { 2024-04-23T10:41:38.089 INFO:teuthology.orchestra.run.smithi204.stdout: "avgcount": 0, 2024-04-23T10:41:38.089 INFO:teuthology.orchestra.run.smithi204.stdout: "sum": 0.000000000, 2024-04-23T10:41:38.089 INFO:teuthology.orchestra.run.smithi204.stdout: "avgtime": 0.000000000 2024-04-23T10:41:38.089 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-04-23T10:41:38.089 INFO:teuthology.orchestra.run.smithi204.stdout: "msgr_recv_encrypted_bytes": 320, 2024-04-23T10:41:38.090 INFO:teuthology.orchestra.run.smithi204.stdout: "msgr_send_encrypted_bytes": 193392 2024-04-23T10:41:38.090 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-04-23T10:41:38.090 INFO:teuthology.orchestra.run.smithi204.stdout: "AsyncMessenger::Worker-2": { 2024-04-23T10:41:38.090 INFO:teuthology.orchestra.run.smithi204.stdout: "msgr_recv_messages": 2, 2024-04-23T10:41:38.090 INFO:teuthology.orchestra.run.smithi204.stdout: "msgr_send_messages": 4, 2024-04-23T10:41:38.090 INFO:teuthology.orchestra.run.smithi204.stdout: "msgr_recv_bytes": 303, 2024-04-23T10:41:38.091 INFO:teuthology.orchestra.run.smithi204.stdout: "msgr_send_bytes": 23311, 2024-04-23T10:41:38.091 INFO:teuthology.orchestra.run.smithi204.stdout: "msgr_created_connections": 4, 2024-04-23T10:41:38.091 INFO:teuthology.orchestra.run.smithi204.stdout: "msgr_active_connections": 0, 2024-04-23T10:41:38.091 INFO:teuthology.orchestra.run.smithi204.stdout: "msgr_running_total_time": 0.001064143, 2024-04-23T10:41:38.091 INFO:teuthology.orchestra.run.smithi204.stdout: "msgr_running_send_time": 0.000055609, 2024-04-23T10:41:38.092 INFO:teuthology.orchestra.run.smithi204.stdout: "msgr_running_recv_time": 0.000901810, 2024-04-23T10:41:38.092 INFO:teuthology.orchestra.run.smithi204.stdout: "msgr_running_fast_dispatch_time": 0.000000000, 2024-04-23T10:41:38.092 INFO:teuthology.orchestra.run.smithi204.stdout: "msgr_send_messages_queue_lat": { 2024-04-23T10:41:38.092 INFO:teuthology.orchestra.run.smithi204.stdout: "avgcount": 2, 2024-04-23T10:41:38.092 INFO:teuthology.orchestra.run.smithi204.stdout: "sum": 0.000032293, 2024-04-23T10:41:38.092 INFO:teuthology.orchestra.run.smithi204.stdout: "avgtime": 0.000016146 2024-04-23T10:41:38.092 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-04-23T10:41:38.093 INFO:teuthology.orchestra.run.smithi204.stdout: "msgr_handle_ack_lat": { 2024-04-23T10:41:38.093 INFO:teuthology.orchestra.run.smithi204.stdout: "avgcount": 2, 2024-04-23T10:41:38.093 INFO:teuthology.orchestra.run.smithi204.stdout: "sum": 0.000000239, 2024-04-23T10:41:38.093 INFO:teuthology.orchestra.run.smithi204.stdout: "avgtime": 0.000000119 2024-04-23T10:41:38.093 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-04-23T10:41:38.093 INFO:teuthology.orchestra.run.smithi204.stdout: "msgr_recv_encrypted_bytes": 0, 2024-04-23T10:41:38.094 INFO:teuthology.orchestra.run.smithi204.stdout: "msgr_send_encrypted_bytes": 0 2024-04-23T10:41:38.094 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-04-23T10:41:38.094 INFO:teuthology.orchestra.run.smithi204.stdout: "cct": { 2024-04-23T10:41:38.094 INFO:teuthology.orchestra.run.smithi204.stdout: "total_workers": 0, 2024-04-23T10:41:38.094 INFO:teuthology.orchestra.run.smithi204.stdout: "unhealthy_workers": 0 2024-04-23T10:41:38.094 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-04-23T10:41:38.095 INFO:teuthology.orchestra.run.smithi204.stdout: "mempool": { 2024-04-23T10:41:38.095 INFO:teuthology.orchestra.run.smithi204.stdout: "bloom_filter_bytes": 0, 2024-04-23T10:41:38.095 INFO:teuthology.orchestra.run.smithi204.stdout: "bloom_filter_items": 0, 2024-04-23T10:41:38.095 INFO:teuthology.orchestra.run.smithi204.stdout: "bluestore_alloc_bytes": 0, 2024-04-23T10:41:38.095 INFO:teuthology.orchestra.run.smithi204.stdout: "bluestore_alloc_items": 0, 2024-04-23T10:41:38.095 INFO:teuthology.orchestra.run.smithi204.stdout: "bluestore_cache_data_bytes": 0, 2024-04-23T10:41:38.095 INFO:teuthology.orchestra.run.smithi204.stdout: "bluestore_cache_data_items": 0, 2024-04-23T10:41:38.096 INFO:teuthology.orchestra.run.smithi204.stdout: "bluestore_cache_onode_bytes": 0, 2024-04-23T10:41:38.096 INFO:teuthology.orchestra.run.smithi204.stdout: "bluestore_cache_onode_items": 0, 2024-04-23T10:41:38.096 INFO:teuthology.orchestra.run.smithi204.stdout: "bluestore_cache_meta_bytes": 0, 2024-04-23T10:41:38.096 INFO:teuthology.orchestra.run.smithi204.stdout: "bluestore_cache_meta_items": 0, 2024-04-23T10:41:38.096 INFO:teuthology.orchestra.run.smithi204.stdout: "bluestore_cache_other_bytes": 0, 2024-04-23T10:41:38.096 INFO:teuthology.orchestra.run.smithi204.stdout: "bluestore_cache_other_items": 0, 2024-04-23T10:41:38.097 INFO:teuthology.orchestra.run.smithi204.stdout: "bluestore_cache_buffer_bytes": 0, 2024-04-23T10:41:38.097 INFO:teuthology.orchestra.run.smithi204.stdout: "bluestore_cache_buffer_items": 0, 2024-04-23T10:41:38.097 INFO:teuthology.orchestra.run.smithi204.stdout: "bluestore_extent_bytes": 0, 2024-04-23T10:41:38.097 INFO:teuthology.orchestra.run.smithi204.stdout: "bluestore_extent_items": 0, 2024-04-23T10:41:38.097 INFO:teuthology.orchestra.run.smithi204.stdout: "bluestore_blob_bytes": 0, 2024-04-23T10:41:38.097 INFO:teuthology.orchestra.run.smithi204.stdout: "bluestore_blob_items": 0, 2024-04-23T10:41:38.097 INFO:teuthology.orchestra.run.smithi204.stdout: "bluestore_shared_blob_bytes": 0, 2024-04-23T10:41:38.098 INFO:teuthology.orchestra.run.smithi204.stdout: "bluestore_shared_blob_items": 0, 2024-04-23T10:41:38.098 INFO:teuthology.orchestra.run.smithi204.stdout: "bluestore_inline_bl_bytes": 0, 2024-04-23T10:41:38.098 INFO:teuthology.orchestra.run.smithi204.stdout: "bluestore_inline_bl_items": 0, 2024-04-23T10:41:38.098 INFO:teuthology.orchestra.run.smithi204.stdout: "bluestore_fsck_bytes": 0, 2024-04-23T10:41:38.098 INFO:teuthology.orchestra.run.smithi204.stdout: "bluestore_fsck_items": 0, 2024-04-23T10:41:38.098 INFO:teuthology.orchestra.run.smithi204.stdout: "bluestore_txc_bytes": 0, 2024-04-23T10:41:38.099 INFO:teuthology.orchestra.run.smithi204.stdout: "bluestore_txc_items": 0, 2024-04-23T10:41:38.099 INFO:teuthology.orchestra.run.smithi204.stdout: "bluestore_writing_deferred_bytes": 0, 2024-04-23T10:41:38.099 INFO:teuthology.orchestra.run.smithi204.stdout: "bluestore_writing_deferred_items": 0, 2024-04-23T10:41:38.099 INFO:teuthology.orchestra.run.smithi204.stdout: "bluestore_writing_bytes": 0, 2024-04-23T10:41:38.099 INFO:teuthology.orchestra.run.smithi204.stdout: "bluestore_writing_items": 0, 2024-04-23T10:41:38.099 INFO:teuthology.orchestra.run.smithi204.stdout: "bluefs_bytes": 0, 2024-04-23T10:41:38.099 INFO:teuthology.orchestra.run.smithi204.stdout: "bluefs_items": 0, 2024-04-23T10:41:38.100 INFO:teuthology.orchestra.run.smithi204.stdout: "bluefs_file_reader_bytes": 0, 2024-04-23T10:41:38.100 INFO:teuthology.orchestra.run.smithi204.stdout: "bluefs_file_reader_items": 0, 2024-04-23T10:41:38.100 INFO:teuthology.orchestra.run.smithi204.stdout: "bluefs_file_writer_bytes": 0, 2024-04-23T10:41:38.100 INFO:teuthology.orchestra.run.smithi204.stdout: "bluefs_file_writer_items": 0, 2024-04-23T10:41:38.100 INFO:teuthology.orchestra.run.smithi204.stdout: "buffer_anon_bytes": 148903, 2024-04-23T10:41:38.100 INFO:teuthology.orchestra.run.smithi204.stdout: "buffer_anon_items": 27, 2024-04-23T10:41:38.101 INFO:teuthology.orchestra.run.smithi204.stdout: "buffer_meta_bytes": 0, 2024-04-23T10:41:38.101 INFO:teuthology.orchestra.run.smithi204.stdout: "buffer_meta_items": 0, 2024-04-23T10:41:38.101 INFO:teuthology.orchestra.run.smithi204.stdout: "osd_bytes": 0, 2024-04-23T10:41:38.101 INFO:teuthology.orchestra.run.smithi204.stdout: "osd_items": 0, 2024-04-23T10:41:38.101 INFO:teuthology.orchestra.run.smithi204.stdout: "osd_mapbl_bytes": 0, 2024-04-23T10:41:38.101 INFO:teuthology.orchestra.run.smithi204.stdout: "osd_mapbl_items": 0, 2024-04-23T10:41:38.102 INFO:teuthology.orchestra.run.smithi204.stdout: "osd_pglog_bytes": 0, 2024-04-23T10:41:38.102 INFO:teuthology.orchestra.run.smithi204.stdout: "osd_pglog_items": 0, 2024-04-23T10:41:38.102 INFO:teuthology.orchestra.run.smithi204.stdout: "osdmap_bytes": 0, 2024-04-23T10:41:38.102 INFO:teuthology.orchestra.run.smithi204.stdout: "osdmap_items": 0, 2024-04-23T10:41:38.102 INFO:teuthology.orchestra.run.smithi204.stdout: "osdmap_mapping_bytes": 0, 2024-04-23T10:41:38.102 INFO:teuthology.orchestra.run.smithi204.stdout: "osdmap_mapping_items": 0, 2024-04-23T10:41:38.103 INFO:teuthology.orchestra.run.smithi204.stdout: "pgmap_bytes": 0, 2024-04-23T10:41:38.103 INFO:teuthology.orchestra.run.smithi204.stdout: "pgmap_items": 0, 2024-04-23T10:41:38.103 INFO:teuthology.orchestra.run.smithi204.stdout: "mds_co_bytes": 928, 2024-04-23T10:41:38.103 INFO:teuthology.orchestra.run.smithi204.stdout: "mds_co_items": 2, 2024-04-23T10:41:38.103 INFO:teuthology.orchestra.run.smithi204.stdout: "unittest_1_bytes": 0, 2024-04-23T10:41:38.103 INFO:teuthology.orchestra.run.smithi204.stdout: "unittest_1_items": 0, 2024-04-23T10:41:38.103 INFO:teuthology.orchestra.run.smithi204.stdout: "unittest_2_bytes": 0, 2024-04-23T10:41:38.104 INFO:teuthology.orchestra.run.smithi204.stdout: "unittest_2_items": 0 2024-04-23T10:41:38.104 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-04-23T10:41:38.104 INFO:teuthology.orchestra.run.smithi204.stdout: "throttle-msgr_dispatch_throttler-mds": { 2024-04-23T10:41:38.104 INFO:teuthology.orchestra.run.smithi204.stdout: "val": 0, 2024-04-23T10:41:38.104 INFO:teuthology.orchestra.run.smithi204.stdout: "max": 104857600, 2024-04-23T10:41:38.104 INFO:teuthology.orchestra.run.smithi204.stdout: "get_started": 0, 2024-04-23T10:41:38.104 INFO:teuthology.orchestra.run.smithi204.stdout: "get": 293, 2024-04-23T10:41:38.105 INFO:teuthology.orchestra.run.smithi204.stdout: "get_sum": 1568113, 2024-04-23T10:41:38.105 INFO:teuthology.orchestra.run.smithi204.stdout: "get_or_fail_fail": 0, 2024-04-23T10:41:38.105 INFO:teuthology.orchestra.run.smithi204.stdout: "get_or_fail_success": 293, 2024-04-23T10:41:38.105 INFO:teuthology.orchestra.run.smithi204.stdout: "take": 0, 2024-04-23T10:41:38.105 INFO:teuthology.orchestra.run.smithi204.stdout: "take_sum": 0, 2024-04-23T10:41:38.105 INFO:teuthology.orchestra.run.smithi204.stdout: "put": 293, 2024-04-23T10:41:38.106 INFO:teuthology.orchestra.run.smithi204.stdout: "put_sum": 1568113, 2024-04-23T10:41:38.106 INFO:teuthology.orchestra.run.smithi204.stdout: "wait": { 2024-04-23T10:41:38.106 INFO:teuthology.orchestra.run.smithi204.stdout: "avgcount": 0, 2024-04-23T10:41:38.106 INFO:teuthology.orchestra.run.smithi204.stdout: "sum": 0.000000000, 2024-04-23T10:41:38.106 INFO:teuthology.orchestra.run.smithi204.stdout: "avgtime": 0.000000000 2024-04-23T10:41:38.106 INFO:teuthology.orchestra.run.smithi204.stdout: } 2024-04-23T10:41:38.106 INFO:teuthology.orchestra.run.smithi204.stdout: } 2024-04-23T10:41:38.107 INFO:teuthology.orchestra.run.smithi204.stdout:} 2024-04-23T10:41:38.795 WARNING:tasks.check_counter:Counter 'mds.exported' not found on daemon mds.f 2024-04-23T10:41:38.795 WARNING:tasks.check_counter:Counter 'mds.imported' not found on daemon mds.f 2024-04-23T10:41:38.795 WARNING:tasks.check_counter:Counter 'mds_cache.dir_update' not found on daemon mds.f 2024-04-23T10:41:38.795 WARNING:tasks.check_counter:Counter 'mds_cache.dir_update_receipt' not found on daemon mds.f 2024-04-23T10:41:38.795 WARNING:tasks.check_counter:Counter 'mds.root_rsnaps' not found on daemon mds.f 2024-04-23T10:41:38.796 WARNING:tasks.check_counter:Counter 'mds_server.req_mksnap_latency.avgcount' not found on daemon mds.f 2024-04-23T10:41:38.796 WARNING:tasks.check_counter:Counter 'mds_server.req_rmsnap_latency.avgcount' not found on daemon mds.f 2024-04-23T10:41:38.796 DEBUG:tasks.check_counter:Getting stats from i 2024-04-23T10:41:38.796 DEBUG:teuthology.orchestra.run.smithi204:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d69f13d827952bb07bb9b7973bca14f97d764d6f shell --fsid aa240572-0159-11ef-bc93-c7b262605968 -- ceph daemon mds.i perf dump 2024-04-23T10:41:39.036 INFO:teuthology.orchestra.run.smithi204.stderr:Inferring config /var/lib/ceph/aa240572-0159-11ef-bc93-c7b262605968/mon.c/config 2024-04-23T10:41:39.828 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 23 10:41:39 smithi204 ceph-mon[32649]: pgmap v475: 97 pgs: 97 active+clean; 551 MiB data, 4.3 GiB used, 1.0 TiB / 1.0 TiB avail; 487 KiB/s rd, 3.1 MiB/s wr, 113 op/s 2024-04-23T10:41:39.992 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 23 10:41:39 smithi119 ceph-mon[32608]: pgmap v475: 97 pgs: 97 active+clean; 551 MiB data, 4.3 GiB used, 1.0 TiB / 1.0 TiB avail; 487 KiB/s rd, 3.1 MiB/s wr, 113 op/s 2024-04-23T10:41:40.004 INFO:journalctl@ceph.mon.a.smithi114.stdout:Apr 23 10:41:39 smithi114 ceph-mon[29255]: pgmap v475: 97 pgs: 97 active+clean; 551 MiB data, 4.3 GiB used, 1.0 TiB / 1.0 TiB avail; 487 KiB/s rd, 3.1 MiB/s wr, 113 op/s 2024-04-23T10:41:40.979 INFO:teuthology.orchestra.run.smithi204.stdout:{ 2024-04-23T10:41:40.979 INFO:teuthology.orchestra.run.smithi204.stdout: "AsyncMessenger::Worker-0": { 2024-04-23T10:41:40.980 INFO:teuthology.orchestra.run.smithi204.stdout: "msgr_recv_messages": 158, 2024-04-23T10:41:40.980 INFO:teuthology.orchestra.run.smithi204.stdout: "msgr_send_messages": 158, 2024-04-23T10:41:40.980 INFO:teuthology.orchestra.run.smithi204.stdout: "msgr_recv_bytes": 29412614, 2024-04-23T10:41:40.980 INFO:teuthology.orchestra.run.smithi204.stdout: "msgr_send_bytes": 49436, 2024-04-23T10:41:40.980 INFO:teuthology.orchestra.run.smithi204.stdout: "msgr_created_connections": 7, 2024-04-23T10:41:40.980 INFO:teuthology.orchestra.run.smithi204.stdout: "msgr_active_connections": 3, 2024-04-23T10:41:40.981 INFO:teuthology.orchestra.run.smithi204.stdout: "msgr_running_total_time": 0.022596950, 2024-04-23T10:41:40.981 INFO:teuthology.orchestra.run.smithi204.stdout: "msgr_running_send_time": 0.003583121, 2024-04-23T10:41:40.981 INFO:teuthology.orchestra.run.smithi204.stdout: "msgr_running_recv_time": 0.013988626, 2024-04-23T10:41:40.981 INFO:teuthology.orchestra.run.smithi204.stdout: "msgr_running_fast_dispatch_time": 0.003311838, 2024-04-23T10:41:40.981 INFO:teuthology.orchestra.run.smithi204.stdout: "msgr_send_messages_queue_lat": { 2024-04-23T10:41:40.981 INFO:teuthology.orchestra.run.smithi204.stdout: "avgcount": 157, 2024-04-23T10:41:40.982 INFO:teuthology.orchestra.run.smithi204.stdout: "sum": 0.006527193, 2024-04-23T10:41:40.982 INFO:teuthology.orchestra.run.smithi204.stdout: "avgtime": 0.000041574 2024-04-23T10:41:40.982 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-04-23T10:41:40.982 INFO:teuthology.orchestra.run.smithi204.stdout: "msgr_handle_ack_lat": { 2024-04-23T10:41:40.982 INFO:teuthology.orchestra.run.smithi204.stdout: "avgcount": 0, 2024-04-23T10:41:40.982 INFO:teuthology.orchestra.run.smithi204.stdout: "sum": 0.000000000, 2024-04-23T10:41:40.983 INFO:teuthology.orchestra.run.smithi204.stdout: "avgtime": 0.000000000 2024-04-23T10:41:40.983 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-04-23T10:41:40.983 INFO:teuthology.orchestra.run.smithi204.stdout: "msgr_recv_encrypted_bytes": 2992, 2024-04-23T10:41:40.983 INFO:teuthology.orchestra.run.smithi204.stdout: "msgr_send_encrypted_bytes": 448 2024-04-23T10:41:40.983 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-04-23T10:41:40.983 INFO:teuthology.orchestra.run.smithi204.stdout: "AsyncMessenger::Worker-1": { 2024-04-23T10:41:40.983 INFO:teuthology.orchestra.run.smithi204.stdout: "msgr_recv_messages": 224, 2024-04-23T10:41:40.984 INFO:teuthology.orchestra.run.smithi204.stdout: "msgr_send_messages": 245, 2024-04-23T10:41:40.984 INFO:teuthology.orchestra.run.smithi204.stdout: "msgr_recv_bytes": 26791665, 2024-04-23T10:41:40.984 INFO:teuthology.orchestra.run.smithi204.stdout: "msgr_send_bytes": 192783, 2024-04-23T10:41:40.984 INFO:teuthology.orchestra.run.smithi204.stdout: "msgr_created_connections": 6, 2024-04-23T10:41:40.984 INFO:teuthology.orchestra.run.smithi204.stdout: "msgr_active_connections": 5, 2024-04-23T10:41:40.984 INFO:teuthology.orchestra.run.smithi204.stdout: "msgr_running_total_time": 0.031338747, 2024-04-23T10:41:40.985 INFO:teuthology.orchestra.run.smithi204.stdout: "msgr_running_send_time": 0.006184813, 2024-04-23T10:41:40.985 INFO:teuthology.orchestra.run.smithi204.stdout: "msgr_running_recv_time": 0.017515299, 2024-04-23T10:41:40.985 INFO:teuthology.orchestra.run.smithi204.stdout: "msgr_running_fast_dispatch_time": 0.005790106, 2024-04-23T10:41:40.985 INFO:teuthology.orchestra.run.smithi204.stdout: "msgr_send_messages_queue_lat": { 2024-04-23T10:41:40.985 INFO:teuthology.orchestra.run.smithi204.stdout: "avgcount": 243, 2024-04-23T10:41:40.985 INFO:teuthology.orchestra.run.smithi204.stdout: "sum": 0.008797486, 2024-04-23T10:41:40.986 INFO:teuthology.orchestra.run.smithi204.stdout: "avgtime": 0.000036203 2024-04-23T10:41:40.986 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-04-23T10:41:40.986 INFO:teuthology.orchestra.run.smithi204.stdout: "msgr_handle_ack_lat": { 2024-04-23T10:41:40.986 INFO:teuthology.orchestra.run.smithi204.stdout: "avgcount": 0, 2024-04-23T10:41:40.986 INFO:teuthology.orchestra.run.smithi204.stdout: "sum": 0.000000000, 2024-04-23T10:41:40.986 INFO:teuthology.orchestra.run.smithi204.stdout: "avgtime": 0.000000000 2024-04-23T10:41:40.987 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-04-23T10:41:40.987 INFO:teuthology.orchestra.run.smithi204.stdout: "msgr_recv_encrypted_bytes": 160, 2024-04-23T10:41:40.987 INFO:teuthology.orchestra.run.smithi204.stdout: "msgr_send_encrypted_bytes": 121808 2024-04-23T10:41:40.987 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-04-23T10:41:40.987 INFO:teuthology.orchestra.run.smithi204.stdout: "AsyncMessenger::Worker-2": { 2024-04-23T10:41:40.987 INFO:teuthology.orchestra.run.smithi204.stdout: "msgr_recv_messages": 71, 2024-04-23T10:41:40.988 INFO:teuthology.orchestra.run.smithi204.stdout: "msgr_send_messages": 68, 2024-04-23T10:41:40.988 INFO:teuthology.orchestra.run.smithi204.stdout: "msgr_recv_bytes": 7884486, 2024-04-23T10:41:40.988 INFO:teuthology.orchestra.run.smithi204.stdout: "msgr_send_bytes": 27380, 2024-04-23T10:41:40.988 INFO:teuthology.orchestra.run.smithi204.stdout: "msgr_created_connections": 4, 2024-04-23T10:41:40.988 INFO:teuthology.orchestra.run.smithi204.stdout: "msgr_active_connections": 3, 2024-04-23T10:41:40.988 INFO:teuthology.orchestra.run.smithi204.stdout: "msgr_running_total_time": 0.011339550, 2024-04-23T10:41:40.989 INFO:teuthology.orchestra.run.smithi204.stdout: "msgr_running_send_time": 0.002078150, 2024-04-23T10:41:40.989 INFO:teuthology.orchestra.run.smithi204.stdout: "msgr_running_recv_time": 0.008857981, 2024-04-23T10:41:40.989 INFO:teuthology.orchestra.run.smithi204.stdout: "msgr_running_fast_dispatch_time": 0.001035181, 2024-04-23T10:41:40.989 INFO:teuthology.orchestra.run.smithi204.stdout: "msgr_send_messages_queue_lat": { 2024-04-23T10:41:40.989 INFO:teuthology.orchestra.run.smithi204.stdout: "avgcount": 67, 2024-04-23T10:41:40.989 INFO:teuthology.orchestra.run.smithi204.stdout: "sum": 0.003266294, 2024-04-23T10:41:40.990 INFO:teuthology.orchestra.run.smithi204.stdout: "avgtime": 0.000048750 2024-04-23T10:41:40.990 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-04-23T10:41:40.990 INFO:teuthology.orchestra.run.smithi204.stdout: "msgr_handle_ack_lat": { 2024-04-23T10:41:40.990 INFO:teuthology.orchestra.run.smithi204.stdout: "avgcount": 0, 2024-04-23T10:41:40.990 INFO:teuthology.orchestra.run.smithi204.stdout: "sum": 0.000000000, 2024-04-23T10:41:40.990 INFO:teuthology.orchestra.run.smithi204.stdout: "avgtime": 0.000000000 2024-04-23T10:41:40.991 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-04-23T10:41:40.991 INFO:teuthology.orchestra.run.smithi204.stdout: "msgr_recv_encrypted_bytes": 173600, 2024-04-23T10:41:40.991 INFO:teuthology.orchestra.run.smithi204.stdout: "msgr_send_encrypted_bytes": 17840 2024-04-23T10:41:40.991 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-04-23T10:41:40.991 INFO:teuthology.orchestra.run.smithi204.stdout: "cct": { 2024-04-23T10:41:40.991 INFO:teuthology.orchestra.run.smithi204.stdout: "total_workers": 1, 2024-04-23T10:41:40.992 INFO:teuthology.orchestra.run.smithi204.stdout: "unhealthy_workers": 0 2024-04-23T10:41:40.992 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-04-23T10:41:40.992 INFO:teuthology.orchestra.run.smithi204.stdout: "finisher-MDSRank": { 2024-04-23T10:41:40.992 INFO:teuthology.orchestra.run.smithi204.stdout: "queue_len": 0, 2024-04-23T10:41:40.992 INFO:teuthology.orchestra.run.smithi204.stdout: "complete_latency": { 2024-04-23T10:41:40.992 INFO:teuthology.orchestra.run.smithi204.stdout: "avgcount": 650, 2024-04-23T10:41:40.993 INFO:teuthology.orchestra.run.smithi204.stdout: "sum": 0.046134641, 2024-04-23T10:41:40.993 INFO:teuthology.orchestra.run.smithi204.stdout: "avgtime": 0.000070976 2024-04-23T10:41:40.993 INFO:teuthology.orchestra.run.smithi204.stdout: } 2024-04-23T10:41:40.993 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-04-23T10:41:40.993 INFO:teuthology.orchestra.run.smithi204.stdout: "finisher-PurgeQueue": { 2024-04-23T10:41:40.993 INFO:teuthology.orchestra.run.smithi204.stdout: "queue_len": 0, 2024-04-23T10:41:40.994 INFO:teuthology.orchestra.run.smithi204.stdout: "complete_latency": { 2024-04-23T10:41:40.994 INFO:teuthology.orchestra.run.smithi204.stdout: "avgcount": 0, 2024-04-23T10:41:40.994 INFO:teuthology.orchestra.run.smithi204.stdout: "sum": 0.000000000, 2024-04-23T10:41:40.994 INFO:teuthology.orchestra.run.smithi204.stdout: "avgtime": 0.000000000 2024-04-23T10:41:40.994 INFO:teuthology.orchestra.run.smithi204.stdout: } 2024-04-23T10:41:40.994 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-04-23T10:41:40.994 INFO:teuthology.orchestra.run.smithi204.stdout: "mds": { 2024-04-23T10:41:40.995 INFO:teuthology.orchestra.run.smithi204.stdout: "request": 0, 2024-04-23T10:41:40.995 INFO:teuthology.orchestra.run.smithi204.stdout: "reply": 0, 2024-04-23T10:41:40.995 INFO:teuthology.orchestra.run.smithi204.stdout: "reply_latency": { 2024-04-23T10:41:40.995 INFO:teuthology.orchestra.run.smithi204.stdout: "avgcount": 0, 2024-04-23T10:41:40.995 INFO:teuthology.orchestra.run.smithi204.stdout: "sum": 0.000000000, 2024-04-23T10:41:40.995 INFO:teuthology.orchestra.run.smithi204.stdout: "avgtime": 0.000000000 2024-04-23T10:41:40.996 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-04-23T10:41:40.996 INFO:teuthology.orchestra.run.smithi204.stdout: "slow_reply": 0, 2024-04-23T10:41:40.996 INFO:teuthology.orchestra.run.smithi204.stdout: "forward": 0, 2024-04-23T10:41:40.996 INFO:teuthology.orchestra.run.smithi204.stdout: "dir_fetch_complete": 0, 2024-04-23T10:41:40.996 INFO:teuthology.orchestra.run.smithi204.stdout: "dir_fetch_keys": 0, 2024-04-23T10:41:40.996 INFO:teuthology.orchestra.run.smithi204.stdout: "dir_commit": 0, 2024-04-23T10:41:40.996 INFO:teuthology.orchestra.run.smithi204.stdout: "dir_split": 0, 2024-04-23T10:41:40.997 INFO:teuthology.orchestra.run.smithi204.stdout: "dir_merge": 0, 2024-04-23T10:41:40.997 INFO:teuthology.orchestra.run.smithi204.stdout: "inodes": 5973, 2024-04-23T10:41:40.997 INFO:teuthology.orchestra.run.smithi204.stdout: "inodes_top": 3399, 2024-04-23T10:41:40.997 INFO:teuthology.orchestra.run.smithi204.stdout: "inodes_bottom": 2574, 2024-04-23T10:41:40.997 INFO:teuthology.orchestra.run.smithi204.stdout: "inodes_pin_tail": 0, 2024-04-23T10:41:40.998 INFO:teuthology.orchestra.run.smithi204.stdout: "inodes_pinned": 1117, 2024-04-23T10:41:40.998 INFO:teuthology.orchestra.run.smithi204.stdout: "inodes_expired": 0, 2024-04-23T10:41:40.998 INFO:teuthology.orchestra.run.smithi204.stdout: "inodes_with_caps": 0, 2024-04-23T10:41:40.998 INFO:teuthology.orchestra.run.smithi204.stdout: "caps": 0, 2024-04-23T10:41:40.998 INFO:teuthology.orchestra.run.smithi204.stdout: "subtrees": 12, 2024-04-23T10:41:40.998 INFO:teuthology.orchestra.run.smithi204.stdout: "traverse": 0, 2024-04-23T10:41:40.999 INFO:teuthology.orchestra.run.smithi204.stdout: "traverse_hit": 0, 2024-04-23T10:41:40.999 INFO:teuthology.orchestra.run.smithi204.stdout: "traverse_forward": 0, 2024-04-23T10:41:40.999 INFO:teuthology.orchestra.run.smithi204.stdout: "traverse_discover": 0, 2024-04-23T10:41:40.999 INFO:teuthology.orchestra.run.smithi204.stdout: "traverse_dir_fetch": 0, 2024-04-23T10:41:40.999 INFO:teuthology.orchestra.run.smithi204.stdout: "traverse_remote_ino": 0, 2024-04-23T10:41:40.999 INFO:teuthology.orchestra.run.smithi204.stdout: "traverse_lock": 0, 2024-04-23T10:41:41.000 INFO:teuthology.orchestra.run.smithi204.stdout: "load_cent": 0, 2024-04-23T10:41:41.000 INFO:teuthology.orchestra.run.smithi204.stdout: "q": 0, 2024-04-23T10:41:41.000 INFO:teuthology.orchestra.run.smithi204.stdout: "exported": 0, 2024-04-23T10:41:41.000 INFO:teuthology.orchestra.run.smithi204.stdout: "exported_inodes": 0, 2024-04-23T10:41:41.000 INFO:teuthology.orchestra.run.smithi204.stdout: "imported": 0, 2024-04-23T10:41:41.000 INFO:teuthology.orchestra.run.smithi204.stdout: "imported_inodes": 0, 2024-04-23T10:41:41.000 INFO:teuthology.orchestra.run.smithi204.stdout: "openino_dir_fetch": 0, 2024-04-23T10:41:41.001 INFO:teuthology.orchestra.run.smithi204.stdout: "openino_backtrace_fetch": 0, 2024-04-23T10:41:41.001 INFO:teuthology.orchestra.run.smithi204.stdout: "openino_peer_discover": 0, 2024-04-23T10:41:41.001 INFO:teuthology.orchestra.run.smithi204.stdout: "root_rfiles": 733, 2024-04-23T10:41:41.001 INFO:teuthology.orchestra.run.smithi204.stdout: "root_rbytes": 217201178, 2024-04-23T10:41:41.001 INFO:teuthology.orchestra.run.smithi204.stdout: "root_rsnaps": 6, 2024-04-23T10:41:41.001 INFO:teuthology.orchestra.run.smithi204.stdout: "scrub_backtrace_fetch": 0, 2024-04-23T10:41:41.002 INFO:teuthology.orchestra.run.smithi204.stdout: "scrub_set_tag": 0, 2024-04-23T10:41:41.002 INFO:teuthology.orchestra.run.smithi204.stdout: "scrub_backtrace_repaired": 0, 2024-04-23T10:41:41.002 INFO:teuthology.orchestra.run.smithi204.stdout: "scrub_inotable_repaired": 0, 2024-04-23T10:41:41.002 INFO:teuthology.orchestra.run.smithi204.stdout: "scrub_dir_inodes": 0, 2024-04-23T10:41:41.002 INFO:teuthology.orchestra.run.smithi204.stdout: "scrub_dir_base_inodes": 0, 2024-04-23T10:41:41.002 INFO:teuthology.orchestra.run.smithi204.stdout: "scrub_dirfrag_rstats": 0, 2024-04-23T10:41:41.002 INFO:teuthology.orchestra.run.smithi204.stdout: "scrub_file_inodes": 0, 2024-04-23T10:41:41.003 INFO:teuthology.orchestra.run.smithi204.stdout: "handle_inode_file_caps": 0, 2024-04-23T10:41:41.003 INFO:teuthology.orchestra.run.smithi204.stdout: "ceph_cap_op_revoke": 0, 2024-04-23T10:41:41.003 INFO:teuthology.orchestra.run.smithi204.stdout: "ceph_cap_op_grant": 0, 2024-04-23T10:41:41.003 INFO:teuthology.orchestra.run.smithi204.stdout: "ceph_cap_op_trunc": 0, 2024-04-23T10:41:41.003 INFO:teuthology.orchestra.run.smithi204.stdout: "ceph_cap_op_flushsnap_ack": 0, 2024-04-23T10:41:41.003 INFO:teuthology.orchestra.run.smithi204.stdout: "ceph_cap_op_flush_ack": 0, 2024-04-23T10:41:41.003 INFO:teuthology.orchestra.run.smithi204.stdout: "handle_client_caps": 0, 2024-04-23T10:41:41.004 INFO:teuthology.orchestra.run.smithi204.stdout: "handle_client_caps_dirty": 0, 2024-04-23T10:41:41.004 INFO:teuthology.orchestra.run.smithi204.stdout: "handle_client_cap_release": 0, 2024-04-23T10:41:41.004 INFO:teuthology.orchestra.run.smithi204.stdout: "process_request_cap_release": 0 2024-04-23T10:41:41.004 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-04-23T10:41:41.004 INFO:teuthology.orchestra.run.smithi204.stdout: "mds_cache": { 2024-04-23T10:41:41.004 INFO:teuthology.orchestra.run.smithi204.stdout: "dir_update": 0, 2024-04-23T10:41:41.004 INFO:teuthology.orchestra.run.smithi204.stdout: "dir_update_receipt": 0, 2024-04-23T10:41:41.005 INFO:teuthology.orchestra.run.smithi204.stdout: "dir_try_discover": 0, 2024-04-23T10:41:41.005 INFO:teuthology.orchestra.run.smithi204.stdout: "dir_send_discover": 0, 2024-04-23T10:41:41.005 INFO:teuthology.orchestra.run.smithi204.stdout: "dir_handle_discover": 0, 2024-04-23T10:41:41.005 INFO:teuthology.orchestra.run.smithi204.stdout: "num_strays": 0, 2024-04-23T10:41:41.005 INFO:teuthology.orchestra.run.smithi204.stdout: "num_strays_delayed": 0, 2024-04-23T10:41:41.005 INFO:teuthology.orchestra.run.smithi204.stdout: "num_strays_enqueuing": 0, 2024-04-23T10:41:41.005 INFO:teuthology.orchestra.run.smithi204.stdout: "strays_created": 0, 2024-04-23T10:41:41.006 INFO:teuthology.orchestra.run.smithi204.stdout: "strays_enqueued": 0, 2024-04-23T10:41:41.006 INFO:teuthology.orchestra.run.smithi204.stdout: "strays_reintegrated": 0, 2024-04-23T10:41:41.006 INFO:teuthology.orchestra.run.smithi204.stdout: "strays_migrated": 0, 2024-04-23T10:41:41.006 INFO:teuthology.orchestra.run.smithi204.stdout: "num_recovering_processing": 0, 2024-04-23T10:41:41.006 INFO:teuthology.orchestra.run.smithi204.stdout: "num_recovering_enqueued": 0, 2024-04-23T10:41:41.006 INFO:teuthology.orchestra.run.smithi204.stdout: "num_recovering_prioritized": 0, 2024-04-23T10:41:41.007 INFO:teuthology.orchestra.run.smithi204.stdout: "recovery_started": 0, 2024-04-23T10:41:41.007 INFO:teuthology.orchestra.run.smithi204.stdout: "recovery_completed": 0, 2024-04-23T10:41:41.007 INFO:teuthology.orchestra.run.smithi204.stdout: "ireq_quiesce_path": 0, 2024-04-23T10:41:41.007 INFO:teuthology.orchestra.run.smithi204.stdout: "ireq_quiesce_inode": 0, 2024-04-23T10:41:41.007 INFO:teuthology.orchestra.run.smithi204.stdout: "ireq_enqueue_scrub": 0, 2024-04-23T10:41:41.007 INFO:teuthology.orchestra.run.smithi204.stdout: "ireq_exportdir": 0, 2024-04-23T10:41:41.007 INFO:teuthology.orchestra.run.smithi204.stdout: "ireq_flush": 0, 2024-04-23T10:41:41.008 INFO:teuthology.orchestra.run.smithi204.stdout: "ireq_fragmentdir": 0, 2024-04-23T10:41:41.008 INFO:teuthology.orchestra.run.smithi204.stdout: "ireq_fragstats": 0, 2024-04-23T10:41:41.008 INFO:teuthology.orchestra.run.smithi204.stdout: "ireq_inodestats": 0 2024-04-23T10:41:41.008 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-04-23T10:41:41.008 INFO:teuthology.orchestra.run.smithi204.stdout: "mds_log": { 2024-04-23T10:41:41.008 INFO:teuthology.orchestra.run.smithi204.stdout: "evlrg": 0, 2024-04-23T10:41:41.008 INFO:teuthology.orchestra.run.smithi204.stdout: "evadd": 0, 2024-04-23T10:41:41.009 INFO:teuthology.orchestra.run.smithi204.stdout: "evex": 0, 2024-04-23T10:41:41.009 INFO:teuthology.orchestra.run.smithi204.stdout: "evtrm": 0, 2024-04-23T10:41:41.009 INFO:teuthology.orchestra.run.smithi204.stdout: "ev": 22437, 2024-04-23T10:41:41.009 INFO:teuthology.orchestra.run.smithi204.stdout: "evexg": 0, 2024-04-23T10:41:41.009 INFO:teuthology.orchestra.run.smithi204.stdout: "evexd": 0, 2024-04-23T10:41:41.009 INFO:teuthology.orchestra.run.smithi204.stdout: "segadd": 0, 2024-04-23T10:41:41.009 INFO:teuthology.orchestra.run.smithi204.stdout: "segex": 0, 2024-04-23T10:41:41.010 INFO:teuthology.orchestra.run.smithi204.stdout: "segtrm": 0, 2024-04-23T10:41:41.010 INFO:teuthology.orchestra.run.smithi204.stdout: "seg": 3, 2024-04-23T10:41:41.010 INFO:teuthology.orchestra.run.smithi204.stdout: "segmjr": 5, 2024-04-23T10:41:41.010 INFO:teuthology.orchestra.run.smithi204.stdout: "segexg": 0, 2024-04-23T10:41:41.010 INFO:teuthology.orchestra.run.smithi204.stdout: "segexd": 0, 2024-04-23T10:41:41.010 INFO:teuthology.orchestra.run.smithi204.stdout: "expos": 412041178, 2024-04-23T10:41:41.011 INFO:teuthology.orchestra.run.smithi204.stdout: "wrpos": 416782532, 2024-04-23T10:41:41.011 INFO:teuthology.orchestra.run.smithi204.stdout: "rdpos": 416771608, 2024-04-23T10:41:41.011 INFO:teuthology.orchestra.run.smithi204.stdout: "jlat": { 2024-04-23T10:41:41.011 INFO:teuthology.orchestra.run.smithi204.stdout: "avgcount": 0, 2024-04-23T10:41:41.011 INFO:teuthology.orchestra.run.smithi204.stdout: "sum": 0.000000000, 2024-04-23T10:41:41.011 INFO:teuthology.orchestra.run.smithi204.stdout: "avgtime": 0.000000000 2024-04-23T10:41:41.011 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-04-23T10:41:41.012 INFO:teuthology.orchestra.run.smithi204.stdout: "replayed": 22437 2024-04-23T10:41:41.012 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-04-23T10:41:41.012 INFO:teuthology.orchestra.run.smithi204.stdout: "mds_mem": { 2024-04-23T10:41:41.012 INFO:teuthology.orchestra.run.smithi204.stdout: "ino": 707, 2024-04-23T10:41:41.012 INFO:teuthology.orchestra.run.smithi204.stdout: "ino+": 5917, 2024-04-23T10:41:41.012 INFO:teuthology.orchestra.run.smithi204.stdout: "ino-": 5210, 2024-04-23T10:41:41.012 INFO:teuthology.orchestra.run.smithi204.stdout: "dir": 131, 2024-04-23T10:41:41.013 INFO:teuthology.orchestra.run.smithi204.stdout: "dir+": 204, 2024-04-23T10:41:41.013 INFO:teuthology.orchestra.run.smithi204.stdout: "dir-": 73, 2024-04-23T10:41:41.013 INFO:teuthology.orchestra.run.smithi204.stdout: "dn": 5973, 2024-04-23T10:41:41.013 INFO:teuthology.orchestra.run.smithi204.stdout: "dn+": 6587, 2024-04-23T10:41:41.013 INFO:teuthology.orchestra.run.smithi204.stdout: "dn-": 614, 2024-04-23T10:41:41.013 INFO:teuthology.orchestra.run.smithi204.stdout: "cap": 0, 2024-04-23T10:41:41.013 INFO:teuthology.orchestra.run.smithi204.stdout: "cap+": 0, 2024-04-23T10:41:41.013 INFO:teuthology.orchestra.run.smithi204.stdout: "cap-": 0, 2024-04-23T10:41:41.014 INFO:teuthology.orchestra.run.smithi204.stdout: "rss": 61084, 2024-04-23T10:41:41.014 INFO:teuthology.orchestra.run.smithi204.stdout: "heap": 215392 2024-04-23T10:41:41.014 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-04-23T10:41:41.014 INFO:teuthology.orchestra.run.smithi204.stdout: "mds_server": { 2024-04-23T10:41:41.014 INFO:teuthology.orchestra.run.smithi204.stdout: "dispatch_client_request": 0, 2024-04-23T10:41:41.014 INFO:teuthology.orchestra.run.smithi204.stdout: "dispatch_server_request": 0, 2024-04-23T10:41:41.014 INFO:teuthology.orchestra.run.smithi204.stdout: "handle_client_request": 0, 2024-04-23T10:41:41.015 INFO:teuthology.orchestra.run.smithi204.stdout: "handle_client_session": 0, 2024-04-23T10:41:41.015 INFO:teuthology.orchestra.run.smithi204.stdout: "handle_peer_request": 0, 2024-04-23T10:41:41.015 INFO:teuthology.orchestra.run.smithi204.stdout: "req_create_latency": { 2024-04-23T10:41:41.015 INFO:teuthology.orchestra.run.smithi204.stdout: "avgcount": 0, 2024-04-23T10:41:41.015 INFO:teuthology.orchestra.run.smithi204.stdout: "sum": 0.000000000, 2024-04-23T10:41:41.015 INFO:teuthology.orchestra.run.smithi204.stdout: "avgtime": 0.000000000 2024-04-23T10:41:41.015 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-04-23T10:41:41.016 INFO:teuthology.orchestra.run.smithi204.stdout: "req_getattr_latency": { 2024-04-23T10:41:41.016 INFO:teuthology.orchestra.run.smithi204.stdout: "avgcount": 0, 2024-04-23T10:41:41.016 INFO:teuthology.orchestra.run.smithi204.stdout: "sum": 0.000000000, 2024-04-23T10:41:41.016 INFO:teuthology.orchestra.run.smithi204.stdout: "avgtime": 0.000000000 2024-04-23T10:41:41.016 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-04-23T10:41:41.016 INFO:teuthology.orchestra.run.smithi204.stdout: "req_getfilelock_latency": { 2024-04-23T10:41:41.016 INFO:teuthology.orchestra.run.smithi204.stdout: "avgcount": 0, 2024-04-23T10:41:41.017 INFO:teuthology.orchestra.run.smithi204.stdout: "sum": 0.000000000, 2024-04-23T10:41:41.017 INFO:teuthology.orchestra.run.smithi204.stdout: "avgtime": 0.000000000 2024-04-23T10:41:41.017 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-04-23T10:41:41.017 INFO:teuthology.orchestra.run.smithi204.stdout: "req_link_latency": { 2024-04-23T10:41:41.017 INFO:teuthology.orchestra.run.smithi204.stdout: "avgcount": 0, 2024-04-23T10:41:41.017 INFO:teuthology.orchestra.run.smithi204.stdout: "sum": 0.000000000, 2024-04-23T10:41:41.017 INFO:teuthology.orchestra.run.smithi204.stdout: "avgtime": 0.000000000 2024-04-23T10:41:41.018 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-04-23T10:41:41.018 INFO:teuthology.orchestra.run.smithi204.stdout: "req_lookup_latency": { 2024-04-23T10:41:41.018 INFO:teuthology.orchestra.run.smithi204.stdout: "avgcount": 0, 2024-04-23T10:41:41.018 INFO:teuthology.orchestra.run.smithi204.stdout: "sum": 0.000000000, 2024-04-23T10:41:41.018 INFO:teuthology.orchestra.run.smithi204.stdout: "avgtime": 0.000000000 2024-04-23T10:41:41.018 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-04-23T10:41:41.018 INFO:teuthology.orchestra.run.smithi204.stdout: "req_lookuphash_latency": { 2024-04-23T10:41:41.019 INFO:teuthology.orchestra.run.smithi204.stdout: "avgcount": 0, 2024-04-23T10:41:41.019 INFO:teuthology.orchestra.run.smithi204.stdout: "sum": 0.000000000, 2024-04-23T10:41:41.019 INFO:teuthology.orchestra.run.smithi204.stdout: "avgtime": 0.000000000 2024-04-23T10:41:41.019 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-04-23T10:41:41.019 INFO:teuthology.orchestra.run.smithi204.stdout: "req_lookupino_latency": { 2024-04-23T10:41:41.019 INFO:teuthology.orchestra.run.smithi204.stdout: "avgcount": 0, 2024-04-23T10:41:41.019 INFO:teuthology.orchestra.run.smithi204.stdout: "sum": 0.000000000, 2024-04-23T10:41:41.020 INFO:teuthology.orchestra.run.smithi204.stdout: "avgtime": 0.000000000 2024-04-23T10:41:41.020 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-04-23T10:41:41.020 INFO:teuthology.orchestra.run.smithi204.stdout: "req_lookupname_latency": { 2024-04-23T10:41:41.020 INFO:teuthology.orchestra.run.smithi204.stdout: "avgcount": 0, 2024-04-23T10:41:41.020 INFO:teuthology.orchestra.run.smithi204.stdout: "sum": 0.000000000, 2024-04-23T10:41:41.020 INFO:teuthology.orchestra.run.smithi204.stdout: "avgtime": 0.000000000 2024-04-23T10:41:41.021 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-04-23T10:41:41.021 INFO:teuthology.orchestra.run.smithi204.stdout: "req_lookupparent_latency": { 2024-04-23T10:41:41.021 INFO:teuthology.orchestra.run.smithi204.stdout: "avgcount": 0, 2024-04-23T10:41:41.021 INFO:teuthology.orchestra.run.smithi204.stdout: "sum": 0.000000000, 2024-04-23T10:41:41.021 INFO:teuthology.orchestra.run.smithi204.stdout: "avgtime": 0.000000000 2024-04-23T10:41:41.021 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-04-23T10:41:41.021 INFO:teuthology.orchestra.run.smithi204.stdout: "req_lookupsnap_latency": { 2024-04-23T10:41:41.021 INFO:teuthology.orchestra.run.smithi204.stdout: "avgcount": 0, 2024-04-23T10:41:41.022 INFO:teuthology.orchestra.run.smithi204.stdout: "sum": 0.000000000, 2024-04-23T10:41:41.022 INFO:teuthology.orchestra.run.smithi204.stdout: "avgtime": 0.000000000 2024-04-23T10:41:41.022 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-04-23T10:41:41.022 INFO:teuthology.orchestra.run.smithi204.stdout: "req_lssnap_latency": { 2024-04-23T10:41:41.022 INFO:teuthology.orchestra.run.smithi204.stdout: "avgcount": 0, 2024-04-23T10:41:41.022 INFO:teuthology.orchestra.run.smithi204.stdout: "sum": 0.000000000, 2024-04-23T10:41:41.023 INFO:teuthology.orchestra.run.smithi204.stdout: "avgtime": 0.000000000 2024-04-23T10:41:41.023 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-04-23T10:41:41.024 INFO:teuthology.orchestra.run.smithi204.stdout: "req_mkdir_latency": { 2024-04-23T10:41:41.024 INFO:teuthology.orchestra.run.smithi204.stdout: "avgcount": 0, 2024-04-23T10:41:41.024 INFO:teuthology.orchestra.run.smithi204.stdout: "sum": 0.000000000, 2024-04-23T10:41:41.024 INFO:teuthology.orchestra.run.smithi204.stdout: "avgtime": 0.000000000 2024-04-23T10:41:41.024 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-04-23T10:41:41.024 INFO:teuthology.orchestra.run.smithi204.stdout: "req_mknod_latency": { 2024-04-23T10:41:41.025 INFO:teuthology.orchestra.run.smithi204.stdout: "avgcount": 0, 2024-04-23T10:41:41.025 INFO:teuthology.orchestra.run.smithi204.stdout: "sum": 0.000000000, 2024-04-23T10:41:41.025 INFO:teuthology.orchestra.run.smithi204.stdout: "avgtime": 0.000000000 2024-04-23T10:41:41.025 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-04-23T10:41:41.025 INFO:teuthology.orchestra.run.smithi204.stdout: "req_mksnap_latency": { 2024-04-23T10:41:41.025 INFO:teuthology.orchestra.run.smithi204.stdout: "avgcount": 0, 2024-04-23T10:41:41.025 INFO:teuthology.orchestra.run.smithi204.stdout: "sum": 0.000000000, 2024-04-23T10:41:41.026 INFO:teuthology.orchestra.run.smithi204.stdout: "avgtime": 0.000000000 2024-04-23T10:41:41.026 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-04-23T10:41:41.026 INFO:teuthology.orchestra.run.smithi204.stdout: "req_open_latency": { 2024-04-23T10:41:41.026 INFO:teuthology.orchestra.run.smithi204.stdout: "avgcount": 0, 2024-04-23T10:41:41.026 INFO:teuthology.orchestra.run.smithi204.stdout: "sum": 0.000000000, 2024-04-23T10:41:41.026 INFO:teuthology.orchestra.run.smithi204.stdout: "avgtime": 0.000000000 2024-04-23T10:41:41.026 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-04-23T10:41:41.027 INFO:teuthology.orchestra.run.smithi204.stdout: "req_readdir_latency": { 2024-04-23T10:41:41.027 INFO:teuthology.orchestra.run.smithi204.stdout: "avgcount": 0, 2024-04-23T10:41:41.027 INFO:teuthology.orchestra.run.smithi204.stdout: "sum": 0.000000000, 2024-04-23T10:41:41.027 INFO:teuthology.orchestra.run.smithi204.stdout: "avgtime": 0.000000000 2024-04-23T10:41:41.027 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-04-23T10:41:41.028 INFO:teuthology.orchestra.run.smithi204.stdout: "req_rename_latency": { 2024-04-23T10:41:41.028 INFO:teuthology.orchestra.run.smithi204.stdout: "avgcount": 0, 2024-04-23T10:41:41.028 INFO:teuthology.orchestra.run.smithi204.stdout: "sum": 0.000000000, 2024-04-23T10:41:41.028 INFO:teuthology.orchestra.run.smithi204.stdout: "avgtime": 0.000000000 2024-04-23T10:41:41.028 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-04-23T10:41:41.028 INFO:teuthology.orchestra.run.smithi204.stdout: "req_renamesnap_latency": { 2024-04-23T10:41:41.028 INFO:teuthology.orchestra.run.smithi204.stdout: "avgcount": 0, 2024-04-23T10:41:41.029 INFO:teuthology.orchestra.run.smithi204.stdout: "sum": 0.000000000, 2024-04-23T10:41:41.029 INFO:teuthology.orchestra.run.smithi204.stdout: "avgtime": 0.000000000 2024-04-23T10:41:41.029 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-04-23T10:41:41.029 INFO:teuthology.orchestra.run.smithi204.stdout: "req_snapdiff_latency": { 2024-04-23T10:41:41.029 INFO:teuthology.orchestra.run.smithi204.stdout: "avgcount": 0, 2024-04-23T10:41:41.029 INFO:teuthology.orchestra.run.smithi204.stdout: "sum": 0.000000000, 2024-04-23T10:41:41.029 INFO:teuthology.orchestra.run.smithi204.stdout: "avgtime": 0.000000000 2024-04-23T10:41:41.029 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-04-23T10:41:41.030 INFO:teuthology.orchestra.run.smithi204.stdout: "req_rmdir_latency": { 2024-04-23T10:41:41.030 INFO:teuthology.orchestra.run.smithi204.stdout: "avgcount": 0, 2024-04-23T10:41:41.030 INFO:teuthology.orchestra.run.smithi204.stdout: "sum": 0.000000000, 2024-04-23T10:41:41.030 INFO:teuthology.orchestra.run.smithi204.stdout: "avgtime": 0.000000000 2024-04-23T10:41:41.030 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-04-23T10:41:41.030 INFO:teuthology.orchestra.run.smithi204.stdout: "req_rmsnap_latency": { 2024-04-23T10:41:41.031 INFO:teuthology.orchestra.run.smithi204.stdout: "avgcount": 0, 2024-04-23T10:41:41.031 INFO:teuthology.orchestra.run.smithi204.stdout: "sum": 0.000000000, 2024-04-23T10:41:41.031 INFO:teuthology.orchestra.run.smithi204.stdout: "avgtime": 0.000000000 2024-04-23T10:41:41.031 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-04-23T10:41:41.031 INFO:teuthology.orchestra.run.smithi204.stdout: "req_rmxattr_latency": { 2024-04-23T10:41:41.031 INFO:teuthology.orchestra.run.smithi204.stdout: "avgcount": 0, 2024-04-23T10:41:41.031 INFO:teuthology.orchestra.run.smithi204.stdout: "sum": 0.000000000, 2024-04-23T10:41:41.032 INFO:teuthology.orchestra.run.smithi204.stdout: "avgtime": 0.000000000 2024-04-23T10:41:41.032 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-04-23T10:41:41.032 INFO:teuthology.orchestra.run.smithi204.stdout: "req_setattr_latency": { 2024-04-23T10:41:41.032 INFO:teuthology.orchestra.run.smithi204.stdout: "avgcount": 0, 2024-04-23T10:41:41.032 INFO:teuthology.orchestra.run.smithi204.stdout: "sum": 0.000000000, 2024-04-23T10:41:41.032 INFO:teuthology.orchestra.run.smithi204.stdout: "avgtime": 0.000000000 2024-04-23T10:41:41.033 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-04-23T10:41:41.033 INFO:teuthology.orchestra.run.smithi204.stdout: "req_setdirlayout_latency": { 2024-04-23T10:41:41.033 INFO:teuthology.orchestra.run.smithi204.stdout: "avgcount": 0, 2024-04-23T10:41:41.033 INFO:teuthology.orchestra.run.smithi204.stdout: "sum": 0.000000000, 2024-04-23T10:41:41.033 INFO:teuthology.orchestra.run.smithi204.stdout: "avgtime": 0.000000000 2024-04-23T10:41:41.033 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-04-23T10:41:41.033 INFO:teuthology.orchestra.run.smithi204.stdout: "req_setfilelock_latency": { 2024-04-23T10:41:41.033 INFO:teuthology.orchestra.run.smithi204.stdout: "avgcount": 0, 2024-04-23T10:41:41.034 INFO:teuthology.orchestra.run.smithi204.stdout: "sum": 0.000000000, 2024-04-23T10:41:41.034 INFO:teuthology.orchestra.run.smithi204.stdout: "avgtime": 0.000000000 2024-04-23T10:41:41.034 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-04-23T10:41:41.034 INFO:teuthology.orchestra.run.smithi204.stdout: "req_setlayout_latency": { 2024-04-23T10:41:41.034 INFO:teuthology.orchestra.run.smithi204.stdout: "avgcount": 0, 2024-04-23T10:41:41.034 INFO:teuthology.orchestra.run.smithi204.stdout: "sum": 0.000000000, 2024-04-23T10:41:41.034 INFO:teuthology.orchestra.run.smithi204.stdout: "avgtime": 0.000000000 2024-04-23T10:41:41.035 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-04-23T10:41:41.035 INFO:teuthology.orchestra.run.smithi204.stdout: "req_setxattr_latency": { 2024-04-23T10:41:41.035 INFO:teuthology.orchestra.run.smithi204.stdout: "avgcount": 0, 2024-04-23T10:41:41.035 INFO:teuthology.orchestra.run.smithi204.stdout: "sum": 0.000000000, 2024-04-23T10:41:41.035 INFO:teuthology.orchestra.run.smithi204.stdout: "avgtime": 0.000000000 2024-04-23T10:41:41.035 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-04-23T10:41:41.035 INFO:teuthology.orchestra.run.smithi204.stdout: "req_symlink_latency": { 2024-04-23T10:41:41.036 INFO:teuthology.orchestra.run.smithi204.stdout: "avgcount": 0, 2024-04-23T10:41:41.036 INFO:teuthology.orchestra.run.smithi204.stdout: "sum": 0.000000000, 2024-04-23T10:41:41.036 INFO:teuthology.orchestra.run.smithi204.stdout: "avgtime": 0.000000000 2024-04-23T10:41:41.036 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-04-23T10:41:41.036 INFO:teuthology.orchestra.run.smithi204.stdout: "req_unlink_latency": { 2024-04-23T10:41:41.036 INFO:teuthology.orchestra.run.smithi204.stdout: "avgcount": 0, 2024-04-23T10:41:41.036 INFO:teuthology.orchestra.run.smithi204.stdout: "sum": 0.000000000, 2024-04-23T10:41:41.037 INFO:teuthology.orchestra.run.smithi204.stdout: "avgtime": 0.000000000 2024-04-23T10:41:41.037 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-04-23T10:41:41.037 INFO:teuthology.orchestra.run.smithi204.stdout: "cap_revoke_eviction": 0, 2024-04-23T10:41:41.037 INFO:teuthology.orchestra.run.smithi204.stdout: "cap_acquisition_throttle": 0, 2024-04-23T10:41:41.037 INFO:teuthology.orchestra.run.smithi204.stdout: "req_getvxattr_latency": { 2024-04-23T10:41:41.037 INFO:teuthology.orchestra.run.smithi204.stdout: "avgcount": 0, 2024-04-23T10:41:41.037 INFO:teuthology.orchestra.run.smithi204.stdout: "sum": 0.000000000, 2024-04-23T10:41:41.037 INFO:teuthology.orchestra.run.smithi204.stdout: "avgtime": 0.000000000 2024-04-23T10:41:41.038 INFO:teuthology.orchestra.run.smithi204.stdout: } 2024-04-23T10:41:41.038 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-04-23T10:41:41.038 INFO:teuthology.orchestra.run.smithi204.stdout: "mds_sessions": { 2024-04-23T10:41:41.038 INFO:teuthology.orchestra.run.smithi204.stdout: "session_count": 2, 2024-04-23T10:41:41.038 INFO:teuthology.orchestra.run.smithi204.stdout: "session_add": 2, 2024-04-23T10:41:41.038 INFO:teuthology.orchestra.run.smithi204.stdout: "session_remove": 0, 2024-04-23T10:41:41.038 INFO:teuthology.orchestra.run.smithi204.stdout: "sessions_open": 0, 2024-04-23T10:41:41.039 INFO:teuthology.orchestra.run.smithi204.stdout: "sessions_stale": 0, 2024-04-23T10:41:41.039 INFO:teuthology.orchestra.run.smithi204.stdout: "total_load": 0, 2024-04-23T10:41:41.039 INFO:teuthology.orchestra.run.smithi204.stdout: "average_load": 0, 2024-04-23T10:41:41.039 INFO:teuthology.orchestra.run.smithi204.stdout: "avg_session_uptime": 2578, 2024-04-23T10:41:41.039 INFO:teuthology.orchestra.run.smithi204.stdout: "mdthresh_evicted": 0 2024-04-23T10:41:41.039 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-04-23T10:41:41.039 INFO:teuthology.orchestra.run.smithi204.stdout: "mempool": { 2024-04-23T10:41:41.039 INFO:teuthology.orchestra.run.smithi204.stdout: "bloom_filter_bytes": 0, 2024-04-23T10:41:41.040 INFO:teuthology.orchestra.run.smithi204.stdout: "bloom_filter_items": 0, 2024-04-23T10:41:41.040 INFO:teuthology.orchestra.run.smithi204.stdout: "bluestore_alloc_bytes": 0, 2024-04-23T10:41:41.040 INFO:teuthology.orchestra.run.smithi204.stdout: "bluestore_alloc_items": 0, 2024-04-23T10:41:41.040 INFO:teuthology.orchestra.run.smithi204.stdout: "bluestore_cache_data_bytes": 0, 2024-04-23T10:41:41.040 INFO:teuthology.orchestra.run.smithi204.stdout: "bluestore_cache_data_items": 0, 2024-04-23T10:41:41.040 INFO:teuthology.orchestra.run.smithi204.stdout: "bluestore_cache_onode_bytes": 0, 2024-04-23T10:41:41.040 INFO:teuthology.orchestra.run.smithi204.stdout: "bluestore_cache_onode_items": 0, 2024-04-23T10:41:41.041 INFO:teuthology.orchestra.run.smithi204.stdout: "bluestore_cache_meta_bytes": 0, 2024-04-23T10:41:41.041 INFO:teuthology.orchestra.run.smithi204.stdout: "bluestore_cache_meta_items": 0, 2024-04-23T10:41:41.041 INFO:teuthology.orchestra.run.smithi204.stdout: "bluestore_cache_other_bytes": 0, 2024-04-23T10:41:41.041 INFO:teuthology.orchestra.run.smithi204.stdout: "bluestore_cache_other_items": 0, 2024-04-23T10:41:41.041 INFO:teuthology.orchestra.run.smithi204.stdout: "bluestore_cache_buffer_bytes": 0, 2024-04-23T10:41:41.041 INFO:teuthology.orchestra.run.smithi204.stdout: "bluestore_cache_buffer_items": 0, 2024-04-23T10:41:41.041 INFO:teuthology.orchestra.run.smithi204.stdout: "bluestore_extent_bytes": 0, 2024-04-23T10:41:41.041 INFO:teuthology.orchestra.run.smithi204.stdout: "bluestore_extent_items": 0, 2024-04-23T10:41:41.042 INFO:teuthology.orchestra.run.smithi204.stdout: "bluestore_blob_bytes": 0, 2024-04-23T10:41:41.042 INFO:teuthology.orchestra.run.smithi204.stdout: "bluestore_blob_items": 0, 2024-04-23T10:41:41.042 INFO:teuthology.orchestra.run.smithi204.stdout: "bluestore_shared_blob_bytes": 0, 2024-04-23T10:41:41.042 INFO:teuthology.orchestra.run.smithi204.stdout: "bluestore_shared_blob_items": 0, 2024-04-23T10:41:41.042 INFO:teuthology.orchestra.run.smithi204.stdout: "bluestore_inline_bl_bytes": 0, 2024-04-23T10:41:41.042 INFO:teuthology.orchestra.run.smithi204.stdout: "bluestore_inline_bl_items": 0, 2024-04-23T10:41:41.042 INFO:teuthology.orchestra.run.smithi204.stdout: "bluestore_fsck_bytes": 0, 2024-04-23T10:41:41.043 INFO:teuthology.orchestra.run.smithi204.stdout: "bluestore_fsck_items": 0, 2024-04-23T10:41:41.043 INFO:teuthology.orchestra.run.smithi204.stdout: "bluestore_txc_bytes": 0, 2024-04-23T10:41:41.043 INFO:teuthology.orchestra.run.smithi204.stdout: "bluestore_txc_items": 0, 2024-04-23T10:41:41.043 INFO:teuthology.orchestra.run.smithi204.stdout: "bluestore_writing_deferred_bytes": 0, 2024-04-23T10:41:41.043 INFO:teuthology.orchestra.run.smithi204.stdout: "bluestore_writing_deferred_items": 0, 2024-04-23T10:41:41.043 INFO:teuthology.orchestra.run.smithi204.stdout: "bluestore_writing_bytes": 0, 2024-04-23T10:41:41.043 INFO:teuthology.orchestra.run.smithi204.stdout: "bluestore_writing_items": 0, 2024-04-23T10:41:41.043 INFO:teuthology.orchestra.run.smithi204.stdout: "bluefs_bytes": 0, 2024-04-23T10:41:41.044 INFO:teuthology.orchestra.run.smithi204.stdout: "bluefs_items": 0, 2024-04-23T10:41:41.044 INFO:teuthology.orchestra.run.smithi204.stdout: "bluefs_file_reader_bytes": 0, 2024-04-23T10:41:41.044 INFO:teuthology.orchestra.run.smithi204.stdout: "bluefs_file_reader_items": 0, 2024-04-23T10:41:41.044 INFO:teuthology.orchestra.run.smithi204.stdout: "bluefs_file_writer_bytes": 0, 2024-04-23T10:41:41.044 INFO:teuthology.orchestra.run.smithi204.stdout: "bluefs_file_writer_items": 0, 2024-04-23T10:41:41.044 INFO:teuthology.orchestra.run.smithi204.stdout: "buffer_anon_bytes": 245387, 2024-04-23T10:41:41.044 INFO:teuthology.orchestra.run.smithi204.stdout: "buffer_anon_items": 941, 2024-04-23T10:41:41.045 INFO:teuthology.orchestra.run.smithi204.stdout: "buffer_meta_bytes": 0, 2024-04-23T10:41:41.045 INFO:teuthology.orchestra.run.smithi204.stdout: "buffer_meta_items": 0, 2024-04-23T10:41:41.045 INFO:teuthology.orchestra.run.smithi204.stdout: "osd_bytes": 0, 2024-04-23T10:41:41.045 INFO:teuthology.orchestra.run.smithi204.stdout: "osd_items": 0, 2024-04-23T10:41:41.045 INFO:teuthology.orchestra.run.smithi204.stdout: "osd_mapbl_bytes": 0, 2024-04-23T10:41:41.045 INFO:teuthology.orchestra.run.smithi204.stdout: "osd_mapbl_items": 0, 2024-04-23T10:41:41.045 INFO:teuthology.orchestra.run.smithi204.stdout: "osd_pglog_bytes": 0, 2024-04-23T10:41:41.045 INFO:teuthology.orchestra.run.smithi204.stdout: "osd_pglog_items": 0, 2024-04-23T10:41:41.046 INFO:teuthology.orchestra.run.smithi204.stdout: "osdmap_bytes": 8416, 2024-04-23T10:41:41.046 INFO:teuthology.orchestra.run.smithi204.stdout: "osdmap_items": 175, 2024-04-23T10:41:41.046 INFO:teuthology.orchestra.run.smithi204.stdout: "osdmap_mapping_bytes": 0, 2024-04-23T10:41:41.046 INFO:teuthology.orchestra.run.smithi204.stdout: "osdmap_mapping_items": 0, 2024-04-23T10:41:41.046 INFO:teuthology.orchestra.run.smithi204.stdout: "pgmap_bytes": 0, 2024-04-23T10:41:41.046 INFO:teuthology.orchestra.run.smithi204.stdout: "pgmap_items": 0, 2024-04-23T10:41:41.046 INFO:teuthology.orchestra.run.smithi204.stdout: "mds_co_bytes": 6237673, 2024-04-23T10:41:41.046 INFO:teuthology.orchestra.run.smithi204.stdout: "mds_co_items": 44202, 2024-04-23T10:41:41.047 INFO:teuthology.orchestra.run.smithi204.stdout: "unittest_1_bytes": 0, 2024-04-23T10:41:41.047 INFO:teuthology.orchestra.run.smithi204.stdout: "unittest_1_items": 0, 2024-04-23T10:41:41.047 INFO:teuthology.orchestra.run.smithi204.stdout: "unittest_2_bytes": 0, 2024-04-23T10:41:41.047 INFO:teuthology.orchestra.run.smithi204.stdout: "unittest_2_items": 0 2024-04-23T10:41:41.047 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-04-23T10:41:41.047 INFO:teuthology.orchestra.run.smithi204.stdout: "objecter": { 2024-04-23T10:41:41.047 INFO:teuthology.orchestra.run.smithi204.stdout: "op_active": 0, 2024-04-23T10:41:41.047 INFO:teuthology.orchestra.run.smithi204.stdout: "op_laggy": 0, 2024-04-23T10:41:41.048 INFO:teuthology.orchestra.run.smithi204.stdout: "op_send": 407, 2024-04-23T10:41:41.048 INFO:teuthology.orchestra.run.smithi204.stdout: "op_send_bytes": 26, 2024-04-23T10:41:41.048 INFO:teuthology.orchestra.run.smithi204.stdout: "op_resend": 4, 2024-04-23T10:41:41.048 INFO:teuthology.orchestra.run.smithi204.stdout: "op_reply": 407, 2024-04-23T10:41:41.048 INFO:teuthology.orchestra.run.smithi204.stdout: "op_latency": { 2024-04-23T10:41:41.048 INFO:teuthology.orchestra.run.smithi204.stdout: "avgcount": 407, 2024-04-23T10:41:41.048 INFO:teuthology.orchestra.run.smithi204.stdout: "sum": 0.417994324, 2024-04-23T10:41:41.049 INFO:teuthology.orchestra.run.smithi204.stdout: "avgtime": 0.001027013 2024-04-23T10:41:41.049 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-04-23T10:41:41.049 INFO:teuthology.orchestra.run.smithi204.stdout: "op_inflight": 0, 2024-04-23T10:41:41.049 INFO:teuthology.orchestra.run.smithi204.stdout: "oplen_avg": { 2024-04-23T10:41:41.049 INFO:teuthology.orchestra.run.smithi204.stdout: "avgcount": 407, 2024-04-23T10:41:41.049 INFO:teuthology.orchestra.run.smithi204.stdout: "sum": 408 2024-04-23T10:41:41.049 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-04-23T10:41:41.049 INFO:teuthology.orchestra.run.smithi204.stdout: "op": 407, 2024-04-23T10:41:41.050 INFO:teuthology.orchestra.run.smithi204.stdout: "op_r": 407, 2024-04-23T10:41:41.050 INFO:teuthology.orchestra.run.smithi204.stdout: "op_w": 0, 2024-04-23T10:41:41.050 INFO:teuthology.orchestra.run.smithi204.stdout: "op_rmw": 0, 2024-04-23T10:41:41.050 INFO:teuthology.orchestra.run.smithi204.stdout: "op_pg": 0, 2024-04-23T10:41:41.050 INFO:teuthology.orchestra.run.smithi204.stdout: "osdop_stat": 226, 2024-04-23T10:41:41.050 INFO:teuthology.orchestra.run.smithi204.stdout: "osdop_create": 0, 2024-04-23T10:41:41.050 INFO:teuthology.orchestra.run.smithi204.stdout: "osdop_read": 178, 2024-04-23T10:41:41.051 INFO:teuthology.orchestra.run.smithi204.stdout: "osdop_write": 0, 2024-04-23T10:41:41.051 INFO:teuthology.orchestra.run.smithi204.stdout: "osdop_writefull": 0, 2024-04-23T10:41:41.051 INFO:teuthology.orchestra.run.smithi204.stdout: "osdop_writesame": 0, 2024-04-23T10:41:41.051 INFO:teuthology.orchestra.run.smithi204.stdout: "osdop_append": 0, 2024-04-23T10:41:41.051 INFO:teuthology.orchestra.run.smithi204.stdout: "osdop_zero": 0, 2024-04-23T10:41:41.051 INFO:teuthology.orchestra.run.smithi204.stdout: "osdop_truncate": 0, 2024-04-23T10:41:41.051 INFO:teuthology.orchestra.run.smithi204.stdout: "osdop_delete": 0, 2024-04-23T10:41:41.051 INFO:teuthology.orchestra.run.smithi204.stdout: "osdop_mapext": 0, 2024-04-23T10:41:41.052 INFO:teuthology.orchestra.run.smithi204.stdout: "osdop_sparse_read": 0, 2024-04-23T10:41:41.052 INFO:teuthology.orchestra.run.smithi204.stdout: "osdop_clonerange": 0, 2024-04-23T10:41:41.052 INFO:teuthology.orchestra.run.smithi204.stdout: "osdop_getxattr": 2, 2024-04-23T10:41:41.052 INFO:teuthology.orchestra.run.smithi204.stdout: "osdop_setxattr": 0, 2024-04-23T10:41:41.052 INFO:teuthology.orchestra.run.smithi204.stdout: "osdop_cmpxattr": 0, 2024-04-23T10:41:41.052 INFO:teuthology.orchestra.run.smithi204.stdout: "osdop_rmxattr": 0, 2024-04-23T10:41:41.052 INFO:teuthology.orchestra.run.smithi204.stdout: "osdop_resetxattrs": 0, 2024-04-23T10:41:41.053 INFO:teuthology.orchestra.run.smithi204.stdout: "osdop_call": 0, 2024-04-23T10:41:41.053 INFO:teuthology.orchestra.run.smithi204.stdout: "osdop_watch": 0, 2024-04-23T10:41:41.053 INFO:teuthology.orchestra.run.smithi204.stdout: "osdop_notify": 0, 2024-04-23T10:41:41.053 INFO:teuthology.orchestra.run.smithi204.stdout: "osdop_src_cmpxattr": 0, 2024-04-23T10:41:41.053 INFO:teuthology.orchestra.run.smithi204.stdout: "osdop_pgls": 0, 2024-04-23T10:41:41.053 INFO:teuthology.orchestra.run.smithi204.stdout: "osdop_pgls_filter": 0, 2024-04-23T10:41:41.053 INFO:teuthology.orchestra.run.smithi204.stdout: "osdop_other": 0, 2024-04-23T10:41:41.053 INFO:teuthology.orchestra.run.smithi204.stdout: "linger_active": 0, 2024-04-23T10:41:41.054 INFO:teuthology.orchestra.run.smithi204.stdout: "linger_send": 0, 2024-04-23T10:41:41.054 INFO:teuthology.orchestra.run.smithi204.stdout: "linger_resend": 0, 2024-04-23T10:41:41.054 INFO:teuthology.orchestra.run.smithi204.stdout: "linger_ping": 0, 2024-04-23T10:41:41.054 INFO:teuthology.orchestra.run.smithi204.stdout: "poolop_active": 0, 2024-04-23T10:41:41.054 INFO:teuthology.orchestra.run.smithi204.stdout: "poolop_send": 0, 2024-04-23T10:41:41.054 INFO:teuthology.orchestra.run.smithi204.stdout: "poolop_resend": 0, 2024-04-23T10:41:41.054 INFO:teuthology.orchestra.run.smithi204.stdout: "poolstat_active": 0, 2024-04-23T10:41:41.054 INFO:teuthology.orchestra.run.smithi204.stdout: "poolstat_send": 0, 2024-04-23T10:41:41.055 INFO:teuthology.orchestra.run.smithi204.stdout: "poolstat_resend": 0, 2024-04-23T10:41:41.055 INFO:teuthology.orchestra.run.smithi204.stdout: "statfs_active": 0, 2024-04-23T10:41:41.055 INFO:teuthology.orchestra.run.smithi204.stdout: "statfs_send": 0, 2024-04-23T10:41:41.055 INFO:teuthology.orchestra.run.smithi204.stdout: "statfs_resend": 0, 2024-04-23T10:41:41.055 INFO:teuthology.orchestra.run.smithi204.stdout: "command_active": 0, 2024-04-23T10:41:41.055 INFO:teuthology.orchestra.run.smithi204.stdout: "command_send": 0, 2024-04-23T10:41:41.055 INFO:teuthology.orchestra.run.smithi204.stdout: "command_resend": 0, 2024-04-23T10:41:41.055 INFO:teuthology.orchestra.run.smithi204.stdout: "map_epoch": 113, 2024-04-23T10:41:41.056 INFO:teuthology.orchestra.run.smithi204.stdout: "map_full": 0, 2024-04-23T10:41:41.056 INFO:teuthology.orchestra.run.smithi204.stdout: "map_inc": 2, 2024-04-23T10:41:41.056 INFO:teuthology.orchestra.run.smithi204.stdout: "osd_sessions": 8, 2024-04-23T10:41:41.056 INFO:teuthology.orchestra.run.smithi204.stdout: "osd_session_open": 8, 2024-04-23T10:41:41.056 INFO:teuthology.orchestra.run.smithi204.stdout: "osd_session_close": 0, 2024-04-23T10:41:41.056 INFO:teuthology.orchestra.run.smithi204.stdout: "osd_laggy": 0, 2024-04-23T10:41:41.056 INFO:teuthology.orchestra.run.smithi204.stdout: "omap_wr": 0, 2024-04-23T10:41:41.057 INFO:teuthology.orchestra.run.smithi204.stdout: "omap_rd": 2, 2024-04-23T10:41:41.057 INFO:teuthology.orchestra.run.smithi204.stdout: "omap_del": 0 2024-04-23T10:41:41.057 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-04-23T10:41:41.057 INFO:teuthology.orchestra.run.smithi204.stdout: "oft": { 2024-04-23T10:41:41.057 INFO:teuthology.orchestra.run.smithi204.stdout: "omap_total_objs": 0, 2024-04-23T10:41:41.057 INFO:teuthology.orchestra.run.smithi204.stdout: "omap_total_kv_pairs": 0, 2024-04-23T10:41:41.057 INFO:teuthology.orchestra.run.smithi204.stdout: "omap_total_updates": 0, 2024-04-23T10:41:41.057 INFO:teuthology.orchestra.run.smithi204.stdout: "omap_total_removes": 0 2024-04-23T10:41:41.058 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-04-23T10:41:41.058 INFO:teuthology.orchestra.run.smithi204.stdout: "osd-slow-ops": { 2024-04-23T10:41:41.058 INFO:teuthology.orchestra.run.smithi204.stdout: "slow_ops_count": 0 2024-04-23T10:41:41.058 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-04-23T10:41:41.058 INFO:teuthology.orchestra.run.smithi204.stdout: "purge_queue": { 2024-04-23T10:41:41.058 INFO:teuthology.orchestra.run.smithi204.stdout: "pq_executing_ops": 0, 2024-04-23T10:41:41.058 INFO:teuthology.orchestra.run.smithi204.stdout: "pq_executing_ops_high_water": 0, 2024-04-23T10:41:41.058 INFO:teuthology.orchestra.run.smithi204.stdout: "pq_executing": 0, 2024-04-23T10:41:41.059 INFO:teuthology.orchestra.run.smithi204.stdout: "pq_executing_high_water": 0, 2024-04-23T10:41:41.059 INFO:teuthology.orchestra.run.smithi204.stdout: "pq_executed_ops": 0, 2024-04-23T10:41:41.059 INFO:teuthology.orchestra.run.smithi204.stdout: "pq_executed": 0, 2024-04-23T10:41:41.059 INFO:teuthology.orchestra.run.smithi204.stdout: "pq_item_in_journal": 0 2024-04-23T10:41:41.059 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-04-23T10:41:41.059 INFO:teuthology.orchestra.run.smithi204.stdout: "throttle-msgr_dispatch_throttler-mds": { 2024-04-23T10:41:41.059 INFO:teuthology.orchestra.run.smithi204.stdout: "val": 0, 2024-04-23T10:41:41.059 INFO:teuthology.orchestra.run.smithi204.stdout: "max": 104857600, 2024-04-23T10:41:41.060 INFO:teuthology.orchestra.run.smithi204.stdout: "get_started": 0, 2024-04-23T10:41:41.060 INFO:teuthology.orchestra.run.smithi204.stdout: "get": 449, 2024-04-23T10:41:41.060 INFO:teuthology.orchestra.run.smithi204.stdout: "get_sum": 64043291, 2024-04-23T10:41:41.060 INFO:teuthology.orchestra.run.smithi204.stdout: "get_or_fail_fail": 0, 2024-04-23T10:41:41.060 INFO:teuthology.orchestra.run.smithi204.stdout: "get_or_fail_success": 449, 2024-04-23T10:41:41.060 INFO:teuthology.orchestra.run.smithi204.stdout: "take": 0, 2024-04-23T10:41:41.060 INFO:teuthology.orchestra.run.smithi204.stdout: "take_sum": 0, 2024-04-23T10:41:41.060 INFO:teuthology.orchestra.run.smithi204.stdout: "put": 449, 2024-04-23T10:41:41.061 INFO:teuthology.orchestra.run.smithi204.stdout: "put_sum": 64043291, 2024-04-23T10:41:41.061 INFO:teuthology.orchestra.run.smithi204.stdout: "wait": { 2024-04-23T10:41:41.061 INFO:teuthology.orchestra.run.smithi204.stdout: "avgcount": 0, 2024-04-23T10:41:41.061 INFO:teuthology.orchestra.run.smithi204.stdout: "sum": 0.000000000, 2024-04-23T10:41:41.061 INFO:teuthology.orchestra.run.smithi204.stdout: "avgtime": 0.000000000 2024-04-23T10:41:41.061 INFO:teuthology.orchestra.run.smithi204.stdout: } 2024-04-23T10:41:41.061 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-04-23T10:41:41.062 INFO:teuthology.orchestra.run.smithi204.stdout: "throttle-objecter_bytes": { 2024-04-23T10:41:41.062 INFO:teuthology.orchestra.run.smithi204.stdout: "val": 0, 2024-04-23T10:41:41.062 INFO:teuthology.orchestra.run.smithi204.stdout: "max": 104857600, 2024-04-23T10:41:41.062 INFO:teuthology.orchestra.run.smithi204.stdout: "get_started": 0, 2024-04-23T10:41:41.062 INFO:teuthology.orchestra.run.smithi204.stdout: "get": 0, 2024-04-23T10:41:41.062 INFO:teuthology.orchestra.run.smithi204.stdout: "get_sum": 0, 2024-04-23T10:41:41.062 INFO:teuthology.orchestra.run.smithi204.stdout: "get_or_fail_fail": 0, 2024-04-23T10:41:41.062 INFO:teuthology.orchestra.run.smithi204.stdout: "get_or_fail_success": 0, 2024-04-23T10:41:41.063 INFO:teuthology.orchestra.run.smithi204.stdout: "take": 407, 2024-04-23T10:41:41.063 INFO:teuthology.orchestra.run.smithi204.stdout: "take_sum": 63784414, 2024-04-23T10:41:41.063 INFO:teuthology.orchestra.run.smithi204.stdout: "put": 407, 2024-04-23T10:41:41.063 INFO:teuthology.orchestra.run.smithi204.stdout: "put_sum": 63784414, 2024-04-23T10:41:41.063 INFO:teuthology.orchestra.run.smithi204.stdout: "wait": { 2024-04-23T10:41:41.063 INFO:teuthology.orchestra.run.smithi204.stdout: "avgcount": 0, 2024-04-23T10:41:41.063 INFO:teuthology.orchestra.run.smithi204.stdout: "sum": 0.000000000, 2024-04-23T10:41:41.063 INFO:teuthology.orchestra.run.smithi204.stdout: "avgtime": 0.000000000 2024-04-23T10:41:41.064 INFO:teuthology.orchestra.run.smithi204.stdout: } 2024-04-23T10:41:41.064 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-04-23T10:41:41.064 INFO:teuthology.orchestra.run.smithi204.stdout: "throttle-objecter_ops": { 2024-04-23T10:41:41.064 INFO:teuthology.orchestra.run.smithi204.stdout: "val": 0, 2024-04-23T10:41:41.064 INFO:teuthology.orchestra.run.smithi204.stdout: "max": 1024, 2024-04-23T10:41:41.064 INFO:teuthology.orchestra.run.smithi204.stdout: "get_started": 0, 2024-04-23T10:41:41.064 INFO:teuthology.orchestra.run.smithi204.stdout: "get": 0, 2024-04-23T10:41:41.064 INFO:teuthology.orchestra.run.smithi204.stdout: "get_sum": 0, 2024-04-23T10:41:41.064 INFO:teuthology.orchestra.run.smithi204.stdout: "get_or_fail_fail": 0, 2024-04-23T10:41:41.065 INFO:teuthology.orchestra.run.smithi204.stdout: "get_or_fail_success": 0, 2024-04-23T10:41:41.065 INFO:teuthology.orchestra.run.smithi204.stdout: "take": 407, 2024-04-23T10:41:41.065 INFO:teuthology.orchestra.run.smithi204.stdout: "take_sum": 407, 2024-04-23T10:41:41.065 INFO:teuthology.orchestra.run.smithi204.stdout: "put": 407, 2024-04-23T10:41:41.065 INFO:teuthology.orchestra.run.smithi204.stdout: "put_sum": 407, 2024-04-23T10:41:41.065 INFO:teuthology.orchestra.run.smithi204.stdout: "wait": { 2024-04-23T10:41:41.065 INFO:teuthology.orchestra.run.smithi204.stdout: "avgcount": 0, 2024-04-23T10:41:41.065 INFO:teuthology.orchestra.run.smithi204.stdout: "sum": 0.000000000, 2024-04-23T10:41:41.066 INFO:teuthology.orchestra.run.smithi204.stdout: "avgtime": 0.000000000 2024-04-23T10:41:41.066 INFO:teuthology.orchestra.run.smithi204.stdout: } 2024-04-23T10:41:41.066 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-04-23T10:41:41.066 INFO:teuthology.orchestra.run.smithi204.stdout: "throttle-write_buf_throttle": { 2024-04-23T10:41:41.066 INFO:teuthology.orchestra.run.smithi204.stdout: "val": 0, 2024-04-23T10:41:41.066 INFO:teuthology.orchestra.run.smithi204.stdout: "max": 3758096384, 2024-04-23T10:41:41.066 INFO:teuthology.orchestra.run.smithi204.stdout: "get_started": 0, 2024-04-23T10:41:41.066 INFO:teuthology.orchestra.run.smithi204.stdout: "get": 0, 2024-04-23T10:41:41.067 INFO:teuthology.orchestra.run.smithi204.stdout: "get_sum": 0, 2024-04-23T10:41:41.067 INFO:teuthology.orchestra.run.smithi204.stdout: "get_or_fail_fail": 0, 2024-04-23T10:41:41.067 INFO:teuthology.orchestra.run.smithi204.stdout: "get_or_fail_success": 0, 2024-04-23T10:41:41.067 INFO:teuthology.orchestra.run.smithi204.stdout: "take": 0, 2024-04-23T10:41:41.067 INFO:teuthology.orchestra.run.smithi204.stdout: "take_sum": 0, 2024-04-23T10:41:41.067 INFO:teuthology.orchestra.run.smithi204.stdout: "put": 0, 2024-04-23T10:41:41.067 INFO:teuthology.orchestra.run.smithi204.stdout: "put_sum": 0, 2024-04-23T10:41:41.067 INFO:teuthology.orchestra.run.smithi204.stdout: "wait": { 2024-04-23T10:41:41.068 INFO:teuthology.orchestra.run.smithi204.stdout: "avgcount": 0, 2024-04-23T10:41:41.068 INFO:teuthology.orchestra.run.smithi204.stdout: "sum": 0.000000000, 2024-04-23T10:41:41.068 INFO:teuthology.orchestra.run.smithi204.stdout: "avgtime": 0.000000000 2024-04-23T10:41:41.068 INFO:teuthology.orchestra.run.smithi204.stdout: } 2024-04-23T10:41:41.068 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-04-23T10:41:41.068 INFO:teuthology.orchestra.run.smithi204.stdout: "throttle-write_buf_throttle-0x5555e34986e0": { 2024-04-23T10:41:41.068 INFO:teuthology.orchestra.run.smithi204.stdout: "val": 0, 2024-04-23T10:41:41.068 INFO:teuthology.orchestra.run.smithi204.stdout: "max": 3758096384, 2024-04-23T10:41:41.069 INFO:teuthology.orchestra.run.smithi204.stdout: "get_started": 0, 2024-04-23T10:41:41.069 INFO:teuthology.orchestra.run.smithi204.stdout: "get": 0, 2024-04-23T10:41:41.069 INFO:teuthology.orchestra.run.smithi204.stdout: "get_sum": 0, 2024-04-23T10:41:41.069 INFO:teuthology.orchestra.run.smithi204.stdout: "get_or_fail_fail": 0, 2024-04-23T10:41:41.069 INFO:teuthology.orchestra.run.smithi204.stdout: "get_or_fail_success": 0, 2024-04-23T10:41:41.069 INFO:teuthology.orchestra.run.smithi204.stdout: "take": 0, 2024-04-23T10:41:41.069 INFO:teuthology.orchestra.run.smithi204.stdout: "take_sum": 0, 2024-04-23T10:41:41.069 INFO:teuthology.orchestra.run.smithi204.stdout: "put": 0, 2024-04-23T10:41:41.070 INFO:teuthology.orchestra.run.smithi204.stdout: "put_sum": 0, 2024-04-23T10:41:41.070 INFO:teuthology.orchestra.run.smithi204.stdout: "wait": { 2024-04-23T10:41:41.070 INFO:teuthology.orchestra.run.smithi204.stdout: "avgcount": 0, 2024-04-23T10:41:41.070 INFO:teuthology.orchestra.run.smithi204.stdout: "sum": 0.000000000, 2024-04-23T10:41:41.070 INFO:teuthology.orchestra.run.smithi204.stdout: "avgtime": 0.000000000 2024-04-23T10:41:41.070 INFO:teuthology.orchestra.run.smithi204.stdout: } 2024-04-23T10:41:41.070 INFO:teuthology.orchestra.run.smithi204.stdout: } 2024-04-23T10:41:41.070 INFO:teuthology.orchestra.run.smithi204.stdout:} 2024-04-23T10:41:41.771 INFO:tasks.check_counter:Daemon mds.i mds.exported=0 2024-04-23T10:41:41.771 INFO:tasks.check_counter:Daemon mds.i mds.imported=0 2024-04-23T10:41:41.772 INFO:tasks.check_counter:Daemon mds.i mds_cache.dir_update=0 2024-04-23T10:41:41.772 INFO:tasks.check_counter:Daemon mds.i mds_cache.dir_update_receipt=0 2024-04-23T10:41:41.772 INFO:tasks.check_counter:Daemon mds.i mds.root_rsnaps=6 2024-04-23T10:41:41.772 INFO:tasks.check_counter:Daemon mds.i mds_server.req_mksnap_latency.avgcount=0 2024-04-23T10:41:41.772 INFO:tasks.check_counter:Daemon mds.i mds_server.req_rmsnap_latency.avgcount=0 2024-04-23T10:41:41.772 DEBUG:tasks.check_counter:Getting stats from l 2024-04-23T10:41:41.773 DEBUG:teuthology.orchestra.run.smithi204:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d69f13d827952bb07bb9b7973bca14f97d764d6f shell --fsid aa240572-0159-11ef-bc93-c7b262605968 -- ceph daemon mds.l perf dump 2024-04-23T10:41:41.796 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 23 10:41:41 smithi204 ceph-mon[32649]: pgmap v476: 97 pgs: 97 active+clean; 551 MiB data, 4.3 GiB used, 1.0 TiB / 1.0 TiB avail; 487 KiB/s rd, 3.1 MiB/s wr, 113 op/s 2024-04-23T10:41:41.992 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 23 10:41:41 smithi119 ceph-mon[32608]: pgmap v476: 97 pgs: 97 active+clean; 551 MiB data, 4.3 GiB used, 1.0 TiB / 1.0 TiB avail; 487 KiB/s rd, 3.1 MiB/s wr, 113 op/s 2024-04-23T10:41:42.004 INFO:journalctl@ceph.mon.a.smithi114.stdout:Apr 23 10:41:41 smithi114 ceph-mon[29255]: pgmap v476: 97 pgs: 97 active+clean; 551 MiB data, 4.3 GiB used, 1.0 TiB / 1.0 TiB avail; 487 KiB/s rd, 3.1 MiB/s wr, 113 op/s 2024-04-23T10:41:42.012 INFO:teuthology.orchestra.run.smithi204.stderr:Inferring config /var/lib/ceph/aa240572-0159-11ef-bc93-c7b262605968/mon.c/config 2024-04-23T10:41:43.711 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 23 10:41:43 smithi204 ceph-mon[32649]: pgmap v477: 97 pgs: 97 active+clean; 551 MiB data, 4.3 GiB used, 1.0 TiB / 1.0 TiB avail; 569 KiB/s rd, 3.1 MiB/s wr, 120 op/s 2024-04-23T10:41:43.711 INFO:teuthology.orchestra.run.smithi204.stdout:{ 2024-04-23T10:41:43.711 INFO:teuthology.orchestra.run.smithi204.stdout: "AsyncMessenger::Worker-0": { 2024-04-23T10:41:43.712 INFO:teuthology.orchestra.run.smithi204.stdout: "msgr_recv_messages": 2, 2024-04-23T10:41:43.712 INFO:teuthology.orchestra.run.smithi204.stdout: "msgr_send_messages": 4, 2024-04-23T10:41:43.712 INFO:teuthology.orchestra.run.smithi204.stdout: "msgr_recv_bytes": 303, 2024-04-23T10:41:43.712 INFO:teuthology.orchestra.run.smithi204.stdout: "msgr_send_bytes": 23311, 2024-04-23T10:41:43.712 INFO:teuthology.orchestra.run.smithi204.stdout: "msgr_created_connections": 6, 2024-04-23T10:41:43.712 INFO:teuthology.orchestra.run.smithi204.stdout: "msgr_active_connections": 18446744073709551615, 2024-04-23T10:41:43.713 INFO:teuthology.orchestra.run.smithi204.stdout: "msgr_running_total_time": 0.002246105, 2024-04-23T10:41:43.713 INFO:teuthology.orchestra.run.smithi204.stdout: "msgr_running_send_time": 0.000057903, 2024-04-23T10:41:43.713 INFO:teuthology.orchestra.run.smithi204.stdout: "msgr_running_recv_time": 0.001068026, 2024-04-23T10:41:43.713 INFO:teuthology.orchestra.run.smithi204.stdout: "msgr_running_fast_dispatch_time": 0.000000000, 2024-04-23T10:41:43.713 INFO:teuthology.orchestra.run.smithi204.stdout: "msgr_send_messages_queue_lat": { 2024-04-23T10:41:43.713 INFO:teuthology.orchestra.run.smithi204.stdout: "avgcount": 2, 2024-04-23T10:41:43.714 INFO:teuthology.orchestra.run.smithi204.stdout: "sum": 0.000031214, 2024-04-23T10:41:43.714 INFO:teuthology.orchestra.run.smithi204.stdout: "avgtime": 0.000015607 2024-04-23T10:41:43.714 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-04-23T10:41:43.714 INFO:teuthology.orchestra.run.smithi204.stdout: "msgr_handle_ack_lat": { 2024-04-23T10:41:43.714 INFO:teuthology.orchestra.run.smithi204.stdout: "avgcount": 2, 2024-04-23T10:41:43.714 INFO:teuthology.orchestra.run.smithi204.stdout: "sum": 0.000000240, 2024-04-23T10:41:43.715 INFO:teuthology.orchestra.run.smithi204.stdout: "avgtime": 0.000000120 2024-04-23T10:41:43.715 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-04-23T10:41:43.715 INFO:teuthology.orchestra.run.smithi204.stdout: "msgr_recv_encrypted_bytes": 0, 2024-04-23T10:41:43.715 INFO:teuthology.orchestra.run.smithi204.stdout: "msgr_send_encrypted_bytes": 0 2024-04-23T10:41:43.715 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-04-23T10:41:43.715 INFO:teuthology.orchestra.run.smithi204.stdout: "AsyncMessenger::Worker-1": { 2024-04-23T10:41:43.716 INFO:teuthology.orchestra.run.smithi204.stdout: "msgr_recv_messages": 297, 2024-04-23T10:41:43.716 INFO:teuthology.orchestra.run.smithi204.stdout: "msgr_send_messages": 284, 2024-04-23T10:41:43.716 INFO:teuthology.orchestra.run.smithi204.stdout: "msgr_recv_bytes": 1612848, 2024-04-23T10:41:43.716 INFO:teuthology.orchestra.run.smithi204.stdout: "msgr_send_bytes": 174112, 2024-04-23T10:41:43.716 INFO:teuthology.orchestra.run.smithi204.stdout: "msgr_created_connections": 2, 2024-04-23T10:41:43.717 INFO:teuthology.orchestra.run.smithi204.stdout: "msgr_active_connections": 1, 2024-04-23T10:41:43.717 INFO:teuthology.orchestra.run.smithi204.stdout: "msgr_running_total_time": 0.052754841, 2024-04-23T10:41:43.717 INFO:teuthology.orchestra.run.smithi204.stdout: "msgr_running_send_time": 0.016399909, 2024-04-23T10:41:43.717 INFO:teuthology.orchestra.run.smithi204.stdout: "msgr_running_recv_time": 0.038951087, 2024-04-23T10:41:43.717 INFO:teuthology.orchestra.run.smithi204.stdout: "msgr_running_fast_dispatch_time": 0.006684889, 2024-04-23T10:41:43.717 INFO:teuthology.orchestra.run.smithi204.stdout: "msgr_send_messages_queue_lat": { 2024-04-23T10:41:43.718 INFO:teuthology.orchestra.run.smithi204.stdout: "avgcount": 284, 2024-04-23T10:41:43.718 INFO:teuthology.orchestra.run.smithi204.stdout: "sum": 0.014237863, 2024-04-23T10:41:43.718 INFO:teuthology.orchestra.run.smithi204.stdout: "avgtime": 0.000050133 2024-04-23T10:41:43.718 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-04-23T10:41:43.718 INFO:teuthology.orchestra.run.smithi204.stdout: "msgr_handle_ack_lat": { 2024-04-23T10:41:43.718 INFO:teuthology.orchestra.run.smithi204.stdout: "avgcount": 0, 2024-04-23T10:41:43.719 INFO:teuthology.orchestra.run.smithi204.stdout: "sum": 0.000000000, 2024-04-23T10:41:43.719 INFO:teuthology.orchestra.run.smithi204.stdout: "avgtime": 0.000000000 2024-04-23T10:41:43.719 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-04-23T10:41:43.719 INFO:teuthology.orchestra.run.smithi204.stdout: "msgr_recv_encrypted_bytes": 1612848, 2024-04-23T10:41:43.719 INFO:teuthology.orchestra.run.smithi204.stdout: "msgr_send_encrypted_bytes": 174112 2024-04-23T10:41:43.719 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-04-23T10:41:43.720 INFO:teuthology.orchestra.run.smithi204.stdout: "AsyncMessenger::Worker-2": { 2024-04-23T10:41:43.720 INFO:teuthology.orchestra.run.smithi204.stdout: "msgr_recv_messages": 3, 2024-04-23T10:41:43.720 INFO:teuthology.orchestra.run.smithi204.stdout: "msgr_send_messages": 217, 2024-04-23T10:41:43.720 INFO:teuthology.orchestra.run.smithi204.stdout: "msgr_recv_bytes": 480, 2024-04-23T10:41:43.720 INFO:teuthology.orchestra.run.smithi204.stdout: "msgr_send_bytes": 272576, 2024-04-23T10:41:43.720 INFO:teuthology.orchestra.run.smithi204.stdout: "msgr_created_connections": 6, 2024-04-23T10:41:43.721 INFO:teuthology.orchestra.run.smithi204.stdout: "msgr_active_connections": 1, 2024-04-23T10:41:43.721 INFO:teuthology.orchestra.run.smithi204.stdout: "msgr_running_total_time": 0.013014278, 2024-04-23T10:41:43.721 INFO:teuthology.orchestra.run.smithi204.stdout: "msgr_running_send_time": 0.008785592, 2024-04-23T10:41:43.721 INFO:teuthology.orchestra.run.smithi204.stdout: "msgr_running_recv_time": 0.001762322, 2024-04-23T10:41:43.721 INFO:teuthology.orchestra.run.smithi204.stdout: "msgr_running_fast_dispatch_time": 0.000000000, 2024-04-23T10:41:43.721 INFO:teuthology.orchestra.run.smithi204.stdout: "msgr_send_messages_queue_lat": { 2024-04-23T10:41:43.722 INFO:teuthology.orchestra.run.smithi204.stdout: "avgcount": 214, 2024-04-23T10:41:43.722 INFO:teuthology.orchestra.run.smithi204.stdout: "sum": 0.013925621, 2024-04-23T10:41:43.722 INFO:teuthology.orchestra.run.smithi204.stdout: "avgtime": 0.000065072 2024-04-23T10:41:43.722 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-04-23T10:41:43.722 INFO:teuthology.orchestra.run.smithi204.stdout: "msgr_handle_ack_lat": { 2024-04-23T10:41:43.722 INFO:teuthology.orchestra.run.smithi204.stdout: "avgcount": 0, 2024-04-23T10:41:43.722 INFO:teuthology.orchestra.run.smithi204.stdout: "sum": 0.000000000, 2024-04-23T10:41:43.723 INFO:teuthology.orchestra.run.smithi204.stdout: "avgtime": 0.000000000 2024-04-23T10:41:43.723 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-04-23T10:41:43.723 INFO:teuthology.orchestra.run.smithi204.stdout: "msgr_recv_encrypted_bytes": 480, 2024-04-23T10:41:43.723 INFO:teuthology.orchestra.run.smithi204.stdout: "msgr_send_encrypted_bytes": 272576 2024-04-23T10:41:43.723 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-04-23T10:41:43.723 INFO:teuthology.orchestra.run.smithi204.stdout: "cct": { 2024-04-23T10:41:43.724 INFO:teuthology.orchestra.run.smithi204.stdout: "total_workers": 0, 2024-04-23T10:41:43.724 INFO:teuthology.orchestra.run.smithi204.stdout: "unhealthy_workers": 0 2024-04-23T10:41:43.724 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-04-23T10:41:43.724 INFO:teuthology.orchestra.run.smithi204.stdout: "mempool": { 2024-04-23T10:41:43.724 INFO:teuthology.orchestra.run.smithi204.stdout: "bloom_filter_bytes": 0, 2024-04-23T10:41:43.724 INFO:teuthology.orchestra.run.smithi204.stdout: "bloom_filter_items": 0, 2024-04-23T10:41:43.724 INFO:teuthology.orchestra.run.smithi204.stdout: "bluestore_alloc_bytes": 0, 2024-04-23T10:41:43.725 INFO:teuthology.orchestra.run.smithi204.stdout: "bluestore_alloc_items": 0, 2024-04-23T10:41:43.725 INFO:teuthology.orchestra.run.smithi204.stdout: "bluestore_cache_data_bytes": 0, 2024-04-23T10:41:43.725 INFO:teuthology.orchestra.run.smithi204.stdout: "bluestore_cache_data_items": 0, 2024-04-23T10:41:43.725 INFO:teuthology.orchestra.run.smithi204.stdout: "bluestore_cache_onode_bytes": 0, 2024-04-23T10:41:43.725 INFO:teuthology.orchestra.run.smithi204.stdout: "bluestore_cache_onode_items": 0, 2024-04-23T10:41:43.725 INFO:teuthology.orchestra.run.smithi204.stdout: "bluestore_cache_meta_bytes": 0, 2024-04-23T10:41:43.726 INFO:teuthology.orchestra.run.smithi204.stdout: "bluestore_cache_meta_items": 0, 2024-04-23T10:41:43.726 INFO:teuthology.orchestra.run.smithi204.stdout: "bluestore_cache_other_bytes": 0, 2024-04-23T10:41:43.726 INFO:teuthology.orchestra.run.smithi204.stdout: "bluestore_cache_other_items": 0, 2024-04-23T10:41:43.726 INFO:teuthology.orchestra.run.smithi204.stdout: "bluestore_cache_buffer_bytes": 0, 2024-04-23T10:41:43.726 INFO:teuthology.orchestra.run.smithi204.stdout: "bluestore_cache_buffer_items": 0, 2024-04-23T10:41:43.726 INFO:teuthology.orchestra.run.smithi204.stdout: "bluestore_extent_bytes": 0, 2024-04-23T10:41:43.727 INFO:teuthology.orchestra.run.smithi204.stdout: "bluestore_extent_items": 0, 2024-04-23T10:41:43.727 INFO:teuthology.orchestra.run.smithi204.stdout: "bluestore_blob_bytes": 0, 2024-04-23T10:41:43.727 INFO:teuthology.orchestra.run.smithi204.stdout: "bluestore_blob_items": 0, 2024-04-23T10:41:43.727 INFO:teuthology.orchestra.run.smithi204.stdout: "bluestore_shared_blob_bytes": 0, 2024-04-23T10:41:43.727 INFO:teuthology.orchestra.run.smithi204.stdout: "bluestore_shared_blob_items": 0, 2024-04-23T10:41:43.727 INFO:teuthology.orchestra.run.smithi204.stdout: "bluestore_inline_bl_bytes": 0, 2024-04-23T10:41:43.727 INFO:teuthology.orchestra.run.smithi204.stdout: "bluestore_inline_bl_items": 0, 2024-04-23T10:41:43.728 INFO:teuthology.orchestra.run.smithi204.stdout: "bluestore_fsck_bytes": 0, 2024-04-23T10:41:43.728 INFO:teuthology.orchestra.run.smithi204.stdout: "bluestore_fsck_items": 0, 2024-04-23T10:41:43.728 INFO:teuthology.orchestra.run.smithi204.stdout: "bluestore_txc_bytes": 0, 2024-04-23T10:41:43.728 INFO:teuthology.orchestra.run.smithi204.stdout: "bluestore_txc_items": 0, 2024-04-23T10:41:43.728 INFO:teuthology.orchestra.run.smithi204.stdout: "bluestore_writing_deferred_bytes": 0, 2024-04-23T10:41:43.728 INFO:teuthology.orchestra.run.smithi204.stdout: "bluestore_writing_deferred_items": 0, 2024-04-23T10:41:43.728 INFO:teuthology.orchestra.run.smithi204.stdout: "bluestore_writing_bytes": 0, 2024-04-23T10:41:43.729 INFO:teuthology.orchestra.run.smithi204.stdout: "bluestore_writing_items": 0, 2024-04-23T10:41:43.729 INFO:teuthology.orchestra.run.smithi204.stdout: "bluefs_bytes": 0, 2024-04-23T10:41:43.729 INFO:teuthology.orchestra.run.smithi204.stdout: "bluefs_items": 0, 2024-04-23T10:41:43.729 INFO:teuthology.orchestra.run.smithi204.stdout: "bluefs_file_reader_bytes": 0, 2024-04-23T10:41:43.729 INFO:teuthology.orchestra.run.smithi204.stdout: "bluefs_file_reader_items": 0, 2024-04-23T10:41:43.729 INFO:teuthology.orchestra.run.smithi204.stdout: "bluefs_file_writer_bytes": 0, 2024-04-23T10:41:43.730 INFO:teuthology.orchestra.run.smithi204.stdout: "bluefs_file_writer_items": 0, 2024-04-23T10:41:43.730 INFO:teuthology.orchestra.run.smithi204.stdout: "buffer_anon_bytes": 147856, 2024-04-23T10:41:43.730 INFO:teuthology.orchestra.run.smithi204.stdout: "buffer_anon_items": 25, 2024-04-23T10:41:43.730 INFO:teuthology.orchestra.run.smithi204.stdout: "buffer_meta_bytes": 0, 2024-04-23T10:41:43.730 INFO:teuthology.orchestra.run.smithi204.stdout: "buffer_meta_items": 0, 2024-04-23T10:41:43.730 INFO:teuthology.orchestra.run.smithi204.stdout: "osd_bytes": 0, 2024-04-23T10:41:43.731 INFO:teuthology.orchestra.run.smithi204.stdout: "osd_items": 0, 2024-04-23T10:41:43.731 INFO:teuthology.orchestra.run.smithi204.stdout: "osd_mapbl_bytes": 0, 2024-04-23T10:41:43.731 INFO:teuthology.orchestra.run.smithi204.stdout: "osd_mapbl_items": 0, 2024-04-23T10:41:43.731 INFO:teuthology.orchestra.run.smithi204.stdout: "osd_pglog_bytes": 0, 2024-04-23T10:41:43.731 INFO:teuthology.orchestra.run.smithi204.stdout: "osd_pglog_items": 0, 2024-04-23T10:41:43.732 INFO:teuthology.orchestra.run.smithi204.stdout: "osdmap_bytes": 0, 2024-04-23T10:41:43.732 INFO:teuthology.orchestra.run.smithi204.stdout: "osdmap_items": 0, 2024-04-23T10:41:43.732 INFO:teuthology.orchestra.run.smithi204.stdout: "osdmap_mapping_bytes": 0, 2024-04-23T10:41:43.732 INFO:teuthology.orchestra.run.smithi204.stdout: "osdmap_mapping_items": 0, 2024-04-23T10:41:43.732 INFO:teuthology.orchestra.run.smithi204.stdout: "pgmap_bytes": 0, 2024-04-23T10:41:43.732 INFO:teuthology.orchestra.run.smithi204.stdout: "pgmap_items": 0, 2024-04-23T10:41:43.732 INFO:teuthology.orchestra.run.smithi204.stdout: "mds_co_bytes": 928, 2024-04-23T10:41:43.733 INFO:teuthology.orchestra.run.smithi204.stdout: "mds_co_items": 2, 2024-04-23T10:41:43.733 INFO:teuthology.orchestra.run.smithi204.stdout: "unittest_1_bytes": 0, 2024-04-23T10:41:43.733 INFO:teuthology.orchestra.run.smithi204.stdout: "unittest_1_items": 0, 2024-04-23T10:41:43.733 INFO:teuthology.orchestra.run.smithi204.stdout: "unittest_2_bytes": 0, 2024-04-23T10:41:43.733 INFO:teuthology.orchestra.run.smithi204.stdout: "unittest_2_items": 0 2024-04-23T10:41:43.733 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-04-23T10:41:43.733 INFO:teuthology.orchestra.run.smithi204.stdout: "throttle-msgr_dispatch_throttler-mds": { 2024-04-23T10:41:43.734 INFO:teuthology.orchestra.run.smithi204.stdout: "val": 0, 2024-04-23T10:41:43.734 INFO:teuthology.orchestra.run.smithi204.stdout: "max": 104857600, 2024-04-23T10:41:43.734 INFO:teuthology.orchestra.run.smithi204.stdout: "get_started": 0, 2024-04-23T10:41:43.734 INFO:teuthology.orchestra.run.smithi204.stdout: "get": 298, 2024-04-23T10:41:43.734 INFO:teuthology.orchestra.run.smithi204.stdout: "get_sum": 1568660, 2024-04-23T10:41:43.734 INFO:teuthology.orchestra.run.smithi204.stdout: "get_or_fail_fail": 0, 2024-04-23T10:41:43.735 INFO:teuthology.orchestra.run.smithi204.stdout: "get_or_fail_success": 298, 2024-04-23T10:41:43.735 INFO:teuthology.orchestra.run.smithi204.stdout: "take": 0, 2024-04-23T10:41:43.735 INFO:teuthology.orchestra.run.smithi204.stdout: "take_sum": 0, 2024-04-23T10:41:43.735 INFO:teuthology.orchestra.run.smithi204.stdout: "put": 298, 2024-04-23T10:41:43.735 INFO:teuthology.orchestra.run.smithi204.stdout: "put_sum": 1568660, 2024-04-23T10:41:43.735 INFO:teuthology.orchestra.run.smithi204.stdout: "wait": { 2024-04-23T10:41:43.735 INFO:teuthology.orchestra.run.smithi204.stdout: "avgcount": 0, 2024-04-23T10:41:43.736 INFO:teuthology.orchestra.run.smithi204.stdout: "sum": 0.000000000, 2024-04-23T10:41:43.736 INFO:teuthology.orchestra.run.smithi204.stdout: "avgtime": 0.000000000 2024-04-23T10:41:43.736 INFO:teuthology.orchestra.run.smithi204.stdout: } 2024-04-23T10:41:43.736 INFO:teuthology.orchestra.run.smithi204.stdout: } 2024-04-23T10:41:43.736 INFO:teuthology.orchestra.run.smithi204.stdout:} 2024-04-23T10:41:43.992 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 23 10:41:43 smithi119 ceph-mon[32608]: pgmap v477: 97 pgs: 97 active+clean; 551 MiB data, 4.3 GiB used, 1.0 TiB / 1.0 TiB avail; 569 KiB/s rd, 3.1 MiB/s wr, 120 op/s 2024-04-23T10:41:44.004 INFO:journalctl@ceph.mon.a.smithi114.stdout:Apr 23 10:41:43 smithi114 ceph-mon[29255]: pgmap v477: 97 pgs: 97 active+clean; 551 MiB data, 4.3 GiB used, 1.0 TiB / 1.0 TiB avail; 569 KiB/s rd, 3.1 MiB/s wr, 120 op/s 2024-04-23T10:41:44.598 WARNING:tasks.check_counter:Counter 'mds.exported' not found on daemon mds.l 2024-04-23T10:41:44.598 WARNING:tasks.check_counter:Counter 'mds.imported' not found on daemon mds.l 2024-04-23T10:41:44.598 WARNING:tasks.check_counter:Counter 'mds_cache.dir_update' not found on daemon mds.l 2024-04-23T10:41:44.598 WARNING:tasks.check_counter:Counter 'mds_cache.dir_update_receipt' not found on daemon mds.l 2024-04-23T10:41:44.599 WARNING:tasks.check_counter:Counter 'mds.root_rsnaps' not found on daemon mds.l 2024-04-23T10:41:44.599 WARNING:tasks.check_counter:Counter 'mds_server.req_mksnap_latency.avgcount' not found on daemon mds.l 2024-04-23T10:41:44.599 WARNING:tasks.check_counter:Counter 'mds_server.req_rmsnap_latency.avgcount' not found on daemon mds.l 2024-04-23T10:41:44.599 DEBUG:teuthology.run_tasks:Unwinding manager kclient 2024-04-23T10:41:44.611 INFO:tasks.kclient:Unmounting kernel clients... 2024-04-23T10:41:44.611 DEBUG:teuthology.orchestra.run.smithi114:> set -ex 2024-04-23T10:41:44.611 DEBUG:teuthology.orchestra.run.smithi114:> dd if=/proc/self/mounts of=/dev/stdout 2024-04-23T10:41:44.633 DEBUG:teuthology.orchestra.run.smithi114:> set -ex 2024-04-23T10:41:44.633 DEBUG:teuthology.orchestra.run.smithi114:> dd if=/proc/self/mounts of=/dev/stdout 2024-04-23T10:41:44.693 DEBUG:teuthology.orchestra.run.smithi114:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-04-23T10:41:44.849 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:41:44.847+0000 7fe549319640 1 -- 172.21.15.114:0/3910819660 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fe54410e980 msgr2=0x7fe544110d70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T10:41:44.849 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:41:44.847+0000 7fe549319640 1 --2- 172.21.15.114:0/3910819660 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fe54410e980 0x7fe544110d70 secure :-1 s=READY pgs=963 cs=0 l=1 rev1=1 crypto rx=0x7fe53c009ca0 tx=0x7fe53c02a8c0 comp rx=0 tx=0).stop 2024-04-23T10:41:44.850 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:41:44.847+0000 7fe549319640 1 -- 172.21.15.114:0/3910819660 shutdown_connections 2024-04-23T10:41:44.850 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:41:44.847+0000 7fe549319640 1 --2- 172.21.15.114:0/3910819660 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fe54410e980 0x7fe544110d70 unknown :-1 s=CLOSED pgs=963 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:41:44.850 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:41:44.847+0000 7fe549319640 1 --2- 172.21.15.114:0/3910819660 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fe544074aa0 0x7fe54410e440 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:41:44.850 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:41:44.847+0000 7fe549319640 1 --2- 172.21.15.114:0/3910819660 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fe544074180 0x7fe544074560 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:41:44.850 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:41:44.847+0000 7fe549319640 1 -- 172.21.15.114:0/3910819660 >> 172.21.15.114:0/3910819660 conn(0x7fe5440790e0 msgr2=0x7fe5440794f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T10:41:44.851 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:41:44.847+0000 7fe549319640 1 -- 172.21.15.114:0/3910819660 shutdown_connections 2024-04-23T10:41:44.851 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:41:44.847+0000 7fe549319640 1 -- 172.21.15.114:0/3910819660 wait complete. 2024-04-23T10:41:44.851 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:41:44.848+0000 7fe549319640 1 Processor -- start 2024-04-23T10:41:44.851 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:41:44.848+0000 7fe549319640 1 -- start start 2024-04-23T10:41:44.851 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:41:44.848+0000 7fe549319640 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fe544074180 0x7fe5441a0650 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T10:41:44.851 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:41:44.848+0000 7fe549319640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fe544074aa0 0x7fe5441a0b90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T10:41:44.852 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:41:44.848+0000 7fe549319640 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fe54410e980 0x7fe54419a760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T10:41:44.852 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:41:44.848+0000 7fe549319640 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x7fe544114380 con 0x7fe544074aa0 2024-04-23T10:41:44.852 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:41:44.848+0000 7fe549319640 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 -- 0x7fe544114200 con 0x7fe544074180 2024-04-23T10:41:44.852 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:41:44.848+0000 7fe549319640 1 -- --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_getmap magic: 0 -- 0x7fe544114500 con 0x7fe54410e980 2024-04-23T10:41:44.852 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:41:44.848+0000 7fe5427fc640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fe544074aa0 0x7fe5441a0b90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T10:41:44.852 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:41:44.848+0000 7fe5427fc640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fe544074aa0 0x7fe5441a0b90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.114:36530/0 (socket says 172.21.15.114:36530) 2024-04-23T10:41:44.852 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:41:44.848+0000 7fe5427fc640 1 -- 172.21.15.114:0/1172586742 learned_addr learned my addr 172.21.15.114:0/1172586742 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-04-23T10:41:44.853 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:41:44.849+0000 7fe542ffd640 1 --2- 172.21.15.114:0/1172586742 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fe544074180 0x7fe5441a0650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T10:41:44.853 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:41:44.849+0000 7fe5437fe640 1 --2- 172.21.15.114:0/1172586742 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fe54410e980 0x7fe54419a760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T10:41:44.853 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:41:44.849+0000 7fe5427fc640 1 -- 172.21.15.114:0/1172586742 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fe54410e980 msgr2=0x7fe54419a760 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T10:41:44.853 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:41:44.849+0000 7fe5427fc640 1 --2- 172.21.15.114:0/1172586742 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fe54410e980 0x7fe54419a760 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:41:44.853 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:41:44.849+0000 7fe5427fc640 1 -- 172.21.15.114:0/1172586742 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fe544074180 msgr2=0x7fe5441a0650 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T10:41:44.853 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:41:44.849+0000 7fe5427fc640 1 --2- 172.21.15.114:0/1172586742 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fe544074180 0x7fe5441a0650 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:41:44.854 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:41:44.849+0000 7fe5427fc640 1 -- 172.21.15.114:0/1172586742 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe54419b020 con 0x7fe544074aa0 2024-04-23T10:41:44.854 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:41:44.849+0000 7fe5427fc640 1 --2- 172.21.15.114:0/1172586742 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fe544074aa0 0x7fe5441a0b90 secure :-1 s=READY pgs=964 cs=0 l=1 rev1=1 crypto rx=0x7fe53800e9f0 tx=0x7fe53800edf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T10:41:44.854 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:41:44.849+0000 7fe52bfff640 1 -- 172.21.15.114:0/1172586742 <== mon.0 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe53800cd80 con 0x7fe544074aa0 2024-04-23T10:41:44.854 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:41:44.849+0000 7fe52bfff640 1 -- 172.21.15.114:0/1172586742 <== mon.0 v2:172.21.15.114:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fe5380054d0 con 0x7fe544074aa0 2024-04-23T10:41:44.854 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:41:44.849+0000 7fe549319640 1 -- 172.21.15.114:0/1172586742 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fe54419b310 con 0x7fe544074aa0 2024-04-23T10:41:44.854 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:41:44.849+0000 7fe52bfff640 1 -- 172.21.15.114:0/1172586742 <== mon.0 v2:172.21.15.114:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe538010690 con 0x7fe544074aa0 2024-04-23T10:41:44.855 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:41:44.849+0000 7fe549319640 1 -- 172.21.15.114:0/1172586742 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fe5441a75d0 con 0x7fe544074aa0 2024-04-23T10:41:44.855 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:41:44.850+0000 7fe549319640 1 -- 172.21.15.114:0/1172586742 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fe5440757f0 con 0x7fe544074aa0 2024-04-23T10:41:44.856 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:41:44.852+0000 7fe52bfff640 1 -- 172.21.15.114:0/1172586742 <== mon.0 v2:172.21.15.114:3300/0 4 ==== mgrmap(e 26) ==== 138778+0+0 (secure 0 0 0) 0x7fe538004e30 con 0x7fe544074aa0 2024-04-23T10:41:44.856 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:41:44.853+0000 7fe52bfff640 1 --2- 172.21.15.114:0/1172586742 >> [v2:172.21.15.114:6800/1395693466,v1:172.21.15.114:6801/1395693466] conn(0x7fe52c0a3d20 0x7fe52c0a61e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T10:41:44.856 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:41:44.853+0000 7fe542ffd640 1 --2- 172.21.15.114:0/1172586742 >> [v2:172.21.15.114:6800/1395693466,v1:172.21.15.114:6801/1395693466] conn(0x7fe52c0a3d20 0x7fe52c0a61e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T10:41:44.857 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:41:44.853+0000 7fe52bfff640 1 -- 172.21.15.114:0/1172586742 <== mon.0 v2:172.21.15.114:3300/0 5 ==== osd_map(113..113 src has 1..113) ==== 8629+0+0 (secure 0 0 0) 0x7fe5380d0dc0 con 0x7fe544074aa0 2024-04-23T10:41:44.857 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:41:44.853+0000 7fe542ffd640 1 --2- 172.21.15.114:0/1172586742 >> [v2:172.21.15.114:6800/1395693466,v1:172.21.15.114:6801/1395693466] conn(0x7fe52c0a3d20 0x7fe52c0a61e0 secure :-1 s=READY pgs=178 cs=0 l=1 rev1=1 crypto rx=0x7fe53400bbe0 tx=0x7fe53400b390 comp rx=0 tx=0).ready entity=mgr.15291 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T10:41:44.857 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:41:44.856+0000 7fe52bfff640 1 -- 172.21.15.114:0/1172586742 <== mon.0 v2:172.21.15.114:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7fe538053ff0 con 0x7fe544074aa0 2024-04-23T10:41:45.033 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:41:45.034+0000 7fe549319640 1 -- 172.21.15.114:0/1172586742 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7fe54419b540 con 0x7fe544074aa0 2024-04-23T10:41:45.035 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:41:45.036+0000 7fe52bfff640 1 -- 172.21.15.114:0/1172586742 <== mon.0 v2:172.21.15.114:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v113) ==== 74+0+23386 (secure 0 0 0) 0x7fe53809d3f0 con 0x7fe544074aa0 2024-04-23T10:41:45.035 INFO:teuthology.orchestra.run.smithi114.stdout: 2024-04-23T10:41:45.036 INFO:teuthology.orchestra.run.smithi114.stdout:{"epoch":113,"fsid":"aa240572-0159-11ef-bc93-c7b262605968","created":"2024-04-23T10:16:53.351136+0000","modified":"2024-04-23T10:41:13.513569+0000","last_up_change":"2024-04-23T10:23:54.153269+0000","last_in_change":"2024-04-23T10:23:32.963085+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":26,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-04-23T10:20:11.970297+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":12,"score_stable":12,"optimal_score":0.25,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-04-23T10:24:41.218540+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"86","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.8799999952316284,"score_stable":1.8799999952316284,"optimal_score":1,"raw_score_acting":1.8799999952316284,"raw_score_stable":1.8799999952316284,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-04-23T10:24:42.341799+0000","flags":8193,"flags_names":"hashpspool,selfmanaged_snaps","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"112","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":25,"snap_epoch":112,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"fd64adde-9445-45a4-a171-ef656e2c5709","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":88,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6802","nonce":736366878},{"type":"v1","addr":"172.21.15.114:6803","nonce":736366878}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6804","nonce":736366878},{"type":"v1","addr":"172.21.15.114:6805","nonce":736366878}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6808","nonce":736366878},{"type":"v1","addr":"172.21.15.114:6809","nonce":736366878}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6806","nonce":736366878},{"type":"v1","addr":"172.21.15.114:6807","nonce":736366878}]},"public_addr":"172.21.15.114:6803/736366878","cluster_addr":"172.21.15.114:6805/736366878","heartbeat_back_addr":"172.21.15.114:6809/736366878","heartbeat_front_addr":"172.21.15.114:6807/736366878","state":["exists","up"]},{"osd":1,"uuid":"e279a1e6-2702-472a-930a-e08ece76a861","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":85,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6800","nonce":1133540779},{"type":"v1","addr":"172.21.15.119:6801","nonce":1133540779}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6802","nonce":1133540779},{"type":"v1","addr":"172.21.15.119:6803","nonce":1133540779}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6806","nonce":1133540779},{"type":"v1","addr":"172.21.15.119:6807","nonce":1133540779}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6804","nonce":1133540779},{"type":"v1","addr":"172.21.15.119:6805","nonce":1133540779}]},"public_addr":"172.21.15.119:6801/1133540779","cluster_addr":"172.21.15.119:6803/1133540779","heartbeat_back_addr":"172.21.15.119:6807/1133540779","heartbeat_front_addr":"172.21.15.119:6805/1133540779","state":["exists","up"]},{"osd":2,"uuid":"f4c9f2b4-ff49-4e14-89cd-b477feeb9382","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":85,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6800","nonce":799652511},{"type":"v1","addr":"172.21.15.204:6801","nonce":799652511}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6802","nonce":799652511},{"type":"v1","addr":"172.21.15.204:6803","nonce":799652511}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6806","nonce":799652511},{"type":"v1","addr":"172.21.15.204:6807","nonce":799652511}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6804","nonce":799652511},{"type":"v1","addr":"172.21.15.204:6805","nonce":799652511}]},"public_addr":"172.21.15.204:6801/799652511","cluster_addr":"172.21.15.204:6803/799652511","heartbeat_back_addr":"172.21.15.204:6807/799652511","heartbeat_front_addr":"172.21.15.204:6805/799652511","state":["exists","up"]},{"osd":3,"uuid":"52d2a566-e71c-4bb3-8d6c-5abdd43177c2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":25,"up_thru":88,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6810","nonce":4275595091},{"type":"v1","addr":"172.21.15.114:6811","nonce":4275595091}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6812","nonce":4275595091},{"type":"v1","addr":"172.21.15.114:6813","nonce":4275595091}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6816","nonce":4275595091},{"type":"v1","addr":"172.21.15.114:6817","nonce":4275595091}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6814","nonce":4275595091},{"type":"v1","addr":"172.21.15.114:6815","nonce":4275595091}]},"public_addr":"172.21.15.114:6811/4275595091","cluster_addr":"172.21.15.114:6813/4275595091","heartbeat_back_addr":"172.21.15.114:6817/4275595091","heartbeat_front_addr":"172.21.15.114:6815/4275595091","state":["exists","up"]},{"osd":4,"uuid":"d911a7c1-4bdc-4d28-b8a6-73896a6e7de6","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":33,"up_thru":85,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6808","nonce":1611007172},{"type":"v1","addr":"172.21.15.119:6809","nonce":1611007172}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6810","nonce":1611007172},{"type":"v1","addr":"172.21.15.119:6811","nonce":1611007172}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6814","nonce":1611007172},{"type":"v1","addr":"172.21.15.119:6815","nonce":1611007172}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6812","nonce":1611007172},{"type":"v1","addr":"172.21.15.119:6813","nonce":1611007172}]},"public_addr":"172.21.15.119:6809/1611007172","cluster_addr":"172.21.15.119:6811/1611007172","heartbeat_back_addr":"172.21.15.119:6815/1611007172","heartbeat_front_addr":"172.21.15.119:6813/1611007172","state":["exists","up"]},{"osd":5,"uuid":"f7590e1a-e93c-4c29-b15f-38ee52d1f921","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":39,"up_thru":85,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6808","nonce":1427563312},{"type":"v1","addr":"172.21.15.204:6809","nonce":1427563312}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6810","nonce":1427563312},{"type":"v1","addr":"172.21.15.204:6811","nonce":1427563312}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6814","nonce":1427563312},{"type":"v1","addr":"172.21.15.204:6815","nonce":1427563312}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6812","nonce":1427563312},{"type":"v1","addr":"172.21.15.204:6813","nonce":1427563312}]},"public_addr":"172.21.15.204:6809/1427563312","cluster_addr":"172.21.15.204:6811/1427563312","heartbeat_back_addr":"172.21.15.204:6815/1427563312","heartbeat_front_addr":"172.21.15.204:6813/1427563312","state":["exists","up"]},{"osd":6,"uuid":"534d20e8-3e54-4db8-a845-6119f49c7a78","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":47,"up_thru":88,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6818","nonce":4072912544},{"type":"v1","addr":"172.21.15.114:6819","nonce":4072912544}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6820","nonce":4072912544},{"type":"v1","addr":"172.21.15.114:6821","nonce":4072912544}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6824","nonce":4072912544},{"type":"v1","addr":"172.21.15.114:6825","nonce":4072912544}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6822","nonce":4072912544},{"type":"v1","addr":"172.21.15.114:6823","nonce":4072912544}]},"public_addr":"172.21.15.114:6819/4072912544","cluster_addr":"172.21.15.114:6821/4072912544","heartbeat_back_addr":"172.21.15.114:6825/4072912544","heartbeat_front_addr":"172.21.15.114:6823/4072912544","state":["exists","up"]},{"osd":7,"uuid":"f0ec22b8-1b48-4859-9d65-b00403aff3ad","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":53,"up_thru":88,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6816","nonce":2237458793},{"type":"v1","addr":"172.21.15.119:6817","nonce":2237458793}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6818","nonce":2237458793},{"type":"v1","addr":"172.21.15.119:6819","nonce":2237458793}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6822","nonce":2237458793},{"type":"v1","addr":"172.21.15.119:6823","nonce":2237458793}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6820","nonce":2237458793},{"type":"v1","addr":"172.21.15.119:6821","nonce":2237458793}]},"public_addr":"172.21.15.119:6817/2237458793","cluster_addr":"172.21.15.119:6819/2237458793","heartbeat_back_addr":"172.21.15.119:6823/2237458793","heartbeat_front_addr":"172.21.15.119:6821/2237458793","state":["exists","up"]},{"osd":8,"uuid":"3ac89686-6d76-4cef-9089-ac29e4b3bcac","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":62,"up_thru":91,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6816","nonce":1426318231},{"type":"v1","addr":"172.21.15.204:6817","nonce":1426318231}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6818","nonce":1426318231},{"type":"v1","addr":"172.21.15.204:6819","nonce":1426318231}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6822","nonce":1426318231},{"type":"v1","addr":"172.21.15.204:6823","nonce":1426318231}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6820","nonce":1426318231},{"type":"v1","addr":"172.21.15.204:6821","nonce":1426318231}]},"public_addr":"172.21.15.204:6817/1426318231","cluster_addr":"172.21.15.204:6819/1426318231","heartbeat_back_addr":"172.21.15.204:6823/1426318231","heartbeat_front_addr":"172.21.15.204:6821/1426318231","state":["exists","up"]},{"osd":9,"uuid":"a80ffe4d-1e98-460f-b046-c4754285cc5a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":69,"up_thru":88,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6826","nonce":2750635721},{"type":"v1","addr":"172.21.15.114:6827","nonce":2750635721}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6828","nonce":2750635721},{"type":"v1","addr":"172.21.15.114:6829","nonce":2750635721}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6832","nonce":2750635721},{"type":"v1","addr":"172.21.15.114:6833","nonce":2750635721}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6830","nonce":2750635721},{"type":"v1","addr":"172.21.15.114:6831","nonce":2750635721}]},"public_addr":"172.21.15.114:6827/2750635721","cluster_addr":"172.21.15.114:6829/2750635721","heartbeat_back_addr":"172.21.15.114:6833/2750635721","heartbeat_front_addr":"172.21.15.114:6831/2750635721","state":["exists","up"]},{"osd":10,"uuid":"cd75a579-bf12-4600-ac09-a8bfce772ced","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":74,"up_thru":85,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6824","nonce":1400187701},{"type":"v1","addr":"172.21.15.119:6825","nonce":1400187701}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6826","nonce":1400187701},{"type":"v1","addr":"172.21.15.119:6827","nonce":1400187701}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6830","nonce":1400187701},{"type":"v1","addr":"172.21.15.119:6831","nonce":1400187701}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6828","nonce":1400187701},{"type":"v1","addr":"172.21.15.119:6829","nonce":1400187701}]},"public_addr":"172.21.15.119:6825/1400187701","cluster_addr":"172.21.15.119:6827/1400187701","heartbeat_back_addr":"172.21.15.119:6831/1400187701","heartbeat_front_addr":"172.21.15.119:6829/1400187701","state":["exists","up"]},{"osd":11,"uuid":"53f037a5-3ec0-483a-80a1-dfee923a68e5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":82,"up_thru":91,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6824","nonce":2407298490},{"type":"v1","addr":"172.21.15.204:6825","nonce":2407298490}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6826","nonce":2407298490},{"type":"v1","addr":"172.21.15.204:6827","nonce":2407298490}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6830","nonce":2407298490},{"type":"v1","addr":"172.21.15.204:6831","nonce":2407298490}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6828","nonce":2407298490},{"type":"v1","addr":"172.21.15.204:6829","nonce":2407298490}]},"public_addr":"172.21.15.204:6825/2407298490","cluster_addr":"172.21.15.204:6827/2407298490","heartbeat_back_addr":"172.21.15.204:6831/2407298490","heartbeat_front_addr":"172.21.15.204:6829/2407298490","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-23T10:19:18.401147+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-23T10:19:42.611299+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-23T10:20:07.277367+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-23T10:20:31.208368+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-23T10:20:55.866225+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-23T10:21:19.961237+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-23T10:21:44.480783+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-23T10:22:09.173641+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-23T10:22:34.030328+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-23T10:22:59.652293+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-23T10:23:25.659673+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-23T10:23:51.168171+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"2.b","mappings":[{"from":8,"to":11}]},{"pgid":"2.e","mappings":[{"from":2,"to":4}]},{"pgid":"2.10","mappings":[{"from":2,"to":1}]},{"pgid":"2.1d","mappings":[{"from":8,"to":11}]},{"pgid":"3.e","mappings":[{"from":5,"to":9}]},{"pgid":"3.12","mappings":[{"from":3,"to":9}]},{"pgid":"3.17","mappings":[{"from":4,"to":6}]},{"pgid":"3.23","mappings":[{"from":4,"to":7}]},{"pgid":"3.27","mappings":[{"from":4,"to":6}]},{"pgid":"3.31","mappings":[{"from":4,"to":8}]},{"pgid":"3.38","mappings":[{"from":3,"to":9}]},{"pgid":"3.39","mappings":[{"from":5,"to":8}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.114:0/480720790":"2024-04-24T10:26:40.272662+0000","172.21.15.114:6801/465333808":"2024-04-24T10:26:40.272662+0000","172.21.15.114:6800/465333808":"2024-04-24T10:26:40.272662+0000","172.21.15.114:0/1787801099":"2024-04-24T10:26:40.272662+0000","172.21.15.114:6801/1604678557":"2024-04-24T10:17:45.874609+0000","172.21.15.114:0/4025385854":"2024-04-24T10:17:45.874609+0000","172.21.15.114:0/2228273469":"2024-04-24T10:17:19.910348+0000","172.21.15.114:0/2713201622":"2024-04-24T10:26:40.272662+0000","172.21.15.114:6800/2831530838":"2024-04-24T10:17:19.910348+0000","172.21.15.114:0/637004095":"2024-04-24T10:26:40.272662+0000","172.21.15.114:0/1333447450":"2024-04-24T10:17:45.874609+0000","172.21.15.114:0/2233026840":"2024-04-24T10:17:19.910348+0000","172.21.15.114:0/1902281545":"2024-04-24T10:17:19.910348+0000","172.21.15.114:6801/2831530838":"2024-04-24T10:17:19.910348+0000","172.21.15.114:6800/1604678557":"2024-04-24T10:17:45.874609+0000","172.21.15.114:0/1623553811":"2024-04-24T10:17:45.874609+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[{"pool":3,"snaps":[{"begin":12,"length":1},{"begin":25,"length":1}]}],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-23T10:41:45.037 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:41:45.038+0000 7fe549319640 1 -- 172.21.15.114:0/1172586742 >> [v2:172.21.15.114:6800/1395693466,v1:172.21.15.114:6801/1395693466] conn(0x7fe52c0a3d20 msgr2=0x7fe52c0a61e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T10:41:45.037 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:41:45.038+0000 7fe549319640 1 --2- 172.21.15.114:0/1172586742 >> [v2:172.21.15.114:6800/1395693466,v1:172.21.15.114:6801/1395693466] conn(0x7fe52c0a3d20 0x7fe52c0a61e0 secure :-1 s=READY pgs=178 cs=0 l=1 rev1=1 crypto rx=0x7fe53400bbe0 tx=0x7fe53400b390 comp rx=0 tx=0).stop 2024-04-23T10:41:45.038 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:41:45.038+0000 7fe549319640 1 -- 172.21.15.114:0/1172586742 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fe544074aa0 msgr2=0x7fe5441a0b90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T10:41:45.038 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:41:45.038+0000 7fe549319640 1 --2- 172.21.15.114:0/1172586742 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fe544074aa0 0x7fe5441a0b90 secure :-1 s=READY pgs=964 cs=0 l=1 rev1=1 crypto rx=0x7fe53800e9f0 tx=0x7fe53800edf0 comp rx=0 tx=0).stop 2024-04-23T10:41:45.038 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:41:45.038+0000 7fe549319640 1 -- 172.21.15.114:0/1172586742 shutdown_connections 2024-04-23T10:41:45.038 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:41:45.038+0000 7fe549319640 1 --2- 172.21.15.114:0/1172586742 >> [v2:172.21.15.114:6800/1395693466,v1:172.21.15.114:6801/1395693466] conn(0x7fe52c0a3d20 0x7fe52c0a61e0 unknown :-1 s=CLOSED pgs=178 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:41:45.038 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:41:45.038+0000 7fe549319640 1 --2- 172.21.15.114:0/1172586742 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fe54410e980 0x7fe54419a760 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:41:45.038 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:41:45.038+0000 7fe549319640 1 --2- 172.21.15.114:0/1172586742 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fe544074aa0 0x7fe5441a0b90 unknown :-1 s=CLOSED pgs=964 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:41:45.039 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:41:45.038+0000 7fe549319640 1 --2- 172.21.15.114:0/1172586742 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fe544074180 0x7fe5441a0650 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:41:45.039 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:41:45.038+0000 7fe549319640 1 -- 172.21.15.114:0/1172586742 >> 172.21.15.114:0/1172586742 conn(0x7fe5440790e0 msgr2=0x7fe5440ff230 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T10:41:45.039 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:41:45.038+0000 7fe549319640 1 -- 172.21.15.114:0/1172586742 shutdown_connections 2024-04-23T10:41:45.039 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:41:45.038+0000 7fe549319640 1 -- 172.21.15.114:0/1172586742 wait complete. 2024-04-23T10:41:45.048 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-04-23T10:20:11.970297+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '22', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 12, 'score_stable': 12, 'optimal_score': 0.25, 'raw_score_acting': 3, 'raw_score_stable': 3, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 2, 'pool_name': 'cephfs_metadata', 'create_time': '2024-04-23T10:24:41.218540+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '86', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.8799999952316284, 'score_stable': 1.8799999952316284, 'optimal_score': 1, 'raw_score_acting': 1.8799999952316284, 'raw_score_stable': 1.8799999952316284, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 3, 'pool_name': 'cephfs_data', 'create_time': '2024-04-23T10:24:42.341799+0000', 'flags': 8193, 'flags_names': 'hashpspool,selfmanaged_snaps', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '112', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 25, 'snap_epoch': 112, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-04-23T10:41:45.049 DEBUG:teuthology.orchestra.run.smithi114:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-04-23T10:41:45.153 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:41:45.150+0000 7f32d2605640 1 -- 172.21.15.114:0/2605713274 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f32cc106170 msgr2=0x7f32cc10dd40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T10:41:45.153 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:41:45.150+0000 7f32d2605640 1 --2- 172.21.15.114:0/2605713274 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f32cc106170 0x7f32cc10dd40 secure :-1 s=READY pgs=965 cs=0 l=1 rev1=1 crypto rx=0x7f32c4009ca0 tx=0x7f32c402a8c0 comp rx=0 tx=0).stop 2024-04-23T10:41:45.154 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:41:45.150+0000 7f32d2605640 1 -- 172.21.15.114:0/2605713274 shutdown_connections 2024-04-23T10:41:45.154 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:41:45.150+0000 7f32d2605640 1 --2- 172.21.15.114:0/2605713274 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f32cc106170 0x7f32cc10dd40 unknown :-1 s=CLOSED pgs=965 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:41:45.154 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:41:45.150+0000 7f32d2605640 1 --2- 172.21.15.114:0/2605713274 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f32cc101a50 0x7f32cc105a90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:41:45.154 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:41:45.150+0000 7f32d2605640 1 --2- 172.21.15.114:0/2605713274 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f32cc1010a0 0x7f32cc101480 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:41:45.154 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:41:45.150+0000 7f32d2605640 1 -- 172.21.15.114:0/2605713274 >> 172.21.15.114:0/2605713274 conn(0x7f32cc077eb0 msgr2=0x7f32cc0782c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T10:41:45.154 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:41:45.150+0000 7f32d2605640 1 -- 172.21.15.114:0/2605713274 shutdown_connections 2024-04-23T10:41:45.154 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:41:45.150+0000 7f32d2605640 1 -- 172.21.15.114:0/2605713274 wait complete. 2024-04-23T10:41:45.154 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:41:45.150+0000 7f32d2605640 1 Processor -- start 2024-04-23T10:41:45.155 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:41:45.150+0000 7f32d2605640 1 -- start start 2024-04-23T10:41:45.155 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:41:45.151+0000 7f32d2605640 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f32cc1010a0 0x7f32cc197330 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T10:41:45.155 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:41:45.151+0000 7f32d2605640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f32cc106170 0x7f32cc197870 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T10:41:45.155 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:41:45.151+0000 7f32d2605640 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f32cc197db0 0x7f32cc1a69b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T10:41:45.155 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:41:45.151+0000 7f32d2605640 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x7f32cc1110e0 con 0x7f32cc106170 2024-04-23T10:41:45.155 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:41:45.151+0000 7f32d2605640 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 -- 0x7f32cc110f60 con 0x7f32cc1010a0 2024-04-23T10:41:45.155 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:41:45.151+0000 7f32d2605640 1 -- --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_getmap magic: 0 -- 0x7f32cc111260 con 0x7f32cc197db0 2024-04-23T10:41:45.155 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:41:45.151+0000 7f32cb7fe640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f32cc106170 0x7f32cc197870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T10:41:45.156 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:41:45.151+0000 7f32cb7fe640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f32cc106170 0x7f32cc197870 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.114:36546/0 (socket says 172.21.15.114:36546) 2024-04-23T10:41:45.156 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:41:45.151+0000 7f32cb7fe640 1 -- 172.21.15.114:0/794934853 learned_addr learned my addr 172.21.15.114:0/794934853 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-04-23T10:41:45.156 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:41:45.151+0000 7f32d0ba2640 1 --2- 172.21.15.114:0/794934853 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f32cc197db0 0x7f32cc1a69b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T10:41:45.156 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:41:45.151+0000 7f32cbfff640 1 --2- 172.21.15.114:0/794934853 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f32cc1010a0 0x7f32cc197330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T10:41:45.156 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:41:45.151+0000 7f32cb7fe640 1 -- 172.21.15.114:0/794934853 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f32cc197db0 msgr2=0x7f32cc1a69b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T10:41:45.156 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:41:45.151+0000 7f32cb7fe640 1 --2- 172.21.15.114:0/794934853 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f32cc197db0 0x7f32cc1a69b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:41:45.156 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:41:45.151+0000 7f32cb7fe640 1 -- 172.21.15.114:0/794934853 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f32cc1010a0 msgr2=0x7f32cc197330 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T10:41:45.156 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:41:45.151+0000 7f32cb7fe640 1 --2- 172.21.15.114:0/794934853 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f32cc1010a0 0x7f32cc197330 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:41:45.156 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:41:45.151+0000 7f32cb7fe640 1 -- 172.21.15.114:0/794934853 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f32cc1a6f70 con 0x7f32cc106170 2024-04-23T10:41:45.157 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:41:45.151+0000 7f32d0ba2640 1 --2- 172.21.15.114:0/794934853 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f32cc197db0 0x7f32cc1a69b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-23T10:41:45.157 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:41:45.151+0000 7f32cbfff640 1 --2- 172.21.15.114:0/794934853 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f32cc1010a0 0x7f32cc197330 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-23T10:41:45.157 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:41:45.151+0000 7f32cb7fe640 1 --2- 172.21.15.114:0/794934853 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f32cc106170 0x7f32cc197870 secure :-1 s=READY pgs=966 cs=0 l=1 rev1=1 crypto rx=0x7f32c000d770 tx=0x7f32c000dc30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T10:41:45.157 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:41:45.151+0000 7f32c97fa640 1 -- 172.21.15.114:0/794934853 <== mon.0 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f32c0017070 con 0x7f32cc106170 2024-04-23T10:41:45.157 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:41:45.152+0000 7f32d2605640 1 -- 172.21.15.114:0/794934853 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f32cc1a7260 con 0x7f32cc106170 2024-04-23T10:41:45.157 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:41:45.152+0000 7f32c97fa640 1 -- 172.21.15.114:0/794934853 <== mon.0 v2:172.21.15.114:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f32c000de80 con 0x7f32cc106170 2024-04-23T10:41:45.157 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:41:45.152+0000 7f32c97fa640 1 -- 172.21.15.114:0/794934853 <== mon.0 v2:172.21.15.114:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f32c0004c30 con 0x7f32cc106170 2024-04-23T10:41:45.157 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:41:45.152+0000 7f32d2605640 1 -- 172.21.15.114:0/794934853 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f32cc1a7760 con 0x7f32cc106170 2024-04-23T10:41:45.157 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:41:45.152+0000 7f32d2605640 1 -- 172.21.15.114:0/794934853 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f32cc0691c0 con 0x7f32cc106170 2024-04-23T10:41:45.158 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:41:45.154+0000 7f32c97fa640 1 -- 172.21.15.114:0/794934853 <== mon.0 v2:172.21.15.114:3300/0 4 ==== mgrmap(e 26) ==== 138778+0+0 (secure 0 0 0) 0x7f32c0004ea0 con 0x7f32cc106170 2024-04-23T10:41:45.158 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:41:45.154+0000 7f32c97fa640 1 --2- 172.21.15.114:0/794934853 >> [v2:172.21.15.114:6800/1395693466,v1:172.21.15.114:6801/1395693466] conn(0x7f32b40a3d20 0x7f32b40a61e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T10:41:45.158 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:41:45.154+0000 7f32cbfff640 1 --2- 172.21.15.114:0/794934853 >> [v2:172.21.15.114:6800/1395693466,v1:172.21.15.114:6801/1395693466] conn(0x7f32b40a3d20 0x7f32b40a61e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T10:41:45.158 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:41:45.154+0000 7f32c97fa640 1 -- 172.21.15.114:0/794934853 <== mon.0 v2:172.21.15.114:3300/0 5 ==== osd_map(113..113 src has 1..113) ==== 8629+0+0 (secure 0 0 0) 0x7f32c00d02d0 con 0x7f32cc106170 2024-04-23T10:41:45.158 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:41:45.155+0000 7f32cbfff640 1 --2- 172.21.15.114:0/794934853 >> [v2:172.21.15.114:6800/1395693466,v1:172.21.15.114:6801/1395693466] conn(0x7f32b40a3d20 0x7f32b40a61e0 secure :-1 s=READY pgs=179 cs=0 l=1 rev1=1 crypto rx=0x7f32bc005d00 tx=0x7f32bc009550 comp rx=0 tx=0).ready entity=mgr.15291 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T10:41:45.159 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:41:45.159+0000 7f32c97fa640 1 -- 172.21.15.114:0/794934853 <== mon.0 v2:172.21.15.114:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f32c009c900 con 0x7f32cc106170 2024-04-23T10:41:45.335 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:41:45.334+0000 7f32d2605640 1 -- 172.21.15.114:0/794934853 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) -- 0x7f32cc101a50 con 0x7f32cc106170 2024-04-23T10:41:45.335 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:41:45.335+0000 7f32c97fa640 1 -- 172.21.15.114:0/794934853 <== mon.0 v2:172.21.15.114:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v113) ==== 93+0+10 (secure 0 0 0) 0x7f32c009c200 con 0x7f32cc106170 2024-04-23T10:41:45.336 INFO:teuthology.orchestra.run.smithi114.stdout:pg_num: 1 2024-04-23T10:41:45.338 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:41:45.336+0000 7f32d2605640 1 -- 172.21.15.114:0/794934853 >> [v2:172.21.15.114:6800/1395693466,v1:172.21.15.114:6801/1395693466] conn(0x7f32b40a3d20 msgr2=0x7f32b40a61e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T10:41:45.338 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:41:45.336+0000 7f32d2605640 1 --2- 172.21.15.114:0/794934853 >> [v2:172.21.15.114:6800/1395693466,v1:172.21.15.114:6801/1395693466] conn(0x7f32b40a3d20 0x7f32b40a61e0 secure :-1 s=READY pgs=179 cs=0 l=1 rev1=1 crypto rx=0x7f32bc005d00 tx=0x7f32bc009550 comp rx=0 tx=0).stop 2024-04-23T10:41:45.338 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:41:45.336+0000 7f32d2605640 1 -- 172.21.15.114:0/794934853 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f32cc106170 msgr2=0x7f32cc197870 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T10:41:45.338 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:41:45.336+0000 7f32d2605640 1 --2- 172.21.15.114:0/794934853 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f32cc106170 0x7f32cc197870 secure :-1 s=READY pgs=966 cs=0 l=1 rev1=1 crypto rx=0x7f32c000d770 tx=0x7f32c000dc30 comp rx=0 tx=0).stop 2024-04-23T10:41:45.339 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:41:45.337+0000 7f32d2605640 1 -- 172.21.15.114:0/794934853 shutdown_connections 2024-04-23T10:41:45.339 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:41:45.337+0000 7f32d2605640 1 --2- 172.21.15.114:0/794934853 >> [v2:172.21.15.114:6800/1395693466,v1:172.21.15.114:6801/1395693466] conn(0x7f32b40a3d20 0x7f32b40a61e0 unknown :-1 s=CLOSED pgs=179 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:41:45.339 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:41:45.337+0000 7f32d2605640 1 --2- 172.21.15.114:0/794934853 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f32cc197db0 0x7f32cc1a69b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:41:45.339 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:41:45.337+0000 7f32d2605640 1 --2- 172.21.15.114:0/794934853 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f32cc106170 0x7f32cc197870 unknown :-1 s=CLOSED pgs=966 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:41:45.339 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:41:45.337+0000 7f32d2605640 1 --2- 172.21.15.114:0/794934853 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f32cc1010a0 0x7f32cc197330 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:41:45.339 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:41:45.337+0000 7f32d2605640 1 -- 172.21.15.114:0/794934853 >> 172.21.15.114:0/794934853 conn(0x7f32cc077eb0 msgr2=0x7f32cc0fede0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T10:41:45.340 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:41:45.337+0000 7f32d2605640 1 -- 172.21.15.114:0/794934853 shutdown_connections 2024-04-23T10:41:45.340 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:41:45.337+0000 7f32d2605640 1 -- 172.21.15.114:0/794934853 wait complete. 2024-04-23T10:41:45.347 DEBUG:teuthology.orchestra.run.smithi114:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2024-04-23T10:41:45.451 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:41:45.450+0000 7f944c474640 1 -- 172.21.15.114:0/786140723 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f9444106160 msgr2=0x7f944410dd30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T10:41:45.452 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:41:45.450+0000 7f944c474640 1 --2- 172.21.15.114:0/786140723 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f9444106160 0x7f944410dd30 secure :-1 s=READY pgs=967 cs=0 l=1 rev1=1 crypto rx=0x7f9434009ca0 tx=0x7f943402a8c0 comp rx=0 tx=0).stop 2024-04-23T10:41:45.452 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:41:45.450+0000 7f944c474640 1 -- 172.21.15.114:0/786140723 shutdown_connections 2024-04-23T10:41:45.452 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:41:45.450+0000 7f944c474640 1 --2- 172.21.15.114:0/786140723 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f9444106160 0x7f944410dd30 unknown :-1 s=CLOSED pgs=967 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:41:45.452 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:41:45.450+0000 7f944c474640 1 --2- 172.21.15.114:0/786140723 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f9444101a40 0x7f9444105a80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:41:45.452 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:41:45.450+0000 7f944c474640 1 --2- 172.21.15.114:0/786140723 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f9444101090 0x7f9444101470 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:41:45.452 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:41:45.450+0000 7f944c474640 1 -- 172.21.15.114:0/786140723 >> 172.21.15.114:0/786140723 conn(0x7f94440771d0 msgr2=0x7f94440775e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T10:41:45.452 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:41:45.450+0000 7f944c474640 1 -- 172.21.15.114:0/786140723 shutdown_connections 2024-04-23T10:41:45.453 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:41:45.450+0000 7f944c474640 1 -- 172.21.15.114:0/786140723 wait complete. 2024-04-23T10:41:45.453 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:41:45.451+0000 7f944c474640 1 Processor -- start 2024-04-23T10:41:45.453 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:41:45.451+0000 7f944c474640 1 -- start start 2024-04-23T10:41:45.453 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:41:45.451+0000 7f944c474640 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f9444101090 0x7f94441995b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T10:41:45.453 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:41:45.451+0000 7f944c474640 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f9444106160 0x7f9444199af0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T10:41:45.453 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:41:45.451+0000 7f944c474640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f944419a030 0x7f94441a4630 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T10:41:45.453 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:41:45.451+0000 7f944c474640 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x7f94441110d0 con 0x7f944419a030 2024-04-23T10:41:45.454 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:41:45.451+0000 7f944c474640 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 -- 0x7f9444110f50 con 0x7f9444106160 2024-04-23T10:41:45.454 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:41:45.451+0000 7f944c474640 1 -- --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_getmap magic: 0 -- 0x7f9444111250 con 0x7f9444101090 2024-04-23T10:41:45.454 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:41:45.451+0000 7f944aa11640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f944419a030 0x7f94441a4630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T10:41:45.454 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:41:45.451+0000 7f944aa11640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f944419a030 0x7f94441a4630 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.114:36560/0 (socket says 172.21.15.114:36560) 2024-04-23T10:41:45.454 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:41:45.451+0000 7f944aa11640 1 -- 172.21.15.114:0/2203823746 learned_addr learned my addr 172.21.15.114:0/2203823746 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-04-23T10:41:45.454 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:41:45.451+0000 7f944a210640 1 --2- 172.21.15.114:0/2203823746 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f9444101090 0x7f94441995b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T10:41:45.454 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:41:45.451+0000 7f9449a0f640 1 --2- 172.21.15.114:0/2203823746 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f9444106160 0x7f9444199af0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T10:41:45.454 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:41:45.451+0000 7f944aa11640 1 -- 172.21.15.114:0/2203823746 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f9444101090 msgr2=0x7f94441995b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T10:41:45.455 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:41:45.451+0000 7f944aa11640 1 --2- 172.21.15.114:0/2203823746 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f9444101090 0x7f94441995b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:41:45.455 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:41:45.451+0000 7f944aa11640 1 -- 172.21.15.114:0/2203823746 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f9444106160 msgr2=0x7f9444199af0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T10:41:45.455 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:41:45.451+0000 7f944aa11640 1 --2- 172.21.15.114:0/2203823746 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f9444106160 0x7f9444199af0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:41:45.455 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:41:45.451+0000 7f944aa11640 1 -- 172.21.15.114:0/2203823746 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f94441a4db0 con 0x7f944419a030 2024-04-23T10:41:45.455 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:41:45.451+0000 7f9449a0f640 1 --2- 172.21.15.114:0/2203823746 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f9444106160 0x7f9444199af0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-23T10:41:45.455 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:41:45.451+0000 7f944aa11640 1 --2- 172.21.15.114:0/2203823746 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f944419a030 0x7f94441a4630 secure :-1 s=READY pgs=968 cs=0 l=1 rev1=1 crypto rx=0x7f94340098d0 tx=0x7f94340317b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T10:41:45.455 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:41:45.452+0000 7f94437fe640 1 -- 172.21.15.114:0/2203823746 <== mon.0 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9434003b70 con 0x7f944419a030 2024-04-23T10:41:45.455 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:41:45.452+0000 7f944c474640 1 -- 172.21.15.114:0/2203823746 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f94441a5040 con 0x7f944419a030 2024-04-23T10:41:45.456 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:41:45.452+0000 7f94437fe640 1 -- 172.21.15.114:0/2203823746 <== mon.0 v2:172.21.15.114:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f9434003d10 con 0x7f944419a030 2024-04-23T10:41:45.456 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:41:45.452+0000 7f94437fe640 1 -- 172.21.15.114:0/2203823746 <== mon.0 v2:172.21.15.114:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f943402fa20 con 0x7f944419a030 2024-04-23T10:41:45.456 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:41:45.452+0000 7f944c474640 1 -- 172.21.15.114:0/2203823746 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f94441a55e0 con 0x7f944419a030 2024-04-23T10:41:45.456 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:41:45.453+0000 7f944c474640 1 -- 172.21.15.114:0/2203823746 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f9444102560 con 0x7f944419a030 2024-04-23T10:41:45.457 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:41:45.454+0000 7f94437fe640 1 -- 172.21.15.114:0/2203823746 <== mon.0 v2:172.21.15.114:3300/0 4 ==== mgrmap(e 26) ==== 138778+0+0 (secure 0 0 0) 0x7f943402fbe0 con 0x7f944419a030 2024-04-23T10:41:45.457 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:41:45.454+0000 7f94437fe640 1 --2- 172.21.15.114:0/2203823746 >> [v2:172.21.15.114:6800/1395693466,v1:172.21.15.114:6801/1395693466] conn(0x7f942c0a3d20 0x7f942c0a61e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T10:41:45.457 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:41:45.454+0000 7f944a210640 1 --2- 172.21.15.114:0/2203823746 >> [v2:172.21.15.114:6800/1395693466,v1:172.21.15.114:6801/1395693466] conn(0x7f942c0a3d20 0x7f942c0a61e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T10:41:45.458 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:41:45.454+0000 7f94437fe640 1 -- 172.21.15.114:0/2203823746 <== mon.0 v2:172.21.15.114:3300/0 5 ==== osd_map(113..113 src has 1..113) ==== 8629+0+0 (secure 0 0 0) 0x7f94340f1ed0 con 0x7f944419a030 2024-04-23T10:41:45.458 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:41:45.455+0000 7f944a210640 1 --2- 172.21.15.114:0/2203823746 >> [v2:172.21.15.114:6800/1395693466,v1:172.21.15.114:6801/1395693466] conn(0x7f942c0a3d20 0x7f942c0a61e0 secure :-1 s=READY pgs=180 cs=0 l=1 rev1=1 crypto rx=0x7f943c009b60 tx=0x7f943c005f90 comp rx=0 tx=0).ready entity=mgr.15291 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T10:41:45.458 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:41:45.458+0000 7f94437fe640 1 -- 172.21.15.114:0/2203823746 <== mon.0 v2:172.21.15.114:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f94340be610 con 0x7f944419a030 2024-04-23T10:41:45.637 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:41:45.638+0000 7f944c474640 1 -- 172.21.15.114:0/2203823746 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) -- 0x7f94441093f0 con 0x7f944419a030 2024-04-23T10:41:45.637 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:41:45.638+0000 7f94437fe640 1 -- 172.21.15.114:0/2203823746 <== mon.0 v2:172.21.15.114:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v113) ==== 104+0+11 (secure 0 0 0) 0x7f94340bdf10 con 0x7f944419a030 2024-04-23T10:41:45.637 INFO:teuthology.orchestra.run.smithi114.stdout:pg_num: 32 2024-04-23T10:41:45.639 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:41:45.640+0000 7f944c474640 1 -- 172.21.15.114:0/2203823746 >> [v2:172.21.15.114:6800/1395693466,v1:172.21.15.114:6801/1395693466] conn(0x7f942c0a3d20 msgr2=0x7f942c0a61e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T10:41:45.639 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:41:45.640+0000 7f944c474640 1 --2- 172.21.15.114:0/2203823746 >> [v2:172.21.15.114:6800/1395693466,v1:172.21.15.114:6801/1395693466] conn(0x7f942c0a3d20 0x7f942c0a61e0 secure :-1 s=READY pgs=180 cs=0 l=1 rev1=1 crypto rx=0x7f943c009b60 tx=0x7f943c005f90 comp rx=0 tx=0).stop 2024-04-23T10:41:45.640 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:41:45.640+0000 7f944c474640 1 -- 172.21.15.114:0/2203823746 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f944419a030 msgr2=0x7f94441a4630 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T10:41:45.640 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:41:45.640+0000 7f944c474640 1 --2- 172.21.15.114:0/2203823746 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f944419a030 0x7f94441a4630 secure :-1 s=READY pgs=968 cs=0 l=1 rev1=1 crypto rx=0x7f94340098d0 tx=0x7f94340317b0 comp rx=0 tx=0).stop 2024-04-23T10:41:45.640 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:41:45.640+0000 7f944c474640 1 -- 172.21.15.114:0/2203823746 shutdown_connections 2024-04-23T10:41:45.640 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:41:45.640+0000 7f944c474640 1 --2- 172.21.15.114:0/2203823746 >> [v2:172.21.15.114:6800/1395693466,v1:172.21.15.114:6801/1395693466] conn(0x7f942c0a3d20 0x7f942c0a61e0 unknown :-1 s=CLOSED pgs=180 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:41:45.640 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:41:45.640+0000 7f944c474640 1 --2- 172.21.15.114:0/2203823746 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f944419a030 0x7f94441a4630 unknown :-1 s=CLOSED pgs=968 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:41:45.640 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:41:45.640+0000 7f944c474640 1 --2- 172.21.15.114:0/2203823746 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f9444106160 0x7f9444199af0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:41:45.640 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:41:45.640+0000 7f944c474640 1 --2- 172.21.15.114:0/2203823746 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f9444101090 0x7f94441995b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:41:45.640 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:41:45.640+0000 7f944c474640 1 -- 172.21.15.114:0/2203823746 >> 172.21.15.114:0/2203823746 conn(0x7f94440771d0 msgr2=0x7f94440fed60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T10:41:45.641 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:41:45.640+0000 7f944c474640 1 -- 172.21.15.114:0/2203823746 shutdown_connections 2024-04-23T10:41:45.641 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:41:45.640+0000 7f944c474640 1 -- 172.21.15.114:0/2203823746 wait complete. 2024-04-23T10:41:45.649 INFO:journalctl@ceph.mon.a.smithi114.stdout:Apr 23 10:41:45 smithi114 ceph-mon[29255]: pgmap v478: 97 pgs: 97 active+clean; 551 MiB data, 4.3 GiB used, 1.0 TiB / 1.0 TiB avail; 316 KiB/s rd, 849 KiB/s wr, 44 op/s 2024-04-23T10:41:45.649 INFO:journalctl@ceph.mon.a.smithi114.stdout:Apr 23 10:41:45 smithi114 ceph-mon[29255]: from='client.? 172.21.15.114:0/1172586742' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-23T10:41:45.649 INFO:journalctl@ceph.mon.a.smithi114.stdout:Apr 23 10:41:45 smithi114 ceph-mon[29255]: from='client.? 172.21.15.114:0/794934853' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-04-23T10:41:45.650 DEBUG:teuthology.orchestra.run.smithi114:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2024-04-23T10:41:45.753 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:41:45.751+0000 7f2ebbf76640 1 -- 172.21.15.114:0/3677303902 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f2eb406a570 msgr2=0x7f2eb406a9f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T10:41:45.753 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:41:45.751+0000 7f2ebbf76640 1 --2- 172.21.15.114:0/3677303902 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f2eb406a570 0x7f2eb406a9f0 secure :-1 s=READY pgs=969 cs=0 l=1 rev1=1 crypto rx=0x7f2ea4009930 tx=0x7f2ea402a420 comp rx=0 tx=0).stop 2024-04-23T10:41:45.753 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:41:45.751+0000 7f2ebbf76640 1 -- 172.21.15.114:0/3677303902 shutdown_connections 2024-04-23T10:41:45.753 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:41:45.751+0000 7f2ebbf76640 1 --2- 172.21.15.114:0/3677303902 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f2eb406af30 0x7f2eb410ee60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:41:45.753 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:41:45.751+0000 7f2ebbf76640 1 --2- 172.21.15.114:0/3677303902 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f2eb406a570 0x7f2eb406a9f0 unknown :-1 s=CLOSED pgs=969 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:41:45.753 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:41:45.751+0000 7f2ebbf76640 1 --2- 172.21.15.114:0/3677303902 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f2eb4106f90 0x7f2eb4107370 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:41:45.753 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:41:45.751+0000 7f2ebbf76640 1 -- 172.21.15.114:0/3677303902 >> 172.21.15.114:0/3677303902 conn(0x7f2eb4077e30 msgr2=0x7f2eb4078240 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T10:41:45.754 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:41:45.751+0000 7f2ebbf76640 1 -- 172.21.15.114:0/3677303902 shutdown_connections 2024-04-23T10:41:45.754 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:41:45.751+0000 7f2ebbf76640 1 -- 172.21.15.114:0/3677303902 wait complete. 2024-04-23T10:41:45.754 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:41:45.752+0000 7f2ebbf76640 1 Processor -- start 2024-04-23T10:41:45.754 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:41:45.752+0000 7f2ebbf76640 1 -- start start 2024-04-23T10:41:45.754 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:41:45.752+0000 7f2ebbf76640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f2eb406a570 0x7f2eb419e410 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T10:41:45.754 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:41:45.752+0000 7f2ebbf76640 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f2eb406af30 0x7f2eb419e950 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T10:41:45.754 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:41:45.752+0000 7f2ebbf76640 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f2eb4106f90 0x7f2eb41985c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T10:41:45.754 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:41:45.752+0000 7f2ebbf76640 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x7f2eb41122c0 con 0x7f2eb406a570 2024-04-23T10:41:45.755 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:41:45.752+0000 7f2ebbf76640 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 -- 0x7f2eb4112140 con 0x7f2eb4106f90 2024-04-23T10:41:45.755 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:41:45.752+0000 7f2ebbf76640 1 -- --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_getmap magic: 0 -- 0x7f2eb4112440 con 0x7f2eb406af30 2024-04-23T10:41:45.755 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:41:45.752+0000 7f2eb9d12640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f2eb406a570 0x7f2eb419e410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T10:41:45.755 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:41:45.752+0000 7f2eb9d12640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f2eb406a570 0x7f2eb419e410 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.114:53860/0 (socket says 172.21.15.114:53860) 2024-04-23T10:41:45.755 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:41:45.752+0000 7f2eb9d12640 1 -- 172.21.15.114:0/3414665942 learned_addr learned my addr 172.21.15.114:0/3414665942 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-04-23T10:41:45.755 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:41:45.752+0000 7f2eb9511640 1 --2- 172.21.15.114:0/3414665942 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f2eb406af30 0x7f2eb419e950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T10:41:45.755 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:41:45.752+0000 7f2eba513640 1 --2- 172.21.15.114:0/3414665942 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f2eb4106f90 0x7f2eb41985c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T10:41:45.755 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:41:45.753+0000 7f2eb9d12640 1 -- 172.21.15.114:0/3414665942 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f2eb406af30 msgr2=0x7f2eb419e950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T10:41:45.755 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:41:45.753+0000 7f2eb9d12640 1 --2- 172.21.15.114:0/3414665942 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f2eb406af30 0x7f2eb419e950 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:41:45.756 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:41:45.753+0000 7f2eb9d12640 1 -- 172.21.15.114:0/3414665942 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f2eb4106f90 msgr2=0x7f2eb41985c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T10:41:45.756 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:41:45.753+0000 7f2eb9d12640 1 --2- 172.21.15.114:0/3414665942 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f2eb4106f90 0x7f2eb41985c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:41:45.756 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:41:45.753+0000 7f2eb9d12640 1 -- 172.21.15.114:0/3414665942 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f2eb4198d40 con 0x7f2eb406a570 2024-04-23T10:41:45.756 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:41:45.753+0000 7f2eb9511640 1 --2- 172.21.15.114:0/3414665942 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f2eb406af30 0x7f2eb419e950 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-23T10:41:45.756 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:41:45.753+0000 7f2eba513640 1 --2- 172.21.15.114:0/3414665942 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f2eb4106f90 0x7f2eb41985c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-23T10:41:45.756 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:41:45.753+0000 7f2eb9d12640 1 --2- 172.21.15.114:0/3414665942 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f2eb406a570 0x7f2eb419e410 secure :-1 s=READY pgs=970 cs=0 l=1 rev1=1 crypto rx=0x7f2eb000d950 tx=0x7f2eb000de10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T10:41:45.756 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:41:45.753+0000 7f2eaaffd640 1 -- 172.21.15.114:0/3414665942 <== mon.0 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2eb0017070 con 0x7f2eb406a570 2024-04-23T10:41:45.756 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:41:45.753+0000 7f2eaaffd640 1 -- 172.21.15.114:0/3414665942 <== mon.0 v2:172.21.15.114:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f2eb0009c30 con 0x7f2eb406a570 2024-04-23T10:41:45.756 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:41:45.753+0000 7f2ebbf76640 1 -- 172.21.15.114:0/3414665942 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f2eb4199030 con 0x7f2eb406a570 2024-04-23T10:41:45.757 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:41:45.753+0000 7f2eaaffd640 1 -- 172.21.15.114:0/3414665942 <== mon.0 v2:172.21.15.114:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2eb000f620 con 0x7f2eb406a570 2024-04-23T10:41:45.757 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:41:45.753+0000 7f2ebbf76640 1 -- 172.21.15.114:0/3414665942 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f2eb41a52d0 con 0x7f2eb406a570 2024-04-23T10:41:45.757 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:41:45.754+0000 7f2ebbf76640 1 -- 172.21.15.114:0/3414665942 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f2eb40691c0 con 0x7f2eb406a570 2024-04-23T10:41:45.758 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:41:45.755+0000 7f2eaaffd640 1 -- 172.21.15.114:0/3414665942 <== mon.0 v2:172.21.15.114:3300/0 4 ==== mgrmap(e 26) ==== 138778+0+0 (secure 0 0 0) 0x7f2eb0010460 con 0x7f2eb406a570 2024-04-23T10:41:45.758 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:41:45.756+0000 7f2eaaffd640 1 --2- 172.21.15.114:0/3414665942 >> [v2:172.21.15.114:6800/1395693466,v1:172.21.15.114:6801/1395693466] conn(0x7f2ea00a3d20 0x7f2ea00a61e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T10:41:45.758 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:41:45.756+0000 7f2eb9511640 1 --2- 172.21.15.114:0/3414665942 >> [v2:172.21.15.114:6800/1395693466,v1:172.21.15.114:6801/1395693466] conn(0x7f2ea00a3d20 0x7f2ea00a61e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T10:41:45.758 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:41:45.756+0000 7f2eaaffd640 1 -- 172.21.15.114:0/3414665942 <== mon.0 v2:172.21.15.114:3300/0 5 ==== osd_map(113..113 src has 1..113) ==== 8629+0+0 (secure 0 0 0) 0x7f2eb00cfeb0 con 0x7f2eb406a570 2024-04-23T10:41:45.758 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:41:45.756+0000 7f2eb9511640 1 --2- 172.21.15.114:0/3414665942 >> [v2:172.21.15.114:6800/1395693466,v1:172.21.15.114:6801/1395693466] conn(0x7f2ea00a3d20 0x7f2ea00a61e0 secure :-1 s=READY pgs=181 cs=0 l=1 rev1=1 crypto rx=0x7f2ea4006010 tx=0x7f2ea4000bc0 comp rx=0 tx=0).ready entity=mgr.15291 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T10:41:45.759 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:41:45.760+0000 7f2eaaffd640 1 -- 172.21.15.114:0/3414665942 <== mon.0 v2:172.21.15.114:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f2eb009c4e0 con 0x7f2eb406a570 2024-04-23T10:41:45.827 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 23 10:41:45 smithi204 ceph-mon[32649]: pgmap v478: 97 pgs: 97 active+clean; 551 MiB data, 4.3 GiB used, 1.0 TiB / 1.0 TiB avail; 316 KiB/s rd, 849 KiB/s wr, 44 op/s 2024-04-23T10:41:45.828 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 23 10:41:45 smithi204 ceph-mon[32649]: from='client.? 172.21.15.114:0/1172586742' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-23T10:41:45.828 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 23 10:41:45 smithi204 ceph-mon[32649]: from='client.? 172.21.15.114:0/794934853' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-04-23T10:41:45.937 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:41:45.938+0000 7f2ebbf76640 1 -- 172.21.15.114:0/3414665942 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) -- 0x7f2eb4109f90 con 0x7f2eb406a570 2024-04-23T10:41:45.937 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:41:45.938+0000 7f2eaaffd640 1 -- 172.21.15.114:0/3414665942 <== mon.0 v2:172.21.15.114:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v113) ==== 100+0+11 (secure 0 0 0) 0x7f2eb0009de0 con 0x7f2eb406a570 2024-04-23T10:41:45.937 INFO:teuthology.orchestra.run.smithi114.stdout:pg_num: 64 2024-04-23T10:41:45.939 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:41:45.940+0000 7f2ebbf76640 1 -- 172.21.15.114:0/3414665942 >> [v2:172.21.15.114:6800/1395693466,v1:172.21.15.114:6801/1395693466] conn(0x7f2ea00a3d20 msgr2=0x7f2ea00a61e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T10:41:45.940 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:41:45.940+0000 7f2ebbf76640 1 --2- 172.21.15.114:0/3414665942 >> [v2:172.21.15.114:6800/1395693466,v1:172.21.15.114:6801/1395693466] conn(0x7f2ea00a3d20 0x7f2ea00a61e0 secure :-1 s=READY pgs=181 cs=0 l=1 rev1=1 crypto rx=0x7f2ea4006010 tx=0x7f2ea4000bc0 comp rx=0 tx=0).stop 2024-04-23T10:41:45.941 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:41:45.940+0000 7f2ebbf76640 1 -- 172.21.15.114:0/3414665942 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f2eb406a570 msgr2=0x7f2eb419e410 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T10:41:45.941 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:41:45.940+0000 7f2ebbf76640 1 --2- 172.21.15.114:0/3414665942 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f2eb406a570 0x7f2eb419e410 secure :-1 s=READY pgs=970 cs=0 l=1 rev1=1 crypto rx=0x7f2eb000d950 tx=0x7f2eb000de10 comp rx=0 tx=0).stop 2024-04-23T10:41:45.942 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:41:45.940+0000 7f2ebbf76640 1 -- 172.21.15.114:0/3414665942 shutdown_connections 2024-04-23T10:41:45.942 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:41:45.940+0000 7f2ebbf76640 1 --2- 172.21.15.114:0/3414665942 >> [v2:172.21.15.114:6800/1395693466,v1:172.21.15.114:6801/1395693466] conn(0x7f2ea00a3d20 0x7f2ea00a61e0 unknown :-1 s=CLOSED pgs=181 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:41:45.942 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:41:45.940+0000 7f2ebbf76640 1 --2- 172.21.15.114:0/3414665942 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f2eb4106f90 0x7f2eb41985c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:41:45.942 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:41:45.940+0000 7f2ebbf76640 1 --2- 172.21.15.114:0/3414665942 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f2eb406af30 0x7f2eb419e950 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:41:45.942 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:41:45.940+0000 7f2ebbf76640 1 --2- 172.21.15.114:0/3414665942 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f2eb406a570 0x7f2eb419e410 unknown :-1 s=CLOSED pgs=970 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:41:45.942 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:41:45.940+0000 7f2ebbf76640 1 -- 172.21.15.114:0/3414665942 >> 172.21.15.114:0/3414665942 conn(0x7f2eb4077e30 msgr2=0x7f2eb40fedf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T10:41:45.942 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:41:45.940+0000 7f2ebbf76640 1 -- 172.21.15.114:0/3414665942 shutdown_connections 2024-04-23T10:41:45.942 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:41:45.940+0000 7f2ebbf76640 1 -- 172.21.15.114:0/3414665942 wait complete. 2024-04-23T10:41:45.951 DEBUG:teuthology.orchestra.run.smithi114:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd blocklist ls 2024-04-23T10:41:45.992 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 23 10:41:45 smithi119 ceph-mon[32608]: pgmap v478: 97 pgs: 97 active+clean; 551 MiB data, 4.3 GiB used, 1.0 TiB / 1.0 TiB avail; 316 KiB/s rd, 849 KiB/s wr, 44 op/s 2024-04-23T10:41:45.992 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 23 10:41:45 smithi119 ceph-mon[32608]: from='client.? 172.21.15.114:0/1172586742' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-23T10:41:45.992 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 23 10:41:45 smithi119 ceph-mon[32608]: from='client.? 172.21.15.114:0/794934853' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-04-23T10:41:46.051 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:41:46.050+0000 7f95670a7640 1 -- 172.21.15.114:0/1212762087 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f9560069d00 msgr2=0x7f9560106b60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T10:41:46.051 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:41:46.050+0000 7f95670a7640 1 --2- 172.21.15.114:0/1212762087 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f9560069d00 0x7f9560106b60 secure :-1 s=READY pgs=971 cs=0 l=1 rev1=1 crypto rx=0x7f954c009900 tx=0x7f954c02a410 comp rx=0 tx=0).stop 2024-04-23T10:41:46.052 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:41:46.050+0000 7f95670a7640 1 -- 172.21.15.114:0/1212762087 shutdown_connections 2024-04-23T10:41:46.052 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:41:46.050+0000 7f95670a7640 1 --2- 172.21.15.114:0/1212762087 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f9560107290 0x7f956010ee30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:41:46.052 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:41:46.050+0000 7f95670a7640 1 --2- 172.21.15.114:0/1212762087 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f9560069d00 0x7f9560106b60 unknown :-1 s=CLOSED pgs=971 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:41:46.052 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:41:46.050+0000 7f95670a7640 1 --2- 172.21.15.114:0/1212762087 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f9560069350 0x7f9560069730 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:41:46.052 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:41:46.050+0000 7f95670a7640 1 -- 172.21.15.114:0/1212762087 >> 172.21.15.114:0/1212762087 conn(0x7f9560077cb0 msgr2=0x7f95600780c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T10:41:46.052 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:41:46.050+0000 7f95670a7640 1 -- 172.21.15.114:0/1212762087 shutdown_connections 2024-04-23T10:41:46.052 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:41:46.050+0000 7f95670a7640 1 -- 172.21.15.114:0/1212762087 wait complete. 2024-04-23T10:41:46.052 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:41:46.051+0000 7f95670a7640 1 Processor -- start 2024-04-23T10:41:46.052 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:41:46.051+0000 7f95670a7640 1 -- start start 2024-04-23T10:41:46.053 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:41:46.051+0000 7f95670a7640 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f9560069350 0x7f9560198750 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T10:41:46.053 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:41:46.051+0000 7f95670a7640 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f9560107290 0x7f9560198c90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T10:41:46.053 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:41:46.051+0000 7f95670a7640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f956019d010 0x7f956019d4c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T10:41:46.053 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:41:46.051+0000 7f95670a7640 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x7f9560112190 con 0x7f956019d010 2024-04-23T10:41:46.053 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:41:46.051+0000 7f95670a7640 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 -- 0x7f9560112010 con 0x7f9560069350 2024-04-23T10:41:46.053 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:41:46.051+0000 7f95670a7640 1 -- --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_getmap magic: 0 -- 0x7f9560112310 con 0x7f9560107290 2024-04-23T10:41:46.053 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:41:46.051+0000 7f9565644640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f956019d010 0x7f956019d4c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T10:41:46.053 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:41:46.051+0000 7f9565644640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f956019d010 0x7f956019d4c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.114:53876/0 (socket says 172.21.15.114:53876) 2024-04-23T10:41:46.053 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:41:46.051+0000 7f9565644640 1 -- 172.21.15.114:0/3684071134 learned_addr learned my addr 172.21.15.114:0/3684071134 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-04-23T10:41:46.054 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:41:46.051+0000 7f955ffff640 1 --2- 172.21.15.114:0/3684071134 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f9560107290 0x7f9560198c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T10:41:46.054 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:41:46.051+0000 7f9564e43640 1 --2- 172.21.15.114:0/3684071134 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f9560069350 0x7f9560198750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T10:41:46.054 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:41:46.052+0000 7f9565644640 1 -- 172.21.15.114:0/3684071134 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f9560107290 msgr2=0x7f9560198c90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T10:41:46.054 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:41:46.052+0000 7f9565644640 1 --2- 172.21.15.114:0/3684071134 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f9560107290 0x7f9560198c90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:41:46.054 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:41:46.052+0000 7f9565644640 1 -- 172.21.15.114:0/3684071134 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f9560069350 msgr2=0x7f9560198750 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T10:41:46.054 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:41:46.052+0000 7f9565644640 1 --2- 172.21.15.114:0/3684071134 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f9560069350 0x7f9560198750 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:41:46.054 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:41:46.052+0000 7f9565644640 1 -- 172.21.15.114:0/3684071134 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f956019db00 con 0x7f956019d010 2024-04-23T10:41:46.054 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:41:46.052+0000 7f9565644640 1 --2- 172.21.15.114:0/3684071134 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f956019d010 0x7f956019d4c0 secure :-1 s=READY pgs=972 cs=0 l=1 rev1=1 crypto rx=0x7f955800cad0 tx=0x7f955800cf90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T10:41:46.055 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:41:46.052+0000 7f955dffb640 1 -- 172.21.15.114:0/3684071134 <== mon.0 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f955800ae20 con 0x7f956019d010 2024-04-23T10:41:46.055 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:41:46.052+0000 7f955dffb640 1 -- 172.21.15.114:0/3684071134 <== mon.0 v2:172.21.15.114:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f9558005020 con 0x7f956019d010 2024-04-23T10:41:46.055 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:41:46.052+0000 7f95670a7640 1 -- 172.21.15.114:0/3684071134 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f956019ddf0 con 0x7f956019d010 2024-04-23T10:41:46.055 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:41:46.052+0000 7f955dffb640 1 -- 172.21.15.114:0/3684071134 <== mon.0 v2:172.21.15.114:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f955800f6b0 con 0x7f956019d010 2024-04-23T10:41:46.055 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:41:46.052+0000 7f95670a7640 1 -- 172.21.15.114:0/3684071134 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f95601a55b0 con 0x7f956019d010 2024-04-23T10:41:46.056 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:41:46.053+0000 7f95670a7640 1 -- 172.21.15.114:0/3684071134 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f9560068d70 con 0x7f956019d010 2024-04-23T10:41:46.056 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:41:46.054+0000 7f955dffb640 1 -- 172.21.15.114:0/3684071134 <== mon.0 v2:172.21.15.114:3300/0 4 ==== mgrmap(e 26) ==== 138778+0+0 (secure 0 0 0) 0x7f95580051d0 con 0x7f956019d010 2024-04-23T10:41:46.056 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:41:46.055+0000 7f955dffb640 1 --2- 172.21.15.114:0/3684071134 >> [v2:172.21.15.114:6800/1395693466,v1:172.21.15.114:6801/1395693466] conn(0x7f95500a3d20 0x7f95500a61e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T10:41:46.056 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:41:46.055+0000 7f9564e43640 1 --2- 172.21.15.114:0/3684071134 >> [v2:172.21.15.114:6800/1395693466,v1:172.21.15.114:6801/1395693466] conn(0x7f95500a3d20 0x7f95500a61e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T10:41:46.056 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:41:46.055+0000 7f955dffb640 1 -- 172.21.15.114:0/3684071134 <== mon.0 v2:172.21.15.114:3300/0 5 ==== osd_map(113..113 src has 1..113) ==== 8629+0+0 (secure 0 0 0) 0x7f95580d07a0 con 0x7f956019d010 2024-04-23T10:41:46.056 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:41:46.055+0000 7f9564e43640 1 --2- 172.21.15.114:0/3684071134 >> [v2:172.21.15.114:6800/1395693466,v1:172.21.15.114:6801/1395693466] conn(0x7f95500a3d20 0x7f95500a61e0 secure :-1 s=READY pgs=182 cs=0 l=1 rev1=1 crypto rx=0x7f955400bbe0 tx=0x7f955400b390 comp rx=0 tx=0).ready entity=mgr.15291 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T10:41:46.058 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:41:46.059+0000 7f955dffb640 1 -- 172.21.15.114:0/3684071134 <== mon.0 v2:172.21.15.114:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f9558014030 con 0x7f956019d010 2024-04-23T10:41:46.247 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:41:46.248+0000 7f95670a7640 1 -- 172.21.15.114:0/3684071134 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_command({"prefix": "osd blocklist ls"} v 0) -- 0x7f9560069d00 con 0x7f956019d010 2024-04-23T10:41:46.248 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:41:46.248+0000 7f955dffb640 1 -- 172.21.15.114:0/3684071134 <== mon.0 v2:172.21.15.114:3300/0 7 ==== mon_command_ack([{"prefix": "osd blocklist ls"}]=0 listed 16 entries v113) ==== 81+0+958 (secure 0 0 0) 0x7f955809c780 con 0x7f956019d010 2024-04-23T10:41:46.248 INFO:teuthology.orchestra.run.smithi114.stdout:172.21.15.114:0/480720790 2024-04-24T10:26:40.272662+0000 2024-04-23T10:41:46.248 INFO:teuthology.orchestra.run.smithi114.stdout:172.21.15.114:6801/465333808 2024-04-24T10:26:40.272662+0000 2024-04-23T10:41:46.248 INFO:teuthology.orchestra.run.smithi114.stdout:172.21.15.114:6800/465333808 2024-04-24T10:26:40.272662+0000 2024-04-23T10:41:46.249 INFO:teuthology.orchestra.run.smithi114.stdout:172.21.15.114:0/1787801099 2024-04-24T10:26:40.272662+0000 2024-04-23T10:41:46.249 INFO:teuthology.orchestra.run.smithi114.stdout:172.21.15.114:6801/1604678557 2024-04-24T10:17:45.874609+0000 2024-04-23T10:41:46.249 INFO:teuthology.orchestra.run.smithi114.stdout:172.21.15.114:0/4025385854 2024-04-24T10:17:45.874609+0000 2024-04-23T10:41:46.249 INFO:teuthology.orchestra.run.smithi114.stdout:172.21.15.114:0/2228273469 2024-04-24T10:17:19.910348+0000 2024-04-23T10:41:46.249 INFO:teuthology.orchestra.run.smithi114.stdout:172.21.15.114:0/2713201622 2024-04-24T10:26:40.272662+0000 2024-04-23T10:41:46.249 INFO:teuthology.orchestra.run.smithi114.stdout:172.21.15.114:6800/2831530838 2024-04-24T10:17:19.910348+0000 2024-04-23T10:41:46.249 INFO:teuthology.orchestra.run.smithi114.stdout:172.21.15.114:0/637004095 2024-04-24T10:26:40.272662+0000 2024-04-23T10:41:46.249 INFO:teuthology.orchestra.run.smithi114.stdout:172.21.15.114:0/1333447450 2024-04-24T10:17:45.874609+0000 2024-04-23T10:41:46.250 INFO:teuthology.orchestra.run.smithi114.stdout:172.21.15.114:0/2233026840 2024-04-24T10:17:19.910348+0000 2024-04-23T10:41:46.250 INFO:teuthology.orchestra.run.smithi114.stdout:172.21.15.114:0/1902281545 2024-04-24T10:17:19.910348+0000 2024-04-23T10:41:46.250 INFO:teuthology.orchestra.run.smithi114.stdout:172.21.15.114:6801/2831530838 2024-04-24T10:17:19.910348+0000 2024-04-23T10:41:46.250 INFO:teuthology.orchestra.run.smithi114.stdout:172.21.15.114:6800/1604678557 2024-04-24T10:17:45.874609+0000 2024-04-23T10:41:46.250 INFO:teuthology.orchestra.run.smithi114.stdout:172.21.15.114:0/1623553811 2024-04-24T10:17:45.874609+0000 2024-04-23T10:41:46.251 INFO:teuthology.orchestra.run.smithi114.stderr:listed 16 entries 2024-04-23T10:41:46.251 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:41:46.250+0000 7f95670a7640 1 -- 172.21.15.114:0/3684071134 >> [v2:172.21.15.114:6800/1395693466,v1:172.21.15.114:6801/1395693466] conn(0x7f95500a3d20 msgr2=0x7f95500a61e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T10:41:46.251 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:41:46.250+0000 7f95670a7640 1 --2- 172.21.15.114:0/3684071134 >> [v2:172.21.15.114:6800/1395693466,v1:172.21.15.114:6801/1395693466] conn(0x7f95500a3d20 0x7f95500a61e0 secure :-1 s=READY pgs=182 cs=0 l=1 rev1=1 crypto rx=0x7f955400bbe0 tx=0x7f955400b390 comp rx=0 tx=0).stop 2024-04-23T10:41:46.251 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:41:46.250+0000 7f95670a7640 1 -- 172.21.15.114:0/3684071134 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f956019d010 msgr2=0x7f956019d4c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T10:41:46.252 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:41:46.250+0000 7f95670a7640 1 --2- 172.21.15.114:0/3684071134 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f956019d010 0x7f956019d4c0 secure :-1 s=READY pgs=972 cs=0 l=1 rev1=1 crypto rx=0x7f955800cad0 tx=0x7f955800cf90 comp rx=0 tx=0).stop 2024-04-23T10:41:46.252 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:41:46.250+0000 7f95670a7640 1 -- 172.21.15.114:0/3684071134 shutdown_connections 2024-04-23T10:41:46.252 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:41:46.250+0000 7f95670a7640 1 --2- 172.21.15.114:0/3684071134 >> [v2:172.21.15.114:6800/1395693466,v1:172.21.15.114:6801/1395693466] conn(0x7f95500a3d20 0x7f95500a61e0 unknown :-1 s=CLOSED pgs=182 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:41:46.252 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:41:46.250+0000 7f95670a7640 1 --2- 172.21.15.114:0/3684071134 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f956019d010 0x7f956019d4c0 unknown :-1 s=CLOSED pgs=972 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:41:46.252 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:41:46.250+0000 7f95670a7640 1 --2- 172.21.15.114:0/3684071134 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f9560107290 0x7f9560198c90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:41:46.252 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:41:46.250+0000 7f95670a7640 1 --2- 172.21.15.114:0/3684071134 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f9560069350 0x7f9560198750 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:41:46.252 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:41:46.250+0000 7f95670a7640 1 -- 172.21.15.114:0/3684071134 >> 172.21.15.114:0/3684071134 conn(0x7f9560077cb0 msgr2=0x7f95600ff610 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T10:41:46.252 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:41:46.250+0000 7f95670a7640 1 -- 172.21.15.114:0/3684071134 shutdown_connections 2024-04-23T10:41:46.253 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:41:46.250+0000 7f95670a7640 1 -- 172.21.15.114:0/3684071134 wait complete. 2024-04-23T10:41:46.260 DEBUG:tasks.cephfs.kernel_mount:Unmounting client client.0... 2024-04-23T10:41:46.260 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-23T10:41:46.260 DEBUG:teuthology.orchestra.run.smithi114:> sudo umount /home/ubuntu/cephtest/mnt.0 2024-04-23T10:41:46.520 INFO:tasks.cephfs.mount:Cleaning up mount ubuntu@smithi114.front.sepia.ceph.com 2024-04-23T10:41:46.521 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-23T10:41:46.521 DEBUG:teuthology.orchestra.run.smithi114:> (cd /home/ubuntu/cephtest && exec rmdir -- /home/ubuntu/cephtest/mnt.0) 2024-04-23T10:41:46.553 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-23T10:41:46.553 DEBUG:teuthology.orchestra.run.smithi114:> ip netns list 2024-04-23T10:41:46.608 INFO:teuthology.orchestra.run.smithi114.stdout:ceph-ns--home-ubuntu-cephtest-mnt.0 (id: 0) 2024-04-23T10:41:46.608 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-23T10:41:46.608 DEBUG:teuthology.orchestra.run.smithi114:> sudo ip netns delete ceph-ns--home-ubuntu-cephtest-mnt.0 2024-04-23T10:41:46.674 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-23T10:41:46.674 DEBUG:teuthology.orchestra.run.smithi114:> sudo ip link delete ceph-brx 2024-04-23T10:41:46.779 INFO:journalctl@ceph.mon.a.smithi114.stdout:Apr 23 10:41:46 smithi114 ceph-mon[29255]: from='client.? 172.21.15.114:0/2203823746' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2024-04-23T10:41:46.779 INFO:journalctl@ceph.mon.a.smithi114.stdout:Apr 23 10:41:46 smithi114 ceph-mon[29255]: from='client.? 172.21.15.114:0/3414665942' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2024-04-23T10:41:46.779 INFO:journalctl@ceph.mon.a.smithi114.stdout:Apr 23 10:41:46 smithi114 ceph-mon[29255]: from='client.? 172.21.15.114:0/3684071134' entity='client.admin' cmd=[{"prefix": "osd blocklist ls"}]: dispatch 2024-04-23T10:41:46.780 INFO:tasks.kclient:Unmounting kernel clients... 2024-04-23T10:41:46.781 DEBUG:teuthology.orchestra.run.smithi114:> set -ex 2024-04-23T10:41:46.781 DEBUG:teuthology.orchestra.run.smithi114:> dd if=/proc/self/mounts of=/dev/stdout 2024-04-23T10:41:46.797 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=4096k,nr_inodes=4056096,mode=755,inode64 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev,inode64 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,size=6497672k,nr_inodes=819200,mode=755,inode64 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,seclabel,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,nosuid,noexec,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=29,pgrp=1,timeout=0,minproto=5,maxproto=5,direct,pipe_ino=18859 0 0 mqueue /dev/mqueue mqueue rw,seclabel,nosuid,nodev,noexec,relatime 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 tracefs /sys/kernel/tracing tracefs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 none /run/credentials/systemd-sysusers.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 none /run/credentials/systemd-sysctl.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 none /run/credentials/systemd-tmpfiles-setup-dev.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 none /run/credentials/systemd-tmpfiles-setup.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3248836k,nr_inodes=812209,mode=700,uid=1000,gid=1316,inode64 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/sda1 /var/lib/containers/storage/overlay ext4 rw,seclabel,relatime 0 0 overlay /var/lib/containers/storage/overlay/7ddc13ba9abd842063f3de490ab2c038e19c63131b2005463d415879835fc4ac/merged overlay rw,context="system_u:object_r:container_file_t:s0:c1022,c1023",relatime,lowerdir=/var/lib/containers/storage/overlay/l/EHTEOGED3HEK3C4KZZWU7FSGVU:/var/lib/containers/storage/overlay/l/P7BFYHZXOECAZF6Y5XOXHT5BI2,upperdir=/var/lib/containers/storage/overlay/7ddc13ba9abd842063f3de490ab2c038e19c63131b2005463d415879835fc4ac/diff,workdir=/var/lib/containers/storage/overlay/7ddc13ba9abd842063f3de490ab2c038e19c63131b2005463d415879835fc4ac/work,metacopy=on,volatile 0 0 overlay /var/lib/containers/storage/overlay/3b945d72431548e4fa7c606123a01976d11200b368a76c76fb1f67cd1925152f/merged overlay rw,context="system_u:object_r:container_file_t:s0:c1022,c1023",nodev,relatime,lowerdir=/var/lib/containers/storage/overlay/l/EHTEOGED3HEK3C4KZZWU7FSGVU:/var/lib/containers/storage/overlay/l/P7BFYHZXOECAZF6Y5XOXHT5BI2,upperdir=/var/lib/containers/storage/overlay/3b945d72431548e4fa7c606123a01976d11200b368a76c76fb1f67cd1925152f/diff,workdir=/var/lib/containers/storage/overlay/3b945d72431548e4fa7c606123a01976d11200b368a76c76fb1f67cd1925152f/work,metacopy=on,volatile 0 0 tmpfs /run/user/0 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3248836k,nr_inodes=812209,mode=700,inode64 0 0 binfmt_misc /proc/sys/fs/binfmt_misc binfmt_misc rw,nosuid,nodev,noexec,relatime 0 0 overlay /var/lib/containers/storage/overlay/2c30893cad24e589460da9f57758b7e43f284ca0098107e462ae62a8b93e6a3f/merged overlay rw,context="system_u:object_r:container_file_t:s0:c1022,c1023",relatime,lowerdir=/var/lib/containers/storage/overlay/l/EHTEOGED3HEK3C4KZZWU7FSGVU:/var/lib/containers/storage/overlay/l/P7BFYHZXOECAZF6Y5XOXHT5BI2,upperdir=/var/lib/containers/storage/overlay/2c30893cad24e589460da9f57758b7e43f284ca0098107e462ae62a8b93e6a3f/diff,workdir=/var/lib/containers/storage/overlay/2c30893cad24e589460da9f57758b7e43f284ca0098107e462ae62a8b93e6a3f/work,metacopy=on,volatile 0 0 overlay /var/lib/containers/storage/overlay/4ee77b49ececa7082e9adac97329fab76adafa8bc9de2ddcc7f0b8a7d9b60edc/merged overlay rw,context="system_u:object_r:container_file_t:s0:c1022,c1023",relatime,lowerdir=/var/lib/containers/storage/overlay/l/EHTEOGED3HEK3C4KZZWU7FSGVU:/var/lib/containers/storage/overlay/l/P7BFYHZXOECAZF6Y5XOXHT5BI2,upperdir=/var/lib/containers/storage/overlay/4ee77b49ececa7082e9adac97329fab76adafa8bc9de2ddcc7f0b8a7d9b60edc/diff,workdir=/var/lib/containers/storage/overlay/4ee77b49ececa7082e9adac97329fab76adafa8bc9de2ddcc7f0b8a7d9b60edc/work,metacopy=on,volatile 0 0 overlay /var/lib/containers/storage/overlay/e2f348501ad90d181d53838452084b307d487d20d6394421d643bfb84c94c847/merged overlay rw,context="system_u:object_r:container_file_t:s0:c1022,c1023",relatime,lowerdir=/var/lib/containers/storage/overlay/l/EHTEOGED3HEK3C4KZZWU7FSGVU:/var/lib/containers/storage/overlay/l/P7BFYHZXOECAZF6Y5XOXHT5BI2,upperdir=/var/lib/containers/storage/overlay/e2f348501ad90d181d53838452084b307d487d20d6394421d643bfb84c94c847/diff,workdir=/var/lib/containers/storage/overlay/e2f348501ad90d181d53838452084b307d487d20d6394421d643bfb84c94c847/work,metacopy=on,volatile 0 0 overlay /var/lib/containers/storage/overlay/6478cfb6899ae517064dc1cf1664178637d15dfbbceee37faa88400d7f3e67f7/merged overlay rw,context="system_u:object_r:container_file_t:s0:c1022,c1023",relatime,lowerdir=/var/lib/containers/storage/overlay/l/EHTEOGED3HEK3C4KZZWU7FSGVU:/var/lib/containers/storage/overlay/l/P7BFYHZXOECAZF6Y5XOXHT5BI2,upperdir=/var/lib/containers/storage/overlay/6478cfb6899ae517064dc1cf1664178637d15dfbbceee37faa88400d7f3e67f7/diff,workdir=/var/lib/containers/storage/overlay/6478cfb6899ae517064dc1cf1664178637d15dfbbceee37faa88400d7f3e67f7/work,metacopy=on,volatile 0 0 overlay /var/lib/containers/storage/overlay/109f37e6b188037ac8a8bcf1b86dbf3383a0e038832394965fbe271c0ec6a5c6/merged overlay rw,context="system_u:object_r:container_file_t:s0:c1022,c1023",nodev,relatime,lowerdir=/var/lib/containers/storage/overlay/l/EHTEOGED3HEK3C4KZZWU7FSGVU:/var/lib/containers/storage/overlay/l/P7BFYHZXOECAZF6Y5XOXHT5BI2,upperdir=/var/lib/containers/storage/overlay/109f37e6b188037ac8a8bcf1b86dbf3383a0e038832394965fbe271c0ec6a5c6/diff,workdir=/var/lib/containers/storage/overlay/109f37e6b188037ac8a8bcf1b86dbf3383a0e038832394965fbe271c0ec6a5c6/work,metacopy=on,volatile 0 0 overlay /var/lib/containers/storage/overlay/ce5bb35438431992242dfeda5872d0e2a3f76a25cdd227da425d084981efb70e/merged overlay rw,context="system_u:object_r:container_file_t:s0:c1022,c1023",nodev,relatime,lowerdir=/var/lib/containers/storage/overlay/l/EHTEOGED3HEK3C4KZZWU7FSGVU:/var/lib/containers/storage/overlay/l/P7BFYHZXOECAZF6Y5XOXHT5BI2,upperdir=/var/lib/containers/storage/overlay/ce5bb35438431992242dfeda5872d0e2a3f76a25cdd227da425d084981efb70e/diff,workdir=/var/lib/containers/storage/overlay/ce5bb35438431992242dfeda5872d0e2a3f76a25cdd227da425d084981efb70e/work,metacopy=on,volatile 0 0 overlay /var/lib/containers/storage/overlay/52b4813a94e7028119e14e7a533734828b46a4f729f5cac376447e8d4493136f/merged overlay rw,context="system_u:object_r:container_file_t:s0:c1022,c1023",nodev,relatime,lowerdir=/var/lib/containers/storage/overlay/l/EHTEOGED3HEK3C4KZZWU7FSGVU:/var/lib/containers/storage/overlay/l/P7BFYHZXOECAZF6Y5XOXHT5BI2,upperdir=/var/lib/containers/storage/overlay/52b4813a94e7028119e14e7a533734828b46a4f729f5cac376447e8d4493136f/diff,workdir=/var/lib/containers/storage/overlay/52b4813a94e7028119e14e7a533734828b46a4f729f5cac376447e8d4493136f/work,metacopy=on,volatile 0 0 overlay /var/lib/containers/storage/overlay/ae06c518042d287a6b6441d2b79437a46bd519b71ba10dc8a115544d0020f774/merged overlay rw,context="system_u:object_r:container_file_t:s0:c1022,c1023",nodev,relatime,lowerdir=/var/lib/containers/storage/overlay/l/EHTEOGED3HEK3C4KZZWU7FSGVU:/var/lib/containers/storage/overlay/l/P7BFYHZXOECAZF6Y5XOXHT5BI2,upperdir=/var/lib/containers/storage/overlay/ae06c518042d287a6b6441d2b79437a46bd519b71ba10dc8a115544d0020f774/diff,workdir=/var/lib/containers/storage/overlay/ae06c518042d287a6b6441d2b79437a46bd519b71ba10dc8a115544d0020f774/work,metacopy=on,volatile 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,size=6497672k,nr_inodes=819200,mode=755,inode64 0 0 2024-04-23T10:41:46.797 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-23T10:41:46.798 DEBUG:teuthology.orchestra.run.smithi114:> ip netns list 2024-04-23T10:41:46.827 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 23 10:41:46 smithi204 ceph-mon[32649]: from='client.? 172.21.15.114:0/2203823746' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2024-04-23T10:41:46.828 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 23 10:41:46 smithi204 ceph-mon[32649]: from='client.? 172.21.15.114:0/3414665942' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2024-04-23T10:41:46.828 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 23 10:41:46 smithi204 ceph-mon[32649]: from='client.? 172.21.15.114:0/3684071134' entity='client.admin' cmd=[{"prefix": "osd blocklist ls"}]: dispatch 2024-04-23T10:41:46.853 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-23T10:41:46.853 DEBUG:teuthology.orchestra.run.smithi114:> sudo ip link delete ceph-brx 2024-04-23T10:41:46.921 INFO:teuthology.orchestra.run.smithi114.stderr:Cannot find device "ceph-brx" 2024-04-23T10:41:46.922 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-23T10:41:46.922 DEBUG:teuthology.run_tasks:Unwinding manager cephadm 2024-04-23T10:41:46.935 INFO:tasks.cephadm:Teardown begin 2024-04-23T10:41:46.935 INFO:tasks.cephfs.filesystem:Destroying file system cephfs and related pools 2024-04-23T10:41:46.935 DEBUG:teuthology.orchestra.run.smithi114:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-23T10:41:46.992 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 23 10:41:46 smithi119 ceph-mon[32608]: from='client.? 172.21.15.114:0/2203823746' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2024-04-23T10:41:46.993 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 23 10:41:46 smithi119 ceph-mon[32608]: from='client.? 172.21.15.114:0/3414665942' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2024-04-23T10:41:46.993 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 23 10:41:46 smithi119 ceph-mon[32608]: from='client.? 172.21.15.114:0/3684071134' entity='client.admin' cmd=[{"prefix": "osd blocklist ls"}]: dispatch 2024-04-23T10:41:47.065 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:41:47.062+0000 7f6bf6520640 1 -- 172.21.15.114:0/1607581873 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f6bf0103b00 msgr2=0x7f6bf010bfe0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T10:41:47.065 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:41:47.062+0000 7f6bf6520640 1 --2- 172.21.15.114:0/1607581873 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f6bf0103b00 0x7f6bf010bfe0 secure :-1 s=READY pgs=973 cs=0 l=1 rev1=1 crypto rx=0x7f6be4009900 tx=0x7f6be402a510 comp rx=0 tx=0).stop 2024-04-23T10:41:47.065 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:41:47.062+0000 7f6bf6520640 1 -- 172.21.15.114:0/1607581873 shutdown_connections 2024-04-23T10:41:47.065 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:41:47.062+0000 7f6bf6520640 1 --2- 172.21.15.114:0/1607581873 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f6bf010c520 0x7f6bf010e9c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:41:47.066 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:41:47.062+0000 7f6bf6520640 1 --2- 172.21.15.114:0/1607581873 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f6bf0103b00 0x7f6bf010bfe0 unknown :-1 s=CLOSED pgs=973 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:41:47.066 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:41:47.062+0000 7f6bf6520640 1 --2- 172.21.15.114:0/1607581873 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f6bf01031e0 0x7f6bf01035c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:41:47.066 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:41:47.062+0000 7f6bf6520640 1 -- 172.21.15.114:0/1607581873 >> 172.21.15.114:0/1607581873 conn(0x7f6bf0077f50 msgr2=0x7f6bf0078360 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T10:41:47.066 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:41:47.063+0000 7f6bf6520640 1 -- 172.21.15.114:0/1607581873 shutdown_connections 2024-04-23T10:41:47.066 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:41:47.063+0000 7f6bf6520640 1 -- 172.21.15.114:0/1607581873 wait complete. 2024-04-23T10:41:47.066 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:41:47.063+0000 7f6bf6520640 1 Processor -- start 2024-04-23T10:41:47.067 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:41:47.063+0000 7f6bf6520640 1 -- start start 2024-04-23T10:41:47.067 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:41:47.063+0000 7f6bf6520640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f6bf01031e0 0x7f6bf0198710 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T10:41:47.067 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:41:47.063+0000 7f6bf6520640 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f6bf0103b00 0x7f6bf0198c50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T10:41:47.067 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:41:47.063+0000 7f6bf6520640 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f6bf010c520 0x7f6bf019cfd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T10:41:47.067 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:41:47.063+0000 7f6bf6520640 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x7f6bf0112190 con 0x7f6bf01031e0 2024-04-23T10:41:47.067 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:41:47.063+0000 7f6bf6520640 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 -- 0x7f6bf0112010 con 0x7f6bf010c520 2024-04-23T10:41:47.067 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:41:47.063+0000 7f6bf6520640 1 -- --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_getmap magic: 0 -- 0x7f6bf0112310 con 0x7f6bf0103b00 2024-04-23T10:41:47.067 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:41:47.063+0000 7f6beffff640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f6bf01031e0 0x7f6bf0198710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T10:41:47.068 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:41:47.064+0000 7f6beffff640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f6bf01031e0 0x7f6bf0198710 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.114:53892/0 (socket says 172.21.15.114:53892) 2024-04-23T10:41:47.068 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:41:47.064+0000 7f6beffff640 1 -- 172.21.15.114:0/1174688085 learned_addr learned my addr 172.21.15.114:0/1174688085 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-04-23T10:41:47.068 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:41:47.064+0000 7f6bf4abd640 1 --2- 172.21.15.114:0/1174688085 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f6bf010c520 0x7f6bf019cfd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T10:41:47.068 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:41:47.064+0000 7f6bef7fe640 1 --2- 172.21.15.114:0/1174688085 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f6bf0103b00 0x7f6bf0198c50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T10:41:47.068 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:41:47.064+0000 7f6beffff640 1 -- 172.21.15.114:0/1174688085 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f6bf0103b00 msgr2=0x7f6bf0198c50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T10:41:47.068 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:41:47.064+0000 7f6beffff640 1 --2- 172.21.15.114:0/1174688085 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f6bf0103b00 0x7f6bf0198c50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:41:47.068 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:41:47.064+0000 7f6beffff640 1 -- 172.21.15.114:0/1174688085 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f6bf010c520 msgr2=0x7f6bf019cfd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T10:41:47.069 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:41:47.064+0000 7f6beffff640 1 --2- 172.21.15.114:0/1174688085 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f6bf010c520 0x7f6bf019cfd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:41:47.069 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:41:47.064+0000 7f6beffff640 1 -- 172.21.15.114:0/1174688085 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f6bf019d700 con 0x7f6bf01031e0 2024-04-23T10:41:47.069 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:41:47.064+0000 7f6bef7fe640 1 --2- 172.21.15.114:0/1174688085 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f6bf0103b00 0x7f6bf0198c50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-23T10:41:47.069 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:41:47.064+0000 7f6bf4abd640 1 --2- 172.21.15.114:0/1174688085 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f6bf010c520 0x7f6bf019cfd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-23T10:41:47.069 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:41:47.064+0000 7f6beffff640 1 --2- 172.21.15.114:0/1174688085 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f6bf01031e0 0x7f6bf0198710 secure :-1 s=READY pgs=974 cs=0 l=1 rev1=1 crypto rx=0x7f6be000e9f0 tx=0x7f6be000eeb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T10:41:47.069 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:41:47.064+0000 7f6bed7fa640 1 -- 172.21.15.114:0/1174688085 <== mon.0 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6be000cd80 con 0x7f6bf01031e0 2024-04-23T10:41:47.069 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:41:47.064+0000 7f6bed7fa640 1 -- 172.21.15.114:0/1174688085 <== mon.0 v2:172.21.15.114:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f6be00108e0 con 0x7f6bf01031e0 2024-04-23T10:41:47.070 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:41:47.064+0000 7f6bf6520640 1 -- 172.21.15.114:0/1174688085 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f6bf019d9f0 con 0x7f6bf01031e0 2024-04-23T10:41:47.070 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:41:47.064+0000 7f6bed7fa640 1 -- 172.21.15.114:0/1174688085 <== mon.0 v2:172.21.15.114:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6be0018980 con 0x7f6bf01031e0 2024-04-23T10:41:47.070 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:41:47.064+0000 7f6bf6520640 1 -- 172.21.15.114:0/1174688085 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f6bf01a5280 con 0x7f6bf01031e0 2024-04-23T10:41:47.070 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:41:47.065+0000 7f6bf6520640 1 -- 172.21.15.114:0/1174688085 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f6bf0068d70 con 0x7f6bf01031e0 2024-04-23T10:41:47.070 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:41:47.066+0000 7f6bed7fa640 1 -- 172.21.15.114:0/1174688085 <== mon.0 v2:172.21.15.114:3300/0 4 ==== mgrmap(e 26) ==== 138778+0+0 (secure 0 0 0) 0x7f6be0010430 con 0x7f6bf01031e0 2024-04-23T10:41:47.071 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:41:47.067+0000 7f6bed7fa640 1 --2- 172.21.15.114:0/1174688085 >> [v2:172.21.15.114:6800/1395693466,v1:172.21.15.114:6801/1395693466] conn(0x7f6bd80a3d20 0x7f6bd80a61e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T10:41:47.072 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:41:47.067+0000 7f6bef7fe640 1 --2- 172.21.15.114:0/1174688085 >> [v2:172.21.15.114:6800/1395693466,v1:172.21.15.114:6801/1395693466] conn(0x7f6bd80a3d20 0x7f6bd80a61e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T10:41:47.072 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:41:47.067+0000 7f6bed7fa640 1 -- 172.21.15.114:0/1174688085 <== mon.0 v2:172.21.15.114:3300/0 5 ==== osd_map(113..113 src has 1..113) ==== 8629+0+0 (secure 0 0 0) 0x7f6be00d0c90 con 0x7f6bf01031e0 2024-04-23T10:41:47.072 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:41:47.067+0000 7f6bef7fe640 1 --2- 172.21.15.114:0/1174688085 >> [v2:172.21.15.114:6800/1395693466,v1:172.21.15.114:6801/1395693466] conn(0x7f6bd80a3d20 0x7f6bd80a61e0 secure :-1 s=READY pgs=183 cs=0 l=1 rev1=1 crypto rx=0x7f6be4000c00 tx=0x7f6be4031400 comp rx=0 tx=0).ready entity=mgr.15291 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T10:41:47.072 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:41:47.072+0000 7f6bed7fa640 1 -- 172.21.15.114:0/1174688085 <== mon.0 v2:172.21.15.114:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f6be009d2c0 con 0x7f6bf01031e0 2024-04-23T10:41:47.285 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:41:47.286+0000 7f6bf6520640 1 -- 172.21.15.114:0/1174688085 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f6bf0199a60 con 0x7f6bf01031e0 2024-04-23T10:41:47.286 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:41:47.287+0000 7f6bed7fa640 1 -- 172.21.15.114:0/1174688085 <== mon.0 v2:172.21.15.114:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 49 v49) ==== 94+0+11891 (secure 0 0 0) 0x7f6be009cbc0 con 0x7f6bf01031e0 2024-04-23T10:41:47.286 INFO:teuthology.orchestra.run.smithi114.stdout: 2024-04-23T10:41:47.287 INFO:teuthology.orchestra.run.smithi114.stdout:{"epoch":49,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14547,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.114:6835/663493580","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6834","nonce":663493580},{"type":"v1","addr":"172.21.15.114:6835","nonce":663493580}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14565,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.114:6837/1439252154","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6836","nonce":1439252154},{"type":"v1","addr":"172.21.15.114:6837","nonce":1439252154}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5},{"gid":14634,"name":"g","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.114:6839/3849796939","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6838","nonce":3849796939},{"type":"v1","addr":"172.21.15.114:6839","nonce":3849796939}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":11},{"gid":14646,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.114:6841/3544523891","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6840","nonce":3544523891},{"type":"v1","addr":"172.21.15.114:6841","nonce":3544523891}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":12},{"gid":24385,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.204:6833/3866561754","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6832","nonce":3866561754},{"type":"v1","addr":"172.21.15.204:6833","nonce":3866561754}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2},{"gid":24427,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.204:6835/2533793172","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6834","nonce":2533793172},{"type":"v1","addr":"172.21.15.204:6835","nonce":2533793172}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":7}],"filesystems":[{"mdsmap":{"epoch":49,"flags":306,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":true},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-04-23T10:24:43.383158+0000","modified":"2024-04-23T10:40:23.446880+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":3,"in":[0,1,2],"up":{"mds_0":24545,"mds_1":24509,"mds_2":24479},"failed":[],"damaged":[],"stopped":[],"info":{"gid_24439":{"gid":24439,"name":"c","rank":2,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.204:6837/513822179","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6836","nonce":513822179},{"type":"v1","addr":"172.21.15.204:6837","nonce":513822179}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24443":{"gid":24443,"name":"k","rank":1,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.119:6833/783244660","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6832","nonce":783244660},{"type":"v1","addr":"172.21.15.119:6833","nonce":783244660}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24479":{"gid":24479,"name":"e","rank":2,"incarnation":19,"state":"up:active","state_seq":9,"addr":"172.21.15.119:6835/2033916709","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6834","nonce":2033916709},{"type":"v1","addr":"172.21.15.119:6835","nonce":2033916709}]},"join_fscid":-1,"export_targets":[0,1],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24509":{"gid":24509,"name":"h","rank":1,"incarnation":17,"state":"up:active","state_seq":6,"addr":"172.21.15.119:6837/1872363128","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6836","nonce":1872363128},{"type":"v1","addr":"172.21.15.119:6837","nonce":1872363128}]},"join_fscid":-1,"export_targets":[0],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24545":{"gid":24545,"name":"b","rank":0,"incarnation":14,"state":"up:active","state_seq":3,"addr":"172.21.15.119:6839/937297340","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6838","nonce":937297340},{"type":"v1","addr":"172.21.15.119:6839","nonce":937297340}]},"join_fscid":-1,"export_targets":[1,2],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_26224":{"gid":26224,"name":"i","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.204:6839/3251736831","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6838","nonce":3251736831},{"type":"v1","addr":"172.21.15.204:6839","nonce":3251736831}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":24545,"qdb_cluster":[24545,24479,24509]},"id":1}]} 2024-04-23T10:41:47.287 INFO:teuthology.orchestra.run.smithi114.stderr:dumped fsmap epoch 49 2024-04-23T10:41:47.288 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:41:47.289+0000 7f6bf6520640 1 -- 172.21.15.114:0/1174688085 >> [v2:172.21.15.114:6800/1395693466,v1:172.21.15.114:6801/1395693466] conn(0x7f6bd80a3d20 msgr2=0x7f6bd80a61e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T10:41:47.289 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:41:47.289+0000 7f6bf6520640 1 --2- 172.21.15.114:0/1174688085 >> [v2:172.21.15.114:6800/1395693466,v1:172.21.15.114:6801/1395693466] conn(0x7f6bd80a3d20 0x7f6bd80a61e0 secure :-1 s=READY pgs=183 cs=0 l=1 rev1=1 crypto rx=0x7f6be4000c00 tx=0x7f6be4031400 comp rx=0 tx=0).stop 2024-04-23T10:41:47.289 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:41:47.289+0000 7f6bf6520640 1 -- 172.21.15.114:0/1174688085 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f6bf01031e0 msgr2=0x7f6bf0198710 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T10:41:47.289 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:41:47.289+0000 7f6bf6520640 1 --2- 172.21.15.114:0/1174688085 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f6bf01031e0 0x7f6bf0198710 secure :-1 s=READY pgs=974 cs=0 l=1 rev1=1 crypto rx=0x7f6be000e9f0 tx=0x7f6be000eeb0 comp rx=0 tx=0).stop 2024-04-23T10:41:47.289 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:41:47.289+0000 7f6bf6520640 1 -- 172.21.15.114:0/1174688085 shutdown_connections 2024-04-23T10:41:47.289 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:41:47.289+0000 7f6bf6520640 1 --2- 172.21.15.114:0/1174688085 >> [v2:172.21.15.114:6800/1395693466,v1:172.21.15.114:6801/1395693466] conn(0x7f6bd80a3d20 0x7f6bd80a61e0 unknown :-1 s=CLOSED pgs=183 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:41:47.290 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:41:47.289+0000 7f6bf6520640 1 --2- 172.21.15.114:0/1174688085 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f6bf010c520 0x7f6bf019cfd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:41:47.290 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:41:47.289+0000 7f6bf6520640 1 --2- 172.21.15.114:0/1174688085 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f6bf0103b00 0x7f6bf0198c50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:41:47.290 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:41:47.289+0000 7f6bf6520640 1 --2- 172.21.15.114:0/1174688085 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f6bf01031e0 0x7f6bf0198710 unknown :-1 s=CLOSED pgs=974 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:41:47.290 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:41:47.289+0000 7f6bf6520640 1 -- 172.21.15.114:0/1174688085 >> 172.21.15.114:0/1174688085 conn(0x7f6bf0077f50 msgr2=0x7f6bf00fe020 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T10:41:47.290 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:41:47.289+0000 7f6bf6520640 1 -- 172.21.15.114:0/1174688085 shutdown_connections 2024-04-23T10:41:47.290 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:41:47.289+0000 7f6bf6520640 1 -- 172.21.15.114:0/1174688085 wait complete. 2024-04-23T10:41:47.300 DEBUG:teuthology.orchestra.run.smithi114:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-23T10:41:47.403 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:41:47.401+0000 7f92a2679640 1 -- 172.21.15.114:0/1893397587 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f929c1021d0 msgr2=0x7f929c1025b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T10:41:47.403 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:41:47.401+0000 7f92a2679640 1 --2- 172.21.15.114:0/1893397587 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f929c1021d0 0x7f929c1025b0 secure :-1 s=READY pgs=975 cs=0 l=1 rev1=1 crypto rx=0x7f928c009900 tx=0x7f928c02a410 comp rx=0 tx=0).stop 2024-04-23T10:41:47.403 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:41:47.401+0000 7f92a2679640 1 -- 172.21.15.114:0/1893397587 shutdown_connections 2024-04-23T10:41:47.403 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:41:47.401+0000 7f92a2679640 1 --2- 172.21.15.114:0/1893397587 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f929c1072a0 0x7f929c10ee70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:41:47.404 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:41:47.401+0000 7f92a2679640 1 --2- 172.21.15.114:0/1893397587 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f929c102b80 0x7f929c106bc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:41:47.404 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:41:47.401+0000 7f92a2679640 1 --2- 172.21.15.114:0/1893397587 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f929c1021d0 0x7f929c1025b0 unknown :-1 s=CLOSED pgs=975 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:41:47.404 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:41:47.401+0000 7f92a2679640 1 -- 172.21.15.114:0/1893397587 >> 172.21.15.114:0/1893397587 conn(0x7f929c0793e0 msgr2=0x7f929c0797f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T10:41:47.404 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:41:47.401+0000 7f92a2679640 1 -- 172.21.15.114:0/1893397587 shutdown_connections 2024-04-23T10:41:47.404 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:41:47.401+0000 7f92a2679640 1 -- 172.21.15.114:0/1893397587 wait complete. 2024-04-23T10:41:47.404 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:41:47.401+0000 7f92a2679640 1 Processor -- start 2024-04-23T10:41:47.405 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:41:47.401+0000 7f92a2679640 1 -- start start 2024-04-23T10:41:47.405 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:41:47.402+0000 7f92a2679640 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f929c1021d0 0x7f929c074250 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T10:41:47.405 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:41:47.402+0000 7f92a2679640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f929c1072a0 0x7f929c074790 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T10:41:47.405 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:41:47.402+0000 7f92a2679640 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f929c074cd0 0x7f929c1a6940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T10:41:47.405 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:41:47.402+0000 7f92a2679640 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x7f929c112210 con 0x7f929c1072a0 2024-04-23T10:41:47.405 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:41:47.402+0000 7f92a2679640 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 -- 0x7f929c112090 con 0x7f929c1021d0 2024-04-23T10:41:47.405 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:41:47.402+0000 7f92a2679640 1 -- --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_getmap magic: 0 -- 0x7f929c112390 con 0x7f929c074cd0 2024-04-23T10:41:47.406 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:41:47.402+0000 7f929b7fe640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f929c1072a0 0x7f929c074790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T10:41:47.406 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:41:47.402+0000 7f929b7fe640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f929c1072a0 0x7f929c074790 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.114:53906/0 (socket says 172.21.15.114:53906) 2024-04-23T10:41:47.406 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:41:47.402+0000 7f929b7fe640 1 -- 172.21.15.114:0/3128400310 learned_addr learned my addr 172.21.15.114:0/3128400310 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-04-23T10:41:47.406 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:41:47.402+0000 7f92a0c16640 1 --2- 172.21.15.114:0/3128400310 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f929c074cd0 0x7f929c1a6940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T10:41:47.406 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:41:47.402+0000 7f929bfff640 1 --2- 172.21.15.114:0/3128400310 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f929c1021d0 0x7f929c074250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T10:41:47.406 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:41:47.402+0000 7f929b7fe640 1 -- 172.21.15.114:0/3128400310 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f929c074cd0 msgr2=0x7f929c1a6940 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T10:41:47.407 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:41:47.402+0000 7f929b7fe640 1 --2- 172.21.15.114:0/3128400310 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f929c074cd0 0x7f929c1a6940 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:41:47.407 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:41:47.402+0000 7f929b7fe640 1 -- 172.21.15.114:0/3128400310 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f929c1021d0 msgr2=0x7f929c074250 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T10:41:47.407 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:41:47.402+0000 7f929b7fe640 1 --2- 172.21.15.114:0/3128400310 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f929c1021d0 0x7f929c074250 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:41:47.407 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:41:47.402+0000 7f929b7fe640 1 -- 172.21.15.114:0/3128400310 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f929c1a7040 con 0x7f929c1072a0 2024-04-23T10:41:47.407 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:41:47.402+0000 7f92a0c16640 1 --2- 172.21.15.114:0/3128400310 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f929c074cd0 0x7f929c1a6940 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-23T10:41:47.407 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:41:47.402+0000 7f929bfff640 1 --2- 172.21.15.114:0/3128400310 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f929c1021d0 0x7f929c074250 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-23T10:41:47.407 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:41:47.402+0000 7f929b7fe640 1 --2- 172.21.15.114:0/3128400310 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f929c1072a0 0x7f929c074790 secure :-1 s=READY pgs=976 cs=0 l=1 rev1=1 crypto rx=0x7f929000d770 tx=0x7f929000dc30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T10:41:47.408 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:41:47.402+0000 7f92997fa640 1 -- 172.21.15.114:0/3128400310 <== mon.0 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9290017070 con 0x7f929c1072a0 2024-04-23T10:41:47.408 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:41:47.402+0000 7f92997fa640 1 -- 172.21.15.114:0/3128400310 <== mon.0 v2:172.21.15.114:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f9290009c30 con 0x7f929c1072a0 2024-04-23T10:41:47.408 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:41:47.402+0000 7f92a2679640 1 -- 172.21.15.114:0/3128400310 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f929c1a7330 con 0x7f929c1072a0 2024-04-23T10:41:47.408 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:41:47.402+0000 7f92997fa640 1 -- 172.21.15.114:0/3128400310 <== mon.0 v2:172.21.15.114:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f929000f620 con 0x7f929c1072a0 2024-04-23T10:41:47.408 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:41:47.402+0000 7f92a2679640 1 -- 172.21.15.114:0/3128400310 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f929c1a7880 con 0x7f929c1072a0 2024-04-23T10:41:47.408 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:41:47.403+0000 7f92a2679640 1 -- 172.21.15.114:0/3128400310 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f929c0684e0 con 0x7f929c1072a0 2024-04-23T10:41:47.409 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:41:47.404+0000 7f92997fa640 1 -- 172.21.15.114:0/3128400310 <== mon.0 v2:172.21.15.114:3300/0 4 ==== mgrmap(e 26) ==== 138778+0+0 (secure 0 0 0) 0x7f929001a070 con 0x7f929c1072a0 2024-04-23T10:41:47.410 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:41:47.405+0000 7f92997fa640 1 --2- 172.21.15.114:0/3128400310 >> [v2:172.21.15.114:6800/1395693466,v1:172.21.15.114:6801/1395693466] conn(0x7f92840a3d50 0x7f92840a6210 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T10:41:47.410 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:41:47.405+0000 7f929bfff640 1 --2- 172.21.15.114:0/3128400310 >> [v2:172.21.15.114:6800/1395693466,v1:172.21.15.114:6801/1395693466] conn(0x7f92840a3d50 0x7f92840a6210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T10:41:47.410 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:41:47.405+0000 7f92997fa640 1 -- 172.21.15.114:0/3128400310 <== mon.0 v2:172.21.15.114:3300/0 5 ==== osd_map(113..113 src has 1..113) ==== 8629+0+0 (secure 0 0 0) 0x7f92900d0420 con 0x7f929c1072a0 2024-04-23T10:41:47.410 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:41:47.406+0000 7f929bfff640 1 --2- 172.21.15.114:0/3128400310 >> [v2:172.21.15.114:6800/1395693466,v1:172.21.15.114:6801/1395693466] conn(0x7f92840a3d50 0x7f92840a6210 secure :-1 s=READY pgs=184 cs=0 l=1 rev1=1 crypto rx=0x7f928c000c00 tx=0x7f928c031400 comp rx=0 tx=0).ready entity=mgr.15291 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T10:41:47.410 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:41:47.410+0000 7f92997fa640 1 -- 172.21.15.114:0/3128400310 <== mon.0 v2:172.21.15.114:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f929009ca50 con 0x7f929c1072a0 2024-04-23T10:41:47.622 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:41:47.623+0000 7f92a2679640 1 -- 172.21.15.114:0/3128400310 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f929c10a4d0 con 0x7f929c1072a0 2024-04-23T10:41:47.623 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:41:47.624+0000 7f92997fa640 1 -- 172.21.15.114:0/3128400310 <== mon.0 v2:172.21.15.114:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 49 v49) ==== 94+0+11891 (secure 0 0 0) 0x7f929009c350 con 0x7f929c1072a0 2024-04-23T10:41:47.623 INFO:teuthology.orchestra.run.smithi114.stdout: 2024-04-23T10:41:47.624 INFO:teuthology.orchestra.run.smithi114.stdout:{"epoch":49,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14547,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.114:6835/663493580","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6834","nonce":663493580},{"type":"v1","addr":"172.21.15.114:6835","nonce":663493580}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14565,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.114:6837/1439252154","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6836","nonce":1439252154},{"type":"v1","addr":"172.21.15.114:6837","nonce":1439252154}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5},{"gid":14634,"name":"g","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.114:6839/3849796939","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6838","nonce":3849796939},{"type":"v1","addr":"172.21.15.114:6839","nonce":3849796939}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":11},{"gid":14646,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.114:6841/3544523891","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6840","nonce":3544523891},{"type":"v1","addr":"172.21.15.114:6841","nonce":3544523891}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":12},{"gid":24385,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.204:6833/3866561754","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6832","nonce":3866561754},{"type":"v1","addr":"172.21.15.204:6833","nonce":3866561754}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2},{"gid":24427,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.204:6835/2533793172","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6834","nonce":2533793172},{"type":"v1","addr":"172.21.15.204:6835","nonce":2533793172}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":7}],"filesystems":[{"mdsmap":{"epoch":49,"flags":306,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":true},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-04-23T10:24:43.383158+0000","modified":"2024-04-23T10:40:23.446880+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":3,"in":[0,1,2],"up":{"mds_0":24545,"mds_1":24509,"mds_2":24479},"failed":[],"damaged":[],"stopped":[],"info":{"gid_24439":{"gid":24439,"name":"c","rank":2,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.204:6837/513822179","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6836","nonce":513822179},{"type":"v1","addr":"172.21.15.204:6837","nonce":513822179}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24443":{"gid":24443,"name":"k","rank":1,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.119:6833/783244660","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6832","nonce":783244660},{"type":"v1","addr":"172.21.15.119:6833","nonce":783244660}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24479":{"gid":24479,"name":"e","rank":2,"incarnation":19,"state":"up:active","state_seq":9,"addr":"172.21.15.119:6835/2033916709","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6834","nonce":2033916709},{"type":"v1","addr":"172.21.15.119:6835","nonce":2033916709}]},"join_fscid":-1,"export_targets":[0,1],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24509":{"gid":24509,"name":"h","rank":1,"incarnation":17,"state":"up:active","state_seq":6,"addr":"172.21.15.119:6837/1872363128","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6836","nonce":1872363128},{"type":"v1","addr":"172.21.15.119:6837","nonce":1872363128}]},"join_fscid":-1,"export_targets":[0],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24545":{"gid":24545,"name":"b","rank":0,"incarnation":14,"state":"up:active","state_seq":3,"addr":"172.21.15.119:6839/937297340","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6838","nonce":937297340},{"type":"v1","addr":"172.21.15.119:6839","nonce":937297340}]},"join_fscid":-1,"export_targets":[1,2],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_26224":{"gid":26224,"name":"i","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.204:6839/3251736831","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6838","nonce":3251736831},{"type":"v1","addr":"172.21.15.204:6839","nonce":3251736831}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":24545,"qdb_cluster":[24545,24479,24509]},"id":1}]} 2024-04-23T10:41:47.624 INFO:teuthology.orchestra.run.smithi114.stderr:dumped fsmap epoch 49 2024-04-23T10:41:47.625 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:41:47.626+0000 7f92a2679640 1 -- 172.21.15.114:0/3128400310 >> [v2:172.21.15.114:6800/1395693466,v1:172.21.15.114:6801/1395693466] conn(0x7f92840a3d50 msgr2=0x7f92840a6210 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T10:41:47.626 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:41:47.626+0000 7f92a2679640 1 --2- 172.21.15.114:0/3128400310 >> [v2:172.21.15.114:6800/1395693466,v1:172.21.15.114:6801/1395693466] conn(0x7f92840a3d50 0x7f92840a6210 secure :-1 s=READY pgs=184 cs=0 l=1 rev1=1 crypto rx=0x7f928c000c00 tx=0x7f928c031400 comp rx=0 tx=0).stop 2024-04-23T10:41:47.626 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:41:47.626+0000 7f92a2679640 1 -- 172.21.15.114:0/3128400310 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f929c1072a0 msgr2=0x7f929c074790 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T10:41:47.626 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:41:47.626+0000 7f92a2679640 1 --2- 172.21.15.114:0/3128400310 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f929c1072a0 0x7f929c074790 secure :-1 s=READY pgs=976 cs=0 l=1 rev1=1 crypto rx=0x7f929000d770 tx=0x7f929000dc30 comp rx=0 tx=0).stop 2024-04-23T10:41:47.626 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:41:47.626+0000 7f92a2679640 1 -- 172.21.15.114:0/3128400310 shutdown_connections 2024-04-23T10:41:47.626 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:41:47.626+0000 7f92a2679640 1 --2- 172.21.15.114:0/3128400310 >> [v2:172.21.15.114:6800/1395693466,v1:172.21.15.114:6801/1395693466] conn(0x7f92840a3d50 0x7f92840a6210 unknown :-1 s=CLOSED pgs=184 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:41:47.626 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:41:47.626+0000 7f92a2679640 1 --2- 172.21.15.114:0/3128400310 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f929c074cd0 0x7f929c1a6940 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:41:47.627 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:41:47.626+0000 7f92a2679640 1 --2- 172.21.15.114:0/3128400310 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f929c1072a0 0x7f929c074790 unknown :-1 s=CLOSED pgs=976 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:41:47.627 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:41:47.626+0000 7f92a2679640 1 --2- 172.21.15.114:0/3128400310 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f929c1021d0 0x7f929c074250 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:41:47.627 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:41:47.626+0000 7f92a2679640 1 -- 172.21.15.114:0/3128400310 >> 172.21.15.114:0/3128400310 conn(0x7f929c0793e0 msgr2=0x7f929c1000d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T10:41:47.627 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:41:47.626+0000 7f92a2679640 1 -- 172.21.15.114:0/3128400310 shutdown_connections 2024-04-23T10:41:47.627 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:41:47.626+0000 7f92a2679640 1 -- 172.21.15.114:0/3128400310 wait complete. 2024-04-23T10:41:47.636 DEBUG:teuthology.orchestra.run.smithi114:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-04-23T10:41:47.738 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:41:47.736+0000 7f85708e8640 1 -- 172.21.15.114:0/1418251227 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f856c101a50 msgr2=0x7f856c105a90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T10:41:47.739 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:41:47.736+0000 7f85708e8640 1 --2- 172.21.15.114:0/1418251227 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f856c101a50 0x7f856c105a90 secure :-1 s=READY pgs=977 cs=0 l=1 rev1=1 crypto rx=0x7f8554009900 tx=0x7f855402a410 comp rx=0 tx=0).stop 2024-04-23T10:41:47.739 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:41:47.736+0000 7f85708e8640 1 -- 172.21.15.114:0/1418251227 shutdown_connections 2024-04-23T10:41:47.739 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:41:47.736+0000 7f85708e8640 1 --2- 172.21.15.114:0/1418251227 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f856c106170 0x7f856c10dd40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:41:47.739 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:41:47.736+0000 7f85708e8640 1 --2- 172.21.15.114:0/1418251227 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f856c101a50 0x7f856c105a90 unknown :-1 s=CLOSED pgs=977 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:41:47.739 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:41:47.736+0000 7f85708e8640 1 --2- 172.21.15.114:0/1418251227 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f856c1010a0 0x7f856c101480 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:41:47.740 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:41:47.736+0000 7f85708e8640 1 -- 172.21.15.114:0/1418251227 >> 172.21.15.114:0/1418251227 conn(0x7f856c077f50 msgr2=0x7f856c078360 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T10:41:47.740 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:41:47.736+0000 7f85708e8640 1 -- 172.21.15.114:0/1418251227 shutdown_connections 2024-04-23T10:41:47.740 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:41:47.736+0000 7f85708e8640 1 -- 172.21.15.114:0/1418251227 wait complete. 2024-04-23T10:41:47.740 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:41:47.737+0000 7f85708e8640 1 Processor -- start 2024-04-23T10:41:47.740 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:41:47.737+0000 7f85708e8640 1 -- start start 2024-04-23T10:41:47.741 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:41:47.737+0000 7f85708e8640 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f856c1010a0 0x7f856c199530 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T10:41:47.741 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:41:47.737+0000 7f85708e8640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f856c106170 0x7f856c199a70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T10:41:47.741 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:41:47.737+0000 7f85708e8640 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f856c19a000 0x7f856c1a45b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T10:41:47.741 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:41:47.737+0000 7f85708e8640 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x7f856c111090 con 0x7f856c106170 2024-04-23T10:41:47.741 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:41:47.737+0000 7f85708e8640 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 -- 0x7f856c110f10 con 0x7f856c19a000 2024-04-23T10:41:47.742 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:41:47.737+0000 7f85708e8640 1 -- --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_getmap magic: 0 -- 0x7f856c111210 con 0x7f856c1010a0 2024-04-23T10:41:47.742 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:41:47.737+0000 7f8563fff640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f856c106170 0x7f856c199a70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T10:41:47.742 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:41:47.737+0000 7f8563fff640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f856c106170 0x7f856c199a70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.114:53934/0 (socket says 172.21.15.114:53934) 2024-04-23T10:41:47.742 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:41:47.737+0000 7f8563fff640 1 -- 172.21.15.114:0/1217424785 learned_addr learned my addr 172.21.15.114:0/1217424785 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-04-23T10:41:47.742 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:41:47.737+0000 7f856a59c640 1 --2- 172.21.15.114:0/1217424785 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f856c1010a0 0x7f856c199530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T10:41:47.742 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:41:47.738+0000 7f856ad9d640 1 --2- 172.21.15.114:0/1217424785 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f856c19a000 0x7f856c1a45b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T10:41:47.743 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:41:47.738+0000 7f8563fff640 1 -- 172.21.15.114:0/1217424785 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f856c1010a0 msgr2=0x7f856c199530 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T10:41:47.743 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:41:47.738+0000 7f8563fff640 1 --2- 172.21.15.114:0/1217424785 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f856c1010a0 0x7f856c199530 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:41:47.743 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:41:47.738+0000 7f8563fff640 1 -- 172.21.15.114:0/1217424785 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f856c19a000 msgr2=0x7f856c1a45b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T10:41:47.743 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:41:47.738+0000 7f8563fff640 1 --2- 172.21.15.114:0/1217424785 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f856c19a000 0x7f856c1a45b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:41:47.743 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:41:47.738+0000 7f8563fff640 1 -- 172.21.15.114:0/1217424785 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f856c1a4db0 con 0x7f856c106170 2024-04-23T10:41:47.744 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:41:47.738+0000 7f8563fff640 1 --2- 172.21.15.114:0/1217424785 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f856c106170 0x7f856c199a70 secure :-1 s=READY pgs=978 cs=0 l=1 rev1=1 crypto rx=0x7f8554032fd0 tx=0x7f8554004270 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T10:41:47.744 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:41:47.738+0000 7f85637fe640 1 -- 172.21.15.114:0/1217424785 <== mon.0 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f855403f030 con 0x7f856c106170 2024-04-23T10:41:47.744 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:41:47.738+0000 7f85637fe640 1 -- 172.21.15.114:0/1217424785 <== mon.0 v2:172.21.15.114:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f8554004490 con 0x7f856c106170 2024-04-23T10:41:47.744 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:41:47.738+0000 7f85708e8640 1 -- 172.21.15.114:0/1217424785 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f856c1a5040 con 0x7f856c106170 2024-04-23T10:41:47.744 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:41:47.738+0000 7f85637fe640 1 -- 172.21.15.114:0/1217424785 <== mon.0 v2:172.21.15.114:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8554035070 con 0x7f856c106170 2024-04-23T10:41:47.744 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:41:47.738+0000 7f85708e8640 1 -- 172.21.15.114:0/1217424785 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f856c1a5530 con 0x7f856c106170 2024-04-23T10:41:47.745 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:41:47.739+0000 7f85708e8640 1 -- 172.21.15.114:0/1217424785 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f856c068d70 con 0x7f856c106170 2024-04-23T10:41:47.746 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 23 10:41:47 smithi119 ceph-mon[32608]: pgmap v479: 97 pgs: 97 active+clean; 552 MiB data, 4.3 GiB used, 1.0 TiB / 1.0 TiB avail; 335 KiB/s rd, 880 KiB/s wr, 50 op/s 2024-04-23T10:41:47.746 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 23 10:41:47 smithi119 ceph-mon[32608]: from='client.? 172.21.15.114:0/1174688085' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-23T10:41:47.747 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:41:47.740+0000 7f85637fe640 1 -- 172.21.15.114:0/1217424785 <== mon.0 v2:172.21.15.114:3300/0 4 ==== mgrmap(e 26) ==== 138778+0+0 (secure 0 0 0) 0x7f8554042030 con 0x7f856c106170 2024-04-23T10:41:47.747 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:41:47.741+0000 7f85637fe640 1 --2- 172.21.15.114:0/1217424785 >> [v2:172.21.15.114:6800/1395693466,v1:172.21.15.114:6801/1395693466] conn(0x7f85580a3d20 0x7f85580a61e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T10:41:47.747 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:41:47.741+0000 7f85637fe640 1 -- 172.21.15.114:0/1217424785 <== mon.0 v2:172.21.15.114:3300/0 5 ==== osd_map(113..113 src has 1..113) ==== 8629+0+0 (secure 0 0 0) 0x7f85540f11a0 con 0x7f856c106170 2024-04-23T10:41:47.747 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:41:47.741+0000 7f856a59c640 1 --2- 172.21.15.114:0/1217424785 >> [v2:172.21.15.114:6800/1395693466,v1:172.21.15.114:6801/1395693466] conn(0x7f85580a3d20 0x7f85580a61e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T10:41:47.747 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:41:47.741+0000 7f856a59c640 1 --2- 172.21.15.114:0/1217424785 >> [v2:172.21.15.114:6800/1395693466,v1:172.21.15.114:6801/1395693466] conn(0x7f85580a3d20 0x7f85580a61e0 secure :-1 s=READY pgs=185 cs=0 l=1 rev1=1 crypto rx=0x7f855c00bbe0 tx=0x7f855c00b390 comp rx=0 tx=0).ready entity=mgr.15291 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T10:41:47.748 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:41:47.744+0000 7f85637fe640 1 -- 172.21.15.114:0/1217424785 <== mon.0 v2:172.21.15.114:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f85540bd7d0 con 0x7f856c106170 2024-04-23T10:41:47.921 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:41:47.922+0000 7f85708e8640 1 -- 172.21.15.114:0/1217424785 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f856c1093a0 con 0x7f856c106170 2024-04-23T10:41:47.923 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:41:47.924+0000 7f85637fe640 1 -- 172.21.15.114:0/1217424785 <== mon.0 v2:172.21.15.114:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v113) ==== 74+0+23386 (secure 0 0 0) 0x7f85540bd0d0 con 0x7f856c106170 2024-04-23T10:41:47.923 INFO:teuthology.orchestra.run.smithi114.stdout: 2024-04-23T10:41:47.924 INFO:teuthology.orchestra.run.smithi114.stdout:{"epoch":113,"fsid":"aa240572-0159-11ef-bc93-c7b262605968","created":"2024-04-23T10:16:53.351136+0000","modified":"2024-04-23T10:41:13.513569+0000","last_up_change":"2024-04-23T10:23:54.153269+0000","last_in_change":"2024-04-23T10:23:32.963085+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":26,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-04-23T10:20:11.970297+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":12,"score_stable":12,"optimal_score":0.25,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-04-23T10:24:41.218540+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"86","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.8799999952316284,"score_stable":1.8799999952316284,"optimal_score":1,"raw_score_acting":1.8799999952316284,"raw_score_stable":1.8799999952316284,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-04-23T10:24:42.341799+0000","flags":8193,"flags_names":"hashpspool,selfmanaged_snaps","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"112","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":25,"snap_epoch":112,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"fd64adde-9445-45a4-a171-ef656e2c5709","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":88,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6802","nonce":736366878},{"type":"v1","addr":"172.21.15.114:6803","nonce":736366878}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6804","nonce":736366878},{"type":"v1","addr":"172.21.15.114:6805","nonce":736366878}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6808","nonce":736366878},{"type":"v1","addr":"172.21.15.114:6809","nonce":736366878}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6806","nonce":736366878},{"type":"v1","addr":"172.21.15.114:6807","nonce":736366878}]},"public_addr":"172.21.15.114:6803/736366878","cluster_addr":"172.21.15.114:6805/736366878","heartbeat_back_addr":"172.21.15.114:6809/736366878","heartbeat_front_addr":"172.21.15.114:6807/736366878","state":["exists","up"]},{"osd":1,"uuid":"e279a1e6-2702-472a-930a-e08ece76a861","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":85,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6800","nonce":1133540779},{"type":"v1","addr":"172.21.15.119:6801","nonce":1133540779}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6802","nonce":1133540779},{"type":"v1","addr":"172.21.15.119:6803","nonce":1133540779}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6806","nonce":1133540779},{"type":"v1","addr":"172.21.15.119:6807","nonce":1133540779}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6804","nonce":1133540779},{"type":"v1","addr":"172.21.15.119:6805","nonce":1133540779}]},"public_addr":"172.21.15.119:6801/1133540779","cluster_addr":"172.21.15.119:6803/1133540779","heartbeat_back_addr":"172.21.15.119:6807/1133540779","heartbeat_front_addr":"172.21.15.119:6805/1133540779","state":["exists","up"]},{"osd":2,"uuid":"f4c9f2b4-ff49-4e14-89cd-b477feeb9382","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":85,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6800","nonce":799652511},{"type":"v1","addr":"172.21.15.204:6801","nonce":799652511}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6802","nonce":799652511},{"type":"v1","addr":"172.21.15.204:6803","nonce":799652511}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6806","nonce":799652511},{"type":"v1","addr":"172.21.15.204:6807","nonce":799652511}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6804","nonce":799652511},{"type":"v1","addr":"172.21.15.204:6805","nonce":799652511}]},"public_addr":"172.21.15.204:6801/799652511","cluster_addr":"172.21.15.204:6803/799652511","heartbeat_back_addr":"172.21.15.204:6807/799652511","heartbeat_front_addr":"172.21.15.204:6805/799652511","state":["exists","up"]},{"osd":3,"uuid":"52d2a566-e71c-4bb3-8d6c-5abdd43177c2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":25,"up_thru":88,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6810","nonce":4275595091},{"type":"v1","addr":"172.21.15.114:6811","nonce":4275595091}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6812","nonce":4275595091},{"type":"v1","addr":"172.21.15.114:6813","nonce":4275595091}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6816","nonce":4275595091},{"type":"v1","addr":"172.21.15.114:6817","nonce":4275595091}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6814","nonce":4275595091},{"type":"v1","addr":"172.21.15.114:6815","nonce":4275595091}]},"public_addr":"172.21.15.114:6811/4275595091","cluster_addr":"172.21.15.114:6813/4275595091","heartbeat_back_addr":"172.21.15.114:6817/4275595091","heartbeat_front_addr":"172.21.15.114:6815/4275595091","state":["exists","up"]},{"osd":4,"uuid":"d911a7c1-4bdc-4d28-b8a6-73896a6e7de6","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":33,"up_thru":85,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6808","nonce":1611007172},{"type":"v1","addr":"172.21.15.119:6809","nonce":1611007172}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6810","nonce":1611007172},{"type":"v1","addr":"172.21.15.119:6811","nonce":1611007172}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6814","nonce":1611007172},{"type":"v1","addr":"172.21.15.119:6815","nonce":1611007172}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6812","nonce":1611007172},{"type":"v1","addr":"172.21.15.119:6813","nonce":1611007172}]},"public_addr":"172.21.15.119:6809/1611007172","cluster_addr":"172.21.15.119:6811/1611007172","heartbeat_back_addr":"172.21.15.119:6815/1611007172","heartbeat_front_addr":"172.21.15.119:6813/1611007172","state":["exists","up"]},{"osd":5,"uuid":"f7590e1a-e93c-4c29-b15f-38ee52d1f921","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":39,"up_thru":85,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6808","nonce":1427563312},{"type":"v1","addr":"172.21.15.204:6809","nonce":1427563312}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6810","nonce":1427563312},{"type":"v1","addr":"172.21.15.204:6811","nonce":1427563312}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6814","nonce":1427563312},{"type":"v1","addr":"172.21.15.204:6815","nonce":1427563312}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6812","nonce":1427563312},{"type":"v1","addr":"172.21.15.204:6813","nonce":1427563312}]},"public_addr":"172.21.15.204:6809/1427563312","cluster_addr":"172.21.15.204:6811/1427563312","heartbeat_back_addr":"172.21.15.204:6815/1427563312","heartbeat_front_addr":"172.21.15.204:6813/1427563312","state":["exists","up"]},{"osd":6,"uuid":"534d20e8-3e54-4db8-a845-6119f49c7a78","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":47,"up_thru":88,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6818","nonce":4072912544},{"type":"v1","addr":"172.21.15.114:6819","nonce":4072912544}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6820","nonce":4072912544},{"type":"v1","addr":"172.21.15.114:6821","nonce":4072912544}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6824","nonce":4072912544},{"type":"v1","addr":"172.21.15.114:6825","nonce":4072912544}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6822","nonce":4072912544},{"type":"v1","addr":"172.21.15.114:6823","nonce":4072912544}]},"public_addr":"172.21.15.114:6819/4072912544","cluster_addr":"172.21.15.114:6821/4072912544","heartbeat_back_addr":"172.21.15.114:6825/4072912544","heartbeat_front_addr":"172.21.15.114:6823/4072912544","state":["exists","up"]},{"osd":7,"uuid":"f0ec22b8-1b48-4859-9d65-b00403aff3ad","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":53,"up_thru":88,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6816","nonce":2237458793},{"type":"v1","addr":"172.21.15.119:6817","nonce":2237458793}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6818","nonce":2237458793},{"type":"v1","addr":"172.21.15.119:6819","nonce":2237458793}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6822","nonce":2237458793},{"type":"v1","addr":"172.21.15.119:6823","nonce":2237458793}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6820","nonce":2237458793},{"type":"v1","addr":"172.21.15.119:6821","nonce":2237458793}]},"public_addr":"172.21.15.119:6817/2237458793","cluster_addr":"172.21.15.119:6819/2237458793","heartbeat_back_addr":"172.21.15.119:6823/2237458793","heartbeat_front_addr":"172.21.15.119:6821/2237458793","state":["exists","up"]},{"osd":8,"uuid":"3ac89686-6d76-4cef-9089-ac29e4b3bcac","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":62,"up_thru":91,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6816","nonce":1426318231},{"type":"v1","addr":"172.21.15.204:6817","nonce":1426318231}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6818","nonce":1426318231},{"type":"v1","addr":"172.21.15.204:6819","nonce":1426318231}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6822","nonce":1426318231},{"type":"v1","addr":"172.21.15.204:6823","nonce":1426318231}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6820","nonce":1426318231},{"type":"v1","addr":"172.21.15.204:6821","nonce":1426318231}]},"public_addr":"172.21.15.204:6817/1426318231","cluster_addr":"172.21.15.204:6819/1426318231","heartbeat_back_addr":"172.21.15.204:6823/1426318231","heartbeat_front_addr":"172.21.15.204:6821/1426318231","state":["exists","up"]},{"osd":9,"uuid":"a80ffe4d-1e98-460f-b046-c4754285cc5a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":69,"up_thru":88,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6826","nonce":2750635721},{"type":"v1","addr":"172.21.15.114:6827","nonce":2750635721}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6828","nonce":2750635721},{"type":"v1","addr":"172.21.15.114:6829","nonce":2750635721}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6832","nonce":2750635721},{"type":"v1","addr":"172.21.15.114:6833","nonce":2750635721}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6830","nonce":2750635721},{"type":"v1","addr":"172.21.15.114:6831","nonce":2750635721}]},"public_addr":"172.21.15.114:6827/2750635721","cluster_addr":"172.21.15.114:6829/2750635721","heartbeat_back_addr":"172.21.15.114:6833/2750635721","heartbeat_front_addr":"172.21.15.114:6831/2750635721","state":["exists","up"]},{"osd":10,"uuid":"cd75a579-bf12-4600-ac09-a8bfce772ced","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":74,"up_thru":85,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6824","nonce":1400187701},{"type":"v1","addr":"172.21.15.119:6825","nonce":1400187701}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6826","nonce":1400187701},{"type":"v1","addr":"172.21.15.119:6827","nonce":1400187701}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6830","nonce":1400187701},{"type":"v1","addr":"172.21.15.119:6831","nonce":1400187701}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6828","nonce":1400187701},{"type":"v1","addr":"172.21.15.119:6829","nonce":1400187701}]},"public_addr":"172.21.15.119:6825/1400187701","cluster_addr":"172.21.15.119:6827/1400187701","heartbeat_back_addr":"172.21.15.119:6831/1400187701","heartbeat_front_addr":"172.21.15.119:6829/1400187701","state":["exists","up"]},{"osd":11,"uuid":"53f037a5-3ec0-483a-80a1-dfee923a68e5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":82,"up_thru":91,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6824","nonce":2407298490},{"type":"v1","addr":"172.21.15.204:6825","nonce":2407298490}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6826","nonce":2407298490},{"type":"v1","addr":"172.21.15.204:6827","nonce":2407298490}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6830","nonce":2407298490},{"type":"v1","addr":"172.21.15.204:6831","nonce":2407298490}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6828","nonce":2407298490},{"type":"v1","addr":"172.21.15.204:6829","nonce":2407298490}]},"public_addr":"172.21.15.204:6825/2407298490","cluster_addr":"172.21.15.204:6827/2407298490","heartbeat_back_addr":"172.21.15.204:6831/2407298490","heartbeat_front_addr":"172.21.15.204:6829/2407298490","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-23T10:19:18.401147+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-23T10:19:42.611299+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-23T10:20:07.277367+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-23T10:20:31.208368+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-23T10:20:55.866225+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-23T10:21:19.961237+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-23T10:21:44.480783+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-23T10:22:09.173641+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-23T10:22:34.030328+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-23T10:22:59.652293+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-23T10:23:25.659673+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-23T10:23:51.168171+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"2.b","mappings":[{"from":8,"to":11}]},{"pgid":"2.e","mappings":[{"from":2,"to":4}]},{"pgid":"2.10","mappings":[{"from":2,"to":1}]},{"pgid":"2.1d","mappings":[{"from":8,"to":11}]},{"pgid":"3.e","mappings":[{"from":5,"to":9}]},{"pgid":"3.12","mappings":[{"from":3,"to":9}]},{"pgid":"3.17","mappings":[{"from":4,"to":6}]},{"pgid":"3.23","mappings":[{"from":4,"to":7}]},{"pgid":"3.27","mappings":[{"from":4,"to":6}]},{"pgid":"3.31","mappings":[{"from":4,"to":8}]},{"pgid":"3.38","mappings":[{"from":3,"to":9}]},{"pgid":"3.39","mappings":[{"from":5,"to":8}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.114:0/480720790":"2024-04-24T10:26:40.272662+0000","172.21.15.114:6801/465333808":"2024-04-24T10:26:40.272662+0000","172.21.15.114:6800/465333808":"2024-04-24T10:26:40.272662+0000","172.21.15.114:0/1787801099":"2024-04-24T10:26:40.272662+0000","172.21.15.114:6801/1604678557":"2024-04-24T10:17:45.874609+0000","172.21.15.114:0/4025385854":"2024-04-24T10:17:45.874609+0000","172.21.15.114:0/2228273469":"2024-04-24T10:17:19.910348+0000","172.21.15.114:0/2713201622":"2024-04-24T10:26:40.272662+0000","172.21.15.114:6800/2831530838":"2024-04-24T10:17:19.910348+0000","172.21.15.114:0/637004095":"2024-04-24T10:26:40.272662+0000","172.21.15.114:0/1333447450":"2024-04-24T10:17:45.874609+0000","172.21.15.114:0/2233026840":"2024-04-24T10:17:19.910348+0000","172.21.15.114:0/1902281545":"2024-04-24T10:17:19.910348+0000","172.21.15.114:6801/2831530838":"2024-04-24T10:17:19.910348+0000","172.21.15.114:6800/1604678557":"2024-04-24T10:17:45.874609+0000","172.21.15.114:0/1623553811":"2024-04-24T10:17:45.874609+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[{"pool":3,"snaps":[{"begin":12,"length":1},{"begin":25,"length":1}]}],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-23T10:41:47.925 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:41:47.926+0000 7f85708e8640 1 -- 172.21.15.114:0/1217424785 >> [v2:172.21.15.114:6800/1395693466,v1:172.21.15.114:6801/1395693466] conn(0x7f85580a3d20 msgr2=0x7f85580a61e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T10:41:47.925 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:41:47.926+0000 7f85708e8640 1 --2- 172.21.15.114:0/1217424785 >> [v2:172.21.15.114:6800/1395693466,v1:172.21.15.114:6801/1395693466] conn(0x7f85580a3d20 0x7f85580a61e0 secure :-1 s=READY pgs=185 cs=0 l=1 rev1=1 crypto rx=0x7f855c00bbe0 tx=0x7f855c00b390 comp rx=0 tx=0).stop 2024-04-23T10:41:47.925 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:41:47.926+0000 7f85708e8640 1 -- 172.21.15.114:0/1217424785 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f856c106170 msgr2=0x7f856c199a70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T10:41:47.926 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:41:47.926+0000 7f85708e8640 1 --2- 172.21.15.114:0/1217424785 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f856c106170 0x7f856c199a70 secure :-1 s=READY pgs=978 cs=0 l=1 rev1=1 crypto rx=0x7f8554032fd0 tx=0x7f8554004270 comp rx=0 tx=0).stop 2024-04-23T10:41:47.926 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:41:47.926+0000 7f85708e8640 1 -- 172.21.15.114:0/1217424785 shutdown_connections 2024-04-23T10:41:47.926 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:41:47.926+0000 7f85708e8640 1 --2- 172.21.15.114:0/1217424785 >> [v2:172.21.15.114:6800/1395693466,v1:172.21.15.114:6801/1395693466] conn(0x7f85580a3d20 0x7f85580a61e0 unknown :-1 s=CLOSED pgs=185 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:41:47.926 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:41:47.926+0000 7f85708e8640 1 --2- 172.21.15.114:0/1217424785 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f856c19a000 0x7f856c1a45b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:41:47.926 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:41:47.926+0000 7f85708e8640 1 --2- 172.21.15.114:0/1217424785 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f856c106170 0x7f856c199a70 unknown :-1 s=CLOSED pgs=978 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:41:47.927 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:41:47.926+0000 7f85708e8640 1 --2- 172.21.15.114:0/1217424785 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f856c1010a0 0x7f856c199530 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:41:47.927 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:41:47.926+0000 7f85708e8640 1 -- 172.21.15.114:0/1217424785 >> 172.21.15.114:0/1217424785 conn(0x7f856c077f50 msgr2=0x7f856c0ff000 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T10:41:47.927 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:41:47.926+0000 7f85708e8640 1 -- 172.21.15.114:0/1217424785 shutdown_connections 2024-04-23T10:41:47.927 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:41:47.926+0000 7f85708e8640 1 -- 172.21.15.114:0/1217424785 wait complete. 2024-04-23T10:41:47.935 INFO:journalctl@ceph.mon.a.smithi114.stdout:Apr 23 10:41:47 smithi114 ceph-mon[29255]: pgmap v479: 97 pgs: 97 active+clean; 552 MiB data, 4.3 GiB used, 1.0 TiB / 1.0 TiB avail; 335 KiB/s rd, 880 KiB/s wr, 50 op/s 2024-04-23T10:41:47.935 INFO:journalctl@ceph.mon.a.smithi114.stdout:Apr 23 10:41:47 smithi114 ceph-mon[29255]: from='client.? 172.21.15.114:0/1174688085' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-23T10:41:47.937 DEBUG:teuthology.orchestra.run.smithi114:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs fail cephfs 2024-04-23T10:41:48.039 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:41:48.037+0000 7fca8327c640 1 -- 172.21.15.114:0/2959253658 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fca7c100b10 msgr2=0x7fca7c100ef0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T10:41:48.040 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:41:48.037+0000 7fca8327c640 1 --2- 172.21.15.114:0/2959253658 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fca7c100b10 0x7fca7c100ef0 secure :-1 s=READY pgs=979 cs=0 l=1 rev1=1 crypto rx=0x7fca78009900 tx=0x7fca7802a410 comp rx=0 tx=0).stop 2024-04-23T10:41:48.040 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:41:48.037+0000 7fca8327c640 1 -- 172.21.15.114:0/2959253658 shutdown_connections 2024-04-23T10:41:48.040 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:41:48.037+0000 7fca8327c640 1 --2- 172.21.15.114:0/2959253658 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fca7c1072d0 0x7fca7c10eec0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:41:48.040 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:41:48.037+0000 7fca8327c640 1 --2- 172.21.15.114:0/2959253658 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fca7c1014c0 0x7fca7c106bf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:41:48.041 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:41:48.037+0000 7fca8327c640 1 --2- 172.21.15.114:0/2959253658 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fca7c100b10 0x7fca7c100ef0 unknown :-1 s=CLOSED pgs=979 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:41:48.041 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:41:48.037+0000 7fca8327c640 1 -- 172.21.15.114:0/2959253658 >> 172.21.15.114:0/2959253658 conn(0x7fca7c076130 msgr2=0x7fca7c076540 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T10:41:48.041 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:41:48.037+0000 7fca8327c640 1 -- 172.21.15.114:0/2959253658 shutdown_connections 2024-04-23T10:41:48.041 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:41:48.037+0000 7fca8327c640 1 -- 172.21.15.114:0/2959253658 wait complete. 2024-04-23T10:41:48.041 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:41:48.038+0000 7fca8327c640 1 Processor -- start 2024-04-23T10:41:48.042 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:41:48.038+0000 7fca8327c640 1 -- start start 2024-04-23T10:41:48.042 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:41:48.038+0000 7fca8327c640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fca7c100b10 0x7fca7c19e5d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T10:41:48.042 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:41:48.038+0000 7fca8327c640 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fca7c1072d0 0x7fca7c19eb10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T10:41:48.042 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:41:48.038+0000 7fca8327c640 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fca7c198690 0x7fca7c198b10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T10:41:48.042 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:41:48.038+0000 7fca8327c640 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x7fca7c1122d0 con 0x7fca7c100b10 2024-04-23T10:41:48.043 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:41:48.038+0000 7fca8327c640 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 -- 0x7fca7c112150 con 0x7fca7c198690 2024-04-23T10:41:48.043 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:41:48.038+0000 7fca8327c640 1 -- --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_getmap magic: 0 -- 0x7fca7c112450 con 0x7fca7c1072d0 2024-04-23T10:41:48.043 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:41:48.038+0000 7fca81018640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fca7c100b10 0x7fca7c19e5d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T10:41:48.043 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:41:48.038+0000 7fca81018640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fca7c100b10 0x7fca7c19e5d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.114:53954/0 (socket says 172.21.15.114:53954) 2024-04-23T10:41:48.043 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:41:48.038+0000 7fca81018640 1 -- 172.21.15.114:0/3099043748 learned_addr learned my addr 172.21.15.114:0/3099043748 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-04-23T10:41:48.043 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:41:48.038+0000 7fca80817640 1 --2- 172.21.15.114:0/3099043748 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fca7c1072d0 0x7fca7c19eb10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T10:41:48.044 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:41:48.038+0000 7fca81819640 1 --2- 172.21.15.114:0/3099043748 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fca7c198690 0x7fca7c198b10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T10:41:48.044 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:41:48.038+0000 7fca81018640 1 -- 172.21.15.114:0/3099043748 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fca7c1072d0 msgr2=0x7fca7c19eb10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T10:41:48.044 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:41:48.038+0000 7fca81018640 1 --2- 172.21.15.114:0/3099043748 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fca7c1072d0 0x7fca7c19eb10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:41:48.044 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:41:48.038+0000 7fca81018640 1 -- 172.21.15.114:0/3099043748 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fca7c198690 msgr2=0x7fca7c198b10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T10:41:48.044 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:41:48.038+0000 7fca81018640 1 --2- 172.21.15.114:0/3099043748 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fca7c198690 0x7fca7c198b10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:41:48.044 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:41:48.038+0000 7fca81018640 1 -- 172.21.15.114:0/3099043748 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fca7c199290 con 0x7fca7c100b10 2024-04-23T10:41:48.045 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:41:48.038+0000 7fca81819640 1 --2- 172.21.15.114:0/3099043748 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fca7c198690 0x7fca7c198b10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-23T10:41:48.045 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:41:48.038+0000 7fca81018640 1 --2- 172.21.15.114:0/3099043748 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fca7c100b10 0x7fca7c19e5d0 secure :-1 s=READY pgs=980 cs=0 l=1 rev1=1 crypto rx=0x7fca78006010 tx=0x7fca780042f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T10:41:48.045 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:41:48.039+0000 7fca727fc640 1 -- 172.21.15.114:0/3099043748 <== mon.0 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fca7803f030 con 0x7fca7c100b10 2024-04-23T10:41:48.045 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:41:48.039+0000 7fca727fc640 1 -- 172.21.15.114:0/3099043748 <== mon.0 v2:172.21.15.114:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fca78003800 con 0x7fca7c100b10 2024-04-23T10:41:48.045 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:41:48.039+0000 7fca8327c640 1 -- 172.21.15.114:0/3099043748 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fca7c199520 con 0x7fca7c100b10 2024-04-23T10:41:48.045 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:41:48.039+0000 7fca727fc640 1 -- 172.21.15.114:0/3099043748 <== mon.0 v2:172.21.15.114:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fca78035070 con 0x7fca7c100b10 2024-04-23T10:41:48.045 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:41:48.039+0000 7fca8327c640 1 -- 172.21.15.114:0/3099043748 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fca7c1a5660 con 0x7fca7c100b10 2024-04-23T10:41:48.046 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:41:48.040+0000 7fca8327c640 1 -- 172.21.15.114:0/3099043748 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fca7c06a0a0 con 0x7fca7c100b10 2024-04-23T10:41:48.046 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:41:48.041+0000 7fca727fc640 1 -- 172.21.15.114:0/3099043748 <== mon.0 v2:172.21.15.114:3300/0 4 ==== mgrmap(e 26) ==== 138778+0+0 (secure 0 0 0) 0x7fca78003cb0 con 0x7fca7c100b10 2024-04-23T10:41:48.047 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:41:48.042+0000 7fca727fc640 1 --2- 172.21.15.114:0/3099043748 >> [v2:172.21.15.114:6800/1395693466,v1:172.21.15.114:6801/1395693466] conn(0x7fca680a3d20 0x7fca680a61e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T10:41:48.047 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:41:48.042+0000 7fca80817640 1 --2- 172.21.15.114:0/3099043748 >> [v2:172.21.15.114:6800/1395693466,v1:172.21.15.114:6801/1395693466] conn(0x7fca680a3d20 0x7fca680a61e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T10:41:48.047 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:41:48.042+0000 7fca727fc640 1 -- 172.21.15.114:0/3099043748 <== mon.0 v2:172.21.15.114:3300/0 5 ==== osd_map(113..113 src has 1..113) ==== 8629+0+0 (secure 0 0 0) 0x7fca780f12b0 con 0x7fca7c100b10 2024-04-23T10:41:48.048 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:41:48.042+0000 7fca80817640 1 --2- 172.21.15.114:0/3099043748 >> [v2:172.21.15.114:6800/1395693466,v1:172.21.15.114:6801/1395693466] conn(0x7fca680a3d20 0x7fca680a61e0 secure :-1 s=READY pgs=186 cs=0 l=1 rev1=1 crypto rx=0x7fca7c199ee0 tx=0x7fca6c008040 comp rx=0 tx=0).ready entity=mgr.15291 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T10:41:48.048 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:41:48.046+0000 7fca727fc640 1 -- 172.21.15.114:0/3099043748 <== mon.0 v2:172.21.15.114:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7fca780bd8e0 con 0x7fca7c100b10 2024-04-23T10:41:48.078 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 23 10:41:47 smithi204 ceph-mon[32649]: pgmap v479: 97 pgs: 97 active+clean; 552 MiB data, 4.3 GiB used, 1.0 TiB / 1.0 TiB avail; 335 KiB/s rd, 880 KiB/s wr, 50 op/s 2024-04-23T10:41:48.078 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 23 10:41:47 smithi204 ceph-mon[32649]: from='client.? 172.21.15.114:0/1174688085' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-23T10:41:48.264 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:41:48.265+0000 7fca8327c640 1 -- 172.21.15.114:0/3099043748 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_command({"prefix": "fs fail", "fs_name": "cephfs"} v 0) -- 0x7fca7c10a480 con 0x7fca7c100b10 2024-04-23T10:41:48.266 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:41:48.268+0000 7fca727fc640 1 -- 172.21.15.114:0/3099043748 <== mon.0 v2:172.21.15.114:3300/0 7 ==== mon_command_ack([{"prefix": "fs fail", "fs_name": "cephfs"}]=0 cephfs marked not joinable; MDS cannot join the cluster. All MDS ranks marked failed. v50) ==== 161+0+0 (secure 0 0 0) 0x7fca780bd1e0 con 0x7fca7c100b10 2024-04-23T10:41:48.267 INFO:teuthology.orchestra.run.smithi114.stderr:cephfs marked not joinable; MDS cannot join the cluster. All MDS ranks marked failed. 2024-04-23T10:41:48.270 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:41:48.270+0000 7fca8327c640 1 -- 172.21.15.114:0/3099043748 >> [v2:172.21.15.114:6800/1395693466,v1:172.21.15.114:6801/1395693466] conn(0x7fca680a3d20 msgr2=0x7fca680a61e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T10:41:48.270 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:41:48.270+0000 7fca8327c640 1 --2- 172.21.15.114:0/3099043748 >> [v2:172.21.15.114:6800/1395693466,v1:172.21.15.114:6801/1395693466] conn(0x7fca680a3d20 0x7fca680a61e0 secure :-1 s=READY pgs=186 cs=0 l=1 rev1=1 crypto rx=0x7fca7c199ee0 tx=0x7fca6c008040 comp rx=0 tx=0).stop 2024-04-23T10:41:48.270 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:41:48.270+0000 7fca8327c640 1 -- 172.21.15.114:0/3099043748 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fca7c100b10 msgr2=0x7fca7c19e5d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T10:41:48.270 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:41:48.270+0000 7fca8327c640 1 --2- 172.21.15.114:0/3099043748 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fca7c100b10 0x7fca7c19e5d0 secure :-1 s=READY pgs=980 cs=0 l=1 rev1=1 crypto rx=0x7fca78006010 tx=0x7fca780042f0 comp rx=0 tx=0).stop 2024-04-23T10:41:48.270 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:41:48.271+0000 7fca8327c640 1 -- 172.21.15.114:0/3099043748 shutdown_connections 2024-04-23T10:41:48.271 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:41:48.271+0000 7fca8327c640 1 --2- 172.21.15.114:0/3099043748 >> [v2:172.21.15.114:6800/1395693466,v1:172.21.15.114:6801/1395693466] conn(0x7fca680a3d20 0x7fca680a61e0 unknown :-1 s=CLOSED pgs=186 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:41:48.271 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:41:48.271+0000 7fca8327c640 1 --2- 172.21.15.114:0/3099043748 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fca7c198690 0x7fca7c198b10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:41:48.271 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:41:48.271+0000 7fca8327c640 1 --2- 172.21.15.114:0/3099043748 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7fca7c1072d0 0x7fca7c19eb10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:41:48.271 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:41:48.271+0000 7fca8327c640 1 --2- 172.21.15.114:0/3099043748 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fca7c100b10 0x7fca7c19e5d0 unknown :-1 s=CLOSED pgs=980 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:41:48.271 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:41:48.271+0000 7fca8327c640 1 -- 172.21.15.114:0/3099043748 >> 172.21.15.114:0/3099043748 conn(0x7fca7c076130 msgr2=0x7fca7c0fed10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T10:41:48.271 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:41:48.271+0000 7fca8327c640 1 -- 172.21.15.114:0/3099043748 shutdown_connections 2024-04-23T10:41:48.272 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:41:48.271+0000 7fca8327c640 1 -- 172.21.15.114:0/3099043748 wait complete. 2024-04-23T10:41:48.281 DEBUG:teuthology.orchestra.run.smithi114:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs rm cephfs --yes-i-really-mean-it 2024-04-23T10:41:48.386 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:41:48.384+0000 7f9324920640 1 -- 172.21.15.114:0/3034192860 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f9320101ad0 msgr2=0x7f932010dcf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T10:41:48.386 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:41:48.384+0000 7f9324920640 1 --2- 172.21.15.114:0/3034192860 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f9320101ad0 0x7f932010dcf0 secure :-1 s=READY pgs=984 cs=0 l=1 rev1=1 crypto rx=0x7f9318009ca0 tx=0x7f931802a8c0 comp rx=0 tx=0).stop 2024-04-23T10:41:48.386 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:41:48.384+0000 7f9324920640 1 -- 172.21.15.114:0/3034192860 shutdown_connections 2024-04-23T10:41:48.386 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:41:48.384+0000 7f9324920640 1 --2- 172.21.15.114:0/3034192860 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f9320101ad0 0x7f932010dcf0 unknown :-1 s=CLOSED pgs=984 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:41:48.387 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:41:48.384+0000 7f9324920640 1 --2- 172.21.15.114:0/3034192860 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f9320101130 0x7f9320101590 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:41:48.387 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:41:48.384+0000 7f9324920640 1 --2- 172.21.15.114:0/3034192860 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f9320105ea0 0x7f9320106280 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:41:48.387 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:41:48.384+0000 7f9324920640 1 -- 172.21.15.114:0/3034192860 >> 172.21.15.114:0/3034192860 conn(0x7f9320077e30 msgr2=0x7f9320078240 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T10:41:48.387 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:41:48.384+0000 7f9324920640 1 -- 172.21.15.114:0/3034192860 shutdown_connections 2024-04-23T10:41:48.387 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:41:48.384+0000 7f9324920640 1 -- 172.21.15.114:0/3034192860 wait complete. 2024-04-23T10:41:48.387 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:41:48.385+0000 7f9324920640 1 Processor -- start 2024-04-23T10:41:48.387 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:41:48.385+0000 7f9324920640 1 -- start start 2024-04-23T10:41:48.388 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:41:48.385+0000 7f9324920640 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f9320101130 0x7f932019e410 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T10:41:48.388 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:41:48.385+0000 7f9324920640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f9320101ad0 0x7f932019e950 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T10:41:48.388 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:41:48.385+0000 7f9324920640 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f9320105ea0 0x7f93201985c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T10:41:48.388 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:41:48.385+0000 7f9324920640 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x7f9320112320 con 0x7f9320101ad0 2024-04-23T10:41:48.388 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:41:48.385+0000 7f9324920640 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 -- 0x7f93201121a0 con 0x7f9320101130 2024-04-23T10:41:48.388 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:41:48.385+0000 7f9324920640 1 -- --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_getmap magic: 0 -- 0x7f93201124a0 con 0x7f9320105ea0 2024-04-23T10:41:48.388 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:41:48.385+0000 7f931dd9b640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f9320101ad0 0x7f932019e950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T10:41:48.389 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:41:48.385+0000 7f931dd9b640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f9320101ad0 0x7f932019e950 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.114:53968/0 (socket says 172.21.15.114:53968) 2024-04-23T10:41:48.389 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:41:48.385+0000 7f931dd9b640 1 -- 172.21.15.114:0/3205085398 learned_addr learned my addr 172.21.15.114:0/3205085398 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-04-23T10:41:48.389 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:41:48.385+0000 7f931ed9d640 1 --2- 172.21.15.114:0/3205085398 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f9320105ea0 0x7f93201985c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T10:41:48.389 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:41:48.385+0000 7f931e59c640 1 --2- 172.21.15.114:0/3205085398 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f9320101130 0x7f932019e410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T10:41:48.389 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:41:48.386+0000 7f931dd9b640 1 -- 172.21.15.114:0/3205085398 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f9320105ea0 msgr2=0x7f93201985c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T10:41:48.389 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:41:48.386+0000 7f931dd9b640 1 --2- 172.21.15.114:0/3205085398 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f9320105ea0 0x7f93201985c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:41:48.389 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:41:48.386+0000 7f931dd9b640 1 -- 172.21.15.114:0/3205085398 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f9320101130 msgr2=0x7f932019e410 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T10:41:48.390 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:41:48.386+0000 7f931dd9b640 1 --2- 172.21.15.114:0/3205085398 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f9320101130 0x7f932019e410 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:41:48.390 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:41:48.386+0000 7f931dd9b640 1 -- 172.21.15.114:0/3205085398 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f9320198d40 con 0x7f9320101ad0 2024-04-23T10:41:48.390 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:41:48.386+0000 7f931e59c640 1 --2- 172.21.15.114:0/3205085398 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f9320101130 0x7f932019e410 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-23T10:41:48.390 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:41:48.386+0000 7f931dd9b640 1 --2- 172.21.15.114:0/3205085398 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f9320101ad0 0x7f932019e950 secure :-1 s=READY pgs=985 cs=0 l=1 rev1=1 crypto rx=0x7f931400d770 tx=0x7f931400dc30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T10:41:48.390 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:41:48.386+0000 7f930f7fe640 1 -- 172.21.15.114:0/3205085398 <== mon.0 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9314017070 con 0x7f9320101ad0 2024-04-23T10:41:48.390 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:41:48.386+0000 7f9324920640 1 -- 172.21.15.114:0/3205085398 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f9320199030 con 0x7f9320101ad0 2024-04-23T10:41:48.391 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:41:48.386+0000 7f9324920640 1 -- 172.21.15.114:0/3205085398 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f93201a52d0 con 0x7f9320101ad0 2024-04-23T10:41:48.391 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:41:48.386+0000 7f930f7fe640 1 -- 172.21.15.114:0/3205085398 <== mon.0 v2:172.21.15.114:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f9314009c30 con 0x7f9320101ad0 2024-04-23T10:41:48.391 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:41:48.386+0000 7f930f7fe640 1 -- 172.21.15.114:0/3205085398 <== mon.0 v2:172.21.15.114:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9314020060 con 0x7f9320101ad0 2024-04-23T10:41:48.391 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:41:48.387+0000 7f9324920640 1 -- 172.21.15.114:0/3205085398 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f9300005290 con 0x7f9320101ad0 2024-04-23T10:41:48.392 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:41:48.388+0000 7f930f7fe640 1 -- 172.21.15.114:0/3205085398 <== mon.0 v2:172.21.15.114:3300/0 4 ==== mgrmap(e 26) ==== 138778+0+0 (secure 0 0 0) 0x7f9314010460 con 0x7f9320101ad0 2024-04-23T10:41:48.392 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:41:48.389+0000 7f930f7fe640 1 --2- 172.21.15.114:0/3205085398 >> [v2:172.21.15.114:6800/1395693466,v1:172.21.15.114:6801/1395693466] conn(0x7f93080a3d20 0x7f93080a61e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T10:41:48.393 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:41:48.389+0000 7f931e59c640 1 --2- 172.21.15.114:0/3205085398 >> [v2:172.21.15.114:6800/1395693466,v1:172.21.15.114:6801/1395693466] conn(0x7f93080a3d20 0x7f93080a61e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T10:41:48.393 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:41:48.389+0000 7f930f7fe640 1 -- 172.21.15.114:0/3205085398 <== mon.0 v2:172.21.15.114:3300/0 5 ==== osd_map(114..114 src has 1..114) ==== 8843+0+0 (secure 0 0 0) 0x7f93140d0210 con 0x7f9320101ad0 2024-04-23T10:41:48.393 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:41:48.389+0000 7f931e59c640 1 --2- 172.21.15.114:0/3205085398 >> [v2:172.21.15.114:6800/1395693466,v1:172.21.15.114:6801/1395693466] conn(0x7f93080a3d20 0x7f93080a61e0 secure :-1 s=READY pgs=187 cs=0 l=1 rev1=1 crypto rx=0x7f93100058c0 tx=0x7f9310005ec0 comp rx=0 tx=0).ready entity=mgr.15291 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T10:41:48.393 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:41:48.392+0000 7f930f7fe640 1 -- 172.21.15.114:0/3205085398 <== mon.0 v2:172.21.15.114:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f931409c810 con 0x7f9320101ad0 2024-04-23T10:41:48.578 INFO:journalctl@ceph.mds.i.smithi204.stdout:Apr 23 10:41:48 smithi204 ceph-aa240572-0159-11ef-bc93-c7b262605968-mds-i[63304]: ignoring --setuser ceph since I am not root 2024-04-23T10:41:48.578 INFO:journalctl@ceph.mds.i.smithi204.stdout:Apr 23 10:41:48 smithi204 ceph-aa240572-0159-11ef-bc93-c7b262605968-mds-i[63304]: ignoring --setgroup ceph since I am not root 2024-04-23T10:41:48.578 INFO:journalctl@ceph.mds.i.smithi204.stdout:Apr 23 10:41:48 smithi204 ceph-mds[63308]: -- 172.21.15.204:0/2492762517 <== mon.1 v2:172.21.15.204:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 194+0+0 (secure 0 0 0) 0x564888633a40 con 0x564888607000 2024-04-23T10:41:48.578 INFO:journalctl@ceph.mds.i.smithi204.stdout:Apr 23 10:41:48 smithi204 ceph-aa240572-0159-11ef-bc93-c7b262605968-mds-i[63304]: starting mds.i at 2024-04-23T10:41:48.578 INFO:journalctl@ceph.mds.c.smithi204.stdout:Apr 23 10:41:48 smithi204 ceph-aa240572-0159-11ef-bc93-c7b262605968-mds-c[62990]: ignoring --setuser ceph since I am not root 2024-04-23T10:41:48.579 INFO:journalctl@ceph.mds.c.smithi204.stdout:Apr 23 10:41:48 smithi204 ceph-aa240572-0159-11ef-bc93-c7b262605968-mds-c[62990]: ignoring --setgroup ceph since I am not root 2024-04-23T10:41:48.579 INFO:journalctl@ceph.mds.c.smithi204.stdout:Apr 23 10:41:48 smithi204 ceph-mds[62994]: -- 172.21.15.204:0/4039142987 <== mon.1 v2:172.21.15.204:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 194+0+0 (secure 0 0 0) 0x55660b08ba40 con 0x55660b05f000 2024-04-23T10:41:48.579 INFO:journalctl@ceph.mds.c.smithi204.stdout:Apr 23 10:41:48 smithi204 ceph-aa240572-0159-11ef-bc93-c7b262605968-mds-c[62990]: starting mds.c at 2024-04-23T10:41:48.584 INFO:journalctl@ceph.mon.a.smithi114.stdout:Apr 23 10:41:48 smithi114 ceph-aa240572-0159-11ef-bc93-c7b262605968-mon-a[29251]: 2024-04-23T10:41:48.266+0000 7efe7c7d9700 -1 log_channel(cluster) log [ERR] : Health check failed: 1 filesystem is offline (MDS_ALL_DOWN) 2024-04-23T10:41:48.585 INFO:journalctl@ceph.mds.b.smithi119.stdout:Apr 23 10:41:48 smithi119 ceph-aa240572-0159-11ef-bc93-c7b262605968-mds-b[62689]: ignoring --setuser ceph since I am not root 2024-04-23T10:41:48.586 INFO:journalctl@ceph.mds.b.smithi119.stdout:Apr 23 10:41:48 smithi119 ceph-aa240572-0159-11ef-bc93-c7b262605968-mds-b[62689]: ignoring --setgroup ceph since I am not root 2024-04-23T10:41:48.586 INFO:journalctl@ceph.mds.b.smithi119.stdout:Apr 23 10:41:48 smithi119 ceph-mds[62693]: -- 172.21.15.119:0/4066953359 <== mon.2 v2:172.21.15.119:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 194+0+0 (secure 0 0 0) 0x55d3f255fa40 con 0x55d3f2532c00 2024-04-23T10:41:48.586 INFO:journalctl@ceph.mds.b.smithi119.stdout:Apr 23 10:41:48 smithi119 ceph-aa240572-0159-11ef-bc93-c7b262605968-mds-b[62689]: starting mds.b at 2024-04-23T10:41:48.586 INFO:journalctl@ceph.mds.k.smithi119.stdout:Apr 23 10:41:48 smithi119 ceph-aa240572-0159-11ef-bc93-c7b262605968-mds-k[61746]: ignoring --setuser ceph since I am not root 2024-04-23T10:41:48.586 INFO:journalctl@ceph.mds.k.smithi119.stdout:Apr 23 10:41:48 smithi119 ceph-aa240572-0159-11ef-bc93-c7b262605968-mds-k[61746]: ignoring --setgroup ceph since I am not root 2024-04-23T10:41:48.587 INFO:journalctl@ceph.mds.k.smithi119.stdout:Apr 23 10:41:48 smithi119 ceph-mds[61750]: -- 172.21.15.119:0/1295720084 <== mon.0 v2:172.21.15.114:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 194+0+0 (secure 0 0 0) 0x562d852b3a40 con 0x562d85286800 2024-04-23T10:41:48.587 INFO:journalctl@ceph.mds.k.smithi119.stdout:Apr 23 10:41:48 smithi119 ceph-aa240572-0159-11ef-bc93-c7b262605968-mds-k[61746]: starting mds.k at 2024-04-23T10:41:48.587 INFO:journalctl@ceph.mds.e.smithi119.stdout:Apr 23 10:41:48 smithi119 ceph-aa240572-0159-11ef-bc93-c7b262605968-mds-e[62063]: ignoring --setuser ceph since I am not root 2024-04-23T10:41:48.587 INFO:journalctl@ceph.mds.e.smithi119.stdout:Apr 23 10:41:48 smithi119 ceph-aa240572-0159-11ef-bc93-c7b262605968-mds-e[62063]: ignoring --setgroup ceph since I am not root 2024-04-23T10:41:48.587 INFO:journalctl@ceph.mds.e.smithi119.stdout:Apr 23 10:41:48 smithi119 ceph-mds[62067]: -- 172.21.15.119:0/2610796484 <== mon.0 v2:172.21.15.114:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 194+0+0 (secure 0 0 0) 0x55c785cbba40 con 0x55c785c8ec00 2024-04-23T10:41:48.588 INFO:journalctl@ceph.mds.e.smithi119.stdout:Apr 23 10:41:48 smithi119 ceph-aa240572-0159-11ef-bc93-c7b262605968-mds-e[62063]: starting mds.e at 2024-04-23T10:41:48.588 INFO:journalctl@ceph.mds.h.smithi119.stdout:Apr 23 10:41:48 smithi119 ceph-aa240572-0159-11ef-bc93-c7b262605968-mds-h[62388]: ignoring --setuser ceph since I am not root 2024-04-23T10:41:48.588 INFO:journalctl@ceph.mds.h.smithi119.stdout:Apr 23 10:41:48 smithi119 ceph-aa240572-0159-11ef-bc93-c7b262605968-mds-h[62388]: ignoring --setgroup ceph since I am not root 2024-04-23T10:41:48.588 INFO:journalctl@ceph.mds.h.smithi119.stdout:Apr 23 10:41:48 smithi119 ceph-mds[62392]: -- 172.21.15.119:0/1836883886 <== mon.2 v2:172.21.15.119:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 194+0+0 (secure 0 0 0) 0x5571643f9a40 con 0x5571643cc800 2024-04-23T10:41:48.588 INFO:journalctl@ceph.mds.h.smithi119.stdout:Apr 23 10:41:48 smithi119 ceph-aa240572-0159-11ef-bc93-c7b262605968-mds-h[62388]: starting mds.h at 2024-04-23T10:41:48.612 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:41:48.613+0000 7f9324920640 1 -- 172.21.15.114:0/3205085398 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_command({"prefix": "fs rm", "fs_name": "cephfs", "yes_i_really_mean_it": true} v 0) -- 0x7f9300005850 con 0x7f9320101ad0 2024-04-23T10:41:48.992 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 23 10:41:48 smithi119 ceph-mon[32608]: from='client.? 172.21.15.114:0/3128400310' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-23T10:41:48.992 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 23 10:41:48 smithi119 ceph-mon[32608]: from='client.? 172.21.15.114:0/1217424785' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-23T10:41:48.992 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 23 10:41:48 smithi119 ceph-mon[32608]: from='client.? 172.21.15.114:0/3099043748' entity='client.admin' cmd=[{"prefix": "fs fail", "fs_name": "cephfs"}]: dispatch 2024-04-23T10:41:48.992 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 23 10:41:48 smithi119 ceph-mon[32608]: Health check failed: 1 filesystem is degraded (FS_DEGRADED) 2024-04-23T10:41:48.992 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 23 10:41:48 smithi119 ceph-mon[32608]: Health check failed: 1 filesystem is offline (MDS_ALL_DOWN) 2024-04-23T10:41:48.993 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 23 10:41:48 smithi119 ceph-mon[32608]: from='client.? 172.21.15.114:0/3099043748' entity='client.admin' cmd='[{"prefix": "fs fail", "fs_name": "cephfs"}]': finished 2024-04-23T10:41:48.993 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 23 10:41:48 smithi119 ceph-mon[32608]: osdmap e114: 12 total, 12 up, 12 in 2024-04-23T10:41:48.993 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 23 10:41:48 smithi119 ceph-mon[32608]: fsmap cephfs:0/3 6 up:standby, 3 failed 2024-04-23T10:41:49.004 INFO:journalctl@ceph.mon.a.smithi114.stdout:Apr 23 10:41:48 smithi114 ceph-mon[29255]: from='client.? 172.21.15.114:0/3128400310' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-23T10:41:49.004 INFO:journalctl@ceph.mon.a.smithi114.stdout:Apr 23 10:41:48 smithi114 ceph-mon[29255]: from='client.? 172.21.15.114:0/1217424785' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-23T10:41:49.004 INFO:journalctl@ceph.mon.a.smithi114.stdout:Apr 23 10:41:48 smithi114 ceph-mon[29255]: from='client.? 172.21.15.114:0/3099043748' entity='client.admin' cmd=[{"prefix": "fs fail", "fs_name": "cephfs"}]: dispatch 2024-04-23T10:41:49.004 INFO:journalctl@ceph.mon.a.smithi114.stdout:Apr 23 10:41:48 smithi114 ceph-mon[29255]: Health check failed: 1 filesystem is degraded (FS_DEGRADED) 2024-04-23T10:41:49.004 INFO:journalctl@ceph.mon.a.smithi114.stdout:Apr 23 10:41:48 smithi114 ceph-mon[29255]: Health check failed: 1 filesystem is offline (MDS_ALL_DOWN) 2024-04-23T10:41:49.004 INFO:journalctl@ceph.mon.a.smithi114.stdout:Apr 23 10:41:48 smithi114 ceph-mon[29255]: from='client.? 172.21.15.114:0/3099043748' entity='client.admin' cmd='[{"prefix": "fs fail", "fs_name": "cephfs"}]': finished 2024-04-23T10:41:49.004 INFO:journalctl@ceph.mon.a.smithi114.stdout:Apr 23 10:41:48 smithi114 ceph-mon[29255]: osdmap e114: 12 total, 12 up, 12 in 2024-04-23T10:41:49.005 INFO:journalctl@ceph.mon.a.smithi114.stdout:Apr 23 10:41:48 smithi114 ceph-mon[29255]: fsmap cephfs:0/3 6 up:standby, 3 failed 2024-04-23T10:41:49.078 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 23 10:41:48 smithi204 ceph-mon[32649]: from='client.? 172.21.15.114:0/3128400310' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-23T10:41:49.078 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 23 10:41:48 smithi204 ceph-mon[32649]: from='client.? 172.21.15.114:0/1217424785' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-23T10:41:49.078 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 23 10:41:48 smithi204 ceph-mon[32649]: from='client.? 172.21.15.114:0/3099043748' entity='client.admin' cmd=[{"prefix": "fs fail", "fs_name": "cephfs"}]: dispatch 2024-04-23T10:41:49.078 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 23 10:41:48 smithi204 ceph-mon[32649]: Health check failed: 1 filesystem is degraded (FS_DEGRADED) 2024-04-23T10:41:49.078 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 23 10:41:48 smithi204 ceph-mon[32649]: Health check failed: 1 filesystem is offline (MDS_ALL_DOWN) 2024-04-23T10:41:49.078 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 23 10:41:48 smithi204 ceph-mon[32649]: from='client.? 172.21.15.114:0/3099043748' entity='client.admin' cmd='[{"prefix": "fs fail", "fs_name": "cephfs"}]': finished 2024-04-23T10:41:49.079 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 23 10:41:48 smithi204 ceph-mon[32649]: osdmap e114: 12 total, 12 up, 12 in 2024-04-23T10:41:49.079 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 23 10:41:48 smithi204 ceph-mon[32649]: fsmap cephfs:0/3 6 up:standby, 3 failed 2024-04-23T10:41:49.274 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:41:49.275+0000 7f930f7fe640 1 -- 172.21.15.114:0/3205085398 <== mon.0 v2:172.21.15.114:3300/0 7 ==== mon_command_ack([{"prefix": "fs rm", "fs_name": "cephfs", "yes_i_really_mean_it": true}]=0 v51) ==== 104+0+0 (secure 0 0 0) 0x7f931409c110 con 0x7f9320101ad0 2024-04-23T10:41:49.278 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:41:49.278+0000 7f9324920640 1 -- 172.21.15.114:0/3205085398 >> [v2:172.21.15.114:6800/1395693466,v1:172.21.15.114:6801/1395693466] conn(0x7f93080a3d20 msgr2=0x7f93080a61e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T10:41:49.278 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:41:49.278+0000 7f9324920640 1 --2- 172.21.15.114:0/3205085398 >> [v2:172.21.15.114:6800/1395693466,v1:172.21.15.114:6801/1395693466] conn(0x7f93080a3d20 0x7f93080a61e0 secure :-1 s=READY pgs=187 cs=0 l=1 rev1=1 crypto rx=0x7f93100058c0 tx=0x7f9310005ec0 comp rx=0 tx=0).stop 2024-04-23T10:41:49.278 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:41:49.278+0000 7f9324920640 1 -- 172.21.15.114:0/3205085398 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f9320101ad0 msgr2=0x7f932019e950 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T10:41:49.278 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:41:49.278+0000 7f9324920640 1 --2- 172.21.15.114:0/3205085398 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f9320101ad0 0x7f932019e950 secure :-1 s=READY pgs=985 cs=0 l=1 rev1=1 crypto rx=0x7f931400d770 tx=0x7f931400dc30 comp rx=0 tx=0).stop 2024-04-23T10:41:49.278 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:41:49.278+0000 7f9324920640 1 -- 172.21.15.114:0/3205085398 shutdown_connections 2024-04-23T10:41:49.279 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:41:49.278+0000 7f9324920640 1 --2- 172.21.15.114:0/3205085398 >> [v2:172.21.15.114:6800/1395693466,v1:172.21.15.114:6801/1395693466] conn(0x7f93080a3d20 0x7f93080a61e0 unknown :-1 s=CLOSED pgs=187 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:41:49.279 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:41:49.278+0000 7f9324920640 1 --2- 172.21.15.114:0/3205085398 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f9320105ea0 0x7f93201985c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:41:49.279 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:41:49.278+0000 7f9324920640 1 --2- 172.21.15.114:0/3205085398 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f9320101ad0 0x7f932019e950 unknown :-1 s=CLOSED pgs=985 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:41:49.279 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:41:49.278+0000 7f9324920640 1 --2- 172.21.15.114:0/3205085398 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f9320101130 0x7f932019e410 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:41:49.279 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:41:49.278+0000 7f9324920640 1 -- 172.21.15.114:0/3205085398 >> 172.21.15.114:0/3205085398 conn(0x7f9320077e30 msgr2=0x7f93200fee10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T10:41:49.279 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:41:49.279+0000 7f9324920640 1 -- 172.21.15.114:0/3205085398 shutdown_connections 2024-04-23T10:41:49.279 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:41:49.279+0000 7f9324920640 1 -- 172.21.15.114:0/3205085398 wait complete. 2024-04-23T10:41:49.292 DEBUG:teuthology.orchestra.run.smithi114:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool rm cephfs_metadata cephfs_metadata --yes-i-really-really-mean-it 2024-04-23T10:41:49.402 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:41:49.401+0000 7f9cab2cf640 1 -- 172.21.15.114:0/1250749141 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f9ca4100d20 msgr2=0x7f9ca4101100 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T10:41:49.403 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:41:49.401+0000 7f9cab2cf640 1 --2- 172.21.15.114:0/1250749141 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f9ca4100d20 0x7f9ca4101100 secure :-1 s=READY pgs=986 cs=0 l=1 rev1=1 crypto rx=0x7f9ca0009900 tx=0x7f9ca002a410 comp rx=0 tx=0).stop 2024-04-23T10:41:49.403 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:41:49.401+0000 7f9cab2cf640 1 -- 172.21.15.114:0/1250749141 shutdown_connections 2024-04-23T10:41:49.403 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:41:49.401+0000 7f9cab2cf640 1 --2- 172.21.15.114:0/1250749141 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f9ca410c4f0 0x7f9ca410e9b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:41:49.403 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:41:49.401+0000 7f9cab2cf640 1 --2- 172.21.15.114:0/1250749141 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f9ca41016d0 0x7f9ca410be30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:41:49.403 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:41:49.401+0000 7f9cab2cf640 1 --2- 172.21.15.114:0/1250749141 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f9ca4100d20 0x7f9ca4101100 unknown :-1 s=CLOSED pgs=986 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:41:49.403 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:41:49.401+0000 7f9cab2cf640 1 -- 172.21.15.114:0/1250749141 >> 172.21.15.114:0/1250749141 conn(0x7f9ca4076150 msgr2=0x7f9ca4076560 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T10:41:49.404 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:41:49.401+0000 7f9cab2cf640 1 -- 172.21.15.114:0/1250749141 shutdown_connections 2024-04-23T10:41:49.404 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:41:49.401+0000 7f9cab2cf640 1 -- 172.21.15.114:0/1250749141 wait complete. 2024-04-23T10:41:49.404 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:41:49.402+0000 7f9cab2cf640 1 Processor -- start 2024-04-23T10:41:49.404 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:41:49.402+0000 7f9cab2cf640 1 -- start start 2024-04-23T10:41:49.404 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:41:49.402+0000 7f9cab2cf640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f9ca4100d20 0x7f9ca4101c90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T10:41:49.404 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:41:49.402+0000 7f9cab2cf640 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f9ca410c4f0 0x7f9ca41021d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T10:41:49.404 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:41:49.402+0000 7f9cab2cf640 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f9ca4102710 0x7f9ca41a6850 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T10:41:49.404 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:41:49.402+0000 7f9cab2cf640 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x7f9ca41121c0 con 0x7f9ca4100d20 2024-04-23T10:41:49.405 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:41:49.402+0000 7f9cab2cf640 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 -- 0x7f9ca4112040 con 0x7f9ca410c4f0 2024-04-23T10:41:49.405 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:41:49.402+0000 7f9cab2cf640 1 -- --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_getmap magic: 0 -- 0x7f9ca4112340 con 0x7f9ca4102710 2024-04-23T10:41:49.405 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:41:49.402+0000 7f9ca906b640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f9ca4100d20 0x7f9ca4101c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T10:41:49.405 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:41:49.402+0000 7f9ca906b640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f9ca4100d20 0x7f9ca4101c90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.114:53990/0 (socket says 172.21.15.114:53990) 2024-04-23T10:41:49.405 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:41:49.402+0000 7f9ca906b640 1 -- 172.21.15.114:0/293849017 learned_addr learned my addr 172.21.15.114:0/293849017 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-04-23T10:41:49.405 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:41:49.402+0000 7f9ca886a640 1 --2- 172.21.15.114:0/293849017 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f9ca410c4f0 0x7f9ca41021d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T10:41:49.405 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:41:49.402+0000 7f9ca986c640 1 --2- 172.21.15.114:0/293849017 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f9ca4102710 0x7f9ca41a6850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T10:41:49.405 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:41:49.402+0000 7f9ca906b640 1 -- 172.21.15.114:0/293849017 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f9ca4102710 msgr2=0x7f9ca41a6850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T10:41:49.406 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:41:49.402+0000 7f9ca906b640 1 --2- 172.21.15.114:0/293849017 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f9ca4102710 0x7f9ca41a6850 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:41:49.406 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:41:49.402+0000 7f9ca906b640 1 -- 172.21.15.114:0/293849017 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f9ca410c4f0 msgr2=0x7f9ca41021d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T10:41:49.406 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:41:49.402+0000 7f9ca906b640 1 --2- 172.21.15.114:0/293849017 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f9ca410c4f0 0x7f9ca41021d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:41:49.406 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:41:49.402+0000 7f9ca906b640 1 -- 172.21.15.114:0/293849017 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f9ca41a6f50 con 0x7f9ca4100d20 2024-04-23T10:41:49.406 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:41:49.402+0000 7f9ca986c640 1 --2- 172.21.15.114:0/293849017 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f9ca4102710 0x7f9ca41a6850 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-23T10:41:49.406 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:41:49.402+0000 7f9ca906b640 1 --2- 172.21.15.114:0/293849017 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f9ca4100d20 0x7f9ca4101c90 secure :-1 s=READY pgs=987 cs=0 l=1 rev1=1 crypto rx=0x7f9ca0005940 tx=0x7f9ca00042f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T10:41:49.406 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:41:49.403+0000 7f9c9a7fc640 1 -- 172.21.15.114:0/293849017 <== mon.0 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9ca003f030 con 0x7f9ca4100d20 2024-04-23T10:41:49.406 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:41:49.403+0000 7f9cab2cf640 1 -- 172.21.15.114:0/293849017 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f9ca41a71e0 con 0x7f9ca4100d20 2024-04-23T10:41:49.407 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:41:49.403+0000 7f9c9a7fc640 1 -- 172.21.15.114:0/293849017 <== mon.0 v2:172.21.15.114:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f9ca0003cb0 con 0x7f9ca4100d20 2024-04-23T10:41:49.407 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:41:49.403+0000 7f9c9a7fc640 1 -- 172.21.15.114:0/293849017 <== mon.0 v2:172.21.15.114:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9ca0035070 con 0x7f9ca4100d20 2024-04-23T10:41:49.407 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:41:49.403+0000 7f9cab2cf640 1 -- 172.21.15.114:0/293849017 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f9ca41a7720 con 0x7f9ca4100d20 2024-04-23T10:41:49.407 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:41:49.404+0000 7f9cab2cf640 1 -- 172.21.15.114:0/293849017 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f9ca40691c0 con 0x7f9ca4100d20 2024-04-23T10:41:49.408 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:41:49.405+0000 7f9c9a7fc640 1 -- 172.21.15.114:0/293849017 <== mon.0 v2:172.21.15.114:3300/0 4 ==== mgrmap(e 26) ==== 138778+0+0 (secure 0 0 0) 0x7f9ca0042030 con 0x7f9ca4100d20 2024-04-23T10:41:49.408 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:41:49.405+0000 7f9c9a7fc640 1 --2- 172.21.15.114:0/293849017 >> [v2:172.21.15.114:6800/1395693466,v1:172.21.15.114:6801/1395693466] conn(0x7f9c900a3d50 0x7f9c900a6210 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T10:41:49.408 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:41:49.405+0000 7f9ca886a640 1 --2- 172.21.15.114:0/293849017 >> [v2:172.21.15.114:6800/1395693466,v1:172.21.15.114:6801/1395693466] conn(0x7f9c900a3d50 0x7f9c900a6210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T10:41:49.408 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:41:49.405+0000 7f9c9a7fc640 1 -- 172.21.15.114:0/293849017 <== mon.0 v2:172.21.15.114:3300/0 5 ==== osd_map(114..114 src has 1..114) ==== 8843+0+0 (secure 0 0 0) 0x7f9ca00f0ec0 con 0x7f9ca4100d20 2024-04-23T10:41:49.408 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:41:49.406+0000 7f9ca886a640 1 --2- 172.21.15.114:0/293849017 >> [v2:172.21.15.114:6800/1395693466,v1:172.21.15.114:6801/1395693466] conn(0x7f9c900a3d50 0x7f9c900a6210 secure :-1 s=READY pgs=194 cs=0 l=1 rev1=1 crypto rx=0x7f9c94005e10 tx=0x7f9c94005da0 comp rx=0 tx=0).ready entity=mgr.15291 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T10:41:49.409 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:41:49.410+0000 7f9c9a7fc640 1 -- 172.21.15.114:0/293849017 <== mon.0 v2:172.21.15.114:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f9ca00bd420 con 0x7f9ca4100d20 2024-04-23T10:41:49.588 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:41:49.590+0000 7f9cab2cf640 1 -- 172.21.15.114:0/293849017 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_command({"prefix": "osd pool rm", "pool": "cephfs_metadata", "pool2": "cephfs_metadata", "yes_i_really_really_mean_it": true} v 0) -- 0x7f9ca41074f0 con 0x7f9ca4100d20 2024-04-23T10:41:49.992 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 23 10:41:49 smithi119 ceph-mon[32608]: pgmap v481: 97 pgs: 97 active+clean; 552 MiB data, 4.3 GiB used, 1.0 TiB / 1.0 TiB avail; 120 KiB/s rd, 133 KiB/s wr, 15 op/s 2024-04-23T10:41:49.992 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 23 10:41:49 smithi119 ceph-mon[32608]: from='client.? 172.21.15.114:0/3205085398' entity='client.admin' cmd=[{"prefix": "fs rm", "fs_name": "cephfs", "yes_i_really_mean_it": true}]: dispatch 2024-04-23T10:41:49.992 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 23 10:41:49 smithi119 ceph-mon[32608]: Health check cleared: FS_DEGRADED (was: 1 filesystem is degraded) 2024-04-23T10:41:49.993 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 23 10:41:49 smithi119 ceph-mon[32608]: Health check cleared: MDS_ALL_DOWN (was: 1 filesystem is offline) 2024-04-23T10:41:49.993 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 23 10:41:49 smithi119 ceph-mon[32608]: Cluster is now healthy 2024-04-23T10:41:49.993 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 23 10:41:49 smithi119 ceph-mon[32608]: from='client.? 172.21.15.114:0/3205085398' entity='client.admin' cmd='[{"prefix": "fs rm", "fs_name": "cephfs", "yes_i_really_mean_it": true}]': finished 2024-04-23T10:41:49.993 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 23 10:41:49 smithi119 ceph-mon[32608]: mds.? [v2:172.21.15.204:6836/3661929443,v1:172.21.15.204:6838/3661929443] up:boot 2024-04-23T10:41:49.993 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 23 10:41:49 smithi119 ceph-mon[32608]: mds.? [v2:172.21.15.204:6837/600984176,v1:172.21.15.204:6839/600984176] up:boot 2024-04-23T10:41:49.993 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 23 10:41:49 smithi119 ceph-mon[32608]: mds.? [v2:172.21.15.119:6832/2586077425,v1:172.21.15.119:6833/2586077425] up:boot 2024-04-23T10:41:49.993 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 23 10:41:49 smithi119 ceph-mon[32608]: mds.? [v2:172.21.15.119:6834/2741130544,v1:172.21.15.119:6835/2741130544] up:boot 2024-04-23T10:41:49.993 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 23 10:41:49 smithi119 ceph-mon[32608]: mds.? [v2:172.21.15.119:6836/2436374018,v1:172.21.15.119:6837/2436374018] up:boot 2024-04-23T10:41:49.994 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 23 10:41:49 smithi119 ceph-mon[32608]: mds.? [v2:172.21.15.119:6838/2553879171,v1:172.21.15.119:6839/2553879171] up:boot 2024-04-23T10:41:49.994 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 23 10:41:49 smithi119 ceph-mon[32608]: fsmap 12 up:standby 2024-04-23T10:41:49.994 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 23 10:41:49 smithi119 ceph-mon[32608]: from='mgr.15291 172.21.15.114:0/1095789187' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "h"}]: dispatch 2024-04-23T10:41:49.994 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 23 10:41:49 smithi119 ceph-mon[32608]: from='mgr.15291 172.21.15.114:0/1095789187' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "c"}]: dispatch 2024-04-23T10:41:49.994 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 23 10:41:49 smithi119 ceph-mon[32608]: from='mgr.15291 172.21.15.114:0/1095789187' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "i"}]: dispatch 2024-04-23T10:41:49.994 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 23 10:41:49 smithi119 ceph-mon[32608]: from='mgr.15291 172.21.15.114:0/1095789187' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "k"}]: dispatch 2024-04-23T10:41:49.994 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 23 10:41:49 smithi119 ceph-mon[32608]: from='mgr.15291 172.21.15.114:0/1095789187' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "b"}]: dispatch 2024-04-23T10:41:49.994 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 23 10:41:49 smithi119 ceph-mon[32608]: from='mgr.15291 172.21.15.114:0/1095789187' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "e"}]: dispatch 2024-04-23T10:41:50.004 INFO:journalctl@ceph.mon.a.smithi114.stdout:Apr 23 10:41:49 smithi114 ceph-mon[29255]: pgmap v481: 97 pgs: 97 active+clean; 552 MiB data, 4.3 GiB used, 1.0 TiB / 1.0 TiB avail; 120 KiB/s rd, 133 KiB/s wr, 15 op/s 2024-04-23T10:41:50.004 INFO:journalctl@ceph.mon.a.smithi114.stdout:Apr 23 10:41:49 smithi114 ceph-mon[29255]: from='client.? 172.21.15.114:0/3205085398' entity='client.admin' cmd=[{"prefix": "fs rm", "fs_name": "cephfs", "yes_i_really_mean_it": true}]: dispatch 2024-04-23T10:41:50.004 INFO:journalctl@ceph.mon.a.smithi114.stdout:Apr 23 10:41:49 smithi114 ceph-mon[29255]: Health check cleared: FS_DEGRADED (was: 1 filesystem is degraded) 2024-04-23T10:41:50.004 INFO:journalctl@ceph.mon.a.smithi114.stdout:Apr 23 10:41:49 smithi114 ceph-mon[29255]: Health check cleared: MDS_ALL_DOWN (was: 1 filesystem is offline) 2024-04-23T10:41:50.004 INFO:journalctl@ceph.mon.a.smithi114.stdout:Apr 23 10:41:49 smithi114 ceph-mon[29255]: Cluster is now healthy 2024-04-23T10:41:50.004 INFO:journalctl@ceph.mon.a.smithi114.stdout:Apr 23 10:41:49 smithi114 ceph-mon[29255]: from='client.? 172.21.15.114:0/3205085398' entity='client.admin' cmd='[{"prefix": "fs rm", "fs_name": "cephfs", "yes_i_really_mean_it": true}]': finished 2024-04-23T10:41:50.005 INFO:journalctl@ceph.mon.a.smithi114.stdout:Apr 23 10:41:49 smithi114 ceph-mon[29255]: mds.? [v2:172.21.15.204:6836/3661929443,v1:172.21.15.204:6838/3661929443] up:boot 2024-04-23T10:41:50.005 INFO:journalctl@ceph.mon.a.smithi114.stdout:Apr 23 10:41:49 smithi114 ceph-mon[29255]: mds.? [v2:172.21.15.204:6837/600984176,v1:172.21.15.204:6839/600984176] up:boot 2024-04-23T10:41:50.005 INFO:journalctl@ceph.mon.a.smithi114.stdout:Apr 23 10:41:49 smithi114 ceph-mon[29255]: mds.? [v2:172.21.15.119:6832/2586077425,v1:172.21.15.119:6833/2586077425] up:boot 2024-04-23T10:41:50.005 INFO:journalctl@ceph.mon.a.smithi114.stdout:Apr 23 10:41:49 smithi114 ceph-mon[29255]: mds.? [v2:172.21.15.119:6834/2741130544,v1:172.21.15.119:6835/2741130544] up:boot 2024-04-23T10:41:50.005 INFO:journalctl@ceph.mon.a.smithi114.stdout:Apr 23 10:41:49 smithi114 ceph-mon[29255]: mds.? [v2:172.21.15.119:6836/2436374018,v1:172.21.15.119:6837/2436374018] up:boot 2024-04-23T10:41:50.005 INFO:journalctl@ceph.mon.a.smithi114.stdout:Apr 23 10:41:49 smithi114 ceph-mon[29255]: mds.? [v2:172.21.15.119:6838/2553879171,v1:172.21.15.119:6839/2553879171] up:boot 2024-04-23T10:41:50.005 INFO:journalctl@ceph.mon.a.smithi114.stdout:Apr 23 10:41:49 smithi114 ceph-mon[29255]: fsmap 12 up:standby 2024-04-23T10:41:50.005 INFO:journalctl@ceph.mon.a.smithi114.stdout:Apr 23 10:41:49 smithi114 ceph-mon[29255]: from='mgr.15291 172.21.15.114:0/1095789187' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "h"}]: dispatch 2024-04-23T10:41:50.005 INFO:journalctl@ceph.mon.a.smithi114.stdout:Apr 23 10:41:49 smithi114 ceph-mon[29255]: from='mgr.15291 172.21.15.114:0/1095789187' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "c"}]: dispatch 2024-04-23T10:41:50.006 INFO:journalctl@ceph.mon.a.smithi114.stdout:Apr 23 10:41:49 smithi114 ceph-mon[29255]: from='mgr.15291 172.21.15.114:0/1095789187' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "i"}]: dispatch 2024-04-23T10:41:50.006 INFO:journalctl@ceph.mon.a.smithi114.stdout:Apr 23 10:41:49 smithi114 ceph-mon[29255]: from='mgr.15291 172.21.15.114:0/1095789187' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "k"}]: dispatch 2024-04-23T10:41:50.006 INFO:journalctl@ceph.mon.a.smithi114.stdout:Apr 23 10:41:49 smithi114 ceph-mon[29255]: from='mgr.15291 172.21.15.114:0/1095789187' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "b"}]: dispatch 2024-04-23T10:41:50.006 INFO:journalctl@ceph.mon.a.smithi114.stdout:Apr 23 10:41:49 smithi114 ceph-mon[29255]: from='mgr.15291 172.21.15.114:0/1095789187' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "e"}]: dispatch 2024-04-23T10:41:50.078 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 23 10:41:49 smithi204 ceph-mon[32649]: pgmap v481: 97 pgs: 97 active+clean; 552 MiB data, 4.3 GiB used, 1.0 TiB / 1.0 TiB avail; 120 KiB/s rd, 133 KiB/s wr, 15 op/s 2024-04-23T10:41:50.078 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 23 10:41:49 smithi204 ceph-mon[32649]: from='client.? 172.21.15.114:0/3205085398' entity='client.admin' cmd=[{"prefix": "fs rm", "fs_name": "cephfs", "yes_i_really_mean_it": true}]: dispatch 2024-04-23T10:41:50.078 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 23 10:41:49 smithi204 ceph-mon[32649]: Health check cleared: FS_DEGRADED (was: 1 filesystem is degraded) 2024-04-23T10:41:50.078 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 23 10:41:49 smithi204 ceph-mon[32649]: Health check cleared: MDS_ALL_DOWN (was: 1 filesystem is offline) 2024-04-23T10:41:50.078 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 23 10:41:49 smithi204 ceph-mon[32649]: Cluster is now healthy 2024-04-23T10:41:50.078 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 23 10:41:49 smithi204 ceph-mon[32649]: from='client.? 172.21.15.114:0/3205085398' entity='client.admin' cmd='[{"prefix": "fs rm", "fs_name": "cephfs", "yes_i_really_mean_it": true}]': finished 2024-04-23T10:41:50.079 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 23 10:41:49 smithi204 ceph-mon[32649]: mds.? [v2:172.21.15.204:6836/3661929443,v1:172.21.15.204:6838/3661929443] up:boot 2024-04-23T10:41:50.079 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 23 10:41:49 smithi204 ceph-mon[32649]: mds.? [v2:172.21.15.204:6837/600984176,v1:172.21.15.204:6839/600984176] up:boot 2024-04-23T10:41:50.079 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 23 10:41:49 smithi204 ceph-mon[32649]: mds.? [v2:172.21.15.119:6832/2586077425,v1:172.21.15.119:6833/2586077425] up:boot 2024-04-23T10:41:50.079 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 23 10:41:49 smithi204 ceph-mon[32649]: mds.? [v2:172.21.15.119:6834/2741130544,v1:172.21.15.119:6835/2741130544] up:boot 2024-04-23T10:41:50.079 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 23 10:41:49 smithi204 ceph-mon[32649]: mds.? [v2:172.21.15.119:6836/2436374018,v1:172.21.15.119:6837/2436374018] up:boot 2024-04-23T10:41:50.079 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 23 10:41:49 smithi204 ceph-mon[32649]: mds.? [v2:172.21.15.119:6838/2553879171,v1:172.21.15.119:6839/2553879171] up:boot 2024-04-23T10:41:50.079 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 23 10:41:49 smithi204 ceph-mon[32649]: fsmap 12 up:standby 2024-04-23T10:41:50.079 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 23 10:41:49 smithi204 ceph-mon[32649]: from='mgr.15291 172.21.15.114:0/1095789187' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "h"}]: dispatch 2024-04-23T10:41:50.079 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 23 10:41:49 smithi204 ceph-mon[32649]: from='mgr.15291 172.21.15.114:0/1095789187' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "c"}]: dispatch 2024-04-23T10:41:50.080 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 23 10:41:49 smithi204 ceph-mon[32649]: from='mgr.15291 172.21.15.114:0/1095789187' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "i"}]: dispatch 2024-04-23T10:41:50.080 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 23 10:41:49 smithi204 ceph-mon[32649]: from='mgr.15291 172.21.15.114:0/1095789187' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "k"}]: dispatch 2024-04-23T10:41:50.080 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 23 10:41:49 smithi204 ceph-mon[32649]: from='mgr.15291 172.21.15.114:0/1095789187' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "b"}]: dispatch 2024-04-23T10:41:50.080 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 23 10:41:49 smithi204 ceph-mon[32649]: from='mgr.15291 172.21.15.114:0/1095789187' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "e"}]: dispatch 2024-04-23T10:41:50.592 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:41:50.593+0000 7f9c9a7fc640 1 -- 172.21.15.114:0/293849017 <== mon.0 v2:172.21.15.114:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool rm", "pool": "cephfs_metadata", "pool2": "cephfs_metadata", "yes_i_really_really_mean_it": true}]=0 pool 'cephfs_metadata' removed v115) ==== 181+0+0 (secure 0 0 0) 0x7f9ca00bcd20 con 0x7f9ca4100d20 2024-04-23T10:41:50.592 INFO:teuthology.orchestra.run.smithi114.stderr:pool 'cephfs_metadata' removed 2024-04-23T10:41:50.596 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:41:50.597+0000 7f9cab2cf640 1 -- 172.21.15.114:0/293849017 >> [v2:172.21.15.114:6800/1395693466,v1:172.21.15.114:6801/1395693466] conn(0x7f9c900a3d50 msgr2=0x7f9c900a6210 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T10:41:50.596 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:41:50.597+0000 7f9cab2cf640 1 --2- 172.21.15.114:0/293849017 >> [v2:172.21.15.114:6800/1395693466,v1:172.21.15.114:6801/1395693466] conn(0x7f9c900a3d50 0x7f9c900a6210 secure :-1 s=READY pgs=194 cs=0 l=1 rev1=1 crypto rx=0x7f9c94005e10 tx=0x7f9c94005da0 comp rx=0 tx=0).stop 2024-04-23T10:41:50.597 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:41:50.597+0000 7f9cab2cf640 1 -- 172.21.15.114:0/293849017 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f9ca4100d20 msgr2=0x7f9ca4101c90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T10:41:50.597 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:41:50.597+0000 7f9cab2cf640 1 --2- 172.21.15.114:0/293849017 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f9ca4100d20 0x7f9ca4101c90 secure :-1 s=READY pgs=987 cs=0 l=1 rev1=1 crypto rx=0x7f9ca0005940 tx=0x7f9ca00042f0 comp rx=0 tx=0).stop 2024-04-23T10:41:50.597 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:41:50.597+0000 7f9cab2cf640 1 -- 172.21.15.114:0/293849017 shutdown_connections 2024-04-23T10:41:50.597 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:41:50.597+0000 7f9cab2cf640 1 --2- 172.21.15.114:0/293849017 >> [v2:172.21.15.114:6800/1395693466,v1:172.21.15.114:6801/1395693466] conn(0x7f9c900a3d50 0x7f9c900a6210 unknown :-1 s=CLOSED pgs=194 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:41:50.597 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:41:50.597+0000 7f9cab2cf640 1 --2- 172.21.15.114:0/293849017 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f9ca4102710 0x7f9ca41a6850 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:41:50.597 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:41:50.597+0000 7f9cab2cf640 1 --2- 172.21.15.114:0/293849017 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f9ca410c4f0 0x7f9ca41021d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:41:50.597 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:41:50.597+0000 7f9cab2cf640 1 --2- 172.21.15.114:0/293849017 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f9ca4100d20 0x7f9ca4101c90 unknown :-1 s=CLOSED pgs=987 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:41:50.598 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:41:50.597+0000 7f9cab2cf640 1 -- 172.21.15.114:0/293849017 >> 172.21.15.114:0/293849017 conn(0x7f9ca4076150 msgr2=0x7f9ca40fdc30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T10:41:50.598 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:41:50.597+0000 7f9cab2cf640 1 -- 172.21.15.114:0/293849017 shutdown_connections 2024-04-23T10:41:50.598 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:41:50.597+0000 7f9cab2cf640 1 -- 172.21.15.114:0/293849017 wait complete. 2024-04-23T10:41:50.614 DEBUG:teuthology.orchestra.run.smithi114:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool rm cephfs_data cephfs_data --yes-i-really-really-mean-it 2024-04-23T10:41:50.725 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:41:50.723+0000 7f2f7e1bb640 1 -- 172.21.15.114:0/747451085 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f2f78106380 msgr2=0x7f2f7810df50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T10:41:50.725 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:41:50.723+0000 7f2f7e1bb640 1 --2- 172.21.15.114:0/747451085 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f2f78106380 0x7f2f7810df50 secure :-1 s=READY pgs=988 cs=0 l=1 rev1=1 crypto rx=0x7f2f70009ca0 tx=0x7f2f7002a8c0 comp rx=0 tx=0).stop 2024-04-23T10:41:50.725 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:41:50.723+0000 7f2f7e1bb640 1 -- 172.21.15.114:0/747451085 shutdown_connections 2024-04-23T10:41:50.725 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:41:50.723+0000 7f2f7e1bb640 1 --2- 172.21.15.114:0/747451085 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f2f78106380 0x7f2f7810df50 unknown :-1 s=CLOSED pgs=988 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:41:50.726 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:41:50.723+0000 7f2f7e1bb640 1 --2- 172.21.15.114:0/747451085 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f2f78102180 0x7f2f78105ca0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:41:50.726 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:41:50.723+0000 7f2f7e1bb640 1 --2- 172.21.15.114:0/747451085 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f2f781017d0 0x7f2f78101bb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:41:50.726 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:41:50.723+0000 7f2f7e1bb640 1 -- 172.21.15.114:0/747451085 >> 172.21.15.114:0/747451085 conn(0x7f2f78077410 msgr2=0x7f2f780fe760 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T10:41:50.726 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:41:50.723+0000 7f2f7e1bb640 1 -- 172.21.15.114:0/747451085 shutdown_connections 2024-04-23T10:41:50.726 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:41:50.723+0000 7f2f7e1bb640 1 -- 172.21.15.114:0/747451085 wait complete. 2024-04-23T10:41:50.727 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:41:50.723+0000 7f2f7e1bb640 1 Processor -- start 2024-04-23T10:41:50.727 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:41:50.724+0000 7f2f7e1bb640 1 -- start start 2024-04-23T10:41:50.727 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:41:50.724+0000 7f2f7e1bb640 1 --2- >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f2f781017d0 0x7f2f78199800 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T10:41:50.727 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:41:50.724+0000 7f2f7e1bb640 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f2f78106380 0x7f2f78199d40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T10:41:50.727 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:41:50.724+0000 7f2f7e1bb640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f2f7819a2d0 0x7f2f781a4880 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T10:41:50.727 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:41:50.724+0000 7f2f7e1bb640 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x7f2f781112f0 con 0x7f2f7819a2d0 2024-04-23T10:41:50.728 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:41:50.724+0000 7f2f7e1bb640 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 -- 0x7f2f78111170 con 0x7f2f78106380 2024-04-23T10:41:50.728 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:41:50.724+0000 7f2f7e1bb640 1 -- --> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] -- mon_getmap magic: 0 -- 0x7f2f78111470 con 0x7f2f781017d0 2024-04-23T10:41:50.728 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:41:50.724+0000 7f2f77fff640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f2f7819a2d0 0x7f2f781a4880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T10:41:50.728 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:41:50.724+0000 7f2f77fff640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f2f7819a2d0 0x7f2f781a4880 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.114:54012/0 (socket says 172.21.15.114:54012) 2024-04-23T10:41:50.728 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:41:50.724+0000 7f2f77fff640 1 -- 172.21.15.114:0/2996258361 learned_addr learned my addr 172.21.15.114:0/2996258361 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-04-23T10:41:50.729 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:41:50.724+0000 7f2f76ffd640 1 --2- 172.21.15.114:0/2996258361 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f2f78106380 0x7f2f78199d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T10:41:50.729 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:41:50.724+0000 7f2f777fe640 1 --2- 172.21.15.114:0/2996258361 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f2f781017d0 0x7f2f78199800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T10:41:50.729 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:41:50.724+0000 7f2f77fff640 1 -- 172.21.15.114:0/2996258361 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f2f781017d0 msgr2=0x7f2f78199800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T10:41:50.729 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:41:50.724+0000 7f2f77fff640 1 --2- 172.21.15.114:0/2996258361 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f2f781017d0 0x7f2f78199800 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:41:50.729 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:41:50.724+0000 7f2f77fff640 1 -- 172.21.15.114:0/2996258361 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f2f78106380 msgr2=0x7f2f78199d40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T10:41:50.729 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:41:50.724+0000 7f2f77fff640 1 --2- 172.21.15.114:0/2996258361 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f2f78106380 0x7f2f78199d40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:41:50.730 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:41:50.724+0000 7f2f77fff640 1 -- 172.21.15.114:0/2996258361 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f2f781a5080 con 0x7f2f7819a2d0 2024-04-23T10:41:50.730 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:41:50.725+0000 7f2f77fff640 1 --2- 172.21.15.114:0/2996258361 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f2f7819a2d0 0x7f2f781a4880 secure :-1 s=READY pgs=989 cs=0 l=1 rev1=1 crypto rx=0x7f2f700098d0 tx=0x7f2f700317b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T10:41:50.730 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:41:50.725+0000 7f2f74ff9640 1 -- 172.21.15.114:0/2996258361 <== mon.0 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2f70003b70 con 0x7f2f7819a2d0 2024-04-23T10:41:50.730 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:41:50.725+0000 7f2f74ff9640 1 -- 172.21.15.114:0/2996258361 <== mon.0 v2:172.21.15.114:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f2f70003d10 con 0x7f2f7819a2d0 2024-04-23T10:41:50.730 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:41:50.725+0000 7f2f74ff9640 1 -- 172.21.15.114:0/2996258361 <== mon.0 v2:172.21.15.114:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2f7002fa20 con 0x7f2f7819a2d0 2024-04-23T10:41:50.730 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:41:50.725+0000 7f2f7e1bb640 1 -- 172.21.15.114:0/2996258361 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f2f781a5310 con 0x7f2f7819a2d0 2024-04-23T10:41:50.731 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:41:50.725+0000 7f2f7e1bb640 1 -- 172.21.15.114:0/2996258361 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f2f781a5860 con 0x7f2f7819a2d0 2024-04-23T10:41:50.731 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:41:50.726+0000 7f2f7e1bb640 1 -- 172.21.15.114:0/2996258361 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f2f780691c0 con 0x7f2f7819a2d0 2024-04-23T10:41:50.732 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:41:50.727+0000 7f2f74ff9640 1 -- 172.21.15.114:0/2996258361 <== mon.0 v2:172.21.15.114:3300/0 4 ==== mgrmap(e 26) ==== 138778+0+0 (secure 0 0 0) 0x7f2f7002fc40 con 0x7f2f7819a2d0 2024-04-23T10:41:50.732 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:41:50.727+0000 7f2f74ff9640 1 --2- 172.21.15.114:0/2996258361 >> [v2:172.21.15.114:6800/1395693466,v1:172.21.15.114:6801/1395693466] conn(0x7f2f600a3d20 0x7f2f600a61e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-23T10:41:50.732 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:41:50.728+0000 7f2f777fe640 1 --2- 172.21.15.114:0/2996258361 >> [v2:172.21.15.114:6800/1395693466,v1:172.21.15.114:6801/1395693466] conn(0x7f2f600a3d20 0x7f2f600a61e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-23T10:41:50.733 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:41:50.728+0000 7f2f74ff9640 1 -- 172.21.15.114:0/2996258361 <== mon.0 v2:172.21.15.114:3300/0 5 ==== osd_map(115..115 src has 1..115) ==== 8289+0+0 (secure 0 0 0) 0x7f2f700f1a40 con 0x7f2f7819a2d0 2024-04-23T10:41:50.733 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:41:50.728+0000 7f2f777fe640 1 --2- 172.21.15.114:0/2996258361 >> [v2:172.21.15.114:6800/1395693466,v1:172.21.15.114:6801/1395693466] conn(0x7f2f600a3d20 0x7f2f600a61e0 secure :-1 s=READY pgs=195 cs=0 l=1 rev1=1 crypto rx=0x7f2f68009bd0 tx=0x7f2f68005f90 comp rx=0 tx=0).ready entity=mgr.15291 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-23T10:41:50.733 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:41:50.732+0000 7f2f74ff9640 1 -- 172.21.15.114:0/2996258361 <== mon.0 v2:172.21.15.114:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f2f700be2d0 con 0x7f2f7819a2d0 2024-04-23T10:41:50.913 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:41:50.914+0000 7f2f7e1bb640 1 -- 172.21.15.114:0/2996258361 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_command({"prefix": "osd pool rm", "pool": "cephfs_data", "pool2": "cephfs_data", "yes_i_really_really_mean_it": true} v 0) -- 0x7f2f7806a370 con 0x7f2f7819a2d0 2024-04-23T10:41:50.992 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 23 10:41:50 smithi119 ceph-mon[32608]: from='client.? 172.21.15.114:0/293849017' entity='client.admin' cmd=[{"prefix": "osd pool rm", "pool": "cephfs_metadata", "pool2": "cephfs_metadata", "yes_i_really_really_mean_it": true}]: dispatch 2024-04-23T10:41:51.004 INFO:journalctl@ceph.mon.a.smithi114.stdout:Apr 23 10:41:50 smithi114 ceph-mon[29255]: from='client.? 172.21.15.114:0/293849017' entity='client.admin' cmd=[{"prefix": "osd pool rm", "pool": "cephfs_metadata", "pool2": "cephfs_metadata", "yes_i_really_really_mean_it": true}]: dispatch 2024-04-23T10:41:51.077 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 23 10:41:50 smithi204 ceph-mon[32649]: from='client.? 172.21.15.114:0/293849017' entity='client.admin' cmd=[{"prefix": "osd pool rm", "pool": "cephfs_metadata", "pool2": "cephfs_metadata", "yes_i_really_really_mean_it": true}]: dispatch 2024-04-23T10:41:51.596 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:41:51.597+0000 7f2f74ff9640 1 -- 172.21.15.114:0/2996258361 <== mon.0 v2:172.21.15.114:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool rm", "pool": "cephfs_data", "pool2": "cephfs_data", "yes_i_really_really_mean_it": true}]=0 pool 'cephfs_data' removed v116) ==== 169+0+0 (secure 0 0 0) 0x7f2f700bdbd0 con 0x7f2f7819a2d0 2024-04-23T10:41:51.597 INFO:teuthology.orchestra.run.smithi114.stderr:pool 'cephfs_data' removed 2024-04-23T10:41:51.599 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:41:51.599+0000 7f2f7e1bb640 1 -- 172.21.15.114:0/2996258361 >> [v2:172.21.15.114:6800/1395693466,v1:172.21.15.114:6801/1395693466] conn(0x7f2f600a3d20 msgr2=0x7f2f600a61e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T10:41:51.599 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:41:51.599+0000 7f2f7e1bb640 1 --2- 172.21.15.114:0/2996258361 >> [v2:172.21.15.114:6800/1395693466,v1:172.21.15.114:6801/1395693466] conn(0x7f2f600a3d20 0x7f2f600a61e0 secure :-1 s=READY pgs=195 cs=0 l=1 rev1=1 crypto rx=0x7f2f68009bd0 tx=0x7f2f68005f90 comp rx=0 tx=0).stop 2024-04-23T10:41:51.599 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:41:51.599+0000 7f2f7e1bb640 1 -- 172.21.15.114:0/2996258361 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f2f7819a2d0 msgr2=0x7f2f781a4880 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-23T10:41:51.599 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:41:51.599+0000 7f2f7e1bb640 1 --2- 172.21.15.114:0/2996258361 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f2f7819a2d0 0x7f2f781a4880 secure :-1 s=READY pgs=989 cs=0 l=1 rev1=1 crypto rx=0x7f2f700098d0 tx=0x7f2f700317b0 comp rx=0 tx=0).stop 2024-04-23T10:41:51.600 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:41:51.599+0000 7f2f7e1bb640 1 -- 172.21.15.114:0/2996258361 shutdown_connections 2024-04-23T10:41:51.600 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:41:51.599+0000 7f2f7e1bb640 1 --2- 172.21.15.114:0/2996258361 >> [v2:172.21.15.114:6800/1395693466,v1:172.21.15.114:6801/1395693466] conn(0x7f2f600a3d20 0x7f2f600a61e0 unknown :-1 s=CLOSED pgs=195 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:41:51.600 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:41:51.599+0000 7f2f7e1bb640 1 --2- 172.21.15.114:0/2996258361 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f2f7819a2d0 0x7f2f781a4880 unknown :-1 s=CLOSED pgs=989 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:41:51.600 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:41:51.599+0000 7f2f7e1bb640 1 --2- 172.21.15.114:0/2996258361 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f2f78106380 0x7f2f78199d40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:41:51.600 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:41:51.599+0000 7f2f7e1bb640 1 --2- 172.21.15.114:0/2996258361 >> [v2:172.21.15.204:3300/0,v1:172.21.15.204:6789/0] conn(0x7f2f781017d0 0x7f2f78199800 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-23T10:41:51.600 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:41:51.599+0000 7f2f7e1bb640 1 -- 172.21.15.114:0/2996258361 >> 172.21.15.114:0/2996258361 conn(0x7f2f78077410 msgr2=0x7f2f78104610 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-23T10:41:51.601 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:41:51.600+0000 7f2f7e1bb640 1 -- 172.21.15.114:0/2996258361 shutdown_connections 2024-04-23T10:41:51.601 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-23T10:41:51.600+0000 7f2f7e1bb640 1 -- 172.21.15.114:0/2996258361 wait complete. 2024-04-23T10:41:51.611 DEBUG:teuthology.orchestra.run.smithi114:> sudo rm -f /etc/ceph/ceph.conf /etc/ceph/ceph.client.admin.keyring 2024-04-23T10:41:51.635 DEBUG:teuthology.orchestra.run.smithi119:> sudo rm -f /etc/ceph/ceph.conf /etc/ceph/ceph.client.admin.keyring 2024-04-23T10:41:51.663 DEBUG:teuthology.orchestra.run.smithi204:> sudo rm -f /etc/ceph/ceph.conf /etc/ceph/ceph.client.admin.keyring 2024-04-23T10:41:51.689 INFO:tasks.cephadm:Cleaning up testdir ceph.* files... 2024-04-23T10:41:51.690 DEBUG:teuthology.orchestra.run.smithi114:> rm -f /home/ubuntu/cephtest/seed.ceph.conf /home/ubuntu/cephtest/ceph.pub 2024-04-23T10:41:51.704 DEBUG:teuthology.orchestra.run.smithi119:> rm -f /home/ubuntu/cephtest/seed.ceph.conf /home/ubuntu/cephtest/ceph.pub 2024-04-23T10:41:51.767 DEBUG:teuthology.orchestra.run.smithi204:> rm -f /home/ubuntu/cephtest/seed.ceph.conf /home/ubuntu/cephtest/ceph.pub 2024-04-23T10:41:51.782 INFO:tasks.cephadm:Stopping all daemons... 2024-04-23T10:41:51.782 INFO:tasks.cephadm.mon.a:Stopping mon.a... 2024-04-23T10:41:51.782 DEBUG:teuthology.orchestra.run.smithi114:> sudo systemctl stop ceph-aa240572-0159-11ef-bc93-c7b262605968@mon.a 2024-04-23T10:41:51.991 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 23 10:41:51 smithi119 ceph-mon[32608]: pgmap v482: 97 pgs: 97 active+clean; 552 MiB data, 4.3 GiB used, 1.0 TiB / 1.0 TiB avail; 120 KiB/s rd, 133 KiB/s wr, 15 op/s 2024-04-23T10:41:51.992 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 23 10:41:51 smithi119 ceph-mon[32608]: from='client.? 172.21.15.114:0/293849017' entity='client.admin' cmd='[{"prefix": "osd pool rm", "pool": "cephfs_metadata", "pool2": "cephfs_metadata", "yes_i_really_really_mean_it": true}]': finished 2024-04-23T10:41:51.992 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 23 10:41:51 smithi119 ceph-mon[32608]: osdmap e115: 12 total, 12 up, 12 in 2024-04-23T10:41:51.992 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 23 10:41:51 smithi119 ceph-mon[32608]: from='client.? 172.21.15.114:0/2996258361' entity='client.admin' cmd=[{"prefix": "osd pool rm", "pool": "cephfs_data", "pool2": "cephfs_data", "yes_i_really_really_mean_it": true}]: dispatch 2024-04-23T10:41:52.004 INFO:journalctl@ceph.mon.a.smithi114.stdout:Apr 23 10:41:51 smithi114 ceph-mon[29255]: pgmap v482: 97 pgs: 97 active+clean; 552 MiB data, 4.3 GiB used, 1.0 TiB / 1.0 TiB avail; 120 KiB/s rd, 133 KiB/s wr, 15 op/s 2024-04-23T10:41:52.004 INFO:journalctl@ceph.mon.a.smithi114.stdout:Apr 23 10:41:51 smithi114 ceph-mon[29255]: from='client.? 172.21.15.114:0/293849017' entity='client.admin' cmd='[{"prefix": "osd pool rm", "pool": "cephfs_metadata", "pool2": "cephfs_metadata", "yes_i_really_really_mean_it": true}]': finished 2024-04-23T10:41:52.004 INFO:journalctl@ceph.mon.a.smithi114.stdout:Apr 23 10:41:51 smithi114 ceph-mon[29255]: osdmap e115: 12 total, 12 up, 12 in 2024-04-23T10:41:52.004 INFO:journalctl@ceph.mon.a.smithi114.stdout:Apr 23 10:41:51 smithi114 ceph-mon[29255]: from='client.? 172.21.15.114:0/2996258361' entity='client.admin' cmd=[{"prefix": "osd pool rm", "pool": "cephfs_data", "pool2": "cephfs_data", "yes_i_really_really_mean_it": true}]: dispatch 2024-04-23T10:41:52.004 INFO:journalctl@ceph.mon.a.smithi114.stdout:Apr 23 10:41:51 smithi114 systemd[1]: Stopping Ceph mon.a for aa240572-0159-11ef-bc93-c7b262605968... 2024-04-23T10:41:52.077 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 23 10:41:51 smithi204 ceph-mon[32649]: pgmap v482: 97 pgs: 97 active+clean; 552 MiB data, 4.3 GiB used, 1.0 TiB / 1.0 TiB avail; 120 KiB/s rd, 133 KiB/s wr, 15 op/s 2024-04-23T10:41:52.078 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 23 10:41:51 smithi204 ceph-mon[32649]: from='client.? 172.21.15.114:0/293849017' entity='client.admin' cmd='[{"prefix": "osd pool rm", "pool": "cephfs_metadata", "pool2": "cephfs_metadata", "yes_i_really_really_mean_it": true}]': finished 2024-04-23T10:41:52.078 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 23 10:41:51 smithi204 ceph-mon[32649]: osdmap e115: 12 total, 12 up, 12 in 2024-04-23T10:41:52.078 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 23 10:41:51 smithi204 ceph-mon[32649]: from='client.? 172.21.15.114:0/2996258361' entity='client.admin' cmd=[{"prefix": "osd pool rm", "pool": "cephfs_data", "pool2": "cephfs_data", "yes_i_really_really_mean_it": true}]: dispatch 2024-04-23T10:41:52.504 INFO:journalctl@ceph.mon.a.smithi114.stdout:Apr 23 10:41:52 smithi114 ceph-aa240572-0159-11ef-bc93-c7b262605968-mon-a[29251]: 2024-04-23T10:41:52.091+0000 7efe81fe4700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-mon -n mon.a -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false --default-mon-cluster-log-to-file=false --default-mon-cluster-log-to-journald=true --default-mon-cluster-log-to-stderr=false (PID: 1) UID: 0 2024-04-23T10:41:52.504 INFO:journalctl@ceph.mon.a.smithi114.stdout:Apr 23 10:41:52 smithi114 ceph-aa240572-0159-11ef-bc93-c7b262605968-mon-a[29251]: 2024-04-23T10:41:52.091+0000 7efe81fe4700 -1 mon.a@0(leader) e3 *** Got Signal Terminated *** 2024-04-23T10:41:52.504 INFO:journalctl@ceph.mon.a.smithi114.stdout:Apr 23 10:41:52 smithi114 podman[90643]: 2024-04-23 10:41:52.316292255 +0000 UTC m=+0.346102644 container died 3c977ab8b867b227b23fcd666ab2002050abf6e75f25f3bb9a7d96eb2b2e973b (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d69f13d827952bb07bb9b7973bca14f97d764d6f, name=ceph-aa240572-0159-11ef-bc93-c7b262605968-mon-a, GIT_BRANCH=HEAD, io.buildah.version=1.33.5, org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240416, GIT_CLEAN=True, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, ceph=True, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, org.label-schema.vendor=CentOS, org.label-schema.name=CentOS Stream 8 Base Image, RELEASE=wip-lusov-quiescer-d69f13d) 2024-04-23T10:41:52.821 INFO:journalctl@ceph.mon.a.smithi114.stdout:Apr 23 10:41:52 smithi114 podman[90643]: 2024-04-23 10:41:52.726233076 +0000 UTC m=+0.756043465 container cleanup 3c977ab8b867b227b23fcd666ab2002050abf6e75f25f3bb9a7d96eb2b2e973b (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d69f13d827952bb07bb9b7973bca14f97d764d6f, name=ceph-aa240572-0159-11ef-bc93-c7b262605968-mon-a, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.license=GPLv2, RELEASE=wip-lusov-quiescer-d69f13d, org.label-schema.schema-version=1.0, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, ceph=True, org.label-schema.build-date=20240416, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux , GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS, io.buildah.version=1.33.5) 2024-04-23T10:41:52.821 INFO:journalctl@ceph.mon.a.smithi114.stdout:Apr 23 10:41:52 smithi114 bash[90643]: ceph-aa240572-0159-11ef-bc93-c7b262605968-mon-a 2024-04-23T10:41:53.070 DEBUG:teuthology.orchestra.run.smithi114:> sudo pkill -f 'journalctl -f -n 0 -u ceph-aa240572-0159-11ef-bc93-c7b262605968@mon.a.service' 2024-04-23T10:41:53.101 INFO:journalctl@ceph.mon.a.smithi114.stdout:Apr 23 10:41:52 smithi114 podman[90657]: 2024-04-23 10:41:52.944328907 +0000 UTC m=+0.629387179 container remove 3c977ab8b867b227b23fcd666ab2002050abf6e75f25f3bb9a7d96eb2b2e973b (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d69f13d827952bb07bb9b7973bca14f97d764d6f, name=ceph-aa240572-0159-11ef-bc93-c7b262605968-mon-a, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, org.label-schema.schema-version=1.0, org.label-schema.build-date=20240416, GIT_BRANCH=HEAD, GIT_CLEAN=True, io.buildah.version=1.33.5, maintainer=Guillaume Abrioux , ceph=True, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, RELEASE=wip-lusov-quiescer-d69f13d, org.label-schema.name=CentOS Stream 8 Base Image) 2024-04-23T10:41:53.101 INFO:journalctl@ceph.mon.a.smithi114.stdout:Apr 23 10:41:53 smithi114 systemd[1]: ceph-aa240572-0159-11ef-bc93-c7b262605968@mon.a.service: Deactivated successfully. 2024-04-23T10:41:53.101 INFO:journalctl@ceph.mon.a.smithi114.stdout:Apr 23 10:41:53 smithi114 systemd[1]: Stopped Ceph mon.a for aa240572-0159-11ef-bc93-c7b262605968. 2024-04-23T10:41:53.102 INFO:journalctl@ceph.mon.a.smithi114.stdout:Apr 23 10:41:53 smithi114 systemd[1]: ceph-aa240572-0159-11ef-bc93-c7b262605968@mon.a.service: Consumed 24.340s CPU time. 2024-04-23T10:41:53.136 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-04-23T10:41:53.137 INFO:tasks.cephadm.mon.a:Stopped mon.a 2024-04-23T10:41:53.137 INFO:tasks.cephadm.mon.c:Stopping mon.b... 2024-04-23T10:41:53.137 DEBUG:teuthology.orchestra.run.smithi119:> sudo systemctl stop ceph-aa240572-0159-11ef-bc93-c7b262605968@mon.b 2024-04-23T10:41:53.441 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 23 10:41:53 smithi119 systemd[1]: Stopping Ceph mon.b for aa240572-0159-11ef-bc93-c7b262605968... 2024-04-23T10:41:53.742 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 23 10:41:53 smithi119 ceph-aa240572-0159-11ef-bc93-c7b262605968-mon-b[32603]: 2024-04-23T10:41:53.441+0000 7f9e61e6f700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-mon -n mon.b -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false --default-mon-cluster-log-to-file=false --default-mon-cluster-log-to-journald=true --default-mon-cluster-log-to-stderr=false (PID: 1) UID: 0 2024-04-23T10:41:53.742 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 23 10:41:53 smithi119 ceph-aa240572-0159-11ef-bc93-c7b262605968-mon-b[32603]: 2024-04-23T10:41:53.441+0000 7f9e61e6f700 -1 mon.b@2(peon) e3 *** Got Signal Terminated *** 2024-04-23T10:41:53.742 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 23 10:41:53 smithi119 podman[69169]: 2024-04-23 10:41:53.615994741 +0000 UTC m=+0.277725725 container died 903c3f3bd1b4516ae98306d6c9d5ed5e42d049cd068de3764e2881d530a5c7ad (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:068a558e3af4932cf1c60dd321c0bdf500075eb52ea1bf455e0973ba114fbfd4, name=ceph-aa240572-0159-11ef-bc93-c7b262605968-mon-b, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240416, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=, RELEASE=wip-lusov-quiescer-d69f13d, org.label-schema.license=GPLv2, io.buildah.version=1.33.5, GIT_CLEAN=True, ceph=True) 2024-04-23T10:41:54.242 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 23 10:41:53 smithi119 podman[69169]: 2024-04-23 10:41:53.984522117 +0000 UTC m=+0.646253101 container cleanup 903c3f3bd1b4516ae98306d6c9d5ed5e42d049cd068de3764e2881d530a5c7ad (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:068a558e3af4932cf1c60dd321c0bdf500075eb52ea1bf455e0973ba114fbfd4, name=ceph-aa240572-0159-11ef-bc93-c7b262605968-mon-b, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.license=GPLv2, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240416, org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS, GIT_CLEAN=True, io.buildah.version=1.33.5, CEPH_POINT_RELEASE=, ceph=True, GIT_BRANCH=HEAD, RELEASE=wip-lusov-quiescer-d69f13d) 2024-04-23T10:41:54.242 INFO:journalctl@ceph.mon.b.smithi119.stdout:Apr 23 10:41:53 smithi119 bash[69169]: ceph-aa240572-0159-11ef-bc93-c7b262605968-mon-b 2024-04-23T10:41:54.476 DEBUG:teuthology.orchestra.run.smithi119:> sudo pkill -f 'journalctl -f -n 0 -u ceph-aa240572-0159-11ef-bc93-c7b262605968@mon.b.service' 2024-04-23T10:41:54.531 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-04-23T10:41:54.532 INFO:tasks.cephadm.mon.c:Stopped mon.b 2024-04-23T10:41:54.532 INFO:tasks.cephadm.mon.c:Stopping mon.c... 2024-04-23T10:41:54.532 DEBUG:teuthology.orchestra.run.smithi204:> sudo systemctl stop ceph-aa240572-0159-11ef-bc93-c7b262605968@mon.c 2024-04-23T10:41:54.828 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 23 10:41:54 smithi204 systemd[1]: Stopping Ceph mon.c for aa240572-0159-11ef-bc93-c7b262605968... 2024-04-23T10:41:55.328 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 23 10:41:54 smithi204 ceph-aa240572-0159-11ef-bc93-c7b262605968-mon-c[32645]: 2024-04-23T10:41:54.869+0000 7f4a84f1b700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-mon -n mon.c -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false --default-mon-cluster-log-to-file=false --default-mon-cluster-log-to-journald=true --default-mon-cluster-log-to-stderr=false (PID: 1) UID: 0 2024-04-23T10:41:55.329 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 23 10:41:54 smithi204 ceph-aa240572-0159-11ef-bc93-c7b262605968-mon-c[32645]: 2024-04-23T10:41:54.869+0000 7f4a84f1b700 -1 mon.c@1(peon) e3 *** Got Signal Terminated *** 2024-04-23T10:41:55.329 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 23 10:41:55 smithi204 podman[70581]: 2024-04-23 10:41:55.095080067 +0000 UTC m=+0.359809011 container died adbe74a807a5b0af3405c83cbe2d8a0564a1166fa90cea7fd099dcf9651880f8 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:068a558e3af4932cf1c60dd321c0bdf500075eb52ea1bf455e0973ba114fbfd4, name=ceph-aa240572-0159-11ef-bc93-c7b262605968-mon-c, org.label-schema.build-date=20240416, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, io.buildah.version=1.33.5, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=wip-lusov-quiescer-d69f13d, ceph=True, GIT_CLEAN=True, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 8 Base Image, maintainer=Guillaume Abrioux ) 2024-04-23T10:41:55.645 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 23 10:41:55 smithi204 podman[70581]: 2024-04-23 10:41:55.428698882 +0000 UTC m=+0.693427827 container cleanup adbe74a807a5b0af3405c83cbe2d8a0564a1166fa90cea7fd099dcf9651880f8 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:068a558e3af4932cf1c60dd321c0bdf500075eb52ea1bf455e0973ba114fbfd4, name=ceph-aa240572-0159-11ef-bc93-c7b262605968-mon-c, org.label-schema.vendor=CentOS, io.buildah.version=1.33.5, GIT_CLEAN=True, org.label-schema.build-date=20240416, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, GIT_BRANCH=HEAD, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 8 Base Image, maintainer=Guillaume Abrioux , GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, ceph=True, RELEASE=wip-lusov-quiescer-d69f13d) 2024-04-23T10:41:55.645 INFO:journalctl@ceph.mon.c.smithi204.stdout:Apr 23 10:41:55 smithi204 bash[70581]: ceph-aa240572-0159-11ef-bc93-c7b262605968-mon-c 2024-04-23T10:41:55.867 DEBUG:teuthology.orchestra.run.smithi204:> sudo pkill -f 'journalctl -f -n 0 -u ceph-aa240572-0159-11ef-bc93-c7b262605968@mon.c.service' 2024-04-23T10:41:55.915 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-04-23T10:41:55.916 INFO:tasks.cephadm.mon.c:Stopped mon.c 2024-04-23T10:41:55.916 INFO:tasks.cephadm.mgr.x:Stopping mgr.x... 2024-04-23T10:41:55.916 DEBUG:teuthology.orchestra.run.smithi114:> sudo systemctl stop ceph-aa240572-0159-11ef-bc93-c7b262605968@mgr.x 2024-04-23T10:41:56.253 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Apr 23 10:41:55 smithi114 systemd[1]: Stopping Ceph mgr.x for aa240572-0159-11ef-bc93-c7b262605968... 2024-04-23T10:41:56.504 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Apr 23 10:41:56 smithi114 ceph-aa240572-0159-11ef-bc93-c7b262605968-mgr-x[29490]: 2024-04-23T10:41:56.253+0000 7f21c9c93700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-mgr -n mgr.x -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-04-23T10:41:56.504 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Apr 23 10:41:56 smithi114 podman[90775]: 2024-04-23 10:41:56.285273176 +0000 UTC m=+0.170828640 container died df0cd55890f902524b61e6bff467fb930ea622cca9459930fb8257a973ac26cd (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d69f13d827952bb07bb9b7973bca14f97d764d6f, name=ceph-aa240572-0159-11ef-bc93-c7b262605968-mgr-x, io.buildah.version=1.33.5, org.label-schema.name=CentOS Stream 8 Base Image, ceph=True, RELEASE=wip-lusov-quiescer-d69f13d, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, GIT_CLEAN=True, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS, org.label-schema.build-date=20240416, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD) 2024-04-23T10:41:56.937 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Apr 23 10:41:56 smithi114 podman[90775]: 2024-04-23 10:41:56.67957678 +0000 UTC m=+0.565132242 container cleanup df0cd55890f902524b61e6bff467fb930ea622cca9459930fb8257a973ac26cd (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d69f13d827952bb07bb9b7973bca14f97d764d6f, name=ceph-aa240572-0159-11ef-bc93-c7b262605968-mgr-x, GIT_CLEAN=True, RELEASE=wip-lusov-quiescer-d69f13d, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, CEPH_POINT_RELEASE=, io.buildah.version=1.33.5, org.label-schema.build-date=20240416, ceph=True, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0, org.label-schema.name=CentOS Stream 8 Base Image) 2024-04-23T10:41:56.938 INFO:journalctl@ceph.mgr.x.smithi114.stdout:Apr 23 10:41:56 smithi114 bash[90775]: ceph-aa240572-0159-11ef-bc93-c7b262605968-mgr-x 2024-04-23T10:41:57.156 DEBUG:teuthology.orchestra.run.smithi114:> sudo pkill -f 'journalctl -f -n 0 -u ceph-aa240572-0159-11ef-bc93-c7b262605968@mgr.x.service' 2024-04-23T10:41:57.193 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-04-23T10:41:57.194 INFO:tasks.cephadm.mgr.x:Stopped mgr.x 2024-04-23T10:41:57.194 INFO:tasks.cephadm.mgr.y:Stopping mgr.y... 2024-04-23T10:41:57.194 DEBUG:teuthology.orchestra.run.smithi119:> sudo systemctl stop ceph-aa240572-0159-11ef-bc93-c7b262605968@mgr.y 2024-04-23T10:41:57.492 INFO:journalctl@ceph.mgr.y.smithi119.stdout:Apr 23 10:41:57 smithi119 systemd[1]: Stopping Ceph mgr.y for aa240572-0159-11ef-bc93-c7b262605968... 2024-04-23T10:41:57.964 INFO:journalctl@ceph.mgr.y.smithi119.stdout:Apr 23 10:41:57 smithi119 podman[69291]: 2024-04-23 10:41:57.518314967 +0000 UTC m=+0.142629210 container died c8a099050ba05aa9e6840787cdb34c260b1a6299c3be20f3499f5cf3f0bc522d (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:068a558e3af4932cf1c60dd321c0bdf500075eb52ea1bf455e0973ba114fbfd4, name=ceph-aa240572-0159-11ef-bc93-c7b262605968-mgr-y, maintainer=Guillaume Abrioux , GIT_CLEAN=True, org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, io.buildah.version=1.33.5, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=wip-lusov-quiescer-d69f13d, GIT_BRANCH=HEAD, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, org.label-schema.name=CentOS Stream 8 Base Image, ceph=True, org.label-schema.build-date=20240416, CEPH_POINT_RELEASE=) 2024-04-23T10:41:58.242 INFO:journalctl@ceph.mgr.y.smithi119.stdout:Apr 23 10:41:58 smithi119 podman[69291]: 2024-04-23 10:41:58.056626821 +0000 UTC m=+0.680941104 container cleanup c8a099050ba05aa9e6840787cdb34c260b1a6299c3be20f3499f5cf3f0bc522d (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:068a558e3af4932cf1c60dd321c0bdf500075eb52ea1bf455e0973ba114fbfd4, name=ceph-aa240572-0159-11ef-bc93-c7b262605968-mgr-y, org.label-schema.build-date=20240416, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.schema-version=1.0, RELEASE=wip-lusov-quiescer-d69f13d, ceph=True, CEPH_POINT_RELEASE=, io.buildah.version=1.33.5, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_BRANCH=HEAD, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, org.label-schema.license=GPLv2, GIT_CLEAN=True) 2024-04-23T10:41:58.242 INFO:journalctl@ceph.mgr.y.smithi119.stdout:Apr 23 10:41:58 smithi119 bash[69291]: ceph-aa240572-0159-11ef-bc93-c7b262605968-mgr-y 2024-04-23T10:41:58.537 DEBUG:teuthology.orchestra.run.smithi119:> sudo pkill -f 'journalctl -f -n 0 -u ceph-aa240572-0159-11ef-bc93-c7b262605968@mgr.y.service' 2024-04-23T10:41:58.558 INFO:journalctl@ceph.mgr.y.smithi119.stdout:Apr 23 10:41:58 smithi119 podman[69305]: 2024-04-23 10:41:58.291644563 +0000 UTC m=+0.764905429 container remove c8a099050ba05aa9e6840787cdb34c260b1a6299c3be20f3499f5cf3f0bc522d (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:068a558e3af4932cf1c60dd321c0bdf500075eb52ea1bf455e0973ba114fbfd4, name=ceph-aa240572-0159-11ef-bc93-c7b262605968-mgr-y, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.33.5, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.schema-version=1.0, GIT_CLEAN=True, org.label-schema.vendor=CentOS, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, ceph=True, org.label-schema.build-date=20240416, RELEASE=wip-lusov-quiescer-d69f13d, GIT_BRANCH=HEAD, org.label-schema.license=GPLv2) 2024-04-23T10:41:58.558 INFO:journalctl@ceph.mgr.y.smithi119.stdout:Apr 23 10:41:58 smithi119 systemd[1]: ceph-aa240572-0159-11ef-bc93-c7b262605968@mgr.y.service: Main process exited, code=exited, status=143/n/a 2024-04-23T10:41:58.558 INFO:journalctl@ceph.mgr.y.smithi119.stdout:Apr 23 10:41:58 smithi119 systemd[1]: ceph-aa240572-0159-11ef-bc93-c7b262605968@mgr.y.service: Failed with result 'exit-code'. 2024-04-23T10:41:58.558 INFO:journalctl@ceph.mgr.y.smithi119.stdout:Apr 23 10:41:58 smithi119 systemd[1]: Stopped Ceph mgr.y for aa240572-0159-11ef-bc93-c7b262605968. 2024-04-23T10:41:58.559 INFO:journalctl@ceph.mgr.y.smithi119.stdout:Apr 23 10:41:58 smithi119 systemd[1]: ceph-aa240572-0159-11ef-bc93-c7b262605968@mgr.y.service: Consumed 20.646s CPU time. 2024-04-23T10:41:58.571 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-04-23T10:41:58.571 INFO:tasks.cephadm.mgr.y:Stopped mgr.y 2024-04-23T10:41:58.572 INFO:tasks.cephadm.mgr.z:Stopping mgr.z... 2024-04-23T10:41:58.572 DEBUG:teuthology.orchestra.run.smithi204:> sudo systemctl stop ceph-aa240572-0159-11ef-bc93-c7b262605968@mgr.z 2024-04-23T10:41:58.902 INFO:journalctl@ceph.mgr.z.smithi204.stdout:Apr 23 10:41:58 smithi204 systemd[1]: Stopping Ceph mgr.z for aa240572-0159-11ef-bc93-c7b262605968... 2024-04-23T10:41:59.214 INFO:journalctl@ceph.mgr.z.smithi204.stdout:Apr 23 10:41:58 smithi204 podman[70704]: 2024-04-23 10:41:58.904389594 +0000 UTC m=+0.136710364 container died 8a04d6bd3ee9040467b0c927f8cfdfa384601c95813ea4b16200a29e028e95c2 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:068a558e3af4932cf1c60dd321c0bdf500075eb52ea1bf455e0973ba114fbfd4, name=ceph-aa240572-0159-11ef-bc93-c7b262605968-mgr-z, org.label-schema.license=GPLv2, io.buildah.version=1.33.5, org.label-schema.build-date=20240416, maintainer=Guillaume Abrioux , ceph=True, org.label-schema.vendor=CentOS, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, GIT_BRANCH=HEAD, GIT_CLEAN=True, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 8 Base Image, RELEASE=wip-lusov-quiescer-d69f13d, org.label-schema.schema-version=1.0) 2024-04-23T10:41:59.534 INFO:journalctl@ceph.mgr.z.smithi204.stdout:Apr 23 10:41:59 smithi204 podman[70704]: 2024-04-23 10:41:59.216823967 +0000 UTC m=+0.449144727 container cleanup 8a04d6bd3ee9040467b0c927f8cfdfa384601c95813ea4b16200a29e028e95c2 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:068a558e3af4932cf1c60dd321c0bdf500075eb52ea1bf455e0973ba114fbfd4, name=ceph-aa240572-0159-11ef-bc93-c7b262605968-mgr-z, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 8 Base Image, ceph=True, org.label-schema.license=GPLv2, io.buildah.version=1.33.5, RELEASE=wip-lusov-quiescer-d69f13d, CEPH_POINT_RELEASE=, org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, org.label-schema.build-date=20240416, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD, GIT_CLEAN=True) 2024-04-23T10:41:59.534 INFO:journalctl@ceph.mgr.z.smithi204.stdout:Apr 23 10:41:59 smithi204 bash[70704]: ceph-aa240572-0159-11ef-bc93-c7b262605968-mgr-z 2024-04-23T10:41:59.534 INFO:journalctl@ceph.mgr.z.smithi204.stdout:Apr 23 10:41:59 smithi204 podman[70718]: 2024-04-23 10:41:59.443115262 +0000 UTC m=+0.533351446 container remove 8a04d6bd3ee9040467b0c927f8cfdfa384601c95813ea4b16200a29e028e95c2 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:068a558e3af4932cf1c60dd321c0bdf500075eb52ea1bf455e0973ba114fbfd4, name=ceph-aa240572-0159-11ef-bc93-c7b262605968-mgr-z, org.label-schema.build-date=20240416, org.label-schema.vendor=CentOS, GIT_CLEAN=True, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, org.label-schema.name=CentOS Stream 8 Base Image, CEPH_POINT_RELEASE=, org.label-schema.schema-version=1.0, ceph=True, org.label-schema.license=GPLv2, RELEASE=wip-lusov-quiescer-d69f13d, io.buildah.version=1.33.5, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux ) 2024-04-23T10:41:59.534 INFO:journalctl@ceph.mgr.z.smithi204.stdout:Apr 23 10:41:59 smithi204 systemd[1]: ceph-aa240572-0159-11ef-bc93-c7b262605968@mgr.z.service: Main process exited, code=exited, status=143/n/a 2024-04-23T10:41:59.553 DEBUG:teuthology.orchestra.run.smithi204:> sudo pkill -f 'journalctl -f -n 0 -u ceph-aa240572-0159-11ef-bc93-c7b262605968@mgr.z.service' 2024-04-23T10:41:59.587 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-04-23T10:41:59.587 INFO:tasks.cephadm.mgr.z:Stopped mgr.z 2024-04-23T10:41:59.587 INFO:tasks.cephadm.osd.0:Stopping osd.0... 2024-04-23T10:41:59.588 DEBUG:teuthology.orchestra.run.smithi114:> sudo systemctl stop ceph-aa240572-0159-11ef-bc93-c7b262605968@osd.0 2024-04-23T10:41:59.958 INFO:journalctl@ceph.osd.0.smithi114.stdout:Apr 23 10:41:59 smithi114 systemd[1]: Stopping Ceph osd.0 for aa240572-0159-11ef-bc93-c7b262605968... 2024-04-23T10:42:00.254 INFO:journalctl@ceph.osd.0.smithi114.stdout:Apr 23 10:41:59 smithi114 ceph-aa240572-0159-11ef-bc93-c7b262605968-osd-0[41269]: 2024-04-23T10:41:59.958+0000 7f5e81986700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.0 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-04-23T10:42:00.254 INFO:journalctl@ceph.osd.0.smithi114.stdout:Apr 23 10:41:59 smithi114 ceph-aa240572-0159-11ef-bc93-c7b262605968-osd-0[41269]: 2024-04-23T10:41:59.958+0000 7f5e81986700 -1 osd.0 116 *** Got signal Terminated *** 2024-04-23T10:42:00.254 INFO:journalctl@ceph.osd.0.smithi114.stdout:Apr 23 10:41:59 smithi114 ceph-aa240572-0159-11ef-bc93-c7b262605968-osd-0[41269]: 2024-04-23T10:41:59.958+0000 7f5e81986700 -1 osd.0 116 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-04-23T10:42:05.674 INFO:journalctl@ceph.osd.0.smithi114.stdout:Apr 23 10:42:05 smithi114 podman[90895]: 2024-04-23 10:42:05.404365634 +0000 UTC m=+5.627881272 container died 69855b5983dec8107f11ba9bd6f825425ab6cf07019acfedc4f314f8127788ff (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:068a558e3af4932cf1c60dd321c0bdf500075eb52ea1bf455e0973ba114fbfd4, name=ceph-aa240572-0159-11ef-bc93-c7b262605968-osd-0, org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 8 Base Image, ceph=True, GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux , RELEASE=wip-lusov-quiescer-d69f13d, GIT_CLEAN=True, io.buildah.version=1.33.5, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, org.label-schema.build-date=20240416) 2024-04-23T10:42:06.004 INFO:journalctl@ceph.osd.0.smithi114.stdout:Apr 23 10:42:05 smithi114 podman[90895]: 2024-04-23 10:42:05.77586802 +0000 UTC m=+5.999383659 container cleanup 69855b5983dec8107f11ba9bd6f825425ab6cf07019acfedc4f314f8127788ff (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:068a558e3af4932cf1c60dd321c0bdf500075eb52ea1bf455e0973ba114fbfd4, name=ceph-aa240572-0159-11ef-bc93-c7b262605968-osd-0, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, org.label-schema.vendor=CentOS, GIT_CLEAN=True, org.label-schema.build-date=20240416, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, ceph=True, io.buildah.version=1.33.5, maintainer=Guillaume Abrioux , RELEASE=wip-lusov-quiescer-d69f13d, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-04-23T10:42:06.004 INFO:journalctl@ceph.osd.0.smithi114.stdout:Apr 23 10:42:05 smithi114 bash[90895]: ceph-aa240572-0159-11ef-bc93-c7b262605968-osd-0 2024-04-23T10:42:06.392 INFO:journalctl@ceph.osd.0.smithi114.stdout:Apr 23 10:42:06 smithi114 podman[90912]: 2024-04-23 10:42:06.085625166 +0000 UTC m=+0.675708633 container remove 69855b5983dec8107f11ba9bd6f825425ab6cf07019acfedc4f314f8127788ff (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:068a558e3af4932cf1c60dd321c0bdf500075eb52ea1bf455e0973ba114fbfd4, name=ceph-aa240572-0159-11ef-bc93-c7b262605968-osd-0, RELEASE=wip-lusov-quiescer-d69f13d, GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0, ceph=True, org.label-schema.build-date=20240416, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, io.buildah.version=1.33.5, GIT_CLEAN=True, org.label-schema.name=CentOS Stream 8 Base Image, maintainer=Guillaume Abrioux ) 2024-04-23T10:42:06.695 INFO:journalctl@ceph.osd.0.smithi114.stdout:Apr 23 10:42:06 smithi114 podman[91006]: 2024-04-23 10:42:06.394667793 +0000 UTC m=+0.104272081 container create 903f82cef1f5235d703c500d7b034c8b60abb935393a6c58444a21e4a0a5debc (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:068a558e3af4932cf1c60dd321c0bdf500075eb52ea1bf455e0973ba114fbfd4, name=ceph-aa240572-0159-11ef-bc93-c7b262605968-osd-0-deactivate, ceph=True, org.label-schema.build-date=20240416, GIT_BRANCH=HEAD, org.label-schema.license=GPLv2, io.buildah.version=1.33.5, maintainer=Guillaume Abrioux , GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=wip-lusov-quiescer-d69f13d, GIT_CLEAN=True, org.label-schema.name=CentOS Stream 8 Base Image, CEPH_POINT_RELEASE=) 2024-04-23T10:42:06.696 INFO:journalctl@ceph.osd.0.smithi114.stdout:Apr 23 10:42:06 smithi114 podman[91006]: 2024-04-23 10:42:06.308455191 +0000 UTC m=+0.018059489 image pull da4b004c09e820713483d039bc76fbd660be678f2ea3c03e059a6161b74dab76 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:068a558e3af4932cf1c60dd321c0bdf500075eb52ea1bf455e0973ba114fbfd4 2024-04-23T10:42:06.696 INFO:journalctl@ceph.osd.0.smithi114.stdout:Apr 23 10:42:06 smithi114 podman[91006]: 2024-04-23 10:42:06.622645732 +0000 UTC m=+0.332250023 container init 903f82cef1f5235d703c500d7b034c8b60abb935393a6c58444a21e4a0a5debc (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:068a558e3af4932cf1c60dd321c0bdf500075eb52ea1bf455e0973ba114fbfd4, name=ceph-aa240572-0159-11ef-bc93-c7b262605968-osd-0-deactivate, GIT_BRANCH=HEAD, RELEASE=wip-lusov-quiescer-d69f13d, io.buildah.version=1.33.5, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, org.label-schema.vendor=CentOS, org.label-schema.name=CentOS Stream 8 Base Image, maintainer=Guillaume Abrioux , GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=, org.label-schema.build-date=20240416, ceph=True, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0) 2024-04-23T10:42:06.696 INFO:journalctl@ceph.osd.0.smithi114.stdout:Apr 23 10:42:06 smithi114 podman[91006]: 2024-04-23 10:42:06.626104349 +0000 UTC m=+0.335708640 container start 903f82cef1f5235d703c500d7b034c8b60abb935393a6c58444a21e4a0a5debc (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:068a558e3af4932cf1c60dd321c0bdf500075eb52ea1bf455e0973ba114fbfd4, name=ceph-aa240572-0159-11ef-bc93-c7b262605968-osd-0-deactivate, org.label-schema.license=GPLv2, maintainer=Guillaume Abrioux , GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, CEPH_POINT_RELEASE=, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=wip-lusov-quiescer-d69f13d, ceph=True, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 8 Base Image, io.buildah.version=1.33.5, GIT_CLEAN=True, org.label-schema.build-date=20240416, org.label-schema.schema-version=1.0) 2024-04-23T10:42:06.696 INFO:journalctl@ceph.osd.0.smithi114.stdout:Apr 23 10:42:06 smithi114 podman[91006]: 2024-04-23 10:42:06.696931654 +0000 UTC m=+0.406535954 container attach 903f82cef1f5235d703c500d7b034c8b60abb935393a6c58444a21e4a0a5debc (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:068a558e3af4932cf1c60dd321c0bdf500075eb52ea1bf455e0973ba114fbfd4, name=ceph-aa240572-0159-11ef-bc93-c7b262605968-osd-0-deactivate, ceph=True, org.label-schema.license=GPLv2, org.label-schema.build-date=20240416, RELEASE=wip-lusov-quiescer-d69f13d, org.label-schema.name=CentOS Stream 8 Base Image, CEPH_POINT_RELEASE=, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, org.label-schema.vendor=CentOS, io.buildah.version=1.33.5) 2024-04-23T10:42:07.004 INFO:journalctl@ceph.osd.0.smithi114.stdout:Apr 23 10:42:06 smithi114 podman[91006]: 2024-04-23 10:42:06.916096063 +0000 UTC m=+0.625700354 container died 903f82cef1f5235d703c500d7b034c8b60abb935393a6c58444a21e4a0a5debc (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:068a558e3af4932cf1c60dd321c0bdf500075eb52ea1bf455e0973ba114fbfd4, name=ceph-aa240572-0159-11ef-bc93-c7b262605968-osd-0-deactivate, GIT_CLEAN=True, ceph=True, io.buildah.version=1.33.5, org.label-schema.name=CentOS Stream 8 Base Image, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0, RELEASE=wip-lusov-quiescer-d69f13d, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, CEPH_POINT_RELEASE=, org.label-schema.build-date=20240416) 2024-04-23T10:42:07.457 INFO:journalctl@ceph.osd.0.smithi114.stdout:Apr 23 10:42:07 smithi114 podman[91033]: 2024-04-23 10:42:07.441806472 +0000 UTC m=+0.519887829 container remove 903f82cef1f5235d703c500d7b034c8b60abb935393a6c58444a21e4a0a5debc (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:068a558e3af4932cf1c60dd321c0bdf500075eb52ea1bf455e0973ba114fbfd4, name=ceph-aa240572-0159-11ef-bc93-c7b262605968-osd-0-deactivate, GIT_CLEAN=True, ceph=True, org.label-schema.schema-version=1.0, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.license=GPLv2, RELEASE=wip-lusov-quiescer-d69f13d, io.buildah.version=1.33.5, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240416, GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-04-23T10:42:07.462 DEBUG:teuthology.orchestra.run.smithi114:> sudo pkill -f 'journalctl -f -n 0 -u ceph-aa240572-0159-11ef-bc93-c7b262605968@osd.0.service' 2024-04-23T10:42:07.501 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-04-23T10:42:07.502 INFO:tasks.cephadm.osd.0:Stopped osd.0 2024-04-23T10:42:07.502 INFO:tasks.cephadm.osd.1:Stopping osd.1... 2024-04-23T10:42:07.502 DEBUG:teuthology.orchestra.run.smithi119:> sudo systemctl stop ceph-aa240572-0159-11ef-bc93-c7b262605968@osd.1 2024-04-23T10:42:07.809 INFO:journalctl@ceph.osd.1.smithi119.stdout:Apr 23 10:42:07 smithi119 systemd[1]: Stopping Ceph osd.1 for aa240572-0159-11ef-bc93-c7b262605968... 2024-04-23T10:42:08.220 INFO:journalctl@ceph.osd.1.smithi119.stdout:Apr 23 10:42:07 smithi119 ceph-aa240572-0159-11ef-bc93-c7b262605968-osd-1[37220]: 2024-04-23T10:42:07.810+0000 7fefc0918700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.1 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-04-23T10:42:08.220 INFO:journalctl@ceph.osd.1.smithi119.stdout:Apr 23 10:42:07 smithi119 ceph-aa240572-0159-11ef-bc93-c7b262605968-osd-1[37220]: 2024-04-23T10:42:07.810+0000 7fefc0918700 -1 osd.1 116 *** Got signal Terminated *** 2024-04-23T10:42:08.220 INFO:journalctl@ceph.osd.1.smithi119.stdout:Apr 23 10:42:07 smithi119 ceph-aa240572-0159-11ef-bc93-c7b262605968-osd-1[37220]: 2024-04-23T10:42:07.810+0000 7fefc0918700 -1 osd.1 116 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-04-23T10:42:13.652 INFO:journalctl@ceph.osd.1.smithi119.stdout:Apr 23 10:42:13 smithi119 podman[69424]: 2024-04-23 10:42:13.366819541 +0000 UTC m=+5.657016714 container died fb83cfbb8ef0fea61f4ae8fc60bf9c30e630aa47f43b44ce5427a5c2faaa01a0 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:068a558e3af4932cf1c60dd321c0bdf500075eb52ea1bf455e0973ba114fbfd4, name=ceph-aa240572-0159-11ef-bc93-c7b262605968-osd-1, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, GIT_CLEAN=True, RELEASE=wip-lusov-quiescer-d69f13d, io.buildah.version=1.33.5, ceph=True, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2, org.label-schema.build-date=20240416, org.label-schema.name=CentOS Stream 8 Base Image) 2024-04-23T10:42:13.985 INFO:journalctl@ceph.osd.1.smithi119.stdout:Apr 23 10:42:13 smithi119 podman[69424]: 2024-04-23 10:42:13.752501298 +0000 UTC m=+6.042698470 container cleanup fb83cfbb8ef0fea61f4ae8fc60bf9c30e630aa47f43b44ce5427a5c2faaa01a0 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:068a558e3af4932cf1c60dd321c0bdf500075eb52ea1bf455e0973ba114fbfd4, name=ceph-aa240572-0159-11ef-bc93-c7b262605968-osd-1, CEPH_POINT_RELEASE=, RELEASE=wip-lusov-quiescer-d69f13d, org.label-schema.vendor=CentOS, org.label-schema.build-date=20240416, org.label-schema.name=CentOS Stream 8 Base Image, io.buildah.version=1.33.5, GIT_CLEAN=True, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD) 2024-04-23T10:42:13.985 INFO:journalctl@ceph.osd.1.smithi119.stdout:Apr 23 10:42:13 smithi119 bash[69424]: ceph-aa240572-0159-11ef-bc93-c7b262605968-osd-1 2024-04-23T10:42:14.242 INFO:journalctl@ceph.osd.1.smithi119.stdout:Apr 23 10:42:13 smithi119 podman[69442]: 2024-04-23 10:42:13.98690142 +0000 UTC m=+0.623105109 container remove fb83cfbb8ef0fea61f4ae8fc60bf9c30e630aa47f43b44ce5427a5c2faaa01a0 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:068a558e3af4932cf1c60dd321c0bdf500075eb52ea1bf455e0973ba114fbfd4, name=ceph-aa240572-0159-11ef-bc93-c7b262605968-osd-1, org.label-schema.license=GPLv2, ceph=True, GIT_CLEAN=True, org.label-schema.build-date=20240416, io.buildah.version=1.33.5, org.label-schema.name=CentOS Stream 8 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, org.label-schema.schema-version=1.0, maintainer=Guillaume Abrioux , RELEASE=wip-lusov-quiescer-d69f13d, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD) 2024-04-23T10:42:14.597 INFO:journalctl@ceph.osd.1.smithi119.stdout:Apr 23 10:42:14 smithi119 podman[69533]: 2024-04-23 10:42:14.2234172 +0000 UTC m=+0.018049803 image pull da4b004c09e820713483d039bc76fbd660be678f2ea3c03e059a6161b74dab76 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:068a558e3af4932cf1c60dd321c0bdf500075eb52ea1bf455e0973ba114fbfd4 2024-04-23T10:42:14.598 INFO:journalctl@ceph.osd.1.smithi119.stdout:Apr 23 10:42:14 smithi119 podman[69533]: 2024-04-23 10:42:14.354511942 +0000 UTC m=+0.149144551 container create 64cb1cea7918da80faa2841d50ce6ebb3bbd4975b886aadae217e06c214cbb7a (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:068a558e3af4932cf1c60dd321c0bdf500075eb52ea1bf455e0973ba114fbfd4, name=ceph-aa240572-0159-11ef-bc93-c7b262605968-osd-1-deactivate, org.label-schema.build-date=20240416, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 8 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=, GIT_CLEAN=True, org.label-schema.license=GPLv2, RELEASE=wip-lusov-quiescer-d69f13d, org.label-schema.schema-version=1.0, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, io.buildah.version=1.33.5, org.label-schema.vendor=CentOS, ceph=True) 2024-04-23T10:42:14.940 INFO:journalctl@ceph.osd.1.smithi119.stdout:Apr 23 10:42:14 smithi119 podman[69533]: 2024-04-23 10:42:14.598788125 +0000 UTC m=+0.393420730 container init 64cb1cea7918da80faa2841d50ce6ebb3bbd4975b886aadae217e06c214cbb7a (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:068a558e3af4932cf1c60dd321c0bdf500075eb52ea1bf455e0973ba114fbfd4, name=ceph-aa240572-0159-11ef-bc93-c7b262605968-osd-1-deactivate, GIT_BRANCH=HEAD, org.label-schema.build-date=20240416, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, CEPH_POINT_RELEASE=, org.label-schema.vendor=CentOS, org.label-schema.name=CentOS Stream 8 Base Image, ceph=True, RELEASE=wip-lusov-quiescer-d69f13d, org.label-schema.schema-version=1.0, maintainer=Guillaume Abrioux , io.buildah.version=1.33.5, GIT_CLEAN=True, org.label-schema.license=GPLv2) 2024-04-23T10:42:14.941 INFO:journalctl@ceph.osd.1.smithi119.stdout:Apr 23 10:42:14 smithi119 podman[69533]: 2024-04-23 10:42:14.602591228 +0000 UTC m=+0.397223833 container start 64cb1cea7918da80faa2841d50ce6ebb3bbd4975b886aadae217e06c214cbb7a (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:068a558e3af4932cf1c60dd321c0bdf500075eb52ea1bf455e0973ba114fbfd4, name=ceph-aa240572-0159-11ef-bc93-c7b262605968-osd-1-deactivate, GIT_BRANCH=HEAD, GIT_CLEAN=True, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, org.label-schema.schema-version=1.0, maintainer=Guillaume Abrioux , io.buildah.version=1.33.5, RELEASE=wip-lusov-quiescer-d69f13d, org.label-schema.name=CentOS Stream 8 Base Image, ceph=True, org.label-schema.license=GPLv2, org.label-schema.build-date=20240416) 2024-04-23T10:42:14.941 INFO:journalctl@ceph.osd.1.smithi119.stdout:Apr 23 10:42:14 smithi119 podman[69533]: 2024-04-23 10:42:14.707326881 +0000 UTC m=+0.501959502 container attach 64cb1cea7918da80faa2841d50ce6ebb3bbd4975b886aadae217e06c214cbb7a (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:068a558e3af4932cf1c60dd321c0bdf500075eb52ea1bf455e0973ba114fbfd4, name=ceph-aa240572-0159-11ef-bc93-c7b262605968-osd-1-deactivate, ceph=True, GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux , GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, org.label-schema.build-date=20240416, org.label-schema.vendor=CentOS, RELEASE=wip-lusov-quiescer-d69f13d, GIT_BRANCH=HEAD, GIT_CLEAN=True, org.label-schema.name=CentOS Stream 8 Base Image, io.buildah.version=1.33.5, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0) 2024-04-23T10:42:15.241 INFO:journalctl@ceph.osd.1.smithi119.stdout:Apr 23 10:42:14 smithi119 podman[69533]: 2024-04-23 10:42:14.942592959 +0000 UTC m=+0.737225563 container died 64cb1cea7918da80faa2841d50ce6ebb3bbd4975b886aadae217e06c214cbb7a (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:068a558e3af4932cf1c60dd321c0bdf500075eb52ea1bf455e0973ba114fbfd4, name=ceph-aa240572-0159-11ef-bc93-c7b262605968-osd-1-deactivate, GIT_CLEAN=True, org.label-schema.name=CentOS Stream 8 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2, io.buildah.version=1.33.5, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, org.label-schema.build-date=20240416, ceph=True, RELEASE=wip-lusov-quiescer-d69f13d, org.label-schema.schema-version=1.0) 2024-04-23T10:42:15.698 DEBUG:teuthology.orchestra.run.smithi119:> sudo pkill -f 'journalctl -f -n 0 -u ceph-aa240572-0159-11ef-bc93-c7b262605968@osd.1.service' 2024-04-23T10:42:15.741 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-04-23T10:42:15.741 INFO:tasks.cephadm.osd.1:Stopped osd.1 2024-04-23T10:42:15.742 INFO:tasks.cephadm.osd.2:Stopping osd.2... 2024-04-23T10:42:15.742 DEBUG:teuthology.orchestra.run.smithi204:> sudo systemctl stop ceph-aa240572-0159-11ef-bc93-c7b262605968@osd.2 2024-04-23T10:42:16.036 INFO:journalctl@ceph.osd.2.smithi204.stdout:Apr 23 10:42:15 smithi204 systemd[1]: Stopping Ceph osd.2 for aa240572-0159-11ef-bc93-c7b262605968... 2024-04-23T10:42:16.328 INFO:journalctl@ceph.osd.2.smithi204.stdout:Apr 23 10:42:16 smithi204 ceph-aa240572-0159-11ef-bc93-c7b262605968-osd-2[37811]: 2024-04-23T10:42:16.037+0000 7fd9c47d3700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.2 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-04-23T10:42:16.328 INFO:journalctl@ceph.osd.2.smithi204.stdout:Apr 23 10:42:16 smithi204 ceph-aa240572-0159-11ef-bc93-c7b262605968-osd-2[37811]: 2024-04-23T10:42:16.037+0000 7fd9c47d3700 -1 osd.2 116 *** Got signal Terminated *** 2024-04-23T10:42:16.328 INFO:journalctl@ceph.osd.2.smithi204.stdout:Apr 23 10:42:16 smithi204 ceph-aa240572-0159-11ef-bc93-c7b262605968-osd-2[37811]: 2024-04-23T10:42:16.037+0000 7fd9c47d3700 -1 osd.2 116 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-04-23T10:42:21.770 INFO:journalctl@ceph.osd.2.smithi204.stdout:Apr 23 10:42:21 smithi204 podman[70842]: 2024-04-23 10:42:21.518613874 +0000 UTC m=+5.583304661 container died b5133a0335f786c6a7b7b09746e36007fbb116ce7ce85cba1e466bb322b7a929 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:068a558e3af4932cf1c60dd321c0bdf500075eb52ea1bf455e0973ba114fbfd4, name=ceph-aa240572-0159-11ef-bc93-c7b262605968-osd-2, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.33.5, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, GIT_CLEAN=True, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, org.label-schema.build-date=20240416, ceph=True, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, RELEASE=wip-lusov-quiescer-d69f13d) 2024-04-23T10:42:22.054 INFO:journalctl@ceph.osd.2.smithi204.stdout:Apr 23 10:42:21 smithi204 podman[70842]: 2024-04-23 10:42:21.863144415 +0000 UTC m=+5.927835188 container cleanup b5133a0335f786c6a7b7b09746e36007fbb116ce7ce85cba1e466bb322b7a929 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:068a558e3af4932cf1c60dd321c0bdf500075eb52ea1bf455e0973ba114fbfd4, name=ceph-aa240572-0159-11ef-bc93-c7b262605968-osd-2, GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=, org.label-schema.schema-version=1.0, ceph=True, io.buildah.version=1.33.5, GIT_BRANCH=HEAD, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 8 Base Image, maintainer=Guillaume Abrioux , RELEASE=wip-lusov-quiescer-d69f13d, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, org.label-schema.vendor=CentOS, GIT_CLEAN=True, org.label-schema.build-date=20240416) 2024-04-23T10:42:22.054 INFO:journalctl@ceph.osd.2.smithi204.stdout:Apr 23 10:42:21 smithi204 bash[70842]: ceph-aa240572-0159-11ef-bc93-c7b262605968-osd-2 2024-04-23T10:42:22.328 INFO:journalctl@ceph.osd.2.smithi204.stdout:Apr 23 10:42:22 smithi204 podman[70859]: 2024-04-23 10:42:22.055936591 +0000 UTC m=+0.536095310 container remove b5133a0335f786c6a7b7b09746e36007fbb116ce7ce85cba1e466bb322b7a929 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:068a558e3af4932cf1c60dd321c0bdf500075eb52ea1bf455e0973ba114fbfd4, name=ceph-aa240572-0159-11ef-bc93-c7b262605968-osd-2, io.buildah.version=1.33.5, GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=, GIT_CLEAN=True, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.vendor=CentOS, ceph=True, org.label-schema.license=GPLv2, RELEASE=wip-lusov-quiescer-d69f13d, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, org.label-schema.schema-version=1.0, org.label-schema.build-date=20240416) 2024-04-23T10:42:22.799 INFO:journalctl@ceph.osd.2.smithi204.stdout:Apr 23 10:42:22 smithi204 podman[70949]: 2024-04-23 10:42:22.378116628 +0000 UTC m=+0.018182545 image pull da4b004c09e820713483d039bc76fbd660be678f2ea3c03e059a6161b74dab76 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:068a558e3af4932cf1c60dd321c0bdf500075eb52ea1bf455e0973ba114fbfd4 2024-04-23T10:42:22.799 INFO:journalctl@ceph.osd.2.smithi204.stdout:Apr 23 10:42:22 smithi204 podman[70949]: 2024-04-23 10:42:22.490148179 +0000 UTC m=+0.130214086 container create 595c69b42049e836a63ee9a6a461fe0a70cddc4d3e36a28cd4009f219bf1116c (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:068a558e3af4932cf1c60dd321c0bdf500075eb52ea1bf455e0973ba114fbfd4, name=ceph-aa240572-0159-11ef-bc93-c7b262605968-osd-2-deactivate, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, io.buildah.version=1.33.5, org.label-schema.build-date=20240416, org.label-schema.name=CentOS Stream 8 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=, GIT_CLEAN=True, RELEASE=wip-lusov-quiescer-d69f13d, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, ceph=True) 2024-04-23T10:42:22.799 INFO:journalctl@ceph.osd.2.smithi204.stdout:Apr 23 10:42:22 smithi204 podman[70949]: 2024-04-23 10:42:22.717859707 +0000 UTC m=+0.357925618 container init 595c69b42049e836a63ee9a6a461fe0a70cddc4d3e36a28cd4009f219bf1116c (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:068a558e3af4932cf1c60dd321c0bdf500075eb52ea1bf455e0973ba114fbfd4, name=ceph-aa240572-0159-11ef-bc93-c7b262605968-osd-2-deactivate, org.label-schema.license=GPLv2, RELEASE=wip-lusov-quiescer-d69f13d, org.label-schema.build-date=20240416, GIT_BRANCH=HEAD, CEPH_POINT_RELEASE=, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux , io.buildah.version=1.33.5, org.label-schema.schema-version=1.0, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, GIT_CLEAN=True, org.label-schema.name=CentOS Stream 8 Base Image, ceph=True, GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-04-23T10:42:22.799 INFO:journalctl@ceph.osd.2.smithi204.stdout:Apr 23 10:42:22 smithi204 podman[70949]: 2024-04-23 10:42:22.721077393 +0000 UTC m=+0.361143302 container start 595c69b42049e836a63ee9a6a461fe0a70cddc4d3e36a28cd4009f219bf1116c (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:068a558e3af4932cf1c60dd321c0bdf500075eb52ea1bf455e0973ba114fbfd4, name=ceph-aa240572-0159-11ef-bc93-c7b262605968-osd-2-deactivate, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.vendor=CentOS, ceph=True, GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0, org.label-schema.build-date=20240416, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_CLEAN=True, io.buildah.version=1.33.5, RELEASE=wip-lusov-quiescer-d69f13d, org.label-schema.license=GPLv2, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, maintainer=Guillaume Abrioux ) 2024-04-23T10:42:22.799 INFO:journalctl@ceph.osd.2.smithi204.stdout:Apr 23 10:42:22 smithi204 podman[70949]: 2024-04-23 10:42:22.801037892 +0000 UTC m=+0.441103801 container attach 595c69b42049e836a63ee9a6a461fe0a70cddc4d3e36a28cd4009f219bf1116c (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:068a558e3af4932cf1c60dd321c0bdf500075eb52ea1bf455e0973ba114fbfd4, name=ceph-aa240572-0159-11ef-bc93-c7b262605968-osd-2-deactivate, org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_BRANCH=HEAD, CEPH_POINT_RELEASE=, org.label-schema.build-date=20240416, io.buildah.version=1.33.5, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux , ceph=True, org.label-schema.license=GPLv2, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, GIT_CLEAN=True, RELEASE=wip-lusov-quiescer-d69f13d) 2024-04-23T10:42:23.077 INFO:journalctl@ceph.osd.2.smithi204.stdout:Apr 23 10:42:23 smithi204 podman[70949]: 2024-04-23 10:42:23.015675578 +0000 UTC m=+0.655741496 container died 595c69b42049e836a63ee9a6a461fe0a70cddc4d3e36a28cd4009f219bf1116c (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:068a558e3af4932cf1c60dd321c0bdf500075eb52ea1bf455e0973ba114fbfd4, name=ceph-aa240572-0159-11ef-bc93-c7b262605968-osd-2-deactivate, GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0, RELEASE=wip-lusov-quiescer-d69f13d, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, ceph=True, CEPH_POINT_RELEASE=, io.buildah.version=1.33.5, org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2, GIT_CLEAN=True, org.label-schema.build-date=20240416, maintainer=Guillaume Abrioux , GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 8 Base Image) 2024-04-23T10:42:23.559 DEBUG:teuthology.orchestra.run.smithi204:> sudo pkill -f 'journalctl -f -n 0 -u ceph-aa240572-0159-11ef-bc93-c7b262605968@osd.2.service' 2024-04-23T10:42:23.577 INFO:journalctl@ceph.osd.2.smithi204.stdout:Apr 23 10:42:23 smithi204 podman[70975]: 2024-04-23 10:42:23.546083749 +0000 UTC m=+0.525182227 container remove 595c69b42049e836a63ee9a6a461fe0a70cddc4d3e36a28cd4009f219bf1116c (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:068a558e3af4932cf1c60dd321c0bdf500075eb52ea1bf455e0973ba114fbfd4, name=ceph-aa240572-0159-11ef-bc93-c7b262605968-osd-2-deactivate, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, org.label-schema.build-date=20240416, GIT_CLEAN=True, io.buildah.version=1.33.5, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, RELEASE=wip-lusov-quiescer-d69f13d, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD, ceph=True) 2024-04-23T10:42:23.578 INFO:journalctl@ceph.osd.2.smithi204.stdout:Apr 23 10:42:23 smithi204 systemd[1]: ceph-aa240572-0159-11ef-bc93-c7b262605968@osd.2.service: Deactivated successfully. 2024-04-23T10:42:23.578 INFO:journalctl@ceph.osd.2.smithi204.stdout:Apr 23 10:42:23 smithi204 systemd[1]: Stopped Ceph osd.2 for aa240572-0159-11ef-bc93-c7b262605968. 2024-04-23T10:42:23.578 INFO:journalctl@ceph.osd.2.smithi204.stdout:Apr 23 10:42:23 smithi204 systemd[1]: ceph-aa240572-0159-11ef-bc93-c7b262605968@osd.2.service: Consumed 1min 32.454s CPU time. 2024-04-23T10:42:23.596 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-04-23T10:42:23.596 INFO:tasks.cephadm.osd.2:Stopped osd.2 2024-04-23T10:42:23.596 INFO:tasks.cephadm.osd.3:Stopping osd.3... 2024-04-23T10:42:23.597 DEBUG:teuthology.orchestra.run.smithi114:> sudo systemctl stop ceph-aa240572-0159-11ef-bc93-c7b262605968@osd.3 2024-04-23T10:42:23.901 INFO:journalctl@ceph.osd.3.smithi114.stdout:Apr 23 10:42:23 smithi114 systemd[1]: Stopping Ceph osd.3 for aa240572-0159-11ef-bc93-c7b262605968... 2024-04-23T10:42:24.164 INFO:journalctl@ceph.osd.3.smithi114.stdout:Apr 23 10:42:23 smithi114 ceph-aa240572-0159-11ef-bc93-c7b262605968-osd-3[48091]: 2024-04-23T10:42:23.901+0000 7f1aa4efa700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.3 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-04-23T10:42:24.164 INFO:journalctl@ceph.osd.3.smithi114.stdout:Apr 23 10:42:23 smithi114 ceph-aa240572-0159-11ef-bc93-c7b262605968-osd-3[48091]: 2024-04-23T10:42:23.901+0000 7f1aa4efa700 -1 osd.3 116 *** Got signal Terminated *** 2024-04-23T10:42:24.164 INFO:journalctl@ceph.osd.3.smithi114.stdout:Apr 23 10:42:23 smithi114 ceph-aa240572-0159-11ef-bc93-c7b262605968-osd-3[48091]: 2024-04-23T10:42:23.901+0000 7f1aa4efa700 -1 osd.3 116 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-04-23T10:42:26.328 INFO:journalctl@ceph.osd.11.smithi204.stdout:Apr 23 10:42:25 smithi204 ceph-aa240572-0159-11ef-bc93-c7b262605968-osd-11[57644]: 2024-04-23T10:42:25.890+0000 7fcbc4b63700 -1 osd.11 116 heartbeat_check: no reply from 172.21.15.114:6806 osd.0 since back 2024-04-23T10:42:00.675680+0000 front 2024-04-23T10:42:00.675649+0000 (oldest deadline 2024-04-23T10:42:25.876359+0000) 2024-04-23T10:42:27.004 INFO:journalctl@ceph.osd.6.smithi114.stdout:Apr 23 10:42:26 smithi114 ceph-aa240572-0159-11ef-bc93-c7b262605968-osd-6[55044]: 2024-04-23T10:42:26.515+0000 7fdba4fa4700 -1 osd.6 116 heartbeat_check: no reply from 172.21.15.114:6806 osd.0 since back 2024-04-23T10:42:01.662289+0000 front 2024-04-23T10:42:01.662441+0000 (oldest deadline 2024-04-23T10:42:26.362236+0000) 2024-04-23T10:42:27.327 INFO:journalctl@ceph.osd.11.smithi204.stdout:Apr 23 10:42:26 smithi204 ceph-aa240572-0159-11ef-bc93-c7b262605968-osd-11[57644]: 2024-04-23T10:42:26.894+0000 7fcbc4b63700 -1 osd.11 116 heartbeat_check: no reply from 172.21.15.114:6806 osd.0 since back 2024-04-23T10:42:00.675680+0000 front 2024-04-23T10:42:00.675649+0000 (oldest deadline 2024-04-23T10:42:25.876359+0000) 2024-04-23T10:42:27.741 INFO:journalctl@ceph.osd.10.smithi119.stdout:Apr 23 10:42:27 smithi119 ceph-aa240572-0159-11ef-bc93-c7b262605968-osd-10[57454]: 2024-04-23T10:42:27.335+0000 7fb1849bf700 -1 osd.10 116 heartbeat_check: no reply from 172.21.15.114:6806 osd.0 since back 2024-04-23T10:42:01.813174+0000 front 2024-04-23T10:42:01.813138+0000 (oldest deadline 2024-04-23T10:42:27.113300+0000) 2024-04-23T10:42:28.003 INFO:journalctl@ceph.osd.6.smithi114.stdout:Apr 23 10:42:27 smithi114 ceph-aa240572-0159-11ef-bc93-c7b262605968-osd-6[55044]: 2024-04-23T10:42:27.559+0000 7fdba4fa4700 -1 osd.6 116 heartbeat_check: no reply from 172.21.15.114:6806 osd.0 since back 2024-04-23T10:42:01.662289+0000 front 2024-04-23T10:42:01.662441+0000 (oldest deadline 2024-04-23T10:42:26.362236+0000) 2024-04-23T10:42:28.304 INFO:journalctl@ceph.osd.11.smithi204.stdout:Apr 23 10:42:27 smithi204 ceph-aa240572-0159-11ef-bc93-c7b262605968-osd-11[57644]: 2024-04-23T10:42:27.889+0000 7fcbc4b63700 -1 osd.11 116 heartbeat_check: no reply from 172.21.15.114:6806 osd.0 since back 2024-04-23T10:42:00.675680+0000 front 2024-04-23T10:42:00.675649+0000 (oldest deadline 2024-04-23T10:42:25.876359+0000) 2024-04-23T10:42:28.578 INFO:journalctl@ceph.osd.8.smithi204.stdout:Apr 23 10:42:28 smithi204 ceph-aa240572-0159-11ef-bc93-c7b262605968-osd-8[51050]: 2024-04-23T10:42:28.348+0000 7f8d662cd700 -1 osd.8 116 heartbeat_check: no reply from 172.21.15.114:6806 osd.0 since back 2024-04-23T10:42:03.759281+0000 front 2024-04-23T10:42:03.759348+0000 (oldest deadline 2024-04-23T10:42:27.859455+0000) 2024-04-23T10:42:28.578 INFO:journalctl@ceph.osd.5.smithi204.stdout:Apr 23 10:42:28 smithi204 ceph-aa240572-0159-11ef-bc93-c7b262605968-osd-5[44334]: 2024-04-23T10:42:28.305+0000 7fbf0215c700 -1 osd.5 116 heartbeat_check: no reply from 172.21.15.114:6806 osd.0 since back 2024-04-23T10:42:02.285250+0000 front 2024-04-23T10:42:02.285119+0000 (oldest deadline 2024-04-23T10:42:28.185247+0000) 2024-04-23T10:42:28.742 INFO:journalctl@ceph.osd.10.smithi119.stdout:Apr 23 10:42:28 smithi119 ceph-aa240572-0159-11ef-bc93-c7b262605968-osd-10[57454]: 2024-04-23T10:42:28.380+0000 7fb1849bf700 -1 osd.10 116 heartbeat_check: no reply from 172.21.15.114:6806 osd.0 since back 2024-04-23T10:42:01.813174+0000 front 2024-04-23T10:42:01.813138+0000 (oldest deadline 2024-04-23T10:42:27.113300+0000) 2024-04-23T10:42:29.004 INFO:journalctl@ceph.osd.6.smithi114.stdout:Apr 23 10:42:28 smithi114 ceph-aa240572-0159-11ef-bc93-c7b262605968-osd-6[55044]: 2024-04-23T10:42:28.548+0000 7fdba4fa4700 -1 osd.6 116 heartbeat_check: no reply from 172.21.15.114:6806 osd.0 since back 2024-04-23T10:42:01.662289+0000 front 2024-04-23T10:42:01.662441+0000 (oldest deadline 2024-04-23T10:42:26.362236+0000) 2024-04-23T10:42:29.254 INFO:journalctl@ceph.osd.11.smithi204.stdout:Apr 23 10:42:28 smithi204 ceph-aa240572-0159-11ef-bc93-c7b262605968-osd-11[57644]: 2024-04-23T10:42:28.885+0000 7fcbc4b63700 -1 osd.11 116 heartbeat_check: no reply from 172.21.15.114:6806 osd.0 since back 2024-04-23T10:42:00.675680+0000 front 2024-04-23T10:42:00.675649+0000 (oldest deadline 2024-04-23T10:42:25.876359+0000) 2024-04-23T10:42:29.578 INFO:journalctl@ceph.osd.8.smithi204.stdout:Apr 23 10:42:29 smithi204 ceph-aa240572-0159-11ef-bc93-c7b262605968-osd-8[51050]: 2024-04-23T10:42:29.315+0000 7f8d662cd700 -1 osd.8 116 heartbeat_check: no reply from 172.21.15.114:6806 osd.0 since back 2024-04-23T10:42:03.759281+0000 front 2024-04-23T10:42:03.759348+0000 (oldest deadline 2024-04-23T10:42:27.859455+0000) 2024-04-23T10:42:29.578 INFO:journalctl@ceph.osd.5.smithi204.stdout:Apr 23 10:42:29 smithi204 ceph-aa240572-0159-11ef-bc93-c7b262605968-osd-5[44334]: 2024-04-23T10:42:29.256+0000 7fbf0215c700 -1 osd.5 116 heartbeat_check: no reply from 172.21.15.114:6806 osd.0 since back 2024-04-23T10:42:02.285250+0000 front 2024-04-23T10:42:02.285119+0000 (oldest deadline 2024-04-23T10:42:28.185247+0000) 2024-04-23T10:42:29.601 INFO:journalctl@ceph.osd.3.smithi114.stdout:Apr 23 10:42:29 smithi114 podman[91112]: 2024-04-23 10:42:29.32168855 +0000 UTC m=+5.536001495 container died 2139eacb0b2d9aac4ebee209be20ca96dd318de02425ddb32e96122bd4c0c5b6 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:068a558e3af4932cf1c60dd321c0bdf500075eb52ea1bf455e0973ba114fbfd4, name=ceph-aa240572-0159-11ef-bc93-c7b262605968-osd-3, RELEASE=wip-lusov-quiescer-d69f13d, org.label-schema.build-date=20240416, io.buildah.version=1.33.5, GIT_CLEAN=True, ceph=True, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 8 Base Image, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS) 2024-04-23T10:42:29.601 INFO:journalctl@ceph.osd.6.smithi114.stdout:Apr 23 10:42:29 smithi114 ceph-aa240572-0159-11ef-bc93-c7b262605968-osd-6[55044]: 2024-04-23T10:42:29.549+0000 7fdba4fa4700 -1 osd.6 116 heartbeat_check: no reply from 172.21.15.114:6806 osd.0 since back 2024-04-23T10:42:01.662289+0000 front 2024-04-23T10:42:01.662441+0000 (oldest deadline 2024-04-23T10:42:26.362236+0000) 2024-04-23T10:42:29.602 INFO:journalctl@ceph.osd.9.smithi114.stdout:Apr 23 10:42:29 smithi114 ceph-aa240572-0159-11ef-bc93-c7b262605968-osd-9[62058]: 2024-04-23T10:42:29.448+0000 7feeb4bc7700 -1 osd.9 116 heartbeat_check: no reply from 172.21.15.114:6806 osd.0 since back 2024-04-23T10:42:04.136736+0000 front 2024-04-23T10:42:04.136764+0000 (oldest deadline 2024-04-23T10:42:28.836926+0000) 2024-04-23T10:42:29.742 INFO:journalctl@ceph.osd.10.smithi119.stdout:Apr 23 10:42:29 smithi119 ceph-aa240572-0159-11ef-bc93-c7b262605968-osd-10[57454]: 2024-04-23T10:42:29.416+0000 7fb1849bf700 -1 osd.10 116 heartbeat_check: no reply from 172.21.15.114:6806 osd.0 since back 2024-04-23T10:42:01.813174+0000 front 2024-04-23T10:42:01.813138+0000 (oldest deadline 2024-04-23T10:42:27.113300+0000) 2024-04-23T10:42:29.943 INFO:journalctl@ceph.osd.3.smithi114.stdout:Apr 23 10:42:29 smithi114 podman[91112]: 2024-04-23 10:42:29.702206086 +0000 UTC m=+5.916519044 container cleanup 2139eacb0b2d9aac4ebee209be20ca96dd318de02425ddb32e96122bd4c0c5b6 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:068a558e3af4932cf1c60dd321c0bdf500075eb52ea1bf455e0973ba114fbfd4, name=ceph-aa240572-0159-11ef-bc93-c7b262605968-osd-3, RELEASE=wip-lusov-quiescer-d69f13d, ceph=True, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=, org.label-schema.license=GPLv2, io.buildah.version=1.33.5, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 8 Base Image, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, GIT_CLEAN=True, org.label-schema.build-date=20240416, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS) 2024-04-23T10:42:29.943 INFO:journalctl@ceph.osd.3.smithi114.stdout:Apr 23 10:42:29 smithi114 bash[91112]: ceph-aa240572-0159-11ef-bc93-c7b262605968-osd-3 2024-04-23T10:42:30.254 INFO:journalctl@ceph.osd.3.smithi114.stdout:Apr 23 10:42:29 smithi114 podman[91144]: 2024-04-23 10:42:29.944994409 +0000 UTC m=+0.623747226 container remove 2139eacb0b2d9aac4ebee209be20ca96dd318de02425ddb32e96122bd4c0c5b6 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:068a558e3af4932cf1c60dd321c0bdf500075eb52ea1bf455e0973ba114fbfd4, name=ceph-aa240572-0159-11ef-bc93-c7b262605968-osd-3, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, GIT_CLEAN=True, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 8 Base Image, io.buildah.version=1.33.5, maintainer=Guillaume Abrioux , GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, ceph=True, org.label-schema.build-date=20240416, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=wip-lusov-quiescer-d69f13d, CEPH_POINT_RELEASE=, org.label-schema.schema-version=1.0) 2024-04-23T10:42:30.276 INFO:journalctl@ceph.osd.11.smithi204.stdout:Apr 23 10:42:29 smithi204 ceph-aa240572-0159-11ef-bc93-c7b262605968-osd-11[57644]: 2024-04-23T10:42:29.915+0000 7fcbc4b63700 -1 osd.11 116 heartbeat_check: no reply from 172.21.15.114:6806 osd.0 since back 2024-04-23T10:42:00.675680+0000 front 2024-04-23T10:42:00.675649+0000 (oldest deadline 2024-04-23T10:42:25.876359+0000) 2024-04-23T10:42:30.526 INFO:journalctl@ceph.osd.9.smithi114.stdout:Apr 23 10:42:30 smithi114 ceph-aa240572-0159-11ef-bc93-c7b262605968-osd-9[62058]: 2024-04-23T10:42:30.446+0000 7feeb4bc7700 -1 osd.9 116 heartbeat_check: no reply from 172.21.15.114:6806 osd.0 since back 2024-04-23T10:42:04.136736+0000 front 2024-04-23T10:42:04.136764+0000 (oldest deadline 2024-04-23T10:42:28.836926+0000) 2024-04-23T10:42:30.527 INFO:journalctl@ceph.osd.3.smithi114.stdout:Apr 23 10:42:30 smithi114 podman[91236]: 2024-04-23 10:42:30.262526809 +0000 UTC m=+0.112336428 container create a2d315fe06dd78e2dfdc48d3290f642a3e89e8108f58098f95c8b27d8e45f499 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:068a558e3af4932cf1c60dd321c0bdf500075eb52ea1bf455e0973ba114fbfd4, name=ceph-aa240572-0159-11ef-bc93-c7b262605968-osd-3-deactivate, CEPH_POINT_RELEASE=, org.label-schema.schema-version=1.0, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, io.buildah.version=1.33.5, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=wip-lusov-quiescer-d69f13d, ceph=True, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux , GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, org.label-schema.build-date=20240416, GIT_CLEAN=True, org.label-schema.name=CentOS Stream 8 Base Image) 2024-04-23T10:42:30.527 INFO:journalctl@ceph.osd.3.smithi114.stdout:Apr 23 10:42:30 smithi114 podman[91236]: 2024-04-23 10:42:30.168692691 +0000 UTC m=+0.018502269 image pull da4b004c09e820713483d039bc76fbd660be678f2ea3c03e059a6161b74dab76 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:068a558e3af4932cf1c60dd321c0bdf500075eb52ea1bf455e0973ba114fbfd4 2024-04-23T10:42:30.578 INFO:journalctl@ceph.osd.8.smithi204.stdout:Apr 23 10:42:30 smithi204 ceph-aa240572-0159-11ef-bc93-c7b262605968-osd-8[51050]: 2024-04-23T10:42:30.278+0000 7f8d662cd700 -1 osd.8 116 heartbeat_check: no reply from 172.21.15.114:6806 osd.0 since back 2024-04-23T10:42:03.759281+0000 front 2024-04-23T10:42:03.759348+0000 (oldest deadline 2024-04-23T10:42:27.859455+0000) 2024-04-23T10:42:30.578 INFO:journalctl@ceph.osd.5.smithi204.stdout:Apr 23 10:42:30 smithi204 ceph-aa240572-0159-11ef-bc93-c7b262605968-osd-5[44334]: 2024-04-23T10:42:30.289+0000 7fbf0215c700 -1 osd.5 116 heartbeat_check: no reply from 172.21.15.114:6806 osd.0 since back 2024-04-23T10:42:02.285250+0000 front 2024-04-23T10:42:02.285119+0000 (oldest deadline 2024-04-23T10:42:28.185247+0000) 2024-04-23T10:42:30.742 INFO:journalctl@ceph.osd.10.smithi119.stdout:Apr 23 10:42:30 smithi119 ceph-aa240572-0159-11ef-bc93-c7b262605968-osd-10[57454]: 2024-04-23T10:42:30.446+0000 7fb1849bf700 -1 osd.10 116 heartbeat_check: no reply from 172.21.15.114:6806 osd.0 since back 2024-04-23T10:42:01.813174+0000 front 2024-04-23T10:42:01.813138+0000 (oldest deadline 2024-04-23T10:42:27.113300+0000) 2024-04-23T10:42:30.827 INFO:journalctl@ceph.osd.6.smithi114.stdout:Apr 23 10:42:30 smithi114 ceph-aa240572-0159-11ef-bc93-c7b262605968-osd-6[55044]: 2024-04-23T10:42:30.597+0000 7fdba4fa4700 -1 osd.6 116 heartbeat_check: no reply from 172.21.15.114:6806 osd.0 since back 2024-04-23T10:42:01.662289+0000 front 2024-04-23T10:42:01.662441+0000 (oldest deadline 2024-04-23T10:42:26.362236+0000) 2024-04-23T10:42:30.828 INFO:journalctl@ceph.osd.3.smithi114.stdout:Apr 23 10:42:30 smithi114 podman[91236]: 2024-04-23 10:42:30.528503675 +0000 UTC m=+0.378313241 container init a2d315fe06dd78e2dfdc48d3290f642a3e89e8108f58098f95c8b27d8e45f499 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:068a558e3af4932cf1c60dd321c0bdf500075eb52ea1bf455e0973ba114fbfd4, name=ceph-aa240572-0159-11ef-bc93-c7b262605968-osd-3-deactivate, CEPH_POINT_RELEASE=, io.buildah.version=1.33.5, GIT_CLEAN=True, org.label-schema.license=GPLv2, ceph=True, GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, org.label-schema.name=CentOS Stream 8 Base Image, maintainer=Guillaume Abrioux , GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=wip-lusov-quiescer-d69f13d, org.label-schema.build-date=20240416, org.label-schema.vendor=CentOS) 2024-04-23T10:42:30.828 INFO:journalctl@ceph.osd.3.smithi114.stdout:Apr 23 10:42:30 smithi114 podman[91236]: 2024-04-23 10:42:30.531713286 +0000 UTC m=+0.381522857 container start a2d315fe06dd78e2dfdc48d3290f642a3e89e8108f58098f95c8b27d8e45f499 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:068a558e3af4932cf1c60dd321c0bdf500075eb52ea1bf455e0973ba114fbfd4, name=ceph-aa240572-0159-11ef-bc93-c7b262605968-osd-3-deactivate, io.buildah.version=1.33.5, GIT_BRANCH=HEAD, RELEASE=wip-lusov-quiescer-d69f13d, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, GIT_CLEAN=True, ceph=True, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.vendor=CentOS, org.label-schema.build-date=20240416, org.label-schema.license=GPLv2, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=, org.label-schema.schema-version=1.0) 2024-04-23T10:42:30.828 INFO:journalctl@ceph.osd.3.smithi114.stdout:Apr 23 10:42:30 smithi114 podman[91236]: 2024-04-23 10:42:30.590052972 +0000 UTC m=+0.439862539 container attach a2d315fe06dd78e2dfdc48d3290f642a3e89e8108f58098f95c8b27d8e45f499 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:068a558e3af4932cf1c60dd321c0bdf500075eb52ea1bf455e0973ba114fbfd4, name=ceph-aa240572-0159-11ef-bc93-c7b262605968-osd-3-deactivate, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, GIT_CLEAN=True, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240416, ceph=True, io.buildah.version=1.33.5, RELEASE=wip-lusov-quiescer-d69f13d, org.label-schema.name=CentOS Stream 8 Base Image, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS) 2024-04-23T10:42:31.191 INFO:journalctl@ceph.osd.3.smithi114.stdout:Apr 23 10:42:30 smithi114 podman[91236]: 2024-04-23 10:42:30.829563099 +0000 UTC m=+0.679372666 container died a2d315fe06dd78e2dfdc48d3290f642a3e89e8108f58098f95c8b27d8e45f499 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:068a558e3af4932cf1c60dd321c0bdf500075eb52ea1bf455e0973ba114fbfd4, name=ceph-aa240572-0159-11ef-bc93-c7b262605968-osd-3-deactivate, org.label-schema.name=CentOS Stream 8 Base Image, GIT_BRANCH=HEAD, io.buildah.version=1.33.5, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240416, RELEASE=wip-lusov-quiescer-d69f13d, ceph=True, org.label-schema.schema-version=1.0, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, GIT_CLEAN=True, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=) 2024-04-23T10:42:31.243 INFO:journalctl@ceph.osd.11.smithi204.stdout:Apr 23 10:42:30 smithi204 ceph-aa240572-0159-11ef-bc93-c7b262605968-osd-11[57644]: 2024-04-23T10:42:30.881+0000 7fcbc4b63700 -1 osd.11 116 heartbeat_check: no reply from 172.21.15.114:6806 osd.0 since back 2024-04-23T10:42:00.675680+0000 front 2024-04-23T10:42:00.675649+0000 (oldest deadline 2024-04-23T10:42:25.876359+0000) 2024-04-23T10:42:31.450 INFO:journalctl@ceph.osd.9.smithi114.stdout:Apr 23 10:42:31 smithi114 ceph-aa240572-0159-11ef-bc93-c7b262605968-osd-9[62058]: 2024-04-23T10:42:31.418+0000 7feeb4bc7700 -1 osd.9 116 heartbeat_check: no reply from 172.21.15.114:6806 osd.0 since back 2024-04-23T10:42:04.136736+0000 front 2024-04-23T10:42:04.136764+0000 (oldest deadline 2024-04-23T10:42:28.836926+0000) 2024-04-23T10:42:31.470 DEBUG:teuthology.orchestra.run.smithi114:> sudo pkill -f 'journalctl -f -n 0 -u ceph-aa240572-0159-11ef-bc93-c7b262605968@osd.3.service' 2024-04-23T10:42:31.510 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-04-23T10:42:31.510 INFO:tasks.cephadm.osd.3:Stopped osd.3 2024-04-23T10:42:31.511 INFO:tasks.cephadm.osd.4:Stopping osd.4... 2024-04-23T10:42:31.511 DEBUG:teuthology.orchestra.run.smithi119:> sudo systemctl stop ceph-aa240572-0159-11ef-bc93-c7b262605968@osd.4 2024-04-23T10:42:31.578 INFO:journalctl@ceph.osd.8.smithi204.stdout:Apr 23 10:42:31 smithi204 ceph-aa240572-0159-11ef-bc93-c7b262605968-osd-8[51050]: 2024-04-23T10:42:31.254+0000 7f8d662cd700 -1 osd.8 116 heartbeat_check: no reply from 172.21.15.114:6806 osd.0 since back 2024-04-23T10:42:03.759281+0000 front 2024-04-23T10:42:03.759348+0000 (oldest deadline 2024-04-23T10:42:27.859455+0000) 2024-04-23T10:42:31.578 INFO:journalctl@ceph.osd.5.smithi204.stdout:Apr 23 10:42:31 smithi204 ceph-aa240572-0159-11ef-bc93-c7b262605968-osd-5[44334]: 2024-04-23T10:42:31.244+0000 7fbf0215c700 -1 osd.5 116 heartbeat_check: no reply from 172.21.15.114:6806 osd.0 since back 2024-04-23T10:42:02.285250+0000 front 2024-04-23T10:42:02.285119+0000 (oldest deadline 2024-04-23T10:42:28.185247+0000) 2024-04-23T10:42:31.742 INFO:journalctl@ceph.osd.4.smithi119.stdout:Apr 23 10:42:31 smithi119 systemd[1]: Stopping Ceph osd.4 for aa240572-0159-11ef-bc93-c7b262605968... 2024-04-23T10:42:31.743 INFO:journalctl@ceph.osd.10.smithi119.stdout:Apr 23 10:42:31 smithi119 ceph-aa240572-0159-11ef-bc93-c7b262605968-osd-10[57454]: 2024-04-23T10:42:31.487+0000 7fb1849bf700 -1 osd.10 116 heartbeat_check: no reply from 172.21.15.114:6806 osd.0 since back 2024-04-23T10:42:01.813174+0000 front 2024-04-23T10:42:01.813138+0000 (oldest deadline 2024-04-23T10:42:27.113300+0000) 2024-04-23T10:42:31.754 INFO:journalctl@ceph.osd.6.smithi114.stdout:Apr 23 10:42:31 smithi114 ceph-aa240572-0159-11ef-bc93-c7b262605968-osd-6[55044]: 2024-04-23T10:42:31.579+0000 7fdba4fa4700 -1 osd.6 116 heartbeat_check: no reply from 172.21.15.114:6806 osd.0 since back 2024-04-23T10:42:01.662289+0000 front 2024-04-23T10:42:01.662441+0000 (oldest deadline 2024-04-23T10:42:26.362236+0000) 2024-04-23T10:42:32.239 INFO:journalctl@ceph.osd.11.smithi204.stdout:Apr 23 10:42:31 smithi204 ceph-aa240572-0159-11ef-bc93-c7b262605968-osd-11[57644]: 2024-04-23T10:42:31.841+0000 7fcbc4b63700 -1 osd.11 116 heartbeat_check: no reply from 172.21.15.114:6806 osd.0 since back 2024-04-23T10:42:00.675680+0000 front 2024-04-23T10:42:00.675649+0000 (oldest deadline 2024-04-23T10:42:25.876359+0000) 2024-04-23T10:42:32.242 INFO:journalctl@ceph.osd.4.smithi119.stdout:Apr 23 10:42:31 smithi119 ceph-aa240572-0159-11ef-bc93-c7b262605968-osd-4[43889]: 2024-04-23T10:42:31.803+0000 7f4e9e9a4700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.4 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-04-23T10:42:32.242 INFO:journalctl@ceph.osd.4.smithi119.stdout:Apr 23 10:42:31 smithi119 ceph-aa240572-0159-11ef-bc93-c7b262605968-osd-4[43889]: 2024-04-23T10:42:31.803+0000 7f4e9e9a4700 -1 osd.4 116 *** Got signal Terminated *** 2024-04-23T10:42:32.242 INFO:journalctl@ceph.osd.4.smithi119.stdout:Apr 23 10:42:31 smithi119 ceph-aa240572-0159-11ef-bc93-c7b262605968-osd-4[43889]: 2024-04-23T10:42:31.803+0000 7f4e9e9a4700 -1 osd.4 116 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-04-23T10:42:32.578 INFO:journalctl@ceph.osd.8.smithi204.stdout:Apr 23 10:42:32 smithi204 ceph-aa240572-0159-11ef-bc93-c7b262605968-osd-8[51050]: 2024-04-23T10:42:32.240+0000 7f8d662cd700 -1 osd.8 116 heartbeat_check: no reply from 172.21.15.114:6806 osd.0 since back 2024-04-23T10:42:03.759281+0000 front 2024-04-23T10:42:03.759348+0000 (oldest deadline 2024-04-23T10:42:27.859455+0000) 2024-04-23T10:42:32.578 INFO:journalctl@ceph.osd.5.smithi204.stdout:Apr 23 10:42:32 smithi204 ceph-aa240572-0159-11ef-bc93-c7b262605968-osd-5[44334]: 2024-04-23T10:42:32.280+0000 7fbf0215c700 -1 osd.5 116 heartbeat_check: no reply from 172.21.15.114:6806 osd.0 since back 2024-04-23T10:42:02.285250+0000 front 2024-04-23T10:42:02.285119+0000 (oldest deadline 2024-04-23T10:42:28.185247+0000) 2024-04-23T10:42:32.742 INFO:journalctl@ceph.osd.10.smithi119.stdout:Apr 23 10:42:32 smithi119 ceph-aa240572-0159-11ef-bc93-c7b262605968-osd-10[57454]: 2024-04-23T10:42:32.460+0000 7fb1849bf700 -1 osd.10 116 heartbeat_check: no reply from 172.21.15.114:6806 osd.0 since back 2024-04-23T10:42:01.813174+0000 front 2024-04-23T10:42:01.813138+0000 (oldest deadline 2024-04-23T10:42:27.113300+0000) 2024-04-23T10:42:32.754 INFO:journalctl@ceph.osd.6.smithi114.stdout:Apr 23 10:42:32 smithi114 ceph-aa240572-0159-11ef-bc93-c7b262605968-osd-6[55044]: 2024-04-23T10:42:32.537+0000 7fdba4fa4700 -1 osd.6 116 heartbeat_check: no reply from 172.21.15.114:6806 osd.0 since back 2024-04-23T10:42:01.662289+0000 front 2024-04-23T10:42:01.662441+0000 (oldest deadline 2024-04-23T10:42:26.362236+0000) 2024-04-23T10:42:32.754 INFO:journalctl@ceph.osd.9.smithi114.stdout:Apr 23 10:42:32 smithi114 ceph-aa240572-0159-11ef-bc93-c7b262605968-osd-9[62058]: 2024-04-23T10:42:32.378+0000 7feeb4bc7700 -1 osd.9 116 heartbeat_check: no reply from 172.21.15.114:6806 osd.0 since back 2024-04-23T10:42:04.136736+0000 front 2024-04-23T10:42:04.136764+0000 (oldest deadline 2024-04-23T10:42:28.836926+0000) 2024-04-23T10:42:33.243 INFO:journalctl@ceph.osd.11.smithi204.stdout:Apr 23 10:42:32 smithi204 ceph-aa240572-0159-11ef-bc93-c7b262605968-osd-11[57644]: 2024-04-23T10:42:32.877+0000 7fcbc4b63700 -1 osd.11 116 heartbeat_check: no reply from 172.21.15.114:6806 osd.0 since back 2024-04-23T10:42:00.675680+0000 front 2024-04-23T10:42:00.675649+0000 (oldest deadline 2024-04-23T10:42:25.876359+0000) 2024-04-23T10:42:33.579 INFO:journalctl@ceph.osd.8.smithi204.stdout:Apr 23 10:42:33 smithi204 ceph-aa240572-0159-11ef-bc93-c7b262605968-osd-8[51050]: 2024-04-23T10:42:33.284+0000 7f8d662cd700 -1 osd.8 116 heartbeat_check: no reply from 172.21.15.114:6806 osd.0 since back 2024-04-23T10:42:03.759281+0000 front 2024-04-23T10:42:03.759348+0000 (oldest deadline 2024-04-23T10:42:27.859455+0000) 2024-04-23T10:42:33.579 INFO:journalctl@ceph.osd.5.smithi204.stdout:Apr 23 10:42:33 smithi204 ceph-aa240572-0159-11ef-bc93-c7b262605968-osd-5[44334]: 2024-04-23T10:42:33.244+0000 7fbf0215c700 -1 osd.5 116 heartbeat_check: no reply from 172.21.15.114:6806 osd.0 since back 2024-04-23T10:42:02.285250+0000 front 2024-04-23T10:42:02.285119+0000 (oldest deadline 2024-04-23T10:42:28.185247+0000) 2024-04-23T10:42:33.742 INFO:journalctl@ceph.osd.10.smithi119.stdout:Apr 23 10:42:33 smithi119 ceph-aa240572-0159-11ef-bc93-c7b262605968-osd-10[57454]: 2024-04-23T10:42:33.456+0000 7fb1849bf700 -1 osd.10 116 heartbeat_check: no reply from 172.21.15.114:6806 osd.0 since back 2024-04-23T10:42:01.813174+0000 front 2024-04-23T10:42:01.813138+0000 (oldest deadline 2024-04-23T10:42:27.113300+0000) 2024-04-23T10:42:33.754 INFO:journalctl@ceph.osd.9.smithi114.stdout:Apr 23 10:42:33 smithi114 ceph-aa240572-0159-11ef-bc93-c7b262605968-osd-9[62058]: 2024-04-23T10:42:33.379+0000 7feeb4bc7700 -1 osd.9 116 heartbeat_check: no reply from 172.21.15.114:6806 osd.0 since back 2024-04-23T10:42:04.136736+0000 front 2024-04-23T10:42:04.136764+0000 (oldest deadline 2024-04-23T10:42:28.836926+0000) 2024-04-23T10:42:33.754 INFO:journalctl@ceph.osd.6.smithi114.stdout:Apr 23 10:42:33 smithi114 ceph-aa240572-0159-11ef-bc93-c7b262605968-osd-6[55044]: 2024-04-23T10:42:33.565+0000 7fdba4fa4700 -1 osd.6 116 heartbeat_check: no reply from 172.21.15.114:6806 osd.0 since back 2024-04-23T10:42:01.662289+0000 front 2024-04-23T10:42:01.662441+0000 (oldest deadline 2024-04-23T10:42:26.362236+0000) 2024-04-23T10:42:34.234 INFO:journalctl@ceph.osd.11.smithi204.stdout:Apr 23 10:42:33 smithi204 ceph-aa240572-0159-11ef-bc93-c7b262605968-osd-11[57644]: 2024-04-23T10:42:33.885+0000 7fcbc4b63700 -1 osd.11 116 heartbeat_check: no reply from 172.21.15.114:6806 osd.0 since back 2024-04-23T10:42:00.675680+0000 front 2024-04-23T10:42:00.675649+0000 (oldest deadline 2024-04-23T10:42:25.876359+0000) 2024-04-23T10:42:34.234 INFO:journalctl@ceph.osd.11.smithi204.stdout:Apr 23 10:42:33 smithi204 ceph-aa240572-0159-11ef-bc93-c7b262605968-osd-11[57644]: 2024-04-23T10:42:33.885+0000 7fcbc4b63700 -1 osd.11 116 heartbeat_check: no reply from 172.21.15.119:6804 osd.1 since back 2024-04-23T10:42:09.278512+0000 front 2024-04-23T10:42:09.278494+0000 (oldest deadline 2024-04-23T10:42:33.378557+0000) 2024-04-23T10:42:34.578 INFO:journalctl@ceph.osd.8.smithi204.stdout:Apr 23 10:42:34 smithi204 ceph-aa240572-0159-11ef-bc93-c7b262605968-osd-8[51050]: 2024-04-23T10:42:34.279+0000 7f8d662cd700 -1 osd.8 116 heartbeat_check: no reply from 172.21.15.114:6806 osd.0 since back 2024-04-23T10:42:03.759281+0000 front 2024-04-23T10:42:03.759348+0000 (oldest deadline 2024-04-23T10:42:27.859455+0000) 2024-04-23T10:42:34.578 INFO:journalctl@ceph.osd.5.smithi204.stdout:Apr 23 10:42:34 smithi204 ceph-aa240572-0159-11ef-bc93-c7b262605968-osd-5[44334]: 2024-04-23T10:42:34.235+0000 7fbf0215c700 -1 osd.5 116 heartbeat_check: no reply from 172.21.15.114:6806 osd.0 since back 2024-04-23T10:42:02.285250+0000 front 2024-04-23T10:42:02.285119+0000 (oldest deadline 2024-04-23T10:42:28.185247+0000) 2024-04-23T10:42:34.742 INFO:journalctl@ceph.osd.10.smithi119.stdout:Apr 23 10:42:34 smithi119 ceph-aa240572-0159-11ef-bc93-c7b262605968-osd-10[57454]: 2024-04-23T10:42:34.480+0000 7fb1849bf700 -1 osd.10 116 heartbeat_check: no reply from 172.21.15.114:6806 osd.0 since back 2024-04-23T10:42:01.813174+0000 front 2024-04-23T10:42:01.813138+0000 (oldest deadline 2024-04-23T10:42:27.113300+0000) 2024-04-23T10:42:34.742 INFO:journalctl@ceph.osd.10.smithi119.stdout:Apr 23 10:42:34 smithi119 ceph-aa240572-0159-11ef-bc93-c7b262605968-osd-10[57454]: 2024-04-23T10:42:34.480+0000 7fb1849bf700 -1 osd.10 116 heartbeat_check: no reply from 172.21.15.119:6804 osd.1 since back 2024-04-23T10:42:10.614599+0000 front 2024-04-23T10:42:10.614622+0000 (oldest deadline 2024-04-23T10:42:34.114881+0000) 2024-04-23T10:42:34.754 INFO:journalctl@ceph.osd.9.smithi114.stdout:Apr 23 10:42:34 smithi114 ceph-aa240572-0159-11ef-bc93-c7b262605968-osd-9[62058]: 2024-04-23T10:42:34.416+0000 7feeb4bc7700 -1 osd.9 116 heartbeat_check: no reply from 172.21.15.114:6806 osd.0 since back 2024-04-23T10:42:04.136736+0000 front 2024-04-23T10:42:04.136764+0000 (oldest deadline 2024-04-23T10:42:28.836926+0000) 2024-04-23T10:42:34.754 INFO:journalctl@ceph.osd.6.smithi114.stdout:Apr 23 10:42:34 smithi114 ceph-aa240572-0159-11ef-bc93-c7b262605968-osd-6[55044]: 2024-04-23T10:42:34.581+0000 7fdba4fa4700 -1 osd.6 116 heartbeat_check: no reply from 172.21.15.114:6806 osd.0 since back 2024-04-23T10:42:01.662289+0000 front 2024-04-23T10:42:01.662441+0000 (oldest deadline 2024-04-23T10:42:26.362236+0000) 2024-04-23T10:42:35.269 INFO:journalctl@ceph.osd.11.smithi204.stdout:Apr 23 10:42:34 smithi204 ceph-aa240572-0159-11ef-bc93-c7b262605968-osd-11[57644]: 2024-04-23T10:42:34.851+0000 7fcbc4b63700 -1 osd.11 116 heartbeat_check: no reply from 172.21.15.114:6806 osd.0 since back 2024-04-23T10:42:00.675680+0000 front 2024-04-23T10:42:00.675649+0000 (oldest deadline 2024-04-23T10:42:25.876359+0000) 2024-04-23T10:42:35.269 INFO:journalctl@ceph.osd.11.smithi204.stdout:Apr 23 10:42:34 smithi204 ceph-aa240572-0159-11ef-bc93-c7b262605968-osd-11[57644]: 2024-04-23T10:42:34.851+0000 7fcbc4b63700 -1 osd.11 116 heartbeat_check: no reply from 172.21.15.119:6804 osd.1 since back 2024-04-23T10:42:09.278512+0000 front 2024-04-23T10:42:09.278494+0000 (oldest deadline 2024-04-23T10:42:33.378557+0000) 2024-04-23T10:42:35.578 INFO:journalctl@ceph.osd.8.smithi204.stdout:Apr 23 10:42:35 smithi204 ceph-aa240572-0159-11ef-bc93-c7b262605968-osd-8[51050]: 2024-04-23T10:42:35.287+0000 7f8d662cd700 -1 osd.8 116 heartbeat_check: no reply from 172.21.15.114:6806 osd.0 since back 2024-04-23T10:42:03.759281+0000 front 2024-04-23T10:42:03.759348+0000 (oldest deadline 2024-04-23T10:42:27.859455+0000) 2024-04-23T10:42:35.578 INFO:journalctl@ceph.osd.5.smithi204.stdout:Apr 23 10:42:35 smithi204 ceph-aa240572-0159-11ef-bc93-c7b262605968-osd-5[44334]: 2024-04-23T10:42:35.270+0000 7fbf0215c700 -1 osd.5 116 heartbeat_check: no reply from 172.21.15.114:6806 osd.0 since back 2024-04-23T10:42:02.285250+0000 front 2024-04-23T10:42:02.285119+0000 (oldest deadline 2024-04-23T10:42:28.185247+0000) 2024-04-23T10:42:35.578 INFO:journalctl@ceph.osd.5.smithi204.stdout:Apr 23 10:42:35 smithi204 ceph-aa240572-0159-11ef-bc93-c7b262605968-osd-5[44334]: 2024-04-23T10:42:35.270+0000 7fbf0215c700 -1 osd.5 116 heartbeat_check: no reply from 172.21.15.119:6804 osd.1 since back 2024-04-23T10:42:09.886215+0000 front 2024-04-23T10:42:09.886359+0000 (oldest deadline 2024-04-23T10:42:35.186188+0000) 2024-04-23T10:42:35.754 INFO:journalctl@ceph.osd.9.smithi114.stdout:Apr 23 10:42:35 smithi114 ceph-aa240572-0159-11ef-bc93-c7b262605968-osd-9[62058]: 2024-04-23T10:42:35.395+0000 7feeb4bc7700 -1 osd.9 116 heartbeat_check: no reply from 172.21.15.114:6806 osd.0 since back 2024-04-23T10:42:04.136736+0000 front 2024-04-23T10:42:04.136764+0000 (oldest deadline 2024-04-23T10:42:28.836926+0000) 2024-04-23T10:42:35.754 INFO:journalctl@ceph.osd.9.smithi114.stdout:Apr 23 10:42:35 smithi114 ceph-aa240572-0159-11ef-bc93-c7b262605968-osd-9[62058]: 2024-04-23T10:42:35.395+0000 7feeb4bc7700 -1 osd.9 116 heartbeat_check: no reply from 172.21.15.119:6804 osd.1 since back 2024-04-23T10:42:08.837665+0000 front 2024-04-23T10:42:08.837503+0000 (oldest deadline 2024-04-23T10:42:34.737571+0000) 2024-04-23T10:42:35.755 INFO:journalctl@ceph.osd.6.smithi114.stdout:Apr 23 10:42:35 smithi114 ceph-aa240572-0159-11ef-bc93-c7b262605968-osd-6[55044]: 2024-04-23T10:42:35.602+0000 7fdba4fa4700 -1 osd.6 116 heartbeat_check: no reply from 172.21.15.114:6806 osd.0 since back 2024-04-23T10:42:01.662289+0000 front 2024-04-23T10:42:01.662441+0000 (oldest deadline 2024-04-23T10:42:26.362236+0000) 2024-04-23T10:42:35.755 INFO:journalctl@ceph.osd.6.smithi114.stdout:Apr 23 10:42:35 smithi114 ceph-aa240572-0159-11ef-bc93-c7b262605968-osd-6[55044]: 2024-04-23T10:42:35.602+0000 7fdba4fa4700 -1 osd.6 116 heartbeat_check: no reply from 172.21.15.119:6804 osd.1 since back 2024-04-23T10:42:09.763752+0000 front 2024-04-23T10:42:09.763645+0000 (oldest deadline 2024-04-23T10:42:35.063704+0000) 2024-04-23T10:42:35.992 INFO:journalctl@ceph.osd.4.smithi119.stdout:Apr 23 10:42:35 smithi119 ceph-aa240572-0159-11ef-bc93-c7b262605968-osd-4[43889]: 2024-04-23T10:42:35.551+0000 7f4e9af98700 -1 osd.4 116 heartbeat_check: no reply from 172.21.15.119:6804 osd.1 since back 2024-04-23T10:42:09.563044+0000 front 2024-04-23T10:42:09.563092+0000 (oldest deadline 2024-04-23T10:42:34.863196+0000) 2024-04-23T10:42:35.992 INFO:journalctl@ceph.osd.10.smithi119.stdout:Apr 23 10:42:35 smithi119 ceph-aa240572-0159-11ef-bc93-c7b262605968-osd-10[57454]: 2024-04-23T10:42:35.526+0000 7fb1849bf700 -1 osd.10 116 heartbeat_check: no reply from 172.21.15.114:6806 osd.0 since back 2024-04-23T10:42:01.813174+0000 front 2024-04-23T10:42:01.813138+0000 (oldest deadline 2024-04-23T10:42:27.113300+0000) 2024-04-23T10:42:35.993 INFO:journalctl@ceph.osd.10.smithi119.stdout:Apr 23 10:42:35 smithi119 ceph-aa240572-0159-11ef-bc93-c7b262605968-osd-10[57454]: 2024-04-23T10:42:35.526+0000 7fb1849bf700 -1 osd.10 116 heartbeat_check: no reply from 172.21.15.119:6804 osd.1 since back 2024-04-23T10:42:10.614599+0000 front 2024-04-23T10:42:10.614622+0000 (oldest deadline 2024-04-23T10:42:34.114881+0000) 2024-04-23T10:42:36.281 INFO:journalctl@ceph.osd.11.smithi204.stdout:Apr 23 10:42:35 smithi204 ceph-aa240572-0159-11ef-bc93-c7b262605968-osd-11[57644]: 2024-04-23T10:42:35.894+0000 7fcbc4b63700 -1 osd.11 116 heartbeat_check: no reply from 172.21.15.114:6806 osd.0 since back 2024-04-23T10:42:00.675680+0000 front 2024-04-23T10:42:00.675649+0000 (oldest deadline 2024-04-23T10:42:25.876359+0000) 2024-04-23T10:42:36.282 INFO:journalctl@ceph.osd.11.smithi204.stdout:Apr 23 10:42:35 smithi204 ceph-aa240572-0159-11ef-bc93-c7b262605968-osd-11[57644]: 2024-04-23T10:42:35.894+0000 7fcbc4b63700 -1 osd.11 116 heartbeat_check: no reply from 172.21.15.119:6804 osd.1 since back 2024-04-23T10:42:09.278512+0000 front 2024-04-23T10:42:09.278494+0000 (oldest deadline 2024-04-23T10:42:33.378557+0000) 2024-04-23T10:42:36.566 INFO:journalctl@ceph.osd.8.smithi204.stdout:Apr 23 10:42:36 smithi204 ceph-aa240572-0159-11ef-bc93-c7b262605968-osd-8[51050]: 2024-04-23T10:42:36.283+0000 7f8d662cd700 -1 osd.8 116 heartbeat_check: no reply from 172.21.15.114:6806 osd.0 since back 2024-04-23T10:42:03.759281+0000 front 2024-04-23T10:42:03.759348+0000 (oldest deadline 2024-04-23T10:42:27.859455+0000) 2024-04-23T10:42:36.566 INFO:journalctl@ceph.osd.5.smithi204.stdout:Apr 23 10:42:36 smithi204 ceph-aa240572-0159-11ef-bc93-c7b262605968-osd-5[44334]: 2024-04-23T10:42:36.300+0000 7fbf0215c700 -1 osd.5 116 heartbeat_check: no reply from 172.21.15.114:6806 osd.0 since back 2024-04-23T10:42:02.285250+0000 front 2024-04-23T10:42:02.285119+0000 (oldest deadline 2024-04-23T10:42:28.185247+0000) 2024-04-23T10:42:36.567 INFO:journalctl@ceph.osd.5.smithi204.stdout:Apr 23 10:42:36 smithi204 ceph-aa240572-0159-11ef-bc93-c7b262605968-osd-5[44334]: 2024-04-23T10:42:36.301+0000 7fbf0215c700 -1 osd.5 116 heartbeat_check: no reply from 172.21.15.119:6804 osd.1 since back 2024-04-23T10:42:09.886215+0000 front 2024-04-23T10:42:09.886359+0000 (oldest deadline 2024-04-23T10:42:35.186188+0000) 2024-04-23T10:42:36.615 INFO:journalctl@ceph.osd.9.smithi114.stdout:Apr 23 10:42:36 smithi114 ceph-aa240572-0159-11ef-bc93-c7b262605968-osd-9[62058]: 2024-04-23T10:42:36.347+0000 7feeb4bc7700 -1 osd.9 116 heartbeat_check: no reply from 172.21.15.114:6806 osd.0 since back 2024-04-23T10:42:04.136736+0000 front 2024-04-23T10:42:04.136764+0000 (oldest deadline 2024-04-23T10:42:28.836926+0000) 2024-04-23T10:42:36.615 INFO:journalctl@ceph.osd.9.smithi114.stdout:Apr 23 10:42:36 smithi114 ceph-aa240572-0159-11ef-bc93-c7b262605968-osd-9[62058]: 2024-04-23T10:42:36.347+0000 7feeb4bc7700 -1 osd.9 116 heartbeat_check: no reply from 172.21.15.119:6804 osd.1 since back 2024-04-23T10:42:08.837665+0000 front 2024-04-23T10:42:08.837503+0000 (oldest deadline 2024-04-23T10:42:34.737571+0000) 2024-04-23T10:42:36.742 INFO:journalctl@ceph.osd.4.smithi119.stdout:Apr 23 10:42:36 smithi119 ceph-aa240572-0159-11ef-bc93-c7b262605968-osd-4[43889]: 2024-04-23T10:42:36.599+0000 7f4e9af98700 -1 osd.4 116 heartbeat_check: no reply from 172.21.15.119:6804 osd.1 since back 2024-04-23T10:42:09.563044+0000 front 2024-04-23T10:42:09.563092+0000 (oldest deadline 2024-04-23T10:42:34.863196+0000) 2024-04-23T10:42:36.743 INFO:journalctl@ceph.osd.10.smithi119.stdout:Apr 23 10:42:36 smithi119 ceph-aa240572-0159-11ef-bc93-c7b262605968-osd-10[57454]: 2024-04-23T10:42:36.490+0000 7fb1849bf700 -1 osd.10 116 heartbeat_check: no reply from 172.21.15.114:6806 osd.0 since back 2024-04-23T10:42:01.813174+0000 front 2024-04-23T10:42:01.813138+0000 (oldest deadline 2024-04-23T10:42:27.113300+0000) 2024-04-23T10:42:36.743 INFO:journalctl@ceph.osd.10.smithi119.stdout:Apr 23 10:42:36 smithi119 ceph-aa240572-0159-11ef-bc93-c7b262605968-osd-10[57454]: 2024-04-23T10:42:36.490+0000 7fb1849bf700 -1 osd.10 116 heartbeat_check: no reply from 172.21.15.119:6804 osd.1 since back 2024-04-23T10:42:10.614599+0000 front 2024-04-23T10:42:10.614622+0000 (oldest deadline 2024-04-23T10:42:34.114881+0000) 2024-04-23T10:42:37.004 INFO:journalctl@ceph.osd.6.smithi114.stdout:Apr 23 10:42:36 smithi114 ceph-aa240572-0159-11ef-bc93-c7b262605968-osd-6[55044]: 2024-04-23T10:42:36.616+0000 7fdba4fa4700 -1 osd.6 116 heartbeat_check: no reply from 172.21.15.114:6806 osd.0 since back 2024-04-23T10:42:01.662289+0000 front 2024-04-23T10:42:01.662441+0000 (oldest deadline 2024-04-23T10:42:26.362236+0000) 2024-04-23T10:42:37.004 INFO:journalctl@ceph.osd.6.smithi114.stdout:Apr 23 10:42:36 smithi114 ceph-aa240572-0159-11ef-bc93-c7b262605968-osd-6[55044]: 2024-04-23T10:42:36.616+0000 7fdba4fa4700 -1 osd.6 116 heartbeat_check: no reply from 172.21.15.119:6804 osd.1 since back 2024-04-23T10:42:09.763752+0000 front 2024-04-23T10:42:09.763645+0000 (oldest deadline 2024-04-23T10:42:35.063704+0000) 2024-04-23T10:42:37.246 INFO:journalctl@ceph.osd.11.smithi204.stdout:Apr 23 10:42:36 smithi204 ceph-aa240572-0159-11ef-bc93-c7b262605968-osd-11[57644]: 2024-04-23T10:42:36.936+0000 7fcbc4b63700 -1 osd.11 116 heartbeat_check: no reply from 172.21.15.114:6806 osd.0 since back 2024-04-23T10:42:00.675680+0000 front 2024-04-23T10:42:00.675649+0000 (oldest deadline 2024-04-23T10:42:25.876359+0000) 2024-04-23T10:42:37.246 INFO:journalctl@ceph.osd.11.smithi204.stdout:Apr 23 10:42:36 smithi204 ceph-aa240572-0159-11ef-bc93-c7b262605968-osd-11[57644]: 2024-04-23T10:42:36.936+0000 7fcbc4b63700 -1 osd.11 116 heartbeat_check: no reply from 172.21.15.119:6804 osd.1 since back 2024-04-23T10:42:09.278512+0000 front 2024-04-23T10:42:09.278494+0000 (oldest deadline 2024-04-23T10:42:33.378557+0000) 2024-04-23T10:42:37.501 INFO:journalctl@ceph.osd.8.smithi204.stdout:Apr 23 10:42:37 smithi204 ceph-aa240572-0159-11ef-bc93-c7b262605968-osd-8[51050]: 2024-04-23T10:42:37.247+0000 7f8d662cd700 -1 osd.8 116 heartbeat_check: no reply from 172.21.15.114:6806 osd.0 since back 2024-04-23T10:42:03.759281+0000 front 2024-04-23T10:42:03.759348+0000 (oldest deadline 2024-04-23T10:42:27.859455+0000) 2024-04-23T10:42:37.502 INFO:journalctl@ceph.osd.5.smithi204.stdout:Apr 23 10:42:37 smithi204 ceph-aa240572-0159-11ef-bc93-c7b262605968-osd-5[44334]: 2024-04-23T10:42:37.319+0000 7fbf0215c700 -1 osd.5 116 heartbeat_check: no reply from 172.21.15.114:6806 osd.0 since back 2024-04-23T10:42:02.285250+0000 front 2024-04-23T10:42:02.285119+0000 (oldest deadline 2024-04-23T10:42:28.185247+0000) 2024-04-23T10:42:37.502 INFO:journalctl@ceph.osd.5.smithi204.stdout:Apr 23 10:42:37 smithi204 ceph-aa240572-0159-11ef-bc93-c7b262605968-osd-5[44334]: 2024-04-23T10:42:37.319+0000 7fbf0215c700 -1 osd.5 116 heartbeat_check: no reply from 172.21.15.119:6804 osd.1 since back 2024-04-23T10:42:09.886215+0000 front 2024-04-23T10:42:09.886359+0000 (oldest deadline 2024-04-23T10:42:35.186188+0000) 2024-04-23T10:42:37.613 INFO:journalctl@ceph.osd.10.smithi119.stdout:Apr 23 10:42:37 smithi119 ceph-aa240572-0159-11ef-bc93-c7b262605968-osd-10[57454]: 2024-04-23T10:42:37.480+0000 7fb1849bf700 -1 osd.10 116 heartbeat_check: no reply from 172.21.15.114:6806 osd.0 since back 2024-04-23T10:42:01.813174+0000 front 2024-04-23T10:42:01.813138+0000 (oldest deadline 2024-04-23T10:42:27.113300+0000) 2024-04-23T10:42:37.613 INFO:journalctl@ceph.osd.10.smithi119.stdout:Apr 23 10:42:37 smithi119 ceph-aa240572-0159-11ef-bc93-c7b262605968-osd-10[57454]: 2024-04-23T10:42:37.480+0000 7fb1849bf700 -1 osd.10 116 heartbeat_check: no reply from 172.21.15.119:6804 osd.1 since back 2024-04-23T10:42:10.614599+0000 front 2024-04-23T10:42:10.614622+0000 (oldest deadline 2024-04-23T10:42:34.114881+0000) 2024-04-23T10:42:37.613 INFO:journalctl@ceph.osd.4.smithi119.stdout:Apr 23 10:42:37 smithi119 podman[69640]: 2024-04-23 10:42:37.322968081 +0000 UTC m=+5.620609060 container died 350ca9fa13c198f642f57a2dd795e3f10730f339c6ec42a8188dafc46280e6fb (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:068a558e3af4932cf1c60dd321c0bdf500075eb52ea1bf455e0973ba114fbfd4, name=ceph-aa240572-0159-11ef-bc93-c7b262605968-osd-4, GIT_BRANCH=HEAD, ceph=True, io.buildah.version=1.33.5, org.label-schema.vendor=CentOS, org.label-schema.name=CentOS Stream 8 Base Image, GIT_CLEAN=True, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=wip-lusov-quiescer-d69f13d, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=, org.label-schema.build-date=20240416, org.label-schema.schema-version=1.0) 2024-04-23T10:42:37.754 INFO:journalctl@ceph.osd.9.smithi114.stdout:Apr 23 10:42:37 smithi114 ceph-aa240572-0159-11ef-bc93-c7b262605968-osd-9[62058]: 2024-04-23T10:42:37.382+0000 7feeb4bc7700 -1 osd.9 116 heartbeat_check: no reply from 172.21.15.114:6806 osd.0 since back 2024-04-23T10:42:04.136736+0000 front 2024-04-23T10:42:04.136764+0000 (oldest deadline 2024-04-23T10:42:28.836926+0000) 2024-04-23T10:42:37.754 INFO:journalctl@ceph.osd.9.smithi114.stdout:Apr 23 10:42:37 smithi114 ceph-aa240572-0159-11ef-bc93-c7b262605968-osd-9[62058]: 2024-04-23T10:42:37.382+0000 7feeb4bc7700 -1 osd.9 116 heartbeat_check: no reply from 172.21.15.119:6804 osd.1 since back 2024-04-23T10:42:08.837665+0000 front 2024-04-23T10:42:08.837503+0000 (oldest deadline 2024-04-23T10:42:34.737571+0000) 2024-04-23T10:42:37.755 INFO:journalctl@ceph.osd.6.smithi114.stdout:Apr 23 10:42:37 smithi114 ceph-aa240572-0159-11ef-bc93-c7b262605968-osd-6[55044]: 2024-04-23T10:42:37.604+0000 7fdba4fa4700 -1 osd.6 116 heartbeat_check: no reply from 172.21.15.114:6806 osd.0 since back 2024-04-23T10:42:01.662289+0000 front 2024-04-23T10:42:01.662441+0000 (oldest deadline 2024-04-23T10:42:26.362236+0000) 2024-04-23T10:42:37.755 INFO:journalctl@ceph.osd.6.smithi114.stdout:Apr 23 10:42:37 smithi114 ceph-aa240572-0159-11ef-bc93-c7b262605968-osd-6[55044]: 2024-04-23T10:42:37.604+0000 7fdba4fa4700 -1 osd.6 116 heartbeat_check: no reply from 172.21.15.119:6804 osd.1 since back 2024-04-23T10:42:09.763752+0000 front 2024-04-23T10:42:09.763645+0000 (oldest deadline 2024-04-23T10:42:35.063704+0000) 2024-04-23T10:42:37.992 INFO:journalctl@ceph.osd.4.smithi119.stdout:Apr 23 10:42:37 smithi119 podman[69640]: 2024-04-23 10:42:37.713430316 +0000 UTC m=+6.011071298 container cleanup 350ca9fa13c198f642f57a2dd795e3f10730f339c6ec42a8188dafc46280e6fb (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:068a558e3af4932cf1c60dd321c0bdf500075eb52ea1bf455e0973ba114fbfd4, name=ceph-aa240572-0159-11ef-bc93-c7b262605968-osd-4, org.label-schema.schema-version=1.0, io.buildah.version=1.33.5, org.label-schema.build-date=20240416, CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 8 Base Image, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, ceph=True, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=wip-lusov-quiescer-d69f13d, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2, GIT_CLEAN=True, GIT_BRANCH=HEAD) 2024-04-23T10:42:37.992 INFO:journalctl@ceph.osd.4.smithi119.stdout:Apr 23 10:42:37 smithi119 bash[69640]: ceph-aa240572-0159-11ef-bc93-c7b262605968-osd-4 2024-04-23T10:42:38.279 INFO:journalctl@ceph.osd.11.smithi204.stdout:Apr 23 10:42:37 smithi204 ceph-aa240572-0159-11ef-bc93-c7b262605968-osd-11[57644]: 2024-04-23T10:42:37.903+0000 7fcbc4b63700 -1 osd.11 116 heartbeat_check: no reply from 172.21.15.114:6806 osd.0 since back 2024-04-23T10:42:00.675680+0000 front 2024-04-23T10:42:00.675649+0000 (oldest deadline 2024-04-23T10:42:25.876359+0000) 2024-04-23T10:42:38.279 INFO:journalctl@ceph.osd.11.smithi204.stdout:Apr 23 10:42:37 smithi204 ceph-aa240572-0159-11ef-bc93-c7b262605968-osd-11[57644]: 2024-04-23T10:42:37.903+0000 7fcbc4b63700 -1 osd.11 116 heartbeat_check: no reply from 172.21.15.119:6804 osd.1 since back 2024-04-23T10:42:09.278512+0000 front 2024-04-23T10:42:09.278494+0000 (oldest deadline 2024-04-23T10:42:33.378557+0000) 2024-04-23T10:42:38.329 INFO:journalctl@ceph.osd.4.smithi119.stdout:Apr 23 10:42:37 smithi119 podman[69657]: 2024-04-23 10:42:37.997930991 +0000 UTC m=+0.667059271 container remove 350ca9fa13c198f642f57a2dd795e3f10730f339c6ec42a8188dafc46280e6fb (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:068a558e3af4932cf1c60dd321c0bdf500075eb52ea1bf455e0973ba114fbfd4, name=ceph-aa240572-0159-11ef-bc93-c7b262605968-osd-4, ceph=True, GIT_CLEAN=True, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, RELEASE=wip-lusov-quiescer-d69f13d, GIT_BRANCH=HEAD, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240416, org.label-schema.name=CentOS Stream 8 Base Image, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.33.5) 2024-04-23T10:42:38.578 INFO:journalctl@ceph.osd.5.smithi204.stdout:Apr 23 10:42:38 smithi204 ceph-aa240572-0159-11ef-bc93-c7b262605968-osd-5[44334]: 2024-04-23T10:42:38.366+0000 7fbf0215c700 -1 osd.5 116 heartbeat_check: no reply from 172.21.15.114:6806 osd.0 since back 2024-04-23T10:42:02.285250+0000 front 2024-04-23T10:42:02.285119+0000 (oldest deadline 2024-04-23T10:42:28.185247+0000) 2024-04-23T10:42:38.578 INFO:journalctl@ceph.osd.5.smithi204.stdout:Apr 23 10:42:38 smithi204 ceph-aa240572-0159-11ef-bc93-c7b262605968-osd-5[44334]: 2024-04-23T10:42:38.366+0000 7fbf0215c700 -1 osd.5 116 heartbeat_check: no reply from 172.21.15.119:6804 osd.1 since back 2024-04-23T10:42:09.886215+0000 front 2024-04-23T10:42:09.886359+0000 (oldest deadline 2024-04-23T10:42:35.186188+0000) 2024-04-23T10:42:38.578 INFO:journalctl@ceph.osd.8.smithi204.stdout:Apr 23 10:42:38 smithi204 ceph-aa240572-0159-11ef-bc93-c7b262605968-osd-8[51050]: 2024-04-23T10:42:38.280+0000 7f8d662cd700 -1 osd.8 116 heartbeat_check: no reply from 172.21.15.114:6806 osd.0 since back 2024-04-23T10:42:03.759281+0000 front 2024-04-23T10:42:03.759348+0000 (oldest deadline 2024-04-23T10:42:27.859455+0000) 2024-04-23T10:42:38.579 INFO:journalctl@ceph.osd.8.smithi204.stdout:Apr 23 10:42:38 smithi204 ceph-aa240572-0159-11ef-bc93-c7b262605968-osd-8[51050]: 2024-04-23T10:42:38.280+0000 7f8d662cd700 -1 osd.8 116 heartbeat_check: no reply from 172.21.15.119:6804 osd.1 since back 2024-04-23T10:42:09.560739+0000 front 2024-04-23T10:42:09.560541+0000 (oldest deadline 2024-04-23T10:42:37.761119+0000) 2024-04-23T10:42:38.608 INFO:journalctl@ceph.osd.10.smithi119.stdout:Apr 23 10:42:38 smithi119 ceph-aa240572-0159-11ef-bc93-c7b262605968-osd-10[57454]: 2024-04-23T10:42:38.502+0000 7fb1849bf700 -1 osd.10 116 heartbeat_check: no reply from 172.21.15.114:6806 osd.0 since back 2024-04-23T10:42:01.813174+0000 front 2024-04-23T10:42:01.813138+0000 (oldest deadline 2024-04-23T10:42:27.113300+0000) 2024-04-23T10:42:38.608 INFO:journalctl@ceph.osd.10.smithi119.stdout:Apr 23 10:42:38 smithi119 ceph-aa240572-0159-11ef-bc93-c7b262605968-osd-10[57454]: 2024-04-23T10:42:38.502+0000 7fb1849bf700 -1 osd.10 116 heartbeat_check: no reply from 172.21.15.119:6804 osd.1 since back 2024-04-23T10:42:10.614599+0000 front 2024-04-23T10:42:10.614622+0000 (oldest deadline 2024-04-23T10:42:34.114881+0000) 2024-04-23T10:42:38.609 INFO:journalctl@ceph.osd.4.smithi119.stdout:Apr 23 10:42:38 smithi119 podman[69742]: 2024-04-23 10:42:38.234720213 +0000 UTC m=+0.017939612 image pull da4b004c09e820713483d039bc76fbd660be678f2ea3c03e059a6161b74dab76 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:068a558e3af4932cf1c60dd321c0bdf500075eb52ea1bf455e0973ba114fbfd4 2024-04-23T10:42:38.609 INFO:journalctl@ceph.osd.4.smithi119.stdout:Apr 23 10:42:38 smithi119 podman[69742]: 2024-04-23 10:42:38.340421533 +0000 UTC m=+0.123640943 container create 8fe0e12c39896beee4ccca89c032d514e20841545f69cbdd4c18b9ed10c55a7e (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:068a558e3af4932cf1c60dd321c0bdf500075eb52ea1bf455e0973ba114fbfd4, name=ceph-aa240572-0159-11ef-bc93-c7b262605968-osd-4-deactivate, org.label-schema.build-date=20240416, io.buildah.version=1.33.5, ceph=True, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=, GIT_CLEAN=True, RELEASE=wip-lusov-quiescer-d69f13d, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 8 Base Image) 2024-04-23T10:42:38.609 INFO:journalctl@ceph.osd.4.smithi119.stdout:Apr 23 10:42:38 smithi119 podman[69742]: 2024-04-23 10:42:38.609778429 +0000 UTC m=+0.392997794 container init 8fe0e12c39896beee4ccca89c032d514e20841545f69cbdd4c18b9ed10c55a7e (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:068a558e3af4932cf1c60dd321c0bdf500075eb52ea1bf455e0973ba114fbfd4, name=ceph-aa240572-0159-11ef-bc93-c7b262605968-osd-4-deactivate, org.label-schema.schema-version=1.0, org.label-schema.license=GPLv2, io.buildah.version=1.33.5, CEPH_POINT_RELEASE=, org.label-schema.vendor=CentOS, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, ceph=True, GIT_CLEAN=True, org.label-schema.build-date=20240416, RELEASE=wip-lusov-quiescer-d69f13d, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 8 Base Image, maintainer=Guillaume Abrioux , GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-04-23T10:42:38.754 INFO:journalctl@ceph.osd.9.smithi114.stdout:Apr 23 10:42:38 smithi114 ceph-aa240572-0159-11ef-bc93-c7b262605968-osd-9[62058]: 2024-04-23T10:42:38.424+0000 7feeb4bc7700 -1 osd.9 116 heartbeat_check: no reply from 172.21.15.114:6806 osd.0 since back 2024-04-23T10:42:04.136736+0000 front 2024-04-23T10:42:04.136764+0000 (oldest deadline 2024-04-23T10:42:28.836926+0000) 2024-04-23T10:42:38.754 INFO:journalctl@ceph.osd.9.smithi114.stdout:Apr 23 10:42:38 smithi114 ceph-aa240572-0159-11ef-bc93-c7b262605968-osd-9[62058]: 2024-04-23T10:42:38.424+0000 7feeb4bc7700 -1 osd.9 116 heartbeat_check: no reply from 172.21.15.119:6804 osd.1 since back 2024-04-23T10:42:08.837665+0000 front 2024-04-23T10:42:08.837503+0000 (oldest deadline 2024-04-23T10:42:34.737571+0000) 2024-04-23T10:42:38.754 INFO:journalctl@ceph.osd.6.smithi114.stdout:Apr 23 10:42:38 smithi114 ceph-aa240572-0159-11ef-bc93-c7b262605968-osd-6[55044]: 2024-04-23T10:42:38.557+0000 7fdba4fa4700 -1 osd.6 116 heartbeat_check: no reply from 172.21.15.114:6806 osd.0 since back 2024-04-23T10:42:01.662289+0000 front 2024-04-23T10:42:01.662441+0000 (oldest deadline 2024-04-23T10:42:26.362236+0000) 2024-04-23T10:42:38.755 INFO:journalctl@ceph.osd.6.smithi114.stdout:Apr 23 10:42:38 smithi114 ceph-aa240572-0159-11ef-bc93-c7b262605968-osd-6[55044]: 2024-04-23T10:42:38.557+0000 7fdba4fa4700 -1 osd.6 116 heartbeat_check: no reply from 172.21.15.119:6804 osd.1 since back 2024-04-23T10:42:09.763752+0000 front 2024-04-23T10:42:09.763645+0000 (oldest deadline 2024-04-23T10:42:35.063704+0000) 2024-04-23T10:42:38.896 INFO:journalctl@ceph.osd.4.smithi119.stdout:Apr 23 10:42:38 smithi119 podman[69742]: 2024-04-23 10:42:38.613737235 +0000 UTC m=+0.396956606 container start 8fe0e12c39896beee4ccca89c032d514e20841545f69cbdd4c18b9ed10c55a7e (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:068a558e3af4932cf1c60dd321c0bdf500075eb52ea1bf455e0973ba114fbfd4, name=ceph-aa240572-0159-11ef-bc93-c7b262605968-osd-4-deactivate, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, GIT_CLEAN=True, RELEASE=wip-lusov-quiescer-d69f13d, org.label-schema.build-date=20240416, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.vendor=CentOS, io.buildah.version=1.33.5, ceph=True, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67) 2024-04-23T10:42:38.897 INFO:journalctl@ceph.osd.4.smithi119.stdout:Apr 23 10:42:38 smithi119 podman[69742]: 2024-04-23 10:42:38.692933144 +0000 UTC m=+0.476152521 container attach 8fe0e12c39896beee4ccca89c032d514e20841545f69cbdd4c18b9ed10c55a7e (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:068a558e3af4932cf1c60dd321c0bdf500075eb52ea1bf455e0973ba114fbfd4, name=ceph-aa240572-0159-11ef-bc93-c7b262605968-osd-4-deactivate, io.buildah.version=1.33.5, CEPH_POINT_RELEASE=, org.label-schema.schema-version=1.0, GIT_CLEAN=True, org.label-schema.build-date=20240416, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=wip-lusov-quiescer-d69f13d, org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2, ceph=True, org.label-schema.name=CentOS Stream 8 Base Image, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67) 2024-04-23T10:42:38.897 INFO:journalctl@ceph.osd.7.smithi119.stdout:Apr 23 10:42:38 smithi119 ceph-aa240572-0159-11ef-bc93-c7b262605968-osd-7[50859]: 2024-04-23T10:42:38.754+0000 7f015f194700 -1 osd.7 116 heartbeat_check: no reply from 172.21.15.119:6804 osd.1 since back 2024-04-23T10:42:09.664009+0000 front 2024-04-23T10:42:09.664029+0000 (oldest deadline 2024-04-23T10:42:38.464808+0000) 2024-04-23T10:42:39.202 INFO:journalctl@ceph.osd.4.smithi119.stdout:Apr 23 10:42:38 smithi119 podman[69742]: 2024-04-23 10:42:38.898591771 +0000 UTC m=+0.681811156 container died 8fe0e12c39896beee4ccca89c032d514e20841545f69cbdd4c18b9ed10c55a7e (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:068a558e3af4932cf1c60dd321c0bdf500075eb52ea1bf455e0973ba114fbfd4, name=ceph-aa240572-0159-11ef-bc93-c7b262605968-osd-4-deactivate, maintainer=Guillaume Abrioux , GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 8 Base Image, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, org.label-schema.build-date=20240416, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, GIT_CLEAN=True, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, io.buildah.version=1.33.5, ceph=True, RELEASE=wip-lusov-quiescer-d69f13d, org.label-schema.license=GPLv2) 2024-04-23T10:42:39.308 INFO:journalctl@ceph.osd.11.smithi204.stdout:Apr 23 10:42:38 smithi204 ceph-aa240572-0159-11ef-bc93-c7b262605968-osd-11[57644]: 2024-04-23T10:42:38.896+0000 7fcbc4b63700 -1 osd.11 116 heartbeat_check: no reply from 172.21.15.114:6806 osd.0 since back 2024-04-23T10:42:00.675680+0000 front 2024-04-23T10:42:00.675649+0000 (oldest deadline 2024-04-23T10:42:25.876359+0000) 2024-04-23T10:42:39.308 INFO:journalctl@ceph.osd.11.smithi204.stdout:Apr 23 10:42:38 smithi204 ceph-aa240572-0159-11ef-bc93-c7b262605968-osd-11[57644]: 2024-04-23T10:42:38.896+0000 7fcbc4b63700 -1 osd.11 116 heartbeat_check: no reply from 172.21.15.119:6804 osd.1 since back 2024-04-23T10:42:09.278512+0000 front 2024-04-23T10:42:09.278494+0000 (oldest deadline 2024-04-23T10:42:33.378557+0000) 2024-04-23T10:42:39.490 DEBUG:teuthology.orchestra.run.smithi119:> sudo pkill -f 'journalctl -f -n 0 -u ceph-aa240572-0159-11ef-bc93-c7b262605968@osd.4.service' 2024-04-23T10:42:39.526 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-04-23T10:42:39.527 INFO:tasks.cephadm.osd.4:Stopped osd.4 2024-04-23T10:42:39.527 INFO:tasks.cephadm.osd.5:Stopping osd.5... 2024-04-23T10:42:39.527 DEBUG:teuthology.orchestra.run.smithi204:> sudo systemctl stop ceph-aa240572-0159-11ef-bc93-c7b262605968@osd.5 2024-04-23T10:42:39.573 INFO:journalctl@ceph.osd.5.smithi204.stdout:Apr 23 10:42:39 smithi204 ceph-aa240572-0159-11ef-bc93-c7b262605968-osd-5[44334]: 2024-04-23T10:42:39.379+0000 7fbf0215c700 -1 osd.5 116 heartbeat_check: no reply from 172.21.15.114:6806 osd.0 since back 2024-04-23T10:42:02.285250+0000 front 2024-04-23T10:42:02.285119+0000 (oldest deadline 2024-04-23T10:42:28.185247+0000) 2024-04-23T10:42:39.574 INFO:journalctl@ceph.osd.5.smithi204.stdout:Apr 23 10:42:39 smithi204 ceph-aa240572-0159-11ef-bc93-c7b262605968-osd-5[44334]: 2024-04-23T10:42:39.379+0000 7fbf0215c700 -1 osd.5 116 heartbeat_check: no reply from 172.21.15.119:6804 osd.1 since back 2024-04-23T10:42:09.886215+0000 front 2024-04-23T10:42:09.886359+0000 (oldest deadline 2024-04-23T10:42:35.186188+0000) 2024-04-23T10:42:39.574 INFO:journalctl@ceph.osd.8.smithi204.stdout:Apr 23 10:42:39 smithi204 ceph-aa240572-0159-11ef-bc93-c7b262605968-osd-8[51050]: 2024-04-23T10:42:39.309+0000 7f8d662cd700 -1 osd.8 116 heartbeat_check: no reply from 172.21.15.114:6806 osd.0 since back 2024-04-23T10:42:03.759281+0000 front 2024-04-23T10:42:03.759348+0000 (oldest deadline 2024-04-23T10:42:27.859455+0000) 2024-04-23T10:42:39.574 INFO:journalctl@ceph.osd.8.smithi204.stdout:Apr 23 10:42:39 smithi204 ceph-aa240572-0159-11ef-bc93-c7b262605968-osd-8[51050]: 2024-04-23T10:42:39.309+0000 7f8d662cd700 -1 osd.8 116 heartbeat_check: no reply from 172.21.15.119:6804 osd.1 since back 2024-04-23T10:42:09.560739+0000 front 2024-04-23T10:42:09.560541+0000 (oldest deadline 2024-04-23T10:42:37.761119+0000) 2024-04-23T10:42:39.742 INFO:journalctl@ceph.osd.10.smithi119.stdout:Apr 23 10:42:39 smithi119 ceph-aa240572-0159-11ef-bc93-c7b262605968-osd-10[57454]: 2024-04-23T10:42:39.541+0000 7fb1849bf700 -1 osd.10 116 heartbeat_check: no reply from 172.21.15.114:6806 osd.0 since back 2024-04-23T10:42:01.813174+0000 front 2024-04-23T10:42:01.813138+0000 (oldest deadline 2024-04-23T10:42:27.113300+0000) 2024-04-23T10:42:39.742 INFO:journalctl@ceph.osd.10.smithi119.stdout:Apr 23 10:42:39 smithi119 ceph-aa240572-0159-11ef-bc93-c7b262605968-osd-10[57454]: 2024-04-23T10:42:39.541+0000 7fb1849bf700 -1 osd.10 116 heartbeat_check: no reply from 172.21.15.119:6804 osd.1 since back 2024-04-23T10:42:10.614599+0000 front 2024-04-23T10:42:10.614622+0000 (oldest deadline 2024-04-23T10:42:34.114881+0000) 2024-04-23T10:42:39.754 INFO:journalctl@ceph.osd.9.smithi114.stdout:Apr 23 10:42:39 smithi114 ceph-aa240572-0159-11ef-bc93-c7b262605968-osd-9[62058]: 2024-04-23T10:42:39.411+0000 7feeb4bc7700 -1 osd.9 116 heartbeat_check: no reply from 172.21.15.114:6806 osd.0 since back 2024-04-23T10:42:04.136736+0000 front 2024-04-23T10:42:04.136764+0000 (oldest deadline 2024-04-23T10:42:28.836926+0000) 2024-04-23T10:42:39.754 INFO:journalctl@ceph.osd.9.smithi114.stdout:Apr 23 10:42:39 smithi114 ceph-aa240572-0159-11ef-bc93-c7b262605968-osd-9[62058]: 2024-04-23T10:42:39.411+0000 7feeb4bc7700 -1 osd.9 116 heartbeat_check: no reply from 172.21.15.119:6804 osd.1 since back 2024-04-23T10:42:08.837665+0000 front 2024-04-23T10:42:08.837503+0000 (oldest deadline 2024-04-23T10:42:34.737571+0000) 2024-04-23T10:42:39.755 INFO:journalctl@ceph.osd.6.smithi114.stdout:Apr 23 10:42:39 smithi114 ceph-aa240572-0159-11ef-bc93-c7b262605968-osd-6[55044]: 2024-04-23T10:42:39.540+0000 7fdba4fa4700 -1 osd.6 116 heartbeat_check: no reply from 172.21.15.114:6806 osd.0 since back 2024-04-23T10:42:01.662289+0000 front 2024-04-23T10:42:01.662441+0000 (oldest deadline 2024-04-23T10:42:26.362236+0000) 2024-04-23T10:42:39.755 INFO:journalctl@ceph.osd.6.smithi114.stdout:Apr 23 10:42:39 smithi114 ceph-aa240572-0159-11ef-bc93-c7b262605968-osd-6[55044]: 2024-04-23T10:42:39.540+0000 7fdba4fa4700 -1 osd.6 116 heartbeat_check: no reply from 172.21.15.119:6804 osd.1 since back 2024-04-23T10:42:09.763752+0000 front 2024-04-23T10:42:09.763645+0000 (oldest deadline 2024-04-23T10:42:35.063704+0000) 2024-04-23T10:42:39.827 INFO:journalctl@ceph.osd.5.smithi204.stdout:Apr 23 10:42:39 smithi204 systemd[1]: Stopping Ceph osd.5 for aa240572-0159-11ef-bc93-c7b262605968... 2024-04-23T10:42:40.242 INFO:journalctl@ceph.osd.7.smithi119.stdout:Apr 23 10:42:39 smithi119 ceph-aa240572-0159-11ef-bc93-c7b262605968-osd-7[50859]: 2024-04-23T10:42:39.791+0000 7f015f194700 -1 osd.7 116 heartbeat_check: no reply from 172.21.15.119:6804 osd.1 since back 2024-04-23T10:42:09.664009+0000 front 2024-04-23T10:42:09.664029+0000 (oldest deadline 2024-04-23T10:42:38.464808+0000) 2024-04-23T10:42:40.283 INFO:journalctl@ceph.osd.11.smithi204.stdout:Apr 23 10:42:39 smithi204 ceph-aa240572-0159-11ef-bc93-c7b262605968-osd-11[57644]: 2024-04-23T10:42:39.863+0000 7fcbc4b63700 -1 osd.11 116 heartbeat_check: no reply from 172.21.15.114:6806 osd.0 since back 2024-04-23T10:42:00.675680+0000 front 2024-04-23T10:42:00.675649+0000 (oldest deadline 2024-04-23T10:42:25.876359+0000) 2024-04-23T10:42:40.283 INFO:journalctl@ceph.osd.11.smithi204.stdout:Apr 23 10:42:39 smithi204 ceph-aa240572-0159-11ef-bc93-c7b262605968-osd-11[57644]: 2024-04-23T10:42:39.863+0000 7fcbc4b63700 -1 osd.11 116 heartbeat_check: no reply from 172.21.15.119:6804 osd.1 since back 2024-04-23T10:42:09.278512+0000 front 2024-04-23T10:42:09.278494+0000 (oldest deadline 2024-04-23T10:42:33.378557+0000) 2024-04-23T10:42:40.284 INFO:journalctl@ceph.osd.5.smithi204.stdout:Apr 23 10:42:39 smithi204 ceph-aa240572-0159-11ef-bc93-c7b262605968-osd-5[44334]: 2024-04-23T10:42:39.843+0000 7fbf09b70700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.5 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-04-23T10:42:40.284 INFO:journalctl@ceph.osd.5.smithi204.stdout:Apr 23 10:42:39 smithi204 ceph-aa240572-0159-11ef-bc93-c7b262605968-osd-5[44334]: 2024-04-23T10:42:39.843+0000 7fbf09b70700 -1 osd.5 116 *** Got signal Terminated *** 2024-04-23T10:42:40.284 INFO:journalctl@ceph.osd.5.smithi204.stdout:Apr 23 10:42:39 smithi204 ceph-aa240572-0159-11ef-bc93-c7b262605968-osd-5[44334]: 2024-04-23T10:42:39.843+0000 7fbf09b70700 -1 osd.5 116 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-04-23T10:42:40.578 INFO:journalctl@ceph.osd.8.smithi204.stdout:Apr 23 10:42:40 smithi204 ceph-aa240572-0159-11ef-bc93-c7b262605968-osd-8[51050]: 2024-04-23T10:42:40.284+0000 7f8d662cd700 -1 osd.8 116 heartbeat_check: no reply from 172.21.15.114:6806 osd.0 since back 2024-04-23T10:42:03.759281+0000 front 2024-04-23T10:42:03.759348+0000 (oldest deadline 2024-04-23T10:42:27.859455+0000) 2024-04-23T10:42:40.578 INFO:journalctl@ceph.osd.8.smithi204.stdout:Apr 23 10:42:40 smithi204 ceph-aa240572-0159-11ef-bc93-c7b262605968-osd-8[51050]: 2024-04-23T10:42:40.284+0000 7f8d662cd700 -1 osd.8 116 heartbeat_check: no reply from 172.21.15.119:6804 osd.1 since back 2024-04-23T10:42:09.560739+0000 front 2024-04-23T10:42:09.560541+0000 (oldest deadline 2024-04-23T10:42:37.761119+0000) 2024-04-23T10:42:40.579 INFO:journalctl@ceph.osd.5.smithi204.stdout:Apr 23 10:42:40 smithi204 ceph-aa240572-0159-11ef-bc93-c7b262605968-osd-5[44334]: 2024-04-23T10:42:40.350+0000 7fbf0215c700 -1 osd.5 116 heartbeat_check: no reply from 172.21.15.114:6806 osd.0 since back 2024-04-23T10:42:02.285250+0000 front 2024-04-23T10:42:02.285119+0000 (oldest deadline 2024-04-23T10:42:28.185247+0000) 2024-04-23T10:42:40.579 INFO:journalctl@ceph.osd.5.smithi204.stdout:Apr 23 10:42:40 smithi204 ceph-aa240572-0159-11ef-bc93-c7b262605968-osd-5[44334]: 2024-04-23T10:42:40.350+0000 7fbf0215c700 -1 osd.5 116 heartbeat_check: no reply from 172.21.15.119:6804 osd.1 since back 2024-04-23T10:42:09.886215+0000 front 2024-04-23T10:42:09.886359+0000 (oldest deadline 2024-04-23T10:42:35.186188+0000) 2024-04-23T10:42:40.754 INFO:journalctl@ceph.osd.9.smithi114.stdout:Apr 23 10:42:40 smithi114 ceph-aa240572-0159-11ef-bc93-c7b262605968-osd-9[62058]: 2024-04-23T10:42:40.428+0000 7feeb4bc7700 -1 osd.9 116 heartbeat_check: no reply from 172.21.15.114:6806 osd.0 since back 2024-04-23T10:42:04.136736+0000 front 2024-04-23T10:42:04.136764+0000 (oldest deadline 2024-04-23T10:42:28.836926+0000) 2024-04-23T10:42:40.754 INFO:journalctl@ceph.osd.9.smithi114.stdout:Apr 23 10:42:40 smithi114 ceph-aa240572-0159-11ef-bc93-c7b262605968-osd-9[62058]: 2024-04-23T10:42:40.428+0000 7feeb4bc7700 -1 osd.9 116 heartbeat_check: no reply from 172.21.15.119:6804 osd.1 since back 2024-04-23T10:42:08.837665+0000 front 2024-04-23T10:42:08.837503+0000 (oldest deadline 2024-04-23T10:42:34.737571+0000) 2024-04-23T10:42:40.755 INFO:journalctl@ceph.osd.6.smithi114.stdout:Apr 23 10:42:40 smithi114 ceph-aa240572-0159-11ef-bc93-c7b262605968-osd-6[55044]: 2024-04-23T10:42:40.512+0000 7fdba4fa4700 -1 osd.6 116 heartbeat_check: no reply from 172.21.15.114:6806 osd.0 since back 2024-04-23T10:42:01.662289+0000 front 2024-04-23T10:42:01.662441+0000 (oldest deadline 2024-04-23T10:42:26.362236+0000) 2024-04-23T10:42:40.755 INFO:journalctl@ceph.osd.6.smithi114.stdout:Apr 23 10:42:40 smithi114 ceph-aa240572-0159-11ef-bc93-c7b262605968-osd-6[55044]: 2024-04-23T10:42:40.512+0000 7fdba4fa4700 -1 osd.6 116 heartbeat_check: no reply from 172.21.15.119:6804 osd.1 since back 2024-04-23T10:42:09.763752+0000 front 2024-04-23T10:42:09.763645+0000 (oldest deadline 2024-04-23T10:42:35.063704+0000) 2024-04-23T10:42:40.992 INFO:journalctl@ceph.osd.7.smithi119.stdout:Apr 23 10:42:40 smithi119 ceph-aa240572-0159-11ef-bc93-c7b262605968-osd-7[50859]: 2024-04-23T10:42:40.746+0000 7f015f194700 -1 osd.7 116 heartbeat_check: no reply from 172.21.15.119:6804 osd.1 since back 2024-04-23T10:42:09.664009+0000 front 2024-04-23T10:42:09.664029+0000 (oldest deadline 2024-04-23T10:42:38.464808+0000) 2024-04-23T10:42:40.992 INFO:journalctl@ceph.osd.10.smithi119.stdout:Apr 23 10:42:40 smithi119 ceph-aa240572-0159-11ef-bc93-c7b262605968-osd-10[57454]: 2024-04-23T10:42:40.554+0000 7fb1849bf700 -1 osd.10 116 heartbeat_check: no reply from 172.21.15.114:6806 osd.0 since back 2024-04-23T10:42:01.813174+0000 front 2024-04-23T10:42:01.813138+0000 (oldest deadline 2024-04-23T10:42:27.113300+0000) 2024-04-23T10:42:40.993 INFO:journalctl@ceph.osd.10.smithi119.stdout:Apr 23 10:42:40 smithi119 ceph-aa240572-0159-11ef-bc93-c7b262605968-osd-10[57454]: 2024-04-23T10:42:40.554+0000 7fb1849bf700 -1 osd.10 116 heartbeat_check: no reply from 172.21.15.119:6804 osd.1 since back 2024-04-23T10:42:10.614599+0000 front 2024-04-23T10:42:10.614622+0000 (oldest deadline 2024-04-23T10:42:34.114881+0000) 2024-04-23T10:42:41.248 INFO:journalctl@ceph.osd.11.smithi204.stdout:Apr 23 10:42:40 smithi204 ceph-aa240572-0159-11ef-bc93-c7b262605968-osd-11[57644]: 2024-04-23T10:42:40.870+0000 7fcbc4b63700 -1 osd.11 116 heartbeat_check: no reply from 172.21.15.114:6806 osd.0 since back 2024-04-23T10:42:00.675680+0000 front 2024-04-23T10:42:00.675649+0000 (oldest deadline 2024-04-23T10:42:25.876359+0000) 2024-04-23T10:42:41.249 INFO:journalctl@ceph.osd.11.smithi204.stdout:Apr 23 10:42:40 smithi204 ceph-aa240572-0159-11ef-bc93-c7b262605968-osd-11[57644]: 2024-04-23T10:42:40.870+0000 7fcbc4b63700 -1 osd.11 116 heartbeat_check: no reply from 172.21.15.119:6804 osd.1 since back 2024-04-23T10:42:09.278512+0000 front 2024-04-23T10:42:09.278494+0000 (oldest deadline 2024-04-23T10:42:33.378557+0000) 2024-04-23T10:42:41.578 INFO:journalctl@ceph.osd.8.smithi204.stdout:Apr 23 10:42:41 smithi204 ceph-aa240572-0159-11ef-bc93-c7b262605968-osd-8[51050]: 2024-04-23T10:42:41.250+0000 7f8d662cd700 -1 osd.8 116 heartbeat_check: no reply from 172.21.15.114:6806 osd.0 since back 2024-04-23T10:42:03.759281+0000 front 2024-04-23T10:42:03.759348+0000 (oldest deadline 2024-04-23T10:42:27.859455+0000) 2024-04-23T10:42:41.578 INFO:journalctl@ceph.osd.8.smithi204.stdout:Apr 23 10:42:41 smithi204 ceph-aa240572-0159-11ef-bc93-c7b262605968-osd-8[51050]: 2024-04-23T10:42:41.250+0000 7f8d662cd700 -1 osd.8 116 heartbeat_check: no reply from 172.21.15.119:6804 osd.1 since back 2024-04-23T10:42:09.560739+0000 front 2024-04-23T10:42:09.560541+0000 (oldest deadline 2024-04-23T10:42:37.761119+0000) 2024-04-23T10:42:41.578 INFO:journalctl@ceph.osd.5.smithi204.stdout:Apr 23 10:42:41 smithi204 ceph-aa240572-0159-11ef-bc93-c7b262605968-osd-5[44334]: 2024-04-23T10:42:41.351+0000 7fbf0215c700 -1 osd.5 116 heartbeat_check: no reply from 172.21.15.114:6806 osd.0 since back 2024-04-23T10:42:02.285250+0000 front 2024-04-23T10:42:02.285119+0000 (oldest deadline 2024-04-23T10:42:28.185247+0000) 2024-04-23T10:42:41.579 INFO:journalctl@ceph.osd.5.smithi204.stdout:Apr 23 10:42:41 smithi204 ceph-aa240572-0159-11ef-bc93-c7b262605968-osd-5[44334]: 2024-04-23T10:42:41.351+0000 7fbf0215c700 -1 osd.5 116 heartbeat_check: no reply from 172.21.15.119:6804 osd.1 since back 2024-04-23T10:42:09.886215+0000 front 2024-04-23T10:42:09.886359+0000 (oldest deadline 2024-04-23T10:42:35.186188+0000) 2024-04-23T10:42:41.754 INFO:journalctl@ceph.osd.6.smithi114.stdout:Apr 23 10:42:41 smithi114 ceph-aa240572-0159-11ef-bc93-c7b262605968-osd-6[55044]: 2024-04-23T10:42:41.496+0000 7fdba4fa4700 -1 osd.6 116 heartbeat_check: no reply from 172.21.15.114:6806 osd.0 since back 2024-04-23T10:42:01.662289+0000 front 2024-04-23T10:42:01.662441+0000 (oldest deadline 2024-04-23T10:42:26.362236+0000) 2024-04-23T10:42:41.754 INFO:journalctl@ceph.osd.6.smithi114.stdout:Apr 23 10:42:41 smithi114 ceph-aa240572-0159-11ef-bc93-c7b262605968-osd-6[55044]: 2024-04-23T10:42:41.496+0000 7fdba4fa4700 -1 osd.6 116 heartbeat_check: no reply from 172.21.15.119:6804 osd.1 since back 2024-04-23T10:42:09.763752+0000 front 2024-04-23T10:42:09.763645+0000 (oldest deadline 2024-04-23T10:42:35.063704+0000) 2024-04-23T10:42:41.754 INFO:journalctl@ceph.osd.9.smithi114.stdout:Apr 23 10:42:41 smithi114 ceph-aa240572-0159-11ef-bc93-c7b262605968-osd-9[62058]: 2024-04-23T10:42:41.392+0000 7feeb4bc7700 -1 osd.9 116 heartbeat_check: no reply from 172.21.15.114:6806 osd.0 since back 2024-04-23T10:42:04.136736+0000 front 2024-04-23T10:42:04.136764+0000 (oldest deadline 2024-04-23T10:42:28.836926+0000) 2024-04-23T10:42:41.754 INFO:journalctl@ceph.osd.9.smithi114.stdout:Apr 23 10:42:41 smithi114 ceph-aa240572-0159-11ef-bc93-c7b262605968-osd-9[62058]: 2024-04-23T10:42:41.392+0000 7feeb4bc7700 -1 osd.9 116 heartbeat_check: no reply from 172.21.15.119:6804 osd.1 since back 2024-04-23T10:42:08.837665+0000 front 2024-04-23T10:42:08.837503+0000 (oldest deadline 2024-04-23T10:42:34.737571+0000) 2024-04-23T10:42:41.992 INFO:journalctl@ceph.osd.7.smithi119.stdout:Apr 23 10:42:41 smithi119 ceph-aa240572-0159-11ef-bc93-c7b262605968-osd-7[50859]: 2024-04-23T10:42:41.707+0000 7f015f194700 -1 osd.7 116 heartbeat_check: no reply from 172.21.15.119:6804 osd.1 since back 2024-04-23T10:42:09.664009+0000 front 2024-04-23T10:42:09.664029+0000 (oldest deadline 2024-04-23T10:42:38.464808+0000) 2024-04-23T10:42:41.992 INFO:journalctl@ceph.osd.10.smithi119.stdout:Apr 23 10:42:41 smithi119 ceph-aa240572-0159-11ef-bc93-c7b262605968-osd-10[57454]: 2024-04-23T10:42:41.535+0000 7fb1849bf700 -1 osd.10 116 heartbeat_check: no reply from 172.21.15.114:6806 osd.0 since back 2024-04-23T10:42:01.813174+0000 front 2024-04-23T10:42:01.813138+0000 (oldest deadline 2024-04-23T10:42:27.113300+0000) 2024-04-23T10:42:41.992 INFO:journalctl@ceph.osd.10.smithi119.stdout:Apr 23 10:42:41 smithi119 ceph-aa240572-0159-11ef-bc93-c7b262605968-osd-10[57454]: 2024-04-23T10:42:41.535+0000 7fb1849bf700 -1 osd.10 116 heartbeat_check: no reply from 172.21.15.119:6804 osd.1 since back 2024-04-23T10:42:10.614599+0000 front 2024-04-23T10:42:10.614622+0000 (oldest deadline 2024-04-23T10:42:34.114881+0000) 2024-04-23T10:42:42.280 INFO:journalctl@ceph.osd.11.smithi204.stdout:Apr 23 10:42:41 smithi204 ceph-aa240572-0159-11ef-bc93-c7b262605968-osd-11[57644]: 2024-04-23T10:42:41.905+0000 7fcbc4b63700 -1 osd.11 116 heartbeat_check: no reply from 172.21.15.114:6806 osd.0 since back 2024-04-23T10:42:00.675680+0000 front 2024-04-23T10:42:00.675649+0000 (oldest deadline 2024-04-23T10:42:25.876359+0000) 2024-04-23T10:42:42.281 INFO:journalctl@ceph.osd.11.smithi204.stdout:Apr 23 10:42:41 smithi204 ceph-aa240572-0159-11ef-bc93-c7b262605968-osd-11[57644]: 2024-04-23T10:42:41.905+0000 7fcbc4b63700 -1 osd.11 116 heartbeat_check: no reply from 172.21.15.119:6804 osd.1 since back 2024-04-23T10:42:09.278512+0000 front 2024-04-23T10:42:09.278494+0000 (oldest deadline 2024-04-23T10:42:33.378557+0000) 2024-04-23T10:42:42.281 INFO:journalctl@ceph.osd.11.smithi204.stdout:Apr 23 10:42:41 smithi204 ceph-aa240572-0159-11ef-bc93-c7b262605968-osd-11[57644]: 2024-04-23T10:42:41.905+0000 7fcbc4b63700 -1 osd.11 116 heartbeat_check: no reply from 172.21.15.204:6804 osd.2 since back 2024-04-23T10:42:18.679708+0000 front 2024-04-23T10:42:18.679577+0000 (oldest deadline 2024-04-23T10:42:41.579766+0000) 2024-04-23T10:42:42.578 INFO:journalctl@ceph.osd.8.smithi204.stdout:Apr 23 10:42:42 smithi204 ceph-aa240572-0159-11ef-bc93-c7b262605968-osd-8[51050]: 2024-04-23T10:42:42.282+0000 7f8d662cd700 -1 osd.8 116 heartbeat_check: no reply from 172.21.15.114:6806 osd.0 since back 2024-04-23T10:42:03.759281+0000 front 2024-04-23T10:42:03.759348+0000 (oldest deadline 2024-04-23T10:42:27.859455+0000) 2024-04-23T10:42:42.578 INFO:journalctl@ceph.osd.8.smithi204.stdout:Apr 23 10:42:42 smithi204 ceph-aa240572-0159-11ef-bc93-c7b262605968-osd-8[51050]: 2024-04-23T10:42:42.282+0000 7f8d662cd700 -1 osd.8 116 heartbeat_check: no reply from 172.21.15.119:6804 osd.1 since back 2024-04-23T10:42:09.560739+0000 front 2024-04-23T10:42:09.560541+0000 (oldest deadline 2024-04-23T10:42:37.761119+0000) 2024-04-23T10:42:42.578 INFO:journalctl@ceph.osd.5.smithi204.stdout:Apr 23 10:42:42 smithi204 ceph-aa240572-0159-11ef-bc93-c7b262605968-osd-5[44334]: 2024-04-23T10:42:42.401+0000 7fbf0215c700 -1 osd.5 116 heartbeat_check: no reply from 172.21.15.114:6806 osd.0 since back 2024-04-23T10:42:02.285250+0000 front 2024-04-23T10:42:02.285119+0000 (oldest deadline 2024-04-23T10:42:28.185247+0000) 2024-04-23T10:42:42.578 INFO:journalctl@ceph.osd.5.smithi204.stdout:Apr 23 10:42:42 smithi204 ceph-aa240572-0159-11ef-bc93-c7b262605968-osd-5[44334]: 2024-04-23T10:42:42.401+0000 7fbf0215c700 -1 osd.5 116 heartbeat_check: no reply from 172.21.15.119:6804 osd.1 since back 2024-04-23T10:42:09.886215+0000 front 2024-04-23T10:42:09.886359+0000 (oldest deadline 2024-04-23T10:42:35.186188+0000) 2024-04-23T10:42:42.578 INFO:journalctl@ceph.osd.5.smithi204.stdout:Apr 23 10:42:42 smithi204 ceph-aa240572-0159-11ef-bc93-c7b262605968-osd-5[44334]: 2024-04-23T10:42:42.401+0000 7fbf0215c700 -1 osd.5 116 heartbeat_check: no reply from 172.21.15.204:6804 osd.2 since back 2024-04-23T10:42:20.487575+0000 front 2024-04-23T10:42:20.487602+0000 (oldest deadline 2024-04-23T10:42:41.587793+0000) 2024-04-23T10:42:42.754 INFO:journalctl@ceph.osd.6.smithi114.stdout:Apr 23 10:42:42 smithi114 ceph-aa240572-0159-11ef-bc93-c7b262605968-osd-6[55044]: 2024-04-23T10:42:42.529+0000 7fdba4fa4700 -1 osd.6 116 heartbeat_check: no reply from 172.21.15.114:6806 osd.0 since back 2024-04-23T10:42:01.662289+0000 front 2024-04-23T10:42:01.662441+0000 (oldest deadline 2024-04-23T10:42:26.362236+0000) 2024-04-23T10:42:42.754 INFO:journalctl@ceph.osd.6.smithi114.stdout:Apr 23 10:42:42 smithi114 ceph-aa240572-0159-11ef-bc93-c7b262605968-osd-6[55044]: 2024-04-23T10:42:42.529+0000 7fdba4fa4700 -1 osd.6 116 heartbeat_check: no reply from 172.21.15.119:6804 osd.1 since back 2024-04-23T10:42:09.763752+0000 front 2024-04-23T10:42:09.763645+0000 (oldest deadline 2024-04-23T10:42:35.063704+0000) 2024-04-23T10:42:42.754 INFO:journalctl@ceph.osd.9.smithi114.stdout:Apr 23 10:42:42 smithi114 ceph-aa240572-0159-11ef-bc93-c7b262605968-osd-9[62058]: 2024-04-23T10:42:42.396+0000 7feeb4bc7700 -1 osd.9 116 heartbeat_check: no reply from 172.21.15.114:6806 osd.0 since back 2024-04-23T10:42:04.136736+0000 front 2024-04-23T10:42:04.136764+0000 (oldest deadline 2024-04-23T10:42:28.836926+0000) 2024-04-23T10:42:42.754 INFO:journalctl@ceph.osd.9.smithi114.stdout:Apr 23 10:42:42 smithi114 ceph-aa240572-0159-11ef-bc93-c7b262605968-osd-9[62058]: 2024-04-23T10:42:42.396+0000 7feeb4bc7700 -1 osd.9 116 heartbeat_check: no reply from 172.21.15.119:6804 osd.1 since back 2024-04-23T10:42:08.837665+0000 front 2024-04-23T10:42:08.837503+0000 (oldest deadline 2024-04-23T10:42:34.737571+0000) 2024-04-23T10:42:42.992 INFO:journalctl@ceph.osd.7.smithi119.stdout:Apr 23 10:42:42 smithi119 ceph-aa240572-0159-11ef-bc93-c7b262605968-osd-7[50859]: 2024-04-23T10:42:42.727+0000 7f015f194700 -1 osd.7 116 heartbeat_check: no reply from 172.21.15.119:6804 osd.1 since back 2024-04-23T10:42:09.664009+0000 front 2024-04-23T10:42:09.664029+0000 (oldest deadline 2024-04-23T10:42:38.464808+0000) 2024-04-23T10:42:42.992 INFO:journalctl@ceph.osd.7.smithi119.stdout:Apr 23 10:42:42 smithi119 ceph-aa240572-0159-11ef-bc93-c7b262605968-osd-7[50859]: 2024-04-23T10:42:42.727+0000 7f015f194700 -1 osd.7 116 heartbeat_check: no reply from 172.21.15.204:6804 osd.2 since back 2024-04-23T10:42:19.566017+0000 front 2024-04-23T10:42:19.566039+0000 (oldest deadline 2024-04-23T10:42:41.766809+0000) 2024-04-23T10:42:42.992 INFO:journalctl@ceph.osd.10.smithi119.stdout:Apr 23 10:42:42 smithi119 ceph-aa240572-0159-11ef-bc93-c7b262605968-osd-10[57454]: 2024-04-23T10:42:42.555+0000 7fb1849bf700 -1 osd.10 116 heartbeat_check: no reply from 172.21.15.114:6806 osd.0 since back 2024-04-23T10:42:01.813174+0000 front 2024-04-23T10:42:01.813138+0000 (oldest deadline 2024-04-23T10:42:27.113300+0000) 2024-04-23T10:42:42.992 INFO:journalctl@ceph.osd.10.smithi119.stdout:Apr 23 10:42:42 smithi119 ceph-aa240572-0159-11ef-bc93-c7b262605968-osd-10[57454]: 2024-04-23T10:42:42.555+0000 7fb1849bf700 -1 osd.10 116 heartbeat_check: no reply from 172.21.15.119:6804 osd.1 since back 2024-04-23T10:42:10.614599+0000 front 2024-04-23T10:42:10.614622+0000 (oldest deadline 2024-04-23T10:42:34.114881+0000) 2024-04-23T10:42:42.992 INFO:journalctl@ceph.osd.10.smithi119.stdout:Apr 23 10:42:42 smithi119 ceph-aa240572-0159-11ef-bc93-c7b262605968-osd-10[57454]: 2024-04-23T10:42:42.555+0000 7fb1849bf700 -1 osd.10 116 heartbeat_check: no reply from 172.21.15.204:6804 osd.2 since back 2024-04-23T10:42:19.416108+0000 front 2024-04-23T10:42:19.416241+0000 (oldest deadline 2024-04-23T10:42:42.316071+0000) 2024-04-23T10:42:43.292 INFO:journalctl@ceph.osd.11.smithi204.stdout:Apr 23 10:42:42 smithi204 ceph-aa240572-0159-11ef-bc93-c7b262605968-osd-11[57644]: 2024-04-23T10:42:42.916+0000 7fcbc4b63700 -1 osd.11 116 heartbeat_check: no reply from 172.21.15.114:6806 osd.0 since back 2024-04-23T10:42:00.675680+0000 front 2024-04-23T10:42:00.675649+0000 (oldest deadline 2024-04-23T10:42:25.876359+0000) 2024-04-23T10:42:43.292 INFO:journalctl@ceph.osd.11.smithi204.stdout:Apr 23 10:42:42 smithi204 ceph-aa240572-0159-11ef-bc93-c7b262605968-osd-11[57644]: 2024-04-23T10:42:42.916+0000 7fcbc4b63700 -1 osd.11 116 heartbeat_check: no reply from 172.21.15.119:6804 osd.1 since back 2024-04-23T10:42:09.278512+0000 front 2024-04-23T10:42:09.278494+0000 (oldest deadline 2024-04-23T10:42:33.378557+0000) 2024-04-23T10:42:43.292 INFO:journalctl@ceph.osd.11.smithi204.stdout:Apr 23 10:42:42 smithi204 ceph-aa240572-0159-11ef-bc93-c7b262605968-osd-11[57644]: 2024-04-23T10:42:42.916+0000 7fcbc4b63700 -1 osd.11 116 heartbeat_check: no reply from 172.21.15.204:6804 osd.2 since back 2024-04-23T10:42:18.679708+0000 front 2024-04-23T10:42:18.679577+0000 (oldest deadline 2024-04-23T10:42:41.579766+0000) 2024-04-23T10:42:43.578 INFO:journalctl@ceph.osd.8.smithi204.stdout:Apr 23 10:42:43 smithi204 ceph-aa240572-0159-11ef-bc93-c7b262605968-osd-8[51050]: 2024-04-23T10:42:43.293+0000 7f8d662cd700 -1 osd.8 116 heartbeat_check: no reply from 172.21.15.114:6806 osd.0 since back 2024-04-23T10:42:03.759281+0000 front 2024-04-23T10:42:03.759348+0000 (oldest deadline 2024-04-23T10:42:27.859455+0000) 2024-04-23T10:42:43.578 INFO:journalctl@ceph.osd.8.smithi204.stdout:Apr 23 10:42:43 smithi204 ceph-aa240572-0159-11ef-bc93-c7b262605968-osd-8[51050]: 2024-04-23T10:42:43.293+0000 7f8d662cd700 -1 osd.8 116 heartbeat_check: no reply from 172.21.15.119:6804 osd.1 since back 2024-04-23T10:42:09.560739+0000 front 2024-04-23T10:42:09.560541+0000 (oldest deadline 2024-04-23T10:42:37.761119+0000) 2024-04-23T10:42:43.578 INFO:journalctl@ceph.osd.5.smithi204.stdout:Apr 23 10:42:43 smithi204 ceph-aa240572-0159-11ef-bc93-c7b262605968-osd-5[44334]: 2024-04-23T10:42:43.407+0000 7fbf0215c700 -1 osd.5 116 heartbeat_check: no reply from 172.21.15.114:6806 osd.0 since back 2024-04-23T10:42:02.285250+0000 front 2024-04-23T10:42:02.285119+0000 (oldest deadline 2024-04-23T10:42:28.185247+0000) 2024-04-23T10:42:43.578 INFO:journalctl@ceph.osd.5.smithi204.stdout:Apr 23 10:42:43 smithi204 ceph-aa240572-0159-11ef-bc93-c7b262605968-osd-5[44334]: 2024-04-23T10:42:43.407+0000 7fbf0215c700 -1 osd.5 116 heartbeat_check: no reply from 172.21.15.119:6804 osd.1 since back 2024-04-23T10:42:09.886215+0000 front 2024-04-23T10:42:09.886359+0000 (oldest deadline 2024-04-23T10:42:35.186188+0000) 2024-04-23T10:42:43.579 INFO:journalctl@ceph.osd.5.smithi204.stdout:Apr 23 10:42:43 smithi204 ceph-aa240572-0159-11ef-bc93-c7b262605968-osd-5[44334]: 2024-04-23T10:42:43.407+0000 7fbf0215c700 -1 osd.5 116 heartbeat_check: no reply from 172.21.15.204:6804 osd.2 since back 2024-04-23T10:42:20.487575+0000 front 2024-04-23T10:42:20.487602+0000 (oldest deadline 2024-04-23T10:42:41.587793+0000) 2024-04-23T10:42:43.754 INFO:journalctl@ceph.osd.6.smithi114.stdout:Apr 23 10:42:43 smithi114 ceph-aa240572-0159-11ef-bc93-c7b262605968-osd-6[55044]: 2024-04-23T10:42:43.559+0000 7fdba4fa4700 -1 osd.6 116 heartbeat_check: no reply from 172.21.15.114:6806 osd.0 since back 2024-04-23T10:42:01.662289+0000 front 2024-04-23T10:42:01.662441+0000 (oldest deadline 2024-04-23T10:42:26.362236+0000) 2024-04-23T10:42:43.754 INFO:journalctl@ceph.osd.6.smithi114.stdout:Apr 23 10:42:43 smithi114 ceph-aa240572-0159-11ef-bc93-c7b262605968-osd-6[55044]: 2024-04-23T10:42:43.559+0000 7fdba4fa4700 -1 osd.6 116 heartbeat_check: no reply from 172.21.15.119:6804 osd.1 since back 2024-04-23T10:42:09.763752+0000 front 2024-04-23T10:42:09.763645+0000 (oldest deadline 2024-04-23T10:42:35.063704+0000) 2024-04-23T10:42:43.755 INFO:journalctl@ceph.osd.9.smithi114.stdout:Apr 23 10:42:43 smithi114 ceph-aa240572-0159-11ef-bc93-c7b262605968-osd-9[62058]: 2024-04-23T10:42:43.388+0000 7feeb4bc7700 -1 osd.9 116 heartbeat_check: no reply from 172.21.15.114:6806 osd.0 since back 2024-04-23T10:42:04.136736+0000 front 2024-04-23T10:42:04.136764+0000 (oldest deadline 2024-04-23T10:42:28.836926+0000) 2024-04-23T10:42:43.755 INFO:journalctl@ceph.osd.9.smithi114.stdout:Apr 23 10:42:43 smithi114 ceph-aa240572-0159-11ef-bc93-c7b262605968-osd-9[62058]: 2024-04-23T10:42:43.388+0000 7feeb4bc7700 -1 osd.9 116 heartbeat_check: no reply from 172.21.15.119:6804 osd.1 since back 2024-04-23T10:42:08.837665+0000 front 2024-04-23T10:42:08.837503+0000 (oldest deadline 2024-04-23T10:42:34.737571+0000) 2024-04-23T10:42:43.992 INFO:journalctl@ceph.osd.7.smithi119.stdout:Apr 23 10:42:43 smithi119 ceph-aa240572-0159-11ef-bc93-c7b262605968-osd-7[50859]: 2024-04-23T10:42:43.777+0000 7f015f194700 -1 osd.7 116 heartbeat_check: no reply from 172.21.15.119:6804 osd.1 since back 2024-04-23T10:42:09.664009+0000 front 2024-04-23T10:42:09.664029+0000 (oldest deadline 2024-04-23T10:42:38.464808+0000) 2024-04-23T10:42:43.992 INFO:journalctl@ceph.osd.7.smithi119.stdout:Apr 23 10:42:43 smithi119 ceph-aa240572-0159-11ef-bc93-c7b262605968-osd-7[50859]: 2024-04-23T10:42:43.777+0000 7f015f194700 -1 osd.7 116 heartbeat_check: no reply from 172.21.15.204:6804 osd.2 since back 2024-04-23T10:42:19.566017+0000 front 2024-04-23T10:42:19.566039+0000 (oldest deadline 2024-04-23T10:42:41.766809+0000) 2024-04-23T10:42:43.992 INFO:journalctl@ceph.osd.10.smithi119.stdout:Apr 23 10:42:43 smithi119 ceph-aa240572-0159-11ef-bc93-c7b262605968-osd-10[57454]: 2024-04-23T10:42:43.559+0000 7fb1849bf700 -1 osd.10 116 heartbeat_check: no reply from 172.21.15.114:6806 osd.0 since back 2024-04-23T10:42:01.813174+0000 front 2024-04-23T10:42:01.813138+0000 (oldest deadline 2024-04-23T10:42:27.113300+0000) 2024-04-23T10:42:43.993 INFO:journalctl@ceph.osd.10.smithi119.stdout:Apr 23 10:42:43 smithi119 ceph-aa240572-0159-11ef-bc93-c7b262605968-osd-10[57454]: 2024-04-23T10:42:43.559+0000 7fb1849bf700 -1 osd.10 116 heartbeat_check: no reply from 172.21.15.119:6804 osd.1 since back 2024-04-23T10:42:10.614599+0000 front 2024-04-23T10:42:10.614622+0000 (oldest deadline 2024-04-23T10:42:34.114881+0000) 2024-04-23T10:42:43.993 INFO:journalctl@ceph.osd.10.smithi119.stdout:Apr 23 10:42:43 smithi119 ceph-aa240572-0159-11ef-bc93-c7b262605968-osd-10[57454]: 2024-04-23T10:42:43.559+0000 7fb1849bf700 -1 osd.10 116 heartbeat_check: no reply from 172.21.15.204:6804 osd.2 since back 2024-04-23T10:42:19.416108+0000 front 2024-04-23T10:42:19.416241+0000 (oldest deadline 2024-04-23T10:42:42.316071+0000) 2024-04-23T10:42:44.328 INFO:journalctl@ceph.osd.11.smithi204.stdout:Apr 23 10:42:43 smithi204 ceph-aa240572-0159-11ef-bc93-c7b262605968-osd-11[57644]: 2024-04-23T10:42:43.901+0000 7fcbc4b63700 -1 osd.11 116 heartbeat_check: no reply from 172.21.15.114:6806 osd.0 since back 2024-04-23T10:42:00.675680+0000 front 2024-04-23T10:42:00.675649+0000 (oldest deadline 2024-04-23T10:42:25.876359+0000) 2024-04-23T10:42:44.328 INFO:journalctl@ceph.osd.11.smithi204.stdout:Apr 23 10:42:43 smithi204 ceph-aa240572-0159-11ef-bc93-c7b262605968-osd-11[57644]: 2024-04-23T10:42:43.901+0000 7fcbc4b63700 -1 osd.11 116 heartbeat_check: no reply from 172.21.15.119:6804 osd.1 since back 2024-04-23T10:42:09.278512+0000 front 2024-04-23T10:42:09.278494+0000 (oldest deadline 2024-04-23T10:42:33.378557+0000) 2024-04-23T10:42:44.328 INFO:journalctl@ceph.osd.11.smithi204.stdout:Apr 23 10:42:43 smithi204 ceph-aa240572-0159-11ef-bc93-c7b262605968-osd-11[57644]: 2024-04-23T10:42:43.901+0000 7fcbc4b63700 -1 osd.11 116 heartbeat_check: no reply from 172.21.15.204:6804 osd.2 since back 2024-04-23T10:42:18.679708+0000 front 2024-04-23T10:42:18.679577+0000 (oldest deadline 2024-04-23T10:42:41.579766+0000) 2024-04-23T10:42:44.754 INFO:journalctl@ceph.osd.6.smithi114.stdout:Apr 23 10:42:44 smithi114 ceph-aa240572-0159-11ef-bc93-c7b262605968-osd-6[55044]: 2024-04-23T10:42:44.551+0000 7fdba4fa4700 -1 osd.6 116 heartbeat_check: no reply from 172.21.15.114:6806 osd.0 since back 2024-04-23T10:42:01.662289+0000 front 2024-04-23T10:42:01.662441+0000 (oldest deadline 2024-04-23T10:42:26.362236+0000) 2024-04-23T10:42:44.754 INFO:journalctl@ceph.osd.6.smithi114.stdout:Apr 23 10:42:44 smithi114 ceph-aa240572-0159-11ef-bc93-c7b262605968-osd-6[55044]: 2024-04-23T10:42:44.551+0000 7fdba4fa4700 -1 osd.6 116 heartbeat_check: no reply from 172.21.15.119:6804 osd.1 since back 2024-04-23T10:42:09.763752+0000 front 2024-04-23T10:42:09.763645+0000 (oldest deadline 2024-04-23T10:42:35.063704+0000) 2024-04-23T10:42:44.755 INFO:journalctl@ceph.osd.9.smithi114.stdout:Apr 23 10:42:44 smithi114 ceph-aa240572-0159-11ef-bc93-c7b262605968-osd-9[62058]: 2024-04-23T10:42:44.424+0000 7feeb4bc7700 -1 osd.9 116 heartbeat_check: no reply from 172.21.15.114:6806 osd.0 since back 2024-04-23T10:42:04.136736+0000 front 2024-04-23T10:42:04.136764+0000 (oldest deadline 2024-04-23T10:42:28.836926+0000) 2024-04-23T10:42:44.755 INFO:journalctl@ceph.osd.9.smithi114.stdout:Apr 23 10:42:44 smithi114 ceph-aa240572-0159-11ef-bc93-c7b262605968-osd-9[62058]: 2024-04-23T10:42:44.424+0000 7feeb4bc7700 -1 osd.9 116 heartbeat_check: no reply from 172.21.15.119:6804 osd.1 since back 2024-04-23T10:42:08.837665+0000 front 2024-04-23T10:42:08.837503+0000 (oldest deadline 2024-04-23T10:42:34.737571+0000) 2024-04-23T10:42:44.755 INFO:journalctl@ceph.osd.9.smithi114.stdout:Apr 23 10:42:44 smithi114 ceph-aa240572-0159-11ef-bc93-c7b262605968-osd-9[62058]: 2024-04-23T10:42:44.424+0000 7feeb4bc7700 -1 osd.9 116 heartbeat_check: no reply from 172.21.15.204:6804 osd.2 since back 2024-04-23T10:42:18.139450+0000 front 2024-04-23T10:42:18.139388+0000 (oldest deadline 2024-04-23T10:42:43.439473+0000) 2024-04-23T10:42:44.828 INFO:journalctl@ceph.osd.8.smithi204.stdout:Apr 23 10:42:44 smithi204 ceph-aa240572-0159-11ef-bc93-c7b262605968-osd-8[51050]: 2024-04-23T10:42:44.340+0000 7f8d662cd700 -1 osd.8 116 heartbeat_check: no reply from 172.21.15.114:6806 osd.0 since back 2024-04-23T10:42:03.759281+0000 front 2024-04-23T10:42:03.759348+0000 (oldest deadline 2024-04-23T10:42:27.859455+0000) 2024-04-23T10:42:44.828 INFO:journalctl@ceph.osd.8.smithi204.stdout:Apr 23 10:42:44 smithi204 ceph-aa240572-0159-11ef-bc93-c7b262605968-osd-8[51050]: 2024-04-23T10:42:44.340+0000 7f8d662cd700 -1 osd.8 116 heartbeat_check: no reply from 172.21.15.119:6804 osd.1 since back 2024-04-23T10:42:09.560739+0000 front 2024-04-23T10:42:09.560541+0000 (oldest deadline 2024-04-23T10:42:37.761119+0000) 2024-04-23T10:42:44.828 INFO:journalctl@ceph.osd.5.smithi204.stdout:Apr 23 10:42:44 smithi204 ceph-aa240572-0159-11ef-bc93-c7b262605968-osd-5[44334]: 2024-04-23T10:42:44.452+0000 7fbf0215c700 -1 osd.5 116 heartbeat_check: no reply from 172.21.15.114:6806 osd.0 since back 2024-04-23T10:42:02.285250+0000 front 2024-04-23T10:42:02.285119+0000 (oldest deadline 2024-04-23T10:42:28.185247+0000) 2024-04-23T10:42:44.828 INFO:journalctl@ceph.osd.5.smithi204.stdout:Apr 23 10:42:44 smithi204 ceph-aa240572-0159-11ef-bc93-c7b262605968-osd-5[44334]: 2024-04-23T10:42:44.452+0000 7fbf0215c700 -1 osd.5 116 heartbeat_check: no reply from 172.21.15.119:6804 osd.1 since back 2024-04-23T10:42:09.886215+0000 front 2024-04-23T10:42:09.886359+0000 (oldest deadline 2024-04-23T10:42:35.186188+0000) 2024-04-23T10:42:44.828 INFO:journalctl@ceph.osd.5.smithi204.stdout:Apr 23 10:42:44 smithi204 ceph-aa240572-0159-11ef-bc93-c7b262605968-osd-5[44334]: 2024-04-23T10:42:44.452+0000 7fbf0215c700 -1 osd.5 116 heartbeat_check: no reply from 172.21.15.204:6804 osd.2 since back 2024-04-23T10:42:20.487575+0000 front 2024-04-23T10:42:20.487602+0000 (oldest deadline 2024-04-23T10:42:41.587793+0000) 2024-04-23T10:42:44.991 INFO:journalctl@ceph.osd.7.smithi119.stdout:Apr 23 10:42:44 smithi119 ceph-aa240572-0159-11ef-bc93-c7b262605968-osd-7[50859]: 2024-04-23T10:42:44.814+0000 7f015f194700 -1 osd.7 116 heartbeat_check: no reply from 172.21.15.119:6804 osd.1 since back 2024-04-23T10:42:09.664009+0000 front 2024-04-23T10:42:09.664029+0000 (oldest deadline 2024-04-23T10:42:38.464808+0000) 2024-04-23T10:42:44.992 INFO:journalctl@ceph.osd.7.smithi119.stdout:Apr 23 10:42:44 smithi119 ceph-aa240572-0159-11ef-bc93-c7b262605968-osd-7[50859]: 2024-04-23T10:42:44.814+0000 7f015f194700 -1 osd.7 116 heartbeat_check: no reply from 172.21.15.204:6804 osd.2 since back 2024-04-23T10:42:19.566017+0000 front 2024-04-23T10:42:19.566039+0000 (oldest deadline 2024-04-23T10:42:41.766809+0000) 2024-04-23T10:42:44.992 INFO:journalctl@ceph.osd.10.smithi119.stdout:Apr 23 10:42:44 smithi119 ceph-aa240572-0159-11ef-bc93-c7b262605968-osd-10[57454]: 2024-04-23T10:42:44.586+0000 7fb1849bf700 -1 osd.10 116 heartbeat_check: no reply from 172.21.15.114:6806 osd.0 since back 2024-04-23T10:42:01.813174+0000 front 2024-04-23T10:42:01.813138+0000 (oldest deadline 2024-04-23T10:42:27.113300+0000) 2024-04-23T10:42:44.992 INFO:journalctl@ceph.osd.10.smithi119.stdout:Apr 23 10:42:44 smithi119 ceph-aa240572-0159-11ef-bc93-c7b262605968-osd-10[57454]: 2024-04-23T10:42:44.586+0000 7fb1849bf700 -1 osd.10 116 heartbeat_check: no reply from 172.21.15.119:6804 osd.1 since back 2024-04-23T10:42:10.614599+0000 front 2024-04-23T10:42:10.614622+0000 (oldest deadline 2024-04-23T10:42:34.114881+0000) 2024-04-23T10:42:44.992 INFO:journalctl@ceph.osd.10.smithi119.stdout:Apr 23 10:42:44 smithi119 ceph-aa240572-0159-11ef-bc93-c7b262605968-osd-10[57454]: 2024-04-23T10:42:44.586+0000 7fb1849bf700 -1 osd.10 116 heartbeat_check: no reply from 172.21.15.204:6804 osd.2 since back 2024-04-23T10:42:19.416108+0000 front 2024-04-23T10:42:19.416241+0000 (oldest deadline 2024-04-23T10:42:42.316071+0000) 2024-04-23T10:42:45.328 INFO:journalctl@ceph.osd.11.smithi204.stdout:Apr 23 10:42:44 smithi204 ceph-aa240572-0159-11ef-bc93-c7b262605968-osd-11[57644]: 2024-04-23T10:42:44.853+0000 7fcbc4b63700 -1 osd.11 116 heartbeat_check: no reply from 172.21.15.114:6806 osd.0 since back 2024-04-23T10:42:00.675680+0000 front 2024-04-23T10:42:00.675649+0000 (oldest deadline 2024-04-23T10:42:25.876359+0000) 2024-04-23T10:42:45.328 INFO:journalctl@ceph.osd.11.smithi204.stdout:Apr 23 10:42:44 smithi204 ceph-aa240572-0159-11ef-bc93-c7b262605968-osd-11[57644]: 2024-04-23T10:42:44.853+0000 7fcbc4b63700 -1 osd.11 116 heartbeat_check: no reply from 172.21.15.119:6804 osd.1 since back 2024-04-23T10:42:09.278512+0000 front 2024-04-23T10:42:09.278494+0000 (oldest deadline 2024-04-23T10:42:33.378557+0000) 2024-04-23T10:42:45.328 INFO:journalctl@ceph.osd.11.smithi204.stdout:Apr 23 10:42:44 smithi204 ceph-aa240572-0159-11ef-bc93-c7b262605968-osd-11[57644]: 2024-04-23T10:42:44.853+0000 7fcbc4b63700 -1 osd.11 116 heartbeat_check: no reply from 172.21.15.204:6804 osd.2 since back 2024-04-23T10:42:18.679708+0000 front 2024-04-23T10:42:18.679577+0000 (oldest deadline 2024-04-23T10:42:41.579766+0000) 2024-04-23T10:42:45.743 INFO:journalctl@ceph.osd.5.smithi204.stdout:Apr 23 10:42:45 smithi204 podman[71065]: 2024-04-23 10:42:45.445098033 +0000 UTC m=+5.722252702 container died 9ba3dca2bfccbf433bcf7655377b8858bf19451efb67dc5c87db976d38eed87c (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:068a558e3af4932cf1c60dd321c0bdf500075eb52ea1bf455e0973ba114fbfd4, name=ceph-aa240572-0159-11ef-bc93-c7b262605968-osd-5, GIT_CLEAN=True, io.buildah.version=1.33.5, org.label-schema.build-date=20240416, CEPH_POINT_RELEASE=, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 8 Base Image, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, ceph=True, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=wip-lusov-quiescer-d69f13d, org.label-schema.schema-version=1.0) 2024-04-23T10:42:45.743 INFO:journalctl@ceph.osd.8.smithi204.stdout:Apr 23 10:42:45 smithi204 ceph-aa240572-0159-11ef-bc93-c7b262605968-osd-8[51050]: 2024-04-23T10:42:45.347+0000 7f8d662cd700 -1 osd.8 116 heartbeat_check: no reply from 172.21.15.114:6806 osd.0 since back 2024-04-23T10:42:03.759281+0000 front 2024-04-23T10:42:03.759348+0000 (oldest deadline 2024-04-23T10:42:27.859455+0000) 2024-04-23T10:42:45.743 INFO:journalctl@ceph.osd.8.smithi204.stdout:Apr 23 10:42:45 smithi204 ceph-aa240572-0159-11ef-bc93-c7b262605968-osd-8[51050]: 2024-04-23T10:42:45.347+0000 7f8d662cd700 -1 osd.8 116 heartbeat_check: no reply from 172.21.15.119:6804 osd.1 since back 2024-04-23T10:42:09.560739+0000 front 2024-04-23T10:42:09.560541+0000 (oldest deadline 2024-04-23T10:42:37.761119+0000) 2024-04-23T10:42:45.754 INFO:journalctl@ceph.osd.6.smithi114.stdout:Apr 23 10:42:45 smithi114 ceph-aa240572-0159-11ef-bc93-c7b262605968-osd-6[55044]: 2024-04-23T10:42:45.523+0000 7fdba4fa4700 -1 osd.6 116 heartbeat_check: no reply from 172.21.15.114:6806 osd.0 since back 2024-04-23T10:42:01.662289+0000 front 2024-04-23T10:42:01.662441+0000 (oldest deadline 2024-04-23T10:42:26.362236+0000) 2024-04-23T10:42:45.754 INFO:journalctl@ceph.osd.6.smithi114.stdout:Apr 23 10:42:45 smithi114 ceph-aa240572-0159-11ef-bc93-c7b262605968-osd-6[55044]: 2024-04-23T10:42:45.523+0000 7fdba4fa4700 -1 osd.6 116 heartbeat_check: no reply from 172.21.15.119:6804 osd.1 since back 2024-04-23T10:42:09.763752+0000 front 2024-04-23T10:42:09.763645+0000 (oldest deadline 2024-04-23T10:42:35.063704+0000) 2024-04-23T10:42:45.754 INFO:journalctl@ceph.osd.9.smithi114.stdout:Apr 23 10:42:45 smithi114 ceph-aa240572-0159-11ef-bc93-c7b262605968-osd-9[62058]: 2024-04-23T10:42:45.375+0000 7feeb4bc7700 -1 osd.9 116 heartbeat_check: no reply from 172.21.15.114:6806 osd.0 since back 2024-04-23T10:42:04.136736+0000 front 2024-04-23T10:42:04.136764+0000 (oldest deadline 2024-04-23T10:42:28.836926+0000) 2024-04-23T10:42:45.755 INFO:journalctl@ceph.osd.9.smithi114.stdout:Apr 23 10:42:45 smithi114 ceph-aa240572-0159-11ef-bc93-c7b262605968-osd-9[62058]: 2024-04-23T10:42:45.375+0000 7feeb4bc7700 -1 osd.9 116 heartbeat_check: no reply from 172.21.15.119:6804 osd.1 since back 2024-04-23T10:42:08.837665+0000 front 2024-04-23T10:42:08.837503+0000 (oldest deadline 2024-04-23T10:42:34.737571+0000) 2024-04-23T10:42:45.755 INFO:journalctl@ceph.osd.9.smithi114.stdout:Apr 23 10:42:45 smithi114 ceph-aa240572-0159-11ef-bc93-c7b262605968-osd-9[62058]: 2024-04-23T10:42:45.375+0000 7feeb4bc7700 -1 osd.9 116 heartbeat_check: no reply from 172.21.15.204:6804 osd.2 since back 2024-04-23T10:42:18.139450+0000 front 2024-04-23T10:42:18.139388+0000 (oldest deadline 2024-04-23T10:42:43.439473+0000) 2024-04-23T10:42:45.992 INFO:journalctl@ceph.osd.7.smithi119.stdout:Apr 23 10:42:45 smithi119 ceph-aa240572-0159-11ef-bc93-c7b262605968-osd-7[50859]: 2024-04-23T10:42:45.825+0000 7f015f194700 -1 osd.7 116 heartbeat_check: no reply from 172.21.15.119:6804 osd.1 since back 2024-04-23T10:42:09.664009+0000 front 2024-04-23T10:42:09.664029+0000 (oldest deadline 2024-04-23T10:42:38.464808+0000) 2024-04-23T10:42:45.992 INFO:journalctl@ceph.osd.7.smithi119.stdout:Apr 23 10:42:45 smithi119 ceph-aa240572-0159-11ef-bc93-c7b262605968-osd-7[50859]: 2024-04-23T10:42:45.825+0000 7f015f194700 -1 osd.7 116 heartbeat_check: no reply from 172.21.15.204:6804 osd.2 since back 2024-04-23T10:42:19.566017+0000 front 2024-04-23T10:42:19.566039+0000 (oldest deadline 2024-04-23T10:42:41.766809+0000) 2024-04-23T10:42:45.992 INFO:journalctl@ceph.osd.10.smithi119.stdout:Apr 23 10:42:45 smithi119 ceph-aa240572-0159-11ef-bc93-c7b262605968-osd-10[57454]: 2024-04-23T10:42:45.630+0000 7fb1849bf700 -1 osd.10 116 heartbeat_check: no reply from 172.21.15.114:6806 osd.0 since back 2024-04-23T10:42:01.813174+0000 front 2024-04-23T10:42:01.813138+0000 (oldest deadline 2024-04-23T10:42:27.113300+0000) 2024-04-23T10:42:45.993 INFO:journalctl@ceph.osd.10.smithi119.stdout:Apr 23 10:42:45 smithi119 ceph-aa240572-0159-11ef-bc93-c7b262605968-osd-10[57454]: 2024-04-23T10:42:45.630+0000 7fb1849bf700 -1 osd.10 116 heartbeat_check: no reply from 172.21.15.119:6804 osd.1 since back 2024-04-23T10:42:10.614599+0000 front 2024-04-23T10:42:10.614622+0000 (oldest deadline 2024-04-23T10:42:34.114881+0000) 2024-04-23T10:42:45.993 INFO:journalctl@ceph.osd.10.smithi119.stdout:Apr 23 10:42:45 smithi119 ceph-aa240572-0159-11ef-bc93-c7b262605968-osd-10[57454]: 2024-04-23T10:42:45.630+0000 7fb1849bf700 -1 osd.10 116 heartbeat_check: no reply from 172.21.15.204:6804 osd.2 since back 2024-04-23T10:42:19.416108+0000 front 2024-04-23T10:42:19.416241+0000 (oldest deadline 2024-04-23T10:42:42.316071+0000) 2024-04-23T10:42:46.042 INFO:journalctl@ceph.osd.5.smithi204.stdout:Apr 23 10:42:45 smithi204 podman[71065]: 2024-04-23 10:42:45.835018108 +0000 UTC m=+6.112172839 container cleanup 9ba3dca2bfccbf433bcf7655377b8858bf19451efb67dc5c87db976d38eed87c (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:068a558e3af4932cf1c60dd321c0bdf500075eb52ea1bf455e0973ba114fbfd4, name=ceph-aa240572-0159-11ef-bc93-c7b262605968-osd-5, GIT_CLEAN=True, io.buildah.version=1.33.5, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, org.label-schema.build-date=20240416, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 8 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , RELEASE=wip-lusov-quiescer-d69f13d, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, org.label-schema.schema-version=1.0, ceph=True) 2024-04-23T10:42:46.042 INFO:journalctl@ceph.osd.5.smithi204.stdout:Apr 23 10:42:45 smithi204 bash[71065]: ceph-aa240572-0159-11ef-bc93-c7b262605968-osd-5 2024-04-23T10:42:46.043 INFO:journalctl@ceph.osd.11.smithi204.stdout:Apr 23 10:42:45 smithi204 ceph-aa240572-0159-11ef-bc93-c7b262605968-osd-11[57644]: 2024-04-23T10:42:45.835+0000 7fcbc4b63700 -1 osd.11 116 heartbeat_check: no reply from 172.21.15.114:6806 osd.0 since back 2024-04-23T10:42:00.675680+0000 front 2024-04-23T10:42:00.675649+0000 (oldest deadline 2024-04-23T10:42:25.876359+0000) 2024-04-23T10:42:46.043 INFO:journalctl@ceph.osd.11.smithi204.stdout:Apr 23 10:42:45 smithi204 ceph-aa240572-0159-11ef-bc93-c7b262605968-osd-11[57644]: 2024-04-23T10:42:45.835+0000 7fcbc4b63700 -1 osd.11 116 heartbeat_check: no reply from 172.21.15.119:6804 osd.1 since back 2024-04-23T10:42:09.278512+0000 front 2024-04-23T10:42:09.278494+0000 (oldest deadline 2024-04-23T10:42:33.378557+0000) 2024-04-23T10:42:46.043 INFO:journalctl@ceph.osd.11.smithi204.stdout:Apr 23 10:42:45 smithi204 ceph-aa240572-0159-11ef-bc93-c7b262605968-osd-11[57644]: 2024-04-23T10:42:45.835+0000 7fcbc4b63700 -1 osd.11 116 heartbeat_check: no reply from 172.21.15.204:6804 osd.2 since back 2024-04-23T10:42:18.679708+0000 front 2024-04-23T10:42:18.679577+0000 (oldest deadline 2024-04-23T10:42:41.579766+0000) 2024-04-23T10:42:46.327 INFO:journalctl@ceph.osd.5.smithi204.stdout:Apr 23 10:42:46 smithi204 podman[71082]: 2024-04-23 10:42:46.04438981 +0000 UTC m=+0.600903147 container remove 9ba3dca2bfccbf433bcf7655377b8858bf19451efb67dc5c87db976d38eed87c (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:068a558e3af4932cf1c60dd321c0bdf500075eb52ea1bf455e0973ba114fbfd4, name=ceph-aa240572-0159-11ef-bc93-c7b262605968-osd-5, org.label-schema.schema-version=1.0, org.label-schema.license=GPLv2, RELEASE=wip-lusov-quiescer-d69f13d, ceph=True, maintainer=Guillaume Abrioux , GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, CEPH_POINT_RELEASE=, io.buildah.version=1.33.5, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240416, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, GIT_CLEAN=True) 2024-04-23T10:42:46.588 INFO:journalctl@ceph.osd.8.smithi204.stdout:Apr 23 10:42:46 smithi204 ceph-aa240572-0159-11ef-bc93-c7b262605968-osd-8[51050]: 2024-04-23T10:42:46.363+0000 7f8d662cd700 -1 osd.8 116 heartbeat_check: no reply from 172.21.15.114:6806 osd.0 since back 2024-04-23T10:42:03.759281+0000 front 2024-04-23T10:42:03.759348+0000 (oldest deadline 2024-04-23T10:42:27.859455+0000) 2024-04-23T10:42:46.588 INFO:journalctl@ceph.osd.8.smithi204.stdout:Apr 23 10:42:46 smithi204 ceph-aa240572-0159-11ef-bc93-c7b262605968-osd-8[51050]: 2024-04-23T10:42:46.363+0000 7f8d662cd700 -1 osd.8 116 heartbeat_check: no reply from 172.21.15.119:6804 osd.1 since back 2024-04-23T10:42:09.560739+0000 front 2024-04-23T10:42:09.560541+0000 (oldest deadline 2024-04-23T10:42:37.761119+0000) 2024-04-23T10:42:46.588 INFO:journalctl@ceph.osd.5.smithi204.stdout:Apr 23 10:42:46 smithi204 podman[71171]: 2024-04-23 10:42:46.239153405 +0000 UTC m=+0.018158655 image pull da4b004c09e820713483d039bc76fbd660be678f2ea3c03e059a6161b74dab76 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:068a558e3af4932cf1c60dd321c0bdf500075eb52ea1bf455e0973ba114fbfd4 2024-04-23T10:42:46.588 INFO:journalctl@ceph.osd.5.smithi204.stdout:Apr 23 10:42:46 smithi204 podman[71171]: 2024-04-23 10:42:46.353764254 +0000 UTC m=+0.132769489 container create 2c02f461dffc245a8bc0469db88264978a6102e340c5b46bb4e16a749f417dba (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:068a558e3af4932cf1c60dd321c0bdf500075eb52ea1bf455e0973ba114fbfd4, name=ceph-aa240572-0159-11ef-bc93-c7b262605968-osd-5-deactivate, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, org.label-schema.build-date=20240416, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, io.buildah.version=1.33.5, maintainer=Guillaume Abrioux , GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_CLEAN=True, RELEASE=wip-lusov-quiescer-d69f13d, ceph=True, CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 8 Base Image) 2024-04-23T10:42:46.589 INFO:journalctl@ceph.osd.5.smithi204.stdout:Apr 23 10:42:46 smithi204 podman[71171]: 2024-04-23 10:42:46.589862713 +0000 UTC m=+0.368867948 container init 2c02f461dffc245a8bc0469db88264978a6102e340c5b46bb4e16a749f417dba (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:068a558e3af4932cf1c60dd321c0bdf500075eb52ea1bf455e0973ba114fbfd4, name=ceph-aa240572-0159-11ef-bc93-c7b262605968-osd-5-deactivate, org.label-schema.license=GPLv2, org.label-schema.build-date=20240416, maintainer=Guillaume Abrioux , GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_CLEAN=True, ceph=True, io.buildah.version=1.33.5, CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 8 Base Image, RELEASE=wip-lusov-quiescer-d69f13d, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, GIT_BRANCH=HEAD) 2024-04-23T10:42:46.754 INFO:journalctl@ceph.osd.6.smithi114.stdout:Apr 23 10:42:46 smithi114 ceph-aa240572-0159-11ef-bc93-c7b262605968-osd-6[55044]: 2024-04-23T10:42:46.515+0000 7fdba4fa4700 -1 osd.6 116 heartbeat_check: no reply from 172.21.15.114:6806 osd.0 since back 2024-04-23T10:42:01.662289+0000 front 2024-04-23T10:42:01.662441+0000 (oldest deadline 2024-04-23T10:42:26.362236+0000) 2024-04-23T10:42:46.754 INFO:journalctl@ceph.osd.6.smithi114.stdout:Apr 23 10:42:46 smithi114 ceph-aa240572-0159-11ef-bc93-c7b262605968-osd-6[55044]: 2024-04-23T10:42:46.515+0000 7fdba4fa4700 -1 osd.6 116 heartbeat_check: no reply from 172.21.15.119:6804 osd.1 since back 2024-04-23T10:42:09.763752+0000 front 2024-04-23T10:42:09.763645+0000 (oldest deadline 2024-04-23T10:42:35.063704+0000) 2024-04-23T10:42:46.755 INFO:journalctl@ceph.osd.9.smithi114.stdout:Apr 23 10:42:46 smithi114 ceph-aa240572-0159-11ef-bc93-c7b262605968-osd-9[62058]: 2024-04-23T10:42:46.328+0000 7feeb4bc7700 -1 osd.9 116 heartbeat_check: no reply from 172.21.15.114:6806 osd.0 since back 2024-04-23T10:42:04.136736+0000 front 2024-04-23T10:42:04.136764+0000 (oldest deadline 2024-04-23T10:42:28.836926+0000) 2024-04-23T10:42:46.755 INFO:journalctl@ceph.osd.9.smithi114.stdout:Apr 23 10:42:46 smithi114 ceph-aa240572-0159-11ef-bc93-c7b262605968-osd-9[62058]: 2024-04-23T10:42:46.328+0000 7feeb4bc7700 -1 osd.9 116 heartbeat_check: no reply from 172.21.15.119:6804 osd.1 since back 2024-04-23T10:42:08.837665+0000 front 2024-04-23T10:42:08.837503+0000 (oldest deadline 2024-04-23T10:42:34.737571+0000) 2024-04-23T10:42:46.755 INFO:journalctl@ceph.osd.9.smithi114.stdout:Apr 23 10:42:46 smithi114 ceph-aa240572-0159-11ef-bc93-c7b262605968-osd-9[62058]: 2024-04-23T10:42:46.328+0000 7feeb4bc7700 -1 osd.9 116 heartbeat_check: no reply from 172.21.15.204:6804 osd.2 since back 2024-04-23T10:42:18.139450+0000 front 2024-04-23T10:42:18.139388+0000 (oldest deadline 2024-04-23T10:42:43.439473+0000) 2024-04-23T10:42:46.885 INFO:journalctl@ceph.osd.11.smithi204.stdout:Apr 23 10:42:46 smithi204 ceph-aa240572-0159-11ef-bc93-c7b262605968-osd-11[57644]: 2024-04-23T10:42:46.809+0000 7fcbc4b63700 -1 osd.11 116 heartbeat_check: no reply from 172.21.15.114:6806 osd.0 since back 2024-04-23T10:42:00.675680+0000 front 2024-04-23T10:42:00.675649+0000 (oldest deadline 2024-04-23T10:42:25.876359+0000) 2024-04-23T10:42:46.885 INFO:journalctl@ceph.osd.11.smithi204.stdout:Apr 23 10:42:46 smithi204 ceph-aa240572-0159-11ef-bc93-c7b262605968-osd-11[57644]: 2024-04-23T10:42:46.809+0000 7fcbc4b63700 -1 osd.11 116 heartbeat_check: no reply from 172.21.15.119:6804 osd.1 since back 2024-04-23T10:42:09.278512+0000 front 2024-04-23T10:42:09.278494+0000 (oldest deadline 2024-04-23T10:42:33.378557+0000) 2024-04-23T10:42:46.885 INFO:journalctl@ceph.osd.11.smithi204.stdout:Apr 23 10:42:46 smithi204 ceph-aa240572-0159-11ef-bc93-c7b262605968-osd-11[57644]: 2024-04-23T10:42:46.809+0000 7fcbc4b63700 -1 osd.11 116 heartbeat_check: no reply from 172.21.15.204:6804 osd.2 since back 2024-04-23T10:42:18.679708+0000 front 2024-04-23T10:42:18.679577+0000 (oldest deadline 2024-04-23T10:42:41.579766+0000) 2024-04-23T10:42:46.885 INFO:journalctl@ceph.osd.5.smithi204.stdout:Apr 23 10:42:46 smithi204 podman[71171]: 2024-04-23 10:42:46.593091442 +0000 UTC m=+0.372096677 container start 2c02f461dffc245a8bc0469db88264978a6102e340c5b46bb4e16a749f417dba (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:068a558e3af4932cf1c60dd321c0bdf500075eb52ea1bf455e0973ba114fbfd4, name=ceph-aa240572-0159-11ef-bc93-c7b262605968-osd-5-deactivate, GIT_CLEAN=True, RELEASE=wip-lusov-quiescer-d69f13d, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0, org.label-schema.name=CentOS Stream 8 Base Image, ceph=True, io.buildah.version=1.33.5, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2, org.label-schema.build-date=20240416) 2024-04-23T10:42:46.886 INFO:journalctl@ceph.osd.5.smithi204.stdout:Apr 23 10:42:46 smithi204 podman[71171]: 2024-04-23 10:42:46.690221292 +0000 UTC m=+0.469226536 container attach 2c02f461dffc245a8bc0469db88264978a6102e340c5b46bb4e16a749f417dba (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:068a558e3af4932cf1c60dd321c0bdf500075eb52ea1bf455e0973ba114fbfd4, name=ceph-aa240572-0159-11ef-bc93-c7b262605968-osd-5-deactivate, org.label-schema.vendor=CentOS, org.label-schema.name=CentOS Stream 8 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, GIT_CLEAN=True, ceph=True, io.buildah.version=1.33.5, RELEASE=wip-lusov-quiescer-d69f13d, org.label-schema.schema-version=1.0, org.label-schema.build-date=20240416, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=) 2024-04-23T10:42:46.992 INFO:journalctl@ceph.osd.7.smithi119.stdout:Apr 23 10:42:46 smithi119 ceph-aa240572-0159-11ef-bc93-c7b262605968-osd-7[50859]: 2024-04-23T10:42:46.805+0000 7f015f194700 -1 osd.7 116 heartbeat_check: no reply from 172.21.15.119:6804 osd.1 since back 2024-04-23T10:42:09.664009+0000 front 2024-04-23T10:42:09.664029+0000 (oldest deadline 2024-04-23T10:42:38.464808+0000) 2024-04-23T10:42:46.992 INFO:journalctl@ceph.osd.7.smithi119.stdout:Apr 23 10:42:46 smithi119 ceph-aa240572-0159-11ef-bc93-c7b262605968-osd-7[50859]: 2024-04-23T10:42:46.805+0000 7f015f194700 -1 osd.7 116 heartbeat_check: no reply from 172.21.15.204:6804 osd.2 since back 2024-04-23T10:42:19.566017+0000 front 2024-04-23T10:42:19.566039+0000 (oldest deadline 2024-04-23T10:42:41.766809+0000) 2024-04-23T10:42:46.992 INFO:journalctl@ceph.osd.10.smithi119.stdout:Apr 23 10:42:46 smithi119 ceph-aa240572-0159-11ef-bc93-c7b262605968-osd-10[57454]: 2024-04-23T10:42:46.664+0000 7fb1849bf700 -1 osd.10 116 heartbeat_check: no reply from 172.21.15.114:6806 osd.0 since back 2024-04-23T10:42:01.813174+0000 front 2024-04-23T10:42:01.813138+0000 (oldest deadline 2024-04-23T10:42:27.113300+0000) 2024-04-23T10:42:46.993 INFO:journalctl@ceph.osd.10.smithi119.stdout:Apr 23 10:42:46 smithi119 ceph-aa240572-0159-11ef-bc93-c7b262605968-osd-10[57454]: 2024-04-23T10:42:46.664+0000 7fb1849bf700 -1 osd.10 116 heartbeat_check: no reply from 172.21.15.119:6804 osd.1 since back 2024-04-23T10:42:10.614599+0000 front 2024-04-23T10:42:10.614622+0000 (oldest deadline 2024-04-23T10:42:34.114881+0000) 2024-04-23T10:42:46.993 INFO:journalctl@ceph.osd.10.smithi119.stdout:Apr 23 10:42:46 smithi119 ceph-aa240572-0159-11ef-bc93-c7b262605968-osd-10[57454]: 2024-04-23T10:42:46.664+0000 7fb1849bf700 -1 osd.10 116 heartbeat_check: no reply from 172.21.15.204:6804 osd.2 since back 2024-04-23T10:42:19.416108+0000 front 2024-04-23T10:42:19.416241+0000 (oldest deadline 2024-04-23T10:42:42.316071+0000) 2024-04-23T10:42:47.167 INFO:journalctl@ceph.osd.5.smithi204.stdout:Apr 23 10:42:46 smithi204 podman[71171]: 2024-04-23 10:42:46.88721548 +0000 UTC m=+0.666220724 container died 2c02f461dffc245a8bc0469db88264978a6102e340c5b46bb4e16a749f417dba (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:068a558e3af4932cf1c60dd321c0bdf500075eb52ea1bf455e0973ba114fbfd4, name=ceph-aa240572-0159-11ef-bc93-c7b262605968-osd-5-deactivate, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 8 Base Image, GIT_CLEAN=True, RELEASE=wip-lusov-quiescer-d69f13d, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0, io.buildah.version=1.33.5, org.label-schema.build-date=20240416, org.label-schema.vendor=CentOS, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, ceph=True) 2024-04-23T10:42:47.425 INFO:journalctl@ceph.osd.8.smithi204.stdout:Apr 23 10:42:47 smithi204 ceph-aa240572-0159-11ef-bc93-c7b262605968-osd-8[51050]: 2024-04-23T10:42:47.356+0000 7f8d662cd700 -1 osd.8 116 heartbeat_check: no reply from 172.21.15.114:6806 osd.0 since back 2024-04-23T10:42:03.759281+0000 front 2024-04-23T10:42:03.759348+0000 (oldest deadline 2024-04-23T10:42:27.859455+0000) 2024-04-23T10:42:47.425 INFO:journalctl@ceph.osd.8.smithi204.stdout:Apr 23 10:42:47 smithi204 ceph-aa240572-0159-11ef-bc93-c7b262605968-osd-8[51050]: 2024-04-23T10:42:47.356+0000 7f8d662cd700 -1 osd.8 116 heartbeat_check: no reply from 172.21.15.119:6804 osd.1 since back 2024-04-23T10:42:09.560739+0000 front 2024-04-23T10:42:09.560541+0000 (oldest deadline 2024-04-23T10:42:37.761119+0000) 2024-04-23T10:42:47.441 DEBUG:teuthology.orchestra.run.smithi204:> sudo pkill -f 'journalctl -f -n 0 -u ceph-aa240572-0159-11ef-bc93-c7b262605968@osd.5.service' 2024-04-23T10:42:47.478 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-04-23T10:42:47.479 INFO:tasks.cephadm.osd.5:Stopped osd.5 2024-04-23T10:42:47.479 INFO:tasks.cephadm.osd.6:Stopping osd.6... 2024-04-23T10:42:47.479 DEBUG:teuthology.orchestra.run.smithi114:> sudo systemctl stop ceph-aa240572-0159-11ef-bc93-c7b262605968@osd.6 2024-04-23T10:42:47.754 INFO:journalctl@ceph.osd.9.smithi114.stdout:Apr 23 10:42:47 smithi114 ceph-aa240572-0159-11ef-bc93-c7b262605968-osd-9[62058]: 2024-04-23T10:42:47.356+0000 7feeb4bc7700 -1 osd.9 116 heartbeat_check: no reply from 172.21.15.114:6806 osd.0 since back 2024-04-23T10:42:04.136736+0000 front 2024-04-23T10:42:04.136764+0000 (oldest deadline 2024-04-23T10:42:28.836926+0000) 2024-04-23T10:42:47.755 INFO:journalctl@ceph.osd.9.smithi114.stdout:Apr 23 10:42:47 smithi114 ceph-aa240572-0159-11ef-bc93-c7b262605968-osd-9[62058]: 2024-04-23T10:42:47.356+0000 7feeb4bc7700 -1 osd.9 116 heartbeat_check: no reply from 172.21.15.119:6804 osd.1 since back 2024-04-23T10:42:08.837665+0000 front 2024-04-23T10:42:08.837503+0000 (oldest deadline 2024-04-23T10:42:34.737571+0000) 2024-04-23T10:42:47.755 INFO:journalctl@ceph.osd.9.smithi114.stdout:Apr 23 10:42:47 smithi114 ceph-aa240572-0159-11ef-bc93-c7b262605968-osd-9[62058]: 2024-04-23T10:42:47.356+0000 7feeb4bc7700 -1 osd.9 116 heartbeat_check: no reply from 172.21.15.204:6804 osd.2 since back 2024-04-23T10:42:18.139450+0000 front 2024-04-23T10:42:18.139388+0000 (oldest deadline 2024-04-23T10:42:43.439473+0000) 2024-04-23T10:42:47.755 INFO:journalctl@ceph.osd.6.smithi114.stdout:Apr 23 10:42:47 smithi114 ceph-aa240572-0159-11ef-bc93-c7b262605968-osd-6[55044]: 2024-04-23T10:42:47.511+0000 7fdba4fa4700 -1 osd.6 116 heartbeat_check: no reply from 172.21.15.114:6806 osd.0 since back 2024-04-23T10:42:01.662289+0000 front 2024-04-23T10:42:01.662441+0000 (oldest deadline 2024-04-23T10:42:26.362236+0000) 2024-04-23T10:42:47.755 INFO:journalctl@ceph.osd.6.smithi114.stdout:Apr 23 10:42:47 smithi114 ceph-aa240572-0159-11ef-bc93-c7b262605968-osd-6[55044]: 2024-04-23T10:42:47.511+0000 7fdba4fa4700 -1 osd.6 116 heartbeat_check: no reply from 172.21.15.119:6804 osd.1 since back 2024-04-23T10:42:09.763752+0000 front 2024-04-23T10:42:09.763645+0000 (oldest deadline 2024-04-23T10:42:35.063704+0000) 2024-04-23T10:42:47.756 INFO:journalctl@ceph.osd.6.smithi114.stdout:Apr 23 10:42:47 smithi114 ceph-aa240572-0159-11ef-bc93-c7b262605968-osd-6[55044]: 2024-04-23T10:42:47.511+0000 7fdba4fa4700 -1 osd.6 116 heartbeat_check: no reply from 172.21.15.204:6804 osd.2 since back 2024-04-23T10:42:17.364953+0000 front 2024-04-23T10:42:17.365063+0000 (oldest deadline 2024-04-23T10:42:47.365241+0000) 2024-04-23T10:42:47.756 INFO:journalctl@ceph.osd.6.smithi114.stdout:Apr 23 10:42:47 smithi114 systemd[1]: Stopping Ceph osd.6 for aa240572-0159-11ef-bc93-c7b262605968... 2024-04-23T10:42:47.992 INFO:journalctl@ceph.osd.7.smithi119.stdout:Apr 23 10:42:47 smithi119 ceph-aa240572-0159-11ef-bc93-c7b262605968-osd-7[50859]: 2024-04-23T10:42:47.816+0000 7f015f194700 -1 osd.7 116 heartbeat_check: no reply from 172.21.15.119:6804 osd.1 since back 2024-04-23T10:42:09.664009+0000 front 2024-04-23T10:42:09.664029+0000 (oldest deadline 2024-04-23T10:42:38.464808+0000) 2024-04-23T10:42:47.993 INFO:journalctl@ceph.osd.7.smithi119.stdout:Apr 23 10:42:47 smithi119 ceph-aa240572-0159-11ef-bc93-c7b262605968-osd-7[50859]: 2024-04-23T10:42:47.816+0000 7f015f194700 -1 osd.7 116 heartbeat_check: no reply from 172.21.15.204:6804 osd.2 since back 2024-04-23T10:42:19.566017+0000 front 2024-04-23T10:42:19.566039+0000 (oldest deadline 2024-04-23T10:42:41.766809+0000) 2024-04-23T10:42:47.993 INFO:journalctl@ceph.osd.10.smithi119.stdout:Apr 23 10:42:47 smithi119 ceph-aa240572-0159-11ef-bc93-c7b262605968-osd-10[57454]: 2024-04-23T10:42:47.669+0000 7fb1849bf700 -1 osd.10 116 heartbeat_check: no reply from 172.21.15.114:6806 osd.0 since back 2024-04-23T10:42:01.813174+0000 front 2024-04-23T10:42:01.813138+0000 (oldest deadline 2024-04-23T10:42:27.113300+0000) 2024-04-23T10:42:47.993 INFO:journalctl@ceph.osd.10.smithi119.stdout:Apr 23 10:42:47 smithi119 ceph-aa240572-0159-11ef-bc93-c7b262605968-osd-10[57454]: 2024-04-23T10:42:47.669+0000 7fb1849bf700 -1 osd.10 116 heartbeat_check: no reply from 172.21.15.119:6804 osd.1 since back 2024-04-23T10:42:10.614599+0000 front 2024-04-23T10:42:10.614622+0000 (oldest deadline 2024-04-23T10:42:34.114881+0000) 2024-04-23T10:42:47.993 INFO:journalctl@ceph.osd.10.smithi119.stdout:Apr 23 10:42:47 smithi119 ceph-aa240572-0159-11ef-bc93-c7b262605968-osd-10[57454]: 2024-04-23T10:42:47.669+0000 7fb1849bf700 -1 osd.10 116 heartbeat_check: no reply from 172.21.15.204:6804 osd.2 since back 2024-04-23T10:42:19.416108+0000 front 2024-04-23T10:42:19.416241+0000 (oldest deadline 2024-04-23T10:42:42.316071+0000) 2024-04-23T10:42:48.078 INFO:journalctl@ceph.osd.11.smithi204.stdout:Apr 23 10:42:47 smithi204 ceph-aa240572-0159-11ef-bc93-c7b262605968-osd-11[57644]: 2024-04-23T10:42:47.768+0000 7fcbc4b63700 -1 osd.11 116 heartbeat_check: no reply from 172.21.15.114:6806 osd.0 since back 2024-04-23T10:42:00.675680+0000 front 2024-04-23T10:42:00.675649+0000 (oldest deadline 2024-04-23T10:42:25.876359+0000) 2024-04-23T10:42:48.078 INFO:journalctl@ceph.osd.11.smithi204.stdout:Apr 23 10:42:47 smithi204 ceph-aa240572-0159-11ef-bc93-c7b262605968-osd-11[57644]: 2024-04-23T10:42:47.768+0000 7fcbc4b63700 -1 osd.11 116 heartbeat_check: no reply from 172.21.15.119:6804 osd.1 since back 2024-04-23T10:42:09.278512+0000 front 2024-04-23T10:42:09.278494+0000 (oldest deadline 2024-04-23T10:42:33.378557+0000) 2024-04-23T10:42:48.078 INFO:journalctl@ceph.osd.11.smithi204.stdout:Apr 23 10:42:47 smithi204 ceph-aa240572-0159-11ef-bc93-c7b262605968-osd-11[57644]: 2024-04-23T10:42:47.768+0000 7fcbc4b63700 -1 osd.11 116 heartbeat_check: no reply from 172.21.15.204:6804 osd.2 since back 2024-04-23T10:42:18.679708+0000 front 2024-04-23T10:42:18.679577+0000 (oldest deadline 2024-04-23T10:42:41.579766+0000) 2024-04-23T10:42:48.254 INFO:journalctl@ceph.osd.6.smithi114.stdout:Apr 23 10:42:47 smithi114 ceph-aa240572-0159-11ef-bc93-c7b262605968-osd-6[55044]: 2024-04-23T10:42:47.770+0000 7fdbac9b8700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.6 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-04-23T10:42:48.254 INFO:journalctl@ceph.osd.6.smithi114.stdout:Apr 23 10:42:47 smithi114 ceph-aa240572-0159-11ef-bc93-c7b262605968-osd-6[55044]: 2024-04-23T10:42:47.770+0000 7fdbac9b8700 -1 osd.6 116 *** Got signal Terminated *** 2024-04-23T10:42:48.254 INFO:journalctl@ceph.osd.6.smithi114.stdout:Apr 23 10:42:47 smithi114 ceph-aa240572-0159-11ef-bc93-c7b262605968-osd-6[55044]: 2024-04-23T10:42:47.770+0000 7fdbac9b8700 -1 osd.6 116 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-04-23T10:42:48.722 INFO:journalctl@ceph.osd.8.smithi204.stdout:Apr 23 10:42:48 smithi204 ceph-aa240572-0159-11ef-bc93-c7b262605968-osd-8[51050]: 2024-04-23T10:42:48.369+0000 7f8d662cd700 -1 osd.8 116 heartbeat_check: no reply from 172.21.15.114:6806 osd.0 since back 2024-04-23T10:42:03.759281+0000 front 2024-04-23T10:42:03.759348+0000 (oldest deadline 2024-04-23T10:42:27.859455+0000) 2024-04-23T10:42:48.722 INFO:journalctl@ceph.osd.8.smithi204.stdout:Apr 23 10:42:48 smithi204 ceph-aa240572-0159-11ef-bc93-c7b262605968-osd-8[51050]: 2024-04-23T10:42:48.369+0000 7f8d662cd700 -1 osd.8 116 heartbeat_check: no reply from 172.21.15.119:6804 osd.1 since back 2024-04-23T10:42:09.560739+0000 front 2024-04-23T10:42:09.560541+0000 (oldest deadline 2024-04-23T10:42:37.761119+0000) 2024-04-23T10:42:48.754 INFO:journalctl@ceph.osd.9.smithi114.stdout:Apr 23 10:42:48 smithi114 ceph-aa240572-0159-11ef-bc93-c7b262605968-osd-9[62058]: 2024-04-23T10:42:48.356+0000 7feeb4bc7700 -1 osd.9 116 heartbeat_check: no reply from 172.21.15.114:6806 osd.0 since back 2024-04-23T10:42:04.136736+0000 front 2024-04-23T10:42:04.136764+0000 (oldest deadline 2024-04-23T10:42:28.836926+0000) 2024-04-23T10:42:48.754 INFO:journalctl@ceph.osd.9.smithi114.stdout:Apr 23 10:42:48 smithi114 ceph-aa240572-0159-11ef-bc93-c7b262605968-osd-9[62058]: 2024-04-23T10:42:48.356+0000 7feeb4bc7700 -1 osd.9 116 heartbeat_check: no reply from 172.21.15.119:6804 osd.1 since back 2024-04-23T10:42:08.837665+0000 front 2024-04-23T10:42:08.837503+0000 (oldest deadline 2024-04-23T10:42:34.737571+0000) 2024-04-23T10:42:48.755 INFO:journalctl@ceph.osd.9.smithi114.stdout:Apr 23 10:42:48 smithi114 ceph-aa240572-0159-11ef-bc93-c7b262605968-osd-9[62058]: 2024-04-23T10:42:48.356+0000 7feeb4bc7700 -1 osd.9 116 heartbeat_check: no reply from 172.21.15.204:6804 osd.2 since back 2024-04-23T10:42:18.139450+0000 front 2024-04-23T10:42:18.139388+0000 (oldest deadline 2024-04-23T10:42:43.439473+0000) 2024-04-23T10:42:48.755 INFO:journalctl@ceph.osd.6.smithi114.stdout:Apr 23 10:42:48 smithi114 ceph-aa240572-0159-11ef-bc93-c7b262605968-osd-6[55044]: 2024-04-23T10:42:48.542+0000 7fdba4fa4700 -1 osd.6 116 heartbeat_check: no reply from 172.21.15.114:6806 osd.0 since back 2024-04-23T10:42:01.662289+0000 front 2024-04-23T10:42:01.662441+0000 (oldest deadline 2024-04-23T10:42:26.362236+0000) 2024-04-23T10:42:48.755 INFO:journalctl@ceph.osd.6.smithi114.stdout:Apr 23 10:42:48 smithi114 ceph-aa240572-0159-11ef-bc93-c7b262605968-osd-6[55044]: 2024-04-23T10:42:48.542+0000 7fdba4fa4700 -1 osd.6 116 heartbeat_check: no reply from 172.21.15.119:6804 osd.1 since back 2024-04-23T10:42:09.763752+0000 front 2024-04-23T10:42:09.763645+0000 (oldest deadline 2024-04-23T10:42:35.063704+0000) 2024-04-23T10:42:48.755 INFO:journalctl@ceph.osd.6.smithi114.stdout:Apr 23 10:42:48 smithi114 ceph-aa240572-0159-11ef-bc93-c7b262605968-osd-6[55044]: 2024-04-23T10:42:48.542+0000 7fdba4fa4700 -1 osd.6 116 heartbeat_check: no reply from 172.21.15.204:6804 osd.2 since back 2024-04-23T10:42:17.364953+0000 front 2024-04-23T10:42:17.365063+0000 (oldest deadline 2024-04-23T10:42:47.365241+0000) 2024-04-23T10:42:48.992 INFO:journalctl@ceph.osd.7.smithi119.stdout:Apr 23 10:42:48 smithi119 ceph-aa240572-0159-11ef-bc93-c7b262605968-osd-7[50859]: 2024-04-23T10:42:48.783+0000 7f015f194700 -1 osd.7 116 heartbeat_check: no reply from 172.21.15.119:6804 osd.1 since back 2024-04-23T10:42:09.664009+0000 front 2024-04-23T10:42:09.664029+0000 (oldest deadline 2024-04-23T10:42:38.464808+0000) 2024-04-23T10:42:48.992 INFO:journalctl@ceph.osd.7.smithi119.stdout:Apr 23 10:42:48 smithi119 ceph-aa240572-0159-11ef-bc93-c7b262605968-osd-7[50859]: 2024-04-23T10:42:48.783+0000 7f015f194700 -1 osd.7 116 heartbeat_check: no reply from 172.21.15.204:6804 osd.2 since back 2024-04-23T10:42:19.566017+0000 front 2024-04-23T10:42:19.566039+0000 (oldest deadline 2024-04-23T10:42:41.766809+0000) 2024-04-23T10:42:48.993 INFO:journalctl@ceph.osd.10.smithi119.stdout:Apr 23 10:42:48 smithi119 ceph-aa240572-0159-11ef-bc93-c7b262605968-osd-10[57454]: 2024-04-23T10:42:48.711+0000 7fb1849bf700 -1 osd.10 116 heartbeat_check: no reply from 172.21.15.114:6806 osd.0 since back 2024-04-23T10:42:01.813174+0000 front 2024-04-23T10:42:01.813138+0000 (oldest deadline 2024-04-23T10:42:27.113300+0000) 2024-04-23T10:42:48.993 INFO:journalctl@ceph.osd.10.smithi119.stdout:Apr 23 10:42:48 smithi119 ceph-aa240572-0159-11ef-bc93-c7b262605968-osd-10[57454]: 2024-04-23T10:42:48.711+0000 7fb1849bf700 -1 osd.10 116 heartbeat_check: no reply from 172.21.15.119:6804 osd.1 since back 2024-04-23T10:42:10.614599+0000 front 2024-04-23T10:42:10.614622+0000 (oldest deadline 2024-04-23T10:42:34.114881+0000) 2024-04-23T10:42:48.993 INFO:journalctl@ceph.osd.10.smithi119.stdout:Apr 23 10:42:48 smithi119 ceph-aa240572-0159-11ef-bc93-c7b262605968-osd-10[57454]: 2024-04-23T10:42:48.711+0000 7fb1849bf700 -1 osd.10 116 heartbeat_check: no reply from 172.21.15.204:6804 osd.2 since back 2024-04-23T10:42:19.416108+0000 front 2024-04-23T10:42:19.416241+0000 (oldest deadline 2024-04-23T10:42:42.316071+0000) 2024-04-23T10:42:49.078 INFO:journalctl@ceph.osd.11.smithi204.stdout:Apr 23 10:42:48 smithi204 ceph-aa240572-0159-11ef-bc93-c7b262605968-osd-11[57644]: 2024-04-23T10:42:48.722+0000 7fcbc4b63700 -1 osd.11 116 heartbeat_check: no reply from 172.21.15.114:6806 osd.0 since back 2024-04-23T10:42:00.675680+0000 front 2024-04-23T10:42:00.675649+0000 (oldest deadline 2024-04-23T10:42:25.876359+0000) 2024-04-23T10:42:49.078 INFO:journalctl@ceph.osd.11.smithi204.stdout:Apr 23 10:42:48 smithi204 ceph-aa240572-0159-11ef-bc93-c7b262605968-osd-11[57644]: 2024-04-23T10:42:48.722+0000 7fcbc4b63700 -1 osd.11 116 heartbeat_check: no reply from 172.21.15.119:6804 osd.1 since back 2024-04-23T10:42:09.278512+0000 front 2024-04-23T10:42:09.278494+0000 (oldest deadline 2024-04-23T10:42:33.378557+0000) 2024-04-23T10:42:49.078 INFO:journalctl@ceph.osd.11.smithi204.stdout:Apr 23 10:42:48 smithi204 ceph-aa240572-0159-11ef-bc93-c7b262605968-osd-11[57644]: 2024-04-23T10:42:48.722+0000 7fcbc4b63700 -1 osd.11 116 heartbeat_check: no reply from 172.21.15.204:6804 osd.2 since back 2024-04-23T10:42:18.679708+0000 front 2024-04-23T10:42:18.679577+0000 (oldest deadline 2024-04-23T10:42:41.579766+0000) 2024-04-23T10:42:49.754 INFO:journalctl@ceph.osd.9.smithi114.stdout:Apr 23 10:42:49 smithi114 ceph-aa240572-0159-11ef-bc93-c7b262605968-osd-9[62058]: 2024-04-23T10:42:49.331+0000 7feeb4bc7700 -1 osd.9 116 heartbeat_check: no reply from 172.21.15.114:6806 osd.0 since back 2024-04-23T10:42:04.136736+0000 front 2024-04-23T10:42:04.136764+0000 (oldest deadline 2024-04-23T10:42:28.836926+0000) 2024-04-23T10:42:49.754 INFO:journalctl@ceph.osd.9.smithi114.stdout:Apr 23 10:42:49 smithi114 ceph-aa240572-0159-11ef-bc93-c7b262605968-osd-9[62058]: 2024-04-23T10:42:49.331+0000 7feeb4bc7700 -1 osd.9 116 heartbeat_check: no reply from 172.21.15.119:6804 osd.1 since back 2024-04-23T10:42:08.837665+0000 front 2024-04-23T10:42:08.837503+0000 (oldest deadline 2024-04-23T10:42:34.737571+0000) 2024-04-23T10:42:49.755 INFO:journalctl@ceph.osd.9.smithi114.stdout:Apr 23 10:42:49 smithi114 ceph-aa240572-0159-11ef-bc93-c7b262605968-osd-9[62058]: 2024-04-23T10:42:49.331+0000 7feeb4bc7700 -1 osd.9 116 heartbeat_check: no reply from 172.21.15.204:6804 osd.2 since back 2024-04-23T10:42:18.139450+0000 front 2024-04-23T10:42:18.139388+0000 (oldest deadline 2024-04-23T10:42:43.439473+0000) 2024-04-23T10:42:49.755 INFO:journalctl@ceph.osd.6.smithi114.stdout:Apr 23 10:42:49 smithi114 ceph-aa240572-0159-11ef-bc93-c7b262605968-osd-6[55044]: 2024-04-23T10:42:49.541+0000 7fdba4fa4700 -1 osd.6 116 heartbeat_check: no reply from 172.21.15.114:6806 osd.0 since back 2024-04-23T10:42:01.662289+0000 front 2024-04-23T10:42:01.662441+0000 (oldest deadline 2024-04-23T10:42:26.362236+0000) 2024-04-23T10:42:49.755 INFO:journalctl@ceph.osd.6.smithi114.stdout:Apr 23 10:42:49 smithi114 ceph-aa240572-0159-11ef-bc93-c7b262605968-osd-6[55044]: 2024-04-23T10:42:49.541+0000 7fdba4fa4700 -1 osd.6 116 heartbeat_check: no reply from 172.21.15.119:6804 osd.1 since back 2024-04-23T10:42:09.763752+0000 front 2024-04-23T10:42:09.763645+0000 (oldest deadline 2024-04-23T10:42:35.063704+0000) 2024-04-23T10:42:49.755 INFO:journalctl@ceph.osd.6.smithi114.stdout:Apr 23 10:42:49 smithi114 ceph-aa240572-0159-11ef-bc93-c7b262605968-osd-6[55044]: 2024-04-23T10:42:49.541+0000 7fdba4fa4700 -1 osd.6 116 heartbeat_check: no reply from 172.21.15.204:6804 osd.2 since back 2024-04-23T10:42:17.364953+0000 front 2024-04-23T10:42:17.365063+0000 (oldest deadline 2024-04-23T10:42:47.365241+0000) 2024-04-23T10:42:49.764 INFO:journalctl@ceph.osd.8.smithi204.stdout:Apr 23 10:42:49 smithi204 ceph-aa240572-0159-11ef-bc93-c7b262605968-osd-8[51050]: 2024-04-23T10:42:49.393+0000 7f8d662cd700 -1 osd.8 116 heartbeat_check: no reply from 172.21.15.114:6806 osd.0 since back 2024-04-23T10:42:03.759281+0000 front 2024-04-23T10:42:03.759348+0000 (oldest deadline 2024-04-23T10:42:27.859455+0000) 2024-04-23T10:42:49.764 INFO:journalctl@ceph.osd.8.smithi204.stdout:Apr 23 10:42:49 smithi204 ceph-aa240572-0159-11ef-bc93-c7b262605968-osd-8[51050]: 2024-04-23T10:42:49.393+0000 7f8d662cd700 -1 osd.8 116 heartbeat_check: no reply from 172.21.15.119:6804 osd.1 since back 2024-04-23T10:42:09.560739+0000 front 2024-04-23T10:42:09.560541+0000 (oldest deadline 2024-04-23T10:42:37.761119+0000) 2024-04-23T10:42:49.992 INFO:journalctl@ceph.osd.7.smithi119.stdout:Apr 23 10:42:49 smithi119 ceph-aa240572-0159-11ef-bc93-c7b262605968-osd-7[50859]: 2024-04-23T10:42:49.796+0000 7f015f194700 -1 osd.7 116 heartbeat_check: no reply from 172.21.15.119:6804 osd.1 since back 2024-04-23T10:42:09.664009+0000 front 2024-04-23T10:42:09.664029+0000 (oldest deadline 2024-04-23T10:42:38.464808+0000) 2024-04-23T10:42:49.992 INFO:journalctl@ceph.osd.7.smithi119.stdout:Apr 23 10:42:49 smithi119 ceph-aa240572-0159-11ef-bc93-c7b262605968-osd-7[50859]: 2024-04-23T10:42:49.796+0000 7f015f194700 -1 osd.7 116 heartbeat_check: no reply from 172.21.15.204:6804 osd.2 since back 2024-04-23T10:42:19.566017+0000 front 2024-04-23T10:42:19.566039+0000 (oldest deadline 2024-04-23T10:42:41.766809+0000) 2024-04-23T10:42:49.992 INFO:journalctl@ceph.osd.10.smithi119.stdout:Apr 23 10:42:49 smithi119 ceph-aa240572-0159-11ef-bc93-c7b262605968-osd-10[57454]: 2024-04-23T10:42:49.732+0000 7fb1849bf700 -1 osd.10 116 heartbeat_check: no reply from 172.21.15.114:6806 osd.0 since back 2024-04-23T10:42:01.813174+0000 front 2024-04-23T10:42:01.813138+0000 (oldest deadline 2024-04-23T10:42:27.113300+0000) 2024-04-23T10:42:49.993 INFO:journalctl@ceph.osd.10.smithi119.stdout:Apr 23 10:42:49 smithi119 ceph-aa240572-0159-11ef-bc93-c7b262605968-osd-10[57454]: 2024-04-23T10:42:49.732+0000 7fb1849bf700 -1 osd.10 116 heartbeat_check: no reply from 172.21.15.119:6804 osd.1 since back 2024-04-23T10:42:10.614599+0000 front 2024-04-23T10:42:10.614622+0000 (oldest deadline 2024-04-23T10:42:34.114881+0000) 2024-04-23T10:42:49.993 INFO:journalctl@ceph.osd.10.smithi119.stdout:Apr 23 10:42:49 smithi119 ceph-aa240572-0159-11ef-bc93-c7b262605968-osd-10[57454]: 2024-04-23T10:42:49.732+0000 7fb1849bf700 -1 osd.10 116 heartbeat_check: no reply from 172.21.15.204:6804 osd.2 since back 2024-04-23T10:42:19.416108+0000 front 2024-04-23T10:42:19.416241+0000 (oldest deadline 2024-04-23T10:42:42.316071+0000) 2024-04-23T10:42:50.078 INFO:journalctl@ceph.osd.11.smithi204.stdout:Apr 23 10:42:49 smithi204 ceph-aa240572-0159-11ef-bc93-c7b262605968-osd-11[57644]: 2024-04-23T10:42:49.766+0000 7fcbc4b63700 -1 osd.11 116 heartbeat_check: no reply from 172.21.15.114:6806 osd.0 since back 2024-04-23T10:42:00.675680+0000 front 2024-04-23T10:42:00.675649+0000 (oldest deadline 2024-04-23T10:42:25.876359+0000) 2024-04-23T10:42:50.078 INFO:journalctl@ceph.osd.11.smithi204.stdout:Apr 23 10:42:49 smithi204 ceph-aa240572-0159-11ef-bc93-c7b262605968-osd-11[57644]: 2024-04-23T10:42:49.766+0000 7fcbc4b63700 -1 osd.11 116 heartbeat_check: no reply from 172.21.15.119:6804 osd.1 since back 2024-04-23T10:42:09.278512+0000 front 2024-04-23T10:42:09.278494+0000 (oldest deadline 2024-04-23T10:42:33.378557+0000) 2024-04-23T10:42:50.078 INFO:journalctl@ceph.osd.11.smithi204.stdout:Apr 23 10:42:49 smithi204 ceph-aa240572-0159-11ef-bc93-c7b262605968-osd-11[57644]: 2024-04-23T10:42:49.766+0000 7fcbc4b63700 -1 osd.11 116 heartbeat_check: no reply from 172.21.15.204:6804 osd.2 since back 2024-04-23T10:42:18.679708+0000 front 2024-04-23T10:42:18.679577+0000 (oldest deadline 2024-04-23T10:42:41.579766+0000) 2024-04-23T10:42:50.754 INFO:journalctl@ceph.osd.9.smithi114.stdout:Apr 23 10:42:50 smithi114 ceph-aa240572-0159-11ef-bc93-c7b262605968-osd-9[62058]: 2024-04-23T10:42:50.308+0000 7feeb4bc7700 -1 osd.9 116 heartbeat_check: no reply from 172.21.15.114:6806 osd.0 since back 2024-04-23T10:42:04.136736+0000 front 2024-04-23T10:42:04.136764+0000 (oldest deadline 2024-04-23T10:42:28.836926+0000) 2024-04-23T10:42:50.755 INFO:journalctl@ceph.osd.9.smithi114.stdout:Apr 23 10:42:50 smithi114 ceph-aa240572-0159-11ef-bc93-c7b262605968-osd-9[62058]: 2024-04-23T10:42:50.308+0000 7feeb4bc7700 -1 osd.9 116 heartbeat_check: no reply from 172.21.15.119:6804 osd.1 since back 2024-04-23T10:42:08.837665+0000 front 2024-04-23T10:42:08.837503+0000 (oldest deadline 2024-04-23T10:42:34.737571+0000) 2024-04-23T10:42:50.755 INFO:journalctl@ceph.osd.9.smithi114.stdout:Apr 23 10:42:50 smithi114 ceph-aa240572-0159-11ef-bc93-c7b262605968-osd-9[62058]: 2024-04-23T10:42:50.308+0000 7feeb4bc7700 -1 osd.9 116 heartbeat_check: no reply from 172.21.15.204:6804 osd.2 since back 2024-04-23T10:42:18.139450+0000 front 2024-04-23T10:42:18.139388+0000 (oldest deadline 2024-04-23T10:42:43.439473+0000) 2024-04-23T10:42:50.755 INFO:journalctl@ceph.osd.6.smithi114.stdout:Apr 23 10:42:50 smithi114 ceph-aa240572-0159-11ef-bc93-c7b262605968-osd-6[55044]: 2024-04-23T10:42:50.552+0000 7fdba4fa4700 -1 osd.6 116 heartbeat_check: no reply from 172.21.15.114:6806 osd.0 since back 2024-04-23T10:42:01.662289+0000 front 2024-04-23T10:42:01.662441+0000 (oldest deadline 2024-04-23T10:42:26.362236+0000) 2024-04-23T10:42:50.755 INFO:journalctl@ceph.osd.6.smithi114.stdout:Apr 23 10:42:50 smithi114 ceph-aa240572-0159-11ef-bc93-c7b262605968-osd-6[55044]: 2024-04-23T10:42:50.552+0000 7fdba4fa4700 -1 osd.6 116 heartbeat_check: no reply from 172.21.15.119:6804 osd.1 since back 2024-04-23T10:42:09.763752+0000 front 2024-04-23T10:42:09.763645+0000 (oldest deadline 2024-04-23T10:42:35.063704+0000) 2024-04-23T10:42:50.756 INFO:journalctl@ceph.osd.6.smithi114.stdout:Apr 23 10:42:50 smithi114 ceph-aa240572-0159-11ef-bc93-c7b262605968-osd-6[55044]: 2024-04-23T10:42:50.552+0000 7fdba4fa4700 -1 osd.6 116 heartbeat_check: no reply from 172.21.15.204:6804 osd.2 since back 2024-04-23T10:42:17.364953+0000 front 2024-04-23T10:42:17.365063+0000 (oldest deadline 2024-04-23T10:42:47.365241+0000) 2024-04-23T10:42:50.792 INFO:journalctl@ceph.osd.8.smithi204.stdout:Apr 23 10:42:50 smithi204 ceph-aa240572-0159-11ef-bc93-c7b262605968-osd-8[51050]: 2024-04-23T10:42:50.377+0000 7f8d662cd700 -1 osd.8 116 heartbeat_check: no reply from 172.21.15.114:6806 osd.0 since back 2024-04-23T10:42:03.759281+0000 front 2024-04-23T10:42:03.759348+0000 (oldest deadline 2024-04-23T10:42:27.859455+0000) 2024-04-23T10:42:50.792 INFO:journalctl@ceph.osd.8.smithi204.stdout:Apr 23 10:42:50 smithi204 ceph-aa240572-0159-11ef-bc93-c7b262605968-osd-8[51050]: 2024-04-23T10:42:50.377+0000 7f8d662cd700 -1 osd.8 116 heartbeat_check: no reply from 172.21.15.119:6804 osd.1 since back 2024-04-23T10:42:09.560739+0000 front 2024-04-23T10:42:09.560541+0000 (oldest deadline 2024-04-23T10:42:37.761119+0000) 2024-04-23T10:42:50.793 INFO:journalctl@ceph.osd.8.smithi204.stdout:Apr 23 10:42:50 smithi204 ceph-aa240572-0159-11ef-bc93-c7b262605968-osd-8[51050]: 2024-04-23T10:42:50.377+0000 7f8d662cd700 -1 osd.8 116 heartbeat_check: no reply from 172.21.15.114:6814 osd.3 since back 2024-04-23T10:42:24.762812+0000 front 2024-04-23T10:42:24.762866+0000 (oldest deadline 2024-04-23T10:42:50.062939+0000) 2024-04-23T10:42:50.992 INFO:journalctl@ceph.osd.7.smithi119.stdout:Apr 23 10:42:50 smithi119 ceph-aa240572-0159-11ef-bc93-c7b262605968-osd-7[50859]: 2024-04-23T10:42:50.752+0000 7f015f194700 -1 osd.7 116 heartbeat_check: no reply from 172.21.15.119:6804 osd.1 since back 2024-04-23T10:42:09.664009+0000 front 2024-04-23T10:42:09.664029+0000 (oldest deadline 2024-04-23T10:42:38.464808+0000) 2024-04-23T10:42:50.992 INFO:journalctl@ceph.osd.7.smithi119.stdout:Apr 23 10:42:50 smithi119 ceph-aa240572-0159-11ef-bc93-c7b262605968-osd-7[50859]: 2024-04-23T10:42:50.752+0000 7f015f194700 -1 osd.7 116 heartbeat_check: no reply from 172.21.15.204:6804 osd.2 since back 2024-04-23T10:42:19.566017+0000 front 2024-04-23T10:42:19.566039+0000 (oldest deadline 2024-04-23T10:42:41.766809+0000) 2024-04-23T10:42:50.992 INFO:journalctl@ceph.osd.10.smithi119.stdout:Apr 23 10:42:50 smithi119 ceph-aa240572-0159-11ef-bc93-c7b262605968-osd-10[57454]: 2024-04-23T10:42:50.712+0000 7fb1849bf700 -1 osd.10 116 heartbeat_check: no reply from 172.21.15.114:6806 osd.0 since back 2024-04-23T10:42:01.813174+0000 front 2024-04-23T10:42:01.813138+0000 (oldest deadline 2024-04-23T10:42:27.113300+0000) 2024-04-23T10:42:50.993 INFO:journalctl@ceph.osd.10.smithi119.stdout:Apr 23 10:42:50 smithi119 ceph-aa240572-0159-11ef-bc93-c7b262605968-osd-10[57454]: 2024-04-23T10:42:50.712+0000 7fb1849bf700 -1 osd.10 116 heartbeat_check: no reply from 172.21.15.119:6804 osd.1 since back 2024-04-23T10:42:10.614599+0000 front 2024-04-23T10:42:10.614622+0000 (oldest deadline 2024-04-23T10:42:34.114881+0000) 2024-04-23T10:42:50.993 INFO:journalctl@ceph.osd.10.smithi119.stdout:Apr 23 10:42:50 smithi119 ceph-aa240572-0159-11ef-bc93-c7b262605968-osd-10[57454]: 2024-04-23T10:42:50.712+0000 7fb1849bf700 -1 osd.10 116 heartbeat_check: no reply from 172.21.15.204:6804 osd.2 since back 2024-04-23T10:42:19.416108+0000 front 2024-04-23T10:42:19.416241+0000 (oldest deadline 2024-04-23T10:42:42.316071+0000) 2024-04-23T10:42:50.993 INFO:journalctl@ceph.osd.10.smithi119.stdout:Apr 23 10:42:50 smithi119 ceph-aa240572-0159-11ef-bc93-c7b262605968-osd-10[57454]: 2024-04-23T10:42:50.712+0000 7fb1849bf700 -1 osd.10 116 heartbeat_check: no reply from 172.21.15.114:6814 osd.3 since back 2024-04-23T10:42:25.217210+0000 front 2024-04-23T10:42:25.217268+0000 (oldest deadline 2024-04-23T10:42:49.917291+0000) 2024-04-23T10:42:51.078 INFO:journalctl@ceph.osd.11.smithi204.stdout:Apr 23 10:42:50 smithi204 ceph-aa240572-0159-11ef-bc93-c7b262605968-osd-11[57644]: 2024-04-23T10:42:50.793+0000 7fcbc4b63700 -1 osd.11 116 heartbeat_check: no reply from 172.21.15.114:6806 osd.0 since back 2024-04-23T10:42:00.675680+0000 front 2024-04-23T10:42:00.675649+0000 (oldest deadline 2024-04-23T10:42:25.876359+0000) 2024-04-23T10:42:51.078 INFO:journalctl@ceph.osd.11.smithi204.stdout:Apr 23 10:42:50 smithi204 ceph-aa240572-0159-11ef-bc93-c7b262605968-osd-11[57644]: 2024-04-23T10:42:50.793+0000 7fcbc4b63700 -1 osd.11 116 heartbeat_check: no reply from 172.21.15.119:6804 osd.1 since back 2024-04-23T10:42:09.278512+0000 front 2024-04-23T10:42:09.278494+0000 (oldest deadline 2024-04-23T10:42:33.378557+0000) 2024-04-23T10:42:51.078 INFO:journalctl@ceph.osd.11.smithi204.stdout:Apr 23 10:42:50 smithi204 ceph-aa240572-0159-11ef-bc93-c7b262605968-osd-11[57644]: 2024-04-23T10:42:50.793+0000 7fcbc4b63700 -1 osd.11 116 heartbeat_check: no reply from 172.21.15.204:6804 osd.2 since back 2024-04-23T10:42:18.679708+0000 front 2024-04-23T10:42:18.679577+0000 (oldest deadline 2024-04-23T10:42:41.579766+0000) 2024-04-23T10:42:51.754 INFO:journalctl@ceph.osd.9.smithi114.stdout:Apr 23 10:42:51 smithi114 ceph-aa240572-0159-11ef-bc93-c7b262605968-osd-9[62058]: 2024-04-23T10:42:51.327+0000 7feeb4bc7700 -1 osd.9 116 heartbeat_check: no reply from 172.21.15.114:6806 osd.0 since back 2024-04-23T10:42:04.136736+0000 front 2024-04-23T10:42:04.136764+0000 (oldest deadline 2024-04-23T10:42:28.836926+0000) 2024-04-23T10:42:51.755 INFO:journalctl@ceph.osd.9.smithi114.stdout:Apr 23 10:42:51 smithi114 ceph-aa240572-0159-11ef-bc93-c7b262605968-osd-9[62058]: 2024-04-23T10:42:51.327+0000 7feeb4bc7700 -1 osd.9 116 heartbeat_check: no reply from 172.21.15.119:6804 osd.1 since back 2024-04-23T10:42:08.837665+0000 front 2024-04-23T10:42:08.837503+0000 (oldest deadline 2024-04-23T10:42:34.737571+0000) 2024-04-23T10:42:51.755 INFO:journalctl@ceph.osd.9.smithi114.stdout:Apr 23 10:42:51 smithi114 ceph-aa240572-0159-11ef-bc93-c7b262605968-osd-9[62058]: 2024-04-23T10:42:51.327+0000 7feeb4bc7700 -1 osd.9 116 heartbeat_check: no reply from 172.21.15.204:6804 osd.2 since back 2024-04-23T10:42:18.139450+0000 front 2024-04-23T10:42:18.139388+0000 (oldest deadline 2024-04-23T10:42:43.439473+0000) 2024-04-23T10:42:51.755 INFO:journalctl@ceph.osd.6.smithi114.stdout:Apr 23 10:42:51 smithi114 ceph-aa240572-0159-11ef-bc93-c7b262605968-osd-6[55044]: 2024-04-23T10:42:51.510+0000 7fdba4fa4700 -1 osd.6 116 heartbeat_check: no reply from 172.21.15.114:6806 osd.0 since back 2024-04-23T10:42:01.662289+0000 front 2024-04-23T10:42:01.662441+0000 (oldest deadline 2024-04-23T10:42:26.362236+0000) 2024-04-23T10:42:51.756 INFO:journalctl@ceph.osd.6.smithi114.stdout:Apr 23 10:42:51 smithi114 ceph-aa240572-0159-11ef-bc93-c7b262605968-osd-6[55044]: 2024-04-23T10:42:51.510+0000 7fdba4fa4700 -1 osd.6 116 heartbeat_check: no reply from 172.21.15.119:6804 osd.1 since back 2024-04-23T10:42:09.763752+0000 front 2024-04-23T10:42:09.763645+0000 (oldest deadline 2024-04-23T10:42:35.063704+0000) 2024-04-23T10:42:51.756 INFO:journalctl@ceph.osd.6.smithi114.stdout:Apr 23 10:42:51 smithi114 ceph-aa240572-0159-11ef-bc93-c7b262605968-osd-6[55044]: 2024-04-23T10:42:51.510+0000 7fdba4fa4700 -1 osd.6 116 heartbeat_check: no reply from 172.21.15.204:6804 osd.2 since back 2024-04-23T10:42:17.364953+0000 front 2024-04-23T10:42:17.365063+0000 (oldest deadline 2024-04-23T10:42:47.365241+0000) 2024-04-23T10:42:51.819 INFO:journalctl@ceph.osd.8.smithi204.stdout:Apr 23 10:42:51 smithi204 ceph-aa240572-0159-11ef-bc93-c7b262605968-osd-8[51050]: 2024-04-23T10:42:51.376+0000 7f8d662cd700 -1 osd.8 116 heartbeat_check: no reply from 172.21.15.114:6806 osd.0 since back 2024-04-23T10:42:03.759281+0000 front 2024-04-23T10:42:03.759348+0000 (oldest deadline 2024-04-23T10:42:27.859455+0000) 2024-04-23T10:42:51.819 INFO:journalctl@ceph.osd.8.smithi204.stdout:Apr 23 10:42:51 smithi204 ceph-aa240572-0159-11ef-bc93-c7b262605968-osd-8[51050]: 2024-04-23T10:42:51.376+0000 7f8d662cd700 -1 osd.8 116 heartbeat_check: no reply from 172.21.15.119:6804 osd.1 since back 2024-04-23T10:42:09.560739+0000 front 2024-04-23T10:42:09.560541+0000 (oldest deadline 2024-04-23T10:42:37.761119+0000) 2024-04-23T10:42:51.819 INFO:journalctl@ceph.osd.8.smithi204.stdout:Apr 23 10:42:51 smithi204 ceph-aa240572-0159-11ef-bc93-c7b262605968-osd-8[51050]: 2024-04-23T10:42:51.376+0000 7f8d662cd700 -1 osd.8 116 heartbeat_check: no reply from 172.21.15.114:6814 osd.3 since back 2024-04-23T10:42:24.762812+0000 front 2024-04-23T10:42:24.762866+0000 (oldest deadline 2024-04-23T10:42:50.062939+0000) 2024-04-23T10:42:51.992 INFO:journalctl@ceph.osd.7.smithi119.stdout:Apr 23 10:42:51 smithi119 ceph-aa240572-0159-11ef-bc93-c7b262605968-osd-7[50859]: 2024-04-23T10:42:51.760+0000 7f015f194700 -1 osd.7 116 heartbeat_check: no reply from 172.21.15.119:6804 osd.1 since back 2024-04-23T10:42:09.664009+0000 front 2024-04-23T10:42:09.664029+0000 (oldest deadline 2024-04-23T10:42:38.464808+0000) 2024-04-23T10:42:51.992 INFO:journalctl@ceph.osd.7.smithi119.stdout:Apr 23 10:42:51 smithi119 ceph-aa240572-0159-11ef-bc93-c7b262605968-osd-7[50859]: 2024-04-23T10:42:51.760+0000 7f015f194700 -1 osd.7 116 heartbeat_check: no reply from 172.21.15.204:6804 osd.2 since back 2024-04-23T10:42:19.566017+0000 front 2024-04-23T10:42:19.566039+0000 (oldest deadline 2024-04-23T10:42:41.766809+0000) 2024-04-23T10:42:51.992 INFO:journalctl@ceph.osd.10.smithi119.stdout:Apr 23 10:42:51 smithi119 ceph-aa240572-0159-11ef-bc93-c7b262605968-osd-10[57454]: 2024-04-23T10:42:51.732+0000 7fb1849bf700 -1 osd.10 116 heartbeat_check: no reply from 172.21.15.114:6806 osd.0 since back 2024-04-23T10:42:01.813174+0000 front 2024-04-23T10:42:01.813138+0000 (oldest deadline 2024-04-23T10:42:27.113300+0000) 2024-04-23T10:42:51.992 INFO:journalctl@ceph.osd.10.smithi119.stdout:Apr 23 10:42:51 smithi119 ceph-aa240572-0159-11ef-bc93-c7b262605968-osd-10[57454]: 2024-04-23T10:42:51.732+0000 7fb1849bf700 -1 osd.10 116 heartbeat_check: no reply from 172.21.15.119:6804 osd.1 since back 2024-04-23T10:42:10.614599+0000 front 2024-04-23T10:42:10.614622+0000 (oldest deadline 2024-04-23T10:42:34.114881+0000) 2024-04-23T10:42:51.993 INFO:journalctl@ceph.osd.10.smithi119.stdout:Apr 23 10:42:51 smithi119 ceph-aa240572-0159-11ef-bc93-c7b262605968-osd-10[57454]: 2024-04-23T10:42:51.732+0000 7fb1849bf700 -1 osd.10 116 heartbeat_check: no reply from 172.21.15.204:6804 osd.2 since back 2024-04-23T10:42:19.416108+0000 front 2024-04-23T10:42:19.416241+0000 (oldest deadline 2024-04-23T10:42:42.316071+0000) 2024-04-23T10:42:51.993 INFO:journalctl@ceph.osd.10.smithi119.stdout:Apr 23 10:42:51 smithi119 ceph-aa240572-0159-11ef-bc93-c7b262605968-osd-10[57454]: 2024-04-23T10:42:51.732+0000 7fb1849bf700 -1 osd.10 116 heartbeat_check: no reply from 172.21.15.114:6814 osd.3 since back 2024-04-23T10:42:25.217210+0000 front 2024-04-23T10:42:25.217268+0000 (oldest deadline 2024-04-23T10:42:49.917291+0000) 2024-04-23T10:42:52.078 INFO:journalctl@ceph.osd.11.smithi204.stdout:Apr 23 10:42:51 smithi204 ceph-aa240572-0159-11ef-bc93-c7b262605968-osd-11[57644]: 2024-04-23T10:42:51.820+0000 7fcbc4b63700 -1 osd.11 116 heartbeat_check: no reply from 172.21.15.114:6806 osd.0 since back 2024-04-23T10:42:00.675680+0000 front 2024-04-23T10:42:00.675649+0000 (oldest deadline 2024-04-23T10:42:25.876359+0000) 2024-04-23T10:42:52.078 INFO:journalctl@ceph.osd.11.smithi204.stdout:Apr 23 10:42:51 smithi204 ceph-aa240572-0159-11ef-bc93-c7b262605968-osd-11[57644]: 2024-04-23T10:42:51.820+0000 7fcbc4b63700 -1 osd.11 116 heartbeat_check: no reply from 172.21.15.119:6804 osd.1 since back 2024-04-23T10:42:09.278512+0000 front 2024-04-23T10:42:09.278494+0000 (oldest deadline 2024-04-23T10:42:33.378557+0000) 2024-04-23T10:42:52.078 INFO:journalctl@ceph.osd.11.smithi204.stdout:Apr 23 10:42:51 smithi204 ceph-aa240572-0159-11ef-bc93-c7b262605968-osd-11[57644]: 2024-04-23T10:42:51.820+0000 7fcbc4b63700 -1 osd.11 116 heartbeat_check: no reply from 172.21.15.204:6804 osd.2 since back 2024-04-23T10:42:18.679708+0000 front 2024-04-23T10:42:18.679577+0000 (oldest deadline 2024-04-23T10:42:41.579766+0000) 2024-04-23T10:42:52.545 INFO:journalctl@ceph.osd.9.smithi114.stdout:Apr 23 10:42:52 smithi114 ceph-aa240572-0159-11ef-bc93-c7b262605968-osd-9[62058]: 2024-04-23T10:42:52.280+0000 7feeb4bc7700 -1 osd.9 116 heartbeat_check: no reply from 172.21.15.114:6806 osd.0 since back 2024-04-23T10:42:04.136736+0000 front 2024-04-23T10:42:04.136764+0000 (oldest deadline 2024-04-23T10:42:28.836926+0000) 2024-04-23T10:42:52.545 INFO:journalctl@ceph.osd.9.smithi114.stdout:Apr 23 10:42:52 smithi114 ceph-aa240572-0159-11ef-bc93-c7b262605968-osd-9[62058]: 2024-04-23T10:42:52.280+0000 7feeb4bc7700 -1 osd.9 116 heartbeat_check: no reply from 172.21.15.119:6804 osd.1 since back 2024-04-23T10:42:08.837665+0000 front 2024-04-23T10:42:08.837503+0000 (oldest deadline 2024-04-23T10:42:34.737571+0000) 2024-04-23T10:42:52.545 INFO:journalctl@ceph.osd.9.smithi114.stdout:Apr 23 10:42:52 smithi114 ceph-aa240572-0159-11ef-bc93-c7b262605968-osd-9[62058]: 2024-04-23T10:42:52.280+0000 7feeb4bc7700 -1 osd.9 116 heartbeat_check: no reply from 172.21.15.204:6804 osd.2 since back 2024-04-23T10:42:18.139450+0000 front 2024-04-23T10:42:18.139388+0000 (oldest deadline 2024-04-23T10:42:43.439473+0000) 2024-04-23T10:42:52.776 INFO:journalctl@ceph.osd.8.smithi204.stdout:Apr 23 10:42:52 smithi204 ceph-aa240572-0159-11ef-bc93-c7b262605968-osd-8[51050]: 2024-04-23T10:42:52.385+0000 7f8d662cd700 -1 osd.8 116 heartbeat_check: no reply from 172.21.15.114:6806 osd.0 since back 2024-04-23T10:42:03.759281+0000 front 2024-04-23T10:42:03.759348+0000 (oldest deadline 2024-04-23T10:42:27.859455+0000) 2024-04-23T10:42:52.777 INFO:journalctl@ceph.osd.8.smithi204.stdout:Apr 23 10:42:52 smithi204 ceph-aa240572-0159-11ef-bc93-c7b262605968-osd-8[51050]: 2024-04-23T10:42:52.385+0000 7f8d662cd700 -1 osd.8 116 heartbeat_check: no reply from 172.21.15.119:6804 osd.1 since back 2024-04-23T10:42:09.560739+0000 front 2024-04-23T10:42:09.560541+0000 (oldest deadline 2024-04-23T10:42:37.761119+0000) 2024-04-23T10:42:52.777 INFO:journalctl@ceph.osd.8.smithi204.stdout:Apr 23 10:42:52 smithi204 ceph-aa240572-0159-11ef-bc93-c7b262605968-osd-8[51050]: 2024-04-23T10:42:52.385+0000 7f8d662cd700 -1 osd.8 116 heartbeat_check: no reply from 172.21.15.114:6814 osd.3 since back 2024-04-23T10:42:24.762812+0000 front 2024-04-23T10:42:24.762866+0000 (oldest deadline 2024-04-23T10:42:50.062939+0000) 2024-04-23T10:42:52.992 INFO:journalctl@ceph.osd.7.smithi119.stdout:Apr 23 10:42:52 smithi119 ceph-aa240572-0159-11ef-bc93-c7b262605968-osd-7[50859]: 2024-04-23T10:42:52.733+0000 7f015f194700 -1 osd.7 116 heartbeat_check: no reply from 172.21.15.119:6804 osd.1 since back 2024-04-23T10:42:09.664009+0000 front 2024-04-23T10:42:09.664029+0000 (oldest deadline 2024-04-23T10:42:38.464808+0000) 2024-04-23T10:42:52.992 INFO:journalctl@ceph.osd.7.smithi119.stdout:Apr 23 10:42:52 smithi119 ceph-aa240572-0159-11ef-bc93-c7b262605968-osd-7[50859]: 2024-04-23T10:42:52.733+0000 7f015f194700 -1 osd.7 116 heartbeat_check: no reply from 172.21.15.204:6804 osd.2 since back 2024-04-23T10:42:19.566017+0000 front 2024-04-23T10:42:19.566039+0000 (oldest deadline 2024-04-23T10:42:41.766809+0000) 2024-04-23T10:42:52.992 INFO:journalctl@ceph.osd.7.smithi119.stdout:Apr 23 10:42:52 smithi119 ceph-aa240572-0159-11ef-bc93-c7b262605968-osd-7[50859]: 2024-04-23T10:42:52.733+0000 7f015f194700 -1 osd.7 116 heartbeat_check: no reply from 172.21.15.114:6814 osd.3 since back 2024-04-23T10:42:26.468167+0000 front 2024-04-23T10:42:26.468303+0000 (oldest deadline 2024-04-23T10:42:51.768175+0000) 2024-04-23T10:42:52.993 INFO:journalctl@ceph.osd.10.smithi119.stdout:Apr 23 10:42:52 smithi119 ceph-aa240572-0159-11ef-bc93-c7b262605968-osd-10[57454]: 2024-04-23T10:42:52.720+0000 7fb1849bf700 -1 osd.10 116 heartbeat_check: no reply from 172.21.15.114:6806 osd.0 since back 2024-04-23T10:42:01.813174+0000 front 2024-04-23T10:42:01.813138+0000 (oldest deadline 2024-04-23T10:42:27.113300+0000) 2024-04-23T10:42:52.993 INFO:journalctl@ceph.osd.10.smithi119.stdout:Apr 23 10:42:52 smithi119 ceph-aa240572-0159-11ef-bc93-c7b262605968-osd-10[57454]: 2024-04-23T10:42:52.720+0000 7fb1849bf700 -1 osd.10 116 heartbeat_check: no reply from 172.21.15.119:6804 osd.1 since back 2024-04-23T10:42:10.614599+0000 front 2024-04-23T10:42:10.614622+0000 (oldest deadline 2024-04-23T10:42:34.114881+0000) 2024-04-23T10:42:52.993 INFO:journalctl@ceph.osd.10.smithi119.stdout:Apr 23 10:42:52 smithi119 ceph-aa240572-0159-11ef-bc93-c7b262605968-osd-10[57454]: 2024-04-23T10:42:52.720+0000 7fb1849bf700 -1 osd.10 116 heartbeat_check: no reply from 172.21.15.204:6804 osd.2 since back 2024-04-23T10:42:19.416108+0000 front 2024-04-23T10:42:19.416241+0000 (oldest deadline 2024-04-23T10:42:42.316071+0000) 2024-04-23T10:42:52.993 INFO:journalctl@ceph.osd.10.smithi119.stdout:Apr 23 10:42:52 smithi119 ceph-aa240572-0159-11ef-bc93-c7b262605968-osd-10[57454]: 2024-04-23T10:42:52.720+0000 7fb1849bf700 -1 osd.10 116 heartbeat_check: no reply from 172.21.15.114:6814 osd.3 since back 2024-04-23T10:42:25.217210+0000 front 2024-04-23T10:42:25.217268+0000 (oldest deadline 2024-04-23T10:42:49.917291+0000) 2024-04-23T10:42:53.004 INFO:journalctl@ceph.osd.6.smithi114.stdout:Apr 23 10:42:52 smithi114 ceph-aa240572-0159-11ef-bc93-c7b262605968-osd-6[55044]: 2024-04-23T10:42:52.546+0000 7fdba4fa4700 -1 osd.6 116 heartbeat_check: no reply from 172.21.15.114:6806 osd.0 since back 2024-04-23T10:42:01.662289+0000 front 2024-04-23T10:42:01.662441+0000 (oldest deadline 2024-04-23T10:42:26.362236+0000) 2024-04-23T10:42:53.004 INFO:journalctl@ceph.osd.6.smithi114.stdout:Apr 23 10:42:52 smithi114 ceph-aa240572-0159-11ef-bc93-c7b262605968-osd-6[55044]: 2024-04-23T10:42:52.546+0000 7fdba4fa4700 -1 osd.6 116 heartbeat_check: no reply from 172.21.15.119:6804 osd.1 since back 2024-04-23T10:42:09.763752+0000 front 2024-04-23T10:42:09.763645+0000 (oldest deadline 2024-04-23T10:42:35.063704+0000) 2024-04-23T10:42:53.004 INFO:journalctl@ceph.osd.6.smithi114.stdout:Apr 23 10:42:52 smithi114 ceph-aa240572-0159-11ef-bc93-c7b262605968-osd-6[55044]: 2024-04-23T10:42:52.546+0000 7fdba4fa4700 -1 osd.6 116 heartbeat_check: no reply from 172.21.15.204:6804 osd.2 since back 2024-04-23T10:42:17.364953+0000 front 2024-04-23T10:42:17.365063+0000 (oldest deadline 2024-04-23T10:42:47.365241+0000) 2024-04-23T10:42:53.078 INFO:journalctl@ceph.osd.11.smithi204.stdout:Apr 23 10:42:52 smithi204 ceph-aa240572-0159-11ef-bc93-c7b262605968-osd-11[57644]: 2024-04-23T10:42:52.778+0000 7fcbc4b63700 -1 osd.11 116 heartbeat_check: no reply from 172.21.15.114:6806 osd.0 since back 2024-04-23T10:42:00.675680+0000 front 2024-04-23T10:42:00.675649+0000 (oldest deadline 2024-04-23T10:42:25.876359+0000) 2024-04-23T10:42:53.078 INFO:journalctl@ceph.osd.11.smithi204.stdout:Apr 23 10:42:52 smithi204 ceph-aa240572-0159-11ef-bc93-c7b262605968-osd-11[57644]: 2024-04-23T10:42:52.778+0000 7fcbc4b63700 -1 osd.11 116 heartbeat_check: no reply from 172.21.15.119:6804 osd.1 since back 2024-04-23T10:42:09.278512+0000 front 2024-04-23T10:42:09.278494+0000 (oldest deadline 2024-04-23T10:42:33.378557+0000) 2024-04-23T10:42:53.078 INFO:journalctl@ceph.osd.11.smithi204.stdout:Apr 23 10:42:52 smithi204 ceph-aa240572-0159-11ef-bc93-c7b262605968-osd-11[57644]: 2024-04-23T10:42:52.778+0000 7fcbc4b63700 -1 osd.11 116 heartbeat_check: no reply from 172.21.15.204:6804 osd.2 since back 2024-04-23T10:42:18.679708+0000 front 2024-04-23T10:42:18.679577+0000 (oldest deadline 2024-04-23T10:42:41.579766+0000) 2024-04-23T10:42:53.289 INFO:journalctl@ceph.osd.6.smithi114.stdout:Apr 23 10:42:53 smithi114 podman[91344]: 2024-04-23 10:42:53.168155698 +0000 UTC m=+5.498146072 container died 8d9b1fa9358030ad2110a6492773781128596d9c88a76243f06ca652d488eb1a (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:068a558e3af4932cf1c60dd321c0bdf500075eb52ea1bf455e0973ba114fbfd4, name=ceph-aa240572-0159-11ef-bc93-c7b262605968-osd-6, org.label-schema.build-date=20240416, org.label-schema.name=CentOS Stream 8 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=wip-lusov-quiescer-d69f13d, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, ceph=True, CEPH_POINT_RELEASE=, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, io.buildah.version=1.33.5, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, GIT_CLEAN=True) 2024-04-23T10:42:53.539 INFO:journalctl@ceph.osd.6.smithi114.stdout:Apr 23 10:42:53 smithi114 podman[91344]: 2024-04-23 10:42:53.529451918 +0000 UTC m=+5.859442284 container cleanup 8d9b1fa9358030ad2110a6492773781128596d9c88a76243f06ca652d488eb1a (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:068a558e3af4932cf1c60dd321c0bdf500075eb52ea1bf455e0973ba114fbfd4, name=ceph-aa240572-0159-11ef-bc93-c7b262605968-osd-6, org.label-schema.build-date=20240416, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0, RELEASE=wip-lusov-quiescer-d69f13d, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS, io.buildah.version=1.33.5, org.label-schema.license=GPLv2, GIT_CLEAN=True, ceph=True, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 8 Base Image) 2024-04-23T10:42:53.539 INFO:journalctl@ceph.osd.6.smithi114.stdout:Apr 23 10:42:53 smithi114 bash[91344]: ceph-aa240572-0159-11ef-bc93-c7b262605968-osd-6 2024-04-23T10:42:53.540 INFO:journalctl@ceph.osd.9.smithi114.stdout:Apr 23 10:42:53 smithi114 ceph-aa240572-0159-11ef-bc93-c7b262605968-osd-9[62058]: 2024-04-23T10:42:53.288+0000 7feeb4bc7700 -1 osd.9 116 heartbeat_check: no reply from 172.21.15.114:6806 osd.0 since back 2024-04-23T10:42:04.136736+0000 front 2024-04-23T10:42:04.136764+0000 (oldest deadline 2024-04-23T10:42:28.836926+0000) 2024-04-23T10:42:53.540 INFO:journalctl@ceph.osd.9.smithi114.stdout:Apr 23 10:42:53 smithi114 ceph-aa240572-0159-11ef-bc93-c7b262605968-osd-9[62058]: 2024-04-23T10:42:53.288+0000 7feeb4bc7700 -1 osd.9 116 heartbeat_check: no reply from 172.21.15.119:6804 osd.1 since back 2024-04-23T10:42:08.837665+0000 front 2024-04-23T10:42:08.837503+0000 (oldest deadline 2024-04-23T10:42:34.737571+0000) 2024-04-23T10:42:53.540 INFO:journalctl@ceph.osd.9.smithi114.stdout:Apr 23 10:42:53 smithi114 ceph-aa240572-0159-11ef-bc93-c7b262605968-osd-9[62058]: 2024-04-23T10:42:53.288+0000 7feeb4bc7700 -1 osd.9 116 heartbeat_check: no reply from 172.21.15.204:6804 osd.2 since back 2024-04-23T10:42:18.139450+0000 front 2024-04-23T10:42:18.139388+0000 (oldest deadline 2024-04-23T10:42:43.439473+0000) 2024-04-23T10:42:53.540 INFO:journalctl@ceph.osd.9.smithi114.stdout:Apr 23 10:42:53 smithi114 ceph-aa240572-0159-11ef-bc93-c7b262605968-osd-9[62058]: 2024-04-23T10:42:53.288+0000 7feeb4bc7700 -1 osd.9 116 heartbeat_check: no reply from 172.21.15.114:6814 osd.3 since back 2024-04-23T10:42:28.641242+0000 front 2024-04-23T10:42:28.641354+0000 (oldest deadline 2024-04-23T10:42:52.741420+0000) 2024-04-23T10:42:53.824 INFO:journalctl@ceph.osd.8.smithi204.stdout:Apr 23 10:42:53 smithi204 ceph-aa240572-0159-11ef-bc93-c7b262605968-osd-8[51050]: 2024-04-23T10:42:53.424+0000 7f8d662cd700 -1 osd.8 116 heartbeat_check: no reply from 172.21.15.114:6806 osd.0 since back 2024-04-23T10:42:03.759281+0000 front 2024-04-23T10:42:03.759348+0000 (oldest deadline 2024-04-23T10:42:27.859455+0000) 2024-04-23T10:42:53.824 INFO:journalctl@ceph.osd.8.smithi204.stdout:Apr 23 10:42:53 smithi204 ceph-aa240572-0159-11ef-bc93-c7b262605968-osd-8[51050]: 2024-04-23T10:42:53.424+0000 7f8d662cd700 -1 osd.8 116 heartbeat_check: no reply from 172.21.15.119:6804 osd.1 since back 2024-04-23T10:42:09.560739+0000 front 2024-04-23T10:42:09.560541+0000 (oldest deadline 2024-04-23T10:42:37.761119+0000) 2024-04-23T10:42:53.824 INFO:journalctl@ceph.osd.8.smithi204.stdout:Apr 23 10:42:53 smithi204 ceph-aa240572-0159-11ef-bc93-c7b262605968-osd-8[51050]: 2024-04-23T10:42:53.424+0000 7f8d662cd700 -1 osd.8 116 heartbeat_check: no reply from 172.21.15.114:6814 osd.3 since back 2024-04-23T10:42:24.762812+0000 front 2024-04-23T10:42:24.762866+0000 (oldest deadline 2024-04-23T10:42:50.062939+0000) 2024-04-23T10:42:53.992 INFO:journalctl@ceph.osd.7.smithi119.stdout:Apr 23 10:42:53 smithi119 ceph-aa240572-0159-11ef-bc93-c7b262605968-osd-7[50859]: 2024-04-23T10:42:53.737+0000 7f015f194700 -1 osd.7 116 heartbeat_check: no reply from 172.21.15.119:6804 osd.1 since back 2024-04-23T10:42:09.664009+0000 front 2024-04-23T10:42:09.664029+0000 (oldest deadline 2024-04-23T10:42:38.464808+0000) 2024-04-23T10:42:53.993 INFO:journalctl@ceph.osd.7.smithi119.stdout:Apr 23 10:42:53 smithi119 ceph-aa240572-0159-11ef-bc93-c7b262605968-osd-7[50859]: 2024-04-23T10:42:53.737+0000 7f015f194700 -1 osd.7 116 heartbeat_check: no reply from 172.21.15.204:6804 osd.2 since back 2024-04-23T10:42:19.566017+0000 front 2024-04-23T10:42:19.566039+0000 (oldest deadline 2024-04-23T10:42:41.766809+0000) 2024-04-23T10:42:53.993 INFO:journalctl@ceph.osd.7.smithi119.stdout:Apr 23 10:42:53 smithi119 ceph-aa240572-0159-11ef-bc93-c7b262605968-osd-7[50859]: 2024-04-23T10:42:53.737+0000 7f015f194700 -1 osd.7 116 heartbeat_check: no reply from 172.21.15.114:6814 osd.3 since back 2024-04-23T10:42:26.468167+0000 front 2024-04-23T10:42:26.468303+0000 (oldest deadline 2024-04-23T10:42:51.768175+0000) 2024-04-23T10:42:53.993 INFO:journalctl@ceph.osd.10.smithi119.stdout:Apr 23 10:42:53 smithi119 ceph-aa240572-0159-11ef-bc93-c7b262605968-osd-10[57454]: 2024-04-23T10:42:53.724+0000 7fb1849bf700 -1 osd.10 116 heartbeat_check: no reply from 172.21.15.114:6806 osd.0 since back 2024-04-23T10:42:01.813174+0000 front 2024-04-23T10:42:01.813138+0000 (oldest deadline 2024-04-23T10:42:27.113300+0000) 2024-04-23T10:42:53.993 INFO:journalctl@ceph.osd.10.smithi119.stdout:Apr 23 10:42:53 smithi119 ceph-aa240572-0159-11ef-bc93-c7b262605968-osd-10[57454]: 2024-04-23T10:42:53.724+0000 7fb1849bf700 -1 osd.10 116 heartbeat_check: no reply from 172.21.15.119:6804 osd.1 since back 2024-04-23T10:42:10.614599+0000 front 2024-04-23T10:42:10.614622+0000 (oldest deadline 2024-04-23T10:42:34.114881+0000) 2024-04-23T10:42:53.994 INFO:journalctl@ceph.osd.10.smithi119.stdout:Apr 23 10:42:53 smithi119 ceph-aa240572-0159-11ef-bc93-c7b262605968-osd-10[57454]: 2024-04-23T10:42:53.724+0000 7fb1849bf700 -1 osd.10 116 heartbeat_check: no reply from 172.21.15.204:6804 osd.2 since back 2024-04-23T10:42:19.416108+0000 front 2024-04-23T10:42:19.416241+0000 (oldest deadline 2024-04-23T10:42:42.316071+0000) 2024-04-23T10:42:53.994 INFO:journalctl@ceph.osd.10.smithi119.stdout:Apr 23 10:42:53 smithi119 ceph-aa240572-0159-11ef-bc93-c7b262605968-osd-10[57454]: 2024-04-23T10:42:53.724+0000 7fb1849bf700 -1 osd.10 116 heartbeat_check: no reply from 172.21.15.114:6814 osd.3 since back 2024-04-23T10:42:25.217210+0000 front 2024-04-23T10:42:25.217268+0000 (oldest deadline 2024-04-23T10:42:49.917291+0000) 2024-04-23T10:42:54.078 INFO:journalctl@ceph.osd.11.smithi204.stdout:Apr 23 10:42:53 smithi204 ceph-aa240572-0159-11ef-bc93-c7b262605968-osd-11[57644]: 2024-04-23T10:42:53.825+0000 7fcbc4b63700 -1 osd.11 116 heartbeat_check: no reply from 172.21.15.114:6806 osd.0 since back 2024-04-23T10:42:00.675680+0000 front 2024-04-23T10:42:00.675649+0000 (oldest deadline 2024-04-23T10:42:25.876359+0000) 2024-04-23T10:42:54.078 INFO:journalctl@ceph.osd.11.smithi204.stdout:Apr 23 10:42:53 smithi204 ceph-aa240572-0159-11ef-bc93-c7b262605968-osd-11[57644]: 2024-04-23T10:42:53.825+0000 7fcbc4b63700 -1 osd.11 116 heartbeat_check: no reply from 172.21.15.119:6804 osd.1 since back 2024-04-23T10:42:09.278512+0000 front 2024-04-23T10:42:09.278494+0000 (oldest deadline 2024-04-23T10:42:33.378557+0000) 2024-04-23T10:42:54.078 INFO:journalctl@ceph.osd.11.smithi204.stdout:Apr 23 10:42:53 smithi204 ceph-aa240572-0159-11ef-bc93-c7b262605968-osd-11[57644]: 2024-04-23T10:42:53.825+0000 7fcbc4b63700 -1 osd.11 116 heartbeat_check: no reply from 172.21.15.204:6804 osd.2 since back 2024-04-23T10:42:18.679708+0000 front 2024-04-23T10:42:18.679577+0000 (oldest deadline 2024-04-23T10:42:41.579766+0000) 2024-04-23T10:42:54.147 INFO:journalctl@ceph.osd.6.smithi114.stdout:Apr 23 10:42:53 smithi114 podman[91362]: 2024-04-23 10:42:53.805752836 +0000 UTC m=+0.641587093 container remove 8d9b1fa9358030ad2110a6492773781128596d9c88a76243f06ca652d488eb1a (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:068a558e3af4932cf1c60dd321c0bdf500075eb52ea1bf455e0973ba114fbfd4, name=ceph-aa240572-0159-11ef-bc93-c7b262605968-osd-6, io.buildah.version=1.33.5, ceph=True, org.label-schema.name=CentOS Stream 8 Base Image, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, RELEASE=wip-lusov-quiescer-d69f13d, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0, maintainer=Guillaume Abrioux , GIT_CLEAN=True, CEPH_POINT_RELEASE=, org.label-schema.build-date=20240416) 2024-04-23T10:42:54.448 INFO:journalctl@ceph.osd.6.smithi114.stdout:Apr 23 10:42:54 smithi114 podman[91461]: 2024-04-23 10:42:54.052946997 +0000 UTC m=+0.019579449 image pull da4b004c09e820713483d039bc76fbd660be678f2ea3c03e059a6161b74dab76 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:068a558e3af4932cf1c60dd321c0bdf500075eb52ea1bf455e0973ba114fbfd4 2024-04-23T10:42:54.449 INFO:journalctl@ceph.osd.6.smithi114.stdout:Apr 23 10:42:54 smithi114 podman[91461]: 2024-04-23 10:42:54.164815679 +0000 UTC m=+0.131448118 container create 170233a53040afa6f306b1886c913f0ea47f1a8a52dfd816b91e1e10ee6f35e3 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:068a558e3af4932cf1c60dd321c0bdf500075eb52ea1bf455e0973ba114fbfd4, name=ceph-aa240572-0159-11ef-bc93-c7b262605968-osd-6-deactivate, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, RELEASE=wip-lusov-quiescer-d69f13d, CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.vendor=CentOS, GIT_CLEAN=True, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, org.label-schema.build-date=20240416, GIT_BRANCH=HEAD, io.buildah.version=1.33.5, ceph=True, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux ) 2024-04-23T10:42:54.449 INFO:journalctl@ceph.osd.6.smithi114.stdout:Apr 23 10:42:54 smithi114 podman[91461]: 2024-04-23 10:42:54.450564636 +0000 UTC m=+0.417197077 container init 170233a53040afa6f306b1886c913f0ea47f1a8a52dfd816b91e1e10ee6f35e3 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:068a558e3af4932cf1c60dd321c0bdf500075eb52ea1bf455e0973ba114fbfd4, name=ceph-aa240572-0159-11ef-bc93-c7b262605968-osd-6-deactivate, ceph=True, GIT_CLEAN=True, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, CEPH_POINT_RELEASE=, org.label-schema.build-date=20240416, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2, io.buildah.version=1.33.5, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, RELEASE=wip-lusov-quiescer-d69f13d, GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-04-23T10:42:54.449 INFO:journalctl@ceph.osd.9.smithi114.stdout:Apr 23 10:42:54 smithi114 ceph-aa240572-0159-11ef-bc93-c7b262605968-osd-9[62058]: 2024-04-23T10:42:54.348+0000 7feeb4bc7700 -1 osd.9 116 heartbeat_check: no reply from 172.21.15.114:6806 osd.0 since back 2024-04-23T10:42:04.136736+0000 front 2024-04-23T10:42:04.136764+0000 (oldest deadline 2024-04-23T10:42:28.836926+0000) 2024-04-23T10:42:54.449 INFO:journalctl@ceph.osd.9.smithi114.stdout:Apr 23 10:42:54 smithi114 ceph-aa240572-0159-11ef-bc93-c7b262605968-osd-9[62058]: 2024-04-23T10:42:54.348+0000 7feeb4bc7700 -1 osd.9 116 heartbeat_check: no reply from 172.21.15.119:6804 osd.1 since back 2024-04-23T10:42:08.837665+0000 front 2024-04-23T10:42:08.837503+0000 (oldest deadline 2024-04-23T10:42:34.737571+0000) 2024-04-23T10:42:54.450 INFO:journalctl@ceph.osd.9.smithi114.stdout:Apr 23 10:42:54 smithi114 ceph-aa240572-0159-11ef-bc93-c7b262605968-osd-9[62058]: 2024-04-23T10:42:54.348+0000 7feeb4bc7700 -1 osd.9 116 heartbeat_check: no reply from 172.21.15.204:6804 osd.2 since back 2024-04-23T10:42:18.139450+0000 front 2024-04-23T10:42:18.139388+0000 (oldest deadline 2024-04-23T10:42:43.439473+0000) 2024-04-23T10:42:54.450 INFO:journalctl@ceph.osd.9.smithi114.stdout:Apr 23 10:42:54 smithi114 ceph-aa240572-0159-11ef-bc93-c7b262605968-osd-9[62058]: 2024-04-23T10:42:54.348+0000 7feeb4bc7700 -1 osd.9 116 heartbeat_check: no reply from 172.21.15.114:6814 osd.3 since back 2024-04-23T10:42:28.641242+0000 front 2024-04-23T10:42:28.641354+0000 (oldest deadline 2024-04-23T10:42:52.741420+0000) 2024-04-23T10:42:54.742 INFO:journalctl@ceph.osd.6.smithi114.stdout:Apr 23 10:42:54 smithi114 podman[91461]: 2024-04-23 10:42:54.453909708 +0000 UTC m=+0.420542155 container start 170233a53040afa6f306b1886c913f0ea47f1a8a52dfd816b91e1e10ee6f35e3 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:068a558e3af4932cf1c60dd321c0bdf500075eb52ea1bf455e0973ba114fbfd4, name=ceph-aa240572-0159-11ef-bc93-c7b262605968-osd-6-deactivate, RELEASE=wip-lusov-quiescer-d69f13d, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240416, ceph=True, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, io.buildah.version=1.33.5, org.label-schema.name=CentOS Stream 8 Base Image, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS) 2024-04-23T10:42:54.743 INFO:journalctl@ceph.osd.6.smithi114.stdout:Apr 23 10:42:54 smithi114 podman[91461]: 2024-04-23 10:42:54.542123199 +0000 UTC m=+0.508755648 container attach 170233a53040afa6f306b1886c913f0ea47f1a8a52dfd816b91e1e10ee6f35e3 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:068a558e3af4932cf1c60dd321c0bdf500075eb52ea1bf455e0973ba114fbfd4, name=ceph-aa240572-0159-11ef-bc93-c7b262605968-osd-6-deactivate, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, org.label-schema.build-date=20240416, io.buildah.version=1.33.5, GIT_CLEAN=True, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, org.label-schema.name=CentOS Stream 8 Base Image, GIT_BRANCH=HEAD, ceph=True, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux , RELEASE=wip-lusov-quiescer-d69f13d, GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=) 2024-04-23T10:42:54.776 INFO:journalctl@ceph.osd.8.smithi204.stdout:Apr 23 10:42:54 smithi204 ceph-aa240572-0159-11ef-bc93-c7b262605968-osd-8[51050]: 2024-04-23T10:42:54.449+0000 7f8d662cd700 -1 osd.8 116 heartbeat_check: no reply from 172.21.15.114:6806 osd.0 since back 2024-04-23T10:42:03.759281+0000 front 2024-04-23T10:42:03.759348+0000 (oldest deadline 2024-04-23T10:42:27.859455+0000) 2024-04-23T10:42:54.776 INFO:journalctl@ceph.osd.8.smithi204.stdout:Apr 23 10:42:54 smithi204 ceph-aa240572-0159-11ef-bc93-c7b262605968-osd-8[51050]: 2024-04-23T10:42:54.449+0000 7f8d662cd700 -1 osd.8 116 heartbeat_check: no reply from 172.21.15.119:6804 osd.1 since back 2024-04-23T10:42:09.560739+0000 front 2024-04-23T10:42:09.560541+0000 (oldest deadline 2024-04-23T10:42:37.761119+0000) 2024-04-23T10:42:54.776 INFO:journalctl@ceph.osd.8.smithi204.stdout:Apr 23 10:42:54 smithi204 ceph-aa240572-0159-11ef-bc93-c7b262605968-osd-8[51050]: 2024-04-23T10:42:54.449+0000 7f8d662cd700 -1 osd.8 116 heartbeat_check: no reply from 172.21.15.114:6814 osd.3 since back 2024-04-23T10:42:24.762812+0000 front 2024-04-23T10:42:24.762866+0000 (oldest deadline 2024-04-23T10:42:50.062939+0000) 2024-04-23T10:42:54.992 INFO:journalctl@ceph.osd.7.smithi119.stdout:Apr 23 10:42:54 smithi119 ceph-aa240572-0159-11ef-bc93-c7b262605968-osd-7[50859]: 2024-04-23T10:42:54.743+0000 7f015f194700 -1 osd.7 116 heartbeat_check: no reply from 172.21.15.119:6804 osd.1 since back 2024-04-23T10:42:09.664009+0000 front 2024-04-23T10:42:09.664029+0000 (oldest deadline 2024-04-23T10:42:38.464808+0000) 2024-04-23T10:42:54.992 INFO:journalctl@ceph.osd.7.smithi119.stdout:Apr 23 10:42:54 smithi119 ceph-aa240572-0159-11ef-bc93-c7b262605968-osd-7[50859]: 2024-04-23T10:42:54.743+0000 7f015f194700 -1 osd.7 116 heartbeat_check: no reply from 172.21.15.204:6804 osd.2 since back 2024-04-23T10:42:19.566017+0000 front 2024-04-23T10:42:19.566039+0000 (oldest deadline 2024-04-23T10:42:41.766809+0000) 2024-04-23T10:42:54.993 INFO:journalctl@ceph.osd.7.smithi119.stdout:Apr 23 10:42:54 smithi119 ceph-aa240572-0159-11ef-bc93-c7b262605968-osd-7[50859]: 2024-04-23T10:42:54.743+0000 7f015f194700 -1 osd.7 116 heartbeat_check: no reply from 172.21.15.114:6814 osd.3 since back 2024-04-23T10:42:26.468167+0000 front 2024-04-23T10:42:26.468303+0000 (oldest deadline 2024-04-23T10:42:51.768175+0000) 2024-04-23T10:42:54.993 INFO:journalctl@ceph.osd.10.smithi119.stdout:Apr 23 10:42:54 smithi119 ceph-aa240572-0159-11ef-bc93-c7b262605968-osd-10[57454]: 2024-04-23T10:42:54.726+0000 7fb1849bf700 -1 osd.10 116 heartbeat_check: no reply from 172.21.15.114:6806 osd.0 since back 2024-04-23T10:42:01.813174+0000 front 2024-04-23T10:42:01.813138+0000 (oldest deadline 2024-04-23T10:42:27.113300+0000) 2024-04-23T10:42:54.993 INFO:journalctl@ceph.osd.10.smithi119.stdout:Apr 23 10:42:54 smithi119 ceph-aa240572-0159-11ef-bc93-c7b262605968-osd-10[57454]: 2024-04-23T10:42:54.726+0000 7fb1849bf700 -1 osd.10 116 heartbeat_check: no reply from 172.21.15.119:6804 osd.1 since back 2024-04-23T10:42:10.614599+0000 front 2024-04-23T10:42:10.614622+0000 (oldest deadline 2024-04-23T10:42:34.114881+0000) 2024-04-23T10:42:54.993 INFO:journalctl@ceph.osd.10.smithi119.stdout:Apr 23 10:42:54 smithi119 ceph-aa240572-0159-11ef-bc93-c7b262605968-osd-10[57454]: 2024-04-23T10:42:54.726+0000 7fb1849bf700 -1 osd.10 116 heartbeat_check: no reply from 172.21.15.204:6804 osd.2 since back 2024-04-23T10:42:19.416108+0000 front 2024-04-23T10:42:19.416241+0000 (oldest deadline 2024-04-23T10:42:42.316071+0000) 2024-04-23T10:42:54.993 INFO:journalctl@ceph.osd.10.smithi119.stdout:Apr 23 10:42:54 smithi119 ceph-aa240572-0159-11ef-bc93-c7b262605968-osd-10[57454]: 2024-04-23T10:42:54.726+0000 7fb1849bf700 -1 osd.10 116 heartbeat_check: no reply from 172.21.15.114:6814 osd.3 since back 2024-04-23T10:42:25.217210+0000 front 2024-04-23T10:42:25.217268+0000 (oldest deadline 2024-04-23T10:42:49.917291+0000) 2024-04-23T10:42:55.004 INFO:journalctl@ceph.osd.6.smithi114.stdout:Apr 23 10:42:54 smithi114 podman[91461]: 2024-04-23 10:42:54.744730264 +0000 UTC m=+0.711362709 container died 170233a53040afa6f306b1886c913f0ea47f1a8a52dfd816b91e1e10ee6f35e3 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:068a558e3af4932cf1c60dd321c0bdf500075eb52ea1bf455e0973ba114fbfd4, name=ceph-aa240572-0159-11ef-bc93-c7b262605968-osd-6-deactivate, org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240416, io.buildah.version=1.33.5, ceph=True, org.label-schema.name=CentOS Stream 8 Base Image, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, RELEASE=wip-lusov-quiescer-d69f13d, org.label-schema.schema-version=1.0) 2024-04-23T10:42:55.078 INFO:journalctl@ceph.osd.11.smithi204.stdout:Apr 23 10:42:54 smithi204 ceph-aa240572-0159-11ef-bc93-c7b262605968-osd-11[57644]: 2024-04-23T10:42:54.778+0000 7fcbc4b63700 -1 osd.11 116 heartbeat_check: no reply from 172.21.15.114:6806 osd.0 since back 2024-04-23T10:42:00.675680+0000 front 2024-04-23T10:42:00.675649+0000 (oldest deadline 2024-04-23T10:42:25.876359+0000) 2024-04-23T10:42:55.078 INFO:journalctl@ceph.osd.11.smithi204.stdout:Apr 23 10:42:54 smithi204 ceph-aa240572-0159-11ef-bc93-c7b262605968-osd-11[57644]: 2024-04-23T10:42:54.778+0000 7fcbc4b63700 -1 osd.11 116 heartbeat_check: no reply from 172.21.15.119:6804 osd.1 since back 2024-04-23T10:42:09.278512+0000 front 2024-04-23T10:42:09.278494+0000 (oldest deadline 2024-04-23T10:42:33.378557+0000) 2024-04-23T10:42:55.078 INFO:journalctl@ceph.osd.11.smithi204.stdout:Apr 23 10:42:54 smithi204 ceph-aa240572-0159-11ef-bc93-c7b262605968-osd-11[57644]: 2024-04-23T10:42:54.778+0000 7fcbc4b63700 -1 osd.11 116 heartbeat_check: no reply from 172.21.15.204:6804 osd.2 since back 2024-04-23T10:42:18.679708+0000 front 2024-04-23T10:42:18.679577+0000 (oldest deadline 2024-04-23T10:42:41.579766+0000) 2024-04-23T10:42:55.322 DEBUG:teuthology.orchestra.run.smithi114:> sudo pkill -f 'journalctl -f -n 0 -u ceph-aa240572-0159-11ef-bc93-c7b262605968@osd.6.service' 2024-04-23T10:42:55.360 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-04-23T10:42:55.361 INFO:tasks.cephadm.osd.6:Stopped osd.6 2024-04-23T10:42:55.361 INFO:tasks.cephadm.osd.7:Stopping osd.7... 2024-04-23T10:42:55.361 DEBUG:teuthology.orchestra.run.smithi119:> sudo systemctl stop ceph-aa240572-0159-11ef-bc93-c7b262605968@osd.7 2024-04-23T10:42:55.680 INFO:journalctl@ceph.osd.7.smithi119.stdout:Apr 23 10:42:55 smithi119 systemd[1]: Stopping Ceph osd.7 for aa240572-0159-11ef-bc93-c7b262605968... 2024-04-23T10:42:55.737 INFO:journalctl@ceph.osd.8.smithi204.stdout:Apr 23 10:42:55 smithi204 ceph-aa240572-0159-11ef-bc93-c7b262605968-osd-8[51050]: 2024-04-23T10:42:55.451+0000 7f8d662cd700 -1 osd.8 116 heartbeat_check: no reply from 172.21.15.114:6806 osd.0 since back 2024-04-23T10:42:03.759281+0000 front 2024-04-23T10:42:03.759348+0000 (oldest deadline 2024-04-23T10:42:27.859455+0000) 2024-04-23T10:42:55.737 INFO:journalctl@ceph.osd.8.smithi204.stdout:Apr 23 10:42:55 smithi204 ceph-aa240572-0159-11ef-bc93-c7b262605968-osd-8[51050]: 2024-04-23T10:42:55.451+0000 7f8d662cd700 -1 osd.8 116 heartbeat_check: no reply from 172.21.15.119:6804 osd.1 since back 2024-04-23T10:42:09.560739+0000 front 2024-04-23T10:42:09.560541+0000 (oldest deadline 2024-04-23T10:42:37.761119+0000) 2024-04-23T10:42:55.737 INFO:journalctl@ceph.osd.8.smithi204.stdout:Apr 23 10:42:55 smithi204 ceph-aa240572-0159-11ef-bc93-c7b262605968-osd-8[51050]: 2024-04-23T10:42:55.451+0000 7f8d662cd700 -1 osd.8 116 heartbeat_check: no reply from 172.21.15.114:6814 osd.3 since back 2024-04-23T10:42:24.762812+0000 front 2024-04-23T10:42:24.762866+0000 (oldest deadline 2024-04-23T10:42:50.062939+0000) 2024-04-23T10:42:55.754 INFO:journalctl@ceph.osd.9.smithi114.stdout:Apr 23 10:42:55 smithi114 ceph-aa240572-0159-11ef-bc93-c7b262605968-osd-9[62058]: 2024-04-23T10:42:55.332+0000 7feeb4bc7700 -1 osd.9 116 heartbeat_check: no reply from 172.21.15.114:6806 osd.0 since back 2024-04-23T10:42:04.136736+0000 front 2024-04-23T10:42:04.136764+0000 (oldest deadline 2024-04-23T10:42:28.836926+0000) 2024-04-23T10:42:55.754 INFO:journalctl@ceph.osd.9.smithi114.stdout:Apr 23 10:42:55 smithi114 ceph-aa240572-0159-11ef-bc93-c7b262605968-osd-9[62058]: 2024-04-23T10:42:55.332+0000 7feeb4bc7700 -1 osd.9 116 heartbeat_check: no reply from 172.21.15.119:6804 osd.1 since back 2024-04-23T10:42:08.837665+0000 front 2024-04-23T10:42:08.837503+0000 (oldest deadline 2024-04-23T10:42:34.737571+0000) 2024-04-23T10:42:55.754 INFO:journalctl@ceph.osd.9.smithi114.stdout:Apr 23 10:42:55 smithi114 ceph-aa240572-0159-11ef-bc93-c7b262605968-osd-9[62058]: 2024-04-23T10:42:55.332+0000 7feeb4bc7700 -1 osd.9 116 heartbeat_check: no reply from 172.21.15.204:6804 osd.2 since back 2024-04-23T10:42:18.139450+0000 front 2024-04-23T10:42:18.139388+0000 (oldest deadline 2024-04-23T10:42:43.439473+0000) 2024-04-23T10:42:55.754 INFO:journalctl@ceph.osd.9.smithi114.stdout:Apr 23 10:42:55 smithi114 ceph-aa240572-0159-11ef-bc93-c7b262605968-osd-9[62058]: 2024-04-23T10:42:55.332+0000 7feeb4bc7700 -1 osd.9 116 heartbeat_check: no reply from 172.21.15.114:6814 osd.3 since back 2024-04-23T10:42:28.641242+0000 front 2024-04-23T10:42:28.641354+0000 (oldest deadline 2024-04-23T10:42:52.741420+0000) 2024-04-23T10:42:55.992 INFO:journalctl@ceph.osd.7.smithi119.stdout:Apr 23 10:42:55 smithi119 ceph-aa240572-0159-11ef-bc93-c7b262605968-osd-7[50859]: 2024-04-23T10:42:55.681+0000 7f0166ba8700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.7 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-04-23T10:42:55.992 INFO:journalctl@ceph.osd.7.smithi119.stdout:Apr 23 10:42:55 smithi119 ceph-aa240572-0159-11ef-bc93-c7b262605968-osd-7[50859]: 2024-04-23T10:42:55.681+0000 7f0166ba8700 -1 osd.7 116 *** Got signal Terminated *** 2024-04-23T10:42:55.992 INFO:journalctl@ceph.osd.7.smithi119.stdout:Apr 23 10:42:55 smithi119 ceph-aa240572-0159-11ef-bc93-c7b262605968-osd-7[50859]: 2024-04-23T10:42:55.681+0000 7f0166ba8700 -1 osd.7 116 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-04-23T10:42:55.992 INFO:journalctl@ceph.osd.7.smithi119.stdout:Apr 23 10:42:55 smithi119 ceph-aa240572-0159-11ef-bc93-c7b262605968-osd-7[50859]: 2024-04-23T10:42:55.769+0000 7f015f194700 -1 osd.7 116 heartbeat_check: no reply from 172.21.15.119:6804 osd.1 since back 2024-04-23T10:42:09.664009+0000 front 2024-04-23T10:42:09.664029+0000 (oldest deadline 2024-04-23T10:42:38.464808+0000) 2024-04-23T10:42:55.992 INFO:journalctl@ceph.osd.7.smithi119.stdout:Apr 23 10:42:55 smithi119 ceph-aa240572-0159-11ef-bc93-c7b262605968-osd-7[50859]: 2024-04-23T10:42:55.769+0000 7f015f194700 -1 osd.7 116 heartbeat_check: no reply from 172.21.15.204:6804 osd.2 since back 2024-04-23T10:42:19.566017+0000 front 2024-04-23T10:42:19.566039+0000 (oldest deadline 2024-04-23T10:42:41.766809+0000) 2024-04-23T10:42:55.992 INFO:journalctl@ceph.osd.7.smithi119.stdout:Apr 23 10:42:55 smithi119 ceph-aa240572-0159-11ef-bc93-c7b262605968-osd-7[50859]: 2024-04-23T10:42:55.769+0000 7f015f194700 -1 osd.7 116 heartbeat_check: no reply from 172.21.15.114:6814 osd.3 since back 2024-04-23T10:42:26.468167+0000 front 2024-04-23T10:42:26.468303+0000 (oldest deadline 2024-04-23T10:42:51.768175+0000) 2024-04-23T10:42:55.992 INFO:journalctl@ceph.osd.10.smithi119.stdout:Apr 23 10:42:55 smithi119 ceph-aa240572-0159-11ef-bc93-c7b262605968-osd-10[57454]: 2024-04-23T10:42:55.734+0000 7fb1849bf700 -1 osd.10 116 heartbeat_check: no reply from 172.21.15.114:6806 osd.0 since back 2024-04-23T10:42:01.813174+0000 front 2024-04-23T10:42:01.813138+0000 (oldest deadline 2024-04-23T10:42:27.113300+0000) 2024-04-23T10:42:55.993 INFO:journalctl@ceph.osd.10.smithi119.stdout:Apr 23 10:42:55 smithi119 ceph-aa240572-0159-11ef-bc93-c7b262605968-osd-10[57454]: 2024-04-23T10:42:55.734+0000 7fb1849bf700 -1 osd.10 116 heartbeat_check: no reply from 172.21.15.119:6804 osd.1 since back 2024-04-23T10:42:10.614599+0000 front 2024-04-23T10:42:10.614622+0000 (oldest deadline 2024-04-23T10:42:34.114881+0000) 2024-04-23T10:42:55.993 INFO:journalctl@ceph.osd.10.smithi119.stdout:Apr 23 10:42:55 smithi119 ceph-aa240572-0159-11ef-bc93-c7b262605968-osd-10[57454]: 2024-04-23T10:42:55.734+0000 7fb1849bf700 -1 osd.10 116 heartbeat_check: no reply from 172.21.15.204:6804 osd.2 since back 2024-04-23T10:42:19.416108+0000 front 2024-04-23T10:42:19.416241+0000 (oldest deadline 2024-04-23T10:42:42.316071+0000) 2024-04-23T10:42:55.993 INFO:journalctl@ceph.osd.10.smithi119.stdout:Apr 23 10:42:55 smithi119 ceph-aa240572-0159-11ef-bc93-c7b262605968-osd-10[57454]: 2024-04-23T10:42:55.734+0000 7fb1849bf700 -1 osd.10 116 heartbeat_check: no reply from 172.21.15.114:6814 osd.3 since back 2024-04-23T10:42:25.217210+0000 front 2024-04-23T10:42:25.217268+0000 (oldest deadline 2024-04-23T10:42:49.917291+0000) 2024-04-23T10:42:56.078 INFO:journalctl@ceph.osd.11.smithi204.stdout:Apr 23 10:42:55 smithi204 ceph-aa240572-0159-11ef-bc93-c7b262605968-osd-11[57644]: 2024-04-23T10:42:55.737+0000 7fcbc4b63700 -1 osd.11 116 heartbeat_check: no reply from 172.21.15.114:6806 osd.0 since back 2024-04-23T10:42:00.675680+0000 front 2024-04-23T10:42:00.675649+0000 (oldest deadline 2024-04-23T10:42:25.876359+0000) 2024-04-23T10:42:56.078 INFO:journalctl@ceph.osd.11.smithi204.stdout:Apr 23 10:42:55 smithi204 ceph-aa240572-0159-11ef-bc93-c7b262605968-osd-11[57644]: 2024-04-23T10:42:55.737+0000 7fcbc4b63700 -1 osd.11 116 heartbeat_check: no reply from 172.21.15.119:6804 osd.1 since back 2024-04-23T10:42:09.278512+0000 front 2024-04-23T10:42:09.278494+0000 (oldest deadline 2024-04-23T10:42:33.378557+0000) 2024-04-23T10:42:56.078 INFO:journalctl@ceph.osd.11.smithi204.stdout:Apr 23 10:42:55 smithi204 ceph-aa240572-0159-11ef-bc93-c7b262605968-osd-11[57644]: 2024-04-23T10:42:55.737+0000 7fcbc4b63700 -1 osd.11 116 heartbeat_check: no reply from 172.21.15.204:6804 osd.2 since back 2024-04-23T10:42:18.679708+0000 front 2024-04-23T10:42:18.679577+0000 (oldest deadline 2024-04-23T10:42:41.579766+0000) 2024-04-23T10:42:56.695 INFO:journalctl@ceph.osd.8.smithi204.stdout:Apr 23 10:42:56 smithi204 ceph-aa240572-0159-11ef-bc93-c7b262605968-osd-8[51050]: 2024-04-23T10:42:56.420+0000 7f8d662cd700 -1 osd.8 116 heartbeat_check: no reply from 172.21.15.114:6806 osd.0 since back 2024-04-23T10:42:03.759281+0000 front 2024-04-23T10:42:03.759348+0000 (oldest deadline 2024-04-23T10:42:27.859455+0000) 2024-04-23T10:42:56.695 INFO:journalctl@ceph.osd.8.smithi204.stdout:Apr 23 10:42:56 smithi204 ceph-aa240572-0159-11ef-bc93-c7b262605968-osd-8[51050]: 2024-04-23T10:42:56.420+0000 7f8d662cd700 -1 osd.8 116 heartbeat_check: no reply from 172.21.15.119:6804 osd.1 since back 2024-04-23T10:42:09.560739+0000 front 2024-04-23T10:42:09.560541+0000 (oldest deadline 2024-04-23T10:42:37.761119+0000) 2024-04-23T10:42:56.695 INFO:journalctl@ceph.osd.8.smithi204.stdout:Apr 23 10:42:56 smithi204 ceph-aa240572-0159-11ef-bc93-c7b262605968-osd-8[51050]: 2024-04-23T10:42:56.420+0000 7f8d662cd700 -1 osd.8 116 heartbeat_check: no reply from 172.21.15.114:6814 osd.3 since back 2024-04-23T10:42:24.762812+0000 front 2024-04-23T10:42:24.762866+0000 (oldest deadline 2024-04-23T10:42:50.062939+0000) 2024-04-23T10:42:56.754 INFO:journalctl@ceph.osd.9.smithi114.stdout:Apr 23 10:42:56 smithi114 ceph-aa240572-0159-11ef-bc93-c7b262605968-osd-9[62058]: 2024-04-23T10:42:56.373+0000 7feeb4bc7700 -1 osd.9 116 heartbeat_check: no reply from 172.21.15.114:6806 osd.0 since back 2024-04-23T10:42:04.136736+0000 front 2024-04-23T10:42:04.136764+0000 (oldest deadline 2024-04-23T10:42:28.836926+0000) 2024-04-23T10:42:56.754 INFO:journalctl@ceph.osd.9.smithi114.stdout:Apr 23 10:42:56 smithi114 ceph-aa240572-0159-11ef-bc93-c7b262605968-osd-9[62058]: 2024-04-23T10:42:56.373+0000 7feeb4bc7700 -1 osd.9 116 heartbeat_check: no reply from 172.21.15.119:6804 osd.1 since back 2024-04-23T10:42:08.837665+0000 front 2024-04-23T10:42:08.837503+0000 (oldest deadline 2024-04-23T10:42:34.737571+0000) 2024-04-23T10:42:56.754 INFO:journalctl@ceph.osd.9.smithi114.stdout:Apr 23 10:42:56 smithi114 ceph-aa240572-0159-11ef-bc93-c7b262605968-osd-9[62058]: 2024-04-23T10:42:56.373+0000 7feeb4bc7700 -1 osd.9 116 heartbeat_check: no reply from 172.21.15.204:6804 osd.2 since back 2024-04-23T10:42:18.139450+0000 front 2024-04-23T10:42:18.139388+0000 (oldest deadline 2024-04-23T10:42:43.439473+0000) 2024-04-23T10:42:56.754 INFO:journalctl@ceph.osd.9.smithi114.stdout:Apr 23 10:42:56 smithi114 ceph-aa240572-0159-11ef-bc93-c7b262605968-osd-9[62058]: 2024-04-23T10:42:56.373+0000 7feeb4bc7700 -1 osd.9 116 heartbeat_check: no reply from 172.21.15.114:6814 osd.3 since back 2024-04-23T10:42:28.641242+0000 front 2024-04-23T10:42:28.641354+0000 (oldest deadline 2024-04-23T10:42:52.741420+0000) 2024-04-23T10:42:56.992 INFO:journalctl@ceph.osd.7.smithi119.stdout:Apr 23 10:42:56 smithi119 ceph-aa240572-0159-11ef-bc93-c7b262605968-osd-7[50859]: 2024-04-23T10:42:56.817+0000 7f015f194700 -1 osd.7 116 heartbeat_check: no reply from 172.21.15.119:6804 osd.1 since back 2024-04-23T10:42:09.664009+0000 front 2024-04-23T10:42:09.664029+0000 (oldest deadline 2024-04-23T10:42:38.464808+0000) 2024-04-23T10:42:56.992 INFO:journalctl@ceph.osd.7.smithi119.stdout:Apr 23 10:42:56 smithi119 ceph-aa240572-0159-11ef-bc93-c7b262605968-osd-7[50859]: 2024-04-23T10:42:56.817+0000 7f015f194700 -1 osd.7 116 heartbeat_check: no reply from 172.21.15.204:6804 osd.2 since back 2024-04-23T10:42:19.566017+0000 front 2024-04-23T10:42:19.566039+0000 (oldest deadline 2024-04-23T10:42:41.766809+0000) 2024-04-23T10:42:56.992 INFO:journalctl@ceph.osd.7.smithi119.stdout:Apr 23 10:42:56 smithi119 ceph-aa240572-0159-11ef-bc93-c7b262605968-osd-7[50859]: 2024-04-23T10:42:56.817+0000 7f015f194700 -1 osd.7 116 heartbeat_check: no reply from 172.21.15.114:6814 osd.3 since back 2024-04-23T10:42:26.468167+0000 front 2024-04-23T10:42:26.468303+0000 (oldest deadline 2024-04-23T10:42:51.768175+0000) 2024-04-23T10:42:56.992 INFO:journalctl@ceph.osd.10.smithi119.stdout:Apr 23 10:42:56 smithi119 ceph-aa240572-0159-11ef-bc93-c7b262605968-osd-10[57454]: 2024-04-23T10:42:56.695+0000 7fb1849bf700 -1 osd.10 116 heartbeat_check: no reply from 172.21.15.114:6806 osd.0 since back 2024-04-23T10:42:01.813174+0000 front 2024-04-23T10:42:01.813138+0000 (oldest deadline 2024-04-23T10:42:27.113300+0000) 2024-04-23T10:42:56.992 INFO:journalctl@ceph.osd.10.smithi119.stdout:Apr 23 10:42:56 smithi119 ceph-aa240572-0159-11ef-bc93-c7b262605968-osd-10[57454]: 2024-04-23T10:42:56.695+0000 7fb1849bf700 -1 osd.10 116 heartbeat_check: no reply from 172.21.15.119:6804 osd.1 since back 2024-04-23T10:42:10.614599+0000 front 2024-04-23T10:42:10.614622+0000 (oldest deadline 2024-04-23T10:42:34.114881+0000) 2024-04-23T10:42:56.993 INFO:journalctl@ceph.osd.10.smithi119.stdout:Apr 23 10:42:56 smithi119 ceph-aa240572-0159-11ef-bc93-c7b262605968-osd-10[57454]: 2024-04-23T10:42:56.695+0000 7fb1849bf700 -1 osd.10 116 heartbeat_check: no reply from 172.21.15.204:6804 osd.2 since back 2024-04-23T10:42:19.416108+0000 front 2024-04-23T10:42:19.416241+0000 (oldest deadline 2024-04-23T10:42:42.316071+0000) 2024-04-23T10:42:56.993 INFO:journalctl@ceph.osd.10.smithi119.stdout:Apr 23 10:42:56 smithi119 ceph-aa240572-0159-11ef-bc93-c7b262605968-osd-10[57454]: 2024-04-23T10:42:56.695+0000 7fb1849bf700 -1 osd.10 116 heartbeat_check: no reply from 172.21.15.114:6814 osd.3 since back 2024-04-23T10:42:25.217210+0000 front 2024-04-23T10:42:25.217268+0000 (oldest deadline 2024-04-23T10:42:49.917291+0000) 2024-04-23T10:42:57.077 INFO:journalctl@ceph.osd.11.smithi204.stdout:Apr 23 10:42:56 smithi204 ceph-aa240572-0159-11ef-bc93-c7b262605968-osd-11[57644]: 2024-04-23T10:42:56.697+0000 7fcbc4b63700 -1 osd.11 116 heartbeat_check: no reply from 172.21.15.114:6806 osd.0 since back 2024-04-23T10:42:00.675680+0000 front 2024-04-23T10:42:00.675649+0000 (oldest deadline 2024-04-23T10:42:25.876359+0000) 2024-04-23T10:42:57.078 INFO:journalctl@ceph.osd.11.smithi204.stdout:Apr 23 10:42:56 smithi204 ceph-aa240572-0159-11ef-bc93-c7b262605968-osd-11[57644]: 2024-04-23T10:42:56.697+0000 7fcbc4b63700 -1 osd.11 116 heartbeat_check: no reply from 172.21.15.119:6804 osd.1 since back 2024-04-23T10:42:09.278512+0000 front 2024-04-23T10:42:09.278494+0000 (oldest deadline 2024-04-23T10:42:33.378557+0000) 2024-04-23T10:42:57.078 INFO:journalctl@ceph.osd.11.smithi204.stdout:Apr 23 10:42:56 smithi204 ceph-aa240572-0159-11ef-bc93-c7b262605968-osd-11[57644]: 2024-04-23T10:42:56.697+0000 7fcbc4b63700 -1 osd.11 116 heartbeat_check: no reply from 172.21.15.204:6804 osd.2 since back 2024-04-23T10:42:18.679708+0000 front 2024-04-23T10:42:18.679577+0000 (oldest deadline 2024-04-23T10:42:41.579766+0000) 2024-04-23T10:42:57.754 INFO:journalctl@ceph.osd.9.smithi114.stdout:Apr 23 10:42:57 smithi114 ceph-aa240572-0159-11ef-bc93-c7b262605968-osd-9[62058]: 2024-04-23T10:42:57.365+0000 7feeb4bc7700 -1 osd.9 116 heartbeat_check: no reply from 172.21.15.114:6806 osd.0 since back 2024-04-23T10:42:04.136736+0000 front 2024-04-23T10:42:04.136764+0000 (oldest deadline 2024-04-23T10:42:28.836926+0000) 2024-04-23T10:42:57.755 INFO:journalctl@ceph.osd.9.smithi114.stdout:Apr 23 10:42:57 smithi114 ceph-aa240572-0159-11ef-bc93-c7b262605968-osd-9[62058]: 2024-04-23T10:42:57.365+0000 7feeb4bc7700 -1 osd.9 116 heartbeat_check: no reply from 172.21.15.119:6804 osd.1 since back 2024-04-23T10:42:08.837665+0000 front 2024-04-23T10:42:08.837503+0000 (oldest deadline 2024-04-23T10:42:34.737571+0000) 2024-04-23T10:42:57.755 INFO:journalctl@ceph.osd.9.smithi114.stdout:Apr 23 10:42:57 smithi114 ceph-aa240572-0159-11ef-bc93-c7b262605968-osd-9[62058]: 2024-04-23T10:42:57.365+0000 7feeb4bc7700 -1 osd.9 116 heartbeat_check: no reply from 172.21.15.204:6804 osd.2 since back 2024-04-23T10:42:18.139450+0000 front 2024-04-23T10:42:18.139388+0000 (oldest deadline 2024-04-23T10:42:43.439473+0000) 2024-04-23T10:42:57.755 INFO:journalctl@ceph.osd.9.smithi114.stdout:Apr 23 10:42:57 smithi114 ceph-aa240572-0159-11ef-bc93-c7b262605968-osd-9[62058]: 2024-04-23T10:42:57.365+0000 7feeb4bc7700 -1 osd.9 116 heartbeat_check: no reply from 172.21.15.114:6814 osd.3 since back 2024-04-23T10:42:28.641242+0000 front 2024-04-23T10:42:28.641354+0000 (oldest deadline 2024-04-23T10:42:52.741420+0000) 2024-04-23T10:42:57.828 INFO:journalctl@ceph.osd.8.smithi204.stdout:Apr 23 10:42:57 smithi204 ceph-aa240572-0159-11ef-bc93-c7b262605968-osd-8[51050]: 2024-04-23T10:42:57.466+0000 7f8d662cd700 -1 osd.8 116 heartbeat_check: no reply from 172.21.15.114:6806 osd.0 since back 2024-04-23T10:42:03.759281+0000 front 2024-04-23T10:42:03.759348+0000 (oldest deadline 2024-04-23T10:42:27.859455+0000) 2024-04-23T10:42:57.830 INFO:journalctl@ceph.osd.8.smithi204.stdout:Apr 23 10:42:57 smithi204 ceph-aa240572-0159-11ef-bc93-c7b262605968-osd-8[51050]: 2024-04-23T10:42:57.466+0000 7f8d662cd700 -1 osd.8 116 heartbeat_check: no reply from 172.21.15.119:6804 osd.1 since back 2024-04-23T10:42:09.560739+0000 front 2024-04-23T10:42:09.560541+0000 (oldest deadline 2024-04-23T10:42:37.761119+0000) 2024-04-23T10:42:57.830 INFO:journalctl@ceph.osd.8.smithi204.stdout:Apr 23 10:42:57 smithi204 ceph-aa240572-0159-11ef-bc93-c7b262605968-osd-8[51050]: 2024-04-23T10:42:57.466+0000 7f8d662cd700 -1 osd.8 116 heartbeat_check: no reply from 172.21.15.114:6814 osd.3 since back 2024-04-23T10:42:24.762812+0000 front 2024-04-23T10:42:24.762866+0000 (oldest deadline 2024-04-23T10:42:50.062939+0000) 2024-04-23T10:42:57.830 INFO:journalctl@ceph.osd.11.smithi204.stdout:Apr 23 10:42:57 smithi204 ceph-aa240572-0159-11ef-bc93-c7b262605968-osd-11[57644]: 2024-04-23T10:42:57.705+0000 7fcbc4b63700 -1 osd.11 116 heartbeat_check: no reply from 172.21.15.114:6806 osd.0 since back 2024-04-23T10:42:00.675680+0000 front 2024-04-23T10:42:00.675649+0000 (oldest deadline 2024-04-23T10:42:25.876359+0000) 2024-04-23T10:42:57.830 INFO:journalctl@ceph.osd.11.smithi204.stdout:Apr 23 10:42:57 smithi204 ceph-aa240572-0159-11ef-bc93-c7b262605968-osd-11[57644]: 2024-04-23T10:42:57.705+0000 7fcbc4b63700 -1 osd.11 116 heartbeat_check: no reply from 172.21.15.119:6804 osd.1 since back 2024-04-23T10:42:09.278512+0000 front 2024-04-23T10:42:09.278494+0000 (oldest deadline 2024-04-23T10:42:33.378557+0000) 2024-04-23T10:42:57.831 INFO:journalctl@ceph.osd.11.smithi204.stdout:Apr 23 10:42:57 smithi204 ceph-aa240572-0159-11ef-bc93-c7b262605968-osd-11[57644]: 2024-04-23T10:42:57.705+0000 7fcbc4b63700 -1 osd.11 116 heartbeat_check: no reply from 172.21.15.204:6804 osd.2 since back 2024-04-23T10:42:18.679708+0000 front 2024-04-23T10:42:18.679577+0000 (oldest deadline 2024-04-23T10:42:41.579766+0000) 2024-04-23T10:42:57.992 INFO:journalctl@ceph.osd.7.smithi119.stdout:Apr 23 10:42:57 smithi119 ceph-aa240572-0159-11ef-bc93-c7b262605968-osd-7[50859]: 2024-04-23T10:42:57.818+0000 7f015f194700 -1 osd.7 116 heartbeat_check: no reply from 172.21.15.119:6804 osd.1 since back 2024-04-23T10:42:09.664009+0000 front 2024-04-23T10:42:09.664029+0000 (oldest deadline 2024-04-23T10:42:38.464808+0000) 2024-04-23T10:42:57.992 INFO:journalctl@ceph.osd.7.smithi119.stdout:Apr 23 10:42:57 smithi119 ceph-aa240572-0159-11ef-bc93-c7b262605968-osd-7[50859]: 2024-04-23T10:42:57.818+0000 7f015f194700 -1 osd.7 116 heartbeat_check: no reply from 172.21.15.204:6804 osd.2 since back 2024-04-23T10:42:19.566017+0000 front 2024-04-23T10:42:19.566039+0000 (oldest deadline 2024-04-23T10:42:41.766809+0000) 2024-04-23T10:42:57.992 INFO:journalctl@ceph.osd.7.smithi119.stdout:Apr 23 10:42:57 smithi119 ceph-aa240572-0159-11ef-bc93-c7b262605968-osd-7[50859]: 2024-04-23T10:42:57.818+0000 7f015f194700 -1 osd.7 116 heartbeat_check: no reply from 172.21.15.114:6814 osd.3 since back 2024-04-23T10:42:26.468167+0000 front 2024-04-23T10:42:26.468303+0000 (oldest deadline 2024-04-23T10:42:51.768175+0000) 2024-04-23T10:42:57.993 INFO:journalctl@ceph.osd.10.smithi119.stdout:Apr 23 10:42:57 smithi119 ceph-aa240572-0159-11ef-bc93-c7b262605968-osd-10[57454]: 2024-04-23T10:42:57.653+0000 7fb1849bf700 -1 osd.10 116 heartbeat_check: no reply from 172.21.15.114:6806 osd.0 since back 2024-04-23T10:42:01.813174+0000 front 2024-04-23T10:42:01.813138+0000 (oldest deadline 2024-04-23T10:42:27.113300+0000) 2024-04-23T10:42:57.993 INFO:journalctl@ceph.osd.10.smithi119.stdout:Apr 23 10:42:57 smithi119 ceph-aa240572-0159-11ef-bc93-c7b262605968-osd-10[57454]: 2024-04-23T10:42:57.653+0000 7fb1849bf700 -1 osd.10 116 heartbeat_check: no reply from 172.21.15.119:6804 osd.1 since back 2024-04-23T10:42:10.614599+0000 front 2024-04-23T10:42:10.614622+0000 (oldest deadline 2024-04-23T10:42:34.114881+0000) 2024-04-23T10:42:57.993 INFO:journalctl@ceph.osd.10.smithi119.stdout:Apr 23 10:42:57 smithi119 ceph-aa240572-0159-11ef-bc93-c7b262605968-osd-10[57454]: 2024-04-23T10:42:57.653+0000 7fb1849bf700 -1 osd.10 116 heartbeat_check: no reply from 172.21.15.204:6804 osd.2 since back 2024-04-23T10:42:19.416108+0000 front 2024-04-23T10:42:19.416241+0000 (oldest deadline 2024-04-23T10:42:42.316071+0000) 2024-04-23T10:42:57.993 INFO:journalctl@ceph.osd.10.smithi119.stdout:Apr 23 10:42:57 smithi119 ceph-aa240572-0159-11ef-bc93-c7b262605968-osd-10[57454]: 2024-04-23T10:42:57.653+0000 7fb1849bf700 -1 osd.10 116 heartbeat_check: no reply from 172.21.15.114:6814 osd.3 since back 2024-04-23T10:42:25.217210+0000 front 2024-04-23T10:42:25.217268+0000 (oldest deadline 2024-04-23T10:42:49.917291+0000) 2024-04-23T10:42:58.754 INFO:journalctl@ceph.osd.9.smithi114.stdout:Apr 23 10:42:58 smithi114 ceph-aa240572-0159-11ef-bc93-c7b262605968-osd-9[62058]: 2024-04-23T10:42:58.322+0000 7feeb4bc7700 -1 osd.9 116 heartbeat_check: no reply from 172.21.15.114:6806 osd.0 since back 2024-04-23T10:42:04.136736+0000 front 2024-04-23T10:42:04.136764+0000 (oldest deadline 2024-04-23T10:42:28.836926+0000) 2024-04-23T10:42:58.754 INFO:journalctl@ceph.osd.9.smithi114.stdout:Apr 23 10:42:58 smithi114 ceph-aa240572-0159-11ef-bc93-c7b262605968-osd-9[62058]: 2024-04-23T10:42:58.322+0000 7feeb4bc7700 -1 osd.9 116 heartbeat_check: no reply from 172.21.15.119:6804 osd.1 since back 2024-04-23T10:42:08.837665+0000 front 2024-04-23T10:42:08.837503+0000 (oldest deadline 2024-04-23T10:42:34.737571+0000) 2024-04-23T10:42:58.754 INFO:journalctl@ceph.osd.9.smithi114.stdout:Apr 23 10:42:58 smithi114 ceph-aa240572-0159-11ef-bc93-c7b262605968-osd-9[62058]: 2024-04-23T10:42:58.322+0000 7feeb4bc7700 -1 osd.9 116 heartbeat_check: no reply from 172.21.15.204:6804 osd.2 since back 2024-04-23T10:42:18.139450+0000 front 2024-04-23T10:42:18.139388+0000 (oldest deadline 2024-04-23T10:42:43.439473+0000) 2024-04-23T10:42:58.755 INFO:journalctl@ceph.osd.9.smithi114.stdout:Apr 23 10:42:58 smithi114 ceph-aa240572-0159-11ef-bc93-c7b262605968-osd-9[62058]: 2024-04-23T10:42:58.322+0000 7feeb4bc7700 -1 osd.9 116 heartbeat_check: no reply from 172.21.15.114:6814 osd.3 since back 2024-04-23T10:42:28.641242+0000 front 2024-04-23T10:42:28.641354+0000 (oldest deadline 2024-04-23T10:42:52.741420+0000) 2024-04-23T10:42:58.828 INFO:journalctl@ceph.osd.11.smithi204.stdout:Apr 23 10:42:58 smithi204 ceph-aa240572-0159-11ef-bc93-c7b262605968-osd-11[57644]: 2024-04-23T10:42:58.658+0000 7fcbc4b63700 -1 osd.11 116 heartbeat_check: no reply from 172.21.15.114:6806 osd.0 since back 2024-04-23T10:42:00.675680+0000 front 2024-04-23T10:42:00.675649+0000 (oldest deadline 2024-04-23T10:42:25.876359+0000) 2024-04-23T10:42:58.828 INFO:journalctl@ceph.osd.11.smithi204.stdout:Apr 23 10:42:58 smithi204 ceph-aa240572-0159-11ef-bc93-c7b262605968-osd-11[57644]: 2024-04-23T10:42:58.658+0000 7fcbc4b63700 -1 osd.11 116 heartbeat_check: no reply from 172.21.15.119:6804 osd.1 since back 2024-04-23T10:42:09.278512+0000 front 2024-04-23T10:42:09.278494+0000 (oldest deadline 2024-04-23T10:42:33.378557+0000) 2024-04-23T10:42:58.828 INFO:journalctl@ceph.osd.11.smithi204.stdout:Apr 23 10:42:58 smithi204 ceph-aa240572-0159-11ef-bc93-c7b262605968-osd-11[57644]: 2024-04-23T10:42:58.658+0000 7fcbc4b63700 -1 osd.11 116 heartbeat_check: no reply from 172.21.15.204:6804 osd.2 since back 2024-04-23T10:42:18.679708+0000 front 2024-04-23T10:42:18.679577+0000 (oldest deadline 2024-04-23T10:42:41.579766+0000) 2024-04-23T10:42:58.828 INFO:journalctl@ceph.osd.8.smithi204.stdout:Apr 23 10:42:58 smithi204 ceph-aa240572-0159-11ef-bc93-c7b262605968-osd-8[51050]: 2024-04-23T10:42:58.464+0000 7f8d662cd700 -1 osd.8 116 heartbeat_check: no reply from 172.21.15.114:6806 osd.0 since back 2024-04-23T10:42:03.759281+0000 front 2024-04-23T10:42:03.759348+0000 (oldest deadline 2024-04-23T10:42:27.859455+0000) 2024-04-23T10:42:58.829 INFO:journalctl@ceph.osd.8.smithi204.stdout:Apr 23 10:42:58 smithi204 ceph-aa240572-0159-11ef-bc93-c7b262605968-osd-8[51050]: 2024-04-23T10:42:58.464+0000 7f8d662cd700 -1 osd.8 116 heartbeat_check: no reply from 172.21.15.119:6804 osd.1 since back 2024-04-23T10:42:09.560739+0000 front 2024-04-23T10:42:09.560541+0000 (oldest deadline 2024-04-23T10:42:37.761119+0000) 2024-04-23T10:42:58.829 INFO:journalctl@ceph.osd.8.smithi204.stdout:Apr 23 10:42:58 smithi204 ceph-aa240572-0159-11ef-bc93-c7b262605968-osd-8[51050]: 2024-04-23T10:42:58.464+0000 7f8d662cd700 -1 osd.8 116 heartbeat_check: no reply from 172.21.15.114:6814 osd.3 since back 2024-04-23T10:42:24.762812+0000 front 2024-04-23T10:42:24.762866+0000 (oldest deadline 2024-04-23T10:42:50.062939+0000) 2024-04-23T10:42:58.992 INFO:journalctl@ceph.osd.7.smithi119.stdout:Apr 23 10:42:58 smithi119 ceph-aa240572-0159-11ef-bc93-c7b262605968-osd-7[50859]: 2024-04-23T10:42:58.821+0000 7f015f194700 -1 osd.7 116 heartbeat_check: no reply from 172.21.15.119:6804 osd.1 since back 2024-04-23T10:42:09.664009+0000 front 2024-04-23T10:42:09.664029+0000 (oldest deadline 2024-04-23T10:42:38.464808+0000) 2024-04-23T10:42:58.992 INFO:journalctl@ceph.osd.7.smithi119.stdout:Apr 23 10:42:58 smithi119 ceph-aa240572-0159-11ef-bc93-c7b262605968-osd-7[50859]: 2024-04-23T10:42:58.821+0000 7f015f194700 -1 osd.7 116 heartbeat_check: no reply from 172.21.15.204:6804 osd.2 since back 2024-04-23T10:42:19.566017+0000 front 2024-04-23T10:42:19.566039+0000 (oldest deadline 2024-04-23T10:42:41.766809+0000) 2024-04-23T10:42:58.992 INFO:journalctl@ceph.osd.7.smithi119.stdout:Apr 23 10:42:58 smithi119 ceph-aa240572-0159-11ef-bc93-c7b262605968-osd-7[50859]: 2024-04-23T10:42:58.821+0000 7f015f194700 -1 osd.7 116 heartbeat_check: no reply from 172.21.15.114:6814 osd.3 since back 2024-04-23T10:42:26.468167+0000 front 2024-04-23T10:42:26.468303+0000 (oldest deadline 2024-04-23T10:42:51.768175+0000) 2024-04-23T10:42:58.993 INFO:journalctl@ceph.osd.10.smithi119.stdout:Apr 23 10:42:58 smithi119 ceph-aa240572-0159-11ef-bc93-c7b262605968-osd-10[57454]: 2024-04-23T10:42:58.693+0000 7fb1849bf700 -1 osd.10 116 heartbeat_check: no reply from 172.21.15.114:6806 osd.0 since back 2024-04-23T10:42:01.813174+0000 front 2024-04-23T10:42:01.813138+0000 (oldest deadline 2024-04-23T10:42:27.113300+0000) 2024-04-23T10:42:58.993 INFO:journalctl@ceph.osd.10.smithi119.stdout:Apr 23 10:42:58 smithi119 ceph-aa240572-0159-11ef-bc93-c7b262605968-osd-10[57454]: 2024-04-23T10:42:58.693+0000 7fb1849bf700 -1 osd.10 116 heartbeat_check: no reply from 172.21.15.119:6804 osd.1 since back 2024-04-23T10:42:10.614599+0000 front 2024-04-23T10:42:10.614622+0000 (oldest deadline 2024-04-23T10:42:34.114881+0000) 2024-04-23T10:42:58.993 INFO:journalctl@ceph.osd.10.smithi119.stdout:Apr 23 10:42:58 smithi119 ceph-aa240572-0159-11ef-bc93-c7b262605968-osd-10[57454]: 2024-04-23T10:42:58.693+0000 7fb1849bf700 -1 osd.10 116 heartbeat_check: no reply from 172.21.15.204:6804 osd.2 since back 2024-04-23T10:42:19.416108+0000 front 2024-04-23T10:42:19.416241+0000 (oldest deadline 2024-04-23T10:42:42.316071+0000) 2024-04-23T10:42:58.993 INFO:journalctl@ceph.osd.10.smithi119.stdout:Apr 23 10:42:58 smithi119 ceph-aa240572-0159-11ef-bc93-c7b262605968-osd-10[57454]: 2024-04-23T10:42:58.693+0000 7fb1849bf700 -1 osd.10 116 heartbeat_check: no reply from 172.21.15.114:6814 osd.3 since back 2024-04-23T10:42:25.217210+0000 front 2024-04-23T10:42:25.217268+0000 (oldest deadline 2024-04-23T10:42:49.917291+0000) 2024-04-23T10:42:59.754 INFO:journalctl@ceph.osd.9.smithi114.stdout:Apr 23 10:42:59 smithi114 ceph-aa240572-0159-11ef-bc93-c7b262605968-osd-9[62058]: 2024-04-23T10:42:59.297+0000 7feeb4bc7700 -1 osd.9 116 heartbeat_check: no reply from 172.21.15.114:6806 osd.0 since back 2024-04-23T10:42:04.136736+0000 front 2024-04-23T10:42:04.136764+0000 (oldest deadline 2024-04-23T10:42:28.836926+0000) 2024-04-23T10:42:59.755 INFO:journalctl@ceph.osd.9.smithi114.stdout:Apr 23 10:42:59 smithi114 ceph-aa240572-0159-11ef-bc93-c7b262605968-osd-9[62058]: 2024-04-23T10:42:59.297+0000 7feeb4bc7700 -1 osd.9 116 heartbeat_check: no reply from 172.21.15.119:6804 osd.1 since back 2024-04-23T10:42:08.837665+0000 front 2024-04-23T10:42:08.837503+0000 (oldest deadline 2024-04-23T10:42:34.737571+0000) 2024-04-23T10:42:59.755 INFO:journalctl@ceph.osd.9.smithi114.stdout:Apr 23 10:42:59 smithi114 ceph-aa240572-0159-11ef-bc93-c7b262605968-osd-9[62058]: 2024-04-23T10:42:59.297+0000 7feeb4bc7700 -1 osd.9 116 heartbeat_check: no reply from 172.21.15.204:6804 osd.2 since back 2024-04-23T10:42:18.139450+0000 front 2024-04-23T10:42:18.139388+0000 (oldest deadline 2024-04-23T10:42:43.439473+0000) 2024-04-23T10:42:59.755 INFO:journalctl@ceph.osd.9.smithi114.stdout:Apr 23 10:42:59 smithi114 ceph-aa240572-0159-11ef-bc93-c7b262605968-osd-9[62058]: 2024-04-23T10:42:59.297+0000 7feeb4bc7700 -1 osd.9 116 heartbeat_check: no reply from 172.21.15.114:6814 osd.3 since back 2024-04-23T10:42:28.641242+0000 front 2024-04-23T10:42:28.641354+0000 (oldest deadline 2024-04-23T10:42:52.741420+0000) 2024-04-23T10:42:59.755 INFO:journalctl@ceph.osd.9.smithi114.stdout:Apr 23 10:42:59 smithi114 ceph-aa240572-0159-11ef-bc93-c7b262605968-osd-9[62058]: 2024-04-23T10:42:59.297+0000 7feeb4bc7700 -1 osd.9 116 heartbeat_check: no reply from 172.21.15.119:6812 osd.4 since back 2024-04-23T10:42:32.742101+0000 front 2024-04-23T10:42:32.742087+0000 (oldest deadline 2024-04-23T10:42:58.642130+0000) 2024-04-23T10:42:59.828 INFO:journalctl@ceph.osd.11.smithi204.stdout:Apr 23 10:42:59 smithi204 ceph-aa240572-0159-11ef-bc93-c7b262605968-osd-11[57644]: 2024-04-23T10:42:59.645+0000 7fcbc4b63700 -1 osd.11 116 heartbeat_check: no reply from 172.21.15.114:6806 osd.0 since back 2024-04-23T10:42:00.675680+0000 front 2024-04-23T10:42:00.675649+0000 (oldest deadline 2024-04-23T10:42:25.876359+0000) 2024-04-23T10:42:59.828 INFO:journalctl@ceph.osd.11.smithi204.stdout:Apr 23 10:42:59 smithi204 ceph-aa240572-0159-11ef-bc93-c7b262605968-osd-11[57644]: 2024-04-23T10:42:59.645+0000 7fcbc4b63700 -1 osd.11 116 heartbeat_check: no reply from 172.21.15.119:6804 osd.1 since back 2024-04-23T10:42:09.278512+0000 front 2024-04-23T10:42:09.278494+0000 (oldest deadline 2024-04-23T10:42:33.378557+0000) 2024-04-23T10:42:59.828 INFO:journalctl@ceph.osd.11.smithi204.stdout:Apr 23 10:42:59 smithi204 ceph-aa240572-0159-11ef-bc93-c7b262605968-osd-11[57644]: 2024-04-23T10:42:59.645+0000 7fcbc4b63700 -1 osd.11 116 heartbeat_check: no reply from 172.21.15.204:6804 osd.2 since back 2024-04-23T10:42:18.679708+0000 front 2024-04-23T10:42:18.679577+0000 (oldest deadline 2024-04-23T10:42:41.579766+0000) 2024-04-23T10:42:59.829 INFO:journalctl@ceph.osd.8.smithi204.stdout:Apr 23 10:42:59 smithi204 ceph-aa240572-0159-11ef-bc93-c7b262605968-osd-8[51050]: 2024-04-23T10:42:59.416+0000 7f8d662cd700 -1 osd.8 116 heartbeat_check: no reply from 172.21.15.114:6806 osd.0 since back 2024-04-23T10:42:03.759281+0000 front 2024-04-23T10:42:03.759348+0000 (oldest deadline 2024-04-23T10:42:27.859455+0000) 2024-04-23T10:42:59.829 INFO:journalctl@ceph.osd.8.smithi204.stdout:Apr 23 10:42:59 smithi204 ceph-aa240572-0159-11ef-bc93-c7b262605968-osd-8[51050]: 2024-04-23T10:42:59.416+0000 7f8d662cd700 -1 osd.8 116 heartbeat_check: no reply from 172.21.15.119:6804 osd.1 since back 2024-04-23T10:42:09.560739+0000 front 2024-04-23T10:42:09.560541+0000 (oldest deadline 2024-04-23T10:42:37.761119+0000) 2024-04-23T10:42:59.829 INFO:journalctl@ceph.osd.8.smithi204.stdout:Apr 23 10:42:59 smithi204 ceph-aa240572-0159-11ef-bc93-c7b262605968-osd-8[51050]: 2024-04-23T10:42:59.416+0000 7f8d662cd700 -1 osd.8 116 heartbeat_check: no reply from 172.21.15.114:6814 osd.3 since back 2024-04-23T10:42:24.762812+0000 front 2024-04-23T10:42:24.762866+0000 (oldest deadline 2024-04-23T10:42:50.062939+0000) 2024-04-23T10:42:59.992 INFO:journalctl@ceph.osd.7.smithi119.stdout:Apr 23 10:42:59 smithi119 ceph-aa240572-0159-11ef-bc93-c7b262605968-osd-7[50859]: 2024-04-23T10:42:59.774+0000 7f015f194700 -1 osd.7 116 heartbeat_check: no reply from 172.21.15.119:6804 osd.1 since back 2024-04-23T10:42:09.664009+0000 front 2024-04-23T10:42:09.664029+0000 (oldest deadline 2024-04-23T10:42:38.464808+0000) 2024-04-23T10:42:59.992 INFO:journalctl@ceph.osd.7.smithi119.stdout:Apr 23 10:42:59 smithi119 ceph-aa240572-0159-11ef-bc93-c7b262605968-osd-7[50859]: 2024-04-23T10:42:59.774+0000 7f015f194700 -1 osd.7 116 heartbeat_check: no reply from 172.21.15.204:6804 osd.2 since back 2024-04-23T10:42:19.566017+0000 front 2024-04-23T10:42:19.566039+0000 (oldest deadline 2024-04-23T10:42:41.766809+0000) 2024-04-23T10:42:59.992 INFO:journalctl@ceph.osd.7.smithi119.stdout:Apr 23 10:42:59 smithi119 ceph-aa240572-0159-11ef-bc93-c7b262605968-osd-7[50859]: 2024-04-23T10:42:59.774+0000 7f015f194700 -1 osd.7 116 heartbeat_check: no reply from 172.21.15.114:6814 osd.3 since back 2024-04-23T10:42:26.468167+0000 front 2024-04-23T10:42:26.468303+0000 (oldest deadline 2024-04-23T10:42:51.768175+0000) 2024-04-23T10:42:59.992 INFO:journalctl@ceph.osd.10.smithi119.stdout:Apr 23 10:42:59 smithi119 ceph-aa240572-0159-11ef-bc93-c7b262605968-osd-10[57454]: 2024-04-23T10:42:59.685+0000 7fb1849bf700 -1 osd.10 116 heartbeat_check: no reply from 172.21.15.114:6806 osd.0 since back 2024-04-23T10:42:01.813174+0000 front 2024-04-23T10:42:01.813138+0000 (oldest deadline 2024-04-23T10:42:27.113300+0000) 2024-04-23T10:42:59.993 INFO:journalctl@ceph.osd.10.smithi119.stdout:Apr 23 10:42:59 smithi119 ceph-aa240572-0159-11ef-bc93-c7b262605968-osd-10[57454]: 2024-04-23T10:42:59.685+0000 7fb1849bf700 -1 osd.10 116 heartbeat_check: no reply from 172.21.15.119:6804 osd.1 since back 2024-04-23T10:42:10.614599+0000 front 2024-04-23T10:42:10.614622+0000 (oldest deadline 2024-04-23T10:42:34.114881+0000) 2024-04-23T10:42:59.993 INFO:journalctl@ceph.osd.10.smithi119.stdout:Apr 23 10:42:59 smithi119 ceph-aa240572-0159-11ef-bc93-c7b262605968-osd-10[57454]: 2024-04-23T10:42:59.685+0000 7fb1849bf700 -1 osd.10 116 heartbeat_check: no reply from 172.21.15.204:6804 osd.2 since back 2024-04-23T10:42:19.416108+0000 front 2024-04-23T10:42:19.416241+0000 (oldest deadline 2024-04-23T10:42:42.316071+0000) 2024-04-23T10:42:59.993 INFO:journalctl@ceph.osd.10.smithi119.stdout:Apr 23 10:42:59 smithi119 ceph-aa240572-0159-11ef-bc93-c7b262605968-osd-10[57454]: 2024-04-23T10:42:59.685+0000 7fb1849bf700 -1 osd.10 116 heartbeat_check: no reply from 172.21.15.114:6814 osd.3 since back 2024-04-23T10:42:25.217210+0000 front 2024-04-23T10:42:25.217268+0000 (oldest deadline 2024-04-23T10:42:49.917291+0000) 2024-04-23T10:43:00.754 INFO:journalctl@ceph.osd.9.smithi114.stdout:Apr 23 10:43:00 smithi114 ceph-aa240572-0159-11ef-bc93-c7b262605968-osd-9[62058]: 2024-04-23T10:43:00.255+0000 7feeb4bc7700 -1 osd.9 116 heartbeat_check: no reply from 172.21.15.114:6806 osd.0 since back 2024-04-23T10:42:04.136736+0000 front 2024-04-23T10:42:04.136764+0000 (oldest deadline 2024-04-23T10:42:28.836926+0000) 2024-04-23T10:43:00.754 INFO:journalctl@ceph.osd.9.smithi114.stdout:Apr 23 10:43:00 smithi114 ceph-aa240572-0159-11ef-bc93-c7b262605968-osd-9[62058]: 2024-04-23T10:43:00.255+0000 7feeb4bc7700 -1 osd.9 116 heartbeat_check: no reply from 172.21.15.119:6804 osd.1 since back 2024-04-23T10:42:08.837665+0000 front 2024-04-23T10:42:08.837503+0000 (oldest deadline 2024-04-23T10:42:34.737571+0000) 2024-04-23T10:43:00.754 INFO:journalctl@ceph.osd.9.smithi114.stdout:Apr 23 10:43:00 smithi114 ceph-aa240572-0159-11ef-bc93-c7b262605968-osd-9[62058]: 2024-04-23T10:43:00.255+0000 7feeb4bc7700 -1 osd.9 116 heartbeat_check: no reply from 172.21.15.204:6804 osd.2 since back 2024-04-23T10:42:18.139450+0000 front 2024-04-23T10:42:18.139388+0000 (oldest deadline 2024-04-23T10:42:43.439473+0000) 2024-04-23T10:43:00.754 INFO:journalctl@ceph.osd.9.smithi114.stdout:Apr 23 10:43:00 smithi114 ceph-aa240572-0159-11ef-bc93-c7b262605968-osd-9[62058]: 2024-04-23T10:43:00.255+0000 7feeb4bc7700 -1 osd.9 116 heartbeat_check: no reply from 172.21.15.114:6814 osd.3 since back 2024-04-23T10:42:28.641242+0000 front 2024-04-23T10:42:28.641354+0000 (oldest deadline 2024-04-23T10:42:52.741420+0000) 2024-04-23T10:43:00.754 INFO:journalctl@ceph.osd.9.smithi114.stdout:Apr 23 10:43:00 smithi114 ceph-aa240572-0159-11ef-bc93-c7b262605968-osd-9[62058]: 2024-04-23T10:43:00.255+0000 7feeb4bc7700 -1 osd.9 116 heartbeat_check: no reply from 172.21.15.119:6812 osd.4 since back 2024-04-23T10:42:32.742101+0000 front 2024-04-23T10:42:32.742087+0000 (oldest deadline 2024-04-23T10:42:58.642130+0000) 2024-04-23T10:43:00.828 INFO:journalctl@ceph.osd.8.smithi204.stdout:Apr 23 10:43:00 smithi204 ceph-aa240572-0159-11ef-bc93-c7b262605968-osd-8[51050]: 2024-04-23T10:43:00.460+0000 7f8d662cd700 -1 osd.8 116 heartbeat_check: no reply from 172.21.15.114:6806 osd.0 since back 2024-04-23T10:42:03.759281+0000 front 2024-04-23T10:42:03.759348+0000 (oldest deadline 2024-04-23T10:42:27.859455+0000) 2024-04-23T10:43:00.828 INFO:journalctl@ceph.osd.8.smithi204.stdout:Apr 23 10:43:00 smithi204 ceph-aa240572-0159-11ef-bc93-c7b262605968-osd-8[51050]: 2024-04-23T10:43:00.460+0000 7f8d662cd700 -1 osd.8 116 heartbeat_check: no reply from 172.21.15.119:6804 osd.1 since back 2024-04-23T10:42:09.560739+0000 front 2024-04-23T10:42:09.560541+0000 (oldest deadline 2024-04-23T10:42:37.761119+0000) 2024-04-23T10:43:00.828 INFO:journalctl@ceph.osd.8.smithi204.stdout:Apr 23 10:43:00 smithi204 ceph-aa240572-0159-11ef-bc93-c7b262605968-osd-8[51050]: 2024-04-23T10:43:00.460+0000 7f8d662cd700 -1 osd.8 116 heartbeat_check: no reply from 172.21.15.114:6814 osd.3 since back 2024-04-23T10:42:24.762812+0000 front 2024-04-23T10:42:24.762866+0000 (oldest deadline 2024-04-23T10:42:50.062939+0000) 2024-04-23T10:43:00.828 INFO:journalctl@ceph.osd.11.smithi204.stdout:Apr 23 10:43:00 smithi204 ceph-aa240572-0159-11ef-bc93-c7b262605968-osd-11[57644]: 2024-04-23T10:43:00.646+0000 7fcbc4b63700 -1 osd.11 116 heartbeat_check: no reply from 172.21.15.114:6806 osd.0 since back 2024-04-23T10:42:00.675680+0000 front 2024-04-23T10:42:00.675649+0000 (oldest deadline 2024-04-23T10:42:25.876359+0000) 2024-04-23T10:43:00.828 INFO:journalctl@ceph.osd.11.smithi204.stdout:Apr 23 10:43:00 smithi204 ceph-aa240572-0159-11ef-bc93-c7b262605968-osd-11[57644]: 2024-04-23T10:43:00.646+0000 7fcbc4b63700 -1 osd.11 116 heartbeat_check: no reply from 172.21.15.119:6804 osd.1 since back 2024-04-23T10:42:09.278512+0000 front 2024-04-23T10:42:09.278494+0000 (oldest deadline 2024-04-23T10:42:33.378557+0000) 2024-04-23T10:43:00.828 INFO:journalctl@ceph.osd.11.smithi204.stdout:Apr 23 10:43:00 smithi204 ceph-aa240572-0159-11ef-bc93-c7b262605968-osd-11[57644]: 2024-04-23T10:43:00.646+0000 7fcbc4b63700 -1 osd.11 116 heartbeat_check: no reply from 172.21.15.204:6804 osd.2 since back 2024-04-23T10:42:18.679708+0000 front 2024-04-23T10:42:18.679577+0000 (oldest deadline 2024-04-23T10:42:41.579766+0000) 2024-04-23T10:43:00.828 INFO:journalctl@ceph.osd.11.smithi204.stdout:Apr 23 10:43:00 smithi204 ceph-aa240572-0159-11ef-bc93-c7b262605968-osd-11[57644]: 2024-04-23T10:43:00.646+0000 7fcbc4b63700 -1 osd.11 116 heartbeat_check: no reply from 172.21.15.119:6812 osd.4 since back 2024-04-23T10:42:35.082088+0000 front 2024-04-23T10:42:35.082102+0000 (oldest deadline 2024-04-23T10:42:59.782202+0000) 2024-04-23T10:43:00.992 INFO:journalctl@ceph.osd.10.smithi119.stdout:Apr 23 10:43:00 smithi119 ceph-aa240572-0159-11ef-bc93-c7b262605968-osd-10[57454]: 2024-04-23T10:43:00.664+0000 7fb1849bf700 -1 osd.10 116 heartbeat_check: no reply from 172.21.15.114:6806 osd.0 since back 2024-04-23T10:42:01.813174+0000 front 2024-04-23T10:42:01.813138+0000 (oldest deadline 2024-04-23T10:42:27.113300+0000) 2024-04-23T10:43:00.992 INFO:journalctl@ceph.osd.10.smithi119.stdout:Apr 23 10:43:00 smithi119 ceph-aa240572-0159-11ef-bc93-c7b262605968-osd-10[57454]: 2024-04-23T10:43:00.664+0000 7fb1849bf700 -1 osd.10 116 heartbeat_check: no reply from 172.21.15.119:6804 osd.1 since back 2024-04-23T10:42:10.614599+0000 front 2024-04-23T10:42:10.614622+0000 (oldest deadline 2024-04-23T10:42:34.114881+0000) 2024-04-23T10:43:00.992 INFO:journalctl@ceph.osd.10.smithi119.stdout:Apr 23 10:43:00 smithi119 ceph-aa240572-0159-11ef-bc93-c7b262605968-osd-10[57454]: 2024-04-23T10:43:00.664+0000 7fb1849bf700 -1 osd.10 116 heartbeat_check: no reply from 172.21.15.204:6804 osd.2 since back 2024-04-23T10:42:19.416108+0000 front 2024-04-23T10:42:19.416241+0000 (oldest deadline 2024-04-23T10:42:42.316071+0000) 2024-04-23T10:43:00.992 INFO:journalctl@ceph.osd.10.smithi119.stdout:Apr 23 10:43:00 smithi119 ceph-aa240572-0159-11ef-bc93-c7b262605968-osd-10[57454]: 2024-04-23T10:43:00.664+0000 7fb1849bf700 -1 osd.10 116 heartbeat_check: no reply from 172.21.15.114:6814 osd.3 since back 2024-04-23T10:42:25.217210+0000 front 2024-04-23T10:42:25.217268+0000 (oldest deadline 2024-04-23T10:42:49.917291+0000) 2024-04-23T10:43:01.313 INFO:journalctl@ceph.osd.7.smithi119.stdout:Apr 23 10:43:01 smithi119 podman[69847]: 2024-04-23 10:43:01.053500397 +0000 UTC m=+5.495306721 container died 1891cc449f689e0a58efab3c3718bbf83ac511b3b65450de9b598b4d51a9b783 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:068a558e3af4932cf1c60dd321c0bdf500075eb52ea1bf455e0973ba114fbfd4, name=ceph-aa240572-0159-11ef-bc93-c7b262605968-osd-7, GIT_CLEAN=True, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240416, org.label-schema.name=CentOS Stream 8 Base Image, io.buildah.version=1.33.5, GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0, RELEASE=wip-lusov-quiescer-d69f13d, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True) 2024-04-23T10:43:01.627 INFO:journalctl@ceph.osd.7.smithi119.stdout:Apr 23 10:43:01 smithi119 podman[69847]: 2024-04-23 10:43:01.406320474 +0000 UTC m=+5.848126787 container cleanup 1891cc449f689e0a58efab3c3718bbf83ac511b3b65450de9b598b4d51a9b783 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:068a558e3af4932cf1c60dd321c0bdf500075eb52ea1bf455e0973ba114fbfd4, name=ceph-aa240572-0159-11ef-bc93-c7b262605968-osd-7, GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0, ceph=True, org.label-schema.name=CentOS Stream 8 Base Image, CEPH_POINT_RELEASE=, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, maintainer=Guillaume Abrioux , RELEASE=wip-lusov-quiescer-d69f13d, org.label-schema.build-date=20240416, org.label-schema.license=GPLv2, io.buildah.version=1.33.5, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_CLEAN=True) 2024-04-23T10:43:01.628 INFO:journalctl@ceph.osd.7.smithi119.stdout:Apr 23 10:43:01 smithi119 bash[69847]: ceph-aa240572-0159-11ef-bc93-c7b262605968-osd-7 2024-04-23T10:43:01.738 INFO:journalctl@ceph.osd.9.smithi114.stdout:Apr 23 10:43:01 smithi114 ceph-aa240572-0159-11ef-bc93-c7b262605968-osd-9[62058]: 2024-04-23T10:43:01.301+0000 7feeb4bc7700 -1 osd.9 116 heartbeat_check: no reply from 172.21.15.114:6806 osd.0 since back 2024-04-23T10:42:04.136736+0000 front 2024-04-23T10:42:04.136764+0000 (oldest deadline 2024-04-23T10:42:28.836926+0000) 2024-04-23T10:43:01.738 INFO:journalctl@ceph.osd.9.smithi114.stdout:Apr 23 10:43:01 smithi114 ceph-aa240572-0159-11ef-bc93-c7b262605968-osd-9[62058]: 2024-04-23T10:43:01.301+0000 7feeb4bc7700 -1 osd.9 116 heartbeat_check: no reply from 172.21.15.119:6804 osd.1 since back 2024-04-23T10:42:08.837665+0000 front 2024-04-23T10:42:08.837503+0000 (oldest deadline 2024-04-23T10:42:34.737571+0000) 2024-04-23T10:43:01.738 INFO:journalctl@ceph.osd.9.smithi114.stdout:Apr 23 10:43:01 smithi114 ceph-aa240572-0159-11ef-bc93-c7b262605968-osd-9[62058]: 2024-04-23T10:43:01.301+0000 7feeb4bc7700 -1 osd.9 116 heartbeat_check: no reply from 172.21.15.204:6804 osd.2 since back 2024-04-23T10:42:18.139450+0000 front 2024-04-23T10:42:18.139388+0000 (oldest deadline 2024-04-23T10:42:43.439473+0000) 2024-04-23T10:43:01.738 INFO:journalctl@ceph.osd.9.smithi114.stdout:Apr 23 10:43:01 smithi114 ceph-aa240572-0159-11ef-bc93-c7b262605968-osd-9[62058]: 2024-04-23T10:43:01.301+0000 7feeb4bc7700 -1 osd.9 116 heartbeat_check: no reply from 172.21.15.114:6814 osd.3 since back 2024-04-23T10:42:28.641242+0000 front 2024-04-23T10:42:28.641354+0000 (oldest deadline 2024-04-23T10:42:52.741420+0000) 2024-04-23T10:43:01.738 INFO:journalctl@ceph.osd.9.smithi114.stdout:Apr 23 10:43:01 smithi114 ceph-aa240572-0159-11ef-bc93-c7b262605968-osd-9[62058]: 2024-04-23T10:43:01.301+0000 7feeb4bc7700 -1 osd.9 116 heartbeat_check: no reply from 172.21.15.119:6812 osd.4 since back 2024-04-23T10:42:32.742101+0000 front 2024-04-23T10:42:32.742087+0000 (oldest deadline 2024-04-23T10:42:58.642130+0000) 2024-04-23T10:43:01.828 INFO:journalctl@ceph.osd.8.smithi204.stdout:Apr 23 10:43:01 smithi204 ceph-aa240572-0159-11ef-bc93-c7b262605968-osd-8[51050]: 2024-04-23T10:43:01.500+0000 7f8d662cd700 -1 osd.8 116 heartbeat_check: no reply from 172.21.15.114:6806 osd.0 since back 2024-04-23T10:42:03.759281+0000 front 2024-04-23T10:42:03.759348+0000 (oldest deadline 2024-04-23T10:42:27.859455+0000) 2024-04-23T10:43:01.828 INFO:journalctl@ceph.osd.8.smithi204.stdout:Apr 23 10:43:01 smithi204 ceph-aa240572-0159-11ef-bc93-c7b262605968-osd-8[51050]: 2024-04-23T10:43:01.500+0000 7f8d662cd700 -1 osd.8 116 heartbeat_check: no reply from 172.21.15.119:6804 osd.1 since back 2024-04-23T10:42:09.560739+0000 front 2024-04-23T10:42:09.560541+0000 (oldest deadline 2024-04-23T10:42:37.761119+0000) 2024-04-23T10:43:01.828 INFO:journalctl@ceph.osd.8.smithi204.stdout:Apr 23 10:43:01 smithi204 ceph-aa240572-0159-11ef-bc93-c7b262605968-osd-8[51050]: 2024-04-23T10:43:01.500+0000 7f8d662cd700 -1 osd.8 116 heartbeat_check: no reply from 172.21.15.114:6814 osd.3 since back 2024-04-23T10:42:24.762812+0000 front 2024-04-23T10:42:24.762866+0000 (oldest deadline 2024-04-23T10:42:50.062939+0000) 2024-04-23T10:43:01.828 INFO:journalctl@ceph.osd.11.smithi204.stdout:Apr 23 10:43:01 smithi204 ceph-aa240572-0159-11ef-bc93-c7b262605968-osd-11[57644]: 2024-04-23T10:43:01.681+0000 7fcbc4b63700 -1 osd.11 116 heartbeat_check: no reply from 172.21.15.114:6806 osd.0 since back 2024-04-23T10:42:00.675680+0000 front 2024-04-23T10:42:00.675649+0000 (oldest deadline 2024-04-23T10:42:25.876359+0000) 2024-04-23T10:43:01.828 INFO:journalctl@ceph.osd.11.smithi204.stdout:Apr 23 10:43:01 smithi204 ceph-aa240572-0159-11ef-bc93-c7b262605968-osd-11[57644]: 2024-04-23T10:43:01.681+0000 7fcbc4b63700 -1 osd.11 116 heartbeat_check: no reply from 172.21.15.119:6804 osd.1 since back 2024-04-23T10:42:09.278512+0000 front 2024-04-23T10:42:09.278494+0000 (oldest deadline 2024-04-23T10:42:33.378557+0000) 2024-04-23T10:43:01.829 INFO:journalctl@ceph.osd.11.smithi204.stdout:Apr 23 10:43:01 smithi204 ceph-aa240572-0159-11ef-bc93-c7b262605968-osd-11[57644]: 2024-04-23T10:43:01.681+0000 7fcbc4b63700 -1 osd.11 116 heartbeat_check: no reply from 172.21.15.204:6804 osd.2 since back 2024-04-23T10:42:18.679708+0000 front 2024-04-23T10:42:18.679577+0000 (oldest deadline 2024-04-23T10:42:41.579766+0000) 2024-04-23T10:43:01.829 INFO:journalctl@ceph.osd.11.smithi204.stdout:Apr 23 10:43:01 smithi204 ceph-aa240572-0159-11ef-bc93-c7b262605968-osd-11[57644]: 2024-04-23T10:43:01.681+0000 7fcbc4b63700 -1 osd.11 116 heartbeat_check: no reply from 172.21.15.119:6812 osd.4 since back 2024-04-23T10:42:35.082088+0000 front 2024-04-23T10:42:35.082102+0000 (oldest deadline 2024-04-23T10:42:59.782202+0000) 2024-04-23T10:43:01.992 INFO:journalctl@ceph.osd.7.smithi119.stdout:Apr 23 10:43:01 smithi119 podman[69866]: 2024-04-23 10:43:01.691058199 +0000 UTC m=+0.639208118 container remove 1891cc449f689e0a58efab3c3718bbf83ac511b3b65450de9b598b4d51a9b783 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:068a558e3af4932cf1c60dd321c0bdf500075eb52ea1bf455e0973ba114fbfd4, name=ceph-aa240572-0159-11ef-bc93-c7b262605968-osd-7, org.label-schema.license=GPLv2, GIT_CLEAN=True, RELEASE=wip-lusov-quiescer-d69f13d, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 8 Base Image, io.buildah.version=1.33.5, ceph=True, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=, org.label-schema.schema-version=1.0, org.label-schema.build-date=20240416, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67) 2024-04-23T10:43:01.992 INFO:journalctl@ceph.osd.10.smithi119.stdout:Apr 23 10:43:01 smithi119 ceph-aa240572-0159-11ef-bc93-c7b262605968-osd-10[57454]: 2024-04-23T10:43:01.628+0000 7fb1849bf700 -1 osd.10 116 heartbeat_check: no reply from 172.21.15.114:6806 osd.0 since back 2024-04-23T10:42:01.813174+0000 front 2024-04-23T10:42:01.813138+0000 (oldest deadline 2024-04-23T10:42:27.113300+0000) 2024-04-23T10:43:01.992 INFO:journalctl@ceph.osd.10.smithi119.stdout:Apr 23 10:43:01 smithi119 ceph-aa240572-0159-11ef-bc93-c7b262605968-osd-10[57454]: 2024-04-23T10:43:01.628+0000 7fb1849bf700 -1 osd.10 116 heartbeat_check: no reply from 172.21.15.119:6804 osd.1 since back 2024-04-23T10:42:10.614599+0000 front 2024-04-23T10:42:10.614622+0000 (oldest deadline 2024-04-23T10:42:34.114881+0000) 2024-04-23T10:43:01.993 INFO:journalctl@ceph.osd.10.smithi119.stdout:Apr 23 10:43:01 smithi119 ceph-aa240572-0159-11ef-bc93-c7b262605968-osd-10[57454]: 2024-04-23T10:43:01.628+0000 7fb1849bf700 -1 osd.10 116 heartbeat_check: no reply from 172.21.15.204:6804 osd.2 since back 2024-04-23T10:42:19.416108+0000 front 2024-04-23T10:42:19.416241+0000 (oldest deadline 2024-04-23T10:42:42.316071+0000) 2024-04-23T10:43:01.993 INFO:journalctl@ceph.osd.10.smithi119.stdout:Apr 23 10:43:01 smithi119 ceph-aa240572-0159-11ef-bc93-c7b262605968-osd-10[57454]: 2024-04-23T10:43:01.628+0000 7fb1849bf700 -1 osd.10 116 heartbeat_check: no reply from 172.21.15.114:6814 osd.3 since back 2024-04-23T10:42:25.217210+0000 front 2024-04-23T10:42:25.217268+0000 (oldest deadline 2024-04-23T10:42:49.917291+0000) 2024-04-23T10:43:01.993 INFO:journalctl@ceph.osd.10.smithi119.stdout:Apr 23 10:43:01 smithi119 ceph-aa240572-0159-11ef-bc93-c7b262605968-osd-10[57454]: 2024-04-23T10:43:01.628+0000 7fb1849bf700 -1 osd.10 116 heartbeat_check: no reply from 172.21.15.119:6812 osd.4 since back 2024-04-23T10:42:36.620525+0000 front 2024-04-23T10:42:36.620567+0000 (oldest deadline 2024-04-23T10:43:00.720710+0000) 2024-04-23T10:43:02.334 INFO:journalctl@ceph.osd.7.smithi119.stdout:Apr 23 10:43:02 smithi119 podman[69955]: 2024-04-23 10:43:01.921202918 +0000 UTC m=+0.018109074 image pull da4b004c09e820713483d039bc76fbd660be678f2ea3c03e059a6161b74dab76 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:068a558e3af4932cf1c60dd321c0bdf500075eb52ea1bf455e0973ba114fbfd4 2024-04-23T10:43:02.334 INFO:journalctl@ceph.osd.7.smithi119.stdout:Apr 23 10:43:02 smithi119 podman[69955]: 2024-04-23 10:43:02.033738025 +0000 UTC m=+0.130644181 container create 86f9aec7f467f2fc0a28e6177f9f752a93fb68c6258dca8491d9ba79c3111fac (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:068a558e3af4932cf1c60dd321c0bdf500075eb52ea1bf455e0973ba114fbfd4, name=ceph-aa240572-0159-11ef-bc93-c7b262605968-osd-7-deactivate, org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.schema-version=1.0, ceph=True, maintainer=Guillaume Abrioux , io.buildah.version=1.33.5, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=, org.label-schema.build-date=20240416, RELEASE=wip-lusov-quiescer-d69f13d, GIT_BRANCH=HEAD) 2024-04-23T10:43:02.334 INFO:journalctl@ceph.osd.7.smithi119.stdout:Apr 23 10:43:02 smithi119 podman[69955]: 2024-04-23 10:43:02.252834711 +0000 UTC m=+0.349740867 container init 86f9aec7f467f2fc0a28e6177f9f752a93fb68c6258dca8491d9ba79c3111fac (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:068a558e3af4932cf1c60dd321c0bdf500075eb52ea1bf455e0973ba114fbfd4, name=ceph-aa240572-0159-11ef-bc93-c7b262605968-osd-7-deactivate, org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 8 Base Image, io.buildah.version=1.33.5, GIT_CLEAN=True, org.label-schema.build-date=20240416, RELEASE=wip-lusov-quiescer-d69f13d, maintainer=Guillaume Abrioux , GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=, ceph=True, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD) 2024-04-23T10:43:02.335 INFO:journalctl@ceph.osd.7.smithi119.stdout:Apr 23 10:43:02 smithi119 podman[69955]: 2024-04-23 10:43:02.255980373 +0000 UTC m=+0.352886529 container start 86f9aec7f467f2fc0a28e6177f9f752a93fb68c6258dca8491d9ba79c3111fac (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:068a558e3af4932cf1c60dd321c0bdf500075eb52ea1bf455e0973ba114fbfd4, name=ceph-aa240572-0159-11ef-bc93-c7b262605968-osd-7-deactivate, org.label-schema.schema-version=1.0, org.label-schema.build-date=20240416, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, RELEASE=wip-lusov-quiescer-d69f13d, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 8 Base Image, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux , io.buildah.version=1.33.5, GIT_CLEAN=True) 2024-04-23T10:43:02.335 INFO:journalctl@ceph.osd.7.smithi119.stdout:Apr 23 10:43:02 smithi119 podman[69955]: 2024-04-23 10:43:02.33607158 +0000 UTC m=+0.432977739 container attach 86f9aec7f467f2fc0a28e6177f9f752a93fb68c6258dca8491d9ba79c3111fac (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:068a558e3af4932cf1c60dd321c0bdf500075eb52ea1bf455e0973ba114fbfd4, name=ceph-aa240572-0159-11ef-bc93-c7b262605968-osd-7-deactivate, ceph=True, org.label-schema.name=CentOS Stream 8 Base Image, GIT_BRANCH=HEAD, io.buildah.version=1.33.5, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, org.label-schema.build-date=20240416, RELEASE=wip-lusov-quiescer-d69f13d, maintainer=Guillaume Abrioux , GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2) 2024-04-23T10:43:02.586 INFO:journalctl@ceph.osd.7.smithi119.stdout:Apr 23 10:43:02 smithi119 podman[69955]: 2024-04-23 10:43:02.544494978 +0000 UTC m=+0.641401126 container died 86f9aec7f467f2fc0a28e6177f9f752a93fb68c6258dca8491d9ba79c3111fac (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:068a558e3af4932cf1c60dd321c0bdf500075eb52ea1bf455e0973ba114fbfd4, name=ceph-aa240572-0159-11ef-bc93-c7b262605968-osd-7-deactivate, org.label-schema.name=CentOS Stream 8 Base Image, GIT_BRANCH=HEAD, GIT_CLEAN=True, org.label-schema.build-date=20240416, RELEASE=wip-lusov-quiescer-d69f13d, org.label-schema.license=GPLv2, io.buildah.version=1.33.5, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, ceph=True, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux , GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67) 2024-04-23T10:43:02.720 INFO:journalctl@ceph.osd.8.smithi204.stdout:Apr 23 10:43:02 smithi204 ceph-aa240572-0159-11ef-bc93-c7b262605968-osd-8[51050]: 2024-04-23T10:43:02.455+0000 7f8d662cd700 -1 osd.8 116 heartbeat_check: no reply from 172.21.15.114:6806 osd.0 since back 2024-04-23T10:42:03.759281+0000 front 2024-04-23T10:42:03.759348+0000 (oldest deadline 2024-04-23T10:42:27.859455+0000) 2024-04-23T10:43:02.720 INFO:journalctl@ceph.osd.8.smithi204.stdout:Apr 23 10:43:02 smithi204 ceph-aa240572-0159-11ef-bc93-c7b262605968-osd-8[51050]: 2024-04-23T10:43:02.455+0000 7f8d662cd700 -1 osd.8 116 heartbeat_check: no reply from 172.21.15.119:6804 osd.1 since back 2024-04-23T10:42:09.560739+0000 front 2024-04-23T10:42:09.560541+0000 (oldest deadline 2024-04-23T10:42:37.761119+0000) 2024-04-23T10:43:02.720 INFO:journalctl@ceph.osd.8.smithi204.stdout:Apr 23 10:43:02 smithi204 ceph-aa240572-0159-11ef-bc93-c7b262605968-osd-8[51050]: 2024-04-23T10:43:02.455+0000 7f8d662cd700 -1 osd.8 116 heartbeat_check: no reply from 172.21.15.114:6814 osd.3 since back 2024-04-23T10:42:24.762812+0000 front 2024-04-23T10:42:24.762866+0000 (oldest deadline 2024-04-23T10:42:50.062939+0000) 2024-04-23T10:43:02.720 INFO:journalctl@ceph.osd.8.smithi204.stdout:Apr 23 10:43:02 smithi204 ceph-aa240572-0159-11ef-bc93-c7b262605968-osd-8[51050]: 2024-04-23T10:43:02.455+0000 7f8d662cd700 -1 osd.8 116 heartbeat_check: no reply from 172.21.15.119:6812 osd.4 since back 2024-04-23T10:42:36.464758+0000 front 2024-04-23T10:42:36.464792+0000 (oldest deadline 2024-04-23T10:43:01.764563+0000) 2024-04-23T10:43:02.754 INFO:journalctl@ceph.osd.9.smithi114.stdout:Apr 23 10:43:02 smithi114 ceph-aa240572-0159-11ef-bc93-c7b262605968-osd-9[62058]: 2024-04-23T10:43:02.336+0000 7feeb4bc7700 -1 osd.9 116 heartbeat_check: no reply from 172.21.15.114:6806 osd.0 since back 2024-04-23T10:42:04.136736+0000 front 2024-04-23T10:42:04.136764+0000 (oldest deadline 2024-04-23T10:42:28.836926+0000) 2024-04-23T10:43:02.754 INFO:journalctl@ceph.osd.9.smithi114.stdout:Apr 23 10:43:02 smithi114 ceph-aa240572-0159-11ef-bc93-c7b262605968-osd-9[62058]: 2024-04-23T10:43:02.336+0000 7feeb4bc7700 -1 osd.9 116 heartbeat_check: no reply from 172.21.15.119:6804 osd.1 since back 2024-04-23T10:42:08.837665+0000 front 2024-04-23T10:42:08.837503+0000 (oldest deadline 2024-04-23T10:42:34.737571+0000) 2024-04-23T10:43:02.754 INFO:journalctl@ceph.osd.9.smithi114.stdout:Apr 23 10:43:02 smithi114 ceph-aa240572-0159-11ef-bc93-c7b262605968-osd-9[62058]: 2024-04-23T10:43:02.336+0000 7feeb4bc7700 -1 osd.9 116 heartbeat_check: no reply from 172.21.15.204:6804 osd.2 since back 2024-04-23T10:42:18.139450+0000 front 2024-04-23T10:42:18.139388+0000 (oldest deadline 2024-04-23T10:42:43.439473+0000) 2024-04-23T10:43:02.754 INFO:journalctl@ceph.osd.9.smithi114.stdout:Apr 23 10:43:02 smithi114 ceph-aa240572-0159-11ef-bc93-c7b262605968-osd-9[62058]: 2024-04-23T10:43:02.336+0000 7feeb4bc7700 -1 osd.9 116 heartbeat_check: no reply from 172.21.15.114:6814 osd.3 since back 2024-04-23T10:42:28.641242+0000 front 2024-04-23T10:42:28.641354+0000 (oldest deadline 2024-04-23T10:42:52.741420+0000) 2024-04-23T10:43:02.755 INFO:journalctl@ceph.osd.9.smithi114.stdout:Apr 23 10:43:02 smithi114 ceph-aa240572-0159-11ef-bc93-c7b262605968-osd-9[62058]: 2024-04-23T10:43:02.336+0000 7feeb4bc7700 -1 osd.9 116 heartbeat_check: no reply from 172.21.15.119:6812 osd.4 since back 2024-04-23T10:42:32.742101+0000 front 2024-04-23T10:42:32.742087+0000 (oldest deadline 2024-04-23T10:42:58.642130+0000) 2024-04-23T10:43:02.992 INFO:journalctl@ceph.osd.10.smithi119.stdout:Apr 23 10:43:02 smithi119 ceph-aa240572-0159-11ef-bc93-c7b262605968-osd-10[57454]: 2024-04-23T10:43:02.586+0000 7fb1849bf700 -1 osd.10 116 heartbeat_check: no reply from 172.21.15.114:6806 osd.0 since back 2024-04-23T10:42:01.813174+0000 front 2024-04-23T10:42:01.813138+0000 (oldest deadline 2024-04-23T10:42:27.113300+0000) 2024-04-23T10:43:02.992 INFO:journalctl@ceph.osd.10.smithi119.stdout:Apr 23 10:43:02 smithi119 ceph-aa240572-0159-11ef-bc93-c7b262605968-osd-10[57454]: 2024-04-23T10:43:02.586+0000 7fb1849bf700 -1 osd.10 116 heartbeat_check: no reply from 172.21.15.119:6804 osd.1 since back 2024-04-23T10:42:10.614599+0000 front 2024-04-23T10:42:10.614622+0000 (oldest deadline 2024-04-23T10:42:34.114881+0000) 2024-04-23T10:43:02.992 INFO:journalctl@ceph.osd.10.smithi119.stdout:Apr 23 10:43:02 smithi119 ceph-aa240572-0159-11ef-bc93-c7b262605968-osd-10[57454]: 2024-04-23T10:43:02.586+0000 7fb1849bf700 -1 osd.10 116 heartbeat_check: no reply from 172.21.15.204:6804 osd.2 since back 2024-04-23T10:42:19.416108+0000 front 2024-04-23T10:42:19.416241+0000 (oldest deadline 2024-04-23T10:42:42.316071+0000) 2024-04-23T10:43:02.993 INFO:journalctl@ceph.osd.10.smithi119.stdout:Apr 23 10:43:02 smithi119 ceph-aa240572-0159-11ef-bc93-c7b262605968-osd-10[57454]: 2024-04-23T10:43:02.586+0000 7fb1849bf700 -1 osd.10 116 heartbeat_check: no reply from 172.21.15.114:6814 osd.3 since back 2024-04-23T10:42:25.217210+0000 front 2024-04-23T10:42:25.217268+0000 (oldest deadline 2024-04-23T10:42:49.917291+0000) 2024-04-23T10:43:02.993 INFO:journalctl@ceph.osd.10.smithi119.stdout:Apr 23 10:43:02 smithi119 ceph-aa240572-0159-11ef-bc93-c7b262605968-osd-10[57454]: 2024-04-23T10:43:02.586+0000 7fb1849bf700 -1 osd.10 116 heartbeat_check: no reply from 172.21.15.119:6812 osd.4 since back 2024-04-23T10:42:36.620525+0000 front 2024-04-23T10:42:36.620567+0000 (oldest deadline 2024-04-23T10:43:00.720710+0000) 2024-04-23T10:43:03.078 INFO:journalctl@ceph.osd.11.smithi204.stdout:Apr 23 10:43:02 smithi204 ceph-aa240572-0159-11ef-bc93-c7b262605968-osd-11[57644]: 2024-04-23T10:43:02.720+0000 7fcbc4b63700 -1 osd.11 116 heartbeat_check: no reply from 172.21.15.114:6806 osd.0 since back 2024-04-23T10:42:00.675680+0000 front 2024-04-23T10:42:00.675649+0000 (oldest deadline 2024-04-23T10:42:25.876359+0000) 2024-04-23T10:43:03.078 INFO:journalctl@ceph.osd.11.smithi204.stdout:Apr 23 10:43:02 smithi204 ceph-aa240572-0159-11ef-bc93-c7b262605968-osd-11[57644]: 2024-04-23T10:43:02.720+0000 7fcbc4b63700 -1 osd.11 116 heartbeat_check: no reply from 172.21.15.119:6804 osd.1 since back 2024-04-23T10:42:09.278512+0000 front 2024-04-23T10:42:09.278494+0000 (oldest deadline 2024-04-23T10:42:33.378557+0000) 2024-04-23T10:43:03.078 INFO:journalctl@ceph.osd.11.smithi204.stdout:Apr 23 10:43:02 smithi204 ceph-aa240572-0159-11ef-bc93-c7b262605968-osd-11[57644]: 2024-04-23T10:43:02.720+0000 7fcbc4b63700 -1 osd.11 116 heartbeat_check: no reply from 172.21.15.204:6804 osd.2 since back 2024-04-23T10:42:18.679708+0000 front 2024-04-23T10:42:18.679577+0000 (oldest deadline 2024-04-23T10:42:41.579766+0000) 2024-04-23T10:43:03.078 INFO:journalctl@ceph.osd.11.smithi204.stdout:Apr 23 10:43:02 smithi204 ceph-aa240572-0159-11ef-bc93-c7b262605968-osd-11[57644]: 2024-04-23T10:43:02.720+0000 7fcbc4b63700 -1 osd.11 116 heartbeat_check: no reply from 172.21.15.119:6812 osd.4 since back 2024-04-23T10:42:35.082088+0000 front 2024-04-23T10:42:35.082102+0000 (oldest deadline 2024-04-23T10:42:59.782202+0000) 2024-04-23T10:43:03.104 DEBUG:teuthology.orchestra.run.smithi119:> sudo pkill -f 'journalctl -f -n 0 -u ceph-aa240572-0159-11ef-bc93-c7b262605968@osd.7.service' 2024-04-23T10:43:03.142 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-04-23T10:43:03.143 INFO:tasks.cephadm.osd.7:Stopped osd.7 2024-04-23T10:43:03.143 INFO:tasks.cephadm.osd.8:Stopping osd.8... 2024-04-23T10:43:03.143 DEBUG:teuthology.orchestra.run.smithi204:> sudo systemctl stop ceph-aa240572-0159-11ef-bc93-c7b262605968@osd.8 2024-04-23T10:43:03.445 INFO:journalctl@ceph.osd.8.smithi204.stdout:Apr 23 10:43:03 smithi204 systemd[1]: Stopping Ceph osd.8 for aa240572-0159-11ef-bc93-c7b262605968... 2024-04-23T10:43:03.735 INFO:journalctl@ceph.osd.8.smithi204.stdout:Apr 23 10:43:03 smithi204 ceph-aa240572-0159-11ef-bc93-c7b262605968-osd-8[51050]: 2024-04-23T10:43:03.445+0000 7f8d6dce1700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.8 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-04-23T10:43:03.735 INFO:journalctl@ceph.osd.8.smithi204.stdout:Apr 23 10:43:03 smithi204 ceph-aa240572-0159-11ef-bc93-c7b262605968-osd-8[51050]: 2024-04-23T10:43:03.445+0000 7f8d6dce1700 -1 osd.8 116 *** Got signal Terminated *** 2024-04-23T10:43:03.735 INFO:journalctl@ceph.osd.8.smithi204.stdout:Apr 23 10:43:03 smithi204 ceph-aa240572-0159-11ef-bc93-c7b262605968-osd-8[51050]: 2024-04-23T10:43:03.445+0000 7f8d6dce1700 -1 osd.8 116 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-04-23T10:43:03.735 INFO:journalctl@ceph.osd.8.smithi204.stdout:Apr 23 10:43:03 smithi204 ceph-aa240572-0159-11ef-bc93-c7b262605968-osd-8[51050]: 2024-04-23T10:43:03.482+0000 7f8d662cd700 -1 osd.8 116 heartbeat_check: no reply from 172.21.15.114:6806 osd.0 since back 2024-04-23T10:42:03.759281+0000 front 2024-04-23T10:42:03.759348+0000 (oldest deadline 2024-04-23T10:42:27.859455+0000) 2024-04-23T10:43:03.735 INFO:journalctl@ceph.osd.8.smithi204.stdout:Apr 23 10:43:03 smithi204 ceph-aa240572-0159-11ef-bc93-c7b262605968-osd-8[51050]: 2024-04-23T10:43:03.482+0000 7f8d662cd700 -1 osd.8 116 heartbeat_check: no reply from 172.21.15.119:6804 osd.1 since back 2024-04-23T10:42:09.560739+0000 front 2024-04-23T10:42:09.560541+0000 (oldest deadline 2024-04-23T10:42:37.761119+0000) 2024-04-23T10:43:03.736 INFO:journalctl@ceph.osd.8.smithi204.stdout:Apr 23 10:43:03 smithi204 ceph-aa240572-0159-11ef-bc93-c7b262605968-osd-8[51050]: 2024-04-23T10:43:03.482+0000 7f8d662cd700 -1 osd.8 116 heartbeat_check: no reply from 172.21.15.114:6814 osd.3 since back 2024-04-23T10:42:24.762812+0000 front 2024-04-23T10:42:24.762866+0000 (oldest deadline 2024-04-23T10:42:50.062939+0000) 2024-04-23T10:43:03.736 INFO:journalctl@ceph.osd.8.smithi204.stdout:Apr 23 10:43:03 smithi204 ceph-aa240572-0159-11ef-bc93-c7b262605968-osd-8[51050]: 2024-04-23T10:43:03.482+0000 7f8d662cd700 -1 osd.8 116 heartbeat_check: no reply from 172.21.15.119:6812 osd.4 since back 2024-04-23T10:42:36.464758+0000 front 2024-04-23T10:42:36.464792+0000 (oldest deadline 2024-04-23T10:43:01.764563+0000) 2024-04-23T10:43:03.754 INFO:journalctl@ceph.osd.9.smithi114.stdout:Apr 23 10:43:03 smithi114 ceph-aa240572-0159-11ef-bc93-c7b262605968-osd-9[62058]: 2024-04-23T10:43:03.345+0000 7feeb4bc7700 -1 osd.9 116 heartbeat_check: no reply from 172.21.15.114:6806 osd.0 since back 2024-04-23T10:42:04.136736+0000 front 2024-04-23T10:42:04.136764+0000 (oldest deadline 2024-04-23T10:42:28.836926+0000) 2024-04-23T10:43:03.754 INFO:journalctl@ceph.osd.9.smithi114.stdout:Apr 23 10:43:03 smithi114 ceph-aa240572-0159-11ef-bc93-c7b262605968-osd-9[62058]: 2024-04-23T10:43:03.345+0000 7feeb4bc7700 -1 osd.9 116 heartbeat_check: no reply from 172.21.15.119:6804 osd.1 since back 2024-04-23T10:42:08.837665+0000 front 2024-04-23T10:42:08.837503+0000 (oldest deadline 2024-04-23T10:42:34.737571+0000) 2024-04-23T10:43:03.754 INFO:journalctl@ceph.osd.9.smithi114.stdout:Apr 23 10:43:03 smithi114 ceph-aa240572-0159-11ef-bc93-c7b262605968-osd-9[62058]: 2024-04-23T10:43:03.345+0000 7feeb4bc7700 -1 osd.9 116 heartbeat_check: no reply from 172.21.15.204:6804 osd.2 since back 2024-04-23T10:42:18.139450+0000 front 2024-04-23T10:42:18.139388+0000 (oldest deadline 2024-04-23T10:42:43.439473+0000) 2024-04-23T10:43:03.754 INFO:journalctl@ceph.osd.9.smithi114.stdout:Apr 23 10:43:03 smithi114 ceph-aa240572-0159-11ef-bc93-c7b262605968-osd-9[62058]: 2024-04-23T10:43:03.345+0000 7feeb4bc7700 -1 osd.9 116 heartbeat_check: no reply from 172.21.15.114:6814 osd.3 since back 2024-04-23T10:42:28.641242+0000 front 2024-04-23T10:42:28.641354+0000 (oldest deadline 2024-04-23T10:42:52.741420+0000) 2024-04-23T10:43:03.755 INFO:journalctl@ceph.osd.9.smithi114.stdout:Apr 23 10:43:03 smithi114 ceph-aa240572-0159-11ef-bc93-c7b262605968-osd-9[62058]: 2024-04-23T10:43:03.345+0000 7feeb4bc7700 -1 osd.9 116 heartbeat_check: no reply from 172.21.15.119:6812 osd.4 since back 2024-04-23T10:42:32.742101+0000 front 2024-04-23T10:42:32.742087+0000 (oldest deadline 2024-04-23T10:42:58.642130+0000) 2024-04-23T10:43:03.992 INFO:journalctl@ceph.osd.10.smithi119.stdout:Apr 23 10:43:03 smithi119 ceph-aa240572-0159-11ef-bc93-c7b262605968-osd-10[57454]: 2024-04-23T10:43:03.609+0000 7fb1849bf700 -1 osd.10 116 heartbeat_check: no reply from 172.21.15.114:6806 osd.0 since back 2024-04-23T10:42:01.813174+0000 front 2024-04-23T10:42:01.813138+0000 (oldest deadline 2024-04-23T10:42:27.113300+0000) 2024-04-23T10:43:03.992 INFO:journalctl@ceph.osd.10.smithi119.stdout:Apr 23 10:43:03 smithi119 ceph-aa240572-0159-11ef-bc93-c7b262605968-osd-10[57454]: 2024-04-23T10:43:03.609+0000 7fb1849bf700 -1 osd.10 116 heartbeat_check: no reply from 172.21.15.119:6804 osd.1 since back 2024-04-23T10:42:10.614599+0000 front 2024-04-23T10:42:10.614622+0000 (oldest deadline 2024-04-23T10:42:34.114881+0000) 2024-04-23T10:43:03.992 INFO:journalctl@ceph.osd.10.smithi119.stdout:Apr 23 10:43:03 smithi119 ceph-aa240572-0159-11ef-bc93-c7b262605968-osd-10[57454]: 2024-04-23T10:43:03.609+0000 7fb1849bf700 -1 osd.10 116 heartbeat_check: no reply from 172.21.15.204:6804 osd.2 since back 2024-04-23T10:42:19.416108+0000 front 2024-04-23T10:42:19.416241+0000 (oldest deadline 2024-04-23T10:42:42.316071+0000) 2024-04-23T10:43:03.993 INFO:journalctl@ceph.osd.10.smithi119.stdout:Apr 23 10:43:03 smithi119 ceph-aa240572-0159-11ef-bc93-c7b262605968-osd-10[57454]: 2024-04-23T10:43:03.609+0000 7fb1849bf700 -1 osd.10 116 heartbeat_check: no reply from 172.21.15.114:6814 osd.3 since back 2024-04-23T10:42:25.217210+0000 front 2024-04-23T10:42:25.217268+0000 (oldest deadline 2024-04-23T10:42:49.917291+0000) 2024-04-23T10:43:03.993 INFO:journalctl@ceph.osd.10.smithi119.stdout:Apr 23 10:43:03 smithi119 ceph-aa240572-0159-11ef-bc93-c7b262605968-osd-10[57454]: 2024-04-23T10:43:03.609+0000 7fb1849bf700 -1 osd.10 116 heartbeat_check: no reply from 172.21.15.119:6812 osd.4 since back 2024-04-23T10:42:36.620525+0000 front 2024-04-23T10:42:36.620567+0000 (oldest deadline 2024-04-23T10:43:00.720710+0000) 2024-04-23T10:43:04.077 INFO:journalctl@ceph.osd.11.smithi204.stdout:Apr 23 10:43:03 smithi204 ceph-aa240572-0159-11ef-bc93-c7b262605968-osd-11[57644]: 2024-04-23T10:43:03.735+0000 7fcbc4b63700 -1 osd.11 116 heartbeat_check: no reply from 172.21.15.114:6806 osd.0 since back 2024-04-23T10:42:00.675680+0000 front 2024-04-23T10:42:00.675649+0000 (oldest deadline 2024-04-23T10:42:25.876359+0000) 2024-04-23T10:43:04.078 INFO:journalctl@ceph.osd.11.smithi204.stdout:Apr 23 10:43:03 smithi204 ceph-aa240572-0159-11ef-bc93-c7b262605968-osd-11[57644]: 2024-04-23T10:43:03.735+0000 7fcbc4b63700 -1 osd.11 116 heartbeat_check: no reply from 172.21.15.119:6804 osd.1 since back 2024-04-23T10:42:09.278512+0000 front 2024-04-23T10:42:09.278494+0000 (oldest deadline 2024-04-23T10:42:33.378557+0000) 2024-04-23T10:43:04.078 INFO:journalctl@ceph.osd.11.smithi204.stdout:Apr 23 10:43:03 smithi204 ceph-aa240572-0159-11ef-bc93-c7b262605968-osd-11[57644]: 2024-04-23T10:43:03.735+0000 7fcbc4b63700 -1 osd.11 116 heartbeat_check: no reply from 172.21.15.204:6804 osd.2 since back 2024-04-23T10:42:18.679708+0000 front 2024-04-23T10:42:18.679577+0000 (oldest deadline 2024-04-23T10:42:41.579766+0000) 2024-04-23T10:43:04.078 INFO:journalctl@ceph.osd.11.smithi204.stdout:Apr 23 10:43:03 smithi204 ceph-aa240572-0159-11ef-bc93-c7b262605968-osd-11[57644]: 2024-04-23T10:43:03.735+0000 7fcbc4b63700 -1 osd.11 116 heartbeat_check: no reply from 172.21.15.119:6812 osd.4 since back 2024-04-23T10:42:35.082088+0000 front 2024-04-23T10:42:35.082102+0000 (oldest deadline 2024-04-23T10:42:59.782202+0000) 2024-04-23T10:43:04.754 INFO:journalctl@ceph.osd.9.smithi114.stdout:Apr 23 10:43:04 smithi114 ceph-aa240572-0159-11ef-bc93-c7b262605968-osd-9[62058]: 2024-04-23T10:43:04.373+0000 7feeb4bc7700 -1 osd.9 116 heartbeat_check: no reply from 172.21.15.114:6806 osd.0 since back 2024-04-23T10:42:04.136736+0000 front 2024-04-23T10:42:04.136764+0000 (oldest deadline 2024-04-23T10:42:28.836926+0000) 2024-04-23T10:43:04.754 INFO:journalctl@ceph.osd.9.smithi114.stdout:Apr 23 10:43:04 smithi114 ceph-aa240572-0159-11ef-bc93-c7b262605968-osd-9[62058]: 2024-04-23T10:43:04.373+0000 7feeb4bc7700 -1 osd.9 116 heartbeat_check: no reply from 172.21.15.119:6804 osd.1 since back 2024-04-23T10:42:08.837665+0000 front 2024-04-23T10:42:08.837503+0000 (oldest deadline 2024-04-23T10:42:34.737571+0000) 2024-04-23T10:43:04.755 INFO:journalctl@ceph.osd.9.smithi114.stdout:Apr 23 10:43:04 smithi114 ceph-aa240572-0159-11ef-bc93-c7b262605968-osd-9[62058]: 2024-04-23T10:43:04.373+0000 7feeb4bc7700 -1 osd.9 116 heartbeat_check: no reply from 172.21.15.204:6804 osd.2 since back 2024-04-23T10:42:18.139450+0000 front 2024-04-23T10:42:18.139388+0000 (oldest deadline 2024-04-23T10:42:43.439473+0000) 2024-04-23T10:43:04.755 INFO:journalctl@ceph.osd.9.smithi114.stdout:Apr 23 10:43:04 smithi114 ceph-aa240572-0159-11ef-bc93-c7b262605968-osd-9[62058]: 2024-04-23T10:43:04.373+0000 7feeb4bc7700 -1 osd.9 116 heartbeat_check: no reply from 172.21.15.114:6814 osd.3 since back 2024-04-23T10:42:28.641242+0000 front 2024-04-23T10:42:28.641354+0000 (oldest deadline 2024-04-23T10:42:52.741420+0000) 2024-04-23T10:43:04.755 INFO:journalctl@ceph.osd.9.smithi114.stdout:Apr 23 10:43:04 smithi114 ceph-aa240572-0159-11ef-bc93-c7b262605968-osd-9[62058]: 2024-04-23T10:43:04.373+0000 7feeb4bc7700 -1 osd.9 116 heartbeat_check: no reply from 172.21.15.119:6812 osd.4 since back 2024-04-23T10:42:32.742101+0000 front 2024-04-23T10:42:32.742087+0000 (oldest deadline 2024-04-23T10:42:58.642130+0000) 2024-04-23T10:43:04.765 INFO:journalctl@ceph.osd.8.smithi204.stdout:Apr 23 10:43:04 smithi204 ceph-aa240572-0159-11ef-bc93-c7b262605968-osd-8[51050]: 2024-04-23T10:43:04.488+0000 7f8d662cd700 -1 osd.8 116 heartbeat_check: no reply from 172.21.15.114:6806 osd.0 since back 2024-04-23T10:42:03.759281+0000 front 2024-04-23T10:42:03.759348+0000 (oldest deadline 2024-04-23T10:42:27.859455+0000) 2024-04-23T10:43:04.765 INFO:journalctl@ceph.osd.8.smithi204.stdout:Apr 23 10:43:04 smithi204 ceph-aa240572-0159-11ef-bc93-c7b262605968-osd-8[51050]: 2024-04-23T10:43:04.488+0000 7f8d662cd700 -1 osd.8 116 heartbeat_check: no reply from 172.21.15.119:6804 osd.1 since back 2024-04-23T10:42:09.560739+0000 front 2024-04-23T10:42:09.560541+0000 (oldest deadline 2024-04-23T10:42:37.761119+0000) 2024-04-23T10:43:04.766 INFO:journalctl@ceph.osd.8.smithi204.stdout:Apr 23 10:43:04 smithi204 ceph-aa240572-0159-11ef-bc93-c7b262605968-osd-8[51050]: 2024-04-23T10:43:04.488+0000 7f8d662cd700 -1 osd.8 116 heartbeat_check: no reply from 172.21.15.114:6814 osd.3 since back 2024-04-23T10:42:24.762812+0000 front 2024-04-23T10:42:24.762866+0000 (oldest deadline 2024-04-23T10:42:50.062939+0000) 2024-04-23T10:43:04.766 INFO:journalctl@ceph.osd.8.smithi204.stdout:Apr 23 10:43:04 smithi204 ceph-aa240572-0159-11ef-bc93-c7b262605968-osd-8[51050]: 2024-04-23T10:43:04.488+0000 7f8d662cd700 -1 osd.8 116 heartbeat_check: no reply from 172.21.15.119:6812 osd.4 since back 2024-04-23T10:42:36.464758+0000 front 2024-04-23T10:42:36.464792+0000 (oldest deadline 2024-04-23T10:43:01.764563+0000) 2024-04-23T10:43:04.992 INFO:journalctl@ceph.osd.10.smithi119.stdout:Apr 23 10:43:04 smithi119 ceph-aa240572-0159-11ef-bc93-c7b262605968-osd-10[57454]: 2024-04-23T10:43:04.624+0000 7fb1849bf700 -1 osd.10 116 heartbeat_check: no reply from 172.21.15.114:6806 osd.0 since back 2024-04-23T10:42:01.813174+0000 front 2024-04-23T10:42:01.813138+0000 (oldest deadline 2024-04-23T10:42:27.113300+0000) 2024-04-23T10:43:04.992 INFO:journalctl@ceph.osd.10.smithi119.stdout:Apr 23 10:43:04 smithi119 ceph-aa240572-0159-11ef-bc93-c7b262605968-osd-10[57454]: 2024-04-23T10:43:04.624+0000 7fb1849bf700 -1 osd.10 116 heartbeat_check: no reply from 172.21.15.119:6804 osd.1 since back 2024-04-23T10:42:10.614599+0000 front 2024-04-23T10:42:10.614622+0000 (oldest deadline 2024-04-23T10:42:34.114881+0000) 2024-04-23T10:43:04.992 INFO:journalctl@ceph.osd.10.smithi119.stdout:Apr 23 10:43:04 smithi119 ceph-aa240572-0159-11ef-bc93-c7b262605968-osd-10[57454]: 2024-04-23T10:43:04.624+0000 7fb1849bf700 -1 osd.10 116 heartbeat_check: no reply from 172.21.15.204:6804 osd.2 since back 2024-04-23T10:42:19.416108+0000 front 2024-04-23T10:42:19.416241+0000 (oldest deadline 2024-04-23T10:42:42.316071+0000) 2024-04-23T10:43:04.992 INFO:journalctl@ceph.osd.10.smithi119.stdout:Apr 23 10:43:04 smithi119 ceph-aa240572-0159-11ef-bc93-c7b262605968-osd-10[57454]: 2024-04-23T10:43:04.624+0000 7fb1849bf700 -1 osd.10 116 heartbeat_check: no reply from 172.21.15.114:6814 osd.3 since back 2024-04-23T10:42:25.217210+0000 front 2024-04-23T10:42:25.217268+0000 (oldest deadline 2024-04-23T10:42:49.917291+0000) 2024-04-23T10:43:04.993 INFO:journalctl@ceph.osd.10.smithi119.stdout:Apr 23 10:43:04 smithi119 ceph-aa240572-0159-11ef-bc93-c7b262605968-osd-10[57454]: 2024-04-23T10:43:04.624+0000 7fb1849bf700 -1 osd.10 116 heartbeat_check: no reply from 172.21.15.119:6812 osd.4 since back 2024-04-23T10:42:36.620525+0000 front 2024-04-23T10:42:36.620567+0000 (oldest deadline 2024-04-23T10:43:00.720710+0000) 2024-04-23T10:43:05.077 INFO:journalctl@ceph.osd.11.smithi204.stdout:Apr 23 10:43:04 smithi204 ceph-aa240572-0159-11ef-bc93-c7b262605968-osd-11[57644]: 2024-04-23T10:43:04.766+0000 7fcbc4b63700 -1 osd.11 116 heartbeat_check: no reply from 172.21.15.114:6806 osd.0 since back 2024-04-23T10:42:00.675680+0000 front 2024-04-23T10:42:00.675649+0000 (oldest deadline 2024-04-23T10:42:25.876359+0000) 2024-04-23T10:43:05.078 INFO:journalctl@ceph.osd.11.smithi204.stdout:Apr 23 10:43:04 smithi204 ceph-aa240572-0159-11ef-bc93-c7b262605968-osd-11[57644]: 2024-04-23T10:43:04.766+0000 7fcbc4b63700 -1 osd.11 116 heartbeat_check: no reply from 172.21.15.119:6804 osd.1 since back 2024-04-23T10:42:09.278512+0000 front 2024-04-23T10:42:09.278494+0000 (oldest deadline 2024-04-23T10:42:33.378557+0000) 2024-04-23T10:43:05.078 INFO:journalctl@ceph.osd.11.smithi204.stdout:Apr 23 10:43:04 smithi204 ceph-aa240572-0159-11ef-bc93-c7b262605968-osd-11[57644]: 2024-04-23T10:43:04.766+0000 7fcbc4b63700 -1 osd.11 116 heartbeat_check: no reply from 172.21.15.204:6804 osd.2 since back 2024-04-23T10:42:18.679708+0000 front 2024-04-23T10:42:18.679577+0000 (oldest deadline 2024-04-23T10:42:41.579766+0000) 2024-04-23T10:43:05.078 INFO:journalctl@ceph.osd.11.smithi204.stdout:Apr 23 10:43:04 smithi204 ceph-aa240572-0159-11ef-bc93-c7b262605968-osd-11[57644]: 2024-04-23T10:43:04.766+0000 7fcbc4b63700 -1 osd.11 116 heartbeat_check: no reply from 172.21.15.119:6812 osd.4 since back 2024-04-23T10:42:35.082088+0000 front 2024-04-23T10:42:35.082102+0000 (oldest deadline 2024-04-23T10:42:59.782202+0000) 2024-04-23T10:43:05.754 INFO:journalctl@ceph.osd.9.smithi114.stdout:Apr 23 10:43:05 smithi114 ceph-aa240572-0159-11ef-bc93-c7b262605968-osd-9[62058]: 2024-04-23T10:43:05.381+0000 7feeb4bc7700 -1 osd.9 116 heartbeat_check: no reply from 172.21.15.114:6806 osd.0 since back 2024-04-23T10:42:04.136736+0000 front 2024-04-23T10:42:04.136764+0000 (oldest deadline 2024-04-23T10:42:28.836926+0000) 2024-04-23T10:43:05.754 INFO:journalctl@ceph.osd.9.smithi114.stdout:Apr 23 10:43:05 smithi114 ceph-aa240572-0159-11ef-bc93-c7b262605968-osd-9[62058]: 2024-04-23T10:43:05.381+0000 7feeb4bc7700 -1 osd.9 116 heartbeat_check: no reply from 172.21.15.119:6804 osd.1 since back 2024-04-23T10:42:08.837665+0000 front 2024-04-23T10:42:08.837503+0000 (oldest deadline 2024-04-23T10:42:34.737571+0000) 2024-04-23T10:43:05.754 INFO:journalctl@ceph.osd.9.smithi114.stdout:Apr 23 10:43:05 smithi114 ceph-aa240572-0159-11ef-bc93-c7b262605968-osd-9[62058]: 2024-04-23T10:43:05.381+0000 7feeb4bc7700 -1 osd.9 116 heartbeat_check: no reply from 172.21.15.204:6804 osd.2 since back 2024-04-23T10:42:18.139450+0000 front 2024-04-23T10:42:18.139388+0000 (oldest deadline 2024-04-23T10:42:43.439473+0000) 2024-04-23T10:43:05.755 INFO:journalctl@ceph.osd.9.smithi114.stdout:Apr 23 10:43:05 smithi114 ceph-aa240572-0159-11ef-bc93-c7b262605968-osd-9[62058]: 2024-04-23T10:43:05.381+0000 7feeb4bc7700 -1 osd.9 116 heartbeat_check: no reply from 172.21.15.114:6814 osd.3 since back 2024-04-23T10:42:28.641242+0000 front 2024-04-23T10:42:28.641354+0000 (oldest deadline 2024-04-23T10:42:52.741420+0000) 2024-04-23T10:43:05.755 INFO:journalctl@ceph.osd.9.smithi114.stdout:Apr 23 10:43:05 smithi114 ceph-aa240572-0159-11ef-bc93-c7b262605968-osd-9[62058]: 2024-04-23T10:43:05.381+0000 7feeb4bc7700 -1 osd.9 116 heartbeat_check: no reply from 172.21.15.119:6812 osd.4 since back 2024-04-23T10:42:32.742101+0000 front 2024-04-23T10:42:32.742087+0000 (oldest deadline 2024-04-23T10:42:58.642130+0000) 2024-04-23T10:43:05.828 INFO:journalctl@ceph.osd.11.smithi204.stdout:Apr 23 10:43:05 smithi204 ceph-aa240572-0159-11ef-bc93-c7b262605968-osd-11[57644]: 2024-04-23T10:43:05.757+0000 7fcbc4b63700 -1 osd.11 116 heartbeat_check: no reply from 172.21.15.114:6806 osd.0 since back 2024-04-23T10:42:00.675680+0000 front 2024-04-23T10:42:00.675649+0000 (oldest deadline 2024-04-23T10:42:25.876359+0000) 2024-04-23T10:43:05.828 INFO:journalctl@ceph.osd.11.smithi204.stdout:Apr 23 10:43:05 smithi204 ceph-aa240572-0159-11ef-bc93-c7b262605968-osd-11[57644]: 2024-04-23T10:43:05.757+0000 7fcbc4b63700 -1 osd.11 116 heartbeat_check: no reply from 172.21.15.119:6804 osd.1 since back 2024-04-23T10:42:09.278512+0000 front 2024-04-23T10:42:09.278494+0000 (oldest deadline 2024-04-23T10:42:33.378557+0000) 2024-04-23T10:43:05.828 INFO:journalctl@ceph.osd.11.smithi204.stdout:Apr 23 10:43:05 smithi204 ceph-aa240572-0159-11ef-bc93-c7b262605968-osd-11[57644]: 2024-04-23T10:43:05.757+0000 7fcbc4b63700 -1 osd.11 116 heartbeat_check: no reply from 172.21.15.204:6804 osd.2 since back 2024-04-23T10:42:18.679708+0000 front 2024-04-23T10:42:18.679577+0000 (oldest deadline 2024-04-23T10:42:41.579766+0000) 2024-04-23T10:43:05.828 INFO:journalctl@ceph.osd.11.smithi204.stdout:Apr 23 10:43:05 smithi204 ceph-aa240572-0159-11ef-bc93-c7b262605968-osd-11[57644]: 2024-04-23T10:43:05.757+0000 7fcbc4b63700 -1 osd.11 116 heartbeat_check: no reply from 172.21.15.119:6812 osd.4 since back 2024-04-23T10:42:35.082088+0000 front 2024-04-23T10:42:35.082102+0000 (oldest deadline 2024-04-23T10:42:59.782202+0000) 2024-04-23T10:43:05.828 INFO:journalctl@ceph.osd.8.smithi204.stdout:Apr 23 10:43:05 smithi204 ceph-aa240572-0159-11ef-bc93-c7b262605968-osd-8[51050]: 2024-04-23T10:43:05.537+0000 7f8d662cd700 -1 osd.8 116 heartbeat_check: no reply from 172.21.15.114:6806 osd.0 since back 2024-04-23T10:42:03.759281+0000 front 2024-04-23T10:42:03.759348+0000 (oldest deadline 2024-04-23T10:42:27.859455+0000) 2024-04-23T10:43:05.829 INFO:journalctl@ceph.osd.8.smithi204.stdout:Apr 23 10:43:05 smithi204 ceph-aa240572-0159-11ef-bc93-c7b262605968-osd-8[51050]: 2024-04-23T10:43:05.537+0000 7f8d662cd700 -1 osd.8 116 heartbeat_check: no reply from 172.21.15.119:6804 osd.1 since back 2024-04-23T10:42:09.560739+0000 front 2024-04-23T10:42:09.560541+0000 (oldest deadline 2024-04-23T10:42:37.761119+0000) 2024-04-23T10:43:05.829 INFO:journalctl@ceph.osd.8.smithi204.stdout:Apr 23 10:43:05 smithi204 ceph-aa240572-0159-11ef-bc93-c7b262605968-osd-8[51050]: 2024-04-23T10:43:05.537+0000 7f8d662cd700 -1 osd.8 116 heartbeat_check: no reply from 172.21.15.114:6814 osd.3 since back 2024-04-23T10:42:24.762812+0000 front 2024-04-23T10:42:24.762866+0000 (oldest deadline 2024-04-23T10:42:50.062939+0000) 2024-04-23T10:43:05.829 INFO:journalctl@ceph.osd.8.smithi204.stdout:Apr 23 10:43:05 smithi204 ceph-aa240572-0159-11ef-bc93-c7b262605968-osd-8[51050]: 2024-04-23T10:43:05.537+0000 7f8d662cd700 -1 osd.8 116 heartbeat_check: no reply from 172.21.15.119:6812 osd.4 since back 2024-04-23T10:42:36.464758+0000 front 2024-04-23T10:42:36.464792+0000 (oldest deadline 2024-04-23T10:43:01.764563+0000) 2024-04-23T10:43:05.992 INFO:journalctl@ceph.osd.10.smithi119.stdout:Apr 23 10:43:05 smithi119 ceph-aa240572-0159-11ef-bc93-c7b262605968-osd-10[57454]: 2024-04-23T10:43:05.618+0000 7fb1849bf700 -1 osd.10 116 heartbeat_check: no reply from 172.21.15.114:6806 osd.0 since back 2024-04-23T10:42:01.813174+0000 front 2024-04-23T10:42:01.813138+0000 (oldest deadline 2024-04-23T10:42:27.113300+0000) 2024-04-23T10:43:05.992 INFO:journalctl@ceph.osd.10.smithi119.stdout:Apr 23 10:43:05 smithi119 ceph-aa240572-0159-11ef-bc93-c7b262605968-osd-10[57454]: 2024-04-23T10:43:05.618+0000 7fb1849bf700 -1 osd.10 116 heartbeat_check: no reply from 172.21.15.119:6804 osd.1 since back 2024-04-23T10:42:10.614599+0000 front 2024-04-23T10:42:10.614622+0000 (oldest deadline 2024-04-23T10:42:34.114881+0000) 2024-04-23T10:43:05.992 INFO:journalctl@ceph.osd.10.smithi119.stdout:Apr 23 10:43:05 smithi119 ceph-aa240572-0159-11ef-bc93-c7b262605968-osd-10[57454]: 2024-04-23T10:43:05.618+0000 7fb1849bf700 -1 osd.10 116 heartbeat_check: no reply from 172.21.15.204:6804 osd.2 since back 2024-04-23T10:42:19.416108+0000 front 2024-04-23T10:42:19.416241+0000 (oldest deadline 2024-04-23T10:42:42.316071+0000) 2024-04-23T10:43:05.992 INFO:journalctl@ceph.osd.10.smithi119.stdout:Apr 23 10:43:05 smithi119 ceph-aa240572-0159-11ef-bc93-c7b262605968-osd-10[57454]: 2024-04-23T10:43:05.618+0000 7fb1849bf700 -1 osd.10 116 heartbeat_check: no reply from 172.21.15.114:6814 osd.3 since back 2024-04-23T10:42:25.217210+0000 front 2024-04-23T10:42:25.217268+0000 (oldest deadline 2024-04-23T10:42:49.917291+0000) 2024-04-23T10:43:05.993 INFO:journalctl@ceph.osd.10.smithi119.stdout:Apr 23 10:43:05 smithi119 ceph-aa240572-0159-11ef-bc93-c7b262605968-osd-10[57454]: 2024-04-23T10:43:05.618+0000 7fb1849bf700 -1 osd.10 116 heartbeat_check: no reply from 172.21.15.119:6812 osd.4 since back 2024-04-23T10:42:36.620525+0000 front 2024-04-23T10:42:36.620567+0000 (oldest deadline 2024-04-23T10:43:00.720710+0000) 2024-04-23T10:43:06.754 INFO:journalctl@ceph.osd.9.smithi114.stdout:Apr 23 10:43:06 smithi114 ceph-aa240572-0159-11ef-bc93-c7b262605968-osd-9[62058]: 2024-04-23T10:43:06.415+0000 7feeb4bc7700 -1 osd.9 116 heartbeat_check: no reply from 172.21.15.114:6806 osd.0 since back 2024-04-23T10:42:04.136736+0000 front 2024-04-23T10:42:04.136764+0000 (oldest deadline 2024-04-23T10:42:28.836926+0000) 2024-04-23T10:43:06.754 INFO:journalctl@ceph.osd.9.smithi114.stdout:Apr 23 10:43:06 smithi114 ceph-aa240572-0159-11ef-bc93-c7b262605968-osd-9[62058]: 2024-04-23T10:43:06.415+0000 7feeb4bc7700 -1 osd.9 116 heartbeat_check: no reply from 172.21.15.119:6804 osd.1 since back 2024-04-23T10:42:08.837665+0000 front 2024-04-23T10:42:08.837503+0000 (oldest deadline 2024-04-23T10:42:34.737571+0000) 2024-04-23T10:43:06.755 INFO:journalctl@ceph.osd.9.smithi114.stdout:Apr 23 10:43:06 smithi114 ceph-aa240572-0159-11ef-bc93-c7b262605968-osd-9[62058]: 2024-04-23T10:43:06.415+0000 7feeb4bc7700 -1 osd.9 116 heartbeat_check: no reply from 172.21.15.204:6804 osd.2 since back 2024-04-23T10:42:18.139450+0000 front 2024-04-23T10:42:18.139388+0000 (oldest deadline 2024-04-23T10:42:43.439473+0000) 2024-04-23T10:43:06.755 INFO:journalctl@ceph.osd.9.smithi114.stdout:Apr 23 10:43:06 smithi114 ceph-aa240572-0159-11ef-bc93-c7b262605968-osd-9[62058]: 2024-04-23T10:43:06.415+0000 7feeb4bc7700 -1 osd.9 116 heartbeat_check: no reply from 172.21.15.114:6814 osd.3 since back 2024-04-23T10:42:28.641242+0000 front 2024-04-23T10:42:28.641354+0000 (oldest deadline 2024-04-23T10:42:52.741420+0000) 2024-04-23T10:43:06.755 INFO:journalctl@ceph.osd.9.smithi114.stdout:Apr 23 10:43:06 smithi114 ceph-aa240572-0159-11ef-bc93-c7b262605968-osd-9[62058]: 2024-04-23T10:43:06.415+0000 7feeb4bc7700 -1 osd.9 116 heartbeat_check: no reply from 172.21.15.119:6812 osd.4 since back 2024-04-23T10:42:32.742101+0000 front 2024-04-23T10:42:32.742087+0000 (oldest deadline 2024-04-23T10:42:58.642130+0000) 2024-04-23T10:43:06.755 INFO:journalctl@ceph.osd.9.smithi114.stdout:Apr 23 10:43:06 smithi114 ceph-aa240572-0159-11ef-bc93-c7b262605968-osd-9[62058]: 2024-04-23T10:43:06.415+0000 7feeb4bc7700 -1 osd.9 116 heartbeat_check: no reply from 172.21.15.204:6812 osd.5 since back 2024-04-23T10:42:43.943328+0000 front 2024-04-23T10:42:43.943282+0000 (oldest deadline 2024-04-23T10:43:06.243220+0000) 2024-04-23T10:43:06.828 INFO:journalctl@ceph.osd.11.smithi204.stdout:Apr 23 10:43:06 smithi204 ceph-aa240572-0159-11ef-bc93-c7b262605968-osd-11[57644]: 2024-04-23T10:43:06.738+0000 7fcbc4b63700 -1 osd.11 116 heartbeat_check: no reply from 172.21.15.114:6806 osd.0 since back 2024-04-23T10:42:00.675680+0000 front 2024-04-23T10:42:00.675649+0000 (oldest deadline 2024-04-23T10:42:25.876359+0000) 2024-04-23T10:43:06.828 INFO:journalctl@ceph.osd.11.smithi204.stdout:Apr 23 10:43:06 smithi204 ceph-aa240572-0159-11ef-bc93-c7b262605968-osd-11[57644]: 2024-04-23T10:43:06.738+0000 7fcbc4b63700 -1 osd.11 116 heartbeat_check: no reply from 172.21.15.119:6804 osd.1 since back 2024-04-23T10:42:09.278512+0000 front 2024-04-23T10:42:09.278494+0000 (oldest deadline 2024-04-23T10:42:33.378557+0000) 2024-04-23T10:43:06.828 INFO:journalctl@ceph.osd.11.smithi204.stdout:Apr 23 10:43:06 smithi204 ceph-aa240572-0159-11ef-bc93-c7b262605968-osd-11[57644]: 2024-04-23T10:43:06.738+0000 7fcbc4b63700 -1 osd.11 116 heartbeat_check: no reply from 172.21.15.204:6804 osd.2 since back 2024-04-23T10:42:18.679708+0000 front 2024-04-23T10:42:18.679577+0000 (oldest deadline 2024-04-23T10:42:41.579766+0000) 2024-04-23T10:43:06.828 INFO:journalctl@ceph.osd.11.smithi204.stdout:Apr 23 10:43:06 smithi204 ceph-aa240572-0159-11ef-bc93-c7b262605968-osd-11[57644]: 2024-04-23T10:43:06.738+0000 7fcbc4b63700 -1 osd.11 116 heartbeat_check: no reply from 172.21.15.119:6812 osd.4 since back 2024-04-23T10:42:35.082088+0000 front 2024-04-23T10:42:35.082102+0000 (oldest deadline 2024-04-23T10:42:59.782202+0000) 2024-04-23T10:43:06.829 INFO:journalctl@ceph.osd.8.smithi204.stdout:Apr 23 10:43:06 smithi204 ceph-aa240572-0159-11ef-bc93-c7b262605968-osd-8[51050]: 2024-04-23T10:43:06.520+0000 7f8d662cd700 -1 osd.8 116 heartbeat_check: no reply from 172.21.15.114:6806 osd.0 since back 2024-04-23T10:42:03.759281+0000 front 2024-04-23T10:42:03.759348+0000 (oldest deadline 2024-04-23T10:42:27.859455+0000) 2024-04-23T10:43:06.829 INFO:journalctl@ceph.osd.8.smithi204.stdout:Apr 23 10:43:06 smithi204 ceph-aa240572-0159-11ef-bc93-c7b262605968-osd-8[51050]: 2024-04-23T10:43:06.520+0000 7f8d662cd700 -1 osd.8 116 heartbeat_check: no reply from 172.21.15.119:6804 osd.1 since back 2024-04-23T10:42:09.560739+0000 front 2024-04-23T10:42:09.560541+0000 (oldest deadline 2024-04-23T10:42:37.761119+0000) 2024-04-23T10:43:06.829 INFO:journalctl@ceph.osd.8.smithi204.stdout:Apr 23 10:43:06 smithi204 ceph-aa240572-0159-11ef-bc93-c7b262605968-osd-8[51050]: 2024-04-23T10:43:06.520+0000 7f8d662cd700 -1 osd.8 116 heartbeat_check: no reply from 172.21.15.114:6814 osd.3 since back 2024-04-23T10:42:24.762812+0000 front 2024-04-23T10:42:24.762866+0000 (oldest deadline 2024-04-23T10:42:50.062939+0000) 2024-04-23T10:43:06.829 INFO:journalctl@ceph.osd.8.smithi204.stdout:Apr 23 10:43:06 smithi204 ceph-aa240572-0159-11ef-bc93-c7b262605968-osd-8[51050]: 2024-04-23T10:43:06.520+0000 7f8d662cd700 -1 osd.8 116 heartbeat_check: no reply from 172.21.15.119:6812 osd.4 since back 2024-04-23T10:42:36.464758+0000 front 2024-04-23T10:42:36.464792+0000 (oldest deadline 2024-04-23T10:43:01.764563+0000) 2024-04-23T10:43:06.829 INFO:journalctl@ceph.osd.8.smithi204.stdout:Apr 23 10:43:06 smithi204 ceph-aa240572-0159-11ef-bc93-c7b262605968-osd-8[51050]: 2024-04-23T10:43:06.520+0000 7f8d662cd700 -1 osd.8 116 heartbeat_check: no reply from 172.21.15.204:6812 osd.5 since back 2024-04-23T10:42:41.764997+0000 front 2024-04-23T10:42:41.765024+0000 (oldest deadline 2024-04-23T10:43:06.465056+0000) 2024-04-23T10:43:06.893 INFO:journalctl@ceph.osd.10.smithi119.stdout:Apr 23 10:43:06 smithi119 ceph-aa240572-0159-11ef-bc93-c7b262605968-osd-10[57454]: 2024-04-23T10:43:06.622+0000 7fb1849bf700 -1 osd.10 116 heartbeat_check: no reply from 172.21.15.114:6806 osd.0 since back 2024-04-23T10:42:01.813174+0000 front 2024-04-23T10:42:01.813138+0000 (oldest deadline 2024-04-23T10:42:27.113300+0000) 2024-04-23T10:43:06.893 INFO:journalctl@ceph.osd.10.smithi119.stdout:Apr 23 10:43:06 smithi119 ceph-aa240572-0159-11ef-bc93-c7b262605968-osd-10[57454]: 2024-04-23T10:43:06.622+0000 7fb1849bf700 -1 osd.10 116 heartbeat_check: no reply from 172.21.15.119:6804 osd.1 since back 2024-04-23T10:42:10.614599+0000 front 2024-04-23T10:42:10.614622+0000 (oldest deadline 2024-04-23T10:42:34.114881+0000) 2024-04-23T10:43:06.894 INFO:journalctl@ceph.osd.10.smithi119.stdout:Apr 23 10:43:06 smithi119 ceph-aa240572-0159-11ef-bc93-c7b262605968-osd-10[57454]: 2024-04-23T10:43:06.622+0000 7fb1849bf700 -1 osd.10 116 heartbeat_check: no reply from 172.21.15.204:6804 osd.2 since back 2024-04-23T10:42:19.416108+0000 front 2024-04-23T10:42:19.416241+0000 (oldest deadline 2024-04-23T10:42:42.316071+0000) 2024-04-23T10:43:06.894 INFO:journalctl@ceph.osd.10.smithi119.stdout:Apr 23 10:43:06 smithi119 ceph-aa240572-0159-11ef-bc93-c7b262605968-osd-10[57454]: 2024-04-23T10:43:06.622+0000 7fb1849bf700 -1 osd.10 116 heartbeat_check: no reply from 172.21.15.114:6814 osd.3 since back 2024-04-23T10:42:25.217210+0000 front 2024-04-23T10:42:25.217268+0000 (oldest deadline 2024-04-23T10:42:49.917291+0000) 2024-04-23T10:43:06.894 INFO:journalctl@ceph.osd.10.smithi119.stdout:Apr 23 10:43:06 smithi119 ceph-aa240572-0159-11ef-bc93-c7b262605968-osd-10[57454]: 2024-04-23T10:43:06.622+0000 7fb1849bf700 -1 osd.10 116 heartbeat_check: no reply from 172.21.15.119:6812 osd.4 since back 2024-04-23T10:42:36.620525+0000 front 2024-04-23T10:42:36.620567+0000 (oldest deadline 2024-04-23T10:43:00.720710+0000) 2024-04-23T10:43:06.894 INFO:journalctl@ceph.osd.10.smithi119.stdout:Apr 23 10:43:06 smithi119 ceph-aa240572-0159-11ef-bc93-c7b262605968-osd-10[57454]: 2024-04-23T10:43:06.622+0000 7fb1849bf700 -1 osd.10 116 heartbeat_check: no reply from 172.21.15.204:6812 osd.5 since back 2024-04-23T10:42:40.721579+0000 front 2024-04-23T10:42:40.721424+0000 (oldest deadline 2024-04-23T10:43:06.621309+0000) 2024-04-23T10:43:07.754 INFO:journalctl@ceph.osd.9.smithi114.stdout:Apr 23 10:43:07 smithi114 ceph-aa240572-0159-11ef-bc93-c7b262605968-osd-9[62058]: 2024-04-23T10:43:07.421+0000 7feeb4bc7700 -1 osd.9 116 heartbeat_check: no reply from 172.21.15.114:6806 osd.0 since back 2024-04-23T10:42:04.136736+0000 front 2024-04-23T10:42:04.136764+0000 (oldest deadline 2024-04-23T10:42:28.836926+0000) 2024-04-23T10:43:07.754 INFO:journalctl@ceph.osd.9.smithi114.stdout:Apr 23 10:43:07 smithi114 ceph-aa240572-0159-11ef-bc93-c7b262605968-osd-9[62058]: 2024-04-23T10:43:07.421+0000 7feeb4bc7700 -1 osd.9 116 heartbeat_check: no reply from 172.21.15.119:6804 osd.1 since back 2024-04-23T10:42:08.837665+0000 front 2024-04-23T10:42:08.837503+0000 (oldest deadline 2024-04-23T10:42:34.737571+0000) 2024-04-23T10:43:07.754 INFO:journalctl@ceph.osd.9.smithi114.stdout:Apr 23 10:43:07 smithi114 ceph-aa240572-0159-11ef-bc93-c7b262605968-osd-9[62058]: 2024-04-23T10:43:07.421+0000 7feeb4bc7700 -1 osd.9 116 heartbeat_check: no reply from 172.21.15.204:6804 osd.2 since back 2024-04-23T10:42:18.139450+0000 front 2024-04-23T10:42:18.139388+0000 (oldest deadline 2024-04-23T10:42:43.439473+0000) 2024-04-23T10:43:07.754 INFO:journalctl@ceph.osd.9.smithi114.stdout:Apr 23 10:43:07 smithi114 ceph-aa240572-0159-11ef-bc93-c7b262605968-osd-9[62058]: 2024-04-23T10:43:07.421+0000 7feeb4bc7700 -1 osd.9 116 heartbeat_check: no reply from 172.21.15.114:6814 osd.3 since back 2024-04-23T10:42:28.641242+0000 front 2024-04-23T10:42:28.641354+0000 (oldest deadline 2024-04-23T10:42:52.741420+0000) 2024-04-23T10:43:07.755 INFO:journalctl@ceph.osd.9.smithi114.stdout:Apr 23 10:43:07 smithi114 ceph-aa240572-0159-11ef-bc93-c7b262605968-osd-9[62058]: 2024-04-23T10:43:07.421+0000 7feeb4bc7700 -1 osd.9 116 heartbeat_check: no reply from 172.21.15.119:6812 osd.4 since back 2024-04-23T10:42:32.742101+0000 front 2024-04-23T10:42:32.742087+0000 (oldest deadline 2024-04-23T10:42:58.642130+0000) 2024-04-23T10:43:07.755 INFO:journalctl@ceph.osd.9.smithi114.stdout:Apr 23 10:43:07 smithi114 ceph-aa240572-0159-11ef-bc93-c7b262605968-osd-9[62058]: 2024-04-23T10:43:07.421+0000 7feeb4bc7700 -1 osd.9 116 heartbeat_check: no reply from 172.21.15.204:6812 osd.5 since back 2024-04-23T10:42:43.943328+0000 front 2024-04-23T10:42:43.943282+0000 (oldest deadline 2024-04-23T10:43:06.243220+0000) 2024-04-23T10:43:07.828 INFO:journalctl@ceph.osd.8.smithi204.stdout:Apr 23 10:43:07 smithi204 ceph-aa240572-0159-11ef-bc93-c7b262605968-osd-8[51050]: 2024-04-23T10:43:07.541+0000 7f8d662cd700 -1 osd.8 116 heartbeat_check: no reply from 172.21.15.114:6806 osd.0 since back 2024-04-23T10:42:03.759281+0000 front 2024-04-23T10:42:03.759348+0000 (oldest deadline 2024-04-23T10:42:27.859455+0000) 2024-04-23T10:43:07.828 INFO:journalctl@ceph.osd.8.smithi204.stdout:Apr 23 10:43:07 smithi204 ceph-aa240572-0159-11ef-bc93-c7b262605968-osd-8[51050]: 2024-04-23T10:43:07.541+0000 7f8d662cd700 -1 osd.8 116 heartbeat_check: no reply from 172.21.15.119:6804 osd.1 since back 2024-04-23T10:42:09.560739+0000 front 2024-04-23T10:42:09.560541+0000 (oldest deadline 2024-04-23T10:42:37.761119+0000) 2024-04-23T10:43:07.828 INFO:journalctl@ceph.osd.8.smithi204.stdout:Apr 23 10:43:07 smithi204 ceph-aa240572-0159-11ef-bc93-c7b262605968-osd-8[51050]: 2024-04-23T10:43:07.541+0000 7f8d662cd700 -1 osd.8 116 heartbeat_check: no reply from 172.21.15.114:6814 osd.3 since back 2024-04-23T10:42:24.762812+0000 front 2024-04-23T10:42:24.762866+0000 (oldest deadline 2024-04-23T10:42:50.062939+0000) 2024-04-23T10:43:07.828 INFO:journalctl@ceph.osd.8.smithi204.stdout:Apr 23 10:43:07 smithi204 ceph-aa240572-0159-11ef-bc93-c7b262605968-osd-8[51050]: 2024-04-23T10:43:07.541+0000 7f8d662cd700 -1 osd.8 116 heartbeat_check: no reply from 172.21.15.119:6812 osd.4 since back 2024-04-23T10:42:36.464758+0000 front 2024-04-23T10:42:36.464792+0000 (oldest deadline 2024-04-23T10:43:01.764563+0000) 2024-04-23T10:43:07.828 INFO:journalctl@ceph.osd.8.smithi204.stdout:Apr 23 10:43:07 smithi204 ceph-aa240572-0159-11ef-bc93-c7b262605968-osd-8[51050]: 2024-04-23T10:43:07.541+0000 7f8d662cd700 -1 osd.8 116 heartbeat_check: no reply from 172.21.15.204:6812 osd.5 since back 2024-04-23T10:42:41.764997+0000 front 2024-04-23T10:42:41.765024+0000 (oldest deadline 2024-04-23T10:43:06.465056+0000) 2024-04-23T10:43:07.828 INFO:journalctl@ceph.osd.11.smithi204.stdout:Apr 23 10:43:07 smithi204 ceph-aa240572-0159-11ef-bc93-c7b262605968-osd-11[57644]: 2024-04-23T10:43:07.786+0000 7fcbc4b63700 -1 osd.11 116 heartbeat_check: no reply from 172.21.15.114:6806 osd.0 since back 2024-04-23T10:42:00.675680+0000 front 2024-04-23T10:42:00.675649+0000 (oldest deadline 2024-04-23T10:42:25.876359+0000) 2024-04-23T10:43:07.828 INFO:journalctl@ceph.osd.11.smithi204.stdout:Apr 23 10:43:07 smithi204 ceph-aa240572-0159-11ef-bc93-c7b262605968-osd-11[57644]: 2024-04-23T10:43:07.786+0000 7fcbc4b63700 -1 osd.11 116 heartbeat_check: no reply from 172.21.15.119:6804 osd.1 since back 2024-04-23T10:42:09.278512+0000 front 2024-04-23T10:42:09.278494+0000 (oldest deadline 2024-04-23T10:42:33.378557+0000) 2024-04-23T10:43:07.829 INFO:journalctl@ceph.osd.11.smithi204.stdout:Apr 23 10:43:07 smithi204 ceph-aa240572-0159-11ef-bc93-c7b262605968-osd-11[57644]: 2024-04-23T10:43:07.786+0000 7fcbc4b63700 -1 osd.11 116 heartbeat_check: no reply from 172.21.15.204:6804 osd.2 since back 2024-04-23T10:42:18.679708+0000 front 2024-04-23T10:42:18.679577+0000 (oldest deadline 2024-04-23T10:42:41.579766+0000) 2024-04-23T10:43:07.829 INFO:journalctl@ceph.osd.11.smithi204.stdout:Apr 23 10:43:07 smithi204 ceph-aa240572-0159-11ef-bc93-c7b262605968-osd-11[57644]: 2024-04-23T10:43:07.786+0000 7fcbc4b63700 -1 osd.11 116 heartbeat_check: no reply from 172.21.15.119:6812 osd.4 since back 2024-04-23T10:42:35.082088+0000 front 2024-04-23T10:42:35.082102+0000 (oldest deadline 2024-04-23T10:42:59.782202+0000) 2024-04-23T10:43:07.992 INFO:journalctl@ceph.osd.10.smithi119.stdout:Apr 23 10:43:07 smithi119 ceph-aa240572-0159-11ef-bc93-c7b262605968-osd-10[57454]: 2024-04-23T10:43:07.587+0000 7fb1849bf700 -1 osd.10 116 heartbeat_check: no reply from 172.21.15.114:6806 osd.0 since back 2024-04-23T10:42:01.813174+0000 front 2024-04-23T10:42:01.813138+0000 (oldest deadline 2024-04-23T10:42:27.113300+0000) 2024-04-23T10:43:07.992 INFO:journalctl@ceph.osd.10.smithi119.stdout:Apr 23 10:43:07 smithi119 ceph-aa240572-0159-11ef-bc93-c7b262605968-osd-10[57454]: 2024-04-23T10:43:07.587+0000 7fb1849bf700 -1 osd.10 116 heartbeat_check: no reply from 172.21.15.119:6804 osd.1 since back 2024-04-23T10:42:10.614599+0000 front 2024-04-23T10:42:10.614622+0000 (oldest deadline 2024-04-23T10:42:34.114881+0000) 2024-04-23T10:43:07.992 INFO:journalctl@ceph.osd.10.smithi119.stdout:Apr 23 10:43:07 smithi119 ceph-aa240572-0159-11ef-bc93-c7b262605968-osd-10[57454]: 2024-04-23T10:43:07.587+0000 7fb1849bf700 -1 osd.10 116 heartbeat_check: no reply from 172.21.15.204:6804 osd.2 since back 2024-04-23T10:42:19.416108+0000 front 2024-04-23T10:42:19.416241+0000 (oldest deadline 2024-04-23T10:42:42.316071+0000) 2024-04-23T10:43:07.992 INFO:journalctl@ceph.osd.10.smithi119.stdout:Apr 23 10:43:07 smithi119 ceph-aa240572-0159-11ef-bc93-c7b262605968-osd-10[57454]: 2024-04-23T10:43:07.587+0000 7fb1849bf700 -1 osd.10 116 heartbeat_check: no reply from 172.21.15.114:6814 osd.3 since back 2024-04-23T10:42:25.217210+0000 front 2024-04-23T10:42:25.217268+0000 (oldest deadline 2024-04-23T10:42:49.917291+0000) 2024-04-23T10:43:07.992 INFO:journalctl@ceph.osd.10.smithi119.stdout:Apr 23 10:43:07 smithi119 ceph-aa240572-0159-11ef-bc93-c7b262605968-osd-10[57454]: 2024-04-23T10:43:07.587+0000 7fb1849bf700 -1 osd.10 116 heartbeat_check: no reply from 172.21.15.119:6812 osd.4 since back 2024-04-23T10:42:36.620525+0000 front 2024-04-23T10:42:36.620567+0000 (oldest deadline 2024-04-23T10:43:00.720710+0000) 2024-04-23T10:43:07.992 INFO:journalctl@ceph.osd.10.smithi119.stdout:Apr 23 10:43:07 smithi119 ceph-aa240572-0159-11ef-bc93-c7b262605968-osd-10[57454]: 2024-04-23T10:43:07.587+0000 7fb1849bf700 -1 osd.10 116 heartbeat_check: no reply from 172.21.15.204:6812 osd.5 since back 2024-04-23T10:42:40.721579+0000 front 2024-04-23T10:42:40.721424+0000 (oldest deadline 2024-04-23T10:43:06.621309+0000) 2024-04-23T10:43:08.754 INFO:journalctl@ceph.osd.9.smithi114.stdout:Apr 23 10:43:08 smithi114 ceph-aa240572-0159-11ef-bc93-c7b262605968-osd-9[62058]: 2024-04-23T10:43:08.391+0000 7feeb4bc7700 -1 osd.9 116 heartbeat_check: no reply from 172.21.15.114:6806 osd.0 since back 2024-04-23T10:42:04.136736+0000 front 2024-04-23T10:42:04.136764+0000 (oldest deadline 2024-04-23T10:42:28.836926+0000) 2024-04-23T10:43:08.754 INFO:journalctl@ceph.osd.9.smithi114.stdout:Apr 23 10:43:08 smithi114 ceph-aa240572-0159-11ef-bc93-c7b262605968-osd-9[62058]: 2024-04-23T10:43:08.391+0000 7feeb4bc7700 -1 osd.9 116 heartbeat_check: no reply from 172.21.15.119:6804 osd.1 since back 2024-04-23T10:42:08.837665+0000 front 2024-04-23T10:42:08.837503+0000 (oldest deadline 2024-04-23T10:42:34.737571+0000) 2024-04-23T10:43:08.755 INFO:journalctl@ceph.osd.9.smithi114.stdout:Apr 23 10:43:08 smithi114 ceph-aa240572-0159-11ef-bc93-c7b262605968-osd-9[62058]: 2024-04-23T10:43:08.391+0000 7feeb4bc7700 -1 osd.9 116 heartbeat_check: no reply from 172.21.15.204:6804 osd.2 since back 2024-04-23T10:42:18.139450+0000 front 2024-04-23T10:42:18.139388+0000 (oldest deadline 2024-04-23T10:42:43.439473+0000) 2024-04-23T10:43:08.755 INFO:journalctl@ceph.osd.9.smithi114.stdout:Apr 23 10:43:08 smithi114 ceph-aa240572-0159-11ef-bc93-c7b262605968-osd-9[62058]: 2024-04-23T10:43:08.392+0000 7feeb4bc7700 -1 osd.9 116 heartbeat_check: no reply from 172.21.15.114:6814 osd.3 since back 2024-04-23T10:42:28.641242+0000 front 2024-04-23T10:42:28.641354+0000 (oldest deadline 2024-04-23T10:42:52.741420+0000) 2024-04-23T10:43:08.755 INFO:journalctl@ceph.osd.9.smithi114.stdout:Apr 23 10:43:08 smithi114 ceph-aa240572-0159-11ef-bc93-c7b262605968-osd-9[62058]: 2024-04-23T10:43:08.392+0000 7feeb4bc7700 -1 osd.9 116 heartbeat_check: no reply from 172.21.15.119:6812 osd.4 since back 2024-04-23T10:42:32.742101+0000 front 2024-04-23T10:42:32.742087+0000 (oldest deadline 2024-04-23T10:42:58.642130+0000) 2024-04-23T10:43:08.755 INFO:journalctl@ceph.osd.9.smithi114.stdout:Apr 23 10:43:08 smithi114 ceph-aa240572-0159-11ef-bc93-c7b262605968-osd-9[62058]: 2024-04-23T10:43:08.392+0000 7feeb4bc7700 -1 osd.9 116 heartbeat_check: no reply from 172.21.15.204:6812 osd.5 since back 2024-04-23T10:42:43.943328+0000 front 2024-04-23T10:42:43.943282+0000 (oldest deadline 2024-04-23T10:43:06.243220+0000) 2024-04-23T10:43:08.992 INFO:journalctl@ceph.osd.10.smithi119.stdout:Apr 23 10:43:08 smithi119 ceph-aa240572-0159-11ef-bc93-c7b262605968-osd-10[57454]: 2024-04-23T10:43:08.597+0000 7fb1849bf700 -1 osd.10 116 heartbeat_check: no reply from 172.21.15.114:6806 osd.0 since back 2024-04-23T10:42:01.813174+0000 front 2024-04-23T10:42:01.813138+0000 (oldest deadline 2024-04-23T10:42:27.113300+0000) 2024-04-23T10:43:08.992 INFO:journalctl@ceph.osd.10.smithi119.stdout:Apr 23 10:43:08 smithi119 ceph-aa240572-0159-11ef-bc93-c7b262605968-osd-10[57454]: 2024-04-23T10:43:08.597+0000 7fb1849bf700 -1 osd.10 116 heartbeat_check: no reply from 172.21.15.119:6804 osd.1 since back 2024-04-23T10:42:10.614599+0000 front 2024-04-23T10:42:10.614622+0000 (oldest deadline 2024-04-23T10:42:34.114881+0000) 2024-04-23T10:43:08.992 INFO:journalctl@ceph.osd.10.smithi119.stdout:Apr 23 10:43:08 smithi119 ceph-aa240572-0159-11ef-bc93-c7b262605968-osd-10[57454]: 2024-04-23T10:43:08.597+0000 7fb1849bf700 -1 osd.10 116 heartbeat_check: no reply from 172.21.15.204:6804 osd.2 since back 2024-04-23T10:42:19.416108+0000 front 2024-04-23T10:42:19.416241+0000 (oldest deadline 2024-04-23T10:42:42.316071+0000) 2024-04-23T10:43:08.993 INFO:journalctl@ceph.osd.10.smithi119.stdout:Apr 23 10:43:08 smithi119 ceph-aa240572-0159-11ef-bc93-c7b262605968-osd-10[57454]: 2024-04-23T10:43:08.597+0000 7fb1849bf700 -1 osd.10 116 heartbeat_check: no reply from 172.21.15.114:6814 osd.3 since back 2024-04-23T10:42:25.217210+0000 front 2024-04-23T10:42:25.217268+0000 (oldest deadline 2024-04-23T10:42:49.917291+0000) 2024-04-23T10:43:08.993 INFO:journalctl@ceph.osd.10.smithi119.stdout:Apr 23 10:43:08 smithi119 ceph-aa240572-0159-11ef-bc93-c7b262605968-osd-10[57454]: 2024-04-23T10:43:08.597+0000 7fb1849bf700 -1 osd.10 116 heartbeat_check: no reply from 172.21.15.119:6812 osd.4 since back 2024-04-23T10:42:36.620525+0000 front 2024-04-23T10:42:36.620567+0000 (oldest deadline 2024-04-23T10:43:00.720710+0000) 2024-04-23T10:43:08.993 INFO:journalctl@ceph.osd.10.smithi119.stdout:Apr 23 10:43:08 smithi119 ceph-aa240572-0159-11ef-bc93-c7b262605968-osd-10[57454]: 2024-04-23T10:43:08.597+0000 7fb1849bf700 -1 osd.10 116 heartbeat_check: no reply from 172.21.15.204:6812 osd.5 since back 2024-04-23T10:42:40.721579+0000 front 2024-04-23T10:42:40.721424+0000 (oldest deadline 2024-04-23T10:43:06.621309+0000) 2024-04-23T10:43:09.078 INFO:journalctl@ceph.osd.8.smithi204.stdout:Apr 23 10:43:08 smithi204 podman[71276]: 2024-04-23 10:43:08.969381114 +0000 UTC m=+5.627864402 container died 91762dd15696d3e85c850e24def20b86f7446266f49ad6b289aa40097811e525 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:068a558e3af4932cf1c60dd321c0bdf500075eb52ea1bf455e0973ba114fbfd4, name=ceph-aa240572-0159-11ef-bc93-c7b262605968-osd-8, org.label-schema.schema-version=1.0, io.buildah.version=1.33.5, org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 8 Base Image, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, org.label-schema.build-date=20240416, ceph=True, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , GIT_CLEAN=True, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, RELEASE=wip-lusov-quiescer-d69f13d) 2024-04-23T10:43:09.079 INFO:journalctl@ceph.osd.11.smithi204.stdout:Apr 23 10:43:08 smithi204 ceph-aa240572-0159-11ef-bc93-c7b262605968-osd-11[57644]: 2024-04-23T10:43:08.825+0000 7fcbc4b63700 -1 osd.11 116 heartbeat_check: no reply from 172.21.15.114:6806 osd.0 since back 2024-04-23T10:42:00.675680+0000 front 2024-04-23T10:42:00.675649+0000 (oldest deadline 2024-04-23T10:42:25.876359+0000) 2024-04-23T10:43:09.079 INFO:journalctl@ceph.osd.11.smithi204.stdout:Apr 23 10:43:08 smithi204 ceph-aa240572-0159-11ef-bc93-c7b262605968-osd-11[57644]: 2024-04-23T10:43:08.825+0000 7fcbc4b63700 -1 osd.11 116 heartbeat_check: no reply from 172.21.15.119:6804 osd.1 since back 2024-04-23T10:42:09.278512+0000 front 2024-04-23T10:42:09.278494+0000 (oldest deadline 2024-04-23T10:42:33.378557+0000) 2024-04-23T10:43:09.079 INFO:journalctl@ceph.osd.11.smithi204.stdout:Apr 23 10:43:08 smithi204 ceph-aa240572-0159-11ef-bc93-c7b262605968-osd-11[57644]: 2024-04-23T10:43:08.825+0000 7fcbc4b63700 -1 osd.11 116 heartbeat_check: no reply from 172.21.15.204:6804 osd.2 since back 2024-04-23T10:42:18.679708+0000 front 2024-04-23T10:42:18.679577+0000 (oldest deadline 2024-04-23T10:42:41.579766+0000) 2024-04-23T10:43:09.079 INFO:journalctl@ceph.osd.11.smithi204.stdout:Apr 23 10:43:08 smithi204 ceph-aa240572-0159-11ef-bc93-c7b262605968-osd-11[57644]: 2024-04-23T10:43:08.825+0000 7fcbc4b63700 -1 osd.11 116 heartbeat_check: no reply from 172.21.15.119:6812 osd.4 since back 2024-04-23T10:42:35.082088+0000 front 2024-04-23T10:42:35.082102+0000 (oldest deadline 2024-04-23T10:42:59.782202+0000) 2024-04-23T10:43:09.079 INFO:journalctl@ceph.osd.11.smithi204.stdout:Apr 23 10:43:08 smithi204 ceph-aa240572-0159-11ef-bc93-c7b262605968-osd-11[57644]: 2024-04-23T10:43:08.825+0000 7fcbc4b63700 -1 osd.11 116 heartbeat_check: no reply from 172.21.15.204:6812 osd.5 since back 2024-04-23T10:42:39.782680+0000 front 2024-04-23T10:42:39.782723+0000 (oldest deadline 2024-04-23T10:43:08.583479+0000) 2024-04-23T10:43:09.578 INFO:journalctl@ceph.osd.8.smithi204.stdout:Apr 23 10:43:09 smithi204 podman[71276]: 2024-04-23 10:43:09.330219445 +0000 UTC m=+5.988702733 container cleanup 91762dd15696d3e85c850e24def20b86f7446266f49ad6b289aa40097811e525 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:068a558e3af4932cf1c60dd321c0bdf500075eb52ea1bf455e0973ba114fbfd4, name=ceph-aa240572-0159-11ef-bc93-c7b262605968-osd-8, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 8 Base Image, GIT_BRANCH=HEAD, io.buildah.version=1.33.5, ceph=True, GIT_CLEAN=True, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240416, org.label-schema.schema-version=1.0, RELEASE=wip-lusov-quiescer-d69f13d, maintainer=Guillaume Abrioux ) 2024-04-23T10:43:09.579 INFO:journalctl@ceph.osd.8.smithi204.stdout:Apr 23 10:43:09 smithi204 bash[71276]: ceph-aa240572-0159-11ef-bc93-c7b262605968-osd-8 2024-04-23T10:43:09.754 INFO:journalctl@ceph.osd.9.smithi114.stdout:Apr 23 10:43:09 smithi114 ceph-aa240572-0159-11ef-bc93-c7b262605968-osd-9[62058]: 2024-04-23T10:43:09.433+0000 7feeb4bc7700 -1 osd.9 116 heartbeat_check: no reply from 172.21.15.114:6806 osd.0 since back 2024-04-23T10:42:04.136736+0000 front 2024-04-23T10:42:04.136764+0000 (oldest deadline 2024-04-23T10:42:28.836926+0000) 2024-04-23T10:43:09.754 INFO:journalctl@ceph.osd.9.smithi114.stdout:Apr 23 10:43:09 smithi114 ceph-aa240572-0159-11ef-bc93-c7b262605968-osd-9[62058]: 2024-04-23T10:43:09.433+0000 7feeb4bc7700 -1 osd.9 116 heartbeat_check: no reply from 172.21.15.119:6804 osd.1 since back 2024-04-23T10:42:08.837665+0000 front 2024-04-23T10:42:08.837503+0000 (oldest deadline 2024-04-23T10:42:34.737571+0000) 2024-04-23T10:43:09.754 INFO:journalctl@ceph.osd.9.smithi114.stdout:Apr 23 10:43:09 smithi114 ceph-aa240572-0159-11ef-bc93-c7b262605968-osd-9[62058]: 2024-04-23T10:43:09.433+0000 7feeb4bc7700 -1 osd.9 116 heartbeat_check: no reply from 172.21.15.204:6804 osd.2 since back 2024-04-23T10:42:18.139450+0000 front 2024-04-23T10:42:18.139388+0000 (oldest deadline 2024-04-23T10:42:43.439473+0000) 2024-04-23T10:43:09.755 INFO:journalctl@ceph.osd.9.smithi114.stdout:Apr 23 10:43:09 smithi114 ceph-aa240572-0159-11ef-bc93-c7b262605968-osd-9[62058]: 2024-04-23T10:43:09.433+0000 7feeb4bc7700 -1 osd.9 116 heartbeat_check: no reply from 172.21.15.114:6814 osd.3 since back 2024-04-23T10:42:28.641242+0000 front 2024-04-23T10:42:28.641354+0000 (oldest deadline 2024-04-23T10:42:52.741420+0000) 2024-04-23T10:43:09.755 INFO:journalctl@ceph.osd.9.smithi114.stdout:Apr 23 10:43:09 smithi114 ceph-aa240572-0159-11ef-bc93-c7b262605968-osd-9[62058]: 2024-04-23T10:43:09.433+0000 7feeb4bc7700 -1 osd.9 116 heartbeat_check: no reply from 172.21.15.119:6812 osd.4 since back 2024-04-23T10:42:32.742101+0000 front 2024-04-23T10:42:32.742087+0000 (oldest deadline 2024-04-23T10:42:58.642130+0000) 2024-04-23T10:43:09.755 INFO:journalctl@ceph.osd.9.smithi114.stdout:Apr 23 10:43:09 smithi114 ceph-aa240572-0159-11ef-bc93-c7b262605968-osd-9[62058]: 2024-04-23T10:43:09.433+0000 7feeb4bc7700 -1 osd.9 116 heartbeat_check: no reply from 172.21.15.204:6812 osd.5 since back 2024-04-23T10:42:43.943328+0000 front 2024-04-23T10:42:43.943282+0000 (oldest deadline 2024-04-23T10:43:06.243220+0000) 2024-04-23T10:43:09.930 INFO:journalctl@ceph.osd.8.smithi204.stdout:Apr 23 10:43:09 smithi204 podman[71306]: 2024-04-23 10:43:09.582063212 +0000 UTC m=+0.604121192 container remove 91762dd15696d3e85c850e24def20b86f7446266f49ad6b289aa40097811e525 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:068a558e3af4932cf1c60dd321c0bdf500075eb52ea1bf455e0973ba114fbfd4, name=ceph-aa240572-0159-11ef-bc93-c7b262605968-osd-8, io.buildah.version=1.33.5, org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 8 Base Image, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, ceph=True, org.label-schema.license=GPLv2, GIT_CLEAN=True, RELEASE=wip-lusov-quiescer-d69f13d, GIT_BRANCH=HEAD, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240416, org.label-schema.vendor=CentOS) 2024-04-23T10:43:09.931 INFO:journalctl@ceph.osd.11.smithi204.stdout:Apr 23 10:43:09 smithi204 ceph-aa240572-0159-11ef-bc93-c7b262605968-osd-11[57644]: 2024-04-23T10:43:09.806+0000 7fcbc4b63700 -1 osd.11 116 heartbeat_check: no reply from 172.21.15.114:6806 osd.0 since back 2024-04-23T10:42:00.675680+0000 front 2024-04-23T10:42:00.675649+0000 (oldest deadline 2024-04-23T10:42:25.876359+0000) 2024-04-23T10:43:09.931 INFO:journalctl@ceph.osd.11.smithi204.stdout:Apr 23 10:43:09 smithi204 ceph-aa240572-0159-11ef-bc93-c7b262605968-osd-11[57644]: 2024-04-23T10:43:09.806+0000 7fcbc4b63700 -1 osd.11 116 heartbeat_check: no reply from 172.21.15.119:6804 osd.1 since back 2024-04-23T10:42:09.278512+0000 front 2024-04-23T10:42:09.278494+0000 (oldest deadline 2024-04-23T10:42:33.378557+0000) 2024-04-23T10:43:09.931 INFO:journalctl@ceph.osd.11.smithi204.stdout:Apr 23 10:43:09 smithi204 ceph-aa240572-0159-11ef-bc93-c7b262605968-osd-11[57644]: 2024-04-23T10:43:09.806+0000 7fcbc4b63700 -1 osd.11 116 heartbeat_check: no reply from 172.21.15.204:6804 osd.2 since back 2024-04-23T10:42:18.679708+0000 front 2024-04-23T10:42:18.679577+0000 (oldest deadline 2024-04-23T10:42:41.579766+0000) 2024-04-23T10:43:09.931 INFO:journalctl@ceph.osd.11.smithi204.stdout:Apr 23 10:43:09 smithi204 ceph-aa240572-0159-11ef-bc93-c7b262605968-osd-11[57644]: 2024-04-23T10:43:09.806+0000 7fcbc4b63700 -1 osd.11 116 heartbeat_check: no reply from 172.21.15.119:6812 osd.4 since back 2024-04-23T10:42:35.082088+0000 front 2024-04-23T10:42:35.082102+0000 (oldest deadline 2024-04-23T10:42:59.782202+0000) 2024-04-23T10:43:09.931 INFO:journalctl@ceph.osd.11.smithi204.stdout:Apr 23 10:43:09 smithi204 ceph-aa240572-0159-11ef-bc93-c7b262605968-osd-11[57644]: 2024-04-23T10:43:09.806+0000 7fcbc4b63700 -1 osd.11 116 heartbeat_check: no reply from 172.21.15.204:6812 osd.5 since back 2024-04-23T10:42:39.782680+0000 front 2024-04-23T10:42:39.782723+0000 (oldest deadline 2024-04-23T10:43:08.583479+0000) 2024-04-23T10:43:09.992 INFO:journalctl@ceph.osd.10.smithi119.stdout:Apr 23 10:43:09 smithi119 ceph-aa240572-0159-11ef-bc93-c7b262605968-osd-10[57454]: 2024-04-23T10:43:09.602+0000 7fb1849bf700 -1 osd.10 116 heartbeat_check: no reply from 172.21.15.114:6806 osd.0 since back 2024-04-23T10:42:01.813174+0000 front 2024-04-23T10:42:01.813138+0000 (oldest deadline 2024-04-23T10:42:27.113300+0000) 2024-04-23T10:43:09.993 INFO:journalctl@ceph.osd.10.smithi119.stdout:Apr 23 10:43:09 smithi119 ceph-aa240572-0159-11ef-bc93-c7b262605968-osd-10[57454]: 2024-04-23T10:43:09.602+0000 7fb1849bf700 -1 osd.10 116 heartbeat_check: no reply from 172.21.15.119:6804 osd.1 since back 2024-04-23T10:42:10.614599+0000 front 2024-04-23T10:42:10.614622+0000 (oldest deadline 2024-04-23T10:42:34.114881+0000) 2024-04-23T10:43:09.993 INFO:journalctl@ceph.osd.10.smithi119.stdout:Apr 23 10:43:09 smithi119 ceph-aa240572-0159-11ef-bc93-c7b262605968-osd-10[57454]: 2024-04-23T10:43:09.602+0000 7fb1849bf700 -1 osd.10 116 heartbeat_check: no reply from 172.21.15.204:6804 osd.2 since back 2024-04-23T10:42:19.416108+0000 front 2024-04-23T10:42:19.416241+0000 (oldest deadline 2024-04-23T10:42:42.316071+0000) 2024-04-23T10:43:09.993 INFO:journalctl@ceph.osd.10.smithi119.stdout:Apr 23 10:43:09 smithi119 ceph-aa240572-0159-11ef-bc93-c7b262605968-osd-10[57454]: 2024-04-23T10:43:09.602+0000 7fb1849bf700 -1 osd.10 116 heartbeat_check: no reply from 172.21.15.114:6814 osd.3 since back 2024-04-23T10:42:25.217210+0000 front 2024-04-23T10:42:25.217268+0000 (oldest deadline 2024-04-23T10:42:49.917291+0000) 2024-04-23T10:43:09.993 INFO:journalctl@ceph.osd.10.smithi119.stdout:Apr 23 10:43:09 smithi119 ceph-aa240572-0159-11ef-bc93-c7b262605968-osd-10[57454]: 2024-04-23T10:43:09.602+0000 7fb1849bf700 -1 osd.10 116 heartbeat_check: no reply from 172.21.15.119:6812 osd.4 since back 2024-04-23T10:42:36.620525+0000 front 2024-04-23T10:42:36.620567+0000 (oldest deadline 2024-04-23T10:43:00.720710+0000) 2024-04-23T10:43:09.993 INFO:journalctl@ceph.osd.10.smithi119.stdout:Apr 23 10:43:09 smithi119 ceph-aa240572-0159-11ef-bc93-c7b262605968-osd-10[57454]: 2024-04-23T10:43:09.602+0000 7fb1849bf700 -1 osd.10 116 heartbeat_check: no reply from 172.21.15.204:6812 osd.5 since back 2024-04-23T10:42:40.721579+0000 front 2024-04-23T10:42:40.721424+0000 (oldest deadline 2024-04-23T10:43:06.621309+0000) 2024-04-23T10:43:10.225 INFO:journalctl@ceph.osd.8.smithi204.stdout:Apr 23 10:43:09 smithi204 podman[71400]: 2024-04-23 10:43:09.835812275 +0000 UTC m=+0.018288482 image pull da4b004c09e820713483d039bc76fbd660be678f2ea3c03e059a6161b74dab76 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:068a558e3af4932cf1c60dd321c0bdf500075eb52ea1bf455e0973ba114fbfd4 2024-04-23T10:43:10.225 INFO:journalctl@ceph.osd.8.smithi204.stdout:Apr 23 10:43:09 smithi204 podman[71400]: 2024-04-23 10:43:09.932797006 +0000 UTC m=+0.115273208 container create 12e8f00fd683a6e23a4b92fb19ccf33a2b4028fc7cac3725aea6e72237de20bb (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:068a558e3af4932cf1c60dd321c0bdf500075eb52ea1bf455e0973ba114fbfd4, name=ceph-aa240572-0159-11ef-bc93-c7b262605968-osd-8-deactivate, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.schema-version=1.0, ceph=True, RELEASE=wip-lusov-quiescer-d69f13d, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, org.label-schema.license=GPLv2, GIT_CLEAN=True, io.buildah.version=1.33.5, org.label-schema.build-date=20240416, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux ) 2024-04-23T10:43:10.225 INFO:journalctl@ceph.osd.8.smithi204.stdout:Apr 23 10:43:10 smithi204 podman[71400]: 2024-04-23 10:43:10.143796375 +0000 UTC m=+0.326272575 container init 12e8f00fd683a6e23a4b92fb19ccf33a2b4028fc7cac3725aea6e72237de20bb (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:068a558e3af4932cf1c60dd321c0bdf500075eb52ea1bf455e0973ba114fbfd4, name=ceph-aa240572-0159-11ef-bc93-c7b262605968-osd-8-deactivate, org.label-schema.name=CentOS Stream 8 Base Image, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0, RELEASE=wip-lusov-quiescer-d69f13d, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_CLEAN=True, org.label-schema.build-date=20240416, org.label-schema.license=GPLv2, ceph=True, maintainer=Guillaume Abrioux , io.buildah.version=1.33.5, CEPH_POINT_RELEASE=) 2024-04-23T10:43:10.226 INFO:journalctl@ceph.osd.8.smithi204.stdout:Apr 23 10:43:10 smithi204 podman[71400]: 2024-04-23 10:43:10.146798214 +0000 UTC m=+0.329274421 container start 12e8f00fd683a6e23a4b92fb19ccf33a2b4028fc7cac3725aea6e72237de20bb (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:068a558e3af4932cf1c60dd321c0bdf500075eb52ea1bf455e0973ba114fbfd4, name=ceph-aa240572-0159-11ef-bc93-c7b262605968-osd-8-deactivate, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 8 Base Image, ceph=True, org.label-schema.build-date=20240416, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, org.label-schema.schema-version=1.0, org.label-schema.license=GPLv2, RELEASE=wip-lusov-quiescer-d69f13d, org.label-schema.vendor=CentOS, GIT_CLEAN=True, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.33.5, maintainer=Guillaume Abrioux ) 2024-04-23T10:43:10.226 INFO:journalctl@ceph.osd.8.smithi204.stdout:Apr 23 10:43:10 smithi204 podman[71400]: 2024-04-23 10:43:10.226920894 +0000 UTC m=+0.409397117 container attach 12e8f00fd683a6e23a4b92fb19ccf33a2b4028fc7cac3725aea6e72237de20bb (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:068a558e3af4932cf1c60dd321c0bdf500075eb52ea1bf455e0973ba114fbfd4, name=ceph-aa240572-0159-11ef-bc93-c7b262605968-osd-8-deactivate, org.label-schema.license=GPLv2, org.label-schema.build-date=20240416, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.33.5, GIT_BRANCH=HEAD, GIT_CLEAN=True, org.label-schema.schema-version=1.0, maintainer=Guillaume Abrioux , ceph=True, RELEASE=wip-lusov-quiescer-d69f13d, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, org.label-schema.name=CentOS Stream 8 Base Image) 2024-04-23T10:43:10.578 INFO:journalctl@ceph.osd.8.smithi204.stdout:Apr 23 10:43:10 smithi204 podman[71400]: 2024-04-23 10:43:10.445657961 +0000 UTC m=+0.628134172 container died 12e8f00fd683a6e23a4b92fb19ccf33a2b4028fc7cac3725aea6e72237de20bb (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:068a558e3af4932cf1c60dd321c0bdf500075eb52ea1bf455e0973ba114fbfd4, name=ceph-aa240572-0159-11ef-bc93-c7b262605968-osd-8-deactivate, RELEASE=wip-lusov-quiescer-d69f13d, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, org.label-schema.schema-version=1.0, io.buildah.version=1.33.5, org.label-schema.build-date=20240416, org.label-schema.license=GPLv2, ceph=True, GIT_CLEAN=True, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 8 Base Image, CEPH_POINT_RELEASE=) 2024-04-23T10:43:10.754 INFO:journalctl@ceph.osd.9.smithi114.stdout:Apr 23 10:43:10 smithi114 ceph-aa240572-0159-11ef-bc93-c7b262605968-osd-9[62058]: 2024-04-23T10:43:10.456+0000 7feeb4bc7700 -1 osd.9 116 heartbeat_check: no reply from 172.21.15.114:6806 osd.0 since back 2024-04-23T10:42:04.136736+0000 front 2024-04-23T10:42:04.136764+0000 (oldest deadline 2024-04-23T10:42:28.836926+0000) 2024-04-23T10:43:10.754 INFO:journalctl@ceph.osd.9.smithi114.stdout:Apr 23 10:43:10 smithi114 ceph-aa240572-0159-11ef-bc93-c7b262605968-osd-9[62058]: 2024-04-23T10:43:10.456+0000 7feeb4bc7700 -1 osd.9 116 heartbeat_check: no reply from 172.21.15.119:6804 osd.1 since back 2024-04-23T10:42:08.837665+0000 front 2024-04-23T10:42:08.837503+0000 (oldest deadline 2024-04-23T10:42:34.737571+0000) 2024-04-23T10:43:10.754 INFO:journalctl@ceph.osd.9.smithi114.stdout:Apr 23 10:43:10 smithi114 ceph-aa240572-0159-11ef-bc93-c7b262605968-osd-9[62058]: 2024-04-23T10:43:10.456+0000 7feeb4bc7700 -1 osd.9 116 heartbeat_check: no reply from 172.21.15.204:6804 osd.2 since back 2024-04-23T10:42:18.139450+0000 front 2024-04-23T10:42:18.139388+0000 (oldest deadline 2024-04-23T10:42:43.439473+0000) 2024-04-23T10:43:10.754 INFO:journalctl@ceph.osd.9.smithi114.stdout:Apr 23 10:43:10 smithi114 ceph-aa240572-0159-11ef-bc93-c7b262605968-osd-9[62058]: 2024-04-23T10:43:10.456+0000 7feeb4bc7700 -1 osd.9 116 heartbeat_check: no reply from 172.21.15.114:6814 osd.3 since back 2024-04-23T10:42:28.641242+0000 front 2024-04-23T10:42:28.641354+0000 (oldest deadline 2024-04-23T10:42:52.741420+0000) 2024-04-23T10:43:10.755 INFO:journalctl@ceph.osd.9.smithi114.stdout:Apr 23 10:43:10 smithi114 ceph-aa240572-0159-11ef-bc93-c7b262605968-osd-9[62058]: 2024-04-23T10:43:10.456+0000 7feeb4bc7700 -1 osd.9 116 heartbeat_check: no reply from 172.21.15.119:6812 osd.4 since back 2024-04-23T10:42:32.742101+0000 front 2024-04-23T10:42:32.742087+0000 (oldest deadline 2024-04-23T10:42:58.642130+0000) 2024-04-23T10:43:10.755 INFO:journalctl@ceph.osd.9.smithi114.stdout:Apr 23 10:43:10 smithi114 ceph-aa240572-0159-11ef-bc93-c7b262605968-osd-9[62058]: 2024-04-23T10:43:10.456+0000 7feeb4bc7700 -1 osd.9 116 heartbeat_check: no reply from 172.21.15.204:6812 osd.5 since back 2024-04-23T10:42:43.943328+0000 front 2024-04-23T10:42:43.943282+0000 (oldest deadline 2024-04-23T10:43:06.243220+0000) 2024-04-23T10:43:10.992 INFO:journalctl@ceph.osd.10.smithi119.stdout:Apr 23 10:43:10 smithi119 ceph-aa240572-0159-11ef-bc93-c7b262605968-osd-10[57454]: 2024-04-23T10:43:10.642+0000 7fb1849bf700 -1 osd.10 116 heartbeat_check: no reply from 172.21.15.114:6806 osd.0 since back 2024-04-23T10:42:01.813174+0000 front 2024-04-23T10:42:01.813138+0000 (oldest deadline 2024-04-23T10:42:27.113300+0000) 2024-04-23T10:43:10.992 INFO:journalctl@ceph.osd.10.smithi119.stdout:Apr 23 10:43:10 smithi119 ceph-aa240572-0159-11ef-bc93-c7b262605968-osd-10[57454]: 2024-04-23T10:43:10.642+0000 7fb1849bf700 -1 osd.10 116 heartbeat_check: no reply from 172.21.15.119:6804 osd.1 since back 2024-04-23T10:42:10.614599+0000 front 2024-04-23T10:42:10.614622+0000 (oldest deadline 2024-04-23T10:42:34.114881+0000) 2024-04-23T10:43:10.992 INFO:journalctl@ceph.osd.10.smithi119.stdout:Apr 23 10:43:10 smithi119 ceph-aa240572-0159-11ef-bc93-c7b262605968-osd-10[57454]: 2024-04-23T10:43:10.642+0000 7fb1849bf700 -1 osd.10 116 heartbeat_check: no reply from 172.21.15.204:6804 osd.2 since back 2024-04-23T10:42:19.416108+0000 front 2024-04-23T10:42:19.416241+0000 (oldest deadline 2024-04-23T10:42:42.316071+0000) 2024-04-23T10:43:10.992 INFO:journalctl@ceph.osd.10.smithi119.stdout:Apr 23 10:43:10 smithi119 ceph-aa240572-0159-11ef-bc93-c7b262605968-osd-10[57454]: 2024-04-23T10:43:10.642+0000 7fb1849bf700 -1 osd.10 116 heartbeat_check: no reply from 172.21.15.114:6814 osd.3 since back 2024-04-23T10:42:25.217210+0000 front 2024-04-23T10:42:25.217268+0000 (oldest deadline 2024-04-23T10:42:49.917291+0000) 2024-04-23T10:43:10.993 INFO:journalctl@ceph.osd.10.smithi119.stdout:Apr 23 10:43:10 smithi119 ceph-aa240572-0159-11ef-bc93-c7b262605968-osd-10[57454]: 2024-04-23T10:43:10.642+0000 7fb1849bf700 -1 osd.10 116 heartbeat_check: no reply from 172.21.15.119:6812 osd.4 since back 2024-04-23T10:42:36.620525+0000 front 2024-04-23T10:42:36.620567+0000 (oldest deadline 2024-04-23T10:43:00.720710+0000) 2024-04-23T10:43:10.993 INFO:journalctl@ceph.osd.10.smithi119.stdout:Apr 23 10:43:10 smithi119 ceph-aa240572-0159-11ef-bc93-c7b262605968-osd-10[57454]: 2024-04-23T10:43:10.642+0000 7fb1849bf700 -1 osd.10 116 heartbeat_check: no reply from 172.21.15.204:6812 osd.5 since back 2024-04-23T10:42:40.721579+0000 front 2024-04-23T10:42:40.721424+0000 (oldest deadline 2024-04-23T10:43:06.621309+0000) 2024-04-23T10:43:11.032 INFO:journalctl@ceph.osd.8.smithi204.stdout:Apr 23 10:43:11 smithi204 podman[71426]: 2024-04-23 10:43:11.014331541 +0000 UTC m=+0.563612999 container remove 12e8f00fd683a6e23a4b92fb19ccf33a2b4028fc7cac3725aea6e72237de20bb (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:068a558e3af4932cf1c60dd321c0bdf500075eb52ea1bf455e0973ba114fbfd4, name=ceph-aa240572-0159-11ef-bc93-c7b262605968-osd-8-deactivate, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, GIT_CLEAN=True, org.label-schema.build-date=20240416, org.label-schema.name=CentOS Stream 8 Base Image, io.buildah.version=1.33.5, RELEASE=wip-lusov-quiescer-d69f13d, org.label-schema.schema-version=1.0, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD, org.label-schema.license=GPLv2) 2024-04-23T10:43:11.033 INFO:journalctl@ceph.osd.11.smithi204.stdout:Apr 23 10:43:10 smithi204 ceph-aa240572-0159-11ef-bc93-c7b262605968-osd-11[57644]: 2024-04-23T10:43:10.852+0000 7fcbc4b63700 -1 osd.11 116 heartbeat_check: no reply from 172.21.15.114:6806 osd.0 since back 2024-04-23T10:42:00.675680+0000 front 2024-04-23T10:42:00.675649+0000 (oldest deadline 2024-04-23T10:42:25.876359+0000) 2024-04-23T10:43:11.033 INFO:journalctl@ceph.osd.11.smithi204.stdout:Apr 23 10:43:10 smithi204 ceph-aa240572-0159-11ef-bc93-c7b262605968-osd-11[57644]: 2024-04-23T10:43:10.852+0000 7fcbc4b63700 -1 osd.11 116 heartbeat_check: no reply from 172.21.15.119:6804 osd.1 since back 2024-04-23T10:42:09.278512+0000 front 2024-04-23T10:42:09.278494+0000 (oldest deadline 2024-04-23T10:42:33.378557+0000) 2024-04-23T10:43:11.033 INFO:journalctl@ceph.osd.11.smithi204.stdout:Apr 23 10:43:10 smithi204 ceph-aa240572-0159-11ef-bc93-c7b262605968-osd-11[57644]: 2024-04-23T10:43:10.852+0000 7fcbc4b63700 -1 osd.11 116 heartbeat_check: no reply from 172.21.15.204:6804 osd.2 since back 2024-04-23T10:42:18.679708+0000 front 2024-04-23T10:42:18.679577+0000 (oldest deadline 2024-04-23T10:42:41.579766+0000) 2024-04-23T10:43:11.033 INFO:journalctl@ceph.osd.11.smithi204.stdout:Apr 23 10:43:10 smithi204 ceph-aa240572-0159-11ef-bc93-c7b262605968-osd-11[57644]: 2024-04-23T10:43:10.852+0000 7fcbc4b63700 -1 osd.11 116 heartbeat_check: no reply from 172.21.15.119:6812 osd.4 since back 2024-04-23T10:42:35.082088+0000 front 2024-04-23T10:42:35.082102+0000 (oldest deadline 2024-04-23T10:42:59.782202+0000) 2024-04-23T10:43:11.034 INFO:journalctl@ceph.osd.11.smithi204.stdout:Apr 23 10:43:10 smithi204 ceph-aa240572-0159-11ef-bc93-c7b262605968-osd-11[57644]: 2024-04-23T10:43:10.852+0000 7fcbc4b63700 -1 osd.11 116 heartbeat_check: no reply from 172.21.15.204:6812 osd.5 since back 2024-04-23T10:42:39.782680+0000 front 2024-04-23T10:42:39.782723+0000 (oldest deadline 2024-04-23T10:43:08.583479+0000) 2024-04-23T10:43:11.038 DEBUG:teuthology.orchestra.run.smithi204:> sudo pkill -f 'journalctl -f -n 0 -u ceph-aa240572-0159-11ef-bc93-c7b262605968@osd.8.service' 2024-04-23T10:43:11.079 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-04-23T10:43:11.080 INFO:tasks.cephadm.osd.8:Stopped osd.8 2024-04-23T10:43:11.080 INFO:tasks.cephadm.osd.9:Stopping osd.9... 2024-04-23T10:43:11.080 DEBUG:teuthology.orchestra.run.smithi114:> sudo systemctl stop ceph-aa240572-0159-11ef-bc93-c7b262605968@osd.9 2024-04-23T10:43:11.396 INFO:journalctl@ceph.osd.9.smithi114.stdout:Apr 23 10:43:11 smithi114 systemd[1]: Stopping Ceph osd.9 for aa240572-0159-11ef-bc93-c7b262605968... 2024-04-23T10:43:11.396 INFO:journalctl@ceph.osd.9.smithi114.stdout:Apr 23 10:43:11 smithi114 ceph-aa240572-0159-11ef-bc93-c7b262605968-osd-9[62058]: 2024-04-23T10:43:11.397+0000 7feeb85d3700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.9 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-04-23T10:43:11.396 INFO:journalctl@ceph.osd.9.smithi114.stdout:Apr 23 10:43:11 smithi114 ceph-aa240572-0159-11ef-bc93-c7b262605968-osd-9[62058]: 2024-04-23T10:43:11.397+0000 7feeb85d3700 -1 osd.9 116 *** Got signal Terminated *** 2024-04-23T10:43:11.754 INFO:journalctl@ceph.osd.9.smithi114.stdout:Apr 23 10:43:11 smithi114 ceph-aa240572-0159-11ef-bc93-c7b262605968-osd-9[62058]: 2024-04-23T10:43:11.397+0000 7feeb85d3700 -1 osd.9 116 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-04-23T10:43:11.754 INFO:journalctl@ceph.osd.9.smithi114.stdout:Apr 23 10:43:11 smithi114 ceph-aa240572-0159-11ef-bc93-c7b262605968-osd-9[62058]: 2024-04-23T10:43:11.472+0000 7feeb4bc7700 -1 osd.9 116 heartbeat_check: no reply from 172.21.15.114:6806 osd.0 since back 2024-04-23T10:42:04.136736+0000 front 2024-04-23T10:42:04.136764+0000 (oldest deadline 2024-04-23T10:42:28.836926+0000) 2024-04-23T10:43:11.754 INFO:journalctl@ceph.osd.9.smithi114.stdout:Apr 23 10:43:11 smithi114 ceph-aa240572-0159-11ef-bc93-c7b262605968-osd-9[62058]: 2024-04-23T10:43:11.472+0000 7feeb4bc7700 -1 osd.9 116 heartbeat_check: no reply from 172.21.15.119:6804 osd.1 since back 2024-04-23T10:42:08.837665+0000 front 2024-04-23T10:42:08.837503+0000 (oldest deadline 2024-04-23T10:42:34.737571+0000) 2024-04-23T10:43:11.755 INFO:journalctl@ceph.osd.9.smithi114.stdout:Apr 23 10:43:11 smithi114 ceph-aa240572-0159-11ef-bc93-c7b262605968-osd-9[62058]: 2024-04-23T10:43:11.472+0000 7feeb4bc7700 -1 osd.9 116 heartbeat_check: no reply from 172.21.15.204:6804 osd.2 since back 2024-04-23T10:42:18.139450+0000 front 2024-04-23T10:42:18.139388+0000 (oldest deadline 2024-04-23T10:42:43.439473+0000) 2024-04-23T10:43:11.755 INFO:journalctl@ceph.osd.9.smithi114.stdout:Apr 23 10:43:11 smithi114 ceph-aa240572-0159-11ef-bc93-c7b262605968-osd-9[62058]: 2024-04-23T10:43:11.472+0000 7feeb4bc7700 -1 osd.9 116 heartbeat_check: no reply from 172.21.15.114:6814 osd.3 since back 2024-04-23T10:42:28.641242+0000 front 2024-04-23T10:42:28.641354+0000 (oldest deadline 2024-04-23T10:42:52.741420+0000) 2024-04-23T10:43:11.755 INFO:journalctl@ceph.osd.9.smithi114.stdout:Apr 23 10:43:11 smithi114 ceph-aa240572-0159-11ef-bc93-c7b262605968-osd-9[62058]: 2024-04-23T10:43:11.472+0000 7feeb4bc7700 -1 osd.9 116 heartbeat_check: no reply from 172.21.15.119:6812 osd.4 since back 2024-04-23T10:42:32.742101+0000 front 2024-04-23T10:42:32.742087+0000 (oldest deadline 2024-04-23T10:42:58.642130+0000) 2024-04-23T10:43:11.755 INFO:journalctl@ceph.osd.9.smithi114.stdout:Apr 23 10:43:11 smithi114 ceph-aa240572-0159-11ef-bc93-c7b262605968-osd-9[62058]: 2024-04-23T10:43:11.472+0000 7feeb4bc7700 -1 osd.9 116 heartbeat_check: no reply from 172.21.15.204:6812 osd.5 since back 2024-04-23T10:42:43.943328+0000 front 2024-04-23T10:42:43.943282+0000 (oldest deadline 2024-04-23T10:43:06.243220+0000) 2024-04-23T10:43:11.992 INFO:journalctl@ceph.osd.10.smithi119.stdout:Apr 23 10:43:11 smithi119 ceph-aa240572-0159-11ef-bc93-c7b262605968-osd-10[57454]: 2024-04-23T10:43:11.670+0000 7fb1849bf700 -1 osd.10 116 heartbeat_check: no reply from 172.21.15.114:6806 osd.0 since back 2024-04-23T10:42:01.813174+0000 front 2024-04-23T10:42:01.813138+0000 (oldest deadline 2024-04-23T10:42:27.113300+0000) 2024-04-23T10:43:11.992 INFO:journalctl@ceph.osd.10.smithi119.stdout:Apr 23 10:43:11 smithi119 ceph-aa240572-0159-11ef-bc93-c7b262605968-osd-10[57454]: 2024-04-23T10:43:11.670+0000 7fb1849bf700 -1 osd.10 116 heartbeat_check: no reply from 172.21.15.119:6804 osd.1 since back 2024-04-23T10:42:10.614599+0000 front 2024-04-23T10:42:10.614622+0000 (oldest deadline 2024-04-23T10:42:34.114881+0000) 2024-04-23T10:43:11.992 INFO:journalctl@ceph.osd.10.smithi119.stdout:Apr 23 10:43:11 smithi119 ceph-aa240572-0159-11ef-bc93-c7b262605968-osd-10[57454]: 2024-04-23T10:43:11.670+0000 7fb1849bf700 -1 osd.10 116 heartbeat_check: no reply from 172.21.15.204:6804 osd.2 since back 2024-04-23T10:42:19.416108+0000 front 2024-04-23T10:42:19.416241+0000 (oldest deadline 2024-04-23T10:42:42.316071+0000) 2024-04-23T10:43:11.992 INFO:journalctl@ceph.osd.10.smithi119.stdout:Apr 23 10:43:11 smithi119 ceph-aa240572-0159-11ef-bc93-c7b262605968-osd-10[57454]: 2024-04-23T10:43:11.670+0000 7fb1849bf700 -1 osd.10 116 heartbeat_check: no reply from 172.21.15.114:6814 osd.3 since back 2024-04-23T10:42:25.217210+0000 front 2024-04-23T10:42:25.217268+0000 (oldest deadline 2024-04-23T10:42:49.917291+0000) 2024-04-23T10:43:11.992 INFO:journalctl@ceph.osd.10.smithi119.stdout:Apr 23 10:43:11 smithi119 ceph-aa240572-0159-11ef-bc93-c7b262605968-osd-10[57454]: 2024-04-23T10:43:11.670+0000 7fb1849bf700 -1 osd.10 116 heartbeat_check: no reply from 172.21.15.119:6812 osd.4 since back 2024-04-23T10:42:36.620525+0000 front 2024-04-23T10:42:36.620567+0000 (oldest deadline 2024-04-23T10:43:00.720710+0000) 2024-04-23T10:43:11.993 INFO:journalctl@ceph.osd.10.smithi119.stdout:Apr 23 10:43:11 smithi119 ceph-aa240572-0159-11ef-bc93-c7b262605968-osd-10[57454]: 2024-04-23T10:43:11.670+0000 7fb1849bf700 -1 osd.10 116 heartbeat_check: no reply from 172.21.15.204:6812 osd.5 since back 2024-04-23T10:42:40.721579+0000 front 2024-04-23T10:42:40.721424+0000 (oldest deadline 2024-04-23T10:43:06.621309+0000) 2024-04-23T10:43:12.328 INFO:journalctl@ceph.osd.11.smithi204.stdout:Apr 23 10:43:11 smithi204 ceph-aa240572-0159-11ef-bc93-c7b262605968-osd-11[57644]: 2024-04-23T10:43:11.876+0000 7fcbc4b63700 -1 osd.11 116 heartbeat_check: no reply from 172.21.15.114:6806 osd.0 since back 2024-04-23T10:42:00.675680+0000 front 2024-04-23T10:42:00.675649+0000 (oldest deadline 2024-04-23T10:42:25.876359+0000) 2024-04-23T10:43:12.328 INFO:journalctl@ceph.osd.11.smithi204.stdout:Apr 23 10:43:11 smithi204 ceph-aa240572-0159-11ef-bc93-c7b262605968-osd-11[57644]: 2024-04-23T10:43:11.876+0000 7fcbc4b63700 -1 osd.11 116 heartbeat_check: no reply from 172.21.15.119:6804 osd.1 since back 2024-04-23T10:42:09.278512+0000 front 2024-04-23T10:42:09.278494+0000 (oldest deadline 2024-04-23T10:42:33.378557+0000) 2024-04-23T10:43:12.328 INFO:journalctl@ceph.osd.11.smithi204.stdout:Apr 23 10:43:11 smithi204 ceph-aa240572-0159-11ef-bc93-c7b262605968-osd-11[57644]: 2024-04-23T10:43:11.876+0000 7fcbc4b63700 -1 osd.11 116 heartbeat_check: no reply from 172.21.15.204:6804 osd.2 since back 2024-04-23T10:42:18.679708+0000 front 2024-04-23T10:42:18.679577+0000 (oldest deadline 2024-04-23T10:42:41.579766+0000) 2024-04-23T10:43:12.328 INFO:journalctl@ceph.osd.11.smithi204.stdout:Apr 23 10:43:11 smithi204 ceph-aa240572-0159-11ef-bc93-c7b262605968-osd-11[57644]: 2024-04-23T10:43:11.876+0000 7fcbc4b63700 -1 osd.11 116 heartbeat_check: no reply from 172.21.15.119:6812 osd.4 since back 2024-04-23T10:42:35.082088+0000 front 2024-04-23T10:42:35.082102+0000 (oldest deadline 2024-04-23T10:42:59.782202+0000) 2024-04-23T10:43:12.328 INFO:journalctl@ceph.osd.11.smithi204.stdout:Apr 23 10:43:11 smithi204 ceph-aa240572-0159-11ef-bc93-c7b262605968-osd-11[57644]: 2024-04-23T10:43:11.876+0000 7fcbc4b63700 -1 osd.11 116 heartbeat_check: no reply from 172.21.15.204:6812 osd.5 since back 2024-04-23T10:42:39.782680+0000 front 2024-04-23T10:42:39.782723+0000 (oldest deadline 2024-04-23T10:43:08.583479+0000) 2024-04-23T10:43:12.754 INFO:journalctl@ceph.osd.9.smithi114.stdout:Apr 23 10:43:12 smithi114 ceph-aa240572-0159-11ef-bc93-c7b262605968-osd-9[62058]: 2024-04-23T10:43:12.496+0000 7feeb4bc7700 -1 osd.9 116 heartbeat_check: no reply from 172.21.15.114:6806 osd.0 since back 2024-04-23T10:42:04.136736+0000 front 2024-04-23T10:42:04.136764+0000 (oldest deadline 2024-04-23T10:42:28.836926+0000) 2024-04-23T10:43:12.754 INFO:journalctl@ceph.osd.9.smithi114.stdout:Apr 23 10:43:12 smithi114 ceph-aa240572-0159-11ef-bc93-c7b262605968-osd-9[62058]: 2024-04-23T10:43:12.496+0000 7feeb4bc7700 -1 osd.9 116 heartbeat_check: no reply from 172.21.15.119:6804 osd.1 since back 2024-04-23T10:42:08.837665+0000 front 2024-04-23T10:42:08.837503+0000 (oldest deadline 2024-04-23T10:42:34.737571+0000) 2024-04-23T10:43:12.754 INFO:journalctl@ceph.osd.9.smithi114.stdout:Apr 23 10:43:12 smithi114 ceph-aa240572-0159-11ef-bc93-c7b262605968-osd-9[62058]: 2024-04-23T10:43:12.496+0000 7feeb4bc7700 -1 osd.9 116 heartbeat_check: no reply from 172.21.15.204:6804 osd.2 since back 2024-04-23T10:42:18.139450+0000 front 2024-04-23T10:42:18.139388+0000 (oldest deadline 2024-04-23T10:42:43.439473+0000) 2024-04-23T10:43:12.754 INFO:journalctl@ceph.osd.9.smithi114.stdout:Apr 23 10:43:12 smithi114 ceph-aa240572-0159-11ef-bc93-c7b262605968-osd-9[62058]: 2024-04-23T10:43:12.496+0000 7feeb4bc7700 -1 osd.9 116 heartbeat_check: no reply from 172.21.15.114:6814 osd.3 since back 2024-04-23T10:42:28.641242+0000 front 2024-04-23T10:42:28.641354+0000 (oldest deadline 2024-04-23T10:42:52.741420+0000) 2024-04-23T10:43:12.754 INFO:journalctl@ceph.osd.9.smithi114.stdout:Apr 23 10:43:12 smithi114 ceph-aa240572-0159-11ef-bc93-c7b262605968-osd-9[62058]: 2024-04-23T10:43:12.496+0000 7feeb4bc7700 -1 osd.9 116 heartbeat_check: no reply from 172.21.15.119:6812 osd.4 since back 2024-04-23T10:42:32.742101+0000 front 2024-04-23T10:42:32.742087+0000 (oldest deadline 2024-04-23T10:42:58.642130+0000) 2024-04-23T10:43:12.754 INFO:journalctl@ceph.osd.9.smithi114.stdout:Apr 23 10:43:12 smithi114 ceph-aa240572-0159-11ef-bc93-c7b262605968-osd-9[62058]: 2024-04-23T10:43:12.496+0000 7feeb4bc7700 -1 osd.9 116 heartbeat_check: no reply from 172.21.15.204:6812 osd.5 since back 2024-04-23T10:42:43.943328+0000 front 2024-04-23T10:42:43.943282+0000 (oldest deadline 2024-04-23T10:43:06.243220+0000) 2024-04-23T10:43:12.992 INFO:journalctl@ceph.osd.10.smithi119.stdout:Apr 23 10:43:12 smithi119 ceph-aa240572-0159-11ef-bc93-c7b262605968-osd-10[57454]: 2024-04-23T10:43:12.667+0000 7fb1849bf700 -1 osd.10 116 heartbeat_check: no reply from 172.21.15.114:6806 osd.0 since back 2024-04-23T10:42:01.813174+0000 front 2024-04-23T10:42:01.813138+0000 (oldest deadline 2024-04-23T10:42:27.113300+0000) 2024-04-23T10:43:12.992 INFO:journalctl@ceph.osd.10.smithi119.stdout:Apr 23 10:43:12 smithi119 ceph-aa240572-0159-11ef-bc93-c7b262605968-osd-10[57454]: 2024-04-23T10:43:12.667+0000 7fb1849bf700 -1 osd.10 116 heartbeat_check: no reply from 172.21.15.119:6804 osd.1 since back 2024-04-23T10:42:10.614599+0000 front 2024-04-23T10:42:10.614622+0000 (oldest deadline 2024-04-23T10:42:34.114881+0000) 2024-04-23T10:43:12.992 INFO:journalctl@ceph.osd.10.smithi119.stdout:Apr 23 10:43:12 smithi119 ceph-aa240572-0159-11ef-bc93-c7b262605968-osd-10[57454]: 2024-04-23T10:43:12.667+0000 7fb1849bf700 -1 osd.10 116 heartbeat_check: no reply from 172.21.15.204:6804 osd.2 since back 2024-04-23T10:42:19.416108+0000 front 2024-04-23T10:42:19.416241+0000 (oldest deadline 2024-04-23T10:42:42.316071+0000) 2024-04-23T10:43:12.992 INFO:journalctl@ceph.osd.10.smithi119.stdout:Apr 23 10:43:12 smithi119 ceph-aa240572-0159-11ef-bc93-c7b262605968-osd-10[57454]: 2024-04-23T10:43:12.667+0000 7fb1849bf700 -1 osd.10 116 heartbeat_check: no reply from 172.21.15.114:6814 osd.3 since back 2024-04-23T10:42:25.217210+0000 front 2024-04-23T10:42:25.217268+0000 (oldest deadline 2024-04-23T10:42:49.917291+0000) 2024-04-23T10:43:12.992 INFO:journalctl@ceph.osd.10.smithi119.stdout:Apr 23 10:43:12 smithi119 ceph-aa240572-0159-11ef-bc93-c7b262605968-osd-10[57454]: 2024-04-23T10:43:12.667+0000 7fb1849bf700 -1 osd.10 116 heartbeat_check: no reply from 172.21.15.119:6812 osd.4 since back 2024-04-23T10:42:36.620525+0000 front 2024-04-23T10:42:36.620567+0000 (oldest deadline 2024-04-23T10:43:00.720710+0000) 2024-04-23T10:43:12.992 INFO:journalctl@ceph.osd.10.smithi119.stdout:Apr 23 10:43:12 smithi119 ceph-aa240572-0159-11ef-bc93-c7b262605968-osd-10[57454]: 2024-04-23T10:43:12.667+0000 7fb1849bf700 -1 osd.10 116 heartbeat_check: no reply from 172.21.15.204:6812 osd.5 since back 2024-04-23T10:42:40.721579+0000 front 2024-04-23T10:42:40.721424+0000 (oldest deadline 2024-04-23T10:43:06.621309+0000) 2024-04-23T10:43:13.328 INFO:journalctl@ceph.osd.11.smithi204.stdout:Apr 23 10:43:12 smithi204 ceph-aa240572-0159-11ef-bc93-c7b262605968-osd-11[57644]: 2024-04-23T10:43:12.865+0000 7fcbc4b63700 -1 osd.11 116 heartbeat_check: no reply from 172.21.15.114:6806 osd.0 since back 2024-04-23T10:42:00.675680+0000 front 2024-04-23T10:42:00.675649+0000 (oldest deadline 2024-04-23T10:42:25.876359+0000) 2024-04-23T10:43:13.328 INFO:journalctl@ceph.osd.11.smithi204.stdout:Apr 23 10:43:12 smithi204 ceph-aa240572-0159-11ef-bc93-c7b262605968-osd-11[57644]: 2024-04-23T10:43:12.865+0000 7fcbc4b63700 -1 osd.11 116 heartbeat_check: no reply from 172.21.15.119:6804 osd.1 since back 2024-04-23T10:42:09.278512+0000 front 2024-04-23T10:42:09.278494+0000 (oldest deadline 2024-04-23T10:42:33.378557+0000) 2024-04-23T10:43:13.328 INFO:journalctl@ceph.osd.11.smithi204.stdout:Apr 23 10:43:12 smithi204 ceph-aa240572-0159-11ef-bc93-c7b262605968-osd-11[57644]: 2024-04-23T10:43:12.865+0000 7fcbc4b63700 -1 osd.11 116 heartbeat_check: no reply from 172.21.15.204:6804 osd.2 since back 2024-04-23T10:42:18.679708+0000 front 2024-04-23T10:42:18.679577+0000 (oldest deadline 2024-04-23T10:42:41.579766+0000) 2024-04-23T10:43:13.328 INFO:journalctl@ceph.osd.11.smithi204.stdout:Apr 23 10:43:12 smithi204 ceph-aa240572-0159-11ef-bc93-c7b262605968-osd-11[57644]: 2024-04-23T10:43:12.865+0000 7fcbc4b63700 -1 osd.11 116 heartbeat_check: no reply from 172.21.15.119:6812 osd.4 since back 2024-04-23T10:42:35.082088+0000 front 2024-04-23T10:42:35.082102+0000 (oldest deadline 2024-04-23T10:42:59.782202+0000) 2024-04-23T10:43:13.328 INFO:journalctl@ceph.osd.11.smithi204.stdout:Apr 23 10:43:12 smithi204 ceph-aa240572-0159-11ef-bc93-c7b262605968-osd-11[57644]: 2024-04-23T10:43:12.865+0000 7fcbc4b63700 -1 osd.11 116 heartbeat_check: no reply from 172.21.15.204:6812 osd.5 since back 2024-04-23T10:42:39.782680+0000 front 2024-04-23T10:42:39.782723+0000 (oldest deadline 2024-04-23T10:43:08.583479+0000) 2024-04-23T10:43:13.754 INFO:journalctl@ceph.osd.9.smithi114.stdout:Apr 23 10:43:13 smithi114 ceph-aa240572-0159-11ef-bc93-c7b262605968-osd-9[62058]: 2024-04-23T10:43:13.493+0000 7feeb4bc7700 -1 osd.9 116 heartbeat_check: no reply from 172.21.15.114:6806 osd.0 since back 2024-04-23T10:42:04.136736+0000 front 2024-04-23T10:42:04.136764+0000 (oldest deadline 2024-04-23T10:42:28.836926+0000) 2024-04-23T10:43:13.754 INFO:journalctl@ceph.osd.9.smithi114.stdout:Apr 23 10:43:13 smithi114 ceph-aa240572-0159-11ef-bc93-c7b262605968-osd-9[62058]: 2024-04-23T10:43:13.493+0000 7feeb4bc7700 -1 osd.9 116 heartbeat_check: no reply from 172.21.15.119:6804 osd.1 since back 2024-04-23T10:42:08.837665+0000 front 2024-04-23T10:42:08.837503+0000 (oldest deadline 2024-04-23T10:42:34.737571+0000) 2024-04-23T10:43:13.754 INFO:journalctl@ceph.osd.9.smithi114.stdout:Apr 23 10:43:13 smithi114 ceph-aa240572-0159-11ef-bc93-c7b262605968-osd-9[62058]: 2024-04-23T10:43:13.493+0000 7feeb4bc7700 -1 osd.9 116 heartbeat_check: no reply from 172.21.15.204:6804 osd.2 since back 2024-04-23T10:42:18.139450+0000 front 2024-04-23T10:42:18.139388+0000 (oldest deadline 2024-04-23T10:42:43.439473+0000) 2024-04-23T10:43:13.754 INFO:journalctl@ceph.osd.9.smithi114.stdout:Apr 23 10:43:13 smithi114 ceph-aa240572-0159-11ef-bc93-c7b262605968-osd-9[62058]: 2024-04-23T10:43:13.493+0000 7feeb4bc7700 -1 osd.9 116 heartbeat_check: no reply from 172.21.15.114:6814 osd.3 since back 2024-04-23T10:42:28.641242+0000 front 2024-04-23T10:42:28.641354+0000 (oldest deadline 2024-04-23T10:42:52.741420+0000) 2024-04-23T10:43:13.755 INFO:journalctl@ceph.osd.9.smithi114.stdout:Apr 23 10:43:13 smithi114 ceph-aa240572-0159-11ef-bc93-c7b262605968-osd-9[62058]: 2024-04-23T10:43:13.493+0000 7feeb4bc7700 -1 osd.9 116 heartbeat_check: no reply from 172.21.15.119:6812 osd.4 since back 2024-04-23T10:42:32.742101+0000 front 2024-04-23T10:42:32.742087+0000 (oldest deadline 2024-04-23T10:42:58.642130+0000) 2024-04-23T10:43:13.755 INFO:journalctl@ceph.osd.9.smithi114.stdout:Apr 23 10:43:13 smithi114 ceph-aa240572-0159-11ef-bc93-c7b262605968-osd-9[62058]: 2024-04-23T10:43:13.493+0000 7feeb4bc7700 -1 osd.9 116 heartbeat_check: no reply from 172.21.15.204:6812 osd.5 since back 2024-04-23T10:42:43.943328+0000 front 2024-04-23T10:42:43.943282+0000 (oldest deadline 2024-04-23T10:43:06.243220+0000) 2024-04-23T10:43:13.992 INFO:journalctl@ceph.osd.10.smithi119.stdout:Apr 23 10:43:13 smithi119 ceph-aa240572-0159-11ef-bc93-c7b262605968-osd-10[57454]: 2024-04-23T10:43:13.638+0000 7fb1849bf700 -1 osd.10 116 heartbeat_check: no reply from 172.21.15.114:6806 osd.0 since back 2024-04-23T10:42:01.813174+0000 front 2024-04-23T10:42:01.813138+0000 (oldest deadline 2024-04-23T10:42:27.113300+0000) 2024-04-23T10:43:13.992 INFO:journalctl@ceph.osd.10.smithi119.stdout:Apr 23 10:43:13 smithi119 ceph-aa240572-0159-11ef-bc93-c7b262605968-osd-10[57454]: 2024-04-23T10:43:13.638+0000 7fb1849bf700 -1 osd.10 116 heartbeat_check: no reply from 172.21.15.119:6804 osd.1 since back 2024-04-23T10:42:10.614599+0000 front 2024-04-23T10:42:10.614622+0000 (oldest deadline 2024-04-23T10:42:34.114881+0000) 2024-04-23T10:43:13.992 INFO:journalctl@ceph.osd.10.smithi119.stdout:Apr 23 10:43:13 smithi119 ceph-aa240572-0159-11ef-bc93-c7b262605968-osd-10[57454]: 2024-04-23T10:43:13.638+0000 7fb1849bf700 -1 osd.10 116 heartbeat_check: no reply from 172.21.15.204:6804 osd.2 since back 2024-04-23T10:42:19.416108+0000 front 2024-04-23T10:42:19.416241+0000 (oldest deadline 2024-04-23T10:42:42.316071+0000) 2024-04-23T10:43:13.992 INFO:journalctl@ceph.osd.10.smithi119.stdout:Apr 23 10:43:13 smithi119 ceph-aa240572-0159-11ef-bc93-c7b262605968-osd-10[57454]: 2024-04-23T10:43:13.638+0000 7fb1849bf700 -1 osd.10 116 heartbeat_check: no reply from 172.21.15.114:6814 osd.3 since back 2024-04-23T10:42:25.217210+0000 front 2024-04-23T10:42:25.217268+0000 (oldest deadline 2024-04-23T10:42:49.917291+0000) 2024-04-23T10:43:13.992 INFO:journalctl@ceph.osd.10.smithi119.stdout:Apr 23 10:43:13 smithi119 ceph-aa240572-0159-11ef-bc93-c7b262605968-osd-10[57454]: 2024-04-23T10:43:13.638+0000 7fb1849bf700 -1 osd.10 116 heartbeat_check: no reply from 172.21.15.119:6812 osd.4 since back 2024-04-23T10:42:36.620525+0000 front 2024-04-23T10:42:36.620567+0000 (oldest deadline 2024-04-23T10:43:00.720710+0000) 2024-04-23T10:43:13.992 INFO:journalctl@ceph.osd.10.smithi119.stdout:Apr 23 10:43:13 smithi119 ceph-aa240572-0159-11ef-bc93-c7b262605968-osd-10[57454]: 2024-04-23T10:43:13.638+0000 7fb1849bf700 -1 osd.10 116 heartbeat_check: no reply from 172.21.15.204:6812 osd.5 since back 2024-04-23T10:42:40.721579+0000 front 2024-04-23T10:42:40.721424+0000 (oldest deadline 2024-04-23T10:43:06.621309+0000) 2024-04-23T10:43:14.328 INFO:journalctl@ceph.osd.11.smithi204.stdout:Apr 23 10:43:13 smithi204 ceph-aa240572-0159-11ef-bc93-c7b262605968-osd-11[57644]: 2024-04-23T10:43:13.866+0000 7fcbc4b63700 -1 osd.11 116 heartbeat_check: no reply from 172.21.15.114:6806 osd.0 since back 2024-04-23T10:42:00.675680+0000 front 2024-04-23T10:42:00.675649+0000 (oldest deadline 2024-04-23T10:42:25.876359+0000) 2024-04-23T10:43:14.328 INFO:journalctl@ceph.osd.11.smithi204.stdout:Apr 23 10:43:13 smithi204 ceph-aa240572-0159-11ef-bc93-c7b262605968-osd-11[57644]: 2024-04-23T10:43:13.866+0000 7fcbc4b63700 -1 osd.11 116 heartbeat_check: no reply from 172.21.15.119:6804 osd.1 since back 2024-04-23T10:42:09.278512+0000 front 2024-04-23T10:42:09.278494+0000 (oldest deadline 2024-04-23T10:42:33.378557+0000) 2024-04-23T10:43:14.328 INFO:journalctl@ceph.osd.11.smithi204.stdout:Apr 23 10:43:13 smithi204 ceph-aa240572-0159-11ef-bc93-c7b262605968-osd-11[57644]: 2024-04-23T10:43:13.866+0000 7fcbc4b63700 -1 osd.11 116 heartbeat_check: no reply from 172.21.15.204:6804 osd.2 since back 2024-04-23T10:42:18.679708+0000 front 2024-04-23T10:42:18.679577+0000 (oldest deadline 2024-04-23T10:42:41.579766+0000) 2024-04-23T10:43:14.328 INFO:journalctl@ceph.osd.11.smithi204.stdout:Apr 23 10:43:13 smithi204 ceph-aa240572-0159-11ef-bc93-c7b262605968-osd-11[57644]: 2024-04-23T10:43:13.866+0000 7fcbc4b63700 -1 osd.11 116 heartbeat_check: no reply from 172.21.15.119:6812 osd.4 since back 2024-04-23T10:42:35.082088+0000 front 2024-04-23T10:42:35.082102+0000 (oldest deadline 2024-04-23T10:42:59.782202+0000) 2024-04-23T10:43:14.328 INFO:journalctl@ceph.osd.11.smithi204.stdout:Apr 23 10:43:13 smithi204 ceph-aa240572-0159-11ef-bc93-c7b262605968-osd-11[57644]: 2024-04-23T10:43:13.866+0000 7fcbc4b63700 -1 osd.11 116 heartbeat_check: no reply from 172.21.15.204:6812 osd.5 since back 2024-04-23T10:42:39.782680+0000 front 2024-04-23T10:42:39.782723+0000 (oldest deadline 2024-04-23T10:43:08.583479+0000) 2024-04-23T10:43:14.992 INFO:journalctl@ceph.osd.10.smithi119.stdout:Apr 23 10:43:14 smithi119 ceph-aa240572-0159-11ef-bc93-c7b262605968-osd-10[57454]: 2024-04-23T10:43:14.654+0000 7fb1849bf700 -1 osd.10 116 heartbeat_check: no reply from 172.21.15.114:6806 osd.0 since back 2024-04-23T10:42:01.813174+0000 front 2024-04-23T10:42:01.813138+0000 (oldest deadline 2024-04-23T10:42:27.113300+0000) 2024-04-23T10:43:14.992 INFO:journalctl@ceph.osd.10.smithi119.stdout:Apr 23 10:43:14 smithi119 ceph-aa240572-0159-11ef-bc93-c7b262605968-osd-10[57454]: 2024-04-23T10:43:14.654+0000 7fb1849bf700 -1 osd.10 116 heartbeat_check: no reply from 172.21.15.119:6804 osd.1 since back 2024-04-23T10:42:10.614599+0000 front 2024-04-23T10:42:10.614622+0000 (oldest deadline 2024-04-23T10:42:34.114881+0000) 2024-04-23T10:43:14.992 INFO:journalctl@ceph.osd.10.smithi119.stdout:Apr 23 10:43:14 smithi119 ceph-aa240572-0159-11ef-bc93-c7b262605968-osd-10[57454]: 2024-04-23T10:43:14.654+0000 7fb1849bf700 -1 osd.10 116 heartbeat_check: no reply from 172.21.15.204:6804 osd.2 since back 2024-04-23T10:42:19.416108+0000 front 2024-04-23T10:42:19.416241+0000 (oldest deadline 2024-04-23T10:42:42.316071+0000) 2024-04-23T10:43:14.992 INFO:journalctl@ceph.osd.10.smithi119.stdout:Apr 23 10:43:14 smithi119 ceph-aa240572-0159-11ef-bc93-c7b262605968-osd-10[57454]: 2024-04-23T10:43:14.654+0000 7fb1849bf700 -1 osd.10 116 heartbeat_check: no reply from 172.21.15.114:6814 osd.3 since back 2024-04-23T10:42:25.217210+0000 front 2024-04-23T10:42:25.217268+0000 (oldest deadline 2024-04-23T10:42:49.917291+0000) 2024-04-23T10:43:14.992 INFO:journalctl@ceph.osd.10.smithi119.stdout:Apr 23 10:43:14 smithi119 ceph-aa240572-0159-11ef-bc93-c7b262605968-osd-10[57454]: 2024-04-23T10:43:14.654+0000 7fb1849bf700 -1 osd.10 116 heartbeat_check: no reply from 172.21.15.119:6812 osd.4 since back 2024-04-23T10:42:36.620525+0000 front 2024-04-23T10:42:36.620567+0000 (oldest deadline 2024-04-23T10:43:00.720710+0000) 2024-04-23T10:43:14.992 INFO:journalctl@ceph.osd.10.smithi119.stdout:Apr 23 10:43:14 smithi119 ceph-aa240572-0159-11ef-bc93-c7b262605968-osd-10[57454]: 2024-04-23T10:43:14.654+0000 7fb1849bf700 -1 osd.10 116 heartbeat_check: no reply from 172.21.15.204:6812 osd.5 since back 2024-04-23T10:42:40.721579+0000 front 2024-04-23T10:42:40.721424+0000 (oldest deadline 2024-04-23T10:43:06.621309+0000) 2024-04-23T10:43:15.004 INFO:journalctl@ceph.osd.9.smithi114.stdout:Apr 23 10:43:14 smithi114 ceph-aa240572-0159-11ef-bc93-c7b262605968-osd-9[62058]: 2024-04-23T10:43:14.536+0000 7feeb4bc7700 -1 osd.9 116 heartbeat_check: no reply from 172.21.15.114:6806 osd.0 since back 2024-04-23T10:42:04.136736+0000 front 2024-04-23T10:42:04.136764+0000 (oldest deadline 2024-04-23T10:42:28.836926+0000) 2024-04-23T10:43:15.004 INFO:journalctl@ceph.osd.9.smithi114.stdout:Apr 23 10:43:14 smithi114 ceph-aa240572-0159-11ef-bc93-c7b262605968-osd-9[62058]: 2024-04-23T10:43:14.536+0000 7feeb4bc7700 -1 osd.9 116 heartbeat_check: no reply from 172.21.15.119:6804 osd.1 since back 2024-04-23T10:42:08.837665+0000 front 2024-04-23T10:42:08.837503+0000 (oldest deadline 2024-04-23T10:42:34.737571+0000) 2024-04-23T10:43:15.004 INFO:journalctl@ceph.osd.9.smithi114.stdout:Apr 23 10:43:14 smithi114 ceph-aa240572-0159-11ef-bc93-c7b262605968-osd-9[62058]: 2024-04-23T10:43:14.536+0000 7feeb4bc7700 -1 osd.9 116 heartbeat_check: no reply from 172.21.15.204:6804 osd.2 since back 2024-04-23T10:42:18.139450+0000 front 2024-04-23T10:42:18.139388+0000 (oldest deadline 2024-04-23T10:42:43.439473+0000) 2024-04-23T10:43:15.004 INFO:journalctl@ceph.osd.9.smithi114.stdout:Apr 23 10:43:14 smithi114 ceph-aa240572-0159-11ef-bc93-c7b262605968-osd-9[62058]: 2024-04-23T10:43:14.536+0000 7feeb4bc7700 -1 osd.9 116 heartbeat_check: no reply from 172.21.15.114:6814 osd.3 since back 2024-04-23T10:42:28.641242+0000 front 2024-04-23T10:42:28.641354+0000 (oldest deadline 2024-04-23T10:42:52.741420+0000) 2024-04-23T10:43:15.004 INFO:journalctl@ceph.osd.9.smithi114.stdout:Apr 23 10:43:14 smithi114 ceph-aa240572-0159-11ef-bc93-c7b262605968-osd-9[62058]: 2024-04-23T10:43:14.536+0000 7feeb4bc7700 -1 osd.9 116 heartbeat_check: no reply from 172.21.15.119:6812 osd.4 since back 2024-04-23T10:42:32.742101+0000 front 2024-04-23T10:42:32.742087+0000 (oldest deadline 2024-04-23T10:42:58.642130+0000) 2024-04-23T10:43:15.004 INFO:journalctl@ceph.osd.9.smithi114.stdout:Apr 23 10:43:14 smithi114 ceph-aa240572-0159-11ef-bc93-c7b262605968-osd-9[62058]: 2024-04-23T10:43:14.536+0000 7feeb4bc7700 -1 osd.9 116 heartbeat_check: no reply from 172.21.15.204:6812 osd.5 since back 2024-04-23T10:42:43.943328+0000 front 2024-04-23T10:42:43.943282+0000 (oldest deadline 2024-04-23T10:43:06.243220+0000) 2024-04-23T10:43:15.005 INFO:journalctl@ceph.osd.9.smithi114.stdout:Apr 23 10:43:14 smithi114 ceph-aa240572-0159-11ef-bc93-c7b262605968-osd-9[62058]: 2024-04-23T10:43:14.536+0000 7feeb4bc7700 -1 osd.9 116 heartbeat_check: no reply from 172.21.15.114:6822 osd.6 since back 2024-04-23T10:42:49.545129+0000 front 2024-04-23T10:42:49.545176+0000 (oldest deadline 2024-04-23T10:43:13.645217+0000) 2024-04-23T10:43:15.329 INFO:journalctl@ceph.osd.11.smithi204.stdout:Apr 23 10:43:14 smithi204 ceph-aa240572-0159-11ef-bc93-c7b262605968-osd-11[57644]: 2024-04-23T10:43:14.888+0000 7fcbc4b63700 -1 osd.11 116 heartbeat_check: no reply from 172.21.15.114:6806 osd.0 since back 2024-04-23T10:42:00.675680+0000 front 2024-04-23T10:42:00.675649+0000 (oldest deadline 2024-04-23T10:42:25.876359+0000) 2024-04-23T10:43:15.329 INFO:journalctl@ceph.osd.11.smithi204.stdout:Apr 23 10:43:14 smithi204 ceph-aa240572-0159-11ef-bc93-c7b262605968-osd-11[57644]: 2024-04-23T10:43:14.888+0000 7fcbc4b63700 -1 osd.11 116 heartbeat_check: no reply from 172.21.15.119:6804 osd.1 since back 2024-04-23T10:42:09.278512+0000 front 2024-04-23T10:42:09.278494+0000 (oldest deadline 2024-04-23T10:42:33.378557+0000) 2024-04-23T10:43:15.329 INFO:journalctl@ceph.osd.11.smithi204.stdout:Apr 23 10:43:14 smithi204 ceph-aa240572-0159-11ef-bc93-c7b262605968-osd-11[57644]: 2024-04-23T10:43:14.888+0000 7fcbc4b63700 -1 osd.11 116 heartbeat_check: no reply from 172.21.15.204:6804 osd.2 since back 2024-04-23T10:42:18.679708+0000 front 2024-04-23T10:42:18.679577+0000 (oldest deadline 2024-04-23T10:42:41.579766+0000) 2024-04-23T10:43:15.330 INFO:journalctl@ceph.osd.11.smithi204.stdout:Apr 23 10:43:14 smithi204 ceph-aa240572-0159-11ef-bc93-c7b262605968-osd-11[57644]: 2024-04-23T10:43:14.888+0000 7fcbc4b63700 -1 osd.11 116 heartbeat_check: no reply from 172.21.15.119:6812 osd.4 since back 2024-04-23T10:42:35.082088+0000 front 2024-04-23T10:42:35.082102+0000 (oldest deadline 2024-04-23T10:42:59.782202+0000) 2024-04-23T10:43:15.330 INFO:journalctl@ceph.osd.11.smithi204.stdout:Apr 23 10:43:14 smithi204 ceph-aa240572-0159-11ef-bc93-c7b262605968-osd-11[57644]: 2024-04-23T10:43:14.888+0000 7fcbc4b63700 -1 osd.11 116 heartbeat_check: no reply from 172.21.15.204:6812 osd.5 since back 2024-04-23T10:42:39.782680+0000 front 2024-04-23T10:42:39.782723+0000 (oldest deadline 2024-04-23T10:43:08.583479+0000) 2024-04-23T10:43:15.330 INFO:journalctl@ceph.osd.11.smithi204.stdout:Apr 23 10:43:14 smithi204 ceph-aa240572-0159-11ef-bc93-c7b262605968-osd-11[57644]: 2024-04-23T10:43:14.888+0000 7fcbc4b63700 -1 osd.11 116 heartbeat_check: no reply from 172.21.15.114:6822 osd.6 since back 2024-04-23T10:42:48.584157+0000 front 2024-04-23T10:42:48.584212+0000 (oldest deadline 2024-04-23T10:43:13.884025+0000) 2024-04-23T10:43:15.993 INFO:journalctl@ceph.osd.10.smithi119.stdout:Apr 23 10:43:15 smithi119 ceph-aa240572-0159-11ef-bc93-c7b262605968-osd-10[57454]: 2024-04-23T10:43:15.659+0000 7fb1849bf700 -1 osd.10 116 heartbeat_check: no reply from 172.21.15.114:6806 osd.0 since back 2024-04-23T10:42:01.813174+0000 front 2024-04-23T10:42:01.813138+0000 (oldest deadline 2024-04-23T10:42:27.113300+0000) 2024-04-23T10:43:15.993 INFO:journalctl@ceph.osd.10.smithi119.stdout:Apr 23 10:43:15 smithi119 ceph-aa240572-0159-11ef-bc93-c7b262605968-osd-10[57454]: 2024-04-23T10:43:15.659+0000 7fb1849bf700 -1 osd.10 116 heartbeat_check: no reply from 172.21.15.119:6804 osd.1 since back 2024-04-23T10:42:10.614599+0000 front 2024-04-23T10:42:10.614622+0000 (oldest deadline 2024-04-23T10:42:34.114881+0000) 2024-04-23T10:43:15.993 INFO:journalctl@ceph.osd.10.smithi119.stdout:Apr 23 10:43:15 smithi119 ceph-aa240572-0159-11ef-bc93-c7b262605968-osd-10[57454]: 2024-04-23T10:43:15.659+0000 7fb1849bf700 -1 osd.10 116 heartbeat_check: no reply from 172.21.15.204:6804 osd.2 since back 2024-04-23T10:42:19.416108+0000 front 2024-04-23T10:42:19.416241+0000 (oldest deadline 2024-04-23T10:42:42.316071+0000) 2024-04-23T10:43:15.993 INFO:journalctl@ceph.osd.10.smithi119.stdout:Apr 23 10:43:15 smithi119 ceph-aa240572-0159-11ef-bc93-c7b262605968-osd-10[57454]: 2024-04-23T10:43:15.659+0000 7fb1849bf700 -1 osd.10 116 heartbeat_check: no reply from 172.21.15.114:6814 osd.3 since back 2024-04-23T10:42:25.217210+0000 front 2024-04-23T10:42:25.217268+0000 (oldest deadline 2024-04-23T10:42:49.917291+0000) 2024-04-23T10:43:15.993 INFO:journalctl@ceph.osd.10.smithi119.stdout:Apr 23 10:43:15 smithi119 ceph-aa240572-0159-11ef-bc93-c7b262605968-osd-10[57454]: 2024-04-23T10:43:15.659+0000 7fb1849bf700 -1 osd.10 116 heartbeat_check: no reply from 172.21.15.119:6812 osd.4 since back 2024-04-23T10:42:36.620525+0000 front 2024-04-23T10:42:36.620567+0000 (oldest deadline 2024-04-23T10:43:00.720710+0000) 2024-04-23T10:43:15.994 INFO:journalctl@ceph.osd.10.smithi119.stdout:Apr 23 10:43:15 smithi119 ceph-aa240572-0159-11ef-bc93-c7b262605968-osd-10[57454]: 2024-04-23T10:43:15.659+0000 7fb1849bf700 -1 osd.10 116 heartbeat_check: no reply from 172.21.15.204:6812 osd.5 since back 2024-04-23T10:42:40.721579+0000 front 2024-04-23T10:42:40.721424+0000 (oldest deadline 2024-04-23T10:43:06.621309+0000) 2024-04-23T10:43:15.994 INFO:journalctl@ceph.osd.10.smithi119.stdout:Apr 23 10:43:15 smithi119 ceph-aa240572-0159-11ef-bc93-c7b262605968-osd-10[57454]: 2024-04-23T10:43:15.659+0000 7fb1849bf700 -1 osd.10 116 heartbeat_check: no reply from 172.21.15.114:6822 osd.6 since back 2024-04-23T10:42:52.423086+0000 front 2024-04-23T10:42:52.422931+0000 (oldest deadline 2024-04-23T10:43:15.322873+0000) 2024-04-23T10:43:16.004 INFO:journalctl@ceph.osd.9.smithi114.stdout:Apr 23 10:43:15 smithi114 ceph-aa240572-0159-11ef-bc93-c7b262605968-osd-9[62058]: 2024-04-23T10:43:15.574+0000 7feeb4bc7700 -1 osd.9 116 heartbeat_check: no reply from 172.21.15.114:6806 osd.0 since back 2024-04-23T10:42:04.136736+0000 front 2024-04-23T10:42:04.136764+0000 (oldest deadline 2024-04-23T10:42:28.836926+0000) 2024-04-23T10:43:16.004 INFO:journalctl@ceph.osd.9.smithi114.stdout:Apr 23 10:43:15 smithi114 ceph-aa240572-0159-11ef-bc93-c7b262605968-osd-9[62058]: 2024-04-23T10:43:15.574+0000 7feeb4bc7700 -1 osd.9 116 heartbeat_check: no reply from 172.21.15.119:6804 osd.1 since back 2024-04-23T10:42:08.837665+0000 front 2024-04-23T10:42:08.837503+0000 (oldest deadline 2024-04-23T10:42:34.737571+0000) 2024-04-23T10:43:16.004 INFO:journalctl@ceph.osd.9.smithi114.stdout:Apr 23 10:43:15 smithi114 ceph-aa240572-0159-11ef-bc93-c7b262605968-osd-9[62058]: 2024-04-23T10:43:15.574+0000 7feeb4bc7700 -1 osd.9 116 heartbeat_check: no reply from 172.21.15.204:6804 osd.2 since back 2024-04-23T10:42:18.139450+0000 front 2024-04-23T10:42:18.139388+0000 (oldest deadline 2024-04-23T10:42:43.439473+0000) 2024-04-23T10:43:16.004 INFO:journalctl@ceph.osd.9.smithi114.stdout:Apr 23 10:43:15 smithi114 ceph-aa240572-0159-11ef-bc93-c7b262605968-osd-9[62058]: 2024-04-23T10:43:15.574+0000 7feeb4bc7700 -1 osd.9 116 heartbeat_check: no reply from 172.21.15.114:6814 osd.3 since back 2024-04-23T10:42:28.641242+0000 front 2024-04-23T10:42:28.641354+0000 (oldest deadline 2024-04-23T10:42:52.741420+0000) 2024-04-23T10:43:16.004 INFO:journalctl@ceph.osd.9.smithi114.stdout:Apr 23 10:43:15 smithi114 ceph-aa240572-0159-11ef-bc93-c7b262605968-osd-9[62058]: 2024-04-23T10:43:15.574+0000 7feeb4bc7700 -1 osd.9 116 heartbeat_check: no reply from 172.21.15.119:6812 osd.4 since back 2024-04-23T10:42:32.742101+0000 front 2024-04-23T10:42:32.742087+0000 (oldest deadline 2024-04-23T10:42:58.642130+0000) 2024-04-23T10:43:16.005 INFO:journalctl@ceph.osd.9.smithi114.stdout:Apr 23 10:43:15 smithi114 ceph-aa240572-0159-11ef-bc93-c7b262605968-osd-9[62058]: 2024-04-23T10:43:15.574+0000 7feeb4bc7700 -1 osd.9 116 heartbeat_check: no reply from 172.21.15.204:6812 osd.5 since back 2024-04-23T10:42:43.943328+0000 front 2024-04-23T10:42:43.943282+0000 (oldest deadline 2024-04-23T10:43:06.243220+0000) 2024-04-23T10:43:16.005 INFO:journalctl@ceph.osd.9.smithi114.stdout:Apr 23 10:43:15 smithi114 ceph-aa240572-0159-11ef-bc93-c7b262605968-osd-9[62058]: 2024-04-23T10:43:15.574+0000 7feeb4bc7700 -1 osd.9 116 heartbeat_check: no reply from 172.21.15.114:6822 osd.6 since back 2024-04-23T10:42:49.545129+0000 front 2024-04-23T10:42:49.545176+0000 (oldest deadline 2024-04-23T10:43:13.645217+0000) 2024-04-23T10:43:16.328 INFO:journalctl@ceph.osd.11.smithi204.stdout:Apr 23 10:43:15 smithi204 ceph-aa240572-0159-11ef-bc93-c7b262605968-osd-11[57644]: 2024-04-23T10:43:15.850+0000 7fcbc4b63700 -1 osd.11 116 heartbeat_check: no reply from 172.21.15.114:6806 osd.0 since back 2024-04-23T10:42:00.675680+0000 front 2024-04-23T10:42:00.675649+0000 (oldest deadline 2024-04-23T10:42:25.876359+0000) 2024-04-23T10:43:16.328 INFO:journalctl@ceph.osd.11.smithi204.stdout:Apr 23 10:43:15 smithi204 ceph-aa240572-0159-11ef-bc93-c7b262605968-osd-11[57644]: 2024-04-23T10:43:15.850+0000 7fcbc4b63700 -1 osd.11 116 heartbeat_check: no reply from 172.21.15.119:6804 osd.1 since back 2024-04-23T10:42:09.278512+0000 front 2024-04-23T10:42:09.278494+0000 (oldest deadline 2024-04-23T10:42:33.378557+0000) 2024-04-23T10:43:16.328 INFO:journalctl@ceph.osd.11.smithi204.stdout:Apr 23 10:43:15 smithi204 ceph-aa240572-0159-11ef-bc93-c7b262605968-osd-11[57644]: 2024-04-23T10:43:15.850+0000 7fcbc4b63700 -1 osd.11 116 heartbeat_check: no reply from 172.21.15.204:6804 osd.2 since back 2024-04-23T10:42:18.679708+0000 front 2024-04-23T10:42:18.679577+0000 (oldest deadline 2024-04-23T10:42:41.579766+0000) 2024-04-23T10:43:16.328 INFO:journalctl@ceph.osd.11.smithi204.stdout:Apr 23 10:43:15 smithi204 ceph-aa240572-0159-11ef-bc93-c7b262605968-osd-11[57644]: 2024-04-23T10:43:15.850+0000 7fcbc4b63700 -1 osd.11 116 heartbeat_check: no reply from 172.21.15.119:6812 osd.4 since back 2024-04-23T10:42:35.082088+0000 front 2024-04-23T10:42:35.082102+0000 (oldest deadline 2024-04-23T10:42:59.782202+0000) 2024-04-23T10:43:16.328 INFO:journalctl@ceph.osd.11.smithi204.stdout:Apr 23 10:43:15 smithi204 ceph-aa240572-0159-11ef-bc93-c7b262605968-osd-11[57644]: 2024-04-23T10:43:15.850+0000 7fcbc4b63700 -1 osd.11 116 heartbeat_check: no reply from 172.21.15.204:6812 osd.5 since back 2024-04-23T10:42:39.782680+0000 front 2024-04-23T10:42:39.782723+0000 (oldest deadline 2024-04-23T10:43:08.583479+0000) 2024-04-23T10:43:16.328 INFO:journalctl@ceph.osd.11.smithi204.stdout:Apr 23 10:43:15 smithi204 ceph-aa240572-0159-11ef-bc93-c7b262605968-osd-11[57644]: 2024-04-23T10:43:15.850+0000 7fcbc4b63700 -1 osd.11 116 heartbeat_check: no reply from 172.21.15.114:6822 osd.6 since back 2024-04-23T10:42:48.584157+0000 front 2024-04-23T10:42:48.584212+0000 (oldest deadline 2024-04-23T10:43:13.884025+0000) 2024-04-23T10:43:16.992 INFO:journalctl@ceph.osd.10.smithi119.stdout:Apr 23 10:43:16 smithi119 ceph-aa240572-0159-11ef-bc93-c7b262605968-osd-10[57454]: 2024-04-23T10:43:16.660+0000 7fb1849bf700 -1 osd.10 116 heartbeat_check: no reply from 172.21.15.114:6806 osd.0 since back 2024-04-23T10:42:01.813174+0000 front 2024-04-23T10:42:01.813138+0000 (oldest deadline 2024-04-23T10:42:27.113300+0000) 2024-04-23T10:43:16.992 INFO:journalctl@ceph.osd.10.smithi119.stdout:Apr 23 10:43:16 smithi119 ceph-aa240572-0159-11ef-bc93-c7b262605968-osd-10[57454]: 2024-04-23T10:43:16.660+0000 7fb1849bf700 -1 osd.10 116 heartbeat_check: no reply from 172.21.15.119:6804 osd.1 since back 2024-04-23T10:42:10.614599+0000 front 2024-04-23T10:42:10.614622+0000 (oldest deadline 2024-04-23T10:42:34.114881+0000) 2024-04-23T10:43:16.992 INFO:journalctl@ceph.osd.10.smithi119.stdout:Apr 23 10:43:16 smithi119 ceph-aa240572-0159-11ef-bc93-c7b262605968-osd-10[57454]: 2024-04-23T10:43:16.660+0000 7fb1849bf700 -1 osd.10 116 heartbeat_check: no reply from 172.21.15.204:6804 osd.2 since back 2024-04-23T10:42:19.416108+0000 front 2024-04-23T10:42:19.416241+0000 (oldest deadline 2024-04-23T10:42:42.316071+0000) 2024-04-23T10:43:16.992 INFO:journalctl@ceph.osd.10.smithi119.stdout:Apr 23 10:43:16 smithi119 ceph-aa240572-0159-11ef-bc93-c7b262605968-osd-10[57454]: 2024-04-23T10:43:16.660+0000 7fb1849bf700 -1 osd.10 116 heartbeat_check: no reply from 172.21.15.114:6814 osd.3 since back 2024-04-23T10:42:25.217210+0000 front 2024-04-23T10:42:25.217268+0000 (oldest deadline 2024-04-23T10:42:49.917291+0000) 2024-04-23T10:43:16.993 INFO:journalctl@ceph.osd.10.smithi119.stdout:Apr 23 10:43:16 smithi119 ceph-aa240572-0159-11ef-bc93-c7b262605968-osd-10[57454]: 2024-04-23T10:43:16.660+0000 7fb1849bf700 -1 osd.10 116 heartbeat_check: no reply from 172.21.15.119:6812 osd.4 since back 2024-04-23T10:42:36.620525+0000 front 2024-04-23T10:42:36.620567+0000 (oldest deadline 2024-04-23T10:43:00.720710+0000) 2024-04-23T10:43:16.993 INFO:journalctl@ceph.osd.10.smithi119.stdout:Apr 23 10:43:16 smithi119 ceph-aa240572-0159-11ef-bc93-c7b262605968-osd-10[57454]: 2024-04-23T10:43:16.660+0000 7fb1849bf700 -1 osd.10 116 heartbeat_check: no reply from 172.21.15.204:6812 osd.5 since back 2024-04-23T10:42:40.721579+0000 front 2024-04-23T10:42:40.721424+0000 (oldest deadline 2024-04-23T10:43:06.621309+0000) 2024-04-23T10:43:16.993 INFO:journalctl@ceph.osd.10.smithi119.stdout:Apr 23 10:43:16 smithi119 ceph-aa240572-0159-11ef-bc93-c7b262605968-osd-10[57454]: 2024-04-23T10:43:16.660+0000 7fb1849bf700 -1 osd.10 116 heartbeat_check: no reply from 172.21.15.114:6822 osd.6 since back 2024-04-23T10:42:52.423086+0000 front 2024-04-23T10:42:52.422931+0000 (oldest deadline 2024-04-23T10:43:15.322873+0000) 2024-04-23T10:43:17.078 INFO:journalctl@ceph.osd.11.smithi204.stdout:Apr 23 10:43:16 smithi204 ceph-aa240572-0159-11ef-bc93-c7b262605968-osd-11[57644]: 2024-04-23T10:43:16.828+0000 7fcbc4b63700 -1 osd.11 116 heartbeat_check: no reply from 172.21.15.114:6806 osd.0 since back 2024-04-23T10:42:00.675680+0000 front 2024-04-23T10:42:00.675649+0000 (oldest deadline 2024-04-23T10:42:25.876359+0000) 2024-04-23T10:43:17.078 INFO:journalctl@ceph.osd.11.smithi204.stdout:Apr 23 10:43:16 smithi204 ceph-aa240572-0159-11ef-bc93-c7b262605968-osd-11[57644]: 2024-04-23T10:43:16.828+0000 7fcbc4b63700 -1 osd.11 116 heartbeat_check: no reply from 172.21.15.119:6804 osd.1 since back 2024-04-23T10:42:09.278512+0000 front 2024-04-23T10:42:09.278494+0000 (oldest deadline 2024-04-23T10:42:33.378557+0000) 2024-04-23T10:43:17.078 INFO:journalctl@ceph.osd.11.smithi204.stdout:Apr 23 10:43:16 smithi204 ceph-aa240572-0159-11ef-bc93-c7b262605968-osd-11[57644]: 2024-04-23T10:43:16.828+0000 7fcbc4b63700 -1 osd.11 116 heartbeat_check: no reply from 172.21.15.204:6804 osd.2 since back 2024-04-23T10:42:18.679708+0000 front 2024-04-23T10:42:18.679577+0000 (oldest deadline 2024-04-23T10:42:41.579766+0000) 2024-04-23T10:43:17.078 INFO:journalctl@ceph.osd.11.smithi204.stdout:Apr 23 10:43:16 smithi204 ceph-aa240572-0159-11ef-bc93-c7b262605968-osd-11[57644]: 2024-04-23T10:43:16.828+0000 7fcbc4b63700 -1 osd.11 116 heartbeat_check: no reply from 172.21.15.119:6812 osd.4 since back 2024-04-23T10:42:35.082088+0000 front 2024-04-23T10:42:35.082102+0000 (oldest deadline 2024-04-23T10:42:59.782202+0000) 2024-04-23T10:43:17.078 INFO:journalctl@ceph.osd.11.smithi204.stdout:Apr 23 10:43:16 smithi204 ceph-aa240572-0159-11ef-bc93-c7b262605968-osd-11[57644]: 2024-04-23T10:43:16.828+0000 7fcbc4b63700 -1 osd.11 116 heartbeat_check: no reply from 172.21.15.204:6812 osd.5 since back 2024-04-23T10:42:39.782680+0000 front 2024-04-23T10:42:39.782723+0000 (oldest deadline 2024-04-23T10:43:08.583479+0000) 2024-04-23T10:43:17.079 INFO:journalctl@ceph.osd.11.smithi204.stdout:Apr 23 10:43:16 smithi204 ceph-aa240572-0159-11ef-bc93-c7b262605968-osd-11[57644]: 2024-04-23T10:43:16.828+0000 7fcbc4b63700 -1 osd.11 116 heartbeat_check: no reply from 172.21.15.114:6822 osd.6 since back 2024-04-23T10:42:48.584157+0000 front 2024-04-23T10:42:48.584212+0000 (oldest deadline 2024-04-23T10:43:13.884025+0000) 2024-04-23T10:43:17.146 INFO:journalctl@ceph.osd.9.smithi114.stdout:Apr 23 10:43:16 smithi114 podman[91570]: 2024-04-23 10:43:16.882564836 +0000 UTC m=+5.598400120 container died 4c9e36e99c22a46cff718163e31f336e690d425cc1ed9fce3d6012c2f78c5597 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:068a558e3af4932cf1c60dd321c0bdf500075eb52ea1bf455e0973ba114fbfd4, name=ceph-aa240572-0159-11ef-bc93-c7b262605968-osd-9, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240416, org.label-schema.name=CentOS Stream 8 Base Image, RELEASE=wip-lusov-quiescer-d69f13d, GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, io.buildah.version=1.33.5, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=) 2024-04-23T10:43:17.413 INFO:journalctl@ceph.osd.9.smithi114.stdout:Apr 23 10:43:17 smithi114 podman[91570]: 2024-04-23 10:43:17.22224203 +0000 UTC m=+5.938077312 container cleanup 4c9e36e99c22a46cff718163e31f336e690d425cc1ed9fce3d6012c2f78c5597 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:068a558e3af4932cf1c60dd321c0bdf500075eb52ea1bf455e0973ba114fbfd4, name=ceph-aa240572-0159-11ef-bc93-c7b262605968-osd-9, RELEASE=wip-lusov-quiescer-d69f13d, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, ceph=True, io.buildah.version=1.33.5, org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, org.label-schema.build-date=20240416, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux , GIT_CLEAN=True) 2024-04-23T10:43:17.413 INFO:journalctl@ceph.osd.9.smithi114.stdout:Apr 23 10:43:17 smithi114 bash[91570]: ceph-aa240572-0159-11ef-bc93-c7b262605968-osd-9 2024-04-23T10:43:17.754 INFO:journalctl@ceph.osd.9.smithi114.stdout:Apr 23 10:43:17 smithi114 podman[91589]: 2024-04-23 10:43:17.41500748 +0000 UTC m=+0.530289835 container remove 4c9e36e99c22a46cff718163e31f336e690d425cc1ed9fce3d6012c2f78c5597 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:068a558e3af4932cf1c60dd321c0bdf500075eb52ea1bf455e0973ba114fbfd4, name=ceph-aa240572-0159-11ef-bc93-c7b262605968-osd-9, GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0, maintainer=Guillaume Abrioux , GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.build-date=20240416, io.buildah.version=1.33.5, RELEASE=wip-lusov-quiescer-d69f13d, CEPH_POINT_RELEASE=, org.label-schema.vendor=CentOS, ceph=True, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_CLEAN=True, org.label-schema.license=GPLv2) 2024-04-23T10:43:17.992 INFO:journalctl@ceph.osd.10.smithi119.stdout:Apr 23 10:43:17 smithi119 ceph-aa240572-0159-11ef-bc93-c7b262605968-osd-10[57454]: 2024-04-23T10:43:17.643+0000 7fb1849bf700 -1 osd.10 116 heartbeat_check: no reply from 172.21.15.114:6806 osd.0 since back 2024-04-23T10:42:01.813174+0000 front 2024-04-23T10:42:01.813138+0000 (oldest deadline 2024-04-23T10:42:27.113300+0000) 2024-04-23T10:43:17.992 INFO:journalctl@ceph.osd.10.smithi119.stdout:Apr 23 10:43:17 smithi119 ceph-aa240572-0159-11ef-bc93-c7b262605968-osd-10[57454]: 2024-04-23T10:43:17.643+0000 7fb1849bf700 -1 osd.10 116 heartbeat_check: no reply from 172.21.15.119:6804 osd.1 since back 2024-04-23T10:42:10.614599+0000 front 2024-04-23T10:42:10.614622+0000 (oldest deadline 2024-04-23T10:42:34.114881+0000) 2024-04-23T10:43:17.993 INFO:journalctl@ceph.osd.10.smithi119.stdout:Apr 23 10:43:17 smithi119 ceph-aa240572-0159-11ef-bc93-c7b262605968-osd-10[57454]: 2024-04-23T10:43:17.643+0000 7fb1849bf700 -1 osd.10 116 heartbeat_check: no reply from 172.21.15.204:6804 osd.2 since back 2024-04-23T10:42:19.416108+0000 front 2024-04-23T10:42:19.416241+0000 (oldest deadline 2024-04-23T10:42:42.316071+0000) 2024-04-23T10:43:17.993 INFO:journalctl@ceph.osd.10.smithi119.stdout:Apr 23 10:43:17 smithi119 ceph-aa240572-0159-11ef-bc93-c7b262605968-osd-10[57454]: 2024-04-23T10:43:17.643+0000 7fb1849bf700 -1 osd.10 116 heartbeat_check: no reply from 172.21.15.114:6814 osd.3 since back 2024-04-23T10:42:25.217210+0000 front 2024-04-23T10:42:25.217268+0000 (oldest deadline 2024-04-23T10:42:49.917291+0000) 2024-04-23T10:43:17.993 INFO:journalctl@ceph.osd.10.smithi119.stdout:Apr 23 10:43:17 smithi119 ceph-aa240572-0159-11ef-bc93-c7b262605968-osd-10[57454]: 2024-04-23T10:43:17.643+0000 7fb1849bf700 -1 osd.10 116 heartbeat_check: no reply from 172.21.15.119:6812 osd.4 since back 2024-04-23T10:42:36.620525+0000 front 2024-04-23T10:42:36.620567+0000 (oldest deadline 2024-04-23T10:43:00.720710+0000) 2024-04-23T10:43:17.993 INFO:journalctl@ceph.osd.10.smithi119.stdout:Apr 23 10:43:17 smithi119 ceph-aa240572-0159-11ef-bc93-c7b262605968-osd-10[57454]: 2024-04-23T10:43:17.643+0000 7fb1849bf700 -1 osd.10 116 heartbeat_check: no reply from 172.21.15.204:6812 osd.5 since back 2024-04-23T10:42:40.721579+0000 front 2024-04-23T10:42:40.721424+0000 (oldest deadline 2024-04-23T10:43:06.621309+0000) 2024-04-23T10:43:17.993 INFO:journalctl@ceph.osd.10.smithi119.stdout:Apr 23 10:43:17 smithi119 ceph-aa240572-0159-11ef-bc93-c7b262605968-osd-10[57454]: 2024-04-23T10:43:17.643+0000 7fb1849bf700 -1 osd.10 116 heartbeat_check: no reply from 172.21.15.114:6822 osd.6 since back 2024-04-23T10:42:52.423086+0000 front 2024-04-23T10:42:52.422931+0000 (oldest deadline 2024-04-23T10:43:15.322873+0000) 2024-04-23T10:43:18.078 INFO:journalctl@ceph.osd.11.smithi204.stdout:Apr 23 10:43:17 smithi204 ceph-aa240572-0159-11ef-bc93-c7b262605968-osd-11[57644]: 2024-04-23T10:43:17.814+0000 7fcbc4b63700 -1 osd.11 116 heartbeat_check: no reply from 172.21.15.114:6806 osd.0 since back 2024-04-23T10:42:00.675680+0000 front 2024-04-23T10:42:00.675649+0000 (oldest deadline 2024-04-23T10:42:25.876359+0000) 2024-04-23T10:43:18.078 INFO:journalctl@ceph.osd.11.smithi204.stdout:Apr 23 10:43:17 smithi204 ceph-aa240572-0159-11ef-bc93-c7b262605968-osd-11[57644]: 2024-04-23T10:43:17.814+0000 7fcbc4b63700 -1 osd.11 116 heartbeat_check: no reply from 172.21.15.119:6804 osd.1 since back 2024-04-23T10:42:09.278512+0000 front 2024-04-23T10:42:09.278494+0000 (oldest deadline 2024-04-23T10:42:33.378557+0000) 2024-04-23T10:43:18.078 INFO:journalctl@ceph.osd.11.smithi204.stdout:Apr 23 10:43:17 smithi204 ceph-aa240572-0159-11ef-bc93-c7b262605968-osd-11[57644]: 2024-04-23T10:43:17.814+0000 7fcbc4b63700 -1 osd.11 116 heartbeat_check: no reply from 172.21.15.204:6804 osd.2 since back 2024-04-23T10:42:18.679708+0000 front 2024-04-23T10:42:18.679577+0000 (oldest deadline 2024-04-23T10:42:41.579766+0000) 2024-04-23T10:43:18.079 INFO:journalctl@ceph.osd.11.smithi204.stdout:Apr 23 10:43:17 smithi204 ceph-aa240572-0159-11ef-bc93-c7b262605968-osd-11[57644]: 2024-04-23T10:43:17.814+0000 7fcbc4b63700 -1 osd.11 116 heartbeat_check: no reply from 172.21.15.119:6812 osd.4 since back 2024-04-23T10:42:35.082088+0000 front 2024-04-23T10:42:35.082102+0000 (oldest deadline 2024-04-23T10:42:59.782202+0000) 2024-04-23T10:43:18.079 INFO:journalctl@ceph.osd.11.smithi204.stdout:Apr 23 10:43:17 smithi204 ceph-aa240572-0159-11ef-bc93-c7b262605968-osd-11[57644]: 2024-04-23T10:43:17.814+0000 7fcbc4b63700 -1 osd.11 116 heartbeat_check: no reply from 172.21.15.204:6812 osd.5 since back 2024-04-23T10:42:39.782680+0000 front 2024-04-23T10:42:39.782723+0000 (oldest deadline 2024-04-23T10:43:08.583479+0000) 2024-04-23T10:43:18.079 INFO:journalctl@ceph.osd.11.smithi204.stdout:Apr 23 10:43:17 smithi204 ceph-aa240572-0159-11ef-bc93-c7b262605968-osd-11[57644]: 2024-04-23T10:43:17.814+0000 7fcbc4b63700 -1 osd.11 116 heartbeat_check: no reply from 172.21.15.114:6822 osd.6 since back 2024-04-23T10:42:48.584157+0000 front 2024-04-23T10:42:48.584212+0000 (oldest deadline 2024-04-23T10:43:13.884025+0000) 2024-04-23T10:43:18.157 INFO:journalctl@ceph.osd.9.smithi114.stdout:Apr 23 10:43:17 smithi114 podman[91680]: 2024-04-23 10:43:17.745345206 +0000 UTC m=+0.018526067 image pull da4b004c09e820713483d039bc76fbd660be678f2ea3c03e059a6161b74dab76 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:068a558e3af4932cf1c60dd321c0bdf500075eb52ea1bf455e0973ba114fbfd4 2024-04-23T10:43:18.158 INFO:journalctl@ceph.osd.9.smithi114.stdout:Apr 23 10:43:17 smithi114 podman[91680]: 2024-04-23 10:43:17.857464556 +0000 UTC m=+0.130645403 container create 2b174416f5df938cb18eee701fb8c2f15b841431012369f2cc4cb8aabe6e65f1 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:068a558e3af4932cf1c60dd321c0bdf500075eb52ea1bf455e0973ba114fbfd4, name=ceph-aa240572-0159-11ef-bc93-c7b262605968-osd-9-deactivate, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=, GIT_CLEAN=True, org.label-schema.schema-version=1.0, org.label-schema.build-date=20240416, RELEASE=wip-lusov-quiescer-d69f13d, io.buildah.version=1.33.5, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 8 Base Image, maintainer=Guillaume Abrioux , ceph=True) 2024-04-23T10:43:18.158 INFO:journalctl@ceph.osd.9.smithi114.stdout:Apr 23 10:43:18 smithi114 podman[91680]: 2024-04-23 10:43:18.068195956 +0000 UTC m=+0.341376804 container init 2b174416f5df938cb18eee701fb8c2f15b841431012369f2cc4cb8aabe6e65f1 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:068a558e3af4932cf1c60dd321c0bdf500075eb52ea1bf455e0973ba114fbfd4, name=ceph-aa240572-0159-11ef-bc93-c7b262605968-osd-9-deactivate, io.buildah.version=1.33.5, org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0, org.label-schema.license=GPLv2, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, org.label-schema.build-date=20240416, GIT_CLEAN=True, RELEASE=wip-lusov-quiescer-d69f13d, maintainer=Guillaume Abrioux , ceph=True, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 8 Base Image, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-04-23T10:43:18.158 INFO:journalctl@ceph.osd.9.smithi114.stdout:Apr 23 10:43:18 smithi114 podman[91680]: 2024-04-23 10:43:18.071264696 +0000 UTC m=+0.344445551 container start 2b174416f5df938cb18eee701fb8c2f15b841431012369f2cc4cb8aabe6e65f1 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:068a558e3af4932cf1c60dd321c0bdf500075eb52ea1bf455e0973ba114fbfd4, name=ceph-aa240572-0159-11ef-bc93-c7b262605968-osd-9-deactivate, RELEASE=wip-lusov-quiescer-d69f13d, org.label-schema.name=CentOS Stream 8 Base Image, CEPH_POINT_RELEASE=, GIT_CLEAN=True, GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS, ceph=True, org.label-schema.license=GPLv2, maintainer=Guillaume Abrioux , io.buildah.version=1.33.5, org.label-schema.build-date=20240416, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-04-23T10:43:18.158 INFO:journalctl@ceph.osd.9.smithi114.stdout:Apr 23 10:43:18 smithi114 podman[91680]: 2024-04-23 10:43:18.159782898 +0000 UTC m=+0.432963757 container attach 2b174416f5df938cb18eee701fb8c2f15b841431012369f2cc4cb8aabe6e65f1 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:068a558e3af4932cf1c60dd321c0bdf500075eb52ea1bf455e0973ba114fbfd4, name=ceph-aa240572-0159-11ef-bc93-c7b262605968-osd-9-deactivate, io.buildah.version=1.33.5, org.label-schema.vendor=CentOS, ceph=True, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_BRANCH=HEAD, RELEASE=wip-lusov-quiescer-d69f13d, GIT_CLEAN=True, org.label-schema.license=GPLv2, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, org.label-schema.build-date=20240416, org.label-schema.name=CentOS Stream 8 Base Image) 2024-04-23T10:43:18.504 INFO:journalctl@ceph.osd.9.smithi114.stdout:Apr 23 10:43:18 smithi114 podman[91680]: 2024-04-23 10:43:18.361917584 +0000 UTC m=+0.635098442 container died 2b174416f5df938cb18eee701fb8c2f15b841431012369f2cc4cb8aabe6e65f1 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:068a558e3af4932cf1c60dd321c0bdf500075eb52ea1bf455e0973ba114fbfd4, name=ceph-aa240572-0159-11ef-bc93-c7b262605968-osd-9-deactivate, GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=, GIT_CLEAN=True, org.label-schema.schema-version=1.0, maintainer=Guillaume Abrioux , io.buildah.version=1.33.5, RELEASE=wip-lusov-quiescer-d69f13d, org.label-schema.vendor=CentOS, org.label-schema.name=CentOS Stream 8 Base Image, GIT_BRANCH=HEAD, ceph=True, org.label-schema.build-date=20240416, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, org.label-schema.license=GPLv2) 2024-04-23T10:43:18.992 INFO:journalctl@ceph.osd.10.smithi119.stdout:Apr 23 10:43:18 smithi119 ceph-aa240572-0159-11ef-bc93-c7b262605968-osd-10[57454]: 2024-04-23T10:43:18.684+0000 7fb1849bf700 -1 osd.10 116 heartbeat_check: no reply from 172.21.15.114:6806 osd.0 since back 2024-04-23T10:42:01.813174+0000 front 2024-04-23T10:42:01.813138+0000 (oldest deadline 2024-04-23T10:42:27.113300+0000) 2024-04-23T10:43:18.992 INFO:journalctl@ceph.osd.10.smithi119.stdout:Apr 23 10:43:18 smithi119 ceph-aa240572-0159-11ef-bc93-c7b262605968-osd-10[57454]: 2024-04-23T10:43:18.684+0000 7fb1849bf700 -1 osd.10 116 heartbeat_check: no reply from 172.21.15.119:6804 osd.1 since back 2024-04-23T10:42:10.614599+0000 front 2024-04-23T10:42:10.614622+0000 (oldest deadline 2024-04-23T10:42:34.114881+0000) 2024-04-23T10:43:18.992 INFO:journalctl@ceph.osd.10.smithi119.stdout:Apr 23 10:43:18 smithi119 ceph-aa240572-0159-11ef-bc93-c7b262605968-osd-10[57454]: 2024-04-23T10:43:18.684+0000 7fb1849bf700 -1 osd.10 116 heartbeat_check: no reply from 172.21.15.204:6804 osd.2 since back 2024-04-23T10:42:19.416108+0000 front 2024-04-23T10:42:19.416241+0000 (oldest deadline 2024-04-23T10:42:42.316071+0000) 2024-04-23T10:43:18.993 INFO:journalctl@ceph.osd.10.smithi119.stdout:Apr 23 10:43:18 smithi119 ceph-aa240572-0159-11ef-bc93-c7b262605968-osd-10[57454]: 2024-04-23T10:43:18.684+0000 7fb1849bf700 -1 osd.10 116 heartbeat_check: no reply from 172.21.15.114:6814 osd.3 since back 2024-04-23T10:42:25.217210+0000 front 2024-04-23T10:42:25.217268+0000 (oldest deadline 2024-04-23T10:42:49.917291+0000) 2024-04-23T10:43:18.993 INFO:journalctl@ceph.osd.10.smithi119.stdout:Apr 23 10:43:18 smithi119 ceph-aa240572-0159-11ef-bc93-c7b262605968-osd-10[57454]: 2024-04-23T10:43:18.684+0000 7fb1849bf700 -1 osd.10 116 heartbeat_check: no reply from 172.21.15.119:6812 osd.4 since back 2024-04-23T10:42:36.620525+0000 front 2024-04-23T10:42:36.620567+0000 (oldest deadline 2024-04-23T10:43:00.720710+0000) 2024-04-23T10:43:18.993 INFO:journalctl@ceph.osd.10.smithi119.stdout:Apr 23 10:43:18 smithi119 ceph-aa240572-0159-11ef-bc93-c7b262605968-osd-10[57454]: 2024-04-23T10:43:18.684+0000 7fb1849bf700 -1 osd.10 116 heartbeat_check: no reply from 172.21.15.204:6812 osd.5 since back 2024-04-23T10:42:40.721579+0000 front 2024-04-23T10:42:40.721424+0000 (oldest deadline 2024-04-23T10:43:06.621309+0000) 2024-04-23T10:43:18.993 INFO:journalctl@ceph.osd.10.smithi119.stdout:Apr 23 10:43:18 smithi119 ceph-aa240572-0159-11ef-bc93-c7b262605968-osd-10[57454]: 2024-04-23T10:43:18.684+0000 7fb1849bf700 -1 osd.10 116 heartbeat_check: no reply from 172.21.15.114:6822 osd.6 since back 2024-04-23T10:42:52.423086+0000 front 2024-04-23T10:42:52.422931+0000 (oldest deadline 2024-04-23T10:43:15.322873+0000) 2024-04-23T10:43:19.038 DEBUG:teuthology.orchestra.run.smithi114:> sudo pkill -f 'journalctl -f -n 0 -u ceph-aa240572-0159-11ef-bc93-c7b262605968@osd.9.service' 2024-04-23T10:43:19.077 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-04-23T10:43:19.078 INFO:tasks.cephadm.osd.9:Stopped osd.9 2024-04-23T10:43:19.078 INFO:tasks.cephadm.osd.10:Stopping osd.10... 2024-04-23T10:43:19.079 DEBUG:teuthology.orchestra.run.smithi119:> sudo systemctl stop ceph-aa240572-0159-11ef-bc93-c7b262605968@osd.10 2024-04-23T10:43:19.328 INFO:journalctl@ceph.osd.11.smithi204.stdout:Apr 23 10:43:18 smithi204 ceph-aa240572-0159-11ef-bc93-c7b262605968-osd-11[57644]: 2024-04-23T10:43:18.863+0000 7fcbc4b63700 -1 osd.11 116 heartbeat_check: no reply from 172.21.15.114:6806 osd.0 since back 2024-04-23T10:42:00.675680+0000 front 2024-04-23T10:42:00.675649+0000 (oldest deadline 2024-04-23T10:42:25.876359+0000) 2024-04-23T10:43:19.328 INFO:journalctl@ceph.osd.11.smithi204.stdout:Apr 23 10:43:18 smithi204 ceph-aa240572-0159-11ef-bc93-c7b262605968-osd-11[57644]: 2024-04-23T10:43:18.863+0000 7fcbc4b63700 -1 osd.11 116 heartbeat_check: no reply from 172.21.15.119:6804 osd.1 since back 2024-04-23T10:42:09.278512+0000 front 2024-04-23T10:42:09.278494+0000 (oldest deadline 2024-04-23T10:42:33.378557+0000) 2024-04-23T10:43:19.328 INFO:journalctl@ceph.osd.11.smithi204.stdout:Apr 23 10:43:18 smithi204 ceph-aa240572-0159-11ef-bc93-c7b262605968-osd-11[57644]: 2024-04-23T10:43:18.863+0000 7fcbc4b63700 -1 osd.11 116 heartbeat_check: no reply from 172.21.15.204:6804 osd.2 since back 2024-04-23T10:42:18.679708+0000 front 2024-04-23T10:42:18.679577+0000 (oldest deadline 2024-04-23T10:42:41.579766+0000) 2024-04-23T10:43:19.328 INFO:journalctl@ceph.osd.11.smithi204.stdout:Apr 23 10:43:18 smithi204 ceph-aa240572-0159-11ef-bc93-c7b262605968-osd-11[57644]: 2024-04-23T10:43:18.863+0000 7fcbc4b63700 -1 osd.11 116 heartbeat_check: no reply from 172.21.15.119:6812 osd.4 since back 2024-04-23T10:42:35.082088+0000 front 2024-04-23T10:42:35.082102+0000 (oldest deadline 2024-04-23T10:42:59.782202+0000) 2024-04-23T10:43:19.329 INFO:journalctl@ceph.osd.11.smithi204.stdout:Apr 23 10:43:18 smithi204 ceph-aa240572-0159-11ef-bc93-c7b262605968-osd-11[57644]: 2024-04-23T10:43:18.863+0000 7fcbc4b63700 -1 osd.11 116 heartbeat_check: no reply from 172.21.15.204:6812 osd.5 since back 2024-04-23T10:42:39.782680+0000 front 2024-04-23T10:42:39.782723+0000 (oldest deadline 2024-04-23T10:43:08.583479+0000) 2024-04-23T10:43:19.329 INFO:journalctl@ceph.osd.11.smithi204.stdout:Apr 23 10:43:18 smithi204 ceph-aa240572-0159-11ef-bc93-c7b262605968-osd-11[57644]: 2024-04-23T10:43:18.863+0000 7fcbc4b63700 -1 osd.11 116 heartbeat_check: no reply from 172.21.15.114:6822 osd.6 since back 2024-04-23T10:42:48.584157+0000 front 2024-04-23T10:42:48.584212+0000 (oldest deadline 2024-04-23T10:43:13.884025+0000) 2024-04-23T10:43:19.373 INFO:journalctl@ceph.osd.10.smithi119.stdout:Apr 23 10:43:19 smithi119 systemd[1]: Stopping Ceph osd.10 for aa240572-0159-11ef-bc93-c7b262605968... 2024-04-23T10:43:19.668 INFO:journalctl@ceph.osd.10.smithi119.stdout:Apr 23 10:43:19 smithi119 ceph-aa240572-0159-11ef-bc93-c7b262605968-osd-10[57454]: 2024-04-23T10:43:19.375+0000 7fb1883cb700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.10 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-04-23T10:43:19.668 INFO:journalctl@ceph.osd.10.smithi119.stdout:Apr 23 10:43:19 smithi119 ceph-aa240572-0159-11ef-bc93-c7b262605968-osd-10[57454]: 2024-04-23T10:43:19.375+0000 7fb1883cb700 -1 osd.10 116 *** Got signal Terminated *** 2024-04-23T10:43:19.669 INFO:journalctl@ceph.osd.10.smithi119.stdout:Apr 23 10:43:19 smithi119 ceph-aa240572-0159-11ef-bc93-c7b262605968-osd-10[57454]: 2024-04-23T10:43:19.375+0000 7fb1883cb700 -1 osd.10 116 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-04-23T10:43:19.669 INFO:journalctl@ceph.osd.10.smithi119.stdout:Apr 23 10:43:19 smithi119 ceph-aa240572-0159-11ef-bc93-c7b262605968-osd-10[57454]: 2024-04-23T10:43:19.669+0000 7fb1849bf700 -1 osd.10 116 heartbeat_check: no reply from 172.21.15.114:6806 osd.0 since back 2024-04-23T10:42:01.813174+0000 front 2024-04-23T10:42:01.813138+0000 (oldest deadline 2024-04-23T10:42:27.113300+0000) 2024-04-23T10:43:19.669 INFO:journalctl@ceph.osd.10.smithi119.stdout:Apr 23 10:43:19 smithi119 ceph-aa240572-0159-11ef-bc93-c7b262605968-osd-10[57454]: 2024-04-23T10:43:19.669+0000 7fb1849bf700 -1 osd.10 116 heartbeat_check: no reply from 172.21.15.119:6804 osd.1 since back 2024-04-23T10:42:10.614599+0000 front 2024-04-23T10:42:10.614622+0000 (oldest deadline 2024-04-23T10:42:34.114881+0000) 2024-04-23T10:43:19.669 INFO:journalctl@ceph.osd.10.smithi119.stdout:Apr 23 10:43:19 smithi119 ceph-aa240572-0159-11ef-bc93-c7b262605968-osd-10[57454]: 2024-04-23T10:43:19.669+0000 7fb1849bf700 -1 osd.10 116 heartbeat_check: no reply from 172.21.15.204:6804 osd.2 since back 2024-04-23T10:42:19.416108+0000 front 2024-04-23T10:42:19.416241+0000 (oldest deadline 2024-04-23T10:42:42.316071+0000) 2024-04-23T10:43:19.669 INFO:journalctl@ceph.osd.10.smithi119.stdout:Apr 23 10:43:19 smithi119 ceph-aa240572-0159-11ef-bc93-c7b262605968-osd-10[57454]: 2024-04-23T10:43:19.669+0000 7fb1849bf700 -1 osd.10 116 heartbeat_check: no reply from 172.21.15.114:6814 osd.3 since back 2024-04-23T10:42:25.217210+0000 front 2024-04-23T10:42:25.217268+0000 (oldest deadline 2024-04-23T10:42:49.917291+0000) 2024-04-23T10:43:19.670 INFO:journalctl@ceph.osd.10.smithi119.stdout:Apr 23 10:43:19 smithi119 ceph-aa240572-0159-11ef-bc93-c7b262605968-osd-10[57454]: 2024-04-23T10:43:19.669+0000 7fb1849bf700 -1 osd.10 116 heartbeat_check: no reply from 172.21.15.119:6812 osd.4 since back 2024-04-23T10:42:36.620525+0000 front 2024-04-23T10:42:36.620567+0000 (oldest deadline 2024-04-23T10:43:00.720710+0000) 2024-04-23T10:43:19.670 INFO:journalctl@ceph.osd.10.smithi119.stdout:Apr 23 10:43:19 smithi119 ceph-aa240572-0159-11ef-bc93-c7b262605968-osd-10[57454]: 2024-04-23T10:43:19.669+0000 7fb1849bf700 -1 osd.10 116 heartbeat_check: no reply from 172.21.15.204:6812 osd.5 since back 2024-04-23T10:42:40.721579+0000 front 2024-04-23T10:42:40.721424+0000 (oldest deadline 2024-04-23T10:43:06.621309+0000) 2024-04-23T10:43:19.670 INFO:journalctl@ceph.osd.10.smithi119.stdout:Apr 23 10:43:19 smithi119 ceph-aa240572-0159-11ef-bc93-c7b262605968-osd-10[57454]: 2024-04-23T10:43:19.669+0000 7fb1849bf700 -1 osd.10 116 heartbeat_check: no reply from 172.21.15.114:6822 osd.6 since back 2024-04-23T10:42:52.423086+0000 front 2024-04-23T10:42:52.422931+0000 (oldest deadline 2024-04-23T10:43:15.322873+0000) 2024-04-23T10:43:20.078 INFO:journalctl@ceph.osd.11.smithi204.stdout:Apr 23 10:43:19 smithi204 ceph-aa240572-0159-11ef-bc93-c7b262605968-osd-11[57644]: 2024-04-23T10:43:19.821+0000 7fcbc4b63700 -1 osd.11 116 heartbeat_check: no reply from 172.21.15.114:6806 osd.0 since back 2024-04-23T10:42:00.675680+0000 front 2024-04-23T10:42:00.675649+0000 (oldest deadline 2024-04-23T10:42:25.876359+0000) 2024-04-23T10:43:20.079 INFO:journalctl@ceph.osd.11.smithi204.stdout:Apr 23 10:43:19 smithi204 ceph-aa240572-0159-11ef-bc93-c7b262605968-osd-11[57644]: 2024-04-23T10:43:19.821+0000 7fcbc4b63700 -1 osd.11 116 heartbeat_check: no reply from 172.21.15.119:6804 osd.1 since back 2024-04-23T10:42:09.278512+0000 front 2024-04-23T10:42:09.278494+0000 (oldest deadline 2024-04-23T10:42:33.378557+0000) 2024-04-23T10:43:20.079 INFO:journalctl@ceph.osd.11.smithi204.stdout:Apr 23 10:43:19 smithi204 ceph-aa240572-0159-11ef-bc93-c7b262605968-osd-11[57644]: 2024-04-23T10:43:19.821+0000 7fcbc4b63700 -1 osd.11 116 heartbeat_check: no reply from 172.21.15.204:6804 osd.2 since back 2024-04-23T10:42:18.679708+0000 front 2024-04-23T10:42:18.679577+0000 (oldest deadline 2024-04-23T10:42:41.579766+0000) 2024-04-23T10:43:20.079 INFO:journalctl@ceph.osd.11.smithi204.stdout:Apr 23 10:43:19 smithi204 ceph-aa240572-0159-11ef-bc93-c7b262605968-osd-11[57644]: 2024-04-23T10:43:19.821+0000 7fcbc4b63700 -1 osd.11 116 heartbeat_check: no reply from 172.21.15.119:6812 osd.4 since back 2024-04-23T10:42:35.082088+0000 front 2024-04-23T10:42:35.082102+0000 (oldest deadline 2024-04-23T10:42:59.782202+0000) 2024-04-23T10:43:20.079 INFO:journalctl@ceph.osd.11.smithi204.stdout:Apr 23 10:43:19 smithi204 ceph-aa240572-0159-11ef-bc93-c7b262605968-osd-11[57644]: 2024-04-23T10:43:19.821+0000 7fcbc4b63700 -1 osd.11 116 heartbeat_check: no reply from 172.21.15.204:6812 osd.5 since back 2024-04-23T10:42:39.782680+0000 front 2024-04-23T10:42:39.782723+0000 (oldest deadline 2024-04-23T10:43:08.583479+0000) 2024-04-23T10:43:20.080 INFO:journalctl@ceph.osd.11.smithi204.stdout:Apr 23 10:43:19 smithi204 ceph-aa240572-0159-11ef-bc93-c7b262605968-osd-11[57644]: 2024-04-23T10:43:19.821+0000 7fcbc4b63700 -1 osd.11 116 heartbeat_check: no reply from 172.21.15.114:6822 osd.6 since back 2024-04-23T10:42:48.584157+0000 front 2024-04-23T10:42:48.584212+0000 (oldest deadline 2024-04-23T10:43:13.884025+0000) 2024-04-23T10:43:20.992 INFO:journalctl@ceph.osd.10.smithi119.stdout:Apr 23 10:43:20 smithi119 ceph-aa240572-0159-11ef-bc93-c7b262605968-osd-10[57454]: 2024-04-23T10:43:20.637+0000 7fb1849bf700 -1 osd.10 116 heartbeat_check: no reply from 172.21.15.114:6806 osd.0 since back 2024-04-23T10:42:01.813174+0000 front 2024-04-23T10:42:01.813138+0000 (oldest deadline 2024-04-23T10:42:27.113300+0000) 2024-04-23T10:43:20.993 INFO:journalctl@ceph.osd.10.smithi119.stdout:Apr 23 10:43:20 smithi119 ceph-aa240572-0159-11ef-bc93-c7b262605968-osd-10[57454]: 2024-04-23T10:43:20.637+0000 7fb1849bf700 -1 osd.10 116 heartbeat_check: no reply from 172.21.15.119:6804 osd.1 since back 2024-04-23T10:42:10.614599+0000 front 2024-04-23T10:42:10.614622+0000 (oldest deadline 2024-04-23T10:42:34.114881+0000) 2024-04-23T10:43:20.993 INFO:journalctl@ceph.osd.10.smithi119.stdout:Apr 23 10:43:20 smithi119 ceph-aa240572-0159-11ef-bc93-c7b262605968-osd-10[57454]: 2024-04-23T10:43:20.637+0000 7fb1849bf700 -1 osd.10 116 heartbeat_check: no reply from 172.21.15.204:6804 osd.2 since back 2024-04-23T10:42:19.416108+0000 front 2024-04-23T10:42:19.416241+0000 (oldest deadline 2024-04-23T10:42:42.316071+0000) 2024-04-23T10:43:20.993 INFO:journalctl@ceph.osd.10.smithi119.stdout:Apr 23 10:43:20 smithi119 ceph-aa240572-0159-11ef-bc93-c7b262605968-osd-10[57454]: 2024-04-23T10:43:20.637+0000 7fb1849bf700 -1 osd.10 116 heartbeat_check: no reply from 172.21.15.114:6814 osd.3 since back 2024-04-23T10:42:25.217210+0000 front 2024-04-23T10:42:25.217268+0000 (oldest deadline 2024-04-23T10:42:49.917291+0000) 2024-04-23T10:43:20.993 INFO:journalctl@ceph.osd.10.smithi119.stdout:Apr 23 10:43:20 smithi119 ceph-aa240572-0159-11ef-bc93-c7b262605968-osd-10[57454]: 2024-04-23T10:43:20.637+0000 7fb1849bf700 -1 osd.10 116 heartbeat_check: no reply from 172.21.15.119:6812 osd.4 since back 2024-04-23T10:42:36.620525+0000 front 2024-04-23T10:42:36.620567+0000 (oldest deadline 2024-04-23T10:43:00.720710+0000) 2024-04-23T10:43:20.993 INFO:journalctl@ceph.osd.10.smithi119.stdout:Apr 23 10:43:20 smithi119 ceph-aa240572-0159-11ef-bc93-c7b262605968-osd-10[57454]: 2024-04-23T10:43:20.637+0000 7fb1849bf700 -1 osd.10 116 heartbeat_check: no reply from 172.21.15.204:6812 osd.5 since back 2024-04-23T10:42:40.721579+0000 front 2024-04-23T10:42:40.721424+0000 (oldest deadline 2024-04-23T10:43:06.621309+0000) 2024-04-23T10:43:20.993 INFO:journalctl@ceph.osd.10.smithi119.stdout:Apr 23 10:43:20 smithi119 ceph-aa240572-0159-11ef-bc93-c7b262605968-osd-10[57454]: 2024-04-23T10:43:20.637+0000 7fb1849bf700 -1 osd.10 116 heartbeat_check: no reply from 172.21.15.114:6822 osd.6 since back 2024-04-23T10:42:52.423086+0000 front 2024-04-23T10:42:52.422931+0000 (oldest deadline 2024-04-23T10:43:15.322873+0000) 2024-04-23T10:43:21.093 INFO:journalctl@ceph.osd.11.smithi204.stdout:Apr 23 10:43:20 smithi204 ceph-aa240572-0159-11ef-bc93-c7b262605968-osd-11[57644]: 2024-04-23T10:43:20.844+0000 7fcbc4b63700 -1 osd.11 116 heartbeat_check: no reply from 172.21.15.114:6806 osd.0 since back 2024-04-23T10:42:00.675680+0000 front 2024-04-23T10:42:00.675649+0000 (oldest deadline 2024-04-23T10:42:25.876359+0000) 2024-04-23T10:43:21.093 INFO:journalctl@ceph.osd.11.smithi204.stdout:Apr 23 10:43:20 smithi204 ceph-aa240572-0159-11ef-bc93-c7b262605968-osd-11[57644]: 2024-04-23T10:43:20.844+0000 7fcbc4b63700 -1 osd.11 116 heartbeat_check: no reply from 172.21.15.119:6804 osd.1 since back 2024-04-23T10:42:09.278512+0000 front 2024-04-23T10:42:09.278494+0000 (oldest deadline 2024-04-23T10:42:33.378557+0000) 2024-04-23T10:43:21.094 INFO:journalctl@ceph.osd.11.smithi204.stdout:Apr 23 10:43:20 smithi204 ceph-aa240572-0159-11ef-bc93-c7b262605968-osd-11[57644]: 2024-04-23T10:43:20.844+0000 7fcbc4b63700 -1 osd.11 116 heartbeat_check: no reply from 172.21.15.204:6804 osd.2 since back 2024-04-23T10:42:18.679708+0000 front 2024-04-23T10:42:18.679577+0000 (oldest deadline 2024-04-23T10:42:41.579766+0000) 2024-04-23T10:43:21.094 INFO:journalctl@ceph.osd.11.smithi204.stdout:Apr 23 10:43:20 smithi204 ceph-aa240572-0159-11ef-bc93-c7b262605968-osd-11[57644]: 2024-04-23T10:43:20.844+0000 7fcbc4b63700 -1 osd.11 116 heartbeat_check: no reply from 172.21.15.119:6812 osd.4 since back 2024-04-23T10:42:35.082088+0000 front 2024-04-23T10:42:35.082102+0000 (oldest deadline 2024-04-23T10:42:59.782202+0000) 2024-04-23T10:43:21.094 INFO:journalctl@ceph.osd.11.smithi204.stdout:Apr 23 10:43:20 smithi204 ceph-aa240572-0159-11ef-bc93-c7b262605968-osd-11[57644]: 2024-04-23T10:43:20.844+0000 7fcbc4b63700 -1 osd.11 116 heartbeat_check: no reply from 172.21.15.204:6812 osd.5 since back 2024-04-23T10:42:39.782680+0000 front 2024-04-23T10:42:39.782723+0000 (oldest deadline 2024-04-23T10:43:08.583479+0000) 2024-04-23T10:43:21.094 INFO:journalctl@ceph.osd.11.smithi204.stdout:Apr 23 10:43:20 smithi204 ceph-aa240572-0159-11ef-bc93-c7b262605968-osd-11[57644]: 2024-04-23T10:43:20.844+0000 7fcbc4b63700 -1 osd.11 116 heartbeat_check: no reply from 172.21.15.114:6822 osd.6 since back 2024-04-23T10:42:48.584157+0000 front 2024-04-23T10:42:48.584212+0000 (oldest deadline 2024-04-23T10:43:13.884025+0000) 2024-04-23T10:43:21.992 INFO:journalctl@ceph.osd.10.smithi119.stdout:Apr 23 10:43:21 smithi119 ceph-aa240572-0159-11ef-bc93-c7b262605968-osd-10[57454]: 2024-04-23T10:43:21.623+0000 7fb1849bf700 -1 osd.10 116 heartbeat_check: no reply from 172.21.15.114:6806 osd.0 since back 2024-04-23T10:42:01.813174+0000 front 2024-04-23T10:42:01.813138+0000 (oldest deadline 2024-04-23T10:42:27.113300+0000) 2024-04-23T10:43:21.992 INFO:journalctl@ceph.osd.10.smithi119.stdout:Apr 23 10:43:21 smithi119 ceph-aa240572-0159-11ef-bc93-c7b262605968-osd-10[57454]: 2024-04-23T10:43:21.623+0000 7fb1849bf700 -1 osd.10 116 heartbeat_check: no reply from 172.21.15.119:6804 osd.1 since back 2024-04-23T10:42:10.614599+0000 front 2024-04-23T10:42:10.614622+0000 (oldest deadline 2024-04-23T10:42:34.114881+0000) 2024-04-23T10:43:21.992 INFO:journalctl@ceph.osd.10.smithi119.stdout:Apr 23 10:43:21 smithi119 ceph-aa240572-0159-11ef-bc93-c7b262605968-osd-10[57454]: 2024-04-23T10:43:21.623+0000 7fb1849bf700 -1 osd.10 116 heartbeat_check: no reply from 172.21.15.204:6804 osd.2 since back 2024-04-23T10:42:19.416108+0000 front 2024-04-23T10:42:19.416241+0000 (oldest deadline 2024-04-23T10:42:42.316071+0000) 2024-04-23T10:43:21.992 INFO:journalctl@ceph.osd.10.smithi119.stdout:Apr 23 10:43:21 smithi119 ceph-aa240572-0159-11ef-bc93-c7b262605968-osd-10[57454]: 2024-04-23T10:43:21.623+0000 7fb1849bf700 -1 osd.10 116 heartbeat_check: no reply from 172.21.15.114:6814 osd.3 since back 2024-04-23T10:42:25.217210+0000 front 2024-04-23T10:42:25.217268+0000 (oldest deadline 2024-04-23T10:42:49.917291+0000) 2024-04-23T10:43:21.993 INFO:journalctl@ceph.osd.10.smithi119.stdout:Apr 23 10:43:21 smithi119 ceph-aa240572-0159-11ef-bc93-c7b262605968-osd-10[57454]: 2024-04-23T10:43:21.623+0000 7fb1849bf700 -1 osd.10 116 heartbeat_check: no reply from 172.21.15.119:6812 osd.4 since back 2024-04-23T10:42:36.620525+0000 front 2024-04-23T10:42:36.620567+0000 (oldest deadline 2024-04-23T10:43:00.720710+0000) 2024-04-23T10:43:21.993 INFO:journalctl@ceph.osd.10.smithi119.stdout:Apr 23 10:43:21 smithi119 ceph-aa240572-0159-11ef-bc93-c7b262605968-osd-10[57454]: 2024-04-23T10:43:21.623+0000 7fb1849bf700 -1 osd.10 116 heartbeat_check: no reply from 172.21.15.204:6812 osd.5 since back 2024-04-23T10:42:40.721579+0000 front 2024-04-23T10:42:40.721424+0000 (oldest deadline 2024-04-23T10:43:06.621309+0000) 2024-04-23T10:43:21.993 INFO:journalctl@ceph.osd.10.smithi119.stdout:Apr 23 10:43:21 smithi119 ceph-aa240572-0159-11ef-bc93-c7b262605968-osd-10[57454]: 2024-04-23T10:43:21.623+0000 7fb1849bf700 -1 osd.10 116 heartbeat_check: no reply from 172.21.15.114:6822 osd.6 since back 2024-04-23T10:42:52.423086+0000 front 2024-04-23T10:42:52.422931+0000 (oldest deadline 2024-04-23T10:43:15.322873+0000) 2024-04-23T10:43:22.078 INFO:journalctl@ceph.osd.11.smithi204.stdout:Apr 23 10:43:21 smithi204 ceph-aa240572-0159-11ef-bc93-c7b262605968-osd-11[57644]: 2024-04-23T10:43:21.810+0000 7fcbc4b63700 -1 osd.11 116 heartbeat_check: no reply from 172.21.15.114:6806 osd.0 since back 2024-04-23T10:42:00.675680+0000 front 2024-04-23T10:42:00.675649+0000 (oldest deadline 2024-04-23T10:42:25.876359+0000) 2024-04-23T10:43:22.078 INFO:journalctl@ceph.osd.11.smithi204.stdout:Apr 23 10:43:21 smithi204 ceph-aa240572-0159-11ef-bc93-c7b262605968-osd-11[57644]: 2024-04-23T10:43:21.810+0000 7fcbc4b63700 -1 osd.11 116 heartbeat_check: no reply from 172.21.15.119:6804 osd.1 since back 2024-04-23T10:42:09.278512+0000 front 2024-04-23T10:42:09.278494+0000 (oldest deadline 2024-04-23T10:42:33.378557+0000) 2024-04-23T10:43:22.078 INFO:journalctl@ceph.osd.11.smithi204.stdout:Apr 23 10:43:21 smithi204 ceph-aa240572-0159-11ef-bc93-c7b262605968-osd-11[57644]: 2024-04-23T10:43:21.810+0000 7fcbc4b63700 -1 osd.11 116 heartbeat_check: no reply from 172.21.15.204:6804 osd.2 since back 2024-04-23T10:42:18.679708+0000 front 2024-04-23T10:42:18.679577+0000 (oldest deadline 2024-04-23T10:42:41.579766+0000) 2024-04-23T10:43:22.078 INFO:journalctl@ceph.osd.11.smithi204.stdout:Apr 23 10:43:21 smithi204 ceph-aa240572-0159-11ef-bc93-c7b262605968-osd-11[57644]: 2024-04-23T10:43:21.810+0000 7fcbc4b63700 -1 osd.11 116 heartbeat_check: no reply from 172.21.15.119:6812 osd.4 since back 2024-04-23T10:42:35.082088+0000 front 2024-04-23T10:42:35.082102+0000 (oldest deadline 2024-04-23T10:42:59.782202+0000) 2024-04-23T10:43:22.078 INFO:journalctl@ceph.osd.11.smithi204.stdout:Apr 23 10:43:21 smithi204 ceph-aa240572-0159-11ef-bc93-c7b262605968-osd-11[57644]: 2024-04-23T10:43:21.810+0000 7fcbc4b63700 -1 osd.11 116 heartbeat_check: no reply from 172.21.15.204:6812 osd.5 since back 2024-04-23T10:42:39.782680+0000 front 2024-04-23T10:42:39.782723+0000 (oldest deadline 2024-04-23T10:43:08.583479+0000) 2024-04-23T10:43:22.078 INFO:journalctl@ceph.osd.11.smithi204.stdout:Apr 23 10:43:21 smithi204 ceph-aa240572-0159-11ef-bc93-c7b262605968-osd-11[57644]: 2024-04-23T10:43:21.810+0000 7fcbc4b63700 -1 osd.11 116 heartbeat_check: no reply from 172.21.15.114:6822 osd.6 since back 2024-04-23T10:42:48.584157+0000 front 2024-04-23T10:42:48.584212+0000 (oldest deadline 2024-04-23T10:43:13.884025+0000) 2024-04-23T10:43:22.992 INFO:journalctl@ceph.osd.10.smithi119.stdout:Apr 23 10:43:22 smithi119 ceph-aa240572-0159-11ef-bc93-c7b262605968-osd-10[57454]: 2024-04-23T10:43:22.620+0000 7fb1849bf700 -1 osd.10 116 heartbeat_check: no reply from 172.21.15.114:6806 osd.0 since back 2024-04-23T10:42:01.813174+0000 front 2024-04-23T10:42:01.813138+0000 (oldest deadline 2024-04-23T10:42:27.113300+0000) 2024-04-23T10:43:22.992 INFO:journalctl@ceph.osd.10.smithi119.stdout:Apr 23 10:43:22 smithi119 ceph-aa240572-0159-11ef-bc93-c7b262605968-osd-10[57454]: 2024-04-23T10:43:22.620+0000 7fb1849bf700 -1 osd.10 116 heartbeat_check: no reply from 172.21.15.119:6804 osd.1 since back 2024-04-23T10:42:10.614599+0000 front 2024-04-23T10:42:10.614622+0000 (oldest deadline 2024-04-23T10:42:34.114881+0000) 2024-04-23T10:43:22.992 INFO:journalctl@ceph.osd.10.smithi119.stdout:Apr 23 10:43:22 smithi119 ceph-aa240572-0159-11ef-bc93-c7b262605968-osd-10[57454]: 2024-04-23T10:43:22.620+0000 7fb1849bf700 -1 osd.10 116 heartbeat_check: no reply from 172.21.15.204:6804 osd.2 since back 2024-04-23T10:42:19.416108+0000 front 2024-04-23T10:42:19.416241+0000 (oldest deadline 2024-04-23T10:42:42.316071+0000) 2024-04-23T10:43:22.992 INFO:journalctl@ceph.osd.10.smithi119.stdout:Apr 23 10:43:22 smithi119 ceph-aa240572-0159-11ef-bc93-c7b262605968-osd-10[57454]: 2024-04-23T10:43:22.620+0000 7fb1849bf700 -1 osd.10 116 heartbeat_check: no reply from 172.21.15.114:6814 osd.3 since back 2024-04-23T10:42:25.217210+0000 front 2024-04-23T10:42:25.217268+0000 (oldest deadline 2024-04-23T10:42:49.917291+0000) 2024-04-23T10:43:22.992 INFO:journalctl@ceph.osd.10.smithi119.stdout:Apr 23 10:43:22 smithi119 ceph-aa240572-0159-11ef-bc93-c7b262605968-osd-10[57454]: 2024-04-23T10:43:22.620+0000 7fb1849bf700 -1 osd.10 116 heartbeat_check: no reply from 172.21.15.119:6812 osd.4 since back 2024-04-23T10:42:36.620525+0000 front 2024-04-23T10:42:36.620567+0000 (oldest deadline 2024-04-23T10:43:00.720710+0000) 2024-04-23T10:43:22.993 INFO:journalctl@ceph.osd.10.smithi119.stdout:Apr 23 10:43:22 smithi119 ceph-aa240572-0159-11ef-bc93-c7b262605968-osd-10[57454]: 2024-04-23T10:43:22.620+0000 7fb1849bf700 -1 osd.10 116 heartbeat_check: no reply from 172.21.15.204:6812 osd.5 since back 2024-04-23T10:42:40.721579+0000 front 2024-04-23T10:42:40.721424+0000 (oldest deadline 2024-04-23T10:43:06.621309+0000) 2024-04-23T10:43:22.993 INFO:journalctl@ceph.osd.10.smithi119.stdout:Apr 23 10:43:22 smithi119 ceph-aa240572-0159-11ef-bc93-c7b262605968-osd-10[57454]: 2024-04-23T10:43:22.620+0000 7fb1849bf700 -1 osd.10 116 heartbeat_check: no reply from 172.21.15.114:6822 osd.6 since back 2024-04-23T10:42:52.423086+0000 front 2024-04-23T10:42:52.422931+0000 (oldest deadline 2024-04-23T10:43:15.322873+0000) 2024-04-23T10:43:23.328 INFO:journalctl@ceph.osd.11.smithi204.stdout:Apr 23 10:43:22 smithi204 ceph-aa240572-0159-11ef-bc93-c7b262605968-osd-11[57644]: 2024-04-23T10:43:22.843+0000 7fcbc4b63700 -1 osd.11 116 heartbeat_check: no reply from 172.21.15.114:6806 osd.0 since back 2024-04-23T10:42:00.675680+0000 front 2024-04-23T10:42:00.675649+0000 (oldest deadline 2024-04-23T10:42:25.876359+0000) 2024-04-23T10:43:23.328 INFO:journalctl@ceph.osd.11.smithi204.stdout:Apr 23 10:43:22 smithi204 ceph-aa240572-0159-11ef-bc93-c7b262605968-osd-11[57644]: 2024-04-23T10:43:22.843+0000 7fcbc4b63700 -1 osd.11 116 heartbeat_check: no reply from 172.21.15.119:6804 osd.1 since back 2024-04-23T10:42:09.278512+0000 front 2024-04-23T10:42:09.278494+0000 (oldest deadline 2024-04-23T10:42:33.378557+0000) 2024-04-23T10:43:23.328 INFO:journalctl@ceph.osd.11.smithi204.stdout:Apr 23 10:43:22 smithi204 ceph-aa240572-0159-11ef-bc93-c7b262605968-osd-11[57644]: 2024-04-23T10:43:22.843+0000 7fcbc4b63700 -1 osd.11 116 heartbeat_check: no reply from 172.21.15.204:6804 osd.2 since back 2024-04-23T10:42:18.679708+0000 front 2024-04-23T10:42:18.679577+0000 (oldest deadline 2024-04-23T10:42:41.579766+0000) 2024-04-23T10:43:23.328 INFO:journalctl@ceph.osd.11.smithi204.stdout:Apr 23 10:43:22 smithi204 ceph-aa240572-0159-11ef-bc93-c7b262605968-osd-11[57644]: 2024-04-23T10:43:22.843+0000 7fcbc4b63700 -1 osd.11 116 heartbeat_check: no reply from 172.21.15.119:6812 osd.4 since back 2024-04-23T10:42:35.082088+0000 front 2024-04-23T10:42:35.082102+0000 (oldest deadline 2024-04-23T10:42:59.782202+0000) 2024-04-23T10:43:23.329 INFO:journalctl@ceph.osd.11.smithi204.stdout:Apr 23 10:43:22 smithi204 ceph-aa240572-0159-11ef-bc93-c7b262605968-osd-11[57644]: 2024-04-23T10:43:22.843+0000 7fcbc4b63700 -1 osd.11 116 heartbeat_check: no reply from 172.21.15.204:6812 osd.5 since back 2024-04-23T10:42:39.782680+0000 front 2024-04-23T10:42:39.782723+0000 (oldest deadline 2024-04-23T10:43:08.583479+0000) 2024-04-23T10:43:23.329 INFO:journalctl@ceph.osd.11.smithi204.stdout:Apr 23 10:43:22 smithi204 ceph-aa240572-0159-11ef-bc93-c7b262605968-osd-11[57644]: 2024-04-23T10:43:22.843+0000 7fcbc4b63700 -1 osd.11 116 heartbeat_check: no reply from 172.21.15.114:6822 osd.6 since back 2024-04-23T10:42:48.584157+0000 front 2024-04-23T10:42:48.584212+0000 (oldest deadline 2024-04-23T10:43:13.884025+0000) 2024-04-23T10:43:23.329 INFO:journalctl@ceph.osd.11.smithi204.stdout:Apr 23 10:43:22 smithi204 ceph-aa240572-0159-11ef-bc93-c7b262605968-osd-11[57644]: 2024-04-23T10:43:22.843+0000 7fcbc4b63700 -1 osd.11 116 heartbeat_check: no reply from 172.21.15.119:6820 osd.7 since back 2024-04-23T10:42:59.085770+0000 front 2024-04-23T10:42:59.085820+0000 (oldest deadline 2024-04-23T10:43:22.585479+0000) 2024-04-23T10:43:23.992 INFO:journalctl@ceph.osd.10.smithi119.stdout:Apr 23 10:43:23 smithi119 ceph-aa240572-0159-11ef-bc93-c7b262605968-osd-10[57454]: 2024-04-23T10:43:23.658+0000 7fb1849bf700 -1 osd.10 116 heartbeat_check: no reply from 172.21.15.114:6806 osd.0 since back 2024-04-23T10:42:01.813174+0000 front 2024-04-23T10:42:01.813138+0000 (oldest deadline 2024-04-23T10:42:27.113300+0000) 2024-04-23T10:43:23.992 INFO:journalctl@ceph.osd.10.smithi119.stdout:Apr 23 10:43:23 smithi119 ceph-aa240572-0159-11ef-bc93-c7b262605968-osd-10[57454]: 2024-04-23T10:43:23.658+0000 7fb1849bf700 -1 osd.10 116 heartbeat_check: no reply from 172.21.15.119:6804 osd.1 since back 2024-04-23T10:42:10.614599+0000 front 2024-04-23T10:42:10.614622+0000 (oldest deadline 2024-04-23T10:42:34.114881+0000) 2024-04-23T10:43:23.992 INFO:journalctl@ceph.osd.10.smithi119.stdout:Apr 23 10:43:23 smithi119 ceph-aa240572-0159-11ef-bc93-c7b262605968-osd-10[57454]: 2024-04-23T10:43:23.658+0000 7fb1849bf700 -1 osd.10 116 heartbeat_check: no reply from 172.21.15.204:6804 osd.2 since back 2024-04-23T10:42:19.416108+0000 front 2024-04-23T10:42:19.416241+0000 (oldest deadline 2024-04-23T10:42:42.316071+0000) 2024-04-23T10:43:23.993 INFO:journalctl@ceph.osd.10.smithi119.stdout:Apr 23 10:43:23 smithi119 ceph-aa240572-0159-11ef-bc93-c7b262605968-osd-10[57454]: 2024-04-23T10:43:23.658+0000 7fb1849bf700 -1 osd.10 116 heartbeat_check: no reply from 172.21.15.114:6814 osd.3 since back 2024-04-23T10:42:25.217210+0000 front 2024-04-23T10:42:25.217268+0000 (oldest deadline 2024-04-23T10:42:49.917291+0000) 2024-04-23T10:43:23.993 INFO:journalctl@ceph.osd.10.smithi119.stdout:Apr 23 10:43:23 smithi119 ceph-aa240572-0159-11ef-bc93-c7b262605968-osd-10[57454]: 2024-04-23T10:43:23.658+0000 7fb1849bf700 -1 osd.10 116 heartbeat_check: no reply from 172.21.15.119:6812 osd.4 since back 2024-04-23T10:42:36.620525+0000 front 2024-04-23T10:42:36.620567+0000 (oldest deadline 2024-04-23T10:43:00.720710+0000) 2024-04-23T10:43:23.993 INFO:journalctl@ceph.osd.10.smithi119.stdout:Apr 23 10:43:23 smithi119 ceph-aa240572-0159-11ef-bc93-c7b262605968-osd-10[57454]: 2024-04-23T10:43:23.658+0000 7fb1849bf700 -1 osd.10 116 heartbeat_check: no reply from 172.21.15.204:6812 osd.5 since back 2024-04-23T10:42:40.721579+0000 front 2024-04-23T10:42:40.721424+0000 (oldest deadline 2024-04-23T10:43:06.621309+0000) 2024-04-23T10:43:23.993 INFO:journalctl@ceph.osd.10.smithi119.stdout:Apr 23 10:43:23 smithi119 ceph-aa240572-0159-11ef-bc93-c7b262605968-osd-10[57454]: 2024-04-23T10:43:23.658+0000 7fb1849bf700 -1 osd.10 116 heartbeat_check: no reply from 172.21.15.114:6822 osd.6 since back 2024-04-23T10:42:52.423086+0000 front 2024-04-23T10:42:52.422931+0000 (oldest deadline 2024-04-23T10:43:15.322873+0000) 2024-04-23T10:43:24.078 INFO:journalctl@ceph.osd.11.smithi204.stdout:Apr 23 10:43:23 smithi204 ceph-aa240572-0159-11ef-bc93-c7b262605968-osd-11[57644]: 2024-04-23T10:43:23.797+0000 7fcbc4b63700 -1 osd.11 116 heartbeat_check: no reply from 172.21.15.114:6806 osd.0 since back 2024-04-23T10:42:00.675680+0000 front 2024-04-23T10:42:00.675649+0000 (oldest deadline 2024-04-23T10:42:25.876359+0000) 2024-04-23T10:43:24.078 INFO:journalctl@ceph.osd.11.smithi204.stdout:Apr 23 10:43:23 smithi204 ceph-aa240572-0159-11ef-bc93-c7b262605968-osd-11[57644]: 2024-04-23T10:43:23.797+0000 7fcbc4b63700 -1 osd.11 116 heartbeat_check: no reply from 172.21.15.119:6804 osd.1 since back 2024-04-23T10:42:09.278512+0000 front 2024-04-23T10:42:09.278494+0000 (oldest deadline 2024-04-23T10:42:33.378557+0000) 2024-04-23T10:43:24.078 INFO:journalctl@ceph.osd.11.smithi204.stdout:Apr 23 10:43:23 smithi204 ceph-aa240572-0159-11ef-bc93-c7b262605968-osd-11[57644]: 2024-04-23T10:43:23.797+0000 7fcbc4b63700 -1 osd.11 116 heartbeat_check: no reply from 172.21.15.204:6804 osd.2 since back 2024-04-23T10:42:18.679708+0000 front 2024-04-23T10:42:18.679577+0000 (oldest deadline 2024-04-23T10:42:41.579766+0000) 2024-04-23T10:43:24.078 INFO:journalctl@ceph.osd.11.smithi204.stdout:Apr 23 10:43:23 smithi204 ceph-aa240572-0159-11ef-bc93-c7b262605968-osd-11[57644]: 2024-04-23T10:43:23.797+0000 7fcbc4b63700 -1 osd.11 116 heartbeat_check: no reply from 172.21.15.119:6812 osd.4 since back 2024-04-23T10:42:35.082088+0000 front 2024-04-23T10:42:35.082102+0000 (oldest deadline 2024-04-23T10:42:59.782202+0000) 2024-04-23T10:43:24.079 INFO:journalctl@ceph.osd.11.smithi204.stdout:Apr 23 10:43:23 smithi204 ceph-aa240572-0159-11ef-bc93-c7b262605968-osd-11[57644]: 2024-04-23T10:43:23.797+0000 7fcbc4b63700 -1 osd.11 116 heartbeat_check: no reply from 172.21.15.204:6812 osd.5 since back 2024-04-23T10:42:39.782680+0000 front 2024-04-23T10:42:39.782723+0000 (oldest deadline 2024-04-23T10:43:08.583479+0000) 2024-04-23T10:43:24.079 INFO:journalctl@ceph.osd.11.smithi204.stdout:Apr 23 10:43:23 smithi204 ceph-aa240572-0159-11ef-bc93-c7b262605968-osd-11[57644]: 2024-04-23T10:43:23.797+0000 7fcbc4b63700 -1 osd.11 116 heartbeat_check: no reply from 172.21.15.114:6822 osd.6 since back 2024-04-23T10:42:48.584157+0000 front 2024-04-23T10:42:48.584212+0000 (oldest deadline 2024-04-23T10:43:13.884025+0000) 2024-04-23T10:43:24.079 INFO:journalctl@ceph.osd.11.smithi204.stdout:Apr 23 10:43:23 smithi204 ceph-aa240572-0159-11ef-bc93-c7b262605968-osd-11[57644]: 2024-04-23T10:43:23.797+0000 7fcbc4b63700 -1 osd.11 116 heartbeat_check: no reply from 172.21.15.119:6820 osd.7 since back 2024-04-23T10:42:59.085770+0000 front 2024-04-23T10:42:59.085820+0000 (oldest deadline 2024-04-23T10:43:22.585479+0000) 2024-04-23T10:43:24.992 INFO:journalctl@ceph.osd.10.smithi119.stdout:Apr 23 10:43:24 smithi119 podman[70072]: 2024-04-23 10:43:24.738952083 +0000 UTC m=+5.468985674 container died 44ce77aa1dade60d99a8056cb1fe9ab8398d25be451bbe2dc787125d0adfa776 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:068a558e3af4932cf1c60dd321c0bdf500075eb52ea1bf455e0973ba114fbfd4, name=ceph-aa240572-0159-11ef-bc93-c7b262605968-osd-10, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux , io.buildah.version=1.33.5, org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, CEPH_POINT_RELEASE=, RELEASE=wip-lusov-quiescer-d69f13d, org.label-schema.build-date=20240416, org.label-schema.license=GPLv2, ceph=True, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.vendor=CentOS, GIT_CLEAN=True) 2024-04-23T10:43:25.078 INFO:journalctl@ceph.osd.11.smithi204.stdout:Apr 23 10:43:24 smithi204 ceph-aa240572-0159-11ef-bc93-c7b262605968-osd-11[57644]: 2024-04-23T10:43:24.799+0000 7fcbc4b63700 -1 osd.11 116 heartbeat_check: no reply from 172.21.15.114:6806 osd.0 since back 2024-04-23T10:42:00.675680+0000 front 2024-04-23T10:42:00.675649+0000 (oldest deadline 2024-04-23T10:42:25.876359+0000) 2024-04-23T10:43:25.078 INFO:journalctl@ceph.osd.11.smithi204.stdout:Apr 23 10:43:24 smithi204 ceph-aa240572-0159-11ef-bc93-c7b262605968-osd-11[57644]: 2024-04-23T10:43:24.799+0000 7fcbc4b63700 -1 osd.11 116 heartbeat_check: no reply from 172.21.15.119:6804 osd.1 since back 2024-04-23T10:42:09.278512+0000 front 2024-04-23T10:42:09.278494+0000 (oldest deadline 2024-04-23T10:42:33.378557+0000) 2024-04-23T10:43:25.078 INFO:journalctl@ceph.osd.11.smithi204.stdout:Apr 23 10:43:24 smithi204 ceph-aa240572-0159-11ef-bc93-c7b262605968-osd-11[57644]: 2024-04-23T10:43:24.799+0000 7fcbc4b63700 -1 osd.11 116 heartbeat_check: no reply from 172.21.15.204:6804 osd.2 since back 2024-04-23T10:42:18.679708+0000 front 2024-04-23T10:42:18.679577+0000 (oldest deadline 2024-04-23T10:42:41.579766+0000) 2024-04-23T10:43:25.078 INFO:journalctl@ceph.osd.11.smithi204.stdout:Apr 23 10:43:24 smithi204 ceph-aa240572-0159-11ef-bc93-c7b262605968-osd-11[57644]: 2024-04-23T10:43:24.799+0000 7fcbc4b63700 -1 osd.11 116 heartbeat_check: no reply from 172.21.15.119:6812 osd.4 since back 2024-04-23T10:42:35.082088+0000 front 2024-04-23T10:42:35.082102+0000 (oldest deadline 2024-04-23T10:42:59.782202+0000) 2024-04-23T10:43:25.079 INFO:journalctl@ceph.osd.11.smithi204.stdout:Apr 23 10:43:24 smithi204 ceph-aa240572-0159-11ef-bc93-c7b262605968-osd-11[57644]: 2024-04-23T10:43:24.799+0000 7fcbc4b63700 -1 osd.11 116 heartbeat_check: no reply from 172.21.15.204:6812 osd.5 since back 2024-04-23T10:42:39.782680+0000 front 2024-04-23T10:42:39.782723+0000 (oldest deadline 2024-04-23T10:43:08.583479+0000) 2024-04-23T10:43:25.079 INFO:journalctl@ceph.osd.11.smithi204.stdout:Apr 23 10:43:24 smithi204 ceph-aa240572-0159-11ef-bc93-c7b262605968-osd-11[57644]: 2024-04-23T10:43:24.799+0000 7fcbc4b63700 -1 osd.11 116 heartbeat_check: no reply from 172.21.15.114:6822 osd.6 since back 2024-04-23T10:42:48.584157+0000 front 2024-04-23T10:42:48.584212+0000 (oldest deadline 2024-04-23T10:43:13.884025+0000) 2024-04-23T10:43:25.079 INFO:journalctl@ceph.osd.11.smithi204.stdout:Apr 23 10:43:24 smithi204 ceph-aa240572-0159-11ef-bc93-c7b262605968-osd-11[57644]: 2024-04-23T10:43:24.799+0000 7fcbc4b63700 -1 osd.11 116 heartbeat_check: no reply from 172.21.15.119:6820 osd.7 since back 2024-04-23T10:42:59.085770+0000 front 2024-04-23T10:42:59.085820+0000 (oldest deadline 2024-04-23T10:43:22.585479+0000) 2024-04-23T10:43:25.399 INFO:journalctl@ceph.osd.10.smithi119.stdout:Apr 23 10:43:25 smithi119 podman[70072]: 2024-04-23 10:43:25.149407921 +0000 UTC m=+5.879441511 container cleanup 44ce77aa1dade60d99a8056cb1fe9ab8398d25be451bbe2dc787125d0adfa776 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:068a558e3af4932cf1c60dd321c0bdf500075eb52ea1bf455e0973ba114fbfd4, name=ceph-aa240572-0159-11ef-bc93-c7b262605968-osd-10, org.label-schema.build-date=20240416, CEPH_POINT_RELEASE=, ceph=True, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.schema-version=1.0, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, RELEASE=wip-lusov-quiescer-d69f13d, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, GIT_CLEAN=True, maintainer=Guillaume Abrioux , io.buildah.version=1.33.5, org.label-schema.vendor=CentOS) 2024-04-23T10:43:25.399 INFO:journalctl@ceph.osd.10.smithi119.stdout:Apr 23 10:43:25 smithi119 bash[70072]: ceph-aa240572-0159-11ef-bc93-c7b262605968-osd-10 2024-04-23T10:43:25.708 INFO:journalctl@ceph.osd.10.smithi119.stdout:Apr 23 10:43:25 smithi119 podman[70089]: 2024-04-23 10:43:25.400557876 +0000 UTC m=+0.656854191 container remove 44ce77aa1dade60d99a8056cb1fe9ab8398d25be451bbe2dc787125d0adfa776 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:068a558e3af4932cf1c60dd321c0bdf500075eb52ea1bf455e0973ba114fbfd4, name=ceph-aa240572-0159-11ef-bc93-c7b262605968-osd-10, GIT_BRANCH=HEAD, CEPH_POINT_RELEASE=, ceph=True, org.label-schema.build-date=20240416, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS, RELEASE=wip-lusov-quiescer-d69f13d, io.buildah.version=1.33.5, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, GIT_CLEAN=True) 2024-04-23T10:43:25.992 INFO:journalctl@ceph.osd.10.smithi119.stdout:Apr 23 10:43:25 smithi119 podman[70180]: 2024-04-23 10:43:25.613141469 +0000 UTC m=+0.017679412 image pull da4b004c09e820713483d039bc76fbd660be678f2ea3c03e059a6161b74dab76 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:068a558e3af4932cf1c60dd321c0bdf500075eb52ea1bf455e0973ba114fbfd4 2024-04-23T10:43:25.993 INFO:journalctl@ceph.osd.10.smithi119.stdout:Apr 23 10:43:25 smithi119 podman[70180]: 2024-04-23 10:43:25.734684858 +0000 UTC m=+0.139222833 container create ddce1846ca7f2ac367eb27656ca8e99f68338efa5733bbc520afd6576420d345 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:068a558e3af4932cf1c60dd321c0bdf500075eb52ea1bf455e0973ba114fbfd4, name=ceph-aa240572-0159-11ef-bc93-c7b262605968-osd-10-deactivate, GIT_CLEAN=True, ceph=True, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240416, org.label-schema.vendor=CentOS, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, RELEASE=wip-lusov-quiescer-d69f13d, io.buildah.version=1.33.5) 2024-04-23T10:43:25.993 INFO:journalctl@ceph.osd.10.smithi119.stdout:Apr 23 10:43:25 smithi119 podman[70180]: 2024-04-23 10:43:25.946019907 +0000 UTC m=+0.350557838 container init ddce1846ca7f2ac367eb27656ca8e99f68338efa5733bbc520afd6576420d345 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:068a558e3af4932cf1c60dd321c0bdf500075eb52ea1bf455e0973ba114fbfd4, name=ceph-aa240572-0159-11ef-bc93-c7b262605968-osd-10-deactivate, org.label-schema.build-date=20240416, org.label-schema.name=CentOS Stream 8 Base Image, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.33.5, org.label-schema.vendor=CentOS, RELEASE=wip-lusov-quiescer-d69f13d, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, org.label-schema.schema-version=1.0, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, GIT_CLEAN=True, ceph=True) 2024-04-23T10:43:25.993 INFO:journalctl@ceph.osd.10.smithi119.stdout:Apr 23 10:43:25 smithi119 podman[70180]: 2024-04-23 10:43:25.949187852 +0000 UTC m=+0.353725782 container start ddce1846ca7f2ac367eb27656ca8e99f68338efa5733bbc520afd6576420d345 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:068a558e3af4932cf1c60dd321c0bdf500075eb52ea1bf455e0973ba114fbfd4, name=ceph-aa240572-0159-11ef-bc93-c7b262605968-osd-10-deactivate, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=wip-lusov-quiescer-d69f13d, GIT_CLEAN=True, maintainer=Guillaume Abrioux , ceph=True, org.label-schema.build-date=20240416, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, io.buildah.version=1.33.5, org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.license=GPLv2) 2024-04-23T10:43:26.078 INFO:journalctl@ceph.osd.11.smithi204.stdout:Apr 23 10:43:25 smithi204 ceph-aa240572-0159-11ef-bc93-c7b262605968-osd-11[57644]: 2024-04-23T10:43:25.777+0000 7fcbc4b63700 -1 osd.11 116 heartbeat_check: no reply from 172.21.15.114:6806 osd.0 since back 2024-04-23T10:42:00.675680+0000 front 2024-04-23T10:42:00.675649+0000 (oldest deadline 2024-04-23T10:42:25.876359+0000) 2024-04-23T10:43:26.078 INFO:journalctl@ceph.osd.11.smithi204.stdout:Apr 23 10:43:25 smithi204 ceph-aa240572-0159-11ef-bc93-c7b262605968-osd-11[57644]: 2024-04-23T10:43:25.777+0000 7fcbc4b63700 -1 osd.11 116 heartbeat_check: no reply from 172.21.15.119:6804 osd.1 since back 2024-04-23T10:42:09.278512+0000 front 2024-04-23T10:42:09.278494+0000 (oldest deadline 2024-04-23T10:42:33.378557+0000) 2024-04-23T10:43:26.078 INFO:journalctl@ceph.osd.11.smithi204.stdout:Apr 23 10:43:25 smithi204 ceph-aa240572-0159-11ef-bc93-c7b262605968-osd-11[57644]: 2024-04-23T10:43:25.777+0000 7fcbc4b63700 -1 osd.11 116 heartbeat_check: no reply from 172.21.15.204:6804 osd.2 since back 2024-04-23T10:42:18.679708+0000 front 2024-04-23T10:42:18.679577+0000 (oldest deadline 2024-04-23T10:42:41.579766+0000) 2024-04-23T10:43:26.078 INFO:journalctl@ceph.osd.11.smithi204.stdout:Apr 23 10:43:25 smithi204 ceph-aa240572-0159-11ef-bc93-c7b262605968-osd-11[57644]: 2024-04-23T10:43:25.777+0000 7fcbc4b63700 -1 osd.11 116 heartbeat_check: no reply from 172.21.15.119:6812 osd.4 since back 2024-04-23T10:42:35.082088+0000 front 2024-04-23T10:42:35.082102+0000 (oldest deadline 2024-04-23T10:42:59.782202+0000) 2024-04-23T10:43:26.079 INFO:journalctl@ceph.osd.11.smithi204.stdout:Apr 23 10:43:25 smithi204 ceph-aa240572-0159-11ef-bc93-c7b262605968-osd-11[57644]: 2024-04-23T10:43:25.777+0000 7fcbc4b63700 -1 osd.11 116 heartbeat_check: no reply from 172.21.15.204:6812 osd.5 since back 2024-04-23T10:42:39.782680+0000 front 2024-04-23T10:42:39.782723+0000 (oldest deadline 2024-04-23T10:43:08.583479+0000) 2024-04-23T10:43:26.079 INFO:journalctl@ceph.osd.11.smithi204.stdout:Apr 23 10:43:25 smithi204 ceph-aa240572-0159-11ef-bc93-c7b262605968-osd-11[57644]: 2024-04-23T10:43:25.777+0000 7fcbc4b63700 -1 osd.11 116 heartbeat_check: no reply from 172.21.15.114:6822 osd.6 since back 2024-04-23T10:42:48.584157+0000 front 2024-04-23T10:42:48.584212+0000 (oldest deadline 2024-04-23T10:43:13.884025+0000) 2024-04-23T10:43:26.079 INFO:journalctl@ceph.osd.11.smithi204.stdout:Apr 23 10:43:25 smithi204 ceph-aa240572-0159-11ef-bc93-c7b262605968-osd-11[57644]: 2024-04-23T10:43:25.777+0000 7fcbc4b63700 -1 osd.11 116 heartbeat_check: no reply from 172.21.15.119:6820 osd.7 since back 2024-04-23T10:42:59.085770+0000 front 2024-04-23T10:42:59.085820+0000 (oldest deadline 2024-04-23T10:43:22.585479+0000) 2024-04-23T10:43:26.492 INFO:journalctl@ceph.osd.10.smithi119.stdout:Apr 23 10:43:26 smithi119 podman[70180]: 2024-04-23 10:43:26.037054675 +0000 UTC m=+0.441592617 container attach ddce1846ca7f2ac367eb27656ca8e99f68338efa5733bbc520afd6576420d345 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:068a558e3af4932cf1c60dd321c0bdf500075eb52ea1bf455e0973ba114fbfd4, name=ceph-aa240572-0159-11ef-bc93-c7b262605968-osd-10-deactivate, org.label-schema.license=GPLv2, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, org.label-schema.schema-version=1.0, org.label-schema.name=CentOS Stream 8 Base Image, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=, org.label-schema.build-date=20240416, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.33.5, org.label-schema.vendor=CentOS, RELEASE=wip-lusov-quiescer-d69f13d, GIT_BRANCH=HEAD, ceph=True) 2024-04-23T10:43:26.492 INFO:journalctl@ceph.osd.10.smithi119.stdout:Apr 23 10:43:26 smithi119 podman[70180]: 2024-04-23 10:43:26.237297104 +0000 UTC m=+0.641835028 container died ddce1846ca7f2ac367eb27656ca8e99f68338efa5733bbc520afd6576420d345 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:068a558e3af4932cf1c60dd321c0bdf500075eb52ea1bf455e0973ba114fbfd4, name=ceph-aa240572-0159-11ef-bc93-c7b262605968-osd-10-deactivate, ceph=True, GIT_CLEAN=True, GIT_BRANCH=HEAD, io.buildah.version=1.33.5, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.license=GPLv2, org.label-schema.build-date=20240416, RELEASE=wip-lusov-quiescer-d69f13d, org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=) 2024-04-23T10:43:26.849 DEBUG:teuthology.orchestra.run.smithi119:> sudo pkill -f 'journalctl -f -n 0 -u ceph-aa240572-0159-11ef-bc93-c7b262605968@osd.10.service' 2024-04-23T10:43:26.885 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-04-23T10:43:26.886 INFO:tasks.cephadm.osd.10:Stopped osd.10 2024-04-23T10:43:26.886 INFO:tasks.cephadm.osd.11:Stopping osd.11... 2024-04-23T10:43:26.886 DEBUG:teuthology.orchestra.run.smithi204:> sudo systemctl stop ceph-aa240572-0159-11ef-bc93-c7b262605968@osd.11 2024-04-23T10:43:27.078 INFO:journalctl@ceph.osd.11.smithi204.stdout:Apr 23 10:43:26 smithi204 ceph-aa240572-0159-11ef-bc93-c7b262605968-osd-11[57644]: 2024-04-23T10:43:26.823+0000 7fcbc4b63700 -1 osd.11 116 heartbeat_check: no reply from 172.21.15.114:6806 osd.0 since back 2024-04-23T10:42:00.675680+0000 front 2024-04-23T10:42:00.675649+0000 (oldest deadline 2024-04-23T10:42:25.876359+0000) 2024-04-23T10:43:27.078 INFO:journalctl@ceph.osd.11.smithi204.stdout:Apr 23 10:43:26 smithi204 ceph-aa240572-0159-11ef-bc93-c7b262605968-osd-11[57644]: 2024-04-23T10:43:26.823+0000 7fcbc4b63700 -1 osd.11 116 heartbeat_check: no reply from 172.21.15.119:6804 osd.1 since back 2024-04-23T10:42:09.278512+0000 front 2024-04-23T10:42:09.278494+0000 (oldest deadline 2024-04-23T10:42:33.378557+0000) 2024-04-23T10:43:27.078 INFO:journalctl@ceph.osd.11.smithi204.stdout:Apr 23 10:43:26 smithi204 ceph-aa240572-0159-11ef-bc93-c7b262605968-osd-11[57644]: 2024-04-23T10:43:26.823+0000 7fcbc4b63700 -1 osd.11 116 heartbeat_check: no reply from 172.21.15.204:6804 osd.2 since back 2024-04-23T10:42:18.679708+0000 front 2024-04-23T10:42:18.679577+0000 (oldest deadline 2024-04-23T10:42:41.579766+0000) 2024-04-23T10:43:27.078 INFO:journalctl@ceph.osd.11.smithi204.stdout:Apr 23 10:43:26 smithi204 ceph-aa240572-0159-11ef-bc93-c7b262605968-osd-11[57644]: 2024-04-23T10:43:26.823+0000 7fcbc4b63700 -1 osd.11 116 heartbeat_check: no reply from 172.21.15.119:6812 osd.4 since back 2024-04-23T10:42:35.082088+0000 front 2024-04-23T10:42:35.082102+0000 (oldest deadline 2024-04-23T10:42:59.782202+0000) 2024-04-23T10:43:27.079 INFO:journalctl@ceph.osd.11.smithi204.stdout:Apr 23 10:43:26 smithi204 ceph-aa240572-0159-11ef-bc93-c7b262605968-osd-11[57644]: 2024-04-23T10:43:26.823+0000 7fcbc4b63700 -1 osd.11 116 heartbeat_check: no reply from 172.21.15.204:6812 osd.5 since back 2024-04-23T10:42:39.782680+0000 front 2024-04-23T10:42:39.782723+0000 (oldest deadline 2024-04-23T10:43:08.583479+0000) 2024-04-23T10:43:27.079 INFO:journalctl@ceph.osd.11.smithi204.stdout:Apr 23 10:43:26 smithi204 ceph-aa240572-0159-11ef-bc93-c7b262605968-osd-11[57644]: 2024-04-23T10:43:26.823+0000 7fcbc4b63700 -1 osd.11 116 heartbeat_check: no reply from 172.21.15.114:6822 osd.6 since back 2024-04-23T10:42:48.584157+0000 front 2024-04-23T10:42:48.584212+0000 (oldest deadline 2024-04-23T10:43:13.884025+0000) 2024-04-23T10:43:27.079 INFO:journalctl@ceph.osd.11.smithi204.stdout:Apr 23 10:43:26 smithi204 ceph-aa240572-0159-11ef-bc93-c7b262605968-osd-11[57644]: 2024-04-23T10:43:26.823+0000 7fcbc4b63700 -1 osd.11 116 heartbeat_check: no reply from 172.21.15.119:6820 osd.7 since back 2024-04-23T10:42:59.085770+0000 front 2024-04-23T10:42:59.085820+0000 (oldest deadline 2024-04-23T10:43:22.585479+0000) 2024-04-23T10:43:27.079 INFO:journalctl@ceph.osd.11.smithi204.stdout:Apr 23 10:43:26 smithi204 systemd[1]: Stopping Ceph osd.11 for aa240572-0159-11ef-bc93-c7b262605968... 2024-04-23T10:43:27.578 INFO:journalctl@ceph.osd.11.smithi204.stdout:Apr 23 10:43:27 smithi204 ceph-aa240572-0159-11ef-bc93-c7b262605968-osd-11[57644]: 2024-04-23T10:43:27.191+0000 7fcbcc577700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.11 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-04-23T10:43:27.578 INFO:journalctl@ceph.osd.11.smithi204.stdout:Apr 23 10:43:27 smithi204 ceph-aa240572-0159-11ef-bc93-c7b262605968-osd-11[57644]: 2024-04-23T10:43:27.191+0000 7fcbcc577700 -1 osd.11 116 *** Got signal Terminated *** 2024-04-23T10:43:27.578 INFO:journalctl@ceph.osd.11.smithi204.stdout:Apr 23 10:43:27 smithi204 ceph-aa240572-0159-11ef-bc93-c7b262605968-osd-11[57644]: 2024-04-23T10:43:27.191+0000 7fcbcc577700 -1 osd.11 116 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-04-23T10:43:28.077 INFO:journalctl@ceph.osd.11.smithi204.stdout:Apr 23 10:43:27 smithi204 ceph-aa240572-0159-11ef-bc93-c7b262605968-osd-11[57644]: 2024-04-23T10:43:27.800+0000 7fcbc4b63700 -1 osd.11 116 heartbeat_check: no reply from 172.21.15.114:6806 osd.0 since back 2024-04-23T10:42:00.675680+0000 front 2024-04-23T10:42:00.675649+0000 (oldest deadline 2024-04-23T10:42:25.876359+0000) 2024-04-23T10:43:28.078 INFO:journalctl@ceph.osd.11.smithi204.stdout:Apr 23 10:43:27 smithi204 ceph-aa240572-0159-11ef-bc93-c7b262605968-osd-11[57644]: 2024-04-23T10:43:27.800+0000 7fcbc4b63700 -1 osd.11 116 heartbeat_check: no reply from 172.21.15.119:6804 osd.1 since back 2024-04-23T10:42:09.278512+0000 front 2024-04-23T10:42:09.278494+0000 (oldest deadline 2024-04-23T10:42:33.378557+0000) 2024-04-23T10:43:28.078 INFO:journalctl@ceph.osd.11.smithi204.stdout:Apr 23 10:43:27 smithi204 ceph-aa240572-0159-11ef-bc93-c7b262605968-osd-11[57644]: 2024-04-23T10:43:27.800+0000 7fcbc4b63700 -1 osd.11 116 heartbeat_check: no reply from 172.21.15.204:6804 osd.2 since back 2024-04-23T10:42:18.679708+0000 front 2024-04-23T10:42:18.679577+0000 (oldest deadline 2024-04-23T10:42:41.579766+0000) 2024-04-23T10:43:28.078 INFO:journalctl@ceph.osd.11.smithi204.stdout:Apr 23 10:43:27 smithi204 ceph-aa240572-0159-11ef-bc93-c7b262605968-osd-11[57644]: 2024-04-23T10:43:27.800+0000 7fcbc4b63700 -1 osd.11 116 heartbeat_check: no reply from 172.21.15.119:6812 osd.4 since back 2024-04-23T10:42:35.082088+0000 front 2024-04-23T10:42:35.082102+0000 (oldest deadline 2024-04-23T10:42:59.782202+0000) 2024-04-23T10:43:28.078 INFO:journalctl@ceph.osd.11.smithi204.stdout:Apr 23 10:43:27 smithi204 ceph-aa240572-0159-11ef-bc93-c7b262605968-osd-11[57644]: 2024-04-23T10:43:27.800+0000 7fcbc4b63700 -1 osd.11 116 heartbeat_check: no reply from 172.21.15.204:6812 osd.5 since back 2024-04-23T10:42:39.782680+0000 front 2024-04-23T10:42:39.782723+0000 (oldest deadline 2024-04-23T10:43:08.583479+0000) 2024-04-23T10:43:28.078 INFO:journalctl@ceph.osd.11.smithi204.stdout:Apr 23 10:43:27 smithi204 ceph-aa240572-0159-11ef-bc93-c7b262605968-osd-11[57644]: 2024-04-23T10:43:27.800+0000 7fcbc4b63700 -1 osd.11 116 heartbeat_check: no reply from 172.21.15.114:6822 osd.6 since back 2024-04-23T10:42:48.584157+0000 front 2024-04-23T10:42:48.584212+0000 (oldest deadline 2024-04-23T10:43:13.884025+0000) 2024-04-23T10:43:28.078 INFO:journalctl@ceph.osd.11.smithi204.stdout:Apr 23 10:43:27 smithi204 ceph-aa240572-0159-11ef-bc93-c7b262605968-osd-11[57644]: 2024-04-23T10:43:27.800+0000 7fcbc4b63700 -1 osd.11 116 heartbeat_check: no reply from 172.21.15.119:6820 osd.7 since back 2024-04-23T10:42:59.085770+0000 front 2024-04-23T10:42:59.085820+0000 (oldest deadline 2024-04-23T10:43:22.585479+0000) 2024-04-23T10:43:29.328 INFO:journalctl@ceph.osd.11.smithi204.stdout:Apr 23 10:43:28 smithi204 ceph-aa240572-0159-11ef-bc93-c7b262605968-osd-11[57644]: 2024-04-23T10:43:28.843+0000 7fcbc4b63700 -1 osd.11 116 heartbeat_check: no reply from 172.21.15.114:6806 osd.0 since back 2024-04-23T10:42:00.675680+0000 front 2024-04-23T10:42:00.675649+0000 (oldest deadline 2024-04-23T10:42:25.876359+0000) 2024-04-23T10:43:29.328 INFO:journalctl@ceph.osd.11.smithi204.stdout:Apr 23 10:43:28 smithi204 ceph-aa240572-0159-11ef-bc93-c7b262605968-osd-11[57644]: 2024-04-23T10:43:28.843+0000 7fcbc4b63700 -1 osd.11 116 heartbeat_check: no reply from 172.21.15.119:6804 osd.1 since back 2024-04-23T10:42:09.278512+0000 front 2024-04-23T10:42:09.278494+0000 (oldest deadline 2024-04-23T10:42:33.378557+0000) 2024-04-23T10:43:29.329 INFO:journalctl@ceph.osd.11.smithi204.stdout:Apr 23 10:43:28 smithi204 ceph-aa240572-0159-11ef-bc93-c7b262605968-osd-11[57644]: 2024-04-23T10:43:28.843+0000 7fcbc4b63700 -1 osd.11 116 heartbeat_check: no reply from 172.21.15.204:6804 osd.2 since back 2024-04-23T10:42:18.679708+0000 front 2024-04-23T10:42:18.679577+0000 (oldest deadline 2024-04-23T10:42:41.579766+0000) 2024-04-23T10:43:29.329 INFO:journalctl@ceph.osd.11.smithi204.stdout:Apr 23 10:43:28 smithi204 ceph-aa240572-0159-11ef-bc93-c7b262605968-osd-11[57644]: 2024-04-23T10:43:28.843+0000 7fcbc4b63700 -1 osd.11 116 heartbeat_check: no reply from 172.21.15.119:6812 osd.4 since back 2024-04-23T10:42:35.082088+0000 front 2024-04-23T10:42:35.082102+0000 (oldest deadline 2024-04-23T10:42:59.782202+0000) 2024-04-23T10:43:29.329 INFO:journalctl@ceph.osd.11.smithi204.stdout:Apr 23 10:43:28 smithi204 ceph-aa240572-0159-11ef-bc93-c7b262605968-osd-11[57644]: 2024-04-23T10:43:28.843+0000 7fcbc4b63700 -1 osd.11 116 heartbeat_check: no reply from 172.21.15.204:6812 osd.5 since back 2024-04-23T10:42:39.782680+0000 front 2024-04-23T10:42:39.782723+0000 (oldest deadline 2024-04-23T10:43:08.583479+0000) 2024-04-23T10:43:29.329 INFO:journalctl@ceph.osd.11.smithi204.stdout:Apr 23 10:43:28 smithi204 ceph-aa240572-0159-11ef-bc93-c7b262605968-osd-11[57644]: 2024-04-23T10:43:28.843+0000 7fcbc4b63700 -1 osd.11 116 heartbeat_check: no reply from 172.21.15.114:6822 osd.6 since back 2024-04-23T10:42:48.584157+0000 front 2024-04-23T10:42:48.584212+0000 (oldest deadline 2024-04-23T10:43:13.884025+0000) 2024-04-23T10:43:29.329 INFO:journalctl@ceph.osd.11.smithi204.stdout:Apr 23 10:43:28 smithi204 ceph-aa240572-0159-11ef-bc93-c7b262605968-osd-11[57644]: 2024-04-23T10:43:28.843+0000 7fcbc4b63700 -1 osd.11 116 heartbeat_check: no reply from 172.21.15.119:6820 osd.7 since back 2024-04-23T10:42:59.085770+0000 front 2024-04-23T10:42:59.085820+0000 (oldest deadline 2024-04-23T10:43:22.585479+0000) 2024-04-23T10:43:30.328 INFO:journalctl@ceph.osd.11.smithi204.stdout:Apr 23 10:43:29 smithi204 ceph-aa240572-0159-11ef-bc93-c7b262605968-osd-11[57644]: 2024-04-23T10:43:29.856+0000 7fcbc4b63700 -1 osd.11 116 heartbeat_check: no reply from 172.21.15.114:6806 osd.0 since back 2024-04-23T10:42:00.675680+0000 front 2024-04-23T10:42:00.675649+0000 (oldest deadline 2024-04-23T10:42:25.876359+0000) 2024-04-23T10:43:30.328 INFO:journalctl@ceph.osd.11.smithi204.stdout:Apr 23 10:43:29 smithi204 ceph-aa240572-0159-11ef-bc93-c7b262605968-osd-11[57644]: 2024-04-23T10:43:29.856+0000 7fcbc4b63700 -1 osd.11 116 heartbeat_check: no reply from 172.21.15.119:6804 osd.1 since back 2024-04-23T10:42:09.278512+0000 front 2024-04-23T10:42:09.278494+0000 (oldest deadline 2024-04-23T10:42:33.378557+0000) 2024-04-23T10:43:30.329 INFO:journalctl@ceph.osd.11.smithi204.stdout:Apr 23 10:43:29 smithi204 ceph-aa240572-0159-11ef-bc93-c7b262605968-osd-11[57644]: 2024-04-23T10:43:29.856+0000 7fcbc4b63700 -1 osd.11 116 heartbeat_check: no reply from 172.21.15.204:6804 osd.2 since back 2024-04-23T10:42:18.679708+0000 front 2024-04-23T10:42:18.679577+0000 (oldest deadline 2024-04-23T10:42:41.579766+0000) 2024-04-23T10:43:30.329 INFO:journalctl@ceph.osd.11.smithi204.stdout:Apr 23 10:43:29 smithi204 ceph-aa240572-0159-11ef-bc93-c7b262605968-osd-11[57644]: 2024-04-23T10:43:29.856+0000 7fcbc4b63700 -1 osd.11 116 heartbeat_check: no reply from 172.21.15.119:6812 osd.4 since back 2024-04-23T10:42:35.082088+0000 front 2024-04-23T10:42:35.082102+0000 (oldest deadline 2024-04-23T10:42:59.782202+0000) 2024-04-23T10:43:30.329 INFO:journalctl@ceph.osd.11.smithi204.stdout:Apr 23 10:43:29 smithi204 ceph-aa240572-0159-11ef-bc93-c7b262605968-osd-11[57644]: 2024-04-23T10:43:29.856+0000 7fcbc4b63700 -1 osd.11 116 heartbeat_check: no reply from 172.21.15.204:6812 osd.5 since back 2024-04-23T10:42:39.782680+0000 front 2024-04-23T10:42:39.782723+0000 (oldest deadline 2024-04-23T10:43:08.583479+0000) 2024-04-23T10:43:30.329 INFO:journalctl@ceph.osd.11.smithi204.stdout:Apr 23 10:43:29 smithi204 ceph-aa240572-0159-11ef-bc93-c7b262605968-osd-11[57644]: 2024-04-23T10:43:29.856+0000 7fcbc4b63700 -1 osd.11 116 heartbeat_check: no reply from 172.21.15.114:6822 osd.6 since back 2024-04-23T10:42:48.584157+0000 front 2024-04-23T10:42:48.584212+0000 (oldest deadline 2024-04-23T10:43:13.884025+0000) 2024-04-23T10:43:30.329 INFO:journalctl@ceph.osd.11.smithi204.stdout:Apr 23 10:43:29 smithi204 ceph-aa240572-0159-11ef-bc93-c7b262605968-osd-11[57644]: 2024-04-23T10:43:29.856+0000 7fcbc4b63700 -1 osd.11 116 heartbeat_check: no reply from 172.21.15.119:6820 osd.7 since back 2024-04-23T10:42:59.085770+0000 front 2024-04-23T10:42:59.085820+0000 (oldest deadline 2024-04-23T10:43:22.585479+0000) 2024-04-23T10:43:31.328 INFO:journalctl@ceph.osd.11.smithi204.stdout:Apr 23 10:43:30 smithi204 ceph-aa240572-0159-11ef-bc93-c7b262605968-osd-11[57644]: 2024-04-23T10:43:30.849+0000 7fcbc4b63700 -1 osd.11 116 heartbeat_check: no reply from 172.21.15.114:6806 osd.0 since back 2024-04-23T10:42:00.675680+0000 front 2024-04-23T10:42:00.675649+0000 (oldest deadline 2024-04-23T10:42:25.876359+0000) 2024-04-23T10:43:31.328 INFO:journalctl@ceph.osd.11.smithi204.stdout:Apr 23 10:43:30 smithi204 ceph-aa240572-0159-11ef-bc93-c7b262605968-osd-11[57644]: 2024-04-23T10:43:30.849+0000 7fcbc4b63700 -1 osd.11 116 heartbeat_check: no reply from 172.21.15.119:6804 osd.1 since back 2024-04-23T10:42:09.278512+0000 front 2024-04-23T10:42:09.278494+0000 (oldest deadline 2024-04-23T10:42:33.378557+0000) 2024-04-23T10:43:31.328 INFO:journalctl@ceph.osd.11.smithi204.stdout:Apr 23 10:43:30 smithi204 ceph-aa240572-0159-11ef-bc93-c7b262605968-osd-11[57644]: 2024-04-23T10:43:30.849+0000 7fcbc4b63700 -1 osd.11 116 heartbeat_check: no reply from 172.21.15.204:6804 osd.2 since back 2024-04-23T10:42:18.679708+0000 front 2024-04-23T10:42:18.679577+0000 (oldest deadline 2024-04-23T10:42:41.579766+0000) 2024-04-23T10:43:31.329 INFO:journalctl@ceph.osd.11.smithi204.stdout:Apr 23 10:43:30 smithi204 ceph-aa240572-0159-11ef-bc93-c7b262605968-osd-11[57644]: 2024-04-23T10:43:30.849+0000 7fcbc4b63700 -1 osd.11 116 heartbeat_check: no reply from 172.21.15.119:6812 osd.4 since back 2024-04-23T10:42:35.082088+0000 front 2024-04-23T10:42:35.082102+0000 (oldest deadline 2024-04-23T10:42:59.782202+0000) 2024-04-23T10:43:31.329 INFO:journalctl@ceph.osd.11.smithi204.stdout:Apr 23 10:43:30 smithi204 ceph-aa240572-0159-11ef-bc93-c7b262605968-osd-11[57644]: 2024-04-23T10:43:30.849+0000 7fcbc4b63700 -1 osd.11 116 heartbeat_check: no reply from 172.21.15.204:6812 osd.5 since back 2024-04-23T10:42:39.782680+0000 front 2024-04-23T10:42:39.782723+0000 (oldest deadline 2024-04-23T10:43:08.583479+0000) 2024-04-23T10:43:31.329 INFO:journalctl@ceph.osd.11.smithi204.stdout:Apr 23 10:43:30 smithi204 ceph-aa240572-0159-11ef-bc93-c7b262605968-osd-11[57644]: 2024-04-23T10:43:30.849+0000 7fcbc4b63700 -1 osd.11 116 heartbeat_check: no reply from 172.21.15.114:6822 osd.6 since back 2024-04-23T10:42:48.584157+0000 front 2024-04-23T10:42:48.584212+0000 (oldest deadline 2024-04-23T10:43:13.884025+0000) 2024-04-23T10:43:31.329 INFO:journalctl@ceph.osd.11.smithi204.stdout:Apr 23 10:43:30 smithi204 ceph-aa240572-0159-11ef-bc93-c7b262605968-osd-11[57644]: 2024-04-23T10:43:30.849+0000 7fcbc4b63700 -1 osd.11 116 heartbeat_check: no reply from 172.21.15.119:6820 osd.7 since back 2024-04-23T10:42:59.085770+0000 front 2024-04-23T10:42:59.085820+0000 (oldest deadline 2024-04-23T10:43:22.585479+0000) 2024-04-23T10:43:32.328 INFO:journalctl@ceph.osd.11.smithi204.stdout:Apr 23 10:43:31 smithi204 ceph-aa240572-0159-11ef-bc93-c7b262605968-osd-11[57644]: 2024-04-23T10:43:31.831+0000 7fcbc4b63700 -1 osd.11 116 heartbeat_check: no reply from 172.21.15.114:6806 osd.0 since back 2024-04-23T10:42:00.675680+0000 front 2024-04-23T10:42:00.675649+0000 (oldest deadline 2024-04-23T10:42:25.876359+0000) 2024-04-23T10:43:32.328 INFO:journalctl@ceph.osd.11.smithi204.stdout:Apr 23 10:43:31 smithi204 ceph-aa240572-0159-11ef-bc93-c7b262605968-osd-11[57644]: 2024-04-23T10:43:31.831+0000 7fcbc4b63700 -1 osd.11 116 heartbeat_check: no reply from 172.21.15.119:6804 osd.1 since back 2024-04-23T10:42:09.278512+0000 front 2024-04-23T10:42:09.278494+0000 (oldest deadline 2024-04-23T10:42:33.378557+0000) 2024-04-23T10:43:32.328 INFO:journalctl@ceph.osd.11.smithi204.stdout:Apr 23 10:43:31 smithi204 ceph-aa240572-0159-11ef-bc93-c7b262605968-osd-11[57644]: 2024-04-23T10:43:31.831+0000 7fcbc4b63700 -1 osd.11 116 heartbeat_check: no reply from 172.21.15.204:6804 osd.2 since back 2024-04-23T10:42:18.679708+0000 front 2024-04-23T10:42:18.679577+0000 (oldest deadline 2024-04-23T10:42:41.579766+0000) 2024-04-23T10:43:32.328 INFO:journalctl@ceph.osd.11.smithi204.stdout:Apr 23 10:43:31 smithi204 ceph-aa240572-0159-11ef-bc93-c7b262605968-osd-11[57644]: 2024-04-23T10:43:31.831+0000 7fcbc4b63700 -1 osd.11 116 heartbeat_check: no reply from 172.21.15.119:6812 osd.4 since back 2024-04-23T10:42:35.082088+0000 front 2024-04-23T10:42:35.082102+0000 (oldest deadline 2024-04-23T10:42:59.782202+0000) 2024-04-23T10:43:32.329 INFO:journalctl@ceph.osd.11.smithi204.stdout:Apr 23 10:43:31 smithi204 ceph-aa240572-0159-11ef-bc93-c7b262605968-osd-11[57644]: 2024-04-23T10:43:31.831+0000 7fcbc4b63700 -1 osd.11 116 heartbeat_check: no reply from 172.21.15.204:6812 osd.5 since back 2024-04-23T10:42:39.782680+0000 front 2024-04-23T10:42:39.782723+0000 (oldest deadline 2024-04-23T10:43:08.583479+0000) 2024-04-23T10:43:32.329 INFO:journalctl@ceph.osd.11.smithi204.stdout:Apr 23 10:43:31 smithi204 ceph-aa240572-0159-11ef-bc93-c7b262605968-osd-11[57644]: 2024-04-23T10:43:31.831+0000 7fcbc4b63700 -1 osd.11 116 heartbeat_check: no reply from 172.21.15.114:6822 osd.6 since back 2024-04-23T10:42:48.584157+0000 front 2024-04-23T10:42:48.584212+0000 (oldest deadline 2024-04-23T10:43:13.884025+0000) 2024-04-23T10:43:32.329 INFO:journalctl@ceph.osd.11.smithi204.stdout:Apr 23 10:43:31 smithi204 ceph-aa240572-0159-11ef-bc93-c7b262605968-osd-11[57644]: 2024-04-23T10:43:31.831+0000 7fcbc4b63700 -1 osd.11 116 heartbeat_check: no reply from 172.21.15.119:6820 osd.7 since back 2024-04-23T10:42:59.085770+0000 front 2024-04-23T10:42:59.085820+0000 (oldest deadline 2024-04-23T10:43:22.585479+0000) 2024-04-23T10:43:32.891 INFO:journalctl@ceph.osd.11.smithi204.stdout:Apr 23 10:43:32 smithi204 podman[71506]: 2024-04-23 10:43:32.635650647 +0000 UTC m=+5.555864055 container died 683e1a039db532eaee9124d32cdf4ae19dcdeca1243fa7fd31b2df7b0ea7386c (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:068a558e3af4932cf1c60dd321c0bdf500075eb52ea1bf455e0973ba114fbfd4, name=ceph-aa240572-0159-11ef-bc93-c7b262605968-osd-11, org.label-schema.build-date=20240416, org.label-schema.license=GPLv2, RELEASE=wip-lusov-quiescer-d69f13d, org.label-schema.name=CentOS Stream 8 Base Image, ceph=True, GIT_CLEAN=True, io.buildah.version=1.33.5, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux , GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67) 2024-04-23T10:43:33.166 INFO:journalctl@ceph.osd.11.smithi204.stdout:Apr 23 10:43:32 smithi204 podman[71506]: 2024-04-23 10:43:32.984174553 +0000 UTC m=+5.904387971 container cleanup 683e1a039db532eaee9124d32cdf4ae19dcdeca1243fa7fd31b2df7b0ea7386c (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:068a558e3af4932cf1c60dd321c0bdf500075eb52ea1bf455e0973ba114fbfd4, name=ceph-aa240572-0159-11ef-bc93-c7b262605968-osd-11, GIT_CLEAN=True, io.buildah.version=1.33.5, org.label-schema.name=CentOS Stream 8 Base Image, ceph=True, org.label-schema.build-date=20240416, RELEASE=wip-lusov-quiescer-d69f13d, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, org.label-schema.schema-version=1.0, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-04-23T10:43:33.167 INFO:journalctl@ceph.osd.11.smithi204.stdout:Apr 23 10:43:32 smithi204 bash[71506]: ceph-aa240572-0159-11ef-bc93-c7b262605968-osd-11 2024-04-23T10:43:33.578 INFO:journalctl@ceph.osd.11.smithi204.stdout:Apr 23 10:43:33 smithi204 podman[71523]: 2024-04-23 10:43:33.168443756 +0000 UTC m=+0.528206906 container remove 683e1a039db532eaee9124d32cdf4ae19dcdeca1243fa7fd31b2df7b0ea7386c (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:068a558e3af4932cf1c60dd321c0bdf500075eb52ea1bf455e0973ba114fbfd4, name=ceph-aa240572-0159-11ef-bc93-c7b262605968-osd-11, GIT_CLEAN=True, io.buildah.version=1.33.5, RELEASE=wip-lusov-quiescer-d69f13d, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, org.label-schema.build-date=20240416, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux , ceph=True, org.label-schema.name=CentOS Stream 8 Base Image) 2024-04-23T10:43:33.895 INFO:journalctl@ceph.osd.11.smithi204.stdout:Apr 23 10:43:33 smithi204 podman[71617]: 2024-04-23 10:43:33.497659065 +0000 UTC m=+0.017985420 image pull da4b004c09e820713483d039bc76fbd660be678f2ea3c03e059a6161b74dab76 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:068a558e3af4932cf1c60dd321c0bdf500075eb52ea1bf455e0973ba114fbfd4 2024-04-23T10:43:33.896 INFO:journalctl@ceph.osd.11.smithi204.stdout:Apr 23 10:43:33 smithi204 podman[71617]: 2024-04-23 10:43:33.60274696 +0000 UTC m=+0.123073350 container create eb2cfc84c1db283aad7ee61428974ed0eee083bf8f4ac3e4a79c5a63f06585cc (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:068a558e3af4932cf1c60dd321c0bdf500075eb52ea1bf455e0973ba114fbfd4, name=ceph-aa240572-0159-11ef-bc93-c7b262605968-osd-11-deactivate, maintainer=Guillaume Abrioux , GIT_CLEAN=True, io.buildah.version=1.33.5, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0, ceph=True, org.label-schema.license=GPLv2, RELEASE=wip-lusov-quiescer-d69f13d, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS, org.label-schema.build-date=20240416, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, org.label-schema.name=CentOS Stream 8 Base Image) 2024-04-23T10:43:33.896 INFO:journalctl@ceph.osd.11.smithi204.stdout:Apr 23 10:43:33 smithi204 podman[71617]: 2024-04-23 10:43:33.89734 +0000 UTC m=+0.417666348 container init eb2cfc84c1db283aad7ee61428974ed0eee083bf8f4ac3e4a79c5a63f06585cc (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:068a558e3af4932cf1c60dd321c0bdf500075eb52ea1bf455e0973ba114fbfd4, name=ceph-aa240572-0159-11ef-bc93-c7b262605968-osd-11-deactivate, RELEASE=wip-lusov-quiescer-d69f13d, io.buildah.version=1.33.5, ceph=True, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, org.label-schema.name=CentOS Stream 8 Base Image, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240416, org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0, GIT_CLEAN=True) 2024-04-23T10:43:34.193 INFO:journalctl@ceph.osd.11.smithi204.stdout:Apr 23 10:43:33 smithi204 podman[71617]: 2024-04-23 10:43:33.900416159 +0000 UTC m=+0.420742504 container start eb2cfc84c1db283aad7ee61428974ed0eee083bf8f4ac3e4a79c5a63f06585cc (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:068a558e3af4932cf1c60dd321c0bdf500075eb52ea1bf455e0973ba114fbfd4, name=ceph-aa240572-0159-11ef-bc93-c7b262605968-osd-11-deactivate, maintainer=Guillaume Abrioux , GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, GIT_BRANCH=HEAD, io.buildah.version=1.33.5, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.build-date=20240416, org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0, RELEASE=wip-lusov-quiescer-d69f13d, org.label-schema.license=GPLv2, GIT_CLEAN=True, CEPH_POINT_RELEASE=, ceph=True) 2024-04-23T10:43:34.194 INFO:journalctl@ceph.osd.11.smithi204.stdout:Apr 23 10:43:34 smithi204 podman[71617]: 2024-04-23 10:43:34.00583533 +0000 UTC m=+0.526161688 container attach eb2cfc84c1db283aad7ee61428974ed0eee083bf8f4ac3e4a79c5a63f06585cc (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:068a558e3af4932cf1c60dd321c0bdf500075eb52ea1bf455e0973ba114fbfd4, name=ceph-aa240572-0159-11ef-bc93-c7b262605968-osd-11-deactivate, org.label-schema.build-date=20240416, io.buildah.version=1.33.5, GIT_BRANCH=HEAD, ceph=True, maintainer=Guillaume Abrioux , GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, GIT_CLEAN=True, org.label-schema.license=GPLv2, RELEASE=wip-lusov-quiescer-d69f13d, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 8 Base Image, CEPH_POINT_RELEASE=, org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0) 2024-04-23T10:43:34.490 INFO:journalctl@ceph.osd.11.smithi204.stdout:Apr 23 10:43:34 smithi204 podman[71617]: 2024-04-23 10:43:34.19532113 +0000 UTC m=+0.715647467 container died eb2cfc84c1db283aad7ee61428974ed0eee083bf8f4ac3e4a79c5a63f06585cc (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:068a558e3af4932cf1c60dd321c0bdf500075eb52ea1bf455e0973ba114fbfd4, name=ceph-aa240572-0159-11ef-bc93-c7b262605968-osd-11-deactivate, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, io.buildah.version=1.33.5, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 8 Base Image, GIT_CLEAN=True, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0, ceph=True, org.label-schema.build-date=20240416, org.label-schema.license=GPLv2, RELEASE=wip-lusov-quiescer-d69f13d, CEPH_POINT_RELEASE=) 2024-04-23T10:43:34.769 DEBUG:teuthology.orchestra.run.smithi204:> sudo pkill -f 'journalctl -f -n 0 -u ceph-aa240572-0159-11ef-bc93-c7b262605968@osd.11.service' 2024-04-23T10:43:34.806 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-04-23T10:43:34.807 INFO:tasks.cephadm.osd.11:Stopped osd.11 2024-04-23T10:43:34.807 INFO:tasks.cephadm.mds.a:Stopping mds.a... 2024-04-23T10:43:34.807 DEBUG:teuthology.orchestra.run.smithi114:> sudo systemctl stop ceph-aa240572-0159-11ef-bc93-c7b262605968@mds.a 2024-04-23T10:43:35.188 INFO:journalctl@ceph.mds.a.smithi114.stdout:Apr 23 10:43:34 smithi114 systemd[1]: Stopping Ceph mds.a for aa240572-0159-11ef-bc93-c7b262605968... 2024-04-23T10:43:35.504 INFO:journalctl@ceph.mds.a.smithi114.stdout:Apr 23 10:43:35 smithi114 ceph-aa240572-0159-11ef-bc93-c7b262605968-mds-a[68604]: 2024-04-23T10:43:35.189+0000 7fb6a06ab700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-mds -n mds.a -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-04-23T10:43:35.504 INFO:journalctl@ceph.mds.a.smithi114.stdout:Apr 23 10:43:35 smithi114 ceph-aa240572-0159-11ef-bc93-c7b262605968-mds-a[68604]: 2024-04-23T10:43:35.189+0000 7fb6a06ab700 -1 mds.a *** got signal Terminated *** 2024-04-23T10:43:39.446 INFO:journalctl@ceph.mds.a.smithi114.stdout:Apr 23 10:43:39 smithi114 podman[91804]: 2024-04-23 10:43:39.195075157 +0000 UTC m=+4.193449033 container died b4bc3a39121733c1ecb40711b4f4880211c6ddc3656a0c4f1887d99d32bb941a (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:068a558e3af4932cf1c60dd321c0bdf500075eb52ea1bf455e0973ba114fbfd4, name=ceph-aa240572-0159-11ef-bc93-c7b262605968-mds-a, org.label-schema.vendor=CentOS, RELEASE=wip-lusov-quiescer-d69f13d, ceph=True, io.buildah.version=1.33.5, GIT_BRANCH=HEAD, org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=, GIT_CLEAN=True, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 8 Base Image, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, org.label-schema.build-date=20240416, org.label-schema.schema-version=1.0) 2024-04-23T10:43:39.747 INFO:journalctl@ceph.mds.a.smithi114.stdout:Apr 23 10:43:39 smithi114 podman[91804]: 2024-04-23 10:43:39.539057723 +0000 UTC m=+4.537431589 container cleanup b4bc3a39121733c1ecb40711b4f4880211c6ddc3656a0c4f1887d99d32bb941a (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:068a558e3af4932cf1c60dd321c0bdf500075eb52ea1bf455e0973ba114fbfd4, name=ceph-aa240572-0159-11ef-bc93-c7b262605968-mds-a, GIT_BRANCH=HEAD, ceph=True, io.buildah.version=1.33.5, maintainer=Guillaume Abrioux , GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS, GIT_CLEAN=True, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, CEPH_POINT_RELEASE=, RELEASE=wip-lusov-quiescer-d69f13d, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, org.label-schema.build-date=20240416) 2024-04-23T10:43:39.747 INFO:journalctl@ceph.mds.a.smithi114.stdout:Apr 23 10:43:39 smithi114 bash[91804]: ceph-aa240572-0159-11ef-bc93-c7b262605968-mds-a 2024-04-23T10:43:39.961 DEBUG:teuthology.orchestra.run.smithi114:> sudo pkill -f 'journalctl -f -n 0 -u ceph-aa240572-0159-11ef-bc93-c7b262605968@mds.a.service' 2024-04-23T10:43:39.995 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-04-23T10:43:39.995 INFO:tasks.cephadm.mds.a:Stopped mds.a 2024-04-23T10:43:39.996 INFO:tasks.cephadm.mds.d:Stopping mds.d... 2024-04-23T10:43:39.996 DEBUG:teuthology.orchestra.run.smithi114:> sudo systemctl stop ceph-aa240572-0159-11ef-bc93-c7b262605968@mds.d 2024-04-23T10:43:40.333 INFO:journalctl@ceph.mds.d.smithi114.stdout:Apr 23 10:43:40 smithi114 systemd[1]: Stopping Ceph mds.d for aa240572-0159-11ef-bc93-c7b262605968... 2024-04-23T10:43:40.753 INFO:journalctl@ceph.mds.d.smithi114.stdout:Apr 23 10:43:40 smithi114 ceph-aa240572-0159-11ef-bc93-c7b262605968-mds-d[67643]: 2024-04-23T10:43:40.334+0000 7fa6cb082700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-mds -n mds.d -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-04-23T10:43:40.754 INFO:journalctl@ceph.mds.d.smithi114.stdout:Apr 23 10:43:40 smithi114 ceph-aa240572-0159-11ef-bc93-c7b262605968-mds-d[67643]: 2024-04-23T10:43:40.334+0000 7fa6cb082700 -1 mds.d *** got signal Terminated *** 2024-04-23T10:43:44.600 INFO:journalctl@ceph.mds.d.smithi114.stdout:Apr 23 10:43:44 smithi114 podman[91925]: 2024-04-23 10:43:44.340023776 +0000 UTC m=+4.120317683 container died 6fd229fdfcca3f474f5659a8f9c7602d73c060537c379b7946717b0039b87e17 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:068a558e3af4932cf1c60dd321c0bdf500075eb52ea1bf455e0973ba114fbfd4, name=ceph-aa240572-0159-11ef-bc93-c7b262605968-mds-d, GIT_BRANCH=HEAD, GIT_CLEAN=True, ceph=True, io.buildah.version=1.33.5, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, org.label-schema.name=CentOS Stream 8 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=wip-lusov-quiescer-d69f13d, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240416, org.label-schema.vendor=CentOS) 2024-04-23T10:43:44.975 INFO:journalctl@ceph.mds.d.smithi114.stdout:Apr 23 10:43:44 smithi114 podman[91925]: 2024-04-23 10:43:44.701018793 +0000 UTC m=+4.481312701 container cleanup 6fd229fdfcca3f474f5659a8f9c7602d73c060537c379b7946717b0039b87e17 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:068a558e3af4932cf1c60dd321c0bdf500075eb52ea1bf455e0973ba114fbfd4, name=ceph-aa240572-0159-11ef-bc93-c7b262605968-mds-d, GIT_CLEAN=True, maintainer=Guillaume Abrioux , io.buildah.version=1.33.5, GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS, ceph=True, org.label-schema.build-date=20240416, org.label-schema.schema-version=1.0, RELEASE=wip-lusov-quiescer-d69f13d, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 8 Base Image, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67) 2024-04-23T10:43:44.976 INFO:journalctl@ceph.mds.d.smithi114.stdout:Apr 23 10:43:44 smithi114 bash[91925]: ceph-aa240572-0159-11ef-bc93-c7b262605968-mds-d 2024-04-23T10:43:45.085 DEBUG:teuthology.orchestra.run.smithi114:> sudo pkill -f 'journalctl -f -n 0 -u ceph-aa240572-0159-11ef-bc93-c7b262605968@mds.d.service' 2024-04-23T10:43:45.117 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-04-23T10:43:45.117 INFO:tasks.cephadm.mds.d:Stopped mds.d 2024-04-23T10:43:45.117 INFO:tasks.cephadm.mds.g:Stopping mds.g... 2024-04-23T10:43:45.118 DEBUG:teuthology.orchestra.run.smithi114:> sudo systemctl stop ceph-aa240572-0159-11ef-bc93-c7b262605968@mds.g 2024-04-23T10:43:45.254 INFO:journalctl@ceph.mds.g.smithi114.stdout:Apr 23 10:43:45 smithi114 systemd[1]: Stopping Ceph mds.g for aa240572-0159-11ef-bc93-c7b262605968... 2024-04-23T10:43:45.754 INFO:journalctl@ceph.mds.g.smithi114.stdout:Apr 23 10:43:45 smithi114 ceph-aa240572-0159-11ef-bc93-c7b262605968-mds-g[68253]: 2024-04-23T10:43:45.437+0000 7f8211807700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-mds -n mds.g -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-04-23T10:43:45.754 INFO:journalctl@ceph.mds.g.smithi114.stdout:Apr 23 10:43:45 smithi114 ceph-aa240572-0159-11ef-bc93-c7b262605968-mds-g[68253]: 2024-04-23T10:43:45.437+0000 7f8211807700 -1 mds.g *** got signal Terminated *** 2024-04-23T10:43:49.254 INFO:journalctl@ceph.mds.g.smithi114.stdout:Apr 23 10:43:49 smithi114 podman[92044]: 2024-04-23 10:43:49.003580101 +0000 UTC m=+3.660953486 container died 9c754624738df274e46e58ec1fd175dcb4b4ca97ac7da78b9741766662e88227 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:068a558e3af4932cf1c60dd321c0bdf500075eb52ea1bf455e0973ba114fbfd4, name=ceph-aa240572-0159-11ef-bc93-c7b262605968-mds-g, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, GIT_CLEAN=True, RELEASE=wip-lusov-quiescer-d69f13d, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.build-date=20240416, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, ceph=True, org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.33.5, maintainer=Guillaume Abrioux ) 2024-04-23T10:43:49.628 INFO:journalctl@ceph.mds.g.smithi114.stdout:Apr 23 10:43:49 smithi114 podman[92044]: 2024-04-23 10:43:49.370806018 +0000 UTC m=+4.028179458 container cleanup 9c754624738df274e46e58ec1fd175dcb4b4ca97ac7da78b9741766662e88227 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:068a558e3af4932cf1c60dd321c0bdf500075eb52ea1bf455e0973ba114fbfd4, name=ceph-aa240572-0159-11ef-bc93-c7b262605968-mds-g, RELEASE=wip-lusov-quiescer-d69f13d, maintainer=Guillaume Abrioux , io.buildah.version=1.33.5, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, GIT_CLEAN=True, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240416, org.label-schema.license=GPLv2, ceph=True, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 8 Base Image) 2024-04-23T10:43:49.628 INFO:journalctl@ceph.mds.g.smithi114.stdout:Apr 23 10:43:49 smithi114 bash[92044]: ceph-aa240572-0159-11ef-bc93-c7b262605968-mds-g 2024-04-23T10:43:49.794 DEBUG:teuthology.orchestra.run.smithi114:> sudo pkill -f 'journalctl -f -n 0 -u ceph-aa240572-0159-11ef-bc93-c7b262605968@mds.g.service' 2024-04-23T10:43:49.827 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-04-23T10:43:49.827 INFO:tasks.cephadm.mds.g:Stopped mds.g 2024-04-23T10:43:49.827 INFO:tasks.cephadm.mds.j:Stopping mds.j... 2024-04-23T10:43:49.827 DEBUG:teuthology.orchestra.run.smithi114:> sudo systemctl stop ceph-aa240572-0159-11ef-bc93-c7b262605968@mds.j 2024-04-23T10:43:50.139 INFO:journalctl@ceph.mds.j.smithi114.stdout:Apr 23 10:43:49 smithi114 systemd[1]: Stopping Ceph mds.j for aa240572-0159-11ef-bc93-c7b262605968... 2024-04-23T10:43:50.504 INFO:journalctl@ceph.mds.j.smithi114.stdout:Apr 23 10:43:50 smithi114 ceph-aa240572-0159-11ef-bc93-c7b262605968-mds-j[67257]: 2024-04-23T10:43:50.140+0000 7fe797f95700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-mds -n mds.j -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-04-23T10:43:50.504 INFO:journalctl@ceph.mds.j.smithi114.stdout:Apr 23 10:43:50 smithi114 ceph-aa240572-0159-11ef-bc93-c7b262605968-mds-j[67257]: 2024-04-23T10:43:50.140+0000 7fe797f95700 -1 mds.j *** got signal Terminated *** 2024-04-23T10:43:53.890 INFO:journalctl@ceph.mds.j.smithi114.stdout:Apr 23 10:43:53 smithi114 podman[92166]: 2024-04-23 10:43:53.634786286 +0000 UTC m=+3.594237677 container died 383afeb57c86c4986b4fbb48e40044e048c5fe4426ad7fb6e6913f5ab2228fb1 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:068a558e3af4932cf1c60dd321c0bdf500075eb52ea1bf455e0973ba114fbfd4, name=ceph-aa240572-0159-11ef-bc93-c7b262605968-mds-j, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 8 Base Image, io.buildah.version=1.33.5, CEPH_POINT_RELEASE=, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, RELEASE=wip-lusov-quiescer-d69f13d, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, org.label-schema.schema-version=1.0, GIT_CLEAN=True, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240416, org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2) 2024-04-23T10:43:54.253 INFO:journalctl@ceph.mds.j.smithi114.stdout:Apr 23 10:43:53 smithi114 podman[92166]: 2024-04-23 10:43:53.991198856 +0000 UTC m=+3.950650245 container cleanup 383afeb57c86c4986b4fbb48e40044e048c5fe4426ad7fb6e6913f5ab2228fb1 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:068a558e3af4932cf1c60dd321c0bdf500075eb52ea1bf455e0973ba114fbfd4, name=ceph-aa240572-0159-11ef-bc93-c7b262605968-mds-j, io.buildah.version=1.33.5, org.label-schema.license=GPLv2, RELEASE=wip-lusov-quiescer-d69f13d, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, GIT_BRANCH=HEAD, org.label-schema.build-date=20240416, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 8 Base Image, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, CEPH_POINT_RELEASE=) 2024-04-23T10:43:54.254 INFO:journalctl@ceph.mds.j.smithi114.stdout:Apr 23 10:43:53 smithi114 bash[92166]: ceph-aa240572-0159-11ef-bc93-c7b262605968-mds-j 2024-04-23T10:43:54.409 DEBUG:teuthology.orchestra.run.smithi114:> sudo pkill -f 'journalctl -f -n 0 -u ceph-aa240572-0159-11ef-bc93-c7b262605968@mds.j.service' 2024-04-23T10:43:54.439 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-04-23T10:43:54.439 INFO:tasks.cephadm.mds.j:Stopped mds.j 2024-04-23T10:43:54.440 INFO:tasks.cephadm.mds.b:Stopping mds.b... 2024-04-23T10:43:54.440 DEBUG:teuthology.orchestra.run.smithi119:> sudo systemctl stop ceph-aa240572-0159-11ef-bc93-c7b262605968@mds.b 2024-04-23T10:43:54.742 INFO:journalctl@ceph.mds.b.smithi119.stdout:Apr 23 10:43:54 smithi119 systemd[1]: Stopping Ceph mds.b for aa240572-0159-11ef-bc93-c7b262605968... 2024-04-23T10:43:55.003 INFO:journalctl@ceph.mds.b.smithi119.stdout:Apr 23 10:43:54 smithi119 ceph-aa240572-0159-11ef-bc93-c7b262605968-mds-b[62689]: 2024-04-23T10:43:54.831+0000 7fd52655d700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-mds -n mds.b -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-04-23T10:43:55.003 INFO:journalctl@ceph.mds.b.smithi119.stdout:Apr 23 10:43:54 smithi119 ceph-aa240572-0159-11ef-bc93-c7b262605968-mds-b[62689]: 2024-04-23T10:43:54.831+0000 7fd52655d700 -1 mds.b *** got signal Terminated *** 2024-04-23T10:43:56.640 INFO:journalctl@ceph.mds.b.smithi119.stdout:Apr 23 10:43:56 smithi119 podman[70286]: 2024-04-23 10:43:56.34995575 +0000 UTC m=+1.709937655 container died c3c18069909792cef80b3deb85df8e647afc0cad3b2249403961570a05f53d8e (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:068a558e3af4932cf1c60dd321c0bdf500075eb52ea1bf455e0973ba114fbfd4, name=ceph-aa240572-0159-11ef-bc93-c7b262605968-mds-b, GIT_BRANCH=HEAD, io.buildah.version=1.33.5, maintainer=Guillaume Abrioux , GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, ceph=True, GIT_CLEAN=True, org.label-schema.build-date=20240416, org.label-schema.name=CentOS Stream 8 Base Image, RELEASE=wip-lusov-quiescer-d69f13d) 2024-04-23T10:43:56.991 INFO:journalctl@ceph.mds.b.smithi119.stdout:Apr 23 10:43:56 smithi119 podman[70286]: 2024-04-23 10:43:56.750327412 +0000 UTC m=+2.110309369 container cleanup c3c18069909792cef80b3deb85df8e647afc0cad3b2249403961570a05f53d8e (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:068a558e3af4932cf1c60dd321c0bdf500075eb52ea1bf455e0973ba114fbfd4, name=ceph-aa240572-0159-11ef-bc93-c7b262605968-mds-b, io.buildah.version=1.33.5, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_BRANCH=HEAD, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, org.label-schema.name=CentOS Stream 8 Base Image, ceph=True, CEPH_POINT_RELEASE=, GIT_CLEAN=True, maintainer=Guillaume Abrioux , RELEASE=wip-lusov-quiescer-d69f13d, org.label-schema.build-date=20240416, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2) 2024-04-23T10:43:56.991 INFO:journalctl@ceph.mds.b.smithi119.stdout:Apr 23 10:43:56 smithi119 bash[70286]: ceph-aa240572-0159-11ef-bc93-c7b262605968-mds-b 2024-04-23T10:43:57.104 DEBUG:teuthology.orchestra.run.smithi119:> sudo pkill -f 'journalctl -f -n 0 -u ceph-aa240572-0159-11ef-bc93-c7b262605968@mds.b.service' 2024-04-23T10:43:57.138 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-04-23T10:43:57.138 INFO:tasks.cephadm.mds.b:Stopped mds.b 2024-04-23T10:43:57.138 INFO:tasks.cephadm.mds.e:Stopping mds.e... 2024-04-23T10:43:57.139 DEBUG:teuthology.orchestra.run.smithi119:> sudo systemctl stop ceph-aa240572-0159-11ef-bc93-c7b262605968@mds.e 2024-04-23T10:43:57.241 INFO:journalctl@ceph.mds.e.smithi119.stdout:Apr 23 10:43:57 smithi119 systemd[1]: Stopping Ceph mds.e for aa240572-0159-11ef-bc93-c7b262605968... 2024-04-23T10:43:57.742 INFO:journalctl@ceph.mds.e.smithi119.stdout:Apr 23 10:43:57 smithi119 ceph-aa240572-0159-11ef-bc93-c7b262605968-mds-e[62063]: 2024-04-23T10:43:57.462+0000 7f37cc90d700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-mds -n mds.e -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-04-23T10:43:57.742 INFO:journalctl@ceph.mds.e.smithi119.stdout:Apr 23 10:43:57 smithi119 ceph-aa240572-0159-11ef-bc93-c7b262605968-mds-e[62063]: 2024-04-23T10:43:57.462+0000 7f37cc90d700 -1 mds.e *** got signal Terminated *** 2024-04-23T10:44:00.679 INFO:journalctl@ceph.mds.e.smithi119.stdout:Apr 23 10:44:00 smithi119 podman[70420]: 2024-04-23 10:44:00.353128152 +0000 UTC m=+2.993950005 container died d5e5a1bdb2a73aa81c32c18b1ed7110745620f6bcb082deb1c33a64c184a08b7 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:068a558e3af4932cf1c60dd321c0bdf500075eb52ea1bf455e0973ba114fbfd4, name=ceph-aa240572-0159-11ef-bc93-c7b262605968-mds-e, org.label-schema.vendor=CentOS, RELEASE=wip-lusov-quiescer-d69f13d, org.label-schema.schema-version=1.0, org.label-schema.license=GPLv2, ceph=True, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=, org.label-schema.build-date=20240416, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, org.label-schema.name=CentOS Stream 8 Base Image, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.33.5, GIT_CLEAN=True) 2024-04-23T10:44:00.992 INFO:journalctl@ceph.mds.e.smithi119.stdout:Apr 23 10:44:00 smithi119 podman[70420]: 2024-04-23 10:44:00.762944946 +0000 UTC m=+3.403766841 container cleanup d5e5a1bdb2a73aa81c32c18b1ed7110745620f6bcb082deb1c33a64c184a08b7 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:068a558e3af4932cf1c60dd321c0bdf500075eb52ea1bf455e0973ba114fbfd4, name=ceph-aa240572-0159-11ef-bc93-c7b262605968-mds-e, ceph=True, org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, io.buildah.version=1.33.5, org.label-schema.build-date=20240416, org.label-schema.name=CentOS Stream 8 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=, GIT_CLEAN=True, RELEASE=wip-lusov-quiescer-d69f13d) 2024-04-23T10:44:00.992 INFO:journalctl@ceph.mds.e.smithi119.stdout:Apr 23 10:44:00 smithi119 bash[70420]: ceph-aa240572-0159-11ef-bc93-c7b262605968-mds-e 2024-04-23T10:44:01.121 DEBUG:teuthology.orchestra.run.smithi119:> sudo pkill -f 'journalctl -f -n 0 -u ceph-aa240572-0159-11ef-bc93-c7b262605968@mds.e.service' 2024-04-23T10:44:01.152 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-04-23T10:44:01.152 INFO:tasks.cephadm.mds.e:Stopped mds.e 2024-04-23T10:44:01.153 INFO:tasks.cephadm.mds.h:Stopping mds.h... 2024-04-23T10:44:01.153 DEBUG:teuthology.orchestra.run.smithi119:> sudo systemctl stop ceph-aa240572-0159-11ef-bc93-c7b262605968@mds.h 2024-04-23T10:44:01.473 INFO:journalctl@ceph.mds.h.smithi119.stdout:Apr 23 10:44:01 smithi119 systemd[1]: Stopping Ceph mds.h for aa240572-0159-11ef-bc93-c7b262605968... 2024-04-23T10:44:01.741 INFO:journalctl@ceph.mds.h.smithi119.stdout:Apr 23 10:44:01 smithi119 ceph-aa240572-0159-11ef-bc93-c7b262605968-mds-h[62388]: 2024-04-23T10:44:01.474+0000 7f66819c9700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-mds -n mds.h -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-04-23T10:44:01.741 INFO:journalctl@ceph.mds.h.smithi119.stdout:Apr 23 10:44:01 smithi119 ceph-aa240572-0159-11ef-bc93-c7b262605968-mds-h[62388]: 2024-04-23T10:44:01.474+0000 7f66819c9700 -1 mds.h *** got signal Terminated *** 2024-04-23T10:44:04.742 INFO:journalctl@ceph.mds.h.smithi119.stdout:Apr 23 10:44:04 smithi119 podman[70540]: 2024-04-23 10:44:04.331443287 +0000 UTC m=+2.958680581 container died 97a4ae8c500223d3a62979d0cbfc9a5668b8348b50dfb9379078b80ff3e7eeb6 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:068a558e3af4932cf1c60dd321c0bdf500075eb52ea1bf455e0973ba114fbfd4, name=ceph-aa240572-0159-11ef-bc93-c7b262605968-mds-h, RELEASE=wip-lusov-quiescer-d69f13d, org.label-schema.license=GPLv2, GIT_CLEAN=True, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, ceph=True, org.label-schema.name=CentOS Stream 8 Base Image, io.buildah.version=1.33.5, org.label-schema.build-date=20240416, org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=) 2024-04-23T10:44:05.058 INFO:journalctl@ceph.mds.h.smithi119.stdout:Apr 23 10:44:04 smithi119 podman[70540]: 2024-04-23 10:44:04.842010202 +0000 UTC m=+3.469247495 container cleanup 97a4ae8c500223d3a62979d0cbfc9a5668b8348b50dfb9379078b80ff3e7eeb6 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:068a558e3af4932cf1c60dd321c0bdf500075eb52ea1bf455e0973ba114fbfd4, name=ceph-aa240572-0159-11ef-bc93-c7b262605968-mds-h, org.label-schema.license=GPLv2, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, ceph=True, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240416, GIT_CLEAN=True, GIT_BRANCH=HEAD, RELEASE=wip-lusov-quiescer-d69f13d, io.buildah.version=1.33.5, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 8 Base Image) 2024-04-23T10:44:05.058 INFO:journalctl@ceph.mds.h.smithi119.stdout:Apr 23 10:44:04 smithi119 bash[70540]: ceph-aa240572-0159-11ef-bc93-c7b262605968-mds-h 2024-04-23T10:44:05.184 DEBUG:teuthology.orchestra.run.smithi119:> sudo pkill -f 'journalctl -f -n 0 -u ceph-aa240572-0159-11ef-bc93-c7b262605968@mds.h.service' 2024-04-23T10:44:05.218 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-04-23T10:44:05.218 INFO:tasks.cephadm.mds.h:Stopped mds.h 2024-04-23T10:44:05.218 INFO:tasks.cephadm.mds.k:Stopping mds.k... 2024-04-23T10:44:05.218 DEBUG:teuthology.orchestra.run.smithi119:> sudo systemctl stop ceph-aa240572-0159-11ef-bc93-c7b262605968@mds.k 2024-04-23T10:44:05.426 INFO:journalctl@ceph.mds.k.smithi119.stdout:Apr 23 10:44:05 smithi119 systemd[1]: Stopping Ceph mds.k for aa240572-0159-11ef-bc93-c7b262605968... 2024-04-23T10:44:05.741 INFO:journalctl@ceph.mds.k.smithi119.stdout:Apr 23 10:44:05 smithi119 ceph-aa240572-0159-11ef-bc93-c7b262605968-mds-k[61746]: 2024-04-23T10:44:05.545+0000 7faf44cb3700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-mds -n mds.k -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-04-23T10:44:05.742 INFO:journalctl@ceph.mds.k.smithi119.stdout:Apr 23 10:44:05 smithi119 ceph-aa240572-0159-11ef-bc93-c7b262605968-mds-k[61746]: 2024-04-23T10:44:05.545+0000 7faf44cb3700 -1 mds.k *** got signal Terminated *** 2024-04-23T10:44:08.603 INFO:journalctl@ceph.mds.k.smithi119.stdout:Apr 23 10:44:08 smithi119 podman[70661]: 2024-04-23 10:44:08.342663614 +0000 UTC m=+2.898968192 container died ef9a53e050a9ec51cd1080c4cb18e56466bd97e8d997e444ca4d15b372981bcd (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:068a558e3af4932cf1c60dd321c0bdf500075eb52ea1bf455e0973ba114fbfd4, name=ceph-aa240572-0159-11ef-bc93-c7b262605968-mds-k, org.label-schema.schema-version=1.0, GIT_CLEAN=True, CEPH_POINT_RELEASE=, ceph=True, org.label-schema.license=GPLv2, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 8 Base Image, RELEASE=wip-lusov-quiescer-d69f13d, io.buildah.version=1.33.5, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_BRANCH=HEAD, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, org.label-schema.vendor=CentOS, org.label-schema.build-date=20240416) 2024-04-23T10:44:08.992 INFO:journalctl@ceph.mds.k.smithi119.stdout:Apr 23 10:44:08 smithi119 podman[70661]: 2024-04-23 10:44:08.712873444 +0000 UTC m=+3.269178015 container cleanup ef9a53e050a9ec51cd1080c4cb18e56466bd97e8d997e444ca4d15b372981bcd (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:068a558e3af4932cf1c60dd321c0bdf500075eb52ea1bf455e0973ba114fbfd4, name=ceph-aa240572-0159-11ef-bc93-c7b262605968-mds-k, GIT_BRANCH=HEAD, RELEASE=wip-lusov-quiescer-d69f13d, org.label-schema.vendor=CentOS, io.buildah.version=1.33.5, org.label-schema.license=GPLv2, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240416, org.label-schema.name=CentOS Stream 8 Base Image, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=, ceph=True, org.label-schema.schema-version=1.0) 2024-04-23T10:44:08.992 INFO:journalctl@ceph.mds.k.smithi119.stdout:Apr 23 10:44:08 smithi119 bash[70661]: ceph-aa240572-0159-11ef-bc93-c7b262605968-mds-k 2024-04-23T10:44:09.136 DEBUG:teuthology.orchestra.run.smithi119:> sudo pkill -f 'journalctl -f -n 0 -u ceph-aa240572-0159-11ef-bc93-c7b262605968@mds.k.service' 2024-04-23T10:44:09.167 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-04-23T10:44:09.167 INFO:tasks.cephadm.mds.k:Stopped mds.k 2024-04-23T10:44:09.167 INFO:tasks.cephadm.mds.c:Stopping mds.c... 2024-04-23T10:44:09.168 DEBUG:teuthology.orchestra.run.smithi204:> sudo systemctl stop ceph-aa240572-0159-11ef-bc93-c7b262605968@mds.c 2024-04-23T10:44:09.579 INFO:journalctl@ceph.mds.c.smithi204.stdout:Apr 23 10:44:09 smithi204 systemd[1]: Stopping Ceph mds.c for aa240572-0159-11ef-bc93-c7b262605968... 2024-04-23T10:44:10.078 INFO:journalctl@ceph.mds.c.smithi204.stdout:Apr 23 10:44:09 smithi204 ceph-aa240572-0159-11ef-bc93-c7b262605968-mds-c[62990]: 2024-04-23T10:44:09.727+0000 7fe9dc8fe700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-mds -n mds.c -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-04-23T10:44:10.078 INFO:journalctl@ceph.mds.c.smithi204.stdout:Apr 23 10:44:09 smithi204 ceph-aa240572-0159-11ef-bc93-c7b262605968-mds-c[62990]: 2024-04-23T10:44:09.727+0000 7fe9dc8fe700 -1 mds.c *** got signal Terminated *** 2024-04-23T10:44:12.618 INFO:journalctl@ceph.mds.c.smithi204.stdout:Apr 23 10:44:12 smithi204 podman[71723]: 2024-04-23 10:44:12.334559357 +0000 UTC m=+2.973624453 container died 084acf2b16c700cde87a30383114cd49b4d1e8dfe3c3df1fa7a6952468a190b9 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:068a558e3af4932cf1c60dd321c0bdf500075eb52ea1bf455e0973ba114fbfd4, name=ceph-aa240572-0159-11ef-bc93-c7b262605968-mds-c, GIT_CLEAN=True, ceph=True, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, RELEASE=wip-lusov-quiescer-d69f13d, org.label-schema.build-date=20240416, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.schema-version=1.0, io.buildah.version=1.33.5) 2024-04-23T10:44:12.976 INFO:journalctl@ceph.mds.c.smithi204.stdout:Apr 23 10:44:12 smithi204 podman[71723]: 2024-04-23 10:44:12.718447555 +0000 UTC m=+3.357512644 container cleanup 084acf2b16c700cde87a30383114cd49b4d1e8dfe3c3df1fa7a6952468a190b9 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:068a558e3af4932cf1c60dd321c0bdf500075eb52ea1bf455e0973ba114fbfd4, name=ceph-aa240572-0159-11ef-bc93-c7b262605968-mds-c, org.label-schema.build-date=20240416, io.buildah.version=1.33.5, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, ceph=True, org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=wip-lusov-quiescer-d69f13d, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, GIT_CLEAN=True) 2024-04-23T10:44:12.976 INFO:journalctl@ceph.mds.c.smithi204.stdout:Apr 23 10:44:12 smithi204 bash[71723]: ceph-aa240572-0159-11ef-bc93-c7b262605968-mds-c 2024-04-23T10:44:13.138 DEBUG:teuthology.orchestra.run.smithi204:> sudo pkill -f 'journalctl -f -n 0 -u ceph-aa240572-0159-11ef-bc93-c7b262605968@mds.c.service' 2024-04-23T10:44:13.172 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-04-23T10:44:13.173 INFO:tasks.cephadm.mds.c:Stopped mds.c 2024-04-23T10:44:13.173 INFO:tasks.cephadm.mds.f:Stopping mds.f... 2024-04-23T10:44:13.173 DEBUG:teuthology.orchestra.run.smithi204:> sudo systemctl stop ceph-aa240572-0159-11ef-bc93-c7b262605968@mds.f 2024-04-23T10:44:13.487 INFO:journalctl@ceph.mds.f.smithi204.stdout:Apr 23 10:44:13 smithi204 systemd[1]: Stopping Ceph mds.f for aa240572-0159-11ef-bc93-c7b262605968... 2024-04-23T10:44:13.828 INFO:journalctl@ceph.mds.f.smithi204.stdout:Apr 23 10:44:13 smithi204 ceph-aa240572-0159-11ef-bc93-c7b262605968-mds-f[62678]: 2024-04-23T10:44:13.487+0000 7f51638b1700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-mds -n mds.f -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-04-23T10:44:13.828 INFO:journalctl@ceph.mds.f.smithi204.stdout:Apr 23 10:44:13 smithi204 ceph-aa240572-0159-11ef-bc93-c7b262605968-mds-f[62678]: 2024-04-23T10:44:13.487+0000 7f51638b1700 -1 mds.f *** got signal Terminated *** 2024-04-23T10:44:17.820 INFO:journalctl@ceph.mds.f.smithi204.stdout:Apr 23 10:44:17 smithi204 podman[71855]: 2024-04-23 10:44:17.495971608 +0000 UTC m=+4.107157633 container died 4fd9d2764c4fb78010077cffd6d0ec8d8ec5fcfe8503b3bd2e85eac2f95b4010 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:068a558e3af4932cf1c60dd321c0bdf500075eb52ea1bf455e0973ba114fbfd4, name=ceph-aa240572-0159-11ef-bc93-c7b262605968-mds-f, GIT_BRANCH=HEAD, ceph=True, RELEASE=wip-lusov-quiescer-d69f13d, org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.33.5, GIT_CLEAN=True, org.label-schema.vendor=CentOS, org.label-schema.build-date=20240416, CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 8 Base Image, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, org.label-schema.license=GPLv2, maintainer=Guillaume Abrioux ) 2024-04-23T10:44:18.078 INFO:journalctl@ceph.mds.f.smithi204.stdout:Apr 23 10:44:18 smithi204 podman[71855]: 2024-04-23 10:44:18.005569357 +0000 UTC m=+4.616755381 container cleanup 4fd9d2764c4fb78010077cffd6d0ec8d8ec5fcfe8503b3bd2e85eac2f95b4010 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:068a558e3af4932cf1c60dd321c0bdf500075eb52ea1bf455e0973ba114fbfd4, name=ceph-aa240572-0159-11ef-bc93-c7b262605968-mds-f, ceph=True, CEPH_POINT_RELEASE=, org.label-schema.build-date=20240416, org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.vendor=CentOS, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, io.buildah.version=1.33.5, GIT_CLEAN=True, RELEASE=wip-lusov-quiescer-d69f13d, org.label-schema.schema-version=1.0) 2024-04-23T10:44:18.078 INFO:journalctl@ceph.mds.f.smithi204.stdout:Apr 23 10:44:18 smithi204 bash[71855]: ceph-aa240572-0159-11ef-bc93-c7b262605968-mds-f 2024-04-23T10:44:18.438 DEBUG:teuthology.orchestra.run.smithi204:> sudo pkill -f 'journalctl -f -n 0 -u ceph-aa240572-0159-11ef-bc93-c7b262605968@mds.f.service' 2024-04-23T10:44:18.470 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-04-23T10:44:18.471 INFO:tasks.cephadm.mds.f:Stopped mds.f 2024-04-23T10:44:18.471 INFO:tasks.cephadm.mds.i:Stopping mds.i... 2024-04-23T10:44:18.471 DEBUG:teuthology.orchestra.run.smithi204:> sudo systemctl stop ceph-aa240572-0159-11ef-bc93-c7b262605968@mds.i 2024-04-23T10:44:18.790 INFO:journalctl@ceph.mds.i.smithi204.stdout:Apr 23 10:44:18 smithi204 systemd[1]: Stopping Ceph mds.i for aa240572-0159-11ef-bc93-c7b262605968... 2024-04-23T10:44:19.077 INFO:journalctl@ceph.mds.i.smithi204.stdout:Apr 23 10:44:18 smithi204 ceph-aa240572-0159-11ef-bc93-c7b262605968-mds-i[63304]: 2024-04-23T10:44:18.791+0000 7f605e6bc700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-mds -n mds.i -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-04-23T10:44:19.078 INFO:journalctl@ceph.mds.i.smithi204.stdout:Apr 23 10:44:18 smithi204 ceph-aa240572-0159-11ef-bc93-c7b262605968-mds-i[63304]: 2024-04-23T10:44:18.791+0000 7f605e6bc700 -1 mds.i *** got signal Terminated *** 2024-04-23T10:44:20.683 INFO:journalctl@ceph.mds.i.smithi204.stdout:Apr 23 10:44:20 smithi204 podman[71978]: 2024-04-23 10:44:20.339294965 +0000 UTC m=+1.643981373 container died 6e31504a857cb22736200eb8bd364a6ae52fae3d213bde84ebb24ffddd2652a3 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:068a558e3af4932cf1c60dd321c0bdf500075eb52ea1bf455e0973ba114fbfd4, name=ceph-aa240572-0159-11ef-bc93-c7b262605968-mds-i, org.label-schema.name=CentOS Stream 8 Base Image, io.buildah.version=1.33.5, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, GIT_CLEAN=True, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=wip-lusov-quiescer-d69f13d, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, maintainer=Guillaume Abrioux , ceph=True, org.label-schema.build-date=20240416) 2024-04-23T10:44:20.961 INFO:journalctl@ceph.mds.i.smithi204.stdout:Apr 23 10:44:20 smithi204 podman[71978]: 2024-04-23 10:44:20.759305673 +0000 UTC m=+2.063992079 container cleanup 6e31504a857cb22736200eb8bd364a6ae52fae3d213bde84ebb24ffddd2652a3 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:068a558e3af4932cf1c60dd321c0bdf500075eb52ea1bf455e0973ba114fbfd4, name=ceph-aa240572-0159-11ef-bc93-c7b262605968-mds-i, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, RELEASE=wip-lusov-quiescer-d69f13d, ceph=True, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.license=GPLv2, io.buildah.version=1.33.5, org.label-schema.build-date=20240416, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, maintainer=Guillaume Abrioux , GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_CLEAN=True, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD) 2024-04-23T10:44:20.962 INFO:journalctl@ceph.mds.i.smithi204.stdout:Apr 23 10:44:20 smithi204 bash[71978]: ceph-aa240572-0159-11ef-bc93-c7b262605968-mds-i 2024-04-23T10:44:21.179 DEBUG:teuthology.orchestra.run.smithi204:> sudo pkill -f 'journalctl -f -n 0 -u ceph-aa240572-0159-11ef-bc93-c7b262605968@mds.i.service' 2024-04-23T10:44:21.211 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-04-23T10:44:21.211 INFO:tasks.cephadm.mds.i:Stopped mds.i 2024-04-23T10:44:21.211 INFO:tasks.cephadm.mds.l:Stopping mds.l... 2024-04-23T10:44:21.211 DEBUG:teuthology.orchestra.run.smithi204:> sudo systemctl stop ceph-aa240572-0159-11ef-bc93-c7b262605968@mds.l 2024-04-23T10:44:21.561 INFO:journalctl@ceph.mds.l.smithi204.stdout:Apr 23 10:44:21 smithi204 systemd[1]: Stopping Ceph mds.l for aa240572-0159-11ef-bc93-c7b262605968... 2024-04-23T10:44:21.828 INFO:journalctl@ceph.mds.l.smithi204.stdout:Apr 23 10:44:21 smithi204 ceph-aa240572-0159-11ef-bc93-c7b262605968-mds-l[62372]: 2024-04-23T10:44:21.562+0000 7f247a4a6700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-mds -n mds.l -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-04-23T10:44:21.828 INFO:journalctl@ceph.mds.l.smithi204.stdout:Apr 23 10:44:21 smithi204 ceph-aa240572-0159-11ef-bc93-c7b262605968-mds-l[62372]: 2024-04-23T10:44:21.562+0000 7f247a4a6700 -1 mds.l *** got signal Terminated *** 2024-04-23T10:44:24.445 INFO:journalctl@ceph.mds.l.smithi204.stdout:Apr 23 10:44:24 smithi204 podman[72101]: 2024-04-23 10:44:24.165172893 +0000 UTC m=+2.737832417 container died ffce6fd328936cd910c50e0e68d0bb3ac574b03fff9108e0718d8090fdb12017 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:068a558e3af4932cf1c60dd321c0bdf500075eb52ea1bf455e0973ba114fbfd4, name=ceph-aa240572-0159-11ef-bc93-c7b262605968-mds-l, RELEASE=wip-lusov-quiescer-d69f13d, GIT_BRANCH=HEAD, org.label-schema.license=GPLv2, ceph=True, GIT_CLEAN=True, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux , GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.33.5, CEPH_POINT_RELEASE=, org.label-schema.build-date=20240416, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.schema-version=1.0) 2024-04-23T10:44:24.770 INFO:journalctl@ceph.mds.l.smithi204.stdout:Apr 23 10:44:24 smithi204 podman[72101]: 2024-04-23 10:44:24.530494713 +0000 UTC m=+3.103154237 container cleanup ffce6fd328936cd910c50e0e68d0bb3ac574b03fff9108e0718d8090fdb12017 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:068a558e3af4932cf1c60dd321c0bdf500075eb52ea1bf455e0973ba114fbfd4, name=ceph-aa240572-0159-11ef-bc93-c7b262605968-mds-l, CEPH_POINT_RELEASE=, org.label-schema.build-date=20240416, org.label-schema.vendor=CentOS, ceph=True, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 8 Base Image, RELEASE=wip-lusov-quiescer-d69f13d, io.buildah.version=1.33.5, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, GIT_CLEAN=True) 2024-04-23T10:44:24.771 INFO:journalctl@ceph.mds.l.smithi204.stdout:Apr 23 10:44:24 smithi204 bash[72101]: ceph-aa240572-0159-11ef-bc93-c7b262605968-mds-l 2024-04-23T10:44:24.900 DEBUG:teuthology.orchestra.run.smithi204:> sudo pkill -f 'journalctl -f -n 0 -u ceph-aa240572-0159-11ef-bc93-c7b262605968@mds.l.service' 2024-04-23T10:44:24.930 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-04-23T10:44:24.930 INFO:tasks.cephadm.mds.l:Stopped mds.l 2024-04-23T10:44:24.931 DEBUG:teuthology.orchestra.run.smithi114:> sudo /home/ubuntu/cephtest/cephadm rm-cluster --fsid aa240572-0159-11ef-bc93-c7b262605968 --force --keep-logs 2024-04-23T10:44:25.119 INFO:teuthology.orchestra.run.smithi114.stdout:Deleting cluster with fsid: aa240572-0159-11ef-bc93-c7b262605968 2024-04-23T10:44:27.581 DEBUG:teuthology.orchestra.run.smithi119:> sudo /home/ubuntu/cephtest/cephadm rm-cluster --fsid aa240572-0159-11ef-bc93-c7b262605968 --force --keep-logs 2024-04-23T10:44:27.769 INFO:teuthology.orchestra.run.smithi119.stdout:Deleting cluster with fsid: aa240572-0159-11ef-bc93-c7b262605968 2024-04-23T10:44:30.369 DEBUG:teuthology.orchestra.run.smithi204:> sudo /home/ubuntu/cephtest/cephadm rm-cluster --fsid aa240572-0159-11ef-bc93-c7b262605968 --force --keep-logs 2024-04-23T10:44:30.558 INFO:teuthology.orchestra.run.smithi204.stdout:Deleting cluster with fsid: aa240572-0159-11ef-bc93-c7b262605968 2024-04-23T10:44:32.862 DEBUG:teuthology.orchestra.run.smithi114:> sudo rm -f /etc/ceph/ceph.conf /etc/ceph/ceph.client.admin.keyring 2024-04-23T10:44:32.894 DEBUG:teuthology.orchestra.run.smithi119:> sudo rm -f /etc/ceph/ceph.conf /etc/ceph/ceph.client.admin.keyring 2024-04-23T10:44:32.925 DEBUG:teuthology.orchestra.run.smithi204:> sudo rm -f /etc/ceph/ceph.conf /etc/ceph/ceph.client.admin.keyring 2024-04-23T10:44:32.951 INFO:tasks.cephadm:Archiving crash dumps... 2024-04-23T10:44:32.953 DEBUG:teuthology.misc:Transferring archived files from smithi114:/var/lib/ceph/aa240572-0159-11ef-bc93-c7b262605968/crash to /home/teuthworker/archive/leonidus-2024-04-23_09:49:14-fs-wip-lusov-quiescer-distro-default-smithi/7669513/remote/smithi114/crash 2024-04-23T10:44:32.954 DEBUG:teuthology.orchestra.run.smithi114:> sudo tar c -f - -C /var/lib/ceph/aa240572-0159-11ef-bc93-c7b262605968/crash -- . 2024-04-23T10:44:33.019 INFO:teuthology.orchestra.run.smithi114.stderr:tar: /var/lib/ceph/aa240572-0159-11ef-bc93-c7b262605968/crash: Cannot open: No such file or directory 2024-04-23T10:44:33.019 INFO:teuthology.orchestra.run.smithi114.stderr:tar: Error is not recoverable: exiting now 2024-04-23T10:44:33.021 DEBUG:teuthology.misc:Transferring archived files from smithi119:/var/lib/ceph/aa240572-0159-11ef-bc93-c7b262605968/crash to /home/teuthworker/archive/leonidus-2024-04-23_09:49:14-fs-wip-lusov-quiescer-distro-default-smithi/7669513/remote/smithi119/crash 2024-04-23T10:44:33.022 DEBUG:teuthology.orchestra.run.smithi119:> sudo tar c -f - -C /var/lib/ceph/aa240572-0159-11ef-bc93-c7b262605968/crash -- . 2024-04-23T10:44:33.060 INFO:teuthology.orchestra.run.smithi119.stderr:tar: /var/lib/ceph/aa240572-0159-11ef-bc93-c7b262605968/crash: Cannot open: No such file or directory 2024-04-23T10:44:33.061 INFO:teuthology.orchestra.run.smithi119.stderr:tar: Error is not recoverable: exiting now 2024-04-23T10:44:33.063 DEBUG:teuthology.misc:Transferring archived files from smithi204:/var/lib/ceph/aa240572-0159-11ef-bc93-c7b262605968/crash to /home/teuthworker/archive/leonidus-2024-04-23_09:49:14-fs-wip-lusov-quiescer-distro-default-smithi/7669513/remote/smithi204/crash 2024-04-23T10:44:33.064 DEBUG:teuthology.orchestra.run.smithi204:> sudo tar c -f - -C /var/lib/ceph/aa240572-0159-11ef-bc93-c7b262605968/crash -- . 2024-04-23T10:44:33.109 INFO:teuthology.orchestra.run.smithi204.stderr:tar: /var/lib/ceph/aa240572-0159-11ef-bc93-c7b262605968/crash: Cannot open: No such file or directory 2024-04-23T10:44:33.110 INFO:teuthology.orchestra.run.smithi204.stderr:tar: Error is not recoverable: exiting now 2024-04-23T10:44:33.111 INFO:tasks.cephadm:Checking cluster log for badness... 2024-04-23T10:44:33.111 DEBUG:teuthology.orchestra.run.smithi114:> sudo egrep '\[ERR\]|\[WRN\]|\[SEC\]' /var/log/ceph/aa240572-0159-11ef-bc93-c7b262605968/ceph.log | egrep -v '\(MDS_ALL_DOWN\)' | egrep -v '\(MDS_UP_LESS_THAN_MAX\)' | egrep -v FS_DEGRADED | egrep -v FS_INLINE_DATA_DEPRECATED | egrep -v FS_WITH_FAILED_MDS | egrep -v MDS_ALL_DOWN | egrep -v MDS_DAMAGE | egrep -v MDS_DEGRADED | egrep -v MDS_FAILED | egrep -v MDS_INSUFFICIENT_STANDBY | egrep -v MDS_UP_LESS_THAN_MAX | egrep -v 'filesystem is online with fewer MDS than max_mds' | egrep -v POOL_APP_NOT_ENABLED | egrep -v 'do not have an application enabled' | egrep -v 'overall HEALTH_' | egrep -v 'Replacing daemon' | egrep -v 'deprecated feature inline_data' | egrep -v 'overall HEALTH_' | egrep -v '\(OSD_DOWN\)' | egrep -v '\(OSD_' | egrep -v 'but it is still running' | egrep -v 'is not responding' | egrep -v PG_AVAILABILITY | egrep -v PG_DEGRADED | egrep -v 'Reduced data availability' | head -n 1 2024-04-23T10:44:33.148 INFO:tasks.cephadm:Compressing logs... 2024-04-23T10:44:33.148 DEBUG:teuthology.orchestra.run.smithi114:> time sudo find /var/log/ceph /var/log/rbd-target-api -name '*.log' -print0 | sudo xargs --max-args=1 --max-procs=0 --verbose -0 --no-run-if-empty -- gzip -5 --verbose -- 2024-04-23T10:44:33.191 DEBUG:teuthology.orchestra.run.smithi119:> time sudo find /var/log/ceph /var/log/rbd-target-api -name '*.log' -print0 | sudo xargs --max-args=1 --max-procs=0 --verbose -0 --no-run-if-empty -- gzip -5 --verbose -- 2024-04-23T10:44:33.193 DEBUG:teuthology.orchestra.run.smithi204:> time sudo find /var/log/ceph /var/log/rbd-target-api -name '*.log' -print0 | sudo xargs --max-args=1 --max-procs=0 --verbose -0 --no-run-if-empty -- gzip -5 --verbose -- 2024-04-23T10:44:33.215 INFO:teuthology.orchestra.run.smithi114.stderr:find: ‘/var/log/rbd-target-api’gzip -5 --verbose -- /var/log/ceph/aa240572-0159-11ef-bc93-c7b262605968/ceph.log 2024-04-23T10:44:33.215 INFO:teuthology.orchestra.run.smithi114.stderr:: No such file or directory 2024-04-23T10:44:33.215 INFO:teuthology.orchestra.run.smithi114.stderr:gzip -5 --verbose -- /var/log/ceph/aa240572-0159-11ef-bc93-c7b262605968/ceph.audit.log 2024-04-23T10:44:33.215 INFO:teuthology.orchestra.run.smithi114.stderr:/var/log/ceph/aa240572-0159-11ef-bc93-c7b262605968/ceph.log: gzip -5 --verbose -- /var/log/ceph/aa240572-0159-11ef-bc93-c7b262605968/ceph-mds.g.log 2024-04-23T10:44:33.216 INFO:teuthology.orchestra.run.smithi114.stderr:/var/log/ceph/aa240572-0159-11ef-bc93-c7b262605968/ceph.audit.log: gzip -5 --verbose -- /var/log/ceph/aa240572-0159-11ef-bc93-c7b262605968/ceph-mds.j.log 2024-04-23T10:44:33.216 INFO:teuthology.orchestra.run.smithi114.stderr:/var/log/ceph/aa240572-0159-11ef-bc93-c7b262605968/ceph-mds.g.log: gzip -5 --verbose -- /var/log/ceph/aa240572-0159-11ef-bc93-c7b262605968/ceph-osd.3.log 2024-04-23T10:44:33.233 INFO:teuthology.orchestra.run.smithi114.stderr:gzip -5 --verbose -- /var/log/ceph/aa240572-0159-11ef-bc93-c7b262605968/ceph-osd.0.log 2024-04-23T10:44:33.233 INFO:teuthology.orchestra.run.smithi114.stderr:/var/log/ceph/aa240572-0159-11ef-bc93-c7b262605968/ceph-mds.j.log: /var/log/ceph/aa240572-0159-11ef-bc93-c7b262605968/ceph-osd.3.log: gzip -5 --verbose -- /var/log/ceph/aa240572-0159-11ef-bc93-c7b262605968/ceph-mgr.x.log 2024-04-23T10:44:33.233 INFO:teuthology.orchestra.run.smithi114.stderr:/var/log/ceph/aa240572-0159-11ef-bc93-c7b262605968/ceph-osd.0.log: gzip -5 --verbose -- /var/log/ceph/aa240572-0159-11ef-bc93-c7b262605968/ceph-mon.a.log 2024-04-23T10:44:33.233 INFO:teuthology.orchestra.run.smithi114.stderr:/var/log/ceph/aa240572-0159-11ef-bc93-c7b262605968/ceph-mgr.x.log: gzip -5 --verbose -- /var/log/ceph/aa240572-0159-11ef-bc93-c7b262605968/ceph-osd.6.log 2024-04-23T10:44:33.234 INFO:teuthology.orchestra.run.smithi114.stderr:/var/log/ceph/aa240572-0159-11ef-bc93-c7b262605968/ceph-mon.a.log: 90.4% 86.9% -- replaced with /var/log/ceph/aa240572-0159-11ef-bc93-c7b262605968/ceph.log.gz 2024-04-23T10:44:33.234 INFO:teuthology.orchestra.run.smithi114.stderr: -- replaced with /var/log/ceph/aa240572-0159-11ef-bc93-c7b262605968/ceph.audit.log.gz 2024-04-23T10:44:33.234 INFO:teuthology.orchestra.run.smithi114.stderr:gzip -5 --verbose -- /var/log/ceph/aa240572-0159-11ef-bc93-c7b262605968/ceph.cephadm.log 2024-04-23T10:44:33.234 INFO:teuthology.orchestra.run.smithi114.stderr:/var/log/ceph/aa240572-0159-11ef-bc93-c7b262605968/ceph-osd.6.log: gzip -5 --verbose -- /var/log/ceph/aa240572-0159-11ef-bc93-c7b262605968/ceph-mds.a.log 2024-04-23T10:44:33.234 INFO:teuthology.orchestra.run.smithi114.stderr:/var/log/ceph/aa240572-0159-11ef-bc93-c7b262605968/ceph.cephadm.log: 84.1% -- replaced with /var/log/ceph/aa240572-0159-11ef-bc93-c7b262605968/ceph.cephadm.log.gz 2024-04-23T10:44:33.234 INFO:teuthology.orchestra.run.smithi114.stderr:gzip -5 --verbose -- /var/log/ceph/aa240572-0159-11ef-bc93-c7b262605968/ceph-volume.log 2024-04-23T10:44:33.234 INFO:teuthology.orchestra.run.smithi114.stderr: 94.5% -- replaced with /var/log/ceph/aa240572-0159-11ef-bc93-c7b262605968/ceph-mds.g.log.gz 2024-04-23T10:44:33.234 INFO:teuthology.orchestra.run.smithi114.stderr:/var/log/ceph/aa240572-0159-11ef-bc93-c7b262605968/ceph-mds.a.log: 94.6% -- replaced with /var/log/ceph/aa240572-0159-11ef-bc93-c7b262605968/ceph-mds.j.log.gz 2024-04-23T10:44:33.235 INFO:teuthology.orchestra.run.smithi114.stderr:gzip -5 --verbose -- /var/log/ceph/aa240572-0159-11ef-bc93-c7b262605968/ceph-mds.d.log 2024-04-23T10:44:33.235 INFO:teuthology.orchestra.run.smithi114.stderr:/var/log/ceph/aa240572-0159-11ef-bc93-c7b262605968/ceph-volume.log: gzip -5 --verbose -- /var/log/ceph/aa240572-0159-11ef-bc93-c7b262605968/ceph-osd.9.log 2024-04-23T10:44:33.235 INFO:teuthology.orchestra.run.smithi114.stderr:/var/log/ceph/aa240572-0159-11ef-bc93-c7b262605968/ceph-mds.d.log: 94.4% -- replaced with /var/log/ceph/aa240572-0159-11ef-bc93-c7b262605968/ceph-mds.a.log.gz 2024-04-23T10:44:33.235 INFO:teuthology.orchestra.run.smithi114.stderr:gzip -5 --verbose -- /var/log/ceph/cephadm.log 2024-04-23T10:44:33.235 INFO:teuthology.orchestra.run.smithi114.stderr:/var/log/ceph/aa240572-0159-11ef-bc93-c7b262605968/ceph-osd.9.log: 94.5% -- replaced with /var/log/ceph/aa240572-0159-11ef-bc93-c7b262605968/ceph-mds.d.log.gz 2024-04-23T10:44:33.236 INFO:teuthology.orchestra.run.smithi119.stderr:find: ‘/var/log/rbd-target-api’gzip -5 --verbose -- /var/log/ceph/aa240572-0159-11ef-bc93-c7b262605968/ceph.log 2024-04-23T10:44:33.236 INFO:teuthology.orchestra.run.smithi119.stderr:: No such file or directory 2024-04-23T10:44:33.237 INFO:teuthology.orchestra.run.smithi119.stderr:gzip -5 --verbose -- /var/log/ceph/aa240572-0159-11ef-bc93-c7b262605968/ceph.audit.log 2024-04-23T10:44:33.237 INFO:teuthology.orchestra.run.smithi119.stderr:/var/log/ceph/aa240572-0159-11ef-bc93-c7b262605968/ceph.log: gzip -5 --verbose -- /var/log/ceph/aa240572-0159-11ef-bc93-c7b262605968/ceph-osd.7.log 2024-04-23T10:44:33.237 INFO:teuthology.orchestra.run.smithi119.stderr:/var/log/ceph/aa240572-0159-11ef-bc93-c7b262605968/ceph.audit.log: gzip -5 --verbose -- /var/log/ceph/aa240572-0159-11ef-bc93-c7b262605968/ceph-mds.b.log 2024-04-23T10:44:33.237 INFO:teuthology.orchestra.run.smithi119.stderr:/var/log/ceph/aa240572-0159-11ef-bc93-c7b262605968/ceph-osd.7.log: gzip -5 --verbose -- /var/log/ceph/aa240572-0159-11ef-bc93-c7b262605968/ceph-osd.4.log 2024-04-23T10:44:33.237 INFO:teuthology.orchestra.run.smithi119.stderr:/var/log/ceph/aa240572-0159-11ef-bc93-c7b262605968/ceph-mds.b.log: gzip -5 --verbose -- /var/log/ceph/aa240572-0159-11ef-bc93-c7b262605968/ceph-osd.1.log 2024-04-23T10:44:33.237 INFO:teuthology.orchestra.run.smithi119.stderr:/var/log/ceph/aa240572-0159-11ef-bc93-c7b262605968/ceph-osd.4.log: gzip -5 --verbose -- /var/log/ceph/aa240572-0159-11ef-bc93-c7b262605968/ceph-mgr.y.log 2024-04-23T10:44:33.237 INFO:teuthology.orchestra.run.smithi119.stderr:/var/log/ceph/aa240572-0159-11ef-bc93-c7b262605968/ceph-osd.1.log: gzip -5 --verbose -- /var/log/ceph/aa240572-0159-11ef-bc93-c7b262605968/ceph-mds.h.log 2024-04-23T10:44:33.237 INFO:teuthology.orchestra.run.smithi119.stderr:/var/log/ceph/aa240572-0159-11ef-bc93-c7b262605968/ceph-mgr.y.log: gzip -5 --verbose -- /var/log/ceph/aa240572-0159-11ef-bc93-c7b262605968/ceph-osd.10.log 2024-04-23T10:44:33.237 INFO:teuthology.orchestra.run.smithi119.stderr:/var/log/ceph/aa240572-0159-11ef-bc93-c7b262605968/ceph-mds.h.log: gzip -5 --verbose -- /var/log/ceph/aa240572-0159-11ef-bc93-c7b262605968/ceph.cephadm.log 2024-04-23T10:44:33.237 INFO:teuthology.orchestra.run.smithi119.stderr:/var/log/ceph/aa240572-0159-11ef-bc93-c7b262605968/ceph-osd.10.log: gzip -5 --verbose -- /var/log/ceph/aa240572-0159-11ef-bc93-c7b262605968/ceph-mds.e.log 2024-04-23T10:44:33.238 INFO:teuthology.orchestra.run.smithi119.stderr:/var/log/ceph/aa240572-0159-11ef-bc93-c7b262605968/ceph.cephadm.log: gzip -5 --verbose -- /var/log/ceph/aa240572-0159-11ef-bc93-c7b262605968/ceph-volume.log 2024-04-23T10:44:33.238 INFO:teuthology.orchestra.run.smithi119.stderr:/var/log/ceph/aa240572-0159-11ef-bc93-c7b262605968/ceph-mds.e.log: gzip -5 --verbose -- /var/log/ceph/aa240572-0159-11ef-bc93-c7b262605968/ceph-mon.b.log 2024-04-23T10:44:33.238 INFO:teuthology.orchestra.run.smithi119.stderr:/var/log/ceph/aa240572-0159-11ef-bc93-c7b262605968/ceph-volume.log: gzip -5 --verbose -- /var/log/ceph/aa240572-0159-11ef-bc93-c7b262605968/ceph-mds.k.log 2024-04-23T10:44:33.238 INFO:teuthology.orchestra.run.smithi119.stderr:/var/log/ceph/aa240572-0159-11ef-bc93-c7b262605968/ceph-mon.b.log: gzip -5 --verbose -- /var/log/ceph/cephadm.log 2024-04-23T10:44:33.238 INFO:teuthology.orchestra.run.smithi119.stderr:/var/log/ceph/aa240572-0159-11ef-bc93-c7b262605968/ceph-mds.k.log: /var/log/ceph/cephadm.log: 83.7% -- replaced with /var/log/ceph/aa240572-0159-11ef-bc93-c7b262605968/ceph.cephadm.log.gz 2024-04-23T10:44:33.239 INFO:teuthology.orchestra.run.smithi114.stderr: 91.0% -- replaced with /var/log/ceph/aa240572-0159-11ef-bc93-c7b262605968/ceph-volume.log.gz 2024-04-23T10:44:33.239 INFO:teuthology.orchestra.run.smithi204.stderr:find: ‘/var/log/rbd-target-api’gzip -5 --verbose -- /var/log/ceph/aa240572-0159-11ef-bc93-c7b262605968/ceph.log 2024-04-23T10:44:33.239 INFO:teuthology.orchestra.run.smithi204.stderr:: No such file or directory 2024-04-23T10:44:33.239 INFO:teuthology.orchestra.run.smithi204.stderr:gzip -5 --verbose -- /var/log/ceph/aa240572-0159-11ef-bc93-c7b262605968/ceph-osd.2.log 2024-04-23T10:44:33.239 INFO:teuthology.orchestra.run.smithi204.stderr:gzip -5 --verbose -- /var/log/ceph/aa240572-0159-11ef-bc93-c7b262605968/ceph-mds.f.log 2024-04-23T10:44:33.239 INFO:teuthology.orchestra.run.smithi204.stderr:/var/log/ceph/aa240572-0159-11ef-bc93-c7b262605968/ceph.log: /var/log/ceph/aa240572-0159-11ef-bc93-c7b262605968/ceph-osd.2.log: gzip -5 --verbose -- /var/log/ceph/aa240572-0159-11ef-bc93-c7b262605968/ceph.audit.log 2024-04-23T10:44:33.239 INFO:teuthology.orchestra.run.smithi204.stderr:/var/log/ceph/aa240572-0159-11ef-bc93-c7b262605968/ceph-mds.f.log: gzip -5 --verbose -- /var/log/ceph/aa240572-0159-11ef-bc93-c7b262605968/ceph-osd.5.log 2024-04-23T10:44:33.240 INFO:teuthology.orchestra.run.smithi204.stderr:gzip -5 --verbose -- /var/log/ceph/aa240572-0159-11ef-bc93-c7b262605968/ceph-mds.c.log 2024-04-23T10:44:33.240 INFO:teuthology.orchestra.run.smithi204.stderr:/var/log/ceph/aa240572-0159-11ef-bc93-c7b262605968/ceph.audit.log: /var/log/ceph/aa240572-0159-11ef-bc93-c7b262605968/ceph-osd.5.log: gzip -5 --verbose -- /var/log/ceph/aa240572-0159-11ef-bc93-c7b262605968/ceph-mon.c.log 2024-04-23T10:44:33.240 INFO:teuthology.orchestra.run.smithi204.stderr:/var/log/ceph/aa240572-0159-11ef-bc93-c7b262605968/ceph-mds.c.log: gzip -5 --verbose -- /var/log/ceph/aa240572-0159-11ef-bc93-c7b262605968/ceph-mgr.z.log 2024-04-23T10:44:33.240 INFO:teuthology.orchestra.run.smithi204.stderr:/var/log/ceph/aa240572-0159-11ef-bc93-c7b262605968/ceph-mon.c.log: gzip -5 --verbose -- /var/log/ceph/aa240572-0159-11ef-bc93-c7b262605968/ceph-mds.i.log 2024-04-23T10:44:33.240 INFO:teuthology.orchestra.run.smithi204.stderr:/var/log/ceph/aa240572-0159-11ef-bc93-c7b262605968/ceph-mgr.z.log: 86.9% -- replaced with /var/log/ceph/aa240572-0159-11ef-bc93-c7b262605968/ceph.log.gz 2024-04-23T10:44:33.240 INFO:teuthology.orchestra.run.smithi204.stderr:gzip -5 --verbose -- /var/log/ceph/aa240572-0159-11ef-bc93-c7b262605968/ceph.cephadm.log 2024-04-23T10:44:33.240 INFO:teuthology.orchestra.run.smithi204.stderr:/var/log/ceph/aa240572-0159-11ef-bc93-c7b262605968/ceph-mds.i.log: 90.6% -- replaced with /var/log/ceph/aa240572-0159-11ef-bc93-c7b262605968/ceph.audit.log.gz 2024-04-23T10:44:33.240 INFO:teuthology.orchestra.run.smithi204.stderr: 94.4% -- replaced with /var/log/ceph/aa240572-0159-11ef-bc93-c7b262605968/ceph-mds.f.log.gz 2024-04-23T10:44:33.240 INFO:teuthology.orchestra.run.smithi204.stderr:gzip -5 --verbose -- /var/log/ceph/aa240572-0159-11ef-bc93-c7b262605968/ceph-osd.11.log 2024-04-23T10:44:33.241 INFO:teuthology.orchestra.run.smithi204.stderr:/var/log/ceph/aa240572-0159-11ef-bc93-c7b262605968/ceph.cephadm.log: gzip -5 --verbose -- /var/log/ceph/aa240572-0159-11ef-bc93-c7b262605968/ceph-osd.8.log 2024-04-23T10:44:33.241 INFO:teuthology.orchestra.run.smithi204.stderr: 83.6% -- replaced with /var/log/ceph/aa240572-0159-11ef-bc93-c7b262605968/ceph.cephadm.log.gz 2024-04-23T10:44:33.241 INFO:teuthology.orchestra.run.smithi204.stderr:/var/log/ceph/aa240572-0159-11ef-bc93-c7b262605968/ceph-osd.11.log: 92.2% -- replaced with /var/log/ceph/aa240572-0159-11ef-bc93-c7b262605968/ceph-mgr.z.log.gz 2024-04-23T10:44:33.241 INFO:teuthology.orchestra.run.smithi204.stderr:gzip -5 --verbose -- /var/log/ceph/aa240572-0159-11ef-bc93-c7b262605968/ceph-volume.log 2024-04-23T10:44:33.241 INFO:teuthology.orchestra.run.smithi204.stderr:/var/log/ceph/aa240572-0159-11ef-bc93-c7b262605968/ceph-osd.8.log: gzip -5 --verbose -- /var/log/ceph/aa240572-0159-11ef-bc93-c7b262605968/ceph-mds.l.log 2024-04-23T10:44:33.244 INFO:teuthology.orchestra.run.smithi204.stderr:/var/log/ceph/aa240572-0159-11ef-bc93-c7b262605968/ceph-volume.log: gzip -5 --verbose -- /var/log/ceph/cephadm.log 2024-04-23T10:44:33.244 INFO:teuthology.orchestra.run.smithi114.stderr:/var/log/ceph/cephadm.log: 93.4% -- replaced with /var/log/ceph/cephadm.log.gz 2024-04-23T10:44:33.251 INFO:teuthology.orchestra.run.smithi204.stderr:/var/log/ceph/aa240572-0159-11ef-bc93-c7b262605968/ceph-mds.l.log: /var/log/ceph/cephadm.log: 91.0% -- replaced with /var/log/ceph/aa240572-0159-11ef-bc93-c7b262605968/ceph-volume.log.gz 2024-04-23T10:44:33.251 INFO:teuthology.orchestra.run.smithi204.stderr: 94.4% -- replaced with /var/log/ceph/aa240572-0159-11ef-bc93-c7b262605968/ceph-mds.l.log.gz 2024-04-23T10:44:33.253 INFO:teuthology.orchestra.run.smithi204.stderr: 92.4% -- replaced with /var/log/ceph/cephadm.log.gz 2024-04-23T10:44:33.288 INFO:teuthology.orchestra.run.smithi119.stderr: 86.9% -- replaced with /var/log/ceph/aa240572-0159-11ef-bc93-c7b262605968/ceph.log.gz 2024-04-23T10:44:33.357 INFO:teuthology.orchestra.run.smithi119.stderr: 92.1% -- replaced with /var/log/ceph/cephadm.log.gz 2024-04-23T10:44:33.388 INFO:teuthology.orchestra.run.smithi119.stderr: 90.6% -- replaced with /var/log/ceph/aa240572-0159-11ef-bc93-c7b262605968/ceph.audit.log.gz 2024-04-23T10:44:33.428 INFO:teuthology.orchestra.run.smithi119.stderr: 92.1% -- replaced with /var/log/ceph/aa240572-0159-11ef-bc93-c7b262605968/ceph-mgr.y.log.gz 2024-04-23T10:44:33.566 INFO:teuthology.orchestra.run.smithi204.stderr: 92.0% -- replaced with /var/log/ceph/aa240572-0159-11ef-bc93-c7b262605968/ceph-mon.c.log.gz 2024-04-23T10:44:33.613 INFO:teuthology.orchestra.run.smithi119.stderr: 91.1% -- replaced with /var/log/ceph/aa240572-0159-11ef-bc93-c7b262605968/ceph-volume.log.gz 2024-04-23T10:44:33.642 INFO:teuthology.orchestra.run.smithi114.stderr: 90.2% -- replaced with /var/log/ceph/aa240572-0159-11ef-bc93-c7b262605968/ceph-mgr.x.log.gz 2024-04-23T10:44:35.143 INFO:teuthology.orchestra.run.smithi119.stderr: 92.1% -- replaced with /var/log/ceph/aa240572-0159-11ef-bc93-c7b262605968/ceph-mon.b.log.gz 2024-04-23T10:44:35.194 INFO:teuthology.orchestra.run.smithi114.stderr: 91.5% -- replaced with /var/log/ceph/aa240572-0159-11ef-bc93-c7b262605968/ceph-mon.a.log.gz 2024-04-23T10:44:37.570 INFO:teuthology.orchestra.run.smithi204.stderr: 95.6% -- replaced with /var/log/ceph/aa240572-0159-11ef-bc93-c7b262605968/ceph-mds.c.log.gz 2024-04-23T10:44:38.008 INFO:teuthology.orchestra.run.smithi204.stderr: 95.2% -- replaced with /var/log/ceph/aa240572-0159-11ef-bc93-c7b262605968/ceph-mds.i.log.gz 2024-04-23T10:44:38.735 INFO:teuthology.orchestra.run.smithi114.stderr: 94.1% -- replaced with /var/log/ceph/aa240572-0159-11ef-bc93-c7b262605968/ceph-osd.9.log.gz 2024-04-23T10:44:38.925 INFO:teuthology.orchestra.run.smithi114.stderr: 93.8% -- replaced with /var/log/ceph/aa240572-0159-11ef-bc93-c7b262605968/ceph-osd.0.log.gz 2024-04-23T10:44:39.588 INFO:teuthology.orchestra.run.smithi114.stderr: 94.1% -- replaced with /var/log/ceph/aa240572-0159-11ef-bc93-c7b262605968/ceph-osd.6.log.gz 2024-04-23T10:44:39.717 INFO:teuthology.orchestra.run.smithi204.stderr: 94.1% -- replaced with /var/log/ceph/aa240572-0159-11ef-bc93-c7b262605968/ceph-osd.8.log.gz 2024-04-23T10:44:40.142 INFO:teuthology.orchestra.run.smithi114.stderr: 94.0% -- replaced with /var/log/ceph/aa240572-0159-11ef-bc93-c7b262605968/ceph-osd.3.log.gz 2024-04-23T10:44:40.145 INFO:teuthology.orchestra.run.smithi114.stderr: 2024-04-23T10:44:40.145 INFO:teuthology.orchestra.run.smithi114.stderr:real 0m6.940s 2024-04-23T10:44:40.145 INFO:teuthology.orchestra.run.smithi114.stderr:user 0m25.967s 2024-04-23T10:44:40.145 INFO:teuthology.orchestra.run.smithi114.stderr:sys 0m0.976s 2024-04-23T10:44:41.468 INFO:teuthology.orchestra.run.smithi204.stderr: 94.1% -- replaced with /var/log/ceph/aa240572-0159-11ef-bc93-c7b262605968/ceph-osd.2.log.gz 2024-04-23T10:44:41.736 INFO:teuthology.orchestra.run.smithi204.stderr: 94.5% -- replaced with /var/log/ceph/aa240572-0159-11ef-bc93-c7b262605968/ceph-osd.11.log.gz 2024-04-23T10:44:42.627 INFO:teuthology.orchestra.run.smithi204.stderr: 94.2% -- replaced with /var/log/ceph/aa240572-0159-11ef-bc93-c7b262605968/ceph-osd.5.log.gz 2024-04-23T10:44:42.629 INFO:teuthology.orchestra.run.smithi204.stderr: 2024-04-23T10:44:42.629 INFO:teuthology.orchestra.run.smithi204.stderr:real 0m9.421s 2024-04-23T10:44:42.629 INFO:teuthology.orchestra.run.smithi204.stderr:user 0m40.676s 2024-04-23T10:44:42.629 INFO:teuthology.orchestra.run.smithi204.stderr:sys 0m1.457s 2024-04-23T10:44:43.973 INFO:teuthology.orchestra.run.smithi119.stderr: 93.9% -- replaced with /var/log/ceph/aa240572-0159-11ef-bc93-c7b262605968/ceph-osd.1.log.gz 2024-04-23T10:44:45.385 INFO:teuthology.orchestra.run.smithi119.stderr: 95.3% -- replaced with /var/log/ceph/aa240572-0159-11ef-bc93-c7b262605968/ceph-mds.k.log.gz 2024-04-23T10:44:46.941 INFO:teuthology.orchestra.run.smithi119.stderr: 94.1% -- replaced with /var/log/ceph/aa240572-0159-11ef-bc93-c7b262605968/ceph-osd.10.log.gz 2024-04-23T10:44:47.847 INFO:teuthology.orchestra.run.smithi119.stderr: 94.2% -- replaced with /var/log/ceph/aa240572-0159-11ef-bc93-c7b262605968/ceph-osd.7.log.gz 2024-04-23T10:44:54.141 INFO:teuthology.orchestra.run.smithi119.stderr: 94.0% -- replaced with /var/log/ceph/aa240572-0159-11ef-bc93-c7b262605968/ceph-osd.4.log.gz 2024-04-23T10:45:17.895 INFO:teuthology.orchestra.run.smithi119.stderr: 93.8% -- replaced with /var/log/ceph/aa240572-0159-11ef-bc93-c7b262605968/ceph-mds.h.log.gz 2024-04-23T10:45:54.013 INFO:teuthology.orchestra.run.smithi119.stderr: 93.8% -- replaced with /var/log/ceph/aa240572-0159-11ef-bc93-c7b262605968/ceph-mds.e.log.gz 2024-04-23T10:46:29.425 INFO:teuthology.orchestra.run.smithi119.stderr: 93.5% -- replaced with /var/log/ceph/aa240572-0159-11ef-bc93-c7b262605968/ceph-mds.b.log.gz 2024-04-23T10:46:29.428 INFO:teuthology.orchestra.run.smithi119.stderr: 2024-04-23T10:46:29.428 INFO:teuthology.orchestra.run.smithi119.stderr:real 1m56.220s 2024-04-23T10:46:29.428 INFO:teuthology.orchestra.run.smithi119.stderr:user 3m50.838s 2024-04-23T10:46:29.428 INFO:teuthology.orchestra.run.smithi119.stderr:sys 0m8.094s 2024-04-23T10:46:29.428 INFO:tasks.cephadm:Archiving logs... 2024-04-23T10:46:29.429 DEBUG:teuthology.misc:Transferring archived files from smithi114:/var/log/ceph to /home/teuthworker/archive/leonidus-2024-04-23_09:49:14-fs-wip-lusov-quiescer-distro-default-smithi/7669513/remote/smithi114/log 2024-04-23T10:46:29.430 DEBUG:teuthology.orchestra.run.smithi114:> sudo tar c -f - -C /var/log/ceph -- . 2024-04-23T10:46:32.813 DEBUG:teuthology.misc:Transferring archived files from smithi119:/var/log/ceph to /home/teuthworker/archive/leonidus-2024-04-23_09:49:14-fs-wip-lusov-quiescer-distro-default-smithi/7669513/remote/smithi119/log 2024-04-23T10:46:32.814 DEBUG:teuthology.orchestra.run.smithi119:> sudo tar c -f - -C /var/log/ceph -- . 2024-04-23T10:47:04.257 DEBUG:teuthology.misc:Transferring archived files from smithi204:/var/log/ceph to /home/teuthworker/archive/leonidus-2024-04-23_09:49:14-fs-wip-lusov-quiescer-distro-default-smithi/7669513/remote/smithi204/log 2024-04-23T10:47:04.258 DEBUG:teuthology.orchestra.run.smithi204:> sudo tar c -f - -C /var/log/ceph -- . 2024-04-23T10:47:08.723 INFO:tasks.cephadm:Removing cluster... 2024-04-23T10:47:08.723 DEBUG:teuthology.orchestra.run.smithi114:> sudo /home/ubuntu/cephtest/cephadm rm-cluster --fsid aa240572-0159-11ef-bc93-c7b262605968 --force 2024-04-23T10:47:08.915 INFO:teuthology.orchestra.run.smithi114.stdout:Deleting cluster with fsid: aa240572-0159-11ef-bc93-c7b262605968 2024-04-23T10:47:09.210 DEBUG:teuthology.orchestra.run.smithi119:> sudo /home/ubuntu/cephtest/cephadm rm-cluster --fsid aa240572-0159-11ef-bc93-c7b262605968 --force 2024-04-23T10:47:09.401 INFO:teuthology.orchestra.run.smithi119.stdout:Deleting cluster with fsid: aa240572-0159-11ef-bc93-c7b262605968 2024-04-23T10:47:09.934 DEBUG:teuthology.orchestra.run.smithi204:> sudo /home/ubuntu/cephtest/cephadm rm-cluster --fsid aa240572-0159-11ef-bc93-c7b262605968 --force 2024-04-23T10:47:10.126 INFO:teuthology.orchestra.run.smithi204.stdout:Deleting cluster with fsid: aa240572-0159-11ef-bc93-c7b262605968 2024-04-23T10:47:10.421 INFO:tasks.cephadm:Removing cephadm ... 2024-04-23T10:47:10.421 DEBUG:teuthology.orchestra.run.smithi114:> rm -rf /home/ubuntu/cephtest/cephadm 2024-04-23T10:47:10.437 DEBUG:teuthology.orchestra.run.smithi119:> rm -rf /home/ubuntu/cephtest/cephadm 2024-04-23T10:47:10.455 DEBUG:teuthology.orchestra.run.smithi204:> rm -rf /home/ubuntu/cephtest/cephadm 2024-04-23T10:47:10.470 INFO:tasks.cephadm:Teardown complete 2024-04-23T10:47:10.471 DEBUG:teuthology.run_tasks:Unwinding manager install 2024-04-23T10:47:10.484 INFO:teuthology.task.install.util:Removing shipped files: /home/ubuntu/cephtest/valgrind.supp /usr/bin/daemon-helper /usr/bin/adjust-ulimits /usr/bin/stdin-killer... 2024-04-23T10:47:10.484 DEBUG:teuthology.orchestra.run.smithi114:> sudo rm -f -- /home/ubuntu/cephtest/valgrind.supp /usr/bin/daemon-helper /usr/bin/adjust-ulimits /usr/bin/stdin-killer 2024-04-23T10:47:10.487 DEBUG:teuthology.orchestra.run.smithi119:> sudo rm -f -- /home/ubuntu/cephtest/valgrind.supp /usr/bin/daemon-helper /usr/bin/adjust-ulimits /usr/bin/stdin-killer 2024-04-23T10:47:10.497 DEBUG:teuthology.orchestra.run.smithi204:> sudo rm -f -- /home/ubuntu/cephtest/valgrind.supp /usr/bin/daemon-helper /usr/bin/adjust-ulimits /usr/bin/stdin-killer 2024-04-23T10:47:10.652 DEBUG:teuthology.run_tasks:Unwinding manager clock 2024-04-23T10:47:10.663 INFO:teuthology.task.clock:Checking final clock skew... 2024-04-23T10:47:10.664 DEBUG:teuthology.orchestra.run.smithi114:> PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-04-23T10:47:10.667 DEBUG:teuthology.orchestra.run.smithi119:> PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-04-23T10:47:10.670 DEBUG:teuthology.orchestra.run.smithi204:> PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-04-23T10:47:10.678 INFO:teuthology.orchestra.run.smithi114.stderr:bash: line 1: ntpq: command not found 2024-04-23T10:47:10.682 INFO:teuthology.orchestra.run.smithi119.stderr:bash: line 1: ntpq: command not found 2024-04-23T10:47:10.684 INFO:teuthology.orchestra.run.smithi114.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2024-04-23T10:47:10.684 INFO:teuthology.orchestra.run.smithi114.stdout:=============================================================================== 2024-04-23T10:47:10.684 INFO:teuthology.orchestra.run.smithi114.stdout:^- hv01.front.sepia.ceph.com 2 6 377 22 +1745us[+1759us] +/- 595ms 2024-04-23T10:47:10.684 INFO:teuthology.orchestra.run.smithi114.stdout:^* hv02.front.sepia.ceph.com 3 6 377 20 +153us[ +167us] +/- 60ms 2024-04-23T10:47:10.685 INFO:teuthology.orchestra.run.smithi114.stdout:^+ hv03.front.sepia.ceph.com 3 6 377 20 -176us[ -162us] +/- 75ms 2024-04-23T10:47:10.685 INFO:teuthology.orchestra.run.smithi114.stdout:^? hv04.front.sepia.ceph.com 0 9 0 - +0ns[ +0ns] +/- 0ns 2024-04-23T10:47:10.685 INFO:teuthology.orchestra.run.smithi204.stderr:bash: line 1: ntpq: command not found 2024-04-23T10:47:10.689 INFO:teuthology.orchestra.run.smithi204.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2024-04-23T10:47:10.689 INFO:teuthology.orchestra.run.smithi204.stdout:=============================================================================== 2024-04-23T10:47:10.689 INFO:teuthology.orchestra.run.smithi204.stdout:^- hv01.front.sepia.ceph.com 2 6 377 23 +1738us[+1738us] +/- 595ms 2024-04-23T10:47:10.689 INFO:teuthology.orchestra.run.smithi204.stdout:^* hv02.front.sepia.ceph.com 3 6 377 24 +160us[ +172us] +/- 60ms 2024-04-23T10:47:10.690 INFO:teuthology.orchestra.run.smithi204.stdout:^+ hv03.front.sepia.ceph.com 3 6 377 22 -170us[ -170us] +/- 75ms 2024-04-23T10:47:10.690 INFO:teuthology.orchestra.run.smithi204.stdout:^? hv04.front.sepia.ceph.com 0 9 0 - +0ns[ +0ns] +/- 0ns 2024-04-23T10:47:10.702 INFO:teuthology.orchestra.run.smithi119.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2024-04-23T10:47:10.703 INFO:teuthology.orchestra.run.smithi119.stdout:=============================================================================== 2024-04-23T10:47:10.703 INFO:teuthology.orchestra.run.smithi119.stdout:^- hv01.front.sepia.ceph.com 2 6 377 22 +1745us[+1760us] +/- 595ms 2024-04-23T10:47:10.703 INFO:teuthology.orchestra.run.smithi119.stdout:^* hv02.front.sepia.ceph.com 3 6 377 22 +163us[ +178us] +/- 60ms 2024-04-23T10:47:10.703 INFO:teuthology.orchestra.run.smithi119.stdout:^+ hv03.front.sepia.ceph.com 3 6 377 21 -184us[ -184us] +/- 75ms 2024-04-23T10:47:10.703 INFO:teuthology.orchestra.run.smithi119.stdout:^? hv04.front.sepia.ceph.com 0 9 0 - +0ns[ +0ns] +/- 0ns 2024-04-23T10:47:10.704 DEBUG:teuthology.run_tasks:Unwinding manager ansible.cephlab 2024-04-23T10:47:10.715 INFO:teuthology.task.ansible:Skipping ansible cleanup... 2024-04-23T10:47:10.716 DEBUG:teuthology.run_tasks:Unwinding manager selinux 2024-04-23T10:47:10.727 DEBUG:teuthology.orchestra.run.smithi114:> mkdir /home/ubuntu/cephtest/archive/audit && sudo cp /var/log/audit/audit.log /home/ubuntu/cephtest/archive/audit && sudo chown $USER /home/ubuntu/cephtest/archive/audit/audit.log && gzip /home/ubuntu/cephtest/archive/audit/audit.log 2024-04-23T10:47:10.799 DEBUG:teuthology.orchestra.run.smithi119:> mkdir /home/ubuntu/cephtest/archive/audit && sudo cp /var/log/audit/audit.log /home/ubuntu/cephtest/archive/audit && sudo chown $USER /home/ubuntu/cephtest/archive/audit/audit.log && gzip /home/ubuntu/cephtest/archive/audit/audit.log 2024-04-23T10:47:10.903 DEBUG:teuthology.orchestra.run.smithi204:> mkdir /home/ubuntu/cephtest/archive/audit && sudo cp /var/log/audit/audit.log /home/ubuntu/cephtest/archive/audit && sudo chown $USER /home/ubuntu/cephtest/archive/audit/audit.log && gzip /home/ubuntu/cephtest/archive/audit/audit.log 2024-04-23T10:47:10.968 DEBUG:teuthology.orchestra.run.smithi114:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 -e scontext=system_u:system_r:logrotate_t:s0 -e scontext=system_u:system_r:getty_t:s0 -e scontext=system_u:system_r:getty_t:s0 2024-04-23T10:47:10.995 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-23T10:47:10.995 DEBUG:teuthology.orchestra.run.smithi119:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 -e scontext=system_u:system_r:logrotate_t:s0 -e scontext=system_u:system_r:getty_t:s0 -e scontext=system_u:system_r:getty_t:s0 2024-04-23T10:47:11.023 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-23T10:47:11.024 DEBUG:teuthology.orchestra.run.smithi204:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 -e scontext=system_u:system_r:logrotate_t:s0 -e scontext=system_u:system_r:getty_t:s0 -e scontext=system_u:system_r:getty_t:s0 2024-04-23T10:47:11.051 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-23T10:47:11.051 DEBUG:teuthology.run_tasks:Unwinding manager pcp 2024-04-23T10:47:11.063 DEBUG:teuthology.run_tasks:Unwinding manager internal.timer 2024-04-23T10:47:11.074 INFO:teuthology.task.internal:Duration was 2825.954349 seconds 2024-04-23T10:47:11.075 DEBUG:teuthology.run_tasks:Unwinding manager internal.syslog 2024-04-23T10:47:11.086 INFO:teuthology.task.internal.syslog:Shutting down syslog monitoring... 2024-04-23T10:47:11.086 DEBUG:teuthology.orchestra.run.smithi114:> sudo rm -f -- /etc/rsyslog.d/80-cephtest.conf && sudo service rsyslog restart 2024-04-23T10:47:11.089 DEBUG:teuthology.orchestra.run.smithi119:> sudo rm -f -- /etc/rsyslog.d/80-cephtest.conf && sudo service rsyslog restart 2024-04-23T10:47:11.092 DEBUG:teuthology.orchestra.run.smithi204:> sudo rm -f -- /etc/rsyslog.d/80-cephtest.conf && sudo service rsyslog restart 2024-04-23T10:47:11.122 INFO:teuthology.orchestra.run.smithi114.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2024-04-23T10:47:11.129 INFO:teuthology.orchestra.run.smithi204.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2024-04-23T10:47:11.148 INFO:teuthology.orchestra.run.smithi119.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2024-04-23T10:47:11.542 INFO:teuthology.task.internal.syslog:Checking logs for errors... 2024-04-23T10:47:11.543 DEBUG:teuthology.task.internal.syslog:Checking ubuntu@smithi114.front.sepia.ceph.com 2024-04-23T10:47:11.543 DEBUG:teuthology.orchestra.run.smithi114:> egrep --binary-files=text '\bBUG\b|\bINFO\b|\bDEADLOCK\b' /home/ubuntu/cephtest/archive/syslog/kern.log | grep -v 'task .* blocked for more than .* seconds' | grep -v 'lockdep is turned off' | grep -v 'trying to register non-static key' | grep -v 'DEBUG: fsize' | grep -v CRON | grep -v 'BUG: bad unlock balance detected' | grep -v 'inconsistent lock state' | grep -v '*** DEADLOCK ***' | grep -v 'INFO: possible irq lock inversion dependency detected' | grep -v 'INFO: NMI handler (perf_event_nmi_handler) took too long to run' | grep -v 'INFO: recovery required on readonly' | grep -v 'ceph-create-keys: INFO' | grep -v INFO:ceph-create-keys | grep -v 'Loaded datasource DataSourceOpenStack' | grep -v 'container-storage-setup: INFO: Volume group backing root filesystem could not be determined' | egrep -v '\bsalt-master\b|\bsalt-minion\b|\bsalt-api\b' | grep -v ceph-crash | egrep -v '\btcmu-runner\b.*\bINFO\b' | head -n 1 2024-04-23T10:47:11.603 DEBUG:teuthology.task.internal.syslog:Checking ubuntu@smithi119.front.sepia.ceph.com 2024-04-23T10:47:11.604 DEBUG:teuthology.orchestra.run.smithi119:> egrep --binary-files=text '\bBUG\b|\bINFO\b|\bDEADLOCK\b' /home/ubuntu/cephtest/archive/syslog/kern.log | grep -v 'task .* blocked for more than .* seconds' | grep -v 'lockdep is turned off' | grep -v 'trying to register non-static key' | grep -v 'DEBUG: fsize' | grep -v CRON | grep -v 'BUG: bad unlock balance detected' | grep -v 'inconsistent lock state' | grep -v '*** DEADLOCK ***' | grep -v 'INFO: possible irq lock inversion dependency detected' | grep -v 'INFO: NMI handler (perf_event_nmi_handler) took too long to run' | grep -v 'INFO: recovery required on readonly' | grep -v 'ceph-create-keys: INFO' | grep -v INFO:ceph-create-keys | grep -v 'Loaded datasource DataSourceOpenStack' | grep -v 'container-storage-setup: INFO: Volume group backing root filesystem could not be determined' | egrep -v '\bsalt-master\b|\bsalt-minion\b|\bsalt-api\b' | grep -v ceph-crash | egrep -v '\btcmu-runner\b.*\bINFO\b' | head -n 1 2024-04-23T10:47:11.675 DEBUG:teuthology.task.internal.syslog:Checking ubuntu@smithi204.front.sepia.ceph.com 2024-04-23T10:47:11.675 DEBUG:teuthology.orchestra.run.smithi204:> egrep --binary-files=text '\bBUG\b|\bINFO\b|\bDEADLOCK\b' /home/ubuntu/cephtest/archive/syslog/kern.log | grep -v 'task .* blocked for more than .* seconds' | grep -v 'lockdep is turned off' | grep -v 'trying to register non-static key' | grep -v 'DEBUG: fsize' | grep -v CRON | grep -v 'BUG: bad unlock balance detected' | grep -v 'inconsistent lock state' | grep -v '*** DEADLOCK ***' | grep -v 'INFO: possible irq lock inversion dependency detected' | grep -v 'INFO: NMI handler (perf_event_nmi_handler) took too long to run' | grep -v 'INFO: recovery required on readonly' | grep -v 'ceph-create-keys: INFO' | grep -v INFO:ceph-create-keys | grep -v 'Loaded datasource DataSourceOpenStack' | grep -v 'container-storage-setup: INFO: Volume group backing root filesystem could not be determined' | egrep -v '\bsalt-master\b|\bsalt-minion\b|\bsalt-api\b' | grep -v ceph-crash | egrep -v '\btcmu-runner\b.*\bINFO\b' | head -n 1 2024-04-23T10:47:11.704 INFO:teuthology.task.internal.syslog:Compressing syslogs... 2024-04-23T10:47:11.704 DEBUG:teuthology.orchestra.run.smithi114:> find /home/ubuntu/cephtest/archive/syslog -name '*.log' -print0 | sudo xargs -0 --no-run-if-empty -- gzip -- 2024-04-23T10:47:11.707 DEBUG:teuthology.orchestra.run.smithi119:> find /home/ubuntu/cephtest/archive/syslog -name '*.log' -print0 | sudo xargs -0 --no-run-if-empty -- gzip -- 2024-04-23T10:47:11.710 DEBUG:teuthology.orchestra.run.smithi204:> find /home/ubuntu/cephtest/archive/syslog -name '*.log' -print0 | sudo xargs -0 --no-run-if-empty -- gzip -- 2024-04-23T10:47:11.770 INFO:teuthology.task.internal.syslog:Gathering journactl -b0... 2024-04-23T10:47:11.770 DEBUG:teuthology.orchestra.run.smithi114:> sudo journalctl -b0 | gzip -9 > /home/ubuntu/cephtest/archive/syslog/journalctl-b0.gz 2024-04-23T10:47:11.774 DEBUG:teuthology.orchestra.run.smithi119:> sudo journalctl -b0 | gzip -9 > /home/ubuntu/cephtest/archive/syslog/journalctl-b0.gz 2024-04-23T10:47:11.778 DEBUG:teuthology.orchestra.run.smithi204:> sudo journalctl -b0 | gzip -9 > /home/ubuntu/cephtest/archive/syslog/journalctl-b0.gz 2024-04-23T10:47:12.160 DEBUG:teuthology.run_tasks:Unwinding manager internal.sudo 2024-04-23T10:47:12.173 INFO:teuthology.task.internal:Restoring /etc/sudoers... 2024-04-23T10:47:12.174 DEBUG:teuthology.orchestra.run.smithi114:> sudo mv -f /etc/sudoers.orig.teuthology /etc/sudoers 2024-04-23T10:47:12.198 DEBUG:teuthology.orchestra.run.smithi119:> sudo mv -f /etc/sudoers.orig.teuthology /etc/sudoers 2024-04-23T10:47:12.227 DEBUG:teuthology.orchestra.run.smithi204:> sudo mv -f /etc/sudoers.orig.teuthology /etc/sudoers 2024-04-23T10:47:12.251 DEBUG:teuthology.run_tasks:Unwinding manager internal.coredump 2024-04-23T10:47:12.263 DEBUG:teuthology.orchestra.run.smithi114:> sudo sysctl -w kernel.core_pattern=core && sudo bash -c 'for f in `find /home/ubuntu/cephtest/archive/coredump -type f`; do file $f | grep -q systemd-sysusers && rm $f || true ; done' && rmdir --ignore-fail-on-non-empty -- /home/ubuntu/cephtest/archive/coredump 2024-04-23T10:47:12.266 DEBUG:teuthology.orchestra.run.smithi119:> sudo sysctl -w kernel.core_pattern=core && sudo bash -c 'for f in `find /home/ubuntu/cephtest/archive/coredump -type f`; do file $f | grep -q systemd-sysusers && rm $f || true ; done' && rmdir --ignore-fail-on-non-empty -- /home/ubuntu/cephtest/archive/coredump 2024-04-23T10:47:12.270 DEBUG:teuthology.orchestra.run.smithi204:> sudo sysctl -w kernel.core_pattern=core && sudo bash -c 'for f in `find /home/ubuntu/cephtest/archive/coredump -type f`; do file $f | grep -q systemd-sysusers && rm $f || true ; done' && rmdir --ignore-fail-on-non-empty -- /home/ubuntu/cephtest/archive/coredump 2024-04-23T10:47:12.285 INFO:teuthology.orchestra.run.smithi114.stdout:kernel.core_pattern = core 2024-04-23T10:47:12.298 INFO:teuthology.orchestra.run.smithi119.stdout:kernel.core_pattern = core 2024-04-23T10:47:12.314 INFO:teuthology.orchestra.run.smithi204.stdout:kernel.core_pattern = core 2024-04-23T10:47:12.337 DEBUG:teuthology.orchestra.run.smithi114:> test -e /home/ubuntu/cephtest/archive/coredump 2024-04-23T10:47:12.351 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-23T10:47:12.352 DEBUG:teuthology.orchestra.run.smithi119:> test -e /home/ubuntu/cephtest/archive/coredump 2024-04-23T10:47:12.373 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-23T10:47:12.374 DEBUG:teuthology.orchestra.run.smithi204:> test -e /home/ubuntu/cephtest/archive/coredump 2024-04-23T10:47:12.391 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-23T10:47:12.392 DEBUG:teuthology.run_tasks:Unwinding manager internal.archive 2024-04-23T10:47:12.404 INFO:teuthology.task.internal:Transferring archived files... 2024-04-23T10:47:12.405 DEBUG:teuthology.misc:Transferring archived files from smithi114:/home/ubuntu/cephtest/archive to /home/teuthworker/archive/leonidus-2024-04-23_09:49:14-fs-wip-lusov-quiescer-distro-default-smithi/7669513/remote/smithi114 2024-04-23T10:47:12.405 DEBUG:teuthology.orchestra.run.smithi114:> sudo tar c -f - -C /home/ubuntu/cephtest/archive -- . 2024-04-23T10:47:12.450 DEBUG:teuthology.misc:Transferring archived files from smithi119:/home/ubuntu/cephtest/archive to /home/teuthworker/archive/leonidus-2024-04-23_09:49:14-fs-wip-lusov-quiescer-distro-default-smithi/7669513/remote/smithi119 2024-04-23T10:47:12.451 DEBUG:teuthology.orchestra.run.smithi119:> sudo tar c -f - -C /home/ubuntu/cephtest/archive -- . 2024-04-23T10:47:12.489 DEBUG:teuthology.misc:Transferring archived files from smithi204:/home/ubuntu/cephtest/archive to /home/teuthworker/archive/leonidus-2024-04-23_09:49:14-fs-wip-lusov-quiescer-distro-default-smithi/7669513/remote/smithi204 2024-04-23T10:47:12.489 DEBUG:teuthology.orchestra.run.smithi204:> sudo tar c -f - -C /home/ubuntu/cephtest/archive -- . 2024-04-23T10:47:12.527 INFO:teuthology.task.internal:Removing archive directory... 2024-04-23T10:47:12.527 DEBUG:teuthology.orchestra.run.smithi114:> rm -rf -- /home/ubuntu/cephtest/archive 2024-04-23T10:47:12.530 DEBUG:teuthology.orchestra.run.smithi119:> rm -rf -- /home/ubuntu/cephtest/archive 2024-04-23T10:47:12.532 DEBUG:teuthology.orchestra.run.smithi204:> rm -rf -- /home/ubuntu/cephtest/archive 2024-04-23T10:47:12.574 DEBUG:teuthology.run_tasks:Unwinding manager internal.archive_upload 2024-04-23T10:47:12.588 INFO:teuthology.task.internal:Not uploading archives. 2024-04-23T10:47:12.588 DEBUG:teuthology.run_tasks:Unwinding manager internal.base 2024-04-23T10:47:12.599 INFO:teuthology.task.internal:Tidying up after the test... 2024-04-23T10:47:12.599 DEBUG:teuthology.orchestra.run.smithi114:> find /home/ubuntu/cephtest -ls ; rmdir -- /home/ubuntu/cephtest 2024-04-23T10:47:12.602 DEBUG:teuthology.orchestra.run.smithi119:> find /home/ubuntu/cephtest -ls ; rmdir -- /home/ubuntu/cephtest 2024-04-23T10:47:12.604 DEBUG:teuthology.orchestra.run.smithi204:> find /home/ubuntu/cephtest -ls ; rmdir -- /home/ubuntu/cephtest 2024-04-23T10:47:12.614 INFO:teuthology.orchestra.run.smithi114.stdout: 262213 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Apr 23 10:47 /home/ubuntu/cephtest 2024-04-23T10:47:12.616 INFO:teuthology.orchestra.run.smithi119.stdout: 262161 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Apr 23 10:47 /home/ubuntu/cephtest 2024-04-23T10:47:12.629 INFO:teuthology.orchestra.run.smithi204.stdout: 262161 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Apr 23 10:47 /home/ubuntu/cephtest 2024-04-23T10:47:12.630 DEBUG:teuthology.run_tasks:Unwinding manager kernel 2024-04-23T10:47:12.642 DEBUG:teuthology.run_tasks:Unwinding manager console_log 2024-04-23T10:47:12.666 INFO:teuthology.nuke:Checking targets against current locks 2024-04-23T10:47:12.738 DEBUG:teuthology.nuke:shortname: smithi114 2024-04-23T10:47:12.738 INFO:teuthology.task.internal.check_lock:Checking locks... 2024-04-23T10:47:12.761 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi114.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/leonidus-2024-04-23_09:49:14-fs-wip-lusov-quiescer-distro-default-smithi/7669513', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'centos', 'os_version': '9', 'arch': 'x86_64', 'locked': True, 'locked_since': '2024-04-23 09:51:12.360716', 'locked_by': 'scheduled_leonidus@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBD6u3Zno1foX48Y3lv7disXfCFtZ6vZInKbwWm5POHIwlsBe/WU1nLpfVJKdmeqy65KimOB1dpAXAgWeGaqkS/w='} 2024-04-23T10:47:12.765 DEBUG:teuthology.nuke:shortname: smithi119 2024-04-23T10:47:12.766 INFO:teuthology.task.internal.check_lock:Checking locks... 2024-04-23T10:47:12.789 DEBUG:teuthology.nuke:shortname: smithi204 2024-04-23T10:47:12.790 INFO:teuthology.task.internal.check_lock:Checking locks... 2024-04-23T10:47:12.795 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi119.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/leonidus-2024-04-23_09:49:14-fs-wip-lusov-quiescer-distro-default-smithi/7669513', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'centos', 'os_version': '9', 'arch': 'x86_64', 'locked': True, 'locked_since': '2024-04-23 09:51:12.357810', 'locked_by': 'scheduled_leonidus@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBMhNGhl/IjEU4Y4EZBE962ZOct6Ib/Whe2OtYifLHYv4C+WpTQ0fnZDi2CQLAMiR1n59QXuw1R+mG+tvEvCiy3U='} 2024-04-23T10:47:12.824 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi204.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/leonidus-2024-04-23_09:49:14-fs-wip-lusov-quiescer-distro-default-smithi/7669513', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'centos', 'os_version': '9', 'arch': 'x86_64', 'locked': True, 'locked_since': '2024-04-23 09:51:12.359497', 'locked_by': 'scheduled_leonidus@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBAzDTSNOL1U1qUNZcRmM1lyk+mFB8Dzhokg4oU5SJYqBEDEPFordLxqpcqVZSf+Rq8cCRhZupwETwiXjgbP2VNs='} 2024-04-23T10:47:12.831 INFO:teuthology.orchestra.console.smithi114:Power off 2024-04-23T10:47:12.832 DEBUG:teuthology.orchestra.console.smithi114:pexpect command: ipmitool -H smithi114.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power off 2024-04-23T10:47:12.855 DEBUG:teuthology.orchestra.console.smithi114:power off output: Chassis Power Control: Down/Off 2024-04-23T10:47:12.855 DEBUG:teuthology.orchestra.console.smithi114:pexpect command: ipmitool -H smithi114.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-04-23T10:47:12.869 INFO:teuthology.orchestra.console.smithi119:Power off 2024-04-23T10:47:12.870 DEBUG:teuthology.orchestra.console.smithi119:pexpect command: ipmitool -H smithi119.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power off 2024-04-23T10:47:12.884 DEBUG:teuthology.orchestra.console.smithi114:check power output: Chassis Power is on 2024-04-23T10:47:12.904 DEBUG:teuthology.orchestra.console.smithi119:power off output: Chassis Power Control: Down/Off 2024-04-23T10:47:12.904 DEBUG:teuthology.orchestra.console.smithi119:pexpect command: ipmitool -H smithi119.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-04-23T10:47:12.917 INFO:teuthology.orchestra.console.smithi204:Power off 2024-04-23T10:47:12.918 DEBUG:teuthology.orchestra.console.smithi204:pexpect command: ipmitool -H smithi204.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power off 2024-04-23T10:47:12.927 DEBUG:teuthology.orchestra.console.smithi119:check power output: Chassis Power is on 2024-04-23T10:47:12.940 DEBUG:teuthology.orchestra.console.smithi204:power off output: Chassis Power Control: Down/Off 2024-04-23T10:47:12.941 DEBUG:teuthology.orchestra.console.smithi204:pexpect command: ipmitool -H smithi204.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-04-23T10:47:12.965 DEBUG:teuthology.orchestra.console.smithi204:check power output: Chassis Power is on 2024-04-23T10:47:16.885 DEBUG:teuthology.orchestra.console.smithi114:pexpect command: ipmitool -H smithi114.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-04-23T10:47:16.929 DEBUG:teuthology.orchestra.console.smithi119:pexpect command: ipmitool -H smithi119.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-04-23T10:47:16.966 DEBUG:teuthology.orchestra.console.smithi204:pexpect command: ipmitool -H smithi204.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-04-23T10:47:16.995 DEBUG:teuthology.orchestra.console.smithi114:check power output: Chassis Power is on 2024-04-23T10:47:17.037 DEBUG:teuthology.orchestra.console.smithi119:check power output: Chassis Power is on 2024-04-23T10:47:17.075 DEBUG:teuthology.orchestra.console.smithi204:check power output: Chassis Power is on 2024-04-23T10:47:20.996 DEBUG:teuthology.orchestra.console.smithi114:pexpect command: ipmitool -H smithi114.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-04-23T10:47:21.038 DEBUG:teuthology.orchestra.console.smithi119:pexpect command: ipmitool -H smithi119.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-04-23T10:47:21.076 DEBUG:teuthology.orchestra.console.smithi204:pexpect command: ipmitool -H smithi204.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-04-23T10:47:21.105 DEBUG:teuthology.orchestra.console.smithi114:check power output: Chassis Power is off 2024-04-23T10:47:21.147 DEBUG:teuthology.orchestra.console.smithi119:check power output: Chassis Power is off 2024-04-23T10:47:21.185 DEBUG:teuthology.orchestra.console.smithi204:check power output: Chassis Power is on 2024-04-23T10:47:21.206 INFO:teuthology.orchestra.console.smithi114:Power off completed 2024-04-23T10:47:21.248 INFO:teuthology.orchestra.console.smithi119:Power off completed 2024-04-23T10:47:25.189 DEBUG:teuthology.orchestra.console.smithi204:pexpect command: ipmitool -H smithi204.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-04-23T10:47:25.303 DEBUG:teuthology.orchestra.console.smithi204:check power output: Chassis Power is off 2024-04-23T10:47:25.403 INFO:teuthology.orchestra.console.smithi204:Power off completed 2024-04-23T10:47:25.510 INFO:teuthology.run:Summary data: description: fs/workload/{0-centos_9.stream begin/{0-install 1-cephadm 2-logrotate 3-modules} clusters/1a11s-mds-1c-client-3node conf/{client mds mgr mon osd} mount/kclient/{base/{mount-syntax/{v1} mount overrides/{distro/stock/{centos_9.stream k-stock} ms-die-on-skipped}} ms_mode/crc wsync/no} objectstore-ec/bluestore-bitmap omap_limit/10000 overrides/{cephsqlite-timeout frag ignorelist_health ignorelist_wrongly_marked_down osd-asserts pg_health session_timeout} ranks/multi/{balancer/automatic export-check n/3 replication/always} standby-replay tasks/{0-subvolume/{with-no-extra-options} 1-check-counter 2-scrub/no 3-snaps/yes 4-flush/yes 5-quiesce/with-quiesce 6-workunit/suites/dbench}} duration: 2825.9543488025665 failure_reason: 'Command failed (workunit test suites/dbench.sh) on smithi114 with status 1: ''mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=0a6c3ed699031b80a2b419e7e795368719871394 TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 CEPH_MNT=/home/ubuntu/cephtest/mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clone.client.0/qa/workunits/suites/dbench.sh''' flavor: default owner: scheduled_leonidus@teuthology sentry_event: https://sentry.ceph.com/organizations/ceph/?query=0ffdb51de7e0431ca064be5e59f5473a status: fail success: false 2024-04-23T10:47:25.510 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/ 2024-04-23T10:47:25.588 INFO:teuthology.run:FAIL